diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,1398 +1,1026 @@ # CHANGELOG for crypton -## 2.0.1 - -* fix(cpu): stop reading Intel's SDBG bit as AMD's XOP, which crashed SHA-512 - and ChaCha20 on Broadwell and later. The vendored assembly dispatches on a - capability word this library fills, and reads bit 11 of its second dword as - XOP -- which is CPUID leaf 0x80000001 ECX bit 11, an AMD extended leaf. - crypton put the raw leaf 1 ECX there, whose bit 11 is SDBG, the silicon - debug interface, which Intel has reported since Broadwell. On such a - processor `sha512-x86_64.S` took `.Lxop_shortcut` and - `chacha-x86_64.S` took `.Lcrypton_chacha20_asm_4xop`, and the first - `vprotq` is an invalid opcode: SIGILL, for SHA-512 and SHA-384, and for - ChaCha20 and so ChaCha20-Poly1305. OpenSSL clears leaf 1's bit 11 before - merging the real flag in; crypton now clears it and leaves it clear, so the - XOP paths are never taken. Nothing is lost -- XOP ran from AMD's Bulldozer - to Excavator and Zen dropped it -- and it is the reason the AVX-512 bits - beside it are cleared as well. Reported by @lucasdicioccio, who - disassembled the trap - [#202](https://github.com/kazu-yamamoto/crypton/issues/202) - -## 2.0.0 - -* fix(docs): export the names the documentation already referred to. - `Crypto.Number.ModArithmetic` throws `CoprimesAssertionError` and - `ModulusAssertionError` from `inverseCoprimes` and `squareRoot`, and said so - in the haddock, without exporting either, so a caller could not name the - exception it was told to expect; `Crypto.PubKey.Rabin.Types.generatePrimes` - takes a `PrimeCondition` in its exported signature and that synonym was not - exported either. All three are now exported - [#195](https://github.com/kazu-yamamoto/crypton/pull/195) - -* Breaking change: fix(bcrypt): refuse a cost bcrypt does not have rather than - substituting one. A cost below 4 came back as a cost-10 hash and a cost - above 31 as a cost-31 one, with nothing said either way, so a caller asking - for something bcrypt does not do was answered with something else and had no - way to tell -- `hashPassword 3` and `hashPassword 10` returned the same - thing. Both ends are now reported as `CryptoError_ParameterInvalid`, which - is what every other KDF here already did for a refused parameter. - `hashPassword` can therefore fail where it could not before, so - `tryHashPassword` is added beside it; the salt it generates is always the - right length, so the cost is the only thing it can report - [#59](https://github.com/kazu-yamamoto/crypton/issues/59) - -* Breaking change: fix(poly1305): take a checked key, so that initializing - cannot fail. A Poly1305 key is thirty-two bytes and nothing else about it - can be wrong, so `initialize` returning a `CryptoFailable` put an error case - in front of every caller for a length most of them know is right -- and they - answered it with `throwCryptoError`, this library included: the one in - `Crypto.Cipher.ChaChaPoly1305` guarded a `B.take 32`, and `auth` did not - even do that, it called `error`. There is now a `Key` with `key` to build - one, the length is checked there, and `initialize :: Key -> State` and - `auth :: Key -> ba -> Auth` are total. A caller checks once and then - initializes as often as it likes with nothing to handle. `initialize k` - becomes `initialize <$> key k` where the length is unknown, and where it is - known the check moves to where the key is made. `Key` has no `Show`, as - key material should not - [#28](https://github.com/kazu-yamamoto/crypton/issues/28) - -* fix(pubkey): stop printing private keys. `Show` is what `print`, a message - built with `error`, an exception and a test framework's failure output all - reach for, so it is the instance a key travels on when nobody meant to send - it anywhere; the library already kept that promise for the secret keys held - in a `ScrubbedBytes`, and the documentation for `ScrubbedBytes` advertises - it, while ten other types printed theirs in full. Those ten -- - `RSA.PrivateKey` and `RSA.KeyPair`, `DSA.PrivateKey` and `DSA.KeyPair`, - `ECDSA.PrivateKey` and `ECDSA.KeyPair`, `DH.PrivateNumber`, and the - `PrivateKey` of `Rabin.Basic`, `Rabin.Modified` and `Rabin.RW` -- now render - the public part and `` for the rest. Nothing else about them - changes: `Read`, `Eq`, `Data`, `Generic` and `NFData` are all still derived. - The new `Crypto.Debug` exports a class `DebugShow` whose `debugShow` returns - exactly what the derived `Show` used to return, for those ten and for the - five `ScrubbedBytes` secret keys as well, which never had a `Show` that - spoke. **Code that serialized a key through `show` has to say `debugShow` - instead**; `read (debugShow k) == k` still holds, but `read` given the - output of `show` will now fail at run time, which the compiler cannot point - at - [#72](https://github.com/kazu-yamamoto/crypton/issues/72) - -* deprecate(ecc): the curves over a binary field. They are obsolete, they are - the curves a cofactor makes delicate, and pyca/cryptography deprecated them - for removal in the release that fixed CVE-2026-26007. A `DEPRECATED` pragma - now covers the eighteen `SEC_t*` constructors of `CurveName` and the - eighteen types of the same names in `Crypto.ECC.Simple.Types`; nothing is - removed, so the only effect is a warning where one of them is named, and - they will go in a later major version. The prime curves with a cofactor, - `SEC_p112r2` and `SEC_p128r2`, are not deprecated: the check above covers - them - [#66](https://github.com/kazu-yamamoto/crypton/issues/66) - -* fix(ecc): refuse a public point outside the prime-order subgroup. A point - that satisfies the curve equation is not necessarily in the subgroup the - base point generates; the two coincide only where the cofactor is 1. Of the - curves in `CurveName` twenty have a cofactor -- the eighteen binary ones, - and `SEC_p112r2` and `SEC_p128r2`, whose cofactor is 4 -- and on those the - other party could offer a point of small order, at which point the value - that came back depended on our private number only through its residue - modulo that order, and offering it and watching the answer handed them those - bits. This is the flaw pyca/cryptography fixed as CVE-2026-26007; what is - fixed here is the same one, found by following that report. - `Crypto.PubKey.ECC.DH.getShared` and `tryGetShared`, and - `Crypto.ECC.Simple.Prim.pointFromIntegers`, now require the point to be in - the subgroup and report `CryptoError_PointSubgroupInvalid` when it is not. - The check is `isPointInSubgroup`, newly exported from both prim modules: - where the cofactor is 1 it answers without work, and otherwise it multiplies - by the group order and requires the point at infinity, which is what - OpenSSL's `EC_KEY_check_key` does and costs one further scalar - multiplication -- an exchange on an affected curve is about twice the price, - and one on every other curve is unchanged. The typed `Crypto.ECC` interface - offers only cofactor-1 curves and dedicated implementations, so nothing - reaching elliptic curves through it, `tls` among them, was affected - -* perf(p256): inline the field arithmetic on AArch64. `felem_mul` and - `felem_square` end in `felem_reduce_degree`, a carry chain the whole width - of the number, and that chain is what their latency is: one product feeding - the next costs 18.1 ns on an Apple M4, while four independent ones cost 11.4 - ns each. The curve arithmetic has independent products to offer -- the two - squarings that open a point doubling, the multiplication and the squaring - that close it -- but only if the compiler inlines the reduction instead of - calling it, since a call is a fence. Plain `inline` does not change its - mind; `always_inline` does, and it is worth asking where there are registers - to hold two carry chains at once and not where there are not: 1.23x on an - M4, 1.12x and 1.05x on a Neoverse under clang and gcc, and 0.95x and 0.82x - on x86-64, whose fifteen general-purpose registers are not enough. So it is - gated on the architecture and x86-64 is left byte-identical. ECDH P-256 on - an M4: 69.16 to 56.24 us, against openssl's 24.68, so 0.36 becomes 0.44; - ECDSA P-256 signing and verification move with it. The cost is code, 30 to - 116 kilobytes of it - [#188](https://github.com/kazu-yamamoto/crypton/pull/188) -* perf(number): count bytes from the bit count, not from base 256. `numBytes` - asked GMP how many base-256 digits a number has, and GHC's bignum answers - that by dividing the number down to nothing, one digit at a time, where the - same question in base two is a look at the highest limb. On a 2048-bit - `Integer` that is 1.65 us against 0.01. Every serialization here asks for - the size before it allocates and `i2ospOf` asks twice, so the cost landed on - every RSA, DSA and DH operation leaving the `Integer` world: `i2ospOf_` at - 256 bytes goes from 2.87 to 0.09 us and RSA-2048 verification from 18.4 to - 15.5 us on an M4. Signing moves by a percent; it is two exponentiations and - hardly touches this - [#187](https://github.com/kazu-yamamoto/crypton/pull/187) -* perf(ecc): stop sharing the doublings in the double multiplication. - `pointAddTwoMuls` was Shamir's trick, one pass over the bits of both scalars - at once in `Integer` arithmetic, which is the right trade when the two - multiplications would cost the same. They have not for a while: `pointMul` - goes to C, and over a prime field it multiplies the base point through a - table of its multiples at about a third of the price -- and the base point - is one of the two, since ECDSA verification is the only caller. Doing them - separately and adding: ECDSA P-384 verification 1397 to 698 us on an M4 in - the typed API, 9839 to 738 in the older one, and a curve over a binary field - 641 ms to 2.6. P-256 keeps the double multiplication it has in C - [#186](https://github.com/kazu-yamamoto/crypton/pull/186) -* perf(sha3): take the CRYPTOGAMS Keccak for x86-64 as well. The same module - as [#181](https://github.com/kazu-yamamoto/crypton/pull/181) on the other - architecture, and the reason it was not taken at the time was a measurement - taken on the wrong machine: crypton on Apple silicon against openssl on - x86-64, which said there was nothing to gain. Measured on one machine there - was: SHA3-256 on an EPYC 7763 goes from 109 to 421 MB/s, against openssl's - 426, so 0.26 becomes 0.99 - [#184](https://github.com/kazu-yamamoto/crypton/pull/184) -* perf(sha1): take the CRYPTOGAMS SHA-1 for AArch64. The instructions are the - ones [#170](https://github.com/kazu-yamamoto/crypton/pull/170) put in, and - the arrangement is what the module has over them: the message schedule of - the next four rounds runs against the rounds of this one, which is not - something a C function is going to be made to do -- - [#179](https://github.com/kazu-yamamoto/crypton/pull/179) tried the one - thing C can do here, handing over a run of blocks, and on this processor it - measured nothing at all. The entry point for processors that have the - instructions is not exported, so the module's own dispatch picks it and the - answer to the runtime question goes into the word that dispatch reads. On - Apple silicon: 3155 to 3379 MB/s at 16 KiB, against openssl's 3350 on the - same machine, so where this was at 0.94 it is now a shade ahead. The - intrinsics stay for the block a message ends with, and for any processor - that has the instructions but is built without the assembly - [#182](https://github.com/kazu-yamamoto/crypton/pull/182) -* perf(sha3): take the CRYPTOGAMS Keccak for AArch64. The instructions are - the ones [#171](https://github.com/kazu-yamamoto/crypton/pull/171) put in -- - EOR3, RAX1, XAR and BCAX -- and what this module does with them is take a - run of blocks rather than one at a time, and schedule the round it is in - against the next one. The absorb loop hands over the whole run, which also - drops the alignment trampoline on that path: the assembly reads the message - as bytes. SHA3-256 on Apple silicon: 1002 to 1104 MB/s at 16 KiB, against - openssl's 1058 on the same machine. Only the absorb side is handed over; - the squeeze, which SHAKE uses to produce output, is entangled with this - side's buffer bookkeeping and is not where the time goes - [#181](https://github.com/kazu-yamamoto/crypton/pull/181) -* perf(xts): double the XTS tweak in the integer registers. The tweak - advances by doubling in GF(2^128) once per block, and it was doing that in a - vector register: six operations on the same units that are running the - rounds and the exclusive ors, in a chain where each waits for the one - before. On a processor whose AES is fast that is not a detail -- taking the - doubling out of a diagnostic build, which gives the wrong answer but says - where the time goes, left XTS running at the speed of ECB. It costs three - integer operations instead, and the integer units have nothing else to do - here; what crosses over is one move per block. On x86-64 that also gets - eight values out of a register file with sixteen entries, so the round keys - stay where they were. AES-128-XTS at 16 KiB: 9644 to 18646 MB/s on Apple - silicon and 4504 to 7660 on a Haswell-generation x86-64, against openssl's - 17382 and 6997 on the same machines, so both are now a little ahead where - they were at 0.55 and 0.64. No assembly: the AArch64 module in CRYPTOGAMS - has no XTS, and the x86-64 one's is inside a module this does not otherwise - want - [#180](https://github.com/kazu-yamamoto/crypton/pull/180) -* perf(sha1): hand the SHA-1 block loop a run of blocks rather than one at a - time. A block at a time means the state goes out to memory and comes back - either side of every block, with the two shuffles that put it in the order - the instructions want; against the hundred-odd cycles a block costs with the - SHA extensions that is most of what stood between this and openssl. On an - EPYC 7763: 1364 to 1677 MB/s, against openssl's 1670 on the same machine, - and on a Xeon 8370C 1506 to 1619. On Apple silicon it measures nothing at - all -- that processor hides the cost -- and is kept there only so the two - paths have one shape. The intended file for this was CRYPTOGAMS' - `sha1-x86_64.pl`, which turns out to be the 2006 scalar implementation: no - SSSE3, no AVX, no SHA extensions. Processors without the extensions are - therefore where they were, 664 MB/s against openssl's 791 on a Haswell - [#179](https://github.com/kazu-yamamoto/crypton/pull/179) -* perf(sha2): take the CRYPTOGAMS SHA-256 and SHA-512 for x86-64. One - generator gives both, as on AArch64, and each dispatches on what the - processor has: the SHA extensions, AVX2, AVX, SSSE3 or plain integer code. - That replaces everything written here for x86-64 -- `sha256_x86.c` and - `sha512_x86.c` go -- since it is ahead of all of it either way. At 16 KiB: - on an EPYC 7763, SHA-256 1430 to 1584 MB/s and SHA-512 423 to 769; on a - Haswell-generation part, SHA-256 318 to 379 and SHA-512 488 to 593, where - openssl reports 378 and 589. The block loops hand over the whole run of - blocks rather than one at a time, and the alignment trampoline goes with it. - This also fixes a bug in the capability word - [#176](https://github.com/kazu-yamamoto/crypton/pull/176) added: bit 29 of - leaf 7 EBX is the SHA extensions, not an AVX-512 bit, and was being cleared - along with them -- which cost the SHA-256 assembly two thirds of its speed - on a processor that has them, and which no machine here could have shown, - since none has them - [#178](https://github.com/kazu-yamamoto/crypton/pull/178) -* perf(chacha): take the CRYPTOGAMS ChaCha20 for x86-64 as well. The C here - vectorises from eight blocks up and takes anything shorter one block at a - time, so a message of a few hundred bytes -- a QUIC packet, a small TLS - record -- ran at a fifth of the bulk rate. The module has vector code for - those lengths and is a few per cent ahead in bulk besides: 494 to 1091 MB/s - at 256 bytes, 477 to 701 at 128, and 2201 to 2374 at 16 KiB, which is - openssl's 2389 on the same machine. It is handed everything from one block - up, where the AArch64 module is handed nothing below three, that one's - scalar path measuring level with the C. Keystream generation is still the - C on both, having no input to exclusive-or - [#177](https://github.com/kazu-yamamoto/crypton/pull/177) -* perf(poly1305): take the CRYPTOGAMS Poly1305 for x86-64 as well. The same - module for the other architecture, through the same three functions, so what - this adds is the capability word: where the AArch64 one reads - `crypton_armcap_P`, this one reads `crypton_ia32cap_P`, which is cpuid's own - words in the order OpenSSL keeps them, filled with the bits for anything the - operating system will not preserve cleared. What it brings over the AVX2 - written here is a hand-scheduled scalar path, which is what a message of a - few hundred bytes actually uses, and an AVX path for machines with no AVX2: - on a Haswell-generation x86-64, 4298 to 5345 MB/s at 16 KiB and 556 to 1573 - at 64 bytes, against the roughly 5270 openssl reaches there. `poly1305_avx2.c` - goes the way the NEON did. The module's AVX-512 paths are not taken: the - generator chooses what to emit from the version of the assembler it is told - about, and it is now told one that predates them, no machine here being able - to run them and an assembler still in use being unable to assemble them. - Pinning that version also makes the checked-in assembly independent of the - host that produced it - [#176](https://github.com/kazu-yamamoto/crypton/pull/176) -* perf(sha256): take the CRYPTOGAMS SHA-256 for AArch64. The instructions are - the ones the intrinsics here already use; what the module does with them is - schedule them across a whole run of blocks rather than one at a time, and - keep the message schedule of the next block moving while the rounds of this - one are still going, which a function that is handed one block and returns - cannot do whatever it is written in. So the block loop hands over the whole - run, which also drops the alignment trampoline on this path -- the assembly - reads the message as bytes and wants neither the alignment nor the copy. On - Apple silicon: 2637 to 3279 MB/s at 16 KiB, against openssl's 3323 on the - same machine, and 1576 to 1966 at 64 bytes. SHA-512, which the same - generator emits, is not taken: 1876 here against openssl's 1880, the - ARMv8.2 instructions for it having gone in with #110 - [#175](https://github.com/kazu-yamamoto/crypton/pull/175) -* perf(poly1305): take the CRYPTOGAMS Poly1305 for AArch64. One - multiplication modulo 2^130 - 5 depends on the one before it, so what there - is to win is in how the multiplies and the carries are laid against each - other, and in keeping the accumulator in whichever base costs less: the - module works in base 2^64 while the message is short and switches to base - 2^26 for the four-way vector loop, deciding that for itself. Unlike the - other two it is the whole of the arithmetic rather than a bulk loop bolted - to the side, so the context now holds either the 26-bit limbs the C works in - or the 192 bytes the assembly keeps, as a union, and grows from 84 bytes to - 232. On Apple silicon: 4269 to 8060 MB/s at 16 KiB and 1542 to 4355 at 64 - bytes, and ChaCha20-Poly1305 together, which is what this is for, 1416 to - 2284 against openssl's 2180 on the same machine. `poly1305_neon.c`, which - [#169](https://github.com/kazu-yamamoto/crypton/pull/169) added, goes: the - assembly is faster at every length on every target that gets it, and the - scalar C remains for the targets that do not. The tests came first and - found that the chunking property here had been testing nothing -- it used - the all-zero key, whose r is zero, so both sides were the nonce whatever - they did, which is how it came to feed the chunks to `update` in reverse - order and pass - [#174](https://github.com/kazu-yamamoto/crypton/pull/174) -* perf(chacha): take the CRYPTOGAMS ChaCha20 for AArch64. The vector - registers hold four ChaCha states and there is no room for a fifth, so once - four blocks are in flight the only place further parallelism can come from - is the integer side: that module runs a fifth block through the general - registers alongside four in the vector ones, and above 512 bytes two - alongside six. Which register holds which word is the whole of the trick - and C has no way to say it, which is why the intrinsics here sat at about - 0.63 of what openssl gets out of this very file. On Apple silicon, a - message per call: 1911 to 3069 MB/s at 512 bytes, 1913 to 3093 at 4 KiB and - 2056 to 3112 at 64 KiB, against openssl 3.6's 3164 on the same machine, - which is this code. It is handed only the states it fits -- twenty rounds, - a 256-bit key, and as many blocks as the 32-bit counter has room for, since - crypton's counter is 64 bits wide and carries where the assembly wraps -- - and nothing below 192 bytes, where its own vector path starts. The tests - came first: the properties here generated one shape of state, so the - 256-bit constants were never exercised by them - [#173](https://github.com/kazu-yamamoto/crypton/pull/173) -* perf(gcm): take the CRYPTOGAMS stitched AES-GCM for x86-64, which is the - first assembly in the package. Counter-mode AES and GHASH do not compete - for the same execution ports, so a loop that interleaves them at - instruction granularity runs both in about the time the rounds alone take; - written in C that interleaving does not survive the compiler, which sinks - every multiply to the end of the group, and the disassembly of what - [#160](https://github.com/kazu-yamamoto/crypton/pull/160) produced says so. - On a Haswell-generation x86-64, a message per call, AES-128-GCM: 2672 to - 3172 MB/s at 1152 bytes, 3394 to 4271 at 4 KiB and 3657 to 5110 at 16 KiB, - where openssl speed on the same machine reports 4896; decryption within a - couple of points of that, and AES-256-GCM 3147 to 4297 at 16 KiB against - openssl's 4206. `cbits/asm` holds the module, the translator it needs and the - generated assembly, one file per object format, so that building needs no - perl; `cbits/asm/README.md` records where it came from and what was done to - it, which is to rename the entry points, a program linking both crypton - and openssl being entitled to object to two definitions of - `aesni_gcm_encrypt`. What the assembly reads is laid out OpenSSL's way and - is built per message in `cbits/aes/gcm_x86_asm.c`, the powers of H being - the ones crypton already has, shifted up a bit. Short messages are not - handed over at all. `cabal-version` is now 3.0, for `asm-sources` - [#172](https://github.com/kazu-yamamoto/crypton/pull/172) -* perf(sha3): use the ARMv8.2 SHA-3 instructions. Keccak was the plain C - everywhere, a round at a time over tables of rotation amounts and lane - positions, at half of what openssl manages on the same machine. EOR3, - RAX1, XAR and BCAX exist for exactly this permutation and take a round from - around a hundred and fifty operations to sixty-six; they come with the - SHA-512 extension the tree already asks for. Rho and pi move one lane of - every row into every other row, so the round cannot be done in place, and - four rounds go in an iteration, which is worth a fifth over one. SHA3-256 - 551 to 991 MB/s (openssl 1064), SHAKE128 700 to 1166, Keccak-256 559 to - 944. The body is generated from the definitions in FIPS 202 rather than - copied in, and the script that worked out the rotations and the lane - permutation checked itself against the published digests of the empty - string and of "abc" before emitting any C, which is how a first attempt - with chi reading lanes another row had already overwritten was caught. x86 - is untouched: nothing there has instructions for this - [#171](https://github.com/kazu-yamamoto/crypton/pull/171) -* perf(sha1): use the ARMv8 SHA-1 instructions. The AArch64 paths for - SHA-256 and SHA-512 went in with #104 and #110 and x86 got its SHA-1 - instructions in [#165](https://github.com/kazu-yamamoto/crypton/pull/165), - but the AArch64 SHA-1 ones were never used -- and they are part of the same - optional feature as the SHA-256 ones, so every processor that has those has - these. SHA1C, SHA1P and SHA1M each do four rounds with one of the three - round functions, SHA1H carries E from one group to the next, and SHA1SU0 - and SHA1SU1 do the message schedule between them. On Apple silicon: 1272 - to 3180 MB/s, against openssl 3.6's 3350 on the same machine. Checked - against the hardware rather than through an emulation of the instructions, - the machine here having them: the digests agree with the generic - implementation over every message length from 0 to 2000, with each input - split in two updates - [#170](https://github.com/kazu-yamamoto/crypton/pull/170) -* perf(poly1305): four blocks at a time with NEON. AArch64 had only the - scalar loop, whose five 26-bit limbs and 32-bit multiplies are the shape a - 32-bit machine wants. This is the arithmetic of the AVX2 path in NEON, - written as a transliteration of that file rather than a fresh formulation, - since the maths there is already pinned by the known-answer tests; what - differs is the width, AVX2 holding four 64-bit products in a register where - NEON holds two, so each product becomes a pair and the limbs are packed - back into four 32-bit lanes before the next multiply. On Apple silicon: - Poly1305 2783 to 4840 MB/s, and ChaCha20-Poly1305 together 1164 to 1368. - Checked against the scalar implementation over forty keys and every message - length from 0 to 400, with each input split in two updates. Also measured - and left alone: BLAKE2b at 1612 MB/s against openssl's 1378, the reference - C being the faster of the two - [#169](https://github.com/kazu-yamamoto/crypton/pull/169) -* perf(modes): stop the generic cipher modes allocating per byte. Counter - mode with a cipher whose modes are not in C ran at a third of what the same - cipher managed in ECB, and at an eighth for Blowfish, for two reasons - outside the cipher. The counters were built one at a time by `ivAdd`, - which allocates a block and walks the whole width of the counter from the - original for each of them; they are now one buffer filled in place. And - the exclusive or was `Data.ByteArray`'s, which walks a byte at a time - through an IO applicative -- 420 MB of heap for 8 MiB of counter mode, - against 17 MB for the same data through ECB, which is fifty bytes allocated - per byte produced and cost more than the cipher did. There is a - `crypton_memxor` to call instead, a pass of words, which the modes and CMAC - use. The serial modes also took each block as a copy and take shared - slices now. On Apple silicon, counter mode: Camellia-128 79.7 to 285.9 - MB/s, Blowfish 60.4 to 282.6, DES 46.2 to 114.9, CAST5 40.2 to 86.6, - Twofish-128 37.4 to 57.6, 3DES 25.2 to 36.3, and CBC and CMAC by a third to - a half as much again. AES is unchanged: its modes are in C and never came - this way - [#168](https://github.com/kazu-yamamoto/crypton/pull/168) -* build: compile the C at -O3, which is what came of looking at P-256 against - openssl. The comparison in the problem list was wrong -- a base point - multiplication here against openssl's ECDH, which is a variable point one -- - and measured properly P-256 is 2.8 to 3.3 times slower rather than the 1.27 - claimed. The time is in the field arithmetic, five 51-bit limbs in - Montgomery form at 44.4 ns a multiplication, against hand-written assembly - using `mulx`, `adcx` and `adox`; a four-limb saturated Montgomery - multiplication written in C to see what a compiler would give measured 41.3 - ns, so that is not the way in. What did move is the optimisation level GHC - passes: a P-256 base point multiplication goes from 71.0 to 59.8 us on x86-64 - and 26.0 to 24.3 on Apple silicon, AES-128-GCM from 3455 to 3708 MB/s and - AES-128-OCB from 2187 to 2484, with ChaCha20, Poly1305, SHA-1 and MD5 within - a couple of per cent either way. The masked selections in the curve and - field code compile to no conditional jumps at either level - [#167](https://github.com/kazu-yamamoto/crypton/pull/167) -* refactor(aes): drop the keystream generator nobody can call. `genCTR` and - `genCounter` are exported from a module in `other-modules`, so nothing - outside the library could reach them and nothing inside used them; the only - mention left was a test commented out since the cryptonite days. They were - also the slowest thing in the file, a block at a time through the - single-block entry point at 715 MB/s where counter mode does 5788, and the - three ways of fixing that are each worse than removing them: counter mode - over zeros costs Apple silicon a fifth, counters through ECB costs both, and - a keystream loop written out per key size is eighty lines for an API no - caller can see. Also declares `crypton_aes_encrypt_ctr` and - `crypton_aes_encrypt_c32` in the header, which had them defined and imported - but never declared - [#166](https://github.com/kazu-yamamoto/crypton/pull/166) -* perf(sha1): use the Intel SHA extensions on x86-64. The extension that - carries the SHA-256 instructions carries four for SHA-1 as well, and the same - cpuid bit answers for both, so this is one file and one branch. On an AMD - EPYC 7763: 725.1 to 1363.8 MB/s, against openssl's 1668.2 on the same - machine. 1.9x, where the SHA-256 instructions were worth 4.8x -- SHA-1's - rounds are cheaper to begin with, so there is less for an instruction to - replace. The sequence was checked by replacing the four instructions with C - that follows the SDM and comparing against the generic implementation over - every length from 0 to 1024, which found the same missing schedule step - [#155](https://github.com/kazu-yamamoto/crypton/pull/155) had - [#165](https://github.com/kazu-yamamoto/crypton/pull/165) -* docs(sidechannel): say what the modules that still work in `Integer` keep - from the clock, and fix the two places where something could be done about - it. ElGamal inverted the shared secret with the extended Euclidean - algorithm, whose steps follow the bits it is given -- the modulus is prime, - so Fermat reaches it. Its signing inverts the ephemeral value modulo an even - number, where Fermat does not reach, so `sign` blinds instead: the algorithm - is handed that value times a fresh random unit and the blinder divided out - afterwards. What is left is written down rather than fixed -- the Jacobi - symbols Rabin takes modulo its private primes, and the cost of `Integer` - arithmetic following the size of the numbers -- and `Crypto.Cipher.AES` now - says which implementation a machine gets and that the fallback, being - table-driven, is not constant time - [#164](https://github.com/kazu-yamamoto/crypton/pull/164) -* perf(poly1305): shorten the carry chain and stop the AVX2 loop spilling. The - carries go in pairs, since the two halves of that chain do not depend on each - other; the powers of r are read from memory, there being sixteen registers - and ten of them wanted for the accumulator and the products; and the message - is added limb by limb as the block comes apart rather than five limbs being - formed first. 4203 to 4452 MB/s, and ChaCha20-Poly1305 together from 1412 to - 1488. What is left is the instruction count: 107 per 64 bytes, of which 25 - are the multiply - [#163](https://github.com/kazu-yamamoto/crypton/pull/163) -* perf(chacha): combine as the keystream comes out of the registers. All three - vector implementations wrote it to a buffer on the stack and read it back to - exclusive-or it with the input, which is a pass over every byte for something - the registers were already holding. 2128 to 2230 MB/s on x86-64, and nothing - on Apple silicon, where the round trip was free. Measured while doing it: - the AVX2 path already did eight blocks at a time, and what is left of the gap - to openssl in this AEAD is Poly1305 rather than the cipher - [#162](https://github.com/kazu-yamamoto/crypton/pull/162) -* perf(sha): compute the message schedule in vector registers on x86. SHA-512 - has no instruction there and SHA-256 has none on a processor older than - Goldmont or Zen, which includes the Ice Lake and Cascade Lake server parts. - The rounds are a chain and stay where they are; the schedule is a quarter of - the work, comes out four words at a time and depends on nothing but the - message, so it goes into the vector registers and runs alongside rounds that - need the general ones. SHA-256 228 to 314 MB/s, SHA-512 354 to 480 - [#161](https://github.com/kazu-yamamoto/crypton/pull/161) -* perf(gcm): take the GHASH of the group before, alongside this group's rounds. - Held a group apart the multiply and the rounds run through each other, where - in step neither could start until the other finished. With it, the multiply - called directly rather than through a branch pointer the compiler cannot see - through, and the round keys read from memory rather than spilled: AES-128-GCM - 2797 to 3458 MB/s and AES-256-GCM 2458 to 3053. openssl does 4895 and 4205 - on the same machine; the rest of that is instruction-level interleaving, - which does not survive being written in intrinsics - [#160](https://github.com/kazu-yamamoto/crypton/pull/160) -* perf(ocb): drive OCB through the ECB paths a group at a time. It ran one - block at a time through the single-block entry point and so cost four times - what GCM costs, for a mode that does less work than GCM. The offsets have to - be worked out in order but the block cipher calls under them do not depend on - each other, so eight go through ECB together. OCB-128 1130 to 3500 MB/s on - Apple silicon and 694 to 2173 on x86-64, the authenticated data 1141 to 5900 - and 692 to 2526. CCM is unchanged and stays that way: what is left there is - CBC-MAC, where each block waits for the one before it - [#159](https://github.com/kazu-yamamoto/crypton/pull/159) -* test(aes): run the XTS vectors, and add OCB and CCM at 192 and 256 bits. The - XTS known-answer tests never ran: the call was commented out and the test it - would have called did not compile, so vectors at both key sizes sat in the - tree unused. XTS is defined only for a 128-bit block, which the general KAT - runner cannot promise, so it gains a counterpart for a cipher that can. OCB - and CCM had vectors at 128 bits only. 2613 examples to 2679 - [#158](https://github.com/kazu-yamamoto/crypton/pull/158) -* perf(aes): build the AArch64 key schedule with the instructions rather than - the S-box table. The AArch64 path expanded a key by calling the generic - implementation and then inverting the round keys, so every schedule went - through sixteen lookups at addresses derived from the key -- a small thing - next to the per-block indexing the extensions exist to remove, but a key - schedule is what an attacker most wants out of a cache, and x86 has never - needed the table. AArch64 has no counterpart to AESKEYGENASSIST, but AESE - against a zero key is SubBytes and ShiftRows, and a word given to it in all - four columns comes back as SubWord in each of them. The words stay in - vector registers throughout, which is what makes it free: moving each one to - a general register for the instruction and back cost more than the - instruction did, 87 to 144 ns for an AES-128 schedule, where keeping them in - registers gives 81.4 - [#157](https://github.com/kazu-yamamoto/crypton/pull/157) -* perf(aes): AES-192 through the processor's AES instructions. Every 192-bit - slot in the branch table was left at the generic code, on x86 and on AArch64 - alike, so a 192-bit key got the table-driven software AES while 128 and 256 - got the instructions. It was 164 times slower for counter mode on the x86 - machine measured and 62 on Apple silicon, and it was also the only key size - whose data path indexes a table with bytes derived from the key -- a caller - who picks AES-192 over AES-128 for a wider margin was quietly given a weaker - one. Counter mode then GCM, before and after: Apple silicon 152.7 to 9452.0 - MB/s and 112.3 to 7049.3, x86-64 40.4 to 6635.1 and 39.9 to 2633.5. Both - implementations were already written once per key size, so this instantiates - them again at twelve rounds; x86 also needed the 192-bit schedule, which - does not fall into 128-bit pieces the way the other two do - [#156](https://github.com/kazu-yamamoto/crypton/pull/156) -* perf(sha256): use the Intel SHA extensions on x86-64, which is what issue - [#31](https://github.com/kazu-yamamoto/crypton/issues/31) reports -- SHA-256 - four to eight times slower than sha256sum and openssl, both of which use the - processor's instructions. AArch64 got its instructions in #104 and is at - parity with them; x86 had nothing. SHA256RNDS2 does two rounds at a time and - SHA256MSG1 and SHA256MSG2 help with the message schedule, so a block costs - four groups of sixteen instructions instead of sixty-four rounds of scalar - work. On an AMD EPYC 9V74: 338.3 to 1612.7 MB/s, against openssl's 1783.8 on - the same machine. The extensions arrived with Goldmont and Ice Lake at Intel - and with Zen at AMD, far later than AES-NI, so a processor without them is - ordinary rather than ancient: the code sits behind a target attribute and a - cpuid question, and the plain C stays for everything else - [#155](https://github.com/kazu-yamamoto/crypton/pull/155) -* perf(bcrypt): Blowfish, and the key setup bcrypt wraps it in, in C. bcrypt - is a cost parameter and a promise that the cost is paid, and what pays it is - the Blowfish key schedule; in Haskell that cost about twice what the usual - implementations charge, so a hash of a given length of time had to be asked - for with a lower cost than elsewhere. Cost 8 goes from 25.97 to 9.98 ms, - cost 10 from 102.01 to 39.79, cost 12 from 418.78 to 159.41, `bcrypt_pbkdf` - from 109.76 to 40.38, and Blowfish over 4 KiB from 0.05 to 0.01 -- at cost - 10 that is 39.8 ms against the 52 `htpasswd` takes on the same machine. The - Haskell cipher goes with it, so there is one implementation rather than two, - and nothing exposed changes - [#154](https://github.com/kazu-yamamoto/crypton/pull/154) -* perf(prime): fewer Miller-Rabin rounds for a candidate nobody chose. A - number handed over may have been built to pass, and against that the only - thing to go on is that a round catches three quarters of the composites - there are, so `isProbablyPrime`, `findPrimeFrom` and `findPrimeFromWith`, - which all take their number from the caller, keep their thirty rounds. A - candidate drawn here is the case Damgard, Landrock and Pomerance worked out - and Table 4.4 of the Handbook of Applied Cryptography tabulates: - `generatePrime` and `generateSafePrime` now use twice what it asks for one - chance in 2^80, capped at the thirty they had, which leaves the chance far - under one in 2^100 at every size. With the candidates held fixed, - `generatePrime 1024` goes from 28.5 to 16.9 ms and an RSA-2048 key from 52.9 - to 39.2 - [#153](https://github.com/kazu-yamamoto/crypton/pull/153) -* fix(rsa): work the private exponent out without the extended Euclidean - algorithm. The modulus is the secret there, so multiplying the value by a - random number hides nothing; what does is that `e` is public. Whatever `d` - is, `e * d = 1 + k * phi` for some `k` under `e`, and reading that modulo - `e` gives `k` as an inverse modulo a number of a handful of bits, which for - a prime `e` is Fermat; `d` is then an exact division. What phi touches is a - remainder and a division, and nothing in either follows it - [#152](https://github.com/kazu-yamamoto/crypton/pull/152) -* perf(f2m): ask aarch64 for its carry-less multiply as well. #148 used PMULL - only where the compiler had been told the machine has the crypto - extensions, which is so on Apple and not on a Linux built for the bare - ARMv8 baseline, though every processor that runs such a build has it. It is - now compiled behind an attribute and the machine asked at run time, through - the auxiliary vector on Linux and Android, elf_aux_info on FreeBSD and a - sysctl on Apple: sect283k1 421.2 to 168.9 us there, sect571r1 2289.2 to - 579.9 - [#151](https://github.com/kazu-yamamoto/crypton/pull/151) -* refactor(ecc): one multiplication for both of the curve APIs, and one place - for each buffer's size. Which path a point multiplication takes was written - out twice, and the copy in `Crypto.ECC.Simple.Prim` cannot be reached from - outside the library on a curve over a binary field, so the suite never ran - it; it moves to the internal module both already share, which makes the copy - nobody can call the same code everybody runs. The two buffers for a C call - that still had their size written out separately from the offsets into them - now take both from one list, as the one that was wrong in #141 does -- the - note there records that neither valgrind nor the debug RTS catches that - mistake, both having been tried - [#150](https://github.com/kazu-yamamoto/crypton/pull/150) -* perf(f2m): use the x86 carry-less multiply where the processor has it. - PCLMULQDQ is not part of the x86-64 baseline, so the cpuid the package - already runs for AES-NI reports one more bit and the multiplication that - uses the instruction sits behind an attribute. Measured through Rosetta, - which translates rather than runs it, so the ratio is what to read: - sect283k1 560.4 to 177.5 us, sect571r1 3049.6 to 623.9 - [#149](https://github.com/kazu-yamamoto/crypton/pull/149) -* perf(f2m): do the binary field arithmetic in C. The ladder of #142 spent - nearly all its time on one thing -- a carry-less multiplication, which - ordinary arithmetic does not give and which in Haskell was `Integer` shifts - and exclusive ors, about 4 us for a 283-bit multiplication. The field and - the ladder over it are now C, with the processor's instruction where there - is one and four interleaved groups of bits where there is not, folding for - the reduction and Fermat for the inverse. sect163k1 3431 to 67.4 us with - the instruction and 117.3 without, sect283k1 10181 to 157.6 and 386.2, - sect571r1 40469 to 521.6 and 2101.0. It is also constant time, which the - Haskell ladder was not - [#148](https://github.com/kazu-yamamoto/crypton/pull/148) -* perf(bignum): start the doubling for `R^2 mod m` at the highest power of two - under the modulus rather than at one, which for a modulus that fills its - limbs is half the steps. Two to three percent of a curve operation, and - every curve operation and every `expSafe` pays for it once. Folding instead - of Montgomery for the primes shaped `2^k - c` was written and measured - alongside it and is not here: it is slower in this representation, 87.8 ns - against 76.8 for a 521-bit multiplication, because the shift down by `k` - costs more than the reduction pass it replaces when `k` does not land on a - limb boundary - [#147](https://github.com/kazu-yamamoto/crypton/pull/147) -* perf(ecc): keep a table of the multiples of each curve's base point, which - is the point signing and making a key multiply and the only one worth a - table. A multiplication with it is one addition per four bits and no - doublings: secp256k1 211.1 to 59.8 us, secp384r1 519.3 to 144.2, secp521r1 - 1047.7 to 283.0, and ECDSA P-384 signing 556.4 to 179.9 on both elliptic - curve APIs, which share the table. A table is built when a curve is first - asked for one -- 2.8 ms for secp256k1, 5.5 for secp384r1, 10.6 for secp521r1 - -- and is 221 KB and 456 KB for the last two, so it pays for itself after - about fifteen multiplications - [#146](https://github.com/kazu-yamamoto/crypton/pull/146) -* perf(bignum): take the limbs four and two at a time as well as eight in the - loop every modular multiplication is built out of. Four and six limbs, which - is what most of the curves want, fell entirely to the one-at-a-time tail - before: a field multiplication at six limbs goes from about 58 to 49 ns, - secp384r1 scalar multiplication from 596.7 to 519.3 us and ECDSA P-384 - signing from 645.0 to 556.4. Specialising the sizes further, which is what a - generated implementation would do, measures about 4% more and is not here - [#145](https://github.com/kazu-yamamoto/crypton/pull/145) -* fix(rsa): keep the blinding factor out of the extended Euclidean algorithm. - The blinder is a random number and its inverse, and the inverse went through - an algorithm whose steps follow the number handed to it -- the number the - blinding rests on, and unlike the other inverses this one is worked out once - per operation rather than once per key. `n` being composite leaves no - Fermat to fall back on, so the algorithm is handed the factor multiplied by - sixteen fresh random bytes and its answer multiplied by them again, which - leaves the inverse wanted and shows the algorithm nothing to do with it. In - IO, where every draw of randomness goes to the system, `generateBlinder` - goes from 74 to about 120 us and a PKCS#1 v1.5 `signSafer` from 719 to about - 765; under a DRG the caller carries, 24.7 to 24.9 - [#144](https://github.com/kazu-yamamoto/crypton/pull/144) -* fix(rsa): work `qinv` out without the extended Euclidean algorithm. Making - a key inverts one prime modulo the other and both of them are the key - itself, so that inverse is now Fermat's little theorem through `expSafe`, - which the other prime being prime allows: 308.6 us against 10.1, on a key - that takes tens of milliseconds to make. Making a key cannot be constant - time -- the search for the primes takes as long as it takes -- but what that - leaks is about the search rather than about the primes it settles on, and - the haddock now says which is which - [#143](https://github.com/kazu-yamamoto/crypton/pull/143) -* perf(ecc): a ladder for the curves over a binary field. These were the last - multiplication whose cost followed the scalar: an affine double-and-add, one - addition for every bit that was set and none for the others, which on - sect283k1 ran from 9665 us for a scalar with two bits set to 17958 for one - with 270. It is now Montgomery's ladder, which carries the multiples of two - consecutive numbers -- their difference being the point is what lets it - carry only their x coordinates -- and spends one addition and one doubling - on every bit whichever way it goes, working the y out at the end from the - two x it is left with, so one division does for the whole multiplication - where the affine code had one per step. The multiplication is now flat, and - quicker: sect163k1 4428 to 3431 us, sect233r1 9304 to 6806, sect283k1 13797 - to 10181, sect409k1 30826 to 20584, sect571r1 61931 to 40469. Uniform is - not constant time -- these are `Integer` operations, whose cost follows the - values -- and the point with no x, which is its own negation, keeps the code - that was there - [#142](https://github.com/kazu-yamamoto/crypton/pull/142) -* perf(ecc): multiply points in C on curves over a prime field. P-256 has had - a C implementation all along; every other prime curve -- P-384, P-521, - secp256k1 and the rest -- multiplied points with `Integer` arithmetic, which - cannot be constant time, since what an `Integer` operation costs follows the - value it is given. The C walks four bits of scalar at a time, taking the - multiple to add from a table of sixteen that it reads by touching every - entry and keeping one with a mask, and its addition and doubling are the - complete formulas of Renes, Costello and Batina, which answer for every pair - of points with no case to choose between. A P-384 multiplication goes from - 1557 to 585 us and no longer follows the scalar, ECDSA P-384 signing from - 1700 to 636 us, P-521 from 1942 to 1123. Binary curves are unchanged, and a - point that is not on the curve keeps the answer the Haskell gives it - [#141](https://github.com/kazu-yamamoto/crypton/pull/141) -* fix(ecc): add at every bit in the prime-curve multiplication, which laziness - was skipping. The multiplication adds at every bit, set or not, so that its - cost follows the width of the curve's order rather than the scalar -- but - the addition was a binding only one branch of the following `if` used, so at - a bit that was not set it stayed a thunk and was never worked out. The cost - followed the number of bits set in the scalar, which is the nonce when - signing and the private key in ECDH: on P-384, 765.8 us for a scalar with - two bits set against 1671.5 for one with 383, in a straight line between. - Both copies of the multiplication had it, so both elliptic curve APIs were - affected on every prime curve but P-256 - [#140](https://github.com/kazu-yamamoto/crypton/pull/140) -* fix(ecdsa): keep the P-256 signature out of `Integer` arithmetic. The - scalar handed to the C implementation was reduced with `mod`, a division, - whose steps follow the number being divided -- the nonce when signing, the - private key in ECDH. Twice the order is more than 256 bits hold, so a - scalar that fits is brought under the order by one masked subtraction - instead. The second half of a signature, `kInv * (z + r * d)`, was - `Integer` arithmetic as well, and now goes through `scalarAdd` and - `scalarMul`, which on P-256 are the C implementation's fixed-width - arithmetic. What is left on that curve is the conversion between `Integer` - and fixed-width scalars, which is also what it costs: signing goes from 34.1 - to 39.3 us, and ECDH and the other curves are unchanged - [#139](https://github.com/kazu-yamamoto/crypton/pull/139) -* fix(dsa,ecdsa): invert the signing nonce without a side channel. Both - inverted it with the extended Euclidean algorithm, whose step count and - branches follow the bits of what it is given -- and a handful of signatures - whose nonces are partly known give the private key away, so the nonce is - worth as much as the key. `Crypto.Number.ModArithmetic.inverseSafe` works - the inverse out with Fermat's little theorem through `expSafe` instead, - falling back on `inverse` when the modulus turns out not to be prime, so - every answer is the one it was. On P-256 the C implementation does it. - Signing costs a little more: ECDSA P-256 30.6 to 34.1 us, ECDSA P-384 678.7 - to 703.4, DSA-2048 422.5 to 437.1. Verification inverts a value that - arrives in the signature and is left alone - [#138](https://github.com/kazu-yamamoto/crypton/pull/138) -* perf(number): square, and multiply, faster in `expSafe`. The product and - the Montgomery reduction are now a full product followed by a reduction - rather than interleaved, built out of one loop that takes its limbs eight at - a time, and squaring works out only the products on one side of the diagonal - and doubles their sum. At 2048 bits the constant-time exponentiation goes - from 3.59 to 2.15 ms, which is 1.4x GMP's own rather than 2.4x; RSA-2048 - signing goes from 0.80 to 0.63 ms and DH-2048 `getShared` from 2.43 to 1.67 - [#137](https://github.com/kazu-yamamoto/crypton/pull/137) -* fix(number): make `expSafe` hide the exponent again. It asked integer-gmp - for `powModSecInteger` and fell back on the ordinary `powModInteger` when - that was missing; since integer-gmp 1.1 it is always missing, so on every - GHC this package supports `expSafe` was the same windowed exponentiation as - `expFast`, table indexed by the exponent's bits, for RSA, DSA, DH, ElGamal - and Rabin alike. It now goes to C: four bits of exponent at a time, the - table of sixteen read by touching every entry and keeping one with a mask, - and a Montgomery multiplication whose final subtraction is masked too. The - exponent's length is still visible, rounded up to a whole 64-bit word, which - is what GMP's own `mpz_powm_sec` lets slip. Hiding the exponent costs 1.6x - at 512 bits and 2.4x at 2048: RSA-2048 signing goes from 0.46 to 0.80 ms and - DH-2048 `getShared` from 1.04 to 2.43 - [#136](https://github.com/kazu-yamamoto/crypton/pull/136) -* perf(prime): stop running a Fermat test that Miller-Rabin subsumes. Every - candidate was tested to base 2 before the Miller-Rabin rounds, which begin - with the same base and prove more; the primes it passed paid for it twice - and the composites it caught were nearly all caught by trial division first. - RSA-2048 key generation goes from 55.0 to 32.8 ms - [#135](https://github.com/kazu-yamamoto/crypton/pull/135) -* perf(f2m): reduce the binary field by folding the top back in rather than - taking a step per bit of excess, square a byte at a time through a table of - the patterns a byte spreads into, and take four bits of a multiplier at a - time rather than one. On the 283-bit field, squaring goes from 5440 to 2068 - ns and multiplication from 7526 to 4086. A scalar multiplication there is - still affine, so it inverts once per addition, which is where its time now - goes - [#134](https://github.com/kazu-yamamoto/crypton/pull/134) -* perf(ecc): fold instead of dividing in the generic prime-curve arithmetic, - and add the point being multiplied as the affine point it is. These primes - are `2^k - c` with `c` far smaller, so the top half of a product folds back - in with a shift, a multiplication and an addition, where dividing costs four - times as much -- above 256 bits, below which the folding costs more than it - saves. P-521 scalar multiplication goes from 892 to 492 us and P-384 from - 684 to 572 - [#133](https://github.com/kazu-yamamoto/crypton/pull/133) -* perf(ecc): route P-256 through the C implementation the library already had. - `Crypto.PubKey.ECDSA` reached `cbits/p256`; `Crypto.PubKey.ECC.*`, the older - and more widely used API, never did. ECDSA signing goes from 590 to 28.7 us, - verification from 726 to 91.4, and `getShared` from 1177 to 96. On P-256 - that multiplication is now constant time, where the generic code branches on - the scalar at every bit - [#132](https://github.com/kazu-yamamoto/crypton/pull/132) -* Breaking change: perf(camellia): put Camellia in C, 40 to 321 MiB/s. The - round function ran a byte at a time in Haskell; generating the tables that - take a byte straight to its contribution gained 14%, and the rest was the - language. Input that is not a whole number of blocks now raises, where the - tail of the answer used to be uninitialised memory - [#131](https://github.com/kazu-yamamoto/crypton/pull/131) -* Breaking change: perf(twofish): walk the blocks once and carry them in words - rather than appending each result to what came before and going through lists - per block. 2 MiB goes from 0.14 to 56 MiB/s, and the rate no longer falls as - the message grows. Input that is not a whole number of blocks now raises, - where it used to come back longer than it went in - [#130](https://github.com/kazu-yamamoto/crypton/pull/130) -* perf(modes): cut the message without copying the rest of it in the generic - block cipher modes, which every cipher but AES uses, and hand whole slices to - the cipher in the modes whose blocks do not depend on one another. Camellia - in CBC goes from 1.8 to 22.9 MiB/s at 1 MiB, DES CBC decryption from 0.5 to - 83, and every figure is now flat in the message length where it used to fall - [#129](https://github.com/kazu-yamamoto/crypton/pull/129) -* Breaking change: perf(des): put DES in C. It was carried over lists of - `Bool`, one cons cell per bit, with the key schedule recomputed for every - block: 0.04 MiB/s, and 3DES 0.013, against 105 and 41 for OpenSSL. They are - now 112 and 37. Input that is not a whole number of blocks now raises, where - the tail of the answer used to be uninitialised memory - [#128](https://github.com/kazu-yamamoto/crypton/pull/128) -* perf(cmac): slice the message rather than copying what is left of it once per - block, and chain through CBC, which is what CMAC's chaining is. A MAC over - 4 MiB goes from 0.36 to 1628 MiB/s, which is the speed of AES-CBC itself - [#127](https://github.com/kazu-yamamoto/crypton/pull/127) -* fix(rabin): decode OAEP without early exits, as - `Crypto.PubKey.RSA.OAEP.unpad` has since #91. The difference is not - measurable against the cost of mask generation, and is structural: the scan - across the padding no longer depends on the data - [#126](https://github.com/kazu-yamamoto/crypton/pull/126) -* Breaking change: fix(rabin): refuse a ciphertext or a signature that is not - below the modulus, and a ciphertext carrying a leading zero octet. Squaring - and the square roots that undo it work modulo n, so Basic and Rabin-Williams - decrypted `c + n` to whatever `c` decrypted to, and all three schemes verified - `s + n`, and `-s`, wherever they verified `s`. `Basic.signWith` also refuses a - padding whose first octet is zero, which the signature cannot carry: about one - signature in 256 was one its own `verify` rejected - [#125](https://github.com/kazu-yamamoto/crypton/pull/125) -* fix(prime): derive the Miller-Rabin witnesses from the number being tested and - from a secret drawn once per process. They came from one generator made once - and shared by every call, so the witnesses for one number were the witnesses - for every number, and testing a number again told the caller nothing it had - not already been told. This is the path every GHC since 9.0 takes, integer-gmp - 1.1 having no Miller-Rabin of its own - [#124](https://github.com/kazu-yamamoto/crypton/pull/124) -* docs(elgamal): say what `signWith` requires of its ephemeral value: the range - is 1 to p-2, not the "between 0 and p-1" the haddock claimed, and the value is - a private key that a signature discloses if it is reused or revealed - [#123](https://github.com/kazu-yamamoto/crypton/pull/123) -* Breaking change: fix(afis): give `split` and `merge` one answer for a parameter - they cannot use. They had four between them, including a division by zero for - an expand count of zero and, for a count of one, handing the diffused data back - as though it were the secret - [#122](https://github.com/kazu-yamamoto/crypton/pull/122) -* Breaking change: fix(rsa): refuse a ciphertext or a signature whose integer - representative is not below the modulus, which RFC 8017 requires in sections - 5.1.2 and 5.2.2. `PKCS15.decrypt` and `OAEP.decrypt` decrypted `c + n` to the - same message as `c`, and `PSS.verifyDigest` accepted `s + n` wherever it - accepted `s` - [#121](https://github.com/kazu-yamamoto/crypton/pull/121) -* fix(otp): search the HOTP resynchronization window without early exits. The - time taken read out both where in the window the client's counter was found - and how many of the submitted values were right -- the second of which the - answer itself does not give, being `Nothing` either way. A call now costs one - HMAC per counter in the window plus one per extra value, every time - [#120](https://github.com/kazu-yamamoto/crypton/pull/120) -* Breaking change: fix(kdf): report a refused parameter as a `CryptoError` rather - than as an `ErrorCall` carrying a string, with a `'`-suffixed variant of each - entry point returning `CryptoFailable`. PBKDF2 had no validation at all: a - negative output length reached `memSet` and killed the process with SIGBUS, and - an iteration count of zero returned 32 bytes of zeroes - [#119](https://github.com/kazu-yamamoto/crypton/pull/119) -* perf(xts): take eight blocks at a time on AArch64 and x86-64, and dispatch XTS - decryption through the branch table, which it had never used. AArch64 goes - from 1200 to 7742 MiB/s encrypting and 1166 to 7763 decrypting, x86-64 from - 1220 to 3464 and from 594 to 3461 - [#118](https://github.com/kazu-yamamoto/crypton/pull/118) -* perf(poly1305): take four blocks at a time with AVX2 on x86-64, folding the - lanes back together weighted by the powers of r. 1347 to 4137 MiB/s - [#117](https://github.com/kazu-yamamoto/crypton/pull/117) -* perf(ecc): work in Jacobian coordinates in both generic prime-field scalar - multiplications, and say in `Crypto.ECC` which curves branch on a secret - scalar. P-384 and P-521 ECDSA are 2.3x: signing goes from 3.36 to 1.46 ms and - from 5.96 to 2.61 ms. P-256, which has its own C implementation, is unaffected - [#116](https://github.com/kazu-yamamoto/crypton/pull/116) -* Breaking change: fix(padding): bound PKCS#7 padding by the block rather than by - the whole input, which had let a block of sixteen accept a claim of twenty, and - refuse a `ZERO` size of zero rather than dividing by it. What `ZERO` can and - cannot undo is now written down - [#115](https://github.com/kazu-yamamoto/crypton/pull/115) -* perf(gcm): give x86 its own GCM decryption loop. It fell to the generic one, - which calls the block function once per block, and ran at a quarter the speed - of encryption; both directions now take eight blocks at a time and fold their - GHASH into one reduction. AES-256-GCM decryption goes from 561 to 2733 MiB/s - and AES-128 from 667 to 3150 - [#114](https://github.com/kazu-yamamoto/crypton/pull/114) -* perf(chacha): take eight blocks at a time with AVX2 where the machine has it, - with the cpuid and XGETBV checks that decide. ChaCha20 on x86-64 goes from - 900 to 2074 MiB/s - [#113](https://github.com/kazu-yamamoto/crypton/pull/113) -* perf(chacha): do four blocks at a time with SSE2 on x86-64, where the cipher - had no vector code at all. ChaCha20 goes from 493 to 900 MiB/s - [#112](https://github.com/kazu-yamamoto/crypton/pull/112) -* perf(chacha): do four blocks at a time with NEON on AArch64. ChaCha20 goes - from 1025 to 1955 MiB/s - [#111](https://github.com/kazu-yamamoto/crypton/pull/111) -* feat(sha512): use the ARMv8.2 SHA-512 instructions on AArch64, which SHA-384 - and the truncated SHA-512/t variants share. Hashing 1 MiB goes from 1.53 ms - to 597 us. The extension is optional, so it is asked for at runtime on both - Apple and Linux rather than assumed - [#110](https://github.com/kazu-yamamoto/crypton/pull/110) -* perf(gcm): drive GCM from AArch64 rather than the generic loop, with a group - of eight blocks folding into a single GHASH reduction. AES-128-GCM goes from - 4030 to 8266 MiB/s and AES-256 from 4043 to 7172 - [#109](https://github.com/kazu-yamamoto/crypton/pull/109) -* perf(aes): specialise the AArch64 code by key size and interleave eight - blocks, and give CTR its own loop. AES-256 ECB goes from 3886 to 15991 - MiB/s, CTR from 2935 to 13567 and CBC decryption from 4366 to 15807 - [#108](https://github.com/kazu-yamamoto/crypton/pull/108) - -* perf(aes): build the AES-NI paths on Windows, which was missing from the list of - systems that compile them. Windows builds have been doing AES, and GHASH with it, - in the generic C - [#107](https://github.com/kazu-yamamoto/crypton/pull/107) -* fix(armv8): compile the AArch64 sources on a toolchain whose baseline lacks the - crypto extensions. They had not built with GCC on AArch64 Linux since #100; CI now - builds and tests there - [#106](https://github.com/kazu-yamamoto/crypton/pull/106) -* perf(gcm): fold four GHASH blocks into one reduction. AES-256-GCM is 1.6x at 1 KiB - and 2.6x at 64 KiB on Apple silicon, and the x86 paths gain the same structure - [#105](https://github.com/kazu-yamamoto/crypton/pull/105) -* perf(sha256): use the ARMv8 SHA-2 instructions on AArch64. SHA-256 and SHA-224 are - 5.5x - [#104](https://github.com/kazu-yamamoto/crypton/pull/104) -* ci: keep the macOS jobs from queueing behind each other, and supersede a branch's - earlier run - [#103](https://github.com/kazu-yamamoto/crypton/pull/103) -* perf(aes): use PMULL for GHASH on AArch64 - [#102](https://github.com/kazu-yamamoto/crypton/pull/102) -* ci: ask cabal where its caches live rather than assuming, and keep the build - products in the cache - [#101](https://github.com/kazu-yamamoto/crypton/pull/101) -* perf(aes): use the ARMv8 cryptographic extensions on AArch64. With the GHASH work - in #102 and #105, AES-256-ECB goes from 121 to 2992 MiB/s and AES-256-GCM from 92 to - 2318 MiB/s on Apple silicon - [#100](https://github.com/kazu-yamamoto/crypton/pull/100) -* build(bench): move the benchmarks from gauge, which is no longer maintained, to - tasty-bench, and let them resolve on a current GHC - [#99](https://github.com/kazu-yamamoto/crypton/pull/99) -* Breaking change: fix(padding): reject a `PKCS7` block size outside 1..255. `pad` - raises and `unpad` returns `Nothing`, where both previously narrowed the size to a - `Word8` and silently agreed on the wrong value - [#98](https://github.com/kazu-yamamoto/crypton/pull/98) -* feat(elgamal): fix `Crypto.PubKey.ElGamal` and expose it - [#97](https://github.com/kazu-yamamoto/crypton/pull/97) -* docs(bcrypt): say that only the first 72 bytes of a password count - [#96](https://github.com/kazu-yamamoto/crypton/pull/96) -* test: move the test suite from tasty to hspec, with hspec-discover. `cabal-version` - is now 2.0 - [#95](https://github.com/kazu-yamamoto/crypton/pull/95) - -* feat(aead): add `tryAeadSimpleDecrypt`, which takes the tag length as its own argument instead of reading it off the supplied tag - [#94](https://github.com/kazu-yamamoto/crypton/pull/94) -* Breaking change: feat(dh): add `tryGetShared` to `Crypto.PubKey.DH` and `Crypto.PubKey.ECC.DH`, reporting a rejected peer value as `CryptoFailable`; `getShared` is now defined in terms of it and so raises a `CryptoError` rather than an `ErrorCall` - [#93](https://github.com/kazu-yamamoto/crypton/pull/93) -* fix(otp): compare TOTP candidates without an early exit - [#92](https://github.com/kazu-yamamoto/crypton/pull/92) -* fix(rsa): drop the early exits from PKCS#1 v1.5 and OAEP unpadding - [#91](https://github.com/kazu-yamamoto/crypton/pull/91) -* Breaking change: fix(argon2): report invalid options as `CryptoFailed` rather than raising, adding `CryptoError_ParameterInvalid` to `CryptoError` - [#90](https://github.com/kazu-yamamoto/crypton/pull/90) -* Breaking change: fix(dh): validate the peer public number, and size the shared secret from `p` rather than `params_bits` - [#89](https://github.com/kazu-yamamoto/crypton/pull/89) -* fix(dsa): do not crash on values that are not invertible modulo `q` - [#88](https://github.com/kazu-yamamoto/crypton/pull/88) -* Breaking change: fix(ecdh): validate the peer point before the exchange - [#87](https://github.com/kazu-yamamoto/crypton/pull/87) -* Breaking change: fix(pkcs15): reject PKCS#1 v1.5 signatures of the wrong length or out of range - [#86](https://github.com/kazu-yamamoto/crypton/pull/86) -* Breaking change: fix(otp): require a digest long enough for RFC 4226 dynamic truncation, which was reading past the end of the MAC - [#85](https://github.com/kazu-yamamoto/crypton/pull/85) -* fix(ecc): accept zero-x P-256 shared secret - [#84](https://github.com/kazu-yamamoto/crypton/pull/84) -* fix(p256): accept valid edge-case points - [#83](https://github.com/kazu-yamamoto/crypton/pull/83) -* Breaking change: fix(hkdf): enforce output length limit - [#82](https://github.com/kazu-yamamoto/crypton/pull/82) -* Breaking change: fix(ed25519): reject non-canonical signatures - [#81](https://github.com/kazu-yamamoto/crypton/pull/81) -* Support GHC 9.14; `tested-with` now covers 9.10.2, 9.12.4 and 9.14.1 - [#74](https://github.com/kazu-yamamoto/crypton/pull/74) - -### API changes - -* New exports: `Crypto.OTP.minimumDigestSize`, `Crypto.PubKey.DH.tryGetShared`, - `Crypto.PubKey.ECC.DH.tryGetShared`, `Crypto.Cipher.Types.AEAD.tryAeadSimpleDecrypt`, - `Crypto.Number.ModArithmetic.inverseSafe`, `Crypto.PubKey.ECC.Prim.scalarInverse`, - `scalarAdd` and `scalarMul`, `Crypto.PubKey.ECC.P256.scalarReduce`, - and the whole of `Crypto.PubKey.ElGamal`, which was present but not exposed. - The variant of an entry point that reports a refusal rather than raising is - named `try` followed by the name it varies, `tryExpand` beside `expand`. A - trailing apostrophe was the obvious spelling and is what these were called - until shortly before release; it collides too easily, since a caller that - imports one of these modules unqualified and has its own `expand'` or - `split'` no longer compiles, and `tls` did. `Safe` was considered and set - aside: this library already uses that suffix for something else, in - `Crypto.Number.ModArithmetic.expSafe` and `inverseSafe` and in - `Crypto.PubKey.ECC.P256.scalarInvSafe`, where it means the value being - worked on stays out of the timing. - The KDFs gained a variant of each entry point that can refuse its parameters, - returning `CryptoFailable` instead of raising: `Crypto.KDF.Scrypt.tryGenerate`, - `Crypto.KDF.BCrypt.tryBcrypt`, `Crypto.KDF.BCryptPBKDF.tryGenerate` and - `tryHashInternal`, `Crypto.KDF.HKDF.tryExpand`, `Crypto.KDF.PBKDF2.tryGenerate` and - `tryFastPBKDF2_SHA1`, `tryFastPBKDF2_SHA256` and `tryFastPBKDF2_SHA512`, and - `Crypto.Data.AFIS.trySplit` and `tryMerge`. These are additions and break nothing. -* Breaking change: `CryptoError_ParameterInvalid` is added to `CryptoError`. It is - appended, so the `Enum` values of the existing constructors are unchanged, but an - exhaustive `case` without a wildcard will warn. Adding a constructor to an exported - datatype is what requires a major version bump under the PVP, which would have been - 1.2.0; this release goes to 2.0.0. Everything else below changes behaviour rather - than types. -* Breaking change: `getShared` in both DH modules raises a `CryptoError` where it - previously raised an `ErrorCall`, since it is now defined in terms of `tryGetShared`. - The same is now true of `Crypto.KDF.Scrypt.generate`, `Crypto.KDF.BCrypt.bcrypt`, - `Crypto.KDF.BCryptPBKDF.generate` and `hashInternal`, and `Crypto.Data.AFIS.split` - and `merge`, each of which is defined in terms of the variant above. -* Breaking change: input that used to be accepted is now rejected -- a digest shorter - than 20 bytes in `Crypto.OTP.hotp`, a signature of the wrong length or out of range - in `Crypto.PubKey.RSA.PKCS15.verify`, an off-curve peer point or a peer public number - outside `1 < y < p-1` in `getShared`, an output beyond 255 blocks in - `Crypto.KDF.HKDF.expand`, a non-canonical Ed25519 signature, and `Options` the - implementation refuses in `Crypto.KDF.Argon2.hash`. -* Breaking change: a value at or above the modulus is now rejected where it used to be - reduced and accepted -- a ciphertext in `Crypto.PubKey.RSA.PKCS15.decrypt` and - `Crypto.PubKey.RSA.OAEP.decrypt`, a signature in `Crypto.PubKey.RSA.PSS.verify`, and - both, along with a negated signature and a ciphertext with a leading zero octet, in - the three `Crypto.PubKey.Rabin.*` schemes. -* Breaking change: parameters that used to be accepted are now refused -- an iteration - count below one or a negative output length in `Crypto.KDF.PBKDF2`, an expand count - below two or a secret of no bytes in `Crypto.Data.AFIS`, a `PKCS7` claim longer than - the block and a `ZERO` size of zero in `Crypto.Data.Padding`, and a signature padding - whose first octet is zero in `Crypto.PubKey.Rabin.Basic.signWith`. -* Breaking change: DES, 3DES, Twofish and Camellia now raise on input that is not a - whole number of blocks, as AES already did. Before, DES and Camellia returned an - answer whose tail was never written -- uninitialised memory -- and Twofish returned - more than it was given, the missing bytes read as zero. -* Breaking change: `Crypto.Data.Padding.pad` raises on a `PKCS7` block size outside - 1..255, and `unpad` returns `Nothing` for one, where both used to narrow the size to - a `Word8` and hand back something other than what was padded. -* No exported function changed its signature. - -## 1.1.5 - -* fix(aead): reject undersized tags - [#80](https://github.com/kazu-yamamoto/crypton/pull/80) -* fix(aes): refuse a zero-length AES-GCM IV - [#79](https://github.com/kazu-yamamoto/crypton/pull/79) -* fix(p256): prevent crashes when validating valid points - [#78](https://github.com/kazu-yamamoto/crypton/pull/78) -* feat(asn1): add SHA-3 HashAlgorithmASN1 instances for PKCS#1 v1.5 - [#77](https://github.com/kazu-yamamoto/crypton/pull/77) -* OCB3 conformance - [#76](https://github.com/kazu-yamamoto/crypton/pull/76) - -## 1.1.4 - -* Generic instance for RSA PublicKey and PrivateKey - -## 1.1.3 - -* Ensure that `pointAdd` in `PubKey.ECC.P256` treats the point at infinity as the additive identity. - [#73](https://github.com/kazu-yamamoto/crypton/pull/73) - -## 1.1.2 - -* Preparing `ram` v0.22. -* Generalizing RSA encrypt/decrypt to manipulate ScrubbedBytes directly. - -## 1.1.1 - -* On iOS, ScrubbedBytes based hashing is used for seedNew. On other - plateforms, entropy is used directly as used to be. - [#71](https://github.com/kazu-yamamoto/crypton/pull/71) - -## 1.1.0 - -* Removing "basement" and "memory". - [#67](https://github.com/kazu-yamamoto/crypton/pull/67) - - -## 1.0.7 - -* Stop depending on basement, use upstream dependencies instead -* Stop transitively depending on basement by depending on ram. - -## 1.0.6 - -* Fix test failures on less common 64-bit arches. - [#65](https://github.com/kazu-yamamoto/crypton/pull/65) - -## 1.0.5 - -* Setter/Getter for ChaCha counter. - [#63](https://github.com/kazu-yamamoto/crypton/pull/63) -* Add simple interface to generate full blocks - [#60](https://github.com/kazu-yamamoto/crypton/pull/60) -* Avoid `ghc-prim` dependency. - [#61](https://github.com/kazu-yamamoto/crypton/pull/61) - -## 1.0.4 - -* Ed448.sign: avoid extra re-derive of public key. - [#48](https://github.com/kazu-yamamoto/crypton/pull/48) - -## 1.0.3 - -* Make sign of Ed25519/Ed448 safer. The public key parameter is - ignored and its public key is generated from the secret key - parameter to prevent Double Public Key Signing Function Oracle - Attack. - [#47](https://github.com/kazu-yamamoto/crypton/pull/47) - -## 1.0.2 - -* Deterministic Nonce Generation for ECDSA - [#46](https://github.com/kazu-yamamoto/crypton/pull/46) -* ECDSA Signature Normalization. - [#45](https://github.com/kazu-yamamoto/crypton/pull/45) -* Add Full Test Suite from RFC 6979. - [#44](https://github.com/kazu-yamamoto/crypton/pull/44) -* ECDSA with Public Key Recovery. - [#43](https://github.com/kazu-yamamoto/crypton/pull/43) -* Providing necessary features for HPKE. - [#42](https://github.com/kazu-yamamoto/crypton/pull/42) - -## 1.0.1 - -* Update decaf library. - [#38](https://github.com/kazu-yamamoto/crypton/pull/38) -* Add TypeOperators language extension to EdDSA.hs. - [#36](https://github.com/kazu-yamamoto/crypton/pull/36) - -## 1.0.0 - -* Versions follow the standard version policy. -* Removing pthread stuff. - [#32](https://github.com/kazu-yamamoto/crypton/pull/32) - -## 0.34 - -* Hashing getRandomBytes before using as Seed for ChaChaDRG - [#24](https://github.com/kazu-yamamoto/crypton/pull/24) -* Add support for XChaCha and XChaChaPoly1305 - [#18](https://github.com/kazu-yamamoto/crypton/pull/18) -* Strict byteArray of IV c - [#16](https://github.com/kazu-yamamoto/crypton/pull/16) - -## 0.33 - -* Add "crypton_" prefix to the final C symbols. - [#9](https://github.com/kazu-yamamoto/crypton/pull/9) - -## 0.32 - -* All C symbols now have the "crypton_" prefix. - [#7](https://github.com/kazu-yamamoto/crypton/pull/7) - [#8](https://github.com/kazu-yamamoto/crypton/pull/8) - -## 0.31 - -* Crypton is forked from cryptonite with the original authors permission. -* Ignoring exceptons from hClose to read the next entropy - [#1](https://github.com/kazu-yamamoto/crypton/pull/1) -* Enabling the support_pclmuldq flag by default. - -## 0.30 - -* Fix some C symbol blake2b prefix to be cryptonite_ prefix (fix mixing with other C library) -* add hmac-lazy -* Fix compilation with GHC 9.2 -* Drop support for GHC8.0, GHC8.2, GHC8.4, GHC8.6 - -## 0.29 - -* advance compilation with gmp breakage due to change upstream -* Add native EdDSA support - -## 0.28 - -* Add hash constant time capability -* Prevent possible overflow during hashing by hashing in 4GB chunks - -## 0.27 - -* Optimise AES GCM and CCM -* Optimise P256R1 implementation -* Various AES-NI building improvements -* Add better ECDSA support -* Add XSalsa derive -* Implement square roots for ECC binary curve -* Various tests and benchmarks - -## 0.26 - -* Add Rabin cryptosystem (and variants) -* Add bcrypt_pbkdf key derivation function -* Optimize Blowfish implementation -* Add KMAC (Keccak Message Authentication Code) -* Add ECDSA sign/verify digest APIs -* Hash algorithms with runtime output length -* Update blake2 to latest upstream version -* RSA-PSS with arbitrary key size -* SHAKE with output length not divisible by 8 -* Add Read and Data instances for Digest type -* Improve P256 scalar primitives -* Fix hash truncation bug in DSA -* Fix cost parsing for bcrypt -* Fix ECC failures on arm64 -* Correction to PKCS#1 v1.5 padding -* Use powModSecInteger when available -* Drop GHC 7.8 and GHC 7.10 support, refer to pkg-guidelines -* Optimise GCM mode -* Add little endian serialization of integer - -## 0.25 - -* Improve digest binary conversion efficiency -* AES CCM support -* Add MonadFailure instance for CryptoFailable -* Various misc improvements on documentation -* Edwards25519 lowlevel arithmetic support -* P256 add point negation -* Improvement in ECC (benchmark, better normalization) -* Blake2 improvements to context size -* Use gauge instead of criterion -* Use haskell-ci for CI scripts -* Improve Digest memory representation to be 2 less Ints and one less boxing - moving from `UArray` to `Block` - -## 0.24 - -* Ed25519: generateSecret & Documentation updates -* Repair tutorial -* RSA: Allow signing digest directly -* IV add: fix overflow behavior -* P256: validate point when decoding -* Compilation fix with deepseq disabled -* Improve Curve448 and use decaf for Ed448 -* Compilation flag blake2 sse merged in sse support -* Process unaligned data better in hashes and AES, on architecture needing alignment -* Drop support for ghc 7.6 -* Add ability to create random generator Seed from binary data and - loosen constraint on ChaChaDRG seed from ByteArray to ByteArrayAccess. -* Add 3 associated types with the HashAlgorithm class, to get - access to the constant for BlockSize, DigestSize and ContextSize at the type level. - the related function that this replaced will be deprecated in later release, and - eventually removed. - -API CHANGES: - -* Improve ECDH safety to return failure for bad inputs (e.g. public point in small order subgroup). - To go back to previous behavior you can replace `ecdh` by `ecdhRaw`. It's recommended to - use `ecdh` and handle the error appropriately. -* Users defining their own HashAlgorithm needs to define the - HashBlockSize, HashDigest, HashInternalContextSize associated types - -## 0.23 - -* Digest memory usage improvement by using unpinned memory -* Fix generateBetween to generate within the right bounds -* Add pure Twofish implementation -* Fix memory allocation in P256 when using a temp point -* Consolidate hash benchmark code -* Add Nat-length Blake2 support (GHC > 8.0) -* Update tutorial - -## 0.22 - -* Add Argon2 (Password Hashing Competition winner) hash function -* Update blake2 to latest upstream version -* Add extra blake2 hashing size -* Add faster PBKDF2 functions for SHA1/SHA256/SHA512 -* Add SHAKE128 and SHAKE256 -* Cleanup prime generation, and add tests -* Add Time-based One Time Password (TOTP) and HMAC-based One Time Password (HOTP) -* Rename Ed448 module name to Curve448, old module name still valid for now - -## 0.21 - -* Drop automated tests with GHC 7.0, GHC 7.4, GHC 7.6. support dropped, but probably still working. -* Improve non-aligned support in C sources, ChaCha and SHA3 now probably work on arch without support for unaligned access. not complete or tested. -* Add another ECC framework that is more flexible, allowing different implementations to work instead of - the existing Pure haskell NIST implementation. -* Add ECIES basic primitives -* Add XSalsa20 stream cipher -* Process partial buffer correctly with Poly1305 - -## 0.20 - -* Fixed hash truncation used in ECDSA signature & verification (Olivier Chéron) -* Fix ECDH when scalar and coordinate bit sizes differ (Olivier Chéron) -* Speed up ECDSA verification using Shamir's trick (Olivier Chéron) -* Fix rdrand on windows - -## 0.19 - -* Add tutorial (Yann Esposito) -* Derive Show instance for better interaction with Show pretty printer (Eric Mertens) - -## 0.18 - -* Re-used standard rdrand instructions instead of bytedump of rdrand instruction -* Improvement to F2m, including lots of tests (Andrew Lelechenko) -* Add error check on salt length in bcrypt - -## 0.17 - -* Add Miyaguchi-Preneel construction (Kei Hibino) -* Fix buffer length in scrypt (Luke Taylor) -* build fixes for i686 and arm related to rdrand - -## 0.16 - -* Fix basepoint for Ed448 - -* Enable 64-bit Curve25519 implementation - -## 0.15 - -* Fix serialization of DH and ECDH - -## 0.14 - -* Reduce size of SHA3 context instead of allocating all-size fit memory. save - up to 72 bytes of memory per context for SHA3-512. -* Add a Seed capability to the main DRG, to be able to debug/reproduce randomized program - where you would want to disable the randomness. -* Add support for Cipher-based Message Authentication Code (CMAC) (Kei Hibino) -* *CHANGE* Change the `SharedKey` for `Crypto.PubKey.DH` and `Crypto.PubKey.ECC.DH`, - from an Integer newtype to a ScrubbedBytes newtype. Prevent mistake where the - bytes representation is generated without the right padding (when needed). -* *CHANGE* Keep The field size in bits, in the `Params` in `Crypto.PubKey.DH`, - moving from 2 elements to 3 elements in the structure. - -## 0.13 - -* *SECURITY* Fix buffer overflow issue in SHA384, copying 16 extra bytes from - the SHA512 context to the destination memory pointer leading to memory - corruption, segfault. (Mikael Bung) - -## 0.12 - -* Fix compilation issue with Ed448 on 32 bits machine. - -## 0.11 - -* Truncate hashing correctly for DSA -* Add support for HKDF (RFC 5869) -* Add support for Ed448 -* Extends support for Blake2s to 224 bits version. -* Compilation workaround for old distribution (RHEL 4.1) -* Compilation fix for AIX -* Compilation fix with AESNI and ghci compiling C source in a weird order. -* Fix example compilation, typo, and warning - -## 0.10 - -* Add reference implementation of blake2 for non-SSE2 platform -* Add support\_blake2\_sse flag - -## 0.9 - -* Quiet down unused module imports -* Move Curve25519 over to Crypto.Error instead of using Either String. -* Add documentation for ChaChaPoly1305 -* Add missing documentation for various modules -* Add a way to create Poly1305 Auth tag. -* Added support for the BLAKE2 family of hash algorithms -* Fix endianness of incrementNonce function for ChaChaPoly1305 - -## 0.8 - -* Add support for ChaChaPoly1305 Nonce Increment (John Galt) -* Move repository to the haskell-crypto organisation - -## 0.7 - -* Add PKCS5 / PKCS7 padding and unpadding methods -* Fix ChaChaPoly1305 Decryption -* Add support for BCrypt (Luke Taylor) - -## 0.6 - -* Add ChaChaPoly1305 AE cipher -* Add instructions in README for building on old OSX -* Fix blocking /dev/random Andrey Sverdlichenko - -## 0.5 - -* Fix all strays exports to all be under the cryptonite prefix. - -## 0.4 - -* Add a System DRG that represent a referentially transparent of evaluated bytes - while using lazy evaluation for future entropy values. - -## 0.3 - -* Allow drgNew to run in any MonadRandom, providing cascading initialization -* Remove Crypto.PubKey.HashDescr in favor of just having the algorithm - specified in PKCS15 RSA function. -* Fix documentation in cipher sub section (Luke Taylor) -* Cleanup AES dead functions (Luke Taylor) -* Fix Show instance of Digest to display without quotes similar to cryptohash -* Use scrubbed bytes instead of bytes for P256 scalar - -## 0.2 - -* Fix P256 compilation and exactness, + add tests -* Add a raw memory number serialization capability (i2osp, os2ip) -* Improve tests for number serialization -* Improve tests for ECC arithmetics -* Add Ord instance for Digest (Nicolas Di Prima) -* Fix entropy compilation on windows 64 bits. - -## 0.1 - -* Initial release +## 2.2.5 + +**RSA private-key operations check their CRT result.** A fault in one +half of the computation could hand out a signature that reveals a prime of +the key; the result is now raised back to the public exponent first, as +OpenSSL does, at a cost of two or three per cent on signing. + +* perf(rsa): verification in C at every RSA size on Apple AArch64 + [#340](https://github.com/kazu-yamamoto/crypton/pull/340) +* perf(rsa): a generated squaring at 48 limbs on AArch64 + [#341](https://github.com/kazu-yamamoto/crypton/pull/341) +* perf(rsa): verification without lists or Integer round trips + [#342](https://github.com/kazu-yamamoto/crypton/pull/342) +* ci: time out instead of waiting on a stalled package mirror + [#343](https://github.com/kazu-yamamoto/crypton/pull/343) +* fix(rsa): check the CRT result before letting it out + [#344](https://github.com/kazu-yamamoto/crypton/pull/344) +* perf(hash): one context for a one-shot hash and for MGF1 + [#345](https://github.com/kazu-yamamoto/crypton/pull/345) +* perf(rsa): RSA-3072's 24-limb square from 8-limb pieces on x86-64 + [#346](https://github.com/kazu-yamamoto/crypton/pull/346) + +## 2.2.4 + +**New module `Crypto.PubKey.RSA.Blinded`.** A `BlindedKey` carries RSA +blinding from one signature or decryption to the next. The existing +`*Safer` functions use it through a small process-wide table, and +verification keeps R^2 per modulus in another. + +* perf(rsa): keep the blinding between operations + [#335](https://github.com/kazu-yamamoto/crypton/pull/335) +* perf(rsa): a generated Montgomery multiplication for AArch64 + [#336](https://github.com/kazu-yamamoto/crypton/pull/336) +* fix: the C warnings CI reports + [#337](https://github.com/kazu-yamamoto/crypton/pull/337) +* perf(rsa): both CRT halves at once with AVX-512 IFMA + [#338](https://github.com/kazu-yamamoto/crypton/pull/338) +* perf(rsa): keep R^2 per modulus for verification + [#339](https://github.com/kazu-yamamoto/crypton/pull/339) + +## 2.2.3 + +**crypton builds on Windows again.** 2.2.2's tarball lacked a header that +only Windows includes, so it built from git but not from Hackage. With +2.2.2, `-f-support_s2n_bignum` works around it. + +* bench: RSA-3072 in the tables + [#331](https://github.com/kazu-yamamoto/crypton/pull/331) +* perf(rsa): a generated Montgomery squaring for AArch64 + [#333](https://github.com/kazu-yamamoto/crypton/pull/333) +* fix: ship the Windows wrapper for s2n-bignum + [#334](https://github.com/kazu-yamamoto/crypton/pull/334) + +## 2.2.2 + +**crypton builds on an Intel Mac again.** From 2.1.2 onwards, Apple's +assembler rejected one of the vendored RSA assembly files, so the default +configuration did not build on x86-64 macOS. With an older release, +`-f-support_s2n_bignum` works around it. + +* doc: the Ed25519 verify step was the AVX2 flag, and the tables move to 2.2.1 + [#318](https://github.com/kazu-yamamoto/crypton/pull/318) +* test: the window the exponentiation walks in, and the macro that hid it + [#319](https://github.com/kazu-yamamoto/crypton/pull/319) +* perf: the Montgomery constant in assembly, on both architectures + [#320](https://github.com/kazu-yamamoto/crypton/pull/320) +* perf: verification need not hide an exponent that is public + [#321](https://github.com/kazu-yamamoto/crypton/pull/321) +* doc: the generic AArch64 routines, measured at last + [#322](https://github.com/kazu-yamamoto/crypton/pull/322) +* doc: a VM's CPUID is not the hardware: two descriptions that said otherwise + [#323](https://github.com/kazu-yamamoto/crypton/pull/323) +* test: switches to assume what CPUID hides, for testing only + [#324](https://github.com/kazu-yamamoto/crypton/pull/324) +* perf: RSA-3072's halves through s2n-bignum on x86-64 + [#325](https://github.com/kazu-yamamoto/crypton/pull/325) +* perf: 3072- and 4096-bit secure exponentiation through s2n-bignum + [#326](https://github.com/kazu-yamamoto/crypton/pull/326) +* test: the two AES differential tests, in CI + [#327](https://github.com/kazu-yamamoto/crypton/pull/327) +* perf: s2n-bignum on x86-64 Windows + [#328](https://github.com/kazu-yamamoto/crypton/pull/328) +* fix: crypton builds on an Intel Mac again + [#329](https://github.com/kazu-yamamoto/crypton/pull/329) +* ci: link with the Command Line Tools on macOS + [#330](https://github.com/kazu-yamamoto/crypton/pull/330) + +## 2.2.1 + +**A binary built on one x86-64 machine runs on another again.** Since +2.1.8 the `-mavx2 -mbmi2` written for the ML-KEM and ML-DSA backends have +been compiling every C file in the library -- `cc-options` in a conditional +block belong to the component, not to the block -- so the compiler put AVX2 +and BMI2 into ordinary code that has no dispatch and asks nothing at +runtime: Ed25519, Argon2, Curve25519, P-256, Ed448, and the older hashes +among them. A binary built where AVX2 exists then died with `SIGILL` where +it does not, in Ed25519 rather than in anything post-quantum. + +The two backends also chose their implementation when the library was +compiled and called it unchecked. They ask the processor now, as everything +else in crypton that uses an instruction beyond the baseline already did, +and fall back to the portable C where the answer is no. AArch64 asks about +the ARMv8.4 Keccak instructions for the same reason. + +Nothing in the API changes. + +* fix: stop raising the x86-64 baseline for the whole library + [#317](https://github.com/kazu-yamamoto/crypton/pull/317) +* fix: ask the processor before using the ML-KEM and ML-DSA backends + [#317](https://github.com/kazu-yamamoto/crypton/pull/317) +* test(ct): link the processor check into the ML-KEM backend build + [#317](https://github.com/kazu-yamamoto/crypton/pull/317) + +## 2.2.0 + +`ProcessorOption` is no longer a sum of constructors. It is a number with +pattern synonyms for the names it knows, so it has `Ord` now and has lost +`Enum` and `Data`, and a `case` over the names needs a catch-all. `RDRAND` +is gone from it, and with it the `support_rdrand` flag. An AArch64 machine +reports `ARMAES` and `ARMPMULL` where it used to report `AESNI` and `PCLMUL`. + +Two more things change under a caller without the type checker saying so: +`MonadRandom IO` draws from a per-thread generator rather than reading the +system on every call, and a missing or failing entropy source raises +`EntropyError` rather than an `ErrorCall` or an `IOException`. + +* feat: take entropy from the kernel without a file descriptor + [#300](https://github.com/kazu-yamamoto/crypton/pull/300) +* feat: a ChaCha20 generator per operating system thread + [#300](https://github.com/kazu-yamamoto/crypton/pull/300) +* feat: MonadRandom IO goes through the per-thread generator + [#300](https://github.com/kazu-yamamoto/crypton/pull/300) +* fix: say that the system gave no entropy, rather than error + [#300](https://github.com/kazu-yamamoto/crypton/pull/300) +* fix: stop using RDRAND at all, since it cannot add anything + [#300](https://github.com/kazu-yamamoto/crypton/pull/300) +* feat: name every processor feature crypton dispatches on, not three of them + [#302](https://github.com/kazu-yamamoto/crypton/pull/302) +* fix: an AArch64 machine no longer reports AESNI and PCLMUL + [#302](https://github.com/kazu-yamamoto/crypton/pull/302) +* chore(rsa): deprecate `sign`, `signSafer` and `verify` in favour of them + [#305](https://github.com/kazu-yamamoto/crypton/pull/305) +* fix: compute the portable AES and GHASH without a table + [#308](https://github.com/kazu-yamamoto/crypton/pull/308) +* feat: use the AES instructions on 32-bit ARM + [#309](https://github.com/kazu-yamamoto/crypton/pull/309) +* test: skip the accelerated constant-time driver where the processor has no instructions + [#310](https://github.com/kazu-yamamoto/crypton/pull/310) +* feat: use the POWER8 AES and GHASH instructions on ppc64le + [#311](https://github.com/kazu-yamamoto/crypton/pull/311) +* test: the big-endian harness now asks about AES + [#312](https://github.com/kazu-yamamoto/crypton/pull/312) +* doc(tutorial): compile the examples, and teach the things people come for + [#313](https://github.com/kazu-yamamoto/crypton/pull/313) + +## 2.1.10 + +* build: ask the processor for AES-NI on every x86 system, not four of them + [#307](https://github.com/kazu-yamamoto/crypton/pull/307) +* fix: ask one place which ARMv8 instruction sets the machine has + [#307](https://github.com/kazu-yamamoto/crypton/pull/307) + +## 2.1.9 + +* feat(rsa): PKCS#1 v1.5 operations that take a digest, and ones that take a DigestInfo + [#305](https://github.com/kazu-yamamoto/crypton/pull/305) +* feat(dsa): sign and verify over a digest + [#305](https://github.com/kazu-yamamoto/crypton/pull/305) +* feat(elgamal): sign and verify over a digest + [#305](https://github.com/kazu-yamamoto/crypton/pull/305) +* feat(rabin): Rabin-Williams and Modified Rabin sign and verify over a digest + [#305](https://github.com/kazu-yamamoto/crypton/pull/305) + +## 2.1.8 + +* chore: stop hiding foldl' from Prelude + [#294](https://github.com/kazu-yamamoto/crypton/pull/294) +* chore: ask for hidden visibility only where the format has it + [#295](https://github.com/kazu-yamamoto/crypton/pull/295) +* feat: ML-KEM and ML-DSA, through mlkem-native and mldsa-native + [#297](https://github.com/kazu-yamamoto/crypton/pull/297) + +## 2.1.7 + +RSA-PSS verification was accepting an encoding RFC 8017 says to refuse. It +is a conformance fault rather than a forgery: producing such a signature takes +the private key, so a third party holding a valid signature cannot turn it +into one of these. + +* chore(p256): drop two declarations nothing defines + [#292](https://github.com/kazu-yamamoto/crypton/pull/292) +* fix(pss): refuse an encoding with a bit set outside emBits + [#293](https://github.com/kazu-yamamoto/crypton/pull/293) + +## 2.1.6 + +The `license:` field now says what the tree holds -- `BSD-3-Clause AND MIT +AND ISC` -- and `license-files:` lists all five texts. **Nothing is required +of a user that was not required before**; the field was simply incomplete. + +* Carry the MIT notice for the parts that follow fusion, and say what the tree holds + [#267](https://github.com/kazu-yamamoto/crypton/pull/267) +* fix: two preconditions the Haskell layer did not enforce + [#288](https://github.com/kazu-yamamoto/crypton/pull/288) +* fix(x86): stop casting a packed block to __m128i * + [#289](https://github.com/kazu-yamamoto/crypton/pull/289) +* fix(c): say that the digest pointers are never null + [#290](https://github.com/kazu-yamamoto/crypton/pull/290) +* fix(pbkdf2): refuse a digest larger than its block at compile time + [#291](https://github.com/kazu-yamamoto/crypton/pull/291) + +## 2.1.5 + +2.1.3 and 2.1.4 cannot be built with GCC 14 or newer. This release is +that fix. + +* Watch for a primitive fallen off its fast path + [#277](https://github.com/kazu-yamamoto/crypton/pull/277) +* The performance tables for 2.1.4 + [#278](https://github.com/kazu-yamamoto/crypton/pull/278) +* The x86-64 table, on the machine the README names + [#279](https://github.com/kazu-yamamoto/crypton/pull/279) +* perf(armv8): GHASH against a twisted H, a quarter faster + [#281](https://github.com/kazu-yamamoto/crypton/pull/281) +* ct: run the constant-time harness on AArch64, and put its AES to it + [#283](https://github.com/kazu-yamamoto/crypton/pull/283) +* Fix crypton_sha1_x86_do_chunk signature + [#284](https://github.com/kazu-yamamoto/crypton/pull/284) +* ci: build the C with a compiler stricter than this matrix has + [#285](https://github.com/kazu-yamamoto/crypton/pull/285) + +## 2.1.4 + +2.1.3 could not be built from Hackage in the default configuration, and is +deprecated there. This release is that fix. + +* Add p256 header files to cabal extra-source-files + [#271](https://github.com/kazu-yamamoto/crypton/pull/271) +* Build what is published, not only what is checked out + [#272](https://github.com/kazu-yamamoto/crypton/pull/272) +* SHA-256 on AArch64 is 5.4x slower in 2.1.3 than in 2.1.2 + [#274](https://github.com/kazu-yamamoto/crypton/pull/274) +* The target attribute spelling gcc before 13 understands + [#276](https://github.com/kazu-yamamoto/crypton/pull/276) + +## 2.1.3 + +* fix(aead): reject missing and oversized authentication tags + [#233](https://github.com/kazu-yamamoto/crypton/pull/233) +* fix(chacha): remove redundant Word8 import + [#234](https://github.com/kazu-yamamoto/crypton/pull/234) +* test(number): cover the two modulus sizes the assembly runs at + [#235](https://github.com/kazu-yamamoto/crypton/pull/235) +* perf(rsa): swap the buffers instead of copying them back + [#236](https://github.com/kazu-yamamoto/crypton/pull/236) +* perf(rsa): scan the exponentiation's table four limbs at a time + [#237](https://github.com/kazu-yamamoto/crypton/pull/237) +* fix(gcm): write the field doubling from its definition + [#238](https://github.com/kazu-yamamoto/crypton/pull/238) +* doc(gcm): carry the MIT notice for the parts that follow fusion + [#239](https://github.com/kazu-yamamoto/crypton/pull/239) +* perf(ed25519): the base point multiplication through s2n-bignum + [#240](https://github.com/kazu-yamamoto/crypton/pull/240) +* perf(gcm): AES-GCM through the 512-bit VAES and VPCLMULQDQ + [#241](https://github.com/kazu-yamamoto/crypton/pull/241) +* doc: the README said AVX-512 was not used, and it is + [#242](https://github.com/kazu-yamamoto/crypton/pull/242) +* perf(ecdsa): P-256 verification multiplies both scalars at once + [#243](https://github.com/kazu-yamamoto/crypton/pull/243) +* perf(rsa): four limbs and two carry chains on AArch64 + [#244](https://github.com/kazu-yamamoto/crypton/pull/244) +* perf(rsa): write out the ragged end of the AArch64 row + [#245](https://github.com/kazu-yamamoto/crypton/pull/245) +* perf(rsa): build R^2 by squaring, not by doubling + [#246](https://github.com/kazu-yamamoto/crypton/pull/246) +* perf(rsa): stop clearing the scratch a Montgomery multiply writes over + [#247](https://github.com/kazu-yamamoto/crypton/pull/247) +* doc(gcm): write down what the AArch64 GHASH is short of + [#248](https://github.com/kazu-yamamoto/crypton/pull/248) +* security(aes): refuse a nonce of no bytes in Crypto.Cipher.AES.GCM + [#250](https://github.com/kazu-yamamoto/crypton/pull/250) +* ci: build and test the C the other architectures use + [#251](https://github.com/kazu-yamamoto/crypton/pull/251) +* perf(p256): five teeth to a comb block, over the signed representation + [#252](https://github.com/kazu-yamamoto/crypton/pull/252) +* security(cipher): stop truncating message lengths on the way to the C + [#253](https://github.com/kazu-yamamoto/crypton/pull/253) +* fix(c): two left shifts the standard leaves undefined + [#254](https://github.com/kazu-yamamoto/crypton/pull/254) +* ci: run the C under the sanitizers + [#255](https://github.com/kazu-yamamoto/crypton/pull/255) +* fix(c): read words out of a block rather than pointing at it + [#256](https://github.com/kazu-yamamoto/crypton/pull/256) +* fix(internal): drop an import nothing uses any more + [#257](https://github.com/kazu-yamamoto/crypton/pull/257) +* fix(c): decide the dispatch table once, not on every key + [#258](https://github.com/kazu-yamamoto/crypton/pull/258) +* Fix the three cabal flag settings that were broken + [#259](https://github.com/kazu-yamamoto/crypton/pull/259) +* Run the C that only 32-bit architectures get, and fix what that found + [#260](https://github.com/kazu-yamamoto/crypton/pull/260) +* Let the last addition of each scalar multiplication be a complete one + [#261](https://github.com/kazu-yamamoto/crypton/pull/261) +* Ask whether the secrets decide anything + [#262](https://github.com/kazu-yamamoto/crypton/pull/262) +* Ask a big-endian machine the same questions + [#263](https://github.com/kazu-yamamoto/crypton/pull/263) +* Ask what the secrets leave behind + [#264](https://github.com/kazu-yamamoto/crypton/pull/264) +* Stop taking eighteen runner slots to test three things + [#265](https://github.com/kazu-yamamoto/crypton/pull/265) +* Make the scrubs ones the compiler cannot drop, and finish round ten + [#266](https://github.com/kazu-yamamoto/crypton/pull/266) +* Feed the parsers bytes nobody chose + [#268](https://github.com/kazu-yamamoto/crypton/pull/268) + +## 2.1.2 + +* perf(p256): 255 squarings for the field inversion, not 287 + [#223](https://github.com/kazu-yamamoto/crypton/pull/223) +* perf(p256): ECDH through s2n-bignum, 2.7x + [#224](https://github.com/kazu-yamamoto/crypton/pull/224) +* perf(ecc): P-384 and P-521 through s2n-bignum, 7x and 9x + [#225](https://github.com/kazu-yamamoto/crypton/pull/225) +* perf(p256): ECDSA signing 2.4x and verification 2.2x + [#226](https://github.com/kazu-yamamoto/crypton/pull/226) +* perf(rsa): the Montgomery multiplication through s2n-bignum on x86-64 + [#227](https://github.com/kazu-yamamoto/crypton/pull/227) +* perf(ecdsa): invert modulo the order in division steps, not an exponentiation + [#228](https://github.com/kazu-yamamoto/crypton/pull/228) +* perf(x25519): X25519 through s2n-bignum, and a table for key generation + [#229](https://github.com/kazu-yamamoto/crypton/pull/229) +* perf(gcm): AES-GCM through VAES and VPCLMULQDQ + [#230](https://github.com/kazu-yamamoto/crypton/pull/230) +* perf(gcm): compile the wide loop once per key length + [#231](https://github.com/kazu-yamamoto/crypton/pull/231) + +## 2.1.1 + +* feat(ecdsa): RFC 6979 deterministic nonces for Crypto.PubKey.ECDSA + [#219](https://github.com/kazu-yamamoto/crypton/pull/219) +* feat(gcm): a decrypt that hands back the tag instead of comparing it + [#220](https://github.com/kazu-yamamoto/crypton/pull/220) +* feat(chachapoly): ChaCha20-Poly1305 a message at a time + [#221](https://github.com/kazu-yamamoto/crypton/pull/221) +* docs(rsa): say in the haddock what the optional blinder covers + [#222](https://github.com/kazu-yamamoto/crypton/pull/222) + +## 2.1.0 + +* fix(cpu): stop reading Intel's SDBG bit as AMD's XOP + [#204](https://github.com/kazu-yamamoto/crypton/pull/204) +* fix(bench): build the benchmark against the checked ChaCha20-Poly1305 key + [#205](https://github.com/kazu-yamamoto/crypton/pull/205) +* ci: key the cache on the package version + [#206](https://github.com/kazu-yamamoto/crypton/pull/206) +* ci: build the benchmarks + [#207](https://github.com/kazu-yamamoto/crypton/pull/207) +* perf(gcm): a fused AES-GCM for x86-64 + [#208](https://github.com/kazu-yamamoto/crypton/pull/208) +* perf(gcm): a fused AES-GCM for AArch64 + [#209](https://github.com/kazu-yamamoto/crypton/pull/209) +* perf(gcm): build the counter in vector registers + [#210](https://github.com/kazu-yamamoto/crypton/pull/210) +* perf(gcm): a spare lane for E(K,Y0), and a cheaper short block + [#211](https://github.com/kazu-yamamoto/crypton/pull/211) +* perf(gcm): unroll the tail pass, and take its blocks from registers + [#212](https://github.com/kazu-yamamoto/crypton/pull/212) +* perf(gcm): read a short block where it lies + [#213](https://github.com/kazu-yamamoto/crypton/pull/213) +* perf(gcm): the length block and the counter, in registers + [#214](https://github.com/kazu-yamamoto/crypton/pull/214) +* perf(gcm): let the one-call interface specialise + [#215](https://github.com/kazu-yamamoto/crypton/pull/215) +* perf(gcm): decryption takes the fused path too + [#216](https://github.com/kazu-yamamoto/crypton/pull/216) +* perf(gcm): GHASH takes the ciphertext from the output buffer + [#217](https://github.com/kazu-yamamoto/crypton/pull/217) +* perf(p256): a signed five-bit window for the variable-point multiply + [#218](https://github.com/kazu-yamamoto/crypton/pull/218) + +## 2.0.1 + +* feat(hash): Skein with the digest size as a type parameter + [#197](https://github.com/kazu-yamamoto/crypton/pull/197) +* fix(chachapoly1305): take a checked key, so that initializing cannot fail + [#198](https://github.com/kazu-yamamoto/crypton/pull/198) +* feat(aes): Crypto.Cipher.AES.GCM, for many short messages under one key + [#199](https://github.com/kazu-yamamoto/crypton/pull/199) +* build: say which platforms the fallback AES sources are for + [#200](https://github.com/kazu-yamamoto/crypton/pull/200) +* feat(aes): encryptWithMask, for the QUIC header protection mask + [#201](https://github.com/kazu-yamamoto/crypton/pull/201) +* fix(cpu): stop reading Intel's SDBG bit as AMD's XOP + [#203](https://github.com/kazu-yamamoto/crypton/pull/203) + +## 2.0.0 + +**Breaking changes.** Input that used to be accepted is now refused: a value +at or above an RSA or Rabin modulus, a signature of the wrong length or out of +range, a digest too short for HOTP's dynamic truncation, a non-canonical +Ed25519 signature, a PKCS#7 block size outside 1..255, and block cipher input +that is not a whole number of blocks. A refused KDF, Argon2 or bcrypt +parameter is reported as a `CryptoError` rather than raised as an `ErrorCall`, +and `CryptoError_ParameterInvalid` is appended to `CryptoError`; +`tryGetShared` is added beside `getShared`. No exported function changed its +signature. + +**Deprecated.** The eighteen curves over a binary field in +`Crypto.ECC.Simple.Types`. They are obsolete, they are the curves whose +cofactor is not 1, and they will go in a later major version. Prefer a prime +curve, or X25519. + +* Add GHC 9.14 to CI + [#74](https://github.com/kazu-yamamoto/crypton/pull/74) +* fix(ed25519): reject non-canonical signatures + [#81](https://github.com/kazu-yamamoto/crypton/pull/81) +* fix(hkdf): enforce RFC 5869 output limit + [#82](https://github.com/kazu-yamamoto/crypton/pull/82) +* fix(p256): accept valid edge-case points + [#83](https://github.com/kazu-yamamoto/crypton/pull/83) +* fix(ecc): accept zero-x P-256 shared secrets + [#84](https://github.com/kazu-yamamoto/crypton/pull/84) +* fix(otp): require a digest long enough for dynamic truncation + [#85](https://github.com/kazu-yamamoto/crypton/pull/85) +* fix(pkcs15): reject malformed PKCS#1 v1.5 signatures + [#86](https://github.com/kazu-yamamoto/crypton/pull/86) +* fix(ecdh): validate the peer point before the exchange + [#87](https://github.com/kazu-yamamoto/crypton/pull/87) +* fix(dsa): do not crash on non-invertible values + [#88](https://github.com/kazu-yamamoto/crypton/pull/88) +* fix(dh): validate the peer public number + [#89](https://github.com/kazu-yamamoto/crypton/pull/89) +* fix(argon2): report invalid options as CryptoFailed + [#90](https://github.com/kazu-yamamoto/crypton/pull/90) +* fix(rsa): drop the early exits from PKCS#1 v1.5 and OAEP unpadding + [#91](https://github.com/kazu-yamamoto/crypton/pull/91) +* fix(otp): compare TOTP candidates without an early exit + [#92](https://github.com/kazu-yamamoto/crypton/pull/92) +* feat(dh): add getShared' reporting rejections as CryptoFailable + [#93](https://github.com/kazu-yamamoto/crypton/pull/93) +* feat(aead): add aeadSimpleDecrypt' taking the tag length + [#94](https://github.com/kazu-yamamoto/crypton/pull/94) +* test: move the suite to hspec, with hspec-discover + [#95](https://github.com/kazu-yamamoto/crypton/pull/95) +* docs(bcrypt): say that only the first 72 bytes of a password count + [#96](https://github.com/kazu-yamamoto/crypton/pull/96) +* feat(elgamal): fix and expose Crypto.PubKey.ElGamal + [#97](https://github.com/kazu-yamamoto/crypton/pull/97) +* fix(padding): reject a PKCS7 block size outside 1..255 + [#98](https://github.com/kazu-yamamoto/crypton/pull/98) +* build(bench): move the benchmarks from gauge to tasty-bench + [#99](https://github.com/kazu-yamamoto/crypton/pull/99) +* feat(aes): use the ARMv8 cryptographic extensions on AArch64 + [#100](https://github.com/kazu-yamamoto/crypton/pull/100) +* ci: stop throwing the cache away, and keep the build products in it + [#101](https://github.com/kazu-yamamoto/crypton/pull/101) +* feat(aes): use PMULL for GHASH on AArch64 + [#102](https://github.com/kazu-yamamoto/crypton/pull/102) +* ci: cut the macOS queueing and supersede stale branch runs + [#103](https://github.com/kazu-yamamoto/crypton/pull/103) +* feat(sha256): use the ARMv8 SHA-2 instructions on AArch64 + [#104](https://github.com/kazu-yamamoto/crypton/pull/104) +* perf(gcm): fold four GHASH blocks into one reduction + [#105](https://github.com/kazu-yamamoto/crypton/pull/105) +* ci: build and test on aarch64 Linux + [#106](https://github.com/kazu-yamamoto/crypton/pull/106) +* fix(cabal): build the AES-NI paths on Windows too + [#107](https://github.com/kazu-yamamoto/crypton/pull/107) +* perf(aes): specialise by key size and interleave eight blocks on AArch64 + [#108](https://github.com/kazu-yamamoto/crypton/pull/108) +* perf(gcm): drive GCM from AArch64 rather than the generic loop + [#109](https://github.com/kazu-yamamoto/crypton/pull/109) +* feat(sha512): use the ARMv8.2 SHA-512 instructions on AArch64 + [#110](https://github.com/kazu-yamamoto/crypton/pull/110) +* perf(chacha): do four blocks at a time with NEON on AArch64 + [#111](https://github.com/kazu-yamamoto/crypton/pull/111) +* perf(chacha): do four blocks at a time with SSE2 on x86-64 + [#112](https://github.com/kazu-yamamoto/crypton/pull/112) +* perf(chacha): take eight blocks with AVX2 where the machine has it + [#113](https://github.com/kazu-yamamoto/crypton/pull/113) +* perf(gcm): give x86 its own decryption loop, and eight blocks either way + [#114](https://github.com/kazu-yamamoto/crypton/pull/114) +* fix(padding): bound PKCS7 padding by the block, and check ZERO's size + [#115](https://github.com/kazu-yamamoto/crypton/pull/115) +* ecc: say which curves branch on a secret scalar, and work in Jacobian coordinates + [#116](https://github.com/kazu-yamamoto/crypton/pull/116) +* perf(poly1305): take four blocks at a time with AVX2 on x86-64 + [#117](https://github.com/kazu-yamamoto/crypton/pull/117) +* perf(xts): drive XTS eight blocks at a time, and dispatch its decryption + [#118](https://github.com/kazu-yamamoto/crypton/pull/118) +* Report refused KDF parameters as CryptoError, and fix a PBKDF2 SIGBUS + [#119](https://github.com/kazu-yamamoto/crypton/pull/119) +* Search the HOTP resynchronization window without early exits + [#120](https://github.com/kazu-yamamoto/crypton/pull/120) +* Refuse an RSA representative that is not below the modulus + [#121](https://github.com/kazu-yamamoto/crypton/pull/121) +* Give AFIS one answer for a parameter it cannot use + [#122](https://github.com/kazu-yamamoto/crypton/pull/122) +* Say what ElGamal's signWith requires of k + [#123](https://github.com/kazu-yamamoto/crypton/pull/123) +* Draw Miller-Rabin witnesses per number, not once per process + [#124](https://github.com/kazu-yamamoto/crypton/pull/124) +* Refuse Rabin values that are not below the modulus, and keep the padding that was signed + [#125](https://github.com/kazu-yamamoto/crypton/pull/125) +* Decode Rabin's OAEP without early exits + [#126](https://github.com/kazu-yamamoto/crypton/pull/126) +* Make CMAC linear, and chain it through CBC + [#127](https://github.com/kazu-yamamoto/crypton/pull/127) +* Put DES in C + [#128](https://github.com/kazu-yamamoto/crypton/pull/128) +* Make the generic block cipher modes linear, and bulk where the blocks allow + [#129](https://github.com/kazu-yamamoto/crypton/pull/129) +* Walk Twofish's blocks once, and carry them in words + [#130](https://github.com/kazu-yamamoto/crypton/pull/130) +* Put Camellia in C + [#131](https://github.com/kazu-yamamoto/crypton/pull/131) +* Route P-256 through the C implementation it already had + [#132](https://github.com/kazu-yamamoto/crypton/pull/132) +* Fold instead of dividing in the generic curve arithmetic + [#133](https://github.com/kazu-yamamoto/crypton/pull/133) +* Reduce the binary field by folding, and work a byte and a nibble at a time + [#134](https://github.com/kazu-yamamoto/crypton/pull/134) +* Stop running a Fermat test Miller-Rabin subsumes + [#135](https://github.com/kazu-yamamoto/crypton/pull/135) +* Make expSafe hide the exponent again + [#136](https://github.com/kazu-yamamoto/crypton/pull/136) +* Square, and multiply, faster in expSafe + [#137](https://github.com/kazu-yamamoto/crypton/pull/137) +* Invert the signing nonce without a side channel + [#138](https://github.com/kazu-yamamoto/crypton/pull/138) +* Keep the P-256 signature out of Integer arithmetic + [#139](https://github.com/kazu-yamamoto/crypton/pull/139) +* Add at every bit in the prime-curve multiplication, which laziness was skipping + [#140](https://github.com/kazu-yamamoto/crypton/pull/140) +* Multiply points in C on curves over a prime field + [#141](https://github.com/kazu-yamamoto/crypton/pull/141) +* A ladder for the curves over a binary field + [#142](https://github.com/kazu-yamamoto/crypton/pull/142) +* Work RSA's qinv out without the extended Euclidean algorithm + [#143](https://github.com/kazu-yamamoto/crypton/pull/143) +* Keep the RSA blinding factor out of the extended algorithm + [#144](https://github.com/kazu-yamamoto/crypton/pull/144) +* Unroll the inner loop at four and two as well + [#145](https://github.com/kazu-yamamoto/crypton/pull/145) +* Keep a table for each curve's base point + [#146](https://github.com/kazu-yamamoto/crypton/pull/146) +* Start R squared at the top of the modulus, and why folding did not pay + [#147](https://github.com/kazu-yamamoto/crypton/pull/147) +* Do the binary field arithmetic in C + [#148](https://github.com/kazu-yamamoto/crypton/pull/148) +* Use the x86 carry-less multiply where the processor has it + [#149](https://github.com/kazu-yamamoto/crypton/pull/149) +* Close the two testing gaps: one multiplication for both APIs, one place for each buffer's size + [#150](https://github.com/kazu-yamamoto/crypton/pull/150) +* Ask aarch64 for its carry-less multiply as well + [#151](https://github.com/kazu-yamamoto/crypton/pull/151) +* Work the RSA private exponent out without the extended algorithm + [#152](https://github.com/kazu-yamamoto/crypton/pull/152) +* Fewer Miller-Rabin rounds for a candidate nobody chose + [#153](https://github.com/kazu-yamamoto/crypton/pull/153) +* Blowfish, and the key setup bcrypt wraps it in, in C + [#154](https://github.com/kazu-yamamoto/crypton/pull/154) +* perf(sha256): use the Intel SHA extensions on x86-64 + [#155](https://github.com/kazu-yamamoto/crypton/pull/155) +* perf(aes): AES-192 through the processor's AES instructions + [#156](https://github.com/kazu-yamamoto/crypton/pull/156) +* perf(aes): build the AArch64 key schedule with AESE, not the S-box table + [#157](https://github.com/kazu-yamamoto/crypton/pull/157) +* test(aes): run the XTS vectors, and OCB and CCM at 192 and 256 bits + [#158](https://github.com/kazu-yamamoto/crypton/pull/158) +* perf(ocb): drive OCB through the ECB paths a group at a time + [#159](https://github.com/kazu-yamamoto/crypton/pull/159) +* perf(gcm): take the GHASH of the group before, alongside this group's rounds + [#160](https://github.com/kazu-yamamoto/crypton/pull/160) +* perf(sha): compute the message schedule in vector registers on x86 + [#161](https://github.com/kazu-yamamoto/crypton/pull/161) +* perf(chacha): combine as the keystream comes out of the registers + [#162](https://github.com/kazu-yamamoto/crypton/pull/162) +* perf(poly1305): shorten the carry chain and stop spilling the loop + [#163](https://github.com/kazu-yamamoto/crypton/pull/163) +* docs(sidechannel): say what the prime-field modules keep from the clock + [#164](https://github.com/kazu-yamamoto/crypton/pull/164) +* perf(sha1): use the Intel SHA extensions on x86-64 + [#165](https://github.com/kazu-yamamoto/crypton/pull/165) +* refactor(aes): drop the keystream generator nobody can call + [#166](https://github.com/kazu-yamamoto/crypton/pull/166) +* build: compile the C at -O3 + [#167](https://github.com/kazu-yamamoto/crypton/pull/167) +* perf(modes): stop the generic cipher modes allocating per byte + [#168](https://github.com/kazu-yamamoto/crypton/pull/168) +* perf(poly1305): four blocks at a time with NEON + [#169](https://github.com/kazu-yamamoto/crypton/pull/169) +* perf(sha1): use the ARMv8 SHA-1 instructions + [#170](https://github.com/kazu-yamamoto/crypton/pull/170) +* perf(sha3): use the ARMv8.2 SHA-3 instructions + [#171](https://github.com/kazu-yamamoto/crypton/pull/171) +* perf(gcm): the CRYPTOGAMS stitched AES-GCM on x86-64 + [#172](https://github.com/kazu-yamamoto/crypton/pull/172) +* perf(chacha): the CRYPTOGAMS ChaCha20 on AArch64 + [#173](https://github.com/kazu-yamamoto/crypton/pull/173) +* perf(poly1305): the CRYPTOGAMS Poly1305 on AArch64 + [#174](https://github.com/kazu-yamamoto/crypton/pull/174) +* perf(sha256): the CRYPTOGAMS SHA-256 on AArch64 + [#175](https://github.com/kazu-yamamoto/crypton/pull/175) +* perf(poly1305): the CRYPTOGAMS Poly1305 on x86-64 too + [#176](https://github.com/kazu-yamamoto/crypton/pull/176) +* perf(chacha): the CRYPTOGAMS ChaCha20 on x86-64 too + [#177](https://github.com/kazu-yamamoto/crypton/pull/177) +* perf(sha2): the CRYPTOGAMS SHA-256 and SHA-512 on x86-64 + [#178](https://github.com/kazu-yamamoto/crypton/pull/178) +* perf(sha1): hand the SHA-1 block loop a run of blocks, not one at a time + [#179](https://github.com/kazu-yamamoto/crypton/pull/179) +* perf(xts): double the tweak in the integer registers + [#180](https://github.com/kazu-yamamoto/crypton/pull/180) +* perf(sha3): take the CRYPTOGAMS Keccak for AArch64 + [#181](https://github.com/kazu-yamamoto/crypton/pull/181) +* perf(sha1): take the CRYPTOGAMS SHA-1 for AArch64 + [#182](https://github.com/kazu-yamamoto/crypton/pull/182) +* docs: put the performance tables in the README + [#183](https://github.com/kazu-yamamoto/crypton/pull/183) +* perf(sha3): take the CRYPTOGAMS Keccak for x86-64 as well + [#184](https://github.com/kazu-yamamoto/crypton/pull/184) +* docs: rebuild the performance tables + [#185](https://github.com/kazu-yamamoto/crypton/pull/185) +* perf(ecc): stop sharing the doublings in the double multiplication + [#186](https://github.com/kazu-yamamoto/crypton/pull/186) +* perf(number): count bytes from the bit count, not from base 256 + [#187](https://github.com/kazu-yamamoto/crypton/pull/187) +* perf(p256): inline the field arithmetic on AArch64 + [#188](https://github.com/kazu-yamamoto/crypton/pull/188) +* fix(api): name the reporting variants try..., not with an apostrophe + [#189](https://github.com/kazu-yamamoto/crypton/pull/189) +* fix(ecc): require a public point to be in the prime-order subgroup + [#190](https://github.com/kazu-yamamoto/crypton/pull/190) +* fix(pubkey): stop printing private keys, and add Crypto.Debug + [#191](https://github.com/kazu-yamamoto/crypton/pull/191) +* fix(poly1305): take a checked key, so that initializing cannot fail + [#192](https://github.com/kazu-yamamoto/crypton/pull/192) +* fix(bcrypt): refuse a cost bcrypt does not have rather than substituting one + [#194](https://github.com/kazu-yamamoto/crypton/pull/194) +* chore: build without a warning + [#195](https://github.com/kazu-yamamoto/crypton/pull/195) +* docs: build the documentation without a warning + [#196](https://github.com/kazu-yamamoto/crypton/pull/196) + +## 1.1.5 + +* fix(aead): reject undersized tags + [#80](https://github.com/kazu-yamamoto/crypton/pull/80) +* fix(aes): refuse a zero-length AES-GCM IV + [#79](https://github.com/kazu-yamamoto/crypton/pull/79) +* fix(p256): prevent crashes when validating valid points + [#78](https://github.com/kazu-yamamoto/crypton/pull/78) +* feat(asn1): add SHA-3 HashAlgorithmASN1 instances for PKCS#1 v1.5 + [#77](https://github.com/kazu-yamamoto/crypton/pull/77) +* OCB3 conformance + [#76](https://github.com/kazu-yamamoto/crypton/pull/76) + +## 1.1.4 + +* Generic instance for RSA PublicKey and PrivateKey + +## 1.1.3 + +* Ensure that `pointAdd` in `PubKey.ECC.P256` treats the point at infinity as the additive identity. + [#73](https://github.com/kazu-yamamoto/crypton/pull/73) + +## 1.1.2 + +* Preparing `ram` v0.22. +* Generalizing RSA encrypt/decrypt to manipulate ScrubbedBytes directly. + +## 1.1.1 + +* On iOS, ScrubbedBytes based hashing is used for seedNew. On other + plateforms, entropy is used directly as used to be. + [#71](https://github.com/kazu-yamamoto/crypton/pull/71) + +## 1.1.0 + +* Removing "basement" and "memory". + [#67](https://github.com/kazu-yamamoto/crypton/pull/67) + + +## 1.0.7 + +* Stop depending on basement, use upstream dependencies instead +* Stop transitively depending on basement by depending on ram. + +## 1.0.6 + +* Fix test failures on less common 64-bit arches. + [#65](https://github.com/kazu-yamamoto/crypton/pull/65) + +## 1.0.5 + +* Setter/Getter for ChaCha counter. + [#63](https://github.com/kazu-yamamoto/crypton/pull/63) +* Add simple interface to generate full blocks + [#60](https://github.com/kazu-yamamoto/crypton/pull/60) +* Avoid `ghc-prim` dependency. + [#61](https://github.com/kazu-yamamoto/crypton/pull/61) + +## 1.0.4 + +* Ed448.sign: avoid extra re-derive of public key. + [#48](https://github.com/kazu-yamamoto/crypton/pull/48) + +## 1.0.3 + +* Make sign of Ed25519/Ed448 safer. The public key parameter is + ignored and its public key is generated from the secret key + parameter to prevent Double Public Key Signing Function Oracle + Attack. + [#47](https://github.com/kazu-yamamoto/crypton/pull/47) + +## 1.0.2 + +* Deterministic Nonce Generation for ECDSA + [#46](https://github.com/kazu-yamamoto/crypton/pull/46) +* ECDSA Signature Normalization. + [#45](https://github.com/kazu-yamamoto/crypton/pull/45) +* Add Full Test Suite from RFC 6979. + [#44](https://github.com/kazu-yamamoto/crypton/pull/44) +* ECDSA with Public Key Recovery. + [#43](https://github.com/kazu-yamamoto/crypton/pull/43) +* Providing necessary features for HPKE. + [#42](https://github.com/kazu-yamamoto/crypton/pull/42) + +## 1.0.1 + +* Update decaf library. + [#38](https://github.com/kazu-yamamoto/crypton/pull/38) +* Add TypeOperators language extension to EdDSA.hs. + [#36](https://github.com/kazu-yamamoto/crypton/pull/36) + +## 1.0.0 + +* Versions follow the standard version policy. +* Removing pthread stuff. + [#32](https://github.com/kazu-yamamoto/crypton/pull/32) + +## 0.34 + +* Hashing getRandomBytes before using as Seed for ChaChaDRG + [#24](https://github.com/kazu-yamamoto/crypton/pull/24) +* Add support for XChaCha and XChaChaPoly1305 + [#18](https://github.com/kazu-yamamoto/crypton/pull/18) +* Strict byteArray of IV c + [#16](https://github.com/kazu-yamamoto/crypton/pull/16) + +## 0.33 + +* Add "crypton_" prefix to the final C symbols. + [#9](https://github.com/kazu-yamamoto/crypton/pull/9) + +## 0.32 + +* All C symbols now have the "crypton_" prefix. + [#7](https://github.com/kazu-yamamoto/crypton/pull/7) + [#8](https://github.com/kazu-yamamoto/crypton/pull/8) + +## 0.31 + +* Crypton is forked from cryptonite with the original authors permission. +* Ignoring exceptons from hClose to read the next entropy + [#1](https://github.com/kazu-yamamoto/crypton/pull/1) +* Enabling the support_pclmuldq flag by default. + +## 0.30 + +* Fix some C symbol blake2b prefix to be cryptonite_ prefix (fix mixing with other C library) +* add hmac-lazy +* Fix compilation with GHC 9.2 +* Drop support for GHC8.0, GHC8.2, GHC8.4, GHC8.6 + +## 0.29 + +* advance compilation with gmp breakage due to change upstream +* Add native EdDSA support + +## 0.28 + +* Add hash constant time capability +* Prevent possible overflow during hashing by hashing in 4GB chunks + +## 0.27 + +* Optimise AES GCM and CCM +* Optimise P256R1 implementation +* Various AES-NI building improvements +* Add better ECDSA support +* Add XSalsa derive +* Implement square roots for ECC binary curve +* Various tests and benchmarks + +## 0.26 + +* Add Rabin cryptosystem (and variants) +* Add bcrypt_pbkdf key derivation function +* Optimize Blowfish implementation +* Add KMAC (Keccak Message Authentication Code) +* Add ECDSA sign/verify digest APIs +* Hash algorithms with runtime output length +* Update blake2 to latest upstream version +* RSA-PSS with arbitrary key size +* SHAKE with output length not divisible by 8 +* Add Read and Data instances for Digest type +* Improve P256 scalar primitives +* Fix hash truncation bug in DSA +* Fix cost parsing for bcrypt +* Fix ECC failures on arm64 +* Correction to PKCS#1 v1.5 padding +* Use powModSecInteger when available +* Drop GHC 7.8 and GHC 7.10 support, refer to pkg-guidelines +* Optimise GCM mode +* Add little endian serialization of integer + +## 0.25 + +* Improve digest binary conversion efficiency +* AES CCM support +* Add MonadFailure instance for CryptoFailable +* Various misc improvements on documentation +* Edwards25519 lowlevel arithmetic support +* P256 add point negation +* Improvement in ECC (benchmark, better normalization) +* Blake2 improvements to context size +* Use gauge instead of criterion +* Use haskell-ci for CI scripts +* Improve Digest memory representation to be 2 less Ints and one less boxing + moving from `UArray` to `Block` + +## 0.24 + +* Ed25519: generateSecret & Documentation updates +* Repair tutorial +* RSA: Allow signing digest directly +* IV add: fix overflow behavior +* P256: validate point when decoding +* Compilation fix with deepseq disabled +* Improve Curve448 and use decaf for Ed448 +* Compilation flag blake2 sse merged in sse support +* Process unaligned data better in hashes and AES, on architecture needing alignment +* Drop support for ghc 7.6 +* Add ability to create random generator Seed from binary data and + loosen constraint on ChaChaDRG seed from ByteArray to ByteArrayAccess. +* Add 3 associated types with the HashAlgorithm class, to get + access to the constant for BlockSize, DigestSize and ContextSize at the type level. + the related function that this replaced will be deprecated in later release, and + eventually removed. + +API CHANGES: + +* Improve ECDH safety to return failure for bad inputs (e.g. public point in small order subgroup). + To go back to previous behavior you can replace `ecdh` by `ecdhRaw`. It's recommended to + use `ecdh` and handle the error appropriately. +* Users defining their own HashAlgorithm needs to define the + HashBlockSize, HashDigest, HashInternalContextSize associated types + +## 0.23 + +* Digest memory usage improvement by using unpinned memory +* Fix generateBetween to generate within the right bounds +* Add pure Twofish implementation +* Fix memory allocation in P256 when using a temp point +* Consolidate hash benchmark code +* Add Nat-length Blake2 support (GHC > 8.0) +* Update tutorial + +## 0.22 + +* Add Argon2 (Password Hashing Competition winner) hash function +* Update blake2 to latest upstream version +* Add extra blake2 hashing size +* Add faster PBKDF2 functions for SHA1/SHA256/SHA512 +* Add SHAKE128 and SHAKE256 +* Cleanup prime generation, and add tests +* Add Time-based One Time Password (TOTP) and HMAC-based One Time Password (HOTP) +* Rename Ed448 module name to Curve448, old module name still valid for now + +## 0.21 + +* Drop automated tests with GHC 7.0, GHC 7.4, GHC 7.6. support dropped, but probably still working. +* Improve non-aligned support in C sources, ChaCha and SHA3 now probably work on arch without support for unaligned access. not complete or tested. +* Add another ECC framework that is more flexible, allowing different implementations to work instead of + the existing Pure haskell NIST implementation. +* Add ECIES basic primitives +* Add XSalsa20 stream cipher +* Process partial buffer correctly with Poly1305 + +## 0.20 + +* Fixed hash truncation used in ECDSA signature & verification (Olivier Chéron) +* Fix ECDH when scalar and coordinate bit sizes differ (Olivier Chéron) +* Speed up ECDSA verification using Shamir's trick (Olivier Chéron) +* Fix rdrand on windows + +## 0.19 + +* Add tutorial (Yann Esposito) +* Derive Show instance for better interaction with Show pretty printer (Eric Mertens) + +## 0.18 + +* Re-used standard rdrand instructions instead of bytedump of rdrand instruction +* Improvement to F2m, including lots of tests (Andrew Lelechenko) +* Add error check on salt length in bcrypt + +## 0.17 + +* Add Miyaguchi-Preneel construction (Kei Hibino) +* Fix buffer length in scrypt (Luke Taylor) +* build fixes for i686 and arm related to rdrand + +## 0.16 + +* Fix basepoint for Ed448 + +* Enable 64-bit Curve25519 implementation + +## 0.15 + +* Fix serialization of DH and ECDH + +## 0.14 + +* Reduce size of SHA3 context instead of allocating all-size fit memory. save + up to 72 bytes of memory per context for SHA3-512. +* Add a Seed capability to the main DRG, to be able to debug/reproduce randomized program + where you would want to disable the randomness. +* Add support for Cipher-based Message Authentication Code (CMAC) (Kei Hibino) +* *CHANGE* Change the `SharedKey` for `Crypto.PubKey.DH` and `Crypto.PubKey.ECC.DH`, + from an Integer newtype to a ScrubbedBytes newtype. Prevent mistake where the + bytes representation is generated without the right padding (when needed). +* *CHANGE* Keep The field size in bits, in the `Params` in `Crypto.PubKey.DH`, + moving from 2 elements to 3 elements in the structure. + +## 0.13 + +* *SECURITY* Fix buffer overflow issue in SHA384, copying 16 extra bytes from + the SHA512 context to the destination memory pointer leading to memory + corruption, segfault. (Mikael Bung) + +## 0.12 + +* Fix compilation issue with Ed448 on 32 bits machine. + +## 0.11 + +* Truncate hashing correctly for DSA +* Add support for HKDF (RFC 5869) +* Add support for Ed448 +* Extends support for Blake2s to 224 bits version. +* Compilation workaround for old distribution (RHEL 4.1) +* Compilation fix for AIX +* Compilation fix with AESNI and ghci compiling C source in a weird order. +* Fix example compilation, typo, and warning + +## 0.10 + +* Add reference implementation of blake2 for non-SSE2 platform +* Add support\_blake2\_sse flag + +## 0.9 + +* Quiet down unused module imports +* Move Curve25519 over to Crypto.Error instead of using Either String. +* Add documentation for ChaChaPoly1305 +* Add missing documentation for various modules +* Add a way to create Poly1305 Auth tag. +* Added support for the BLAKE2 family of hash algorithms +* Fix endianness of incrementNonce function for ChaChaPoly1305 + +## 0.8 + +* Add support for ChaChaPoly1305 Nonce Increment (John Galt) +* Move repository to the haskell-crypto organisation + +## 0.7 + +* Add PKCS5 / PKCS7 padding and unpadding methods +* Fix ChaChaPoly1305 Decryption +* Add support for BCrypt (Luke Taylor) + +## 0.6 + +* Add ChaChaPoly1305 AE cipher +* Add instructions in README for building on old OSX +* Fix blocking /dev/random Andrey Sverdlichenko + +## 0.5 + +* Fix all strays exports to all be under the cryptonite prefix. + +## 0.4 + +* Add a System DRG that represent a referentially transparent of evaluated bytes + while using lazy evaluation for future entropy values. + +## 0.3 + +* Allow drgNew to run in any MonadRandom, providing cascading initialization +* Remove Crypto.PubKey.HashDescr in favor of just having the algorithm + specified in PKCS15 RSA function. +* Fix documentation in cipher sub section (Luke Taylor) +* Cleanup AES dead functions (Luke Taylor) +* Fix Show instance of Digest to display without quotes similar to cryptohash +* Use scrubbed bytes instead of bytes for P256 scalar + +## 0.2 + +* Fix P256 compilation and exactness, + add tests +* Add a raw memory number serialization capability (i2osp, os2ip) +* Improve tests for number serialization +* Improve tests for ECC arithmetics +* Add Ord instance for Digest (Nicolas Di Prima) +* Fix entropy compilation on windows 64 bits. + +## 0.1 + +* Initial release + diff --git a/Crypto/Cipher/AES/GCM.hs b/Crypto/Cipher/AES/GCM.hs new file mode 100644 --- /dev/null +++ b/Crypto/Cipher/AES/GCM.hs @@ -0,0 +1,256 @@ +-- | +-- Module : Crypto.Cipher.AES.GCM +-- License : BSD-style +-- Maintainer : Kazu Yamamoto +-- Stability : experimental +-- Portability : unknown +-- +-- AES-GCM for callers that send many short messages under one key, which is +-- what a datagram transport does. +-- +-- The interface in "Crypto.Cipher.Types" builds a state from the key /and/ +-- the nonce and then walks it through appending the additional data, +-- encrypting and finalizing, copying the state at each step. For a stream +-- that is nothing next to the encryption. For a QUIC packet it is most of +-- the work: the key schedule and the table of multiples of @H@ depend on the +-- key alone, and rebuilding them for every nonce costs more than encrypting +-- 1440 bytes. +-- +-- So here a t'Context' is built from the key once and holds both, and +-- 'encrypt' takes a nonce and a whole message and answers in one call. +-- +-- > ctx <- throwCryptoError <$> pure (newContext key) +-- > let packet = encrypt ctx nonce header plaintext 16 +-- +-- This runs on AES-NI and carry-less multiply, or on the ARMv8 cryptographic +-- extension, and makes no branch and no memory access that depends on the key +-- or on the data. Where the processor has neither, AES falls back to a table +-- driven implementation that is /not/ constant time; see the side channels +-- section of the README, and 'Crypto.System.CPU.processorOptions' for which is +-- in use. +-- +-- The result is the ciphertext with the tag after it, which is the shape a +-- packet wants. 'decrypt' takes that shape back, compares the tag itself and +-- answers 'Nothing' when it does not match. +-- +-- This computes the same thing as the general interface; the tests hold it to +-- that on the same vectors. +module Crypto.Cipher.AES.GCM ( + Context, + newContext, + encrypt, + decrypt, + decryptWithTag, + + -- * Header protection + HeaderKey, + newHeaderKey, + encryptWithMask, +) where + +import Crypto.Cipher.AES.Primitive ( + AES, + AESGCMKey, + gcmFullDecrypt, + gcmFullDecryptTag, + gcmFullEncrypt, + gcmFullEncryptMask, + gcmKeyInit, + initAES, + ) +import Crypto.Cipher.Types (AuthTag) +import Crypto.Cipher.Types.AEAD (minimumTagLength) +import Crypto.Error +import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess) +import qualified Crypto.Internal.ByteArray as B +import Data.Word (Word8) +import Foreign.Ptr (Ptr) + +-- | Everything a key determines: the AES key schedule and the table of +-- multiples of @H@. Build it once and encrypt as many messages under it as +-- the key is good for. +data Context = Context !AES !AESGCMKey + +-- | Take a key of 16, 24 or 32 bytes. Any other length is reported as +-- 'CryptoError_KeySizeInvalid'. +newContext :: ByteArrayAccess key => key -> CryptoFailable Context +newContext k = do + aes <- initAES k + return $ Context aes (gcmKeyInit aes) + +-- | Encrypt one message: the nonce, the additional data that is +-- authenticated but not encrypted, the plaintext, and how many bytes of tag +-- to produce, which GCM allows between 4 and 16. Any other length throws +-- 'CryptoError_AuthenticationTagSizeInvalid', and so does 'decryptWithTag'. +-- +-- The answer is the ciphertext followed by the tag. +-- +-- A nonce must not be used twice with the same t'Context'. Twelve bytes is +-- the size GCM is defined for and the only one that does not cost a further +-- pass. A nonce of no bytes is refused: it would hand out the key GCM +-- authenticates with, so 'encrypt' and 'decryptWithTag' throw +-- 'CryptoError_IvSizeInvalid' for it, 'decrypt' gives 'Nothing' and +-- 'encryptWithMask' gives 'False'. +{-# INLINABLE encrypt #-} +encrypt + :: ( ByteArrayAccess nonce + , ByteArrayAccess aad + , ByteArrayAccess ba + , ByteArray output + ) + => Context + -> nonce + -> aad + -> ba + -> Int + -> output +encrypt (Context aes gk) nonce aad input taglen + | tooLongForC aad input = + throwCryptoError (CryptoFailed CryptoError_ParameterInvalid) + | badNonce nonce = + throwCryptoError (CryptoFailed CryptoError_IvSizeInvalid) + | badTagLength taglen = + throwCryptoError (CryptoFailed CryptoError_AuthenticationTagSizeInvalid) + | otherwise = gcmFullEncrypt aes gk nonce aad input taglen + +-- | Decrypt one message, in the shape 'encrypt' produced: the ciphertext with +-- its tag after it. The tag is compared here, every byte of it whatever the +-- answer, and a message whose tag does not match gives 'Nothing' rather than +-- the plaintext. +-- +-- 'Nothing' also comes back when the nonce has no bytes, the input is +-- shorter than the tag, or the tag length is outside 4 to 16. +{-# INLINABLE decrypt #-} +decrypt + :: (ByteArrayAccess nonce, ByteArrayAccess aad, ByteArray ba) + => Context + -> nonce + -> aad + -> ba + -> Int + -> Maybe ba +decrypt (Context aes gk) nonce aad input taglen + | tooLongForC aad input = Nothing + | badNonce nonce || badTagLength taglen || B.length input < taglen = Nothing + | otherwise = gcmFullDecrypt aes gk nonce aad body tag + where + (body, tag) = B.splitAt (B.length input - taglen) input + +-- | Decrypt one message, the tag kept apart, and hand back the tag this end +-- computed. +-- +-- For a caller whose protocol hands it the tag separately from the +-- ciphertext, so that 'decrypt' -- which wants the two together and compares +-- them itself -- does not fit. Compare the two tags with '=='; the 'Eq' +-- instance of t'AuthTag' is a constant-time comparison, and taking them apart +-- to compare the bytes is how this goes wrong. +-- +-- Nothing here says whether the message is authentic. Until the comparison +-- is made and has come out equal, what this returns is not plaintext, it is +-- what the ciphertext turns into, and a caller must not act on it. +{-# INLINABLE decryptWithTag #-} +decryptWithTag + :: (ByteArrayAccess nonce, ByteArrayAccess aad, ByteArray ba) + => Context + -> nonce + -> aad + -> ba + -> Int + -> (ba, AuthTag) +decryptWithTag (Context aes gk) nonce aad input taglen + | tooLongForC aad input = + throwCryptoError (CryptoFailed CryptoError_ParameterInvalid) + | badNonce nonce = + throwCryptoError (CryptoFailed CryptoError_IvSizeInvalid) + | badTagLength taglen = + throwCryptoError (CryptoFailed CryptoError_AuthenticationTagSizeInvalid) + | otherwise = gcmFullDecryptTag aes gk nonce aad input taglen + +---------------------------------------------------------------- + +-- | The key schedule for header protection, which QUIC keeps separately from +-- the one it encrypts with. Built once, like a t'Context'. +newtype HeaderKey = HeaderKey AES + +-- | Take a header protection key of 16, 24 or 32 bytes. +newHeaderKey :: ByteArrayAccess key => key -> CryptoFailable HeaderKey +newHeaderKey k = HeaderKey <$> initAES k + +-- | Encrypt one message and, from a sample of the ciphertext it just +-- produced, make the header protection mask -- in one call, into two buffers +-- the caller already has. +-- +-- QUIC takes its sample from the ciphertext, so the mask cannot be had before +-- the encryption. It can be had before coming back, and with the buffers +-- already there nothing is allocated for either. On an Apple M4 the mask +-- then costs about 0.02 us, where asking for it separately costs 0.11. +-- +-- The sealed message wants @length input + taglen@ bytes and the mask +-- sixteen. @sampleOffset@ says where the sixteen bytes of sample begin in +-- the sealed message, counting the tag as part of it. +-- +-- 'False' comes back, and nothing is written, when the nonce has no bytes, +-- the sample would not fit, or the tag length is outside 4 to 16. +{-# INLINABLE encryptWithMask #-} +encryptWithMask + :: (ByteArrayAccess nonce, ByteArrayAccess aad, ByteArrayAccess ba) + => Context + -> HeaderKey + -> nonce + -> aad + -> ba + -> Int + -- ^ tag length + -> Int + -- ^ sample offset + -> Ptr Word8 + -- ^ where the sealed message goes + -> Ptr Word8 + -- ^ where the sixteen bytes of mask go + -> IO Bool +encryptWithMask (Context aes gk) (HeaderKey hp) nonce aad input taglen off outp maskp + | tooLongForC aad input = return False + | badNonce nonce = return False + | off < 0 || badTagLength taglen || off + 16 > B.length input + taglen = + return False + | otherwise = do + gcmFullEncryptMask aes gk hp nonce aad input taglen off outp maskp + return True + +-- | The C behind all four takes its lengths as @uint32_t@, so a message or +-- its additional data from 2^32 bytes up cannot be handed to it: the length +-- would be truncated and most of the buffer left untouched, with nothing to +-- say so. There is no splitting the work here -- the C does the whole +-- message in one call, tag and all -- so such a message is refused. +tooLongForC + :: (ByteArrayAccess aad, ByteArrayAccess ba) => aad -> ba -> Bool +tooLongForC aad input = + B.overCLength (B.length aad) || B.overCLength (B.length input) + +-- | SP 800-38D 5.2.1.1 asks for at least one byte of IV, and this is why. +-- +-- GCM builds its pre-counter block from a nonce that is not twelve bytes as +-- @J0 = GHASH_H(IV || 0^s || [0]_64 || [len(IV)]_64)@. For an empty IV that +-- input is one block of zeros, so @J0@ is zero, and the tag of a message +-- becomes @GHASH_H(A, C) XOR E(K, 0^128)@ -- where @E(K, 0^128)@ is the +-- definition of @H@. The tag of an empty message under an empty nonce is +-- therefore @H@ itself, and any other full tag gives @H@ as the root of a +-- known polynomial. +-- +-- @H@ belongs to the key, not to the nonce. An attacker holding it, plus one +-- genuine message under any nonce, has @E(K, J0)@ for that nonce and can make +-- a tag that verifies for data of their own -- under a correct twelve-byte +-- nonce, and for every nonce they have seen. One encryption with an empty +-- nonce and a full tag ends the authenticity of everything under the key. +-- +-- 'Crypto.Cipher.AES.Primitive.gcmAeadInit' refuses the empty IV for this +-- reason, and these four have to as well. Only the empty one is refused: +-- SP 800-38D allows every length from one byte up. +badNonce :: ByteArrayAccess nonce => nonce -> Bool +badNonce nonce = B.length nonce == 0 + +-- | GCM makes a sixteen-byte tag and a shorter one is a prefix of it. Below +-- 'minimumTagLength' it authenticates next to nothing, and past sixteen the +-- C code would read beyond the tag it computed. +badTagLength :: Int -> Bool +badTagLength t = t < minimumTagLength || t > 16 diff --git a/Crypto/Cipher/AES/Primitive.hs b/Crypto/Cipher/AES/Primitive.hs --- a/Crypto/Cipher/AES/Primitive.hs +++ b/Crypto/Cipher/AES/Primitive.hs @@ -40,6 +40,12 @@ -- * Incremental GCM gcmMode, gcmInit, + AESGCMKey, + gcmKeyInit, + gcmFullEncrypt, + gcmFullEncryptMask, + gcmFullDecrypt, + gcmFullDecryptTag, gcmAeadInit, -- * Incremental OCB @@ -158,6 +164,14 @@ sizeGCM :: Int sizeGCM = 320 +-- | The size of what a key determines, which is the 320 bytes above and the +-- powers of H the fused path reads: sixteen of them, and sixteen more for +-- the term the Karatsuba multiplication would otherwise work out every time. +-- The same on every platform, so that this is one number rather than one per +-- architecture; the powers are filled only where that path is compiled in. +sizeGCMKey :: Int +sizeGCMKey = 832 + sizeOCB :: Int sizeOCB = 160 @@ -257,6 +271,7 @@ -- ^ ciphertext output encryptCTR ctx iv input | len <= 0 = B.empty + | B.overCLength len = error tooLongMessage | B.length iv /= 16 = error $ "AES error: IV length must be block size (16). Its length is: " @@ -351,6 +366,20 @@ c_aes_encrypt_c32 (castPtr o) k v i (fromIntegral len) len = B.length input +-- | What the AES modes say when a message cannot be given to the C, whose +-- lengths are @uint32_t@. Above that the length is truncated on the way +-- down and most of the buffer is left as it was found, with nothing to say +-- so, which is worse than refusing. +-- +-- Unlike the stream ciphers, these cannot be done in pieces: the C is handed +-- the IV and does not hand it back, so a second call would start from the +-- wrong place. ECB, CBC and XTS count blocks rather than bytes, so their +-- limit is sixteen times further out than CTR's. +tooLongMessage :: String +tooLongMessage = + "AES error: message too long for this implementation, whose C takes its " + ++ "lengths as uint32_t" + {-# INLINE doECB #-} doECB :: ByteArray ba @@ -359,6 +388,7 @@ -> ba -> ba doECB f ctx input + | B.overCLength nbBlocks = error tooLongMessage | len == 0 = B.empty | r /= 0 = error $ @@ -382,6 +412,7 @@ -> ba -> ba doCBC f ctx (IV iv) input + | B.overCLength nbBlocks = error tooLongMessage | len == 0 = B.empty | r /= 0 = error $ @@ -405,6 +436,7 @@ -> ba -> ba doXTS f (key1, key2) iv spoint input + | B.overCLength nbBlocks = error tooLongMessage | len == 0 = B.empty | r /= 0 = error $ @@ -429,6 +461,190 @@ c_aes_gcm_init (castPtr gcmStPtr) k v (fromIntegral $ B.length iv) return $ AESGCM sm +-- | How long a message may be and still be handed to an unsafe foreign call. +-- Four kibibytes is about half a microsecond of work, and it takes in a +-- datagram of any size a network will carry. +shortMessage :: Int +shortMessage = 4096 + +-- | The part of a GCM state the key alone determines: H, which is the key +-- applied to a block of zeroes, and the table of its multiples. That is 256 +-- of the 320 bytes of a GCM state, and it is the same for every message sent +-- under one key, so a caller that keeps a key can build this once rather than +-- once for every message. +newtype AESGCMKey = AESGCMKey ScrubbedBytes + +-- | Build the key part of a GCM state. +{-# NOINLINE gcmKeyInit #-} +gcmKeyInit :: AES -> AESGCMKey +gcmKeyInit ctx = AESGCMKey $ B.allocAndFreeze sizeGCMKey $ \p -> + keyToPtr ctx $ \k -> c_aes_gcm_key_init (castPtr p) k + +-- | Authenticate and encrypt one message in a single call: the nonce, the +-- additional data, the plaintext and the tag, with no state crossing back +-- into Haskell in between. The result is the ciphertext followed by the tag. +{-# INLINABLE gcmFullEncrypt #-} +gcmFullEncrypt + :: (ByteArrayAccess iv, ByteArrayAccess aad, ByteArrayAccess ba, ByteArray output) + => AES -> AESGCMKey -> iv -> aad -> ba -> Int -> output +gcmFullEncrypt ctx (AESGCMKey gk) iv aad input taglen = + B.allocAndFreeze (B.length input + taglen) $ \out -> + B.withByteArray gk $ \gkp -> + keyToPtr ctx $ \k -> + B.withByteArray iv $ \ivp -> + B.withByteArray aad $ \aadp -> + B.withByteArray input $ \inp -> + call + out + (castPtr gkp) + k + ivp + (fromIntegral $ B.length iv) + aadp + (fromIntegral $ B.length aad) + inp + (fromIntegral $ B.length input) + (fromIntegral taglen) + where + -- An unsafe call keeps a capability for as long as it runs, so it is only + -- right for work that is over quickly. A message this side of + -- 'shortMessage' is, and it is the short ones the saving matters for: a + -- safe call costs about 0.075 us whatever the length, which is a fifth of + -- a 1440-byte packet and a percent of a 16 KiB record. + call + | B.length input <= shortMessage = c_aes_gcm_full_encrypt_unsafe + | otherwise = c_aes_gcm_full_encrypt + +-- | Encrypt, and from a sample of the ciphertext just produced make the +-- header protection mask, into buffers the caller owns. QUIC takes its +-- sample from the ciphertext, so the mask cannot be had before the +-- encryption; it can be had before coming back, and with the buffers already +-- there nothing is allocated for either. +-- +-- @sampleoff@ is where the sixteen bytes of sample begin in the output. +{-# INLINABLE gcmFullEncryptMask #-} +gcmFullEncryptMask + :: (ByteArrayAccess iv, ByteArrayAccess aad, ByteArrayAccess ba) + => AES + -> AESGCMKey + -> AES + -> iv + -> aad + -> ba + -> Int + -> Int + -> Ptr Word8 + -> Ptr Word8 + -> IO () +gcmFullEncryptMask ctx (AESGCMKey gk) hpctx iv aad input taglen sampleoff outp maskp = + B.withByteArray gk $ \gkp -> + keyToPtr ctx $ \k -> + keyToPtr hpctx $ \hk -> + B.withByteArray iv $ \ivp -> + B.withByteArray aad $ \aadp -> + B.withByteArray input $ \inp -> + call + outp + (castPtr gkp) + k + ivp + (fromIntegral $ B.length iv) + aadp + (fromIntegral $ B.length aad) + inp + (fromIntegral $ B.length input) + (fromIntegral taglen) + hk + (fromIntegral sampleoff) + maskp + where + call + | B.length input <= shortMessage = c_aes_gcm_full_encrypt_mask_unsafe + | otherwise = c_aes_gcm_full_encrypt_mask + +-- | The same the other way, with the tag compared here rather than by the +-- caller: 'Nothing' when it does not match, and every byte of it is looked at +-- either way. The ciphertext comes in without its tag, which is given +-- separately. +{-# INLINABLE gcmFullDecrypt #-} +gcmFullDecrypt + :: ( ByteArrayAccess iv + , ByteArrayAccess aad + , ByteArrayAccess ba + , ByteArrayAccess tag + , ByteArray output + ) + => AES -> AESGCMKey -> iv -> aad -> ba -> tag -> Maybe output +gcmFullDecrypt ctx (AESGCMKey gk) iv aad input tag = unsafeDoIO $ do + (r, out) <- B.allocRet (B.length input) $ \outp -> + B.withByteArray gk $ \gkp -> + keyToPtr ctx $ \k -> + B.withByteArray iv $ \ivp -> + B.withByteArray aad $ \aadp -> + B.withByteArray input $ \inp -> + B.withByteArray tag $ \tagp -> + call + outp + (castPtr gkp) + k + ivp + (fromIntegral $ B.length iv) + aadp + (fromIntegral $ B.length aad) + inp + (fromIntegral $ B.length input) + tagp + (fromIntegral $ B.length tag) + return $ if r /= 0 then Just out else Nothing + where + call + | B.length input <= shortMessage = c_aes_gcm_full_decrypt_unsafe + | otherwise = c_aes_gcm_full_decrypt + +-- | Decrypt one message and hand back the tag that was computed over it, +-- rather than comparing it here. +-- +-- For a caller that holds the expected tag in a form of its own and will +-- compare it itself. Compare the two t'AuthTag's with '==', whose instance +-- for that type is a constant-time comparison; taking them apart and +-- comparing the bytes is how this goes wrong. +-- +-- Where the tag simply arrives after the ciphertext, 'gcmFullDecrypt' is the +-- one to use: it compares in C and never puts a tag in the caller's hands. +{-# INLINABLE gcmFullDecryptTag #-} +gcmFullDecryptTag + :: ( ByteArrayAccess iv + , ByteArrayAccess aad + , ByteArrayAccess ba + , ByteArray output + ) + => AES -> AESGCMKey -> iv -> aad -> ba -> Int -> (output, AuthTag) +gcmFullDecryptTag ctx (AESGCMKey gk) iv aad input taglen = unsafeDoIO $ do + (tagbs, out) <- B.allocRet (B.length input) $ \outp -> + B.alloc taglen $ \tagp -> + B.withByteArray gk $ \gkp -> + keyToPtr ctx $ \k -> + B.withByteArray iv $ \ivp -> + B.withByteArray aad $ \aadp -> + B.withByteArray input $ \inp -> + call + outp + tagp + (castPtr gkp) + k + ivp + (fromIntegral $ B.length iv) + aadp + (fromIntegral $ B.length aad) + inp + (fromIntegral $ B.length input) + (fromIntegral taglen) + return (out, AuthTag $ B.convert (tagbs :: B.Bytes)) + where + call + | B.length input <= shortMessage = c_aes_gcm_full_decrypt_tag_unsafe + | otherwise = c_aes_gcm_full_decrypt_tag + -- | append data which is only going to be authenticated to the GCM context. -- -- needs to happen after initialization and before appending encryption/decryption data. @@ -439,7 +655,8 @@ doAppend = withNewGCMSt gcmSt $ \gcmStPtr -> withByteArray input $ \i -> - c_aes_gcm_aad gcmStPtr i (fromIntegral $ B.length input) + B.inCLengths (B.length input) $ \off n -> + c_aes_gcm_aad gcmStPtr (i `plusPtr` off) (fromIntegral n) -- | append data to encrypt and append to the GCM context -- @@ -453,7 +670,13 @@ doEnc gcmStPtr aesPtr = B.alloc len $ \o -> withByteArray input $ \i -> - c_aes_gcm_encrypt (castPtr o) gcmStPtr aesPtr i (fromIntegral len) + B.inCLengths len $ \off n -> + c_aes_gcm_encrypt + (castPtr o `plusPtr` off) + gcmStPtr + aesPtr + (i `plusPtr` off) + (fromIntegral n) -- | append data to decrypt and append to the GCM context -- @@ -467,7 +690,13 @@ doDec gcmStPtr aesPtr = B.alloc len $ \o -> withByteArray input $ \i -> - c_aes_gcm_decrypt (castPtr o) gcmStPtr aesPtr i (fromIntegral len) + B.inCLengths len $ \off n -> + c_aes_gcm_decrypt + (castPtr o `plusPtr` off) + gcmStPtr + aesPtr + (i `plusPtr` off) + (fromIntegral n) -- | Generate the Tag from GCM context {-# NOINLINE gcmFinish #-} @@ -500,9 +729,11 @@ -- The tag length is expressed in bytes and must be in [0..16]. -- The IV length must be in [1..15] bytes per RFC 7253. {-# NOINLINE ocbInitWithTagLength #-} -ocbInitWithTagLength :: ByteArrayAccess iv => AES -> iv -> Int -> CryptoFailable AESOCB +ocbInitWithTagLength + :: ByteArrayAccess iv => AES -> iv -> Int -> CryptoFailable AESOCB ocbInitWithTagLength ctx iv taglen - | taglen < 0 || taglen > 16 = CryptoFailed CryptoError_AuthenticationTagSizeInvalid + | taglen < 0 || taglen > 16 = + CryptoFailed CryptoError_AuthenticationTagSizeInvalid | ivlen < 1 || ivlen > 15 = CryptoFailed CryptoError_IvSizeInvalid | otherwise = CryptoPassed $ unsafeDoIO $ do sm <- B.alloc sizeOCB $ \ocbStPtr -> @@ -522,7 +753,9 @@ -- need to happen after initialization and before appending encryption/decryption data. {-# NOINLINE ocbAppendAAD #-} ocbAppendAAD :: ByteArrayAccess aad => AES -> AESOCB -> aad -> AESOCB -ocbAppendAAD ctx ocb input = unsafeDoIO (snd `fmap` withOCBKeyAndCopySt ctx ocb doAppend) +ocbAppendAAD ctx ocb input + | B.overCLength (B.length input) = error tooLongMessage + | otherwise = unsafeDoIO (snd `fmap` withOCBKeyAndCopySt ctx ocb doAppend) where doAppend ocbStPtr aesPtr = withByteArray input $ \i -> @@ -534,7 +767,9 @@ -- need to happen after AAD appending, or after initialization if no AAD data. {-# NOINLINE ocbAppendEncrypt #-} ocbAppendEncrypt :: ByteArray ba => AES -> AESOCB -> ba -> (ba, AESOCB) -ocbAppendEncrypt ctx ocb input = unsafeDoIO $ withOCBKeyAndCopySt ctx ocb doEnc +ocbAppendEncrypt ctx ocb input + | B.overCLength (B.length input) = error tooLongMessage + | otherwise = unsafeDoIO $ withOCBKeyAndCopySt ctx ocb doEnc where len = B.length input doEnc ocbStPtr aesPtr = @@ -548,7 +783,9 @@ -- need to happen after AAD appending, or after initialization if no AAD data. {-# NOINLINE ocbAppendDecrypt #-} ocbAppendDecrypt :: ByteArray ba => AES -> AESOCB -> ba -> (ba, AESOCB) -ocbAppendDecrypt ctx ocb input = unsafeDoIO $ withOCBKeyAndCopySt ctx ocb doDec +ocbAppendDecrypt ctx ocb input + | B.overCLength (B.length input) = error tooLongMessage + | otherwise = unsafeDoIO $ withOCBKeyAndCopySt ctx ocb doDec where len = B.length input doDec ocbStPtr aesPtr = @@ -608,7 +845,9 @@ -- needs to happen after initialization and before appending encryption/decryption data. {-# NOINLINE ccmAppendAAD #-} ccmAppendAAD :: ByteArrayAccess aad => AES -> AESCCM -> aad -> AESCCM -ccmAppendAAD ctx ccm input = unsafeDoIO $ snd <$> withCCMKeyAndCopySt ctx ccm doAppend +ccmAppendAAD ctx ccm input + | B.overCLength (B.length input) = error tooLongMessage + | otherwise = unsafeDoIO $ snd <$> withCCMKeyAndCopySt ctx ccm doAppend where doAppend ccmStPtr aesPtr = withByteArray input $ \i -> c_aes_ccm_aad ccmStPtr aesPtr i (fromIntegral $ B.length input) @@ -619,7 +858,9 @@ -- needs to happen after AAD appending, or after initialization if no AAD data. {-# NOINLINE ccmEncrypt #-} ccmEncrypt :: ByteArray ba => AES -> AESCCM -> ba -> (ba, AESCCM) -ccmEncrypt ctx ccm input = unsafeDoIO $ withCCMKeyAndCopySt ctx ccm cbcmacAndIv +ccmEncrypt ctx ccm input + | B.overCLength (B.length input) = error tooLongMessage + | otherwise = unsafeDoIO $ withCCMKeyAndCopySt ctx ccm cbcmacAndIv where len = B.length input cbcmacAndIv ccmStPtr aesPtr = @@ -633,7 +874,9 @@ -- needs to happen after AAD appending, or after initialization if no AAD data. {-# NOINLINE ccmDecrypt #-} ccmDecrypt :: ByteArray ba => AES -> AESCCM -> ba -> (ba, AESCCM) -ccmDecrypt ctx ccm input = unsafeDoIO $ withCCMKeyAndCopySt ctx ccm cbcmacAndIv +ccmDecrypt ctx ccm input + | B.overCLength (B.length input) = error tooLongMessage + | otherwise = unsafeDoIO $ withCCMKeyAndCopySt ctx ccm cbcmacAndIv where len = B.length input cbcmacAndIv ccmStPtr aesPtr = @@ -682,6 +925,131 @@ foreign import ccall "crypton_aes.h crypton_aes_encrypt_c32" c_aes_encrypt_c32 :: CString -> Ptr AES -> Ptr Word8 -> CString -> CUInt -> IO () + +foreign import ccall unsafe "crypton_aes.h crypton_aes_gcm_key_init" + c_aes_gcm_key_init :: Ptr AESGCM -> Ptr AES -> IO () + +foreign import ccall "crypton_aes.h crypton_aes_gcm_full_encrypt" + c_aes_gcm_full_encrypt + :: Ptr Word8 + -> Ptr AESGCM + -> Ptr AES + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> CUInt + -> IO () + +foreign import ccall "crypton_aes.h crypton_aes_gcm_full_decrypt" + c_aes_gcm_full_decrypt + :: Ptr Word8 + -> Ptr AESGCM + -> Ptr AES + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> IO CInt + +foreign import ccall "crypton_aes.h crypton_aes_gcm_full_decrypt_tag" + c_aes_gcm_full_decrypt_tag + :: Ptr Word8 + -> Ptr Word8 + -> Ptr AESGCM + -> Ptr AES + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> CUInt + -> IO () + +foreign import ccall unsafe "crypton_aes.h crypton_aes_gcm_full_decrypt_tag" + c_aes_gcm_full_decrypt_tag_unsafe + :: Ptr Word8 + -> Ptr Word8 + -> Ptr AESGCM + -> Ptr AES + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> CUInt + -> IO () + +foreign import ccall unsafe "crypton_aes.h crypton_aes_gcm_full_encrypt" + c_aes_gcm_full_encrypt_unsafe + :: Ptr Word8 + -> Ptr AESGCM + -> Ptr AES + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> CUInt + -> IO () + +foreign import ccall unsafe "crypton_aes.h crypton_aes_gcm_full_decrypt" + c_aes_gcm_full_decrypt_unsafe + :: Ptr Word8 + -> Ptr AESGCM + -> Ptr AES + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> IO CInt + +foreign import ccall "crypton_aes.h crypton_aes_gcm_full_encrypt_mask" + c_aes_gcm_full_encrypt_mask + :: Ptr Word8 + -> Ptr AESGCM + -> Ptr AES + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> CUInt + -> Ptr AES + -> CUInt + -> Ptr Word8 + -> IO () + +foreign import ccall unsafe "crypton_aes.h crypton_aes_gcm_full_encrypt_mask" + c_aes_gcm_full_encrypt_mask_unsafe + :: Ptr Word8 + -> Ptr AESGCM + -> Ptr AES + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> CUInt + -> Ptr AES + -> CUInt + -> Ptr Word8 + -> IO () foreign import ccall "crypton_aes.h crypton_aes_gcm_init" c_aes_gcm_init :: Ptr AESGCM -> Ptr AES -> Ptr Word8 -> CUInt -> IO () diff --git a/Crypto/Cipher/ChaCha.hs b/Crypto/Cipher/ChaCha.hs --- a/Crypto/Cipher/ChaCha.hs +++ b/Crypto/Cipher/ChaCha.hs @@ -183,10 +183,19 @@ | B.null src = (B.empty, prevSt) | otherwise = unsafeDoIO $ do (out, st) <- B.copyRet prevStMem $ \ctx -> - B.alloc (B.length src) $ \dstPtr -> + B.alloc n $ \dstPtr -> B.withByteArray src $ \srcPtr -> - ccrypton_chacha_combine dstPtr ctx srcPtr (fromIntegral $ B.length src) + -- in pieces the C's uint32_t length can hold; it carries + -- the state in ctx, so it can simply be called again + B.inCLengths n $ \off len -> + ccrypton_chacha_combine + (dstPtr `plusPtr` off) + ctx + (srcPtr `plusPtr` off) + (fromIntegral len) return (out, State st) + where + n = B.length src -- | Generate a number of bytes from the ChaCha output directly generate @@ -201,7 +210,11 @@ | otherwise = unsafeDoIO $ do (out, st) <- B.copyRet prevStMem $ \ctx -> B.alloc len $ \dstPtr -> - ccrypton_chacha_generate dstPtr ctx (fromIntegral len) + B.inCLengths len $ \off n -> + ccrypton_chacha_generate + (dstPtr `plusPtr` off) + ctx + (fromIntegral n) return (out, State st) -- | similar to 'generate' but assume certains values diff --git a/Crypto/Cipher/ChaCha/Poly1305.hs b/Crypto/Cipher/ChaCha/Poly1305.hs new file mode 100644 --- /dev/null +++ b/Crypto/Cipher/ChaCha/Poly1305.hs @@ -0,0 +1,335 @@ +-- | +-- Module : Crypto.Cipher.ChaCha.Poly1305 +-- License : BSD-style +-- Maintainer : Kazu Yamamoto +-- Stability : experimental +-- Portability : Good +-- +-- ChaCha20-Poly1305 (RFC 8439) a message at a time. +-- +-- "Crypto.Cipher.ChaChaPoly1305" takes a message in pieces: a state is +-- started, the additional data appended, the body encrypted and the tag +-- taken, each a step of its own. That is what a protocol wants when the +-- message arrives in pieces, and it is eight foreign calls and the +-- allocations between them when the message was already whole. +-- +-- Here the whole message goes in one call. +-- +-- The functions are the same shape as "Crypto.Cipher.AES.GCM", so a protocol +-- that offers both ciphers can hold them the same way. +module Crypto.Cipher.ChaCha.Poly1305 ( + Context, + newContext, + encrypt, + decrypt, + decryptWithTag, +) where + +import Crypto.Cipher.Types (AuthTag (..)) +import Crypto.Error +import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess) +import qualified Crypto.Internal.ByteArray as B +import Crypto.Internal.Compat (unsafeDoIO) +import Crypto.Internal.Imports +import Foreign.C.Types (CInt (..), CUInt (..)) +import Foreign.Ptr (Ptr, plusPtr) + +-- | A key, checked once. +-- +-- ChaCha20-Poly1305 has nothing to precompute from a key: the one-time +-- Poly1305 key comes from the nonce, so it differs for every message. This +-- holds the thirty-two bytes and the knowledge that they are thirty-two, and +-- exists so that the interface is the one "Crypto.Cipher.AES.GCM" has. +newtype Context = Context B.ScrubbedBytes + +instance NFData Context where + rnf (Context k) = k `seq` () + +-- | Take a key of 32 bytes. Any other length is reported as +-- 'CryptoError_KeySizeInvalid'. +newContext :: ByteArrayAccess key => key -> CryptoFailable Context +newContext k + | B.length k /= 32 = CryptoFailed CryptoError_KeySizeInvalid + | otherwise = CryptoPassed $ Context (B.convert k) +{-# INLINABLE newContext #-} + +-- | Encrypt one message. The result is the ciphertext with the tag after it, +-- which is the shape 'decrypt' expects. +-- +-- The nonce is the twelve bytes RFC 8439 defines; any other length gives +-- 'CryptoError_IvSizeInvalid'. RFC 8439 requires a 16-byte tag. +{-# INLINABLE encrypt #-} +encrypt + :: ( ByteArrayAccess nonce + , ByteArrayAccess aad + , ByteArrayAccess ba + , ByteArray output + ) + => Context + -> nonce + -> aad + -> ba + -> Int + -> CryptoFailable output +encrypt (Context k) nonce aad input taglen + | tooLongForC aad input = CryptoFailed CryptoError_ParameterInvalid + | not (validNonce nonce) = CryptoFailed CryptoError_IvSizeInvalid + | badTag taglen = CryptoFailed CryptoError_AuthenticationTagSizeInvalid + | otherwise = + CryptoPassed $ + unsafeDoIO $ + B.alloc (B.length input + taglen) $ \out -> + B.withByteArray k $ \kp -> + B.withByteArray nonce $ \np -> + B.withByteArray aad $ \ap -> + B.withByteArray input $ \ip -> + (callE (B.length input)) + out + (out `plusPtr` B.length input) + (fromIntegral taglen) + kp + np + (fromIntegral $ B.length nonce) + ap + (fromIntegral $ B.length aad) + ip + (fromIntegral $ B.length input) + +-- | Decrypt one message, in the shape 'encrypt' produced: the ciphertext with +-- its tag after it. The tag is compared here, every byte of it whatever the +-- answer, and a message whose tag does not match gives 'Nothing' rather than +-- the plaintext. +-- +-- 'Nothing' also comes back when the input is shorter than the tag, or the +-- nonce is not twelve bytes. +{-# INLINABLE decrypt #-} +decrypt + :: (ByteArrayAccess nonce, ByteArrayAccess aad, ByteArray ba) + => Context + -> nonce + -> aad + -> ba + -> Int + -> Maybe ba +decrypt (Context k) nonce aad input taglen + | tooLongForC aad input = Nothing + | not (validNonce nonce) = Nothing + | badTag taglen || B.length input < taglen = Nothing + | otherwise = unsafeDoIO $ do + (r, out) <- B.allocRet bodylen $ \outp -> + B.withByteArray k $ \kp -> + B.withByteArray nonce $ \np -> + B.withByteArray aad $ \ap -> + B.withByteArray body $ \ip -> + B.withByteArray tag $ \tp -> + (callD bodylen) + outp + tp + (fromIntegral taglen) + kp + np + (fromIntegral $ B.length nonce) + ap + (fromIntegral $ B.length aad) + ip + (fromIntegral bodylen) + return $ if r /= 0 then Just out else Nothing + where + bodylen = B.length input - taglen + (body, tag) = B.splitAt bodylen input + +-- | Decrypt one message, the tag kept apart, and hand back the tag this end +-- computed. +-- +-- For a caller whose protocol carries the tag separately from the ciphertext, +-- so that 'decrypt' -- which wants the two together and compares them itself +-- -- does not fit. Compare the two tags with '=='; the 'Eq' instance of +-- t'AuthTag' is a constant-time comparison, and taking them apart to compare +-- the bytes is how this goes wrong. +-- +-- Nothing here says whether the message is authentic. Until the comparison +-- is made and has come out equal, what this returns is not plaintext, it is +-- what the ciphertext turns into, and a caller must not act on it. +{-# INLINABLE decryptWithTag #-} +decryptWithTag + :: (ByteArrayAccess nonce, ByteArrayAccess aad, ByteArray ba) + => Context + -> nonce + -> aad + -> ba + -> Int + -> CryptoFailable (ba, AuthTag) +decryptWithTag (Context k) nonce aad input taglen + | tooLongForC aad input = CryptoFailed CryptoError_ParameterInvalid + | not (validNonce nonce) = CryptoFailed CryptoError_IvSizeInvalid + | badTag taglen = CryptoFailed CryptoError_AuthenticationTagSizeInvalid + | otherwise = CryptoPassed $ unsafeDoIO $ do + (tagbs, out) <- B.allocRet (B.length input) $ \outp -> + B.alloc taglen $ \tagp -> + B.withByteArray k $ \kp -> + B.withByteArray nonce $ \np -> + B.withByteArray aad $ \ap -> + B.withByteArray input $ \ip -> + (callT (B.length input)) + outp + tagp + (fromIntegral taglen) + kp + np + (fromIntegral $ B.length nonce) + ap + (fromIntegral $ B.length aad) + ip + (fromIntegral $ B.length input) + return (out, AuthTag $ B.convert (tagbs :: B.Bytes)) + +-- | The C takes its lengths as @uint32_t@, so a message or its additional +-- data from 2^32 bytes up cannot be handed to it: the length would be +-- truncated and most of the buffer left untouched, with nothing to say so. +-- The C does the whole message in one call, so there is no splitting it. +tooLongForC + :: (ByteArrayAccess aad, ByteArrayAccess ba) => aad -> ba -> Bool +tooLongForC aad input = + B.overCLength (B.length aad) || B.overCLength (B.length input) + +-- RFC 8439 is the twelve-byte nonce. ChaCha20 will take eight, but that is +-- the other construction, with a 64-bit block counter, and it is not what +-- this AEAD is defined over -- so it is refused here rather than quietly +-- encrypting under a scheme nobody asked for. +validNonce :: ByteArrayAccess nonce => nonce -> Bool +validNonce n = B.length n == 12 + +badTag :: Int -> Bool +badTag t = t /= 16 + +-- | An unsafe call keeps a capability for as long as it runs, so it is only +-- for a message short enough that the run is short. Four kibibytes is what +-- the AES side uses, and it takes in a datagram of any size a network will +-- carry. +shortMessage :: Int +shortMessage = 4096 + +callE :: Int -> CEncrypt +callE n + | n <= shortMessage = c_chachapoly_encrypt_unsafe + | otherwise = c_chachapoly_encrypt + +callD :: Int -> CDecrypt +callD n + | n <= shortMessage = c_chachapoly_decrypt_unsafe + | otherwise = c_chachapoly_decrypt + +callT :: Int -> CEncrypt +callT n + | n <= shortMessage = c_chachapoly_decrypt_tag_unsafe + | otherwise = c_chachapoly_decrypt_tag + +type CEncrypt = + Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> IO () + +type CDecrypt = + Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> IO CInt + +foreign import ccall "crypton_chachapoly.h crypton_chachapoly_encrypt" + c_chachapoly_encrypt + :: Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> IO () + +foreign import ccall unsafe "crypton_chachapoly.h crypton_chachapoly_encrypt" + c_chachapoly_encrypt_unsafe + :: Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> IO () + +foreign import ccall "crypton_chachapoly.h crypton_chachapoly_decrypt" + c_chachapoly_decrypt + :: Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> IO CInt + +foreign import ccall unsafe "crypton_chachapoly.h crypton_chachapoly_decrypt" + c_chachapoly_decrypt_unsafe + :: Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> IO CInt + +foreign import ccall "crypton_chachapoly.h crypton_chachapoly_decrypt_tag" + c_chachapoly_decrypt_tag + :: Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> IO () + +foreign import ccall unsafe "crypton_chachapoly.h crypton_chachapoly_decrypt_tag" + c_chachapoly_decrypt_tag_unsafe + :: Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> Ptr Word8 + -> CUInt + -> IO () diff --git a/Crypto/Cipher/ChaChaPoly1305.hs b/Crypto/Cipher/ChaChaPoly1305.hs --- a/Crypto/Cipher/ChaChaPoly1305.hs +++ b/Crypto/Cipher/ChaChaPoly1305.hs @@ -1,3 +1,5 @@ +{-# LANGUAGE GeneralizedNewtypeDeriving #-} + -- | -- Module : Crypto.Cipher.ChaChaPoly1305 -- License : BSD-style @@ -28,7 +30,7 @@ -- > -> ByteString -- input plaintext to be encrypted -- > -> CryptoFailable ByteString -- ciphertext with a 128-bit tag attached -- >encrypt nonce key header plaintext = do --- > st1 <- C.nonce12 nonce >>= C.initialize key +-- > st1 <- C.initialize <$> C.key key <*> C.nonce12 nonce -- > let -- > st2 = C.finalizeAAD $ C.appendAAD header st1 -- > (out, st3) = C.encrypt plaintext st2 @@ -41,6 +43,8 @@ -- * Low level State, + Key, + key, Nonce, XNonce, nonce12, @@ -68,6 +72,7 @@ ) import qualified Crypto.Internal.ByteArray as B import Crypto.Internal.Imports +import qualified Crypto.Internal.Poly1305 as PolyKey import qualified Crypto.MAC.Poly1305 as Poly1305 import qualified Data.ByteArray.Pack as P import Data.Memory.Endian @@ -174,44 +179,39 @@ -- -- The key length need to be 256 bits, and the nonce -- procured using either `nonce8` or `nonce12` -initialize - :: ByteArrayAccess key - => key -> Nonce -> CryptoFailable State -initialize key (Nonce8 nonce) = initialize' key nonce -initialize key (Nonce12 nonce) = initialize' key nonce +-- | A ChaCha20Poly1305 key: thirty-two bytes, checked once here rather than +-- at every use, so that 'initialize' and 'initializeX' cannot fail. +newtype Key = Key ScrubbedBytes + deriving (ByteArrayAccess, Eq, NFData) -initialize' - :: ByteArrayAccess key - => key -> Bytes -> CryptoFailable State -initialize' key nonce - | B.length key /= 32 = CryptoFailed CryptoError_KeySizeInvalid - | otherwise = CryptoPassed $ initFromRootState rootState - where - rootState = ChaCha.initialize 20 key nonce +-- | Take thirty-two bytes for a key. A different length is reported as +-- 'CryptoError_KeySizeInvalid'; nothing else about a key can be wrong. +key :: ByteArrayAccess ba => ba -> CryptoFailable Key +key k + | B.length k /= 32 = CryptoFailed CryptoError_KeySizeInvalid + | otherwise = CryptoPassed $ Key $ B.convert k +initialize :: Key -> Nonce -> State +initialize k (Nonce8 nonce) = initialize' k nonce +initialize k (Nonce12 nonce) = initialize' k nonce + +initialize' :: Key -> Bytes -> State +initialize' k nonce = initFromRootState (ChaCha.initialize 20 k nonce) + initFromRootState :: ChaCha.State -> State initFromRootState rootState = State encState polyState 0 0 where (polyKey, encState) = ChaCha.generate rootState 64 - -- 64 bytes are generated so the ChaCha state advances a whole block; the - -- first 32 of them are the key, so the length is right by construction - polyState = - Poly1305.initialize $ - throwCryptoError $ - Poly1305.key (B.take 32 polyKey :: ScrubbedBytes) + -- 64 bytes are generated so the ChaCha state advances a whole block, and + -- the first 32 of them are the key, so there is no length left to check + polyState = Poly1305.initialize (PolyKey.Key (B.take 32 polyKey)) -- | Initialize a new XChaChaPoly1305 State -- -- The key length needs to be 256 bits, and the nonce -- procured using `nonce24`. -initializeX - :: ByteArrayAccess key - => key -> XNonce -> CryptoFailable State -initializeX key (Nonce24 nonce) - | B.length key /= 32 = CryptoFailed CryptoError_KeySizeInvalid - | otherwise = CryptoPassed $ initFromRootState rootState - where - rootState = ChaCha.initializeX 20 key nonce +initializeX :: Key -> XNonce -> State +initializeX k (Nonce24 nonce) = initFromRootState (ChaCha.initializeX 20 k nonce) -- | Append Authenticated Data to the State and return -- the new modified State. @@ -267,8 +267,8 @@ aeadChacha20poly1305Init :: (ByteArrayAccess k, ByteArrayAccess n) => k -> n -> CryptoFailable (AEAD ChaCha20Poly1305) -aeadChacha20poly1305Init key nonce = do - st0 <- nonce12 nonce >>= initialize key +aeadChacha20poly1305Init k nonce = do + st0 <- initialize <$> key k <*> nonce12 nonce return $ AEAD model st0 where model = diff --git a/Crypto/Cipher/RC4.hs b/Crypto/Cipher/RC4.hs --- a/Crypto/Cipher/RC4.hs +++ b/Crypto/Cipher/RC4.hs @@ -98,7 +98,14 @@ B.allocRet len $ \outptr -> B.withByteArray clearText $ \clearPtr -> do st <- B.copy prevSt $ \stPtr -> - c_rc4_combine (castPtr stPtr) clearPtr (fromIntegral len) outptr + -- in pieces the C's uint32_t length can hold; the state it + -- keeps means it can simply be called again + B.inCLengths len $ \off n -> + c_rc4_combine + (castPtr stPtr) + (clearPtr `plusPtr` off) + (fromIntegral n) + (outptr `plusPtr` off) return $! State st where -- return $! (State st, B.PS outfptr 0 len) diff --git a/Crypto/Cipher/Salsa.hs b/Crypto/Cipher/Salsa.hs --- a/Crypto/Cipher/Salsa.hs +++ b/Crypto/Cipher/Salsa.hs @@ -70,10 +70,19 @@ | B.null src = (B.empty, prevSt) | otherwise = unsafeDoIO $ do (out, st) <- B.copyRet prevStMem $ \ctx -> - B.alloc (B.length src) $ \dstPtr -> - B.withByteArray src $ \srcPtr -> do - ccrypton_salsa_combine dstPtr ctx srcPtr (fromIntegral $ B.length src) + B.alloc n $ \dstPtr -> + B.withByteArray src $ \srcPtr -> + -- in pieces the C's uint32_t length can hold; it carries + -- the state in ctx, so it can simply be called again + B.inCLengths n $ \off len -> + ccrypton_salsa_combine + (dstPtr `plusPtr` off) + ctx + (srcPtr `plusPtr` off) + (fromIntegral len) return (out, State st) + where + n = B.length src -- | Generate a number of bytes from the Salsa output directly generate @@ -88,7 +97,11 @@ | otherwise = unsafeDoIO $ do (out, st) <- B.copyRet prevStMem $ \ctx -> B.alloc len $ \dstPtr -> - ccrypton_salsa_generate dstPtr ctx (fromIntegral len) + B.inCLengths len $ \off n -> + ccrypton_salsa_generate + (dstPtr `plusPtr` off) + ctx + (fromIntegral n) return (out, State st) foreign import ccall "crypton_salsa_init" diff --git a/Crypto/Cipher/Twofish/Primitive.hs b/Crypto/Cipher/Twofish/Primitive.hs --- a/Crypto/Cipher/Twofish/Primitive.hs +++ b/Crypto/Cipher/Twofish/Primitive.hs @@ -15,9 +15,8 @@ import Crypto.Internal.WordArray import Crypto.Internal.Words (Word128 (..)) import Data.Bits -import Data.List (foldl') +import qualified Data.List as L import Data.Word -import Prelude hiding (foldl') -- Based on the Golang referance implementation -- https://github.com/golang/crypto/blob/master/twofish/twofish.go @@ -113,7 +112,7 @@ b' = b `xor` arrayRead32 ks 1 c' = c `xor` arrayRead32 ks 2 d' = d `xor` arrayRead32 ks 3 - (!a'', !b'', !c'', !d'') = foldl' shuffle (a', b', c', d') [0 .. 7] + (!a'', !b'', !c'', !d'') = L.foldl' shuffle (a', b', c', d') [0 .. 7] ts = ( c'' `xor` arrayRead32 ks 4 , d'' `xor` arrayRead32 ks 5 @@ -180,7 +179,7 @@ b' = d `xor` arrayRead32 ks 7 c' = a `xor` arrayRead32 ks 4 d' = b `xor` arrayRead32 ks 5 - (!a'', !b'', !c'', !d'') = foldl' unshuffle (a', b', c', d') [8, 7 .. 1] + (!a'', !b'', !c'', !d'') = L.foldl' unshuffle (a', b', c', d') [8, 7 .. 1] ixs = ( a'' `xor` arrayRead32 ks 0 , b'' `xor` arrayRead32 ks 1 @@ -284,7 +283,7 @@ ( \wordIndex -> map ( \rsRow -> - foldl' + L.foldl' ( \acc (!rsVal, !colIndex) -> acc `xor` gfMult rsPolynomial (B.index key $ 8 * wordIndex + colIndex) rsVal ) @@ -443,7 +442,7 @@ b' = rotateL b 8 h :: ByteArray ba => [Word8] -> KeyPackage ba -> Int -> Word32 -h input keyPackage offset = foldl' xorMdsColMult 0 $ zip [y0f, y1f, y2f, y3f] $ enumFrom Zero +h input keyPackage offset = L.foldl' xorMdsColMult 0 $ zip [y0f, y1f, y2f, y3f] $ enumFrom Zero where key = rawKeyBytes keyPackage [y0, y1, y2, y3] = take 4 input diff --git a/Crypto/ConstructHash/MiyaguchiPreneel.hs b/Crypto/ConstructHash/MiyaguchiPreneel.hs --- a/Crypto/ConstructHash/MiyaguchiPreneel.hs +++ b/Crypto/ConstructHash/MiyaguchiPreneel.hs @@ -15,8 +15,7 @@ MiyaguchiPreneel, ) where -import Data.List (foldl') -import Prelude hiding (foldl') +import qualified Data.List as L import Crypto.Cipher.Types import Crypto.Cipher.Types.Utils (chunk) @@ -41,7 +40,7 @@ -> MiyaguchiPreneel cipher -- ^ output tag compute' g = - MP . foldl' (step $ g) (B.replicate bsz 0) . chunks . pad (ZERO bsz) . B.convert + MP . L.foldl' (step $ g) (B.replicate bsz 0) . chunks . pad (ZERO bsz) . B.convert where bsz = blockSize (g B.empty {- dummy to get block size -}) -- 'chunk' slices rather than splitting the message, which copied whatever diff --git a/Crypto/ECC.hs b/Crypto/ECC.hs --- a/Crypto/ECC.hs +++ b/Crypto/ECC.hs @@ -49,13 +49,12 @@ import qualified Crypto.ECC.Simple.Prim as Simple import qualified Crypto.ECC.Simple.Types as Simple import Crypto.Error +import Crypto.KEM (SharedSecret (..)) import Crypto.Internal.ByteArray ( ByteArray, ByteArrayAccess, - ScrubbedBytes, ) import qualified Crypto.Internal.ByteArray as B -import Crypto.Internal.Imports import Crypto.Number.Basic (numBits) import Crypto.Number.Serialize (i2ospOf_, os2ip) import qualified Crypto.Number.Serialize.LE as LE @@ -74,16 +73,6 @@ { keypairGetPublic :: !(Point curve) , keypairGetPrivate :: !(Scalar curve) } - --- | Secret shared via key exchange -newtype SharedSecret = SharedSecret ScrubbedBytes - deriving (Eq, ByteArrayAccess, NFData) - -instance Semigroup SharedSecret where - SharedSecret x <> SharedSecret y = SharedSecret (x <> y) - -instance Monoid SharedSecret where - mempty = SharedSecret mempty class EllipticCurve curve where -- | Point on an Elliptic Curve diff --git a/Crypto/Error/Types.hs b/Crypto/Error/Types.hs --- a/Crypto/Error/Types.hs +++ b/Crypto/Error/Types.hs @@ -57,6 +57,10 @@ -- the base point generates, so multiplying it would answer modulo a -- small order. Appended for the same reason as the constructor above. CryptoError_PointSubgroupInvalid + | -- | A public key is the right length but is not a well-formed encoding + -- of one, so no honest party produced it. Appended for the same + -- reason as the two constructors above. + CryptoError_PublicKeyStructureInvalid deriving (Show, Eq, Enum, Data) instance E.Exception CryptoError diff --git a/Crypto/Hash.hs b/Crypto/Hash.hs --- a/Crypto/Hash.hs +++ b/Crypto/Hash.hs @@ -53,12 +53,20 @@ import qualified Crypto.Internal.ByteArray as B import qualified Data.ByteString.Lazy as L import Data.Int (Int32) +import Data.Word (Word8) +import Foreign.Marshal.Alloc (allocaBytes) import qualified Foreign.Marshal.Utils as FMU import Foreign.Ptr (Ptr, castPtr, plusPtr) -- | Hash a strict bytestring into a digest. -hash :: (ByteArrayAccess ba, HashAlgorithm a) => ba -> Digest a -hash bs = hashFinalize $ hashUpdate hashInit bs +hash :: forall ba a. (ByteArrayAccess ba, HashAlgorithm a) => ba -> Digest a +hash bs = Digest $ + allocAndFreezePrim (hashDigestSize (undefined :: a)) $ \(dig :: Ptr (Digest a)) -> + -- one context, used in place: the pure steps would copy it three times + allocaBytes (hashInternalContextSize (undefined :: a)) $ \(ctx :: Ptr (Context a)) -> do + hashInternalInit ctx + B.withByteArray bs (updateBlocks ctx (B.length bs)) + hashInternalFinalize ctx dig -- | Hash the first N bytes of a bytestring, with code path independent from N. hashPrefix @@ -92,20 +100,19 @@ hashUpdates c l | null ls = c | otherwise = Context $ B.copyAndFreeze c $ \(ctx :: Ptr (Context a)) -> - mapM_ (\b -> B.withByteArray b (processBlocks ctx (B.length b))) ls + mapM_ (\b -> B.withByteArray b (updateBlocks ctx (B.length b))) ls where ls = filter (not . B.null) l - -- process the data in 2GB chunks to fit in uint32_t and Int on 32 bit systems - processBlocks ctx bytesLeft dataPtr - | bytesLeft == 0 = return () - | otherwise = do - hashInternalUpdate ctx dataPtr (fromIntegral actuallyProcessed) - processBlocks - ctx - (bytesLeft - actuallyProcessed) - (dataPtr `plusPtr` actuallyProcessed) - where - actuallyProcessed = min bytesLeft (fromIntegral (maxBound :: Int32)) + +-- | Process the data in 2GB chunks to fit in uint32_t and Int on 32 bit systems +updateBlocks :: HashAlgorithm a => Ptr (Context a) -> Int -> Ptr Word8 -> IO () +updateBlocks ctx bytesLeft dataPtr + | bytesLeft == 0 = return () + | otherwise = do + hashInternalUpdate ctx dataPtr (fromIntegral actuallyProcessed) + updateBlocks ctx (bytesLeft - actuallyProcessed) (dataPtr `plusPtr` actuallyProcessed) + where + actuallyProcessed = min bytesLeft (fromIntegral (maxBound :: Int32)) -- | Finalize a context and return a digest. hashFinalize diff --git a/Crypto/Hash/Algorithms.hs b/Crypto/Hash/Algorithms.hs --- a/Crypto/Hash/Algorithms.hs +++ b/Crypto/Hash/Algorithms.hs @@ -48,8 +48,10 @@ Blake2bp (..), Blake2s (..), Blake2sp (..), + Skein256 (..), Skein256_224 (..), Skein256_256 (..), + Skein512 (..), Skein512_224 (..), Skein512_256 (..), Skein512_384 (..), diff --git a/Crypto/Hash/Skein256.hs b/Crypto/Hash/Skein256.hs --- a/Crypto/Hash/Skein256.hs +++ b/Crypto/Hash/Skein256.hs @@ -1,7 +1,11 @@ {-# LANGUAGE DataKinds #-} {-# LANGUAGE DeriveDataTypeable #-} {-# LANGUAGE ForeignFunctionInterface #-} +{-# LANGUAGE KindSignatures #-} +{-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeFamilies #-} +{-# LANGUAGE TypeOperators #-} +{-# LANGUAGE UndecidableInstances #-} -- | -- Module : Crypto.Hash.Skein256 @@ -13,14 +17,17 @@ -- Module containing the binding functions to work with the -- Skein256 cryptographic hash. module Crypto.Hash.Skein256 ( + Skein256 (..), Skein256_224 (..), Skein256_256 (..), ) where import Crypto.Hash.Types +import Crypto.Internal.Nat import Data.Data import Data.Word (Word32, Word8) import Foreign.Ptr (Ptr) +import GHC.TypeLits (KnownNat, Nat, type (+)) -- | Skein256 (224 bits) cryptographic hash algorithm data Skein256_224 = Skein256_224 @@ -51,6 +58,38 @@ hashInternalInit p = c_skein256_init p 256 hashInternalUpdate = c_skein256_update hashInternalFinalize p = c_skein256_finalize p 256 + +-- | Skein256 with the digest size given as a type parameter of kind 'Nat', +-- in bits. @t'Skein256' 256@ is @t'Skein256_256'@; the sizes with a type of +-- their own +-- above are there for their names, and this one also takes the sizes that +-- have none. +-- +-- A size that is not a whole number of bytes is rounded up to the next one, +-- as the implementation underneath does. +-- +-- The output is produced in counter mode, a block of it per Threefish call, +-- so one large digest is a good deal cheaper than the same number of bytes +-- taken from repeated small ones: on an Apple M4, 512 KiB arrives at 947 MB/s +-- in one digest against 172 MB/s as 8192 separate @t'Skein256_256'@ ones. +-- +-- Note the digest size goes into the configuration block, so it changes the +-- value the message is hashed from: a longer digest is /not/ an extension of +-- a shorter one. That is the opposite of how t'Crypto.Hash.SHAKE.SHAKE128' +-- behaves. +data Skein256 (bitlen :: Nat) = Skein256 + deriving (Show, Data) + +instance KnownNat bitlen => HashAlgorithm (Skein256 bitlen) where + type HashBlockSize (Skein256 bitlen) = 32 + type HashDigestSize (Skein256 bitlen) = Div8 (bitlen + 7) + type HashInternalContextSize (Skein256 bitlen) = 96 + hashBlockSize _ = 32 + hashDigestSize _ = byteLen (Proxy :: Proxy bitlen) + hashInternalContextSize _ = 96 + hashInternalInit p = c_skein256_init p (integralNatVal (Proxy :: Proxy bitlen)) + hashInternalUpdate = c_skein256_update + hashInternalFinalize p = c_skein256_finalize p (integralNatVal (Proxy :: Proxy bitlen)) foreign import ccall unsafe "crypton_skein256_init" c_skein256_init :: Ptr (Context a) -> Word32 -> IO () diff --git a/Crypto/Hash/Skein512.hs b/Crypto/Hash/Skein512.hs --- a/Crypto/Hash/Skein512.hs +++ b/Crypto/Hash/Skein512.hs @@ -1,7 +1,11 @@ {-# LANGUAGE DataKinds #-} {-# LANGUAGE DeriveDataTypeable #-} {-# LANGUAGE ForeignFunctionInterface #-} +{-# LANGUAGE KindSignatures #-} +{-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeFamilies #-} +{-# LANGUAGE TypeOperators #-} +{-# LANGUAGE UndecidableInstances #-} -- | -- Module : Crypto.Hash.Skein512 @@ -13,6 +17,7 @@ -- Module containing the binding functions to work with the -- Skein512 cryptographic hash. module Crypto.Hash.Skein512 ( + Skein512 (..), Skein512_224 (..), Skein512_256 (..), Skein512_384 (..), @@ -20,9 +25,11 @@ ) where import Crypto.Hash.Types +import Crypto.Internal.Nat import Data.Data import Data.Word (Word32, Word8) import Foreign.Ptr (Ptr) +import GHC.TypeLits (KnownNat, Nat, type (+)) -- | Skein512 (224 bits) cryptographic hash algorithm data Skein512_224 = Skein512_224 @@ -83,6 +90,38 @@ hashInternalInit p = c_skein512_init p 512 hashInternalUpdate = c_skein512_update hashInternalFinalize p = c_skein512_finalize p 512 + +-- | Skein512 with the digest size given as a type parameter of kind 'Nat', +-- in bits. @t'Skein512' 512@ is @t'Skein512_512'@; the sizes with a type of +-- their own +-- above are there for their names, and this one also takes the sizes that +-- have none. +-- +-- A size that is not a whole number of bytes is rounded up to the next one, +-- as the implementation underneath does. +-- +-- The output is produced in counter mode, a block of it per Threefish call, +-- so one large digest is a good deal cheaper than the same number of bytes +-- taken from repeated small ones: on an Apple M4, 512 KiB arrives at 947 MB/s +-- in one digest against 172 MB/s as 8192 separate @t'Skein512_512'@ ones. +-- +-- Note the digest size goes into the configuration block, so it changes the +-- value the message is hashed from: a longer digest is /not/ an extension of +-- a shorter one. That is the opposite of how t'Crypto.Hash.SHAKE.SHAKE128' +-- behaves. +data Skein512 (bitlen :: Nat) = Skein512 + deriving (Show, Data) + +instance KnownNat bitlen => HashAlgorithm (Skein512 bitlen) where + type HashBlockSize (Skein512 bitlen) = 64 + type HashDigestSize (Skein512 bitlen) = Div8 (bitlen + 7) + type HashInternalContextSize (Skein512 bitlen) = 160 + hashBlockSize _ = 64 + hashDigestSize _ = byteLen (Proxy :: Proxy bitlen) + hashInternalContextSize _ = 160 + hashInternalInit p = c_skein512_init p (integralNatVal (Proxy :: Proxy bitlen)) + hashInternalUpdate = c_skein512_update + hashInternalFinalize p = c_skein512_finalize p (integralNatVal (Proxy :: Proxy bitlen)) foreign import ccall unsafe "crypton_skein512_init" c_skein512_init :: Ptr (Context a) -> Word32 -> IO () diff --git a/Crypto/Hash/Types.hs b/Crypto/Hash/Types.hs --- a/Crypto/Hash/Types.hs +++ b/Crypto/Hash/Types.hs @@ -103,6 +103,26 @@ -- layout is architecture dependent, may contain uninitialized data fragments, -- and change in future versions. The bytearray should not be used as input to -- cryptographic algorithms. +-- +-- __A context is not erased when it is finished with.__ A hash algorithm +-- buffers its input a block at a time, and finalizing does not clear what is +-- left there. How much survives depends on where the message ended relative +-- to the block: with SHA-256, a 32-byte message is still in the context in +-- full afterwards, and a 100-byte one leaves its last 36 bytes. +-- @hashFinalize@ works on a copy, so the caller's own context keeps what it +-- had as well. Nothing clears either of them: this is 'Bytes' rather than +-- @ScrubbedBytes@, and the C clears nothing. They go to the garbage +-- collector as they are, and a core file, a crash dump or a swapped page can +-- carry them away afterwards. +-- +-- That is a deliberate trade rather than an oversight, and there is no way +-- to ask for the other side of it: no operation here clears a context. +-- Scrubbing them all was measured at about 70% of a 32-byte hash and a third +-- of an incremental one, because the allocation is most of the work when the +-- message is short -- and short hashes are the common case, in HMAC, in +-- HKDF, and anywhere a key or an identifier is hashed. A 64 KB hash does +-- not notice it. Anything that must not be left in memory this way is +-- better not hashed through this interface at all. newtype Context a = Context Bytes deriving (ByteArrayAccess, NFData) diff --git a/Crypto/Internal/ByteArray.hs b/Crypto/Internal/ByteArray.hs --- a/Crypto/Internal/ByteArray.hs +++ b/Crypto/Internal/ByteArray.hs @@ -14,6 +14,8 @@ module Data.ByteArray.Mapping, module Data.ByteArray.Encoding, constAllZero, + overCLength, + inCLengths, allocAndFreezePrimIO, allocAndFreezePrim, bxor, @@ -30,6 +32,51 @@ import Foreign.Storable (peekByteOff) import Crypto.Internal.Compat (unsafeDoIO) + +-- | Whether a length is too large to reach the C, which takes its lengths as +-- @uint32_t@. +-- +-- From 2^32 up the value is truncated on the way down, and the C then works +-- on the low bits of it and leaves the rest of the buffer as it found it -- +-- which for a fresh allocation is zeros. What comes back is as long as the +-- caller asked for, with nothing to say that most of it was never written: +-- a 4 GiB message through Crypto.Cipher.ChaCha.combine came back with 2^32 +-- bytes of zeros where the ciphertext should have been. +-- +-- Every place that hands a caller's length to the C either turns it away +-- with this or cuts the work into pieces small enough to pass. +-- +-- The round trip through 'Word32' rather than a comparison against 2^32, +-- which a 32-bit 'Int' cannot hold. There every non-negative 'Int' passes, +-- which is the right answer: there is no such buffer to be had. +overCLength :: Int -> Bool +overCLength n = fromIntegral (fromIntegral n :: Word32) /= n + +-- | Walk a length in pieces small enough to reach the C, calling the action +-- with the offset and the size of each. +-- +-- The same 2 GiB step "Crypto.Hash" has always taken, and for the same +-- reason: the C takes its lengths as @uint32_t@, and a 32-bit 'Int' cannot +-- hold a whole one either. This is for the C that keeps its state in a +-- context and can simply be called again -- the stream ciphers, the MACs -- +-- where a long message can be enciphered in pieces rather than refused. +inCLengths :: Int -> (Int -> Int -> IO ()) -> IO () +inCLengths total f = go 0 + where + go !off + | off >= total = return () + | otherwise = f off n >> go (off + n) + where + !n = min (total - off) cChunk + +-- | The step 'inCLengths' takes: the largest multiple of 64 that a signed +-- 32-bit integer holds. +-- +-- Under 2^31 because a 32-bit 'Int' cannot hold more, and a +-- multiple of 64 because some of the C this feeds -- the AEAD modes -- will +-- take a piece that is not a whole number of blocks only as the last one. +cChunk :: Int +cChunk = 0x7fffffc0 -- | Allocate a pinned 'Prim.ByteArray' of the given size, populate it via a -- 'Ptr', then freeze and return it. The pointer must not be retained after diff --git a/Crypto/Internal/Poly1305.hs b/Crypto/Internal/Poly1305.hs new file mode 100644 --- /dev/null +++ b/Crypto/Internal/Poly1305.hs @@ -0,0 +1,37 @@ +{-# LANGUAGE GeneralizedNewtypeDeriving #-} + +-- | +-- Module : Crypto.Internal.Poly1305 +-- License : BSD-style +-- Maintainer : Kazu Yamamoto +-- Stability : experimental +-- Portability : unknown +-- +-- The Poly1305 key with its constructor, for the modules here that build one +-- from bytes whose length they already know. "Crypto.MAC.Poly1305" exports +-- the type without the constructor, so that outside this library a key can +-- only be made by 'key', which checks. +module Crypto.Internal.Poly1305 ( + Key (..), + key, +) where + +import Crypto.Error +import Crypto.Internal.ByteArray (ByteArrayAccess, ScrubbedBytes) +import qualified Crypto.Internal.ByteArray as B +import Crypto.Internal.DeepSeq + +-- | A Poly1305 key: thirty-two bytes, and the length is checked here rather +-- than at every use. 'Crypto.MAC.Poly1305.initialize' and +-- 'Crypto.MAC.Poly1305.auth' take one of these and cannot fail, so a caller +-- that holds a key does not carry an error case for a length it already knows +-- is right. +newtype Key = Key ScrubbedBytes + deriving (ByteArrayAccess, Eq, NFData) + +-- | Take thirty-two bytes for a key. A different length is reported as +-- 'CryptoError_MacKeyInvalid'; nothing else about a key can be wrong. +key :: ByteArrayAccess ba => ba -> CryptoFailable Key +key k + | B.length k /= 32 = CryptoFailed CryptoError_MacKeyInvalid + | otherwise = CryptoPassed $ Key $ B.convert k diff --git a/Crypto/KEM.hs b/Crypto/KEM.hs new file mode 100644 --- /dev/null +++ b/Crypto/KEM.hs @@ -0,0 +1,121 @@ +-- | +-- Module : Crypto.KEM +-- License : BSD-style +-- Maintainer : Kazu Yamamoto +-- Stability : experimental +-- Portability : unknown +-- +-- Key encapsulation: one side publishes a key, the other draws a secret and +-- returns something only the first side can turn back into it. +-- +-- The shape is not ML-KEM's alone, which is why this is a class in a module +-- of its own rather than part of any one algorithm: 'Crypto.PubKey.MLKEM' +-- is one instance of it, and a caller that does not care which it has can +-- be written against this. +-- +-- A bare Diffie-Hellman exchange is deliberately __not__ an instance, even +-- though the shapes line up. The KEM a Diffie-Hellman group gives is +-- DHKEM, of RFC 9180 section 4.1, and that is not the raw exchange: its +-- shared secret is the exchange's output run through HKDF with the +-- ephemeral and the recipient public keys as context, under labels that +-- name the ciphersuite. The binding to those two keys, and the separation +-- between suites, are what the KEM security argument rests on; the raw +-- value has neither. A protocol can supply the binding at its own level -- +-- TLS 1.3 does, in the key schedule -- but then the binding belongs to the +-- protocol, not to this class, and offering the raw exchange here would +-- invite its use somewhere that supplies nothing. +-- +-- DHKEM proper is an instance, and it is not here either: the labels it +-- derives under carry the HPKE ciphersuite identifier, which is an IANA +-- registry value rather than anything a primitive knows, so the instances +-- live beside the registry in the @hpke@ package. +{-# LANGUAGE GeneralizedNewtypeDeriving #-} +{-# LANGUAGE TypeFamilies #-} + +module Crypto.KEM ( + KEM (..), + SharedSecret (..), +) where + +import Data.Kind (Type) + +import Crypto.Error (CryptoFailable) +import Crypto.Internal.ByteArray (ByteArrayAccess, ScrubbedBytes) +import Crypto.Internal.Imports +import Crypto.Random (MonadRandom) + +-- | Secret shared via key exchange. +newtype SharedSecret = SharedSecret ScrubbedBytes + deriving (Eq, ByteArrayAccess, NFData) + +instance Show SharedSecret where + show _ = "SharedSecret " + +instance Semigroup SharedSecret where + SharedSecret x <> SharedSecret y = SharedSecret (x <> y) + +instance Monoid SharedSecret where + mempty = SharedSecret mempty + +-- | A key encapsulation mechanism. +-- +-- The three values are named for what they do rather than for what they are +-- in any one algorithm. In ML-KEM the encapsulation key and the ciphertext +-- are what the names say; in a Diffie-Hellman exchange both are public +-- values of the group, and the \"ciphertext\" is the ephemeral one the +-- encapsulating side generates. Keeping them apart in the types is what +-- stops one being passed where the other belongs, which they are not +-- interchangeable for even where they have the same representation. +class KEM kem where + -- | What the encapsulating side is given. + type EncapsulationKey kem :: Type + + -- | What the decapsulating side keeps. + type DecapsulationKey kem :: Type + + -- | What travels back, and is decapsulated. + type Ciphertext kem :: Type + + -- | The randomness 'encapsulate' draws, for the instances that let a + -- caller supply it. In ML-KEM it is @m@ of FIPS 203, a string of + -- bytes; in DHKEM it is the ephemeral secret key, a scalar of the + -- group. Both are secret, and both determine the shared secret + -- completely. + type Coins kem :: Type + + -- | Generate a key pair for the decapsulating side. + generateKeyPair + :: MonadRandom m + => proxy kem -> m (EncapsulationKey kem, DecapsulationKey kem) + + -- | Draw a secret and encapsulate it against the key. + -- + -- This can fail, and does for some instances: a Diffie-Hellman exchange + -- refuses a peer value that would make the secret degenerate, where + -- ML-KEM has nothing to refuse. + encapsulate + :: MonadRandom m + => proxy kem + -> EncapsulationKey kem + -> m (CryptoFailable (Ciphertext kem, SharedSecret)) + + -- | Encapsulate with the randomness supplied rather than drawn. + -- + -- The secret this produces is a deterministic function of the key and + -- these coins, so they must come from a source no other party can + -- predict or repeat, and must not be used twice. 'encapsulate' is the + -- entry point for ordinary use; this one is for test vectors, and for a + -- protocol that has to name the ephemeral value it used -- HPKE lets a + -- sender supply its own, and the RFC 9180 vectors are written that way. + encapsulateWith + :: proxy kem + -> EncapsulationKey kem + -> Coins kem + -> CryptoFailable (Ciphertext kem, SharedSecret) + + -- | Recover the secret. + decapsulate + :: proxy kem + -> DecapsulationKey kem + -> Ciphertext kem + -> CryptoFailable SharedSecret diff --git a/Crypto/MAC/Poly1305.hs b/Crypto/MAC/Poly1305.hs --- a/Crypto/MAC/Poly1305.hs +++ b/Crypto/MAC/Poly1305.hs @@ -35,6 +35,7 @@ ) import qualified Crypto.Internal.ByteArray as B import Crypto.Internal.DeepSeq +import Crypto.Internal.Poly1305 (Key (..), key) import Data.Word import Foreign.C.Types import Foreign.Ptr @@ -48,20 +49,6 @@ newtype State = State ScrubbedBytes deriving (ByteArrayAccess) --- | A Poly1305 key: thirty-two bytes, and the length is checked here rather --- than at every use. 'initialize' and 'auth' take one of these and cannot --- fail, so a caller that holds a key does not carry an error case for a --- length it already knows is right. -newtype Key = Key ScrubbedBytes - deriving (ByteArrayAccess, Eq, NFData) - --- | Take thirty-two bytes for a key. A different length is reported as --- 'CryptoError_MacKeyInvalid'; nothing else about a key can be wrong. -key :: ByteArrayAccess ba => ba -> CryptoFailable Key -key k - | B.length k /= 32 = CryptoFailed CryptoError_MacKeyInvalid - | otherwise = CryptoPassed $ Key $ B.convert k - -- | Poly1305 State. use State instead of Ctx type Ctx = State @@ -105,7 +92,10 @@ update :: ByteArrayAccess ba => State -> ba -> State update (State prevCtx) d = State $ B.copyAndFreeze prevCtx $ \ctxPtr -> B.withByteArray d $ \dataPtr -> - c_poly1305_update (castPtr ctxPtr) dataPtr (fromIntegral $ B.length d) + -- in pieces the C's uint32_t length can hold; the context carries + -- across, so it can simply be called again + B.inCLengths (B.length d) $ \off n -> + c_poly1305_update (castPtr ctxPtr) (dataPtr `plusPtr` off) (fromIntegral n) {-# NOINLINE update #-} -- | updates a context with multiples bytestring @@ -114,7 +104,9 @@ where loop [] _ = return () loop (x : xs) ctxPtr = do - B.withByteArray x $ \dataPtr -> c_poly1305_update ctxPtr dataPtr (fromIntegral $ B.length x) + B.withByteArray x $ \dataPtr -> + B.inCLengths (B.length x) $ \off n -> + c_poly1305_update ctxPtr (dataPtr `plusPtr` off) (fromIntegral n) loop xs ctxPtr {-# NOINLINE updates #-} @@ -137,5 +129,9 @@ B.withByteArray k $ \keyPtr -> do c_poly1305_init (castPtr ctxPtr) keyPtr B.withByteArray d $ \dataPtr -> - c_poly1305_update (castPtr ctxPtr) dataPtr (fromIntegral $ B.length d) + B.inCLengths (B.length d) $ \off n -> + c_poly1305_update + (castPtr ctxPtr) + (dataPtr `plusPtr` off) + (fromIntegral n) c_poly1305_finalize dst (castPtr ctxPtr) diff --git a/Crypto/Number/Basic.hs b/Crypto/Number/Basic.hs --- a/Crypto/Number/Basic.hs +++ b/Crypto/Number/Basic.hs @@ -85,7 +85,12 @@ -- | Compute the number of bits for an integer numBits :: Integer -> Int -numBits n = gmpSizeInBits n `onGmpUnsupported` (if n == 0 then 1 else computeBits 0 n) +-- GMP sizes the magnitude and calls zero zero bits, and every caller here -- +-- 'numBytes' above all -- is written against that. The fallback used to +-- answer 1 for zero, and to divide a negative number by 256 forever, because +-- the quotient never reaches zero. +numBits n = + gmpSizeInBits n `onGmpUnsupported` (if n == 0 then 0 else computeBits 0 (abs n)) where computeBits !acc i | q == 0 = diff --git a/Crypto/Number/F2m.hs b/Crypto/Number/F2m.hs --- a/Crypto/Number/F2m.hs +++ b/Crypto/Number/F2m.hs @@ -36,9 +36,8 @@ (.&.), (.|.), ) -import Data.List (foldl') +import qualified Data.List as L import Data.Word (Word32) -import Prelude hiding (foldl') -- | Binary Polynomial represented by an integer type BinaryPolynomial = Integer @@ -89,7 +88,7 @@ | otherwise = fold es - (foldl' (\acc e -> acc `xor` (hi `shiftL` e)) (n .&. mask) es) + (L.foldl' (\acc e -> acc `xor` (hi `shiftL` e)) (n .&. mask) es) where hi = n `shiftR` lfx {-# INLINE modF2m #-} @@ -211,7 +210,7 @@ where spread :: Int -> Word32 spread b = - foldl' (\acc i -> if testBit b i then setBit acc (2 * i) else acc) 0 [0 .. 7] + L.foldl' (\acc i -> if testBit b i then setBit acc (2 * i) else acc) 0 [0 .. 7] {-# NOINLINE spreadTable #-} {-# INLINE squareF2m' #-} diff --git a/Crypto/Number/ModArithmetic.hs b/Crypto/Number/ModArithmetic.hs --- a/Crypto/Number/ModArithmetic.hs +++ b/Crypto/Number/ModArithmetic.hs @@ -43,6 +43,7 @@ import Crypto.Internal.Compat (unsafeDoIO) import Crypto.Number.Basic import Crypto.Number.Compat +import Crypto.Number.ModArithmetic.Pub (expPub) import qualified Crypto.Number.Serialize.Internal as Internal import Data.Memory.PtrMethods (memSet) import Data.Word (Word32, Word8) @@ -141,6 +142,7 @@ -- what GMP's own secure exponentiation lets slip !eLen = 8 * ((numBytes e + 7) `div` 8) + foreign import ccall safe "crypton_powm_sec" c_powm_sec :: Ptr Word8 @@ -158,6 +160,14 @@ -- -- Use this function when all the parameters are public, -- otherwise 'expSafe' should be preferred. +-- +-- A short exponent on an odd modulus -- RSA verification, where the exponent +-- is 65537 and comes out of the public key -- is walked a bit at a time in +-- C, squaring at each step and multiplying where a bit is set, which reads +-- the exponent and branches on it and is why this is not 'expSafe'. That +-- path is taken only where it is the faster of the two, which the C decides; +-- everywhere else, and for every longer exponent, this is GMP's windowed +-- exponentiation as it has always been. expFast :: Integer -- ^ base @@ -167,13 +177,29 @@ -- ^ modulo -> Integer -- ^ result -expFast b e m = gmpPowModInteger b e m `onGmpUnsupported` exponentiation b e m +expFast b e m + | odd m && m > 1 && e > 0 = case expPub (b `mod` m) e m of + Just r -> r + Nothing -> slow + | otherwise = slow + where + slow = gmpPowModInteger b e m `onGmpUnsupported` exponentiation b e m -- | @exponentiation@ computes modular exponentiation as /b^e mod m/ -- using repetitive squaring. +-- +-- The corner cases are held to what GMP answers, since that is what this +-- computes on every build that has it: a modulus of one is zero whatever +-- else is asked, and a negative exponent is a request for the inverse of +-- the base raised to its magnitude, which is zero when no inverse exists. +-- Read literally, the recursion below walked a negative exponent from -1 to +-- -2 and back for as long as the stack held. exponentiation :: Integer -> Integer -> Integer -> Integer exponentiation b e m - | b == 1 = b + | m == 1 = 0 + | e < 0 = + maybe 0 (\bInv -> exponentiation bInv (negate e) m) (inverse (b `mod` m) m) + | b == 1 = 1 | e == 0 = 1 | e == 1 = b `mod` m | even e = @@ -240,20 +266,56 @@ -- bits it is given, and a nonce inverted that way has been taken apart before -- by watching the steps go by. -- --- The moduli this is for -- the order of a group -- are prime, so the inverse --- comes from 'inverseFermat' instead. When the modulus is not prime, that --- answer is not an inverse, and the result is checked and 'inverse' asked --- instead, so this agrees with 'inverse' on every input. That fallback is --- reached only by parameters that are already broken. +-- The answer comes from a fixed number of division steps where the assembly +-- for them is built, and from 'inverseFermat' where it is not. Either way it +-- is checked here by multiplying out: neither one says when the number has no +-- inverse -- the first returns something that is not one and the second +-- returns something that is not one either -- so the check is what makes this +-- agree with 'inverse' on every input, and 'inverse' is asked when it fails. +-- That fallback is reached only by parameters that are already broken. -- --- It costs what an exponentiation costs: around thirty times an 'inverse' --- for a 256-bit modulus. +-- The division steps cost about a twentieth of the exponentiation: on an +-- Apple M4, inverting modulo the P-256 group order is 0.80 microseconds +-- against 6.02, and modulo the P-521 one 2.05 against 63.2. inverseSafe :: Integer -> Integer -> Maybe Integer inverseSafe g m | m > 1 && (g * r) `mod` m == 1 = Just r | otherwise = inverse g m where - r = inverseFermat g m + r = case inverseSec g m of + Just v -> v + Nothing -> inverseFermat g m + +-- | The inverse in a fixed number of division steps, from the vendored +-- assembly. 'Nothing' when that is not built, when the modulus is even -- +-- where the routine answers without saying it cannot -- or when the numbers +-- are larger than it keeps room for. The answer is not checked here; the +-- caller does that. +inverseSec :: Integer -> Integer -> Maybe Integer +inverseSec g m + | m <= 1 || even m || g < 0 = Nothing + | otherwise = unsafeDoIO $ + allocaBytes (3 * mLen) $ \out -> do + let gp = out `plusPtr` mLen + mp = gp `plusPtr` mLen + _ <- Internal.i2ospOf (g `mod` m) gp mLen + _ <- Internal.i2ospOf m mp mLen + r <- c_modinv_sec out gp mp (fromIntegral mLen) + if r == 0 + then do + !v <- Internal.os2ip out mLen + return (Just v) + else return Nothing + where + !mLen = numBytes m + +foreign import ccall unsafe "crypton_modinv_sec" + c_modinv_sec + :: Ptr Word8 + -> Ptr Word8 + -> Ptr Word8 + -> Word32 + -> IO CInt -- | Raised when the assumption about the modulus is invalid. data ModulusAssertionError = ModulusAssertionError diff --git a/Crypto/Number/ModArithmetic/Pub.hs b/Crypto/Number/ModArithmetic/Pub.hs new file mode 100644 --- /dev/null +++ b/Crypto/Number/ModArithmetic/Pub.hs @@ -0,0 +1,158 @@ +{-# LANGUAGE BangPatterns #-} + +-- | +-- Module : Crypto.Number.ModArithmetic.Pub +-- License : BSD-style +-- Stability : experimental +-- Portability : Good +-- +-- The C exponentiation for an exponent that is not a secret, and the R^2 +-- it keeps per modulus. Internal: 'expPub' is behind 'expFast', and +-- 'expPubBytes' behind RSA's public-key operation. +module Crypto.Number.ModArithmetic.Pub ( + expPub, + expPubBytes, +) where + +import Crypto.Internal.ByteArray ( + ByteArray, + ByteArrayAccess, + Bytes, + allocRet, + withByteArray, + ) +import qualified Crypto.Internal.ByteArray as B +import Crypto.Internal.Compat (unsafeDoIO) +import Crypto.Number.Basic +import qualified Crypto.Number.Serialize.Internal as Internal +import Data.IORef (IORef, atomicModifyIORef', newIORef) +import Data.Word (Word32, Word8) +import Foreign.C.Types (CInt (..)) +import Foreign.Marshal.Alloc (allocaBytes) +import Foreign.Ptr (Ptr, plusPtr) +import System.IO.Unsafe (unsafePerformIO) + +-- | The windowless exponentiation in C, for an exponent that is not a +-- secret. 'Nothing' when the C declines, which it does on every machine +-- where what the caller already has is faster -- so this is an +-- optimisation and never the only path. The base has to be reduced and +-- the exponent positive, which the caller sees to. +expPub :: Integer -> Integer -> Integer -> Maybe Integer +expPub b e m = unsafeDoIO $ do + known <- r2For m + case known of + Nothing -> return Nothing + Just (r2, modu) -> + allocaBytes (sum widths) $ \start -> case scanl plusPtr start widths of + (out : base : expo : _) -> do + _ <- Internal.i2ospOf b base mLen + _ <- Internal.i2ospOf e expo eLen + r <- withByteArray modu $ \pm -> withByteArray r2 $ \pr2 -> + c_powm_pub_r2 + out + base + (fromIntegral mLen) + expo + (fromIntegral eLen) + pm + (fromIntegral mLen) + pr2 + if r == 0 + then do + !v <- Internal.os2ip out mLen + return (Just v) + else return Nothing + _ -> return Nothing + where + !mLen = numBytes m + !eLen = numBytes e + widths = [mLen, mLen, eLen] + +-- | The same with the base and the answer as big-endian bytes, as long as +-- the modulus: what RSA verification has in hand, so neither goes through +-- an 'Integer'. 'Nothing' also when the base is not below the modulus. +expPubBytes + :: (ByteArrayAccess bi, ByteArray bo) => Integer -> Integer -> bi -> Maybe bo +expPubBytes e m b + | e <= 0 || m <= 1 || even m || B.length b /= mLen = Nothing + | otherwise = unsafeDoIO $ do + known <- r2For m + case known of + Nothing -> return Nothing + Just (r2, modu) + | not (below b modu) -> return Nothing + | otherwise -> allocaBytes eLen $ \expo -> do + _ <- Internal.i2ospOf e expo eLen + (r, out) <- allocRet mLen $ \pout -> + withByteArray b $ \pb -> withByteArray modu $ \pm -> + withByteArray r2 $ \pr2 -> + c_powm_pub_r2 + pout + pb + (fromIntegral mLen) + expo + (fromIntegral eLen) + pm + (fromIntegral mLen) + pr2 + return (if r == 0 then Just out else Nothing) + where + !mLen = numBytes m + !eLen = numBytes e + +-- big-endian numbers of one length: the first byte that differs decides +below :: (ByteArrayAccess a, ByteArrayAccess b) => a -> b -> Bool +below x y = go 0 + where + n = B.length x + go i + | i >= n = False + | otherwise = case compare (B.index x i) (B.index y i) of + EQ -> go (i + 1) + o -> o == LT + +-- | R^2 mod m for the public path, and m in bytes, kept for the last few +-- moduli, since a program verifies under the same keys again and again. +-- 'Nothing' is kept too: it means the C declines this modulus, which is +-- then not asked again. +r2For :: Integer -> IO (Maybe (Bytes, Bytes)) +r2For m = do + found <- atomicModifyIORef' r2Table $ \ks -> case break ((== m) . fst) ks of + (before, hit : after) -> (hit : before ++ after, Just (snd hit)) + _ -> (ks, Nothing) + case found of + Just r2 -> return r2 + Nothing -> do + let mLen = numBytes m + (r2, modu) <- allocRet mLen $ \modu -> do + _ <- Internal.i2ospOf m modu mLen + (r, out) <- allocRet mLen $ \pout -> c_powm_r2 pout modu (fromIntegral mLen) + return (if r == 0 then Just out else Nothing) + let kept = fmap (\x -> (x, modu)) r2 + atomicModifyIORef' r2Table $ \ks -> (take r2TableSize ((m, kept) : ks), ()) + return kept + +r2TableSize :: Int +r2TableSize = 8 + +r2Table :: IORef [(Integer, Maybe (Bytes, Bytes))] +r2Table = unsafePerformIO (newIORef []) +{-# NOINLINE r2Table #-} + +-- safe, as the secure one is: the call is tens of microseconds at RSA +-- sizes, and the hundred nanoseconds this costs over an unsafe call is +-- nothing beside holding a capability for that long in a server +foreign import ccall safe "crypton_powm_pub_r2" + c_powm_pub_r2 + :: Ptr Word8 + -> Ptr Word8 + -> Word32 + -> Ptr Word8 + -> Word32 + -> Ptr Word8 + -> Word32 + -> Ptr Word8 + -> IO CInt + +foreign import ccall unsafe "crypton_powm_r2" + c_powm_r2 :: Ptr Word8 -> Ptr Word8 -> Word32 -> IO CInt diff --git a/Crypto/Number/Serialize/Internal.hs b/Crypto/Number/Serialize/Internal.hs --- a/Crypto/Number/Serialize/Internal.hs +++ b/Crypto/Number/Serialize/Internal.hs @@ -53,7 +53,12 @@ !padSz = ptrSz - sz fillPtr :: Ptr Word8 -> Int -> Integer -> IO () -fillPtr p sz m = gmpExportInteger m p `onGmpUnsupported` export (sz - 1) m +fillPtr p sz m + -- zero is no bytes wide, and the callers above have already written the + -- room out as zeros. Without this the loop below starts at offset -1, + -- never meets the 0 it stops at, and walks backwards out of the buffer. + | sz <= 0 = return () + | otherwise = gmpExportInteger m p `onGmpUnsupported` export (sz - 1) m where export ofs i | ofs == 0 = pokeByteOff p ofs (fromIntegral i :: Word8) diff --git a/Crypto/OTP.hs b/Crypto/OTP.hs --- a/Crypto/OTP.hs +++ b/Crypto/OTP.hs @@ -49,9 +49,8 @@ import Crypto.MAC.HMAC import Data.Bits (complement, shiftL, shiftR, xor, (.&.), (.|.)) import Data.ByteArray.Mapping (fromW64BE) -import Data.List (foldl') +import qualified Data.List as L import Data.Word -import Prelude hiding (foldl') -- | A one-time password which is a sequence of 4 to 9 digits. type OTP = Word32 @@ -150,7 +149,7 @@ range = map (hotp h d k) [c .. c + fromIntegral s] -- the offset of the first match, accumulated without stopping there - (matched, offset) = foldl' pick (0, 0) (zip [0 ..] range) + (matched, offset) = L.foldl' pick (0, 0) (zip [0 ..] range) pick (!m, !off) (i, candidate) = (m .|. hit, off .|. (hit .&. i)) where -- zero once something has matched, so only the first match counts @@ -162,7 +161,7 @@ -- the counters continue past the window, and wrap where the old -- 'checkExtraOtps' wrapped extrasMatched = - foldl' step (complement 0) (zip (iterate (+ 1) afterFirst) extras) + L.foldl' step (complement 0) (zip (iterate (+ 1) afterFirst) extras) step acc (ctr, p) = acc .&. eqMask (hotp h d k ctr) p -- | All ones when the two values are equal, zero otherwise, without branching @@ -250,7 +249,7 @@ -- every candidate is compared, and none of the comparisons stops early, so -- neither which step matched nor how far a mismatch got is visible in how -- long this takes - matched = foldl' step 0 (map (hotp h d k) (range window [])) + matched = L.foldl' step 0 (map (hotp h d k) (range window [])) step acc candidate = acc .|. eqMask candidate otp timeToCounter :: Word64 -> Word64 -> Word16 -> Word64 diff --git a/Crypto/PubKey/Curve25519.hs b/Crypto/PubKey/Curve25519.hs --- a/Crypto/PubKey/Curve25519.hs +++ b/Crypto/PubKey/Curve25519.hs @@ -1,5 +1,4 @@ {-# LANGUAGE GeneralizedNewtypeDeriving #-} -{-# LANGUAGE MagicHash #-} {-# LANGUAGE ScopedTypeVariables #-} -- | @@ -31,7 +30,6 @@ import Data.Word import Foreign.Ptr import Foreign.Storable -import GHC.Ptr import Crypto.Error import Crypto.Internal.ByteArray ( @@ -114,20 +112,19 @@ $ \result -> withByteArray sec $ \psec -> withByteArray pub $ \ppub -> - ccrypton_curve25519 result psec ppub + ccrypton_x25519 result psec ppub {-# NOINLINE dh #-} -- | Create a public key from a secret key +-- The base point does not go in: where the assembly is built there is a table +-- for this, and it is four to five times less work than multiplying the point +-- 9 the general way. toPublic :: SecretKey -> PublicKey toPublic (SecretKey sec) = PublicKey <$> B.allocAndFreeze 32 $ \result -> withByteArray sec $ \psec -> - ccrypton_curve25519 result psec basePoint - where - basePoint = - Ptr - "\x09\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"# + ccrypton_x25519_base result psec {-# NOINLINE toPublic #-} -- | Generate a secret key. @@ -142,12 +139,20 @@ modifyByte :: Ptr Word8 -> Int -> (Word8 -> Word8) -> IO () modifyByte p n f = peekByteOff p n >>= pokeByteOff p n . f -foreign import ccall "crypton_curve25519_donna" - ccrypton_curve25519 +foreign import ccall "crypton_x25519" + ccrypton_x25519 :: Ptr Word8 -- ^ public -> Ptr Word8 -- ^ secret -> Ptr Word8 -- ^ basepoint + -> IO () + +foreign import ccall "crypton_x25519_base" + ccrypton_x25519_base + :: Ptr Word8 + -- ^ public + -> Ptr Word8 + -- ^ secret -> IO () diff --git a/Crypto/PubKey/DSA.hs b/Crypto/PubKey/DSA.hs --- a/Crypto/PubKey/DSA.hs +++ b/Crypto/PubKey/DSA.hs @@ -41,9 +41,12 @@ -- * Signature primitive sign, signWith, + signDigest, + signDigestWith, -- * Verification primitive verify, + verifyDigest, -- * Key pair KeyPair (..), @@ -59,7 +62,7 @@ import Crypto.Internal.Imports import Crypto.Number.Generate import Crypto.Number.ModArithmetic (expFast, expSafe, inverse, inverseSafe) -import Crypto.PubKey.Internal (dsaTruncHash) +import Crypto.PubKey.Internal (dsaTruncHashDigest) import Crypto.Random.Types -- | DSA Public Number, usually embedded in DSA Public Key @@ -194,14 +197,31 @@ -> msg -- ^ message to sign -> Maybe Signature -signWith k pk hashAlg msg = do +signWith k pk hashAlg msg = signDigestWith k pk (hashWith hashAlg msg) + +-- | Sign a digest using the private key and an explicit k number. +-- +-- The digest's type says which algorithm made it, so this needs nothing else +-- to name one. 'signWith' takes a @hash@ value and never reads it -- it is +-- there to fix the type -- which leaves a caller that is itself polymorphic +-- in the algorithm with nothing to pass. +signDigestWith + :: HashAlgorithm hash + => Integer + -- ^ k random number + -> PrivateKey + -- ^ private key + -> Digest hash + -- ^ digest of the message to sign + -> Maybe Signature +signDigestWith k pk digest = do -- k comes from the caller and is only invertible when it is coprime with -- q, which the caller cannot check without knowing q is prime. It is also -- a secret worth as much as the private key, so it is inverted without -- the extended Euclidean algorithm, whose steps follow the bits of what -- it is given kInv <- inverseSafe k q - let hm = dsaTruncHash hashAlg msg q + let hm = dsaTruncHashDigest digest q r = expSafe g k p `mod` q s = (kInv * (hm + x * r)) `mod` q if r == 0 || s == 0 then Nothing else Just $ Signature r s @@ -214,10 +234,17 @@ sign :: (ByteArrayAccess msg, HashAlgorithm hash, MonadRandom m) => PrivateKey -> hash -> msg -> m Signature -sign pk hashAlg msg = do +sign pk hashAlg msg = signDigest pk (hashWith hashAlg msg) + +-- | Sign a digest using the private key. See 'signDigestWith' for why a +-- digest rather than a @hash@ value. +signDigest + :: (HashAlgorithm hash, MonadRandom m) + => PrivateKey -> Digest hash -> m Signature +signDigest pk digest = do k <- generateMax q - case signWith k pk hashAlg msg of - Nothing -> sign pk hashAlg msg + case signDigestWith k pk digest of + Nothing -> signDigest pk digest Just sig -> return sig where (Params _ _ q) = private_params pk @@ -226,7 +253,13 @@ verify :: (ByteArrayAccess msg, HashAlgorithm hash) => hash -> PublicKey -> Signature -> msg -> Bool -verify hashAlg pk (Signature r s) m +verify hashAlg pk sig m = verifyDigest pk sig (hashWith hashAlg m) + +-- | Verify a signature over a digest. See 'signDigestWith' for why a digest +-- rather than a @hash@ value. +verifyDigest + :: HashAlgorithm hash => PublicKey -> Signature -> Digest hash -> Bool +verifyDigest pk (Signature r s) digest -- Reject the signature if either 0 < r < q or 0 < s < q is not satisfied. | r <= 0 || r >= q || s <= 0 || s >= q = False -- s is invertible for every 0 < s < q when q is prime, but the parameters @@ -235,7 +268,7 @@ where (Params p g q) = public_params pk y = public_y pk - hm = dsaTruncHash hashAlg m q + hm = dsaTruncHashDigest digest q v = do w <- inverse s q let u1 = (hm * w) `mod` q diff --git a/Crypto/PubKey/ECC/P256.hs b/Crypto/PubKey/ECC/P256.hs --- a/Crypto/PubKey/ECC/P256.hs +++ b/Crypto/PubKey/ECC/P256.hs @@ -413,8 +413,6 @@ foreign import ccall "crypton_p256e_scalar_invert" ccrypton_p256e_scalar_invert :: Ptr P256Scalar -> Ptr P256Scalar -> IO () --- foreign import ccall "crypton_p256_modinv" --- ccrypton_p256_modinv :: Ptr P256Scalar -> Ptr P256Scalar -> Ptr P256Scalar -> IO () foreign import ccall "crypton_p256_modinv_vartime" ccrypton_p256_modinv_vartime :: Ptr P256Scalar -> Ptr P256Scalar -> Ptr P256Scalar -> IO () diff --git a/Crypto/PubKey/ECDSA.hs b/Crypto/PubKey/ECDSA.hs --- a/Crypto/PubKey/ECDSA.hs +++ b/Crypto/PubKey/ECDSA.hs @@ -48,6 +48,11 @@ signDigest, verify, verifyDigest, + + -- * Deterministic nonces + deterministicNonce, + signDeterministic, + signDigestDeterministic, ) where import Control.Monad @@ -58,8 +63,10 @@ import Crypto.Hash import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess) import Crypto.Internal.Imports -import Crypto.Number.ModArithmetic (inverseFermat) +import Crypto.Number.Generate (generatePrefix) +import Crypto.Number.ModArithmetic (inverseSafe) import qualified Crypto.PubKey.ECC.P256 as P256 +import Crypto.Random.HmacDRG (initial, update) import Crypto.Random.Types import Data.Bits @@ -257,6 +264,75 @@ verify prx hashAlg q sig msg = verifyDigest prx q sig (hashWith hashAlg msg) -- | Truncate a digest based on curve order size. +-- | Deterministic nonce generation according to RFC 6979. +-- +-- The nonce is derived from the private key and the message alone, so a +-- signature made this way needs no random number generator and cannot be the +-- one that repeats a nonce -- which, for ECDSA, hands over the private key. +-- +-- The hash used to seed the generator is given separately from the one the +-- message was digested with, as RFC 6979 allows. +-- +-- The last argument is what to do with a candidate nonce. It may answer +-- 'Nothing', in which case another candidate is drawn, which is what +-- 'signDigestDeterministic' does for the r or s that comes out zero: +-- +-- > deterministicNonce prx SHA256 priv digest (\k -> signDigestWith prx k priv digest) +deterministicNonce + :: (EllipticCurveECDSA curve, HashAlgorithm hashDRG, HashAlgorithm hashDigest) + => proxy curve + -> hashDRG + -> PrivateKey curve + -> Digest hashDigest + -> (Scalar curve -> Maybe a) + -> a +deterministicNonce prx alg d digest go = fst $ withDRG state run + where + state = update seed $ initial alg + -- RFC 6979 section 3.2 step d: int2octets(x) || bits2octets(h1). The + -- second is the truncated digest taken modulo the order, which is what + -- scalarAdd with zero does, its contract being to reduce there. + seed = + B.append (encodeScalar prx d) (encodeScalar prx z) + :: B.ScrubbedBytes + z = scalarAdd prx (tHashDigest prx digest) zeroScalar + zeroScalar = throwCryptoError $ scalarFromInteger prx 0 + run = do + k <- generatePrefix (curveOrderBits prx) + case scalarFromInteger prx k of + CryptoPassed s + | scalarIsValid prx s -> maybe run pure (go s) + _ -> run + +-- | Sign a digest with a nonce derived from the private key and the digest, +-- as RFC 6979 says, rather than from a random number generator. +signDigestDeterministic + :: (EllipticCurveECDSA curve, HashAlgorithm hashDRG, HashAlgorithm hashDigest) + => proxy curve + -> hashDRG + -> PrivateKey curve + -> Digest hashDigest + -> Signature curve +signDigestDeterministic prx alg d digest = + deterministicNonce prx alg d digest $ \k -> signDigestWith prx k d digest + +-- | Sign a message with a nonce derived from the private key and the message, +-- as RFC 6979 says, rather than from a random number generator. +signDeterministic + :: ( EllipticCurveECDSA curve + , HashAlgorithm hashDRG + , HashAlgorithm hash + , ByteArrayAccess msg + ) + => proxy curve + -> hashDRG + -> PrivateKey curve + -> hash + -> msg + -> Signature curve +signDeterministic prx alg d hashAlg msg = + signDigestDeterministic prx alg d (hashWith hashAlg msg) + tHashDigest :: (EllipticCurveECDSA curve, HashAlgorithm hash) => proxy curve -> Digest hash -> Scalar curve @@ -296,15 +372,16 @@ => Simple.Scalar curve -> Bool ecScalarIsZero (Simple.Scalar a) = a == 0 +-- | 'inverseSafe' is the one that takes a fixed number of division steps +-- where the assembly for them is built, and checks whatever it gets by +-- multiplying out. It answers 'Nothing' exactly where the exponentiation +-- this used to do answered zero. ecScalarInv :: Simple.Curve c => proxy c -> Simple.Scalar c -> Maybe (Simple.Scalar c) -ecScalarInv prx (Simple.Scalar s) - | i == 0 = Nothing - | otherwise = Just $ Simple.Scalar i +ecScalarInv prx (Simple.Scalar s) = Simple.Scalar <$> inverseSafe s n where n = Simple.curveEccN $ Simple.curveParameters prx - i = inverseFermat s n ecPointX :: Simple.Curve c diff --git a/Crypto/PubKey/ElGamal.hs b/Crypto/PubKey/ElGamal.hs --- a/Crypto/PubKey/ElGamal.hs +++ b/Crypto/PubKey/ElGamal.hs @@ -57,10 +57,13 @@ -- * Signature primitives signWith, + signDigestWith, sign, + signDigest, -- * Verification primitives verify, + verifyDigest, ) where import Crypto.Error @@ -194,8 +197,28 @@ -> msg -- ^ message to sign -> Maybe Signature -signWith = signWithBlinder 1 +signWith k params priv hashAlg msg = + signDigestWith k params priv (hashWith hashAlg msg) +-- | Sign a digest with an explicit ephemeral value. +-- +-- The digest's type says which algorithm made it, so this needs nothing else +-- to name one. 'signWith' takes a @hash@ value and never reads it -- it is +-- there to fix the type -- which leaves a caller that is itself polymorphic +-- in the algorithm with nothing to pass. +signDigestWith + :: HashAlgorithm hash + => Integer + -- ^ ephemeral value k, in [1, p-2] and coprime with p-1 + -> Params + -- ^ DH params (p,g) + -> PrivateNumber + -- ^ DH private key + -> Digest hash + -- ^ digest of the message to sign + -> Maybe Signature +signDigestWith = signWithBlinder 1 + -- | The same with a blinder for the inversion of @k@. -- -- @k@ is inverted modulo @p-1@, which is even, so Fermat's little theorem @@ -210,15 +233,15 @@ -- When @b@ shares a factor with @p-1@ the algorithm reports it the same way -- it reports one in @k@, and the answer is the same: draw again. signWithBlinder - :: (ByteArrayAccess msg, HashAlgorithm hash) - => Integer -> Integer -> Params -> PrivateNumber -> hash -> msg -> Maybe Signature -signWithBlinder b k (Params p g _) (PrivateNumber x) hashAlg msg + :: HashAlgorithm hash + => Integer -> Integer -> Params -> PrivateNumber -> Digest hash -> Maybe Signature +signWithBlinder b k (Params p g _) (PrivateNumber x) digest | k <= 0 || k >= p - 1 || b <= 0 || d > 1 = Nothing | s == 0 = Nothing | otherwise = Just $ Signature r s where r = expSafe g k p - h = os2ip $ hashWith hashAlg msg + h = os2ip digest s = ((h - x * r) * kInv) `mod` (p - 1) kInv = (kbInv * b) `mod` (p - 1) (kbInv, _, d) = gcde ((k * b) `mod` (p - 1)) (p - 1) @@ -239,13 +262,20 @@ -> msg -- ^ message to sign -> m Signature -sign params@(Params p _ _) priv hashAlg msg = do +sign params priv hashAlg msg = signDigest params priv (hashWith hashAlg msg) + +-- | Sign a digest, drawing the ephemeral value. See 'signDigestWith' for +-- why a digest rather than a @hash@ value. +signDigest + :: (HashAlgorithm hash, MonadRandom m) + => Params -> PrivateNumber -> Digest hash -> m Signature +signDigest params@(Params p _ _) priv digest = do k <- generateMax (p - 1) -- and a blinder for the inversion of k, which is the one step here that -- the extended Euclidean algorithm has to do b <- generateMax (p - 1) - case signWithBlinder b k params priv hashAlg msg of - Nothing -> sign params priv hashAlg msg + case signWithBlinder b k params priv digest of + Nothing -> signDigest params priv digest Just sig -> return sig -- | verify a signature @@ -257,10 +287,18 @@ -> msg -> Signature -> Bool -verify (Params p g _) (PublicNumber y) hashAlg msg (Signature r s) +verify params pub hashAlg msg sig = + verifyDigest params pub (hashWith hashAlg msg) sig + +-- | Verify a signature over a digest. See 'signDigestWith' for why a digest +-- rather than a @hash@ value. +verifyDigest + :: HashAlgorithm hash + => Params -> PublicNumber -> Digest hash -> Signature -> Bool +verifyDigest (Params p g _) (PublicNumber y) digest (Signature r s) | or [r <= 0, r >= p, s <= 0, s >= (p - 1)] = False | otherwise = lhs == rhs where - h = os2ip $ hashWith hashAlg msg + h = os2ip digest lhs = expFast g h p rhs = (expFast y r p * expFast r s p) `mod` p diff --git a/Crypto/PubKey/Internal.hs b/Crypto/PubKey/Internal.hs --- a/Crypto/PubKey/Internal.hs +++ b/Crypto/PubKey/Internal.hs @@ -12,8 +12,7 @@ ) where import Data.Bits (shiftR) -import Data.List (foldl') -import Prelude hiding (foldl') +import qualified Data.List as L import Crypto.Hash import Crypto.Internal.ByteArray (ByteArrayAccess) @@ -22,7 +21,7 @@ -- | This is a strict version of and and' :: [Bool] -> Bool -and' l = foldl' (&&!) True l +and' l = L.foldl' (&&!) True l -- | This is a strict version of &&. (&&!) :: Bool -> Bool -> Bool diff --git a/Crypto/PubKey/MLDSA.hs b/Crypto/PubKey/MLDSA.hs new file mode 100644 --- /dev/null +++ b/Crypto/PubKey/MLDSA.hs @@ -0,0 +1,656 @@ +-- | +-- Module : Crypto.PubKey.MLDSA +-- License : BSD-style +-- Maintainer : Kazu Yamamoto +-- Stability : experimental +-- Portability : unknown +-- +-- ML-DSA, the Module-Lattice-Based Digital Signature Algorithm of +-- , in all three +-- parameter sets. +-- +-- > (vk, sk) <- generateKeyPair MLDSA65 +-- > sig <- sign sk emptyContext message +-- > verify vk emptyContext message sig +-- +-- What 'generateKeyPair' and 'sign' draw their randomness from is the +-- 'Crypto.Random.MonadRandom' instance in use. Its documentation says what +-- an instance of your own has to be. +-- +-- The parameter set is a type, so an ML-DSA-65 key cannot be passed where +-- an ML-DSA-87 one is expected. The three are fixed by FIPS 204 and the +-- class has no other instances. +-- +-- This is pure ML-DSA: the message goes in whole. The pre-hash variant +-- (HashML-DSA) is a different algorithm with a different domain separator +-- and is not offered here. +{-# LANGUAGE DataKinds #-} +{-# LANGUAGE GeneralizedNewtypeDeriving #-} +{-# LANGUAGE ScopedTypeVariables #-} + +module Crypto.PubKey.MLDSA ( + -- * Parameter sets + MLDSA44 (..), + MLDSA65 (..), + MLDSA87 (..), + MLDSA (verificationKeySize, signingKeySize, signatureSize), + + -- * Keys and signatures + VerificationKey, + SigningKey, + Signature, + + -- * Smart constructors + verificationKey, + signingKey, + signature, + + -- * Generating a key pair + generateKeyPair, + generateKeyPairAndSeed, + keyPairFromSeed, + toPublic, + + -- * The context string + Context, + context, + emptyContext, + + -- * The message representative + Mu, + mu, + messageRepresentative, + + -- ** A message that does not arrive in one piece + MuContext, + muInit, + muUpdate, + muUpdates, + muFinalize, + + -- * Signing and verifying + sign, + signWith, + signDeterministic, + verify, + + -- * Signing and verifying a message representative + signExternalMu, + signExternalMuWith, + signExternalMuDeterministic, + verifyExternalMu, + + -- * Sizes + seedSize, + signingRandomnessSize, + maxContextLength, + muSize, +) where + +import Data.Proxy (Proxy (..)) +import Foreign.C.Types (CInt (..), CSize (..)) +import Foreign.Ptr (Ptr, nullPtr) + +import Crypto.Debug (DebugShow (..), debugShowBytes) +import Crypto.Hash (Digest, hash, hashFinalize, hashInit, hashUpdate, hashUpdates) +import qualified Crypto.Hash as Hash (Context) +import Crypto.Hash.Algorithms (SHAKE256 (..)) +import Crypto.Error +import Crypto.Internal.ByteArray ( + ByteArrayAccess, + Bytes, + ScrubbedBytes, + withByteArray, + ) +import qualified Crypto.Internal.ByteArray as B +import Crypto.Internal.Compat (unsafeDoIO) +import Crypto.Internal.Imports +import Crypto.Random (MonadRandom, getRandomBytes) + +-- | ML-DSA-44. +data MLDSA44 = MLDSA44 deriving (Show, Eq) + +-- | ML-DSA-65. +data MLDSA65 = MLDSA65 deriving (Show, Eq) + +-- | ML-DSA-87. +data MLDSA87 = MLDSA87 deriving (Show, Eq) + +-- | The three parameter sets of FIPS 204. +-- +-- Named for the algorithm rather than \"DSA\", which is a different one that +-- crypton also has, in "Crypto.PubKey.DSA". It is the three sets FIPS 204 +-- defines, closed, carrying their sizes and the calls into the +-- implementation; only the sizes are exported. +class MLDSA p where + -- | Size in bytes of a 'VerificationKey' of this parameter set. + verificationKeySize :: proxy p -> Int + + -- | Size in bytes of a 'SigningKey' of this parameter set. + signingKeySize :: proxy p -> Int + + -- | Size in bytes of a 'Signature' of this parameter set. + signatureSize :: proxy p -> Int + + c_keypair :: proxy p -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt + c_sign + :: proxy p + -> Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize + -> Ptr Word8 -> Ptr Word8 -> CInt -> IO CInt + c_verify + :: proxy p + -> Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize + -> Ptr Word8 -> CInt -> IO CInt + c_pkFromSk :: proxy p -> Ptr Word8 -> Ptr Word8 -> IO CInt + +-- | A public verification key. +newtype VerificationKey p = VerificationKey Bytes + deriving (Show, Eq, ByteArrayAccess, NFData) + +-- | A private signing key. +newtype SigningKey p = SigningKey ScrubbedBytes + deriving (Eq, ByteArrayAccess, NFData) + +instance Show (SigningKey p) where + show _ = "SigningKey " + +instance DebugShow (SigningKey p) where + debugShow = debugShowBytes "SigningKey" + +-- | A signature. +newtype Signature p = Signature Bytes + deriving (Show, Eq, ByteArrayAccess, NFData) + +-- | The context string a signature is bound to, at most +-- 'maxContextLength' bytes. +-- +-- FIPS 204 mixes it into what is signed, so a signature made under one +-- context does not verify under another. Two uses of one key that don't +-- agree on a context string cannot be made to accept each other's +-- signatures. Use 'emptyContext' where there is nothing to separate -- TLS, +-- for one, signs with an empty context. +newtype Context = Context Bytes + deriving (Show, Eq, ByteArrayAccess, NFData) + +-- | The context string of length zero, which is what to sign under when +-- there is nothing to separate. +-- +-- It is a context and not the absence of one: 'sign' always takes one, and +-- what this separates from is every non-empty context there is. +emptyContext :: Context +emptyContext = Context B.empty + +-- | Try to build a context string. +context :: ByteArrayAccess ba => ba -> CryptoFailable Context +context bs + | B.length bs <= maxContextLength = + CryptoPassed $ Context $ B.copyAndFreeze bs (\_ -> return ()) + | otherwise = CryptoFailed CryptoError_ParameterInvalid + +-- | The longest context string FIPS 204 allows, which is 255 bytes because +-- its length is encoded in one byte. +maxContextLength :: Int +maxContextLength = 255 + +-- | The message representative, @mu@ in FIPS 204: a 64-byte commitment to +-- the verification key, the context string and the message, and the only +-- part of them that signing and verification actually read. +-- +-- Signing it directly is the "external mu" interface. It is for a caller +-- that has the representative without having the message in one piece: a +-- message arriving as a stream, or hashed on another machine, or by a +-- device that holds the key and is handed only this. TLS does not need it. +newtype Mu = Mu Bytes + deriving (Show, Eq, ByteArrayAccess, NFData) + +-- | Size in bytes of a 'Mu'. +muSize :: Int +muSize = 64 + +-- | Try to read a message representative. +mu :: ByteArrayAccess ba => ba -> CryptoFailable Mu +mu bs + | B.length bs == muSize = CryptoPassed $ Mu $ B.copyAndFreeze bs (\_ -> return ()) + | otherwise = CryptoFailed CryptoError_ParameterInvalid + +-- | Compute the message representative, for a caller that wants to make it +-- here and sign it later, or sign it elsewhere. +-- +-- @'signExternalMuDeterministic' sk ('messageRepresentative' ('toPublic' sk) ctx msg)@ +-- and @'signDeterministic' sk ctx msg@ are the same signature. +messageRepresentative + :: (MLDSA p, ByteArrayAccess msg) + => VerificationKey p -> Context -> msg -> Mu +messageRepresentative vk ctx msg = muFinalize (muUpdate (muInit vk ctx) msg) + +-- | A 'Mu' being computed, with the message going in a piece at a time. +-- +-- The name is not 'Context': that is ML-DSA's context string, which this +-- is built from and is not. +newtype MuContext = MuContext (Hash.Context (SHAKE256 512)) + +-- | Begin a message representative. The key and the context string are +-- what it is bound to, and they are all that is needed before the message. +-- +-- > muFinalize (muUpdates (muInit vk ctx) chunks) +-- +-- is 'messageRepresentative' of the chunks joined, so a message too large +-- to hold at once never has to be. +muInit :: MLDSA p => VerificationKey p -> Context -> MuContext +muInit vk ctx = + -- FIPS 204: tr <- H(pk, 64) at key generation, and mu <- H(tr || M', 64) + -- when signing, with M' the domain-separated message. Everything up to + -- the message itself is absorbed here. + MuContext $ hashUpdates hashInit [tr, domainPrefix ctx] + where + tr = B.convert (shake64 (B.convert vk :: Bytes)) :: Bytes + +-- | Absorb a piece of the message. +muUpdate :: ByteArrayAccess msg => MuContext -> msg -> MuContext +muUpdate (MuContext c) msg = MuContext (hashUpdate c msg) + +-- | Absorb several pieces, which is 'muUpdate' one after the other. +muUpdates :: ByteArrayAccess msg => MuContext -> [msg] -> MuContext +muUpdates (MuContext c) msgs = MuContext (hashUpdates c msgs) + +-- | The message representative of everything absorbed so far. +muFinalize :: MuContext -> Mu +muFinalize (MuContext c) = Mu (B.convert (hashFinalize c :: Digest (SHAKE256 512))) + +shake64 :: ByteArrayAccess ba => ba -> Digest (SHAKE256 512) +shake64 = hash + +-- | Size in bytes of the seed 'keyPairFromSeed' takes, @xi@ in FIPS 204. +seedSize :: Int +seedSize = 32 + +-- | Size in bytes of the randomness 'signWith' takes. +signingRandomnessSize :: Int +signingRandomnessSize = 32 + +-- | Try to read a verification key. Only the length is checked: a +-- verification key is a packed encoding with no redundancy to test, and one +-- that is not a real key simply verifies nothing. +verificationKey + :: forall p ba + . (MLDSA p, ByteArrayAccess ba) + => ba -> CryptoFailable (VerificationKey p) +verificationKey bs + | B.length bs == verificationKeySize (Proxy :: Proxy p) = + CryptoPassed $ VerificationKey $ B.copyAndFreeze bs (\_ -> return ()) + | otherwise = CryptoFailed CryptoError_PublicKeySizeInvalid + +-- | Try to read a signing key. +-- +-- Beyond the length this runs the validity checks of the implementation: +-- the secret polynomials must have coefficients in range, and the +-- commitment and the public-key hash the key carries must match what is +-- recomputed from the rest of it. A key that fails has been damaged or was +-- never a key, and signing with it would produce signatures nothing +-- verifies. +signingKey + :: forall p ba + . (MLDSA p, ByteArrayAccess ba) + => ba -> CryptoFailable (SigningKey p) +signingKey bs + | B.length bs /= signingKeySize p = CryptoFailed CryptoError_SecretKeySizeInvalid + | otherwise = unsafeDoIO $ do + (r, _ :: Bytes) <- B.allocRet (verificationKeySize p) $ \ppk -> + withByteArray bs $ \psk -> c_pkFromSk p ppk psk + return $ + if r == 0 + then CryptoPassed $ SigningKey $ B.copyAndFreeze bs (\_ -> return ()) + else CryptoFailed CryptoError_SecretKeyStructureInvalid + where + p = Proxy :: Proxy p +{-# NOINLINE signingKey #-} + +-- | Try to read a signature. Only the length is checked; whether it is a +-- signature of anything is what 'verify' answers. +signature + :: forall p ba + . (MLDSA p, ByteArrayAccess ba) + => ba -> CryptoFailable (Signature p) +signature bs + | B.length bs == signatureSize (Proxy :: Proxy p) = + CryptoPassed $ Signature $ B.copyAndFreeze bs (\_ -> return ()) + | otherwise = CryptoFailed CryptoError_ParameterInvalid + +-- | Recover the verification key a signing key was made with. +toPublic :: forall p. MLDSA p => SigningKey p -> VerificationKey p +toPublic sk = VerificationKey $ unsafeDoIO $ do + (_ :: CInt, pk) <- B.allocRet (verificationKeySize p) $ \ppk -> + withByteArray sk $ \psk -> c_pkFromSk p ppk psk + return pk + where + p = Proxy :: Proxy p +{-# NOINLINE toPublic #-} + +-- | Generate a key pair. +-- +-- The seed it is derived from is drawn here and thrown away. Use +-- 'generateKeyPairAndSeed' where it has to be kept. +generateKeyPair + :: forall p proxy m + . (MLDSA p, MonadRandom m) + => proxy p -> m (VerificationKey p, SigningKey p) +generateKeyPair p = do + (vk, sk, _) <- generateKeyPairAndSeed p + return (vk, sk) + +-- | Generate a key pair and hand back the seed it was derived from, @xi@ +-- in FIPS 204. +-- +-- A 'SigningKey' is the expanded key and nothing else, so the seed cannot +-- be recovered from a pair afterwards. An application that has to write +-- the key out in a form that keeps the seed -- RFC 9881 lets an ML-DSA +-- private key be the seed, the expanded key, or both -- has to generate it +-- here: +-- +-- > (vk, sk, seed) <- generateKeyPairAndSeed MLDSA65 +-- +-- The seed is as secret as the signing key: 'keyPairFromSeed' turns it +-- back into the same pair. +generateKeyPairAndSeed + :: forall p proxy m + . (MLDSA p, MonadRandom m) + => proxy p -> m (VerificationKey p, SigningKey p, ScrubbedBytes) +generateKeyPairAndSeed p = do + seed <- getRandomBytes seedSize :: m ScrubbedBytes + case keyPairFromSeed p seed of + CryptoPassed (vk, sk) -> return (vk, sk, seed) + CryptoFailed e -> + error ("Crypto.PubKey.MLDSA.generateKeyPairAndSeed: " ++ show e) + +-- | Derive a key pair from a seed, @xi@ in FIPS 204, which must be +-- 'seedSize' bytes. +keyPairFromSeed + :: forall p proxy ba + . (MLDSA p, ByteArrayAccess ba) + => proxy p -> ba -> CryptoFailable (VerificationKey p, SigningKey p) +keyPairFromSeed p seed + | B.length seed /= seedSize = CryptoFailed CryptoError_SeedSizeInvalid + | otherwise = unsafeDoIO $ do + -- Not zeroed, and does not need to be: the C writes the whole + -- buffer, and on a non-zero return the result is discarded without + -- being read. Anything that is *read* before being written has to + -- use B.zero instead -- see signInternal in Crypto.PubKey.MLDSA. + sk <- B.alloc (signingKeySize p) (\_ -> return ()) :: IO ScrubbedBytes + (r, vk) <- B.allocRet (verificationKeySize p) $ \pvk -> + withByteArray sk $ \psk -> + withByteArray seed $ \pseed -> + c_keypair p pvk psk pseed + return $ + if r == 0 + then CryptoPassed (VerificationKey vk, SigningKey sk) + else CryptoFailed CryptoError_ParameterInvalid +{-# NOINLINE keyPairFromSeed #-} + +-- | Sign a message. +-- +-- This is the hedged signing FIPS 204 recommends: fresh randomness goes in +-- alongside the key and the message, so two signatures of one message +-- differ and a fault in one reveals less. Verification does not care which +-- of the three entry points made the signature. +sign + :: forall p m msg + . (MLDSA p, MonadRandom m, ByteArrayAccess msg) + => SigningKey p -> Context -> msg -> m (Signature p) +sign sk ctx msg = do + rnd <- getRandomBytes signingRandomnessSize :: m ScrubbedBytes + case signWith sk ctx msg rnd of + CryptoPassed s -> return s + CryptoFailed e -> error ("Crypto.PubKey.MLDSA.sign: " ++ show e) + +-- | Sign with the randomness supplied, which must be +-- 'signingRandomnessSize' bytes. +-- +-- For test vectors, and for callers who draw their own randomness. Ordinary +-- use wants 'sign'. +signWith + :: forall p msg rnd + . (MLDSA p, ByteArrayAccess msg, ByteArrayAccess rnd) + => SigningKey p -> Context -> msg -> rnd -> CryptoFailable (Signature p) +signWith sk ctx msg rnd + | B.length rnd /= signingRandomnessSize = CryptoFailed CryptoError_SeedSizeInvalid + | otherwise = signInternal sk ctx msg (Just rnd) + +-- | Sign deterministically, as FIPS 204 section 3.4 allows: the randomness +-- is replaced by zeroes, so one key and one message always give one +-- signature. +-- +-- This is what test vectors are written against, and what to use where the +-- signature must be reproducible. It gives up what hedging buys, so where +-- there is a usable random source 'sign' is the better default. +signDeterministic + :: forall p msg + . (MLDSA p, ByteArrayAccess msg) + => SigningKey p -> Context -> msg -> Signature p +signDeterministic sk ctx msg = + case signInternal sk ctx msg (Nothing :: Maybe Bytes) of + CryptoPassed s -> s + CryptoFailed e -> error ("Crypto.PubKey.MLDSA.signDeterministic: " ++ show e) + +signInternal + :: forall p msg rnd + . (MLDSA p, ByteArrayAccess msg, ByteArrayAccess rnd) + => SigningKey p -> Context -> msg -> Maybe rnd -> CryptoFailable (Signature p) +signInternal sk ctx msg mrnd = unsafeDoIO $ do + -- B.zero, not B.alloc with an empty action: alloc hands back whatever + -- was in the memory. That made signDeterministic sign with the last + -- caller's bytes and produce a different signature every time, which the + -- ACVP vectors caught only once the whole suite ran and the allocator + -- stopped handing out fresh zeroed pages. + let zeroes = B.zero signingRandomnessSize :: ScrubbedBytes + withRnd f = case mrnd of + Just r -> withByteArray r f + Nothing -> withByteArray zeroes f + (r, sig) <- B.allocRet (signatureSize p) $ \psig -> + withByteArray msg $ \pmsg -> + withByteArray pre $ \ppre -> + withRnd $ \prnd -> + withByteArray sk $ \psk -> + c_sign + p + psig + pmsg + (fromIntegral (B.length msg)) + ppre + (fromIntegral (B.length pre)) + prnd + psk + 0 + return $ + if r == 0 + then CryptoPassed (Signature sig) + else CryptoFailed CryptoError_ParameterInvalid + where + p = Proxy :: Proxy p + pre = domainPrefix ctx +{-# NOINLINE signInternal #-} + +-- | Sign a message representative, drawing the randomness. +-- +-- The context string is already inside the representative, which is why +-- this does not take one. +signExternalMu + :: forall p m + . (MLDSA p, MonadRandom m) + => SigningKey p -> Mu -> m (Signature p) +signExternalMu sk m = do + rnd <- getRandomBytes signingRandomnessSize :: m ScrubbedBytes + case signExternalMuWith sk m rnd of + CryptoPassed s -> return s + CryptoFailed e -> error ("Crypto.PubKey.MLDSA.signExternalMu: " ++ show e) + +-- | Sign a message representative with the randomness supplied. +signExternalMuWith + :: (MLDSA p, ByteArrayAccess rnd) + => SigningKey p -> Mu -> rnd -> CryptoFailable (Signature p) +signExternalMuWith sk m rnd + | B.length rnd /= signingRandomnessSize = CryptoFailed CryptoError_SeedSizeInvalid + | otherwise = signMu sk m (Just rnd) + +-- | Sign a message representative deterministically. +signExternalMuDeterministic + :: MLDSA p => SigningKey p -> Mu -> Signature p +signExternalMuDeterministic sk m = + case signMu sk m (Nothing :: Maybe Bytes) of + CryptoPassed s -> s + CryptoFailed e -> + error ("Crypto.PubKey.MLDSA.signExternalMuDeterministic: " ++ show e) + +-- | Verify a signature of a message representative. +verifyExternalMu + :: forall p. MLDSA p => VerificationKey p -> Mu -> Signature p -> Bool +verifyExternalMu vk m sig + | B.length sig /= signatureSize p = False + | otherwise = unsafeDoIO $ + withByteArray sig $ \psig -> + withByteArray m $ \pmu -> + withByteArray vk $ \pvk -> do + r <- + c_verify + p + psig + pmu + (fromIntegral muSize) + nullPtr + 0 + pvk + 1 + return (r == 0) + where + p = Proxy :: Proxy p +{-# NOINLINE verifyExternalMu #-} + +-- The external-mu entry points are the ordinary ones with the last argument +-- set: the representative goes in where the message would, there is no +-- domain separation prefix to prepend because it is already inside, and the +-- implementation is told so. +signMu + :: forall p rnd + . (MLDSA p, ByteArrayAccess rnd) + => SigningKey p -> Mu -> Maybe rnd -> CryptoFailable (Signature p) +signMu sk m mrnd = unsafeDoIO $ do + let zeroes = B.zero signingRandomnessSize :: ScrubbedBytes + withRnd f = case mrnd of + Just r -> withByteArray r f + Nothing -> withByteArray zeroes f + (r, sig) <- B.allocRet (signatureSize p) $ \psig -> + withByteArray m $ \pmu -> + withRnd $ \prnd -> + withByteArray sk $ \psk -> + c_sign p psig pmu (fromIntegral muSize) nullPtr 0 prnd psk 1 + return $ + if r == 0 + then CryptoPassed (Signature sig) + else CryptoFailed CryptoError_ParameterInvalid + where + p = Proxy :: Proxy p +{-# NOINLINE signMu #-} + +-- | Verify a signature. +-- +-- The context must be the one it was signed under; anything else is a +-- rejection, which is what the context is for. +verify + :: forall p msg + . (MLDSA p, ByteArrayAccess msg) + => VerificationKey p -> Context -> msg -> Signature p -> Bool +verify vk ctx msg sig + | B.length sig /= signatureSize p = False + | otherwise = unsafeDoIO $ + withByteArray sig $ \psig -> + withByteArray msg $ \pmsg -> + withByteArray pre $ \ppre -> + withByteArray vk $ \pvk -> do + r <- + c_verify + p + psig + pmsg + (fromIntegral (B.length msg)) + ppre + (fromIntegral (B.length pre)) + pvk + 0 + return (r == 0) + where + p = Proxy :: Proxy p + pre = domainPrefix ctx +{-# NOINLINE verify #-} + +-- | The domain separation prefix of FIPS 204 for pure ML-DSA, which is a +-- zero byte, the context's length and the context itself. It is built here +-- rather than taken from the implementation because it is three bytes of +-- concatenation and doing it here keeps one fewer foreign call. +domainPrefix :: Context -> Bytes +domainPrefix (Context ctx) = + B.concat [B.pack [0, fromIntegral (B.length ctx)] :: Bytes, B.convert ctx] + +instance MLDSA MLDSA44 where + verificationKeySize _ = 1312 + signingKeySize _ = 2560 + signatureSize _ = 2420 + c_keypair _ = c_mldsa44_keypair + c_sign _ = c_mldsa44_sign + c_verify _ = c_mldsa44_verify + c_pkFromSk _ = c_mldsa44_pk_from_sk + +instance MLDSA MLDSA65 where + verificationKeySize _ = 1952 + signingKeySize _ = 4032 + signatureSize _ = 3309 + c_keypair _ = c_mldsa65_keypair + c_sign _ = c_mldsa65_sign + c_verify _ = c_mldsa65_verify + c_pkFromSk _ = c_mldsa65_pk_from_sk + +instance MLDSA MLDSA87 where + verificationKeySize _ = 2592 + signingKeySize _ = 4896 + signatureSize _ = 4627 + c_keypair _ = c_mldsa87_keypair + c_sign _ = c_mldsa87_sign + c_verify _ = c_mldsa87_verify + c_pkFromSk _ = c_mldsa87_pk_from_sk + +foreign import ccall unsafe "crypton_mldsa44_keypair_internal" + c_mldsa44_keypair :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt +foreign import ccall unsafe "crypton_mldsa44_signature_internal" + c_mldsa44_sign + :: Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize + -> Ptr Word8 -> Ptr Word8 -> CInt -> IO CInt +foreign import ccall unsafe "crypton_mldsa44_verify_internal" + c_mldsa44_verify + :: Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize + -> Ptr Word8 -> CInt -> IO CInt +foreign import ccall unsafe "crypton_mldsa44_pk_from_sk" + c_mldsa44_pk_from_sk :: Ptr Word8 -> Ptr Word8 -> IO CInt + +foreign import ccall unsafe "crypton_mldsa65_keypair_internal" + c_mldsa65_keypair :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt +foreign import ccall unsafe "crypton_mldsa65_signature_internal" + c_mldsa65_sign + :: Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize + -> Ptr Word8 -> Ptr Word8 -> CInt -> IO CInt +foreign import ccall unsafe "crypton_mldsa65_verify_internal" + c_mldsa65_verify + :: Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize + -> Ptr Word8 -> CInt -> IO CInt +foreign import ccall unsafe "crypton_mldsa65_pk_from_sk" + c_mldsa65_pk_from_sk :: Ptr Word8 -> Ptr Word8 -> IO CInt + +foreign import ccall unsafe "crypton_mldsa87_keypair_internal" + c_mldsa87_keypair :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt +foreign import ccall unsafe "crypton_mldsa87_signature_internal" + c_mldsa87_sign + :: Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize + -> Ptr Word8 -> Ptr Word8 -> CInt -> IO CInt +foreign import ccall unsafe "crypton_mldsa87_verify_internal" + c_mldsa87_verify + :: Ptr Word8 -> Ptr Word8 -> CSize -> Ptr Word8 -> CSize + -> Ptr Word8 -> CInt -> IO CInt +foreign import ccall unsafe "crypton_mldsa87_pk_from_sk" + c_mldsa87_pk_from_sk :: Ptr Word8 -> Ptr Word8 -> IO CInt diff --git a/Crypto/PubKey/MLKEM.hs b/Crypto/PubKey/MLKEM.hs new file mode 100644 --- /dev/null +++ b/Crypto/PubKey/MLKEM.hs @@ -0,0 +1,448 @@ +-- | +-- Module : Crypto.PubKey.MLKEM +-- License : BSD-style +-- Maintainer : Kazu Yamamoto +-- Stability : experimental +-- Portability : unknown +-- +-- ML-KEM, the Module-Lattice-Based Key-Encapsulation Mechanism of +-- , in all three +-- parameter sets. +-- +-- A key encapsulation mechanism is not a Diffie-Hellman: there is no shared +-- secret to be computed from two key pairs. One side publishes an +-- 'EncapsulationKey'; the other calls 'encapsulate' on it, which draws a +-- fresh secret and returns it along with a 'Ciphertext' that only the holder +-- of the matching 'DecapsulationKey' can turn back into that secret. +-- +-- > (ek, dk) <- generateKeyPair MLKEM768 -- the receiver +-- > (ct, ss) <- encapsulate ek -- the sender +-- > let ss' = decapsulate dk ct -- the receiver, again +-- > ss == ss' +-- +-- What 'generateKeyPair' and 'encapsulate' draw their randomness from is +-- the 'Crypto.Random.MonadRandom' instance in use. Its documentation says +-- what an instance of your own has to be. +-- +-- The parameter set is a type, so an ML-KEM-768 key cannot be passed where +-- an ML-KEM-1024 one is expected. The three are fixed by FIPS 203 and the +-- class has no other instances. +{-# LANGUAGE GeneralizedNewtypeDeriving #-} +{-# LANGUAGE TypeFamilies #-} +{-# LANGUAGE TypeOperators #-} +{-# LANGUAGE ScopedTypeVariables #-} + +module Crypto.PubKey.MLKEM ( + -- * Parameter sets + MLKEM512 (..), + MLKEM768 (..), + MLKEM1024 (..), + MLKEM (encapsulationKeySize, decapsulationKeySize, ciphertextSize), + + -- * Keys, ciphertexts and shared secrets + -- + -- | These are the associated types of 'KEM', re-exported so that a + -- caller of this module alone has them. + KEM (..), + SharedSecret (..), + + -- * Smart constructors + encapsulationKey, + decapsulationKey, + ciphertext, + + -- * What ML-KEM has beyond the class + generateKeyPairAndSeed, + keyPairFromSeed, + + -- * Sizes + seedSize, + encapsulationCoinsSize, + sharedSecretSize, +) where + +import Data.Proxy (Proxy (..)) +import Foreign.C.Types (CInt (..)) +import Foreign.Ptr (Ptr) + +import Crypto.Debug (DebugShow (..), debugShowBytes) +import Crypto.Error +import Crypto.KEM +import Crypto.Internal.ByteArray ( + ByteArrayAccess, + Bytes, + ScrubbedBytes, + withByteArray, + ) +import qualified Crypto.Internal.ByteArray as B +import Crypto.Internal.Compat (unsafeDoIO) +import Crypto.Internal.Imports +import Crypto.Random (MonadRandom, getRandomBytes) + +-- | ML-KEM-512. +data MLKEM512 = MLKEM512 deriving (Show, Eq) + +-- | ML-KEM-768. This is the set TLS uses, on its own and as the +-- lattice half of the hybrid groups. +data MLKEM768 = MLKEM768 deriving (Show, Eq) + +-- | ML-KEM-1024. +data MLKEM1024 = MLKEM1024 deriving (Show, Eq) + +-- | The three parameter sets of FIPS 203. +-- +-- This is not an abstract KEM interface and does not try to be: it is the +-- three sets FIPS 203 defines, closed, carrying their sizes and the calls +-- into the implementation. Only the sizes are exported. If crypton grows +-- a second KEM and an interface common to both is wanted, that belongs in +-- a module of its own, with this as one of its instances. +class + ( KEM p + , EncapsulationKey p ~ MLKEMEncapsulationKey p + , DecapsulationKey p ~ MLKEMDecapsulationKey p + , Ciphertext p ~ MLKEMCiphertext p + , Coins p ~ ScrubbedBytes + ) => + MLKEM p + where + -- | Size in bytes of an 'EncapsulationKey' of this parameter set. + encapsulationKeySize :: proxy p -> Int + + -- | Size in bytes of a 'DecapsulationKey' of this parameter set. + decapsulationKeySize :: proxy p -> Int + + -- | Size in bytes of a 'Ciphertext' of this parameter set. + ciphertextSize :: proxy p -> Int + + c_keypair :: proxy p -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt + c_enc :: proxy p -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt + c_dec :: proxy p -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt + c_checkPk :: proxy p -> Ptr Word8 -> IO CInt + c_checkSk :: proxy p -> Ptr Word8 -> IO CInt + +-- | A public encapsulation key, @ek@ in FIPS 203. +newtype MLKEMEncapsulationKey p = MLKEMEncapsulationKey Bytes + deriving (Show, Eq, ByteArrayAccess, NFData) + +-- | A private decapsulation key, @dk@ in FIPS 203. It embeds the matching +-- encapsulation key, which is why it is the larger of the two. +newtype MLKEMDecapsulationKey p = MLKEMDecapsulationKey ScrubbedBytes + deriving (Eq, ByteArrayAccess, NFData) + +instance Show (MLKEMDecapsulationKey p) where + show _ = "DecapsulationKey " + +instance DebugShow (MLKEMDecapsulationKey p) where + debugShow = debugShowBytes "DecapsulationKey" + +-- | The value 'encapsulate' produces and 'decapsulate' consumes. +newtype MLKEMCiphertext p = MLKEMCiphertext Bytes + deriving (Show, Eq, ByteArrayAccess, NFData) + +-- | Size in bytes of the seed 'keyPairFromSeed' takes, which is @d@ and @z@ +-- of FIPS 203 one after the other. +seedSize :: Int +seedSize = 64 + +-- | Size in bytes of the randomness 'encapsulateWith' takes, @m@ in +-- FIPS 203. +encapsulationCoinsSize :: Int +encapsulationCoinsSize = 32 + +-- | Size in bytes of a 'SharedSecret'. +sharedSecretSize :: Int +sharedSecretSize = 32 + +-- | Try to read an encapsulation key. +-- +-- Beyond the length this runs the check of FIPS 203 section 7.2: the key +-- must be the encoding of coefficients that are all in range, which is to +-- say it must survive a decode and re-encode unchanged. A key that fails +-- it is not one any honest party produced. +encapsulationKey + :: forall p ba + . (MLKEM p, ByteArrayAccess ba) + => ba -> CryptoFailable (EncapsulationKey p) +encapsulationKey bs + | B.length bs /= encapsulationKeySize p = CryptoFailed CryptoError_PublicKeySizeInvalid + | otherwise = unsafeDoIO $ withByteArray bs $ \inp -> do + r <- c_checkPk p inp + return $ + if r == 0 + then CryptoPassed $ MLKEMEncapsulationKey $ B.copyAndFreeze bs (\_ -> return ()) + else CryptoFailed CryptoError_PublicKeyStructureInvalid + where + p = Proxy :: Proxy p +{-# NOINLINE encapsulationKey #-} + +-- | Try to read a decapsulation key. +-- +-- Beyond the length this runs the check of FIPS 203 section 7.3: the hash +-- of the encapsulation key the private key embeds must match the copy of +-- that hash it also embeds. The two disagreeing means the key was not +-- produced as a pair, and decapsulating with it would silently answer with +-- the implicit rejection every time. +decapsulationKey + :: forall p ba + . (MLKEM p, ByteArrayAccess ba) + => ba -> CryptoFailable (DecapsulationKey p) +decapsulationKey bs + | B.length bs /= decapsulationKeySize p = CryptoFailed CryptoError_SecretKeySizeInvalid + | otherwise = unsafeDoIO $ withByteArray bs $ \inp -> do + r <- c_checkSk p inp + return $ + if r == 0 + then CryptoPassed $ MLKEMDecapsulationKey $ B.copyAndFreeze bs (\_ -> return ()) + else CryptoFailed CryptoError_SecretKeyStructureInvalid + where + p = Proxy :: Proxy p +{-# NOINLINE decapsulationKey #-} + +-- | Try to read a ciphertext. Only the length is checked; every string of +-- the right length is a ciphertext that 'decapsulate' will answer. +ciphertext + :: forall p ba + . (MLKEM p, ByteArrayAccess ba) + => ba -> CryptoFailable (Ciphertext p) +ciphertext bs + | B.length bs == ciphertextSize (Proxy :: Proxy p) = + CryptoPassed $ MLKEMCiphertext $ B.copyAndFreeze bs (\_ -> return ()) + | otherwise = CryptoFailed CryptoError_PointSizeInvalid + +-- | Generate a key pair. +-- +-- The seed it is derived from is drawn here and thrown away. Use +-- 'generateKeyPairAndSeed' where it has to be kept. +mlkemGenerateKeyPair + :: forall p proxy m + . (MLKEM p, MonadRandom m) + => proxy p -> m (MLKEMEncapsulationKey p, DecapsulationKey p) +mlkemGenerateKeyPair p = do + (ek, dk, _) <- generateKeyPairAndSeed p + return (ek, dk) + +-- | Generate a key pair and hand back the seed it was derived from, @d@ +-- and @z@ of FIPS 203 one after the other. +-- +-- A 'DecapsulationKey' is the expanded key and nothing else, so the seed +-- cannot be recovered from a pair afterwards. An application that has to +-- write the key out in a form that keeps the seed has to generate it here: +-- +-- > (ek, dk, seed) <- generateKeyPairAndSeed MLKEM768 +-- +-- The seed is as secret as the decapsulation key: 'keyPairFromSeed' turns +-- it back into the same pair. +generateKeyPairAndSeed + :: forall p proxy m + . (MLKEM p, MonadRandom m) + => proxy p + -> m (MLKEMEncapsulationKey p, DecapsulationKey p, ScrubbedBytes) +generateKeyPairAndSeed p = do + seed <- getRandomBytes seedSize :: m ScrubbedBytes + case keyPairFromSeed p seed of + CryptoPassed (ek, dk) -> return (ek, dk, seed) + CryptoFailed e -> + error ("Crypto.PubKey.MLKEM.generateKeyPairAndSeed: " ++ show e) + +-- | Derive a key pair from a seed, which is @d@ and @z@ of FIPS 203 one +-- after the other and must be 'seedSize' bytes. +-- +-- This is the entry point to use when the seed comes from somewhere +-- particular -- a test vector, or a store that keeps seeds rather than +-- expanded keys. For an ordinary key, 'generateKeyPair' draws the seed +-- itself. +keyPairFromSeed + :: forall p proxy ba + . (MLKEM p, ByteArrayAccess ba) + => proxy p + -> ba + -> CryptoFailable (MLKEMEncapsulationKey p, DecapsulationKey p) +keyPairFromSeed p seed + | B.length seed /= seedSize = CryptoFailed CryptoError_SeedSizeInvalid + | otherwise = unsafeDoIO $ do + -- Not zeroed, and does not need to be: the C writes the whole + -- buffer, and on a non-zero return the result is discarded without + -- being read. Anything that is *read* before being written has to + -- use B.zero instead -- see signInternal in Crypto.PubKey.MLDSA. + dk <- B.alloc (decapsulationKeySize p) (\_ -> return ()) :: IO ScrubbedBytes + (r, ek) <- B.allocRet (encapsulationKeySize p) $ \pek -> + withByteArray dk $ \pdk -> + withByteArray seed $ \pseed -> + c_keypair p pek pdk pseed + return $ + if r == 0 + then CryptoPassed (MLKEMEncapsulationKey ek, MLKEMDecapsulationKey dk) + else CryptoFailed CryptoError_ParameterInvalid +{-# NOINLINE keyPairFromSeed #-} + +-- | Encapsulate against a public key, drawing the randomness. +mlkemEncapsulate + :: forall p m + . (MLKEM p, MonadRandom m) + => MLKEMEncapsulationKey p + -> m (CryptoFailable (Ciphertext p, SharedSecret)) +mlkemEncapsulate ek = do + coins <- getRandomBytes encapsulationCoinsSize :: m ScrubbedBytes + return (mlkemEncapsulateWith ek coins) + +-- The class's 'encapsulateWith' for ML-KEM, where the coins are @m@ of +-- FIPS 203 and must be 'encapsulationCoinsSize' bytes. +mlkemEncapsulateWith + :: forall p + . MLKEM p + => MLKEMEncapsulationKey p + -> ScrubbedBytes + -> CryptoFailable (Ciphertext p, SharedSecret) +mlkemEncapsulateWith ek coins + | B.length coins /= encapsulationCoinsSize = CryptoFailed CryptoError_SeedSizeInvalid + | otherwise = unsafeDoIO $ do + ss <- B.alloc sharedSecretSize (\_ -> return ()) :: IO ScrubbedBytes + (r, ct) <- B.allocRet (ciphertextSize p) $ \pct -> + withByteArray ss $ \pss -> + withByteArray ek $ \pek -> + withByteArray coins $ \pcoins -> + c_enc p pct pss pek pcoins + return $ + if r == 0 + then CryptoPassed (MLKEMCiphertext ct, SharedSecret ss) + else CryptoFailed CryptoError_ParameterInvalid + where + p = Proxy :: Proxy p +{-# NOINLINE mlkemEncapsulateWith #-} + +-- | Recover the shared secret from a ciphertext. +-- +-- A ciphertext that was not produced by encapsulating against the matching +-- key is not an error. ML-KEM rejects implicitly: it yields a secret +-- derived from the private key and the ciphertext, and the caller cannot +-- tell that case from the other one, which is the point -- telling them +-- apart is what a chosen-ciphertext attack needs. A ciphertext that does +-- not belong here shows up later, as the two sides failing to agree on +-- anything. +-- +-- The checks FIPS 203 does require are at the point where bytes become a +-- value of these types, which is where they can be reported: +-- +-- * The ciphertext type check of section 7.3 is its length, and +-- 'ciphertext' is the only way to build a 'Ciphertext' from bytes. There +-- is nothing else to check: a ciphertext's coefficients are compressed to +-- fewer than twelve bits, so every bit pattern decodes to a value in +-- range. +-- * The hash check of section 7.3 is on the decapsulation key, and +-- 'decapsulationKey' runs it; a key from 'generateKeyPair' or +-- 'keyPairFromSeed' satisfies it by construction. +-- +-- So the result is 'CryptoPassed' for every key and ciphertext this module +-- can produce. It is 'CryptoFailable' rather than a bare 'SharedSecret' +-- because the implementation checks the key again on its way through, and +-- what it finds is better reported than turned into an exception. +mlkemDecapsulate + :: forall p + . MLKEM p + => DecapsulationKey p -> Ciphertext p -> CryptoFailable SharedSecret +mlkemDecapsulate dk ct = unsafeDoIO $ do + (r, ss) <- B.allocRet sharedSecretSize $ \pss -> + withByteArray ct $ \pct -> + withByteArray dk $ \pdk -> + c_dec (Proxy :: Proxy p) pss pct pdk + return $ + if r == (0 :: CInt) + then CryptoPassed (SharedSecret ss) + else CryptoFailed CryptoError_SecretKeyStructureInvalid +{-# NOINLINE mlkemDecapsulate #-} + +-- The class's view of the three sets. The operations are the ones above; +-- only the shape of the arguments differs, because the class takes the +-- mechanism as a proxy. +instance KEM MLKEM512 where + type EncapsulationKey MLKEM512 = MLKEMEncapsulationKey MLKEM512 + type DecapsulationKey MLKEM512 = MLKEMDecapsulationKey MLKEM512 + type Ciphertext MLKEM512 = MLKEMCiphertext MLKEM512 + type Coins MLKEM512 = ScrubbedBytes + generateKeyPair = mlkemGenerateKeyPair + encapsulate _ = mlkemEncapsulate + encapsulateWith _ = mlkemEncapsulateWith + decapsulate _ = mlkemDecapsulate + +instance KEM MLKEM768 where + type EncapsulationKey MLKEM768 = MLKEMEncapsulationKey MLKEM768 + type DecapsulationKey MLKEM768 = MLKEMDecapsulationKey MLKEM768 + type Ciphertext MLKEM768 = MLKEMCiphertext MLKEM768 + type Coins MLKEM768 = ScrubbedBytes + generateKeyPair = mlkemGenerateKeyPair + encapsulate _ = mlkemEncapsulate + encapsulateWith _ = mlkemEncapsulateWith + decapsulate _ = mlkemDecapsulate + +instance KEM MLKEM1024 where + type EncapsulationKey MLKEM1024 = MLKEMEncapsulationKey MLKEM1024 + type DecapsulationKey MLKEM1024 = MLKEMDecapsulationKey MLKEM1024 + type Ciphertext MLKEM1024 = MLKEMCiphertext MLKEM1024 + type Coins MLKEM1024 = ScrubbedBytes + generateKeyPair = mlkemGenerateKeyPair + encapsulate _ = mlkemEncapsulate + encapsulateWith _ = mlkemEncapsulateWith + decapsulate _ = mlkemDecapsulate + +instance MLKEM MLKEM512 where + encapsulationKeySize _ = 800 + decapsulationKeySize _ = 1632 + ciphertextSize _ = 768 + c_keypair _ = c_mlkem512_keypair + c_enc _ = c_mlkem512_enc + c_dec _ = c_mlkem512_dec + c_checkPk _ = c_mlkem512_check_pk + c_checkSk _ = c_mlkem512_check_sk + +instance MLKEM MLKEM768 where + encapsulationKeySize _ = 1184 + decapsulationKeySize _ = 2400 + ciphertextSize _ = 1088 + c_keypair _ = c_mlkem768_keypair + c_enc _ = c_mlkem768_enc + c_dec _ = c_mlkem768_dec + c_checkPk _ = c_mlkem768_check_pk + c_checkSk _ = c_mlkem768_check_sk + +instance MLKEM MLKEM1024 where + encapsulationKeySize _ = 1568 + decapsulationKeySize _ = 3168 + ciphertextSize _ = 1568 + c_keypair _ = c_mlkem1024_keypair + c_enc _ = c_mlkem1024_enc + c_dec _ = c_mlkem1024_dec + c_checkPk _ = c_mlkem1024_check_pk + c_checkSk _ = c_mlkem1024_check_sk + +foreign import ccall unsafe "crypton_mlkem512_keypair_derand" + c_mlkem512_keypair :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt +foreign import ccall unsafe "crypton_mlkem512_enc_derand" + c_mlkem512_enc :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt +foreign import ccall unsafe "crypton_mlkem512_dec" + c_mlkem512_dec :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt +foreign import ccall unsafe "crypton_mlkem512_check_pk" + c_mlkem512_check_pk :: Ptr Word8 -> IO CInt +foreign import ccall unsafe "crypton_mlkem512_check_sk" + c_mlkem512_check_sk :: Ptr Word8 -> IO CInt + +foreign import ccall unsafe "crypton_mlkem768_keypair_derand" + c_mlkem768_keypair :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt +foreign import ccall unsafe "crypton_mlkem768_enc_derand" + c_mlkem768_enc :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt +foreign import ccall unsafe "crypton_mlkem768_dec" + c_mlkem768_dec :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt +foreign import ccall unsafe "crypton_mlkem768_check_pk" + c_mlkem768_check_pk :: Ptr Word8 -> IO CInt +foreign import ccall unsafe "crypton_mlkem768_check_sk" + c_mlkem768_check_sk :: Ptr Word8 -> IO CInt + +foreign import ccall unsafe "crypton_mlkem1024_keypair_derand" + c_mlkem1024_keypair :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt +foreign import ccall unsafe "crypton_mlkem1024_enc_derand" + c_mlkem1024_enc :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt +foreign import ccall unsafe "crypton_mlkem1024_dec" + c_mlkem1024_dec :: Ptr Word8 -> Ptr Word8 -> Ptr Word8 -> IO CInt +foreign import ccall unsafe "crypton_mlkem1024_check_pk" + c_mlkem1024_check_pk :: Ptr Word8 -> IO CInt +foreign import ccall unsafe "crypton_mlkem1024_check_sk" + c_mlkem1024_check_sk :: Ptr Word8 -> IO CInt diff --git a/Crypto/PubKey/MaskGenFunction.hs b/Crypto/PubKey/MaskGenFunction.hs --- a/Crypto/PubKey/MaskGenFunction.hs +++ b/Crypto/PubKey/MaskGenFunction.hs @@ -1,4 +1,4 @@ -{-# LANGUAGE BangPatterns #-} +{-# LANGUAGE ScopedTypeVariables #-} -- | -- Module : Crypto.PubKey.MaskGenFunction @@ -11,10 +11,17 @@ mgf1, ) where -import Crypto.Hash -import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess, Bytes) +import Control.Monad (forM_) +import Crypto.Hash (Context, Digest) +import Crypto.Hash.IO (HashAlgorithm (..)) +import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess) import qualified Crypto.Internal.ByteArray as B -import Crypto.Number.Serialize (i2ospOf_) +import Data.Bits (shiftR) +import Data.Word (Word8) +import Foreign.Marshal.Alloc (allocaBytes) +import Foreign.Marshal.Utils (copyBytes) +import Foreign.Ptr (Ptr, castPtr, plusPtr) +import Foreign.Storable (pokeByteOff) -- | Represent a mask generation algorithm type MaskGenAlgorithm seed output = @@ -26,20 +33,33 @@ -- | Mask generation algorithm MGF1 mgf1 - :: (ByteArrayAccess seed, ByteArray output, HashAlgorithm hashAlg) + :: forall seed output hashAlg + . (ByteArrayAccess seed, ByteArray output, HashAlgorithm hashAlg) => hashAlg -> seed -> Int -> output -mgf1 hashAlg seed len = - let !seededCtx = hashUpdate (hashInitWith hashAlg) seed - in B.take len $ - B.concat $ - map (hashCounter seededCtx) [0 .. fromIntegral (maxCounter - 1)] +mgf1 hashAlg seed len + | len <= 0 = B.empty + | otherwise = B.allocAndFreeze len $ \out -> + -- the seeded context is kept, and copied for each block rather than + -- rebuilt; each block's digest is written straight into the output + allocaBytes ctxSize $ \(seeded :: Ptr (Context hashAlg)) -> + allocaBytes ctxSize $ \(ctx :: Ptr (Context hashAlg)) -> + allocaBytes digestLen $ \(dig :: Ptr (Digest hashAlg)) -> + allocaBytes 4 $ \(cnt :: Ptr Word8) -> do + hashInternalInit seeded + B.withByteArray seed $ \p -> + hashInternalUpdate seeded p (fromIntegral (B.length seed)) + forM_ [0 .. maxCounter - 1] $ \i -> do + copyBytes ctx seeded ctxSize + forM_ [0 .. 3] $ \k -> + pokeByteOff cnt k (fromIntegral (i `shiftR` (8 * (3 - k))) :: Word8) + hashInternalUpdate ctx cnt 4 + hashInternalFinalize ctx dig + let off = i * digestLen + copyBytes (out `plusPtr` off) (castPtr dig) (min digestLen (len - off)) where + ctxSize = hashInternalContextSize hashAlg digestLen = hashDigestSize hashAlg - (chunks, left) = len `divMod` digestLen - maxCounter = if left > 0 then chunks + 1 else chunks - - hashCounter :: HashAlgorithm a => Context a -> Integer -> Digest a - hashCounter ctx counter = hashFinalize $ hashUpdate ctx (i2ospOf_ 4 counter :: Bytes) + maxCounter = (len + digestLen - 1) `div` digestLen diff --git a/Crypto/PubKey/RSA/Blinded.hs b/Crypto/PubKey/RSA/Blinded.hs new file mode 100644 --- /dev/null +++ b/Crypto/PubKey/RSA/Blinded.hs @@ -0,0 +1,29 @@ +-- | +-- Module : Crypto.PubKey.RSA.Blinded +-- License : BSD-style +-- Stability : experimental +-- Portability : Good +-- +-- An RSA private key that keeps its blinding between operations. +-- +-- Blinding hides the private-key operation's input from timing: it is +-- multiplied by @r^e@ for a random @r@ first, and the result by @r^-1@ +-- after. Drawing a new @r@ costs a modular inverse and an exponentiation, +-- about a tenth of an RSA-2048 signature. A t'BlindedKey' draws one and +-- squares the pair for each use after it, which is what OpenSSL does, and +-- draws a new one every 32 uses, from the 'Crypto.Random.Types.MonadRandom' +-- the operation runs in. +-- +-- The key is threaded through the operations as a hash context is: each +-- function in "Crypto.PubKey.RSA.PKCS15", "Crypto.PubKey.RSA.PSS" and +-- "Crypto.PubKey.RSA.OAEP" whose name ends in @Blinded@ returns the key for +-- the next one. The @Safer@ functions there keep a t'BlindedKey' per key +-- themselves, for the last few keys used, so they get the same saving +-- without one being passed. +module Crypto.PubKey.RSA.Blinded ( + BlindedKey, + blindedKey, + blindedPrivateKey, +) where + +import Crypto.PubKey.RSA.Blinded.Internal diff --git a/Crypto/PubKey/RSA/Blinded/Internal.hs b/Crypto/PubKey/RSA/Blinded/Internal.hs new file mode 100644 --- /dev/null +++ b/Crypto/PubKey/RSA/Blinded/Internal.hs @@ -0,0 +1,136 @@ +{-# LANGUAGE BangPatterns #-} +{-# LANGUAGE ScopedTypeVariables #-} +{-# OPTIONS_GHC -fno-cse -fno-full-laziness #-} + +-- | +-- Module : Crypto.PubKey.RSA.Blinded.Internal +-- License : BSD-style +-- Stability : experimental +-- Portability : Good +-- +-- A private key with a blinding pair that is reused by squaring, as OpenSSL +-- does, rather than generated afresh for every operation. The key is a value +-- threaded through the operations, as a hash 'Crypto.Hash.Context' is. +module Crypto.PubKey.RSA.Blinded.Internal ( + BlindedKey (..), + blindedKey, + dpBlinded, + Draw, + drawFor, + withBlindingFor, + dpWithPair, +) where + +import Data.IORef +import System.IO.Unsafe (unsafePerformIO) + +import Crypto.Internal.ByteArray (ByteArray, ScrubbedBytes) +import Crypto.Number.Generate (generateMax) +import Crypto.Number.ModArithmetic (expFast, inverseCoprimes) +import Crypto.Number.Serialize (i2ospOf_, os2ip) +import Crypto.PubKey.RSA.Prim (dp) +import Crypto.PubKey.RSA.Types +import Crypto.Random.Types + +-- | A private key together with the blinding its next operation uses. +-- +-- Each operation multiplies the input by @r^e@ and the output by @r^-1@, and +-- returns the key with both squared for the next one, so a pair costs two +-- multiplications after the first. A new @r@ is drawn every 32 operations. +-- Use the key an operation returns: handing the same one to two operations +-- gives both the same pair. +data BlindedKey = BlindedKey + { blindedPrivateKey :: PrivateKey + -- ^ the key it was made from + , blindedPair :: Pair + } + +-- | @r^e@ and @r^-1@ modulo n, and how many more uses they have after this. +data Pair = Pair !Int !Integer !Integer | Spent + +-- | Uses of one @r@ before another is drawn, OpenSSL's number. +refresh :: Int +refresh = 32 + +-- | A t'BlindedKey' for a private key. The first pair is drawn when it is +-- first used. +blindedKey :: PrivateKey -> BlindedKey +blindedKey pk = BlindedKey pk Spent + +-- | The random numbers a new pair is made from: drawing them is cheap, and +-- making the pair, an inverse and an exponentiation, is done only if needed. +data Draw = Draw !Integer !Integer + +-- | Draw for a key, as 'Crypto.PubKey.RSA.generateBlinder' does. +drawFor :: forall m. MonadRandom m => PrivateKey -> m Draw +drawFor pk = do + r <- generateMax (private_n pk) + u <- os2ip <$> (getRandomBytes 16 :: m ScrubbedBytes) + return (Draw r u) + +-- | @r^e@ and @r^-1@. The inverse is taken of @r*u@ and multiplied by @u@, +-- so the extended Euclidean algorithm never sees @r@, as in +-- 'Crypto.PubKey.RSA.generateBlinder'. +makePair :: PrivateKey -> Draw -> (Integer, Integer) +makePair pk (Draw r u) = + (expFast r (private_e pk) n, (inverseCoprimes ((r * u) `mod` n) n * u) `mod` n) + where + n = private_n pk + +-- | The pair for this operation, and the key holding the next one. The +-- draw is used only when a new pair is due. +takePair :: Draw -> BlindedKey -> ((Integer, Integer), BlindedKey) +takePair d (BlindedKey pk pair) = case pair of + Pair k a ai -> ((a, ai), BlindedKey pk (next k a ai)) + Spent -> case makePair pk d of + (a, ai) -> ((a, ai), BlindedKey pk (next refresh a ai)) + where + n = private_n pk + next k a ai + | k > 1 = Pair (k - 1) ((a * a) `mod` n) ((ai * ai) `mod` n) + | otherwise = Spent + +-- | The decryption primitive, blinded with a given pair. +dpWithPair :: ByteArray ba => (Integer, Integer) -> PrivateKey -> ba -> ba +dpWithPair (a, ai) pk c = i2ospOf_ k ((ai * os2ip m) `mod` n) + where + n = private_n pk + k = private_size pk + m = dp Nothing pk (i2ospOf_ k ((a * os2ip c) `mod` n) `asTypeOf` c) + +-- | The decryption primitive, blinded with the key's pair, and the key for +-- the next operation. Randomness is drawn only when a new pair is due. +dpBlinded + :: (MonadRandom m, ByteArray ba) => BlindedKey -> ba -> m (ba, BlindedKey) +dpBlinded bk c = do + d <- case blindedPair bk of + Pair {} -> return (Draw 0 0) + Spent -> drawFor (blindedPrivateKey bk) + let (pair, bk') = takePair d bk + return (dpWithPair pair (blindedPrivateKey bk) c, bk') + +-- | Run an operation with the pair kept for a key, for the functions that +-- take a bare t'PrivateKey'. One t'BlindedKey' is kept per public key, for +-- the last few keys used; taking its pair and leaving the next is atomic, so +-- no two operations get the same pair. The caller draws every time, since +-- whether a new pair is due is known only in here. +withBlindingFor :: PrivateKey -> Draw -> ((Integer, Integer) -> a) -> a +withBlindingFor pk d f = unsafePerformIO $ do + pair <- atomicModifyIORef' table step + return (f pair) + where + pub = private_pub pk + step ks = case break ((== pub) . private_pub . blindedPrivateKey) ks of + (before, bk : after) -> found bk (before ++ after) + _ -> found (blindedKey pk) ks + found bk rest = case takePair d bk of + (pair@(!_, !_), bk') -> (take tableSize (bk' : rest), pair) +{-# NOINLINE withBlindingFor #-} + +-- | How many keys 'withBlindingFor' remembers. +tableSize :: Int +tableSize = 4 + +table :: IORef [BlindedKey] +table = unsafePerformIO (newIORef []) +{-# NOINLINE table #-} diff --git a/Crypto/PubKey/RSA/OAEP.hs b/Crypto/PubKey/RSA/OAEP.hs --- a/Crypto/PubKey/RSA/OAEP.hs +++ b/Crypto/PubKey/RSA/OAEP.hs @@ -18,22 +18,22 @@ -- * OAEP decryption decrypt, decryptSafer, + decryptBlinded, ) where import Crypto.Hash import Crypto.Number.Serialize (os2ip) import Crypto.PubKey.Internal (and') import Crypto.PubKey.MaskGenFunction -import Crypto.PubKey.RSA (generateBlinder) +import Crypto.PubKey.RSA.Blinded.Internal import Crypto.PubKey.RSA.Prim import Crypto.PubKey.RSA.Types import Crypto.Random.Types import Data.Bits (complement, shiftR, xor, (.&.), (.|.)) import Data.ByteString (ByteString) import qualified Data.ByteString as B -import Data.List (foldl') +import qualified Data.List as L import Data.Word (Word32) -import Prelude hiding (foldl') import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess) import qualified Crypto.Internal.ByteArray as B (constEq, convert) @@ -156,7 +156,7 @@ -- is zero; all of them are looked at either way oneIndex = fst $ - foldl' + L.foldl' step (fromIntegral (B.length db1) :: Word32, 1 :: Word32) (zip [0 ..] (B.unpack db1)) @@ -181,10 +181,9 @@ -- | Decrypt a ciphertext using OAEP -- --- When the signature is not in a context where an attacker could gain --- information from the timing of the operation, the blinder can be set to None. --- --- If unsure always set a blinder or use decryptSafer +-- The blinder is optional and 'Nothing' is accepted, but see t'Blinder' for +-- what it covers and when leaving it out is a decision rather than a default. +-- 'decryptSafer' generates one for you. -- -- Following RFC 8017, the ciphertext is rejected unless it is exactly as long -- as the modulus (section 7.1.2, step 1) and its integer representative is @@ -214,7 +213,8 @@ k = private_size pk hashLen = hashDigestSize (oaepHash oaep) --- | Decrypt a ciphertext using OAEP and by automatically generating a blinder. +-- | Decrypt a ciphertext using OAEP, blinded with the pair kept for this key; +-- see "Crypto.PubKey.RSA.Blinded". decryptSafer :: (HashAlgorithm hash, MonadRandom m) => OAEPParams hash ByteString ByteString @@ -224,6 +224,34 @@ -> ByteString -- ^ Cipher text -> m (Either Error ByteString) -decryptSafer oaep pk cipher = do - blinder <- generateBlinder (private_n pk) - return (decrypt (Just blinder) oaep pk cipher) +decryptSafer oaep pk cipher + | B.length cipher /= k = return (Left MessageSizeIncorrect) + | os2ip cipher >= private_n pk = return (Left MessageSizeIncorrect) + | k < 2 * hashLen + 2 = return (Left InvalidParameters) + | otherwise = do + d <- drawFor pk + return $ withBlindingFor pk d $ \pair -> unpad oaep k (dpWithPair pair pk cipher) + where + k = private_size pk + hashLen = hashDigestSize (oaepHash oaep) + +-- | 'decrypt' with the blinding a t'BlindedKey' keeps, and the key for the +-- next operation. +decryptBlinded + :: (HashAlgorithm hash, MonadRandom m) + => OAEPParams hash ByteString ByteString + -> BlindedKey + -> ByteString + -- ^ Cipher text + -> m (Either Error ByteString, BlindedKey) +decryptBlinded oaep bk cipher + | B.length cipher /= k = return (Left MessageSizeIncorrect, bk) + | os2ip cipher >= private_n pk = return (Left MessageSizeIncorrect, bk) + | k < 2 * hashLen + 2 = return (Left InvalidParameters, bk) + | otherwise = do + (m, bk') <- dpBlinded bk cipher + return (unpad oaep k m, bk') + where + pk = blindedPrivateKey bk + k = private_size pk + hashLen = hashDigestSize (oaepHash oaep) diff --git a/Crypto/PubKey/RSA/PKCS15.hs b/Crypto/PubKey/RSA/PKCS15.hs --- a/Crypto/PubKey/RSA/PKCS15.hs +++ b/Crypto/PubKey/RSA/PKCS15.hs @@ -15,10 +15,21 @@ decryptSafer, sign, signSafer, + signDigest, + signSaferDigest, + signDigestInfo, + signSaferDigestInfo, + -- * Private key operations with a kept blinding + decryptBlinded, + signDigestBlinded, + signDigestInfoBlinded, + -- * Public key operations encrypt, verify, + verifyDigest, + verifyDigestInfo, -- * Hash ASN1 description HashAlgorithmASN1, @@ -27,7 +38,7 @@ import Crypto.Hash import Crypto.Number.Serialize (os2ip) import Crypto.PubKey.Internal (and') -import Crypto.PubKey.RSA (generateBlinder) +import Crypto.PubKey.RSA.Blinded.Internal import Crypto.PubKey.RSA.Prim import Crypto.PubKey.RSA.Types import Crypto.Random.Types @@ -38,8 +49,7 @@ import Crypto.Internal.ByteArray (ByteArray, Bytes) import qualified Crypto.Internal.ByteArray as B -import Data.List (foldl') -import Prelude hiding (foldl') +import qualified Data.List as L -- | A specialized class for hash algorithm that can product -- a ASN1 wrapped description the algorithm plus the content @@ -405,10 +415,10 @@ :: ByteArray signature => Int -> signature -> Either Error signature padSignature klen signature | klen < siglen + 11 = Left SignatureTooLong - | otherwise = Right (B.pack padding `B.append` signature) + | otherwise = + Right (B.concat [B.pack [0, 1], B.replicate (klen - siglen - 3) 0xff, B.pack [0], signature]) where siglen = B.length signature - padding = 0 : 1 : (replicate (klen - siglen - 3) 0xff ++ [0]) -- | Try to remove a standard PKCS1.5 encryption padding. -- @@ -431,7 +441,7 @@ -- index of the first zero octet in ps0m, counted from the start of packed, -- or len when there is none; every octet is looked at either way - zeroIndex = fst $ foldl' step (fromIntegral len :: Word32, 1 :: Word32) indexed + zeroIndex = fst $ L.foldl' step (fromIntegral len :: Word32, 1 :: Word32) indexed indexed = zip [2 ..] (B.unpack ps0m) step (idx, unseen) (i, b) = (select found i idx, unseen .&. complement found) where @@ -453,10 +463,9 @@ -- | decrypt message using the private key. -- --- When the decryption is not in a context where an attacker could gain --- information from the timing of the operation, the blinder can be set to None. --- --- If unsure always set a blinder or use decryptSafer +-- The blinder is optional and 'Nothing' is accepted, but see t'Blinder' for +-- what it covers and when leaving it out is a decision rather than a default. +-- 'decryptSafer' generates one for you. -- -- The message is returned un-padded. -- @@ -482,7 +491,8 @@ -- "convert" must be apply to "c". | otherwise = unpad $ dp blinder pk $ B.convert c --- | decrypt message using the private key and by automatically generating a blinder. +-- | decrypt message using the private key, blinded with the pair kept for +-- this key; see "Crypto.PubKey.RSA.Blinded". decryptSafer :: (MonadRandom m, ByteArray ba) => PrivateKey @@ -490,10 +500,30 @@ -> ByteString -- ^ cipher text -> m (Either Error ba) -decryptSafer pk b = do - blinder <- generateBlinder (private_n pk) - return (decrypt (Just blinder) pk b) +decryptSafer pk b + | B.length b /= private_size pk = return (Left MessageSizeIncorrect) + | os2ip b >= private_n pk = return (Left MessageSizeIncorrect) + | otherwise = do + d <- drawFor pk + return $ withBlindingFor pk d $ \pair -> unpad (dpWithPair pair pk (B.convert b)) +-- | 'decrypt' with the blinding a t'BlindedKey' keeps, and the key for the +-- next operation. +decryptBlinded + :: (MonadRandom m, ByteArray ba) + => BlindedKey + -> ByteString + -- ^ cipher text + -> m (Either Error ba, BlindedKey) +decryptBlinded bk c + | B.length c /= private_size pk = return (Left MessageSizeIncorrect, bk) + | os2ip c >= private_n pk = return (Left MessageSizeIncorrect, bk) + | otherwise = do + (m, bk') <- dpBlinded bk (B.convert c) + return (unpad m, bk') + where + pk = blindedPrivateKey bk + -- | encrypt a bytestring using the public key. -- -- The message needs to be smaller than the key size - 11. @@ -508,10 +538,15 @@ -- | sign message using private key, a hash and its ASN1 description -- --- When the signature is not in a context where an attacker could gain --- information from the timing of the operation, the blinder can be set to None. +-- __Deprecated.__ Use 'signDigest', or 'signDigestInfo' where this would +-- have been given 'Nothing'. The @Maybe hashAlg@ argument says both \"hash +-- it with this\" and \"do not hash it\", and down the first path the value +-- inside the 'Just' is never read: it is there to fix the type, which +-- leaves a caller that has only a type with nothing to pass. -- --- If unsure always set a blinder or use signSafer +-- The blinder is optional and 'Nothing' is accepted, but see t'Blinder' for +-- what it covers and when leaving it out is a decision rather than a default. +-- 'signSafer' generates one for you. sign :: HashAlgorithmASN1 hashAlg => Maybe Blinder @@ -524,8 +559,15 @@ -- ^ message to sign -> Either Error ByteString sign blinder hashDescr pk m = dp blinder pk `fmap` makeSignature hashDescr (private_size pk) m +{-# DEPRECATED + sign + "Use signDigest, or signDigestInfo when the message is already a DigestInfo" + #-} --- | sign message using the private key and by automatically generating a blinder. +-- | sign message using the private key, blinded with the pair kept for this +-- key; see "Crypto.PubKey.RSA.Blinded". +-- +-- __Deprecated.__ Use 'signSaferDigest', or 'signSaferDigestInfo'. signSafer :: (HashAlgorithmASN1 hashAlg, MonadRandom m) => Maybe hashAlg @@ -536,11 +578,18 @@ -- ^ message to sign -> m (Either Error ByteString) signSafer hashAlg pk m = do - blinder <- generateBlinder (private_n pk) - return (sign (Just blinder) hashAlg pk m) + d <- drawFor pk + return $ signWithBlindingFor pk d (makeSignature hashAlg (private_size pk) m) +{-# DEPRECATED + signSafer + "Use signSaferDigest, or signSaferDigestInfo when the message is already a DigestInfo" + #-} -- | verify message with the signed message -- +-- __Deprecated.__ Use 'verifyDigest', or 'verifyDigestInfo' where this +-- would have been given 'Nothing'. +-- -- Following RFC 8017, the signature is rejected unless it is exactly as long -- as the modulus (section 8.2.2, step 1) and its integer representative is -- below the modulus (section 5.2.2, step 1). Verification works by @@ -557,13 +606,170 @@ -> ByteString -- ^ Signature -> Bool -verify hashAlg pk m sm +verify hashAlg pk m sm = + verifyEncoded pk (makeSignature hashAlg (public_size pk) m) sm +{-# DEPRECATED + verify + "Use verifyDigest, or verifyDigestInfo when the message is already a DigestInfo" + #-} + +-- | The two checks of RFC 8017 and the comparison, shared by the three +-- verification entry points. The expected encoding is a thunk and is +-- forced only once the checks have passed, as it was when this was written +-- out inside 'verify'. +verifyEncoded + :: PublicKey + -> Either Error ByteString + -- ^ the encoding a signature of this message would have + -> ByteString + -- ^ signature + -> Bool +verifyEncoded pk expected sm | B.length sm /= public_size pk = False | os2ip sm >= public_n pk = False | otherwise = - case makeSignature hashAlg (public_size pk) m of + case expected of Left _ -> False - Right s -> s == (ep pk sm) + Right s -> s == ep pk sm + +-- | Sign a digest. +-- +-- The digest's type says which algorithm made it, so this needs nothing +-- else to name one: the ASN.1 DigestInfo prefix comes from the +-- 'HashAlgorithmASN1' instance that type selects. 'sign' takes @Maybe +-- hashAlg@ and never reads the value inside the @Just@ -- it is there only +-- to fix the type -- which is no burden when the caller has a value and +-- leaves nothing to pass when the caller is itself polymorphic in the +-- algorithm. +-- +-- > signDigest blinder key (hashWith SHA256 message) +-- +-- The blinder is optional and 'Nothing' is accepted, but see t'Blinder' for +-- what it covers and when leaving it out is a decision rather than a +-- default. 'signSaferDigest' generates one for you. +signDigest + :: HashAlgorithmASN1 hashAlg + => Maybe Blinder + -- ^ optional blinder + -> PrivateKey + -- ^ private key + -> Digest hashAlg + -- ^ digest of the message to sign + -> Either Error ByteString +signDigest blinder pk digest = + dp blinder pk `fmap` padSignature (private_size pk) (hashDigestASN1 digest) + +-- | 'signDigest' blinded with the pair kept for this key, as 'signSafer' is +-- to 'sign'; see "Crypto.PubKey.RSA.Blinded". +signSaferDigest + :: (HashAlgorithmASN1 hashAlg, MonadRandom m) + => PrivateKey + -- ^ private key + -> Digest hashAlg + -- ^ digest of the message to sign + -> m (Either Error ByteString) +signSaferDigest pk digest = do + d <- drawFor pk + return $ + signWithBlindingFor pk d (padSignature (private_size pk) (hashDigestASN1 digest)) + +-- | 'signDigest' with the blinding a t'BlindedKey' keeps, and the key for the +-- next operation. +signDigestBlinded + :: (HashAlgorithmASN1 hashAlg, MonadRandom m) + => BlindedKey + -> Digest hashAlg + -- ^ digest of the message to sign + -> m (Either Error ByteString, BlindedKey) +signDigestBlinded bk digest = + signBlindedEncoded bk $ + padSignature (private_size (blindedPrivateKey bk)) (hashDigestASN1 digest) + +-- | Sign something that is already a DigestInfo, the ASN.1 structure +-- naming a hash algorithm and carrying a digest under it. +-- +-- This is what @'sign' blinder 'Nothing'@ does. It needs no +-- 'HashAlgorithmASN1' constraint, because nothing here hashes or encodes: +-- the caller has done both. @'sign' blinder 'Nothing'@ carries the +-- constraint anyway, and since the type variable then appears nowhere else +-- the caller has to name an algorithm that is never used -- +-- @'sign' blinder ('Nothing' :: 'Maybe' 'Crypto.Hash.SHA256')@ -- to say +-- which one it is not using. +signDigestInfo + :: Maybe Blinder + -- ^ optional blinder + -> PrivateKey + -- ^ private key + -> ByteString + -- ^ a DigestInfo, encoded + -> Either Error ByteString +signDigestInfo blinder pk di = + dp blinder pk `fmap` padSignature (private_size pk) di + +-- | 'signDigestInfo' blinded with the pair kept for this key. +signSaferDigestInfo + :: MonadRandom m + => PrivateKey + -- ^ private key + -> ByteString + -- ^ a DigestInfo, encoded + -> m (Either Error ByteString) +signSaferDigestInfo pk di = do + d <- drawFor pk + return $ signWithBlindingFor pk d (padSignature (private_size pk) di) + +-- | 'signDigestInfo' with the blinding a t'BlindedKey' keeps, and the key for +-- the next operation. +signDigestInfoBlinded + :: MonadRandom m + => BlindedKey + -> ByteString + -- ^ a DigestInfo, encoded + -> m (Either Error ByteString, BlindedKey) +signDigestInfoBlinded bk di = + signBlindedEncoded bk $ padSignature (private_size (blindedPrivateKey bk)) di + +-- | The private-key operation on an encoded signature, if there is one; the +-- key is used up only if there is. +signBlindedEncoded + :: MonadRandom m + => BlindedKey + -> Either Error ByteString + -> m (Either Error ByteString, BlindedKey) +signBlindedEncoded bk (Left err) = return (Left err, bk) +signBlindedEncoded bk (Right em) = do + (s, bk') <- dpBlinded bk em + return (Right s, bk') + +-- | The same, with the pair the @Safer@ functions keep for this key. +signWithBlindingFor + :: PrivateKey -> Draw -> Either Error ByteString -> Either Error ByteString +signWithBlindingFor _ _ (Left err) = Left err +signWithBlindingFor pk d (Right em) = Right (withBlindingFor pk d (\pair -> dpWithPair pair pk em)) + +-- | Verify a signature over a digest. The checks are 'verify's. +verifyDigest + :: HashAlgorithmASN1 hashAlg + => PublicKey + -> Digest hashAlg + -- ^ digest of the message + -> ByteString + -- ^ signature + -> Bool +verifyDigest pk digest sm = + verifyEncoded pk (padSignature (public_size pk) (hashDigestASN1 digest)) sm + +-- | Verify a signature over something that is already a DigestInfo, which +-- is what @'verify' 'Nothing'@ does. +verifyDigestInfo + :: PublicKey + -> ByteString + -- ^ a DigestInfo, encoded + -> ByteString + -- ^ signature + -> Bool +verifyDigestInfo pk di sm = + verifyEncoded pk (padSignature (public_size pk) di) sm -- | make signature digest, used in 'sign' and 'verify' makeSignature diff --git a/Crypto/PubKey/RSA/PSS.hs b/Crypto/PubKey/RSA/PSS.hs --- a/Crypto/PubKey/RSA/PSS.hs +++ b/Crypto/PubKey/RSA/PSS.hs @@ -16,6 +16,8 @@ signDigest, signSafer, signDigestSafer, + signBlinded, + signDigestBlinded, verify, verifyDigest, ) where @@ -24,15 +26,15 @@ import Crypto.Number.Basic (numBits) import Crypto.Number.Serialize (os2ip) import Crypto.PubKey.MaskGenFunction -import Crypto.PubKey.RSA (generateBlinder) +import Crypto.PubKey.RSA.Blinded.Internal import Crypto.PubKey.RSA.Prim import Crypto.PubKey.RSA.Types import Crypto.Random.Types -import Data.Bits (shiftR, xor, (.&.)) +import Data.Bits (complement, shiftR, (.&.)) import Data.Word import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess) -import qualified Crypto.Internal.ByteArray as B (convert, eq) +import qualified Crypto.Internal.ByteArray as B (bxor, convert, eq) import Data.ByteString (ByteString) import qualified Data.ByteString as B @@ -69,6 +71,9 @@ -- | Sign using the PSS parameters and the salt explicitely passed as parameters. -- -- the function ignore SaltLength from the PSS Parameters +-- +-- See t'Blinder' for what the optional blinder covers and when leaving it out +-- is a decision rather than a default. 'signSafer' generates one for you. signDigestWithSalt :: HashAlgorithm hash => ByteString @@ -82,9 +87,20 @@ -> Digest hash -- ^ Message digest -> Either Error ByteString -signDigestWithSalt salt blinder params pk digest +signDigestWithSalt salt blinder params pk digest = + dp blinder pk <$> encodeDigest salt params pk digest + +-- | The encoded message the private-key operation is applied to. +encodeDigest + :: HashAlgorithm hash + => ByteString + -> PSSParams hash ByteString ByteString + -> PrivateKey + -> Digest hash + -> Either Error ByteString +encodeDigest salt params pk digest | emLen < hashLen + saltLen + 2 = Left InvalidParameters - | otherwise = Right $ dp blinder pk em + | otherwise = Right em where k = private_size pk emLen = if emTruncate pubBits then k - 1 else k @@ -97,12 +113,15 @@ h = B.convert $ hashWith (pssHash params) m' db = B.concat [B.replicate (dbLen - saltLen - 1) 0, B.singleton 1, salt] dbmask = pssMaskGenAlg params h dbLen - maskedDB = B.pack $ normalizeToKeySize pubBits $ B.zipWith xor db dbmask + maskedDB = normalizeToKeySize pubBits (B.bxor db dbmask :: ByteString) em = B.concat [maskedDB, h, B.singleton (pssTrailerField params)] -- | Sign using the PSS parameters and the salt explicitely passed as parameters. -- -- the function ignore SaltLength from the PSS Parameters +-- +-- See t'Blinder' for what the optional blinder covers and when leaving it out +-- is a decision rather than a default. 'signSafer' generates one for you. signWithSalt :: HashAlgorithm hash => ByteString @@ -121,6 +140,9 @@ mHash = hashWith (pssHash params) m -- | Sign using the PSS Parameters +-- +-- See t'Blinder' for what the optional blinder covers and when leaving it out +-- is a decision rather than a default. 'signSafer' generates one for you. sign :: (HashAlgorithm hash, MonadRandom m) => Maybe Blinder @@ -137,6 +159,9 @@ return (signWithSalt salt blinder params pk m) -- | Sign using the PSS Parameters +-- +-- See t'Blinder' for what the optional blinder covers and when leaving it out +-- is a decision rather than a default. 'signSafer' generates one for you. signDigest :: (HashAlgorithm hash, MonadRandom m) => Maybe Blinder @@ -152,7 +177,8 @@ salt <- getRandomBytes (pssSaltLength params) return (signDigestWithSalt salt blinder params pk digest) --- | Sign using the PSS Parameters and an automatically generated blinder. +-- | Sign using the PSS Parameters, blinded with the pair kept for this key; +-- see "Crypto.PubKey.RSA.Blinded". signSafer :: (HashAlgorithm hash, MonadRandom m) => PSSParams hash ByteString ByteString @@ -162,11 +188,10 @@ -> ByteString -- ^ message to sign -> m (Either Error ByteString) -signSafer params pk m = do - blinder <- generateBlinder (private_n pk) - sign (Just blinder) params pk m +signSafer params pk m = signDigestSafer params pk (hashWith (pssHash params) m) --- | Sign using the PSS Parameters and an automatically generated blinder. +-- | Sign using the PSS Parameters, blinded with the pair kept for this key; +-- see "Crypto.PubKey.RSA.Blinded". signDigestSafer :: (HashAlgorithm hash, MonadRandom m) => PSSParams hash ByteString ByteString @@ -177,9 +202,40 @@ -- ^ message digst -> m (Either Error ByteString) signDigestSafer params pk digest = do - blinder <- generateBlinder (private_n pk) - signDigest (Just blinder) params pk digest + salt <- getRandomBytes (pssSaltLength params) + d <- drawFor pk + return $ case encodeDigest salt params pk digest of + Left err -> Left err + Right em -> Right (withBlindingFor pk d (\pair -> dpWithPair pair pk em)) +-- | 'signSafer' with the blinding a t'BlindedKey' keeps, and the key for the +-- next operation. +signBlinded + :: (HashAlgorithm hash, MonadRandom m) + => PSSParams hash ByteString ByteString + -> BlindedKey + -> ByteString + -- ^ message to sign + -> m (Either Error ByteString, BlindedKey) +signBlinded params bk m = signDigestBlinded params bk (hashWith (pssHash params) m) + +-- | 'signDigestSafer' with the blinding a t'BlindedKey' keeps, and the key +-- for the next operation. +signDigestBlinded + :: (HashAlgorithm hash, MonadRandom m) + => PSSParams hash ByteString ByteString + -> BlindedKey + -> Digest hash + -- ^ message digest + -> m (Either Error ByteString, BlindedKey) +signDigestBlinded params bk digest = do + salt <- getRandomBytes (pssSaltLength params) + case encodeDigest salt params (blindedPrivateKey bk) digest of + Left err -> return (Left err, bk) + Right em -> do + (s, bk') <- dpBlinded bk em + return (Right s, bk') + -- | Verify a signature using the PSS Parameters verify :: HashAlgorithm hash @@ -221,6 +277,7 @@ | B.length s /= k = False | os2ip s >= public_n pk = False | B.any (/= 0) pre = False + | B.any (\x -> x .&. topBits /= 0) (B.take 1 maskedDB) = False | B.last em /= pssTrailerField params = False | B.any (/= 0) ps0 = False | b1 /= B.singleton 1 = False @@ -234,12 +291,19 @@ emLen = if emTruncate pubBits then k - 1 else k dbLen = emLen - hashLen - 1 pubBits = numBits (public_n pk) + -- RFC 8017 9.1.2 step 6: the leftmost 8*emLen - emBits bits of the + -- leftmost octet of maskedDB have to be zero already. Step 9 clears + -- them in DB, which is what normalizeToKeySize does below, and clearing + -- is not checking: without this an encoding with the top bit set -- one + -- the standard calls inconsistent -- verifies as though it were sound, + -- because the bit that made it wrong is thrown away before it is read. + topBits = complement (normalizeMask pubBits) -- unmarshall fields (pre, em) = B.splitAt (k - emLen) (ep pk s) -- drop 0..1 byte maskedDB = B.take dbLen em h = B.take hashLen $ B.drop (B.length maskedDB) em dbmask = pssMaskGenAlg params h dbLen - db = B.pack $ normalizeToKeySize pubBits $ B.zipWith xor maskedDB dbmask + db = normalizeToKeySize pubBits (B.bxor maskedDB dbmask :: ByteString) (ps0, z) = B.break (== 1) db (b1, salt) = B.splitAt 1 z m' = B.concat [B.replicate 8 0, mHash, salt] @@ -249,9 +313,15 @@ emTruncate :: Int -> Bool emTruncate bits = ((bits - 1) .&. 0x7) == 0 -normalizeToKeySize :: Int -> [Word8] -> [Word8] -normalizeToKeySize _ [] = [] -- very unlikely -normalizeToKeySize bits (x : xs) = x .&. mask : xs +normalizeToKeySize :: Int -> ByteString -> ByteString +normalizeToKeySize bits x = case B.uncons x of + Nothing -> x -- very unlikely + Just (h, t) -> B.cons (h .&. normalizeMask bits) t + +-- | The bits of the leftmost octet that belong to the encoding: the low +-- @emBits `mod` 8@ of them, or all eight when that is zero. Its complement +-- is the bits RFC 8017 requires to be zero. +normalizeMask :: Int -> Word8 +normalizeMask bits = if sh > 0 then 0xff `shiftR` (8 - sh) else 0xff where - mask = if sh > 0 then 0xff `shiftR` (8 - sh) else 0xff sh = (bits - 1) .&. 0x7 diff --git a/Crypto/PubKey/RSA/Prim.hs b/Crypto/PubKey/RSA/Prim.hs --- a/Crypto/PubKey/RSA/Prim.hs +++ b/Crypto/PubKey/RSA/Prim.hs @@ -1,3 +1,5 @@ +{-# LANGUAGE BangPatterns #-} + -- | -- Module : Crypto.PubKey.RSA.Prim -- License : BSD-style @@ -12,9 +14,19 @@ ep, ) where +import Data.Memory.PtrMethods (memSet) +import Data.Word (Word32, Word8) +import Foreign.C.Types (CInt (..)) +import Foreign.Marshal.Alloc (allocaBytes) +import Foreign.Ptr (Ptr, plusPtr) + import Crypto.Internal.ByteArray (ByteArray) +import Crypto.Internal.Compat (unsafeDoIO) +import Crypto.Number.Basic (numBytes) import Crypto.Number.ModArithmetic (expFast, expSafe) +import Crypto.Number.ModArithmetic.Pub (expPubBytes) import Crypto.Number.Serialize (i2ospOf_, os2ip) +import qualified Crypto.Number.Serialize.Internal as Internal import Crypto.PubKey.RSA.Types {- dpSlow computes the decrypted message not using any precomputed cache value. @@ -29,22 +41,78 @@ dpFast (Blinder r rm1) pk c = i2ospOf_ (private_size pk) - (multiplication rm1 (m2 + h * (private_q pk)) (private_n pk)) + (multiplication rm1 (crtChecked pk iC (m2 + h * (private_q pk))) (private_n pk)) where re = expFast r (public_e $ private_pub pk) (private_n pk) iC = multiplication re (os2ip c) (private_n pk) - m1 = expSafe iC (private_dP pk) (private_p pk) - m2 = expSafe iC (private_dQ pk) (private_q pk) + (m1, m2) = crtHalves iC pk h = ((private_qinv pk) * (m1 - m2)) `mod` (private_p pk) dpFastNoBlinder :: ByteArray ba => PrivateKey -> ba -> ba -dpFastNoBlinder pk c = i2ospOf_ (private_size pk) (m2 + h * (private_q pk)) +dpFastNoBlinder pk c = i2ospOf_ (private_size pk) (crtChecked pk iC (m2 + h * (private_q pk))) where iC = os2ip c - m1 = expSafe iC (private_dP pk) (private_p pk) - m2 = expSafe iC (private_dQ pk) (private_q pk) + (m1, m2) = crtHalves iC pk h = ((private_qinv pk) * (m1 - m2)) `mod` (private_p pk) +-- | The CRT answer m for c, raised back to e before it is let out: a fault +-- in one half gives an m whose gcd with n is the other prime (Boneh, DeMillo +-- and Lipton). On a mismatch it is computed again without CRT, as OpenSSL does. +crtChecked :: PrivateKey -> Integer -> Integer -> Integer +crtChecked pk c m + | expFast m (public_e (private_pub pk)) n == c `mod` n = m + | private_d pk > 0 = expSafe c (private_d pk) n + | otherwise = m + where + n = private_n pk + +-- | c^dP mod p and c^dQ mod q, the halves of the private-key operation: both +-- at once where the C can (AVX-512 IFMA, and the sizes of RSA-2048, -3072 and +-- -4096), one after the other where it declines. +crtHalves :: Integer -> PrivateKey -> (Integer, Integer) +crtHalves c pk = case expSafeX2 (c `mod` p) dP p (c `mod` q) dQ q of + Just r -> r + Nothing -> (expSafe c dP p, expSafe c dQ q) + where + p = private_p pk + q = private_q pk + dP = private_dP pk + dQ = private_dQ pk + +-- | Two secure exponentiations in one call. The bases are reduced already +-- and the moduli are the same length; both exponents go in at that length, +-- so the length the C sees is the modulus', as 'expSafe' pads to whole words. +expSafeX2 :: Integer -> Integer -> Integer -> Integer -> Integer -> Integer -> Maybe (Integer, Integer) +expSafeX2 b1 e1 m1 b2 e2 m2 + | numBytes m2 /= len || not (odd m1 && odd m2) = Nothing + | e1 < 0 || e2 < 0 || numBytes e1 > len || numBytes e2 > len = Nothing + | otherwise = unsafeDoIO $ + allocaBytes (8 * len) $ \start -> case take 8 (iterate (`plusPtr` len) start) of + [o1, pb1, pe1, pm1, o2, pb2, pe2, pm2] -> do + mapM_ + (\(v, ptr) -> Internal.i2ospOf v ptr len) + [(b1, pb1), (e1, pe1), (m1, pm1), (b2, pb2), (e2, pe2), (m2, pm2)] + r <- c_powm_sec_x2 o1 pb1 l pe1 l pm1 l o2 pb2 l pe2 l pm2 l + -- the exponents are the caller's secrets + memSet pe1 0 len + memSet pe2 0 len + if r /= 0 + then return Nothing + else do + !v1 <- Internal.os2ip o1 len + !v2 <- Internal.os2ip o2 len + return (Just (v1, v2)) + _ -> return Nothing + where + !len = numBytes m1 + l = fromIntegral len :: Word32 + +foreign import ccall safe "crypton_powm_sec_x2" + c_powm_sec_x2 + :: Ptr Word8 -> Ptr Word8 -> Word32 -> Ptr Word8 -> Word32 -> Ptr Word8 -> Word32 + -> Ptr Word8 -> Ptr Word8 -> Word32 -> Ptr Word8 -> Word32 -> Ptr Word8 -> Word32 + -> IO CInt + -- | Compute the RSA decrypt primitive. -- if the p and q numbers are available, then dpFast is used -- otherwise, we use dpSlow which only need d and n. @@ -56,7 +124,9 @@ -- | Compute the RSA encrypt primitive ep :: ByteArray ba => PublicKey -> ba -> ba -ep pk m = i2ospOf_ (public_size pk) $ expFast (os2ip m) (public_e pk) (public_n pk) +ep pk m = case expPubBytes (public_e pk) (public_n pk) m of + Just r -> r + Nothing -> i2ospOf_ (public_size pk) $ expFast (os2ip m) (public_e pk) (public_n pk) -- | multiply 2 integers in Zm only performing the modulo operation if necessary multiplication :: Integer -> Integer -> Integer -> Integer diff --git a/Crypto/PubKey/RSA/Types.hs b/Crypto/PubKey/RSA/Types.hs --- a/Crypto/PubKey/RSA/Types.hs +++ b/Crypto/PubKey/RSA/Types.hs @@ -27,8 +27,32 @@ import GHC.Generics --- | Blinder which is used to obfuscate the timing --- of the decryption primitive (used by decryption and signing). +-- | A blinder, which keeps the timing of the private key operation from +-- saying anything about the number it was given. +-- +-- The private exponent is not what is at risk. 'Crypto.Number.ModArithmetic.expSafe', +-- which the exponentiation goes through, keeps the /value/ of an exponent out +-- of the work it does. +-- +-- What a blinder covers is the other side. Without one, the operation runs +-- on the ciphertext as it arrived, so how long it takes depends on a number +-- an attacker may have chosen and can vary -- which is what a remote timing +-- attack on RSA needs. With one, the input is multiplied by a random value +-- first and that value divided out afterwards, so the timing carries nothing +-- an attacker can steer. +-- +-- Every private key operation here takes a @'Maybe' t'Blinder'@. The +-- @Safer@ form of each -- 'Crypto.PubKey.RSA.PKCS15.decryptSafer', +-- 'Crypto.PubKey.RSA.PKCS15.signSafer' and their kind -- generates one and is +-- the one to reach for. Pass 'Nothing' only where the input is not attacker +-- controlled and you have decided that it is not. +-- +-- A blinder costs one more exponentiation, by the public exponent, which is +-- the cheap direction: measured on an Apple M4, PKCS#1 v1.5 signing goes from +-- about 601 to about 620 microseconds. +-- +-- Use a blinder once. 'Crypto.PubKey.RSA.generateBlinder' makes a fresh one; +-- carrying one across operations is not what it is for. data Blinder = Blinder !Integer !Integer deriving (Show, Eq) diff --git a/Crypto/PubKey/Rabin/Basic.hs b/Crypto/PubKey/Rabin/Basic.hs --- a/Crypto/PubKey/Rabin/Basic.hs +++ b/Crypto/PubKey/Rabin/Basic.hs @@ -25,6 +25,38 @@ -- -- Around all of that is 'Integer' arithmetic, whose cost follows the size of -- the numbers; see "Crypto.PubKey.DSA" for that note at more length. +-- +-- == The hash algorithm is passed as a value, and that is not going to change +-- +-- 'sign', 'signWith' and 'verify' take a @hash@ argument whose value they +-- never read. Every 'Crypto.Hash.HashAlgorithm' instance is a nullary +-- constructor, and the algorithm comes from the type; the value is there to +-- carry the type and nothing else. A caller holding a value loses nothing +-- by it, and a caller that is itself polymorphic in the algorithm has no +-- value to pass. +-- +-- Elsewhere in crypton that is answered by taking a +-- 'Crypto.Hash.Digest' instead: the digest carries the algorithm in its +-- type and the bytes in its value, so nothing is passed only to name a +-- type. "Crypto.PubKey.RSA.PKCS15", "Crypto.PubKey.DSA", +-- "Crypto.PubKey.ElGamal", "Crypto.PubKey.Rabin.RW" and +-- "Crypto.PubKey.Rabin.Modified" all do that. +-- +-- This module cannot. What is hashed here is not the message: +-- +-- > h = os2ip $ hashWith hashAlg $ B.append padding m +-- +-- and the padding is not the caller's either. 'sign' searches for one, +-- drawing eight bytes at a time until the first octet is non-zero and the +-- Jacobi symbols of the hash modulo each private prime are both 1. Which +-- bytes get hashed is therefore decided inside the signing operation, after +-- the caller has handed over the message, so there is no digest for a +-- caller to compute in advance. +-- +-- A proxy argument would work where a digest cannot. It is deliberately +-- not added: it would be the one exception to a rule the rest of the +-- library now follows, and this module has no callers asking for it. See +-- the survey in . module Crypto.PubKey.Rabin.Basic ( PublicKey (..), PrivateKey (..), diff --git a/Crypto/PubKey/Rabin/Modified.hs b/Crypto/PubKey/Rabin/Modified.hs --- a/Crypto/PubKey/Rabin/Modified.hs +++ b/Crypto/PubKey/Rabin/Modified.hs @@ -17,7 +17,9 @@ PrivateKey (..), generate, sign, + signDigest, verify, + verifyDigest, ) where import Crypto.Debug (DebugShow (..)) @@ -109,10 +111,25 @@ -> ByteString -- ^ message to sign -> Either Error Integer -sign pk hashAlg m = +sign pk hashAlg m = signDigest pk (hashWith hashAlg m) + +-- | Sign a digest using the private key. +-- +-- The digest's type says which algorithm made it, so this needs nothing else +-- to name one. 'sign' takes a @hash@ value and never reads it -- it is there +-- to fix the type -- which leaves a caller that is itself polymorphic in the +-- algorithm with nothing to pass. +signDigest + :: HashAlgorithm hash + => PrivateKey + -- ^ private key + -> Digest hash + -- ^ digest of the message to sign + -> Either Error Integer +signDigest pk digest = let d = private_d pk n = public_n $ private_pub pk - h = os2ip $ hashWith hashAlg m + h = os2ip digest limit = (n - 6) `div` 16 in if h > limit then Left MessageTooLong @@ -135,14 +152,27 @@ -> Integer -- ^ signature -> Bool -verify pk hashAlg m s +verify pk hashAlg m s = verifyDigest pk (hashWith hashAlg m) s + +-- | Verify a signature over a digest. See 'signDigest' for why a digest +-- rather than a @hash@ value. +verifyDigest + :: HashAlgorithm hash + => PublicKey + -- ^ public key + -> Digest hash + -- ^ digest of the message + -> Integer + -- ^ signature + -> Bool +verifyDigest pk digest s -- squaring works modulo n, so s + n and -s would verify wherever s does | s < 0 || s >= n = False | otherwise = go where n = public_n pk go = - let h = os2ip $ hashWith hashAlg m + let h = os2ip digest s' = expSafe s 2 n s'' = case s' `mod` 8 of 6 -> s' diff --git a/Crypto/PubKey/Rabin/OAEP.hs b/Crypto/PubKey/Rabin/OAEP.hs --- a/Crypto/PubKey/Rabin/OAEP.hs +++ b/Crypto/PubKey/Rabin/OAEP.hs @@ -17,9 +17,8 @@ import Data.Bits (complement, shiftR, xor, (.&.), (.|.)) import Data.ByteString (ByteString) import qualified Data.ByteString as B -import Data.List (foldl') +import qualified Data.List as L import Data.Word (Word32) -import Prelude hiding (foldl') import Crypto.Hash import Crypto.Internal.ByteArray (ByteArray, ByteArrayAccess) @@ -127,7 +126,7 @@ -- is zero; all of them are looked at either way oneIndex = fst $ - foldl' + L.foldl' step (fromIntegral (B.length db1) :: Word32, 1 :: Word32) (zip [0 ..] (B.unpack db1)) diff --git a/Crypto/PubKey/Rabin/RW.hs b/Crypto/PubKey/Rabin/RW.hs --- a/Crypto/PubKey/Rabin/RW.hs +++ b/Crypto/PubKey/Rabin/RW.hs @@ -21,7 +21,9 @@ encryptWithSeed, decrypt, sign, + signDigest, verify, + verifyDigest, ) where import Crypto.Debug (DebugShow (..)) @@ -181,11 +183,26 @@ -> ByteString -- ^ message to sign -> Either Error Integer -sign pk hashAlg m = +sign pk hashAlg m = signDigest pk (hashWith hashAlg m) + +-- | Sign a digest using the private key. +-- +-- The digest's type says which algorithm made it, so this needs nothing else +-- to name one. 'sign' takes a @hash@ value and never reads it -- it is there +-- to fix the type -- which leaves a caller that is itself polymorphic in the +-- algorithm with nothing to pass. +signDigest + :: HashAlgorithm hash + => PrivateKey + -- ^ private key + -> Digest hash + -- ^ digest of the message to sign + -> Either Error Integer +signDigest pk digest = let d = private_d pk n = public_n $ private_pub pk in do - m' <- ep1 n $ os2ip $ hashWith hashAlg m + m' <- ep1 n $ os2ip digest return $ dp1 d n m' -- | Verify signature using hash algorithm and public key. @@ -200,11 +217,24 @@ -> Integer -- ^ signature -> Bool -verify pk hashAlg m s +verify pk hashAlg m s = verifyDigest pk (hashWith hashAlg m) s + +-- | Verify a signature over a digest. See 'signDigest' for why a digest +-- rather than a @hash@ value. +verifyDigest + :: HashAlgorithm hash + => PublicKey + -- ^ public key + -> Digest hash + -- ^ digest of the message + -> Integer + -- ^ signature + -> Bool +verifyDigest pk digest s -- squaring works modulo n, so s + n and -s would verify wherever s does | s < 0 || s >= n = False | otherwise = - let h = os2ip $ hashWith hashAlg m + let h = os2ip digest h' = dp2 n $ ep2 n s in h' == h where diff --git a/Crypto/Random.hs b/Crypto/Random.hs --- a/Crypto/Random.hs +++ b/Crypto/Random.hs @@ -7,35 +7,159 @@ -- Maintainer : Vincent Hanquez -- Stability : stable -- Portability : good +-- +-- Random bytes, drawn either from the system or from a generator whose +-- seed you hold. +-- +-- == Drawing from the system +-- +-- 'getRandomBytes' in 'IO' is the ordinary way to get bytes nobody can +-- predict. The length is in bytes and the type is any 'ByteArray', so the +-- result is usually pinned down by where it goes: +-- +-- > import Crypto.Random (getRandomBytes) +-- > import Data.ByteString (ByteString) +-- > +-- > nonce <- getRandomBytes 12 :: IO ByteString +-- +-- Everything in this library that needs randomness takes it the same way, +-- through a @MonadRandom m =>@ constraint, so running it in 'IO' is the +-- whole of choosing this source: +-- +-- > import qualified Crypto.PubKey.RSA as RSA +-- > +-- > (pub, priv) <- RSA.generate 256 0x10001 +-- +-- Where the operating system offers @getrandom(2)@ or @getentropy(3)@ the +-- bytes come from a ChaCha20 generator belonging to the operating system +-- thread the call runs on, seeded from the system and reseeded as it goes; +-- see "Crypto.Random.SysDRG" for what that is and what it is not. Where +-- it does not -- Windows, for now -- they come from the system on every +-- call, as they always did. Either way the caller writes the same line. +-- +-- Any Haskell thread may draw, and none has to say which generator it +-- wants: +-- +-- > import Control.Concurrent (forkIO) +-- > +-- > mapM_ (\_ -> forkIO (getRandomBytes 32 >>= use)) [1 .. 64 :: Int] +-- +-- The generator belongs to the operating system thread that happens to be +-- carrying the Haskell thread when the call is made, and is found through +-- that thread's own storage rather than by anything the caller passes. A +-- @forkIO@ thread moves between capabilities, so two draws from one Haskell +-- thread may be answered by two generators; they are seeded independently +-- of each other, so it makes no difference which one answers. +-- +-- == Drawing from a generator you hold +-- +-- A generator of your own is the way to draw many times from one seed. +-- The system is asked once, when the generator is made, and not again. +-- That is how @tls@ does a connection: 'seedNew' when it opens, and +-- everything the handshake needs afterwards from the generator, whose +-- state the connection carries from one draw to the next. +-- +-- > import Crypto.Random (ChaChaDRG, drgNewSeed, seedNew, withDRG) +-- > +-- > data Connection = Connection { connRNG :: ChaChaDRG } +-- > +-- > newConnection :: IO Connection +-- > newConnection = do +-- > seed <- seedNew -- the only draw from the system +-- > return (Connection (drgNewSeed seed)) +-- > +-- > -- every later draw advances the connection's own generator +-- > connRandom :: Int -> Connection -> (ByteString, Connection) +-- > connRandom n conn = +-- > let (bytes, rng') = withDRG (connRNG conn) (getRandomBytes n) +-- > in (bytes, conn{connRNG = rng'}) +-- +-- Inside 'withDRG' the same 'getRandomBytes' resolves to the instance for +-- 'MonadPseudoRandom' rather than the one for 'IO', so it touches neither +-- the system nor the per-thread generator: it advances the generator it was +-- given and hands it back. Keep that generator and the bytes are +-- reproducible from the seed, which is what makes a test repeatable -- and +-- what makes a generator unfit for keys unless its seed came from the +-- system. +-- +-- 'drgNew' is 'seedNew' and 'drgNewSeed' in one step; 'drgNewTest' takes +-- four numbers instead of a seed, for a test that must give the same answer +-- twice. +-- +-- == Drawing from a monad stacked on IO +-- +-- There are two instances, one for 'IO' and one for 'MonadPseudoRandom', +-- and none for the transformers, so a @ReaderT env IO@ or a @StateT s IO@ +-- reaches the system through 'liftIO': +-- +-- > import Control.Monad.IO.Class (liftIO) +-- > import Control.Monad.Trans.Reader (ReaderT) +-- > +-- > newKey :: ReaderT env IO ByteString +-- > newKey = liftIO (getRandomBytes 32) +-- +-- What that draws is what the first section describes, no more and no less: +-- the 'IO' instance, the generator of the operating system thread the call +-- lands on, seeded from the system and reseeded as it goes. 'liftIO' says +-- which monad the draw happens in and nothing about where the bytes come +-- from. +-- +-- Writing an instance for the stack instead would make the choice invisible +-- at the call site, and the class cannot tell a strong source from a weak +-- one -- see 'MonadRandom'. +-- +-- == When the system will not give any +-- +-- Drawing randomness is the one thing here with nothing to fall back on, +-- so the failure is an exception rather than a value: there is no sensible +-- 'Maybe' to return and no partial answer worth having. Since 2.2 it is an +-- 'EntropyError' and not a @Control.Exception.ErrorCall@ or an +-- @Control.Exception.IOException@, which is worth knowing if you were +-- catching one of those: +-- +-- > import Control.Exception (catch) +-- > import Crypto.Random (EntropyError (..), getRandomBytes) +-- > +-- > key <- getRandomBytes 32 `catch` \e -> case e of +-- > NoEntropySource -> fail "this system offers no randomness at all" +-- > EntropyShort w g -> fail (show w ++ " bytes wanted, " ++ show g ++ " arrived") +-- > EntropySourceLost s -> fail ("the source " ++ s ++ " went away") +-- +-- Catching it at all is a decision rather than a default. A program that +-- cannot get randomness cannot make a key, and stopping is usually the +-- honest thing; the reason to catch is to say so in the program's own +-- terms rather than in crypton's. module Crypto.Random ( - -- * Deterministic instances - ChaChaDRG, - SystemDRG, - Seed, + -- * Drawing from the system + MonadRandom (..), - -- * Seed + -- * Drawing from a generator you hold + Seed, seedNew, seedFromInteger, seedToInteger, seedFromBinary, - - -- * Deterministic Random class - getSystemDRG, - drgNew, drgNewSeed, + drgNew, drgNewTest, withDRG, withRandomBytes, + MonadPseudoRandom, DRG (..), - -- * Random abstraction - MonadRandom (..), - MonadPseudoRandom, + -- * The generators + ChaChaDRG, + SystemDRG, + getSystemDRG, + + -- * When the system will not give any + EntropyError (..), ) where import Crypto.Error import Crypto.Internal.Imports import Crypto.Random.ChaChaDRG +import Crypto.Random.Entropy (EntropyError (..)) import Crypto.Random.SystemDRG import Crypto.Random.Types import Data.ByteArray (ByteArray, ByteArrayAccess, ScrubbedBytes) @@ -50,6 +174,9 @@ import Foreign.Ptr (Ptr, castPtr) #endif +-- | The material a deterministic generator is built from. Two generators +-- made from one seed produce the same bytes, which is what 'drgNewSeed' is +-- for and why a seed kept anywhere is as good as the keys drawn from it. newtype Seed = Seed ScrubbedBytes deriving (ByteArrayAccess) diff --git a/Crypto/Random/Entropy.hs b/Crypto/Random/Entropy.hs --- a/Crypto/Random/Entropy.hs +++ b/Crypto/Random/Entropy.hs @@ -6,16 +6,37 @@ -- Portability : Good module Crypto.Random.Entropy ( getEntropy, + EntropyError (..), ) where import Crypto.Internal.ByteArray (ByteArray) import qualified Crypto.Internal.ByteArray as B -import Data.Maybe (catMaybes) +import System.IO.Unsafe (unsafeInterleaveIO, unsafePerformIO) import Crypto.Random.Entropy.Unsafe +-- | The backends this system has, worked out once and no further than +-- needed. +-- +-- Opening one is not free: for a device file it means opening and closing +-- @\/dev\/random@ or @\/dev\/urandom@ just to learn that it is there. This +-- used to be done on every call, and for the whole list before any backend +-- was asked for a byte, so a program paid for both devices even when the +-- first backend answered everything. +-- +-- Once, now, and lazily: 'replenish' stops as soon as the buffer is full, +-- which leaves the tail of this list unforced, so a system where the first +-- backend answers never opens a device at all. +{-# NOINLINE openedBackends #-} +openedBackends :: [EntropyBackend] +openedBackends = unsafePerformIO (openAsNeeded supportedBackends) + where + openAsNeeded [] = return [] + openAsNeeded (o : os) = do + m <- o + rest <- unsafeInterleaveIO (openAsNeeded os) + return $ maybe rest (: rest) m + -- | Get some entropy from the system source of entropy getEntropy :: ByteArray byteArray => Int -> IO byteArray -getEntropy n = do - backends <- catMaybes `fmap` sequence supportedBackends - B.alloc n (replenish n backends) +getEntropy n = B.alloc n (replenish n openedBackends) diff --git a/Crypto/Random/Entropy/Backend.hs b/Crypto/Random/Entropy/Backend.hs --- a/Crypto/Random/Entropy/Backend.hs +++ b/Crypto/Random/Entropy/Backend.hs @@ -11,27 +11,39 @@ ( EntropyBackend , supportedBackends , gatherBackend + , EntropyError(..) ) where import Foreign.Ptr import Data.Proxy import Data.Word (Word8) import Crypto.Random.Entropy.Source -#ifdef SUPPORT_RDRAND -import Crypto.Random.Entropy.RDRand -#endif #ifdef WINDOWS import Crypto.Random.Entropy.Windows #else +import Crypto.Random.Entropy.SysRandom import Crypto.Random.Entropy.Unix #endif --- | All supported backends +-- | All supported backends, best first. +-- +-- The system call comes before everything else: it is the kernel's own +-- generator, it needs no descriptor, and it is what the rest of the world +-- reaches for now. The device files are what is left when the system has +-- no such call. +-- +-- RDRAND is deliberately not here, though it used to be first on x86. A +-- list like this one is a list of alternatives, and whichever answers +-- first decides the bytes on its own -- which is the one thing #298 says +-- RDRAND should not do. Nor does it contribute anywhere else: every +-- system this builds for already feeds the instruction into the pool the +-- call above draws from, so asking it again adds nothing. See the note in +-- @cbits\/crypton_sysdrg.c@. supportedBackends :: [IO (Maybe EntropyBackend)] supportedBackends = [ -#ifdef SUPPORT_RDRAND - openBackend (Proxy :: Proxy RDRand), +#ifndef WINDOWS + openBackend (Proxy :: Proxy SysRandom), #endif #ifdef WINDOWS openBackend (Proxy :: Proxy WinCryptoAPI) diff --git a/Crypto/Random/Entropy/RDRand.hs b/Crypto/Random/Entropy/RDRand.hs deleted file mode 100644 --- a/Crypto/Random/Entropy/RDRand.hs +++ /dev/null @@ -1,39 +0,0 @@ -{-# LANGUAGE ForeignFunctionInterface #-} - --- | --- Module : Crypto.Random.Entropy.RDRand --- License : BSD-style --- Maintainer : Vincent Hanquez --- Stability : experimental --- Portability : Good -module Crypto.Random.Entropy.RDRand ( - RDRand, -) where - -import Crypto.Random.Entropy.Source -import Data.Word (Word8) -import Foreign.C.Types -import Foreign.Ptr - -foreign import ccall unsafe "crypton_cpu_has_rdrand" - c_cpu_has_rdrand :: IO CInt - -foreign import ccall unsafe "crypton_get_rand_bytes" - c_get_rand_bytes :: Ptr Word8 -> CInt -> IO CInt - --- | Fake handle to Intel RDRand entropy CPU instruction -data RDRand = RDRand - -instance EntropySource RDRand where - entropyOpen = rdrandGrab - entropyGather _ = rdrandGetBytes - entropyClose _ = return () - -rdrandGrab :: IO (Maybe RDRand) -rdrandGrab = supported `fmap` c_cpu_has_rdrand - where - supported 0 = Nothing - supported _ = Just RDRand - -rdrandGetBytes :: Ptr Word8 -> Int -> IO Int -rdrandGetBytes ptr sz = fromIntegral `fmap` c_get_rand_bytes ptr (fromIntegral sz) diff --git a/Crypto/Random/Entropy/Source.hs b/Crypto/Random/Entropy/Source.hs --- a/Crypto/Random/Entropy/Source.hs +++ b/Crypto/Random/Entropy/Source.hs @@ -6,8 +6,32 @@ -- Portability : Good module Crypto.Random.Entropy.Source where +import Control.Exception (Exception) import Data.Word (Word8) import Foreign.Ptr + +-- | The system would not give the entropy it was asked for. +-- +-- This is the one failure in the library with nothing to fall back on and +-- nothing sensible to return: a key drawn from bytes that are not random +-- is worse than no key. It used to be reported with 'error' and 'fail', +-- which left a caller no way to tell it from a bug in the library, and no +-- way to say anything useful about it. +data EntropyError + = -- | The system offers no source of entropy at all. On Unix that + -- means no @getrandom(2)@, no @getentropy(3)@ and no @\/dev@ to read + -- from; a container built from nothing would look like this. + NoEntropySource + | -- | The sources between them gave fewer bytes than were asked for, + -- three times over. The two numbers are how many were wanted and + -- how many arrived. + EntropyShort Int Int + | -- | A source that could be opened once could not be opened again. + -- The string names it. + EntropySourceLost String + deriving (Show, Eq) + +instance Exception EntropyError -- | A handle to an entropy maker, either a system capability -- or a hardware generator. diff --git a/Crypto/Random/Entropy/SysRandom.hs b/Crypto/Random/Entropy/SysRandom.hs new file mode 100644 --- /dev/null +++ b/Crypto/Random/Entropy/SysRandom.hs @@ -0,0 +1,50 @@ +{-# LANGUAGE ForeignFunctionInterface #-} + +-- | +-- Module : Crypto.Random.Entropy.SysRandom +-- License : BSD-style +-- Maintainer : Kazu Yamamoto +-- Stability : experimental +-- Portability : Unix +-- +-- The kernel's own generator, reached without a file descriptor: +-- @getrandom(2)@ on Linux and FreeBSD, @getentropy(3)@ where that is what +-- the system has. +-- +-- This is the source to prefer over reading @\/dev\/urandom@. It needs no +-- path and no descriptor, so it still answers where @\/dev@ is not mounted +-- or not populated, and it cannot be defeated by a full descriptor table. +module Crypto.Random.Entropy.SysRandom ( + SysRandom, +) where + +import Crypto.Random.Entropy.Source +import Data.Word (Word8) +import Foreign.C.Types +import Foreign.Ptr + +-- Both are @safe@ rather than @unsafe@: at early boot, before the kernel +-- pool is initialised, these calls block, and an unsafe call that blocks +-- holds the capability it runs on. +foreign import ccall safe "crypton_sysrandom_available" + c_sysrandom_available :: IO CInt + +foreign import ccall safe "crypton_sysrandom_bytes" + c_sysrandom_bytes :: Ptr Word8 -> CInt -> IO CInt + +-- | The system call, where there is one. +data SysRandom = SysRandom + +instance EntropySource SysRandom where + -- Asked at run time and not only at compile time: a binary built where + -- the header declares the call can still run on a kernel that answers + -- ENOSYS. + entropyOpen = available `fmap` c_sysrandom_available + where + available 0 = Nothing + available _ = Just SysRandom + + entropyGather _ ptr n = + fromIntegral `fmap` c_sysrandom_bytes ptr (fromIntegral n) + + entropyClose _ = return () diff --git a/Crypto/Random/Entropy/Unix.hs b/Crypto/Random/Entropy/Unix.hs --- a/Crypto/Random/Entropy/Unix.hs +++ b/Crypto/Random/Entropy/Unix.hs @@ -61,7 +61,7 @@ withDev filepath f = openDev filepath >>= \h -> case h of - Nothing -> error ("device " ++ filepath ++ " cannot be grabbed") + Nothing -> E.throwIO (EntropySourceLost filepath) Just fd -> f fd `E.finally` closeDev fd closeDev :: H -> IO () diff --git a/Crypto/Random/Entropy/Unsafe.hs b/Crypto/Random/Entropy/Unsafe.hs --- a/Crypto/Random/Entropy/Unsafe.hs +++ b/Crypto/Random/Entropy/Unsafe.hs @@ -9,25 +9,27 @@ module Crypto.Random.Entropy.Backend, ) where +import Control.Exception (throwIO) + import Crypto.Random.Entropy.Backend import Data.Word (Word8) import Foreign.Ptr (Ptr, plusPtr) -- | Refill the entropy in a buffer -- --- Call each entropy backend in turn until the buffer has --- been replenished. +-- Call each entropy backend in turn until the buffer has been replenished. -- --- If the buffer cannot be refill after 3 loopings, this will raise --- an User Error exception +-- Throws 'EntropyError': 'NoEntropySource' when there is no backend at all, +-- and 'EntropyShort' when three passes over the backends still leave the +-- buffer unfilled. replenish :: Int -> [EntropyBackend] -> Ptr Word8 -> IO () -replenish _ [] _ = fail "crypton: random: cannot get any source of entropy on this system" +replenish _ [] _ = throwIO NoEntropySource replenish poolSize backends ptr = loop 0 backends ptr poolSize where loop :: Int -> [EntropyBackend] -> Ptr Word8 -> Int -> IO () loop _ _ _ 0 = return () loop retry [] p n - | retry == 3 = error "crypton: random: cannot fully replenish" + | retry == 3 = throwIO $ EntropyShort poolSize (poolSize - n) | otherwise = loop (retry + 1) backends p n loop retry (b : bs) p n = do r <- gatherBackend b p n diff --git a/Crypto/Random/Entropy/Windows.hs b/Crypto/Random/Entropy/Windows.hs --- a/Crypto/Random/Entropy/Windows.hs +++ b/Crypto/Random/Entropy/Windows.hs @@ -23,6 +23,7 @@ import Foreign.Storable (peek) import System.Win32.Types (getLastError) +import Control.Exception (throwIO) import Crypto.Random.Entropy.Source @@ -38,7 +39,8 @@ case mctx of Nothing -> do lastError <- getLastError - fail $ "cannot re-grab win crypto api: error " ++ show lastError + throwIO $ EntropySourceLost + ("the Windows crypto API: error " ++ show lastError) Just ctx -> do r <- cryptGenRandom ctx ptr n cryptReleaseCtx ctx @@ -100,4 +102,5 @@ then return () else do lastError <- getLastError - fail $ "cryptReleaseCtx: error " ++ show lastError + throwIO $ EntropySourceLost + ("cryptReleaseCtx: error " ++ show lastError) diff --git a/Crypto/Random/SysDRG.hs b/Crypto/Random/SysDRG.hs new file mode 100644 --- /dev/null +++ b/Crypto/Random/SysDRG.hs @@ -0,0 +1,82 @@ +{-# LANGUAGE ForeignFunctionInterface #-} + +-- | +-- Module : Crypto.Random.SysDRG +-- License : BSD-style +-- Maintainer : Kazu Yamamoto +-- Stability : experimental +-- Portability : Unix, Windows +-- +-- The generator behind the 'Crypto.Random.MonadRandom' instance for 'IO'. +-- +-- A ChaCha20 generator per operating system thread, seeded from a +-- process-wide generator, seeded in turn from the system entropy pool. +-- The state and the reseeding live in +-- @cbits\/crypton_sysdrg.c@, because a @forkIO@ thread is not an operating +-- system thread -- it moves between capabilities -- so state held against +-- one would be shared by threads running at the same time. +-- +-- == What it is, and what it is not +-- +-- It is not a DRBG of NIST SP 800-90A. That standard names three -- +-- @Hash_DRBG@, @HMAC_DRBG@ and @CTR_DRBG@ -- and none of them is built on a +-- stream cipher. The name here is the older and looser sense of the word. +-- +-- What it is is the shape @arc4random@ on OpenBSD and @get_random_bytes@ in +-- the Linux kernel both have, and the one asked for in +-- . That shape is +-- common; it is not specified anywhere, so the parts a standard would have +-- fixed were chosen here instead: +-- +-- * ChaCha20, rather than AES in counter mode. +-- * SHA-512 over the system's bytes, rather than a derivation function a +-- standard would have named. +-- * A mebibyte per thread, and a mebibyte of issued seed for the process +-- generator, as the points at which to reseed. Those numbers are a +-- choice, not a result. +-- +-- The pieces underneath are specified: ChaCha20 is RFC 8439, SHA-512 is +-- FIPS 180-4. The way they are put together is not. +-- +-- == What a compromised state gives away +-- +-- Each draw ends by taking the next forty bytes of keystream as the key and +-- nonce and dropping the ones that made the output, so a state read after a +-- draw is not the state that produced it. ChaCha20 does not run backwards +-- and the key that would be needed is gone, which is backtracking +-- resistance in the terms of SP 800-90A. Without it, a key stands until +-- the next reseed and anyone holding the state can wind the counter back +-- over everything issued since -- a mebibyte of output that was meant to be +-- secret. It is what @arc4random@ does, and for the same reason. +-- +-- Prediction resistance is what the reseeding gives. A reseed draws from +-- the system again, so a state that has been read does not determine what +-- comes after one. +module Crypto.Random.SysDRG ( + sysDRGBytes, +) where + +import Data.Word (Word8) +import Foreign.C.Types (CInt (..)) +import Foreign.Ptr (Ptr) + +import Crypto.Internal.ByteArray (ByteArray) +import qualified Crypto.Internal.ByteArray as B + +-- Safe, not unsafe: seeding can reach a getrandom(2) that blocks until the +-- kernel pool is ready, and an unsafe call that blocks holds the capability +-- it runs on. +foreign import ccall safe "crypton_sysdrg_bytes" + c_sysdrg_bytes :: Ptr Word8 -> CInt -> IO CInt + +-- | Draw bytes, or 'Nothing' if the generator cannot be seeded. +-- +-- It cannot be seeded where the system has no @getrandom(2)@ or +-- @getentropy(3)@; the caller falls back to +-- 'Crypto.Random.Entropy.getEntropy' there, which is the path this system +-- had before the generator existed. +sysDRGBytes :: ByteArray byteArray => Int -> IO (Maybe byteArray) +sysDRGBytes n = do + (got, out) <- B.allocRet n $ \ptr -> + fromIntegral `fmap` c_sysdrg_bytes ptr (fromIntegral n) + return $ if got == n then Just out else Nothing diff --git a/Crypto/Random/Types.hs b/Crypto/Random/Types.hs --- a/Crypto/Random/Types.hs +++ b/Crypto/Random/Types.hs @@ -13,8 +13,21 @@ import Crypto.Internal.ByteArray import Crypto.Random.Entropy +import Crypto.Random.SysDRG (sysDRGBytes) -- | A monad constraint that allows to generate random bytes +-- +-- Everything in this library that draws a key, a nonce or a signature's +-- randomness draws it through this class, and cannot tell a strong source +-- from a weak one. The two instances below are the ones to reach for: +-- @IO@ reads the system entropy source, and @MonadPseudoRandom@ runs a +-- 'DRG' seeded from it. +-- +-- An instance of your own is held to the same standard. A generator that +-- another party can predict, or that repeats, yields keys and signatures +-- that give away what they are meant to keep -- so the deliberately +-- repeatable instance that makes a test reproducible is not one to ship +-- with. class Monad m => MonadRandom m where getRandomBytes :: ByteArray byteArray => Int -> m byteArray @@ -23,8 +36,13 @@ -- | Generate N bytes of randomness from a DRG randomBytesGenerate :: ByteArray byteArray => Int -> gen -> (byteArray, gen) +-- | Through the generator of 'Crypto.Random.SysDRG': a ChaCha20 generator +-- per operating system thread, reseeded from the system. Where that +-- cannot be seeded -- a system with no @getrandom(2)@ or @getentropy(3)@, +-- which includes Windows for now -- this is the system entropy source +-- directly, as it was before the generator existed. instance MonadRandom IO where - getRandomBytes = getEntropy + getRandomBytes n = sysDRGBytes n >>= maybe (getEntropy n) return -- | A simple Monad class very similar to a State Monad -- with the state being a DRG. diff --git a/Crypto/System/CPU.hs b/Crypto/System/CPU.hs --- a/Crypto/System/CPU.hs +++ b/Crypto/System/CPU.hs @@ -1,6 +1,6 @@ {-# LANGUAGE CPP #-} -{-# LANGUAGE DeriveDataTypeable #-} {-# LANGUAGE ForeignFunctionInterface #-} +{-# LANGUAGE PatternSynonyms #-} -- | -- Module : Crypto.System.CPU @@ -11,57 +11,218 @@ -- -- Gives information about crypton runtime environment. module Crypto.System.CPU ( - ProcessorOption (..), + -- The names are bundled with the type rather than listed as + -- `pattern' exports, so an importer writes ProcessorOption (..), or + -- names the ones it wants, as it would for a type with constructors. + ProcessorOption ( + -- x86 + AESNI, + PCLMUL, + SSSE3, + AVX, + AVX2, + SHANI, + MOVBE, + ADX, + VAES, + VAES512, + AVX512IFMA, + -- AArch64 + NEON, + ARMAES, + ARMPMULL, + ARMSHA1, + ARMSHA2, + ARMSHA512, + -- PowerISA + PPCAES, + PPCVPMSUM + ), processorOptions, -) where -import Data.Data -import Data.List (findIndices) -#ifdef SUPPORT_RDRAND -import Data.Maybe (isJust) -#endif -import Data.Word (Word8) -import Foreign.Ptr -import Foreign.Storable + -- * Questions that do not name an architecture + hasAESAcceleration, + hasGHASHAcceleration, +) where +import Control.Monad (filterM) +import Data.List (sort) +import Data.Word (Word16) +import Foreign.C.Types (CInt (..), CUInt (..)) import Crypto.Internal.Compat -#ifdef SUPPORT_RDRAND -import Crypto.Random.Entropy.RDRand -import Crypto.Random.Entropy.Source -#endif +-- | A processor feature crypton looked for, and dispatches on where it +-- finds it. +-- +-- This is a number with names rather than a sum of constructors, and the +-- names are pattern synonyms with no @COMPLETE@ pragma, so a @case@ over +-- them needs a catch-all and a feature named in a later release breaks +-- nothing that compiled against this one. The same reason 'Show' is +-- written out below: a program built against an older crypton still says +-- something useful about a value from a newer one. +-- +-- The names are the processor's, not the operation's. 'AESNI' is x86's +-- and 'ARMAES' is AArch64's, and a machine reports only the ones it has; +-- ask 'hasAESAcceleration' if the question is whether AES is fast here. +-- +-- They are bundled with the type in the export list, so @ProcessorOption +-- (..)@ brings in all of them and naming one brings in that one, as for a +-- type with constructors. The constructor underneath is not exported: +-- these values say what the processor was found to have, and a caller has +-- nothing to build. +newtype ProcessorOption = ProcessorOption Word16 + deriving (Eq, Ord) --- | CPU options impacting cryptography implementation and library performance. -data ProcessorOption - = -- | Support for AES instructions, with flag @support_aesni@ - AESNI - | -- | Support for CLMUL instructions, with flag @support_pclmuldq@ - PCLMUL - | -- | Support for RDRAND instruction, with flag @support_rdrand@ - RDRAND - deriving (Show, Eq, Enum, Data) +-- | Support for AES instructions, with flag @support_aesni@. +pattern AESNI :: ProcessorOption +pattern AESNI = ProcessorOption 0 --- | Options which have been enabled at compile time and are supported by the --- current CPU. +-- | Support for CLMUL instructions, with flag @support_pclmuldq@. +pattern PCLMUL :: ProcessorOption +pattern PCLMUL = ProcessorOption 1 + +-- | Supplemental SSE3. +pattern SSSE3 :: ProcessorOption +pattern SSSE3 = ProcessorOption 3 + +-- | AVX, and an operating system that saves its registers. +pattern AVX :: ProcessorOption +pattern AVX = ProcessorOption 4 + +-- | AVX2, and an operating system that saves its registers. +pattern AVX2 :: ProcessorOption +pattern AVX2 = ProcessorOption 5 + +-- | The SHA extensions, @sha1rnds4@ and @sha256rnds2@ and their neighbours. +pattern SHANI :: ProcessorOption +pattern SHANI = ProcessorOption 6 + +-- | The byte-swapping load. +pattern MOVBE :: ProcessorOption +pattern MOVBE = ProcessorOption 7 + +-- | @MULX@, @ADCX@ and @ADOX@: the two independent carry chains. +pattern ADX :: ProcessorOption +pattern ADX = ProcessorOption 8 + +-- | The AES and carry-less multiply instructions in their 256-bit form. +pattern VAES :: ProcessorOption +pattern VAES = ProcessorOption 9 + +-- | The same pair in their 512-bit form. +pattern VAES512 :: ProcessorOption +pattern VAES512 = ProcessorOption 10 + +-- | The AVX-512 52-bit integer multiply-adds, with AVX-512 F and VL, and the +-- operating system saving their state: RSA's private-key operation uses them. +pattern AVX512IFMA :: ProcessorOption +pattern AVX512IFMA = ProcessorOption 19 + +-- | Advanced SIMD, which is not optional on AArch64. +pattern NEON :: ProcessorOption +pattern NEON = ProcessorOption 11 + +-- | The ARMv8 AES instructions. +pattern ARMAES :: ProcessorOption +pattern ARMAES = ProcessorOption 12 + +-- | @PMULL@, the ARMv8 carry-less multiply. +pattern ARMPMULL :: ProcessorOption +pattern ARMPMULL = ProcessorOption 13 + +-- | The ARMv8 SHA-1 instructions. +pattern ARMSHA1 :: ProcessorOption +pattern ARMSHA1 = ProcessorOption 14 + +-- | The ARMv8 SHA-256 instructions. +pattern ARMSHA2 :: ProcessorOption +pattern ARMSHA2 = ProcessorOption 15 + +-- | The ARMv8.2 SHA-512 instructions, which are optional where SHA-256's +-- are not. +pattern ARMSHA512 :: ProcessorOption +pattern ARMSHA512 = ProcessorOption 16 + +-- | Support for the PowerISA 2.07 vector AES instructions, which POWER8 was +-- the first to implement. +pattern PPCAES :: ProcessorOption +pattern PPCAES = ProcessorOption 17 + +-- | Support for @vpmsumd@, the vector carry-less multiply that came with +-- them, which is what makes GHASH fast. +pattern PPCVPMSUM :: ProcessorOption +pattern PPCVPMSUM = ProcessorOption 18 + +-- | Named where the name is known, numbered where it is not, so that a +-- binary built against an older crypton can still print a value a newer one +-- produced. +instance Show ProcessorOption where + show AESNI = "AESNI" + show PCLMUL = "PCLMUL" + show SSSE3 = "SSSE3" + show AVX = "AVX" + show AVX2 = "AVX2" + show SHANI = "SHANI" + show MOVBE = "MOVBE" + show ADX = "ADX" + show VAES = "VAES" + show VAES512 = "VAES512" + show AVX512IFMA = "AVX512IFMA" + show NEON = "NEON" + show ARMAES = "ARMAES" + show ARMPMULL = "ARMPMULL" + show ARMSHA1 = "ARMSHA1" + show ARMSHA2 = "ARMSHA2" + show ARMSHA512 = "ARMSHA512" + show PPCAES = "PPCAES" + show PPCVPMSUM = "PPCVPMSUM" + show (ProcessorOption n) = "ProcessorOption " ++ show n + +-- | Options which have been enabled at compile time and which this machine +-- says may be used: on x86-64 what the processor declares through CPUID, and +-- for the vector extensions only where the operating system saves their +-- state as well; elsewhere what the operating system reports of the +-- processor. +-- +-- That is what crypton itself decides by, and it is not always everything +-- the core underneath executes. A guest under a hypervisor may be shown an +-- older processor model than the host's, so that it can be moved to another +-- host -- and an instruction used because it happens to run would fail +-- there. So an option missing here means crypton will not use it, not that +-- the hardware lacks it. +-- +-- Sorted, and without repeats. A machine reports the names of its own +-- architecture only: an AArch64 processor with AES says 'ARMAES', not +-- 'AESNI', which it does not have. processorOptions :: [ProcessorOption] -processorOptions = unsafeDoIO $ do - p <- crypton_aes_cpu_init - options <- traverse (getOption p) aesOptions - rdrand <- hasRDRand - return (decodeOptions options ++ [RDRAND | rdrand]) +processorOptions = unsafeDoIO (sort <$> filterM askC allOptions) where - aesOptions = [AESNI .. PCLMUL] - getOption p = peekElemOff p . fromEnum - decodeOptions = map toEnum . findIndices (> 0) + -- 2 is not asked for and has no name: it was RDRAND, which crypton no + -- longer dispatches on. The number is left out rather than reused, so + -- that the others keep the values they had. + allOptions = [ProcessorOption n | n <- [0 .. 19], n /= 2] + askC (ProcessorOption n) = + (/= 0) <$> crypton_cpu_option (fromIntegral n) {-# NOINLINE processorOptions #-} -hasRDRand :: IO Bool -#ifdef SUPPORT_RDRAND -hasRDRand = fmap isJust getRDRand - where getRDRand = entropyOpen :: IO (Maybe RDRand) -#else -hasRDRand = return False -#endif +-- | Is there hardware AES on this machine? +-- +-- The instructions have different names on different architectures, and a +-- caller that wants to know whether AES-GCM will be fast wants this rather +-- than either name. +hasAESAcceleration :: Bool +hasAESAcceleration = + any + (`elem` processorOptions) + [AESNI, ARMAES, PPCAES] -foreign import ccall unsafe "crypton_aes_cpu_init" - crypton_aes_cpu_init :: IO (Ptr Word8) +-- | Is there a hardware carry-less multiply, which is what GHASH, and so +-- AES-GCM, spends its time in once AES itself is fast? +hasGHASHAcceleration :: Bool +hasGHASHAcceleration = + any + (`elem` processorOptions) + [PCLMUL, ARMPMULL, PPCVPMSUM] + +foreign import ccall unsafe "crypton_cpu_option" + crypton_cpu_option :: CUInt -> IO CInt diff --git a/Crypto/Tutorial.hs b/Crypto/Tutorial.hs --- a/Crypto/Tutorial.hs +++ b/Crypto/Tutorial.hs @@ -1,4 +1,7 @@ -- | Examples of how to use @crypton@. +-- +-- Every code block here is extracted and compiled against this version of +-- the library by @tests\/tutorial\/run.sh@, so what is written below builds. module Crypto.Tutorial ( -- * API design -- $api_design @@ -6,9 +9,21 @@ -- * Hash algorithms -- $hash_algorithms - -- * Symmetric block ciphers - -- $symmetric_block_ciphers + -- * Authenticated encryption + -- $authenticated_encryption + -- * Comparing secrets + -- $comparing_secrets + + -- * Password storage + -- $password_storage + + -- * Key derivation + -- $key_derivation + + -- * Digital signatures + -- $digital_signatures + -- * Combining primitives -- $combining_primitives ) where @@ -31,6 +46,14 @@ -- Error conditions are returned with data type 'Crypto.Error.CryptoFailable'. -- Functions in module "Crypto.Error" can convert those values to runtime -- exceptions, 'Maybe' or 'Either' values. +-- +-- Types that hold a secret do not print it. A private key's 'Show' renders +-- whatever is public and @\@ or @\@ for the rest, +-- because 'Show' is what @print@, @error@, an exception and a failing test +-- all reach for, and a key arriving in a log that way is an accident nobody +-- asked for. "Crypto.Debug" is how one is printed when printing it is what +-- was meant. Several of those types keep their bytes in +-- 'Data.ByteArray.ScrubbedBytes', which is wiped when it is collected. -- $hash_algorithms -- @@ -90,72 +113,258 @@ -- > hashMutableUpdate ctx ("dog" :: ByteString) -- > hashMutableFinalize ctx >>= print --- $symmetric_block_ciphers +-- $authenticated_encryption -- +-- Encrypting hides a message; it does not stop anyone changing it. Under a +-- counter or stream mode, flipping a bit of the ciphertext flips the same +-- bit of the plaintext, and the receiver has no way to tell. An AEAD mode +-- binds the message, and anything else named as associated data, to a short +-- authentication tag, and decrypting something that does not match that tag +-- returns nothing at all. +-- +-- That is the mode to use. The unauthenticated ones are in this library +-- for protocols that authenticate separately, not as a starting point. +-- -- > {-# LANGUAGE OverloadedStrings #-} --- > {-# LANGUAGE ScopedTypeVariables #-} --- > {-# LANGUAGE GADTs #-} -- > -- > import Crypto.Cipher.AES (AES256) --- > import Crypto.Cipher.Types (BlockCipher(..), Cipher(..), nullIV, KeySizeSpecifier(..), IV, makeIV) --- > import Crypto.Error (CryptoFailable(..), CryptoError(..)) +-- > import Crypto.Cipher.Types +-- > ( AEADMode (AEAD_GCM) +-- > , AuthTag +-- > , BlockCipher (aeadInit) +-- > , Cipher (cipherInit, cipherKeySize) +-- > , KeySizeSpecifier (..) +-- > , aeadSimpleDecrypt +-- > , aeadSimpleEncrypt +-- > ) +-- > import Crypto.Error (CryptoError, eitherCryptoError) +-- > import qualified Crypto.Random.Types as CRT -- > +-- > import Data.ByteArray (ByteArray, ByteArrayAccess, ScrubbedBytes) +-- > import Data.ByteString (ByteString) +-- > +-- > -- | A key of the length the cipher asks for, rather than a length +-- > -- written out here. ScrubbedBytes rather than ByteString, so that it +-- > -- is wiped when it is collected and does not print. +-- > genSecretKey :: (Cipher c, CRT.MonadRandom m) => c -> m ScrubbedBytes +-- > genSecretKey c = CRT.getRandomBytes (longest (cipherKeySize c)) +-- > where +-- > longest (KeySizeFixed n) = n +-- > longest (KeySizeRange _ n) = n +-- > longest (KeySizeEnum ns) = maximum ns +-- > +-- > -- | A fresh nonce for every message. GCM must never see one twice +-- > -- under the same key: a repeat does not just expose those two messages, +-- > -- it hands over the key that authenticates all of them. Twelve random +-- > -- bytes, sent along with the ciphertext. +-- > genNonce :: CRT.MonadRandom m => m ByteString +-- > genNonce = CRT.getRandomBytes 12 +-- > +-- > -- | Encrypt and authenticate. The associated data is authenticated but +-- > -- not encrypted: it is for what the receiver can already see and must +-- > -- not have had altered, such as a header or an address. +-- > encrypt +-- > :: (ByteArray key, ByteArrayAccess nonce, ByteArrayAccess aad, ByteArray ba) +-- > => key -> nonce -> aad -> ba -> Either CryptoError (AuthTag, ba) +-- > encrypt key nonce aad plaintext = do +-- > cipher <- eitherCryptoError (cipherInit key) :: Either CryptoError AES256 +-- > aead <- eitherCryptoError (aeadInit AEAD_GCM cipher nonce) +-- > return (aeadSimpleEncrypt aead aad plaintext 16) +-- > +-- > -- | And back, with two different failures. Left is this code used +-- > -- wrongly -- a key of the wrong length, a mode the cipher has not got. +-- > -- Nothing is a message that is not the one that was sent; it carries no +-- > -- plaintext and says nothing about which byte was wrong, both of which +-- > -- are the point. +-- > decrypt +-- > :: (ByteArray key, ByteArrayAccess nonce, ByteArrayAccess aad, ByteArray ba) +-- > => key -> nonce -> aad -> AuthTag -> ba -> Either CryptoError (Maybe ba) +-- > decrypt key nonce aad tag ciphertext = do +-- > cipher <- eitherCryptoError (cipherInit key) :: Either CryptoError AES256 +-- > aead <- eitherCryptoError (aeadInit AEAD_GCM cipher nonce) +-- > return (aeadSimpleDecrypt aead aad ciphertext tag) +-- > +-- > exampleAES256GCM :: ByteString -> IO () +-- > exampleAES256GCM msg = do +-- > key <- genSecretKey (undefined :: AES256) +-- > nonce <- genNonce +-- > let aad = "to: alice" :: ByteString +-- > case encrypt key nonce aad msg of +-- > Left err -> error (show err) +-- > Right (tag, ciphertext) -> do +-- > putStrLn $ "ciphertext: " ++ show ciphertext +-- > putStrLn $ " tag: " ++ show tag +-- > putStrLn $ " recovered: " +-- > ++ show (decrypt key nonce aad tag ciphertext) +-- > -- The same bytes and the same tag, with one thing changed +-- > -- that was never encrypted: Right Nothing. +-- > putStrLn $ "redirected: " +-- > ++ show (decrypt key nonce ("to: eve" :: ByteString) tag ciphertext) +-- +-- The two functions above work for any cipher that has an AEAD mode; what +-- changes is the mode given to 'Crypto.Cipher.Types.aeadInit'. +-- "Crypto.Cipher.ChaChaPoly1305" is the one to prefer on a machine with no +-- AES instructions, and "Crypto.Cipher.AESGCMSIV" is the one that survives +-- a repeated nonce, at the price of needing the whole message before it can +-- begin. + +-- $comparing_secrets +-- +-- Comparing two byte strings with '==' stops at the first byte that +-- differs, so how long it takes says where that byte was. Against an +-- authentication tag that is the whole secret: someone who can send a guess +-- and time the answer finds the first byte in a few hundred tries, then the +-- second, and has a tag that was supposed to cost 2^128 in a few thousand. +-- +-- crypton's own authentication types already compare in constant time, so +-- for those there is nothing to do: 'Crypto.MAC.HMAC.HMAC', +-- 'Crypto.MAC.CMAC.CMAC', 'Crypto.MAC.Poly1305.Auth' and +-- 'Crypto.Cipher.Types.AuthTag' have an 'Eq' that looks at every byte +-- whatever it finds. What needs care is a tag that arrives as bytes, and +-- 'Data.ByteArray.constEq' is the comparison for it. +-- +-- > {-# LANGUAGE OverloadedStrings #-} +-- > +-- > import Crypto.Hash.Algorithms (SHA256) +-- > import Crypto.MAC.HMAC (HMAC, hmac) +-- > +-- > import qualified Data.ByteArray as BA +-- > import Data.ByteString (ByteString) +-- > +-- > -- | The tag came off the wire as bytes, so it is compared as bytes. +-- > authentic :: ByteString -> ByteString -> ByteString -> Bool +-- > authentic key message tag = +-- > BA.constEq tag (hmac key message :: HMAC SHA256) +-- > +-- > -- | Once it has been parsed into the library's own type, (==) is +-- > -- already the constant-time comparison. +-- > authentic' :: ByteString -> ByteString -> HMAC SHA256 -> Bool +-- > authentic' key message tag = tag == hmac key message + +-- $password_storage +-- +-- A password is not a key. It is short and it is guessable, and whoever +-- takes the database can try every likely one without being watched. What +-- answers that is a function that is deliberately expensive to compute, and +-- crypton has four: "Crypto.KDF.BCrypt", "Crypto.KDF.Scrypt", +-- "Crypto.KDF.Argon2" and "Crypto.KDF.PBKDF2". A plain hash is not one of +-- them, however many times it is applied by hand. +-- +-- bcrypt leaves the least to get wrong, because the record it returns +-- carries the salt and the cost inside it: +-- +-- > import Crypto.KDF.BCrypt (hashPassword, validatePassword) +-- > +-- > import Data.ByteString (ByteString) +-- > +-- > -- | What goes in the database. The salt is drawn inside and ends up in +-- > -- the result, so two accounts with the same password do not look alike. +-- > register :: ByteString -> IO ByteString +-- > register password = hashPassword 12 password +-- > +-- > -- | And what is checked against it. The cost comes out of the stored +-- > -- record, so raising it for new accounts leaves the old ones working. +-- > login :: ByteString -> ByteString -> Bool +-- > login password stored = validatePassword password stored +-- +-- Argon2 is the stronger choice and the one to pick for something new: it +-- asks for memory as well as time, which is what takes the advantage away +-- from the hardware that bcrypt's small working set leaves room for. +-- Nothing is encoded for the caller, though -- the salt and the options are +-- theirs to store, and without all three the hash cannot be recomputed when +-- the user comes back. +-- +-- > import Crypto.Error (CryptoFailable) +-- > import qualified Crypto.KDF.Argon2 as Argon2 -- > import qualified Crypto.Random.Types as CRT -- > --- > import Data.ByteArray (ByteArray) -- > import Data.ByteString (ByteString) -- > --- > -- | Not required, but most general implementation --- > data Key c a where --- > Key :: (BlockCipher c, ByteArray a) => a -> Key c a +-- > -- | Argon2id, which is the variant to prefer: it resists both a machine +-- > -- built to guess and a process watching the cache. +-- > options :: Argon2.Options +-- > options = Argon2.defaultOptions{Argon2.variant = Argon2.Argon2id} -- > --- > -- | Generates a string of bytes (key) of a specific length for a given block cipher --- > genSecretKey :: forall m c a. (CRT.MonadRandom m, BlockCipher c, ByteArray a) => c -> Int -> m (Key c a) --- > genSecretKey _ = fmap Key . CRT.getRandomBytes +-- > newSalt :: CRT.MonadRandom m => m ByteString +-- > newSalt = CRT.getRandomBytes 16 -- > --- > -- | Generate a random initialization vector for a given block cipher --- > genRandomIV :: forall m c. (CRT.MonadRandom m, BlockCipher c) => c -> m (Maybe (IV c)) --- > genRandomIV _ = do --- > bytes :: ByteString <- CRT.getRandomBytes $ blockSize (undefined :: c) --- > return $ makeIV bytes +-- > derive :: ByteString -> ByteString -> CryptoFailable ByteString +-- > derive salt password = Argon2.hash options password salt 32 + +-- $key_derivation +-- +-- HKDF turns one secret into as many keys as a protocol needs. It is for +-- material that is already unguessable -- what comes out of a +-- Diffie-Hellman, or a key already agreed -- and it is deliberately cheap, +-- which is exactly what makes it the wrong thing for a password. Those go +-- to the section above. +-- +-- The info string is what keeps the outputs independent: the same secret +-- with a different info gives an unrelated key, so each use of a secret +-- names itself there. +-- +-- > {-# LANGUAGE OverloadedStrings #-} -- > --- > -- | Initialize a block cipher --- > initCipher :: (BlockCipher c, ByteArray a) => Key c a -> Either CryptoError c --- > initCipher (Key k) = case cipherInit k of --- > CryptoFailed e -> Left e --- > CryptoPassed a -> Right a +-- > import Crypto.Hash.Algorithms (SHA256) +-- > import qualified Crypto.KDF.HKDF as HKDF -- > --- > encrypt :: (BlockCipher c, ByteArray a) => Key c a -> IV c -> a -> Either CryptoError a --- > encrypt secretKey initIV msg = --- > case initCipher secretKey of --- > Left e -> Left e --- > Right c -> Right $ ctrCombine c initIV msg +-- > import Data.ByteArray (ScrubbedBytes) +-- > import Data.ByteString (ByteString) -- > --- > decrypt :: (BlockCipher c, ByteArray a) => Key c a -> IV c -> a -> Either CryptoError a --- > decrypt = encrypt +-- > -- | One shared secret in, two unrelated keys out. +-- > directionKeys :: ByteString -> ByteString -> (ScrubbedBytes, ScrubbedBytes) +-- > directionKeys salt shared = (keyFor "client write", keyFor "server write") +-- > where +-- > prk = HKDF.extract salt shared :: HKDF.PRK SHA256 +-- > keyFor info = HKDF.expand prk (info :: ByteString) 32 + +-- $digital_signatures +-- +-- Ed25519 is the one to reach for. The keys are thirty-two bytes, there is +-- nothing to choose and nothing to encode, and signing needs no randomness, +-- so it cannot be ruined by a bad source of it. "Crypto.PubKey.Ed448" is +-- the same shape at a larger size, "Crypto.PubKey.ECDSA" and +-- "Crypto.PubKey.RSA.PSS" are there for protocols that ask for them, and +-- "Crypto.PubKey.MLDSA" is the post-quantum one. +-- +-- > {-# LANGUAGE OverloadedStrings #-} -- > --- > exampleAES256 :: ByteString -> IO () --- > exampleAES256 msg = do --- > -- secret key needs 256 bits (32 * 8) --- > secretKey <- genSecretKey (undefined :: AES256) 32 --- > mInitIV <- genRandomIV (undefined :: AES256) --- > case mInitIV of --- > Nothing -> error "Failed to generate and initialization vector." --- > Just initIV -> do --- > let encryptedMsg = encrypt secretKey initIV msg --- > decryptedMsg = decrypt secretKey initIV =<< encryptedMsg --- > case (,) <$> encryptedMsg <*> decryptedMsg of --- > Left err -> error $ show err --- > Right (eMsg, dMsg) -> do --- > putStrLn $ "Original Message: " ++ show msg --- > putStrLn $ "Message after encryption: " ++ show eMsg --- > putStrLn $ "Message after decryption: " ++ show dMsg +-- > import Crypto.Error (throwCryptoError) +-- > import qualified Crypto.PubKey.Ed25519 as Ed25519 +-- > +-- > import qualified Data.ByteArray as BA +-- > import Data.ByteString (ByteString) +-- > +-- > exampleEd25519 :: ByteString -> IO () +-- > exampleEd25519 msg = do +-- > sk <- Ed25519.generateSecretKey +-- > let pk = Ed25519.toPublic sk +-- > sig = Ed25519.sign sk pk msg +-- > print (Ed25519.verify pk msg sig) +-- > -- The same signature against a message one byte longer: False. +-- > print (Ed25519.verify pk (msg <> "!") sig) +-- > +-- > -- | A secret key on its way to storage. Printing one does not reveal +-- > -- it -- see the first section -- so this is the way out. +-- > store :: Ed25519.SecretKey -> ByteString +-- > store = BA.convert +-- > +-- > -- | And the way back in, which is checked, because the bytes read from +-- > -- a file may be anything at all. +-- > load :: ByteString -> Ed25519.SecretKey +-- > load = throwCryptoError . Ed25519.secretKey -- $combining_primitives -- -- This example shows how to use Curve25519, XSalsa and Poly1305 primitives to -- emulate NaCl's @crypto_box@ construct. -- +-- It is here to show how the pieces fit together, not as something to +-- deploy. An authenticated encryption scheme assembled by hand is the kind +-- of code that is wrong in ways no test notices; for actual use, +-- "Crypto.Cipher.ChaChaPoly1305" does this job with the mistakes already +-- made. +-- -- > import qualified Data.ByteArray as BA -- > import Data.ByteString (ByteString) -- > import qualified Data.ByteString as B @@ -167,6 +376,9 @@ -- > -- > -- | Build a @crypto_box@ packet encrypting the specified content with a -- > -- 192-bit nonce, receiver public key and sender private key. +-- > crypto_box +-- > :: ByteString -> ByteString -> X25519.PublicKey -> X25519.SecretKey +-- > -> ByteString -- > crypto_box content nonce pk sk = BA.convert tag `B.append` c -- > where -- > zero = B.replicate 16 0 @@ -181,6 +393,9 @@ -- > -- > -- | Try to open a @crypto_box@ packet and recover the content using the -- > -- 192-bit nonce, sender public key and receiver private key. +-- > crypto_box_open +-- > :: ByteString -> ByteString -> X25519.PublicKey -> X25519.SecretKey +-- > -> Maybe ByteString -- > crypto_box_open packet nonce pk sk -- > | B.length packet < 16 = Nothing -- > | BA.constEq tag' tag = Just content diff --git a/LICENSE b/LICENSE --- a/LICENSE +++ b/LICENSE @@ -1,4 +1,5 @@ Copyright (c) 2006-2015 Vincent Hanquez +Copyright (c) 2023-2026 Kazu Yamamoto All rights reserved. diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -3,24 +3,98 @@ crypton ========== -Crypton is a fork from cryptonite with the original author's permission. +`crypton` is a fork from `cryptonite` with the original author's permission. -Crypton is a haskell repository of cryptographic primitives. Each crypto -algorithm has specificities that are hard to wrap in common APIs and types, -so instead of trying to provide a common ground for algorithms, this package -provides a non-consistent low-level API. -If you have no idea what you're doing, please do not use this directly. -Instead, rely on higher level protocols or implementations. +`crypton` is a low-level cryptography library. To achieve high +performance, it utilizes C and assembly language to define FFI +bindings, structuring them in a way that makes them easy to use. + +Side channels +------------- + +### AES + +AES is where this matters most, and which implementation runs is decided at +runtime from what the processor has. + +On x86-64 with AES-NI and carry-less multiply, on AArch64 and on 32-bit ARM +with the ARMv8 cryptographic extension, and on ppc64le with the vector +instructions PowerISA 2.07 brought to POWER8, AES and GHASH are instructions +rather than software. crypton's AES and AES-GCM then make no branch and no +memory access that depends on the key or on the data: the secrets stay in +vector registers and never reach one a branch can test. Every x86-64 part +since about 2010 and every AArch64 part in ordinary use has these, and the +ppc64le path is little-endian only. + +Where none of them is there, crypton computes AES and GHASH without a table. +The portable AES is bitsliced -- the S-box is boolean algebra over four +blocks held across eight 64-bit words -- and the GF(2^128) multiply is built +from shifts, masks and integer multiplies. Neither looks anything up, so +neither derives an address from a secret, and **the portable path is constant +time as well**. The code is BearSSL's, under MIT, in `cbits/bearssl`. + +The constant-time harness in `cbits/tests/ct` runs the same driver against +the portable implementation and against each architecture's instructions, and +every one of them is required to report nothing at all. + +So the answer does not depend on the machine any more, and what is left to +ask is whether AES is fast on it. `hasAESAcceleration` and +`hasGHASHAcceleration` answer that without naming an architecture. +`processorOptions` says what was found, in the processor's own names -- +`AESNI` and `PCLMUL` on x86, `ARMAES` and `ARMPMULL` on AArch64 and 32-bit +ARM, `PPCAES` and `PPCVPMSUM` on ppc64le -- along with everything else it was +asked about, which is unrelated to this. + + ghci> import Crypto.System.CPU + ghci> hasAESAcceleration + True + ghci> processorOptions -- on an x86-64 machine + [AESNI,PCLMUL,SSSE3,AVX,AVX2,SHANI,MOVBE,ADX,VAES] + ghci> processorOptions -- and on an AArch64 one + [NEON,ARMAES,ARMPMULL,ARMSHA1,ARMSHA2,ARMSHA512] + +### RSA + +RSA is the other place to know about, and there the choice is the caller's. +The private key operations in `Crypto.PubKey.RSA.PKCS15`, `.OAEP` and `.PSS` +take a `Maybe Blinder`, and `Nothing` is no harder to write than the safe +form: + + decrypt :: Maybe Blinder -> PrivateKey -> ByteString -> ... + decryptSafer :: MonadRandom m => PrivateKey -> ByteString -> m ... + +The exponent itself is not what is at risk. `expSafe` keeps the *value* of an +exponent out of the work it does, so the private exponent does not leak +through the exponentiation. What a blinder covers is the other side: without +one, the operation runs on the ciphertext the caller was handed, so how long +it takes depends on a number an attacker may have chosen and can vary. That +is what a remote timing attack on RSA needs. With a blinder the input is +multiplied by a random value first and the result divided out afterwards, so +the timing carries nothing an attacker can steer. + +`decryptSafer` and `signSafer` generate the blinder themselves and are the +ones to reach for. Pass `Nothing` only where the input is not attacker +controlled and you have decided that it is not. + +The RSA rows in the tables below are the unblinded path. A fresh blinder +costs one more exponentiation, by the public exponent, which is the cheap +direction: measured on the M4, signing goes from about 316 to about 331 +microseconds, under five per cent. `signSafer` and `decryptSafer` pay less +than that from 2.2.4: they keep the blinder of a key they have seen and +square it for the next use, as OpenSSL does, drawing a fresh one every 32 +uses. `Crypto.PubKey.RSA.Blinded` gives a caller the same thing to hold, a +`BlindedKey` passed along from one operation to the next. + Performance ----------- -The algorithms a TLS connection uses, measured against the previous release -and against OpenSSL on the same machine. Throughput is over 16 KiB messages; -the public key operations are one operation each; every figure is the best of -several runs, and crypton and OpenSSL are run alternately so that neither gets -the quieter machine. +The algorithms a TLS connection uses, measured against the last release +before the rewrite and against OpenSSL on the same machine. Throughput is +over 16 KiB messages; the public key operations are one operation each; every +figure is the best of several runs, and crypton and OpenSSL are run +alternately so that neither gets the quieter machine. Bulk encryption and hashing are measured through crypton's C layer, as `openssl speed` measures OpenSSL's. The public key operations are measured @@ -28,125 +102,209 @@ is what a program actually calls; the Haskell layer adds well under a microsecond, which the X25519 and ECDH P-256 rows confirm by agreeing with a C-level measurement to within a percent. Both releases of crypton are built -the same way -- `-optc-O3`, which is what both of them ask for -- and each -column of a table comes from one run on the machine named above it. +the same way -- `-optc-O3`, which is what each asks for -- and by +`cabal build`, since a copy of the sources compiled by hand does not measure +what a program linking the library gets, and leaves out whole implementations +without saying so. Each column of a table comes from one run on the machine +named above it. ### x86-64 -An AMD EPYC 7763, which has AES-NI, PCLMULQDQ, AVX2, ADX and the SHA -extensions, against OpenSSL 3.0.13. +An AMD EPYC 7763, which has AES-NI, PCLMULQDQ, AVX2, ADX, VAES, VPCLMULQDQ +and the SHA extensions, against OpenSSL 4.0.3. Throughput in MB/s, **higher is better**: -| | crypton 1.1.5 | crypton 2.0.0 | OpenSSL | 2.0.0 / OpenSSL | +| | crypton 1.1.5 | crypton 2.2.4 | OpenSSL | 2.2.4 / OpenSSL | | --- | ---: | ---: | ---: | ---: | -| AES-128-GCM | 1331 | 4118 | 4264 | 0.97 | -| AES-256-GCM | 1090 | 3810 | 3951 | 0.96 | -| ChaCha20-Poly1305 | 398 | 2195 | 2191 | 1.00 | -| SHA-1 | 738 | 1678 | 1672 | 1.00 | -| SHA-256 | 286 | 1585 | 1570 | 1.01 | -| SHA-512 | 448 | 769 | 746 | 1.03 | -| SHA3-256 | 109 | 421 | 426 | 0.99 | +| AES-128-GCM | 1335 | **6041** | 4066 | 1.49 | +| AES-256-GCM | 1093 | **5492** | 3766 | 1.46 | +| ChaCha20-Poly1305 | 399 | 2195 | 2222 | 0.99 | +| SHA-1 | 727 | 1663 | 1672 | 0.99 | +| SHA-256 | 289 | 1585 | 1576 | 1.01 | +| SHA-512 | 448 | 770 | 749 | 1.03 | +| SHA3-256 | 109 | 421 | 429 | 0.98 | -Time per operation in microseconds, **lower is better** -- so the last column -divides OpenSSL's time by crypton's, and is again better the larger it is: +Time per operation in microseconds, **lower is better**: -| | crypton 1.1.5 | crypton 2.0.0 | OpenSSL | OpenSSL / 2.0.0 | +| | crypton 1.1.5 | crypton 2.2.4 | OpenSSL | OpenSSL / 2.2.4 | | --- | ---: | ---: | ---: | ---: | -| X25519 | 43.57 | 43.52 | 36.58 | 0.84 | -| ECDH P-256 | 163.8 | 163.7 | 52.36 | 0.32 | -| ECDH P-384 | 2241 | 1101 | 857.1 | 0.78 | -| Ed25519 sign | 28.52 | 28.25 | 43.49 | 1.54 | -| Ed25519 verify | 46.06 | 46.16 | 119.3 | 2.59 | -| ECDSA P-256 sign | 76.04 | 75.02 | 22.91 | 0.31 | -| ECDSA P-256 verify | 229.2 | 228.5 | 67.98 | 0.30 | -| ECDSA P-384 sign | 2271 | 387.6 | 904.3 | 2.33 | -| ECDSA P-384 verify | 2667 | 1493 | 746.2 | 0.50 | -| RSA-2048 sign/decrypt | 759.8 | 1311 | 660.1 | 0.50 | -| RSA-2048 verify/encrypt | 31.62 | 28.59 | 18.63 | 0.65 | +| X25519 | 44.38 | 28.29 | 36.61 | 1.29 | +| ECDH P-256 | 164.0 | 51.31 | 51.70 | 1.01 | +| ECDH P-384 | 2242 | **164.3** | 836.1 | 5.09 | +| Ed25519 sign | 43.86 | 18.50 | 33.74 | 1.82 | +| Ed25519 verify | 48.30 | 48.43 | 109.8 | 2.27 | +| ECDSA P-256 sign | 81.80 | 19.72 | 21.84 | 1.11 | +| ECDSA P-256 verify | 231.8 | 70.47 | 67.55 | 0.96 | +| ECDSA P-384 sign | 2267 | **305.2** | 879.5 | 2.88 | +| ECDSA P-384 verify | 2698 | **473.6** | 725.2 | 1.53 | +| RSA-2048 sign/decrypt | 760.1 | 612.3 | 659.6 | 1.08 | +| RSA-2048 verify/encrypt | 33.61 | 23.50 | 18.88 | 0.80 | +| RSA-3072 sign/decrypt | 2499 | 2324 | 2012 | 0.87 | +| RSA-3072 verify/encrypt | 66.80 | 61.91 | 40.14 | 0.65 | ### AArch64 An Apple M4, which has the AES, PMULL, SHA-1, SHA-2, SHA-512 and SHA-3 -instructions, against OpenSSL 3.6.4. +instructions, against OpenSSL 4.0.3. Throughput in MB/s, **higher is better**: -| | crypton 1.1.5 | crypton 2.0.0 | OpenSSL | 2.0.0 / OpenSSL | +| | crypton 1.1.5 | crypton 2.2.4 | OpenSSL | 2.2.4 / OpenSSL | | --- | ---: | ---: | ---: | ---: | -| AES-128-GCM | 126 | 8702 | 10719 | 0.81 | -| AES-256-GCM | 98 | 7648 | 9154 | 0.84 | -| ChaCha20-Poly1305 | 758 | 2323 | 2244 | 1.04 | -| SHA-1 | 1199 | 3380 | 3346 | 1.01 | -| SHA-256 | 467 | 3394 | 3352 | 1.01 | -| SHA-512 | 723 | 1868 | 1851 | 1.01 | -| SHA3-256 | 548 | 1091 | 1054 | 1.04 | +| AES-128-GCM | 131 | **12052** | 10922 | 1.10 | +| AES-256-GCM | 99 | **10192** | 9208 | 1.11 | +| ChaCha20-Poly1305 | 783 | 2342 | 2261 | 1.04 | +| SHA-1 | 1234 | 3383 | 3347 | 1.01 | +| SHA-256 | 486 | 3392 | 3354 | 1.01 | +| SHA-512 | 742 | 1869 | 1860 | 1.00 | +| SHA3-256 | 552 | 1096 | 1092 | 1.00 | -Time per operation in microseconds, **lower is better**; the last column again -divides OpenSSL's time by crypton's: +Time per operation in microseconds, **lower is better**: -| | crypton 1.1.5 | crypton 2.0.0 | OpenSSL | OpenSSL / 2.0.0 | +| | crypton 1.1.5 | crypton 2.2.4 | OpenSSL | OpenSSL / 2.2.4 | | --- | ---: | ---: | ---: | ---: | -| X25519 | 18.44 | 18.41 | 18.41 | 1.00 | -| ECDH P-256 | 69.46 | 56.24 | 24.68 | 0.44 | -| ECDH P-384 | 3252 | 511.1 | 379.7 | 0.74 | -| Ed25519 sign | 13.75 | 13.14 | 15.90 | 1.21 | -| Ed25519 verify | 18.17 | 18.04 | 39.27 | 2.18 | -| ECDSA P-256 sign | 32.56 | 27.91 | 11.05 | 0.40 | -| ECDSA P-256 verify | 96.50 | 80.19 | 32.82 | 0.41 | -| ECDSA P-384 sign | 3219 | 169.3 | 403.4 | 2.38 | -| ECDSA P-384 verify | 3807 | 688.1 | 335.1 | 0.49 | -| RSA-2048 sign/decrypt | 451.8 | 605.4 | 325.0 | 0.54 | -| RSA-2048 verify/encrypt | 18.32 | 15.28 | 8.50 | 0.56 | +| X25519 | 17.92 | **12.02** | 15.32 | 1.28 | +| ECDH P-256 | 67.50 | **19.05** | 24.36 | 1.28 | +| ECDH P-384 | 3256 | **71.78** | 373.1 | 5.20 | +| Ed25519 sign | 13.38 | **7.717** | 13.31 | 1.72 | +| Ed25519 verify | 18.32 | 18.18 | 34.77 | 1.91 | +| ECDSA P-256 sign | 31.08 | **6.500** | 10.66 | 1.64 | +| ECDSA P-256 verify | 94.27 | **26.58** | 32.25 | 1.21 | +| ECDSA P-384 sign | 3225 | **124.0** | 394.2 | 3.18 | +| ECDSA P-384 verify | 3866 | **203.4** | 332.3 | 1.63 | +| RSA-2048 sign/decrypt | 451.3 | 318.4 | 321.5 | 1.01 | +| RSA-2048 verify/encrypt | 18.15 | 11.12 | 8.405 | 0.76 | +| RSA-3072 sign/decrypt | 1473 | 937.4 | 931.1 | 0.99 | +| RSA-3072 verify/encrypt | 47.95 | 43.63 | 17.89 | 0.41 | ### What the numbers say -1.1.5 had no AArch64 code of its own at all, which is why AES-GCM there is -sixty-nine times what it was. On x86-64 it had AES-NI and nothing else. The -curves over a prime field other than P-256 moved from Haskell `Integer` -arithmetic into C, which is the nineteenfold change in ECDSA P-384 signing on -the M4. X25519 and Ed25519 are unchanged between the two releases, and the -rows say so: where they differ by half a percent, that is the measurement and -not the code. P-256 is unchanged on x86-64 and a fifth faster on AArch64, -which is the paragraph below. +There are two changes behind the 1.1.5 column and the 2.2.4 one, not a +single steady improvement. -Where crypton is behind, it is behind for three separate reasons. +The first, in 2.0.0, was a rewrite: the bulk algorithms moved into C, the +curves other than P-256 moved out of Haskell `Integer` arithmetic, and +everything that touches a secret was made to take the same time whatever the +secret is. 1.1.5 had no AArch64 code of its own at all, which is why AES-GCM +there is close to a hundred times what it was, and on x86-64 it had AES-NI +and nothing else. -*P-256.* crypton's field arithmetic is C where OpenSSL's is hand-written -assembly, and that is what is left of the difference: the two differ by about -the same factor on every P-256 row, and nothing above the field -- a wider -window, a different addition formula, another field representation -- recovers -a useful part of it. +The second, from 2.1.0 onwards, is assembly, for the operations where C +cannot reach. Which of the two a row owes its gain to is not the same +everywhere: ECDSA P-384 signing took nineteenfold from the rewrite and a +further fifth from the assembly, while X25519 waited for the assembly +entirely and ECDH P-384 is almost all of it. -The AArch64 rows are better than the x86-64 ones because of where a field -multiplication's latency goes. It ends in a carry chain the width of the -number, and the curve arithmetic has independent products that could cover -that chain -- but only if the compiler inlines the reduction instead of -calling it, since a call is a fence. Asking it to costs code and pays where -there are registers enough to hold two chains at once: a quarter on AArch64, -where there are thirty-one, and nothing on x86-64, where there are fifteen and -the same request makes it slower. So x86-64 is left to the compiler's own -judgement and stays at 0.3. +Most of that assembly is not crypton's. The prime curves, the inverse modulo +a group order, X25519, and RSA's Montgomery multiplication on x86-64 go +through [s2n-bignum](https://github.com/awslabs/s2n-bignum), vendored in +`cbits/s2n`. Every routine in it carries a machine-checked proof in +HOL-Light that it computes what it says, and is written in a constant-time +style. It is `Apache-2.0 OR ISC OR MIT-0`, and crypton takes it under ISC. -*RSA signing.* 2.0.0 is slower than 1.1.5 here on purpose. Its modular -exponentiation no longer indexes a table with the bits of the exponent, and -hiding the exponent is what the difference buys. What is left of the gap -against OpenSSL is the Montgomery multiplication, which is assembly there and -C here. +That licence is why any of this was possible. The obvious assembly to reach +for is OpenSSL's and BoringSSL's `ecp_nistz256`, and it cannot be used here: +it is Apache-2.0 only, and Intel and CloudFlare hold copyright in it besides +OpenSSL, so nobody is in a position to relicense it. -*The AVX-512 instructions.* Neither machine above has them. On one that does --- an EPYC 9V74, measured the same way -- OpenSSL uses them for AES-GCM and -ChaCha20 and reaches 12003 and 3789 MB/s, against 4745 and 2372 for crypton, -whose vendored assembly is generated without them. Those ratios are 0.40 and -0.63 rather than 0.97 and 1.00. Nothing else in either table moves by more -than a few percent between the two processors. +crypton is behind in a few rows now: RSA verification on both architectures, +and RSA-3072 signing and ECDSA P-256 verification on x86-64. The AES-GCM rows +were the other half of this section until 2.1.5; they are ahead on both +machines now, and what the instructions do is still worth setting out. +*RSA.* 2.0.0 made signing slower than 1.1.5 on purpose: its modular +exponentiation stopped indexing a table with the bits of the exponent, and +hiding the exponent is what the difference bought. 2.2.4 has repaid that +cost on both machines, by different means. On x86-64 s2n-bignum's +Montgomery multiplication is twice the C's, because the C cannot form the two +carry chains `ADCX` and `ADOX` give, and 2.2.4 signs in less than 1.1.5 +took, at 3072 bits as well as 2048. On AArch64 s2n-bignum's routines measure +no better than the C, so the gain there is the C's own: the multiplication +and squaring for each key size are generated and unrolled, with the +reduction folded into each column as in Koç, Acar and Kaliski's FIPS method, +and signing is level with OpenSSL. The measurements are in +[#275](https://github.com/kazu-yamamoto/crypton/issues/275). + +On processors with AVX-512 IFMA, Intel Xeons from Ice Lake and AMD from Zen 4, +2.2.4 computes the two halves of the CRT together in 52-bit digits, Gueron and +Krasnov's method and the one OpenSSL takes there. Neither machine above has +IFMA, so neither table shows it. + +Verification's exponent is 65537, seventeen bits, so there is little +exponentiation to speed up and the work around it counts. 2.2.4 keeps R^2, +the Montgomery constant of a modulus, for the moduli it has seen, which is +most of what the RSA-2048 verification rows gained. + +*The wide AES instructions.* `VAES` and `VPCLMULQDQ` do two blocks where +`AES-NI` and `PCLMULQDQ` do one, and four in their 512-bit form. crypton uses +the 256-bit form where the processor has it, which is Zen 3 and Ice Lake +onwards, and the 512-bit form where that is worth having, which is Ice Lake and +Zen 5 onwards. There was nothing to borrow: the wide AES-GCM in OpenSSL, +BoringSSL and AWS-LC is Apache-2.0 and s2n-bignum has no GCM, so both files are +crypton's own. + +Having the 256-bit one is where the 1.49 in the x86-64 table comes from, and +it is narrower than it sounds. The EPYC 7763 is Zen 3: VAES and VPCLMULQDQ, +no AVX-512. OpenSSL's x86-64 AES-GCM is `aesni-gcm-x86_64.pl`, which is +128-bit -- its `vaesenc`s are the VEX encoding of `AESENC` on `xmm`, and +there is not one `ymm` in the file -- or `aes-gcm-avx512.pl`, which wants +`AVX512VAES`. There is no rung between them, so on this processor OpenSSL +takes a block at a time where crypton takes two. The same idea as theirs, +one step further down the feature ladder; not a better one. + +The 512-bit path arrived after 2.1.2, so it is in the 2.2.4 column -- but +neither machine in the tables above has AVX-512, so neither column shows it. +On the runners that do, measured over 16 KiB in MB/s: an EPYC 9V45 (Zen 5) +goes from 9616 to 14268 with it, a Xeon 6973P-C from 8095 to 9848, a Xeon +8573C from 6983 to 8447. OpenSSL on those machines is ahead still -- 25760 +on the first of them -- because it interleaves the GHASH with the AES where +crypton does them in turn. Zen 4 keeps the 256-bit path: its 512-bit +instructions are two passes through a 256-bit datapath, so the wider encoding +buys nothing there and costs a little. + +AArch64 has no counterpart to any of these: one AES block and one GHASH +multiplication at a time is all the instruction set offers. Its AES-GCM +rows were 0.85 and 0.87 until 2.1.5, for that reason. What closed it was +not width but the GHASH's representation -- H is twisted once at key setup +so that GCM's bit reflection is already undone, which turns a reduction of +some twenty-five shifts and XORs into two PMULL and six EOR and makes +Karatsuba worth taking. The scheme is ARM's, from the BSD-3-Clause part of +[AArch64cryptolib](https://github.com/ARM-software/AArch64cryptolib), +written out in crypton's own intrinsics. The AES there is ahead of +OpenSSL's and always was; it was the GHASH beside it that was behind. + One row wants a word of its own: crypton's `Ed25519.sign` derives the public key from the secret key every time it signs, so that a caller who passes a public key that does not match cannot be made to leak the private one. That -costs a second scalar multiplication, which OpenSSL's signing does not pay. +costs a second scalar multiplication, which OpenSSL's signing does not pay -- +and the row is still 1.72 on AArch64 and 1.82 on x86-64, so the safety is had +for nothing here rather than paid for. +Ed25519 verification wanted one too, for an unhappier reason, and 2.2.1 +settles it. On x86-64 the row was about eight per cent slower from 2.1.8 +onwards: 2.1.5, 2.1.6 and 2.1.7 all measure near 48 microseconds there, +2.1.8, 2.1.10 and 2.2.0 all measure near 52, and 2.2.1 is near 48 again. +Nothing in Ed25519 ever changed -- no commit touches `cbits/ed25519` across +any of it, and AArch64 never moved, the compiled `ed25519.o` there being +byte for byte the same object either side of 2.1.8. What moved was a line +in this package's cabal file. The ML-KEM and ML-DSA backends that 2.1.8 +vendored want `-mavx2 -mbmi2`, and they were given as `cc-options` inside an +`if arch(x86_64)` block -- but cc-options in a conditional belong to the +component and not to the block, so every C file in the library was compiled +with AVX2 allowed, `cbits/ed25519` among them, and what the compiler made of +it that way is slower on this machine than what it replaced. Signing, which +is the same file and a different routine, is where it always was. 2.2.1 +narrowed the flags to the assembly that asked for them, which is also what +stopped a binary built on one x86-64 machine from dying on another, and the +row came back. + SHA-1 is in the tables because a number of protocols and file formats still ask for it, not because it is a good choice for anything new. The algorithms that nothing should ask for any more -- MD5, 3DES, RC4, CBC mode -- are left out. + +ML-KEM and ML-DSA are left out too, though 2.1.8 brought both and they are +fast -- `mlkem-native` and `mldsa-native`, from the PQ Code Package. No +comparison is reported here: the implementations to measure against are +still developing. diff --git a/benchs/Bench.hs b/benchs/Bench.hs --- a/benchs/Bench.hs +++ b/benchs/Bench.hs @@ -195,7 +195,9 @@ where cp k (ini, plain) = let iniState = - throwCryptoError $ CP.initialize k (throwCryptoError $ CP.nonce12 nonce12) + CP.initialize + (throwCryptoError $ CP.key k) + (throwCryptoError $ CP.nonce12 nonce12) afterAAD = CP.finalizeAAD (CP.appendAAD ini iniState) (out, afterEncrypt) = CP.encrypt plain afterAAD outtag = CP.finalize afterEncrypt diff --git a/cbits/LICENSE.go b/cbits/LICENSE.go new file mode 100644 --- /dev/null +++ b/cbits/LICENSE.go @@ -0,0 +1,38 @@ +The arrangement of the AArch64 multiply-accumulate loop in +cbits/crypton_bignum.h -- four limbs to an iteration, the low halves of the +products and the high halves accumulated in two chains -- follows +addMulVVWx in Go's crypto/internal/fips140/bigmod/nat_arm64.s, written out +in the assembler that file's compiler speaks. Go is at +https://github.com/golang/go and carries the licence below. That file's +own header reads "Copyright 2013 The Go Authors. All rights reserved. Use +of this source code is governed by a BSD-style license that can be found in +the LICENSE file", and the LICENSE file it means is this one. + + +Copyright 2009 The Go Authors. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google LLC nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/cbits/aes/LICENSE.fusion b/cbits/aes/LICENSE.fusion new file mode 100644 --- /dev/null +++ b/cbits/aes/LICENSE.fusion @@ -0,0 +1,29 @@ +Parts of cbits/aes/gcm_fused_x86.c follow the AES-GCM implementation in +picotls, lib/fusion.c, which is under the MIT license reproduced below. +The design is described by its author at + + http://blog.kazuhooku.com/2020/06/quicaes-gcm-12.html + http://blog.kazuhooku.com/2020/06/quicaes-gcm-22.html + +and the source is at https://github.com/h2o/picotls. + + +Copyright (c) 2020-2022 Fastly, Kazuho Oku + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to +deal in the Software without restriction, including without limitation the +rights to use, copy, modify, merge, publish, distribute, sublicense, and/or +sell copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +IN THE SOFTWARE. diff --git a/cbits/aes/armv8.c b/cbits/aes/armv8.c --- a/cbits/aes/armv8.c +++ b/cbits/aes/armv8.c @@ -20,12 +20,9 @@ #include #include #include -#if defined(__linux__) -#include -#include -#endif #include "crypton_aes.h" #include "crypton_bitfn.h" +#include "crypton_cpu.h" /* * The AES and PMULL instructions are extensions, so a translation unit @@ -39,11 +36,7 @@ * * "+crypto" rather than "crypto": GCC rejects the latter. */ -#ifdef WITH_TARGET_ATTRIBUTES -#define TARGET_ARMV8_CRYPTO __attribute__((target("+crypto"))) -#else -#define TARGET_ARMV8_CRYPTO -#endif +#include "crypton_armv8_target.h" /* forward round keys: nbr + 1 of them, written by the generic key expansion */ #define FWD(key) ((const uint8_t *) (key)->data) @@ -70,14 +63,14 @@ * instructions exist to remove -- but a key schedule is the one thing an * attacker most wants and it costs little to keep it out of the cache. */ -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO static uint32x4_t sub_word(uint32x4_t w) { return vreinterpretq_u32_u8( vaeseq_u8(vreinterpretq_u8_u32(w), vdupq_n_u8(0))); } -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO static uint32x4_t sub_rot_word(uint32x4_t w) { const uint8x16_t s = vreinterpretq_u8_u32(sub_word(w)); @@ -85,7 +78,7 @@ return vreinterpretq_u32_u8(vextq_u8(s, s, 1)); } -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void crypton_aes_armv8_init(aes_key *key, uint8_t *origkey, uint8_t size) { /* 2^0 .. 2^9 in GF(2^8), which is as far as any key size reaches */ @@ -127,191 +120,194 @@ } /* - * Whether the extensions are actually present. - * - * They are mandatory on Apple silicon, and on other AArch64 systems the - * kernel reports them through the auxiliary vector. A system without them - * keeps the generic implementation. + * Whether the extensions are actually present. crypton_cpu.c asks the + * system once, in the one place that knows how each system answers; a + * processor without them keeps the generic implementation. */ int crypton_aes_armv8_available(void) { -#if defined(__APPLE__) - return 1; -#elif defined(__linux__) - return (getauxval(AT_HWCAP) & HWCAP_AES) != 0; -#else - return 0; -#endif + return (crypton_arm_features() & CRYPTON_ARM_AES) != 0; } /* * GHASH using PMULL, the AArch64 counterpart to PCLMULQDQ. * - * This is a transliteration of gfmul_pclmuldq in x86ni.c rather than a fresh - * formulation: that code is already pinned by the GCM known-answer tests, and - * every operation it uses has a direct NEON equivalent, so translating it is - * easier to check than reasoning about a new reduction from scratch. + * Not the transliteration of gfmul_pclmuldq in x86ni.c this used to be. The + * x86 formulation keeps H in the order GCM writes it and pays, at the end of + * every batch, a reduction that first has to undo GCM's bit reflection: some + * twenty-five shifts and XORs, and a batch of eight costs it once. * - * _mm_shuffle_epi8 with a reversing mask -> vrev64q_u8 then vextq_u8 - * _mm_clmulepi64_si128 -> vmull_p64 / vmull_high_p64 - * _mm_slli_si128 / _mm_srli_si128 -> vextq_u8 against zero - * _mm_slli_epi32 / _mm_srli_epi32 -> vshlq_n_u32 / vshrq_n_u32 + * Instead H is twisted once, at key setup, so that the reflection is already + * undone and a reversed-polynomial multiply lands in the right place. Two + * things follow. The reduction becomes two PMULL against 0xC2000..0 and six + * EOR, a third of what it was. And because nothing has to be byte-reversed + * back and forth, Karatsuba pays: three PMULL a block rather than four, with + * the middle terms accumulated in a third register and tidied up once per + * batch. + * + * crypton tried Karatsuba in the old representation and measured it 1.6 per + * cent slower -- the saved PMULL did not cover the extra EOR when the + * reduction stayed as expensive as it was. It is the pair that pays. + * Measured over 16 KiB messages, this against the old code: + * + * Apple M4 Neoverse N2 + * AES-128-GCM 1.30 1.25 + * AES-192-GCM 1.22 1.25 + * AES-256-GCM 1.19 1.24 + * + * The scheme is ARM's, from the 'big' AES-GCM kernel of + * https://github.com/ARM-software/AArch64cryptolib, which is BSD-3-Clause, + * (c) 2018-2019 ARM Limited. Their kernels under AArch64cryptolib_opt_bigger + * are faster again and are NOT under that licence, whatever the repository's + * LICENSE.md says; nothing here comes from those files. + * + * The table holds the twisted powers H^1 .. H^8 at htable[0 .. 7] and the + * Karatsuba half of each -- its high 64 bits XOR its low -- in the first + * eight bytes of htable[8 .. 15]. The running tag is kept the way GCM + * writes it at every boundary, and swapped into the internal form on the way + * in and out, which is two instructions. */ -/* reverse all 16 bytes */ -TARGET_ARMV8_CRYPTO -static inline uint8x16_t bswap128(uint8x16_t v) -{ - return vextq_u8(vrev64q_u8(v), vrev64q_u8(v), 8); -} - -/* shift the whole register left by n bytes, as _mm_slli_si128 does */ -#define SHIFT_LEFT_BYTES(v, n) vextq_u8(vdupq_n_u8(0), (v), 16 - (n)) -/* and right, as _mm_srli_si128 does */ -#define SHIFT_RIGHT_BYTES(v, n) vextq_u8((v), vdupq_n_u8(0), (n)) - -#define SHL32(v, n) vreinterpretq_u8_u32(vshlq_n_u32(vreinterpretq_u32_u8(v), (n))) -#define SHR32(v, n) vreinterpretq_u8_u32(vshrq_n_u32(vreinterpretq_u32_u8(v), (n))) - -TARGET_ARMV8_CRYPTO -static inline uint8x16_t clmul_ll(uint8x16_t a, uint8x16_t b) -{ - return vreinterpretq_u8_p128(vmull_p64( - (poly64_t) vgetq_lane_u64(vreinterpretq_u64_u8(a), 0), - (poly64_t) vgetq_lane_u64(vreinterpretq_u64_u8(b), 0))); -} +#define GHASH_MODC ((poly64_t) 0xC200000000000000ul) -TARGET_ARMV8_CRYPTO -static inline uint8x16_t clmul_lh(uint8x16_t a, uint8x16_t b) +/* the internal accumulator form, and back again: its own inverse */ +CRYPTON_TARGET_ARMV8_CRYPTO +static inline uint8x16_t ghash_swap(uint8x16_t t) { - return vreinterpretq_u8_p128(vmull_p64( - (poly64_t) vgetq_lane_u64(vreinterpretq_u64_u8(a), 0), - (poly64_t) vgetq_lane_u64(vreinterpretq_u64_u8(b), 1))); + t = vrev64q_u8(t); + return vextq_u8(t, t, 8); } -TARGET_ARMV8_CRYPTO -static inline uint8x16_t clmul_hl(uint8x16_t a, uint8x16_t b) +/* the high and low halves XORed together, which is what Karatsuba wants */ +CRYPTON_TARGET_ARMV8_CRYPTO +static inline poly64_t ghash_karat(poly64x2_t v) { - return vreinterpretq_u8_p128(vmull_p64( - (poly64_t) vgetq_lane_u64(vreinterpretq_u64_u8(a), 1), - (poly64_t) vgetq_lane_u64(vreinterpretq_u64_u8(b), 0))); + return (poly64_t) veor_u64(vget_high_u64(vreinterpretq_u64_p64(v)), + vget_low_u64(vreinterpretq_u64_p64(v))); } -TARGET_ARMV8_CRYPTO -static inline uint8x16_t clmul_hh(uint8x16_t a, uint8x16_t b) -{ - return vreinterpretq_u8_p128(vmull_high_p64( - vreinterpretq_p64_u8(a), vreinterpretq_p64_u8(b))); -} +/* the twisted power H^(i+1) */ +#define GHASH_POW(ht, i) \ + vreinterpretq_p64_u8(vld1q_u8((const uint8_t *) &(ht)[i])) +/* and its Karatsuba half */ +#define GHASH_KARAT(ht, i) \ + ((poly64_t) vgetq_lane_u64( \ + vreinterpretq_u64_u8(vld1q_u8((const uint8_t *) &(ht)[8 + (i)])), 0)) /* - * The 256-bit carry-less product of a (normal byte order) and b (already - * reversed, as it sits in the table), before the reflection fixup and the - * reduction. Split out from the reduction because both of those are linear - * over XOR: several products can be added together and fixed up just once, - * which is what gf_mul4 below does. + * One block's three partial products, XORed into the accumulators. b is + * already in the internal form; hp and hk are the power it is to meet. */ -TARGET_ARMV8_CRYPTO -static inline void clmul_pmull(uint8x16_t a, uint8x16_t b, - uint8x16_t *lo, uint8x16_t *hi) -{ - uint8x16_t t3, t4, t5, t6; - - a = bswap128(a); - - t3 = clmul_ll(a, b); - t4 = clmul_lh(a, b); - t5 = clmul_hl(a, b); - t6 = clmul_hh(a, b); - - t4 = veorq_u8(t4, t5); - t5 = SHIFT_LEFT_BYTES(t4, 8); - t4 = SHIFT_RIGHT_BYTES(t4, 8); - - *lo = veorq_u8(t3, t5); - *hi = veorq_u8(t6, t4); -} +#define GHASH_MUL(b, hp, hk, H, M, L) \ + do { \ + poly64x2_t b__ = (b); \ + poly64x2_t hp__ = (hp); \ + (H) = veorq_u64((H), vreinterpretq_u64_p128( \ + vmull_high_p64(b__, hp__))); \ + (L) = veorq_u64((L), vreinterpretq_u64_p128(vmull_p64( \ + (poly64_t) vgetq_lane_u64(vreinterpretq_u64_p64(b__), 0), \ + (poly64_t) vgetq_lane_u64(vreinterpretq_u64_p64(hp__), 0)))); \ + (M) = veorq_u64((M), vreinterpretq_u64_p128( \ + vmull_p64(ghash_karat(b__), (hk)))); \ + } while (0) -/* Shift the 256-bit product left by one to undo GCM's bit reflection, then - * reduce modulo the GCM polynomial. This is the expensive half. */ -TARGET_ARMV8_CRYPTO -static inline uint8x16_t gfred_pmull(uint8x16_t t3, uint8x16_t t6) +/* + * Finish the Karatsuba -- the middle accumulator still holds only the + * (ah^al)(bh^bl) terms and wants the other two taken out of it -- and reduce + * the 256 bits modulo the GCM polynomial. The result is in internal form. + */ +CRYPTON_TARGET_ARMV8_CRYPTO +static inline uint64x2_t ghash_reduce(uint64x2_t H, uint64x2_t M, uint64x2_t L) { - uint8x16_t t2, t4, t5, t7, t8, t9; - - t7 = SHR32(t3, 31); - t8 = SHR32(t6, 31); - t3 = SHL32(t3, 1); - t6 = SHL32(t6, 1); - - t9 = SHIFT_RIGHT_BYTES(t7, 12); - t8 = SHIFT_LEFT_BYTES(t8, 4); - t7 = SHIFT_LEFT_BYTES(t7, 4); - t3 = vorrq_u8(t3, t7); - t6 = vorrq_u8(t6, t8); - t6 = vorrq_u8(t6, t9); - - t7 = SHL32(t3, 31); - t8 = SHL32(t3, 30); - t9 = SHL32(t3, 25); + uint64x2_t t; - t7 = veorq_u8(t7, t8); - t7 = veorq_u8(t7, t9); - t8 = SHIFT_RIGHT_BYTES(t7, 4); - t7 = SHIFT_LEFT_BYTES(t7, 12); - t3 = veorq_u8(t3, t7); + M = veorq_u64(M, H); + M = veorq_u64(M, L); - t2 = SHR32(t3, 1); - t4 = SHR32(t3, 2); - t5 = SHR32(t3, 7); - t2 = veorq_u8(t2, t4); - t2 = veorq_u8(t2, t5); - t2 = veorq_u8(t2, t8); - t3 = veorq_u8(t3, t2); - t6 = veorq_u8(t6, t3); + t = vreinterpretq_u64_p128(vmull_p64( + (poly64_t) vgetq_lane_u64(H, 0), GHASH_MODC)); + H = vreinterpretq_u64_u8(vextq_u8(vreinterpretq_u8_u64(H), + vreinterpretq_u8_u64(H), 8)); + M = veorq_u64(M, t); + M = veorq_u64(M, H); - return bswap128(t6); + t = vreinterpretq_u64_p128(vmull_p64( + (poly64_t) vgetq_lane_u64(M, 0), GHASH_MODC)); + M = vreinterpretq_u64_u8(vextq_u8(vreinterpretq_u8_u64(M), + vreinterpretq_u8_u64(M), 8)); + L = veorq_u64(L, t); + return veorq_u64(L, M); } -TARGET_ARMV8_CRYPTO -static uint8x16_t gfmul_pmull(uint8x16_t a, const uint8_t *htable) +/* a single block against H^1, accumulator in internal form */ +CRYPTON_TARGET_ARMV8_CRYPTO +static inline uint64x2_t ghash_one(uint64x2_t acc, uint8x16_t blk, + const block128 *ht) { - uint8x16_t lo, hi; + uint64x2_t H = vdupq_n_u64(0), M = H, L = H; + poly64x2_t b; - clmul_pmull(a, vld1q_u8(htable), &lo, &hi); - return gfred_pmull(lo, hi); + acc = vreinterpretq_u64_u8(vextq_u8(vreinterpretq_u8_u64(acc), + vreinterpretq_u8_u64(acc), 8)); + b = vreinterpretq_p64_u64(veorq_u64( + vreinterpretq_u64_u8(vrev64q_u8(blk)), acc)); + GHASH_MUL(b, GHASH_POW(ht, 0), GHASH_KARAT(ht, 0), H, M, L); + return ghash_reduce(H, M, L); } /* - * With PMULL there is no 4-bit table to fill: H goes in at index 0, byte - * reversed, so that gfmul_pmull does not have to swap it every time. This - * mirrors crypton_aesni_hinit_pclmul. + * Twist H and raise it to the powers a batch needs. * - * Indices 1..7 get H^2 .. H^8, which is what lets a group of blocks fold - * into one reduction: gf_mul4 uses the first four, the GCM loop all eight. - * The table has sixteen slots, so they are free. + * The twist is a shift left by one with 0xC2000..01 folded back in when a + * bit falls off the top -- the same correction the old reduction applied to + * every product, done once here instead. Each further power is one multiply + * in the twisted domain; the result comes out of ghash_reduce with its + * halves swapped, which a batch undoes on the way in, so a stored power has + * to be swapped back. */ -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void crypton_aes_armv8_hinit_pmull(block128 *htable, const block128 *h) { - uint8x16_t p; + uint8x16_t hk = vrev64q_u8(vld1q_u8((const uint8_t *) h)); + uint64x2_t shl = vshlq_n_u64(vreinterpretq_u64_u8(hk), 1); + uint64x2_t shr = vreinterpretq_u64_s64( + vshrq_n_s64(vreinterpretq_s64_u8(hk), 63)); + uint8x16_t mask = vextq_u8(vreinterpretq_u8_u64(shr), + vreinterpretq_u8_u64(shr), 12); + uint64x2_t tc = vdupq_n_u64(0); + poly64x2_t base, p; int i; - htable[0].q[0] = bitfn_swap64(h->q[1]); - htable[0].q[1] = bitfn_swap64(h->q[0]); + tc = vsetq_lane_u64(0xC200000000000001ul, tc, 0); + tc = vsetq_lane_u64(1, tc, 1); + tc = vandq_u64(vreinterpretq_u64_u8(mask), tc); + base = vreinterpretq_p64_u64(veorq_u64(tc, shl)); - p = vld1q_u8((const uint8_t *) h); - for (i = 1; i < 8; i++) { - p = gfmul_pmull(p, (const uint8_t *) &htable[0]); - vst1q_u8((uint8_t *) &htable[i], bswap128(p)); + p = base; + for (i = 0; i < 8; i++) { + uint64x2_t H = vdupq_n_u64(0), M = H, L = H, r; + + vst1q_u8((uint8_t *) &htable[i], + vreinterpretq_u8_p64(p)); + vst1q_u8((uint8_t *) &htable[8 + i], + vreinterpretq_u8_u64( + vdupq_n_u64((uint64_t) ghash_karat(p)))); + + GHASH_MUL(p, base, ghash_karat(base), H, M, L); + r = ghash_reduce(H, M, L); + p = vreinterpretq_p64_u8(vextq_u8(vreinterpretq_u8_u64(r), + vreinterpretq_u8_u64(r), 8)); } } -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void crypton_aes_armv8_gf_mul_pmull(block128 *a, const block128 *htable) { - vst1q_u8((uint8_t *) a, - gfmul_pmull(vld1q_u8((const uint8_t *) a), (const uint8_t *) htable)); + uint64x2_t acc = vreinterpretq_u64_u8( + ghash_swap(vld1q_u8((const uint8_t *) a))); + + acc = ghash_one(acc, vdupq_n_u8(0), htable); + vst1q_u8((uint8_t *) a, ghash_swap(vreinterpretq_u8_u64(acc))); } /* @@ -320,36 +316,38 @@ * four products can be summed first and reduced once, which is where the * time goes. Aggregated reduction, from the Intel GCM paper. */ -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void crypton_aes_armv8_gf_mul4_pmull(block128 *a, const block128 *blocks, const block128 *htable) { - uint8x16_t lo, hi, l, h; + uint64x2_t acc = vreinterpretq_u64_u8( + ghash_swap(vld1q_u8((const uint8_t *) a))); + uint64x2_t H = vdupq_n_u64(0), M = H, L = H; + poly64x2_t b; int i; - clmul_pmull(veorq_u8(vld1q_u8((const uint8_t *) a), - vld1q_u8((const uint8_t *) &blocks[0])), - vld1q_u8((const uint8_t *) &htable[3]), &lo, &hi); + acc = vreinterpretq_u64_u8(vextq_u8(vreinterpretq_u8_u64(acc), + vreinterpretq_u8_u64(acc), 8)); + b = vreinterpretq_p64_u64(veorq_u64( + vreinterpretq_u64_u8(vrev64q_u8( + vld1q_u8((const uint8_t *) &blocks[0]))), acc)); + GHASH_MUL(b, GHASH_POW(htable, 3), GHASH_KARAT(htable, 3), H, M, L); for (i = 1; i < 4; i++) { - clmul_pmull(vld1q_u8((const uint8_t *) &blocks[i]), - vld1q_u8((const uint8_t *) &htable[3 - i]), &l, &h); - lo = veorq_u8(lo, l); - hi = veorq_u8(hi, h); + b = vreinterpretq_p64_u8(vrev64q_u8( + vld1q_u8((const uint8_t *) &blocks[i]))); + GHASH_MUL(b, GHASH_POW(htable, 3 - i), + GHASH_KARAT(htable, 3 - i), H, M, L); } - vst1q_u8((uint8_t *) a, gfred_pmull(lo, hi)); + acc = ghash_reduce(H, M, L); + vst1q_u8((uint8_t *) a, ghash_swap(vreinterpretq_u8_u64(acc))); } + int crypton_aes_armv8_pmull_available(void) { -#if defined(__APPLE__) - return 1; -#elif defined(__linux__) - return (getauxval(AT_HWCAP) & HWCAP_PMULL) != 0; -#else - return 0; -#endif + return (crypton_arm_features() & CRYPTON_ARM_PMULL) != 0; } /* @@ -359,7 +357,7 @@ * the high half, and fold the bit that leaves the top back in as 0x87. The * block is little-endian, so lane 0 is the low half. */ -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO static inline uint8x16_t gfmulx_neon(uint8x16_t v) { const uint64x2_t x = vreinterpretq_u64_u8(v); @@ -396,3 +394,59 @@ #include #undef SIZED #undef NBR + +/* + * The fused entry point, over the three key sizes. Each was generated with + * its round count fixed, which is what lets the eight chains stay in + * registers; the choice between them is made once per message here. + */ +CRYPTON_TARGET_ARMV8_CRYPTO +void crypton_aes_armv8_gcm_fused(uint8_t *out, const block128 *ht, + aes_key *key, const uint8_t *nonce, + const uint8_t *aad, uint32_t aadlen, + const uint8_t *in, uint32_t inlen, + uint32_t taglen, aes_key *hpkey, + uint32_t sampleoff, uint8_t *mask) +{ + switch (key->strength) { + case 0: + crypton_aes_armv8_gcm_fused128(out, ht, key, nonce, aad, aadlen, + in, inlen, taglen, hpkey, + sampleoff, mask); + break; + case 1: + crypton_aes_armv8_gcm_fused192(out, ht, key, nonce, aad, aadlen, + in, inlen, taglen, hpkey, + sampleoff, mask); + break; + default: + crypton_aes_armv8_gcm_fused256(out, ht, key, nonce, aad, aadlen, + in, inlen, taglen, hpkey, + sampleoff, mask); + break; + } +} + +CRYPTON_TARGET_ARMV8_CRYPTO +int crypton_aes_armv8_gcm_fused_dec(uint8_t *out, const block128 *ht, + aes_key *key, const uint8_t *nonce, + const uint8_t *aad, uint32_t aadlen, + const uint8_t *in, uint32_t inlen, + const uint8_t *tag, uint32_t taglen, + uint8_t *outtag) +{ + switch (key->strength) { + case 0: + return crypton_aes_armv8_gcm_fused_dec128(out, ht, key, nonce, + aad, aadlen, in, inlen, + tag, taglen, outtag); + case 1: + return crypton_aes_armv8_gcm_fused_dec192(out, ht, key, nonce, + aad, aadlen, in, inlen, + tag, taglen, outtag); + default: + return crypton_aes_armv8_gcm_fused_dec256(out, ht, key, nonce, + aad, aadlen, in, inlen, + tag, taglen, outtag); + } +} diff --git a/cbits/aes/armv8_impl.c b/cbits/aes/armv8_impl.c --- a/cbits/aes/armv8_impl.c +++ b/cbits/aes/armv8_impl.c @@ -25,8 +25,6 @@ #define EACH1(m) m(0) #define EACH8(m) m(0) m(1) m(2) m(3) m(4) m(5) m(6) m(7) -/* the blocks after the first; GHASH folds block 0 in with the tag */ -#define EACH7(m) m(1) m(2) m(3) m(4) m(5) m(6) m(7) #define LOAD_IN(i) s[i] = vld1q_u8((const uint8_t *) (input + (i))); #define STORE_OUT(i) vst1q_u8((uint8_t *) (output + (i)), s[i]); @@ -74,7 +72,7 @@ } \ } while (0) -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_encrypt_block)(aes_block *output, aes_key *key, aes_block *input) { const uint8_t *rk = FWD(key); @@ -85,7 +83,7 @@ EACH1(STORE_OUT); } -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_decrypt_block)(aes_block *output, aes_key *key, aes_block *input) { const uint8_t *fwd = FWD(key); @@ -97,7 +95,7 @@ EACH1(STORE_OUT); } -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_encrypt_ecb)(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks) { const uint8_t *rk = FWD(key); @@ -115,7 +113,7 @@ } } -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_decrypt_ecb)(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks) { const uint8_t *fwd = FWD(key); @@ -136,7 +134,7 @@ /* CBC encryption chains, so there is nothing to interleave. It still gains * the round keys staying put. */ -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_encrypt_cbc)(aes_block *output, aes_key *key, aes_block *_iv, aes_block *input, uint32_t nb_blocks) { const uint8_t *rk = FWD(key); @@ -158,7 +156,7 @@ #define CBC_KEEP(i) c[(i) + 1] = s[i]; #define CBC_XOR(i) vst1q_u8((uint8_t *) (output + (i)), veorq_u8(s[i], c[i])); -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_decrypt_cbc)(aes_block *output, aes_key *key, aes_block *_iv, aes_block *input, uint32_t nb_blocks) { const uint8_t *fwd = FWD(key); @@ -192,7 +190,7 @@ #define CTR_XOR(i) vst1q_u8(output + 16 * (i), \ veorq_u8(s[i], vld1q_u8(input + 16 * (i)))); -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_encrypt_ctr)(uint8_t *output, aes_key *key, aes_block *iv, uint8_t *input, uint32_t len) { const uint8_t *rk = FWD(key); @@ -262,6 +260,47 @@ * * GCM's counter is the low 32 bits only and wraps there, so unlike CTR * there is no carry to chase: the top twelve bytes never move. + * + * What this loop is short of is not overlap but instructions. A group of + * eight blocks compiles to 383 of them on an Apple M4, and only 152 are the + * cipher -- eighty AESE and seventy-two AESMC. The rest is the GHASH and + * the counters. Measured against OpenSSL on the same machine, 16 KiB + * messages under AES-128: + * + * crypton, this loop with the GHASH taken out 19883 MB/s + * OpenSSL, AES-128-CTR 17176 + * OpenSSL, AES-128-GCM 9869 + * crypton, AES-128-GCM 8641 + * + * The cipher here is ahead of OpenSSL's; all of the 0.87 is the GHASH. + * Four ways of closing it were tried and every one measured worse, so they + * are written down here rather than tried again. Each was checked to give + * the same ciphertext and tag as this code for three key sizes and + * thirty-five lengths, encrypt and decrypt, before being timed: + * + * the GHASH held back a group and spread through the next -1.3% + * group's AES rounds, so that the two do not queue to -3.6% + * Karatsuba: three multiplications for the two halves -1.6% + * instead of four + * the same with every product on a lane the instruction 0.0% + * reaches, which does remove sixteen fmov a group + * the same again with the H powers' halves added together -3% + * already, in the spare half of htable + * + * The first fails because there was nothing to gain: a group's AES depends + * on nothing in the group before it, so a wide out-of-order core already + * runs the two together, and holding a group back only adds copies. The + * rest fail for one reason -- none of them makes the loop shorter. + * Karatsuba buys a PMULL for two EOR and an EXT, and the folded table turns + * sixteen `dup` into sixteen `ld1r` and sixteen more address adds. + * + * That last sentence used to end by saying a count which does come down + * wants the data laid out differently. It does, and since the GHASH was + * rewritten against a twisted H -- see cbits/aes/armv8.c -- it is laid out + * differently, so the figures above are what the *previous* GHASH gave. + * Karatsuba pays now that the reduction it has to carry is a third of what + * it was; the other three are untried in the new representation and the + * first of them has no more reason to work than it had. */ #define GCM_CTR(i) s[i] = vreinterpretq_u8_u32(vsetq_lane_u32(cpu_to_be32(c + 1 + (i)), base, 3)); #define GCM_ENC(i) { const uint8x16_t m_ = vld1q_u8(input + 16 * (i)); \ @@ -270,27 +309,33 @@ #define GCM_DEC(i) { const uint8x16_t m_ = vld1q_u8(input + 16 * (i)); \ vst1q_u8(output + 16 * (i), veorq_u8(s[i], m_)); \ s[i] = m_; } -#define GCM_GHASH(i) { uint8x16_t l_, h_; \ - clmul_pmull(s[i], vld1q_u8((const uint8_t *) &ht[WAY - 1 - (i)]), \ - &l_, &h_); \ - glo = veorq_u8(glo, l_); ghi = veorq_u8(ghi, h_); } +#define GCM_GHASH(i) \ + { \ + poly64x2_t b_ = vreinterpretq_p64_u8(vrev64q_u8(s[i])); \ + if ((i) == 0) \ + b_ = vreinterpretq_p64_u64(veorq_u64( \ + vreinterpretq_u64_p64(b_), acc)); \ + GHASH_MUL(b_, GHASH_POW(ht, WAY - 1 - (i)), \ + GHASH_KARAT(ht, WAY - 1 - (i)), gH, gM, gL); \ + } /* the eight blocks now in s[] are the ciphertext; fold them into the tag */ #define GCM_FOLD() \ do { \ - uint8x16_t glo, ghi; \ - clmul_pmull(veorq_u8(tag, s[0]), \ - vld1q_u8((const uint8_t *) &ht[WAY - 1]), \ - &glo, &ghi); \ - EACH7(GCM_GHASH) \ - tag = gfred_pmull(glo, ghi); \ + uint64x2_t gH = vdupq_n_u64(0), gM = gH, gL = gH; \ + acc = vreinterpretq_u64_u8( \ + vextq_u8(vreinterpretq_u8_u64(acc), \ + vreinterpretq_u8_u64(acc), 8)); \ + EACH8(GCM_GHASH) \ + acc = ghash_reduce(gH, gM, gL); \ } while (0) #define GCM_PROLOGUE \ const uint8_t *rk = FWD(key); \ const block128 *ht = gcm->htable; \ uint8x16_t s[WAY]; \ - uint8x16_t tag = vld1q_u8((const uint8_t *) &gcm->tag); \ + uint64x2_t acc = vreinterpretq_u64_u8( \ + ghash_swap(vld1q_u8((const uint8_t *) &gcm->tag))); \ uint32_t c = be32_to_cpu(gcm->civ.d[3]); \ uint32x4_t base = vreinterpretq_u32_u8(vld1q_u8((const uint8_t *) &gcm->civ)) @@ -303,16 +348,17 @@ ENC_ROUNDS(EACH1); \ s[0] = veorq_u8(s[0], m_); \ (store_c); \ - tag = gfmul_pmull(veorq_u8(tag, (ghash_of)), (const uint8_t *) ht); \ + acc = ghash_one(acc, (ghash_of), ht); \ } while (0) #define GCM_EPILOGUE \ do { \ gcm->civ.d[3] = cpu_to_be32(c); \ - vst1q_u8((uint8_t *) &gcm->tag, tag); \ + vst1q_u8((uint8_t *) &gcm->tag, \ + ghash_swap(vreinterpretq_u8_u64(acc))); \ } while (0) -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_gcm_encrypt)(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length) { GCM_PROLOGUE; @@ -343,13 +389,12 @@ block128_zero(&m); for (i = 0; i < length; i++) output[i] = m.b[i] = o.b[i]; - tag = gfmul_pmull(veorq_u8(tag, vld1q_u8((const uint8_t *) &m)), - (const uint8_t *) ht); + acc = ghash_one(acc, vld1q_u8((const uint8_t *) &m), ht); } GCM_EPILOGUE; } -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_gcm_decrypt)(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length) { GCM_PROLOGUE; @@ -381,8 +426,7 @@ vst1q_u8((uint8_t *) &o, s[0]); for (i = 0; i < length; i++) output[i] = o.b[i]; - tag = gfmul_pmull(veorq_u8(tag, vld1q_u8((const uint8_t *) &m)), - (const uint8_t *) ht); + acc = ghash_one(acc, vld1q_u8((const uint8_t *) &m), ht); } GCM_EPILOGUE; } @@ -433,7 +477,7 @@ } while (0); #define XTS_TWEAK_ROLL(i) do { t[i] = tn[i]; } while (0); -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_encrypt_xts)(aes_block *output, aes_key *key, aes_key *key2, aes_block *dataunit, uint32_t spoint, aes_block *input, uint32_t nb_blocks) { const uint8_t *rk = FWD(key); @@ -477,7 +521,7 @@ } } -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void SIZED(crypton_aes_armv8_decrypt_xts)(aes_block *output, aes_key *key, aes_key *key2, aes_block *dataunit, uint32_t spoint, aes_block *input, uint32_t nb_blocks) { const uint8_t *fwd = FWD(key); @@ -523,9 +567,237 @@ } } +/* + * One message, one call: the additional data, the counter-mode encryption, + * the tag and the QUIC header protection mask, with the running tag and the + * counter kept in registers from end to end. + * + * What this saves over composing crypton_aes_gcm_aad, _encrypt and _finish is + * not the arithmetic but the boundaries. Each of those reaches its + * primitives through a branch table, so the 128-bit state goes back to memory + * at every step and a header of one block pays a reduction of its own. On an + * Apple M4 that framing was 0.07 of the 0.112 microseconds a 100-byte packet + * cost -- more than the encryption of the packet itself. + * + * The GHASH is taken in batches of WAY against the powers of H the key + * already holds, so a batch costs one reduction rather than one per block, + * and the additional data and the length block ride in the same batches as + * the ciphertext instead of being multiplied on their own. + */ + +/* start a batch, or continue one; blen is how many blocks this batch holds */ +#define FG_ABSORB(blk) \ + do { \ + poly64x2_t b_ = vreinterpretq_p64_u8(vrev64q_u8(blk)); \ + if (bn == 0) { \ + uint32_t left_ = gtotal - gidx; \ + blen = left_ < WAY ? left_ : WAY; \ + acc = vreinterpretq_u64_u8( \ + vextq_u8(vreinterpretq_u8_u64(acc), \ + vreinterpretq_u8_u64(acc), 8)); \ + b_ = vreinterpretq_p64_u64(veorq_u64( \ + vreinterpretq_u64_p64(b_), acc)); \ + gH = vdupq_n_u64(0); \ + gM = gH; \ + gL = gH; \ + } \ + GHASH_MUL(b_, GHASH_POW(ht, blen - bn - 1), \ + GHASH_KARAT(ht, blen - bn - 1), gH, gM, gL); \ + gidx++; bn++; \ + if (bn == blen) { acc = ghash_reduce(gH, gM, gL); bn = 0; } \ + } while (0) + +/* a block that is short, zero padded, as GHASH wants it */ +#define FG_PARTIAL(p, n) \ + ({ uint8_t buf_[16]; memset(buf_, 0, 16); memcpy(buf_, (p), (n)); \ + vld1q_u8(buf_); }) + +CRYPTON_TARGET_ARMV8_CRYPTO +void SIZED(crypton_aes_armv8_gcm_fused)(uint8_t *out, const block128 *ht, + aes_key *key, const uint8_t *nonce, + const uint8_t *aad, uint32_t aadlen, + const uint8_t *in, uint32_t inlen, + uint32_t taglen, aes_key *hpkey, + uint32_t sampleoff, uint8_t *mask) +{ + const uint8_t *rk = FWD(key); + uint8x16_t s[WAY]; + uint8x16_t ek0; + uint64x2_t acc = vdupq_n_u64(0), gH = acc, gM = acc, gL = acc; + uint32x4_t base; + uint32_t c = 1, bn = 0, blen = 0, gidx = 0; + uint32_t gtotal = (aadlen + 15) / 16 + (inlen + 15) / 16 + 1; + uint32_t i, done; + uint8_t y0[16], lenb[16]; + uint64_t la, lc; + + memcpy(y0, nonce, 12); + y0[12] = 0; y0[13] = 0; y0[14] = 0; y0[15] = 1; + base = vreinterpretq_u32_u8(vld1q_u8(y0)); + + s[0] = vld1q_u8(y0); + ENC_ROUNDS(EACH1); + ek0 = s[0]; + + for (i = 0; i + 16 <= aadlen; i += 16) + FG_ABSORB(vld1q_u8(aad + i)); + if (i < aadlen) + FG_ABSORB(FG_PARTIAL(aad + i, aadlen - i)); + + for (done = 0; done + 16 * WAY <= inlen; done += 16 * WAY) { + const uint8_t *p = in + done; + uint8_t *q = out + done; + + EACH8(GCM_CTR); + c += WAY; + ENC_ROUNDS(EACH8); + { + const uint8_t *input = p; + uint8_t *output = q; + EACH8(GCM_ENC); + } + FG_ABSORB(s[0]); FG_ABSORB(s[1]); FG_ABSORB(s[2]); FG_ABSORB(s[3]); + FG_ABSORB(s[4]); FG_ABSORB(s[5]); FG_ABSORB(s[6]); FG_ABSORB(s[7]); + } + + for (; done < inlen; done += 16) { + uint32_t n = inlen - done < 16 ? inlen - done : 16; + uint8x16_t m_ = n == 16 ? vld1q_u8(in + done) + : FG_PARTIAL(in + done, n); + c++; + s[0] = vreinterpretq_u8_u32(vsetq_lane_u32(cpu_to_be32(c), base, 3)); + ENC_ROUNDS(EACH1); + s[0] = veorq_u8(s[0], m_); + if (n == 16) { + vst1q_u8(out + done, s[0]); + } else { + uint8_t buf_[16]; + vst1q_u8(buf_, s[0]); + memcpy(out + done, buf_, n); + memset(buf_ + n, 0, 16 - n); + s[0] = vld1q_u8(buf_); + } + FG_ABSORB(s[0]); + } + + la = (uint64_t) aadlen << 3; + lc = (uint64_t) inlen << 3; + for (i = 0; i < 8; i++) lenb[i] = (uint8_t) (la >> (56 - 8 * i)); + for (i = 0; i < 8; i++) lenb[8 + i] = (uint8_t) (lc >> (56 - 8 * i)); + FG_ABSORB(vld1q_u8(lenb)); + + { + uint8_t tbuf[16]; + vst1q_u8(tbuf, veorq_u8(ghash_swap(vreinterpretq_u8_u64(acc)), ek0)); + memcpy(out + inlen, tbuf, taglen); + } + + if (hpkey != 0 && mask != 0) { + block128 sample, m; + memcpy(&sample, out + sampleoff, 16); + crypton_aes_encrypt_ecb(&m, hpkey, &sample, 1); + memcpy(mask, &m, 16); + } +} + + +/* + * The same for decryption. GCM_DEC leaves the ciphertext in s[] once it has + * written the plaintext out, which is what GHASH wants, so the only other + * difference is the end: the tag is compared here rather than written, every + * byte of it whichever way the answer goes. + */ +CRYPTON_TARGET_ARMV8_CRYPTO +int SIZED(crypton_aes_armv8_gcm_fused_dec)(uint8_t *out, const block128 *ht, + aes_key *key, const uint8_t *nonce, + const uint8_t *aad, uint32_t aadlen, + const uint8_t *in, uint32_t inlen, + const uint8_t *tagp, uint32_t taglen, + uint8_t *outtag) +{ + const uint8_t *rk = FWD(key); + uint8x16_t s[WAY]; + uint8x16_t ek0; + uint64x2_t acc = vdupq_n_u64(0), gH = acc, gM = acc, gL = acc; + uint32x4_t base; + uint32_t c = 1, bn = 0, blen = 0, gidx = 0; + uint32_t gtotal = (aadlen + 15) / 16 + (inlen + 15) / 16 + 1; + uint32_t i, done; + uint8_t y0[16], lenb[16], want[16]; + uint64_t la, lc; + uint8_t diff = 0; + + memcpy(y0, nonce, 12); + y0[12] = 0; y0[13] = 0; y0[14] = 0; y0[15] = 1; + base = vreinterpretq_u32_u8(vld1q_u8(y0)); + + s[0] = vld1q_u8(y0); + ENC_ROUNDS(EACH1); + ek0 = s[0]; + + for (i = 0; i + 16 <= aadlen; i += 16) + FG_ABSORB(vld1q_u8(aad + i)); + if (i < aadlen) + FG_ABSORB(FG_PARTIAL(aad + i, aadlen - i)); + + for (done = 0; done + 16 * WAY <= inlen; done += 16 * WAY) { + const uint8_t *p = in + done; + uint8_t *q = out + done; + + EACH8(GCM_CTR); + c += WAY; + ENC_ROUNDS(EACH8); + { + const uint8_t *input = p; + uint8_t *output = q; + EACH8(GCM_DEC); + } + FG_ABSORB(s[0]); FG_ABSORB(s[1]); FG_ABSORB(s[2]); FG_ABSORB(s[3]); + FG_ABSORB(s[4]); FG_ABSORB(s[5]); FG_ABSORB(s[6]); FG_ABSORB(s[7]); + } + + for (; done < inlen; done += 16) { + uint32_t n = inlen - done < 16 ? inlen - done : 16; + uint8x16_t m_ = n == 16 ? vld1q_u8(in + done) + : FG_PARTIAL(in + done, n); + c++; + s[0] = vreinterpretq_u8_u32(vsetq_lane_u32(cpu_to_be32(c), base, 3)); + ENC_ROUNDS(EACH1); + { + uint8x16_t pl = veorq_u8(s[0], m_); + if (n == 16) { + vst1q_u8(out + done, pl); + } else { + uint8_t buf_[16]; + vst1q_u8(buf_, pl); + memcpy(out + done, buf_, n); + } + } + FG_ABSORB(m_); + } + + la = (uint64_t) aadlen << 3; + lc = (uint64_t) inlen << 3; + for (i = 0; i < 8; i++) lenb[i] = (uint8_t) (la >> (56 - 8 * i)); + for (i = 0; i < 8; i++) lenb[8 + i] = (uint8_t) (lc >> (56 - 8 * i)); + FG_ABSORB(vld1q_u8(lenb)); + + vst1q_u8(want, veorq_u8(ghash_swap(vreinterpretq_u8_u64(acc)), ek0)); + if (outtag) { + /* The caller holds the expected tag and will compare it itself. */ + memcpy(outtag, want, taglen); + return 1; + } + for (i = 0; i < taglen; i++) + diff |= (uint8_t) (want[i] ^ tagp[i]); + return diff == 0; +} + +#undef FG_ABSORB +#undef FG_PARTIAL + #undef WAY #undef EACH1 -#undef EACH7 #undef EACH8 #undef LOAD_IN #undef STORE_OUT diff --git a/cbits/aes/block128.h b/cbits/aes/block128.h --- a/cbits/aes/block128.h +++ b/cbits/aes/block128.h @@ -34,7 +34,21 @@ #include #include -typedef union { +/* Packed, so that the union asks nothing of the address it is at. + * + * Several callers here make one of these out of a pointer of their own -- a + * ciphertext, a tag, a nonce -- and without this that cast produces a pointer + * the standard says may not exist, and reading q[] out of it is a member + * access at an address the type is not aligned for. crypton_align.h used to + * answer that with need_alignment, which is zero on i386 and x86-64: the two + * places the access is architecturally fine and the standard still says + * nothing about it. UndefinedBehaviorSanitizer reported it. + * + * With the alignment declared to be one, the compiler is the one that knows + * what the target can do: on i386, x86-64 and AArch64 it emits the same load + * and store it emitted before, and where a target cannot read a word off an + * odd address it emits what that target needs. */ +typedef union __attribute__((packed)) { uint64_t q[2]; uint32_t d[4]; uint16_t w[8]; diff --git a/cbits/aes/gcm_fused_x86.c b/cbits/aes/gcm_fused_x86.c new file mode 100644 --- /dev/null +++ b/cbits/aes/gcm_fused_x86.c @@ -0,0 +1,1013 @@ +/* + * A fused AES-GCM for x86-64, following the design Kazuho Oku sets out in + * "QUICむけにAES-GCM実装を最適化した話" and implements in picotls's + * lib/fusion.c: keep AES-NI issuing every clock and fit everything else -- + * the additional data, the tag, the QUIC header protection mask -- into the + * gaps it leaves. Written in C with intrinsics rather than assembly, for + * the same reason he gives: the scheduling is what is complicated here, and + * it has to stay readable to stay correct. + * + * Parts of this file follow fusion closely enough to say so: `loadn` and the + * two tables it reads, `loadn_page_end` and `storen` are its `loadn128`, + * `loadn_end_of_page` and `storen128` in another spelling, and the + * reduction of a 256-bit product is the sequence fusion takes from Gueron's + * "AES-GCM for Efficient Authenticated Encryption". fusion is under the MIT + * license, which is in cbits/aes/LICENSE.fusion beside this file. + * + * The powers of H are built once per key, so the additional data, the + * ciphertext and the length block are absorbed against them in batches that + * share one reduction, rather than each block paying for a reduction of its + * own. How many powers, and so how large a batch, is + * CRYPTON_GCM_FUSED_POWERS in crypton_aes.h. + * + * Only messages shorter than CRYPTON_GCM_FUSED_MAX_MESSAGE come here. Above + * that the stitched assembly in cbits/asm is faster, and crypton_aes.c sends + * them there instead; below it, that assembly will not start at all. + */ + +#include + +#ifdef WITH_GCM_FUSED + +#include +#include +#include +#include +#include + +#include +#include + +/* + * aes_key is a struct of bytes, so its round keys sit wherever the members + * before them leave them -- eight bytes in, as it happens. A __m128i * + * pointed at that gets an aligned load and a fault, so each round key is + * fetched with an unaligned load instead. Copying them somewhere aligned + * would cost a copy per call, which at these message lengths is a tenth of + * the whole; the load is free from L1 and the round key is fetched once for + * all six lanes. + */ +#define RK(p, i) \ + _mm_loadu_si128((const __m128i *) ((const uint8_t *) (p) + 16 * (size_t) (i))) + +/* a full sixteen-byte reversal: mask bytes 15,14,...,0 */ +static const __m128i BSWAP = {0x08090a0b0c0d0e0fLL, 0x0001020304050607LL}; + + +#define TGT __attribute__((target("aes,pclmul,sse4.1"))) + +/* the two halves of a value added together: the term Karatsuba needs, and it + * does not depend on what the value is multiplied by */ +TGT static inline __m128i fold(__m128i a) +{ + return _mm_xor_si128(a, _mm_unpackhi_epi64(a, a)); +} + +/* H multiplied by x in the field, which is what puts H and its powers one + * bit up: GCM numbers the bits of a field element the other way round from + * the way the carry-less multiply does, and pre-shifting is what saves the + * correction after every multiply. + * + * Written from the definition. The field is GF(2)[x] modulo x^128 + x^127 + + * x^126 + x^121 + 1, so multiplying by x is a shift of one place, and the + * term that leaves the top comes back as the other four. */ +TGT static __m128i twist(__m128i h) +{ + /* x^127 + x^126 + x^121 + 1, the terms x^128 is congruent to */ + const __m128i poly = _mm_set_epi64x(0xc200000000000000ULL, 1); + /* the top bit of each half */ + __m128i tops = _mm_srli_epi64(h, 63); + /* doubling a polynomial is a shift by one: each half doubles, and the + * low half's top bit becomes the high half's bottom bit */ + __m128i doubled = _mm_or_si128(_mm_add_epi64(h, h), + _mm_slli_si128(tops, 8)); + /* bit 127 is the one that leaves the field; spread it to a mask by + * subtracting it from zero, and it brings the four terms back */ + __m128i mask = _mm_sub_epi64(_mm_setzero_si128(), + _mm_unpackhi_epi64(tops, tops)); + + return _mm_xor_si128(doubled, _mm_and_si128(mask, poly)); +} + +/* one reduction of a 256-bit product back into the field */ +TGT static __m128i reduce256(__m128i lo, __m128i hi) +{ + const __m128i poly = _mm_set_epi64x(0xc200000000000000ULL, 1); + __m128i t; + + t = _mm_clmulepi64_si128(lo, poly, 0x10); + lo = _mm_xor_si128(_mm_shuffle_epi32(lo, 0x4e), t); + t = _mm_clmulepi64_si128(lo, poly, 0x10); + lo = _mm_xor_si128(_mm_shuffle_epi32(lo, 0x4e), t); + return _mm_xor_si128(hi, lo); +} + +/* + * One multiply in exactly the form the hot loop uses it: the left operand + * plain, the right one already twisted. Building the table with the same + * multiply that consumes it is the only way the two conventions cannot drift + * apart. + */ +TGT static __m128i mul_twisted(__m128i a, __m128i ht) +{ + __m128i lo = _mm_clmulepi64_si128(a, ht, 0x00); + __m128i hi = _mm_clmulepi64_si128(a, ht, 0x11); + __m128i mid = _mm_clmulepi64_si128(fold(a), fold(ht), 0x00); + + mid = _mm_xor_si128(mid, _mm_xor_si128(lo, hi)); + lo = _mm_xor_si128(lo, _mm_slli_si128(mid, 8)); + hi = _mm_xor_si128(hi, _mm_srli_si128(mid, 8)); + return reduce256(lo, hi); +} + +TGT void crypton_gcm_fused_key_init(aes_gcm_fused *fk, const aes_key *key) +{ + const uint8_t *rk = key->data; + const int rounds = key->nbr; + __m128i h, p; + int i; + + /* H = E_K(0) */ + h = RK(rk, 0); + for (i = 1; i < rounds; i++) h = _mm_aesenc_si128(h, RK(rk, i)); + h = _mm_aesenclast_si128(h, RK(rk, rounds)); + h = _mm_shuffle_epi8(h, BSWAP); + + { + __m128i ht = twist(h); + p = h; + for (i = 0; i < CRYPTON_GCM_FUSED_POWERS; i++) { + __m128i t = twist(p); + _mm_storeu_si128((__m128i *) &fk->p[i].h, t); + _mm_storeu_si128((__m128i *) &fk->p[i].r, fold(t)); + p = mul_twisted(p, ht); + } + } +} + +/* + * The running product. Three plain locals and a macro, not a struct behind + * a pointer: taking the address of the accumulators is enough to keep them + * out of registers, and then every multiply reloads and restores them. That + * is the same mistake as reaching a table through an index the compiler + * cannot fold, and it costs more here because it is on the inner path. + */ +#define GHASH_DECL __m128i glo = _mm_setzero_si128(), \ + ghi = _mm_setzero_si128(), \ + gmid = _mm_setzero_si128(), \ + gtag = _mm_setzero_si128(); \ + int gidx = 0, gblen = 0, gbpos = 0 + +/* + * Absorb one block. Blocks are taken in batches of at most CRYPTON_GCM_FUSED_POWERS: the + * first of a batch carries in the value the batch before it reduced to, the + * rest go in against descending powers, and the batch ends with the one + * reduction they share. With the batch as long as the message this is + * picotls's single reduction; with it fixed, the state stays a fixed size. + */ +#define GHASH_ONE(blk, unused_power) \ + do { \ + __m128i _b = (blk); \ + __m128i _h, _r; \ + if (gbpos == 0) { \ + int _left = gtotal - gidx; \ + gblen = _left < CRYPTON_GCM_FUSED_POWERS ? _left : CRYPTON_GCM_FUSED_POWERS; \ + _b = _mm_xor_si128(_b, gtag); \ + glo = ghi = gmid = _mm_setzero_si128(); \ + } \ + _h = _mm_loadu_si128((const __m128i *) &fk->p[gblen-gbpos-1].h); \ + _r = _mm_loadu_si128((const __m128i *) &fk->p[gblen-gbpos-1].r); \ + glo = _mm_xor_si128(glo, _mm_clmulepi64_si128(_b, _h, 0x00)); \ + ghi = _mm_xor_si128(ghi, _mm_clmulepi64_si128(_b, _h, 0x11)); \ + gmid = _mm_xor_si128(gmid, \ + _mm_clmulepi64_si128(fold(_b), _r, 0x00)); \ + gidx++; gbpos++; \ + if (gbpos == gblen) { \ + gtag = ghash_reduce(glo, ghi, gmid); \ + gbpos = 0; \ + } \ + } while (0) + +TGT static __m128i ghash_reduce(__m128i glo, __m128i ghi, __m128i gmid) +{ + __m128i mid = _mm_xor_si128(gmid, _mm_xor_si128(glo, ghi)); + __m128i lo = _mm_xor_si128(glo, _mm_slli_si128(mid, 8)); + __m128i hi = _mm_xor_si128(ghi, _mm_srli_si128(mid, 8)); + + return reduce256(lo, hi); +} + +/* zero every byte from n onwards, so a partial block can be fed to GHASH + * without being written out and read back */ +TGT static __m128i clampn(__m128i v, size_t n) +{ + const __m128i idx = {0x0706050403020100LL, 0x0f0e0d0c0b0a0908LL}; + return _mm_and_si128(v, _mm_cmpgt_epi8(_mm_set1_epi8((char) n), idx)); +} + +/* + * A short block, zero padded, without going through the stack. + * + * The obvious way -- zero sixteen bytes, copy n in, load them back -- is + * three trips to memory with a store the load has to wait for, and at these + * lengths that is a tenth of the whole call. Reading the sixteen bytes and + * masking off what is above n is two instructions, but it reads past the end + * of what the caller gave, so it has to be sure those bytes exist. + * + * They do unless the block ends a page. A load of sixteen bytes that starts + * at least sixteen from the end of a page stays inside it; and if the n bytes + * asked for themselves cross the boundary then the next page is there to be + * read as well. What is left is a block near the end of a page whose own + * bytes stop short of it, and that is read aligned -- which cannot leave the + * page -- and shuffled down. This is how picotls's fusion does it. + */ + +/* thirty-two bytes of ones, then thirty-one of zeros: sixteen loaded from + * 32 - n give n bytes of ones and the rest zeros */ +static const uint8_t loadn_mask[63] = { + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}; + +/* the first sixteen map to byte offsets, the rest to zero */ +static const uint8_t loadn_shuffle[31] = { + 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, + 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, + 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, + 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80}; + +#if defined(__has_feature) +#if __has_feature(address_sanitizer) +#define NO_ASAN __attribute__((no_sanitize_address)) +#endif +#elif defined(__SANITIZE_ADDRESS__) +#define NO_ASAN __attribute__((no_sanitize_address)) +#endif +#ifndef NO_ASAN +#define NO_ASAN +#endif + +TGT NO_ASAN static __m128i loadn_page_end(const uint8_t *p, size_t n) +{ + uintptr_t shift = (uintptr_t) p & 15; + __m128i pattern = _mm_loadu_si128((const __m128i *) (loadn_shuffle + shift)); + + (void) n; + return _mm_shuffle_epi8( + _mm_load_si128((const __m128i *) ((uintptr_t) p - shift)), pattern); +} + +TGT NO_ASAN static __m128i loadn(const uint8_t *p, size_t n) +{ + __m128i mask = _mm_loadu_si128((const __m128i *) (loadn_mask + 32 - n)); + uintptr_t mod4k = (uintptr_t) p % 4096; + __m128i v; + + if (mod4k <= 4096 - 16 || mod4k + n > 4096) + v = _mm_loadu_si128((const __m128i *) p); + else + v = loadn_page_end(p, n); + return _mm_and_si128(v, mask); +} + +TGT static void storen(uint8_t *p, __m128i v, size_t n) +{ + uint8_t buf[16]; + _mm_storeu_si128((__m128i *) buf, v); + memcpy(p, buf, n); +} + +/* One block. The ten rounds of the common case are written out for the + * same reason the six-wide group is: a loop over a round count that lives in + * the key leaves every round key fetched through an index the compiler + * cannot fold, and adds a branch to a chain that is already latency-bound. */ +TGT static __m128i aes_one_block(const uint8_t *rk, int rounds, __m128i v) +{ + const uint8_t *k = rk; + int i; + + if (rounds == 10) { + v = _mm_xor_si128(v, RK(k, 0)); + v = _mm_aesenc_si128(v, RK(k, 1)); + v = _mm_aesenc_si128(v, RK(k, 2)); + v = _mm_aesenc_si128(v, RK(k, 3)); + v = _mm_aesenc_si128(v, RK(k, 4)); + v = _mm_aesenc_si128(v, RK(k, 5)); + v = _mm_aesenc_si128(v, RK(k, 6)); + v = _mm_aesenc_si128(v, RK(k, 7)); + v = _mm_aesenc_si128(v, RK(k, 8)); + v = _mm_aesenc_si128(v, RK(k, 9)); + return _mm_aesenclast_si128(v, RK(k, 10)); + } + v = _mm_xor_si128(v, RK(k, 0)); + for (i = 1; i < rounds; i++) v = _mm_aesenc_si128(v, RK(k, i)); + return _mm_aesenclast_si128(v, RK(k, rounds)); +} + +/* + * Six blocks at once, with lane 5 free to run a different key schedule from + * the rest. Which schedule that lane uses is chosen once, into a pointer, + * rather than tested inside the rounds, and the six live in named variables + * rather than an array -- an array indexed by a running variable goes to + * memory, and then every round is a load and a store instead of a register + * to register operation, which is the whole of what this is trying to avoid. + * + * Lanes beyond what the caller needs still run. Six are in flight whatever + * the message length, so the spare ones cost nothing, and that is exactly + * why the header protection mask and E(K,Y0) are worth putting in them + * instead of giving each a dependent chain of its own. + */ +#define WIDE6(alt) \ + do { \ + const uint8_t *ak = (alt); \ + int r; \ + t0 = _mm_xor_si128(t0, RK(rk, 0)); \ + t1 = _mm_xor_si128(t1, RK(rk, 0)); \ + t2 = _mm_xor_si128(t2, RK(rk, 0)); \ + t3 = _mm_xor_si128(t3, RK(rk, 0)); \ + t4 = _mm_xor_si128(t4, RK(rk, 0)); \ + t5 = _mm_xor_si128(t5, RK(ak, 0)); \ + for (r = 1; r < rounds; r++) { \ + __m128i k = RK(rk, r); \ + t0 = _mm_aesenc_si128(t0, k); \ + t1 = _mm_aesenc_si128(t1, k); \ + t2 = _mm_aesenc_si128(t2, k); \ + t3 = _mm_aesenc_si128(t3, k); \ + t4 = _mm_aesenc_si128(t4, k); \ + t5 = _mm_aesenc_si128(t5, RK(ak, r)); \ + GSTEP(); \ + } \ + { \ + __m128i k = RK(rk, rounds); \ + t0 = _mm_aesenclast_si128(t0, k); \ + t1 = _mm_aesenclast_si128(t1, k); \ + t2 = _mm_aesenclast_si128(t2, k); \ + t3 = _mm_aesenclast_si128(t3, k); \ + t4 = _mm_aesenclast_si128(t4, k); \ + t5 = _mm_aesenclast_si128(t5, RK(ak, rounds)); \ + } \ + } while (0) + +/* the same with nothing queued to absorb: the decryption side takes its + * GHASH straight from the input and has no queue to drain */ +#define WIDE6_NOQ(alt) \ + do { \ + const uint8_t *ak = (alt); \ + int r; \ + t0 = _mm_xor_si128(t0, RK(rk, 0)); \ + t1 = _mm_xor_si128(t1, RK(rk, 0)); \ + t2 = _mm_xor_si128(t2, RK(rk, 0)); \ + t3 = _mm_xor_si128(t3, RK(rk, 0)); \ + t4 = _mm_xor_si128(t4, RK(rk, 0)); \ + t5 = _mm_xor_si128(t5, RK(ak, 0)); \ + for (r = 1; r < rounds; r++) { \ + __m128i k = RK(rk, r); \ + t0 = _mm_aesenc_si128(t0, k); \ + t1 = _mm_aesenc_si128(t1, k); \ + t2 = _mm_aesenc_si128(t2, k); \ + t3 = _mm_aesenc_si128(t3, k); \ + t4 = _mm_aesenc_si128(t4, k); \ + t5 = _mm_aesenc_si128(t5, RK(ak, r)); \ + } \ + { \ + __m128i k = RK(rk, rounds); \ + t0 = _mm_aesenclast_si128(t0, k); \ + t1 = _mm_aesenclast_si128(t1, k); \ + t2 = _mm_aesenclast_si128(t2, k); \ + t3 = _mm_aesenclast_si128(t3, k); \ + t4 = _mm_aesenclast_si128(t4, k); \ + t5 = _mm_aesenclast_si128(t5, RK(ak, rounds)); \ + } \ + } while (0) + +/* + * The same written out for the ten rounds of AES-128, with a slot for one + * queued multiply between each of the first six. The loop above cannot take + * them: the round count is a value in the key, so there is no place the + * compiler knows is a round apart from the next, and a test before each + * multiply would end the basic block the scheduler works inside. This pass + * runs with a group's multiplies still waiting, and on a short message that + * is most of what it has to do. + */ +#define WROUND6(r, ak) \ + do { \ + __m128i k = RK(rk, r); \ + t0 = _mm_aesenc_si128(t0, k); \ + t1 = _mm_aesenc_si128(t1, k); \ + t2 = _mm_aesenc_si128(t2, k); \ + t3 = _mm_aesenc_si128(t3, k); \ + t4 = _mm_aesenc_si128(t4, k); \ + t5 = _mm_aesenc_si128(t5, RK(ak, r)); \ + } while (0) + +#define WIDE6_10(alt) \ + do { \ + const uint8_t *ak = (alt); \ + t0 = _mm_xor_si128(t0, RK(rk, 0)); \ + t1 = _mm_xor_si128(t1, RK(rk, 0)); \ + t2 = _mm_xor_si128(t2, RK(rk, 0)); \ + t3 = _mm_xor_si128(t3, RK(rk, 0)); \ + t4 = _mm_xor_si128(t4, RK(rk, 0)); \ + t5 = _mm_xor_si128(t5, RK(ak, 0)); \ + WROUND6(1, ak); GAT(0); \ + WROUND6(2, ak); GAT(1); \ + WROUND6(3, ak); GAT(2); \ + WROUND6(4, ak); GAT(3); \ + WROUND6(5, ak); GAT(4); \ + WROUND6(6, ak); GAT(5); \ + WROUND6(7, ak); \ + WROUND6(8, ak); \ + WROUND6(9, ak); \ + { \ + __m128i k = RK(rk, 10); \ + t0 = _mm_aesenclast_si128(t0, k); \ + t1 = _mm_aesenclast_si128(t1, k); \ + t2 = _mm_aesenclast_si128(t2, k); \ + t3 = _mm_aesenclast_si128(t3, k); \ + t4 = _mm_aesenclast_si128(t4, k); \ + t5 = _mm_aesenclast_si128(t5, RK(ak, 10)); \ + } \ + } while (0) + +/* + * v2: six blocks of AES in flight at once, so the ten rounds of one block no + * longer wait on each other -- AES-NI is pipelined and will take one + * instruction a clock as long as the instructions in flight are independent. + * The GHASH multiplies of the group just finished are issued between the + * rounds of the group now running, which is the stitching: they do not want + * the same execution port, so held against each other they cost about what + * the rounds alone cost. + */ + +/* + * The counter block, built without leaving the vector registers. + * + * GCM counts in the low 32 bits of the block, big endian, and wraps there. + * ctr holds the block with its bytes reversed, so those four bytes are the + * low lane and _mm_add_epi32 steps them without carrying into the nonce + * above -- which is the wrap GCM asks for. A shuffle puts the bytes back. + * + * The obvious way -- increment a uint32_t, byte swap it, pinsrd it in -- + * costs a move from a general register to a vector one for every lane, six + * to a group, and those do not come free. + */ +#define CTR6(j) \ + do { \ + ctr = _mm_add_epi32(ctr, one32); \ + b##j = _mm_xor_si128(_mm_shuffle_epi8(ctr, BSWAP), RK(rk, 0)); \ + } while (0) + +#define ROUND6(r) \ + do { \ + __m128i k = RK(rk, r); \ + b0 = _mm_aesenc_si128(b0, k); \ + b1 = _mm_aesenc_si128(b1, k); \ + b2 = _mm_aesenc_si128(b2, k); \ + b3 = _mm_aesenc_si128(b3, k); \ + b4 = _mm_aesenc_si128(b4, k); \ + b5 = _mm_aesenc_si128(b5, k); \ + } while (0) + +#define LAST6(r) \ + do { \ + __m128i k = RK(rk, r); \ + b0 = _mm_aesenclast_si128(b0, k); \ + b1 = _mm_aesenclast_si128(b1, k); \ + b2 = _mm_aesenclast_si128(b2, k); \ + b3 = _mm_aesenclast_si128(b3, k); \ + b4 = _mm_aesenclast_si128(b4, k); \ + b5 = _mm_aesenclast_si128(b5, k); \ + } while (0) + +/* one GHASH multiply, taken from a queue of blocks waiting to be absorbed, + * to be issued in the gaps between AES rounds */ +/* A ring, so that a block queued while others are still waiting costs an + * index and not a move: the queue is walked from both ends and never + * compacted. */ +#define GQ_MASK 15 + +#define GPUSH(v) \ + do { gq[gw] = (v); gw++; gn++; } while (0) + +#define GSTEP() \ + do { \ + if (gn > 0) { \ + GHASH_ONE(gq[gi], gp); \ + gi++; gp--; gn--; \ + } \ + } while (0) + +/* the same at a slot the compiler can see, for the unrolled group below */ +/* + * One queued block, at a slot the compiler can see and with nothing to test + * before it. A test here would end the basic block, and the scheduler works + * inside one: six tests turn the group into twelve blocks and the multiplies + * can no longer be moved up among the rounds, which is the whole point of + * writing them there. The group below is entered only when the queue is + * full, so there is nothing to test. + */ +/* + * The block a slot names, read from the output where the group before it + * left the ciphertext rather than from a copy kept beside it. The copy + * cost six stores a group for bytes already in memory; picotls's fusion + * points its GHASH at the output it has just written for the same reason. + */ +#define GAT(j) GHASH_ONE(_mm_shuffle_epi8( \ + _mm_loadu_si128((const __m128i *) (prev + 16 * (j))), BSWAP), gp - (j)) + +TGT void crypton_gcm_fused_encrypt(uint8_t *out, const aes_gcm_fused *fk, + const aes_key *key, const uint8_t *nonce, + const uint8_t *aad, size_t aadlen, + const uint8_t *in, size_t inlen, size_t taglen, + const aes_key *hpkey, size_t sampleoff, + uint8_t *mask) +{ + const uint8_t *rk = key->data; + const uint8_t *hprk = hpkey != 0 ? hpkey->data : rk; + const int rounds = key->nbr; + const int hprounds = hpkey != 0 ? hpkey->nbr : rounds; + GHASH_DECL; + __m128i ctrbase, ctr, one32, ek0, tag, b0, b1, b2, b3, b4, b5; + const int ntail_pre = (int) ((inlen % 96 + 15) / 16); + int lane_ek0; + __m128i gq[6]; + unsigned gi = 0, gw = 0; + int gn = 0; + size_t nblk = (aadlen + 15) / 16 + (inlen + 15) / 16 + 1; + const int gtotal = (int) nblk; + int gp = (int) nblk; + size_t i; + size_t done; + int lane_mask = 0; + + /* + * Y0: the twelve bytes of nonce and a counter of one. loadn reads the + * nonce where it lies and masks what is above it, so this is one load + * rather than three of four bytes each and a set built from them. + */ + ctrbase = _mm_insert_epi32(loadn(nonce, 12), (int) __builtin_bswap32(1), 3); + ctr = _mm_shuffle_epi8(ctrbase, BSWAP); + one32 = _mm_set_epi32(0, 0, 0, 1); + + lane_ek0 = ntail_pre > 0 && ntail_pre <= 4; + if (!lane_ek0) + ek0 = aes_one_block(rk, rounds, ctrbase); + + /* The additional data goes in first and takes the highest powers, but it + * is only queued here: absorbing it takes multiplies, and the multiplies + * belong in the gaps between the AES rounds below rather than in front + * of them where nothing else is running. */ + /* + * The additional data goes in first and takes the highest powers. It is + * absorbed here rather than queued: what the queue is for is giving the + * rounds below something to interleave with, and a queue that sometimes + * holds the header and sometimes does not forces a test before every + * multiply -- which is what stopped the interleaving from happening at + * all. See the peeled first group below. + */ + { + size_t nfull = aadlen / 16; + size_t rest = aadlen % 16; + + for (i = 0; i < nfull; i++) + GHASH_ONE(_mm_shuffle_epi8( + _mm_loadu_si128((const __m128i *) (aad + i * 16)), BSWAP), 0); + if (rest) + GHASH_ONE(_mm_shuffle_epi8(loadn(aad + nfull * 16, rest), BSWAP), 0); + } + + /* Whole groups of six. The rounds are written out rather than looped: + * the number of them is a value in the key, so a loop over it leaves the + * compiler fetching each round key through an index it cannot fold, and + * the six lanes go to memory with them. Written out, the whole group + * stays in registers, and the six multiplies of the group before can be + * placed between the rounds by hand -- which is the stitching: AES-NI + * and PCLMULQDQ do not contend for the same port, so the multiplies are + * very nearly free. + */ + done = 0; + if (rounds == 10) { + if (done + 96 <= inlen) { + const uint8_t *p = in + done; + uint8_t *q = out + done; + + CTR6(0); CTR6(1); CTR6(2); CTR6(3); CTR6(4); CTR6(5); + ROUND6(1); + ROUND6(2); + ROUND6(3); + ROUND6(4); + ROUND6(5); + ROUND6(6); + ROUND6(7); + ROUND6(8); + ROUND6(9); + LAST6(10); + gn = 0; gi = 0; gw = 0; + + b0 = _mm_xor_si128(b0, _mm_loadu_si128((const __m128i *) p)); + b1 = _mm_xor_si128(b1, _mm_loadu_si128((const __m128i *) (p + 16))); + b2 = _mm_xor_si128(b2, _mm_loadu_si128((const __m128i *) (p + 32))); + b3 = _mm_xor_si128(b3, _mm_loadu_si128((const __m128i *) (p + 48))); + b4 = _mm_xor_si128(b4, _mm_loadu_si128((const __m128i *) (p + 64))); + b5 = _mm_xor_si128(b5, _mm_loadu_si128((const __m128i *) (p + 80))); + _mm_storeu_si128((__m128i *) q, b0); + _mm_storeu_si128((__m128i *) (q + 16), b1); + _mm_storeu_si128((__m128i *) (q + 32), b2); + _mm_storeu_si128((__m128i *) (q + 48), b3); + _mm_storeu_si128((__m128i *) (q + 64), b4); + _mm_storeu_si128((__m128i *) (q + 80), b5); + + done += 96; + } + for (; done + 96 <= inlen; done += 96) { + const uint8_t *p = in + done; + uint8_t *q = out + done; + const uint8_t *prev = out + done - 96; + + CTR6(0); CTR6(1); CTR6(2); CTR6(3); CTR6(4); CTR6(5); + ROUND6(1); GAT(0); + ROUND6(2); GAT(1); + ROUND6(3); GAT(2); + ROUND6(4); GAT(3); + ROUND6(5); GAT(4); + ROUND6(6); GAT(5); + ROUND6(7); + ROUND6(8); + ROUND6(9); + LAST6(10); + gp -= 6; + + b0 = _mm_xor_si128(b0, _mm_loadu_si128((const __m128i *) p)); + b1 = _mm_xor_si128(b1, _mm_loadu_si128((const __m128i *) (p + 16))); + b2 = _mm_xor_si128(b2, _mm_loadu_si128((const __m128i *) (p + 32))); + b3 = _mm_xor_si128(b3, _mm_loadu_si128((const __m128i *) (p + 48))); + b4 = _mm_xor_si128(b4, _mm_loadu_si128((const __m128i *) (p + 64))); + b5 = _mm_xor_si128(b5, _mm_loadu_si128((const __m128i *) (p + 80))); + _mm_storeu_si128((__m128i *) q, b0); + _mm_storeu_si128((__m128i *) (q + 16), b1); + _mm_storeu_si128((__m128i *) (q + 32), b2); + _mm_storeu_si128((__m128i *) (q + 48), b3); + _mm_storeu_si128((__m128i *) (q + 64), b4); + _mm_storeu_si128((__m128i *) (q + 80), b5); + + /* straight from the registers the last round left them in: the + * queue existed only to hold them until the next group's rounds + * could hide the multiplies, and that is 192 bytes of store and + * load per 96 bytes of payload */ + } + + } else { + for (done = 0; done + 96 <= inlen; done += 96) { + const uint8_t *p = in + done; + uint8_t *q = out + done; + int r; + + CTR6(0); CTR6(1); CTR6(2); CTR6(3); CTR6(4); CTR6(5); + for (r = 1; r < rounds; r++) { + ROUND6(r); + GSTEP(); + } + LAST6(rounds); + + b0 = _mm_xor_si128(b0, _mm_loadu_si128((const __m128i *) p)); + b1 = _mm_xor_si128(b1, _mm_loadu_si128((const __m128i *) (p + 16))); + b2 = _mm_xor_si128(b2, _mm_loadu_si128((const __m128i *) (p + 32))); + b3 = _mm_xor_si128(b3, _mm_loadu_si128((const __m128i *) (p + 48))); + b4 = _mm_xor_si128(b4, _mm_loadu_si128((const __m128i *) (p + 64))); + b5 = _mm_xor_si128(b5, _mm_loadu_si128((const __m128i *) (p + 80))); + _mm_storeu_si128((__m128i *) q, b0); + _mm_storeu_si128((__m128i *) (q + 16), b1); + _mm_storeu_si128((__m128i *) (q + 32), b2); + _mm_storeu_si128((__m128i *) (q + 48), b3); + _mm_storeu_si128((__m128i *) (q + 64), b4); + _mm_storeu_si128((__m128i *) (q + 80), b5); + + while (gn > 0) GSTEP(); + gi = 0; gw = 0; + gq[0] = _mm_shuffle_epi8(b0, BSWAP); + gq[1] = _mm_shuffle_epi8(b1, BSWAP); + gq[2] = _mm_shuffle_epi8(b2, BSWAP); + gq[3] = _mm_shuffle_epi8(b3, BSWAP); + gq[4] = _mm_shuffle_epi8(b4, BSWAP); + gq[5] = _mm_shuffle_epi8(b5, BSWAP); + gn = 6; + } + } + + /* The tail. The wide pass below runs only when there are blocks for it: + * sixty AES instructions to fill one lane is not worth it, so a message + * that ends on a group boundary leaves E(K,Y0) the chain it was given + * above and takes the mask on one of its own. + * + * The spare lane carries the mask only when two things hold. The sample + * has to lie entirely in output the groups above have already written: + * this pass reads it while it runs, and the blocks it is itself + * computing are stored after it, so a sample reaching into them would be + * read before it exists. And the two key schedules have to have the + * same number of rounds, because the lanes share the loop that counts + * them and a shorter schedule would be read past its end. TLS and QUIC + * satisfy both; anything else gets the mask on a chain of its own, which + * is what it would have had anyway. */ + { + __m128i t0, t1, t2, t3, t4, t5; + __m128i tv[6]; + size_t toff[6]; + int ntail = 0, j; + + if (done >= inlen) goto no_tail; + + for (i = done; i < inlen; i += 16) { + toff[ntail] = i; + ctr = _mm_add_epi32(ctr, one32); + tv[ntail] = _mm_shuffle_epi8(ctr, BSWAP); + ntail++; + } + + lane_mask = ntail > 0 && ntail <= 5 && hpkey != 0 && mask != 0 + && hprounds == rounds + && sampleoff + 16 <= done; + + if (ntail > 0) { + t0 = tv[0]; + t1 = ntail > 1 ? tv[1] : ctrbase; + t2 = ntail > 2 ? tv[2] : ctrbase; + t3 = ntail > 3 ? tv[3] : ctrbase; + t4 = lane_ek0 ? ctrbase : (ntail > 4 ? tv[4] : ctrbase); + t5 = lane_mask + ? _mm_loadu_si128((const __m128i *) (out + sampleoff)) + : (ntail > 5 ? tv[5] : ctrbase); + /* A group leaves exactly six queued, which is what lets the + * slots below be named at compile time. Where no group ran + * there is nothing to place and the plain pass will do. */ + if (rounds == 10 && done >= 96) { + const uint8_t *prev = out + done - 96; + WIDE6_10(lane_mask ? hprk : rk); + } else { + WIDE6(lane_mask ? hprk : rk); + } + if (lane_ek0) + ek0 = t4; + else + tv[4] = t4; + if (lane_mask) + _mm_storeu_si128((__m128i *) mask, t5); + else if (ntail > 5) + tv[5] = t5; + } +no_tail: + while (gn > 0) GSTEP(); + + /* The last group's ciphertext is absorbed by the pass above where + * there is one to absorb it. A message that ends on a group + * boundary has no such pass, so it is taken here. */ + if (rounds == 10 && done >= 96 && ntail == 0) { + const uint8_t *prev = out + done - 96; + GAT(0); GAT(1); GAT(2); GAT(3); GAT(4); GAT(5); + } + + /* + * The tail blocks, from the registers the pass left them in. They + * were going through an array indexed by the loop variable, which + * the compiler cannot see through and so keeps in memory: every + * block then reloaded its own keystream. Named one per block and + * reached by a test on a count instead, they stay where they are. + */ +#define TAILBLK(j, reg) \ + do { \ + size_t off = done + 16 * (j); \ + size_t n = inlen - off < 16 ? inlen - off : 16; \ + __m128i c = _mm_xor_si128(reg, n == 16 \ + ? _mm_loadu_si128((const __m128i *) (in + off)) \ + : loadn(in + off, n)); \ + if (n == 16) { \ + _mm_storeu_si128((__m128i *) (out + off), c); \ + } else { \ + /* the tag goes in directly above, so those bytes are \ + * written over anyway where there are sixteen to spare */ \ + if (n + taglen >= 16) \ + _mm_storeu_si128((__m128i *) (out + off), c); \ + else \ + storen(out + off, c, n); \ + c = clampn(c, n); \ + } \ + GHASH_ONE(_mm_shuffle_epi8(c, BSWAP), gp); \ + gp--; \ + } while (0) + + if (ntail > 0) TAILBLK(0, t0); + if (ntail > 1) TAILBLK(1, t1); + if (ntail > 2) TAILBLK(2, t2); + if (ntail > 3) TAILBLK(3, t3); + if (ntail > 4) TAILBLK(4, tv[4]); + if (ntail > 5) TAILBLK(5, tv[5]); +#undef TAILBLK + } + + { + /* + * The length block: the additional data's bit count and the + * message's, each big endian in a half, and then reversed like + * every other block on its way to GHASH. + * + * Reversed, that block is the two counts as ordinary little endian + * words with the message's in the low half -- which is one set, and + * no shuffle. Sixteen byte stores to the stack and a load back is + * what it cost before. + */ + GHASH_ONE(_mm_set_epi64x((long long) ((uint64_t) aadlen << 3), + (long long) ((uint64_t) inlen << 3)), gp); + } + + tag = _mm_shuffle_epi8(gtag, BSWAP); + tag = _mm_xor_si128(tag, ek0); + if (taglen == 16) + _mm_storeu_si128((__m128i *) (out + inlen), tag); + else + storen(out + inlen, tag, taglen); + + /* A sample the pass above could not reach -- because it covered blocks + * that pass was still computing, or the tag, which is written just now + * -- is taken here instead, where everything it can cover exists. */ + if (!lane_mask && hpkey != 0 && mask != 0) + _mm_storeu_si128((__m128i *) mask, + aes_one_block(hprk, hprounds, _mm_loadu_si128( + (const __m128i *) (out + sampleoff)))); +} + + +/* + * The same for decryption, which is the simpler of the two. + * + * What GHASH absorbs here is the ciphertext, and the ciphertext is the + * input: it is there before any of the AES has run. So there is no queue -- + * the multiplies of a group go between the rounds of that same group rather + * than waiting for the one after, and nothing is stored and loaded back to + * carry them across. + * + * The tag is compared here, every byte of it whichever way the answer goes, + * and the answer is 1 for a message whose tag matched. + */ + +/* one ciphertext block straight from the input, at a slot named here */ +#define DAT(j) GHASH_ONE(_mm_shuffle_epi8( \ + _mm_loadu_si128((const __m128i *) (p + 16 * (j))), BSWAP), 0) + +/* the next counter block, into a named register */ +#define CTRT(t) \ + do { ctr = _mm_add_epi32(ctr, one32); \ + t = _mm_shuffle_epi8(ctr, BSWAP); } while (0) + +TGT int crypton_gcm_fused_decrypt(uint8_t *out, const aes_gcm_fused *fk, + const aes_key *key, const uint8_t *nonce, + const uint8_t *aad, size_t aadlen, + const uint8_t *in, size_t inlen, + const uint8_t *tag, size_t taglen, + uint8_t *outtag) +{ + const uint8_t *rk = key->data; + const int rounds = key->nbr; + GHASH_DECL; + __m128i ctrbase, ctr, one32, ek0, want, b0, b1, b2, b3, b4, b5; + const int ntail_pre = (int) ((inlen % 96 + 15) / 16); + int lane_ek0, gp = 0; + const int gtotal = (int) ((aadlen + 15) / 16 + (inlen + 15) / 16 + 1); + size_t i; + size_t done; + uint8_t diff = 0; + + ctrbase = _mm_insert_epi32(loadn(nonce, 12), (int) __builtin_bswap32(1), 3); + ctr = _mm_shuffle_epi8(ctrbase, BSWAP); + one32 = _mm_set_epi32(0, 0, 0, 1); + + lane_ek0 = ntail_pre > 0 && ntail_pre <= 5; + if (!lane_ek0) + ek0 = aes_one_block(rk, rounds, ctrbase); + + { + size_t nfull = aadlen / 16; + size_t rest = aadlen % 16; + + for (i = 0; i < nfull; i++) + GHASH_ONE(_mm_shuffle_epi8( + _mm_loadu_si128((const __m128i *) (aad + i * 16)), BSWAP), 0); + if (rest) + GHASH_ONE(_mm_shuffle_epi8(loadn(aad + nfull * 16, rest), BSWAP), 0); + } + + done = 0; + if (rounds == 10) { + for (; done + 96 <= inlen; done += 96) { + const uint8_t *p = in + done; + uint8_t *q = out + done; + + CTR6(0); CTR6(1); CTR6(2); CTR6(3); CTR6(4); CTR6(5); + ROUND6(1); DAT(0); + ROUND6(2); DAT(1); + ROUND6(3); DAT(2); + ROUND6(4); DAT(3); + ROUND6(5); DAT(4); + ROUND6(6); DAT(5); + ROUND6(7); + ROUND6(8); + ROUND6(9); + LAST6(10); + + _mm_storeu_si128((__m128i *) q, + _mm_xor_si128(b0, _mm_loadu_si128((const __m128i *) p))); + _mm_storeu_si128((__m128i *) (q + 16), + _mm_xor_si128(b1, _mm_loadu_si128((const __m128i *) (p + 16)))); + _mm_storeu_si128((__m128i *) (q + 32), + _mm_xor_si128(b2, _mm_loadu_si128((const __m128i *) (p + 32)))); + _mm_storeu_si128((__m128i *) (q + 48), + _mm_xor_si128(b3, _mm_loadu_si128((const __m128i *) (p + 48)))); + _mm_storeu_si128((__m128i *) (q + 64), + _mm_xor_si128(b4, _mm_loadu_si128((const __m128i *) (p + 64)))); + _mm_storeu_si128((__m128i *) (q + 80), + _mm_xor_si128(b5, _mm_loadu_si128((const __m128i *) (p + 80)))); + } + } else { + for (; done + 96 <= inlen; done += 96) { + const uint8_t *p = in + done; + uint8_t *q = out + done; + int r; + + CTR6(0); CTR6(1); CTR6(2); CTR6(3); CTR6(4); CTR6(5); + for (r = 1; r < rounds; r++) { + ROUND6(r); + if (r <= 6) DAT(r - 1); + } + LAST6(rounds); + + _mm_storeu_si128((__m128i *) q, + _mm_xor_si128(b0, _mm_loadu_si128((const __m128i *) p))); + _mm_storeu_si128((__m128i *) (q + 16), + _mm_xor_si128(b1, _mm_loadu_si128((const __m128i *) (p + 16)))); + _mm_storeu_si128((__m128i *) (q + 32), + _mm_xor_si128(b2, _mm_loadu_si128((const __m128i *) (p + 32)))); + _mm_storeu_si128((__m128i *) (q + 48), + _mm_xor_si128(b3, _mm_loadu_si128((const __m128i *) (p + 48)))); + _mm_storeu_si128((__m128i *) (q + 64), + _mm_xor_si128(b4, _mm_loadu_si128((const __m128i *) (p + 64)))); + _mm_storeu_si128((__m128i *) (q + 80), + _mm_xor_si128(b5, _mm_loadu_si128((const __m128i *) (p + 80)))); + } + } + + /* the tail, with E(K,Y0) in a lane the length leaves idle */ + { + __m128i t0, t1, t2, t3, t4, t5; + int ntail = (int) ((inlen - done + 15) / 16), j; + + /* the counter is where the groups left it */ + t0 = t1 = t2 = t3 = t4 = t5 = ctrbase; + if (ntail > 0) CTRT(t0); + if (ntail > 1) CTRT(t1); + if (ntail > 2) CTRT(t2); + if (ntail > 3) CTRT(t3); + if (ntail > 4) CTRT(t4); + if (ntail > 5) CTRT(t5); + + if (ntail > 0) { + WIDE6_NOQ(rk); + if (lane_ek0) ek0 = t5; + } + + for (j = 0; j < ntail; j++) { + size_t off = done + 16 * (size_t) j; + size_t n = inlen - off < 16 ? inlen - off : 16; + __m128i c = n == 16 ? _mm_loadu_si128((const __m128i *) (in + off)) + : loadn(in + off, n); + __m128i ks = j == 0 ? t0 : j == 1 ? t1 : j == 2 ? t2 + : j == 3 ? t3 : j == 4 ? t4 : t5; + + GHASH_ONE(_mm_shuffle_epi8(c, BSWAP), 0); + { + __m128i pl = _mm_xor_si128(ks, c); + if (n == 16) + _mm_storeu_si128((__m128i *) (out + off), pl); + else + storen(out + off, pl, n); + } + } + } + + GHASH_ONE(_mm_set_epi64x((long long) ((uint64_t) aadlen << 3), + (long long) ((uint64_t) inlen << 3)), gp); + + want = _mm_xor_si128(_mm_shuffle_epi8(gtag, BSWAP), ek0); + { + uint8_t got[16]; + _mm_storeu_si128((__m128i *) got, want); + if (outtag) { + /* The caller holds the expected tag and will compare it itself. */ + memcpy(outtag, got, taglen); + return 1; + } + for (i = 0; i < taglen; i++) + diff |= (uint8_t) (got[i] ^ tag[i]); + } + return diff == 0; +} + +#endif /* WITH_GCM_FUSED */ diff --git a/cbits/aes/gcm_fused_x86.h b/cbits/aes/gcm_fused_x86.h new file mode 100644 --- /dev/null +++ b/cbits/aes/gcm_fused_x86.h @@ -0,0 +1,55 @@ +/* + * Copyright (c) 2026 Kazu Yamamoto + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of the author nor the names of his contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#ifndef CRYPTON_GCM_FUSED_X86_H +#define CRYPTON_GCM_FUSED_X86_H + +#include +#include +#include + +void crypton_gcm_fused_key_init(aes_gcm_fused *fk, const aes_key *key); + +void crypton_gcm_fused_encrypt(uint8_t *out, const aes_gcm_fused *fk, + const aes_key *key, + const uint8_t *nonce, + const uint8_t *aad, size_t aadlen, + const uint8_t *in, size_t inlen, size_t taglen, + const aes_key *hpkey, size_t sampleoff, + uint8_t *mask); + +int crypton_gcm_fused_decrypt(uint8_t *out, const aes_gcm_fused *fk, + const aes_key *key, const uint8_t *nonce, + const uint8_t *aad, size_t aadlen, + const uint8_t *in, size_t inlen, + const uint8_t *tag, size_t taglen, + uint8_t *outtag); + +#endif diff --git a/cbits/aes/gcm_vaes512_x86.c b/cbits/aes/gcm_vaes512_x86.c new file mode 100644 --- /dev/null +++ b/cbits/aes/gcm_vaes512_x86.c @@ -0,0 +1,364 @@ +/* + * AES-GCM through VAES and VPCLMULQDQ in their 512-bit form, which takes + * four blocks where the 256-bit form in cbits/aes/gcm_vaes_x86.c takes two + * and the 128-bit one takes one. The instruction rate is the same, so the + * work per group halves again. + * + * This is that file widened and nothing else: the same group of sixteen + * blocks, the same descending powers of H sharing one reduction, the same + * round keys read from memory rather than held in registers. Four blocks to + * a register means the group fills four of them rather than eight, which is + * what leaves room for the group's own ciphertext to be kept for the GHASH + * when encrypting. + * + * Nothing here is borrowed. OpenSSL's and BoringSSL's AVX-512 AES-GCM is + * Apache-2.0 and s2n-bignum has no GCM at all. + * + * The reduction at the end is a copy of the one in gcm_vaes_x86.c rather + * than a call to it: the two files are compiled for different instruction + * sets, and a function compiled for one cannot be inlined into the other. + */ +#include "aes/gcm_vaes512_x86.h" + +#ifdef WITH_GCM_VAES512 + +#include +#include + +#include +#include + +#if defined(__clang__) || defined(__GNUC__) +#define V512_TARGET \ + __attribute__((target("avx512f,avx512bw,avx512vl,aes,pclmul,vaes,vpclmulqdq"))) +#else +#define V512_TARGET +#endif + +/* + * Thirty-two blocks to a group, four to a register, so eight registers are + * in flight. The number of registers is what matters as much as the blocks + * per instruction: AES-NI has a latency of four cycles against a throughput + * of one, so it takes eight independent chains to keep two ports busy. Four + * registers of four blocks was written first and measured *slower* than the + * 256-bit path -- the blocks per instruction had doubled and the chains had + * halved. + * + * The table holds sixteen powers of H, so the GHASH of a group is two passes + * of sixteen blocks, the second picking up the tag the first leaves. + */ +#define V512WIDE 8 +#define V512HALF 4 +#define V512BYTES 512 + +/* + * The 128-bit multiply of cbits/aes/x86ni.c, done in all four lanes at once. + * Every shuffle and shift here works inside its own 128-bit lane, so the + * four products never mix: what comes out is four independent carry-less + * products, accumulated by the caller and reduced together at the end. + */ +V512_TARGET +static inline void clmul512(__m512i a, __m512i b, __m512i *lo, __m512i *hi) +{ + const __m512i bswap = _mm512_set4_epi32( + 0x00010203, 0x04050607, 0x08090a0b, 0x0c0d0e0f); + __m512i t3, t4, t5, t6; + + a = _mm512_shuffle_epi8(a, bswap); + + /* Karatsuba, as in the 128-bit one: three multiplies, not four */ + t3 = _mm512_clmulepi64_epi128(a, b, 0x00); + t6 = _mm512_clmulepi64_epi128(a, b, 0x11); + t4 = _mm512_clmulepi64_epi128( + _mm512_xor_si512(a, _mm512_shuffle_epi32(a, _MM_PERM_BADC)), + _mm512_xor_si512(b, _mm512_shuffle_epi32(b, _MM_PERM_BADC)), + 0x00); + t4 = _mm512_xor_si512(t4, _mm512_xor_si512(t3, t6)); + + t5 = _mm512_bslli_epi128(t4, 8); + t4 = _mm512_bsrli_epi128(t4, 8); + + *lo = _mm512_xor_si512(t3, t5); + *hi = _mm512_xor_si512(t6, t4); +} + +/* + * The reduction of cbits/aes/x86ni.c. By the time it runs the four lanes + * have been folded into one, so there is one 256-bit product to reduce. + */ +V512_TARGET +static inline __m128i gfred512(__m128i t3, __m128i t6) +{ + const __m128i bswap = _mm_set_epi8(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15); + __m128i t2, t4, t5, t7, t8, t9; + + t7 = _mm_srli_epi32(t3, 31); + t8 = _mm_srli_epi32(t6, 31); + t3 = _mm_slli_epi32(t3, 1); + t6 = _mm_slli_epi32(t6, 1); + + t9 = _mm_srli_si128(t7, 12); + t8 = _mm_slli_si128(t8, 4); + t7 = _mm_slli_si128(t7, 4); + t3 = _mm_or_si128(t3, t7); + t6 = _mm_or_si128(t6, t8); + t6 = _mm_or_si128(t6, t9); + + t7 = _mm_slli_epi32(t3, 31); + t8 = _mm_slli_epi32(t3, 30); + t9 = _mm_slli_epi32(t3, 25); + + t7 = _mm_xor_si128(t7, t8); + t7 = _mm_xor_si128(t7, t9); + t8 = _mm_srli_si128(t7, 4); + t7 = _mm_slli_si128(t7, 12); + t3 = _mm_xor_si128(t3, t7); + + t2 = _mm_srli_epi32(t3, 1); + t4 = _mm_srli_epi32(t3, 2); + t5 = _mm_srli_epi32(t3, 7); + t2 = _mm_xor_si128(t2, t4); + t2 = _mm_xor_si128(t2, t5); + t2 = _mm_xor_si128(t2, t8); + t3 = _mm_xor_si128(t3, t2); + t6 = _mm_xor_si128(t6, t3); + + return _mm_shuffle_epi8(t6, bswap); +} + +/* the four 128-bit lanes of a register added together */ +V512_TARGET +static inline __m128i fold512(__m512i v) +{ + __m256i h = _mm256_xor_si256(_mm512_castsi512_si256(v), + _mm512_extracti64x4_epi64(v, 1)); + + return _mm_xor_si128(_mm256_castsi256_si128(h), + _mm256_extracti128_si256(h, 1)); +} + +/* + * Sixteen blocks against H^16 .. H^1, one reduction. v[j] holds blocks 4j + * to 4j+3 in its four lanes, so the powers for it are H^(16-4j) down to + * H^(13-4j) -- the table's own order the other way round, hence the four + * 128-bit loads rather than one 512-bit one. + */ +V512_TARGET +static inline __m128i ghash16(__m128i tag, const table_4bit htable, + const __m512i *v, int fromwire) +{ + __m512i lo = _mm512_setzero_si512(), hi = _mm512_setzero_si512(); + __m512i l, h, b; + int j; + + for (j = 0; j < V512HALF; j++) { + const __m128i p0 = + _mm_loadu_si128((const __m128i *) &htable[15 - 4 * j]); + const __m128i p1 = + _mm_loadu_si128((const __m128i *) &htable[14 - 4 * j]); + const __m128i p2 = + _mm_loadu_si128((const __m128i *) &htable[13 - 4 * j]); + const __m128i p3 = + _mm_loadu_si128((const __m128i *) &htable[12 - 4 * j]); + __m512i hp = _mm512_castsi128_si512(p0); + + hp = _mm512_inserti32x4(hp, p1, 1); + hp = _mm512_inserti32x4(hp, p2, 2); + hp = _mm512_inserti32x4(hp, p3, 3); + + b = fromwire ? _mm512_loadu_si512(v + j) : v[j]; + if (j == 0) /* the running tag joins the first block */ + b = _mm512_xor_si512( + b, _mm512_inserti32x4( + _mm512_setzero_si512(), tag, 0)); + clmul512(b, hp, &l, &h); + lo = _mm512_xor_si512(lo, l); + hi = _mm512_xor_si512(hi, h); + } + + /* the four lanes are independent products of the same sum: fold them */ + return gfred512(fold512(lo), fold512(hi)); +} + +#define KK512(r) _mm512_broadcast_i32x4(_mm_loadu_si128(k_ + (r))) + +#define AESENC32(K) \ + do { \ + const __m512i rk = (K); \ + v[0] = _mm512_aesenc_epi128(v[0], rk); \ + v[1] = _mm512_aesenc_epi128(v[1], rk); \ + v[2] = _mm512_aesenc_epi128(v[2], rk); \ + v[3] = _mm512_aesenc_epi128(v[3], rk); \ + v[4] = _mm512_aesenc_epi128(v[4], rk); \ + v[5] = _mm512_aesenc_epi128(v[5], rk); \ + v[6] = _mm512_aesenc_epi128(v[6], rk); \ + v[7] = _mm512_aesenc_epi128(v[7], rk); \ + } while (0) + +#define AESLAST32(K) \ + do { \ + const __m512i rk = (K); \ + v[0] = _mm512_aesenclast_epi128(v[0], rk); \ + v[1] = _mm512_aesenclast_epi128(v[1], rk); \ + v[2] = _mm512_aesenclast_epi128(v[2], rk); \ + v[3] = _mm512_aesenclast_epi128(v[3], rk); \ + v[4] = _mm512_aesenclast_epi128(v[4], rk); \ + v[5] = _mm512_aesenclast_epi128(v[5], rk); \ + v[6] = _mm512_aesenclast_epi128(v[6], rk); \ + v[7] = _mm512_aesenclast_epi128(v[7], rk); \ + } while (0) + +#define XOR32(K) \ + do { \ + const __m512i rk = (K); \ + v[0] = _mm512_xor_si512(v[0], rk); \ + v[1] = _mm512_xor_si512(v[1], rk); \ + v[2] = _mm512_xor_si512(v[2], rk); \ + v[3] = _mm512_xor_si512(v[3], rk); \ + v[4] = _mm512_xor_si512(v[4], rk); \ + v[5] = _mm512_xor_si512(v[5], rk); \ + v[6] = _mm512_xor_si512(v[6], rk); \ + v[7] = _mm512_xor_si512(v[7], rk); \ + } while (0) + +/* + * The rounds are written out rather than looped for the reason the 128-bit + * loop gives: the count is a value in the key, and a loop over it leaves the + * round key reached through an index the compiler cannot fold. + */ +V512_TARGET +static inline __attribute__((always_inline)) void +rounds32(__m512i *v, const uint8_t *k, const int nbr) +{ + const __m128i *k_ = (const __m128i *) k; + + XOR32(KK512(0)); + AESENC32(KK512(1)); AESENC32(KK512(2)); AESENC32(KK512(3)); + AESENC32(KK512(4)); AESENC32(KK512(5)); AESENC32(KK512(6)); + AESENC32(KK512(7)); AESENC32(KK512(8)); AESENC32(KK512(9)); + if (nbr > 10) { + AESENC32(KK512(10)); AESENC32(KK512(11)); + if (nbr > 12) { + AESENC32(KK512(12)); AESENC32(KK512(13)); + } + } + AESLAST32(_mm512_broadcast_i32x4(_mm_loadu_si128(k_ + nbr))); +} + +/* sixteen consecutive counters, four to a register. GCM counts in the low + * thirty-two bits and wraps there, which is what _mm_add_epi32 does. */ +V512_TARGET +static inline __m128i counters32(__m512i *v, __m128i iv, __m128i one, + __m128i bswap) +{ + int j; + + for (j = 0; j < V512WIDE; j++) { + __m128i c0, c1, c2, c3; + __m512i c; + + iv = _mm_add_epi32(iv, one); + c0 = _mm_shuffle_epi8(iv, bswap); + iv = _mm_add_epi32(iv, one); + c1 = _mm_shuffle_epi8(iv, bswap); + iv = _mm_add_epi32(iv, one); + c2 = _mm_shuffle_epi8(iv, bswap); + iv = _mm_add_epi32(iv, one); + c3 = _mm_shuffle_epi8(iv, bswap); + + c = _mm512_castsi128_si512(c0); + c = _mm512_inserti32x4(c, c1, 1); + c = _mm512_inserti32x4(c, c2, 2); + c = _mm512_inserti32x4(c, c3, 3); + v[j] = c; + } + return iv; +} + +/* + * Inlined into three callers with the round count a constant in each, which + * folds away the tests inside the group loop -- the same reason the 256-bit + * file gives, where without it AES-256 lost what AES-128 gained. + */ +V512_TARGET +static inline __attribute__((always_inline)) uint32_t +bulk_n(uint8_t *output, aes_gcm *gcm, const aes_key *key, + const uint8_t *input, uint32_t length, int decrypt, const int nbr) +{ + const __m128i bswap = _mm_setr_epi8(7,6,5,4,3,2,1,0,15,14,13,12,11,10,9,8); + const __m128i one = _mm_set_epi32(0, 1, 0, 0); + __m512i v[V512WIDE]; + __m128i iv, tag; + uint32_t groups = length / V512BYTES; + uint32_t done = 0; + uint32_t g; + int j; + + if (groups == 0) + return 0; + + iv = _mm_shuffle_epi8(_mm_loadu_si128((const __m128i *) &gcm->civ), bswap); + tag = _mm_loadu_si128((const __m128i *) &gcm->tag); + + for (g = 0; g < groups; g++, input += V512BYTES, output += V512BYTES, + done += V512BYTES) { + iv = counters32(v, iv, one, bswap); + rounds32(v, key->data, nbr); + + for (j = 0; j < V512WIDE; j++) { + const __m512i in = + _mm512_loadu_si512((const __m512i *) (input + 64 * j)); + + v[j] = _mm512_xor_si512(v[j], in); + _mm512_storeu_si512((__m512i *) (output + 64 * j), v[j]); + } + /* sixteen blocks to a pass, since that is how many powers of + * H the table holds; the second picks up the tag the first + * leaves */ + tag = ghash16(tag, gcm->htable, + decrypt ? (const __m512i *) input : v, + decrypt); + tag = ghash16(tag, gcm->htable, + decrypt ? (const __m512i *) (input + 256) + : v + V512HALF, + decrypt); + } + + _mm_storeu_si128((__m128i *) &gcm->civ, _mm_shuffle_epi8(iv, bswap)); + _mm_storeu_si128((__m128i *) &gcm->tag, tag); + return done; +} + +V512_TARGET +static uint32_t bulk(uint8_t *output, aes_gcm *gcm, const aes_key *key, + const uint8_t *input, uint32_t length, int decrypt) +{ + switch (key->nbr) { + case 10: + return bulk_n(output, gcm, key, input, length, decrypt, 10); + case 12: + return bulk_n(output, gcm, key, input, length, decrypt, 12); + case 14: + return bulk_n(output, gcm, key, input, length, decrypt, 14); + default: + return 0; /* not a key length AES has */ + } +} + +uint32_t crypton_gcm_vaes512_bulk_encrypt(uint8_t *output, aes_gcm *gcm, + const aes_key *key, + const uint8_t *input, + uint32_t length) +{ + return bulk(output, gcm, key, input, length, 0); +} + +uint32_t crypton_gcm_vaes512_bulk_decrypt(uint8_t *output, aes_gcm *gcm, + const aes_key *key, + const uint8_t *input, + uint32_t length) +{ + return bulk(output, gcm, key, input, length, 1); +} + +#endif diff --git a/cbits/aes/gcm_vaes512_x86.h b/cbits/aes/gcm_vaes512_x86.h new file mode 100644 --- /dev/null +++ b/cbits/aes/gcm_vaes512_x86.h @@ -0,0 +1,37 @@ +/* + * AES-GCM in the 512-bit form of the AES and carry-less multiply + * instructions, which do four blocks where the 128-bit ones do one and the + * 256-bit ones in cbits/aes/gcm_vaes_x86.c do two. + */ +#ifndef CRYPTON_GCM_VAES512_X86_H +#define CRYPTON_GCM_VAES512_X86_H + +#include + +#if defined(ARCH_X86) && defined(__x86_64__) && defined(WITH_AESNI) \ + && defined(WITH_PCLMUL) +#define WITH_GCM_VAES512 +#endif + +#ifdef WITH_GCM_VAES512 + +#include +#include + +/* The same contract as the 256-bit pair: whole groups off the front, the + * counter left in gcm->civ and the running tag in gcm->tag, and the number + * of bytes taken returned, a multiple of 512 and possibly zero. */ +uint32_t crypton_gcm_vaes512_bulk_encrypt(uint8_t *output, aes_gcm *gcm, + const aes_key *key, + const uint8_t *input, + uint32_t length); +uint32_t crypton_gcm_vaes512_bulk_decrypt(uint8_t *output, aes_gcm *gcm, + const aes_key *key, + const uint8_t *input, + uint32_t length); + +/* Thirty-two blocks is the least it will start on. */ +#define GCM_VAES512_MIN_BLOCKS 32 + +#endif +#endif diff --git a/cbits/aes/gcm_vaes_x86.c b/cbits/aes/gcm_vaes_x86.c new file mode 100644 --- /dev/null +++ b/cbits/aes/gcm_vaes_x86.c @@ -0,0 +1,325 @@ +/* + * AES-GCM through VAES and VPCLMULQDQ: the same AES and carry-less multiply + * instructions the rest of this directory uses, in their 256-bit form, which + * takes two blocks where the 128-bit form takes one. The instruction rate is + * the same, so the throughput is twice -- measured at 2.00 on an EPYC 9V74, + * for both halves, with nothing else in the loop. + * + * Nothing here is borrowed. OpenSSL's and BoringSSL's wide AES-GCM is + * Apache-2.0, s2n-bignum has no GCM at all, and the CRYPTOGAMS assembly in + * cbits/asm is 128-bit throughout -- its `vaesenc` is the VEX encoding of + * AESENC on XMM, not the VAES extension. So this is the 128-bit loop in + * cbits/aes/x86ni_impl.c widened, and it keeps that loop's shape: a group of + * counters through the rounds together, the round keys read from memory + * rather than held in registers, and the group's GHASH folded against + * descending powers of H so that sixteen blocks share one reduction. + * + * The powers come from the table crypton_aesni_hinit_pclmul fills. It has + * sixteen slots and the 128-bit loop uses eight of them; this uses all + * sixteen, which is why that function now fills them. + */ +#include "aes/gcm_vaes_x86.h" + +#ifdef WITH_GCM_VAES + +#include +#include + +#include +#include + +#if defined(__clang__) || defined(__GNUC__) +#define VAES_TARGET __attribute__((target("avx2,aes,pclmul,vaes,vpclmulqdq"))) +#else +#define VAES_TARGET +#endif + +/* sixteen blocks to a group, two to a register */ +#define VWIDE 8 + +/* + * The 128-bit multiply of cbits/aes/x86ni.c, done in both lanes at once. + * Every shuffle and shift here works inside its own 128-bit half, so the two + * products never mix: what comes out is two independent carry-less products, + * accumulated by the caller and reduced together at the end. + */ +VAES_TARGET +static inline void clmul256(__m256i a, __m256i b, __m256i *lo, __m256i *hi) +{ + const __m256i bswap = _mm256_setr_epi8( + 15,14,13,12,11,10,9,8,7,6,5,4,3,2,1,0, + 15,14,13,12,11,10,9,8,7,6,5,4,3,2,1,0); + __m256i t3, t4, t5, t6; + + a = _mm256_shuffle_epi8(a, bswap); + + /* Karatsuba, as in the 128-bit one: three multiplies, not four */ + t3 = _mm256_clmulepi64_epi128(a, b, 0x00); + t6 = _mm256_clmulepi64_epi128(a, b, 0x11); + t4 = _mm256_clmulepi64_epi128( + _mm256_xor_si256(a, _mm256_shuffle_epi32(a, 0x4e)), + _mm256_xor_si256(b, _mm256_shuffle_epi32(b, 0x4e)), 0x00); + t4 = _mm256_xor_si256(t4, _mm256_xor_si256(t3, t6)); + + t5 = _mm256_slli_si256(t4, 8); + t4 = _mm256_srli_si256(t4, 8); + + *lo = _mm256_xor_si256(t3, t5); + *hi = _mm256_xor_si256(t6, t4); +} + +/* + * The reduction of cbits/aes/x86ni.c, unchanged: by the time it runs the two + * lanes have been folded into one, so there is one 256-bit product to reduce + * and no reason to do it twice. + */ +VAES_TARGET +static inline __m128i gfred(__m128i t3, __m128i t6) +{ + const __m128i bswap = _mm_set_epi8(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15); + __m128i t2, t4, t5, t7, t8, t9; + + t7 = _mm_srli_epi32(t3, 31); + t8 = _mm_srli_epi32(t6, 31); + t3 = _mm_slli_epi32(t3, 1); + t6 = _mm_slli_epi32(t6, 1); + + t9 = _mm_srli_si128(t7, 12); + t8 = _mm_slli_si128(t8, 4); + t7 = _mm_slli_si128(t7, 4); + t3 = _mm_or_si128(t3, t7); + t6 = _mm_or_si128(t6, t8); + t6 = _mm_or_si128(t6, t9); + + t7 = _mm_slli_epi32(t3, 31); + t8 = _mm_slli_epi32(t3, 30); + t9 = _mm_slli_epi32(t3, 25); + + t7 = _mm_xor_si128(t7, t8); + t7 = _mm_xor_si128(t7, t9); + t8 = _mm_srli_si128(t7, 4); + t7 = _mm_slli_si128(t7, 12); + t3 = _mm_xor_si128(t3, t7); + + t2 = _mm_srli_epi32(t3, 1); + t4 = _mm_srli_epi32(t3, 2); + t5 = _mm_srli_epi32(t3, 7); + t2 = _mm_xor_si128(t2, t4); + t2 = _mm_xor_si128(t2, t5); + t2 = _mm_xor_si128(t2, t8); + t3 = _mm_xor_si128(t3, t2); + t6 = _mm_xor_si128(t6, t3); + + return _mm_shuffle_epi8(t6, bswap); +} + +/* + * Sixteen blocks against H^16 .. H^1, one reduction. v[j] holds blocks 2j + * and 2j+1 in its low and high halves, so the powers for it are H^(16-2j) + * low and H^(15-2j) high -- the table's own order the other way round, hence + * the pair of 128-bit loads rather than one 256-bit one. + */ +VAES_TARGET +static inline __m128i ghash16(__m128i tag, const table_4bit htable, + const __m256i *v, int fromwire) +{ + __m256i lo = _mm256_setzero_si256(), hi = _mm256_setzero_si256(); + __m256i l, h, b; + int j; + + for (j = 0; j < VWIDE; j++) { + const __m256i hp = _mm256_set_m128i( + _mm_loadu_si128((const __m128i *) &htable[14 - 2 * j]), + _mm_loadu_si128((const __m128i *) &htable[15 - 2 * j])); + + b = fromwire ? _mm256_loadu_si256(v + j) : v[j]; + if (j == 0) /* the running tag joins the first block */ + b = _mm256_xor_si256( + b, _mm256_inserti128_si256( + _mm256_setzero_si256(), tag, 0)); + clmul256(b, hp, &l, &h); + lo = _mm256_xor_si256(lo, l); + hi = _mm256_xor_si256(hi, h); + } + + /* the two lanes are independent products of the same sum: fold them */ + return gfred(_mm_xor_si128(_mm256_castsi256_si128(lo), + _mm256_extracti128_si256(lo, 1)), + _mm_xor_si128(_mm256_castsi256_si128(hi), + _mm256_extracti128_si256(hi, 1))); +} + +#define KK(r) _mm256_broadcastsi128_si256(_mm_loadu_si128(k_ + (r))) + +#define AESENC16(K) \ + do { \ + const __m256i rk = (K); \ + v[0] = _mm256_aesenc_epi128(v[0], rk); \ + v[1] = _mm256_aesenc_epi128(v[1], rk); \ + v[2] = _mm256_aesenc_epi128(v[2], rk); \ + v[3] = _mm256_aesenc_epi128(v[3], rk); \ + v[4] = _mm256_aesenc_epi128(v[4], rk); \ + v[5] = _mm256_aesenc_epi128(v[5], rk); \ + v[6] = _mm256_aesenc_epi128(v[6], rk); \ + v[7] = _mm256_aesenc_epi128(v[7], rk); \ + } while (0) + +#define AESLAST16(K) \ + do { \ + const __m256i rk = (K); \ + v[0] = _mm256_aesenclast_epi128(v[0], rk); \ + v[1] = _mm256_aesenclast_epi128(v[1], rk); \ + v[2] = _mm256_aesenclast_epi128(v[2], rk); \ + v[3] = _mm256_aesenclast_epi128(v[3], rk); \ + v[4] = _mm256_aesenclast_epi128(v[4], rk); \ + v[5] = _mm256_aesenclast_epi128(v[5], rk); \ + v[6] = _mm256_aesenclast_epi128(v[6], rk); \ + v[7] = _mm256_aesenclast_epi128(v[7], rk); \ + } while (0) + +#define XOR16(K) \ + do { \ + const __m256i rk = (K); \ + v[0] = _mm256_xor_si256(v[0], rk); \ + v[1] = _mm256_xor_si256(v[1], rk); \ + v[2] = _mm256_xor_si256(v[2], rk); \ + v[3] = _mm256_xor_si256(v[3], rk); \ + v[4] = _mm256_xor_si256(v[4], rk); \ + v[5] = _mm256_xor_si256(v[5], rk); \ + v[6] = _mm256_xor_si256(v[6], rk); \ + v[7] = _mm256_xor_si256(v[7], rk); \ + } while (0) + +/* + * The rounds are written out rather than looped for the reason the 128-bit + * loop gives: the count is a value in the key, and a loop over it leaves the + * round key reached through an index the compiler cannot fold. + */ +VAES_TARGET +static inline __attribute__((always_inline)) void +rounds16(__m256i *v, const uint8_t *k, const int nbr) +{ + const __m128i *k_ = (const __m128i *) k; + + XOR16(KK(0)); + AESENC16(KK(1)); AESENC16(KK(2)); AESENC16(KK(3)); + AESENC16(KK(4)); AESENC16(KK(5)); AESENC16(KK(6)); + AESENC16(KK(7)); AESENC16(KK(8)); AESENC16(KK(9)); + if (nbr > 10) { + AESENC16(KK(10)); AESENC16(KK(11)); + if (nbr > 12) { + AESENC16(KK(12)); AESENC16(KK(13)); + } + } + AESLAST16(_mm256_broadcastsi128_si256(_mm_loadu_si128(k_ + nbr))); +} + +/* sixteen consecutive counters, two to a register. GCM counts in the low + * thirty-two bits and wraps there, which is what _mm_add_epi32 does. */ +VAES_TARGET +static inline __m128i counters16(__m256i *v, __m128i iv, __m128i one, + __m128i bswap) +{ + int j; + + for (j = 0; j < VWIDE; j++) { + __m128i c0, c1; + + iv = _mm_add_epi32(iv, one); + c0 = _mm_shuffle_epi8(iv, bswap); + iv = _mm_add_epi32(iv, one); + c1 = _mm_shuffle_epi8(iv, bswap); + v[j] = _mm256_set_m128i(c1, c0); + } + return iv; +} + +/* + * The round count is a value in the key, and a test on it inside the group + * loop is a branch the 128-bit path does not have: that one compiles a + * separate function for each key length through the SIZED macro. This does + * the same thing by being inlined into three callers with the count a + * constant in each, which folds the tests away. Without it AES-256 lost + * what AES-128 gained. + */ +VAES_TARGET +static inline __attribute__((always_inline)) uint32_t +bulk_n(uint8_t *output, aes_gcm *gcm, const aes_key *key, + const uint8_t *input, uint32_t length, int decrypt, const int nbr) +{ + const __m128i bswap = _mm_setr_epi8(7,6,5,4,3,2,1,0,15,14,13,12,11,10,9,8); + const __m128i one = _mm_set_epi32(0, 1, 0, 0); + __m256i v[VWIDE]; + __m128i iv, tag; + uint32_t groups = length / 256; + uint32_t done = 0; + uint32_t g; + int j; + + if (groups == 0) + return 0; + + iv = _mm_shuffle_epi8(_mm_loadu_si128((const __m128i *) &gcm->civ), bswap); + tag = _mm_loadu_si128((const __m128i *) &gcm->tag); + + for (g = 0; g < groups; g++, input += 256, output += 256, done += 256) { + iv = counters16(v, iv, one, bswap); + rounds16(v, key->data, nbr); + + /* + * The ciphertext is what the tag is taken over, and after + * this exclusive or it is in v itself when encrypting. When + * decrypting it is the input, which the GHASH below reads + * again rather than keeping: there are sixteen vector + * registers, the group fills eight of them, and a second + * eight held aside is what makes the compiler spill. The + * input is in L1 from the load a moment ago. + */ + for (j = 0; j < VWIDE; j++) { + const __m256i in = + _mm256_loadu_si256((const __m256i *) (input + 32 * j)); + + v[j] = _mm256_xor_si256(v[j], in); + _mm256_storeu_si256((__m256i *) (output + 32 * j), v[j]); + } + tag = ghash16(tag, gcm->htable, + decrypt ? (const __m256i *) input : v, + decrypt); + } + + _mm_storeu_si128((__m128i *) &gcm->civ, _mm_shuffle_epi8(iv, bswap)); + _mm_storeu_si128((__m128i *) &gcm->tag, tag); + return done; +} + +VAES_TARGET +static uint32_t bulk(uint8_t *output, aes_gcm *gcm, const aes_key *key, + const uint8_t *input, uint32_t length, int decrypt) +{ + switch (key->nbr) { + case 10: + return bulk_n(output, gcm, key, input, length, decrypt, 10); + case 12: + return bulk_n(output, gcm, key, input, length, decrypt, 12); + case 14: + return bulk_n(output, gcm, key, input, length, decrypt, 14); + default: + return 0; /* not a key length AES has */ + } +} + +uint32_t crypton_gcm_vaes_bulk_encrypt(uint8_t *output, aes_gcm *gcm, + const aes_key *key, + const uint8_t *input, uint32_t length) +{ + return bulk(output, gcm, key, input, length, 0); +} + +uint32_t crypton_gcm_vaes_bulk_decrypt(uint8_t *output, aes_gcm *gcm, + const aes_key *key, + const uint8_t *input, uint32_t length) +{ + return bulk(output, gcm, key, input, length, 1); +} + +#endif diff --git a/cbits/aes/gcm_vaes_x86.h b/cbits/aes/gcm_vaes_x86.h new file mode 100644 --- /dev/null +++ b/cbits/aes/gcm_vaes_x86.h @@ -0,0 +1,35 @@ +/* + * AES-GCM in the 256-bit form of the AES and carry-less multiply + * instructions, which do two blocks where the 128-bit ones do one. + */ +#ifndef CRYPTON_GCM_VAES_X86_H +#define CRYPTON_GCM_VAES_X86_H + +#include + +#if defined(ARCH_X86) && defined(__x86_64__) && defined(WITH_AESNI) \ + && defined(WITH_PCLMUL) +#define WITH_GCM_VAES +#endif + +#ifdef WITH_GCM_VAES + +#include +#include + +/* The bulk of a message in whole groups of sixteen blocks, leaving the + * counter in gcm->civ and the running tag in gcm->tag where the caller's own + * loop expects to find them. Returns the number of bytes taken, which is a + * multiple of 256 and may be zero. */ +uint32_t crypton_gcm_vaes_bulk_encrypt(uint8_t *output, aes_gcm *gcm, + const aes_key *key, + const uint8_t *input, uint32_t length); +uint32_t crypton_gcm_vaes_bulk_decrypt(uint8_t *output, aes_gcm *gcm, + const aes_key *key, + const uint8_t *input, uint32_t length); + +/* Sixteen blocks is the least it will start on. */ +#define GCM_VAES_MIN_BLOCKS 16 + +#endif +#endif diff --git a/cbits/aes/gcm_x86_asm.c b/cbits/aes/gcm_x86_asm.c --- a/cbits/aes/gcm_x86_asm.c +++ b/cbits/aes/gcm_x86_asm.c @@ -92,19 +92,29 @@ * H, and every power of it, is kept shifted up by one bit: GCM numbers the * bits of a field element the other way round from the way the carry-less * multiply does, and pre-shifting the operand is what saves the correction - * after each multiply. The bit that falls off the top is the one that the - * polynomial reduces. + * after each multiply. + * + * Written from the definition. The field is GF(2)[x] modulo x^128 + x^127 + + * x^126 + x^121 + 1, so multiplying by x is a shift of one place, and the + * term that leaves the top comes back as the other four. */ TARGET_PCLMUL static __m128i twist(__m128i h) { + /* x^127 + x^126 + x^121 + 1, the terms x^128 is congruent to */ const __m128i poly = _mm_set_epi64x(0xc200000000000000ULL, 1); - __m128i carried = _mm_slli_si128(_mm_srli_epi64(h, 63), 8); - __m128i top = _mm_shuffle_epi32(h, 0xff); - __m128i reduce = _mm_cmpgt_epi32(_mm_setzero_si128(), top); + /* the top bit of each half */ + __m128i tops = _mm_srli_epi64(h, 63); + /* doubling a polynomial is a shift by one: each half doubles, and the + * low half's top bit becomes the high half's bottom bit */ + __m128i doubled = _mm_or_si128(_mm_add_epi64(h, h), + _mm_slli_si128(tops, 8)); + /* bit 127 is the one that leaves the field; spread it to a mask by + * subtracting it from zero, and it brings the four terms back */ + __m128i mask = _mm_sub_epi64(_mm_setzero_si128(), + _mm_unpackhi_epi64(tops, tops)); - h = _mm_or_si128(_mm_slli_epi64(h, 1), carried); - return _mm_xor_si128(h, _mm_and_si128(reduce, poly)); + return _mm_xor_si128(doubled, _mm_and_si128(mask, poly)); } /* the two halves of a value added together, which is the term Karatsuba diff --git a/cbits/aes/generic.c b/cbits/aes/generic.c --- a/cbits/aes/generic.c +++ b/cbits/aes/generic.c @@ -27,420 +27,191 @@ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * - * AES implementation + * AES, portable, for machines whose processor has no AES instructions or + * whose build was not given them. + * + * This was a table-driven implementation: an S-box indexed by a byte of the + * state, in every round and in the key expansion, which is what made it fast + * and what made it variable-time. It is now BearSSL's bitsliced aes_ct64, + * which holds four blocks interleaved across eight 64-bit words and computes + * the S-box as boolean algebra -- no table, and so no address derived from a + * secret. See cbits/bearssl/README.md. + * + * What is here is only the glue. Nothing in this file branches or indexes on + * the key or the data. */ #include -#include +#include #include -#include - -static uint8_t sbox[256] = { - 0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5, 0x30, 0x01, 0x67, 0x2b, 0xfe, - 0xd7, 0xab, 0x76, 0xca, 0x82, 0xc9, 0x7d, 0xfa, 0x59, 0x47, 0xf0, 0xad, 0xd4, - 0xa2, 0xaf, 0x9c, 0xa4, 0x72, 0xc0, 0xb7, 0xfd, 0x93, 0x26, 0x36, 0x3f, 0xf7, - 0xcc, 0x34, 0xa5, 0xe5, 0xf1, 0x71, 0xd8, 0x31, 0x15, 0x04, 0xc7, 0x23, 0xc3, - 0x18, 0x96, 0x05, 0x9a, 0x07, 0x12, 0x80, 0xe2, 0xeb, 0x27, 0xb2, 0x75, 0x09, - 0x83, 0x2c, 0x1a, 0x1b, 0x6e, 0x5a, 0xa0, 0x52, 0x3b, 0xd6, 0xb3, 0x29, 0xe3, - 0x2f, 0x84, 0x53, 0xd1, 0x00, 0xed, 0x20, 0xfc, 0xb1, 0x5b, 0x6a, 0xcb, 0xbe, - 0x39, 0x4a, 0x4c, 0x58, 0xcf, 0xd0, 0xef, 0xaa, 0xfb, 0x43, 0x4d, 0x33, 0x85, - 0x45, 0xf9, 0x02, 0x7f, 0x50, 0x3c, 0x9f, 0xa8, 0x51, 0xa3, 0x40, 0x8f, 0x92, - 0x9d, 0x38, 0xf5, 0xbc, 0xb6, 0xda, 0x21, 0x10, 0xff, 0xf3, 0xd2, 0xcd, 0x0c, - 0x13, 0xec, 0x5f, 0x97, 0x44, 0x17, 0xc4, 0xa7, 0x7e, 0x3d, 0x64, 0x5d, 0x19, - 0x73, 0x60, 0x81, 0x4f, 0xdc, 0x22, 0x2a, 0x90, 0x88, 0x46, 0xee, 0xb8, 0x14, - 0xde, 0x5e, 0x0b, 0xdb, 0xe0, 0x32, 0x3a, 0x0a, 0x49, 0x06, 0x24, 0x5c, 0xc2, - 0xd3, 0xac, 0x62, 0x91, 0x95, 0xe4, 0x79, 0xe7, 0xc8, 0x37, 0x6d, 0x8d, 0xd5, - 0x4e, 0xa9, 0x6c, 0x56, 0xf4, 0xea, 0x65, 0x7a, 0xae, 0x08, 0xba, 0x78, 0x25, - 0x2e, 0x1c, 0xa6, 0xb4, 0xc6, 0xe8, 0xdd, 0x74, 0x1f, 0x4b, 0xbd, 0x8b, 0x8a, - 0x70, 0x3e, 0xb5, 0x66, 0x48, 0x03, 0xf6, 0x0e, 0x61, 0x35, 0x57, 0xb9, 0x86, - 0xc1, 0x1d, 0x9e, 0xe1, 0xf8, 0x98, 0x11, 0x69, 0xd9, 0x8e, 0x94, 0x9b, 0x1e, - 0x87, 0xe9, 0xce, 0x55, 0x28, 0xdf, 0x8c, 0xa1, 0x89, 0x0d, 0xbf, 0xe6, 0x42, - 0x68, 0x41, 0x99, 0x2d, 0x0f, 0xb0, 0x54, 0xbb, 0x16 -}; - -static uint8_t rsbox[256] = { - 0x52, 0x09, 0x6a, 0xd5, 0x30, 0x36, 0xa5, 0x38, 0xbf, 0x40, 0xa3, 0x9e, 0x81, - 0xf3, 0xd7, 0xfb, 0x7c, 0xe3, 0x39, 0x82, 0x9b, 0x2f, 0xff, 0x87, 0x34, 0x8e, - 0x43, 0x44, 0xc4, 0xde, 0xe9, 0xcb, 0x54, 0x7b, 0x94, 0x32, 0xa6, 0xc2, 0x23, - 0x3d, 0xee, 0x4c, 0x95, 0x0b, 0x42, 0xfa, 0xc3, 0x4e, 0x08, 0x2e, 0xa1, 0x66, - 0x28, 0xd9, 0x24, 0xb2, 0x76, 0x5b, 0xa2, 0x49, 0x6d, 0x8b, 0xd1, 0x25, 0x72, - 0xf8, 0xf6, 0x64, 0x86, 0x68, 0x98, 0x16, 0xd4, 0xa4, 0x5c, 0xcc, 0x5d, 0x65, - 0xb6, 0x92, 0x6c, 0x70, 0x48, 0x50, 0xfd, 0xed, 0xb9, 0xda, 0x5e, 0x15, 0x46, - 0x57, 0xa7, 0x8d, 0x9d, 0x84, 0x90, 0xd8, 0xab, 0x00, 0x8c, 0xbc, 0xd3, 0x0a, - 0xf7, 0xe4, 0x58, 0x05, 0xb8, 0xb3, 0x45, 0x06, 0xd0, 0x2c, 0x1e, 0x8f, 0xca, - 0x3f, 0x0f, 0x02, 0xc1, 0xaf, 0xbd, 0x03, 0x01, 0x13, 0x8a, 0x6b, 0x3a, 0x91, - 0x11, 0x41, 0x4f, 0x67, 0xdc, 0xea, 0x97, 0xf2, 0xcf, 0xce, 0xf0, 0xb4, 0xe6, - 0x73, 0x96, 0xac, 0x74, 0x22, 0xe7, 0xad, 0x35, 0x85, 0xe2, 0xf9, 0x37, 0xe8, - 0x1c, 0x75, 0xdf, 0x6e, 0x47, 0xf1, 0x1a, 0x71, 0x1d, 0x29, 0xc5, 0x89, 0x6f, - 0xb7, 0x62, 0x0e, 0xaa, 0x18, 0xbe, 0x1b, 0xfc, 0x56, 0x3e, 0x4b, 0xc6, 0xd2, - 0x79, 0x20, 0x9a, 0xdb, 0xc0, 0xfe, 0x78, 0xcd, 0x5a, 0xf4, 0x1f, 0xdd, 0xa8, - 0x33, 0x88, 0x07, 0xc7, 0x31, 0xb1, 0x12, 0x10, 0x59, 0x27, 0x80, 0xec, 0x5f, - 0x60, 0x51, 0x7f, 0xa9, 0x19, 0xb5, 0x4a, 0x0d, 0x2d, 0xe5, 0x7a, 0x9f, 0x93, - 0xc9, 0x9c, 0xef, 0xa0, 0xe0, 0x3b, 0x4d, 0xae, 0x2a, 0xf5, 0xb0, 0xc8, 0xeb, - 0xbb, 0x3c, 0x83, 0x53, 0x99, 0x61, 0x17, 0x2b, 0x04, 0x7e, 0xba, 0x77, 0xd6, - 0x26, 0xe1, 0x69, 0x14, 0x63, 0x55, 0x21, 0x0c, 0x7d -}; - -static uint8_t Rcon[] = { - 0x8d, 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36, 0x6c, 0xd8, - 0xab, 0x4d, 0x9a, 0x2f, 0x5e, 0xbc, 0x63, 0xc6, 0x97, 0x35, 0x6a, 0xd4, 0xb3, - 0x7d, 0xfa, 0xef, 0xc5, 0x91, 0x39, 0x72, 0xe4, 0xd3, 0xbd, 0x61, 0xc2, 0x9f, - 0x25, 0x4a, 0x94, 0x33, 0x66, 0xcc, 0x83, 0x1d, 0x3a, 0x74, 0xe8, 0xcb, -}; +#include "bearssl/inner.h" +#include "aes/block128.h" +#include "aes/generic.h" -#define G(a,b,c,d,e,f) { a,b,c,d,e,f } -static uint8_t gmtab[256][6] = -{ - G(0x00, 0x00, 0x00, 0x00, 0x00, 0x00), G(0x02, 0x03, 0x09, 0x0b, 0x0d, 0x0e), - G(0x04, 0x06, 0x12, 0x16, 0x1a, 0x1c), G(0x06, 0x05, 0x1b, 0x1d, 0x17, 0x12), - G(0x08, 0x0c, 0x24, 0x2c, 0x34, 0x38), G(0x0a, 0x0f, 0x2d, 0x27, 0x39, 0x36), - G(0x0c, 0x0a, 0x36, 0x3a, 0x2e, 0x24), G(0x0e, 0x09, 0x3f, 0x31, 0x23, 0x2a), - G(0x10, 0x18, 0x48, 0x58, 0x68, 0x70), G(0x12, 0x1b, 0x41, 0x53, 0x65, 0x7e), - G(0x14, 0x1e, 0x5a, 0x4e, 0x72, 0x6c), G(0x16, 0x1d, 0x53, 0x45, 0x7f, 0x62), - G(0x18, 0x14, 0x6c, 0x74, 0x5c, 0x48), G(0x1a, 0x17, 0x65, 0x7f, 0x51, 0x46), - G(0x1c, 0x12, 0x7e, 0x62, 0x46, 0x54), G(0x1e, 0x11, 0x77, 0x69, 0x4b, 0x5a), - G(0x20, 0x30, 0x90, 0xb0, 0xd0, 0xe0), G(0x22, 0x33, 0x99, 0xbb, 0xdd, 0xee), - G(0x24, 0x36, 0x82, 0xa6, 0xca, 0xfc), G(0x26, 0x35, 0x8b, 0xad, 0xc7, 0xf2), - G(0x28, 0x3c, 0xb4, 0x9c, 0xe4, 0xd8), G(0x2a, 0x3f, 0xbd, 0x97, 0xe9, 0xd6), - G(0x2c, 0x3a, 0xa6, 0x8a, 0xfe, 0xc4), G(0x2e, 0x39, 0xaf, 0x81, 0xf3, 0xca), - G(0x30, 0x28, 0xd8, 0xe8, 0xb8, 0x90), G(0x32, 0x2b, 0xd1, 0xe3, 0xb5, 0x9e), - G(0x34, 0x2e, 0xca, 0xfe, 0xa2, 0x8c), G(0x36, 0x2d, 0xc3, 0xf5, 0xaf, 0x82), - G(0x38, 0x24, 0xfc, 0xc4, 0x8c, 0xa8), G(0x3a, 0x27, 0xf5, 0xcf, 0x81, 0xa6), - G(0x3c, 0x22, 0xee, 0xd2, 0x96, 0xb4), G(0x3e, 0x21, 0xe7, 0xd9, 0x9b, 0xba), - G(0x40, 0x60, 0x3b, 0x7b, 0xbb, 0xdb), G(0x42, 0x63, 0x32, 0x70, 0xb6, 0xd5), - G(0x44, 0x66, 0x29, 0x6d, 0xa1, 0xc7), G(0x46, 0x65, 0x20, 0x66, 0xac, 0xc9), - G(0x48, 0x6c, 0x1f, 0x57, 0x8f, 0xe3), G(0x4a, 0x6f, 0x16, 0x5c, 0x82, 0xed), - G(0x4c, 0x6a, 0x0d, 0x41, 0x95, 0xff), G(0x4e, 0x69, 0x04, 0x4a, 0x98, 0xf1), - G(0x50, 0x78, 0x73, 0x23, 0xd3, 0xab), G(0x52, 0x7b, 0x7a, 0x28, 0xde, 0xa5), - G(0x54, 0x7e, 0x61, 0x35, 0xc9, 0xb7), G(0x56, 0x7d, 0x68, 0x3e, 0xc4, 0xb9), - G(0x58, 0x74, 0x57, 0x0f, 0xe7, 0x93), G(0x5a, 0x77, 0x5e, 0x04, 0xea, 0x9d), - G(0x5c, 0x72, 0x45, 0x19, 0xfd, 0x8f), G(0x5e, 0x71, 0x4c, 0x12, 0xf0, 0x81), - G(0x60, 0x50, 0xab, 0xcb, 0x6b, 0x3b), G(0x62, 0x53, 0xa2, 0xc0, 0x66, 0x35), - G(0x64, 0x56, 0xb9, 0xdd, 0x71, 0x27), G(0x66, 0x55, 0xb0, 0xd6, 0x7c, 0x29), - G(0x68, 0x5c, 0x8f, 0xe7, 0x5f, 0x03), G(0x6a, 0x5f, 0x86, 0xec, 0x52, 0x0d), - G(0x6c, 0x5a, 0x9d, 0xf1, 0x45, 0x1f), G(0x6e, 0x59, 0x94, 0xfa, 0x48, 0x11), - G(0x70, 0x48, 0xe3, 0x93, 0x03, 0x4b), G(0x72, 0x4b, 0xea, 0x98, 0x0e, 0x45), - G(0x74, 0x4e, 0xf1, 0x85, 0x19, 0x57), G(0x76, 0x4d, 0xf8, 0x8e, 0x14, 0x59), - G(0x78, 0x44, 0xc7, 0xbf, 0x37, 0x73), G(0x7a, 0x47, 0xce, 0xb4, 0x3a, 0x7d), - G(0x7c, 0x42, 0xd5, 0xa9, 0x2d, 0x6f), G(0x7e, 0x41, 0xdc, 0xa2, 0x20, 0x61), - G(0x80, 0xc0, 0x76, 0xf6, 0x6d, 0xad), G(0x82, 0xc3, 0x7f, 0xfd, 0x60, 0xa3), - G(0x84, 0xc6, 0x64, 0xe0, 0x77, 0xb1), G(0x86, 0xc5, 0x6d, 0xeb, 0x7a, 0xbf), - G(0x88, 0xcc, 0x52, 0xda, 0x59, 0x95), G(0x8a, 0xcf, 0x5b, 0xd1, 0x54, 0x9b), - G(0x8c, 0xca, 0x40, 0xcc, 0x43, 0x89), G(0x8e, 0xc9, 0x49, 0xc7, 0x4e, 0x87), - G(0x90, 0xd8, 0x3e, 0xae, 0x05, 0xdd), G(0x92, 0xdb, 0x37, 0xa5, 0x08, 0xd3), - G(0x94, 0xde, 0x2c, 0xb8, 0x1f, 0xc1), G(0x96, 0xdd, 0x25, 0xb3, 0x12, 0xcf), - G(0x98, 0xd4, 0x1a, 0x82, 0x31, 0xe5), G(0x9a, 0xd7, 0x13, 0x89, 0x3c, 0xeb), - G(0x9c, 0xd2, 0x08, 0x94, 0x2b, 0xf9), G(0x9e, 0xd1, 0x01, 0x9f, 0x26, 0xf7), - G(0xa0, 0xf0, 0xe6, 0x46, 0xbd, 0x4d), G(0xa2, 0xf3, 0xef, 0x4d, 0xb0, 0x43), - G(0xa4, 0xf6, 0xf4, 0x50, 0xa7, 0x51), G(0xa6, 0xf5, 0xfd, 0x5b, 0xaa, 0x5f), - G(0xa8, 0xfc, 0xc2, 0x6a, 0x89, 0x75), G(0xaa, 0xff, 0xcb, 0x61, 0x84, 0x7b), - G(0xac, 0xfa, 0xd0, 0x7c, 0x93, 0x69), G(0xae, 0xf9, 0xd9, 0x77, 0x9e, 0x67), - G(0xb0, 0xe8, 0xae, 0x1e, 0xd5, 0x3d), G(0xb2, 0xeb, 0xa7, 0x15, 0xd8, 0x33), - G(0xb4, 0xee, 0xbc, 0x08, 0xcf, 0x21), G(0xb6, 0xed, 0xb5, 0x03, 0xc2, 0x2f), - G(0xb8, 0xe4, 0x8a, 0x32, 0xe1, 0x05), G(0xba, 0xe7, 0x83, 0x39, 0xec, 0x0b), - G(0xbc, 0xe2, 0x98, 0x24, 0xfb, 0x19), G(0xbe, 0xe1, 0x91, 0x2f, 0xf6, 0x17), - G(0xc0, 0xa0, 0x4d, 0x8d, 0xd6, 0x76), G(0xc2, 0xa3, 0x44, 0x86, 0xdb, 0x78), - G(0xc4, 0xa6, 0x5f, 0x9b, 0xcc, 0x6a), G(0xc6, 0xa5, 0x56, 0x90, 0xc1, 0x64), - G(0xc8, 0xac, 0x69, 0xa1, 0xe2, 0x4e), G(0xca, 0xaf, 0x60, 0xaa, 0xef, 0x40), - G(0xcc, 0xaa, 0x7b, 0xb7, 0xf8, 0x52), G(0xce, 0xa9, 0x72, 0xbc, 0xf5, 0x5c), - G(0xd0, 0xb8, 0x05, 0xd5, 0xbe, 0x06), G(0xd2, 0xbb, 0x0c, 0xde, 0xb3, 0x08), - G(0xd4, 0xbe, 0x17, 0xc3, 0xa4, 0x1a), G(0xd6, 0xbd, 0x1e, 0xc8, 0xa9, 0x14), - G(0xd8, 0xb4, 0x21, 0xf9, 0x8a, 0x3e), G(0xda, 0xb7, 0x28, 0xf2, 0x87, 0x30), - G(0xdc, 0xb2, 0x33, 0xef, 0x90, 0x22), G(0xde, 0xb1, 0x3a, 0xe4, 0x9d, 0x2c), - G(0xe0, 0x90, 0xdd, 0x3d, 0x06, 0x96), G(0xe2, 0x93, 0xd4, 0x36, 0x0b, 0x98), - G(0xe4, 0x96, 0xcf, 0x2b, 0x1c, 0x8a), G(0xe6, 0x95, 0xc6, 0x20, 0x11, 0x84), - G(0xe8, 0x9c, 0xf9, 0x11, 0x32, 0xae), G(0xea, 0x9f, 0xf0, 0x1a, 0x3f, 0xa0), - G(0xec, 0x9a, 0xeb, 0x07, 0x28, 0xb2), G(0xee, 0x99, 0xe2, 0x0c, 0x25, 0xbc), - G(0xf0, 0x88, 0x95, 0x65, 0x6e, 0xe6), G(0xf2, 0x8b, 0x9c, 0x6e, 0x63, 0xe8), - G(0xf4, 0x8e, 0x87, 0x73, 0x74, 0xfa), G(0xf6, 0x8d, 0x8e, 0x78, 0x79, 0xf4), - G(0xf8, 0x84, 0xb1, 0x49, 0x5a, 0xde), G(0xfa, 0x87, 0xb8, 0x42, 0x57, 0xd0), - G(0xfc, 0x82, 0xa3, 0x5f, 0x40, 0xc2), G(0xfe, 0x81, 0xaa, 0x54, 0x4d, 0xcc), - G(0x1b, 0x9b, 0xec, 0xf7, 0xda, 0x41), G(0x19, 0x98, 0xe5, 0xfc, 0xd7, 0x4f), - G(0x1f, 0x9d, 0xfe, 0xe1, 0xc0, 0x5d), G(0x1d, 0x9e, 0xf7, 0xea, 0xcd, 0x53), - G(0x13, 0x97, 0xc8, 0xdb, 0xee, 0x79), G(0x11, 0x94, 0xc1, 0xd0, 0xe3, 0x77), - G(0x17, 0x91, 0xda, 0xcd, 0xf4, 0x65), G(0x15, 0x92, 0xd3, 0xc6, 0xf9, 0x6b), - G(0x0b, 0x83, 0xa4, 0xaf, 0xb2, 0x31), G(0x09, 0x80, 0xad, 0xa4, 0xbf, 0x3f), - G(0x0f, 0x85, 0xb6, 0xb9, 0xa8, 0x2d), G(0x0d, 0x86, 0xbf, 0xb2, 0xa5, 0x23), - G(0x03, 0x8f, 0x80, 0x83, 0x86, 0x09), G(0x01, 0x8c, 0x89, 0x88, 0x8b, 0x07), - G(0x07, 0x89, 0x92, 0x95, 0x9c, 0x15), G(0x05, 0x8a, 0x9b, 0x9e, 0x91, 0x1b), - G(0x3b, 0xab, 0x7c, 0x47, 0x0a, 0xa1), G(0x39, 0xa8, 0x75, 0x4c, 0x07, 0xaf), - G(0x3f, 0xad, 0x6e, 0x51, 0x10, 0xbd), G(0x3d, 0xae, 0x67, 0x5a, 0x1d, 0xb3), - G(0x33, 0xa7, 0x58, 0x6b, 0x3e, 0x99), G(0x31, 0xa4, 0x51, 0x60, 0x33, 0x97), - G(0x37, 0xa1, 0x4a, 0x7d, 0x24, 0x85), G(0x35, 0xa2, 0x43, 0x76, 0x29, 0x8b), - G(0x2b, 0xb3, 0x34, 0x1f, 0x62, 0xd1), G(0x29, 0xb0, 0x3d, 0x14, 0x6f, 0xdf), - G(0x2f, 0xb5, 0x26, 0x09, 0x78, 0xcd), G(0x2d, 0xb6, 0x2f, 0x02, 0x75, 0xc3), - G(0x23, 0xbf, 0x10, 0x33, 0x56, 0xe9), G(0x21, 0xbc, 0x19, 0x38, 0x5b, 0xe7), - G(0x27, 0xb9, 0x02, 0x25, 0x4c, 0xf5), G(0x25, 0xba, 0x0b, 0x2e, 0x41, 0xfb), - G(0x5b, 0xfb, 0xd7, 0x8c, 0x61, 0x9a), G(0x59, 0xf8, 0xde, 0x87, 0x6c, 0x94), - G(0x5f, 0xfd, 0xc5, 0x9a, 0x7b, 0x86), G(0x5d, 0xfe, 0xcc, 0x91, 0x76, 0x88), - G(0x53, 0xf7, 0xf3, 0xa0, 0x55, 0xa2), G(0x51, 0xf4, 0xfa, 0xab, 0x58, 0xac), - G(0x57, 0xf1, 0xe1, 0xb6, 0x4f, 0xbe), G(0x55, 0xf2, 0xe8, 0xbd, 0x42, 0xb0), - G(0x4b, 0xe3, 0x9f, 0xd4, 0x09, 0xea), G(0x49, 0xe0, 0x96, 0xdf, 0x04, 0xe4), - G(0x4f, 0xe5, 0x8d, 0xc2, 0x13, 0xf6), G(0x4d, 0xe6, 0x84, 0xc9, 0x1e, 0xf8), - G(0x43, 0xef, 0xbb, 0xf8, 0x3d, 0xd2), G(0x41, 0xec, 0xb2, 0xf3, 0x30, 0xdc), - G(0x47, 0xe9, 0xa9, 0xee, 0x27, 0xce), G(0x45, 0xea, 0xa0, 0xe5, 0x2a, 0xc0), - G(0x7b, 0xcb, 0x47, 0x3c, 0xb1, 0x7a), G(0x79, 0xc8, 0x4e, 0x37, 0xbc, 0x74), - G(0x7f, 0xcd, 0x55, 0x2a, 0xab, 0x66), G(0x7d, 0xce, 0x5c, 0x21, 0xa6, 0x68), - G(0x73, 0xc7, 0x63, 0x10, 0x85, 0x42), G(0x71, 0xc4, 0x6a, 0x1b, 0x88, 0x4c), - G(0x77, 0xc1, 0x71, 0x06, 0x9f, 0x5e), G(0x75, 0xc2, 0x78, 0x0d, 0x92, 0x50), - G(0x6b, 0xd3, 0x0f, 0x64, 0xd9, 0x0a), G(0x69, 0xd0, 0x06, 0x6f, 0xd4, 0x04), - G(0x6f, 0xd5, 0x1d, 0x72, 0xc3, 0x16), G(0x6d, 0xd6, 0x14, 0x79, 0xce, 0x18), - G(0x63, 0xdf, 0x2b, 0x48, 0xed, 0x32), G(0x61, 0xdc, 0x22, 0x43, 0xe0, 0x3c), - G(0x67, 0xd9, 0x39, 0x5e, 0xf7, 0x2e), G(0x65, 0xda, 0x30, 0x55, 0xfa, 0x20), - G(0x9b, 0x5b, 0x9a, 0x01, 0xb7, 0xec), G(0x99, 0x58, 0x93, 0x0a, 0xba, 0xe2), - G(0x9f, 0x5d, 0x88, 0x17, 0xad, 0xf0), G(0x9d, 0x5e, 0x81, 0x1c, 0xa0, 0xfe), - G(0x93, 0x57, 0xbe, 0x2d, 0x83, 0xd4), G(0x91, 0x54, 0xb7, 0x26, 0x8e, 0xda), - G(0x97, 0x51, 0xac, 0x3b, 0x99, 0xc8), G(0x95, 0x52, 0xa5, 0x30, 0x94, 0xc6), - G(0x8b, 0x43, 0xd2, 0x59, 0xdf, 0x9c), G(0x89, 0x40, 0xdb, 0x52, 0xd2, 0x92), - G(0x8f, 0x45, 0xc0, 0x4f, 0xc5, 0x80), G(0x8d, 0x46, 0xc9, 0x44, 0xc8, 0x8e), - G(0x83, 0x4f, 0xf6, 0x75, 0xeb, 0xa4), G(0x81, 0x4c, 0xff, 0x7e, 0xe6, 0xaa), - G(0x87, 0x49, 0xe4, 0x63, 0xf1, 0xb8), G(0x85, 0x4a, 0xed, 0x68, 0xfc, 0xb6), - G(0xbb, 0x6b, 0x0a, 0xb1, 0x67, 0x0c), G(0xb9, 0x68, 0x03, 0xba, 0x6a, 0x02), - G(0xbf, 0x6d, 0x18, 0xa7, 0x7d, 0x10), G(0xbd, 0x6e, 0x11, 0xac, 0x70, 0x1e), - G(0xb3, 0x67, 0x2e, 0x9d, 0x53, 0x34), G(0xb1, 0x64, 0x27, 0x96, 0x5e, 0x3a), - G(0xb7, 0x61, 0x3c, 0x8b, 0x49, 0x28), G(0xb5, 0x62, 0x35, 0x80, 0x44, 0x26), - G(0xab, 0x73, 0x42, 0xe9, 0x0f, 0x7c), G(0xa9, 0x70, 0x4b, 0xe2, 0x02, 0x72), - G(0xaf, 0x75, 0x50, 0xff, 0x15, 0x60), G(0xad, 0x76, 0x59, 0xf4, 0x18, 0x6e), - G(0xa3, 0x7f, 0x66, 0xc5, 0x3b, 0x44), G(0xa1, 0x7c, 0x6f, 0xce, 0x36, 0x4a), - G(0xa7, 0x79, 0x74, 0xd3, 0x21, 0x58), G(0xa5, 0x7a, 0x7d, 0xd8, 0x2c, 0x56), - G(0xdb, 0x3b, 0xa1, 0x7a, 0x0c, 0x37), G(0xd9, 0x38, 0xa8, 0x71, 0x01, 0x39), - G(0xdf, 0x3d, 0xb3, 0x6c, 0x16, 0x2b), G(0xdd, 0x3e, 0xba, 0x67, 0x1b, 0x25), - G(0xd3, 0x37, 0x85, 0x56, 0x38, 0x0f), G(0xd1, 0x34, 0x8c, 0x5d, 0x35, 0x01), - G(0xd7, 0x31, 0x97, 0x40, 0x22, 0x13), G(0xd5, 0x32, 0x9e, 0x4b, 0x2f, 0x1d), - G(0xcb, 0x23, 0xe9, 0x22, 0x64, 0x47), G(0xc9, 0x20, 0xe0, 0x29, 0x69, 0x49), - G(0xcf, 0x25, 0xfb, 0x34, 0x7e, 0x5b), G(0xcd, 0x26, 0xf2, 0x3f, 0x73, 0x55), - G(0xc3, 0x2f, 0xcd, 0x0e, 0x50, 0x7f), G(0xc1, 0x2c, 0xc4, 0x05, 0x5d, 0x71), - G(0xc7, 0x29, 0xdf, 0x18, 0x4a, 0x63), G(0xc5, 0x2a, 0xd6, 0x13, 0x47, 0x6d), - G(0xfb, 0x0b, 0x31, 0xca, 0xdc, 0xd7), G(0xf9, 0x08, 0x38, 0xc1, 0xd1, 0xd9), - G(0xff, 0x0d, 0x23, 0xdc, 0xc6, 0xcb), G(0xfd, 0x0e, 0x2a, 0xd7, 0xcb, 0xc5), - G(0xf3, 0x07, 0x15, 0xe6, 0xe8, 0xef), G(0xf1, 0x04, 0x1c, 0xed, 0xe5, 0xe1), - G(0xf7, 0x01, 0x07, 0xf0, 0xf2, 0xf3), G(0xf5, 0x02, 0x0e, 0xfb, 0xff, 0xfd), - G(0xeb, 0x13, 0x79, 0x92, 0xb4, 0xa7), G(0xe9, 0x10, 0x70, 0x99, 0xb9, 0xa9), - G(0xef, 0x15, 0x6b, 0x84, 0xae, 0xbb), G(0xed, 0x16, 0x62, 0x8f, 0xa3, 0xb5), - G(0xe3, 0x1f, 0x5d, 0xbe, 0x80, 0x9f), G(0xe1, 0x1c, 0x54, 0xb5, 0x8d, 0x91), - G(0xe7, 0x19, 0x4f, 0xa8, 0x9a, 0x83), G(0xe5, 0x1a, 0x46, 0xa3, 0x97, 0x8d), -}; -#undef G +/* + * The schedule is kept in its compressed form, 30 words of it, inside the + * aes_key the caller already has -- comfortably inside the 448 bytes that + * held the round keys before. br_aes_ct64_skey_expand blows it up to 120 + * words on the stack once per call, which is how BearSSL's own CTR and CBC + * use it. + * + * It travels through memcpy rather than a cast because aes_key is all + * uint8_t and so carries no alignment of its own, whatever the allocator + * happens to give it. + */ +#define COMP_SKEY_WORDS 30 -static void expand_key(uint8_t *expandedKey, uint8_t *key, int size, size_t expandedKeySize) +void crypton_aes_generic_schedule(aes_sched *sched, const aes_key *key) { - int csz; - int i; - uint8_t t[4] = { 0 }; - - for (i = 0; i < size; i++) - expandedKey[i] = key[i]; - csz = size; + uint64_t comp_skey[COMP_SKEY_WORDS]; - i = 1; - while (csz < expandedKeySize) { - t[0] = expandedKey[(csz - 4) + 0]; - t[1] = expandedKey[(csz - 4) + 1]; - t[2] = expandedKey[(csz - 4) + 2]; - t[3] = expandedKey[(csz - 4) + 3]; + memcpy(comp_skey, key->data, sizeof comp_skey); + sched->nbr = key->nbr; + br_aes_ct64_skey_expand(sched->sk_exp, sched->nbr, comp_skey); +} - if (csz % size == 0) { - uint8_t tmp; +/* + * Up to four blocks through the bitsliced core at once. Fewer than four is + * the same work as four -- the lanes are there whether anything is in them + * -- so a caller with four to offer gets them for what one used to cost. + */ +static void pass(uint8_t *output, const uint8_t *input, unsigned n, + const aes_sched *sched, int decrypt) +{ + uint32_t w[16]; + uint64_t q[8]; + unsigned i; - tmp = t[0]; - t[0] = sbox[t[1]] ^ Rcon[i++ % sizeof(Rcon)]; - t[1] = sbox[t[2]]; - t[2] = sbox[t[3]]; - t[3] = sbox[tmp]; - } + memset(w, 0, sizeof w); + for (i = 0; i < n; i++) { + w[4 * i] = br_dec32le(input + 16 * i); + w[4 * i + 1] = br_dec32le(input + 16 * i + 4); + w[4 * i + 2] = br_dec32le(input + 16 * i + 8); + w[4 * i + 3] = br_dec32le(input + 16 * i + 12); + } - if (size == 32 && ((csz % size) == 16)) { - t[0] = sbox[t[0]]; - t[1] = sbox[t[1]]; - t[2] = sbox[t[2]]; - t[3] = sbox[t[3]]; - } + for (i = 0; i < 4; i++) + br_aes_ct64_interleave_in(&q[i], &q[i + 4], w + 4 * i); + br_aes_ct64_ortho(q); + if (decrypt) + br_aes_ct64_bitslice_decrypt(sched->nbr, sched->sk_exp, q); + else + br_aes_ct64_bitslice_encrypt(sched->nbr, sched->sk_exp, q); + br_aes_ct64_ortho(q); + for (i = 0; i < 4; i++) + br_aes_ct64_interleave_out(w + 4 * i, q[i], q[i + 4]); - expandedKey[csz] = expandedKey[csz - size] ^ t[0]; csz++; - expandedKey[csz] = expandedKey[csz - size] ^ t[1]; csz++; - expandedKey[csz] = expandedKey[csz - size] ^ t[2]; csz++; - expandedKey[csz] = expandedKey[csz - size] ^ t[3]; csz++; + for (i = 0; i < n; i++) { + br_enc32le(output + 16 * i, w[4 * i]); + br_enc32le(output + 16 * i + 4, w[4 * i + 1]); + br_enc32le(output + 16 * i + 8, w[4 * i + 2]); + br_enc32le(output + 16 * i + 12, w[4 * i + 3]); } } -static void shift_rows(uint8_t *state) +void crypton_aes_generic_blocks(uint8_t *output, const uint8_t *input, + uint32_t nb_blocks, const aes_sched *sched, + int decrypt) { - uint32_t *s32; - int i; - - for (i = 0; i < 16; i++) - state[i] = sbox[state[i]]; - s32 = (uint32_t *) state; - s32[1] = rol32_be(s32[1], 8); - s32[2] = rol32_be(s32[2], 16); - s32[3] = rol32_be(s32[3], 24); + while (nb_blocks >= 4) { + pass(output, input, 4, sched, decrypt); + output += 64; + input += 64; + nb_blocks -= 4; + } + if (nb_blocks > 0) + pass(output, input, (unsigned) nb_blocks, sched, decrypt); } -static void add_round_key(uint8_t *state, uint8_t *rk) +static void one(aes_block *output, aes_key *key, aes_block *input, int decrypt) { - uint32_t *s32, *r32; + aes_sched sched; - s32 = (uint32_t *) state; - r32 = (uint32_t *) rk; - s32[0] ^= r32[0]; - s32[1] ^= r32[1]; - s32[2] ^= r32[2]; - s32[3] ^= r32[3]; + crypton_aes_generic_schedule(&sched, key); + crypton_aes_generic_blocks((uint8_t *) output, (const uint8_t *) input, + 1, &sched, decrypt); } -#define gm1(a) (a) -#define gm2(a) gmtab[a][0] -#define gm3(a) gmtab[a][1] -#define gm9(a) gmtab[a][2] -#define gm11(a) gmtab[a][3] -#define gm13(a) gmtab[a][4] -#define gm14(a) gmtab[a][5] - -static void mix_columns(uint8_t *state) +void crypton_aes_generic_encrypt_block(aes_block *output, aes_key *key, aes_block *input) { - int i; - uint8_t cpy[4]; - - for (i = 0; i < 4; i++) { - cpy[0] = state[0 * 4 + i]; - cpy[1] = state[1 * 4 + i]; - cpy[2] = state[2 * 4 + i]; - cpy[3] = state[3 * 4 + i]; - state[i] = gm2(cpy[0]) ^ gm1(cpy[3]) ^ gm1(cpy[2]) ^ gm3(cpy[1]); - state[4+i] = gm2(cpy[1]) ^ gm1(cpy[0]) ^ gm1(cpy[3]) ^ gm3(cpy[2]); - state[8+i] = gm2(cpy[2]) ^ gm1(cpy[1]) ^ gm1(cpy[0]) ^ gm3(cpy[3]); - state[12+i] = gm2(cpy[3]) ^ gm1(cpy[2]) ^ gm1(cpy[1]) ^ gm3(cpy[0]); - } + one(output, key, input, 0); } -static void create_round_key(uint8_t *expandedKey, uint8_t *rk) +void crypton_aes_generic_decrypt_block(aes_block *output, aes_key *key, aes_block *input) { - int i,j; - for (i = 0; i < 4; i++) - for (j = 0; j < 4; j++) - rk[i + j * 4] = expandedKey[i * 4 + j]; + one(output, key, input, 1); } -static void aes_main(aes_key *key, uint8_t *state) +void crypton_aes_generic_init(aes_key *key, uint8_t *origkey, uint8_t size) { - int i = 0; - uint32_t rk[4]; - uint8_t *rkptr = (uint8_t *) rk; - - create_round_key(key->data, rkptr); - add_round_key(state, rkptr); - - for (i = 1; i < key->nbr; i++) { - create_round_key(key->data + 16 * i, rkptr); - shift_rows(state); - mix_columns(state); - add_round_key(state, rkptr); - } - - create_round_key(key->data + 16 * key->nbr, rkptr); - shift_rows(state); - add_round_key(state, rkptr); -} + uint64_t comp_skey[COMP_SKEY_WORDS]; + unsigned nbr; -static void shift_rows_inv(uint8_t *state) -{ - uint32_t *s32; - int i; + /* 0 for a key length that is not 16, 24 or 32; the old code returned + * without touching the key in that case and so does this */ + nbr = br_aes_ct64_keysched(comp_skey, origkey, size); + if (nbr == 0) + return; - s32 = (uint32_t *) state; - s32[1] = ror32_be(s32[1], 8); - s32[2] = ror32_be(s32[2], 16); - s32[3] = ror32_be(s32[3], 24); - for (i = 0; i < 16; i++) - state[i] = rsbox[state[i]]; + key->nbr = (uint8_t) nbr; + memcpy(key->data, comp_skey, sizeof comp_skey); } -static void mix_columns_inv(uint8_t *state) +/* + * CTR, four counter blocks at a time. The counter itself is serial, but + * nothing about it depends on the keystream, so the four blocks it will + * reach next can be written down before any of them is encrypted. + */ +static void ctr(uint8_t *output, aes_key *key, aes_block *iv, + uint8_t *input, uint32_t len, int c32) { - int i; - uint8_t cpy[4]; + aes_sched sched; + aes_block counter; + uint8_t ks[64]; + uint32_t nb_blocks = len / 16; + uint32_t tail = len % 16; + uint32_t i; - for (i = 0; i < 4; i++) { - cpy[0] = state[0 * 4 + i]; - cpy[1] = state[1 * 4 + i]; - cpy[2] = state[2 * 4 + i]; - cpy[3] = state[3 * 4 + i]; - state[i] = gm14(cpy[0]) ^ gm9(cpy[3]) ^ gm13(cpy[2]) ^ gm11(cpy[1]); - state[4+i] = gm14(cpy[1]) ^ gm9(cpy[0]) ^ gm13(cpy[3]) ^ gm11(cpy[2]); - state[8+i] = gm14(cpy[2]) ^ gm9(cpy[1]) ^ gm13(cpy[0]) ^ gm11(cpy[3]); - state[12+i] = gm14(cpy[3]) ^ gm9(cpy[2]) ^ gm13(cpy[1]) ^ gm11(cpy[0]); - } -} + crypton_aes_generic_schedule(&sched, key); + block128_copy(&counter, iv); -static void aes_main_inv(aes_key *key, uint8_t *state) -{ - int i = 0; - uint32_t rk[4]; - uint8_t *rkptr = (uint8_t *) rk; + while (nb_blocks > 0) { + uint32_t n = nb_blocks < 4 ? nb_blocks : 4; - create_round_key(key->data + 16 * key->nbr, rkptr); - add_round_key(state, rkptr); + for (i = 0; i < n; i++) { + block128_copy((block128 *) (ks + 16 * i), &counter); + if (c32) + block128_inc32_le(&counter); + else + block128_inc_be(&counter); + } + crypton_aes_generic_blocks(ks, ks, n, &sched, 0); + for (i = 0; i < n * 16; i++) + output[i] = ks[i] ^ input[i]; - for (i = key->nbr - 1; i > 0; i--) { - create_round_key(key->data + 16 * i, rkptr); - shift_rows_inv(state); - add_round_key(state, rkptr); - mix_columns_inv(state); + output += n * 16; + input += n * 16; + nb_blocks -= n; } - create_round_key(key->data, rkptr); - shift_rows_inv(state); - add_round_key(state, rkptr); -} - -/* Set the block values, for the block: - * a0,0 a0,1 a0,2 a0,3 - * a1,0 a1,1 a1,2 a1,3 -> a0,0 a1,0 a2,0 a3,0 a0,1 a1,1 ... a2,3 a3,3 - * a2,0 a2,1 a2,2 a2,3 - * a3,0 a3,1 a3,2 a3,3 - */ -#define swap_block(t, f) \ - t[0] = f[0]; t[4] = f[1]; t[8] = f[2]; t[12] = f[3]; \ - t[1] = f[4]; t[5] = f[5]; t[9] = f[6]; t[13] = f[7]; \ - t[2] = f[8]; t[6] = f[9]; t[10] = f[10]; t[14] = f[11]; \ - t[3] = f[12]; t[7] = f[13]; t[11] = f[14]; t[15] = f[15] - -void crypton_aes_generic_encrypt_block(aes_block *output, aes_key *key, aes_block *input) -{ - uint32_t block[4]; - uint8_t *iptr, *optr, *bptr; - - iptr = (uint8_t *) input; - optr = (uint8_t *) output; - bptr = (uint8_t *) block; - swap_block(bptr, iptr); - aes_main(key, bptr); - swap_block(optr, bptr); + if (tail != 0) { + block128_copy((block128 *) ks, &counter); + crypton_aes_generic_blocks(ks, ks, 1, &sched, 0); + for (i = 0; i < tail; i++) + output[i] = ks[i] ^ input[i]; + } } -void crypton_aes_generic_decrypt_block(aes_block *output, aes_key *key, aes_block *input) +void crypton_aes_bitsliced_encrypt_ctr(uint8_t *output, aes_key *key, + aes_block *iv, uint8_t *input, + uint32_t len) { - uint32_t block[4]; - uint8_t *iptr, *optr, *bptr; - - iptr = (uint8_t *) input; - optr = (uint8_t *) output; - bptr = (uint8_t *) block; - swap_block(bptr, iptr); - aes_main_inv(key, bptr); - swap_block(optr, bptr); + ctr(output, key, iv, input, len, 0); } -void crypton_aes_generic_init(aes_key *key, uint8_t *origkey, uint8_t size) +void crypton_aes_bitsliced_encrypt_c32(uint8_t *output, aes_key *key, + aes_block *iv, uint8_t *input, + uint32_t len) { - int esz; - - switch (size) { - case 16: key->nbr = 10; esz = 176; break; - case 24: key->nbr = 12; esz = 208; break; - case 32: key->nbr = 14; esz = 240; break; - default: return; - } - expand_key(key->data, origkey, size, esz); - return; + ctr(output, key, iv, input, len, 1); } diff --git a/cbits/aes/generic.h b/cbits/aes/generic.h --- a/cbits/aes/generic.h +++ b/cbits/aes/generic.h @@ -32,3 +32,35 @@ void crypton_aes_generic_encrypt_block(aes_block *output, aes_key *key, aes_block *input); void crypton_aes_generic_decrypt_block(aes_block *output, aes_key *key, aes_block *input); void crypton_aes_generic_init(aes_key *key, uint8_t *origkey, uint8_t size); + +/* + * The bitsliced core takes four blocks at a time, and the schedule it reads + * is the expanded one rather than the compressed form the aes_key holds. + * Expanding costs about what a block costs, so a mode with more than one + * block to do expands once, here, and hands the result to every group. + */ +typedef struct { + uint64_t sk_exp[120]; + unsigned nbr; +} aes_sched; + +void crypton_aes_generic_schedule(aes_sched *sched, const aes_key *key); + +/* nb_blocks of them, four at a pass; decrypt selects the direction */ +void crypton_aes_generic_blocks(uint8_t *output, const uint8_t *input, + uint32_t nb_blocks, const aes_sched *sched, + int decrypt); + +/* + * CTR with the two counters crypton uses. These are not the generic + * entries: those go through the branch table for the block itself and so + * run on accelerated machines too, where the key holds a different + * schedule. crypton_aes.c installs these only when nothing was + * accelerated. + */ +void crypton_aes_bitsliced_encrypt_ctr(uint8_t *output, aes_key *key, + aes_block *iv, uint8_t *input, + uint32_t len); +void crypton_aes_bitsliced_encrypt_c32(uint8_t *output, aes_key *key, + aes_block *iv, uint8_t *input, + uint32_t len); diff --git a/cbits/aes/gf.c b/cbits/aes/gf.c --- a/cbits/aes/gf.c +++ b/cbits/aes/gf.c @@ -33,6 +33,7 @@ #include #include #include +#include "bearssl/inner.h" /* inplace GFMUL for xts mode */ void crypton_aes_generic_gf_mulx(block128 *a) @@ -45,118 +46,43 @@ /* - * GF multiplication with Shoup's method and 4-bit table. + * GHASH, without a table. * - * We precompute the products of H with all 4-bit polynomials and store them in - * a 'table_4bit' array. To avoid unnecessary byte swapping, the 16 blocks are - * written to the table with qwords already converted to CPU order. Table - * indices use the reflected bit ordering, i.e. polynomials X^0, X^1, X^2, X^3 - * map to bit positions 3, 2, 1, 0 respectively. + * This was Shoup's method: the products of H with all sixteen 4-bit + * polynomials, precomputed, and thirty-two lookups a block at indices taken + * from the accumulator -- which is to say, thirty-two addresses derived from + * a secret. It is now BearSSL's ghash_ctmul64, which builds the GF(2^128) + * multiply out of shifts, masks and integer multiplies and looks nothing up. + * See cbits/bearssl/README.md. * - * To multiply an arbitrary block with H, the input block is decomposed in 4-bit - * segments. We get the final result after 32 table lookups and additions, one - * for each segment, interleaving multiplication by P(X)=X^4. + * The table_4bit the interface names is sixteen blocks wide because the + * table needed it. This keeps H in the first and leaves the rest alone; the + * PMULL and PCLMUL implementations keep their own powers of H in that same + * space, so the width stays as it is. */ -/* convert block128 qwords between BE and CPU order */ -static inline void block128_cpu_swap_be(block128 *a, const block128 *b) -{ - a->q[1] = cpu_to_be64(b->q[1]); - a->q[0] = cpu_to_be64(b->q[0]); -} - -/* multiplication by P(X)=X, assuming qwords already in CPU order */ -static inline void cpu_gf_mulx(block128 *a, const block128 *b) -{ - uint64_t v0 = b->q[0]; - uint64_t v1 = b->q[1]; - a->q[1] = v1 >> 1 | v0 << 63; - a->q[0] = v0 >> 1 ^ ((0-(v1 & 1)) & 0xe100000000000000ULL); -} - -static const uint64_t r4_0[] = - { 0x0000000000000000ULL, 0x1c20000000000000ULL - , 0x3840000000000000ULL, 0x2460000000000000ULL - , 0x7080000000000000ULL, 0x6ca0000000000000ULL - , 0x48c0000000000000ULL, 0x54e0000000000000ULL - , 0xe100000000000000ULL, 0xfd20000000000000ULL - , 0xd940000000000000ULL, 0xc560000000000000ULL - , 0x9180000000000000ULL, 0x8da0000000000000ULL - , 0xa9c0000000000000ULL, 0xb5e0000000000000ULL - }; - -/* multiplication by P(X)=X^4, assuming qwords already in CPU order */ -static inline void cpu_gf_mulx4(block128 *a, const block128 *b) -{ - uint64_t v0 = b->q[0]; - uint64_t v1 = b->q[1]; - a->q[1] = v1 >> 4 | v0 << 60; - a->q[0] = v0 >> 4 ^ r4_0[v1 & 0xf]; -} - -/* initialize the 4-bit table given H */ +/* remember H */ void crypton_aes_generic_hinit(table_4bit htable, const block128 *h) { - block128 v, *p; - int i, j; - - /* multiplication by 0 is 0 */ - block128_zero(&htable[0]); - - /* at index 8=2^3 we have H.X^0 = H */ - i = 8; - block128_cpu_swap_be(&htable[i], h); /* in CPU order */ - p = &htable[i]; - - /* for other powers of 2, repeat multiplication by P(X)=X */ - for (i = 4; i > 0; i >>= 1) - { - cpu_gf_mulx(&htable[i], p); - p = &htable[i]; - } - - /* remaining elements are linear combinations */ - for (i = 2; i < 16; i <<= 1) { - p = &htable[i]; - v = *p; - for (j = 1; j < i; j++) { - p[j] = v; - block128_xor_aligned(&p[j], &htable[j]); - } - } + block128_copy(&htable[0], h); } -/* multiply a block with H */ +/* + * br_ghash_ctmul64 computes y = (y ^ x) * H for each block x it is given, so + * a block of zeros is the bare multiply this entry is asked for. + */ void crypton_aes_generic_gf_mul(block128 *a, const table_4bit htable) { - block128 b; - int i; - block128_zero(&b); - for (i = 15; i >= 0; i--) - { - uint8_t v = a->b[i]; - block128_xor_aligned(&b, &htable[v & 0xf]); /* high bits (reflected) */ - cpu_gf_mulx4(&b, &b); - block128_xor_aligned(&b, &htable[v >> 4]); /* low bits (reflected) */ - if (i > 0) - cpu_gf_mulx4(&b, &b); - else - block128_cpu_swap_be(a, &b); /* restore BE order when done */ - } + static const uint8_t zero[16] = { 0 }; + + br_ghash_ctmul64(a, &htable[0], zero, sizeof zero); } /* - * Four GHASH steps at once. The generic table-driven multiply has no cheaper - * way to do this than one block at a time; the point of the entry is that the - * PMULL and PCLMUL versions can fold the four products into one reduction, so - * the GCM loops hand over four blocks whenever they have them. + * Four GHASH steps at once, which here is one call rather than four: the + * loop inside br_ghash_ctmul64 takes the blocks as they come. */ void crypton_aes_generic_gf_mul4(block128 *a, const block128 *blocks, const table_4bit htable) { - int i; - - for (i = 0; i < 4; i++) { - block128_xor(a, &blocks[i]); - crypton_aes_generic_gf_mul(a, htable); - } + br_ghash_ctmul64(a, &htable[0], blocks, 4 * sizeof(block128)); } diff --git a/cbits/aes/ppc8.c b/cbits/aes/ppc8.c new file mode 100644 --- /dev/null +++ b/cbits/aes/ppc8.c @@ -0,0 +1,287 @@ +/* + * AES and GHASH using the PowerISA 2.07 vector instructions, first + * implemented by POWER8. + * + * The instructions themselves come from CRYPTOGAMS, assembled from + * cbits/asm/aesp8-ppc-*.S and cbits/asm/ghashp8-ppc-*.S; what is here is the + * glue that puts them behind crypton's branch table. Nothing in this file + * branches or indexes on a key or on data. + * + * == Where the key schedule lives + * + * The assembly takes OpenSSL's AES_KEY -- sixty round-key words and a round + * count, 244 bytes -- and its set_decrypt_key writes a second, complete one + * rather than sharing ends with the first the way cbits/aes/armv8.c does. + * Two of those are 488 bytes and aes_key.data is 448, so both do not fit. + * + * So the forward schedule is kept there, with the key the caller gave after + * it, and the inverse is built on the stack by the operations that need it. + * Those are all bulk -- ECB, CBC and XTS decryption -- so it is one key + * schedule per call rather than per block. Single-block decryption pays for + * one too, and is reached by nothing that runs in a loop: OCB and CCM drive + * the block function in the encrypting direction. + */ + +#include +#include +#include +#include +#include "aes/block128.h" +#include "aes/gf.h" +#include "aes/ppc8.h" + +/* OpenSSL's AES_KEY, which is what the assembly was written against */ +typedef struct { + unsigned int rd_key[60]; + int rounds; +} p8_key; + +int crypton_aes_p8_set_encrypt_key(const unsigned char *, int, p8_key *); +int crypton_aes_p8_set_decrypt_key(const unsigned char *, int, p8_key *); +void crypton_aes_p8_encrypt(const unsigned char *, unsigned char *, const p8_key *); +void crypton_aes_p8_decrypt(const unsigned char *, unsigned char *, const p8_key *); +void crypton_aes_p8_cbc_encrypt(const unsigned char *, unsigned char *, size_t, + const p8_key *, unsigned char *, int); +void crypton_aes_p8_ctr32_encrypt_blocks(const unsigned char *, unsigned char *, + size_t, const p8_key *, + const unsigned char *); +void crypton_aes_p8_xts_encrypt(const unsigned char *, unsigned char *, size_t, + const p8_key *, const p8_key *, + const unsigned char *); +void crypton_aes_p8_xts_decrypt(const unsigned char *, unsigned char *, size_t, + const p8_key *, const p8_key *, + const unsigned char *); +/* void * rather than uint64_t *: the assembly loads these with lvx_u and so + * does not want the alignment a uint64_t pointer would promise, and + * block128 is packed. */ +void crypton_gcm_init_p8(void *Htable, const void *H); +void crypton_gcm_gmult_p8(void *Xi, const void *Htable); +void crypton_gcm_ghash_p8(void *Xi, const void *Htable, + const unsigned char *inp, size_t len); + +#define FORWARD(k) ((p8_key *) (void *) (k)->data) +#define USERKEY(k) ((uint8_t *) (k)->data + sizeof(p8_key)) +#define USERLEN(k) (*((uint8_t *) (k)->data + sizeof(p8_key) + 32)) + +static void inverse(p8_key *dk, aes_key *key) +{ + crypton_aes_p8_set_decrypt_key(USERKEY(key), USERLEN(key) * 8, dk); +} + +void crypton_aes_ppc8_init(aes_key *key, uint8_t *origkey, uint8_t size) +{ + if (size != 16 && size != 24 && size != 32) + return; + crypton_aes_p8_set_encrypt_key(origkey, size * 8, FORWARD(key)); + memcpy(USERKEY(key), origkey, size); + USERLEN(key) = size; +} + +void crypton_aes_ppc8_encrypt_block(aes_block *output, aes_key *key, aes_block *input) +{ + crypton_aes_p8_encrypt((const unsigned char *) input, + (unsigned char *) output, FORWARD(key)); +} + +void crypton_aes_ppc8_decrypt_block(aes_block *output, aes_key *key, aes_block *input) +{ + p8_key dk; + + inverse(&dk, key); + crypton_aes_p8_decrypt((const unsigned char *) input, + (unsigned char *) output, &dk); + memset(&dk, 0, sizeof dk); +} + +/* The assembly has no ECB entry: it is the block function in a loop, which + * is what the generic implementation does too. */ +void crypton_aes_ppc8_encrypt_ecb(aes_block *output, aes_key *key, aes_block *input, + uint32_t nb_blocks) +{ + for (; nb_blocks-- > 0; input++, output++) + crypton_aes_p8_encrypt((const unsigned char *) input, + (unsigned char *) output, FORWARD(key)); +} + +void crypton_aes_ppc8_decrypt_ecb(aes_block *output, aes_key *key, aes_block *input, + uint32_t nb_blocks) +{ + p8_key dk; + + inverse(&dk, key); + for (; nb_blocks-- > 0; input++, output++) + crypton_aes_p8_decrypt((const unsigned char *) input, + (unsigned char *) output, &dk); + memset(&dk, 0, sizeof dk); +} + +void crypton_aes_ppc8_encrypt_cbc(aes_block *output, aes_key *key, aes_block *iv, + aes_block *input, uint32_t nb_blocks) +{ + uint8_t ivbuf[16]; + + /* a copy, because the assembly writes the last block back through this + * pointer and the callers of this entry do not expect their IV touched */ + memcpy(ivbuf, iv, 16); + crypton_aes_p8_cbc_encrypt((const unsigned char *) input, + (unsigned char *) output, + (size_t) nb_blocks * 16, FORWARD(key), ivbuf, 1); +} + +void crypton_aes_ppc8_decrypt_cbc(aes_block *output, aes_key *key, aes_block *iv, + aes_block *input, uint32_t nb_blocks) +{ + p8_key dk; + uint8_t ivbuf[16]; + + inverse(&dk, key); + memcpy(ivbuf, iv, 16); + crypton_aes_p8_cbc_encrypt((const unsigned char *) input, + (unsigned char *) output, + (size_t) nb_blocks * 16, &dk, ivbuf, 0); + memset(&dk, 0, sizeof dk); +} + +/* + * CTR, which is where the two counters have to be reconciled. + * + * crypton counts with block128_inc_be, over the whole 128 bits; the assembly + * counts over the low 32 only. They agree until that word wraps, so the + * work is handed over in runs that stop there, and the carry into the upper + * bits is done here. This is the same arrangement OpenSSL makes for the + * same reason. + */ +void crypton_aes_ppc8_encrypt_ctr(uint8_t *output, aes_key *key, aes_block *iv, + uint8_t *input, uint32_t len) +{ + block128 ctr; + uint32_t nb_blocks = len / 16; + uint32_t tail = len % 16; + uint32_t i; + + block128_copy(&ctr, iv); + + while (nb_blocks > 0) { + uint64_t low = (uint64_t) be32_to_cpu(ctr.d[3]); + uint64_t room = 0x100000000ULL - low; /* blocks before it wraps */ + uint32_t n; + + /* room is 2^32 when the word is at zero, which does not fit the + * type the comparison would narrow it to -- and a zero n here + * would not advance */ + if (room > (uint64_t) nb_blocks) + room = (uint64_t) nb_blocks; + n = (uint32_t) room; + + crypton_aes_p8_ctr32_encrypt_blocks((const unsigned char *) input, + (unsigned char *) output, + n, FORWARD(key), + (const unsigned char *) &ctr); + /* the assembly does not write the counter back */ + for (i = 0; i < n; i++) + block128_inc_be(&ctr); + + output += (size_t) n * 16; + input += (size_t) n * 16; + nb_blocks -= n; + } + + if (tail != 0) { + block128 ks; + + crypton_aes_p8_encrypt((const unsigned char *) &ctr, + (unsigned char *) &ks, FORWARD(key)); + for (i = 0; i < tail; i++) + output[i] = ks.b[i] ^ input[i]; + } +} + +/* + * XTS. The assembly encrypts the tweak with its second key, unless that key + * is NULL, in which case it takes the tweak already encrypted -- which is + * what this needs, because crypton's entry also carries a starting point and + * the tweak has to be advanced that many doublings before any block is + * enciphered. + */ +static void xts_tweak(block128 *tweak, aes_key *k2, aes_block *dataunit, + uint32_t spoint) +{ + block128_copy(tweak, dataunit); + crypton_aes_p8_encrypt((const unsigned char *) tweak, + (unsigned char *) tweak, FORWARD(k2)); + while (spoint-- > 0) + crypton_aes_generic_gf_mulx(tweak); +} + +void crypton_aes_ppc8_encrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, + aes_block *dataunit, uint32_t spoint, + aes_block *input, uint32_t nb_blocks) +{ + block128 tweak; + + xts_tweak(&tweak, k2, dataunit, spoint); + crypton_aes_p8_xts_encrypt((const unsigned char *) input, + (unsigned char *) output, + (size_t) nb_blocks * 16, FORWARD(k1), NULL, + (const unsigned char *) &tweak); +} + +void crypton_aes_ppc8_decrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, + aes_block *dataunit, uint32_t spoint, + aes_block *input, uint32_t nb_blocks) +{ + block128 tweak; + p8_key dk; + + /* the tweak is enciphered with k2 forwards either way */ + xts_tweak(&tweak, k2, dataunit, spoint); + inverse(&dk, k1); + crypton_aes_p8_xts_decrypt((const unsigned char *) input, + (unsigned char *) output, + (size_t) nb_blocks * 16, &dk, NULL, + (const unsigned char *) &tweak); + memset(&dk, 0, sizeof dk); +} + +/* + * GHASH. The table the assembly builds is 192 bytes, which fits the sixteen + * blocks the interface names. + * + * The two arguments do not take the same convention, which is worth saying + * because it does not look like an accident until it is checked. H arrives + * here as the sixteen bytes GCM defines, and gcm_init_p8 wants those as a + * pair of host-order words -- so they are swapped on a little-endian machine + * and left alone on a big-endian one, which be64_to_cpu does. The + * accumulator is the byte string throughout, in and out, and is passed as it + * stands. OpenSSL makes the same pair of choices. + * + * Checked rather than assumed: against crypton's own GHASH, over two + * different H and with a non-zero accumulator, so that neither a symmetric + * value nor a zero start could hide a wrong one. + */ +void crypton_aes_ppc8_hinit(table_4bit htable, const block128 *h) +{ + uint64_t H[2]; + + memcpy(H, h, sizeof H); + H[0] = be64_to_cpu(H[0]); + H[1] = be64_to_cpu(H[1]); + crypton_gcm_init_p8(htable, H); +} + +void crypton_aes_ppc8_gf_mul(block128 *a, const table_4bit htable) +{ + crypton_gcm_gmult_p8(a, htable); +} + +void crypton_aes_ppc8_gf_mul4(block128 *a, const block128 *blocks, + const table_4bit htable) +{ + crypton_gcm_ghash_p8(a, htable, (const unsigned char *) blocks, + 4 * sizeof(block128)); +} + +int crypton_aes_ppc8_available(void) +{ + return (crypton_ppc_features() & CRYPTON_PPC_VCRYPTO) != 0; +} diff --git a/cbits/aes/ppc8.h b/cbits/aes/ppc8.h new file mode 100644 --- /dev/null +++ b/cbits/aes/ppc8.h @@ -0,0 +1,29 @@ +/* + * AES and GHASH on PowerISA 2.07, as implemented by cbits/aes/ppc8.c over + * the CRYPTOGAMS assembly. crypton_aes.c installs these when + * crypton_aes_ppc8_available says the processor has the instructions. + */ +#ifndef CRYPTON_AES_PPC8_H +#define CRYPTON_AES_PPC8_H + +#include "crypton_aes.h" +#include "aes/gf.h" +#include "crypton_cpu.h" + +int crypton_aes_ppc8_available(void); + +void crypton_aes_ppc8_init(aes_key *key, uint8_t *origkey, uint8_t size); +void crypton_aes_ppc8_encrypt_block(aes_block *output, aes_key *key, aes_block *input); +void crypton_aes_ppc8_decrypt_block(aes_block *output, aes_key *key, aes_block *input); +void crypton_aes_ppc8_encrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks); +void crypton_aes_ppc8_decrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks); +void crypton_aes_ppc8_encrypt_cbc(aes_block *output, aes_key *key, aes_block *iv, aes_block *input, uint32_t nb_blocks); +void crypton_aes_ppc8_decrypt_cbc(aes_block *output, aes_key *key, aes_block *iv, aes_block *input, uint32_t nb_blocks); +void crypton_aes_ppc8_encrypt_ctr(uint8_t *output, aes_key *key, aes_block *iv, uint8_t *input, uint32_t len); +void crypton_aes_ppc8_encrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, aes_block *dataunit, uint32_t spoint, aes_block *input, uint32_t nb_blocks); +void crypton_aes_ppc8_decrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, aes_block *dataunit, uint32_t spoint, aes_block *input, uint32_t nb_blocks); +void crypton_aes_ppc8_hinit(table_4bit htable, const block128 *h); +void crypton_aes_ppc8_gf_mul(block128 *a, const table_4bit htable); +void crypton_aes_ppc8_gf_mul4(block128 *a, const block128 *blocks, const table_4bit htable); + +#endif diff --git a/cbits/aes/x86ni.c b/cbits/aes/x86ni.c --- a/cbits/aes/x86ni.c +++ b/cbits/aes/x86ni.c @@ -37,6 +37,8 @@ #include #include #include +#include +#include #include #include @@ -207,13 +209,23 @@ return v; } +/* memcpy rather than a cast, as everything else that moves bytes between a + * crypton structure and a word does since block128 was packed. The cast + * this replaces was written in 2014, when block128 was a plain union and + * taking a __m128i * to one promised nothing the type did not already + * offer. Packing it dropped its alignment to one, and the promise with it: + * gcc has reported the cast ever since, and it is right to -- the attribute + * on the local below is what makes the promise true, and nothing obliges + * the next edit to keep it. Sixteen bytes of memcpy between a __m128i and + * a sixteen-byte object is one movdqu, or nothing at all when both stay in + * registers. */ TARGET_AESNI static __m128i gfmul_generic(__m128i tag, const table_4bit htable) { - aes_block _t ALIGNMENT(16); - _mm_store_si128((__m128i *) &_t, tag); + aes_block _t; + memcpy(&_t, &tag, sizeof _t); crypton_aes_generic_gf_mul(&_t, htable); - tag = _mm_load_si128((__m128i *) &_t); + memcpy(&tag, &_t, sizeof tag); return tag; } @@ -357,11 +369,14 @@ htable[0].q[0] = bitfn_swap64(h->q[1]); htable[0].q[1] = bitfn_swap64(h->q[0]); - /* Indices 1..7 get H^2 .. H^8, which is what lets a group of blocks - * fold into one reduction: gf_mul4 uses the first four, the GCM loop - * all eight. The table has sixteen slots. */ + /* Indices 1..15 get H^2 .. H^16, which is what lets a group of blocks + * fold into one reduction: gf_mul4 uses the first four, the 128-bit + * GCM loop eight, and the 256-bit one all sixteen. The table has + * sixteen slots and now they are all used. Filling the upper half + * costs eight multiplies once per key, which is nothing beside a + * message. */ p = _mm_loadu_si128((const __m128i *) h); - for (i = 1; i < 8; i++) { + for (i = 1; i < 16; i++) { p = gfmul_pclmuldq(p, htable); _mm_storeu_si128((__m128i *) &htable[i], _mm_shuffle_epi8(p, bswap_mask)); diff --git a/cbits/aes/x86ni_impl.c b/cbits/aes/x86ni_impl.c --- a/cbits/aes/x86ni_impl.c +++ b/cbits/aes/x86ni_impl.c @@ -335,9 +335,43 @@ gcm->length_input += length; +#ifdef WITH_GCM_VAES512 + /* + * The widest form the processor has, first: four blocks to an + * instruction where the 256-bit one below takes two and the assembly + * after that takes one. Same contract throughout -- whole groups off + * the front, the counter and the tag left behind. + */ + if (nb_blocks >= GCM_VAES512_MIN_BLOCKS + && (crypton_x86_simd_features() & CRYPTON_X86_VAES512)) { + uint32_t done = crypton_gcm_vaes512_bulk_encrypt( + output, gcm, key, input, nb_blocks * 16); + + output += done; + input += done; + nb_blocks -= done / 16; + } +#endif +#ifdef WITH_GCM_VAES + /* + * The 256-bit instructions next: they take two blocks where the ones + * below take one, and the assembly that follows is 128-bit + * throughout. + */ + if (nb_blocks >= GCM_VAES_MIN_BLOCKS + && (crypton_x86_simd_features() & CRYPTON_X86_VAES)) { + uint32_t done = crypton_gcm_vaes_bulk_encrypt(output, gcm, key, + input, + nb_blocks * 16); + + output += done; + input += done; + nb_blocks -= done / 16; + } +#endif #if defined(WITH_X86_GCM_ASM) && defined(WITH_PCLMUL) /* - * The stitched assembly first, which takes whole groups of six + * The stitched assembly next, which takes whole groups of six * blocks off the front of the message and leaves the counter and the * running tag where the loop below expects to find them. It wants * eighteen blocks before it will start, and answers with what it did. @@ -483,6 +517,31 @@ gcm->length_input += length; +#ifdef WITH_GCM_VAES512 + /* as in encryption; the tag is taken over the input here */ + if (nb_blocks >= GCM_VAES512_MIN_BLOCKS + && (crypton_x86_simd_features() & CRYPTON_X86_VAES512)) { + uint32_t done = crypton_gcm_vaes512_bulk_decrypt( + output, gcm, key, input, nb_blocks * 16); + + output += done; + input += done; + nb_blocks -= done / 16; + } +#endif +#ifdef WITH_GCM_VAES + /* as in encryption; the tag is taken over the input here */ + if (nb_blocks >= GCM_VAES_MIN_BLOCKS + && (crypton_x86_simd_features() & CRYPTON_X86_VAES)) { + uint32_t done = crypton_gcm_vaes_bulk_decrypt(output, gcm, key, + input, + nb_blocks * 16); + + output += done; + input += done; + nb_blocks -= done / 16; + } +#endif #if defined(WITH_X86_GCM_ASM) && defined(WITH_PCLMUL) /* the same as encryption, except that decryption has nothing to * hold back and so will start on six blocks */ diff --git a/cbits/asm/README.md b/cbits/asm/README.md --- a/cbits/asm/README.md +++ b/cbits/asm/README.md @@ -2,7 +2,7 @@ ## What is here -Two modules from [CRYPTOGAMS](https://github.com/dot-asm/cryptogams), by Andy +Modules from [CRYPTOGAMS](https://github.com/dot-asm/cryptogams), by Andy Polyakov, checked in unmodified together with the translators they need: | generator | what it is | @@ -17,6 +17,8 @@ | `sha1-armv8.pl` | SHA-1 for AArch64 | | `sha512-armv8.pl` | SHA-256 for AArch64 (the generator emits SHA-512 or SHA-256 according to the name it is given, and only the latter is wanted) | | `keccak1600-armv8.pl` | Keccak for AArch64 | +| `aesp8-ppc.pl` | AES for PowerISA 2.07, which POWER8 was the first to implement | +| `ghashp8-ppc.pl` | GHASH for the same, over `vpmsumd` | `x86_64-xlate.pl`, `arm-xlate.pl` and `arm_arch.h` are the machinery those modules use. `generate.sh` runs the generators to produce the `.S` files, which are diff --git a/cbits/asm/aesp8-ppc-linux64le.S b/cbits/asm/aesp8-ppc-linux64le.S new file mode 100644 --- /dev/null +++ b/cbits/asm/aesp8-ppc-linux64le.S @@ -0,0 +1,3658 @@ +.machine "any" + +.abiversion 2 +.text + +.align 7 +rcon: +.byte 0x00,0x00,0x00,0x01,0x00,0x00,0x00,0x01,0x00,0x00,0x00,0x01,0x00,0x00,0x00,0x01 +.byte 0x00,0x00,0x00,0x1b,0x00,0x00,0x00,0x1b,0x00,0x00,0x00,0x1b,0x00,0x00,0x00,0x1b +.byte 0x0c,0x0f,0x0e,0x0d,0x0c,0x0f,0x0e,0x0d,0x0c,0x0f,0x0e,0x0d,0x0c,0x0f,0x0e,0x0d +.byte 0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00 +.Lconsts: + mflr 0 + bcl 20,31,$+4 + mflr 6 + addi 6,6,-0x48 + mtlr 0 + blr +.long 0 +.byte 0,12,0x14,0,0,0,0,0 +.byte 65,69,83,32,102,111,114,32,80,111,119,101,114,73,83,65,32,50,46,48,55,44,67,82,89,80,84,79,71,65,77,83,32,98,121,32,60,97,112,112,114,111,64,111,112,101,110,115,115,108,46,111,114,103,62,0 +.align 2 + +.globl crypton_aes_p8_set_encrypt_key +.type crypton_aes_p8_set_encrypt_key,@function +.align 5 +crypton_aes_p8_set_encrypt_key: +.localentry crypton_aes_p8_set_encrypt_key,0 + +.Lset_encrypt_key: + mflr 11 + std 11,16(1) + + li 6,-1 + cmpldi 3,0 + beq- .Lenc_key_abort + cmpldi 5,0 + beq- .Lenc_key_abort + li 6,-2 + cmpwi 4,128 + blt- .Lenc_key_abort + cmpwi 4,256 + bgt- .Lenc_key_abort + andi. 0,4,0x3f + bne- .Lenc_key_abort + + lis 0,0xfff0 + li 12,-1 + or 0,0,0 + + bl .Lconsts + mtlr 11 + + neg 9,3 + lvx 1,0,3 + addi 3,3,15 + lvsr 3,0,9 + li 8,0x20 + cmpwi 4,192 + lvx 2,0,3 + vspltisb 5,0x0f + lvx 4,0,6 + vxor 3,3,5 + lvx 5,8,6 + addi 6,6,0x10 + vperm 1,1,2,3 + li 7,8 + vxor 0,0,0 + mtctr 7 + + lvsl 8,0,5 + vspltisb 9,-1 + lvx 10,0,5 + vperm 9,9,0,8 + + blt .Loop128 + addi 3,3,8 + beq .L192 + addi 3,3,8 + b .L256 + +.align 4 +.Loop128: + vperm 3,1,1,5 + vsldoi 6,0,1,12 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + .long 0x10632509 + stvx 7,0,5 + addi 5,5,16 + + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vadduwm 4,4,4 + vxor 1,1,3 + bdnz .Loop128 + + lvx 4,0,6 + + vperm 3,1,1,5 + vsldoi 6,0,1,12 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + .long 0x10632509 + stvx 7,0,5 + addi 5,5,16 + + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vadduwm 4,4,4 + vxor 1,1,3 + + vperm 3,1,1,5 + vsldoi 6,0,1,12 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + .long 0x10632509 + stvx 7,0,5 + addi 5,5,16 + + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vxor 1,1,3 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + stvx 7,0,5 + + addi 3,5,15 + addi 5,5,0x50 + + li 8,10 + b .Ldone + +.align 4 +.L192: + lvx 6,0,3 + li 7,4 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + stvx 7,0,5 + addi 5,5,16 + vperm 2,2,6,3 + vspltisb 3,8 + mtctr 7 + vsububm 5,5,3 + +.Loop192: + vperm 3,2,2,5 + vsldoi 6,0,1,12 + .long 0x10632509 + + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + + vsldoi 7,0,2,8 + vspltw 6,1,3 + vxor 6,6,2 + vsldoi 2,0,2,12 + vadduwm 4,4,4 + vxor 2,2,6 + vxor 1,1,3 + vxor 2,2,3 + vsldoi 7,7,1,8 + + vperm 3,2,2,5 + vsldoi 6,0,1,12 + vperm 11,7,7,8 + vsel 7,10,11,9 + vor 10,11,11 + .long 0x10632509 + stvx 7,0,5 + addi 5,5,16 + + vsldoi 7,1,2,8 + vxor 1,1,6 + vsldoi 6,0,6,12 + vperm 11,7,7,8 + vsel 7,10,11,9 + vor 10,11,11 + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + stvx 7,0,5 + addi 5,5,16 + + vspltw 6,1,3 + vxor 6,6,2 + vsldoi 2,0,2,12 + vadduwm 4,4,4 + vxor 2,2,6 + vxor 1,1,3 + vxor 2,2,3 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + stvx 7,0,5 + addi 3,5,15 + addi 5,5,16 + bdnz .Loop192 + + li 8,12 + addi 5,5,0x20 + b .Ldone + +.align 4 +.L256: + lvx 6,0,3 + li 7,7 + li 8,14 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + stvx 7,0,5 + addi 5,5,16 + vperm 2,2,6,3 + mtctr 7 + +.Loop256: + vperm 3,2,2,5 + vsldoi 6,0,1,12 + vperm 11,2,2,8 + vsel 7,10,11,9 + vor 10,11,11 + .long 0x10632509 + stvx 7,0,5 + addi 5,5,16 + + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vsldoi 6,0,6,12 + vxor 1,1,6 + vadduwm 4,4,4 + vxor 1,1,3 + vperm 11,1,1,8 + vsel 7,10,11,9 + vor 10,11,11 + stvx 7,0,5 + addi 3,5,15 + addi 5,5,16 + bdz .Ldone + + vspltw 3,1,3 + vsldoi 6,0,2,12 + .long 0x106305C8 + + vxor 2,2,6 + vsldoi 6,0,6,12 + vxor 2,2,6 + vsldoi 6,0,6,12 + vxor 2,2,6 + + vxor 2,2,3 + b .Loop256 + +.align 4 +.Ldone: + lvx 2,0,3 + vsel 2,10,2,9 + stvx 2,0,3 + li 6,0 + or 12,12,12 + stw 8,0(5) + +.Lenc_key_abort: + mr 3,6 + blr +.long 0 +.byte 0,12,0x14,1,0,0,3,0 +.long 0 +.size crypton_aes_p8_set_encrypt_key,.-crypton_aes_p8_set_encrypt_key + +.globl crypton_aes_p8_set_decrypt_key +.type crypton_aes_p8_set_decrypt_key,@function +.align 5 +crypton_aes_p8_set_decrypt_key: +.localentry crypton_aes_p8_set_decrypt_key,0 + + stdu 1,-64(1) + mflr 10 + std 10,64+16(1) + bl .Lset_encrypt_key + mtlr 10 + + cmpwi 3,0 + bne- .Ldec_key_abort + + slwi 7,8,4 + subi 3,5,240 + srwi 8,8,1 + add 5,3,7 + mtctr 8 + +.Ldeckey: + lwz 0, 0(3) + lwz 6, 4(3) + lwz 7, 8(3) + lwz 8, 12(3) + addi 3,3,16 + lwz 9, 0(5) + lwz 10,4(5) + lwz 11,8(5) + lwz 12,12(5) + stw 0, 0(5) + stw 6, 4(5) + stw 7, 8(5) + stw 8, 12(5) + subi 5,5,16 + stw 9, -16(3) + stw 10,-12(3) + stw 11,-8(3) + stw 12,-4(3) + bdnz .Ldeckey + + xor 3,3,3 +.Ldec_key_abort: + addi 1,1,64 + blr +.long 0 +.byte 0,12,4,1,0x80,0,3,0 +.long 0 +.size crypton_aes_p8_set_decrypt_key,.-crypton_aes_p8_set_decrypt_key +.globl crypton_aes_p8_encrypt +.type crypton_aes_p8_encrypt,@function +.align 5 +crypton_aes_p8_encrypt: +.localentry crypton_aes_p8_encrypt,0 + + lwz 6,240(5) + lis 0,0xfc00 + li 12,-1 + li 7,15 + or 0,0,0 + + lvx 0,0,3 + neg 11,4 + lvx 1,7,3 + lvsl 2,0,3 + vspltisb 4,0x0f + lvsr 3,0,11 + vxor 2,2,4 + li 7,16 + vperm 0,0,1,2 + lvx 1,0,5 + lvsr 5,0,5 + srwi 6,6,1 + lvx 2,7,5 + addi 7,7,16 + subi 6,6,1 + vperm 1,2,1,5 + + vxor 0,0,1 + lvx 1,7,5 + addi 7,7,16 + mtctr 6 + +.Loop_enc: + vperm 2,1,2,5 + .long 0x10001508 + lvx 2,7,5 + addi 7,7,16 + vperm 1,2,1,5 + .long 0x10000D08 + lvx 1,7,5 + addi 7,7,16 + bdnz .Loop_enc + + vperm 2,1,2,5 + .long 0x10001508 + lvx 2,7,5 + vperm 1,2,1,5 + .long 0x10000D09 + + vspltisb 2,-1 + vxor 1,1,1 + li 7,15 + vperm 2,2,1,3 + vxor 3,3,4 + lvx 1,0,4 + vperm 0,0,0,3 + vsel 1,1,0,2 + lvx 4,7,4 + stvx 1,0,4 + vsel 0,0,4,2 + stvx 0,7,4 + + or 12,12,12 + blr +.long 0 +.byte 0,12,0x14,0,0,0,3,0 +.long 0 +.size crypton_aes_p8_encrypt,.-crypton_aes_p8_encrypt +.globl crypton_aes_p8_decrypt +.type crypton_aes_p8_decrypt,@function +.align 5 +crypton_aes_p8_decrypt: +.localentry crypton_aes_p8_decrypt,0 + + lwz 6,240(5) + lis 0,0xfc00 + li 12,-1 + li 7,15 + or 0,0,0 + + lvx 0,0,3 + neg 11,4 + lvx 1,7,3 + lvsl 2,0,3 + vspltisb 4,0x0f + lvsr 3,0,11 + vxor 2,2,4 + li 7,16 + vperm 0,0,1,2 + lvx 1,0,5 + lvsr 5,0,5 + srwi 6,6,1 + lvx 2,7,5 + addi 7,7,16 + subi 6,6,1 + vperm 1,2,1,5 + + vxor 0,0,1 + lvx 1,7,5 + addi 7,7,16 + mtctr 6 + +.Loop_dec: + vperm 2,1,2,5 + .long 0x10001548 + lvx 2,7,5 + addi 7,7,16 + vperm 1,2,1,5 + .long 0x10000D48 + lvx 1,7,5 + addi 7,7,16 + bdnz .Loop_dec + + vperm 2,1,2,5 + .long 0x10001548 + lvx 2,7,5 + vperm 1,2,1,5 + .long 0x10000D49 + + vspltisb 2,-1 + vxor 1,1,1 + li 7,15 + vperm 2,2,1,3 + vxor 3,3,4 + lvx 1,0,4 + vperm 0,0,0,3 + vsel 1,1,0,2 + lvx 4,7,4 + stvx 1,0,4 + vsel 0,0,4,2 + stvx 0,7,4 + + or 12,12,12 + blr +.long 0 +.byte 0,12,0x14,0,0,0,3,0 +.long 0 +.size crypton_aes_p8_decrypt,.-crypton_aes_p8_decrypt +.globl crypton_aes_p8_cbc_encrypt +.type crypton_aes_p8_cbc_encrypt,@function +.align 5 +crypton_aes_p8_cbc_encrypt: +.localentry crypton_aes_p8_cbc_encrypt,0 + + cmpldi 5,16 + .long 0x4dc00020 + + cmpwi 8,0 + lis 0,0xffe0 + li 12,-1 + or 0,0,0 + + li 10,15 + vxor 0,0,0 + vspltisb 3,0x0f + + lvx 4,0,7 + lvsl 6,0,7 + lvx 5,10,7 + vxor 6,6,3 + vperm 4,4,5,6 + + neg 11,3 + lvsr 10,0,6 + lwz 9,240(6) + + lvsr 6,0,11 + lvx 5,0,3 + addi 3,3,15 + vxor 6,6,3 + + lvsl 8,0,4 + vspltisb 9,-1 + lvx 7,0,4 + vperm 9,9,0,8 + vxor 8,8,3 + + srwi 9,9,1 + li 10,16 + subi 9,9,1 + beq .Lcbc_dec + +.Lcbc_enc: + vor 2,5,5 + lvx 5,0,3 + addi 3,3,16 + mtctr 9 + subi 5,5,16 + + lvx 0,0,6 + vperm 2,2,5,6 + lvx 1,10,6 + addi 10,10,16 + vperm 0,1,0,10 + vxor 2,2,0 + lvx 0,10,6 + addi 10,10,16 + vxor 2,2,4 + +.Loop_cbc_enc: + vperm 1,0,1,10 + .long 0x10420D08 + lvx 1,10,6 + addi 10,10,16 + vperm 0,1,0,10 + .long 0x10420508 + lvx 0,10,6 + addi 10,10,16 + bdnz .Loop_cbc_enc + + vperm 1,0,1,10 + .long 0x10420D08 + lvx 1,10,6 + li 10,16 + vperm 0,1,0,10 + .long 0x10820509 + cmpldi 5,16 + + vperm 3,4,4,8 + vsel 2,7,3,9 + vor 7,3,3 + stvx 2,0,4 + addi 4,4,16 + bge .Lcbc_enc + + b .Lcbc_done + +.align 4 +.Lcbc_dec: + cmpldi 5,128 + bge _aesp8_cbc_decrypt8x + vor 3,5,5 + lvx 5,0,3 + addi 3,3,16 + mtctr 9 + subi 5,5,16 + + lvx 0,0,6 + vperm 3,3,5,6 + lvx 1,10,6 + addi 10,10,16 + vperm 0,1,0,10 + vxor 2,3,0 + lvx 0,10,6 + addi 10,10,16 + +.Loop_cbc_dec: + vperm 1,0,1,10 + .long 0x10420D48 + lvx 1,10,6 + addi 10,10,16 + vperm 0,1,0,10 + .long 0x10420548 + lvx 0,10,6 + addi 10,10,16 + bdnz .Loop_cbc_dec + + vperm 1,0,1,10 + .long 0x10420D48 + lvx 1,10,6 + li 10,16 + vperm 0,1,0,10 + .long 0x10420549 + cmpldi 5,16 + + vxor 2,2,4 + vor 4,3,3 + vperm 3,2,2,8 + vsel 2,7,3,9 + vor 7,3,3 + stvx 2,0,4 + addi 4,4,16 + bge .Lcbc_dec + +.Lcbc_done: + addi 4,4,-1 + lvx 2,0,4 + vsel 2,7,2,9 + stvx 2,0,4 + + neg 8,7 + li 10,15 + vxor 0,0,0 + vspltisb 9,-1 + vspltisb 3,0x0f + lvsr 8,0,8 + vperm 9,9,0,8 + vxor 8,8,3 + lvx 7,0,7 + vperm 4,4,4,8 + vsel 2,7,4,9 + lvx 5,10,7 + stvx 2,0,7 + vsel 2,4,5,9 + stvx 2,10,7 + + or 12,12,12 + blr +.long 0 +.byte 0,12,0x14,0,0,0,6,0 +.long 0 +.align 5 +_aesp8_cbc_decrypt8x: + stdu 1,-448(1) + li 10,207 + li 11,223 + stvx 20,10,1 + addi 10,10,32 + stvx 21,11,1 + addi 11,11,32 + stvx 22,10,1 + addi 10,10,32 + stvx 23,11,1 + addi 11,11,32 + stvx 24,10,1 + addi 10,10,32 + stvx 25,11,1 + addi 11,11,32 + stvx 26,10,1 + addi 10,10,32 + stvx 27,11,1 + addi 11,11,32 + stvx 28,10,1 + addi 10,10,32 + stvx 29,11,1 + addi 11,11,32 + stvx 30,10,1 + stvx 31,11,1 + li 0,-1 + stw 12,396(1) + li 8,0x10 + std 26,400(1) + li 26,0x20 + std 27,408(1) + li 27,0x30 + std 28,416(1) + li 28,0x40 + std 29,424(1) + li 29,0x50 + std 30,432(1) + li 30,0x60 + std 31,440(1) + li 31,0x70 + or 0,0,0 + + subi 9,9,3 + subi 5,5,128 + + lvx 23,0,6 + lvx 30,8,6 + addi 6,6,0x20 + lvx 31,0,6 + vperm 23,30,23,10 + addi 11,1,64+15 + mtctr 9 + +.Load_cbc_dec_key: + vperm 24,31,30,10 + lvx 30,8,6 + addi 6,6,0x20 + stvx 24,0,11 + vperm 25,30,31,10 + lvx 31,0,6 + stvx 25,8,11 + addi 11,11,0x20 + bdnz .Load_cbc_dec_key + + lvx 26,8,6 + vperm 24,31,30,10 + lvx 27,26,6 + stvx 24,0,11 + vperm 25,26,31,10 + lvx 28,27,6 + stvx 25,8,11 + addi 11,1,64+15 + vperm 26,27,26,10 + lvx 29,28,6 + vperm 27,28,27,10 + lvx 30,29,6 + vperm 28,29,28,10 + lvx 31,30,6 + vperm 29,30,29,10 + lvx 14,31,6 + vperm 30,31,30,10 + lvx 24,0,11 + vperm 31,14,31,10 + lvx 25,8,11 + + + + subi 3,3,15 + + li 10,8 + .long 0x7C001E99 + lvsl 6,0,10 + vspltisb 3,0x0f + .long 0x7C281E99 + vxor 6,6,3 + .long 0x7C5A1E99 + vperm 0,0,0,6 + .long 0x7C7B1E99 + vperm 1,1,1,6 + .long 0x7D5C1E99 + vperm 2,2,2,6 + vxor 14,0,23 + .long 0x7D7D1E99 + vperm 3,3,3,6 + vxor 15,1,23 + .long 0x7D9E1E99 + vperm 10,10,10,6 + vxor 16,2,23 + .long 0x7DBF1E99 + addi 3,3,0x80 + vperm 11,11,11,6 + vxor 17,3,23 + vperm 12,12,12,6 + vxor 18,10,23 + vperm 13,13,13,6 + vxor 19,11,23 + vxor 20,12,23 + vxor 21,13,23 + + mtctr 9 + b .Loop_cbc_dec8x +.align 5 +.Loop_cbc_dec8x: + .long 0x11CEC548 + .long 0x11EFC548 + .long 0x1210C548 + .long 0x1231C548 + .long 0x1252C548 + .long 0x1273C548 + .long 0x1294C548 + .long 0x12B5C548 + lvx 24,26,11 + addi 11,11,0x20 + + .long 0x11CECD48 + .long 0x11EFCD48 + .long 0x1210CD48 + .long 0x1231CD48 + .long 0x1252CD48 + .long 0x1273CD48 + .long 0x1294CD48 + .long 0x12B5CD48 + lvx 25,8,11 + bdnz .Loop_cbc_dec8x + + subic 5,5,128 + .long 0x11CEC548 + .long 0x11EFC548 + .long 0x1210C548 + .long 0x1231C548 + .long 0x1252C548 + .long 0x1273C548 + .long 0x1294C548 + .long 0x12B5C548 + + subfe. 0,0,0 + .long 0x11CECD48 + .long 0x11EFCD48 + .long 0x1210CD48 + .long 0x1231CD48 + .long 0x1252CD48 + .long 0x1273CD48 + .long 0x1294CD48 + .long 0x12B5CD48 + + and 0,0,5 + .long 0x11CED548 + .long 0x11EFD548 + .long 0x1210D548 + .long 0x1231D548 + .long 0x1252D548 + .long 0x1273D548 + .long 0x1294D548 + .long 0x12B5D548 + + add 3,3,0 + + + + .long 0x11CEDD48 + .long 0x11EFDD48 + .long 0x1210DD48 + .long 0x1231DD48 + .long 0x1252DD48 + .long 0x1273DD48 + .long 0x1294DD48 + .long 0x12B5DD48 + + addi 11,1,64+15 + .long 0x11CEE548 + .long 0x11EFE548 + .long 0x1210E548 + .long 0x1231E548 + .long 0x1252E548 + .long 0x1273E548 + .long 0x1294E548 + .long 0x12B5E548 + lvx 24,0,11 + + .long 0x11CEED48 + .long 0x11EFED48 + .long 0x1210ED48 + .long 0x1231ED48 + .long 0x1252ED48 + .long 0x1273ED48 + .long 0x1294ED48 + .long 0x12B5ED48 + lvx 25,8,11 + + .long 0x11CEF548 + vxor 4,4,31 + .long 0x11EFF548 + vxor 0,0,31 + .long 0x1210F548 + vxor 1,1,31 + .long 0x1231F548 + vxor 2,2,31 + .long 0x1252F548 + vxor 3,3,31 + .long 0x1273F548 + vxor 10,10,31 + .long 0x1294F548 + vxor 11,11,31 + .long 0x12B5F548 + vxor 12,12,31 + + .long 0x11CE2549 + .long 0x11EF0549 + .long 0x7C001E99 + .long 0x12100D49 + .long 0x7C281E99 + .long 0x12311549 + vperm 0,0,0,6 + .long 0x7C5A1E99 + .long 0x12521D49 + vperm 1,1,1,6 + .long 0x7C7B1E99 + .long 0x12735549 + vperm 2,2,2,6 + .long 0x7D5C1E99 + .long 0x12945D49 + vperm 3,3,3,6 + .long 0x7D7D1E99 + .long 0x12B56549 + vperm 10,10,10,6 + .long 0x7D9E1E99 + vor 4,13,13 + vperm 11,11,11,6 + .long 0x7DBF1E99 + addi 3,3,0x80 + + vperm 14,14,14,6 + vperm 15,15,15,6 + .long 0x7DC02799 + vperm 12,12,12,6 + vxor 14,0,23 + vperm 16,16,16,6 + .long 0x7DE82799 + vperm 13,13,13,6 + vxor 15,1,23 + vperm 17,17,17,6 + .long 0x7E1A2799 + vxor 16,2,23 + vperm 18,18,18,6 + .long 0x7E3B2799 + vxor 17,3,23 + vperm 19,19,19,6 + .long 0x7E5C2799 + vxor 18,10,23 + vperm 20,20,20,6 + .long 0x7E7D2799 + vxor 19,11,23 + vperm 21,21,21,6 + .long 0x7E9E2799 + vxor 20,12,23 + .long 0x7EBF2799 + addi 4,4,0x80 + vxor 21,13,23 + + mtctr 9 + beq .Loop_cbc_dec8x + + addic. 5,5,128 + beq .Lcbc_dec8x_done + nop + nop + +.Loop_cbc_dec8x_tail: + .long 0x11EFC548 + .long 0x1210C548 + .long 0x1231C548 + .long 0x1252C548 + .long 0x1273C548 + .long 0x1294C548 + .long 0x12B5C548 + lvx 24,26,11 + addi 11,11,0x20 + + .long 0x11EFCD48 + .long 0x1210CD48 + .long 0x1231CD48 + .long 0x1252CD48 + .long 0x1273CD48 + .long 0x1294CD48 + .long 0x12B5CD48 + lvx 25,8,11 + bdnz .Loop_cbc_dec8x_tail + + .long 0x11EFC548 + .long 0x1210C548 + .long 0x1231C548 + .long 0x1252C548 + .long 0x1273C548 + .long 0x1294C548 + .long 0x12B5C548 + + .long 0x11EFCD48 + .long 0x1210CD48 + .long 0x1231CD48 + .long 0x1252CD48 + .long 0x1273CD48 + .long 0x1294CD48 + .long 0x12B5CD48 + + .long 0x11EFD548 + .long 0x1210D548 + .long 0x1231D548 + .long 0x1252D548 + .long 0x1273D548 + .long 0x1294D548 + .long 0x12B5D548 + + .long 0x11EFDD48 + .long 0x1210DD48 + .long 0x1231DD48 + .long 0x1252DD48 + .long 0x1273DD48 + .long 0x1294DD48 + .long 0x12B5DD48 + + .long 0x11EFE548 + .long 0x1210E548 + .long 0x1231E548 + .long 0x1252E548 + .long 0x1273E548 + .long 0x1294E548 + .long 0x12B5E548 + + .long 0x11EFED48 + .long 0x1210ED48 + .long 0x1231ED48 + .long 0x1252ED48 + .long 0x1273ED48 + .long 0x1294ED48 + .long 0x12B5ED48 + + .long 0x11EFF548 + vxor 4,4,31 + .long 0x1210F548 + vxor 1,1,31 + .long 0x1231F548 + vxor 2,2,31 + .long 0x1252F548 + vxor 3,3,31 + .long 0x1273F548 + vxor 10,10,31 + .long 0x1294F548 + vxor 11,11,31 + .long 0x12B5F548 + vxor 12,12,31 + + cmplwi 5,32 + blt .Lcbc_dec8x_one + nop + beq .Lcbc_dec8x_two + cmplwi 5,64 + blt .Lcbc_dec8x_three + nop + beq .Lcbc_dec8x_four + cmplwi 5,96 + blt .Lcbc_dec8x_five + nop + beq .Lcbc_dec8x_six + +.Lcbc_dec8x_seven: + .long 0x11EF2549 + .long 0x12100D49 + .long 0x12311549 + .long 0x12521D49 + .long 0x12735549 + .long 0x12945D49 + .long 0x12B56549 + vor 4,13,13 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + vperm 17,17,17,6 + .long 0x7E082799 + vperm 18,18,18,6 + .long 0x7E3A2799 + vperm 19,19,19,6 + .long 0x7E5B2799 + vperm 20,20,20,6 + .long 0x7E7C2799 + vperm 21,21,21,6 + .long 0x7E9D2799 + .long 0x7EBE2799 + addi 4,4,0x70 + b .Lcbc_dec8x_done + +.align 5 +.Lcbc_dec8x_six: + .long 0x12102549 + .long 0x12311549 + .long 0x12521D49 + .long 0x12735549 + .long 0x12945D49 + .long 0x12B56549 + vor 4,13,13 + + vperm 16,16,16,6 + vperm 17,17,17,6 + .long 0x7E002799 + vperm 18,18,18,6 + .long 0x7E282799 + vperm 19,19,19,6 + .long 0x7E5A2799 + vperm 20,20,20,6 + .long 0x7E7B2799 + vperm 21,21,21,6 + .long 0x7E9C2799 + .long 0x7EBD2799 + addi 4,4,0x60 + b .Lcbc_dec8x_done + +.align 5 +.Lcbc_dec8x_five: + .long 0x12312549 + .long 0x12521D49 + .long 0x12735549 + .long 0x12945D49 + .long 0x12B56549 + vor 4,13,13 + + vperm 17,17,17,6 + vperm 18,18,18,6 + .long 0x7E202799 + vperm 19,19,19,6 + .long 0x7E482799 + vperm 20,20,20,6 + .long 0x7E7A2799 + vperm 21,21,21,6 + .long 0x7E9B2799 + .long 0x7EBC2799 + addi 4,4,0x50 + b .Lcbc_dec8x_done + +.align 5 +.Lcbc_dec8x_four: + .long 0x12522549 + .long 0x12735549 + .long 0x12945D49 + .long 0x12B56549 + vor 4,13,13 + + vperm 18,18,18,6 + vperm 19,19,19,6 + .long 0x7E402799 + vperm 20,20,20,6 + .long 0x7E682799 + vperm 21,21,21,6 + .long 0x7E9A2799 + .long 0x7EBB2799 + addi 4,4,0x40 + b .Lcbc_dec8x_done + +.align 5 +.Lcbc_dec8x_three: + .long 0x12732549 + .long 0x12945D49 + .long 0x12B56549 + vor 4,13,13 + + vperm 19,19,19,6 + vperm 20,20,20,6 + .long 0x7E602799 + vperm 21,21,21,6 + .long 0x7E882799 + .long 0x7EBA2799 + addi 4,4,0x30 + b .Lcbc_dec8x_done + +.align 5 +.Lcbc_dec8x_two: + .long 0x12942549 + .long 0x12B56549 + vor 4,13,13 + + vperm 20,20,20,6 + vperm 21,21,21,6 + .long 0x7E802799 + .long 0x7EA82799 + addi 4,4,0x20 + b .Lcbc_dec8x_done + +.align 5 +.Lcbc_dec8x_one: + .long 0x12B52549 + vor 4,13,13 + + vperm 21,21,21,6 + .long 0x7EA02799 + addi 4,4,0x10 + +.Lcbc_dec8x_done: + vperm 4,4,4,6 + .long 0x7C803F99 + + li 10,79 + li 11,95 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + + or 12,12,12 + lvx 20,10,1 + addi 10,10,32 + lvx 21,11,1 + addi 11,11,32 + lvx 22,10,1 + addi 10,10,32 + lvx 23,11,1 + addi 11,11,32 + lvx 24,10,1 + addi 10,10,32 + lvx 25,11,1 + addi 11,11,32 + lvx 26,10,1 + addi 10,10,32 + lvx 27,11,1 + addi 11,11,32 + lvx 28,10,1 + addi 10,10,32 + lvx 29,11,1 + addi 11,11,32 + lvx 30,10,1 + lvx 31,11,1 + ld 26,400(1) + ld 27,408(1) + ld 28,416(1) + ld 29,424(1) + ld 30,432(1) + ld 31,440(1) + addi 1,1,448 + blr +.long 0 +.byte 0,12,0x04,0,0x80,6,6,0 +.long 0 +.size crypton_aes_p8_cbc_encrypt,.-crypton_aes_p8_cbc_encrypt +.globl crypton_aes_p8_ctr32_encrypt_blocks +.type crypton_aes_p8_ctr32_encrypt_blocks,@function +.align 5 +crypton_aes_p8_ctr32_encrypt_blocks: +.localentry crypton_aes_p8_ctr32_encrypt_blocks,0 + + cmpldi 5,1 + .long 0x4dc00020 + + lis 0,0xfff0 + li 12,-1 + or 0,0,0 + + li 10,15 + vxor 0,0,0 + vspltisb 3,0x0f + + lvx 4,0,7 + lvsl 6,0,7 + lvx 5,10,7 + vspltisb 11,1 + vxor 6,6,3 + vperm 4,4,5,6 + vsldoi 11,0,11,1 + + neg 11,3 + lvsr 10,0,6 + lwz 9,240(6) + + lvsr 6,0,11 + lvx 5,0,3 + addi 3,3,15 + vxor 6,6,3 + + srwi 9,9,1 + li 10,16 + subi 9,9,1 + + cmpldi 5,8 + bge _aesp8_ctr32_encrypt8x + + lvsl 8,0,4 + vspltisb 9,-1 + lvx 7,0,4 + vperm 9,9,0,8 + vxor 8,8,3 + + lvx 0,0,6 + mtctr 9 + lvx 1,10,6 + addi 10,10,16 + vperm 0,1,0,10 + vxor 2,4,0 + lvx 0,10,6 + addi 10,10,16 + b .Loop_ctr32_enc + +.align 5 +.Loop_ctr32_enc: + vperm 1,0,1,10 + .long 0x10420D08 + lvx 1,10,6 + addi 10,10,16 + vperm 0,1,0,10 + .long 0x10420508 + lvx 0,10,6 + addi 10,10,16 + bdnz .Loop_ctr32_enc + + vadduwm 4,4,11 + vor 3,5,5 + lvx 5,0,3 + addi 3,3,16 + subic. 5,5,1 + + vperm 1,0,1,10 + .long 0x10420D08 + lvx 1,10,6 + vperm 3,3,5,6 + li 10,16 + vperm 1,1,0,10 + lvx 0,0,6 + vxor 3,3,1 + .long 0x10421D09 + + lvx 1,10,6 + addi 10,10,16 + vperm 2,2,2,8 + vsel 3,7,2,9 + mtctr 9 + vperm 0,1,0,10 + vor 7,2,2 + vxor 2,4,0 + lvx 0,10,6 + addi 10,10,16 + stvx 3,0,4 + addi 4,4,16 + bne .Loop_ctr32_enc + + addi 4,4,-1 + lvx 2,0,4 + vsel 2,7,2,9 + stvx 2,0,4 + + or 12,12,12 + blr +.long 0 +.byte 0,12,0x14,0,0,0,6,0 +.long 0 +.align 5 +_aesp8_ctr32_encrypt8x: + stdu 1,-448(1) + li 10,207 + li 11,223 + stvx 20,10,1 + addi 10,10,32 + stvx 21,11,1 + addi 11,11,32 + stvx 22,10,1 + addi 10,10,32 + stvx 23,11,1 + addi 11,11,32 + stvx 24,10,1 + addi 10,10,32 + stvx 25,11,1 + addi 11,11,32 + stvx 26,10,1 + addi 10,10,32 + stvx 27,11,1 + addi 11,11,32 + stvx 28,10,1 + addi 10,10,32 + stvx 29,11,1 + addi 11,11,32 + stvx 30,10,1 + stvx 31,11,1 + li 0,-1 + stw 12,396(1) + li 8,0x10 + std 26,400(1) + li 26,0x20 + std 27,408(1) + li 27,0x30 + std 28,416(1) + li 28,0x40 + std 29,424(1) + li 29,0x50 + std 30,432(1) + li 30,0x60 + std 31,440(1) + li 31,0x70 + or 0,0,0 + + subi 9,9,3 + + lvx 23,0,6 + lvx 30,8,6 + addi 6,6,0x20 + lvx 31,0,6 + vperm 23,30,23,10 + addi 11,1,64+15 + mtctr 9 + +.Load_ctr32_enc_key: + vperm 24,31,30,10 + lvx 30,8,6 + addi 6,6,0x20 + stvx 24,0,11 + vperm 25,30,31,10 + lvx 31,0,6 + stvx 25,8,11 + addi 11,11,0x20 + bdnz .Load_ctr32_enc_key + + lvx 26,8,6 + vperm 24,31,30,10 + lvx 27,26,6 + stvx 24,0,11 + vperm 25,26,31,10 + lvx 28,27,6 + stvx 25,8,11 + addi 11,1,64+15 + vperm 26,27,26,10 + lvx 29,28,6 + vperm 27,28,27,10 + lvx 30,29,6 + vperm 28,29,28,10 + lvx 31,30,6 + vperm 29,30,29,10 + lvx 15,31,6 + vperm 30,31,30,10 + lvx 24,0,11 + vperm 31,15,31,10 + lvx 25,8,11 + + vadduwm 7,11,11 + subi 3,3,15 + sldi 5,5,4 + + vadduwm 16,4,11 + vadduwm 17,4,7 + vxor 15,4,23 + li 10,8 + vadduwm 18,16,7 + vxor 16,16,23 + lvsl 6,0,10 + vadduwm 19,17,7 + vxor 17,17,23 + vspltisb 3,0x0f + vadduwm 20,18,7 + vxor 18,18,23 + vxor 6,6,3 + vadduwm 21,19,7 + vxor 19,19,23 + vadduwm 22,20,7 + vxor 20,20,23 + vadduwm 4,21,7 + vxor 21,21,23 + vxor 22,22,23 + + mtctr 9 + b .Loop_ctr32_enc8x +.align 5 +.Loop_ctr32_enc8x: + .long 0x11EFC508 + .long 0x1210C508 + .long 0x1231C508 + .long 0x1252C508 + .long 0x1273C508 + .long 0x1294C508 + .long 0x12B5C508 + .long 0x12D6C508 +.Loop_ctr32_enc8x_middle: + lvx 24,26,11 + addi 11,11,0x20 + + .long 0x11EFCD08 + .long 0x1210CD08 + .long 0x1231CD08 + .long 0x1252CD08 + .long 0x1273CD08 + .long 0x1294CD08 + .long 0x12B5CD08 + .long 0x12D6CD08 + lvx 25,8,11 + bdnz .Loop_ctr32_enc8x + + subic 11,5,256 + .long 0x11EFC508 + .long 0x1210C508 + .long 0x1231C508 + .long 0x1252C508 + .long 0x1273C508 + .long 0x1294C508 + .long 0x12B5C508 + .long 0x12D6C508 + + subfe 0,0,0 + .long 0x11EFCD08 + .long 0x1210CD08 + .long 0x1231CD08 + .long 0x1252CD08 + .long 0x1273CD08 + .long 0x1294CD08 + .long 0x12B5CD08 + .long 0x12D6CD08 + + and 0,0,11 + addi 11,1,64+15 + .long 0x11EFD508 + .long 0x1210D508 + .long 0x1231D508 + .long 0x1252D508 + .long 0x1273D508 + .long 0x1294D508 + .long 0x12B5D508 + .long 0x12D6D508 + lvx 24,0,11 + + subic 5,5,129 + .long 0x11EFDD08 + addi 5,5,1 + .long 0x1210DD08 + .long 0x1231DD08 + .long 0x1252DD08 + .long 0x1273DD08 + .long 0x1294DD08 + .long 0x12B5DD08 + .long 0x12D6DD08 + lvx 25,8,11 + + .long 0x11EFE508 + .long 0x7C001E99 + .long 0x1210E508 + .long 0x7C281E99 + .long 0x1231E508 + .long 0x7C5A1E99 + .long 0x1252E508 + .long 0x7C7B1E99 + .long 0x1273E508 + .long 0x7D5C1E99 + .long 0x1294E508 + .long 0x7D9D1E99 + .long 0x12B5E508 + .long 0x7DBE1E99 + .long 0x12D6E508 + .long 0x7DDF1E99 + addi 3,3,0x80 + + .long 0x11EFED08 + vperm 0,0,0,6 + .long 0x1210ED08 + vperm 1,1,1,6 + .long 0x1231ED08 + vperm 2,2,2,6 + .long 0x1252ED08 + vperm 3,3,3,6 + .long 0x1273ED08 + vperm 10,10,10,6 + .long 0x1294ED08 + vperm 12,12,12,6 + .long 0x12B5ED08 + vperm 13,13,13,6 + .long 0x12D6ED08 + vperm 14,14,14,6 + + add 3,3,0 + + + + subfe. 0,0,0 + .long 0x11EFF508 + vxor 0,0,31 + .long 0x1210F508 + vxor 1,1,31 + .long 0x1231F508 + vxor 2,2,31 + .long 0x1252F508 + vxor 3,3,31 + .long 0x1273F508 + vxor 10,10,31 + .long 0x1294F508 + vxor 12,12,31 + .long 0x12B5F508 + vxor 13,13,31 + .long 0x12D6F508 + vxor 14,14,31 + + bne .Lctr32_enc8x_break + + .long 0x100F0509 + .long 0x10300D09 + vadduwm 16,4,11 + .long 0x10511509 + vadduwm 17,4,7 + vxor 15,4,23 + .long 0x10721D09 + vadduwm 18,16,7 + vxor 16,16,23 + .long 0x11535509 + vadduwm 19,17,7 + vxor 17,17,23 + .long 0x11946509 + vadduwm 20,18,7 + vxor 18,18,23 + .long 0x11B56D09 + vadduwm 21,19,7 + vxor 19,19,23 + .long 0x11D67509 + vadduwm 22,20,7 + vxor 20,20,23 + vperm 0,0,0,6 + vadduwm 4,21,7 + vxor 21,21,23 + vperm 1,1,1,6 + vxor 22,22,23 + mtctr 9 + + .long 0x11EFC508 + .long 0x7C002799 + vperm 2,2,2,6 + .long 0x1210C508 + .long 0x7C282799 + vperm 3,3,3,6 + .long 0x1231C508 + .long 0x7C5A2799 + vperm 10,10,10,6 + .long 0x1252C508 + .long 0x7C7B2799 + vperm 12,12,12,6 + .long 0x1273C508 + .long 0x7D5C2799 + vperm 13,13,13,6 + .long 0x1294C508 + .long 0x7D9D2799 + vperm 14,14,14,6 + .long 0x12B5C508 + .long 0x7DBE2799 + .long 0x12D6C508 + .long 0x7DDF2799 + addi 4,4,0x80 + + b .Loop_ctr32_enc8x_middle + +.align 5 +.Lctr32_enc8x_break: + cmpwi 5,-0x60 + blt .Lctr32_enc8x_one + nop + beq .Lctr32_enc8x_two + cmpwi 5,-0x40 + blt .Lctr32_enc8x_three + nop + beq .Lctr32_enc8x_four + cmpwi 5,-0x20 + blt .Lctr32_enc8x_five + nop + beq .Lctr32_enc8x_six + cmpwi 5,0x00 + blt .Lctr32_enc8x_seven + +.Lctr32_enc8x_eight: + .long 0x11EF0509 + .long 0x12100D09 + .long 0x12311509 + .long 0x12521D09 + .long 0x12735509 + .long 0x12946509 + .long 0x12B56D09 + .long 0x12D67509 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + vperm 17,17,17,6 + .long 0x7E082799 + vperm 18,18,18,6 + .long 0x7E3A2799 + vperm 19,19,19,6 + .long 0x7E5B2799 + vperm 20,20,20,6 + .long 0x7E7C2799 + vperm 21,21,21,6 + .long 0x7E9D2799 + vperm 22,22,22,6 + .long 0x7EBE2799 + .long 0x7EDF2799 + addi 4,4,0x80 + b .Lctr32_enc8x_done + +.align 5 +.Lctr32_enc8x_seven: + .long 0x11EF0D09 + .long 0x12101509 + .long 0x12311D09 + .long 0x12525509 + .long 0x12736509 + .long 0x12946D09 + .long 0x12B57509 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + vperm 17,17,17,6 + .long 0x7E082799 + vperm 18,18,18,6 + .long 0x7E3A2799 + vperm 19,19,19,6 + .long 0x7E5B2799 + vperm 20,20,20,6 + .long 0x7E7C2799 + vperm 21,21,21,6 + .long 0x7E9D2799 + .long 0x7EBE2799 + addi 4,4,0x70 + b .Lctr32_enc8x_done + +.align 5 +.Lctr32_enc8x_six: + .long 0x11EF1509 + .long 0x12101D09 + .long 0x12315509 + .long 0x12526509 + .long 0x12736D09 + .long 0x12947509 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + vperm 17,17,17,6 + .long 0x7E082799 + vperm 18,18,18,6 + .long 0x7E3A2799 + vperm 19,19,19,6 + .long 0x7E5B2799 + vperm 20,20,20,6 + .long 0x7E7C2799 + .long 0x7E9D2799 + addi 4,4,0x60 + b .Lctr32_enc8x_done + +.align 5 +.Lctr32_enc8x_five: + .long 0x11EF1D09 + .long 0x12105509 + .long 0x12316509 + .long 0x12526D09 + .long 0x12737509 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + vperm 17,17,17,6 + .long 0x7E082799 + vperm 18,18,18,6 + .long 0x7E3A2799 + vperm 19,19,19,6 + .long 0x7E5B2799 + .long 0x7E7C2799 + addi 4,4,0x50 + b .Lctr32_enc8x_done + +.align 5 +.Lctr32_enc8x_four: + .long 0x11EF5509 + .long 0x12106509 + .long 0x12316D09 + .long 0x12527509 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + vperm 17,17,17,6 + .long 0x7E082799 + vperm 18,18,18,6 + .long 0x7E3A2799 + .long 0x7E5B2799 + addi 4,4,0x40 + b .Lctr32_enc8x_done + +.align 5 +.Lctr32_enc8x_three: + .long 0x11EF6509 + .long 0x12106D09 + .long 0x12317509 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + vperm 17,17,17,6 + .long 0x7E082799 + .long 0x7E3A2799 + addi 4,4,0x30 + b .Lctr32_enc8x_done + +.align 5 +.Lctr32_enc8x_two: + .long 0x11EF6D09 + .long 0x12107509 + + vperm 15,15,15,6 + vperm 16,16,16,6 + .long 0x7DE02799 + .long 0x7E082799 + addi 4,4,0x20 + b .Lctr32_enc8x_done + +.align 5 +.Lctr32_enc8x_one: + .long 0x11EF7509 + + vperm 15,15,15,6 + .long 0x7DE02799 + addi 4,4,0x10 + +.Lctr32_enc8x_done: + li 10,79 + li 11,95 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + stvx 6,10,1 + addi 10,10,32 + stvx 6,11,1 + addi 11,11,32 + + or 12,12,12 + lvx 20,10,1 + addi 10,10,32 + lvx 21,11,1 + addi 11,11,32 + lvx 22,10,1 + addi 10,10,32 + lvx 23,11,1 + addi 11,11,32 + lvx 24,10,1 + addi 10,10,32 + lvx 25,11,1 + addi 11,11,32 + lvx 26,10,1 + addi 10,10,32 + lvx 27,11,1 + addi 11,11,32 + lvx 28,10,1 + addi 10,10,32 + lvx 29,11,1 + addi 11,11,32 + lvx 30,10,1 + lvx 31,11,1 + ld 26,400(1) + ld 27,408(1) + ld 28,416(1) + ld 29,424(1) + ld 30,432(1) + ld 31,440(1) + addi 1,1,448 + blr +.long 0 +.byte 0,12,0x04,0,0x80,6,6,0 +.long 0 +.size crypton_aes_p8_ctr32_encrypt_blocks,.-crypton_aes_p8_ctr32_encrypt_blocks +.globl crypton_aes_p8_xts_encrypt +.type crypton_aes_p8_xts_encrypt,@function +.align 5 +crypton_aes_p8_xts_encrypt: +.localentry crypton_aes_p8_xts_encrypt,0 + + mr 10,3 + li 3,-1 + cmpldi 5,16 + .long 0x4dc00020 + + lis 0,0xfff0 + li 12,-1 + li 11,0 + or 0,0,0 + + vspltisb 9,0x07 + lvsl 6,11,11 + vspltisb 11,0x0f + vxor 6,6,9 + + li 3,15 + lvx 8,0,8 + lvsl 5,0,8 + lvx 4,3,8 + vxor 5,5,11 + vperm 8,8,4,5 + + neg 11,10 + lvsr 5,0,11 + lvx 2,0,10 + addi 10,10,15 + vxor 5,5,11 + + cmpldi 7,0 + beq .Lxts_enc_no_key2 + + lvsr 7,0,7 + lwz 9,240(7) + srwi 9,9,1 + subi 9,9,1 + li 3,16 + + lvx 0,0,7 + lvx 1,3,7 + addi 3,3,16 + vperm 0,1,0,7 + vxor 8,8,0 + lvx 0,3,7 + addi 3,3,16 + mtctr 9 + +.Ltweak_xts_enc: + vperm 1,0,1,7 + .long 0x11080D08 + lvx 1,3,7 + addi 3,3,16 + vperm 0,1,0,7 + .long 0x11080508 + lvx 0,3,7 + addi 3,3,16 + bdnz .Ltweak_xts_enc + + vperm 1,0,1,7 + .long 0x11080D08 + lvx 1,3,7 + vperm 0,1,0,7 + .long 0x11080509 + + li 8,0 + b .Lxts_enc + +.Lxts_enc_no_key2: + li 3,-16 + and 5,5,3 + + +.Lxts_enc: + lvx 4,0,10 + addi 10,10,16 + + lvsr 7,0,6 + lwz 9,240(6) + srwi 9,9,1 + subi 9,9,1 + li 3,16 + + vslb 10,9,9 + vor 10,10,9 + vspltisb 11,1 + vsldoi 10,10,11,15 + + cmpldi 5,96 + bge _aesp8_xts_encrypt6x + + andi. 7,5,15 + subic 0,5,32 + subi 7,7,16 + subfe 0,0,0 + and 0,0,7 + add 10,10,0 + + lvx 0,0,6 + lvx 1,3,6 + addi 3,3,16 + vperm 2,2,4,5 + vperm 0,1,0,7 + vxor 2,2,8 + vxor 2,2,0 + lvx 0,3,6 + addi 3,3,16 + mtctr 9 + b .Loop_xts_enc + +.align 5 +.Loop_xts_enc: + vperm 1,0,1,7 + .long 0x10420D08 + lvx 1,3,6 + addi 3,3,16 + vperm 0,1,0,7 + .long 0x10420508 + lvx 0,3,6 + addi 3,3,16 + bdnz .Loop_xts_enc + + vperm 1,0,1,7 + .long 0x10420D08 + lvx 1,3,6 + li 3,16 + vperm 0,1,0,7 + vxor 0,0,8 + .long 0x10620509 + + vperm 11,3,3,6 + + .long 0x7D602799 + + addi 4,4,16 + + subic. 5,5,16 + beq .Lxts_enc_done + + vor 2,4,4 + lvx 4,0,10 + addi 10,10,16 + lvx 0,0,6 + lvx 1,3,6 + addi 3,3,16 + + subic 0,5,32 + subfe 0,0,0 + and 0,0,7 + add 10,10,0 + + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vand 11,11,10 + vxor 8,8,11 + + vperm 2,2,4,5 + vperm 0,1,0,7 + vxor 2,2,8 + vxor 3,3,0 + vxor 2,2,0 + lvx 0,3,6 + addi 3,3,16 + + mtctr 9 + cmpldi 5,16 + bge .Loop_xts_enc + + vxor 3,3,8 + lvsr 5,0,5 + vxor 4,4,4 + vspltisb 11,-1 + vperm 4,4,11,5 + vsel 2,2,3,4 + + subi 11,4,17 + subi 4,4,16 + mtctr 5 + li 5,16 +.Loop_xts_enc_steal: + lbzu 0,1(11) + stb 0,16(11) + bdnz .Loop_xts_enc_steal + + mtctr 9 + b .Loop_xts_enc + +.Lxts_enc_done: + cmpldi 8,0 + beq .Lxts_enc_ret + + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vand 11,11,10 + vxor 8,8,11 + + vperm 8,8,8,6 + .long 0x7D004799 + +.Lxts_enc_ret: + or 12,12,12 + li 3,0 + blr +.long 0 +.byte 0,12,0x04,0,0x80,6,6,0 +.long 0 +.size crypton_aes_p8_xts_encrypt,.-crypton_aes_p8_xts_encrypt + +.globl crypton_aes_p8_xts_decrypt +.type crypton_aes_p8_xts_decrypt,@function +.align 5 +crypton_aes_p8_xts_decrypt: +.localentry crypton_aes_p8_xts_decrypt,0 + + mr 10,3 + li 3,-1 + cmpldi 5,16 + .long 0x4dc00020 + + lis 0,0xfff8 + li 12,-1 + li 11,0 + or 0,0,0 + + andi. 0,5,15 + neg 0,0 + andi. 0,0,16 + sub 5,5,0 + + vspltisb 9,0x07 + lvsl 6,11,11 + vspltisb 11,0x0f + vxor 6,6,9 + + li 3,15 + lvx 8,0,8 + lvsl 5,0,8 + lvx 4,3,8 + vxor 5,5,11 + vperm 8,8,4,5 + + neg 11,10 + lvsr 5,0,11 + lvx 2,0,10 + addi 10,10,15 + vxor 5,5,11 + + cmpldi 7,0 + beq .Lxts_dec_no_key2 + + lvsr 7,0,7 + lwz 9,240(7) + srwi 9,9,1 + subi 9,9,1 + li 3,16 + + lvx 0,0,7 + lvx 1,3,7 + addi 3,3,16 + vperm 0,1,0,7 + vxor 8,8,0 + lvx 0,3,7 + addi 3,3,16 + mtctr 9 + +.Ltweak_xts_dec: + vperm 1,0,1,7 + .long 0x11080D08 + lvx 1,3,7 + addi 3,3,16 + vperm 0,1,0,7 + .long 0x11080508 + lvx 0,3,7 + addi 3,3,16 + bdnz .Ltweak_xts_dec + + vperm 1,0,1,7 + .long 0x11080D08 + lvx 1,3,7 + vperm 0,1,0,7 + .long 0x11080509 + + li 8,0 + b .Lxts_dec + +.Lxts_dec_no_key2: + neg 3,5 + andi. 3,3,15 + add 5,5,3 + + +.Lxts_dec: + lvx 4,0,10 + addi 10,10,16 + + lvsr 7,0,6 + lwz 9,240(6) + srwi 9,9,1 + subi 9,9,1 + li 3,16 + + vslb 10,9,9 + vor 10,10,9 + vspltisb 11,1 + vsldoi 10,10,11,15 + + cmpldi 5,96 + bge _aesp8_xts_decrypt6x + + lvx 0,0,6 + lvx 1,3,6 + addi 3,3,16 + vperm 2,2,4,5 + vperm 0,1,0,7 + vxor 2,2,8 + vxor 2,2,0 + lvx 0,3,6 + addi 3,3,16 + mtctr 9 + + cmpldi 5,16 + blt .Ltail_xts_dec + + +.align 5 +.Loop_xts_dec: + vperm 1,0,1,7 + .long 0x10420D48 + lvx 1,3,6 + addi 3,3,16 + vperm 0,1,0,7 + .long 0x10420548 + lvx 0,3,6 + addi 3,3,16 + bdnz .Loop_xts_dec + + vperm 1,0,1,7 + .long 0x10420D48 + lvx 1,3,6 + li 3,16 + vperm 0,1,0,7 + vxor 0,0,8 + .long 0x10620549 + + vperm 11,3,3,6 + + .long 0x7D602799 + + addi 4,4,16 + + subic. 5,5,16 + beq .Lxts_dec_done + + vor 2,4,4 + lvx 4,0,10 + addi 10,10,16 + lvx 0,0,6 + lvx 1,3,6 + addi 3,3,16 + + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vand 11,11,10 + vxor 8,8,11 + + vperm 2,2,4,5 + vperm 0,1,0,7 + vxor 2,2,8 + vxor 2,2,0 + lvx 0,3,6 + addi 3,3,16 + + mtctr 9 + cmpldi 5,16 + bge .Loop_xts_dec + +.Ltail_xts_dec: + vsrab 11,8,9 + vaddubm 12,8,8 + vsldoi 11,11,11,15 + vand 11,11,10 + vxor 12,12,11 + + subi 10,10,16 + add 10,10,5 + + vxor 2,2,8 + vxor 2,2,12 + +.Loop_xts_dec_short: + vperm 1,0,1,7 + .long 0x10420D48 + lvx 1,3,6 + addi 3,3,16 + vperm 0,1,0,7 + .long 0x10420548 + lvx 0,3,6 + addi 3,3,16 + bdnz .Loop_xts_dec_short + + vperm 1,0,1,7 + .long 0x10420D48 + lvx 1,3,6 + li 3,16 + vperm 0,1,0,7 + vxor 0,0,12 + .long 0x10620549 + + vperm 11,3,3,6 + + .long 0x7D602799 + + + vor 2,4,4 + lvx 4,0,10 + + lvx 0,0,6 + lvx 1,3,6 + addi 3,3,16 + vperm 2,2,4,5 + vperm 0,1,0,7 + + lvsr 5,0,5 + vxor 4,4,4 + vspltisb 11,-1 + vperm 4,4,11,5 + vsel 2,2,3,4 + + vxor 0,0,8 + vxor 2,2,0 + lvx 0,3,6 + addi 3,3,16 + + subi 11,4,1 + mtctr 5 + li 5,16 +.Loop_xts_dec_steal: + lbzu 0,1(11) + stb 0,16(11) + bdnz .Loop_xts_dec_steal + + mtctr 9 + b .Loop_xts_dec + +.Lxts_dec_done: + cmpldi 8,0 + beq .Lxts_dec_ret + + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vand 11,11,10 + vxor 8,8,11 + + vperm 8,8,8,6 + .long 0x7D004799 + +.Lxts_dec_ret: + or 12,12,12 + li 3,0 + blr +.long 0 +.byte 0,12,0x04,0,0x80,6,6,0 +.long 0 +.size crypton_aes_p8_xts_decrypt,.-crypton_aes_p8_xts_decrypt +.align 5 +_aesp8_xts_encrypt6x: + stdu 1,-448(1) + mflr 11 + li 7,207 + li 3,223 + std 11,464(1) + stvx 20,7,1 + addi 7,7,32 + stvx 21,3,1 + addi 3,3,32 + stvx 22,7,1 + addi 7,7,32 + stvx 23,3,1 + addi 3,3,32 + stvx 24,7,1 + addi 7,7,32 + stvx 25,3,1 + addi 3,3,32 + stvx 26,7,1 + addi 7,7,32 + stvx 27,3,1 + addi 3,3,32 + stvx 28,7,1 + addi 7,7,32 + stvx 29,3,1 + addi 3,3,32 + stvx 30,7,1 + stvx 31,3,1 + li 0,-1 + stw 12,396(1) + li 3,0x10 + std 26,400(1) + li 26,0x20 + std 27,408(1) + li 27,0x30 + std 28,416(1) + li 28,0x40 + std 29,424(1) + li 29,0x50 + std 30,432(1) + li 30,0x60 + std 31,440(1) + li 31,0x70 + or 0,0,0 + + subi 9,9,3 + + lvx 23,0,6 + lvx 30,3,6 + addi 6,6,0x20 + lvx 31,0,6 + vperm 23,30,23,7 + addi 7,1,64+15 + mtctr 9 + +.Load_xts_enc_key: + vperm 24,31,30,7 + lvx 30,3,6 + addi 6,6,0x20 + stvx 24,0,7 + vperm 25,30,31,7 + lvx 31,0,6 + stvx 25,3,7 + addi 7,7,0x20 + bdnz .Load_xts_enc_key + + lvx 26,3,6 + vperm 24,31,30,7 + lvx 27,26,6 + stvx 24,0,7 + vperm 25,26,31,7 + lvx 28,27,6 + stvx 25,3,7 + addi 7,1,64+15 + vperm 26,27,26,7 + lvx 29,28,6 + vperm 27,28,27,7 + lvx 30,29,6 + vperm 28,29,28,7 + lvx 31,30,6 + vperm 29,30,29,7 + lvx 22,31,6 + vperm 30,31,30,7 + lvx 24,0,7 + vperm 31,22,31,7 + lvx 25,3,7 + + vperm 0,2,4,5 + subi 10,10,31 + vxor 17,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vand 11,11,10 + vxor 7,0,17 + vxor 8,8,11 + + .long 0x7C235699 + vxor 18,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 1,1,1,6 + vand 11,11,10 + vxor 12,1,18 + vxor 8,8,11 + + .long 0x7C5A5699 + andi. 31,5,15 + vxor 19,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 2,2,2,6 + vand 11,11,10 + vxor 13,2,19 + vxor 8,8,11 + + .long 0x7C7B5699 + sub 5,5,31 + vxor 20,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 3,3,3,6 + vand 11,11,10 + vxor 14,3,20 + vxor 8,8,11 + + .long 0x7C9C5699 + subi 5,5,0x60 + vxor 21,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 4,4,4,6 + vand 11,11,10 + vxor 15,4,21 + vxor 8,8,11 + + .long 0x7CBD5699 + addi 10,10,0x60 + vxor 22,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 5,5,5,6 + vand 11,11,10 + vxor 16,5,22 + vxor 8,8,11 + + vxor 31,31,23 + mtctr 9 + b .Loop_xts_enc6x + +.align 5 +.Loop_xts_enc6x: + .long 0x10E7C508 + .long 0x118CC508 + .long 0x11ADC508 + .long 0x11CEC508 + .long 0x11EFC508 + .long 0x1210C508 + lvx 24,26,7 + addi 7,7,0x20 + + .long 0x10E7CD08 + .long 0x118CCD08 + .long 0x11ADCD08 + .long 0x11CECD08 + .long 0x11EFCD08 + .long 0x1210CD08 + lvx 25,3,7 + bdnz .Loop_xts_enc6x + + subic 5,5,96 + vxor 0,17,31 + .long 0x10E7C508 + .long 0x118CC508 + vsrab 11,8,9 + vxor 17,8,23 + vaddubm 8,8,8 + .long 0x11ADC508 + .long 0x11CEC508 + vsldoi 11,11,11,15 + .long 0x11EFC508 + .long 0x1210C508 + + subfe. 0,0,0 + vand 11,11,10 + .long 0x10E7CD08 + .long 0x118CCD08 + vxor 8,8,11 + .long 0x11ADCD08 + .long 0x11CECD08 + vxor 1,18,31 + vsrab 11,8,9 + vxor 18,8,23 + .long 0x11EFCD08 + .long 0x1210CD08 + + and 0,0,5 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + .long 0x10E7D508 + .long 0x118CD508 + vand 11,11,10 + .long 0x11ADD508 + .long 0x11CED508 + vxor 8,8,11 + .long 0x11EFD508 + .long 0x1210D508 + + add 10,10,0 + + + + vxor 2,19,31 + vsrab 11,8,9 + vxor 19,8,23 + vaddubm 8,8,8 + .long 0x10E7DD08 + .long 0x118CDD08 + vsldoi 11,11,11,15 + .long 0x11ADDD08 + .long 0x11CEDD08 + vand 11,11,10 + .long 0x11EFDD08 + .long 0x1210DD08 + + addi 7,1,64+15 + vxor 8,8,11 + .long 0x10E7E508 + .long 0x118CE508 + vxor 3,20,31 + vsrab 11,8,9 + vxor 20,8,23 + .long 0x11ADE508 + .long 0x11CEE508 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + .long 0x11EFE508 + .long 0x1210E508 + lvx 24,0,7 + vand 11,11,10 + + .long 0x10E7ED08 + .long 0x118CED08 + vxor 8,8,11 + .long 0x11ADED08 + .long 0x11CEED08 + vxor 4,21,31 + vsrab 11,8,9 + vxor 21,8,23 + .long 0x11EFED08 + .long 0x1210ED08 + lvx 25,3,7 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + + .long 0x10E7F508 + .long 0x118CF508 + vand 11,11,10 + .long 0x11ADF508 + .long 0x11CEF508 + vxor 8,8,11 + .long 0x11EFF508 + .long 0x1210F508 + vxor 5,22,31 + vsrab 11,8,9 + vxor 22,8,23 + + .long 0x10E70509 + .long 0x7C005699 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + .long 0x118C0D09 + .long 0x7C235699 + .long 0x11AD1509 + vperm 0,0,0,6 + .long 0x7C5A5699 + vand 11,11,10 + .long 0x11CE1D09 + vperm 1,1,1,6 + .long 0x7C7B5699 + .long 0x11EF2509 + vperm 2,2,2,6 + .long 0x7C9C5699 + vxor 8,8,11 + .long 0x11702D09 + + vperm 3,3,3,6 + .long 0x7CBD5699 + addi 10,10,0x60 + vperm 4,4,4,6 + vperm 5,5,5,6 + + vperm 7,7,7,6 + vperm 12,12,12,6 + .long 0x7CE02799 + vxor 7,0,17 + vperm 13,13,13,6 + .long 0x7D832799 + vxor 12,1,18 + vperm 14,14,14,6 + .long 0x7DBA2799 + vxor 13,2,19 + vperm 15,15,15,6 + .long 0x7DDB2799 + vxor 14,3,20 + vperm 16,11,11,6 + .long 0x7DFC2799 + vxor 15,4,21 + .long 0x7E1D2799 + + vxor 16,5,22 + addi 4,4,0x60 + + mtctr 9 + beq .Loop_xts_enc6x + + addic. 5,5,0x60 + beq .Lxts_enc6x_zero + cmpwi 5,0x20 + blt .Lxts_enc6x_one + nop + beq .Lxts_enc6x_two + cmpwi 5,0x40 + blt .Lxts_enc6x_three + nop + beq .Lxts_enc6x_four + +.Lxts_enc6x_five: + vxor 7,1,17 + vxor 12,2,18 + vxor 13,3,19 + vxor 14,4,20 + vxor 15,5,21 + + bl _aesp8_xts_enc5x + + vperm 7,7,7,6 + vor 17,22,22 + vperm 12,12,12,6 + .long 0x7CE02799 + vperm 13,13,13,6 + .long 0x7D832799 + vperm 14,14,14,6 + .long 0x7DBA2799 + vxor 11,15,22 + vperm 15,15,15,6 + .long 0x7DDB2799 + .long 0x7DFC2799 + addi 4,4,0x50 + bne .Lxts_enc6x_steal + b .Lxts_enc6x_done + +.align 4 +.Lxts_enc6x_four: + vxor 7,2,17 + vxor 12,3,18 + vxor 13,4,19 + vxor 14,5,20 + vxor 15,15,15 + + bl _aesp8_xts_enc5x + + vperm 7,7,7,6 + vor 17,21,21 + vperm 12,12,12,6 + .long 0x7CE02799 + vperm 13,13,13,6 + .long 0x7D832799 + vxor 11,14,21 + vperm 14,14,14,6 + .long 0x7DBA2799 + .long 0x7DDB2799 + addi 4,4,0x40 + bne .Lxts_enc6x_steal + b .Lxts_enc6x_done + +.align 4 +.Lxts_enc6x_three: + vxor 7,3,17 + vxor 12,4,18 + vxor 13,5,19 + vxor 14,14,14 + vxor 15,15,15 + + bl _aesp8_xts_enc5x + + vperm 7,7,7,6 + vor 17,20,20 + vperm 12,12,12,6 + .long 0x7CE02799 + vxor 11,13,20 + vperm 13,13,13,6 + .long 0x7D832799 + .long 0x7DBA2799 + addi 4,4,0x30 + bne .Lxts_enc6x_steal + b .Lxts_enc6x_done + +.align 4 +.Lxts_enc6x_two: + vxor 7,4,17 + vxor 12,5,18 + vxor 13,13,13 + vxor 14,14,14 + vxor 15,15,15 + + bl _aesp8_xts_enc5x + + vperm 7,7,7,6 + vor 17,19,19 + vxor 11,12,19 + vperm 12,12,12,6 + .long 0x7CE02799 + .long 0x7D832799 + addi 4,4,0x20 + bne .Lxts_enc6x_steal + b .Lxts_enc6x_done + +.align 4 +.Lxts_enc6x_one: + vxor 7,5,17 + nop +.Loop_xts_enc1x: + .long 0x10E7C508 + lvx 24,26,7 + addi 7,7,0x20 + + .long 0x10E7CD08 + lvx 25,3,7 + bdnz .Loop_xts_enc1x + + add 10,10,31 + cmpwi 31,0 + .long 0x10E7C508 + + subi 10,10,16 + .long 0x10E7CD08 + + lvsr 5,0,31 + .long 0x10E7D508 + + .long 0x7C005699 + .long 0x10E7DD08 + + addi 7,1,64+15 + .long 0x10E7E508 + lvx 24,0,7 + + .long 0x10E7ED08 + lvx 25,3,7 + vxor 17,17,31 + + vperm 0,0,0,6 + .long 0x10E7F508 + + vperm 0,0,0,5 + .long 0x10E78D09 + + vor 17,18,18 + vxor 11,7,18 + vperm 7,7,7,6 + .long 0x7CE02799 + addi 4,4,0x10 + bne .Lxts_enc6x_steal + b .Lxts_enc6x_done + +.align 4 +.Lxts_enc6x_zero: + cmpwi 31,0 + beq .Lxts_enc6x_done + + add 10,10,31 + subi 10,10,16 + .long 0x7C005699 + lvsr 5,0,31 + vperm 0,0,0,6 + vperm 0,0,0,5 + vxor 11,11,17 +.Lxts_enc6x_steal: + vxor 0,0,17 + vxor 7,7,7 + vspltisb 12,-1 + vperm 7,7,12,5 + vsel 7,0,11,7 + + subi 30,4,17 + subi 4,4,16 + mtctr 31 +.Loop_xts_enc6x_steal: + lbzu 0,1(30) + stb 0,16(30) + bdnz .Loop_xts_enc6x_steal + + li 31,0 + mtctr 9 + b .Loop_xts_enc1x + +.align 4 +.Lxts_enc6x_done: + cmpldi 8,0 + beq .Lxts_enc6x_ret + + vxor 8,17,23 + vperm 8,8,8,6 + .long 0x7D004799 + +.Lxts_enc6x_ret: + mtlr 11 + li 10,79 + li 11,95 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + + or 12,12,12 + lvx 20,10,1 + addi 10,10,32 + lvx 21,11,1 + addi 11,11,32 + lvx 22,10,1 + addi 10,10,32 + lvx 23,11,1 + addi 11,11,32 + lvx 24,10,1 + addi 10,10,32 + lvx 25,11,1 + addi 11,11,32 + lvx 26,10,1 + addi 10,10,32 + lvx 27,11,1 + addi 11,11,32 + lvx 28,10,1 + addi 10,10,32 + lvx 29,11,1 + addi 11,11,32 + lvx 30,10,1 + lvx 31,11,1 + ld 26,400(1) + ld 27,408(1) + ld 28,416(1) + ld 29,424(1) + ld 30,432(1) + ld 31,440(1) + addi 1,1,448 + blr +.long 0 +.byte 0,12,0x04,1,0x80,6,6,0 +.long 0 + +.align 5 +_aesp8_xts_enc5x: + .long 0x10E7C508 + .long 0x118CC508 + .long 0x11ADC508 + .long 0x11CEC508 + .long 0x11EFC508 + lvx 24,26,7 + addi 7,7,0x20 + + .long 0x10E7CD08 + .long 0x118CCD08 + .long 0x11ADCD08 + .long 0x11CECD08 + .long 0x11EFCD08 + lvx 25,3,7 + bdnz _aesp8_xts_enc5x + + add 10,10,31 + cmpwi 31,0 + .long 0x10E7C508 + .long 0x118CC508 + .long 0x11ADC508 + .long 0x11CEC508 + .long 0x11EFC508 + + subi 10,10,16 + .long 0x10E7CD08 + .long 0x118CCD08 + .long 0x11ADCD08 + .long 0x11CECD08 + .long 0x11EFCD08 + vxor 17,17,31 + + .long 0x10E7D508 + lvsr 5,0,31 + .long 0x118CD508 + .long 0x11ADD508 + .long 0x11CED508 + .long 0x11EFD508 + vxor 1,18,31 + + .long 0x10E7DD08 + .long 0x7C005699 + .long 0x118CDD08 + .long 0x11ADDD08 + .long 0x11CEDD08 + .long 0x11EFDD08 + vxor 2,19,31 + + addi 7,1,64+15 + .long 0x10E7E508 + .long 0x118CE508 + .long 0x11ADE508 + .long 0x11CEE508 + .long 0x11EFE508 + lvx 24,0,7 + vxor 3,20,31 + + .long 0x10E7ED08 + vperm 0,0,0,6 + .long 0x118CED08 + .long 0x11ADED08 + .long 0x11CEED08 + .long 0x11EFED08 + lvx 25,3,7 + vxor 4,21,31 + + .long 0x10E7F508 + vperm 0,0,0,5 + .long 0x118CF508 + .long 0x11ADF508 + .long 0x11CEF508 + .long 0x11EFF508 + + .long 0x10E78D09 + .long 0x118C0D09 + .long 0x11AD1509 + .long 0x11CE1D09 + .long 0x11EF2509 + blr +.long 0 +.byte 0,12,0x14,0,0,0,0,0 + +.align 5 +_aesp8_xts_decrypt6x: + stdu 1,-448(1) + mflr 11 + li 7,207 + li 3,223 + std 11,464(1) + stvx 20,7,1 + addi 7,7,32 + stvx 21,3,1 + addi 3,3,32 + stvx 22,7,1 + addi 7,7,32 + stvx 23,3,1 + addi 3,3,32 + stvx 24,7,1 + addi 7,7,32 + stvx 25,3,1 + addi 3,3,32 + stvx 26,7,1 + addi 7,7,32 + stvx 27,3,1 + addi 3,3,32 + stvx 28,7,1 + addi 7,7,32 + stvx 29,3,1 + addi 3,3,32 + stvx 30,7,1 + stvx 31,3,1 + li 0,-1 + stw 12,396(1) + li 3,0x10 + std 26,400(1) + li 26,0x20 + std 27,408(1) + li 27,0x30 + std 28,416(1) + li 28,0x40 + std 29,424(1) + li 29,0x50 + std 30,432(1) + li 30,0x60 + std 31,440(1) + li 31,0x70 + or 0,0,0 + + subi 9,9,3 + + lvx 23,0,6 + lvx 30,3,6 + addi 6,6,0x20 + lvx 31,0,6 + vperm 23,30,23,7 + addi 7,1,64+15 + mtctr 9 + +.Load_xts_dec_key: + vperm 24,31,30,7 + lvx 30,3,6 + addi 6,6,0x20 + stvx 24,0,7 + vperm 25,30,31,7 + lvx 31,0,6 + stvx 25,3,7 + addi 7,7,0x20 + bdnz .Load_xts_dec_key + + lvx 26,3,6 + vperm 24,31,30,7 + lvx 27,26,6 + stvx 24,0,7 + vperm 25,26,31,7 + lvx 28,27,6 + stvx 25,3,7 + addi 7,1,64+15 + vperm 26,27,26,7 + lvx 29,28,6 + vperm 27,28,27,7 + lvx 30,29,6 + vperm 28,29,28,7 + lvx 31,30,6 + vperm 29,30,29,7 + lvx 22,31,6 + vperm 30,31,30,7 + lvx 24,0,7 + vperm 31,22,31,7 + lvx 25,3,7 + + vperm 0,2,4,5 + subi 10,10,31 + vxor 17,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vand 11,11,10 + vxor 7,0,17 + vxor 8,8,11 + + .long 0x7C235699 + vxor 18,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 1,1,1,6 + vand 11,11,10 + vxor 12,1,18 + vxor 8,8,11 + + .long 0x7C5A5699 + andi. 31,5,15 + vxor 19,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 2,2,2,6 + vand 11,11,10 + vxor 13,2,19 + vxor 8,8,11 + + .long 0x7C7B5699 + sub 5,5,31 + vxor 20,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 3,3,3,6 + vand 11,11,10 + vxor 14,3,20 + vxor 8,8,11 + + .long 0x7C9C5699 + subi 5,5,0x60 + vxor 21,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 4,4,4,6 + vand 11,11,10 + vxor 15,4,21 + vxor 8,8,11 + + .long 0x7CBD5699 + addi 10,10,0x60 + vxor 22,8,23 + vsrab 11,8,9 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + vperm 5,5,5,6 + vand 11,11,10 + vxor 16,5,22 + vxor 8,8,11 + + vxor 31,31,23 + mtctr 9 + b .Loop_xts_dec6x + +.align 5 +.Loop_xts_dec6x: + .long 0x10E7C548 + .long 0x118CC548 + .long 0x11ADC548 + .long 0x11CEC548 + .long 0x11EFC548 + .long 0x1210C548 + lvx 24,26,7 + addi 7,7,0x20 + + .long 0x10E7CD48 + .long 0x118CCD48 + .long 0x11ADCD48 + .long 0x11CECD48 + .long 0x11EFCD48 + .long 0x1210CD48 + lvx 25,3,7 + bdnz .Loop_xts_dec6x + + subic 5,5,96 + vxor 0,17,31 + .long 0x10E7C548 + .long 0x118CC548 + vsrab 11,8,9 + vxor 17,8,23 + vaddubm 8,8,8 + .long 0x11ADC548 + .long 0x11CEC548 + vsldoi 11,11,11,15 + .long 0x11EFC548 + .long 0x1210C548 + + subfe. 0,0,0 + vand 11,11,10 + .long 0x10E7CD48 + .long 0x118CCD48 + vxor 8,8,11 + .long 0x11ADCD48 + .long 0x11CECD48 + vxor 1,18,31 + vsrab 11,8,9 + vxor 18,8,23 + .long 0x11EFCD48 + .long 0x1210CD48 + + and 0,0,5 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + .long 0x10E7D548 + .long 0x118CD548 + vand 11,11,10 + .long 0x11ADD548 + .long 0x11CED548 + vxor 8,8,11 + .long 0x11EFD548 + .long 0x1210D548 + + add 10,10,0 + + + + vxor 2,19,31 + vsrab 11,8,9 + vxor 19,8,23 + vaddubm 8,8,8 + .long 0x10E7DD48 + .long 0x118CDD48 + vsldoi 11,11,11,15 + .long 0x11ADDD48 + .long 0x11CEDD48 + vand 11,11,10 + .long 0x11EFDD48 + .long 0x1210DD48 + + addi 7,1,64+15 + vxor 8,8,11 + .long 0x10E7E548 + .long 0x118CE548 + vxor 3,20,31 + vsrab 11,8,9 + vxor 20,8,23 + .long 0x11ADE548 + .long 0x11CEE548 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + .long 0x11EFE548 + .long 0x1210E548 + lvx 24,0,7 + vand 11,11,10 + + .long 0x10E7ED48 + .long 0x118CED48 + vxor 8,8,11 + .long 0x11ADED48 + .long 0x11CEED48 + vxor 4,21,31 + vsrab 11,8,9 + vxor 21,8,23 + .long 0x11EFED48 + .long 0x1210ED48 + lvx 25,3,7 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + + .long 0x10E7F548 + .long 0x118CF548 + vand 11,11,10 + .long 0x11ADF548 + .long 0x11CEF548 + vxor 8,8,11 + .long 0x11EFF548 + .long 0x1210F548 + vxor 5,22,31 + vsrab 11,8,9 + vxor 22,8,23 + + .long 0x10E70549 + .long 0x7C005699 + vaddubm 8,8,8 + vsldoi 11,11,11,15 + .long 0x118C0D49 + .long 0x7C235699 + .long 0x11AD1549 + vperm 0,0,0,6 + .long 0x7C5A5699 + vand 11,11,10 + .long 0x11CE1D49 + vperm 1,1,1,6 + .long 0x7C7B5699 + .long 0x11EF2549 + vperm 2,2,2,6 + .long 0x7C9C5699 + vxor 8,8,11 + .long 0x12102D49 + vperm 3,3,3,6 + .long 0x7CBD5699 + addi 10,10,0x60 + vperm 4,4,4,6 + vperm 5,5,5,6 + + vperm 7,7,7,6 + vperm 12,12,12,6 + .long 0x7CE02799 + vxor 7,0,17 + vperm 13,13,13,6 + .long 0x7D832799 + vxor 12,1,18 + vperm 14,14,14,6 + .long 0x7DBA2799 + vxor 13,2,19 + vperm 15,15,15,6 + .long 0x7DDB2799 + vxor 14,3,20 + vperm 16,16,16,6 + .long 0x7DFC2799 + vxor 15,4,21 + .long 0x7E1D2799 + vxor 16,5,22 + addi 4,4,0x60 + + mtctr 9 + beq .Loop_xts_dec6x + + addic. 5,5,0x60 + beq .Lxts_dec6x_zero + cmpwi 5,0x20 + blt .Lxts_dec6x_one + nop + beq .Lxts_dec6x_two + cmpwi 5,0x40 + blt .Lxts_dec6x_three + nop + beq .Lxts_dec6x_four + +.Lxts_dec6x_five: + vxor 7,1,17 + vxor 12,2,18 + vxor 13,3,19 + vxor 14,4,20 + vxor 15,5,21 + + bl _aesp8_xts_dec5x + + vperm 7,7,7,6 + vor 17,22,22 + vxor 18,8,23 + vperm 12,12,12,6 + .long 0x7CE02799 + vxor 7,0,18 + vperm 13,13,13,6 + .long 0x7D832799 + vperm 14,14,14,6 + .long 0x7DBA2799 + vperm 15,15,15,6 + .long 0x7DDB2799 + .long 0x7DFC2799 + addi 4,4,0x50 + bne .Lxts_dec6x_steal + b .Lxts_dec6x_done + +.align 4 +.Lxts_dec6x_four: + vxor 7,2,17 + vxor 12,3,18 + vxor 13,4,19 + vxor 14,5,20 + vxor 15,15,15 + + bl _aesp8_xts_dec5x + + vperm 7,7,7,6 + vor 17,21,21 + vor 18,22,22 + vperm 12,12,12,6 + .long 0x7CE02799 + vxor 7,0,22 + vperm 13,13,13,6 + .long 0x7D832799 + vperm 14,14,14,6 + .long 0x7DBA2799 + .long 0x7DDB2799 + addi 4,4,0x40 + bne .Lxts_dec6x_steal + b .Lxts_dec6x_done + +.align 4 +.Lxts_dec6x_three: + vxor 7,3,17 + vxor 12,4,18 + vxor 13,5,19 + vxor 14,14,14 + vxor 15,15,15 + + bl _aesp8_xts_dec5x + + vperm 7,7,7,6 + vor 17,20,20 + vor 18,21,21 + vperm 12,12,12,6 + .long 0x7CE02799 + vxor 7,0,21 + vperm 13,13,13,6 + .long 0x7D832799 + .long 0x7DBA2799 + addi 4,4,0x30 + bne .Lxts_dec6x_steal + b .Lxts_dec6x_done + +.align 4 +.Lxts_dec6x_two: + vxor 7,4,17 + vxor 12,5,18 + vxor 13,13,13 + vxor 14,14,14 + vxor 15,15,15 + + bl _aesp8_xts_dec5x + + vperm 7,7,7,6 + vor 17,19,19 + vor 18,20,20 + vperm 12,12,12,6 + .long 0x7CE02799 + vxor 7,0,20 + .long 0x7D832799 + addi 4,4,0x20 + bne .Lxts_dec6x_steal + b .Lxts_dec6x_done + +.align 4 +.Lxts_dec6x_one: + vxor 7,5,17 + nop +.Loop_xts_dec1x: + .long 0x10E7C548 + lvx 24,26,7 + addi 7,7,0x20 + + .long 0x10E7CD48 + lvx 25,3,7 + bdnz .Loop_xts_dec1x + + subi 0,31,1 + .long 0x10E7C548 + + andi. 0,0,16 + cmpwi 31,0 + .long 0x10E7CD48 + + sub 10,10,0 + .long 0x10E7D548 + + .long 0x7C005699 + .long 0x10E7DD48 + + addi 7,1,64+15 + .long 0x10E7E548 + lvx 24,0,7 + + .long 0x10E7ED48 + lvx 25,3,7 + vxor 17,17,31 + + vperm 0,0,0,6 + .long 0x10E7F548 + + mtctr 9 + .long 0x10E78D49 + + vor 17,18,18 + vor 18,19,19 + vperm 7,7,7,6 + .long 0x7CE02799 + addi 4,4,0x10 + vxor 7,0,19 + bne .Lxts_dec6x_steal + b .Lxts_dec6x_done + +.align 4 +.Lxts_dec6x_zero: + cmpwi 31,0 + beq .Lxts_dec6x_done + + .long 0x7C005699 + vperm 0,0,0,6 + vxor 7,0,18 +.Lxts_dec6x_steal: + .long 0x10E7C548 + lvx 24,26,7 + addi 7,7,0x20 + + .long 0x10E7CD48 + lvx 25,3,7 + bdnz .Lxts_dec6x_steal + + add 10,10,31 + .long 0x10E7C548 + + cmpwi 31,0 + .long 0x10E7CD48 + + .long 0x7C005699 + .long 0x10E7D548 + + lvsr 5,0,31 + .long 0x10E7DD48 + + addi 7,1,64+15 + .long 0x10E7E548 + lvx 24,0,7 + + .long 0x10E7ED48 + lvx 25,3,7 + vxor 18,18,31 + + vperm 0,0,0,6 + .long 0x10E7F548 + + vperm 0,0,0,5 + .long 0x11679549 + + vperm 7,11,11,6 + .long 0x7CE02799 + + + vxor 7,7,7 + vspltisb 12,-1 + vperm 7,7,12,5 + vsel 7,0,11,7 + vxor 7,7,17 + + subi 30,4,1 + mtctr 31 +.Loop_xts_dec6x_steal: + lbzu 0,1(30) + stb 0,16(30) + bdnz .Loop_xts_dec6x_steal + + li 31,0 + mtctr 9 + b .Loop_xts_dec1x + +.align 4 +.Lxts_dec6x_done: + cmpldi 8,0 + beq .Lxts_dec6x_ret + + vxor 8,17,23 + vperm 8,8,8,6 + .long 0x7D004799 + +.Lxts_dec6x_ret: + mtlr 11 + li 10,79 + li 11,95 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + stvx 9,10,1 + addi 10,10,32 + stvx 9,11,1 + addi 11,11,32 + + or 12,12,12 + lvx 20,10,1 + addi 10,10,32 + lvx 21,11,1 + addi 11,11,32 + lvx 22,10,1 + addi 10,10,32 + lvx 23,11,1 + addi 11,11,32 + lvx 24,10,1 + addi 10,10,32 + lvx 25,11,1 + addi 11,11,32 + lvx 26,10,1 + addi 10,10,32 + lvx 27,11,1 + addi 11,11,32 + lvx 28,10,1 + addi 10,10,32 + lvx 29,11,1 + addi 11,11,32 + lvx 30,10,1 + lvx 31,11,1 + ld 26,400(1) + ld 27,408(1) + ld 28,416(1) + ld 29,424(1) + ld 30,432(1) + ld 31,440(1) + addi 1,1,448 + blr +.long 0 +.byte 0,12,0x04,1,0x80,6,6,0 +.long 0 + +.align 5 +_aesp8_xts_dec5x: + .long 0x10E7C548 + .long 0x118CC548 + .long 0x11ADC548 + .long 0x11CEC548 + .long 0x11EFC548 + lvx 24,26,7 + addi 7,7,0x20 + + .long 0x10E7CD48 + .long 0x118CCD48 + .long 0x11ADCD48 + .long 0x11CECD48 + .long 0x11EFCD48 + lvx 25,3,7 + bdnz _aesp8_xts_dec5x + + subi 0,31,1 + .long 0x10E7C548 + .long 0x118CC548 + .long 0x11ADC548 + .long 0x11CEC548 + .long 0x11EFC548 + + andi. 0,0,16 + cmpwi 31,0 + .long 0x10E7CD48 + .long 0x118CCD48 + .long 0x11ADCD48 + .long 0x11CECD48 + .long 0x11EFCD48 + vxor 17,17,31 + + sub 10,10,0 + .long 0x10E7D548 + .long 0x118CD548 + .long 0x11ADD548 + .long 0x11CED548 + .long 0x11EFD548 + vxor 1,18,31 + + .long 0x10E7DD48 + .long 0x7C005699 + .long 0x118CDD48 + .long 0x11ADDD48 + .long 0x11CEDD48 + .long 0x11EFDD48 + vxor 2,19,31 + + addi 7,1,64+15 + .long 0x10E7E548 + .long 0x118CE548 + .long 0x11ADE548 + .long 0x11CEE548 + .long 0x11EFE548 + lvx 24,0,7 + vxor 3,20,31 + + .long 0x10E7ED48 + vperm 0,0,0,6 + .long 0x118CED48 + .long 0x11ADED48 + .long 0x11CEED48 + .long 0x11EFED48 + lvx 25,3,7 + vxor 4,21,31 + + .long 0x10E7F548 + .long 0x118CF548 + .long 0x11ADF548 + .long 0x11CEF548 + .long 0x11EFF548 + + .long 0x10E78D49 + .long 0x118C0D49 + .long 0x11AD1549 + .long 0x11CE1D49 + .long 0x11EF2549 + mtctr 9 + blr +.long 0 +.byte 0,12,0x14,0,0,0,0,0 diff --git a/cbits/asm/aesp8-ppc.pl b/cbits/asm/aesp8-ppc.pl new file mode 100644 --- /dev/null +++ b/cbits/asm/aesp8-ppc.pl @@ -0,0 +1,3801 @@ +#!/usr/bin/env perl +# +# ==================================================================== +# Written by Andy Polyakov for the OpenSSL +# project. The module is, however, dual licensed under OpenSSL and +# CRYPTOGAMS licenses depending on where you obtain it. For further +# details see http://www.openssl.org/~appro/cryptogams/. +# ==================================================================== +# +# This module implements support for AES instructions as per PowerISA +# specification version 2.07, first implemented by POWER8 processor. +# The module is endian-agnostic in sense that it supports both big- +# and little-endian cases. Data alignment in parallelizable modes is +# handled with VSX loads and stores, which implies MSR.VSX flag being +# set. It should also be noted that ISA specification doesn't prohibit +# alignment exceptions for these instructions on page boundaries. +# Initially alignment was handled in pure AltiVec/VMX way [when data +# is aligned programmatically, which in turn guarantees exception- +# free execution], but it turned to hamper performance when vcipher +# instructions are interleaved. It's reckoned that eventual +# misalignment penalties at page boundaries are in average lower +# than additional overhead in pure AltiVec approach. +# +# May 2016 +# +# Add XTS subroutine, 9x on little- and 12x improvement on big-endian +# systems were measured. +# +###################################################################### +# Current large-block performance in cycles per byte processed with +# 128-bit key (less is better). +# +# CBC en-/decrypt CTR XTS +# POWER8[le] 3.96/0.72 0.74 1.1 +# POWER8[be] 3.75/0.65 0.66 1.0 +# POWER9[le] 4.02/0.86 0.84 1.05 +# POWER9[be] 3.99/0.78 0.79 0.97 + + +$flavour = shift; + +if ($flavour =~ /64/) { + $SIZE_T =8; + $LRSAVE =2*$SIZE_T; + $STU ="stdu"; + $POP ="ld"; + $PUSH ="std"; + $UCMP ="cmpld"; + $SHL ="sldi"; +} elsif ($flavour =~ /32/) { + $SIZE_T =4; + $LRSAVE =$SIZE_T; + $STU ="stwu"; + $POP ="lwz"; + $PUSH ="stw"; + $UCMP ="cmplw"; + $SHL ="slwi"; +} else { die "nonsense $flavour"; } + +$LITTLE_ENDIAN = ($flavour=~/le$/) ? $SIZE_T : 0; + +$0 =~ m/(.*[\/\\])[^\/\\]+$/; $dir=$1; +( $xlate="${dir}ppc-xlate.pl" and -f $xlate ) or +( $xlate="${dir}../../perlasm/ppc-xlate.pl" and -f $xlate) or +die "can't locate ppc-xlate.pl"; + +open STDOUT,"| $^X $xlate $flavour ".shift || die "can't call $xlate: $!"; + +$FRAME=8*$SIZE_T; +$prefix="aes_p8"; + +$sp="r1"; +$vrsave="r12"; + +######################################################################### +{{{ # Key setup procedures # +my ($inp,$bits,$out,$ptr,$cnt,$rounds)=map("r$_",(3..8)); +my ($zero,$in0,$in1,$key,$rcon,$mask,$tmp)=map("v$_",(0..6)); +my ($stage,$outperm,$outmask,$outhead,$outtail)=map("v$_",(7..11)); + +$code.=<<___; +.machine "any" + +.text + +.align 7 +rcon: +.long 0x01000000, 0x01000000, 0x01000000, 0x01000000 ?rev +.long 0x1b000000, 0x1b000000, 0x1b000000, 0x1b000000 ?rev +.long 0x0d0e0f0c, 0x0d0e0f0c, 0x0d0e0f0c, 0x0d0e0f0c ?rev +.long 0,0,0,0 ?asis +Lconsts: + mflr r0 + bcl 20,31,\$+4 + mflr $ptr #vvvvv "distance between . and rcon + addi $ptr,$ptr,-0x48 + mtlr r0 + blr + .long 0 + .byte 0,12,0x14,0,0,0,0,0 +.asciz "AES for PowerISA 2.07, CRYPTOGAMS by " + +.globl .${prefix}_set_encrypt_key +.align 5 +.${prefix}_set_encrypt_key: +Lset_encrypt_key: + mflr r11 + $PUSH r11,$LRSAVE($sp) + + li $ptr,-1 + ${UCMP}i $inp,0 + beq- Lenc_key_abort # if ($inp==0) return -1; + ${UCMP}i $out,0 + beq- Lenc_key_abort # if ($out==0) return -1; + li $ptr,-2 + cmpwi $bits,128 + blt- Lenc_key_abort + cmpwi $bits,256 + bgt- Lenc_key_abort + andi. r0,$bits,0x3f + bne- Lenc_key_abort + + lis r0,0xfff0 + mfspr $vrsave,256 + mtspr 256,r0 + + bl Lconsts + mtlr r11 + + neg r9,$inp + lvx $in0,0,$inp + addi $inp,$inp,15 # 15 is not typo + lvsr $key,0,r9 # borrow $key + li r8,0x20 + cmpwi $bits,192 + lvx $in1,0,$inp + le?vspltisb $mask,0x0f # borrow $mask + lvx $rcon,0,$ptr + le?vxor $key,$key,$mask # adjust for byte swap + lvx $mask,r8,$ptr + addi $ptr,$ptr,0x10 + vperm $in0,$in0,$in1,$key # align [and byte swap in LE] + li $cnt,8 + vxor $zero,$zero,$zero + mtctr $cnt + + ?lvsr $outperm,0,$out + vspltisb $outmask,-1 + lvx $outhead,0,$out + ?vperm $outmask,$zero,$outmask,$outperm + + blt Loop128 + addi $inp,$inp,8 + beq L192 + addi $inp,$inp,8 + b L256 + +.align 4 +Loop128: + vperm $key,$in0,$in0,$mask # rotate-n-splat + vsldoi $tmp,$zero,$in0,12 # >>32 + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + vcipherlast $key,$key,$rcon + stvx $stage,0,$out + addi $out,$out,16 + + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vadduwm $rcon,$rcon,$rcon + vxor $in0,$in0,$key + bdnz Loop128 + + lvx $rcon,0,$ptr # last two round keys + + vperm $key,$in0,$in0,$mask # rotate-n-splat + vsldoi $tmp,$zero,$in0,12 # >>32 + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + vcipherlast $key,$key,$rcon + stvx $stage,0,$out + addi $out,$out,16 + + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vadduwm $rcon,$rcon,$rcon + vxor $in0,$in0,$key + + vperm $key,$in0,$in0,$mask # rotate-n-splat + vsldoi $tmp,$zero,$in0,12 # >>32 + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + vcipherlast $key,$key,$rcon + stvx $stage,0,$out + addi $out,$out,16 + + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vxor $in0,$in0,$key + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + stvx $stage,0,$out + + addi $inp,$out,15 # 15 is not typo + addi $out,$out,0x50 + + li $rounds,10 + b Ldone + +.align 4 +L192: + lvx $tmp,0,$inp + li $cnt,4 + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + stvx $stage,0,$out + addi $out,$out,16 + vperm $in1,$in1,$tmp,$key # align [and byte swap in LE] + vspltisb $key,8 # borrow $key + mtctr $cnt + vsububm $mask,$mask,$key # adjust the mask + +Loop192: + vperm $key,$in1,$in1,$mask # roate-n-splat + vsldoi $tmp,$zero,$in0,12 # >>32 + vcipherlast $key,$key,$rcon + + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + + vsldoi $stage,$zero,$in1,8 + vspltw $tmp,$in0,3 + vxor $tmp,$tmp,$in1 + vsldoi $in1,$zero,$in1,12 # >>32 + vadduwm $rcon,$rcon,$rcon + vxor $in1,$in1,$tmp + vxor $in0,$in0,$key + vxor $in1,$in1,$key + vsldoi $stage,$stage,$in0,8 + + vperm $key,$in1,$in1,$mask # rotate-n-splat + vsldoi $tmp,$zero,$in0,12 # >>32 + vperm $outtail,$stage,$stage,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + vcipherlast $key,$key,$rcon + stvx $stage,0,$out + addi $out,$out,16 + + vsldoi $stage,$in0,$in1,8 + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vperm $outtail,$stage,$stage,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + stvx $stage,0,$out + addi $out,$out,16 + + vspltw $tmp,$in0,3 + vxor $tmp,$tmp,$in1 + vsldoi $in1,$zero,$in1,12 # >>32 + vadduwm $rcon,$rcon,$rcon + vxor $in1,$in1,$tmp + vxor $in0,$in0,$key + vxor $in1,$in1,$key + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + stvx $stage,0,$out + addi $inp,$out,15 # 15 is not typo + addi $out,$out,16 + bdnz Loop192 + + li $rounds,12 + addi $out,$out,0x20 + b Ldone + +.align 4 +L256: + lvx $tmp,0,$inp + li $cnt,7 + li $rounds,14 + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + stvx $stage,0,$out + addi $out,$out,16 + vperm $in1,$in1,$tmp,$key # align [and byte swap in LE] + mtctr $cnt + +Loop256: + vperm $key,$in1,$in1,$mask # rotate-n-splat + vsldoi $tmp,$zero,$in0,12 # >>32 + vperm $outtail,$in1,$in1,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + vcipherlast $key,$key,$rcon + stvx $stage,0,$out + addi $out,$out,16 + + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in0,$in0,$tmp + vadduwm $rcon,$rcon,$rcon + vxor $in0,$in0,$key + vperm $outtail,$in0,$in0,$outperm # rotate + vsel $stage,$outhead,$outtail,$outmask + vmr $outhead,$outtail + stvx $stage,0,$out + addi $inp,$out,15 # 15 is not typo + addi $out,$out,16 + bdz Ldone + + vspltw $key,$in0,3 # just splat + vsldoi $tmp,$zero,$in1,12 # >>32 + vsbox $key,$key + + vxor $in1,$in1,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in1,$in1,$tmp + vsldoi $tmp,$zero,$tmp,12 # >>32 + vxor $in1,$in1,$tmp + + vxor $in1,$in1,$key + b Loop256 + +.align 4 +Ldone: + lvx $in1,0,$inp # redundant in aligned case + vsel $in1,$outhead,$in1,$outmask + stvx $in1,0,$inp + li $ptr,0 + mtspr 256,$vrsave + stw $rounds,0($out) + +Lenc_key_abort: + mr r3,$ptr + blr + .long 0 + .byte 0,12,0x14,1,0,0,3,0 + .long 0 +.size .${prefix}_set_encrypt_key,.-.${prefix}_set_encrypt_key + +.globl .${prefix}_set_decrypt_key +.align 5 +.${prefix}_set_decrypt_key: + $STU $sp,-$FRAME($sp) + mflr r10 + $PUSH r10,$FRAME+$LRSAVE($sp) + bl Lset_encrypt_key + mtlr r10 + + cmpwi r3,0 + bne- Ldec_key_abort + + slwi $cnt,$rounds,4 + subi $inp,$out,240 # first round key + srwi $rounds,$rounds,1 + add $out,$inp,$cnt # last round key + mtctr $rounds + +Ldeckey: + lwz r0, 0($inp) + lwz r6, 4($inp) + lwz r7, 8($inp) + lwz r8, 12($inp) + addi $inp,$inp,16 + lwz r9, 0($out) + lwz r10,4($out) + lwz r11,8($out) + lwz r12,12($out) + stw r0, 0($out) + stw r6, 4($out) + stw r7, 8($out) + stw r8, 12($out) + subi $out,$out,16 + stw r9, -16($inp) + stw r10,-12($inp) + stw r11,-8($inp) + stw r12,-4($inp) + bdnz Ldeckey + + xor r3,r3,r3 # return value +Ldec_key_abort: + addi $sp,$sp,$FRAME + blr + .long 0 + .byte 0,12,4,1,0x80,0,3,0 + .long 0 +.size .${prefix}_set_decrypt_key,.-.${prefix}_set_decrypt_key +___ +}}} +######################################################################### +{{{ # Single block en- and decrypt procedures # +sub gen_block () { +my $dir = shift; +my $n = $dir eq "de" ? "n" : ""; +my ($inp,$out,$key,$rounds,$idx)=map("r$_",(3..7)); + +$code.=<<___; +.globl .${prefix}_${dir}crypt +.align 5 +.${prefix}_${dir}crypt: + lwz $rounds,240($key) + lis r0,0xfc00 + mfspr $vrsave,256 + li $idx,15 # 15 is not typo + mtspr 256,r0 + + lvx v0,0,$inp + neg r11,$out + lvx v1,$idx,$inp + lvsl v2,0,$inp # inpperm + le?vspltisb v4,0x0f + ?lvsl v3,0,r11 # outperm + le?vxor v2,v2,v4 + li $idx,16 + vperm v0,v0,v1,v2 # align [and byte swap in LE] + lvx v1,0,$key + ?lvsl v5,0,$key # keyperm + srwi $rounds,$rounds,1 + lvx v2,$idx,$key + addi $idx,$idx,16 + subi $rounds,$rounds,1 + ?vperm v1,v1,v2,v5 # align round key + + vxor v0,v0,v1 + lvx v1,$idx,$key + addi $idx,$idx,16 + mtctr $rounds + +Loop_${dir}c: + ?vperm v2,v2,v1,v5 + v${n}cipher v0,v0,v2 + lvx v2,$idx,$key + addi $idx,$idx,16 + ?vperm v1,v1,v2,v5 + v${n}cipher v0,v0,v1 + lvx v1,$idx,$key + addi $idx,$idx,16 + bdnz Loop_${dir}c + + ?vperm v2,v2,v1,v5 + v${n}cipher v0,v0,v2 + lvx v2,$idx,$key + ?vperm v1,v1,v2,v5 + v${n}cipherlast v0,v0,v1 + + vspltisb v2,-1 + vxor v1,v1,v1 + li $idx,15 # 15 is not typo + ?vperm v2,v1,v2,v3 # outmask + le?vxor v3,v3,v4 + lvx v1,0,$out # outhead + vperm v0,v0,v0,v3 # rotate [and byte swap in LE] + vsel v1,v1,v0,v2 + lvx v4,$idx,$out + stvx v1,0,$out + vsel v0,v0,v4,v2 + stvx v0,$idx,$out + + mtspr 256,$vrsave + blr + .long 0 + .byte 0,12,0x14,0,0,0,3,0 + .long 0 +.size .${prefix}_${dir}crypt,.-.${prefix}_${dir}crypt +___ +} +&gen_block("en"); +&gen_block("de"); +}}} +######################################################################### +{{{ # CBC en- and decrypt procedures # +my ($inp,$out,$len,$key,$ivp,$enc,$rounds,$idx)=map("r$_",(3..10)); +my ($rndkey0,$rndkey1,$inout,$tmp)= map("v$_",(0..3)); +my ($ivec,$inptail,$inpperm,$outhead,$outperm,$outmask,$keyperm)= + map("v$_",(4..10)); +$code.=<<___; +.globl .${prefix}_cbc_encrypt +.align 5 +.${prefix}_cbc_encrypt: + ${UCMP}i $len,16 + bltlr- + + cmpwi $enc,0 # test direction + lis r0,0xffe0 + mfspr $vrsave,256 + mtspr 256,r0 + + li $idx,15 + vxor $rndkey0,$rndkey0,$rndkey0 + le?vspltisb $tmp,0x0f + + lvx $ivec,0,$ivp # load [unaligned] iv + lvsl $inpperm,0,$ivp + lvx $inptail,$idx,$ivp + le?vxor $inpperm,$inpperm,$tmp + vperm $ivec,$ivec,$inptail,$inpperm + + neg r11,$inp + ?lvsl $keyperm,0,$key # prepare for unaligned key + lwz $rounds,240($key) + + lvsr $inpperm,0,r11 # prepare for unaligned load + lvx $inptail,0,$inp + addi $inp,$inp,15 # 15 is not typo + le?vxor $inpperm,$inpperm,$tmp + + ?lvsr $outperm,0,$out # prepare for unaligned store + vspltisb $outmask,-1 + lvx $outhead,0,$out + ?vperm $outmask,$rndkey0,$outmask,$outperm + le?vxor $outperm,$outperm,$tmp + + srwi $rounds,$rounds,1 + li $idx,16 + subi $rounds,$rounds,1 + beq Lcbc_dec + +Lcbc_enc: + vmr $inout,$inptail + lvx $inptail,0,$inp + addi $inp,$inp,16 + mtctr $rounds + subi $len,$len,16 # len-=16 + + lvx $rndkey0,0,$key + vperm $inout,$inout,$inptail,$inpperm + lvx $rndkey1,$idx,$key + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key + addi $idx,$idx,16 + vxor $inout,$inout,$ivec + +Loop_cbc_enc: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipher $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key + addi $idx,$idx,16 + bdnz Loop_cbc_enc + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key + li $idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipherlast $ivec,$inout,$rndkey0 + ${UCMP}i $len,16 + + vperm $tmp,$ivec,$ivec,$outperm + vsel $inout,$outhead,$tmp,$outmask + vmr $outhead,$tmp + stvx $inout,0,$out + addi $out,$out,16 + bge Lcbc_enc + + b Lcbc_done + +.align 4 +Lcbc_dec: + ${UCMP}i $len,128 + bge _aesp8_cbc_decrypt8x + vmr $tmp,$inptail + lvx $inptail,0,$inp + addi $inp,$inp,16 + mtctr $rounds + subi $len,$len,16 # len-=16 + + lvx $rndkey0,0,$key + vperm $tmp,$tmp,$inptail,$inpperm + lvx $rndkey1,$idx,$key + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $inout,$tmp,$rndkey0 + lvx $rndkey0,$idx,$key + addi $idx,$idx,16 + +Loop_cbc_dec: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vncipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vncipher $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key + addi $idx,$idx,16 + bdnz Loop_cbc_dec + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vncipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key + li $idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vncipherlast $inout,$inout,$rndkey0 + ${UCMP}i $len,16 + + vxor $inout,$inout,$ivec + vmr $ivec,$tmp + vperm $tmp,$inout,$inout,$outperm + vsel $inout,$outhead,$tmp,$outmask + vmr $outhead,$tmp + stvx $inout,0,$out + addi $out,$out,16 + bge Lcbc_dec + +Lcbc_done: + addi $out,$out,-1 + lvx $inout,0,$out # redundant in aligned case + vsel $inout,$outhead,$inout,$outmask + stvx $inout,0,$out + + neg $enc,$ivp # write [unaligned] iv + li $idx,15 # 15 is not typo + vxor $rndkey0,$rndkey0,$rndkey0 + vspltisb $outmask,-1 + le?vspltisb $tmp,0x0f + ?lvsl $outperm,0,$enc + ?vperm $outmask,$rndkey0,$outmask,$outperm + le?vxor $outperm,$outperm,$tmp + lvx $outhead,0,$ivp + vperm $ivec,$ivec,$ivec,$outperm + vsel $inout,$outhead,$ivec,$outmask + lvx $inptail,$idx,$ivp + stvx $inout,0,$ivp + vsel $inout,$ivec,$inptail,$outmask + stvx $inout,$idx,$ivp + + mtspr 256,$vrsave + blr + .long 0 + .byte 0,12,0x14,0,0,0,6,0 + .long 0 +___ +######################################################################### +{{ # Optimized CBC decrypt procedure # +my $key_="r11"; +my ($x00,$x10,$x20,$x30,$x40,$x50,$x60,$x70)=map("r$_",(0,8,26..31)); + $x00=0 if ($flavour =~ /osx/); +my ($in0, $in1, $in2, $in3, $in4, $in5, $in6, $in7 )=map("v$_",(0..3,10..13)); +my ($out0,$out1,$out2,$out3,$out4,$out5,$out6,$out7)=map("v$_",(14..21)); +my $rndkey0="v23"; # v24-v25 rotating buffer for first found keys + # v26-v31 last 6 round keys +my ($tmp,$keyperm)=($in3,$in4); # aliases with "caller", redundant assignment + +$code.=<<___; +.align 5 +_aesp8_cbc_decrypt8x: + $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp) + li r10,`$FRAME+8*16+15` + li r11,`$FRAME+8*16+31` + stvx v20,r10,$sp # ABI says so + addi r10,r10,32 + stvx v21,r11,$sp + addi r11,r11,32 + stvx v22,r10,$sp + addi r10,r10,32 + stvx v23,r11,$sp + addi r11,r11,32 + stvx v24,r10,$sp + addi r10,r10,32 + stvx v25,r11,$sp + addi r11,r11,32 + stvx v26,r10,$sp + addi r10,r10,32 + stvx v27,r11,$sp + addi r11,r11,32 + stvx v28,r10,$sp + addi r10,r10,32 + stvx v29,r11,$sp + addi r11,r11,32 + stvx v30,r10,$sp + stvx v31,r11,$sp + li r0,-1 + stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave + li $x10,0x10 + $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp) + li $x20,0x20 + $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp) + li $x30,0x30 + $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp) + li $x40,0x40 + $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp) + li $x50,0x50 + $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp) + li $x60,0x60 + $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp) + li $x70,0x70 + mtspr 256,r0 + + subi $rounds,$rounds,3 # -4 in total + subi $len,$len,128 # bias + + lvx $rndkey0,$x00,$key # load key schedule + lvx v30,$x10,$key + addi $key,$key,0x20 + lvx v31,$x00,$key + ?vperm $rndkey0,$rndkey0,v30,$keyperm + addi $key_,$sp,$FRAME+15 + mtctr $rounds + +Load_cbc_dec_key: + ?vperm v24,v30,v31,$keyperm + lvx v30,$x10,$key + addi $key,$key,0x20 + stvx v24,$x00,$key_ # off-load round[1] + ?vperm v25,v31,v30,$keyperm + lvx v31,$x00,$key + stvx v25,$x10,$key_ # off-load round[2] + addi $key_,$key_,0x20 + bdnz Load_cbc_dec_key + + lvx v26,$x10,$key + ?vperm v24,v30,v31,$keyperm + lvx v27,$x20,$key + stvx v24,$x00,$key_ # off-load round[3] + ?vperm v25,v31,v26,$keyperm + lvx v28,$x30,$key + stvx v25,$x10,$key_ # off-load round[4] + addi $key_,$sp,$FRAME+15 # rewind $key_ + ?vperm v26,v26,v27,$keyperm + lvx v29,$x40,$key + ?vperm v27,v27,v28,$keyperm + lvx v30,$x50,$key + ?vperm v28,v28,v29,$keyperm + lvx v31,$x60,$key + ?vperm v29,v29,v30,$keyperm + lvx $out0,$x70,$key # borrow $out0 + ?vperm v30,v30,v31,$keyperm + lvx v24,$x00,$key_ # pre-load round[1] + ?vperm v31,v31,$out0,$keyperm + lvx v25,$x10,$key_ # pre-load round[2] + + #lvx $inptail,0,$inp # "caller" already did this + #addi $inp,$inp,15 # 15 is not typo + subi $inp,$inp,15 # undo "caller" + + le?li $idx,8 + lvx_u $in0,$x00,$inp # load first 8 "words" + le?lvsl $inpperm,0,$idx + le?vspltisb $tmp,0x0f + lvx_u $in1,$x10,$inp + le?vxor $inpperm,$inpperm,$tmp # transform for lvx_u/stvx_u + lvx_u $in2,$x20,$inp + le?vperm $in0,$in0,$in0,$inpperm + lvx_u $in3,$x30,$inp + le?vperm $in1,$in1,$in1,$inpperm + lvx_u $in4,$x40,$inp + le?vperm $in2,$in2,$in2,$inpperm + vxor $out0,$in0,$rndkey0 + lvx_u $in5,$x50,$inp + le?vperm $in3,$in3,$in3,$inpperm + vxor $out1,$in1,$rndkey0 + lvx_u $in6,$x60,$inp + le?vperm $in4,$in4,$in4,$inpperm + vxor $out2,$in2,$rndkey0 + lvx_u $in7,$x70,$inp + addi $inp,$inp,0x80 + le?vperm $in5,$in5,$in5,$inpperm + vxor $out3,$in3,$rndkey0 + le?vperm $in6,$in6,$in6,$inpperm + vxor $out4,$in4,$rndkey0 + le?vperm $in7,$in7,$in7,$inpperm + vxor $out5,$in5,$rndkey0 + vxor $out6,$in6,$rndkey0 + vxor $out7,$in7,$rndkey0 + + mtctr $rounds + b Loop_cbc_dec8x +.align 5 +Loop_cbc_dec8x: + vncipher $out0,$out0,v24 + vncipher $out1,$out1,v24 + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vncipher $out4,$out4,v24 + vncipher $out5,$out5,v24 + vncipher $out6,$out6,v24 + vncipher $out7,$out7,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vncipher $out0,$out0,v25 + vncipher $out1,$out1,v25 + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vncipher $out4,$out4,v25 + vncipher $out5,$out5,v25 + vncipher $out6,$out6,v25 + vncipher $out7,$out7,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Loop_cbc_dec8x + + subic $len,$len,128 # $len-=128 + vncipher $out0,$out0,v24 + vncipher $out1,$out1,v24 + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vncipher $out4,$out4,v24 + vncipher $out5,$out5,v24 + vncipher $out6,$out6,v24 + vncipher $out7,$out7,v24 + + subfe. r0,r0,r0 # borrow?-1:0 + vncipher $out0,$out0,v25 + vncipher $out1,$out1,v25 + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vncipher $out4,$out4,v25 + vncipher $out5,$out5,v25 + vncipher $out6,$out6,v25 + vncipher $out7,$out7,v25 + + and r0,r0,$len + vncipher $out0,$out0,v26 + vncipher $out1,$out1,v26 + vncipher $out2,$out2,v26 + vncipher $out3,$out3,v26 + vncipher $out4,$out4,v26 + vncipher $out5,$out5,v26 + vncipher $out6,$out6,v26 + vncipher $out7,$out7,v26 + + add $inp,$inp,r0 # $inp is adjusted in such + # way that at exit from the + # loop inX-in7 are loaded + # with last "words" + vncipher $out0,$out0,v27 + vncipher $out1,$out1,v27 + vncipher $out2,$out2,v27 + vncipher $out3,$out3,v27 + vncipher $out4,$out4,v27 + vncipher $out5,$out5,v27 + vncipher $out6,$out6,v27 + vncipher $out7,$out7,v27 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vncipher $out0,$out0,v28 + vncipher $out1,$out1,v28 + vncipher $out2,$out2,v28 + vncipher $out3,$out3,v28 + vncipher $out4,$out4,v28 + vncipher $out5,$out5,v28 + vncipher $out6,$out6,v28 + vncipher $out7,$out7,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + + vncipher $out0,$out0,v29 + vncipher $out1,$out1,v29 + vncipher $out2,$out2,v29 + vncipher $out3,$out3,v29 + vncipher $out4,$out4,v29 + vncipher $out5,$out5,v29 + vncipher $out6,$out6,v29 + vncipher $out7,$out7,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + + vncipher $out0,$out0,v30 + vxor $ivec,$ivec,v31 # xor with last round key + vncipher $out1,$out1,v30 + vxor $in0,$in0,v31 + vncipher $out2,$out2,v30 + vxor $in1,$in1,v31 + vncipher $out3,$out3,v30 + vxor $in2,$in2,v31 + vncipher $out4,$out4,v30 + vxor $in3,$in3,v31 + vncipher $out5,$out5,v30 + vxor $in4,$in4,v31 + vncipher $out6,$out6,v30 + vxor $in5,$in5,v31 + vncipher $out7,$out7,v30 + vxor $in6,$in6,v31 + + vncipherlast $out0,$out0,$ivec + vncipherlast $out1,$out1,$in0 + lvx_u $in0,$x00,$inp # load next input block + vncipherlast $out2,$out2,$in1 + lvx_u $in1,$x10,$inp + vncipherlast $out3,$out3,$in2 + le?vperm $in0,$in0,$in0,$inpperm + lvx_u $in2,$x20,$inp + vncipherlast $out4,$out4,$in3 + le?vperm $in1,$in1,$in1,$inpperm + lvx_u $in3,$x30,$inp + vncipherlast $out5,$out5,$in4 + le?vperm $in2,$in2,$in2,$inpperm + lvx_u $in4,$x40,$inp + vncipherlast $out6,$out6,$in5 + le?vperm $in3,$in3,$in3,$inpperm + lvx_u $in5,$x50,$inp + vncipherlast $out7,$out7,$in6 + le?vperm $in4,$in4,$in4,$inpperm + lvx_u $in6,$x60,$inp + vmr $ivec,$in7 + le?vperm $in5,$in5,$in5,$inpperm + lvx_u $in7,$x70,$inp + addi $inp,$inp,0x80 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + le?vperm $in6,$in6,$in6,$inpperm + vxor $out0,$in0,$rndkey0 + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x10,$out + le?vperm $in7,$in7,$in7,$inpperm + vxor $out1,$in1,$rndkey0 + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x20,$out + vxor $out2,$in2,$rndkey0 + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x30,$out + vxor $out3,$in3,$rndkey0 + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x40,$out + vxor $out4,$in4,$rndkey0 + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x50,$out + vxor $out5,$in5,$rndkey0 + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x60,$out + vxor $out6,$in6,$rndkey0 + stvx_u $out7,$x70,$out + addi $out,$out,0x80 + vxor $out7,$in7,$rndkey0 + + mtctr $rounds + beq Loop_cbc_dec8x # did $len-=128 borrow? + + addic. $len,$len,128 + beq Lcbc_dec8x_done + nop + nop + +Loop_cbc_dec8x_tail: # up to 7 "words" tail... + vncipher $out1,$out1,v24 + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vncipher $out4,$out4,v24 + vncipher $out5,$out5,v24 + vncipher $out6,$out6,v24 + vncipher $out7,$out7,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vncipher $out1,$out1,v25 + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vncipher $out4,$out4,v25 + vncipher $out5,$out5,v25 + vncipher $out6,$out6,v25 + vncipher $out7,$out7,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Loop_cbc_dec8x_tail + + vncipher $out1,$out1,v24 + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vncipher $out4,$out4,v24 + vncipher $out5,$out5,v24 + vncipher $out6,$out6,v24 + vncipher $out7,$out7,v24 + + vncipher $out1,$out1,v25 + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vncipher $out4,$out4,v25 + vncipher $out5,$out5,v25 + vncipher $out6,$out6,v25 + vncipher $out7,$out7,v25 + + vncipher $out1,$out1,v26 + vncipher $out2,$out2,v26 + vncipher $out3,$out3,v26 + vncipher $out4,$out4,v26 + vncipher $out5,$out5,v26 + vncipher $out6,$out6,v26 + vncipher $out7,$out7,v26 + + vncipher $out1,$out1,v27 + vncipher $out2,$out2,v27 + vncipher $out3,$out3,v27 + vncipher $out4,$out4,v27 + vncipher $out5,$out5,v27 + vncipher $out6,$out6,v27 + vncipher $out7,$out7,v27 + + vncipher $out1,$out1,v28 + vncipher $out2,$out2,v28 + vncipher $out3,$out3,v28 + vncipher $out4,$out4,v28 + vncipher $out5,$out5,v28 + vncipher $out6,$out6,v28 + vncipher $out7,$out7,v28 + + vncipher $out1,$out1,v29 + vncipher $out2,$out2,v29 + vncipher $out3,$out3,v29 + vncipher $out4,$out4,v29 + vncipher $out5,$out5,v29 + vncipher $out6,$out6,v29 + vncipher $out7,$out7,v29 + + vncipher $out1,$out1,v30 + vxor $ivec,$ivec,v31 # last round key + vncipher $out2,$out2,v30 + vxor $in1,$in1,v31 + vncipher $out3,$out3,v30 + vxor $in2,$in2,v31 + vncipher $out4,$out4,v30 + vxor $in3,$in3,v31 + vncipher $out5,$out5,v30 + vxor $in4,$in4,v31 + vncipher $out6,$out6,v30 + vxor $in5,$in5,v31 + vncipher $out7,$out7,v30 + vxor $in6,$in6,v31 + + cmplwi $len,32 # switch($len) + blt Lcbc_dec8x_one + nop + beq Lcbc_dec8x_two + cmplwi $len,64 + blt Lcbc_dec8x_three + nop + beq Lcbc_dec8x_four + cmplwi $len,96 + blt Lcbc_dec8x_five + nop + beq Lcbc_dec8x_six + +Lcbc_dec8x_seven: + vncipherlast $out1,$out1,$ivec + vncipherlast $out2,$out2,$in1 + vncipherlast $out3,$out3,$in2 + vncipherlast $out4,$out4,$in3 + vncipherlast $out5,$out5,$in4 + vncipherlast $out6,$out6,$in5 + vncipherlast $out7,$out7,$in6 + vmr $ivec,$in7 + + le?vperm $out1,$out1,$out1,$inpperm + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x00,$out + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x10,$out + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x20,$out + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x30,$out + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x40,$out + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x50,$out + stvx_u $out7,$x60,$out + addi $out,$out,0x70 + b Lcbc_dec8x_done + +.align 5 +Lcbc_dec8x_six: + vncipherlast $out2,$out2,$ivec + vncipherlast $out3,$out3,$in2 + vncipherlast $out4,$out4,$in3 + vncipherlast $out5,$out5,$in4 + vncipherlast $out6,$out6,$in5 + vncipherlast $out7,$out7,$in6 + vmr $ivec,$in7 + + le?vperm $out2,$out2,$out2,$inpperm + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x00,$out + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x10,$out + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x20,$out + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x30,$out + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x40,$out + stvx_u $out7,$x50,$out + addi $out,$out,0x60 + b Lcbc_dec8x_done + +.align 5 +Lcbc_dec8x_five: + vncipherlast $out3,$out3,$ivec + vncipherlast $out4,$out4,$in3 + vncipherlast $out5,$out5,$in4 + vncipherlast $out6,$out6,$in5 + vncipherlast $out7,$out7,$in6 + vmr $ivec,$in7 + + le?vperm $out3,$out3,$out3,$inpperm + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x00,$out + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x10,$out + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x20,$out + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x30,$out + stvx_u $out7,$x40,$out + addi $out,$out,0x50 + b Lcbc_dec8x_done + +.align 5 +Lcbc_dec8x_four: + vncipherlast $out4,$out4,$ivec + vncipherlast $out5,$out5,$in4 + vncipherlast $out6,$out6,$in5 + vncipherlast $out7,$out7,$in6 + vmr $ivec,$in7 + + le?vperm $out4,$out4,$out4,$inpperm + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x00,$out + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x10,$out + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x20,$out + stvx_u $out7,$x30,$out + addi $out,$out,0x40 + b Lcbc_dec8x_done + +.align 5 +Lcbc_dec8x_three: + vncipherlast $out5,$out5,$ivec + vncipherlast $out6,$out6,$in5 + vncipherlast $out7,$out7,$in6 + vmr $ivec,$in7 + + le?vperm $out5,$out5,$out5,$inpperm + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x00,$out + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x10,$out + stvx_u $out7,$x20,$out + addi $out,$out,0x30 + b Lcbc_dec8x_done + +.align 5 +Lcbc_dec8x_two: + vncipherlast $out6,$out6,$ivec + vncipherlast $out7,$out7,$in6 + vmr $ivec,$in7 + + le?vperm $out6,$out6,$out6,$inpperm + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x00,$out + stvx_u $out7,$x10,$out + addi $out,$out,0x20 + b Lcbc_dec8x_done + +.align 5 +Lcbc_dec8x_one: + vncipherlast $out7,$out7,$ivec + vmr $ivec,$in7 + + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out7,0,$out + addi $out,$out,0x10 + +Lcbc_dec8x_done: + le?vperm $ivec,$ivec,$ivec,$inpperm + stvx_u $ivec,0,$ivp # write [unaligned] iv + + li r10,`$FRAME+15` + li r11,`$FRAME+31` + stvx $inpperm,r10,$sp # wipe copies of round keys + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + stvx $inpperm,r10,$sp + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + stvx $inpperm,r10,$sp + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + stvx $inpperm,r10,$sp + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + + mtspr 256,$vrsave + lvx v20,r10,$sp # ABI says so + addi r10,r10,32 + lvx v21,r11,$sp + addi r11,r11,32 + lvx v22,r10,$sp + addi r10,r10,32 + lvx v23,r11,$sp + addi r11,r11,32 + lvx v24,r10,$sp + addi r10,r10,32 + lvx v25,r11,$sp + addi r11,r11,32 + lvx v26,r10,$sp + addi r10,r10,32 + lvx v27,r11,$sp + addi r11,r11,32 + lvx v28,r10,$sp + addi r10,r10,32 + lvx v29,r11,$sp + addi r11,r11,32 + lvx v30,r10,$sp + lvx v31,r11,$sp + $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp) + $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp) + $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp) + $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp) + $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp) + $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp) + addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T` + blr + .long 0 + .byte 0,12,0x04,0,0x80,6,6,0 + .long 0 +.size .${prefix}_cbc_encrypt,.-.${prefix}_cbc_encrypt +___ +}} }}} + +######################################################################### +{{{ # CTR procedure[s] # +my ($inp,$out,$len,$key,$ivp,$x10,$rounds,$idx)=map("r$_",(3..10)); +my ($rndkey0,$rndkey1,$inout,$tmp)= map("v$_",(0..3)); +my ($ivec,$inptail,$inpperm,$outhead,$outperm,$outmask,$keyperm,$one)= + map("v$_",(4..11)); +my $dat=$tmp; + +$code.=<<___; +.globl .${prefix}_ctr32_encrypt_blocks +.align 5 +.${prefix}_ctr32_encrypt_blocks: + ${UCMP}i $len,1 + bltlr- + + lis r0,0xfff0 + mfspr $vrsave,256 + mtspr 256,r0 + + li $idx,15 + vxor $rndkey0,$rndkey0,$rndkey0 + le?vspltisb $tmp,0x0f + + lvx $ivec,0,$ivp # load [unaligned] iv + lvsl $inpperm,0,$ivp + lvx $inptail,$idx,$ivp + vspltisb $one,1 + le?vxor $inpperm,$inpperm,$tmp + vperm $ivec,$ivec,$inptail,$inpperm + vsldoi $one,$rndkey0,$one,1 + + neg r11,$inp + ?lvsl $keyperm,0,$key # prepare for unaligned key + lwz $rounds,240($key) + + lvsr $inpperm,0,r11 # prepare for unaligned load + lvx $inptail,0,$inp + addi $inp,$inp,15 # 15 is not typo + le?vxor $inpperm,$inpperm,$tmp + + srwi $rounds,$rounds,1 + li $idx,16 + subi $rounds,$rounds,1 + + ${UCMP}i $len,8 + bge _aesp8_ctr32_encrypt8x + + ?lvsr $outperm,0,$out # prepare for unaligned store + vspltisb $outmask,-1 + lvx $outhead,0,$out + ?vperm $outmask,$rndkey0,$outmask,$outperm + le?vxor $outperm,$outperm,$tmp + + lvx $rndkey0,0,$key + mtctr $rounds + lvx $rndkey1,$idx,$key + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $inout,$ivec,$rndkey0 + lvx $rndkey0,$idx,$key + addi $idx,$idx,16 + b Loop_ctr32_enc + +.align 5 +Loop_ctr32_enc: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipher $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key + addi $idx,$idx,16 + bdnz Loop_ctr32_enc + + vadduwm $ivec,$ivec,$one + vmr $dat,$inptail + lvx $inptail,0,$inp + addi $inp,$inp,16 + subic. $len,$len,1 # blocks-- + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key + vperm $dat,$dat,$inptail,$inpperm + li $idx,16 + ?vperm $rndkey1,$rndkey0,$rndkey1,$keyperm + lvx $rndkey0,0,$key + vxor $dat,$dat,$rndkey1 # last round key + vcipherlast $inout,$inout,$dat + + lvx $rndkey1,$idx,$key + addi $idx,$idx,16 + vperm $inout,$inout,$inout,$outperm + vsel $dat,$outhead,$inout,$outmask + mtctr $rounds + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vmr $outhead,$inout + vxor $inout,$ivec,$rndkey0 + lvx $rndkey0,$idx,$key + addi $idx,$idx,16 + stvx $dat,0,$out + addi $out,$out,16 + bne Loop_ctr32_enc + + addi $out,$out,-1 + lvx $inout,0,$out # redundant in aligned case + vsel $inout,$outhead,$inout,$outmask + stvx $inout,0,$out + + mtspr 256,$vrsave + blr + .long 0 + .byte 0,12,0x14,0,0,0,6,0 + .long 0 +___ +######################################################################### +{{ # Optimized CTR procedure # +my $key_="r11"; +my ($x00,$x10,$x20,$x30,$x40,$x50,$x60,$x70)=map("r$_",(0,8,26..31)); + $x00=0 if ($flavour =~ /osx/); +my ($in0, $in1, $in2, $in3, $in4, $in5, $in6, $in7 )=map("v$_",(0..3,10,12..14)); +my ($out0,$out1,$out2,$out3,$out4,$out5,$out6,$out7)=map("v$_",(15..22)); +my $rndkey0="v23"; # v24-v25 rotating buffer for first found keys + # v26-v31 last 6 round keys +my ($tmp,$keyperm)=($in3,$in4); # aliases with "caller", redundant assignment +my ($two,$three,$four)=($outhead,$outperm,$outmask); + +$code.=<<___; +.align 5 +_aesp8_ctr32_encrypt8x: + $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp) + li r10,`$FRAME+8*16+15` + li r11,`$FRAME+8*16+31` + stvx v20,r10,$sp # ABI says so + addi r10,r10,32 + stvx v21,r11,$sp + addi r11,r11,32 + stvx v22,r10,$sp + addi r10,r10,32 + stvx v23,r11,$sp + addi r11,r11,32 + stvx v24,r10,$sp + addi r10,r10,32 + stvx v25,r11,$sp + addi r11,r11,32 + stvx v26,r10,$sp + addi r10,r10,32 + stvx v27,r11,$sp + addi r11,r11,32 + stvx v28,r10,$sp + addi r10,r10,32 + stvx v29,r11,$sp + addi r11,r11,32 + stvx v30,r10,$sp + stvx v31,r11,$sp + li r0,-1 + stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave + li $x10,0x10 + $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp) + li $x20,0x20 + $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp) + li $x30,0x30 + $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp) + li $x40,0x40 + $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp) + li $x50,0x50 + $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp) + li $x60,0x60 + $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp) + li $x70,0x70 + mtspr 256,r0 + + subi $rounds,$rounds,3 # -4 in total + + lvx $rndkey0,$x00,$key # load key schedule + lvx v30,$x10,$key + addi $key,$key,0x20 + lvx v31,$x00,$key + ?vperm $rndkey0,$rndkey0,v30,$keyperm + addi $key_,$sp,$FRAME+15 + mtctr $rounds + +Load_ctr32_enc_key: + ?vperm v24,v30,v31,$keyperm + lvx v30,$x10,$key + addi $key,$key,0x20 + stvx v24,$x00,$key_ # off-load round[1] + ?vperm v25,v31,v30,$keyperm + lvx v31,$x00,$key + stvx v25,$x10,$key_ # off-load round[2] + addi $key_,$key_,0x20 + bdnz Load_ctr32_enc_key + + lvx v26,$x10,$key + ?vperm v24,v30,v31,$keyperm + lvx v27,$x20,$key + stvx v24,$x00,$key_ # off-load round[3] + ?vperm v25,v31,v26,$keyperm + lvx v28,$x30,$key + stvx v25,$x10,$key_ # off-load round[4] + addi $key_,$sp,$FRAME+15 # rewind $key_ + ?vperm v26,v26,v27,$keyperm + lvx v29,$x40,$key + ?vperm v27,v27,v28,$keyperm + lvx v30,$x50,$key + ?vperm v28,v28,v29,$keyperm + lvx v31,$x60,$key + ?vperm v29,v29,v30,$keyperm + lvx $out0,$x70,$key # borrow $out0 + ?vperm v30,v30,v31,$keyperm + lvx v24,$x00,$key_ # pre-load round[1] + ?vperm v31,v31,$out0,$keyperm + lvx v25,$x10,$key_ # pre-load round[2] + + vadduwm $two,$one,$one + subi $inp,$inp,15 # undo "caller" + $SHL $len,$len,4 + + vadduwm $out1,$ivec,$one # counter values ... + vadduwm $out2,$ivec,$two + vxor $out0,$ivec,$rndkey0 # ... xored with rndkey[0] + le?li $idx,8 + vadduwm $out3,$out1,$two + vxor $out1,$out1,$rndkey0 + le?lvsl $inpperm,0,$idx + vadduwm $out4,$out2,$two + vxor $out2,$out2,$rndkey0 + le?vspltisb $tmp,0x0f + vadduwm $out5,$out3,$two + vxor $out3,$out3,$rndkey0 + le?vxor $inpperm,$inpperm,$tmp # transform for lvx_u/stvx_u + vadduwm $out6,$out4,$two + vxor $out4,$out4,$rndkey0 + vadduwm $out7,$out5,$two + vxor $out5,$out5,$rndkey0 + vadduwm $ivec,$out6,$two # next counter value + vxor $out6,$out6,$rndkey0 + vxor $out7,$out7,$rndkey0 + + mtctr $rounds + b Loop_ctr32_enc8x +.align 5 +Loop_ctr32_enc8x: + vcipher $out0,$out0,v24 + vcipher $out1,$out1,v24 + vcipher $out2,$out2,v24 + vcipher $out3,$out3,v24 + vcipher $out4,$out4,v24 + vcipher $out5,$out5,v24 + vcipher $out6,$out6,v24 + vcipher $out7,$out7,v24 +Loop_ctr32_enc8x_middle: + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vcipher $out0,$out0,v25 + vcipher $out1,$out1,v25 + vcipher $out2,$out2,v25 + vcipher $out3,$out3,v25 + vcipher $out4,$out4,v25 + vcipher $out5,$out5,v25 + vcipher $out6,$out6,v25 + vcipher $out7,$out7,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Loop_ctr32_enc8x + + subic r11,$len,256 # $len-256, borrow $key_ + vcipher $out0,$out0,v24 + vcipher $out1,$out1,v24 + vcipher $out2,$out2,v24 + vcipher $out3,$out3,v24 + vcipher $out4,$out4,v24 + vcipher $out5,$out5,v24 + vcipher $out6,$out6,v24 + vcipher $out7,$out7,v24 + + subfe r0,r0,r0 # borrow?-1:0 + vcipher $out0,$out0,v25 + vcipher $out1,$out1,v25 + vcipher $out2,$out2,v25 + vcipher $out3,$out3,v25 + vcipher $out4,$out4,v25 + vcipher $out5,$out5,v25 + vcipher $out6,$out6,v25 + vcipher $out7,$out7,v25 + + and r0,r0,r11 + addi $key_,$sp,$FRAME+15 # rewind $key_ + vcipher $out0,$out0,v26 + vcipher $out1,$out1,v26 + vcipher $out2,$out2,v26 + vcipher $out3,$out3,v26 + vcipher $out4,$out4,v26 + vcipher $out5,$out5,v26 + vcipher $out6,$out6,v26 + vcipher $out7,$out7,v26 + lvx v24,$x00,$key_ # re-pre-load round[1] + + subic $len,$len,129 # $len-=129 + vcipher $out0,$out0,v27 + addi $len,$len,1 # $len-=128 really + vcipher $out1,$out1,v27 + vcipher $out2,$out2,v27 + vcipher $out3,$out3,v27 + vcipher $out4,$out4,v27 + vcipher $out5,$out5,v27 + vcipher $out6,$out6,v27 + vcipher $out7,$out7,v27 + lvx v25,$x10,$key_ # re-pre-load round[2] + + vcipher $out0,$out0,v28 + lvx_u $in0,$x00,$inp # load input + vcipher $out1,$out1,v28 + lvx_u $in1,$x10,$inp + vcipher $out2,$out2,v28 + lvx_u $in2,$x20,$inp + vcipher $out3,$out3,v28 + lvx_u $in3,$x30,$inp + vcipher $out4,$out4,v28 + lvx_u $in4,$x40,$inp + vcipher $out5,$out5,v28 + lvx_u $in5,$x50,$inp + vcipher $out6,$out6,v28 + lvx_u $in6,$x60,$inp + vcipher $out7,$out7,v28 + lvx_u $in7,$x70,$inp + addi $inp,$inp,0x80 + + vcipher $out0,$out0,v29 + le?vperm $in0,$in0,$in0,$inpperm + vcipher $out1,$out1,v29 + le?vperm $in1,$in1,$in1,$inpperm + vcipher $out2,$out2,v29 + le?vperm $in2,$in2,$in2,$inpperm + vcipher $out3,$out3,v29 + le?vperm $in3,$in3,$in3,$inpperm + vcipher $out4,$out4,v29 + le?vperm $in4,$in4,$in4,$inpperm + vcipher $out5,$out5,v29 + le?vperm $in5,$in5,$in5,$inpperm + vcipher $out6,$out6,v29 + le?vperm $in6,$in6,$in6,$inpperm + vcipher $out7,$out7,v29 + le?vperm $in7,$in7,$in7,$inpperm + + add $inp,$inp,r0 # $inp is adjusted in such + # way that at exit from the + # loop inX-in7 are loaded + # with last "words" + subfe. r0,r0,r0 # borrow?-1:0 + vcipher $out0,$out0,v30 + vxor $in0,$in0,v31 # xor with last round key + vcipher $out1,$out1,v30 + vxor $in1,$in1,v31 + vcipher $out2,$out2,v30 + vxor $in2,$in2,v31 + vcipher $out3,$out3,v30 + vxor $in3,$in3,v31 + vcipher $out4,$out4,v30 + vxor $in4,$in4,v31 + vcipher $out5,$out5,v30 + vxor $in5,$in5,v31 + vcipher $out6,$out6,v30 + vxor $in6,$in6,v31 + vcipher $out7,$out7,v30 + vxor $in7,$in7,v31 + + bne Lctr32_enc8x_break # did $len-129 borrow? + + vcipherlast $in0,$out0,$in0 + vcipherlast $in1,$out1,$in1 + vadduwm $out1,$ivec,$one # counter values ... + vcipherlast $in2,$out2,$in2 + vadduwm $out2,$ivec,$two + vxor $out0,$ivec,$rndkey0 # ... xored with rndkey[0] + vcipherlast $in3,$out3,$in3 + vadduwm $out3,$out1,$two + vxor $out1,$out1,$rndkey0 + vcipherlast $in4,$out4,$in4 + vadduwm $out4,$out2,$two + vxor $out2,$out2,$rndkey0 + vcipherlast $in5,$out5,$in5 + vadduwm $out5,$out3,$two + vxor $out3,$out3,$rndkey0 + vcipherlast $in6,$out6,$in6 + vadduwm $out6,$out4,$two + vxor $out4,$out4,$rndkey0 + vcipherlast $in7,$out7,$in7 + vadduwm $out7,$out5,$two + vxor $out5,$out5,$rndkey0 + le?vperm $in0,$in0,$in0,$inpperm + vadduwm $ivec,$out6,$two # next counter value + vxor $out6,$out6,$rndkey0 + le?vperm $in1,$in1,$in1,$inpperm + vxor $out7,$out7,$rndkey0 + mtctr $rounds + + vcipher $out0,$out0,v24 + stvx_u $in0,$x00,$out + le?vperm $in2,$in2,$in2,$inpperm + vcipher $out1,$out1,v24 + stvx_u $in1,$x10,$out + le?vperm $in3,$in3,$in3,$inpperm + vcipher $out2,$out2,v24 + stvx_u $in2,$x20,$out + le?vperm $in4,$in4,$in4,$inpperm + vcipher $out3,$out3,v24 + stvx_u $in3,$x30,$out + le?vperm $in5,$in5,$in5,$inpperm + vcipher $out4,$out4,v24 + stvx_u $in4,$x40,$out + le?vperm $in6,$in6,$in6,$inpperm + vcipher $out5,$out5,v24 + stvx_u $in5,$x50,$out + le?vperm $in7,$in7,$in7,$inpperm + vcipher $out6,$out6,v24 + stvx_u $in6,$x60,$out + vcipher $out7,$out7,v24 + stvx_u $in7,$x70,$out + addi $out,$out,0x80 + + b Loop_ctr32_enc8x_middle + +.align 5 +Lctr32_enc8x_break: + cmpwi $len,-0x60 + blt Lctr32_enc8x_one + nop + beq Lctr32_enc8x_two + cmpwi $len,-0x40 + blt Lctr32_enc8x_three + nop + beq Lctr32_enc8x_four + cmpwi $len,-0x20 + blt Lctr32_enc8x_five + nop + beq Lctr32_enc8x_six + cmpwi $len,0x00 + blt Lctr32_enc8x_seven + +Lctr32_enc8x_eight: + vcipherlast $out0,$out0,$in0 + vcipherlast $out1,$out1,$in1 + vcipherlast $out2,$out2,$in2 + vcipherlast $out3,$out3,$in3 + vcipherlast $out4,$out4,$in4 + vcipherlast $out5,$out5,$in5 + vcipherlast $out6,$out6,$in6 + vcipherlast $out7,$out7,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x20,$out + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x30,$out + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x40,$out + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x50,$out + le?vperm $out7,$out7,$out7,$inpperm + stvx_u $out6,$x60,$out + stvx_u $out7,$x70,$out + addi $out,$out,0x80 + b Lctr32_enc8x_done + +.align 5 +Lctr32_enc8x_seven: + vcipherlast $out0,$out0,$in1 + vcipherlast $out1,$out1,$in2 + vcipherlast $out2,$out2,$in3 + vcipherlast $out3,$out3,$in4 + vcipherlast $out4,$out4,$in5 + vcipherlast $out5,$out5,$in6 + vcipherlast $out6,$out6,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x20,$out + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x30,$out + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x40,$out + le?vperm $out6,$out6,$out6,$inpperm + stvx_u $out5,$x50,$out + stvx_u $out6,$x60,$out + addi $out,$out,0x70 + b Lctr32_enc8x_done + +.align 5 +Lctr32_enc8x_six: + vcipherlast $out0,$out0,$in2 + vcipherlast $out1,$out1,$in3 + vcipherlast $out2,$out2,$in4 + vcipherlast $out3,$out3,$in5 + vcipherlast $out4,$out4,$in6 + vcipherlast $out5,$out5,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x20,$out + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x30,$out + le?vperm $out5,$out5,$out5,$inpperm + stvx_u $out4,$x40,$out + stvx_u $out5,$x50,$out + addi $out,$out,0x60 + b Lctr32_enc8x_done + +.align 5 +Lctr32_enc8x_five: + vcipherlast $out0,$out0,$in3 + vcipherlast $out1,$out1,$in4 + vcipherlast $out2,$out2,$in5 + vcipherlast $out3,$out3,$in6 + vcipherlast $out4,$out4,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x20,$out + le?vperm $out4,$out4,$out4,$inpperm + stvx_u $out3,$x30,$out + stvx_u $out4,$x40,$out + addi $out,$out,0x50 + b Lctr32_enc8x_done + +.align 5 +Lctr32_enc8x_four: + vcipherlast $out0,$out0,$in4 + vcipherlast $out1,$out1,$in5 + vcipherlast $out2,$out2,$in6 + vcipherlast $out3,$out3,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$inpperm + stvx_u $out2,$x20,$out + stvx_u $out3,$x30,$out + addi $out,$out,0x40 + b Lctr32_enc8x_done + +.align 5 +Lctr32_enc8x_three: + vcipherlast $out0,$out0,$in5 + vcipherlast $out1,$out1,$in6 + vcipherlast $out2,$out2,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + le?vperm $out2,$out2,$out2,$inpperm + stvx_u $out1,$x10,$out + stvx_u $out2,$x20,$out + addi $out,$out,0x30 + b Lctr32_enc8x_done + +.align 5 +Lctr32_enc8x_two: + vcipherlast $out0,$out0,$in6 + vcipherlast $out1,$out1,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + le?vperm $out1,$out1,$out1,$inpperm + stvx_u $out0,$x00,$out + stvx_u $out1,$x10,$out + addi $out,$out,0x20 + b Lctr32_enc8x_done + +.align 5 +Lctr32_enc8x_one: + vcipherlast $out0,$out0,$in7 + + le?vperm $out0,$out0,$out0,$inpperm + stvx_u $out0,0,$out + addi $out,$out,0x10 + +Lctr32_enc8x_done: + li r10,`$FRAME+15` + li r11,`$FRAME+31` + stvx $inpperm,r10,$sp # wipe copies of round keys + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + stvx $inpperm,r10,$sp + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + stvx $inpperm,r10,$sp + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + stvx $inpperm,r10,$sp + addi r10,r10,32 + stvx $inpperm,r11,$sp + addi r11,r11,32 + + mtspr 256,$vrsave + lvx v20,r10,$sp # ABI says so + addi r10,r10,32 + lvx v21,r11,$sp + addi r11,r11,32 + lvx v22,r10,$sp + addi r10,r10,32 + lvx v23,r11,$sp + addi r11,r11,32 + lvx v24,r10,$sp + addi r10,r10,32 + lvx v25,r11,$sp + addi r11,r11,32 + lvx v26,r10,$sp + addi r10,r10,32 + lvx v27,r11,$sp + addi r11,r11,32 + lvx v28,r10,$sp + addi r10,r10,32 + lvx v29,r11,$sp + addi r11,r11,32 + lvx v30,r10,$sp + lvx v31,r11,$sp + $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp) + $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp) + $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp) + $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp) + $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp) + $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp) + addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T` + blr + .long 0 + .byte 0,12,0x04,0,0x80,6,6,0 + .long 0 +.size .${prefix}_ctr32_encrypt_blocks,.-.${prefix}_ctr32_encrypt_blocks +___ +}} }}} + +######################################################################### +{{{ # XTS procedures # +# int aes_p8_xts_[en|de]crypt(const char *inp, char *out, size_t len, # +# const AES_KEY *key1, const AES_KEY *key2, # +# [const] unsigned char iv[16]); # +# If $key2 is NULL, then a "tweak chaining" mode is engaged, in which # +# input tweak value is assumed to be encrypted already, and last tweak # +# value, one suitable for consecutive call on same chunk of data, is # +# written back to original buffer. In addition, in "tweak chaining" # +# mode only complete input blocks are processed. # + +my ($inp,$out,$len,$key1,$key2,$ivp,$rounds,$idx) = map("r$_",(3..10)); +my ($rndkey0,$rndkey1,$inout) = map("v$_",(0..2)); +my ($output,$inptail,$inpperm,$leperm,$keyperm) = map("v$_",(3..7)); +my ($tweak,$seven,$eighty7,$tmp,$tweak1) = map("v$_",(8..12)); +my $taillen = $key2; + + ($inp,$idx) = ($idx,$inp); # reassign + +$code.=<<___; +.globl .${prefix}_xts_encrypt +.align 5 +.${prefix}_xts_encrypt: + mr $inp,r3 # reassign + li r3,-1 + ${UCMP}i $len,16 + bltlr- + + lis r0,0xfff0 + mfspr r12,256 # save vrsave + li r11,0 + mtspr 256,r0 + + vspltisb $seven,0x07 # 0x070707..07 + le?lvsl $leperm,r11,r11 + le?vspltisb $tmp,0x0f + le?vxor $leperm,$leperm,$seven + + li $idx,15 + lvx $tweak,0,$ivp # load [unaligned] iv + lvsl $inpperm,0,$ivp + lvx $inptail,$idx,$ivp + le?vxor $inpperm,$inpperm,$tmp + vperm $tweak,$tweak,$inptail,$inpperm + + neg r11,$inp + lvsr $inpperm,0,r11 # prepare for unaligned load + lvx $inout,0,$inp + addi $inp,$inp,15 # 15 is not typo + le?vxor $inpperm,$inpperm,$tmp + + ${UCMP}i $key2,0 # key2==NULL? + beq Lxts_enc_no_key2 + + ?lvsl $keyperm,0,$key2 # prepare for unaligned key + lwz $rounds,240($key2) + srwi $rounds,$rounds,1 + subi $rounds,$rounds,1 + li $idx,16 + + lvx $rndkey0,0,$key2 + lvx $rndkey1,$idx,$key2 + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $tweak,$tweak,$rndkey0 + lvx $rndkey0,$idx,$key2 + addi $idx,$idx,16 + mtctr $rounds + +Ltweak_xts_enc: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $tweak,$tweak,$rndkey1 + lvx $rndkey1,$idx,$key2 + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipher $tweak,$tweak,$rndkey0 + lvx $rndkey0,$idx,$key2 + addi $idx,$idx,16 + bdnz Ltweak_xts_enc + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $tweak,$tweak,$rndkey1 + lvx $rndkey1,$idx,$key2 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipherlast $tweak,$tweak,$rndkey0 + + li $ivp,0 # don't chain the tweak + b Lxts_enc + +Lxts_enc_no_key2: + li $idx,-16 + and $len,$len,$idx # in "tweak chaining" + # mode only complete + # blocks are processed +Lxts_enc: + lvx $inptail,0,$inp + addi $inp,$inp,16 + + ?lvsl $keyperm,0,$key1 # prepare for unaligned key + lwz $rounds,240($key1) + srwi $rounds,$rounds,1 + subi $rounds,$rounds,1 + li $idx,16 + + vslb $eighty7,$seven,$seven # 0x808080..80 + vor $eighty7,$eighty7,$seven # 0x878787..87 + vspltisb $tmp,1 # 0x010101..01 + vsldoi $eighty7,$eighty7,$tmp,15 # 0x870101..01 + + ${UCMP}i $len,96 + bge _aesp8_xts_encrypt6x + + andi. $taillen,$len,15 + subic r0,$len,32 + subi $taillen,$taillen,16 + subfe r0,r0,r0 + and r0,r0,$taillen + add $inp,$inp,r0 + + lvx $rndkey0,0,$key1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + vperm $inout,$inout,$inptail,$inpperm + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $inout,$inout,$tweak + vxor $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + mtctr $rounds + b Loop_xts_enc + +.align 5 +Loop_xts_enc: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipher $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + bdnz Loop_xts_enc + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key1 + li $idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $rndkey0,$rndkey0,$tweak + vcipherlast $output,$inout,$rndkey0 + + le?vperm $tmp,$output,$output,$leperm + be?nop + le?stvx_u $tmp,0,$out + be?stvx_u $output,0,$out + addi $out,$out,16 + + subic. $len,$len,16 + beq Lxts_enc_done + + vmr $inout,$inptail + lvx $inptail,0,$inp + addi $inp,$inp,16 + lvx $rndkey0,0,$key1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + + subic r0,$len,32 + subfe r0,r0,r0 + and r0,r0,$taillen + add $inp,$inp,r0 + + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vand $tmp,$tmp,$eighty7 + vxor $tweak,$tweak,$tmp + + vperm $inout,$inout,$inptail,$inpperm + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $inout,$inout,$tweak + vxor $output,$output,$rndkey0 # just in case $len<16 + vxor $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + + mtctr $rounds + ${UCMP}i $len,16 + bge Loop_xts_enc + + vxor $output,$output,$tweak + lvsr $inpperm,0,$len # $inpperm is no longer needed + vxor $inptail,$inptail,$inptail # $inptail is no longer needed + vspltisb $tmp,-1 + vperm $inptail,$inptail,$tmp,$inpperm + vsel $inout,$inout,$output,$inptail + + subi r11,$out,17 + subi $out,$out,16 + mtctr $len + li $len,16 +Loop_xts_enc_steal: + lbzu r0,1(r11) + stb r0,16(r11) + bdnz Loop_xts_enc_steal + + mtctr $rounds + b Loop_xts_enc # one more time... + +Lxts_enc_done: + ${UCMP}i $ivp,0 + beq Lxts_enc_ret + + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vand $tmp,$tmp,$eighty7 + vxor $tweak,$tweak,$tmp + + le?vperm $tweak,$tweak,$tweak,$leperm + stvx_u $tweak,0,$ivp + +Lxts_enc_ret: + mtspr 256,r12 # restore vrsave + li r3,0 + blr + .long 0 + .byte 0,12,0x04,0,0x80,6,6,0 + .long 0 +.size .${prefix}_xts_encrypt,.-.${prefix}_xts_encrypt + +.globl .${prefix}_xts_decrypt +.align 5 +.${prefix}_xts_decrypt: + mr $inp,r3 # reassign + li r3,-1 + ${UCMP}i $len,16 + bltlr- + + lis r0,0xfff8 + mfspr r12,256 # save vrsave + li r11,0 + mtspr 256,r0 + + andi. r0,$len,15 + neg r0,r0 + andi. r0,r0,16 + sub $len,$len,r0 + + vspltisb $seven,0x07 # 0x070707..07 + le?lvsl $leperm,r11,r11 + le?vspltisb $tmp,0x0f + le?vxor $leperm,$leperm,$seven + + li $idx,15 + lvx $tweak,0,$ivp # load [unaligned] iv + lvsl $inpperm,0,$ivp + lvx $inptail,$idx,$ivp + le?vxor $inpperm,$inpperm,$tmp + vperm $tweak,$tweak,$inptail,$inpperm + + neg r11,$inp + lvsr $inpperm,0,r11 # prepare for unaligned load + lvx $inout,0,$inp + addi $inp,$inp,15 # 15 is not typo + le?vxor $inpperm,$inpperm,$tmp + + ${UCMP}i $key2,0 # key2==NULL? + beq Lxts_dec_no_key2 + + ?lvsl $keyperm,0,$key2 # prepare for unaligned key + lwz $rounds,240($key2) + srwi $rounds,$rounds,1 + subi $rounds,$rounds,1 + li $idx,16 + + lvx $rndkey0,0,$key2 + lvx $rndkey1,$idx,$key2 + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $tweak,$tweak,$rndkey0 + lvx $rndkey0,$idx,$key2 + addi $idx,$idx,16 + mtctr $rounds + +Ltweak_xts_dec: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $tweak,$tweak,$rndkey1 + lvx $rndkey1,$idx,$key2 + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipher $tweak,$tweak,$rndkey0 + lvx $rndkey0,$idx,$key2 + addi $idx,$idx,16 + bdnz Ltweak_xts_dec + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vcipher $tweak,$tweak,$rndkey1 + lvx $rndkey1,$idx,$key2 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vcipherlast $tweak,$tweak,$rndkey0 + + li $ivp,0 # don't chain the tweak + b Lxts_dec + +Lxts_dec_no_key2: + neg $idx,$len + andi. $idx,$idx,15 + add $len,$len,$idx # in "tweak chaining" + # mode only complete + # blocks are processed +Lxts_dec: + lvx $inptail,0,$inp + addi $inp,$inp,16 + + ?lvsl $keyperm,0,$key1 # prepare for unaligned key + lwz $rounds,240($key1) + srwi $rounds,$rounds,1 + subi $rounds,$rounds,1 + li $idx,16 + + vslb $eighty7,$seven,$seven # 0x808080..80 + vor $eighty7,$eighty7,$seven # 0x878787..87 + vspltisb $tmp,1 # 0x010101..01 + vsldoi $eighty7,$eighty7,$tmp,15 # 0x870101..01 + + ${UCMP}i $len,96 + bge _aesp8_xts_decrypt6x + + lvx $rndkey0,0,$key1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + vperm $inout,$inout,$inptail,$inpperm + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $inout,$inout,$tweak + vxor $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + mtctr $rounds + + ${UCMP}i $len,16 + blt Ltail_xts_dec + be?b Loop_xts_dec + +.align 5 +Loop_xts_dec: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vncipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vncipher $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + bdnz Loop_xts_dec + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vncipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key1 + li $idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $rndkey0,$rndkey0,$tweak + vncipherlast $output,$inout,$rndkey0 + + le?vperm $tmp,$output,$output,$leperm + be?nop + le?stvx_u $tmp,0,$out + be?stvx_u $output,0,$out + addi $out,$out,16 + + subic. $len,$len,16 + beq Lxts_dec_done + + vmr $inout,$inptail + lvx $inptail,0,$inp + addi $inp,$inp,16 + lvx $rndkey0,0,$key1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vand $tmp,$tmp,$eighty7 + vxor $tweak,$tweak,$tmp + + vperm $inout,$inout,$inptail,$inpperm + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $inout,$inout,$tweak + vxor $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + + mtctr $rounds + ${UCMP}i $len,16 + bge Loop_xts_dec + +Ltail_xts_dec: + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak1,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vand $tmp,$tmp,$eighty7 + vxor $tweak1,$tweak1,$tmp + + subi $inp,$inp,16 + add $inp,$inp,$len + + vxor $inout,$inout,$tweak # :-( + vxor $inout,$inout,$tweak1 # :-) + +Loop_xts_dec_short: + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vncipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vncipher $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + bdnz Loop_xts_dec_short + + ?vperm $rndkey1,$rndkey1,$rndkey0,$keyperm + vncipher $inout,$inout,$rndkey1 + lvx $rndkey1,$idx,$key1 + li $idx,16 + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + vxor $rndkey0,$rndkey0,$tweak1 + vncipherlast $output,$inout,$rndkey0 + + le?vperm $tmp,$output,$output,$leperm + be?nop + le?stvx_u $tmp,0,$out + be?stvx_u $output,0,$out + + vmr $inout,$inptail + lvx $inptail,0,$inp + #addi $inp,$inp,16 + lvx $rndkey0,0,$key1 + lvx $rndkey1,$idx,$key1 + addi $idx,$idx,16 + vperm $inout,$inout,$inptail,$inpperm + ?vperm $rndkey0,$rndkey0,$rndkey1,$keyperm + + lvsr $inpperm,0,$len # $inpperm is no longer needed + vxor $inptail,$inptail,$inptail # $inptail is no longer needed + vspltisb $tmp,-1 + vperm $inptail,$inptail,$tmp,$inpperm + vsel $inout,$inout,$output,$inptail + + vxor $rndkey0,$rndkey0,$tweak + vxor $inout,$inout,$rndkey0 + lvx $rndkey0,$idx,$key1 + addi $idx,$idx,16 + + subi r11,$out,1 + mtctr $len + li $len,16 +Loop_xts_dec_steal: + lbzu r0,1(r11) + stb r0,16(r11) + bdnz Loop_xts_dec_steal + + mtctr $rounds + b Loop_xts_dec # one more time... + +Lxts_dec_done: + ${UCMP}i $ivp,0 + beq Lxts_dec_ret + + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vand $tmp,$tmp,$eighty7 + vxor $tweak,$tweak,$tmp + + le?vperm $tweak,$tweak,$tweak,$leperm + stvx_u $tweak,0,$ivp + +Lxts_dec_ret: + mtspr 256,r12 # restore vrsave + li r3,0 + blr + .long 0 + .byte 0,12,0x04,0,0x80,6,6,0 + .long 0 +.size .${prefix}_xts_decrypt,.-.${prefix}_xts_decrypt +___ +######################################################################### +{{ # Optimized XTS procedures # +my $key_=$key2; +my ($x00,$x10,$x20,$x30,$x40,$x50,$x60,$x70)=map("r$_",(0,3,26..31)); + $x00=0 if ($flavour =~ /osx/); +my ($in0, $in1, $in2, $in3, $in4, $in5 )=map("v$_",(0..5)); +my ($out0, $out1, $out2, $out3, $out4, $out5)=map("v$_",(7,12..16)); +my ($twk0, $twk1, $twk2, $twk3, $twk4, $twk5)=map("v$_",(17..22)); +my $rndkey0="v23"; # v24-v25 rotating buffer for first found keys + # v26-v31 last 6 round keys +my ($keyperm)=($out0); # aliases with "caller", redundant assignment +my $taillen=$x70; + +$code.=<<___; +.align 5 +_aesp8_xts_encrypt6x: + $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp) + mflr r11 + li r7,`$FRAME+8*16+15` + li r3,`$FRAME+8*16+31` + $PUSH r11,`$FRAME+21*16+6*$SIZE_T+$LRSAVE`($sp) + stvx v20,r7,$sp # ABI says so + addi r7,r7,32 + stvx v21,r3,$sp + addi r3,r3,32 + stvx v22,r7,$sp + addi r7,r7,32 + stvx v23,r3,$sp + addi r3,r3,32 + stvx v24,r7,$sp + addi r7,r7,32 + stvx v25,r3,$sp + addi r3,r3,32 + stvx v26,r7,$sp + addi r7,r7,32 + stvx v27,r3,$sp + addi r3,r3,32 + stvx v28,r7,$sp + addi r7,r7,32 + stvx v29,r3,$sp + addi r3,r3,32 + stvx v30,r7,$sp + stvx v31,r3,$sp + li r0,-1 + stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave + li $x10,0x10 + $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp) + li $x20,0x20 + $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp) + li $x30,0x30 + $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp) + li $x40,0x40 + $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp) + li $x50,0x50 + $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp) + li $x60,0x60 + $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp) + li $x70,0x70 + mtspr 256,r0 + + subi $rounds,$rounds,3 # -4 in total + + lvx $rndkey0,$x00,$key1 # load key schedule + lvx v30,$x10,$key1 + addi $key1,$key1,0x20 + lvx v31,$x00,$key1 + ?vperm $rndkey0,$rndkey0,v30,$keyperm + addi $key_,$sp,$FRAME+15 + mtctr $rounds + +Load_xts_enc_key: + ?vperm v24,v30,v31,$keyperm + lvx v30,$x10,$key1 + addi $key1,$key1,0x20 + stvx v24,$x00,$key_ # off-load round[1] + ?vperm v25,v31,v30,$keyperm + lvx v31,$x00,$key1 + stvx v25,$x10,$key_ # off-load round[2] + addi $key_,$key_,0x20 + bdnz Load_xts_enc_key + + lvx v26,$x10,$key1 + ?vperm v24,v30,v31,$keyperm + lvx v27,$x20,$key1 + stvx v24,$x00,$key_ # off-load round[3] + ?vperm v25,v31,v26,$keyperm + lvx v28,$x30,$key1 + stvx v25,$x10,$key_ # off-load round[4] + addi $key_,$sp,$FRAME+15 # rewind $key_ + ?vperm v26,v26,v27,$keyperm + lvx v29,$x40,$key1 + ?vperm v27,v27,v28,$keyperm + lvx v30,$x50,$key1 + ?vperm v28,v28,v29,$keyperm + lvx v31,$x60,$key1 + ?vperm v29,v29,v30,$keyperm + lvx $twk5,$x70,$key1 # borrow $twk5 + ?vperm v30,v30,v31,$keyperm + lvx v24,$x00,$key_ # pre-load round[1] + ?vperm v31,v31,$twk5,$keyperm + lvx v25,$x10,$key_ # pre-load round[2] + + vperm $in0,$inout,$inptail,$inpperm + subi $inp,$inp,31 # undo "caller" + vxor $twk0,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vand $tmp,$tmp,$eighty7 + vxor $out0,$in0,$twk0 + vxor $tweak,$tweak,$tmp + + lvx_u $in1,$x10,$inp + vxor $twk1,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in1,$in1,$in1,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out1,$in1,$twk1 + vxor $tweak,$tweak,$tmp + + lvx_u $in2,$x20,$inp + andi. $taillen,$len,15 + vxor $twk2,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in2,$in2,$in2,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out2,$in2,$twk2 + vxor $tweak,$tweak,$tmp + + lvx_u $in3,$x30,$inp + sub $len,$len,$taillen + vxor $twk3,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in3,$in3,$in3,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out3,$in3,$twk3 + vxor $tweak,$tweak,$tmp + + lvx_u $in4,$x40,$inp + subi $len,$len,0x60 + vxor $twk4,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in4,$in4,$in4,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out4,$in4,$twk4 + vxor $tweak,$tweak,$tmp + + lvx_u $in5,$x50,$inp + addi $inp,$inp,0x60 + vxor $twk5,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in5,$in5,$in5,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out5,$in5,$twk5 + vxor $tweak,$tweak,$tmp + + vxor v31,v31,$rndkey0 + mtctr $rounds + b Loop_xts_enc6x + +.align 5 +Loop_xts_enc6x: + vcipher $out0,$out0,v24 + vcipher $out1,$out1,v24 + vcipher $out2,$out2,v24 + vcipher $out3,$out3,v24 + vcipher $out4,$out4,v24 + vcipher $out5,$out5,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vcipher $out0,$out0,v25 + vcipher $out1,$out1,v25 + vcipher $out2,$out2,v25 + vcipher $out3,$out3,v25 + vcipher $out4,$out4,v25 + vcipher $out5,$out5,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Loop_xts_enc6x + + subic $len,$len,96 # $len-=96 + vxor $in0,$twk0,v31 # xor with last round key + vcipher $out0,$out0,v24 + vcipher $out1,$out1,v24 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk0,$tweak,$rndkey0 + vaddubm $tweak,$tweak,$tweak + vcipher $out2,$out2,v24 + vcipher $out3,$out3,v24 + vsldoi $tmp,$tmp,$tmp,15 + vcipher $out4,$out4,v24 + vcipher $out5,$out5,v24 + + subfe. r0,r0,r0 # borrow?-1:0 + vand $tmp,$tmp,$eighty7 + vcipher $out0,$out0,v25 + vcipher $out1,$out1,v25 + vxor $tweak,$tweak,$tmp + vcipher $out2,$out2,v25 + vcipher $out3,$out3,v25 + vxor $in1,$twk1,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk1,$tweak,$rndkey0 + vcipher $out4,$out4,v25 + vcipher $out5,$out5,v25 + + and r0,r0,$len + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vcipher $out0,$out0,v26 + vcipher $out1,$out1,v26 + vand $tmp,$tmp,$eighty7 + vcipher $out2,$out2,v26 + vcipher $out3,$out3,v26 + vxor $tweak,$tweak,$tmp + vcipher $out4,$out4,v26 + vcipher $out5,$out5,v26 + + add $inp,$inp,r0 # $inp is adjusted in such + # way that at exit from the + # loop inX-in5 are loaded + # with last "words" + vxor $in2,$twk2,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk2,$tweak,$rndkey0 + vaddubm $tweak,$tweak,$tweak + vcipher $out0,$out0,v27 + vcipher $out1,$out1,v27 + vsldoi $tmp,$tmp,$tmp,15 + vcipher $out2,$out2,v27 + vcipher $out3,$out3,v27 + vand $tmp,$tmp,$eighty7 + vcipher $out4,$out4,v27 + vcipher $out5,$out5,v27 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vxor $tweak,$tweak,$tmp + vcipher $out0,$out0,v28 + vcipher $out1,$out1,v28 + vxor $in3,$twk3,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk3,$tweak,$rndkey0 + vcipher $out2,$out2,v28 + vcipher $out3,$out3,v28 + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vcipher $out4,$out4,v28 + vcipher $out5,$out5,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + vand $tmp,$tmp,$eighty7 + + vcipher $out0,$out0,v29 + vcipher $out1,$out1,v29 + vxor $tweak,$tweak,$tmp + vcipher $out2,$out2,v29 + vcipher $out3,$out3,v29 + vxor $in4,$twk4,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk4,$tweak,$rndkey0 + vcipher $out4,$out4,v29 + vcipher $out5,$out5,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + + vcipher $out0,$out0,v30 + vcipher $out1,$out1,v30 + vand $tmp,$tmp,$eighty7 + vcipher $out2,$out2,v30 + vcipher $out3,$out3,v30 + vxor $tweak,$tweak,$tmp + vcipher $out4,$out4,v30 + vcipher $out5,$out5,v30 + vxor $in5,$twk5,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk5,$tweak,$rndkey0 + + vcipherlast $out0,$out0,$in0 + lvx_u $in0,$x00,$inp # load next input block + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vcipherlast $out1,$out1,$in1 + lvx_u $in1,$x10,$inp + vcipherlast $out2,$out2,$in2 + le?vperm $in0,$in0,$in0,$leperm + lvx_u $in2,$x20,$inp + vand $tmp,$tmp,$eighty7 + vcipherlast $out3,$out3,$in3 + le?vperm $in1,$in1,$in1,$leperm + lvx_u $in3,$x30,$inp + vcipherlast $out4,$out4,$in4 + le?vperm $in2,$in2,$in2,$leperm + lvx_u $in4,$x40,$inp + vxor $tweak,$tweak,$tmp + vcipherlast $tmp,$out5,$in5 # last block might be needed + # in stealing mode + le?vperm $in3,$in3,$in3,$leperm + lvx_u $in5,$x50,$inp + addi $inp,$inp,0x60 + le?vperm $in4,$in4,$in4,$leperm + le?vperm $in5,$in5,$in5,$leperm + + le?vperm $out0,$out0,$out0,$leperm + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + vxor $out0,$in0,$twk0 + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + vxor $out1,$in1,$twk1 + le?vperm $out3,$out3,$out3,$leperm + stvx_u $out2,$x20,$out + vxor $out2,$in2,$twk2 + le?vperm $out4,$out4,$out4,$leperm + stvx_u $out3,$x30,$out + vxor $out3,$in3,$twk3 + le?vperm $out5,$tmp,$tmp,$leperm + stvx_u $out4,$x40,$out + vxor $out4,$in4,$twk4 + le?stvx_u $out5,$x50,$out + be?stvx_u $tmp, $x50,$out + vxor $out5,$in5,$twk5 + addi $out,$out,0x60 + + mtctr $rounds + beq Loop_xts_enc6x # did $len-=96 borrow? + + addic. $len,$len,0x60 + beq Lxts_enc6x_zero + cmpwi $len,0x20 + blt Lxts_enc6x_one + nop + beq Lxts_enc6x_two + cmpwi $len,0x40 + blt Lxts_enc6x_three + nop + beq Lxts_enc6x_four + +Lxts_enc6x_five: + vxor $out0,$in1,$twk0 + vxor $out1,$in2,$twk1 + vxor $out2,$in3,$twk2 + vxor $out3,$in4,$twk3 + vxor $out4,$in5,$twk4 + + bl _aesp8_xts_enc5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk5 # unused tweak + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$leperm + stvx_u $out2,$x20,$out + vxor $tmp,$out4,$twk5 # last block prep for stealing + le?vperm $out4,$out4,$out4,$leperm + stvx_u $out3,$x30,$out + stvx_u $out4,$x40,$out + addi $out,$out,0x50 + bne Lxts_enc6x_steal + b Lxts_enc6x_done + +.align 4 +Lxts_enc6x_four: + vxor $out0,$in2,$twk0 + vxor $out1,$in3,$twk1 + vxor $out2,$in4,$twk2 + vxor $out3,$in5,$twk3 + vxor $out4,$out4,$out4 + + bl _aesp8_xts_enc5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk4 # unused tweak + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + vxor $tmp,$out3,$twk4 # last block prep for stealing + le?vperm $out3,$out3,$out3,$leperm + stvx_u $out2,$x20,$out + stvx_u $out3,$x30,$out + addi $out,$out,0x40 + bne Lxts_enc6x_steal + b Lxts_enc6x_done + +.align 4 +Lxts_enc6x_three: + vxor $out0,$in3,$twk0 + vxor $out1,$in4,$twk1 + vxor $out2,$in5,$twk2 + vxor $out3,$out3,$out3 + vxor $out4,$out4,$out4 + + bl _aesp8_xts_enc5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk3 # unused tweak + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + vxor $tmp,$out2,$twk3 # last block prep for stealing + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + stvx_u $out2,$x20,$out + addi $out,$out,0x30 + bne Lxts_enc6x_steal + b Lxts_enc6x_done + +.align 4 +Lxts_enc6x_two: + vxor $out0,$in4,$twk0 + vxor $out1,$in5,$twk1 + vxor $out2,$out2,$out2 + vxor $out3,$out3,$out3 + vxor $out4,$out4,$out4 + + bl _aesp8_xts_enc5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk2 # unused tweak + vxor $tmp,$out1,$twk2 # last block prep for stealing + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + stvx_u $out1,$x10,$out + addi $out,$out,0x20 + bne Lxts_enc6x_steal + b Lxts_enc6x_done + +.align 4 +Lxts_enc6x_one: + vxor $out0,$in5,$twk0 + nop +Loop_xts_enc1x: + vcipher $out0,$out0,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vcipher $out0,$out0,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Loop_xts_enc1x + + add $inp,$inp,$taillen + cmpwi $taillen,0 + vcipher $out0,$out0,v24 + + subi $inp,$inp,16 + vcipher $out0,$out0,v25 + + lvsr $inpperm,0,$taillen + vcipher $out0,$out0,v26 + + lvx_u $in0,0,$inp + vcipher $out0,$out0,v27 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vcipher $out0,$out0,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + + vcipher $out0,$out0,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + vxor $twk0,$twk0,v31 + + le?vperm $in0,$in0,$in0,$leperm + vcipher $out0,$out0,v30 + + vperm $in0,$in0,$in0,$inpperm + vcipherlast $out0,$out0,$twk0 + + vmr $twk0,$twk1 # unused tweak + vxor $tmp,$out0,$twk1 # last block prep for stealing + le?vperm $out0,$out0,$out0,$leperm + stvx_u $out0,$x00,$out # store output + addi $out,$out,0x10 + bne Lxts_enc6x_steal + b Lxts_enc6x_done + +.align 4 +Lxts_enc6x_zero: + cmpwi $taillen,0 + beq Lxts_enc6x_done + + add $inp,$inp,$taillen + subi $inp,$inp,16 + lvx_u $in0,0,$inp + lvsr $inpperm,0,$taillen # $in5 is no more + le?vperm $in0,$in0,$in0,$leperm + vperm $in0,$in0,$in0,$inpperm + vxor $tmp,$tmp,$twk0 +Lxts_enc6x_steal: + vxor $in0,$in0,$twk0 + vxor $out0,$out0,$out0 + vspltisb $out1,-1 + vperm $out0,$out0,$out1,$inpperm + vsel $out0,$in0,$tmp,$out0 # $tmp is last block, remember? + + subi r30,$out,17 + subi $out,$out,16 + mtctr $taillen +Loop_xts_enc6x_steal: + lbzu r0,1(r30) + stb r0,16(r30) + bdnz Loop_xts_enc6x_steal + + li $taillen,0 + mtctr $rounds + b Loop_xts_enc1x # one more time... + +.align 4 +Lxts_enc6x_done: + ${UCMP}i $ivp,0 + beq Lxts_enc6x_ret + + vxor $tweak,$twk0,$rndkey0 + le?vperm $tweak,$tweak,$tweak,$leperm + stvx_u $tweak,0,$ivp + +Lxts_enc6x_ret: + mtlr r11 + li r10,`$FRAME+15` + li r11,`$FRAME+31` + stvx $seven,r10,$sp # wipe copies of round keys + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + stvx $seven,r10,$sp + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + stvx $seven,r10,$sp + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + stvx $seven,r10,$sp + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + + mtspr 256,$vrsave + lvx v20,r10,$sp # ABI says so + addi r10,r10,32 + lvx v21,r11,$sp + addi r11,r11,32 + lvx v22,r10,$sp + addi r10,r10,32 + lvx v23,r11,$sp + addi r11,r11,32 + lvx v24,r10,$sp + addi r10,r10,32 + lvx v25,r11,$sp + addi r11,r11,32 + lvx v26,r10,$sp + addi r10,r10,32 + lvx v27,r11,$sp + addi r11,r11,32 + lvx v28,r10,$sp + addi r10,r10,32 + lvx v29,r11,$sp + addi r11,r11,32 + lvx v30,r10,$sp + lvx v31,r11,$sp + $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp) + $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp) + $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp) + $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp) + $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp) + $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp) + addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T` + blr + .long 0 + .byte 0,12,0x04,1,0x80,6,6,0 + .long 0 + +.align 5 +_aesp8_xts_enc5x: + vcipher $out0,$out0,v24 + vcipher $out1,$out1,v24 + vcipher $out2,$out2,v24 + vcipher $out3,$out3,v24 + vcipher $out4,$out4,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vcipher $out0,$out0,v25 + vcipher $out1,$out1,v25 + vcipher $out2,$out2,v25 + vcipher $out3,$out3,v25 + vcipher $out4,$out4,v25 + lvx v25,$x10,$key_ # round[4] + bdnz _aesp8_xts_enc5x + + add $inp,$inp,$taillen + cmpwi $taillen,0 + vcipher $out0,$out0,v24 + vcipher $out1,$out1,v24 + vcipher $out2,$out2,v24 + vcipher $out3,$out3,v24 + vcipher $out4,$out4,v24 + + subi $inp,$inp,16 + vcipher $out0,$out0,v25 + vcipher $out1,$out1,v25 + vcipher $out2,$out2,v25 + vcipher $out3,$out3,v25 + vcipher $out4,$out4,v25 + vxor $twk0,$twk0,v31 + + vcipher $out0,$out0,v26 + lvsr $inpperm,0,$taillen # $in5 is no more + vcipher $out1,$out1,v26 + vcipher $out2,$out2,v26 + vcipher $out3,$out3,v26 + vcipher $out4,$out4,v26 + vxor $in1,$twk1,v31 + + vcipher $out0,$out0,v27 + lvx_u $in0,0,$inp + vcipher $out1,$out1,v27 + vcipher $out2,$out2,v27 + vcipher $out3,$out3,v27 + vcipher $out4,$out4,v27 + vxor $in2,$twk2,v31 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vcipher $out0,$out0,v28 + vcipher $out1,$out1,v28 + vcipher $out2,$out2,v28 + vcipher $out3,$out3,v28 + vcipher $out4,$out4,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + vxor $in3,$twk3,v31 + + vcipher $out0,$out0,v29 + le?vperm $in0,$in0,$in0,$leperm + vcipher $out1,$out1,v29 + vcipher $out2,$out2,v29 + vcipher $out3,$out3,v29 + vcipher $out4,$out4,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + vxor $in4,$twk4,v31 + + vcipher $out0,$out0,v30 + vperm $in0,$in0,$in0,$inpperm + vcipher $out1,$out1,v30 + vcipher $out2,$out2,v30 + vcipher $out3,$out3,v30 + vcipher $out4,$out4,v30 + + vcipherlast $out0,$out0,$twk0 + vcipherlast $out1,$out1,$in1 + vcipherlast $out2,$out2,$in2 + vcipherlast $out3,$out3,$in3 + vcipherlast $out4,$out4,$in4 + blr + .long 0 + .byte 0,12,0x14,0,0,0,0,0 + +.align 5 +_aesp8_xts_decrypt6x: + $STU $sp,-`($FRAME+21*16+6*$SIZE_T)`($sp) + mflr r11 + li r7,`$FRAME+8*16+15` + li r3,`$FRAME+8*16+31` + $PUSH r11,`$FRAME+21*16+6*$SIZE_T+$LRSAVE`($sp) + stvx v20,r7,$sp # ABI says so + addi r7,r7,32 + stvx v21,r3,$sp + addi r3,r3,32 + stvx v22,r7,$sp + addi r7,r7,32 + stvx v23,r3,$sp + addi r3,r3,32 + stvx v24,r7,$sp + addi r7,r7,32 + stvx v25,r3,$sp + addi r3,r3,32 + stvx v26,r7,$sp + addi r7,r7,32 + stvx v27,r3,$sp + addi r3,r3,32 + stvx v28,r7,$sp + addi r7,r7,32 + stvx v29,r3,$sp + addi r3,r3,32 + stvx v30,r7,$sp + stvx v31,r3,$sp + li r0,-1 + stw $vrsave,`$FRAME+21*16-4`($sp) # save vrsave + li $x10,0x10 + $PUSH r26,`$FRAME+21*16+0*$SIZE_T`($sp) + li $x20,0x20 + $PUSH r27,`$FRAME+21*16+1*$SIZE_T`($sp) + li $x30,0x30 + $PUSH r28,`$FRAME+21*16+2*$SIZE_T`($sp) + li $x40,0x40 + $PUSH r29,`$FRAME+21*16+3*$SIZE_T`($sp) + li $x50,0x50 + $PUSH r30,`$FRAME+21*16+4*$SIZE_T`($sp) + li $x60,0x60 + $PUSH r31,`$FRAME+21*16+5*$SIZE_T`($sp) + li $x70,0x70 + mtspr 256,r0 + + subi $rounds,$rounds,3 # -4 in total + + lvx $rndkey0,$x00,$key1 # load key schedule + lvx v30,$x10,$key1 + addi $key1,$key1,0x20 + lvx v31,$x00,$key1 + ?vperm $rndkey0,$rndkey0,v30,$keyperm + addi $key_,$sp,$FRAME+15 + mtctr $rounds + +Load_xts_dec_key: + ?vperm v24,v30,v31,$keyperm + lvx v30,$x10,$key1 + addi $key1,$key1,0x20 + stvx v24,$x00,$key_ # off-load round[1] + ?vperm v25,v31,v30,$keyperm + lvx v31,$x00,$key1 + stvx v25,$x10,$key_ # off-load round[2] + addi $key_,$key_,0x20 + bdnz Load_xts_dec_key + + lvx v26,$x10,$key1 + ?vperm v24,v30,v31,$keyperm + lvx v27,$x20,$key1 + stvx v24,$x00,$key_ # off-load round[3] + ?vperm v25,v31,v26,$keyperm + lvx v28,$x30,$key1 + stvx v25,$x10,$key_ # off-load round[4] + addi $key_,$sp,$FRAME+15 # rewind $key_ + ?vperm v26,v26,v27,$keyperm + lvx v29,$x40,$key1 + ?vperm v27,v27,v28,$keyperm + lvx v30,$x50,$key1 + ?vperm v28,v28,v29,$keyperm + lvx v31,$x60,$key1 + ?vperm v29,v29,v30,$keyperm + lvx $twk5,$x70,$key1 # borrow $twk5 + ?vperm v30,v30,v31,$keyperm + lvx v24,$x00,$key_ # pre-load round[1] + ?vperm v31,v31,$twk5,$keyperm + lvx v25,$x10,$key_ # pre-load round[2] + + vperm $in0,$inout,$inptail,$inpperm + subi $inp,$inp,31 # undo "caller" + vxor $twk0,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vand $tmp,$tmp,$eighty7 + vxor $out0,$in0,$twk0 + vxor $tweak,$tweak,$tmp + + lvx_u $in1,$x10,$inp + vxor $twk1,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in1,$in1,$in1,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out1,$in1,$twk1 + vxor $tweak,$tweak,$tmp + + lvx_u $in2,$x20,$inp + andi. $taillen,$len,15 + vxor $twk2,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in2,$in2,$in2,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out2,$in2,$twk2 + vxor $tweak,$tweak,$tmp + + lvx_u $in3,$x30,$inp + sub $len,$len,$taillen + vxor $twk3,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in3,$in3,$in3,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out3,$in3,$twk3 + vxor $tweak,$tweak,$tmp + + lvx_u $in4,$x40,$inp + subi $len,$len,0x60 + vxor $twk4,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in4,$in4,$in4,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out4,$in4,$twk4 + vxor $tweak,$tweak,$tmp + + lvx_u $in5,$x50,$inp + addi $inp,$inp,0x60 + vxor $twk5,$tweak,$rndkey0 + vsrab $tmp,$tweak,$seven # next tweak value + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + le?vperm $in5,$in5,$in5,$leperm + vand $tmp,$tmp,$eighty7 + vxor $out5,$in5,$twk5 + vxor $tweak,$tweak,$tmp + + vxor v31,v31,$rndkey0 + mtctr $rounds + b Loop_xts_dec6x + +.align 5 +Loop_xts_dec6x: + vncipher $out0,$out0,v24 + vncipher $out1,$out1,v24 + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vncipher $out4,$out4,v24 + vncipher $out5,$out5,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vncipher $out0,$out0,v25 + vncipher $out1,$out1,v25 + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vncipher $out4,$out4,v25 + vncipher $out5,$out5,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Loop_xts_dec6x + + subic $len,$len,96 # $len-=96 + vxor $in0,$twk0,v31 # xor with last round key + vncipher $out0,$out0,v24 + vncipher $out1,$out1,v24 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk0,$tweak,$rndkey0 + vaddubm $tweak,$tweak,$tweak + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vsldoi $tmp,$tmp,$tmp,15 + vncipher $out4,$out4,v24 + vncipher $out5,$out5,v24 + + subfe. r0,r0,r0 # borrow?-1:0 + vand $tmp,$tmp,$eighty7 + vncipher $out0,$out0,v25 + vncipher $out1,$out1,v25 + vxor $tweak,$tweak,$tmp + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vxor $in1,$twk1,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk1,$tweak,$rndkey0 + vncipher $out4,$out4,v25 + vncipher $out5,$out5,v25 + + and r0,r0,$len + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vncipher $out0,$out0,v26 + vncipher $out1,$out1,v26 + vand $tmp,$tmp,$eighty7 + vncipher $out2,$out2,v26 + vncipher $out3,$out3,v26 + vxor $tweak,$tweak,$tmp + vncipher $out4,$out4,v26 + vncipher $out5,$out5,v26 + + add $inp,$inp,r0 # $inp is adjusted in such + # way that at exit from the + # loop inX-in5 are loaded + # with last "words" + vxor $in2,$twk2,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk2,$tweak,$rndkey0 + vaddubm $tweak,$tweak,$tweak + vncipher $out0,$out0,v27 + vncipher $out1,$out1,v27 + vsldoi $tmp,$tmp,$tmp,15 + vncipher $out2,$out2,v27 + vncipher $out3,$out3,v27 + vand $tmp,$tmp,$eighty7 + vncipher $out4,$out4,v27 + vncipher $out5,$out5,v27 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vxor $tweak,$tweak,$tmp + vncipher $out0,$out0,v28 + vncipher $out1,$out1,v28 + vxor $in3,$twk3,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk3,$tweak,$rndkey0 + vncipher $out2,$out2,v28 + vncipher $out3,$out3,v28 + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vncipher $out4,$out4,v28 + vncipher $out5,$out5,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + vand $tmp,$tmp,$eighty7 + + vncipher $out0,$out0,v29 + vncipher $out1,$out1,v29 + vxor $tweak,$tweak,$tmp + vncipher $out2,$out2,v29 + vncipher $out3,$out3,v29 + vxor $in4,$twk4,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk4,$tweak,$rndkey0 + vncipher $out4,$out4,v29 + vncipher $out5,$out5,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + + vncipher $out0,$out0,v30 + vncipher $out1,$out1,v30 + vand $tmp,$tmp,$eighty7 + vncipher $out2,$out2,v30 + vncipher $out3,$out3,v30 + vxor $tweak,$tweak,$tmp + vncipher $out4,$out4,v30 + vncipher $out5,$out5,v30 + vxor $in5,$twk5,v31 + vsrab $tmp,$tweak,$seven # next tweak value + vxor $twk5,$tweak,$rndkey0 + + vncipherlast $out0,$out0,$in0 + lvx_u $in0,$x00,$inp # load next input block + vaddubm $tweak,$tweak,$tweak + vsldoi $tmp,$tmp,$tmp,15 + vncipherlast $out1,$out1,$in1 + lvx_u $in1,$x10,$inp + vncipherlast $out2,$out2,$in2 + le?vperm $in0,$in0,$in0,$leperm + lvx_u $in2,$x20,$inp + vand $tmp,$tmp,$eighty7 + vncipherlast $out3,$out3,$in3 + le?vperm $in1,$in1,$in1,$leperm + lvx_u $in3,$x30,$inp + vncipherlast $out4,$out4,$in4 + le?vperm $in2,$in2,$in2,$leperm + lvx_u $in4,$x40,$inp + vxor $tweak,$tweak,$tmp + vncipherlast $out5,$out5,$in5 + le?vperm $in3,$in3,$in3,$leperm + lvx_u $in5,$x50,$inp + addi $inp,$inp,0x60 + le?vperm $in4,$in4,$in4,$leperm + le?vperm $in5,$in5,$in5,$leperm + + le?vperm $out0,$out0,$out0,$leperm + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + vxor $out0,$in0,$twk0 + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + vxor $out1,$in1,$twk1 + le?vperm $out3,$out3,$out3,$leperm + stvx_u $out2,$x20,$out + vxor $out2,$in2,$twk2 + le?vperm $out4,$out4,$out4,$leperm + stvx_u $out3,$x30,$out + vxor $out3,$in3,$twk3 + le?vperm $out5,$out5,$out5,$leperm + stvx_u $out4,$x40,$out + vxor $out4,$in4,$twk4 + stvx_u $out5,$x50,$out + vxor $out5,$in5,$twk5 + addi $out,$out,0x60 + + mtctr $rounds + beq Loop_xts_dec6x # did $len-=96 borrow? + + addic. $len,$len,0x60 + beq Lxts_dec6x_zero + cmpwi $len,0x20 + blt Lxts_dec6x_one + nop + beq Lxts_dec6x_two + cmpwi $len,0x40 + blt Lxts_dec6x_three + nop + beq Lxts_dec6x_four + +Lxts_dec6x_five: + vxor $out0,$in1,$twk0 + vxor $out1,$in2,$twk1 + vxor $out2,$in3,$twk2 + vxor $out3,$in4,$twk3 + vxor $out4,$in5,$twk4 + + bl _aesp8_xts_dec5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk5 # unused tweak + vxor $twk1,$tweak,$rndkey0 + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + vxor $out0,$in0,$twk1 + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$leperm + stvx_u $out2,$x20,$out + le?vperm $out4,$out4,$out4,$leperm + stvx_u $out3,$x30,$out + stvx_u $out4,$x40,$out + addi $out,$out,0x50 + bne Lxts_dec6x_steal + b Lxts_dec6x_done + +.align 4 +Lxts_dec6x_four: + vxor $out0,$in2,$twk0 + vxor $out1,$in3,$twk1 + vxor $out2,$in4,$twk2 + vxor $out3,$in5,$twk3 + vxor $out4,$out4,$out4 + + bl _aesp8_xts_dec5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk4 # unused tweak + vmr $twk1,$twk5 + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + vxor $out0,$in0,$twk5 + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + le?vperm $out3,$out3,$out3,$leperm + stvx_u $out2,$x20,$out + stvx_u $out3,$x30,$out + addi $out,$out,0x40 + bne Lxts_dec6x_steal + b Lxts_dec6x_done + +.align 4 +Lxts_dec6x_three: + vxor $out0,$in3,$twk0 + vxor $out1,$in4,$twk1 + vxor $out2,$in5,$twk2 + vxor $out3,$out3,$out3 + vxor $out4,$out4,$out4 + + bl _aesp8_xts_dec5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk3 # unused tweak + vmr $twk1,$twk4 + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + vxor $out0,$in0,$twk4 + le?vperm $out2,$out2,$out2,$leperm + stvx_u $out1,$x10,$out + stvx_u $out2,$x20,$out + addi $out,$out,0x30 + bne Lxts_dec6x_steal + b Lxts_dec6x_done + +.align 4 +Lxts_dec6x_two: + vxor $out0,$in4,$twk0 + vxor $out1,$in5,$twk1 + vxor $out2,$out2,$out2 + vxor $out3,$out3,$out3 + vxor $out4,$out4,$out4 + + bl _aesp8_xts_dec5x + + le?vperm $out0,$out0,$out0,$leperm + vmr $twk0,$twk2 # unused tweak + vmr $twk1,$twk3 + le?vperm $out1,$out1,$out1,$leperm + stvx_u $out0,$x00,$out # store output + vxor $out0,$in0,$twk3 + stvx_u $out1,$x10,$out + addi $out,$out,0x20 + bne Lxts_dec6x_steal + b Lxts_dec6x_done + +.align 4 +Lxts_dec6x_one: + vxor $out0,$in5,$twk0 + nop +Loop_xts_dec1x: + vncipher $out0,$out0,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vncipher $out0,$out0,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Loop_xts_dec1x + + subi r0,$taillen,1 + vncipher $out0,$out0,v24 + + andi. r0,r0,16 + cmpwi $taillen,0 + vncipher $out0,$out0,v25 + + sub $inp,$inp,r0 + vncipher $out0,$out0,v26 + + lvx_u $in0,0,$inp + vncipher $out0,$out0,v27 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vncipher $out0,$out0,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + + vncipher $out0,$out0,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + vxor $twk0,$twk0,v31 + + le?vperm $in0,$in0,$in0,$leperm + vncipher $out0,$out0,v30 + + mtctr $rounds + vncipherlast $out0,$out0,$twk0 + + vmr $twk0,$twk1 # unused tweak + vmr $twk1,$twk2 + le?vperm $out0,$out0,$out0,$leperm + stvx_u $out0,$x00,$out # store output + addi $out,$out,0x10 + vxor $out0,$in0,$twk2 + bne Lxts_dec6x_steal + b Lxts_dec6x_done + +.align 4 +Lxts_dec6x_zero: + cmpwi $taillen,0 + beq Lxts_dec6x_done + + lvx_u $in0,0,$inp + le?vperm $in0,$in0,$in0,$leperm + vxor $out0,$in0,$twk1 +Lxts_dec6x_steal: + vncipher $out0,$out0,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vncipher $out0,$out0,v25 + lvx v25,$x10,$key_ # round[4] + bdnz Lxts_dec6x_steal + + add $inp,$inp,$taillen + vncipher $out0,$out0,v24 + + cmpwi $taillen,0 + vncipher $out0,$out0,v25 + + lvx_u $in0,0,$inp + vncipher $out0,$out0,v26 + + lvsr $inpperm,0,$taillen # $in5 is no more + vncipher $out0,$out0,v27 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vncipher $out0,$out0,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + + vncipher $out0,$out0,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + vxor $twk1,$twk1,v31 + + le?vperm $in0,$in0,$in0,$leperm + vncipher $out0,$out0,v30 + + vperm $in0,$in0,$in0,$inpperm + vncipherlast $tmp,$out0,$twk1 + + le?vperm $out0,$tmp,$tmp,$leperm + le?stvx_u $out0,0,$out + be?stvx_u $tmp,0,$out + + vxor $out0,$out0,$out0 + vspltisb $out1,-1 + vperm $out0,$out0,$out1,$inpperm + vsel $out0,$in0,$tmp,$out0 + vxor $out0,$out0,$twk0 + + subi r30,$out,1 + mtctr $taillen +Loop_xts_dec6x_steal: + lbzu r0,1(r30) + stb r0,16(r30) + bdnz Loop_xts_dec6x_steal + + li $taillen,0 + mtctr $rounds + b Loop_xts_dec1x # one more time... + +.align 4 +Lxts_dec6x_done: + ${UCMP}i $ivp,0 + beq Lxts_dec6x_ret + + vxor $tweak,$twk0,$rndkey0 + le?vperm $tweak,$tweak,$tweak,$leperm + stvx_u $tweak,0,$ivp + +Lxts_dec6x_ret: + mtlr r11 + li r10,`$FRAME+15` + li r11,`$FRAME+31` + stvx $seven,r10,$sp # wipe copies of round keys + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + stvx $seven,r10,$sp + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + stvx $seven,r10,$sp + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + stvx $seven,r10,$sp + addi r10,r10,32 + stvx $seven,r11,$sp + addi r11,r11,32 + + mtspr 256,$vrsave + lvx v20,r10,$sp # ABI says so + addi r10,r10,32 + lvx v21,r11,$sp + addi r11,r11,32 + lvx v22,r10,$sp + addi r10,r10,32 + lvx v23,r11,$sp + addi r11,r11,32 + lvx v24,r10,$sp + addi r10,r10,32 + lvx v25,r11,$sp + addi r11,r11,32 + lvx v26,r10,$sp + addi r10,r10,32 + lvx v27,r11,$sp + addi r11,r11,32 + lvx v28,r10,$sp + addi r10,r10,32 + lvx v29,r11,$sp + addi r11,r11,32 + lvx v30,r10,$sp + lvx v31,r11,$sp + $POP r26,`$FRAME+21*16+0*$SIZE_T`($sp) + $POP r27,`$FRAME+21*16+1*$SIZE_T`($sp) + $POP r28,`$FRAME+21*16+2*$SIZE_T`($sp) + $POP r29,`$FRAME+21*16+3*$SIZE_T`($sp) + $POP r30,`$FRAME+21*16+4*$SIZE_T`($sp) + $POP r31,`$FRAME+21*16+5*$SIZE_T`($sp) + addi $sp,$sp,`$FRAME+21*16+6*$SIZE_T` + blr + .long 0 + .byte 0,12,0x04,1,0x80,6,6,0 + .long 0 + +.align 5 +_aesp8_xts_dec5x: + vncipher $out0,$out0,v24 + vncipher $out1,$out1,v24 + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vncipher $out4,$out4,v24 + lvx v24,$x20,$key_ # round[3] + addi $key_,$key_,0x20 + + vncipher $out0,$out0,v25 + vncipher $out1,$out1,v25 + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vncipher $out4,$out4,v25 + lvx v25,$x10,$key_ # round[4] + bdnz _aesp8_xts_dec5x + + subi r0,$taillen,1 + vncipher $out0,$out0,v24 + vncipher $out1,$out1,v24 + vncipher $out2,$out2,v24 + vncipher $out3,$out3,v24 + vncipher $out4,$out4,v24 + + andi. r0,r0,16 + cmpwi $taillen,0 + vncipher $out0,$out0,v25 + vncipher $out1,$out1,v25 + vncipher $out2,$out2,v25 + vncipher $out3,$out3,v25 + vncipher $out4,$out4,v25 + vxor $twk0,$twk0,v31 + + sub $inp,$inp,r0 + vncipher $out0,$out0,v26 + vncipher $out1,$out1,v26 + vncipher $out2,$out2,v26 + vncipher $out3,$out3,v26 + vncipher $out4,$out4,v26 + vxor $in1,$twk1,v31 + + vncipher $out0,$out0,v27 + lvx_u $in0,0,$inp + vncipher $out1,$out1,v27 + vncipher $out2,$out2,v27 + vncipher $out3,$out3,v27 + vncipher $out4,$out4,v27 + vxor $in2,$twk2,v31 + + addi $key_,$sp,$FRAME+15 # rewind $key_ + vncipher $out0,$out0,v28 + vncipher $out1,$out1,v28 + vncipher $out2,$out2,v28 + vncipher $out3,$out3,v28 + vncipher $out4,$out4,v28 + lvx v24,$x00,$key_ # re-pre-load round[1] + vxor $in3,$twk3,v31 + + vncipher $out0,$out0,v29 + le?vperm $in0,$in0,$in0,$leperm + vncipher $out1,$out1,v29 + vncipher $out2,$out2,v29 + vncipher $out3,$out3,v29 + vncipher $out4,$out4,v29 + lvx v25,$x10,$key_ # re-pre-load round[2] + vxor $in4,$twk4,v31 + + vncipher $out0,$out0,v30 + vncipher $out1,$out1,v30 + vncipher $out2,$out2,v30 + vncipher $out3,$out3,v30 + vncipher $out4,$out4,v30 + + vncipherlast $out0,$out0,$twk0 + vncipherlast $out1,$out1,$in1 + vncipherlast $out2,$out2,$in2 + vncipherlast $out3,$out3,$in3 + vncipherlast $out4,$out4,$in4 + mtctr $rounds + blr + .long 0 + .byte 0,12,0x14,0,0,0,0,0 +___ +}} }}} + +my $consts=1; +foreach(split("\n",$code)) { + s/\`([^\`]*)\`/eval($1)/geo; + + # constants table endian-specific conversion + if ($consts && m/\.(long|byte)\s+(.+)\s+(\?[a-z]*)$/o) { + my $conv=$3; + my @bytes=(); + + # convert to endian-agnostic format + if ($1 eq "long") { + foreach (split(/,\s*/,$2)) { + my $l = /^0/?oct:int; + push @bytes,($l>>24)&0xff,($l>>16)&0xff,($l>>8)&0xff,$l&0xff; + } + } else { + @bytes = map(/^0/?oct:int,split(/,\s*/,$2)); + } + + # little-endian conversion + if ($flavour =~ /le$/o) { + SWITCH: for($conv) { + /\?inv/ && do { @bytes=map($_^0xf,@bytes); last; }; + /\?rev/ && do { @bytes=reverse(@bytes); last; }; + } + } + + #emit + print ".byte\t",join(',',map (sprintf("0x%02x",$_),@bytes)),"\n"; + next; + } + $consts=0 if (m/Lconsts:/o); # end of table + + # instructions prefixed with '?' are endian-specific and need + # to be adjusted accordingly... + if ($flavour =~ /le$/o) { # little-endian + s/le\?//o or + s/be\?/#be#/o or + s/\?lvsr/lvsl/o or + s/\?lvsl/lvsr/o or + s/\?(vperm\s+v[0-9]+,\s*)(v[0-9]+,\s*)(v[0-9]+,\s*)(v[0-9]+)/$1$3$2$4/o or + s/\?(vsldoi\s+v[0-9]+,\s*)(v[0-9]+,)\s*(v[0-9]+,\s*)([0-9]+)/$1$3$2 16-$4/o or + s/\?(vspltw\s+v[0-9]+,\s*)(v[0-9]+,)\s*([0-9])/$1$2 3-$3/o; + } else { # big-endian + s/le\?/#le#/o or + s/be\?//o or + s/\?([a-z]+)/$1/o; + } + + print $_,"\n"; +} + +close STDOUT; diff --git a/cbits/asm/generate.sh b/cbits/asm/generate.sh --- a/cbits/asm/generate.sh +++ b/cbits/asm/generate.sh @@ -12,6 +12,8 @@ # arm/sha1-armv8.pl arm/sha512-armv8.pl # arm/keccak1600-armv8.pl arm/arm-xlate.pl # arm/arm_arch.h +# ppc/aesp8-ppc.pl ppc/ghashp8-ppc.pl +# ppc/ppc-xlate.pl # # The .pl files are the generator, not the product: each one emits # assembly for a given "flavour", which is the calling convention and the @@ -150,4 +152,25 @@ .section .note.GNU-stack,"",%progbits NOTE +done + +# POWER8's AES and GHASH, little-endian only. The generator emits a +# big-endian flavour from the same source, and crypton does not build for +# big-endian POWER: nothing here has been able to run that code, and an AES +# path that has never been executed is not one to check in. Adding +# "linux64" to the list below is what it would take. +# +# These two read no capability word of their own -- unlike the ARM and x86-64 +# modules above -- so the entry points are all that is renamed. +for flavour in linux64le; do + perl aesp8-ppc.pl $flavour tmp-$flavour.S + sed -e 's/\baes_p8_/crypton_aes_p8_/g' \ + tmp-$flavour.S > aesp8-ppc-$flavour.S + + perl ghashp8-ppc.pl $flavour tmp-$flavour.S + sed -e 's/\bgcm_init_p8/crypton_gcm_init_p8/g' \ + -e 's/\bgcm_gmult_p8/crypton_gcm_gmult_p8/g' \ + -e 's/\bgcm_ghash_p8/crypton_gcm_ghash_p8/g' \ + tmp-$flavour.S > ghashp8-ppc-$flavour.S + rm -f tmp-$flavour.S done diff --git a/cbits/asm/ghashp8-ppc-linux64le.S b/cbits/asm/ghashp8-ppc-linux64le.S new file mode 100644 --- /dev/null +++ b/cbits/asm/ghashp8-ppc-linux64le.S @@ -0,0 +1,575 @@ +.machine "any" + +.abiversion 2 +.text + +.globl crypton_gcm_init_p8 +.type crypton_gcm_init_p8,@function +.align 5 +crypton_gcm_init_p8: +.localentry crypton_gcm_init_p8,0 + + li 0,-4096 + li 8,0x10 + li 12,-1 + li 9,0x20 + or 0,0,0 + li 10,0x30 + .long 0x7D202699 + + vspltisb 8,-16 + vspltisb 5,1 + vaddubm 8,8,8 + vxor 4,4,4 + vor 8,8,5 + vsldoi 8,8,4,15 + vsldoi 6,4,5,1 + vaddubm 8,8,8 + vspltisb 7,7 + vor 8,8,6 + vspltb 6,9,0 + vsl 9,9,5 + vsrab 6,6,7 + vand 6,6,8 + vxor 3,9,6 + + vsldoi 9,3,3,8 + vsldoi 8,4,8,8 + vsldoi 11,4,9,8 + vsldoi 10,9,4,8 + + .long 0x7D001F99 + .long 0x7D681F99 + li 8,0x40 + .long 0x7D291F99 + li 9,0x50 + .long 0x7D4A1F99 + li 10,0x60 + + .long 0x10035CC8 + .long 0x10234CC8 + .long 0x104354C8 + + .long 0x10E044C8 + + vsldoi 5,1,4,8 + vsldoi 6,4,1,8 + vxor 0,0,5 + vxor 2,2,6 + + vsldoi 0,0,0,8 + vxor 0,0,7 + + vsldoi 6,0,0,8 + .long 0x100044C8 + vxor 6,6,2 + vxor 16,0,6 + + vsldoi 17,16,16,8 + vsldoi 19,4,17,8 + vsldoi 18,17,4,8 + + .long 0x7E681F99 + li 8,0x70 + .long 0x7E291F99 + li 9,0x80 + .long 0x7E4A1F99 + li 10,0x90 + .long 0x10039CC8 + .long 0x11B09CC8 + .long 0x10238CC8 + .long 0x11D08CC8 + .long 0x104394C8 + .long 0x11F094C8 + + .long 0x10E044C8 + .long 0x114D44C8 + + vsldoi 5,1,4,8 + vsldoi 6,4,1,8 + vsldoi 11,14,4,8 + vsldoi 9,4,14,8 + vxor 0,0,5 + vxor 2,2,6 + vxor 13,13,11 + vxor 15,15,9 + + vsldoi 0,0,0,8 + vsldoi 13,13,13,8 + vxor 0,0,7 + vxor 13,13,10 + + vsldoi 6,0,0,8 + vsldoi 9,13,13,8 + .long 0x100044C8 + .long 0x11AD44C8 + vxor 6,6,2 + vxor 9,9,15 + vxor 0,0,6 + vxor 13,13,9 + + vsldoi 9,0,0,8 + vsldoi 17,13,13,8 + vsldoi 11,4,9,8 + vsldoi 10,9,4,8 + vsldoi 19,4,17,8 + vsldoi 18,17,4,8 + + .long 0x7D681F99 + li 8,0xa0 + .long 0x7D291F99 + li 9,0xb0 + .long 0x7D4A1F99 + li 10,0xc0 + .long 0x7E681F99 + .long 0x7E291F99 + .long 0x7E4A1F99 + + or 12,12,12 + blr +.long 0 +.byte 0,12,0x14,0,0,0,2,0 +.long 0 +.size crypton_gcm_init_p8,.-crypton_gcm_init_p8 +.globl crypton_gcm_gmult_p8 +.type crypton_gcm_gmult_p8,@function +.align 5 +crypton_gcm_gmult_p8: +.localentry crypton_gcm_gmult_p8,0 + + lis 0,0xfff8 + li 8,0x10 + li 12,-1 + li 9,0x20 + or 0,0,0 + li 10,0x30 + .long 0x7C601E99 + + .long 0x7D682699 + lvsl 12,0,0 + .long 0x7D292699 + vspltisb 5,0x07 + .long 0x7D4A2699 + vxor 12,12,5 + .long 0x7D002699 + vperm 3,3,3,12 + vxor 4,4,4 + + .long 0x10035CC8 + .long 0x10234CC8 + .long 0x104354C8 + + .long 0x10E044C8 + + vsldoi 5,1,4,8 + vsldoi 6,4,1,8 + vxor 0,0,5 + vxor 2,2,6 + + vsldoi 0,0,0,8 + vxor 0,0,7 + + vsldoi 6,0,0,8 + .long 0x100044C8 + vxor 6,6,2 + vxor 0,0,6 + + vperm 0,0,0,12 + .long 0x7C001F99 + + or 12,12,12 + blr +.long 0 +.byte 0,12,0x14,0,0,0,2,0 +.long 0 +.size crypton_gcm_gmult_p8,.-crypton_gcm_gmult_p8 + +.globl crypton_gcm_ghash_p8 +.type crypton_gcm_ghash_p8,@function +.align 5 +crypton_gcm_ghash_p8: +.localentry crypton_gcm_ghash_p8,0 + + li 0,-4096 + li 8,0x10 + li 12,-1 + li 9,0x20 + or 0,0,0 + li 10,0x30 + .long 0x7C001E99 + + .long 0x7D682699 + li 8,0x40 + lvsl 12,0,0 + .long 0x7D292699 + li 9,0x50 + vspltisb 5,0x07 + .long 0x7D4A2699 + li 10,0x60 + vxor 12,12,5 + .long 0x7D002699 + vperm 0,0,0,12 + vxor 4,4,4 + + cmpldi 6,64 + bge .Lgcm_ghash_p8_4x + + .long 0x7C602E99 + addi 5,5,16 + subic. 6,6,16 + vperm 3,3,3,12 + vxor 3,3,0 + beq .Lshort + + .long 0x7E682699 + li 8,16 + .long 0x7E292699 + add 9,5,6 + .long 0x7E4A2699 + + +.align 5 +.Loop_2x: + .long 0x7E002E99 + vperm 16,16,16,12 + + subic 6,6,32 + .long 0x10039CC8 + .long 0x11B05CC8 + subfe 0,0,0 + .long 0x10238CC8 + .long 0x11D04CC8 + and 0,0,6 + .long 0x104394C8 + .long 0x11F054C8 + add 5,5,0 + + vxor 0,0,13 + vxor 1,1,14 + + .long 0x10E044C8 + + vsldoi 5,1,4,8 + vsldoi 6,4,1,8 + vxor 2,2,15 + vxor 0,0,5 + vxor 2,2,6 + + vsldoi 0,0,0,8 + vxor 0,0,7 + .long 0x7C682E99 + addi 5,5,32 + + vsldoi 6,0,0,8 + .long 0x100044C8 + vperm 3,3,3,12 + vxor 6,6,2 + vxor 3,3,6 + vxor 3,3,0 + cmpld 9,5 + bgt .Loop_2x + + cmplwi 6,0 + bne .Leven + +.Lshort: + .long 0x10035CC8 + .long 0x10234CC8 + .long 0x104354C8 + + .long 0x10E044C8 + + vsldoi 5,1,4,8 + vsldoi 6,4,1,8 + vxor 0,0,5 + vxor 2,2,6 + + vsldoi 0,0,0,8 + vxor 0,0,7 + + vsldoi 6,0,0,8 + .long 0x100044C8 + vxor 6,6,2 + +.Leven: + vxor 0,0,6 + vperm 0,0,0,12 + .long 0x7C001F99 + + or 12,12,12 + blr +.long 0 +.byte 0,12,0x14,0,0,0,4,0 +.long 0 +.align 5 +.crypton_gcm_ghash_p8_4x: +.Lgcm_ghash_p8_4x: + stdu 1,-256(1) + li 10,63 + li 11,79 + stvx 20,10,1 + addi 10,10,32 + stvx 21,11,1 + addi 11,11,32 + stvx 22,10,1 + addi 10,10,32 + stvx 23,11,1 + addi 11,11,32 + stvx 24,10,1 + addi 10,10,32 + stvx 25,11,1 + addi 11,11,32 + stvx 26,10,1 + addi 10,10,32 + stvx 27,11,1 + addi 11,11,32 + stvx 28,10,1 + addi 10,10,32 + stvx 29,11,1 + addi 11,11,32 + stvx 30,10,1 + li 10,0x60 + stvx 31,11,1 + li 0,-1 + stw 12,252(1) + or 0,0,0 + + lvsl 5,0,8 + + li 8,0x70 + .long 0x7E292699 + li 9,0x80 + vspltisb 6,8 + + li 10,0x90 + .long 0x7EE82699 + li 8,0xa0 + .long 0x7F092699 + li 9,0xb0 + .long 0x7F2A2699 + li 10,0xc0 + .long 0x7FA82699 + li 8,0x10 + .long 0x7FC92699 + li 9,0x20 + .long 0x7FEA2699 + li 10,0x30 + + vsldoi 7,4,6,8 + vaddubm 18,5,7 + vaddubm 19,6,18 + + srdi 6,6,4 + + .long 0x7C602E99 + .long 0x7E082E99 + subic. 6,6,8 + .long 0x7EC92E99 + .long 0x7F8A2E99 + addi 5,5,0x40 + vperm 3,3,3,12 + vperm 16,16,16,12 + vperm 22,22,22,12 + vperm 28,28,28,12 + + vxor 2,3,0 + + .long 0x11B0BCC8 + .long 0x11D0C4C8 + .long 0x11F0CCC8 + + vperm 11,17,9,18 + vperm 5,22,28,19 + vperm 10,17,9,19 + vperm 6,22,28,18 + .long 0x12B68CC8 + .long 0x12855CC8 + .long 0x137C4CC8 + .long 0x134654C8 + + vxor 21,21,14 + vxor 20,20,13 + vxor 27,27,21 + vxor 26,26,15 + + blt .Ltail_4x + +.Loop_4x: + .long 0x7C602E99 + .long 0x7E082E99 + subic. 6,6,4 + .long 0x7EC92E99 + .long 0x7F8A2E99 + addi 5,5,0x40 + vperm 16,16,16,12 + vperm 22,22,22,12 + vperm 28,28,28,12 + vperm 3,3,3,12 + + .long 0x1002ECC8 + .long 0x1022F4C8 + .long 0x1042FCC8 + .long 0x11B0BCC8 + .long 0x11D0C4C8 + .long 0x11F0CCC8 + + vxor 0,0,20 + vxor 1,1,27 + vxor 2,2,26 + vperm 5,22,28,19 + vperm 6,22,28,18 + + .long 0x10E044C8 + .long 0x12855CC8 + .long 0x134654C8 + + vsldoi 5,1,4,8 + vsldoi 6,4,1,8 + vxor 0,0,5 + vxor 2,2,6 + + vsldoi 0,0,0,8 + vxor 0,0,7 + + vsldoi 6,0,0,8 + .long 0x12B68CC8 + .long 0x137C4CC8 + .long 0x100044C8 + + vxor 20,20,13 + vxor 26,26,15 + vxor 2,2,3 + vxor 21,21,14 + vxor 2,2,6 + vxor 27,27,21 + vxor 2,2,0 + bge .Loop_4x + +.Ltail_4x: + .long 0x1002ECC8 + .long 0x1022F4C8 + .long 0x1042FCC8 + + vxor 0,0,20 + vxor 1,1,27 + + .long 0x10E044C8 + + vsldoi 5,1,4,8 + vsldoi 6,4,1,8 + vxor 2,2,26 + vxor 0,0,5 + vxor 2,2,6 + + vsldoi 0,0,0,8 + vxor 0,0,7 + + vsldoi 6,0,0,8 + .long 0x100044C8 + vxor 6,6,2 + vxor 0,0,6 + + addic. 6,6,4 + beq .Ldone_4x + + .long 0x7C602E99 + cmpldi 6,2 + li 6,-4 + blt .Lone + .long 0x7E082E99 + beq .Ltwo + +.Lthree: + .long 0x7EC92E99 + vperm 3,3,3,12 + vperm 16,16,16,12 + vperm 22,22,22,12 + + vxor 2,3,0 + vor 29,23,23 + vor 30,24,24 + vor 31,25,25 + + vperm 5,16,22,19 + vperm 6,16,22,18 + .long 0x12B08CC8 + .long 0x13764CC8 + .long 0x12855CC8 + .long 0x134654C8 + + vxor 27,27,21 + b .Ltail_4x + +.align 4 +.Ltwo: + vperm 3,3,3,12 + vperm 16,16,16,12 + + vxor 2,3,0 + vperm 5,4,16,19 + vperm 6,4,16,18 + + vsldoi 29,4,17,8 + vor 30,17,17 + vsldoi 31,17,4,8 + + .long 0x12855CC8 + .long 0x13704CC8 + .long 0x134654C8 + + b .Ltail_4x + +.align 4 +.Lone: + vperm 3,3,3,12 + + vsldoi 29,4,9,8 + vor 30,9,9 + vsldoi 31,9,4,8 + + vxor 2,3,0 + vxor 20,20,20 + vxor 27,27,27 + vxor 26,26,26 + + b .Ltail_4x + +.Ldone_4x: + vperm 0,0,0,12 + .long 0x7C001F99 + + li 10,63 + li 11,79 + or 12,12,12 + lvx 20,10,1 + addi 10,10,32 + lvx 21,11,1 + addi 11,11,32 + lvx 22,10,1 + addi 10,10,32 + lvx 23,11,1 + addi 11,11,32 + lvx 24,10,1 + addi 10,10,32 + lvx 25,11,1 + addi 11,11,32 + lvx 26,10,1 + addi 10,10,32 + lvx 27,11,1 + addi 11,11,32 + lvx 28,10,1 + addi 10,10,32 + lvx 29,11,1 + addi 11,11,32 + lvx 30,10,1 + lvx 31,11,1 + addi 1,1,256 + blr +.long 0 +.byte 0,12,0x04,0,0x80,0,4,0 +.long 0 +.size crypton_gcm_ghash_p8,.-crypton_gcm_ghash_p8 + +.byte 71,72,65,83,72,32,102,111,114,32,80,111,119,101,114,73,83,65,32,50,46,48,55,44,67,82,89,80,84,79,71,65,77,83,32,98,121,32,60,97,112,112,114,111,64,111,112,101,110,115,115,108,46,111,114,103,62,0 +.align 2 +.align 2 diff --git a/cbits/asm/ghashp8-ppc.pl b/cbits/asm/ghashp8-ppc.pl new file mode 100644 --- /dev/null +++ b/cbits/asm/ghashp8-ppc.pl @@ -0,0 +1,664 @@ +#!/usr/bin/env perl +# +# ==================================================================== +# Written by Andy Polyakov, @dot-asm, initially for use in the OpenSSL +# project. The module is dual licensed under OpenSSL and CRYPTOGAMS +# licenses depending on where you obtain it. For further details see +# https://github.com/dot-asm/cryptogams/. +# ==================================================================== +# +# GHASH for for PowerISA v2.07. +# +# July 2014 +# +# Accurate performance measurements are problematic, because it's +# always virtualized setup with possibly throttled processor. +# Relative comparison is therefore more informative. This initial +# version is ~2.1x slower than hardware-assisted AES-128-CTR, ~12x +# faster than "4-bit" integer-only compiler-generated 64-bit code. +# "Initial version" means that there is room for further improvement. + +# May 2016 +# +# 2x aggregated reduction improves performance by 50% (resulting +# performance on POWER8 is 1 cycle per processed byte), and 4x +# aggregated reduction - by 170% or 2.7x (resulting in 0.55 cpb). +# POWER9 delivers 0.51 cpb. + +$flavour=shift; +$output =shift; + +if ($flavour =~ /64/) { + $SIZE_T=8; + $LRSAVE=2*$SIZE_T; + $STU="stdu"; + $POP="ld"; + $PUSH="std"; + $UCMP="cmpld"; + $SHRI="srdi"; +} elsif ($flavour =~ /32/) { + $SIZE_T=4; + $LRSAVE=$SIZE_T; + $STU="stwu"; + $POP="lwz"; + $PUSH="stw"; + $UCMP="cmplw"; + $SHRI="srwi"; +} else { die "nonsense $flavour"; } + +$sp="r1"; +$FRAME=6*$SIZE_T+13*16; # 13*16 is for v20-v31 offload + +$0 =~ m/(.*[\/\\])[^\/\\]+$/; $dir=$1; +( $xlate="${dir}ppc-xlate.pl" and -f $xlate ) or +( $xlate="${dir}../../perlasm/ppc-xlate.pl" and -f $xlate) or +die "can't locate ppc-xlate.pl"; + +open STDOUT,"| $^X $xlate $flavour $output" || die "can't call $xlate: $!"; + +my ($Xip,$Htbl,$inp,$len)=map("r$_",(3..6)); # argument block + +my ($Xl,$Xm,$Xh,$IN)=map("v$_",(0..3)); +my ($zero,$t0,$t1,$t2,$xC2,$H,$Hh,$Hl,$lemask)=map("v$_",(4..12)); +my ($Xl1,$Xm1,$Xh1,$IN1,$H2,$H2h,$H2l)=map("v$_",(13..19)); +my $vrsave="r12"; + +$code=<<___; +.machine "any" + +.text + +.globl .gcm_init_p8 +.align 5 +.gcm_init_p8: + li r0,-4096 + li r8,0x10 + mfspr $vrsave,256 + li r9,0x20 + mtspr 256,r0 + li r10,0x30 + lvx_u $H,0,r4 # load H + + vspltisb $xC2,-16 # 0xf0 + vspltisb $t0,1 # one + vaddubm $xC2,$xC2,$xC2 # 0xe0 + vxor $zero,$zero,$zero + vor $xC2,$xC2,$t0 # 0xe1 + vsldoi $xC2,$xC2,$zero,15 # 0xe1... + vsldoi $t1,$zero,$t0,1 # ...1 + vaddubm $xC2,$xC2,$xC2 # 0xc2... + vspltisb $t2,7 + vor $xC2,$xC2,$t1 # 0xc2....01 + vspltb $t1,$H,0 # most significant byte + vsl $H,$H,$t0 # H<<=1 + vsrab $t1,$t1,$t2 # broadcast carry bit + vand $t1,$t1,$xC2 + vxor $IN,$H,$t1 # twisted H + + vsldoi $H,$IN,$IN,8 # twist even more ... + vsldoi $xC2,$zero,$xC2,8 # 0xc2.0 + vsldoi $Hl,$zero,$H,8 # ... and split + vsldoi $Hh,$H,$zero,8 + + stvx_u $xC2,0,r3 # save pre-computed table + stvx_u $Hl,r8,r3 + li r8,0x40 + stvx_u $H, r9,r3 + li r9,0x50 + stvx_u $Hh,r10,r3 + li r10,0x60 + + vpmsumd $Xl,$IN,$Hl # H.lo·H.lo + vpmsumd $Xm,$IN,$H # H.hi·H.lo+H.lo·H.hi + vpmsumd $Xh,$IN,$Hh # H.hi·H.hi + + vpmsumd $t2,$Xl,$xC2 # 1st reduction phase + + vsldoi $t0,$Xm,$zero,8 + vsldoi $t1,$zero,$Xm,8 + vxor $Xl,$Xl,$t0 + vxor $Xh,$Xh,$t1 + + vsldoi $Xl,$Xl,$Xl,8 + vxor $Xl,$Xl,$t2 + + vsldoi $t1,$Xl,$Xl,8 # 2nd reduction phase + vpmsumd $Xl,$Xl,$xC2 + vxor $t1,$t1,$Xh + vxor $IN1,$Xl,$t1 + + vsldoi $H2,$IN1,$IN1,8 + vsldoi $H2l,$zero,$H2,8 + vsldoi $H2h,$H2,$zero,8 + + stvx_u $H2l,r8,r3 # save H^2 + li r8,0x70 + stvx_u $H2,r9,r3 + li r9,0x80 + stvx_u $H2h,r10,r3 + li r10,0x90 +___ +{ +my ($t4,$t5,$t6) = ($Hl,$H,$Hh); +$code.=<<___; + vpmsumd $Xl,$IN,$H2l # H.lo·H^2.lo + vpmsumd $Xl1,$IN1,$H2l # H^2.lo·H^2.lo + vpmsumd $Xm,$IN,$H2 # H.hi·H^2.lo+H.lo·H^2.hi + vpmsumd $Xm1,$IN1,$H2 # H^2.hi·H^2.lo+H^2.lo·H^2.hi + vpmsumd $Xh,$IN,$H2h # H.hi·H^2.hi + vpmsumd $Xh1,$IN1,$H2h # H^2.hi·H^2.hi + + vpmsumd $t2,$Xl,$xC2 # 1st reduction phase + vpmsumd $t6,$Xl1,$xC2 # 1st reduction phase + + vsldoi $t0,$Xm,$zero,8 + vsldoi $t1,$zero,$Xm,8 + vsldoi $t4,$Xm1,$zero,8 + vsldoi $t5,$zero,$Xm1,8 + vxor $Xl,$Xl,$t0 + vxor $Xh,$Xh,$t1 + vxor $Xl1,$Xl1,$t4 + vxor $Xh1,$Xh1,$t5 + + vsldoi $Xl,$Xl,$Xl,8 + vsldoi $Xl1,$Xl1,$Xl1,8 + vxor $Xl,$Xl,$t2 + vxor $Xl1,$Xl1,$t6 + + vsldoi $t1,$Xl,$Xl,8 # 2nd reduction phase + vsldoi $t5,$Xl1,$Xl1,8 # 2nd reduction phase + vpmsumd $Xl,$Xl,$xC2 + vpmsumd $Xl1,$Xl1,$xC2 + vxor $t1,$t1,$Xh + vxor $t5,$t5,$Xh1 + vxor $Xl,$Xl,$t1 + vxor $Xl1,$Xl1,$t5 + + vsldoi $H,$Xl,$Xl,8 + vsldoi $H2,$Xl1,$Xl1,8 + vsldoi $Hl,$zero,$H,8 + vsldoi $Hh,$H,$zero,8 + vsldoi $H2l,$zero,$H2,8 + vsldoi $H2h,$H2,$zero,8 + + stvx_u $Hl,r8,r3 # save H^3 + li r8,0xa0 + stvx_u $H,r9,r3 + li r9,0xb0 + stvx_u $Hh,r10,r3 + li r10,0xc0 + stvx_u $H2l,r8,r3 # save H^4 + stvx_u $H2,r9,r3 + stvx_u $H2h,r10,r3 + + mtspr 256,$vrsave + blr + .long 0 + .byte 0,12,0x14,0,0,0,2,0 + .long 0 +.size .gcm_init_p8,.-.gcm_init_p8 +___ +} +$code.=<<___; +.globl .gcm_gmult_p8 +.align 5 +.gcm_gmult_p8: + lis r0,0xfff8 + li r8,0x10 + mfspr $vrsave,256 + li r9,0x20 + mtspr 256,r0 + li r10,0x30 + lvx_u $IN,0,$Xip # load Xi + + lvx_u $Hl,r8,$Htbl # load pre-computed table + le?lvsl $lemask,r0,r0 + lvx_u $H, r9,$Htbl + le?vspltisb $t0,0x07 + lvx_u $Hh,r10,$Htbl + le?vxor $lemask,$lemask,$t0 + lvx_u $xC2,0,$Htbl + le?vperm $IN,$IN,$IN,$lemask + vxor $zero,$zero,$zero + + vpmsumd $Xl,$IN,$Hl # H.lo·Xi.lo + vpmsumd $Xm,$IN,$H # H.hi·Xi.lo+H.lo·Xi.hi + vpmsumd $Xh,$IN,$Hh # H.hi·Xi.hi + + vpmsumd $t2,$Xl,$xC2 # 1st reduction phase + + vsldoi $t0,$Xm,$zero,8 + vsldoi $t1,$zero,$Xm,8 + vxor $Xl,$Xl,$t0 + vxor $Xh,$Xh,$t1 + + vsldoi $Xl,$Xl,$Xl,8 + vxor $Xl,$Xl,$t2 + + vsldoi $t1,$Xl,$Xl,8 # 2nd reduction phase + vpmsumd $Xl,$Xl,$xC2 + vxor $t1,$t1,$Xh + vxor $Xl,$Xl,$t1 + + le?vperm $Xl,$Xl,$Xl,$lemask + stvx_u $Xl,0,$Xip # write out Xi + + mtspr 256,$vrsave + blr + .long 0 + .byte 0,12,0x14,0,0,0,2,0 + .long 0 +.size .gcm_gmult_p8,.-.gcm_gmult_p8 + +.globl .gcm_ghash_p8 +.align 5 +.gcm_ghash_p8: + li r0,-4096 + li r8,0x10 + mfspr $vrsave,256 + li r9,0x20 + mtspr 256,r0 + li r10,0x30 + lvx_u $Xl,0,$Xip # load Xi + + lvx_u $Hl,r8,$Htbl # load pre-computed table + li r8,0x40 + le?lvsl $lemask,r0,r0 + lvx_u $H, r9,$Htbl + li r9,0x50 + le?vspltisb $t0,0x07 + lvx_u $Hh,r10,$Htbl + li r10,0x60 + le?vxor $lemask,$lemask,$t0 + lvx_u $xC2,0,$Htbl + le?vperm $Xl,$Xl,$Xl,$lemask + vxor $zero,$zero,$zero + + ${UCMP}i $len,64 + bge Lgcm_ghash_p8_4x + + lvx_u $IN,0,$inp + addi $inp,$inp,16 + subic. $len,$len,16 + le?vperm $IN,$IN,$IN,$lemask + vxor $IN,$IN,$Xl + beq Lshort + + lvx_u $H2l,r8,$Htbl # load H^2 + li r8,16 + lvx_u $H2, r9,$Htbl + add r9,$inp,$len # end of input + lvx_u $H2h,r10,$Htbl + be?b Loop_2x + +.align 5 +Loop_2x: + lvx_u $IN1,0,$inp + le?vperm $IN1,$IN1,$IN1,$lemask + + subic $len,$len,32 + vpmsumd $Xl,$IN,$H2l # H^2.lo·Xi.lo + vpmsumd $Xl1,$IN1,$Hl # H.lo·Xi+1.lo + subfe r0,r0,r0 # borrow?-1:0 + vpmsumd $Xm,$IN,$H2 # H^2.hi·Xi.lo+H^2.lo·Xi.hi + vpmsumd $Xm1,$IN1,$H # H.hi·Xi+1.lo+H.lo·Xi+1.hi + and r0,r0,$len + vpmsumd $Xh,$IN,$H2h # H^2.hi·Xi.hi + vpmsumd $Xh1,$IN1,$Hh # H.hi·Xi+1.hi + add $inp,$inp,r0 + + vxor $Xl,$Xl,$Xl1 + vxor $Xm,$Xm,$Xm1 + + vpmsumd $t2,$Xl,$xC2 # 1st reduction phase + + vsldoi $t0,$Xm,$zero,8 + vsldoi $t1,$zero,$Xm,8 + vxor $Xh,$Xh,$Xh1 + vxor $Xl,$Xl,$t0 + vxor $Xh,$Xh,$t1 + + vsldoi $Xl,$Xl,$Xl,8 + vxor $Xl,$Xl,$t2 + lvx_u $IN,r8,$inp + addi $inp,$inp,32 + + vsldoi $t1,$Xl,$Xl,8 # 2nd reduction phase + vpmsumd $Xl,$Xl,$xC2 + le?vperm $IN,$IN,$IN,$lemask + vxor $t1,$t1,$Xh + vxor $IN,$IN,$t1 + vxor $IN,$IN,$Xl + $UCMP r9,$inp + bgt Loop_2x # done yet? + + cmplwi $len,0 + bne Leven + +Lshort: + vpmsumd $Xl,$IN,$Hl # H.lo·Xi.lo + vpmsumd $Xm,$IN,$H # H.hi·Xi.lo+H.lo·Xi.hi + vpmsumd $Xh,$IN,$Hh # H.hi·Xi.hi + + vpmsumd $t2,$Xl,$xC2 # 1st reduction phase + + vsldoi $t0,$Xm,$zero,8 + vsldoi $t1,$zero,$Xm,8 + vxor $Xl,$Xl,$t0 + vxor $Xh,$Xh,$t1 + + vsldoi $Xl,$Xl,$Xl,8 + vxor $Xl,$Xl,$t2 + + vsldoi $t1,$Xl,$Xl,8 # 2nd reduction phase + vpmsumd $Xl,$Xl,$xC2 + vxor $t1,$t1,$Xh + +Leven: + vxor $Xl,$Xl,$t1 + le?vperm $Xl,$Xl,$Xl,$lemask + stvx_u $Xl,0,$Xip # write out Xi + + mtspr 256,$vrsave + blr + .long 0 + .byte 0,12,0x14,0,0,0,4,0 + .long 0 +___ +{ +my ($Xl3,$Xm2,$IN2,$H3l,$H3,$H3h, + $Xh3,$Xm3,$IN3,$H4l,$H4,$H4h) = map("v$_",(20..31)); +my $IN0=$IN; +my ($H21l,$H21h,$loperm,$hiperm) = ($Hl,$Hh,$H2l,$H2h); + +$code.=<<___; +.align 5 +.gcm_ghash_p8_4x: +Lgcm_ghash_p8_4x: + $STU $sp,-$FRAME($sp) + li r10,`15+6*$SIZE_T` + li r11,`31+6*$SIZE_T` + stvx v20,r10,$sp + addi r10,r10,32 + stvx v21,r11,$sp + addi r11,r11,32 + stvx v22,r10,$sp + addi r10,r10,32 + stvx v23,r11,$sp + addi r11,r11,32 + stvx v24,r10,$sp + addi r10,r10,32 + stvx v25,r11,$sp + addi r11,r11,32 + stvx v26,r10,$sp + addi r10,r10,32 + stvx v27,r11,$sp + addi r11,r11,32 + stvx v28,r10,$sp + addi r10,r10,32 + stvx v29,r11,$sp + addi r11,r11,32 + stvx v30,r10,$sp + li r10,0x60 + stvx v31,r11,$sp + li r0,-1 + stw $vrsave,`$FRAME-4`($sp) # save vrsave + mtspr 256,r0 # preserve all AltiVec registers + + lvsl $t0,0,r8 # 0x0001..0e0f + #lvx_u $H2l,r8,$Htbl # load H^2 + li r8,0x70 + lvx_u $H2, r9,$Htbl + li r9,0x80 + vspltisb $t1,8 # 0x0808..0808 + #lvx_u $H2h,r10,$Htbl + li r10,0x90 + lvx_u $H3l,r8,$Htbl # load H^3 + li r8,0xa0 + lvx_u $H3, r9,$Htbl + li r9,0xb0 + lvx_u $H3h,r10,$Htbl + li r10,0xc0 + lvx_u $H4l,r8,$Htbl # load H^4 + li r8,0x10 + lvx_u $H4, r9,$Htbl + li r9,0x20 + lvx_u $H4h,r10,$Htbl + li r10,0x30 + + vsldoi $t2,$zero,$t1,8 # 0x0000..0808 + vaddubm $hiperm,$t0,$t2 # 0x0001..1617 + vaddubm $loperm,$t1,$hiperm # 0x0809..1e1f + + $SHRI $len,$len,4 # this allows to use sign bit + # as carry + lvx_u $IN0,0,$inp # load input + lvx_u $IN1,r8,$inp + subic. $len,$len,8 + lvx_u $IN2,r9,$inp + lvx_u $IN3,r10,$inp + addi $inp,$inp,0x40 + le?vperm $IN0,$IN0,$IN0,$lemask + le?vperm $IN1,$IN1,$IN1,$lemask + le?vperm $IN2,$IN2,$IN2,$lemask + le?vperm $IN3,$IN3,$IN3,$lemask + + vxor $Xh,$IN0,$Xl + + vpmsumd $Xl1,$IN1,$H3l + vpmsumd $Xm1,$IN1,$H3 + vpmsumd $Xh1,$IN1,$H3h + + vperm $H21l,$H2,$H,$hiperm + vperm $t0,$IN2,$IN3,$loperm + vperm $H21h,$H2,$H,$loperm + vperm $t1,$IN2,$IN3,$hiperm + vpmsumd $Xm2,$IN2,$H2 # H^2.lo·Xi+2.hi+H^2.hi·Xi+2.lo + vpmsumd $Xl3,$t0,$H21l # H^2.lo·Xi+2.lo+H.lo·Xi+3.lo + vpmsumd $Xm3,$IN3,$H # H.hi·Xi+3.lo +H.lo·Xi+3.hi + vpmsumd $Xh3,$t1,$H21h # H^2.hi·Xi+2.hi+H.hi·Xi+3.hi + + vxor $Xm2,$Xm2,$Xm1 + vxor $Xl3,$Xl3,$Xl1 + vxor $Xm3,$Xm3,$Xm2 + vxor $Xh3,$Xh3,$Xh1 + + blt Ltail_4x + +Loop_4x: + lvx_u $IN0,0,$inp + lvx_u $IN1,r8,$inp + subic. $len,$len,4 + lvx_u $IN2,r9,$inp + lvx_u $IN3,r10,$inp + addi $inp,$inp,0x40 + le?vperm $IN1,$IN1,$IN1,$lemask + le?vperm $IN2,$IN2,$IN2,$lemask + le?vperm $IN3,$IN3,$IN3,$lemask + le?vperm $IN0,$IN0,$IN0,$lemask + + vpmsumd $Xl,$Xh,$H4l # H^4.lo·Xi.lo + vpmsumd $Xm,$Xh,$H4 # H^4.hi·Xi.lo+H^4.lo·Xi.hi + vpmsumd $Xh,$Xh,$H4h # H^4.hi·Xi.hi + vpmsumd $Xl1,$IN1,$H3l + vpmsumd $Xm1,$IN1,$H3 + vpmsumd $Xh1,$IN1,$H3h + + vxor $Xl,$Xl,$Xl3 + vxor $Xm,$Xm,$Xm3 + vxor $Xh,$Xh,$Xh3 + vperm $t0,$IN2,$IN3,$loperm + vperm $t1,$IN2,$IN3,$hiperm + + vpmsumd $t2,$Xl,$xC2 # 1st reduction phase + vpmsumd $Xl3,$t0,$H21l # H.lo·Xi+3.lo +H^2.lo·Xi+2.lo + vpmsumd $Xh3,$t1,$H21h # H.hi·Xi+3.hi +H^2.hi·Xi+2.hi + + vsldoi $t0,$Xm,$zero,8 + vsldoi $t1,$zero,$Xm,8 + vxor $Xl,$Xl,$t0 + vxor $Xh,$Xh,$t1 + + vsldoi $Xl,$Xl,$Xl,8 + vxor $Xl,$Xl,$t2 + + vsldoi $t1,$Xl,$Xl,8 # 2nd reduction phase + vpmsumd $Xm2,$IN2,$H2 # H^2.hi·Xi+2.lo+H^2.lo·Xi+2.hi + vpmsumd $Xm3,$IN3,$H # H.hi·Xi+3.lo +H.lo·Xi+3.hi + vpmsumd $Xl,$Xl,$xC2 + + vxor $Xl3,$Xl3,$Xl1 + vxor $Xh3,$Xh3,$Xh1 + vxor $Xh,$Xh,$IN0 + vxor $Xm2,$Xm2,$Xm1 + vxor $Xh,$Xh,$t1 + vxor $Xm3,$Xm3,$Xm2 + vxor $Xh,$Xh,$Xl + bge Loop_4x + +Ltail_4x: + vpmsumd $Xl,$Xh,$H4l # H^4.lo·Xi.lo + vpmsumd $Xm,$Xh,$H4 # H^4.hi·Xi.lo+H^4.lo·Xi.hi + vpmsumd $Xh,$Xh,$H4h # H^4.hi·Xi.hi + + vxor $Xl,$Xl,$Xl3 + vxor $Xm,$Xm,$Xm3 + + vpmsumd $t2,$Xl,$xC2 # 1st reduction phase + + vsldoi $t0,$Xm,$zero,8 + vsldoi $t1,$zero,$Xm,8 + vxor $Xh,$Xh,$Xh3 + vxor $Xl,$Xl,$t0 + vxor $Xh,$Xh,$t1 + + vsldoi $Xl,$Xl,$Xl,8 + vxor $Xl,$Xl,$t2 + + vsldoi $t1,$Xl,$Xl,8 # 2nd reduction phase + vpmsumd $Xl,$Xl,$xC2 + vxor $t1,$t1,$Xh + vxor $Xl,$Xl,$t1 + + addic. $len,$len,4 + beq Ldone_4x + + lvx_u $IN0,0,$inp + ${UCMP}i $len,2 + li $len,-4 + blt Lone + lvx_u $IN1,r8,$inp + beq Ltwo + +Lthree: + lvx_u $IN2,r9,$inp + le?vperm $IN0,$IN0,$IN0,$lemask + le?vperm $IN1,$IN1,$IN1,$lemask + le?vperm $IN2,$IN2,$IN2,$lemask + + vxor $Xh,$IN0,$Xl + vmr $H4l,$H3l + vmr $H4, $H3 + vmr $H4h,$H3h + + vperm $t0,$IN1,$IN2,$loperm + vperm $t1,$IN1,$IN2,$hiperm + vpmsumd $Xm2,$IN1,$H2 # H^2.lo·Xi+1.hi+H^2.hi·Xi+1.lo + vpmsumd $Xm3,$IN2,$H # H.hi·Xi+2.lo +H.lo·Xi+2.hi + vpmsumd $Xl3,$t0,$H21l # H^2.lo·Xi+1.lo+H.lo·Xi+2.lo + vpmsumd $Xh3,$t1,$H21h # H^2.hi·Xi+1.hi+H.hi·Xi+2.hi + + vxor $Xm3,$Xm3,$Xm2 + b Ltail_4x + +.align 4 +Ltwo: + le?vperm $IN0,$IN0,$IN0,$lemask + le?vperm $IN1,$IN1,$IN1,$lemask + + vxor $Xh,$IN0,$Xl + vperm $t0,$zero,$IN1,$loperm + vperm $t1,$zero,$IN1,$hiperm + + vsldoi $H4l,$zero,$H2,8 + vmr $H4, $H2 + vsldoi $H4h,$H2,$zero,8 + + vpmsumd $Xl3,$t0, $H21l # H.lo·Xi+1.lo + vpmsumd $Xm3,$IN1,$H # H.hi·Xi+1.lo+H.lo·Xi+2.hi + vpmsumd $Xh3,$t1, $H21h # H.hi·Xi+1.hi + + b Ltail_4x + +.align 4 +Lone: + le?vperm $IN0,$IN0,$IN0,$lemask + + vsldoi $H4l,$zero,$H,8 + vmr $H4, $H + vsldoi $H4h,$H,$zero,8 + + vxor $Xh,$IN0,$Xl + vxor $Xl3,$Xl3,$Xl3 + vxor $Xm3,$Xm3,$Xm3 + vxor $Xh3,$Xh3,$Xh3 + + b Ltail_4x + +Ldone_4x: + le?vperm $Xl,$Xl,$Xl,$lemask + stvx_u $Xl,0,$Xip # write out Xi + + li r10,`15+6*$SIZE_T` + li r11,`31+6*$SIZE_T` + mtspr 256,$vrsave + lvx v20,r10,$sp + addi r10,r10,32 + lvx v21,r11,$sp + addi r11,r11,32 + lvx v22,r10,$sp + addi r10,r10,32 + lvx v23,r11,$sp + addi r11,r11,32 + lvx v24,r10,$sp + addi r10,r10,32 + lvx v25,r11,$sp + addi r11,r11,32 + lvx v26,r10,$sp + addi r10,r10,32 + lvx v27,r11,$sp + addi r11,r11,32 + lvx v28,r10,$sp + addi r10,r10,32 + lvx v29,r11,$sp + addi r11,r11,32 + lvx v30,r10,$sp + lvx v31,r11,$sp + addi $sp,$sp,$FRAME + blr + .long 0 + .byte 0,12,0x04,0,0x80,0,4,0 + .long 0 +___ +} +$code.=<<___; +.size .gcm_ghash_p8,.-.gcm_ghash_p8 + +.asciz "GHASH for PowerISA 2.07, CRYPTOGAMS by " +.align 2 +___ + +foreach (split("\n",$code)) { + s/\`([^\`]*)\`/eval $1/geo; + + if ($flavour =~ /le$/o) { # little-endian + s/le\?//o or + s/be\?/#be#/o; + } else { + s/le\?/#le#/o or + s/be\?//o; + } + print $_,"\n"; +} + +close STDOUT; # enforce flush diff --git a/cbits/asm/ppc-xlate.pl b/cbits/asm/ppc-xlate.pl new file mode 100644 --- /dev/null +++ b/cbits/asm/ppc-xlate.pl @@ -0,0 +1,352 @@ +#!/usr/bin/env perl + +# PowerPC assembler distiller by \@dot-asm. + +################################################################ +# Recognized "flavour"-s are: +# +# linux{32|64}[le] GNU assembler and ELF symbol decorations, +# with little-endian option +# linux64v2 GNU asssembler and big-endian instantiation +# of latest ELF specification +# aix{32|64} AIX assembler and symbol decorations +# osx{32|64} Mac OS X assembler and symbol decoratons + +my $flavour = shift; +my $output = shift; +open STDOUT,">$output" || die "can't open $output: $!"; + +my %GLOBALS; +my %TYPES; +my $dotinlocallabels=($flavour=~/linux/)?1:0; + +################################################################ +# directives which need special treatment on different platforms +################################################################ +my $type = sub { + my ($dir,$name,$type) = @_; + + $TYPES{$name} = $type; + if ($flavour =~ /linux/) { + $name =~ s|^\.||; + ".type $name,$type"; + } else { + ""; + } +}; +my $globl = sub { + my $junk = shift; + my $name = shift; + my $global = \$GLOBALS{$name}; + my $type = \$TYPES{$name}; + my $ret; + + $name =~ s|^\.||; + + SWITCH: for ($flavour) { + /aix/ && do { if (!$$type) { + $$type = "\@function"; + } + if ($$type =~ /function/) { + $name = ".$name"; + } + last; + }; + /osx/ && do { $name = "_$name"; + last; + }; + /linux.*(32|64(le|v2))/ + && do { $ret .= ".globl $name"; + if (!$$type) { + $ret .= "\n.type $name,\@function"; + $$type = "\@function"; + } + last; + }; + /linux.*64/ && do { $ret .= ".globl $name"; + if (!$$type) { + $ret .= "\n.type $name,\@function"; + $$type = "\@function"; + } + if ($$type =~ /function/) { + $ret .= "\n.section \".opd\",\"aw\""; + $ret .= "\n.align 3"; + $ret .= "\n$name:"; + $ret .= "\n.quad .$name,.TOC.\@tocbase,0"; + $ret .= "\n.previous"; + $name = ".$name"; + } + last; + }; + } + + $ret = ".globl $name" if (!$ret); + $$global = $name; + $ret; +}; +my $text = sub { + my $ret = ($flavour =~ /aix/) ? ".csect\t.text[PR],7" : ".text"; + $ret = ".abiversion 2\n".$ret if ($flavour =~ /linux.*64(le|v2)/); + $ret; +}; +my $machine = sub { + my $junk = shift; + my $arch = shift; + if ($flavour =~ /osx/) + { $arch =~ s/\"//g; + $arch = ($flavour=~/64/) ? "ppc970-64" : "ppc970" if ($arch eq "any"); + } + ".machine $arch"; +}; +my $size = sub { + if ($flavour =~ /linux/) + { shift; + my $name = shift; + my $real = $GLOBALS{$name} ? \$GLOBALS{$name} : \$name; + my $ret = ".size $$real,.-$$real"; + $name =~ s|^\.||; + if ($$real ne $name) { + $ret .= "\n.size $name,.-$$real"; + } + $ret; + } + else + { ""; } +}; +my $asciz = sub { + shift; + my $line = join(",",@_); + if ($line =~ /^"(.*)"$/) + { ".byte " . join(",",unpack("C*",$1),0) . "\n.align 2"; } + else + { ""; } +}; +my $quad = sub { + shift; + my @ret; + my ($hi,$lo); + for (@_) { + if (/^0x([0-9a-f]*?)([0-9a-f]{1,8})$/io) + { $hi=$1?"0x$1":"0"; $lo="0x$2"; } + elsif (/^([0-9]+)$/o) + { $hi=$1>>32; $lo=$1&0xffffffff; } # error-prone with 32-bit perl + else + { $hi=undef; $lo=$_; } + + if (defined($hi)) + { push(@ret,$flavour=~/le$/o?".long\t$lo,$hi":".long\t$hi,$lo"); } + else + { push(@ret,".quad $lo"); } + } + join("\n",@ret); +}; + +################################################################ +# simplified mnemonics not handled by at least one assembler +################################################################ +my $cmplw = sub { + my $f = shift; + my $cr = 0; $cr = shift if ($#_>1); + # Some out-of-date 32-bit GNU assembler just can't handle cmplw... + ($flavour =~ /linux.*32/) ? + " .long ".sprintf "0x%x",31<<26|$cr<<23|$_[0]<<16|$_[1]<<11|64 : + " cmplw ".join(',',$cr,@_); +}; +my $bdnz = sub { + my $f = shift; + my $bo = $f=~/[\+\-]/ ? 16+9 : 16; # optional "to be taken" hint + " bc $bo,0,".shift; +} if ($flavour!~/linux/); +my $bltlr = sub { + my $f = shift; + my $bo = $f=~/\-/ ? 12+2 : 12; # optional "not to be taken" hint + ($flavour =~ /linux/) ? # GNU as doesn't allow most recent hints + " .long ".sprintf "0x%x",19<<26|$bo<<21|16<<1 : + " bclr $bo,0"; +}; +my $bnelr = sub { + my $f = shift; + my $bo = $f=~/\-/ ? 4+2 : 4; # optional "not to be taken" hint + ($flavour =~ /linux/) ? # GNU as doesn't allow most recent hints + " .long ".sprintf "0x%x",19<<26|$bo<<21|2<<16|16<<1 : + " bclr $bo,2"; +}; +my $beqlr = sub { + my $f = shift; + my $bo = $f=~/-/ ? 12+2 : 12; # optional "not to be taken" hint + ($flavour =~ /linux/) ? # GNU as doesn't allow most recent hints + " .long ".sprintf "0x%X",19<<26|$bo<<21|2<<16|16<<1 : + " bclr $bo,2"; +}; +# GNU assembler can't handle extrdi rA,rS,16,48, or when sum of last two +# arguments is 64, with "operand out of range" error. +my $extrdi = sub { + my ($f,$ra,$rs,$n,$b) = @_; + $b = ($b+$n)&63; $n = 64-$n; + " rldicl $ra,$rs,$b,$n"; +}; +my $vmr = sub { + my ($f,$vx,$vy) = @_; + " vor $vx,$vy,$vy"; +}; + +# Some ABIs specify vrsave, special-purpose register #256, as reserved +# for system use. +my $no_vrsave = ($flavour =~ /aix|linux64(le|v2)/); +my $mtspr = sub { + my ($f,$idx,$ra) = @_; + if ($idx == 256 && $no_vrsave) { + " or $ra,$ra,$ra"; + } else { + " mtspr $idx,$ra"; + } +}; +my $mfspr = sub { + my ($f,$rd,$idx) = @_; + if ($idx == 256 && $no_vrsave) { + " li $rd,-1"; + } else { + " mfspr $rd,$idx"; + } +}; + +# PowerISA 2.06 stuff +sub vsxmem_op { + my ($f, $vrt, $ra, $rb, $op) = @_; + " .long ".sprintf "0x%X",(31<<26)|($vrt<<21)|($ra<<16)|($rb<<11)|($op*2+1); +} +# made-up unaligned memory reference AltiVec/VMX instructions +my $lvx_u = sub { vsxmem_op(@_, 844); }; # lxvd2x +my $stvx_u = sub { vsxmem_op(@_, 972); }; # stxvd2x +my $lvdx_u = sub { vsxmem_op(@_, 588); }; # lxsdx +my $stvdx_u = sub { vsxmem_op(@_, 716); }; # stxsdx +my $lvx_4w = sub { vsxmem_op(@_, 780); }; # lxvw4x +my $stvx_4w = sub { vsxmem_op(@_, 908); }; # stxvw4x +my $lvx_splt = sub { vsxmem_op(@_, 332); }; # lxvdsx +# VSX instruction[s] masqueraded as made-up AltiVec/VMX +my $vpermdi = sub { # xxpermdi + my ($f, $vrt, $vra, $vrb, $dm) = @_; + $dm = oct($dm) if ($dm =~ /^0/); + " .long ".sprintf "0x%X",(60<<26)|($vrt<<21)|($vra<<16)|($vrb<<11)|($dm<<8)|(10<<3)|7; +}; + +# PowerISA 2.07 stuff +sub vcrypto_op { + my ($f, $vrt, $vra, $vrb, $op) = @_; + " .long ".sprintf "0x%X",(4<<26)|($vrt<<21)|($vra<<16)|($vrb<<11)|$op; +} +sub vfour { + my ($f, $vrt, $vra, $vrb, $vrc, $op) = @_; + " .long ".sprintf "0x%X",(4<<26)|($vrt<<21)|($vra<<16)|($vrb<<11)|($vrc<<6)|$op; +}; +my $vcipher = sub { vcrypto_op(@_, 1288); }; +my $vcipherlast = sub { vcrypto_op(@_, 1289); }; +my $vncipher = sub { vcrypto_op(@_, 1352); }; +my $vncipherlast= sub { vcrypto_op(@_, 1353); }; +my $vsbox = sub { vcrypto_op(@_, 0, 1480); }; +my $vshasigmad = sub { my ($st,$six)=splice(@_,-2); vcrypto_op(@_, $st<<4|$six, 1730); }; +my $vshasigmaw = sub { my ($st,$six)=splice(@_,-2); vcrypto_op(@_, $st<<4|$six, 1666); }; +my $vpmsumb = sub { vcrypto_op(@_, 1032); }; +my $vpmsumd = sub { vcrypto_op(@_, 1224); }; +my $vpmsubh = sub { vcrypto_op(@_, 1096); }; +my $vpmsumw = sub { vcrypto_op(@_, 1160); }; +# These are not really crypto, but vcrypto_op template works +my $vaddudm = sub { vcrypto_op(@_, 192); }; +my $vadduqm = sub { vcrypto_op(@_, 256); }; +my $vmuleuw = sub { vcrypto_op(@_, 648); }; +my $vmulouw = sub { vcrypto_op(@_, 136); }; +my $vrld = sub { vcrypto_op(@_, 196); }; +my $vsld = sub { vcrypto_op(@_, 1476); }; +my $vsrd = sub { vcrypto_op(@_, 1732); }; +my $vsubudm = sub { vcrypto_op(@_, 1216); }; +my $vaddcuq = sub { vcrypto_op(@_, 320); }; +my $vaddeuqm = sub { vfour(@_,60); }; +my $vaddecuq = sub { vfour(@_,61); }; +my $vmrgew = sub { vfour(@_,0,1932); }; +my $vmrgow = sub { vfour(@_,0,1676); }; + +my $mtsle = sub { + my ($f, $arg) = @_; + " .long ".sprintf "0x%X",(31<<26)|($arg<<21)|(147*2); +}; + +# VSX instructions masqueraded as AltiVec/VMX +my $mtvrd = sub { + my ($f, $vrt, $ra) = @_; + " .long ".sprintf "0x%X",(31<<26)|($vrt<<21)|($ra<<16)|(179<<1)|1; +}; +my $mtvrwz = sub { + my ($f, $vrt, $ra) = @_; + " .long ".sprintf "0x%X",(31<<26)|($vrt<<21)|($ra<<16)|(243<<1)|1; +}; +my $lvwzx_u = sub { vsxmem_op(@_, 12); }; # lxsiwzx +my $stvwx_u = sub { vsxmem_op(@_, 140); }; # stxsiwx + +# PowerISA 3.0 stuff +my $maddhdu = sub { vfour(@_,49); }; +my $maddld = sub { vfour(@_,51); }; +my $darn = sub { + my ($f, $rt, $l) = @_; + " .long ".sprintf "0x%X",(31<<26)|($rt<<21)|($l<<16)|(755<<1); +}; +my $iseleq = sub { + my ($f, $rt, $ra, $rb) = @_; + " .long ".sprintf "0x%X",(31<<26)|($rt<<21)|($ra<<16)|($rb<<11)|(2<<6)|30; +}; +# VSX instruction[s] masqueraded as made-up AltiVec/VMX +my $vspltib = sub { # xxspltib + my ($f, $vrt, $imm8) = @_; + $imm8 = oct($imm8) if ($imm8 =~ /^0/); + $imm8 &= 0xff; + " .long ".sprintf "0x%X",(60<<26)|($vrt<<21)|($imm8<<11)|(360<<1)|1; +}; + +# PowerISA 3.0B stuff +my $addex = sub { + my ($f, $rt, $ra, $rb, $cy) = @_; # only cy==0 is specified in 3.0B + " .long ".sprintf "0x%X",(31<<26)|($rt<<21)|($ra<<16)|($rb<<11)|($cy<<9)|(170<<1); +}; +my $vmsumudm = sub { vfour(@_,35); }; + +while($line=<>) { + + $line =~ s|[#!;].*$||; # get rid of asm-style comments... + $line =~ s|/\*.*\*/||; # ... and C-style comments... + $line =~ s|^\s+||; # ... and skip white spaces in beginning... + $line =~ s|\s+$||; # ... and at the end + + { + $line =~ s|\.L(\w+)|L$1|g; # common denominator for Locallabel + $line =~ s|\bL(\w+)|\.L$1|g if ($dotinlocallabels); + } + + { + $line =~ s|(^[\.\w]+)\:\s*||; + my $label = $1; + if ($label) { + my $xlated = ($GLOBALS{$label} or $label); + print "$xlated:"; + if ($flavour =~ /linux.*64(le|v2)/) { + if ($TYPES{$label} =~ /function/) { + printf "\n.localentry %s,0\n",$xlated; + } + } + } + } + + { + $line =~ s|^\s*(\.?)(\w+)([\.\+\-]?)\s*||; + my $c = $1; $c = "\t" if ($c eq ""); + my $mnemonic = $2; + my $f = $3; + my $opcode = eval("\$$mnemonic"); + $line =~ s/\b(c?[rf]|v|vs)([0-9]+)\b/$2/g if ($c ne "." and $flavour !~ /osx/); + if (ref($opcode) eq 'CODE') { $line = &$opcode($f,split(/,\s*/,$line)); } + elsif ($mnemonic) { $line = $c.$mnemonic.$f."\t".$line; } + } + + print $line if ($line); + print "\n"; +} + +close STDOUT; diff --git a/cbits/bearssl/LICENSE b/cbits/bearssl/LICENSE new file mode 100644 --- /dev/null +++ b/cbits/bearssl/LICENSE @@ -0,0 +1,21 @@ +Copyright (c) 2016 Thomas Pornin + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS +BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/cbits/bearssl/README.md b/cbits/bearssl/README.md new file mode 100644 --- /dev/null +++ b/cbits/bearssl/README.md @@ -0,0 +1,57 @@ +# BearSSL + +Constant-time AES and GHASH from [BearSSL](https://bearssl.org/), vendored +here. `VERSION` holds the upstream release these files came from, and +`import.sh` fetches them again. + +## Why + +crypton's portable AES and GHASH are table-driven, and both index their +tables with a secret: `cbits/aes/generic.c` with a byte of the state, in +every round and in the key expansion, and `cbits/aes/gf.c` with a nibble of +the GHASH accumulator. Both are variable-time by construction, and the +cache-timing attacks on that shape of code are old and well documented. + +The processors that have AES instructions do not run any of it -- crypton +asks them first. What runs it is everything else: ppc64le and s390x, where +the instructions exist but crypton has no path to them; 32-bit ARM, where +the same is true; riscv64 and loongarch64; and the boards that genuinely +have no AES instructions at all, of which the Raspberry Pi 3 and 4 are by +some distance the largest population. + +BearSSL's answer is bitslicing. `aes_ct64` holds four blocks interleaved +across eight 64-bit words and computes the S-box as boolean algebra, so +there is no table and no address derived from a secret; `ghash_ctmul64` +builds the GF(2^128) multiply out of shifts, masks and integer multiplies +rather than a table of H. Both are plain C99 and assume nothing beyond +`uint64_t`. + +## What is here + +| file | from | +| --- | --- | +| `aes_ct64.c` | `src/symcipher/aes_ct64.c` | +| `aes_ct64_enc.c` | `src/symcipher/aes_ct64_enc.c` | +| `aes_ct64_dec.c` | `src/symcipher/aes_ct64_dec.c` | +| `ghash_ctmul64.c` | `src/hash/ghash_ctmul64.c` | +| `dec32le.c` | `src/codec/dec32le.c`, for `br_range_dec32le` | +| `LICENSE` | `LICENSE.txt` -- MIT, (c) 2016 Thomas Pornin | + +`inner.h` is **crypton's, not upstream's**. Each of those five opens with +`#include "inner.h"`, and upstream's is some two thousand lines declaring +the whole library; these five want six things from it. So this one gives +those six and nothing else, which is what lets the five stay byte for byte +what upstream ships. `import.sh` does not overwrite it. + +The byte-order helpers in it are written rather than copied, in their plain +portable form without upstream's unaligned-access fast paths, so that no +platform configuration comes with them. + +## Keeping it honest + +`cbits/tests/bearssl_diff.c` checks this against the implementation it +replaces: the FIPS-197 vectors first, so that agreement means AES and not +merely that both sides compute the same wrong thing, then a run of random +keys and blocks, then GHASH against the 4-bit table. Given any argument it +corrupts three results on purpose and the comparison has to notice -- a +differential test that cannot fail has said nothing. diff --git a/cbits/bearssl/VERSION b/cbits/bearssl/VERSION new file mode 100644 --- /dev/null +++ b/cbits/bearssl/VERSION @@ -0,0 +1,1 @@ +0.6 diff --git a/cbits/bearssl/aes_ct64.c b/cbits/bearssl/aes_ct64.c new file mode 100644 --- /dev/null +++ b/cbits/bearssl/aes_ct64.c @@ -0,0 +1,398 @@ +/* + * Copyright (c) 2016 Thomas Pornin + * + * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the + * "Software"), to deal in the Software without restriction, including + * without limitation the rights to use, copy, modify, merge, publish, + * distribute, sublicense, and/or sell copies of the Software, and to + * permit persons to whom the Software is furnished to do so, subject to + * the following conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include "inner.h" + +/* see inner.h */ +void +br_aes_ct64_bitslice_Sbox(uint64_t *q) +{ + /* + * This S-box implementation is a straightforward translation of + * the circuit described by Boyar and Peralta in "A new + * combinational logic minimization technique with applications + * to cryptology" (https://eprint.iacr.org/2009/191.pdf). + * + * Note that variables x* (input) and s* (output) are numbered + * in "reverse" order (x0 is the high bit, x7 is the low bit). + */ + + uint64_t x0, x1, x2, x3, x4, x5, x6, x7; + uint64_t y1, y2, y3, y4, y5, y6, y7, y8, y9; + uint64_t y10, y11, y12, y13, y14, y15, y16, y17, y18, y19; + uint64_t y20, y21; + uint64_t z0, z1, z2, z3, z4, z5, z6, z7, z8, z9; + uint64_t z10, z11, z12, z13, z14, z15, z16, z17; + uint64_t t0, t1, t2, t3, t4, t5, t6, t7, t8, t9; + uint64_t t10, t11, t12, t13, t14, t15, t16, t17, t18, t19; + uint64_t t20, t21, t22, t23, t24, t25, t26, t27, t28, t29; + uint64_t t30, t31, t32, t33, t34, t35, t36, t37, t38, t39; + uint64_t t40, t41, t42, t43, t44, t45, t46, t47, t48, t49; + uint64_t t50, t51, t52, t53, t54, t55, t56, t57, t58, t59; + uint64_t t60, t61, t62, t63, t64, t65, t66, t67; + uint64_t s0, s1, s2, s3, s4, s5, s6, s7; + + x0 = q[7]; + x1 = q[6]; + x2 = q[5]; + x3 = q[4]; + x4 = q[3]; + x5 = q[2]; + x6 = q[1]; + x7 = q[0]; + + /* + * Top linear transformation. + */ + y14 = x3 ^ x5; + y13 = x0 ^ x6; + y9 = x0 ^ x3; + y8 = x0 ^ x5; + t0 = x1 ^ x2; + y1 = t0 ^ x7; + y4 = y1 ^ x3; + y12 = y13 ^ y14; + y2 = y1 ^ x0; + y5 = y1 ^ x6; + y3 = y5 ^ y8; + t1 = x4 ^ y12; + y15 = t1 ^ x5; + y20 = t1 ^ x1; + y6 = y15 ^ x7; + y10 = y15 ^ t0; + y11 = y20 ^ y9; + y7 = x7 ^ y11; + y17 = y10 ^ y11; + y19 = y10 ^ y8; + y16 = t0 ^ y11; + y21 = y13 ^ y16; + y18 = x0 ^ y16; + + /* + * Non-linear section. + */ + t2 = y12 & y15; + t3 = y3 & y6; + t4 = t3 ^ t2; + t5 = y4 & x7; + t6 = t5 ^ t2; + t7 = y13 & y16; + t8 = y5 & y1; + t9 = t8 ^ t7; + t10 = y2 & y7; + t11 = t10 ^ t7; + t12 = y9 & y11; + t13 = y14 & y17; + t14 = t13 ^ t12; + t15 = y8 & y10; + t16 = t15 ^ t12; + t17 = t4 ^ t14; + t18 = t6 ^ t16; + t19 = t9 ^ t14; + t20 = t11 ^ t16; + t21 = t17 ^ y20; + t22 = t18 ^ y19; + t23 = t19 ^ y21; + t24 = t20 ^ y18; + + t25 = t21 ^ t22; + t26 = t21 & t23; + t27 = t24 ^ t26; + t28 = t25 & t27; + t29 = t28 ^ t22; + t30 = t23 ^ t24; + t31 = t22 ^ t26; + t32 = t31 & t30; + t33 = t32 ^ t24; + t34 = t23 ^ t33; + t35 = t27 ^ t33; + t36 = t24 & t35; + t37 = t36 ^ t34; + t38 = t27 ^ t36; + t39 = t29 & t38; + t40 = t25 ^ t39; + + t41 = t40 ^ t37; + t42 = t29 ^ t33; + t43 = t29 ^ t40; + t44 = t33 ^ t37; + t45 = t42 ^ t41; + z0 = t44 & y15; + z1 = t37 & y6; + z2 = t33 & x7; + z3 = t43 & y16; + z4 = t40 & y1; + z5 = t29 & y7; + z6 = t42 & y11; + z7 = t45 & y17; + z8 = t41 & y10; + z9 = t44 & y12; + z10 = t37 & y3; + z11 = t33 & y4; + z12 = t43 & y13; + z13 = t40 & y5; + z14 = t29 & y2; + z15 = t42 & y9; + z16 = t45 & y14; + z17 = t41 & y8; + + /* + * Bottom linear transformation. + */ + t46 = z15 ^ z16; + t47 = z10 ^ z11; + t48 = z5 ^ z13; + t49 = z9 ^ z10; + t50 = z2 ^ z12; + t51 = z2 ^ z5; + t52 = z7 ^ z8; + t53 = z0 ^ z3; + t54 = z6 ^ z7; + t55 = z16 ^ z17; + t56 = z12 ^ t48; + t57 = t50 ^ t53; + t58 = z4 ^ t46; + t59 = z3 ^ t54; + t60 = t46 ^ t57; + t61 = z14 ^ t57; + t62 = t52 ^ t58; + t63 = t49 ^ t58; + t64 = z4 ^ t59; + t65 = t61 ^ t62; + t66 = z1 ^ t63; + s0 = t59 ^ t63; + s6 = t56 ^ ~t62; + s7 = t48 ^ ~t60; + t67 = t64 ^ t65; + s3 = t53 ^ t66; + s4 = t51 ^ t66; + s5 = t47 ^ t65; + s1 = t64 ^ ~s3; + s2 = t55 ^ ~t67; + + q[7] = s0; + q[6] = s1; + q[5] = s2; + q[4] = s3; + q[3] = s4; + q[2] = s5; + q[1] = s6; + q[0] = s7; +} + +/* see inner.h */ +void +br_aes_ct64_ortho(uint64_t *q) +{ +#define SWAPN(cl, ch, s, x, y) do { \ + uint64_t a, b; \ + a = (x); \ + b = (y); \ + (x) = (a & (uint64_t)cl) | ((b & (uint64_t)cl) << (s)); \ + (y) = ((a & (uint64_t)ch) >> (s)) | (b & (uint64_t)ch); \ + } while (0) + +#define SWAP2(x, y) SWAPN(0x5555555555555555, 0xAAAAAAAAAAAAAAAA, 1, x, y) +#define SWAP4(x, y) SWAPN(0x3333333333333333, 0xCCCCCCCCCCCCCCCC, 2, x, y) +#define SWAP8(x, y) SWAPN(0x0F0F0F0F0F0F0F0F, 0xF0F0F0F0F0F0F0F0, 4, x, y) + + SWAP2(q[0], q[1]); + SWAP2(q[2], q[3]); + SWAP2(q[4], q[5]); + SWAP2(q[6], q[7]); + + SWAP4(q[0], q[2]); + SWAP4(q[1], q[3]); + SWAP4(q[4], q[6]); + SWAP4(q[5], q[7]); + + SWAP8(q[0], q[4]); + SWAP8(q[1], q[5]); + SWAP8(q[2], q[6]); + SWAP8(q[3], q[7]); +} + +/* see inner.h */ +void +br_aes_ct64_interleave_in(uint64_t *q0, uint64_t *q1, const uint32_t *w) +{ + uint64_t x0, x1, x2, x3; + + x0 = w[0]; + x1 = w[1]; + x2 = w[2]; + x3 = w[3]; + x0 |= (x0 << 16); + x1 |= (x1 << 16); + x2 |= (x2 << 16); + x3 |= (x3 << 16); + x0 &= (uint64_t)0x0000FFFF0000FFFF; + x1 &= (uint64_t)0x0000FFFF0000FFFF; + x2 &= (uint64_t)0x0000FFFF0000FFFF; + x3 &= (uint64_t)0x0000FFFF0000FFFF; + x0 |= (x0 << 8); + x1 |= (x1 << 8); + x2 |= (x2 << 8); + x3 |= (x3 << 8); + x0 &= (uint64_t)0x00FF00FF00FF00FF; + x1 &= (uint64_t)0x00FF00FF00FF00FF; + x2 &= (uint64_t)0x00FF00FF00FF00FF; + x3 &= (uint64_t)0x00FF00FF00FF00FF; + *q0 = x0 | (x2 << 8); + *q1 = x1 | (x3 << 8); +} + +/* see inner.h */ +void +br_aes_ct64_interleave_out(uint32_t *w, uint64_t q0, uint64_t q1) +{ + uint64_t x0, x1, x2, x3; + + x0 = q0 & (uint64_t)0x00FF00FF00FF00FF; + x1 = q1 & (uint64_t)0x00FF00FF00FF00FF; + x2 = (q0 >> 8) & (uint64_t)0x00FF00FF00FF00FF; + x3 = (q1 >> 8) & (uint64_t)0x00FF00FF00FF00FF; + x0 |= (x0 >> 8); + x1 |= (x1 >> 8); + x2 |= (x2 >> 8); + x3 |= (x3 >> 8); + x0 &= (uint64_t)0x0000FFFF0000FFFF; + x1 &= (uint64_t)0x0000FFFF0000FFFF; + x2 &= (uint64_t)0x0000FFFF0000FFFF; + x3 &= (uint64_t)0x0000FFFF0000FFFF; + w[0] = (uint32_t)x0 | (uint32_t)(x0 >> 16); + w[1] = (uint32_t)x1 | (uint32_t)(x1 >> 16); + w[2] = (uint32_t)x2 | (uint32_t)(x2 >> 16); + w[3] = (uint32_t)x3 | (uint32_t)(x3 >> 16); +} + +static const unsigned char Rcon[] = { + 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1B, 0x36 +}; + +static uint32_t +sub_word(uint32_t x) +{ + uint64_t q[8]; + + memset(q, 0, sizeof q); + q[0] = x; + br_aes_ct64_ortho(q); + br_aes_ct64_bitslice_Sbox(q); + br_aes_ct64_ortho(q); + return (uint32_t)q[0]; +} + +/* see inner.h */ +unsigned +br_aes_ct64_keysched(uint64_t *comp_skey, const void *key, size_t key_len) +{ + unsigned num_rounds; + int i, j, k, nk, nkf; + uint32_t tmp; + uint32_t skey[60]; + + switch (key_len) { + case 16: + num_rounds = 10; + break; + case 24: + num_rounds = 12; + break; + case 32: + num_rounds = 14; + break; + default: + /* abort(); */ + return 0; + } + nk = (int)(key_len >> 2); + nkf = (int)((num_rounds + 1) << 2); + br_range_dec32le(skey, (key_len >> 2), key); + tmp = skey[(key_len >> 2) - 1]; + for (i = nk, j = 0, k = 0; i < nkf; i ++) { + if (j == 0) { + tmp = (tmp << 24) | (tmp >> 8); + tmp = sub_word(tmp) ^ Rcon[k]; + } else if (nk > 6 && j == 4) { + tmp = sub_word(tmp); + } + tmp ^= skey[i - nk]; + skey[i] = tmp; + if (++ j == nk) { + j = 0; + k ++; + } + } + + for (i = 0, j = 0; i < nkf; i += 4, j += 2) { + uint64_t q[8]; + + br_aes_ct64_interleave_in(&q[0], &q[4], skey + i); + q[1] = q[0]; + q[2] = q[0]; + q[3] = q[0]; + q[5] = q[4]; + q[6] = q[4]; + q[7] = q[4]; + br_aes_ct64_ortho(q); + comp_skey[j + 0] = + (q[0] & (uint64_t)0x1111111111111111) + | (q[1] & (uint64_t)0x2222222222222222) + | (q[2] & (uint64_t)0x4444444444444444) + | (q[3] & (uint64_t)0x8888888888888888); + comp_skey[j + 1] = + (q[4] & (uint64_t)0x1111111111111111) + | (q[5] & (uint64_t)0x2222222222222222) + | (q[6] & (uint64_t)0x4444444444444444) + | (q[7] & (uint64_t)0x8888888888888888); + } + return num_rounds; +} + +/* see inner.h */ +void +br_aes_ct64_skey_expand(uint64_t *skey, + unsigned num_rounds, const uint64_t *comp_skey) +{ + unsigned u, v, n; + + n = (num_rounds + 1) << 1; + for (u = 0, v = 0; u < n; u ++, v += 4) { + uint64_t x0, x1, x2, x3; + + x0 = x1 = x2 = x3 = comp_skey[u]; + x0 &= (uint64_t)0x1111111111111111; + x1 &= (uint64_t)0x2222222222222222; + x2 &= (uint64_t)0x4444444444444444; + x3 &= (uint64_t)0x8888888888888888; + x1 >>= 1; + x2 >>= 2; + x3 >>= 3; + skey[v + 0] = (x0 << 4) - x0; + skey[v + 1] = (x1 << 4) - x1; + skey[v + 2] = (x2 << 4) - x2; + skey[v + 3] = (x3 << 4) - x3; + } +} diff --git a/cbits/bearssl/aes_ct64_dec.c b/cbits/bearssl/aes_ct64_dec.c new file mode 100644 --- /dev/null +++ b/cbits/bearssl/aes_ct64_dec.c @@ -0,0 +1,159 @@ +/* + * Copyright (c) 2016 Thomas Pornin + * + * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the + * "Software"), to deal in the Software without restriction, including + * without limitation the rights to use, copy, modify, merge, publish, + * distribute, sublicense, and/or sell copies of the Software, and to + * permit persons to whom the Software is furnished to do so, subject to + * the following conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include "inner.h" + +/* see inner.h */ +void +br_aes_ct64_bitslice_invSbox(uint64_t *q) +{ + /* + * See br_aes_ct_bitslice_invSbox(). This is the natural extension + * to 64-bit registers. + */ + uint64_t q0, q1, q2, q3, q4, q5, q6, q7; + + q0 = ~q[0]; + q1 = ~q[1]; + q2 = q[2]; + q3 = q[3]; + q4 = q[4]; + q5 = ~q[5]; + q6 = ~q[6]; + q7 = q[7]; + q[7] = q1 ^ q4 ^ q6; + q[6] = q0 ^ q3 ^ q5; + q[5] = q7 ^ q2 ^ q4; + q[4] = q6 ^ q1 ^ q3; + q[3] = q5 ^ q0 ^ q2; + q[2] = q4 ^ q7 ^ q1; + q[1] = q3 ^ q6 ^ q0; + q[0] = q2 ^ q5 ^ q7; + + br_aes_ct64_bitslice_Sbox(q); + + q0 = ~q[0]; + q1 = ~q[1]; + q2 = q[2]; + q3 = q[3]; + q4 = q[4]; + q5 = ~q[5]; + q6 = ~q[6]; + q7 = q[7]; + q[7] = q1 ^ q4 ^ q6; + q[6] = q0 ^ q3 ^ q5; + q[5] = q7 ^ q2 ^ q4; + q[4] = q6 ^ q1 ^ q3; + q[3] = q5 ^ q0 ^ q2; + q[2] = q4 ^ q7 ^ q1; + q[1] = q3 ^ q6 ^ q0; + q[0] = q2 ^ q5 ^ q7; +} + +static void +add_round_key(uint64_t *q, const uint64_t *sk) +{ + int i; + + for (i = 0; i < 8; i ++) { + q[i] ^= sk[i]; + } +} + +static void +inv_shift_rows(uint64_t *q) +{ + int i; + + for (i = 0; i < 8; i ++) { + uint64_t x; + + x = q[i]; + q[i] = (x & (uint64_t)0x000000000000FFFF) + | ((x & (uint64_t)0x000000000FFF0000) << 4) + | ((x & (uint64_t)0x00000000F0000000) >> 12) + | ((x & (uint64_t)0x000000FF00000000) << 8) + | ((x & (uint64_t)0x0000FF0000000000) >> 8) + | ((x & (uint64_t)0x000F000000000000) << 12) + | ((x & (uint64_t)0xFFF0000000000000) >> 4); + } +} + +static inline uint64_t +rotr32(uint64_t x) +{ + return (x << 32) | (x >> 32); +} + +static void +inv_mix_columns(uint64_t *q) +{ + uint64_t q0, q1, q2, q3, q4, q5, q6, q7; + uint64_t r0, r1, r2, r3, r4, r5, r6, r7; + + q0 = q[0]; + q1 = q[1]; + q2 = q[2]; + q3 = q[3]; + q4 = q[4]; + q5 = q[5]; + q6 = q[6]; + q7 = q[7]; + r0 = (q0 >> 16) | (q0 << 48); + r1 = (q1 >> 16) | (q1 << 48); + r2 = (q2 >> 16) | (q2 << 48); + r3 = (q3 >> 16) | (q3 << 48); + r4 = (q4 >> 16) | (q4 << 48); + r5 = (q5 >> 16) | (q5 << 48); + r6 = (q6 >> 16) | (q6 << 48); + r7 = (q7 >> 16) | (q7 << 48); + + q[0] = q5 ^ q6 ^ q7 ^ r0 ^ r5 ^ r7 ^ rotr32(q0 ^ q5 ^ q6 ^ r0 ^ r5); + q[1] = q0 ^ q5 ^ r0 ^ r1 ^ r5 ^ r6 ^ r7 ^ rotr32(q1 ^ q5 ^ q7 ^ r1 ^ r5 ^ r6); + q[2] = q0 ^ q1 ^ q6 ^ r1 ^ r2 ^ r6 ^ r7 ^ rotr32(q0 ^ q2 ^ q6 ^ r2 ^ r6 ^ r7); + q[3] = q0 ^ q1 ^ q2 ^ q5 ^ q6 ^ r0 ^ r2 ^ r3 ^ r5 ^ rotr32(q0 ^ q1 ^ q3 ^ q5 ^ q6 ^ q7 ^ r0 ^ r3 ^ r5 ^ r7); + q[4] = q1 ^ q2 ^ q3 ^ q5 ^ r1 ^ r3 ^ r4 ^ r5 ^ r6 ^ r7 ^ rotr32(q1 ^ q2 ^ q4 ^ q5 ^ q7 ^ r1 ^ r4 ^ r5 ^ r6); + q[5] = q2 ^ q3 ^ q4 ^ q6 ^ r2 ^ r4 ^ r5 ^ r6 ^ r7 ^ rotr32(q2 ^ q3 ^ q5 ^ q6 ^ r2 ^ r5 ^ r6 ^ r7); + q[6] = q3 ^ q4 ^ q5 ^ q7 ^ r3 ^ r5 ^ r6 ^ r7 ^ rotr32(q3 ^ q4 ^ q6 ^ q7 ^ r3 ^ r6 ^ r7); + q[7] = q4 ^ q5 ^ q6 ^ r4 ^ r6 ^ r7 ^ rotr32(q4 ^ q5 ^ q7 ^ r4 ^ r7); +} + +/* see inner.h */ +void +br_aes_ct64_bitslice_decrypt(unsigned num_rounds, + const uint64_t *skey, uint64_t *q) +{ + unsigned u; + + add_round_key(q, skey + (num_rounds << 3)); + for (u = num_rounds - 1; u > 0; u --) { + inv_shift_rows(q); + br_aes_ct64_bitslice_invSbox(q); + add_round_key(q, skey + (u << 3)); + inv_mix_columns(q); + } + inv_shift_rows(q); + br_aes_ct64_bitslice_invSbox(q); + add_round_key(q, skey); +} diff --git a/cbits/bearssl/aes_ct64_enc.c b/cbits/bearssl/aes_ct64_enc.c new file mode 100644 --- /dev/null +++ b/cbits/bearssl/aes_ct64_enc.c @@ -0,0 +1,115 @@ +/* + * Copyright (c) 2016 Thomas Pornin + * + * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the + * "Software"), to deal in the Software without restriction, including + * without limitation the rights to use, copy, modify, merge, publish, + * distribute, sublicense, and/or sell copies of the Software, and to + * permit persons to whom the Software is furnished to do so, subject to + * the following conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include "inner.h" + +static inline void +add_round_key(uint64_t *q, const uint64_t *sk) +{ + q[0] ^= sk[0]; + q[1] ^= sk[1]; + q[2] ^= sk[2]; + q[3] ^= sk[3]; + q[4] ^= sk[4]; + q[5] ^= sk[5]; + q[6] ^= sk[6]; + q[7] ^= sk[7]; +} + +static inline void +shift_rows(uint64_t *q) +{ + int i; + + for (i = 0; i < 8; i ++) { + uint64_t x; + + x = q[i]; + q[i] = (x & (uint64_t)0x000000000000FFFF) + | ((x & (uint64_t)0x00000000FFF00000) >> 4) + | ((x & (uint64_t)0x00000000000F0000) << 12) + | ((x & (uint64_t)0x0000FF0000000000) >> 8) + | ((x & (uint64_t)0x000000FF00000000) << 8) + | ((x & (uint64_t)0xF000000000000000) >> 12) + | ((x & (uint64_t)0x0FFF000000000000) << 4); + } +} + +static inline uint64_t +rotr32(uint64_t x) +{ + return (x << 32) | (x >> 32); +} + +static inline void +mix_columns(uint64_t *q) +{ + uint64_t q0, q1, q2, q3, q4, q5, q6, q7; + uint64_t r0, r1, r2, r3, r4, r5, r6, r7; + + q0 = q[0]; + q1 = q[1]; + q2 = q[2]; + q3 = q[3]; + q4 = q[4]; + q5 = q[5]; + q6 = q[6]; + q7 = q[7]; + r0 = (q0 >> 16) | (q0 << 48); + r1 = (q1 >> 16) | (q1 << 48); + r2 = (q2 >> 16) | (q2 << 48); + r3 = (q3 >> 16) | (q3 << 48); + r4 = (q4 >> 16) | (q4 << 48); + r5 = (q5 >> 16) | (q5 << 48); + r6 = (q6 >> 16) | (q6 << 48); + r7 = (q7 >> 16) | (q7 << 48); + + q[0] = q7 ^ r7 ^ r0 ^ rotr32(q0 ^ r0); + q[1] = q0 ^ r0 ^ q7 ^ r7 ^ r1 ^ rotr32(q1 ^ r1); + q[2] = q1 ^ r1 ^ r2 ^ rotr32(q2 ^ r2); + q[3] = q2 ^ r2 ^ q7 ^ r7 ^ r3 ^ rotr32(q3 ^ r3); + q[4] = q3 ^ r3 ^ q7 ^ r7 ^ r4 ^ rotr32(q4 ^ r4); + q[5] = q4 ^ r4 ^ r5 ^ rotr32(q5 ^ r5); + q[6] = q5 ^ r5 ^ r6 ^ rotr32(q6 ^ r6); + q[7] = q6 ^ r6 ^ r7 ^ rotr32(q7 ^ r7); +} + +/* see inner.h */ +void +br_aes_ct64_bitslice_encrypt(unsigned num_rounds, + const uint64_t *skey, uint64_t *q) +{ + unsigned u; + + add_round_key(q, skey); + for (u = 1; u < num_rounds; u ++) { + br_aes_ct64_bitslice_Sbox(q); + shift_rows(q); + mix_columns(q); + add_round_key(q, skey + (u << 3)); + } + br_aes_ct64_bitslice_Sbox(q); + shift_rows(q); + add_round_key(q, skey + (num_rounds << 3)); +} diff --git a/cbits/bearssl/dec32le.c b/cbits/bearssl/dec32le.c new file mode 100644 --- /dev/null +++ b/cbits/bearssl/dec32le.c @@ -0,0 +1,38 @@ +/* + * Copyright (c) 2016 Thomas Pornin + * + * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the + * "Software"), to deal in the Software without restriction, including + * without limitation the rights to use, copy, modify, merge, publish, + * distribute, sublicense, and/or sell copies of the Software, and to + * permit persons to whom the Software is furnished to do so, subject to + * the following conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include "inner.h" + +/* see inner.h */ +void +br_range_dec32le(uint32_t *v, size_t num, const void *src) +{ + const unsigned char *buf; + + buf = src; + while (num -- > 0) { + *v ++ = br_dec32le(buf); + buf += 4; + } +} diff --git a/cbits/bearssl/ghash_ctmul64.c b/cbits/bearssl/ghash_ctmul64.c new file mode 100644 --- /dev/null +++ b/cbits/bearssl/ghash_ctmul64.c @@ -0,0 +1,154 @@ +/* + * Copyright (c) 2016 Thomas Pornin + * + * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the + * "Software"), to deal in the Software without restriction, including + * without limitation the rights to use, copy, modify, merge, publish, + * distribute, sublicense, and/or sell copies of the Software, and to + * permit persons to whom the Software is furnished to do so, subject to + * the following conditions: + * + * The above copyright notice and this permission notice shall be + * included in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#include "inner.h" + +/* + * This is the 64-bit variant of br_ghash_ctmul32(), with 64-bit operands + * and bit reversal of 64-bit words. + */ + +static inline uint64_t +bmul64(uint64_t x, uint64_t y) +{ + uint64_t x0, x1, x2, x3; + uint64_t y0, y1, y2, y3; + uint64_t z0, z1, z2, z3; + + x0 = x & (uint64_t)0x1111111111111111; + x1 = x & (uint64_t)0x2222222222222222; + x2 = x & (uint64_t)0x4444444444444444; + x3 = x & (uint64_t)0x8888888888888888; + y0 = y & (uint64_t)0x1111111111111111; + y1 = y & (uint64_t)0x2222222222222222; + y2 = y & (uint64_t)0x4444444444444444; + y3 = y & (uint64_t)0x8888888888888888; + z0 = (x0 * y0) ^ (x1 * y3) ^ (x2 * y2) ^ (x3 * y1); + z1 = (x0 * y1) ^ (x1 * y0) ^ (x2 * y3) ^ (x3 * y2); + z2 = (x0 * y2) ^ (x1 * y1) ^ (x2 * y0) ^ (x3 * y3); + z3 = (x0 * y3) ^ (x1 * y2) ^ (x2 * y1) ^ (x3 * y0); + z0 &= (uint64_t)0x1111111111111111; + z1 &= (uint64_t)0x2222222222222222; + z2 &= (uint64_t)0x4444444444444444; + z3 &= (uint64_t)0x8888888888888888; + return z0 | z1 | z2 | z3; +} + +static uint64_t +rev64(uint64_t x) +{ +#define RMS(m, s) do { \ + x = ((x & (uint64_t)(m)) << (s)) \ + | ((x >> (s)) & (uint64_t)(m)); \ + } while (0) + + RMS(0x5555555555555555, 1); + RMS(0x3333333333333333, 2); + RMS(0x0F0F0F0F0F0F0F0F, 4); + RMS(0x00FF00FF00FF00FF, 8); + RMS(0x0000FFFF0000FFFF, 16); + return (x << 32) | (x >> 32); + +#undef RMS +} + +/* see bearssl_ghash.h */ +void +br_ghash_ctmul64(void *y, const void *h, const void *data, size_t len) +{ + const unsigned char *buf, *hb; + unsigned char *yb; + uint64_t y0, y1; + uint64_t h0, h1, h2, h0r, h1r, h2r; + + buf = data; + yb = y; + hb = h; + y1 = br_dec64be(yb); + y0 = br_dec64be(yb + 8); + h1 = br_dec64be(hb); + h0 = br_dec64be(hb + 8); + h0r = rev64(h0); + h1r = rev64(h1); + h2 = h0 ^ h1; + h2r = h0r ^ h1r; + while (len > 0) { + const unsigned char *src; + unsigned char tmp[16]; + uint64_t y0r, y1r, y2, y2r; + uint64_t z0, z1, z2, z0h, z1h, z2h; + uint64_t v0, v1, v2, v3; + + if (len >= 16) { + src = buf; + buf += 16; + len -= 16; + } else { + memcpy(tmp, buf, len); + memset(tmp + len, 0, (sizeof tmp) - len); + src = tmp; + len = 0; + } + y1 ^= br_dec64be(src); + y0 ^= br_dec64be(src + 8); + + y0r = rev64(y0); + y1r = rev64(y1); + y2 = y0 ^ y1; + y2r = y0r ^ y1r; + + z0 = bmul64(y0, h0); + z1 = bmul64(y1, h1); + z2 = bmul64(y2, h2); + z0h = bmul64(y0r, h0r); + z1h = bmul64(y1r, h1r); + z2h = bmul64(y2r, h2r); + z2 ^= z0 ^ z1; + z2h ^= z0h ^ z1h; + z0h = rev64(z0h) >> 1; + z1h = rev64(z1h) >> 1; + z2h = rev64(z2h) >> 1; + + v0 = z0; + v1 = z0h ^ z2; + v2 = z1 ^ z2h; + v3 = z1h; + + v3 = (v3 << 1) | (v2 >> 63); + v2 = (v2 << 1) | (v1 >> 63); + v1 = (v1 << 1) | (v0 >> 63); + v0 = (v0 << 1); + + v2 ^= v0 ^ (v0 >> 1) ^ (v0 >> 2) ^ (v0 >> 7); + v1 ^= (v0 << 63) ^ (v0 << 62) ^ (v0 << 57); + v3 ^= v1 ^ (v1 >> 1) ^ (v1 >> 2) ^ (v1 >> 7); + v2 ^= (v1 << 63) ^ (v1 << 62) ^ (v1 << 57); + + y0 = v2; + y1 = v3; + } + + br_enc64be(yb, y1); + br_enc64be(yb + 8, y0); +} diff --git a/cbits/bearssl/import.sh b/cbits/bearssl/import.sh new file mode 100644 --- /dev/null +++ b/cbits/bearssl/import.sh @@ -0,0 +1,34 @@ +#!/bin/sh +# Re-import the vendored parts of BearSSL. +# +# Only the five files crypton calls are kept, and they are kept unmodified, +# so that `diff` against a new release is readable. What they want from +# upstream's two-thousand-line src/inner.h is supplied by the inner.h here, +# which is crypton's and is NOT overwritten by this script. Run this from +# cbits/bearssl: +# +# ./import.sh [version] +# +# and commit the result together with the VERSION line it writes, so that +# the tree always says which upstream release it holds. +set -eu + +VER=${1:-0.6} +HERE=$(cd "$(dirname "$0")" && pwd) +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT + +curl -sSL "https://bearssl.org/bearssl-$VER.tar.gz" -o "$TMP/b.tgz" +tar xzf "$TMP/b.tgz" -C "$TMP" +SRC="$TMP/bearssl-$VER" + +cp "$SRC/LICENSE.txt" "$HERE/LICENSE" +cp "$SRC/src/symcipher/aes_ct64.c" "$HERE/" +cp "$SRC/src/symcipher/aes_ct64_enc.c" "$HERE/" +cp "$SRC/src/symcipher/aes_ct64_dec.c" "$HERE/" +cp "$SRC/src/hash/ghash_ctmul64.c" "$HERE/" +cp "$SRC/src/codec/dec32le.c" "$HERE/" +echo "$VER" > "$HERE/VERSION" + +echo "imported BearSSL $VER" +echo "now run the differential test in cbits/tests/bearssl_diff.c" diff --git a/cbits/bearssl/inner.h b/cbits/bearssl/inner.h new file mode 100644 --- /dev/null +++ b/cbits/bearssl/inner.h @@ -0,0 +1,92 @@ +/* + * Stands in for BearSSL's own src/inner.h. + * + * The five .c files beside this one are upstream's, byte for byte, and each + * of them opens with #include "inner.h". Upstream's is some two thousand + * lines and declares the whole library; these five want six things from it. + * So this header gives those six and nothing else, and the upstream files + * stay unmodified -- which is what makes `diff` against a new BearSSL + * release readable. See README.md. + * + * The byte-order helpers are written here rather than copied, in their plain + * portable form without upstream's unaligned-access fast paths, so that they + * carry no platform configuration with them. cbits/tests/bearssl_diff.c + * checks the whole thing against crypton's existing implementation. + */ +#ifndef CRYPTON_BEARSSL_INNER_H +#define CRYPTON_BEARSSL_INNER_H + +#include +#include +#include + +static inline uint32_t +br_dec32le(const void *src) +{ + const unsigned char *b = src; + + return (uint32_t)b[0] + | ((uint32_t)b[1] << 8) + | ((uint32_t)b[2] << 16) + | ((uint32_t)b[3] << 24); +} + +static inline void +br_enc32le(void *dst, uint32_t x) +{ + unsigned char *b = dst; + + b[0] = (unsigned char)x; + b[1] = (unsigned char)(x >> 8); + b[2] = (unsigned char)(x >> 16); + b[3] = (unsigned char)(x >> 24); +} + +static inline uint64_t +br_dec64be(const void *src) +{ + const unsigned char *b = src; + uint64_t x = 0; + int i; + + for (i = 0; i < 8; i++) + x = (x << 8) | (uint64_t)b[i]; + return x; +} + +static inline void +br_enc64be(void *dst, uint64_t x) +{ + unsigned char *b = dst; + int i; + + for (i = 7; i >= 0; i--) { + b[i] = (unsigned char)(x & 0xff); + x >>= 8; + } +} + +/* dec32le.c */ +void br_range_dec32le(uint32_t *v, size_t num, const void *src); + +/* aes_ct64.c */ +void br_aes_ct64_bitslice_Sbox(uint64_t *q); +void br_aes_ct64_ortho(uint64_t *q); +void br_aes_ct64_interleave_in(uint64_t *q0, uint64_t *q1, const uint32_t *w); +void br_aes_ct64_interleave_out(uint32_t *w, uint64_t q0, uint64_t q1); +unsigned br_aes_ct64_keysched(uint64_t *comp_skey, const void *key, + size_t key_len); +void br_aes_ct64_skey_expand(uint64_t *skey, unsigned num_rounds, + const uint64_t *comp_skey); + +/* aes_ct64_enc.c, aes_ct64_dec.c */ +void br_aes_ct64_bitslice_encrypt(unsigned num_rounds, const uint64_t *skey, + uint64_t *q); +void br_aes_ct64_bitslice_invSbox(uint64_t *q); +void br_aes_ct64_bitslice_decrypt(unsigned num_rounds, const uint64_t *skey, + uint64_t *q); + +/* ghash_ctmul64.c */ +void br_ghash_ctmul64(void *y, const void *h, const void *data, size_t len); + +#endif diff --git a/cbits/crypton_aes.c b/cbits/crypton_aes.c --- a/cbits/crypton_aes.c +++ b/cbits/crypton_aes.c @@ -38,6 +38,12 @@ #include #include #include +#ifdef WITH_PPC8_CRYPTO +#include +#endif +#ifdef WITH_GCM_FUSED +#include +#endif void crypton_aes_generic_encrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks); void crypton_aes_generic_decrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks); @@ -51,6 +57,8 @@ uint32_t spoint, aes_block *input, uint32_t nb_blocks); void crypton_aes_generic_gcm_encrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length); void crypton_aes_generic_gcm_decrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length); +void crypton_aes_bitsliced_gcm_encrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length); +void crypton_aes_bitsliced_gcm_decrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length); void crypton_aes_generic_ocb_encrypt(uint8_t *output, aes_ocb *ocb, aes_key *key, uint8_t *input, uint32_t length); void crypton_aes_generic_ocb_decrypt(uint8_t *output, aes_ocb *ocb, aes_key *key, uint8_t *input, uint32_t length); void crypton_aes_generic_ccm_encrypt(uint8_t *output, aes_ccm *ccm, aes_key *key, uint8_t *input, uint32_t length); @@ -58,6 +66,18 @@ #ifdef WITH_ARMV8_CRYPTO void crypton_aes_armv8_init(aes_key *key, uint8_t *origkey, uint8_t size); +int crypton_aes_armv8_gcm_fused_dec(uint8_t *out, const block128 *ht, + aes_key *key, const uint8_t *nonce, + const uint8_t *aad, uint32_t aadlen, + const uint8_t *in, uint32_t inlen, + const uint8_t *tag, uint32_t taglen, + uint8_t *outtag); +void crypton_aes_armv8_gcm_fused(uint8_t *out, const block128 *ht, + aes_key *key, const uint8_t *nonce, + const uint8_t *aad, uint32_t aadlen, + const uint8_t *in, uint32_t inlen, + uint32_t taglen, aes_key *hpkey, + uint32_t sampleoff, uint8_t *mask); #define ARMV8_DECLS(sz) \ void crypton_aes_armv8_encrypt_block##sz(aes_block *output, aes_key *key, aes_block *input); \ void crypton_aes_armv8_decrypt_block##sz(aes_block *output, aes_key *key, aes_block *input); \ @@ -193,7 +213,7 @@ typedef void (*gf_mul_f)(block128 *a, const table_4bit htable); typedef void (*gf_mul4_f)(block128 *a, const block128 *blocks, const table_4bit htable); -#if defined(WITH_AESNI) || defined(WITH_ARMV8_CRYPTO) +#if defined(WITH_AESNI) || defined(WITH_ARMV8_CRYPTO) || defined(WITH_PPC8_CRYPTO) #define GET_INIT(strength) \ ((init_f) (crypton_aes_branch_table[INIT_128 + strength])) #define GET_ECB_ENCRYPT(strength) \ @@ -240,12 +260,16 @@ #define GET_ECB_DECRYPT(strength) crypton_aes_generic_decrypt_ecb #define GET_CBC_ENCRYPT(strength) crypton_aes_generic_encrypt_cbc #define GET_CBC_DECRYPT(strength) crypton_aes_generic_decrypt_cbc -#define GET_CTR_ENCRYPT(strength) crypton_aes_generic_encrypt_ctr -#define GET_C32_ENCRYPT(strength) crypton_aes_generic_encrypt_c32 +/* No accelerator is compiled in, so the portable schedule is the only one + * a key can hold and the four-block CTR is named here rather than installed + * at run time. This is the build ppc64le, s390x, riscv64, 32-bit ARM and + * -f-support_aesni all get, and nothing in it reads the branch table. */ +#define GET_CTR_ENCRYPT(strength) crypton_aes_bitsliced_encrypt_ctr +#define GET_C32_ENCRYPT(strength) crypton_aes_bitsliced_encrypt_c32 #define GET_XTS_ENCRYPT(strength) crypton_aes_generic_encrypt_xts #define GET_XTS_DECRYPT(strength) crypton_aes_generic_decrypt_xts -#define GET_GCM_ENCRYPT(strength) crypton_aes_generic_gcm_encrypt -#define GET_GCM_DECRYPT(strength) crypton_aes_generic_gcm_decrypt +#define GET_GCM_ENCRYPT(strength) crypton_aes_bitsliced_gcm_encrypt +#define GET_GCM_DECRYPT(strength) crypton_aes_bitsliced_gcm_decrypt #define GET_OCB_ENCRYPT(strength) crypton_aes_generic_ocb_encrypt #define GET_OCB_DECRYPT(strength) crypton_aes_generic_ocb_decrypt #define GET_CCM_ENCRYPT(strength) crypton_aes_generic_ccm_encrypt @@ -257,10 +281,6 @@ #define crypton_gf_mul4(a,b,t) crypton_aes_generic_gf_mul4(a,b,t) #endif -#define CPU_AESNI 0 -#define CPU_PCLMUL 1 -#define CPU_OPTION_COUNT 2 - static uint8_t crypton_aes_cpu_options[CPU_OPTION_COUNT] = {}; #if defined(ARCH_X86) && defined(WITH_AESNI) @@ -410,14 +430,112 @@ } #endif -uint8_t *crypton_aes_cpu_init(void) +/* Which implementation each entry of the branch table names is decided once, + * before anything else runs. + * + * It used to be decided again on every crypton_aes_initkey, which meant two + * threads taking a key at the same time were writing the whole table at the + * same time -- a data race for as long as the program used AES, not just at + * the start. ThreadSanitizer reports forty-two of them for eight threads + * doing nothing but taking keys. The values written are the same ones every + * time and the table starts out holding valid generic implementations, so + * nothing has ever come of it; it is a race the standard gives no meaning to + * all the same. + * + * A constructor runs while there is one thread, which is the cheapest way to + * have no race at all: no flag to test, no lock to take, and one less thing + * for crypton_aes_initkey to do per key. */ +/* + * The two CTR entries, where the portable implementation is the one that + * will run. This is for the build that has an accelerator compiled in and + * did not find it on the processor; a build with no accelerator at all + * names them directly in the GET_ macros above and never reads this table. + * + * They cannot be the defaults. crypton_aes_generic_encrypt_c32 is not + * overridden on AArch64 -- the ARMv8 table has no C32 of its own -- and + * neither CCM nor OCB is overridden anywhere, so those reach the block + * function through the branch table and work whatever is installed. A CTR + * that read the portable schedule directly would be wrong on exactly those + * machines. So these go in only once nothing has claimed AES. + */ +static void initialize_table_bitsliced(void) { + crypton_aes_branch_table[ENCRYPT_CTR_128] = crypton_aes_bitsliced_encrypt_ctr; + crypton_aes_branch_table[ENCRYPT_CTR_192] = crypton_aes_bitsliced_encrypt_ctr; + crypton_aes_branch_table[ENCRYPT_CTR_256] = crypton_aes_bitsliced_encrypt_ctr; + crypton_aes_branch_table[ENCRYPT_C32_128] = crypton_aes_bitsliced_encrypt_c32; + crypton_aes_branch_table[ENCRYPT_C32_192] = crypton_aes_bitsliced_encrypt_c32; + crypton_aes_branch_table[ENCRYPT_C32_256] = crypton_aes_bitsliced_encrypt_c32; + crypton_aes_branch_table[ENCRYPT_GCM_128] = crypton_aes_bitsliced_gcm_encrypt; + crypton_aes_branch_table[ENCRYPT_GCM_192] = crypton_aes_bitsliced_gcm_encrypt; + crypton_aes_branch_table[ENCRYPT_GCM_256] = crypton_aes_bitsliced_gcm_encrypt; + crypton_aes_branch_table[DECRYPT_GCM_128] = crypton_aes_bitsliced_gcm_decrypt; + crypton_aes_branch_table[DECRYPT_GCM_192] = crypton_aes_bitsliced_gcm_decrypt; + crypton_aes_branch_table[DECRYPT_GCM_256] = crypton_aes_bitsliced_gcm_decrypt; +} + +#ifdef WITH_PPC8_CRYPTO +/* + * POWER8. One function per operation rather than three: the assembly reads + * the round count out of the key, so the same entry serves every key size. + * + * GCM and the 32-bit counter are left where they are. Neither has an entry + * in the assembly, and the generic loops reach the block function and the + * GHASH through this table, so both get the instructions anyway. + */ +static void initialize_table_ppc8(void) +{ + int sz; + + if (!crypton_aes_ppc8_available()) + return; + /* what stops initialize_table_bitsliced below from putting its own CTR + * in, which would read this key as a bitsliced schedule */ + crypton_aes_cpu_options[CPU_AESNI] = 1; + crypton_aes_cpu_options[CPU_PCLMUL] = 1; + + for (sz = 0; sz < 3; sz++) { + crypton_aes_branch_table[INIT_128 + sz] = crypton_aes_ppc8_init; + crypton_aes_branch_table[ENCRYPT_BLOCK_128 + sz] = crypton_aes_ppc8_encrypt_block; + crypton_aes_branch_table[DECRYPT_BLOCK_128 + sz] = crypton_aes_ppc8_decrypt_block; + crypton_aes_branch_table[ENCRYPT_ECB_128 + sz] = crypton_aes_ppc8_encrypt_ecb; + crypton_aes_branch_table[DECRYPT_ECB_128 + sz] = crypton_aes_ppc8_decrypt_ecb; + crypton_aes_branch_table[ENCRYPT_CBC_128 + sz] = crypton_aes_ppc8_encrypt_cbc; + crypton_aes_branch_table[DECRYPT_CBC_128 + sz] = crypton_aes_ppc8_decrypt_cbc; + crypton_aes_branch_table[ENCRYPT_CTR_128 + sz] = crypton_aes_ppc8_encrypt_ctr; + crypton_aes_branch_table[ENCRYPT_XTS_128 + sz] = crypton_aes_ppc8_encrypt_xts; + crypton_aes_branch_table[DECRYPT_XTS_128 + sz] = crypton_aes_ppc8_decrypt_xts; + } + + crypton_aes_branch_table[GHASH_HINIT] = crypton_aes_ppc8_hinit; + crypton_aes_branch_table[GHASH_GF_MUL] = crypton_aes_ppc8_gf_mul; + crypton_aes_branch_table[GHASH_GF_MUL4] = crypton_aes_ppc8_gf_mul4; +} +#endif + +static void crypton_aes_cpu_setup(void) +{ #if defined(ARCH_X86) && defined(WITH_AESNI) crypton_aesni_initialize_hw(initialize_table_ni); #endif #ifdef WITH_ARMV8_CRYPTO initialize_table_armv8(); #endif +#ifdef WITH_PPC8_CRYPTO + initialize_table_ppc8(); +#endif + if (crypton_aes_cpu_options[CPU_AESNI] == 0) + initialize_table_bitsliced(); +} + +__attribute__((constructor)) +static void crypton_aes_cpu_ctor(void) +{ + crypton_aes_cpu_setup(); +} + +uint8_t *crypton_aes_cpu_init(void) +{ return crypton_aes_cpu_options; } @@ -428,7 +546,6 @@ case 24: key->nbr = 12; key->strength = 1; break; case 32: key->nbr = 14; key->strength = 2; break; } - crypton_aes_cpu_init(); init_f _init = GET_INIT(key->strength); _init(key, origkey, size); } @@ -532,20 +649,33 @@ crypton_gf_mul4(&gcm->tag, b, gcm->htable); } -void crypton_aes_gcm_init(aes_gcm *gcm, aes_key *key, uint8_t *iv, uint32_t len) +/* The part of the state that depends on the key alone: H = encrypt_K(0^128) + * and the table of its multiples. It is 256 of the 320 bytes, and a caller + * that keeps a key can compute it once instead of once per message. */ +void crypton_aes_gcm_key_init(aes_gcm_key *gk, aes_key *key) { block128 h; + + block128_zero(&h); + crypton_aes_encrypt_block(&h, key, &h); + crypton_hinit(gk->gcm.htable, &h); +#ifdef WITH_GCM_FUSED + if (crypton_aes_cpu_options[CPU_AESNI] + && crypton_aes_cpu_options[CPU_PCLMUL]) + crypton_gcm_fused_key_init(&gk->fused, key); +#endif +} + +/* Everything else: what the nonce and the message determine. Leaves htable + * alone, so it runs on a state whose key part is already there. */ +static void gcm_message_init(aes_gcm *gcm, uint8_t *iv, uint32_t len) +{ gcm->length_aad = 0; gcm->length_input = 0; - block128_zero(&h); block128_zero(&gcm->tag); block128_zero(&gcm->iv); - /* prepare H : encrypt_K(0^128) */ - crypton_aes_encrypt_block(&h, key, &h); - crypton_hinit(gcm->htable, &h); - if (len == 12) { block128_copy_bytes(&gcm->iv, iv, 12); gcm->iv.b[15] = 0x01; @@ -568,6 +698,16 @@ block128_copy_aligned(&gcm->civ, &gcm->iv); } +void crypton_aes_gcm_init(aes_gcm *gcm, aes_key *key, uint8_t *iv, uint32_t len) +{ + block128 h; + + block128_zero(&h); + crypton_aes_encrypt_block(&h, key, &h); + crypton_hinit(gcm->htable, &h); + gcm_message_init(gcm, iv, len); +} + void crypton_aes_gcm_aad(aes_gcm *gcm, uint8_t *input, uint32_t length) { gcm->length_aad += length; @@ -604,6 +744,195 @@ } } +/* One message, one call. The key part of the state comes in already built, + * the rest is set up on the stack, and the additional data, the encryption + * and the tag all happen before returning, so nothing crosses a language + * boundary between them and no intermediate state is copied out. The output + * buffer takes the ciphertext and then the tag, so it wants length + taglen + * bytes. */ +void crypton_aes_gcm_full_encrypt(uint8_t *output, const aes_gcm_key *gcmkey, aes_key *key, + uint8_t *iv, uint32_t ivlen, + uint8_t *aad, uint32_t aadlen, + uint8_t *input, uint32_t length, uint32_t taglen) +{ + aes_gcm gcm; + uint8_t tag[16]; + +#ifdef WITH_GCM_FUSED + /* Short messages go the other way: the assembly below will not start + * on anything under 288 bytes, and under about 1.5 KB the fused path + * is ahead of it even where it does. */ + if (ivlen == 12 && length <= CRYPTON_GCM_FUSED_MAX_MESSAGE + && crypton_aes_cpu_options[CPU_AESNI] + && crypton_aes_cpu_options[CPU_PCLMUL]) { + crypton_gcm_fused_encrypt(output, &gcmkey->fused, key, iv, + aad, aadlen, input, length, taglen, + NULL, 0, NULL); + return; + } +#endif +#ifdef WITH_ARMV8_CRYPTO + /* The same on AArch64, where the framing is what costs: composing the + * additional data, the encryption and the tag reaches each through the + * branch table, so the running state goes back to memory between them + * and a one-block header pays a reduction of its own. Measured on an + * Apple M4, a 100-byte packet is 3.0x faster taken in one call. + * + * No length limit, unlike x86: there is no vendored assembly on this + * side for a long message to be handed to instead, and measured + * against the path this replaces it is never slower -- 1.25x at 1440 + * bytes, level from about 6 KB up. */ + if (ivlen == 12 + && crypton_aes_cpu_options[CPU_AESNI] + && crypton_aes_cpu_options[CPU_PCLMUL]) { + crypton_aes_armv8_gcm_fused(output, gcmkey->gcm.htable, key, iv, + aad, aadlen, input, length, taglen, + NULL, 0, NULL); + return; + } +#endif + memcpy(gcm.htable, gcmkey->gcm.htable, sizeof(gcm.htable)); + gcm_message_init(&gcm, iv, ivlen); + if (aadlen) + crypton_aes_gcm_aad(&gcm, aad, aadlen); + if (length) + crypton_aes_gcm_encrypt(output, &gcm, key, input, length); + crypton_aes_gcm_finish(tag, &gcm, key); + memcpy(output + length, tag, taglen); +} + +/* The same, and then the header protection mask. QUIC takes its sample from + * the ciphertext, so the mask cannot be had before the encryption -- but it + * can be had before returning, which saves a second crossing for one AES + * block. The block itself is about a nanosecond; what it saves is the call. + * sampleoff is where the sixteen bytes of sample start in the output. */ +void crypton_aes_gcm_full_encrypt_mask(uint8_t *output, const aes_gcm_key *gcmkey, aes_key *key, + uint8_t *iv, uint32_t ivlen, + uint8_t *aad, uint32_t aadlen, + uint8_t *input, uint32_t length, uint32_t taglen, + aes_key *hpkey, uint32_t sampleoff, uint8_t *mask) +{ + block128 sample, m; + +#ifdef WITH_GCM_FUSED + /* Here the mask rides in a lane of the AES pipeline that the message + * length leaves idle, so it costs very nearly nothing on top of the + * encryption rather than a block of its own. */ + if (ivlen == 12 && length <= CRYPTON_GCM_FUSED_MAX_MESSAGE + && crypton_aes_cpu_options[CPU_AESNI] + && crypton_aes_cpu_options[CPU_PCLMUL]) { + crypton_gcm_fused_encrypt(output, &gcmkey->fused, key, iv, + aad, aadlen, input, length, taglen, + hpkey, sampleoff, mask); + return; + } +#endif +#ifdef WITH_ARMV8_CRYPTO + /* The same on AArch64, where the framing is what costs: composing the + * additional data, the encryption and the tag reaches each through the + * branch table, so the running state goes back to memory between them + * and a one-block header pays a reduction of its own. Measured on an + * Apple M4, a 100-byte packet is 3.3x faster taken in one call. */ + if (ivlen == 12 + && crypton_aes_cpu_options[CPU_AESNI] + && crypton_aes_cpu_options[CPU_PCLMUL]) { + crypton_aes_armv8_gcm_fused(output, gcmkey->gcm.htable, key, iv, + aad, aadlen, input, length, taglen, + hpkey, sampleoff, mask); + return; + } +#endif + crypton_aes_gcm_full_encrypt(output, gcmkey, key, iv, ivlen, aad, aadlen, + input, length, taglen); + /* copied rather than cast: the sample lands wherever the header put it + * and a block128 is read as 64-bit words */ + memcpy(&sample, output + sampleoff, 16); + crypton_aes_encrypt_block(&m, hpkey, &sample); + memcpy(mask, &m, 16); +} + +/* The same the other way, with the tag checked here rather than by the + * caller: returns 1 when it matches and 0 when it does not, comparing every + * byte either way. The plaintext is written whatever the answer, so a caller + * that gets 0 must not use it. */ +/* Decrypt, and either compare the tag or hand it back. + * + * With outtag NULL this is the verifying form: the tag is compared here, a + * byte at a time over its whole length whichever way the answer goes, and the + * answer is the return value. With outtag not NULL the computed tag is + * written there instead and the return value is 1 -- for a caller that holds + * the expected tag in a form of its own and will compare it itself. */ +static int gcm_full_decrypt(uint8_t *output, const aes_gcm_key *gcmkey, aes_key *key, + uint8_t *iv, uint32_t ivlen, + uint8_t *aad, uint32_t aadlen, + uint8_t *input, uint32_t length, + const uint8_t *tag, uint32_t taglen, + uint8_t *outtag) +{ + aes_gcm gcm; + uint8_t expected[16]; + uint32_t i; + uint8_t diff = 0; + +#ifdef WITH_GCM_FUSED + /* The same as the encryption side, and simpler: what GHASH absorbs + * here is the ciphertext, which is the input, so the multiplies need + * not wait for anything. Measured on an Intel Haswell, a 100-byte + * packet was three times the cost of encrypting one before this. */ + if (ivlen == 12 && length <= CRYPTON_GCM_FUSED_MAX_MESSAGE + && crypton_aes_cpu_options[CPU_AESNI] + && crypton_aes_cpu_options[CPU_PCLMUL]) + return crypton_gcm_fused_decrypt(output, &gcmkey->fused, key, + iv, aad, aadlen, input, + length, tag, taglen, outtag); +#endif +#ifdef WITH_ARMV8_CRYPTO + if (ivlen == 12 + && crypton_aes_cpu_options[CPU_AESNI] + && crypton_aes_cpu_options[CPU_PCLMUL]) + return crypton_aes_armv8_gcm_fused_dec(output, gcmkey->gcm.htable, + key, iv, aad, aadlen, + input, length, tag, taglen, + outtag); +#endif + memcpy(gcm.htable, gcmkey->gcm.htable, sizeof(gcm.htable)); + gcm_message_init(&gcm, iv, ivlen); + if (aadlen) + crypton_aes_gcm_aad(&gcm, aad, aadlen); + if (length) + crypton_aes_gcm_decrypt(output, &gcm, key, input, length); + crypton_aes_gcm_finish(expected, &gcm, key); + + if (outtag) { + memcpy(outtag, expected, taglen); + return 1; + } + for (i = 0; i < taglen; i++) + diff |= (uint8_t) (expected[i] ^ tag[i]); + return diff == 0; +} + +int crypton_aes_gcm_full_decrypt(uint8_t *output, const aes_gcm_key *gcmkey, aes_key *key, + uint8_t *iv, uint32_t ivlen, + uint8_t *aad, uint32_t aadlen, + uint8_t *input, uint32_t length, + const uint8_t *tag, uint32_t taglen) +{ + return gcm_full_decrypt(output, gcmkey, key, iv, ivlen, aad, aadlen, + input, length, tag, taglen, NULL); +} + +void crypton_aes_gcm_full_decrypt_tag(uint8_t *output, uint8_t *outtag, + const aes_gcm_key *gcmkey, aes_key *key, + uint8_t *iv, uint32_t ivlen, + uint8_t *aad, uint32_t aadlen, + uint8_t *input, uint32_t length, + uint32_t taglen) +{ + (void) gcm_full_decrypt(output, gcmkey, key, iv, ivlen, aad, aadlen, + input, length, NULL, taglen, outtag); +} + static inline uint8_t ccm_b0_flags(uint32_t has_adata, uint32_t m, uint32_t l) { return 8*m + l + (has_adata? 64: 0); @@ -885,18 +1214,30 @@ block128_xor((block128 *) tag, &ocb->sum_aad); } +/* + * These three reach cbits/aes/generic.c directly rather than through the + * branch table, so they run only where the portable implementation was the + * one installed -- which is what lets them read its schedule and hand its + * core four blocks at a time. The CTR entries below cannot do the same: + * they go through the branch table for the block, and so run on accelerated + * machines too. + */ void crypton_aes_generic_encrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks) { - for ( ; nb_blocks-- > 0; input++, output++) { - crypton_aes_generic_encrypt_block(output, key, input); - } + aes_sched sched; + + crypton_aes_generic_schedule(&sched, key); + crypton_aes_generic_blocks((uint8_t *) output, (const uint8_t *) input, + nb_blocks, &sched, 0); } void crypton_aes_generic_decrypt_ecb(aes_block *output, aes_key *key, aes_block *input, uint32_t nb_blocks) { - for ( ; nb_blocks-- > 0; input++, output++) { - crypton_aes_generic_decrypt_block(output, key, input); - } + aes_sched sched; + + crypton_aes_generic_schedule(&sched, key); + crypton_aes_generic_blocks((uint8_t *) output, (const uint8_t *) input, + nb_blocks, &sched, 1); } void crypton_aes_generic_encrypt_cbc(aes_block *output, aes_key *key, aes_block *iv, aes_block *input, uint32_t nb_blocks) @@ -912,18 +1253,35 @@ } } +/* + * Decryption, unlike encryption, does not wait for the block before it: + * every ciphertext block is ready at once and the chaining is a XOR + * afterwards. So four at a pass, with the ciphertext copied aside first, + * since the caller is allowed to decrypt in place and the next group's IV + * is the last ciphertext block of this one. + */ void crypton_aes_generic_decrypt_cbc(aes_block *output, aes_key *key, aes_block *ivini, aes_block *input, uint32_t nb_blocks) { - aes_block block, blocko; + aes_sched sched; aes_block iv; + uint8_t ct[64], plain[64]; - /* preload IV in block */ + crypton_aes_generic_schedule(&sched, key); block128_copy(&iv, ivini); - for ( ; nb_blocks-- > 0; input++, output++) { - block128_copy(&block, (block128 *) input); - crypton_aes_generic_decrypt_block(&blocko, key, &block); - block128_vxor((block128 *) output, &blocko, &iv); - block128_copy(&iv, &block); + while (nb_blocks > 0) { + uint32_t n = nb_blocks < 4 ? nb_blocks : 4; + uint32_t i; + + memcpy(ct, input, n * 16); + crypton_aes_generic_blocks(plain, ct, n, &sched, 1); + for (i = 0; i < n; i++) { + block128_vxor((block128 *) (output + i), + (block128 *) (plain + 16 * i), &iv); + block128_copy(&iv, (block128 *) (ct + 16 * i)); + } + input += n; + output += n; + nb_blocks -= n; } } @@ -978,7 +1336,9 @@ void crypton_aes_generic_encrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, aes_block *dataunit, uint32_t spoint, aes_block *input, uint32_t nb_blocks) { - aes_block block, tweak; + aes_sched sched; + aes_block tweak; + uint8_t buf[64], tw[64]; /* load IV and encrypt it using k2 as the tweak */ block128_copy(&tweak, dataunit); @@ -988,17 +1348,34 @@ while (spoint-- > 0) crypton_aes_generic_gf_mulx(&tweak); - for ( ; nb_blocks-- > 0; input++, output++, crypton_aes_generic_gf_mulx(&tweak)) { - block128_vxor(&block, input, &tweak); - crypton_aes_encrypt_block(&block, k1, &block); - block128_vxor(output, &block, &tweak); + crypton_aes_generic_schedule(&sched, k1); + while (nb_blocks > 0) { + uint32_t n = nb_blocks < 4 ? nb_blocks : 4; + uint32_t i; + + /* the tweaks for a group depend on nothing but each other, so + * all four are known before any block is enciphered */ + for (i = 0; i < n; i++) { + block128_copy((block128 *) (tw + 16 * i), &tweak); + block128_vxor((block128 *) (buf + 16 * i), input + i, &tweak); + crypton_aes_generic_gf_mulx(&tweak); + } + crypton_aes_generic_blocks(buf, buf, n, &sched, 0); + for (i = 0; i < n; i++) + block128_vxor(output + i, (block128 *) (buf + 16 * i), + (block128 *) (tw + 16 * i)); + input += n; + output += n; + nb_blocks -= n; } } void crypton_aes_generic_decrypt_xts(aes_block *output, aes_key *k1, aes_key *k2, aes_block *dataunit, uint32_t spoint, aes_block *input, uint32_t nb_blocks) { - aes_block block, tweak; + aes_sched sched; + aes_block tweak; + uint8_t buf[64], tw[64]; /* load IV and encrypt it using k2 as the tweak */ block128_copy(&tweak, dataunit); @@ -1008,10 +1385,25 @@ while (spoint-- > 0) crypton_aes_generic_gf_mulx(&tweak); - for ( ; nb_blocks-- > 0; input++, output++, crypton_aes_generic_gf_mulx(&tweak)) { - block128_vxor(&block, input, &tweak); - crypton_aes_decrypt_block(&block, k1, &block); - block128_vxor(output, &block, &tweak); + crypton_aes_generic_schedule(&sched, k1); + while (nb_blocks > 0) { + uint32_t n = nb_blocks < 4 ? nb_blocks : 4; + uint32_t i; + + /* the tweaks for a group depend on nothing but each other, so + * all four are known before any block is enciphered */ + for (i = 0; i < n; i++) { + block128_copy((block128 *) (tw + 16 * i), &tweak); + block128_vxor((block128 *) (buf + 16 * i), input + i, &tweak); + crypton_aes_generic_gf_mulx(&tweak); + } + crypton_aes_generic_blocks(buf, buf, n, &sched, 1); + for (i = 0; i < n; i++) + block128_vxor(output + i, (block128 *) (buf + 16 * i), + (block128 *) (tw + 16 * i)); + input += n; + output += n; + nb_blocks -= n; } } @@ -1240,6 +1632,116 @@ block128_zero(&tmp); block128_copy_bytes(&tmp, input, length); ccm_cbcmac_add(ccm, key, &tmp); + } +} + +/* + * GCM where the portable implementation is the one that will run. + * + * The same shape as the two above, with two differences: the four counter + * blocks of a group are encrypted together, which is what the bitsliced + * core is for, and the schedule is expanded once for the message rather + * than once per block. + * + * These cannot replace the generic pair. That pair also runs on an x86 + * machine that has AES-NI and no carry-less multiply, where crypton_aes.c + * installs the AES-NI block function and leaves GCM alone -- and there the + * key holds the AES-NI schedule, not this one. So the choice is made where + * the rest of the portable entries are chosen. + */ +void crypton_aes_bitsliced_gcm_encrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length) +{ + aes_sched sched; + aes_block out; + + crypton_aes_generic_schedule(&sched, key); + gcm->length_input += length; + for (; length >= 64; input += 64, output += 64, length -= 64) { + aes_block buf[4]; + int i; + + for (i = 0; i < 4; i++) { + block128_inc32_be(&gcm->civ); + block128_copy(&buf[i], &gcm->civ); + } + crypton_aes_generic_blocks((uint8_t *) buf, (const uint8_t *) buf, + 4, &sched, 0); + for (i = 0; i < 4; i++) + block128_xor(&buf[i], (block128 *) (input + 16 * i)); + gcm_ghash_add4(gcm, buf); + for (i = 0; i < 4; i++) + block128_copy((block128 *) (output + 16 * i), &buf[i]); + } + for (; length >= 16; input += 16, output += 16, length -= 16) { + block128_inc32_be(&gcm->civ); + crypton_aes_generic_blocks((uint8_t *) &out, + (const uint8_t *) &gcm->civ, 1, &sched, 0); + block128_xor(&out, (block128 *) input); + gcm_ghash_add(gcm, &out); + block128_copy((block128 *) output, &out); + } + if (length > 0) { + aes_block tmp; + uint32_t i; + + block128_inc32_be(&gcm->civ); + crypton_aes_generic_blocks((uint8_t *) &out, + (const uint8_t *) &gcm->civ, 1, &sched, 0); + block128_zero(&tmp); + block128_copy_bytes(&tmp, input, length); + block128_xor_bytes(&tmp, out.b, length); + gcm_ghash_add(gcm, &tmp); + for (i = 0; i < length; i++) + output[i] = tmp.b[i]; + } +} + +void crypton_aes_bitsliced_gcm_decrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length) +{ + aes_sched sched; + aes_block out; + + crypton_aes_generic_schedule(&sched, key); + gcm->length_input += length; + /* GHASH all four ciphertext blocks before writing any plaintext, since + * output may be input */ + for (; length >= 64; input += 64, output += 64, length -= 64) { + aes_block buf[4]; + int i; + + gcm_ghash_add4(gcm, (const block128 *) input); + for (i = 0; i < 4; i++) { + block128_inc32_be(&gcm->civ); + block128_copy(&buf[i], &gcm->civ); + } + crypton_aes_generic_blocks((uint8_t *) buf, (const uint8_t *) buf, + 4, &sched, 0); + for (i = 0; i < 4; i++) { + block128_xor(&buf[i], (block128 *) (input + 16 * i)); + block128_copy((block128 *) (output + 16 * i), &buf[i]); + } + } + for (; length >= 16; input += 16, output += 16, length -= 16) { + block128_inc32_be(&gcm->civ); + crypton_aes_generic_blocks((uint8_t *) &out, + (const uint8_t *) &gcm->civ, 1, &sched, 0); + gcm_ghash_add(gcm, (block128 *) input); + block128_xor(&out, (block128 *) input); + block128_copy((block128 *) output, &out); + } + if (length > 0) { + aes_block tmp; + uint32_t i; + + block128_inc32_be(&gcm->civ); + crypton_aes_generic_blocks((uint8_t *) &out, + (const uint8_t *) &gcm->civ, 1, &sched, 0); + block128_zero(&tmp); + block128_copy_bytes(&tmp, input, length); + gcm_ghash_add(gcm, &tmp); + block128_xor_bytes(&tmp, out.b, length); + for (i = 0; i < length; i++) + output[i] = tmp.b[i]; } } diff --git a/cbits/crypton_aes.h b/cbits/crypton_aes.h --- a/cbits/crypton_aes.h +++ b/cbits/crypton_aes.h @@ -55,6 +55,55 @@ uint64_t length_input; } aes_gcm; +/* + * How many powers of H a key keeps for the fused path in + * cbits/aes/gcm_fused_x86.c. A power for every block of the message would + * fold its whole GHASH into one reduction, which is what picotls does, but + * then the state grows with the longest message a caller might send and a + * server holding many keys pays it for each. A fixed count costs one + * reduction per this many blocks and keeps the state one size. Sixteen was + * measured against 6, 8, 32, 64, 96 and 256: above eight the choice is worth + * about two per cent, since only messages short enough to take this path at + * all reach a second batch. Six is worth avoiding -- at 1440 bytes it is + * slower than not taking the path. + */ +#define CRYPTON_GCM_FUSED_POWERS 16 + +/* + * Beyond this many bytes the stitched assembly in cbits/asm is faster than + * the fused path, so longer messages go there instead. Measured on an Intel + * Haswell: even at 1440 bytes, the assembly ahead by 12 per cent at 3 KB and + * 20 per cent at 16 KB, and the fused path ahead by 1.9x at 100 bytes and + * 1.16x at 1200. QUIC packets fall below this; TLS records do not. + */ +#define CRYPTON_GCM_FUSED_MAX_MESSAGE 1536 + +/* + * The powers themselves, each shifted up by one bit, and beside each the + * halves of it added together for the Karatsuba term. The two are kept + * adjacent rather than in two arrays: a multiply wants both, and two arrays + * put them 256 bytes apart, which is two cache lines where this is one. + * + * Defined on every platform so that the key state below is one size + * everywhere; filled only where the fused path is compiled in. + */ +typedef struct { + struct { + aes_block h; + aes_block r; + } p[CRYPTON_GCM_FUSED_POWERS]; +} aes_gcm_fused; + +/* + * Everything a key determines, built once by crypton_aes_gcm_key_init and + * read by every message sent under that key: the key half of a GCM state, + * and the powers of H the fused path reads. 832 bytes. + */ +typedef struct { + aes_gcm gcm; + aes_gcm_fused fused; +} aes_gcm_key; + /* size = 4*16+4*4= 80 */ typedef struct { aes_block xi; @@ -104,6 +153,27 @@ uint32_t spoint, aes_block *input, uint32_t nb_blocks); void crypton_aes_gcm_init(aes_gcm *gcm, aes_key *key, uint8_t *iv, uint32_t len); +void crypton_aes_gcm_key_init(aes_gcm_key *gk, aes_key *key); +void crypton_aes_gcm_full_encrypt(uint8_t *output, const aes_gcm_key *gcmkey, aes_key *key, + uint8_t *iv, uint32_t ivlen, + uint8_t *aad, uint32_t aadlen, + uint8_t *input, uint32_t length, uint32_t taglen); +void crypton_aes_gcm_full_encrypt_mask(uint8_t *output, const aes_gcm_key *gcmkey, aes_key *key, + uint8_t *iv, uint32_t ivlen, + uint8_t *aad, uint32_t aadlen, + uint8_t *input, uint32_t length, uint32_t taglen, + aes_key *hpkey, uint32_t sampleoff, uint8_t *mask); +int crypton_aes_gcm_full_decrypt(uint8_t *output, const aes_gcm_key *gcmkey, aes_key *key, + uint8_t *iv, uint32_t ivlen, + uint8_t *aad, uint32_t aadlen, + uint8_t *input, uint32_t length, + const uint8_t *tag, uint32_t taglen); +void crypton_aes_gcm_full_decrypt_tag(uint8_t *output, uint8_t *outtag, + const aes_gcm_key *gcmkey, aes_key *key, + uint8_t *iv, uint32_t ivlen, + uint8_t *aad, uint32_t aadlen, + uint8_t *input, uint32_t length, + uint32_t taglen); void crypton_aes_gcm_aad(aes_gcm *gcm, uint8_t *input, uint32_t length); void crypton_aes_gcm_encrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length); void crypton_aes_gcm_decrypt(uint8_t *output, aes_gcm *gcm, aes_key *key, uint8_t *input, uint32_t length); diff --git a/cbits/crypton_align.h b/cbits/crypton_align.h --- a/cbits/crypton_align.h +++ b/cbits/crypton_align.h @@ -3,6 +3,8 @@ #include "crypton_bitfn.h" +#include + #if (defined(__i386__)) # define UNALIGNED_ACCESS_OK #elif defined(__x86_64__) @@ -34,107 +36,119 @@ #define need_alignment(p,n) IS_ALIGNED(p,n) #endif -static inline uint32_t load_le32_aligned(const uint8_t *p) -{ - return le32_to_cpu(*((uint32_t *) p)); -} +/* + * Reading and writing a 32- or 64-bit word at a byte pointer. + * + * Through memcpy, not a cast to uint32_t * or uint64_t *. A cast is two + * things the standard does not allow -- a read of the value through the + * wrong type, and a read at an address that type is not aligned for -- and + * this file used to do both wherever UNALIGNED_ACCESS_OK is defined, which + * is i386 and x86-64. UndefinedBehaviorSanitizer reported seventy-eight + * lines of it. + * + * Every compiler crypton is built with turns a memcpy of four or eight bytes + * into the one load or store the cast used to be, so this is the same code + * with none of the licence. Where the target cannot do an unaligned load, + * the compiler is the one that knows, and it emits what the target needs -- + * which is what the byte-at-a-time versions this replaces were for. + * + * The _aligned names stay because nineteen files use them. They no longer + * ask anything of the pointer. + */ -static inline void store_le32_aligned(uint8_t *dst, const uint32_t v) +static inline uint32_t load_le32(const uint8_t *p) { - *((uint32_t *) dst) = cpu_to_le32(v); -} + uint32_t v; -static inline void xor_le32_aligned(uint8_t *dst, const uint32_t v) -{ - *((uint32_t *) dst) ^= cpu_to_le32(v); + memcpy(&v, p, sizeof(v)); + return le32_to_cpu(v); } -static inline void store_be32_aligned(uint8_t *dst, const uint32_t v) +static inline uint64_t load_le64(const uint8_t *p) { - *((uint32_t *) dst) = cpu_to_be32(v); -} + uint64_t v; -static inline void xor_be32_aligned(uint8_t *dst, const uint32_t v) -{ - *((uint32_t *) dst) ^= cpu_to_be32(v); + memcpy(&v, p, sizeof(v)); + return le64_to_cpu(v); } -static inline void store_le64_aligned(uint8_t *dst, const uint64_t v) +static inline uint32_t load_be32(const uint8_t *p) { - *((uint64_t *) dst) = cpu_to_le64(v); -} + uint32_t v; -static inline void store_be64_aligned(uint8_t *dst, const uint64_t v) -{ - *((uint64_t *) dst) = cpu_to_be64(v); + memcpy(&v, p, sizeof(v)); + return be32_to_cpu(v); } -static inline void xor_be64_aligned(uint8_t *dst, const uint64_t v) +static inline uint64_t load_be64(const uint8_t *p) { - *((uint64_t *) dst) ^= cpu_to_be64(v); -} + uint64_t v; -#ifdef UNALIGNED_ACCESS_OK -#define load_le32(a) load_le32_aligned(a) -#else -static inline uint32_t load_le32(const uint8_t *p) -{ - return ((uint32_t)p[0]) | ((uint32_t)p[1] << 8) | ((uint32_t)p[2] << 16) | ((uint32_t)p[3] << 24); + memcpy(&v, p, sizeof(v)); + return be64_to_cpu(v); } -#endif -#ifdef UNALIGNED_ACCESS_OK -#define store_le32(a, b) store_le32_aligned(a, b) -#define xor_le32(a, b) xor_le32_aligned(a, b) -#else static inline void store_le32(uint8_t *dst, const uint32_t v) { - dst[0] = v; dst[1] = v >> 8; dst[2] = v >> 16; dst[3] = v >> 24; + uint32_t w = cpu_to_le32(v); + + memcpy(dst, &w, sizeof(w)); } + static inline void xor_le32(uint8_t *dst, const uint32_t v) { - dst[0] ^= v; dst[1] ^= v >> 8; dst[2] ^= v >> 16; dst[3] ^= v >> 24; + store_le32(dst, le32_to_cpu(load_le32(dst)) ^ v); } -#endif -#ifdef UNALIGNED_ACCESS_OK -#define store_be32(a, b) store_be32_aligned(a, b) -#define xor_be32(a, b) xor_be32_aligned(a, b) -#else static inline void store_be32(uint8_t *dst, const uint32_t v) { - dst[3] = v; dst[2] = v >> 8; dst[1] = v >> 16; dst[0] = v >> 24; + uint32_t w = cpu_to_be32(v); + + memcpy(dst, &w, sizeof(w)); } + static inline void xor_be32(uint8_t *dst, const uint32_t v) { - dst[3] ^= v; dst[2] ^= v >> 8; dst[1] ^= v >> 16; dst[0] ^= v >> 24; + uint32_t w; + + memcpy(&w, dst, sizeof(w)); + w ^= cpu_to_be32(v); + memcpy(dst, &w, sizeof(w)); } -#endif -#ifdef UNALIGNED_ACCESS_OK -#define store_le64(a, b) store_le64_aligned(a, b) -#else static inline void store_le64(uint8_t *dst, const uint64_t v) { - dst[0] = v ; dst[1] = v >> 8 ; dst[2] = v >> 16; dst[3] = v >> 24; - dst[4] = v >> 32; dst[5] = v >> 40; dst[6] = v >> 48; dst[7] = v >> 56; + uint64_t w = cpu_to_le64(v); + + memcpy(dst, &w, sizeof(w)); } -#endif -#ifdef UNALIGNED_ACCESS_OK -#define store_be64(a, b) store_be64_aligned(a, b) -#define xor_be64(a, b) xor_be64_aligned(a, b) -#else static inline void store_be64(uint8_t *dst, const uint64_t v) { - dst[7] = v ; dst[6] = v >> 8 ; dst[5] = v >> 16; dst[4] = v >> 24; - dst[3] = v >> 32; dst[2] = v >> 40; dst[1] = v >> 48; dst[0] = v >> 56; + uint64_t w = cpu_to_be64(v); + + memcpy(dst, &w, sizeof(w)); } + static inline void xor_be64(uint8_t *dst, const uint64_t v) { - dst[7] ^= v ; dst[6] ^= v >> 8 ; dst[5] ^= v >> 16; dst[4] ^= v >> 24; - dst[3] ^= v >> 32; dst[2] ^= v >> 40; dst[1] ^= v >> 48; dst[0] ^= v >> 56; + uint64_t w; + + memcpy(&w, dst, sizeof(w)); + w ^= cpu_to_be64(v); + memcpy(dst, &w, sizeof(w)); } -#endif + +#define load_le32_aligned(p) load_le32(p) +#define load_le64_aligned(p) load_le64(p) +#define load_be32_aligned(p) load_be32(p) +#define load_be64_aligned(p) load_be64(p) +#define store_le32_aligned(d, v) store_le32(d, v) +#define xor_le32_aligned(d, v) xor_le32(d, v) +#define store_be32_aligned(d, v) store_be32(d, v) +#define xor_be32_aligned(d, v) xor_be32(d, v) +#define store_le64_aligned(d, v) store_le64(d, v) +#define store_be64_aligned(d, v) store_be64(d, v) +#define xor_be64_aligned(d, v) xor_be64(d, v) #endif diff --git a/cbits/crypton_armv8_target.h b/cbits/crypton_armv8_target.h new file mode 100644 --- /dev/null +++ b/cbits/crypton_armv8_target.h @@ -0,0 +1,52 @@ +/* + * Asking for an AArch64 extension on one function. + * + * The instructions these files use are extensions, so a translation unit + * compiled for the baseline may not emit them, and the function that does has + * to say which extension it needs. The two compilers spell that differently + * and did not always: + * + * clang target("+crypto") for as long as it matters + * GCC 13 and up target("+crypto") as well + * GCC before 13 target("arch=armv8-a+crypto") -- the bare "+feature" form + * is not understood, and the extension never reaches the + * function, so an always_inline intrinsic that needs it + * fails to inline and the build stops + * + * That last case is #273: gcc 12.2 on an aarch64 Linux could not build + * cbits/sha3_armv8.c at all. Naming the architecture as well as the + * extension is understood by every version of both compilers, so GCC is given + * that spelling and clang keeps the shorter one, which leaves whatever + * baseline the caller chose alone. + */ +#ifndef CRYPTON_ARMV8_TARGET_H +#define CRYPTON_ARMV8_TARGET_H + +#ifdef WITH_TARGET_ATTRIBUTES +#if defined(__arm__) +/* + * AArch32 asks for an FPU rather than an architecture extension, and the + * spelling differs between the compilers in ways that cannot be checked from + * here. Rather than guess, the cabal file passes -mfpu=crypto-neon-fp-armv8 + * for the whole component on this architecture whatever this flag says, and + * these expand to nothing. A global target option is safe because a + * compiler emits these instructions where an intrinsic asks for them and + * nowhere else. + */ +#define CRYPTON_TARGET_ARMV8_CRYPTO +#define CRYPTON_TARGET_ARMV8_SHA3 +#elif defined(__clang__) +#define CRYPTON_TARGET_ARMV8_CRYPTO __attribute__((target("+crypto"))) +#define CRYPTON_TARGET_ARMV8_SHA3 __attribute__((target("+sha3"))) +#else +#define CRYPTON_TARGET_ARMV8_CRYPTO \ + __attribute__((target("arch=armv8-a+crypto"))) +#define CRYPTON_TARGET_ARMV8_SHA3 \ + __attribute__((target("arch=armv8.2-a+sha3"))) +#endif +#else +#define CRYPTON_TARGET_ARMV8_CRYPTO +#define CRYPTON_TARGET_ARMV8_SHA3 +#endif + +#endif diff --git a/cbits/crypton_bignum.h b/cbits/crypton_bignum.h --- a/cbits/crypton_bignum.h +++ b/cbits/crypton_bignum.h @@ -29,11 +29,6 @@ #define LIMB_BYTES (LIMB_BITS / 8) -/* four bits of exponent per window, so a table of sixteen and no leftover - * bits: a byte holds exactly two windows */ -#define WINDOW_BITS 4 -#define TABLE_SIZE (1 << WINDOW_BITS) - /* r = a - b, returning the borrow out of the top */ static inline limb_t sub_n(limb_t *r, const limb_t *a, const limb_t *b, uint32_t n) { @@ -112,9 +107,196 @@ return (limb_t) 0 - inv; } -/* t += a * b over n limbs, returning the carry. This is where nearly all of - * the time goes, so the limbs are taken eight at a time; what is left over at - * the end is taken one at a time. */ +/* + * t += a * b over n limbs, returning the carry. This is where nearly all of + * the time goes. + * + * The C below takes the limbs eight at a time; what is left over at the end + * is taken one at a time. On AArch64 the compiler writes each limb as + * `mul`, `umulh`, `adds`, `cset`, `adds`, `adc`: the carry out of one + * 128-bit addition leaves the flags for a general register and is added back + * in the next, because in C each addition is a statement of its own. Two of + * those instructions are that round trip. + * + * Three attempts to take them back measured worse than the C, and are + * written down here so that they are not tried again -- one RSA-2048 CRT + * private operation on an Apple M4, best of many: + * + * this loop in inline assembly, a carry chain per limb 654.7 us + * the same with the loads hoisted out of the chain 644.5 + * the multiply interleaved with its reduction (CIOS) 604.1 + * the C below 590.1 + * what the AArch64 block does, four limbs and two chains 514.9 + * the same, with the ragged end of the row written out too 503.4 + * + * The first two lose because a chain per limb serialises what the spare + * `cset` lets overlap: `adds`, `adc`, `adds`, `adc` is four dependent steps + * per limb, and a wide out-of-order core would rather have the extra + * instruction than the dependency. The third loses because shifting the + * accumulator down a limb each round costs more than the round trip through + * 2n limbs that it saves. What works is neither: four limbs to an + * iteration with the flags carrying through two long chains, one for the low + * halves of the products and one for the high halves a place up. + * + * There is more still there. OpenSSL's armv8-mont.pl runs a 16-limb + * Montgomery multiplication at about 0.98 multiply-accumulates per cycle; + * this file was at 0.55 and the block below brings it to 0.64, where 1.0 is + * the ceiling -- a multiply-accumulate is two instructions and the machine + * issues two multiplies a cycle. That code cannot be borrowed: it is in + * OpenSSL's tree only, under Apache-2.0, and CRYPTOGAMS, which this library + * does vendor from, publishes no Montgomery generator at all. BearSSL's + * only ARM assembly is 32-bit Thumb for Cortex-M0 to M3, with fifteen-bit + * limbs for cores that have no fast multiplier, and Botan's AArch64 inline + * assembly is the `mul`/`umulh`/`adds`/`adc` primitive the compiler already + * emits. + */ +#if defined(__aarch64__) && LIMB_BITS == 64 \ + && (defined(__GNUC__) || defined(__clang__)) +/* + * Four limbs to an iteration, accumulated in two chains rather than one per + * limb: the low halves of the four products, with the carry coming in, are + * one run of `adds` and `adcs`, and the high halves shifted up a place are + * another. The flags carry the whole way through each, which is what the C + * above cannot say and what it pays for in `cset` and an extra add. + * + * The arrangement is the one in Go's crypto/internal/fips140/bigmod + * (nat_arm64.s, addMulVVWx), which is BSD-3-Clause like this library -- + * cbits/LICENSE.go carries its notice. Written out here in the assembler + * this file's compiler speaks. + */ +static inline limb_t addmul_1(limb_t *t, const limb_t *a, uint32_t n, limb_t b) +{ + uint64_t carry = 0; + uint64_t x0, x1, x2, x3, z0, z1, z2, z3; + uint64_t l0, l1, l2, l3, h0, h1, h2, h3; + uint64_t blocks = n / 4, left = n % 4; + + if (blocks) { + __asm__ volatile( + "1:\n\t" + "ldp %[x0], %[x1], [%[a]], #16\n\t" + "ldp %[x2], %[x3], [%[a]], #16\n\t" + "ldp %[z0], %[z1], [%[t]]\n\t" + /* the low halves, one place up from the second chain, with + * the carry that came in */ + "adds %[z0], %[z0], %[c]\n\t" + "mul %[l1], %[x1], %[b]\n\t" + "adcs %[z1], %[z1], %[l1]\n\t" + "mul %[l2], %[x2], %[b]\n\t" + "ldp %[z2], %[z3], [%[t], #16]\n\t" + "adcs %[z2], %[z2], %[l2]\n\t" + "mul %[l3], %[x3], %[b]\n\t" + "adcs %[z3], %[z3], %[l3]\n\t" + "umulh %[h3], %[x3], %[b]\n\t" + "adc %[h3], %[h3], xzr\n\t" + /* and the high halves, which is where this block's own carry + * ends up */ + "mul %[l0], %[x0], %[b]\n\t" + "adds %[z0], %[z0], %[l0]\n\t" + "umulh %[h0], %[x0], %[b]\n\t" + "adcs %[z1], %[z1], %[h0]\n\t" + "umulh %[h1], %[x1], %[b]\n\t" + "stp %[z0], %[z1], [%[t]], #16\n\t" + "adcs %[z2], %[z2], %[h1]\n\t" + "umulh %[h2], %[x2], %[b]\n\t" + "adcs %[z3], %[z3], %[h2]\n\t" + "stp %[z2], %[z3], [%[t]], #16\n\t" + "adc %[c], %[h3], xzr\n\t" + "subs %[k], %[k], #1\n\t" + "b.ne 1b\n\t" + : [a] "+r"(a), [t] "+r"(t), [c] "+r"(carry), [k] "+r"(blocks), + [x0] "=&r"(x0), [x1] "=&r"(x1), [x2] "=&r"(x2), + [x3] "=&r"(x3), [z0] "=&r"(z0), [z1] "=&r"(z1), + [z2] "=&r"(z2), [z3] "=&r"(z3), [l0] "=&r"(l0), + [l1] "=&r"(l1), [l2] "=&r"(l2), [l3] "=&r"(l3), + [h0] "=&r"(h0), [h1] "=&r"(h1), [h2] "=&r"(h2), + [h3] "=&r"(h3) + : [b] "r"(b) + : "cc", "memory"); + } + /* What is left of the row: three limbs, two, or one, each the same two + * chains cut short. This is not a rare case to be handed back to C -- + * mont_sqr asks for every length from n-1 down to 1, so three rows in + * four end ragged. Only 4.7% of the limbs in a 1024-bit exponentiation + * arrive here, but they were the dearer ones, and writing them out is + * worth the last two per cent in the table above. The three lengths + * are spelled out rather than run as 2+1, because chaining two short + * blocks makes the second wait on the first: that costs two thirds of + * the gain. + */ + if (left == 3) { + __asm__ volatile( + "ldp %[x0], %[x1], [%[a]]\n\t" + "ldr %[x2], [%[a], #16]\n\t" + "add %[a], %[a], #24\n\t" + "ldp %[z0], %[z1], [%[t]]\n\t" + "ldr %[z2], [%[t], #16]\n\t" + "adds %[z0], %[z0], %[c]\n\t" + "mul %[l1], %[x1], %[b]\n\t" + "adcs %[z1], %[z1], %[l1]\n\t" + "mul %[l2], %[x2], %[b]\n\t" + "adcs %[z2], %[z2], %[l2]\n\t" + "umulh %[h2], %[x2], %[b]\n\t" + "adc %[h2], %[h2], xzr\n\t" + "mul %[l0], %[x0], %[b]\n\t" + "adds %[z0], %[z0], %[l0]\n\t" + "umulh %[h0], %[x0], %[b]\n\t" + "adcs %[z1], %[z1], %[h0]\n\t" + "umulh %[h1], %[x1], %[b]\n\t" + "stp %[z0], %[z1], [%[t]], #16\n\t" + "adcs %[z2], %[z2], %[h1]\n\t" + "str %[z2], [%[t]], #8\n\t" + "adc %[c], %[h2], xzr\n\t" + : [a] "+r"(a), [t] "+r"(t), [c] "+r"(carry), + [x0] "=&r"(x0), [x1] "=&r"(x1), [x2] "=&r"(x2), + [z0] "=&r"(z0), [z1] "=&r"(z1), [z2] "=&r"(z2), + [l0] "=&r"(l0), [l1] "=&r"(l1), [l2] "=&r"(l2), + [h0] "=&r"(h0), [h1] "=&r"(h1), [h2] "=&r"(h2) + : [b] "r"(b) + : "cc", "memory"); + } else if (left == 2) { + __asm__ volatile( + "ldp %[x0], %[x1], [%[a]], #16\n\t" + "ldp %[z0], %[z1], [%[t]]\n\t" + "adds %[z0], %[z0], %[c]\n\t" + "mul %[l1], %[x1], %[b]\n\t" + "adcs %[z1], %[z1], %[l1]\n\t" + "umulh %[h1], %[x1], %[b]\n\t" + "adc %[h1], %[h1], xzr\n\t" + "mul %[l0], %[x0], %[b]\n\t" + "adds %[z0], %[z0], %[l0]\n\t" + "umulh %[h0], %[x0], %[b]\n\t" + "adcs %[z1], %[z1], %[h0]\n\t" + "stp %[z0], %[z1], [%[t]], #16\n\t" + "adc %[c], %[h1], xzr\n\t" + : [a] "+r"(a), [t] "+r"(t), [c] "+r"(carry), + [x0] "=&r"(x0), [x1] "=&r"(x1), [z0] "=&r"(z0), + [z1] "=&r"(z1), [l0] "=&r"(l0), [l1] "=&r"(l1), + [h0] "=&r"(h0), [h1] "=&r"(h1) + : [b] "r"(b) + : "cc", "memory"); + } else if (left == 1) { + __asm__ volatile( + "ldr %[x0], [%[a]], #8\n\t" + "ldr %[z0], [%[t]]\n\t" + "mul %[l0], %[x0], %[b]\n\t" + "adds %[z0], %[z0], %[c]\n\t" + "umulh %[h0], %[x0], %[b]\n\t" + "adc %[h0], %[h0], xzr\n\t" + "adds %[z0], %[z0], %[l0]\n\t" + "str %[z0], [%[t]], #8\n\t" + "adc %[c], %[h0], xzr\n\t" + : [a] "+r"(a), [t] "+r"(t), [c] "+r"(carry), + [x0] "=&r"(x0), [z0] "=&r"(z0), [l0] "=&r"(l0), + [h0] "=&r"(h0) + : [b] "r"(b) + : "cc", "memory"); + } + return carry; +} +#else +/* one limb of it, so that the loops below can say how many they take at a + * time without saying the rest of it four times over */ #define ADDMUL_STEP(k) \ p = (dlimb_t) a[i + (k)] * b + t[i + (k)] + carry; \ t[i + (k)] = (limb_t) p; \ @@ -141,6 +323,7 @@ } return carry; } +#endif /* r = t * R^-1 mod m, with t of 2n limbs and destroyed on the way */ static inline void mont_reduce(limb_t *r, limb_t *t, const limb_t *m, limb_t n0, @@ -165,14 +348,33 @@ select_n(r, r, t + n, take & 1, n); } +/* t = a * b, the low n limbs, returning the limb above them: addmul_1 with + * nothing to add to, for the row of a product that lands on empty space. */ +static inline limb_t mul_1(limb_t *t, const limb_t *a, uint32_t n, limb_t b) +{ + limb_t carry = 0; + uint32_t i; + + for (i = 0; i < n; i++) { + dlimb_t p = (dlimb_t) a[i] * b + carry; + + t[i] = (limb_t) p; + carry = (limb_t) (p >> LIMB_BITS); + } + return carry; +} + /* r = a * b * R^-1 mod m, with t of 2n limbs */ static inline void mont_mul(limb_t *r, const limb_t *a, const limb_t *b, const limb_t *m, limb_t n0, uint32_t n, limb_t *t) { uint32_t i; - memset(t, 0, 2 * n * sizeof(limb_t)); - for (i = 0; i < n; i++) + /* Nothing has to be cleared first. The first row lands on empty space + * and writes t[0 .. n], and every row after it reads t[i .. i+n-1], + * whose top limb is the one the row before it wrote. */ + t[n] = mul_1(t, a, n, b[0]); + for (i = 1; i < n; i++) t[n + i] = addmul_1(t + i, a, n, b[i]); mont_reduce(r, t, m, n0, n); } @@ -186,8 +388,14 @@ limb_t carry = 0; uint32_t i; - memset(t, 0, 2 * n * sizeof(limb_t)); - for (i = 0; i + 1 < n; i++) + /* The first row lands on empty space here too, on t[1 .. n], and the + * rows between them write t[1 .. 2n-2] before any of it is read. The + * diagonal below is the only reader of the two ends. */ + t[0] = 0; + t[2 * n - 1] = 0; + if (n > 1) + t[n] = mul_1(t + 1, a + 1, n - 1, a[0]); + for (i = 1; i + 1 < n; i++) t[n + i] = addmul_1(t + i + i + 1, a + i + 1, n - 1 - i, a[i]); shl1(t, 2 * n); /* their sum is under half of what 2n limbs hold */ for (i = 0; i < n; i++) { @@ -201,15 +409,54 @@ mont_reduce(r, t, m, n0, n); } -/* r2 = R^2 mod m, by doubling +/* a = 2a mod m, for an a already under m */ +static inline void dbl_mod(limb_t *a, const limb_t *m, uint32_t n, limb_t *tmp) +{ + limb_t carry = shl1(a, n); + limb_t borrow = sub_n(tmp, a, m, n); + + select_n(a, tmp, a, (carry | (borrow ^ 1)) & 1, n); +} + +/* r2 = R^2 mod m, where R is 2^(n * LIMB_BITS) * + * Doubling the whole way there is 2n * LIMB_BITS steps, and at RSA sizes + * that is a tenth of the exponentiation it is setting up for. Only the + * first half of it has to be done a bit at a time. + * + * Write a value as 2^(lgR + d) mod m. A Montgomery squaring divides by R, + * so it takes that to 2(lgR + d) - lgR = lgR + 2d: it doubles d. So double + * up to R mod m, where d is zero, take one more step to make d one, and then + * climb to d = lgR by the binary expansion of lgR -- a squaring for each bit + * and one more doubling where the bit is set. For a 1024-bit modulus that + * is ten squarings in place of a thousand and twenty-five doublings, and on + * an Apple M4 that is 2.1 microseconds against 24.7. + * * Doubling starts at the highest power of two under the modulus rather than * at one, since everything below that power is where doubling would go - * anyway: for a modulus that fills its limbs that is half the steps. + * anyway: for a modulus that fills its limbs, that first half is one step. + * + * What the trip counts depend on is the modulus' length, which is what the + * doubling loop showed as well, and nothing else about it. */ -static inline void mont_r2(limb_t *r2, const limb_t *m, uint32_t n, limb_t *tmp) +/* s2n-bignum's bignum_montifier computes the same constant in assembly, on + * both architectures, and is the faster of the two by a good margin -- the + * C cannot form the carry chains it does. It is used where it is built, + * and the C below is what everywhere else gets and what it is checked + * against. It needs a scratch buffer of n limbs, which every caller + * already passes. */ +#if defined(CRYPTON_S2N_BIGNUM) && !defined(CRYPTON_NO_MONTIFIER) \ + && (defined(__x86_64__) || defined(__aarch64__)) && LIMB_BITS == 64 +#define CRYPTON_MONTIFIER 1 +extern void bignum_montifier(uint64_t k, uint64_t *z, const uint64_t *m, + uint64_t *t); +#endif + +static inline void mont_r2_c(limb_t *r2, const limb_t *m, limb_t n0, uint32_t n, + limb_t *t) { - uint32_t i, k = 0, steps; + uint32_t lgr = n * LIMB_BITS; + uint32_t i, k = 0, msb = 0; for (i = n; i > 0 && k == 0; i--) if (m[i - 1] != 0) { @@ -225,12 +472,28 @@ if (k == 0) return; /* a modulus of nothing, which the caller rules out */ r2[(k - 1) / LIMB_BITS] = (limb_t) 1 << ((k - 1) % LIMB_BITS); - steps = 2 * n * LIMB_BITS - (k - 1); - for (i = 0; i < steps; i++) { - limb_t carry = shl1(r2, n); - limb_t borrow = sub_n(tmp, r2, m, n); - select_n(r2, tmp, r2, (carry | (borrow ^ 1)) & 1, n); + for (i = k - 1; i < lgr; i++) + dbl_mod(r2, m, n, t); + /* r2 is R mod m, so d is zero and squaring would leave it there */ + dbl_mod(r2, m, n, t); + while ((lgr >> (msb + 1)) != 0) + msb++; + for (i = msb; i > 0; i--) { + mont_sqr(r2, r2, m, n0, n, t); + if ((lgr >> (i - 1)) & 1) + dbl_mod(r2, m, n, t); } +} + +static inline void mont_r2(limb_t *r2, const limb_t *m, limb_t n0, uint32_t n, + limb_t *t) +{ +#ifdef CRYPTON_MONTIFIER + (void) n0; + bignum_montifier(n, r2, m, t); +#else + mont_r2_c(r2, m, n0, n, t); +#endif } /* big-endian bytes into limbs, least significant limb first; anything above diff --git a/cbits/crypton_bitfn.h b/cbits/crypton_bitfn.h --- a/cbits/crypton_bitfn.h +++ b/cbits/crypton_bitfn.h @@ -127,23 +127,6 @@ } #endif -#ifndef ARCH_HAS_MEMORY_ZERO -static inline void memory_zero(void *ptr, uint32_t len) -{ - uint32_t *ptr32 = ptr; - uint8_t *ptr8; - int i; - - for (i = 0; i < len / 4; i++) - *ptr32++ = 0; - if (len % 4) { - ptr8 = (uint8_t *) ptr32; - for (i = len % 4; i >= 0; i--) - ptr8[i] = 0; - } -} -#endif - #ifndef ARCH_HAS_ARRAY_COPY32 static inline void array_copy32(uint32_t *d, uint32_t *s, uint32_t nb) { diff --git a/cbits/crypton_bzero.h b/cbits/crypton_bzero.h new file mode 100644 --- /dev/null +++ b/cbits/crypton_bzero.h @@ -0,0 +1,27 @@ +/* + * Erasing memory that the compiler is entitled to decide nobody reads. + * + * memset on an object that is about to die -- freed, or a local going out of + * scope -- is a store to memory nothing can observe, and an optimizer may + * drop it. That is the whole reason explicit_bzero and memset_s exist. + * Neither is everywhere, so this writes through a volatile pointer, which the + * standard says cannot be elided. + * + * Use it wherever key material stops being needed. Plain memset is still + * right for memory that is about to be read again. + */ +#ifndef CRYPTON_BZERO_H +#define CRYPTON_BZERO_H + +#include +#include + +static inline void crypton_bzero(void *p, size_t n) +{ + volatile uint8_t *q = (volatile uint8_t *)p; + + while (n--) + *q++ = 0; +} + +#endif diff --git a/cbits/crypton_chacha.c b/cbits/crypton_chacha.c --- a/cbits/crypton_chacha.c +++ b/cbits/crypton_chacha.c @@ -51,7 +51,7 @@ const crypton_chacha_state *in); void crypton_chacha_simd_generate(int rounds, uint8_t *dst, const crypton_chacha_state *in); -/* The counters in a group must not carry into d[13], which the block loop +/* The counters in a group must not carry into d[13], which the crypton_chacha_block loop * below handles and the vector one does not; that is one run in 2^29. */ #define CHACHA_SIMD_OK(st, n) ((st)->d[12] <= 0xffffffffU - (uint32_t) (n)) #endif @@ -89,7 +89,7 @@ * How much is worth handing over. On AArch64 the module's vector path * starts at three blocks and below that its scalar path measures level with * the C here, so there is nothing to gain; on x86-64 it is ahead from one - * block, the C there having no vector path until eight. + * crypton_chacha_block, the C there having no vector path until eight. */ #ifndef CHACHA_ASM_MIN_BLOCKS #ifdef WITH_X86_CHACHA_ASM @@ -109,10 +109,11 @@ #define ALIGNED64(PTR) \ (((uintptr_t)(const void *)(PTR)) % 8 == 0) -static const uint8_t sigma[16] = "expand 32-byte k"; -static const uint8_t tau[16] = "expand 16-byte k"; +/* sixteen bytes are read; the seventeenth is the string's terminator */ +static const uint8_t sigma[17] = "expand 32-byte k"; +static const uint8_t tau[17] = "expand 16-byte k"; -static void chacha_core(int rounds, block *out, const crypton_chacha_state *in) +static void chacha_core(int rounds, crypton_chacha_block *out, const crypton_chacha_state *in) { uint32_t x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, x14, x15; int i; @@ -296,7 +297,7 @@ void crypton_chacha_combine(uint8_t *dst, crypton_chacha_context *ctx, const uint8_t *src, uint32_t bytes) { - block out; + crypton_chacha_block out; crypton_chacha_state *st; int i; @@ -384,7 +385,7 @@ } /* xor as much as needed */ - for (i = 0; i < bytes; i++) + for (i = 0; i < (int) bytes; i++) dst[i] = src[i] ^ out.b[i]; /* copy the left over in the buffer */ @@ -434,7 +435,7 @@ void crypton_chacha_generate(uint8_t *dst, crypton_chacha_context *ctx, uint32_t bytes) { crypton_chacha_state *st; - block out; + crypton_chacha_block out; int i; if (!bytes) @@ -474,7 +475,7 @@ /* xor new 64-bytes chunks and store the left over if any */ for (; bytes >= 64; bytes -= 64, dst += 64) { /* generate new chunk and update state */ - chacha_core(ctx->nb_rounds, (block *) dst, st); + chacha_core(ctx->nb_rounds, (crypton_chacha_block *) dst, st); uint32_t t0 = le32_to_cpu(st->d[12]); st->d[12] = cpu_to_le32(t0 + 1); if (st->d[12] == 0) { @@ -510,7 +511,7 @@ } /* xor as much as needed */ - for (i = 0; i < bytes; i++) + for (i = 0; i < (int) bytes; i++) dst[i] = out.b[i]; /* copy the left over in the buffer */ @@ -524,9 +525,9 @@ void crypton_chacha_generate_simple_block(uint8_t *dst, crypton_chacha_state *st, uint8_t rounds) { if (ALIGNED64(dst)) { - chacha_core(rounds, (block *) dst, st); + chacha_core(rounds, (crypton_chacha_block *) dst, st); } else { - block out; + crypton_chacha_block out; int i; chacha_core(rounds, &out, st); for (i = 0; i < 64; ++i) { @@ -544,7 +545,7 @@ void crypton_chacha_random(uint32_t rounds, uint8_t *dst, crypton_chacha_state *st, uint32_t bytes) { - block out; + crypton_chacha_block out; if (!bytes) return; diff --git a/cbits/crypton_chacha.h b/cbits/crypton_chacha.h --- a/cbits/crypton_chacha.h +++ b/cbits/crypton_chacha.h @@ -34,9 +34,9 @@ uint64_t q[8]; uint32_t d[16]; uint8_t b[64]; -} block; +} crypton_chacha_block; -typedef block crypton_chacha_state; +typedef crypton_chacha_block crypton_chacha_state; typedef struct { crypton_chacha_state st; diff --git a/cbits/crypton_chachapoly.c b/cbits/crypton_chachapoly.c new file mode 100644 --- /dev/null +++ b/cbits/crypton_chachapoly.c @@ -0,0 +1,156 @@ +/* + * Copyright (c) 2026 Kazu Yamamoto + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS + * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + +#include +#include + +#include "crypton_chacha.h" +#include "crypton_chachapoly.h" +#include "crypton_poly1305.h" + +/* RFC 8439. The one-time Poly1305 key is the first 32 bytes of the ChaCha20 + * keystream at counter 0; a whole 64-byte block is generated so the counter + * lands on 1, which is where the message starts. */ +static void chachapoly_start(crypton_chacha_context *cctx, poly1305_ctx *pctx, + const uint8_t *key, + const uint8_t *nonce, uint32_t noncelen) +{ + uint8_t block[64]; + + crypton_chacha_init(cctx, 20, 32, key, noncelen, nonce); + crypton_chacha_generate(block, cctx, sizeof(block)); + crypton_poly1305_init(pctx, (poly1305_key *) block); + memset(block, 0, sizeof(block)); +} + +/* Poly1305 over an associated or encrypted part, then zeros up to the next + * multiple of sixteen. */ +static void absorb_padded(poly1305_ctx *pctx, const uint8_t *p, uint32_t len) +{ + static const uint8_t zeros[16] = {0}; + uint32_t rem; + + if (len) + crypton_poly1305_update(pctx, (uint8_t *) p, len); + rem = len % 16; + if (rem) + crypton_poly1305_update(pctx, (uint8_t *) zeros, 16 - rem); +} + +/* The two lengths, little endian, eight bytes each, which is what the tag + * ends on. */ +static void absorb_lengths(poly1305_ctx *pctx, uint32_t aadlen, uint32_t inlen) +{ + uint8_t lens[16]; + int i; + + for (i = 0; i < 8; i++) + lens[i] = (uint8_t) (((uint64_t) aadlen) >> (8 * i)); + for (i = 0; i < 8; i++) + lens[8 + i] = (uint8_t) (((uint64_t) inlen) >> (8 * i)); + crypton_poly1305_update(pctx, lens, sizeof(lens)); +} + +void crypton_chachapoly_encrypt(uint8_t *out, uint8_t *tag, uint32_t taglen, + const uint8_t *key, + const uint8_t *nonce, uint32_t noncelen, + const uint8_t *aad, uint32_t aadlen, + const uint8_t *input, uint32_t inlen) +{ + crypton_chacha_context cctx; + poly1305_ctx pctx; + poly1305_mac mac; + + chachapoly_start(&cctx, &pctx, key, nonce, noncelen); + absorb_padded(&pctx, aad, aadlen); + if (inlen) + crypton_chacha_combine(out, &cctx, input, inlen); + /* what the tag covers is the ciphertext, which is now in out */ + absorb_padded(&pctx, out, inlen); + absorb_lengths(&pctx, aadlen, inlen); + crypton_poly1305_finalize(mac, &pctx); + memcpy(tag, mac, taglen); + + memset(&cctx, 0, sizeof(cctx)); + memset(&pctx, 0, sizeof(pctx)); +} + +/* Shared by the two decrypting entry points: with outtag NULL the tag is + * compared here and the answer returned, otherwise it is written there. */ +static int chachapoly_decrypt(uint8_t *out, const uint8_t *tag, uint32_t taglen, + uint8_t *outtag, const uint8_t *key, + const uint8_t *nonce, uint32_t noncelen, + const uint8_t *aad, uint32_t aadlen, + const uint8_t *input, uint32_t inlen) +{ + crypton_chacha_context cctx; + poly1305_ctx pctx; + poly1305_mac mac; + uint8_t diff = 0; + uint32_t i; + + chachapoly_start(&cctx, &pctx, key, nonce, noncelen); + absorb_padded(&pctx, aad, aadlen); + /* here the ciphertext is the input, so the tag can be taken before the + * plaintext is written and out may alias input */ + absorb_padded(&pctx, input, inlen); + absorb_lengths(&pctx, aadlen, inlen); + crypton_poly1305_finalize(mac, &pctx); + + if (inlen) + crypton_chacha_combine(out, &cctx, input, inlen); + + memset(&cctx, 0, sizeof(cctx)); + memset(&pctx, 0, sizeof(pctx)); + + if (outtag) { + memcpy(outtag, mac, taglen); + return 1; + } + for (i = 0; i < taglen; i++) + diff |= (uint8_t) (mac[i] ^ tag[i]); + return diff == 0; +} + +int crypton_chachapoly_decrypt(uint8_t *out, + const uint8_t *tag, uint32_t taglen, + const uint8_t *key, + const uint8_t *nonce, uint32_t noncelen, + const uint8_t *aad, uint32_t aadlen, + const uint8_t *input, uint32_t inlen) +{ + return chachapoly_decrypt(out, tag, taglen, NULL, key, nonce, noncelen, + aad, aadlen, input, inlen); +} + +void crypton_chachapoly_decrypt_tag(uint8_t *out, uint8_t *outtag, + uint32_t taglen, const uint8_t *key, + const uint8_t *nonce, uint32_t noncelen, + const uint8_t *aad, uint32_t aadlen, + const uint8_t *input, uint32_t inlen) +{ + (void) chachapoly_decrypt(out, NULL, taglen, outtag, key, nonce, + noncelen, aad, aadlen, input, inlen); +} diff --git a/cbits/crypton_chachapoly.h b/cbits/crypton_chachapoly.h new file mode 100644 --- /dev/null +++ b/cbits/crypton_chachapoly.h @@ -0,0 +1,62 @@ +/* + * Copyright (c) 2026 Kazu Yamamoto + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS + * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + +#ifndef CRYPTON_CHACHAPOLY_H +#define CRYPTON_CHACHAPOLY_H + +#include + +/* ChaCha20-Poly1305 (RFC 8439) as one call. + * + * The pieces are the ChaCha20 and Poly1305 already here; what these do is + * hold them together, which the Haskell above used to do at the cost of eight + * foreign calls and the allocations between them. + * + * The nonce is the twelve bytes RFC 8439 defines. taglen is at most 16. + */ + +void crypton_chachapoly_encrypt(uint8_t *out, uint8_t *tag, uint32_t taglen, + const uint8_t *key, + const uint8_t *nonce, uint32_t noncelen, + const uint8_t *aad, uint32_t aadlen, + const uint8_t *input, uint32_t inlen); + +/* Decrypt and compare, a byte at a time over the whole tag whichever way the + * answer goes. Returns non-zero when the tag matched. */ +int crypton_chachapoly_decrypt(uint8_t *out, + const uint8_t *tag, uint32_t taglen, + const uint8_t *key, + const uint8_t *nonce, uint32_t noncelen, + const uint8_t *aad, uint32_t aadlen, + const uint8_t *input, uint32_t inlen); + +/* Decrypt and hand the computed tag back rather than comparing it. */ +void crypton_chachapoly_decrypt_tag(uint8_t *out, uint8_t *outtag, + uint32_t taglen, const uint8_t *key, + const uint8_t *nonce, uint32_t noncelen, + const uint8_t *aad, uint32_t aadlen, + const uint8_t *input, uint32_t inlen); + +#endif diff --git a/cbits/crypton_cpu.c b/cbits/crypton_cpu.c --- a/cbits/crypton_cpu.c +++ b/cbits/crypton_cpu.c @@ -32,6 +32,23 @@ #include /* + * PE has no way to say "hidden": every symbol in an object is local to the + * image unless something exports it, which is what hidden asks for + * elsewhere, so nothing is lost by dropping the attribute here. Saying it + * anyway is not harmless -- the gcc that GHC 9.2 ships for Windows parses + * the attribute, discards it and warns, and it is the only warning crypton's + * own code produces anywhere in the CI matrix. Measured on mingw gcc 13.2.0 + * and clang 14.0.6 (the compiler GHC 9.4 and later ship): gcc warns for + * "hidden" and is silent for "default", clang is silent for both, which is + * why the vendored decaf and argon2 headers ask for "default" unnoticed. + */ +#if defined(_WIN32) || defined(__CYGWIN__) +#define CRYPTON_HIDDEN +#else +#define CRYPTON_HIDDEN __attribute__((visibility("hidden"))) +#endif + +/* * The word the assembly reads; crypton_cpu.h says what is in it. Hidden, * so that the reference to it from the assembly resolves at link time in a * shared object as well as a static one. The SHA-256 bit is set by @@ -39,10 +56,226 @@ * instructions. */ #ifdef CRYPTON_ARM_ASM -__attribute__((visibility("hidden"))) unsigned int crypton_armcap_P = +CRYPTON_HIDDEN unsigned int crypton_armcap_P = CRYPTON_ARMCAP_NEON; #endif +#if defined(__aarch64__) || defined(__arm__) +#if defined(__APPLE__) +#include +#elif defined(__linux__) +#include +#include +#elif defined(__FreeBSD__) +#include +#endif + +/* + * The bit positions are the ARM ELF ABI's, so every system that reports + * through the auxiliary vector agrees on them. What differs is the tag -- + * AT_HWCAP is 16 on Linux and 25 on FreeBSD -- and that comes from each + * system's own header, which is why no tag is written out here. Defined + * only where the system's headers did not define them, the way compiler-rt + * does it, so that a system which reports through the auxiliary vector + * without shipping the ARM names still compiles. + * + * The two execution states do not share a word or an order. AArch64 puts + * these in AT_HWCAP; AArch32 has filled that word with older features and + * puts the cryptographic ones in AT_HWCAP2, starting again from bit zero. + */ +#if defined(__aarch64__) +#ifndef HWCAP_AES +#define HWCAP_AES (1 << 3) +#endif +#ifndef HWCAP_PMULL +#define HWCAP_PMULL (1 << 4) +#endif +#ifndef HWCAP_SHA1 +#define HWCAP_SHA1 (1 << 5) +#endif +#ifndef HWCAP_SHA2 +#define HWCAP_SHA2 (1 << 6) +#endif +#ifndef HWCAP_SHA3 +#define HWCAP_SHA3 (1 << 17) +#endif +#ifndef HWCAP_SHA512 +#define HWCAP_SHA512 (1 << 21) +#endif +#else +#ifndef HWCAP2_AES +#define HWCAP2_AES (1 << 0) +#endif +#ifndef HWCAP2_PMULL +#define HWCAP2_PMULL (1 << 1) +#endif +#ifndef HWCAP2_SHA1 +#define HWCAP2_SHA1 (1 << 2) +#endif +#ifndef HWCAP2_SHA2 +#define HWCAP2_SHA2 (1 << 3) +#endif +#endif + +#if defined(__APPLE__) +static int apple_has(const char *name) +{ + int v = 0; + size_t n = sizeof(v); + + if (sysctlbyname(name, &v, &n, NULL, 0) != 0) + return 0; + return v != 0; +} +#endif + +/* + * Systems still answering zero, and what each would need: + * + * OpenBSD, NetBSD sysctl on machdep.id_aa64isar0, which hands out the + * ID_AA64ISAR0_EL1 fields rather than a HWCAP word, so + * it is a different shape of answer rather than another + * tag. + * Windows on ARM IsProcessorFeaturePresent with + * PF_ARM_V8_CRYPTO_INSTRUCTIONS_AVAILABLE, which covers + * AES, PMULL, SHA-1 and SHA-256 as one bit and says + * nothing about SHA-512 or SHA-3. An ARM64 Windows + * guest answers 1 to it, but GHC has no native ARM64 + * Windows target: there it builds x86-64 and the + * emulator reports AES-NI, so this file is not reached. + * + * Neither is written here because neither can be built and run to see it + * work, and an untested answer about whether a machine has AES is worse + * than the honest zero it replaces. + */ +unsigned int crypton_arm_features(void) +{ + static unsigned int features; + static int resolved; + + if (!resolved) { + unsigned int f = 0; +#if defined(__APPLE__) + /* AES, PMULL, SHA-1 and SHA-256 are not optional on Apple + * silicon. The two later ones are. */ + f = CRYPTON_ARM_AES | CRYPTON_ARM_PMULL + | CRYPTON_ARM_SHA1 | CRYPTON_ARM_SHA2; + if (apple_has("hw.optional.arm.FEAT_SHA512")) + f |= CRYPTON_ARM_SHA512; + if (apple_has("hw.optional.arm.FEAT_SHA3")) + f |= CRYPTON_ARM_SHA3; +#elif defined(__aarch64__) + unsigned long cap = 0; + +#if defined(__linux__) + cap = getauxval(AT_HWCAP); +#elif defined(__FreeBSD__) + if (elf_aux_info(AT_HWCAP, &cap, sizeof(cap)) != 0) + cap = 0; +#endif + if (cap & HWCAP_AES) f |= CRYPTON_ARM_AES; + if (cap & HWCAP_PMULL) f |= CRYPTON_ARM_PMULL; + if (cap & HWCAP_SHA1) f |= CRYPTON_ARM_SHA1; + if (cap & HWCAP_SHA2) f |= CRYPTON_ARM_SHA2; + if (cap & HWCAP_SHA512) f |= CRYPTON_ARM_SHA512; + if (cap & HWCAP_SHA3) f |= CRYPTON_ARM_SHA3; +#else + /* AArch32: the second word, and nothing later than SHA-256 to + * ask about */ + unsigned long cap = 0; + +#if defined(__linux__) + cap = getauxval(AT_HWCAP2); +#elif defined(__FreeBSD__) + if (elf_aux_info(AT_HWCAP2, &cap, sizeof(cap)) != 0) + cap = 0; +#endif + if (cap & HWCAP2_AES) f |= CRYPTON_ARM_AES; + if (cap & HWCAP2_PMULL) f |= CRYPTON_ARM_PMULL; + if (cap & HWCAP2_SHA1) f |= CRYPTON_ARM_SHA1; + if (cap & HWCAP2_SHA2) f |= CRYPTON_ARM_SHA2; +#endif + features = f; + resolved = 1; + } + return features; +} +#endif /* __aarch64__ || __arm__ */ + +#if defined(__powerpc64__) || defined(__PPC64__) +#if defined(__linux__) +#include +#elif defined(__FreeBSD__) +#include + +#endif + +/* + * PPC_FEATURE2_VEC_CRYPTO, from the Power ELF ABI. Defined here only where + * the system's headers did not, so that a system reporting through the + * auxiliary vector without shipping the name still compiles. + */ +#ifndef PPC_FEATURE2_VEC_CRYPTO +#define PPC_FEATURE2_VEC_CRYPTO 0x02000000 +#endif + +unsigned int crypton_ppc_features(void) +{ + static unsigned int features; + static int resolved; + + if (!resolved) { + unsigned long cap = 0; + +#if defined(__linux__) + cap = getauxval(AT_HWCAP2); +#elif defined(__FreeBSD__) + if (elf_aux_info(AT_HWCAP2, &cap, sizeof(cap)) != 0) + cap = 0; +#endif + features = (cap & PPC_FEATURE2_VEC_CRYPTO) + ? CRYPTON_PPC_VCRYPTO : 0; + resolved = 1; + } + return features; +} +#endif /* __powerpc64__ */ + +/* + * For testing on a machine whose CPUID says less than its core executes, and + * for nothing else: each of these makes the detection below report a + * feature it did not find. + * + * CRYPTON_CPU_TESTING_ADX BMI2 and ADX + * CRYPTON_CPU_TESTING_SHA_NI the SHA extensions + * CRYPTON_CPU_TESTING_VAES VAES and VPCLMULQDQ, 256-bit + * + * A hypervisor may show a guest an older processor model than the host's so + * that the guest can be moved to another host, and the instructions it hides + * then still run. That is the only place this is for: forcing a path that + * CPUID would not choose, to measure it or to test it, on a machine where the + * instructions have been seen to execute. Anywhere they do not, the first + * use is a SIGILL -- which is the bug #317 fixed, made on purpose. + * + * What the operating system has to agree to is still asked. VAES works in + * the wide registers, so it is assumed only where AVX2 was found, which + * means XCR0 says those are saved; the SHA extensions still want SSSE3 and + * SSE4.1 to have been found, as the detection asks. Nothing 512-bit can be + * assumed: whether the operating system saves that state is not something a + * build can promise. + * + * The library is never built with these, and a build that has one says so. + */ +#if defined(CRYPTON_CPU_TESTING_ADX) || defined(CRYPTON_CPU_TESTING_SHA_NI) \ + || defined(CRYPTON_CPU_TESTING_VAES) +/* The code below asks for each switch by its own name, which comes from the + * command line and so is seen everywhere in the file. It once asked for one + * umbrella name defined beside the first function that read it -- and the + * capability vector the assembly reads is filled earlier in the file than + * that, so the SHA bit never reached it, without a word from the compiler. */ +#warning "CRYPTON_CPU_TESTING_*: features assumed, not detected" +#endif + #ifdef ARCH_X86 static void cpuid(uint32_t info, uint32_t *eax, uint32_t *ebx, uint32_t *ecx, uint32_t *edx) { @@ -101,7 +334,7 @@ } #ifdef CRYPTON_X86_ASM -__attribute__((visibility("hidden"))) unsigned int crypton_ia32cap_P[4]; +CRYPTON_HIDDEN unsigned int crypton_ia32cap_P[4]; /* * The AVX-512 bits of leaf 7 EBX -- F, DQ, IFMA, PF, ER, CD, BW and VL, @@ -169,6 +402,16 @@ } if (!(f & CRYPTON_X86_AVX2)) leaf7_ebx &= ~(1u << 5); + /* what the testing switches assumed, told to the assembly as + * well: its SHA-256 reads the SHA bit here, not the word */ +#ifdef CRYPTON_CPU_TESTING_SHA_NI + if (f & CRYPTON_X86_SHA_NI) + leaf7_ebx |= 1u << 29; +#endif +#ifdef CRYPTON_CPU_TESTING_ADX + if (f & CRYPTON_X86_ADX) + leaf7_ebx |= (1u << 8) | (1u << 19); +#endif crypton_ia32cap_P[2] = leaf7_ebx & ~IA32CAP_AVX512; resolved = 1; @@ -182,13 +425,23 @@ static uint32_t features = 0; if (!resolved) { - uint32_t eax, ebx, ecx, edx, leaf1, maxleaf, f = 0; + uint32_t eax, ebx, ecx, edx, leaf1, maxleaf, family, f = 0; + int amd; cpuid(0, &eax, &ebx, &ecx, &edx); maxleaf = eax; + /* "AuthenticAMD" arrives as EBX, EDX, ECX in that order */ + amd = (ebx == 0x68747541 && edx == 0x69746e65 + && ecx == 0x444d4163); cpuid(1, &eax, &ebx, &ecx, &edx); leaf1 = ecx; + /* the family is the base one, and the extended field is + * added to it only when the base reads 0xf, which is how + * every AMD Zen part reports */ + family = (eax >> 8) & 0xf; + if (family == 0xf) + family += (eax >> 20) & 0xff; if (leaf1 & (1 << 9)) f |= CRYPTON_X86_SSSE3; if (leaf1 & (1 << 1)) @@ -223,7 +476,62 @@ if ((ebx & (1 << 5)) && (leaf1 & (1 << 27)) && (leaf1 & (1 << 28)) && ((xcr0() & 6) == 6)) f |= CRYPTON_X86_AVX2; + /* BMI2 for MULX and ADX for ADCX/ADOX. Both are + * wanted together and neither touches vector state, + * so there is nothing to ask the operating system */ + if ((ebx & (1 << 8)) && (ebx & (1 << 19))) + f |= CRYPTON_X86_ADX; + /* VAES and VPCLMULQDQ, leaf 7 ECX bits 9 and 10. + * They are wanted together -- one without the other + * leaves half of AES-GCM narrow -- and they need the + * wide registers, so AVX2 has to have answered first, + * which settles the operating system's part. */ + if ((ecx & (1 << 9)) && (ecx & (1 << 10)) + && (f & CRYPTON_X86_AVX2)) + f |= CRYPTON_X86_VAES; + /* The same two instructions in their 512-bit form, + * which wants AVX-512 F, BW and VL as well -- and + * three more bits of XCR0, for the mask registers and + * the two upper halves of the vector state. A + * machine can report the instructions and still fault + * on them when the operating system has not said it + * saves that state, which is what those bits are. */ + if ((f & CRYPTON_X86_VAES) + && (ebx & (1 << 16)) && (ebx & (1u << 30)) + && (ebx & (1u << 31)) + && ((xcr0() & 0xe6) == 0xe6) + /* Not on Zen 4, which is AMD family 19h with + * AVX-512: there the 512-bit instructions are two + * 256-bit passes through a 256-bit datapath, so + * they carry the wider encoding for none of the + * throughput, and AES-GCM measures 0.6 to 3 + * per cent slower than the 256-bit path. Zen 5 + * is family 1Ah and does have the wide datapath, + * where the same code is half as fast again; + * Zen 3, the other family 19h part, has no + * AVX-512 at all and never reaches here. */ + && !(amd && family == 0x19)) + f |= CRYPTON_X86_VAES512; + /* IFMA (bit 21) with F (16) and VL (31), and the same + * AVX-512 state. Zen 4 is not left out here: RSA's + * multiplication uses the 256-bit form. */ + if ((f & CRYPTON_X86_AVX2) + && (ebx & (1 << 16)) && (ebx & (1 << 21)) + && (ebx & (1u << 31)) + && ((xcr0() & 0xe6) == 0xe6)) + f |= CRYPTON_X86_IFMA; } +#ifdef CRYPTON_CPU_TESTING_ADX + f |= CRYPTON_X86_ADX; +#endif +#ifdef CRYPTON_CPU_TESTING_SHA_NI + if ((leaf1 & (1 << 9)) && (leaf1 & (1 << 19))) + f |= CRYPTON_X86_SHA_NI; +#endif +#ifdef CRYPTON_CPU_TESTING_VAES + if (f & CRYPTON_X86_AVX2) + f |= CRYPTON_X86_VAES; +#endif features = f; resolved = 1; } @@ -250,3 +558,110 @@ #endif #endif + +/* + * What Crypto.System.CPU reports. + * + * The numbers are that module's, and this answers for them. The old shape + * was the other way round -- Haskell read crypton_aes.c's two-entry array by + * the Enum index of its own constructors -- which tied the list of names to + * the indices of an array that had no reason to grow, and so it never did. + * + * Nothing here is cached: every answer is either a static array filled by a + * constructor, a value resolved once and remembered by the function that + * owns it, or a runtime check that does its own remembering. Haskell calls + * this once, for a CAF. + */ +uint8_t *crypton_aes_cpu_init(void); + +#ifdef WITH_ARMV8_CRYPTO +int crypton_aes_armv8_available(void); +int crypton_aes_armv8_pmull_available(void); +#endif +#ifdef WITH_ARMV8_SHA1 +extern int crypton_sha1_armv8_available(void); +#endif +#ifdef WITH_ARMV8_SHA2 +extern int crypton_sha256_armv8_available(void); +#endif +#ifdef WITH_ARMV8_SHA512 +extern int crypton_sha512_armv8_available(void); +#endif + +/* Keep in step with Crypto/System/CPU.hs, which is where these are named. + * 2 is missing: it was RDRAND, which crypton no longer dispatches on. The + * number is left out rather than reused, so that the rest keep the values + * they had. */ +#define OPT_AESNI 0 +#define OPT_PCLMUL 1 +#define OPT_SSSE3 3 +#define OPT_AVX 4 +#define OPT_AVX2 5 +#define OPT_SHANI 6 +#define OPT_MOVBE 7 +#define OPT_ADX 8 +#define OPT_VAES 9 +#define OPT_VAES512 10 +#define OPT_NEON 11 +#define OPT_ARMAES 12 +#define OPT_ARMPMULL 13 +#define OPT_ARMSHA1 14 +#define OPT_ARMSHA2 15 +#define OPT_ARMSHA512 16 +#define OPT_PPCAES 17 +#define OPT_PPCVPMSUM 18 +#define OPT_IFMA 19 + +int crypton_cpu_option(unsigned int option) +{ +#ifdef ARCH_X86 + uint32_t f = crypton_x86_simd_features(); + + switch (option) { + case OPT_AESNI: return crypton_aes_cpu_init()[CPU_AESNI] != 0; + case OPT_PCLMUL: return crypton_aes_cpu_init()[CPU_PCLMUL] != 0; + case OPT_SSSE3: return (f & CRYPTON_X86_SSSE3) != 0; + case OPT_AVX: return (f & CRYPTON_X86_AVX) != 0; + case OPT_AVX2: return (f & CRYPTON_X86_AVX2) != 0; + case OPT_SHANI: return (f & CRYPTON_X86_SHA_NI) != 0; + case OPT_MOVBE: return (f & CRYPTON_X86_MOVBE) != 0; + case OPT_ADX: return (f & CRYPTON_X86_ADX) != 0; + case OPT_VAES: return (f & CRYPTON_X86_VAES) != 0; + case OPT_VAES512: return (f & CRYPTON_X86_VAES512) != 0; + case OPT_IFMA: return (f & CRYPTON_X86_IFMA) != 0; + default: return 0; + } +#else + switch (option) { + /* NEON is not optional on AArch64, and crypton_armcap_P is set from + * the start for it. Where there is no ARM assembly at all there is + * nothing to say. */ +#ifdef CRYPTON_ARM_ASM + case OPT_NEON: return (crypton_armcap_P & CRYPTON_ARMCAP_NEON) != 0; +#endif +#ifdef WITH_ARMV8_CRYPTO + /* The array rather than the check: it is what the branch table was + * actually built from, so it says what will run, not merely what the + * processor has. */ + case OPT_ARMAES: return crypton_aes_cpu_init()[CPU_AESNI] != 0; + case OPT_ARMPMULL: return crypton_aes_cpu_init()[CPU_PCLMUL] != 0; +#endif +#ifdef WITH_ARMV8_SHA1 + case OPT_ARMSHA1: return crypton_sha1_armv8_available() != 0; +#endif +#ifdef WITH_ARMV8_SHA2 + case OPT_ARMSHA2: return crypton_sha256_armv8_available() != 0; +#endif +#ifdef WITH_ARMV8_SHA512 + case OPT_ARMSHA512: return crypton_sha512_armv8_available() != 0; +#endif +#ifdef WITH_PPC8_CRYPTO + /* The array rather than the check, for the same reason as the ARM pair + * above: it says what the branch table was built from. */ + case OPT_PPCAES: return crypton_aes_cpu_init()[CPU_AESNI] != 0; + case OPT_PPCVPMSUM: return crypton_aes_cpu_init()[CPU_PCLMUL] != 0; +#endif + default: return 0; + } +#endif +} diff --git a/cbits/crypton_cpu.h b/cbits/crypton_cpu.h --- a/cbits/crypton_cpu.h +++ b/cbits/crypton_cpu.h @@ -49,6 +49,25 @@ * in, and the byte-swapping load it reads the message with */ #define CRYPTON_X86_AVX 16 #define CRYPTON_X86_MOVBE 32 +/* MULX, ADCX and ADOX together: the two independent carry chains the + * vendored s2n-bignum assembly wants. They are general-purpose register + * instructions, so unlike the vector ones above they ask nothing of the + * operating system. */ +#define CRYPTON_X86_ADX 64 +/* The AES and carry-less multiply instructions in their 256-bit form, which + * do two blocks where the 128-bit ones do one. They are VEX-encoded and use + * the vector registers AVX2 already needs the operating system to save, so + * they ask nothing further of it -- but AVX2 itself is asked about, since + * without it there is nowhere to put them. */ +#define CRYPTON_X86_VAES 128 +/* The same pair in their 512-bit form, four blocks to an instruction. These + * are EVEX-encoded and need the AVX-512 state as well, which is three more + * bits of XCR0 than AVX2 wants: the mask registers and the two upper halves + * of the vector registers. */ +#define CRYPTON_X86_VAES512 256 +/* AVX-512 IFMA, the 52-bit multiply-adds, with F and VL for the 256-bit form + * RSA uses them in: EVEX-encoded, so they want the AVX-512 state saved too. */ +#define CRYPTON_X86_IFMA 512 #ifdef ARCH_X86 uint32_t crypton_x86_simd_features(void); #endif @@ -81,6 +100,64 @@ #define CRYPTON_ARMCAP_SHA256 (1 << 4) extern unsigned int crypton_armcap_P; #endif + +/* + * Which of the optional ARMv8 instruction sets this processor has. + * + * Asked here rather than in each file that wants one. Five files used to + * carry the same three-way conditional -- Apple, Linux, and otherwise zero + * -- and the "otherwise" is not a statement about the processor but about + * which systems someone had thought of. It is what left FreeBSD running + * the table-driven AES and the C SHA on hardware that has the + * instructions. In one place the next system is added once. + * + * AArch32 answers here too. It has the same AES, PMULL, SHA-1 and SHA-256 + * instructions, reports them in a different word of the auxiliary vector, + * and has no SHA-512 or SHA-3 to report at all -- those two are AArch64's + * alone, so the bits exist and are never set. + */ +#if defined(__aarch64__) || defined(__arm__) +#define CRYPTON_ARM_AES (1u << 0) +#define CRYPTON_ARM_PMULL (1u << 1) +#define CRYPTON_ARM_SHA1 (1u << 2) +#define CRYPTON_ARM_SHA2 (1u << 3) +#define CRYPTON_ARM_SHA512 (1u << 4) +#define CRYPTON_ARM_SHA3 (1u << 5) +unsigned int crypton_arm_features(void); +#endif + +/* + * Which of the optional PowerISA instruction sets this processor has. + * + * One bit so far: the vector AES and the vector carry-less multiply that + * PowerISA 2.07 brought and POWER8 was the first to implement. The system + * reports it in the second capability word, as AArch32 does, and with its + * own numbering again. + */ +#if defined(__powerpc64__) || defined(__PPC64__) +#define CRYPTON_PPC_VCRYPTO (1u << 0) +unsigned int crypton_ppc_features(void); +#endif + +/* + * The two slots of the array crypton_aes.c fills and crypton_aes_cpu_init + * hands out. Here rather than in that file because crypton_cpu.c reads + * them too, to answer for Crypto.System.CPU. + */ +#define CPU_AESNI 0 +#define CPU_PCLMUL 1 +#define CPU_OPTION_COUNT 2 + +/* + * One question at a time, numbered the way Crypto.System.CPU numbers its + * ProcessorOption rather than the way any array here is indexed. The + * numbering is the module's to choose and this answers for it; reading an + * array by the Haskell constructor's Enum index is what kept that type at + * three names. Returns 1 if the processor has it and crypton was built to + * use it, 0 otherwise -- including for anything this does not know, so an + * older library answers a newer caller rather than failing to link. + */ +int crypton_cpu_option(unsigned int option); #ifdef USE_AESNI void crypton_aesni_initialize_hw(void (*init_table)(int, int)); diff --git a/cbits/crypton_ecc.c b/cbits/crypton_ecc.c --- a/cbits/crypton_ecc.c +++ b/cbits/crypton_ecc.c @@ -23,8 +23,12 @@ */ #include #include +#include #include #include +#ifdef CRYPTON_S2N_BIGNUM +#include +#endif /* four bits of scalar per window, so a table of sixteen and no leftover * bits: a byte holds exactly two windows */ @@ -220,12 +224,15 @@ static void ctx_free(curve_ctx *c) { + /* crypton_bzero rather than memset: this memory is freed on the next + * line, and a store to memory about to die is one an optimizer may + * drop. Both buffers have held scalars. */ if (c->space != NULL) { - memset(c->space, 0, c->words * sizeof(limb_t)); + crypton_bzero(c->space, c->words * sizeof(limb_t)); free(c->space); } if (c->bytes != NULL) { - memset(c->bytes, 0, 2 * c->plen); + crypton_bzero(c->bytes, 2 * c->plen); free(c->bytes); } c->space = NULL; @@ -284,10 +291,10 @@ ctx_free(c); return -1; } - mont_r2(c->r2, mp, n, mont_t); + c->f.n0 = mont_n0(mp[0]); + mont_r2(c->r2, mp, c->f.n0, n, mont_t); c->f.n = n; - c->f.n0 = mont_n0(mp[0]); c->f.p = mp; c->f.a = ma; c->f.b3 = mb3; @@ -411,6 +418,20 @@ curve_ctx c; uint32_t n, i, j; int ret = -1; + +#ifdef CRYPTON_S2N_BIGNUM + /* Two of the curves that reach here have hand-written assembly, six + * to ten times faster than what follows; see cbits/s2n/README.md. + * Anything else, including those two named with a different a or b, + * goes on down. */ + { + int s2n_ret; + + if (crypton_s2n_ecc_mul(&s2n_ret, outx, outy, px, py, k, klen, + a, b, p, plen)) + return s2n_ret; + } +#endif if (klen == 0 || ctx_init(&c, a, b, p, plen) != 0) return -1; diff --git a/cbits/crypton_ecc_s2n.c b/cbits/crypton_ecc_s2n.c new file mode 100644 --- /dev/null +++ b/cbits/crypton_ecc_s2n.c @@ -0,0 +1,204 @@ +#include + +#include "crypton_ecc_s2n.h" +#include "crypton_ecc_s2n_curves.h" +#include "crypton_cpu.h" + +/* P-384, Montgomery domain, six words a coordinate */ +extern void p384_montjscalarmul(uint64_t *res, const uint64_t *s, const uint64_t *p); +extern void p384_montjscalarmul_alt(uint64_t *res, const uint64_t *s, const uint64_t *p); +extern void bignum_tomont_p384(uint64_t *z, const uint64_t *x); +extern void bignum_tomont_p384_alt(uint64_t *z, const uint64_t *x); +extern void bignum_deamont_p384(uint64_t *z, const uint64_t *x); +extern void bignum_deamont_p384_alt(uint64_t *z, const uint64_t *x); +extern void bignum_montmul_p384(uint64_t *z, const uint64_t *x, const uint64_t *y); +extern void bignum_montmul_p384_alt(uint64_t *z, const uint64_t *x, const uint64_t *y); +extern void bignum_montsqr_p384(uint64_t *z, const uint64_t *x); +extern void bignum_montsqr_p384_alt(uint64_t *z, const uint64_t *x); +extern void bignum_montinv_p384(uint64_t *z, const uint64_t *x); + +/* P-521, ordinary values, nine words a coordinate */ +extern void p521_jscalarmul(uint64_t *res, const uint64_t *s, const uint64_t *p); +extern void p521_jscalarmul_alt(uint64_t *res, const uint64_t *s, const uint64_t *p); +extern void bignum_mul_p521(uint64_t *z, const uint64_t *x, const uint64_t *y); +extern void bignum_mul_p521_alt(uint64_t *z, const uint64_t *x, const uint64_t *y); +extern void bignum_sqr_p521(uint64_t *z, const uint64_t *x); +extern void bignum_sqr_p521_alt(uint64_t *z, const uint64_t *x); +extern void bignum_inv_p521(uint64_t *z, const uint64_t *x); + +/* One flavour or the other, all the way through: on x86-64 the plain form + * wants MULX, ADCX and ADOX and _alt is the fallback, so mixing them would + * fault on a machine without those. */ +struct p384_asm { + void (*tomont)(uint64_t *, const uint64_t *); + void (*deamont)(uint64_t *, const uint64_t *); + void (*montmul)(uint64_t *, const uint64_t *, const uint64_t *); + void (*montsqr)(uint64_t *, const uint64_t *); + void (*jscalarmul)(uint64_t *, const uint64_t *, const uint64_t *); +}; +struct p521_asm { + void (*mul)(uint64_t *, const uint64_t *, const uint64_t *); + void (*sqr)(uint64_t *, const uint64_t *); + void (*jscalarmul)(uint64_t *, const uint64_t *, const uint64_t *); +}; + +static const struct p384_asm p384_std = { + bignum_tomont_p384, bignum_deamont_p384, bignum_montmul_p384, + bignum_montsqr_p384, p384_montjscalarmul +}; +static const struct p384_asm p384_alt = { + bignum_tomont_p384_alt, bignum_deamont_p384_alt, + bignum_montmul_p384_alt, bignum_montsqr_p384_alt, + p384_montjscalarmul_alt +}; +static const struct p521_asm p521_std = { + bignum_mul_p521, bignum_sqr_p521, p521_jscalarmul +}; +static const struct p521_asm p521_alt = { + bignum_mul_p521_alt, bignum_sqr_p521_alt, p521_jscalarmul_alt +}; + +/* The same question as for P-256, answered the same way; see + * cbits/p256/p256_s2n.c. */ +static int use_alt(void) +{ +#if defined(__aarch64__) || defined(__arm64__) +#ifdef __APPLE__ + return 1; +#else + return 0; +#endif +#else + return (crypton_x86_simd_features() & CRYPTON_X86_ADX) == 0; +#endif +} + +#define MAXWORDS 9 + +static void be_to_le64(uint64_t *w, const uint8_t *b, uint32_t len) +{ + uint32_t i; + + for (i = 0; i < MAXWORDS; i++) + w[i] = 0; + for (i = 0; i < len; i++) { + uint32_t pos = len - 1 - i; + w[pos / 8] |= (uint64_t)b[i] << (8 * (pos % 8)); + } +} + +static void le64_to_be(uint8_t *b, uint32_t len, const uint64_t *w) +{ + uint32_t i; + + for (i = 0; i < len; i++) + b[len - 1 - i] = (uint8_t)(w[i / 8] >> (8 * (i % 8))); +} + +static int is_zero(const uint64_t *w, int words) +{ + uint64_t acc = 0; + int i; + + for (i = 0; i < words; i++) + acc |= w[i]; + return acc == 0; +} + +static int mul_p384(uint8_t *outx, uint8_t *outy, const uint8_t *px, + const uint8_t *py, const uint8_t *k, uint32_t klen) +{ + const struct p384_asm *f = use_alt() ? &p384_alt : &p384_std; + uint64_t pt[18], res[18], sc[MAXWORDS], t[MAXWORDS]; + uint64_t zi[6], zi2[6], zi3[6], num[6]; + static const uint64_t one[6] = {1, 0, 0, 0, 0, 0}; + + be_to_le64(t, px, P384_PLEN); + f->tomont(pt, t); + be_to_le64(t, py, P384_PLEN); + f->tomont(pt + 6, t); + f->tomont(pt + 12, one); + be_to_le64(sc, k, klen); + + f->jscalarmul(res, sc, pt); + if (is_zero(res + 12, 6)) + return 1; + + /* affine again: x = X/Z^2, y = Y/Z^3, with the inverse taken in the + * Montgomery domain so that it lands where the rest of these are */ + bignum_montinv_p384(zi, res + 12); + f->montsqr(zi2, zi); + f->montmul(zi3, zi2, zi); + f->montmul(num, res, zi2); + f->deamont(t, num); + le64_to_be(outx, P384_PLEN, t); + f->montmul(num, res + 6, zi3); + f->deamont(t, num); + le64_to_be(outy, P384_PLEN, t); + return 0; +} + +static int mul_p521(uint8_t *outx, uint8_t *outy, const uint8_t *px, + const uint8_t *py, const uint8_t *k, uint32_t klen) +{ + const struct p521_asm *f = use_alt() ? &p521_alt : &p521_std; + uint64_t pt[27], res[27], sc[MAXWORDS], t[MAXWORDS]; + uint64_t zi[9], zi2[9], zi3[9]; + static const uint64_t one[9] = {1, 0, 0, 0, 0, 0, 0, 0, 0}; + + be_to_le64(pt, px, P521_PLEN); + be_to_le64(pt + 9, py, P521_PLEN); + memcpy(pt + 18, one, sizeof(one)); + be_to_le64(sc, k, klen); + + f->jscalarmul(res, sc, pt); + if (is_zero(res + 18, 9)) + return 1; + + bignum_inv_p521(zi, res + 18); + f->sqr(zi2, zi); + f->mul(zi3, zi2, zi); + f->mul(t, res, zi2); + le64_to_be(outx, P521_PLEN, t); + f->mul(t, res + 9, zi3); + le64_to_be(outy, P521_PLEN, t); + return 0; +} + +int crypton_s2n_ecc_mul(int *ret, uint8_t *outx, uint8_t *outy, + const uint8_t *px, const uint8_t *py, + const uint8_t *k, uint32_t klen, + const uint8_t *a, const uint8_t *b, + const uint8_t *p, uint32_t plen) +{ + int is384; + + if (plen == P384_PLEN && memcmp(p, P384_P, plen) == 0 + && memcmp(a, P384_A, plen) == 0 && memcmp(b, P384_B, plen) == 0) + is384 = 1; + else if (plen == P521_PLEN && memcmp(p, P521_P, plen) == 0 + && memcmp(a, P521_A, plen) == 0 + && memcmp(b, P521_B, plen) == 0) + is384 = 0; + else + return 0; /* some other curve; the C answers it */ + + /* A scalar longer than the prime is not something the word arrays + * here hold, and it is not what any caller of these two curves + * sends, so leave it to the C rather than grow a second path. */ + if (klen == 0 || klen > plen) + return 0; + + /* The C does not require the coordinates to be reduced -- it takes + * whatever fits in its limbs and lets the conversion to Montgomery + * form reduce it. Rather than carry a reduction here to match, hand + * that case back: nothing sends one, and this way the two cannot + * disagree about it. (A differential test against the C found this; + * the first version of this check returned -1 and was wrong.) */ + if (memcmp(px, p, plen) >= 0 || memcmp(py, p, plen) >= 0) + return 0; + + *ret = is384 ? mul_p384(outx, outy, px, py, k, klen) + : mul_p521(outx, outy, px, py, k, klen); + return 1; +} diff --git a/cbits/crypton_ecc_s2n.h b/cbits/crypton_ecc_s2n.h new file mode 100644 --- /dev/null +++ b/cbits/crypton_ecc_s2n.h @@ -0,0 +1,27 @@ +/* + * P-384 and P-521 through the vendored s2n-bignum assembly, for the two + * curves it knows among the ones crypton_ecc_mul is asked about. + * + * s2n-bignum has no affine wrapper for these two -- only a scalar + * multiplication on Jacobian points, Montgomery-domain for P-384 and plain + * for P-521 -- so the conversions in and out are built here out of its own + * field operations. See cbits/s2n/README.md. + */ +#ifndef CRYPTON_ECC_S2N_H +#define CRYPTON_ECC_S2N_H + +#include + +/* + * Returns 1 if this was a curve it knows and it answered, with *ret set to + * what crypton_ecc_mul should return -- 0 and the point in outx and outy, 1 + * for the point at infinity, or -1 for arguments it will not take. Returns + * 0 if the curve is not one of its two and nothing was written. + */ +int crypton_s2n_ecc_mul(int *ret, uint8_t *outx, uint8_t *outy, + const uint8_t *px, const uint8_t *py, + const uint8_t *k, uint32_t klen, + const uint8_t *a, const uint8_t *b, + const uint8_t *p, uint32_t plen); + +#endif diff --git a/cbits/crypton_ecc_s2n_curves.h b/cbits/crypton_ecc_s2n_curves.h new file mode 100644 --- /dev/null +++ b/cbits/crypton_ecc_s2n_curves.h @@ -0,0 +1,76 @@ +/* + * p, a and b of the two curves the vendored s2n-bignum assembly knows, as + * big-endian bytes, which is how crypton_ecc_mul is given a curve. They are + * here to be compared against, not computed with: a caller naming some other + * curve with the same sizes has to go to the C. + * + * Generated from `openssl ecparam -name secp384r1 -param_enc explicit -text` + * and the same for secp521r1, rather than transcribed. + */ +#ifndef CRYPTON_ECC_S2N_CURVES_H +#define CRYPTON_ECC_S2N_CURVES_H + +#include + +#define P384_PLEN 48 +static const uint8_t P384_P[48] = { + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xfe, + 0xff,0xff,0xff,0xff,0x00,0x00,0x00,0x00, + 0x00,0x00,0x00,0x00,0xff,0xff,0xff,0xff +}; +static const uint8_t P384_A[48] = { + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xfe, + 0xff,0xff,0xff,0xff,0x00,0x00,0x00,0x00, + 0x00,0x00,0x00,0x00,0xff,0xff,0xff,0xfc +}; +static const uint8_t P384_B[48] = { + 0xb3,0x31,0x2f,0xa7,0xe2,0x3e,0xe7,0xe4, + 0x98,0x8e,0x05,0x6b,0xe3,0xf8,0x2d,0x19, + 0x18,0x1d,0x9c,0x6e,0xfe,0x81,0x41,0x12, + 0x03,0x14,0x08,0x8f,0x50,0x13,0x87,0x5a, + 0xc6,0x56,0x39,0x8d,0x8a,0x2e,0xd1,0x9d, + 0x2a,0x85,0xc8,0xed,0xd3,0xec,0x2a,0xef +}; + +#define P521_PLEN 66 +static const uint8_t P521_P[66] = { + 0x01,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff +}; +static const uint8_t P521_A[66] = { + 0x01,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xfc +}; +static const uint8_t P521_B[66] = { + 0x00,0x51,0x95,0x3e,0xb9,0x61,0x8e,0x1c, + 0x9a,0x1f,0x92,0x9a,0x21,0xa0,0xb6,0x85, + 0x40,0xee,0xa2,0xda,0x72,0x5b,0x99,0xb3, + 0x15,0xf3,0xb8,0xb4,0x89,0x91,0x8e,0xf1, + 0x09,0xe1,0x56,0x19,0x39,0x51,0xec,0x7e, + 0x93,0x7b,0x16,0x52,0xc0,0xbd,0x3b,0xb1, + 0xbf,0x07,0x35,0x73,0xdf,0x88,0x3d,0x2c, + 0x34,0xf1,0xef,0x45,0x1f,0xd4,0x6b,0x50, + 0x3f,0x00 +}; + +#endif diff --git a/cbits/crypton_f2m.c b/cbits/crypton_f2m.c --- a/cbits/crypton_f2m.c +++ b/cbits/crypton_f2m.c @@ -25,6 +25,8 @@ #include #include #include +#include "crypton_armv8_target.h" +#include #include typedef uint64_t limb_t; @@ -89,7 +91,7 @@ #define PMULL_ATTR #define PMULL_ALWAYS 1 #else -#define PMULL_ATTR __attribute__((target("+crypto"))) +#define PMULL_ATTR CRYPTON_TARGET_ARMV8_CRYPTO #define PMULL_ALWAYS 0 #endif @@ -543,8 +545,10 @@ ret = 0; done: + /* freed on the next line, so a plain memset here is a store the + * optimizer may drop */ if (space != NULL) { - memset(space, 0, words * sizeof(limb_t)); + crypton_bzero(space, words * sizeof(limb_t)); free(space); } return ret; diff --git a/cbits/crypton_md4.c b/cbits/crypton_md4.c --- a/cbits/crypton_md4.c +++ b/cbits/crypton_md4.c @@ -48,16 +48,17 @@ #define K3 0x6ED9EBA1 #define R(a,b,c,d,f,k,s,i) (a = rol32(a + f(b,c,d) + w[i] + k, s)) -static void md4_do_chunk(struct md4_ctx *ctx, uint32_t *buf) +/* The words are read out of the block rather than the block being pointed at + * as though it were an array of them; see crypton_md5.c. */ +static void md4_do_chunk(struct md4_ctx *ctx, const uint8_t *buf) { uint32_t a, b, c, d; -#ifdef ARCH_IS_BIG_ENDIAN uint32_t w[16]; - cpu_to_le32_array(w, (uint32_t *) buf, 16); -#else - uint32_t *w = buf; -#endif + int wi; + for (wi = 0; wi < 16; wi++) + w[wi] = load_le32(buf + 4 * wi); + a = ctx->h[0]; b = ctx->h[1]; c = ctx->h[2]; d = ctx->h[3]; R(a, b, c, d, f1, K1, 3, 0); @@ -125,24 +126,15 @@ if (index && len >= to_fill) { memcpy(ctx->buf + index, data, to_fill); - md4_do_chunk(ctx, (uint32_t *) ctx->buf); + md4_do_chunk(ctx, ctx->buf); len -= to_fill; data += to_fill; index = 0; } - if (need_alignment(data, 4)) { - uint32_t tramp[16]; - ASSERT_ALIGNMENT(tramp, 4); - for (; len >= 64; len -= 64, data += 64) { - memcpy(tramp, data, 64); - md4_do_chunk(ctx, tramp); - } - } else { - /* process as much 64-block as possible */ - for (; len >= 64; len -= 64, data += 64) - md4_do_chunk(ctx, (uint32_t *) data); - } + /* No trampoline: load_le32 does not ask for a boundary. */ + for (; len >= 64; len -= 64, data += 64) + md4_do_chunk(ctx, data); /* append data into buf */ if (len) diff --git a/cbits/crypton_md5.c b/cbits/crypton_md5.c --- a/cbits/crypton_md5.c +++ b/cbits/crypton_md5.c @@ -45,15 +45,22 @@ #define f4(x, y, z) (y ^ (x | ~z)) #define R(f, a, b, c, d, i, k, s) a += f(b, c, d) + w[i] + k; a = rol32(a, s); a += b -static void md5_do_chunk(struct md5_ctx *ctx, uint32_t *buf) +/* The sixteen words are read out of the block rather than the block being + * pointed at as though it were an array of them. A caller's pointer cast to + * uint32_t * is a pointer the standard says may not exist unless the address + * is aligned for it, and reading through it is undefined whether or not the + * machine minds; UndefinedBehaviorSanitizer counted sixty-four of these. + * load_le32 is a memcpy, which every compiler here turns into the one load + * the cast used to be, and it takes the endianness with it -- so the two + * arms this replaces are one. */ +static void md5_do_chunk(struct md5_ctx *ctx, const uint8_t *buf) { uint32_t a, b, c, d; -#ifdef ARCH_IS_BIG_ENDIAN uint32_t w[16]; - cpu_to_le32_array(w, buf, 16); -#else - uint32_t *w = buf; -#endif + int wi; + + for (wi = 0; wi < 16; wi++) + w[wi] = load_le32(buf + 4 * wi); a = ctx->h[0]; b = ctx->h[1]; c = ctx->h[2]; d = ctx->h[3]; R(f1, a, b, c, d, 0, 0xd76aa478, 7); @@ -138,24 +145,16 @@ if (index && len >= to_fill) { memcpy(ctx->buf + index, data, to_fill); - md5_do_chunk(ctx, (uint32_t *) ctx->buf); + md5_do_chunk(ctx, ctx->buf); len -= to_fill; data += to_fill; index = 0; } - if (need_alignment(data, 4)) { - uint32_t tramp[16]; - ASSERT_ALIGNMENT(tramp, 4); - for (; len >= 64; len -= 64, data += 64) { - memcpy(tramp, data, 64); - md5_do_chunk(ctx, tramp); - } - } else { - /* process as much 64-block as possible */ - for (; len >= 64; len -= 64, data += 64) - md5_do_chunk(ctx, (uint32_t *) data); - } + /* No trampoline for a block that is not on a four-byte boundary: the + * words are read with load_le32 now, which does not ask. */ + for (; len >= 64; len -= 64, data += 64) + md5_do_chunk(ctx, data); /* append data into buf */ if (len) diff --git a/cbits/crypton_modinv.c b/cbits/crypton_modinv.c new file mode 100644 --- /dev/null +++ b/cbits/crypton_modinv.c @@ -0,0 +1,74 @@ +/* + * Inversion modulo an odd number through s2n-bignum, whose routine takes a + * fixed number of division steps rather than an exponentiation: twenty to + * thirty times less work than Fermat's little theorem at the sizes here. + * Measured on an Apple M4, inverting modulo a curve order: + * + * Fermat this + * P-256 6.02 us 0.80 + * P-384 31.3 1.20 + * P-521 63.2 2.05 + * + * It uses no instruction beyond the base architecture on either x86-64 or + * AArch64, so unlike the rest of the vendored assembly there is nothing to + * ask the processor first. + */ +#include + +#include "crypton_modinv.h" + +#ifdef CRYPTON_S2N_BIGNUM + +extern void bignum_modinv(uint64_t k, uint64_t *z, const uint64_t *a, + const uint64_t *b, uint64_t *t); + +/* 4096 bits and no more, which covers every modulus that reaches here -- the + * order of a curve, or a prime factor of an RSA modulus -- and keeps the + * working space on the stack. Anything larger is handed back. */ +#define MODINV_MAXWORDS 64 + +int crypton_modinv_sec(uint8_t *z, const uint8_t *a, const uint8_t *m, + uint32_t len) +{ + uint64_t aw[MODINV_MAXWORDS], mw[MODINV_MAXWORDS]; + uint64_t zw[MODINV_MAXWORDS], t[3 * MODINV_MAXWORDS]; + uint32_t k = (len + 7) / 8; + uint32_t i; + + /* An even modulus is the one case it answers without saying it + * cannot: it returns a number that is not an inverse rather than + * failing, so keep it away from here. Every caller's modulus is odd. */ + if (len == 0 || k > MODINV_MAXWORDS || (m[len - 1] & 1) == 0) + return 1; + + for (i = 0; i < k; i++) { + aw[i] = 0; + mw[i] = 0; + } + for (i = 0; i < len; i++) { + uint32_t pos = len - 1 - i; + + aw[pos / 8] |= (uint64_t)a[i] << (8 * (pos % 8)); + mw[pos / 8] |= (uint64_t)m[i] << (8 * (pos % 8)); + } + + bignum_modinv(k, zw, aw, mw, t); + + for (i = 0; i < len; i++) + z[len - 1 - i] = (uint8_t)(zw[i / 8] >> (8 * (i % 8))); + return 0; +} + +#else + +int crypton_modinv_sec(uint8_t *z, const uint8_t *a, const uint8_t *m, + uint32_t len) +{ + (void)z; + (void)a; + (void)m; + (void)len; + return 1; +} + +#endif diff --git a/cbits/crypton_modinv.h b/cbits/crypton_modinv.h new file mode 100644 --- /dev/null +++ b/cbits/crypton_modinv.h @@ -0,0 +1,22 @@ +#ifndef CRYPTON_MODINV_H +#define CRYPTON_MODINV_H + +#include + +/* z = a^-1 mod m, all three big-endian byte strings of len bytes. + * + * Returns 0 with the answer in z, and 1 without touching z when it will not + * do this one: the assembly it needs is not built, the modulus is even, or + * the numbers are larger than it keeps room for. A 1 is not an error, it is + * "ask something else". + * + * The answer is not checked here. When a has no inverse the routine + * underneath returns something that is not one rather than saying so, so the + * caller has to multiply out and look -- which is what Crypto.Number. + * ModArithmetic.inverseSafe already did for the exponentiation this + * replaces. + */ +int crypton_modinv_sec(uint8_t *z, const uint8_t *a, const uint8_t *m, + uint32_t len); + +#endif diff --git a/cbits/crypton_mont.h b/cbits/crypton_mont.h new file mode 100644 --- /dev/null +++ b/cbits/crypton_mont.h @@ -0,0 +1,13445 @@ +/* Generated by cbits/crypton_mont.py; do not edit. + * + * Montgomery multiplication and squaring for AArch64 at 16, 24 and 32 limbs, + * the halves of RSA-2048, -3072 and -4096, and squaring at 48, RSA-3072's + * modulus. Each answers what mont_mul and mont_sqr answer. */ +#ifndef CRYPTON_MONT_H +#define CRYPTON_MONT_H + +#include "crypton_bignum.h" + +/* (t2:t1:t0) += x * y */ +#define MONT_MAC(t0, t1, t2, x, y) \ + do { \ + limb_t lo_, hi_; \ + __asm__("mul\t%0, %5, %6\n\t" \ + "umulh\t%1, %5, %6\n\t" \ + "adds\t%2, %2, %0\n\t" \ + "adcs\t%3, %3, %1\n\t" \ + "adc\t%4, %4, xzr" \ + : "=&r"(lo_), "=&r"(hi_), "+r"(t0), "+r"(t1), "+r"(t2) \ + : "r"(x), "r"(y) \ + : "cc"); \ + } while (0) + +/* (s2:s1:s0) = x * y */ +#define MONT_MUL(s0, s1, s2, x, y) \ + __asm__("mul\t%0, %3, %4\n\t" \ + "umulh\t%1, %3, %4\n\t" \ + "mov\t%2, xzr" \ + : "=&r"(s0), "=&r"(s1), "=r"(s2) \ + : "r"(x), "r"(y)) + +/* (t2:t1:t0) += (s2:s1:s0) */ +#define MONT_ADD(t0, t1, t2, s0, s1, s2) \ + __asm__("adds\t%0, %0, %3\n\t" \ + "adcs\t%1, %1, %4\n\t" \ + "adc\t%2, %2, %5" \ + : "+r"(t0), "+r"(t1), "+r"(t2) \ + : "r"(s0), "r"(s1), "r"(s2) \ + : "cc") + +/* (s2:s1:s0) *= 2 */ +#define MONT_DBL(s0, s1, s2) \ + do { \ + s2 = (s2 << 1) | (s1 >> 63); \ + s1 = (s1 << 1) | (s0 >> 63); \ + s0 <<= 1; \ + } while (0) + +/* r = a * b * R^-1 mod m, for a and b under m; t is 2n limbs of scratch */ +static void crypton_mont_mul16(limb_t *r, const limb_t *a, const limb_t *b, + const limb_t *m, limb_t n0, limb_t *t) +{ + limb_t *u = t, *z = t + 16; + limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take; + + /* column 0 */ + MONT_MAC(t0, t1, t2, a[0], b[0]); + u[0] = t0 * n0; + MONT_MAC(t0, t1, t2, u[0], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 1 */ + MONT_MAC(t0, t1, t2, a[0], b[1]); + MONT_MUL(v0, v1, v2, a[1], b[0]); + MONT_MUL(w0, w1, w2, u[0], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[1] = t0 * n0; + MONT_MAC(t0, t1, t2, u[1], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 2 */ + MONT_MAC(t0, t1, t2, a[0], b[2]); + MONT_MUL(v0, v1, v2, a[1], b[1]); + MONT_MUL(w0, w1, w2, a[2], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[2]); + MONT_MAC(v0, v1, v2, u[1], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[2] = t0 * n0; + MONT_MAC(t0, t1, t2, u[2], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 3 */ + MONT_MAC(t0, t1, t2, a[0], b[3]); + MONT_MUL(v0, v1, v2, a[1], b[2]); + MONT_MUL(w0, w1, w2, a[2], b[1]); + MONT_MAC(t0, t1, t2, a[3], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[3]); + MONT_MAC(w0, w1, w2, u[1], m[2]); + MONT_MAC(t0, t1, t2, u[2], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[3] = t0 * n0; + MONT_MAC(t0, t1, t2, u[3], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 4 */ + MONT_MAC(t0, t1, t2, a[0], b[4]); + MONT_MUL(v0, v1, v2, a[1], b[3]); + MONT_MUL(w0, w1, w2, a[2], b[2]); + MONT_MAC(t0, t1, t2, a[3], b[1]); + MONT_MAC(v0, v1, v2, a[4], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[4]); + MONT_MAC(t0, t1, t2, u[1], m[3]); + MONT_MAC(v0, v1, v2, u[2], m[2]); + MONT_MAC(w0, w1, w2, u[3], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[4] = t0 * n0; + MONT_MAC(t0, t1, t2, u[4], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 5 */ + MONT_MAC(t0, t1, t2, a[0], b[5]); + MONT_MUL(v0, v1, v2, a[1], b[4]); + MONT_MUL(w0, w1, w2, a[2], b[3]); + MONT_MAC(t0, t1, t2, a[3], b[2]); + MONT_MAC(v0, v1, v2, a[4], b[1]); + MONT_MAC(w0, w1, w2, a[5], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[5]); + MONT_MAC(v0, v1, v2, u[1], m[4]); + MONT_MAC(w0, w1, w2, u[2], m[3]); + MONT_MAC(t0, t1, t2, u[3], m[2]); + MONT_MAC(v0, v1, v2, u[4], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[5] = t0 * n0; + MONT_MAC(t0, t1, t2, u[5], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 6 */ + MONT_MAC(t0, t1, t2, a[0], b[6]); + MONT_MUL(v0, v1, v2, a[1], b[5]); + MONT_MUL(w0, w1, w2, a[2], b[4]); + MONT_MAC(t0, t1, t2, a[3], b[3]); + MONT_MAC(v0, v1, v2, a[4], b[2]); + MONT_MAC(w0, w1, w2, a[5], b[1]); + MONT_MAC(t0, t1, t2, a[6], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[6]); + MONT_MAC(w0, w1, w2, u[1], m[5]); + MONT_MAC(t0, t1, t2, u[2], m[4]); + MONT_MAC(v0, v1, v2, u[3], m[3]); + MONT_MAC(w0, w1, w2, u[4], m[2]); + MONT_MAC(t0, t1, t2, u[5], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[6] = t0 * n0; + MONT_MAC(t0, t1, t2, u[6], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 7 */ + MONT_MAC(t0, t1, t2, a[0], b[7]); + MONT_MUL(v0, v1, v2, a[1], b[6]); + MONT_MUL(w0, w1, w2, a[2], b[5]); + MONT_MAC(t0, t1, t2, a[3], b[4]); + MONT_MAC(v0, v1, v2, a[4], b[3]); + MONT_MAC(w0, w1, w2, a[5], b[2]); + MONT_MAC(t0, t1, t2, a[6], b[1]); + MONT_MAC(v0, v1, v2, a[7], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[7]); + MONT_MAC(t0, t1, t2, u[1], m[6]); + MONT_MAC(v0, v1, v2, u[2], m[5]); + MONT_MAC(w0, w1, w2, u[3], m[4]); + MONT_MAC(t0, t1, t2, u[4], m[3]); + MONT_MAC(v0, v1, v2, u[5], m[2]); + MONT_MAC(w0, w1, w2, u[6], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[7] = t0 * n0; + MONT_MAC(t0, t1, t2, u[7], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 8 */ + MONT_MAC(t0, t1, t2, a[0], b[8]); + MONT_MUL(v0, v1, v2, a[1], b[7]); + MONT_MUL(w0, w1, w2, a[2], b[6]); + MONT_MAC(t0, t1, t2, a[3], b[5]); + MONT_MAC(v0, v1, v2, a[4], b[4]); + MONT_MAC(w0, w1, w2, a[5], b[3]); + MONT_MAC(t0, t1, t2, a[6], b[2]); + MONT_MAC(v0, v1, v2, a[7], b[1]); + MONT_MAC(w0, w1, w2, a[8], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[8]); + MONT_MAC(v0, v1, v2, u[1], m[7]); + MONT_MAC(w0, w1, w2, u[2], m[6]); + MONT_MAC(t0, t1, t2, u[3], m[5]); + MONT_MAC(v0, v1, v2, u[4], m[4]); + MONT_MAC(w0, w1, w2, u[5], m[3]); + MONT_MAC(t0, t1, t2, u[6], m[2]); + MONT_MAC(v0, v1, v2, u[7], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[8] = t0 * n0; + MONT_MAC(t0, t1, t2, u[8], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 9 */ + MONT_MAC(t0, t1, t2, a[0], b[9]); + MONT_MUL(v0, v1, v2, a[1], b[8]); + MONT_MUL(w0, w1, w2, a[2], b[7]); + MONT_MAC(t0, t1, t2, a[3], b[6]); + MONT_MAC(v0, v1, v2, a[4], b[5]); + MONT_MAC(w0, w1, w2, a[5], b[4]); + MONT_MAC(t0, t1, t2, a[6], b[3]); + MONT_MAC(v0, v1, v2, a[7], b[2]); + MONT_MAC(w0, w1, w2, a[8], b[1]); + MONT_MAC(t0, t1, t2, a[9], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[9]); + MONT_MAC(w0, w1, w2, u[1], m[8]); + MONT_MAC(t0, t1, t2, u[2], m[7]); + MONT_MAC(v0, v1, v2, u[3], m[6]); + MONT_MAC(w0, w1, w2, u[4], m[5]); + MONT_MAC(t0, t1, t2, u[5], m[4]); + MONT_MAC(v0, v1, v2, u[6], m[3]); + MONT_MAC(w0, w1, w2, u[7], m[2]); + MONT_MAC(t0, t1, t2, u[8], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[9] = t0 * n0; + MONT_MAC(t0, t1, t2, u[9], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 10 */ + MONT_MAC(t0, t1, t2, a[0], b[10]); + MONT_MUL(v0, v1, v2, a[1], b[9]); + MONT_MUL(w0, w1, w2, a[2], b[8]); + MONT_MAC(t0, t1, t2, a[3], b[7]); + MONT_MAC(v0, v1, v2, a[4], b[6]); + MONT_MAC(w0, w1, w2, a[5], b[5]); + MONT_MAC(t0, t1, t2, a[6], b[4]); + MONT_MAC(v0, v1, v2, a[7], b[3]); + MONT_MAC(w0, w1, w2, a[8], b[2]); + MONT_MAC(t0, t1, t2, a[9], b[1]); + MONT_MAC(v0, v1, v2, a[10], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[10]); + MONT_MAC(t0, t1, t2, u[1], m[9]); + MONT_MAC(v0, v1, v2, u[2], m[8]); + MONT_MAC(w0, w1, w2, u[3], m[7]); + MONT_MAC(t0, t1, t2, u[4], m[6]); + MONT_MAC(v0, v1, v2, u[5], m[5]); + MONT_MAC(w0, w1, w2, u[6], m[4]); + MONT_MAC(t0, t1, t2, u[7], m[3]); + MONT_MAC(v0, v1, v2, u[8], m[2]); + MONT_MAC(w0, w1, w2, u[9], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[10] = t0 * n0; + MONT_MAC(t0, t1, t2, u[10], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 11 */ + MONT_MAC(t0, t1, t2, a[0], b[11]); + MONT_MUL(v0, v1, v2, a[1], b[10]); + MONT_MUL(w0, w1, w2, a[2], b[9]); + MONT_MAC(t0, t1, t2, a[3], b[8]); + MONT_MAC(v0, v1, v2, a[4], b[7]); + MONT_MAC(w0, w1, w2, a[5], b[6]); + MONT_MAC(t0, t1, t2, a[6], b[5]); + MONT_MAC(v0, v1, v2, a[7], b[4]); + MONT_MAC(w0, w1, w2, a[8], b[3]); + MONT_MAC(t0, t1, t2, a[9], b[2]); + MONT_MAC(v0, v1, v2, a[10], b[1]); + MONT_MAC(w0, w1, w2, a[11], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[11]); + MONT_MAC(v0, v1, v2, u[1], m[10]); + MONT_MAC(w0, w1, w2, u[2], m[9]); + MONT_MAC(t0, t1, t2, u[3], m[8]); + MONT_MAC(v0, v1, v2, u[4], m[7]); + MONT_MAC(w0, w1, w2, u[5], m[6]); + MONT_MAC(t0, t1, t2, u[6], m[5]); + MONT_MAC(v0, v1, v2, u[7], m[4]); + MONT_MAC(w0, w1, w2, u[8], m[3]); + MONT_MAC(t0, t1, t2, u[9], m[2]); + MONT_MAC(v0, v1, v2, u[10], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[11] = t0 * n0; + MONT_MAC(t0, t1, t2, u[11], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 12 */ + MONT_MAC(t0, t1, t2, a[0], b[12]); + MONT_MUL(v0, v1, v2, a[1], b[11]); + MONT_MUL(w0, w1, w2, a[2], b[10]); + MONT_MAC(t0, t1, t2, a[3], b[9]); + MONT_MAC(v0, v1, v2, a[4], b[8]); + MONT_MAC(w0, w1, w2, a[5], b[7]); + MONT_MAC(t0, t1, t2, a[6], b[6]); + MONT_MAC(v0, v1, v2, a[7], b[5]); + MONT_MAC(w0, w1, w2, a[8], b[4]); + MONT_MAC(t0, t1, t2, a[9], b[3]); + MONT_MAC(v0, v1, v2, a[10], b[2]); + MONT_MAC(w0, w1, w2, a[11], b[1]); + MONT_MAC(t0, t1, t2, a[12], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[12]); + MONT_MAC(w0, w1, w2, u[1], m[11]); + MONT_MAC(t0, t1, t2, u[2], m[10]); + MONT_MAC(v0, v1, v2, u[3], m[9]); + MONT_MAC(w0, w1, w2, u[4], m[8]); + MONT_MAC(t0, t1, t2, u[5], m[7]); + MONT_MAC(v0, v1, v2, u[6], m[6]); + MONT_MAC(w0, w1, w2, u[7], m[5]); + MONT_MAC(t0, t1, t2, u[8], m[4]); + MONT_MAC(v0, v1, v2, u[9], m[3]); + MONT_MAC(w0, w1, w2, u[10], m[2]); + MONT_MAC(t0, t1, t2, u[11], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[12] = t0 * n0; + MONT_MAC(t0, t1, t2, u[12], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 13 */ + MONT_MAC(t0, t1, t2, a[0], b[13]); + MONT_MUL(v0, v1, v2, a[1], b[12]); + MONT_MUL(w0, w1, w2, a[2], b[11]); + MONT_MAC(t0, t1, t2, a[3], b[10]); + MONT_MAC(v0, v1, v2, a[4], b[9]); + MONT_MAC(w0, w1, w2, a[5], b[8]); + MONT_MAC(t0, t1, t2, a[6], b[7]); + MONT_MAC(v0, v1, v2, a[7], b[6]); + MONT_MAC(w0, w1, w2, a[8], b[5]); + MONT_MAC(t0, t1, t2, a[9], b[4]); + MONT_MAC(v0, v1, v2, a[10], b[3]); + MONT_MAC(w0, w1, w2, a[11], b[2]); + MONT_MAC(t0, t1, t2, a[12], b[1]); + MONT_MAC(v0, v1, v2, a[13], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[13]); + MONT_MAC(t0, t1, t2, u[1], m[12]); + MONT_MAC(v0, v1, v2, u[2], m[11]); + MONT_MAC(w0, w1, w2, u[3], m[10]); + MONT_MAC(t0, t1, t2, u[4], m[9]); + MONT_MAC(v0, v1, v2, u[5], m[8]); + MONT_MAC(w0, w1, w2, u[6], m[7]); + MONT_MAC(t0, t1, t2, u[7], m[6]); + MONT_MAC(v0, v1, v2, u[8], m[5]); + MONT_MAC(w0, w1, w2, u[9], m[4]); + MONT_MAC(t0, t1, t2, u[10], m[3]); + MONT_MAC(v0, v1, v2, u[11], m[2]); + MONT_MAC(w0, w1, w2, u[12], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[13] = t0 * n0; + MONT_MAC(t0, t1, t2, u[13], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 14 */ + MONT_MAC(t0, t1, t2, a[0], b[14]); + MONT_MUL(v0, v1, v2, a[1], b[13]); + MONT_MUL(w0, w1, w2, a[2], b[12]); + MONT_MAC(t0, t1, t2, a[3], b[11]); + MONT_MAC(v0, v1, v2, a[4], b[10]); + MONT_MAC(w0, w1, w2, a[5], b[9]); + MONT_MAC(t0, t1, t2, a[6], b[8]); + MONT_MAC(v0, v1, v2, a[7], b[7]); + MONT_MAC(w0, w1, w2, a[8], b[6]); + MONT_MAC(t0, t1, t2, a[9], b[5]); + MONT_MAC(v0, v1, v2, a[10], b[4]); + MONT_MAC(w0, w1, w2, a[11], b[3]); + MONT_MAC(t0, t1, t2, a[12], b[2]); + MONT_MAC(v0, v1, v2, a[13], b[1]); + MONT_MAC(w0, w1, w2, a[14], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[14]); + MONT_MAC(v0, v1, v2, u[1], m[13]); + MONT_MAC(w0, w1, w2, u[2], m[12]); + MONT_MAC(t0, t1, t2, u[3], m[11]); + MONT_MAC(v0, v1, v2, u[4], m[10]); + MONT_MAC(w0, w1, w2, u[5], m[9]); + MONT_MAC(t0, t1, t2, u[6], m[8]); + MONT_MAC(v0, v1, v2, u[7], m[7]); + MONT_MAC(w0, w1, w2, u[8], m[6]); + MONT_MAC(t0, t1, t2, u[9], m[5]); + MONT_MAC(v0, v1, v2, u[10], m[4]); + MONT_MAC(w0, w1, w2, u[11], m[3]); + MONT_MAC(t0, t1, t2, u[12], m[2]); + MONT_MAC(v0, v1, v2, u[13], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[14] = t0 * n0; + MONT_MAC(t0, t1, t2, u[14], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 15 */ + MONT_MAC(t0, t1, t2, a[0], b[15]); + MONT_MUL(v0, v1, v2, a[1], b[14]); + MONT_MUL(w0, w1, w2, a[2], b[13]); + MONT_MAC(t0, t1, t2, a[3], b[12]); + MONT_MAC(v0, v1, v2, a[4], b[11]); + MONT_MAC(w0, w1, w2, a[5], b[10]); + MONT_MAC(t0, t1, t2, a[6], b[9]); + MONT_MAC(v0, v1, v2, a[7], b[8]); + MONT_MAC(w0, w1, w2, a[8], b[7]); + MONT_MAC(t0, t1, t2, a[9], b[6]); + MONT_MAC(v0, v1, v2, a[10], b[5]); + MONT_MAC(w0, w1, w2, a[11], b[4]); + MONT_MAC(t0, t1, t2, a[12], b[3]); + MONT_MAC(v0, v1, v2, a[13], b[2]); + MONT_MAC(w0, w1, w2, a[14], b[1]); + MONT_MAC(t0, t1, t2, a[15], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[15]); + MONT_MAC(w0, w1, w2, u[1], m[14]); + MONT_MAC(t0, t1, t2, u[2], m[13]); + MONT_MAC(v0, v1, v2, u[3], m[12]); + MONT_MAC(w0, w1, w2, u[4], m[11]); + MONT_MAC(t0, t1, t2, u[5], m[10]); + MONT_MAC(v0, v1, v2, u[6], m[9]); + MONT_MAC(w0, w1, w2, u[7], m[8]); + MONT_MAC(t0, t1, t2, u[8], m[7]); + MONT_MAC(v0, v1, v2, u[9], m[6]); + MONT_MAC(w0, w1, w2, u[10], m[5]); + MONT_MAC(t0, t1, t2, u[11], m[4]); + MONT_MAC(v0, v1, v2, u[12], m[3]); + MONT_MAC(w0, w1, w2, u[13], m[2]); + MONT_MAC(t0, t1, t2, u[14], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[15] = t0 * n0; + MONT_MAC(t0, t1, t2, u[15], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 16 */ + MONT_MAC(t0, t1, t2, a[1], b[15]); + MONT_MUL(v0, v1, v2, a[2], b[14]); + MONT_MUL(w0, w1, w2, a[3], b[13]); + MONT_MAC(t0, t1, t2, a[4], b[12]); + MONT_MAC(v0, v1, v2, a[5], b[11]); + MONT_MAC(w0, w1, w2, a[6], b[10]); + MONT_MAC(t0, t1, t2, a[7], b[9]); + MONT_MAC(v0, v1, v2, a[8], b[8]); + MONT_MAC(w0, w1, w2, a[9], b[7]); + MONT_MAC(t0, t1, t2, a[10], b[6]); + MONT_MAC(v0, v1, v2, a[11], b[5]); + MONT_MAC(w0, w1, w2, a[12], b[4]); + MONT_MAC(t0, t1, t2, a[13], b[3]); + MONT_MAC(v0, v1, v2, a[14], b[2]); + MONT_MAC(w0, w1, w2, a[15], b[1]); + MONT_MAC(t0, t1, t2, u[1], m[15]); + MONT_MAC(v0, v1, v2, u[2], m[14]); + MONT_MAC(w0, w1, w2, u[3], m[13]); + MONT_MAC(t0, t1, t2, u[4], m[12]); + MONT_MAC(v0, v1, v2, u[5], m[11]); + MONT_MAC(w0, w1, w2, u[6], m[10]); + MONT_MAC(t0, t1, t2, u[7], m[9]); + MONT_MAC(v0, v1, v2, u[8], m[8]); + MONT_MAC(w0, w1, w2, u[9], m[7]); + MONT_MAC(t0, t1, t2, u[10], m[6]); + MONT_MAC(v0, v1, v2, u[11], m[5]); + MONT_MAC(w0, w1, w2, u[12], m[4]); + MONT_MAC(t0, t1, t2, u[13], m[3]); + MONT_MAC(v0, v1, v2, u[14], m[2]); + MONT_MAC(w0, w1, w2, u[15], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[0] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 17 */ + MONT_MAC(t0, t1, t2, a[2], b[15]); + MONT_MUL(v0, v1, v2, a[3], b[14]); + MONT_MUL(w0, w1, w2, a[4], b[13]); + MONT_MAC(t0, t1, t2, a[5], b[12]); + MONT_MAC(v0, v1, v2, a[6], b[11]); + MONT_MAC(w0, w1, w2, a[7], b[10]); + MONT_MAC(t0, t1, t2, a[8], b[9]); + MONT_MAC(v0, v1, v2, a[9], b[8]); + MONT_MAC(w0, w1, w2, a[10], b[7]); + MONT_MAC(t0, t1, t2, a[11], b[6]); + MONT_MAC(v0, v1, v2, a[12], b[5]); + MONT_MAC(w0, w1, w2, a[13], b[4]); + MONT_MAC(t0, t1, t2, a[14], b[3]); + MONT_MAC(v0, v1, v2, a[15], b[2]); + MONT_MAC(w0, w1, w2, u[2], m[15]); + MONT_MAC(t0, t1, t2, u[3], m[14]); + MONT_MAC(v0, v1, v2, u[4], m[13]); + MONT_MAC(w0, w1, w2, u[5], m[12]); + MONT_MAC(t0, t1, t2, u[6], m[11]); + MONT_MAC(v0, v1, v2, u[7], m[10]); + MONT_MAC(w0, w1, w2, u[8], m[9]); + MONT_MAC(t0, t1, t2, u[9], m[8]); + MONT_MAC(v0, v1, v2, u[10], m[7]); + MONT_MAC(w0, w1, w2, u[11], m[6]); + MONT_MAC(t0, t1, t2, u[12], m[5]); + MONT_MAC(v0, v1, v2, u[13], m[4]); + MONT_MAC(w0, w1, w2, u[14], m[3]); + MONT_MAC(t0, t1, t2, u[15], m[2]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[1] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 18 */ + MONT_MAC(t0, t1, t2, a[3], b[15]); + MONT_MUL(v0, v1, v2, a[4], b[14]); + MONT_MUL(w0, w1, w2, a[5], b[13]); + MONT_MAC(t0, t1, t2, a[6], b[12]); + MONT_MAC(v0, v1, v2, a[7], b[11]); + MONT_MAC(w0, w1, w2, a[8], b[10]); + MONT_MAC(t0, t1, t2, a[9], b[9]); + MONT_MAC(v0, v1, v2, a[10], b[8]); + MONT_MAC(w0, w1, w2, a[11], b[7]); + MONT_MAC(t0, t1, t2, a[12], b[6]); + MONT_MAC(v0, v1, v2, a[13], b[5]); + MONT_MAC(w0, w1, w2, a[14], b[4]); + MONT_MAC(t0, t1, t2, a[15], b[3]); + MONT_MAC(v0, v1, v2, u[3], m[15]); + MONT_MAC(w0, w1, w2, u[4], m[14]); + MONT_MAC(t0, t1, t2, u[5], m[13]); + MONT_MAC(v0, v1, v2, u[6], m[12]); + MONT_MAC(w0, w1, w2, u[7], m[11]); + MONT_MAC(t0, t1, t2, u[8], m[10]); + MONT_MAC(v0, v1, v2, u[9], m[9]); + MONT_MAC(w0, w1, w2, u[10], m[8]); + MONT_MAC(t0, t1, t2, u[11], m[7]); + MONT_MAC(v0, v1, v2, u[12], m[6]); + MONT_MAC(w0, w1, w2, u[13], m[5]); + MONT_MAC(t0, t1, t2, u[14], m[4]); + MONT_MAC(v0, v1, v2, u[15], m[3]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[2] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 19 */ + MONT_MAC(t0, t1, t2, a[4], b[15]); + MONT_MUL(v0, v1, v2, a[5], b[14]); + MONT_MUL(w0, w1, w2, a[6], b[13]); + MONT_MAC(t0, t1, t2, a[7], b[12]); + MONT_MAC(v0, v1, v2, a[8], b[11]); + MONT_MAC(w0, w1, w2, a[9], b[10]); + MONT_MAC(t0, t1, t2, a[10], b[9]); + MONT_MAC(v0, v1, v2, a[11], b[8]); + MONT_MAC(w0, w1, w2, a[12], b[7]); + MONT_MAC(t0, t1, t2, a[13], b[6]); + MONT_MAC(v0, v1, v2, a[14], b[5]); + MONT_MAC(w0, w1, w2, a[15], b[4]); + MONT_MAC(t0, t1, t2, u[4], m[15]); + MONT_MAC(v0, v1, v2, u[5], m[14]); + MONT_MAC(w0, w1, w2, u[6], m[13]); + MONT_MAC(t0, t1, t2, u[7], m[12]); + MONT_MAC(v0, v1, v2, u[8], m[11]); + MONT_MAC(w0, w1, w2, u[9], m[10]); + MONT_MAC(t0, t1, t2, u[10], m[9]); + MONT_MAC(v0, v1, v2, u[11], m[8]); + MONT_MAC(w0, w1, w2, u[12], m[7]); + MONT_MAC(t0, t1, t2, u[13], m[6]); + MONT_MAC(v0, v1, v2, u[14], m[5]); + MONT_MAC(w0, w1, w2, u[15], m[4]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[3] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 20 */ + MONT_MAC(t0, t1, t2, a[5], b[15]); + MONT_MUL(v0, v1, v2, a[6], b[14]); + MONT_MUL(w0, w1, w2, a[7], b[13]); + MONT_MAC(t0, t1, t2, a[8], b[12]); + MONT_MAC(v0, v1, v2, a[9], b[11]); + MONT_MAC(w0, w1, w2, a[10], b[10]); + MONT_MAC(t0, t1, t2, a[11], b[9]); + MONT_MAC(v0, v1, v2, a[12], b[8]); + MONT_MAC(w0, w1, w2, a[13], b[7]); + MONT_MAC(t0, t1, t2, a[14], b[6]); + MONT_MAC(v0, v1, v2, a[15], b[5]); + MONT_MAC(w0, w1, w2, u[5], m[15]); + MONT_MAC(t0, t1, t2, u[6], m[14]); + MONT_MAC(v0, v1, v2, u[7], m[13]); + MONT_MAC(w0, w1, w2, u[8], m[12]); + MONT_MAC(t0, t1, t2, u[9], m[11]); + MONT_MAC(v0, v1, v2, u[10], m[10]); + MONT_MAC(w0, w1, w2, u[11], m[9]); + MONT_MAC(t0, t1, t2, u[12], m[8]); + MONT_MAC(v0, v1, v2, u[13], m[7]); + MONT_MAC(w0, w1, w2, u[14], m[6]); + MONT_MAC(t0, t1, t2, u[15], m[5]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[4] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 21 */ + MONT_MAC(t0, t1, t2, a[6], b[15]); + MONT_MUL(v0, v1, v2, a[7], b[14]); + MONT_MUL(w0, w1, w2, a[8], b[13]); + MONT_MAC(t0, t1, t2, a[9], b[12]); + MONT_MAC(v0, v1, v2, a[10], b[11]); + MONT_MAC(w0, w1, w2, a[11], b[10]); + MONT_MAC(t0, t1, t2, a[12], b[9]); + MONT_MAC(v0, v1, v2, a[13], b[8]); + MONT_MAC(w0, w1, w2, a[14], b[7]); + MONT_MAC(t0, t1, t2, a[15], b[6]); + MONT_MAC(v0, v1, v2, u[6], m[15]); + MONT_MAC(w0, w1, w2, u[7], m[14]); + MONT_MAC(t0, t1, t2, u[8], m[13]); + MONT_MAC(v0, v1, v2, u[9], m[12]); + MONT_MAC(w0, w1, w2, u[10], m[11]); + MONT_MAC(t0, t1, t2, u[11], m[10]); + MONT_MAC(v0, v1, v2, u[12], m[9]); + MONT_MAC(w0, w1, w2, u[13], m[8]); + MONT_MAC(t0, t1, t2, u[14], m[7]); + MONT_MAC(v0, v1, v2, u[15], m[6]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[5] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 22 */ + MONT_MAC(t0, t1, t2, a[7], b[15]); + MONT_MUL(v0, v1, v2, a[8], b[14]); + MONT_MUL(w0, w1, w2, a[9], b[13]); + MONT_MAC(t0, t1, t2, a[10], b[12]); + MONT_MAC(v0, v1, v2, a[11], b[11]); + MONT_MAC(w0, w1, w2, a[12], b[10]); + MONT_MAC(t0, t1, t2, a[13], b[9]); + MONT_MAC(v0, v1, v2, a[14], b[8]); + MONT_MAC(w0, w1, w2, a[15], b[7]); + MONT_MAC(t0, t1, t2, u[7], m[15]); + MONT_MAC(v0, v1, v2, u[8], m[14]); + MONT_MAC(w0, w1, w2, u[9], m[13]); + MONT_MAC(t0, t1, t2, u[10], m[12]); + MONT_MAC(v0, v1, v2, u[11], m[11]); + MONT_MAC(w0, w1, w2, u[12], m[10]); + MONT_MAC(t0, t1, t2, u[13], m[9]); + MONT_MAC(v0, v1, v2, u[14], m[8]); + MONT_MAC(w0, w1, w2, u[15], m[7]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[6] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 23 */ + MONT_MAC(t0, t1, t2, a[8], b[15]); + MONT_MUL(v0, v1, v2, a[9], b[14]); + MONT_MUL(w0, w1, w2, a[10], b[13]); + MONT_MAC(t0, t1, t2, a[11], b[12]); + MONT_MAC(v0, v1, v2, a[12], b[11]); + MONT_MAC(w0, w1, w2, a[13], b[10]); + MONT_MAC(t0, t1, t2, a[14], b[9]); + MONT_MAC(v0, v1, v2, a[15], b[8]); + MONT_MAC(w0, w1, w2, u[8], m[15]); + MONT_MAC(t0, t1, t2, u[9], m[14]); + MONT_MAC(v0, v1, v2, u[10], m[13]); + MONT_MAC(w0, w1, w2, u[11], m[12]); + MONT_MAC(t0, t1, t2, u[12], m[11]); + MONT_MAC(v0, v1, v2, u[13], m[10]); + MONT_MAC(w0, w1, w2, u[14], m[9]); + MONT_MAC(t0, t1, t2, u[15], m[8]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[7] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 24 */ + MONT_MAC(t0, t1, t2, a[9], b[15]); + MONT_MUL(v0, v1, v2, a[10], b[14]); + MONT_MUL(w0, w1, w2, a[11], b[13]); + MONT_MAC(t0, t1, t2, a[12], b[12]); + MONT_MAC(v0, v1, v2, a[13], b[11]); + MONT_MAC(w0, w1, w2, a[14], b[10]); + MONT_MAC(t0, t1, t2, a[15], b[9]); + MONT_MAC(v0, v1, v2, u[9], m[15]); + MONT_MAC(w0, w1, w2, u[10], m[14]); + MONT_MAC(t0, t1, t2, u[11], m[13]); + MONT_MAC(v0, v1, v2, u[12], m[12]); + MONT_MAC(w0, w1, w2, u[13], m[11]); + MONT_MAC(t0, t1, t2, u[14], m[10]); + MONT_MAC(v0, v1, v2, u[15], m[9]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[8] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 25 */ + MONT_MAC(t0, t1, t2, a[10], b[15]); + MONT_MUL(v0, v1, v2, a[11], b[14]); + MONT_MUL(w0, w1, w2, a[12], b[13]); + MONT_MAC(t0, t1, t2, a[13], b[12]); + MONT_MAC(v0, v1, v2, a[14], b[11]); + MONT_MAC(w0, w1, w2, a[15], b[10]); + MONT_MAC(t0, t1, t2, u[10], m[15]); + MONT_MAC(v0, v1, v2, u[11], m[14]); + MONT_MAC(w0, w1, w2, u[12], m[13]); + MONT_MAC(t0, t1, t2, u[13], m[12]); + MONT_MAC(v0, v1, v2, u[14], m[11]); + MONT_MAC(w0, w1, w2, u[15], m[10]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[9] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 26 */ + MONT_MAC(t0, t1, t2, a[11], b[15]); + MONT_MUL(v0, v1, v2, a[12], b[14]); + MONT_MUL(w0, w1, w2, a[13], b[13]); + MONT_MAC(t0, t1, t2, a[14], b[12]); + MONT_MAC(v0, v1, v2, a[15], b[11]); + MONT_MAC(w0, w1, w2, u[11], m[15]); + MONT_MAC(t0, t1, t2, u[12], m[14]); + MONT_MAC(v0, v1, v2, u[13], m[13]); + MONT_MAC(w0, w1, w2, u[14], m[12]); + MONT_MAC(t0, t1, t2, u[15], m[11]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[10] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 27 */ + MONT_MAC(t0, t1, t2, a[12], b[15]); + MONT_MUL(v0, v1, v2, a[13], b[14]); + MONT_MUL(w0, w1, w2, a[14], b[13]); + MONT_MAC(t0, t1, t2, a[15], b[12]); + MONT_MAC(v0, v1, v2, u[12], m[15]); + MONT_MAC(w0, w1, w2, u[13], m[14]); + MONT_MAC(t0, t1, t2, u[14], m[13]); + MONT_MAC(v0, v1, v2, u[15], m[12]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[11] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 28 */ + MONT_MAC(t0, t1, t2, a[13], b[15]); + MONT_MUL(v0, v1, v2, a[14], b[14]); + MONT_MUL(w0, w1, w2, a[15], b[13]); + MONT_MAC(t0, t1, t2, u[13], m[15]); + MONT_MAC(v0, v1, v2, u[14], m[14]); + MONT_MAC(w0, w1, w2, u[15], m[13]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[12] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 29 */ + MONT_MAC(t0, t1, t2, a[14], b[15]); + MONT_MUL(v0, v1, v2, a[15], b[14]); + MONT_MUL(w0, w1, w2, u[14], m[15]); + MONT_MAC(t0, t1, t2, u[15], m[14]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[13] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 30 */ + MONT_MAC(t0, t1, t2, a[15], b[15]); + MONT_MUL(v0, v1, v2, u[15], m[15]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + z[14] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + z[15] = t0; + + /* under 2m, with t1 the bit above n limbs: at most one subtraction */ + borrow = sub_n(r, z, m, 16); + take = t1 | (borrow ^ 1); + select_n(r, r, z, take & 1, 16); +} + +/* r = a * a * R^-1 mod m, for a under m; t is 2n limbs of scratch */ +static void crypton_mont_sqr16(limb_t *r, const limb_t *a, + const limb_t *m, limb_t n0, limb_t *t) +{ + limb_t *u = t, *z = t + 16; + limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take; + limb_t s0, s1, s2, c0, c1, c2; + + /* column 0 */ + MONT_MAC(t0, t1, t2, a[0], a[0]); + u[0] = t0 * n0; + MONT_MAC(t0, t1, t2, u[0], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 1 */ + MONT_MUL(s0, s1, s2, a[0], a[1]); + MONT_MAC(t0, t1, t2, u[0], m[1]); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[1] = t0 * n0; + MONT_MAC(t0, t1, t2, u[1], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 2 */ + MONT_MUL(s0, s1, s2, a[0], a[2]); + MONT_MAC(t0, t1, t2, a[1], a[1]); + MONT_MUL(v0, v1, v2, u[0], m[2]); + MONT_MUL(w0, w1, w2, u[1], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[2] = t0 * n0; + MONT_MAC(t0, t1, t2, u[2], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 3 */ + MONT_MUL(s0, s1, s2, a[0], a[3]); + MONT_MUL(c0, c1, c2, a[1], a[2]); + MONT_MAC(t0, t1, t2, u[0], m[3]); + MONT_MUL(v0, v1, v2, u[1], m[2]); + MONT_MUL(w0, w1, w2, u[2], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[3] = t0 * n0; + MONT_MAC(t0, t1, t2, u[3], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 4 */ + MONT_MUL(s0, s1, s2, a[0], a[4]); + MONT_MUL(c0, c1, c2, a[1], a[3]); + MONT_MAC(t0, t1, t2, a[2], a[2]); + MONT_MUL(v0, v1, v2, u[0], m[4]); + MONT_MUL(w0, w1, w2, u[1], m[3]); + MONT_MAC(t0, t1, t2, u[2], m[2]); + MONT_MAC(v0, v1, v2, u[3], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[4] = t0 * n0; + MONT_MAC(t0, t1, t2, u[4], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 5 */ + MONT_MUL(s0, s1, s2, a[0], a[5]); + MONT_MUL(c0, c1, c2, a[1], a[4]); + MONT_MAC(s0, s1, s2, a[2], a[3]); + MONT_MAC(t0, t1, t2, u[0], m[5]); + MONT_MUL(v0, v1, v2, u[1], m[4]); + MONT_MUL(w0, w1, w2, u[2], m[3]); + MONT_MAC(t0, t1, t2, u[3], m[2]); + MONT_MAC(v0, v1, v2, u[4], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[5] = t0 * n0; + MONT_MAC(t0, t1, t2, u[5], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 6 */ + MONT_MUL(s0, s1, s2, a[0], a[6]); + MONT_MUL(c0, c1, c2, a[1], a[5]); + MONT_MAC(s0, s1, s2, a[2], a[4]); + MONT_MAC(t0, t1, t2, a[3], a[3]); + MONT_MUL(v0, v1, v2, u[0], m[6]); + MONT_MUL(w0, w1, w2, u[1], m[5]); + MONT_MAC(t0, t1, t2, u[2], m[4]); + MONT_MAC(v0, v1, v2, u[3], m[3]); + MONT_MAC(w0, w1, w2, u[4], m[2]); + MONT_MAC(t0, t1, t2, u[5], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[6] = t0 * n0; + MONT_MAC(t0, t1, t2, u[6], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 7 */ + MONT_MUL(s0, s1, s2, a[0], a[7]); + MONT_MUL(c0, c1, c2, a[1], a[6]); + MONT_MAC(s0, s1, s2, a[2], a[5]); + MONT_MAC(c0, c1, c2, a[3], a[4]); + MONT_MAC(t0, t1, t2, u[0], m[7]); + MONT_MUL(v0, v1, v2, u[1], m[6]); + MONT_MUL(w0, w1, w2, u[2], m[5]); + MONT_MAC(t0, t1, t2, u[3], m[4]); + MONT_MAC(v0, v1, v2, u[4], m[3]); + MONT_MAC(w0, w1, w2, u[5], m[2]); + MONT_MAC(t0, t1, t2, u[6], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[7] = t0 * n0; + MONT_MAC(t0, t1, t2, u[7], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 8 */ + MONT_MUL(s0, s1, s2, a[0], a[8]); + MONT_MUL(c0, c1, c2, a[1], a[7]); + MONT_MAC(s0, s1, s2, a[2], a[6]); + MONT_MAC(c0, c1, c2, a[3], a[5]); + MONT_MAC(t0, t1, t2, a[4], a[4]); + MONT_MUL(v0, v1, v2, u[0], m[8]); + MONT_MUL(w0, w1, w2, u[1], m[7]); + MONT_MAC(t0, t1, t2, u[2], m[6]); + MONT_MAC(v0, v1, v2, u[3], m[5]); + MONT_MAC(w0, w1, w2, u[4], m[4]); + MONT_MAC(t0, t1, t2, u[5], m[3]); + MONT_MAC(v0, v1, v2, u[6], m[2]); + MONT_MAC(w0, w1, w2, u[7], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[8] = t0 * n0; + MONT_MAC(t0, t1, t2, u[8], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 9 */ + MONT_MUL(s0, s1, s2, a[0], a[9]); + MONT_MUL(c0, c1, c2, a[1], a[8]); + MONT_MAC(s0, s1, s2, a[2], a[7]); + MONT_MAC(c0, c1, c2, a[3], a[6]); + MONT_MAC(s0, s1, s2, a[4], a[5]); + MONT_MAC(t0, t1, t2, u[0], m[9]); + MONT_MUL(v0, v1, v2, u[1], m[8]); + MONT_MUL(w0, w1, w2, u[2], m[7]); + MONT_MAC(t0, t1, t2, u[3], m[6]); + MONT_MAC(v0, v1, v2, u[4], m[5]); + MONT_MAC(w0, w1, w2, u[5], m[4]); + MONT_MAC(t0, t1, t2, u[6], m[3]); + MONT_MAC(v0, v1, v2, u[7], m[2]); + MONT_MAC(w0, w1, w2, u[8], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[9] = t0 * n0; + MONT_MAC(t0, t1, t2, u[9], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 10 */ + MONT_MUL(s0, s1, s2, a[0], a[10]); + MONT_MUL(c0, c1, c2, a[1], a[9]); + MONT_MAC(s0, s1, s2, a[2], a[8]); + MONT_MAC(c0, c1, c2, a[3], a[7]); + MONT_MAC(s0, s1, s2, a[4], a[6]); + MONT_MAC(t0, t1, t2, a[5], a[5]); + MONT_MUL(v0, v1, v2, u[0], m[10]); + MONT_MUL(w0, w1, w2, u[1], m[9]); + MONT_MAC(t0, t1, t2, u[2], m[8]); + MONT_MAC(v0, v1, v2, u[3], m[7]); + MONT_MAC(w0, w1, w2, u[4], m[6]); + MONT_MAC(t0, t1, t2, u[5], m[5]); + MONT_MAC(v0, v1, v2, u[6], m[4]); + MONT_MAC(w0, w1, w2, u[7], m[3]); + MONT_MAC(t0, t1, t2, u[8], m[2]); + MONT_MAC(v0, v1, v2, u[9], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[10] = t0 * n0; + MONT_MAC(t0, t1, t2, u[10], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 11 */ + MONT_MUL(s0, s1, s2, a[0], a[11]); + MONT_MUL(c0, c1, c2, a[1], a[10]); + MONT_MAC(s0, s1, s2, a[2], a[9]); + MONT_MAC(c0, c1, c2, a[3], a[8]); + MONT_MAC(s0, s1, s2, a[4], a[7]); + MONT_MAC(c0, c1, c2, a[5], a[6]); + MONT_MAC(t0, t1, t2, u[0], m[11]); + MONT_MUL(v0, v1, v2, u[1], m[10]); + MONT_MUL(w0, w1, w2, u[2], m[9]); + MONT_MAC(t0, t1, t2, u[3], m[8]); + MONT_MAC(v0, v1, v2, u[4], m[7]); + MONT_MAC(w0, w1, w2, u[5], m[6]); + MONT_MAC(t0, t1, t2, u[6], m[5]); + MONT_MAC(v0, v1, v2, u[7], m[4]); + MONT_MAC(w0, w1, w2, u[8], m[3]); + MONT_MAC(t0, t1, t2, u[9], m[2]); + MONT_MAC(v0, v1, v2, u[10], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[11] = t0 * n0; + MONT_MAC(t0, t1, t2, u[11], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 12 */ + MONT_MUL(s0, s1, s2, a[0], a[12]); + MONT_MUL(c0, c1, c2, a[1], a[11]); + MONT_MAC(s0, s1, s2, a[2], a[10]); + MONT_MAC(c0, c1, c2, a[3], a[9]); + MONT_MAC(s0, s1, s2, a[4], a[8]); + MONT_MAC(c0, c1, c2, a[5], a[7]); + MONT_MAC(t0, t1, t2, a[6], a[6]); + MONT_MUL(v0, v1, v2, u[0], m[12]); + MONT_MUL(w0, w1, w2, u[1], m[11]); + MONT_MAC(t0, t1, t2, u[2], m[10]); + MONT_MAC(v0, v1, v2, u[3], m[9]); + MONT_MAC(w0, w1, w2, u[4], m[8]); + MONT_MAC(t0, t1, t2, u[5], m[7]); + MONT_MAC(v0, v1, v2, u[6], m[6]); + MONT_MAC(w0, w1, w2, u[7], m[5]); + MONT_MAC(t0, t1, t2, u[8], m[4]); + MONT_MAC(v0, v1, v2, u[9], m[3]); + MONT_MAC(w0, w1, w2, u[10], m[2]); + MONT_MAC(t0, t1, t2, u[11], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[12] = t0 * n0; + MONT_MAC(t0, t1, t2, u[12], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 13 */ + MONT_MUL(s0, s1, s2, a[0], a[13]); + MONT_MUL(c0, c1, c2, a[1], a[12]); + MONT_MAC(s0, s1, s2, a[2], a[11]); + MONT_MAC(c0, c1, c2, a[3], a[10]); + MONT_MAC(s0, s1, s2, a[4], a[9]); + MONT_MAC(c0, c1, c2, a[5], a[8]); + MONT_MAC(s0, s1, s2, a[6], a[7]); + MONT_MAC(t0, t1, t2, u[0], m[13]); + MONT_MUL(v0, v1, v2, u[1], m[12]); + MONT_MUL(w0, w1, w2, u[2], m[11]); + MONT_MAC(t0, t1, t2, u[3], m[10]); + MONT_MAC(v0, v1, v2, u[4], m[9]); + MONT_MAC(w0, w1, w2, u[5], m[8]); + MONT_MAC(t0, t1, t2, u[6], m[7]); + MONT_MAC(v0, v1, v2, u[7], m[6]); + MONT_MAC(w0, w1, w2, u[8], m[5]); + MONT_MAC(t0, t1, t2, u[9], m[4]); + MONT_MAC(v0, v1, v2, u[10], m[3]); + MONT_MAC(w0, w1, w2, u[11], m[2]); + MONT_MAC(t0, t1, t2, u[12], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[13] = t0 * n0; + MONT_MAC(t0, t1, t2, u[13], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 14 */ + MONT_MUL(s0, s1, s2, a[0], a[14]); + MONT_MUL(c0, c1, c2, a[1], a[13]); + MONT_MAC(s0, s1, s2, a[2], a[12]); + MONT_MAC(c0, c1, c2, a[3], a[11]); + MONT_MAC(s0, s1, s2, a[4], a[10]); + MONT_MAC(c0, c1, c2, a[5], a[9]); + MONT_MAC(s0, s1, s2, a[6], a[8]); + MONT_MAC(t0, t1, t2, a[7], a[7]); + MONT_MUL(v0, v1, v2, u[0], m[14]); + MONT_MUL(w0, w1, w2, u[1], m[13]); + MONT_MAC(t0, t1, t2, u[2], m[12]); + MONT_MAC(v0, v1, v2, u[3], m[11]); + MONT_MAC(w0, w1, w2, u[4], m[10]); + MONT_MAC(t0, t1, t2, u[5], m[9]); + MONT_MAC(v0, v1, v2, u[6], m[8]); + MONT_MAC(w0, w1, w2, u[7], m[7]); + MONT_MAC(t0, t1, t2, u[8], m[6]); + MONT_MAC(v0, v1, v2, u[9], m[5]); + MONT_MAC(w0, w1, w2, u[10], m[4]); + MONT_MAC(t0, t1, t2, u[11], m[3]); + MONT_MAC(v0, v1, v2, u[12], m[2]); + MONT_MAC(w0, w1, w2, u[13], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[14] = t0 * n0; + MONT_MAC(t0, t1, t2, u[14], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 15 */ + MONT_MUL(s0, s1, s2, a[0], a[15]); + MONT_MUL(c0, c1, c2, a[1], a[14]); + MONT_MAC(s0, s1, s2, a[2], a[13]); + MONT_MAC(c0, c1, c2, a[3], a[12]); + MONT_MAC(s0, s1, s2, a[4], a[11]); + MONT_MAC(c0, c1, c2, a[5], a[10]); + MONT_MAC(s0, s1, s2, a[6], a[9]); + MONT_MAC(c0, c1, c2, a[7], a[8]); + MONT_MAC(t0, t1, t2, u[0], m[15]); + MONT_MUL(v0, v1, v2, u[1], m[14]); + MONT_MUL(w0, w1, w2, u[2], m[13]); + MONT_MAC(t0, t1, t2, u[3], m[12]); + MONT_MAC(v0, v1, v2, u[4], m[11]); + MONT_MAC(w0, w1, w2, u[5], m[10]); + MONT_MAC(t0, t1, t2, u[6], m[9]); + MONT_MAC(v0, v1, v2, u[7], m[8]); + MONT_MAC(w0, w1, w2, u[8], m[7]); + MONT_MAC(t0, t1, t2, u[9], m[6]); + MONT_MAC(v0, v1, v2, u[10], m[5]); + MONT_MAC(w0, w1, w2, u[11], m[4]); + MONT_MAC(t0, t1, t2, u[12], m[3]); + MONT_MAC(v0, v1, v2, u[13], m[2]); + MONT_MAC(w0, w1, w2, u[14], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[15] = t0 * n0; + MONT_MAC(t0, t1, t2, u[15], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 16 */ + MONT_MUL(s0, s1, s2, a[1], a[15]); + MONT_MUL(c0, c1, c2, a[2], a[14]); + MONT_MAC(s0, s1, s2, a[3], a[13]); + MONT_MAC(c0, c1, c2, a[4], a[12]); + MONT_MAC(s0, s1, s2, a[5], a[11]); + MONT_MAC(c0, c1, c2, a[6], a[10]); + MONT_MAC(s0, s1, s2, a[7], a[9]); + MONT_MAC(t0, t1, t2, a[8], a[8]); + MONT_MUL(v0, v1, v2, u[1], m[15]); + MONT_MUL(w0, w1, w2, u[2], m[14]); + MONT_MAC(t0, t1, t2, u[3], m[13]); + MONT_MAC(v0, v1, v2, u[4], m[12]); + MONT_MAC(w0, w1, w2, u[5], m[11]); + MONT_MAC(t0, t1, t2, u[6], m[10]); + MONT_MAC(v0, v1, v2, u[7], m[9]); + MONT_MAC(w0, w1, w2, u[8], m[8]); + MONT_MAC(t0, t1, t2, u[9], m[7]); + MONT_MAC(v0, v1, v2, u[10], m[6]); + MONT_MAC(w0, w1, w2, u[11], m[5]); + MONT_MAC(t0, t1, t2, u[12], m[4]); + MONT_MAC(v0, v1, v2, u[13], m[3]); + MONT_MAC(w0, w1, w2, u[14], m[2]); + MONT_MAC(t0, t1, t2, u[15], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[0] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 17 */ + MONT_MUL(s0, s1, s2, a[2], a[15]); + MONT_MUL(c0, c1, c2, a[3], a[14]); + MONT_MAC(s0, s1, s2, a[4], a[13]); + MONT_MAC(c0, c1, c2, a[5], a[12]); + MONT_MAC(s0, s1, s2, a[6], a[11]); + MONT_MAC(c0, c1, c2, a[7], a[10]); + MONT_MAC(s0, s1, s2, a[8], a[9]); + MONT_MAC(t0, t1, t2, u[2], m[15]); + MONT_MUL(v0, v1, v2, u[3], m[14]); + MONT_MUL(w0, w1, w2, u[4], m[13]); + MONT_MAC(t0, t1, t2, u[5], m[12]); + MONT_MAC(v0, v1, v2, u[6], m[11]); + MONT_MAC(w0, w1, w2, u[7], m[10]); + MONT_MAC(t0, t1, t2, u[8], m[9]); + MONT_MAC(v0, v1, v2, u[9], m[8]); + MONT_MAC(w0, w1, w2, u[10], m[7]); + MONT_MAC(t0, t1, t2, u[11], m[6]); + MONT_MAC(v0, v1, v2, u[12], m[5]); + MONT_MAC(w0, w1, w2, u[13], m[4]); + MONT_MAC(t0, t1, t2, u[14], m[3]); + MONT_MAC(v0, v1, v2, u[15], m[2]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[1] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 18 */ + MONT_MUL(s0, s1, s2, a[3], a[15]); + MONT_MUL(c0, c1, c2, a[4], a[14]); + MONT_MAC(s0, s1, s2, a[5], a[13]); + MONT_MAC(c0, c1, c2, a[6], a[12]); + MONT_MAC(s0, s1, s2, a[7], a[11]); + MONT_MAC(c0, c1, c2, a[8], a[10]); + MONT_MAC(t0, t1, t2, a[9], a[9]); + MONT_MUL(v0, v1, v2, u[3], m[15]); + MONT_MUL(w0, w1, w2, u[4], m[14]); + MONT_MAC(t0, t1, t2, u[5], m[13]); + MONT_MAC(v0, v1, v2, u[6], m[12]); + MONT_MAC(w0, w1, w2, u[7], m[11]); + MONT_MAC(t0, t1, t2, u[8], m[10]); + MONT_MAC(v0, v1, v2, u[9], m[9]); + MONT_MAC(w0, w1, w2, u[10], m[8]); + MONT_MAC(t0, t1, t2, u[11], m[7]); + MONT_MAC(v0, v1, v2, u[12], m[6]); + MONT_MAC(w0, w1, w2, u[13], m[5]); + MONT_MAC(t0, t1, t2, u[14], m[4]); + MONT_MAC(v0, v1, v2, u[15], m[3]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[2] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 19 */ + MONT_MUL(s0, s1, s2, a[4], a[15]); + MONT_MUL(c0, c1, c2, a[5], a[14]); + MONT_MAC(s0, s1, s2, a[6], a[13]); + MONT_MAC(c0, c1, c2, a[7], a[12]); + MONT_MAC(s0, s1, s2, a[8], a[11]); + MONT_MAC(c0, c1, c2, a[9], a[10]); + MONT_MAC(t0, t1, t2, u[4], m[15]); + MONT_MUL(v0, v1, v2, u[5], m[14]); + MONT_MUL(w0, w1, w2, u[6], m[13]); + MONT_MAC(t0, t1, t2, u[7], m[12]); + MONT_MAC(v0, v1, v2, u[8], m[11]); + MONT_MAC(w0, w1, w2, u[9], m[10]); + MONT_MAC(t0, t1, t2, u[10], m[9]); + MONT_MAC(v0, v1, v2, u[11], m[8]); + MONT_MAC(w0, w1, w2, u[12], m[7]); + MONT_MAC(t0, t1, t2, u[13], m[6]); + MONT_MAC(v0, v1, v2, u[14], m[5]); + MONT_MAC(w0, w1, w2, u[15], m[4]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[3] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 20 */ + MONT_MUL(s0, s1, s2, a[5], a[15]); + MONT_MUL(c0, c1, c2, a[6], a[14]); + MONT_MAC(s0, s1, s2, a[7], a[13]); + MONT_MAC(c0, c1, c2, a[8], a[12]); + MONT_MAC(s0, s1, s2, a[9], a[11]); + MONT_MAC(t0, t1, t2, a[10], a[10]); + MONT_MUL(v0, v1, v2, u[5], m[15]); + MONT_MUL(w0, w1, w2, u[6], m[14]); + MONT_MAC(t0, t1, t2, u[7], m[13]); + MONT_MAC(v0, v1, v2, u[8], m[12]); + MONT_MAC(w0, w1, w2, u[9], m[11]); + MONT_MAC(t0, t1, t2, u[10], m[10]); + MONT_MAC(v0, v1, v2, u[11], m[9]); + MONT_MAC(w0, w1, w2, u[12], m[8]); + MONT_MAC(t0, t1, t2, u[13], m[7]); + MONT_MAC(v0, v1, v2, u[14], m[6]); + MONT_MAC(w0, w1, w2, u[15], m[5]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[4] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 21 */ + MONT_MUL(s0, s1, s2, a[6], a[15]); + MONT_MUL(c0, c1, c2, a[7], a[14]); + MONT_MAC(s0, s1, s2, a[8], a[13]); + MONT_MAC(c0, c1, c2, a[9], a[12]); + MONT_MAC(s0, s1, s2, a[10], a[11]); + MONT_MAC(t0, t1, t2, u[6], m[15]); + MONT_MUL(v0, v1, v2, u[7], m[14]); + MONT_MUL(w0, w1, w2, u[8], m[13]); + MONT_MAC(t0, t1, t2, u[9], m[12]); + MONT_MAC(v0, v1, v2, u[10], m[11]); + MONT_MAC(w0, w1, w2, u[11], m[10]); + MONT_MAC(t0, t1, t2, u[12], m[9]); + MONT_MAC(v0, v1, v2, u[13], m[8]); + MONT_MAC(w0, w1, w2, u[14], m[7]); + MONT_MAC(t0, t1, t2, u[15], m[6]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[5] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 22 */ + MONT_MUL(s0, s1, s2, a[7], a[15]); + MONT_MUL(c0, c1, c2, a[8], a[14]); + MONT_MAC(s0, s1, s2, a[9], a[13]); + MONT_MAC(c0, c1, c2, a[10], a[12]); + MONT_MAC(t0, t1, t2, a[11], a[11]); + MONT_MUL(v0, v1, v2, u[7], m[15]); + MONT_MUL(w0, w1, w2, u[8], m[14]); + MONT_MAC(t0, t1, t2, u[9], m[13]); + MONT_MAC(v0, v1, v2, u[10], m[12]); + MONT_MAC(w0, w1, w2, u[11], m[11]); + MONT_MAC(t0, t1, t2, u[12], m[10]); + MONT_MAC(v0, v1, v2, u[13], m[9]); + MONT_MAC(w0, w1, w2, u[14], m[8]); + MONT_MAC(t0, t1, t2, u[15], m[7]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[6] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 23 */ + MONT_MUL(s0, s1, s2, a[8], a[15]); + MONT_MUL(c0, c1, c2, a[9], a[14]); + MONT_MAC(s0, s1, s2, a[10], a[13]); + MONT_MAC(c0, c1, c2, a[11], a[12]); + MONT_MAC(t0, t1, t2, u[8], m[15]); + MONT_MUL(v0, v1, v2, u[9], m[14]); + MONT_MUL(w0, w1, w2, u[10], m[13]); + MONT_MAC(t0, t1, t2, u[11], m[12]); + MONT_MAC(v0, v1, v2, u[12], m[11]); + MONT_MAC(w0, w1, w2, u[13], m[10]); + MONT_MAC(t0, t1, t2, u[14], m[9]); + MONT_MAC(v0, v1, v2, u[15], m[8]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[7] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 24 */ + MONT_MUL(s0, s1, s2, a[9], a[15]); + MONT_MUL(c0, c1, c2, a[10], a[14]); + MONT_MAC(s0, s1, s2, a[11], a[13]); + MONT_MAC(t0, t1, t2, a[12], a[12]); + MONT_MUL(v0, v1, v2, u[9], m[15]); + MONT_MUL(w0, w1, w2, u[10], m[14]); + MONT_MAC(t0, t1, t2, u[11], m[13]); + MONT_MAC(v0, v1, v2, u[12], m[12]); + MONT_MAC(w0, w1, w2, u[13], m[11]); + MONT_MAC(t0, t1, t2, u[14], m[10]); + MONT_MAC(v0, v1, v2, u[15], m[9]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[8] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 25 */ + MONT_MUL(s0, s1, s2, a[10], a[15]); + MONT_MUL(c0, c1, c2, a[11], a[14]); + MONT_MAC(s0, s1, s2, a[12], a[13]); + MONT_MAC(t0, t1, t2, u[10], m[15]); + MONT_MUL(v0, v1, v2, u[11], m[14]); + MONT_MUL(w0, w1, w2, u[12], m[13]); + MONT_MAC(t0, t1, t2, u[13], m[12]); + MONT_MAC(v0, v1, v2, u[14], m[11]); + MONT_MAC(w0, w1, w2, u[15], m[10]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[9] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 26 */ + MONT_MUL(s0, s1, s2, a[11], a[15]); + MONT_MUL(c0, c1, c2, a[12], a[14]); + MONT_MAC(t0, t1, t2, a[13], a[13]); + MONT_MUL(v0, v1, v2, u[11], m[15]); + MONT_MUL(w0, w1, w2, u[12], m[14]); + MONT_MAC(t0, t1, t2, u[13], m[13]); + MONT_MAC(v0, v1, v2, u[14], m[12]); + MONT_MAC(w0, w1, w2, u[15], m[11]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[10] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 27 */ + MONT_MUL(s0, s1, s2, a[12], a[15]); + MONT_MUL(c0, c1, c2, a[13], a[14]); + MONT_MAC(t0, t1, t2, u[12], m[15]); + MONT_MUL(v0, v1, v2, u[13], m[14]); + MONT_MUL(w0, w1, w2, u[14], m[13]); + MONT_MAC(t0, t1, t2, u[15], m[12]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[11] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 28 */ + MONT_MUL(s0, s1, s2, a[13], a[15]); + MONT_MAC(t0, t1, t2, a[14], a[14]); + MONT_MUL(v0, v1, v2, u[13], m[15]); + MONT_MUL(w0, w1, w2, u[14], m[14]); + MONT_MAC(t0, t1, t2, u[15], m[13]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[12] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 29 */ + MONT_MUL(s0, s1, s2, a[14], a[15]); + MONT_MAC(t0, t1, t2, u[14], m[15]); + MONT_MUL(v0, v1, v2, u[15], m[14]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[13] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 30 */ + MONT_MAC(t0, t1, t2, a[15], a[15]); + MONT_MUL(v0, v1, v2, u[15], m[15]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + z[14] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + z[15] = t0; + + /* under 2m, with t1 the bit above n limbs: at most one subtraction */ + borrow = sub_n(r, z, m, 16); + take = t1 | (borrow ^ 1); + select_n(r, r, z, take & 1, 16); +} + +/* r = a * b * R^-1 mod m, for a and b under m; t is 2n limbs of scratch */ +static void crypton_mont_mul24(limb_t *r, const limb_t *a, const limb_t *b, + const limb_t *m, limb_t n0, limb_t *t) +{ + limb_t *u = t, *z = t + 24; + limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take; + + /* column 0 */ + MONT_MAC(t0, t1, t2, a[0], b[0]); + u[0] = t0 * n0; + MONT_MAC(t0, t1, t2, u[0], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 1 */ + MONT_MAC(t0, t1, t2, a[0], b[1]); + MONT_MUL(v0, v1, v2, a[1], b[0]); + MONT_MUL(w0, w1, w2, u[0], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[1] = t0 * n0; + MONT_MAC(t0, t1, t2, u[1], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 2 */ + MONT_MAC(t0, t1, t2, a[0], b[2]); + MONT_MUL(v0, v1, v2, a[1], b[1]); + MONT_MUL(w0, w1, w2, a[2], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[2]); + MONT_MAC(v0, v1, v2, u[1], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[2] = t0 * n0; + MONT_MAC(t0, t1, t2, u[2], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 3 */ + MONT_MAC(t0, t1, t2, a[0], b[3]); + MONT_MUL(v0, v1, v2, a[1], b[2]); + MONT_MUL(w0, w1, w2, a[2], b[1]); + MONT_MAC(t0, t1, t2, a[3], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[3]); + MONT_MAC(w0, w1, w2, u[1], m[2]); + MONT_MAC(t0, t1, t2, u[2], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[3] = t0 * n0; + MONT_MAC(t0, t1, t2, u[3], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 4 */ + MONT_MAC(t0, t1, t2, a[0], b[4]); + MONT_MUL(v0, v1, v2, a[1], b[3]); + MONT_MUL(w0, w1, w2, a[2], b[2]); + MONT_MAC(t0, t1, t2, a[3], b[1]); + MONT_MAC(v0, v1, v2, a[4], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[4]); + MONT_MAC(t0, t1, t2, u[1], m[3]); + MONT_MAC(v0, v1, v2, u[2], m[2]); + MONT_MAC(w0, w1, w2, u[3], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[4] = t0 * n0; + MONT_MAC(t0, t1, t2, u[4], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 5 */ + MONT_MAC(t0, t1, t2, a[0], b[5]); + MONT_MUL(v0, v1, v2, a[1], b[4]); + MONT_MUL(w0, w1, w2, a[2], b[3]); + MONT_MAC(t0, t1, t2, a[3], b[2]); + MONT_MAC(v0, v1, v2, a[4], b[1]); + MONT_MAC(w0, w1, w2, a[5], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[5]); + MONT_MAC(v0, v1, v2, u[1], m[4]); + MONT_MAC(w0, w1, w2, u[2], m[3]); + MONT_MAC(t0, t1, t2, u[3], m[2]); + MONT_MAC(v0, v1, v2, u[4], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[5] = t0 * n0; + MONT_MAC(t0, t1, t2, u[5], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 6 */ + MONT_MAC(t0, t1, t2, a[0], b[6]); + MONT_MUL(v0, v1, v2, a[1], b[5]); + MONT_MUL(w0, w1, w2, a[2], b[4]); + MONT_MAC(t0, t1, t2, a[3], b[3]); + MONT_MAC(v0, v1, v2, a[4], b[2]); + MONT_MAC(w0, w1, w2, a[5], b[1]); + MONT_MAC(t0, t1, t2, a[6], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[6]); + MONT_MAC(w0, w1, w2, u[1], m[5]); + MONT_MAC(t0, t1, t2, u[2], m[4]); + MONT_MAC(v0, v1, v2, u[3], m[3]); + MONT_MAC(w0, w1, w2, u[4], m[2]); + MONT_MAC(t0, t1, t2, u[5], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[6] = t0 * n0; + MONT_MAC(t0, t1, t2, u[6], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 7 */ + MONT_MAC(t0, t1, t2, a[0], b[7]); + MONT_MUL(v0, v1, v2, a[1], b[6]); + MONT_MUL(w0, w1, w2, a[2], b[5]); + MONT_MAC(t0, t1, t2, a[3], b[4]); + MONT_MAC(v0, v1, v2, a[4], b[3]); + MONT_MAC(w0, w1, w2, a[5], b[2]); + MONT_MAC(t0, t1, t2, a[6], b[1]); + MONT_MAC(v0, v1, v2, a[7], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[7]); + MONT_MAC(t0, t1, t2, u[1], m[6]); + MONT_MAC(v0, v1, v2, u[2], m[5]); + MONT_MAC(w0, w1, w2, u[3], m[4]); + MONT_MAC(t0, t1, t2, u[4], m[3]); + MONT_MAC(v0, v1, v2, u[5], m[2]); + MONT_MAC(w0, w1, w2, u[6], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[7] = t0 * n0; + MONT_MAC(t0, t1, t2, u[7], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 8 */ + MONT_MAC(t0, t1, t2, a[0], b[8]); + MONT_MUL(v0, v1, v2, a[1], b[7]); + MONT_MUL(w0, w1, w2, a[2], b[6]); + MONT_MAC(t0, t1, t2, a[3], b[5]); + MONT_MAC(v0, v1, v2, a[4], b[4]); + MONT_MAC(w0, w1, w2, a[5], b[3]); + MONT_MAC(t0, t1, t2, a[6], b[2]); + MONT_MAC(v0, v1, v2, a[7], b[1]); + MONT_MAC(w0, w1, w2, a[8], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[8]); + MONT_MAC(v0, v1, v2, u[1], m[7]); + MONT_MAC(w0, w1, w2, u[2], m[6]); + MONT_MAC(t0, t1, t2, u[3], m[5]); + MONT_MAC(v0, v1, v2, u[4], m[4]); + MONT_MAC(w0, w1, w2, u[5], m[3]); + MONT_MAC(t0, t1, t2, u[6], m[2]); + MONT_MAC(v0, v1, v2, u[7], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[8] = t0 * n0; + MONT_MAC(t0, t1, t2, u[8], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 9 */ + MONT_MAC(t0, t1, t2, a[0], b[9]); + MONT_MUL(v0, v1, v2, a[1], b[8]); + MONT_MUL(w0, w1, w2, a[2], b[7]); + MONT_MAC(t0, t1, t2, a[3], b[6]); + MONT_MAC(v0, v1, v2, a[4], b[5]); + MONT_MAC(w0, w1, w2, a[5], b[4]); + MONT_MAC(t0, t1, t2, a[6], b[3]); + MONT_MAC(v0, v1, v2, a[7], b[2]); + MONT_MAC(w0, w1, w2, a[8], b[1]); + MONT_MAC(t0, t1, t2, a[9], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[9]); + MONT_MAC(w0, w1, w2, u[1], m[8]); + MONT_MAC(t0, t1, t2, u[2], m[7]); + MONT_MAC(v0, v1, v2, u[3], m[6]); + MONT_MAC(w0, w1, w2, u[4], m[5]); + MONT_MAC(t0, t1, t2, u[5], m[4]); + MONT_MAC(v0, v1, v2, u[6], m[3]); + MONT_MAC(w0, w1, w2, u[7], m[2]); + MONT_MAC(t0, t1, t2, u[8], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[9] = t0 * n0; + MONT_MAC(t0, t1, t2, u[9], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 10 */ + MONT_MAC(t0, t1, t2, a[0], b[10]); + MONT_MUL(v0, v1, v2, a[1], b[9]); + MONT_MUL(w0, w1, w2, a[2], b[8]); + MONT_MAC(t0, t1, t2, a[3], b[7]); + MONT_MAC(v0, v1, v2, a[4], b[6]); + MONT_MAC(w0, w1, w2, a[5], b[5]); + MONT_MAC(t0, t1, t2, a[6], b[4]); + MONT_MAC(v0, v1, v2, a[7], b[3]); + MONT_MAC(w0, w1, w2, a[8], b[2]); + MONT_MAC(t0, t1, t2, a[9], b[1]); + MONT_MAC(v0, v1, v2, a[10], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[10]); + MONT_MAC(t0, t1, t2, u[1], m[9]); + MONT_MAC(v0, v1, v2, u[2], m[8]); + MONT_MAC(w0, w1, w2, u[3], m[7]); + MONT_MAC(t0, t1, t2, u[4], m[6]); + MONT_MAC(v0, v1, v2, u[5], m[5]); + MONT_MAC(w0, w1, w2, u[6], m[4]); + MONT_MAC(t0, t1, t2, u[7], m[3]); + MONT_MAC(v0, v1, v2, u[8], m[2]); + MONT_MAC(w0, w1, w2, u[9], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[10] = t0 * n0; + MONT_MAC(t0, t1, t2, u[10], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 11 */ + MONT_MAC(t0, t1, t2, a[0], b[11]); + MONT_MUL(v0, v1, v2, a[1], b[10]); + MONT_MUL(w0, w1, w2, a[2], b[9]); + MONT_MAC(t0, t1, t2, a[3], b[8]); + MONT_MAC(v0, v1, v2, a[4], b[7]); + MONT_MAC(w0, w1, w2, a[5], b[6]); + MONT_MAC(t0, t1, t2, a[6], b[5]); + MONT_MAC(v0, v1, v2, a[7], b[4]); + MONT_MAC(w0, w1, w2, a[8], b[3]); + MONT_MAC(t0, t1, t2, a[9], b[2]); + MONT_MAC(v0, v1, v2, a[10], b[1]); + MONT_MAC(w0, w1, w2, a[11], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[11]); + MONT_MAC(v0, v1, v2, u[1], m[10]); + MONT_MAC(w0, w1, w2, u[2], m[9]); + MONT_MAC(t0, t1, t2, u[3], m[8]); + MONT_MAC(v0, v1, v2, u[4], m[7]); + MONT_MAC(w0, w1, w2, u[5], m[6]); + MONT_MAC(t0, t1, t2, u[6], m[5]); + MONT_MAC(v0, v1, v2, u[7], m[4]); + MONT_MAC(w0, w1, w2, u[8], m[3]); + MONT_MAC(t0, t1, t2, u[9], m[2]); + MONT_MAC(v0, v1, v2, u[10], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[11] = t0 * n0; + MONT_MAC(t0, t1, t2, u[11], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 12 */ + MONT_MAC(t0, t1, t2, a[0], b[12]); + MONT_MUL(v0, v1, v2, a[1], b[11]); + MONT_MUL(w0, w1, w2, a[2], b[10]); + MONT_MAC(t0, t1, t2, a[3], b[9]); + MONT_MAC(v0, v1, v2, a[4], b[8]); + MONT_MAC(w0, w1, w2, a[5], b[7]); + MONT_MAC(t0, t1, t2, a[6], b[6]); + MONT_MAC(v0, v1, v2, a[7], b[5]); + MONT_MAC(w0, w1, w2, a[8], b[4]); + MONT_MAC(t0, t1, t2, a[9], b[3]); + MONT_MAC(v0, v1, v2, a[10], b[2]); + MONT_MAC(w0, w1, w2, a[11], b[1]); + MONT_MAC(t0, t1, t2, a[12], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[12]); + MONT_MAC(w0, w1, w2, u[1], m[11]); + MONT_MAC(t0, t1, t2, u[2], m[10]); + MONT_MAC(v0, v1, v2, u[3], m[9]); + MONT_MAC(w0, w1, w2, u[4], m[8]); + MONT_MAC(t0, t1, t2, u[5], m[7]); + MONT_MAC(v0, v1, v2, u[6], m[6]); + MONT_MAC(w0, w1, w2, u[7], m[5]); + MONT_MAC(t0, t1, t2, u[8], m[4]); + MONT_MAC(v0, v1, v2, u[9], m[3]); + MONT_MAC(w0, w1, w2, u[10], m[2]); + MONT_MAC(t0, t1, t2, u[11], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[12] = t0 * n0; + MONT_MAC(t0, t1, t2, u[12], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 13 */ + MONT_MAC(t0, t1, t2, a[0], b[13]); + MONT_MUL(v0, v1, v2, a[1], b[12]); + MONT_MUL(w0, w1, w2, a[2], b[11]); + MONT_MAC(t0, t1, t2, a[3], b[10]); + MONT_MAC(v0, v1, v2, a[4], b[9]); + MONT_MAC(w0, w1, w2, a[5], b[8]); + MONT_MAC(t0, t1, t2, a[6], b[7]); + MONT_MAC(v0, v1, v2, a[7], b[6]); + MONT_MAC(w0, w1, w2, a[8], b[5]); + MONT_MAC(t0, t1, t2, a[9], b[4]); + MONT_MAC(v0, v1, v2, a[10], b[3]); + MONT_MAC(w0, w1, w2, a[11], b[2]); + MONT_MAC(t0, t1, t2, a[12], b[1]); + MONT_MAC(v0, v1, v2, a[13], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[13]); + MONT_MAC(t0, t1, t2, u[1], m[12]); + MONT_MAC(v0, v1, v2, u[2], m[11]); + MONT_MAC(w0, w1, w2, u[3], m[10]); + MONT_MAC(t0, t1, t2, u[4], m[9]); + MONT_MAC(v0, v1, v2, u[5], m[8]); + MONT_MAC(w0, w1, w2, u[6], m[7]); + MONT_MAC(t0, t1, t2, u[7], m[6]); + MONT_MAC(v0, v1, v2, u[8], m[5]); + MONT_MAC(w0, w1, w2, u[9], m[4]); + MONT_MAC(t0, t1, t2, u[10], m[3]); + MONT_MAC(v0, v1, v2, u[11], m[2]); + MONT_MAC(w0, w1, w2, u[12], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[13] = t0 * n0; + MONT_MAC(t0, t1, t2, u[13], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 14 */ + MONT_MAC(t0, t1, t2, a[0], b[14]); + MONT_MUL(v0, v1, v2, a[1], b[13]); + MONT_MUL(w0, w1, w2, a[2], b[12]); + MONT_MAC(t0, t1, t2, a[3], b[11]); + MONT_MAC(v0, v1, v2, a[4], b[10]); + MONT_MAC(w0, w1, w2, a[5], b[9]); + MONT_MAC(t0, t1, t2, a[6], b[8]); + MONT_MAC(v0, v1, v2, a[7], b[7]); + MONT_MAC(w0, w1, w2, a[8], b[6]); + MONT_MAC(t0, t1, t2, a[9], b[5]); + MONT_MAC(v0, v1, v2, a[10], b[4]); + MONT_MAC(w0, w1, w2, a[11], b[3]); + MONT_MAC(t0, t1, t2, a[12], b[2]); + MONT_MAC(v0, v1, v2, a[13], b[1]); + MONT_MAC(w0, w1, w2, a[14], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[14]); + MONT_MAC(v0, v1, v2, u[1], m[13]); + MONT_MAC(w0, w1, w2, u[2], m[12]); + MONT_MAC(t0, t1, t2, u[3], m[11]); + MONT_MAC(v0, v1, v2, u[4], m[10]); + MONT_MAC(w0, w1, w2, u[5], m[9]); + MONT_MAC(t0, t1, t2, u[6], m[8]); + MONT_MAC(v0, v1, v2, u[7], m[7]); + MONT_MAC(w0, w1, w2, u[8], m[6]); + MONT_MAC(t0, t1, t2, u[9], m[5]); + MONT_MAC(v0, v1, v2, u[10], m[4]); + MONT_MAC(w0, w1, w2, u[11], m[3]); + MONT_MAC(t0, t1, t2, u[12], m[2]); + MONT_MAC(v0, v1, v2, u[13], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[14] = t0 * n0; + MONT_MAC(t0, t1, t2, u[14], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 15 */ + MONT_MAC(t0, t1, t2, a[0], b[15]); + MONT_MUL(v0, v1, v2, a[1], b[14]); + MONT_MUL(w0, w1, w2, a[2], b[13]); + MONT_MAC(t0, t1, t2, a[3], b[12]); + MONT_MAC(v0, v1, v2, a[4], b[11]); + MONT_MAC(w0, w1, w2, a[5], b[10]); + MONT_MAC(t0, t1, t2, a[6], b[9]); + MONT_MAC(v0, v1, v2, a[7], b[8]); + MONT_MAC(w0, w1, w2, a[8], b[7]); + MONT_MAC(t0, t1, t2, a[9], b[6]); + MONT_MAC(v0, v1, v2, a[10], b[5]); + MONT_MAC(w0, w1, w2, a[11], b[4]); + MONT_MAC(t0, t1, t2, a[12], b[3]); + MONT_MAC(v0, v1, v2, a[13], b[2]); + MONT_MAC(w0, w1, w2, a[14], b[1]); + MONT_MAC(t0, t1, t2, a[15], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[15]); + MONT_MAC(w0, w1, w2, u[1], m[14]); + MONT_MAC(t0, t1, t2, u[2], m[13]); + MONT_MAC(v0, v1, v2, u[3], m[12]); + MONT_MAC(w0, w1, w2, u[4], m[11]); + MONT_MAC(t0, t1, t2, u[5], m[10]); + MONT_MAC(v0, v1, v2, u[6], m[9]); + MONT_MAC(w0, w1, w2, u[7], m[8]); + MONT_MAC(t0, t1, t2, u[8], m[7]); + MONT_MAC(v0, v1, v2, u[9], m[6]); + MONT_MAC(w0, w1, w2, u[10], m[5]); + MONT_MAC(t0, t1, t2, u[11], m[4]); + MONT_MAC(v0, v1, v2, u[12], m[3]); + MONT_MAC(w0, w1, w2, u[13], m[2]); + MONT_MAC(t0, t1, t2, u[14], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[15] = t0 * n0; + MONT_MAC(t0, t1, t2, u[15], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 16 */ + MONT_MAC(t0, t1, t2, a[0], b[16]); + MONT_MUL(v0, v1, v2, a[1], b[15]); + MONT_MUL(w0, w1, w2, a[2], b[14]); + MONT_MAC(t0, t1, t2, a[3], b[13]); + MONT_MAC(v0, v1, v2, a[4], b[12]); + MONT_MAC(w0, w1, w2, a[5], b[11]); + MONT_MAC(t0, t1, t2, a[6], b[10]); + MONT_MAC(v0, v1, v2, a[7], b[9]); + MONT_MAC(w0, w1, w2, a[8], b[8]); + MONT_MAC(t0, t1, t2, a[9], b[7]); + MONT_MAC(v0, v1, v2, a[10], b[6]); + MONT_MAC(w0, w1, w2, a[11], b[5]); + MONT_MAC(t0, t1, t2, a[12], b[4]); + MONT_MAC(v0, v1, v2, a[13], b[3]); + MONT_MAC(w0, w1, w2, a[14], b[2]); + MONT_MAC(t0, t1, t2, a[15], b[1]); + MONT_MAC(v0, v1, v2, a[16], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[16]); + MONT_MAC(t0, t1, t2, u[1], m[15]); + MONT_MAC(v0, v1, v2, u[2], m[14]); + MONT_MAC(w0, w1, w2, u[3], m[13]); + MONT_MAC(t0, t1, t2, u[4], m[12]); + MONT_MAC(v0, v1, v2, u[5], m[11]); + MONT_MAC(w0, w1, w2, u[6], m[10]); + MONT_MAC(t0, t1, t2, u[7], m[9]); + MONT_MAC(v0, v1, v2, u[8], m[8]); + MONT_MAC(w0, w1, w2, u[9], m[7]); + MONT_MAC(t0, t1, t2, u[10], m[6]); + MONT_MAC(v0, v1, v2, u[11], m[5]); + MONT_MAC(w0, w1, w2, u[12], m[4]); + MONT_MAC(t0, t1, t2, u[13], m[3]); + MONT_MAC(v0, v1, v2, u[14], m[2]); + MONT_MAC(w0, w1, w2, u[15], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[16] = t0 * n0; + MONT_MAC(t0, t1, t2, u[16], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 17 */ + MONT_MAC(t0, t1, t2, a[0], b[17]); + MONT_MUL(v0, v1, v2, a[1], b[16]); + MONT_MUL(w0, w1, w2, a[2], b[15]); + MONT_MAC(t0, t1, t2, a[3], b[14]); + MONT_MAC(v0, v1, v2, a[4], b[13]); + MONT_MAC(w0, w1, w2, a[5], b[12]); + MONT_MAC(t0, t1, t2, a[6], b[11]); + MONT_MAC(v0, v1, v2, a[7], b[10]); + MONT_MAC(w0, w1, w2, a[8], b[9]); + MONT_MAC(t0, t1, t2, a[9], b[8]); + MONT_MAC(v0, v1, v2, a[10], b[7]); + MONT_MAC(w0, w1, w2, a[11], b[6]); + MONT_MAC(t0, t1, t2, a[12], b[5]); + MONT_MAC(v0, v1, v2, a[13], b[4]); + MONT_MAC(w0, w1, w2, a[14], b[3]); + MONT_MAC(t0, t1, t2, a[15], b[2]); + MONT_MAC(v0, v1, v2, a[16], b[1]); + MONT_MAC(w0, w1, w2, a[17], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[17]); + MONT_MAC(v0, v1, v2, u[1], m[16]); + MONT_MAC(w0, w1, w2, u[2], m[15]); + MONT_MAC(t0, t1, t2, u[3], m[14]); + MONT_MAC(v0, v1, v2, u[4], m[13]); + MONT_MAC(w0, w1, w2, u[5], m[12]); + MONT_MAC(t0, t1, t2, u[6], m[11]); + MONT_MAC(v0, v1, v2, u[7], m[10]); + MONT_MAC(w0, w1, w2, u[8], m[9]); + MONT_MAC(t0, t1, t2, u[9], m[8]); + MONT_MAC(v0, v1, v2, u[10], m[7]); + MONT_MAC(w0, w1, w2, u[11], m[6]); + MONT_MAC(t0, t1, t2, u[12], m[5]); + MONT_MAC(v0, v1, v2, u[13], m[4]); + MONT_MAC(w0, w1, w2, u[14], m[3]); + MONT_MAC(t0, t1, t2, u[15], m[2]); + MONT_MAC(v0, v1, v2, u[16], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[17] = t0 * n0; + MONT_MAC(t0, t1, t2, u[17], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 18 */ + MONT_MAC(t0, t1, t2, a[0], b[18]); + MONT_MUL(v0, v1, v2, a[1], b[17]); + MONT_MUL(w0, w1, w2, a[2], b[16]); + MONT_MAC(t0, t1, t2, a[3], b[15]); + MONT_MAC(v0, v1, v2, a[4], b[14]); + MONT_MAC(w0, w1, w2, a[5], b[13]); + MONT_MAC(t0, t1, t2, a[6], b[12]); + MONT_MAC(v0, v1, v2, a[7], b[11]); + MONT_MAC(w0, w1, w2, a[8], b[10]); + MONT_MAC(t0, t1, t2, a[9], b[9]); + MONT_MAC(v0, v1, v2, a[10], b[8]); + MONT_MAC(w0, w1, w2, a[11], b[7]); + MONT_MAC(t0, t1, t2, a[12], b[6]); + MONT_MAC(v0, v1, v2, a[13], b[5]); + MONT_MAC(w0, w1, w2, a[14], b[4]); + MONT_MAC(t0, t1, t2, a[15], b[3]); + MONT_MAC(v0, v1, v2, a[16], b[2]); + MONT_MAC(w0, w1, w2, a[17], b[1]); + MONT_MAC(t0, t1, t2, a[18], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[18]); + MONT_MAC(w0, w1, w2, u[1], m[17]); + MONT_MAC(t0, t1, t2, u[2], m[16]); + MONT_MAC(v0, v1, v2, u[3], m[15]); + MONT_MAC(w0, w1, w2, u[4], m[14]); + MONT_MAC(t0, t1, t2, u[5], m[13]); + MONT_MAC(v0, v1, v2, u[6], m[12]); + MONT_MAC(w0, w1, w2, u[7], m[11]); + MONT_MAC(t0, t1, t2, u[8], m[10]); + MONT_MAC(v0, v1, v2, u[9], m[9]); + MONT_MAC(w0, w1, w2, u[10], m[8]); + MONT_MAC(t0, t1, t2, u[11], m[7]); + MONT_MAC(v0, v1, v2, u[12], m[6]); + MONT_MAC(w0, w1, w2, u[13], m[5]); + MONT_MAC(t0, t1, t2, u[14], m[4]); + MONT_MAC(v0, v1, v2, u[15], m[3]); + MONT_MAC(w0, w1, w2, u[16], m[2]); + MONT_MAC(t0, t1, t2, u[17], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[18] = t0 * n0; + MONT_MAC(t0, t1, t2, u[18], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 19 */ + MONT_MAC(t0, t1, t2, a[0], b[19]); + MONT_MUL(v0, v1, v2, a[1], b[18]); + MONT_MUL(w0, w1, w2, a[2], b[17]); + MONT_MAC(t0, t1, t2, a[3], b[16]); + MONT_MAC(v0, v1, v2, a[4], b[15]); + MONT_MAC(w0, w1, w2, a[5], b[14]); + MONT_MAC(t0, t1, t2, a[6], b[13]); + MONT_MAC(v0, v1, v2, a[7], b[12]); + MONT_MAC(w0, w1, w2, a[8], b[11]); + MONT_MAC(t0, t1, t2, a[9], b[10]); + MONT_MAC(v0, v1, v2, a[10], b[9]); + MONT_MAC(w0, w1, w2, a[11], b[8]); + MONT_MAC(t0, t1, t2, a[12], b[7]); + MONT_MAC(v0, v1, v2, a[13], b[6]); + MONT_MAC(w0, w1, w2, a[14], b[5]); + MONT_MAC(t0, t1, t2, a[15], b[4]); + MONT_MAC(v0, v1, v2, a[16], b[3]); + MONT_MAC(w0, w1, w2, a[17], b[2]); + MONT_MAC(t0, t1, t2, a[18], b[1]); + MONT_MAC(v0, v1, v2, a[19], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[19]); + MONT_MAC(t0, t1, t2, u[1], m[18]); + MONT_MAC(v0, v1, v2, u[2], m[17]); + MONT_MAC(w0, w1, w2, u[3], m[16]); + MONT_MAC(t0, t1, t2, u[4], m[15]); + MONT_MAC(v0, v1, v2, u[5], m[14]); + MONT_MAC(w0, w1, w2, u[6], m[13]); + MONT_MAC(t0, t1, t2, u[7], m[12]); + MONT_MAC(v0, v1, v2, u[8], m[11]); + MONT_MAC(w0, w1, w2, u[9], m[10]); + MONT_MAC(t0, t1, t2, u[10], m[9]); + MONT_MAC(v0, v1, v2, u[11], m[8]); + MONT_MAC(w0, w1, w2, u[12], m[7]); + MONT_MAC(t0, t1, t2, u[13], m[6]); + MONT_MAC(v0, v1, v2, u[14], m[5]); + MONT_MAC(w0, w1, w2, u[15], m[4]); + MONT_MAC(t0, t1, t2, u[16], m[3]); + MONT_MAC(v0, v1, v2, u[17], m[2]); + MONT_MAC(w0, w1, w2, u[18], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[19] = t0 * n0; + MONT_MAC(t0, t1, t2, u[19], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 20 */ + MONT_MAC(t0, t1, t2, a[0], b[20]); + MONT_MUL(v0, v1, v2, a[1], b[19]); + MONT_MUL(w0, w1, w2, a[2], b[18]); + MONT_MAC(t0, t1, t2, a[3], b[17]); + MONT_MAC(v0, v1, v2, a[4], b[16]); + MONT_MAC(w0, w1, w2, a[5], b[15]); + MONT_MAC(t0, t1, t2, a[6], b[14]); + MONT_MAC(v0, v1, v2, a[7], b[13]); + MONT_MAC(w0, w1, w2, a[8], b[12]); + MONT_MAC(t0, t1, t2, a[9], b[11]); + MONT_MAC(v0, v1, v2, a[10], b[10]); + MONT_MAC(w0, w1, w2, a[11], b[9]); + MONT_MAC(t0, t1, t2, a[12], b[8]); + MONT_MAC(v0, v1, v2, a[13], b[7]); + MONT_MAC(w0, w1, w2, a[14], b[6]); + MONT_MAC(t0, t1, t2, a[15], b[5]); + MONT_MAC(v0, v1, v2, a[16], b[4]); + MONT_MAC(w0, w1, w2, a[17], b[3]); + MONT_MAC(t0, t1, t2, a[18], b[2]); + MONT_MAC(v0, v1, v2, a[19], b[1]); + MONT_MAC(w0, w1, w2, a[20], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[20]); + MONT_MAC(v0, v1, v2, u[1], m[19]); + MONT_MAC(w0, w1, w2, u[2], m[18]); + MONT_MAC(t0, t1, t2, u[3], m[17]); + MONT_MAC(v0, v1, v2, u[4], m[16]); + MONT_MAC(w0, w1, w2, u[5], m[15]); + MONT_MAC(t0, t1, t2, u[6], m[14]); + MONT_MAC(v0, v1, v2, u[7], m[13]); + MONT_MAC(w0, w1, w2, u[8], m[12]); + MONT_MAC(t0, t1, t2, u[9], m[11]); + MONT_MAC(v0, v1, v2, u[10], m[10]); + MONT_MAC(w0, w1, w2, u[11], m[9]); + MONT_MAC(t0, t1, t2, u[12], m[8]); + MONT_MAC(v0, v1, v2, u[13], m[7]); + MONT_MAC(w0, w1, w2, u[14], m[6]); + MONT_MAC(t0, t1, t2, u[15], m[5]); + MONT_MAC(v0, v1, v2, u[16], m[4]); + MONT_MAC(w0, w1, w2, u[17], m[3]); + MONT_MAC(t0, t1, t2, u[18], m[2]); + MONT_MAC(v0, v1, v2, u[19], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[20] = t0 * n0; + MONT_MAC(t0, t1, t2, u[20], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 21 */ + MONT_MAC(t0, t1, t2, a[0], b[21]); + MONT_MUL(v0, v1, v2, a[1], b[20]); + MONT_MUL(w0, w1, w2, a[2], b[19]); + MONT_MAC(t0, t1, t2, a[3], b[18]); + MONT_MAC(v0, v1, v2, a[4], b[17]); + MONT_MAC(w0, w1, w2, a[5], b[16]); + MONT_MAC(t0, t1, t2, a[6], b[15]); + MONT_MAC(v0, v1, v2, a[7], b[14]); + MONT_MAC(w0, w1, w2, a[8], b[13]); + MONT_MAC(t0, t1, t2, a[9], b[12]); + MONT_MAC(v0, v1, v2, a[10], b[11]); + MONT_MAC(w0, w1, w2, a[11], b[10]); + MONT_MAC(t0, t1, t2, a[12], b[9]); + MONT_MAC(v0, v1, v2, a[13], b[8]); + MONT_MAC(w0, w1, w2, a[14], b[7]); + MONT_MAC(t0, t1, t2, a[15], b[6]); + MONT_MAC(v0, v1, v2, a[16], b[5]); + MONT_MAC(w0, w1, w2, a[17], b[4]); + MONT_MAC(t0, t1, t2, a[18], b[3]); + MONT_MAC(v0, v1, v2, a[19], b[2]); + MONT_MAC(w0, w1, w2, a[20], b[1]); + MONT_MAC(t0, t1, t2, a[21], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[21]); + MONT_MAC(w0, w1, w2, u[1], m[20]); + MONT_MAC(t0, t1, t2, u[2], m[19]); + MONT_MAC(v0, v1, v2, u[3], m[18]); + MONT_MAC(w0, w1, w2, u[4], m[17]); + MONT_MAC(t0, t1, t2, u[5], m[16]); + MONT_MAC(v0, v1, v2, u[6], m[15]); + MONT_MAC(w0, w1, w2, u[7], m[14]); + MONT_MAC(t0, t1, t2, u[8], m[13]); + MONT_MAC(v0, v1, v2, u[9], m[12]); + MONT_MAC(w0, w1, w2, u[10], m[11]); + MONT_MAC(t0, t1, t2, u[11], m[10]); + MONT_MAC(v0, v1, v2, u[12], m[9]); + MONT_MAC(w0, w1, w2, u[13], m[8]); + MONT_MAC(t0, t1, t2, u[14], m[7]); + MONT_MAC(v0, v1, v2, u[15], m[6]); + MONT_MAC(w0, w1, w2, u[16], m[5]); + MONT_MAC(t0, t1, t2, u[17], m[4]); + MONT_MAC(v0, v1, v2, u[18], m[3]); + MONT_MAC(w0, w1, w2, u[19], m[2]); + MONT_MAC(t0, t1, t2, u[20], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[21] = t0 * n0; + MONT_MAC(t0, t1, t2, u[21], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 22 */ + MONT_MAC(t0, t1, t2, a[0], b[22]); + MONT_MUL(v0, v1, v2, a[1], b[21]); + MONT_MUL(w0, w1, w2, a[2], b[20]); + MONT_MAC(t0, t1, t2, a[3], b[19]); + MONT_MAC(v0, v1, v2, a[4], b[18]); + MONT_MAC(w0, w1, w2, a[5], b[17]); + MONT_MAC(t0, t1, t2, a[6], b[16]); + MONT_MAC(v0, v1, v2, a[7], b[15]); + MONT_MAC(w0, w1, w2, a[8], b[14]); + MONT_MAC(t0, t1, t2, a[9], b[13]); + MONT_MAC(v0, v1, v2, a[10], b[12]); + MONT_MAC(w0, w1, w2, a[11], b[11]); + MONT_MAC(t0, t1, t2, a[12], b[10]); + MONT_MAC(v0, v1, v2, a[13], b[9]); + MONT_MAC(w0, w1, w2, a[14], b[8]); + MONT_MAC(t0, t1, t2, a[15], b[7]); + MONT_MAC(v0, v1, v2, a[16], b[6]); + MONT_MAC(w0, w1, w2, a[17], b[5]); + MONT_MAC(t0, t1, t2, a[18], b[4]); + MONT_MAC(v0, v1, v2, a[19], b[3]); + MONT_MAC(w0, w1, w2, a[20], b[2]); + MONT_MAC(t0, t1, t2, a[21], b[1]); + MONT_MAC(v0, v1, v2, a[22], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[22]); + MONT_MAC(t0, t1, t2, u[1], m[21]); + MONT_MAC(v0, v1, v2, u[2], m[20]); + MONT_MAC(w0, w1, w2, u[3], m[19]); + MONT_MAC(t0, t1, t2, u[4], m[18]); + MONT_MAC(v0, v1, v2, u[5], m[17]); + MONT_MAC(w0, w1, w2, u[6], m[16]); + MONT_MAC(t0, t1, t2, u[7], m[15]); + MONT_MAC(v0, v1, v2, u[8], m[14]); + MONT_MAC(w0, w1, w2, u[9], m[13]); + MONT_MAC(t0, t1, t2, u[10], m[12]); + MONT_MAC(v0, v1, v2, u[11], m[11]); + MONT_MAC(w0, w1, w2, u[12], m[10]); + MONT_MAC(t0, t1, t2, u[13], m[9]); + MONT_MAC(v0, v1, v2, u[14], m[8]); + MONT_MAC(w0, w1, w2, u[15], m[7]); + MONT_MAC(t0, t1, t2, u[16], m[6]); + MONT_MAC(v0, v1, v2, u[17], m[5]); + MONT_MAC(w0, w1, w2, u[18], m[4]); + MONT_MAC(t0, t1, t2, u[19], m[3]); + MONT_MAC(v0, v1, v2, u[20], m[2]); + MONT_MAC(w0, w1, w2, u[21], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[22] = t0 * n0; + MONT_MAC(t0, t1, t2, u[22], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 23 */ + MONT_MAC(t0, t1, t2, a[0], b[23]); + MONT_MUL(v0, v1, v2, a[1], b[22]); + MONT_MUL(w0, w1, w2, a[2], b[21]); + MONT_MAC(t0, t1, t2, a[3], b[20]); + MONT_MAC(v0, v1, v2, a[4], b[19]); + MONT_MAC(w0, w1, w2, a[5], b[18]); + MONT_MAC(t0, t1, t2, a[6], b[17]); + MONT_MAC(v0, v1, v2, a[7], b[16]); + MONT_MAC(w0, w1, w2, a[8], b[15]); + MONT_MAC(t0, t1, t2, a[9], b[14]); + MONT_MAC(v0, v1, v2, a[10], b[13]); + MONT_MAC(w0, w1, w2, a[11], b[12]); + MONT_MAC(t0, t1, t2, a[12], b[11]); + MONT_MAC(v0, v1, v2, a[13], b[10]); + MONT_MAC(w0, w1, w2, a[14], b[9]); + MONT_MAC(t0, t1, t2, a[15], b[8]); + MONT_MAC(v0, v1, v2, a[16], b[7]); + MONT_MAC(w0, w1, w2, a[17], b[6]); + MONT_MAC(t0, t1, t2, a[18], b[5]); + MONT_MAC(v0, v1, v2, a[19], b[4]); + MONT_MAC(w0, w1, w2, a[20], b[3]); + MONT_MAC(t0, t1, t2, a[21], b[2]); + MONT_MAC(v0, v1, v2, a[22], b[1]); + MONT_MAC(w0, w1, w2, a[23], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[23]); + MONT_MAC(v0, v1, v2, u[1], m[22]); + MONT_MAC(w0, w1, w2, u[2], m[21]); + MONT_MAC(t0, t1, t2, u[3], m[20]); + MONT_MAC(v0, v1, v2, u[4], m[19]); + MONT_MAC(w0, w1, w2, u[5], m[18]); + MONT_MAC(t0, t1, t2, u[6], m[17]); + MONT_MAC(v0, v1, v2, u[7], m[16]); + MONT_MAC(w0, w1, w2, u[8], m[15]); + MONT_MAC(t0, t1, t2, u[9], m[14]); + MONT_MAC(v0, v1, v2, u[10], m[13]); + MONT_MAC(w0, w1, w2, u[11], m[12]); + MONT_MAC(t0, t1, t2, u[12], m[11]); + MONT_MAC(v0, v1, v2, u[13], m[10]); + MONT_MAC(w0, w1, w2, u[14], m[9]); + MONT_MAC(t0, t1, t2, u[15], m[8]); + MONT_MAC(v0, v1, v2, u[16], m[7]); + MONT_MAC(w0, w1, w2, u[17], m[6]); + MONT_MAC(t0, t1, t2, u[18], m[5]); + MONT_MAC(v0, v1, v2, u[19], m[4]); + MONT_MAC(w0, w1, w2, u[20], m[3]); + MONT_MAC(t0, t1, t2, u[21], m[2]); + MONT_MAC(v0, v1, v2, u[22], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[23] = t0 * n0; + MONT_MAC(t0, t1, t2, u[23], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 24 */ + MONT_MAC(t0, t1, t2, a[1], b[23]); + MONT_MUL(v0, v1, v2, a[2], b[22]); + MONT_MUL(w0, w1, w2, a[3], b[21]); + MONT_MAC(t0, t1, t2, a[4], b[20]); + MONT_MAC(v0, v1, v2, a[5], b[19]); + MONT_MAC(w0, w1, w2, a[6], b[18]); + MONT_MAC(t0, t1, t2, a[7], b[17]); + MONT_MAC(v0, v1, v2, a[8], b[16]); + MONT_MAC(w0, w1, w2, a[9], b[15]); + MONT_MAC(t0, t1, t2, a[10], b[14]); + MONT_MAC(v0, v1, v2, a[11], b[13]); + MONT_MAC(w0, w1, w2, a[12], b[12]); + MONT_MAC(t0, t1, t2, a[13], b[11]); + MONT_MAC(v0, v1, v2, a[14], b[10]); + MONT_MAC(w0, w1, w2, a[15], b[9]); + MONT_MAC(t0, t1, t2, a[16], b[8]); + MONT_MAC(v0, v1, v2, a[17], b[7]); + MONT_MAC(w0, w1, w2, a[18], b[6]); + MONT_MAC(t0, t1, t2, a[19], b[5]); + MONT_MAC(v0, v1, v2, a[20], b[4]); + MONT_MAC(w0, w1, w2, a[21], b[3]); + MONT_MAC(t0, t1, t2, a[22], b[2]); + MONT_MAC(v0, v1, v2, a[23], b[1]); + MONT_MAC(w0, w1, w2, u[1], m[23]); + MONT_MAC(t0, t1, t2, u[2], m[22]); + MONT_MAC(v0, v1, v2, u[3], m[21]); + MONT_MAC(w0, w1, w2, u[4], m[20]); + MONT_MAC(t0, t1, t2, u[5], m[19]); + MONT_MAC(v0, v1, v2, u[6], m[18]); + MONT_MAC(w0, w1, w2, u[7], m[17]); + MONT_MAC(t0, t1, t2, u[8], m[16]); + MONT_MAC(v0, v1, v2, u[9], m[15]); + MONT_MAC(w0, w1, w2, u[10], m[14]); + MONT_MAC(t0, t1, t2, u[11], m[13]); + MONT_MAC(v0, v1, v2, u[12], m[12]); + MONT_MAC(w0, w1, w2, u[13], m[11]); + MONT_MAC(t0, t1, t2, u[14], m[10]); + MONT_MAC(v0, v1, v2, u[15], m[9]); + MONT_MAC(w0, w1, w2, u[16], m[8]); + MONT_MAC(t0, t1, t2, u[17], m[7]); + MONT_MAC(v0, v1, v2, u[18], m[6]); + MONT_MAC(w0, w1, w2, u[19], m[5]); + MONT_MAC(t0, t1, t2, u[20], m[4]); + MONT_MAC(v0, v1, v2, u[21], m[3]); + MONT_MAC(w0, w1, w2, u[22], m[2]); + MONT_MAC(t0, t1, t2, u[23], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[0] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 25 */ + MONT_MAC(t0, t1, t2, a[2], b[23]); + MONT_MUL(v0, v1, v2, a[3], b[22]); + MONT_MUL(w0, w1, w2, a[4], b[21]); + MONT_MAC(t0, t1, t2, a[5], b[20]); + MONT_MAC(v0, v1, v2, a[6], b[19]); + MONT_MAC(w0, w1, w2, a[7], b[18]); + MONT_MAC(t0, t1, t2, a[8], b[17]); + MONT_MAC(v0, v1, v2, a[9], b[16]); + MONT_MAC(w0, w1, w2, a[10], b[15]); + MONT_MAC(t0, t1, t2, a[11], b[14]); + MONT_MAC(v0, v1, v2, a[12], b[13]); + MONT_MAC(w0, w1, w2, a[13], b[12]); + MONT_MAC(t0, t1, t2, a[14], b[11]); + MONT_MAC(v0, v1, v2, a[15], b[10]); + MONT_MAC(w0, w1, w2, a[16], b[9]); + MONT_MAC(t0, t1, t2, a[17], b[8]); + MONT_MAC(v0, v1, v2, a[18], b[7]); + MONT_MAC(w0, w1, w2, a[19], b[6]); + MONT_MAC(t0, t1, t2, a[20], b[5]); + MONT_MAC(v0, v1, v2, a[21], b[4]); + MONT_MAC(w0, w1, w2, a[22], b[3]); + MONT_MAC(t0, t1, t2, a[23], b[2]); + MONT_MAC(v0, v1, v2, u[2], m[23]); + MONT_MAC(w0, w1, w2, u[3], m[22]); + MONT_MAC(t0, t1, t2, u[4], m[21]); + MONT_MAC(v0, v1, v2, u[5], m[20]); + MONT_MAC(w0, w1, w2, u[6], m[19]); + MONT_MAC(t0, t1, t2, u[7], m[18]); + MONT_MAC(v0, v1, v2, u[8], m[17]); + MONT_MAC(w0, w1, w2, u[9], m[16]); + MONT_MAC(t0, t1, t2, u[10], m[15]); + MONT_MAC(v0, v1, v2, u[11], m[14]); + MONT_MAC(w0, w1, w2, u[12], m[13]); + MONT_MAC(t0, t1, t2, u[13], m[12]); + MONT_MAC(v0, v1, v2, u[14], m[11]); + MONT_MAC(w0, w1, w2, u[15], m[10]); + MONT_MAC(t0, t1, t2, u[16], m[9]); + MONT_MAC(v0, v1, v2, u[17], m[8]); + MONT_MAC(w0, w1, w2, u[18], m[7]); + MONT_MAC(t0, t1, t2, u[19], m[6]); + MONT_MAC(v0, v1, v2, u[20], m[5]); + MONT_MAC(w0, w1, w2, u[21], m[4]); + MONT_MAC(t0, t1, t2, u[22], m[3]); + MONT_MAC(v0, v1, v2, u[23], m[2]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[1] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 26 */ + MONT_MAC(t0, t1, t2, a[3], b[23]); + MONT_MUL(v0, v1, v2, a[4], b[22]); + MONT_MUL(w0, w1, w2, a[5], b[21]); + MONT_MAC(t0, t1, t2, a[6], b[20]); + MONT_MAC(v0, v1, v2, a[7], b[19]); + MONT_MAC(w0, w1, w2, a[8], b[18]); + MONT_MAC(t0, t1, t2, a[9], b[17]); + MONT_MAC(v0, v1, v2, a[10], b[16]); + MONT_MAC(w0, w1, w2, a[11], b[15]); + MONT_MAC(t0, t1, t2, a[12], b[14]); + MONT_MAC(v0, v1, v2, a[13], b[13]); + MONT_MAC(w0, w1, w2, a[14], b[12]); + MONT_MAC(t0, t1, t2, a[15], b[11]); + MONT_MAC(v0, v1, v2, a[16], b[10]); + MONT_MAC(w0, w1, w2, a[17], b[9]); + MONT_MAC(t0, t1, t2, a[18], b[8]); + MONT_MAC(v0, v1, v2, a[19], b[7]); + MONT_MAC(w0, w1, w2, a[20], b[6]); + MONT_MAC(t0, t1, t2, a[21], b[5]); + MONT_MAC(v0, v1, v2, a[22], b[4]); + MONT_MAC(w0, w1, w2, a[23], b[3]); + MONT_MAC(t0, t1, t2, u[3], m[23]); + MONT_MAC(v0, v1, v2, u[4], m[22]); + MONT_MAC(w0, w1, w2, u[5], m[21]); + MONT_MAC(t0, t1, t2, u[6], m[20]); + MONT_MAC(v0, v1, v2, u[7], m[19]); + MONT_MAC(w0, w1, w2, u[8], m[18]); + MONT_MAC(t0, t1, t2, u[9], m[17]); + MONT_MAC(v0, v1, v2, u[10], m[16]); + MONT_MAC(w0, w1, w2, u[11], m[15]); + MONT_MAC(t0, t1, t2, u[12], m[14]); + MONT_MAC(v0, v1, v2, u[13], m[13]); + MONT_MAC(w0, w1, w2, u[14], m[12]); + MONT_MAC(t0, t1, t2, u[15], m[11]); + MONT_MAC(v0, v1, v2, u[16], m[10]); + MONT_MAC(w0, w1, w2, u[17], m[9]); + MONT_MAC(t0, t1, t2, u[18], m[8]); + MONT_MAC(v0, v1, v2, u[19], m[7]); + MONT_MAC(w0, w1, w2, u[20], m[6]); + MONT_MAC(t0, t1, t2, u[21], m[5]); + MONT_MAC(v0, v1, v2, u[22], m[4]); + MONT_MAC(w0, w1, w2, u[23], m[3]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[2] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 27 */ + MONT_MAC(t0, t1, t2, a[4], b[23]); + MONT_MUL(v0, v1, v2, a[5], b[22]); + MONT_MUL(w0, w1, w2, a[6], b[21]); + MONT_MAC(t0, t1, t2, a[7], b[20]); + MONT_MAC(v0, v1, v2, a[8], b[19]); + MONT_MAC(w0, w1, w2, a[9], b[18]); + MONT_MAC(t0, t1, t2, a[10], b[17]); + MONT_MAC(v0, v1, v2, a[11], b[16]); + MONT_MAC(w0, w1, w2, a[12], b[15]); + MONT_MAC(t0, t1, t2, a[13], b[14]); + MONT_MAC(v0, v1, v2, a[14], b[13]); + MONT_MAC(w0, w1, w2, a[15], b[12]); + MONT_MAC(t0, t1, t2, a[16], b[11]); + MONT_MAC(v0, v1, v2, a[17], b[10]); + MONT_MAC(w0, w1, w2, a[18], b[9]); + MONT_MAC(t0, t1, t2, a[19], b[8]); + MONT_MAC(v0, v1, v2, a[20], b[7]); + MONT_MAC(w0, w1, w2, a[21], b[6]); + MONT_MAC(t0, t1, t2, a[22], b[5]); + MONT_MAC(v0, v1, v2, a[23], b[4]); + MONT_MAC(w0, w1, w2, u[4], m[23]); + MONT_MAC(t0, t1, t2, u[5], m[22]); + MONT_MAC(v0, v1, v2, u[6], m[21]); + MONT_MAC(w0, w1, w2, u[7], m[20]); + MONT_MAC(t0, t1, t2, u[8], m[19]); + MONT_MAC(v0, v1, v2, u[9], m[18]); + MONT_MAC(w0, w1, w2, u[10], m[17]); + MONT_MAC(t0, t1, t2, u[11], m[16]); + MONT_MAC(v0, v1, v2, u[12], m[15]); + MONT_MAC(w0, w1, w2, u[13], m[14]); + MONT_MAC(t0, t1, t2, u[14], m[13]); + MONT_MAC(v0, v1, v2, u[15], m[12]); + MONT_MAC(w0, w1, w2, u[16], m[11]); + MONT_MAC(t0, t1, t2, u[17], m[10]); + MONT_MAC(v0, v1, v2, u[18], m[9]); + MONT_MAC(w0, w1, w2, u[19], m[8]); + MONT_MAC(t0, t1, t2, u[20], m[7]); + MONT_MAC(v0, v1, v2, u[21], m[6]); + MONT_MAC(w0, w1, w2, u[22], m[5]); + MONT_MAC(t0, t1, t2, u[23], m[4]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[3] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 28 */ + MONT_MAC(t0, t1, t2, a[5], b[23]); + MONT_MUL(v0, v1, v2, a[6], b[22]); + MONT_MUL(w0, w1, w2, a[7], b[21]); + MONT_MAC(t0, t1, t2, a[8], b[20]); + MONT_MAC(v0, v1, v2, a[9], b[19]); + MONT_MAC(w0, w1, w2, a[10], b[18]); + MONT_MAC(t0, t1, t2, a[11], b[17]); + MONT_MAC(v0, v1, v2, a[12], b[16]); + MONT_MAC(w0, w1, w2, a[13], b[15]); + MONT_MAC(t0, t1, t2, a[14], b[14]); + MONT_MAC(v0, v1, v2, a[15], b[13]); + MONT_MAC(w0, w1, w2, a[16], b[12]); + MONT_MAC(t0, t1, t2, a[17], b[11]); + MONT_MAC(v0, v1, v2, a[18], b[10]); + MONT_MAC(w0, w1, w2, a[19], b[9]); + MONT_MAC(t0, t1, t2, a[20], b[8]); + MONT_MAC(v0, v1, v2, a[21], b[7]); + MONT_MAC(w0, w1, w2, a[22], b[6]); + MONT_MAC(t0, t1, t2, a[23], b[5]); + MONT_MAC(v0, v1, v2, u[5], m[23]); + MONT_MAC(w0, w1, w2, u[6], m[22]); + MONT_MAC(t0, t1, t2, u[7], m[21]); + MONT_MAC(v0, v1, v2, u[8], m[20]); + MONT_MAC(w0, w1, w2, u[9], m[19]); + MONT_MAC(t0, t1, t2, u[10], m[18]); + MONT_MAC(v0, v1, v2, u[11], m[17]); + MONT_MAC(w0, w1, w2, u[12], m[16]); + MONT_MAC(t0, t1, t2, u[13], m[15]); + MONT_MAC(v0, v1, v2, u[14], m[14]); + MONT_MAC(w0, w1, w2, u[15], m[13]); + MONT_MAC(t0, t1, t2, u[16], m[12]); + MONT_MAC(v0, v1, v2, u[17], m[11]); + MONT_MAC(w0, w1, w2, u[18], m[10]); + MONT_MAC(t0, t1, t2, u[19], m[9]); + MONT_MAC(v0, v1, v2, u[20], m[8]); + MONT_MAC(w0, w1, w2, u[21], m[7]); + MONT_MAC(t0, t1, t2, u[22], m[6]); + MONT_MAC(v0, v1, v2, u[23], m[5]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[4] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 29 */ + MONT_MAC(t0, t1, t2, a[6], b[23]); + MONT_MUL(v0, v1, v2, a[7], b[22]); + MONT_MUL(w0, w1, w2, a[8], b[21]); + MONT_MAC(t0, t1, t2, a[9], b[20]); + MONT_MAC(v0, v1, v2, a[10], b[19]); + MONT_MAC(w0, w1, w2, a[11], b[18]); + MONT_MAC(t0, t1, t2, a[12], b[17]); + MONT_MAC(v0, v1, v2, a[13], b[16]); + MONT_MAC(w0, w1, w2, a[14], b[15]); + MONT_MAC(t0, t1, t2, a[15], b[14]); + MONT_MAC(v0, v1, v2, a[16], b[13]); + MONT_MAC(w0, w1, w2, a[17], b[12]); + MONT_MAC(t0, t1, t2, a[18], b[11]); + MONT_MAC(v0, v1, v2, a[19], b[10]); + MONT_MAC(w0, w1, w2, a[20], b[9]); + MONT_MAC(t0, t1, t2, a[21], b[8]); + MONT_MAC(v0, v1, v2, a[22], b[7]); + MONT_MAC(w0, w1, w2, a[23], b[6]); + MONT_MAC(t0, t1, t2, u[6], m[23]); + MONT_MAC(v0, v1, v2, u[7], m[22]); + MONT_MAC(w0, w1, w2, u[8], m[21]); + MONT_MAC(t0, t1, t2, u[9], m[20]); + MONT_MAC(v0, v1, v2, u[10], m[19]); + MONT_MAC(w0, w1, w2, u[11], m[18]); + MONT_MAC(t0, t1, t2, u[12], m[17]); + MONT_MAC(v0, v1, v2, u[13], m[16]); + MONT_MAC(w0, w1, w2, u[14], m[15]); + MONT_MAC(t0, t1, t2, u[15], m[14]); + MONT_MAC(v0, v1, v2, u[16], m[13]); + MONT_MAC(w0, w1, w2, u[17], m[12]); + MONT_MAC(t0, t1, t2, u[18], m[11]); + MONT_MAC(v0, v1, v2, u[19], m[10]); + MONT_MAC(w0, w1, w2, u[20], m[9]); + MONT_MAC(t0, t1, t2, u[21], m[8]); + MONT_MAC(v0, v1, v2, u[22], m[7]); + MONT_MAC(w0, w1, w2, u[23], m[6]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[5] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 30 */ + MONT_MAC(t0, t1, t2, a[7], b[23]); + MONT_MUL(v0, v1, v2, a[8], b[22]); + MONT_MUL(w0, w1, w2, a[9], b[21]); + MONT_MAC(t0, t1, t2, a[10], b[20]); + MONT_MAC(v0, v1, v2, a[11], b[19]); + MONT_MAC(w0, w1, w2, a[12], b[18]); + MONT_MAC(t0, t1, t2, a[13], b[17]); + MONT_MAC(v0, v1, v2, a[14], b[16]); + MONT_MAC(w0, w1, w2, a[15], b[15]); + MONT_MAC(t0, t1, t2, a[16], b[14]); + MONT_MAC(v0, v1, v2, a[17], b[13]); + MONT_MAC(w0, w1, w2, a[18], b[12]); + MONT_MAC(t0, t1, t2, a[19], b[11]); + MONT_MAC(v0, v1, v2, a[20], b[10]); + MONT_MAC(w0, w1, w2, a[21], b[9]); + MONT_MAC(t0, t1, t2, a[22], b[8]); + MONT_MAC(v0, v1, v2, a[23], b[7]); + MONT_MAC(w0, w1, w2, u[7], m[23]); + MONT_MAC(t0, t1, t2, u[8], m[22]); + MONT_MAC(v0, v1, v2, u[9], m[21]); + MONT_MAC(w0, w1, w2, u[10], m[20]); + MONT_MAC(t0, t1, t2, u[11], m[19]); + MONT_MAC(v0, v1, v2, u[12], m[18]); + MONT_MAC(w0, w1, w2, u[13], m[17]); + MONT_MAC(t0, t1, t2, u[14], m[16]); + MONT_MAC(v0, v1, v2, u[15], m[15]); + MONT_MAC(w0, w1, w2, u[16], m[14]); + MONT_MAC(t0, t1, t2, u[17], m[13]); + MONT_MAC(v0, v1, v2, u[18], m[12]); + MONT_MAC(w0, w1, w2, u[19], m[11]); + MONT_MAC(t0, t1, t2, u[20], m[10]); + MONT_MAC(v0, v1, v2, u[21], m[9]); + MONT_MAC(w0, w1, w2, u[22], m[8]); + MONT_MAC(t0, t1, t2, u[23], m[7]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[6] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 31 */ + MONT_MAC(t0, t1, t2, a[8], b[23]); + MONT_MUL(v0, v1, v2, a[9], b[22]); + MONT_MUL(w0, w1, w2, a[10], b[21]); + MONT_MAC(t0, t1, t2, a[11], b[20]); + MONT_MAC(v0, v1, v2, a[12], b[19]); + MONT_MAC(w0, w1, w2, a[13], b[18]); + MONT_MAC(t0, t1, t2, a[14], b[17]); + MONT_MAC(v0, v1, v2, a[15], b[16]); + MONT_MAC(w0, w1, w2, a[16], b[15]); + MONT_MAC(t0, t1, t2, a[17], b[14]); + MONT_MAC(v0, v1, v2, a[18], b[13]); + MONT_MAC(w0, w1, w2, a[19], b[12]); + MONT_MAC(t0, t1, t2, a[20], b[11]); + MONT_MAC(v0, v1, v2, a[21], b[10]); + MONT_MAC(w0, w1, w2, a[22], b[9]); + MONT_MAC(t0, t1, t2, a[23], b[8]); + MONT_MAC(v0, v1, v2, u[8], m[23]); + MONT_MAC(w0, w1, w2, u[9], m[22]); + MONT_MAC(t0, t1, t2, u[10], m[21]); + MONT_MAC(v0, v1, v2, u[11], m[20]); + MONT_MAC(w0, w1, w2, u[12], m[19]); + MONT_MAC(t0, t1, t2, u[13], m[18]); + MONT_MAC(v0, v1, v2, u[14], m[17]); + MONT_MAC(w0, w1, w2, u[15], m[16]); + MONT_MAC(t0, t1, t2, u[16], m[15]); + MONT_MAC(v0, v1, v2, u[17], m[14]); + MONT_MAC(w0, w1, w2, u[18], m[13]); + MONT_MAC(t0, t1, t2, u[19], m[12]); + MONT_MAC(v0, v1, v2, u[20], m[11]); + MONT_MAC(w0, w1, w2, u[21], m[10]); + MONT_MAC(t0, t1, t2, u[22], m[9]); + MONT_MAC(v0, v1, v2, u[23], m[8]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[7] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 32 */ + MONT_MAC(t0, t1, t2, a[9], b[23]); + MONT_MUL(v0, v1, v2, a[10], b[22]); + MONT_MUL(w0, w1, w2, a[11], b[21]); + MONT_MAC(t0, t1, t2, a[12], b[20]); + MONT_MAC(v0, v1, v2, a[13], b[19]); + MONT_MAC(w0, w1, w2, a[14], b[18]); + MONT_MAC(t0, t1, t2, a[15], b[17]); + MONT_MAC(v0, v1, v2, a[16], b[16]); + MONT_MAC(w0, w1, w2, a[17], b[15]); + MONT_MAC(t0, t1, t2, a[18], b[14]); + MONT_MAC(v0, v1, v2, a[19], b[13]); + MONT_MAC(w0, w1, w2, a[20], b[12]); + MONT_MAC(t0, t1, t2, a[21], b[11]); + MONT_MAC(v0, v1, v2, a[22], b[10]); + MONT_MAC(w0, w1, w2, a[23], b[9]); + MONT_MAC(t0, t1, t2, u[9], m[23]); + MONT_MAC(v0, v1, v2, u[10], m[22]); + MONT_MAC(w0, w1, w2, u[11], m[21]); + MONT_MAC(t0, t1, t2, u[12], m[20]); + MONT_MAC(v0, v1, v2, u[13], m[19]); + MONT_MAC(w0, w1, w2, u[14], m[18]); + MONT_MAC(t0, t1, t2, u[15], m[17]); + MONT_MAC(v0, v1, v2, u[16], m[16]); + MONT_MAC(w0, w1, w2, u[17], m[15]); + MONT_MAC(t0, t1, t2, u[18], m[14]); + MONT_MAC(v0, v1, v2, u[19], m[13]); + MONT_MAC(w0, w1, w2, u[20], m[12]); + MONT_MAC(t0, t1, t2, u[21], m[11]); + MONT_MAC(v0, v1, v2, u[22], m[10]); + MONT_MAC(w0, w1, w2, u[23], m[9]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[8] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 33 */ + MONT_MAC(t0, t1, t2, a[10], b[23]); + MONT_MUL(v0, v1, v2, a[11], b[22]); + MONT_MUL(w0, w1, w2, a[12], b[21]); + MONT_MAC(t0, t1, t2, a[13], b[20]); + MONT_MAC(v0, v1, v2, a[14], b[19]); + MONT_MAC(w0, w1, w2, a[15], b[18]); + MONT_MAC(t0, t1, t2, a[16], b[17]); + MONT_MAC(v0, v1, v2, a[17], b[16]); + MONT_MAC(w0, w1, w2, a[18], b[15]); + MONT_MAC(t0, t1, t2, a[19], b[14]); + MONT_MAC(v0, v1, v2, a[20], b[13]); + MONT_MAC(w0, w1, w2, a[21], b[12]); + MONT_MAC(t0, t1, t2, a[22], b[11]); + MONT_MAC(v0, v1, v2, a[23], b[10]); + MONT_MAC(w0, w1, w2, u[10], m[23]); + MONT_MAC(t0, t1, t2, u[11], m[22]); + MONT_MAC(v0, v1, v2, u[12], m[21]); + MONT_MAC(w0, w1, w2, u[13], m[20]); + MONT_MAC(t0, t1, t2, u[14], m[19]); + MONT_MAC(v0, v1, v2, u[15], m[18]); + MONT_MAC(w0, w1, w2, u[16], m[17]); + MONT_MAC(t0, t1, t2, u[17], m[16]); + MONT_MAC(v0, v1, v2, u[18], m[15]); + MONT_MAC(w0, w1, w2, u[19], m[14]); + MONT_MAC(t0, t1, t2, u[20], m[13]); + MONT_MAC(v0, v1, v2, u[21], m[12]); + MONT_MAC(w0, w1, w2, u[22], m[11]); + MONT_MAC(t0, t1, t2, u[23], m[10]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[9] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 34 */ + MONT_MAC(t0, t1, t2, a[11], b[23]); + MONT_MUL(v0, v1, v2, a[12], b[22]); + MONT_MUL(w0, w1, w2, a[13], b[21]); + MONT_MAC(t0, t1, t2, a[14], b[20]); + MONT_MAC(v0, v1, v2, a[15], b[19]); + MONT_MAC(w0, w1, w2, a[16], b[18]); + MONT_MAC(t0, t1, t2, a[17], b[17]); + MONT_MAC(v0, v1, v2, a[18], b[16]); + MONT_MAC(w0, w1, w2, a[19], b[15]); + MONT_MAC(t0, t1, t2, a[20], b[14]); + MONT_MAC(v0, v1, v2, a[21], b[13]); + MONT_MAC(w0, w1, w2, a[22], b[12]); + MONT_MAC(t0, t1, t2, a[23], b[11]); + MONT_MAC(v0, v1, v2, u[11], m[23]); + MONT_MAC(w0, w1, w2, u[12], m[22]); + MONT_MAC(t0, t1, t2, u[13], m[21]); + MONT_MAC(v0, v1, v2, u[14], m[20]); + MONT_MAC(w0, w1, w2, u[15], m[19]); + MONT_MAC(t0, t1, t2, u[16], m[18]); + MONT_MAC(v0, v1, v2, u[17], m[17]); + MONT_MAC(w0, w1, w2, u[18], m[16]); + MONT_MAC(t0, t1, t2, u[19], m[15]); + MONT_MAC(v0, v1, v2, u[20], m[14]); + MONT_MAC(w0, w1, w2, u[21], m[13]); + MONT_MAC(t0, t1, t2, u[22], m[12]); + MONT_MAC(v0, v1, v2, u[23], m[11]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[10] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 35 */ + MONT_MAC(t0, t1, t2, a[12], b[23]); + MONT_MUL(v0, v1, v2, a[13], b[22]); + MONT_MUL(w0, w1, w2, a[14], b[21]); + MONT_MAC(t0, t1, t2, a[15], b[20]); + MONT_MAC(v0, v1, v2, a[16], b[19]); + MONT_MAC(w0, w1, w2, a[17], b[18]); + MONT_MAC(t0, t1, t2, a[18], b[17]); + MONT_MAC(v0, v1, v2, a[19], b[16]); + MONT_MAC(w0, w1, w2, a[20], b[15]); + MONT_MAC(t0, t1, t2, a[21], b[14]); + MONT_MAC(v0, v1, v2, a[22], b[13]); + MONT_MAC(w0, w1, w2, a[23], b[12]); + MONT_MAC(t0, t1, t2, u[12], m[23]); + MONT_MAC(v0, v1, v2, u[13], m[22]); + MONT_MAC(w0, w1, w2, u[14], m[21]); + MONT_MAC(t0, t1, t2, u[15], m[20]); + MONT_MAC(v0, v1, v2, u[16], m[19]); + MONT_MAC(w0, w1, w2, u[17], m[18]); + MONT_MAC(t0, t1, t2, u[18], m[17]); + MONT_MAC(v0, v1, v2, u[19], m[16]); + MONT_MAC(w0, w1, w2, u[20], m[15]); + MONT_MAC(t0, t1, t2, u[21], m[14]); + MONT_MAC(v0, v1, v2, u[22], m[13]); + MONT_MAC(w0, w1, w2, u[23], m[12]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[11] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 36 */ + MONT_MAC(t0, t1, t2, a[13], b[23]); + MONT_MUL(v0, v1, v2, a[14], b[22]); + MONT_MUL(w0, w1, w2, a[15], b[21]); + MONT_MAC(t0, t1, t2, a[16], b[20]); + MONT_MAC(v0, v1, v2, a[17], b[19]); + MONT_MAC(w0, w1, w2, a[18], b[18]); + MONT_MAC(t0, t1, t2, a[19], b[17]); + MONT_MAC(v0, v1, v2, a[20], b[16]); + MONT_MAC(w0, w1, w2, a[21], b[15]); + MONT_MAC(t0, t1, t2, a[22], b[14]); + MONT_MAC(v0, v1, v2, a[23], b[13]); + MONT_MAC(w0, w1, w2, u[13], m[23]); + MONT_MAC(t0, t1, t2, u[14], m[22]); + MONT_MAC(v0, v1, v2, u[15], m[21]); + MONT_MAC(w0, w1, w2, u[16], m[20]); + MONT_MAC(t0, t1, t2, u[17], m[19]); + MONT_MAC(v0, v1, v2, u[18], m[18]); + MONT_MAC(w0, w1, w2, u[19], m[17]); + MONT_MAC(t0, t1, t2, u[20], m[16]); + MONT_MAC(v0, v1, v2, u[21], m[15]); + MONT_MAC(w0, w1, w2, u[22], m[14]); + MONT_MAC(t0, t1, t2, u[23], m[13]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[12] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 37 */ + MONT_MAC(t0, t1, t2, a[14], b[23]); + MONT_MUL(v0, v1, v2, a[15], b[22]); + MONT_MUL(w0, w1, w2, a[16], b[21]); + MONT_MAC(t0, t1, t2, a[17], b[20]); + MONT_MAC(v0, v1, v2, a[18], b[19]); + MONT_MAC(w0, w1, w2, a[19], b[18]); + MONT_MAC(t0, t1, t2, a[20], b[17]); + MONT_MAC(v0, v1, v2, a[21], b[16]); + MONT_MAC(w0, w1, w2, a[22], b[15]); + MONT_MAC(t0, t1, t2, a[23], b[14]); + MONT_MAC(v0, v1, v2, u[14], m[23]); + MONT_MAC(w0, w1, w2, u[15], m[22]); + MONT_MAC(t0, t1, t2, u[16], m[21]); + MONT_MAC(v0, v1, v2, u[17], m[20]); + MONT_MAC(w0, w1, w2, u[18], m[19]); + MONT_MAC(t0, t1, t2, u[19], m[18]); + MONT_MAC(v0, v1, v2, u[20], m[17]); + MONT_MAC(w0, w1, w2, u[21], m[16]); + MONT_MAC(t0, t1, t2, u[22], m[15]); + MONT_MAC(v0, v1, v2, u[23], m[14]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[13] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 38 */ + MONT_MAC(t0, t1, t2, a[15], b[23]); + MONT_MUL(v0, v1, v2, a[16], b[22]); + MONT_MUL(w0, w1, w2, a[17], b[21]); + MONT_MAC(t0, t1, t2, a[18], b[20]); + MONT_MAC(v0, v1, v2, a[19], b[19]); + MONT_MAC(w0, w1, w2, a[20], b[18]); + MONT_MAC(t0, t1, t2, a[21], b[17]); + MONT_MAC(v0, v1, v2, a[22], b[16]); + MONT_MAC(w0, w1, w2, a[23], b[15]); + MONT_MAC(t0, t1, t2, u[15], m[23]); + MONT_MAC(v0, v1, v2, u[16], m[22]); + MONT_MAC(w0, w1, w2, u[17], m[21]); + MONT_MAC(t0, t1, t2, u[18], m[20]); + MONT_MAC(v0, v1, v2, u[19], m[19]); + MONT_MAC(w0, w1, w2, u[20], m[18]); + MONT_MAC(t0, t1, t2, u[21], m[17]); + MONT_MAC(v0, v1, v2, u[22], m[16]); + MONT_MAC(w0, w1, w2, u[23], m[15]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[14] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 39 */ + MONT_MAC(t0, t1, t2, a[16], b[23]); + MONT_MUL(v0, v1, v2, a[17], b[22]); + MONT_MUL(w0, w1, w2, a[18], b[21]); + MONT_MAC(t0, t1, t2, a[19], b[20]); + MONT_MAC(v0, v1, v2, a[20], b[19]); + MONT_MAC(w0, w1, w2, a[21], b[18]); + MONT_MAC(t0, t1, t2, a[22], b[17]); + MONT_MAC(v0, v1, v2, a[23], b[16]); + MONT_MAC(w0, w1, w2, u[16], m[23]); + MONT_MAC(t0, t1, t2, u[17], m[22]); + MONT_MAC(v0, v1, v2, u[18], m[21]); + MONT_MAC(w0, w1, w2, u[19], m[20]); + MONT_MAC(t0, t1, t2, u[20], m[19]); + MONT_MAC(v0, v1, v2, u[21], m[18]); + MONT_MAC(w0, w1, w2, u[22], m[17]); + MONT_MAC(t0, t1, t2, u[23], m[16]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[15] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 40 */ + MONT_MAC(t0, t1, t2, a[17], b[23]); + MONT_MUL(v0, v1, v2, a[18], b[22]); + MONT_MUL(w0, w1, w2, a[19], b[21]); + MONT_MAC(t0, t1, t2, a[20], b[20]); + MONT_MAC(v0, v1, v2, a[21], b[19]); + MONT_MAC(w0, w1, w2, a[22], b[18]); + MONT_MAC(t0, t1, t2, a[23], b[17]); + MONT_MAC(v0, v1, v2, u[17], m[23]); + MONT_MAC(w0, w1, w2, u[18], m[22]); + MONT_MAC(t0, t1, t2, u[19], m[21]); + MONT_MAC(v0, v1, v2, u[20], m[20]); + MONT_MAC(w0, w1, w2, u[21], m[19]); + MONT_MAC(t0, t1, t2, u[22], m[18]); + MONT_MAC(v0, v1, v2, u[23], m[17]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[16] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 41 */ + MONT_MAC(t0, t1, t2, a[18], b[23]); + MONT_MUL(v0, v1, v2, a[19], b[22]); + MONT_MUL(w0, w1, w2, a[20], b[21]); + MONT_MAC(t0, t1, t2, a[21], b[20]); + MONT_MAC(v0, v1, v2, a[22], b[19]); + MONT_MAC(w0, w1, w2, a[23], b[18]); + MONT_MAC(t0, t1, t2, u[18], m[23]); + MONT_MAC(v0, v1, v2, u[19], m[22]); + MONT_MAC(w0, w1, w2, u[20], m[21]); + MONT_MAC(t0, t1, t2, u[21], m[20]); + MONT_MAC(v0, v1, v2, u[22], m[19]); + MONT_MAC(w0, w1, w2, u[23], m[18]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[17] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 42 */ + MONT_MAC(t0, t1, t2, a[19], b[23]); + MONT_MUL(v0, v1, v2, a[20], b[22]); + MONT_MUL(w0, w1, w2, a[21], b[21]); + MONT_MAC(t0, t1, t2, a[22], b[20]); + MONT_MAC(v0, v1, v2, a[23], b[19]); + MONT_MAC(w0, w1, w2, u[19], m[23]); + MONT_MAC(t0, t1, t2, u[20], m[22]); + MONT_MAC(v0, v1, v2, u[21], m[21]); + MONT_MAC(w0, w1, w2, u[22], m[20]); + MONT_MAC(t0, t1, t2, u[23], m[19]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[18] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 43 */ + MONT_MAC(t0, t1, t2, a[20], b[23]); + MONT_MUL(v0, v1, v2, a[21], b[22]); + MONT_MUL(w0, w1, w2, a[22], b[21]); + MONT_MAC(t0, t1, t2, a[23], b[20]); + MONT_MAC(v0, v1, v2, u[20], m[23]); + MONT_MAC(w0, w1, w2, u[21], m[22]); + MONT_MAC(t0, t1, t2, u[22], m[21]); + MONT_MAC(v0, v1, v2, u[23], m[20]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[19] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 44 */ + MONT_MAC(t0, t1, t2, a[21], b[23]); + MONT_MUL(v0, v1, v2, a[22], b[22]); + MONT_MUL(w0, w1, w2, a[23], b[21]); + MONT_MAC(t0, t1, t2, u[21], m[23]); + MONT_MAC(v0, v1, v2, u[22], m[22]); + MONT_MAC(w0, w1, w2, u[23], m[21]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[20] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 45 */ + MONT_MAC(t0, t1, t2, a[22], b[23]); + MONT_MUL(v0, v1, v2, a[23], b[22]); + MONT_MUL(w0, w1, w2, u[22], m[23]); + MONT_MAC(t0, t1, t2, u[23], m[22]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[21] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 46 */ + MONT_MAC(t0, t1, t2, a[23], b[23]); + MONT_MUL(v0, v1, v2, u[23], m[23]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + z[22] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + z[23] = t0; + + /* under 2m, with t1 the bit above n limbs: at most one subtraction */ + borrow = sub_n(r, z, m, 24); + take = t1 | (borrow ^ 1); + select_n(r, r, z, take & 1, 24); +} + +/* r = a * a * R^-1 mod m, for a under m; t is 2n limbs of scratch */ +static void crypton_mont_sqr24(limb_t *r, const limb_t *a, + const limb_t *m, limb_t n0, limb_t *t) +{ + limb_t *u = t, *z = t + 24; + limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take; + limb_t s0, s1, s2, c0, c1, c2; + + /* column 0 */ + MONT_MAC(t0, t1, t2, a[0], a[0]); + u[0] = t0 * n0; + MONT_MAC(t0, t1, t2, u[0], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 1 */ + MONT_MUL(s0, s1, s2, a[0], a[1]); + MONT_MAC(t0, t1, t2, u[0], m[1]); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[1] = t0 * n0; + MONT_MAC(t0, t1, t2, u[1], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 2 */ + MONT_MUL(s0, s1, s2, a[0], a[2]); + MONT_MAC(t0, t1, t2, a[1], a[1]); + MONT_MUL(v0, v1, v2, u[0], m[2]); + MONT_MUL(w0, w1, w2, u[1], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[2] = t0 * n0; + MONT_MAC(t0, t1, t2, u[2], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 3 */ + MONT_MUL(s0, s1, s2, a[0], a[3]); + MONT_MUL(c0, c1, c2, a[1], a[2]); + MONT_MAC(t0, t1, t2, u[0], m[3]); + MONT_MUL(v0, v1, v2, u[1], m[2]); + MONT_MUL(w0, w1, w2, u[2], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[3] = t0 * n0; + MONT_MAC(t0, t1, t2, u[3], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 4 */ + MONT_MUL(s0, s1, s2, a[0], a[4]); + MONT_MUL(c0, c1, c2, a[1], a[3]); + MONT_MAC(t0, t1, t2, a[2], a[2]); + MONT_MUL(v0, v1, v2, u[0], m[4]); + MONT_MUL(w0, w1, w2, u[1], m[3]); + MONT_MAC(t0, t1, t2, u[2], m[2]); + MONT_MAC(v0, v1, v2, u[3], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[4] = t0 * n0; + MONT_MAC(t0, t1, t2, u[4], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 5 */ + MONT_MUL(s0, s1, s2, a[0], a[5]); + MONT_MUL(c0, c1, c2, a[1], a[4]); + MONT_MAC(s0, s1, s2, a[2], a[3]); + MONT_MAC(t0, t1, t2, u[0], m[5]); + MONT_MUL(v0, v1, v2, u[1], m[4]); + MONT_MUL(w0, w1, w2, u[2], m[3]); + MONT_MAC(t0, t1, t2, u[3], m[2]); + MONT_MAC(v0, v1, v2, u[4], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[5] = t0 * n0; + MONT_MAC(t0, t1, t2, u[5], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 6 */ + MONT_MUL(s0, s1, s2, a[0], a[6]); + MONT_MUL(c0, c1, c2, a[1], a[5]); + MONT_MAC(s0, s1, s2, a[2], a[4]); + MONT_MAC(t0, t1, t2, a[3], a[3]); + MONT_MUL(v0, v1, v2, u[0], m[6]); + MONT_MUL(w0, w1, w2, u[1], m[5]); + MONT_MAC(t0, t1, t2, u[2], m[4]); + MONT_MAC(v0, v1, v2, u[3], m[3]); + MONT_MAC(w0, w1, w2, u[4], m[2]); + MONT_MAC(t0, t1, t2, u[5], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[6] = t0 * n0; + MONT_MAC(t0, t1, t2, u[6], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 7 */ + MONT_MUL(s0, s1, s2, a[0], a[7]); + MONT_MUL(c0, c1, c2, a[1], a[6]); + MONT_MAC(s0, s1, s2, a[2], a[5]); + MONT_MAC(c0, c1, c2, a[3], a[4]); + MONT_MAC(t0, t1, t2, u[0], m[7]); + MONT_MUL(v0, v1, v2, u[1], m[6]); + MONT_MUL(w0, w1, w2, u[2], m[5]); + MONT_MAC(t0, t1, t2, u[3], m[4]); + MONT_MAC(v0, v1, v2, u[4], m[3]); + MONT_MAC(w0, w1, w2, u[5], m[2]); + MONT_MAC(t0, t1, t2, u[6], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[7] = t0 * n0; + MONT_MAC(t0, t1, t2, u[7], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 8 */ + MONT_MUL(s0, s1, s2, a[0], a[8]); + MONT_MUL(c0, c1, c2, a[1], a[7]); + MONT_MAC(s0, s1, s2, a[2], a[6]); + MONT_MAC(c0, c1, c2, a[3], a[5]); + MONT_MAC(t0, t1, t2, a[4], a[4]); + MONT_MUL(v0, v1, v2, u[0], m[8]); + MONT_MUL(w0, w1, w2, u[1], m[7]); + MONT_MAC(t0, t1, t2, u[2], m[6]); + MONT_MAC(v0, v1, v2, u[3], m[5]); + MONT_MAC(w0, w1, w2, u[4], m[4]); + MONT_MAC(t0, t1, t2, u[5], m[3]); + MONT_MAC(v0, v1, v2, u[6], m[2]); + MONT_MAC(w0, w1, w2, u[7], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[8] = t0 * n0; + MONT_MAC(t0, t1, t2, u[8], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 9 */ + MONT_MUL(s0, s1, s2, a[0], a[9]); + MONT_MUL(c0, c1, c2, a[1], a[8]); + MONT_MAC(s0, s1, s2, a[2], a[7]); + MONT_MAC(c0, c1, c2, a[3], a[6]); + MONT_MAC(s0, s1, s2, a[4], a[5]); + MONT_MAC(t0, t1, t2, u[0], m[9]); + MONT_MUL(v0, v1, v2, u[1], m[8]); + MONT_MUL(w0, w1, w2, u[2], m[7]); + MONT_MAC(t0, t1, t2, u[3], m[6]); + MONT_MAC(v0, v1, v2, u[4], m[5]); + MONT_MAC(w0, w1, w2, u[5], m[4]); + MONT_MAC(t0, t1, t2, u[6], m[3]); + MONT_MAC(v0, v1, v2, u[7], m[2]); + MONT_MAC(w0, w1, w2, u[8], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[9] = t0 * n0; + MONT_MAC(t0, t1, t2, u[9], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 10 */ + MONT_MUL(s0, s1, s2, a[0], a[10]); + MONT_MUL(c0, c1, c2, a[1], a[9]); + MONT_MAC(s0, s1, s2, a[2], a[8]); + MONT_MAC(c0, c1, c2, a[3], a[7]); + MONT_MAC(s0, s1, s2, a[4], a[6]); + MONT_MAC(t0, t1, t2, a[5], a[5]); + MONT_MUL(v0, v1, v2, u[0], m[10]); + MONT_MUL(w0, w1, w2, u[1], m[9]); + MONT_MAC(t0, t1, t2, u[2], m[8]); + MONT_MAC(v0, v1, v2, u[3], m[7]); + MONT_MAC(w0, w1, w2, u[4], m[6]); + MONT_MAC(t0, t1, t2, u[5], m[5]); + MONT_MAC(v0, v1, v2, u[6], m[4]); + MONT_MAC(w0, w1, w2, u[7], m[3]); + MONT_MAC(t0, t1, t2, u[8], m[2]); + MONT_MAC(v0, v1, v2, u[9], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[10] = t0 * n0; + MONT_MAC(t0, t1, t2, u[10], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 11 */ + MONT_MUL(s0, s1, s2, a[0], a[11]); + MONT_MUL(c0, c1, c2, a[1], a[10]); + MONT_MAC(s0, s1, s2, a[2], a[9]); + MONT_MAC(c0, c1, c2, a[3], a[8]); + MONT_MAC(s0, s1, s2, a[4], a[7]); + MONT_MAC(c0, c1, c2, a[5], a[6]); + MONT_MAC(t0, t1, t2, u[0], m[11]); + MONT_MUL(v0, v1, v2, u[1], m[10]); + MONT_MUL(w0, w1, w2, u[2], m[9]); + MONT_MAC(t0, t1, t2, u[3], m[8]); + MONT_MAC(v0, v1, v2, u[4], m[7]); + MONT_MAC(w0, w1, w2, u[5], m[6]); + MONT_MAC(t0, t1, t2, u[6], m[5]); + MONT_MAC(v0, v1, v2, u[7], m[4]); + MONT_MAC(w0, w1, w2, u[8], m[3]); + MONT_MAC(t0, t1, t2, u[9], m[2]); + MONT_MAC(v0, v1, v2, u[10], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[11] = t0 * n0; + MONT_MAC(t0, t1, t2, u[11], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 12 */ + MONT_MUL(s0, s1, s2, a[0], a[12]); + MONT_MUL(c0, c1, c2, a[1], a[11]); + MONT_MAC(s0, s1, s2, a[2], a[10]); + MONT_MAC(c0, c1, c2, a[3], a[9]); + MONT_MAC(s0, s1, s2, a[4], a[8]); + MONT_MAC(c0, c1, c2, a[5], a[7]); + MONT_MAC(t0, t1, t2, a[6], a[6]); + MONT_MUL(v0, v1, v2, u[0], m[12]); + MONT_MUL(w0, w1, w2, u[1], m[11]); + MONT_MAC(t0, t1, t2, u[2], m[10]); + MONT_MAC(v0, v1, v2, u[3], m[9]); + MONT_MAC(w0, w1, w2, u[4], m[8]); + MONT_MAC(t0, t1, t2, u[5], m[7]); + MONT_MAC(v0, v1, v2, u[6], m[6]); + MONT_MAC(w0, w1, w2, u[7], m[5]); + MONT_MAC(t0, t1, t2, u[8], m[4]); + MONT_MAC(v0, v1, v2, u[9], m[3]); + MONT_MAC(w0, w1, w2, u[10], m[2]); + MONT_MAC(t0, t1, t2, u[11], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[12] = t0 * n0; + MONT_MAC(t0, t1, t2, u[12], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 13 */ + MONT_MUL(s0, s1, s2, a[0], a[13]); + MONT_MUL(c0, c1, c2, a[1], a[12]); + MONT_MAC(s0, s1, s2, a[2], a[11]); + MONT_MAC(c0, c1, c2, a[3], a[10]); + MONT_MAC(s0, s1, s2, a[4], a[9]); + MONT_MAC(c0, c1, c2, a[5], a[8]); + MONT_MAC(s0, s1, s2, a[6], a[7]); + MONT_MAC(t0, t1, t2, u[0], m[13]); + MONT_MUL(v0, v1, v2, u[1], m[12]); + MONT_MUL(w0, w1, w2, u[2], m[11]); + MONT_MAC(t0, t1, t2, u[3], m[10]); + MONT_MAC(v0, v1, v2, u[4], m[9]); + MONT_MAC(w0, w1, w2, u[5], m[8]); + MONT_MAC(t0, t1, t2, u[6], m[7]); + MONT_MAC(v0, v1, v2, u[7], m[6]); + MONT_MAC(w0, w1, w2, u[8], m[5]); + MONT_MAC(t0, t1, t2, u[9], m[4]); + MONT_MAC(v0, v1, v2, u[10], m[3]); + MONT_MAC(w0, w1, w2, u[11], m[2]); + MONT_MAC(t0, t1, t2, u[12], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[13] = t0 * n0; + MONT_MAC(t0, t1, t2, u[13], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 14 */ + MONT_MUL(s0, s1, s2, a[0], a[14]); + MONT_MUL(c0, c1, c2, a[1], a[13]); + MONT_MAC(s0, s1, s2, a[2], a[12]); + MONT_MAC(c0, c1, c2, a[3], a[11]); + MONT_MAC(s0, s1, s2, a[4], a[10]); + MONT_MAC(c0, c1, c2, a[5], a[9]); + MONT_MAC(s0, s1, s2, a[6], a[8]); + MONT_MAC(t0, t1, t2, a[7], a[7]); + MONT_MUL(v0, v1, v2, u[0], m[14]); + MONT_MUL(w0, w1, w2, u[1], m[13]); + MONT_MAC(t0, t1, t2, u[2], m[12]); + MONT_MAC(v0, v1, v2, u[3], m[11]); + MONT_MAC(w0, w1, w2, u[4], m[10]); + MONT_MAC(t0, t1, t2, u[5], m[9]); + MONT_MAC(v0, v1, v2, u[6], m[8]); + MONT_MAC(w0, w1, w2, u[7], m[7]); + MONT_MAC(t0, t1, t2, u[8], m[6]); + MONT_MAC(v0, v1, v2, u[9], m[5]); + MONT_MAC(w0, w1, w2, u[10], m[4]); + MONT_MAC(t0, t1, t2, u[11], m[3]); + MONT_MAC(v0, v1, v2, u[12], m[2]); + MONT_MAC(w0, w1, w2, u[13], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[14] = t0 * n0; + MONT_MAC(t0, t1, t2, u[14], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 15 */ + MONT_MUL(s0, s1, s2, a[0], a[15]); + MONT_MUL(c0, c1, c2, a[1], a[14]); + MONT_MAC(s0, s1, s2, a[2], a[13]); + MONT_MAC(c0, c1, c2, a[3], a[12]); + MONT_MAC(s0, s1, s2, a[4], a[11]); + MONT_MAC(c0, c1, c2, a[5], a[10]); + MONT_MAC(s0, s1, s2, a[6], a[9]); + MONT_MAC(c0, c1, c2, a[7], a[8]); + MONT_MAC(t0, t1, t2, u[0], m[15]); + MONT_MUL(v0, v1, v2, u[1], m[14]); + MONT_MUL(w0, w1, w2, u[2], m[13]); + MONT_MAC(t0, t1, t2, u[3], m[12]); + MONT_MAC(v0, v1, v2, u[4], m[11]); + MONT_MAC(w0, w1, w2, u[5], m[10]); + MONT_MAC(t0, t1, t2, u[6], m[9]); + MONT_MAC(v0, v1, v2, u[7], m[8]); + MONT_MAC(w0, w1, w2, u[8], m[7]); + MONT_MAC(t0, t1, t2, u[9], m[6]); + MONT_MAC(v0, v1, v2, u[10], m[5]); + MONT_MAC(w0, w1, w2, u[11], m[4]); + MONT_MAC(t0, t1, t2, u[12], m[3]); + MONT_MAC(v0, v1, v2, u[13], m[2]); + MONT_MAC(w0, w1, w2, u[14], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[15] = t0 * n0; + MONT_MAC(t0, t1, t2, u[15], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 16 */ + MONT_MUL(s0, s1, s2, a[0], a[16]); + MONT_MUL(c0, c1, c2, a[1], a[15]); + MONT_MAC(s0, s1, s2, a[2], a[14]); + MONT_MAC(c0, c1, c2, a[3], a[13]); + MONT_MAC(s0, s1, s2, a[4], a[12]); + MONT_MAC(c0, c1, c2, a[5], a[11]); + MONT_MAC(s0, s1, s2, a[6], a[10]); + MONT_MAC(c0, c1, c2, a[7], a[9]); + MONT_MAC(t0, t1, t2, a[8], a[8]); + MONT_MUL(v0, v1, v2, u[0], m[16]); + MONT_MUL(w0, w1, w2, u[1], m[15]); + MONT_MAC(t0, t1, t2, u[2], m[14]); + MONT_MAC(v0, v1, v2, u[3], m[13]); + MONT_MAC(w0, w1, w2, u[4], m[12]); + MONT_MAC(t0, t1, t2, u[5], m[11]); + MONT_MAC(v0, v1, v2, u[6], m[10]); + MONT_MAC(w0, w1, w2, u[7], m[9]); + MONT_MAC(t0, t1, t2, u[8], m[8]); + MONT_MAC(v0, v1, v2, u[9], m[7]); + MONT_MAC(w0, w1, w2, u[10], m[6]); + MONT_MAC(t0, t1, t2, u[11], m[5]); + MONT_MAC(v0, v1, v2, u[12], m[4]); + MONT_MAC(w0, w1, w2, u[13], m[3]); + MONT_MAC(t0, t1, t2, u[14], m[2]); + MONT_MAC(v0, v1, v2, u[15], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[16] = t0 * n0; + MONT_MAC(t0, t1, t2, u[16], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 17 */ + MONT_MUL(s0, s1, s2, a[0], a[17]); + MONT_MUL(c0, c1, c2, a[1], a[16]); + MONT_MAC(s0, s1, s2, a[2], a[15]); + MONT_MAC(c0, c1, c2, a[3], a[14]); + MONT_MAC(s0, s1, s2, a[4], a[13]); + MONT_MAC(c0, c1, c2, a[5], a[12]); + MONT_MAC(s0, s1, s2, a[6], a[11]); + MONT_MAC(c0, c1, c2, a[7], a[10]); + MONT_MAC(s0, s1, s2, a[8], a[9]); + MONT_MAC(t0, t1, t2, u[0], m[17]); + MONT_MUL(v0, v1, v2, u[1], m[16]); + MONT_MUL(w0, w1, w2, u[2], m[15]); + MONT_MAC(t0, t1, t2, u[3], m[14]); + MONT_MAC(v0, v1, v2, u[4], m[13]); + MONT_MAC(w0, w1, w2, u[5], m[12]); + MONT_MAC(t0, t1, t2, u[6], m[11]); + MONT_MAC(v0, v1, v2, u[7], m[10]); + MONT_MAC(w0, w1, w2, u[8], m[9]); + MONT_MAC(t0, t1, t2, u[9], m[8]); + MONT_MAC(v0, v1, v2, u[10], m[7]); + MONT_MAC(w0, w1, w2, u[11], m[6]); + MONT_MAC(t0, t1, t2, u[12], m[5]); + MONT_MAC(v0, v1, v2, u[13], m[4]); + MONT_MAC(w0, w1, w2, u[14], m[3]); + MONT_MAC(t0, t1, t2, u[15], m[2]); + MONT_MAC(v0, v1, v2, u[16], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[17] = t0 * n0; + MONT_MAC(t0, t1, t2, u[17], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 18 */ + MONT_MUL(s0, s1, s2, a[0], a[18]); + MONT_MUL(c0, c1, c2, a[1], a[17]); + MONT_MAC(s0, s1, s2, a[2], a[16]); + MONT_MAC(c0, c1, c2, a[3], a[15]); + MONT_MAC(s0, s1, s2, a[4], a[14]); + MONT_MAC(c0, c1, c2, a[5], a[13]); + MONT_MAC(s0, s1, s2, a[6], a[12]); + MONT_MAC(c0, c1, c2, a[7], a[11]); + MONT_MAC(s0, s1, s2, a[8], a[10]); + MONT_MAC(t0, t1, t2, a[9], a[9]); + MONT_MUL(v0, v1, v2, u[0], m[18]); + MONT_MUL(w0, w1, w2, u[1], m[17]); + MONT_MAC(t0, t1, t2, u[2], m[16]); + MONT_MAC(v0, v1, v2, u[3], m[15]); + MONT_MAC(w0, w1, w2, u[4], m[14]); + MONT_MAC(t0, t1, t2, u[5], m[13]); + MONT_MAC(v0, v1, v2, u[6], m[12]); + MONT_MAC(w0, w1, w2, u[7], m[11]); + MONT_MAC(t0, t1, t2, u[8], m[10]); + MONT_MAC(v0, v1, v2, u[9], m[9]); + MONT_MAC(w0, w1, w2, u[10], m[8]); + MONT_MAC(t0, t1, t2, u[11], m[7]); + MONT_MAC(v0, v1, v2, u[12], m[6]); + MONT_MAC(w0, w1, w2, u[13], m[5]); + MONT_MAC(t0, t1, t2, u[14], m[4]); + MONT_MAC(v0, v1, v2, u[15], m[3]); + MONT_MAC(w0, w1, w2, u[16], m[2]); + MONT_MAC(t0, t1, t2, u[17], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[18] = t0 * n0; + MONT_MAC(t0, t1, t2, u[18], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 19 */ + MONT_MUL(s0, s1, s2, a[0], a[19]); + MONT_MUL(c0, c1, c2, a[1], a[18]); + MONT_MAC(s0, s1, s2, a[2], a[17]); + MONT_MAC(c0, c1, c2, a[3], a[16]); + MONT_MAC(s0, s1, s2, a[4], a[15]); + MONT_MAC(c0, c1, c2, a[5], a[14]); + MONT_MAC(s0, s1, s2, a[6], a[13]); + MONT_MAC(c0, c1, c2, a[7], a[12]); + MONT_MAC(s0, s1, s2, a[8], a[11]); + MONT_MAC(c0, c1, c2, a[9], a[10]); + MONT_MAC(t0, t1, t2, u[0], m[19]); + MONT_MUL(v0, v1, v2, u[1], m[18]); + MONT_MUL(w0, w1, w2, u[2], m[17]); + MONT_MAC(t0, t1, t2, u[3], m[16]); + MONT_MAC(v0, v1, v2, u[4], m[15]); + MONT_MAC(w0, w1, w2, u[5], m[14]); + MONT_MAC(t0, t1, t2, u[6], m[13]); + MONT_MAC(v0, v1, v2, u[7], m[12]); + MONT_MAC(w0, w1, w2, u[8], m[11]); + MONT_MAC(t0, t1, t2, u[9], m[10]); + MONT_MAC(v0, v1, v2, u[10], m[9]); + MONT_MAC(w0, w1, w2, u[11], m[8]); + MONT_MAC(t0, t1, t2, u[12], m[7]); + MONT_MAC(v0, v1, v2, u[13], m[6]); + MONT_MAC(w0, w1, w2, u[14], m[5]); + MONT_MAC(t0, t1, t2, u[15], m[4]); + MONT_MAC(v0, v1, v2, u[16], m[3]); + MONT_MAC(w0, w1, w2, u[17], m[2]); + MONT_MAC(t0, t1, t2, u[18], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[19] = t0 * n0; + MONT_MAC(t0, t1, t2, u[19], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 20 */ + MONT_MUL(s0, s1, s2, a[0], a[20]); + MONT_MUL(c0, c1, c2, a[1], a[19]); + MONT_MAC(s0, s1, s2, a[2], a[18]); + MONT_MAC(c0, c1, c2, a[3], a[17]); + MONT_MAC(s0, s1, s2, a[4], a[16]); + MONT_MAC(c0, c1, c2, a[5], a[15]); + MONT_MAC(s0, s1, s2, a[6], a[14]); + MONT_MAC(c0, c1, c2, a[7], a[13]); + MONT_MAC(s0, s1, s2, a[8], a[12]); + MONT_MAC(c0, c1, c2, a[9], a[11]); + MONT_MAC(t0, t1, t2, a[10], a[10]); + MONT_MUL(v0, v1, v2, u[0], m[20]); + MONT_MUL(w0, w1, w2, u[1], m[19]); + MONT_MAC(t0, t1, t2, u[2], m[18]); + MONT_MAC(v0, v1, v2, u[3], m[17]); + MONT_MAC(w0, w1, w2, u[4], m[16]); + MONT_MAC(t0, t1, t2, u[5], m[15]); + MONT_MAC(v0, v1, v2, u[6], m[14]); + MONT_MAC(w0, w1, w2, u[7], m[13]); + MONT_MAC(t0, t1, t2, u[8], m[12]); + MONT_MAC(v0, v1, v2, u[9], m[11]); + MONT_MAC(w0, w1, w2, u[10], m[10]); + MONT_MAC(t0, t1, t2, u[11], m[9]); + MONT_MAC(v0, v1, v2, u[12], m[8]); + MONT_MAC(w0, w1, w2, u[13], m[7]); + MONT_MAC(t0, t1, t2, u[14], m[6]); + MONT_MAC(v0, v1, v2, u[15], m[5]); + MONT_MAC(w0, w1, w2, u[16], m[4]); + MONT_MAC(t0, t1, t2, u[17], m[3]); + MONT_MAC(v0, v1, v2, u[18], m[2]); + MONT_MAC(w0, w1, w2, u[19], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[20] = t0 * n0; + MONT_MAC(t0, t1, t2, u[20], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 21 */ + MONT_MUL(s0, s1, s2, a[0], a[21]); + MONT_MUL(c0, c1, c2, a[1], a[20]); + MONT_MAC(s0, s1, s2, a[2], a[19]); + MONT_MAC(c0, c1, c2, a[3], a[18]); + MONT_MAC(s0, s1, s2, a[4], a[17]); + MONT_MAC(c0, c1, c2, a[5], a[16]); + MONT_MAC(s0, s1, s2, a[6], a[15]); + MONT_MAC(c0, c1, c2, a[7], a[14]); + MONT_MAC(s0, s1, s2, a[8], a[13]); + MONT_MAC(c0, c1, c2, a[9], a[12]); + MONT_MAC(s0, s1, s2, a[10], a[11]); + MONT_MAC(t0, t1, t2, u[0], m[21]); + MONT_MUL(v0, v1, v2, u[1], m[20]); + MONT_MUL(w0, w1, w2, u[2], m[19]); + MONT_MAC(t0, t1, t2, u[3], m[18]); + MONT_MAC(v0, v1, v2, u[4], m[17]); + MONT_MAC(w0, w1, w2, u[5], m[16]); + MONT_MAC(t0, t1, t2, u[6], m[15]); + MONT_MAC(v0, v1, v2, u[7], m[14]); + MONT_MAC(w0, w1, w2, u[8], m[13]); + MONT_MAC(t0, t1, t2, u[9], m[12]); + MONT_MAC(v0, v1, v2, u[10], m[11]); + MONT_MAC(w0, w1, w2, u[11], m[10]); + MONT_MAC(t0, t1, t2, u[12], m[9]); + MONT_MAC(v0, v1, v2, u[13], m[8]); + MONT_MAC(w0, w1, w2, u[14], m[7]); + MONT_MAC(t0, t1, t2, u[15], m[6]); + MONT_MAC(v0, v1, v2, u[16], m[5]); + MONT_MAC(w0, w1, w2, u[17], m[4]); + MONT_MAC(t0, t1, t2, u[18], m[3]); + MONT_MAC(v0, v1, v2, u[19], m[2]); + MONT_MAC(w0, w1, w2, u[20], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[21] = t0 * n0; + MONT_MAC(t0, t1, t2, u[21], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 22 */ + MONT_MUL(s0, s1, s2, a[0], a[22]); + MONT_MUL(c0, c1, c2, a[1], a[21]); + MONT_MAC(s0, s1, s2, a[2], a[20]); + MONT_MAC(c0, c1, c2, a[3], a[19]); + MONT_MAC(s0, s1, s2, a[4], a[18]); + MONT_MAC(c0, c1, c2, a[5], a[17]); + MONT_MAC(s0, s1, s2, a[6], a[16]); + MONT_MAC(c0, c1, c2, a[7], a[15]); + MONT_MAC(s0, s1, s2, a[8], a[14]); + MONT_MAC(c0, c1, c2, a[9], a[13]); + MONT_MAC(s0, s1, s2, a[10], a[12]); + MONT_MAC(t0, t1, t2, a[11], a[11]); + MONT_MUL(v0, v1, v2, u[0], m[22]); + MONT_MUL(w0, w1, w2, u[1], m[21]); + MONT_MAC(t0, t1, t2, u[2], m[20]); + MONT_MAC(v0, v1, v2, u[3], m[19]); + MONT_MAC(w0, w1, w2, u[4], m[18]); + MONT_MAC(t0, t1, t2, u[5], m[17]); + MONT_MAC(v0, v1, v2, u[6], m[16]); + MONT_MAC(w0, w1, w2, u[7], m[15]); + MONT_MAC(t0, t1, t2, u[8], m[14]); + MONT_MAC(v0, v1, v2, u[9], m[13]); + MONT_MAC(w0, w1, w2, u[10], m[12]); + MONT_MAC(t0, t1, t2, u[11], m[11]); + MONT_MAC(v0, v1, v2, u[12], m[10]); + MONT_MAC(w0, w1, w2, u[13], m[9]); + MONT_MAC(t0, t1, t2, u[14], m[8]); + MONT_MAC(v0, v1, v2, u[15], m[7]); + MONT_MAC(w0, w1, w2, u[16], m[6]); + MONT_MAC(t0, t1, t2, u[17], m[5]); + MONT_MAC(v0, v1, v2, u[18], m[4]); + MONT_MAC(w0, w1, w2, u[19], m[3]); + MONT_MAC(t0, t1, t2, u[20], m[2]); + MONT_MAC(v0, v1, v2, u[21], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[22] = t0 * n0; + MONT_MAC(t0, t1, t2, u[22], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 23 */ + MONT_MUL(s0, s1, s2, a[0], a[23]); + MONT_MUL(c0, c1, c2, a[1], a[22]); + MONT_MAC(s0, s1, s2, a[2], a[21]); + MONT_MAC(c0, c1, c2, a[3], a[20]); + MONT_MAC(s0, s1, s2, a[4], a[19]); + MONT_MAC(c0, c1, c2, a[5], a[18]); + MONT_MAC(s0, s1, s2, a[6], a[17]); + MONT_MAC(c0, c1, c2, a[7], a[16]); + MONT_MAC(s0, s1, s2, a[8], a[15]); + MONT_MAC(c0, c1, c2, a[9], a[14]); + MONT_MAC(s0, s1, s2, a[10], a[13]); + MONT_MAC(c0, c1, c2, a[11], a[12]); + MONT_MAC(t0, t1, t2, u[0], m[23]); + MONT_MUL(v0, v1, v2, u[1], m[22]); + MONT_MUL(w0, w1, w2, u[2], m[21]); + MONT_MAC(t0, t1, t2, u[3], m[20]); + MONT_MAC(v0, v1, v2, u[4], m[19]); + MONT_MAC(w0, w1, w2, u[5], m[18]); + MONT_MAC(t0, t1, t2, u[6], m[17]); + MONT_MAC(v0, v1, v2, u[7], m[16]); + MONT_MAC(w0, w1, w2, u[8], m[15]); + MONT_MAC(t0, t1, t2, u[9], m[14]); + MONT_MAC(v0, v1, v2, u[10], m[13]); + MONT_MAC(w0, w1, w2, u[11], m[12]); + MONT_MAC(t0, t1, t2, u[12], m[11]); + MONT_MAC(v0, v1, v2, u[13], m[10]); + MONT_MAC(w0, w1, w2, u[14], m[9]); + MONT_MAC(t0, t1, t2, u[15], m[8]); + MONT_MAC(v0, v1, v2, u[16], m[7]); + MONT_MAC(w0, w1, w2, u[17], m[6]); + MONT_MAC(t0, t1, t2, u[18], m[5]); + MONT_MAC(v0, v1, v2, u[19], m[4]); + MONT_MAC(w0, w1, w2, u[20], m[3]); + MONT_MAC(t0, t1, t2, u[21], m[2]); + MONT_MAC(v0, v1, v2, u[22], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[23] = t0 * n0; + MONT_MAC(t0, t1, t2, u[23], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 24 */ + MONT_MUL(s0, s1, s2, a[1], a[23]); + MONT_MUL(c0, c1, c2, a[2], a[22]); + MONT_MAC(s0, s1, s2, a[3], a[21]); + MONT_MAC(c0, c1, c2, a[4], a[20]); + MONT_MAC(s0, s1, s2, a[5], a[19]); + MONT_MAC(c0, c1, c2, a[6], a[18]); + MONT_MAC(s0, s1, s2, a[7], a[17]); + MONT_MAC(c0, c1, c2, a[8], a[16]); + MONT_MAC(s0, s1, s2, a[9], a[15]); + MONT_MAC(c0, c1, c2, a[10], a[14]); + MONT_MAC(s0, s1, s2, a[11], a[13]); + MONT_MAC(t0, t1, t2, a[12], a[12]); + MONT_MUL(v0, v1, v2, u[1], m[23]); + MONT_MUL(w0, w1, w2, u[2], m[22]); + MONT_MAC(t0, t1, t2, u[3], m[21]); + MONT_MAC(v0, v1, v2, u[4], m[20]); + MONT_MAC(w0, w1, w2, u[5], m[19]); + MONT_MAC(t0, t1, t2, u[6], m[18]); + MONT_MAC(v0, v1, v2, u[7], m[17]); + MONT_MAC(w0, w1, w2, u[8], m[16]); + MONT_MAC(t0, t1, t2, u[9], m[15]); + MONT_MAC(v0, v1, v2, u[10], m[14]); + MONT_MAC(w0, w1, w2, u[11], m[13]); + MONT_MAC(t0, t1, t2, u[12], m[12]); + MONT_MAC(v0, v1, v2, u[13], m[11]); + MONT_MAC(w0, w1, w2, u[14], m[10]); + MONT_MAC(t0, t1, t2, u[15], m[9]); + MONT_MAC(v0, v1, v2, u[16], m[8]); + MONT_MAC(w0, w1, w2, u[17], m[7]); + MONT_MAC(t0, t1, t2, u[18], m[6]); + MONT_MAC(v0, v1, v2, u[19], m[5]); + MONT_MAC(w0, w1, w2, u[20], m[4]); + MONT_MAC(t0, t1, t2, u[21], m[3]); + MONT_MAC(v0, v1, v2, u[22], m[2]); + MONT_MAC(w0, w1, w2, u[23], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[0] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 25 */ + MONT_MUL(s0, s1, s2, a[2], a[23]); + MONT_MUL(c0, c1, c2, a[3], a[22]); + MONT_MAC(s0, s1, s2, a[4], a[21]); + MONT_MAC(c0, c1, c2, a[5], a[20]); + MONT_MAC(s0, s1, s2, a[6], a[19]); + MONT_MAC(c0, c1, c2, a[7], a[18]); + MONT_MAC(s0, s1, s2, a[8], a[17]); + MONT_MAC(c0, c1, c2, a[9], a[16]); + MONT_MAC(s0, s1, s2, a[10], a[15]); + MONT_MAC(c0, c1, c2, a[11], a[14]); + MONT_MAC(s0, s1, s2, a[12], a[13]); + MONT_MAC(t0, t1, t2, u[2], m[23]); + MONT_MUL(v0, v1, v2, u[3], m[22]); + MONT_MUL(w0, w1, w2, u[4], m[21]); + MONT_MAC(t0, t1, t2, u[5], m[20]); + MONT_MAC(v0, v1, v2, u[6], m[19]); + MONT_MAC(w0, w1, w2, u[7], m[18]); + MONT_MAC(t0, t1, t2, u[8], m[17]); + MONT_MAC(v0, v1, v2, u[9], m[16]); + MONT_MAC(w0, w1, w2, u[10], m[15]); + MONT_MAC(t0, t1, t2, u[11], m[14]); + MONT_MAC(v0, v1, v2, u[12], m[13]); + MONT_MAC(w0, w1, w2, u[13], m[12]); + MONT_MAC(t0, t1, t2, u[14], m[11]); + MONT_MAC(v0, v1, v2, u[15], m[10]); + MONT_MAC(w0, w1, w2, u[16], m[9]); + MONT_MAC(t0, t1, t2, u[17], m[8]); + MONT_MAC(v0, v1, v2, u[18], m[7]); + MONT_MAC(w0, w1, w2, u[19], m[6]); + MONT_MAC(t0, t1, t2, u[20], m[5]); + MONT_MAC(v0, v1, v2, u[21], m[4]); + MONT_MAC(w0, w1, w2, u[22], m[3]); + MONT_MAC(t0, t1, t2, u[23], m[2]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[1] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 26 */ + MONT_MUL(s0, s1, s2, a[3], a[23]); + MONT_MUL(c0, c1, c2, a[4], a[22]); + MONT_MAC(s0, s1, s2, a[5], a[21]); + MONT_MAC(c0, c1, c2, a[6], a[20]); + MONT_MAC(s0, s1, s2, a[7], a[19]); + MONT_MAC(c0, c1, c2, a[8], a[18]); + MONT_MAC(s0, s1, s2, a[9], a[17]); + MONT_MAC(c0, c1, c2, a[10], a[16]); + MONT_MAC(s0, s1, s2, a[11], a[15]); + MONT_MAC(c0, c1, c2, a[12], a[14]); + MONT_MAC(t0, t1, t2, a[13], a[13]); + MONT_MUL(v0, v1, v2, u[3], m[23]); + MONT_MUL(w0, w1, w2, u[4], m[22]); + MONT_MAC(t0, t1, t2, u[5], m[21]); + MONT_MAC(v0, v1, v2, u[6], m[20]); + MONT_MAC(w0, w1, w2, u[7], m[19]); + MONT_MAC(t0, t1, t2, u[8], m[18]); + MONT_MAC(v0, v1, v2, u[9], m[17]); + MONT_MAC(w0, w1, w2, u[10], m[16]); + MONT_MAC(t0, t1, t2, u[11], m[15]); + MONT_MAC(v0, v1, v2, u[12], m[14]); + MONT_MAC(w0, w1, w2, u[13], m[13]); + MONT_MAC(t0, t1, t2, u[14], m[12]); + MONT_MAC(v0, v1, v2, u[15], m[11]); + MONT_MAC(w0, w1, w2, u[16], m[10]); + MONT_MAC(t0, t1, t2, u[17], m[9]); + MONT_MAC(v0, v1, v2, u[18], m[8]); + MONT_MAC(w0, w1, w2, u[19], m[7]); + MONT_MAC(t0, t1, t2, u[20], m[6]); + MONT_MAC(v0, v1, v2, u[21], m[5]); + MONT_MAC(w0, w1, w2, u[22], m[4]); + MONT_MAC(t0, t1, t2, u[23], m[3]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[2] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 27 */ + MONT_MUL(s0, s1, s2, a[4], a[23]); + MONT_MUL(c0, c1, c2, a[5], a[22]); + MONT_MAC(s0, s1, s2, a[6], a[21]); + MONT_MAC(c0, c1, c2, a[7], a[20]); + MONT_MAC(s0, s1, s2, a[8], a[19]); + MONT_MAC(c0, c1, c2, a[9], a[18]); + MONT_MAC(s0, s1, s2, a[10], a[17]); + MONT_MAC(c0, c1, c2, a[11], a[16]); + MONT_MAC(s0, s1, s2, a[12], a[15]); + MONT_MAC(c0, c1, c2, a[13], a[14]); + MONT_MAC(t0, t1, t2, u[4], m[23]); + MONT_MUL(v0, v1, v2, u[5], m[22]); + MONT_MUL(w0, w1, w2, u[6], m[21]); + MONT_MAC(t0, t1, t2, u[7], m[20]); + MONT_MAC(v0, v1, v2, u[8], m[19]); + MONT_MAC(w0, w1, w2, u[9], m[18]); + MONT_MAC(t0, t1, t2, u[10], m[17]); + MONT_MAC(v0, v1, v2, u[11], m[16]); + MONT_MAC(w0, w1, w2, u[12], m[15]); + MONT_MAC(t0, t1, t2, u[13], m[14]); + MONT_MAC(v0, v1, v2, u[14], m[13]); + MONT_MAC(w0, w1, w2, u[15], m[12]); + MONT_MAC(t0, t1, t2, u[16], m[11]); + MONT_MAC(v0, v1, v2, u[17], m[10]); + MONT_MAC(w0, w1, w2, u[18], m[9]); + MONT_MAC(t0, t1, t2, u[19], m[8]); + MONT_MAC(v0, v1, v2, u[20], m[7]); + MONT_MAC(w0, w1, w2, u[21], m[6]); + MONT_MAC(t0, t1, t2, u[22], m[5]); + MONT_MAC(v0, v1, v2, u[23], m[4]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[3] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 28 */ + MONT_MUL(s0, s1, s2, a[5], a[23]); + MONT_MUL(c0, c1, c2, a[6], a[22]); + MONT_MAC(s0, s1, s2, a[7], a[21]); + MONT_MAC(c0, c1, c2, a[8], a[20]); + MONT_MAC(s0, s1, s2, a[9], a[19]); + MONT_MAC(c0, c1, c2, a[10], a[18]); + MONT_MAC(s0, s1, s2, a[11], a[17]); + MONT_MAC(c0, c1, c2, a[12], a[16]); + MONT_MAC(s0, s1, s2, a[13], a[15]); + MONT_MAC(t0, t1, t2, a[14], a[14]); + MONT_MUL(v0, v1, v2, u[5], m[23]); + MONT_MUL(w0, w1, w2, u[6], m[22]); + MONT_MAC(t0, t1, t2, u[7], m[21]); + MONT_MAC(v0, v1, v2, u[8], m[20]); + MONT_MAC(w0, w1, w2, u[9], m[19]); + MONT_MAC(t0, t1, t2, u[10], m[18]); + MONT_MAC(v0, v1, v2, u[11], m[17]); + MONT_MAC(w0, w1, w2, u[12], m[16]); + MONT_MAC(t0, t1, t2, u[13], m[15]); + MONT_MAC(v0, v1, v2, u[14], m[14]); + MONT_MAC(w0, w1, w2, u[15], m[13]); + MONT_MAC(t0, t1, t2, u[16], m[12]); + MONT_MAC(v0, v1, v2, u[17], m[11]); + MONT_MAC(w0, w1, w2, u[18], m[10]); + MONT_MAC(t0, t1, t2, u[19], m[9]); + MONT_MAC(v0, v1, v2, u[20], m[8]); + MONT_MAC(w0, w1, w2, u[21], m[7]); + MONT_MAC(t0, t1, t2, u[22], m[6]); + MONT_MAC(v0, v1, v2, u[23], m[5]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[4] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 29 */ + MONT_MUL(s0, s1, s2, a[6], a[23]); + MONT_MUL(c0, c1, c2, a[7], a[22]); + MONT_MAC(s0, s1, s2, a[8], a[21]); + MONT_MAC(c0, c1, c2, a[9], a[20]); + MONT_MAC(s0, s1, s2, a[10], a[19]); + MONT_MAC(c0, c1, c2, a[11], a[18]); + MONT_MAC(s0, s1, s2, a[12], a[17]); + MONT_MAC(c0, c1, c2, a[13], a[16]); + MONT_MAC(s0, s1, s2, a[14], a[15]); + MONT_MAC(t0, t1, t2, u[6], m[23]); + MONT_MUL(v0, v1, v2, u[7], m[22]); + MONT_MUL(w0, w1, w2, u[8], m[21]); + MONT_MAC(t0, t1, t2, u[9], m[20]); + MONT_MAC(v0, v1, v2, u[10], m[19]); + MONT_MAC(w0, w1, w2, u[11], m[18]); + MONT_MAC(t0, t1, t2, u[12], m[17]); + MONT_MAC(v0, v1, v2, u[13], m[16]); + MONT_MAC(w0, w1, w2, u[14], m[15]); + MONT_MAC(t0, t1, t2, u[15], m[14]); + MONT_MAC(v0, v1, v2, u[16], m[13]); + MONT_MAC(w0, w1, w2, u[17], m[12]); + MONT_MAC(t0, t1, t2, u[18], m[11]); + MONT_MAC(v0, v1, v2, u[19], m[10]); + MONT_MAC(w0, w1, w2, u[20], m[9]); + MONT_MAC(t0, t1, t2, u[21], m[8]); + MONT_MAC(v0, v1, v2, u[22], m[7]); + MONT_MAC(w0, w1, w2, u[23], m[6]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[5] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 30 */ + MONT_MUL(s0, s1, s2, a[7], a[23]); + MONT_MUL(c0, c1, c2, a[8], a[22]); + MONT_MAC(s0, s1, s2, a[9], a[21]); + MONT_MAC(c0, c1, c2, a[10], a[20]); + MONT_MAC(s0, s1, s2, a[11], a[19]); + MONT_MAC(c0, c1, c2, a[12], a[18]); + MONT_MAC(s0, s1, s2, a[13], a[17]); + MONT_MAC(c0, c1, c2, a[14], a[16]); + MONT_MAC(t0, t1, t2, a[15], a[15]); + MONT_MUL(v0, v1, v2, u[7], m[23]); + MONT_MUL(w0, w1, w2, u[8], m[22]); + MONT_MAC(t0, t1, t2, u[9], m[21]); + MONT_MAC(v0, v1, v2, u[10], m[20]); + MONT_MAC(w0, w1, w2, u[11], m[19]); + MONT_MAC(t0, t1, t2, u[12], m[18]); + MONT_MAC(v0, v1, v2, u[13], m[17]); + MONT_MAC(w0, w1, w2, u[14], m[16]); + MONT_MAC(t0, t1, t2, u[15], m[15]); + MONT_MAC(v0, v1, v2, u[16], m[14]); + MONT_MAC(w0, w1, w2, u[17], m[13]); + MONT_MAC(t0, t1, t2, u[18], m[12]); + MONT_MAC(v0, v1, v2, u[19], m[11]); + MONT_MAC(w0, w1, w2, u[20], m[10]); + MONT_MAC(t0, t1, t2, u[21], m[9]); + MONT_MAC(v0, v1, v2, u[22], m[8]); + MONT_MAC(w0, w1, w2, u[23], m[7]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[6] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 31 */ + MONT_MUL(s0, s1, s2, a[8], a[23]); + MONT_MUL(c0, c1, c2, a[9], a[22]); + MONT_MAC(s0, s1, s2, a[10], a[21]); + MONT_MAC(c0, c1, c2, a[11], a[20]); + MONT_MAC(s0, s1, s2, a[12], a[19]); + MONT_MAC(c0, c1, c2, a[13], a[18]); + MONT_MAC(s0, s1, s2, a[14], a[17]); + MONT_MAC(c0, c1, c2, a[15], a[16]); + MONT_MAC(t0, t1, t2, u[8], m[23]); + MONT_MUL(v0, v1, v2, u[9], m[22]); + MONT_MUL(w0, w1, w2, u[10], m[21]); + MONT_MAC(t0, t1, t2, u[11], m[20]); + MONT_MAC(v0, v1, v2, u[12], m[19]); + MONT_MAC(w0, w1, w2, u[13], m[18]); + MONT_MAC(t0, t1, t2, u[14], m[17]); + MONT_MAC(v0, v1, v2, u[15], m[16]); + MONT_MAC(w0, w1, w2, u[16], m[15]); + MONT_MAC(t0, t1, t2, u[17], m[14]); + MONT_MAC(v0, v1, v2, u[18], m[13]); + MONT_MAC(w0, w1, w2, u[19], m[12]); + MONT_MAC(t0, t1, t2, u[20], m[11]); + MONT_MAC(v0, v1, v2, u[21], m[10]); + MONT_MAC(w0, w1, w2, u[22], m[9]); + MONT_MAC(t0, t1, t2, u[23], m[8]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[7] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 32 */ + MONT_MUL(s0, s1, s2, a[9], a[23]); + MONT_MUL(c0, c1, c2, a[10], a[22]); + MONT_MAC(s0, s1, s2, a[11], a[21]); + MONT_MAC(c0, c1, c2, a[12], a[20]); + MONT_MAC(s0, s1, s2, a[13], a[19]); + MONT_MAC(c0, c1, c2, a[14], a[18]); + MONT_MAC(s0, s1, s2, a[15], a[17]); + MONT_MAC(t0, t1, t2, a[16], a[16]); + MONT_MUL(v0, v1, v2, u[9], m[23]); + MONT_MUL(w0, w1, w2, u[10], m[22]); + MONT_MAC(t0, t1, t2, u[11], m[21]); + MONT_MAC(v0, v1, v2, u[12], m[20]); + MONT_MAC(w0, w1, w2, u[13], m[19]); + MONT_MAC(t0, t1, t2, u[14], m[18]); + MONT_MAC(v0, v1, v2, u[15], m[17]); + MONT_MAC(w0, w1, w2, u[16], m[16]); + MONT_MAC(t0, t1, t2, u[17], m[15]); + MONT_MAC(v0, v1, v2, u[18], m[14]); + MONT_MAC(w0, w1, w2, u[19], m[13]); + MONT_MAC(t0, t1, t2, u[20], m[12]); + MONT_MAC(v0, v1, v2, u[21], m[11]); + MONT_MAC(w0, w1, w2, u[22], m[10]); + MONT_MAC(t0, t1, t2, u[23], m[9]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[8] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 33 */ + MONT_MUL(s0, s1, s2, a[10], a[23]); + MONT_MUL(c0, c1, c2, a[11], a[22]); + MONT_MAC(s0, s1, s2, a[12], a[21]); + MONT_MAC(c0, c1, c2, a[13], a[20]); + MONT_MAC(s0, s1, s2, a[14], a[19]); + MONT_MAC(c0, c1, c2, a[15], a[18]); + MONT_MAC(s0, s1, s2, a[16], a[17]); + MONT_MAC(t0, t1, t2, u[10], m[23]); + MONT_MUL(v0, v1, v2, u[11], m[22]); + MONT_MUL(w0, w1, w2, u[12], m[21]); + MONT_MAC(t0, t1, t2, u[13], m[20]); + MONT_MAC(v0, v1, v2, u[14], m[19]); + MONT_MAC(w0, w1, w2, u[15], m[18]); + MONT_MAC(t0, t1, t2, u[16], m[17]); + MONT_MAC(v0, v1, v2, u[17], m[16]); + MONT_MAC(w0, w1, w2, u[18], m[15]); + MONT_MAC(t0, t1, t2, u[19], m[14]); + MONT_MAC(v0, v1, v2, u[20], m[13]); + MONT_MAC(w0, w1, w2, u[21], m[12]); + MONT_MAC(t0, t1, t2, u[22], m[11]); + MONT_MAC(v0, v1, v2, u[23], m[10]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[9] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 34 */ + MONT_MUL(s0, s1, s2, a[11], a[23]); + MONT_MUL(c0, c1, c2, a[12], a[22]); + MONT_MAC(s0, s1, s2, a[13], a[21]); + MONT_MAC(c0, c1, c2, a[14], a[20]); + MONT_MAC(s0, s1, s2, a[15], a[19]); + MONT_MAC(c0, c1, c2, a[16], a[18]); + MONT_MAC(t0, t1, t2, a[17], a[17]); + MONT_MUL(v0, v1, v2, u[11], m[23]); + MONT_MUL(w0, w1, w2, u[12], m[22]); + MONT_MAC(t0, t1, t2, u[13], m[21]); + MONT_MAC(v0, v1, v2, u[14], m[20]); + MONT_MAC(w0, w1, w2, u[15], m[19]); + MONT_MAC(t0, t1, t2, u[16], m[18]); + MONT_MAC(v0, v1, v2, u[17], m[17]); + MONT_MAC(w0, w1, w2, u[18], m[16]); + MONT_MAC(t0, t1, t2, u[19], m[15]); + MONT_MAC(v0, v1, v2, u[20], m[14]); + MONT_MAC(w0, w1, w2, u[21], m[13]); + MONT_MAC(t0, t1, t2, u[22], m[12]); + MONT_MAC(v0, v1, v2, u[23], m[11]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[10] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 35 */ + MONT_MUL(s0, s1, s2, a[12], a[23]); + MONT_MUL(c0, c1, c2, a[13], a[22]); + MONT_MAC(s0, s1, s2, a[14], a[21]); + MONT_MAC(c0, c1, c2, a[15], a[20]); + MONT_MAC(s0, s1, s2, a[16], a[19]); + MONT_MAC(c0, c1, c2, a[17], a[18]); + MONT_MAC(t0, t1, t2, u[12], m[23]); + MONT_MUL(v0, v1, v2, u[13], m[22]); + MONT_MUL(w0, w1, w2, u[14], m[21]); + MONT_MAC(t0, t1, t2, u[15], m[20]); + MONT_MAC(v0, v1, v2, u[16], m[19]); + MONT_MAC(w0, w1, w2, u[17], m[18]); + MONT_MAC(t0, t1, t2, u[18], m[17]); + MONT_MAC(v0, v1, v2, u[19], m[16]); + MONT_MAC(w0, w1, w2, u[20], m[15]); + MONT_MAC(t0, t1, t2, u[21], m[14]); + MONT_MAC(v0, v1, v2, u[22], m[13]); + MONT_MAC(w0, w1, w2, u[23], m[12]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[11] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 36 */ + MONT_MUL(s0, s1, s2, a[13], a[23]); + MONT_MUL(c0, c1, c2, a[14], a[22]); + MONT_MAC(s0, s1, s2, a[15], a[21]); + MONT_MAC(c0, c1, c2, a[16], a[20]); + MONT_MAC(s0, s1, s2, a[17], a[19]); + MONT_MAC(t0, t1, t2, a[18], a[18]); + MONT_MUL(v0, v1, v2, u[13], m[23]); + MONT_MUL(w0, w1, w2, u[14], m[22]); + MONT_MAC(t0, t1, t2, u[15], m[21]); + MONT_MAC(v0, v1, v2, u[16], m[20]); + MONT_MAC(w0, w1, w2, u[17], m[19]); + MONT_MAC(t0, t1, t2, u[18], m[18]); + MONT_MAC(v0, v1, v2, u[19], m[17]); + MONT_MAC(w0, w1, w2, u[20], m[16]); + MONT_MAC(t0, t1, t2, u[21], m[15]); + MONT_MAC(v0, v1, v2, u[22], m[14]); + MONT_MAC(w0, w1, w2, u[23], m[13]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[12] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 37 */ + MONT_MUL(s0, s1, s2, a[14], a[23]); + MONT_MUL(c0, c1, c2, a[15], a[22]); + MONT_MAC(s0, s1, s2, a[16], a[21]); + MONT_MAC(c0, c1, c2, a[17], a[20]); + MONT_MAC(s0, s1, s2, a[18], a[19]); + MONT_MAC(t0, t1, t2, u[14], m[23]); + MONT_MUL(v0, v1, v2, u[15], m[22]); + MONT_MUL(w0, w1, w2, u[16], m[21]); + MONT_MAC(t0, t1, t2, u[17], m[20]); + MONT_MAC(v0, v1, v2, u[18], m[19]); + MONT_MAC(w0, w1, w2, u[19], m[18]); + MONT_MAC(t0, t1, t2, u[20], m[17]); + MONT_MAC(v0, v1, v2, u[21], m[16]); + MONT_MAC(w0, w1, w2, u[22], m[15]); + MONT_MAC(t0, t1, t2, u[23], m[14]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[13] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 38 */ + MONT_MUL(s0, s1, s2, a[15], a[23]); + MONT_MUL(c0, c1, c2, a[16], a[22]); + MONT_MAC(s0, s1, s2, a[17], a[21]); + MONT_MAC(c0, c1, c2, a[18], a[20]); + MONT_MAC(t0, t1, t2, a[19], a[19]); + MONT_MUL(v0, v1, v2, u[15], m[23]); + MONT_MUL(w0, w1, w2, u[16], m[22]); + MONT_MAC(t0, t1, t2, u[17], m[21]); + MONT_MAC(v0, v1, v2, u[18], m[20]); + MONT_MAC(w0, w1, w2, u[19], m[19]); + MONT_MAC(t0, t1, t2, u[20], m[18]); + MONT_MAC(v0, v1, v2, u[21], m[17]); + MONT_MAC(w0, w1, w2, u[22], m[16]); + MONT_MAC(t0, t1, t2, u[23], m[15]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[14] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 39 */ + MONT_MUL(s0, s1, s2, a[16], a[23]); + MONT_MUL(c0, c1, c2, a[17], a[22]); + MONT_MAC(s0, s1, s2, a[18], a[21]); + MONT_MAC(c0, c1, c2, a[19], a[20]); + MONT_MAC(t0, t1, t2, u[16], m[23]); + MONT_MUL(v0, v1, v2, u[17], m[22]); + MONT_MUL(w0, w1, w2, u[18], m[21]); + MONT_MAC(t0, t1, t2, u[19], m[20]); + MONT_MAC(v0, v1, v2, u[20], m[19]); + MONT_MAC(w0, w1, w2, u[21], m[18]); + MONT_MAC(t0, t1, t2, u[22], m[17]); + MONT_MAC(v0, v1, v2, u[23], m[16]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[15] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 40 */ + MONT_MUL(s0, s1, s2, a[17], a[23]); + MONT_MUL(c0, c1, c2, a[18], a[22]); + MONT_MAC(s0, s1, s2, a[19], a[21]); + MONT_MAC(t0, t1, t2, a[20], a[20]); + MONT_MUL(v0, v1, v2, u[17], m[23]); + MONT_MUL(w0, w1, w2, u[18], m[22]); + MONT_MAC(t0, t1, t2, u[19], m[21]); + MONT_MAC(v0, v1, v2, u[20], m[20]); + MONT_MAC(w0, w1, w2, u[21], m[19]); + MONT_MAC(t0, t1, t2, u[22], m[18]); + MONT_MAC(v0, v1, v2, u[23], m[17]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[16] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 41 */ + MONT_MUL(s0, s1, s2, a[18], a[23]); + MONT_MUL(c0, c1, c2, a[19], a[22]); + MONT_MAC(s0, s1, s2, a[20], a[21]); + MONT_MAC(t0, t1, t2, u[18], m[23]); + MONT_MUL(v0, v1, v2, u[19], m[22]); + MONT_MUL(w0, w1, w2, u[20], m[21]); + MONT_MAC(t0, t1, t2, u[21], m[20]); + MONT_MAC(v0, v1, v2, u[22], m[19]); + MONT_MAC(w0, w1, w2, u[23], m[18]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[17] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 42 */ + MONT_MUL(s0, s1, s2, a[19], a[23]); + MONT_MUL(c0, c1, c2, a[20], a[22]); + MONT_MAC(t0, t1, t2, a[21], a[21]); + MONT_MUL(v0, v1, v2, u[19], m[23]); + MONT_MUL(w0, w1, w2, u[20], m[22]); + MONT_MAC(t0, t1, t2, u[21], m[21]); + MONT_MAC(v0, v1, v2, u[22], m[20]); + MONT_MAC(w0, w1, w2, u[23], m[19]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[18] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 43 */ + MONT_MUL(s0, s1, s2, a[20], a[23]); + MONT_MUL(c0, c1, c2, a[21], a[22]); + MONT_MAC(t0, t1, t2, u[20], m[23]); + MONT_MUL(v0, v1, v2, u[21], m[22]); + MONT_MUL(w0, w1, w2, u[22], m[21]); + MONT_MAC(t0, t1, t2, u[23], m[20]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[19] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 44 */ + MONT_MUL(s0, s1, s2, a[21], a[23]); + MONT_MAC(t0, t1, t2, a[22], a[22]); + MONT_MUL(v0, v1, v2, u[21], m[23]); + MONT_MUL(w0, w1, w2, u[22], m[22]); + MONT_MAC(t0, t1, t2, u[23], m[21]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[20] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 45 */ + MONT_MUL(s0, s1, s2, a[22], a[23]); + MONT_MAC(t0, t1, t2, u[22], m[23]); + MONT_MUL(v0, v1, v2, u[23], m[22]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[21] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 46 */ + MONT_MAC(t0, t1, t2, a[23], a[23]); + MONT_MUL(v0, v1, v2, u[23], m[23]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + z[22] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + z[23] = t0; + + /* under 2m, with t1 the bit above n limbs: at most one subtraction */ + borrow = sub_n(r, z, m, 24); + take = t1 | (borrow ^ 1); + select_n(r, r, z, take & 1, 24); +} + +/* r = a * b * R^-1 mod m, for a and b under m; t is 2n limbs of scratch */ +static void crypton_mont_mul32(limb_t *r, const limb_t *a, const limb_t *b, + const limb_t *m, limb_t n0, limb_t *t) +{ + limb_t *u = t, *z = t + 32; + limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take; + + /* column 0 */ + MONT_MAC(t0, t1, t2, a[0], b[0]); + u[0] = t0 * n0; + MONT_MAC(t0, t1, t2, u[0], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 1 */ + MONT_MAC(t0, t1, t2, a[0], b[1]); + MONT_MUL(v0, v1, v2, a[1], b[0]); + MONT_MUL(w0, w1, w2, u[0], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[1] = t0 * n0; + MONT_MAC(t0, t1, t2, u[1], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 2 */ + MONT_MAC(t0, t1, t2, a[0], b[2]); + MONT_MUL(v0, v1, v2, a[1], b[1]); + MONT_MUL(w0, w1, w2, a[2], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[2]); + MONT_MAC(v0, v1, v2, u[1], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[2] = t0 * n0; + MONT_MAC(t0, t1, t2, u[2], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 3 */ + MONT_MAC(t0, t1, t2, a[0], b[3]); + MONT_MUL(v0, v1, v2, a[1], b[2]); + MONT_MUL(w0, w1, w2, a[2], b[1]); + MONT_MAC(t0, t1, t2, a[3], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[3]); + MONT_MAC(w0, w1, w2, u[1], m[2]); + MONT_MAC(t0, t1, t2, u[2], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[3] = t0 * n0; + MONT_MAC(t0, t1, t2, u[3], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 4 */ + MONT_MAC(t0, t1, t2, a[0], b[4]); + MONT_MUL(v0, v1, v2, a[1], b[3]); + MONT_MUL(w0, w1, w2, a[2], b[2]); + MONT_MAC(t0, t1, t2, a[3], b[1]); + MONT_MAC(v0, v1, v2, a[4], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[4]); + MONT_MAC(t0, t1, t2, u[1], m[3]); + MONT_MAC(v0, v1, v2, u[2], m[2]); + MONT_MAC(w0, w1, w2, u[3], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[4] = t0 * n0; + MONT_MAC(t0, t1, t2, u[4], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 5 */ + MONT_MAC(t0, t1, t2, a[0], b[5]); + MONT_MUL(v0, v1, v2, a[1], b[4]); + MONT_MUL(w0, w1, w2, a[2], b[3]); + MONT_MAC(t0, t1, t2, a[3], b[2]); + MONT_MAC(v0, v1, v2, a[4], b[1]); + MONT_MAC(w0, w1, w2, a[5], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[5]); + MONT_MAC(v0, v1, v2, u[1], m[4]); + MONT_MAC(w0, w1, w2, u[2], m[3]); + MONT_MAC(t0, t1, t2, u[3], m[2]); + MONT_MAC(v0, v1, v2, u[4], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[5] = t0 * n0; + MONT_MAC(t0, t1, t2, u[5], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 6 */ + MONT_MAC(t0, t1, t2, a[0], b[6]); + MONT_MUL(v0, v1, v2, a[1], b[5]); + MONT_MUL(w0, w1, w2, a[2], b[4]); + MONT_MAC(t0, t1, t2, a[3], b[3]); + MONT_MAC(v0, v1, v2, a[4], b[2]); + MONT_MAC(w0, w1, w2, a[5], b[1]); + MONT_MAC(t0, t1, t2, a[6], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[6]); + MONT_MAC(w0, w1, w2, u[1], m[5]); + MONT_MAC(t0, t1, t2, u[2], m[4]); + MONT_MAC(v0, v1, v2, u[3], m[3]); + MONT_MAC(w0, w1, w2, u[4], m[2]); + MONT_MAC(t0, t1, t2, u[5], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[6] = t0 * n0; + MONT_MAC(t0, t1, t2, u[6], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 7 */ + MONT_MAC(t0, t1, t2, a[0], b[7]); + MONT_MUL(v0, v1, v2, a[1], b[6]); + MONT_MUL(w0, w1, w2, a[2], b[5]); + MONT_MAC(t0, t1, t2, a[3], b[4]); + MONT_MAC(v0, v1, v2, a[4], b[3]); + MONT_MAC(w0, w1, w2, a[5], b[2]); + MONT_MAC(t0, t1, t2, a[6], b[1]); + MONT_MAC(v0, v1, v2, a[7], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[7]); + MONT_MAC(t0, t1, t2, u[1], m[6]); + MONT_MAC(v0, v1, v2, u[2], m[5]); + MONT_MAC(w0, w1, w2, u[3], m[4]); + MONT_MAC(t0, t1, t2, u[4], m[3]); + MONT_MAC(v0, v1, v2, u[5], m[2]); + MONT_MAC(w0, w1, w2, u[6], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[7] = t0 * n0; + MONT_MAC(t0, t1, t2, u[7], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 8 */ + MONT_MAC(t0, t1, t2, a[0], b[8]); + MONT_MUL(v0, v1, v2, a[1], b[7]); + MONT_MUL(w0, w1, w2, a[2], b[6]); + MONT_MAC(t0, t1, t2, a[3], b[5]); + MONT_MAC(v0, v1, v2, a[4], b[4]); + MONT_MAC(w0, w1, w2, a[5], b[3]); + MONT_MAC(t0, t1, t2, a[6], b[2]); + MONT_MAC(v0, v1, v2, a[7], b[1]); + MONT_MAC(w0, w1, w2, a[8], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[8]); + MONT_MAC(v0, v1, v2, u[1], m[7]); + MONT_MAC(w0, w1, w2, u[2], m[6]); + MONT_MAC(t0, t1, t2, u[3], m[5]); + MONT_MAC(v0, v1, v2, u[4], m[4]); + MONT_MAC(w0, w1, w2, u[5], m[3]); + MONT_MAC(t0, t1, t2, u[6], m[2]); + MONT_MAC(v0, v1, v2, u[7], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[8] = t0 * n0; + MONT_MAC(t0, t1, t2, u[8], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 9 */ + MONT_MAC(t0, t1, t2, a[0], b[9]); + MONT_MUL(v0, v1, v2, a[1], b[8]); + MONT_MUL(w0, w1, w2, a[2], b[7]); + MONT_MAC(t0, t1, t2, a[3], b[6]); + MONT_MAC(v0, v1, v2, a[4], b[5]); + MONT_MAC(w0, w1, w2, a[5], b[4]); + MONT_MAC(t0, t1, t2, a[6], b[3]); + MONT_MAC(v0, v1, v2, a[7], b[2]); + MONT_MAC(w0, w1, w2, a[8], b[1]); + MONT_MAC(t0, t1, t2, a[9], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[9]); + MONT_MAC(w0, w1, w2, u[1], m[8]); + MONT_MAC(t0, t1, t2, u[2], m[7]); + MONT_MAC(v0, v1, v2, u[3], m[6]); + MONT_MAC(w0, w1, w2, u[4], m[5]); + MONT_MAC(t0, t1, t2, u[5], m[4]); + MONT_MAC(v0, v1, v2, u[6], m[3]); + MONT_MAC(w0, w1, w2, u[7], m[2]); + MONT_MAC(t0, t1, t2, u[8], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[9] = t0 * n0; + MONT_MAC(t0, t1, t2, u[9], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 10 */ + MONT_MAC(t0, t1, t2, a[0], b[10]); + MONT_MUL(v0, v1, v2, a[1], b[9]); + MONT_MUL(w0, w1, w2, a[2], b[8]); + MONT_MAC(t0, t1, t2, a[3], b[7]); + MONT_MAC(v0, v1, v2, a[4], b[6]); + MONT_MAC(w0, w1, w2, a[5], b[5]); + MONT_MAC(t0, t1, t2, a[6], b[4]); + MONT_MAC(v0, v1, v2, a[7], b[3]); + MONT_MAC(w0, w1, w2, a[8], b[2]); + MONT_MAC(t0, t1, t2, a[9], b[1]); + MONT_MAC(v0, v1, v2, a[10], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[10]); + MONT_MAC(t0, t1, t2, u[1], m[9]); + MONT_MAC(v0, v1, v2, u[2], m[8]); + MONT_MAC(w0, w1, w2, u[3], m[7]); + MONT_MAC(t0, t1, t2, u[4], m[6]); + MONT_MAC(v0, v1, v2, u[5], m[5]); + MONT_MAC(w0, w1, w2, u[6], m[4]); + MONT_MAC(t0, t1, t2, u[7], m[3]); + MONT_MAC(v0, v1, v2, u[8], m[2]); + MONT_MAC(w0, w1, w2, u[9], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[10] = t0 * n0; + MONT_MAC(t0, t1, t2, u[10], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 11 */ + MONT_MAC(t0, t1, t2, a[0], b[11]); + MONT_MUL(v0, v1, v2, a[1], b[10]); + MONT_MUL(w0, w1, w2, a[2], b[9]); + MONT_MAC(t0, t1, t2, a[3], b[8]); + MONT_MAC(v0, v1, v2, a[4], b[7]); + MONT_MAC(w0, w1, w2, a[5], b[6]); + MONT_MAC(t0, t1, t2, a[6], b[5]); + MONT_MAC(v0, v1, v2, a[7], b[4]); + MONT_MAC(w0, w1, w2, a[8], b[3]); + MONT_MAC(t0, t1, t2, a[9], b[2]); + MONT_MAC(v0, v1, v2, a[10], b[1]); + MONT_MAC(w0, w1, w2, a[11], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[11]); + MONT_MAC(v0, v1, v2, u[1], m[10]); + MONT_MAC(w0, w1, w2, u[2], m[9]); + MONT_MAC(t0, t1, t2, u[3], m[8]); + MONT_MAC(v0, v1, v2, u[4], m[7]); + MONT_MAC(w0, w1, w2, u[5], m[6]); + MONT_MAC(t0, t1, t2, u[6], m[5]); + MONT_MAC(v0, v1, v2, u[7], m[4]); + MONT_MAC(w0, w1, w2, u[8], m[3]); + MONT_MAC(t0, t1, t2, u[9], m[2]); + MONT_MAC(v0, v1, v2, u[10], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[11] = t0 * n0; + MONT_MAC(t0, t1, t2, u[11], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 12 */ + MONT_MAC(t0, t1, t2, a[0], b[12]); + MONT_MUL(v0, v1, v2, a[1], b[11]); + MONT_MUL(w0, w1, w2, a[2], b[10]); + MONT_MAC(t0, t1, t2, a[3], b[9]); + MONT_MAC(v0, v1, v2, a[4], b[8]); + MONT_MAC(w0, w1, w2, a[5], b[7]); + MONT_MAC(t0, t1, t2, a[6], b[6]); + MONT_MAC(v0, v1, v2, a[7], b[5]); + MONT_MAC(w0, w1, w2, a[8], b[4]); + MONT_MAC(t0, t1, t2, a[9], b[3]); + MONT_MAC(v0, v1, v2, a[10], b[2]); + MONT_MAC(w0, w1, w2, a[11], b[1]); + MONT_MAC(t0, t1, t2, a[12], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[12]); + MONT_MAC(w0, w1, w2, u[1], m[11]); + MONT_MAC(t0, t1, t2, u[2], m[10]); + MONT_MAC(v0, v1, v2, u[3], m[9]); + MONT_MAC(w0, w1, w2, u[4], m[8]); + MONT_MAC(t0, t1, t2, u[5], m[7]); + MONT_MAC(v0, v1, v2, u[6], m[6]); + MONT_MAC(w0, w1, w2, u[7], m[5]); + MONT_MAC(t0, t1, t2, u[8], m[4]); + MONT_MAC(v0, v1, v2, u[9], m[3]); + MONT_MAC(w0, w1, w2, u[10], m[2]); + MONT_MAC(t0, t1, t2, u[11], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[12] = t0 * n0; + MONT_MAC(t0, t1, t2, u[12], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 13 */ + MONT_MAC(t0, t1, t2, a[0], b[13]); + MONT_MUL(v0, v1, v2, a[1], b[12]); + MONT_MUL(w0, w1, w2, a[2], b[11]); + MONT_MAC(t0, t1, t2, a[3], b[10]); + MONT_MAC(v0, v1, v2, a[4], b[9]); + MONT_MAC(w0, w1, w2, a[5], b[8]); + MONT_MAC(t0, t1, t2, a[6], b[7]); + MONT_MAC(v0, v1, v2, a[7], b[6]); + MONT_MAC(w0, w1, w2, a[8], b[5]); + MONT_MAC(t0, t1, t2, a[9], b[4]); + MONT_MAC(v0, v1, v2, a[10], b[3]); + MONT_MAC(w0, w1, w2, a[11], b[2]); + MONT_MAC(t0, t1, t2, a[12], b[1]); + MONT_MAC(v0, v1, v2, a[13], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[13]); + MONT_MAC(t0, t1, t2, u[1], m[12]); + MONT_MAC(v0, v1, v2, u[2], m[11]); + MONT_MAC(w0, w1, w2, u[3], m[10]); + MONT_MAC(t0, t1, t2, u[4], m[9]); + MONT_MAC(v0, v1, v2, u[5], m[8]); + MONT_MAC(w0, w1, w2, u[6], m[7]); + MONT_MAC(t0, t1, t2, u[7], m[6]); + MONT_MAC(v0, v1, v2, u[8], m[5]); + MONT_MAC(w0, w1, w2, u[9], m[4]); + MONT_MAC(t0, t1, t2, u[10], m[3]); + MONT_MAC(v0, v1, v2, u[11], m[2]); + MONT_MAC(w0, w1, w2, u[12], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[13] = t0 * n0; + MONT_MAC(t0, t1, t2, u[13], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 14 */ + MONT_MAC(t0, t1, t2, a[0], b[14]); + MONT_MUL(v0, v1, v2, a[1], b[13]); + MONT_MUL(w0, w1, w2, a[2], b[12]); + MONT_MAC(t0, t1, t2, a[3], b[11]); + MONT_MAC(v0, v1, v2, a[4], b[10]); + MONT_MAC(w0, w1, w2, a[5], b[9]); + MONT_MAC(t0, t1, t2, a[6], b[8]); + MONT_MAC(v0, v1, v2, a[7], b[7]); + MONT_MAC(w0, w1, w2, a[8], b[6]); + MONT_MAC(t0, t1, t2, a[9], b[5]); + MONT_MAC(v0, v1, v2, a[10], b[4]); + MONT_MAC(w0, w1, w2, a[11], b[3]); + MONT_MAC(t0, t1, t2, a[12], b[2]); + MONT_MAC(v0, v1, v2, a[13], b[1]); + MONT_MAC(w0, w1, w2, a[14], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[14]); + MONT_MAC(v0, v1, v2, u[1], m[13]); + MONT_MAC(w0, w1, w2, u[2], m[12]); + MONT_MAC(t0, t1, t2, u[3], m[11]); + MONT_MAC(v0, v1, v2, u[4], m[10]); + MONT_MAC(w0, w1, w2, u[5], m[9]); + MONT_MAC(t0, t1, t2, u[6], m[8]); + MONT_MAC(v0, v1, v2, u[7], m[7]); + MONT_MAC(w0, w1, w2, u[8], m[6]); + MONT_MAC(t0, t1, t2, u[9], m[5]); + MONT_MAC(v0, v1, v2, u[10], m[4]); + MONT_MAC(w0, w1, w2, u[11], m[3]); + MONT_MAC(t0, t1, t2, u[12], m[2]); + MONT_MAC(v0, v1, v2, u[13], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[14] = t0 * n0; + MONT_MAC(t0, t1, t2, u[14], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 15 */ + MONT_MAC(t0, t1, t2, a[0], b[15]); + MONT_MUL(v0, v1, v2, a[1], b[14]); + MONT_MUL(w0, w1, w2, a[2], b[13]); + MONT_MAC(t0, t1, t2, a[3], b[12]); + MONT_MAC(v0, v1, v2, a[4], b[11]); + MONT_MAC(w0, w1, w2, a[5], b[10]); + MONT_MAC(t0, t1, t2, a[6], b[9]); + MONT_MAC(v0, v1, v2, a[7], b[8]); + MONT_MAC(w0, w1, w2, a[8], b[7]); + MONT_MAC(t0, t1, t2, a[9], b[6]); + MONT_MAC(v0, v1, v2, a[10], b[5]); + MONT_MAC(w0, w1, w2, a[11], b[4]); + MONT_MAC(t0, t1, t2, a[12], b[3]); + MONT_MAC(v0, v1, v2, a[13], b[2]); + MONT_MAC(w0, w1, w2, a[14], b[1]); + MONT_MAC(t0, t1, t2, a[15], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[15]); + MONT_MAC(w0, w1, w2, u[1], m[14]); + MONT_MAC(t0, t1, t2, u[2], m[13]); + MONT_MAC(v0, v1, v2, u[3], m[12]); + MONT_MAC(w0, w1, w2, u[4], m[11]); + MONT_MAC(t0, t1, t2, u[5], m[10]); + MONT_MAC(v0, v1, v2, u[6], m[9]); + MONT_MAC(w0, w1, w2, u[7], m[8]); + MONT_MAC(t0, t1, t2, u[8], m[7]); + MONT_MAC(v0, v1, v2, u[9], m[6]); + MONT_MAC(w0, w1, w2, u[10], m[5]); + MONT_MAC(t0, t1, t2, u[11], m[4]); + MONT_MAC(v0, v1, v2, u[12], m[3]); + MONT_MAC(w0, w1, w2, u[13], m[2]); + MONT_MAC(t0, t1, t2, u[14], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[15] = t0 * n0; + MONT_MAC(t0, t1, t2, u[15], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 16 */ + MONT_MAC(t0, t1, t2, a[0], b[16]); + MONT_MUL(v0, v1, v2, a[1], b[15]); + MONT_MUL(w0, w1, w2, a[2], b[14]); + MONT_MAC(t0, t1, t2, a[3], b[13]); + MONT_MAC(v0, v1, v2, a[4], b[12]); + MONT_MAC(w0, w1, w2, a[5], b[11]); + MONT_MAC(t0, t1, t2, a[6], b[10]); + MONT_MAC(v0, v1, v2, a[7], b[9]); + MONT_MAC(w0, w1, w2, a[8], b[8]); + MONT_MAC(t0, t1, t2, a[9], b[7]); + MONT_MAC(v0, v1, v2, a[10], b[6]); + MONT_MAC(w0, w1, w2, a[11], b[5]); + MONT_MAC(t0, t1, t2, a[12], b[4]); + MONT_MAC(v0, v1, v2, a[13], b[3]); + MONT_MAC(w0, w1, w2, a[14], b[2]); + MONT_MAC(t0, t1, t2, a[15], b[1]); + MONT_MAC(v0, v1, v2, a[16], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[16]); + MONT_MAC(t0, t1, t2, u[1], m[15]); + MONT_MAC(v0, v1, v2, u[2], m[14]); + MONT_MAC(w0, w1, w2, u[3], m[13]); + MONT_MAC(t0, t1, t2, u[4], m[12]); + MONT_MAC(v0, v1, v2, u[5], m[11]); + MONT_MAC(w0, w1, w2, u[6], m[10]); + MONT_MAC(t0, t1, t2, u[7], m[9]); + MONT_MAC(v0, v1, v2, u[8], m[8]); + MONT_MAC(w0, w1, w2, u[9], m[7]); + MONT_MAC(t0, t1, t2, u[10], m[6]); + MONT_MAC(v0, v1, v2, u[11], m[5]); + MONT_MAC(w0, w1, w2, u[12], m[4]); + MONT_MAC(t0, t1, t2, u[13], m[3]); + MONT_MAC(v0, v1, v2, u[14], m[2]); + MONT_MAC(w0, w1, w2, u[15], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[16] = t0 * n0; + MONT_MAC(t0, t1, t2, u[16], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 17 */ + MONT_MAC(t0, t1, t2, a[0], b[17]); + MONT_MUL(v0, v1, v2, a[1], b[16]); + MONT_MUL(w0, w1, w2, a[2], b[15]); + MONT_MAC(t0, t1, t2, a[3], b[14]); + MONT_MAC(v0, v1, v2, a[4], b[13]); + MONT_MAC(w0, w1, w2, a[5], b[12]); + MONT_MAC(t0, t1, t2, a[6], b[11]); + MONT_MAC(v0, v1, v2, a[7], b[10]); + MONT_MAC(w0, w1, w2, a[8], b[9]); + MONT_MAC(t0, t1, t2, a[9], b[8]); + MONT_MAC(v0, v1, v2, a[10], b[7]); + MONT_MAC(w0, w1, w2, a[11], b[6]); + MONT_MAC(t0, t1, t2, a[12], b[5]); + MONT_MAC(v0, v1, v2, a[13], b[4]); + MONT_MAC(w0, w1, w2, a[14], b[3]); + MONT_MAC(t0, t1, t2, a[15], b[2]); + MONT_MAC(v0, v1, v2, a[16], b[1]); + MONT_MAC(w0, w1, w2, a[17], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[17]); + MONT_MAC(v0, v1, v2, u[1], m[16]); + MONT_MAC(w0, w1, w2, u[2], m[15]); + MONT_MAC(t0, t1, t2, u[3], m[14]); + MONT_MAC(v0, v1, v2, u[4], m[13]); + MONT_MAC(w0, w1, w2, u[5], m[12]); + MONT_MAC(t0, t1, t2, u[6], m[11]); + MONT_MAC(v0, v1, v2, u[7], m[10]); + MONT_MAC(w0, w1, w2, u[8], m[9]); + MONT_MAC(t0, t1, t2, u[9], m[8]); + MONT_MAC(v0, v1, v2, u[10], m[7]); + MONT_MAC(w0, w1, w2, u[11], m[6]); + MONT_MAC(t0, t1, t2, u[12], m[5]); + MONT_MAC(v0, v1, v2, u[13], m[4]); + MONT_MAC(w0, w1, w2, u[14], m[3]); + MONT_MAC(t0, t1, t2, u[15], m[2]); + MONT_MAC(v0, v1, v2, u[16], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[17] = t0 * n0; + MONT_MAC(t0, t1, t2, u[17], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 18 */ + MONT_MAC(t0, t1, t2, a[0], b[18]); + MONT_MUL(v0, v1, v2, a[1], b[17]); + MONT_MUL(w0, w1, w2, a[2], b[16]); + MONT_MAC(t0, t1, t2, a[3], b[15]); + MONT_MAC(v0, v1, v2, a[4], b[14]); + MONT_MAC(w0, w1, w2, a[5], b[13]); + MONT_MAC(t0, t1, t2, a[6], b[12]); + MONT_MAC(v0, v1, v2, a[7], b[11]); + MONT_MAC(w0, w1, w2, a[8], b[10]); + MONT_MAC(t0, t1, t2, a[9], b[9]); + MONT_MAC(v0, v1, v2, a[10], b[8]); + MONT_MAC(w0, w1, w2, a[11], b[7]); + MONT_MAC(t0, t1, t2, a[12], b[6]); + MONT_MAC(v0, v1, v2, a[13], b[5]); + MONT_MAC(w0, w1, w2, a[14], b[4]); + MONT_MAC(t0, t1, t2, a[15], b[3]); + MONT_MAC(v0, v1, v2, a[16], b[2]); + MONT_MAC(w0, w1, w2, a[17], b[1]); + MONT_MAC(t0, t1, t2, a[18], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[18]); + MONT_MAC(w0, w1, w2, u[1], m[17]); + MONT_MAC(t0, t1, t2, u[2], m[16]); + MONT_MAC(v0, v1, v2, u[3], m[15]); + MONT_MAC(w0, w1, w2, u[4], m[14]); + MONT_MAC(t0, t1, t2, u[5], m[13]); + MONT_MAC(v0, v1, v2, u[6], m[12]); + MONT_MAC(w0, w1, w2, u[7], m[11]); + MONT_MAC(t0, t1, t2, u[8], m[10]); + MONT_MAC(v0, v1, v2, u[9], m[9]); + MONT_MAC(w0, w1, w2, u[10], m[8]); + MONT_MAC(t0, t1, t2, u[11], m[7]); + MONT_MAC(v0, v1, v2, u[12], m[6]); + MONT_MAC(w0, w1, w2, u[13], m[5]); + MONT_MAC(t0, t1, t2, u[14], m[4]); + MONT_MAC(v0, v1, v2, u[15], m[3]); + MONT_MAC(w0, w1, w2, u[16], m[2]); + MONT_MAC(t0, t1, t2, u[17], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[18] = t0 * n0; + MONT_MAC(t0, t1, t2, u[18], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 19 */ + MONT_MAC(t0, t1, t2, a[0], b[19]); + MONT_MUL(v0, v1, v2, a[1], b[18]); + MONT_MUL(w0, w1, w2, a[2], b[17]); + MONT_MAC(t0, t1, t2, a[3], b[16]); + MONT_MAC(v0, v1, v2, a[4], b[15]); + MONT_MAC(w0, w1, w2, a[5], b[14]); + MONT_MAC(t0, t1, t2, a[6], b[13]); + MONT_MAC(v0, v1, v2, a[7], b[12]); + MONT_MAC(w0, w1, w2, a[8], b[11]); + MONT_MAC(t0, t1, t2, a[9], b[10]); + MONT_MAC(v0, v1, v2, a[10], b[9]); + MONT_MAC(w0, w1, w2, a[11], b[8]); + MONT_MAC(t0, t1, t2, a[12], b[7]); + MONT_MAC(v0, v1, v2, a[13], b[6]); + MONT_MAC(w0, w1, w2, a[14], b[5]); + MONT_MAC(t0, t1, t2, a[15], b[4]); + MONT_MAC(v0, v1, v2, a[16], b[3]); + MONT_MAC(w0, w1, w2, a[17], b[2]); + MONT_MAC(t0, t1, t2, a[18], b[1]); + MONT_MAC(v0, v1, v2, a[19], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[19]); + MONT_MAC(t0, t1, t2, u[1], m[18]); + MONT_MAC(v0, v1, v2, u[2], m[17]); + MONT_MAC(w0, w1, w2, u[3], m[16]); + MONT_MAC(t0, t1, t2, u[4], m[15]); + MONT_MAC(v0, v1, v2, u[5], m[14]); + MONT_MAC(w0, w1, w2, u[6], m[13]); + MONT_MAC(t0, t1, t2, u[7], m[12]); + MONT_MAC(v0, v1, v2, u[8], m[11]); + MONT_MAC(w0, w1, w2, u[9], m[10]); + MONT_MAC(t0, t1, t2, u[10], m[9]); + MONT_MAC(v0, v1, v2, u[11], m[8]); + MONT_MAC(w0, w1, w2, u[12], m[7]); + MONT_MAC(t0, t1, t2, u[13], m[6]); + MONT_MAC(v0, v1, v2, u[14], m[5]); + MONT_MAC(w0, w1, w2, u[15], m[4]); + MONT_MAC(t0, t1, t2, u[16], m[3]); + MONT_MAC(v0, v1, v2, u[17], m[2]); + MONT_MAC(w0, w1, w2, u[18], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[19] = t0 * n0; + MONT_MAC(t0, t1, t2, u[19], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 20 */ + MONT_MAC(t0, t1, t2, a[0], b[20]); + MONT_MUL(v0, v1, v2, a[1], b[19]); + MONT_MUL(w0, w1, w2, a[2], b[18]); + MONT_MAC(t0, t1, t2, a[3], b[17]); + MONT_MAC(v0, v1, v2, a[4], b[16]); + MONT_MAC(w0, w1, w2, a[5], b[15]); + MONT_MAC(t0, t1, t2, a[6], b[14]); + MONT_MAC(v0, v1, v2, a[7], b[13]); + MONT_MAC(w0, w1, w2, a[8], b[12]); + MONT_MAC(t0, t1, t2, a[9], b[11]); + MONT_MAC(v0, v1, v2, a[10], b[10]); + MONT_MAC(w0, w1, w2, a[11], b[9]); + MONT_MAC(t0, t1, t2, a[12], b[8]); + MONT_MAC(v0, v1, v2, a[13], b[7]); + MONT_MAC(w0, w1, w2, a[14], b[6]); + MONT_MAC(t0, t1, t2, a[15], b[5]); + MONT_MAC(v0, v1, v2, a[16], b[4]); + MONT_MAC(w0, w1, w2, a[17], b[3]); + MONT_MAC(t0, t1, t2, a[18], b[2]); + MONT_MAC(v0, v1, v2, a[19], b[1]); + MONT_MAC(w0, w1, w2, a[20], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[20]); + MONT_MAC(v0, v1, v2, u[1], m[19]); + MONT_MAC(w0, w1, w2, u[2], m[18]); + MONT_MAC(t0, t1, t2, u[3], m[17]); + MONT_MAC(v0, v1, v2, u[4], m[16]); + MONT_MAC(w0, w1, w2, u[5], m[15]); + MONT_MAC(t0, t1, t2, u[6], m[14]); + MONT_MAC(v0, v1, v2, u[7], m[13]); + MONT_MAC(w0, w1, w2, u[8], m[12]); + MONT_MAC(t0, t1, t2, u[9], m[11]); + MONT_MAC(v0, v1, v2, u[10], m[10]); + MONT_MAC(w0, w1, w2, u[11], m[9]); + MONT_MAC(t0, t1, t2, u[12], m[8]); + MONT_MAC(v0, v1, v2, u[13], m[7]); + MONT_MAC(w0, w1, w2, u[14], m[6]); + MONT_MAC(t0, t1, t2, u[15], m[5]); + MONT_MAC(v0, v1, v2, u[16], m[4]); + MONT_MAC(w0, w1, w2, u[17], m[3]); + MONT_MAC(t0, t1, t2, u[18], m[2]); + MONT_MAC(v0, v1, v2, u[19], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[20] = t0 * n0; + MONT_MAC(t0, t1, t2, u[20], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 21 */ + MONT_MAC(t0, t1, t2, a[0], b[21]); + MONT_MUL(v0, v1, v2, a[1], b[20]); + MONT_MUL(w0, w1, w2, a[2], b[19]); + MONT_MAC(t0, t1, t2, a[3], b[18]); + MONT_MAC(v0, v1, v2, a[4], b[17]); + MONT_MAC(w0, w1, w2, a[5], b[16]); + MONT_MAC(t0, t1, t2, a[6], b[15]); + MONT_MAC(v0, v1, v2, a[7], b[14]); + MONT_MAC(w0, w1, w2, a[8], b[13]); + MONT_MAC(t0, t1, t2, a[9], b[12]); + MONT_MAC(v0, v1, v2, a[10], b[11]); + MONT_MAC(w0, w1, w2, a[11], b[10]); + MONT_MAC(t0, t1, t2, a[12], b[9]); + MONT_MAC(v0, v1, v2, a[13], b[8]); + MONT_MAC(w0, w1, w2, a[14], b[7]); + MONT_MAC(t0, t1, t2, a[15], b[6]); + MONT_MAC(v0, v1, v2, a[16], b[5]); + MONT_MAC(w0, w1, w2, a[17], b[4]); + MONT_MAC(t0, t1, t2, a[18], b[3]); + MONT_MAC(v0, v1, v2, a[19], b[2]); + MONT_MAC(w0, w1, w2, a[20], b[1]); + MONT_MAC(t0, t1, t2, a[21], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[21]); + MONT_MAC(w0, w1, w2, u[1], m[20]); + MONT_MAC(t0, t1, t2, u[2], m[19]); + MONT_MAC(v0, v1, v2, u[3], m[18]); + MONT_MAC(w0, w1, w2, u[4], m[17]); + MONT_MAC(t0, t1, t2, u[5], m[16]); + MONT_MAC(v0, v1, v2, u[6], m[15]); + MONT_MAC(w0, w1, w2, u[7], m[14]); + MONT_MAC(t0, t1, t2, u[8], m[13]); + MONT_MAC(v0, v1, v2, u[9], m[12]); + MONT_MAC(w0, w1, w2, u[10], m[11]); + MONT_MAC(t0, t1, t2, u[11], m[10]); + MONT_MAC(v0, v1, v2, u[12], m[9]); + MONT_MAC(w0, w1, w2, u[13], m[8]); + MONT_MAC(t0, t1, t2, u[14], m[7]); + MONT_MAC(v0, v1, v2, u[15], m[6]); + MONT_MAC(w0, w1, w2, u[16], m[5]); + MONT_MAC(t0, t1, t2, u[17], m[4]); + MONT_MAC(v0, v1, v2, u[18], m[3]); + MONT_MAC(w0, w1, w2, u[19], m[2]); + MONT_MAC(t0, t1, t2, u[20], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[21] = t0 * n0; + MONT_MAC(t0, t1, t2, u[21], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 22 */ + MONT_MAC(t0, t1, t2, a[0], b[22]); + MONT_MUL(v0, v1, v2, a[1], b[21]); + MONT_MUL(w0, w1, w2, a[2], b[20]); + MONT_MAC(t0, t1, t2, a[3], b[19]); + MONT_MAC(v0, v1, v2, a[4], b[18]); + MONT_MAC(w0, w1, w2, a[5], b[17]); + MONT_MAC(t0, t1, t2, a[6], b[16]); + MONT_MAC(v0, v1, v2, a[7], b[15]); + MONT_MAC(w0, w1, w2, a[8], b[14]); + MONT_MAC(t0, t1, t2, a[9], b[13]); + MONT_MAC(v0, v1, v2, a[10], b[12]); + MONT_MAC(w0, w1, w2, a[11], b[11]); + MONT_MAC(t0, t1, t2, a[12], b[10]); + MONT_MAC(v0, v1, v2, a[13], b[9]); + MONT_MAC(w0, w1, w2, a[14], b[8]); + MONT_MAC(t0, t1, t2, a[15], b[7]); + MONT_MAC(v0, v1, v2, a[16], b[6]); + MONT_MAC(w0, w1, w2, a[17], b[5]); + MONT_MAC(t0, t1, t2, a[18], b[4]); + MONT_MAC(v0, v1, v2, a[19], b[3]); + MONT_MAC(w0, w1, w2, a[20], b[2]); + MONT_MAC(t0, t1, t2, a[21], b[1]); + MONT_MAC(v0, v1, v2, a[22], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[22]); + MONT_MAC(t0, t1, t2, u[1], m[21]); + MONT_MAC(v0, v1, v2, u[2], m[20]); + MONT_MAC(w0, w1, w2, u[3], m[19]); + MONT_MAC(t0, t1, t2, u[4], m[18]); + MONT_MAC(v0, v1, v2, u[5], m[17]); + MONT_MAC(w0, w1, w2, u[6], m[16]); + MONT_MAC(t0, t1, t2, u[7], m[15]); + MONT_MAC(v0, v1, v2, u[8], m[14]); + MONT_MAC(w0, w1, w2, u[9], m[13]); + MONT_MAC(t0, t1, t2, u[10], m[12]); + MONT_MAC(v0, v1, v2, u[11], m[11]); + MONT_MAC(w0, w1, w2, u[12], m[10]); + MONT_MAC(t0, t1, t2, u[13], m[9]); + MONT_MAC(v0, v1, v2, u[14], m[8]); + MONT_MAC(w0, w1, w2, u[15], m[7]); + MONT_MAC(t0, t1, t2, u[16], m[6]); + MONT_MAC(v0, v1, v2, u[17], m[5]); + MONT_MAC(w0, w1, w2, u[18], m[4]); + MONT_MAC(t0, t1, t2, u[19], m[3]); + MONT_MAC(v0, v1, v2, u[20], m[2]); + MONT_MAC(w0, w1, w2, u[21], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[22] = t0 * n0; + MONT_MAC(t0, t1, t2, u[22], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 23 */ + MONT_MAC(t0, t1, t2, a[0], b[23]); + MONT_MUL(v0, v1, v2, a[1], b[22]); + MONT_MUL(w0, w1, w2, a[2], b[21]); + MONT_MAC(t0, t1, t2, a[3], b[20]); + MONT_MAC(v0, v1, v2, a[4], b[19]); + MONT_MAC(w0, w1, w2, a[5], b[18]); + MONT_MAC(t0, t1, t2, a[6], b[17]); + MONT_MAC(v0, v1, v2, a[7], b[16]); + MONT_MAC(w0, w1, w2, a[8], b[15]); + MONT_MAC(t0, t1, t2, a[9], b[14]); + MONT_MAC(v0, v1, v2, a[10], b[13]); + MONT_MAC(w0, w1, w2, a[11], b[12]); + MONT_MAC(t0, t1, t2, a[12], b[11]); + MONT_MAC(v0, v1, v2, a[13], b[10]); + MONT_MAC(w0, w1, w2, a[14], b[9]); + MONT_MAC(t0, t1, t2, a[15], b[8]); + MONT_MAC(v0, v1, v2, a[16], b[7]); + MONT_MAC(w0, w1, w2, a[17], b[6]); + MONT_MAC(t0, t1, t2, a[18], b[5]); + MONT_MAC(v0, v1, v2, a[19], b[4]); + MONT_MAC(w0, w1, w2, a[20], b[3]); + MONT_MAC(t0, t1, t2, a[21], b[2]); + MONT_MAC(v0, v1, v2, a[22], b[1]); + MONT_MAC(w0, w1, w2, a[23], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[23]); + MONT_MAC(v0, v1, v2, u[1], m[22]); + MONT_MAC(w0, w1, w2, u[2], m[21]); + MONT_MAC(t0, t1, t2, u[3], m[20]); + MONT_MAC(v0, v1, v2, u[4], m[19]); + MONT_MAC(w0, w1, w2, u[5], m[18]); + MONT_MAC(t0, t1, t2, u[6], m[17]); + MONT_MAC(v0, v1, v2, u[7], m[16]); + MONT_MAC(w0, w1, w2, u[8], m[15]); + MONT_MAC(t0, t1, t2, u[9], m[14]); + MONT_MAC(v0, v1, v2, u[10], m[13]); + MONT_MAC(w0, w1, w2, u[11], m[12]); + MONT_MAC(t0, t1, t2, u[12], m[11]); + MONT_MAC(v0, v1, v2, u[13], m[10]); + MONT_MAC(w0, w1, w2, u[14], m[9]); + MONT_MAC(t0, t1, t2, u[15], m[8]); + MONT_MAC(v0, v1, v2, u[16], m[7]); + MONT_MAC(w0, w1, w2, u[17], m[6]); + MONT_MAC(t0, t1, t2, u[18], m[5]); + MONT_MAC(v0, v1, v2, u[19], m[4]); + MONT_MAC(w0, w1, w2, u[20], m[3]); + MONT_MAC(t0, t1, t2, u[21], m[2]); + MONT_MAC(v0, v1, v2, u[22], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[23] = t0 * n0; + MONT_MAC(t0, t1, t2, u[23], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 24 */ + MONT_MAC(t0, t1, t2, a[0], b[24]); + MONT_MUL(v0, v1, v2, a[1], b[23]); + MONT_MUL(w0, w1, w2, a[2], b[22]); + MONT_MAC(t0, t1, t2, a[3], b[21]); + MONT_MAC(v0, v1, v2, a[4], b[20]); + MONT_MAC(w0, w1, w2, a[5], b[19]); + MONT_MAC(t0, t1, t2, a[6], b[18]); + MONT_MAC(v0, v1, v2, a[7], b[17]); + MONT_MAC(w0, w1, w2, a[8], b[16]); + MONT_MAC(t0, t1, t2, a[9], b[15]); + MONT_MAC(v0, v1, v2, a[10], b[14]); + MONT_MAC(w0, w1, w2, a[11], b[13]); + MONT_MAC(t0, t1, t2, a[12], b[12]); + MONT_MAC(v0, v1, v2, a[13], b[11]); + MONT_MAC(w0, w1, w2, a[14], b[10]); + MONT_MAC(t0, t1, t2, a[15], b[9]); + MONT_MAC(v0, v1, v2, a[16], b[8]); + MONT_MAC(w0, w1, w2, a[17], b[7]); + MONT_MAC(t0, t1, t2, a[18], b[6]); + MONT_MAC(v0, v1, v2, a[19], b[5]); + MONT_MAC(w0, w1, w2, a[20], b[4]); + MONT_MAC(t0, t1, t2, a[21], b[3]); + MONT_MAC(v0, v1, v2, a[22], b[2]); + MONT_MAC(w0, w1, w2, a[23], b[1]); + MONT_MAC(t0, t1, t2, a[24], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[24]); + MONT_MAC(w0, w1, w2, u[1], m[23]); + MONT_MAC(t0, t1, t2, u[2], m[22]); + MONT_MAC(v0, v1, v2, u[3], m[21]); + MONT_MAC(w0, w1, w2, u[4], m[20]); + MONT_MAC(t0, t1, t2, u[5], m[19]); + MONT_MAC(v0, v1, v2, u[6], m[18]); + MONT_MAC(w0, w1, w2, u[7], m[17]); + MONT_MAC(t0, t1, t2, u[8], m[16]); + MONT_MAC(v0, v1, v2, u[9], m[15]); + MONT_MAC(w0, w1, w2, u[10], m[14]); + MONT_MAC(t0, t1, t2, u[11], m[13]); + MONT_MAC(v0, v1, v2, u[12], m[12]); + MONT_MAC(w0, w1, w2, u[13], m[11]); + MONT_MAC(t0, t1, t2, u[14], m[10]); + MONT_MAC(v0, v1, v2, u[15], m[9]); + MONT_MAC(w0, w1, w2, u[16], m[8]); + MONT_MAC(t0, t1, t2, u[17], m[7]); + MONT_MAC(v0, v1, v2, u[18], m[6]); + MONT_MAC(w0, w1, w2, u[19], m[5]); + MONT_MAC(t0, t1, t2, u[20], m[4]); + MONT_MAC(v0, v1, v2, u[21], m[3]); + MONT_MAC(w0, w1, w2, u[22], m[2]); + MONT_MAC(t0, t1, t2, u[23], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[24] = t0 * n0; + MONT_MAC(t0, t1, t2, u[24], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 25 */ + MONT_MAC(t0, t1, t2, a[0], b[25]); + MONT_MUL(v0, v1, v2, a[1], b[24]); + MONT_MUL(w0, w1, w2, a[2], b[23]); + MONT_MAC(t0, t1, t2, a[3], b[22]); + MONT_MAC(v0, v1, v2, a[4], b[21]); + MONT_MAC(w0, w1, w2, a[5], b[20]); + MONT_MAC(t0, t1, t2, a[6], b[19]); + MONT_MAC(v0, v1, v2, a[7], b[18]); + MONT_MAC(w0, w1, w2, a[8], b[17]); + MONT_MAC(t0, t1, t2, a[9], b[16]); + MONT_MAC(v0, v1, v2, a[10], b[15]); + MONT_MAC(w0, w1, w2, a[11], b[14]); + MONT_MAC(t0, t1, t2, a[12], b[13]); + MONT_MAC(v0, v1, v2, a[13], b[12]); + MONT_MAC(w0, w1, w2, a[14], b[11]); + MONT_MAC(t0, t1, t2, a[15], b[10]); + MONT_MAC(v0, v1, v2, a[16], b[9]); + MONT_MAC(w0, w1, w2, a[17], b[8]); + MONT_MAC(t0, t1, t2, a[18], b[7]); + MONT_MAC(v0, v1, v2, a[19], b[6]); + MONT_MAC(w0, w1, w2, a[20], b[5]); + MONT_MAC(t0, t1, t2, a[21], b[4]); + MONT_MAC(v0, v1, v2, a[22], b[3]); + MONT_MAC(w0, w1, w2, a[23], b[2]); + MONT_MAC(t0, t1, t2, a[24], b[1]); + MONT_MAC(v0, v1, v2, a[25], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[25]); + MONT_MAC(t0, t1, t2, u[1], m[24]); + MONT_MAC(v0, v1, v2, u[2], m[23]); + MONT_MAC(w0, w1, w2, u[3], m[22]); + MONT_MAC(t0, t1, t2, u[4], m[21]); + MONT_MAC(v0, v1, v2, u[5], m[20]); + MONT_MAC(w0, w1, w2, u[6], m[19]); + MONT_MAC(t0, t1, t2, u[7], m[18]); + MONT_MAC(v0, v1, v2, u[8], m[17]); + MONT_MAC(w0, w1, w2, u[9], m[16]); + MONT_MAC(t0, t1, t2, u[10], m[15]); + MONT_MAC(v0, v1, v2, u[11], m[14]); + MONT_MAC(w0, w1, w2, u[12], m[13]); + MONT_MAC(t0, t1, t2, u[13], m[12]); + MONT_MAC(v0, v1, v2, u[14], m[11]); + MONT_MAC(w0, w1, w2, u[15], m[10]); + MONT_MAC(t0, t1, t2, u[16], m[9]); + MONT_MAC(v0, v1, v2, u[17], m[8]); + MONT_MAC(w0, w1, w2, u[18], m[7]); + MONT_MAC(t0, t1, t2, u[19], m[6]); + MONT_MAC(v0, v1, v2, u[20], m[5]); + MONT_MAC(w0, w1, w2, u[21], m[4]); + MONT_MAC(t0, t1, t2, u[22], m[3]); + MONT_MAC(v0, v1, v2, u[23], m[2]); + MONT_MAC(w0, w1, w2, u[24], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[25] = t0 * n0; + MONT_MAC(t0, t1, t2, u[25], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 26 */ + MONT_MAC(t0, t1, t2, a[0], b[26]); + MONT_MUL(v0, v1, v2, a[1], b[25]); + MONT_MUL(w0, w1, w2, a[2], b[24]); + MONT_MAC(t0, t1, t2, a[3], b[23]); + MONT_MAC(v0, v1, v2, a[4], b[22]); + MONT_MAC(w0, w1, w2, a[5], b[21]); + MONT_MAC(t0, t1, t2, a[6], b[20]); + MONT_MAC(v0, v1, v2, a[7], b[19]); + MONT_MAC(w0, w1, w2, a[8], b[18]); + MONT_MAC(t0, t1, t2, a[9], b[17]); + MONT_MAC(v0, v1, v2, a[10], b[16]); + MONT_MAC(w0, w1, w2, a[11], b[15]); + MONT_MAC(t0, t1, t2, a[12], b[14]); + MONT_MAC(v0, v1, v2, a[13], b[13]); + MONT_MAC(w0, w1, w2, a[14], b[12]); + MONT_MAC(t0, t1, t2, a[15], b[11]); + MONT_MAC(v0, v1, v2, a[16], b[10]); + MONT_MAC(w0, w1, w2, a[17], b[9]); + MONT_MAC(t0, t1, t2, a[18], b[8]); + MONT_MAC(v0, v1, v2, a[19], b[7]); + MONT_MAC(w0, w1, w2, a[20], b[6]); + MONT_MAC(t0, t1, t2, a[21], b[5]); + MONT_MAC(v0, v1, v2, a[22], b[4]); + MONT_MAC(w0, w1, w2, a[23], b[3]); + MONT_MAC(t0, t1, t2, a[24], b[2]); + MONT_MAC(v0, v1, v2, a[25], b[1]); + MONT_MAC(w0, w1, w2, a[26], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[26]); + MONT_MAC(v0, v1, v2, u[1], m[25]); + MONT_MAC(w0, w1, w2, u[2], m[24]); + MONT_MAC(t0, t1, t2, u[3], m[23]); + MONT_MAC(v0, v1, v2, u[4], m[22]); + MONT_MAC(w0, w1, w2, u[5], m[21]); + MONT_MAC(t0, t1, t2, u[6], m[20]); + MONT_MAC(v0, v1, v2, u[7], m[19]); + MONT_MAC(w0, w1, w2, u[8], m[18]); + MONT_MAC(t0, t1, t2, u[9], m[17]); + MONT_MAC(v0, v1, v2, u[10], m[16]); + MONT_MAC(w0, w1, w2, u[11], m[15]); + MONT_MAC(t0, t1, t2, u[12], m[14]); + MONT_MAC(v0, v1, v2, u[13], m[13]); + MONT_MAC(w0, w1, w2, u[14], m[12]); + MONT_MAC(t0, t1, t2, u[15], m[11]); + MONT_MAC(v0, v1, v2, u[16], m[10]); + MONT_MAC(w0, w1, w2, u[17], m[9]); + MONT_MAC(t0, t1, t2, u[18], m[8]); + MONT_MAC(v0, v1, v2, u[19], m[7]); + MONT_MAC(w0, w1, w2, u[20], m[6]); + MONT_MAC(t0, t1, t2, u[21], m[5]); + MONT_MAC(v0, v1, v2, u[22], m[4]); + MONT_MAC(w0, w1, w2, u[23], m[3]); + MONT_MAC(t0, t1, t2, u[24], m[2]); + MONT_MAC(v0, v1, v2, u[25], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[26] = t0 * n0; + MONT_MAC(t0, t1, t2, u[26], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 27 */ + MONT_MAC(t0, t1, t2, a[0], b[27]); + MONT_MUL(v0, v1, v2, a[1], b[26]); + MONT_MUL(w0, w1, w2, a[2], b[25]); + MONT_MAC(t0, t1, t2, a[3], b[24]); + MONT_MAC(v0, v1, v2, a[4], b[23]); + MONT_MAC(w0, w1, w2, a[5], b[22]); + MONT_MAC(t0, t1, t2, a[6], b[21]); + MONT_MAC(v0, v1, v2, a[7], b[20]); + MONT_MAC(w0, w1, w2, a[8], b[19]); + MONT_MAC(t0, t1, t2, a[9], b[18]); + MONT_MAC(v0, v1, v2, a[10], b[17]); + MONT_MAC(w0, w1, w2, a[11], b[16]); + MONT_MAC(t0, t1, t2, a[12], b[15]); + MONT_MAC(v0, v1, v2, a[13], b[14]); + MONT_MAC(w0, w1, w2, a[14], b[13]); + MONT_MAC(t0, t1, t2, a[15], b[12]); + MONT_MAC(v0, v1, v2, a[16], b[11]); + MONT_MAC(w0, w1, w2, a[17], b[10]); + MONT_MAC(t0, t1, t2, a[18], b[9]); + MONT_MAC(v0, v1, v2, a[19], b[8]); + MONT_MAC(w0, w1, w2, a[20], b[7]); + MONT_MAC(t0, t1, t2, a[21], b[6]); + MONT_MAC(v0, v1, v2, a[22], b[5]); + MONT_MAC(w0, w1, w2, a[23], b[4]); + MONT_MAC(t0, t1, t2, a[24], b[3]); + MONT_MAC(v0, v1, v2, a[25], b[2]); + MONT_MAC(w0, w1, w2, a[26], b[1]); + MONT_MAC(t0, t1, t2, a[27], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[27]); + MONT_MAC(w0, w1, w2, u[1], m[26]); + MONT_MAC(t0, t1, t2, u[2], m[25]); + MONT_MAC(v0, v1, v2, u[3], m[24]); + MONT_MAC(w0, w1, w2, u[4], m[23]); + MONT_MAC(t0, t1, t2, u[5], m[22]); + MONT_MAC(v0, v1, v2, u[6], m[21]); + MONT_MAC(w0, w1, w2, u[7], m[20]); + MONT_MAC(t0, t1, t2, u[8], m[19]); + MONT_MAC(v0, v1, v2, u[9], m[18]); + MONT_MAC(w0, w1, w2, u[10], m[17]); + MONT_MAC(t0, t1, t2, u[11], m[16]); + MONT_MAC(v0, v1, v2, u[12], m[15]); + MONT_MAC(w0, w1, w2, u[13], m[14]); + MONT_MAC(t0, t1, t2, u[14], m[13]); + MONT_MAC(v0, v1, v2, u[15], m[12]); + MONT_MAC(w0, w1, w2, u[16], m[11]); + MONT_MAC(t0, t1, t2, u[17], m[10]); + MONT_MAC(v0, v1, v2, u[18], m[9]); + MONT_MAC(w0, w1, w2, u[19], m[8]); + MONT_MAC(t0, t1, t2, u[20], m[7]); + MONT_MAC(v0, v1, v2, u[21], m[6]); + MONT_MAC(w0, w1, w2, u[22], m[5]); + MONT_MAC(t0, t1, t2, u[23], m[4]); + MONT_MAC(v0, v1, v2, u[24], m[3]); + MONT_MAC(w0, w1, w2, u[25], m[2]); + MONT_MAC(t0, t1, t2, u[26], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[27] = t0 * n0; + MONT_MAC(t0, t1, t2, u[27], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 28 */ + MONT_MAC(t0, t1, t2, a[0], b[28]); + MONT_MUL(v0, v1, v2, a[1], b[27]); + MONT_MUL(w0, w1, w2, a[2], b[26]); + MONT_MAC(t0, t1, t2, a[3], b[25]); + MONT_MAC(v0, v1, v2, a[4], b[24]); + MONT_MAC(w0, w1, w2, a[5], b[23]); + MONT_MAC(t0, t1, t2, a[6], b[22]); + MONT_MAC(v0, v1, v2, a[7], b[21]); + MONT_MAC(w0, w1, w2, a[8], b[20]); + MONT_MAC(t0, t1, t2, a[9], b[19]); + MONT_MAC(v0, v1, v2, a[10], b[18]); + MONT_MAC(w0, w1, w2, a[11], b[17]); + MONT_MAC(t0, t1, t2, a[12], b[16]); + MONT_MAC(v0, v1, v2, a[13], b[15]); + MONT_MAC(w0, w1, w2, a[14], b[14]); + MONT_MAC(t0, t1, t2, a[15], b[13]); + MONT_MAC(v0, v1, v2, a[16], b[12]); + MONT_MAC(w0, w1, w2, a[17], b[11]); + MONT_MAC(t0, t1, t2, a[18], b[10]); + MONT_MAC(v0, v1, v2, a[19], b[9]); + MONT_MAC(w0, w1, w2, a[20], b[8]); + MONT_MAC(t0, t1, t2, a[21], b[7]); + MONT_MAC(v0, v1, v2, a[22], b[6]); + MONT_MAC(w0, w1, w2, a[23], b[5]); + MONT_MAC(t0, t1, t2, a[24], b[4]); + MONT_MAC(v0, v1, v2, a[25], b[3]); + MONT_MAC(w0, w1, w2, a[26], b[2]); + MONT_MAC(t0, t1, t2, a[27], b[1]); + MONT_MAC(v0, v1, v2, a[28], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[28]); + MONT_MAC(t0, t1, t2, u[1], m[27]); + MONT_MAC(v0, v1, v2, u[2], m[26]); + MONT_MAC(w0, w1, w2, u[3], m[25]); + MONT_MAC(t0, t1, t2, u[4], m[24]); + MONT_MAC(v0, v1, v2, u[5], m[23]); + MONT_MAC(w0, w1, w2, u[6], m[22]); + MONT_MAC(t0, t1, t2, u[7], m[21]); + MONT_MAC(v0, v1, v2, u[8], m[20]); + MONT_MAC(w0, w1, w2, u[9], m[19]); + MONT_MAC(t0, t1, t2, u[10], m[18]); + MONT_MAC(v0, v1, v2, u[11], m[17]); + MONT_MAC(w0, w1, w2, u[12], m[16]); + MONT_MAC(t0, t1, t2, u[13], m[15]); + MONT_MAC(v0, v1, v2, u[14], m[14]); + MONT_MAC(w0, w1, w2, u[15], m[13]); + MONT_MAC(t0, t1, t2, u[16], m[12]); + MONT_MAC(v0, v1, v2, u[17], m[11]); + MONT_MAC(w0, w1, w2, u[18], m[10]); + MONT_MAC(t0, t1, t2, u[19], m[9]); + MONT_MAC(v0, v1, v2, u[20], m[8]); + MONT_MAC(w0, w1, w2, u[21], m[7]); + MONT_MAC(t0, t1, t2, u[22], m[6]); + MONT_MAC(v0, v1, v2, u[23], m[5]); + MONT_MAC(w0, w1, w2, u[24], m[4]); + MONT_MAC(t0, t1, t2, u[25], m[3]); + MONT_MAC(v0, v1, v2, u[26], m[2]); + MONT_MAC(w0, w1, w2, u[27], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[28] = t0 * n0; + MONT_MAC(t0, t1, t2, u[28], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 29 */ + MONT_MAC(t0, t1, t2, a[0], b[29]); + MONT_MUL(v0, v1, v2, a[1], b[28]); + MONT_MUL(w0, w1, w2, a[2], b[27]); + MONT_MAC(t0, t1, t2, a[3], b[26]); + MONT_MAC(v0, v1, v2, a[4], b[25]); + MONT_MAC(w0, w1, w2, a[5], b[24]); + MONT_MAC(t0, t1, t2, a[6], b[23]); + MONT_MAC(v0, v1, v2, a[7], b[22]); + MONT_MAC(w0, w1, w2, a[8], b[21]); + MONT_MAC(t0, t1, t2, a[9], b[20]); + MONT_MAC(v0, v1, v2, a[10], b[19]); + MONT_MAC(w0, w1, w2, a[11], b[18]); + MONT_MAC(t0, t1, t2, a[12], b[17]); + MONT_MAC(v0, v1, v2, a[13], b[16]); + MONT_MAC(w0, w1, w2, a[14], b[15]); + MONT_MAC(t0, t1, t2, a[15], b[14]); + MONT_MAC(v0, v1, v2, a[16], b[13]); + MONT_MAC(w0, w1, w2, a[17], b[12]); + MONT_MAC(t0, t1, t2, a[18], b[11]); + MONT_MAC(v0, v1, v2, a[19], b[10]); + MONT_MAC(w0, w1, w2, a[20], b[9]); + MONT_MAC(t0, t1, t2, a[21], b[8]); + MONT_MAC(v0, v1, v2, a[22], b[7]); + MONT_MAC(w0, w1, w2, a[23], b[6]); + MONT_MAC(t0, t1, t2, a[24], b[5]); + MONT_MAC(v0, v1, v2, a[25], b[4]); + MONT_MAC(w0, w1, w2, a[26], b[3]); + MONT_MAC(t0, t1, t2, a[27], b[2]); + MONT_MAC(v0, v1, v2, a[28], b[1]); + MONT_MAC(w0, w1, w2, a[29], b[0]); + MONT_MAC(t0, t1, t2, u[0], m[29]); + MONT_MAC(v0, v1, v2, u[1], m[28]); + MONT_MAC(w0, w1, w2, u[2], m[27]); + MONT_MAC(t0, t1, t2, u[3], m[26]); + MONT_MAC(v0, v1, v2, u[4], m[25]); + MONT_MAC(w0, w1, w2, u[5], m[24]); + MONT_MAC(t0, t1, t2, u[6], m[23]); + MONT_MAC(v0, v1, v2, u[7], m[22]); + MONT_MAC(w0, w1, w2, u[8], m[21]); + MONT_MAC(t0, t1, t2, u[9], m[20]); + MONT_MAC(v0, v1, v2, u[10], m[19]); + MONT_MAC(w0, w1, w2, u[11], m[18]); + MONT_MAC(t0, t1, t2, u[12], m[17]); + MONT_MAC(v0, v1, v2, u[13], m[16]); + MONT_MAC(w0, w1, w2, u[14], m[15]); + MONT_MAC(t0, t1, t2, u[15], m[14]); + MONT_MAC(v0, v1, v2, u[16], m[13]); + MONT_MAC(w0, w1, w2, u[17], m[12]); + MONT_MAC(t0, t1, t2, u[18], m[11]); + MONT_MAC(v0, v1, v2, u[19], m[10]); + MONT_MAC(w0, w1, w2, u[20], m[9]); + MONT_MAC(t0, t1, t2, u[21], m[8]); + MONT_MAC(v0, v1, v2, u[22], m[7]); + MONT_MAC(w0, w1, w2, u[23], m[6]); + MONT_MAC(t0, t1, t2, u[24], m[5]); + MONT_MAC(v0, v1, v2, u[25], m[4]); + MONT_MAC(w0, w1, w2, u[26], m[3]); + MONT_MAC(t0, t1, t2, u[27], m[2]); + MONT_MAC(v0, v1, v2, u[28], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[29] = t0 * n0; + MONT_MAC(t0, t1, t2, u[29], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 30 */ + MONT_MAC(t0, t1, t2, a[0], b[30]); + MONT_MUL(v0, v1, v2, a[1], b[29]); + MONT_MUL(w0, w1, w2, a[2], b[28]); + MONT_MAC(t0, t1, t2, a[3], b[27]); + MONT_MAC(v0, v1, v2, a[4], b[26]); + MONT_MAC(w0, w1, w2, a[5], b[25]); + MONT_MAC(t0, t1, t2, a[6], b[24]); + MONT_MAC(v0, v1, v2, a[7], b[23]); + MONT_MAC(w0, w1, w2, a[8], b[22]); + MONT_MAC(t0, t1, t2, a[9], b[21]); + MONT_MAC(v0, v1, v2, a[10], b[20]); + MONT_MAC(w0, w1, w2, a[11], b[19]); + MONT_MAC(t0, t1, t2, a[12], b[18]); + MONT_MAC(v0, v1, v2, a[13], b[17]); + MONT_MAC(w0, w1, w2, a[14], b[16]); + MONT_MAC(t0, t1, t2, a[15], b[15]); + MONT_MAC(v0, v1, v2, a[16], b[14]); + MONT_MAC(w0, w1, w2, a[17], b[13]); + MONT_MAC(t0, t1, t2, a[18], b[12]); + MONT_MAC(v0, v1, v2, a[19], b[11]); + MONT_MAC(w0, w1, w2, a[20], b[10]); + MONT_MAC(t0, t1, t2, a[21], b[9]); + MONT_MAC(v0, v1, v2, a[22], b[8]); + MONT_MAC(w0, w1, w2, a[23], b[7]); + MONT_MAC(t0, t1, t2, a[24], b[6]); + MONT_MAC(v0, v1, v2, a[25], b[5]); + MONT_MAC(w0, w1, w2, a[26], b[4]); + MONT_MAC(t0, t1, t2, a[27], b[3]); + MONT_MAC(v0, v1, v2, a[28], b[2]); + MONT_MAC(w0, w1, w2, a[29], b[1]); + MONT_MAC(t0, t1, t2, a[30], b[0]); + MONT_MAC(v0, v1, v2, u[0], m[30]); + MONT_MAC(w0, w1, w2, u[1], m[29]); + MONT_MAC(t0, t1, t2, u[2], m[28]); + MONT_MAC(v0, v1, v2, u[3], m[27]); + MONT_MAC(w0, w1, w2, u[4], m[26]); + MONT_MAC(t0, t1, t2, u[5], m[25]); + MONT_MAC(v0, v1, v2, u[6], m[24]); + MONT_MAC(w0, w1, w2, u[7], m[23]); + MONT_MAC(t0, t1, t2, u[8], m[22]); + MONT_MAC(v0, v1, v2, u[9], m[21]); + MONT_MAC(w0, w1, w2, u[10], m[20]); + MONT_MAC(t0, t1, t2, u[11], m[19]); + MONT_MAC(v0, v1, v2, u[12], m[18]); + MONT_MAC(w0, w1, w2, u[13], m[17]); + MONT_MAC(t0, t1, t2, u[14], m[16]); + MONT_MAC(v0, v1, v2, u[15], m[15]); + MONT_MAC(w0, w1, w2, u[16], m[14]); + MONT_MAC(t0, t1, t2, u[17], m[13]); + MONT_MAC(v0, v1, v2, u[18], m[12]); + MONT_MAC(w0, w1, w2, u[19], m[11]); + MONT_MAC(t0, t1, t2, u[20], m[10]); + MONT_MAC(v0, v1, v2, u[21], m[9]); + MONT_MAC(w0, w1, w2, u[22], m[8]); + MONT_MAC(t0, t1, t2, u[23], m[7]); + MONT_MAC(v0, v1, v2, u[24], m[6]); + MONT_MAC(w0, w1, w2, u[25], m[5]); + MONT_MAC(t0, t1, t2, u[26], m[4]); + MONT_MAC(v0, v1, v2, u[27], m[3]); + MONT_MAC(w0, w1, w2, u[28], m[2]); + MONT_MAC(t0, t1, t2, u[29], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[30] = t0 * n0; + MONT_MAC(t0, t1, t2, u[30], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 31 */ + MONT_MAC(t0, t1, t2, a[0], b[31]); + MONT_MUL(v0, v1, v2, a[1], b[30]); + MONT_MUL(w0, w1, w2, a[2], b[29]); + MONT_MAC(t0, t1, t2, a[3], b[28]); + MONT_MAC(v0, v1, v2, a[4], b[27]); + MONT_MAC(w0, w1, w2, a[5], b[26]); + MONT_MAC(t0, t1, t2, a[6], b[25]); + MONT_MAC(v0, v1, v2, a[7], b[24]); + MONT_MAC(w0, w1, w2, a[8], b[23]); + MONT_MAC(t0, t1, t2, a[9], b[22]); + MONT_MAC(v0, v1, v2, a[10], b[21]); + MONT_MAC(w0, w1, w2, a[11], b[20]); + MONT_MAC(t0, t1, t2, a[12], b[19]); + MONT_MAC(v0, v1, v2, a[13], b[18]); + MONT_MAC(w0, w1, w2, a[14], b[17]); + MONT_MAC(t0, t1, t2, a[15], b[16]); + MONT_MAC(v0, v1, v2, a[16], b[15]); + MONT_MAC(w0, w1, w2, a[17], b[14]); + MONT_MAC(t0, t1, t2, a[18], b[13]); + MONT_MAC(v0, v1, v2, a[19], b[12]); + MONT_MAC(w0, w1, w2, a[20], b[11]); + MONT_MAC(t0, t1, t2, a[21], b[10]); + MONT_MAC(v0, v1, v2, a[22], b[9]); + MONT_MAC(w0, w1, w2, a[23], b[8]); + MONT_MAC(t0, t1, t2, a[24], b[7]); + MONT_MAC(v0, v1, v2, a[25], b[6]); + MONT_MAC(w0, w1, w2, a[26], b[5]); + MONT_MAC(t0, t1, t2, a[27], b[4]); + MONT_MAC(v0, v1, v2, a[28], b[3]); + MONT_MAC(w0, w1, w2, a[29], b[2]); + MONT_MAC(t0, t1, t2, a[30], b[1]); + MONT_MAC(v0, v1, v2, a[31], b[0]); + MONT_MAC(w0, w1, w2, u[0], m[31]); + MONT_MAC(t0, t1, t2, u[1], m[30]); + MONT_MAC(v0, v1, v2, u[2], m[29]); + MONT_MAC(w0, w1, w2, u[3], m[28]); + MONT_MAC(t0, t1, t2, u[4], m[27]); + MONT_MAC(v0, v1, v2, u[5], m[26]); + MONT_MAC(w0, w1, w2, u[6], m[25]); + MONT_MAC(t0, t1, t2, u[7], m[24]); + MONT_MAC(v0, v1, v2, u[8], m[23]); + MONT_MAC(w0, w1, w2, u[9], m[22]); + MONT_MAC(t0, t1, t2, u[10], m[21]); + MONT_MAC(v0, v1, v2, u[11], m[20]); + MONT_MAC(w0, w1, w2, u[12], m[19]); + MONT_MAC(t0, t1, t2, u[13], m[18]); + MONT_MAC(v0, v1, v2, u[14], m[17]); + MONT_MAC(w0, w1, w2, u[15], m[16]); + MONT_MAC(t0, t1, t2, u[16], m[15]); + MONT_MAC(v0, v1, v2, u[17], m[14]); + MONT_MAC(w0, w1, w2, u[18], m[13]); + MONT_MAC(t0, t1, t2, u[19], m[12]); + MONT_MAC(v0, v1, v2, u[20], m[11]); + MONT_MAC(w0, w1, w2, u[21], m[10]); + MONT_MAC(t0, t1, t2, u[22], m[9]); + MONT_MAC(v0, v1, v2, u[23], m[8]); + MONT_MAC(w0, w1, w2, u[24], m[7]); + MONT_MAC(t0, t1, t2, u[25], m[6]); + MONT_MAC(v0, v1, v2, u[26], m[5]); + MONT_MAC(w0, w1, w2, u[27], m[4]); + MONT_MAC(t0, t1, t2, u[28], m[3]); + MONT_MAC(v0, v1, v2, u[29], m[2]); + MONT_MAC(w0, w1, w2, u[30], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + u[31] = t0 * n0; + MONT_MAC(t0, t1, t2, u[31], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 32 */ + MONT_MAC(t0, t1, t2, a[1], b[31]); + MONT_MUL(v0, v1, v2, a[2], b[30]); + MONT_MUL(w0, w1, w2, a[3], b[29]); + MONT_MAC(t0, t1, t2, a[4], b[28]); + MONT_MAC(v0, v1, v2, a[5], b[27]); + MONT_MAC(w0, w1, w2, a[6], b[26]); + MONT_MAC(t0, t1, t2, a[7], b[25]); + MONT_MAC(v0, v1, v2, a[8], b[24]); + MONT_MAC(w0, w1, w2, a[9], b[23]); + MONT_MAC(t0, t1, t2, a[10], b[22]); + MONT_MAC(v0, v1, v2, a[11], b[21]); + MONT_MAC(w0, w1, w2, a[12], b[20]); + MONT_MAC(t0, t1, t2, a[13], b[19]); + MONT_MAC(v0, v1, v2, a[14], b[18]); + MONT_MAC(w0, w1, w2, a[15], b[17]); + MONT_MAC(t0, t1, t2, a[16], b[16]); + MONT_MAC(v0, v1, v2, a[17], b[15]); + MONT_MAC(w0, w1, w2, a[18], b[14]); + MONT_MAC(t0, t1, t2, a[19], b[13]); + MONT_MAC(v0, v1, v2, a[20], b[12]); + MONT_MAC(w0, w1, w2, a[21], b[11]); + MONT_MAC(t0, t1, t2, a[22], b[10]); + MONT_MAC(v0, v1, v2, a[23], b[9]); + MONT_MAC(w0, w1, w2, a[24], b[8]); + MONT_MAC(t0, t1, t2, a[25], b[7]); + MONT_MAC(v0, v1, v2, a[26], b[6]); + MONT_MAC(w0, w1, w2, a[27], b[5]); + MONT_MAC(t0, t1, t2, a[28], b[4]); + MONT_MAC(v0, v1, v2, a[29], b[3]); + MONT_MAC(w0, w1, w2, a[30], b[2]); + MONT_MAC(t0, t1, t2, a[31], b[1]); + MONT_MAC(v0, v1, v2, u[1], m[31]); + MONT_MAC(w0, w1, w2, u[2], m[30]); + MONT_MAC(t0, t1, t2, u[3], m[29]); + MONT_MAC(v0, v1, v2, u[4], m[28]); + MONT_MAC(w0, w1, w2, u[5], m[27]); + MONT_MAC(t0, t1, t2, u[6], m[26]); + MONT_MAC(v0, v1, v2, u[7], m[25]); + MONT_MAC(w0, w1, w2, u[8], m[24]); + MONT_MAC(t0, t1, t2, u[9], m[23]); + MONT_MAC(v0, v1, v2, u[10], m[22]); + MONT_MAC(w0, w1, w2, u[11], m[21]); + MONT_MAC(t0, t1, t2, u[12], m[20]); + MONT_MAC(v0, v1, v2, u[13], m[19]); + MONT_MAC(w0, w1, w2, u[14], m[18]); + MONT_MAC(t0, t1, t2, u[15], m[17]); + MONT_MAC(v0, v1, v2, u[16], m[16]); + MONT_MAC(w0, w1, w2, u[17], m[15]); + MONT_MAC(t0, t1, t2, u[18], m[14]); + MONT_MAC(v0, v1, v2, u[19], m[13]); + MONT_MAC(w0, w1, w2, u[20], m[12]); + MONT_MAC(t0, t1, t2, u[21], m[11]); + MONT_MAC(v0, v1, v2, u[22], m[10]); + MONT_MAC(w0, w1, w2, u[23], m[9]); + MONT_MAC(t0, t1, t2, u[24], m[8]); + MONT_MAC(v0, v1, v2, u[25], m[7]); + MONT_MAC(w0, w1, w2, u[26], m[6]); + MONT_MAC(t0, t1, t2, u[27], m[5]); + MONT_MAC(v0, v1, v2, u[28], m[4]); + MONT_MAC(w0, w1, w2, u[29], m[3]); + MONT_MAC(t0, t1, t2, u[30], m[2]); + MONT_MAC(v0, v1, v2, u[31], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[0] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 33 */ + MONT_MAC(t0, t1, t2, a[2], b[31]); + MONT_MUL(v0, v1, v2, a[3], b[30]); + MONT_MUL(w0, w1, w2, a[4], b[29]); + MONT_MAC(t0, t1, t2, a[5], b[28]); + MONT_MAC(v0, v1, v2, a[6], b[27]); + MONT_MAC(w0, w1, w2, a[7], b[26]); + MONT_MAC(t0, t1, t2, a[8], b[25]); + MONT_MAC(v0, v1, v2, a[9], b[24]); + MONT_MAC(w0, w1, w2, a[10], b[23]); + MONT_MAC(t0, t1, t2, a[11], b[22]); + MONT_MAC(v0, v1, v2, a[12], b[21]); + MONT_MAC(w0, w1, w2, a[13], b[20]); + MONT_MAC(t0, t1, t2, a[14], b[19]); + MONT_MAC(v0, v1, v2, a[15], b[18]); + MONT_MAC(w0, w1, w2, a[16], b[17]); + MONT_MAC(t0, t1, t2, a[17], b[16]); + MONT_MAC(v0, v1, v2, a[18], b[15]); + MONT_MAC(w0, w1, w2, a[19], b[14]); + MONT_MAC(t0, t1, t2, a[20], b[13]); + MONT_MAC(v0, v1, v2, a[21], b[12]); + MONT_MAC(w0, w1, w2, a[22], b[11]); + MONT_MAC(t0, t1, t2, a[23], b[10]); + MONT_MAC(v0, v1, v2, a[24], b[9]); + MONT_MAC(w0, w1, w2, a[25], b[8]); + MONT_MAC(t0, t1, t2, a[26], b[7]); + MONT_MAC(v0, v1, v2, a[27], b[6]); + MONT_MAC(w0, w1, w2, a[28], b[5]); + MONT_MAC(t0, t1, t2, a[29], b[4]); + MONT_MAC(v0, v1, v2, a[30], b[3]); + MONT_MAC(w0, w1, w2, a[31], b[2]); + MONT_MAC(t0, t1, t2, u[2], m[31]); + MONT_MAC(v0, v1, v2, u[3], m[30]); + MONT_MAC(w0, w1, w2, u[4], m[29]); + MONT_MAC(t0, t1, t2, u[5], m[28]); + MONT_MAC(v0, v1, v2, u[6], m[27]); + MONT_MAC(w0, w1, w2, u[7], m[26]); + MONT_MAC(t0, t1, t2, u[8], m[25]); + MONT_MAC(v0, v1, v2, u[9], m[24]); + MONT_MAC(w0, w1, w2, u[10], m[23]); + MONT_MAC(t0, t1, t2, u[11], m[22]); + MONT_MAC(v0, v1, v2, u[12], m[21]); + MONT_MAC(w0, w1, w2, u[13], m[20]); + MONT_MAC(t0, t1, t2, u[14], m[19]); + MONT_MAC(v0, v1, v2, u[15], m[18]); + MONT_MAC(w0, w1, w2, u[16], m[17]); + MONT_MAC(t0, t1, t2, u[17], m[16]); + MONT_MAC(v0, v1, v2, u[18], m[15]); + MONT_MAC(w0, w1, w2, u[19], m[14]); + MONT_MAC(t0, t1, t2, u[20], m[13]); + MONT_MAC(v0, v1, v2, u[21], m[12]); + MONT_MAC(w0, w1, w2, u[22], m[11]); + MONT_MAC(t0, t1, t2, u[23], m[10]); + MONT_MAC(v0, v1, v2, u[24], m[9]); + MONT_MAC(w0, w1, w2, u[25], m[8]); + MONT_MAC(t0, t1, t2, u[26], m[7]); + MONT_MAC(v0, v1, v2, u[27], m[6]); + MONT_MAC(w0, w1, w2, u[28], m[5]); + MONT_MAC(t0, t1, t2, u[29], m[4]); + MONT_MAC(v0, v1, v2, u[30], m[3]); + MONT_MAC(w0, w1, w2, u[31], m[2]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[1] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 34 */ + MONT_MAC(t0, t1, t2, a[3], b[31]); + MONT_MUL(v0, v1, v2, a[4], b[30]); + MONT_MUL(w0, w1, w2, a[5], b[29]); + MONT_MAC(t0, t1, t2, a[6], b[28]); + MONT_MAC(v0, v1, v2, a[7], b[27]); + MONT_MAC(w0, w1, w2, a[8], b[26]); + MONT_MAC(t0, t1, t2, a[9], b[25]); + MONT_MAC(v0, v1, v2, a[10], b[24]); + MONT_MAC(w0, w1, w2, a[11], b[23]); + MONT_MAC(t0, t1, t2, a[12], b[22]); + MONT_MAC(v0, v1, v2, a[13], b[21]); + MONT_MAC(w0, w1, w2, a[14], b[20]); + MONT_MAC(t0, t1, t2, a[15], b[19]); + MONT_MAC(v0, v1, v2, a[16], b[18]); + MONT_MAC(w0, w1, w2, a[17], b[17]); + MONT_MAC(t0, t1, t2, a[18], b[16]); + MONT_MAC(v0, v1, v2, a[19], b[15]); + MONT_MAC(w0, w1, w2, a[20], b[14]); + MONT_MAC(t0, t1, t2, a[21], b[13]); + MONT_MAC(v0, v1, v2, a[22], b[12]); + MONT_MAC(w0, w1, w2, a[23], b[11]); + MONT_MAC(t0, t1, t2, a[24], b[10]); + MONT_MAC(v0, v1, v2, a[25], b[9]); + MONT_MAC(w0, w1, w2, a[26], b[8]); + MONT_MAC(t0, t1, t2, a[27], b[7]); + MONT_MAC(v0, v1, v2, a[28], b[6]); + MONT_MAC(w0, w1, w2, a[29], b[5]); + MONT_MAC(t0, t1, t2, a[30], b[4]); + MONT_MAC(v0, v1, v2, a[31], b[3]); + MONT_MAC(w0, w1, w2, u[3], m[31]); + MONT_MAC(t0, t1, t2, u[4], m[30]); + MONT_MAC(v0, v1, v2, u[5], m[29]); + MONT_MAC(w0, w1, w2, u[6], m[28]); + MONT_MAC(t0, t1, t2, u[7], m[27]); + MONT_MAC(v0, v1, v2, u[8], m[26]); + MONT_MAC(w0, w1, w2, u[9], m[25]); + MONT_MAC(t0, t1, t2, u[10], m[24]); + MONT_MAC(v0, v1, v2, u[11], m[23]); + MONT_MAC(w0, w1, w2, u[12], m[22]); + MONT_MAC(t0, t1, t2, u[13], m[21]); + MONT_MAC(v0, v1, v2, u[14], m[20]); + MONT_MAC(w0, w1, w2, u[15], m[19]); + MONT_MAC(t0, t1, t2, u[16], m[18]); + MONT_MAC(v0, v1, v2, u[17], m[17]); + MONT_MAC(w0, w1, w2, u[18], m[16]); + MONT_MAC(t0, t1, t2, u[19], m[15]); + MONT_MAC(v0, v1, v2, u[20], m[14]); + MONT_MAC(w0, w1, w2, u[21], m[13]); + MONT_MAC(t0, t1, t2, u[22], m[12]); + MONT_MAC(v0, v1, v2, u[23], m[11]); + MONT_MAC(w0, w1, w2, u[24], m[10]); + MONT_MAC(t0, t1, t2, u[25], m[9]); + MONT_MAC(v0, v1, v2, u[26], m[8]); + MONT_MAC(w0, w1, w2, u[27], m[7]); + MONT_MAC(t0, t1, t2, u[28], m[6]); + MONT_MAC(v0, v1, v2, u[29], m[5]); + MONT_MAC(w0, w1, w2, u[30], m[4]); + MONT_MAC(t0, t1, t2, u[31], m[3]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[2] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 35 */ + MONT_MAC(t0, t1, t2, a[4], b[31]); + MONT_MUL(v0, v1, v2, a[5], b[30]); + MONT_MUL(w0, w1, w2, a[6], b[29]); + MONT_MAC(t0, t1, t2, a[7], b[28]); + MONT_MAC(v0, v1, v2, a[8], b[27]); + MONT_MAC(w0, w1, w2, a[9], b[26]); + MONT_MAC(t0, t1, t2, a[10], b[25]); + MONT_MAC(v0, v1, v2, a[11], b[24]); + MONT_MAC(w0, w1, w2, a[12], b[23]); + MONT_MAC(t0, t1, t2, a[13], b[22]); + MONT_MAC(v0, v1, v2, a[14], b[21]); + MONT_MAC(w0, w1, w2, a[15], b[20]); + MONT_MAC(t0, t1, t2, a[16], b[19]); + MONT_MAC(v0, v1, v2, a[17], b[18]); + MONT_MAC(w0, w1, w2, a[18], b[17]); + MONT_MAC(t0, t1, t2, a[19], b[16]); + MONT_MAC(v0, v1, v2, a[20], b[15]); + MONT_MAC(w0, w1, w2, a[21], b[14]); + MONT_MAC(t0, t1, t2, a[22], b[13]); + MONT_MAC(v0, v1, v2, a[23], b[12]); + MONT_MAC(w0, w1, w2, a[24], b[11]); + MONT_MAC(t0, t1, t2, a[25], b[10]); + MONT_MAC(v0, v1, v2, a[26], b[9]); + MONT_MAC(w0, w1, w2, a[27], b[8]); + MONT_MAC(t0, t1, t2, a[28], b[7]); + MONT_MAC(v0, v1, v2, a[29], b[6]); + MONT_MAC(w0, w1, w2, a[30], b[5]); + MONT_MAC(t0, t1, t2, a[31], b[4]); + MONT_MAC(v0, v1, v2, u[4], m[31]); + MONT_MAC(w0, w1, w2, u[5], m[30]); + MONT_MAC(t0, t1, t2, u[6], m[29]); + MONT_MAC(v0, v1, v2, u[7], m[28]); + MONT_MAC(w0, w1, w2, u[8], m[27]); + MONT_MAC(t0, t1, t2, u[9], m[26]); + MONT_MAC(v0, v1, v2, u[10], m[25]); + MONT_MAC(w0, w1, w2, u[11], m[24]); + MONT_MAC(t0, t1, t2, u[12], m[23]); + MONT_MAC(v0, v1, v2, u[13], m[22]); + MONT_MAC(w0, w1, w2, u[14], m[21]); + MONT_MAC(t0, t1, t2, u[15], m[20]); + MONT_MAC(v0, v1, v2, u[16], m[19]); + MONT_MAC(w0, w1, w2, u[17], m[18]); + MONT_MAC(t0, t1, t2, u[18], m[17]); + MONT_MAC(v0, v1, v2, u[19], m[16]); + MONT_MAC(w0, w1, w2, u[20], m[15]); + MONT_MAC(t0, t1, t2, u[21], m[14]); + MONT_MAC(v0, v1, v2, u[22], m[13]); + MONT_MAC(w0, w1, w2, u[23], m[12]); + MONT_MAC(t0, t1, t2, u[24], m[11]); + MONT_MAC(v0, v1, v2, u[25], m[10]); + MONT_MAC(w0, w1, w2, u[26], m[9]); + MONT_MAC(t0, t1, t2, u[27], m[8]); + MONT_MAC(v0, v1, v2, u[28], m[7]); + MONT_MAC(w0, w1, w2, u[29], m[6]); + MONT_MAC(t0, t1, t2, u[30], m[5]); + MONT_MAC(v0, v1, v2, u[31], m[4]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[3] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 36 */ + MONT_MAC(t0, t1, t2, a[5], b[31]); + MONT_MUL(v0, v1, v2, a[6], b[30]); + MONT_MUL(w0, w1, w2, a[7], b[29]); + MONT_MAC(t0, t1, t2, a[8], b[28]); + MONT_MAC(v0, v1, v2, a[9], b[27]); + MONT_MAC(w0, w1, w2, a[10], b[26]); + MONT_MAC(t0, t1, t2, a[11], b[25]); + MONT_MAC(v0, v1, v2, a[12], b[24]); + MONT_MAC(w0, w1, w2, a[13], b[23]); + MONT_MAC(t0, t1, t2, a[14], b[22]); + MONT_MAC(v0, v1, v2, a[15], b[21]); + MONT_MAC(w0, w1, w2, a[16], b[20]); + MONT_MAC(t0, t1, t2, a[17], b[19]); + MONT_MAC(v0, v1, v2, a[18], b[18]); + MONT_MAC(w0, w1, w2, a[19], b[17]); + MONT_MAC(t0, t1, t2, a[20], b[16]); + MONT_MAC(v0, v1, v2, a[21], b[15]); + MONT_MAC(w0, w1, w2, a[22], b[14]); + MONT_MAC(t0, t1, t2, a[23], b[13]); + MONT_MAC(v0, v1, v2, a[24], b[12]); + MONT_MAC(w0, w1, w2, a[25], b[11]); + MONT_MAC(t0, t1, t2, a[26], b[10]); + MONT_MAC(v0, v1, v2, a[27], b[9]); + MONT_MAC(w0, w1, w2, a[28], b[8]); + MONT_MAC(t0, t1, t2, a[29], b[7]); + MONT_MAC(v0, v1, v2, a[30], b[6]); + MONT_MAC(w0, w1, w2, a[31], b[5]); + MONT_MAC(t0, t1, t2, u[5], m[31]); + MONT_MAC(v0, v1, v2, u[6], m[30]); + MONT_MAC(w0, w1, w2, u[7], m[29]); + MONT_MAC(t0, t1, t2, u[8], m[28]); + MONT_MAC(v0, v1, v2, u[9], m[27]); + MONT_MAC(w0, w1, w2, u[10], m[26]); + MONT_MAC(t0, t1, t2, u[11], m[25]); + MONT_MAC(v0, v1, v2, u[12], m[24]); + MONT_MAC(w0, w1, w2, u[13], m[23]); + MONT_MAC(t0, t1, t2, u[14], m[22]); + MONT_MAC(v0, v1, v2, u[15], m[21]); + MONT_MAC(w0, w1, w2, u[16], m[20]); + MONT_MAC(t0, t1, t2, u[17], m[19]); + MONT_MAC(v0, v1, v2, u[18], m[18]); + MONT_MAC(w0, w1, w2, u[19], m[17]); + MONT_MAC(t0, t1, t2, u[20], m[16]); + MONT_MAC(v0, v1, v2, u[21], m[15]); + MONT_MAC(w0, w1, w2, u[22], m[14]); + MONT_MAC(t0, t1, t2, u[23], m[13]); + MONT_MAC(v0, v1, v2, u[24], m[12]); + MONT_MAC(w0, w1, w2, u[25], m[11]); + MONT_MAC(t0, t1, t2, u[26], m[10]); + MONT_MAC(v0, v1, v2, u[27], m[9]); + MONT_MAC(w0, w1, w2, u[28], m[8]); + MONT_MAC(t0, t1, t2, u[29], m[7]); + MONT_MAC(v0, v1, v2, u[30], m[6]); + MONT_MAC(w0, w1, w2, u[31], m[5]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[4] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 37 */ + MONT_MAC(t0, t1, t2, a[6], b[31]); + MONT_MUL(v0, v1, v2, a[7], b[30]); + MONT_MUL(w0, w1, w2, a[8], b[29]); + MONT_MAC(t0, t1, t2, a[9], b[28]); + MONT_MAC(v0, v1, v2, a[10], b[27]); + MONT_MAC(w0, w1, w2, a[11], b[26]); + MONT_MAC(t0, t1, t2, a[12], b[25]); + MONT_MAC(v0, v1, v2, a[13], b[24]); + MONT_MAC(w0, w1, w2, a[14], b[23]); + MONT_MAC(t0, t1, t2, a[15], b[22]); + MONT_MAC(v0, v1, v2, a[16], b[21]); + MONT_MAC(w0, w1, w2, a[17], b[20]); + MONT_MAC(t0, t1, t2, a[18], b[19]); + MONT_MAC(v0, v1, v2, a[19], b[18]); + MONT_MAC(w0, w1, w2, a[20], b[17]); + MONT_MAC(t0, t1, t2, a[21], b[16]); + MONT_MAC(v0, v1, v2, a[22], b[15]); + MONT_MAC(w0, w1, w2, a[23], b[14]); + MONT_MAC(t0, t1, t2, a[24], b[13]); + MONT_MAC(v0, v1, v2, a[25], b[12]); + MONT_MAC(w0, w1, w2, a[26], b[11]); + MONT_MAC(t0, t1, t2, a[27], b[10]); + MONT_MAC(v0, v1, v2, a[28], b[9]); + MONT_MAC(w0, w1, w2, a[29], b[8]); + MONT_MAC(t0, t1, t2, a[30], b[7]); + MONT_MAC(v0, v1, v2, a[31], b[6]); + MONT_MAC(w0, w1, w2, u[6], m[31]); + MONT_MAC(t0, t1, t2, u[7], m[30]); + MONT_MAC(v0, v1, v2, u[8], m[29]); + MONT_MAC(w0, w1, w2, u[9], m[28]); + MONT_MAC(t0, t1, t2, u[10], m[27]); + MONT_MAC(v0, v1, v2, u[11], m[26]); + MONT_MAC(w0, w1, w2, u[12], m[25]); + MONT_MAC(t0, t1, t2, u[13], m[24]); + MONT_MAC(v0, v1, v2, u[14], m[23]); + MONT_MAC(w0, w1, w2, u[15], m[22]); + MONT_MAC(t0, t1, t2, u[16], m[21]); + MONT_MAC(v0, v1, v2, u[17], m[20]); + MONT_MAC(w0, w1, w2, u[18], m[19]); + MONT_MAC(t0, t1, t2, u[19], m[18]); + MONT_MAC(v0, v1, v2, u[20], m[17]); + MONT_MAC(w0, w1, w2, u[21], m[16]); + MONT_MAC(t0, t1, t2, u[22], m[15]); + MONT_MAC(v0, v1, v2, u[23], m[14]); + MONT_MAC(w0, w1, w2, u[24], m[13]); + MONT_MAC(t0, t1, t2, u[25], m[12]); + MONT_MAC(v0, v1, v2, u[26], m[11]); + MONT_MAC(w0, w1, w2, u[27], m[10]); + MONT_MAC(t0, t1, t2, u[28], m[9]); + MONT_MAC(v0, v1, v2, u[29], m[8]); + MONT_MAC(w0, w1, w2, u[30], m[7]); + MONT_MAC(t0, t1, t2, u[31], m[6]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[5] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 38 */ + MONT_MAC(t0, t1, t2, a[7], b[31]); + MONT_MUL(v0, v1, v2, a[8], b[30]); + MONT_MUL(w0, w1, w2, a[9], b[29]); + MONT_MAC(t0, t1, t2, a[10], b[28]); + MONT_MAC(v0, v1, v2, a[11], b[27]); + MONT_MAC(w0, w1, w2, a[12], b[26]); + MONT_MAC(t0, t1, t2, a[13], b[25]); + MONT_MAC(v0, v1, v2, a[14], b[24]); + MONT_MAC(w0, w1, w2, a[15], b[23]); + MONT_MAC(t0, t1, t2, a[16], b[22]); + MONT_MAC(v0, v1, v2, a[17], b[21]); + MONT_MAC(w0, w1, w2, a[18], b[20]); + MONT_MAC(t0, t1, t2, a[19], b[19]); + MONT_MAC(v0, v1, v2, a[20], b[18]); + MONT_MAC(w0, w1, w2, a[21], b[17]); + MONT_MAC(t0, t1, t2, a[22], b[16]); + MONT_MAC(v0, v1, v2, a[23], b[15]); + MONT_MAC(w0, w1, w2, a[24], b[14]); + MONT_MAC(t0, t1, t2, a[25], b[13]); + MONT_MAC(v0, v1, v2, a[26], b[12]); + MONT_MAC(w0, w1, w2, a[27], b[11]); + MONT_MAC(t0, t1, t2, a[28], b[10]); + MONT_MAC(v0, v1, v2, a[29], b[9]); + MONT_MAC(w0, w1, w2, a[30], b[8]); + MONT_MAC(t0, t1, t2, a[31], b[7]); + MONT_MAC(v0, v1, v2, u[7], m[31]); + MONT_MAC(w0, w1, w2, u[8], m[30]); + MONT_MAC(t0, t1, t2, u[9], m[29]); + MONT_MAC(v0, v1, v2, u[10], m[28]); + MONT_MAC(w0, w1, w2, u[11], m[27]); + MONT_MAC(t0, t1, t2, u[12], m[26]); + MONT_MAC(v0, v1, v2, u[13], m[25]); + MONT_MAC(w0, w1, w2, u[14], m[24]); + MONT_MAC(t0, t1, t2, u[15], m[23]); + MONT_MAC(v0, v1, v2, u[16], m[22]); + MONT_MAC(w0, w1, w2, u[17], m[21]); + MONT_MAC(t0, t1, t2, u[18], m[20]); + MONT_MAC(v0, v1, v2, u[19], m[19]); + MONT_MAC(w0, w1, w2, u[20], m[18]); + MONT_MAC(t0, t1, t2, u[21], m[17]); + MONT_MAC(v0, v1, v2, u[22], m[16]); + MONT_MAC(w0, w1, w2, u[23], m[15]); + MONT_MAC(t0, t1, t2, u[24], m[14]); + MONT_MAC(v0, v1, v2, u[25], m[13]); + MONT_MAC(w0, w1, w2, u[26], m[12]); + MONT_MAC(t0, t1, t2, u[27], m[11]); + MONT_MAC(v0, v1, v2, u[28], m[10]); + MONT_MAC(w0, w1, w2, u[29], m[9]); + MONT_MAC(t0, t1, t2, u[30], m[8]); + MONT_MAC(v0, v1, v2, u[31], m[7]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[6] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 39 */ + MONT_MAC(t0, t1, t2, a[8], b[31]); + MONT_MUL(v0, v1, v2, a[9], b[30]); + MONT_MUL(w0, w1, w2, a[10], b[29]); + MONT_MAC(t0, t1, t2, a[11], b[28]); + MONT_MAC(v0, v1, v2, a[12], b[27]); + MONT_MAC(w0, w1, w2, a[13], b[26]); + MONT_MAC(t0, t1, t2, a[14], b[25]); + MONT_MAC(v0, v1, v2, a[15], b[24]); + MONT_MAC(w0, w1, w2, a[16], b[23]); + MONT_MAC(t0, t1, t2, a[17], b[22]); + MONT_MAC(v0, v1, v2, a[18], b[21]); + MONT_MAC(w0, w1, w2, a[19], b[20]); + MONT_MAC(t0, t1, t2, a[20], b[19]); + MONT_MAC(v0, v1, v2, a[21], b[18]); + MONT_MAC(w0, w1, w2, a[22], b[17]); + MONT_MAC(t0, t1, t2, a[23], b[16]); + MONT_MAC(v0, v1, v2, a[24], b[15]); + MONT_MAC(w0, w1, w2, a[25], b[14]); + MONT_MAC(t0, t1, t2, a[26], b[13]); + MONT_MAC(v0, v1, v2, a[27], b[12]); + MONT_MAC(w0, w1, w2, a[28], b[11]); + MONT_MAC(t0, t1, t2, a[29], b[10]); + MONT_MAC(v0, v1, v2, a[30], b[9]); + MONT_MAC(w0, w1, w2, a[31], b[8]); + MONT_MAC(t0, t1, t2, u[8], m[31]); + MONT_MAC(v0, v1, v2, u[9], m[30]); + MONT_MAC(w0, w1, w2, u[10], m[29]); + MONT_MAC(t0, t1, t2, u[11], m[28]); + MONT_MAC(v0, v1, v2, u[12], m[27]); + MONT_MAC(w0, w1, w2, u[13], m[26]); + MONT_MAC(t0, t1, t2, u[14], m[25]); + MONT_MAC(v0, v1, v2, u[15], m[24]); + MONT_MAC(w0, w1, w2, u[16], m[23]); + MONT_MAC(t0, t1, t2, u[17], m[22]); + MONT_MAC(v0, v1, v2, u[18], m[21]); + MONT_MAC(w0, w1, w2, u[19], m[20]); + MONT_MAC(t0, t1, t2, u[20], m[19]); + MONT_MAC(v0, v1, v2, u[21], m[18]); + MONT_MAC(w0, w1, w2, u[22], m[17]); + MONT_MAC(t0, t1, t2, u[23], m[16]); + MONT_MAC(v0, v1, v2, u[24], m[15]); + MONT_MAC(w0, w1, w2, u[25], m[14]); + MONT_MAC(t0, t1, t2, u[26], m[13]); + MONT_MAC(v0, v1, v2, u[27], m[12]); + MONT_MAC(w0, w1, w2, u[28], m[11]); + MONT_MAC(t0, t1, t2, u[29], m[10]); + MONT_MAC(v0, v1, v2, u[30], m[9]); + MONT_MAC(w0, w1, w2, u[31], m[8]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[7] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 40 */ + MONT_MAC(t0, t1, t2, a[9], b[31]); + MONT_MUL(v0, v1, v2, a[10], b[30]); + MONT_MUL(w0, w1, w2, a[11], b[29]); + MONT_MAC(t0, t1, t2, a[12], b[28]); + MONT_MAC(v0, v1, v2, a[13], b[27]); + MONT_MAC(w0, w1, w2, a[14], b[26]); + MONT_MAC(t0, t1, t2, a[15], b[25]); + MONT_MAC(v0, v1, v2, a[16], b[24]); + MONT_MAC(w0, w1, w2, a[17], b[23]); + MONT_MAC(t0, t1, t2, a[18], b[22]); + MONT_MAC(v0, v1, v2, a[19], b[21]); + MONT_MAC(w0, w1, w2, a[20], b[20]); + MONT_MAC(t0, t1, t2, a[21], b[19]); + MONT_MAC(v0, v1, v2, a[22], b[18]); + MONT_MAC(w0, w1, w2, a[23], b[17]); + MONT_MAC(t0, t1, t2, a[24], b[16]); + MONT_MAC(v0, v1, v2, a[25], b[15]); + MONT_MAC(w0, w1, w2, a[26], b[14]); + MONT_MAC(t0, t1, t2, a[27], b[13]); + MONT_MAC(v0, v1, v2, a[28], b[12]); + MONT_MAC(w0, w1, w2, a[29], b[11]); + MONT_MAC(t0, t1, t2, a[30], b[10]); + MONT_MAC(v0, v1, v2, a[31], b[9]); + MONT_MAC(w0, w1, w2, u[9], m[31]); + MONT_MAC(t0, t1, t2, u[10], m[30]); + MONT_MAC(v0, v1, v2, u[11], m[29]); + MONT_MAC(w0, w1, w2, u[12], m[28]); + MONT_MAC(t0, t1, t2, u[13], m[27]); + MONT_MAC(v0, v1, v2, u[14], m[26]); + MONT_MAC(w0, w1, w2, u[15], m[25]); + MONT_MAC(t0, t1, t2, u[16], m[24]); + MONT_MAC(v0, v1, v2, u[17], m[23]); + MONT_MAC(w0, w1, w2, u[18], m[22]); + MONT_MAC(t0, t1, t2, u[19], m[21]); + MONT_MAC(v0, v1, v2, u[20], m[20]); + MONT_MAC(w0, w1, w2, u[21], m[19]); + MONT_MAC(t0, t1, t2, u[22], m[18]); + MONT_MAC(v0, v1, v2, u[23], m[17]); + MONT_MAC(w0, w1, w2, u[24], m[16]); + MONT_MAC(t0, t1, t2, u[25], m[15]); + MONT_MAC(v0, v1, v2, u[26], m[14]); + MONT_MAC(w0, w1, w2, u[27], m[13]); + MONT_MAC(t0, t1, t2, u[28], m[12]); + MONT_MAC(v0, v1, v2, u[29], m[11]); + MONT_MAC(w0, w1, w2, u[30], m[10]); + MONT_MAC(t0, t1, t2, u[31], m[9]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[8] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 41 */ + MONT_MAC(t0, t1, t2, a[10], b[31]); + MONT_MUL(v0, v1, v2, a[11], b[30]); + MONT_MUL(w0, w1, w2, a[12], b[29]); + MONT_MAC(t0, t1, t2, a[13], b[28]); + MONT_MAC(v0, v1, v2, a[14], b[27]); + MONT_MAC(w0, w1, w2, a[15], b[26]); + MONT_MAC(t0, t1, t2, a[16], b[25]); + MONT_MAC(v0, v1, v2, a[17], b[24]); + MONT_MAC(w0, w1, w2, a[18], b[23]); + MONT_MAC(t0, t1, t2, a[19], b[22]); + MONT_MAC(v0, v1, v2, a[20], b[21]); + MONT_MAC(w0, w1, w2, a[21], b[20]); + MONT_MAC(t0, t1, t2, a[22], b[19]); + MONT_MAC(v0, v1, v2, a[23], b[18]); + MONT_MAC(w0, w1, w2, a[24], b[17]); + MONT_MAC(t0, t1, t2, a[25], b[16]); + MONT_MAC(v0, v1, v2, a[26], b[15]); + MONT_MAC(w0, w1, w2, a[27], b[14]); + MONT_MAC(t0, t1, t2, a[28], b[13]); + MONT_MAC(v0, v1, v2, a[29], b[12]); + MONT_MAC(w0, w1, w2, a[30], b[11]); + MONT_MAC(t0, t1, t2, a[31], b[10]); + MONT_MAC(v0, v1, v2, u[10], m[31]); + MONT_MAC(w0, w1, w2, u[11], m[30]); + MONT_MAC(t0, t1, t2, u[12], m[29]); + MONT_MAC(v0, v1, v2, u[13], m[28]); + MONT_MAC(w0, w1, w2, u[14], m[27]); + MONT_MAC(t0, t1, t2, u[15], m[26]); + MONT_MAC(v0, v1, v2, u[16], m[25]); + MONT_MAC(w0, w1, w2, u[17], m[24]); + MONT_MAC(t0, t1, t2, u[18], m[23]); + MONT_MAC(v0, v1, v2, u[19], m[22]); + MONT_MAC(w0, w1, w2, u[20], m[21]); + MONT_MAC(t0, t1, t2, u[21], m[20]); + MONT_MAC(v0, v1, v2, u[22], m[19]); + MONT_MAC(w0, w1, w2, u[23], m[18]); + MONT_MAC(t0, t1, t2, u[24], m[17]); + MONT_MAC(v0, v1, v2, u[25], m[16]); + MONT_MAC(w0, w1, w2, u[26], m[15]); + MONT_MAC(t0, t1, t2, u[27], m[14]); + MONT_MAC(v0, v1, v2, u[28], m[13]); + MONT_MAC(w0, w1, w2, u[29], m[12]); + MONT_MAC(t0, t1, t2, u[30], m[11]); + MONT_MAC(v0, v1, v2, u[31], m[10]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[9] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 42 */ + MONT_MAC(t0, t1, t2, a[11], b[31]); + MONT_MUL(v0, v1, v2, a[12], b[30]); + MONT_MUL(w0, w1, w2, a[13], b[29]); + MONT_MAC(t0, t1, t2, a[14], b[28]); + MONT_MAC(v0, v1, v2, a[15], b[27]); + MONT_MAC(w0, w1, w2, a[16], b[26]); + MONT_MAC(t0, t1, t2, a[17], b[25]); + MONT_MAC(v0, v1, v2, a[18], b[24]); + MONT_MAC(w0, w1, w2, a[19], b[23]); + MONT_MAC(t0, t1, t2, a[20], b[22]); + MONT_MAC(v0, v1, v2, a[21], b[21]); + MONT_MAC(w0, w1, w2, a[22], b[20]); + MONT_MAC(t0, t1, t2, a[23], b[19]); + MONT_MAC(v0, v1, v2, a[24], b[18]); + MONT_MAC(w0, w1, w2, a[25], b[17]); + MONT_MAC(t0, t1, t2, a[26], b[16]); + MONT_MAC(v0, v1, v2, a[27], b[15]); + MONT_MAC(w0, w1, w2, a[28], b[14]); + MONT_MAC(t0, t1, t2, a[29], b[13]); + MONT_MAC(v0, v1, v2, a[30], b[12]); + MONT_MAC(w0, w1, w2, a[31], b[11]); + MONT_MAC(t0, t1, t2, u[11], m[31]); + MONT_MAC(v0, v1, v2, u[12], m[30]); + MONT_MAC(w0, w1, w2, u[13], m[29]); + MONT_MAC(t0, t1, t2, u[14], m[28]); + MONT_MAC(v0, v1, v2, u[15], m[27]); + MONT_MAC(w0, w1, w2, u[16], m[26]); + MONT_MAC(t0, t1, t2, u[17], m[25]); + MONT_MAC(v0, v1, v2, u[18], m[24]); + MONT_MAC(w0, w1, w2, u[19], m[23]); + MONT_MAC(t0, t1, t2, u[20], m[22]); + MONT_MAC(v0, v1, v2, u[21], m[21]); + MONT_MAC(w0, w1, w2, u[22], m[20]); + MONT_MAC(t0, t1, t2, u[23], m[19]); + MONT_MAC(v0, v1, v2, u[24], m[18]); + MONT_MAC(w0, w1, w2, u[25], m[17]); + MONT_MAC(t0, t1, t2, u[26], m[16]); + MONT_MAC(v0, v1, v2, u[27], m[15]); + MONT_MAC(w0, w1, w2, u[28], m[14]); + MONT_MAC(t0, t1, t2, u[29], m[13]); + MONT_MAC(v0, v1, v2, u[30], m[12]); + MONT_MAC(w0, w1, w2, u[31], m[11]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[10] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 43 */ + MONT_MAC(t0, t1, t2, a[12], b[31]); + MONT_MUL(v0, v1, v2, a[13], b[30]); + MONT_MUL(w0, w1, w2, a[14], b[29]); + MONT_MAC(t0, t1, t2, a[15], b[28]); + MONT_MAC(v0, v1, v2, a[16], b[27]); + MONT_MAC(w0, w1, w2, a[17], b[26]); + MONT_MAC(t0, t1, t2, a[18], b[25]); + MONT_MAC(v0, v1, v2, a[19], b[24]); + MONT_MAC(w0, w1, w2, a[20], b[23]); + MONT_MAC(t0, t1, t2, a[21], b[22]); + MONT_MAC(v0, v1, v2, a[22], b[21]); + MONT_MAC(w0, w1, w2, a[23], b[20]); + MONT_MAC(t0, t1, t2, a[24], b[19]); + MONT_MAC(v0, v1, v2, a[25], b[18]); + MONT_MAC(w0, w1, w2, a[26], b[17]); + MONT_MAC(t0, t1, t2, a[27], b[16]); + MONT_MAC(v0, v1, v2, a[28], b[15]); + MONT_MAC(w0, w1, w2, a[29], b[14]); + MONT_MAC(t0, t1, t2, a[30], b[13]); + MONT_MAC(v0, v1, v2, a[31], b[12]); + MONT_MAC(w0, w1, w2, u[12], m[31]); + MONT_MAC(t0, t1, t2, u[13], m[30]); + MONT_MAC(v0, v1, v2, u[14], m[29]); + MONT_MAC(w0, w1, w2, u[15], m[28]); + MONT_MAC(t0, t1, t2, u[16], m[27]); + MONT_MAC(v0, v1, v2, u[17], m[26]); + MONT_MAC(w0, w1, w2, u[18], m[25]); + MONT_MAC(t0, t1, t2, u[19], m[24]); + MONT_MAC(v0, v1, v2, u[20], m[23]); + MONT_MAC(w0, w1, w2, u[21], m[22]); + MONT_MAC(t0, t1, t2, u[22], m[21]); + MONT_MAC(v0, v1, v2, u[23], m[20]); + MONT_MAC(w0, w1, w2, u[24], m[19]); + MONT_MAC(t0, t1, t2, u[25], m[18]); + MONT_MAC(v0, v1, v2, u[26], m[17]); + MONT_MAC(w0, w1, w2, u[27], m[16]); + MONT_MAC(t0, t1, t2, u[28], m[15]); + MONT_MAC(v0, v1, v2, u[29], m[14]); + MONT_MAC(w0, w1, w2, u[30], m[13]); + MONT_MAC(t0, t1, t2, u[31], m[12]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[11] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 44 */ + MONT_MAC(t0, t1, t2, a[13], b[31]); + MONT_MUL(v0, v1, v2, a[14], b[30]); + MONT_MUL(w0, w1, w2, a[15], b[29]); + MONT_MAC(t0, t1, t2, a[16], b[28]); + MONT_MAC(v0, v1, v2, a[17], b[27]); + MONT_MAC(w0, w1, w2, a[18], b[26]); + MONT_MAC(t0, t1, t2, a[19], b[25]); + MONT_MAC(v0, v1, v2, a[20], b[24]); + MONT_MAC(w0, w1, w2, a[21], b[23]); + MONT_MAC(t0, t1, t2, a[22], b[22]); + MONT_MAC(v0, v1, v2, a[23], b[21]); + MONT_MAC(w0, w1, w2, a[24], b[20]); + MONT_MAC(t0, t1, t2, a[25], b[19]); + MONT_MAC(v0, v1, v2, a[26], b[18]); + MONT_MAC(w0, w1, w2, a[27], b[17]); + MONT_MAC(t0, t1, t2, a[28], b[16]); + MONT_MAC(v0, v1, v2, a[29], b[15]); + MONT_MAC(w0, w1, w2, a[30], b[14]); + MONT_MAC(t0, t1, t2, a[31], b[13]); + MONT_MAC(v0, v1, v2, u[13], m[31]); + MONT_MAC(w0, w1, w2, u[14], m[30]); + MONT_MAC(t0, t1, t2, u[15], m[29]); + MONT_MAC(v0, v1, v2, u[16], m[28]); + MONT_MAC(w0, w1, w2, u[17], m[27]); + MONT_MAC(t0, t1, t2, u[18], m[26]); + MONT_MAC(v0, v1, v2, u[19], m[25]); + MONT_MAC(w0, w1, w2, u[20], m[24]); + MONT_MAC(t0, t1, t2, u[21], m[23]); + MONT_MAC(v0, v1, v2, u[22], m[22]); + MONT_MAC(w0, w1, w2, u[23], m[21]); + MONT_MAC(t0, t1, t2, u[24], m[20]); + MONT_MAC(v0, v1, v2, u[25], m[19]); + MONT_MAC(w0, w1, w2, u[26], m[18]); + MONT_MAC(t0, t1, t2, u[27], m[17]); + MONT_MAC(v0, v1, v2, u[28], m[16]); + MONT_MAC(w0, w1, w2, u[29], m[15]); + MONT_MAC(t0, t1, t2, u[30], m[14]); + MONT_MAC(v0, v1, v2, u[31], m[13]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[12] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 45 */ + MONT_MAC(t0, t1, t2, a[14], b[31]); + MONT_MUL(v0, v1, v2, a[15], b[30]); + MONT_MUL(w0, w1, w2, a[16], b[29]); + MONT_MAC(t0, t1, t2, a[17], b[28]); + MONT_MAC(v0, v1, v2, a[18], b[27]); + MONT_MAC(w0, w1, w2, a[19], b[26]); + MONT_MAC(t0, t1, t2, a[20], b[25]); + MONT_MAC(v0, v1, v2, a[21], b[24]); + MONT_MAC(w0, w1, w2, a[22], b[23]); + MONT_MAC(t0, t1, t2, a[23], b[22]); + MONT_MAC(v0, v1, v2, a[24], b[21]); + MONT_MAC(w0, w1, w2, a[25], b[20]); + MONT_MAC(t0, t1, t2, a[26], b[19]); + MONT_MAC(v0, v1, v2, a[27], b[18]); + MONT_MAC(w0, w1, w2, a[28], b[17]); + MONT_MAC(t0, t1, t2, a[29], b[16]); + MONT_MAC(v0, v1, v2, a[30], b[15]); + MONT_MAC(w0, w1, w2, a[31], b[14]); + MONT_MAC(t0, t1, t2, u[14], m[31]); + MONT_MAC(v0, v1, v2, u[15], m[30]); + MONT_MAC(w0, w1, w2, u[16], m[29]); + MONT_MAC(t0, t1, t2, u[17], m[28]); + MONT_MAC(v0, v1, v2, u[18], m[27]); + MONT_MAC(w0, w1, w2, u[19], m[26]); + MONT_MAC(t0, t1, t2, u[20], m[25]); + MONT_MAC(v0, v1, v2, u[21], m[24]); + MONT_MAC(w0, w1, w2, u[22], m[23]); + MONT_MAC(t0, t1, t2, u[23], m[22]); + MONT_MAC(v0, v1, v2, u[24], m[21]); + MONT_MAC(w0, w1, w2, u[25], m[20]); + MONT_MAC(t0, t1, t2, u[26], m[19]); + MONT_MAC(v0, v1, v2, u[27], m[18]); + MONT_MAC(w0, w1, w2, u[28], m[17]); + MONT_MAC(t0, t1, t2, u[29], m[16]); + MONT_MAC(v0, v1, v2, u[30], m[15]); + MONT_MAC(w0, w1, w2, u[31], m[14]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[13] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 46 */ + MONT_MAC(t0, t1, t2, a[15], b[31]); + MONT_MUL(v0, v1, v2, a[16], b[30]); + MONT_MUL(w0, w1, w2, a[17], b[29]); + MONT_MAC(t0, t1, t2, a[18], b[28]); + MONT_MAC(v0, v1, v2, a[19], b[27]); + MONT_MAC(w0, w1, w2, a[20], b[26]); + MONT_MAC(t0, t1, t2, a[21], b[25]); + MONT_MAC(v0, v1, v2, a[22], b[24]); + MONT_MAC(w0, w1, w2, a[23], b[23]); + MONT_MAC(t0, t1, t2, a[24], b[22]); + MONT_MAC(v0, v1, v2, a[25], b[21]); + MONT_MAC(w0, w1, w2, a[26], b[20]); + MONT_MAC(t0, t1, t2, a[27], b[19]); + MONT_MAC(v0, v1, v2, a[28], b[18]); + MONT_MAC(w0, w1, w2, a[29], b[17]); + MONT_MAC(t0, t1, t2, a[30], b[16]); + MONT_MAC(v0, v1, v2, a[31], b[15]); + MONT_MAC(w0, w1, w2, u[15], m[31]); + MONT_MAC(t0, t1, t2, u[16], m[30]); + MONT_MAC(v0, v1, v2, u[17], m[29]); + MONT_MAC(w0, w1, w2, u[18], m[28]); + MONT_MAC(t0, t1, t2, u[19], m[27]); + MONT_MAC(v0, v1, v2, u[20], m[26]); + MONT_MAC(w0, w1, w2, u[21], m[25]); + MONT_MAC(t0, t1, t2, u[22], m[24]); + MONT_MAC(v0, v1, v2, u[23], m[23]); + MONT_MAC(w0, w1, w2, u[24], m[22]); + MONT_MAC(t0, t1, t2, u[25], m[21]); + MONT_MAC(v0, v1, v2, u[26], m[20]); + MONT_MAC(w0, w1, w2, u[27], m[19]); + MONT_MAC(t0, t1, t2, u[28], m[18]); + MONT_MAC(v0, v1, v2, u[29], m[17]); + MONT_MAC(w0, w1, w2, u[30], m[16]); + MONT_MAC(t0, t1, t2, u[31], m[15]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[14] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 47 */ + MONT_MAC(t0, t1, t2, a[16], b[31]); + MONT_MUL(v0, v1, v2, a[17], b[30]); + MONT_MUL(w0, w1, w2, a[18], b[29]); + MONT_MAC(t0, t1, t2, a[19], b[28]); + MONT_MAC(v0, v1, v2, a[20], b[27]); + MONT_MAC(w0, w1, w2, a[21], b[26]); + MONT_MAC(t0, t1, t2, a[22], b[25]); + MONT_MAC(v0, v1, v2, a[23], b[24]); + MONT_MAC(w0, w1, w2, a[24], b[23]); + MONT_MAC(t0, t1, t2, a[25], b[22]); + MONT_MAC(v0, v1, v2, a[26], b[21]); + MONT_MAC(w0, w1, w2, a[27], b[20]); + MONT_MAC(t0, t1, t2, a[28], b[19]); + MONT_MAC(v0, v1, v2, a[29], b[18]); + MONT_MAC(w0, w1, w2, a[30], b[17]); + MONT_MAC(t0, t1, t2, a[31], b[16]); + MONT_MAC(v0, v1, v2, u[16], m[31]); + MONT_MAC(w0, w1, w2, u[17], m[30]); + MONT_MAC(t0, t1, t2, u[18], m[29]); + MONT_MAC(v0, v1, v2, u[19], m[28]); + MONT_MAC(w0, w1, w2, u[20], m[27]); + MONT_MAC(t0, t1, t2, u[21], m[26]); + MONT_MAC(v0, v1, v2, u[22], m[25]); + MONT_MAC(w0, w1, w2, u[23], m[24]); + MONT_MAC(t0, t1, t2, u[24], m[23]); + MONT_MAC(v0, v1, v2, u[25], m[22]); + MONT_MAC(w0, w1, w2, u[26], m[21]); + MONT_MAC(t0, t1, t2, u[27], m[20]); + MONT_MAC(v0, v1, v2, u[28], m[19]); + MONT_MAC(w0, w1, w2, u[29], m[18]); + MONT_MAC(t0, t1, t2, u[30], m[17]); + MONT_MAC(v0, v1, v2, u[31], m[16]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[15] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 48 */ + MONT_MAC(t0, t1, t2, a[17], b[31]); + MONT_MUL(v0, v1, v2, a[18], b[30]); + MONT_MUL(w0, w1, w2, a[19], b[29]); + MONT_MAC(t0, t1, t2, a[20], b[28]); + MONT_MAC(v0, v1, v2, a[21], b[27]); + MONT_MAC(w0, w1, w2, a[22], b[26]); + MONT_MAC(t0, t1, t2, a[23], b[25]); + MONT_MAC(v0, v1, v2, a[24], b[24]); + MONT_MAC(w0, w1, w2, a[25], b[23]); + MONT_MAC(t0, t1, t2, a[26], b[22]); + MONT_MAC(v0, v1, v2, a[27], b[21]); + MONT_MAC(w0, w1, w2, a[28], b[20]); + MONT_MAC(t0, t1, t2, a[29], b[19]); + MONT_MAC(v0, v1, v2, a[30], b[18]); + MONT_MAC(w0, w1, w2, a[31], b[17]); + MONT_MAC(t0, t1, t2, u[17], m[31]); + MONT_MAC(v0, v1, v2, u[18], m[30]); + MONT_MAC(w0, w1, w2, u[19], m[29]); + MONT_MAC(t0, t1, t2, u[20], m[28]); + MONT_MAC(v0, v1, v2, u[21], m[27]); + MONT_MAC(w0, w1, w2, u[22], m[26]); + MONT_MAC(t0, t1, t2, u[23], m[25]); + MONT_MAC(v0, v1, v2, u[24], m[24]); + MONT_MAC(w0, w1, w2, u[25], m[23]); + MONT_MAC(t0, t1, t2, u[26], m[22]); + MONT_MAC(v0, v1, v2, u[27], m[21]); + MONT_MAC(w0, w1, w2, u[28], m[20]); + MONT_MAC(t0, t1, t2, u[29], m[19]); + MONT_MAC(v0, v1, v2, u[30], m[18]); + MONT_MAC(w0, w1, w2, u[31], m[17]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[16] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 49 */ + MONT_MAC(t0, t1, t2, a[18], b[31]); + MONT_MUL(v0, v1, v2, a[19], b[30]); + MONT_MUL(w0, w1, w2, a[20], b[29]); + MONT_MAC(t0, t1, t2, a[21], b[28]); + MONT_MAC(v0, v1, v2, a[22], b[27]); + MONT_MAC(w0, w1, w2, a[23], b[26]); + MONT_MAC(t0, t1, t2, a[24], b[25]); + MONT_MAC(v0, v1, v2, a[25], b[24]); + MONT_MAC(w0, w1, w2, a[26], b[23]); + MONT_MAC(t0, t1, t2, a[27], b[22]); + MONT_MAC(v0, v1, v2, a[28], b[21]); + MONT_MAC(w0, w1, w2, a[29], b[20]); + MONT_MAC(t0, t1, t2, a[30], b[19]); + MONT_MAC(v0, v1, v2, a[31], b[18]); + MONT_MAC(w0, w1, w2, u[18], m[31]); + MONT_MAC(t0, t1, t2, u[19], m[30]); + MONT_MAC(v0, v1, v2, u[20], m[29]); + MONT_MAC(w0, w1, w2, u[21], m[28]); + MONT_MAC(t0, t1, t2, u[22], m[27]); + MONT_MAC(v0, v1, v2, u[23], m[26]); + MONT_MAC(w0, w1, w2, u[24], m[25]); + MONT_MAC(t0, t1, t2, u[25], m[24]); + MONT_MAC(v0, v1, v2, u[26], m[23]); + MONT_MAC(w0, w1, w2, u[27], m[22]); + MONT_MAC(t0, t1, t2, u[28], m[21]); + MONT_MAC(v0, v1, v2, u[29], m[20]); + MONT_MAC(w0, w1, w2, u[30], m[19]); + MONT_MAC(t0, t1, t2, u[31], m[18]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[17] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 50 */ + MONT_MAC(t0, t1, t2, a[19], b[31]); + MONT_MUL(v0, v1, v2, a[20], b[30]); + MONT_MUL(w0, w1, w2, a[21], b[29]); + MONT_MAC(t0, t1, t2, a[22], b[28]); + MONT_MAC(v0, v1, v2, a[23], b[27]); + MONT_MAC(w0, w1, w2, a[24], b[26]); + MONT_MAC(t0, t1, t2, a[25], b[25]); + MONT_MAC(v0, v1, v2, a[26], b[24]); + MONT_MAC(w0, w1, w2, a[27], b[23]); + MONT_MAC(t0, t1, t2, a[28], b[22]); + MONT_MAC(v0, v1, v2, a[29], b[21]); + MONT_MAC(w0, w1, w2, a[30], b[20]); + MONT_MAC(t0, t1, t2, a[31], b[19]); + MONT_MAC(v0, v1, v2, u[19], m[31]); + MONT_MAC(w0, w1, w2, u[20], m[30]); + MONT_MAC(t0, t1, t2, u[21], m[29]); + MONT_MAC(v0, v1, v2, u[22], m[28]); + MONT_MAC(w0, w1, w2, u[23], m[27]); + MONT_MAC(t0, t1, t2, u[24], m[26]); + MONT_MAC(v0, v1, v2, u[25], m[25]); + MONT_MAC(w0, w1, w2, u[26], m[24]); + MONT_MAC(t0, t1, t2, u[27], m[23]); + MONT_MAC(v0, v1, v2, u[28], m[22]); + MONT_MAC(w0, w1, w2, u[29], m[21]); + MONT_MAC(t0, t1, t2, u[30], m[20]); + MONT_MAC(v0, v1, v2, u[31], m[19]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[18] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 51 */ + MONT_MAC(t0, t1, t2, a[20], b[31]); + MONT_MUL(v0, v1, v2, a[21], b[30]); + MONT_MUL(w0, w1, w2, a[22], b[29]); + MONT_MAC(t0, t1, t2, a[23], b[28]); + MONT_MAC(v0, v1, v2, a[24], b[27]); + MONT_MAC(w0, w1, w2, a[25], b[26]); + MONT_MAC(t0, t1, t2, a[26], b[25]); + MONT_MAC(v0, v1, v2, a[27], b[24]); + MONT_MAC(w0, w1, w2, a[28], b[23]); + MONT_MAC(t0, t1, t2, a[29], b[22]); + MONT_MAC(v0, v1, v2, a[30], b[21]); + MONT_MAC(w0, w1, w2, a[31], b[20]); + MONT_MAC(t0, t1, t2, u[20], m[31]); + MONT_MAC(v0, v1, v2, u[21], m[30]); + MONT_MAC(w0, w1, w2, u[22], m[29]); + MONT_MAC(t0, t1, t2, u[23], m[28]); + MONT_MAC(v0, v1, v2, u[24], m[27]); + MONT_MAC(w0, w1, w2, u[25], m[26]); + MONT_MAC(t0, t1, t2, u[26], m[25]); + MONT_MAC(v0, v1, v2, u[27], m[24]); + MONT_MAC(w0, w1, w2, u[28], m[23]); + MONT_MAC(t0, t1, t2, u[29], m[22]); + MONT_MAC(v0, v1, v2, u[30], m[21]); + MONT_MAC(w0, w1, w2, u[31], m[20]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[19] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 52 */ + MONT_MAC(t0, t1, t2, a[21], b[31]); + MONT_MUL(v0, v1, v2, a[22], b[30]); + MONT_MUL(w0, w1, w2, a[23], b[29]); + MONT_MAC(t0, t1, t2, a[24], b[28]); + MONT_MAC(v0, v1, v2, a[25], b[27]); + MONT_MAC(w0, w1, w2, a[26], b[26]); + MONT_MAC(t0, t1, t2, a[27], b[25]); + MONT_MAC(v0, v1, v2, a[28], b[24]); + MONT_MAC(w0, w1, w2, a[29], b[23]); + MONT_MAC(t0, t1, t2, a[30], b[22]); + MONT_MAC(v0, v1, v2, a[31], b[21]); + MONT_MAC(w0, w1, w2, u[21], m[31]); + MONT_MAC(t0, t1, t2, u[22], m[30]); + MONT_MAC(v0, v1, v2, u[23], m[29]); + MONT_MAC(w0, w1, w2, u[24], m[28]); + MONT_MAC(t0, t1, t2, u[25], m[27]); + MONT_MAC(v0, v1, v2, u[26], m[26]); + MONT_MAC(w0, w1, w2, u[27], m[25]); + MONT_MAC(t0, t1, t2, u[28], m[24]); + MONT_MAC(v0, v1, v2, u[29], m[23]); + MONT_MAC(w0, w1, w2, u[30], m[22]); + MONT_MAC(t0, t1, t2, u[31], m[21]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[20] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 53 */ + MONT_MAC(t0, t1, t2, a[22], b[31]); + MONT_MUL(v0, v1, v2, a[23], b[30]); + MONT_MUL(w0, w1, w2, a[24], b[29]); + MONT_MAC(t0, t1, t2, a[25], b[28]); + MONT_MAC(v0, v1, v2, a[26], b[27]); + MONT_MAC(w0, w1, w2, a[27], b[26]); + MONT_MAC(t0, t1, t2, a[28], b[25]); + MONT_MAC(v0, v1, v2, a[29], b[24]); + MONT_MAC(w0, w1, w2, a[30], b[23]); + MONT_MAC(t0, t1, t2, a[31], b[22]); + MONT_MAC(v0, v1, v2, u[22], m[31]); + MONT_MAC(w0, w1, w2, u[23], m[30]); + MONT_MAC(t0, t1, t2, u[24], m[29]); + MONT_MAC(v0, v1, v2, u[25], m[28]); + MONT_MAC(w0, w1, w2, u[26], m[27]); + MONT_MAC(t0, t1, t2, u[27], m[26]); + MONT_MAC(v0, v1, v2, u[28], m[25]); + MONT_MAC(w0, w1, w2, u[29], m[24]); + MONT_MAC(t0, t1, t2, u[30], m[23]); + MONT_MAC(v0, v1, v2, u[31], m[22]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[21] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 54 */ + MONT_MAC(t0, t1, t2, a[23], b[31]); + MONT_MUL(v0, v1, v2, a[24], b[30]); + MONT_MUL(w0, w1, w2, a[25], b[29]); + MONT_MAC(t0, t1, t2, a[26], b[28]); + MONT_MAC(v0, v1, v2, a[27], b[27]); + MONT_MAC(w0, w1, w2, a[28], b[26]); + MONT_MAC(t0, t1, t2, a[29], b[25]); + MONT_MAC(v0, v1, v2, a[30], b[24]); + MONT_MAC(w0, w1, w2, a[31], b[23]); + MONT_MAC(t0, t1, t2, u[23], m[31]); + MONT_MAC(v0, v1, v2, u[24], m[30]); + MONT_MAC(w0, w1, w2, u[25], m[29]); + MONT_MAC(t0, t1, t2, u[26], m[28]); + MONT_MAC(v0, v1, v2, u[27], m[27]); + MONT_MAC(w0, w1, w2, u[28], m[26]); + MONT_MAC(t0, t1, t2, u[29], m[25]); + MONT_MAC(v0, v1, v2, u[30], m[24]); + MONT_MAC(w0, w1, w2, u[31], m[23]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[22] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 55 */ + MONT_MAC(t0, t1, t2, a[24], b[31]); + MONT_MUL(v0, v1, v2, a[25], b[30]); + MONT_MUL(w0, w1, w2, a[26], b[29]); + MONT_MAC(t0, t1, t2, a[27], b[28]); + MONT_MAC(v0, v1, v2, a[28], b[27]); + MONT_MAC(w0, w1, w2, a[29], b[26]); + MONT_MAC(t0, t1, t2, a[30], b[25]); + MONT_MAC(v0, v1, v2, a[31], b[24]); + MONT_MAC(w0, w1, w2, u[24], m[31]); + MONT_MAC(t0, t1, t2, u[25], m[30]); + MONT_MAC(v0, v1, v2, u[26], m[29]); + MONT_MAC(w0, w1, w2, u[27], m[28]); + MONT_MAC(t0, t1, t2, u[28], m[27]); + MONT_MAC(v0, v1, v2, u[29], m[26]); + MONT_MAC(w0, w1, w2, u[30], m[25]); + MONT_MAC(t0, t1, t2, u[31], m[24]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[23] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 56 */ + MONT_MAC(t0, t1, t2, a[25], b[31]); + MONT_MUL(v0, v1, v2, a[26], b[30]); + MONT_MUL(w0, w1, w2, a[27], b[29]); + MONT_MAC(t0, t1, t2, a[28], b[28]); + MONT_MAC(v0, v1, v2, a[29], b[27]); + MONT_MAC(w0, w1, w2, a[30], b[26]); + MONT_MAC(t0, t1, t2, a[31], b[25]); + MONT_MAC(v0, v1, v2, u[25], m[31]); + MONT_MAC(w0, w1, w2, u[26], m[30]); + MONT_MAC(t0, t1, t2, u[27], m[29]); + MONT_MAC(v0, v1, v2, u[28], m[28]); + MONT_MAC(w0, w1, w2, u[29], m[27]); + MONT_MAC(t0, t1, t2, u[30], m[26]); + MONT_MAC(v0, v1, v2, u[31], m[25]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[24] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 57 */ + MONT_MAC(t0, t1, t2, a[26], b[31]); + MONT_MUL(v0, v1, v2, a[27], b[30]); + MONT_MUL(w0, w1, w2, a[28], b[29]); + MONT_MAC(t0, t1, t2, a[29], b[28]); + MONT_MAC(v0, v1, v2, a[30], b[27]); + MONT_MAC(w0, w1, w2, a[31], b[26]); + MONT_MAC(t0, t1, t2, u[26], m[31]); + MONT_MAC(v0, v1, v2, u[27], m[30]); + MONT_MAC(w0, w1, w2, u[28], m[29]); + MONT_MAC(t0, t1, t2, u[29], m[28]); + MONT_MAC(v0, v1, v2, u[30], m[27]); + MONT_MAC(w0, w1, w2, u[31], m[26]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[25] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 58 */ + MONT_MAC(t0, t1, t2, a[27], b[31]); + MONT_MUL(v0, v1, v2, a[28], b[30]); + MONT_MUL(w0, w1, w2, a[29], b[29]); + MONT_MAC(t0, t1, t2, a[30], b[28]); + MONT_MAC(v0, v1, v2, a[31], b[27]); + MONT_MAC(w0, w1, w2, u[27], m[31]); + MONT_MAC(t0, t1, t2, u[28], m[30]); + MONT_MAC(v0, v1, v2, u[29], m[29]); + MONT_MAC(w0, w1, w2, u[30], m[28]); + MONT_MAC(t0, t1, t2, u[31], m[27]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[26] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 59 */ + MONT_MAC(t0, t1, t2, a[28], b[31]); + MONT_MUL(v0, v1, v2, a[29], b[30]); + MONT_MUL(w0, w1, w2, a[30], b[29]); + MONT_MAC(t0, t1, t2, a[31], b[28]); + MONT_MAC(v0, v1, v2, u[28], m[31]); + MONT_MAC(w0, w1, w2, u[29], m[30]); + MONT_MAC(t0, t1, t2, u[30], m[29]); + MONT_MAC(v0, v1, v2, u[31], m[28]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[27] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 60 */ + MONT_MAC(t0, t1, t2, a[29], b[31]); + MONT_MUL(v0, v1, v2, a[30], b[30]); + MONT_MUL(w0, w1, w2, a[31], b[29]); + MONT_MAC(t0, t1, t2, u[29], m[31]); + MONT_MAC(v0, v1, v2, u[30], m[30]); + MONT_MAC(w0, w1, w2, u[31], m[29]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[28] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 61 */ + MONT_MAC(t0, t1, t2, a[30], b[31]); + MONT_MUL(v0, v1, v2, a[31], b[30]); + MONT_MUL(w0, w1, w2, u[30], m[31]); + MONT_MAC(t0, t1, t2, u[31], m[30]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + z[29] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 62 */ + MONT_MAC(t0, t1, t2, a[31], b[31]); + MONT_MUL(v0, v1, v2, u[31], m[31]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + z[30] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + z[31] = t0; + + /* under 2m, with t1 the bit above n limbs: at most one subtraction */ + borrow = sub_n(r, z, m, 32); + take = t1 | (borrow ^ 1); + select_n(r, r, z, take & 1, 32); +} + +/* r = a * a * R^-1 mod m, for a under m; t is 2n limbs of scratch */ +static void crypton_mont_sqr32(limb_t *r, const limb_t *a, + const limb_t *m, limb_t n0, limb_t *t) +{ + limb_t *u = t, *z = t + 32; + limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take; + limb_t s0, s1, s2, c0, c1, c2; + + /* column 0 */ + MONT_MAC(t0, t1, t2, a[0], a[0]); + u[0] = t0 * n0; + MONT_MAC(t0, t1, t2, u[0], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 1 */ + MONT_MUL(s0, s1, s2, a[0], a[1]); + MONT_MAC(t0, t1, t2, u[0], m[1]); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[1] = t0 * n0; + MONT_MAC(t0, t1, t2, u[1], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 2 */ + MONT_MUL(s0, s1, s2, a[0], a[2]); + MONT_MAC(t0, t1, t2, a[1], a[1]); + MONT_MUL(v0, v1, v2, u[0], m[2]); + MONT_MUL(w0, w1, w2, u[1], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[2] = t0 * n0; + MONT_MAC(t0, t1, t2, u[2], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 3 */ + MONT_MUL(s0, s1, s2, a[0], a[3]); + MONT_MUL(c0, c1, c2, a[1], a[2]); + MONT_MAC(t0, t1, t2, u[0], m[3]); + MONT_MUL(v0, v1, v2, u[1], m[2]); + MONT_MUL(w0, w1, w2, u[2], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[3] = t0 * n0; + MONT_MAC(t0, t1, t2, u[3], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 4 */ + MONT_MUL(s0, s1, s2, a[0], a[4]); + MONT_MUL(c0, c1, c2, a[1], a[3]); + MONT_MAC(t0, t1, t2, a[2], a[2]); + MONT_MUL(v0, v1, v2, u[0], m[4]); + MONT_MUL(w0, w1, w2, u[1], m[3]); + MONT_MAC(t0, t1, t2, u[2], m[2]); + MONT_MAC(v0, v1, v2, u[3], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[4] = t0 * n0; + MONT_MAC(t0, t1, t2, u[4], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 5 */ + MONT_MUL(s0, s1, s2, a[0], a[5]); + MONT_MUL(c0, c1, c2, a[1], a[4]); + MONT_MAC(s0, s1, s2, a[2], a[3]); + MONT_MAC(t0, t1, t2, u[0], m[5]); + MONT_MUL(v0, v1, v2, u[1], m[4]); + MONT_MUL(w0, w1, w2, u[2], m[3]); + MONT_MAC(t0, t1, t2, u[3], m[2]); + MONT_MAC(v0, v1, v2, u[4], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[5] = t0 * n0; + MONT_MAC(t0, t1, t2, u[5], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 6 */ + MONT_MUL(s0, s1, s2, a[0], a[6]); + MONT_MUL(c0, c1, c2, a[1], a[5]); + MONT_MAC(s0, s1, s2, a[2], a[4]); + MONT_MAC(t0, t1, t2, a[3], a[3]); + MONT_MUL(v0, v1, v2, u[0], m[6]); + MONT_MUL(w0, w1, w2, u[1], m[5]); + MONT_MAC(t0, t1, t2, u[2], m[4]); + MONT_MAC(v0, v1, v2, u[3], m[3]); + MONT_MAC(w0, w1, w2, u[4], m[2]); + MONT_MAC(t0, t1, t2, u[5], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[6] = t0 * n0; + MONT_MAC(t0, t1, t2, u[6], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 7 */ + MONT_MUL(s0, s1, s2, a[0], a[7]); + MONT_MUL(c0, c1, c2, a[1], a[6]); + MONT_MAC(s0, s1, s2, a[2], a[5]); + MONT_MAC(c0, c1, c2, a[3], a[4]); + MONT_MAC(t0, t1, t2, u[0], m[7]); + MONT_MUL(v0, v1, v2, u[1], m[6]); + MONT_MUL(w0, w1, w2, u[2], m[5]); + MONT_MAC(t0, t1, t2, u[3], m[4]); + MONT_MAC(v0, v1, v2, u[4], m[3]); + MONT_MAC(w0, w1, w2, u[5], m[2]); + MONT_MAC(t0, t1, t2, u[6], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[7] = t0 * n0; + MONT_MAC(t0, t1, t2, u[7], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 8 */ + MONT_MUL(s0, s1, s2, a[0], a[8]); + MONT_MUL(c0, c1, c2, a[1], a[7]); + MONT_MAC(s0, s1, s2, a[2], a[6]); + MONT_MAC(c0, c1, c2, a[3], a[5]); + MONT_MAC(t0, t1, t2, a[4], a[4]); + MONT_MUL(v0, v1, v2, u[0], m[8]); + MONT_MUL(w0, w1, w2, u[1], m[7]); + MONT_MAC(t0, t1, t2, u[2], m[6]); + MONT_MAC(v0, v1, v2, u[3], m[5]); + MONT_MAC(w0, w1, w2, u[4], m[4]); + MONT_MAC(t0, t1, t2, u[5], m[3]); + MONT_MAC(v0, v1, v2, u[6], m[2]); + MONT_MAC(w0, w1, w2, u[7], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[8] = t0 * n0; + MONT_MAC(t0, t1, t2, u[8], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 9 */ + MONT_MUL(s0, s1, s2, a[0], a[9]); + MONT_MUL(c0, c1, c2, a[1], a[8]); + MONT_MAC(s0, s1, s2, a[2], a[7]); + MONT_MAC(c0, c1, c2, a[3], a[6]); + MONT_MAC(s0, s1, s2, a[4], a[5]); + MONT_MAC(t0, t1, t2, u[0], m[9]); + MONT_MUL(v0, v1, v2, u[1], m[8]); + MONT_MUL(w0, w1, w2, u[2], m[7]); + MONT_MAC(t0, t1, t2, u[3], m[6]); + MONT_MAC(v0, v1, v2, u[4], m[5]); + MONT_MAC(w0, w1, w2, u[5], m[4]); + MONT_MAC(t0, t1, t2, u[6], m[3]); + MONT_MAC(v0, v1, v2, u[7], m[2]); + MONT_MAC(w0, w1, w2, u[8], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[9] = t0 * n0; + MONT_MAC(t0, t1, t2, u[9], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 10 */ + MONT_MUL(s0, s1, s2, a[0], a[10]); + MONT_MUL(c0, c1, c2, a[1], a[9]); + MONT_MAC(s0, s1, s2, a[2], a[8]); + MONT_MAC(c0, c1, c2, a[3], a[7]); + MONT_MAC(s0, s1, s2, a[4], a[6]); + MONT_MAC(t0, t1, t2, a[5], a[5]); + MONT_MUL(v0, v1, v2, u[0], m[10]); + MONT_MUL(w0, w1, w2, u[1], m[9]); + MONT_MAC(t0, t1, t2, u[2], m[8]); + MONT_MAC(v0, v1, v2, u[3], m[7]); + MONT_MAC(w0, w1, w2, u[4], m[6]); + MONT_MAC(t0, t1, t2, u[5], m[5]); + MONT_MAC(v0, v1, v2, u[6], m[4]); + MONT_MAC(w0, w1, w2, u[7], m[3]); + MONT_MAC(t0, t1, t2, u[8], m[2]); + MONT_MAC(v0, v1, v2, u[9], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[10] = t0 * n0; + MONT_MAC(t0, t1, t2, u[10], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 11 */ + MONT_MUL(s0, s1, s2, a[0], a[11]); + MONT_MUL(c0, c1, c2, a[1], a[10]); + MONT_MAC(s0, s1, s2, a[2], a[9]); + MONT_MAC(c0, c1, c2, a[3], a[8]); + MONT_MAC(s0, s1, s2, a[4], a[7]); + MONT_MAC(c0, c1, c2, a[5], a[6]); + MONT_MAC(t0, t1, t2, u[0], m[11]); + MONT_MUL(v0, v1, v2, u[1], m[10]); + MONT_MUL(w0, w1, w2, u[2], m[9]); + MONT_MAC(t0, t1, t2, u[3], m[8]); + MONT_MAC(v0, v1, v2, u[4], m[7]); + MONT_MAC(w0, w1, w2, u[5], m[6]); + MONT_MAC(t0, t1, t2, u[6], m[5]); + MONT_MAC(v0, v1, v2, u[7], m[4]); + MONT_MAC(w0, w1, w2, u[8], m[3]); + MONT_MAC(t0, t1, t2, u[9], m[2]); + MONT_MAC(v0, v1, v2, u[10], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[11] = t0 * n0; + MONT_MAC(t0, t1, t2, u[11], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 12 */ + MONT_MUL(s0, s1, s2, a[0], a[12]); + MONT_MUL(c0, c1, c2, a[1], a[11]); + MONT_MAC(s0, s1, s2, a[2], a[10]); + MONT_MAC(c0, c1, c2, a[3], a[9]); + MONT_MAC(s0, s1, s2, a[4], a[8]); + MONT_MAC(c0, c1, c2, a[5], a[7]); + MONT_MAC(t0, t1, t2, a[6], a[6]); + MONT_MUL(v0, v1, v2, u[0], m[12]); + MONT_MUL(w0, w1, w2, u[1], m[11]); + MONT_MAC(t0, t1, t2, u[2], m[10]); + MONT_MAC(v0, v1, v2, u[3], m[9]); + MONT_MAC(w0, w1, w2, u[4], m[8]); + MONT_MAC(t0, t1, t2, u[5], m[7]); + MONT_MAC(v0, v1, v2, u[6], m[6]); + MONT_MAC(w0, w1, w2, u[7], m[5]); + MONT_MAC(t0, t1, t2, u[8], m[4]); + MONT_MAC(v0, v1, v2, u[9], m[3]); + MONT_MAC(w0, w1, w2, u[10], m[2]); + MONT_MAC(t0, t1, t2, u[11], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[12] = t0 * n0; + MONT_MAC(t0, t1, t2, u[12], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 13 */ + MONT_MUL(s0, s1, s2, a[0], a[13]); + MONT_MUL(c0, c1, c2, a[1], a[12]); + MONT_MAC(s0, s1, s2, a[2], a[11]); + MONT_MAC(c0, c1, c2, a[3], a[10]); + MONT_MAC(s0, s1, s2, a[4], a[9]); + MONT_MAC(c0, c1, c2, a[5], a[8]); + MONT_MAC(s0, s1, s2, a[6], a[7]); + MONT_MAC(t0, t1, t2, u[0], m[13]); + MONT_MUL(v0, v1, v2, u[1], m[12]); + MONT_MUL(w0, w1, w2, u[2], m[11]); + MONT_MAC(t0, t1, t2, u[3], m[10]); + MONT_MAC(v0, v1, v2, u[4], m[9]); + MONT_MAC(w0, w1, w2, u[5], m[8]); + MONT_MAC(t0, t1, t2, u[6], m[7]); + MONT_MAC(v0, v1, v2, u[7], m[6]); + MONT_MAC(w0, w1, w2, u[8], m[5]); + MONT_MAC(t0, t1, t2, u[9], m[4]); + MONT_MAC(v0, v1, v2, u[10], m[3]); + MONT_MAC(w0, w1, w2, u[11], m[2]); + MONT_MAC(t0, t1, t2, u[12], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[13] = t0 * n0; + MONT_MAC(t0, t1, t2, u[13], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 14 */ + MONT_MUL(s0, s1, s2, a[0], a[14]); + MONT_MUL(c0, c1, c2, a[1], a[13]); + MONT_MAC(s0, s1, s2, a[2], a[12]); + MONT_MAC(c0, c1, c2, a[3], a[11]); + MONT_MAC(s0, s1, s2, a[4], a[10]); + MONT_MAC(c0, c1, c2, a[5], a[9]); + MONT_MAC(s0, s1, s2, a[6], a[8]); + MONT_MAC(t0, t1, t2, a[7], a[7]); + MONT_MUL(v0, v1, v2, u[0], m[14]); + MONT_MUL(w0, w1, w2, u[1], m[13]); + MONT_MAC(t0, t1, t2, u[2], m[12]); + MONT_MAC(v0, v1, v2, u[3], m[11]); + MONT_MAC(w0, w1, w2, u[4], m[10]); + MONT_MAC(t0, t1, t2, u[5], m[9]); + MONT_MAC(v0, v1, v2, u[6], m[8]); + MONT_MAC(w0, w1, w2, u[7], m[7]); + MONT_MAC(t0, t1, t2, u[8], m[6]); + MONT_MAC(v0, v1, v2, u[9], m[5]); + MONT_MAC(w0, w1, w2, u[10], m[4]); + MONT_MAC(t0, t1, t2, u[11], m[3]); + MONT_MAC(v0, v1, v2, u[12], m[2]); + MONT_MAC(w0, w1, w2, u[13], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[14] = t0 * n0; + MONT_MAC(t0, t1, t2, u[14], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 15 */ + MONT_MUL(s0, s1, s2, a[0], a[15]); + MONT_MUL(c0, c1, c2, a[1], a[14]); + MONT_MAC(s0, s1, s2, a[2], a[13]); + MONT_MAC(c0, c1, c2, a[3], a[12]); + MONT_MAC(s0, s1, s2, a[4], a[11]); + MONT_MAC(c0, c1, c2, a[5], a[10]); + MONT_MAC(s0, s1, s2, a[6], a[9]); + MONT_MAC(c0, c1, c2, a[7], a[8]); + MONT_MAC(t0, t1, t2, u[0], m[15]); + MONT_MUL(v0, v1, v2, u[1], m[14]); + MONT_MUL(w0, w1, w2, u[2], m[13]); + MONT_MAC(t0, t1, t2, u[3], m[12]); + MONT_MAC(v0, v1, v2, u[4], m[11]); + MONT_MAC(w0, w1, w2, u[5], m[10]); + MONT_MAC(t0, t1, t2, u[6], m[9]); + MONT_MAC(v0, v1, v2, u[7], m[8]); + MONT_MAC(w0, w1, w2, u[8], m[7]); + MONT_MAC(t0, t1, t2, u[9], m[6]); + MONT_MAC(v0, v1, v2, u[10], m[5]); + MONT_MAC(w0, w1, w2, u[11], m[4]); + MONT_MAC(t0, t1, t2, u[12], m[3]); + MONT_MAC(v0, v1, v2, u[13], m[2]); + MONT_MAC(w0, w1, w2, u[14], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[15] = t0 * n0; + MONT_MAC(t0, t1, t2, u[15], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 16 */ + MONT_MUL(s0, s1, s2, a[0], a[16]); + MONT_MUL(c0, c1, c2, a[1], a[15]); + MONT_MAC(s0, s1, s2, a[2], a[14]); + MONT_MAC(c0, c1, c2, a[3], a[13]); + MONT_MAC(s0, s1, s2, a[4], a[12]); + MONT_MAC(c0, c1, c2, a[5], a[11]); + MONT_MAC(s0, s1, s2, a[6], a[10]); + MONT_MAC(c0, c1, c2, a[7], a[9]); + MONT_MAC(t0, t1, t2, a[8], a[8]); + MONT_MUL(v0, v1, v2, u[0], m[16]); + MONT_MUL(w0, w1, w2, u[1], m[15]); + MONT_MAC(t0, t1, t2, u[2], m[14]); + MONT_MAC(v0, v1, v2, u[3], m[13]); + MONT_MAC(w0, w1, w2, u[4], m[12]); + MONT_MAC(t0, t1, t2, u[5], m[11]); + MONT_MAC(v0, v1, v2, u[6], m[10]); + MONT_MAC(w0, w1, w2, u[7], m[9]); + MONT_MAC(t0, t1, t2, u[8], m[8]); + MONT_MAC(v0, v1, v2, u[9], m[7]); + MONT_MAC(w0, w1, w2, u[10], m[6]); + MONT_MAC(t0, t1, t2, u[11], m[5]); + MONT_MAC(v0, v1, v2, u[12], m[4]); + MONT_MAC(w0, w1, w2, u[13], m[3]); + MONT_MAC(t0, t1, t2, u[14], m[2]); + MONT_MAC(v0, v1, v2, u[15], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[16] = t0 * n0; + MONT_MAC(t0, t1, t2, u[16], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 17 */ + MONT_MUL(s0, s1, s2, a[0], a[17]); + MONT_MUL(c0, c1, c2, a[1], a[16]); + MONT_MAC(s0, s1, s2, a[2], a[15]); + MONT_MAC(c0, c1, c2, a[3], a[14]); + MONT_MAC(s0, s1, s2, a[4], a[13]); + MONT_MAC(c0, c1, c2, a[5], a[12]); + MONT_MAC(s0, s1, s2, a[6], a[11]); + MONT_MAC(c0, c1, c2, a[7], a[10]); + MONT_MAC(s0, s1, s2, a[8], a[9]); + MONT_MAC(t0, t1, t2, u[0], m[17]); + MONT_MUL(v0, v1, v2, u[1], m[16]); + MONT_MUL(w0, w1, w2, u[2], m[15]); + MONT_MAC(t0, t1, t2, u[3], m[14]); + MONT_MAC(v0, v1, v2, u[4], m[13]); + MONT_MAC(w0, w1, w2, u[5], m[12]); + MONT_MAC(t0, t1, t2, u[6], m[11]); + MONT_MAC(v0, v1, v2, u[7], m[10]); + MONT_MAC(w0, w1, w2, u[8], m[9]); + MONT_MAC(t0, t1, t2, u[9], m[8]); + MONT_MAC(v0, v1, v2, u[10], m[7]); + MONT_MAC(w0, w1, w2, u[11], m[6]); + MONT_MAC(t0, t1, t2, u[12], m[5]); + MONT_MAC(v0, v1, v2, u[13], m[4]); + MONT_MAC(w0, w1, w2, u[14], m[3]); + MONT_MAC(t0, t1, t2, u[15], m[2]); + MONT_MAC(v0, v1, v2, u[16], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[17] = t0 * n0; + MONT_MAC(t0, t1, t2, u[17], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 18 */ + MONT_MUL(s0, s1, s2, a[0], a[18]); + MONT_MUL(c0, c1, c2, a[1], a[17]); + MONT_MAC(s0, s1, s2, a[2], a[16]); + MONT_MAC(c0, c1, c2, a[3], a[15]); + MONT_MAC(s0, s1, s2, a[4], a[14]); + MONT_MAC(c0, c1, c2, a[5], a[13]); + MONT_MAC(s0, s1, s2, a[6], a[12]); + MONT_MAC(c0, c1, c2, a[7], a[11]); + MONT_MAC(s0, s1, s2, a[8], a[10]); + MONT_MAC(t0, t1, t2, a[9], a[9]); + MONT_MUL(v0, v1, v2, u[0], m[18]); + MONT_MUL(w0, w1, w2, u[1], m[17]); + MONT_MAC(t0, t1, t2, u[2], m[16]); + MONT_MAC(v0, v1, v2, u[3], m[15]); + MONT_MAC(w0, w1, w2, u[4], m[14]); + MONT_MAC(t0, t1, t2, u[5], m[13]); + MONT_MAC(v0, v1, v2, u[6], m[12]); + MONT_MAC(w0, w1, w2, u[7], m[11]); + MONT_MAC(t0, t1, t2, u[8], m[10]); + MONT_MAC(v0, v1, v2, u[9], m[9]); + MONT_MAC(w0, w1, w2, u[10], m[8]); + MONT_MAC(t0, t1, t2, u[11], m[7]); + MONT_MAC(v0, v1, v2, u[12], m[6]); + MONT_MAC(w0, w1, w2, u[13], m[5]); + MONT_MAC(t0, t1, t2, u[14], m[4]); + MONT_MAC(v0, v1, v2, u[15], m[3]); + MONT_MAC(w0, w1, w2, u[16], m[2]); + MONT_MAC(t0, t1, t2, u[17], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[18] = t0 * n0; + MONT_MAC(t0, t1, t2, u[18], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 19 */ + MONT_MUL(s0, s1, s2, a[0], a[19]); + MONT_MUL(c0, c1, c2, a[1], a[18]); + MONT_MAC(s0, s1, s2, a[2], a[17]); + MONT_MAC(c0, c1, c2, a[3], a[16]); + MONT_MAC(s0, s1, s2, a[4], a[15]); + MONT_MAC(c0, c1, c2, a[5], a[14]); + MONT_MAC(s0, s1, s2, a[6], a[13]); + MONT_MAC(c0, c1, c2, a[7], a[12]); + MONT_MAC(s0, s1, s2, a[8], a[11]); + MONT_MAC(c0, c1, c2, a[9], a[10]); + MONT_MAC(t0, t1, t2, u[0], m[19]); + MONT_MUL(v0, v1, v2, u[1], m[18]); + MONT_MUL(w0, w1, w2, u[2], m[17]); + MONT_MAC(t0, t1, t2, u[3], m[16]); + MONT_MAC(v0, v1, v2, u[4], m[15]); + MONT_MAC(w0, w1, w2, u[5], m[14]); + MONT_MAC(t0, t1, t2, u[6], m[13]); + MONT_MAC(v0, v1, v2, u[7], m[12]); + MONT_MAC(w0, w1, w2, u[8], m[11]); + MONT_MAC(t0, t1, t2, u[9], m[10]); + MONT_MAC(v0, v1, v2, u[10], m[9]); + MONT_MAC(w0, w1, w2, u[11], m[8]); + MONT_MAC(t0, t1, t2, u[12], m[7]); + MONT_MAC(v0, v1, v2, u[13], m[6]); + MONT_MAC(w0, w1, w2, u[14], m[5]); + MONT_MAC(t0, t1, t2, u[15], m[4]); + MONT_MAC(v0, v1, v2, u[16], m[3]); + MONT_MAC(w0, w1, w2, u[17], m[2]); + MONT_MAC(t0, t1, t2, u[18], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[19] = t0 * n0; + MONT_MAC(t0, t1, t2, u[19], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 20 */ + MONT_MUL(s0, s1, s2, a[0], a[20]); + MONT_MUL(c0, c1, c2, a[1], a[19]); + MONT_MAC(s0, s1, s2, a[2], a[18]); + MONT_MAC(c0, c1, c2, a[3], a[17]); + MONT_MAC(s0, s1, s2, a[4], a[16]); + MONT_MAC(c0, c1, c2, a[5], a[15]); + MONT_MAC(s0, s1, s2, a[6], a[14]); + MONT_MAC(c0, c1, c2, a[7], a[13]); + MONT_MAC(s0, s1, s2, a[8], a[12]); + MONT_MAC(c0, c1, c2, a[9], a[11]); + MONT_MAC(t0, t1, t2, a[10], a[10]); + MONT_MUL(v0, v1, v2, u[0], m[20]); + MONT_MUL(w0, w1, w2, u[1], m[19]); + MONT_MAC(t0, t1, t2, u[2], m[18]); + MONT_MAC(v0, v1, v2, u[3], m[17]); + MONT_MAC(w0, w1, w2, u[4], m[16]); + MONT_MAC(t0, t1, t2, u[5], m[15]); + MONT_MAC(v0, v1, v2, u[6], m[14]); + MONT_MAC(w0, w1, w2, u[7], m[13]); + MONT_MAC(t0, t1, t2, u[8], m[12]); + MONT_MAC(v0, v1, v2, u[9], m[11]); + MONT_MAC(w0, w1, w2, u[10], m[10]); + MONT_MAC(t0, t1, t2, u[11], m[9]); + MONT_MAC(v0, v1, v2, u[12], m[8]); + MONT_MAC(w0, w1, w2, u[13], m[7]); + MONT_MAC(t0, t1, t2, u[14], m[6]); + MONT_MAC(v0, v1, v2, u[15], m[5]); + MONT_MAC(w0, w1, w2, u[16], m[4]); + MONT_MAC(t0, t1, t2, u[17], m[3]); + MONT_MAC(v0, v1, v2, u[18], m[2]); + MONT_MAC(w0, w1, w2, u[19], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[20] = t0 * n0; + MONT_MAC(t0, t1, t2, u[20], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 21 */ + MONT_MUL(s0, s1, s2, a[0], a[21]); + MONT_MUL(c0, c1, c2, a[1], a[20]); + MONT_MAC(s0, s1, s2, a[2], a[19]); + MONT_MAC(c0, c1, c2, a[3], a[18]); + MONT_MAC(s0, s1, s2, a[4], a[17]); + MONT_MAC(c0, c1, c2, a[5], a[16]); + MONT_MAC(s0, s1, s2, a[6], a[15]); + MONT_MAC(c0, c1, c2, a[7], a[14]); + MONT_MAC(s0, s1, s2, a[8], a[13]); + MONT_MAC(c0, c1, c2, a[9], a[12]); + MONT_MAC(s0, s1, s2, a[10], a[11]); + MONT_MAC(t0, t1, t2, u[0], m[21]); + MONT_MUL(v0, v1, v2, u[1], m[20]); + MONT_MUL(w0, w1, w2, u[2], m[19]); + MONT_MAC(t0, t1, t2, u[3], m[18]); + MONT_MAC(v0, v1, v2, u[4], m[17]); + MONT_MAC(w0, w1, w2, u[5], m[16]); + MONT_MAC(t0, t1, t2, u[6], m[15]); + MONT_MAC(v0, v1, v2, u[7], m[14]); + MONT_MAC(w0, w1, w2, u[8], m[13]); + MONT_MAC(t0, t1, t2, u[9], m[12]); + MONT_MAC(v0, v1, v2, u[10], m[11]); + MONT_MAC(w0, w1, w2, u[11], m[10]); + MONT_MAC(t0, t1, t2, u[12], m[9]); + MONT_MAC(v0, v1, v2, u[13], m[8]); + MONT_MAC(w0, w1, w2, u[14], m[7]); + MONT_MAC(t0, t1, t2, u[15], m[6]); + MONT_MAC(v0, v1, v2, u[16], m[5]); + MONT_MAC(w0, w1, w2, u[17], m[4]); + MONT_MAC(t0, t1, t2, u[18], m[3]); + MONT_MAC(v0, v1, v2, u[19], m[2]); + MONT_MAC(w0, w1, w2, u[20], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[21] = t0 * n0; + MONT_MAC(t0, t1, t2, u[21], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 22 */ + MONT_MUL(s0, s1, s2, a[0], a[22]); + MONT_MUL(c0, c1, c2, a[1], a[21]); + MONT_MAC(s0, s1, s2, a[2], a[20]); + MONT_MAC(c0, c1, c2, a[3], a[19]); + MONT_MAC(s0, s1, s2, a[4], a[18]); + MONT_MAC(c0, c1, c2, a[5], a[17]); + MONT_MAC(s0, s1, s2, a[6], a[16]); + MONT_MAC(c0, c1, c2, a[7], a[15]); + MONT_MAC(s0, s1, s2, a[8], a[14]); + MONT_MAC(c0, c1, c2, a[9], a[13]); + MONT_MAC(s0, s1, s2, a[10], a[12]); + MONT_MAC(t0, t1, t2, a[11], a[11]); + MONT_MUL(v0, v1, v2, u[0], m[22]); + MONT_MUL(w0, w1, w2, u[1], m[21]); + MONT_MAC(t0, t1, t2, u[2], m[20]); + MONT_MAC(v0, v1, v2, u[3], m[19]); + MONT_MAC(w0, w1, w2, u[4], m[18]); + MONT_MAC(t0, t1, t2, u[5], m[17]); + MONT_MAC(v0, v1, v2, u[6], m[16]); + MONT_MAC(w0, w1, w2, u[7], m[15]); + MONT_MAC(t0, t1, t2, u[8], m[14]); + MONT_MAC(v0, v1, v2, u[9], m[13]); + MONT_MAC(w0, w1, w2, u[10], m[12]); + MONT_MAC(t0, t1, t2, u[11], m[11]); + MONT_MAC(v0, v1, v2, u[12], m[10]); + MONT_MAC(w0, w1, w2, u[13], m[9]); + MONT_MAC(t0, t1, t2, u[14], m[8]); + MONT_MAC(v0, v1, v2, u[15], m[7]); + MONT_MAC(w0, w1, w2, u[16], m[6]); + MONT_MAC(t0, t1, t2, u[17], m[5]); + MONT_MAC(v0, v1, v2, u[18], m[4]); + MONT_MAC(w0, w1, w2, u[19], m[3]); + MONT_MAC(t0, t1, t2, u[20], m[2]); + MONT_MAC(v0, v1, v2, u[21], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[22] = t0 * n0; + MONT_MAC(t0, t1, t2, u[22], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 23 */ + MONT_MUL(s0, s1, s2, a[0], a[23]); + MONT_MUL(c0, c1, c2, a[1], a[22]); + MONT_MAC(s0, s1, s2, a[2], a[21]); + MONT_MAC(c0, c1, c2, a[3], a[20]); + MONT_MAC(s0, s1, s2, a[4], a[19]); + MONT_MAC(c0, c1, c2, a[5], a[18]); + MONT_MAC(s0, s1, s2, a[6], a[17]); + MONT_MAC(c0, c1, c2, a[7], a[16]); + MONT_MAC(s0, s1, s2, a[8], a[15]); + MONT_MAC(c0, c1, c2, a[9], a[14]); + MONT_MAC(s0, s1, s2, a[10], a[13]); + MONT_MAC(c0, c1, c2, a[11], a[12]); + MONT_MAC(t0, t1, t2, u[0], m[23]); + MONT_MUL(v0, v1, v2, u[1], m[22]); + MONT_MUL(w0, w1, w2, u[2], m[21]); + MONT_MAC(t0, t1, t2, u[3], m[20]); + MONT_MAC(v0, v1, v2, u[4], m[19]); + MONT_MAC(w0, w1, w2, u[5], m[18]); + MONT_MAC(t0, t1, t2, u[6], m[17]); + MONT_MAC(v0, v1, v2, u[7], m[16]); + MONT_MAC(w0, w1, w2, u[8], m[15]); + MONT_MAC(t0, t1, t2, u[9], m[14]); + MONT_MAC(v0, v1, v2, u[10], m[13]); + MONT_MAC(w0, w1, w2, u[11], m[12]); + MONT_MAC(t0, t1, t2, u[12], m[11]); + MONT_MAC(v0, v1, v2, u[13], m[10]); + MONT_MAC(w0, w1, w2, u[14], m[9]); + MONT_MAC(t0, t1, t2, u[15], m[8]); + MONT_MAC(v0, v1, v2, u[16], m[7]); + MONT_MAC(w0, w1, w2, u[17], m[6]); + MONT_MAC(t0, t1, t2, u[18], m[5]); + MONT_MAC(v0, v1, v2, u[19], m[4]); + MONT_MAC(w0, w1, w2, u[20], m[3]); + MONT_MAC(t0, t1, t2, u[21], m[2]); + MONT_MAC(v0, v1, v2, u[22], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[23] = t0 * n0; + MONT_MAC(t0, t1, t2, u[23], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 24 */ + MONT_MUL(s0, s1, s2, a[0], a[24]); + MONT_MUL(c0, c1, c2, a[1], a[23]); + MONT_MAC(s0, s1, s2, a[2], a[22]); + MONT_MAC(c0, c1, c2, a[3], a[21]); + MONT_MAC(s0, s1, s2, a[4], a[20]); + MONT_MAC(c0, c1, c2, a[5], a[19]); + MONT_MAC(s0, s1, s2, a[6], a[18]); + MONT_MAC(c0, c1, c2, a[7], a[17]); + MONT_MAC(s0, s1, s2, a[8], a[16]); + MONT_MAC(c0, c1, c2, a[9], a[15]); + MONT_MAC(s0, s1, s2, a[10], a[14]); + MONT_MAC(c0, c1, c2, a[11], a[13]); + MONT_MAC(t0, t1, t2, a[12], a[12]); + MONT_MUL(v0, v1, v2, u[0], m[24]); + MONT_MUL(w0, w1, w2, u[1], m[23]); + MONT_MAC(t0, t1, t2, u[2], m[22]); + MONT_MAC(v0, v1, v2, u[3], m[21]); + MONT_MAC(w0, w1, w2, u[4], m[20]); + MONT_MAC(t0, t1, t2, u[5], m[19]); + MONT_MAC(v0, v1, v2, u[6], m[18]); + MONT_MAC(w0, w1, w2, u[7], m[17]); + MONT_MAC(t0, t1, t2, u[8], m[16]); + MONT_MAC(v0, v1, v2, u[9], m[15]); + MONT_MAC(w0, w1, w2, u[10], m[14]); + MONT_MAC(t0, t1, t2, u[11], m[13]); + MONT_MAC(v0, v1, v2, u[12], m[12]); + MONT_MAC(w0, w1, w2, u[13], m[11]); + MONT_MAC(t0, t1, t2, u[14], m[10]); + MONT_MAC(v0, v1, v2, u[15], m[9]); + MONT_MAC(w0, w1, w2, u[16], m[8]); + MONT_MAC(t0, t1, t2, u[17], m[7]); + MONT_MAC(v0, v1, v2, u[18], m[6]); + MONT_MAC(w0, w1, w2, u[19], m[5]); + MONT_MAC(t0, t1, t2, u[20], m[4]); + MONT_MAC(v0, v1, v2, u[21], m[3]); + MONT_MAC(w0, w1, w2, u[22], m[2]); + MONT_MAC(t0, t1, t2, u[23], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[24] = t0 * n0; + MONT_MAC(t0, t1, t2, u[24], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 25 */ + MONT_MUL(s0, s1, s2, a[0], a[25]); + MONT_MUL(c0, c1, c2, a[1], a[24]); + MONT_MAC(s0, s1, s2, a[2], a[23]); + MONT_MAC(c0, c1, c2, a[3], a[22]); + MONT_MAC(s0, s1, s2, a[4], a[21]); + MONT_MAC(c0, c1, c2, a[5], a[20]); + MONT_MAC(s0, s1, s2, a[6], a[19]); + MONT_MAC(c0, c1, c2, a[7], a[18]); + MONT_MAC(s0, s1, s2, a[8], a[17]); + MONT_MAC(c0, c1, c2, a[9], a[16]); + MONT_MAC(s0, s1, s2, a[10], a[15]); + MONT_MAC(c0, c1, c2, a[11], a[14]); + MONT_MAC(s0, s1, s2, a[12], a[13]); + MONT_MAC(t0, t1, t2, u[0], m[25]); + MONT_MUL(v0, v1, v2, u[1], m[24]); + MONT_MUL(w0, w1, w2, u[2], m[23]); + MONT_MAC(t0, t1, t2, u[3], m[22]); + MONT_MAC(v0, v1, v2, u[4], m[21]); + MONT_MAC(w0, w1, w2, u[5], m[20]); + MONT_MAC(t0, t1, t2, u[6], m[19]); + MONT_MAC(v0, v1, v2, u[7], m[18]); + MONT_MAC(w0, w1, w2, u[8], m[17]); + MONT_MAC(t0, t1, t2, u[9], m[16]); + MONT_MAC(v0, v1, v2, u[10], m[15]); + MONT_MAC(w0, w1, w2, u[11], m[14]); + MONT_MAC(t0, t1, t2, u[12], m[13]); + MONT_MAC(v0, v1, v2, u[13], m[12]); + MONT_MAC(w0, w1, w2, u[14], m[11]); + MONT_MAC(t0, t1, t2, u[15], m[10]); + MONT_MAC(v0, v1, v2, u[16], m[9]); + MONT_MAC(w0, w1, w2, u[17], m[8]); + MONT_MAC(t0, t1, t2, u[18], m[7]); + MONT_MAC(v0, v1, v2, u[19], m[6]); + MONT_MAC(w0, w1, w2, u[20], m[5]); + MONT_MAC(t0, t1, t2, u[21], m[4]); + MONT_MAC(v0, v1, v2, u[22], m[3]); + MONT_MAC(w0, w1, w2, u[23], m[2]); + MONT_MAC(t0, t1, t2, u[24], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[25] = t0 * n0; + MONT_MAC(t0, t1, t2, u[25], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 26 */ + MONT_MUL(s0, s1, s2, a[0], a[26]); + MONT_MUL(c0, c1, c2, a[1], a[25]); + MONT_MAC(s0, s1, s2, a[2], a[24]); + MONT_MAC(c0, c1, c2, a[3], a[23]); + MONT_MAC(s0, s1, s2, a[4], a[22]); + MONT_MAC(c0, c1, c2, a[5], a[21]); + MONT_MAC(s0, s1, s2, a[6], a[20]); + MONT_MAC(c0, c1, c2, a[7], a[19]); + MONT_MAC(s0, s1, s2, a[8], a[18]); + MONT_MAC(c0, c1, c2, a[9], a[17]); + MONT_MAC(s0, s1, s2, a[10], a[16]); + MONT_MAC(c0, c1, c2, a[11], a[15]); + MONT_MAC(s0, s1, s2, a[12], a[14]); + MONT_MAC(t0, t1, t2, a[13], a[13]); + MONT_MUL(v0, v1, v2, u[0], m[26]); + MONT_MUL(w0, w1, w2, u[1], m[25]); + MONT_MAC(t0, t1, t2, u[2], m[24]); + MONT_MAC(v0, v1, v2, u[3], m[23]); + MONT_MAC(w0, w1, w2, u[4], m[22]); + MONT_MAC(t0, t1, t2, u[5], m[21]); + MONT_MAC(v0, v1, v2, u[6], m[20]); + MONT_MAC(w0, w1, w2, u[7], m[19]); + MONT_MAC(t0, t1, t2, u[8], m[18]); + MONT_MAC(v0, v1, v2, u[9], m[17]); + MONT_MAC(w0, w1, w2, u[10], m[16]); + MONT_MAC(t0, t1, t2, u[11], m[15]); + MONT_MAC(v0, v1, v2, u[12], m[14]); + MONT_MAC(w0, w1, w2, u[13], m[13]); + MONT_MAC(t0, t1, t2, u[14], m[12]); + MONT_MAC(v0, v1, v2, u[15], m[11]); + MONT_MAC(w0, w1, w2, u[16], m[10]); + MONT_MAC(t0, t1, t2, u[17], m[9]); + MONT_MAC(v0, v1, v2, u[18], m[8]); + MONT_MAC(w0, w1, w2, u[19], m[7]); + MONT_MAC(t0, t1, t2, u[20], m[6]); + MONT_MAC(v0, v1, v2, u[21], m[5]); + MONT_MAC(w0, w1, w2, u[22], m[4]); + MONT_MAC(t0, t1, t2, u[23], m[3]); + MONT_MAC(v0, v1, v2, u[24], m[2]); + MONT_MAC(w0, w1, w2, u[25], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[26] = t0 * n0; + MONT_MAC(t0, t1, t2, u[26], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 27 */ + MONT_MUL(s0, s1, s2, a[0], a[27]); + MONT_MUL(c0, c1, c2, a[1], a[26]); + MONT_MAC(s0, s1, s2, a[2], a[25]); + MONT_MAC(c0, c1, c2, a[3], a[24]); + MONT_MAC(s0, s1, s2, a[4], a[23]); + MONT_MAC(c0, c1, c2, a[5], a[22]); + MONT_MAC(s0, s1, s2, a[6], a[21]); + MONT_MAC(c0, c1, c2, a[7], a[20]); + MONT_MAC(s0, s1, s2, a[8], a[19]); + MONT_MAC(c0, c1, c2, a[9], a[18]); + MONT_MAC(s0, s1, s2, a[10], a[17]); + MONT_MAC(c0, c1, c2, a[11], a[16]); + MONT_MAC(s0, s1, s2, a[12], a[15]); + MONT_MAC(c0, c1, c2, a[13], a[14]); + MONT_MAC(t0, t1, t2, u[0], m[27]); + MONT_MUL(v0, v1, v2, u[1], m[26]); + MONT_MUL(w0, w1, w2, u[2], m[25]); + MONT_MAC(t0, t1, t2, u[3], m[24]); + MONT_MAC(v0, v1, v2, u[4], m[23]); + MONT_MAC(w0, w1, w2, u[5], m[22]); + MONT_MAC(t0, t1, t2, u[6], m[21]); + MONT_MAC(v0, v1, v2, u[7], m[20]); + MONT_MAC(w0, w1, w2, u[8], m[19]); + MONT_MAC(t0, t1, t2, u[9], m[18]); + MONT_MAC(v0, v1, v2, u[10], m[17]); + MONT_MAC(w0, w1, w2, u[11], m[16]); + MONT_MAC(t0, t1, t2, u[12], m[15]); + MONT_MAC(v0, v1, v2, u[13], m[14]); + MONT_MAC(w0, w1, w2, u[14], m[13]); + MONT_MAC(t0, t1, t2, u[15], m[12]); + MONT_MAC(v0, v1, v2, u[16], m[11]); + MONT_MAC(w0, w1, w2, u[17], m[10]); + MONT_MAC(t0, t1, t2, u[18], m[9]); + MONT_MAC(v0, v1, v2, u[19], m[8]); + MONT_MAC(w0, w1, w2, u[20], m[7]); + MONT_MAC(t0, t1, t2, u[21], m[6]); + MONT_MAC(v0, v1, v2, u[22], m[5]); + MONT_MAC(w0, w1, w2, u[23], m[4]); + MONT_MAC(t0, t1, t2, u[24], m[3]); + MONT_MAC(v0, v1, v2, u[25], m[2]); + MONT_MAC(w0, w1, w2, u[26], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[27] = t0 * n0; + MONT_MAC(t0, t1, t2, u[27], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 28 */ + MONT_MUL(s0, s1, s2, a[0], a[28]); + MONT_MUL(c0, c1, c2, a[1], a[27]); + MONT_MAC(s0, s1, s2, a[2], a[26]); + MONT_MAC(c0, c1, c2, a[3], a[25]); + MONT_MAC(s0, s1, s2, a[4], a[24]); + MONT_MAC(c0, c1, c2, a[5], a[23]); + MONT_MAC(s0, s1, s2, a[6], a[22]); + MONT_MAC(c0, c1, c2, a[7], a[21]); + MONT_MAC(s0, s1, s2, a[8], a[20]); + MONT_MAC(c0, c1, c2, a[9], a[19]); + MONT_MAC(s0, s1, s2, a[10], a[18]); + MONT_MAC(c0, c1, c2, a[11], a[17]); + MONT_MAC(s0, s1, s2, a[12], a[16]); + MONT_MAC(c0, c1, c2, a[13], a[15]); + MONT_MAC(t0, t1, t2, a[14], a[14]); + MONT_MUL(v0, v1, v2, u[0], m[28]); + MONT_MUL(w0, w1, w2, u[1], m[27]); + MONT_MAC(t0, t1, t2, u[2], m[26]); + MONT_MAC(v0, v1, v2, u[3], m[25]); + MONT_MAC(w0, w1, w2, u[4], m[24]); + MONT_MAC(t0, t1, t2, u[5], m[23]); + MONT_MAC(v0, v1, v2, u[6], m[22]); + MONT_MAC(w0, w1, w2, u[7], m[21]); + MONT_MAC(t0, t1, t2, u[8], m[20]); + MONT_MAC(v0, v1, v2, u[9], m[19]); + MONT_MAC(w0, w1, w2, u[10], m[18]); + MONT_MAC(t0, t1, t2, u[11], m[17]); + MONT_MAC(v0, v1, v2, u[12], m[16]); + MONT_MAC(w0, w1, w2, u[13], m[15]); + MONT_MAC(t0, t1, t2, u[14], m[14]); + MONT_MAC(v0, v1, v2, u[15], m[13]); + MONT_MAC(w0, w1, w2, u[16], m[12]); + MONT_MAC(t0, t1, t2, u[17], m[11]); + MONT_MAC(v0, v1, v2, u[18], m[10]); + MONT_MAC(w0, w1, w2, u[19], m[9]); + MONT_MAC(t0, t1, t2, u[20], m[8]); + MONT_MAC(v0, v1, v2, u[21], m[7]); + MONT_MAC(w0, w1, w2, u[22], m[6]); + MONT_MAC(t0, t1, t2, u[23], m[5]); + MONT_MAC(v0, v1, v2, u[24], m[4]); + MONT_MAC(w0, w1, w2, u[25], m[3]); + MONT_MAC(t0, t1, t2, u[26], m[2]); + MONT_MAC(v0, v1, v2, u[27], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[28] = t0 * n0; + MONT_MAC(t0, t1, t2, u[28], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 29 */ + MONT_MUL(s0, s1, s2, a[0], a[29]); + MONT_MUL(c0, c1, c2, a[1], a[28]); + MONT_MAC(s0, s1, s2, a[2], a[27]); + MONT_MAC(c0, c1, c2, a[3], a[26]); + MONT_MAC(s0, s1, s2, a[4], a[25]); + MONT_MAC(c0, c1, c2, a[5], a[24]); + MONT_MAC(s0, s1, s2, a[6], a[23]); + MONT_MAC(c0, c1, c2, a[7], a[22]); + MONT_MAC(s0, s1, s2, a[8], a[21]); + MONT_MAC(c0, c1, c2, a[9], a[20]); + MONT_MAC(s0, s1, s2, a[10], a[19]); + MONT_MAC(c0, c1, c2, a[11], a[18]); + MONT_MAC(s0, s1, s2, a[12], a[17]); + MONT_MAC(c0, c1, c2, a[13], a[16]); + MONT_MAC(s0, s1, s2, a[14], a[15]); + MONT_MAC(t0, t1, t2, u[0], m[29]); + MONT_MUL(v0, v1, v2, u[1], m[28]); + MONT_MUL(w0, w1, w2, u[2], m[27]); + MONT_MAC(t0, t1, t2, u[3], m[26]); + MONT_MAC(v0, v1, v2, u[4], m[25]); + MONT_MAC(w0, w1, w2, u[5], m[24]); + MONT_MAC(t0, t1, t2, u[6], m[23]); + MONT_MAC(v0, v1, v2, u[7], m[22]); + MONT_MAC(w0, w1, w2, u[8], m[21]); + MONT_MAC(t0, t1, t2, u[9], m[20]); + MONT_MAC(v0, v1, v2, u[10], m[19]); + MONT_MAC(w0, w1, w2, u[11], m[18]); + MONT_MAC(t0, t1, t2, u[12], m[17]); + MONT_MAC(v0, v1, v2, u[13], m[16]); + MONT_MAC(w0, w1, w2, u[14], m[15]); + MONT_MAC(t0, t1, t2, u[15], m[14]); + MONT_MAC(v0, v1, v2, u[16], m[13]); + MONT_MAC(w0, w1, w2, u[17], m[12]); + MONT_MAC(t0, t1, t2, u[18], m[11]); + MONT_MAC(v0, v1, v2, u[19], m[10]); + MONT_MAC(w0, w1, w2, u[20], m[9]); + MONT_MAC(t0, t1, t2, u[21], m[8]); + MONT_MAC(v0, v1, v2, u[22], m[7]); + MONT_MAC(w0, w1, w2, u[23], m[6]); + MONT_MAC(t0, t1, t2, u[24], m[5]); + MONT_MAC(v0, v1, v2, u[25], m[4]); + MONT_MAC(w0, w1, w2, u[26], m[3]); + MONT_MAC(t0, t1, t2, u[27], m[2]); + MONT_MAC(v0, v1, v2, u[28], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[29] = t0 * n0; + MONT_MAC(t0, t1, t2, u[29], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 30 */ + MONT_MUL(s0, s1, s2, a[0], a[30]); + MONT_MUL(c0, c1, c2, a[1], a[29]); + MONT_MAC(s0, s1, s2, a[2], a[28]); + MONT_MAC(c0, c1, c2, a[3], a[27]); + MONT_MAC(s0, s1, s2, a[4], a[26]); + MONT_MAC(c0, c1, c2, a[5], a[25]); + MONT_MAC(s0, s1, s2, a[6], a[24]); + MONT_MAC(c0, c1, c2, a[7], a[23]); + MONT_MAC(s0, s1, s2, a[8], a[22]); + MONT_MAC(c0, c1, c2, a[9], a[21]); + MONT_MAC(s0, s1, s2, a[10], a[20]); + MONT_MAC(c0, c1, c2, a[11], a[19]); + MONT_MAC(s0, s1, s2, a[12], a[18]); + MONT_MAC(c0, c1, c2, a[13], a[17]); + MONT_MAC(s0, s1, s2, a[14], a[16]); + MONT_MAC(t0, t1, t2, a[15], a[15]); + MONT_MUL(v0, v1, v2, u[0], m[30]); + MONT_MUL(w0, w1, w2, u[1], m[29]); + MONT_MAC(t0, t1, t2, u[2], m[28]); + MONT_MAC(v0, v1, v2, u[3], m[27]); + MONT_MAC(w0, w1, w2, u[4], m[26]); + MONT_MAC(t0, t1, t2, u[5], m[25]); + MONT_MAC(v0, v1, v2, u[6], m[24]); + MONT_MAC(w0, w1, w2, u[7], m[23]); + MONT_MAC(t0, t1, t2, u[8], m[22]); + MONT_MAC(v0, v1, v2, u[9], m[21]); + MONT_MAC(w0, w1, w2, u[10], m[20]); + MONT_MAC(t0, t1, t2, u[11], m[19]); + MONT_MAC(v0, v1, v2, u[12], m[18]); + MONT_MAC(w0, w1, w2, u[13], m[17]); + MONT_MAC(t0, t1, t2, u[14], m[16]); + MONT_MAC(v0, v1, v2, u[15], m[15]); + MONT_MAC(w0, w1, w2, u[16], m[14]); + MONT_MAC(t0, t1, t2, u[17], m[13]); + MONT_MAC(v0, v1, v2, u[18], m[12]); + MONT_MAC(w0, w1, w2, u[19], m[11]); + MONT_MAC(t0, t1, t2, u[20], m[10]); + MONT_MAC(v0, v1, v2, u[21], m[9]); + MONT_MAC(w0, w1, w2, u[22], m[8]); + MONT_MAC(t0, t1, t2, u[23], m[7]); + MONT_MAC(v0, v1, v2, u[24], m[6]); + MONT_MAC(w0, w1, w2, u[25], m[5]); + MONT_MAC(t0, t1, t2, u[26], m[4]); + MONT_MAC(v0, v1, v2, u[27], m[3]); + MONT_MAC(w0, w1, w2, u[28], m[2]); + MONT_MAC(t0, t1, t2, u[29], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[30] = t0 * n0; + MONT_MAC(t0, t1, t2, u[30], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 31 */ + MONT_MUL(s0, s1, s2, a[0], a[31]); + MONT_MUL(c0, c1, c2, a[1], a[30]); + MONT_MAC(s0, s1, s2, a[2], a[29]); + MONT_MAC(c0, c1, c2, a[3], a[28]); + MONT_MAC(s0, s1, s2, a[4], a[27]); + MONT_MAC(c0, c1, c2, a[5], a[26]); + MONT_MAC(s0, s1, s2, a[6], a[25]); + MONT_MAC(c0, c1, c2, a[7], a[24]); + MONT_MAC(s0, s1, s2, a[8], a[23]); + MONT_MAC(c0, c1, c2, a[9], a[22]); + MONT_MAC(s0, s1, s2, a[10], a[21]); + MONT_MAC(c0, c1, c2, a[11], a[20]); + MONT_MAC(s0, s1, s2, a[12], a[19]); + MONT_MAC(c0, c1, c2, a[13], a[18]); + MONT_MAC(s0, s1, s2, a[14], a[17]); + MONT_MAC(c0, c1, c2, a[15], a[16]); + MONT_MAC(t0, t1, t2, u[0], m[31]); + MONT_MUL(v0, v1, v2, u[1], m[30]); + MONT_MUL(w0, w1, w2, u[2], m[29]); + MONT_MAC(t0, t1, t2, u[3], m[28]); + MONT_MAC(v0, v1, v2, u[4], m[27]); + MONT_MAC(w0, w1, w2, u[5], m[26]); + MONT_MAC(t0, t1, t2, u[6], m[25]); + MONT_MAC(v0, v1, v2, u[7], m[24]); + MONT_MAC(w0, w1, w2, u[8], m[23]); + MONT_MAC(t0, t1, t2, u[9], m[22]); + MONT_MAC(v0, v1, v2, u[10], m[21]); + MONT_MAC(w0, w1, w2, u[11], m[20]); + MONT_MAC(t0, t1, t2, u[12], m[19]); + MONT_MAC(v0, v1, v2, u[13], m[18]); + MONT_MAC(w0, w1, w2, u[14], m[17]); + MONT_MAC(t0, t1, t2, u[15], m[16]); + MONT_MAC(v0, v1, v2, u[16], m[15]); + MONT_MAC(w0, w1, w2, u[17], m[14]); + MONT_MAC(t0, t1, t2, u[18], m[13]); + MONT_MAC(v0, v1, v2, u[19], m[12]); + MONT_MAC(w0, w1, w2, u[20], m[11]); + MONT_MAC(t0, t1, t2, u[21], m[10]); + MONT_MAC(v0, v1, v2, u[22], m[9]); + MONT_MAC(w0, w1, w2, u[23], m[8]); + MONT_MAC(t0, t1, t2, u[24], m[7]); + MONT_MAC(v0, v1, v2, u[25], m[6]); + MONT_MAC(w0, w1, w2, u[26], m[5]); + MONT_MAC(t0, t1, t2, u[27], m[4]); + MONT_MAC(v0, v1, v2, u[28], m[3]); + MONT_MAC(w0, w1, w2, u[29], m[2]); + MONT_MAC(t0, t1, t2, u[30], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[31] = t0 * n0; + MONT_MAC(t0, t1, t2, u[31], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 32 */ + MONT_MUL(s0, s1, s2, a[1], a[31]); + MONT_MUL(c0, c1, c2, a[2], a[30]); + MONT_MAC(s0, s1, s2, a[3], a[29]); + MONT_MAC(c0, c1, c2, a[4], a[28]); + MONT_MAC(s0, s1, s2, a[5], a[27]); + MONT_MAC(c0, c1, c2, a[6], a[26]); + MONT_MAC(s0, s1, s2, a[7], a[25]); + MONT_MAC(c0, c1, c2, a[8], a[24]); + MONT_MAC(s0, s1, s2, a[9], a[23]); + MONT_MAC(c0, c1, c2, a[10], a[22]); + MONT_MAC(s0, s1, s2, a[11], a[21]); + MONT_MAC(c0, c1, c2, a[12], a[20]); + MONT_MAC(s0, s1, s2, a[13], a[19]); + MONT_MAC(c0, c1, c2, a[14], a[18]); + MONT_MAC(s0, s1, s2, a[15], a[17]); + MONT_MAC(t0, t1, t2, a[16], a[16]); + MONT_MUL(v0, v1, v2, u[1], m[31]); + MONT_MUL(w0, w1, w2, u[2], m[30]); + MONT_MAC(t0, t1, t2, u[3], m[29]); + MONT_MAC(v0, v1, v2, u[4], m[28]); + MONT_MAC(w0, w1, w2, u[5], m[27]); + MONT_MAC(t0, t1, t2, u[6], m[26]); + MONT_MAC(v0, v1, v2, u[7], m[25]); + MONT_MAC(w0, w1, w2, u[8], m[24]); + MONT_MAC(t0, t1, t2, u[9], m[23]); + MONT_MAC(v0, v1, v2, u[10], m[22]); + MONT_MAC(w0, w1, w2, u[11], m[21]); + MONT_MAC(t0, t1, t2, u[12], m[20]); + MONT_MAC(v0, v1, v2, u[13], m[19]); + MONT_MAC(w0, w1, w2, u[14], m[18]); + MONT_MAC(t0, t1, t2, u[15], m[17]); + MONT_MAC(v0, v1, v2, u[16], m[16]); + MONT_MAC(w0, w1, w2, u[17], m[15]); + MONT_MAC(t0, t1, t2, u[18], m[14]); + MONT_MAC(v0, v1, v2, u[19], m[13]); + MONT_MAC(w0, w1, w2, u[20], m[12]); + MONT_MAC(t0, t1, t2, u[21], m[11]); + MONT_MAC(v0, v1, v2, u[22], m[10]); + MONT_MAC(w0, w1, w2, u[23], m[9]); + MONT_MAC(t0, t1, t2, u[24], m[8]); + MONT_MAC(v0, v1, v2, u[25], m[7]); + MONT_MAC(w0, w1, w2, u[26], m[6]); + MONT_MAC(t0, t1, t2, u[27], m[5]); + MONT_MAC(v0, v1, v2, u[28], m[4]); + MONT_MAC(w0, w1, w2, u[29], m[3]); + MONT_MAC(t0, t1, t2, u[30], m[2]); + MONT_MAC(v0, v1, v2, u[31], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[0] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 33 */ + MONT_MUL(s0, s1, s2, a[2], a[31]); + MONT_MUL(c0, c1, c2, a[3], a[30]); + MONT_MAC(s0, s1, s2, a[4], a[29]); + MONT_MAC(c0, c1, c2, a[5], a[28]); + MONT_MAC(s0, s1, s2, a[6], a[27]); + MONT_MAC(c0, c1, c2, a[7], a[26]); + MONT_MAC(s0, s1, s2, a[8], a[25]); + MONT_MAC(c0, c1, c2, a[9], a[24]); + MONT_MAC(s0, s1, s2, a[10], a[23]); + MONT_MAC(c0, c1, c2, a[11], a[22]); + MONT_MAC(s0, s1, s2, a[12], a[21]); + MONT_MAC(c0, c1, c2, a[13], a[20]); + MONT_MAC(s0, s1, s2, a[14], a[19]); + MONT_MAC(c0, c1, c2, a[15], a[18]); + MONT_MAC(s0, s1, s2, a[16], a[17]); + MONT_MAC(t0, t1, t2, u[2], m[31]); + MONT_MUL(v0, v1, v2, u[3], m[30]); + MONT_MUL(w0, w1, w2, u[4], m[29]); + MONT_MAC(t0, t1, t2, u[5], m[28]); + MONT_MAC(v0, v1, v2, u[6], m[27]); + MONT_MAC(w0, w1, w2, u[7], m[26]); + MONT_MAC(t0, t1, t2, u[8], m[25]); + MONT_MAC(v0, v1, v2, u[9], m[24]); + MONT_MAC(w0, w1, w2, u[10], m[23]); + MONT_MAC(t0, t1, t2, u[11], m[22]); + MONT_MAC(v0, v1, v2, u[12], m[21]); + MONT_MAC(w0, w1, w2, u[13], m[20]); + MONT_MAC(t0, t1, t2, u[14], m[19]); + MONT_MAC(v0, v1, v2, u[15], m[18]); + MONT_MAC(w0, w1, w2, u[16], m[17]); + MONT_MAC(t0, t1, t2, u[17], m[16]); + MONT_MAC(v0, v1, v2, u[18], m[15]); + MONT_MAC(w0, w1, w2, u[19], m[14]); + MONT_MAC(t0, t1, t2, u[20], m[13]); + MONT_MAC(v0, v1, v2, u[21], m[12]); + MONT_MAC(w0, w1, w2, u[22], m[11]); + MONT_MAC(t0, t1, t2, u[23], m[10]); + MONT_MAC(v0, v1, v2, u[24], m[9]); + MONT_MAC(w0, w1, w2, u[25], m[8]); + MONT_MAC(t0, t1, t2, u[26], m[7]); + MONT_MAC(v0, v1, v2, u[27], m[6]); + MONT_MAC(w0, w1, w2, u[28], m[5]); + MONT_MAC(t0, t1, t2, u[29], m[4]); + MONT_MAC(v0, v1, v2, u[30], m[3]); + MONT_MAC(w0, w1, w2, u[31], m[2]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[1] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 34 */ + MONT_MUL(s0, s1, s2, a[3], a[31]); + MONT_MUL(c0, c1, c2, a[4], a[30]); + MONT_MAC(s0, s1, s2, a[5], a[29]); + MONT_MAC(c0, c1, c2, a[6], a[28]); + MONT_MAC(s0, s1, s2, a[7], a[27]); + MONT_MAC(c0, c1, c2, a[8], a[26]); + MONT_MAC(s0, s1, s2, a[9], a[25]); + MONT_MAC(c0, c1, c2, a[10], a[24]); + MONT_MAC(s0, s1, s2, a[11], a[23]); + MONT_MAC(c0, c1, c2, a[12], a[22]); + MONT_MAC(s0, s1, s2, a[13], a[21]); + MONT_MAC(c0, c1, c2, a[14], a[20]); + MONT_MAC(s0, s1, s2, a[15], a[19]); + MONT_MAC(c0, c1, c2, a[16], a[18]); + MONT_MAC(t0, t1, t2, a[17], a[17]); + MONT_MUL(v0, v1, v2, u[3], m[31]); + MONT_MUL(w0, w1, w2, u[4], m[30]); + MONT_MAC(t0, t1, t2, u[5], m[29]); + MONT_MAC(v0, v1, v2, u[6], m[28]); + MONT_MAC(w0, w1, w2, u[7], m[27]); + MONT_MAC(t0, t1, t2, u[8], m[26]); + MONT_MAC(v0, v1, v2, u[9], m[25]); + MONT_MAC(w0, w1, w2, u[10], m[24]); + MONT_MAC(t0, t1, t2, u[11], m[23]); + MONT_MAC(v0, v1, v2, u[12], m[22]); + MONT_MAC(w0, w1, w2, u[13], m[21]); + MONT_MAC(t0, t1, t2, u[14], m[20]); + MONT_MAC(v0, v1, v2, u[15], m[19]); + MONT_MAC(w0, w1, w2, u[16], m[18]); + MONT_MAC(t0, t1, t2, u[17], m[17]); + MONT_MAC(v0, v1, v2, u[18], m[16]); + MONT_MAC(w0, w1, w2, u[19], m[15]); + MONT_MAC(t0, t1, t2, u[20], m[14]); + MONT_MAC(v0, v1, v2, u[21], m[13]); + MONT_MAC(w0, w1, w2, u[22], m[12]); + MONT_MAC(t0, t1, t2, u[23], m[11]); + MONT_MAC(v0, v1, v2, u[24], m[10]); + MONT_MAC(w0, w1, w2, u[25], m[9]); + MONT_MAC(t0, t1, t2, u[26], m[8]); + MONT_MAC(v0, v1, v2, u[27], m[7]); + MONT_MAC(w0, w1, w2, u[28], m[6]); + MONT_MAC(t0, t1, t2, u[29], m[5]); + MONT_MAC(v0, v1, v2, u[30], m[4]); + MONT_MAC(w0, w1, w2, u[31], m[3]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[2] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 35 */ + MONT_MUL(s0, s1, s2, a[4], a[31]); + MONT_MUL(c0, c1, c2, a[5], a[30]); + MONT_MAC(s0, s1, s2, a[6], a[29]); + MONT_MAC(c0, c1, c2, a[7], a[28]); + MONT_MAC(s0, s1, s2, a[8], a[27]); + MONT_MAC(c0, c1, c2, a[9], a[26]); + MONT_MAC(s0, s1, s2, a[10], a[25]); + MONT_MAC(c0, c1, c2, a[11], a[24]); + MONT_MAC(s0, s1, s2, a[12], a[23]); + MONT_MAC(c0, c1, c2, a[13], a[22]); + MONT_MAC(s0, s1, s2, a[14], a[21]); + MONT_MAC(c0, c1, c2, a[15], a[20]); + MONT_MAC(s0, s1, s2, a[16], a[19]); + MONT_MAC(c0, c1, c2, a[17], a[18]); + MONT_MAC(t0, t1, t2, u[4], m[31]); + MONT_MUL(v0, v1, v2, u[5], m[30]); + MONT_MUL(w0, w1, w2, u[6], m[29]); + MONT_MAC(t0, t1, t2, u[7], m[28]); + MONT_MAC(v0, v1, v2, u[8], m[27]); + MONT_MAC(w0, w1, w2, u[9], m[26]); + MONT_MAC(t0, t1, t2, u[10], m[25]); + MONT_MAC(v0, v1, v2, u[11], m[24]); + MONT_MAC(w0, w1, w2, u[12], m[23]); + MONT_MAC(t0, t1, t2, u[13], m[22]); + MONT_MAC(v0, v1, v2, u[14], m[21]); + MONT_MAC(w0, w1, w2, u[15], m[20]); + MONT_MAC(t0, t1, t2, u[16], m[19]); + MONT_MAC(v0, v1, v2, u[17], m[18]); + MONT_MAC(w0, w1, w2, u[18], m[17]); + MONT_MAC(t0, t1, t2, u[19], m[16]); + MONT_MAC(v0, v1, v2, u[20], m[15]); + MONT_MAC(w0, w1, w2, u[21], m[14]); + MONT_MAC(t0, t1, t2, u[22], m[13]); + MONT_MAC(v0, v1, v2, u[23], m[12]); + MONT_MAC(w0, w1, w2, u[24], m[11]); + MONT_MAC(t0, t1, t2, u[25], m[10]); + MONT_MAC(v0, v1, v2, u[26], m[9]); + MONT_MAC(w0, w1, w2, u[27], m[8]); + MONT_MAC(t0, t1, t2, u[28], m[7]); + MONT_MAC(v0, v1, v2, u[29], m[6]); + MONT_MAC(w0, w1, w2, u[30], m[5]); + MONT_MAC(t0, t1, t2, u[31], m[4]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[3] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 36 */ + MONT_MUL(s0, s1, s2, a[5], a[31]); + MONT_MUL(c0, c1, c2, a[6], a[30]); + MONT_MAC(s0, s1, s2, a[7], a[29]); + MONT_MAC(c0, c1, c2, a[8], a[28]); + MONT_MAC(s0, s1, s2, a[9], a[27]); + MONT_MAC(c0, c1, c2, a[10], a[26]); + MONT_MAC(s0, s1, s2, a[11], a[25]); + MONT_MAC(c0, c1, c2, a[12], a[24]); + MONT_MAC(s0, s1, s2, a[13], a[23]); + MONT_MAC(c0, c1, c2, a[14], a[22]); + MONT_MAC(s0, s1, s2, a[15], a[21]); + MONT_MAC(c0, c1, c2, a[16], a[20]); + MONT_MAC(s0, s1, s2, a[17], a[19]); + MONT_MAC(t0, t1, t2, a[18], a[18]); + MONT_MUL(v0, v1, v2, u[5], m[31]); + MONT_MUL(w0, w1, w2, u[6], m[30]); + MONT_MAC(t0, t1, t2, u[7], m[29]); + MONT_MAC(v0, v1, v2, u[8], m[28]); + MONT_MAC(w0, w1, w2, u[9], m[27]); + MONT_MAC(t0, t1, t2, u[10], m[26]); + MONT_MAC(v0, v1, v2, u[11], m[25]); + MONT_MAC(w0, w1, w2, u[12], m[24]); + MONT_MAC(t0, t1, t2, u[13], m[23]); + MONT_MAC(v0, v1, v2, u[14], m[22]); + MONT_MAC(w0, w1, w2, u[15], m[21]); + MONT_MAC(t0, t1, t2, u[16], m[20]); + MONT_MAC(v0, v1, v2, u[17], m[19]); + MONT_MAC(w0, w1, w2, u[18], m[18]); + MONT_MAC(t0, t1, t2, u[19], m[17]); + MONT_MAC(v0, v1, v2, u[20], m[16]); + MONT_MAC(w0, w1, w2, u[21], m[15]); + MONT_MAC(t0, t1, t2, u[22], m[14]); + MONT_MAC(v0, v1, v2, u[23], m[13]); + MONT_MAC(w0, w1, w2, u[24], m[12]); + MONT_MAC(t0, t1, t2, u[25], m[11]); + MONT_MAC(v0, v1, v2, u[26], m[10]); + MONT_MAC(w0, w1, w2, u[27], m[9]); + MONT_MAC(t0, t1, t2, u[28], m[8]); + MONT_MAC(v0, v1, v2, u[29], m[7]); + MONT_MAC(w0, w1, w2, u[30], m[6]); + MONT_MAC(t0, t1, t2, u[31], m[5]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[4] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 37 */ + MONT_MUL(s0, s1, s2, a[6], a[31]); + MONT_MUL(c0, c1, c2, a[7], a[30]); + MONT_MAC(s0, s1, s2, a[8], a[29]); + MONT_MAC(c0, c1, c2, a[9], a[28]); + MONT_MAC(s0, s1, s2, a[10], a[27]); + MONT_MAC(c0, c1, c2, a[11], a[26]); + MONT_MAC(s0, s1, s2, a[12], a[25]); + MONT_MAC(c0, c1, c2, a[13], a[24]); + MONT_MAC(s0, s1, s2, a[14], a[23]); + MONT_MAC(c0, c1, c2, a[15], a[22]); + MONT_MAC(s0, s1, s2, a[16], a[21]); + MONT_MAC(c0, c1, c2, a[17], a[20]); + MONT_MAC(s0, s1, s2, a[18], a[19]); + MONT_MAC(t0, t1, t2, u[6], m[31]); + MONT_MUL(v0, v1, v2, u[7], m[30]); + MONT_MUL(w0, w1, w2, u[8], m[29]); + MONT_MAC(t0, t1, t2, u[9], m[28]); + MONT_MAC(v0, v1, v2, u[10], m[27]); + MONT_MAC(w0, w1, w2, u[11], m[26]); + MONT_MAC(t0, t1, t2, u[12], m[25]); + MONT_MAC(v0, v1, v2, u[13], m[24]); + MONT_MAC(w0, w1, w2, u[14], m[23]); + MONT_MAC(t0, t1, t2, u[15], m[22]); + MONT_MAC(v0, v1, v2, u[16], m[21]); + MONT_MAC(w0, w1, w2, u[17], m[20]); + MONT_MAC(t0, t1, t2, u[18], m[19]); + MONT_MAC(v0, v1, v2, u[19], m[18]); + MONT_MAC(w0, w1, w2, u[20], m[17]); + MONT_MAC(t0, t1, t2, u[21], m[16]); + MONT_MAC(v0, v1, v2, u[22], m[15]); + MONT_MAC(w0, w1, w2, u[23], m[14]); + MONT_MAC(t0, t1, t2, u[24], m[13]); + MONT_MAC(v0, v1, v2, u[25], m[12]); + MONT_MAC(w0, w1, w2, u[26], m[11]); + MONT_MAC(t0, t1, t2, u[27], m[10]); + MONT_MAC(v0, v1, v2, u[28], m[9]); + MONT_MAC(w0, w1, w2, u[29], m[8]); + MONT_MAC(t0, t1, t2, u[30], m[7]); + MONT_MAC(v0, v1, v2, u[31], m[6]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[5] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 38 */ + MONT_MUL(s0, s1, s2, a[7], a[31]); + MONT_MUL(c0, c1, c2, a[8], a[30]); + MONT_MAC(s0, s1, s2, a[9], a[29]); + MONT_MAC(c0, c1, c2, a[10], a[28]); + MONT_MAC(s0, s1, s2, a[11], a[27]); + MONT_MAC(c0, c1, c2, a[12], a[26]); + MONT_MAC(s0, s1, s2, a[13], a[25]); + MONT_MAC(c0, c1, c2, a[14], a[24]); + MONT_MAC(s0, s1, s2, a[15], a[23]); + MONT_MAC(c0, c1, c2, a[16], a[22]); + MONT_MAC(s0, s1, s2, a[17], a[21]); + MONT_MAC(c0, c1, c2, a[18], a[20]); + MONT_MAC(t0, t1, t2, a[19], a[19]); + MONT_MUL(v0, v1, v2, u[7], m[31]); + MONT_MUL(w0, w1, w2, u[8], m[30]); + MONT_MAC(t0, t1, t2, u[9], m[29]); + MONT_MAC(v0, v1, v2, u[10], m[28]); + MONT_MAC(w0, w1, w2, u[11], m[27]); + MONT_MAC(t0, t1, t2, u[12], m[26]); + MONT_MAC(v0, v1, v2, u[13], m[25]); + MONT_MAC(w0, w1, w2, u[14], m[24]); + MONT_MAC(t0, t1, t2, u[15], m[23]); + MONT_MAC(v0, v1, v2, u[16], m[22]); + MONT_MAC(w0, w1, w2, u[17], m[21]); + MONT_MAC(t0, t1, t2, u[18], m[20]); + MONT_MAC(v0, v1, v2, u[19], m[19]); + MONT_MAC(w0, w1, w2, u[20], m[18]); + MONT_MAC(t0, t1, t2, u[21], m[17]); + MONT_MAC(v0, v1, v2, u[22], m[16]); + MONT_MAC(w0, w1, w2, u[23], m[15]); + MONT_MAC(t0, t1, t2, u[24], m[14]); + MONT_MAC(v0, v1, v2, u[25], m[13]); + MONT_MAC(w0, w1, w2, u[26], m[12]); + MONT_MAC(t0, t1, t2, u[27], m[11]); + MONT_MAC(v0, v1, v2, u[28], m[10]); + MONT_MAC(w0, w1, w2, u[29], m[9]); + MONT_MAC(t0, t1, t2, u[30], m[8]); + MONT_MAC(v0, v1, v2, u[31], m[7]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[6] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 39 */ + MONT_MUL(s0, s1, s2, a[8], a[31]); + MONT_MUL(c0, c1, c2, a[9], a[30]); + MONT_MAC(s0, s1, s2, a[10], a[29]); + MONT_MAC(c0, c1, c2, a[11], a[28]); + MONT_MAC(s0, s1, s2, a[12], a[27]); + MONT_MAC(c0, c1, c2, a[13], a[26]); + MONT_MAC(s0, s1, s2, a[14], a[25]); + MONT_MAC(c0, c1, c2, a[15], a[24]); + MONT_MAC(s0, s1, s2, a[16], a[23]); + MONT_MAC(c0, c1, c2, a[17], a[22]); + MONT_MAC(s0, s1, s2, a[18], a[21]); + MONT_MAC(c0, c1, c2, a[19], a[20]); + MONT_MAC(t0, t1, t2, u[8], m[31]); + MONT_MUL(v0, v1, v2, u[9], m[30]); + MONT_MUL(w0, w1, w2, u[10], m[29]); + MONT_MAC(t0, t1, t2, u[11], m[28]); + MONT_MAC(v0, v1, v2, u[12], m[27]); + MONT_MAC(w0, w1, w2, u[13], m[26]); + MONT_MAC(t0, t1, t2, u[14], m[25]); + MONT_MAC(v0, v1, v2, u[15], m[24]); + MONT_MAC(w0, w1, w2, u[16], m[23]); + MONT_MAC(t0, t1, t2, u[17], m[22]); + MONT_MAC(v0, v1, v2, u[18], m[21]); + MONT_MAC(w0, w1, w2, u[19], m[20]); + MONT_MAC(t0, t1, t2, u[20], m[19]); + MONT_MAC(v0, v1, v2, u[21], m[18]); + MONT_MAC(w0, w1, w2, u[22], m[17]); + MONT_MAC(t0, t1, t2, u[23], m[16]); + MONT_MAC(v0, v1, v2, u[24], m[15]); + MONT_MAC(w0, w1, w2, u[25], m[14]); + MONT_MAC(t0, t1, t2, u[26], m[13]); + MONT_MAC(v0, v1, v2, u[27], m[12]); + MONT_MAC(w0, w1, w2, u[28], m[11]); + MONT_MAC(t0, t1, t2, u[29], m[10]); + MONT_MAC(v0, v1, v2, u[30], m[9]); + MONT_MAC(w0, w1, w2, u[31], m[8]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[7] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 40 */ + MONT_MUL(s0, s1, s2, a[9], a[31]); + MONT_MUL(c0, c1, c2, a[10], a[30]); + MONT_MAC(s0, s1, s2, a[11], a[29]); + MONT_MAC(c0, c1, c2, a[12], a[28]); + MONT_MAC(s0, s1, s2, a[13], a[27]); + MONT_MAC(c0, c1, c2, a[14], a[26]); + MONT_MAC(s0, s1, s2, a[15], a[25]); + MONT_MAC(c0, c1, c2, a[16], a[24]); + MONT_MAC(s0, s1, s2, a[17], a[23]); + MONT_MAC(c0, c1, c2, a[18], a[22]); + MONT_MAC(s0, s1, s2, a[19], a[21]); + MONT_MAC(t0, t1, t2, a[20], a[20]); + MONT_MUL(v0, v1, v2, u[9], m[31]); + MONT_MUL(w0, w1, w2, u[10], m[30]); + MONT_MAC(t0, t1, t2, u[11], m[29]); + MONT_MAC(v0, v1, v2, u[12], m[28]); + MONT_MAC(w0, w1, w2, u[13], m[27]); + MONT_MAC(t0, t1, t2, u[14], m[26]); + MONT_MAC(v0, v1, v2, u[15], m[25]); + MONT_MAC(w0, w1, w2, u[16], m[24]); + MONT_MAC(t0, t1, t2, u[17], m[23]); + MONT_MAC(v0, v1, v2, u[18], m[22]); + MONT_MAC(w0, w1, w2, u[19], m[21]); + MONT_MAC(t0, t1, t2, u[20], m[20]); + MONT_MAC(v0, v1, v2, u[21], m[19]); + MONT_MAC(w0, w1, w2, u[22], m[18]); + MONT_MAC(t0, t1, t2, u[23], m[17]); + MONT_MAC(v0, v1, v2, u[24], m[16]); + MONT_MAC(w0, w1, w2, u[25], m[15]); + MONT_MAC(t0, t1, t2, u[26], m[14]); + MONT_MAC(v0, v1, v2, u[27], m[13]); + MONT_MAC(w0, w1, w2, u[28], m[12]); + MONT_MAC(t0, t1, t2, u[29], m[11]); + MONT_MAC(v0, v1, v2, u[30], m[10]); + MONT_MAC(w0, w1, w2, u[31], m[9]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[8] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 41 */ + MONT_MUL(s0, s1, s2, a[10], a[31]); + MONT_MUL(c0, c1, c2, a[11], a[30]); + MONT_MAC(s0, s1, s2, a[12], a[29]); + MONT_MAC(c0, c1, c2, a[13], a[28]); + MONT_MAC(s0, s1, s2, a[14], a[27]); + MONT_MAC(c0, c1, c2, a[15], a[26]); + MONT_MAC(s0, s1, s2, a[16], a[25]); + MONT_MAC(c0, c1, c2, a[17], a[24]); + MONT_MAC(s0, s1, s2, a[18], a[23]); + MONT_MAC(c0, c1, c2, a[19], a[22]); + MONT_MAC(s0, s1, s2, a[20], a[21]); + MONT_MAC(t0, t1, t2, u[10], m[31]); + MONT_MUL(v0, v1, v2, u[11], m[30]); + MONT_MUL(w0, w1, w2, u[12], m[29]); + MONT_MAC(t0, t1, t2, u[13], m[28]); + MONT_MAC(v0, v1, v2, u[14], m[27]); + MONT_MAC(w0, w1, w2, u[15], m[26]); + MONT_MAC(t0, t1, t2, u[16], m[25]); + MONT_MAC(v0, v1, v2, u[17], m[24]); + MONT_MAC(w0, w1, w2, u[18], m[23]); + MONT_MAC(t0, t1, t2, u[19], m[22]); + MONT_MAC(v0, v1, v2, u[20], m[21]); + MONT_MAC(w0, w1, w2, u[21], m[20]); + MONT_MAC(t0, t1, t2, u[22], m[19]); + MONT_MAC(v0, v1, v2, u[23], m[18]); + MONT_MAC(w0, w1, w2, u[24], m[17]); + MONT_MAC(t0, t1, t2, u[25], m[16]); + MONT_MAC(v0, v1, v2, u[26], m[15]); + MONT_MAC(w0, w1, w2, u[27], m[14]); + MONT_MAC(t0, t1, t2, u[28], m[13]); + MONT_MAC(v0, v1, v2, u[29], m[12]); + MONT_MAC(w0, w1, w2, u[30], m[11]); + MONT_MAC(t0, t1, t2, u[31], m[10]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[9] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 42 */ + MONT_MUL(s0, s1, s2, a[11], a[31]); + MONT_MUL(c0, c1, c2, a[12], a[30]); + MONT_MAC(s0, s1, s2, a[13], a[29]); + MONT_MAC(c0, c1, c2, a[14], a[28]); + MONT_MAC(s0, s1, s2, a[15], a[27]); + MONT_MAC(c0, c1, c2, a[16], a[26]); + MONT_MAC(s0, s1, s2, a[17], a[25]); + MONT_MAC(c0, c1, c2, a[18], a[24]); + MONT_MAC(s0, s1, s2, a[19], a[23]); + MONT_MAC(c0, c1, c2, a[20], a[22]); + MONT_MAC(t0, t1, t2, a[21], a[21]); + MONT_MUL(v0, v1, v2, u[11], m[31]); + MONT_MUL(w0, w1, w2, u[12], m[30]); + MONT_MAC(t0, t1, t2, u[13], m[29]); + MONT_MAC(v0, v1, v2, u[14], m[28]); + MONT_MAC(w0, w1, w2, u[15], m[27]); + MONT_MAC(t0, t1, t2, u[16], m[26]); + MONT_MAC(v0, v1, v2, u[17], m[25]); + MONT_MAC(w0, w1, w2, u[18], m[24]); + MONT_MAC(t0, t1, t2, u[19], m[23]); + MONT_MAC(v0, v1, v2, u[20], m[22]); + MONT_MAC(w0, w1, w2, u[21], m[21]); + MONT_MAC(t0, t1, t2, u[22], m[20]); + MONT_MAC(v0, v1, v2, u[23], m[19]); + MONT_MAC(w0, w1, w2, u[24], m[18]); + MONT_MAC(t0, t1, t2, u[25], m[17]); + MONT_MAC(v0, v1, v2, u[26], m[16]); + MONT_MAC(w0, w1, w2, u[27], m[15]); + MONT_MAC(t0, t1, t2, u[28], m[14]); + MONT_MAC(v0, v1, v2, u[29], m[13]); + MONT_MAC(w0, w1, w2, u[30], m[12]); + MONT_MAC(t0, t1, t2, u[31], m[11]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[10] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 43 */ + MONT_MUL(s0, s1, s2, a[12], a[31]); + MONT_MUL(c0, c1, c2, a[13], a[30]); + MONT_MAC(s0, s1, s2, a[14], a[29]); + MONT_MAC(c0, c1, c2, a[15], a[28]); + MONT_MAC(s0, s1, s2, a[16], a[27]); + MONT_MAC(c0, c1, c2, a[17], a[26]); + MONT_MAC(s0, s1, s2, a[18], a[25]); + MONT_MAC(c0, c1, c2, a[19], a[24]); + MONT_MAC(s0, s1, s2, a[20], a[23]); + MONT_MAC(c0, c1, c2, a[21], a[22]); + MONT_MAC(t0, t1, t2, u[12], m[31]); + MONT_MUL(v0, v1, v2, u[13], m[30]); + MONT_MUL(w0, w1, w2, u[14], m[29]); + MONT_MAC(t0, t1, t2, u[15], m[28]); + MONT_MAC(v0, v1, v2, u[16], m[27]); + MONT_MAC(w0, w1, w2, u[17], m[26]); + MONT_MAC(t0, t1, t2, u[18], m[25]); + MONT_MAC(v0, v1, v2, u[19], m[24]); + MONT_MAC(w0, w1, w2, u[20], m[23]); + MONT_MAC(t0, t1, t2, u[21], m[22]); + MONT_MAC(v0, v1, v2, u[22], m[21]); + MONT_MAC(w0, w1, w2, u[23], m[20]); + MONT_MAC(t0, t1, t2, u[24], m[19]); + MONT_MAC(v0, v1, v2, u[25], m[18]); + MONT_MAC(w0, w1, w2, u[26], m[17]); + MONT_MAC(t0, t1, t2, u[27], m[16]); + MONT_MAC(v0, v1, v2, u[28], m[15]); + MONT_MAC(w0, w1, w2, u[29], m[14]); + MONT_MAC(t0, t1, t2, u[30], m[13]); + MONT_MAC(v0, v1, v2, u[31], m[12]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[11] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 44 */ + MONT_MUL(s0, s1, s2, a[13], a[31]); + MONT_MUL(c0, c1, c2, a[14], a[30]); + MONT_MAC(s0, s1, s2, a[15], a[29]); + MONT_MAC(c0, c1, c2, a[16], a[28]); + MONT_MAC(s0, s1, s2, a[17], a[27]); + MONT_MAC(c0, c1, c2, a[18], a[26]); + MONT_MAC(s0, s1, s2, a[19], a[25]); + MONT_MAC(c0, c1, c2, a[20], a[24]); + MONT_MAC(s0, s1, s2, a[21], a[23]); + MONT_MAC(t0, t1, t2, a[22], a[22]); + MONT_MUL(v0, v1, v2, u[13], m[31]); + MONT_MUL(w0, w1, w2, u[14], m[30]); + MONT_MAC(t0, t1, t2, u[15], m[29]); + MONT_MAC(v0, v1, v2, u[16], m[28]); + MONT_MAC(w0, w1, w2, u[17], m[27]); + MONT_MAC(t0, t1, t2, u[18], m[26]); + MONT_MAC(v0, v1, v2, u[19], m[25]); + MONT_MAC(w0, w1, w2, u[20], m[24]); + MONT_MAC(t0, t1, t2, u[21], m[23]); + MONT_MAC(v0, v1, v2, u[22], m[22]); + MONT_MAC(w0, w1, w2, u[23], m[21]); + MONT_MAC(t0, t1, t2, u[24], m[20]); + MONT_MAC(v0, v1, v2, u[25], m[19]); + MONT_MAC(w0, w1, w2, u[26], m[18]); + MONT_MAC(t0, t1, t2, u[27], m[17]); + MONT_MAC(v0, v1, v2, u[28], m[16]); + MONT_MAC(w0, w1, w2, u[29], m[15]); + MONT_MAC(t0, t1, t2, u[30], m[14]); + MONT_MAC(v0, v1, v2, u[31], m[13]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[12] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 45 */ + MONT_MUL(s0, s1, s2, a[14], a[31]); + MONT_MUL(c0, c1, c2, a[15], a[30]); + MONT_MAC(s0, s1, s2, a[16], a[29]); + MONT_MAC(c0, c1, c2, a[17], a[28]); + MONT_MAC(s0, s1, s2, a[18], a[27]); + MONT_MAC(c0, c1, c2, a[19], a[26]); + MONT_MAC(s0, s1, s2, a[20], a[25]); + MONT_MAC(c0, c1, c2, a[21], a[24]); + MONT_MAC(s0, s1, s2, a[22], a[23]); + MONT_MAC(t0, t1, t2, u[14], m[31]); + MONT_MUL(v0, v1, v2, u[15], m[30]); + MONT_MUL(w0, w1, w2, u[16], m[29]); + MONT_MAC(t0, t1, t2, u[17], m[28]); + MONT_MAC(v0, v1, v2, u[18], m[27]); + MONT_MAC(w0, w1, w2, u[19], m[26]); + MONT_MAC(t0, t1, t2, u[20], m[25]); + MONT_MAC(v0, v1, v2, u[21], m[24]); + MONT_MAC(w0, w1, w2, u[22], m[23]); + MONT_MAC(t0, t1, t2, u[23], m[22]); + MONT_MAC(v0, v1, v2, u[24], m[21]); + MONT_MAC(w0, w1, w2, u[25], m[20]); + MONT_MAC(t0, t1, t2, u[26], m[19]); + MONT_MAC(v0, v1, v2, u[27], m[18]); + MONT_MAC(w0, w1, w2, u[28], m[17]); + MONT_MAC(t0, t1, t2, u[29], m[16]); + MONT_MAC(v0, v1, v2, u[30], m[15]); + MONT_MAC(w0, w1, w2, u[31], m[14]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[13] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 46 */ + MONT_MUL(s0, s1, s2, a[15], a[31]); + MONT_MUL(c0, c1, c2, a[16], a[30]); + MONT_MAC(s0, s1, s2, a[17], a[29]); + MONT_MAC(c0, c1, c2, a[18], a[28]); + MONT_MAC(s0, s1, s2, a[19], a[27]); + MONT_MAC(c0, c1, c2, a[20], a[26]); + MONT_MAC(s0, s1, s2, a[21], a[25]); + MONT_MAC(c0, c1, c2, a[22], a[24]); + MONT_MAC(t0, t1, t2, a[23], a[23]); + MONT_MUL(v0, v1, v2, u[15], m[31]); + MONT_MUL(w0, w1, w2, u[16], m[30]); + MONT_MAC(t0, t1, t2, u[17], m[29]); + MONT_MAC(v0, v1, v2, u[18], m[28]); + MONT_MAC(w0, w1, w2, u[19], m[27]); + MONT_MAC(t0, t1, t2, u[20], m[26]); + MONT_MAC(v0, v1, v2, u[21], m[25]); + MONT_MAC(w0, w1, w2, u[22], m[24]); + MONT_MAC(t0, t1, t2, u[23], m[23]); + MONT_MAC(v0, v1, v2, u[24], m[22]); + MONT_MAC(w0, w1, w2, u[25], m[21]); + MONT_MAC(t0, t1, t2, u[26], m[20]); + MONT_MAC(v0, v1, v2, u[27], m[19]); + MONT_MAC(w0, w1, w2, u[28], m[18]); + MONT_MAC(t0, t1, t2, u[29], m[17]); + MONT_MAC(v0, v1, v2, u[30], m[16]); + MONT_MAC(w0, w1, w2, u[31], m[15]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[14] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 47 */ + MONT_MUL(s0, s1, s2, a[16], a[31]); + MONT_MUL(c0, c1, c2, a[17], a[30]); + MONT_MAC(s0, s1, s2, a[18], a[29]); + MONT_MAC(c0, c1, c2, a[19], a[28]); + MONT_MAC(s0, s1, s2, a[20], a[27]); + MONT_MAC(c0, c1, c2, a[21], a[26]); + MONT_MAC(s0, s1, s2, a[22], a[25]); + MONT_MAC(c0, c1, c2, a[23], a[24]); + MONT_MAC(t0, t1, t2, u[16], m[31]); + MONT_MUL(v0, v1, v2, u[17], m[30]); + MONT_MUL(w0, w1, w2, u[18], m[29]); + MONT_MAC(t0, t1, t2, u[19], m[28]); + MONT_MAC(v0, v1, v2, u[20], m[27]); + MONT_MAC(w0, w1, w2, u[21], m[26]); + MONT_MAC(t0, t1, t2, u[22], m[25]); + MONT_MAC(v0, v1, v2, u[23], m[24]); + MONT_MAC(w0, w1, w2, u[24], m[23]); + MONT_MAC(t0, t1, t2, u[25], m[22]); + MONT_MAC(v0, v1, v2, u[26], m[21]); + MONT_MAC(w0, w1, w2, u[27], m[20]); + MONT_MAC(t0, t1, t2, u[28], m[19]); + MONT_MAC(v0, v1, v2, u[29], m[18]); + MONT_MAC(w0, w1, w2, u[30], m[17]); + MONT_MAC(t0, t1, t2, u[31], m[16]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[15] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 48 */ + MONT_MUL(s0, s1, s2, a[17], a[31]); + MONT_MUL(c0, c1, c2, a[18], a[30]); + MONT_MAC(s0, s1, s2, a[19], a[29]); + MONT_MAC(c0, c1, c2, a[20], a[28]); + MONT_MAC(s0, s1, s2, a[21], a[27]); + MONT_MAC(c0, c1, c2, a[22], a[26]); + MONT_MAC(s0, s1, s2, a[23], a[25]); + MONT_MAC(t0, t1, t2, a[24], a[24]); + MONT_MUL(v0, v1, v2, u[17], m[31]); + MONT_MUL(w0, w1, w2, u[18], m[30]); + MONT_MAC(t0, t1, t2, u[19], m[29]); + MONT_MAC(v0, v1, v2, u[20], m[28]); + MONT_MAC(w0, w1, w2, u[21], m[27]); + MONT_MAC(t0, t1, t2, u[22], m[26]); + MONT_MAC(v0, v1, v2, u[23], m[25]); + MONT_MAC(w0, w1, w2, u[24], m[24]); + MONT_MAC(t0, t1, t2, u[25], m[23]); + MONT_MAC(v0, v1, v2, u[26], m[22]); + MONT_MAC(w0, w1, w2, u[27], m[21]); + MONT_MAC(t0, t1, t2, u[28], m[20]); + MONT_MAC(v0, v1, v2, u[29], m[19]); + MONT_MAC(w0, w1, w2, u[30], m[18]); + MONT_MAC(t0, t1, t2, u[31], m[17]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[16] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 49 */ + MONT_MUL(s0, s1, s2, a[18], a[31]); + MONT_MUL(c0, c1, c2, a[19], a[30]); + MONT_MAC(s0, s1, s2, a[20], a[29]); + MONT_MAC(c0, c1, c2, a[21], a[28]); + MONT_MAC(s0, s1, s2, a[22], a[27]); + MONT_MAC(c0, c1, c2, a[23], a[26]); + MONT_MAC(s0, s1, s2, a[24], a[25]); + MONT_MAC(t0, t1, t2, u[18], m[31]); + MONT_MUL(v0, v1, v2, u[19], m[30]); + MONT_MUL(w0, w1, w2, u[20], m[29]); + MONT_MAC(t0, t1, t2, u[21], m[28]); + MONT_MAC(v0, v1, v2, u[22], m[27]); + MONT_MAC(w0, w1, w2, u[23], m[26]); + MONT_MAC(t0, t1, t2, u[24], m[25]); + MONT_MAC(v0, v1, v2, u[25], m[24]); + MONT_MAC(w0, w1, w2, u[26], m[23]); + MONT_MAC(t0, t1, t2, u[27], m[22]); + MONT_MAC(v0, v1, v2, u[28], m[21]); + MONT_MAC(w0, w1, w2, u[29], m[20]); + MONT_MAC(t0, t1, t2, u[30], m[19]); + MONT_MAC(v0, v1, v2, u[31], m[18]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[17] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 50 */ + MONT_MUL(s0, s1, s2, a[19], a[31]); + MONT_MUL(c0, c1, c2, a[20], a[30]); + MONT_MAC(s0, s1, s2, a[21], a[29]); + MONT_MAC(c0, c1, c2, a[22], a[28]); + MONT_MAC(s0, s1, s2, a[23], a[27]); + MONT_MAC(c0, c1, c2, a[24], a[26]); + MONT_MAC(t0, t1, t2, a[25], a[25]); + MONT_MUL(v0, v1, v2, u[19], m[31]); + MONT_MUL(w0, w1, w2, u[20], m[30]); + MONT_MAC(t0, t1, t2, u[21], m[29]); + MONT_MAC(v0, v1, v2, u[22], m[28]); + MONT_MAC(w0, w1, w2, u[23], m[27]); + MONT_MAC(t0, t1, t2, u[24], m[26]); + MONT_MAC(v0, v1, v2, u[25], m[25]); + MONT_MAC(w0, w1, w2, u[26], m[24]); + MONT_MAC(t0, t1, t2, u[27], m[23]); + MONT_MAC(v0, v1, v2, u[28], m[22]); + MONT_MAC(w0, w1, w2, u[29], m[21]); + MONT_MAC(t0, t1, t2, u[30], m[20]); + MONT_MAC(v0, v1, v2, u[31], m[19]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[18] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 51 */ + MONT_MUL(s0, s1, s2, a[20], a[31]); + MONT_MUL(c0, c1, c2, a[21], a[30]); + MONT_MAC(s0, s1, s2, a[22], a[29]); + MONT_MAC(c0, c1, c2, a[23], a[28]); + MONT_MAC(s0, s1, s2, a[24], a[27]); + MONT_MAC(c0, c1, c2, a[25], a[26]); + MONT_MAC(t0, t1, t2, u[20], m[31]); + MONT_MUL(v0, v1, v2, u[21], m[30]); + MONT_MUL(w0, w1, w2, u[22], m[29]); + MONT_MAC(t0, t1, t2, u[23], m[28]); + MONT_MAC(v0, v1, v2, u[24], m[27]); + MONT_MAC(w0, w1, w2, u[25], m[26]); + MONT_MAC(t0, t1, t2, u[26], m[25]); + MONT_MAC(v0, v1, v2, u[27], m[24]); + MONT_MAC(w0, w1, w2, u[28], m[23]); + MONT_MAC(t0, t1, t2, u[29], m[22]); + MONT_MAC(v0, v1, v2, u[30], m[21]); + MONT_MAC(w0, w1, w2, u[31], m[20]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[19] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 52 */ + MONT_MUL(s0, s1, s2, a[21], a[31]); + MONT_MUL(c0, c1, c2, a[22], a[30]); + MONT_MAC(s0, s1, s2, a[23], a[29]); + MONT_MAC(c0, c1, c2, a[24], a[28]); + MONT_MAC(s0, s1, s2, a[25], a[27]); + MONT_MAC(t0, t1, t2, a[26], a[26]); + MONT_MUL(v0, v1, v2, u[21], m[31]); + MONT_MUL(w0, w1, w2, u[22], m[30]); + MONT_MAC(t0, t1, t2, u[23], m[29]); + MONT_MAC(v0, v1, v2, u[24], m[28]); + MONT_MAC(w0, w1, w2, u[25], m[27]); + MONT_MAC(t0, t1, t2, u[26], m[26]); + MONT_MAC(v0, v1, v2, u[27], m[25]); + MONT_MAC(w0, w1, w2, u[28], m[24]); + MONT_MAC(t0, t1, t2, u[29], m[23]); + MONT_MAC(v0, v1, v2, u[30], m[22]); + MONT_MAC(w0, w1, w2, u[31], m[21]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[20] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 53 */ + MONT_MUL(s0, s1, s2, a[22], a[31]); + MONT_MUL(c0, c1, c2, a[23], a[30]); + MONT_MAC(s0, s1, s2, a[24], a[29]); + MONT_MAC(c0, c1, c2, a[25], a[28]); + MONT_MAC(s0, s1, s2, a[26], a[27]); + MONT_MAC(t0, t1, t2, u[22], m[31]); + MONT_MUL(v0, v1, v2, u[23], m[30]); + MONT_MUL(w0, w1, w2, u[24], m[29]); + MONT_MAC(t0, t1, t2, u[25], m[28]); + MONT_MAC(v0, v1, v2, u[26], m[27]); + MONT_MAC(w0, w1, w2, u[27], m[26]); + MONT_MAC(t0, t1, t2, u[28], m[25]); + MONT_MAC(v0, v1, v2, u[29], m[24]); + MONT_MAC(w0, w1, w2, u[30], m[23]); + MONT_MAC(t0, t1, t2, u[31], m[22]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[21] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 54 */ + MONT_MUL(s0, s1, s2, a[23], a[31]); + MONT_MUL(c0, c1, c2, a[24], a[30]); + MONT_MAC(s0, s1, s2, a[25], a[29]); + MONT_MAC(c0, c1, c2, a[26], a[28]); + MONT_MAC(t0, t1, t2, a[27], a[27]); + MONT_MUL(v0, v1, v2, u[23], m[31]); + MONT_MUL(w0, w1, w2, u[24], m[30]); + MONT_MAC(t0, t1, t2, u[25], m[29]); + MONT_MAC(v0, v1, v2, u[26], m[28]); + MONT_MAC(w0, w1, w2, u[27], m[27]); + MONT_MAC(t0, t1, t2, u[28], m[26]); + MONT_MAC(v0, v1, v2, u[29], m[25]); + MONT_MAC(w0, w1, w2, u[30], m[24]); + MONT_MAC(t0, t1, t2, u[31], m[23]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[22] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 55 */ + MONT_MUL(s0, s1, s2, a[24], a[31]); + MONT_MUL(c0, c1, c2, a[25], a[30]); + MONT_MAC(s0, s1, s2, a[26], a[29]); + MONT_MAC(c0, c1, c2, a[27], a[28]); + MONT_MAC(t0, t1, t2, u[24], m[31]); + MONT_MUL(v0, v1, v2, u[25], m[30]); + MONT_MUL(w0, w1, w2, u[26], m[29]); + MONT_MAC(t0, t1, t2, u[27], m[28]); + MONT_MAC(v0, v1, v2, u[28], m[27]); + MONT_MAC(w0, w1, w2, u[29], m[26]); + MONT_MAC(t0, t1, t2, u[30], m[25]); + MONT_MAC(v0, v1, v2, u[31], m[24]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[23] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 56 */ + MONT_MUL(s0, s1, s2, a[25], a[31]); + MONT_MUL(c0, c1, c2, a[26], a[30]); + MONT_MAC(s0, s1, s2, a[27], a[29]); + MONT_MAC(t0, t1, t2, a[28], a[28]); + MONT_MUL(v0, v1, v2, u[25], m[31]); + MONT_MUL(w0, w1, w2, u[26], m[30]); + MONT_MAC(t0, t1, t2, u[27], m[29]); + MONT_MAC(v0, v1, v2, u[28], m[28]); + MONT_MAC(w0, w1, w2, u[29], m[27]); + MONT_MAC(t0, t1, t2, u[30], m[26]); + MONT_MAC(v0, v1, v2, u[31], m[25]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[24] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 57 */ + MONT_MUL(s0, s1, s2, a[26], a[31]); + MONT_MUL(c0, c1, c2, a[27], a[30]); + MONT_MAC(s0, s1, s2, a[28], a[29]); + MONT_MAC(t0, t1, t2, u[26], m[31]); + MONT_MUL(v0, v1, v2, u[27], m[30]); + MONT_MUL(w0, w1, w2, u[28], m[29]); + MONT_MAC(t0, t1, t2, u[29], m[28]); + MONT_MAC(v0, v1, v2, u[30], m[27]); + MONT_MAC(w0, w1, w2, u[31], m[26]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[25] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 58 */ + MONT_MUL(s0, s1, s2, a[27], a[31]); + MONT_MUL(c0, c1, c2, a[28], a[30]); + MONT_MAC(t0, t1, t2, a[29], a[29]); + MONT_MUL(v0, v1, v2, u[27], m[31]); + MONT_MUL(w0, w1, w2, u[28], m[30]); + MONT_MAC(t0, t1, t2, u[29], m[29]); + MONT_MAC(v0, v1, v2, u[30], m[28]); + MONT_MAC(w0, w1, w2, u[31], m[27]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[26] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 59 */ + MONT_MUL(s0, s1, s2, a[28], a[31]); + MONT_MUL(c0, c1, c2, a[29], a[30]); + MONT_MAC(t0, t1, t2, u[28], m[31]); + MONT_MUL(v0, v1, v2, u[29], m[30]); + MONT_MUL(w0, w1, w2, u[30], m[29]); + MONT_MAC(t0, t1, t2, u[31], m[28]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[27] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 60 */ + MONT_MUL(s0, s1, s2, a[29], a[31]); + MONT_MAC(t0, t1, t2, a[30], a[30]); + MONT_MUL(v0, v1, v2, u[29], m[31]); + MONT_MUL(w0, w1, w2, u[30], m[30]); + MONT_MAC(t0, t1, t2, u[31], m[29]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[28] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 61 */ + MONT_MUL(s0, s1, s2, a[30], a[31]); + MONT_MAC(t0, t1, t2, u[30], m[31]); + MONT_MUL(v0, v1, v2, u[31], m[30]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[29] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 62 */ + MONT_MAC(t0, t1, t2, a[31], a[31]); + MONT_MUL(v0, v1, v2, u[31], m[31]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + z[30] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + z[31] = t0; + + /* under 2m, with t1 the bit above n limbs: at most one subtraction */ + borrow = sub_n(r, z, m, 32); + take = t1 | (borrow ^ 1); + select_n(r, r, z, take & 1, 32); +} + +/* r = a * a * R^-1 mod m, for a under m; t is 2n limbs of scratch */ +static void crypton_mont_sqr48(limb_t *r, const limb_t *a, + const limb_t *m, limb_t n0, limb_t *t) +{ + limb_t *u = t, *z = t + 48; + limb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take; + limb_t s0, s1, s2, c0, c1, c2; + + /* column 0 */ + MONT_MAC(t0, t1, t2, a[0], a[0]); + u[0] = t0 * n0; + MONT_MAC(t0, t1, t2, u[0], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 1 */ + MONT_MUL(s0, s1, s2, a[0], a[1]); + MONT_MAC(t0, t1, t2, u[0], m[1]); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[1] = t0 * n0; + MONT_MAC(t0, t1, t2, u[1], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 2 */ + MONT_MUL(s0, s1, s2, a[0], a[2]); + MONT_MAC(t0, t1, t2, a[1], a[1]); + MONT_MUL(v0, v1, v2, u[0], m[2]); + MONT_MUL(w0, w1, w2, u[1], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[2] = t0 * n0; + MONT_MAC(t0, t1, t2, u[2], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 3 */ + MONT_MUL(s0, s1, s2, a[0], a[3]); + MONT_MUL(c0, c1, c2, a[1], a[2]); + MONT_MAC(t0, t1, t2, u[0], m[3]); + MONT_MUL(v0, v1, v2, u[1], m[2]); + MONT_MUL(w0, w1, w2, u[2], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[3] = t0 * n0; + MONT_MAC(t0, t1, t2, u[3], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 4 */ + MONT_MUL(s0, s1, s2, a[0], a[4]); + MONT_MUL(c0, c1, c2, a[1], a[3]); + MONT_MAC(t0, t1, t2, a[2], a[2]); + MONT_MUL(v0, v1, v2, u[0], m[4]); + MONT_MUL(w0, w1, w2, u[1], m[3]); + MONT_MAC(t0, t1, t2, u[2], m[2]); + MONT_MAC(v0, v1, v2, u[3], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[4] = t0 * n0; + MONT_MAC(t0, t1, t2, u[4], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 5 */ + MONT_MUL(s0, s1, s2, a[0], a[5]); + MONT_MUL(c0, c1, c2, a[1], a[4]); + MONT_MAC(s0, s1, s2, a[2], a[3]); + MONT_MAC(t0, t1, t2, u[0], m[5]); + MONT_MUL(v0, v1, v2, u[1], m[4]); + MONT_MUL(w0, w1, w2, u[2], m[3]); + MONT_MAC(t0, t1, t2, u[3], m[2]); + MONT_MAC(v0, v1, v2, u[4], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[5] = t0 * n0; + MONT_MAC(t0, t1, t2, u[5], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 6 */ + MONT_MUL(s0, s1, s2, a[0], a[6]); + MONT_MUL(c0, c1, c2, a[1], a[5]); + MONT_MAC(s0, s1, s2, a[2], a[4]); + MONT_MAC(t0, t1, t2, a[3], a[3]); + MONT_MUL(v0, v1, v2, u[0], m[6]); + MONT_MUL(w0, w1, w2, u[1], m[5]); + MONT_MAC(t0, t1, t2, u[2], m[4]); + MONT_MAC(v0, v1, v2, u[3], m[3]); + MONT_MAC(w0, w1, w2, u[4], m[2]); + MONT_MAC(t0, t1, t2, u[5], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[6] = t0 * n0; + MONT_MAC(t0, t1, t2, u[6], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 7 */ + MONT_MUL(s0, s1, s2, a[0], a[7]); + MONT_MUL(c0, c1, c2, a[1], a[6]); + MONT_MAC(s0, s1, s2, a[2], a[5]); + MONT_MAC(c0, c1, c2, a[3], a[4]); + MONT_MAC(t0, t1, t2, u[0], m[7]); + MONT_MUL(v0, v1, v2, u[1], m[6]); + MONT_MUL(w0, w1, w2, u[2], m[5]); + MONT_MAC(t0, t1, t2, u[3], m[4]); + MONT_MAC(v0, v1, v2, u[4], m[3]); + MONT_MAC(w0, w1, w2, u[5], m[2]); + MONT_MAC(t0, t1, t2, u[6], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[7] = t0 * n0; + MONT_MAC(t0, t1, t2, u[7], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 8 */ + MONT_MUL(s0, s1, s2, a[0], a[8]); + MONT_MUL(c0, c1, c2, a[1], a[7]); + MONT_MAC(s0, s1, s2, a[2], a[6]); + MONT_MAC(c0, c1, c2, a[3], a[5]); + MONT_MAC(t0, t1, t2, a[4], a[4]); + MONT_MUL(v0, v1, v2, u[0], m[8]); + MONT_MUL(w0, w1, w2, u[1], m[7]); + MONT_MAC(t0, t1, t2, u[2], m[6]); + MONT_MAC(v0, v1, v2, u[3], m[5]); + MONT_MAC(w0, w1, w2, u[4], m[4]); + MONT_MAC(t0, t1, t2, u[5], m[3]); + MONT_MAC(v0, v1, v2, u[6], m[2]); + MONT_MAC(w0, w1, w2, u[7], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[8] = t0 * n0; + MONT_MAC(t0, t1, t2, u[8], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 9 */ + MONT_MUL(s0, s1, s2, a[0], a[9]); + MONT_MUL(c0, c1, c2, a[1], a[8]); + MONT_MAC(s0, s1, s2, a[2], a[7]); + MONT_MAC(c0, c1, c2, a[3], a[6]); + MONT_MAC(s0, s1, s2, a[4], a[5]); + MONT_MAC(t0, t1, t2, u[0], m[9]); + MONT_MUL(v0, v1, v2, u[1], m[8]); + MONT_MUL(w0, w1, w2, u[2], m[7]); + MONT_MAC(t0, t1, t2, u[3], m[6]); + MONT_MAC(v0, v1, v2, u[4], m[5]); + MONT_MAC(w0, w1, w2, u[5], m[4]); + MONT_MAC(t0, t1, t2, u[6], m[3]); + MONT_MAC(v0, v1, v2, u[7], m[2]); + MONT_MAC(w0, w1, w2, u[8], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[9] = t0 * n0; + MONT_MAC(t0, t1, t2, u[9], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 10 */ + MONT_MUL(s0, s1, s2, a[0], a[10]); + MONT_MUL(c0, c1, c2, a[1], a[9]); + MONT_MAC(s0, s1, s2, a[2], a[8]); + MONT_MAC(c0, c1, c2, a[3], a[7]); + MONT_MAC(s0, s1, s2, a[4], a[6]); + MONT_MAC(t0, t1, t2, a[5], a[5]); + MONT_MUL(v0, v1, v2, u[0], m[10]); + MONT_MUL(w0, w1, w2, u[1], m[9]); + MONT_MAC(t0, t1, t2, u[2], m[8]); + MONT_MAC(v0, v1, v2, u[3], m[7]); + MONT_MAC(w0, w1, w2, u[4], m[6]); + MONT_MAC(t0, t1, t2, u[5], m[5]); + MONT_MAC(v0, v1, v2, u[6], m[4]); + MONT_MAC(w0, w1, w2, u[7], m[3]); + MONT_MAC(t0, t1, t2, u[8], m[2]); + MONT_MAC(v0, v1, v2, u[9], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[10] = t0 * n0; + MONT_MAC(t0, t1, t2, u[10], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 11 */ + MONT_MUL(s0, s1, s2, a[0], a[11]); + MONT_MUL(c0, c1, c2, a[1], a[10]); + MONT_MAC(s0, s1, s2, a[2], a[9]); + MONT_MAC(c0, c1, c2, a[3], a[8]); + MONT_MAC(s0, s1, s2, a[4], a[7]); + MONT_MAC(c0, c1, c2, a[5], a[6]); + MONT_MAC(t0, t1, t2, u[0], m[11]); + MONT_MUL(v0, v1, v2, u[1], m[10]); + MONT_MUL(w0, w1, w2, u[2], m[9]); + MONT_MAC(t0, t1, t2, u[3], m[8]); + MONT_MAC(v0, v1, v2, u[4], m[7]); + MONT_MAC(w0, w1, w2, u[5], m[6]); + MONT_MAC(t0, t1, t2, u[6], m[5]); + MONT_MAC(v0, v1, v2, u[7], m[4]); + MONT_MAC(w0, w1, w2, u[8], m[3]); + MONT_MAC(t0, t1, t2, u[9], m[2]); + MONT_MAC(v0, v1, v2, u[10], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[11] = t0 * n0; + MONT_MAC(t0, t1, t2, u[11], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 12 */ + MONT_MUL(s0, s1, s2, a[0], a[12]); + MONT_MUL(c0, c1, c2, a[1], a[11]); + MONT_MAC(s0, s1, s2, a[2], a[10]); + MONT_MAC(c0, c1, c2, a[3], a[9]); + MONT_MAC(s0, s1, s2, a[4], a[8]); + MONT_MAC(c0, c1, c2, a[5], a[7]); + MONT_MAC(t0, t1, t2, a[6], a[6]); + MONT_MUL(v0, v1, v2, u[0], m[12]); + MONT_MUL(w0, w1, w2, u[1], m[11]); + MONT_MAC(t0, t1, t2, u[2], m[10]); + MONT_MAC(v0, v1, v2, u[3], m[9]); + MONT_MAC(w0, w1, w2, u[4], m[8]); + MONT_MAC(t0, t1, t2, u[5], m[7]); + MONT_MAC(v0, v1, v2, u[6], m[6]); + MONT_MAC(w0, w1, w2, u[7], m[5]); + MONT_MAC(t0, t1, t2, u[8], m[4]); + MONT_MAC(v0, v1, v2, u[9], m[3]); + MONT_MAC(w0, w1, w2, u[10], m[2]); + MONT_MAC(t0, t1, t2, u[11], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[12] = t0 * n0; + MONT_MAC(t0, t1, t2, u[12], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 13 */ + MONT_MUL(s0, s1, s2, a[0], a[13]); + MONT_MUL(c0, c1, c2, a[1], a[12]); + MONT_MAC(s0, s1, s2, a[2], a[11]); + MONT_MAC(c0, c1, c2, a[3], a[10]); + MONT_MAC(s0, s1, s2, a[4], a[9]); + MONT_MAC(c0, c1, c2, a[5], a[8]); + MONT_MAC(s0, s1, s2, a[6], a[7]); + MONT_MAC(t0, t1, t2, u[0], m[13]); + MONT_MUL(v0, v1, v2, u[1], m[12]); + MONT_MUL(w0, w1, w2, u[2], m[11]); + MONT_MAC(t0, t1, t2, u[3], m[10]); + MONT_MAC(v0, v1, v2, u[4], m[9]); + MONT_MAC(w0, w1, w2, u[5], m[8]); + MONT_MAC(t0, t1, t2, u[6], m[7]); + MONT_MAC(v0, v1, v2, u[7], m[6]); + MONT_MAC(w0, w1, w2, u[8], m[5]); + MONT_MAC(t0, t1, t2, u[9], m[4]); + MONT_MAC(v0, v1, v2, u[10], m[3]); + MONT_MAC(w0, w1, w2, u[11], m[2]); + MONT_MAC(t0, t1, t2, u[12], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[13] = t0 * n0; + MONT_MAC(t0, t1, t2, u[13], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 14 */ + MONT_MUL(s0, s1, s2, a[0], a[14]); + MONT_MUL(c0, c1, c2, a[1], a[13]); + MONT_MAC(s0, s1, s2, a[2], a[12]); + MONT_MAC(c0, c1, c2, a[3], a[11]); + MONT_MAC(s0, s1, s2, a[4], a[10]); + MONT_MAC(c0, c1, c2, a[5], a[9]); + MONT_MAC(s0, s1, s2, a[6], a[8]); + MONT_MAC(t0, t1, t2, a[7], a[7]); + MONT_MUL(v0, v1, v2, u[0], m[14]); + MONT_MUL(w0, w1, w2, u[1], m[13]); + MONT_MAC(t0, t1, t2, u[2], m[12]); + MONT_MAC(v0, v1, v2, u[3], m[11]); + MONT_MAC(w0, w1, w2, u[4], m[10]); + MONT_MAC(t0, t1, t2, u[5], m[9]); + MONT_MAC(v0, v1, v2, u[6], m[8]); + MONT_MAC(w0, w1, w2, u[7], m[7]); + MONT_MAC(t0, t1, t2, u[8], m[6]); + MONT_MAC(v0, v1, v2, u[9], m[5]); + MONT_MAC(w0, w1, w2, u[10], m[4]); + MONT_MAC(t0, t1, t2, u[11], m[3]); + MONT_MAC(v0, v1, v2, u[12], m[2]); + MONT_MAC(w0, w1, w2, u[13], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[14] = t0 * n0; + MONT_MAC(t0, t1, t2, u[14], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 15 */ + MONT_MUL(s0, s1, s2, a[0], a[15]); + MONT_MUL(c0, c1, c2, a[1], a[14]); + MONT_MAC(s0, s1, s2, a[2], a[13]); + MONT_MAC(c0, c1, c2, a[3], a[12]); + MONT_MAC(s0, s1, s2, a[4], a[11]); + MONT_MAC(c0, c1, c2, a[5], a[10]); + MONT_MAC(s0, s1, s2, a[6], a[9]); + MONT_MAC(c0, c1, c2, a[7], a[8]); + MONT_MAC(t0, t1, t2, u[0], m[15]); + MONT_MUL(v0, v1, v2, u[1], m[14]); + MONT_MUL(w0, w1, w2, u[2], m[13]); + MONT_MAC(t0, t1, t2, u[3], m[12]); + MONT_MAC(v0, v1, v2, u[4], m[11]); + MONT_MAC(w0, w1, w2, u[5], m[10]); + MONT_MAC(t0, t1, t2, u[6], m[9]); + MONT_MAC(v0, v1, v2, u[7], m[8]); + MONT_MAC(w0, w1, w2, u[8], m[7]); + MONT_MAC(t0, t1, t2, u[9], m[6]); + MONT_MAC(v0, v1, v2, u[10], m[5]); + MONT_MAC(w0, w1, w2, u[11], m[4]); + MONT_MAC(t0, t1, t2, u[12], m[3]); + MONT_MAC(v0, v1, v2, u[13], m[2]); + MONT_MAC(w0, w1, w2, u[14], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[15] = t0 * n0; + MONT_MAC(t0, t1, t2, u[15], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 16 */ + MONT_MUL(s0, s1, s2, a[0], a[16]); + MONT_MUL(c0, c1, c2, a[1], a[15]); + MONT_MAC(s0, s1, s2, a[2], a[14]); + MONT_MAC(c0, c1, c2, a[3], a[13]); + MONT_MAC(s0, s1, s2, a[4], a[12]); + MONT_MAC(c0, c1, c2, a[5], a[11]); + MONT_MAC(s0, s1, s2, a[6], a[10]); + MONT_MAC(c0, c1, c2, a[7], a[9]); + MONT_MAC(t0, t1, t2, a[8], a[8]); + MONT_MUL(v0, v1, v2, u[0], m[16]); + MONT_MUL(w0, w1, w2, u[1], m[15]); + MONT_MAC(t0, t1, t2, u[2], m[14]); + MONT_MAC(v0, v1, v2, u[3], m[13]); + MONT_MAC(w0, w1, w2, u[4], m[12]); + MONT_MAC(t0, t1, t2, u[5], m[11]); + MONT_MAC(v0, v1, v2, u[6], m[10]); + MONT_MAC(w0, w1, w2, u[7], m[9]); + MONT_MAC(t0, t1, t2, u[8], m[8]); + MONT_MAC(v0, v1, v2, u[9], m[7]); + MONT_MAC(w0, w1, w2, u[10], m[6]); + MONT_MAC(t0, t1, t2, u[11], m[5]); + MONT_MAC(v0, v1, v2, u[12], m[4]); + MONT_MAC(w0, w1, w2, u[13], m[3]); + MONT_MAC(t0, t1, t2, u[14], m[2]); + MONT_MAC(v0, v1, v2, u[15], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[16] = t0 * n0; + MONT_MAC(t0, t1, t2, u[16], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 17 */ + MONT_MUL(s0, s1, s2, a[0], a[17]); + MONT_MUL(c0, c1, c2, a[1], a[16]); + MONT_MAC(s0, s1, s2, a[2], a[15]); + MONT_MAC(c0, c1, c2, a[3], a[14]); + MONT_MAC(s0, s1, s2, a[4], a[13]); + MONT_MAC(c0, c1, c2, a[5], a[12]); + MONT_MAC(s0, s1, s2, a[6], a[11]); + MONT_MAC(c0, c1, c2, a[7], a[10]); + MONT_MAC(s0, s1, s2, a[8], a[9]); + MONT_MAC(t0, t1, t2, u[0], m[17]); + MONT_MUL(v0, v1, v2, u[1], m[16]); + MONT_MUL(w0, w1, w2, u[2], m[15]); + MONT_MAC(t0, t1, t2, u[3], m[14]); + MONT_MAC(v0, v1, v2, u[4], m[13]); + MONT_MAC(w0, w1, w2, u[5], m[12]); + MONT_MAC(t0, t1, t2, u[6], m[11]); + MONT_MAC(v0, v1, v2, u[7], m[10]); + MONT_MAC(w0, w1, w2, u[8], m[9]); + MONT_MAC(t0, t1, t2, u[9], m[8]); + MONT_MAC(v0, v1, v2, u[10], m[7]); + MONT_MAC(w0, w1, w2, u[11], m[6]); + MONT_MAC(t0, t1, t2, u[12], m[5]); + MONT_MAC(v0, v1, v2, u[13], m[4]); + MONT_MAC(w0, w1, w2, u[14], m[3]); + MONT_MAC(t0, t1, t2, u[15], m[2]); + MONT_MAC(v0, v1, v2, u[16], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[17] = t0 * n0; + MONT_MAC(t0, t1, t2, u[17], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 18 */ + MONT_MUL(s0, s1, s2, a[0], a[18]); + MONT_MUL(c0, c1, c2, a[1], a[17]); + MONT_MAC(s0, s1, s2, a[2], a[16]); + MONT_MAC(c0, c1, c2, a[3], a[15]); + MONT_MAC(s0, s1, s2, a[4], a[14]); + MONT_MAC(c0, c1, c2, a[5], a[13]); + MONT_MAC(s0, s1, s2, a[6], a[12]); + MONT_MAC(c0, c1, c2, a[7], a[11]); + MONT_MAC(s0, s1, s2, a[8], a[10]); + MONT_MAC(t0, t1, t2, a[9], a[9]); + MONT_MUL(v0, v1, v2, u[0], m[18]); + MONT_MUL(w0, w1, w2, u[1], m[17]); + MONT_MAC(t0, t1, t2, u[2], m[16]); + MONT_MAC(v0, v1, v2, u[3], m[15]); + MONT_MAC(w0, w1, w2, u[4], m[14]); + MONT_MAC(t0, t1, t2, u[5], m[13]); + MONT_MAC(v0, v1, v2, u[6], m[12]); + MONT_MAC(w0, w1, w2, u[7], m[11]); + MONT_MAC(t0, t1, t2, u[8], m[10]); + MONT_MAC(v0, v1, v2, u[9], m[9]); + MONT_MAC(w0, w1, w2, u[10], m[8]); + MONT_MAC(t0, t1, t2, u[11], m[7]); + MONT_MAC(v0, v1, v2, u[12], m[6]); + MONT_MAC(w0, w1, w2, u[13], m[5]); + MONT_MAC(t0, t1, t2, u[14], m[4]); + MONT_MAC(v0, v1, v2, u[15], m[3]); + MONT_MAC(w0, w1, w2, u[16], m[2]); + MONT_MAC(t0, t1, t2, u[17], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[18] = t0 * n0; + MONT_MAC(t0, t1, t2, u[18], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 19 */ + MONT_MUL(s0, s1, s2, a[0], a[19]); + MONT_MUL(c0, c1, c2, a[1], a[18]); + MONT_MAC(s0, s1, s2, a[2], a[17]); + MONT_MAC(c0, c1, c2, a[3], a[16]); + MONT_MAC(s0, s1, s2, a[4], a[15]); + MONT_MAC(c0, c1, c2, a[5], a[14]); + MONT_MAC(s0, s1, s2, a[6], a[13]); + MONT_MAC(c0, c1, c2, a[7], a[12]); + MONT_MAC(s0, s1, s2, a[8], a[11]); + MONT_MAC(c0, c1, c2, a[9], a[10]); + MONT_MAC(t0, t1, t2, u[0], m[19]); + MONT_MUL(v0, v1, v2, u[1], m[18]); + MONT_MUL(w0, w1, w2, u[2], m[17]); + MONT_MAC(t0, t1, t2, u[3], m[16]); + MONT_MAC(v0, v1, v2, u[4], m[15]); + MONT_MAC(w0, w1, w2, u[5], m[14]); + MONT_MAC(t0, t1, t2, u[6], m[13]); + MONT_MAC(v0, v1, v2, u[7], m[12]); + MONT_MAC(w0, w1, w2, u[8], m[11]); + MONT_MAC(t0, t1, t2, u[9], m[10]); + MONT_MAC(v0, v1, v2, u[10], m[9]); + MONT_MAC(w0, w1, w2, u[11], m[8]); + MONT_MAC(t0, t1, t2, u[12], m[7]); + MONT_MAC(v0, v1, v2, u[13], m[6]); + MONT_MAC(w0, w1, w2, u[14], m[5]); + MONT_MAC(t0, t1, t2, u[15], m[4]); + MONT_MAC(v0, v1, v2, u[16], m[3]); + MONT_MAC(w0, w1, w2, u[17], m[2]); + MONT_MAC(t0, t1, t2, u[18], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[19] = t0 * n0; + MONT_MAC(t0, t1, t2, u[19], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 20 */ + MONT_MUL(s0, s1, s2, a[0], a[20]); + MONT_MUL(c0, c1, c2, a[1], a[19]); + MONT_MAC(s0, s1, s2, a[2], a[18]); + MONT_MAC(c0, c1, c2, a[3], a[17]); + MONT_MAC(s0, s1, s2, a[4], a[16]); + MONT_MAC(c0, c1, c2, a[5], a[15]); + MONT_MAC(s0, s1, s2, a[6], a[14]); + MONT_MAC(c0, c1, c2, a[7], a[13]); + MONT_MAC(s0, s1, s2, a[8], a[12]); + MONT_MAC(c0, c1, c2, a[9], a[11]); + MONT_MAC(t0, t1, t2, a[10], a[10]); + MONT_MUL(v0, v1, v2, u[0], m[20]); + MONT_MUL(w0, w1, w2, u[1], m[19]); + MONT_MAC(t0, t1, t2, u[2], m[18]); + MONT_MAC(v0, v1, v2, u[3], m[17]); + MONT_MAC(w0, w1, w2, u[4], m[16]); + MONT_MAC(t0, t1, t2, u[5], m[15]); + MONT_MAC(v0, v1, v2, u[6], m[14]); + MONT_MAC(w0, w1, w2, u[7], m[13]); + MONT_MAC(t0, t1, t2, u[8], m[12]); + MONT_MAC(v0, v1, v2, u[9], m[11]); + MONT_MAC(w0, w1, w2, u[10], m[10]); + MONT_MAC(t0, t1, t2, u[11], m[9]); + MONT_MAC(v0, v1, v2, u[12], m[8]); + MONT_MAC(w0, w1, w2, u[13], m[7]); + MONT_MAC(t0, t1, t2, u[14], m[6]); + MONT_MAC(v0, v1, v2, u[15], m[5]); + MONT_MAC(w0, w1, w2, u[16], m[4]); + MONT_MAC(t0, t1, t2, u[17], m[3]); + MONT_MAC(v0, v1, v2, u[18], m[2]); + MONT_MAC(w0, w1, w2, u[19], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[20] = t0 * n0; + MONT_MAC(t0, t1, t2, u[20], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 21 */ + MONT_MUL(s0, s1, s2, a[0], a[21]); + MONT_MUL(c0, c1, c2, a[1], a[20]); + MONT_MAC(s0, s1, s2, a[2], a[19]); + MONT_MAC(c0, c1, c2, a[3], a[18]); + MONT_MAC(s0, s1, s2, a[4], a[17]); + MONT_MAC(c0, c1, c2, a[5], a[16]); + MONT_MAC(s0, s1, s2, a[6], a[15]); + MONT_MAC(c0, c1, c2, a[7], a[14]); + MONT_MAC(s0, s1, s2, a[8], a[13]); + MONT_MAC(c0, c1, c2, a[9], a[12]); + MONT_MAC(s0, s1, s2, a[10], a[11]); + MONT_MAC(t0, t1, t2, u[0], m[21]); + MONT_MUL(v0, v1, v2, u[1], m[20]); + MONT_MUL(w0, w1, w2, u[2], m[19]); + MONT_MAC(t0, t1, t2, u[3], m[18]); + MONT_MAC(v0, v1, v2, u[4], m[17]); + MONT_MAC(w0, w1, w2, u[5], m[16]); + MONT_MAC(t0, t1, t2, u[6], m[15]); + MONT_MAC(v0, v1, v2, u[7], m[14]); + MONT_MAC(w0, w1, w2, u[8], m[13]); + MONT_MAC(t0, t1, t2, u[9], m[12]); + MONT_MAC(v0, v1, v2, u[10], m[11]); + MONT_MAC(w0, w1, w2, u[11], m[10]); + MONT_MAC(t0, t1, t2, u[12], m[9]); + MONT_MAC(v0, v1, v2, u[13], m[8]); + MONT_MAC(w0, w1, w2, u[14], m[7]); + MONT_MAC(t0, t1, t2, u[15], m[6]); + MONT_MAC(v0, v1, v2, u[16], m[5]); + MONT_MAC(w0, w1, w2, u[17], m[4]); + MONT_MAC(t0, t1, t2, u[18], m[3]); + MONT_MAC(v0, v1, v2, u[19], m[2]); + MONT_MAC(w0, w1, w2, u[20], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[21] = t0 * n0; + MONT_MAC(t0, t1, t2, u[21], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 22 */ + MONT_MUL(s0, s1, s2, a[0], a[22]); + MONT_MUL(c0, c1, c2, a[1], a[21]); + MONT_MAC(s0, s1, s2, a[2], a[20]); + MONT_MAC(c0, c1, c2, a[3], a[19]); + MONT_MAC(s0, s1, s2, a[4], a[18]); + MONT_MAC(c0, c1, c2, a[5], a[17]); + MONT_MAC(s0, s1, s2, a[6], a[16]); + MONT_MAC(c0, c1, c2, a[7], a[15]); + MONT_MAC(s0, s1, s2, a[8], a[14]); + MONT_MAC(c0, c1, c2, a[9], a[13]); + MONT_MAC(s0, s1, s2, a[10], a[12]); + MONT_MAC(t0, t1, t2, a[11], a[11]); + MONT_MUL(v0, v1, v2, u[0], m[22]); + MONT_MUL(w0, w1, w2, u[1], m[21]); + MONT_MAC(t0, t1, t2, u[2], m[20]); + MONT_MAC(v0, v1, v2, u[3], m[19]); + MONT_MAC(w0, w1, w2, u[4], m[18]); + MONT_MAC(t0, t1, t2, u[5], m[17]); + MONT_MAC(v0, v1, v2, u[6], m[16]); + MONT_MAC(w0, w1, w2, u[7], m[15]); + MONT_MAC(t0, t1, t2, u[8], m[14]); + MONT_MAC(v0, v1, v2, u[9], m[13]); + MONT_MAC(w0, w1, w2, u[10], m[12]); + MONT_MAC(t0, t1, t2, u[11], m[11]); + MONT_MAC(v0, v1, v2, u[12], m[10]); + MONT_MAC(w0, w1, w2, u[13], m[9]); + MONT_MAC(t0, t1, t2, u[14], m[8]); + MONT_MAC(v0, v1, v2, u[15], m[7]); + MONT_MAC(w0, w1, w2, u[16], m[6]); + MONT_MAC(t0, t1, t2, u[17], m[5]); + MONT_MAC(v0, v1, v2, u[18], m[4]); + MONT_MAC(w0, w1, w2, u[19], m[3]); + MONT_MAC(t0, t1, t2, u[20], m[2]); + MONT_MAC(v0, v1, v2, u[21], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[22] = t0 * n0; + MONT_MAC(t0, t1, t2, u[22], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 23 */ + MONT_MUL(s0, s1, s2, a[0], a[23]); + MONT_MUL(c0, c1, c2, a[1], a[22]); + MONT_MAC(s0, s1, s2, a[2], a[21]); + MONT_MAC(c0, c1, c2, a[3], a[20]); + MONT_MAC(s0, s1, s2, a[4], a[19]); + MONT_MAC(c0, c1, c2, a[5], a[18]); + MONT_MAC(s0, s1, s2, a[6], a[17]); + MONT_MAC(c0, c1, c2, a[7], a[16]); + MONT_MAC(s0, s1, s2, a[8], a[15]); + MONT_MAC(c0, c1, c2, a[9], a[14]); + MONT_MAC(s0, s1, s2, a[10], a[13]); + MONT_MAC(c0, c1, c2, a[11], a[12]); + MONT_MAC(t0, t1, t2, u[0], m[23]); + MONT_MUL(v0, v1, v2, u[1], m[22]); + MONT_MUL(w0, w1, w2, u[2], m[21]); + MONT_MAC(t0, t1, t2, u[3], m[20]); + MONT_MAC(v0, v1, v2, u[4], m[19]); + MONT_MAC(w0, w1, w2, u[5], m[18]); + MONT_MAC(t0, t1, t2, u[6], m[17]); + MONT_MAC(v0, v1, v2, u[7], m[16]); + MONT_MAC(w0, w1, w2, u[8], m[15]); + MONT_MAC(t0, t1, t2, u[9], m[14]); + MONT_MAC(v0, v1, v2, u[10], m[13]); + MONT_MAC(w0, w1, w2, u[11], m[12]); + MONT_MAC(t0, t1, t2, u[12], m[11]); + MONT_MAC(v0, v1, v2, u[13], m[10]); + MONT_MAC(w0, w1, w2, u[14], m[9]); + MONT_MAC(t0, t1, t2, u[15], m[8]); + MONT_MAC(v0, v1, v2, u[16], m[7]); + MONT_MAC(w0, w1, w2, u[17], m[6]); + MONT_MAC(t0, t1, t2, u[18], m[5]); + MONT_MAC(v0, v1, v2, u[19], m[4]); + MONT_MAC(w0, w1, w2, u[20], m[3]); + MONT_MAC(t0, t1, t2, u[21], m[2]); + MONT_MAC(v0, v1, v2, u[22], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[23] = t0 * n0; + MONT_MAC(t0, t1, t2, u[23], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 24 */ + MONT_MUL(s0, s1, s2, a[0], a[24]); + MONT_MUL(c0, c1, c2, a[1], a[23]); + MONT_MAC(s0, s1, s2, a[2], a[22]); + MONT_MAC(c0, c1, c2, a[3], a[21]); + MONT_MAC(s0, s1, s2, a[4], a[20]); + MONT_MAC(c0, c1, c2, a[5], a[19]); + MONT_MAC(s0, s1, s2, a[6], a[18]); + MONT_MAC(c0, c1, c2, a[7], a[17]); + MONT_MAC(s0, s1, s2, a[8], a[16]); + MONT_MAC(c0, c1, c2, a[9], a[15]); + MONT_MAC(s0, s1, s2, a[10], a[14]); + MONT_MAC(c0, c1, c2, a[11], a[13]); + MONT_MAC(t0, t1, t2, a[12], a[12]); + MONT_MUL(v0, v1, v2, u[0], m[24]); + MONT_MUL(w0, w1, w2, u[1], m[23]); + MONT_MAC(t0, t1, t2, u[2], m[22]); + MONT_MAC(v0, v1, v2, u[3], m[21]); + MONT_MAC(w0, w1, w2, u[4], m[20]); + MONT_MAC(t0, t1, t2, u[5], m[19]); + MONT_MAC(v0, v1, v2, u[6], m[18]); + MONT_MAC(w0, w1, w2, u[7], m[17]); + MONT_MAC(t0, t1, t2, u[8], m[16]); + MONT_MAC(v0, v1, v2, u[9], m[15]); + MONT_MAC(w0, w1, w2, u[10], m[14]); + MONT_MAC(t0, t1, t2, u[11], m[13]); + MONT_MAC(v0, v1, v2, u[12], m[12]); + MONT_MAC(w0, w1, w2, u[13], m[11]); + MONT_MAC(t0, t1, t2, u[14], m[10]); + MONT_MAC(v0, v1, v2, u[15], m[9]); + MONT_MAC(w0, w1, w2, u[16], m[8]); + MONT_MAC(t0, t1, t2, u[17], m[7]); + MONT_MAC(v0, v1, v2, u[18], m[6]); + MONT_MAC(w0, w1, w2, u[19], m[5]); + MONT_MAC(t0, t1, t2, u[20], m[4]); + MONT_MAC(v0, v1, v2, u[21], m[3]); + MONT_MAC(w0, w1, w2, u[22], m[2]); + MONT_MAC(t0, t1, t2, u[23], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[24] = t0 * n0; + MONT_MAC(t0, t1, t2, u[24], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 25 */ + MONT_MUL(s0, s1, s2, a[0], a[25]); + MONT_MUL(c0, c1, c2, a[1], a[24]); + MONT_MAC(s0, s1, s2, a[2], a[23]); + MONT_MAC(c0, c1, c2, a[3], a[22]); + MONT_MAC(s0, s1, s2, a[4], a[21]); + MONT_MAC(c0, c1, c2, a[5], a[20]); + MONT_MAC(s0, s1, s2, a[6], a[19]); + MONT_MAC(c0, c1, c2, a[7], a[18]); + MONT_MAC(s0, s1, s2, a[8], a[17]); + MONT_MAC(c0, c1, c2, a[9], a[16]); + MONT_MAC(s0, s1, s2, a[10], a[15]); + MONT_MAC(c0, c1, c2, a[11], a[14]); + MONT_MAC(s0, s1, s2, a[12], a[13]); + MONT_MAC(t0, t1, t2, u[0], m[25]); + MONT_MUL(v0, v1, v2, u[1], m[24]); + MONT_MUL(w0, w1, w2, u[2], m[23]); + MONT_MAC(t0, t1, t2, u[3], m[22]); + MONT_MAC(v0, v1, v2, u[4], m[21]); + MONT_MAC(w0, w1, w2, u[5], m[20]); + MONT_MAC(t0, t1, t2, u[6], m[19]); + MONT_MAC(v0, v1, v2, u[7], m[18]); + MONT_MAC(w0, w1, w2, u[8], m[17]); + MONT_MAC(t0, t1, t2, u[9], m[16]); + MONT_MAC(v0, v1, v2, u[10], m[15]); + MONT_MAC(w0, w1, w2, u[11], m[14]); + MONT_MAC(t0, t1, t2, u[12], m[13]); + MONT_MAC(v0, v1, v2, u[13], m[12]); + MONT_MAC(w0, w1, w2, u[14], m[11]); + MONT_MAC(t0, t1, t2, u[15], m[10]); + MONT_MAC(v0, v1, v2, u[16], m[9]); + MONT_MAC(w0, w1, w2, u[17], m[8]); + MONT_MAC(t0, t1, t2, u[18], m[7]); + MONT_MAC(v0, v1, v2, u[19], m[6]); + MONT_MAC(w0, w1, w2, u[20], m[5]); + MONT_MAC(t0, t1, t2, u[21], m[4]); + MONT_MAC(v0, v1, v2, u[22], m[3]); + MONT_MAC(w0, w1, w2, u[23], m[2]); + MONT_MAC(t0, t1, t2, u[24], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[25] = t0 * n0; + MONT_MAC(t0, t1, t2, u[25], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 26 */ + MONT_MUL(s0, s1, s2, a[0], a[26]); + MONT_MUL(c0, c1, c2, a[1], a[25]); + MONT_MAC(s0, s1, s2, a[2], a[24]); + MONT_MAC(c0, c1, c2, a[3], a[23]); + MONT_MAC(s0, s1, s2, a[4], a[22]); + MONT_MAC(c0, c1, c2, a[5], a[21]); + MONT_MAC(s0, s1, s2, a[6], a[20]); + MONT_MAC(c0, c1, c2, a[7], a[19]); + MONT_MAC(s0, s1, s2, a[8], a[18]); + MONT_MAC(c0, c1, c2, a[9], a[17]); + MONT_MAC(s0, s1, s2, a[10], a[16]); + MONT_MAC(c0, c1, c2, a[11], a[15]); + MONT_MAC(s0, s1, s2, a[12], a[14]); + MONT_MAC(t0, t1, t2, a[13], a[13]); + MONT_MUL(v0, v1, v2, u[0], m[26]); + MONT_MUL(w0, w1, w2, u[1], m[25]); + MONT_MAC(t0, t1, t2, u[2], m[24]); + MONT_MAC(v0, v1, v2, u[3], m[23]); + MONT_MAC(w0, w1, w2, u[4], m[22]); + MONT_MAC(t0, t1, t2, u[5], m[21]); + MONT_MAC(v0, v1, v2, u[6], m[20]); + MONT_MAC(w0, w1, w2, u[7], m[19]); + MONT_MAC(t0, t1, t2, u[8], m[18]); + MONT_MAC(v0, v1, v2, u[9], m[17]); + MONT_MAC(w0, w1, w2, u[10], m[16]); + MONT_MAC(t0, t1, t2, u[11], m[15]); + MONT_MAC(v0, v1, v2, u[12], m[14]); + MONT_MAC(w0, w1, w2, u[13], m[13]); + MONT_MAC(t0, t1, t2, u[14], m[12]); + MONT_MAC(v0, v1, v2, u[15], m[11]); + MONT_MAC(w0, w1, w2, u[16], m[10]); + MONT_MAC(t0, t1, t2, u[17], m[9]); + MONT_MAC(v0, v1, v2, u[18], m[8]); + MONT_MAC(w0, w1, w2, u[19], m[7]); + MONT_MAC(t0, t1, t2, u[20], m[6]); + MONT_MAC(v0, v1, v2, u[21], m[5]); + MONT_MAC(w0, w1, w2, u[22], m[4]); + MONT_MAC(t0, t1, t2, u[23], m[3]); + MONT_MAC(v0, v1, v2, u[24], m[2]); + MONT_MAC(w0, w1, w2, u[25], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[26] = t0 * n0; + MONT_MAC(t0, t1, t2, u[26], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 27 */ + MONT_MUL(s0, s1, s2, a[0], a[27]); + MONT_MUL(c0, c1, c2, a[1], a[26]); + MONT_MAC(s0, s1, s2, a[2], a[25]); + MONT_MAC(c0, c1, c2, a[3], a[24]); + MONT_MAC(s0, s1, s2, a[4], a[23]); + MONT_MAC(c0, c1, c2, a[5], a[22]); + MONT_MAC(s0, s1, s2, a[6], a[21]); + MONT_MAC(c0, c1, c2, a[7], a[20]); + MONT_MAC(s0, s1, s2, a[8], a[19]); + MONT_MAC(c0, c1, c2, a[9], a[18]); + MONT_MAC(s0, s1, s2, a[10], a[17]); + MONT_MAC(c0, c1, c2, a[11], a[16]); + MONT_MAC(s0, s1, s2, a[12], a[15]); + MONT_MAC(c0, c1, c2, a[13], a[14]); + MONT_MAC(t0, t1, t2, u[0], m[27]); + MONT_MUL(v0, v1, v2, u[1], m[26]); + MONT_MUL(w0, w1, w2, u[2], m[25]); + MONT_MAC(t0, t1, t2, u[3], m[24]); + MONT_MAC(v0, v1, v2, u[4], m[23]); + MONT_MAC(w0, w1, w2, u[5], m[22]); + MONT_MAC(t0, t1, t2, u[6], m[21]); + MONT_MAC(v0, v1, v2, u[7], m[20]); + MONT_MAC(w0, w1, w2, u[8], m[19]); + MONT_MAC(t0, t1, t2, u[9], m[18]); + MONT_MAC(v0, v1, v2, u[10], m[17]); + MONT_MAC(w0, w1, w2, u[11], m[16]); + MONT_MAC(t0, t1, t2, u[12], m[15]); + MONT_MAC(v0, v1, v2, u[13], m[14]); + MONT_MAC(w0, w1, w2, u[14], m[13]); + MONT_MAC(t0, t1, t2, u[15], m[12]); + MONT_MAC(v0, v1, v2, u[16], m[11]); + MONT_MAC(w0, w1, w2, u[17], m[10]); + MONT_MAC(t0, t1, t2, u[18], m[9]); + MONT_MAC(v0, v1, v2, u[19], m[8]); + MONT_MAC(w0, w1, w2, u[20], m[7]); + MONT_MAC(t0, t1, t2, u[21], m[6]); + MONT_MAC(v0, v1, v2, u[22], m[5]); + MONT_MAC(w0, w1, w2, u[23], m[4]); + MONT_MAC(t0, t1, t2, u[24], m[3]); + MONT_MAC(v0, v1, v2, u[25], m[2]); + MONT_MAC(w0, w1, w2, u[26], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[27] = t0 * n0; + MONT_MAC(t0, t1, t2, u[27], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 28 */ + MONT_MUL(s0, s1, s2, a[0], a[28]); + MONT_MUL(c0, c1, c2, a[1], a[27]); + MONT_MAC(s0, s1, s2, a[2], a[26]); + MONT_MAC(c0, c1, c2, a[3], a[25]); + MONT_MAC(s0, s1, s2, a[4], a[24]); + MONT_MAC(c0, c1, c2, a[5], a[23]); + MONT_MAC(s0, s1, s2, a[6], a[22]); + MONT_MAC(c0, c1, c2, a[7], a[21]); + MONT_MAC(s0, s1, s2, a[8], a[20]); + MONT_MAC(c0, c1, c2, a[9], a[19]); + MONT_MAC(s0, s1, s2, a[10], a[18]); + MONT_MAC(c0, c1, c2, a[11], a[17]); + MONT_MAC(s0, s1, s2, a[12], a[16]); + MONT_MAC(c0, c1, c2, a[13], a[15]); + MONT_MAC(t0, t1, t2, a[14], a[14]); + MONT_MUL(v0, v1, v2, u[0], m[28]); + MONT_MUL(w0, w1, w2, u[1], m[27]); + MONT_MAC(t0, t1, t2, u[2], m[26]); + MONT_MAC(v0, v1, v2, u[3], m[25]); + MONT_MAC(w0, w1, w2, u[4], m[24]); + MONT_MAC(t0, t1, t2, u[5], m[23]); + MONT_MAC(v0, v1, v2, u[6], m[22]); + MONT_MAC(w0, w1, w2, u[7], m[21]); + MONT_MAC(t0, t1, t2, u[8], m[20]); + MONT_MAC(v0, v1, v2, u[9], m[19]); + MONT_MAC(w0, w1, w2, u[10], m[18]); + MONT_MAC(t0, t1, t2, u[11], m[17]); + MONT_MAC(v0, v1, v2, u[12], m[16]); + MONT_MAC(w0, w1, w2, u[13], m[15]); + MONT_MAC(t0, t1, t2, u[14], m[14]); + MONT_MAC(v0, v1, v2, u[15], m[13]); + MONT_MAC(w0, w1, w2, u[16], m[12]); + MONT_MAC(t0, t1, t2, u[17], m[11]); + MONT_MAC(v0, v1, v2, u[18], m[10]); + MONT_MAC(w0, w1, w2, u[19], m[9]); + MONT_MAC(t0, t1, t2, u[20], m[8]); + MONT_MAC(v0, v1, v2, u[21], m[7]); + MONT_MAC(w0, w1, w2, u[22], m[6]); + MONT_MAC(t0, t1, t2, u[23], m[5]); + MONT_MAC(v0, v1, v2, u[24], m[4]); + MONT_MAC(w0, w1, w2, u[25], m[3]); + MONT_MAC(t0, t1, t2, u[26], m[2]); + MONT_MAC(v0, v1, v2, u[27], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[28] = t0 * n0; + MONT_MAC(t0, t1, t2, u[28], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 29 */ + MONT_MUL(s0, s1, s2, a[0], a[29]); + MONT_MUL(c0, c1, c2, a[1], a[28]); + MONT_MAC(s0, s1, s2, a[2], a[27]); + MONT_MAC(c0, c1, c2, a[3], a[26]); + MONT_MAC(s0, s1, s2, a[4], a[25]); + MONT_MAC(c0, c1, c2, a[5], a[24]); + MONT_MAC(s0, s1, s2, a[6], a[23]); + MONT_MAC(c0, c1, c2, a[7], a[22]); + MONT_MAC(s0, s1, s2, a[8], a[21]); + MONT_MAC(c0, c1, c2, a[9], a[20]); + MONT_MAC(s0, s1, s2, a[10], a[19]); + MONT_MAC(c0, c1, c2, a[11], a[18]); + MONT_MAC(s0, s1, s2, a[12], a[17]); + MONT_MAC(c0, c1, c2, a[13], a[16]); + MONT_MAC(s0, s1, s2, a[14], a[15]); + MONT_MAC(t0, t1, t2, u[0], m[29]); + MONT_MUL(v0, v1, v2, u[1], m[28]); + MONT_MUL(w0, w1, w2, u[2], m[27]); + MONT_MAC(t0, t1, t2, u[3], m[26]); + MONT_MAC(v0, v1, v2, u[4], m[25]); + MONT_MAC(w0, w1, w2, u[5], m[24]); + MONT_MAC(t0, t1, t2, u[6], m[23]); + MONT_MAC(v0, v1, v2, u[7], m[22]); + MONT_MAC(w0, w1, w2, u[8], m[21]); + MONT_MAC(t0, t1, t2, u[9], m[20]); + MONT_MAC(v0, v1, v2, u[10], m[19]); + MONT_MAC(w0, w1, w2, u[11], m[18]); + MONT_MAC(t0, t1, t2, u[12], m[17]); + MONT_MAC(v0, v1, v2, u[13], m[16]); + MONT_MAC(w0, w1, w2, u[14], m[15]); + MONT_MAC(t0, t1, t2, u[15], m[14]); + MONT_MAC(v0, v1, v2, u[16], m[13]); + MONT_MAC(w0, w1, w2, u[17], m[12]); + MONT_MAC(t0, t1, t2, u[18], m[11]); + MONT_MAC(v0, v1, v2, u[19], m[10]); + MONT_MAC(w0, w1, w2, u[20], m[9]); + MONT_MAC(t0, t1, t2, u[21], m[8]); + MONT_MAC(v0, v1, v2, u[22], m[7]); + MONT_MAC(w0, w1, w2, u[23], m[6]); + MONT_MAC(t0, t1, t2, u[24], m[5]); + MONT_MAC(v0, v1, v2, u[25], m[4]); + MONT_MAC(w0, w1, w2, u[26], m[3]); + MONT_MAC(t0, t1, t2, u[27], m[2]); + MONT_MAC(v0, v1, v2, u[28], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[29] = t0 * n0; + MONT_MAC(t0, t1, t2, u[29], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 30 */ + MONT_MUL(s0, s1, s2, a[0], a[30]); + MONT_MUL(c0, c1, c2, a[1], a[29]); + MONT_MAC(s0, s1, s2, a[2], a[28]); + MONT_MAC(c0, c1, c2, a[3], a[27]); + MONT_MAC(s0, s1, s2, a[4], a[26]); + MONT_MAC(c0, c1, c2, a[5], a[25]); + MONT_MAC(s0, s1, s2, a[6], a[24]); + MONT_MAC(c0, c1, c2, a[7], a[23]); + MONT_MAC(s0, s1, s2, a[8], a[22]); + MONT_MAC(c0, c1, c2, a[9], a[21]); + MONT_MAC(s0, s1, s2, a[10], a[20]); + MONT_MAC(c0, c1, c2, a[11], a[19]); + MONT_MAC(s0, s1, s2, a[12], a[18]); + MONT_MAC(c0, c1, c2, a[13], a[17]); + MONT_MAC(s0, s1, s2, a[14], a[16]); + MONT_MAC(t0, t1, t2, a[15], a[15]); + MONT_MUL(v0, v1, v2, u[0], m[30]); + MONT_MUL(w0, w1, w2, u[1], m[29]); + MONT_MAC(t0, t1, t2, u[2], m[28]); + MONT_MAC(v0, v1, v2, u[3], m[27]); + MONT_MAC(w0, w1, w2, u[4], m[26]); + MONT_MAC(t0, t1, t2, u[5], m[25]); + MONT_MAC(v0, v1, v2, u[6], m[24]); + MONT_MAC(w0, w1, w2, u[7], m[23]); + MONT_MAC(t0, t1, t2, u[8], m[22]); + MONT_MAC(v0, v1, v2, u[9], m[21]); + MONT_MAC(w0, w1, w2, u[10], m[20]); + MONT_MAC(t0, t1, t2, u[11], m[19]); + MONT_MAC(v0, v1, v2, u[12], m[18]); + MONT_MAC(w0, w1, w2, u[13], m[17]); + MONT_MAC(t0, t1, t2, u[14], m[16]); + MONT_MAC(v0, v1, v2, u[15], m[15]); + MONT_MAC(w0, w1, w2, u[16], m[14]); + MONT_MAC(t0, t1, t2, u[17], m[13]); + MONT_MAC(v0, v1, v2, u[18], m[12]); + MONT_MAC(w0, w1, w2, u[19], m[11]); + MONT_MAC(t0, t1, t2, u[20], m[10]); + MONT_MAC(v0, v1, v2, u[21], m[9]); + MONT_MAC(w0, w1, w2, u[22], m[8]); + MONT_MAC(t0, t1, t2, u[23], m[7]); + MONT_MAC(v0, v1, v2, u[24], m[6]); + MONT_MAC(w0, w1, w2, u[25], m[5]); + MONT_MAC(t0, t1, t2, u[26], m[4]); + MONT_MAC(v0, v1, v2, u[27], m[3]); + MONT_MAC(w0, w1, w2, u[28], m[2]); + MONT_MAC(t0, t1, t2, u[29], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[30] = t0 * n0; + MONT_MAC(t0, t1, t2, u[30], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 31 */ + MONT_MUL(s0, s1, s2, a[0], a[31]); + MONT_MUL(c0, c1, c2, a[1], a[30]); + MONT_MAC(s0, s1, s2, a[2], a[29]); + MONT_MAC(c0, c1, c2, a[3], a[28]); + MONT_MAC(s0, s1, s2, a[4], a[27]); + MONT_MAC(c0, c1, c2, a[5], a[26]); + MONT_MAC(s0, s1, s2, a[6], a[25]); + MONT_MAC(c0, c1, c2, a[7], a[24]); + MONT_MAC(s0, s1, s2, a[8], a[23]); + MONT_MAC(c0, c1, c2, a[9], a[22]); + MONT_MAC(s0, s1, s2, a[10], a[21]); + MONT_MAC(c0, c1, c2, a[11], a[20]); + MONT_MAC(s0, s1, s2, a[12], a[19]); + MONT_MAC(c0, c1, c2, a[13], a[18]); + MONT_MAC(s0, s1, s2, a[14], a[17]); + MONT_MAC(c0, c1, c2, a[15], a[16]); + MONT_MAC(t0, t1, t2, u[0], m[31]); + MONT_MUL(v0, v1, v2, u[1], m[30]); + MONT_MUL(w0, w1, w2, u[2], m[29]); + MONT_MAC(t0, t1, t2, u[3], m[28]); + MONT_MAC(v0, v1, v2, u[4], m[27]); + MONT_MAC(w0, w1, w2, u[5], m[26]); + MONT_MAC(t0, t1, t2, u[6], m[25]); + MONT_MAC(v0, v1, v2, u[7], m[24]); + MONT_MAC(w0, w1, w2, u[8], m[23]); + MONT_MAC(t0, t1, t2, u[9], m[22]); + MONT_MAC(v0, v1, v2, u[10], m[21]); + MONT_MAC(w0, w1, w2, u[11], m[20]); + MONT_MAC(t0, t1, t2, u[12], m[19]); + MONT_MAC(v0, v1, v2, u[13], m[18]); + MONT_MAC(w0, w1, w2, u[14], m[17]); + MONT_MAC(t0, t1, t2, u[15], m[16]); + MONT_MAC(v0, v1, v2, u[16], m[15]); + MONT_MAC(w0, w1, w2, u[17], m[14]); + MONT_MAC(t0, t1, t2, u[18], m[13]); + MONT_MAC(v0, v1, v2, u[19], m[12]); + MONT_MAC(w0, w1, w2, u[20], m[11]); + MONT_MAC(t0, t1, t2, u[21], m[10]); + MONT_MAC(v0, v1, v2, u[22], m[9]); + MONT_MAC(w0, w1, w2, u[23], m[8]); + MONT_MAC(t0, t1, t2, u[24], m[7]); + MONT_MAC(v0, v1, v2, u[25], m[6]); + MONT_MAC(w0, w1, w2, u[26], m[5]); + MONT_MAC(t0, t1, t2, u[27], m[4]); + MONT_MAC(v0, v1, v2, u[28], m[3]); + MONT_MAC(w0, w1, w2, u[29], m[2]); + MONT_MAC(t0, t1, t2, u[30], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[31] = t0 * n0; + MONT_MAC(t0, t1, t2, u[31], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 32 */ + MONT_MUL(s0, s1, s2, a[0], a[32]); + MONT_MUL(c0, c1, c2, a[1], a[31]); + MONT_MAC(s0, s1, s2, a[2], a[30]); + MONT_MAC(c0, c1, c2, a[3], a[29]); + MONT_MAC(s0, s1, s2, a[4], a[28]); + MONT_MAC(c0, c1, c2, a[5], a[27]); + MONT_MAC(s0, s1, s2, a[6], a[26]); + MONT_MAC(c0, c1, c2, a[7], a[25]); + MONT_MAC(s0, s1, s2, a[8], a[24]); + MONT_MAC(c0, c1, c2, a[9], a[23]); + MONT_MAC(s0, s1, s2, a[10], a[22]); + MONT_MAC(c0, c1, c2, a[11], a[21]); + MONT_MAC(s0, s1, s2, a[12], a[20]); + MONT_MAC(c0, c1, c2, a[13], a[19]); + MONT_MAC(s0, s1, s2, a[14], a[18]); + MONT_MAC(c0, c1, c2, a[15], a[17]); + MONT_MAC(t0, t1, t2, a[16], a[16]); + MONT_MUL(v0, v1, v2, u[0], m[32]); + MONT_MUL(w0, w1, w2, u[1], m[31]); + MONT_MAC(t0, t1, t2, u[2], m[30]); + MONT_MAC(v0, v1, v2, u[3], m[29]); + MONT_MAC(w0, w1, w2, u[4], m[28]); + MONT_MAC(t0, t1, t2, u[5], m[27]); + MONT_MAC(v0, v1, v2, u[6], m[26]); + MONT_MAC(w0, w1, w2, u[7], m[25]); + MONT_MAC(t0, t1, t2, u[8], m[24]); + MONT_MAC(v0, v1, v2, u[9], m[23]); + MONT_MAC(w0, w1, w2, u[10], m[22]); + MONT_MAC(t0, t1, t2, u[11], m[21]); + MONT_MAC(v0, v1, v2, u[12], m[20]); + MONT_MAC(w0, w1, w2, u[13], m[19]); + MONT_MAC(t0, t1, t2, u[14], m[18]); + MONT_MAC(v0, v1, v2, u[15], m[17]); + MONT_MAC(w0, w1, w2, u[16], m[16]); + MONT_MAC(t0, t1, t2, u[17], m[15]); + MONT_MAC(v0, v1, v2, u[18], m[14]); + MONT_MAC(w0, w1, w2, u[19], m[13]); + MONT_MAC(t0, t1, t2, u[20], m[12]); + MONT_MAC(v0, v1, v2, u[21], m[11]); + MONT_MAC(w0, w1, w2, u[22], m[10]); + MONT_MAC(t0, t1, t2, u[23], m[9]); + MONT_MAC(v0, v1, v2, u[24], m[8]); + MONT_MAC(w0, w1, w2, u[25], m[7]); + MONT_MAC(t0, t1, t2, u[26], m[6]); + MONT_MAC(v0, v1, v2, u[27], m[5]); + MONT_MAC(w0, w1, w2, u[28], m[4]); + MONT_MAC(t0, t1, t2, u[29], m[3]); + MONT_MAC(v0, v1, v2, u[30], m[2]); + MONT_MAC(w0, w1, w2, u[31], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[32] = t0 * n0; + MONT_MAC(t0, t1, t2, u[32], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 33 */ + MONT_MUL(s0, s1, s2, a[0], a[33]); + MONT_MUL(c0, c1, c2, a[1], a[32]); + MONT_MAC(s0, s1, s2, a[2], a[31]); + MONT_MAC(c0, c1, c2, a[3], a[30]); + MONT_MAC(s0, s1, s2, a[4], a[29]); + MONT_MAC(c0, c1, c2, a[5], a[28]); + MONT_MAC(s0, s1, s2, a[6], a[27]); + MONT_MAC(c0, c1, c2, a[7], a[26]); + MONT_MAC(s0, s1, s2, a[8], a[25]); + MONT_MAC(c0, c1, c2, a[9], a[24]); + MONT_MAC(s0, s1, s2, a[10], a[23]); + MONT_MAC(c0, c1, c2, a[11], a[22]); + MONT_MAC(s0, s1, s2, a[12], a[21]); + MONT_MAC(c0, c1, c2, a[13], a[20]); + MONT_MAC(s0, s1, s2, a[14], a[19]); + MONT_MAC(c0, c1, c2, a[15], a[18]); + MONT_MAC(s0, s1, s2, a[16], a[17]); + MONT_MAC(t0, t1, t2, u[0], m[33]); + MONT_MUL(v0, v1, v2, u[1], m[32]); + MONT_MUL(w0, w1, w2, u[2], m[31]); + MONT_MAC(t0, t1, t2, u[3], m[30]); + MONT_MAC(v0, v1, v2, u[4], m[29]); + MONT_MAC(w0, w1, w2, u[5], m[28]); + MONT_MAC(t0, t1, t2, u[6], m[27]); + MONT_MAC(v0, v1, v2, u[7], m[26]); + MONT_MAC(w0, w1, w2, u[8], m[25]); + MONT_MAC(t0, t1, t2, u[9], m[24]); + MONT_MAC(v0, v1, v2, u[10], m[23]); + MONT_MAC(w0, w1, w2, u[11], m[22]); + MONT_MAC(t0, t1, t2, u[12], m[21]); + MONT_MAC(v0, v1, v2, u[13], m[20]); + MONT_MAC(w0, w1, w2, u[14], m[19]); + MONT_MAC(t0, t1, t2, u[15], m[18]); + MONT_MAC(v0, v1, v2, u[16], m[17]); + MONT_MAC(w0, w1, w2, u[17], m[16]); + MONT_MAC(t0, t1, t2, u[18], m[15]); + MONT_MAC(v0, v1, v2, u[19], m[14]); + MONT_MAC(w0, w1, w2, u[20], m[13]); + MONT_MAC(t0, t1, t2, u[21], m[12]); + MONT_MAC(v0, v1, v2, u[22], m[11]); + MONT_MAC(w0, w1, w2, u[23], m[10]); + MONT_MAC(t0, t1, t2, u[24], m[9]); + MONT_MAC(v0, v1, v2, u[25], m[8]); + MONT_MAC(w0, w1, w2, u[26], m[7]); + MONT_MAC(t0, t1, t2, u[27], m[6]); + MONT_MAC(v0, v1, v2, u[28], m[5]); + MONT_MAC(w0, w1, w2, u[29], m[4]); + MONT_MAC(t0, t1, t2, u[30], m[3]); + MONT_MAC(v0, v1, v2, u[31], m[2]); + MONT_MAC(w0, w1, w2, u[32], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[33] = t0 * n0; + MONT_MAC(t0, t1, t2, u[33], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 34 */ + MONT_MUL(s0, s1, s2, a[0], a[34]); + MONT_MUL(c0, c1, c2, a[1], a[33]); + MONT_MAC(s0, s1, s2, a[2], a[32]); + MONT_MAC(c0, c1, c2, a[3], a[31]); + MONT_MAC(s0, s1, s2, a[4], a[30]); + MONT_MAC(c0, c1, c2, a[5], a[29]); + MONT_MAC(s0, s1, s2, a[6], a[28]); + MONT_MAC(c0, c1, c2, a[7], a[27]); + MONT_MAC(s0, s1, s2, a[8], a[26]); + MONT_MAC(c0, c1, c2, a[9], a[25]); + MONT_MAC(s0, s1, s2, a[10], a[24]); + MONT_MAC(c0, c1, c2, a[11], a[23]); + MONT_MAC(s0, s1, s2, a[12], a[22]); + MONT_MAC(c0, c1, c2, a[13], a[21]); + MONT_MAC(s0, s1, s2, a[14], a[20]); + MONT_MAC(c0, c1, c2, a[15], a[19]); + MONT_MAC(s0, s1, s2, a[16], a[18]); + MONT_MAC(t0, t1, t2, a[17], a[17]); + MONT_MUL(v0, v1, v2, u[0], m[34]); + MONT_MUL(w0, w1, w2, u[1], m[33]); + MONT_MAC(t0, t1, t2, u[2], m[32]); + MONT_MAC(v0, v1, v2, u[3], m[31]); + MONT_MAC(w0, w1, w2, u[4], m[30]); + MONT_MAC(t0, t1, t2, u[5], m[29]); + MONT_MAC(v0, v1, v2, u[6], m[28]); + MONT_MAC(w0, w1, w2, u[7], m[27]); + MONT_MAC(t0, t1, t2, u[8], m[26]); + MONT_MAC(v0, v1, v2, u[9], m[25]); + MONT_MAC(w0, w1, w2, u[10], m[24]); + MONT_MAC(t0, t1, t2, u[11], m[23]); + MONT_MAC(v0, v1, v2, u[12], m[22]); + MONT_MAC(w0, w1, w2, u[13], m[21]); + MONT_MAC(t0, t1, t2, u[14], m[20]); + MONT_MAC(v0, v1, v2, u[15], m[19]); + MONT_MAC(w0, w1, w2, u[16], m[18]); + MONT_MAC(t0, t1, t2, u[17], m[17]); + MONT_MAC(v0, v1, v2, u[18], m[16]); + MONT_MAC(w0, w1, w2, u[19], m[15]); + MONT_MAC(t0, t1, t2, u[20], m[14]); + MONT_MAC(v0, v1, v2, u[21], m[13]); + MONT_MAC(w0, w1, w2, u[22], m[12]); + MONT_MAC(t0, t1, t2, u[23], m[11]); + MONT_MAC(v0, v1, v2, u[24], m[10]); + MONT_MAC(w0, w1, w2, u[25], m[9]); + MONT_MAC(t0, t1, t2, u[26], m[8]); + MONT_MAC(v0, v1, v2, u[27], m[7]); + MONT_MAC(w0, w1, w2, u[28], m[6]); + MONT_MAC(t0, t1, t2, u[29], m[5]); + MONT_MAC(v0, v1, v2, u[30], m[4]); + MONT_MAC(w0, w1, w2, u[31], m[3]); + MONT_MAC(t0, t1, t2, u[32], m[2]); + MONT_MAC(v0, v1, v2, u[33], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[34] = t0 * n0; + MONT_MAC(t0, t1, t2, u[34], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 35 */ + MONT_MUL(s0, s1, s2, a[0], a[35]); + MONT_MUL(c0, c1, c2, a[1], a[34]); + MONT_MAC(s0, s1, s2, a[2], a[33]); + MONT_MAC(c0, c1, c2, a[3], a[32]); + MONT_MAC(s0, s1, s2, a[4], a[31]); + MONT_MAC(c0, c1, c2, a[5], a[30]); + MONT_MAC(s0, s1, s2, a[6], a[29]); + MONT_MAC(c0, c1, c2, a[7], a[28]); + MONT_MAC(s0, s1, s2, a[8], a[27]); + MONT_MAC(c0, c1, c2, a[9], a[26]); + MONT_MAC(s0, s1, s2, a[10], a[25]); + MONT_MAC(c0, c1, c2, a[11], a[24]); + MONT_MAC(s0, s1, s2, a[12], a[23]); + MONT_MAC(c0, c1, c2, a[13], a[22]); + MONT_MAC(s0, s1, s2, a[14], a[21]); + MONT_MAC(c0, c1, c2, a[15], a[20]); + MONT_MAC(s0, s1, s2, a[16], a[19]); + MONT_MAC(c0, c1, c2, a[17], a[18]); + MONT_MAC(t0, t1, t2, u[0], m[35]); + MONT_MUL(v0, v1, v2, u[1], m[34]); + MONT_MUL(w0, w1, w2, u[2], m[33]); + MONT_MAC(t0, t1, t2, u[3], m[32]); + MONT_MAC(v0, v1, v2, u[4], m[31]); + MONT_MAC(w0, w1, w2, u[5], m[30]); + MONT_MAC(t0, t1, t2, u[6], m[29]); + MONT_MAC(v0, v1, v2, u[7], m[28]); + MONT_MAC(w0, w1, w2, u[8], m[27]); + MONT_MAC(t0, t1, t2, u[9], m[26]); + MONT_MAC(v0, v1, v2, u[10], m[25]); + MONT_MAC(w0, w1, w2, u[11], m[24]); + MONT_MAC(t0, t1, t2, u[12], m[23]); + MONT_MAC(v0, v1, v2, u[13], m[22]); + MONT_MAC(w0, w1, w2, u[14], m[21]); + MONT_MAC(t0, t1, t2, u[15], m[20]); + MONT_MAC(v0, v1, v2, u[16], m[19]); + MONT_MAC(w0, w1, w2, u[17], m[18]); + MONT_MAC(t0, t1, t2, u[18], m[17]); + MONT_MAC(v0, v1, v2, u[19], m[16]); + MONT_MAC(w0, w1, w2, u[20], m[15]); + MONT_MAC(t0, t1, t2, u[21], m[14]); + MONT_MAC(v0, v1, v2, u[22], m[13]); + MONT_MAC(w0, w1, w2, u[23], m[12]); + MONT_MAC(t0, t1, t2, u[24], m[11]); + MONT_MAC(v0, v1, v2, u[25], m[10]); + MONT_MAC(w0, w1, w2, u[26], m[9]); + MONT_MAC(t0, t1, t2, u[27], m[8]); + MONT_MAC(v0, v1, v2, u[28], m[7]); + MONT_MAC(w0, w1, w2, u[29], m[6]); + MONT_MAC(t0, t1, t2, u[30], m[5]); + MONT_MAC(v0, v1, v2, u[31], m[4]); + MONT_MAC(w0, w1, w2, u[32], m[3]); + MONT_MAC(t0, t1, t2, u[33], m[2]); + MONT_MAC(v0, v1, v2, u[34], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[35] = t0 * n0; + MONT_MAC(t0, t1, t2, u[35], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 36 */ + MONT_MUL(s0, s1, s2, a[0], a[36]); + MONT_MUL(c0, c1, c2, a[1], a[35]); + MONT_MAC(s0, s1, s2, a[2], a[34]); + MONT_MAC(c0, c1, c2, a[3], a[33]); + MONT_MAC(s0, s1, s2, a[4], a[32]); + MONT_MAC(c0, c1, c2, a[5], a[31]); + MONT_MAC(s0, s1, s2, a[6], a[30]); + MONT_MAC(c0, c1, c2, a[7], a[29]); + MONT_MAC(s0, s1, s2, a[8], a[28]); + MONT_MAC(c0, c1, c2, a[9], a[27]); + MONT_MAC(s0, s1, s2, a[10], a[26]); + MONT_MAC(c0, c1, c2, a[11], a[25]); + MONT_MAC(s0, s1, s2, a[12], a[24]); + MONT_MAC(c0, c1, c2, a[13], a[23]); + MONT_MAC(s0, s1, s2, a[14], a[22]); + MONT_MAC(c0, c1, c2, a[15], a[21]); + MONT_MAC(s0, s1, s2, a[16], a[20]); + MONT_MAC(c0, c1, c2, a[17], a[19]); + MONT_MAC(t0, t1, t2, a[18], a[18]); + MONT_MUL(v0, v1, v2, u[0], m[36]); + MONT_MUL(w0, w1, w2, u[1], m[35]); + MONT_MAC(t0, t1, t2, u[2], m[34]); + MONT_MAC(v0, v1, v2, u[3], m[33]); + MONT_MAC(w0, w1, w2, u[4], m[32]); + MONT_MAC(t0, t1, t2, u[5], m[31]); + MONT_MAC(v0, v1, v2, u[6], m[30]); + MONT_MAC(w0, w1, w2, u[7], m[29]); + MONT_MAC(t0, t1, t2, u[8], m[28]); + MONT_MAC(v0, v1, v2, u[9], m[27]); + MONT_MAC(w0, w1, w2, u[10], m[26]); + MONT_MAC(t0, t1, t2, u[11], m[25]); + MONT_MAC(v0, v1, v2, u[12], m[24]); + MONT_MAC(w0, w1, w2, u[13], m[23]); + MONT_MAC(t0, t1, t2, u[14], m[22]); + MONT_MAC(v0, v1, v2, u[15], m[21]); + MONT_MAC(w0, w1, w2, u[16], m[20]); + MONT_MAC(t0, t1, t2, u[17], m[19]); + MONT_MAC(v0, v1, v2, u[18], m[18]); + MONT_MAC(w0, w1, w2, u[19], m[17]); + MONT_MAC(t0, t1, t2, u[20], m[16]); + MONT_MAC(v0, v1, v2, u[21], m[15]); + MONT_MAC(w0, w1, w2, u[22], m[14]); + MONT_MAC(t0, t1, t2, u[23], m[13]); + MONT_MAC(v0, v1, v2, u[24], m[12]); + MONT_MAC(w0, w1, w2, u[25], m[11]); + MONT_MAC(t0, t1, t2, u[26], m[10]); + MONT_MAC(v0, v1, v2, u[27], m[9]); + MONT_MAC(w0, w1, w2, u[28], m[8]); + MONT_MAC(t0, t1, t2, u[29], m[7]); + MONT_MAC(v0, v1, v2, u[30], m[6]); + MONT_MAC(w0, w1, w2, u[31], m[5]); + MONT_MAC(t0, t1, t2, u[32], m[4]); + MONT_MAC(v0, v1, v2, u[33], m[3]); + MONT_MAC(w0, w1, w2, u[34], m[2]); + MONT_MAC(t0, t1, t2, u[35], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[36] = t0 * n0; + MONT_MAC(t0, t1, t2, u[36], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 37 */ + MONT_MUL(s0, s1, s2, a[0], a[37]); + MONT_MUL(c0, c1, c2, a[1], a[36]); + MONT_MAC(s0, s1, s2, a[2], a[35]); + MONT_MAC(c0, c1, c2, a[3], a[34]); + MONT_MAC(s0, s1, s2, a[4], a[33]); + MONT_MAC(c0, c1, c2, a[5], a[32]); + MONT_MAC(s0, s1, s2, a[6], a[31]); + MONT_MAC(c0, c1, c2, a[7], a[30]); + MONT_MAC(s0, s1, s2, a[8], a[29]); + MONT_MAC(c0, c1, c2, a[9], a[28]); + MONT_MAC(s0, s1, s2, a[10], a[27]); + MONT_MAC(c0, c1, c2, a[11], a[26]); + MONT_MAC(s0, s1, s2, a[12], a[25]); + MONT_MAC(c0, c1, c2, a[13], a[24]); + MONT_MAC(s0, s1, s2, a[14], a[23]); + MONT_MAC(c0, c1, c2, a[15], a[22]); + MONT_MAC(s0, s1, s2, a[16], a[21]); + MONT_MAC(c0, c1, c2, a[17], a[20]); + MONT_MAC(s0, s1, s2, a[18], a[19]); + MONT_MAC(t0, t1, t2, u[0], m[37]); + MONT_MUL(v0, v1, v2, u[1], m[36]); + MONT_MUL(w0, w1, w2, u[2], m[35]); + MONT_MAC(t0, t1, t2, u[3], m[34]); + MONT_MAC(v0, v1, v2, u[4], m[33]); + MONT_MAC(w0, w1, w2, u[5], m[32]); + MONT_MAC(t0, t1, t2, u[6], m[31]); + MONT_MAC(v0, v1, v2, u[7], m[30]); + MONT_MAC(w0, w1, w2, u[8], m[29]); + MONT_MAC(t0, t1, t2, u[9], m[28]); + MONT_MAC(v0, v1, v2, u[10], m[27]); + MONT_MAC(w0, w1, w2, u[11], m[26]); + MONT_MAC(t0, t1, t2, u[12], m[25]); + MONT_MAC(v0, v1, v2, u[13], m[24]); + MONT_MAC(w0, w1, w2, u[14], m[23]); + MONT_MAC(t0, t1, t2, u[15], m[22]); + MONT_MAC(v0, v1, v2, u[16], m[21]); + MONT_MAC(w0, w1, w2, u[17], m[20]); + MONT_MAC(t0, t1, t2, u[18], m[19]); + MONT_MAC(v0, v1, v2, u[19], m[18]); + MONT_MAC(w0, w1, w2, u[20], m[17]); + MONT_MAC(t0, t1, t2, u[21], m[16]); + MONT_MAC(v0, v1, v2, u[22], m[15]); + MONT_MAC(w0, w1, w2, u[23], m[14]); + MONT_MAC(t0, t1, t2, u[24], m[13]); + MONT_MAC(v0, v1, v2, u[25], m[12]); + MONT_MAC(w0, w1, w2, u[26], m[11]); + MONT_MAC(t0, t1, t2, u[27], m[10]); + MONT_MAC(v0, v1, v2, u[28], m[9]); + MONT_MAC(w0, w1, w2, u[29], m[8]); + MONT_MAC(t0, t1, t2, u[30], m[7]); + MONT_MAC(v0, v1, v2, u[31], m[6]); + MONT_MAC(w0, w1, w2, u[32], m[5]); + MONT_MAC(t0, t1, t2, u[33], m[4]); + MONT_MAC(v0, v1, v2, u[34], m[3]); + MONT_MAC(w0, w1, w2, u[35], m[2]); + MONT_MAC(t0, t1, t2, u[36], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[37] = t0 * n0; + MONT_MAC(t0, t1, t2, u[37], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 38 */ + MONT_MUL(s0, s1, s2, a[0], a[38]); + MONT_MUL(c0, c1, c2, a[1], a[37]); + MONT_MAC(s0, s1, s2, a[2], a[36]); + MONT_MAC(c0, c1, c2, a[3], a[35]); + MONT_MAC(s0, s1, s2, a[4], a[34]); + MONT_MAC(c0, c1, c2, a[5], a[33]); + MONT_MAC(s0, s1, s2, a[6], a[32]); + MONT_MAC(c0, c1, c2, a[7], a[31]); + MONT_MAC(s0, s1, s2, a[8], a[30]); + MONT_MAC(c0, c1, c2, a[9], a[29]); + MONT_MAC(s0, s1, s2, a[10], a[28]); + MONT_MAC(c0, c1, c2, a[11], a[27]); + MONT_MAC(s0, s1, s2, a[12], a[26]); + MONT_MAC(c0, c1, c2, a[13], a[25]); + MONT_MAC(s0, s1, s2, a[14], a[24]); + MONT_MAC(c0, c1, c2, a[15], a[23]); + MONT_MAC(s0, s1, s2, a[16], a[22]); + MONT_MAC(c0, c1, c2, a[17], a[21]); + MONT_MAC(s0, s1, s2, a[18], a[20]); + MONT_MAC(t0, t1, t2, a[19], a[19]); + MONT_MUL(v0, v1, v2, u[0], m[38]); + MONT_MUL(w0, w1, w2, u[1], m[37]); + MONT_MAC(t0, t1, t2, u[2], m[36]); + MONT_MAC(v0, v1, v2, u[3], m[35]); + MONT_MAC(w0, w1, w2, u[4], m[34]); + MONT_MAC(t0, t1, t2, u[5], m[33]); + MONT_MAC(v0, v1, v2, u[6], m[32]); + MONT_MAC(w0, w1, w2, u[7], m[31]); + MONT_MAC(t0, t1, t2, u[8], m[30]); + MONT_MAC(v0, v1, v2, u[9], m[29]); + MONT_MAC(w0, w1, w2, u[10], m[28]); + MONT_MAC(t0, t1, t2, u[11], m[27]); + MONT_MAC(v0, v1, v2, u[12], m[26]); + MONT_MAC(w0, w1, w2, u[13], m[25]); + MONT_MAC(t0, t1, t2, u[14], m[24]); + MONT_MAC(v0, v1, v2, u[15], m[23]); + MONT_MAC(w0, w1, w2, u[16], m[22]); + MONT_MAC(t0, t1, t2, u[17], m[21]); + MONT_MAC(v0, v1, v2, u[18], m[20]); + MONT_MAC(w0, w1, w2, u[19], m[19]); + MONT_MAC(t0, t1, t2, u[20], m[18]); + MONT_MAC(v0, v1, v2, u[21], m[17]); + MONT_MAC(w0, w1, w2, u[22], m[16]); + MONT_MAC(t0, t1, t2, u[23], m[15]); + MONT_MAC(v0, v1, v2, u[24], m[14]); + MONT_MAC(w0, w1, w2, u[25], m[13]); + MONT_MAC(t0, t1, t2, u[26], m[12]); + MONT_MAC(v0, v1, v2, u[27], m[11]); + MONT_MAC(w0, w1, w2, u[28], m[10]); + MONT_MAC(t0, t1, t2, u[29], m[9]); + MONT_MAC(v0, v1, v2, u[30], m[8]); + MONT_MAC(w0, w1, w2, u[31], m[7]); + MONT_MAC(t0, t1, t2, u[32], m[6]); + MONT_MAC(v0, v1, v2, u[33], m[5]); + MONT_MAC(w0, w1, w2, u[34], m[4]); + MONT_MAC(t0, t1, t2, u[35], m[3]); + MONT_MAC(v0, v1, v2, u[36], m[2]); + MONT_MAC(w0, w1, w2, u[37], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[38] = t0 * n0; + MONT_MAC(t0, t1, t2, u[38], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 39 */ + MONT_MUL(s0, s1, s2, a[0], a[39]); + MONT_MUL(c0, c1, c2, a[1], a[38]); + MONT_MAC(s0, s1, s2, a[2], a[37]); + MONT_MAC(c0, c1, c2, a[3], a[36]); + MONT_MAC(s0, s1, s2, a[4], a[35]); + MONT_MAC(c0, c1, c2, a[5], a[34]); + MONT_MAC(s0, s1, s2, a[6], a[33]); + MONT_MAC(c0, c1, c2, a[7], a[32]); + MONT_MAC(s0, s1, s2, a[8], a[31]); + MONT_MAC(c0, c1, c2, a[9], a[30]); + MONT_MAC(s0, s1, s2, a[10], a[29]); + MONT_MAC(c0, c1, c2, a[11], a[28]); + MONT_MAC(s0, s1, s2, a[12], a[27]); + MONT_MAC(c0, c1, c2, a[13], a[26]); + MONT_MAC(s0, s1, s2, a[14], a[25]); + MONT_MAC(c0, c1, c2, a[15], a[24]); + MONT_MAC(s0, s1, s2, a[16], a[23]); + MONT_MAC(c0, c1, c2, a[17], a[22]); + MONT_MAC(s0, s1, s2, a[18], a[21]); + MONT_MAC(c0, c1, c2, a[19], a[20]); + MONT_MAC(t0, t1, t2, u[0], m[39]); + MONT_MUL(v0, v1, v2, u[1], m[38]); + MONT_MUL(w0, w1, w2, u[2], m[37]); + MONT_MAC(t0, t1, t2, u[3], m[36]); + MONT_MAC(v0, v1, v2, u[4], m[35]); + MONT_MAC(w0, w1, w2, u[5], m[34]); + MONT_MAC(t0, t1, t2, u[6], m[33]); + MONT_MAC(v0, v1, v2, u[7], m[32]); + MONT_MAC(w0, w1, w2, u[8], m[31]); + MONT_MAC(t0, t1, t2, u[9], m[30]); + MONT_MAC(v0, v1, v2, u[10], m[29]); + MONT_MAC(w0, w1, w2, u[11], m[28]); + MONT_MAC(t0, t1, t2, u[12], m[27]); + MONT_MAC(v0, v1, v2, u[13], m[26]); + MONT_MAC(w0, w1, w2, u[14], m[25]); + MONT_MAC(t0, t1, t2, u[15], m[24]); + MONT_MAC(v0, v1, v2, u[16], m[23]); + MONT_MAC(w0, w1, w2, u[17], m[22]); + MONT_MAC(t0, t1, t2, u[18], m[21]); + MONT_MAC(v0, v1, v2, u[19], m[20]); + MONT_MAC(w0, w1, w2, u[20], m[19]); + MONT_MAC(t0, t1, t2, u[21], m[18]); + MONT_MAC(v0, v1, v2, u[22], m[17]); + MONT_MAC(w0, w1, w2, u[23], m[16]); + MONT_MAC(t0, t1, t2, u[24], m[15]); + MONT_MAC(v0, v1, v2, u[25], m[14]); + MONT_MAC(w0, w1, w2, u[26], m[13]); + MONT_MAC(t0, t1, t2, u[27], m[12]); + MONT_MAC(v0, v1, v2, u[28], m[11]); + MONT_MAC(w0, w1, w2, u[29], m[10]); + MONT_MAC(t0, t1, t2, u[30], m[9]); + MONT_MAC(v0, v1, v2, u[31], m[8]); + MONT_MAC(w0, w1, w2, u[32], m[7]); + MONT_MAC(t0, t1, t2, u[33], m[6]); + MONT_MAC(v0, v1, v2, u[34], m[5]); + MONT_MAC(w0, w1, w2, u[35], m[4]); + MONT_MAC(t0, t1, t2, u[36], m[3]); + MONT_MAC(v0, v1, v2, u[37], m[2]); + MONT_MAC(w0, w1, w2, u[38], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[39] = t0 * n0; + MONT_MAC(t0, t1, t2, u[39], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 40 */ + MONT_MUL(s0, s1, s2, a[0], a[40]); + MONT_MUL(c0, c1, c2, a[1], a[39]); + MONT_MAC(s0, s1, s2, a[2], a[38]); + MONT_MAC(c0, c1, c2, a[3], a[37]); + MONT_MAC(s0, s1, s2, a[4], a[36]); + MONT_MAC(c0, c1, c2, a[5], a[35]); + MONT_MAC(s0, s1, s2, a[6], a[34]); + MONT_MAC(c0, c1, c2, a[7], a[33]); + MONT_MAC(s0, s1, s2, a[8], a[32]); + MONT_MAC(c0, c1, c2, a[9], a[31]); + MONT_MAC(s0, s1, s2, a[10], a[30]); + MONT_MAC(c0, c1, c2, a[11], a[29]); + MONT_MAC(s0, s1, s2, a[12], a[28]); + MONT_MAC(c0, c1, c2, a[13], a[27]); + MONT_MAC(s0, s1, s2, a[14], a[26]); + MONT_MAC(c0, c1, c2, a[15], a[25]); + MONT_MAC(s0, s1, s2, a[16], a[24]); + MONT_MAC(c0, c1, c2, a[17], a[23]); + MONT_MAC(s0, s1, s2, a[18], a[22]); + MONT_MAC(c0, c1, c2, a[19], a[21]); + MONT_MAC(t0, t1, t2, a[20], a[20]); + MONT_MUL(v0, v1, v2, u[0], m[40]); + MONT_MUL(w0, w1, w2, u[1], m[39]); + MONT_MAC(t0, t1, t2, u[2], m[38]); + MONT_MAC(v0, v1, v2, u[3], m[37]); + MONT_MAC(w0, w1, w2, u[4], m[36]); + MONT_MAC(t0, t1, t2, u[5], m[35]); + MONT_MAC(v0, v1, v2, u[6], m[34]); + MONT_MAC(w0, w1, w2, u[7], m[33]); + MONT_MAC(t0, t1, t2, u[8], m[32]); + MONT_MAC(v0, v1, v2, u[9], m[31]); + MONT_MAC(w0, w1, w2, u[10], m[30]); + MONT_MAC(t0, t1, t2, u[11], m[29]); + MONT_MAC(v0, v1, v2, u[12], m[28]); + MONT_MAC(w0, w1, w2, u[13], m[27]); + MONT_MAC(t0, t1, t2, u[14], m[26]); + MONT_MAC(v0, v1, v2, u[15], m[25]); + MONT_MAC(w0, w1, w2, u[16], m[24]); + MONT_MAC(t0, t1, t2, u[17], m[23]); + MONT_MAC(v0, v1, v2, u[18], m[22]); + MONT_MAC(w0, w1, w2, u[19], m[21]); + MONT_MAC(t0, t1, t2, u[20], m[20]); + MONT_MAC(v0, v1, v2, u[21], m[19]); + MONT_MAC(w0, w1, w2, u[22], m[18]); + MONT_MAC(t0, t1, t2, u[23], m[17]); + MONT_MAC(v0, v1, v2, u[24], m[16]); + MONT_MAC(w0, w1, w2, u[25], m[15]); + MONT_MAC(t0, t1, t2, u[26], m[14]); + MONT_MAC(v0, v1, v2, u[27], m[13]); + MONT_MAC(w0, w1, w2, u[28], m[12]); + MONT_MAC(t0, t1, t2, u[29], m[11]); + MONT_MAC(v0, v1, v2, u[30], m[10]); + MONT_MAC(w0, w1, w2, u[31], m[9]); + MONT_MAC(t0, t1, t2, u[32], m[8]); + MONT_MAC(v0, v1, v2, u[33], m[7]); + MONT_MAC(w0, w1, w2, u[34], m[6]); + MONT_MAC(t0, t1, t2, u[35], m[5]); + MONT_MAC(v0, v1, v2, u[36], m[4]); + MONT_MAC(w0, w1, w2, u[37], m[3]); + MONT_MAC(t0, t1, t2, u[38], m[2]); + MONT_MAC(v0, v1, v2, u[39], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[40] = t0 * n0; + MONT_MAC(t0, t1, t2, u[40], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 41 */ + MONT_MUL(s0, s1, s2, a[0], a[41]); + MONT_MUL(c0, c1, c2, a[1], a[40]); + MONT_MAC(s0, s1, s2, a[2], a[39]); + MONT_MAC(c0, c1, c2, a[3], a[38]); + MONT_MAC(s0, s1, s2, a[4], a[37]); + MONT_MAC(c0, c1, c2, a[5], a[36]); + MONT_MAC(s0, s1, s2, a[6], a[35]); + MONT_MAC(c0, c1, c2, a[7], a[34]); + MONT_MAC(s0, s1, s2, a[8], a[33]); + MONT_MAC(c0, c1, c2, a[9], a[32]); + MONT_MAC(s0, s1, s2, a[10], a[31]); + MONT_MAC(c0, c1, c2, a[11], a[30]); + MONT_MAC(s0, s1, s2, a[12], a[29]); + MONT_MAC(c0, c1, c2, a[13], a[28]); + MONT_MAC(s0, s1, s2, a[14], a[27]); + MONT_MAC(c0, c1, c2, a[15], a[26]); + MONT_MAC(s0, s1, s2, a[16], a[25]); + MONT_MAC(c0, c1, c2, a[17], a[24]); + MONT_MAC(s0, s1, s2, a[18], a[23]); + MONT_MAC(c0, c1, c2, a[19], a[22]); + MONT_MAC(s0, s1, s2, a[20], a[21]); + MONT_MAC(t0, t1, t2, u[0], m[41]); + MONT_MUL(v0, v1, v2, u[1], m[40]); + MONT_MUL(w0, w1, w2, u[2], m[39]); + MONT_MAC(t0, t1, t2, u[3], m[38]); + MONT_MAC(v0, v1, v2, u[4], m[37]); + MONT_MAC(w0, w1, w2, u[5], m[36]); + MONT_MAC(t0, t1, t2, u[6], m[35]); + MONT_MAC(v0, v1, v2, u[7], m[34]); + MONT_MAC(w0, w1, w2, u[8], m[33]); + MONT_MAC(t0, t1, t2, u[9], m[32]); + MONT_MAC(v0, v1, v2, u[10], m[31]); + MONT_MAC(w0, w1, w2, u[11], m[30]); + MONT_MAC(t0, t1, t2, u[12], m[29]); + MONT_MAC(v0, v1, v2, u[13], m[28]); + MONT_MAC(w0, w1, w2, u[14], m[27]); + MONT_MAC(t0, t1, t2, u[15], m[26]); + MONT_MAC(v0, v1, v2, u[16], m[25]); + MONT_MAC(w0, w1, w2, u[17], m[24]); + MONT_MAC(t0, t1, t2, u[18], m[23]); + MONT_MAC(v0, v1, v2, u[19], m[22]); + MONT_MAC(w0, w1, w2, u[20], m[21]); + MONT_MAC(t0, t1, t2, u[21], m[20]); + MONT_MAC(v0, v1, v2, u[22], m[19]); + MONT_MAC(w0, w1, w2, u[23], m[18]); + MONT_MAC(t0, t1, t2, u[24], m[17]); + MONT_MAC(v0, v1, v2, u[25], m[16]); + MONT_MAC(w0, w1, w2, u[26], m[15]); + MONT_MAC(t0, t1, t2, u[27], m[14]); + MONT_MAC(v0, v1, v2, u[28], m[13]); + MONT_MAC(w0, w1, w2, u[29], m[12]); + MONT_MAC(t0, t1, t2, u[30], m[11]); + MONT_MAC(v0, v1, v2, u[31], m[10]); + MONT_MAC(w0, w1, w2, u[32], m[9]); + MONT_MAC(t0, t1, t2, u[33], m[8]); + MONT_MAC(v0, v1, v2, u[34], m[7]); + MONT_MAC(w0, w1, w2, u[35], m[6]); + MONT_MAC(t0, t1, t2, u[36], m[5]); + MONT_MAC(v0, v1, v2, u[37], m[4]); + MONT_MAC(w0, w1, w2, u[38], m[3]); + MONT_MAC(t0, t1, t2, u[39], m[2]); + MONT_MAC(v0, v1, v2, u[40], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[41] = t0 * n0; + MONT_MAC(t0, t1, t2, u[41], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 42 */ + MONT_MUL(s0, s1, s2, a[0], a[42]); + MONT_MUL(c0, c1, c2, a[1], a[41]); + MONT_MAC(s0, s1, s2, a[2], a[40]); + MONT_MAC(c0, c1, c2, a[3], a[39]); + MONT_MAC(s0, s1, s2, a[4], a[38]); + MONT_MAC(c0, c1, c2, a[5], a[37]); + MONT_MAC(s0, s1, s2, a[6], a[36]); + MONT_MAC(c0, c1, c2, a[7], a[35]); + MONT_MAC(s0, s1, s2, a[8], a[34]); + MONT_MAC(c0, c1, c2, a[9], a[33]); + MONT_MAC(s0, s1, s2, a[10], a[32]); + MONT_MAC(c0, c1, c2, a[11], a[31]); + MONT_MAC(s0, s1, s2, a[12], a[30]); + MONT_MAC(c0, c1, c2, a[13], a[29]); + MONT_MAC(s0, s1, s2, a[14], a[28]); + MONT_MAC(c0, c1, c2, a[15], a[27]); + MONT_MAC(s0, s1, s2, a[16], a[26]); + MONT_MAC(c0, c1, c2, a[17], a[25]); + MONT_MAC(s0, s1, s2, a[18], a[24]); + MONT_MAC(c0, c1, c2, a[19], a[23]); + MONT_MAC(s0, s1, s2, a[20], a[22]); + MONT_MAC(t0, t1, t2, a[21], a[21]); + MONT_MUL(v0, v1, v2, u[0], m[42]); + MONT_MUL(w0, w1, w2, u[1], m[41]); + MONT_MAC(t0, t1, t2, u[2], m[40]); + MONT_MAC(v0, v1, v2, u[3], m[39]); + MONT_MAC(w0, w1, w2, u[4], m[38]); + MONT_MAC(t0, t1, t2, u[5], m[37]); + MONT_MAC(v0, v1, v2, u[6], m[36]); + MONT_MAC(w0, w1, w2, u[7], m[35]); + MONT_MAC(t0, t1, t2, u[8], m[34]); + MONT_MAC(v0, v1, v2, u[9], m[33]); + MONT_MAC(w0, w1, w2, u[10], m[32]); + MONT_MAC(t0, t1, t2, u[11], m[31]); + MONT_MAC(v0, v1, v2, u[12], m[30]); + MONT_MAC(w0, w1, w2, u[13], m[29]); + MONT_MAC(t0, t1, t2, u[14], m[28]); + MONT_MAC(v0, v1, v2, u[15], m[27]); + MONT_MAC(w0, w1, w2, u[16], m[26]); + MONT_MAC(t0, t1, t2, u[17], m[25]); + MONT_MAC(v0, v1, v2, u[18], m[24]); + MONT_MAC(w0, w1, w2, u[19], m[23]); + MONT_MAC(t0, t1, t2, u[20], m[22]); + MONT_MAC(v0, v1, v2, u[21], m[21]); + MONT_MAC(w0, w1, w2, u[22], m[20]); + MONT_MAC(t0, t1, t2, u[23], m[19]); + MONT_MAC(v0, v1, v2, u[24], m[18]); + MONT_MAC(w0, w1, w2, u[25], m[17]); + MONT_MAC(t0, t1, t2, u[26], m[16]); + MONT_MAC(v0, v1, v2, u[27], m[15]); + MONT_MAC(w0, w1, w2, u[28], m[14]); + MONT_MAC(t0, t1, t2, u[29], m[13]); + MONT_MAC(v0, v1, v2, u[30], m[12]); + MONT_MAC(w0, w1, w2, u[31], m[11]); + MONT_MAC(t0, t1, t2, u[32], m[10]); + MONT_MAC(v0, v1, v2, u[33], m[9]); + MONT_MAC(w0, w1, w2, u[34], m[8]); + MONT_MAC(t0, t1, t2, u[35], m[7]); + MONT_MAC(v0, v1, v2, u[36], m[6]); + MONT_MAC(w0, w1, w2, u[37], m[5]); + MONT_MAC(t0, t1, t2, u[38], m[4]); + MONT_MAC(v0, v1, v2, u[39], m[3]); + MONT_MAC(w0, w1, w2, u[40], m[2]); + MONT_MAC(t0, t1, t2, u[41], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[42] = t0 * n0; + MONT_MAC(t0, t1, t2, u[42], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 43 */ + MONT_MUL(s0, s1, s2, a[0], a[43]); + MONT_MUL(c0, c1, c2, a[1], a[42]); + MONT_MAC(s0, s1, s2, a[2], a[41]); + MONT_MAC(c0, c1, c2, a[3], a[40]); + MONT_MAC(s0, s1, s2, a[4], a[39]); + MONT_MAC(c0, c1, c2, a[5], a[38]); + MONT_MAC(s0, s1, s2, a[6], a[37]); + MONT_MAC(c0, c1, c2, a[7], a[36]); + MONT_MAC(s0, s1, s2, a[8], a[35]); + MONT_MAC(c0, c1, c2, a[9], a[34]); + MONT_MAC(s0, s1, s2, a[10], a[33]); + MONT_MAC(c0, c1, c2, a[11], a[32]); + MONT_MAC(s0, s1, s2, a[12], a[31]); + MONT_MAC(c0, c1, c2, a[13], a[30]); + MONT_MAC(s0, s1, s2, a[14], a[29]); + MONT_MAC(c0, c1, c2, a[15], a[28]); + MONT_MAC(s0, s1, s2, a[16], a[27]); + MONT_MAC(c0, c1, c2, a[17], a[26]); + MONT_MAC(s0, s1, s2, a[18], a[25]); + MONT_MAC(c0, c1, c2, a[19], a[24]); + MONT_MAC(s0, s1, s2, a[20], a[23]); + MONT_MAC(c0, c1, c2, a[21], a[22]); + MONT_MAC(t0, t1, t2, u[0], m[43]); + MONT_MUL(v0, v1, v2, u[1], m[42]); + MONT_MUL(w0, w1, w2, u[2], m[41]); + MONT_MAC(t0, t1, t2, u[3], m[40]); + MONT_MAC(v0, v1, v2, u[4], m[39]); + MONT_MAC(w0, w1, w2, u[5], m[38]); + MONT_MAC(t0, t1, t2, u[6], m[37]); + MONT_MAC(v0, v1, v2, u[7], m[36]); + MONT_MAC(w0, w1, w2, u[8], m[35]); + MONT_MAC(t0, t1, t2, u[9], m[34]); + MONT_MAC(v0, v1, v2, u[10], m[33]); + MONT_MAC(w0, w1, w2, u[11], m[32]); + MONT_MAC(t0, t1, t2, u[12], m[31]); + MONT_MAC(v0, v1, v2, u[13], m[30]); + MONT_MAC(w0, w1, w2, u[14], m[29]); + MONT_MAC(t0, t1, t2, u[15], m[28]); + MONT_MAC(v0, v1, v2, u[16], m[27]); + MONT_MAC(w0, w1, w2, u[17], m[26]); + MONT_MAC(t0, t1, t2, u[18], m[25]); + MONT_MAC(v0, v1, v2, u[19], m[24]); + MONT_MAC(w0, w1, w2, u[20], m[23]); + MONT_MAC(t0, t1, t2, u[21], m[22]); + MONT_MAC(v0, v1, v2, u[22], m[21]); + MONT_MAC(w0, w1, w2, u[23], m[20]); + MONT_MAC(t0, t1, t2, u[24], m[19]); + MONT_MAC(v0, v1, v2, u[25], m[18]); + MONT_MAC(w0, w1, w2, u[26], m[17]); + MONT_MAC(t0, t1, t2, u[27], m[16]); + MONT_MAC(v0, v1, v2, u[28], m[15]); + MONT_MAC(w0, w1, w2, u[29], m[14]); + MONT_MAC(t0, t1, t2, u[30], m[13]); + MONT_MAC(v0, v1, v2, u[31], m[12]); + MONT_MAC(w0, w1, w2, u[32], m[11]); + MONT_MAC(t0, t1, t2, u[33], m[10]); + MONT_MAC(v0, v1, v2, u[34], m[9]); + MONT_MAC(w0, w1, w2, u[35], m[8]); + MONT_MAC(t0, t1, t2, u[36], m[7]); + MONT_MAC(v0, v1, v2, u[37], m[6]); + MONT_MAC(w0, w1, w2, u[38], m[5]); + MONT_MAC(t0, t1, t2, u[39], m[4]); + MONT_MAC(v0, v1, v2, u[40], m[3]); + MONT_MAC(w0, w1, w2, u[41], m[2]); + MONT_MAC(t0, t1, t2, u[42], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[43] = t0 * n0; + MONT_MAC(t0, t1, t2, u[43], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 44 */ + MONT_MUL(s0, s1, s2, a[0], a[44]); + MONT_MUL(c0, c1, c2, a[1], a[43]); + MONT_MAC(s0, s1, s2, a[2], a[42]); + MONT_MAC(c0, c1, c2, a[3], a[41]); + MONT_MAC(s0, s1, s2, a[4], a[40]); + MONT_MAC(c0, c1, c2, a[5], a[39]); + MONT_MAC(s0, s1, s2, a[6], a[38]); + MONT_MAC(c0, c1, c2, a[7], a[37]); + MONT_MAC(s0, s1, s2, a[8], a[36]); + MONT_MAC(c0, c1, c2, a[9], a[35]); + MONT_MAC(s0, s1, s2, a[10], a[34]); + MONT_MAC(c0, c1, c2, a[11], a[33]); + MONT_MAC(s0, s1, s2, a[12], a[32]); + MONT_MAC(c0, c1, c2, a[13], a[31]); + MONT_MAC(s0, s1, s2, a[14], a[30]); + MONT_MAC(c0, c1, c2, a[15], a[29]); + MONT_MAC(s0, s1, s2, a[16], a[28]); + MONT_MAC(c0, c1, c2, a[17], a[27]); + MONT_MAC(s0, s1, s2, a[18], a[26]); + MONT_MAC(c0, c1, c2, a[19], a[25]); + MONT_MAC(s0, s1, s2, a[20], a[24]); + MONT_MAC(c0, c1, c2, a[21], a[23]); + MONT_MAC(t0, t1, t2, a[22], a[22]); + MONT_MUL(v0, v1, v2, u[0], m[44]); + MONT_MUL(w0, w1, w2, u[1], m[43]); + MONT_MAC(t0, t1, t2, u[2], m[42]); + MONT_MAC(v0, v1, v2, u[3], m[41]); + MONT_MAC(w0, w1, w2, u[4], m[40]); + MONT_MAC(t0, t1, t2, u[5], m[39]); + MONT_MAC(v0, v1, v2, u[6], m[38]); + MONT_MAC(w0, w1, w2, u[7], m[37]); + MONT_MAC(t0, t1, t2, u[8], m[36]); + MONT_MAC(v0, v1, v2, u[9], m[35]); + MONT_MAC(w0, w1, w2, u[10], m[34]); + MONT_MAC(t0, t1, t2, u[11], m[33]); + MONT_MAC(v0, v1, v2, u[12], m[32]); + MONT_MAC(w0, w1, w2, u[13], m[31]); + MONT_MAC(t0, t1, t2, u[14], m[30]); + MONT_MAC(v0, v1, v2, u[15], m[29]); + MONT_MAC(w0, w1, w2, u[16], m[28]); + MONT_MAC(t0, t1, t2, u[17], m[27]); + MONT_MAC(v0, v1, v2, u[18], m[26]); + MONT_MAC(w0, w1, w2, u[19], m[25]); + MONT_MAC(t0, t1, t2, u[20], m[24]); + MONT_MAC(v0, v1, v2, u[21], m[23]); + MONT_MAC(w0, w1, w2, u[22], m[22]); + MONT_MAC(t0, t1, t2, u[23], m[21]); + MONT_MAC(v0, v1, v2, u[24], m[20]); + MONT_MAC(w0, w1, w2, u[25], m[19]); + MONT_MAC(t0, t1, t2, u[26], m[18]); + MONT_MAC(v0, v1, v2, u[27], m[17]); + MONT_MAC(w0, w1, w2, u[28], m[16]); + MONT_MAC(t0, t1, t2, u[29], m[15]); + MONT_MAC(v0, v1, v2, u[30], m[14]); + MONT_MAC(w0, w1, w2, u[31], m[13]); + MONT_MAC(t0, t1, t2, u[32], m[12]); + MONT_MAC(v0, v1, v2, u[33], m[11]); + MONT_MAC(w0, w1, w2, u[34], m[10]); + MONT_MAC(t0, t1, t2, u[35], m[9]); + MONT_MAC(v0, v1, v2, u[36], m[8]); + MONT_MAC(w0, w1, w2, u[37], m[7]); + MONT_MAC(t0, t1, t2, u[38], m[6]); + MONT_MAC(v0, v1, v2, u[39], m[5]); + MONT_MAC(w0, w1, w2, u[40], m[4]); + MONT_MAC(t0, t1, t2, u[41], m[3]); + MONT_MAC(v0, v1, v2, u[42], m[2]); + MONT_MAC(w0, w1, w2, u[43], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[44] = t0 * n0; + MONT_MAC(t0, t1, t2, u[44], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 45 */ + MONT_MUL(s0, s1, s2, a[0], a[45]); + MONT_MUL(c0, c1, c2, a[1], a[44]); + MONT_MAC(s0, s1, s2, a[2], a[43]); + MONT_MAC(c0, c1, c2, a[3], a[42]); + MONT_MAC(s0, s1, s2, a[4], a[41]); + MONT_MAC(c0, c1, c2, a[5], a[40]); + MONT_MAC(s0, s1, s2, a[6], a[39]); + MONT_MAC(c0, c1, c2, a[7], a[38]); + MONT_MAC(s0, s1, s2, a[8], a[37]); + MONT_MAC(c0, c1, c2, a[9], a[36]); + MONT_MAC(s0, s1, s2, a[10], a[35]); + MONT_MAC(c0, c1, c2, a[11], a[34]); + MONT_MAC(s0, s1, s2, a[12], a[33]); + MONT_MAC(c0, c1, c2, a[13], a[32]); + MONT_MAC(s0, s1, s2, a[14], a[31]); + MONT_MAC(c0, c1, c2, a[15], a[30]); + MONT_MAC(s0, s1, s2, a[16], a[29]); + MONT_MAC(c0, c1, c2, a[17], a[28]); + MONT_MAC(s0, s1, s2, a[18], a[27]); + MONT_MAC(c0, c1, c2, a[19], a[26]); + MONT_MAC(s0, s1, s2, a[20], a[25]); + MONT_MAC(c0, c1, c2, a[21], a[24]); + MONT_MAC(s0, s1, s2, a[22], a[23]); + MONT_MAC(t0, t1, t2, u[0], m[45]); + MONT_MUL(v0, v1, v2, u[1], m[44]); + MONT_MUL(w0, w1, w2, u[2], m[43]); + MONT_MAC(t0, t1, t2, u[3], m[42]); + MONT_MAC(v0, v1, v2, u[4], m[41]); + MONT_MAC(w0, w1, w2, u[5], m[40]); + MONT_MAC(t0, t1, t2, u[6], m[39]); + MONT_MAC(v0, v1, v2, u[7], m[38]); + MONT_MAC(w0, w1, w2, u[8], m[37]); + MONT_MAC(t0, t1, t2, u[9], m[36]); + MONT_MAC(v0, v1, v2, u[10], m[35]); + MONT_MAC(w0, w1, w2, u[11], m[34]); + MONT_MAC(t0, t1, t2, u[12], m[33]); + MONT_MAC(v0, v1, v2, u[13], m[32]); + MONT_MAC(w0, w1, w2, u[14], m[31]); + MONT_MAC(t0, t1, t2, u[15], m[30]); + MONT_MAC(v0, v1, v2, u[16], m[29]); + MONT_MAC(w0, w1, w2, u[17], m[28]); + MONT_MAC(t0, t1, t2, u[18], m[27]); + MONT_MAC(v0, v1, v2, u[19], m[26]); + MONT_MAC(w0, w1, w2, u[20], m[25]); + MONT_MAC(t0, t1, t2, u[21], m[24]); + MONT_MAC(v0, v1, v2, u[22], m[23]); + MONT_MAC(w0, w1, w2, u[23], m[22]); + MONT_MAC(t0, t1, t2, u[24], m[21]); + MONT_MAC(v0, v1, v2, u[25], m[20]); + MONT_MAC(w0, w1, w2, u[26], m[19]); + MONT_MAC(t0, t1, t2, u[27], m[18]); + MONT_MAC(v0, v1, v2, u[28], m[17]); + MONT_MAC(w0, w1, w2, u[29], m[16]); + MONT_MAC(t0, t1, t2, u[30], m[15]); + MONT_MAC(v0, v1, v2, u[31], m[14]); + MONT_MAC(w0, w1, w2, u[32], m[13]); + MONT_MAC(t0, t1, t2, u[33], m[12]); + MONT_MAC(v0, v1, v2, u[34], m[11]); + MONT_MAC(w0, w1, w2, u[35], m[10]); + MONT_MAC(t0, t1, t2, u[36], m[9]); + MONT_MAC(v0, v1, v2, u[37], m[8]); + MONT_MAC(w0, w1, w2, u[38], m[7]); + MONT_MAC(t0, t1, t2, u[39], m[6]); + MONT_MAC(v0, v1, v2, u[40], m[5]); + MONT_MAC(w0, w1, w2, u[41], m[4]); + MONT_MAC(t0, t1, t2, u[42], m[3]); + MONT_MAC(v0, v1, v2, u[43], m[2]); + MONT_MAC(w0, w1, w2, u[44], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[45] = t0 * n0; + MONT_MAC(t0, t1, t2, u[45], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 46 */ + MONT_MUL(s0, s1, s2, a[0], a[46]); + MONT_MUL(c0, c1, c2, a[1], a[45]); + MONT_MAC(s0, s1, s2, a[2], a[44]); + MONT_MAC(c0, c1, c2, a[3], a[43]); + MONT_MAC(s0, s1, s2, a[4], a[42]); + MONT_MAC(c0, c1, c2, a[5], a[41]); + MONT_MAC(s0, s1, s2, a[6], a[40]); + MONT_MAC(c0, c1, c2, a[7], a[39]); + MONT_MAC(s0, s1, s2, a[8], a[38]); + MONT_MAC(c0, c1, c2, a[9], a[37]); + MONT_MAC(s0, s1, s2, a[10], a[36]); + MONT_MAC(c0, c1, c2, a[11], a[35]); + MONT_MAC(s0, s1, s2, a[12], a[34]); + MONT_MAC(c0, c1, c2, a[13], a[33]); + MONT_MAC(s0, s1, s2, a[14], a[32]); + MONT_MAC(c0, c1, c2, a[15], a[31]); + MONT_MAC(s0, s1, s2, a[16], a[30]); + MONT_MAC(c0, c1, c2, a[17], a[29]); + MONT_MAC(s0, s1, s2, a[18], a[28]); + MONT_MAC(c0, c1, c2, a[19], a[27]); + MONT_MAC(s0, s1, s2, a[20], a[26]); + MONT_MAC(c0, c1, c2, a[21], a[25]); + MONT_MAC(s0, s1, s2, a[22], a[24]); + MONT_MAC(t0, t1, t2, a[23], a[23]); + MONT_MUL(v0, v1, v2, u[0], m[46]); + MONT_MUL(w0, w1, w2, u[1], m[45]); + MONT_MAC(t0, t1, t2, u[2], m[44]); + MONT_MAC(v0, v1, v2, u[3], m[43]); + MONT_MAC(w0, w1, w2, u[4], m[42]); + MONT_MAC(t0, t1, t2, u[5], m[41]); + MONT_MAC(v0, v1, v2, u[6], m[40]); + MONT_MAC(w0, w1, w2, u[7], m[39]); + MONT_MAC(t0, t1, t2, u[8], m[38]); + MONT_MAC(v0, v1, v2, u[9], m[37]); + MONT_MAC(w0, w1, w2, u[10], m[36]); + MONT_MAC(t0, t1, t2, u[11], m[35]); + MONT_MAC(v0, v1, v2, u[12], m[34]); + MONT_MAC(w0, w1, w2, u[13], m[33]); + MONT_MAC(t0, t1, t2, u[14], m[32]); + MONT_MAC(v0, v1, v2, u[15], m[31]); + MONT_MAC(w0, w1, w2, u[16], m[30]); + MONT_MAC(t0, t1, t2, u[17], m[29]); + MONT_MAC(v0, v1, v2, u[18], m[28]); + MONT_MAC(w0, w1, w2, u[19], m[27]); + MONT_MAC(t0, t1, t2, u[20], m[26]); + MONT_MAC(v0, v1, v2, u[21], m[25]); + MONT_MAC(w0, w1, w2, u[22], m[24]); + MONT_MAC(t0, t1, t2, u[23], m[23]); + MONT_MAC(v0, v1, v2, u[24], m[22]); + MONT_MAC(w0, w1, w2, u[25], m[21]); + MONT_MAC(t0, t1, t2, u[26], m[20]); + MONT_MAC(v0, v1, v2, u[27], m[19]); + MONT_MAC(w0, w1, w2, u[28], m[18]); + MONT_MAC(t0, t1, t2, u[29], m[17]); + MONT_MAC(v0, v1, v2, u[30], m[16]); + MONT_MAC(w0, w1, w2, u[31], m[15]); + MONT_MAC(t0, t1, t2, u[32], m[14]); + MONT_MAC(v0, v1, v2, u[33], m[13]); + MONT_MAC(w0, w1, w2, u[34], m[12]); + MONT_MAC(t0, t1, t2, u[35], m[11]); + MONT_MAC(v0, v1, v2, u[36], m[10]); + MONT_MAC(w0, w1, w2, u[37], m[9]); + MONT_MAC(t0, t1, t2, u[38], m[8]); + MONT_MAC(v0, v1, v2, u[39], m[7]); + MONT_MAC(w0, w1, w2, u[40], m[6]); + MONT_MAC(t0, t1, t2, u[41], m[5]); + MONT_MAC(v0, v1, v2, u[42], m[4]); + MONT_MAC(w0, w1, w2, u[43], m[3]); + MONT_MAC(t0, t1, t2, u[44], m[2]); + MONT_MAC(v0, v1, v2, u[45], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[46] = t0 * n0; + MONT_MAC(t0, t1, t2, u[46], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 47 */ + MONT_MUL(s0, s1, s2, a[0], a[47]); + MONT_MUL(c0, c1, c2, a[1], a[46]); + MONT_MAC(s0, s1, s2, a[2], a[45]); + MONT_MAC(c0, c1, c2, a[3], a[44]); + MONT_MAC(s0, s1, s2, a[4], a[43]); + MONT_MAC(c0, c1, c2, a[5], a[42]); + MONT_MAC(s0, s1, s2, a[6], a[41]); + MONT_MAC(c0, c1, c2, a[7], a[40]); + MONT_MAC(s0, s1, s2, a[8], a[39]); + MONT_MAC(c0, c1, c2, a[9], a[38]); + MONT_MAC(s0, s1, s2, a[10], a[37]); + MONT_MAC(c0, c1, c2, a[11], a[36]); + MONT_MAC(s0, s1, s2, a[12], a[35]); + MONT_MAC(c0, c1, c2, a[13], a[34]); + MONT_MAC(s0, s1, s2, a[14], a[33]); + MONT_MAC(c0, c1, c2, a[15], a[32]); + MONT_MAC(s0, s1, s2, a[16], a[31]); + MONT_MAC(c0, c1, c2, a[17], a[30]); + MONT_MAC(s0, s1, s2, a[18], a[29]); + MONT_MAC(c0, c1, c2, a[19], a[28]); + MONT_MAC(s0, s1, s2, a[20], a[27]); + MONT_MAC(c0, c1, c2, a[21], a[26]); + MONT_MAC(s0, s1, s2, a[22], a[25]); + MONT_MAC(c0, c1, c2, a[23], a[24]); + MONT_MAC(t0, t1, t2, u[0], m[47]); + MONT_MUL(v0, v1, v2, u[1], m[46]); + MONT_MUL(w0, w1, w2, u[2], m[45]); + MONT_MAC(t0, t1, t2, u[3], m[44]); + MONT_MAC(v0, v1, v2, u[4], m[43]); + MONT_MAC(w0, w1, w2, u[5], m[42]); + MONT_MAC(t0, t1, t2, u[6], m[41]); + MONT_MAC(v0, v1, v2, u[7], m[40]); + MONT_MAC(w0, w1, w2, u[8], m[39]); + MONT_MAC(t0, t1, t2, u[9], m[38]); + MONT_MAC(v0, v1, v2, u[10], m[37]); + MONT_MAC(w0, w1, w2, u[11], m[36]); + MONT_MAC(t0, t1, t2, u[12], m[35]); + MONT_MAC(v0, v1, v2, u[13], m[34]); + MONT_MAC(w0, w1, w2, u[14], m[33]); + MONT_MAC(t0, t1, t2, u[15], m[32]); + MONT_MAC(v0, v1, v2, u[16], m[31]); + MONT_MAC(w0, w1, w2, u[17], m[30]); + MONT_MAC(t0, t1, t2, u[18], m[29]); + MONT_MAC(v0, v1, v2, u[19], m[28]); + MONT_MAC(w0, w1, w2, u[20], m[27]); + MONT_MAC(t0, t1, t2, u[21], m[26]); + MONT_MAC(v0, v1, v2, u[22], m[25]); + MONT_MAC(w0, w1, w2, u[23], m[24]); + MONT_MAC(t0, t1, t2, u[24], m[23]); + MONT_MAC(v0, v1, v2, u[25], m[22]); + MONT_MAC(w0, w1, w2, u[26], m[21]); + MONT_MAC(t0, t1, t2, u[27], m[20]); + MONT_MAC(v0, v1, v2, u[28], m[19]); + MONT_MAC(w0, w1, w2, u[29], m[18]); + MONT_MAC(t0, t1, t2, u[30], m[17]); + MONT_MAC(v0, v1, v2, u[31], m[16]); + MONT_MAC(w0, w1, w2, u[32], m[15]); + MONT_MAC(t0, t1, t2, u[33], m[14]); + MONT_MAC(v0, v1, v2, u[34], m[13]); + MONT_MAC(w0, w1, w2, u[35], m[12]); + MONT_MAC(t0, t1, t2, u[36], m[11]); + MONT_MAC(v0, v1, v2, u[37], m[10]); + MONT_MAC(w0, w1, w2, u[38], m[9]); + MONT_MAC(t0, t1, t2, u[39], m[8]); + MONT_MAC(v0, v1, v2, u[40], m[7]); + MONT_MAC(w0, w1, w2, u[41], m[6]); + MONT_MAC(t0, t1, t2, u[42], m[5]); + MONT_MAC(v0, v1, v2, u[43], m[4]); + MONT_MAC(w0, w1, w2, u[44], m[3]); + MONT_MAC(t0, t1, t2, u[45], m[2]); + MONT_MAC(v0, v1, v2, u[46], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + u[47] = t0 * n0; + MONT_MAC(t0, t1, t2, u[47], m[0]); + t0 = t1; + t1 = t2; + t2 = 0; + /* column 48 */ + MONT_MUL(s0, s1, s2, a[1], a[47]); + MONT_MUL(c0, c1, c2, a[2], a[46]); + MONT_MAC(s0, s1, s2, a[3], a[45]); + MONT_MAC(c0, c1, c2, a[4], a[44]); + MONT_MAC(s0, s1, s2, a[5], a[43]); + MONT_MAC(c0, c1, c2, a[6], a[42]); + MONT_MAC(s0, s1, s2, a[7], a[41]); + MONT_MAC(c0, c1, c2, a[8], a[40]); + MONT_MAC(s0, s1, s2, a[9], a[39]); + MONT_MAC(c0, c1, c2, a[10], a[38]); + MONT_MAC(s0, s1, s2, a[11], a[37]); + MONT_MAC(c0, c1, c2, a[12], a[36]); + MONT_MAC(s0, s1, s2, a[13], a[35]); + MONT_MAC(c0, c1, c2, a[14], a[34]); + MONT_MAC(s0, s1, s2, a[15], a[33]); + MONT_MAC(c0, c1, c2, a[16], a[32]); + MONT_MAC(s0, s1, s2, a[17], a[31]); + MONT_MAC(c0, c1, c2, a[18], a[30]); + MONT_MAC(s0, s1, s2, a[19], a[29]); + MONT_MAC(c0, c1, c2, a[20], a[28]); + MONT_MAC(s0, s1, s2, a[21], a[27]); + MONT_MAC(c0, c1, c2, a[22], a[26]); + MONT_MAC(s0, s1, s2, a[23], a[25]); + MONT_MAC(t0, t1, t2, a[24], a[24]); + MONT_MUL(v0, v1, v2, u[1], m[47]); + MONT_MUL(w0, w1, w2, u[2], m[46]); + MONT_MAC(t0, t1, t2, u[3], m[45]); + MONT_MAC(v0, v1, v2, u[4], m[44]); + MONT_MAC(w0, w1, w2, u[5], m[43]); + MONT_MAC(t0, t1, t2, u[6], m[42]); + MONT_MAC(v0, v1, v2, u[7], m[41]); + MONT_MAC(w0, w1, w2, u[8], m[40]); + MONT_MAC(t0, t1, t2, u[9], m[39]); + MONT_MAC(v0, v1, v2, u[10], m[38]); + MONT_MAC(w0, w1, w2, u[11], m[37]); + MONT_MAC(t0, t1, t2, u[12], m[36]); + MONT_MAC(v0, v1, v2, u[13], m[35]); + MONT_MAC(w0, w1, w2, u[14], m[34]); + MONT_MAC(t0, t1, t2, u[15], m[33]); + MONT_MAC(v0, v1, v2, u[16], m[32]); + MONT_MAC(w0, w1, w2, u[17], m[31]); + MONT_MAC(t0, t1, t2, u[18], m[30]); + MONT_MAC(v0, v1, v2, u[19], m[29]); + MONT_MAC(w0, w1, w2, u[20], m[28]); + MONT_MAC(t0, t1, t2, u[21], m[27]); + MONT_MAC(v0, v1, v2, u[22], m[26]); + MONT_MAC(w0, w1, w2, u[23], m[25]); + MONT_MAC(t0, t1, t2, u[24], m[24]); + MONT_MAC(v0, v1, v2, u[25], m[23]); + MONT_MAC(w0, w1, w2, u[26], m[22]); + MONT_MAC(t0, t1, t2, u[27], m[21]); + MONT_MAC(v0, v1, v2, u[28], m[20]); + MONT_MAC(w0, w1, w2, u[29], m[19]); + MONT_MAC(t0, t1, t2, u[30], m[18]); + MONT_MAC(v0, v1, v2, u[31], m[17]); + MONT_MAC(w0, w1, w2, u[32], m[16]); + MONT_MAC(t0, t1, t2, u[33], m[15]); + MONT_MAC(v0, v1, v2, u[34], m[14]); + MONT_MAC(w0, w1, w2, u[35], m[13]); + MONT_MAC(t0, t1, t2, u[36], m[12]); + MONT_MAC(v0, v1, v2, u[37], m[11]); + MONT_MAC(w0, w1, w2, u[38], m[10]); + MONT_MAC(t0, t1, t2, u[39], m[9]); + MONT_MAC(v0, v1, v2, u[40], m[8]); + MONT_MAC(w0, w1, w2, u[41], m[7]); + MONT_MAC(t0, t1, t2, u[42], m[6]); + MONT_MAC(v0, v1, v2, u[43], m[5]); + MONT_MAC(w0, w1, w2, u[44], m[4]); + MONT_MAC(t0, t1, t2, u[45], m[3]); + MONT_MAC(v0, v1, v2, u[46], m[2]); + MONT_MAC(w0, w1, w2, u[47], m[1]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[0] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 49 */ + MONT_MUL(s0, s1, s2, a[2], a[47]); + MONT_MUL(c0, c1, c2, a[3], a[46]); + MONT_MAC(s0, s1, s2, a[4], a[45]); + MONT_MAC(c0, c1, c2, a[5], a[44]); + MONT_MAC(s0, s1, s2, a[6], a[43]); + MONT_MAC(c0, c1, c2, a[7], a[42]); + MONT_MAC(s0, s1, s2, a[8], a[41]); + MONT_MAC(c0, c1, c2, a[9], a[40]); + MONT_MAC(s0, s1, s2, a[10], a[39]); + MONT_MAC(c0, c1, c2, a[11], a[38]); + MONT_MAC(s0, s1, s2, a[12], a[37]); + MONT_MAC(c0, c1, c2, a[13], a[36]); + MONT_MAC(s0, s1, s2, a[14], a[35]); + MONT_MAC(c0, c1, c2, a[15], a[34]); + MONT_MAC(s0, s1, s2, a[16], a[33]); + MONT_MAC(c0, c1, c2, a[17], a[32]); + MONT_MAC(s0, s1, s2, a[18], a[31]); + MONT_MAC(c0, c1, c2, a[19], a[30]); + MONT_MAC(s0, s1, s2, a[20], a[29]); + MONT_MAC(c0, c1, c2, a[21], a[28]); + MONT_MAC(s0, s1, s2, a[22], a[27]); + MONT_MAC(c0, c1, c2, a[23], a[26]); + MONT_MAC(s0, s1, s2, a[24], a[25]); + MONT_MAC(t0, t1, t2, u[2], m[47]); + MONT_MUL(v0, v1, v2, u[3], m[46]); + MONT_MUL(w0, w1, w2, u[4], m[45]); + MONT_MAC(t0, t1, t2, u[5], m[44]); + MONT_MAC(v0, v1, v2, u[6], m[43]); + MONT_MAC(w0, w1, w2, u[7], m[42]); + MONT_MAC(t0, t1, t2, u[8], m[41]); + MONT_MAC(v0, v1, v2, u[9], m[40]); + MONT_MAC(w0, w1, w2, u[10], m[39]); + MONT_MAC(t0, t1, t2, u[11], m[38]); + MONT_MAC(v0, v1, v2, u[12], m[37]); + MONT_MAC(w0, w1, w2, u[13], m[36]); + MONT_MAC(t0, t1, t2, u[14], m[35]); + MONT_MAC(v0, v1, v2, u[15], m[34]); + MONT_MAC(w0, w1, w2, u[16], m[33]); + MONT_MAC(t0, t1, t2, u[17], m[32]); + MONT_MAC(v0, v1, v2, u[18], m[31]); + MONT_MAC(w0, w1, w2, u[19], m[30]); + MONT_MAC(t0, t1, t2, u[20], m[29]); + MONT_MAC(v0, v1, v2, u[21], m[28]); + MONT_MAC(w0, w1, w2, u[22], m[27]); + MONT_MAC(t0, t1, t2, u[23], m[26]); + MONT_MAC(v0, v1, v2, u[24], m[25]); + MONT_MAC(w0, w1, w2, u[25], m[24]); + MONT_MAC(t0, t1, t2, u[26], m[23]); + MONT_MAC(v0, v1, v2, u[27], m[22]); + MONT_MAC(w0, w1, w2, u[28], m[21]); + MONT_MAC(t0, t1, t2, u[29], m[20]); + MONT_MAC(v0, v1, v2, u[30], m[19]); + MONT_MAC(w0, w1, w2, u[31], m[18]); + MONT_MAC(t0, t1, t2, u[32], m[17]); + MONT_MAC(v0, v1, v2, u[33], m[16]); + MONT_MAC(w0, w1, w2, u[34], m[15]); + MONT_MAC(t0, t1, t2, u[35], m[14]); + MONT_MAC(v0, v1, v2, u[36], m[13]); + MONT_MAC(w0, w1, w2, u[37], m[12]); + MONT_MAC(t0, t1, t2, u[38], m[11]); + MONT_MAC(v0, v1, v2, u[39], m[10]); + MONT_MAC(w0, w1, w2, u[40], m[9]); + MONT_MAC(t0, t1, t2, u[41], m[8]); + MONT_MAC(v0, v1, v2, u[42], m[7]); + MONT_MAC(w0, w1, w2, u[43], m[6]); + MONT_MAC(t0, t1, t2, u[44], m[5]); + MONT_MAC(v0, v1, v2, u[45], m[4]); + MONT_MAC(w0, w1, w2, u[46], m[3]); + MONT_MAC(t0, t1, t2, u[47], m[2]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[1] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 50 */ + MONT_MUL(s0, s1, s2, a[3], a[47]); + MONT_MUL(c0, c1, c2, a[4], a[46]); + MONT_MAC(s0, s1, s2, a[5], a[45]); + MONT_MAC(c0, c1, c2, a[6], a[44]); + MONT_MAC(s0, s1, s2, a[7], a[43]); + MONT_MAC(c0, c1, c2, a[8], a[42]); + MONT_MAC(s0, s1, s2, a[9], a[41]); + MONT_MAC(c0, c1, c2, a[10], a[40]); + MONT_MAC(s0, s1, s2, a[11], a[39]); + MONT_MAC(c0, c1, c2, a[12], a[38]); + MONT_MAC(s0, s1, s2, a[13], a[37]); + MONT_MAC(c0, c1, c2, a[14], a[36]); + MONT_MAC(s0, s1, s2, a[15], a[35]); + MONT_MAC(c0, c1, c2, a[16], a[34]); + MONT_MAC(s0, s1, s2, a[17], a[33]); + MONT_MAC(c0, c1, c2, a[18], a[32]); + MONT_MAC(s0, s1, s2, a[19], a[31]); + MONT_MAC(c0, c1, c2, a[20], a[30]); + MONT_MAC(s0, s1, s2, a[21], a[29]); + MONT_MAC(c0, c1, c2, a[22], a[28]); + MONT_MAC(s0, s1, s2, a[23], a[27]); + MONT_MAC(c0, c1, c2, a[24], a[26]); + MONT_MAC(t0, t1, t2, a[25], a[25]); + MONT_MUL(v0, v1, v2, u[3], m[47]); + MONT_MUL(w0, w1, w2, u[4], m[46]); + MONT_MAC(t0, t1, t2, u[5], m[45]); + MONT_MAC(v0, v1, v2, u[6], m[44]); + MONT_MAC(w0, w1, w2, u[7], m[43]); + MONT_MAC(t0, t1, t2, u[8], m[42]); + MONT_MAC(v0, v1, v2, u[9], m[41]); + MONT_MAC(w0, w1, w2, u[10], m[40]); + MONT_MAC(t0, t1, t2, u[11], m[39]); + MONT_MAC(v0, v1, v2, u[12], m[38]); + MONT_MAC(w0, w1, w2, u[13], m[37]); + MONT_MAC(t0, t1, t2, u[14], m[36]); + MONT_MAC(v0, v1, v2, u[15], m[35]); + MONT_MAC(w0, w1, w2, u[16], m[34]); + MONT_MAC(t0, t1, t2, u[17], m[33]); + MONT_MAC(v0, v1, v2, u[18], m[32]); + MONT_MAC(w0, w1, w2, u[19], m[31]); + MONT_MAC(t0, t1, t2, u[20], m[30]); + MONT_MAC(v0, v1, v2, u[21], m[29]); + MONT_MAC(w0, w1, w2, u[22], m[28]); + MONT_MAC(t0, t1, t2, u[23], m[27]); + MONT_MAC(v0, v1, v2, u[24], m[26]); + MONT_MAC(w0, w1, w2, u[25], m[25]); + MONT_MAC(t0, t1, t2, u[26], m[24]); + MONT_MAC(v0, v1, v2, u[27], m[23]); + MONT_MAC(w0, w1, w2, u[28], m[22]); + MONT_MAC(t0, t1, t2, u[29], m[21]); + MONT_MAC(v0, v1, v2, u[30], m[20]); + MONT_MAC(w0, w1, w2, u[31], m[19]); + MONT_MAC(t0, t1, t2, u[32], m[18]); + MONT_MAC(v0, v1, v2, u[33], m[17]); + MONT_MAC(w0, w1, w2, u[34], m[16]); + MONT_MAC(t0, t1, t2, u[35], m[15]); + MONT_MAC(v0, v1, v2, u[36], m[14]); + MONT_MAC(w0, w1, w2, u[37], m[13]); + MONT_MAC(t0, t1, t2, u[38], m[12]); + MONT_MAC(v0, v1, v2, u[39], m[11]); + MONT_MAC(w0, w1, w2, u[40], m[10]); + MONT_MAC(t0, t1, t2, u[41], m[9]); + MONT_MAC(v0, v1, v2, u[42], m[8]); + MONT_MAC(w0, w1, w2, u[43], m[7]); + MONT_MAC(t0, t1, t2, u[44], m[6]); + MONT_MAC(v0, v1, v2, u[45], m[5]); + MONT_MAC(w0, w1, w2, u[46], m[4]); + MONT_MAC(t0, t1, t2, u[47], m[3]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[2] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 51 */ + MONT_MUL(s0, s1, s2, a[4], a[47]); + MONT_MUL(c0, c1, c2, a[5], a[46]); + MONT_MAC(s0, s1, s2, a[6], a[45]); + MONT_MAC(c0, c1, c2, a[7], a[44]); + MONT_MAC(s0, s1, s2, a[8], a[43]); + MONT_MAC(c0, c1, c2, a[9], a[42]); + MONT_MAC(s0, s1, s2, a[10], a[41]); + MONT_MAC(c0, c1, c2, a[11], a[40]); + MONT_MAC(s0, s1, s2, a[12], a[39]); + MONT_MAC(c0, c1, c2, a[13], a[38]); + MONT_MAC(s0, s1, s2, a[14], a[37]); + MONT_MAC(c0, c1, c2, a[15], a[36]); + MONT_MAC(s0, s1, s2, a[16], a[35]); + MONT_MAC(c0, c1, c2, a[17], a[34]); + MONT_MAC(s0, s1, s2, a[18], a[33]); + MONT_MAC(c0, c1, c2, a[19], a[32]); + MONT_MAC(s0, s1, s2, a[20], a[31]); + MONT_MAC(c0, c1, c2, a[21], a[30]); + MONT_MAC(s0, s1, s2, a[22], a[29]); + MONT_MAC(c0, c1, c2, a[23], a[28]); + MONT_MAC(s0, s1, s2, a[24], a[27]); + MONT_MAC(c0, c1, c2, a[25], a[26]); + MONT_MAC(t0, t1, t2, u[4], m[47]); + MONT_MUL(v0, v1, v2, u[5], m[46]); + MONT_MUL(w0, w1, w2, u[6], m[45]); + MONT_MAC(t0, t1, t2, u[7], m[44]); + MONT_MAC(v0, v1, v2, u[8], m[43]); + MONT_MAC(w0, w1, w2, u[9], m[42]); + MONT_MAC(t0, t1, t2, u[10], m[41]); + MONT_MAC(v0, v1, v2, u[11], m[40]); + MONT_MAC(w0, w1, w2, u[12], m[39]); + MONT_MAC(t0, t1, t2, u[13], m[38]); + MONT_MAC(v0, v1, v2, u[14], m[37]); + MONT_MAC(w0, w1, w2, u[15], m[36]); + MONT_MAC(t0, t1, t2, u[16], m[35]); + MONT_MAC(v0, v1, v2, u[17], m[34]); + MONT_MAC(w0, w1, w2, u[18], m[33]); + MONT_MAC(t0, t1, t2, u[19], m[32]); + MONT_MAC(v0, v1, v2, u[20], m[31]); + MONT_MAC(w0, w1, w2, u[21], m[30]); + MONT_MAC(t0, t1, t2, u[22], m[29]); + MONT_MAC(v0, v1, v2, u[23], m[28]); + MONT_MAC(w0, w1, w2, u[24], m[27]); + MONT_MAC(t0, t1, t2, u[25], m[26]); + MONT_MAC(v0, v1, v2, u[26], m[25]); + MONT_MAC(w0, w1, w2, u[27], m[24]); + MONT_MAC(t0, t1, t2, u[28], m[23]); + MONT_MAC(v0, v1, v2, u[29], m[22]); + MONT_MAC(w0, w1, w2, u[30], m[21]); + MONT_MAC(t0, t1, t2, u[31], m[20]); + MONT_MAC(v0, v1, v2, u[32], m[19]); + MONT_MAC(w0, w1, w2, u[33], m[18]); + MONT_MAC(t0, t1, t2, u[34], m[17]); + MONT_MAC(v0, v1, v2, u[35], m[16]); + MONT_MAC(w0, w1, w2, u[36], m[15]); + MONT_MAC(t0, t1, t2, u[37], m[14]); + MONT_MAC(v0, v1, v2, u[38], m[13]); + MONT_MAC(w0, w1, w2, u[39], m[12]); + MONT_MAC(t0, t1, t2, u[40], m[11]); + MONT_MAC(v0, v1, v2, u[41], m[10]); + MONT_MAC(w0, w1, w2, u[42], m[9]); + MONT_MAC(t0, t1, t2, u[43], m[8]); + MONT_MAC(v0, v1, v2, u[44], m[7]); + MONT_MAC(w0, w1, w2, u[45], m[6]); + MONT_MAC(t0, t1, t2, u[46], m[5]); + MONT_MAC(v0, v1, v2, u[47], m[4]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[3] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 52 */ + MONT_MUL(s0, s1, s2, a[5], a[47]); + MONT_MUL(c0, c1, c2, a[6], a[46]); + MONT_MAC(s0, s1, s2, a[7], a[45]); + MONT_MAC(c0, c1, c2, a[8], a[44]); + MONT_MAC(s0, s1, s2, a[9], a[43]); + MONT_MAC(c0, c1, c2, a[10], a[42]); + MONT_MAC(s0, s1, s2, a[11], a[41]); + MONT_MAC(c0, c1, c2, a[12], a[40]); + MONT_MAC(s0, s1, s2, a[13], a[39]); + MONT_MAC(c0, c1, c2, a[14], a[38]); + MONT_MAC(s0, s1, s2, a[15], a[37]); + MONT_MAC(c0, c1, c2, a[16], a[36]); + MONT_MAC(s0, s1, s2, a[17], a[35]); + MONT_MAC(c0, c1, c2, a[18], a[34]); + MONT_MAC(s0, s1, s2, a[19], a[33]); + MONT_MAC(c0, c1, c2, a[20], a[32]); + MONT_MAC(s0, s1, s2, a[21], a[31]); + MONT_MAC(c0, c1, c2, a[22], a[30]); + MONT_MAC(s0, s1, s2, a[23], a[29]); + MONT_MAC(c0, c1, c2, a[24], a[28]); + MONT_MAC(s0, s1, s2, a[25], a[27]); + MONT_MAC(t0, t1, t2, a[26], a[26]); + MONT_MUL(v0, v1, v2, u[5], m[47]); + MONT_MUL(w0, w1, w2, u[6], m[46]); + MONT_MAC(t0, t1, t2, u[7], m[45]); + MONT_MAC(v0, v1, v2, u[8], m[44]); + MONT_MAC(w0, w1, w2, u[9], m[43]); + MONT_MAC(t0, t1, t2, u[10], m[42]); + MONT_MAC(v0, v1, v2, u[11], m[41]); + MONT_MAC(w0, w1, w2, u[12], m[40]); + MONT_MAC(t0, t1, t2, u[13], m[39]); + MONT_MAC(v0, v1, v2, u[14], m[38]); + MONT_MAC(w0, w1, w2, u[15], m[37]); + MONT_MAC(t0, t1, t2, u[16], m[36]); + MONT_MAC(v0, v1, v2, u[17], m[35]); + MONT_MAC(w0, w1, w2, u[18], m[34]); + MONT_MAC(t0, t1, t2, u[19], m[33]); + MONT_MAC(v0, v1, v2, u[20], m[32]); + MONT_MAC(w0, w1, w2, u[21], m[31]); + MONT_MAC(t0, t1, t2, u[22], m[30]); + MONT_MAC(v0, v1, v2, u[23], m[29]); + MONT_MAC(w0, w1, w2, u[24], m[28]); + MONT_MAC(t0, t1, t2, u[25], m[27]); + MONT_MAC(v0, v1, v2, u[26], m[26]); + MONT_MAC(w0, w1, w2, u[27], m[25]); + MONT_MAC(t0, t1, t2, u[28], m[24]); + MONT_MAC(v0, v1, v2, u[29], m[23]); + MONT_MAC(w0, w1, w2, u[30], m[22]); + MONT_MAC(t0, t1, t2, u[31], m[21]); + MONT_MAC(v0, v1, v2, u[32], m[20]); + MONT_MAC(w0, w1, w2, u[33], m[19]); + MONT_MAC(t0, t1, t2, u[34], m[18]); + MONT_MAC(v0, v1, v2, u[35], m[17]); + MONT_MAC(w0, w1, w2, u[36], m[16]); + MONT_MAC(t0, t1, t2, u[37], m[15]); + MONT_MAC(v0, v1, v2, u[38], m[14]); + MONT_MAC(w0, w1, w2, u[39], m[13]); + MONT_MAC(t0, t1, t2, u[40], m[12]); + MONT_MAC(v0, v1, v2, u[41], m[11]); + MONT_MAC(w0, w1, w2, u[42], m[10]); + MONT_MAC(t0, t1, t2, u[43], m[9]); + MONT_MAC(v0, v1, v2, u[44], m[8]); + MONT_MAC(w0, w1, w2, u[45], m[7]); + MONT_MAC(t0, t1, t2, u[46], m[6]); + MONT_MAC(v0, v1, v2, u[47], m[5]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[4] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 53 */ + MONT_MUL(s0, s1, s2, a[6], a[47]); + MONT_MUL(c0, c1, c2, a[7], a[46]); + MONT_MAC(s0, s1, s2, a[8], a[45]); + MONT_MAC(c0, c1, c2, a[9], a[44]); + MONT_MAC(s0, s1, s2, a[10], a[43]); + MONT_MAC(c0, c1, c2, a[11], a[42]); + MONT_MAC(s0, s1, s2, a[12], a[41]); + MONT_MAC(c0, c1, c2, a[13], a[40]); + MONT_MAC(s0, s1, s2, a[14], a[39]); + MONT_MAC(c0, c1, c2, a[15], a[38]); + MONT_MAC(s0, s1, s2, a[16], a[37]); + MONT_MAC(c0, c1, c2, a[17], a[36]); + MONT_MAC(s0, s1, s2, a[18], a[35]); + MONT_MAC(c0, c1, c2, a[19], a[34]); + MONT_MAC(s0, s1, s2, a[20], a[33]); + MONT_MAC(c0, c1, c2, a[21], a[32]); + MONT_MAC(s0, s1, s2, a[22], a[31]); + MONT_MAC(c0, c1, c2, a[23], a[30]); + MONT_MAC(s0, s1, s2, a[24], a[29]); + MONT_MAC(c0, c1, c2, a[25], a[28]); + MONT_MAC(s0, s1, s2, a[26], a[27]); + MONT_MAC(t0, t1, t2, u[6], m[47]); + MONT_MUL(v0, v1, v2, u[7], m[46]); + MONT_MUL(w0, w1, w2, u[8], m[45]); + MONT_MAC(t0, t1, t2, u[9], m[44]); + MONT_MAC(v0, v1, v2, u[10], m[43]); + MONT_MAC(w0, w1, w2, u[11], m[42]); + MONT_MAC(t0, t1, t2, u[12], m[41]); + MONT_MAC(v0, v1, v2, u[13], m[40]); + MONT_MAC(w0, w1, w2, u[14], m[39]); + MONT_MAC(t0, t1, t2, u[15], m[38]); + MONT_MAC(v0, v1, v2, u[16], m[37]); + MONT_MAC(w0, w1, w2, u[17], m[36]); + MONT_MAC(t0, t1, t2, u[18], m[35]); + MONT_MAC(v0, v1, v2, u[19], m[34]); + MONT_MAC(w0, w1, w2, u[20], m[33]); + MONT_MAC(t0, t1, t2, u[21], m[32]); + MONT_MAC(v0, v1, v2, u[22], m[31]); + MONT_MAC(w0, w1, w2, u[23], m[30]); + MONT_MAC(t0, t1, t2, u[24], m[29]); + MONT_MAC(v0, v1, v2, u[25], m[28]); + MONT_MAC(w0, w1, w2, u[26], m[27]); + MONT_MAC(t0, t1, t2, u[27], m[26]); + MONT_MAC(v0, v1, v2, u[28], m[25]); + MONT_MAC(w0, w1, w2, u[29], m[24]); + MONT_MAC(t0, t1, t2, u[30], m[23]); + MONT_MAC(v0, v1, v2, u[31], m[22]); + MONT_MAC(w0, w1, w2, u[32], m[21]); + MONT_MAC(t0, t1, t2, u[33], m[20]); + MONT_MAC(v0, v1, v2, u[34], m[19]); + MONT_MAC(w0, w1, w2, u[35], m[18]); + MONT_MAC(t0, t1, t2, u[36], m[17]); + MONT_MAC(v0, v1, v2, u[37], m[16]); + MONT_MAC(w0, w1, w2, u[38], m[15]); + MONT_MAC(t0, t1, t2, u[39], m[14]); + MONT_MAC(v0, v1, v2, u[40], m[13]); + MONT_MAC(w0, w1, w2, u[41], m[12]); + MONT_MAC(t0, t1, t2, u[42], m[11]); + MONT_MAC(v0, v1, v2, u[43], m[10]); + MONT_MAC(w0, w1, w2, u[44], m[9]); + MONT_MAC(t0, t1, t2, u[45], m[8]); + MONT_MAC(v0, v1, v2, u[46], m[7]); + MONT_MAC(w0, w1, w2, u[47], m[6]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[5] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 54 */ + MONT_MUL(s0, s1, s2, a[7], a[47]); + MONT_MUL(c0, c1, c2, a[8], a[46]); + MONT_MAC(s0, s1, s2, a[9], a[45]); + MONT_MAC(c0, c1, c2, a[10], a[44]); + MONT_MAC(s0, s1, s2, a[11], a[43]); + MONT_MAC(c0, c1, c2, a[12], a[42]); + MONT_MAC(s0, s1, s2, a[13], a[41]); + MONT_MAC(c0, c1, c2, a[14], a[40]); + MONT_MAC(s0, s1, s2, a[15], a[39]); + MONT_MAC(c0, c1, c2, a[16], a[38]); + MONT_MAC(s0, s1, s2, a[17], a[37]); + MONT_MAC(c0, c1, c2, a[18], a[36]); + MONT_MAC(s0, s1, s2, a[19], a[35]); + MONT_MAC(c0, c1, c2, a[20], a[34]); + MONT_MAC(s0, s1, s2, a[21], a[33]); + MONT_MAC(c0, c1, c2, a[22], a[32]); + MONT_MAC(s0, s1, s2, a[23], a[31]); + MONT_MAC(c0, c1, c2, a[24], a[30]); + MONT_MAC(s0, s1, s2, a[25], a[29]); + MONT_MAC(c0, c1, c2, a[26], a[28]); + MONT_MAC(t0, t1, t2, a[27], a[27]); + MONT_MUL(v0, v1, v2, u[7], m[47]); + MONT_MUL(w0, w1, w2, u[8], m[46]); + MONT_MAC(t0, t1, t2, u[9], m[45]); + MONT_MAC(v0, v1, v2, u[10], m[44]); + MONT_MAC(w0, w1, w2, u[11], m[43]); + MONT_MAC(t0, t1, t2, u[12], m[42]); + MONT_MAC(v0, v1, v2, u[13], m[41]); + MONT_MAC(w0, w1, w2, u[14], m[40]); + MONT_MAC(t0, t1, t2, u[15], m[39]); + MONT_MAC(v0, v1, v2, u[16], m[38]); + MONT_MAC(w0, w1, w2, u[17], m[37]); + MONT_MAC(t0, t1, t2, u[18], m[36]); + MONT_MAC(v0, v1, v2, u[19], m[35]); + MONT_MAC(w0, w1, w2, u[20], m[34]); + MONT_MAC(t0, t1, t2, u[21], m[33]); + MONT_MAC(v0, v1, v2, u[22], m[32]); + MONT_MAC(w0, w1, w2, u[23], m[31]); + MONT_MAC(t0, t1, t2, u[24], m[30]); + MONT_MAC(v0, v1, v2, u[25], m[29]); + MONT_MAC(w0, w1, w2, u[26], m[28]); + MONT_MAC(t0, t1, t2, u[27], m[27]); + MONT_MAC(v0, v1, v2, u[28], m[26]); + MONT_MAC(w0, w1, w2, u[29], m[25]); + MONT_MAC(t0, t1, t2, u[30], m[24]); + MONT_MAC(v0, v1, v2, u[31], m[23]); + MONT_MAC(w0, w1, w2, u[32], m[22]); + MONT_MAC(t0, t1, t2, u[33], m[21]); + MONT_MAC(v0, v1, v2, u[34], m[20]); + MONT_MAC(w0, w1, w2, u[35], m[19]); + MONT_MAC(t0, t1, t2, u[36], m[18]); + MONT_MAC(v0, v1, v2, u[37], m[17]); + MONT_MAC(w0, w1, w2, u[38], m[16]); + MONT_MAC(t0, t1, t2, u[39], m[15]); + MONT_MAC(v0, v1, v2, u[40], m[14]); + MONT_MAC(w0, w1, w2, u[41], m[13]); + MONT_MAC(t0, t1, t2, u[42], m[12]); + MONT_MAC(v0, v1, v2, u[43], m[11]); + MONT_MAC(w0, w1, w2, u[44], m[10]); + MONT_MAC(t0, t1, t2, u[45], m[9]); + MONT_MAC(v0, v1, v2, u[46], m[8]); + MONT_MAC(w0, w1, w2, u[47], m[7]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[6] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 55 */ + MONT_MUL(s0, s1, s2, a[8], a[47]); + MONT_MUL(c0, c1, c2, a[9], a[46]); + MONT_MAC(s0, s1, s2, a[10], a[45]); + MONT_MAC(c0, c1, c2, a[11], a[44]); + MONT_MAC(s0, s1, s2, a[12], a[43]); + MONT_MAC(c0, c1, c2, a[13], a[42]); + MONT_MAC(s0, s1, s2, a[14], a[41]); + MONT_MAC(c0, c1, c2, a[15], a[40]); + MONT_MAC(s0, s1, s2, a[16], a[39]); + MONT_MAC(c0, c1, c2, a[17], a[38]); + MONT_MAC(s0, s1, s2, a[18], a[37]); + MONT_MAC(c0, c1, c2, a[19], a[36]); + MONT_MAC(s0, s1, s2, a[20], a[35]); + MONT_MAC(c0, c1, c2, a[21], a[34]); + MONT_MAC(s0, s1, s2, a[22], a[33]); + MONT_MAC(c0, c1, c2, a[23], a[32]); + MONT_MAC(s0, s1, s2, a[24], a[31]); + MONT_MAC(c0, c1, c2, a[25], a[30]); + MONT_MAC(s0, s1, s2, a[26], a[29]); + MONT_MAC(c0, c1, c2, a[27], a[28]); + MONT_MAC(t0, t1, t2, u[8], m[47]); + MONT_MUL(v0, v1, v2, u[9], m[46]); + MONT_MUL(w0, w1, w2, u[10], m[45]); + MONT_MAC(t0, t1, t2, u[11], m[44]); + MONT_MAC(v0, v1, v2, u[12], m[43]); + MONT_MAC(w0, w1, w2, u[13], m[42]); + MONT_MAC(t0, t1, t2, u[14], m[41]); + MONT_MAC(v0, v1, v2, u[15], m[40]); + MONT_MAC(w0, w1, w2, u[16], m[39]); + MONT_MAC(t0, t1, t2, u[17], m[38]); + MONT_MAC(v0, v1, v2, u[18], m[37]); + MONT_MAC(w0, w1, w2, u[19], m[36]); + MONT_MAC(t0, t1, t2, u[20], m[35]); + MONT_MAC(v0, v1, v2, u[21], m[34]); + MONT_MAC(w0, w1, w2, u[22], m[33]); + MONT_MAC(t0, t1, t2, u[23], m[32]); + MONT_MAC(v0, v1, v2, u[24], m[31]); + MONT_MAC(w0, w1, w2, u[25], m[30]); + MONT_MAC(t0, t1, t2, u[26], m[29]); + MONT_MAC(v0, v1, v2, u[27], m[28]); + MONT_MAC(w0, w1, w2, u[28], m[27]); + MONT_MAC(t0, t1, t2, u[29], m[26]); + MONT_MAC(v0, v1, v2, u[30], m[25]); + MONT_MAC(w0, w1, w2, u[31], m[24]); + MONT_MAC(t0, t1, t2, u[32], m[23]); + MONT_MAC(v0, v1, v2, u[33], m[22]); + MONT_MAC(w0, w1, w2, u[34], m[21]); + MONT_MAC(t0, t1, t2, u[35], m[20]); + MONT_MAC(v0, v1, v2, u[36], m[19]); + MONT_MAC(w0, w1, w2, u[37], m[18]); + MONT_MAC(t0, t1, t2, u[38], m[17]); + MONT_MAC(v0, v1, v2, u[39], m[16]); + MONT_MAC(w0, w1, w2, u[40], m[15]); + MONT_MAC(t0, t1, t2, u[41], m[14]); + MONT_MAC(v0, v1, v2, u[42], m[13]); + MONT_MAC(w0, w1, w2, u[43], m[12]); + MONT_MAC(t0, t1, t2, u[44], m[11]); + MONT_MAC(v0, v1, v2, u[45], m[10]); + MONT_MAC(w0, w1, w2, u[46], m[9]); + MONT_MAC(t0, t1, t2, u[47], m[8]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[7] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 56 */ + MONT_MUL(s0, s1, s2, a[9], a[47]); + MONT_MUL(c0, c1, c2, a[10], a[46]); + MONT_MAC(s0, s1, s2, a[11], a[45]); + MONT_MAC(c0, c1, c2, a[12], a[44]); + MONT_MAC(s0, s1, s2, a[13], a[43]); + MONT_MAC(c0, c1, c2, a[14], a[42]); + MONT_MAC(s0, s1, s2, a[15], a[41]); + MONT_MAC(c0, c1, c2, a[16], a[40]); + MONT_MAC(s0, s1, s2, a[17], a[39]); + MONT_MAC(c0, c1, c2, a[18], a[38]); + MONT_MAC(s0, s1, s2, a[19], a[37]); + MONT_MAC(c0, c1, c2, a[20], a[36]); + MONT_MAC(s0, s1, s2, a[21], a[35]); + MONT_MAC(c0, c1, c2, a[22], a[34]); + MONT_MAC(s0, s1, s2, a[23], a[33]); + MONT_MAC(c0, c1, c2, a[24], a[32]); + MONT_MAC(s0, s1, s2, a[25], a[31]); + MONT_MAC(c0, c1, c2, a[26], a[30]); + MONT_MAC(s0, s1, s2, a[27], a[29]); + MONT_MAC(t0, t1, t2, a[28], a[28]); + MONT_MUL(v0, v1, v2, u[9], m[47]); + MONT_MUL(w0, w1, w2, u[10], m[46]); + MONT_MAC(t0, t1, t2, u[11], m[45]); + MONT_MAC(v0, v1, v2, u[12], m[44]); + MONT_MAC(w0, w1, w2, u[13], m[43]); + MONT_MAC(t0, t1, t2, u[14], m[42]); + MONT_MAC(v0, v1, v2, u[15], m[41]); + MONT_MAC(w0, w1, w2, u[16], m[40]); + MONT_MAC(t0, t1, t2, u[17], m[39]); + MONT_MAC(v0, v1, v2, u[18], m[38]); + MONT_MAC(w0, w1, w2, u[19], m[37]); + MONT_MAC(t0, t1, t2, u[20], m[36]); + MONT_MAC(v0, v1, v2, u[21], m[35]); + MONT_MAC(w0, w1, w2, u[22], m[34]); + MONT_MAC(t0, t1, t2, u[23], m[33]); + MONT_MAC(v0, v1, v2, u[24], m[32]); + MONT_MAC(w0, w1, w2, u[25], m[31]); + MONT_MAC(t0, t1, t2, u[26], m[30]); + MONT_MAC(v0, v1, v2, u[27], m[29]); + MONT_MAC(w0, w1, w2, u[28], m[28]); + MONT_MAC(t0, t1, t2, u[29], m[27]); + MONT_MAC(v0, v1, v2, u[30], m[26]); + MONT_MAC(w0, w1, w2, u[31], m[25]); + MONT_MAC(t0, t1, t2, u[32], m[24]); + MONT_MAC(v0, v1, v2, u[33], m[23]); + MONT_MAC(w0, w1, w2, u[34], m[22]); + MONT_MAC(t0, t1, t2, u[35], m[21]); + MONT_MAC(v0, v1, v2, u[36], m[20]); + MONT_MAC(w0, w1, w2, u[37], m[19]); + MONT_MAC(t0, t1, t2, u[38], m[18]); + MONT_MAC(v0, v1, v2, u[39], m[17]); + MONT_MAC(w0, w1, w2, u[40], m[16]); + MONT_MAC(t0, t1, t2, u[41], m[15]); + MONT_MAC(v0, v1, v2, u[42], m[14]); + MONT_MAC(w0, w1, w2, u[43], m[13]); + MONT_MAC(t0, t1, t2, u[44], m[12]); + MONT_MAC(v0, v1, v2, u[45], m[11]); + MONT_MAC(w0, w1, w2, u[46], m[10]); + MONT_MAC(t0, t1, t2, u[47], m[9]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[8] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 57 */ + MONT_MUL(s0, s1, s2, a[10], a[47]); + MONT_MUL(c0, c1, c2, a[11], a[46]); + MONT_MAC(s0, s1, s2, a[12], a[45]); + MONT_MAC(c0, c1, c2, a[13], a[44]); + MONT_MAC(s0, s1, s2, a[14], a[43]); + MONT_MAC(c0, c1, c2, a[15], a[42]); + MONT_MAC(s0, s1, s2, a[16], a[41]); + MONT_MAC(c0, c1, c2, a[17], a[40]); + MONT_MAC(s0, s1, s2, a[18], a[39]); + MONT_MAC(c0, c1, c2, a[19], a[38]); + MONT_MAC(s0, s1, s2, a[20], a[37]); + MONT_MAC(c0, c1, c2, a[21], a[36]); + MONT_MAC(s0, s1, s2, a[22], a[35]); + MONT_MAC(c0, c1, c2, a[23], a[34]); + MONT_MAC(s0, s1, s2, a[24], a[33]); + MONT_MAC(c0, c1, c2, a[25], a[32]); + MONT_MAC(s0, s1, s2, a[26], a[31]); + MONT_MAC(c0, c1, c2, a[27], a[30]); + MONT_MAC(s0, s1, s2, a[28], a[29]); + MONT_MAC(t0, t1, t2, u[10], m[47]); + MONT_MUL(v0, v1, v2, u[11], m[46]); + MONT_MUL(w0, w1, w2, u[12], m[45]); + MONT_MAC(t0, t1, t2, u[13], m[44]); + MONT_MAC(v0, v1, v2, u[14], m[43]); + MONT_MAC(w0, w1, w2, u[15], m[42]); + MONT_MAC(t0, t1, t2, u[16], m[41]); + MONT_MAC(v0, v1, v2, u[17], m[40]); + MONT_MAC(w0, w1, w2, u[18], m[39]); + MONT_MAC(t0, t1, t2, u[19], m[38]); + MONT_MAC(v0, v1, v2, u[20], m[37]); + MONT_MAC(w0, w1, w2, u[21], m[36]); + MONT_MAC(t0, t1, t2, u[22], m[35]); + MONT_MAC(v0, v1, v2, u[23], m[34]); + MONT_MAC(w0, w1, w2, u[24], m[33]); + MONT_MAC(t0, t1, t2, u[25], m[32]); + MONT_MAC(v0, v1, v2, u[26], m[31]); + MONT_MAC(w0, w1, w2, u[27], m[30]); + MONT_MAC(t0, t1, t2, u[28], m[29]); + MONT_MAC(v0, v1, v2, u[29], m[28]); + MONT_MAC(w0, w1, w2, u[30], m[27]); + MONT_MAC(t0, t1, t2, u[31], m[26]); + MONT_MAC(v0, v1, v2, u[32], m[25]); + MONT_MAC(w0, w1, w2, u[33], m[24]); + MONT_MAC(t0, t1, t2, u[34], m[23]); + MONT_MAC(v0, v1, v2, u[35], m[22]); + MONT_MAC(w0, w1, w2, u[36], m[21]); + MONT_MAC(t0, t1, t2, u[37], m[20]); + MONT_MAC(v0, v1, v2, u[38], m[19]); + MONT_MAC(w0, w1, w2, u[39], m[18]); + MONT_MAC(t0, t1, t2, u[40], m[17]); + MONT_MAC(v0, v1, v2, u[41], m[16]); + MONT_MAC(w0, w1, w2, u[42], m[15]); + MONT_MAC(t0, t1, t2, u[43], m[14]); + MONT_MAC(v0, v1, v2, u[44], m[13]); + MONT_MAC(w0, w1, w2, u[45], m[12]); + MONT_MAC(t0, t1, t2, u[46], m[11]); + MONT_MAC(v0, v1, v2, u[47], m[10]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[9] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 58 */ + MONT_MUL(s0, s1, s2, a[11], a[47]); + MONT_MUL(c0, c1, c2, a[12], a[46]); + MONT_MAC(s0, s1, s2, a[13], a[45]); + MONT_MAC(c0, c1, c2, a[14], a[44]); + MONT_MAC(s0, s1, s2, a[15], a[43]); + MONT_MAC(c0, c1, c2, a[16], a[42]); + MONT_MAC(s0, s1, s2, a[17], a[41]); + MONT_MAC(c0, c1, c2, a[18], a[40]); + MONT_MAC(s0, s1, s2, a[19], a[39]); + MONT_MAC(c0, c1, c2, a[20], a[38]); + MONT_MAC(s0, s1, s2, a[21], a[37]); + MONT_MAC(c0, c1, c2, a[22], a[36]); + MONT_MAC(s0, s1, s2, a[23], a[35]); + MONT_MAC(c0, c1, c2, a[24], a[34]); + MONT_MAC(s0, s1, s2, a[25], a[33]); + MONT_MAC(c0, c1, c2, a[26], a[32]); + MONT_MAC(s0, s1, s2, a[27], a[31]); + MONT_MAC(c0, c1, c2, a[28], a[30]); + MONT_MAC(t0, t1, t2, a[29], a[29]); + MONT_MUL(v0, v1, v2, u[11], m[47]); + MONT_MUL(w0, w1, w2, u[12], m[46]); + MONT_MAC(t0, t1, t2, u[13], m[45]); + MONT_MAC(v0, v1, v2, u[14], m[44]); + MONT_MAC(w0, w1, w2, u[15], m[43]); + MONT_MAC(t0, t1, t2, u[16], m[42]); + MONT_MAC(v0, v1, v2, u[17], m[41]); + MONT_MAC(w0, w1, w2, u[18], m[40]); + MONT_MAC(t0, t1, t2, u[19], m[39]); + MONT_MAC(v0, v1, v2, u[20], m[38]); + MONT_MAC(w0, w1, w2, u[21], m[37]); + MONT_MAC(t0, t1, t2, u[22], m[36]); + MONT_MAC(v0, v1, v2, u[23], m[35]); + MONT_MAC(w0, w1, w2, u[24], m[34]); + MONT_MAC(t0, t1, t2, u[25], m[33]); + MONT_MAC(v0, v1, v2, u[26], m[32]); + MONT_MAC(w0, w1, w2, u[27], m[31]); + MONT_MAC(t0, t1, t2, u[28], m[30]); + MONT_MAC(v0, v1, v2, u[29], m[29]); + MONT_MAC(w0, w1, w2, u[30], m[28]); + MONT_MAC(t0, t1, t2, u[31], m[27]); + MONT_MAC(v0, v1, v2, u[32], m[26]); + MONT_MAC(w0, w1, w2, u[33], m[25]); + MONT_MAC(t0, t1, t2, u[34], m[24]); + MONT_MAC(v0, v1, v2, u[35], m[23]); + MONT_MAC(w0, w1, w2, u[36], m[22]); + MONT_MAC(t0, t1, t2, u[37], m[21]); + MONT_MAC(v0, v1, v2, u[38], m[20]); + MONT_MAC(w0, w1, w2, u[39], m[19]); + MONT_MAC(t0, t1, t2, u[40], m[18]); + MONT_MAC(v0, v1, v2, u[41], m[17]); + MONT_MAC(w0, w1, w2, u[42], m[16]); + MONT_MAC(t0, t1, t2, u[43], m[15]); + MONT_MAC(v0, v1, v2, u[44], m[14]); + MONT_MAC(w0, w1, w2, u[45], m[13]); + MONT_MAC(t0, t1, t2, u[46], m[12]); + MONT_MAC(v0, v1, v2, u[47], m[11]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[10] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 59 */ + MONT_MUL(s0, s1, s2, a[12], a[47]); + MONT_MUL(c0, c1, c2, a[13], a[46]); + MONT_MAC(s0, s1, s2, a[14], a[45]); + MONT_MAC(c0, c1, c2, a[15], a[44]); + MONT_MAC(s0, s1, s2, a[16], a[43]); + MONT_MAC(c0, c1, c2, a[17], a[42]); + MONT_MAC(s0, s1, s2, a[18], a[41]); + MONT_MAC(c0, c1, c2, a[19], a[40]); + MONT_MAC(s0, s1, s2, a[20], a[39]); + MONT_MAC(c0, c1, c2, a[21], a[38]); + MONT_MAC(s0, s1, s2, a[22], a[37]); + MONT_MAC(c0, c1, c2, a[23], a[36]); + MONT_MAC(s0, s1, s2, a[24], a[35]); + MONT_MAC(c0, c1, c2, a[25], a[34]); + MONT_MAC(s0, s1, s2, a[26], a[33]); + MONT_MAC(c0, c1, c2, a[27], a[32]); + MONT_MAC(s0, s1, s2, a[28], a[31]); + MONT_MAC(c0, c1, c2, a[29], a[30]); + MONT_MAC(t0, t1, t2, u[12], m[47]); + MONT_MUL(v0, v1, v2, u[13], m[46]); + MONT_MUL(w0, w1, w2, u[14], m[45]); + MONT_MAC(t0, t1, t2, u[15], m[44]); + MONT_MAC(v0, v1, v2, u[16], m[43]); + MONT_MAC(w0, w1, w2, u[17], m[42]); + MONT_MAC(t0, t1, t2, u[18], m[41]); + MONT_MAC(v0, v1, v2, u[19], m[40]); + MONT_MAC(w0, w1, w2, u[20], m[39]); + MONT_MAC(t0, t1, t2, u[21], m[38]); + MONT_MAC(v0, v1, v2, u[22], m[37]); + MONT_MAC(w0, w1, w2, u[23], m[36]); + MONT_MAC(t0, t1, t2, u[24], m[35]); + MONT_MAC(v0, v1, v2, u[25], m[34]); + MONT_MAC(w0, w1, w2, u[26], m[33]); + MONT_MAC(t0, t1, t2, u[27], m[32]); + MONT_MAC(v0, v1, v2, u[28], m[31]); + MONT_MAC(w0, w1, w2, u[29], m[30]); + MONT_MAC(t0, t1, t2, u[30], m[29]); + MONT_MAC(v0, v1, v2, u[31], m[28]); + MONT_MAC(w0, w1, w2, u[32], m[27]); + MONT_MAC(t0, t1, t2, u[33], m[26]); + MONT_MAC(v0, v1, v2, u[34], m[25]); + MONT_MAC(w0, w1, w2, u[35], m[24]); + MONT_MAC(t0, t1, t2, u[36], m[23]); + MONT_MAC(v0, v1, v2, u[37], m[22]); + MONT_MAC(w0, w1, w2, u[38], m[21]); + MONT_MAC(t0, t1, t2, u[39], m[20]); + MONT_MAC(v0, v1, v2, u[40], m[19]); + MONT_MAC(w0, w1, w2, u[41], m[18]); + MONT_MAC(t0, t1, t2, u[42], m[17]); + MONT_MAC(v0, v1, v2, u[43], m[16]); + MONT_MAC(w0, w1, w2, u[44], m[15]); + MONT_MAC(t0, t1, t2, u[45], m[14]); + MONT_MAC(v0, v1, v2, u[46], m[13]); + MONT_MAC(w0, w1, w2, u[47], m[12]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[11] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 60 */ + MONT_MUL(s0, s1, s2, a[13], a[47]); + MONT_MUL(c0, c1, c2, a[14], a[46]); + MONT_MAC(s0, s1, s2, a[15], a[45]); + MONT_MAC(c0, c1, c2, a[16], a[44]); + MONT_MAC(s0, s1, s2, a[17], a[43]); + MONT_MAC(c0, c1, c2, a[18], a[42]); + MONT_MAC(s0, s1, s2, a[19], a[41]); + MONT_MAC(c0, c1, c2, a[20], a[40]); + MONT_MAC(s0, s1, s2, a[21], a[39]); + MONT_MAC(c0, c1, c2, a[22], a[38]); + MONT_MAC(s0, s1, s2, a[23], a[37]); + MONT_MAC(c0, c1, c2, a[24], a[36]); + MONT_MAC(s0, s1, s2, a[25], a[35]); + MONT_MAC(c0, c1, c2, a[26], a[34]); + MONT_MAC(s0, s1, s2, a[27], a[33]); + MONT_MAC(c0, c1, c2, a[28], a[32]); + MONT_MAC(s0, s1, s2, a[29], a[31]); + MONT_MAC(t0, t1, t2, a[30], a[30]); + MONT_MUL(v0, v1, v2, u[13], m[47]); + MONT_MUL(w0, w1, w2, u[14], m[46]); + MONT_MAC(t0, t1, t2, u[15], m[45]); + MONT_MAC(v0, v1, v2, u[16], m[44]); + MONT_MAC(w0, w1, w2, u[17], m[43]); + MONT_MAC(t0, t1, t2, u[18], m[42]); + MONT_MAC(v0, v1, v2, u[19], m[41]); + MONT_MAC(w0, w1, w2, u[20], m[40]); + MONT_MAC(t0, t1, t2, u[21], m[39]); + MONT_MAC(v0, v1, v2, u[22], m[38]); + MONT_MAC(w0, w1, w2, u[23], m[37]); + MONT_MAC(t0, t1, t2, u[24], m[36]); + MONT_MAC(v0, v1, v2, u[25], m[35]); + MONT_MAC(w0, w1, w2, u[26], m[34]); + MONT_MAC(t0, t1, t2, u[27], m[33]); + MONT_MAC(v0, v1, v2, u[28], m[32]); + MONT_MAC(w0, w1, w2, u[29], m[31]); + MONT_MAC(t0, t1, t2, u[30], m[30]); + MONT_MAC(v0, v1, v2, u[31], m[29]); + MONT_MAC(w0, w1, w2, u[32], m[28]); + MONT_MAC(t0, t1, t2, u[33], m[27]); + MONT_MAC(v0, v1, v2, u[34], m[26]); + MONT_MAC(w0, w1, w2, u[35], m[25]); + MONT_MAC(t0, t1, t2, u[36], m[24]); + MONT_MAC(v0, v1, v2, u[37], m[23]); + MONT_MAC(w0, w1, w2, u[38], m[22]); + MONT_MAC(t0, t1, t2, u[39], m[21]); + MONT_MAC(v0, v1, v2, u[40], m[20]); + MONT_MAC(w0, w1, w2, u[41], m[19]); + MONT_MAC(t0, t1, t2, u[42], m[18]); + MONT_MAC(v0, v1, v2, u[43], m[17]); + MONT_MAC(w0, w1, w2, u[44], m[16]); + MONT_MAC(t0, t1, t2, u[45], m[15]); + MONT_MAC(v0, v1, v2, u[46], m[14]); + MONT_MAC(w0, w1, w2, u[47], m[13]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[12] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 61 */ + MONT_MUL(s0, s1, s2, a[14], a[47]); + MONT_MUL(c0, c1, c2, a[15], a[46]); + MONT_MAC(s0, s1, s2, a[16], a[45]); + MONT_MAC(c0, c1, c2, a[17], a[44]); + MONT_MAC(s0, s1, s2, a[18], a[43]); + MONT_MAC(c0, c1, c2, a[19], a[42]); + MONT_MAC(s0, s1, s2, a[20], a[41]); + MONT_MAC(c0, c1, c2, a[21], a[40]); + MONT_MAC(s0, s1, s2, a[22], a[39]); + MONT_MAC(c0, c1, c2, a[23], a[38]); + MONT_MAC(s0, s1, s2, a[24], a[37]); + MONT_MAC(c0, c1, c2, a[25], a[36]); + MONT_MAC(s0, s1, s2, a[26], a[35]); + MONT_MAC(c0, c1, c2, a[27], a[34]); + MONT_MAC(s0, s1, s2, a[28], a[33]); + MONT_MAC(c0, c1, c2, a[29], a[32]); + MONT_MAC(s0, s1, s2, a[30], a[31]); + MONT_MAC(t0, t1, t2, u[14], m[47]); + MONT_MUL(v0, v1, v2, u[15], m[46]); + MONT_MUL(w0, w1, w2, u[16], m[45]); + MONT_MAC(t0, t1, t2, u[17], m[44]); + MONT_MAC(v0, v1, v2, u[18], m[43]); + MONT_MAC(w0, w1, w2, u[19], m[42]); + MONT_MAC(t0, t1, t2, u[20], m[41]); + MONT_MAC(v0, v1, v2, u[21], m[40]); + MONT_MAC(w0, w1, w2, u[22], m[39]); + MONT_MAC(t0, t1, t2, u[23], m[38]); + MONT_MAC(v0, v1, v2, u[24], m[37]); + MONT_MAC(w0, w1, w2, u[25], m[36]); + MONT_MAC(t0, t1, t2, u[26], m[35]); + MONT_MAC(v0, v1, v2, u[27], m[34]); + MONT_MAC(w0, w1, w2, u[28], m[33]); + MONT_MAC(t0, t1, t2, u[29], m[32]); + MONT_MAC(v0, v1, v2, u[30], m[31]); + MONT_MAC(w0, w1, w2, u[31], m[30]); + MONT_MAC(t0, t1, t2, u[32], m[29]); + MONT_MAC(v0, v1, v2, u[33], m[28]); + MONT_MAC(w0, w1, w2, u[34], m[27]); + MONT_MAC(t0, t1, t2, u[35], m[26]); + MONT_MAC(v0, v1, v2, u[36], m[25]); + MONT_MAC(w0, w1, w2, u[37], m[24]); + MONT_MAC(t0, t1, t2, u[38], m[23]); + MONT_MAC(v0, v1, v2, u[39], m[22]); + MONT_MAC(w0, w1, w2, u[40], m[21]); + MONT_MAC(t0, t1, t2, u[41], m[20]); + MONT_MAC(v0, v1, v2, u[42], m[19]); + MONT_MAC(w0, w1, w2, u[43], m[18]); + MONT_MAC(t0, t1, t2, u[44], m[17]); + MONT_MAC(v0, v1, v2, u[45], m[16]); + MONT_MAC(w0, w1, w2, u[46], m[15]); + MONT_MAC(t0, t1, t2, u[47], m[14]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[13] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 62 */ + MONT_MUL(s0, s1, s2, a[15], a[47]); + MONT_MUL(c0, c1, c2, a[16], a[46]); + MONT_MAC(s0, s1, s2, a[17], a[45]); + MONT_MAC(c0, c1, c2, a[18], a[44]); + MONT_MAC(s0, s1, s2, a[19], a[43]); + MONT_MAC(c0, c1, c2, a[20], a[42]); + MONT_MAC(s0, s1, s2, a[21], a[41]); + MONT_MAC(c0, c1, c2, a[22], a[40]); + MONT_MAC(s0, s1, s2, a[23], a[39]); + MONT_MAC(c0, c1, c2, a[24], a[38]); + MONT_MAC(s0, s1, s2, a[25], a[37]); + MONT_MAC(c0, c1, c2, a[26], a[36]); + MONT_MAC(s0, s1, s2, a[27], a[35]); + MONT_MAC(c0, c1, c2, a[28], a[34]); + MONT_MAC(s0, s1, s2, a[29], a[33]); + MONT_MAC(c0, c1, c2, a[30], a[32]); + MONT_MAC(t0, t1, t2, a[31], a[31]); + MONT_MUL(v0, v1, v2, u[15], m[47]); + MONT_MUL(w0, w1, w2, u[16], m[46]); + MONT_MAC(t0, t1, t2, u[17], m[45]); + MONT_MAC(v0, v1, v2, u[18], m[44]); + MONT_MAC(w0, w1, w2, u[19], m[43]); + MONT_MAC(t0, t1, t2, u[20], m[42]); + MONT_MAC(v0, v1, v2, u[21], m[41]); + MONT_MAC(w0, w1, w2, u[22], m[40]); + MONT_MAC(t0, t1, t2, u[23], m[39]); + MONT_MAC(v0, v1, v2, u[24], m[38]); + MONT_MAC(w0, w1, w2, u[25], m[37]); + MONT_MAC(t0, t1, t2, u[26], m[36]); + MONT_MAC(v0, v1, v2, u[27], m[35]); + MONT_MAC(w0, w1, w2, u[28], m[34]); + MONT_MAC(t0, t1, t2, u[29], m[33]); + MONT_MAC(v0, v1, v2, u[30], m[32]); + MONT_MAC(w0, w1, w2, u[31], m[31]); + MONT_MAC(t0, t1, t2, u[32], m[30]); + MONT_MAC(v0, v1, v2, u[33], m[29]); + MONT_MAC(w0, w1, w2, u[34], m[28]); + MONT_MAC(t0, t1, t2, u[35], m[27]); + MONT_MAC(v0, v1, v2, u[36], m[26]); + MONT_MAC(w0, w1, w2, u[37], m[25]); + MONT_MAC(t0, t1, t2, u[38], m[24]); + MONT_MAC(v0, v1, v2, u[39], m[23]); + MONT_MAC(w0, w1, w2, u[40], m[22]); + MONT_MAC(t0, t1, t2, u[41], m[21]); + MONT_MAC(v0, v1, v2, u[42], m[20]); + MONT_MAC(w0, w1, w2, u[43], m[19]); + MONT_MAC(t0, t1, t2, u[44], m[18]); + MONT_MAC(v0, v1, v2, u[45], m[17]); + MONT_MAC(w0, w1, w2, u[46], m[16]); + MONT_MAC(t0, t1, t2, u[47], m[15]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[14] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 63 */ + MONT_MUL(s0, s1, s2, a[16], a[47]); + MONT_MUL(c0, c1, c2, a[17], a[46]); + MONT_MAC(s0, s1, s2, a[18], a[45]); + MONT_MAC(c0, c1, c2, a[19], a[44]); + MONT_MAC(s0, s1, s2, a[20], a[43]); + MONT_MAC(c0, c1, c2, a[21], a[42]); + MONT_MAC(s0, s1, s2, a[22], a[41]); + MONT_MAC(c0, c1, c2, a[23], a[40]); + MONT_MAC(s0, s1, s2, a[24], a[39]); + MONT_MAC(c0, c1, c2, a[25], a[38]); + MONT_MAC(s0, s1, s2, a[26], a[37]); + MONT_MAC(c0, c1, c2, a[27], a[36]); + MONT_MAC(s0, s1, s2, a[28], a[35]); + MONT_MAC(c0, c1, c2, a[29], a[34]); + MONT_MAC(s0, s1, s2, a[30], a[33]); + MONT_MAC(c0, c1, c2, a[31], a[32]); + MONT_MAC(t0, t1, t2, u[16], m[47]); + MONT_MUL(v0, v1, v2, u[17], m[46]); + MONT_MUL(w0, w1, w2, u[18], m[45]); + MONT_MAC(t0, t1, t2, u[19], m[44]); + MONT_MAC(v0, v1, v2, u[20], m[43]); + MONT_MAC(w0, w1, w2, u[21], m[42]); + MONT_MAC(t0, t1, t2, u[22], m[41]); + MONT_MAC(v0, v1, v2, u[23], m[40]); + MONT_MAC(w0, w1, w2, u[24], m[39]); + MONT_MAC(t0, t1, t2, u[25], m[38]); + MONT_MAC(v0, v1, v2, u[26], m[37]); + MONT_MAC(w0, w1, w2, u[27], m[36]); + MONT_MAC(t0, t1, t2, u[28], m[35]); + MONT_MAC(v0, v1, v2, u[29], m[34]); + MONT_MAC(w0, w1, w2, u[30], m[33]); + MONT_MAC(t0, t1, t2, u[31], m[32]); + MONT_MAC(v0, v1, v2, u[32], m[31]); + MONT_MAC(w0, w1, w2, u[33], m[30]); + MONT_MAC(t0, t1, t2, u[34], m[29]); + MONT_MAC(v0, v1, v2, u[35], m[28]); + MONT_MAC(w0, w1, w2, u[36], m[27]); + MONT_MAC(t0, t1, t2, u[37], m[26]); + MONT_MAC(v0, v1, v2, u[38], m[25]); + MONT_MAC(w0, w1, w2, u[39], m[24]); + MONT_MAC(t0, t1, t2, u[40], m[23]); + MONT_MAC(v0, v1, v2, u[41], m[22]); + MONT_MAC(w0, w1, w2, u[42], m[21]); + MONT_MAC(t0, t1, t2, u[43], m[20]); + MONT_MAC(v0, v1, v2, u[44], m[19]); + MONT_MAC(w0, w1, w2, u[45], m[18]); + MONT_MAC(t0, t1, t2, u[46], m[17]); + MONT_MAC(v0, v1, v2, u[47], m[16]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[15] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 64 */ + MONT_MUL(s0, s1, s2, a[17], a[47]); + MONT_MUL(c0, c1, c2, a[18], a[46]); + MONT_MAC(s0, s1, s2, a[19], a[45]); + MONT_MAC(c0, c1, c2, a[20], a[44]); + MONT_MAC(s0, s1, s2, a[21], a[43]); + MONT_MAC(c0, c1, c2, a[22], a[42]); + MONT_MAC(s0, s1, s2, a[23], a[41]); + MONT_MAC(c0, c1, c2, a[24], a[40]); + MONT_MAC(s0, s1, s2, a[25], a[39]); + MONT_MAC(c0, c1, c2, a[26], a[38]); + MONT_MAC(s0, s1, s2, a[27], a[37]); + MONT_MAC(c0, c1, c2, a[28], a[36]); + MONT_MAC(s0, s1, s2, a[29], a[35]); + MONT_MAC(c0, c1, c2, a[30], a[34]); + MONT_MAC(s0, s1, s2, a[31], a[33]); + MONT_MAC(t0, t1, t2, a[32], a[32]); + MONT_MUL(v0, v1, v2, u[17], m[47]); + MONT_MUL(w0, w1, w2, u[18], m[46]); + MONT_MAC(t0, t1, t2, u[19], m[45]); + MONT_MAC(v0, v1, v2, u[20], m[44]); + MONT_MAC(w0, w1, w2, u[21], m[43]); + MONT_MAC(t0, t1, t2, u[22], m[42]); + MONT_MAC(v0, v1, v2, u[23], m[41]); + MONT_MAC(w0, w1, w2, u[24], m[40]); + MONT_MAC(t0, t1, t2, u[25], m[39]); + MONT_MAC(v0, v1, v2, u[26], m[38]); + MONT_MAC(w0, w1, w2, u[27], m[37]); + MONT_MAC(t0, t1, t2, u[28], m[36]); + MONT_MAC(v0, v1, v2, u[29], m[35]); + MONT_MAC(w0, w1, w2, u[30], m[34]); + MONT_MAC(t0, t1, t2, u[31], m[33]); + MONT_MAC(v0, v1, v2, u[32], m[32]); + MONT_MAC(w0, w1, w2, u[33], m[31]); + MONT_MAC(t0, t1, t2, u[34], m[30]); + MONT_MAC(v0, v1, v2, u[35], m[29]); + MONT_MAC(w0, w1, w2, u[36], m[28]); + MONT_MAC(t0, t1, t2, u[37], m[27]); + MONT_MAC(v0, v1, v2, u[38], m[26]); + MONT_MAC(w0, w1, w2, u[39], m[25]); + MONT_MAC(t0, t1, t2, u[40], m[24]); + MONT_MAC(v0, v1, v2, u[41], m[23]); + MONT_MAC(w0, w1, w2, u[42], m[22]); + MONT_MAC(t0, t1, t2, u[43], m[21]); + MONT_MAC(v0, v1, v2, u[44], m[20]); + MONT_MAC(w0, w1, w2, u[45], m[19]); + MONT_MAC(t0, t1, t2, u[46], m[18]); + MONT_MAC(v0, v1, v2, u[47], m[17]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[16] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 65 */ + MONT_MUL(s0, s1, s2, a[18], a[47]); + MONT_MUL(c0, c1, c2, a[19], a[46]); + MONT_MAC(s0, s1, s2, a[20], a[45]); + MONT_MAC(c0, c1, c2, a[21], a[44]); + MONT_MAC(s0, s1, s2, a[22], a[43]); + MONT_MAC(c0, c1, c2, a[23], a[42]); + MONT_MAC(s0, s1, s2, a[24], a[41]); + MONT_MAC(c0, c1, c2, a[25], a[40]); + MONT_MAC(s0, s1, s2, a[26], a[39]); + MONT_MAC(c0, c1, c2, a[27], a[38]); + MONT_MAC(s0, s1, s2, a[28], a[37]); + MONT_MAC(c0, c1, c2, a[29], a[36]); + MONT_MAC(s0, s1, s2, a[30], a[35]); + MONT_MAC(c0, c1, c2, a[31], a[34]); + MONT_MAC(s0, s1, s2, a[32], a[33]); + MONT_MAC(t0, t1, t2, u[18], m[47]); + MONT_MUL(v0, v1, v2, u[19], m[46]); + MONT_MUL(w0, w1, w2, u[20], m[45]); + MONT_MAC(t0, t1, t2, u[21], m[44]); + MONT_MAC(v0, v1, v2, u[22], m[43]); + MONT_MAC(w0, w1, w2, u[23], m[42]); + MONT_MAC(t0, t1, t2, u[24], m[41]); + MONT_MAC(v0, v1, v2, u[25], m[40]); + MONT_MAC(w0, w1, w2, u[26], m[39]); + MONT_MAC(t0, t1, t2, u[27], m[38]); + MONT_MAC(v0, v1, v2, u[28], m[37]); + MONT_MAC(w0, w1, w2, u[29], m[36]); + MONT_MAC(t0, t1, t2, u[30], m[35]); + MONT_MAC(v0, v1, v2, u[31], m[34]); + MONT_MAC(w0, w1, w2, u[32], m[33]); + MONT_MAC(t0, t1, t2, u[33], m[32]); + MONT_MAC(v0, v1, v2, u[34], m[31]); + MONT_MAC(w0, w1, w2, u[35], m[30]); + MONT_MAC(t0, t1, t2, u[36], m[29]); + MONT_MAC(v0, v1, v2, u[37], m[28]); + MONT_MAC(w0, w1, w2, u[38], m[27]); + MONT_MAC(t0, t1, t2, u[39], m[26]); + MONT_MAC(v0, v1, v2, u[40], m[25]); + MONT_MAC(w0, w1, w2, u[41], m[24]); + MONT_MAC(t0, t1, t2, u[42], m[23]); + MONT_MAC(v0, v1, v2, u[43], m[22]); + MONT_MAC(w0, w1, w2, u[44], m[21]); + MONT_MAC(t0, t1, t2, u[45], m[20]); + MONT_MAC(v0, v1, v2, u[46], m[19]); + MONT_MAC(w0, w1, w2, u[47], m[18]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[17] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 66 */ + MONT_MUL(s0, s1, s2, a[19], a[47]); + MONT_MUL(c0, c1, c2, a[20], a[46]); + MONT_MAC(s0, s1, s2, a[21], a[45]); + MONT_MAC(c0, c1, c2, a[22], a[44]); + MONT_MAC(s0, s1, s2, a[23], a[43]); + MONT_MAC(c0, c1, c2, a[24], a[42]); + MONT_MAC(s0, s1, s2, a[25], a[41]); + MONT_MAC(c0, c1, c2, a[26], a[40]); + MONT_MAC(s0, s1, s2, a[27], a[39]); + MONT_MAC(c0, c1, c2, a[28], a[38]); + MONT_MAC(s0, s1, s2, a[29], a[37]); + MONT_MAC(c0, c1, c2, a[30], a[36]); + MONT_MAC(s0, s1, s2, a[31], a[35]); + MONT_MAC(c0, c1, c2, a[32], a[34]); + MONT_MAC(t0, t1, t2, a[33], a[33]); + MONT_MUL(v0, v1, v2, u[19], m[47]); + MONT_MUL(w0, w1, w2, u[20], m[46]); + MONT_MAC(t0, t1, t2, u[21], m[45]); + MONT_MAC(v0, v1, v2, u[22], m[44]); + MONT_MAC(w0, w1, w2, u[23], m[43]); + MONT_MAC(t0, t1, t2, u[24], m[42]); + MONT_MAC(v0, v1, v2, u[25], m[41]); + MONT_MAC(w0, w1, w2, u[26], m[40]); + MONT_MAC(t0, t1, t2, u[27], m[39]); + MONT_MAC(v0, v1, v2, u[28], m[38]); + MONT_MAC(w0, w1, w2, u[29], m[37]); + MONT_MAC(t0, t1, t2, u[30], m[36]); + MONT_MAC(v0, v1, v2, u[31], m[35]); + MONT_MAC(w0, w1, w2, u[32], m[34]); + MONT_MAC(t0, t1, t2, u[33], m[33]); + MONT_MAC(v0, v1, v2, u[34], m[32]); + MONT_MAC(w0, w1, w2, u[35], m[31]); + MONT_MAC(t0, t1, t2, u[36], m[30]); + MONT_MAC(v0, v1, v2, u[37], m[29]); + MONT_MAC(w0, w1, w2, u[38], m[28]); + MONT_MAC(t0, t1, t2, u[39], m[27]); + MONT_MAC(v0, v1, v2, u[40], m[26]); + MONT_MAC(w0, w1, w2, u[41], m[25]); + MONT_MAC(t0, t1, t2, u[42], m[24]); + MONT_MAC(v0, v1, v2, u[43], m[23]); + MONT_MAC(w0, w1, w2, u[44], m[22]); + MONT_MAC(t0, t1, t2, u[45], m[21]); + MONT_MAC(v0, v1, v2, u[46], m[20]); + MONT_MAC(w0, w1, w2, u[47], m[19]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[18] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 67 */ + MONT_MUL(s0, s1, s2, a[20], a[47]); + MONT_MUL(c0, c1, c2, a[21], a[46]); + MONT_MAC(s0, s1, s2, a[22], a[45]); + MONT_MAC(c0, c1, c2, a[23], a[44]); + MONT_MAC(s0, s1, s2, a[24], a[43]); + MONT_MAC(c0, c1, c2, a[25], a[42]); + MONT_MAC(s0, s1, s2, a[26], a[41]); + MONT_MAC(c0, c1, c2, a[27], a[40]); + MONT_MAC(s0, s1, s2, a[28], a[39]); + MONT_MAC(c0, c1, c2, a[29], a[38]); + MONT_MAC(s0, s1, s2, a[30], a[37]); + MONT_MAC(c0, c1, c2, a[31], a[36]); + MONT_MAC(s0, s1, s2, a[32], a[35]); + MONT_MAC(c0, c1, c2, a[33], a[34]); + MONT_MAC(t0, t1, t2, u[20], m[47]); + MONT_MUL(v0, v1, v2, u[21], m[46]); + MONT_MUL(w0, w1, w2, u[22], m[45]); + MONT_MAC(t0, t1, t2, u[23], m[44]); + MONT_MAC(v0, v1, v2, u[24], m[43]); + MONT_MAC(w0, w1, w2, u[25], m[42]); + MONT_MAC(t0, t1, t2, u[26], m[41]); + MONT_MAC(v0, v1, v2, u[27], m[40]); + MONT_MAC(w0, w1, w2, u[28], m[39]); + MONT_MAC(t0, t1, t2, u[29], m[38]); + MONT_MAC(v0, v1, v2, u[30], m[37]); + MONT_MAC(w0, w1, w2, u[31], m[36]); + MONT_MAC(t0, t1, t2, u[32], m[35]); + MONT_MAC(v0, v1, v2, u[33], m[34]); + MONT_MAC(w0, w1, w2, u[34], m[33]); + MONT_MAC(t0, t1, t2, u[35], m[32]); + MONT_MAC(v0, v1, v2, u[36], m[31]); + MONT_MAC(w0, w1, w2, u[37], m[30]); + MONT_MAC(t0, t1, t2, u[38], m[29]); + MONT_MAC(v0, v1, v2, u[39], m[28]); + MONT_MAC(w0, w1, w2, u[40], m[27]); + MONT_MAC(t0, t1, t2, u[41], m[26]); + MONT_MAC(v0, v1, v2, u[42], m[25]); + MONT_MAC(w0, w1, w2, u[43], m[24]); + MONT_MAC(t0, t1, t2, u[44], m[23]); + MONT_MAC(v0, v1, v2, u[45], m[22]); + MONT_MAC(w0, w1, w2, u[46], m[21]); + MONT_MAC(t0, t1, t2, u[47], m[20]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[19] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 68 */ + MONT_MUL(s0, s1, s2, a[21], a[47]); + MONT_MUL(c0, c1, c2, a[22], a[46]); + MONT_MAC(s0, s1, s2, a[23], a[45]); + MONT_MAC(c0, c1, c2, a[24], a[44]); + MONT_MAC(s0, s1, s2, a[25], a[43]); + MONT_MAC(c0, c1, c2, a[26], a[42]); + MONT_MAC(s0, s1, s2, a[27], a[41]); + MONT_MAC(c0, c1, c2, a[28], a[40]); + MONT_MAC(s0, s1, s2, a[29], a[39]); + MONT_MAC(c0, c1, c2, a[30], a[38]); + MONT_MAC(s0, s1, s2, a[31], a[37]); + MONT_MAC(c0, c1, c2, a[32], a[36]); + MONT_MAC(s0, s1, s2, a[33], a[35]); + MONT_MAC(t0, t1, t2, a[34], a[34]); + MONT_MUL(v0, v1, v2, u[21], m[47]); + MONT_MUL(w0, w1, w2, u[22], m[46]); + MONT_MAC(t0, t1, t2, u[23], m[45]); + MONT_MAC(v0, v1, v2, u[24], m[44]); + MONT_MAC(w0, w1, w2, u[25], m[43]); + MONT_MAC(t0, t1, t2, u[26], m[42]); + MONT_MAC(v0, v1, v2, u[27], m[41]); + MONT_MAC(w0, w1, w2, u[28], m[40]); + MONT_MAC(t0, t1, t2, u[29], m[39]); + MONT_MAC(v0, v1, v2, u[30], m[38]); + MONT_MAC(w0, w1, w2, u[31], m[37]); + MONT_MAC(t0, t1, t2, u[32], m[36]); + MONT_MAC(v0, v1, v2, u[33], m[35]); + MONT_MAC(w0, w1, w2, u[34], m[34]); + MONT_MAC(t0, t1, t2, u[35], m[33]); + MONT_MAC(v0, v1, v2, u[36], m[32]); + MONT_MAC(w0, w1, w2, u[37], m[31]); + MONT_MAC(t0, t1, t2, u[38], m[30]); + MONT_MAC(v0, v1, v2, u[39], m[29]); + MONT_MAC(w0, w1, w2, u[40], m[28]); + MONT_MAC(t0, t1, t2, u[41], m[27]); + MONT_MAC(v0, v1, v2, u[42], m[26]); + MONT_MAC(w0, w1, w2, u[43], m[25]); + MONT_MAC(t0, t1, t2, u[44], m[24]); + MONT_MAC(v0, v1, v2, u[45], m[23]); + MONT_MAC(w0, w1, w2, u[46], m[22]); + MONT_MAC(t0, t1, t2, u[47], m[21]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[20] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 69 */ + MONT_MUL(s0, s1, s2, a[22], a[47]); + MONT_MUL(c0, c1, c2, a[23], a[46]); + MONT_MAC(s0, s1, s2, a[24], a[45]); + MONT_MAC(c0, c1, c2, a[25], a[44]); + MONT_MAC(s0, s1, s2, a[26], a[43]); + MONT_MAC(c0, c1, c2, a[27], a[42]); + MONT_MAC(s0, s1, s2, a[28], a[41]); + MONT_MAC(c0, c1, c2, a[29], a[40]); + MONT_MAC(s0, s1, s2, a[30], a[39]); + MONT_MAC(c0, c1, c2, a[31], a[38]); + MONT_MAC(s0, s1, s2, a[32], a[37]); + MONT_MAC(c0, c1, c2, a[33], a[36]); + MONT_MAC(s0, s1, s2, a[34], a[35]); + MONT_MAC(t0, t1, t2, u[22], m[47]); + MONT_MUL(v0, v1, v2, u[23], m[46]); + MONT_MUL(w0, w1, w2, u[24], m[45]); + MONT_MAC(t0, t1, t2, u[25], m[44]); + MONT_MAC(v0, v1, v2, u[26], m[43]); + MONT_MAC(w0, w1, w2, u[27], m[42]); + MONT_MAC(t0, t1, t2, u[28], m[41]); + MONT_MAC(v0, v1, v2, u[29], m[40]); + MONT_MAC(w0, w1, w2, u[30], m[39]); + MONT_MAC(t0, t1, t2, u[31], m[38]); + MONT_MAC(v0, v1, v2, u[32], m[37]); + MONT_MAC(w0, w1, w2, u[33], m[36]); + MONT_MAC(t0, t1, t2, u[34], m[35]); + MONT_MAC(v0, v1, v2, u[35], m[34]); + MONT_MAC(w0, w1, w2, u[36], m[33]); + MONT_MAC(t0, t1, t2, u[37], m[32]); + MONT_MAC(v0, v1, v2, u[38], m[31]); + MONT_MAC(w0, w1, w2, u[39], m[30]); + MONT_MAC(t0, t1, t2, u[40], m[29]); + MONT_MAC(v0, v1, v2, u[41], m[28]); + MONT_MAC(w0, w1, w2, u[42], m[27]); + MONT_MAC(t0, t1, t2, u[43], m[26]); + MONT_MAC(v0, v1, v2, u[44], m[25]); + MONT_MAC(w0, w1, w2, u[45], m[24]); + MONT_MAC(t0, t1, t2, u[46], m[23]); + MONT_MAC(v0, v1, v2, u[47], m[22]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[21] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 70 */ + MONT_MUL(s0, s1, s2, a[23], a[47]); + MONT_MUL(c0, c1, c2, a[24], a[46]); + MONT_MAC(s0, s1, s2, a[25], a[45]); + MONT_MAC(c0, c1, c2, a[26], a[44]); + MONT_MAC(s0, s1, s2, a[27], a[43]); + MONT_MAC(c0, c1, c2, a[28], a[42]); + MONT_MAC(s0, s1, s2, a[29], a[41]); + MONT_MAC(c0, c1, c2, a[30], a[40]); + MONT_MAC(s0, s1, s2, a[31], a[39]); + MONT_MAC(c0, c1, c2, a[32], a[38]); + MONT_MAC(s0, s1, s2, a[33], a[37]); + MONT_MAC(c0, c1, c2, a[34], a[36]); + MONT_MAC(t0, t1, t2, a[35], a[35]); + MONT_MUL(v0, v1, v2, u[23], m[47]); + MONT_MUL(w0, w1, w2, u[24], m[46]); + MONT_MAC(t0, t1, t2, u[25], m[45]); + MONT_MAC(v0, v1, v2, u[26], m[44]); + MONT_MAC(w0, w1, w2, u[27], m[43]); + MONT_MAC(t0, t1, t2, u[28], m[42]); + MONT_MAC(v0, v1, v2, u[29], m[41]); + MONT_MAC(w0, w1, w2, u[30], m[40]); + MONT_MAC(t0, t1, t2, u[31], m[39]); + MONT_MAC(v0, v1, v2, u[32], m[38]); + MONT_MAC(w0, w1, w2, u[33], m[37]); + MONT_MAC(t0, t1, t2, u[34], m[36]); + MONT_MAC(v0, v1, v2, u[35], m[35]); + MONT_MAC(w0, w1, w2, u[36], m[34]); + MONT_MAC(t0, t1, t2, u[37], m[33]); + MONT_MAC(v0, v1, v2, u[38], m[32]); + MONT_MAC(w0, w1, w2, u[39], m[31]); + MONT_MAC(t0, t1, t2, u[40], m[30]); + MONT_MAC(v0, v1, v2, u[41], m[29]); + MONT_MAC(w0, w1, w2, u[42], m[28]); + MONT_MAC(t0, t1, t2, u[43], m[27]); + MONT_MAC(v0, v1, v2, u[44], m[26]); + MONT_MAC(w0, w1, w2, u[45], m[25]); + MONT_MAC(t0, t1, t2, u[46], m[24]); + MONT_MAC(v0, v1, v2, u[47], m[23]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[22] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 71 */ + MONT_MUL(s0, s1, s2, a[24], a[47]); + MONT_MUL(c0, c1, c2, a[25], a[46]); + MONT_MAC(s0, s1, s2, a[26], a[45]); + MONT_MAC(c0, c1, c2, a[27], a[44]); + MONT_MAC(s0, s1, s2, a[28], a[43]); + MONT_MAC(c0, c1, c2, a[29], a[42]); + MONT_MAC(s0, s1, s2, a[30], a[41]); + MONT_MAC(c0, c1, c2, a[31], a[40]); + MONT_MAC(s0, s1, s2, a[32], a[39]); + MONT_MAC(c0, c1, c2, a[33], a[38]); + MONT_MAC(s0, s1, s2, a[34], a[37]); + MONT_MAC(c0, c1, c2, a[35], a[36]); + MONT_MAC(t0, t1, t2, u[24], m[47]); + MONT_MUL(v0, v1, v2, u[25], m[46]); + MONT_MUL(w0, w1, w2, u[26], m[45]); + MONT_MAC(t0, t1, t2, u[27], m[44]); + MONT_MAC(v0, v1, v2, u[28], m[43]); + MONT_MAC(w0, w1, w2, u[29], m[42]); + MONT_MAC(t0, t1, t2, u[30], m[41]); + MONT_MAC(v0, v1, v2, u[31], m[40]); + MONT_MAC(w0, w1, w2, u[32], m[39]); + MONT_MAC(t0, t1, t2, u[33], m[38]); + MONT_MAC(v0, v1, v2, u[34], m[37]); + MONT_MAC(w0, w1, w2, u[35], m[36]); + MONT_MAC(t0, t1, t2, u[36], m[35]); + MONT_MAC(v0, v1, v2, u[37], m[34]); + MONT_MAC(w0, w1, w2, u[38], m[33]); + MONT_MAC(t0, t1, t2, u[39], m[32]); + MONT_MAC(v0, v1, v2, u[40], m[31]); + MONT_MAC(w0, w1, w2, u[41], m[30]); + MONT_MAC(t0, t1, t2, u[42], m[29]); + MONT_MAC(v0, v1, v2, u[43], m[28]); + MONT_MAC(w0, w1, w2, u[44], m[27]); + MONT_MAC(t0, t1, t2, u[45], m[26]); + MONT_MAC(v0, v1, v2, u[46], m[25]); + MONT_MAC(w0, w1, w2, u[47], m[24]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[23] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 72 */ + MONT_MUL(s0, s1, s2, a[25], a[47]); + MONT_MUL(c0, c1, c2, a[26], a[46]); + MONT_MAC(s0, s1, s2, a[27], a[45]); + MONT_MAC(c0, c1, c2, a[28], a[44]); + MONT_MAC(s0, s1, s2, a[29], a[43]); + MONT_MAC(c0, c1, c2, a[30], a[42]); + MONT_MAC(s0, s1, s2, a[31], a[41]); + MONT_MAC(c0, c1, c2, a[32], a[40]); + MONT_MAC(s0, s1, s2, a[33], a[39]); + MONT_MAC(c0, c1, c2, a[34], a[38]); + MONT_MAC(s0, s1, s2, a[35], a[37]); + MONT_MAC(t0, t1, t2, a[36], a[36]); + MONT_MUL(v0, v1, v2, u[25], m[47]); + MONT_MUL(w0, w1, w2, u[26], m[46]); + MONT_MAC(t0, t1, t2, u[27], m[45]); + MONT_MAC(v0, v1, v2, u[28], m[44]); + MONT_MAC(w0, w1, w2, u[29], m[43]); + MONT_MAC(t0, t1, t2, u[30], m[42]); + MONT_MAC(v0, v1, v2, u[31], m[41]); + MONT_MAC(w0, w1, w2, u[32], m[40]); + MONT_MAC(t0, t1, t2, u[33], m[39]); + MONT_MAC(v0, v1, v2, u[34], m[38]); + MONT_MAC(w0, w1, w2, u[35], m[37]); + MONT_MAC(t0, t1, t2, u[36], m[36]); + MONT_MAC(v0, v1, v2, u[37], m[35]); + MONT_MAC(w0, w1, w2, u[38], m[34]); + MONT_MAC(t0, t1, t2, u[39], m[33]); + MONT_MAC(v0, v1, v2, u[40], m[32]); + MONT_MAC(w0, w1, w2, u[41], m[31]); + MONT_MAC(t0, t1, t2, u[42], m[30]); + MONT_MAC(v0, v1, v2, u[43], m[29]); + MONT_MAC(w0, w1, w2, u[44], m[28]); + MONT_MAC(t0, t1, t2, u[45], m[27]); + MONT_MAC(v0, v1, v2, u[46], m[26]); + MONT_MAC(w0, w1, w2, u[47], m[25]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[24] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 73 */ + MONT_MUL(s0, s1, s2, a[26], a[47]); + MONT_MUL(c0, c1, c2, a[27], a[46]); + MONT_MAC(s0, s1, s2, a[28], a[45]); + MONT_MAC(c0, c1, c2, a[29], a[44]); + MONT_MAC(s0, s1, s2, a[30], a[43]); + MONT_MAC(c0, c1, c2, a[31], a[42]); + MONT_MAC(s0, s1, s2, a[32], a[41]); + MONT_MAC(c0, c1, c2, a[33], a[40]); + MONT_MAC(s0, s1, s2, a[34], a[39]); + MONT_MAC(c0, c1, c2, a[35], a[38]); + MONT_MAC(s0, s1, s2, a[36], a[37]); + MONT_MAC(t0, t1, t2, u[26], m[47]); + MONT_MUL(v0, v1, v2, u[27], m[46]); + MONT_MUL(w0, w1, w2, u[28], m[45]); + MONT_MAC(t0, t1, t2, u[29], m[44]); + MONT_MAC(v0, v1, v2, u[30], m[43]); + MONT_MAC(w0, w1, w2, u[31], m[42]); + MONT_MAC(t0, t1, t2, u[32], m[41]); + MONT_MAC(v0, v1, v2, u[33], m[40]); + MONT_MAC(w0, w1, w2, u[34], m[39]); + MONT_MAC(t0, t1, t2, u[35], m[38]); + MONT_MAC(v0, v1, v2, u[36], m[37]); + MONT_MAC(w0, w1, w2, u[37], m[36]); + MONT_MAC(t0, t1, t2, u[38], m[35]); + MONT_MAC(v0, v1, v2, u[39], m[34]); + MONT_MAC(w0, w1, w2, u[40], m[33]); + MONT_MAC(t0, t1, t2, u[41], m[32]); + MONT_MAC(v0, v1, v2, u[42], m[31]); + MONT_MAC(w0, w1, w2, u[43], m[30]); + MONT_MAC(t0, t1, t2, u[44], m[29]); + MONT_MAC(v0, v1, v2, u[45], m[28]); + MONT_MAC(w0, w1, w2, u[46], m[27]); + MONT_MAC(t0, t1, t2, u[47], m[26]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[25] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 74 */ + MONT_MUL(s0, s1, s2, a[27], a[47]); + MONT_MUL(c0, c1, c2, a[28], a[46]); + MONT_MAC(s0, s1, s2, a[29], a[45]); + MONT_MAC(c0, c1, c2, a[30], a[44]); + MONT_MAC(s0, s1, s2, a[31], a[43]); + MONT_MAC(c0, c1, c2, a[32], a[42]); + MONT_MAC(s0, s1, s2, a[33], a[41]); + MONT_MAC(c0, c1, c2, a[34], a[40]); + MONT_MAC(s0, s1, s2, a[35], a[39]); + MONT_MAC(c0, c1, c2, a[36], a[38]); + MONT_MAC(t0, t1, t2, a[37], a[37]); + MONT_MUL(v0, v1, v2, u[27], m[47]); + MONT_MUL(w0, w1, w2, u[28], m[46]); + MONT_MAC(t0, t1, t2, u[29], m[45]); + MONT_MAC(v0, v1, v2, u[30], m[44]); + MONT_MAC(w0, w1, w2, u[31], m[43]); + MONT_MAC(t0, t1, t2, u[32], m[42]); + MONT_MAC(v0, v1, v2, u[33], m[41]); + MONT_MAC(w0, w1, w2, u[34], m[40]); + MONT_MAC(t0, t1, t2, u[35], m[39]); + MONT_MAC(v0, v1, v2, u[36], m[38]); + MONT_MAC(w0, w1, w2, u[37], m[37]); + MONT_MAC(t0, t1, t2, u[38], m[36]); + MONT_MAC(v0, v1, v2, u[39], m[35]); + MONT_MAC(w0, w1, w2, u[40], m[34]); + MONT_MAC(t0, t1, t2, u[41], m[33]); + MONT_MAC(v0, v1, v2, u[42], m[32]); + MONT_MAC(w0, w1, w2, u[43], m[31]); + MONT_MAC(t0, t1, t2, u[44], m[30]); + MONT_MAC(v0, v1, v2, u[45], m[29]); + MONT_MAC(w0, w1, w2, u[46], m[28]); + MONT_MAC(t0, t1, t2, u[47], m[27]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[26] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 75 */ + MONT_MUL(s0, s1, s2, a[28], a[47]); + MONT_MUL(c0, c1, c2, a[29], a[46]); + MONT_MAC(s0, s1, s2, a[30], a[45]); + MONT_MAC(c0, c1, c2, a[31], a[44]); + MONT_MAC(s0, s1, s2, a[32], a[43]); + MONT_MAC(c0, c1, c2, a[33], a[42]); + MONT_MAC(s0, s1, s2, a[34], a[41]); + MONT_MAC(c0, c1, c2, a[35], a[40]); + MONT_MAC(s0, s1, s2, a[36], a[39]); + MONT_MAC(c0, c1, c2, a[37], a[38]); + MONT_MAC(t0, t1, t2, u[28], m[47]); + MONT_MUL(v0, v1, v2, u[29], m[46]); + MONT_MUL(w0, w1, w2, u[30], m[45]); + MONT_MAC(t0, t1, t2, u[31], m[44]); + MONT_MAC(v0, v1, v2, u[32], m[43]); + MONT_MAC(w0, w1, w2, u[33], m[42]); + MONT_MAC(t0, t1, t2, u[34], m[41]); + MONT_MAC(v0, v1, v2, u[35], m[40]); + MONT_MAC(w0, w1, w2, u[36], m[39]); + MONT_MAC(t0, t1, t2, u[37], m[38]); + MONT_MAC(v0, v1, v2, u[38], m[37]); + MONT_MAC(w0, w1, w2, u[39], m[36]); + MONT_MAC(t0, t1, t2, u[40], m[35]); + MONT_MAC(v0, v1, v2, u[41], m[34]); + MONT_MAC(w0, w1, w2, u[42], m[33]); + MONT_MAC(t0, t1, t2, u[43], m[32]); + MONT_MAC(v0, v1, v2, u[44], m[31]); + MONT_MAC(w0, w1, w2, u[45], m[30]); + MONT_MAC(t0, t1, t2, u[46], m[29]); + MONT_MAC(v0, v1, v2, u[47], m[28]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[27] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 76 */ + MONT_MUL(s0, s1, s2, a[29], a[47]); + MONT_MUL(c0, c1, c2, a[30], a[46]); + MONT_MAC(s0, s1, s2, a[31], a[45]); + MONT_MAC(c0, c1, c2, a[32], a[44]); + MONT_MAC(s0, s1, s2, a[33], a[43]); + MONT_MAC(c0, c1, c2, a[34], a[42]); + MONT_MAC(s0, s1, s2, a[35], a[41]); + MONT_MAC(c0, c1, c2, a[36], a[40]); + MONT_MAC(s0, s1, s2, a[37], a[39]); + MONT_MAC(t0, t1, t2, a[38], a[38]); + MONT_MUL(v0, v1, v2, u[29], m[47]); + MONT_MUL(w0, w1, w2, u[30], m[46]); + MONT_MAC(t0, t1, t2, u[31], m[45]); + MONT_MAC(v0, v1, v2, u[32], m[44]); + MONT_MAC(w0, w1, w2, u[33], m[43]); + MONT_MAC(t0, t1, t2, u[34], m[42]); + MONT_MAC(v0, v1, v2, u[35], m[41]); + MONT_MAC(w0, w1, w2, u[36], m[40]); + MONT_MAC(t0, t1, t2, u[37], m[39]); + MONT_MAC(v0, v1, v2, u[38], m[38]); + MONT_MAC(w0, w1, w2, u[39], m[37]); + MONT_MAC(t0, t1, t2, u[40], m[36]); + MONT_MAC(v0, v1, v2, u[41], m[35]); + MONT_MAC(w0, w1, w2, u[42], m[34]); + MONT_MAC(t0, t1, t2, u[43], m[33]); + MONT_MAC(v0, v1, v2, u[44], m[32]); + MONT_MAC(w0, w1, w2, u[45], m[31]); + MONT_MAC(t0, t1, t2, u[46], m[30]); + MONT_MAC(v0, v1, v2, u[47], m[29]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[28] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 77 */ + MONT_MUL(s0, s1, s2, a[30], a[47]); + MONT_MUL(c0, c1, c2, a[31], a[46]); + MONT_MAC(s0, s1, s2, a[32], a[45]); + MONT_MAC(c0, c1, c2, a[33], a[44]); + MONT_MAC(s0, s1, s2, a[34], a[43]); + MONT_MAC(c0, c1, c2, a[35], a[42]); + MONT_MAC(s0, s1, s2, a[36], a[41]); + MONT_MAC(c0, c1, c2, a[37], a[40]); + MONT_MAC(s0, s1, s2, a[38], a[39]); + MONT_MAC(t0, t1, t2, u[30], m[47]); + MONT_MUL(v0, v1, v2, u[31], m[46]); + MONT_MUL(w0, w1, w2, u[32], m[45]); + MONT_MAC(t0, t1, t2, u[33], m[44]); + MONT_MAC(v0, v1, v2, u[34], m[43]); + MONT_MAC(w0, w1, w2, u[35], m[42]); + MONT_MAC(t0, t1, t2, u[36], m[41]); + MONT_MAC(v0, v1, v2, u[37], m[40]); + MONT_MAC(w0, w1, w2, u[38], m[39]); + MONT_MAC(t0, t1, t2, u[39], m[38]); + MONT_MAC(v0, v1, v2, u[40], m[37]); + MONT_MAC(w0, w1, w2, u[41], m[36]); + MONT_MAC(t0, t1, t2, u[42], m[35]); + MONT_MAC(v0, v1, v2, u[43], m[34]); + MONT_MAC(w0, w1, w2, u[44], m[33]); + MONT_MAC(t0, t1, t2, u[45], m[32]); + MONT_MAC(v0, v1, v2, u[46], m[31]); + MONT_MAC(w0, w1, w2, u[47], m[30]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[29] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 78 */ + MONT_MUL(s0, s1, s2, a[31], a[47]); + MONT_MUL(c0, c1, c2, a[32], a[46]); + MONT_MAC(s0, s1, s2, a[33], a[45]); + MONT_MAC(c0, c1, c2, a[34], a[44]); + MONT_MAC(s0, s1, s2, a[35], a[43]); + MONT_MAC(c0, c1, c2, a[36], a[42]); + MONT_MAC(s0, s1, s2, a[37], a[41]); + MONT_MAC(c0, c1, c2, a[38], a[40]); + MONT_MAC(t0, t1, t2, a[39], a[39]); + MONT_MUL(v0, v1, v2, u[31], m[47]); + MONT_MUL(w0, w1, w2, u[32], m[46]); + MONT_MAC(t0, t1, t2, u[33], m[45]); + MONT_MAC(v0, v1, v2, u[34], m[44]); + MONT_MAC(w0, w1, w2, u[35], m[43]); + MONT_MAC(t0, t1, t2, u[36], m[42]); + MONT_MAC(v0, v1, v2, u[37], m[41]); + MONT_MAC(w0, w1, w2, u[38], m[40]); + MONT_MAC(t0, t1, t2, u[39], m[39]); + MONT_MAC(v0, v1, v2, u[40], m[38]); + MONT_MAC(w0, w1, w2, u[41], m[37]); + MONT_MAC(t0, t1, t2, u[42], m[36]); + MONT_MAC(v0, v1, v2, u[43], m[35]); + MONT_MAC(w0, w1, w2, u[44], m[34]); + MONT_MAC(t0, t1, t2, u[45], m[33]); + MONT_MAC(v0, v1, v2, u[46], m[32]); + MONT_MAC(w0, w1, w2, u[47], m[31]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[30] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 79 */ + MONT_MUL(s0, s1, s2, a[32], a[47]); + MONT_MUL(c0, c1, c2, a[33], a[46]); + MONT_MAC(s0, s1, s2, a[34], a[45]); + MONT_MAC(c0, c1, c2, a[35], a[44]); + MONT_MAC(s0, s1, s2, a[36], a[43]); + MONT_MAC(c0, c1, c2, a[37], a[42]); + MONT_MAC(s0, s1, s2, a[38], a[41]); + MONT_MAC(c0, c1, c2, a[39], a[40]); + MONT_MAC(t0, t1, t2, u[32], m[47]); + MONT_MUL(v0, v1, v2, u[33], m[46]); + MONT_MUL(w0, w1, w2, u[34], m[45]); + MONT_MAC(t0, t1, t2, u[35], m[44]); + MONT_MAC(v0, v1, v2, u[36], m[43]); + MONT_MAC(w0, w1, w2, u[37], m[42]); + MONT_MAC(t0, t1, t2, u[38], m[41]); + MONT_MAC(v0, v1, v2, u[39], m[40]); + MONT_MAC(w0, w1, w2, u[40], m[39]); + MONT_MAC(t0, t1, t2, u[41], m[38]); + MONT_MAC(v0, v1, v2, u[42], m[37]); + MONT_MAC(w0, w1, w2, u[43], m[36]); + MONT_MAC(t0, t1, t2, u[44], m[35]); + MONT_MAC(v0, v1, v2, u[45], m[34]); + MONT_MAC(w0, w1, w2, u[46], m[33]); + MONT_MAC(t0, t1, t2, u[47], m[32]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[31] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 80 */ + MONT_MUL(s0, s1, s2, a[33], a[47]); + MONT_MUL(c0, c1, c2, a[34], a[46]); + MONT_MAC(s0, s1, s2, a[35], a[45]); + MONT_MAC(c0, c1, c2, a[36], a[44]); + MONT_MAC(s0, s1, s2, a[37], a[43]); + MONT_MAC(c0, c1, c2, a[38], a[42]); + MONT_MAC(s0, s1, s2, a[39], a[41]); + MONT_MAC(t0, t1, t2, a[40], a[40]); + MONT_MUL(v0, v1, v2, u[33], m[47]); + MONT_MUL(w0, w1, w2, u[34], m[46]); + MONT_MAC(t0, t1, t2, u[35], m[45]); + MONT_MAC(v0, v1, v2, u[36], m[44]); + MONT_MAC(w0, w1, w2, u[37], m[43]); + MONT_MAC(t0, t1, t2, u[38], m[42]); + MONT_MAC(v0, v1, v2, u[39], m[41]); + MONT_MAC(w0, w1, w2, u[40], m[40]); + MONT_MAC(t0, t1, t2, u[41], m[39]); + MONT_MAC(v0, v1, v2, u[42], m[38]); + MONT_MAC(w0, w1, w2, u[43], m[37]); + MONT_MAC(t0, t1, t2, u[44], m[36]); + MONT_MAC(v0, v1, v2, u[45], m[35]); + MONT_MAC(w0, w1, w2, u[46], m[34]); + MONT_MAC(t0, t1, t2, u[47], m[33]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[32] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 81 */ + MONT_MUL(s0, s1, s2, a[34], a[47]); + MONT_MUL(c0, c1, c2, a[35], a[46]); + MONT_MAC(s0, s1, s2, a[36], a[45]); + MONT_MAC(c0, c1, c2, a[37], a[44]); + MONT_MAC(s0, s1, s2, a[38], a[43]); + MONT_MAC(c0, c1, c2, a[39], a[42]); + MONT_MAC(s0, s1, s2, a[40], a[41]); + MONT_MAC(t0, t1, t2, u[34], m[47]); + MONT_MUL(v0, v1, v2, u[35], m[46]); + MONT_MUL(w0, w1, w2, u[36], m[45]); + MONT_MAC(t0, t1, t2, u[37], m[44]); + MONT_MAC(v0, v1, v2, u[38], m[43]); + MONT_MAC(w0, w1, w2, u[39], m[42]); + MONT_MAC(t0, t1, t2, u[40], m[41]); + MONT_MAC(v0, v1, v2, u[41], m[40]); + MONT_MAC(w0, w1, w2, u[42], m[39]); + MONT_MAC(t0, t1, t2, u[43], m[38]); + MONT_MAC(v0, v1, v2, u[44], m[37]); + MONT_MAC(w0, w1, w2, u[45], m[36]); + MONT_MAC(t0, t1, t2, u[46], m[35]); + MONT_MAC(v0, v1, v2, u[47], m[34]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[33] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 82 */ + MONT_MUL(s0, s1, s2, a[35], a[47]); + MONT_MUL(c0, c1, c2, a[36], a[46]); + MONT_MAC(s0, s1, s2, a[37], a[45]); + MONT_MAC(c0, c1, c2, a[38], a[44]); + MONT_MAC(s0, s1, s2, a[39], a[43]); + MONT_MAC(c0, c1, c2, a[40], a[42]); + MONT_MAC(t0, t1, t2, a[41], a[41]); + MONT_MUL(v0, v1, v2, u[35], m[47]); + MONT_MUL(w0, w1, w2, u[36], m[46]); + MONT_MAC(t0, t1, t2, u[37], m[45]); + MONT_MAC(v0, v1, v2, u[38], m[44]); + MONT_MAC(w0, w1, w2, u[39], m[43]); + MONT_MAC(t0, t1, t2, u[40], m[42]); + MONT_MAC(v0, v1, v2, u[41], m[41]); + MONT_MAC(w0, w1, w2, u[42], m[40]); + MONT_MAC(t0, t1, t2, u[43], m[39]); + MONT_MAC(v0, v1, v2, u[44], m[38]); + MONT_MAC(w0, w1, w2, u[45], m[37]); + MONT_MAC(t0, t1, t2, u[46], m[36]); + MONT_MAC(v0, v1, v2, u[47], m[35]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[34] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 83 */ + MONT_MUL(s0, s1, s2, a[36], a[47]); + MONT_MUL(c0, c1, c2, a[37], a[46]); + MONT_MAC(s0, s1, s2, a[38], a[45]); + MONT_MAC(c0, c1, c2, a[39], a[44]); + MONT_MAC(s0, s1, s2, a[40], a[43]); + MONT_MAC(c0, c1, c2, a[41], a[42]); + MONT_MAC(t0, t1, t2, u[36], m[47]); + MONT_MUL(v0, v1, v2, u[37], m[46]); + MONT_MUL(w0, w1, w2, u[38], m[45]); + MONT_MAC(t0, t1, t2, u[39], m[44]); + MONT_MAC(v0, v1, v2, u[40], m[43]); + MONT_MAC(w0, w1, w2, u[41], m[42]); + MONT_MAC(t0, t1, t2, u[42], m[41]); + MONT_MAC(v0, v1, v2, u[43], m[40]); + MONT_MAC(w0, w1, w2, u[44], m[39]); + MONT_MAC(t0, t1, t2, u[45], m[38]); + MONT_MAC(v0, v1, v2, u[46], m[37]); + MONT_MAC(w0, w1, w2, u[47], m[36]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[35] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 84 */ + MONT_MUL(s0, s1, s2, a[37], a[47]); + MONT_MUL(c0, c1, c2, a[38], a[46]); + MONT_MAC(s0, s1, s2, a[39], a[45]); + MONT_MAC(c0, c1, c2, a[40], a[44]); + MONT_MAC(s0, s1, s2, a[41], a[43]); + MONT_MAC(t0, t1, t2, a[42], a[42]); + MONT_MUL(v0, v1, v2, u[37], m[47]); + MONT_MUL(w0, w1, w2, u[38], m[46]); + MONT_MAC(t0, t1, t2, u[39], m[45]); + MONT_MAC(v0, v1, v2, u[40], m[44]); + MONT_MAC(w0, w1, w2, u[41], m[43]); + MONT_MAC(t0, t1, t2, u[42], m[42]); + MONT_MAC(v0, v1, v2, u[43], m[41]); + MONT_MAC(w0, w1, w2, u[44], m[40]); + MONT_MAC(t0, t1, t2, u[45], m[39]); + MONT_MAC(v0, v1, v2, u[46], m[38]); + MONT_MAC(w0, w1, w2, u[47], m[37]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[36] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 85 */ + MONT_MUL(s0, s1, s2, a[38], a[47]); + MONT_MUL(c0, c1, c2, a[39], a[46]); + MONT_MAC(s0, s1, s2, a[40], a[45]); + MONT_MAC(c0, c1, c2, a[41], a[44]); + MONT_MAC(s0, s1, s2, a[42], a[43]); + MONT_MAC(t0, t1, t2, u[38], m[47]); + MONT_MUL(v0, v1, v2, u[39], m[46]); + MONT_MUL(w0, w1, w2, u[40], m[45]); + MONT_MAC(t0, t1, t2, u[41], m[44]); + MONT_MAC(v0, v1, v2, u[42], m[43]); + MONT_MAC(w0, w1, w2, u[43], m[42]); + MONT_MAC(t0, t1, t2, u[44], m[41]); + MONT_MAC(v0, v1, v2, u[45], m[40]); + MONT_MAC(w0, w1, w2, u[46], m[39]); + MONT_MAC(t0, t1, t2, u[47], m[38]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[37] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 86 */ + MONT_MUL(s0, s1, s2, a[39], a[47]); + MONT_MUL(c0, c1, c2, a[40], a[46]); + MONT_MAC(s0, s1, s2, a[41], a[45]); + MONT_MAC(c0, c1, c2, a[42], a[44]); + MONT_MAC(t0, t1, t2, a[43], a[43]); + MONT_MUL(v0, v1, v2, u[39], m[47]); + MONT_MUL(w0, w1, w2, u[40], m[46]); + MONT_MAC(t0, t1, t2, u[41], m[45]); + MONT_MAC(v0, v1, v2, u[42], m[44]); + MONT_MAC(w0, w1, w2, u[43], m[43]); + MONT_MAC(t0, t1, t2, u[44], m[42]); + MONT_MAC(v0, v1, v2, u[45], m[41]); + MONT_MAC(w0, w1, w2, u[46], m[40]); + MONT_MAC(t0, t1, t2, u[47], m[39]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[38] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 87 */ + MONT_MUL(s0, s1, s2, a[40], a[47]); + MONT_MUL(c0, c1, c2, a[41], a[46]); + MONT_MAC(s0, s1, s2, a[42], a[45]); + MONT_MAC(c0, c1, c2, a[43], a[44]); + MONT_MAC(t0, t1, t2, u[40], m[47]); + MONT_MUL(v0, v1, v2, u[41], m[46]); + MONT_MUL(w0, w1, w2, u[42], m[45]); + MONT_MAC(t0, t1, t2, u[43], m[44]); + MONT_MAC(v0, v1, v2, u[44], m[43]); + MONT_MAC(w0, w1, w2, u[45], m[42]); + MONT_MAC(t0, t1, t2, u[46], m[41]); + MONT_MAC(v0, v1, v2, u[47], m[40]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[39] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 88 */ + MONT_MUL(s0, s1, s2, a[41], a[47]); + MONT_MUL(c0, c1, c2, a[42], a[46]); + MONT_MAC(s0, s1, s2, a[43], a[45]); + MONT_MAC(t0, t1, t2, a[44], a[44]); + MONT_MUL(v0, v1, v2, u[41], m[47]); + MONT_MUL(w0, w1, w2, u[42], m[46]); + MONT_MAC(t0, t1, t2, u[43], m[45]); + MONT_MAC(v0, v1, v2, u[44], m[44]); + MONT_MAC(w0, w1, w2, u[45], m[43]); + MONT_MAC(t0, t1, t2, u[46], m[42]); + MONT_MAC(v0, v1, v2, u[47], m[41]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[40] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 89 */ + MONT_MUL(s0, s1, s2, a[42], a[47]); + MONT_MUL(c0, c1, c2, a[43], a[46]); + MONT_MAC(s0, s1, s2, a[44], a[45]); + MONT_MAC(t0, t1, t2, u[42], m[47]); + MONT_MUL(v0, v1, v2, u[43], m[46]); + MONT_MUL(w0, w1, w2, u[44], m[45]); + MONT_MAC(t0, t1, t2, u[45], m[44]); + MONT_MAC(v0, v1, v2, u[46], m[43]); + MONT_MAC(w0, w1, w2, u[47], m[42]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[41] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 90 */ + MONT_MUL(s0, s1, s2, a[43], a[47]); + MONT_MUL(c0, c1, c2, a[44], a[46]); + MONT_MAC(t0, t1, t2, a[45], a[45]); + MONT_MUL(v0, v1, v2, u[43], m[47]); + MONT_MUL(w0, w1, w2, u[44], m[46]); + MONT_MAC(t0, t1, t2, u[45], m[45]); + MONT_MAC(v0, v1, v2, u[46], m[44]); + MONT_MAC(w0, w1, w2, u[47], m[43]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[42] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 91 */ + MONT_MUL(s0, s1, s2, a[44], a[47]); + MONT_MUL(c0, c1, c2, a[45], a[46]); + MONT_MAC(t0, t1, t2, u[44], m[47]); + MONT_MUL(v0, v1, v2, u[45], m[46]); + MONT_MUL(w0, w1, w2, u[46], m[45]); + MONT_MAC(t0, t1, t2, u[47], m[44]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_ADD(s0, s1, s2, c0, c1, c2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[43] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 92 */ + MONT_MUL(s0, s1, s2, a[45], a[47]); + MONT_MAC(t0, t1, t2, a[46], a[46]); + MONT_MUL(v0, v1, v2, u[45], m[47]); + MONT_MUL(w0, w1, w2, u[46], m[46]); + MONT_MAC(t0, t1, t2, u[47], m[45]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_ADD(t0, t1, t2, w0, w1, w2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[44] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 93 */ + MONT_MUL(s0, s1, s2, a[46], a[47]); + MONT_MAC(t0, t1, t2, u[46], m[47]); + MONT_MUL(v0, v1, v2, u[47], m[46]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + MONT_DBL(s0, s1, s2); + MONT_ADD(t0, t1, t2, s0, s1, s2); + z[45] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + /* column 94 */ + MONT_MAC(t0, t1, t2, a[47], a[47]); + MONT_MUL(v0, v1, v2, u[47], m[47]); + MONT_ADD(t0, t1, t2, v0, v1, v2); + z[46] = t0; + t0 = t1; + t1 = t2; + t2 = 0; + z[47] = t0; + + /* under 2m, with t1 the bit above n limbs: at most one subtraction */ + borrow = sub_n(r, z, m, 48); + take = t1 | (borrow ^ 1); + select_n(r, r, z, take & 1, 48); +} + +#endif diff --git a/cbits/crypton_mont.py b/cbits/crypton_mont.py new file mode 100644 --- /dev/null +++ b/cbits/crypton_mont.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +"""Generate cbits/crypton_mont.h: Montgomery multiplication and squaring for +AArch64 at the limb counts of RSA's CRT halves, fully unrolled and +product-scanning. + +Usage: python3 cbits/crypton_mont.py > cbits/crypton_mont.h + +The result is a column at a time, with the reduction folded into the same +scan (the FIPS method), so the running sum stays in registers instead of +going through memory a row at a time. A column's products are spread over +three accumulators and added together at its end, since one accumulator's +carry chain is what holds the core back, not its multipliers. In a square, +each product off the diagonal is summed once, doubled, and added in. +""" + +SIZES = (16, 24, 32) +# Squaring at 48 too, RSA-3072's modulus, for the verification that is +# seventeen squarings. Its multiplication, at twice the code, is left out. +SQR_ONLY = (48,) + +HEADER = """\ +/* Generated by cbits/crypton_mont.py; do not edit. + * + * Montgomery multiplication and squaring for AArch64 at 16, 24 and 32 limbs, + * the halves of RSA-2048, -3072 and -4096, and squaring at 48, RSA-3072's + * modulus. Each answers what mont_mul and mont_sqr answer. */ +#ifndef CRYPTON_MONT_H +#define CRYPTON_MONT_H + +#include "crypton_bignum.h" + +/* (t2:t1:t0) += x * y */ +#define MONT_MAC(t0, t1, t2, x, y) \\ + do { \\ + limb_t lo_, hi_; \\ + __asm__("mul\\t%0, %5, %6\\n\\t" \\ + "umulh\\t%1, %5, %6\\n\\t" \\ + "adds\\t%2, %2, %0\\n\\t" \\ + "adcs\\t%3, %3, %1\\n\\t" \\ + "adc\\t%4, %4, xzr" \\ + : "=&r"(lo_), "=&r"(hi_), "+r"(t0), "+r"(t1), "+r"(t2) \\ + : "r"(x), "r"(y) \\ + : "cc"); \\ + } while (0) + +/* (s2:s1:s0) = x * y */ +#define MONT_MUL(s0, s1, s2, x, y) \\ + __asm__("mul\\t%0, %3, %4\\n\\t" \\ + "umulh\\t%1, %3, %4\\n\\t" \\ + "mov\\t%2, xzr" \\ + : "=&r"(s0), "=&r"(s1), "=r"(s2) \\ + : "r"(x), "r"(y)) + +/* (t2:t1:t0) += (s2:s1:s0) */ +#define MONT_ADD(t0, t1, t2, s0, s1, s2) \\ + __asm__("adds\\t%0, %0, %3\\n\\t" \\ + "adcs\\t%1, %1, %4\\n\\t" \\ + "adc\\t%2, %2, %5" \\ + : "+r"(t0), "+r"(t1), "+r"(t2) \\ + : "r"(s0), "r"(s1), "r"(s2) \\ + : "cc") + +/* (s2:s1:s0) *= 2 */ +#define MONT_DBL(s0, s1, s2) \\ + do { \\ + s2 = (s2 << 1) | (s1 >> 63); \\ + s1 = (s1 << 1) | (s0 >> 63); \\ + s0 <<= 1; \\ + } while (0) +""" + +FOOTER = """\ +#endif +""" + + +class Column: + """Products added to accumulators, each set up by its first product.""" + + def __init__(self, w): + self.w = w + self.started = set() + + def add(self, acc, x, y): + if acc == "t" or acc in self.started: + self.w(f"\tMONT_MAC({acc}0, {acc}1, {acc}2, {x}, {y});") + else: + self.w(f"\tMONT_MUL({acc}0, {acc}1, {acc}2, {x}, {y});") + self.started.add(acc) + + def fold(self, into, acc): + if acc in self.started: + self.w(f"\tMONT_ADD({into}0, {into}1, {into}2, {acc}0, {acc}1, {acc}2);") + + +def column(w, singles, cross): + """singles are added once, cross twice: s and c take the cross products, + t, v and w take the rest, and all of them end up in t.""" + col = Column(w) + for k, (x, y) in enumerate(cross): + col.add("s" if k % 2 == 0 else "c", x, y) + for k, (x, y) in enumerate(singles): + col.add("tvw"[k % 3], x, y) + col.fold("t", "v") + col.fold("t", "w") + if cross: + col.fold("s", "c") + w("\tMONT_DBL(s0, s1, s2);") + w("\tMONT_ADD(t0, t1, t2, s0, s1, s2);") + + +def gen(n, square): + out = [] + w = out.append + name = f"crypton_mont_{'sqr' if square else 'mul'}{n}" + args = "const limb_t *a," if square else "const limb_t *a, const limb_t *b," + if square: + w(f"/* r = a * a * R^-1 mod m, for a under m; t is 2n limbs of scratch */") + else: + w(f"/* r = a * b * R^-1 mod m, for a and b under m; t is 2n limbs of scratch */") + w(f"static void {name}(limb_t *r, {args}") + w(f"{' ' * len(f'static void {name}(')}const limb_t *m, limb_t n0, limb_t *t)") + w("{") + w(f"\tlimb_t *u = t, *z = t + {n};") + w("\tlimb_t t0 = 0, t1 = 0, t2 = 0, v0, v1, v2, w0, w1, w2, borrow, take;") + if square: + w("\tlimb_t s0, s1, s2, c0, c1, c2;") + w("") + for i in range(2 * n - 1): + w(f"\t/* column {i} */") + singles, cross = [], [] + for j in range(max(0, i - n + 1), min(i, n - 1) + 1): + k = i - j + if not square: + singles.append((f"a[{j}]", f"b[{k}]")) + elif j < k: + cross.append((f"a[{j}]", f"a[{k}]")) + elif j == k: + singles.append((f"a[{j}]", f"a[{j}]")) + for j in range(max(0, i - n + 1), min(i, n)): + singles.append((f"u[{j}]", f"m[{i - j}]")) + column(w, singles, cross) + if i < n: + w(f"\tu[{i}] = t0 * n0;") + w(f"\tMONT_MAC(t0, t1, t2, u[{i}], m[0]);") + else: + w(f"\tz[{i - n}] = t0;") + w("\tt0 = t1;") + w("\tt1 = t2;") + w("\tt2 = 0;") + w(f"\tz[{n - 1}] = t0;") + w("") + w("\t/* under 2m, with t1 the bit above n limbs: at most one subtraction */") + w(f"\tborrow = sub_n(r, z, m, {n});") + w("\ttake = t1 | (borrow ^ 1);") + w(f"\tselect_n(r, r, z, take & 1, {n});") + w("}") + w("") + return "\n".join(out) + + +def main(): + print(HEADER) + for n in SIZES: + print(gen(n, False)) + print(gen(n, True)) + for n in SQR_ONLY: + print(gen(n, True)) + print(FOOTER, end="") + + +if __name__ == "__main__": + main() diff --git a/cbits/crypton_pbkdf2.c b/cbits/crypton_pbkdf2.c --- a/cbits/crypton_pbkdf2.c +++ b/cbits/crypton_pbkdf2.c @@ -46,6 +46,7 @@ /* Internal function/type names for hash-specific things. */ #define HMAC_CTX(_name) HMAC_ ## _name ## _ctx +#define DIGEST_FITS(_name) pbkdf2_ ## _name ## _digest_fits_in_a_block #define HMAC_INIT(_name) HMAC_ ## _name ## _init #define HMAC_UPDATE(_name) HMAC_ ## _name ## _update #define HMAC_FINAL(_name) HMAC_ ## _name ## _final @@ -79,6 +80,16 @@ */ #define DECL_PBKDF2(_name, _blocksz, _hashsz, _ctx, \ _init, _update, _xform, _final, _xcpy, _xtract, _xxor) \ + /* HMAC_INIT below shortens a key longer than the block by hashing it, \ + * which writes _hashsz bytes into a buffer of _blocksz. An instantiation \ + * whose digest is larger than its block would overflow that buffer, and \ + * would do it before any check inside the function could say so -- which \ + * is where the check used to be. Refuse such an instantiation here \ + * instead, in front of the person writing it. The three below are \ + * SHA-1, SHA-256 and SHA-512, whose digests are 20, 32 and 64 bytes \ + * against blocks of 64, 64 and 128. */ \ + typedef char DIGEST_FITS(_name)[(_hashsz) <= (_blocksz) ? 1 : -1]; \ + \ typedef struct { \ _ctx inner; \ _ctx outer; \ @@ -101,9 +112,6 @@ nkey = _hashsz; \ } \ \ - /* Standard doesn't cover case where blocksz < hashsz. */ \ - assert(nkey <= _blocksz); \ - \ /* Right zero-pad short keys. */ \ if (k != key) \ memcpy(k, key, nkey); \ @@ -192,7 +200,16 @@ uint8_t *out, size_t nout) \ { \ assert(iterations); \ - assert(out && nout); \ + assert(out); \ + \ + /* Zero bytes of derived key is zero bytes of work. RFC 8018 asks for a \ + * positive dkLen and the loop below would write a block regardless, so \ + * this used to be `assert(out && nout)` -- which aborts the process, in \ + * a library built without NDEBUG, on a length the caller chose. \ + * Crypto.KDF.PBKDF2's own tryGenerate returns an empty result for this, \ + * so return and let the two agree. */ \ + if (nout == 0) \ + return; \ \ /* Starting point for inner loop. */ \ HMAC_CTX(_name) ctx; \ diff --git a/cbits/crypton_powm.c b/cbits/crypton_powm.c --- a/cbits/crypton_powm.c +++ b/cbits/crypton_powm.c @@ -20,21 +20,536 @@ #include #include #include +#include -/* four bits of exponent per window, so a table of sixteen and no leftover - * bits: a byte holds exactly two windows */ -#define WINDOW_BITS 4 +/* On AArch64, multiplication and squaring at the RSA half sizes go through + * generated code that scans a column at a time with the reduction folded in; + * see cbits/crypton_mont.py. CRYPTON_NO_MONT_FIXED keeps mont_mul and + * mont_sqr, to compare. */ +#if defined(__aarch64__) && LIMB_BITS == 64 && !defined(CRYPTON_NO_MONT_FIXED) \ + && (defined(__GNUC__) || defined(__clang__)) +#define CRYPTON_MONT_FIXED 1 +#include +#endif + +/* + * At RSA sizes on x86-64, the Montgomery multiplication below is the whole + * cost, and s2n-bignum's is twice as fast because the C cannot form the two + * carry chains ADCX and ADOX give. The window, the table and its masked + * scan are unchanged: only the multiply and the square are swapped, and + * only for the sizes s2n-bignum has a Karatsuba multiplication for. + * + * Not on AArch64, where the C measures 5% faster than the assembly. + * See cbits/s2n/README.md. + */ +/* Not on macOS, where crypton.cabal leaves these files out: Apple's assembler + * cannot assemble bignum_emontredc_8n. See the note beside them there. */ +#if defined(CRYPTON_S2N_BIGNUM) && defined(__x86_64__) && !defined(__APPLE__) +#define CRYPTON_POWM_S2N 1 +#include + +extern void bignum_kmul_16_32(uint64_t *z, const uint64_t *x, + const uint64_t *y, uint64_t *t); +extern void bignum_ksqr_16_32(uint64_t *z, const uint64_t *x, uint64_t *t); +extern void bignum_kmul_32_64(uint64_t *z, const uint64_t *x, + const uint64_t *y, uint64_t *t); +extern void bignum_ksqr_32_64(uint64_t *z, const uint64_t *x, uint64_t *t); +extern uint64_t bignum_emontredc_8n(uint64_t k, uint64_t *z, + const uint64_t *m, uint64_t w); +extern void bignum_mul(uint64_t k, uint64_t *z, uint64_t m, const uint64_t *x, + uint64_t n, const uint64_t *y); +extern void bignum_sqr(uint64_t k, uint64_t *z, uint64_t n, const uint64_t *x); +extern void bignum_sqr_8_16(uint64_t *z, const uint64_t *x); +extern void bignum_mul_8_16(uint64_t *z, const uint64_t *x, const uint64_t *y); + +/* 16 limbs is 1024 bits and 32 is 2048: the halves a CRT exponentiation + * works in for RSA-2048 and RSA-4096, and the whole thing without CRT. 24 + * is 1536, the half of RSA-3072, which has no Karatsuba routine of its own: + * its multiplication is the 32-limb one with the operands padded, and its + * square is built from 8-limb pieces, which measured faster on a whole + * exponentiation than the padded square. Both are still nearly twice the + * C, since the C cannot form ADCX and ADOX's two carry chains at any size. + * + * 48 and 64 are 3072 and 4096 bits: finite-field Diffie-Hellman over the + * RFC 7919 groups of those sizes, and RSA without CRT. There is no + * Karatsuba routine at either, and padding to the next would be wasteful, + * so the multiplication is s2n-bignum's schoolbook one of any size -- + * plain MUL and ADC, no ADX -- and the reduction is the same ADX one. Not + * the gain the Karatsuba sizes have, and still well ahead of the C. + * + * The reduction wants ADX, so the answer is a run-time one. */ +static int powm_s2n_usable(uint32_t n) +{ + return (n == 16 || n == 24 || n == 32 || n == 48 || n == 64) + && (crypton_x86_simd_features() & CRYPTON_X86_ADX) != 0; +} + +/* The scratch the routines ask for, in limbs. kmul_16_32 and kmul_32_64 + * want three times their operand. A 24-limb operand padded to 32 wants the + * 32-limb routine's 64-limb product and its 96 limbs of its own, and the two + * padded operands besides. */ +#define POWM_S2N_PAD_Z 0 +#define POWM_S2N_PAD_T 64 +#define POWM_S2N_PAD_X (64 + 96) +#define POWM_S2N_PAD_Y (64 + 96 + 32) +static uint32_t powm_s2n_scratch(uint32_t n) +{ + return n == 24 ? 64 + 96 + 32 + 32 : 3 * n; +} + +/* a 24-limb operand in the low limbs of 32, the rest zero: zeroed every + * time rather than once, so that nothing else writing the scratch can + * leave a high limb behind */ +static void powm_s2n_pad(limb_t *to, const limb_t *from) +{ + memcpy(to, from, 24 * sizeof(limb_t)); + memset(to + 24, 0, 8 * sizeof(limb_t)); +} + +/* bignum_emontredc_8n leaves the result in the top half of z with one more + * bit as its return value, and what is there is under twice the modulus -- + * the same place mont_reduce ends up, and finished the same way. */ +static void powm_s2n_finish(limb_t *r, limb_t *z, const limb_t *m, + limb_t carry, uint32_t n) +{ + limb_t borrow = sub_n(r, z + n, m, n); + limb_t take = carry | (borrow ^ 1); + + select_n(r, r, z + n, take & 1, n); +} + +static void powm_s2n_mul(limb_t *r, const limb_t *a, const limb_t *b, + const limb_t *m, limb_t n0, uint32_t n, limb_t *z, + limb_t *scratch) +{ + if (n == 24) { + /* the product of two 24-limb numbers is 48 limbs, so the top + * sixteen of the 64 written are zero and the reduction reads + * only what it should */ + z = scratch + POWM_S2N_PAD_Z; + powm_s2n_pad(scratch + POWM_S2N_PAD_X, a); + powm_s2n_pad(scratch + POWM_S2N_PAD_Y, b); + bignum_kmul_32_64(z, scratch + POWM_S2N_PAD_X, + scratch + POWM_S2N_PAD_Y, + scratch + POWM_S2N_PAD_T); + } else if (n == 48 || n == 64) + bignum_mul(2 * n, z, n, a, n, b); + else if (n == 16) + bignum_kmul_16_32(z, a, b, scratch); + else + bignum_kmul_32_64(z, a, b, scratch); + powm_s2n_finish(r, z, m, bignum_emontredc_8n(n, z, m, n0), n); +} + +/* a^2 for a of 24 limbs, from three 8-limb squares and three 8-limb + * products: a = a0 + a1 X + a2 X^2 with X = 2^512, and the cross terms, at + * X to X^3 in c, go in doubled. c is 48 limbs of scratch. */ +static void powm_s2n_sqr24(limb_t *z, const limb_t *a, limb_t *c) +{ + limb_t k; + uint32_t i; + + bignum_sqr_8_16(z, a); + bignum_sqr_8_16(z + 16, a + 8); + bignum_sqr_8_16(z + 32, a + 16); + bignum_mul_8_16(c, a, a + 8); + bignum_mul_8_16(c + 16, a + 8, a + 16); + bignum_mul_8_16(c + 32, a, a + 16); + k = add_n(c + 8, c + 8, c + 32, 16); + for (i = 24; i < 32; i++) { + c[i] += k; + k = c[i] < k; + } + k = shl1(c, 32); + k += add_n(z + 8, z + 8, c, 32); + for (i = 40; i < 48; i++) { + z[i] += k; + k = z[i] < k; + } +} + +static void powm_s2n_sqr(limb_t *r, const limb_t *a, const limb_t *m, + limb_t n0, uint32_t n, limb_t *z, limb_t *scratch) +{ + if (n == 24) { + z = scratch + POWM_S2N_PAD_Z; + powm_s2n_sqr24(z, a, scratch + POWM_S2N_PAD_T); + } else if (n == 48 || n == 64) + bignum_sqr(2 * n, z, n, a); + else if (n == 16) + bignum_ksqr_16_32(z, a, scratch); + else + bignum_ksqr_32_64(z, a, scratch); + powm_s2n_finish(r, z, m, bignum_emontredc_8n(n, z, m, n0), n); +} +#else +static uint32_t powm_s2n_scratch(uint32_t n) +{ + (void) n; + return 0; +} +#endif + +/* One or the other, decided once per call */ +static void powm_mul(limb_t *r, const limb_t *a, const limb_t *b, + const limb_t *m, limb_t n0, uint32_t n, limb_t *t, + limb_t *scratch, int s2n) +{ +#ifdef CRYPTON_POWM_S2N + if (s2n) { + powm_s2n_mul(r, a, b, m, n0, n, t, scratch); + return; + } +#else + (void)scratch; + (void)s2n; +#endif +#ifdef CRYPTON_MONT_FIXED + switch (n) { + case 16: crypton_mont_mul16(r, a, b, m, n0, t); return; + case 24: crypton_mont_mul24(r, a, b, m, n0, t); return; + case 32: crypton_mont_mul32(r, a, b, m, n0, t); return; + } +#endif + mont_mul(r, a, b, m, n0, n, t); +} + +static void powm_sqr(limb_t *r, const limb_t *a, const limb_t *m, limb_t n0, + uint32_t n, limb_t *t, limb_t *scratch, int s2n) +{ +#ifdef CRYPTON_POWM_S2N + if (s2n) { + powm_s2n_sqr(r, a, m, n0, n, t, scratch); + return; + } +#else + (void)scratch; + (void)s2n; +#endif +#ifdef CRYPTON_MONT_FIXED + switch (n) { + case 16: crypton_mont_sqr16(r, a, m, n0, t); return; + case 24: crypton_mont_sqr24(r, a, m, n0, t); return; + case 32: crypton_mont_sqr32(r, a, m, n0, t); return; + case 48: crypton_mont_sqr48(r, a, m, n0, t); return; + } +#endif + mont_sqr(r, a, m, n0, n, t); +} + +/* Four bits of exponent per window, so a table of sixteen. + * + * Five was written and measured and is not here. A wider window saves + * multiplications -- 205 of them against 256 at 1024 bits, with the same + * 1024 squarings -- and pays for it in the masked scan of a table twice as + * long. When that was first measured the scan was the scalar one, and five + * came out 3.5% better on an Apple M4, about 1% worse on an older x86-64 and + * 7% worse where s2n-bignum's multiplication runs, since that makes the scan + * the expensive half. The conclusion written here was that what would make + * a wider window pay is a cheaper scan. + * + * The cheaper scan arrived twenty-one minutes later, in the AVX2 form above, + * and nobody measured the window again. Measured since, on the machine the + * x86-64 table is measured on and with both builds raced in one run: five is + * 0.6% worse than four, beside 0.5% between two builds that differ in + * nothing. So the cheaper scan did not turn it round -- the window stays at + * four, and now for a reason that was checked after the scan rather than + * before it. + * + * The width is POWM_WINDOW_BITS so that it can be measured rather than + * argued about. It had to be renamed to be settable at all: WINDOW_BITS was + * defined here, in cbits/crypton_ecc.c and in cbits/crypton_bignum.h, all + * three to four, and a redefinition to the same value is silent -- so + * -DWINDOW_BITS=5 was overridden by the header and three builds meant to + * differ did not. cbits/tests/powm builds the walk at four, five and six + * and requires the same answers of all three. */ +#ifndef POWM_WINDOW_BITS +#define POWM_WINDOW_BITS 4 +#endif +#define WINDOW_BITS POWM_WINDOW_BITS #define TABLE_SIZE (1 << WINDOW_BITS) +/* Window number k counting from the bottom, where the bottom window holds + * the exponent's least significant WINDOW_BITS bits. The exponent is + * big-endian bytes, and a window may straddle a byte boundary once the width + * stops dividing eight, so this reads the two bytes it can fall across. + * + * Which window is being read is the loop counter, not the exponent, so the + * shifts and the index here are public. + */ +static limb_t exp_window(const uint8_t *exp, uint32_t explen, uint32_t k) +{ + uint32_t lo = k * WINDOW_BITS; /* bit offset from the bottom */ + uint32_t byte = lo / 8, off = lo % 8; + uint32_t v; + + if (byte >= explen) + return 0; + v = exp[explen - 1 - byte]; + if (off + WINDOW_BITS > 8 && byte + 1 < explen) + v |= (uint32_t) exp[explen - 2 - byte] << 8; + return (v >> off) & (TABLE_SIZE - 1); +} + +/* The masked scan of the table: every entry is read and a mask keeps the one + * wanted, so that the address stream does not follow the exponent. At + * RSA-2048's CRT size that is two kilobytes read per window, and the window + * loop runs 256 times per exponentiation, which is why it is worth a vector + * register: removing the scan altogether measures 11% of an exponentiation + * where s2n-bignum's multiplication runs, and the AVX2 form below gets + * essentially all of it. */ +static void scan_table(limb_t *sel, const limb_t *table, uint32_t n, limb_t w) +{ + uint32_t k, l; + + memset(sel, 0, n * sizeof(limb_t)); + for (k = 0; k < TABLE_SIZE; k++) { + limb_t mask = eq_mask(k, w); + + for (l = 0; l < n; l++) + sel[l] |= table[k * n + l] & mask; + } +} + +#if defined(__x86_64__) && defined(WITH_TARGET_ATTRIBUTES) && LIMB_BITS == 64 +#define CRYPTON_POWM_SCAN_AVX2 1 +#include +#include + +/* The same scan four limbs at a time. The sixteen masks are worked out + * once; after that each register of the answer is one pass over the table's + * column, reading every entry exactly as the scalar form does. */ +__attribute__((target("avx2"))) +static void scan_table_avx2(limb_t *sel, const limb_t *table, uint32_t n, + limb_t w) +{ + __m256i masks[TABLE_SIZE]; + uint32_t k, l; + + for (k = 0; k < TABLE_SIZE; k++) + masks[k] = _mm256_cmpeq_epi64( + _mm256_set1_epi64x((long long) k), + _mm256_set1_epi64x((long long) w)); + + for (l = 0; l + 4 <= n; l += 4) { + __m256i acc = _mm256_setzero_si256(); + + for (k = 0; k < TABLE_SIZE; k++) { + __m256i v = _mm256_loadu_si256( + (const __m256i *) (table + k * n + l)); + + acc = _mm256_or_si256(acc, + _mm256_and_si256(v, masks[k])); + } + _mm256_storeu_si256((__m256i *) (sel + l), acc); + } + + /* a modulus whose limbs do not come in fours ends here */ + for (; l < n; l++) { + limb_t v = 0; + + for (k = 0; k < TABLE_SIZE; k++) + v |= table[k * n + l] & eq_mask(k, w); + sel[l] = v; + } +} +#endif + +/* Modular exponentiation for an exponent that is not a secret. + * + * RSA verification raises a signature to the public exponent, which is in + * the public key and is usually 65537. crypton_powm_sec below hides the + * exponent and pays for it twice: a table of sixteen powers built before the + * walk begins, and every entry of that table read at every window. For a + * seventeen-bit exponent the table alone costs more than the exponentiation. + * + * So this one reads the exponent's bits and branches on them, squaring at + * every step and multiplying where a bit is set. It is for public exponents + * only, and nothing in this library may call it with a secret. + * + * It declines -- returning 1 without touching out -- in three cases, so a + * caller must have something to fall back to. + * + * The first is a long exponent. Walking it a bit at a time costs about one + * and a half multiplications per bit where a windowed walk costs a quarter + * less, and the table that buys the difference pays for itself after some + * sixty-odd bits. Past that the caller's own exponentiation is the better + * algorithm and this one would have to grow a table to compete, which is + * the thing it exists not to have. RSA public exponents are seventeen + * bits; the bound is set well above them and well below where a Diffie- + * Hellman or DSA exponent starts. + * + * The second and third are a machine and a size where GMP, which the + * caller has, was measured the faster; see powm_pub_wanted. + * + * Define CRYPTON_POWM_PUB_ALWAYS to make it accept anyway: the differential + * test builds that way, so that this arithmetic is exercised on the + * machines the assembly never reaches. + */ +#define POWM_PUB_MAX_EXP_BYTES 8 + +/* Which sizes this is worth taking at is not which sizes the assembly runs + * at. Against GMP, which is what the caller has, on an EPYC 7763 with ADX + * on both sides: ahead at 2048 bits, and behind everywhere else measured -- + * slightly at 1024, by more at 1536, and by more again at 3072 and 4096, + * which is the size where GMP's own setup and multiplication are the better + * pair. So 2048 alone, which is RSA-2048's modulus and the size nearly + * every RSA public key has. It once took every size the assembly ran at, + * on the strength of 2048 alone, and was slower than GMP at 1024. */ +static int powm_pub_wanted(uint32_t n) +{ +#if defined(CRYPTON_POWM_PUB_ALWAYS) + return n > 0; +#elif defined(CRYPTON_POWM_S2N) + return n == 32 && powm_s2n_usable(n); +#elif defined(CRYPTON_MONT_FIXED) + /* AArch64: the generated code is ahead of GMP at 2048 bits. At the + * other RSA sizes Linux's GMP is level with the C or ahead of it. The + * GMP GHC builds into itself on macOS is compiled without -O, and is + * behind the C from 1024 to 4096 bits on the same M4 */ +#if defined(__APPLE__) + return n >= 16 && n <= 64; +#else + return n == 32; +#endif +#else + (void) n; + return 0; +#endif +} + +int crypton_powm_window_bits(void) +{ + return WINDOW_BITS; +} + +int crypton_powm_pub(uint8_t *out, + const uint8_t *base, uint32_t baselen, + const uint8_t *exp, uint32_t explen, + const uint8_t *mod, uint32_t modlen) +{ + return crypton_powm_pub_r2(out, base, baselen, exp, explen, mod, modlen, NULL); +} + +int crypton_powm_r2(uint8_t *out, const uint8_t *mod, uint32_t modlen) +{ + uint32_t n = (modlen + LIMB_BYTES - 1) / LIMB_BYTES; + limb_t *space, *m, *r2, *t; + + if (modlen == 0 || n == 0 || (mod[modlen - 1] & 1) == 0 || !powm_pub_wanted(n)) + return 1; + space = calloc(5 * n, sizeof(limb_t)); + if (space == NULL) + return 1; + m = space; + r2 = m + n; + t = r2 + n; + if (from_be(m, n, mod, modlen) != 0) { + free(space); + return 1; + } + mont_r2(r2, m, mont_n0(m[0]), n, t); + to_be(out, modlen, r2, n); + free(space); + return 0; +} + +int crypton_powm_pub_r2(uint8_t *out, + const uint8_t *base, uint32_t baselen, + const uint8_t *exp, uint32_t explen, + const uint8_t *mod, uint32_t modlen, + const uint8_t *r2be) +{ + uint32_t n = (modlen + LIMB_BYTES - 1) / LIMB_BYTES; + uint32_t words = (6 + 2) * n + powm_s2n_scratch(n); + limb_t *space, *m, *r2, *acc, *tmp, *bm, *one, *t, *scratch, n0; + uint32_t top, i; + int s2n = 0; + + if (modlen == 0 || n == 0 || (mod[modlen - 1] & 1) == 0) + return 1; + if (explen > POWM_PUB_MAX_EXP_BYTES) + return 1; + if (!powm_pub_wanted(n)) + return 1; +#ifdef CRYPTON_POWM_S2N + s2n = powm_s2n_usable(n); +#endif + + /* the exponent's top set bit, counted from the bottom. An exponent of + * zero is left to the caller, whose answer for it is already right. */ + top = explen * 8; + while (top > 0 && + ((exp[explen - 1 - (top - 1) / 8] >> ((top - 1) % 8)) & 1) == 0) + top--; + if (top == 0) + return 1; + + space = calloc(words, sizeof(limb_t)); + if (space == NULL) + return 1; + m = space; + r2 = m + n; + acc = r2 + n; + tmp = acc + n; + bm = tmp + n; + one = bm + n; + t = one + n; + scratch = t + 2 * n; + + if (from_be(m, n, mod, modlen) != 0) + goto fail; + n0 = mont_n0(m[0]); + if (r2be == NULL) + mont_r2(r2, m, n0, n, t); + else if (from_be(r2, n, r2be, modlen) != 0) + goto fail; + + /* the base into Montgomery form. The accumulator starts there rather + * than at one, because the exponent's top bit is a one by + * construction and the first squaring would be of one. */ + if (from_be(tmp, n, base, baselen) != 0) + goto fail; + powm_mul(bm, tmp, r2, m, n0, n, t, scratch, s2n); + memcpy(acc, bm, n * sizeof(limb_t)); + + for (i = top - 1; i > 0; i--) { + limb_t *swap; + + powm_sqr(tmp, acc, m, n0, n, t, scratch, s2n); + swap = acc; acc = tmp; tmp = swap; + if ((exp[explen - 1 - (i - 1) / 8] >> ((i - 1) % 8)) & 1) { + powm_mul(tmp, acc, bm, m, n0, n, t, scratch, s2n); + swap = acc; acc = tmp; tmp = swap; + } + } + + /* out of Montgomery form */ + one[0] = 1; + powm_mul(tmp, acc, one, m, n0, n, t, scratch, s2n); + to_be(out, modlen, tmp, n); + + free(space); + return 0; + +fail: + free(space); + return 1; +} + int crypton_powm_sec(uint8_t *out, const uint8_t *base, uint32_t baselen, const uint8_t *exp, uint32_t explen, const uint8_t *mod, uint32_t modlen) { uint32_t n = (modlen + LIMB_BYTES - 1) / LIMB_BYTES; - uint32_t words = (TABLE_SIZE + 7) * n; - limb_t *space, *m, *r2, *acc, *sel, *prod, *table, *t, n0; + uint32_t words = (TABLE_SIZE + 7) * n + powm_s2n_scratch(n); + limb_t *space, *m, *r2, *acc, *sel, *prod, *table, *t, *scratch, n0; uint32_t i, j, k; + int s2n = 0; +#ifdef CRYPTON_POWM_SCAN_AVX2 + int avx2 = (crypton_x86_simd_features() & CRYPTON_X86_AVX2) != 0; +#endif if (modlen == 0 || n == 0 || (mod[modlen - 1] & 1) == 0) return 1; @@ -50,62 +565,78 @@ prod = sel + n; t = prod + n; table = t + 2 * n; + scratch = table + TABLE_SIZE * n; +#ifdef CRYPTON_POWM_S2N + s2n = powm_s2n_usable(n); +#endif + if (from_be(m, n, mod, modlen) != 0) goto fail; - mont_r2(r2, m, n, t); n0 = mont_n0(m[0]); + mont_r2(r2, m, n0, n, t); /* table[k] = base^k in Montgomery form, and table[0] = 1 there */ memset(table, 0, n * sizeof(limb_t)); table[0] = 1; - mont_mul(acc, table, r2, m, n0, n, t); + powm_mul(acc, table, r2, m, n0, n, t, scratch, s2n); memcpy(table, acc, n * sizeof(limb_t)); if (from_be(sel, n, base, baselen) != 0) goto fail; - mont_mul(table + n, sel, r2, m, n0, n, t); + powm_mul(table + n, sel, r2, m, n0, n, t, scratch, s2n); for (k = 2; k < TABLE_SIZE; k++) - mont_mul(table + k * n, table + (k - 1) * n, table + n, m, n0, n, t); + powm_mul(table + k * n, table + (k - 1) * n, table + n, m, n0, n, + t, scratch, s2n); memcpy(acc, table, n * sizeof(limb_t)); - for (i = explen * 2; i > 0; i--) { - uint32_t nib = i - 1; - limb_t w = (exp[explen - 1 - nib / 2] >> (4 * (nib % 2))) & 0xf; + for (i = (explen * 8 + WINDOW_BITS - 1) / WINDOW_BITS; i > 0; i--) { + limb_t w = exp_window(exp, explen, i - 1); + /* squaring into the other buffer and swapping the two saves a + * copy of the modulus' width every time; which of the three + * buffers a pointer names is nobody's secret */ for (j = 0; j < WINDOW_BITS; j++) { - mont_sqr(sel, acc, m, n0, n, t); - memcpy(acc, sel, n * sizeof(limb_t)); + limb_t *swap; + + powm_sqr(sel, acc, m, n0, n, t, scratch, s2n); + swap = acc; + acc = sel; + sel = swap; } - /* every entry is read, and a mask keeps the one wanted */ - memset(sel, 0, n * sizeof(limb_t)); - for (k = 0; k < TABLE_SIZE; k++) { - limb_t mask = eq_mask(k, w); - uint32_t l; +#ifdef CRYPTON_POWM_SCAN_AVX2 + if (avx2) + scan_table_avx2(sel, table, n, w); + else +#endif + scan_table(sel, table, n, w); + powm_mul(prod, acc, sel, m, n0, n, t, scratch, s2n); + { + limb_t *swap = acc; - for (l = 0; l < n; l++) - sel[l] |= table[k * n + l] & mask; + acc = prod; + prod = swap; } - mont_mul(prod, acc, sel, m, n0, n, t); - memcpy(acc, prod, n * sizeof(limb_t)); } /* out of Montgomery form */ memset(sel, 0, n * sizeof(limb_t)); sel[0] = 1; - mont_mul(prod, acc, sel, m, n0, n, t); + powm_mul(prod, acc, sel, m, n0, n, t, scratch, s2n); to_be(out, modlen, prod, n); /* nothing here is the caller's secret, but the exponent's bits passed - * through the accumulators */ - memset(space, 0, words * sizeof(limb_t)); + * through the accumulators. crypton_bzero rather than memset, since + * this memory is freed on the next line and a store to memory about to + * die is one an optimizer may drop. */ + crypton_bzero(space, words * sizeof(limb_t)); free(space); return 0; fail: - memset(space, 0, words * sizeof(limb_t)); + crypton_bzero(space, words * sizeof(limb_t)); free(space); return 1; } diff --git a/cbits/crypton_powm.h b/cbits/crypton_powm.h --- a/cbits/crypton_powm.h +++ b/cbits/crypton_powm.h @@ -20,4 +20,53 @@ const uint8_t *exp, uint32_t explen, const uint8_t *mod, uint32_t modlen); +/* The same, for an exponent that is not a secret: it reads the exponent's + * bits and branches on them. Nothing in this library may call it with a + * secret exponent. + * + * It declines -- returning 1 with out untouched -- whenever it is not the + * faster of the two things the caller could do: for an exponent of more + * than eight bytes, where a windowed walk is the better algorithm; + * everywhere the vendored assembly's Montgomery multiplication does not + * run; and at every modulus size but 2048 bits, where GMP was measured the + * faster even with the assembly. So a caller must have something to fall + * back to. + */ +int crypton_powm_pub(uint8_t *out, + const uint8_t *base, uint32_t baselen, + const uint8_t *exp, uint32_t explen, + const uint8_t *mod, uint32_t modlen); + +/* crypton_powm_pub with R^2 mod m given, big-endian in modlen bytes, as + * crypton_powm_r2 computes it; NULL computes it here as crypton_powm_pub does. + * A caller verifying under one key many times keeps it. */ +int crypton_powm_pub_r2(uint8_t *out, + const uint8_t *base, uint32_t baselen, + const uint8_t *exp, uint32_t explen, + const uint8_t *mod, uint32_t modlen, + const uint8_t *r2be); + +/* R^2 mod m for crypton_powm_pub_r2, or nonzero where crypton_powm_pub would + * decline this modulus, so that nothing is kept for it. */ +int crypton_powm_r2(uint8_t *out, const uint8_t *mod, uint32_t modlen); + +/* Two secure exponentiations at once, as the halves of an RSA private-key + * operation by CRT are: each as crypton_powm_sec, with the same contract. + * + * Only on x86-64 with AVX-512 IFMA, for two moduli of 128, 192 or 256 bytes + * and two exponents of the same length. It declines everywhere else -- + * returning nonzero with both outputs untouched -- so a caller must have + * crypton_powm_sec to fall back to. */ +int crypton_powm_sec_x2(uint8_t *out1, const uint8_t *base1, uint32_t baselen1, + const uint8_t *exp1, uint32_t explen1, + const uint8_t *mod1, uint32_t modlen1, + uint8_t *out2, const uint8_t *base2, uint32_t baselen2, + const uint8_t *exp2, uint32_t explen2, + const uint8_t *mod2, uint32_t modlen2); + +/* The window the secure exponentiation walks the exponent in, so that a + * build which did not take -DPOWM_WINDOW_BITS cannot pass for one that + * did. */ +int crypton_powm_window_bits(void); + #endif diff --git a/cbits/crypton_powm_ifma.c b/cbits/crypton_powm_ifma.c new file mode 100644 --- /dev/null +++ b/cbits/crypton_powm_ifma.c @@ -0,0 +1,187 @@ +/* Two secure exponentiations at once with AVX-512 IFMA, the CRT halves of + * RSA-2048, -3072 and -4096: crypton_powm_sec's walk with a five-bit window, + * over the multiplication generated by cbits/crypton_powm_ifma.py. */ +#include +#include +#include +#include +#include + +#if defined(__x86_64__) && (defined(__GNUC__) || defined(__clang__)) +#include +#include "crypton_powm_ifma.h" + +#define WIN 5 + +typedef void (*amm_fn)(uint64_t *, const uint64_t *, const uint64_t *, const uint64_t *, + uint64_t, uint64_t *, const uint64_t *, const uint64_t *, + const uint64_t *, uint64_t); +typedef void (*sel_fn)(uint64_t *, const uint64_t *, uint64_t, uint64_t *, + const uint64_t *, uint64_t); + +/* limbs of 64 bits to digits of 52, d of them; the rest of the w words zero */ +static void to_digits(uint64_t *dg, uint32_t w, const limb_t *l, uint32_t n, uint32_t d) +{ + uint32_t i; + + memset(dg, 0, w * sizeof(uint64_t)); + for (i = 0; i < d; i++) { + uint32_t bit = 52 * i, li = bit / 64, sh = bit % 64; + uint64_t v = li < n ? l[li] >> sh : 0; + + if (sh > 12 && li + 1 < n) + v |= l[li + 1] << (64 - sh); + dg[i] = v & IFMA_MASK52; + } +} + +static void from_digits(limb_t *l, uint32_t n, const uint64_t *dg, uint32_t d) +{ + uint32_t i; + + memset(l, 0, n * sizeof(limb_t)); + for (i = 0; i < d; i++) { + uint32_t bit = 52 * i, li = bit / 64, sh = bit % 64; + + if (li < n) + l[li] |= dg[i] << sh; + if (sh > 12 && li + 1 < n) + l[li + 1] |= dg[i] >> (64 - sh); + } +} + +/* Window k from the bottom of a big-endian exponent of len bytes. Which + * window is read is the loop counter's business, not the exponent's. */ +static uint64_t window(const uint8_t *e, uint32_t len, uint32_t k) +{ + uint64_t v = 0; + uint32_t j; + + for (j = 0; j < WIN; j++) { + uint32_t bit = k * WIN + j; + + if (bit < 8 * len) + v |= (uint64_t) ((e[len - 1 - bit / 8] >> (bit % 8)) & 1) << j; + } + return v; +} + +/* m and k0 in digits, and R^2 mod m for R = 2^(52d): crypton's R^2 is for + * 2^(64n), so it is doubled the rest of the way */ +static int setup(const uint8_t *mod, uint32_t modlen, uint32_t n, uint32_t d, uint32_t w, + limb_t *ml, uint64_t *md, uint64_t *rr, uint64_t *k0, limb_t *t) +{ + limb_t *r2 = t + 4 * n; + uint32_t j; + + if (from_be(ml, n, mod, modlen) != 0 || (ml[0] & 1) == 0) + return 1; + *k0 = mont_n0(ml[0]); + mont_r2(r2, ml, *k0, n, t); + for (j = 0; j < 2 * (52 * d - 64 * n); j++) + dbl_mod(r2, ml, n, t); + to_digits(md, w, ml, n, d); + to_digits(rr, w, r2, n, d); + return 0; +} + +int crypton_powm_sec_x2(uint8_t *out1, const uint8_t *base1, uint32_t baselen1, + const uint8_t *exp1, uint32_t explen1, + const uint8_t *mod1, uint32_t modlen1, + uint8_t *out2, const uint8_t *base2, uint32_t baselen2, + const uint8_t *exp2, uint32_t explen2, + const uint8_t *mod2, uint32_t modlen2) +{ + uint32_t n, d, w, nwin, k, j; + amm_fn amm; + sel_fn sel; + uint64_t *space, *m1, *m2, *rr1, *rr2, *x1, *x2, *s1, *s2, *one, *t1, *t2, k1, k2; + limb_t *l1, *l2, *lt, *lt2; + size_t words; + + if (modlen1 != modlen2 || explen1 != explen2) + return 1; + if ((crypton_x86_simd_features() & CRYPTON_X86_IFMA) == 0) + return 1; + switch (modlen1) { + case 128: d = 20; amm = ifma_amm20_x2; sel = ifma_sel5; break; + case 192: d = 30; amm = ifma_amm30_x2; sel = ifma_sel8; break; + case 256: d = 40; amm = ifma_amm40_x2; sel = ifma_sel10; break; + default: return 1; + } + n = modlen1 / 8; + w = 4 * ((d + 3) / 4); + + /* nine numbers in digits, two tables of 32, and limbs for the setup */ + words = (size_t) w * (9 + 2 * 32) + (size_t) n * 12; + space = calloc(words, sizeof(uint64_t)); + if (space == NULL) + return 1; + m1 = space; m2 = m1 + w; rr1 = m2 + w; rr2 = rr1 + w; + x1 = rr2 + w; x2 = x1 + w; s1 = x2 + w; s2 = s1 + w; one = s2 + w; + t1 = one + w; t2 = t1 + 32 * w; + l1 = (limb_t *)(t2 + 32 * w); l2 = l1 + n; lt = l2 + n; lt2 = lt + n; + + if (setup(mod1, modlen1, n, d, w, l1, m1, rr1, &k1, lt) != 0 + || setup(mod2, modlen2, n, d, w, l2, m2, rr2, &k2, lt) != 0) + goto fail; + + /* the bases, under their moduli as the contract says, into digits */ + if (from_be(lt, n, base1, baselen1) != 0 || from_be(lt2, n, base2, baselen2) != 0) + goto fail; + to_digits(s1, w, lt, n, d); + to_digits(s2, w, lt2, n, d); + + /* table[0] = R and table[1] = b R in Montgomery form, table[k] = b^k R */ + one[0] = 1; + amm(t1, one, rr1, m1, k1, t2, one, rr2, m2, k2); + amm(t1 + w, s1, rr1, m1, k1, t2 + w, s2, rr2, m2, k2); + for (k = 2; k < 32; k++) + amm(t1 + k * w, t1 + (k - 1) * w, t1 + w, m1, k1, + t2 + k * w, t2 + (k - 1) * w, t2 + w, m2, k2); + + memcpy(x1, t1, w * sizeof(uint64_t)); + memcpy(x2, t2, w * sizeof(uint64_t)); + nwin = (8 * explen1 + WIN - 1) / WIN; + for (k = nwin; k > 0; k--) { + for (j = 0; j < WIN; j++) + amm(x1, x1, x1, m1, k1, x2, x2, x2, m2, k2); + sel(s1, t1, window(exp1, explen1, k - 1), s2, t2, window(exp2, explen2, k - 1)); + amm(x1, x1, s1, m1, k1, x2, x2, s2, m2, k2); + } + + /* out of Montgomery form, then under m with one masked subtraction */ + amm(x1, x1, one, m1, k1, x2, x2, one, m2, k2); + from_digits(lt, n, x1, d); + select_n(lt, lt2, lt, sub_n(lt2, lt, l1, n) ^ 1, n); + to_be(out1, modlen1, lt, n); + from_digits(lt, n, x2, d); + select_n(lt, lt2, lt, sub_n(lt2, lt, l2, n) ^ 1, n); + to_be(out2, modlen2, lt, n); + + crypton_bzero(space, words * sizeof(uint64_t)); + free(space); + return 0; + +fail: + crypton_bzero(space, words * sizeof(uint64_t)); + free(space); + return 1; +} + +#else + +int crypton_powm_sec_x2(uint8_t *out1, const uint8_t *base1, uint32_t baselen1, + const uint8_t *exp1, uint32_t explen1, + const uint8_t *mod1, uint32_t modlen1, + uint8_t *out2, const uint8_t *base2, uint32_t baselen2, + const uint8_t *exp2, uint32_t explen2, + const uint8_t *mod2, uint32_t modlen2) +{ + (void) out1; (void) base1; (void) baselen1; (void) exp1; (void) explen1; + (void) mod1; (void) modlen1; (void) out2; (void) base2; (void) baselen2; + (void) exp2; (void) explen2; (void) mod2; (void) modlen2; + return 1; +} + +#endif diff --git a/cbits/crypton_powm_ifma.h b/cbits/crypton_powm_ifma.h new file mode 100644 --- /dev/null +++ b/cbits/crypton_powm_ifma.h @@ -0,0 +1,513 @@ +/* Generated by cbits/crypton_powm_ifma.py; do not edit. + * + * Almost-Montgomery multiplication in radix 2^52, two at once: r = a b + * 2^-52D mod m, under 2m for a and b under 2m. Digits are 52 bits in 64-bit + * words, each number padded with zero words to a whole number of 256-bit + * registers. m is read from memory; at 20 digits a is held in registers, + * above that it is read from memory too. */ +#ifndef CRYPTON_POWM_IFMA_H +#define CRYPTON_POWM_IFMA_H + +#include +#include + +#define IFMA_MASK52 ((UINT64_C(1) << 52) - 1) +#define IFMA_TARGET __attribute__((target("avx2,avx512f,avx512vl,avx512ifma"))) + +/* 20 lanes to 52-bit digits: what is above 52 bits moves up a lane, + * and the one-bit carries that can then remain are resolved together */ +IFMA_TARGET static inline void ifma_norm5(uint64_t *r, __m256i X0, __m256i X1, __m256i X2, __m256i X3, __m256i X4) +{ + const __m256i mk = _mm256_set1_epi64x((long long) IFMA_MASK52), one = _mm256_set1_epi64x(1); + const __m256i zero = _mm256_setzero_si256(); + uint64_t g = 0, p = 0, c; + __m256i C0 = _mm256_srli_epi64(X0, 52); + __m256i C1 = _mm256_srli_epi64(X1, 52); + __m256i C2 = _mm256_srli_epi64(X2, 52); + __m256i C3 = _mm256_srli_epi64(X3, 52); + __m256i C4 = _mm256_srli_epi64(X4, 52); + X0 = _mm256_add_epi64(_mm256_and_si256(X0, mk), _mm256_alignr_epi64(C0, zero, 3)); + X1 = _mm256_add_epi64(_mm256_and_si256(X1, mk), _mm256_alignr_epi64(C1, C0, 3)); + X2 = _mm256_add_epi64(_mm256_and_si256(X2, mk), _mm256_alignr_epi64(C2, C1, 3)); + X3 = _mm256_add_epi64(_mm256_and_si256(X3, mk), _mm256_alignr_epi64(C3, C2, 3)); + X4 = _mm256_add_epi64(_mm256_and_si256(X4, mk), _mm256_alignr_epi64(C4, C3, 3)); + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X0, mk) << 0; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X0, mk) << 0; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X1, mk) << 4; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X1, mk) << 4; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X2, mk) << 8; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X2, mk) << 8; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X3, mk) << 12; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X3, mk) << 12; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X4, mk) << 16; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X4, mk) << 16; + c = ((g << 1) + p) ^ p; + X0 = _mm256_and_si256(_mm256_mask_add_epi64(X0, (__mmask8)(c >> 0 & 15), X0, one), mk); + _mm256_storeu_si256((__m256i *)(r + 0), X0); + X1 = _mm256_and_si256(_mm256_mask_add_epi64(X1, (__mmask8)(c >> 4 & 15), X1, one), mk); + _mm256_storeu_si256((__m256i *)(r + 4), X1); + X2 = _mm256_and_si256(_mm256_mask_add_epi64(X2, (__mmask8)(c >> 8 & 15), X2, one), mk); + _mm256_storeu_si256((__m256i *)(r + 8), X2); + X3 = _mm256_and_si256(_mm256_mask_add_epi64(X3, (__mmask8)(c >> 12 & 15), X3, one), mk); + _mm256_storeu_si256((__m256i *)(r + 12), X3); + X4 = _mm256_and_si256(_mm256_mask_add_epi64(X4, (__mmask8)(c >> 16 & 15), X4, one), mk); + _mm256_storeu_si256((__m256i *)(r + 16), X4); +} + +/* both tables' entries k1 and k2 of 32, every entry read and a mask keeping + * one; entries are 20 words apart */ +IFMA_TARGET static void ifma_sel5(uint64_t *o1, const uint64_t *t1, uint64_t k1, + uint64_t *o2, const uint64_t *t2, uint64_t k2) +{ + const __m256i one = _mm256_set1_epi64x(1); + __m256i v1 = _mm256_set1_epi64x((long long) k1), v2 = _mm256_set1_epi64x((long long) k2); + __m256i kv = _mm256_setzero_si256(); + __m256i P0 = kv, Q0 = kv, P1 = kv, Q1 = kv, P2 = kv, Q2 = kv, P3 = kv, Q3 = kv, P4 = kv, Q4 = kv; + int k; + for (k = 0; k < 32; k++) { + __mmask8 s1 = _mm256_cmpeq_epi64_mask(v1, kv), s2 = _mm256_cmpeq_epi64_mask(v2, kv); + const uint64_t *e1 = t1 + 20 * k, *e2 = t2 + 20 * k; + P0 = _mm256_mask_loadu_epi64(P0, s1, e1 + 0); Q0 = _mm256_mask_loadu_epi64(Q0, s2, e2 + 0); + P1 = _mm256_mask_loadu_epi64(P1, s1, e1 + 4); Q1 = _mm256_mask_loadu_epi64(Q1, s2, e2 + 4); + P2 = _mm256_mask_loadu_epi64(P2, s1, e1 + 8); Q2 = _mm256_mask_loadu_epi64(Q2, s2, e2 + 8); + P3 = _mm256_mask_loadu_epi64(P3, s1, e1 + 12); Q3 = _mm256_mask_loadu_epi64(Q3, s2, e2 + 12); + P4 = _mm256_mask_loadu_epi64(P4, s1, e1 + 16); Q4 = _mm256_mask_loadu_epi64(Q4, s2, e2 + 16); + kv = _mm256_add_epi64(kv, one); + } + _mm256_storeu_si256((__m256i *)(o1 + 0), P0); _mm256_storeu_si256((__m256i *)(o2 + 0), Q0); + _mm256_storeu_si256((__m256i *)(o1 + 4), P1); _mm256_storeu_si256((__m256i *)(o2 + 4), Q1); + _mm256_storeu_si256((__m256i *)(o1 + 8), P2); _mm256_storeu_si256((__m256i *)(o2 + 8), Q2); + _mm256_storeu_si256((__m256i *)(o1 + 12), P3); _mm256_storeu_si256((__m256i *)(o2 + 12), Q3); + _mm256_storeu_si256((__m256i *)(o1 + 16), P4); _mm256_storeu_si256((__m256i *)(o2 + 16), Q4); +} + +/* 32 lanes to 52-bit digits: what is above 52 bits moves up a lane, + * and the one-bit carries that can then remain are resolved together */ +IFMA_TARGET static inline void ifma_norm8(uint64_t *r, __m256i X0, __m256i X1, __m256i X2, __m256i X3, __m256i X4, __m256i X5, __m256i X6, __m256i X7) +{ + const __m256i mk = _mm256_set1_epi64x((long long) IFMA_MASK52), one = _mm256_set1_epi64x(1); + const __m256i zero = _mm256_setzero_si256(); + uint64_t g = 0, p = 0, c; + __m256i C0 = _mm256_srli_epi64(X0, 52); + __m256i C1 = _mm256_srli_epi64(X1, 52); + __m256i C2 = _mm256_srli_epi64(X2, 52); + __m256i C3 = _mm256_srli_epi64(X3, 52); + __m256i C4 = _mm256_srli_epi64(X4, 52); + __m256i C5 = _mm256_srli_epi64(X5, 52); + __m256i C6 = _mm256_srli_epi64(X6, 52); + __m256i C7 = _mm256_srli_epi64(X7, 52); + X0 = _mm256_add_epi64(_mm256_and_si256(X0, mk), _mm256_alignr_epi64(C0, zero, 3)); + X1 = _mm256_add_epi64(_mm256_and_si256(X1, mk), _mm256_alignr_epi64(C1, C0, 3)); + X2 = _mm256_add_epi64(_mm256_and_si256(X2, mk), _mm256_alignr_epi64(C2, C1, 3)); + X3 = _mm256_add_epi64(_mm256_and_si256(X3, mk), _mm256_alignr_epi64(C3, C2, 3)); + X4 = _mm256_add_epi64(_mm256_and_si256(X4, mk), _mm256_alignr_epi64(C4, C3, 3)); + X5 = _mm256_add_epi64(_mm256_and_si256(X5, mk), _mm256_alignr_epi64(C5, C4, 3)); + X6 = _mm256_add_epi64(_mm256_and_si256(X6, mk), _mm256_alignr_epi64(C6, C5, 3)); + X7 = _mm256_add_epi64(_mm256_and_si256(X7, mk), _mm256_alignr_epi64(C7, C6, 3)); + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X0, mk) << 0; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X0, mk) << 0; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X1, mk) << 4; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X1, mk) << 4; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X2, mk) << 8; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X2, mk) << 8; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X3, mk) << 12; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X3, mk) << 12; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X4, mk) << 16; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X4, mk) << 16; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X5, mk) << 20; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X5, mk) << 20; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X6, mk) << 24; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X6, mk) << 24; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X7, mk) << 28; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X7, mk) << 28; + c = ((g << 1) + p) ^ p; + X0 = _mm256_and_si256(_mm256_mask_add_epi64(X0, (__mmask8)(c >> 0 & 15), X0, one), mk); + _mm256_storeu_si256((__m256i *)(r + 0), X0); + X1 = _mm256_and_si256(_mm256_mask_add_epi64(X1, (__mmask8)(c >> 4 & 15), X1, one), mk); + _mm256_storeu_si256((__m256i *)(r + 4), X1); + X2 = _mm256_and_si256(_mm256_mask_add_epi64(X2, (__mmask8)(c >> 8 & 15), X2, one), mk); + _mm256_storeu_si256((__m256i *)(r + 8), X2); + X3 = _mm256_and_si256(_mm256_mask_add_epi64(X3, (__mmask8)(c >> 12 & 15), X3, one), mk); + _mm256_storeu_si256((__m256i *)(r + 12), X3); + X4 = _mm256_and_si256(_mm256_mask_add_epi64(X4, (__mmask8)(c >> 16 & 15), X4, one), mk); + _mm256_storeu_si256((__m256i *)(r + 16), X4); + X5 = _mm256_and_si256(_mm256_mask_add_epi64(X5, (__mmask8)(c >> 20 & 15), X5, one), mk); + _mm256_storeu_si256((__m256i *)(r + 20), X5); + X6 = _mm256_and_si256(_mm256_mask_add_epi64(X6, (__mmask8)(c >> 24 & 15), X6, one), mk); + _mm256_storeu_si256((__m256i *)(r + 24), X6); + X7 = _mm256_and_si256(_mm256_mask_add_epi64(X7, (__mmask8)(c >> 28 & 15), X7, one), mk); + _mm256_storeu_si256((__m256i *)(r + 28), X7); +} + +/* both tables' entries k1 and k2 of 32, every entry read and a mask keeping + * one; entries are 32 words apart */ +IFMA_TARGET static void ifma_sel8(uint64_t *o1, const uint64_t *t1, uint64_t k1, + uint64_t *o2, const uint64_t *t2, uint64_t k2) +{ + const __m256i one = _mm256_set1_epi64x(1); + __m256i v1 = _mm256_set1_epi64x((long long) k1), v2 = _mm256_set1_epi64x((long long) k2); + __m256i kv = _mm256_setzero_si256(); + __m256i P0 = kv, Q0 = kv, P1 = kv, Q1 = kv, P2 = kv, Q2 = kv, P3 = kv, Q3 = kv, P4 = kv, Q4 = kv, P5 = kv, Q5 = kv, P6 = kv, Q6 = kv, P7 = kv, Q7 = kv; + int k; + for (k = 0; k < 32; k++) { + __mmask8 s1 = _mm256_cmpeq_epi64_mask(v1, kv), s2 = _mm256_cmpeq_epi64_mask(v2, kv); + const uint64_t *e1 = t1 + 32 * k, *e2 = t2 + 32 * k; + P0 = _mm256_mask_loadu_epi64(P0, s1, e1 + 0); Q0 = _mm256_mask_loadu_epi64(Q0, s2, e2 + 0); + P1 = _mm256_mask_loadu_epi64(P1, s1, e1 + 4); Q1 = _mm256_mask_loadu_epi64(Q1, s2, e2 + 4); + P2 = _mm256_mask_loadu_epi64(P2, s1, e1 + 8); Q2 = _mm256_mask_loadu_epi64(Q2, s2, e2 + 8); + P3 = _mm256_mask_loadu_epi64(P3, s1, e1 + 12); Q3 = _mm256_mask_loadu_epi64(Q3, s2, e2 + 12); + P4 = _mm256_mask_loadu_epi64(P4, s1, e1 + 16); Q4 = _mm256_mask_loadu_epi64(Q4, s2, e2 + 16); + P5 = _mm256_mask_loadu_epi64(P5, s1, e1 + 20); Q5 = _mm256_mask_loadu_epi64(Q5, s2, e2 + 20); + P6 = _mm256_mask_loadu_epi64(P6, s1, e1 + 24); Q6 = _mm256_mask_loadu_epi64(Q6, s2, e2 + 24); + P7 = _mm256_mask_loadu_epi64(P7, s1, e1 + 28); Q7 = _mm256_mask_loadu_epi64(Q7, s2, e2 + 28); + kv = _mm256_add_epi64(kv, one); + } + _mm256_storeu_si256((__m256i *)(o1 + 0), P0); _mm256_storeu_si256((__m256i *)(o2 + 0), Q0); + _mm256_storeu_si256((__m256i *)(o1 + 4), P1); _mm256_storeu_si256((__m256i *)(o2 + 4), Q1); + _mm256_storeu_si256((__m256i *)(o1 + 8), P2); _mm256_storeu_si256((__m256i *)(o2 + 8), Q2); + _mm256_storeu_si256((__m256i *)(o1 + 12), P3); _mm256_storeu_si256((__m256i *)(o2 + 12), Q3); + _mm256_storeu_si256((__m256i *)(o1 + 16), P4); _mm256_storeu_si256((__m256i *)(o2 + 16), Q4); + _mm256_storeu_si256((__m256i *)(o1 + 20), P5); _mm256_storeu_si256((__m256i *)(o2 + 20), Q5); + _mm256_storeu_si256((__m256i *)(o1 + 24), P6); _mm256_storeu_si256((__m256i *)(o2 + 24), Q6); + _mm256_storeu_si256((__m256i *)(o1 + 28), P7); _mm256_storeu_si256((__m256i *)(o2 + 28), Q7); +} + +/* 40 lanes to 52-bit digits: what is above 52 bits moves up a lane, + * and the one-bit carries that can then remain are resolved together */ +IFMA_TARGET static inline void ifma_norm10(uint64_t *r, __m256i X0, __m256i X1, __m256i X2, __m256i X3, __m256i X4, __m256i X5, __m256i X6, __m256i X7, __m256i X8, __m256i X9) +{ + const __m256i mk = _mm256_set1_epi64x((long long) IFMA_MASK52), one = _mm256_set1_epi64x(1); + const __m256i zero = _mm256_setzero_si256(); + uint64_t g = 0, p = 0, c; + __m256i C0 = _mm256_srli_epi64(X0, 52); + __m256i C1 = _mm256_srli_epi64(X1, 52); + __m256i C2 = _mm256_srli_epi64(X2, 52); + __m256i C3 = _mm256_srli_epi64(X3, 52); + __m256i C4 = _mm256_srli_epi64(X4, 52); + __m256i C5 = _mm256_srli_epi64(X5, 52); + __m256i C6 = _mm256_srli_epi64(X6, 52); + __m256i C7 = _mm256_srli_epi64(X7, 52); + __m256i C8 = _mm256_srli_epi64(X8, 52); + __m256i C9 = _mm256_srli_epi64(X9, 52); + X0 = _mm256_add_epi64(_mm256_and_si256(X0, mk), _mm256_alignr_epi64(C0, zero, 3)); + X1 = _mm256_add_epi64(_mm256_and_si256(X1, mk), _mm256_alignr_epi64(C1, C0, 3)); + X2 = _mm256_add_epi64(_mm256_and_si256(X2, mk), _mm256_alignr_epi64(C2, C1, 3)); + X3 = _mm256_add_epi64(_mm256_and_si256(X3, mk), _mm256_alignr_epi64(C3, C2, 3)); + X4 = _mm256_add_epi64(_mm256_and_si256(X4, mk), _mm256_alignr_epi64(C4, C3, 3)); + X5 = _mm256_add_epi64(_mm256_and_si256(X5, mk), _mm256_alignr_epi64(C5, C4, 3)); + X6 = _mm256_add_epi64(_mm256_and_si256(X6, mk), _mm256_alignr_epi64(C6, C5, 3)); + X7 = _mm256_add_epi64(_mm256_and_si256(X7, mk), _mm256_alignr_epi64(C7, C6, 3)); + X8 = _mm256_add_epi64(_mm256_and_si256(X8, mk), _mm256_alignr_epi64(C8, C7, 3)); + X9 = _mm256_add_epi64(_mm256_and_si256(X9, mk), _mm256_alignr_epi64(C9, C8, 3)); + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X0, mk) << 0; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X0, mk) << 0; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X1, mk) << 4; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X1, mk) << 4; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X2, mk) << 8; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X2, mk) << 8; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X3, mk) << 12; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X3, mk) << 12; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X4, mk) << 16; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X4, mk) << 16; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X5, mk) << 20; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X5, mk) << 20; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X6, mk) << 24; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X6, mk) << 24; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X7, mk) << 28; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X7, mk) << 28; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X8, mk) << 32; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X8, mk) << 32; + g |= (uint64_t) _mm256_cmpgt_epu64_mask(X9, mk) << 36; + p |= (uint64_t) _mm256_cmpeq_epu64_mask(X9, mk) << 36; + c = ((g << 1) + p) ^ p; + X0 = _mm256_and_si256(_mm256_mask_add_epi64(X0, (__mmask8)(c >> 0 & 15), X0, one), mk); + _mm256_storeu_si256((__m256i *)(r + 0), X0); + X1 = _mm256_and_si256(_mm256_mask_add_epi64(X1, (__mmask8)(c >> 4 & 15), X1, one), mk); + _mm256_storeu_si256((__m256i *)(r + 4), X1); + X2 = _mm256_and_si256(_mm256_mask_add_epi64(X2, (__mmask8)(c >> 8 & 15), X2, one), mk); + _mm256_storeu_si256((__m256i *)(r + 8), X2); + X3 = _mm256_and_si256(_mm256_mask_add_epi64(X3, (__mmask8)(c >> 12 & 15), X3, one), mk); + _mm256_storeu_si256((__m256i *)(r + 12), X3); + X4 = _mm256_and_si256(_mm256_mask_add_epi64(X4, (__mmask8)(c >> 16 & 15), X4, one), mk); + _mm256_storeu_si256((__m256i *)(r + 16), X4); + X5 = _mm256_and_si256(_mm256_mask_add_epi64(X5, (__mmask8)(c >> 20 & 15), X5, one), mk); + _mm256_storeu_si256((__m256i *)(r + 20), X5); + X6 = _mm256_and_si256(_mm256_mask_add_epi64(X6, (__mmask8)(c >> 24 & 15), X6, one), mk); + _mm256_storeu_si256((__m256i *)(r + 24), X6); + X7 = _mm256_and_si256(_mm256_mask_add_epi64(X7, (__mmask8)(c >> 28 & 15), X7, one), mk); + _mm256_storeu_si256((__m256i *)(r + 28), X7); + X8 = _mm256_and_si256(_mm256_mask_add_epi64(X8, (__mmask8)(c >> 32 & 15), X8, one), mk); + _mm256_storeu_si256((__m256i *)(r + 32), X8); + X9 = _mm256_and_si256(_mm256_mask_add_epi64(X9, (__mmask8)(c >> 36 & 15), X9, one), mk); + _mm256_storeu_si256((__m256i *)(r + 36), X9); +} + +/* both tables' entries k1 and k2 of 32, every entry read and a mask keeping + * one; entries are 40 words apart */ +IFMA_TARGET static void ifma_sel10(uint64_t *o1, const uint64_t *t1, uint64_t k1, + uint64_t *o2, const uint64_t *t2, uint64_t k2) +{ + const __m256i one = _mm256_set1_epi64x(1); + __m256i v1 = _mm256_set1_epi64x((long long) k1), v2 = _mm256_set1_epi64x((long long) k2); + __m256i kv = _mm256_setzero_si256(); + __m256i P0 = kv, Q0 = kv, P1 = kv, Q1 = kv, P2 = kv, Q2 = kv, P3 = kv, Q3 = kv, P4 = kv, Q4 = kv, P5 = kv, Q5 = kv, P6 = kv, Q6 = kv, P7 = kv, Q7 = kv, P8 = kv, Q8 = kv, P9 = kv, Q9 = kv; + int k; + for (k = 0; k < 32; k++) { + __mmask8 s1 = _mm256_cmpeq_epi64_mask(v1, kv), s2 = _mm256_cmpeq_epi64_mask(v2, kv); + const uint64_t *e1 = t1 + 40 * k, *e2 = t2 + 40 * k; + P0 = _mm256_mask_loadu_epi64(P0, s1, e1 + 0); Q0 = _mm256_mask_loadu_epi64(Q0, s2, e2 + 0); + P1 = _mm256_mask_loadu_epi64(P1, s1, e1 + 4); Q1 = _mm256_mask_loadu_epi64(Q1, s2, e2 + 4); + P2 = _mm256_mask_loadu_epi64(P2, s1, e1 + 8); Q2 = _mm256_mask_loadu_epi64(Q2, s2, e2 + 8); + P3 = _mm256_mask_loadu_epi64(P3, s1, e1 + 12); Q3 = _mm256_mask_loadu_epi64(Q3, s2, e2 + 12); + P4 = _mm256_mask_loadu_epi64(P4, s1, e1 + 16); Q4 = _mm256_mask_loadu_epi64(Q4, s2, e2 + 16); + P5 = _mm256_mask_loadu_epi64(P5, s1, e1 + 20); Q5 = _mm256_mask_loadu_epi64(Q5, s2, e2 + 20); + P6 = _mm256_mask_loadu_epi64(P6, s1, e1 + 24); Q6 = _mm256_mask_loadu_epi64(Q6, s2, e2 + 24); + P7 = _mm256_mask_loadu_epi64(P7, s1, e1 + 28); Q7 = _mm256_mask_loadu_epi64(Q7, s2, e2 + 28); + P8 = _mm256_mask_loadu_epi64(P8, s1, e1 + 32); Q8 = _mm256_mask_loadu_epi64(Q8, s2, e2 + 32); + P9 = _mm256_mask_loadu_epi64(P9, s1, e1 + 36); Q9 = _mm256_mask_loadu_epi64(Q9, s2, e2 + 36); + kv = _mm256_add_epi64(kv, one); + } + _mm256_storeu_si256((__m256i *)(o1 + 0), P0); _mm256_storeu_si256((__m256i *)(o2 + 0), Q0); + _mm256_storeu_si256((__m256i *)(o1 + 4), P1); _mm256_storeu_si256((__m256i *)(o2 + 4), Q1); + _mm256_storeu_si256((__m256i *)(o1 + 8), P2); _mm256_storeu_si256((__m256i *)(o2 + 8), Q2); + _mm256_storeu_si256((__m256i *)(o1 + 12), P3); _mm256_storeu_si256((__m256i *)(o2 + 12), Q3); + _mm256_storeu_si256((__m256i *)(o1 + 16), P4); _mm256_storeu_si256((__m256i *)(o2 + 16), Q4); + _mm256_storeu_si256((__m256i *)(o1 + 20), P5); _mm256_storeu_si256((__m256i *)(o2 + 20), Q5); + _mm256_storeu_si256((__m256i *)(o1 + 24), P6); _mm256_storeu_si256((__m256i *)(o2 + 24), Q6); + _mm256_storeu_si256((__m256i *)(o1 + 28), P7); _mm256_storeu_si256((__m256i *)(o2 + 28), Q7); + _mm256_storeu_si256((__m256i *)(o1 + 32), P8); _mm256_storeu_si256((__m256i *)(o2 + 32), Q8); + _mm256_storeu_si256((__m256i *)(o1 + 36), P9); _mm256_storeu_si256((__m256i *)(o2 + 36), Q9); +} + +/* 20 digits, 5 registers a number */ +IFMA_TARGET static void ifma_amm20_x2(uint64_t *r1, const uint64_t *a1, const uint64_t *b1, + const uint64_t *m1, uint64_t k1, + uint64_t *r2, const uint64_t *a2, const uint64_t *b2, + const uint64_t *m2, uint64_t k2) +{ + const __m256i zero = _mm256_setzero_si256(); +#define L(p, k) _mm256_loadu_si256((const __m256i *)((p) + 4 * (k))) + __m256i X0 = zero, X1 = zero, X2 = zero, X3 = zero, X4 = zero, Y0 = zero, Y1 = zero, Y2 = zero, Y3 = zero, Y4 = zero; + __m256i A0 = L(a1, 0), A1 = L(a1, 1), A2 = L(a1, 2), A3 = L(a1, 3), A4 = L(a1, 4); + __m256i B0 = L(a2, 0), B1 = L(a2, 1), B2 = L(a2, 2), B3 = L(a2, 3), B4 = L(a2, 4); + uint64_t x1 = 0, x2 = 0; + int i; + + for (i = 0; i < 20; i++) { + uint64_t bi = b1[i], ci = b2[i], y, z; + /* lane 0 in full: the products' high halves, which are lane 1's, too */ + unsigned __int128 s1 = (unsigned __int128) a1[0] * bi + x1; + unsigned __int128 s2 = (unsigned __int128) a2[0] * ci + x2; + __m256i bv = _mm256_set1_epi64x((long long) bi), cv = _mm256_set1_epi64x((long long) ci); + __m256i yv, zv; + + y = ((uint64_t) s1 * k1) & IFMA_MASK52; + z = ((uint64_t) s2 * k2) & IFMA_MASK52; + s1 += (unsigned __int128) m1[0] * y; + s2 += (unsigned __int128) m2[0] * z; + yv = _mm256_set1_epi64x((long long) y); + zv = _mm256_set1_epi64x((long long) z); + X0 = _mm256_madd52lo_epu64(X0, A0, bv); Y0 = _mm256_madd52lo_epu64(Y0, B0, cv); + X1 = _mm256_madd52lo_epu64(X1, A1, bv); Y1 = _mm256_madd52lo_epu64(Y1, B1, cv); + X2 = _mm256_madd52lo_epu64(X2, A2, bv); Y2 = _mm256_madd52lo_epu64(Y2, B2, cv); + X3 = _mm256_madd52lo_epu64(X3, A3, bv); Y3 = _mm256_madd52lo_epu64(Y3, B3, cv); + X4 = _mm256_madd52lo_epu64(X4, A4, bv); Y4 = _mm256_madd52lo_epu64(Y4, B4, cv); + X0 = _mm256_madd52lo_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52lo_epu64(Y0, L(m2, 0), zv); + X1 = _mm256_madd52lo_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52lo_epu64(Y1, L(m2, 1), zv); + X2 = _mm256_madd52lo_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52lo_epu64(Y2, L(m2, 2), zv); + X3 = _mm256_madd52lo_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52lo_epu64(Y3, L(m2, 3), zv); + X4 = _mm256_madd52lo_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52lo_epu64(Y4, L(m2, 4), zv); + /* down a lane: lane 0 is a multiple of 2^52, and s carries it */ + X0 = _mm256_alignr_epi64(X1, X0, 1); Y0 = _mm256_alignr_epi64(Y1, Y0, 1); + X1 = _mm256_alignr_epi64(X2, X1, 1); Y1 = _mm256_alignr_epi64(Y2, Y1, 1); + X2 = _mm256_alignr_epi64(X3, X2, 1); Y2 = _mm256_alignr_epi64(Y3, Y2, 1); + X3 = _mm256_alignr_epi64(X4, X3, 1); Y3 = _mm256_alignr_epi64(Y4, Y3, 1); + X4 = _mm256_alignr_epi64(zero, X4, 1); Y4 = _mm256_alignr_epi64(zero, Y4, 1); + x1 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(X0)) + (uint64_t)(s1 >> 52); + x2 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(Y0)) + (uint64_t)(s2 >> 52); + X0 = _mm256_madd52hi_epu64(X0, A0, bv); Y0 = _mm256_madd52hi_epu64(Y0, B0, cv); + X1 = _mm256_madd52hi_epu64(X1, A1, bv); Y1 = _mm256_madd52hi_epu64(Y1, B1, cv); + X2 = _mm256_madd52hi_epu64(X2, A2, bv); Y2 = _mm256_madd52hi_epu64(Y2, B2, cv); + X3 = _mm256_madd52hi_epu64(X3, A3, bv); Y3 = _mm256_madd52hi_epu64(Y3, B3, cv); + X4 = _mm256_madd52hi_epu64(X4, A4, bv); Y4 = _mm256_madd52hi_epu64(Y4, B4, cv); + X0 = _mm256_madd52hi_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52hi_epu64(Y0, L(m2, 0), zv); + X1 = _mm256_madd52hi_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52hi_epu64(Y1, L(m2, 1), zv); + X2 = _mm256_madd52hi_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52hi_epu64(Y2, L(m2, 2), zv); + X3 = _mm256_madd52hi_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52hi_epu64(Y3, L(m2, 3), zv); + X4 = _mm256_madd52hi_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52hi_epu64(Y4, L(m2, 4), zv); + } +#undef L + /* lane 0, with the carry it was never given, is what the scalar holds */ + X0 = _mm256_mask_set1_epi64(X0, 1, (long long) x1); + Y0 = _mm256_mask_set1_epi64(Y0, 1, (long long) x2); + ifma_norm5(r1, X0, X1, X2, X3, X4); + ifma_norm5(r2, Y0, Y1, Y2, Y3, Y4); +} + +/* 30 digits, 8 registers a number */ +IFMA_TARGET static void ifma_amm30_x2(uint64_t *r1, const uint64_t *a1, const uint64_t *b1, + const uint64_t *m1, uint64_t k1, + uint64_t *r2, const uint64_t *a2, const uint64_t *b2, + const uint64_t *m2, uint64_t k2) +{ + const __m256i zero = _mm256_setzero_si256(); +#define L(p, k) _mm256_loadu_si256((const __m256i *)((p) + 4 * (k))) + __m256i X0 = zero, X1 = zero, X2 = zero, X3 = zero, X4 = zero, X5 = zero, X6 = zero, X7 = zero, Y0 = zero, Y1 = zero, Y2 = zero, Y3 = zero, Y4 = zero, Y5 = zero, Y6 = zero, Y7 = zero; + uint64_t x1 = 0, x2 = 0; + int i; + + for (i = 0; i < 30; i++) { + uint64_t bi = b1[i], ci = b2[i], y, z; + /* lane 0 in full: the products' high halves, which are lane 1's, too */ + unsigned __int128 s1 = (unsigned __int128) a1[0] * bi + x1; + unsigned __int128 s2 = (unsigned __int128) a2[0] * ci + x2; + __m256i bv = _mm256_set1_epi64x((long long) bi), cv = _mm256_set1_epi64x((long long) ci); + __m256i yv, zv; + + y = ((uint64_t) s1 * k1) & IFMA_MASK52; + z = ((uint64_t) s2 * k2) & IFMA_MASK52; + s1 += (unsigned __int128) m1[0] * y; + s2 += (unsigned __int128) m2[0] * z; + yv = _mm256_set1_epi64x((long long) y); + zv = _mm256_set1_epi64x((long long) z); + X0 = _mm256_madd52lo_epu64(X0, L(a1, 0), bv); Y0 = _mm256_madd52lo_epu64(Y0, L(a2, 0), cv); + X1 = _mm256_madd52lo_epu64(X1, L(a1, 1), bv); Y1 = _mm256_madd52lo_epu64(Y1, L(a2, 1), cv); + X2 = _mm256_madd52lo_epu64(X2, L(a1, 2), bv); Y2 = _mm256_madd52lo_epu64(Y2, L(a2, 2), cv); + X3 = _mm256_madd52lo_epu64(X3, L(a1, 3), bv); Y3 = _mm256_madd52lo_epu64(Y3, L(a2, 3), cv); + X4 = _mm256_madd52lo_epu64(X4, L(a1, 4), bv); Y4 = _mm256_madd52lo_epu64(Y4, L(a2, 4), cv); + X5 = _mm256_madd52lo_epu64(X5, L(a1, 5), bv); Y5 = _mm256_madd52lo_epu64(Y5, L(a2, 5), cv); + X6 = _mm256_madd52lo_epu64(X6, L(a1, 6), bv); Y6 = _mm256_madd52lo_epu64(Y6, L(a2, 6), cv); + X7 = _mm256_madd52lo_epu64(X7, L(a1, 7), bv); Y7 = _mm256_madd52lo_epu64(Y7, L(a2, 7), cv); + X0 = _mm256_madd52lo_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52lo_epu64(Y0, L(m2, 0), zv); + X1 = _mm256_madd52lo_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52lo_epu64(Y1, L(m2, 1), zv); + X2 = _mm256_madd52lo_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52lo_epu64(Y2, L(m2, 2), zv); + X3 = _mm256_madd52lo_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52lo_epu64(Y3, L(m2, 3), zv); + X4 = _mm256_madd52lo_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52lo_epu64(Y4, L(m2, 4), zv); + X5 = _mm256_madd52lo_epu64(X5, L(m1, 5), yv); Y5 = _mm256_madd52lo_epu64(Y5, L(m2, 5), zv); + X6 = _mm256_madd52lo_epu64(X6, L(m1, 6), yv); Y6 = _mm256_madd52lo_epu64(Y6, L(m2, 6), zv); + X7 = _mm256_madd52lo_epu64(X7, L(m1, 7), yv); Y7 = _mm256_madd52lo_epu64(Y7, L(m2, 7), zv); + /* down a lane: lane 0 is a multiple of 2^52, and s carries it */ + X0 = _mm256_alignr_epi64(X1, X0, 1); Y0 = _mm256_alignr_epi64(Y1, Y0, 1); + X1 = _mm256_alignr_epi64(X2, X1, 1); Y1 = _mm256_alignr_epi64(Y2, Y1, 1); + X2 = _mm256_alignr_epi64(X3, X2, 1); Y2 = _mm256_alignr_epi64(Y3, Y2, 1); + X3 = _mm256_alignr_epi64(X4, X3, 1); Y3 = _mm256_alignr_epi64(Y4, Y3, 1); + X4 = _mm256_alignr_epi64(X5, X4, 1); Y4 = _mm256_alignr_epi64(Y5, Y4, 1); + X5 = _mm256_alignr_epi64(X6, X5, 1); Y5 = _mm256_alignr_epi64(Y6, Y5, 1); + X6 = _mm256_alignr_epi64(X7, X6, 1); Y6 = _mm256_alignr_epi64(Y7, Y6, 1); + X7 = _mm256_alignr_epi64(zero, X7, 1); Y7 = _mm256_alignr_epi64(zero, Y7, 1); + x1 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(X0)) + (uint64_t)(s1 >> 52); + x2 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(Y0)) + (uint64_t)(s2 >> 52); + X0 = _mm256_madd52hi_epu64(X0, L(a1, 0), bv); Y0 = _mm256_madd52hi_epu64(Y0, L(a2, 0), cv); + X1 = _mm256_madd52hi_epu64(X1, L(a1, 1), bv); Y1 = _mm256_madd52hi_epu64(Y1, L(a2, 1), cv); + X2 = _mm256_madd52hi_epu64(X2, L(a1, 2), bv); Y2 = _mm256_madd52hi_epu64(Y2, L(a2, 2), cv); + X3 = _mm256_madd52hi_epu64(X3, L(a1, 3), bv); Y3 = _mm256_madd52hi_epu64(Y3, L(a2, 3), cv); + X4 = _mm256_madd52hi_epu64(X4, L(a1, 4), bv); Y4 = _mm256_madd52hi_epu64(Y4, L(a2, 4), cv); + X5 = _mm256_madd52hi_epu64(X5, L(a1, 5), bv); Y5 = _mm256_madd52hi_epu64(Y5, L(a2, 5), cv); + X6 = _mm256_madd52hi_epu64(X6, L(a1, 6), bv); Y6 = _mm256_madd52hi_epu64(Y6, L(a2, 6), cv); + X7 = _mm256_madd52hi_epu64(X7, L(a1, 7), bv); Y7 = _mm256_madd52hi_epu64(Y7, L(a2, 7), cv); + X0 = _mm256_madd52hi_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52hi_epu64(Y0, L(m2, 0), zv); + X1 = _mm256_madd52hi_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52hi_epu64(Y1, L(m2, 1), zv); + X2 = _mm256_madd52hi_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52hi_epu64(Y2, L(m2, 2), zv); + X3 = _mm256_madd52hi_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52hi_epu64(Y3, L(m2, 3), zv); + X4 = _mm256_madd52hi_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52hi_epu64(Y4, L(m2, 4), zv); + X5 = _mm256_madd52hi_epu64(X5, L(m1, 5), yv); Y5 = _mm256_madd52hi_epu64(Y5, L(m2, 5), zv); + X6 = _mm256_madd52hi_epu64(X6, L(m1, 6), yv); Y6 = _mm256_madd52hi_epu64(Y6, L(m2, 6), zv); + X7 = _mm256_madd52hi_epu64(X7, L(m1, 7), yv); Y7 = _mm256_madd52hi_epu64(Y7, L(m2, 7), zv); + } +#undef L + /* lane 0, with the carry it was never given, is what the scalar holds */ + X0 = _mm256_mask_set1_epi64(X0, 1, (long long) x1); + Y0 = _mm256_mask_set1_epi64(Y0, 1, (long long) x2); + ifma_norm8(r1, X0, X1, X2, X3, X4, X5, X6, X7); + ifma_norm8(r2, Y0, Y1, Y2, Y3, Y4, Y5, Y6, Y7); +} + +/* 40 digits, 10 registers a number */ +IFMA_TARGET static void ifma_amm40_x2(uint64_t *r1, const uint64_t *a1, const uint64_t *b1, + const uint64_t *m1, uint64_t k1, + uint64_t *r2, const uint64_t *a2, const uint64_t *b2, + const uint64_t *m2, uint64_t k2) +{ + const __m256i zero = _mm256_setzero_si256(); +#define L(p, k) _mm256_loadu_si256((const __m256i *)((p) + 4 * (k))) + __m256i X0 = zero, X1 = zero, X2 = zero, X3 = zero, X4 = zero, X5 = zero, X6 = zero, X7 = zero, X8 = zero, X9 = zero, Y0 = zero, Y1 = zero, Y2 = zero, Y3 = zero, Y4 = zero, Y5 = zero, Y6 = zero, Y7 = zero, Y8 = zero, Y9 = zero; + uint64_t x1 = 0, x2 = 0; + int i; + + for (i = 0; i < 40; i++) { + uint64_t bi = b1[i], ci = b2[i], y, z; + /* lane 0 in full: the products' high halves, which are lane 1's, too */ + unsigned __int128 s1 = (unsigned __int128) a1[0] * bi + x1; + unsigned __int128 s2 = (unsigned __int128) a2[0] * ci + x2; + __m256i bv = _mm256_set1_epi64x((long long) bi), cv = _mm256_set1_epi64x((long long) ci); + __m256i yv, zv; + + y = ((uint64_t) s1 * k1) & IFMA_MASK52; + z = ((uint64_t) s2 * k2) & IFMA_MASK52; + s1 += (unsigned __int128) m1[0] * y; + s2 += (unsigned __int128) m2[0] * z; + yv = _mm256_set1_epi64x((long long) y); + zv = _mm256_set1_epi64x((long long) z); + X0 = _mm256_madd52lo_epu64(X0, L(a1, 0), bv); Y0 = _mm256_madd52lo_epu64(Y0, L(a2, 0), cv); + X1 = _mm256_madd52lo_epu64(X1, L(a1, 1), bv); Y1 = _mm256_madd52lo_epu64(Y1, L(a2, 1), cv); + X2 = _mm256_madd52lo_epu64(X2, L(a1, 2), bv); Y2 = _mm256_madd52lo_epu64(Y2, L(a2, 2), cv); + X3 = _mm256_madd52lo_epu64(X3, L(a1, 3), bv); Y3 = _mm256_madd52lo_epu64(Y3, L(a2, 3), cv); + X4 = _mm256_madd52lo_epu64(X4, L(a1, 4), bv); Y4 = _mm256_madd52lo_epu64(Y4, L(a2, 4), cv); + X5 = _mm256_madd52lo_epu64(X5, L(a1, 5), bv); Y5 = _mm256_madd52lo_epu64(Y5, L(a2, 5), cv); + X6 = _mm256_madd52lo_epu64(X6, L(a1, 6), bv); Y6 = _mm256_madd52lo_epu64(Y6, L(a2, 6), cv); + X7 = _mm256_madd52lo_epu64(X7, L(a1, 7), bv); Y7 = _mm256_madd52lo_epu64(Y7, L(a2, 7), cv); + X8 = _mm256_madd52lo_epu64(X8, L(a1, 8), bv); Y8 = _mm256_madd52lo_epu64(Y8, L(a2, 8), cv); + X9 = _mm256_madd52lo_epu64(X9, L(a1, 9), bv); Y9 = _mm256_madd52lo_epu64(Y9, L(a2, 9), cv); + X0 = _mm256_madd52lo_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52lo_epu64(Y0, L(m2, 0), zv); + X1 = _mm256_madd52lo_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52lo_epu64(Y1, L(m2, 1), zv); + X2 = _mm256_madd52lo_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52lo_epu64(Y2, L(m2, 2), zv); + X3 = _mm256_madd52lo_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52lo_epu64(Y3, L(m2, 3), zv); + X4 = _mm256_madd52lo_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52lo_epu64(Y4, L(m2, 4), zv); + X5 = _mm256_madd52lo_epu64(X5, L(m1, 5), yv); Y5 = _mm256_madd52lo_epu64(Y5, L(m2, 5), zv); + X6 = _mm256_madd52lo_epu64(X6, L(m1, 6), yv); Y6 = _mm256_madd52lo_epu64(Y6, L(m2, 6), zv); + X7 = _mm256_madd52lo_epu64(X7, L(m1, 7), yv); Y7 = _mm256_madd52lo_epu64(Y7, L(m2, 7), zv); + X8 = _mm256_madd52lo_epu64(X8, L(m1, 8), yv); Y8 = _mm256_madd52lo_epu64(Y8, L(m2, 8), zv); + X9 = _mm256_madd52lo_epu64(X9, L(m1, 9), yv); Y9 = _mm256_madd52lo_epu64(Y9, L(m2, 9), zv); + /* down a lane: lane 0 is a multiple of 2^52, and s carries it */ + X0 = _mm256_alignr_epi64(X1, X0, 1); Y0 = _mm256_alignr_epi64(Y1, Y0, 1); + X1 = _mm256_alignr_epi64(X2, X1, 1); Y1 = _mm256_alignr_epi64(Y2, Y1, 1); + X2 = _mm256_alignr_epi64(X3, X2, 1); Y2 = _mm256_alignr_epi64(Y3, Y2, 1); + X3 = _mm256_alignr_epi64(X4, X3, 1); Y3 = _mm256_alignr_epi64(Y4, Y3, 1); + X4 = _mm256_alignr_epi64(X5, X4, 1); Y4 = _mm256_alignr_epi64(Y5, Y4, 1); + X5 = _mm256_alignr_epi64(X6, X5, 1); Y5 = _mm256_alignr_epi64(Y6, Y5, 1); + X6 = _mm256_alignr_epi64(X7, X6, 1); Y6 = _mm256_alignr_epi64(Y7, Y6, 1); + X7 = _mm256_alignr_epi64(X8, X7, 1); Y7 = _mm256_alignr_epi64(Y8, Y7, 1); + X8 = _mm256_alignr_epi64(X9, X8, 1); Y8 = _mm256_alignr_epi64(Y9, Y8, 1); + X9 = _mm256_alignr_epi64(zero, X9, 1); Y9 = _mm256_alignr_epi64(zero, Y9, 1); + x1 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(X0)) + (uint64_t)(s1 >> 52); + x2 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(Y0)) + (uint64_t)(s2 >> 52); + X0 = _mm256_madd52hi_epu64(X0, L(a1, 0), bv); Y0 = _mm256_madd52hi_epu64(Y0, L(a2, 0), cv); + X1 = _mm256_madd52hi_epu64(X1, L(a1, 1), bv); Y1 = _mm256_madd52hi_epu64(Y1, L(a2, 1), cv); + X2 = _mm256_madd52hi_epu64(X2, L(a1, 2), bv); Y2 = _mm256_madd52hi_epu64(Y2, L(a2, 2), cv); + X3 = _mm256_madd52hi_epu64(X3, L(a1, 3), bv); Y3 = _mm256_madd52hi_epu64(Y3, L(a2, 3), cv); + X4 = _mm256_madd52hi_epu64(X4, L(a1, 4), bv); Y4 = _mm256_madd52hi_epu64(Y4, L(a2, 4), cv); + X5 = _mm256_madd52hi_epu64(X5, L(a1, 5), bv); Y5 = _mm256_madd52hi_epu64(Y5, L(a2, 5), cv); + X6 = _mm256_madd52hi_epu64(X6, L(a1, 6), bv); Y6 = _mm256_madd52hi_epu64(Y6, L(a2, 6), cv); + X7 = _mm256_madd52hi_epu64(X7, L(a1, 7), bv); Y7 = _mm256_madd52hi_epu64(Y7, L(a2, 7), cv); + X8 = _mm256_madd52hi_epu64(X8, L(a1, 8), bv); Y8 = _mm256_madd52hi_epu64(Y8, L(a2, 8), cv); + X9 = _mm256_madd52hi_epu64(X9, L(a1, 9), bv); Y9 = _mm256_madd52hi_epu64(Y9, L(a2, 9), cv); + X0 = _mm256_madd52hi_epu64(X0, L(m1, 0), yv); Y0 = _mm256_madd52hi_epu64(Y0, L(m2, 0), zv); + X1 = _mm256_madd52hi_epu64(X1, L(m1, 1), yv); Y1 = _mm256_madd52hi_epu64(Y1, L(m2, 1), zv); + X2 = _mm256_madd52hi_epu64(X2, L(m1, 2), yv); Y2 = _mm256_madd52hi_epu64(Y2, L(m2, 2), zv); + X3 = _mm256_madd52hi_epu64(X3, L(m1, 3), yv); Y3 = _mm256_madd52hi_epu64(Y3, L(m2, 3), zv); + X4 = _mm256_madd52hi_epu64(X4, L(m1, 4), yv); Y4 = _mm256_madd52hi_epu64(Y4, L(m2, 4), zv); + X5 = _mm256_madd52hi_epu64(X5, L(m1, 5), yv); Y5 = _mm256_madd52hi_epu64(Y5, L(m2, 5), zv); + X6 = _mm256_madd52hi_epu64(X6, L(m1, 6), yv); Y6 = _mm256_madd52hi_epu64(Y6, L(m2, 6), zv); + X7 = _mm256_madd52hi_epu64(X7, L(m1, 7), yv); Y7 = _mm256_madd52hi_epu64(Y7, L(m2, 7), zv); + X8 = _mm256_madd52hi_epu64(X8, L(m1, 8), yv); Y8 = _mm256_madd52hi_epu64(Y8, L(m2, 8), zv); + X9 = _mm256_madd52hi_epu64(X9, L(m1, 9), yv); Y9 = _mm256_madd52hi_epu64(Y9, L(m2, 9), zv); + } +#undef L + /* lane 0, with the carry it was never given, is what the scalar holds */ + X0 = _mm256_mask_set1_epi64(X0, 1, (long long) x1); + Y0 = _mm256_mask_set1_epi64(Y0, 1, (long long) x2); + ifma_norm10(r1, X0, X1, X2, X3, X4, X5, X6, X7, X8, X9); + ifma_norm10(r2, Y0, Y1, Y2, Y3, Y4, Y5, Y6, Y7, Y8, Y9); +} + +#endif diff --git a/cbits/crypton_powm_ifma.py b/cbits/crypton_powm_ifma.py new file mode 100644 --- /dev/null +++ b/cbits/crypton_powm_ifma.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +"""Generate cbits/crypton_powm_ifma.h: two almost-Montgomery multiplications +at once in radix 2^52 with AVX-512 IFMA on 256-bit registers, at 20, 30 and +40 digits. Usage: python3 cbits/crypton_powm_ifma.py > cbits/crypton_powm_ifma.h + +Gueron and Krasnov's method, a digit of b at a time. Lane 0 is followed in +full in a scalar, so it is read before the high halves go in and they stay +off the chain the next digit waits on. +""" + +SIZES = (20, 30, 40) + +HEADER = """\ +/* Generated by cbits/crypton_powm_ifma.py; do not edit. + * + * Almost-Montgomery multiplication in radix 2^52, two at once: r = a b + * 2^-52D mod m, under 2m for a and b under 2m. Digits are 52 bits in 64-bit + * words, each number padded with zero words to a whole number of 256-bit + * registers. m is read from memory; at 20 digits a is held in registers, + * above that it is read from memory too. */ +#ifndef CRYPTON_POWM_IFMA_H +#define CRYPTON_POWM_IFMA_H + +#include +#include + +#define IFMA_MASK52 ((UINT64_C(1) << 52) - 1) +#define IFMA_TARGET __attribute__((target("avx2,avx512f,avx512vl,avx512ifma"))) +""" + +FOOTER = """\ +#endif +""" + + +def regs(d): + return (d + 3) // 4 + + +def gen_norm(R): + o = [] + w = o.append + xs = [f"X{j}" for j in range(R)] + w(f"/* {4 * R} lanes to 52-bit digits: what is above 52 bits moves up a lane,") + w(f" * and the one-bit carries that can then remain are resolved together */") + w(f"IFMA_TARGET static inline void ifma_norm{R}(uint64_t *r, " + + ", ".join(f"__m256i {x}" for x in xs) + ")") + w("{") + w("\tconst __m256i mk = _mm256_set1_epi64x((long long) IFMA_MASK52), one = _mm256_set1_epi64x(1);") + w("\tconst __m256i zero = _mm256_setzero_si256();") + w("\tuint64_t g = 0, p = 0, c;") + for j in range(R): + w(f"\t__m256i C{j} = _mm256_srli_epi64(X{j}, 52);") + for j in range(R): + prev = f"C{j - 1}" if j else "zero" + w(f"\tX{j} = _mm256_add_epi64(_mm256_and_si256(X{j}, mk), _mm256_alignr_epi64(C{j}, {prev}, 3));") + for j in range(R): + w(f"\tg |= (uint64_t) _mm256_cmpgt_epu64_mask(X{j}, mk) << {4 * j};") + w(f"\tp |= (uint64_t) _mm256_cmpeq_epu64_mask(X{j}, mk) << {4 * j};") + w("\tc = ((g << 1) + p) ^ p;") + for j in range(R): + w(f"\tX{j} = _mm256_and_si256(_mm256_mask_add_epi64(X{j}, (__mmask8)(c >> {4 * j} & 15), X{j}, one), mk);") + w(f"\t_mm256_storeu_si256((__m256i *)(r + {4 * j}), X{j});") + w("}") + w("") + return "\n".join(o) + + +def gen_amm(d): + R = regs(d) + in_regs = R <= 5 + o = [] + w = o.append + w(f"/* {d} digits, {R} registers a number */") + w(f"IFMA_TARGET static void ifma_amm{d}_x2(uint64_t *r1, const uint64_t *a1, const uint64_t *b1,") + w(f" const uint64_t *m1, uint64_t k1,") + w(f" uint64_t *r2, const uint64_t *a2, const uint64_t *b2,") + w(f" const uint64_t *m2, uint64_t k2)") + w("{") + w("\tconst __m256i zero = _mm256_setzero_si256();") + w("#define L(p, k) _mm256_loadu_si256((const __m256i *)((p) + 4 * (k)))") + X = [f"X{j}" for j in range(R)] + Y = [f"Y{j}" for j in range(R)] + w("\t__m256i " + ", ".join(f"{v} = zero" for v in X + Y) + ";") + if in_regs: + w("\t__m256i " + ", ".join(f"A{j} = L(a1, {j})" for j in range(R)) + ";") + w("\t__m256i " + ", ".join(f"B{j} = L(a2, {j})" for j in range(R)) + ";") + A = [f"A{j}" for j in range(R)] + B = [f"B{j}" for j in range(R)] + else: + A = [f"L(a1, {j})" for j in range(R)] + B = [f"L(a2, {j})" for j in range(R)] + M = [f"L(m1, {j})" for j in range(R)] + N = [f"L(m2, {j})" for j in range(R)] + w("\tuint64_t x1 = 0, x2 = 0;") + w("\tint i;") + w("") + w(f"\tfor (i = 0; i < {d}; i++) {{") + w("\t\tuint64_t bi = b1[i], ci = b2[i], y, z;") + w("\t\t/* lane 0 in full: the products' high halves, which are lane 1's, too */") + w("\t\tunsigned __int128 s1 = (unsigned __int128) a1[0] * bi + x1;") + w("\t\tunsigned __int128 s2 = (unsigned __int128) a2[0] * ci + x2;") + w("\t\t__m256i bv = _mm256_set1_epi64x((long long) bi), cv = _mm256_set1_epi64x((long long) ci);") + w("\t\t__m256i yv, zv;") + w("") + w("\t\ty = ((uint64_t) s1 * k1) & IFMA_MASK52;") + w("\t\tz = ((uint64_t) s2 * k2) & IFMA_MASK52;") + w("\t\ts1 += (unsigned __int128) m1[0] * y;") + w("\t\ts2 += (unsigned __int128) m2[0] * z;") + w("\t\tyv = _mm256_set1_epi64x((long long) y);") + w("\t\tzv = _mm256_set1_epi64x((long long) z);") + for j in range(R): + w(f"\t\tX{j} = _mm256_madd52lo_epu64(X{j}, {A[j]}, bv); Y{j} = _mm256_madd52lo_epu64(Y{j}, {B[j]}, cv);") + for j in range(R): + w(f"\t\tX{j} = _mm256_madd52lo_epu64(X{j}, {M[j]}, yv); Y{j} = _mm256_madd52lo_epu64(Y{j}, {N[j]}, zv);") + w("\t\t/* down a lane: lane 0 is a multiple of 2^52, and s carries it */") + for j in range(R): + nx = f"X{j + 1}" if j + 1 < R else "zero" + ny = f"Y{j + 1}" if j + 1 < R else "zero" + w(f"\t\tX{j} = _mm256_alignr_epi64({nx}, X{j}, 1); Y{j} = _mm256_alignr_epi64({ny}, Y{j}, 1);") + w("\t\tx1 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(X0)) + (uint64_t)(s1 >> 52);") + w("\t\tx2 = (uint64_t) _mm_cvtsi128_si64(_mm256_castsi256_si128(Y0)) + (uint64_t)(s2 >> 52);") + for j in range(R): + w(f"\t\tX{j} = _mm256_madd52hi_epu64(X{j}, {A[j]}, bv); Y{j} = _mm256_madd52hi_epu64(Y{j}, {B[j]}, cv);") + for j in range(R): + w(f"\t\tX{j} = _mm256_madd52hi_epu64(X{j}, {M[j]}, yv); Y{j} = _mm256_madd52hi_epu64(Y{j}, {N[j]}, zv);") + w("\t}") + w("#undef L") + w("\t/* lane 0, with the carry it was never given, is what the scalar holds */") + w("\tX0 = _mm256_mask_set1_epi64(X0, 1, (long long) x1);") + w("\tY0 = _mm256_mask_set1_epi64(Y0, 1, (long long) x2);") + w(f"\tifma_norm{R}(r1, " + ", ".join(X) + ");") + w(f"\tifma_norm{R}(r2, " + ", ".join(Y) + ");") + w("}") + w("") + return "\n".join(o) + + +def gen_sel(R): + o = [] + w = o.append + w(f"/* both tables' entries k1 and k2 of 32, every entry read and a mask keeping") + w(f" * one; entries are {4 * R} words apart */") + w(f"IFMA_TARGET static void ifma_sel{R}(uint64_t *o1, const uint64_t *t1, uint64_t k1,") + w(f" uint64_t *o2, const uint64_t *t2, uint64_t k2)") + w("{") + w("\tconst __m256i one = _mm256_set1_epi64x(1);") + w("\t__m256i v1 = _mm256_set1_epi64x((long long) k1), v2 = _mm256_set1_epi64x((long long) k2);") + w("\t__m256i kv = _mm256_setzero_si256();") + w("\t__m256i " + ", ".join(f"P{j} = kv, Q{j} = kv" for j in range(R)) + ";") + w("\tint k;") + w("\tfor (k = 0; k < 32; k++) {") + w("\t\t__mmask8 s1 = _mm256_cmpeq_epi64_mask(v1, kv), s2 = _mm256_cmpeq_epi64_mask(v2, kv);") + w(f"\t\tconst uint64_t *e1 = t1 + {4 * R} * k, *e2 = t2 + {4 * R} * k;") + for j in range(R): + w(f"\t\tP{j} = _mm256_mask_loadu_epi64(P{j}, s1, e1 + {4 * j}); Q{j} = _mm256_mask_loadu_epi64(Q{j}, s2, e2 + {4 * j});") + w("\t\tkv = _mm256_add_epi64(kv, one);") + w("\t}") + for j in range(R): + w(f"\t_mm256_storeu_si256((__m256i *)(o1 + {4 * j}), P{j}); _mm256_storeu_si256((__m256i *)(o2 + {4 * j}), Q{j});") + w("}") + w("") + return "\n".join(o) + + +def main(): + print(HEADER) + rs = sorted({regs(d) for d in SIZES}) + for R in rs: + print(gen_norm(R)) + print(gen_sel(R)) + for d in SIZES: + print(gen_amm(d)) + print(FOOTER, end="") + + +if __name__ == "__main__": + main() diff --git a/cbits/crypton_rdrand.c b/cbits/crypton_rdrand.c deleted file mode 100644 --- a/cbits/crypton_rdrand.c +++ /dev/null @@ -1,126 +0,0 @@ -/* - * Copyright (C) Thomas DuBuisson - * Copyright (C) 2013 Vincent Hanquez - * - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * 3. Neither the name of the author nor the names of his contributors - * may be used to endorse or promote products derived from this software - * without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE - * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL - * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS - * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) - * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY - * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF - * SUCH DAMAGE. - */ - -#include -#include -#include -#include - -int crypton_cpu_has_rdrand() -{ - uint32_t ax,bx,cx,dx,func=1; -#if defined(__PIC__) && defined(__i386__) - __asm__ volatile ("mov %%ebx, %%edi;" "cpuid;" "xchgl %%ebx, %%edi;" - : "=a" (ax), "=D" (bx), "=c" (cx), "=d" (dx) : "a" (func)); -#else - __asm__ volatile ("cpuid": "=a" (ax), "=b" (bx), "=c" (cx), "=d" (dx) : "a" (func)); -#endif - return (cx & 0x40000000); -} - -/* inline encoding of 'rdrand %rax' to cover old binutils - * - no inputs - * - 'cc' to the clobber list as we modify condition code. - * - output of rdrand in rax and have a 8 bit error condition - */ -#define inline_rdrand_rax(val, err) \ - asm(".byte 0x48,0x0f,0xc7,0xf0; setc %1" \ - : "=a" (val), "=q" (err) \ - : \ - : "cc") - -/* inline encoding of 'rdrand %eax' to cover old binutils - * - no inputs - * - 'cc' to the clobber list as we modify condition code. - * - output of rdrand in eax and have a 8 bit error condition - */ -#define inline_rdrand_eax(val, err) \ - asm(".byte 0x0f,0xc7,0xf0; setc %1" \ - : "=a" (val), "=q" (err) \ - : \ - : "cc") - -#ifdef __x86_64__ -# define RDRAND_SZ 8 -# define RDRAND_T uint64_t -#define inline_rdrand(val, err) err = crypton_rdrand_step(&val) -#else -# define RDRAND_SZ 4 -# define RDRAND_T uint32_t -#define inline_rdrand(val, err) err = crypton_rdrand_step(&val) -#endif - -/* sadly many people are still using an old binutils, - * leading to report that instruction is not recognized. - */ -#if 1 -/* Returns 1 on success */ -static inline int crypton_rdrand_step(RDRAND_T *buffer) -{ - unsigned char err; - asm volatile ("rdrand %0; setc %1" : "=r" (*buffer), "=qm" (err)); - return (int) err; -} -#endif - -/* Returns the number of bytes successfully generated */ -int crypton_get_rand_bytes(uint8_t *buffer, size_t len) -{ - RDRAND_T tmp; - int aligned = (intptr_t) buffer % RDRAND_SZ; - int orig_len = len; - int to_alignment = RDRAND_SZ - aligned; - uint8_t ok; - - if (aligned != 0) { - inline_rdrand(tmp, ok); - if (!ok) - return 0; - memcpy(buffer, (uint8_t *) &tmp, to_alignment); - buffer += to_alignment; - len -= to_alignment; - } - - for (; len >= RDRAND_SZ; buffer += RDRAND_SZ, len -= RDRAND_SZ) { - inline_rdrand(tmp, ok); - if (!ok) - return (orig_len - len); - *((RDRAND_T *) buffer) = tmp; - } - - if (len > 0) { - inline_rdrand(tmp, ok); - if (!ok) - return (orig_len - len); - memcpy(buffer, (uint8_t *) &tmp, len); - } - return orig_len; -} diff --git a/cbits/crypton_ripemd.c b/cbits/crypton_ripemd.c --- a/cbits/crypton_ripemd.c +++ b/cbits/crypton_ripemd.c @@ -57,16 +57,17 @@ #define R(a, b, c, d, e, f, k, i, s) \ a += f(b, c, d) + w[i] + k; a = rol32(a, s) + e; c = rol32(c, 10) -static void ripemd160_do_chunk(struct ripemd160_ctx *ctx, uint32_t *buf) +/* The words are read out of the block rather than the block being pointed at + * as though it were an array of them; see crypton_md5.c. */ +static void ripemd160_do_chunk(struct ripemd160_ctx *ctx, const uint8_t *buf) { uint32_t a1, b1, c1, d1, e1, a2, b2, c2, d2, e2; -#ifdef ARCH_IS_BIG_ENDIAN uint32_t w[16]; - cpu_to_le32_array(w, buf, 16); -#else - uint32_t *w = buf; -#endif + int wi; + for (wi = 0; wi < 16; wi++) + w[wi] = load_le32(buf + 4 * wi); + a1 = ctx->h[0]; b1 = ctx->h[1]; c1 = ctx->h[2]; d1 = ctx->h[3]; e1 = ctx->h[4]; a2 = ctx->h[0]; b2 = ctx->h[1]; c2 = ctx->h[2]; d2 = ctx->h[3]; e2 = ctx->h[4]; @@ -260,24 +261,15 @@ ctx->sz += len; if (index && len >= to_fill) { memcpy(ctx->buf + index, data, to_fill); - ripemd160_do_chunk(ctx, (uint32_t *) ctx->buf); + ripemd160_do_chunk(ctx, ctx->buf); len -= to_fill; data += to_fill; index = 0; } - if (need_alignment(data, 4)) { - uint32_t tramp[16]; - ASSERT_ALIGNMENT(tramp, 4); - for (; len >= 64; len -= 64, data += 64) { - memcpy(tramp, data, 64); - ripemd160_do_chunk(ctx, tramp); - } - } else { - /* process as much 64-block as possible */ - for (; len >= 64; len -= 64, data += 64) - ripemd160_do_chunk(ctx, (uint32_t *) data); - } + /* No trampoline: load_le32 does not ask for a boundary. */ + for (; len >= 64; len -= 64, data += 64) + ripemd160_do_chunk(ctx, data); /* append data into buf */ if (len) diff --git a/cbits/crypton_salsa.c b/cbits/crypton_salsa.c --- a/cbits/crypton_salsa.c +++ b/cbits/crypton_salsa.c @@ -59,7 +59,7 @@ QR (x15,x12,x13,x14); \ } -static void salsa_core(int rounds, block *out, const crypton_salsa_state *in) +static void salsa_core(int rounds, crypton_salsa_block *out, const crypton_salsa_state *in) { uint32_t x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, x14, x15; int i; @@ -94,7 +94,7 @@ out->d[15] = cpu_to_le32(x15); } -void crypton_salsa_core_xor(int rounds, block *out, block *in) +void crypton_salsa_core_xor(int rounds, crypton_salsa_block *out, crypton_salsa_block *in) { uint32_t x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, x14, x15; int i; @@ -165,7 +165,7 @@ void crypton_salsa_combine(uint8_t *dst, crypton_salsa_context *ctx, const uint8_t *src, uint32_t bytes) { - block out; + crypton_salsa_block out; crypton_salsa_state *st; int i; @@ -225,7 +225,7 @@ void crypton_salsa_generate(uint8_t *dst, crypton_salsa_context *ctx, uint32_t bytes) { crypton_salsa_state *st; - block out; + crypton_salsa_block out; int i; if (!bytes) @@ -252,7 +252,7 @@ /* xor new 64-bytes chunks and store the left over if any */ for (; bytes >= 64; bytes -= 64, dst += 64) { /* generate new chunk and update state */ - salsa_core(ctx->nb_rounds, (block *) dst, st); + salsa_core(ctx->nb_rounds, (crypton_salsa_block *) dst, st); st->d[8] += 1; if (st->d[8] == 0) st->d[9] += 1; diff --git a/cbits/crypton_salsa.h b/cbits/crypton_salsa.h --- a/cbits/crypton_salsa.h +++ b/cbits/crypton_salsa.h @@ -34,9 +34,9 @@ uint64_t q[8]; uint32_t d[16]; uint8_t b[64]; -} block; +} crypton_salsa_block; -typedef block crypton_salsa_state; +typedef crypton_salsa_block crypton_salsa_state; typedef struct { crypton_salsa_state st; @@ -47,7 +47,7 @@ } crypton_salsa_context; /* for scrypt */ -void crypton_salsa_core_xor(int rounds, block *out, block *in); +void crypton_salsa_core_xor(int rounds, crypton_salsa_block *out, crypton_salsa_block *in); void crypton_salsa_init_core(crypton_salsa_state *st, uint32_t keylen, const uint8_t *key, uint32_t ivlen, const uint8_t *iv); void crypton_salsa_init(crypton_salsa_context *ctx, uint8_t nb_rounds, uint32_t keylen, const uint8_t *key, uint32_t ivlen, const uint8_t *iv); diff --git a/cbits/crypton_scrypt.c b/cbits/crypton_scrypt.c --- a/cbits/crypton_scrypt.c +++ b/cbits/crypton_scrypt.c @@ -37,10 +37,10 @@ array_copy32(X, &in[(2 * r - 1) * 16], 16); for (i = 0; i < 2 * r; i += 2) { - crypton_salsa_core_xor(8, (block *) X, (block *) &in[i*16]); + crypton_salsa_core_xor(8, (crypton_salsa_block *) X, (crypton_salsa_block *) &in[i*16]); array_copy32(&out[i * 8], X, 16); - crypton_salsa_core_xor(8, (block *) X, (block *) &in[i*16+16]); + crypton_salsa_core_xor(8, (crypton_salsa_block *) X, (crypton_salsa_block *) &in[i*16+16]); array_copy32(&out[i * 8 + r * 16], X, 16); } } diff --git a/cbits/crypton_sha1.c b/cbits/crypton_sha1.c --- a/cbits/crypton_sha1.c +++ b/cbits/crypton_sha1.c @@ -32,7 +32,7 @@ * Two threads racing to answer here both write the same value. */ #ifdef WITH_ARMV8_SHA1 -extern void crypton_sha1_armv8_do_chunk(uint32_t state[5], const uint32_t buf[16]); +extern void crypton_sha1_armv8_do_chunk(uint32_t state[5], const uint8_t buf[64]); extern void crypton_sha1_armv8_do_chunks(uint32_t state[5], const uint8_t *data, uint32_t blocks); extern int crypton_sha1_armv8_available(void); @@ -55,8 +55,20 @@ const void *data, size_t blocks); #endif +/* Resolved before there is a second thread; see the constructor in + * cbits/crypton_aes.c for why. The test below then only ever reads. */ static int sha1_use_armv8 = -1; + +__attribute__((constructor)) +static void sha1_armv8_ctor(void) +{ + sha1_use_armv8 = crypton_sha1_armv8_available(); +#ifdef SHA1_ASM + if (sha1_use_armv8) + crypton_armcap_P |= CRYPTON_ARMCAP_SHA1; #endif +} +#endif #ifdef WITH_X86_SHA_NI #include "crypton_cpu.h" @@ -89,11 +101,13 @@ #define M(i) (w[i & 0x0f] = rol32(w[i & 0x0f] ^ w[(i - 14) & 0x0f] \ ^ w[(i - 8) & 0x0f] ^ w[(i - 3) & 0x0f], 1)) -static void sha1_do_chunk_generic(struct sha1_ctx *ctx, uint32_t *buf) +/* The words are read out of the block rather than the block being pointed at + * as though it were an array of them; see crypton_md5.c. */ +static void sha1_do_chunk_generic(struct sha1_ctx *ctx, const uint8_t *buf) { uint32_t a, b, c, d, e; uint32_t w[16]; -#define CPY(i) w[i] = be32_to_cpu(buf[i]) +#define CPY(i) w[i] = load_be32(buf + 4 * (i)) CPY(0); CPY(1); CPY(2); CPY(3); CPY(4); CPY(5); CPY(6); CPY(7); CPY(8); CPY(9); CPY(10); CPY(11); CPY(12); CPY(13); CPY(14); CPY(15); #undef CPY @@ -197,14 +211,14 @@ * They arrived long after the x86-64 baseline, so ask before using them. * Two threads racing to answer here both write the same value. */ -extern void crypton_sha1_x86_do_chunk(uint32_t state[5], const uint32_t buf[16]); +extern void crypton_sha1_x86_do_chunk(uint32_t state[5], const uint8_t buf[64]); extern void crypton_sha1_x86_do_chunks(uint32_t state[5], const uint8_t *data, uint32_t blocks); static int sha1_use_x86 = -1; #endif -static inline void sha1_do_chunk(struct sha1_ctx *ctx, uint32_t *buf) +static inline void sha1_do_chunk(struct sha1_ctx *ctx, const uint8_t *buf) { #ifdef WITH_ARMV8_SHA1 if (sha1_use_armv8 < 0) { @@ -247,7 +261,7 @@ /* process partial buffer if there's enough data to make a block */ if (index && len >= to_fill) { memcpy(ctx->buf + index, data, to_fill); - sha1_do_chunk(ctx, (uint32_t *) ctx->buf); + sha1_do_chunk(ctx, ctx->buf); len -= to_fill; data += to_fill; index = 0; @@ -292,18 +306,9 @@ } #endif - if (need_alignment(data, 4)) { - uint32_t tramp[16]; - ASSERT_ALIGNMENT(tramp, 4); - for (; len >= 64; len -= 64, data += 64) { - memcpy(tramp, data, 64); - sha1_do_chunk(ctx, tramp); - } - } else { - /* process as much 64-block as possible */ - for (; len >= 64; len -= 64, data += 64) - sha1_do_chunk(ctx, (uint32_t *) data); - } + /* No trampoline: load_be32 does not ask for a boundary. */ + for (; len >= 64; len -= 64, data += 64) + sha1_do_chunk(ctx, data); /* append data into buf */ if (len) diff --git a/cbits/crypton_sha256.c b/cbits/crypton_sha256.c --- a/cbits/crypton_sha256.c +++ b/cbits/crypton_sha256.c @@ -78,13 +78,16 @@ #define s0(x) (ror32(x, 7) ^ ror32(x,18) ^ (x >> 3)) #define s1(x) (ror32(x,17) ^ ror32(x,19) ^ (x >> 10)) -static void sha256_do_chunk_generic(struct sha256_ctx *ctx, uint32_t buf[]) +/* The sixteen words are read out of the block rather than the block being + * pointed at as though it were an array of them; see crypton_md5.c. */ +static void sha256_do_chunk_generic(struct sha256_ctx *ctx, const uint8_t *buf) { uint32_t a, b, c, d, e, f, g, h, t1, t2; int i; uint32_t w[64]; - cpu_to_be32_array(w, buf, 16); + for (i = 0; i < 16; i++) + w[i] = load_be32(buf + 4 * i); for (i = 16; i < 64; i++) w[i] = s1(w[i - 2]) + w[i - 7] + s0(w[i - 15]) + w[i - 16]; @@ -120,10 +123,18 @@ * sha256_armv8.c. They are optional in ARMv8.0, so ask before using them. * Two threads racing to answer here both write the same value. */ -extern void crypton_sha256_armv8_do_chunk(uint32_t state[8], const uint32_t buf[16]); +extern void crypton_sha256_armv8_do_chunk(uint32_t state[8], const uint8_t buf[64]); extern int crypton_sha256_armv8_available(void); +/* Resolved before there is a second thread; see the constructor in + * cbits/crypton_aes.c for why. The test below then only ever reads. */ static int sha256_use_armv8 = -1; + +__attribute__((constructor)) +static void sha256_armv8_ctor(void) +{ + sha256_use_armv8 = crypton_sha256_armv8_available(); +} #endif #if (defined(WITH_ARMV8_SHA256_ASM) && defined(WITH_ARMV8_SHA2)) \ @@ -141,22 +152,34 @@ extern void crypton_sha256_asm_block_data_order(uint32_t state[8], const void *data, size_t blocks); +#ifdef WITH_ARMV8_SHA256_ASM +/* The assembly picks its path from crypton_armcap_P, so the answer to the + * runtime check has to reach that word rather than the flag above. It is + * set here, before there is a second thread, for the reason the constructor + * in cbits/crypton_aes.c gives. + * + * This ran from sha256_asm_ready below until 2.1.3, guarded by the flag + * still being unresolved -- which stopped happening when the constructor + * above was added, so the bit was never set and the assembly took its + * generic path. SHA-256 was 5.7 times slower on an Apple M4 for it. */ +__attribute__((constructor)) +static void sha256_armcap_ctor(void) +{ + if (crypton_sha256_armv8_available()) + crypton_armcap_P |= CRYPTON_ARMCAP_SHA256; +} +#endif + static void sha256_asm_ready(void) { -#ifdef WITH_ARMV8_SHA256_ASM - if (sha256_use_armv8 < 0) { - if (crypton_sha256_armv8_available()) - crypton_armcap_P |= CRYPTON_ARMCAP_SHA256; - sha256_use_armv8 = 1; - } -#else +#ifndef WITH_ARMV8_SHA256_ASM crypton_x86_ia32cap_resolve(); #endif } #endif -static void sha256_do_chunk(struct sha256_ctx *ctx, uint32_t buf[]) +static void sha256_do_chunk(struct sha256_ctx *ctx, const uint8_t *buf) { #ifdef SHA256_ASM sha256_asm_ready(); @@ -192,7 +215,7 @@ /* process partial buffer if there's enough data to make a block */ if (index && len >= to_fill) { memcpy(ctx->buf + index, data, to_fill); - sha256_do_chunk(ctx, (uint32_t *) ctx->buf); + sha256_do_chunk(ctx, ctx->buf); len -= to_fill; data += to_fill; index = 0; @@ -211,18 +234,9 @@ len -= (uint32_t) blocks * 64; } #else - if (need_alignment(data, 4)) { - uint32_t tramp[16]; - ASSERT_ALIGNMENT(tramp, 4); - for (; len >= 64; len -= 64, data += 64) { - memcpy(tramp, data, 64); - sha256_do_chunk(ctx, tramp); - } - } else { - /* process as much 64-block as possible */ - for (; len >= 64; len -= 64, data += 64) - sha256_do_chunk(ctx, (uint32_t *) data); - } + /* No trampoline: load_be32 does not ask for a boundary. */ + for (; len >= 64; len -= 64, data += 64) + sha256_do_chunk(ctx, data); #endif /* append data into buf */ diff --git a/cbits/crypton_sha256.h b/cbits/crypton_sha256.h --- a/cbits/crypton_sha256.h +++ b/cbits/crypton_sha256.h @@ -51,6 +51,18 @@ void crypton_sha256_init(struct sha256_ctx *ctx); void crypton_sha256_update(struct sha256_ctx *ctx, const uint8_t *data, uint32_t len); +/* The pointers are all required to be non-null, which is said here so that + * the compiler knows it too. Both of these write their digest through a + * loop -- store_be32(out + 4 * i, ...) -- where sha1 and md5 write theirs at + * constant offsets, and that is the difference that makes gcc's + * -Wstringop-overflow reason about out being null: with + * -fsanitize=undefined, UndefinedBehaviorSanitizer inserts a null check + * before memcpy, because glibc declares memcpy nonnull, and the check puts a + * null path in front of the warning pass, which then reports writing into + * "a region of size 0" at "address zero". Saying the pointer is never null + * removes the path rather than the warning. It costs nothing: compiled as + * the package compiles it, the assembly is identical with and without. */ +__attribute__((nonnull)) void crypton_sha256_finalize(struct sha256_ctx *ctx, uint8_t *out); void crypton_sha256_finalize_prefix(struct sha256_ctx *ctx, const uint8_t *data, uint32_t len, uint32_t n, uint8_t *out); diff --git a/cbits/crypton_sha3.c b/cbits/crypton_sha3.c --- a/cbits/crypton_sha3.c +++ b/cbits/crypton_sha3.c @@ -102,14 +102,16 @@ #define sha3_asm_absorb crypton_keccak_asm_absorb #endif -static inline void sha3_do_chunk(uint64_t state[25], uint64_t buf[], int bufsz) +/* The words are read out of the block rather than the block being pointed at + * as though it were an array of them; see crypton_md5.c. */ +static inline void sha3_do_chunk(uint64_t state[25], const uint8_t *buf, int bufsz) { int i, j, r; uint64_t tmp, bc[5]; /* merge buf with state */ for (i = 0; i < bufsz; i++) - state[i] ^= le64_to_cpu(buf[i]); + state[i] ^= load_le64(buf + 8 * i); #ifdef WITH_ARMV8_SHA3 if (sha3_armv8_ok()) { @@ -180,14 +182,14 @@ to_fill = ctx->bufsz - ctx->bufindex; if (ctx->bufindex == ctx->bufsz) { - sha3_do_chunk(ctx->state, (uint64_t *) ctx->buf, ctx->bufsz / 8); + sha3_do_chunk(ctx->state, ctx->buf, ctx->bufsz / 8); ctx->bufindex = 0; } /* process partial buffer if there's enough data to make a block */ if (ctx->bufindex && len >= to_fill) { memcpy(ctx->buf + ctx->bufindex, data, to_fill); - sha3_do_chunk(ctx->state, (uint64_t *) ctx->buf, ctx->bufsz / 8); + sha3_do_chunk(ctx->state, ctx->buf, ctx->bufsz / 8); len -= to_fill; data += to_fill; ctx->bufindex = 0; @@ -207,18 +209,9 @@ } #endif - if (need_alignment(data, 8)) { - uint64_t tramp[SHA3_BUF_SIZE_MAX/8]; - ASSERT_ALIGNMENT(tramp, 8); - for (; len >= ctx->bufsz; len -= ctx->bufsz, data += ctx->bufsz) { - memcpy(tramp, data, ctx->bufsz); - sha3_do_chunk(ctx->state, tramp, ctx->bufsz / 8); - } - } else { - /* process as much ctx->bufsz-block */ - for (; len >= ctx->bufsz; len -= ctx->bufsz, data += ctx->bufsz) - sha3_do_chunk(ctx->state, (uint64_t *) data, ctx->bufsz / 8); - } + /* No trampoline: load_le64 does not ask for a boundary. */ + for (; len >= ctx->bufsz; len -= ctx->bufsz, data += ctx->bufsz) + sha3_do_chunk(ctx->state, data, ctx->bufsz / 8); /* append data into buf */ @@ -232,7 +225,7 @@ { /* process full buffer if needed */ if (ctx->bufindex == ctx->bufsz) { - sha3_do_chunk(ctx->state, (uint64_t *) ctx->buf, ctx->bufsz / 8); + sha3_do_chunk(ctx->state, ctx->buf, ctx->bufsz / 8); ctx->bufindex = 0; } @@ -242,7 +235,7 @@ ctx->buf[ctx->bufsz - 1] |= 0x80; /* process */ - sha3_do_chunk(ctx->state, (uint64_t *) ctx->buf, ctx->bufsz / 8); + sha3_do_chunk(ctx->state, ctx->buf, ctx->bufsz / 8); ctx->bufindex = 0; } diff --git a/cbits/crypton_sha512.c b/cbits/crypton_sha512.c --- a/cbits/crypton_sha512.c +++ b/cbits/crypton_sha512.c @@ -91,13 +91,16 @@ #define s0(x) (ror64(x, 1) ^ ror64(x, 8) ^ (x >> 7)) #define s1(x) (ror64(x, 19) ^ ror64(x, 61) ^ (x >> 6)) -static void sha512_do_chunk_generic(struct sha512_ctx *ctx, uint64_t *buf) +/* The words are read out of the block rather than the block being pointed at + * as though it were an array of them; see crypton_md5.c. */ +static void sha512_do_chunk_generic(struct sha512_ctx *ctx, const uint8_t *buf) { uint64_t a, b, c, d, e, f, g, h, t1, t2; int i; uint64_t w[80]; - cpu_to_be64_array(w, buf, 16); + for (i = 0; i < 16; i++) + w[i] = load_be64(buf + 8 * i); for (i = 16; i < 80; i++) w[i] = s1(w[i - 2]) + w[i - 7] + s0(w[i - 15]) + w[i - 16]; @@ -135,10 +138,18 @@ * widespread than the SHA-256 ones, so ask before using them. Two threads * racing to answer here both write the same value. */ -extern void crypton_sha512_armv8_do_chunk(uint64_t state[8], const uint64_t buf[16]); +extern void crypton_sha512_armv8_do_chunk(uint64_t state[8], const uint8_t buf[128]); extern int crypton_sha512_armv8_available(void); +/* Resolved before there is a second thread; see the constructor in + * cbits/crypton_aes.c for why. The test below then only ever reads. */ static int sha512_use_armv8 = -1; + +__attribute__((constructor)) +static void sha512_armv8_ctor(void) +{ + sha512_use_armv8 = crypton_sha512_armv8_available(); +} #endif @@ -154,7 +165,7 @@ const void *data, size_t blocks); #endif -static void sha512_do_chunk(struct sha512_ctx *ctx, uint64_t *buf) +static void sha512_do_chunk(struct sha512_ctx *ctx, const uint8_t *buf) { #ifdef SHA512_ASM crypton_x86_ia32cap_resolve(); @@ -192,7 +203,7 @@ /* process partial buffer if there's enough data to make a block */ if (index && len >= to_fill) { memcpy(ctx->buf + index, data, to_fill); - sha512_do_chunk(ctx, (uint64_t *) ctx->buf); + sha512_do_chunk(ctx, ctx->buf); len -= to_fill; data += to_fill; index = 0; @@ -210,18 +221,9 @@ len -= (uint32_t) blocks * 128; } #else - if (need_alignment(data, 8)) { - uint64_t tramp[16]; - ASSERT_ALIGNMENT(tramp, 8); - for (; len >= 128; len -= 128, data += 128) { - memcpy(tramp, data, 128); - sha512_do_chunk(ctx, tramp); - } - } else { - /* process as much 128-block as possible */ - for (; len >= 128; len -= 128, data += 128) - sha512_do_chunk(ctx, (uint64_t *) data); - } + /* No trampoline: load_be64 does not ask for a boundary. */ + for (; len >= 128; len -= 128, data += 128) + sha512_do_chunk(ctx, data); #endif /* append data into buf */ @@ -344,7 +346,7 @@ /* re-init the context, otherwise len is changed */ memset(ctx, 0, sizeof(*ctx)); for (i = 0; i < 8; i++) - ctx->h[i] = cpu_to_be64(((uint64_t *) out)[i]); + ctx->h[i] = load_be64(out + 8 * i); } } } diff --git a/cbits/crypton_sha512.h b/cbits/crypton_sha512.h --- a/cbits/crypton_sha512.h +++ b/cbits/crypton_sha512.h @@ -50,6 +50,18 @@ void crypton_sha512_init(struct sha512_ctx *ctx); void crypton_sha512_update(struct sha512_ctx *ctx, const uint8_t *data, uint32_t len); +/* The pointers are all required to be non-null, which is said here so that + * the compiler knows it too. Both of these write their digest through a + * loop -- store_be32(out + 4 * i, ...) -- where sha1 and md5 write theirs at + * constant offsets, and that is the difference that makes gcc's + * -Wstringop-overflow reason about out being null: with + * -fsanitize=undefined, UndefinedBehaviorSanitizer inserts a null check + * before memcpy, because glibc declares memcpy nonnull, and the check puts a + * null path in front of the warning pass, which then reports writing into + * "a region of size 0" at "address zero". Saying the pointer is never null + * removes the path rather than the warning. It costs nothing: compiled as + * the package compiles it, the assembly is identical with and without. */ +__attribute__((nonnull)) void crypton_sha512_finalize(struct sha512_ctx *ctx, uint8_t *out); void crypton_sha512_finalize_prefix(struct sha512_ctx *ctx, const uint8_t *data, uint32_t len, uint32_t n, uint8_t *out); diff --git a/cbits/crypton_skein256.c b/cbits/crypton_skein256.c --- a/cbits/crypton_skein256.c +++ b/cbits/crypton_skein256.c @@ -37,7 +37,9 @@ static const uint8_t K256_6[2] = { 58, 22, }; static const uint8_t K256_7[2] = { 32, 32, }; -static inline void skein256_do_chunk(struct skein256_ctx *ctx, uint64_t *buf, uint32_t len) +/* The four words are read out of the block rather than the block being + * pointed at as though it were an array of them; see crypton_md5.c. */ +static inline void skein256_do_chunk(struct skein256_ctx *ctx, const uint8_t *bufp, uint32_t len) { uint64_t x[4]; uint64_t ts[3]; @@ -78,11 +80,18 @@ ROUND(0,3,2,1,K256_7); \ INJECTKEY((i*2) + 2) - x[0] = le64_to_cpu(buf[0]) + ks[0]; - x[1] = le64_to_cpu(buf[1]) + ks[1] + ts[0]; - x[2] = le64_to_cpu(buf[2]) + ks[2] + ts[1]; - x[3] = le64_to_cpu(buf[3]) + ks[3]; + uint64_t buf[4]; + buf[0] = load_le64(bufp); + buf[1] = load_le64(bufp + 8); + buf[2] = load_le64(bufp + 16); + buf[3] = load_le64(bufp + 24); + + x[0] = buf[0] + ks[0]; + x[1] = buf[1] + ks[1] + ts[0]; + x[2] = buf[2] + ks[2] + ts[1]; + x[3] = buf[3] + ks[3]; + /* 9 pass of 8 rounds = 72 rounds */ PASS(0); PASS(1); @@ -98,10 +107,10 @@ ctx->t0 = ts[0]; ctx->t1 = ts[1]; - ctx->h[0] = x[0] ^ cpu_to_le64(buf[0]); - ctx->h[1] = x[1] ^ cpu_to_le64(buf[1]); - ctx->h[2] = x[2] ^ cpu_to_le64(buf[2]); - ctx->h[3] = x[3] ^ cpu_to_le64(buf[3]); + ctx->h[0] = x[0] ^ buf[0]; + ctx->h[1] = x[1] ^ buf[1]; + ctx->h[2] = x[2] ^ buf[2]; + ctx->h[3] = x[3] ^ buf[3]; } void crypton_skein256_init(struct skein256_ctx *ctx, uint32_t hashlen) @@ -115,7 +124,7 @@ buf[0] = cpu_to_le64((SKEIN_VERSION << 32) | SKEIN_IDSTRING); buf[1] = cpu_to_le64(hashlen); buf[2] = 0; /* tree info, not implemented */ - skein256_do_chunk(ctx, buf, 4*8); + skein256_do_chunk(ctx, (const uint8_t *) buf, 4*8); SET_TYPE(ctx, FLAG_FIRST | FLAG_TYPE(TYPE_MSG)); } @@ -130,7 +139,7 @@ to_fill = 32 - ctx->bufindex; if (ctx->bufindex == 32) { - skein256_do_chunk(ctx, (uint64_t *) ctx->buf, 32); + skein256_do_chunk(ctx, ctx->buf, 32); ctx->bufindex = 0; } @@ -138,24 +147,17 @@ * and there's without doubt further blocks */ if (ctx->bufindex && len > to_fill) { memcpy(ctx->buf + ctx->bufindex, data, to_fill); - skein256_do_chunk(ctx, (uint64_t *) ctx->buf, 32); + skein256_do_chunk(ctx, ctx->buf, 32); len -= to_fill; data += to_fill; ctx->bufindex = 0; } - if (need_alignment(data, 8)) { - uint64_t tramp[4]; - ASSERT_ALIGNMENT(tramp, 8); - for (; len > 32; len -= 32, data += 32) { - memcpy(tramp, data, 32); - skein256_do_chunk(ctx, tramp, 32); - } - } else { - /* process as much 32-block as possible except the last one in case we finalize */ - for (; len > 32; len -= 32, data += 32) - skein256_do_chunk(ctx, (uint64_t *) data, 32); - } + /* No trampoline for a block that is not on an eight-byte boundary: the + * words are read with load_le64 now, which does not ask. The last + * block is left for the finalisation. */ + for (; len > 32; len -= 32, data += 32) + skein256_do_chunk(ctx, data, 32); /* append data into buf */ if (len) { @@ -174,7 +176,7 @@ /* if buf is not complete pad with 0 bytes */ if (ctx->bufindex < 32) memset(ctx->buf + ctx->bufindex, '\0', 32 - ctx->bufindex); - skein256_do_chunk(ctx, (uint64_t *) ctx->buf, ctx->bufindex); + skein256_do_chunk(ctx, ctx->buf, ctx->bufindex); memset(ctx->buf, '\0', 32); @@ -187,9 +189,9 @@ /* threefish in counter mode, 0 for 1st 64 bytes, 1 for 2nd 64 bytes, .. */ for (i = 0; i*32 < outsize; i++) { uint64_t w[4]; - *((uint64_t *) ctx->buf) = cpu_to_le64(i); + store_le64(ctx->buf, i); SET_TYPE(ctx, FLAG_FIRST | FLAG_FINAL | FLAG_TYPE(TYPE_OUT)); - skein256_do_chunk(ctx, (uint64_t *) ctx->buf, sizeof(uint64_t)); + skein256_do_chunk(ctx, ctx->buf, sizeof(uint64_t)); n = outsize - i * 32; if (n >= 32) n = 32; diff --git a/cbits/crypton_skein256.h b/cbits/crypton_skein256.h --- a/cbits/crypton_skein256.h +++ b/cbits/crypton_skein256.h @@ -37,8 +37,8 @@ #define SKEIN256_CTX_SIZE sizeof(struct skein256_ctx) -void cryponite_skein256_init(struct skein256_ctx *ctx, uint32_t hashlen); -void cryponite_skein256_update(struct skein256_ctx *ctx, const uint8_t *data, uint32_t len); -void cryponite_skein256_finalize(struct skein256_ctx *ctx, uint32_t hashlen, uint8_t *out); +void crypton_skein256_init(struct skein256_ctx *ctx, uint32_t hashlen); +void crypton_skein256_update(struct skein256_ctx *ctx, const uint8_t *data, uint32_t len); +void crypton_skein256_finalize(struct skein256_ctx *ctx, uint32_t hashlen, uint8_t *out); #endif diff --git a/cbits/crypton_skein512.c b/cbits/crypton_skein512.c --- a/cbits/crypton_skein512.c +++ b/cbits/crypton_skein512.c @@ -37,7 +37,9 @@ static const uint8_t K512_6[4] = { 25, 29, 39, 43, }; static const uint8_t K512_7[4] = { 8, 35, 56, 22, }; -static inline void skein512_do_chunk(struct skein512_ctx *ctx, uint64_t *buf, uint32_t len) +/* The words are read out of the block rather than the block being pointed at + * as though it were an array of them; see crypton_md5.c. */ +static inline void skein512_do_chunk(struct skein512_ctx *ctx, const uint8_t *bufp, uint32_t len) { uint64_t x[8]; uint64_t ts[3]; @@ -88,15 +90,21 @@ ROUND(6,1,0,7,2,5,4,3,K512_7); \ INJECTKEY((i*2) + 2) - x[0] = le64_to_cpu(buf[0]) + ks[0]; - x[1] = le64_to_cpu(buf[1]) + ks[1]; - x[2] = le64_to_cpu(buf[2]) + ks[2]; - x[3] = le64_to_cpu(buf[3]) + ks[3]; - x[4] = le64_to_cpu(buf[4]) + ks[4]; - x[5] = le64_to_cpu(buf[5]) + ks[5] + ts[0]; - x[6] = le64_to_cpu(buf[6]) + ks[6] + ts[1]; - x[7] = le64_to_cpu(buf[7]) + ks[7]; + uint64_t buf[8]; + int bi; + for (bi = 0; bi < 8; bi++) + buf[bi] = load_le64(bufp + 8 * bi); + + x[0] = buf[0] + ks[0]; + x[1] = buf[1] + ks[1]; + x[2] = buf[2] + ks[2]; + x[3] = buf[3] + ks[3]; + x[4] = buf[4] + ks[4]; + x[5] = buf[5] + ks[5] + ts[0]; + x[6] = buf[6] + ks[6] + ts[1]; + x[7] = buf[7] + ks[7]; + /* 9 pass of 8 rounds = 72 rounds */ PASS(0); PASS(1); @@ -112,14 +120,14 @@ ctx->t0 = ts[0]; ctx->t1 = ts[1]; - ctx->h[0] = x[0] ^ cpu_to_le64(buf[0]); - ctx->h[1] = x[1] ^ cpu_to_le64(buf[1]); - ctx->h[2] = x[2] ^ cpu_to_le64(buf[2]); - ctx->h[3] = x[3] ^ cpu_to_le64(buf[3]); - ctx->h[4] = x[4] ^ cpu_to_le64(buf[4]); - ctx->h[5] = x[5] ^ cpu_to_le64(buf[5]); - ctx->h[6] = x[6] ^ cpu_to_le64(buf[6]); - ctx->h[7] = x[7] ^ cpu_to_le64(buf[7]); + ctx->h[0] = x[0] ^ buf[0]; + ctx->h[1] = x[1] ^ buf[1]; + ctx->h[2] = x[2] ^ buf[2]; + ctx->h[3] = x[3] ^ buf[3]; + ctx->h[4] = x[4] ^ buf[4]; + ctx->h[5] = x[5] ^ buf[5]; + ctx->h[6] = x[6] ^ buf[6]; + ctx->h[7] = x[7] ^ buf[7]; } void crypton_skein512_init(struct skein512_ctx *ctx, uint32_t hashlen) @@ -133,7 +141,7 @@ buf[0] = cpu_to_le64((SKEIN_VERSION << 32) | SKEIN_IDSTRING); buf[1] = cpu_to_le64(hashlen); buf[2] = 0; /* tree info, not implemented */ - skein512_do_chunk(ctx, buf, 4*8); + skein512_do_chunk(ctx, (const uint8_t *) buf, 4*8); SET_TYPE(ctx, FLAG_FIRST | FLAG_TYPE(TYPE_MSG)); } @@ -148,7 +156,7 @@ to_fill = 64 - ctx->bufindex; if (ctx->bufindex == 64) { - skein512_do_chunk(ctx, (uint64_t *) ctx->buf, 64); + skein512_do_chunk(ctx, ctx->buf, 64); ctx->bufindex = 0; } @@ -156,24 +164,15 @@ * and there's without doubt further blocks */ if (ctx->bufindex && len > to_fill) { memcpy(ctx->buf + ctx->bufindex, data, to_fill); - skein512_do_chunk(ctx, (uint64_t *) ctx->buf, 64); + skein512_do_chunk(ctx, ctx->buf, 64); len -= to_fill; data += to_fill; ctx->bufindex = 0; } - if (need_alignment(data, 8)) { - uint64_t tramp[8]; - ASSERT_ALIGNMENT(tramp, 8); - for (; len > 64; len -= 64, data += 64) { - memcpy(tramp, data, 64); - skein512_do_chunk(ctx, tramp, 64); - } - } else { - /* process as much 64-block as possible except the last one in case we finalize */ - for (; len > 64; len -= 64, data += 64) - skein512_do_chunk(ctx, (uint64_t *) data, 64); - } + /* No trampoline: load_le64 does not ask for a boundary. */ + for (; len > 64; len -= 64, data += 64) + skein512_do_chunk(ctx, data, 64); /* append data into buf */ if (len) { @@ -192,7 +191,7 @@ /* if buf is not complete pad with 0 bytes */ if (ctx->bufindex < 64) memset(ctx->buf + ctx->bufindex, '\0', 64 - ctx->bufindex); - skein512_do_chunk(ctx, (uint64_t *) ctx->buf, ctx->bufindex); + skein512_do_chunk(ctx, ctx->buf, ctx->bufindex); memset(ctx->buf, '\0', 64); @@ -205,9 +204,9 @@ /* threefish in counter mode, 0 for 1st 64 bytes, 1 for 2nd 64 bytes, .. */ for (i = 0; i*64 < outsize; i++) { uint64_t w[8]; - *((uint64_t *) ctx->buf) = cpu_to_le64(i); + store_le64(ctx->buf, i); SET_TYPE(ctx, FLAG_FIRST | FLAG_FINAL | FLAG_TYPE(TYPE_OUT)); - skein512_do_chunk(ctx, (uint64_t *) ctx->buf, sizeof(uint64_t)); + skein512_do_chunk(ctx, ctx->buf, sizeof(uint64_t)); n = outsize - i * 64; if (n >= 64) n = 64; diff --git a/cbits/crypton_skein512.h b/cbits/crypton_skein512.h --- a/cbits/crypton_skein512.h +++ b/cbits/crypton_skein512.h @@ -37,8 +37,8 @@ #define SKEIN512_CTX_SIZE sizeof(struct skein512_ctx) -void cryponite_skein512_init(struct skein512_ctx *ctx, uint32_t hashlen); -void cryponite_skein512_update(struct skein512_ctx *ctx, const uint8_t *data, uint32_t len); -void cryponite_skein512_finalize(struct skein512_ctx *ctx, uint32_t hashlen, uint8_t *out); +void crypton_skein512_init(struct skein512_ctx *ctx, uint32_t hashlen); +void crypton_skein512_update(struct skein512_ctx *ctx, const uint8_t *data, uint32_t len); +void crypton_skein512_finalize(struct skein512_ctx *ctx, uint32_t hashlen, uint8_t *out); #endif diff --git a/cbits/crypton_sysdrg.c b/cbits/crypton_sysdrg.c new file mode 100644 --- /dev/null +++ b/cbits/crypton_sysdrg.c @@ -0,0 +1,388 @@ +/* + * The generator behind MonadRandom IO. + * + * A ChaCha20 DRBG per operating system thread, seeded from a process-wide + * DRBG, which is itself seeded from the system entropy pool. This is the + * shape RFC 9180's neighbours and the other libraries have settled on, and + * it was asked for in #298. + * + * Per operating system thread and not per Haskell thread: a forkIO thread + * moves between capabilities, so state kept against it would be shared by + * threads running at the same time. That is why the state is here and + * reached through pthread_getspecific rather than held in Haskell. + * + * Three things force a reseed: + * + * - a thread has produced CRYPTON_THREAD_RESEED bytes, + * - the process DRBG has issued CRYPTON_GLOBAL_RESEED bytes of seed, + * - the process has forked. + * + * The last one is the one that bites. A child inherits its parent's state + * and would otherwise produce the same stream; the generation counter below + * is bumped in the child by a pthread_atfork handler, and every generator + * compares against it before it answers. + */ + +#include +#include +#include + +#include "crypton_chacha.h" +#include "crypton_sha512.h" + +#ifdef _WIN32 +#include +#else +#include +#include +#endif + +/* from crypton_sysrandom.c */ +int crypton_sysrandom_available(void); +int crypton_sysrandom_bytes(uint8_t *buf, int len); + +#define CHACHA_ROUNDS 20 +#define SEED_KEY 32 +#define SEED_IV 8 +#define SEED_LEN (SEED_KEY + SEED_IV) + +#define CRYPTON_THREAD_RESEED (1u << 20) +#define CRYPTON_GLOBAL_RESEED (1u << 20) + +typedef struct { + crypton_chacha_context ctx; + uint64_t used; + uint32_t generation; + int seeded; +} drg_t; + +static drg_t global_drg; +static volatile uint32_t fork_generation = 0; + +#ifdef _WIN32 +static CRITICAL_SECTION global_lock; +/* Fls and not Tls: TlsAlloc has no destructor, so the state of every thread + * that ever drew a byte would be left allocated and unscrubbed when the + * thread ended. FlsAlloc takes the callback that pthread_key_create does. */ +static DWORD thread_slot; +static INIT_ONCE init_once = INIT_ONCE_STATIC_INIT; +#else +static pthread_mutex_t global_lock = PTHREAD_MUTEX_INITIALIZER; +static pthread_key_t thread_slot; +static pthread_once_t init_once = PTHREAD_ONCE_INIT; +#endif + +static void scrub(void *p, size_t n) +{ + volatile uint8_t *q = (volatile uint8_t *) p; + while (n--) *q++ = 0; +} + +/* + * Seed material for the process DRBG. + * + * What the system gives goes through SHA-512 rather than into the key + * directly, so that the key is a fixed size whatever the call returns, and + * so that a second source could be added without any of this changing shape. + * There is one source today and the note below says why. + */ +static int seed_from_system(uint8_t out[SEED_LEN]) +{ + struct sha512_ctx h; + uint8_t buf[64]; + uint8_t digest[64]; + int got; + + crypton_sha512_init(&h); + + /* + * The system call only. Where there is none -- an old kernel, a BSD + * this does not know -- seeding fails and the caller keeps the path + * it has, rather than this file growing a second copy of the device + * reading that Crypto.Random.Entropy.Unix already does. + * + * And RDRAND is not mixed in beside it, which it was until + * @vdukhovni pointed out that it cannot add anything: every system + * this code runs on already feeds RDRAND into the pool the call below + * draws from. Linux does that whatever random.trust_cpu says -- that + * setting decides whether the contribution is *credited*, not whether + * it is mixed -- so the instruction's output is in this buffer + * already. + * + * The argument for keeping it was defence in depth against that pool + * having gone wrong. It does not survive the above: a pool that has + * gone wrong has gone wrong with RDRAND already in it, so asking the + * instruction a second time covers only the case where the kernel's + * mixing is broken and the instruction is not. That is not worth a + * second code path, a flag, and the standing invitation to read this + * as a second source when it is the same one twice. + */ + got = crypton_sysrandom_available() ? crypton_sysrandom_bytes(buf, 64) : 0; + if (got <= 0) { + /* Nothing else here is a seed on its own, so there is nothing to + * go on with: return before drawing anything that would only be + * thrown away. */ + scrub(&h, sizeof h); + scrub(buf, sizeof buf); + return 0; + } + crypton_sha512_update(&h, buf, (uint32_t) got); + + + crypton_sha512_finalize(&h, digest); + memcpy(out, digest, SEED_LEN); + + scrub(&h, sizeof h); + scrub(buf, sizeof buf); + scrub(digest, sizeof digest); + return 1; +} + +static void drg_seed(drg_t *d, const uint8_t seed[SEED_LEN]) +{ + crypton_chacha_init(&d->ctx, CHACHA_ROUNDS, SEED_KEY, seed, + SEED_IV, seed + SEED_KEY); + d->used = 0; + d->generation = fork_generation; + d->seeded = 1; +} + +/* + * Forget the key that produced the bytes just handed out. + * + * Without this the key stands until the next reseed, and anyone who reads a + * generator's state can wind the counter back and reproduce everything it + * has issued since -- up to CRYPTON_THREAD_RESEED bytes that were meant to + * be secret. Taking the next forty bytes of keystream as the new key and + * nonce, and dropping the old ones, puts that out of reach one step after + * it is issued: ChaCha20 does not run backwards, and the key that would + * have been needed is gone. It is what arc4random does. + * + * crypton_chacha_init memsets the whole context, so the counter returns to + * zero and the tail of a part-used block goes with the old key rather than + * being handed out under the new one. + * + * d->used is not touched. It counts what callers were given, which is what + * the reseed interval is written in terms of; these forty bytes are the + * cost of the rekey and not an answer to anybody. + */ +static void drg_rekey(drg_t *d) +{ + uint8_t next[SEED_LEN]; + + crypton_chacha_generate(next, &d->ctx, SEED_LEN); + crypton_chacha_init(&d->ctx, CHACHA_ROUNDS, SEED_KEY, next, + SEED_IV, next + SEED_KEY); + scrub(next, sizeof next); +} + +static int drg_stale(const drg_t *d, uint64_t limit) +{ + return !d->seeded || d->used >= limit || d->generation != fork_generation; +} + +/* Bytes from the process DRBG, which is only ever asked for seed material. */ +static int global_bytes(uint8_t *out, uint32_t len) +{ + int ok = 1; + +#ifdef _WIN32 + EnterCriticalSection(&global_lock); +#else + pthread_mutex_lock(&global_lock); +#endif + if (drg_stale(&global_drg, CRYPTON_GLOBAL_RESEED)) { + uint8_t seed[SEED_LEN]; + ok = seed_from_system(seed); + if (ok) + drg_seed(&global_drg, seed); + scrub(seed, sizeof seed); + } + if (ok) { + crypton_chacha_generate(out, &global_drg.ctx, len); + global_drg.used += len; + drg_rekey(&global_drg); + } +#ifdef _WIN32 + LeaveCriticalSection(&global_lock); +#else + pthread_mutex_unlock(&global_lock); +#endif + return ok; +} + +#ifdef _WIN32 +static void WINAPI thread_free(void *p) +#else +static void thread_free(void *p) +#endif +{ + if (p) { + scrub(p, sizeof(drg_t)); + free(p); + } +} + +#ifndef _WIN32 +/* All three handlers, not just the child's. The child's first draw has to + * reseed -- the generation has changed -- and reseeding takes global_lock. + * A fork made while another thread held it would hand the child a mutex + * locked by a thread that did not come across, and the child would wait on + * it for ever. So the lock is taken before the fork and released on both + * sides of it, which is the state the child needs it in. */ +static void before_fork(void) +{ + pthread_mutex_lock(&global_lock); +} + +static void after_fork_in_parent(void) +{ + pthread_mutex_unlock(&global_lock); +} + +static void after_fork_in_child(void) +{ + pthread_mutex_unlock(&global_lock); + fork_generation++; +} +#endif + +#ifdef CRYPTON_SYSDRG_TESTING +/* For cbits/tests/sysdrg and nothing else: hold and release the process + * generator's lock, so that a fork can be made to happen while another + * thread holds it. Nothing outside that test declares these, and the + * library is never built with this defined. */ +void crypton_sysdrg_test_lock(void); +void crypton_sysdrg_test_unlock(void); + +static drg_t *this_thread(void); + +/* The calling thread's ChaCha key, which is d[4..11] of the state. A test + * uses it to ask whether the key that produced a draw is still there + * afterwards; see "a draw replaces the key that made it" in + * cbits/tests/sysdrg. */ +void crypton_sysdrg_test_key(uint8_t out[32]); + +void crypton_sysdrg_test_key(uint8_t out[32]) +{ + drg_t *d = this_thread(); + int i; + + if (!d) + return; + for (i = 0; i < 8; i++) { + uint32_t w = d->ctx.st.d[4 + i]; + out[i * 4 + 0] = (uint8_t) (w); + out[i * 4 + 1] = (uint8_t) (w >> 8); + out[i * 4 + 2] = (uint8_t) (w >> 16); + out[i * 4 + 3] = (uint8_t) (w >> 24); + } +} + +void crypton_sysdrg_test_lock(void) +{ +#ifndef _WIN32 + pthread_mutex_lock(&global_lock); +#endif +} + +void crypton_sysdrg_test_unlock(void) +{ +#ifndef _WIN32 + pthread_mutex_unlock(&global_lock); +#endif +} +#endif + +#ifdef _WIN32 +static BOOL CALLBACK init_slot(PINIT_ONCE o, PVOID p, PVOID *c) +{ + (void) o; (void) p; (void) c; + InitializeCriticalSection(&global_lock); + thread_slot = FlsAlloc(thread_free); + return TRUE; +} +#else +static void init_slot(void) +{ + pthread_key_create(&thread_slot, thread_free); + pthread_atfork(before_fork, after_fork_in_parent, after_fork_in_child); +} +#endif + +static drg_t *this_thread(void) +{ + drg_t *d; + +#ifdef _WIN32 + InitOnceExecuteOnce(&init_once, init_slot, NULL, NULL); + if (thread_slot == FLS_OUT_OF_INDEXES) + return NULL; + d = (drg_t *) FlsGetValue(thread_slot); +#else + pthread_once(&init_once, init_slot); + d = (drg_t *) pthread_getspecific(thread_slot); +#endif + if (!d) { + d = (drg_t *) calloc(1, sizeof(drg_t)); + if (!d) + return NULL; +#ifdef _WIN32 + if (!FlsSetValue(thread_slot, d)) { + free(d); + return NULL; + } +#else + if (pthread_setspecific(thread_slot, d) != 0) { + free(d); + return NULL; + } +#endif + } + return d; +} + +/* Returns the number of bytes written, which is len unless there was no + * seed to be had -- and then it is 0, so that the caller can say so rather + * than hand back a buffer it cannot vouch for. */ +int crypton_sysdrg_bytes(uint8_t *out, int len) +{ + drg_t *d; + + if (len < 0) + return 0; + d = this_thread(); + if (!d) + return 0; + + if (drg_stale(d, CRYPTON_THREAD_RESEED)) { + uint8_t seed[SEED_LEN]; + int ok = global_bytes(seed, SEED_LEN); + if (ok) + drg_seed(d, seed); + scrub(seed, sizeof seed); + if (!ok) + return 0; + } + + crypton_chacha_generate(out, &d->ctx, (uint32_t) len); + d->used += (uint64_t) len; + drg_rekey(d); + return len; +} + +/* For the tests: how many times the process has been seen to fork. */ +uint32_t crypton_sysdrg_generation(void) +{ + return fork_generation; +} + +/* For the tests: bytes this thread's generator has produced since it was + * last seeded. A generator shared between threads would carry the first + * thread's count into the second; a per-thread one starts again at zero, + * and that is the only way from outside to tell the two apart. */ +uint64_t crypton_sysdrg_thread_used(void) +{ + drg_t *d = this_thread(); + return d ? d->used : 0; +} diff --git a/cbits/crypton_sysrandom.c b/cbits/crypton_sysrandom.c new file mode 100644 --- /dev/null +++ b/cbits/crypton_sysrandom.c @@ -0,0 +1,119 @@ +/* + * The kernel's own random number generator, reached without a file + * descriptor: getrandom(2) on Linux and FreeBSD, getentropy(3) on the + * systems that have that instead. + * + * This is preferred over reading /dev/urandom because it needs no path and + * no descriptor, so it still works where /dev is not mounted or not + * populated -- a minimal container, a chroot, a sandbox -- and it cannot be + * defeated by exhausting the descriptor table. + * + * Availability is decided at run time as well as at compile time: a binary + * built against headers that declare getrandom can still run on a kernel + * that does not implement it, and says so with ENOSYS. + */ + +#include +#include +#include + +#if defined(__linux__) +#include +#include +#ifdef SYS_getrandom +#define CRYPTON_SYSRANDOM_GETRANDOM 1 +#endif +#elif defined(__FreeBSD__) +#include +#if __FreeBSD_version >= 1200000 +#include +#define CRYPTON_SYSRANDOM_GETRANDOM 1 +#endif +#elif defined(__APPLE__) +/* getentropy(3) is declared in sys/random.h on Darwin, since 10.12 -- but + * macOS only. Crypto.Random says iOS does not allow it, which is why that + * platform builds with INSECURE_ENTROPY; until someone can try it there, + * iOS keeps the path it has rather than gaining an untested one. */ +#include +#if defined(TARGET_OS_OSX) && TARGET_OS_OSX +#include +#define CRYPTON_SYSRANDOM_GETENTROPY 1 +#endif +#elif defined(__OpenBSD__) +#include +#define CRYPTON_SYSRANDOM_GETENTROPY 1 +#elif defined(__NetBSD__) +#include +#if __NetBSD_Version__ >= 1000000000 +#include +#define CRYPTON_SYSRANDOM_GETENTROPY 1 +#endif +#endif + +/* getentropy(3) refuses more than 256 bytes in one call. */ +#define CRYPTON_GETENTROPY_MAX 256 + +#if defined(CRYPTON_SYSRANDOM_GETRANDOM) + +static int sysrandom_once(uint8_t *buf, size_t n) +{ +#if defined(__linux__) + return (int) syscall(SYS_getrandom, buf, n, 0); +#else + return (int) getrandom(buf, n, 0); +#endif +} + +#elif defined(CRYPTON_SYSRANDOM_GETENTROPY) + +static int sysrandom_once(uint8_t *buf, size_t n) +{ + if (n > CRYPTON_GETENTROPY_MAX) + n = CRYPTON_GETENTROPY_MAX; + if (getentropy(buf, n) != 0) + return -1; + return (int) n; +} + +#else + +static int sysrandom_once(uint8_t *buf, size_t n) +{ + (void) buf; (void) n; + errno = ENOSYS; + return -1; +} + +#endif + +/* Is the call there, on this kernel, right now? A zero-length request + * answers that without consuming anything. */ +int crypton_sysrandom_available(void) +{ + uint8_t b; + int r = sysrandom_once(&b, 0); + return r < 0 ? 0 : 1; +} + +/* Fill the buffer. Returns the number of bytes written, which is n unless + * the call failed for a reason other than being interrupted. */ +int crypton_sysrandom_bytes(uint8_t *buf, int len) +{ + size_t n = (size_t) len; + size_t done = 0; + + if (len < 0) + return 0; + while (done < n) { + int r = sysrandom_once(buf + done, n - done); + if (r < 0) { + if (errno == EINTR) + continue; + break; + } + if (r == 0) + break; + done += (size_t) r; + } + return (int) done; +} diff --git a/cbits/crypton_tiger.c b/cbits/crypton_tiger.c --- a/cbits/crypton_tiger.c +++ b/cbits/crypton_tiger.c @@ -306,7 +306,9 @@ ctx->h[2] = 0xf096a5b4c3b2e187ULL; } -static inline void tiger_do_chunk(struct tiger_ctx *ctx, uint64_t *buf) +/* The words are read out of the block rather than the block being pointed at + * as though it were an array of them; see crypton_md5.c. */ +static inline void tiger_do_chunk(struct tiger_ctx *ctx, const uint8_t *buf) { uint64_t x0, x1, x2, x3, x4, x5, x6, x7; uint64_t a,b,c; @@ -314,8 +316,8 @@ b = ctx->h[1]; c = ctx->h[2]; - x0 = cpu_to_le64(buf[0]); x1 = cpu_to_le64(buf[1]); x2 = cpu_to_le64(buf[2]); x3 = cpu_to_le64(buf[3]); - x4 = cpu_to_le64(buf[4]); x5 = cpu_to_le64(buf[5]); x6 = cpu_to_le64(buf[6]); x7 = cpu_to_le64(buf[7]); + x0 = load_le64(buf ); x1 = load_le64(buf + 8); x2 = load_le64(buf + 16); x3 = load_le64(buf + 24); + x4 = load_le64(buf + 32); x5 = load_le64(buf + 40); x6 = load_le64(buf + 48); x7 = load_le64(buf + 56); #define BYTEOF(c, n) ((uint8_t) (c >> ((n * 8)))) @@ -376,24 +378,15 @@ /* process partial buffer if there's enough data to make a block */ if (index && len >= to_fill) { memcpy(ctx->buf + index, data, to_fill); - tiger_do_chunk(ctx, (uint64_t *) ctx->buf); + tiger_do_chunk(ctx, ctx->buf); len -= to_fill; data += to_fill; index = 0; } - if (need_alignment(data, 8)) { - uint64_t tramp[8]; - ASSERT_ALIGNMENT(tramp, 8); - for (; len >= 64; len -= 64, data += 64) { - memcpy(tramp, data, 64); - tiger_do_chunk(ctx, tramp); - } - } else { - /* process as much 64-block as possible */ - for (; len >= 64; len -= 64, data += 64) - tiger_do_chunk(ctx, (uint64_t *) data); - } + /* No trampoline: load_le64 does not ask for a boundary. */ + for (; len >= 64; len -= 64, data += 64) + tiger_do_chunk(ctx, data); /* append data into buf */ if (len) diff --git a/cbits/crypton_xsalsa.c b/cbits/crypton_xsalsa.c --- a/cbits/crypton_xsalsa.c +++ b/cbits/crypton_xsalsa.c @@ -41,7 +41,7 @@ memset(ctx, 0, sizeof(*ctx)); ctx->nb_rounds = nb_rounds; - /* Create initial 512-bit input block: + /* Create initial 512-bit input crypton_salsa_block: (x0, x5, x10, x15) is the Salsa20 constant (x1, x2, x3, x4, x11, x12, x13, x14) is a 256-bit key (x6, x7, x8, x9) is the first 128 bits of a 192-bit nonce @@ -56,7 +56,7 @@ void crypton_xsalsa_derive(crypton_salsa_context *ctx, uint32_t ivlen, const uint8_t *iv) { - /* Finish creating initial 512-bit input block: + /* Finish creating initial 512-bit input crypton_salsa_block: (x6, x7, x8, x9) is the first 128 bits of a 192-bit nonce Except iv has been shifted by 64 bits so there are now only 128 bits ahead. @@ -65,15 +65,15 @@ ctx->st.d[ 9] += load_le32(iv + 4); /* Compute (z0, z1, . . . , z15) = doubleround ^(r/2) (x0, x1, . . . , x15) */ - block hSalsa; - memset(&hSalsa, 0, sizeof(block)); + crypton_salsa_block hSalsa; + memset(&hSalsa, 0, sizeof(crypton_salsa_block)); crypton_salsa_core_xor(ctx->nb_rounds, &hSalsa, &ctx->st); - /* Build a new 512-bit input block (x′0, x′1, . . . , x′15): + /* Build a new 512-bit input crypton_salsa_block (x′0, x′1, . . . , x′15): (x′0, x′5, x′10, x′15) is the Salsa20 constant (x′1,x′2,x′3,x′4,x′11,x′12,x′13,x′14) = (z0,z5,z10,z15,z6,z7,z8,z9) (x′6,x′7) is the last 64 bits of the 192-bit nonce - (x′8, x′9) is a 64-bit block counter. + (x′8, x′9) is a 64-bit crypton_salsa_block counter. */ ctx->st.d[ 1] = hSalsa.d[ 0] - ctx->st.d[ 0]; ctx->st.d[ 2] = hSalsa.d[ 5] - ctx->st.d[ 5]; diff --git a/cbits/curve25519/x25519.c b/cbits/curve25519/x25519.c new file mode 100644 --- /dev/null +++ b/cbits/curve25519/x25519.c @@ -0,0 +1,88 @@ +/* + * X25519 through the vendored s2n-bignum where it is built, and through + * curve25519-donna where it is not. + * + * The fixed-base routine is the interesting half: crypton had none, and asked + * for the public key by multiplying the base point 9 the general way. With a + * table it is four to five times less work, and a TLS handshake generates a + * key every time. Measured on an Apple M4: + * + * donna s2n + * shared secret 18.15 us 12.35 + * key generation 18.15 3.65 + * + * and on an x86-64, 41.19 to 26.96 and 41.18 to 8.54. + */ +#include + +#include "curve25519/x25519.h" + +void crypton_curve25519_donna(uint8_t *mypublic, const uint8_t *secret, + const uint8_t *basepoint); + +/* The assembly takes four little-endian 64-bit words, which is the same bits + * as the 32 little-endian bytes RFC 7748 sends, so the two cross by copying + * -- on a little-endian machine, which is the only kind s2n-bignum is for. */ +#if defined(CRYPTON_S2N_BIGNUM) && defined(__BYTE_ORDER__) \ + && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ +#define CRYPTON_X25519_S2N 1 +#include "crypton_cpu.h" + +extern void curve25519_x25519(uint64_t res[4], const uint64_t scalar[4], + const uint64_t point[4]); +extern void curve25519_x25519_alt(uint64_t res[4], const uint64_t scalar[4], + const uint64_t point[4]); +extern void curve25519_x25519base(uint64_t res[4], const uint64_t scalar[4]); +extern void curve25519_x25519base_alt(uint64_t res[4], const uint64_t scalar[4]); + +/* The same question as everywhere else in cbits/s2n: a microarchitecture one + * on ARM that no feature bit answers, and exactly a feature bit on x86-64. */ +static int use_alt(void) +{ +#if defined(__aarch64__) || defined(__arm64__) +#ifdef __APPLE__ + return 1; +#else + return 0; +#endif +#else + return (crypton_x86_simd_features() & CRYPTON_X86_ADX) == 0; +#endif +} +#endif + +void crypton_x25519(uint8_t out[32], const uint8_t secret[32], + const uint8_t point[32]) +{ +#ifdef CRYPTON_X25519_S2N + uint64_t r[4], s[4], p[4]; + + memcpy(s, secret, 32); + memcpy(p, point, 32); + if (use_alt()) + curve25519_x25519_alt(r, s, p); + else + curve25519_x25519(r, s, p); + memcpy(out, r, 32); +#else + crypton_curve25519_donna(out, secret, point); +#endif +} + +void crypton_x25519_base(uint8_t out[32], const uint8_t secret[32]) +{ +#ifdef CRYPTON_X25519_S2N + uint64_t r[4], s[4]; + + memcpy(s, secret, 32); + if (use_alt()) + curve25519_x25519base_alt(r, s); + else + curve25519_x25519base(r, s); + memcpy(out, r, 32); +#else + static const uint8_t nine[32] = {9}; + + crypton_curve25519_donna(out, secret, nine); +#endif +} diff --git a/cbits/curve25519/x25519.h b/cbits/curve25519/x25519.h new file mode 100644 --- /dev/null +++ b/cbits/curve25519/x25519.h @@ -0,0 +1,16 @@ +#ifndef CRYPTON_X25519_H +#define CRYPTON_X25519_H + +#include + +/* out = secret * point, the X25519 of RFC 7748: three 32-byte little-endian + * strings as they go over the wire. */ +void crypton_x25519(uint8_t out[32], const uint8_t secret[32], + const uint8_t point[32]); + +/* out = secret * G, which is the same thing with the base point 9 -- but + * where the assembly is built this reads a table instead and is four to five + * times faster, which is what a key generation costs. */ +void crypton_x25519_base(uint8_t out[32], const uint8_t secret[32]); + +#endif diff --git a/cbits/decaf/ed448goldilocks/decaf.c b/cbits/decaf/ed448goldilocks/decaf.c --- a/cbits/decaf/ed448goldilocks/decaf.c +++ b/cbits/decaf/ed448goldilocks/decaf.c @@ -1462,7 +1462,10 @@ uint32_t pos = __builtin_ctz((uint32_t)current), odd = (uint32_t)current >> pos; int32_t delta = odd & mask; if (odd & 1<<(table_bits+1)) delta -= (1<<(table_bits+1)); - current -= delta << pos; + /* delta is negative half the time and shifting a negative + * value left is undefined; current is unsigned, so the shift is + * done there and means the same thing. */ + current -= (uint64_t)delta << pos; control[position].power = pos + 16*(w-1); control[position].addend = delta; position--; diff --git a/cbits/decaf/include/word.h b/cbits/decaf/include/word.h --- a/cbits/decaf/include/word.h +++ b/cbits/decaf/include/word.h @@ -68,7 +68,7 @@ * Expand bit 0 of the given uint8_t to a mask_t all 1 or all 0 * The input must be either 0 or 1 */ -CRYPTON_DECAF_INLINE mask_t bit_to_mask(uint8_t bit) { +static CRYPTON_DECAF_INLINE mask_t bit_to_mask(uint8_t bit) { #ifdef _MSC_VER #pragma warning ( push) #pragma warning ( disable : 4146) diff --git a/cbits/ed25519/ed25519.c b/cbits/ed25519/ed25519.c --- a/cbits/ed25519/ed25519.c +++ b/cbits/ed25519/ed25519.c @@ -12,8 +12,33 @@ #include "ed25519-randombytes.h" #include "ed25519-hash.h" #include "ed25519-crypton-exts.h" +#include "ed25519/ed25519_s2n.h" /* + The base point multiplied by a scalar, packed. s2n-bignum's assembly + where it is built -- twice the speed of the table below, and signing + does this twice -- and ed25519-donna's own table where it is not. +*/ +static void +ed25519_base_pack(ed25519_public_key out, const bignum256modm s) { + ge25519 ALIGN(16) p; + +#if defined(CRYPTON_S2N_BIGNUM) && defined(__BYTE_ORDER__) \ + && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ + unsigned char e[32]; + + contract256_modm(e, s); + if (crypton_ed25519_base_mult(out, e)) { + memset(e, 0, sizeof e); + return; + } + memset(e, 0, sizeof e); +#endif + ge25519_scalarmult_base_niels(&p, ge25519_niels_base_multiples, s); + ge25519_pack(out, &p); +} + +/* Generates a (extsk[0..31]) and aExt (extsk[32..63]) */ @@ -38,14 +63,12 @@ void ED25519_FN(ed25519_publickey) (const ed25519_secret_key sk, ed25519_public_key pk) { bignum256modm a; - ge25519 ALIGN(16) A; hash_512bits extsk; /* A = aB */ ed25519_extsk(extsk, sk); expand256_modm(a, extsk, 32); - ge25519_scalarmult_base_niels(&A, ge25519_niels_base_multiples, a); - ge25519_pack(pk, &A); + ed25519_base_pack(pk, a); } @@ -53,7 +76,6 @@ ED25519_FN(ed25519_sign) (const unsigned char *m, size_t mlen, const ed25519_secret_key sk, const ed25519_public_key pk, ed25519_signature RS) { ed25519_hash_context ctx; bignum256modm r, S, a; - ge25519 ALIGN(16) R; hash_512bits extsk, hashr, hram; ed25519_extsk(extsk, sk); @@ -66,8 +88,7 @@ expand256_modm(r, hashr, 64); /* R = rB */ - ge25519_scalarmult_base_niels(&R, ge25519_niels_base_multiples, r); - ge25519_pack(RS, &R); + ed25519_base_pack(RS, r); /* S = H(R,A,m).. */ ed25519_hram(hram, RS, pk, m, mlen); diff --git a/cbits/ed25519/ed25519_s2n.c b/cbits/ed25519/ed25519_s2n.c new file mode 100644 --- /dev/null +++ b/cbits/ed25519/ed25519_s2n.c @@ -0,0 +1,65 @@ +/* + * Ed25519's base point multiplication through the vendored s2n-bignum. + * + * Signing does this twice: once for the nonce's point R, and once for the + * public key, which crypton derives from the secret key at every signature + * rather than trusting the one it is handed. Both go through here, so both + * halves of a signature move at once. + * + * Measured on an Apple M4, one multiplication with the encoding: + * ed25519-donna 7.5 us against s2n-bignum 3.5. + */ +#include + +#include "ed25519/ed25519_s2n.h" + +#if defined(CRYPTON_S2N_BIGNUM) && defined(__BYTE_ORDER__) \ + && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ +#define CRYPTON_ED25519_S2N 1 +#include "crypton_cpu.h" + +extern void edwards25519_scalarmulbase(uint64_t res[8], + const uint64_t scalar[4]); +extern void edwards25519_scalarmulbase_alt(uint64_t res[8], + const uint64_t scalar[4]); +extern void edwards25519_encode(uint8_t z[32], const uint64_t p[8]); + +/* The same question as everywhere else in cbits/s2n: a microarchitecture one + * on ARM that no feature bit answers, and exactly a feature bit on x86-64. */ +static int use_alt(void) +{ +#if defined(__aarch64__) || defined(__arm64__) +#ifdef __APPLE__ + return 1; +#else + return 0; +#endif +#else + return (crypton_x86_simd_features() & CRYPTON_X86_ADX) == 0; +#endif +} +#endif + +int crypton_ed25519_base_mult(uint8_t out[32], const uint8_t scalar[32]) +{ +#ifdef CRYPTON_ED25519_S2N + /* the assembly takes four little-endian 64-bit words, which is the + * same bits as the 32 little-endian bytes the scalar is kept in */ + uint64_t s[4], p[8]; + + memcpy(s, scalar, 32); + if (use_alt()) + edwards25519_scalarmulbase_alt(p, s); + else + edwards25519_scalarmulbase(p, s); + edwards25519_encode(out, p); + + memset(s, 0, sizeof s); + memset(p, 0, sizeof p); + return 1; +#else + (void) out; + (void) scalar; + return 0; +#endif +} diff --git a/cbits/ed25519/ed25519_s2n.h b/cbits/ed25519/ed25519_s2n.h new file mode 100644 --- /dev/null +++ b/cbits/ed25519/ed25519_s2n.h @@ -0,0 +1,14 @@ +#ifndef CRYPTON_ED25519_S2N_H +#define CRYPTON_ED25519_S2N_H + +#include + +/* The base point multiplied by a scalar, packed into Ed25519's 32-byte + * encoding. The scalar is 32 little-endian bytes, already reduced. + * + * Returns 1 when the vendored s2n-bignum did the work and 0 when it is not + * built, in which case the caller multiplies the base point itself. + */ +int crypton_ed25519_base_mult(uint8_t out[32], const uint8_t scalar[32]); + +#endif diff --git a/cbits/include32/p256/p256.h b/cbits/include32/p256/p256.h --- a/cbits/include32/p256/p256.h +++ b/cbits/include32/p256/p256.h @@ -83,16 +83,9 @@ const crypton_p256_int* b, crypton_p256_int* c); -// b := 1 / a % MOD -// MOD best be SECP256r1_n -void crypton_p256_modinv( - const crypton_p256_int* MOD, - const crypton_p256_int* a, - crypton_p256_int* b); - -// b := 1 / a % MOD +// b := 1 / a % MOD, in time that depends on a // MOD best be SECP256r1_n -// Faster than crypton_p256_modinv() +// Answers zero for an a that has no inverse, which is zero and MOD void crypton_p256_modinv_vartime( const crypton_p256_int* MOD, const crypton_p256_int* a, @@ -130,13 +123,6 @@ void crypton_p256_base_point_mul(const crypton_p256_int *n, crypton_p256_int *out_x, crypton_p256_int *out_y); - -// {out_x,out_y} := n{in_x,in_y} -void crypton_p256_point_mul(const crypton_p256_int *n, - const crypton_p256_int *in_x, - const crypton_p256_int *in_y, - crypton_p256_int *out_x, - crypton_p256_int *out_y); // {out_x,out_y} := n1G + n2{in_x,in_y} void crypton_p256_points_mul_vartime( diff --git a/cbits/include32/p256/p256_gf.h b/cbits/include32/p256/p256_gf.h --- a/cbits/include32/p256/p256_gf.h +++ b/cbits/include32/p256/p256_gf.h @@ -76,6 +76,13 @@ 0 }; static const felem kZero = {0}; + +/* the curve's b, in Montgomery form, for the complete addition formula */ +static const felem kB = { + 0x13897bbf, 0x9cdf622, 0x43090d8, 0x2e67c4, + 0x176b5678, 0x2afdc84, 0xd196888, 0xb090e90, + 0xb8600c3 +}; static const felem kP = { 0x1fffffff, 0xfffffff, 0x1fffffff, 0x3ff, 0, 0, 0x200000, 0xf000000, @@ -86,96 +93,99 @@ 0, 0, 0x400000, 0xe000000, 0x1fffffff }; -/* kPrecomputed contains precomputed values to aid the calculation of scalar - * multiples of the base point, G. It's actually two, equal length, tables - * concatenated. +/* kPrecomputed holds the multiples of the base point G that the comb in + * scalar_base_mult reads. Two tables of sixteen affine points, one after the + * other. * - * The first table contains (x,y) felem pairs for 16 multiples of the base - * point, G. + * The comb takes five bits of the signed all-bits-set representation at a + * time, from positions 52 apart, and the two tables are offset from each + * other by 26: * - * Index | Index (binary) | Value - * 0 | 0000 | 0G (all zeros, omitted) - * 1 | 0001 | G - * 2 | 0010 | 2**64G - * 3 | 0011 | 2**64G + G - * 4 | 0100 | 2**128G - * 5 | 0101 | 2**128G + G - * 6 | 0110 | 2**128G + 2**64G - * 7 | 0111 | 2**128G + 2**64G + G - * 8 | 1000 | 2**192G - * 9 | 1001 | 2**192G + G - * 10 | 1010 | 2**192G + 2**64G - * 11 | 1011 | 2**192G + 2**64G + G - * 12 | 1100 | 2**192G + 2**128G - * 13 | 1101 | 2**192G + 2**128G + G - * 14 | 1110 | 2**192G + 2**128G + 2**64G - * 15 | 1111 | 2**192G + 2**128G + 2**64G + G + * first table i, 52+i, 104+i, 156+i, 208+i + * second table 26+i, 78+i, 130+i, 182+i, 234+i * - * The second table follows the same style, but the terms are 2**32G, - * 2**96G, 2**160G, 2**224G. + * for i from 25 down to 0, which covers all 260 bits between them. * + * Every digit of that representation is +-1, so a block of five teeth takes + * one of thirty-two values -- and they come in pairs that differ only by + * sign. So sixteen entries are enough: the top tooth is taken positive, bit + * j of the index says that tooth j agrees with it, and where the top tooth is + * negative the caller negates y, which costs a subtraction. Entry zero is a + * point like any other here, unlike the unsigned table this replaces, where + * it stood for the infinity. + * + * Index | Index (binary) | Value + * 0 | 0000 | 2**208G - 2**156G - 2**104G - 2**52G - G + * 1 | 0001 | 2**208G - 2**156G - 2**104G - 2**52G + G + * ... | ... | ... + * 15 | 1111 | 2**208G + 2**156G + 2**104G + 2**52G + G + * * This is ~2KB of data. */ -static const limb kPrecomputed[NLIMBS * 2 * 15 * 2] = { - 0x11522878, 0xe730d41, 0xdb60179, 0x4afe2ff, 0x12883add, 0xcaddd88, 0x119e7edc, 0xd4a6eab, 0x3120bee, - 0x1d2aac15, 0xf25357c, 0x19e45cdd, 0x5c721d0, 0x1992c5a5, 0xa237487, 0x154ba21, 0x14b10bb, 0xae3fe3, - 0xd41a576, 0x922fc51, 0x234994f, 0x60b60d3, 0x164586ae, 0xce95f18, 0x1fe49073, 0x3fa36cc, 0x5ebcd2c, - 0xb402f2f, 0x15c70bf, 0x1561925c, 0x5a26704, 0xda91e90, 0xcdc1c7f, 0x1ea12446, 0xe1ade1e, 0xec91f22, - 0x26f7778, 0x566847e, 0xa0bec9e, 0x234f453, 0x1a31f21a, 0xd85e75c, 0x56c7109, 0xa267a00, 0xb57c050, - 0x98fb57, 0xaa837cc, 0x60c0792, 0xcfa5e19, 0x61bab9e, 0x589e39b, 0xa324c5, 0x7d6dee7, 0x2976e4b, - 0x1fc4124a, 0xa8c244b, 0x1ce86762, 0xcd61c7e, 0x1831c8e0, 0x75774e1, 0x1d96a5a9, 0x843a649, 0xc3ab0fa, - 0x6e2e7d5, 0x7673a2a, 0x178b65e8, 0x4003e9b, 0x1a1f11c2, 0x7816ea, 0xf643e11, 0x58c43df, 0xf423fc2, - 0x19633ffa, 0x891f2b2, 0x123c231c, 0x46add8c, 0x54700dd, 0x59e2b17, 0x172db40f, 0x83e277d, 0xb0dd609, - 0xfd1da12, 0x35c6e52, 0x19ede20c, 0xd19e0c0, 0x97d0f40, 0xb015b19, 0x449e3f5, 0xe10c9e, 0x33ab581, - 0x56a67ab, 0x577734d, 0x1dddc062, 0xc57b10d, 0x149b39d, 0x26a9e7b, 0xc35df9f, 0x48764cd, 0x76dbcca, - 0xca4b366, 0xe9303ab, 0x1a7480e7, 0x57e9e81, 0x1e13eb50, 0xf466cf3, 0x6f16b20, 0x4ba3173, 0xc168c33, - 0x15cb5439, 0x6a38e11, 0x73658bd, 0xb29564f, 0x3f6dc5b, 0x53b97e, 0x1322c4c0, 0x65dd7ff, 0x3a1e4f6, - 0x14e614aa, 0x9246317, 0x1bc83aca, 0xad97eed, 0xd38ce4a, 0xf82b006, 0x341f077, 0xa6add89, 0x4894acd, - 0x9f162d5, 0xf8410ef, 0x1b266a56, 0xd7f223, 0x3e0cb92, 0xe39b672, 0x6a2901a, 0x69a8556, 0x7e7c0, - 0x9b7d8d3, 0x309a80, 0x1ad05f7f, 0xc2fb5dd, 0xcbfd41d, 0x9ceb638, 0x1051825c, 0xda0cf5b, 0x812e881, - 0x6f35669, 0x6a56f2c, 0x1df8d184, 0x345820, 0x1477d477, 0x1645db1, 0xbe80c51, 0xc22be3e, 0xe35e65a, - 0x1aeb7aa0, 0xc375315, 0xf67bc99, 0x7fdd7b9, 0x191fc1be, 0x61235d, 0x2c184e9, 0x1c5a839, 0x47a1e26, - 0xb7cb456, 0x93e225d, 0x14f3c6ed, 0xccc1ac9, 0x17fe37f3, 0x4988989, 0x1a90c502, 0x2f32042, 0xa17769b, - 0xafd8c7c, 0x8191c6e, 0x1dcdb237, 0x16200c0, 0x107b32a1, 0x66c08db, 0x10d06a02, 0x3fc93, 0x5620023, - 0x16722b27, 0x68b5c59, 0x270fcfc, 0xfad0ecc, 0xe5de1c2, 0xeab466b, 0x2fc513c, 0x407f75c, 0xbaab133, - 0x9705fe9, 0xb88b8e7, 0x734c993, 0x1e1ff8f, 0x19156970, 0xabd0f00, 0x10469ea7, 0x3293ac0, 0xcdc98aa, - 0x1d843fd, 0xe14bfe8, 0x15be825f, 0x8b5212, 0xeb3fb67, 0x81cbd29, 0xbc62f16, 0x2b6fcc7, 0xf5a4e29, - 0x13560b66, 0xc0b6ac2, 0x51ae690, 0xd41e271, 0xf3e9bd4, 0x1d70aab, 0x1029f72, 0x73e1c35, 0xee70fbc, - 0xad81baf, 0x9ecc49a, 0x86c741e, 0xfe6be30, 0x176752e7, 0x23d416, 0x1f83de85, 0x27de188, 0x66f70b8, - 0x181cd51f, 0x96b6e4c, 0x188f2335, 0xa5df759, 0x17a77eb6, 0xfeb0e73, 0x154ae914, 0x2f3ec51, 0x3826b59, - 0xb91f17d, 0x1c72949, 0x1362bf0a, 0xe23fddf, 0xa5614b0, 0xf7d8f, 0x79061, 0x823d9d2, 0x8213f39, - 0x1128ae0b, 0xd095d05, 0xb85c0c2, 0x1ecb2ef, 0x24ddc84, 0xe35e901, 0x18411a4a, 0xf5ddc3d, 0x3786689, - 0x52260e8, 0x5ae3564, 0x542b10d, 0x8d93a45, 0x19952aa4, 0x996cc41, 0x1051a729, 0x4be3499, 0x52b23aa, - 0x109f307e, 0x6f5b6bb, 0x1f84e1e7, 0x77a0cfa, 0x10c4df3f, 0x25a02ea, 0xb048035, 0xe31de66, 0xc6ecaa3, - 0x28ea335, 0x2886024, 0x1372f020, 0xf55d35, 0x15e4684c, 0xf2a9e17, 0x1a4a7529, 0xcb7beb1, 0xb2a78a1, - 0x1ab21f1f, 0x6361ccf, 0x6c9179d, 0xb135627, 0x1267b974, 0x4408bad, 0x1cbff658, 0xe3d6511, 0xc7d76f, - 0x1cc7a69, 0xe7ee31b, 0x54fab4f, 0x2b914f, 0x1ad27a30, 0xcd3579e, 0xc50124c, 0x50daa90, 0xb13f72, - 0xb06aa75, 0x70f5cc6, 0x1649e5aa, 0x84a5312, 0x329043c, 0x41c4011, 0x13d32411, 0xb04a838, 0xd760d2d, - 0x1713b532, 0xbaa0c03, 0x84022ab, 0x6bcf5c1, 0x2f45379, 0x18ae070, 0x18c9e11e, 0x20bca9a, 0x66f496b, - 0x3eef294, 0x67500d2, 0xd7f613c, 0x2dbbeb, 0xb741038, 0xe04133f, 0x1582968d, 0xbe985f7, 0x1acbc1a, - 0x1a6a939f, 0x33e50f6, 0xd665ed4, 0xb4b7bd6, 0x1e5a3799, 0x6b33847, 0x17fa56ff, 0x65ef930, 0x21dc4a, - 0x2b37659, 0x450fe17, 0xb357b65, 0xdf5efac, 0x15397bef, 0x9d35a7f, 0x112ac15f, 0x624e62e, 0xa90ae2f, - 0x107eecd2, 0x1f69bbe, 0x77d6bce, 0x5741394, 0x13c684fc, 0x950c910, 0x725522b, 0xdc78583, 0x40eeabb, - 0x1fde328a, 0xbd61d96, 0xd28c387, 0x9e77d89, 0x12550c40, 0x759cb7d, 0x367ef34, 0xae2a960, 0x91b8bdc, - 0x93462a9, 0xf469ef, 0xb2e9aef, 0xd2ca771, 0x54e1f42, 0x7aaa49, 0x6316abb, 0x2413c8e, 0x5425bf9, - 0x1bed3e3a, 0xf272274, 0x1f5e7326, 0x6416517, 0xea27072, 0x9cedea7, 0x6e7633, 0x7c91952, 0xd806dce, - 0x8e2a7e1, 0xe421e1a, 0x418c9e1, 0x1dbc890, 0x1b395c36, 0xa1dc175, 0x1dc4ef73, 0x8956f34, 0xe4b5cf2, - 0x1b0d3a18, 0x3194a36, 0x6c2641f, 0xe44124c, 0xa2f4eaa, 0xa8c25ba, 0xf927ed7, 0x627b614, 0x7371cca, - 0xba16694, 0x417bc03, 0x7c0a7e3, 0x9c35c19, 0x1168a205, 0x8b6b00d, 0x10e3edc9, 0x9c19bf2, 0x5882229, - 0x1b2b4162, 0xa5cef1a, 0x1543622b, 0x9bd433e, 0x364e04d, 0x7480792, 0x5c9b5b3, 0xe85ff25, 0x408ef57, - 0x1814cfa4, 0x121b41b, 0xd248a0f, 0x3b05222, 0x39bb16a, 0xc75966d, 0xa038113, 0xa4a1769, 0x11fbc6c, - 0x917e50e, 0xeec3da8, 0x169d6eac, 0x10c1699, 0xa416153, 0xf724912, 0x15cd60b7, 0x4acbad9, 0x5efc5fa, - 0xf150ed7, 0x122b51, 0x1104b40a, 0xcb7f442, 0xfbb28ff, 0x6ac53ca, 0x196142cc, 0x7bf0fa9, 0x957651, - 0x4e0f215, 0xed439f8, 0x3f46bd5, 0x5ace82f, 0x110916b6, 0x6db078, 0xffd7d57, 0xf2ecaac, 0xca86dec, - 0x15d6b2da, 0x965ecc9, 0x1c92b4c2, 0x1f3811, 0x1cb080f5, 0x2d8b804, 0x19d1c12d, 0xf20bd46, 0x1951fa7, - 0xa3656c3, 0x523a425, 0xfcd0692, 0xd44ddc8, 0x131f0f5b, 0xaf80e4a, 0xcd9fc74, 0x99bb618, 0x2db944c, - 0xa673090, 0x1c210e1, 0x178c8d23, 0x1474383, 0x10b8743d, 0x985a55b, 0x2e74779, 0x576138, 0x9587927, - 0x133130fa, 0xbe05516, 0x9f4d619, 0xbb62570, 0x99ec591, 0xd9468fe, 0x1d07782d, 0xfc72e0b, 0x701b298, - 0x1863863b, 0x85954b8, 0x121a0c36, 0x9e7fedf, 0xf64b429, 0x9b9d71e, 0x14e2f5d8, 0xf858d3a, 0x942eea8, - 0xda5b765, 0x6edafff, 0xa9d18cc, 0xc65e4ba, 0x1c747e86, 0xe4ea915, 0x1981d7a1, 0x8395659, 0x52ed4e2, - 0x87d43b7, 0x37ab11b, 0x19d292ce, 0xf8d4692, 0x18c3053f, 0x8863e13, 0x4c146c0, 0x6bdf55a, 0x4e4457d, - 0x16152289, 0xac78ec2, 0x1a59c5a2, 0x2028b97, 0x71c2d01, 0x295851f, 0x404747b, 0x878558d, 0x7d29aa4, - 0x13d8341f, 0x8daefd7, 0x139c972d, 0x6b7ea75, 0xd4a9dde, 0xff163d8, 0x81d55d7, 0xa5bef68, 0xb7b30d8, - 0xbe73d6f, 0xaa88141, 0xd976c81, 0x7e7a9cc, 0x18beb771, 0xd773cbd, 0x13f51951, 0x9d0c177, 0x1c49a78, +static const limb kPrecomputed[NLIMBS * 2 * 16 * 2] = { + 0xe01bd76, 0xa0be8b3, 0x8494c1d, 0x609ab3d, 0x1188042f, 0x499c03d, 0x1df7cd26, 0x51b33c5, 0x1fb3bce, + 0x39cdd45, 0xdc0dd9b, 0xe3053d7, 0x1ffaf46, 0x9ac284a, 0xac051d4, 0x1c09fe1b, 0x8227cbf, 0x5bf049b, + 0xb9487d, 0x2ecb75f, 0x194825bf, 0xd70cf28, 0x14e528f3, 0x4d8670c, 0x35bbabb, 0x6b692ca, 0xd96d08, + 0x1db081dc, 0xa87ea7b, 0x190e5549, 0xa4cf420, 0x1e151385, 0xaf3d4bd, 0x4057e9f, 0x5078feb, 0x154519a, + 0xbf15dea, 0x453fa25, 0x1171c85a, 0x576c824, 0x154e7060, 0x71ede6e, 0x160467a2, 0xdea8a44, 0x81ffcb1, + 0x61a56fa, 0x76119b9, 0x110bfb9b, 0x3d527ea, 0x1997bdb4, 0xe1d1253, 0x180ce91c, 0x11950ee, 0x53d5938, + 0x694e7c9, 0xe0cf337, 0x16d8ae50, 0x202517f, 0x4d02e16, 0xd13b5fd, 0xfae97eb, 0xa1c7f60, 0x1206fe8, + 0x11b1c908, 0xf8a82f, 0x6ab17a0, 0x48058e8, 0x2d0feb, 0xfada550, 0x658edb9, 0xa17567a, 0x8daa44d, + 0x6361dc9, 0xec00c0e, 0x151a7b1c, 0xb35a683, 0x1643fe02, 0x70155c0, 0x1f131d45, 0x3998068, 0x25beef8, + 0x88138de, 0x8995ce4, 0x18565c50, 0x60f12b6, 0xc47a656, 0x4a82bd9, 0xb547a17, 0xb333474, 0xe86513a, + 0x7f8d2bc, 0x7f0f16c, 0x8cde475, 0x1ac3d5c, 0x1a832c9a, 0x6a93e7a, 0x19833281, 0xcec82db, 0x4f08cc, + 0x72c4394, 0x4686520, 0x1e845ce, 0xfb181a1, 0xf5135a5, 0xa1265d6, 0x6c63ce8, 0xe81797e, 0x5dbcd5a, + 0x2a5d603, 0x1ad4e91, 0xc86e1b3, 0x793abea, 0x1b8610a4, 0x8d5b975, 0x74dd850, 0xbbca81, 0xd7c35d8, + 0x1bab7afc, 0x4df749, 0x4acb4ea, 0xfae8c89, 0x14552ae5, 0x6dc1c20, 0x14f629f, 0x2368fe5, 0xe5a9cba, + 0x67576f7, 0x9c77c50, 0x1d63c92a, 0xbbb9ef8, 0xa7530d4, 0x963335, 0xfa09c54, 0xb6d03e3, 0xed1a022, + 0x19c59f49, 0x45c823d, 0x17a28df1, 0x80ae516, 0xe2ada82, 0x19b97fb, 0x13a9ebf, 0xf1e7606, 0xde03632, + 0x14e318b9, 0x7b57b83, 0x51a9a92, 0x3378a17, 0x1cde9289, 0x45956c2, 0x2bbab5e, 0x780b1f5, 0x8356034, + 0x75eca28, 0x6f39648, 0xf2fdbda, 0x4c65cd9, 0xb3759e7, 0x710c0b1, 0xda24432, 0x8d236aa, 0xf4c449f, + 0xaf9ba24, 0xb83ea7f, 0x1e46ecc3, 0x3f277b0, 0x6acdecd, 0x1f597d1, 0x8483d72, 0x57d29dd, 0x66d4060, + 0x167c14af, 0xc142ded, 0xc1689ca, 0x651aa52, 0x8d05768, 0x9709cfa, 0x165fd283, 0x9f6f583, 0x9833b54, + 0xd17e2d6, 0x2915c32, 0x1970ef24, 0xe8ca45b, 0x11c29e09, 0x8121f26, 0xb5afbe1, 0x10c80ec, 0x834a25c, + 0xa37f0b0, 0xd6bac16, 0xed484fc, 0x8799206, 0x13db8bb1, 0xdce615c, 0x13320329, 0x79dc25, 0x914e7af, + 0x860a414, 0xb8a9434, 0x50396ce, 0x902d3e6, 0x15c152f3, 0x3753c64, 0x970c055, 0xba296fb, 0x64bd63b, + 0x118cbb94, 0x9d4274f, 0x121cdbfe, 0xb9137cf, 0xc8bddf1, 0xa1598d0, 0x446ab41, 0x11f1df2, 0x68115f1, + 0x2f1f708, 0xee35192, 0x1dfeb3fc, 0x4e1e1a6, 0x1d9adcbb, 0x6688662, 0x63ad21b, 0x9d9a9e1, 0xb0f8b4b, + 0xcd8bc3a, 0x3577d8, 0x1ffcb97d, 0xd31e8d6, 0x1c776310, 0x95b4ef7, 0x185a82ed, 0xe40bbb0, 0xbca1ea, + 0x19462e0b, 0x2252179, 0x14f14f09, 0x565e68d, 0x7ba5f37, 0x4cd1858, 0x167941b3, 0x4d1c7a7, 0x7aef1ab, + 0x1599efe9, 0x658e78d, 0x1ad33917, 0x7e74797, 0x19152edc, 0xdf7dc18, 0xdf677c, 0x9315d96, 0x4ba8eec, + 0xc45cd82, 0x1acfa03, 0xe265b49, 0xcfa6eb, 0x89d7619, 0x7b05, 0x1ba11068, 0xe1672d3, 0x655622a, + 0x1607ca6, 0x339049, 0x5454f70, 0x10edd75, 0x1133ceb7, 0xe3eec39, 0xc263156, 0xeb6ddbe, 0x10360a7, + 0xfd5f981, 0x47dd502, 0x1e66dbbe, 0x8820b63, 0xeee91ef, 0xffde293, 0xb66325d, 0x3a5a2a, 0x6a2acbb, + 0x11e949bc, 0xf6a6d57, 0x6b2ca24, 0xad903ec, 0x1e55de0, 0xe7074ed, 0xe681934, 0xea44010, 0xaa490cc, + 0x512324a, 0x6a5eb00, 0xee5e100, 0xd60a02b, 0x1b89c993, 0x5cffb70, 0xa49030c, 0x405aee, 0xe1b27cc, + 0x2e73a15, 0x9ca8dc0, 0x781dbff, 0x9fd85e0, 0x1884e4c8, 0x40873f6, 0x32d4b69, 0xf42f753, 0xeaf4c38, + 0x2802a4c, 0x1283dac, 0x759100a, 0xcb3ba75, 0xf3203d3, 0xf89b8aa, 0x7caa59e, 0x384a60a, 0x37f69e7, + 0x1e56d569, 0x120c552, 0x181734aa, 0x4fcd9b4, 0x7918e4e, 0xcd7938a, 0x2bbd8b2, 0x19f4f97, 0xa7af533, + 0xbb69948, 0x3eff33e, 0x1f3ac118, 0x3739770, 0x58898fd, 0x623fafb, 0xa7e6d93, 0xd31a676, 0x615192d, + 0xf543d28, 0xe61ce5a, 0x10ae4b39, 0xcd5a8d7, 0x1b34c6de, 0x81997ad, 0x198e2093, 0xd9b6ff7, 0x9a5954f, + 0x16782589, 0xed3c1ab, 0x62dc4a5, 0xac12d0c, 0x8bc8b7c, 0x168ec4e, 0x177e11dc, 0x407df09, 0x4056e85, + 0x9858305, 0xfe445e9, 0x18b1230a, 0x815ee9f, 0xa852eb4, 0x89444e0, 0x1a83481, 0x479359b, 0x2192576, + 0x16d5e61b, 0xfe480c4, 0x1d60b8b7, 0x1c6e798, 0x1a01310, 0x9998572, 0x7c59f75, 0x49dda87, 0xe75e0b6, + 0x1b2b7536, 0xb267d8, 0x15443085, 0x45e5924, 0x7fb947f, 0x296915d, 0x38fc56b, 0x4bae39f, 0xb218e7c, + 0x115c3b16, 0x9d95f0c, 0xa50ac4c, 0xcce8037, 0xc3c7ba3, 0xf02773a, 0xbc2ad54, 0x26914c1, 0x19a4b8d, + 0x3f8a1a6, 0xa0b8459, 0x1f77a521, 0x7d93297, 0x1dddb4b2, 0x9e4cd1c, 0x6e28403, 0xd7ce413, 0x5575b62, + 0x12bb7dc1, 0xbdfb15e, 0xa542867, 0xe943d3e, 0x1367fbdd, 0x37b387, 0x14e4d75f, 0xb90b09d, 0xbf6ec28, + 0xa5182c8, 0x1e5b34e, 0xabe4602, 0x1c13efa, 0x8d1182, 0x1c2947a, 0x1e04e0e3, 0x6caecdb, 0x40f14e8, + 0x1b845e4a, 0xa9fb149, 0xb34f513, 0x3a0fdbb, 0xfad2335, 0x5bb9342, 0x18c5ad62, 0xc97fdc3, 0x31225c7, + 0xb28a9ee, 0x585915, 0x1e355da1, 0x26ed08e, 0xc7d06a, 0xa65f219, 0x1fdf45cd, 0xc8323ea, 0x297b9ee, + 0x1c031098, 0x9c39cf0, 0x1287d79f, 0x69a9e32, 0x10015650, 0x1c3dfc3, 0x12dcd848, 0x3155a59, 0xeff3212, + 0x1a6ecdd0, 0xd26bd07, 0x18e077ab, 0x442b477, 0x46b735f, 0x495d60c, 0x1b57a6e5, 0x76a368a, 0xf53bd50, + 0xf12c5e0, 0x80a9b4, 0x15562060, 0x4102113, 0xa144ab5, 0x5fa4e9, 0x1009f5e9, 0xe34343a, 0x26fda5a, + 0x159e06a4, 0x5fa3aad, 0x10259b5f, 0xa69947d, 0x1190417e, 0x987da3f, 0x14e1e868, 0xdcb7e1e, 0x8890f9f, + 0x14dece80, 0x94bbf5c, 0x18513a17, 0x4ca31ca, 0xc0a2713, 0xbc46074, 0x1536f6a5, 0x43991aa, 0xb9f8f1c, + 0x987ba48, 0xb0829ae, 0xd29d324, 0x6339c35, 0x18ace0a4, 0x5d53b55, 0xff829f4, 0xe882ecf, 0xc05164c, + 0x6bd6bba, 0x9dfc14f, 0x1981cab3, 0xcfebf18, 0x1ddac868, 0x94ec6d4, 0x5abd4b, 0x737fdb3, 0x18f531f, + 0xd3c2a71, 0x337178f, 0x9f7c32e, 0xd9d7fda, 0x137d191f, 0xdd0757b, 0x14b6d65, 0x179f37a, 0x67b10e1, + 0x1bfb2cfd, 0xfe4ca43, 0x1fee2930, 0x98c2aa0, 0x9826788, 0xeaf4ceb, 0x17a6be82, 0xc899ed1, 0x500fb01, + 0x10918e6f, 0x36179ed, 0xbca6643, 0x2d80942, 0xf1ef61, 0xadca21c, 0xbe5b3a7, 0xadae157, 0x12daac, + 0x1337dda8, 0x6326e5d, 0x2738e1b, 0x5cb5c54, 0x98ec8a0, 0x252647d, 0x1d5c173c, 0xbdf848d, 0x9e5217b, + 0x1d64f447, 0xb71d1a, 0xb2c2360, 0xccb6bee, 0x1245995e, 0x94a9130, 0x5b93d91, 0x76c57ff, 0xeaa91d1, + 0x3941881, 0xc2aafbd, 0x1c0540d0, 0x1a938f0, 0x1304b724, 0x8524e10, 0x1bef780f, 0xbc0ea48, 0xbe90dae, + 0x1d10d5d8, 0xca979e2, 0x10db5cc4, 0x54e2493, 0x44d38f3, 0xbcb73b, 0x12dcff4, 0xd0ab219, 0xde69db2, + 0x13594366, 0xc30e05f, 0xfc245d4, 0x8c5b52f, 0x81901c7, 0xa9d1e03, 0x11ead62e, 0xb7be89b, 0xc9c8486, + 0x132a6fa0, 0x56af9b8, 0x41cb561, 0xf74418c, 0x141c461a, 0xbc18514, 0x1d6bbb68, 0x96d43c2, 0x7108696 }; diff --git a/cbits/include64/p256/p256.h b/cbits/include64/p256/p256.h --- a/cbits/include64/p256/p256.h +++ b/cbits/include64/p256/p256.h @@ -83,16 +83,9 @@ const crypton_p256_int* b, crypton_p256_int* c); -// b := 1 / a % MOD -// MOD best be SECP256r1_n -void crypton_p256_modinv( - const crypton_p256_int* MOD, - const crypton_p256_int* a, - crypton_p256_int* b); - -// b := 1 / a % MOD +// b := 1 / a % MOD, in time that depends on a // MOD best be SECP256r1_n -// Faster than crypton_p256_modinv() +// Answers zero for an a that has no inverse, which is zero and MOD void crypton_p256_modinv_vartime( const crypton_p256_int* MOD, const crypton_p256_int* a, @@ -130,13 +123,6 @@ void crypton_p256_base_point_mul(const crypton_p256_int *n, crypton_p256_int *out_x, crypton_p256_int *out_y); - -// {out_x,out_y} := n{in_x,in_y} -void crypton_p256_point_mul(const crypton_p256_int *n, - const crypton_p256_int *in_x, - const crypton_p256_int *in_y, - crypton_p256_int *out_x, - crypton_p256_int *out_y); // {out_x,out_y} := n1G + n2{in_x,in_y} void crypton_p256_points_mul_vartime( diff --git a/cbits/include64/p256/p256_gf.h b/cbits/include64/p256/p256_gf.h --- a/cbits/include64/p256/p256_gf.h +++ b/cbits/include64/p256/p256_gf.h @@ -104,102 +104,111 @@ 2, 0xfc00000000000, 0x7ffffffffffff, 0xfff7fffffffff, 0x7ffff }; static const felem kZero = {0}; + +/* the curve's b, in Montgomery form, for the complete addition formula */ +static const felem kB = { + 0x1bec453897bbf, 0x33e210c243627, 0x484bb5ab3c017, 0x41a32d11055fb, + 0x2e18030ec243a +}; static const felem kP = { 0x7ffffffffffff, 0x1fffffffffff, 0, 0x4000000000, 0x3fffffffc0000 }; static const felem k2P = { 0x7fffffffffffe, 0x3fffffffffff, 0, 0x8000000000, 0x7fffffff80000 }; -/* kPrecomputed contains precomputed values to aid the calculation of scalar - * multiples of the base point, G. It's actually two, equal length, tables - * concatenated. +/* kPrecomputed holds the multiples of the base point G that the comb in + * scalar_base_mult reads. Two tables of sixteen affine points, one after the + * other. * - * The first table contains (x,y) felem pairs for 16 multiples of the base - * point, G. + * The comb takes five bits of the signed all-bits-set representation at a + * time, from positions 52 apart, and the two tables are offset from each + * other by 26: * - * Index | Index (binary) | Value - * 0 | 0000 | 0G (all zeros, omitted) - * 1 | 0001 | G - * 2 | 0010 | 2**64G - * 3 | 0011 | 2**64G + G - * 4 | 0100 | 2**128G - * 5 | 0101 | 2**128G + G - * 6 | 0110 | 2**128G + 2**64G - * 7 | 0111 | 2**128G + 2**64G + G - * 8 | 1000 | 2**192G - * 9 | 1001 | 2**192G + G - * 10 | 1010 | 2**192G + 2**64G - * 11 | 1011 | 2**192G + 2**64G + G - * 12 | 1100 | 2**192G + 2**128G - * 13 | 1101 | 2**192G + 2**128G + G - * 14 | 1110 | 2**192G + 2**128G + 2**64G - * 15 | 1111 | 2**192G + 2**128G + 2**64G + G + * first table i, 52+i, 104+i, 156+i, 208+i + * second table 26+i, 78+i, 130+i, 182+i, 234+i * - * The second table follows the same style, but the terms are 2**32G, - * 2**96G, 2**160G, 2**224G. + * for i from 25 down to 0, which covers all 260 bits between them. * + * Every digit of that representation is +-1, so a block of five teeth takes + * one of thirty-two values -- and they come in pairs that differ only by + * sign. So sixteen entries are enough: the top tooth is taken positive, bit + * j of the index says that tooth j agrees with it, and where the top tooth is + * negative the caller negates y, which costs a subtraction. Entry zero is a + * point like any other here, unlike the unsigned table this replaces, where + * it stood for the infinity. + * + * Index | Index (binary) | Value + * 0 | 0000 | 2**208G - 2**156G - 2**104G - 2**52G - G + * 1 | 0001 | 2**208G - 2**156G - 2**104G - 2**52G + G + * ... | ... | ... + * 15 | 1111 | 2**208G + 2**156G + 2**104G + 2**52G + G + * * This is ~2KB of data. */ -static const limb kPrecomputed[NLIMBS * 2 * 15 * 2] = { - 0x661a831522878, 0xf17fb6d805e79, 0x5889441d6ea57, 0xae33cfdb995bb, 0xc482fbb529ba, - 0x4a6af9d2aac15, 0x90e867917377c, 0x487cc962d2ae3, 0xec2a97443446e, 0x2b8ff8c52c42, - 0x45f8a2d41a576, 0xb06988d2653e4, 0x718b22c357305, 0x33fc920e79d2b, 0x17af34b0fe8db, - 0x38e17eb402f2f, 0x3382558649705, 0x47f6d48f482d1, 0x7bd42488d9b83, 0x3b247c8b86b78, - 0x4d08fc26f7778, 0x7a29a82fb2795, 0x75cd18f90d11a, 0xad8e213b0bc, 0x2d5f0142899e8, - 0x506f98098fb57, 0x2f0c98301e4aa, 0x39b30dd5cf67d, 0x9c146498ab13c, 0xa5db92df5b7b, - 0x184897fc4124a, 0xe3f73a19d8aa, 0x4e1c18e47066b, 0x27b2d4b52eaee, 0x30eac3ea10e99, - 0x4e74546e2e7d5, 0x1f4dde2d97a1d, 0x6ead0f88e1200, 0x7dec87c220f02, 0x3d08ff096310f, - 0x23e5659633ffa, 0x6ec648f08c722, 0x3172a3806ea35, 0xf6e5b681eb3c5, 0x2c3758260f89d, - 0x38dca4fd1da12, 0xf06067b78830d, 0x3194be87a068c, 0x78893c7eb602b, 0xcead60438432, - 0x6ee69a56a67ab, 0xd886f77701895, 0x67b0a4d9cee2b, 0x3586bbf3e4d53, 0x1db6f32921d93, - 0x260756ca4b366, 0x4f40e9d2039fa, 0x4f3f09f5a82bf, 0xccde2d641e8cd, 0x305a30cd2e8c5, - 0x471c235cb5439, 0xab279cd962f5a, 0x17e1fb6e2dd94, 0xfe64589800a77, 0xe8793d99775f, - 0x48c62f4e614aa, 0xbf76ef20eb2a4, 0x669c672556c, 0x24683e0eff056, 0x12252b369ab76, - 0x821de9f162d5, 0xf911ec99a95be, 0x6721f065c906b, 0x58d452035c736, 0x1f9f01a6a15, - 0x6135009b7d8d3, 0xdaeeeb417dfc0, 0x63865fea0ee17, 0x6e0a304b939d6, 0x204ba2076833d, - 0x4ade586f35669, 0x2c1077e34611a, 0x5b1a3bea3b81a, 0xf97d018a22c8b, 0x38d7996b08af8, - 0x6ea62baeb7aa0, 0xebdcbd9ef2670, 0x35dc8fe0df3fe, 0xe458309d20c24, 0x11e87898716a0, - 0x7c44bab7cb456, 0xd64d3cf1bb64, 0x189bff1bf9e66, 0xb5218a049311, 0x285dda6cbcc81, - 0x3238dcafd8c7c, 0x607736c8de0, 0xdb83d99508b1, 0x4e1a0d404cd81, 0x1588008c00ff2, - 0x16b8b36722b27, 0x876609c3f3f1a, 0x66b72ef0e17d6, 0x705f8a279d568, 0x2eaac4cd01fdd, - 0x1171ce9705fe9, 0xffc79cd3264ee, 0x700c8ab4b80f0, 0x208d3d4f57a1, 0x337262a8ca4eb, - 0x297fd01d843fd, 0xa90956fa097f8, 0x529759fdb3845, 0x1d78c5e2d0397, 0x3d6938a4adbf3, - 0x16d5853560b66, 0xf138946b9a430, 0x2ab79f4dea6a0, 0xd42053ee43ae1, 0x3b9c3ef1cf870, - 0x598934ad81baf, 0x5f1821b1d07a7, 0x416bb3a973ff3, 0x23f07bd0a047a, 0x19bdc2e09f786, - 0x56dc9981cd51f, 0xfbace23c8cd65, 0x673bd3bf5b52e, 0x46a95d229fd61, 0xe09ad64bcfb1, - 0xe5292b91f17d, 0xfeefcd8afc287, 0x58f52b0a58711, 0x4800f20c201ef, 0x2084fce608f67, - 0x12ba0b128ae0b, 0x5977ae17030b4, 0x101126ee420f6, 0xf70823495c6bd, 0xde19a27d7770, - 0x5c6ac852260e8, 0x9d22950ac4356, 0x441cca955246c, 0x660a34e5332d9, 0x14ac8ea92f8d2, - 0x6b6d7709f307e, 0x67d7e13879db, 0x2ea8626f9fbbd, 0x99609006a4b40, 0x31bb2a8f8c779, - 0x10c04828ea335, 0xae9acdcbc080a, 0x617af2342607a, 0xc7494ea53e553, 0x2ca9e2872defa, - 0x6c399fab21f1f, 0xab139b245e758, 0x3ad933dcba589, 0x4797fecb08811, 0x31f5dbf8f594, - 0x7dc6361cc7a69, 0xc8a7953ead3f9, 0x79ed693d18015, 0x418a024999a6a, 0x2c4fdc9436aa, - 0x1eb98cb06aa75, 0x2989592796a9c, 0x11194821e425, 0xe27a648228388, 0x35d834b6c12a0, - 0x541807713b532, 0x7ae0a1008aaee, 0x7017a29bcb5e, 0x6b193c23c315c, 0x19bd25ac82f2a, - 0x6a01a43eef294, 0xddf5b5fd84f19, 0x33f5ba081c016, 0xdeb052d1bc082, 0x6b2f06afa617, - 0x7ca1eda6a939f, 0xbdeb35997b50c, 0x47f2d1bccda5, 0xc2ff4adfed667, 0x87712997be4, - 0x21fc2e2b37659, 0xf7d62cd5ed951, 0x27fa9cbdf7efa, 0xba25582bf3a6b, 0x2a42b8bd89398, - 0x6d377d07eecd2, 0x9ca1df5af387, 0x1109e3427e2ba, 0xce4aa4572a19, 0x103baaef71e16, - 0x2c3b2dfde328a, 0xbec4b4a30e1ef, 0x37d92a86204f3, 0x806cfde68eb39, 0x246e2f72b8aa5, - 0x68d3de93462a9, 0x53b8acba6bbc3, 0x2492a70fa1696, 0x38c62d5760f55, 0x15096fe4904f2, - 0x4e44e9bed3e3a, 0xb28bfd79cc9bc, 0x6a77513839320, 0x480dcec6739db, 0x3601b739f2465, - 0x43c348e2a7e1, 0xe448106327879, 0x175d9cae1b0ed, 0xd3b89dee743b8, 0x392d73ca255bc, - 0x32946db0d3a18, 0x9261b09907cc, 0x5ba517a755722, 0x51f24fdaf5184, 0x1cdc732989ed8, - 0x2f7806ba16694, 0xae0c9f029f8d0, 0xd8b45102ce1, 0xca1c7db9316d6, 0x162088a67066f, - 0x39de35b2b4162, 0xa19f550d88ae9, 0x7921b27026cde, 0x94b936b66e900, 0x1023bd5fa17fc, - 0x436837814cfa4, 0x29113492283c4, 0x66d1cdd8b51d8, 0xa540702278eb2, 0x47ef1b29285d, - 0x587b50917e50e, 0xb4cda75bab3b, 0x112520b0a9886, 0x66b9ac16fee49, 0x17bf17e92b2eb, - 0x2456a2f150ed7, 0xfa214412d0280, 0x3ca7dd947fe5b, 0xa72c28598d58a, 0x255d945efc3e, - 0x2873f04e0f215, 0x74178fd1af57b, 0x788848b5b2d6, 0xb1ffafaae0db6, 0x32a1b7b3cbb2a, - 0x4bd9935d6b2da, 0x9c08f24ad30a5, 0x4e58407a80f, 0x1b3a3825a5b17, 0x6547e9fc82f5, - 0x47484aa3656c3, 0x6ee43f341a494, 0x64a98f87adea2, 0x619b3f8e95f01, 0xb6e513266ed8, - 0x421c2a673090, 0xa1c1de32348c7, 0x55b85c3a1e8a3, 0xe05ce8ef330b4, 0x2561e49c15d84, - 0x40aa2d33130fa, 0x12b827d35866f, 0xfe4cf62c8ddb, 0x2fa0ef05bb28d, 0x1c06ca63f1cb8, - 0x32a971863863b, 0xff6fc86830da1, 0x71e7b25a14cf3, 0xea9c5ebb1373a, 0x250bbaa3e1634, - 0x5b5ffeda5b765, 0xf25d2a746331b, 0x115e3a3f43632, 0x67303af43c9d5, 0x14bb538a0e559, - 0x75623687d43b7, 0xa349674a4b38d, 0x613c61829ffc6, 0x689828d8110c7, 0x139115f5af7d5, - 0xf1d856152289, 0x45cbe967168ab, 0x51f38e1680901, 0x34808e8f652b0, 0x1f4a6a921e156, - 0x35dfaf3d8341f, 0xf53ace725cb63, 0x3d86a54eef35b, 0xa103aabaffe2c, 0x2decc36296fbd, - 0x510282be73d6f, 0xd4e6365db206a, 0x4bdc5f5bb8bf3, 0xde7ea32a3aee7, 0x71269e274305, +static const limb kPrecomputed[NLIMBS * 2 * 16 * 2] = { + 0x17d166e01bd76, 0xd59ea12530768, 0x3d8c40217b04, 0x17bef9a4c9338, 0x7ecef3946ccf, + 0x1bb3639cdd45, 0xd7a338c14f5f7, 0x1d44d614250ff, 0xff813fc37580a, 0x16fc126e089f2, + 0x596ebe0b9487d, 0x6794652096fcb, 0x70ca729479eb8, 0x286b775769b0c, 0x365b421ada4b, + 0xfd4f7db081dc, 0x7a1064395526a, 0x4bdf0a89c2d26, 0xac80afd3f5e7a, 0x551466941e3f, + 0x27f44abf15dea, 0x641245c721691, 0x66eaa738302bb, 0x12c08cf44e3db, 0x207ff2c77aa29, + 0x42337261a56fa, 0x93f5442fee6dd, 0x253ccbdeda1ea, 0xbb019d239c3a2, 0x14f564e046543, + 0x19e66e694e7c9, 0x28bfdb62b9438, 0x5fd268170b101, 0x81f5d2fd7a276, 0x481bfa2871fd, + 0x71505f1b1c908, 0x2c741aac5e803, 0x55001687f5a40, 0xe8cb1db73f5b4, 0x236a913685d59, + 0x181c6361dc9, 0xd341d469ec73b, 0x5c0b21ff0159a, 0xa3e263a8ae02a, 0x96fbbe0e6601, + 0x32b9c888138de, 0x895b615971422, 0x3d9623d32b307, 0xd16a8f42e9505, 0x3a1944eacccd1, + 0x61e2d87f8d2bc, 0x1eae233791d5f, 0x67ad41964d0d6, 0x6f3066502d527, 0x13c23333b20b, + 0x50ca4072c4394, 0xc0d087a117391, 0x5d67a89ad2fd8, 0xf8d8c79d1424c, 0x176f356ba05e5, + 0x5a9d222a5d603, 0xd5f5321b86cc6, 0x175dc308523c9, 0x4e9bb0a11ab7, 0x35f0d7602ef2a, + 0x1bee93bab7afc, 0x464492b2d3a81, 0x420a2a9572fd7, 0x9429ec53edb83, 0x396a72e88da3f, + 0xef8a067576f7, 0xcf7c758f24aa7, 0x33553a986a5dd, 0x8df4138a812c6, 0x3b46808adb40f, + 0x39047b9c59f49, 0x728b5e8a37c51, 0x7fb7156d41405, 0x182753d7e3372, 0x3780d8cbc79d8, + 0x6af7074e318b9, 0xc50b946a6a49e, 0x6c2e6f494499b, 0xd457756bc8b2a, 0x20d580d1e02c7, + 0x672c9075eca28, 0x2e6cbcbf6f69b, 0xb159bacf3a63, 0xa9b448864e218, 0x3d31127e348da, + 0x7d4feaf9ba24, 0x3bd8791bb30ee, 0x7d13566f669f9, 0x750907ae43eb2, 0x19b501815f4a7, + 0x285bdb67c14af, 0xd529305a272b0, 0x4fa4682bb4328, 0xecbfa5072e13, 0x260ced527dbd6, + 0x22b864d17e2d6, 0x522de5c3bc90a, 0x7268e14f04f46, 0xb16b5f7c30243, 0x20d2897043203, + 0x57582ca37f0b0, 0xc9033b5213f35, 0x15c9edc5d8c3c, 0x966640653b9cc, 0x24539ebc1e770, + 0x152868860a414, 0x69f3140e5b3ae, 0x464ae0a979c81, 0xed2e180aa6ea7, 0x192f58eee8a5b, + 0x284e9f18cbb94, 0x9be7c8736ffa7, 0xd0645eef8dc8, 0xc888d568342b3, 0x1a0457c447c77, + 0x46a3242f1f708, 0xf0d377facff3b, 0x662ecd6e5da70, 0x84c75a436cd10, 0x2c3e2d2e766a7, + 0x6aefb0cd8bc3a, 0xf46b7ff2e5f40, 0x6f7e3bb188698, 0xc30b505db2b69, 0x2f287ab902ee, + 0x4a42f39462e0b, 0xf346d3c53c248, 0x583dd2f9bab2, 0x9ecf2836699a3, 0x1ebbc6ad3471e, + 0x31cf1b599efe9, 0xa3cbeb4ce45d9, 0x418c8a976e3f3, 0x581becef9befb, 0x12ea3bb24c576, + 0x59f406c45cd82, 0xd375b8996d246, 0x30544ebb0c867, 0x4f74220d0000f, 0x195588ab859cb, + 0x6720921607ca6, 0x6eba95153dc00, 0x439899e75b887, 0xf984c62adc7dd, 0x40d829fadb76, + 0x7baa04fd5f981, 0x5b1f99b6ef91, 0x29377748f7c41, 0xa96cc64bbffbc, 0x1a8ab2ec0e968, + 0x54daaf1e949bc, 0x81f61acb2893d, 0x4ed0f2aef056c, 0x41cd03269ce0e, 0x2a924333a9100, + 0x4bd600512324a, 0x5015bb978401a, 0x370dc4e4c9eb0, 0xb94920618b9ff, 0x386c9f301016b, + 0x151b802e73a15, 0xc2f01e076ffe7, 0x3f6c4272644fe, 0x4c65a96d2810e, 0x3abd30e3d0bdd, + 0x507b582802a4c, 0xdd3a9d6440284, 0xaa79901e9e59, 0x28f954b3df137, 0xdfda79ce1298, + 0x418aa5e56d569, 0x6cda605cd2a84, 0x38a3c8c72727e, 0x5c577b1659af2, 0x29ebd4cc67d3e, + 0x5fe67cbb69948, 0xcbb87ceb0460f, 0x2fb2c44c7e9b9, 0xd94fcdb26c47f, 0x185464b74c699, + 0x439cb4f543d28, 0xd46bc2b92ce79, 0x7add9a636f66a, 0xdf31c41270332, 0x2696553f66dbf, + 0x2783576782589, 0x968618b71297b, 0x44e45e45be560, 0x26efc23b82d1d, 0x1015ba1501f7c, + 0x488bd29858305, 0xf74fe2c48c2bf, 0x4e0542975a40a, 0x6c35069031288, 0x86495d91e4d6, + 0x4901896d5e61b, 0x73cc7582e2dff, 0x5720d009880e3, 0x1cf8b3eeb3330, 0x39d782d92776a, + 0x64cfb1b2b7536, 0x2c925510c2142, 0x15d3fdca3fa2f, 0x7c71f8ad652d2, 0x2c8639f12eb8e, + 0x32be1915c3b16, 0x401ba942b1327, 0x73a61e3dd1e67, 0x57855aa9e04e, 0x6692e349a453, + 0x1708b23f8a1a6, 0x994bfdde94868, 0x51ceeeda593ec, 0x4cdc508073c99, 0x155d6d8b5f390, + 0x3f62bd2bb7dc1, 0x1e9f2950a19ef, 0x3879b3fdeef4a, 0x769c9aebe06f6, 0x2fdbb0a2e42c2, + 0x4b669ca5182c8, 0x9f7d2af918087, 0x47a04688c10e0, 0x6fc09c1c63852, 0x103c53a1b2bb3, + 0x3f6293b845e4a, 0x7eddacd3d44ea, 0x3427d6919a9d0, 0xf18b5ac4b772, 0xc48971f25ff7, + 0x30b22ab28a9ee, 0x684778d576841, 0x219063e835137, 0xabfbe8b9b4cbe, 0xa5ee7bb20c8f, + 0x739e1c031098, 0x4f194a1f5e7e7, 0x7c3800ab2834d, 0x665b9b090387b, 0x3bfcc848c5569, + 0x4d7a0fa6ecdd0, 0x5a3be381deaf4, 0x60c235b9afa21, 0x2b6af4dca92ba, 0x3d4ef541da8da, + 0x15368f12c5e0, 0x1089d55881802, 0x4e950a255aa08, 0xea013ebd20bf4, 0x9bf696b8d0d0, + 0x74755b59e06a4, 0xca3ec0966d7d7, 0x23f8c820bf534, 0x7a9c3d0d130fb, 0x22243e7f72df8, + 0x177eb94dece80, 0x18e56144e85e5, 0x746051389a65, 0xaaa6ded4b788c, 0x2e7e3c710e646, + 0x10535c987ba48, 0xce1ab4a74c92c, 0x355c567052319, 0x3dff053e8baa7, 0x30145933a20bb, + 0x3f829e6bd6bba, 0x5f8c66072ace7, 0x6d4eed643467f, 0xcc0b57a9729d8, 0x63d4c7dcdff6, + 0x6e2f1ed3c2a71, 0xbfed27df0cb8c, 0x57b9be8c8fece, 0xe8296dacbba0e, 0x19ec43845e7cd, + 0x499487bfb2cfd, 0x15507fb8a4c3f, 0x4eb4c133c44c6, 0x46f4d7d05d5e9, 0x1403ec072267b, + 0x42f3db0918e6f, 0x4a12f29990cd, 0x21c078f7b096c, 0x5d7cb674f5b94, 0x4b6ab2b6b85, + 0x64dcbb337dda8, 0xae2a09ce386d8, 0x47d4c764502e5, 0x37ab82e784a4c, 0x279485eef7e12, + 0x6e3a35d64f447, 0xb5f72cb08d802, 0x130922ccaf665, 0xfcb727b232952, 0x3aaa4745db15f, + 0x555f7a3941881, 0x9c78701503430, 0x6109825b920d4, 0x237def01f0a49, 0x2fa436baf03a9, + 0x52f3c5d10d5d8, 0x1249c36d73132, 0x73b2269c79aa7, 0x6425b9fe81796, 0x379a76cb42ac8, + 0x61c0bf3594366, 0xda97bf0917530, 0x60340c80e3c62, 0x6e3d5ac5d53a3, 0x3272121adefa2, + 0x55f37132a6fa0, 0x20c61072d5855, 0x514a0e230d7ba, 0xbad776d17830, 0x1c421a5a5b50f }; diff --git a/cbits/include64/p256/p256_s2n.h b/cbits/include64/p256/p256_s2n.h new file mode 100644 --- /dev/null +++ b/cbits/include64/p256/p256_s2n.h @@ -0,0 +1,25 @@ +/* + * The vendored s2n-bignum assembly, behind one call that picks the variant + * the machine wants. See cbits/s2n/README.md for which and why. + * + * Only declared in the 64-bit field build: s2n-bignum is x86-64 and AArch64 + * only, and this interface is the four little-endian 64-bit words those + * architectures give crypton_p256_int anyway. + */ +#ifndef CRYPTON_P256_S2N_H +#define CRYPTON_P256_S2N_H + +#include + +/* res = scalar * point, all of them affine and not in Montgomery form: + * point is x then y, four words each, and res the same. The point at + * infinity goes in and comes out as (0, 0). */ +void crypton_s2n_p256_scalarmul(uint64_t res[8], const uint64_t scalar[4], + const uint64_t point[8]); + +/* res = scalar * G, the same shape out. The table it reads and the window + * width it was built for are in cbits/p256/p256_base_table.c, which + * cbits/p256/gen_base_table.py writes. */ +void crypton_s2n_p256_scalarmulbase(uint64_t res[8], const uint64_t scalar[4]); + +#endif diff --git a/cbits/mldsa/COMMIT b/cbits/mldsa/COMMIT new file mode 100644 --- /dev/null +++ b/cbits/mldsa/COMMIT @@ -0,0 +1,2 @@ +834a90d5e846ffa1e1611bd24e160bb2e9b86d35 +v2.0.0 diff --git a/cbits/mldsa/LICENSE b/cbits/mldsa/LICENSE new file mode 100644 --- /dev/null +++ b/cbits/mldsa/LICENSE @@ -0,0 +1,305 @@ +mldsa-native is a fork of the public domain Dilithium reference implementation, +available on https://github.com/pq-crystals/dilithium. + +All new files and all files derived from the Dilithium reference +implementation are made available under the Apache-2.0 license OR +the ISC license OR the MIT license. These licenses are +reproduced at the bottom of this file. + +Files outside the library itself may carry different terms. Every file +states its own SPDX-License-Identifier, which determines the terms that +apply to it. In particular: + +The code in test/notrandombytes/*, and its copies in +examples/*/test_only_rng/* and scripts/notrandombytes, is derived from +https://cr.yp.to/papers.html#surf and licensed under +LicenseRef-PD-hp OR CC0-1.0 OR 0BSD OR MIT-0 OR MIT. +It is only used for testing purposes. + +The benchmarking code in test/hal/* carries the +MIT license. It is only used for testing purposes. + +The tiny_sha3 code in +examples/bring_your_own_fips202/custom_fips202/tiny_sha3/* and +examples/custom_backend/mldsa_native/src/fips202/native/custom/src/* +carries the MIT license. It is only used to demonstrate custom +FIPS-202 implementations. + +The proofs in proofs/* are in part derived from Amazon Web Services +verification infrastructure and licensed under +Apache-2.0 OR ISC OR MIT-0, MIT-0, or MIT-0 AND Apache-2.0. The IACR +document class proofs/isabelle/neon_ntt/document/iacrtrans.cls is +licensed under CC0-1.0. None of this is part of the library. + +Documentation is licensed under CC-BY-4.0. + +``` +Copyright (c) The mldsa-native project authors +Copyright (c) The mlkem-native project authors +Copyright (c) 2020 Dougall Johnson +Copyright (c) 2022 Arm Limited +SPDX-License-Identifier: MIT + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +``` + +ISC license for mldsa-native content +------------------------------------ + +Copyright (c) The mldsa-native project authors + +Permission to use, copy, modify, and/or distribute this software for any purpose +with or without fee is hereby granted, provided that the above copyright notice +and this permission notice appear in all copies. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH +REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, +INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS +OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER +TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF +THIS SOFTWARE. + + +MIT license for mldsa-native content +------------------------------------ + +Copyright (c) The mldsa-native project authors + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the “Software”), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Apache-2.0 license for mldsa-native content +------------------------------------------- + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright (c) The mldsa-native project authors + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/cbits/mldsa/README.md b/cbits/mldsa/README.md new file mode 100644 --- /dev/null +++ b/cbits/mldsa/README.md @@ -0,0 +1,60 @@ +# mldsa-native + +ML-DSA (FIPS 204) from the PQ Code Package, vendored here and built for all +three parameter sets. + +## Where it comes from + +. `COMMIT` holds the +revision this tree is at; `import.sh` puts it there and is how the tree is +refreshed. Nothing here is edited by hand -- every choice crypton makes is +made in `crypton_mldsa.h`, `crypton_mldsa.c` and `crypton.cabal`, so that a +re-import is a straight overwrite. + +## Licence + +`Apache-2.0 OR ISC OR MIT`, the same three-way form as `cbits/s2n` and by +some of the same authors. crypton takes it under ISC, which is already in +the package's `license:` field. `LICENSE` is the upstream file and is +listed in `license-files:`. + +## What is taken and what is not + +The whole of `mldsa/`, less the backends for architectures crypton does not +build for: the 32-bit +`src/fips202/native/armv81m`. (Unlike mlkem-native, this one ships only the +AArch64 and x86-64 arithmetic backends, so there is nothing else to drop.) Every reference to those is behind an +`MLD_SYS_` guard that cannot be true on the architectures crypton does +build for, so dropping them changes no build and keeps a few dozen files of +unreachable assembly out of the release tarball. To take one back, delete +its line from `import.sh` and add its directory to `extra-source-files:`. + +## How it is built + +Upstream builds for one parameter set at a time. `crypton_mldsa.c` +includes the amalgamation once per set -- the level-independent half kept by +exactly one of them -- which is how a single crypton offers ML-DSA-44, 65 +and 87. `crypton_mldsa_asm.S` does the same for the assembly, which is +level-independent and so is included once. + +This is the shape `cbits/aes/armv8.c` already uses for the three AES key +sizes, and it has the same hazard: **cabal does not know that the wrapper +depends on the tree it includes.** After changing anything under +`cbits/mldsa`, touch `crypton_mldsa.c`, or the build keeps the object it +already has and the change is not tested. + +The hand-written backends are selected by `CRYPTON_MLDSA_NATIVE_BACKEND`, +which `crypton.cabal` defines on x86-64 and AArch64 other than Windows -- +the same exclusion, and for the same reasons, as `cbits/s2n`. Everywhere +else the portable C is built, which is the same code and passes the same +tests. + +There is no randomised API: `MLD_CONFIG_NO_RANDOMIZED_API` is set, no +`randombytes()` is needed, and randomness is drawn in Haskell through +`MonadRandom` as it is for every other key crypton generates. + +The symbols are `crypton_mldsa44_*`, `crypton_mldsa65_*` and +`crypton_mldsa87_*` rather than upstream's defaults, so that an +application linking another copy of mldsa-native -- through some other +library, or its own -- does not present the linker with two sets of +functions answering to one set of names. diff --git a/cbits/mldsa/crypton_mldsa.c b/cbits/mldsa/crypton_mldsa.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/crypton_mldsa.c @@ -0,0 +1,31 @@ +/* + * All three ML-DSA parameter sets in one translation unit. + * + * mldsa-native is built for one parameter set at a time; a build wanting + * several includes the amalgamation once per set, with the level-independent + * half kept by exactly one of them. This is the same shape as + * cbits/aes/armv8.c, which includes cbits/aes/armv8_impl.c three times for + * the three AES key sizes. + * + * NOTE, as there: cabal does not know that this file depends on the tree it + * includes. After changing anything under cbits/mldsa, touch this file, or + * the build will quietly keep the object it already has. + */ +#include "crypton_mldsa.h" + +#define MLD_CONFIG_MULTILEVEL_WITH_SHARED 1 +#define MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS +#define MLD_CONFIG_PARAMETER_SET 44 +#include "mldsa_native.c" +#undef MLD_CONFIG_MULTILEVEL_WITH_SHARED +#undef MLD_CONFIG_PARAMETER_SET + +#define MLD_CONFIG_MULTILEVEL_NO_SHARED +#define MLD_CONFIG_PARAMETER_SET 65 +#include "mldsa_native.c" +#undef MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS +#undef MLD_CONFIG_PARAMETER_SET + +#define MLD_CONFIG_PARAMETER_SET 87 +#include "mldsa_native.c" +#undef MLD_CONFIG_PARAMETER_SET diff --git a/cbits/mldsa/crypton_mldsa.h b/cbits/mldsa/crypton_mldsa.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/crypton_mldsa.h @@ -0,0 +1,79 @@ +/* + * What crypton asks of the vendored mldsa-native, in one place. The tree + * under cbits/mldsa is upstream's and is overwritten by import.sh, so every + * choice crypton makes is made here instead of by editing it. + * + * Included first by both crypton_mldsa.c and crypton_mldsa_asm.S, so it must + * hold nothing the assembler cannot read: directives, and C behind + * __ASSEMBLER__. + */ +#ifndef CRYPTON_MLDSA_H +#define CRYPTON_MLDSA_H + +/* + * The symbols are crypton's own, not the default PQCP_MLDSA_NATIVE_*. An + * application is free to link another copy of mldsa-native -- through some + * other library, or its own -- and two copies answering to one set of names + * is a problem the linker resolves silently and in nobody's favour. With + * MLD_CONFIG_MULTILEVEL_BUILD the level is appended, so the entry points + * are crypton_mldsa44_*, crypton_mldsa65_* and crypton_mldsa87_*. + */ +#define MLD_CONFIG_NAMESPACE_PREFIX crypton_mldsa +#define MLD_CONFIG_MULTILEVEL_BUILD + +/* + * No randomised API, so no randombytes() to provide. Randomness is drawn + * in Haskell through MonadRandom, the way every other key in crypton is + * generated, and the deterministic entry points are what the FFI calls. + * That also keeps the C free of any opinion about where entropy comes from. + */ +#define MLD_CONFIG_NO_RANDOMIZED_API + +/* + * The hand-written backends, where crypton.cabal says the architecture has + * them. Without this the portable C is built, which is correct everywhere + * and is what every other architecture gets. + */ +#ifdef CRYPTON_MLDSA_NATIVE_BACKEND +#define MLD_CONFIG_USE_NATIVE_BACKEND_ARITH +#define MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 + +/* + * Which of the backend's instructions this processor actually has. + * + * Without this both packages decide at compile time: the backend is built + * because the architecture has the instructions in principle, and is then + * called with no check. Upstream says so plainly -- if the machine that + * runs the binary may differ from the one that built it, this function + * MUST be provided -- and crypton is a library whose builds are copied + * between machines and shipped by distributions. Without it a binary + * built on a host with AVX2 dies with SIGILL on one without (#316). + * + * The dispatch sites are upstream's; each falls back to the portable C + * when this answers no. The answers come from crypton's own detection, + * the same that AES-NI, PCLMUL and the s2n-bignum variants use. + */ +#define MLD_CONFIG_CUSTOM_CAPABILITY_FUNC +#ifndef __ASSEMBLER__ +#include "src/sys.h" +#include "crypton_cpu.h" + +static MLD_INLINE int mld_sys_check_capability(mld_sys_cap cap) +{ +#if defined(__x86_64__) + if (cap == MLD_SYS_CAP_X86_64_AVX2) + return (crypton_x86_simd_features() & CRYPTON_X86_AVX2) != 0; +#elif defined(__aarch64__) + /* NEON is not optional on AArch64; the Keccak instructions are. */ + if (cap == MLD_SYS_CAP_AARCH64_NEON) + return 1; + if (cap == MLD_SYS_CAP_AARCH64_SHA3) + return (crypton_arm_features() & CRYPTON_ARM_SHA3) != 0; +#endif + (void)cap; + return 0; +} +#endif /* !__ASSEMBLER__ */ +#endif + +#endif /* CRYPTON_MLDSA_H */ diff --git a/cbits/mldsa/crypton_mldsa_asm.S b/cbits/mldsa/crypton_mldsa_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/crypton_mldsa_asm.S @@ -0,0 +1,14 @@ +/* + * The assembly half, which is level-independent: it is included once, with + * the shared directives kept, and covers all three parameter sets. See the + * comment at the top of mldsa_native_asm.S. + * + * Built only where crypton.cabal defines CRYPTON_MLDSA_NATIVE_BACKEND; on + * every other architecture this file is not in asm-sources at all. + */ +#include "crypton_mldsa.h" + +#define MLD_CONFIG_MULTILEVEL_WITH_SHARED 1 +#define MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS +#define MLD_CONFIG_PARAMETER_SET 65 +#include "mldsa_native_asm.S" diff --git a/cbits/mldsa/import.sh b/cbits/mldsa/import.sh new file mode 100644 --- /dev/null +++ b/cbits/mldsa/import.sh @@ -0,0 +1,40 @@ +#!/bin/sh +# Re-import the vendored parts of the PQ Code Package's mldsa-native. +# +# The files are kept unmodified. Everything crypton decides -- which +# parameter sets exist, what the symbols are called, that there is no +# randomised API -- is decided in cbits/mldsa/crypton_mldsa.c and in +# crypton.cabal, not by editing anything here. Run this from cbits/mldsa: +# +# ./import.sh [tag-or-commit] +# +# and commit the result together with the COMMIT line it writes, so that the +# tree always says which upstream revision it holds. +set -eu + +REPO=https://github.com/pq-code-package/mldsa-native +REV=${1:-v2.0.0} +HERE=$(cd "$(dirname "$0")" && pwd) +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT + +git clone -q "$REPO" "$TMP/u" +git -C "$TMP/u" checkout -q "$REV" + +rm -rf "$HERE/src" +cp "$TMP/u/mldsa/mldsa_native.c" "$HERE/" +cp "$TMP/u/mldsa/mldsa_native.h" "$HERE/" +cp "$TMP/u/mldsa/mldsa_native_asm.S" "$HERE/" +cp "$TMP/u/mldsa/mldsa_native_config.h" "$HERE/" +cp -R "$TMP/u/mldsa/src" "$HERE/src" +cp "$TMP/u/LICENSE" "$HERE/LICENSE" + +# The 32-bit Armv8.1-M Keccak, for an architecture crypton does not build +# for. Every reference to it is behind an MLD_SYS_ guard that cannot be true +# on the ones it does, so dropping it changes no build and keeps unreachable +# assembly out of the release tarball. +rm -rf "$HERE/src/fips202/native/armv81m" + +git -C "$TMP/u" rev-parse HEAD > "$HERE/COMMIT" +git -C "$TMP/u" describe --tags --exact-match 2>/dev/null >> "$HERE/COMMIT" || true +echo "imported $(head -1 "$HERE/COMMIT")" diff --git a/cbits/mldsa/mldsa_native.c b/cbits/mldsa/mldsa_native.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/mldsa_native.c @@ -0,0 +1,803 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mldsa-native repository. + * Do not modify it directly. + */ + +/****************************************************************************** + * + * Single compilation unit (SCU) for fixed-level build of mldsa-native + * + * This compilation unit bundles together all source files for a build + * of mldsa-native for a fixed security level (MLDSA-44/65/87). + * + * # API + * + * The API exposed by this file is described in mldsa_native.h. + * + * # Multi-level build + * + * If you want an SCU build of mldsa-native with support for multiple security + * levels, you need to include this file multiple times, and set + * MLD_CONFIG_MULTILEVEL_WITH_SHARED and MLD_CONFIG_MULTILEVEL_NO_SHARED + * appropriately. This is exemplified in examples/monolithic_build_multilevel + * and examples/monolithic_build_multilevel_native. + * + * # Configuration + * + * The following options from the mldsa-native configuration are relevant: + * + * - MLD_CONFIG_FIPS202_CUSTOM_HEADER + * Set this option if you use a custom FIPS202 implementation. + * + * - MLD_CONFIG_USE_NATIVE_BACKEND_ARITH + * Set this option if you want to include the native arithmetic backends + * in your build. + * + * - MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 + * Set this option if you want to include the native FIPS202 backends + * in your build. + * + * - MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS + * Set this option if you want to keep the directives defined in + * level-independent headers. This is needed for a multi-level build. + */ + +/* If parts of the mldsa-native source tree are not used, + * consider reducing this header via `unifdef`. + * + * Example: + * ```bash + * unifdef -UMLD_CONFIG_USE_NATIVE_BACKEND_ARITH mldsa_native.c + * ``` + */ + +#include "src/common.h" + +#include "src/ct.c" +#include "src/debug.c" +#include "src/packing.c" +#include "src/poly.c" +#include "src/poly_kl.c" +#include "src/polyvec.c" +#include "src/polyvec_lazy.c" +#include "src/sign.c" + +#if !defined(MLD_CONFIG_FIPS202_CUSTOM_HEADER) +#include "src/fips202/fips202.c" +#include "src/fips202/fips202x4.c" +#include "src/fips202/keccakf1600.c" +#endif + +#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH) +#if defined(MLD_SYS_AARCH64) +#include "src/native/aarch64/src/aarch64_zetas.c" +#include "src/native/aarch64/src/polyz_unpack_table.c" +#include "src/native/aarch64/src/rej_uniform_eta_table.c" +#include "src/native/aarch64/src/rej_uniform_table.c" +#endif /* MLD_SYS_AARCH64 */ +#if defined(MLD_SYS_X86_64) +#include "src/native/x86_64/src/consts.c" +#include "src/native/x86_64/src/rej_uniform_table.c" +#endif +#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_ARITH */ + +#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202) +#if defined(MLD_SYS_AARCH64) +#include "src/fips202/native/aarch64/src/keccakf1600_round_constants.c" +#endif +#if defined(MLD_SYS_X86_64) +#include "src/fips202/native/x86_64/src/keccakf1600_constants.c" +#endif +#if defined(MLD_SYS_ARMV81M_MVE) +#include "src/fips202/native/armv81m/src/keccak_f1600_x4_mve.c" +#include "src/fips202/native/armv81m/src/keccakf1600_round_constants.c" +#endif +#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 */ + +/* Macro #undef's + * + * The following undefines macros from headers + * included by the source files imported above. + * + * This is to allow building and linking multiple builds + * of mldsa-native for varying parameter sets through concatenation + * of this file, as if the files had been compiled separately. + * If this is not relevant to you, you may remove the following. + */ + +/* + * Undefine macros from MLD_CONFIG_PARAMETER_SET-specific files + */ +/* mldsa/mldsa_native.h */ +#undef MLDSA44_BYTES +#undef MLDSA44_CRHBYTES +#undef MLDSA44_PUBLICKEYBYTES +#undef MLDSA44_RNDBYTES +#undef MLDSA44_SECRETKEYBYTES +#undef MLDSA44_SEEDBYTES +#undef MLDSA44_TRBYTES +#undef MLDSA65_BYTES +#undef MLDSA65_CRHBYTES +#undef MLDSA65_PUBLICKEYBYTES +#undef MLDSA65_RNDBYTES +#undef MLDSA65_SECRETKEYBYTES +#undef MLDSA65_SEEDBYTES +#undef MLDSA65_TRBYTES +#undef MLDSA87_BYTES +#undef MLDSA87_CRHBYTES +#undef MLDSA87_PUBLICKEYBYTES +#undef MLDSA87_RNDBYTES +#undef MLDSA87_SECRETKEYBYTES +#undef MLDSA87_SEEDBYTES +#undef MLDSA87_TRBYTES +#undef MLDSA_BYTES +#undef MLDSA_BYTES_ +#undef MLDSA_CRHBYTES +#undef MLDSA_PUBLICKEYBYTES +#undef MLDSA_PUBLICKEYBYTES_ +#undef MLDSA_RNDBYTES +#undef MLDSA_SECRETKEYBYTES +#undef MLDSA_SECRETKEYBYTES_ +#undef MLDSA_SEEDBYTES +#undef MLDSA_TRBYTES +#undef MLD_API_CONCAT +#undef MLD_API_CONCAT_ +#undef MLD_API_CONCAT_UNDERSCORE +#undef MLD_API_MUST_CHECK_RETURN_VALUE +#undef MLD_API_NAMESPACE +#undef MLD_API_NAMESPACE_PREFIX +#undef MLD_API_QUALIFIER +#undef MLD_DOMAIN_SEPARATION_MAX_BYTES +#undef MLD_ERR_FAIL +#undef MLD_ERR_INVALID_ARG +#undef MLD_ERR_INVALID_KEY +#undef MLD_ERR_INVALID_SIGNATURE +#undef MLD_ERR_OUT_OF_MEMORY +#undef MLD_ERR_PCT_FAIL +#undef MLD_ERR_RNG_FAIL +#undef MLD_ERR_SIGNING_PAUSED +#undef MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED +#undef MLD_H +#undef MLD_MAX3_ +#undef MLD_MAX4_ +#undef MLD_PREHASH_NONE +#undef MLD_PREHASH_SHA2_224 +#undef MLD_PREHASH_SHA2_256 +#undef MLD_PREHASH_SHA2_384 +#undef MLD_PREHASH_SHA2_512 +#undef MLD_PREHASH_SHA2_512_224 +#undef MLD_PREHASH_SHA2_512_256 +#undef MLD_PREHASH_SHA3_224 +#undef MLD_PREHASH_SHA3_256 +#undef MLD_PREHASH_SHA3_384 +#undef MLD_PREHASH_SHA3_512 +#undef MLD_PREHASH_SHAKE_128 +#undef MLD_PREHASH_SHAKE_256 +#undef MLD_TOTAL_ALLOC_44 +#undef MLD_TOTAL_ALLOC_44_KEYPAIR +#undef MLD_TOTAL_ALLOC_44_KEYPAIR_NO_PCT +#undef MLD_TOTAL_ALLOC_44_KEYPAIR_PCT +#undef MLD_TOTAL_ALLOC_44_PK_FROM_SK +#undef MLD_TOTAL_ALLOC_44_SIGN +#undef MLD_TOTAL_ALLOC_44_VERIFY +#undef MLD_TOTAL_ALLOC_65 +#undef MLD_TOTAL_ALLOC_65_KEYPAIR +#undef MLD_TOTAL_ALLOC_65_KEYPAIR_NO_PCT +#undef MLD_TOTAL_ALLOC_65_KEYPAIR_PCT +#undef MLD_TOTAL_ALLOC_65_PK_FROM_SK +#undef MLD_TOTAL_ALLOC_65_SIGN +#undef MLD_TOTAL_ALLOC_65_VERIFY +#undef MLD_TOTAL_ALLOC_87 +#undef MLD_TOTAL_ALLOC_87_KEYPAIR +#undef MLD_TOTAL_ALLOC_87_KEYPAIR_NO_PCT +#undef MLD_TOTAL_ALLOC_87_KEYPAIR_PCT +#undef MLD_TOTAL_ALLOC_87_PK_FROM_SK +#undef MLD_TOTAL_ALLOC_87_SIGN +#undef MLD_TOTAL_ALLOC_87_VERIFY +/* mldsa/src/common.h */ +#undef MLD_ADD_PARAM_SET +#undef MLD_ALLOC +#undef MLD_APPLY +#undef MLD_ASM_FN_SIZE +#undef MLD_ASM_FN_SYMBOL +#undef MLD_ASM_NAMESPACE +#undef MLD_BUILD_INTERNAL +#undef MLD_COMMON_H +#undef MLD_CONCAT +#undef MLD_CONCAT_ +#undef MLD_EMPTY_CU +#undef MLD_ERR_FAIL +#undef MLD_ERR_INVALID_ARG +#undef MLD_ERR_INVALID_KEY +#undef MLD_ERR_INVALID_SIGNATURE +#undef MLD_ERR_OUT_OF_MEMORY +#undef MLD_ERR_PCT_FAIL +#undef MLD_ERR_RNG_FAIL +#undef MLD_ERR_SIGNING_PAUSED +#undef MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED +#undef MLD_EXTERNAL_API +#undef MLD_FIPS202X4_HEADER_FILE +#undef MLD_FIPS202_HEADER_FILE +#undef MLD_FREE +#undef MLD_INTERNAL_API +#undef MLD_INTERNAL_DATA_DECLARATION +#undef MLD_INTERNAL_DATA_DEFINITION +#undef MLD_MULTILEVEL_BUILD +#undef MLD_NAMESPACE +#undef MLD_NAMESPACE_KL +#undef MLD_NAMESPACE_PREFIX +#undef MLD_NAMESPACE_PREFIX_KL +#undef mld_memcpy +#undef mld_memset +/* mldsa/src/packing.h */ +#undef MLD_PACKING_H +#undef mld_pack_sig_c +#undef mld_pack_sig_h +#undef mld_pack_sig_z +#undef mld_pack_sk_rho_key_tr_s2 +#undef mld_pack_sk_s1 +#undef mld_sig_unpack_hints +#undef mld_unpack_pk_t1 +#undef mld_unpack_sk +/* mldsa/src/params.h */ +#undef MLDSA_BETA +#undef MLDSA_CRHBYTES +#undef MLDSA_CRYPTO_BYTES +#undef MLDSA_CRYPTO_PUBLICKEYBYTES +#undef MLDSA_CRYPTO_SECRETKEYBYTES +#undef MLDSA_CTILDEBYTES +#undef MLDSA_D +#undef MLDSA_ETA +#undef MLDSA_GAMMA1 +#undef MLDSA_GAMMA2 +#undef MLDSA_GAMMA2_32 +#undef MLDSA_GAMMA2_88 +#undef MLDSA_K +#undef MLDSA_L +#undef MLDSA_N +#undef MLDSA_OMEGA +#undef MLDSA_PK_END +#undef MLDSA_PK_RHO_BYTES +#undef MLDSA_PK_RHO_OFFSET +#undef MLDSA_PK_T1_BYTES +#undef MLDSA_PK_T1_OFFSET +#undef MLDSA_POLYETA_PACKEDBYTES +#undef MLDSA_POLYT0_PACKEDBYTES +#undef MLDSA_POLYT1_PACKEDBYTES +#undef MLDSA_POLYVECH_PACKEDBYTES +#undef MLDSA_POLYW1_PACKEDBYTES +#undef MLDSA_POLYW1_PACKEDBYTES_32 +#undef MLDSA_POLYW1_PACKEDBYTES_88 +#undef MLDSA_POLYZ_PACKEDBYTES +#undef MLDSA_Q +#undef MLDSA_Q_HALF +#undef MLDSA_RNDBYTES +#undef MLDSA_SEEDBYTES +#undef MLDSA_SIG_C_BYTES +#undef MLDSA_SIG_C_OFFSET +#undef MLDSA_SIG_END +#undef MLDSA_SIG_H_BYTES +#undef MLDSA_SIG_H_OFFSET +#undef MLDSA_SIG_Z_BYTES +#undef MLDSA_SIG_Z_OFFSET +#undef MLDSA_SK_END +#undef MLDSA_SK_KEY_BYTES +#undef MLDSA_SK_KEY_OFFSET +#undef MLDSA_SK_RHO_BYTES +#undef MLDSA_SK_RHO_OFFSET +#undef MLDSA_SK_S1_BYTES +#undef MLDSA_SK_S1_OFFSET +#undef MLDSA_SK_S2_BYTES +#undef MLDSA_SK_S2_OFFSET +#undef MLDSA_SK_T0_BYTES +#undef MLDSA_SK_T0_OFFSET +#undef MLDSA_SK_TR_BYTES +#undef MLDSA_SK_TR_OFFSET +#undef MLDSA_TAU +#undef MLDSA_TRBYTES +#undef MLD_MAX_KAPPA +#undef MLD_PARAMS_H +/* mldsa/src/poly_kl.h */ +#undef MLD_POLYETA_UNPACK_LOWER_BOUND +#undef MLD_POLY_KL_H +#undef mld_poly_challenge +#undef mld_poly_decompose +#undef mld_poly_uniform_eta +#undef mld_poly_uniform_eta_4x +#undef mld_poly_uniform_gamma1 +#undef mld_poly_uniform_gamma1_4x +#undef mld_poly_use_hint +#undef mld_polyeta_pack +#undef mld_polyeta_unpack +#undef mld_polyw1_pack +#undef mld_polyz_pack +#undef mld_polyz_unpack +/* mldsa/src/polyvec.h */ +#undef MLD_POLYVEC_H +#undef mld_polyveck +#undef mld_polyveck_caddq +#undef mld_polyveck_chknorm +#undef mld_polyveck_decompose +#undef mld_polyveck_invntt_tomont +#undef mld_polyveck_ntt +#undef mld_polyveck_pack_eta +#undef mld_polyveck_pack_w1 +#undef mld_polyveck_reduce +#undef mld_polyveck_unpack_eta +#undef mld_polyvecl +#undef mld_polyvecl_chknorm +#undef mld_polyvecl_ntt +#undef mld_polyvecl_pack_eta +#undef mld_polyvecl_pointwise_acc_montgomery +#undef mld_polyvecl_uniform_gamma1 +#undef mld_polyvecl_unpack_eta +#undef mld_polyvecl_unpack_z +/* mldsa/src/polyvec_lazy.h */ +#undef MLD_POLYVEC_LAZY_H +#undef mld_poly_permute_bitrev_to_custom_optional +#undef mld_polymat +#undef mld_polymat_eager +#undef mld_polymat_lazy +#undef mld_polyvec_matrix_expand +#undef mld_polyvec_matrix_expand_eager +#undef mld_polyvec_matrix_expand_lazy +#undef mld_polyvec_matrix_pointwise_montgomery +#undef mld_polyvec_matrix_pointwise_montgomery_row +#undef mld_polyvec_matrix_pointwise_montgomery_row_eager +#undef mld_polyvec_matrix_pointwise_montgomery_row_lazy +#undef mld_polyvec_matrix_pointwise_montgomery_yvec +#undef mld_polyvec_matrix_pointwise_montgomery_yvec_eager +#undef mld_polyvec_matrix_pointwise_montgomery_yvec_lazy +#undef mld_sk_s1hat +#undef mld_sk_s1hat_eager +#undef mld_sk_s1hat_get_poly +#undef mld_sk_s1hat_get_poly_eager +#undef mld_sk_s1hat_get_poly_lazy +#undef mld_sk_s1hat_lazy +#undef mld_sk_s2hat +#undef mld_sk_s2hat_eager +#undef mld_sk_s2hat_get_poly +#undef mld_sk_s2hat_get_poly_eager +#undef mld_sk_s2hat_get_poly_lazy +#undef mld_sk_s2hat_lazy +#undef mld_sk_t0hat +#undef mld_sk_t0hat_eager +#undef mld_sk_t0hat_get_poly +#undef mld_sk_t0hat_get_poly_eager +#undef mld_sk_t0hat_get_poly_lazy +#undef mld_sk_t0hat_lazy +#undef mld_unpack_sk_s1hat +#undef mld_unpack_sk_s1hat_eager +#undef mld_unpack_sk_s1hat_lazy +#undef mld_unpack_sk_s2hat +#undef mld_unpack_sk_s2hat_eager +#undef mld_unpack_sk_s2hat_lazy +#undef mld_unpack_sk_t0hat +#undef mld_unpack_sk_t0hat_eager +#undef mld_unpack_sk_t0hat_lazy +#undef mld_yvec +#undef mld_yvec_eager +#undef mld_yvec_get_poly +#undef mld_yvec_get_poly_eager +#undef mld_yvec_get_poly_lazy +#undef mld_yvec_init +#undef mld_yvec_init_eager +#undef mld_yvec_init_lazy +#undef mld_yvec_lazy +/* mldsa/src/rounding.h */ +#undef MLD_2_POW_D +#undef MLD_ROUNDING_H +#undef mld_decompose +#undef mld_make_hint +#undef mld_power2round +#undef mld_use_hint +/* mldsa/src/sign.h */ +#undef MLD_DOMAIN_SEPARATION_MAX_BYTES +#undef MLD_PREHASH_NONE +#undef MLD_PREHASH_SHA2_224 +#undef MLD_PREHASH_SHA2_256 +#undef MLD_PREHASH_SHA2_384 +#undef MLD_PREHASH_SHA2_512 +#undef MLD_PREHASH_SHA2_512_224 +#undef MLD_PREHASH_SHA2_512_256 +#undef MLD_PREHASH_SHA3_224 +#undef MLD_PREHASH_SHA3_256 +#undef MLD_PREHASH_SHA3_384 +#undef MLD_PREHASH_SHA3_512 +#undef MLD_PREHASH_SHAKE_128 +#undef MLD_PREHASH_SHAKE_256 +#undef MLD_SIGN_H +#undef mld_prepare_domain_separation_prefix +#undef mld_sign_keypair +#undef mld_sign_keypair_internal +#undef mld_sign_pk_from_sk +#undef mld_sign_signature +#undef mld_sign_signature_extmu +#undef mld_sign_signature_internal +#undef mld_sign_signature_pre_hash_internal +#undef mld_sign_signature_pre_hash_shake256 +#undef mld_sign_verify +#undef mld_sign_verify_extmu +#undef mld_sign_verify_internal +#undef mld_sign_verify_pre_hash_internal +#undef mld_sign_verify_pre_hash_shake256 + +#if !defined(MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS) +/* + * Undefine macros from MLD_CONFIG_PARAMETER_SET-generic files + */ +/* mldsa/src/context.h */ +#undef MLD_CONTEXT_H +#undef MLD_CONTEXT_PARAMETERS_0 +#undef MLD_CONTEXT_PARAMETERS_1 +#undef MLD_CONTEXT_PARAMETERS_2 +#undef MLD_CONTEXT_PARAMETERS_3 +#undef MLD_CONTEXT_PARAMETERS_4 +#undef MLD_CONTEXT_PARAMETERS_5 +#undef MLD_CONTEXT_PARAMETERS_6 +#undef MLD_CONTEXT_PARAMETERS_7 +#undef MLD_CONTEXT_PARAMETERS_8 +#undef MLD_CONTEXT_PARAMETERS_9 +#undef MLD_CONTEXT_UNUSED +#undef mld_sign_attempt +#undef mld_sign_finish +#undef mld_sign_resume +/* mldsa/src/ct.h */ +#undef MLD_CT_H +#undef MLD_USE_ASM_VALUE_BARRIER +#undef mld_ct_opt_blocker_u64 +/* mldsa/src/debug.h */ +#undef MLD_DEBUG_H +#undef mld_assert +#undef mld_assert_abs_bound +#undef mld_assert_abs_bound_2d +#undef mld_assert_bound +#undef mld_assert_bound_2d +#undef mld_debug_check_assert +#undef mld_debug_check_bounds +/* mldsa/src/poly.h */ +#undef MLD_FQMUL_BOUND +#undef MLD_INTT_BOUND +#undef MLD_NTT_BOUND +#undef MLD_POLY_H +#undef mld_poly_add +#undef mld_poly_caddq +#undef mld_poly_chknorm +#undef mld_poly_invntt_tomont +#undef mld_poly_ntt +#undef mld_poly_pointwise_montgomery +#undef mld_poly_power2round +#undef mld_poly_reduce +#undef mld_poly_shiftl +#undef mld_poly_sub +#undef mld_poly_uniform +#undef mld_poly_uniform_4x +#undef mld_polyt0_pack +#undef mld_polyt0_unpack +#undef mld_polyt1_pack +#undef mld_polyt1_unpack +#undef mld_polyw1_pack_32 +#undef mld_polyw1_pack_88 +/* mldsa/src/randombytes.h */ +#undef MLD_RANDOMBYTES_H +/* mldsa/src/reduce.h */ +#undef MLD_MONT +#undef MLD_REDUCE32_DOMAIN_MAX +#undef MLD_REDUCE32_RANGE_MAX +#undef MLD_REDUCE_H +/* mldsa/src/symmetric.h */ +#undef MLD_STREAM128_BLOCKBYTES +#undef MLD_STREAM256_BLOCKBYTES +#undef MLD_SYMMETRIC_H +#undef mld_xof128_absorb_once +#undef mld_xof128_ctx +#undef mld_xof128_init +#undef mld_xof128_release +#undef mld_xof128_squeezeblocks +#undef mld_xof128_x4_absorb +#undef mld_xof128_x4_ctx +#undef mld_xof128_x4_init +#undef mld_xof128_x4_release +#undef mld_xof128_x4_squeezeblocks +#undef mld_xof256_absorb_once +#undef mld_xof256_ctx +#undef mld_xof256_init +#undef mld_xof256_release +#undef mld_xof256_squeezeblocks +#undef mld_xof256_x4_absorb +#undef mld_xof256_x4_ctx +#undef mld_xof256_x4_init +#undef mld_xof256_x4_release +#undef mld_xof256_x4_squeezeblocks +/* mldsa/src/sys.h */ +#undef MLD_ALIGN +#undef MLD_ALIGN_UP +#undef MLD_ALWAYS_INLINE +#undef MLD_CET_ENDBR +#undef MLD_CT_TESTING_DECLASSIFY +#undef MLD_CT_TESTING_SECRET +#undef MLD_DEFAULT_ALIGN +#undef MLD_HAVE_INLINE_ASM +#undef MLD_INLINE +#undef MLD_MUST_CHECK_RETURN_VALUE +#undef MLD_NOINLINE +#undef MLD_RESTRICT +#undef MLD_STATIC_TESTABLE +#undef MLD_SYSV_ABI +#undef MLD_SYSV_ABI_SUPPORTED +#undef MLD_SYS_AARCH64 +#undef MLD_SYS_AARCH64_EB +#undef MLD_SYS_AARCH64_NEON +#undef MLD_SYS_APPLE +#undef MLD_SYS_ARMV81M_MVE +#undef MLD_SYS_BIG_ENDIAN +#undef MLD_SYS_H +#undef MLD_SYS_LINUX +#undef MLD_SYS_LITTLE_ENDIAN +#undef MLD_SYS_PPC64LE +#undef MLD_SYS_RISCV32 +#undef MLD_SYS_RISCV64 +#undef MLD_SYS_RISCV64_RVV +#undef MLD_SYS_WINDOWS +#undef MLD_SYS_X86_64 +#undef MLD_SYS_X86_64_AVX2 +/* mldsa/src/cbmc.h */ +#undef MLD_CBMC_H +#undef __contract__ +#undef __loop__ + +#if !defined(MLD_CONFIG_FIPS202_CUSTOM_HEADER) +/* + * Undefine macros from FIPS-202 files + */ +/* mldsa/src/fips202/fips202.h */ +#undef MLD_FIPS202_FIPS202_H +#undef MLD_KECCAK_LANES +#undef SHA3_256_HASHBYTES +#undef SHA3_256_RATE +#undef SHA3_512_HASHBYTES +#undef SHA3_512_RATE +#undef SHAKE128_RATE +#undef SHAKE256_RATE +#undef mld_shake128_absorb +#undef mld_shake128_finalize +#undef mld_shake128_init +#undef mld_shake128_release +#undef mld_shake128_squeeze +#undef mld_shake256 +#undef mld_shake256_absorb +#undef mld_shake256_finalize +#undef mld_shake256_init +#undef mld_shake256_release +#undef mld_shake256_squeeze +/* mldsa/src/fips202/fips202x4.h */ +#undef MLD_FIPS202_FIPS202X4_H +#undef mld_shake128x4_absorb_once +#undef mld_shake128x4_init +#undef mld_shake128x4_release +#undef mld_shake128x4_squeezeblocks +#undef mld_shake256x4_absorb_once +#undef mld_shake256x4_init +#undef mld_shake256x4_release +#undef mld_shake256x4_squeezeblocks +/* mldsa/src/fips202/keccakf1600.h */ +#undef MLD_FIPS202_KECCAKF1600_H +#undef MLD_KECCAK_LANES +#undef MLD_KECCAK_WAY +#undef mld_keccakf1600_extract_bytes +#undef mld_keccakf1600_permute +#undef mld_keccakf1600_xor_bytes +#undef mld_keccakf1600x4_extract_bytes +#undef mld_keccakf1600x4_permute +#undef mld_keccakf1600x4_xor_bytes +#endif /* !MLD_CONFIG_FIPS202_CUSTOM_HEADER */ + +#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202) +/* mldsa/src/fips202/native/api.h */ +#undef MLD_FIPS202_NATIVE_API_H +#undef MLD_NATIVE_FUNC_FALLBACK +#undef MLD_NATIVE_FUNC_SUCCESS +/* mldsa/src/fips202/native/auto.h */ +#undef MLD_FIPS202_NATIVE_AUTO_H +#if defined(MLD_SYS_AARCH64) +/* + * Undefine macros from native code (FIPS202, AArch64) + */ +/* mldsa/src/fips202/native/aarch64/auto.h */ +#undef MLD_FIPS202_NATIVE_AARCH64_AUTO_H +/* mldsa/src/fips202/native/aarch64/src/fips202_native_aarch64.h */ +#undef MLD_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H +#undef mld_keccak_f1600_x1_scalar_aarch64_asm +#undef mld_keccak_f1600_x1_v84a_aarch64_asm +#undef mld_keccak_f1600_x2_v84a_aarch64_asm +#undef mld_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm +#undef mld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm +#undef mld_keccakf1600_round_constants +/* mldsa/src/fips202/native/aarch64/x1_scalar.h */ +#undef MLD_FIPS202_AARCH64_NEED_X1_SCALAR +#undef MLD_FIPS202_NATIVE_AARCH64_X1_SCALAR_H +#undef MLD_USE_NATIVE_FIPS202_X1 +/* mldsa/src/fips202/native/aarch64/x1_v84a.h */ +#undef MLD_FIPS202_AARCH64_NEED_X1_V84A +#undef MLD_FIPS202_NATIVE_AARCH64_X1_V84A_H +#undef MLD_USE_NATIVE_FIPS202_X1 +/* mldsa/src/fips202/native/aarch64/x2_v84a.h */ +#undef MLD_FIPS202_AARCH64_NEED_X2_V84A +#undef MLD_FIPS202_NATIVE_AARCH64_X2_V84A_H +#undef MLD_USE_NATIVE_FIPS202_X4 +/* mldsa/src/fips202/native/aarch64/x4_v8a_scalar.h */ +#undef MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID +#undef MLD_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H +#undef MLD_USE_NATIVE_FIPS202_X4 +/* mldsa/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h */ +#undef MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID +#undef MLD_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H +#undef MLD_USE_NATIVE_FIPS202_X4 +#endif /* MLD_SYS_AARCH64 */ +#if defined(MLD_SYS_X86_64) +/* + * Undefine macros from native code (FIPS202, x86_64) + */ +/* mldsa/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h */ +#undef MLD_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H +#undef MLD_FIPS202_X86_64_NEED_X4_AVX2 +#undef MLD_USE_NATIVE_FIPS202_X4 +/* mldsa/src/fips202/native/x86_64/src/fips202_native_x86_64.h */ +#undef MLD_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H +#undef mld_keccak_f1600_x4_avx2_asm +#undef mld_keccak_rho56 +#undef mld_keccak_rho8 +#undef mld_keccakf1600_round_constants +#endif /* MLD_SYS_X86_64 */ +#if defined(MLD_SYS_ARMV81M_MVE) +/* + * Undefine macros from native code (FIPS202, Armv8.1-M) + */ +/* mldsa/src/fips202/native/armv81m/mve.h */ +#undef MLD_FIPS202_ARMV81M_NEED_X4 +#undef MLD_FIPS202_NATIVE_ARMV81M +#undef MLD_FIPS202_NATIVE_ARMV81M_MVE_H +#undef MLD_USE_NATIVE_FIPS202_X4 +#undef MLD_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES +#undef MLD_USE_NATIVE_FIPS202_X4_XOR_BYTES +#undef mld_keccak_f1600_x4_native_impl +/* mldsa/src/fips202/native/armv81m/src/fips202_native_armv81m.h */ +#undef MLD_FIPS202_NATIVE_ARMV81M_SRC_FIPS202_NATIVE_ARMV81M_H +#undef mld_keccak_f1600_x4_mve_asm +#undef mld_keccak_f1600_x4_state_extract_bytes_asm +#undef mld_keccak_f1600_x4_state_xor_bytes_asm +#undef mld_keccakf1600_round_constants +#endif /* MLD_SYS_ARMV81M_MVE */ +#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 */ +#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH) +/* mldsa/src/native/api.h */ +#undef MLD_FQMUL_BOUND +#undef MLD_INTT_BOUND +#undef MLD_NATIVE_API_H +#undef MLD_NATIVE_FUNC_FALLBACK +#undef MLD_NATIVE_FUNC_SUCCESS +#undef MLD_NTT_BOUND +#undef MLD_REDUCE32_RANGE_MAX +/* mldsa/src/native/meta.h */ +#undef MLD_NATIVE_META_H +#if defined(MLD_SYS_AARCH64) +/* + * Undefine macros from native code (Arith, AArch64) + */ +/* mldsa/src/native/aarch64/meta.h */ +#undef MLD_ARITH_BACKEND_AARCH64 +#undef MLD_NATIVE_AARCH64_META_H +#undef MLD_USE_NATIVE_INTT +#undef MLD_USE_NATIVE_NTT +#undef MLD_USE_NATIVE_POINTWISE_MONTGOMERY +#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4 +#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5 +#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7 +#undef MLD_USE_NATIVE_POLYZ_UNPACK_17 +#undef MLD_USE_NATIVE_POLYZ_UNPACK_19 +#undef MLD_USE_NATIVE_POLY_CADDQ +#undef MLD_USE_NATIVE_POLY_CHKNORM +#undef MLD_USE_NATIVE_POLY_DECOMPOSE_32 +#undef MLD_USE_NATIVE_POLY_DECOMPOSE_88 +#undef MLD_USE_NATIVE_POLY_USE_HINT_32 +#undef MLD_USE_NATIVE_POLY_USE_HINT_88 +#undef MLD_USE_NATIVE_REJ_UNIFORM +#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA2 +#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA4 +/* mldsa/src/native/aarch64/src/arith_native_aarch64.h */ +#undef MLD_AARCH64_REJ_UNIFORM_ETA2_BUFLEN +#undef MLD_AARCH64_REJ_UNIFORM_ETA4_BUFLEN +#undef MLD_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H +#undef mld_aarch64_intt_zetas_layer123456 +#undef mld_aarch64_intt_zetas_layer78 +#undef mld_aarch64_ntt_zetas_layer123456 +#undef mld_aarch64_ntt_zetas_layer78 +#undef mld_intt_aarch64_asm +#undef mld_ntt_aarch64_asm +#undef mld_poly_caddq_aarch64_asm +#undef mld_poly_chknorm_aarch64_asm +#undef mld_poly_decompose_32_aarch64_asm +#undef mld_poly_decompose_88_aarch64_asm +#undef mld_poly_pointwise_montgomery_aarch64_asm +#undef mld_poly_use_hint_32_aarch64_asm +#undef mld_poly_use_hint_88_aarch64_asm +#undef mld_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm +#undef mld_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm +#undef mld_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm +#undef mld_polyz_unpack_17_aarch64_asm +#undef mld_polyz_unpack_17_indices +#undef mld_polyz_unpack_19_aarch64_asm +#undef mld_polyz_unpack_19_indices +#undef mld_rej_uniform_aarch64_asm +#undef mld_rej_uniform_eta2_aarch64_asm +#undef mld_rej_uniform_eta4_aarch64_asm +#undef mld_rej_uniform_eta_table +#undef mld_rej_uniform_table +#endif /* MLD_SYS_AARCH64 */ +#if defined(MLD_SYS_X86_64) +/* + * Undefine macros from native code (Arith, X86_64) + */ +/* mldsa/src/native/x86_64/meta.h */ +#undef MLD_ARITH_BACKEND_X86_64_DEFAULT +#undef MLD_NATIVE_X86_64_META_H +#undef MLD_USE_NATIVE_INTT +#undef MLD_USE_NATIVE_NTT +#undef MLD_USE_NATIVE_NTT_CUSTOM_ORDER +#undef MLD_USE_NATIVE_POINTWISE_MONTGOMERY +#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4 +#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5 +#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7 +#undef MLD_USE_NATIVE_POLYZ_UNPACK_17 +#undef MLD_USE_NATIVE_POLYZ_UNPACK_19 +#undef MLD_USE_NATIVE_POLY_CADDQ +#undef MLD_USE_NATIVE_POLY_CHKNORM +#undef MLD_USE_NATIVE_POLY_DECOMPOSE_32 +#undef MLD_USE_NATIVE_POLY_DECOMPOSE_88 +#undef MLD_USE_NATIVE_POLY_USE_HINT_32 +#undef MLD_USE_NATIVE_POLY_USE_HINT_88 +#undef MLD_USE_NATIVE_REJ_UNIFORM +#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA2 +#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA4 +/* mldsa/src/native/x86_64/src/arith_native_x86_64.h */ +#undef MLD_AVX2_REJ_UNIFORM_BUFLEN +#undef MLD_AVX2_REJ_UNIFORM_ETA2_BUFLEN +#undef MLD_AVX2_REJ_UNIFORM_ETA4_BUFLEN +#undef MLD_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H +#undef mld_invntt_avx2_asm +#undef mld_ntt_avx2_asm +#undef mld_nttunpack_avx2_asm +#undef mld_pointwise_acc_l4_avx2_asm +#undef mld_pointwise_acc_l5_avx2_asm +#undef mld_pointwise_acc_l7_avx2_asm +#undef mld_pointwise_avx2_asm +#undef mld_poly_caddq_avx2_asm +#undef mld_poly_chknorm_avx2_asm +#undef mld_poly_decompose_32_avx2_asm +#undef mld_poly_decompose_88_avx2_asm +#undef mld_poly_use_hint_32_avx2_asm +#undef mld_poly_use_hint_88_avx2_asm +#undef mld_polyz_unpack_17_avx2_asm +#undef mld_polyz_unpack_19_avx2_asm +#undef mld_rej_uniform_avx2_asm +#undef mld_rej_uniform_eta2_avx2_asm +#undef mld_rej_uniform_eta4_avx2_asm +#undef mld_rej_uniform_table +/* mldsa/src/native/x86_64/src/consts.h */ +#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XDIV +#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XDIV_QINV +#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XQ +#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XQINV +#undef MLD_AVX2_BACKEND_DATA_OFFSET_ZETAS +#undef MLD_AVX2_BACKEND_DATA_OFFSET_ZETAS_QINV +#undef MLD_NATIVE_X86_64_SRC_CONSTS_H +#undef mld_qdata +#endif /* MLD_SYS_X86_64 */ +#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_ARITH */ +#endif /* !MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS */ diff --git a/cbits/mldsa/mldsa_native.h b/cbits/mldsa/mldsa_native.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/mldsa_native.h @@ -0,0 +1,956 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + */ + +#ifndef MLD_H +#define MLD_H + +/* + * Public API for mldsa-native + * + * This header defines the public API of a single build of mldsa-native. + * + * Make sure the configuration file is in the include path + * (this is "mldsa_native_config.h" by default, or MLD_CONFIG_FILE if defined). + * + * # API conventions + * + * Conventions shared by all functions below (return values, pointer validity, + * output buffers on error) are documented in API-CONVENTIONS.md. + * + * # Multi-level builds + * + * This header specifies a build of mldsa-native for a fixed security level. + * If you need multiple security levels, leave the security level unspecified + * in the configuration file and include this header multiple times, setting + * MLD_CONFIG_PARAMETER_SET accordingly for each, and #undef'ing the MLD_H + * guard to allow multiple inclusions. + */ + +/******************************* Key sizes ************************************/ + +/* Sizes of cryptographic material, per parameter set */ +/* See mldsa/src/params.h for the arithmetic expressions giving rise to these */ +/* check-magic: off */ +#define MLDSA44_SECRETKEYBYTES 2560 +#define MLDSA44_PUBLICKEYBYTES 1312 +#define MLDSA44_BYTES 2420 + +#define MLDSA65_SECRETKEYBYTES 4032 +#define MLDSA65_PUBLICKEYBYTES 1952 +#define MLDSA65_BYTES 3309 + +#define MLDSA87_SECRETKEYBYTES 4896 +#define MLDSA87_PUBLICKEYBYTES 2592 +#define MLDSA87_BYTES 4627 +/* check-magic: on */ + +/* Size of seed and randomness in bytes (level-independent) */ +#define MLDSA_SEEDBYTES 32 +#define MLDSA44_SEEDBYTES MLDSA_SEEDBYTES +#define MLDSA65_SEEDBYTES MLDSA_SEEDBYTES +#define MLDSA87_SEEDBYTES MLDSA_SEEDBYTES + +/* Size of CRH output in bytes (level-independent) */ +#define MLDSA_CRHBYTES 64 +#define MLDSA44_CRHBYTES MLDSA_CRHBYTES +#define MLDSA65_CRHBYTES MLDSA_CRHBYTES +#define MLDSA87_CRHBYTES MLDSA_CRHBYTES + +/* Size of TR in bytes (level-independent) + * + * TR = SHAKE256(pk, 64) is the hash of the public key. Callers of the + * external-mu API (signature_extmu / verify_extmu) that compute the message + * representative mu = SHAKE256(TR || M', MLDSA_CRHBYTES) themselves -- e.g. to + * sign or verify a message that is too large or streamed to hold in memory -- + * need this constant to size the TR buffer. */ +#define MLDSA_TRBYTES 64 +#define MLDSA44_TRBYTES MLDSA_TRBYTES +#define MLDSA65_TRBYTES MLDSA_TRBYTES +#define MLDSA87_TRBYTES MLDSA_TRBYTES + +/* Size of randomness for signing in bytes (level-independent) */ +#define MLDSA_RNDBYTES 32 +#define MLDSA44_RNDBYTES MLDSA_RNDBYTES +#define MLDSA65_RNDBYTES MLDSA_RNDBYTES +#define MLDSA87_RNDBYTES MLDSA_RNDBYTES + +/* Sizes of cryptographic material, as a function of LVL=44,65,87 */ +#define MLDSA_SECRETKEYBYTES_(LVL) MLDSA##LVL##_SECRETKEYBYTES +#define MLDSA_PUBLICKEYBYTES_(LVL) MLDSA##LVL##_PUBLICKEYBYTES +#define MLDSA_BYTES_(LVL) MLDSA##LVL##_BYTES +#define MLDSA_SECRETKEYBYTES(LVL) MLDSA_SECRETKEYBYTES_(LVL) +#define MLDSA_PUBLICKEYBYTES(LVL) MLDSA_PUBLICKEYBYTES_(LVL) +#define MLDSA_BYTES(LVL) MLDSA_BYTES_(LVL) + +/****************************** Error codes ***********************************/ + +/* Generic failure condition, reserved for failures not covered by a more + * specific error code. */ +#define MLD_ERR_FAIL (-1) +/* An allocation failed. This can only happen if MLD_CONFIG_CUSTOM_ALLOC_FREE + * is defined and the provided MLD_CUSTOM_ALLOC can fail. */ +#define MLD_ERR_OUT_OF_MEMORY (-2) +/* An RNG failure occurred. Might be due to insufficient entropy or + * system misconfiguration. */ +#define MLD_ERR_RNG_FAIL (-3) +/* The signing rejection-sampling loop exceeded + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS iterations without producing a valid + * signature. With a FIPS 204 Appendix C compliant bound (>= 821) this + * has probability < 2^-256. */ +#define MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED (-4) +/* Signing was paused before completing, at the request of a caller-provided + * MLD_CONFIG_SIGN_HOOK_ATTEMPT hook (see mldsa_native_config.h). The caller + * resumes by re-invoking signing with the same inputs; the attempt hook, + * together with MLD_CONFIG_SIGN_HOOK_RESUME, decides where to continue. */ +#define MLD_ERR_SIGNING_PAUSED (-5) +/* Signature verification failed: the signature is not valid for the given + * message and public key. Returned by the verification API. */ +#define MLD_ERR_INVALID_SIGNATURE (-6) +/* Secret key validation failed: the secret key is malformed or internally + * inconsistent. Returned by pk_from_sk. */ +#define MLD_ERR_INVALID_KEY (-7) +/* The Pairwise Consistency Test failed. Only possible when + * MLD_CONFIG_KEYGEN_PCT is enabled; signals that the freshly generated key + * pair failed its sign/verify self-test. */ +#define MLD_ERR_PCT_FAIL (-8) +/* An argument was invalid, e.g. an unsupported pre-hash algorithm or a context + * string longer than 255 bytes. */ +#define MLD_ERR_INVALID_ARG (-9) + +/********************* Namespacing and Qualifiers *****************************/ + +#define MLD_API_CONCAT_(x, y) x##y +#define MLD_API_CONCAT(x, y) MLD_API_CONCAT_(x, y) +#define MLD_API_CONCAT_UNDERSCORE(x, y) MLD_API_CONCAT(MLD_API_CONCAT(x, _), y) + +/* You need to make sure the config file is in the include path. */ +#if defined(MLD_CONFIG_FILE) +#include MLD_CONFIG_FILE +#else +#include "mldsa_native_config.h" +#endif + +/* Namespace prefix for the public API symbols. For multi-level builds, the + * parameter set is appended to disambiguate the security levels. */ +#if defined(MLD_CONFIG_MULTILEVEL_BUILD) +#define MLD_API_NAMESPACE_PREFIX \ + MLD_API_CONCAT(MLD_CONFIG_NAMESPACE_PREFIX, MLD_CONFIG_PARAMETER_SET) +#else +#define MLD_API_NAMESPACE_PREFIX MLD_CONFIG_NAMESPACE_PREFIX +#endif + +#define MLD_API_NAMESPACE(sym) \ + MLD_API_CONCAT_UNDERSCORE(MLD_API_NAMESPACE_PREFIX, sym) + +#if defined(__GNUC__) || defined(__clang__) +#define MLD_API_MUST_CHECK_RETURN_VALUE __attribute__((warn_unused_result)) +#else +#define MLD_API_MUST_CHECK_RETURN_VALUE +#endif + +#if defined(MLD_CONFIG_EXTERNAL_API_QUALIFIER) +#define MLD_API_QUALIFIER MLD_CONFIG_EXTERNAL_API_QUALIFIER +#else +#define MLD_API_QUALIFIER +#endif + +/* Hash algorithm constants for domain separation */ +#define MLD_PREHASH_NONE 0 +#define MLD_PREHASH_SHA2_224 1 +#define MLD_PREHASH_SHA2_256 2 +#define MLD_PREHASH_SHA2_384 3 +#define MLD_PREHASH_SHA2_512 4 +#define MLD_PREHASH_SHA2_512_224 5 +#define MLD_PREHASH_SHA2_512_256 6 +#define MLD_PREHASH_SHA3_224 7 +#define MLD_PREHASH_SHA3_256 8 +#define MLD_PREHASH_SHA3_384 9 +#define MLD_PREHASH_SHA3_512 10 +#define MLD_PREHASH_SHAKE_128 11 +#define MLD_PREHASH_SHAKE_256 12 + +/* Maximum formatted domain separation message length */ +#define MLD_DOMAIN_SEPARATION_MAX_BYTES (2 + 255 + 11 + 64) + +/****************************** Function API **********************************/ + +#if !defined(MLD_CONFIG_CONSTANTS_ONLY) + +#include +#include + +#ifdef __cplusplus +extern "C" +{ +#endif + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +/** + * Generate a public-private key pair from a seed. + * + * When MLD_CONFIG_KEYGEN_PCT is set, performs a Pairwise Consistency Test + * (PCT) as required by FIPS 140-3 IG. + * + * @warning The seed must be generated by a cryptographically secure random + * number generator. + * + * @spec{Implements @[FIPS204, Algorithm 6, ML-DSA.KeyGen_internal].} + * + * @param[out] pk Output public key. + * @param[out] sk Output private key. + * @param[in] seed Input random seed. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_RNG_FAIL Random number generation failed + * during the PCT. Only possible when + * MLD_CONFIG_KEYGEN_PCT is enabled. + * @retval MLD_ERR_SIGNING_PAUSED The PCT's signing step was paused by + * a MLD_CONFIG_SIGN_HOOK_ATTEMPT hook. + * This should currently never happen: + * signing hooks require + * MLD_CONFIG_NO_RANDOMIZED_API, which + * is incompatible with + * MLD_CONFIG_KEYGEN_PCT, so the two + * cannot be enabled simultaneously. + * @retval MLD_ERR_PCT_FAIL MLD_CONFIG_KEYGEN_PCT is enabled and + * the PCT check failed. + */ +MLD_API_QUALIFIER +MLD_API_MUST_CHECK_RETURN_VALUE +int MLD_API_NAMESPACE(keypair_internal)( + uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)], + uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)], + const uint8_t seed[MLDSA_SEEDBYTES] +#ifdef MLD_CONFIG_CONTEXT_PARAMETER + , + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); + +#if !defined(MLD_CONFIG_CORE_API_ONLY) +#if !defined(MLD_CONFIG_NO_RANDOMIZED_API) +/** + * Generate a public-private key pair. + * + * When MLD_CONFIG_KEYGEN_PCT is set, performs a Pairwise Consistency Test + * (PCT) as required by FIPS 140-3 IG. + * + * @spec{Implements @[FIPS204, Algorithm 1, ML-DSA.KeyGen].} + * + * @param[out] pk Output public key. + * @param[out] sk Output private key. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_RNG_FAIL Random number generation failed. + * @retval MLD_ERR_SIGNING_PAUSED The PCT's signing step was paused by + * a MLD_CONFIG_SIGN_HOOK_ATTEMPT hook. + * This should currently never happen: + * signing hooks require + * MLD_CONFIG_NO_RANDOMIZED_API, which + * is incompatible with + * MLD_CONFIG_KEYGEN_PCT, so the two + * cannot be enabled simultaneously. + * @retval MLD_ERR_PCT_FAIL MLD_CONFIG_KEYGEN_PCT is enabled and + * the PCT check failed. + */ +MLD_API_QUALIFIER +MLD_API_MUST_CHECK_RETURN_VALUE +int MLD_API_NAMESPACE(keypair)( + uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)], + uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)] +#ifdef MLD_CONFIG_CONTEXT_PARAMETER + , + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); +#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */ +#endif /* !MLD_CONFIG_CORE_API_ONLY */ +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +/** + * Compute signature using a caller-supplied random seed and prefix. + * + * On error (non-zero return value), the signature buffer sig is zeroized. + * + * @spec{Implements @[FIPS204, Algorithm 7, ML-DSA.Sign_internal].} + * + * @warning This function does not perform secret key validation. + * Callers importing serialized keys can use pk_from_sk + * to validate them before signing. + * + * @param[out] sig Pointer to buffer to hold the generated signature of + * MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes. + * @param[in] m Pointer to message to be signed (when + * externalmu == 0), or to a precomputed + * message representative mu (when externalmu != 0). + * @param mlen Length of m. Must equal MLDSA_CRHBYTES when + * externalmu != 0. + * @param[in] pre Pointer to prefix string. Ignored when + * externalmu != 0. + * @param prelen Length of prefix string. Ignored when + * externalmu != 0. + * @param[in] rnd Random seed. + * @param[in] sk Bit-packed secret key; assumed to be valid. + * @param externalmu 0: m/mlen is the raw message; mu = H(tr, pre, m) is + * computed internally. + * non-zero: m points to a precomputed mu of + * MLDSA_CRHBYTES bytes; pre/prelen unused. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS + * iterations. + * @retval MLD_ERR_SIGNING_PAUSED A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook + * paused signing; re-invoke to resume. + */ +MLD_API_QUALIFIER +MLD_API_MUST_CHECK_RETURN_VALUE +int MLD_API_NAMESPACE(signature_internal)( + uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *m, + size_t mlen, const uint8_t *pre, size_t prelen, + const uint8_t rnd[MLDSA_RNDBYTES], + const uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)], + int externalmu +#ifdef MLD_CONFIG_CONTEXT_PARAMETER + , + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); + +#if !defined(MLD_CONFIG_CORE_API_ONLY) +#if !defined(MLD_CONFIG_NO_RANDOMIZED_API) +/** + * Compute signature. This function implements the randomized variant of + * ML-DSA. If you require the deterministic variant, use + * signature_internal directly. + * + * @spec{Implements @[FIPS204, Algorithm 2, ML-DSA.Sign].} + * + * @warning This function does not perform secret key validation. + * Callers importing serialized keys can use pk_from_sk + * to validate them before signing. + * + * @param[out] sig Pointer to buffer to hold the generated signature of + * MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes. + * @param[in] m Pointer to message to be signed. May be NULL if + * mlen == 0. + * @param mlen Length of message. + * @param[in] ctx Pointer to context string. May be NULL if ctxlen == 0. + * @param ctxlen Length of context string. Should be <= 255. + * @param[in] sk Bit-packed secret key; assumed to be valid. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_RNG_FAIL Random number generation failed. + * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS + * iterations. + * @retval MLD_ERR_SIGNING_PAUSED A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook + * paused signing; re-invoke to resume. + * @retval MLD_ERR_INVALID_ARG The context string exceeded 255 + * bytes. + */ +MLD_API_QUALIFIER +MLD_API_MUST_CHECK_RETURN_VALUE +int MLD_API_NAMESPACE(signature)( + uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *m, + size_t mlen, const uint8_t *ctx, size_t ctxlen, + const uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)] +#ifdef MLD_CONFIG_CONTEXT_PARAMETER + , + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); + +/** + * Compute signature in "external mu" mode: the caller has already computed + * the message representative mu = SHAKE256(tr || M', 64), where + * tr = SHAKE256(pk, 64) and M' is the FIPS 204 formatted message (e.g. + * 0x00 || ctxlen || ctx || msg for pure ML-DSA). This is useful when the + * message is large or streamed and cannot be held in memory. + * + * @spec{Implements @[FIPS204, Algorithm 2, ML-DSA.Sign external mu variant].} + * + * @warning This function does not perform secret key validation. + * Callers importing serialized keys can use pk_from_sk + * to validate them before signing. + * + * @param[out] sig Pointer to buffer to hold the generated signature of + * MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes. + * @param[in] mu Precomputed message representative. + * @param[in] sk Bit-packed secret key; assumed to be valid. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_RNG_FAIL Random number generation failed. + * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS + * iterations. + * @retval MLD_ERR_SIGNING_PAUSED A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook + * paused signing; re-invoke to resume. + */ +MLD_API_QUALIFIER +MLD_API_MUST_CHECK_RETURN_VALUE +int MLD_API_NAMESPACE(signature_extmu)( + uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], + const uint8_t mu[MLDSA_CRHBYTES], + const uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)] +#ifdef MLD_CONFIG_CONTEXT_PARAMETER + , + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); + +#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */ +#endif /* !MLD_CONFIG_CORE_API_ONLY */ +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +/** + * Verify signature. Internal API. + * + * @spec{Implements @[FIPS204, Algorithm 8, ML-DSA.Verify_internal].} + * + * @param[in] sig Pointer to input signature of + * MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes. + * @param[in] m Pointer to message (when externalmu == 0), or to a + * precomputed message representative mu (when + * externalmu != 0). + * @param mlen Length of m. Must equal MLDSA_CRHBYTES when + * externalmu != 0. + * @param[in] pre Pointer to prefix string. Ignored when externalmu != 0. + * @param prelen Length of prefix string. Ignored when externalmu != 0. + * @param[in] pk Bit-packed public key. + * @param externalmu 0: m/mlen is the raw message; mu = H(H(pk), pre, m) is + * computed internally. + * non-zero: m points to a precomputed mu of + * MLDSA_CRHBYTES bytes; pre/prelen unused. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed. + */ +MLD_API_QUALIFIER +MLD_API_MUST_CHECK_RETURN_VALUE +int MLD_API_NAMESPACE(verify_internal)( + const uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *m, + size_t mlen, const uint8_t *pre, size_t prelen, + const uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)], + int externalmu +#ifdef MLD_CONFIG_CONTEXT_PARAMETER + , + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); + +#if !defined(MLD_CONFIG_CORE_API_ONLY) +/** + * Verify signature. + * + * @spec{Implements @[FIPS204, Algorithm 3, ML-DSA.Verify].} + * + * @param[in] sig Pointer to input signature of + * MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes. + * @param[in] m Pointer to message. May be NULL if mlen == 0. + * @param mlen Length of message. + * @param[in] ctx Pointer to context string. May be NULL if ctxlen == 0. + * @param ctxlen Length of context string. + * @param[in] pk Bit-packed public key. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed. + * @retval MLD_ERR_INVALID_ARG The context string exceeded 255 bytes. + */ +MLD_API_QUALIFIER +MLD_API_MUST_CHECK_RETURN_VALUE +int MLD_API_NAMESPACE(verify)( + const uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *m, + size_t mlen, const uint8_t *ctx, size_t ctxlen, + const uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)] +#ifdef MLD_CONFIG_CONTEXT_PARAMETER + , + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); + +/** + * Verify signature in "external mu" mode: the caller has already computed + * the message representative mu = SHAKE256(tr || M', 64), where + * tr = SHAKE256(pk, 64) and M' is the FIPS 204 formatted message (e.g. + * 0x00 || ctxlen || ctx || msg for pure ML-DSA). The same mu must have + * been used at signing time. + * + * @spec{Implements @[FIPS204, Algorithm 3, ML-DSA.Verify external mu variant].} + * + * @param[in] sig Pointer to input signature of + * MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes. + * @param[in] mu Precomputed message representative. + * @param[in] pk Bit-packed public key. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed. + */ +MLD_API_QUALIFIER +MLD_API_MUST_CHECK_RETURN_VALUE +int MLD_API_NAMESPACE(verify_extmu)( + const uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], + const uint8_t mu[MLDSA_CRHBYTES], + const uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)] +#ifdef MLD_CONFIG_CONTEXT_PARAMETER + , + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); + +#endif /* !MLD_CONFIG_CORE_API_ONLY */ +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_CORE_API_ONLY) +#if !defined(MLD_CONFIG_NO_SIGN_API) +/** + * Compute signature with pre-hashed message. + * + * @spec{Implements @[FIPS204, Algorithm 4, HashML-DSA.Sign].} + * + * Supported hash algorithm constants: + * MLD_PREHASH_SHA2_224, MLD_PREHASH_SHA2_256, MLD_PREHASH_SHA2_384, + * MLD_PREHASH_SHA2_512, MLD_PREHASH_SHA2_512_224, MLD_PREHASH_SHA2_512_256, + * MLD_PREHASH_SHA3_224, MLD_PREHASH_SHA3_256, MLD_PREHASH_SHA3_384, + * MLD_PREHASH_SHA3_512, MLD_PREHASH_SHAKE_128, MLD_PREHASH_SHAKE_256. + * + * MLD_PREHASH_NONE is rejected by this API. + * + * @warning This is an unstable API that may change in the future. If you need + * a stable API use signature_pre_hash_shake256. + * + * @warning This function does not perform secret key validation. + * Callers importing serialized keys can use pk_from_sk + * to validate them before signing. + * + * @param[out] sig Pointer to buffer to hold the generated signature of + * MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes. + * @param[in] ph Pointer to pre-hashed message. + * @param phlen Length of pre-hashed message. Must match the output + * length of hashalg (the digest size for SHA-2/SHA-3, + * 32 bytes for MLD_PREHASH_SHAKE_128, 64 bytes for + * MLD_PREHASH_SHAKE_256). + * @param[in] ctx Pointer to context string. May be NULL if ctxlen == 0. + * @param ctxlen Length of context string. + * @param[in] rnd Random seed. + * @param[in] sk Bit-packed secret key; assumed to be valid. + * @param hashalg Hash algorithm constant (one of MLD_PREHASH_*). + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS + * iterations. + * @retval MLD_ERR_SIGNING_PAUSED A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook + * paused signing; re-invoke to resume. + * @retval MLD_ERR_INVALID_ARG The pre-hash algorithm was + * MLD_PREHASH_NONE or unsupported, + * phlen did not match the output + * length of hashalg, or the context + * string exceeded 255 bytes. + */ +MLD_API_QUALIFIER +MLD_API_MUST_CHECK_RETURN_VALUE +int MLD_API_NAMESPACE(signature_pre_hash_internal)( + uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *ph, + size_t phlen, const uint8_t *ctx, size_t ctxlen, + const uint8_t rnd[MLDSA_RNDBYTES], + const uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)], + int hashalg +#ifdef MLD_CONFIG_CONTEXT_PARAMETER + , + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +/** + * Verifies signature with pre-hashed message. + * + * @spec{Implements @[FIPS204, Algorithm 5, HashML-DSA.Verify].} + * + * Supported hash algorithm constants: + * MLD_PREHASH_SHA2_224, MLD_PREHASH_SHA2_256, MLD_PREHASH_SHA2_384, + * MLD_PREHASH_SHA2_512, MLD_PREHASH_SHA2_512_224, MLD_PREHASH_SHA2_512_256, + * MLD_PREHASH_SHA3_224, MLD_PREHASH_SHA3_256, MLD_PREHASH_SHA3_384, + * MLD_PREHASH_SHA3_512, MLD_PREHASH_SHAKE_128, MLD_PREHASH_SHAKE_256. + * + * MLD_PREHASH_NONE is rejected by this API. + * + * @warning This is an unstable API that may change in the future. If you need + * a stable API use verify_pre_hash_shake256. + * + * @param[in] sig Pointer to input signature of + * MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes. + * @param[in] ph Pointer to pre-hashed message. + * @param phlen Length of pre-hashed message. Must match the output + * length of hashalg (the digest size for SHA-2/SHA-3, + * 32 bytes for MLD_PREHASH_SHAKE_128, 64 bytes for + * MLD_PREHASH_SHAKE_256). + * @param[in] ctx Pointer to context string. May be NULL if ctxlen == 0. + * @param ctxlen Length of context string. + * @param[in] pk Bit-packed public key. + * @param hashalg Hash algorithm constant (one of MLD_PREHASH_*). + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed. + * @retval MLD_ERR_INVALID_ARG The pre-hash algorithm was + * MLD_PREHASH_NONE or unsupported, phlen + * did not match the output length of + * hashalg, or the context string exceeded + * 255 bytes. + */ +MLD_API_QUALIFIER +MLD_API_MUST_CHECK_RETURN_VALUE +int MLD_API_NAMESPACE(verify_pre_hash_internal)( + const uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *ph, + size_t phlen, const uint8_t *ctx, size_t ctxlen, + const uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)], + int hashalg +#ifdef MLD_CONFIG_CONTEXT_PARAMETER + , + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +/** + * Compute signature with pre-hashed message using SHAKE256. This function + * computes the SHAKE256 hash of the message internally. + * + * @spec{Implements @[FIPS204, Algorithm 4, HashML-DSA.Sign] with SHAKE256 as + * the pre-hash.} + * + * @warning This function does not perform secret key validation. + * Callers importing serialized keys can use pk_from_sk + * to validate them before signing. + * + * @param[out] sig Pointer to buffer to hold the generated signature of + * MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes. + * @param[in] m Pointer to message to be hashed and signed. May be + * NULL if mlen == 0. + * @param mlen Length of message. + * @param[in] ctx Pointer to context string. May be NULL if ctxlen == 0. + * @param ctxlen Length of context string. + * @param[in] rnd Random seed. + * @param[in] sk Bit-packed secret key; assumed to be valid. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS + * iterations. + * @retval MLD_ERR_SIGNING_PAUSED A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook + * paused signing; re-invoke to resume. + * @retval MLD_ERR_INVALID_ARG The context string exceeded 255 + * bytes. + */ +MLD_API_QUALIFIER +MLD_API_MUST_CHECK_RETURN_VALUE +int MLD_API_NAMESPACE(signature_pre_hash_shake256)( + uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *m, + size_t mlen, const uint8_t *ctx, size_t ctxlen, + const uint8_t rnd[MLDSA_RNDBYTES], + const uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)] +#ifdef MLD_CONFIG_CONTEXT_PARAMETER + , + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +/** + * Verify signature with pre-hashed message using SHAKE256. This function + * computes the SHAKE256 hash of the message internally. + * + * @spec{Implements @[FIPS204, Algorithm 5, HashML-DSA.Verify] with SHAKE256 as + * the pre-hash.} + * + * @param[in] sig Pointer to input signature of + * MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) bytes. + * @param[in] m Pointer to message to be hashed and verified. May be + * NULL if mlen == 0. + * @param mlen Length of message. + * @param[in] ctx Pointer to context string. May be NULL if ctxlen == 0. + * @param ctxlen Length of context string. + * @param[in] pk Bit-packed public key. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed. + * @retval MLD_ERR_INVALID_ARG The context string exceeded 255 bytes. + */ +MLD_API_QUALIFIER +MLD_API_MUST_CHECK_RETURN_VALUE +int MLD_API_NAMESPACE(verify_pre_hash_shake256)( + const uint8_t sig[MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET)], const uint8_t *m, + size_t mlen, const uint8_t *ctx, size_t ctxlen, + const uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)] +#ifdef MLD_CONFIG_CONTEXT_PARAMETER + , + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); +#endif /* !MLD_CONFIG_NO_VERIFY_API */ +#endif /* !MLD_CONFIG_CORE_API_ONLY */ + +#if !defined(MLD_CONFIG_CORE_API_ONLY) +/** + * Prepare domain separation prefix for ML-DSA signing. + * + * For pure ML-DSA (hashalg == MLD_PREHASH_NONE): + * Format: 0x00 || ctxlen (1 byte) || ctx. + * + * For HashML-DSA (hashalg != MLD_PREHASH_NONE): + * Format: 0x01 || ctxlen (1 byte) || ctx || oid (11 bytes) || ph. + * + * This function is useful for building incremental signing APIs. + * + * @spec{For HashML-DSA (hashalg != MLD_PREHASH_NONE), implements + * @[FIPS204, Algorithm 4, line 23]. For Pure ML-DSA + * (hashalg == MLD_PREHASH_NONE), implements + * ``` + * M' <- BytesToBits(IntegerToBytes(0, 1) + * || IntegerToBytes(|ctx|, 1) + * || ctx + * ``` + * which is part of @[FIPS204, Algorithm 2, ML-DSA.Sign, line 10] and + * @[FIPS204, Algorithm 3, ML-DSA.Verify, line 5].} + * + * @param[out] prefix Output domain separation prefix buffer. + * @param[in] ph Pointer to pre-hashed message (ignored for pure + * ML-DSA). + * @param phlen Length of pre-hashed message; must match the output + * length of hashalg (ignored for pure ML-DSA). + * @param[in] ctx Pointer to context string. May be NULL if ctxlen == 0. + * @param ctxlen Length of context string. + * @param hashalg Hash algorithm constant (MLD_PREHASH_NONE for pure + * ML-DSA, or MLD_PREHASH_* for HashML-DSA). + * + * @return The total length of the formatted prefix, or 0 on error. + * Errors are: + * - The context string exceeded 255 bytes. + * - For HashML-DSA: hashalg was unsupported, ph was NULL, or phlen + * did not match the output length of hashalg. + */ +MLD_API_QUALIFIER +MLD_API_MUST_CHECK_RETURN_VALUE +size_t MLD_API_NAMESPACE(prepare_domain_separation_prefix)( + uint8_t prefix[MLD_DOMAIN_SEPARATION_MAX_BYTES], const uint8_t *ph, + size_t phlen, const uint8_t *ctx, size_t ctxlen, int hashalg); + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +/** + * Perform basic validity checks on secret key, and derive public key. + * + * Referring to the decoding of the secret key `sk=(rho, K, tr, s1, s2, t0)` + * (cf. @[FIPS204, Algorithm 25, skDecode]), the following checks are + * performed: + * - Check that s1 and s2 have coefficients in [-MLDSA_ETA, MLDSA_ETA]. + * - Check that t0 and tr stored in sk match recomputed values. + * + * @note This function leaks whether the secret key is valid or invalid + * through its return value and timing. + * + * @param[out] pk Output public key. + * @param[in] sk Input secret key. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was used and an + * allocation via MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_INVALID_KEY Secret key validation failed. + */ +MLD_API_QUALIFIER +MLD_API_MUST_CHECK_RETURN_VALUE +int MLD_API_NAMESPACE(pk_from_sk)( + uint8_t pk[MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET)], + const uint8_t sk[MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET)] +#ifdef MLD_CONFIG_CONTEXT_PARAMETER + , + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ +#endif /* !MLD_CONFIG_CORE_API_ONLY */ + +#ifdef __cplusplus +} +#endif + +#undef MLD_API_NAMESPACE_PREFIX + +#endif /* !MLD_CONFIG_CONSTANTS_ONLY */ + +/***************************** Memory Usage **********************************/ + +/* + * By default mldsa-native performs all memory allocations on the stack. + * Alternatively, mldsa-native supports custom allocation of large structures + * through the `MLD_CONFIG_CUSTOM_ALLOC_FREE` configuration option. + * See mldsa_native_config.h for details. + * + * `MLD_TOTAL_ALLOC_{44,65,87}_{KEYPAIR,SIGN,VERIFY}` indicates the maximum + * (accumulative) allocation via MLD_ALLOC for each parameter set and operation. + * Note that some stack allocation remains even + * when using custom allocators, so these values are lower than total stack + * usage with the default stack-only allocation. + * + * These constants may be used to implement custom allocations using a + * fixed-sized buffer and a simple allocator (e.g., bump allocator). + */ +/* check-magic: off */ +#if !defined(MLD_CONFIG_REDUCE_RAM) +#define MLD_TOTAL_ALLOC_44_KEYPAIR_NO_PCT 26912 +#define MLD_TOTAL_ALLOC_44_KEYPAIR_PCT 48480 +#define MLD_TOTAL_ALLOC_44_PK_FROM_SK 28480 +#define MLD_TOTAL_ALLOC_44_SIGN 44704 +#define MLD_TOTAL_ALLOC_44_VERIFY 24448 +#define MLD_TOTAL_ALLOC_65_KEYPAIR_NO_PCT 44320 +#define MLD_TOTAL_ALLOC_65_KEYPAIR_PCT 74624 +#define MLD_TOTAL_ALLOC_65_PK_FROM_SK 46720 +#define MLD_TOTAL_ALLOC_65_SIGN 69312 +#define MLD_TOTAL_ALLOC_65_VERIFY 39872 +#define MLD_TOTAL_ALLOC_87_KEYPAIR_NO_PCT 75040 +#define MLD_TOTAL_ALLOC_87_KEYPAIR_PCT 115488 +#define MLD_TOTAL_ALLOC_87_PK_FROM_SK 78272 +#define MLD_TOTAL_ALLOC_87_SIGN 108224 +#define MLD_TOTAL_ALLOC_87_VERIFY 68800 +#else /* !MLD_CONFIG_REDUCE_RAM */ +#define MLD_TOTAL_ALLOC_44_KEYPAIR_NO_PCT 11584 +#define MLD_TOTAL_ALLOC_44_KEYPAIR_PCT 16896 +#define MLD_TOTAL_ALLOC_44_PK_FROM_SK 13152 +#define MLD_TOTAL_ALLOC_44_SIGN 13120 +#define MLD_TOTAL_ALLOC_44_VERIFY 9120 +#define MLD_TOTAL_ALLOC_65_KEYPAIR_NO_PCT 14656 +#define MLD_TOTAL_ALLOC_65_KEYPAIR_PCT 22560 +#define MLD_TOTAL_ALLOC_65_PK_FROM_SK 17056 +#define MLD_TOTAL_ALLOC_65_SIGN 17248 +#define MLD_TOTAL_ALLOC_65_VERIFY 10208 +#define MLD_TOTAL_ALLOC_87_KEYPAIR_NO_PCT 18752 +#define MLD_TOTAL_ALLOC_87_KEYPAIR_PCT 28608 +#define MLD_TOTAL_ALLOC_87_PK_FROM_SK 21984 +#define MLD_TOTAL_ALLOC_87_SIGN 21344 +#define MLD_TOTAL_ALLOC_87_VERIFY 12512 +#endif /* MLD_CONFIG_REDUCE_RAM */ +/* check-magic: on */ + +/* + * MLD_TOTAL_ALLOC_*_KEYPAIR adapts based on MLD_CONFIG_KEYGEN_PCT. + */ +#if defined(MLD_CONFIG_KEYGEN_PCT) +#define MLD_TOTAL_ALLOC_44_KEYPAIR MLD_TOTAL_ALLOC_44_KEYPAIR_PCT +#define MLD_TOTAL_ALLOC_65_KEYPAIR MLD_TOTAL_ALLOC_65_KEYPAIR_PCT +#define MLD_TOTAL_ALLOC_87_KEYPAIR MLD_TOTAL_ALLOC_87_KEYPAIR_PCT +#else +#define MLD_TOTAL_ALLOC_44_KEYPAIR MLD_TOTAL_ALLOC_44_KEYPAIR_NO_PCT +#define MLD_TOTAL_ALLOC_65_KEYPAIR MLD_TOTAL_ALLOC_65_KEYPAIR_NO_PCT +#define MLD_TOTAL_ALLOC_87_KEYPAIR MLD_TOTAL_ALLOC_87_KEYPAIR_NO_PCT +#endif + +#define MLD_MAX3_(a, b, c) \ + ((a) > (b) ? ((a) > (c) ? (a) : (c)) : ((b) > (c) ? (b) : (c))) +#define MLD_MAX4_(a, b, c, d) MLD_MAX3_((a), (b), MLD_MAX3_((c), (d), (d))) + +/* + * `MLD_TOTAL_ALLOC_{44,65,87}` is the maximum across standard API operations + * (keygen, sign, verify) for each parameter set. + */ +#define MLD_TOTAL_ALLOC_44 \ + MLD_MAX4_(MLD_TOTAL_ALLOC_44_KEYPAIR, MLD_TOTAL_ALLOC_44_PK_FROM_SK, \ + MLD_TOTAL_ALLOC_44_SIGN, MLD_TOTAL_ALLOC_44_VERIFY) +#define MLD_TOTAL_ALLOC_65 \ + MLD_MAX4_(MLD_TOTAL_ALLOC_65_KEYPAIR, MLD_TOTAL_ALLOC_65_PK_FROM_SK, \ + MLD_TOTAL_ALLOC_65_SIGN, MLD_TOTAL_ALLOC_65_VERIFY) +#define MLD_TOTAL_ALLOC_87 \ + MLD_MAX4_(MLD_TOTAL_ALLOC_87_KEYPAIR, MLD_TOTAL_ALLOC_87_PK_FROM_SK, \ + MLD_TOTAL_ALLOC_87_SIGN, MLD_TOTAL_ALLOC_87_VERIFY) + +#endif /* !MLD_H */ diff --git a/cbits/mldsa/mldsa_native_asm.S b/cbits/mldsa/mldsa_native_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/mldsa_native_asm.S @@ -0,0 +1,830 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mldsa-native repository. + * Do not modify it directly. + */ + +/****************************************************************************** + * + * Single assembly unit for fixed-level build of mldsa-native + * + * This assembly unit bundles together all assembly files for a build + * of mldsa-native for a fixed security level (MLDSA-44/65/87). + * + * # Multi-level build + * + * If you want an SCU build of mldsa-native with support for multiple security + * levels, you should include this file once with + * MLD_CONFIG_MULTILEVEL_WITH_SHARED set. + * + * (You could also follow the same pattern as for mldsa_native.c + * and include it for every level, setting MLD_CONFIG_MULTILEVEL_NO_SHARED + * for all but one. For builds with MLD_CONFIG_MULTILEVEL_NO_SHARED, this + * file will then be ignored.) + * + * # Configuration + * + * The following options from the mldsa-native configuration are relevant: + * + * - MLD_CONFIG_FIPS202_CUSTOM_HEADER + * Set this option if you use a custom FIPS202 implementation. + * + * - MLD_CONFIG_USE_NATIVE_BACKEND_ARITH + * Set this option if you want to include the native arithmetic backends + * in your build. + * + * - MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 + * Set this option if you want to include the native FIPS202 backends + * in your build. + * + * - MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS + * Set this option if you want to keep the directives defined in + * level-independent headers. This is needed for a multi-level build. + */ + +/* If parts of the mldsa-native source tree are not used, + * consider reducing this header via `unifdef`. + * + * Example: + * ```bash + * unifdef -UMLD_CONFIG_USE_NATIVE_BACKEND_ARITH mldsa_native_asm.S + * ``` + */ + +#include "src/common.h" + +#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH) +#if defined(MLD_SYS_AARCH64) +#include "src/native/aarch64/src/mldsa_intt_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_ntt_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_pointwise_montgomery_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_poly_caddq_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_poly_chknorm_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_poly_decompose_32_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_poly_decompose_88_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_poly_use_hint_32_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_poly_use_hint_88_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_polyz_unpack_17_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_polyz_unpack_19_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_rej_uniform_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_rej_uniform_eta2_aarch64_asm.S" +#include "src/native/aarch64/src/mldsa_rej_uniform_eta4_aarch64_asm.S" +#endif /* MLD_SYS_AARCH64 */ +#if defined(MLD_SYS_X86_64) +#include "src/native/x86_64/src/mldsa_intt_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_ntt_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_nttunpack_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_pointwise_acc_l4_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_pointwise_acc_l5_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_pointwise_acc_l7_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_pointwise_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_poly_caddq_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_poly_chknorm_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_poly_decompose_32_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_poly_decompose_88_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_poly_use_hint_32_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_poly_use_hint_88_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_polyz_unpack_17_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_polyz_unpack_19_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_rej_uniform_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_rej_uniform_eta2_avx2_asm.S" +#include "src/native/x86_64/src/mldsa_rej_uniform_eta4_avx2_asm.S" +#endif /* MLD_SYS_X86_64 */ +#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_ARITH */ + +#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202) +#if defined(MLD_SYS_AARCH64) +#include "src/fips202/native/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S" +#include "src/fips202/native/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S" +#include "src/fips202/native/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S" +#include "src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S" +#include "src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S" +#endif /* MLD_SYS_AARCH64 */ +#if defined(MLD_SYS_X86_64) +#include "src/fips202/native/x86_64/src/keccak_f1600_x4_avx2_asm.S" +#endif +#if defined(MLD_SYS_ARMV81M_MVE) +#include "src/fips202/native/armv81m/src/keccak_f1600_x4_mve.S" +#include "src/fips202/native/armv81m/src/keccak_f1600_x4_state_extract_bytes_mve.S" +#include "src/fips202/native/armv81m/src/keccak_f1600_x4_state_xor_bytes_mve.S" +#endif +#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 */ + + +/* Macro #undef's + * + * The following undefines macros from headers + * included by the source files imported above. + * + * This is to allow building and linking multiple builds + * of mldsa-native for varying parameter sets through concatenation + * of this file, as if the files had been compiled separately. + * If this is not relevant to you, you may remove the following. + * + * NOTE: This is not needed for the assembly SCU since, at present, + * there is no need to include it multiple times. + * We keep it for uniformity with mldsa_native.c only. + * + * NOTE: To avoid having to distinguish between which headers are included + * from the assembly files, we #undef the same set of directives + * as in mldsa_native.c + */ + +/* + * Undefine macros from MLD_CONFIG_PARAMETER_SET-specific files + */ +/* mldsa/mldsa_native.h */ +#undef MLDSA44_BYTES +#undef MLDSA44_CRHBYTES +#undef MLDSA44_PUBLICKEYBYTES +#undef MLDSA44_RNDBYTES +#undef MLDSA44_SECRETKEYBYTES +#undef MLDSA44_SEEDBYTES +#undef MLDSA44_TRBYTES +#undef MLDSA65_BYTES +#undef MLDSA65_CRHBYTES +#undef MLDSA65_PUBLICKEYBYTES +#undef MLDSA65_RNDBYTES +#undef MLDSA65_SECRETKEYBYTES +#undef MLDSA65_SEEDBYTES +#undef MLDSA65_TRBYTES +#undef MLDSA87_BYTES +#undef MLDSA87_CRHBYTES +#undef MLDSA87_PUBLICKEYBYTES +#undef MLDSA87_RNDBYTES +#undef MLDSA87_SECRETKEYBYTES +#undef MLDSA87_SEEDBYTES +#undef MLDSA87_TRBYTES +#undef MLDSA_BYTES +#undef MLDSA_BYTES_ +#undef MLDSA_CRHBYTES +#undef MLDSA_PUBLICKEYBYTES +#undef MLDSA_PUBLICKEYBYTES_ +#undef MLDSA_RNDBYTES +#undef MLDSA_SECRETKEYBYTES +#undef MLDSA_SECRETKEYBYTES_ +#undef MLDSA_SEEDBYTES +#undef MLDSA_TRBYTES +#undef MLD_API_CONCAT +#undef MLD_API_CONCAT_ +#undef MLD_API_CONCAT_UNDERSCORE +#undef MLD_API_MUST_CHECK_RETURN_VALUE +#undef MLD_API_NAMESPACE +#undef MLD_API_NAMESPACE_PREFIX +#undef MLD_API_QUALIFIER +#undef MLD_DOMAIN_SEPARATION_MAX_BYTES +#undef MLD_ERR_FAIL +#undef MLD_ERR_INVALID_ARG +#undef MLD_ERR_INVALID_KEY +#undef MLD_ERR_INVALID_SIGNATURE +#undef MLD_ERR_OUT_OF_MEMORY +#undef MLD_ERR_PCT_FAIL +#undef MLD_ERR_RNG_FAIL +#undef MLD_ERR_SIGNING_PAUSED +#undef MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED +#undef MLD_H +#undef MLD_MAX3_ +#undef MLD_MAX4_ +#undef MLD_PREHASH_NONE +#undef MLD_PREHASH_SHA2_224 +#undef MLD_PREHASH_SHA2_256 +#undef MLD_PREHASH_SHA2_384 +#undef MLD_PREHASH_SHA2_512 +#undef MLD_PREHASH_SHA2_512_224 +#undef MLD_PREHASH_SHA2_512_256 +#undef MLD_PREHASH_SHA3_224 +#undef MLD_PREHASH_SHA3_256 +#undef MLD_PREHASH_SHA3_384 +#undef MLD_PREHASH_SHA3_512 +#undef MLD_PREHASH_SHAKE_128 +#undef MLD_PREHASH_SHAKE_256 +#undef MLD_TOTAL_ALLOC_44 +#undef MLD_TOTAL_ALLOC_44_KEYPAIR +#undef MLD_TOTAL_ALLOC_44_KEYPAIR_NO_PCT +#undef MLD_TOTAL_ALLOC_44_KEYPAIR_PCT +#undef MLD_TOTAL_ALLOC_44_PK_FROM_SK +#undef MLD_TOTAL_ALLOC_44_SIGN +#undef MLD_TOTAL_ALLOC_44_VERIFY +#undef MLD_TOTAL_ALLOC_65 +#undef MLD_TOTAL_ALLOC_65_KEYPAIR +#undef MLD_TOTAL_ALLOC_65_KEYPAIR_NO_PCT +#undef MLD_TOTAL_ALLOC_65_KEYPAIR_PCT +#undef MLD_TOTAL_ALLOC_65_PK_FROM_SK +#undef MLD_TOTAL_ALLOC_65_SIGN +#undef MLD_TOTAL_ALLOC_65_VERIFY +#undef MLD_TOTAL_ALLOC_87 +#undef MLD_TOTAL_ALLOC_87_KEYPAIR +#undef MLD_TOTAL_ALLOC_87_KEYPAIR_NO_PCT +#undef MLD_TOTAL_ALLOC_87_KEYPAIR_PCT +#undef MLD_TOTAL_ALLOC_87_PK_FROM_SK +#undef MLD_TOTAL_ALLOC_87_SIGN +#undef MLD_TOTAL_ALLOC_87_VERIFY +/* mldsa/src/common.h */ +#undef MLD_ADD_PARAM_SET +#undef MLD_ALLOC +#undef MLD_APPLY +#undef MLD_ASM_FN_SIZE +#undef MLD_ASM_FN_SYMBOL +#undef MLD_ASM_NAMESPACE +#undef MLD_BUILD_INTERNAL +#undef MLD_COMMON_H +#undef MLD_CONCAT +#undef MLD_CONCAT_ +#undef MLD_EMPTY_CU +#undef MLD_ERR_FAIL +#undef MLD_ERR_INVALID_ARG +#undef MLD_ERR_INVALID_KEY +#undef MLD_ERR_INVALID_SIGNATURE +#undef MLD_ERR_OUT_OF_MEMORY +#undef MLD_ERR_PCT_FAIL +#undef MLD_ERR_RNG_FAIL +#undef MLD_ERR_SIGNING_PAUSED +#undef MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED +#undef MLD_EXTERNAL_API +#undef MLD_FIPS202X4_HEADER_FILE +#undef MLD_FIPS202_HEADER_FILE +#undef MLD_FREE +#undef MLD_INTERNAL_API +#undef MLD_INTERNAL_DATA_DECLARATION +#undef MLD_INTERNAL_DATA_DEFINITION +#undef MLD_MULTILEVEL_BUILD +#undef MLD_NAMESPACE +#undef MLD_NAMESPACE_KL +#undef MLD_NAMESPACE_PREFIX +#undef MLD_NAMESPACE_PREFIX_KL +#undef mld_memcpy +#undef mld_memset +/* mldsa/src/packing.h */ +#undef MLD_PACKING_H +#undef mld_pack_sig_c +#undef mld_pack_sig_h +#undef mld_pack_sig_z +#undef mld_pack_sk_rho_key_tr_s2 +#undef mld_pack_sk_s1 +#undef mld_sig_unpack_hints +#undef mld_unpack_pk_t1 +#undef mld_unpack_sk +/* mldsa/src/params.h */ +#undef MLDSA_BETA +#undef MLDSA_CRHBYTES +#undef MLDSA_CRYPTO_BYTES +#undef MLDSA_CRYPTO_PUBLICKEYBYTES +#undef MLDSA_CRYPTO_SECRETKEYBYTES +#undef MLDSA_CTILDEBYTES +#undef MLDSA_D +#undef MLDSA_ETA +#undef MLDSA_GAMMA1 +#undef MLDSA_GAMMA2 +#undef MLDSA_GAMMA2_32 +#undef MLDSA_GAMMA2_88 +#undef MLDSA_K +#undef MLDSA_L +#undef MLDSA_N +#undef MLDSA_OMEGA +#undef MLDSA_PK_END +#undef MLDSA_PK_RHO_BYTES +#undef MLDSA_PK_RHO_OFFSET +#undef MLDSA_PK_T1_BYTES +#undef MLDSA_PK_T1_OFFSET +#undef MLDSA_POLYETA_PACKEDBYTES +#undef MLDSA_POLYT0_PACKEDBYTES +#undef MLDSA_POLYT1_PACKEDBYTES +#undef MLDSA_POLYVECH_PACKEDBYTES +#undef MLDSA_POLYW1_PACKEDBYTES +#undef MLDSA_POLYW1_PACKEDBYTES_32 +#undef MLDSA_POLYW1_PACKEDBYTES_88 +#undef MLDSA_POLYZ_PACKEDBYTES +#undef MLDSA_Q +#undef MLDSA_Q_HALF +#undef MLDSA_RNDBYTES +#undef MLDSA_SEEDBYTES +#undef MLDSA_SIG_C_BYTES +#undef MLDSA_SIG_C_OFFSET +#undef MLDSA_SIG_END +#undef MLDSA_SIG_H_BYTES +#undef MLDSA_SIG_H_OFFSET +#undef MLDSA_SIG_Z_BYTES +#undef MLDSA_SIG_Z_OFFSET +#undef MLDSA_SK_END +#undef MLDSA_SK_KEY_BYTES +#undef MLDSA_SK_KEY_OFFSET +#undef MLDSA_SK_RHO_BYTES +#undef MLDSA_SK_RHO_OFFSET +#undef MLDSA_SK_S1_BYTES +#undef MLDSA_SK_S1_OFFSET +#undef MLDSA_SK_S2_BYTES +#undef MLDSA_SK_S2_OFFSET +#undef MLDSA_SK_T0_BYTES +#undef MLDSA_SK_T0_OFFSET +#undef MLDSA_SK_TR_BYTES +#undef MLDSA_SK_TR_OFFSET +#undef MLDSA_TAU +#undef MLDSA_TRBYTES +#undef MLD_MAX_KAPPA +#undef MLD_PARAMS_H +/* mldsa/src/poly_kl.h */ +#undef MLD_POLYETA_UNPACK_LOWER_BOUND +#undef MLD_POLY_KL_H +#undef mld_poly_challenge +#undef mld_poly_decompose +#undef mld_poly_uniform_eta +#undef mld_poly_uniform_eta_4x +#undef mld_poly_uniform_gamma1 +#undef mld_poly_uniform_gamma1_4x +#undef mld_poly_use_hint +#undef mld_polyeta_pack +#undef mld_polyeta_unpack +#undef mld_polyw1_pack +#undef mld_polyz_pack +#undef mld_polyz_unpack +/* mldsa/src/polyvec.h */ +#undef MLD_POLYVEC_H +#undef mld_polyveck +#undef mld_polyveck_caddq +#undef mld_polyveck_chknorm +#undef mld_polyveck_decompose +#undef mld_polyveck_invntt_tomont +#undef mld_polyveck_ntt +#undef mld_polyveck_pack_eta +#undef mld_polyveck_pack_w1 +#undef mld_polyveck_reduce +#undef mld_polyveck_unpack_eta +#undef mld_polyvecl +#undef mld_polyvecl_chknorm +#undef mld_polyvecl_ntt +#undef mld_polyvecl_pack_eta +#undef mld_polyvecl_pointwise_acc_montgomery +#undef mld_polyvecl_uniform_gamma1 +#undef mld_polyvecl_unpack_eta +#undef mld_polyvecl_unpack_z +/* mldsa/src/polyvec_lazy.h */ +#undef MLD_POLYVEC_LAZY_H +#undef mld_poly_permute_bitrev_to_custom_optional +#undef mld_polymat +#undef mld_polymat_eager +#undef mld_polymat_lazy +#undef mld_polyvec_matrix_expand +#undef mld_polyvec_matrix_expand_eager +#undef mld_polyvec_matrix_expand_lazy +#undef mld_polyvec_matrix_pointwise_montgomery +#undef mld_polyvec_matrix_pointwise_montgomery_row +#undef mld_polyvec_matrix_pointwise_montgomery_row_eager +#undef mld_polyvec_matrix_pointwise_montgomery_row_lazy +#undef mld_polyvec_matrix_pointwise_montgomery_yvec +#undef mld_polyvec_matrix_pointwise_montgomery_yvec_eager +#undef mld_polyvec_matrix_pointwise_montgomery_yvec_lazy +#undef mld_sk_s1hat +#undef mld_sk_s1hat_eager +#undef mld_sk_s1hat_get_poly +#undef mld_sk_s1hat_get_poly_eager +#undef mld_sk_s1hat_get_poly_lazy +#undef mld_sk_s1hat_lazy +#undef mld_sk_s2hat +#undef mld_sk_s2hat_eager +#undef mld_sk_s2hat_get_poly +#undef mld_sk_s2hat_get_poly_eager +#undef mld_sk_s2hat_get_poly_lazy +#undef mld_sk_s2hat_lazy +#undef mld_sk_t0hat +#undef mld_sk_t0hat_eager +#undef mld_sk_t0hat_get_poly +#undef mld_sk_t0hat_get_poly_eager +#undef mld_sk_t0hat_get_poly_lazy +#undef mld_sk_t0hat_lazy +#undef mld_unpack_sk_s1hat +#undef mld_unpack_sk_s1hat_eager +#undef mld_unpack_sk_s1hat_lazy +#undef mld_unpack_sk_s2hat +#undef mld_unpack_sk_s2hat_eager +#undef mld_unpack_sk_s2hat_lazy +#undef mld_unpack_sk_t0hat +#undef mld_unpack_sk_t0hat_eager +#undef mld_unpack_sk_t0hat_lazy +#undef mld_yvec +#undef mld_yvec_eager +#undef mld_yvec_get_poly +#undef mld_yvec_get_poly_eager +#undef mld_yvec_get_poly_lazy +#undef mld_yvec_init +#undef mld_yvec_init_eager +#undef mld_yvec_init_lazy +#undef mld_yvec_lazy +/* mldsa/src/rounding.h */ +#undef MLD_2_POW_D +#undef MLD_ROUNDING_H +#undef mld_decompose +#undef mld_make_hint +#undef mld_power2round +#undef mld_use_hint +/* mldsa/src/sign.h */ +#undef MLD_DOMAIN_SEPARATION_MAX_BYTES +#undef MLD_PREHASH_NONE +#undef MLD_PREHASH_SHA2_224 +#undef MLD_PREHASH_SHA2_256 +#undef MLD_PREHASH_SHA2_384 +#undef MLD_PREHASH_SHA2_512 +#undef MLD_PREHASH_SHA2_512_224 +#undef MLD_PREHASH_SHA2_512_256 +#undef MLD_PREHASH_SHA3_224 +#undef MLD_PREHASH_SHA3_256 +#undef MLD_PREHASH_SHA3_384 +#undef MLD_PREHASH_SHA3_512 +#undef MLD_PREHASH_SHAKE_128 +#undef MLD_PREHASH_SHAKE_256 +#undef MLD_SIGN_H +#undef mld_prepare_domain_separation_prefix +#undef mld_sign_keypair +#undef mld_sign_keypair_internal +#undef mld_sign_pk_from_sk +#undef mld_sign_signature +#undef mld_sign_signature_extmu +#undef mld_sign_signature_internal +#undef mld_sign_signature_pre_hash_internal +#undef mld_sign_signature_pre_hash_shake256 +#undef mld_sign_verify +#undef mld_sign_verify_extmu +#undef mld_sign_verify_internal +#undef mld_sign_verify_pre_hash_internal +#undef mld_sign_verify_pre_hash_shake256 + +#if !defined(MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS) +/* + * Undefine macros from MLD_CONFIG_PARAMETER_SET-generic files + */ +/* mldsa/src/context.h */ +#undef MLD_CONTEXT_H +#undef MLD_CONTEXT_PARAMETERS_0 +#undef MLD_CONTEXT_PARAMETERS_1 +#undef MLD_CONTEXT_PARAMETERS_2 +#undef MLD_CONTEXT_PARAMETERS_3 +#undef MLD_CONTEXT_PARAMETERS_4 +#undef MLD_CONTEXT_PARAMETERS_5 +#undef MLD_CONTEXT_PARAMETERS_6 +#undef MLD_CONTEXT_PARAMETERS_7 +#undef MLD_CONTEXT_PARAMETERS_8 +#undef MLD_CONTEXT_PARAMETERS_9 +#undef MLD_CONTEXT_UNUSED +#undef mld_sign_attempt +#undef mld_sign_finish +#undef mld_sign_resume +/* mldsa/src/ct.h */ +#undef MLD_CT_H +#undef MLD_USE_ASM_VALUE_BARRIER +#undef mld_ct_opt_blocker_u64 +/* mldsa/src/debug.h */ +#undef MLD_DEBUG_H +#undef mld_assert +#undef mld_assert_abs_bound +#undef mld_assert_abs_bound_2d +#undef mld_assert_bound +#undef mld_assert_bound_2d +#undef mld_debug_check_assert +#undef mld_debug_check_bounds +/* mldsa/src/poly.h */ +#undef MLD_FQMUL_BOUND +#undef MLD_INTT_BOUND +#undef MLD_NTT_BOUND +#undef MLD_POLY_H +#undef mld_poly_add +#undef mld_poly_caddq +#undef mld_poly_chknorm +#undef mld_poly_invntt_tomont +#undef mld_poly_ntt +#undef mld_poly_pointwise_montgomery +#undef mld_poly_power2round +#undef mld_poly_reduce +#undef mld_poly_shiftl +#undef mld_poly_sub +#undef mld_poly_uniform +#undef mld_poly_uniform_4x +#undef mld_polyt0_pack +#undef mld_polyt0_unpack +#undef mld_polyt1_pack +#undef mld_polyt1_unpack +#undef mld_polyw1_pack_32 +#undef mld_polyw1_pack_88 +/* mldsa/src/randombytes.h */ +#undef MLD_RANDOMBYTES_H +/* mldsa/src/reduce.h */ +#undef MLD_MONT +#undef MLD_REDUCE32_DOMAIN_MAX +#undef MLD_REDUCE32_RANGE_MAX +#undef MLD_REDUCE_H +/* mldsa/src/symmetric.h */ +#undef MLD_STREAM128_BLOCKBYTES +#undef MLD_STREAM256_BLOCKBYTES +#undef MLD_SYMMETRIC_H +#undef mld_xof128_absorb_once +#undef mld_xof128_ctx +#undef mld_xof128_init +#undef mld_xof128_release +#undef mld_xof128_squeezeblocks +#undef mld_xof128_x4_absorb +#undef mld_xof128_x4_ctx +#undef mld_xof128_x4_init +#undef mld_xof128_x4_release +#undef mld_xof128_x4_squeezeblocks +#undef mld_xof256_absorb_once +#undef mld_xof256_ctx +#undef mld_xof256_init +#undef mld_xof256_release +#undef mld_xof256_squeezeblocks +#undef mld_xof256_x4_absorb +#undef mld_xof256_x4_ctx +#undef mld_xof256_x4_init +#undef mld_xof256_x4_release +#undef mld_xof256_x4_squeezeblocks +/* mldsa/src/sys.h */ +#undef MLD_ALIGN +#undef MLD_ALIGN_UP +#undef MLD_ALWAYS_INLINE +#undef MLD_CET_ENDBR +#undef MLD_CT_TESTING_DECLASSIFY +#undef MLD_CT_TESTING_SECRET +#undef MLD_DEFAULT_ALIGN +#undef MLD_HAVE_INLINE_ASM +#undef MLD_INLINE +#undef MLD_MUST_CHECK_RETURN_VALUE +#undef MLD_NOINLINE +#undef MLD_RESTRICT +#undef MLD_STATIC_TESTABLE +#undef MLD_SYSV_ABI +#undef MLD_SYSV_ABI_SUPPORTED +#undef MLD_SYS_AARCH64 +#undef MLD_SYS_AARCH64_EB +#undef MLD_SYS_AARCH64_NEON +#undef MLD_SYS_APPLE +#undef MLD_SYS_ARMV81M_MVE +#undef MLD_SYS_BIG_ENDIAN +#undef MLD_SYS_H +#undef MLD_SYS_LINUX +#undef MLD_SYS_LITTLE_ENDIAN +#undef MLD_SYS_PPC64LE +#undef MLD_SYS_RISCV32 +#undef MLD_SYS_RISCV64 +#undef MLD_SYS_RISCV64_RVV +#undef MLD_SYS_WINDOWS +#undef MLD_SYS_X86_64 +#undef MLD_SYS_X86_64_AVX2 +/* mldsa/src/cbmc.h */ +#undef MLD_CBMC_H +#undef __contract__ +#undef __loop__ + +#if !defined(MLD_CONFIG_FIPS202_CUSTOM_HEADER) +/* + * Undefine macros from FIPS-202 files + */ +/* mldsa/src/fips202/fips202.h */ +#undef MLD_FIPS202_FIPS202_H +#undef MLD_KECCAK_LANES +#undef SHA3_256_HASHBYTES +#undef SHA3_256_RATE +#undef SHA3_512_HASHBYTES +#undef SHA3_512_RATE +#undef SHAKE128_RATE +#undef SHAKE256_RATE +#undef mld_shake128_absorb +#undef mld_shake128_finalize +#undef mld_shake128_init +#undef mld_shake128_release +#undef mld_shake128_squeeze +#undef mld_shake256 +#undef mld_shake256_absorb +#undef mld_shake256_finalize +#undef mld_shake256_init +#undef mld_shake256_release +#undef mld_shake256_squeeze +/* mldsa/src/fips202/fips202x4.h */ +#undef MLD_FIPS202_FIPS202X4_H +#undef mld_shake128x4_absorb_once +#undef mld_shake128x4_init +#undef mld_shake128x4_release +#undef mld_shake128x4_squeezeblocks +#undef mld_shake256x4_absorb_once +#undef mld_shake256x4_init +#undef mld_shake256x4_release +#undef mld_shake256x4_squeezeblocks +/* mldsa/src/fips202/keccakf1600.h */ +#undef MLD_FIPS202_KECCAKF1600_H +#undef MLD_KECCAK_LANES +#undef MLD_KECCAK_WAY +#undef mld_keccakf1600_extract_bytes +#undef mld_keccakf1600_permute +#undef mld_keccakf1600_xor_bytes +#undef mld_keccakf1600x4_extract_bytes +#undef mld_keccakf1600x4_permute +#undef mld_keccakf1600x4_xor_bytes +#endif /* !MLD_CONFIG_FIPS202_CUSTOM_HEADER */ + +#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202) +/* mldsa/src/fips202/native/api.h */ +#undef MLD_FIPS202_NATIVE_API_H +#undef MLD_NATIVE_FUNC_FALLBACK +#undef MLD_NATIVE_FUNC_SUCCESS +/* mldsa/src/fips202/native/auto.h */ +#undef MLD_FIPS202_NATIVE_AUTO_H +#if defined(MLD_SYS_AARCH64) +/* + * Undefine macros from native code (FIPS202, AArch64) + */ +/* mldsa/src/fips202/native/aarch64/auto.h */ +#undef MLD_FIPS202_NATIVE_AARCH64_AUTO_H +/* mldsa/src/fips202/native/aarch64/src/fips202_native_aarch64.h */ +#undef MLD_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H +#undef mld_keccak_f1600_x1_scalar_aarch64_asm +#undef mld_keccak_f1600_x1_v84a_aarch64_asm +#undef mld_keccak_f1600_x2_v84a_aarch64_asm +#undef mld_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm +#undef mld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm +#undef mld_keccakf1600_round_constants +/* mldsa/src/fips202/native/aarch64/x1_scalar.h */ +#undef MLD_FIPS202_AARCH64_NEED_X1_SCALAR +#undef MLD_FIPS202_NATIVE_AARCH64_X1_SCALAR_H +#undef MLD_USE_NATIVE_FIPS202_X1 +/* mldsa/src/fips202/native/aarch64/x1_v84a.h */ +#undef MLD_FIPS202_AARCH64_NEED_X1_V84A +#undef MLD_FIPS202_NATIVE_AARCH64_X1_V84A_H +#undef MLD_USE_NATIVE_FIPS202_X1 +/* mldsa/src/fips202/native/aarch64/x2_v84a.h */ +#undef MLD_FIPS202_AARCH64_NEED_X2_V84A +#undef MLD_FIPS202_NATIVE_AARCH64_X2_V84A_H +#undef MLD_USE_NATIVE_FIPS202_X4 +/* mldsa/src/fips202/native/aarch64/x4_v8a_scalar.h */ +#undef MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID +#undef MLD_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H +#undef MLD_USE_NATIVE_FIPS202_X4 +/* mldsa/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h */ +#undef MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID +#undef MLD_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H +#undef MLD_USE_NATIVE_FIPS202_X4 +#endif /* MLD_SYS_AARCH64 */ +#if defined(MLD_SYS_X86_64) +/* + * Undefine macros from native code (FIPS202, x86_64) + */ +/* mldsa/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h */ +#undef MLD_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H +#undef MLD_FIPS202_X86_64_NEED_X4_AVX2 +#undef MLD_USE_NATIVE_FIPS202_X4 +/* mldsa/src/fips202/native/x86_64/src/fips202_native_x86_64.h */ +#undef MLD_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H +#undef mld_keccak_f1600_x4_avx2_asm +#undef mld_keccak_rho56 +#undef mld_keccak_rho8 +#undef mld_keccakf1600_round_constants +#endif /* MLD_SYS_X86_64 */ +#if defined(MLD_SYS_ARMV81M_MVE) +/* + * Undefine macros from native code (FIPS202, Armv8.1-M) + */ +/* mldsa/src/fips202/native/armv81m/mve.h */ +#undef MLD_FIPS202_ARMV81M_NEED_X4 +#undef MLD_FIPS202_NATIVE_ARMV81M +#undef MLD_FIPS202_NATIVE_ARMV81M_MVE_H +#undef MLD_USE_NATIVE_FIPS202_X4 +#undef MLD_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES +#undef MLD_USE_NATIVE_FIPS202_X4_XOR_BYTES +#undef mld_keccak_f1600_x4_native_impl +/* mldsa/src/fips202/native/armv81m/src/fips202_native_armv81m.h */ +#undef MLD_FIPS202_NATIVE_ARMV81M_SRC_FIPS202_NATIVE_ARMV81M_H +#undef mld_keccak_f1600_x4_mve_asm +#undef mld_keccak_f1600_x4_state_extract_bytes_asm +#undef mld_keccak_f1600_x4_state_xor_bytes_asm +#undef mld_keccakf1600_round_constants +#endif /* MLD_SYS_ARMV81M_MVE */ +#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 */ +#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH) +/* mldsa/src/native/api.h */ +#undef MLD_FQMUL_BOUND +#undef MLD_INTT_BOUND +#undef MLD_NATIVE_API_H +#undef MLD_NATIVE_FUNC_FALLBACK +#undef MLD_NATIVE_FUNC_SUCCESS +#undef MLD_NTT_BOUND +#undef MLD_REDUCE32_RANGE_MAX +/* mldsa/src/native/meta.h */ +#undef MLD_NATIVE_META_H +#if defined(MLD_SYS_AARCH64) +/* + * Undefine macros from native code (Arith, AArch64) + */ +/* mldsa/src/native/aarch64/meta.h */ +#undef MLD_ARITH_BACKEND_AARCH64 +#undef MLD_NATIVE_AARCH64_META_H +#undef MLD_USE_NATIVE_INTT +#undef MLD_USE_NATIVE_NTT +#undef MLD_USE_NATIVE_POINTWISE_MONTGOMERY +#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4 +#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5 +#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7 +#undef MLD_USE_NATIVE_POLYZ_UNPACK_17 +#undef MLD_USE_NATIVE_POLYZ_UNPACK_19 +#undef MLD_USE_NATIVE_POLY_CADDQ +#undef MLD_USE_NATIVE_POLY_CHKNORM +#undef MLD_USE_NATIVE_POLY_DECOMPOSE_32 +#undef MLD_USE_NATIVE_POLY_DECOMPOSE_88 +#undef MLD_USE_NATIVE_POLY_USE_HINT_32 +#undef MLD_USE_NATIVE_POLY_USE_HINT_88 +#undef MLD_USE_NATIVE_REJ_UNIFORM +#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA2 +#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA4 +/* mldsa/src/native/aarch64/src/arith_native_aarch64.h */ +#undef MLD_AARCH64_REJ_UNIFORM_ETA2_BUFLEN +#undef MLD_AARCH64_REJ_UNIFORM_ETA4_BUFLEN +#undef MLD_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H +#undef mld_aarch64_intt_zetas_layer123456 +#undef mld_aarch64_intt_zetas_layer78 +#undef mld_aarch64_ntt_zetas_layer123456 +#undef mld_aarch64_ntt_zetas_layer78 +#undef mld_intt_aarch64_asm +#undef mld_ntt_aarch64_asm +#undef mld_poly_caddq_aarch64_asm +#undef mld_poly_chknorm_aarch64_asm +#undef mld_poly_decompose_32_aarch64_asm +#undef mld_poly_decompose_88_aarch64_asm +#undef mld_poly_pointwise_montgomery_aarch64_asm +#undef mld_poly_use_hint_32_aarch64_asm +#undef mld_poly_use_hint_88_aarch64_asm +#undef mld_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm +#undef mld_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm +#undef mld_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm +#undef mld_polyz_unpack_17_aarch64_asm +#undef mld_polyz_unpack_17_indices +#undef mld_polyz_unpack_19_aarch64_asm +#undef mld_polyz_unpack_19_indices +#undef mld_rej_uniform_aarch64_asm +#undef mld_rej_uniform_eta2_aarch64_asm +#undef mld_rej_uniform_eta4_aarch64_asm +#undef mld_rej_uniform_eta_table +#undef mld_rej_uniform_table +#endif /* MLD_SYS_AARCH64 */ +#if defined(MLD_SYS_X86_64) +/* + * Undefine macros from native code (Arith, X86_64) + */ +/* mldsa/src/native/x86_64/meta.h */ +#undef MLD_ARITH_BACKEND_X86_64_DEFAULT +#undef MLD_NATIVE_X86_64_META_H +#undef MLD_USE_NATIVE_INTT +#undef MLD_USE_NATIVE_NTT +#undef MLD_USE_NATIVE_NTT_CUSTOM_ORDER +#undef MLD_USE_NATIVE_POINTWISE_MONTGOMERY +#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4 +#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5 +#undef MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7 +#undef MLD_USE_NATIVE_POLYZ_UNPACK_17 +#undef MLD_USE_NATIVE_POLYZ_UNPACK_19 +#undef MLD_USE_NATIVE_POLY_CADDQ +#undef MLD_USE_NATIVE_POLY_CHKNORM +#undef MLD_USE_NATIVE_POLY_DECOMPOSE_32 +#undef MLD_USE_NATIVE_POLY_DECOMPOSE_88 +#undef MLD_USE_NATIVE_POLY_USE_HINT_32 +#undef MLD_USE_NATIVE_POLY_USE_HINT_88 +#undef MLD_USE_NATIVE_REJ_UNIFORM +#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA2 +#undef MLD_USE_NATIVE_REJ_UNIFORM_ETA4 +/* mldsa/src/native/x86_64/src/arith_native_x86_64.h */ +#undef MLD_AVX2_REJ_UNIFORM_BUFLEN +#undef MLD_AVX2_REJ_UNIFORM_ETA2_BUFLEN +#undef MLD_AVX2_REJ_UNIFORM_ETA4_BUFLEN +#undef MLD_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H +#undef mld_invntt_avx2_asm +#undef mld_ntt_avx2_asm +#undef mld_nttunpack_avx2_asm +#undef mld_pointwise_acc_l4_avx2_asm +#undef mld_pointwise_acc_l5_avx2_asm +#undef mld_pointwise_acc_l7_avx2_asm +#undef mld_pointwise_avx2_asm +#undef mld_poly_caddq_avx2_asm +#undef mld_poly_chknorm_avx2_asm +#undef mld_poly_decompose_32_avx2_asm +#undef mld_poly_decompose_88_avx2_asm +#undef mld_poly_use_hint_32_avx2_asm +#undef mld_poly_use_hint_88_avx2_asm +#undef mld_polyz_unpack_17_avx2_asm +#undef mld_polyz_unpack_19_avx2_asm +#undef mld_rej_uniform_avx2_asm +#undef mld_rej_uniform_eta2_avx2_asm +#undef mld_rej_uniform_eta4_avx2_asm +#undef mld_rej_uniform_table +/* mldsa/src/native/x86_64/src/consts.h */ +#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XDIV +#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XDIV_QINV +#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XQ +#undef MLD_AVX2_BACKEND_DATA_OFFSET_8XQINV +#undef MLD_AVX2_BACKEND_DATA_OFFSET_ZETAS +#undef MLD_AVX2_BACKEND_DATA_OFFSET_ZETAS_QINV +#undef MLD_NATIVE_X86_64_SRC_CONSTS_H +#undef mld_qdata +#endif /* MLD_SYS_X86_64 */ +#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_ARITH */ +#endif /* !MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS */ diff --git a/cbits/mldsa/mldsa_native_config.h b/cbits/mldsa/mldsa_native_config.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/mldsa_native_config.h @@ -0,0 +1,855 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS140_3_IG] + * Implementation Guidance for FIPS 140-3 and the Cryptographic Module + * Validation Program + * National Institute of Standards and Technology + * https://csrc.nist.gov/projects/cryptographic-module-validation-program/fips-140-3-ig-announcements + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + * + * - [FIPS204_UPDATES] + * FIPS 204 Potential Updates (Errata) + * National Institute of Standards and Technology + * https://csrc.nist.gov/files/pubs/fips/204/final/docs/fips-204-potential-updates.xlsx + */ + +#ifndef MLD_CONFIG_H +#define MLD_CONFIG_H + +/** + * MLD_CONFIG_PARAMETER_SET + * + * Specifies the parameter set for ML-DSA + * - MLD_CONFIG_PARAMETER_SET=44 corresponds to ML-DSA-44 + * - MLD_CONFIG_PARAMETER_SET=65 corresponds to ML-DSA-65 + * - MLD_CONFIG_PARAMETER_SET=87 corresponds to ML-DSA-87 + * + * If you want to support multiple parameter sets, build the + * library multiple times and set MLD_CONFIG_MULTILEVEL_BUILD. + * See MLD_CONFIG_MULTILEVEL_BUILD for how to do this while + * minimizing code duplication. + * + * This can also be set using CFLAGS. + */ +#ifndef MLD_CONFIG_PARAMETER_SET +#define MLD_CONFIG_PARAMETER_SET \ + 44 /* Change this for different security strengths */ +#endif + +/** + * MLD_CONFIG_FILE + * + * If defined, this is a header that will be included instead + * of the default configuration file mldsa/mldsa_native_config.h. + * + * When you need to build mldsa-native in multiple configurations, + * using varying MLD_CONFIG_FILE can be more convenient + * than configuring everything through CFLAGS. + * + * To use, MLD_CONFIG_FILE _must_ be defined prior + * to the inclusion of any mldsa-native headers. For example, + * it can be set by passing `-DMLD_CONFIG_FILE="..."` + * on the command line. + */ +/* #define MLD_CONFIG_FILE "mldsa_native_config.h" */ + +/** + * MLD_CONFIG_NAMESPACE_PREFIX + * + * The prefix to use to namespace global symbols from mldsa/. + * + * In a multi-level build, level-dependent symbols will + * additionally be prefixed with the parameter set (44/65/87). + * + * This can also be set using CFLAGS. + */ +#if !defined(MLD_CONFIG_NAMESPACE_PREFIX) +#define MLD_CONFIG_NAMESPACE_PREFIX MLD_DEFAULT_NAMESPACE_PREFIX +#endif + +/** + * MLD_CONFIG_MULTILEVEL_BUILD + * + * Set this if the build is part of a multi-level build supporting + * multiple parameter sets. + * + * If you need only a single parameter set, keep this unset. + * + * To build mldsa-native with support for all parameter sets, + * build it three times -- once per parameter set -- and set the + * option MLD_CONFIG_MULTILEVEL_WITH_SHARED for exactly one of + * them, and MLD_CONFIG_MULTILEVEL_NO_SHARED for the others. + * MLD_CONFIG_MULTILEVEL_BUILD should be set for all of them. + * + * See examples/multilevel_build for an example. + * + * This can also be set using CFLAGS. + */ +/* #define MLD_CONFIG_MULTILEVEL_BUILD */ + +/** + * MLD_CONFIG_EXTERNAL_API_QUALIFIER + * + * If set, this option provides an additional function + * qualifier to be added to declarations of mldsa-native's + * public API. + * + * The primary use case for this option are single-CU builds + * where the public API exposed by mldsa-native is wrapped by + * another API in the consuming application. In this case, + * even mldsa-native's public API can be marked `static`. + */ +/* #define MLD_CONFIG_EXTERNAL_API_QUALIFIER */ + +/** + * MLD_CONFIG_NO_KEYPAIR_API + * + * By default, mldsa-native includes support for generating key + * pairs. If you don't need this, set MLD_CONFIG_NO_KEYPAIR_API + * to exclude keypair, keypair_internal, + * pk_from_sk, and all internal APIs only needed by + * those functions. + */ +/* #define MLD_CONFIG_NO_KEYPAIR_API */ + +/** + * MLD_CONFIG_NO_SIGN_API + * + * By default, mldsa-native includes support for creating + * signatures. If you don't need this, set MLD_CONFIG_NO_SIGN_API + * to exclude signature, + * signature_extmu, signature_internal, + * signature_pre_hash_internal, + * signature_pre_hash_shake256, and all internal APIs + * only needed by those functions. + */ +/* #define MLD_CONFIG_NO_SIGN_API */ + +/** + * MLD_CONFIG_NO_VERIFY_API + * + * By default, mldsa-native includes support for verifying + * signatures. If you don't need this, set + * MLD_CONFIG_NO_VERIFY_API to exclude verify, + * verify_extmu, verify_internal, + * verify_pre_hash_internal, + * verify_pre_hash_shake256, and all internal APIs + * only needed by those functions. + */ +/* #define MLD_CONFIG_NO_VERIFY_API */ + +/** + * MLD_CONFIG_CORE_API_ONLY + * + * Set this to remove all public APIs except + * keypair_internal, signature_internal, + * and verify_internal. + */ +/* #define MLD_CONFIG_CORE_API_ONLY */ + +/** + * MLD_CONFIG_NO_RANDOMIZED_API + * + * If this option is set, mldsa-native will be built without the + * randomized API functions (keypair, + * signature, and signature_extmu). + * This allows users to build mldsa-native without providing a + * randombytes() implementation if they only need the + * internal deterministic API + * (keypair_internal, signature_internal). + * + * @note This option is incompatible with MLD_CONFIG_KEYGEN_PCT + * as the current PCT implementation requires + * signature(). + */ +/* #define MLD_CONFIG_NO_RANDOMIZED_API */ + +/** + * MLD_CONFIG_CONSTANTS_ONLY + * + * If you only need the size constants (MLDSA_PUBLICKEYBYTES, etc.) + * but no function declarations, set MLD_CONFIG_CONSTANTS_ONLY. + * + * This only affects the public header mldsa_native.h, not + * the implementation. + */ +/* #define MLD_CONFIG_CONSTANTS_ONLY */ +/****************************************************************************** + * + * Build-only configuration options + * + * The remaining configurations are build-options only. + * They do not affect the API described in mldsa_native.h. + * + *****************************************************************************/ +#if defined(MLD_BUILD_INTERNAL) + +/** + * MLD_CONFIG_MULTILEVEL_WITH_SHARED + * + * This is for multi-level builds of mldsa-native only. If you + * need only a single parameter set, keep this unset. + * + * If this is set, all MLD_CONFIG_PARAMETER_SET-independent + * code will be included in the build, including code needed only + * for other parameter sets. + * + * Example: mld_polyw1_pack_88 is only needed for + * MLD_CONFIG_PARAMETER_SET == 44. Yet, if this option is set for a + * build with MLD_CONFIG_PARAMETER_SET == 65/87, it would be included. + * + * To build mldsa-native with support for all parameter sets, + * build it three times -- once per parameter set -- and set the + * option MLD_CONFIG_MULTILEVEL_WITH_SHARED for exactly one of + * them, and MLD_CONFIG_MULTILEVEL_NO_SHARED for the others. + * MLD_CONFIG_MULTILEVEL_BUILD should be set for all of them. + * + * See examples/multilevel_build for an example. + * + * This can also be set using CFLAGS. + */ +/* #define MLD_CONFIG_MULTILEVEL_WITH_SHARED */ + +/** + * MLD_CONFIG_MULTILEVEL_NO_SHARED + * + * This is for multi-level builds of mldsa-native only. If you + * need only a single parameter set, keep this unset. + * + * If this is set, no MLD_CONFIG_PARAMETER_SET-independent code + * will be included in the build. + * + * To build mldsa-native with support for all parameter sets, + * build it three times -- once per parameter set -- and set the + * option MLD_CONFIG_MULTILEVEL_WITH_SHARED for exactly one of + * them, and MLD_CONFIG_MULTILEVEL_NO_SHARED for the others. + * MLD_CONFIG_MULTILEVEL_BUILD should be set for all of them. + * + * See examples/multilevel_build for an example. + * + * This can also be set using CFLAGS. + */ +/* #define MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +/** + * MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS + * + * This is only relevant for single compilation unit (SCU) + * builds of mldsa-native. In this case, it determines whether + * directives defined in parameter-set-independent headers should + * be #undef'ined or not at the end of the SCU file. This is + * needed in multilevel builds. + * + * See examples/multilevel_build_native for an example. + * + * This can also be set using CFLAGS. + */ +/* #define MLD_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS */ + +/** + * MLD_CONFIG_USE_NATIVE_BACKEND_ARITH + * + * Determines whether a native arithmetic backend should be used. + * + * The arithmetic backend covers performance-critical functions + * such as the number-theoretic transform (NTT). + * + * If this option is unset, the C backend will be used. + * + * If this option is set, the arithmetic backend to be used is + * determined by MLD_CONFIG_ARITH_BACKEND_FILE: If the latter is + * unset, the default backend for your target architecture + * will be used. If set, it must be the name of a backend metadata + * file. + * + * This can also be set using CFLAGS. + */ +#if !defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH) +/* #define MLD_CONFIG_USE_NATIVE_BACKEND_ARITH */ +#endif + +/** + * MLD_CONFIG_ARITH_BACKEND_FILE + * + * The arithmetic backend to use. + * + * If MLD_CONFIG_USE_NATIVE_BACKEND_ARITH is unset, this option + * is ignored. + * + * If MLD_CONFIG_USE_NATIVE_BACKEND_ARITH is set, this option must + * either be undefined or the filename of an arithmetic backend. + * If unset, the default backend will be used. + * + * This can be set using CFLAGS. + */ +#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH) && \ + !defined(MLD_CONFIG_ARITH_BACKEND_FILE) +#define MLD_CONFIG_ARITH_BACKEND_FILE "native/meta.h" +#endif + +/** + * MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 + * + * Determines whether a native FIPS202 backend should be used. + * + * The FIPS202 backend covers 1x/2x/4x-fold Keccak-f1600, which is + * the performance bottleneck of SHA3 and SHAKE. + * + * If this option is unset, the C backend will be used. + * + * If this option is set, the FIPS202 backend to be used is + * determined by MLD_CONFIG_FIPS202_BACKEND_FILE: If the latter is + * unset, the default backend for your target architecture + * will be used. If set, it must be the name of a backend metadata + * file. + * + * This can also be set using CFLAGS. + */ +#if !defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202) +/* #define MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 */ +#endif + +/** + * MLD_CONFIG_FIPS202_BACKEND_FILE + * + * The FIPS-202 backend to use. + * + * If MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 is set, this option + * must either be undefined or the filename of a FIPS202 backend. + * If unset, the default backend will be used. + * + * This can be set using CFLAGS. + */ +#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202) && \ + !defined(MLD_CONFIG_FIPS202_BACKEND_FILE) +#define MLD_CONFIG_FIPS202_BACKEND_FILE "fips202/native/auto.h" +#endif + +/** + * MLD_CONFIG_FIPS202_CUSTOM_HEADER + * + * Custom header to use for FIPS-202 + * + * This should only be set if you intend to use a custom + * FIPS-202 implementation, different from the one shipped + * with mldsa-native. + * + * If set, it must be the name of a file serving as the + * replacement for mldsa/src/fips202/fips202.h, and exposing + * the same API (see FIPS202.md). + */ +/* #define MLD_CONFIG_FIPS202_CUSTOM_HEADER "SOME_FILE.h" */ + +/** + * MLD_CONFIG_FIPS202X4_CUSTOM_HEADER + * + * Custom header to use for FIPS-202-X4 + * + * This should only be set if you intend to use a custom + * FIPS-202 implementation, different from the one shipped + * with mldsa-native. + * + * If set, it must be the name of a file serving as the + * replacement for mldsa/src/fips202/fips202x4.h, and exposing + * the same API (see FIPS202.md). + */ +/* #define MLD_CONFIG_FIPS202X4_CUSTOM_HEADER "SOME_FILE.h" */ + +/** + * MLD_CONFIG_CUSTOM_ZEROIZE + * + * In compliance with @[FIPS204, Section 3.6.3], mldsa-native zeroizes + * intermediate buffers before returning from function calls. By default, + * those buffers are allocated from the stack; if MLD_CONFIG_CUSTOM_ALLOC_FREE + * is set, they are (mostly -- few exceptions remain at present) allocated from + * the configured custom allocator. + * + * mldsa-native also zeroizes caller-owned output buffers as needed to uphold + * the API convention that outputs be either unmodified or zeroized upon + * failure. + * + * Set this option and define `mld_zeroize` if you want to use a custom + * method to zeroize intermediate and output buffers. + * + * The default implementation uses SecureZeroMemory on Windows and a + * memset + compiler barrier otherwise. If neither of those is available on + * the target platform, compilation will fail, and you will need to use + * MLD_CONFIG_CUSTOM_ZEROIZE to provide a custom implementation of + * `mld_zeroize()`. + * + * @warning + * The zeroization conducted by mldsa-native reduces the likelihood of data + * leaking on the stack or custom allocators, but it does not eliminate it. + * For example, the C standard makes no guarantee about where a compiler + * allocates local structures and whether/where it makes copies of them. + * Also, in addition to entire structures, there may also be potentially + * exploitable leakage of individual values on the stack. If you need + * bullet-proof zeroization of the stack, you need to consider additional + * measures instead of what this feature provides. In this case, you can + * set mld_zeroize to a no-op. Note that in this case you are also responsible + * for zeroizing output buffers upon failure. + */ +/* #define MLD_CONFIG_CUSTOM_ZEROIZE + #if !defined(__ASSEMBLER__) + #include + #include "src/src.h" + static MLD_INLINE void mld_zeroize(void *ptr, size_t len) + { + ... your implementation ... + } + #endif +*/ + +/** + * MLD_CONFIG_CUSTOM_RANDOMBYTES + * + * mldsa-native does not provide a secure randombytes + * implementation. Such an implementation has to be provided by + * the consumer. + * + * If this option is not set, mldsa-native expects a function + * int randombytes(uint8_t *out, size_t outlen). + * + * Set this option and define `mld_randombytes` if you want to + * use a custom method to sample randombytes with a different name + * or signature. + */ +/* #define MLD_CONFIG_CUSTOM_RANDOMBYTES + #if !defined(__ASSEMBLER__) + #include + #include "src/src.h" + static MLD_INLINE int mld_randombytes(uint8_t *ptr, size_t len) + { + ... your implementation ... + return 0; + } + #endif +*/ + +/** + * MLD_CONFIG_CUSTOM_CAPABILITY_FUNC + * + * mldsa-native backends may rely on specific hardware features. + * Those backends will only be included in an mldsa-native build + * if support for the respective features is enabled at + * compile-time. However, when building for a heterogeneous set + * of CPUs to run the resulting binary/library on, feature + * detection at _runtime_ is needed to decide whether a backend + * can be used or not. + * + * Set this option and define `mld_sys_check_capability` if you + * want to use a custom method to dispatch between implementations. + * + * Return value 1 indicates that a capability is supported. + * Return value 0 indicates that a capability is not supported. + * + * If this option is not set, mldsa-native uses compile-time + * feature detection only to decide which backend to use. + * + * If you compile mldsa-native on a system with different + * capabilities than the system that the resulting binary/library + * will be run on, you must use this option. + */ +/* #define MLD_CONFIG_CUSTOM_CAPABILITY_FUNC + static MLD_INLINE int mld_sys_check_capability(mld_sys_cap cap) + { + ... your implementation ... + } +*/ + +/** + * MLD_CONFIG_CUSTOM_ALLOC_FREE + * + * Set this option and define `MLD_CUSTOM_ALLOC` and + * `MLD_CUSTOM_FREE` if you want to use custom allocation for + * large local structures or buffers. + * + * By default, all buffers/structures are allocated on the stack. + * If this option is set, most of them will be allocated via + * MLD_CUSTOM_ALLOC. + * + * Parameters to MLD_CUSTOM_ALLOC: + * - T* v: Target pointer to declare. + * - T: Type of structure to be allocated + * - N: Number of elements to be allocated. + * + * Parameters to MLD_CUSTOM_FREE: + * - T* v: Target pointer to free. May be NULL. + * - T: Type of structure to be freed. + * - N: Number of elements to be freed. + * + * @warning This option is experimental. Its scope, configuration and + * function/macro signatures may change at any time. We expect a + * stable API in a future version. + * + * @note Even if this option is set, some allocations further down + * the call stack will still be made from the stack. Those will + * likely be added to the scope of this option in the future. + * + * @note MLD_CUSTOM_ALLOC need not guarantee a successful + * allocation nor include error handling. Upon failure, the + * target pointer should simply be set to NULL. The calling + * code will handle this case and invoke MLD_CUSTOM_FREE. + */ +/* #define MLD_CONFIG_CUSTOM_ALLOC_FREE + #if !defined(__ASSEMBLER__) + #include + #define MLD_CUSTOM_ALLOC(v, T, N) \ + T* (v) = (T *)aligned_alloc(MLD_DEFAULT_ALIGN, \ + MLD_ALIGN_UP(sizeof(T) * (N))) + #define MLD_CUSTOM_FREE(v, T, N) free(v) + #endif +*/ + +/** + * MLD_CONFIG_CUSTOM_MEMCPY + * + * Set this option and define `mld_memcpy` if you want to + * use a custom method to copy memory instead of the standard + * library memcpy function. + * + * The custom implementation must have the same signature and + * behavior as the standard memcpy function: + * void *mld_memcpy(void *dest, const void *src, size_t n) + */ +/* #define MLD_CONFIG_CUSTOM_MEMCPY + #if !defined(__ASSEMBLER__) + #include + #include "src/src.h" + static MLD_INLINE void *mld_memcpy(void *dest, const void *src, size_t n) + { + ... your implementation ... + } + #endif +*/ + +/** + * MLD_CONFIG_CUSTOM_MEMSET + * + * Set this option and define `mld_memset` if you want to + * use a custom method to set memory instead of the standard + * library memset function. + * + * The custom implementation must have the same signature and + * behavior as the standard memset function: + * void *mld_memset(void *s, int c, size_t n) + */ +/* #define MLD_CONFIG_CUSTOM_MEMSET + #if !defined(__ASSEMBLER__) + #include + #include "src/src.h" + static MLD_INLINE void *mld_memset(void *s, int c, size_t n) + { + ... your implementation ... + } + #endif +*/ + +/** + * MLD_CONFIG_INTERNAL_API_QUALIFIER + * + * If set, this option provides an additional qualifier + * to be added to declarations of internal API functions and data. + * + * The primary use case for this option are single-CU builds, + * in which case this option can be set to `static`. + */ +/* #define MLD_CONFIG_INTERNAL_API_QUALIFIER */ + +/** + * MLD_CONFIG_CT_TESTING_ENABLED + * + * If set, mldsa-native annotates data as secret / public using + * valgrind's annotations VALGRIND_MAKE_MEM_UNDEFINED and + * VALGRIND_MAKE_MEM_DEFINED, enabling various checks for secret- + * dependent control flow of variable time execution (depending + * on the exact version of valgrind installed). + */ +/* #define MLD_CONFIG_CT_TESTING_ENABLED */ + +/** + * MLD_CONFIG_NO_ASM + * + * If this option is set, mldsa-native will be built without + * use of native code or inline assembly. + * + * By default, inline assembly is used to implement value barriers. + * Without inline assembly, mldsa-native will use a global volatile + * 'opt blocker' instead; see ct.h. + * + * Inline assembly is also used to implement a secure zeroization + * function on non-Windows platforms. If this option is set and + * the target platform is not Windows, you MUST set + * MLD_CONFIG_CUSTOM_ZEROIZE and provide a custom zeroization + * function. + * + * If this option is set, MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 and + * MLD_CONFIG_USE_NATIVE_BACKEND_ARITH will be ignored, and no + * native backends will be used. + */ +/* #define MLD_CONFIG_NO_ASM */ + +/** + * MLD_CONFIG_NO_ASM_VALUE_BARRIER + * + * If this option is set, mldsa-native will be built without + * use of native code or inline assembly for value barriers. + * + * By default, inline assembly (if available) is used to implement + * value barriers. + * Without inline assembly, mldsa-native will use a global volatile + * 'opt blocker' instead; see ct.h. + */ +/* #define MLD_CONFIG_NO_ASM_VALUE_BARRIER */ + +/** + * MLD_CONFIG_KEYGEN_PCT + * + * Compliance with @[FIPS140_3_IG, p.87] requires a + * Pairwise Consistency Test (PCT) to be carried out on a freshly + * generated keypair before it can be exported. + * + * Set this option if such a check should be implemented. + * In this case, keypair_internal and + * keypair will return MLD_ERR_PCT_FAIL if the + * PCT failed. + * + * @note This feature will drastically lower the performance of + * key generation. + * + * @note This option is incompatible with MLD_CONFIG_NO_SIGN_API + * and MLD_CONFIG_NO_VERIFY_API as the current PCT implementation + * requires signature() and verify(). + */ +/* #define MLD_CONFIG_KEYGEN_PCT */ + +/** + * MLD_CONFIG_KEYGEN_PCT_BREAKAGE_TEST + * + * If this option is set, the user must provide a runtime + * function `static inline int mld_break_pct() { ... }` to + * indicate whether the PCT should be made fail. + * + * This option only has an effect if MLD_CONFIG_KEYGEN_PCT is set. + */ +/* #define MLD_CONFIG_KEYGEN_PCT_BREAKAGE_TEST + #if !defined(__ASSEMBLER__) + #include "src/src.h" + static MLD_INLINE int mld_break_pct(void) + { + ... return 0/1 depending on whether PCT should be broken ... + } + #endif +*/ + +/** + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS + * + * Upper bound on the number of rejection-sampling iterations + * performed by ML-DSA signing (@[FIPS204, Algorithm 7]). + * + * If a valid signature is not produced within this many + * attempts, signing returns MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED. + * This is useful in timing-sensitive environments that + * require a deterministic worst-case bound on signing time. + * + * For FIPS 204 compliance, this value MUST be at least 821, + * cf. @[FIPS204, Appendix C] and @[FIPS204_UPDATES], which is + * chosen so that the signing failure rate is < 2^{-256}. + * + * Default: Largest possible value before internal counters + * would overflow. This is larger than the FIPS204 bound. + * + * In particular, in the default configuration, the signing + * failure rate is < 2^{-256}. + */ +/* #define MLD_CONFIG_MAX_SIGNING_ATTEMPTS 821 */ + +/** + * MLD_CONFIG_SERIAL_FIPS202_ONLY + * + * Set this to use a FIPS202 implementation with global state + * that supports only one active Keccak computation at a time + * (e.g. some hardware accelerators). + * + * If this option is set, ML-DSA will use FIPS202 operations + * serially, ensuring that only one SHAKE context is active + * at any given time. + * + * This allows offloading Keccak computations to a hardware + * accelerator that holds only a single Keccak state locally, + * rather than requiring support for multiple concurrent + * Keccak states. + * + * @note Depending on the target CPU, this may reduce + * performance when using software FIPS202 implementations. + * Only enable this when you have to. + */ +/* #define MLD_CONFIG_SERIAL_FIPS202_ONLY */ + +/** + * MLD_CONFIG_CONTEXT_PARAMETER + * + * Set this to add a caller-supplied context parameter to the public API + * functions, which is then forwarded unchanged to the custom callbacks + * (allocation, and signing hooks below). + * + * When this option is set, every public API function gains a trailing + * parameter + * + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE context + * + * as its last argument; its type is configured via + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE (see below). mldsa-native treats this + * value as opaque: it never dereferences it and only passes it on to the + * configurable hook macros. It is meant to carry per-caller state -- e.g. a + * pointer to a memory pool for the allocation hooks, or the resume state for + * the signing hooks -- into those hooks. + * + * When this option is unset (the default), no extra parameter is added and + * the hook macros never receive a context argument. + * + * The hooks that receive the context are the allocation hooks (see + * MLD_CONFIG_CUSTOM_ALLOC_FREE) and the signing hooks (see + * MLD_CONFIG_SIGN_HOOK_RESUME / _ATTEMPT / _FINISH); each is documented with + * its own option below. + */ +/* #define MLD_CONFIG_CONTEXT_PARAMETER */ + +/** + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE + * + * Set this to define the type of the context parameter added by + * MLD_CONFIG_CONTEXT_PARAMETER. It can be any C type usable as a function + * parameter, e.g. `void *` or a pointer to a caller-defined struct such as + * `struct my_ctx *`. + * + * This option must be defined if and only if MLD_CONFIG_CONTEXT_PARAMETER is + * defined; defining one without the other is a compile-time error. + */ +/* #define MLD_CONFIG_CONTEXT_PARAMETER_TYPE void* */ + +/** + * Signing hooks: MLD_CONFIG_SIGN_HOOK_RESUME / _ATTEMPT / _FINISH + * + * Three optional, independent hooks into the ML-DSA signing rejection-sampling + * loop. Each is enabled by defining the matching option, in which case the + * integration must provide the corresponding function. If a hook needs + * per-operation state, enable MLD_CONFIG_CONTEXT_PARAMETER; the context is then + * appended as the last argument. + * + * @warning This feature is experimental. Its scope, configuration and + * function signatures may change at any time, including after v2. + * + * Enabling any of the hooks requires MLD_CONFIG_NO_RANDOMIZED_API (restricting + * the public API to deterministic operations). This is because the restartable + * signing as enabled by the signing hooks only produces the uninterrupted + * signature when the randomness is fixed across calls. A logging-only use + * (attempt always returns 0; resume/finish merely observe) would be safe with + * the randomized API too, but for now the requirement is imposed uniformly on + * all three hooks. + * + * Note: Randomized signing is a shim wrapper around deterministic signing, and + * all helper functions you need to build it are exposed publicly. Thus, if you + * need a restartable, randomized signing operation, you can build your own by + * replicating the logic and adding the RNG seed to the restart context. In this + * case, please also consider letting the mldsa-native maintainers know of your + * need for randomized, restartable signing, so the feature can be appropriately + * prioritized. + * + * - MLD_CONFIG_SIGN_HOOK_ATTEMPT: int mld_sign_hook_attempt(attempt[, ctxt]) + * Called before each attempt. Returns 0 to proceed, or non-zero to pause: + * signing then returns MLD_ERR_SIGNING_PAUSED with `attempt` as the resume + * point (needs MLD_CONFIG_SIGN_HOOK_RESUME to resume; otherwise just aborts). + * Always returning 0 makes it a logging/benchmarking hook. + * + * - MLD_CONFIG_SIGN_HOOK_RESUME: uint16_t mld_sign_hook_resume([ctxt]) + * Returns the attempt to resume from (0 for a fresh operation), i.e. the one + * recorded when a previous call paused. + * + * - MLD_CONFIG_SIGN_HOOK_FINISH: void mld_sign_hook_finish(attempt[, ctxt]) + * Called on success with the succeeding attempt. Observe-only. + * + * When an option is unset, the hook is a no-op (resume to 0, attempt proceeds), + * i.e. ordinary one-shot signing. + * + * Independent of MLD_CONFIG_MAX_SIGNING_ATTEMPTS, which is a static upper bound + * on the number of signing attempts. + * + * See test/src/test_sign_hook.c for a worked example using all three. + */ +/* #define MLD_CONFIG_SIGN_HOOK_RESUME + #define MLD_CONFIG_SIGN_HOOK_ATTEMPT + #define MLD_CONFIG_SIGN_HOOK_FINISH + #if !defined(__ASSEMBLER__) + #include + #include "src/sys.h" + static MLD_INLINE uint16_t mld_sign_hook_resume(void) + { + ... return the attempt to resume from ... + } + static MLD_INLINE int mld_sign_hook_attempt(uint16_t attempt) + { + ... return non-zero to pause here; for resume, store attempt ... + return 0; + } + static MLD_INLINE void mld_sign_hook_finish(uint16_t attempt) + { + ... mark the operation complete (attempt = successful attempt) ... + } + #endif +*/ + +/** + * MLD_CONFIG_REDUCE_RAM + * + * Set this to reduce RAM usage. This trades memory for performance. + * + * For expected memory usage, see the MLD_TOTAL_ALLOC_* constants defined in + * mldsa_native.h. + * + * This option is useful for embedded systems with tight RAM constraints but + * relaxed performance requirements. + * + */ +/* #define MLD_CONFIG_REDUCE_RAM */ + +/************************* Config internals ********************************/ + +#endif /* MLD_BUILD_INTERNAL */ + +/* Default namespace + * + * Don't change this. If you need a different namespace, re-define + * MLD_CONFIG_NAMESPACE_PREFIX above instead, and remove the following. + * + * The default MLDSA namespace is + * + * PQCP_MLDSA_NATIVE_MLDSA_ + * + * e.g., PQCP_MLDSA_NATIVE_MLDSA44_ + */ + +#if defined(MLD_CONFIG_MULTILEVEL_BUILD) +/* In a multi-level build the parameter set is appended by the namespacing + * machinery, so the default prefix must not embed it. */ +#define MLD_DEFAULT_NAMESPACE_PREFIX PQCP_MLDSA_NATIVE_MLDSA +#elif MLD_CONFIG_PARAMETER_SET == 44 +#define MLD_DEFAULT_NAMESPACE_PREFIX PQCP_MLDSA_NATIVE_MLDSA44 +#elif MLD_CONFIG_PARAMETER_SET == 65 +#define MLD_DEFAULT_NAMESPACE_PREFIX PQCP_MLDSA_NATIVE_MLDSA65 +#elif MLD_CONFIG_PARAMETER_SET == 87 +#define MLD_DEFAULT_NAMESPACE_PREFIX PQCP_MLDSA_NATIVE_MLDSA87 +#endif + +#endif /* !MLD_CONFIG_H */ diff --git a/cbits/mldsa/src/cbmc.h b/cbits/mldsa/src/cbmc.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/cbmc.h @@ -0,0 +1,233 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_CBMC_H +#define MLD_CBMC_H + +/*************************************************** + * Basic replacements for __CPROVER_XXX contracts + ***************************************************/ +/* + * The `__contract__` / `__loop__` annotation macros use a + * leading-double-underscore spelling in line with other CBMC macros. + * clang-tidy flags these as reserved identifiers; we suppress the diagnostic + * at each definition site (NOLINT) rather than disabling the check globally, + * so it stays active for the rest of the tree. + */ +#ifndef CBMC + +/* clang-format off */ +#define __contract__(x) /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */ +#define __loop__(x) /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */ +/* clang-format on */ +#define cassert(x) + +#else /* !CBMC */ + + +/* clang-format off */ +#define __contract__(x) x /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */ +#define __loop__(x) x /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */ +/* clang-format on */ + +/* Conditionally expand to __VA_ARGS__ depending on MLD_CONFIG_REDUCE_RAM. */ +#if defined(MLD_CONFIG_REDUCE_RAM) +#define MLD_IF_REDUCE_RAM(...) __VA_ARGS__ +#define MLD_IF_NOT_REDUCE_RAM(...) +#else +#define MLD_IF_REDUCE_RAM(...) +#define MLD_IF_NOT_REDUCE_RAM(...) __VA_ARGS__ +#endif + +/* https://diffblue.github.io/cbmc/contracts-assigns.html */ +#define assigns(...) __CPROVER_assigns(__VA_ARGS__) + +/* https://diffblue.github.io/cbmc/contracts-requires-ensures.html */ +#define requires(...) __CPROVER_requires(__VA_ARGS__) +#define ensures(...) __CPROVER_ensures(__VA_ARGS__) +/* https://diffblue.github.io/cbmc/contracts-loops.html */ +#define invariant(...) __CPROVER_loop_invariant(__VA_ARGS__) +#define decreases(...) __CPROVER_decreases(__VA_ARGS__) +/* cassert to avoid confusion with in-built assert */ +#define cassert(x) __CPROVER_assert(x, "cbmc assertion failed") +#define assume(...) __CPROVER_assume(__VA_ARGS__) + +/*************************************************** + * Macros for "expression" forms that may appear + * _inside_ top-level contracts. + ***************************************************/ + +/* + * function return value - useful inside ensures + * https://diffblue.github.io/cbmc/contracts-functions.html + */ +#define return_value (__CPROVER_return_value) + +/* + * assigns l-value targets + * https://diffblue.github.io/cbmc/contracts-assigns.html + */ +#define object_whole(...) __CPROVER_object_whole(__VA_ARGS__) +#define memory_slice(...) __CPROVER_object_upto(__VA_ARGS__) +#define same_object(...) __CPROVER_same_object(__VA_ARGS__) + +/* + * Pointer-related predicates + * https://diffblue.github.io/cbmc/contracts-memory-predicates.html + */ +#define memory_no_alias(...) __CPROVER_is_fresh(__VA_ARGS__) +#define readable(...) __CPROVER_r_ok(__VA_ARGS__) +#define writeable(...) __CPROVER_w_ok(__VA_ARGS__) + +/* Maximum supported buffer size + * + * Larger buffers may be supported, but due to internal modeling constraints + * in CBMC, the proofs of memory- and type-safety won't be able to run. + * + * If you find yourself in need for a buffer size larger than this, + * please contact the maintainers, so we can prioritize work to relax + * this somewhat artificial bound. + */ +#define MLD_MAX_BUFFER_SIZE (SIZE_MAX >> 12) + + +/* + * History variables + * https://diffblue.github.io/cbmc/contracts-history-variables.html + */ +#define old(...) __CPROVER_old(__VA_ARGS__) +#define loop_entry(...) __CPROVER_loop_entry(__VA_ARGS__) + +/* + * Quantifiers + * Note that the range on qvar is _exclusive_ between qvar_lb .. qvar_ub + * https://diffblue.github.io/cbmc/contracts-quantifiers.html + * + * The quantified variable is declared as uint32_t, so these macros + * quantify only over indices in [0, UINT32_MAX). Bounds larger than + * UINT32_MAX (4 GiB) are NOT supported: the explicit (uint32_t) casts + * on the bounds will trigger CBMC's conversion check if a wider bound + * (e.g. a size_t > UINT32_MAX) is passed. + * + * Quantifying over size_t (64-bit) was found to blow up SMT proof + * times, so we deliberately keep the index width at 32 bits. Callers + * dealing with size_t-typed buffers must add an explicit + * requires(len <= UINT32_MAX) + * precondition. + */ + +/* + * Prevent clang-format from corrupting CBMC's special ==> operator + */ +/* clang-format off */ +#define forall(qvar, qvar_lb, qvar_ub, predicate) \ + __CPROVER_forall \ + { \ + uint32_t qvar; \ + ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> (predicate) \ + } + +#define exists(qvar, qvar_lb, qvar_ub, predicate) \ + __CPROVER_exists \ + { \ + uint32_t qvar; \ + ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) && (predicate) \ + } +/* clang-format on */ + +/*************************************************** + * Convenience macros for common contract patterns + ***************************************************/ +/* + * Prevent clang-format from corrupting CBMC's special ==> operator + */ +/* clang-format off */ +#define CBMC_CONCAT_(left, right) left##right +#define CBMC_CONCAT(left, right) CBMC_CONCAT_(left, right) + +#define array_bound_core(qvar, qvar_lb, qvar_ub, array_var, \ + value_lb, value_ub) \ + __CPROVER_forall \ + { \ + uint32_t qvar; \ + ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> \ + (((int)(value_lb) <= ((array_var)[(qvar)])) && \ + (((array_var)[(qvar)]) < (int)(value_ub))) \ + } + +#define array_bound(array_var, qvar_lb, qvar_ub, value_lb, value_ub) \ + array_bound_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), (qvar_lb), \ + (qvar_ub), (array_var), (value_lb), (value_ub)) + +#define array_unchanged_core(qvar, qvar_lb, qvar_ub, array_var) \ + __CPROVER_forall \ + { \ + uint32_t qvar; \ + ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> \ + ((array_var)[(qvar)]) == (old(* (int32_t (*)[(qvar_ub)])(array_var)))[(qvar)] \ + } + +#define array_unchanged(array_var, N) \ + array_unchanged_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), 0, (N), (array_var)) + +#define array_unchanged_u64_core(qvar, qvar_lb, qvar_ub, array_var) \ + __CPROVER_forall \ + { \ + uint32_t qvar; \ + ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> \ + ((array_var)[(qvar)]) == (old(* (uint64_t (*)[(qvar_ub)])(array_var)))[(qvar)] \ + } + +#define array_unchanged_u64(array_var, N) \ + array_unchanged_u64_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), 0, (N), (array_var)) + +#define array_unchanged_u8_core(qvar, qvar_lb, qvar_ub, array_var) \ + __CPROVER_forall \ + { \ + uint32_t qvar; \ + ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> \ + ((array_var)[(qvar)]) == (old(* (uint8_t (*)[(qvar_ub)])(array_var)))[(qvar)] \ + } + +#define array_unchanged_u8(array_var, N) \ + array_unchanged_u8_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), 0, (N), (array_var)) + +#define array_zeroized_u8_core(qvar, qvar_lb, qvar_ub, array_var) \ + __CPROVER_forall \ + { \ + uint32_t qvar; \ + ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> \ + ((array_var)[(qvar)]) == 0 \ + } + +#define array_zeroized_u8(array_var, N) \ + array_zeroized_u8_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), 0, (N), (array_var)) +/* clang-format on */ + +/* + * Output-buffer discipline on failure, as documented in API-CONVENTIONS.md: + * when a function fails, each caller-owned output buffer is left either + * fully unchanged or fully zeroized -- never holding partially computed or + * stale data that could be mistaken for a valid result. + * + * Note the disjunction is over the buffer as a whole: it is not enough for + * each byte to be individually either unchanged or zero. + */ +#define array_unchanged_or_zeroized_u8(array_var, N) \ + (array_unchanged_u8((array_var), (N)) || array_zeroized_u8((array_var), (N))) + +/* Wrapper around array_bound operating on absolute values. + * + * The absolute value bound `k` is exclusive. + * + * Note that since the lower bound in array_bound is inclusive, we have to + * raise it by 1 here. + */ +#define array_abs_bound(arr, lb, ub, k) \ + array_bound((arr), (lb), (ub), -((int)(k)) + 1, (k)) + +#endif /* CBMC */ + +#endif /* !MLD_CBMC_H */ diff --git a/cbits/mldsa/src/common.h b/cbits/mldsa/src/common.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/common.h @@ -0,0 +1,301 @@ +/* + * Copyright (c) The mldsa-native project authors + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLD_COMMON_H +#define MLD_COMMON_H + +#ifndef __ASSEMBLER__ +#include +#endif + + +#define MLD_BUILD_INTERNAL + +#if defined(MLD_CONFIG_FILE) +#include MLD_CONFIG_FILE +#else +#include "mldsa_native_config.h" +#endif + +#include "params.h" +#include "sys.h" + +/* Internal and public API have external linkage by default, but + * this can be overwritten by the user, e.g. for single-CU builds. */ +#if !defined(MLD_CONFIG_INTERNAL_API_QUALIFIER) +#define MLD_INTERNAL_API +#define MLD_INTERNAL_DATA_DECLARATION extern +#define MLD_INTERNAL_DATA_DEFINITION +#else +#define MLD_INTERNAL_API MLD_CONFIG_INTERNAL_API_QUALIFIER +#define MLD_INTERNAL_DATA_DECLARATION MLD_CONFIG_INTERNAL_API_QUALIFIER +#define MLD_INTERNAL_DATA_DEFINITION MLD_CONFIG_INTERNAL_API_QUALIFIER +#endif + +#if !defined(MLD_CONFIG_EXTERNAL_API_QUALIFIER) +#define MLD_EXTERNAL_API +#else +#define MLD_EXTERNAL_API MLD_CONFIG_EXTERNAL_API_QUALIFIER +#endif + +#if defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) || \ + defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) +#define MLD_MULTILEVEL_BUILD +#endif + +#define MLD_CONCAT_(x1, x2) x1##x2 +#define MLD_CONCAT(x1, x2) MLD_CONCAT_(x1, x2) + +#if defined(MLD_MULTILEVEL_BUILD) +#define MLD_ADD_PARAM_SET(s) MLD_CONCAT(s, MLD_CONFIG_PARAMETER_SET) +#else +#define MLD_ADD_PARAM_SET(s) s +#endif + +#define MLD_NAMESPACE_PREFIX MLD_CONCAT(MLD_CONFIG_NAMESPACE_PREFIX, _) +#define MLD_NAMESPACE_PREFIX_KL \ + MLD_CONCAT(MLD_ADD_PARAM_SET(MLD_CONFIG_NAMESPACE_PREFIX), _) + +/* Functions are prefixed by MLD_CONFIG_NAMESPACE_PREFIX. + * + * If multiple parameter sets are used, functions depending on the parameter + * set are additionally prefixed with 44/65/87. See mldsa_native_config.h. + * + * Example: If MLD_CONFIG_NAMESPACE_PREFIX is PQCP_MLDSA_NATIVE, then + * MLD_NAMESPACE_KL(keypair) becomes PQCP_MLDSA_NATIVE44_keypair/ + * PQCP_MLDSA_NATIVE65_keypair/PQCP_MLDSA_NATIVE87_keypair. + */ +#define MLD_NAMESPACE(s) MLD_CONCAT(MLD_NAMESPACE_PREFIX, s) +#define MLD_NAMESPACE_KL(s) MLD_CONCAT(MLD_NAMESPACE_PREFIX_KL, s) + +/* On Apple platforms, we need to emit leading underscore + * in front of assembly symbols. We thus introduce a separate + * namespace wrapper for ASM symbols. */ +#if !defined(__APPLE__) +#define MLD_ASM_NAMESPACE(sym) MLD_NAMESPACE(sym) +#else +#define MLD_ASM_NAMESPACE(sym) MLD_CONCAT(_, MLD_NAMESPACE(sym)) +#endif + +/* + * On X86_64 if control-flow protections (CET) are enabled (through + * -fcf-protection=), we add an endbr64 instruction at every global function + * label. See sys.h for more details + */ +#if defined(MLD_SYS_X86_64) +#define MLD_ASM_FN_SYMBOL(sym) MLD_ASM_NAMESPACE(sym) : MLD_CET_ENDBR +#elif defined(MLD_SYS_ARMV81M_MVE) +/* clang-format off */ +#define MLD_ASM_FN_SYMBOL(sym) \ + .type MLD_ASM_NAMESPACE(sym), %function; \ + MLD_ASM_NAMESPACE(sym) : +/* clang-format on */ +#else /* !MLD_SYS_X86_64 && MLD_SYS_ARMV81M_MVE */ +#define MLD_ASM_FN_SYMBOL(sym) MLD_ASM_NAMESPACE(sym) : +#endif /* !MLD_SYS_X86_64 && !MLD_SYS_ARMV81M_MVE */ + +/* + * Output the size of an assembly function. + */ +#if defined(__ELF__) +#define MLD_ASM_FN_SIZE(sym) \ + .size MLD_ASM_NAMESPACE(sym), .- MLD_ASM_NAMESPACE(sym) +#else +#define MLD_ASM_FN_SIZE(sym) +#endif + +/* We aim to simplify the user's life by supporting builds where + * all source files are included, even those that are not needed. + * Those files are appropriately guarded and will be empty when unneeded. + * The following is to avoid compilers complaining about this. */ +#define MLD_EMPTY_CU(s) extern int MLD_NAMESPACE_KL(empty_cu_##s); + +/* MLD_CONFIG_NO_ASM takes precedence over MLD_USE_NATIVE_XXX */ +#if defined(MLD_CONFIG_NO_ASM) +#undef MLD_CONFIG_USE_NATIVE_BACKEND_ARITH +#undef MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 +#endif + +#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH) && \ + !defined(MLD_CONFIG_ARITH_BACKEND_FILE) +#error Bad configuration: MLD_CONFIG_USE_NATIVE_BACKEND_ARITH is set, but MLD_CONFIG_ARITH_BACKEND_FILE is not. +#endif + +#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202) && \ + !defined(MLD_CONFIG_FIPS202_BACKEND_FILE) +#error Bad configuration: MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 is set, but MLD_CONFIG_FIPS202_BACKEND_FILE is not. +#endif + +#if defined(MLD_CONFIG_NO_RANDOMIZED_API) && defined(MLD_CONFIG_KEYGEN_PCT) +#error Bad configuration: MLD_CONFIG_NO_RANDOMIZED_API is incompatible with MLD_CONFIG_KEYGEN_PCT as the current PCT implementation requires signature() +#endif + +#if defined(MLD_CONFIG_NO_SIGN_API) && defined(MLD_CONFIG_KEYGEN_PCT) +#error Bad configuration: MLD_CONFIG_NO_SIGN_API is incompatible with MLD_CONFIG_KEYGEN_PCT as the current PCT implementation requires signature() +#endif + +#if defined(MLD_CONFIG_NO_VERIFY_API) && defined(MLD_CONFIG_KEYGEN_PCT) +#error Bad configuration: MLD_CONFIG_NO_VERIFY_API is incompatible with MLD_CONFIG_KEYGEN_PCT as the current PCT implementation requires verify() +#endif + +#if defined(MLD_CONFIG_CORE_API_ONLY) && defined(MLD_CONFIG_KEYGEN_PCT) +#error Bad configuration: MLD_CONFIG_CORE_API_ONLY is incompatible with MLD_CONFIG_KEYGEN_PCT as the current PCT implementation requires signature() and verify() +#endif + +#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_ARITH) +#include MLD_CONFIG_ARITH_BACKEND_FILE +/* Include to enforce consistency of API and implementation, + * and conduct sanity checks on the backend. + * + * Keep this _after_ the inclusion of the backend; otherwise, + * the sanity checks won't have an effect. */ +#if defined(MLD_CHECK_APIS) && !defined(__ASSEMBLER__) +#include "native/api.h" +#endif +#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_ARITH */ + +#if defined(MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202) +#include MLD_CONFIG_FIPS202_BACKEND_FILE +/* Include to enforce consistency of API and implementation, + * and conduct sanity checks on the backend. + * + * Keep this _after_ the inclusion of the backend; otherwise, + * the sanity checks won't have an effect. */ +#if defined(MLD_CHECK_APIS) && !defined(__ASSEMBLER__) +#include "fips202/native/api.h" +#endif +#endif /* MLD_CONFIG_USE_NATIVE_BACKEND_FIPS202 */ + +#if !defined(MLD_CONFIG_FIPS202_CUSTOM_HEADER) +#define MLD_FIPS202_HEADER_FILE "fips202/fips202.h" +#else +#define MLD_FIPS202_HEADER_FILE MLD_CONFIG_FIPS202_CUSTOM_HEADER +#endif + +#if !defined(MLD_CONFIG_FIPS202X4_CUSTOM_HEADER) +#define MLD_FIPS202X4_HEADER_FILE "fips202/fips202x4.h" +#else +#define MLD_FIPS202X4_HEADER_FILE MLD_CONFIG_FIPS202X4_CUSTOM_HEADER +#endif + +/* Standard library function replacements */ +#if !defined(__ASSEMBLER__) +#if !defined(MLD_CONFIG_CUSTOM_MEMCPY) +#include +#define mld_memcpy memcpy +#endif + +#if !defined(MLD_CONFIG_CUSTOM_MEMSET) +#include +#define mld_memset memset +#endif + +/* Allocation macros for large local structures + * + * MLD_ALLOC(v, T, N) declares T *v and attempts to point it to an T[N] + * MLD_FREE(v, T, N) zeroizes and frees the allocation + * + * Default implementation uses stack allocation. + * Can be overridden by setting the config option MLD_CONFIG_CUSTOM_ALLOC_FREE + * and defining MLD_CUSTOM_ALLOC and MLD_CUSTOM_FREE. + */ +#if defined(MLD_CONFIG_CUSTOM_ALLOC_FREE) != \ + (defined(MLD_CUSTOM_ALLOC) && defined(MLD_CUSTOM_FREE)) +#error Bad configuration: MLD_CONFIG_CUSTOM_ALLOC_FREE must be set together with MLD_CUSTOM_ALLOC and MLD_CUSTOM_FREE +#endif + +/* Context-parameter machinery (MLD_CONTEXT_PARAMETERS_n and related config + * checks). Kept in a separate, level-generic header for readability; included + * here so it is available to the allocation macros below and to all consumers + * of common.h. */ +#include "context.h" + +#if !defined(MLD_CONFIG_CUSTOM_ALLOC_FREE) +/* Default: stack allocation */ + +/* This is a declaration macro, not an expression macro: T is a type and v is + * a declarator, neither of which can be wrapped in parentheses. The + * bugprone-macro-parentheses diagnostic is therefore a false positive here. */ +#define MLD_ALLOC(v, T, N, context) \ + MLD_ALIGN T mld_alloc_##v[N]; \ + T *v = mld_alloc_##v /* NOLINT(bugprone-macro-parentheses) */ + +/* The MLD_FREE macro body references mld_zeroize(), which is declared in + * ct.h. We deliberately do NOT include ct.h here: doing so would create a + * circular dependency (ct.h includes common.h), and common.h itself never + * calls mld_zeroize() -- only the macro expansion does. Each translation + * unit that uses MLD_FREE therefore includes ct.h directly. */ +#define MLD_FREE(v, T, N, context) \ + do \ + { \ + MLD_CONTEXT_UNUSED(context); \ + mld_zeroize(mld_alloc_##v, sizeof(mld_alloc_##v)); \ + (v) = NULL; \ + } while (0) + +#else /* !MLD_CONFIG_CUSTOM_ALLOC_FREE */ + +/* Custom allocation */ + +/* + * The indirection here is necessary to use MLD_CONTEXT_PARAMETERS_3 here. + */ +#define MLD_APPLY(f, args) f args + +#define MLD_ALLOC(v, T, N, context) \ + MLD_APPLY(MLD_CUSTOM_ALLOC, MLD_CONTEXT_PARAMETERS_3(v, T, N, context)) + +#define MLD_FREE(v, T, N, context) \ + do \ + { \ + if (v != NULL) \ + { \ + mld_zeroize(v, sizeof(T) * (N)); \ + MLD_APPLY(MLD_CUSTOM_FREE, MLD_CONTEXT_PARAMETERS_3(v, T, N, context)); \ + v = NULL; \ + } \ + } while (0) + +#endif /* MLD_CONFIG_CUSTOM_ALLOC_FREE */ + +/****************************** Error codes ***********************************/ + +/* Generic failure condition, reserved for failures not covered by a more + * specific error code. */ +#define MLD_ERR_FAIL (-1) +/* An allocation failed. This can only happen if MLD_CONFIG_CUSTOM_ALLOC_FREE + * is defined and the provided MLD_CUSTOM_ALLOC can fail. */ +#define MLD_ERR_OUT_OF_MEMORY (-2) +/* An RNG failure occurred. Might be due to insufficient entropy or + * system misconfiguration. */ +#define MLD_ERR_RNG_FAIL (-3) +/* The signing rejection-sampling loop exceeded + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS iterations without producing a valid + * signature. With a FIPS 204 Appendix C compliant bound (>= 821) this + * has probability < 2^-256. */ +#define MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED (-4) +/* Signing was paused before completing, at the request of a caller-provided + * MLD_CONFIG_SIGN_HOOK_ATTEMPT hook (see mldsa_native_config.h). The caller + * resumes by re-invoking signing with the same inputs; the attempt hook, + * together with MLD_CONFIG_SIGN_HOOK_RESUME, decides where to continue. */ +#define MLD_ERR_SIGNING_PAUSED (-5) +/* Signature verification failed: the signature is not valid for the given + * message and public key. Returned by the verification API. */ +#define MLD_ERR_INVALID_SIGNATURE (-6) +/* Secret key validation failed: the secret key is malformed or internally + * inconsistent. Returned by pk_from_sk. */ +#define MLD_ERR_INVALID_KEY (-7) +/* The Pairwise Consistency Test failed. Only possible when + * MLD_CONFIG_KEYGEN_PCT is enabled; signals that the freshly generated key + * pair failed its sign/verify self-test. */ +#define MLD_ERR_PCT_FAIL (-8) +/* An argument was invalid, e.g. an unsupported pre-hash algorithm or a context + * string longer than 255 bytes. */ +#define MLD_ERR_INVALID_ARG (-9) + + +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLD_COMMON_H */ diff --git a/cbits/mldsa/src/context.h b/cbits/mldsa/src/context.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/context.h @@ -0,0 +1,152 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLD_CONTEXT_H +#define MLD_CONTEXT_H + +/* This header is included by common.h once the configuration has been pulled + * in; it is not meant to be included directly. */ +#if !defined(__ASSEMBLER__) + +#include +#include "cbmc.h" +#include "sys.h" + +/* + * If the integration wants to provide a context parameter for use in + * platform-specific hooks, then it should define this parameter. + * + * The MLD_CONTEXT_PARAMETERS_n macros are intended to be used with macros + * defining the function names and expand to either pass or discard the context + * argument as required by the current build. If there is no context parameter + * requested then these are removed from the prototypes and from all calls. + */ +#ifdef MLD_CONFIG_CONTEXT_PARAMETER +#define MLD_CONTEXT_PARAMETERS_0(context) (context) +#define MLD_CONTEXT_PARAMETERS_1(arg0, context) (arg0, context) +#define MLD_CONTEXT_PARAMETERS_2(arg0, arg1, context) (arg0, arg1, context) +#define MLD_CONTEXT_PARAMETERS_3(arg0, arg1, arg2, context) \ + (arg0, arg1, arg2, context) +#define MLD_CONTEXT_PARAMETERS_4(arg0, arg1, arg2, arg3, context) \ + (arg0, arg1, arg2, arg3, context) +#define MLD_CONTEXT_PARAMETERS_5(arg0, arg1, arg2, arg3, arg4, context) \ + (arg0, arg1, arg2, arg3, arg4, context) +#define MLD_CONTEXT_PARAMETERS_6(arg0, arg1, arg2, arg3, arg4, arg5, context) \ + (arg0, arg1, arg2, arg3, arg4, arg5, context) +#define MLD_CONTEXT_PARAMETERS_7(arg0, arg1, arg2, arg3, arg4, arg5, arg6, \ + context) \ + (arg0, arg1, arg2, arg3, arg4, arg5, arg6, context) +#define MLD_CONTEXT_PARAMETERS_8(arg0, arg1, arg2, arg3, arg4, arg5, arg6, \ + arg7, context) \ + (arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7, context) +#define MLD_CONTEXT_PARAMETERS_9(arg0, arg1, arg2, arg3, arg4, arg5, arg6, \ + arg7, arg8, context) \ + (arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8, context) +#else /* MLD_CONFIG_CONTEXT_PARAMETER */ +#define MLD_CONTEXT_PARAMETERS_0(context) () +#define MLD_CONTEXT_PARAMETERS_1(arg0, context) (arg0) +#define MLD_CONTEXT_PARAMETERS_2(arg0, arg1, context) (arg0, arg1) +#define MLD_CONTEXT_PARAMETERS_3(arg0, arg1, arg2, context) (arg0, arg1, arg2) +#define MLD_CONTEXT_PARAMETERS_4(arg0, arg1, arg2, arg3, context) \ + (arg0, arg1, arg2, arg3) +#define MLD_CONTEXT_PARAMETERS_5(arg0, arg1, arg2, arg3, arg4, context) \ + (arg0, arg1, arg2, arg3, arg4) +#define MLD_CONTEXT_PARAMETERS_6(arg0, arg1, arg2, arg3, arg4, arg5, context) \ + (arg0, arg1, arg2, arg3, arg4, arg5) +#define MLD_CONTEXT_PARAMETERS_7(arg0, arg1, arg2, arg3, arg4, arg5, arg6, \ + context) \ + (arg0, arg1, arg2, arg3, arg4, arg5, arg6) +#define MLD_CONTEXT_PARAMETERS_8(arg0, arg1, arg2, arg3, arg4, arg5, arg6, \ + arg7, context) \ + (arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7) +#define MLD_CONTEXT_PARAMETERS_9(arg0, arg1, arg2, arg3, arg4, arg5, arg6, \ + arg7, arg8, context) \ + (arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7, arg8) +#endif /* !MLD_CONFIG_CONTEXT_PARAMETER */ + +/* Consume a context parameter carried only for the integration's benefit, + * avoiding -Wunused-parameter; expands to nothing when no context is + * configured. */ +#if defined(MLD_CONFIG_CONTEXT_PARAMETER) +#define MLD_CONTEXT_UNUSED(context) ((void)(context)) +#else +#define MLD_CONTEXT_UNUSED(context) ((void)0) +#endif + +#if defined(MLD_CONFIG_CONTEXT_PARAMETER_TYPE) != \ + defined(MLD_CONFIG_CONTEXT_PARAMETER) +#error MLD_CONFIG_CONTEXT_PARAMETER_TYPE must be defined if and only if MLD_CONFIG_CONTEXT_PARAMETER is defined +#endif + +/* The signing hooks tie into the rejection-sampling loop. A pausing attempt + * hook only reproduces the uninterrupted signature if the randomness is fixed + * across calls, and thus requires the deterministic API. + * For now we impose that requirement on all three hooks uniformly: enabling any + * of them requires MLD_CONFIG_NO_RANDOMIZED_API. This also rules out + * MLD_CONFIG_KEYGEN_PCT (whose PCT needs the randomized signature(), see + * common.h). + * + * A logging-only use (attempt always returns 0; resume/finish merely observe) + * would be safe with the randomized API too, but the restriction is applied + * uniformly for now. */ +#if (defined(MLD_CONFIG_SIGN_HOOK_RESUME) || \ + defined(MLD_CONFIG_SIGN_HOOK_ATTEMPT) || \ + defined(MLD_CONFIG_SIGN_HOOK_FINISH)) && \ + !defined(MLD_CONFIG_NO_RANDOMIZED_API) +#error Signing hooks (MLD_CONFIG_SIGN_HOOK_RESUME / _ATTEMPT / _FINISH) require MLD_CONFIG_NO_RANDOMIZED_API +#endif /* (MLD_CONFIG_SIGN_HOOK_RESUME || MLD_CONFIG_SIGN_HOOK_ATTEMPT || \ + MLD_CONFIG_SIGN_HOOK_FINISH) && !MLD_CONFIG_NO_RANDOMIZED_API */ + +/* Signing hooks (MLD_CONFIG_SIGN_HOOK_RESUME / _ATTEMPT / _FINISH; documented + * in mldsa_native_config.h). The following macros route the call sites to + * mld_sign_hook_*, appending or dropping the context argument; each unset hook + * uses the dummy below. */ +#define mld_sign_resume mld_sign_hook_resume MLD_CONTEXT_PARAMETERS_0 +#define mld_sign_attempt mld_sign_hook_attempt MLD_CONTEXT_PARAMETERS_1 +#define mld_sign_finish mld_sign_hook_finish MLD_CONTEXT_PARAMETERS_1 + +/* We don't use mld_sign_resume here because MLD_CONTEXT_PARAMETERS_0 is + * unsuitable for function declarations: it misses `void` as the placeholder + * argument. */ +#if !defined(MLD_CONFIG_SIGN_HOOK_RESUME) +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE uint16_t mld_sign_hook_resume( +#if defined(MLD_CONFIG_CONTEXT_PARAMETER) + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context +#else + void +#endif +) +__contract__(assigns() ensures(1)) +{ + MLD_CONTEXT_UNUSED(context); + return 0; +} +#endif /* !MLD_CONFIG_SIGN_HOOK_RESUME */ + +#if !defined(MLD_CONFIG_SIGN_HOOK_ATTEMPT) +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_sign_attempt( + uint16_t attempt, MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__(assigns() ensures(1)) +{ + ((void)attempt); + MLD_CONTEXT_UNUSED(context); + return 0; +} +#endif /* !MLD_CONFIG_SIGN_HOOK_ATTEMPT */ + +#if !defined(MLD_CONFIG_SIGN_HOOK_FINISH) +static MLD_INLINE void mld_sign_finish( + uint16_t attempt, MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__(assigns() ensures(1)) +{ + ((void)attempt); + MLD_CONTEXT_UNUSED(context); +} +#endif /* !MLD_CONFIG_SIGN_HOOK_FINISH */ + +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLD_CONTEXT_H */ diff --git a/cbits/mldsa/src/ct.c b/cbits/mldsa/src/ct.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/ct.c @@ -0,0 +1,21 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#include "ct.h" + +#if !defined(MLD_USE_ASM_VALUE_BARRIER) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) +/* + * Masking value used in constant-time functions from + * ct.h to block the compiler's range analysis and + * thereby reduce the risk of compiler-introduced branches. + */ +volatile uint64_t mld_ct_opt_blocker_u64 = 0; + +#else /* !MLD_USE_ASM_VALUE_BARRIER && !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +MLD_EMPTY_CU(ct) + +#endif /* !(!MLD_USE_ASM_VALUE_BARRIER && !MLD_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mldsa/src/ct.h b/cbits/mldsa/src/ct.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/ct.h @@ -0,0 +1,373 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + * + * - [libmceliece] + * libmceliece implementation of Classic McEliece + * Bernstein, Chou + * https://lib.mceliece.org/ + * + * - [optblocker] + * PQC forum post on opt-blockers using volatile globals + * Daniel J. Bernstein + * https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/hqbtIGFKIpU/m/H14H0wOlBgAJ + */ + +#ifndef MLD_CT_H +#define MLD_CT_H + +#include "cbmc.h" +#include "common.h" + +/* Constant-time comparisons and conditional operations + + We reduce the risk for compilation into variable-time code + through the use of 'value barriers'. + + Functionally, a value barrier is a no-op. To the compiler, however, + it constitutes an arbitrary modification of its input, and therefore + harden's value propagation and range analysis. + + We consider two approaches to implement a value barrier: + - An empty inline asm block which marks the target value as clobbered. + - XOR'ing with the value of a volatile global that's set to 0; + see @[optblocker] for a discussion of this idea, and + @[libmceliece, inttypes/crypto_intN.h] for an implementation. + + The first approach is cheap because it only prevents the compiler + from reasoning about the value of the variable past the barrier, + but does not directly generate additional instructions. + + The second approach generates redundant loads and XOR operations + and therefore comes at a higher runtime cost. However, it appears + more robust towards optimization, as compilers should never drop + a volatile load. + + We use the empty-ASM value barrier for GCC and clang, and fall + back to the global volatile barrier otherwise. + + The global value barrier can be forced by setting + MLD_CONFIG_NO_ASM_VALUE_BARRIER. + +*/ + +#if defined(MLD_HAVE_INLINE_ASM) && !defined(MLD_CONFIG_NO_ASM_VALUE_BARRIER) +#define MLD_USE_ASM_VALUE_BARRIER +#endif + + +#if !defined(MLD_USE_ASM_VALUE_BARRIER) +/* + * Declaration of global volatile that the global value barrier + * is loading from and masking with. + */ +#define mld_ct_opt_blocker_u64 MLD_NAMESPACE(ct_opt_blocker_u64) +extern volatile uint64_t mld_ct_opt_blocker_u64; + + +/* Helper functions for obtaining global masks of various sizes */ + +/* This contract is not proved but treated as an axiom. + * + * Its validity relies on the assumption that the global opt-blocker + * constant mld_ct_opt_blocker_u64 is not modified. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE uint64_t mld_ct_get_optblocker_u64(void) +__contract__(ensures(return_value == 0)) { return mld_ct_opt_blocker_u64; } + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int64_t mld_ct_get_optblocker_i64(void) +__contract__(ensures(return_value == 0)) { return (int64_t)mld_ct_get_optblocker_u64(); } + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE uint32_t mld_ct_get_optblocker_u32(void) +__contract__(ensures(return_value == 0)) { return (uint32_t)mld_ct_get_optblocker_u64(); } + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE uint8_t mld_ct_get_optblocker_u8(void) +__contract__(ensures(return_value == 0)) { return (uint8_t)mld_ct_get_optblocker_u64(); } + +/* Opt-blocker based implementation of value barriers */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int64_t mld_value_barrier_i64(int64_t b) +__contract__(ensures(return_value == b)) { return (b ^ mld_ct_get_optblocker_i64()); } + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE uint32_t mld_value_barrier_u32(uint32_t b) +__contract__(ensures(return_value == b)) { return (b ^ mld_ct_get_optblocker_u32()); } + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE uint8_t mld_value_barrier_u8(uint8_t b) +__contract__(ensures(return_value == b)) { return (b ^ mld_ct_get_optblocker_u8()); } + + +#else /* !MLD_USE_ASM_VALUE_BARRIER */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int64_t mld_value_barrier_i64(int64_t b) +__contract__(ensures(return_value == b)) +{ + __asm__ volatile("" : "+r"(b)); + return b; +} + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE uint32_t mld_value_barrier_u32(uint32_t b) +__contract__(ensures(return_value == b)) +{ + __asm__ volatile("" : "+r"(b)); + return b; +} + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE uint8_t mld_value_barrier_u8(uint8_t b) +__contract__(ensures(return_value == b)) +{ + __asm__ volatile("" : "+r"(b)); + return b; +} +#endif /* MLD_USE_ASM_VALUE_BARRIER */ + +#ifdef CBMC +#pragma CPROVER check push +#pragma CPROVER check disable "conversion" +#endif + +/** + * Cast uint32 value to int32. + * + * @param x Input value. + * + * @return For uint32_t x, the unique y in int32_t so that x == y mod 2^32. + * Concretely: + * - x < 2^31: returns x + * - x >= 2^31: returns x - 2^32 + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_ALWAYS_INLINE int32_t mld_cast_uint32_to_int32(uint32_t x) +{ + /* + * PORTABILITY: This relies on uint32_t -> int32_t + * being implemented as the inverse of int32_t -> uint32_t, + * which is implementation-defined (C99 6.3.1.3 (3)) + * CBMC (correctly) fails to prove this conversion is OK, + * so we have to suppress that check here + */ + return (int32_t)x; +} + +#ifdef CBMC +#pragma CPROVER check pop +#endif + + +/** + * Cast int64 value to uint32 as per C standard. + * + * @param x Input value. + * + * @return For int64_t x, the unique y in uint32_t so that x == y mod 2^32. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_ALWAYS_INLINE uint32_t mld_cast_int64_to_uint32(int64_t x) +{ + return (uint32_t)(x & (int64_t)UINT32_MAX); +} + +/** + * Cast int32 value to uint32 as per C standard. + * + * @param x Input value. + * + * @return For int32_t x, the unique y in uint32_t so that x == y mod 2^32. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_ALWAYS_INLINE uint32_t mld_cast_int32_to_uint32(int32_t x) +{ + return mld_cast_int64_to_uint32((int64_t)x); +} + +/** + * Functionally equivalent to cond ? a : b, but implemented with guards against + * compiler-introduced branches. + * + * @param a First alternative. + * @param b Second alternative. + * @param cond Condition variable. + * + * @return a if cond is 0xFFFFFFFF, b if cond is 0. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int32_t mld_ct_sel_int32(int32_t a, int32_t b, uint32_t cond) +__contract__( + requires(cond == 0x0 || cond == 0xFFFFFFFF) + ensures(return_value == (cond ? a : b)) +) +{ + uint32_t au = mld_cast_int32_to_uint32(a); + uint32_t bu = mld_cast_int32_to_uint32(b); + uint32_t res = bu ^ (mld_value_barrier_u32(cond) & (au ^ bu)); + return mld_cast_uint32_to_int32(res); +} + +/** + * Return 0 if input is zero, and -1 otherwise. + * + * @param x Value to be converted into a mask. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE uint32_t mld_ct_cmask_nonzero_u32(uint32_t x) +__contract__(ensures(return_value == ((x == 0) ? 0 : 0xFFFFFFFF))) +{ + int64_t tmp = mld_value_barrier_i64(-((int64_t)x)); + /* + * PORTABILITY: Right-shift on a signed integer is + * implementation-defined for negative left argument. + * Here, we assume it's sign-preserving "arithmetic" shift right. + * See (C99 6.5.7 (5)) + */ + tmp >>= 32; + return mld_cast_int64_to_uint32(tmp); +} + +/** + * Return 0 if input is zero, and -1 otherwise. + * + * @param x Value to be converted into a mask. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE uint8_t mld_ct_cmask_nonzero_u8(uint8_t x) +__contract__(ensures(return_value == ((x == 0) ? 0 : 0xFF))) +{ + uint32_t mask = mld_ct_cmask_nonzero_u32((uint32_t)x); + return (uint8_t)(mask & 0xFF); +} + +/** + * Return 0 if input is non-negative, and -1 otherwise. + * + * @param x Value to be converted into a mask. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE uint32_t mld_ct_cmask_neg_i32(int32_t x) +__contract__( + ensures(return_value == ((x < 0) ? 0xFFFFFFFF : 0)) +) +{ + int64_t tmp = mld_value_barrier_i64((int64_t)x); + /* + * PORTABILITY: Right-shift on a signed integer is + * implementation-defined for negative left argument. + * Here, we assume it's sign-preserving "arithmetic" shift right. + * See (C99 6.5.7 (5)) + */ + tmp >>= 31; + return mld_cast_int64_to_uint32(tmp); +} + +/** + * Return -x if x<0, x otherwise. + * + * @param x Input value. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int32_t mld_ct_abs_i32(int32_t x) +__contract__( + requires(x >= -INT32_MAX) + ensures(return_value == ((x < 0) ? -x : x)) +) +{ + return mld_ct_sel_int32(-x, x, mld_ct_cmask_neg_i32(x)); +} + +/** + * Compare two arrays for equality in constant time. + * + * @param[in] a Pointer to first byte array. + * @param[in] b Pointer to second byte array. + * @param len Length of the byte arrays, upper-bounded to UINT16_MAX to + * control proof complexity only. + * + * @return 0 if the byte arrays are equal, 0xFF otherwise. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE uint8_t mld_ct_memcmp(const uint8_t *a, const uint8_t *b, + const size_t len) +__contract__( + requires(len <= UINT16_MAX) + requires(memory_no_alias(a, len)) + requires(memory_no_alias(b, len)) + ensures((return_value == 0) || (return_value == 0xFF)) + ensures((return_value == 0) == forall(i, 0, len, (a[i] == b[i])))) +{ + uint8_t r = 0, s = 0; + unsigned i; + + for (i = 0; i < len; i++) + __loop__( + invariant(i <= len) + invariant((r == 0) == (forall(k, 0, i, (a[k] == b[k])))) + decreases(len - i)) + { + r |= a[i] ^ b[i]; + /* s is useless, but prevents the loop from being aborted once r=0xff. */ + s ^= a[i] ^ b[i]; + } + + /* + * - Convert r into a mask; this may not be necessary, but is an additional + * safeguard + * towards leaking information about a and b. + * - XOR twice with s, separated by a value barrier, to prevent the compile + * from dropping the s computation in the loop. + */ + return (mld_value_barrier_u8(mld_ct_cmask_nonzero_u8(r) ^ s) ^ s); +} + +/** + * Force-zeroize a buffer. + * + * @[FIPS204, Section 3.6.3] Destruction of intermediate values. + * + * @param[out] ptr Pointer to buffer to be zeroed. + * @param len Amount of bytes to be zeroed. + */ +#if !defined(MLD_CONFIG_CUSTOM_ZEROIZE) +#if defined(MLD_SYS_WINDOWS) +#include +#elif !defined(MLD_HAVE_INLINE_ASM) +#error No plausibly-secure implementation of mld_zeroize available. Please provide your own using MLD_CONFIG_CUSTOM_ZEROIZE. +#endif + +static MLD_INLINE void mld_zeroize(void *ptr, size_t len) +__contract__( + requires(len <= UINT32_MAX) + requires(memory_no_alias(ptr, len)) + assigns(memory_slice(ptr, len)) + ensures(array_zeroized_u8((uint8_t *)ptr, len))) +{ +#if defined(MLD_SYS_WINDOWS) + SecureZeroMemory(ptr, len); +#else + mld_memset(ptr, 0, len); + /* This follows OpenSSL and seems sufficient to prevent the compiler + * from optimizing away the memset. + * + * If there was a reliable way to detect availability of memset_s(), + * that would be preferred. */ + __asm__ __volatile__("" : : "r"(ptr) : "memory"); +#endif /* !MLD_SYS_WINDOWS */ +} +#endif /* !MLD_CONFIG_CUSTOM_ZEROIZE */ + +#endif /* !MLD_CT_H */ diff --git a/cbits/mldsa/src/debug.c b/cbits/mldsa/src/debug.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/debug.c @@ -0,0 +1,75 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* NOTE: You can remove this file unless you compile with MLDSA_DEBUG. */ + +#include "common.h" + +#if !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +#if defined(MLDSA_DEBUG) + +#include +#include +#include +#include "debug.h" + +#define MLD_DEBUG_ERROR_HEADER "[ERROR:%s:%04d] " + +MLD_INTERNAL_API +void mld_debug_check_assert(const char *file, int line, const int val) +{ + if (val == 0) + { + fprintf(stderr, MLD_DEBUG_ERROR_HEADER "Assertion failed (value %d)\n", + file, line, val); + exit(1); + } +} + +MLD_INTERNAL_API +void mld_debug_check_bounds(const char *file, int line, const int32_t *ptr, + unsigned len, int64_t lower_bound_exclusive, + int64_t upper_bound_exclusive) +{ + int err = 0; + unsigned i; + for (i = 0; i < len; i++) + { + int32_t val = ptr[i]; + if (!(val > lower_bound_exclusive && val < upper_bound_exclusive)) + { + fprintf(stderr, + MLD_DEBUG_ERROR_HEADER + "Bounds assertion failed: Index %u, value %d out of bounds " + "(%" PRId64 ",%" PRId64 ")\n", + file, line, i, (int)val, lower_bound_exclusive, + upper_bound_exclusive); + err = 1; + } + } + + if (err == 1) + { + exit(1); + } +} + +#else /* MLDSA_DEBUG */ + +MLD_EMPTY_CU(debug) + +#endif /* !MLDSA_DEBUG */ + +#else /* !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +MLD_EMPTY_CU(debug) + +#endif /* MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. + * Don't modify by hand -- this is auto-generated by scripts/autogen. */ +#undef MLD_DEBUG_ERROR_HEADER diff --git a/cbits/mldsa/src/debug.h b/cbits/mldsa/src/debug.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/debug.h @@ -0,0 +1,125 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLD_DEBUG_H +#define MLD_DEBUG_H +#include "common.h" + +#if defined(MLDSA_DEBUG) + +/** + * Check debug assertion. + * + * Prints an error message to stderr and calls exit(1) if not. + * + * @param file Filename. + * @param line Line number. + * @param val Value asserted to be non-zero. + */ +#define mld_debug_check_assert MLD_NAMESPACE(mldsa_debug_assert) +MLD_INTERNAL_API +void mld_debug_check_assert(const char *file, int line, const int val); + +/** + * Check whether values in an array of int32_t are within specified bounds. + * + * Prints an error message to stderr and calls exit(1) if not. + * + * @param file Filename. + * @param line Line number. + * @param[in] ptr Base of array to be checked. + * @param len Number of int32_t in ptr. + * @param lower_bound_exclusive Exclusive lower bound. + * @param upper_bound_exclusive Exclusive upper bound. + */ +#define mld_debug_check_bounds MLD_NAMESPACE(mldsa_debug_check_bounds) +MLD_INTERNAL_API +void mld_debug_check_bounds(const char *file, int line, const int32_t *ptr, + unsigned len, int64_t lower_bound_exclusive, + int64_t upper_bound_exclusive); + +/* Check assertion, calling exit() upon failure + * + * val: Value that's asserted to be non-zero + */ +#define mld_assert(val) mld_debug_check_assert(__FILE__, __LINE__, (val)) + +/* Check bounds in array of int32_t's + * ptr: Base of int32_t array; will be explicitly cast to int32_t*, + * so you may pass a byte-compatible type such as mld_poly or mld_polyvec. + * len: Number of int32_t in array + * value_lb: Inclusive lower value bound + * value_ub: Exclusive upper value bound */ +#define mld_assert_bound(ptr, len, value_lb, value_ub) \ + mld_debug_check_bounds(__FILE__, __LINE__, (const int32_t *)(ptr), (len), \ + ((int64_t)(value_lb)) - 1, (value_ub)) + +/* Check absolute bounds in array of int32_t's + * ptr: Base of array, expression of type int32_t* + * len: Number of int32_t in array + * value_abs_bd: Exclusive absolute upper bound */ +#define mld_assert_abs_bound(ptr, len, value_abs_bd) \ + mld_assert_bound((ptr), (len), (-((int64_t)(value_abs_bd)) + 1), \ + (value_abs_bd)) + +/* Version of bounds assertions for 2-dimensional arrays */ +#define mld_assert_bound_2d(ptr, len0, len1, value_lb, value_ub) \ + mld_assert_bound((ptr), ((len0) * (len1)), (value_lb), (value_ub)) + +#define mld_assert_abs_bound_2d(ptr, len0, len1, value_abs_bd) \ + mld_assert_abs_bound((ptr), ((len0) * (len1)), (value_abs_bd)) + +/* When running CBMC, convert debug assertions into proof obligations */ +#elif defined(CBMC) +#include "cbmc.h" + +#define mld_assert(val) cassert(val) + +#define mld_assert_bound(ptr, len, value_lb, value_ub) \ + cassert(array_bound(((int32_t *)(ptr)), 0, (len), (value_lb), (value_ub))) + +#define mld_assert_abs_bound(ptr, len, value_abs_bd) \ + cassert(array_abs_bound(((int32_t *)(ptr)), 0, (len), (value_abs_bd))) + +/* Because of https://github.com/diffblue/cbmc/issues/8570, we can't + * just use a single flattened array_bound(...) here. */ +#define mld_assert_bound_2d(ptr, M, N, value_lb, value_ub) \ + cassert(forall(kN, 0, (M), \ + array_bound(&((int32_t (*)[(N)])(ptr))[kN][0], 0, (N), \ + (value_lb), (value_ub)))) + +#define mld_assert_abs_bound_2d(ptr, M, N, value_abs_bd) \ + cassert(forall(kN, 0, (M), \ + array_abs_bound(&((int32_t (*)[(N)])(ptr))[kN][0], 0, (N), \ + (value_abs_bd)))) + +#else /* !MLDSA_DEBUG && CBMC */ + +#define mld_assert(val) \ + do \ + { \ + } while (0) +#define mld_assert_bound(ptr, len, value_lb, value_ub) \ + do \ + { \ + } while (0) +#define mld_assert_abs_bound(ptr, len, value_abs_bd) \ + do \ + { \ + } while (0) + +#define mld_assert_bound_2d(ptr, len0, len1, value_lb, value_ub) \ + do \ + { \ + } while (0) + +#define mld_assert_abs_bound_2d(ptr, len0, len1, value_abs_bd) \ + do \ + { \ + } while (0) + + +#endif /* !MLDSA_DEBUG && !CBMC */ +#endif /* !MLD_DEBUG_H */ diff --git a/cbits/mldsa/src/fips202/fips202.c b/cbits/mldsa/src/fips202/fips202.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/fips202.c @@ -0,0 +1,270 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + * + * - [mupq] + * Common files for pqm4, pqm3, pqriscv + * Kannwischer, Petri, Rijneveld, Schwabe, Stoffelen + * https://github.com/mupq/mupq + * + * - [supercop] + * SUPERCOP benchmarking framework + * Daniel J. Bernstein + * http://bench.cr.yp.to/supercop.html + * + * - [tweetfips] + * 'tweetfips202' FIPS202 implementation + * Van Assche, Bernstein, Schwabe + * https://keccak.team/2015/tweetfips202.html + */ + +/* Based on the CC0 implementation from @[mupq] and the public domain + * implementation @[supercop, crypto_hash/keccakc512/simple/] + * by Ronny Van Keer, and the public domain @[tweetfips] implementation. */ + +#include + +#include "../common.h" +#include "../ct.h" +#include "fips202.h" +#include "keccakf1600.h" +#if !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/** + * Initializes the Keccak state. + * + * @param[out] s Pointer to Keccak state. + */ +static void keccak_init(uint64_t s[MLD_KECCAK_LANES]) +__contract__( + requires(memory_no_alias(s, sizeof(uint64_t) * MLD_KECCAK_LANES)) + assigns(memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES)) +) +{ + mld_memset(s, 0, sizeof(uint64_t) * MLD_KECCAK_LANES); +} + +/** + * Absorb step of Keccak; incremental. + * + * @param[in,out] s Pointer to Keccak state. + * @param pos Position in current block to be absorbed. + * @param r Rate in bytes (e.g., 168 for SHAKE128). + * @param[in] in Pointer to input to be absorbed into s. + * @param inlen Length of input in bytes. + * + * @return New position pos in current block. + */ +static unsigned int keccak_absorb(uint64_t s[MLD_KECCAK_LANES], + unsigned int pos, unsigned int r, + const uint8_t *in, size_t inlen) +__contract__( + requires(inlen <= MLD_MAX_BUFFER_SIZE) + requires(r > 0) + requires(r < sizeof(uint64_t) * MLD_KECCAK_LANES) + requires(pos <= r) + requires(memory_no_alias(s, sizeof(uint64_t) * MLD_KECCAK_LANES)) + requires(memory_no_alias(in, inlen)) + assigns(memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES)) + ensures(return_value < r)) +{ + while (inlen >= r - pos) + __loop__( + assigns(pos, in, inlen, + memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES)) + invariant(inlen <= loop_entry(inlen)) + invariant(pos <= r) + invariant(in == loop_entry(in) + (loop_entry(inlen) - inlen)) + decreases(inlen + pos)) + { + mld_keccakf1600_xor_bytes(s, in, pos, r - pos); + inlen -= r - pos; + in += r - pos; + mld_keccakf1600_permute(s); + pos = 0; + } + /* Safety: At this point, inlen < r, so the truncation to unsigned is safe. */ + mld_keccakf1600_xor_bytes(s, in, pos, (unsigned)inlen); + + /* Safety: At this point, inlen < r and pos <= r so the truncation to unsigned + * is safe. */ + return (unsigned)(pos + inlen); +} + +/** + * Finalize absorb step. + * + * @param[in,out] s Pointer to Keccak state. + * @param pos Position in current block to be absorbed. + * @param r Rate in bytes (e.g., 168 for SHAKE128). + * @param p Domain separation byte. + */ +static void keccak_finalize(uint64_t s[MLD_KECCAK_LANES], unsigned int pos, + unsigned int r, uint8_t p) +__contract__( + requires(pos <= r && r < sizeof(uint64_t) * MLD_KECCAK_LANES) + requires((r / 8) >= 1) + requires(memory_no_alias(s, sizeof(uint64_t) * MLD_KECCAK_LANES)) + assigns(memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES)) +) +{ + uint8_t b = 0x80; + mld_keccakf1600_xor_bytes(s, &p, pos, 1); + mld_keccakf1600_xor_bytes(s, &b, r - 1, 1); +} + +/** + * Squeeze step of Keccak. Squeezes arbitrarily many bytes. Modifies the + * state. Can be called multiple times to keep squeezing, i.e., is + * incremental. + * + * @param[out] out Pointer to output data. + * @param outlen Number of bytes to be squeezed (written to out). + * @param[in,out] s Pointer to input/output Keccak state. + * @param pos Number of bytes in current block already squeezed. + * @param r Rate in bytes (e.g., 168 for SHAKE128). + * + * @return New position pos in current block. + */ +static unsigned int keccak_squeeze(uint8_t *out, size_t outlen, + uint64_t s[MLD_KECCAK_LANES], + unsigned int pos, unsigned int r) +__contract__( + requires((r == SHAKE128_RATE && pos <= SHAKE128_RATE) || + (r == SHAKE256_RATE && pos <= SHAKE256_RATE) || + (r == SHA3_512_RATE && pos <= SHA3_512_RATE)) + requires(outlen <= 8 * r /* somewhat arbitrary bound */) + requires(memory_no_alias(s, sizeof(uint64_t) * MLD_KECCAK_LANES)) + requires(memory_no_alias(out, outlen)) + assigns(memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES)) + assigns(memory_slice(out, outlen)) + ensures(return_value <= r)) +{ + unsigned int i; + size_t out_offset = 0; + + /* Reference: This code is re-factored from the reference implementation + * to facilitate proof with CBMC and to improve readability. + * + * Take a mutable copy of outlen to count down the number of bytes + * still to squeeze. The initial value of outlen is needed for the CBMC + * assigns() clauses. */ + size_t bytes_to_go = outlen; + + while (bytes_to_go > 0) + __loop__( + assigns(i, bytes_to_go, pos, out_offset, memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES), memory_slice(out, outlen)) + invariant(bytes_to_go <= outlen) + invariant(out_offset == outlen - bytes_to_go) + invariant(pos <= r) + decreases(bytes_to_go) + ) + { + if (pos == r) + { + mld_keccakf1600_permute(s); + pos = 0; + } + /* Safety: If bytes_to_go < r - pos, truncation to unsigned is safe. */ + i = bytes_to_go < r - pos ? (unsigned)bytes_to_go : r - pos; + mld_keccakf1600_extract_bytes(s, out + out_offset, pos, i); + bytes_to_go -= i; + pos += i; + out_offset += i; + } + + return pos; +} + +MLD_INTERNAL_API +void mld_shake128_init(mld_shake128ctx *state) +{ + keccak_init(state->s); + state->pos = 0; +} + +MLD_INTERNAL_API +void mld_shake128_absorb(mld_shake128ctx *state, const uint8_t *in, + size_t inlen) +{ + state->pos = keccak_absorb(state->s, state->pos, SHAKE128_RATE, in, inlen); +} + +MLD_INTERNAL_API +void mld_shake128_finalize(mld_shake128ctx *state) +{ + keccak_finalize(state->s, state->pos, SHAKE128_RATE, 0x1F); + state->pos = SHAKE128_RATE; +} + +MLD_INTERNAL_API +void mld_shake128_squeeze(uint8_t *out, size_t outlen, mld_shake128ctx *state) +{ + state->pos = keccak_squeeze(out, outlen, state->s, state->pos, SHAKE128_RATE); +} + +MLD_INTERNAL_API +void mld_shake128_release(mld_shake128ctx *state) +{ + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(state, sizeof(mld_shake128ctx)); +} + +MLD_INTERNAL_API +void mld_shake256_init(mld_shake256ctx *state) +{ + keccak_init(state->s); + state->pos = 0; +} + +MLD_INTERNAL_API +void mld_shake256_absorb(mld_shake256ctx *state, const uint8_t *in, + size_t inlen) +{ + state->pos = keccak_absorb(state->s, state->pos, SHAKE256_RATE, in, inlen); +} + +MLD_INTERNAL_API +void mld_shake256_finalize(mld_shake256ctx *state) +{ + keccak_finalize(state->s, state->pos, SHAKE256_RATE, 0x1F); + state->pos = SHAKE256_RATE; +} + +MLD_INTERNAL_API +void mld_shake256_squeeze(uint8_t *out, size_t outlen, mld_shake256ctx *state) +{ + state->pos = keccak_squeeze(out, outlen, state->s, state->pos, SHAKE256_RATE); +} + +MLD_INTERNAL_API +void mld_shake256_release(mld_shake256ctx *state) +{ + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(state, sizeof(mld_shake256ctx)); +} + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_CORE_API_ONLY) +MLD_INTERNAL_API +void mld_shake256(uint8_t *out, size_t outlen, const uint8_t *in, size_t inlen) +{ + mld_shake256ctx state; + + mld_shake256_init(&state); + mld_shake256_absorb(&state, in, inlen); + mld_shake256_finalize(&state); + mld_shake256_squeeze(out, outlen, &state); + mld_shake256_release(&state); +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_CORE_API_ONLY */ + +#endif /* !MLD_CONFIG_MULTILEVEL_NO_SHARED */ diff --git a/cbits/mldsa/src/fips202/fips202.h b/cbits/mldsa/src/fips202/fips202.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/fips202.h @@ -0,0 +1,224 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLD_FIPS202_FIPS202_H +#define MLD_FIPS202_FIPS202_H + +#include +#include "../cbmc.h" +#include "../common.h" + +#define SHAKE128_RATE 168 +#define SHAKE256_RATE 136 +#define SHA3_256_RATE 136 +#define SHA3_512_RATE 72 +#define MLD_KECCAK_LANES 25 +#define SHA3_256_HASHBYTES 32 +#define SHA3_512_HASHBYTES 64 + + +/** Context for the incremental SHAKE128 XOF. */ +typedef struct +{ + uint64_t s[MLD_KECCAK_LANES]; /**< Keccak state. */ + unsigned int pos; /**< Byte position within the current Keccak block. */ +} mld_shake128ctx; + +/** Context for the incremental SHAKE256 XOF. */ +typedef struct +{ + uint64_t s[MLD_KECCAK_LANES]; /**< Keccak state. */ + unsigned int pos; /**< Byte position within the current Keccak block. */ +} mld_shake256ctx; + +#define mld_shake128_init MLD_NAMESPACE(shake128_init) +/** + * Initializes state for use as SHAKE128 XOF. + * + * @param[out] state Pointer to (uninitialized) state. + */ +MLD_INTERNAL_API +void mld_shake128_init(mld_shake128ctx *state) +__contract__( + requires(memory_no_alias(state, sizeof(mld_shake128ctx))) + assigns(memory_slice(state, sizeof(mld_shake128ctx))) + ensures(state->pos == 0) +); + +#define mld_shake128_absorb MLD_NAMESPACE(shake128_absorb) +/** + * Absorb step of the SHAKE128 XOF. Absorbs arbitrarily many bytes. Can be + * called multiple times to absorb multiple chunks of data. + * + * @param[in,out] state Pointer to (initialized) output state. + * @param[in] in Pointer to input to be absorbed into s. + * @param inlen Length of input in bytes. + */ +MLD_INTERNAL_API +void mld_shake128_absorb(mld_shake128ctx *state, const uint8_t *in, + size_t inlen) +__contract__( + requires(inlen <= MLD_MAX_BUFFER_SIZE) + requires(memory_no_alias(state, sizeof(mld_shake128ctx))) + requires(memory_no_alias(in, inlen)) + requires(state->pos <= SHAKE128_RATE) + assigns(memory_slice(state, sizeof(mld_shake128ctx))) + ensures(state->pos <= SHAKE128_RATE) +); + +#define mld_shake128_finalize MLD_NAMESPACE(shake128_finalize) +/** + * Concludes the absorb phase of the SHAKE128 XOF. + * + * @param[in,out] state Pointer to state. + */ +MLD_INTERNAL_API +void mld_shake128_finalize(mld_shake128ctx *state) +__contract__( + requires(memory_no_alias(state, sizeof(mld_shake128ctx))) + requires(state->pos <= SHAKE128_RATE) + assigns(memory_slice(state, sizeof(mld_shake128ctx))) + ensures(state->pos <= SHAKE128_RATE) +); + +#define mld_shake128_squeeze MLD_NAMESPACE(shake128_squeeze) +/** + * Squeeze step of SHAKE128 XOF. Squeezes arbitrarily many bytes. Can be + * called multiple times to keep squeezing. + * + * @param[out] out Pointer to output blocks. + * @param outlen Number of bytes to be squeezed (written to output). + * @param[in,out] state Pointer to input/output state. + */ +MLD_INTERNAL_API +void mld_shake128_squeeze(uint8_t *out, size_t outlen, mld_shake128ctx *state) +__contract__( + requires(outlen <= 8 * SHAKE128_RATE /* somewhat arbitrary bound */) + requires(memory_no_alias(state, sizeof(mld_shake128ctx))) + requires(memory_no_alias(out, outlen)) + requires(state->pos <= SHAKE128_RATE) + assigns(memory_slice(state, sizeof(mld_shake128ctx))) + assigns(memory_slice(out, outlen)) + ensures(state->pos <= SHAKE128_RATE) +); + +#define mld_shake128_release MLD_NAMESPACE(shake128_release) +/** + * Release and securely zero the SHAKE128 state. + * + * @param[in,out] state Pointer to state. + */ +MLD_INTERNAL_API +void mld_shake128_release(mld_shake128ctx *state) +__contract__( + requires(memory_no_alias(state, sizeof(mld_shake128ctx))) + assigns(memory_slice(state, sizeof(mld_shake128ctx))) +); + +#define mld_shake256_init MLD_NAMESPACE(shake256_init) +/** + * Initializes state for use as SHAKE256 XOF. + * + * @param[out] state Pointer to (uninitialized) state. + */ +MLD_INTERNAL_API +void mld_shake256_init(mld_shake256ctx *state) +__contract__( + requires(memory_no_alias(state, sizeof(mld_shake256ctx))) + assigns(memory_slice(state, sizeof(mld_shake256ctx))) + ensures(state->pos == 0) +); + +#define mld_shake256_absorb MLD_NAMESPACE(shake256_absorb) +/** + * Absorb step of the SHAKE256 XOF. Absorbs arbitrarily many bytes. Can be + * called multiple times to absorb multiple chunks of data. + * + * @param[in,out] state Pointer to (initialized) output state. + * @param[in] in Pointer to input to be absorbed into s. + * @param inlen Length of input in bytes. + */ +MLD_INTERNAL_API +void mld_shake256_absorb(mld_shake256ctx *state, const uint8_t *in, + size_t inlen) +__contract__( + requires(inlen <= MLD_MAX_BUFFER_SIZE) + requires(memory_no_alias(state, sizeof(mld_shake256ctx))) + requires(memory_no_alias(in, inlen)) + requires(state->pos <= SHAKE256_RATE) + assigns(memory_slice(state, sizeof(mld_shake256ctx))) + ensures(state->pos <= SHAKE256_RATE) +); + +#define mld_shake256_finalize MLD_NAMESPACE(shake256_finalize) +/** + * Concludes the absorb phase of the SHAKE256 XOF. + * + * @param[in,out] state Pointer to state. + */ +MLD_INTERNAL_API +void mld_shake256_finalize(mld_shake256ctx *state) +__contract__( + requires(memory_no_alias(state, sizeof(mld_shake256ctx))) + requires(state->pos <= SHAKE256_RATE) + assigns(memory_slice(state, sizeof(mld_shake256ctx))) + ensures(state->pos <= SHAKE256_RATE) +); + +#define mld_shake256_squeeze MLD_NAMESPACE(shake256_squeeze) +/** + * Squeeze step of SHAKE256 XOF. Squeezes arbitrarily many bytes. Can be + * called multiple times to keep squeezing. + * + * @param[out] out Pointer to output blocks. + * @param outlen Number of bytes to be squeezed (written to output). + * @param[in,out] state Pointer to input/output state. + */ +MLD_INTERNAL_API +void mld_shake256_squeeze(uint8_t *out, size_t outlen, mld_shake256ctx *state) +__contract__( + requires(outlen <= 8 * SHAKE256_RATE /* somewhat arbitrary bound */) + requires(memory_no_alias(state, sizeof(mld_shake256ctx))) + requires(memory_no_alias(out, outlen)) + requires(state->pos <= SHAKE256_RATE) + assigns(memory_slice(state, sizeof(mld_shake256ctx))) + assigns(memory_slice(out, outlen)) + ensures(state->pos <= SHAKE256_RATE) +); + +#define mld_shake256_release MLD_NAMESPACE(shake256_release) +/** + * Release and securely zero the SHAKE256 state. + * + * @param[in,out] state Pointer to state. + */ +MLD_INTERNAL_API +void mld_shake256_release(mld_shake256ctx *state) +__contract__( + requires(memory_no_alias(state, sizeof(mld_shake256ctx))) + assigns(memory_slice(state, sizeof(mld_shake256ctx))) +); + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_CORE_API_ONLY) +#define mld_shake256 MLD_NAMESPACE(shake256) +/** + * SHAKE256 XOF with non-incremental API. + * + * @param[out] out Pointer to output. + * @param outlen Requested output length in bytes. + * @param[in] in Pointer to input. + * @param inlen Length of input in bytes. + */ +MLD_INTERNAL_API +void mld_shake256(uint8_t *out, size_t outlen, const uint8_t *in, size_t inlen) +__contract__( + requires(inlen <= MLD_MAX_BUFFER_SIZE) + requires(outlen <= 8 * SHAKE256_RATE /* somewhat arbitrary bound */) + requires(memory_no_alias(in, inlen)) + requires(memory_no_alias(out, outlen)) + assigns(memory_slice(out, outlen)) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_CORE_API_ONLY */ + +#endif /* !MLD_FIPS202_FIPS202_H */ diff --git a/cbits/mldsa/src/fips202/fips202x4.c b/cbits/mldsa/src/fips202/fips202x4.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/fips202x4.c @@ -0,0 +1,187 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + */ + +#include "../common.h" +#if !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) + +#include "../ct.h" +#include "fips202.h" +#include "fips202x4.h" +#include "keccakf1600.h" + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_REDUCE_RAM) || \ + defined(MLD_UNIT_TEST) +static void mld_keccak_absorb_once_x4(uint64_t *s, unsigned r, + const uint8_t *in0, const uint8_t *in1, + const uint8_t *in2, const uint8_t *in3, + size_t inlen, uint8_t p) +__contract__( + requires(inlen <= MLD_MAX_BUFFER_SIZE) + requires(memory_no_alias(s, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY)) + requires(r > 0) + requires(r <= sizeof(uint64_t) * MLD_KECCAK_LANES) + requires(memory_no_alias(in0, inlen)) + requires(memory_no_alias(in1, inlen)) + requires(memory_no_alias(in2, inlen)) + requires(memory_no_alias(in3, inlen)) + assigns(memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY))) +{ + while (inlen >= r) + __loop__( + assigns(inlen, in0, in1, in2, in3, memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY)) + invariant(inlen <= loop_entry(inlen)) + invariant(in0 == loop_entry(in0) + (loop_entry(inlen) - inlen)) + invariant(in1 == loop_entry(in1) + (loop_entry(inlen) - inlen)) + invariant(in2 == loop_entry(in2) + (loop_entry(inlen) - inlen)) + invariant(in3 == loop_entry(in3) + (loop_entry(inlen) - inlen)) + decreases(inlen)) + { + mld_keccakf1600x4_xor_bytes(s, in0, in1, in2, in3, 0, r); + mld_keccakf1600x4_permute(s); + + in0 += r; + in1 += r; + in2 += r; + in3 += r; + inlen -= r; + } + + /* Safety: At this point, inlen < r, so the truncations to unsigned are safe + * below. */ + if (inlen > 0) + { + mld_keccakf1600x4_xor_bytes(s, in0, in1, in2, in3, 0, (unsigned)inlen); + } + + if (inlen == r - 1) + { + p |= 128; + mld_keccakf1600x4_xor_bytes(s, &p, &p, &p, &p, (unsigned)inlen, 1); + } + else + { + mld_keccakf1600x4_xor_bytes(s, &p, &p, &p, &p, (unsigned)inlen, 1); + p = 128; + mld_keccakf1600x4_xor_bytes(s, &p, &p, &p, &p, r - 1, 1); + } +} + +static void mld_keccak_squeezeblocks_x4(uint8_t *out0, uint8_t *out1, + uint8_t *out2, uint8_t *out3, + size_t nblocks, uint64_t *s, unsigned r) +__contract__( + requires(r <= sizeof(uint64_t) * MLD_KECCAK_LANES) + requires(nblocks <= 8 /* somewhat arbitrary bound */) + requires(memory_no_alias(s, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY)) + requires(memory_no_alias(out0, nblocks * r)) + requires(memory_no_alias(out1, nblocks * r)) + requires(memory_no_alias(out2, nblocks * r)) + requires(memory_no_alias(out3, nblocks * r)) + assigns(memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY)) + assigns(memory_slice(out0, nblocks * r)) + assigns(memory_slice(out1, nblocks * r)) + assigns(memory_slice(out2, nblocks * r)) + assigns(memory_slice(out3, nblocks * r))) +{ + size_t current_offset = 0; + while (nblocks > 0) + __loop__( + assigns(nblocks, current_offset, + memory_slice(s, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY), + memory_slice(out0, nblocks * r), + memory_slice(out1, nblocks * r), + memory_slice(out2, nblocks * r), + memory_slice(out3, nblocks * r)) + invariant(nblocks <= loop_entry(nblocks)) + invariant(current_offset == (loop_entry(nblocks) - nblocks) * r) + decreases(nblocks)) + { + mld_keccakf1600x4_permute(s); + mld_keccakf1600x4_extract_bytes( + s, &out0[current_offset], &out1[current_offset], &out2[current_offset], + &out3[current_offset], 0, r); + current_offset += r; + nblocks--; + } +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_REDUCE_RAM || \ + MLD_UNIT_TEST */ + +#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +MLD_INTERNAL_API +void mld_shake128x4_absorb_once(mld_shake128x4ctx *state, const uint8_t *in0, + const uint8_t *in1, const uint8_t *in2, + const uint8_t *in3, size_t inlen) +{ + mld_memset(state, 0, sizeof(mld_shake128x4ctx)); + mld_keccak_absorb_once_x4(state->ctx, SHAKE128_RATE, in0, in1, in2, in3, + inlen, 0x1F); +} + +MLD_INTERNAL_API +void mld_shake128x4_squeezeblocks(uint8_t *out0, uint8_t *out1, uint8_t *out2, + uint8_t *out3, size_t nblocks, + mld_shake128x4ctx *state) +{ + mld_keccak_squeezeblocks_x4(out0, out1, out2, out3, nblocks, state->ctx, + SHAKE128_RATE); +} + +MLD_INTERNAL_API +void mld_shake128x4_init(mld_shake128x4ctx *state) { (void)state; } +MLD_INTERNAL_API +void mld_shake128x4_release(mld_shake128x4ctx *state) +{ + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(state, sizeof(mld_shake128x4ctx)); +} +#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || \ + (!defined(MLD_CONFIG_NO_SIGN_API) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))) +MLD_INTERNAL_API +void mld_shake256x4_absorb_once(mld_shake256x4ctx *state, const uint8_t *in0, + const uint8_t *in1, const uint8_t *in2, + const uint8_t *in3, size_t inlen) +{ + mld_memset(state, 0, sizeof(mld_shake256x4ctx)); + mld_keccak_absorb_once_x4(state->ctx, SHAKE256_RATE, in0, in1, in2, in3, + inlen, 0x1F); +} + +MLD_INTERNAL_API +void mld_shake256x4_squeezeblocks(uint8_t *out0, uint8_t *out1, uint8_t *out2, + uint8_t *out3, size_t nblocks, + mld_shake256x4ctx *state) +{ + mld_keccak_squeezeblocks_x4(out0, out1, out2, out3, nblocks, state->ctx, + SHAKE256_RATE); +} + +MLD_INTERNAL_API +void mld_shake256x4_init(mld_shake256x4ctx *state) { (void)state; } +MLD_INTERNAL_API +void mld_shake256x4_release(mld_shake256x4ctx *state) +{ + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(state, sizeof(mld_shake256x4ctx)); +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || (!MLD_CONFIG_NO_SIGN_API && \ + (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST)) */ + +#endif /* !MLD_CONFIG_MULTILEVEL_NO_SHARED && !MLD_CONFIG_SERIAL_FIPS202_ONLY \ + */ diff --git a/cbits/mldsa/src/fips202/fips202x4.h b/cbits/mldsa/src/fips202/fips202x4.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/fips202x4.h @@ -0,0 +1,125 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLD_FIPS202_FIPS202X4_H +#define MLD_FIPS202_FIPS202X4_H + +#include "../common.h" + +#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) + +#include + +#include "../cbmc.h" +#include "fips202.h" +#include "keccakf1600.h" + +/** Context for the non-incremental 4-way SHAKE128 API. */ +typedef struct +{ + uint64_t ctx[MLD_KECCAK_LANES * + MLD_KECCAK_WAY]; /**< 4-way Keccak state, stored sequentially. */ +} mld_shake128x4ctx; + +/** Context for the 4-way batched SHAKE256 XOF. */ +typedef struct +{ + uint64_t ctx[MLD_KECCAK_LANES * + MLD_KECCAK_WAY]; /**< Interleaved 4-way Keccak state. */ +} mld_shake256x4ctx; + +#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +#define mld_shake128x4_absorb_once MLD_NAMESPACE(shake128x4_absorb_once) +MLD_INTERNAL_API +void mld_shake128x4_absorb_once(mld_shake128x4ctx *state, const uint8_t *in0, + const uint8_t *in1, const uint8_t *in2, + const uint8_t *in3, size_t inlen) +__contract__( + requires(inlen <= MLD_MAX_BUFFER_SIZE) + requires(memory_no_alias(state, sizeof(mld_shake128x4ctx))) + requires(memory_no_alias(in0, inlen)) + requires(memory_no_alias(in1, inlen)) + requires(memory_no_alias(in2, inlen)) + requires(memory_no_alias(in3, inlen)) + assigns(memory_slice(state, sizeof(mld_shake128x4ctx))) +); + +#define mld_shake128x4_squeezeblocks MLD_NAMESPACE(shake128x4_squeezeblocks) +MLD_INTERNAL_API +void mld_shake128x4_squeezeblocks(uint8_t *out0, uint8_t *out1, uint8_t *out2, + uint8_t *out3, size_t nblocks, + mld_shake128x4ctx *state) +__contract__( + requires(nblocks <= 8 /* somewhat arbitrary bound */) + requires(memory_no_alias(state, sizeof(mld_shake128x4ctx))) + requires(memory_no_alias(out0, nblocks * SHAKE128_RATE)) + requires(memory_no_alias(out1, nblocks * SHAKE128_RATE)) + requires(memory_no_alias(out2, nblocks * SHAKE128_RATE)) + requires(memory_no_alias(out3, nblocks * SHAKE128_RATE)) + assigns(memory_slice(out0, nblocks * SHAKE128_RATE), + memory_slice(out1, nblocks * SHAKE128_RATE), + memory_slice(out2, nblocks * SHAKE128_RATE), + memory_slice(out3, nblocks * SHAKE128_RATE), + memory_slice(state, sizeof(mld_shake128x4ctx))) +); + +#define mld_shake128x4_init MLD_NAMESPACE(shake128x4_init) +MLD_INTERNAL_API +void mld_shake128x4_init(mld_shake128x4ctx *state); + +#define mld_shake128x4_release MLD_NAMESPACE(shake128x4_release) +MLD_INTERNAL_API +void mld_shake128x4_release(mld_shake128x4ctx *state); +#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || \ + (!defined(MLD_CONFIG_NO_SIGN_API) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))) +#define mld_shake256x4_absorb_once MLD_NAMESPACE(shake256x4_absorb_once) +MLD_INTERNAL_API +void mld_shake256x4_absorb_once(mld_shake256x4ctx *state, const uint8_t *in0, + const uint8_t *in1, const uint8_t *in2, + const uint8_t *in3, size_t inlen) +__contract__( + requires(inlen <= MLD_MAX_BUFFER_SIZE) + requires(memory_no_alias(state, sizeof(mld_shake256x4ctx))) + requires(memory_no_alias(in0, inlen)) + requires(memory_no_alias(in1, inlen)) + requires(memory_no_alias(in2, inlen)) + requires(memory_no_alias(in3, inlen)) + assigns(memory_slice(state, sizeof(mld_shake256x4ctx))) +); + +#define mld_shake256x4_squeezeblocks MLD_NAMESPACE(shake256x4_squeezeblocks) +MLD_INTERNAL_API +void mld_shake256x4_squeezeblocks(uint8_t *out0, uint8_t *out1, uint8_t *out2, + uint8_t *out3, size_t nblocks, + mld_shake256x4ctx *state) +__contract__( + requires(nblocks <= 8 /* somewhat arbitrary bound */) + requires(memory_no_alias(state, sizeof(mld_shake256x4ctx))) + requires(memory_no_alias(out0, nblocks * SHAKE256_RATE)) + requires(memory_no_alias(out1, nblocks * SHAKE256_RATE)) + requires(memory_no_alias(out2, nblocks * SHAKE256_RATE)) + requires(memory_no_alias(out3, nblocks * SHAKE256_RATE)) + assigns(memory_slice(out0, nblocks * SHAKE256_RATE), + memory_slice(out1, nblocks * SHAKE256_RATE), + memory_slice(out2, nblocks * SHAKE256_RATE), + memory_slice(out3, nblocks * SHAKE256_RATE), + memory_slice(state, sizeof(mld_shake256x4ctx))) +); + +#define mld_shake256x4_init MLD_NAMESPACE(shake256x4_init) +MLD_INTERNAL_API +void mld_shake256x4_init(mld_shake256x4ctx *state); + +#define mld_shake256x4_release MLD_NAMESPACE(shake256x4_release) +MLD_INTERNAL_API +void mld_shake256x4_release(mld_shake256x4ctx *state); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || (!MLD_CONFIG_NO_SIGN_API && \ + (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST)) */ + +#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY */ +#endif /* !MLD_FIPS202_FIPS202X4_H */ diff --git a/cbits/mldsa/src/fips202/keccakf1600.c b/cbits/mldsa/src/fips202/keccakf1600.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/keccakf1600.c @@ -0,0 +1,510 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [mupq] + * Common files for pqm4, pqm3, pqriscv + * Kannwischer, Petri, Rijneveld, Schwabe, Stoffelen + * https://github.com/mupq/mupq + * + * - [supercop] + * SUPERCOP benchmarking framework + * Daniel J. Bernstein + * http://bench.cr.yp.to/supercop.html + * + * - [tweetfips] + * 'tweetfips202' FIPS202 implementation + * Van Assche, Bernstein, Schwabe + * https://keccak.team/2015/tweetfips202.html + */ + +/* Based on the CC0 implementation from @[mupq] and the public domain + * implementation @[supercop, crypto_hash/keccakc512/simple/] + * by Ronny Van Keer, and the public domain @[tweetfips] implementation. */ + +#include "keccakf1600.h" +#if !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +#define MLD_KECCAK_NROUNDS 24 +#define MLD_KECCAK_ROL(a, offset) (((a) << (offset)) ^ ((a) >> (64 - (offset)))) + +MLD_INTERNAL_API +void mld_keccakf1600_extract_bytes(uint64_t *state, unsigned char *data, + unsigned offset, unsigned length) +{ + unsigned i; +#if defined(MLD_SYS_LITTLE_ENDIAN) + uint8_t *state_ptr = (uint8_t *)state + offset; + for (i = 0; i < length; i++) + __loop__(invariant(i <= length) + decreases(length - i)) + { + data[i] = state_ptr[i]; + } +#else /* MLD_SYS_LITTLE_ENDIAN */ + /* Portable version */ + for (i = 0; i < length; i++) + __loop__(invariant(i <= length) + decreases(length - i)) + { + data[i] = (state[(offset + i) >> 3] >> (8 * ((offset + i) & 0x07))) & 0xFF; + } +#endif /* !MLD_SYS_LITTLE_ENDIAN */ +} + +MLD_INTERNAL_API +void mld_keccakf1600_xor_bytes(uint64_t *state, const unsigned char *data, + unsigned offset, unsigned length) +{ + unsigned i; +#if defined(MLD_SYS_LITTLE_ENDIAN) + uint8_t *state_ptr = (uint8_t *)state + offset; + for (i = 0; i < length; i++) + __loop__(invariant(i <= length) + decreases(length - i)) + { + state_ptr[i] ^= data[i]; + } +#else /* MLD_SYS_LITTLE_ENDIAN */ + /* Portable version */ + for (i = 0; i < length; i++) + __loop__(invariant(i <= length) + decreases(length - i)) + { + state[(offset + i) >> 3] ^= (uint64_t)data[i] + << (8 * ((offset + i) & 0x07)); + } +#endif /* !MLD_SYS_LITTLE_ENDIAN */ +} + +#if (!defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_REDUCE_RAM) || \ + defined(MLD_UNIT_TEST)) && \ + !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) +static void mld_keccakf1600x4_extract_bytes_c(uint64_t *state, + unsigned char *data0, + unsigned char *data1, + unsigned char *data2, + unsigned char *data3, + unsigned offset, unsigned length) +__contract__( + requires(0 <= offset && offset <= MLD_KECCAK_LANES * sizeof(uint64_t) && + 0 <= length && length <= MLD_KECCAK_LANES * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY)) + requires(memory_no_alias(data0, length)) + requires(memory_no_alias(data1, length)) + requires(memory_no_alias(data2, length)) + requires(memory_no_alias(data3, length)) + assigns(memory_slice(data0, length)) + assigns(memory_slice(data1, length)) + assigns(memory_slice(data2, length)) + assigns(memory_slice(data3, length)) +) +{ + mld_keccakf1600_extract_bytes(state + MLD_KECCAK_LANES * 0, data0, offset, + length); + mld_keccakf1600_extract_bytes(state + MLD_KECCAK_LANES * 1, data1, offset, + length); + mld_keccakf1600_extract_bytes(state + MLD_KECCAK_LANES * 2, data2, offset, + length); + mld_keccakf1600_extract_bytes(state + MLD_KECCAK_LANES * 3, data3, offset, + length); +} + +MLD_INTERNAL_API +void mld_keccakf1600x4_extract_bytes(uint64_t *state, unsigned char *data0, + unsigned char *data1, unsigned char *data2, + unsigned char *data3, unsigned offset, + unsigned length) +{ +#if defined(MLD_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES) + if (mld_keccakf1600_extract_bytes_x4_native(state, data0, data1, data2, data3, + offset, length) == + MLD_NATIVE_FUNC_SUCCESS) + { + return; + } +#endif /* MLD_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES */ + mld_keccakf1600x4_extract_bytes_c(state, data0, data1, data2, data3, offset, + length); +} + +static void mld_keccakf1600x4_xor_bytes_c(uint64_t *state, + const unsigned char *data0, + const unsigned char *data1, + const unsigned char *data2, + const unsigned char *data3, + unsigned offset, unsigned length) +__contract__( + requires(0 <= offset && offset <= MLD_KECCAK_LANES * sizeof(uint64_t) && + 0 <= length && length <= MLD_KECCAK_LANES * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY)) + requires(memory_no_alias(data0, length)) + /* Case 1: all input buffers are distinct; Case 2: All input buffers are the same */ + requires((data0 == data1 && + data0 == data2 && + data0 == data3) || + (memory_no_alias(data1, length) && + memory_no_alias(data2, length) && + memory_no_alias(data3, length))) + assigns(memory_slice(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY)) +) +{ + mld_keccakf1600_xor_bytes(state + MLD_KECCAK_LANES * 0, data0, offset, + length); + mld_keccakf1600_xor_bytes(state + MLD_KECCAK_LANES * 1, data1, offset, + length); + mld_keccakf1600_xor_bytes(state + MLD_KECCAK_LANES * 2, data2, offset, + length); + mld_keccakf1600_xor_bytes(state + MLD_KECCAK_LANES * 3, data3, offset, + length); +} + +MLD_INTERNAL_API +void mld_keccakf1600x4_xor_bytes(uint64_t *state, const unsigned char *data0, + const unsigned char *data1, + const unsigned char *data2, + const unsigned char *data3, unsigned offset, + unsigned length) +{ +#if defined(MLD_USE_NATIVE_FIPS202_X4_XOR_BYTES) + if (mld_keccakf1600_xor_bytes_x4_native(state, data0, data1, data2, data3, + offset, + length) == MLD_NATIVE_FUNC_SUCCESS) + { + return; + } +#endif /* MLD_USE_NATIVE_FIPS202_X4_XOR_BYTES */ + mld_keccakf1600x4_xor_bytes_c(state, data0, data1, data2, data3, offset, + length); +} + +MLD_INTERNAL_API +void mld_keccakf1600x4_permute(uint64_t *state) +{ +#if defined(MLD_USE_NATIVE_FIPS202_X4) + if (mld_keccak_f1600_x4_native(state) == MLD_NATIVE_FUNC_SUCCESS) + { + return; + } +#endif /* MLD_USE_NATIVE_FIPS202_X4 */ + mld_keccakf1600_permute(state + MLD_KECCAK_LANES * 0); + mld_keccakf1600_permute(state + MLD_KECCAK_LANES * 1); + mld_keccakf1600_permute(state + MLD_KECCAK_LANES * 2); + mld_keccakf1600_permute(state + MLD_KECCAK_LANES * 3); +} +#endif /* (!MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_REDUCE_RAM || \ + MLD_UNIT_TEST) && !MLD_CONFIG_SERIAL_FIPS202_ONLY */ + +static const uint64_t mld_KeccakF_RoundConstants[MLD_KECCAK_NROUNDS] = { + (uint64_t)0x0000000000000001ULL, (uint64_t)0x0000000000008082ULL, + (uint64_t)0x800000000000808aULL, (uint64_t)0x8000000080008000ULL, + (uint64_t)0x000000000000808bULL, (uint64_t)0x0000000080000001ULL, + (uint64_t)0x8000000080008081ULL, (uint64_t)0x8000000000008009ULL, + (uint64_t)0x000000000000008aULL, (uint64_t)0x0000000000000088ULL, + (uint64_t)0x0000000080008009ULL, (uint64_t)0x000000008000000aULL, + (uint64_t)0x000000008000808bULL, (uint64_t)0x800000000000008bULL, + (uint64_t)0x8000000000008089ULL, (uint64_t)0x8000000000008003ULL, + (uint64_t)0x8000000000008002ULL, (uint64_t)0x8000000000000080ULL, + (uint64_t)0x000000000000800aULL, (uint64_t)0x800000008000000aULL, + (uint64_t)0x8000000080008081ULL, (uint64_t)0x8000000000008080ULL, + (uint64_t)0x0000000080000001ULL, (uint64_t)0x8000000080008008ULL}; + +MLD_STATIC_TESTABLE +void mld_keccakf1600_permute_c(uint64_t *state) +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES)) + assigns(memory_slice(state, sizeof(uint64_t) * MLD_KECCAK_LANES)) +) +{ + unsigned round; + + uint64_t Aba, Abe, Abi, Abo, Abu; + uint64_t Aga, Age, Agi, Ago, Agu; + uint64_t Aka, Ake, Aki, Ako, Aku; + uint64_t Ama, Ame, Ami, Amo, Amu; + uint64_t Asa, Ase, Asi, Aso, Asu; + uint64_t BCa, BCe, BCi, BCo, BCu; + uint64_t Da, De, Di, Do, Du; + uint64_t Eba, Ebe, Ebi, Ebo, Ebu; + uint64_t Ega, Ege, Egi, Ego, Egu; + uint64_t Eka, Eke, Eki, Eko, Eku; + uint64_t Ema, Eme, Emi, Emo, Emu; + uint64_t Esa, Ese, Esi, Eso, Esu; + + /* copyFromState(A, state) */ + Aba = state[0]; + Abe = state[1]; + Abi = state[2]; + Abo = state[3]; + Abu = state[4]; + Aga = state[5]; + Age = state[6]; + Agi = state[7]; + Ago = state[8]; + Agu = state[9]; + Aka = state[10]; + Ake = state[11]; + Aki = state[12]; + Ako = state[13]; + Aku = state[14]; + Ama = state[15]; + Ame = state[16]; + Ami = state[17]; + Amo = state[18]; + Amu = state[19]; + Asa = state[20]; + Ase = state[21]; + Asi = state[22]; + Aso = state[23]; + Asu = state[24]; + + for (round = 0; round < MLD_KECCAK_NROUNDS; round += 2) + __loop__(invariant(round <= MLD_KECCAK_NROUNDS && round % 2 == 0) + decreases(MLD_KECCAK_NROUNDS - round)) + { + /* prepareTheta */ + BCa = Aba ^ Aga ^ Aka ^ Ama ^ Asa; + BCe = Abe ^ Age ^ Ake ^ Ame ^ Ase; + BCi = Abi ^ Agi ^ Aki ^ Ami ^ Asi; + BCo = Abo ^ Ago ^ Ako ^ Amo ^ Aso; + BCu = Abu ^ Agu ^ Aku ^ Amu ^ Asu; + + /* thetaRhoPiChiIotaPrepareTheta(round, A, E) */ + Da = BCu ^ MLD_KECCAK_ROL(BCe, 1); + De = BCa ^ MLD_KECCAK_ROL(BCi, 1); + Di = BCe ^ MLD_KECCAK_ROL(BCo, 1); + Do = BCi ^ MLD_KECCAK_ROL(BCu, 1); + Du = BCo ^ MLD_KECCAK_ROL(BCa, 1); + + Aba ^= Da; + BCa = Aba; + Age ^= De; + BCe = MLD_KECCAK_ROL(Age, 44); + Aki ^= Di; + BCi = MLD_KECCAK_ROL(Aki, 43); + Amo ^= Do; + BCo = MLD_KECCAK_ROL(Amo, 21); + Asu ^= Du; + BCu = MLD_KECCAK_ROL(Asu, 14); + Eba = BCa ^ ((~BCe) & BCi); + Eba ^= (uint64_t)mld_KeccakF_RoundConstants[round]; + Ebe = BCe ^ ((~BCi) & BCo); + Ebi = BCi ^ ((~BCo) & BCu); + Ebo = BCo ^ ((~BCu) & BCa); + Ebu = BCu ^ ((~BCa) & BCe); + + Abo ^= Do; + BCa = MLD_KECCAK_ROL(Abo, 28); + Agu ^= Du; + BCe = MLD_KECCAK_ROL(Agu, 20); + Aka ^= Da; + BCi = MLD_KECCAK_ROL(Aka, 3); + Ame ^= De; + BCo = MLD_KECCAK_ROL(Ame, 45); + Asi ^= Di; + BCu = MLD_KECCAK_ROL(Asi, 61); + Ega = BCa ^ ((~BCe) & BCi); + Ege = BCe ^ ((~BCi) & BCo); + Egi = BCi ^ ((~BCo) & BCu); + Ego = BCo ^ ((~BCu) & BCa); + Egu = BCu ^ ((~BCa) & BCe); + + Abe ^= De; + BCa = MLD_KECCAK_ROL(Abe, 1); + Agi ^= Di; + BCe = MLD_KECCAK_ROL(Agi, 6); + Ako ^= Do; + BCi = MLD_KECCAK_ROL(Ako, 25); + Amu ^= Du; + BCo = MLD_KECCAK_ROL(Amu, 8); + Asa ^= Da; + BCu = MLD_KECCAK_ROL(Asa, 18); + Eka = BCa ^ ((~BCe) & BCi); + Eke = BCe ^ ((~BCi) & BCo); + Eki = BCi ^ ((~BCo) & BCu); + Eko = BCo ^ ((~BCu) & BCa); + Eku = BCu ^ ((~BCa) & BCe); + + Abu ^= Du; + BCa = MLD_KECCAK_ROL(Abu, 27); + Aga ^= Da; + BCe = MLD_KECCAK_ROL(Aga, 36); + Ake ^= De; + BCi = MLD_KECCAK_ROL(Ake, 10); + Ami ^= Di; + BCo = MLD_KECCAK_ROL(Ami, 15); + Aso ^= Do; + BCu = MLD_KECCAK_ROL(Aso, 56); + Ema = BCa ^ ((~BCe) & BCi); + Eme = BCe ^ ((~BCi) & BCo); + Emi = BCi ^ ((~BCo) & BCu); + Emo = BCo ^ ((~BCu) & BCa); + Emu = BCu ^ ((~BCa) & BCe); + + Abi ^= Di; + BCa = MLD_KECCAK_ROL(Abi, 62); + Ago ^= Do; + BCe = MLD_KECCAK_ROL(Ago, 55); + Aku ^= Du; + BCi = MLD_KECCAK_ROL(Aku, 39); + Ama ^= Da; + BCo = MLD_KECCAK_ROL(Ama, 41); + Ase ^= De; + BCu = MLD_KECCAK_ROL(Ase, 2); + Esa = BCa ^ ((~BCe) & BCi); + Ese = BCe ^ ((~BCi) & BCo); + Esi = BCi ^ ((~BCo) & BCu); + Eso = BCo ^ ((~BCu) & BCa); + Esu = BCu ^ ((~BCa) & BCe); + + /* prepareTheta */ + BCa = Eba ^ Ega ^ Eka ^ Ema ^ Esa; + BCe = Ebe ^ Ege ^ Eke ^ Eme ^ Ese; + BCi = Ebi ^ Egi ^ Eki ^ Emi ^ Esi; + BCo = Ebo ^ Ego ^ Eko ^ Emo ^ Eso; + BCu = Ebu ^ Egu ^ Eku ^ Emu ^ Esu; + + /* thetaRhoPiChiIotaPrepareTheta(round+1, E, A) */ + Da = BCu ^ MLD_KECCAK_ROL(BCe, 1); + De = BCa ^ MLD_KECCAK_ROL(BCi, 1); + Di = BCe ^ MLD_KECCAK_ROL(BCo, 1); + Do = BCi ^ MLD_KECCAK_ROL(BCu, 1); + Du = BCo ^ MLD_KECCAK_ROL(BCa, 1); + + Eba ^= Da; + BCa = Eba; + Ege ^= De; + BCe = MLD_KECCAK_ROL(Ege, 44); + Eki ^= Di; + BCi = MLD_KECCAK_ROL(Eki, 43); + Emo ^= Do; + BCo = MLD_KECCAK_ROL(Emo, 21); + Esu ^= Du; + BCu = MLD_KECCAK_ROL(Esu, 14); + Aba = BCa ^ ((~BCe) & BCi); + Aba ^= (uint64_t)mld_KeccakF_RoundConstants[round + 1]; + Abe = BCe ^ ((~BCi) & BCo); + Abi = BCi ^ ((~BCo) & BCu); + Abo = BCo ^ ((~BCu) & BCa); + Abu = BCu ^ ((~BCa) & BCe); + + Ebo ^= Do; + BCa = MLD_KECCAK_ROL(Ebo, 28); + Egu ^= Du; + BCe = MLD_KECCAK_ROL(Egu, 20); + Eka ^= Da; + BCi = MLD_KECCAK_ROL(Eka, 3); + Eme ^= De; + BCo = MLD_KECCAK_ROL(Eme, 45); + Esi ^= Di; + BCu = MLD_KECCAK_ROL(Esi, 61); + Aga = BCa ^ ((~BCe) & BCi); + Age = BCe ^ ((~BCi) & BCo); + Agi = BCi ^ ((~BCo) & BCu); + Ago = BCo ^ ((~BCu) & BCa); + Agu = BCu ^ ((~BCa) & BCe); + + Ebe ^= De; + BCa = MLD_KECCAK_ROL(Ebe, 1); + Egi ^= Di; + BCe = MLD_KECCAK_ROL(Egi, 6); + Eko ^= Do; + BCi = MLD_KECCAK_ROL(Eko, 25); + Emu ^= Du; + BCo = MLD_KECCAK_ROL(Emu, 8); + Esa ^= Da; + BCu = MLD_KECCAK_ROL(Esa, 18); + Aka = BCa ^ ((~BCe) & BCi); + Ake = BCe ^ ((~BCi) & BCo); + Aki = BCi ^ ((~BCo) & BCu); + Ako = BCo ^ ((~BCu) & BCa); + Aku = BCu ^ ((~BCa) & BCe); + + Ebu ^= Du; + BCa = MLD_KECCAK_ROL(Ebu, 27); + Ega ^= Da; + BCe = MLD_KECCAK_ROL(Ega, 36); + Eke ^= De; + BCi = MLD_KECCAK_ROL(Eke, 10); + Emi ^= Di; + BCo = MLD_KECCAK_ROL(Emi, 15); + Eso ^= Do; + BCu = MLD_KECCAK_ROL(Eso, 56); + Ama = BCa ^ ((~BCe) & BCi); + Ame = BCe ^ ((~BCi) & BCo); + Ami = BCi ^ ((~BCo) & BCu); + Amo = BCo ^ ((~BCu) & BCa); + Amu = BCu ^ ((~BCa) & BCe); + + Ebi ^= Di; + BCa = MLD_KECCAK_ROL(Ebi, 62); + Ego ^= Do; + BCe = MLD_KECCAK_ROL(Ego, 55); + Eku ^= Du; + BCi = MLD_KECCAK_ROL(Eku, 39); + Ema ^= Da; + BCo = MLD_KECCAK_ROL(Ema, 41); + Ese ^= De; + BCu = MLD_KECCAK_ROL(Ese, 2); + Asa = BCa ^ ((~BCe) & BCi); + Ase = BCe ^ ((~BCi) & BCo); + Asi = BCi ^ ((~BCo) & BCu); + Aso = BCo ^ ((~BCu) & BCa); + Asu = BCu ^ ((~BCa) & BCe); + } + + /* copyToState(state, A) */ + state[0] = Aba; + state[1] = Abe; + state[2] = Abi; + state[3] = Abo; + state[4] = Abu; + state[5] = Aga; + state[6] = Age; + state[7] = Agi; + state[8] = Ago; + state[9] = Agu; + state[10] = Aka; + state[11] = Ake; + state[12] = Aki; + state[13] = Ako; + state[14] = Aku; + state[15] = Ama; + state[16] = Ame; + state[17] = Ami; + state[18] = Amo; + state[19] = Amu; + state[20] = Asa; + state[21] = Ase; + state[22] = Asi; + state[23] = Aso; + state[24] = Asu; +} + +MLD_INTERNAL_API +void mld_keccakf1600_permute(uint64_t *state) +{ +#if defined(MLD_USE_NATIVE_FIPS202_X1) + if (mld_keccak_f1600_x1_native(state) == MLD_NATIVE_FUNC_SUCCESS) + { + return; + } +#endif /* MLD_USE_NATIVE_FIPS202_X1 */ + mld_keccakf1600_permute_c(state); +} + +#else /* !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +MLD_EMPTY_CU(keccakf1600) + +#endif /* MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. + * Don't modify by hand -- this is auto-generated by scripts/autogen. */ +#undef MLD_KECCAK_NROUNDS +#undef MLD_KECCAK_ROL diff --git a/cbits/mldsa/src/fips202/keccakf1600.h b/cbits/mldsa/src/fips202/keccakf1600.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/keccakf1600.h @@ -0,0 +1,110 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLD_FIPS202_KECCAKF1600_H +#define MLD_FIPS202_KECCAKF1600_H +#include "../cbmc.h" +#include "../common.h" +#include "fips202.h" + +#define MLD_KECCAK_LANES 25 +#define MLD_KECCAK_WAY 4 + +/* + * WARNING: + * The contents of this structure, including the placement + * and interleaving of Keccak lanes, are IMPLEMENTATION-DEFINED. + * The struct is only exposed here to allow its construction on the stack. + */ + +#define mld_keccakf1600_extract_bytes MLD_NAMESPACE(keccakf1600_extract_bytes) +MLD_INTERNAL_API +void mld_keccakf1600_extract_bytes(uint64_t *state, unsigned char *data, + unsigned offset, unsigned length) +__contract__( + requires(0 <= offset && offset <= MLD_KECCAK_LANES * sizeof(uint64_t) && + 0 <= length && length <= MLD_KECCAK_LANES * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES)) + requires(memory_no_alias(data, length)) + assigns(memory_slice(data, length)) +); + +#define mld_keccakf1600_xor_bytes MLD_NAMESPACE(keccakf1600_xor_bytes) +MLD_INTERNAL_API +void mld_keccakf1600_xor_bytes(uint64_t *state, const unsigned char *data, + unsigned offset, unsigned length) +__contract__( + requires(0 <= offset && offset <= MLD_KECCAK_LANES * sizeof(uint64_t) && + 0 <= length && length <= MLD_KECCAK_LANES * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES)) + requires(memory_no_alias(data, length)) + assigns(memory_slice(state, sizeof(uint64_t) * MLD_KECCAK_LANES)) +); + +#if (!defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_REDUCE_RAM) || \ + defined(MLD_UNIT_TEST)) && \ + !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) +#define mld_keccakf1600x4_extract_bytes \ + MLD_NAMESPACE(keccakf1600x4_extract_bytes) +MLD_INTERNAL_API +void mld_keccakf1600x4_extract_bytes(uint64_t *state, unsigned char *data0, + unsigned char *data1, unsigned char *data2, + unsigned char *data3, unsigned offset, + unsigned length) +__contract__( + requires(0 <= offset && offset <= MLD_KECCAK_LANES * sizeof(uint64_t) && + 0 <= length && length <= MLD_KECCAK_LANES * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY)) + requires(memory_no_alias(data0, length)) + requires(memory_no_alias(data1, length)) + requires(memory_no_alias(data2, length)) + requires(memory_no_alias(data3, length)) + assigns(memory_slice(data0, length)) + assigns(memory_slice(data1, length)) + assigns(memory_slice(data2, length)) + assigns(memory_slice(data3, length)) +); + +#define mld_keccakf1600x4_xor_bytes MLD_NAMESPACE(keccakf1600x4_xor_bytes) +MLD_INTERNAL_API +void mld_keccakf1600x4_xor_bytes(uint64_t *state, const unsigned char *data0, + const unsigned char *data1, + const unsigned char *data2, + const unsigned char *data3, unsigned offset, + unsigned length) +__contract__( + requires(0 <= offset && offset <= MLD_KECCAK_LANES * sizeof(uint64_t) && + 0 <= length && length <= MLD_KECCAK_LANES * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY)) + requires(memory_no_alias(data0, length)) + /* Case 1: all input buffers are distinct; Case 2: All input buffers are the same */ + requires((data0 == data1 && + data0 == data2 && + data0 == data3) || + (memory_no_alias(data1, length) && + memory_no_alias(data2, length) && + memory_no_alias(data3, length))) + assigns(memory_slice(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY)) +); + +#define mld_keccakf1600x4_permute MLD_NAMESPACE(keccakf1600x4_permute) +MLD_INTERNAL_API +void mld_keccakf1600x4_permute(uint64_t *state) +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY)) + assigns(memory_slice(state, sizeof(uint64_t) * MLD_KECCAK_LANES * MLD_KECCAK_WAY)) +); +#endif /* (!MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_REDUCE_RAM || \ + MLD_UNIT_TEST) && !MLD_CONFIG_SERIAL_FIPS202_ONLY */ + +#define mld_keccakf1600_permute MLD_NAMESPACE(keccakf1600_permute) +MLD_INTERNAL_API +void mld_keccakf1600_permute(uint64_t *state) +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * MLD_KECCAK_LANES)) + assigns(memory_slice(state, sizeof(uint64_t) * MLD_KECCAK_LANES)) +); + +#endif /* !MLD_FIPS202_KECCAKF1600_H */ diff --git a/cbits/mldsa/src/fips202/native/aarch64/auto.h b/cbits/mldsa/src/fips202/native/aarch64/auto.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/aarch64/auto.h @@ -0,0 +1,85 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [HYBRID] + * Hybrid scalar/vector implementations of Keccak and SPHINCS+ on AArch64 + * Becker, Kannwischer + * https://eprint.iacr.org/2022/1243 + */ + +#ifndef MLD_FIPS202_NATIVE_AARCH64_AUTO_H +#define MLD_FIPS202_NATIVE_AARCH64_AUTO_H +/* Default FIPS202 assembly profile for AArch64 systems */ + +/* + * Default logic to decide which implementation to use. + * + */ + +/* + * Keccak-f1600 + * + * - On Arm-based Apple CPUs, or CPUs with MLD_SYS_AARCH64_FAST_SHA3 set, + * we pick a pure Neon implementation. + * - Otherwise, unless MLD_SYS_AARCH64_SLOW_BARREL_SHIFTER is set, + * we use lazy-rotation scalar assembly from @[HYBRID]. + * - Otherwise, if MLD_SYS_AARCH64_SLOW_BARREL_SHIFTER is set, we + * fall back to the standard C implementation. + */ +#if defined(__ARM_FEATURE_SHA3) && \ + (defined(__APPLE__) || defined(MLD_SYS_AARCH64_FAST_SHA3)) +#include "x1_v84a.h" +#elif !defined(MLD_SYS_AARCH64_SLOW_BARREL_SHIFTER) +#include "x1_scalar.h" +#endif + +#if (!defined(MLD_CONFIG_NO_KEYPAIR_API) || \ + !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_REDUCE_RAM)) && \ + !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) +/* Batched, SIMD-based Keccak-f1600 implementations. */ +#if defined(MLD_SYS_AARCH64_NEON) + +/* + * Keccak-f1600x2/x4 + * + * The optimal implementation is highly CPU-specific; see @[HYBRID]. + * + * For now, if v8.4-A is not implemented, we fall back to Keccak-f1600. + * If v8.4-A is implemented and we are on an Apple CPU (or a CPU with + * MLD_SYS_AARCH64_FAST_SHA3 set), we use a plain Neon-based implementation. + * If v8.4-A is implemented and we are on neither, we use a + * scalar/Neon/Neon hybrid. + * The reason for this distinction is that Apple CPUs (and other CPUs flagged + * via MLD_SYS_AARCH64_FAST_SHA3) appear to implement the SHA3 instructions on + * all SIMD units, while Arm CPUs prior to Cortex-X4 don't, and ordinary Neon + * instructions are still needed. + */ +#if defined(__ARM_FEATURE_SHA3) +/* + * For Apple-M cores (and other cores flagged via MLD_SYS_AARCH64_FAST_SHA3), + * we use a plain implementation leveraging SHA3 instructions only. + */ +#if defined(__APPLE__) || defined(MLD_SYS_AARCH64_FAST_SHA3) +#include "x2_v84a.h" +#else +#include "x4_v8a_v84a_scalar.h" +#endif + +#else /* __ARM_FEATURE_SHA3 */ + +#include "x4_v8a_scalar.h" + +#endif /* !__ARM_FEATURE_SHA3 */ + +#endif /* MLD_SYS_AARCH64_NEON */ + +#endif /* (!MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API || \ + !MLD_CONFIG_REDUCE_RAM) && !MLD_CONFIG_SERIAL_FIPS202_ONLY */ + +#endif /* !MLD_FIPS202_NATIVE_AARCH64_AUTO_H */ diff --git a/cbits/mldsa/src/fips202/native/aarch64/src/fips202_native_aarch64.h b/cbits/mldsa/src/fips202/native/aarch64/src/fips202_native_aarch64.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/aarch64/src/fips202_native_aarch64.h @@ -0,0 +1,69 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLD_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H +#define MLD_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H + + +#include "../../../../cbmc.h" +#include "../../../../common.h" + + +#define mld_keccakf1600_round_constants \ + MLD_NAMESPACE(keccakf1600_round_constants) +MLD_INTERNAL_DATA_DECLARATION const uint64_t + mld_keccakf1600_round_constants[24]; + +#define mld_keccak_f1600_x1_scalar_aarch64_asm \ + MLD_NAMESPACE(keccak_f1600_x1_scalar_aarch64_asm) +void mld_keccak_f1600_x1_scalar_aarch64_asm(uint64_t state[25], + const uint64_t rc[24]) +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 1)) + requires(rc == mld_keccakf1600_round_constants) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 1)) +); + +#define mld_keccak_f1600_x1_v84a_aarch64_asm \ + MLD_NAMESPACE(keccak_f1600_x1_v84a_aarch64_asm) +void mld_keccak_f1600_x1_v84a_aarch64_asm(uint64_t state[25], + const uint64_t rc[24]) +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 1)) + requires(rc == mld_keccakf1600_round_constants) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 1)) +); + +#define mld_keccak_f1600_x2_v84a_aarch64_asm \ + MLD_NAMESPACE(keccak_f1600_x2_v84a_aarch64_asm) +void mld_keccak_f1600_x2_v84a_aarch64_asm(uint64_t state[50], + const uint64_t rc[24]) +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 2)) + requires(rc == mld_keccakf1600_round_constants) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 2)) +); + +#define mld_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm \ + MLD_NAMESPACE(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm) +void mld_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm(uint64_t state[100], + const uint64_t rc[24]) +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4)) + requires(rc == mld_keccakf1600_round_constants) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4)) +); + +#define mld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm \ + MLD_NAMESPACE(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm) +void mld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm( + uint64_t state[100], const uint64_t rc[24]) +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4)) + requires(rc == mld_keccakf1600_round_constants) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4)) +); + +#endif /* !MLD_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H */ diff --git a/cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S b/cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S @@ -0,0 +1,378 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * Copyright (c) 2021-2022 Arm Limited + * Copyright (c) 2022 Matthias Kannwischer + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +// Author: Hanno Becker +// Author: Matthias Kannwischer + +/*yaml + Name: keccak_f1600_x1_scalar_aarch64_asm + Description: AArch64 scalar implementation of Keccak-f[1600] permutation for single state + Signature: void mld_keccak_f1600_x1_scalar_aarch64_asm(uint64_t state[25], const uint64_t rc[24]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + x0: + type: buffer + size_bytes: 200 + permissions: read/write + c_parameter: uint64_t state[25] + description: Keccak state (25 x uint64_t) + x1: + type: buffer + size_bytes: 192 + permissions: read-only + c_parameter: uint64_t const *rc + description: Round constants (24 x uint64_t) + Stack: + bytes: 128 + description: register preservation and temporary storage +*/ + +#include "../../../../common.h" +#if defined(MLD_FIPS202_AARCH64_NEED_X1_SCALAR) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/fips202/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(keccak_f1600_x1_scalar_aarch64_asm) +MLD_ASM_FN_SYMBOL(keccak_f1600_x1_scalar_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x80 + .cfi_adjust_cfa_offset 0x80 + stp x19, x20, [sp, #0x20] + .cfi_rel_offset x19, 0x20 + .cfi_rel_offset x20, 0x28 + stp x21, x22, [sp, #0x30] + .cfi_rel_offset x21, 0x30 + .cfi_rel_offset x22, 0x38 + stp x23, x24, [sp, #0x40] + .cfi_rel_offset x23, 0x40 + .cfi_rel_offset x24, 0x48 + stp x25, x26, [sp, #0x50] + .cfi_rel_offset x25, 0x50 + .cfi_rel_offset x26, 0x58 + stp x27, x28, [sp, #0x60] + .cfi_rel_offset x27, 0x60 + .cfi_rel_offset x28, 0x68 + stp x29, x30, [sp, #0x70] + .cfi_rel_offset x29, 0x70 + .cfi_rel_offset x30, 0x78 + +Lmld_keccak_f1600_x1_scalar_initial: + mov x26, x1 + str x1, [sp, #0x8] + ldp x1, x6, [x0] + ldp x11, x16, [x0, #0x10] + ldp x21, x2, [x0, #0x20] + ldp x7, x12, [x0, #0x30] + ldp x17, x22, [x0, #0x40] + ldp x3, x8, [x0, #0x50] + ldp x13, x28, [x0, #0x60] + ldp x23, x4, [x0, #0x70] + ldp x9, x14, [x0, #0x80] + ldp x19, x24, [x0, #0x90] + ldp x5, x10, [x0, #0xa0] + ldp x15, x20, [x0, #0xb0] + ldr x25, [x0, #0xc0] + str x0, [sp] + eor x30, x24, x25 + eor x27, x9, x10 + eor x0, x30, x21 + eor x26, x27, x6 + eor x27, x26, x7 + eor x29, x0, x22 + eor x26, x29, x23 + eor x29, x4, x5 + eor x30, x29, x1 + eor x0, x27, x8 + eor x29, x30, x2 + eor x30, x19, x20 + eor x30, x30, x16 + eor x27, x26, x0, ror #63 + eor x4, x4, x27 + eor x30, x30, x17 + eor x30, x30, x28 + eor x29, x29, x3 + eor x0, x0, x30, ror #63 + eor x30, x30, x29, ror #63 + eor x22, x22, x30 + eor x23, x23, x30 + str x23, [sp, #0x18] + eor x23, x14, x15 + eor x14, x14, x0 + eor x23, x23, x11 + eor x15, x15, x0 + eor x1, x1, x27 + eor x23, x23, x12 + eor x23, x23, x13 + eor x11, x11, x0 + eor x29, x29, x23, ror #63 + eor x23, x23, x26, ror #63 + eor x26, x13, x0 + eor x13, x28, x23 + eor x28, x24, x30 + eor x24, x16, x23 + eor x16, x21, x30 + eor x21, x25, x30 + eor x30, x19, x23 + eor x19, x20, x23 + eor x20, x17, x23 + eor x17, x12, x0 + eor x0, x2, x27 + eor x2, x6, x29 + eor x6, x8, x29 + bic x8, x28, x13, ror #47 + eor x12, x3, x27 + bic x3, x13, x17, ror #19 + eor x5, x5, x27 + ldr x27, [sp, #0x18] + bic x25, x17, x2, ror #5 + eor x9, x9, x29 + eor x23, x25, x5, ror #52 + eor x3, x3, x2, ror #24 + eor x8, x8, x17, ror #2 + eor x17, x10, x29 + bic x25, x12, x22, ror #47 + eor x29, x7, x29 + bic x10, x4, x27, ror #2 + bic x7, x5, x28, ror #10 + eor x10, x10, x20, ror #50 + eor x13, x7, x13, ror #57 + bic x7, x2, x5, ror #47 + eor x2, x25, x24, ror #39 + bic x25, x20, x11, ror #57 + bic x5, x17, x4, ror #25 + eor x25, x25, x17, ror #53 + bic x17, x11, x17, ror #60 + eor x28, x7, x28, ror #57 + bic x7, x9, x12, ror #42 + eor x7, x7, x22, ror #25 + bic x22, x22, x24, ror #56 + bic x24, x24, x15, ror #31 + eor x22, x22, x15, ror #23 + bic x20, x27, x20, ror #48 + bic x15, x15, x9, ror #16 + eor x12, x15, x12, ror #58 + eor x15, x5, x27, ror #27 + eor x5, x20, x11, ror #41 + ldr x11, [sp, #0x8] + eor x20, x17, x4, ror #21 + eor x17, x24, x9, ror #47 + mov x24, #0x1 // =1 + bic x9, x0, x16, ror #9 + str x24, [sp, #0x10] + bic x24, x29, x1, ror #44 + bic x27, x1, x21, ror #50 + bic x4, x26, x29, ror #63 + eor x1, x1, x4, ror #21 + ldr x11, [x11] + bic x4, x21, x30, ror #57 + eor x21, x24, x21, ror #30 + eor x24, x9, x19, ror #44 + bic x9, x14, x6, ror #5 + eor x9, x9, x0, ror #43 + bic x0, x6, x0, ror #38 + eor x1, x1, x11 + eor x11, x4, x26, ror #35 + eor x4, x0, x16, ror #47 + bic x0, x16, x19, ror #35 + eor x16, x27, x30, ror #43 + bic x27, x30, x26, ror #42 + bic x26, x19, x14, ror #41 + eor x19, x0, x14, ror #12 + eor x14, x26, x6, ror #46 + eor x6, x27, x29, ror #41 + +Lmld_keccak_f1600_x1_scalar_loop: + eor x0, x15, x11, ror #52 + eor x0, x0, x13, ror #48 + eor x26, x8, x9, ror #57 + eor x27, x0, x14, ror #10 + eor x29, x16, x28, ror #63 + eor x26, x26, x6, ror #51 + eor x30, x23, x22, ror #50 + eor x0, x26, x10, ror #31 + eor x29, x29, x19, ror #37 + eor x27, x27, x12, ror #5 + eor x30, x30, x24, ror #34 + eor x0, x0, x7, ror #27 + eor x26, x30, x21, ror #26 + eor x26, x26, x25, ror #15 + ror x30, x27, #0x3e + eor x30, x30, x26, ror #57 + ror x26, x26, #0x3a + eor x16, x30, x16 + eor x28, x30, x28, ror #63 + str x28, [sp, #0x18] + eor x29, x29, x17, ror #36 + eor x28, x1, x2, ror #61 + eor x19, x30, x19, ror #37 + eor x29, x29, x20, ror #2 + eor x28, x28, x4, ror #54 + eor x26, x26, x0, ror #55 + eor x28, x28, x3, ror #39 + eor x28, x28, x5, ror #25 + ror x0, x0, #0x38 + eor x0, x0, x29, ror #63 + eor x27, x28, x27, ror #61 + eor x13, x0, x13, ror #46 + eor x28, x29, x28, ror #63 + eor x29, x30, x20, ror #2 + eor x20, x26, x3, ror #39 + eor x11, x0, x11, ror #50 + eor x25, x28, x25, ror #9 + eor x3, x28, x21, ror #20 + eor x21, x26, x1 + eor x9, x27, x9, ror #49 + eor x24, x28, x24, ror #28 + eor x1, x30, x17, ror #36 + eor x14, x0, x14, ror #8 + eor x22, x28, x22, ror #44 + eor x8, x27, x8, ror #56 + eor x17, x27, x7, ror #19 + eor x15, x0, x15, ror #62 + bic x7, x20, x22, ror #47 + eor x4, x26, x4, ror #54 + eor x0, x0, x12, ror #3 + eor x28, x28, x23, ror #58 + eor x23, x26, x2, ror #61 + eor x26, x26, x5, ror #25 + eor x2, x7, x16, ror #39 + bic x7, x9, x20, ror #42 + bic x30, x15, x9, ror #16 + eor x7, x7, x22, ror #25 + eor x12, x30, x20, ror #58 + bic x20, x22, x16, ror #56 + eor x30, x27, x6, ror #43 + eor x22, x20, x15, ror #23 + bic x6, x19, x13, ror #42 + eor x6, x6, x17, ror #41 + bic x5, x13, x17, ror #63 + eor x5, x21, x5, ror #21 + bic x17, x17, x21, ror #44 + eor x27, x27, x10, ror #23 + bic x21, x21, x25, ror #50 + bic x20, x27, x4, ror #25 + bic x10, x16, x15, ror #31 + eor x16, x21, x19, ror #43 + eor x21, x17, x25, ror #30 + bic x19, x25, x19, ror #57 + ldr x25, [sp, #0x10] + eor x17, x10, x9, ror #47 + ldr x9, [sp, #0x8] + eor x15, x20, x28, ror #27 + bic x20, x4, x28, ror #2 + eor x10, x20, x1, ror #50 + bic x20, x11, x27, ror #60 + eor x20, x20, x4, ror #21 + bic x4, x28, x1, ror #48 + bic x1, x1, x11, ror #57 + ldr x28, [x9, x25, lsl #3] + ldr x9, [sp, #0x18] + add x25, x25, #0x1 + str x25, [sp, #0x10] + cmp x25, #0x17 + eor x25, x1, x27, ror #53 + bic x27, x30, x26, ror #47 + eor x1, x5, x28 + eor x5, x4, x11, ror #41 + eor x11, x19, x13, ror #35 + bic x13, x26, x24, ror #10 + eor x28, x27, x24, ror #57 + bic x27, x24, x9, ror #47 + bic x19, x23, x3, ror #9 + bic x4, x29, x14, ror #41 + eor x24, x19, x29, ror #44 + bic x29, x3, x29, ror #35 + eor x13, x13, x9, ror #57 + eor x19, x29, x14, ror #12 + bic x29, x9, x0, ror #19 + bic x14, x14, x8, ror #5 + eor x9, x14, x23, ror #43 + eor x14, x4, x8, ror #46 + bic x23, x8, x23, ror #38 + eor x8, x27, x0, ror #2 + eor x4, x23, x3, ror #47 + bic x3, x0, x30, ror #5 + eor x23, x3, x26, ror #52 + eor x3, x29, x30, ror #24 + b.le Lmld_keccak_f1600_x1_scalar_loop + ror x6, x6, #0x2b + ror x11, x11, #0x32 + ror x21, x21, #0x14 + ror x2, x2, #0x3d + ror x7, x7, #0x13 + ror x12, x12, #0x3 + ror x17, x17, #0x24 + ror x22, x22, #0x2c + ror x3, x3, #0x27 + ror x8, x8, #0x38 + ror x13, x13, #0x2e + ror x28, x28, #0x3f + ror x23, x23, #0x3a + ror x4, x4, #0x36 + ror x9, x9, #0x31 + ror x14, x14, #0x8 + ror x19, x19, #0x25 + ror x24, x24, #0x1c + ror x5, x5, #0x19 + ror x10, x10, #0x17 + ror x15, x15, #0x3e + ror x20, x20, #0x2 + ror x25, x25, #0x9 + ldr x0, [sp] + stp x1, x6, [x0] + stp x11, x16, [x0, #0x10] + stp x21, x2, [x0, #0x20] + stp x7, x12, [x0, #0x30] + stp x17, x22, [x0, #0x40] + stp x3, x8, [x0, #0x50] + stp x13, x28, [x0, #0x60] + stp x23, x4, [x0, #0x70] + stp x9, x14, [x0, #0x80] + stp x19, x24, [x0, #0x90] + stp x5, x10, [x0, #0xa0] + stp x15, x20, [x0, #0xb0] + str x25, [x0, #0xc0] + ldp x19, x20, [sp, #0x20] + .cfi_restore x19 + .cfi_restore x20 + ldp x21, x22, [sp, #0x30] + .cfi_restore x21 + .cfi_restore x22 + ldp x23, x24, [sp, #0x40] + .cfi_restore x23 + .cfi_restore x24 + ldp x25, x26, [sp, #0x50] + .cfi_restore x25 + .cfi_restore x26 + ldp x27, x28, [sp, #0x60] + .cfi_restore x27 + .cfi_restore x28 + ldp x29, x30, [sp, #0x70] + .cfi_restore x29 + .cfi_restore x30 + add sp, sp, #0x80 + .cfi_adjust_cfa_offset -0x80 + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(keccak_f1600_x1_scalar_aarch64_asm) + +#endif /* MLD_FIPS202_AARCH64_NEED_X1_SCALAR && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S b/cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S @@ -0,0 +1,207 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * Copyright (c) 2021-2022 Arm Limited + * Copyright (c) 2022 Matthias Kannwischer + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [HYBRID] + * Hybrid scalar/vector implementations of Keccak and SPHINCS+ on AArch64 + * Becker, Kannwischer + * https://eprint.iacr.org/2022/1243 + */ + +/*yaml + Name: keccak_f1600_x1_v84a_aarch64_asm + Description: AArch64 ARMv8.4-A implementation of Keccak-f[1600] permutation for single state + Signature: void mld_keccak_f1600_x1_v84a_aarch64_asm(uint64_t state[25], const uint64_t rc[24]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON, SHA3] + x0: + type: buffer + size_bytes: 200 + permissions: read/write + c_parameter: uint64_t state[25] + description: Keccak state (25 x uint64_t) + x1: + type: buffer + size_bytes: 192 + permissions: read-only + c_parameter: uint64_t const *rc + description: Round constants (24 x uint64_t) + Stack: + bytes: 64 + description: register preservation +*/ + +// +// Author: Hanno Becker +// Author: Matthias Kannwischer +// +// This implementation is essentially from the paper @[HYBRID]. +// The only difference is interleaving/deinterleaving of Keccak state +// during load and store, so that the caller need not do this. +// + +#include "../../../../common.h" +#if defined(MLD_FIPS202_AARCH64_NEED_X1_V84A) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +#if defined(__ARM_FEATURE_SHA3) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/fips202/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(keccak_f1600_x1_v84a_aarch64_asm) +MLD_ASM_FN_SYMBOL(keccak_f1600_x1_v84a_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x40 + .cfi_adjust_cfa_offset 0x40 + stp d8, d9, [sp] + .cfi_rel_offset d8, 0x0 + .cfi_rel_offset d9, 0x8 + stp d10, d11, [sp, #0x10] + .cfi_rel_offset d10, 0x10 + .cfi_rel_offset d11, 0x18 + stp d12, d13, [sp, #0x20] + .cfi_rel_offset d12, 0x20 + .cfi_rel_offset d13, 0x28 + stp d14, d15, [sp, #0x30] + .cfi_rel_offset d14, 0x30 + .cfi_rel_offset d15, 0x38 + ldp d0, d1, [x0] + ldp d2, d3, [x0, #0x10] + ldp d4, d5, [x0, #0x20] + ldp d6, d7, [x0, #0x30] + ldp d8, d9, [x0, #0x40] + ldp d10, d11, [x0, #0x50] + ldp d12, d13, [x0, #0x60] + ldp d14, d15, [x0, #0x70] + ldp d16, d17, [x0, #0x80] + ldp d18, d19, [x0, #0x90] + ldp d20, d21, [x0, #0xa0] + ldp d22, d23, [x0, #0xb0] + ldr d24, [x0, #0xc0] + mov x2, #0x18 // =24 + +Lmld_keccak_f1600_x1_v84a_loop: + eor3 v30.16b, v0.16b, v5.16b, v10.16b + eor3 v29.16b, v1.16b, v6.16b, v11.16b + eor3 v28.16b, v2.16b, v7.16b, v12.16b + eor3 v27.16b, v3.16b, v8.16b, v13.16b + eor3 v26.16b, v4.16b, v9.16b, v14.16b + eor3 v30.16b, v30.16b, v15.16b, v20.16b + eor3 v29.16b, v29.16b, v16.16b, v21.16b + eor3 v28.16b, v28.16b, v17.16b, v22.16b + eor3 v27.16b, v27.16b, v18.16b, v23.16b + eor3 v26.16b, v26.16b, v19.16b, v24.16b + rax1 v25.2d, v30.2d, v28.2d + rax1 v28.2d, v28.2d, v26.2d + rax1 v26.2d, v26.2d, v29.2d + rax1 v29.2d, v29.2d, v27.2d + rax1 v27.2d, v27.2d, v30.2d + eor v30.16b, v0.16b, v26.16b + xar v0.2d, v2.2d, v29.2d, #0x2 + xar v2.2d, v12.2d, v29.2d, #0x15 + xar v12.2d, v13.2d, v28.2d, #0x27 + xar v13.2d, v19.2d, v27.2d, #0x38 + xar v19.2d, v23.2d, v28.2d, #0x8 + xar v23.2d, v15.2d, v26.2d, #0x17 + xar v15.2d, v1.2d, v25.2d, #0x3f + xar v1.2d, v8.2d, v28.2d, #0x9 + xar v8.2d, v16.2d, v25.2d, #0x13 + xar v16.2d, v7.2d, v29.2d, #0x3a + xar v7.2d, v10.2d, v26.2d, #0x3d + xar v10.2d, v3.2d, v28.2d, #0x24 + xar v3.2d, v18.2d, v28.2d, #0x2b + xar v18.2d, v17.2d, v29.2d, #0x31 + xar v17.2d, v11.2d, v25.2d, #0x36 + xar v11.2d, v9.2d, v27.2d, #0x2c + xar v9.2d, v22.2d, v29.2d, #0x3 + xar v22.2d, v14.2d, v27.2d, #0x19 + xar v14.2d, v20.2d, v26.2d, #0x2e + xar v20.2d, v4.2d, v27.2d, #0x25 + xar v4.2d, v24.2d, v27.2d, #0x32 + xar v24.2d, v21.2d, v25.2d, #0x3e + xar v21.2d, v5.2d, v26.2d, #0x1c + xar v27.2d, v6.2d, v25.2d, #0x14 + ld1r { v31.2d }, [x1], #8 + bcax v5.16b, v10.16b, v7.16b, v11.16b + bcax v6.16b, v11.16b, v8.16b, v7.16b + bcax v7.16b, v7.16b, v9.16b, v8.16b + bcax v8.16b, v8.16b, v10.16b, v9.16b + bcax v9.16b, v9.16b, v11.16b, v10.16b + bcax v10.16b, v15.16b, v12.16b, v16.16b + bcax v11.16b, v16.16b, v13.16b, v12.16b + bcax v12.16b, v12.16b, v14.16b, v13.16b + bcax v13.16b, v13.16b, v15.16b, v14.16b + bcax v14.16b, v14.16b, v16.16b, v15.16b + bcax v15.16b, v20.16b, v17.16b, v21.16b + bcax v16.16b, v21.16b, v18.16b, v17.16b + bcax v17.16b, v17.16b, v19.16b, v18.16b + bcax v18.16b, v18.16b, v20.16b, v19.16b + bcax v19.16b, v19.16b, v21.16b, v20.16b + bcax v20.16b, v0.16b, v22.16b, v1.16b + bcax v21.16b, v1.16b, v23.16b, v22.16b + bcax v22.16b, v22.16b, v24.16b, v23.16b + bcax v23.16b, v23.16b, v0.16b, v24.16b + bcax v24.16b, v24.16b, v1.16b, v0.16b + bcax v0.16b, v30.16b, v2.16b, v27.16b + bcax v1.16b, v27.16b, v3.16b, v2.16b + bcax v2.16b, v2.16b, v4.16b, v3.16b + bcax v3.16b, v3.16b, v30.16b, v4.16b + bcax v4.16b, v4.16b, v27.16b, v30.16b + eor v0.16b, v0.16b, v31.16b + sub x2, x2, #0x1 + cbnz x2, Lmld_keccak_f1600_x1_v84a_loop + stp d0, d1, [x0] + stp d2, d3, [x0, #0x10] + stp d4, d5, [x0, #0x20] + stp d6, d7, [x0, #0x30] + stp d8, d9, [x0, #0x40] + stp d10, d11, [x0, #0x50] + stp d12, d13, [x0, #0x60] + stp d14, d15, [x0, #0x70] + stp d16, d17, [x0, #0x80] + stp d18, d19, [x0, #0x90] + stp d20, d21, [x0, #0xa0] + stp d22, d23, [x0, #0xb0] + str d24, [x0, #0xc0] + ldp d8, d9, [sp] + .cfi_restore d8 + .cfi_restore d9 + ldp d10, d11, [sp, #0x10] + .cfi_restore d10 + .cfi_restore d11 + ldp d12, d13, [sp, #0x20] + .cfi_restore d12 + .cfi_restore d13 + ldp d14, d15, [sp, #0x30] + .cfi_restore d14 + .cfi_restore d15 + add sp, sp, #0x40 + .cfi_adjust_cfa_offset -0x40 + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(keccak_f1600_x1_v84a_aarch64_asm) + +#endif /* __ARM_FEATURE_SHA3 */ + +#endif /* MLD_FIPS202_AARCH64_NEED_X1_V84A && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S b/cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S @@ -0,0 +1,262 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * Copyright (c) 2021-2022 Arm Limited + * Copyright (c) 2022 Matthias Kannwischer + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [HYBRID] + * Hybrid scalar/vector implementations of Keccak and SPHINCS+ on AArch64 + * Becker, Kannwischer + * https://eprint.iacr.org/2022/1243 + */ + +/*yaml + Name: keccak_f1600_x2_v84a_aarch64_asm + Description: AArch64 ARMv8.4-A implementation of Keccak-f[1600] permutation for two sequential states + Signature: void mld_keccak_f1600_x2_v84a_aarch64_asm(uint64_t state[50], const uint64_t rc[24]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON, SHA3] + x0: + type: buffer + size_bytes: 400 + permissions: read/write + c_parameter: uint64_t state[50] + description: Two sequential Keccak states (state0[25], state1[25]) + x1: + type: buffer + size_bytes: 192 + permissions: read-only + c_parameter: uint64_t const *rc + description: Round constants (24 x uint64_t) + Stack: + bytes: 64 + description: register preservation +*/ + +// +// Author: Hanno Becker +// Author: Matthias Kannwischer +// +// This implementation is essentially from the paper @[HYBRID]. +// The only difference is interleaving/deinterleaving of Keccak state +// during load and store, so that the caller need not do this. +// + +#include "../../../../common.h" +#if defined(MLD_FIPS202_AARCH64_NEED_X2_V84A) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +#if defined(__ARM_FEATURE_SHA3) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/fips202/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(keccak_f1600_x2_v84a_aarch64_asm) +MLD_ASM_FN_SYMBOL(keccak_f1600_x2_v84a_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x40 + .cfi_adjust_cfa_offset 0x40 + stp d8, d9, [sp] + .cfi_rel_offset d8, 0x0 + .cfi_rel_offset d9, 0x8 + stp d10, d11, [sp, #0x10] + .cfi_rel_offset d10, 0x10 + .cfi_rel_offset d11, 0x18 + stp d12, d13, [sp, #0x20] + .cfi_rel_offset d12, 0x20 + .cfi_rel_offset d13, 0x28 + stp d14, d15, [sp, #0x30] + .cfi_rel_offset d14, 0x30 + .cfi_rel_offset d15, 0x38 + add x2, x0, #0xc8 + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x2], #32 + trn1 v0.2d, v25.2d, v27.2d + trn2 v1.2d, v25.2d, v27.2d + trn1 v2.2d, v26.2d, v28.2d + trn2 v3.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x2], #32 + trn1 v4.2d, v25.2d, v27.2d + trn2 v5.2d, v25.2d, v27.2d + trn1 v6.2d, v26.2d, v28.2d + trn2 v7.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x2], #32 + trn1 v8.2d, v25.2d, v27.2d + trn2 v9.2d, v25.2d, v27.2d + trn1 v10.2d, v26.2d, v28.2d + trn2 v11.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x2], #32 + trn1 v12.2d, v25.2d, v27.2d + trn2 v13.2d, v25.2d, v27.2d + trn1 v14.2d, v26.2d, v28.2d + trn2 v15.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x2], #32 + trn1 v16.2d, v25.2d, v27.2d + trn2 v17.2d, v25.2d, v27.2d + trn1 v18.2d, v26.2d, v28.2d + trn2 v19.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x2], #32 + trn1 v20.2d, v25.2d, v27.2d + trn2 v21.2d, v25.2d, v27.2d + trn1 v22.2d, v26.2d, v28.2d + trn2 v23.2d, v26.2d, v28.2d + ldr d25, [x0] + ldr d27, [x2] + trn1 v24.2d, v25.2d, v27.2d + mov x2, #0x18 // =24 + +Lmld_keccak_f1600_x2_v84a_loop: + eor3 v30.16b, v0.16b, v5.16b, v10.16b + eor3 v29.16b, v1.16b, v6.16b, v11.16b + eor3 v28.16b, v2.16b, v7.16b, v12.16b + eor3 v27.16b, v3.16b, v8.16b, v13.16b + eor3 v26.16b, v4.16b, v9.16b, v14.16b + eor3 v30.16b, v30.16b, v15.16b, v20.16b + eor3 v29.16b, v29.16b, v16.16b, v21.16b + eor3 v28.16b, v28.16b, v17.16b, v22.16b + eor3 v27.16b, v27.16b, v18.16b, v23.16b + eor3 v26.16b, v26.16b, v19.16b, v24.16b + rax1 v25.2d, v30.2d, v28.2d + rax1 v28.2d, v28.2d, v26.2d + rax1 v26.2d, v26.2d, v29.2d + rax1 v29.2d, v29.2d, v27.2d + rax1 v27.2d, v27.2d, v30.2d + eor v30.16b, v0.16b, v26.16b + xar v0.2d, v2.2d, v29.2d, #0x2 + xar v2.2d, v12.2d, v29.2d, #0x15 + xar v12.2d, v13.2d, v28.2d, #0x27 + xar v13.2d, v19.2d, v27.2d, #0x38 + xar v19.2d, v23.2d, v28.2d, #0x8 + xar v23.2d, v15.2d, v26.2d, #0x17 + xar v15.2d, v1.2d, v25.2d, #0x3f + xar v1.2d, v8.2d, v28.2d, #0x9 + xar v8.2d, v16.2d, v25.2d, #0x13 + xar v16.2d, v7.2d, v29.2d, #0x3a + xar v7.2d, v10.2d, v26.2d, #0x3d + xar v10.2d, v3.2d, v28.2d, #0x24 + xar v3.2d, v18.2d, v28.2d, #0x2b + xar v18.2d, v17.2d, v29.2d, #0x31 + xar v17.2d, v11.2d, v25.2d, #0x36 + xar v11.2d, v9.2d, v27.2d, #0x2c + xar v9.2d, v22.2d, v29.2d, #0x3 + xar v22.2d, v14.2d, v27.2d, #0x19 + xar v14.2d, v20.2d, v26.2d, #0x2e + xar v20.2d, v4.2d, v27.2d, #0x25 + xar v4.2d, v24.2d, v27.2d, #0x32 + xar v24.2d, v21.2d, v25.2d, #0x3e + xar v21.2d, v5.2d, v26.2d, #0x1c + xar v27.2d, v6.2d, v25.2d, #0x14 + ld1r { v31.2d }, [x1], #8 + bcax v5.16b, v10.16b, v7.16b, v11.16b + bcax v6.16b, v11.16b, v8.16b, v7.16b + bcax v7.16b, v7.16b, v9.16b, v8.16b + bcax v8.16b, v8.16b, v10.16b, v9.16b + bcax v9.16b, v9.16b, v11.16b, v10.16b + bcax v10.16b, v15.16b, v12.16b, v16.16b + bcax v11.16b, v16.16b, v13.16b, v12.16b + bcax v12.16b, v12.16b, v14.16b, v13.16b + bcax v13.16b, v13.16b, v15.16b, v14.16b + bcax v14.16b, v14.16b, v16.16b, v15.16b + bcax v15.16b, v20.16b, v17.16b, v21.16b + bcax v16.16b, v21.16b, v18.16b, v17.16b + bcax v17.16b, v17.16b, v19.16b, v18.16b + bcax v18.16b, v18.16b, v20.16b, v19.16b + bcax v19.16b, v19.16b, v21.16b, v20.16b + bcax v20.16b, v0.16b, v22.16b, v1.16b + bcax v21.16b, v1.16b, v23.16b, v22.16b + bcax v22.16b, v22.16b, v24.16b, v23.16b + bcax v23.16b, v23.16b, v0.16b, v24.16b + bcax v24.16b, v24.16b, v1.16b, v0.16b + bcax v0.16b, v30.16b, v2.16b, v27.16b + bcax v1.16b, v27.16b, v3.16b, v2.16b + bcax v2.16b, v2.16b, v4.16b, v3.16b + bcax v3.16b, v3.16b, v30.16b, v4.16b + bcax v4.16b, v4.16b, v27.16b, v30.16b + eor v0.16b, v0.16b, v31.16b + sub x2, x2, #0x1 + cbnz x2, Lmld_keccak_f1600_x2_v84a_loop + sub x0, x0, #0xc0 + add x2, x0, #0xc8 + trn1 v25.2d, v0.2d, v1.2d + trn1 v26.2d, v2.2d, v3.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v0.2d, v1.2d + trn2 v28.2d, v2.2d, v3.2d + st1 { v27.2d, v28.2d }, [x2], #32 + trn1 v25.2d, v4.2d, v5.2d + trn1 v26.2d, v6.2d, v7.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v4.2d, v5.2d + trn2 v28.2d, v6.2d, v7.2d + st1 { v27.2d, v28.2d }, [x2], #32 + trn1 v25.2d, v8.2d, v9.2d + trn1 v26.2d, v10.2d, v11.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v8.2d, v9.2d + trn2 v28.2d, v10.2d, v11.2d + st1 { v27.2d, v28.2d }, [x2], #32 + trn1 v25.2d, v12.2d, v13.2d + trn1 v26.2d, v14.2d, v15.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v12.2d, v13.2d + trn2 v28.2d, v14.2d, v15.2d + st1 { v27.2d, v28.2d }, [x2], #32 + trn1 v25.2d, v16.2d, v17.2d + trn1 v26.2d, v18.2d, v19.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v16.2d, v17.2d + trn2 v28.2d, v18.2d, v19.2d + st1 { v27.2d, v28.2d }, [x2], #32 + trn1 v25.2d, v20.2d, v21.2d + trn1 v26.2d, v22.2d, v23.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v20.2d, v21.2d + trn2 v28.2d, v22.2d, v23.2d + st1 { v27.2d, v28.2d }, [x2], #32 + str d24, [x0] + trn2 v25.2d, v24.2d, v24.2d + str d25, [x2] + ldp d8, d9, [sp] + .cfi_restore d8 + .cfi_restore d9 + ldp d10, d11, [sp, #0x10] + .cfi_restore d10 + .cfi_restore d11 + ldp d12, d13, [sp, #0x20] + .cfi_restore d12 + .cfi_restore d13 + ldp d14, d15, [sp, #0x30] + .cfi_restore d14 + .cfi_restore d15 + add sp, sp, #0x40 + .cfi_adjust_cfa_offset -0x40 + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(keccak_f1600_x2_v84a_aarch64_asm) + +#endif /* __ARM_FEATURE_SHA3 */ + +#endif /* MLD_FIPS202_AARCH64_NEED_X2_V84A && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S b/cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S @@ -0,0 +1,1080 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * Copyright (c) 2021-2022 Arm Limited + * Copyright (c) 2022 Matthias Kannwischer + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +// Author: Hanno Becker +// Author: Matthias Kannwischer + +/*yaml + Name: keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm + Description: AArch64 hybrid scalar/vector implementation of Keccak-f[1600] permutation for four sequential states + Signature: void mld_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm(uint64_t state[100], const uint64_t rc[24]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 800 + permissions: read/write + c_parameter: uint64_t state[100] + description: Four sequential Keccak states (state0[25], state1[25], state2[25], state3[25]) + x1: + type: buffer + size_bytes: 192 + permissions: read-only + c_parameter: uint64_t const *rc + description: Round constants (24 x uint64_t) + Stack: + bytes: 224 + description: register preservation and temporary storage +*/ + +#include "../../../../common.h" +#if defined(MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/fips202/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm) +MLD_ASM_FN_SYMBOL(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0xe0 + .cfi_adjust_cfa_offset 0xe0 + stp x19, x20, [sp, #0x30] + .cfi_rel_offset x19, 0x30 + .cfi_rel_offset x20, 0x38 + stp x21, x22, [sp, #0x40] + .cfi_rel_offset x21, 0x40 + .cfi_rel_offset x22, 0x48 + stp x23, x24, [sp, #0x50] + .cfi_rel_offset x23, 0x50 + .cfi_rel_offset x24, 0x58 + stp x25, x26, [sp, #0x60] + .cfi_rel_offset x25, 0x60 + .cfi_rel_offset x26, 0x68 + stp x27, x28, [sp, #0x70] + .cfi_rel_offset x27, 0x70 + .cfi_rel_offset x28, 0x78 + stp x29, x30, [sp, #0x80] + .cfi_rel_offset x29, 0x80 + .cfi_rel_offset x30, 0x88 + stp d8, d9, [sp, #0x90] + .cfi_rel_offset d8, 0x90 + .cfi_rel_offset d9, 0x98 + stp d10, d11, [sp, #0xa0] + .cfi_rel_offset d10, 0xa0 + .cfi_rel_offset d11, 0xa8 + stp d12, d13, [sp, #0xb0] + .cfi_rel_offset d12, 0xb0 + .cfi_rel_offset d13, 0xb8 + stp d14, d15, [sp, #0xc0] + .cfi_rel_offset d14, 0xc0 + .cfi_rel_offset d15, 0xc8 + mov x29, x1 + mov x30, #0x0 // =0 + str x30, [sp, #0x20] + str x29, [sp, #0x8] + str x29, [sp, #0x10] + str x0, [sp] + add x4, x0, #0xc8 + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v0.2d, v25.2d, v27.2d + trn2 v1.2d, v25.2d, v27.2d + trn1 v2.2d, v26.2d, v28.2d + trn2 v3.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v4.2d, v25.2d, v27.2d + trn2 v5.2d, v25.2d, v27.2d + trn1 v6.2d, v26.2d, v28.2d + trn2 v7.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v8.2d, v25.2d, v27.2d + trn2 v9.2d, v25.2d, v27.2d + trn1 v10.2d, v26.2d, v28.2d + trn2 v11.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v12.2d, v25.2d, v27.2d + trn2 v13.2d, v25.2d, v27.2d + trn1 v14.2d, v26.2d, v28.2d + trn2 v15.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v16.2d, v25.2d, v27.2d + trn2 v17.2d, v25.2d, v27.2d + trn1 v18.2d, v26.2d, v28.2d + trn2 v19.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v20.2d, v25.2d, v27.2d + trn2 v21.2d, v25.2d, v27.2d + trn1 v22.2d, v26.2d, v28.2d + trn2 v23.2d, v26.2d, v28.2d + ldr d25, [x0] + ldr d27, [x4] + trn1 v24.2d, v25.2d, v27.2d + sub x0, x0, #0xc0 + add x0, x0, #0x190 + ldp x1, x6, [x0] + ldp x11, x16, [x0, #0x10] + ldp x21, x2, [x0, #0x20] + ldp x7, x12, [x0, #0x30] + ldp x17, x22, [x0, #0x40] + ldp x3, x8, [x0, #0x50] + ldp x13, x28, [x0, #0x60] + ldp x23, x4, [x0, #0x70] + ldp x9, x14, [x0, #0x80] + ldp x19, x24, [x0, #0x90] + ldp x5, x10, [x0, #0xa0] + ldp x15, x20, [x0, #0xb0] + ldr x25, [x0, #0xc0] + sub x0, x0, #0x190 + +Lmld_keccak_f1600_x4_v8a_scalar_hybrid_initial: + eor x30, x24, x25 + eor x27, x9, x10 + eor v30.16b, v0.16b, v5.16b + eor v30.16b, v30.16b, v10.16b + eor x0, x30, x21 + eor v30.16b, v30.16b, v15.16b + eor x26, x27, x6 + eor x27, x26, x7 + eor v30.16b, v30.16b, v20.16b + eor x29, x0, x22 + eor v29.16b, v1.16b, v6.16b + eor x26, x29, x23 + eor v29.16b, v29.16b, v11.16b + eor x29, x4, x5 + eor x30, x29, x1 + eor v29.16b, v29.16b, v16.16b + eor x0, x27, x8 + eor v29.16b, v29.16b, v21.16b + eor x29, x30, x2 + eor v28.16b, v2.16b, v7.16b + eor x30, x19, x20 + eor x30, x30, x16 + eor v28.16b, v28.16b, v12.16b + eor x27, x26, x0, ror #63 + eor v28.16b, v28.16b, v17.16b + eor x4, x4, x27 + eor v28.16b, v28.16b, v22.16b + eor x30, x30, x17 + eor x30, x30, x28 + eor v27.16b, v3.16b, v8.16b + eor x29, x29, x3 + eor v27.16b, v27.16b, v13.16b + eor x0, x0, x30, ror #63 + eor v27.16b, v27.16b, v18.16b + eor x30, x30, x29, ror #63 + eor x22, x22, x30 + eor v27.16b, v27.16b, v23.16b + eor x23, x23, x30 + eor v26.16b, v4.16b, v9.16b + str x23, [sp, #0xd0] + eor v26.16b, v26.16b, v14.16b + eor x23, x14, x15 + eor x14, x14, x0 + eor v26.16b, v26.16b, v19.16b + eor x23, x23, x11 + eor v26.16b, v26.16b, v24.16b + eor x15, x15, x0 + eor x1, x1, x27 + add v31.2d, v28.2d, v28.2d + eor x23, x23, x12 + sri v31.2d, v28.2d, #0x3f + eor x23, x23, x13 + eor v25.16b, v31.16b, v30.16b + eor x11, x11, x0 + eor x29, x29, x23, ror #63 + add v31.2d, v26.2d, v26.2d + eor x23, x23, x26, ror #63 + sri v31.2d, v26.2d, #0x3f + eor x26, x13, x0 + eor v28.16b, v31.16b, v28.16b + eor x13, x28, x23 + eor x28, x24, x30 + add v31.2d, v29.2d, v29.2d + eor x24, x16, x23 + sri v31.2d, v29.2d, #0x3f + eor x16, x21, x30 + eor v26.16b, v31.16b, v26.16b + eor x21, x25, x30 + eor x30, x19, x23 + add v31.2d, v27.2d, v27.2d + eor x19, x20, x23 + sri v31.2d, v27.2d, #0x3f + eor x20, x17, x23 + eor v29.16b, v31.16b, v29.16b + eor x17, x12, x0 + eor x0, x2, x27 + add v31.2d, v30.2d, v30.2d + eor x2, x6, x29 + sri v31.2d, v30.2d, #0x3f + eor x6, x8, x29 + eor v27.16b, v31.16b, v27.16b + bic x8, x28, x13, ror #47 + eor x12, x3, x27 + eor v30.16b, v0.16b, v26.16b + bic x3, x13, x17, ror #19 + eor v31.16b, v2.16b, v29.16b + eor x5, x5, x27 + ldr x27, [sp, #0xd0] + shl v0.2d, v31.2d, #0x3e + bic x25, x17, x2, ror #5 + sri v0.2d, v31.2d, #0x2 + eor x9, x9, x29 + eor v31.16b, v12.16b, v29.16b + eor x23, x25, x5, ror #52 + eor x3, x3, x2, ror #24 + shl v2.2d, v31.2d, #0x2b + eor x8, x8, x17, ror #2 + sri v2.2d, v31.2d, #0x15 + eor x17, x10, x29 + eor v31.16b, v13.16b, v28.16b + bic x25, x12, x22, ror #47 + eor x29, x7, x29 + shl v12.2d, v31.2d, #0x19 + bic x10, x4, x27, ror #2 + sri v12.2d, v31.2d, #0x27 + bic x7, x5, x28, ror #10 + eor v31.16b, v19.16b, v27.16b + eor x10, x10, x20, ror #50 + eor x13, x7, x13, ror #57 + shl v13.2d, v31.2d, #0x8 + bic x7, x2, x5, ror #47 + sri v13.2d, v31.2d, #0x38 + eor x2, x25, x24, ror #39 + eor v31.16b, v23.16b, v28.16b + bic x25, x20, x11, ror #57 + bic x5, x17, x4, ror #25 + shl v19.2d, v31.2d, #0x38 + eor x25, x25, x17, ror #53 + sri v19.2d, v31.2d, #0x8 + bic x17, x11, x17, ror #60 + eor v31.16b, v15.16b, v26.16b + eor x28, x7, x28, ror #57 + bic x7, x9, x12, ror #42 + shl v23.2d, v31.2d, #0x29 + eor x7, x7, x22, ror #25 + sri v23.2d, v31.2d, #0x17 + bic x22, x22, x24, ror #56 + bic x24, x24, x15, ror #31 + eor v31.16b, v1.16b, v25.16b + eor x22, x22, x15, ror #23 + shl v15.2d, v31.2d, #0x1 + bic x20, x27, x20, ror #48 + sri v15.2d, v31.2d, #0x3f + bic x15, x15, x9, ror #16 + eor x12, x15, x12, ror #58 + eor v31.16b, v8.16b, v28.16b + eor x15, x5, x27, ror #27 + shl v1.2d, v31.2d, #0x37 + eor x5, x20, x11, ror #41 + sri v1.2d, v31.2d, #0x9 + ldr x11, [sp, #0x8] + eor x20, x17, x4, ror #21 + eor v31.16b, v16.16b, v25.16b + eor x17, x24, x9, ror #47 + shl v8.2d, v31.2d, #0x2d + mov x24, #0x1 // =1 + sri v8.2d, v31.2d, #0x13 + bic x9, x0, x16, ror #9 + str x24, [sp, #0x18] + eor v31.16b, v7.16b, v29.16b + bic x24, x29, x1, ror #44 + shl v16.2d, v31.2d, #0x6 + bic x27, x1, x21, ror #50 + sri v16.2d, v31.2d, #0x3a + bic x4, x26, x29, ror #63 + eor x1, x1, x4, ror #21 + eor v31.16b, v10.16b, v26.16b + ldr x11, [x11] + shl v7.2d, v31.2d, #0x3 + bic x4, x21, x30, ror #57 + sri v7.2d, v31.2d, #0x3d + eor x21, x24, x21, ror #30 + eor x24, x9, x19, ror #44 + eor v31.16b, v3.16b, v28.16b + bic x9, x14, x6, ror #5 + shl v10.2d, v31.2d, #0x1c + eor x9, x9, x0, ror #43 + sri v10.2d, v31.2d, #0x24 + bic x0, x6, x0, ror #38 + eor x1, x1, x11 + eor v31.16b, v18.16b, v28.16b + eor x11, x4, x26, ror #35 + shl v3.2d, v31.2d, #0x15 + eor x4, x0, x16, ror #47 + bic x0, x16, x19, ror #35 + sri v3.2d, v31.2d, #0x2b + eor x16, x27, x30, ror #43 + eor v31.16b, v17.16b, v29.16b + bic x27, x30, x26, ror #42 + shl v18.2d, v31.2d, #0xf + bic x26, x19, x14, ror #41 + eor x19, x0, x14, ror #12 + sri v18.2d, v31.2d, #0x31 + eor x14, x26, x6, ror #46 + eor v31.16b, v11.16b, v25.16b + eor x6, x27, x29, ror #41 + shl v17.2d, v31.2d, #0xa + eor x0, x15, x11, ror #52 + eor x0, x0, x13, ror #48 + sri v17.2d, v31.2d, #0x36 + eor x26, x8, x9, ror #57 + eor v31.16b, v9.16b, v27.16b + eor x27, x0, x14, ror #10 + shl v11.2d, v31.2d, #0x14 + eor x29, x16, x28, ror #63 + eor x26, x26, x6, ror #51 + sri v11.2d, v31.2d, #0x2c + eor x30, x23, x22, ror #50 + eor v31.16b, v22.16b, v29.16b + eor x0, x26, x10, ror #31 + shl v9.2d, v31.2d, #0x3d + eor x29, x29, x19, ror #37 + eor x27, x27, x12, ror #5 + sri v9.2d, v31.2d, #0x3 + eor x30, x30, x24, ror #34 + eor v31.16b, v14.16b, v27.16b + eor x0, x0, x7, ror #27 + shl v22.2d, v31.2d, #0x27 + eor x26, x30, x21, ror #26 + eor x26, x26, x25, ror #15 + sri v22.2d, v31.2d, #0x19 + ror x30, x27, #0x3e + eor v31.16b, v20.16b, v26.16b + eor x30, x30, x26, ror #57 + ror x26, x26, #0x3a + shl v14.2d, v31.2d, #0x12 + eor x16, x30, x16 + sri v14.2d, v31.2d, #0x2e + eor x28, x30, x28, ror #63 + eor v31.16b, v4.16b, v27.16b + str x28, [sp, #0xd0] + eor x29, x29, x17, ror #36 + shl v20.2d, v31.2d, #0x1b + eor x28, x1, x2, ror #61 + sri v20.2d, v31.2d, #0x25 + eor x19, x30, x19, ror #37 + eor v31.16b, v24.16b, v27.16b + eor x29, x29, x20, ror #2 + eor x28, x28, x4, ror #54 + shl v4.2d, v31.2d, #0xe + eor x26, x26, x0, ror #55 + sri v4.2d, v31.2d, #0x32 + eor x28, x28, x3, ror #39 + eor v31.16b, v21.16b, v25.16b + eor x28, x28, x5, ror #25 + ror x0, x0, #0x38 + shl v24.2d, v31.2d, #0x2 + eor x0, x0, x29, ror #63 + sri v24.2d, v31.2d, #0x3e + eor x27, x28, x27, ror #61 + eor v31.16b, v5.16b, v26.16b + eor x13, x0, x13, ror #46 + eor x28, x29, x28, ror #63 + shl v21.2d, v31.2d, #0x24 + eor x29, x30, x20, ror #2 + sri v21.2d, v31.2d, #0x1c + eor x20, x26, x3, ror #39 + eor v31.16b, v6.16b, v25.16b + eor x11, x0, x11, ror #50 + eor x25, x28, x25, ror #9 + shl v27.2d, v31.2d, #0x2c + eor x3, x28, x21, ror #20 + sri v27.2d, v31.2d, #0x14 + eor x21, x26, x1 + eor x9, x27, x9, ror #49 + bic v31.16b, v7.16b, v11.16b + eor x24, x28, x24, ror #28 + eor v5.16b, v31.16b, v10.16b + eor x1, x30, x17, ror #36 + bic v31.16b, v8.16b, v7.16b + eor x14, x0, x14, ror #8 + eor x22, x28, x22, ror #44 + eor v6.16b, v31.16b, v11.16b + eor x8, x27, x8, ror #56 + bic v31.16b, v9.16b, v8.16b + eor x17, x27, x7, ror #19 + eor v7.16b, v31.16b, v7.16b + eor x15, x0, x15, ror #62 + bic x7, x20, x22, ror #47 + bic v31.16b, v10.16b, v9.16b + eor x4, x26, x4, ror #54 + eor v8.16b, v31.16b, v8.16b + eor x0, x0, x12, ror #3 + bic v31.16b, v11.16b, v10.16b + eor x28, x28, x23, ror #58 + eor x23, x26, x2, ror #61 + eor v9.16b, v31.16b, v9.16b + eor x26, x26, x5, ror #25 + bic v31.16b, v12.16b, v16.16b + eor x2, x7, x16, ror #39 + eor v10.16b, v31.16b, v15.16b + bic x7, x9, x20, ror #42 + bic x30, x15, x9, ror #16 + bic v31.16b, v13.16b, v12.16b + eor x7, x7, x22, ror #25 + eor v11.16b, v31.16b, v16.16b + eor x12, x30, x20, ror #58 + bic v31.16b, v14.16b, v13.16b + bic x20, x22, x16, ror #56 + eor x30, x27, x6, ror #43 + eor v12.16b, v31.16b, v12.16b + eor x22, x20, x15, ror #23 + bic v31.16b, v15.16b, v14.16b + bic x6, x19, x13, ror #42 + eor v13.16b, v31.16b, v13.16b + eor x6, x6, x17, ror #41 + bic x5, x13, x17, ror #63 + bic v31.16b, v16.16b, v15.16b + eor x5, x21, x5, ror #21 + eor v14.16b, v31.16b, v14.16b + bic x17, x17, x21, ror #44 + eor x27, x27, x10, ror #23 + bic v31.16b, v17.16b, v21.16b + bic x21, x21, x25, ror #50 + eor v15.16b, v31.16b, v20.16b + bic x20, x27, x4, ror #25 + bic v31.16b, v18.16b, v17.16b + bic x10, x16, x15, ror #31 + eor x16, x21, x19, ror #43 + eor v16.16b, v31.16b, v21.16b + eor x21, x17, x25, ror #30 + bic v31.16b, v19.16b, v18.16b + bic x19, x25, x19, ror #57 + eor v17.16b, v31.16b, v17.16b + ldr x25, [sp, #0x18] + eor x17, x10, x9, ror #47 + bic v31.16b, v20.16b, v19.16b + ldr x9, [sp, #0x8] + eor v18.16b, v31.16b, v18.16b + eor x15, x20, x28, ror #27 + bic v31.16b, v21.16b, v20.16b + bic x20, x4, x28, ror #2 + eor x10, x20, x1, ror #50 + eor v19.16b, v31.16b, v19.16b + bic x20, x11, x27, ror #60 + bic v31.16b, v22.16b, v1.16b + eor x20, x20, x4, ror #21 + eor v20.16b, v31.16b, v0.16b + bic x4, x28, x1, ror #48 + bic x1, x1, x11, ror #57 + bic v31.16b, v23.16b, v22.16b + ldr x28, [x9, x25, lsl #3] + eor v21.16b, v31.16b, v1.16b + ldr x9, [sp, #0xd0] + bic v31.16b, v24.16b, v23.16b + add x25, x25, #0x1 + str x25, [sp, #0x18] + eor v22.16b, v31.16b, v22.16b + cmp x25, #0x17 + bic v31.16b, v0.16b, v24.16b + eor x25, x1, x27, ror #53 + bic x27, x30, x26, ror #47 + eor v23.16b, v31.16b, v23.16b + eor x1, x5, x28 + bic v31.16b, v1.16b, v0.16b + eor x5, x4, x11, ror #41 + eor v24.16b, v31.16b, v24.16b + eor x11, x19, x13, ror #35 + bic x13, x26, x24, ror #10 + bic v31.16b, v2.16b, v27.16b + eor x28, x27, x24, ror #57 + eor v0.16b, v31.16b, v30.16b + bic x27, x24, x9, ror #47 + bic v31.16b, v3.16b, v2.16b + bic x19, x23, x3, ror #9 + bic x4, x29, x14, ror #41 + eor v1.16b, v31.16b, v27.16b + eor x24, x19, x29, ror #44 + bic v31.16b, v4.16b, v3.16b + bic x29, x3, x29, ror #35 + eor v2.16b, v31.16b, v2.16b + eor x13, x13, x9, ror #57 + eor x19, x29, x14, ror #12 + bic v31.16b, v30.16b, v4.16b + bic x29, x9, x0, ror #19 + eor v3.16b, v31.16b, v3.16b + bic x14, x14, x8, ror #5 + bic v31.16b, v27.16b, v30.16b + eor x9, x14, x23, ror #43 + eor x14, x4, x8, ror #46 + eor v4.16b, v31.16b, v4.16b + bic x23, x8, x23, ror #38 + eor x8, x27, x0, ror #2 + eor x4, x23, x3, ror #47 + bic x3, x0, x30, ror #5 + eor x23, x3, x26, ror #52 + eor x3, x29, x30, ror #24 + ldr x30, [sp, #0x10] + ld1r { v28.2d }, [x30], #8 + str x30, [sp, #0x10] + eor v0.16b, v0.16b, v28.16b + +Lmld_keccak_f1600_x4_v8a_scalar_hybrid_loop: + eor x0, x15, x11, ror #52 + eor x0, x0, x13, ror #48 + eor v30.16b, v0.16b, v5.16b + eor v30.16b, v30.16b, v10.16b + eor x26, x8, x9, ror #57 + eor v30.16b, v30.16b, v15.16b + eor x27, x0, x14, ror #10 + eor x29, x16, x28, ror #63 + eor v30.16b, v30.16b, v20.16b + eor x26, x26, x6, ror #51 + eor v29.16b, v1.16b, v6.16b + eor x30, x23, x22, ror #50 + eor v29.16b, v29.16b, v11.16b + eor x0, x26, x10, ror #31 + eor x29, x29, x19, ror #37 + eor v29.16b, v29.16b, v16.16b + eor x27, x27, x12, ror #5 + eor v29.16b, v29.16b, v21.16b + eor x30, x30, x24, ror #34 + eor x0, x0, x7, ror #27 + eor v28.16b, v2.16b, v7.16b + eor x26, x30, x21, ror #26 + eor v28.16b, v28.16b, v12.16b + eor x26, x26, x25, ror #15 + eor v28.16b, v28.16b, v17.16b + ror x30, x27, #0x3e + eor x30, x30, x26, ror #57 + eor v28.16b, v28.16b, v22.16b + ror x26, x26, #0x3a + eor v27.16b, v3.16b, v8.16b + eor x16, x30, x16 + eor v27.16b, v27.16b, v13.16b + eor x28, x30, x28, ror #63 + str x28, [sp, #0xd0] + eor v27.16b, v27.16b, v18.16b + eor x29, x29, x17, ror #36 + eor v27.16b, v27.16b, v23.16b + eor x28, x1, x2, ror #61 + eor x19, x30, x19, ror #37 + eor v26.16b, v4.16b, v9.16b + eor x29, x29, x20, ror #2 + eor v26.16b, v26.16b, v14.16b + eor x28, x28, x4, ror #54 + eor v26.16b, v26.16b, v19.16b + eor x26, x26, x0, ror #55 + eor x28, x28, x3, ror #39 + eor v26.16b, v26.16b, v24.16b + eor x28, x28, x5, ror #25 + add v31.2d, v28.2d, v28.2d + ror x0, x0, #0x38 + eor x0, x0, x29, ror #63 + sri v31.2d, v28.2d, #0x3f + eor x27, x28, x27, ror #61 + eor v25.16b, v31.16b, v30.16b + eor x13, x0, x13, ror #46 + add v31.2d, v26.2d, v26.2d + eor x28, x29, x28, ror #63 + eor x29, x30, x20, ror #2 + sri v31.2d, v26.2d, #0x3f + eor x20, x26, x3, ror #39 + eor v28.16b, v31.16b, v28.16b + eor x11, x0, x11, ror #50 + add v31.2d, v29.2d, v29.2d + eor x25, x28, x25, ror #9 + eor x3, x28, x21, ror #20 + sri v31.2d, v29.2d, #0x3f + eor x21, x26, x1 + eor v26.16b, v31.16b, v26.16b + eor x9, x27, x9, ror #49 + eor x24, x28, x24, ror #28 + add v31.2d, v27.2d, v27.2d + eor x1, x30, x17, ror #36 + sri v31.2d, v27.2d, #0x3f + eor x14, x0, x14, ror #8 + eor v29.16b, v31.16b, v29.16b + eor x22, x28, x22, ror #44 + eor x8, x27, x8, ror #56 + add v31.2d, v30.2d, v30.2d + eor x17, x27, x7, ror #19 + sri v31.2d, v30.2d, #0x3f + eor x15, x0, x15, ror #62 + bic x7, x20, x22, ror #47 + eor v27.16b, v31.16b, v27.16b + eor x4, x26, x4, ror #54 + eor v30.16b, v0.16b, v26.16b + eor x0, x0, x12, ror #3 + eor v31.16b, v2.16b, v29.16b + eor x28, x28, x23, ror #58 + eor x23, x26, x2, ror #61 + shl v0.2d, v31.2d, #0x3e + eor x26, x26, x5, ror #25 + sri v0.2d, v31.2d, #0x2 + eor x2, x7, x16, ror #39 + eor v31.16b, v12.16b, v29.16b + bic x7, x9, x20, ror #42 + bic x30, x15, x9, ror #16 + shl v2.2d, v31.2d, #0x2b + eor x7, x7, x22, ror #25 + sri v2.2d, v31.2d, #0x15 + eor x12, x30, x20, ror #58 + bic x20, x22, x16, ror #56 + eor v31.16b, v13.16b, v28.16b + eor x30, x27, x6, ror #43 + shl v12.2d, v31.2d, #0x19 + eor x22, x20, x15, ror #23 + sri v12.2d, v31.2d, #0x27 + bic x6, x19, x13, ror #42 + eor x6, x6, x17, ror #41 + eor v31.16b, v19.16b, v27.16b + bic x5, x13, x17, ror #63 + shl v13.2d, v31.2d, #0x8 + eor x5, x21, x5, ror #21 + sri v13.2d, v31.2d, #0x38 + bic x17, x17, x21, ror #44 + eor x27, x27, x10, ror #23 + eor v31.16b, v23.16b, v28.16b + bic x21, x21, x25, ror #50 + shl v19.2d, v31.2d, #0x38 + bic x20, x27, x4, ror #25 + bic x10, x16, x15, ror #31 + sri v19.2d, v31.2d, #0x8 + eor x16, x21, x19, ror #43 + eor v31.16b, v15.16b, v26.16b + eor x21, x17, x25, ror #30 + shl v23.2d, v31.2d, #0x29 + bic x19, x25, x19, ror #57 + ldr x25, [sp, #0x18] + sri v23.2d, v31.2d, #0x17 + eor x17, x10, x9, ror #47 + eor v31.16b, v1.16b, v25.16b + ldr x9, [sp, #0x8] + eor x15, x20, x28, ror #27 + shl v15.2d, v31.2d, #0x1 + bic x20, x4, x28, ror #2 + sri v15.2d, v31.2d, #0x3f + eor x10, x20, x1, ror #50 + eor v31.16b, v8.16b, v28.16b + bic x20, x11, x27, ror #60 + eor x20, x20, x4, ror #21 + shl v1.2d, v31.2d, #0x37 + bic x4, x28, x1, ror #48 + sri v1.2d, v31.2d, #0x9 + bic x1, x1, x11, ror #57 + eor v31.16b, v16.16b, v25.16b + ldr x28, [x9, x25, lsl #3] + ldr x9, [sp, #0xd0] + shl v8.2d, v31.2d, #0x2d + add x25, x25, #0x1 + sri v8.2d, v31.2d, #0x13 + str x25, [sp, #0x18] + cmp x25, #0x17 + eor v31.16b, v7.16b, v29.16b + eor x25, x1, x27, ror #53 + shl v16.2d, v31.2d, #0x6 + bic x27, x30, x26, ror #47 + sri v16.2d, v31.2d, #0x3a + eor x1, x5, x28 + eor x5, x4, x11, ror #41 + eor v31.16b, v10.16b, v26.16b + eor x11, x19, x13, ror #35 + shl v7.2d, v31.2d, #0x3 + bic x13, x26, x24, ror #10 + eor x28, x27, x24, ror #57 + sri v7.2d, v31.2d, #0x3d + bic x27, x24, x9, ror #47 + eor v31.16b, v3.16b, v28.16b + bic x19, x23, x3, ror #9 + shl v10.2d, v31.2d, #0x1c + bic x4, x29, x14, ror #41 + eor x24, x19, x29, ror #44 + sri v10.2d, v31.2d, #0x24 + bic x29, x3, x29, ror #35 + eor v31.16b, v18.16b, v28.16b + eor x13, x13, x9, ror #57 + shl v3.2d, v31.2d, #0x15 + eor x19, x29, x14, ror #12 + bic x29, x9, x0, ror #19 + sri v3.2d, v31.2d, #0x2b + bic x14, x14, x8, ror #5 + eor v31.16b, v17.16b, v29.16b + eor x9, x14, x23, ror #43 + eor x14, x4, x8, ror #46 + shl v18.2d, v31.2d, #0xf + bic x23, x8, x23, ror #38 + sri v18.2d, v31.2d, #0x31 + eor x8, x27, x0, ror #2 + eor v31.16b, v11.16b, v25.16b + eor x4, x23, x3, ror #47 + bic x3, x0, x30, ror #5 + shl v17.2d, v31.2d, #0xa + eor x23, x3, x26, ror #52 + sri v17.2d, v31.2d, #0x36 + eor x3, x29, x30, ror #24 + eor x0, x15, x11, ror #52 + eor v31.16b, v9.16b, v27.16b + eor x0, x0, x13, ror #48 + shl v11.2d, v31.2d, #0x14 + eor x26, x8, x9, ror #57 + sri v11.2d, v31.2d, #0x2c + eor x27, x0, x14, ror #10 + eor x29, x16, x28, ror #63 + eor v31.16b, v22.16b, v29.16b + eor x26, x26, x6, ror #51 + shl v9.2d, v31.2d, #0x3d + eor x30, x23, x22, ror #50 + sri v9.2d, v31.2d, #0x3 + eor x0, x26, x10, ror #31 + eor x29, x29, x19, ror #37 + eor v31.16b, v14.16b, v27.16b + eor x27, x27, x12, ror #5 + shl v22.2d, v31.2d, #0x27 + eor x30, x30, x24, ror #34 + eor x0, x0, x7, ror #27 + sri v22.2d, v31.2d, #0x19 + eor x26, x30, x21, ror #26 + eor v31.16b, v20.16b, v26.16b + eor x26, x26, x25, ror #15 + shl v14.2d, v31.2d, #0x12 + ror x30, x27, #0x3e + eor x30, x30, x26, ror #57 + sri v14.2d, v31.2d, #0x2e + ror x26, x26, #0x3a + eor v31.16b, v4.16b, v27.16b + eor x16, x30, x16 + shl v20.2d, v31.2d, #0x1b + eor x28, x30, x28, ror #63 + str x28, [sp, #0xd0] + sri v20.2d, v31.2d, #0x25 + eor x29, x29, x17, ror #36 + eor v31.16b, v24.16b, v27.16b + eor x28, x1, x2, ror #61 + eor x19, x30, x19, ror #37 + shl v4.2d, v31.2d, #0xe + eor x29, x29, x20, ror #2 + sri v4.2d, v31.2d, #0x32 + eor x28, x28, x4, ror #54 + eor v31.16b, v21.16b, v25.16b + eor x26, x26, x0, ror #55 + eor x28, x28, x3, ror #39 + shl v24.2d, v31.2d, #0x2 + eor x28, x28, x5, ror #25 + sri v24.2d, v31.2d, #0x3e + ror x0, x0, #0x38 + eor x0, x0, x29, ror #63 + eor v31.16b, v5.16b, v26.16b + eor x27, x28, x27, ror #61 + shl v21.2d, v31.2d, #0x24 + eor x13, x0, x13, ror #46 + sri v21.2d, v31.2d, #0x1c + eor x28, x29, x28, ror #63 + eor x29, x30, x20, ror #2 + eor v31.16b, v6.16b, v25.16b + eor x20, x26, x3, ror #39 + shl v27.2d, v31.2d, #0x2c + eor x11, x0, x11, ror #50 + sri v27.2d, v31.2d, #0x14 + eor x25, x28, x25, ror #9 + eor x3, x28, x21, ror #20 + bic v31.16b, v7.16b, v11.16b + eor x21, x26, x1 + eor v5.16b, v31.16b, v10.16b + eor x9, x27, x9, ror #49 + eor x24, x28, x24, ror #28 + bic v31.16b, v8.16b, v7.16b + eor x1, x30, x17, ror #36 + eor v6.16b, v31.16b, v11.16b + eor x14, x0, x14, ror #8 + bic v31.16b, v9.16b, v8.16b + eor x22, x28, x22, ror #44 + eor x8, x27, x8, ror #56 + eor v7.16b, v31.16b, v7.16b + eor x17, x27, x7, ror #19 + bic v31.16b, v10.16b, v9.16b + eor x15, x0, x15, ror #62 + bic x7, x20, x22, ror #47 + eor v8.16b, v31.16b, v8.16b + eor x4, x26, x4, ror #54 + bic v31.16b, v11.16b, v10.16b + eor x0, x0, x12, ror #3 + eor v9.16b, v31.16b, v9.16b + eor x28, x28, x23, ror #58 + eor x23, x26, x2, ror #61 + bic v31.16b, v12.16b, v16.16b + eor x26, x26, x5, ror #25 + eor v10.16b, v31.16b, v15.16b + eor x2, x7, x16, ror #39 + bic v31.16b, v13.16b, v12.16b + bic x7, x9, x20, ror #42 + bic x30, x15, x9, ror #16 + eor v11.16b, v31.16b, v16.16b + eor x7, x7, x22, ror #25 + bic v31.16b, v14.16b, v13.16b + eor x12, x30, x20, ror #58 + bic x20, x22, x16, ror #56 + eor v12.16b, v31.16b, v12.16b + eor x30, x27, x6, ror #43 + bic v31.16b, v15.16b, v14.16b + eor x22, x20, x15, ror #23 + eor v13.16b, v31.16b, v13.16b + bic x6, x19, x13, ror #42 + eor x6, x6, x17, ror #41 + bic v31.16b, v16.16b, v15.16b + bic x5, x13, x17, ror #63 + eor v14.16b, v31.16b, v14.16b + eor x5, x21, x5, ror #21 + bic v31.16b, v17.16b, v21.16b + bic x17, x17, x21, ror #44 + eor x27, x27, x10, ror #23 + eor v15.16b, v31.16b, v20.16b + bic x21, x21, x25, ror #50 + bic v31.16b, v18.16b, v17.16b + bic x20, x27, x4, ror #25 + bic x10, x16, x15, ror #31 + eor v16.16b, v31.16b, v21.16b + eor x16, x21, x19, ror #43 + bic v31.16b, v19.16b, v18.16b + eor x21, x17, x25, ror #30 + eor v17.16b, v31.16b, v17.16b + bic x19, x25, x19, ror #57 + ldr x25, [sp, #0x18] + bic v31.16b, v20.16b, v19.16b + eor x17, x10, x9, ror #47 + eor v18.16b, v31.16b, v18.16b + ldr x9, [sp, #0x8] + eor x15, x20, x28, ror #27 + bic v31.16b, v21.16b, v20.16b + bic x20, x4, x28, ror #2 + eor v19.16b, v31.16b, v19.16b + eor x10, x20, x1, ror #50 + bic v31.16b, v22.16b, v1.16b + bic x20, x11, x27, ror #60 + eor x20, x20, x4, ror #21 + eor v20.16b, v31.16b, v0.16b + bic x4, x28, x1, ror #48 + bic v31.16b, v23.16b, v22.16b + bic x1, x1, x11, ror #57 + eor v21.16b, v31.16b, v1.16b + ldr x28, [x9, x25, lsl #3] + ldr x9, [sp, #0xd0] + bic v31.16b, v24.16b, v23.16b + add x25, x25, #0x1 + eor v22.16b, v31.16b, v22.16b + str x25, [sp, #0x18] + cmp x25, #0x17 + bic v31.16b, v0.16b, v24.16b + eor x25, x1, x27, ror #53 + eor v23.16b, v31.16b, v23.16b + bic x27, x30, x26, ror #47 + bic v31.16b, v1.16b, v0.16b + eor x1, x5, x28 + eor x5, x4, x11, ror #41 + eor v24.16b, v31.16b, v24.16b + eor x11, x19, x13, ror #35 + bic v31.16b, v2.16b, v27.16b + bic x13, x26, x24, ror #10 + eor x28, x27, x24, ror #57 + eor v0.16b, v31.16b, v30.16b + bic x27, x24, x9, ror #47 + bic v31.16b, v3.16b, v2.16b + bic x19, x23, x3, ror #9 + eor v1.16b, v31.16b, v27.16b + bic x4, x29, x14, ror #41 + eor x24, x19, x29, ror #44 + bic v31.16b, v4.16b, v3.16b + bic x29, x3, x29, ror #35 + eor v2.16b, v31.16b, v2.16b + eor x13, x13, x9, ror #57 + bic v31.16b, v30.16b, v4.16b + eor x19, x29, x14, ror #12 + bic x29, x9, x0, ror #19 + eor v3.16b, v31.16b, v3.16b + bic x14, x14, x8, ror #5 + bic v31.16b, v27.16b, v30.16b + eor x9, x14, x23, ror #43 + eor x14, x4, x8, ror #46 + eor v4.16b, v31.16b, v4.16b + bic x23, x8, x23, ror #38 + eor x8, x27, x0, ror #2 + eor x4, x23, x3, ror #47 + bic x3, x0, x30, ror #5 + eor x23, x3, x26, ror #52 + eor x3, x29, x30, ror #24 + ldr x30, [sp, #0x10] + ld1r { v28.2d }, [x30], #8 + str x30, [sp, #0x10] + eor v0.16b, v0.16b, v28.16b + +Lmld_keccak_f1600_x4_v8a_scalar_hybrid_loop_end: + b.le Lmld_keccak_f1600_x4_v8a_scalar_hybrid_loop + ror x2, x2, #0x3d + ror x3, x3, #0x27 + ror x4, x4, #0x36 + ror x5, x5, #0x19 + ror x6, x6, #0x2b + ror x7, x7, #0x13 + ror x8, x8, #0x38 + ror x9, x9, #0x31 + ror x10, x10, #0x17 + ror x11, x11, #0x32 + ror x12, x12, #0x3 + ror x13, x13, #0x2e + ror x14, x14, #0x8 + ror x15, x15, #0x3e + ror x17, x17, #0x24 + ror x28, x28, #0x3f + ror x19, x19, #0x25 + ror x20, x20, #0x2 + ror x21, x21, #0x14 + ror x22, x22, #0x2c + ror x23, x23, #0x3a + ror x24, x24, #0x1c + ror x25, x25, #0x9 + ldr x30, [sp, #0x20] + cmp x30, #0x1 + b.eq Lmld_keccak_f1600_x4_v8a_scalar_hybrid_done + mov x30, #0x1 // =1 + str x30, [sp, #0x20] + ldr x0, [sp] + add x0, x0, #0x190 + stp x1, x6, [x0] + stp x11, x16, [x0, #0x10] + stp x21, x2, [x0, #0x20] + stp x7, x12, [x0, #0x30] + stp x17, x22, [x0, #0x40] + stp x3, x8, [x0, #0x50] + stp x13, x28, [x0, #0x60] + stp x23, x4, [x0, #0x70] + stp x9, x14, [x0, #0x80] + stp x19, x24, [x0, #0x90] + stp x5, x10, [x0, #0xa0] + stp x15, x20, [x0, #0xb0] + str x25, [x0, #0xc0] + sub x0, x0, #0x190 + add x0, x0, #0x258 + ldp x1, x6, [x0] + ldp x11, x16, [x0, #0x10] + ldp x21, x2, [x0, #0x20] + ldp x7, x12, [x0, #0x30] + ldp x17, x22, [x0, #0x40] + ldp x3, x8, [x0, #0x50] + ldp x13, x28, [x0, #0x60] + ldp x23, x4, [x0, #0x70] + ldp x9, x14, [x0, #0x80] + ldp x19, x24, [x0, #0x90] + ldp x5, x10, [x0, #0xa0] + ldp x15, x20, [x0, #0xb0] + ldr x25, [x0, #0xc0] + sub x0, x0, #0x258 + b Lmld_keccak_f1600_x4_v8a_scalar_hybrid_initial + +Lmld_keccak_f1600_x4_v8a_scalar_hybrid_done: + ldr x0, [sp] + add x0, x0, #0x258 + stp x1, x6, [x0] + stp x11, x16, [x0, #0x10] + stp x21, x2, [x0, #0x20] + stp x7, x12, [x0, #0x30] + stp x17, x22, [x0, #0x40] + stp x3, x8, [x0, #0x50] + stp x13, x28, [x0, #0x60] + stp x23, x4, [x0, #0x70] + stp x9, x14, [x0, #0x80] + stp x19, x24, [x0, #0x90] + stp x5, x10, [x0, #0xa0] + stp x15, x20, [x0, #0xb0] + str x25, [x0, #0xc0] + sub x0, x0, #0x258 + add x4, x0, #0xc8 + trn1 v25.2d, v0.2d, v1.2d + trn1 v26.2d, v2.2d, v3.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v0.2d, v1.2d + trn2 v28.2d, v2.2d, v3.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v4.2d, v5.2d + trn1 v26.2d, v6.2d, v7.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v4.2d, v5.2d + trn2 v28.2d, v6.2d, v7.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v8.2d, v9.2d + trn1 v26.2d, v10.2d, v11.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v8.2d, v9.2d + trn2 v28.2d, v10.2d, v11.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v12.2d, v13.2d + trn1 v26.2d, v14.2d, v15.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v12.2d, v13.2d + trn2 v28.2d, v14.2d, v15.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v16.2d, v17.2d + trn1 v26.2d, v18.2d, v19.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v16.2d, v17.2d + trn2 v28.2d, v18.2d, v19.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v20.2d, v21.2d + trn1 v26.2d, v22.2d, v23.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v20.2d, v21.2d + trn2 v28.2d, v22.2d, v23.2d + st1 { v27.2d, v28.2d }, [x4], #32 + str d24, [x0] + trn2 v25.2d, v24.2d, v24.2d + str d25, [x4] + ldp d8, d9, [sp, #0x90] + .cfi_restore d8 + .cfi_restore d9 + ldp d10, d11, [sp, #0xa0] + .cfi_restore d10 + .cfi_restore d11 + ldp d12, d13, [sp, #0xb0] + .cfi_restore d12 + .cfi_restore d13 + ldp d14, d15, [sp, #0xc0] + .cfi_restore d14 + .cfi_restore d15 + ldp x19, x20, [sp, #0x30] + .cfi_restore x19 + .cfi_restore x20 + ldp x21, x22, [sp, #0x40] + .cfi_restore x21 + .cfi_restore x22 + ldp x23, x24, [sp, #0x50] + .cfi_restore x23 + .cfi_restore x24 + ldp x25, x26, [sp, #0x60] + .cfi_restore x25 + .cfi_restore x26 + ldp x27, x28, [sp, #0x70] + .cfi_restore x27 + .cfi_restore x28 + ldp x29, x30, [sp, #0x80] + .cfi_restore x29 + .cfi_restore x30 + add sp, sp, #0xe0 + .cfi_adjust_cfa_offset -0xe0 + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm) + +#endif /* MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S b/cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S @@ -0,0 +1,990 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * Copyright (c) 2021-2022 Arm Limited + * Copyright (c) 2022 Matthias Kannwischer + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +// Author: Hanno Becker +// Author: Matthias Kannwischer + +/*yaml + Name: keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm + Description: AArch64 hybrid scalar/vector implementation of Keccak-f[1600] permutation for four sequential states with ARMv8.4-A optimizations + Signature: void mld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm(uint64_t state[100], const uint64_t rc[24]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON, SHA3] + x0: + type: buffer + size_bytes: 800 + permissions: read/write + c_parameter: uint64_t state[100] + description: Four sequential Keccak states (state0[25], state1[25], state2[25], state3[25]) + x1: + type: buffer + size_bytes: 192 + permissions: read-only + c_parameter: uint64_t const *rc + description: Round constants (24 x uint64_t) + Stack: + bytes: 224 + description: register preservation and temporary storage +*/ + +#include "../../../../common.h" +#if defined(MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +#if defined(__ARM_FEATURE_SHA3) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/fips202/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm) +MLD_ASM_FN_SYMBOL(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0xe0 + .cfi_adjust_cfa_offset 0xe0 + stp x19, x20, [sp, #0x30] + .cfi_rel_offset x19, 0x30 + .cfi_rel_offset x20, 0x38 + stp x21, x22, [sp, #0x40] + .cfi_rel_offset x21, 0x40 + .cfi_rel_offset x22, 0x48 + stp x23, x24, [sp, #0x50] + .cfi_rel_offset x23, 0x50 + .cfi_rel_offset x24, 0x58 + stp x25, x26, [sp, #0x60] + .cfi_rel_offset x25, 0x60 + .cfi_rel_offset x26, 0x68 + stp x27, x28, [sp, #0x70] + .cfi_rel_offset x27, 0x70 + .cfi_rel_offset x28, 0x78 + stp x29, x30, [sp, #0x80] + .cfi_rel_offset x29, 0x80 + .cfi_rel_offset x30, 0x88 + stp d8, d9, [sp, #0x90] + .cfi_rel_offset d8, 0x90 + .cfi_rel_offset d9, 0x98 + stp d10, d11, [sp, #0xa0] + .cfi_rel_offset d10, 0xa0 + .cfi_rel_offset d11, 0xa8 + stp d12, d13, [sp, #0xb0] + .cfi_rel_offset d12, 0xb0 + .cfi_rel_offset d13, 0xb8 + stp d14, d15, [sp, #0xc0] + .cfi_rel_offset d14, 0xc0 + .cfi_rel_offset d15, 0xc8 + mov x29, x1 + mov x30, #0x0 // =0 + str x30, [sp, #0x20] + str x29, [sp, #0x8] + str x29, [sp, #0x10] + str x0, [sp] + add x4, x0, #0xc8 + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v0.2d, v25.2d, v27.2d + trn2 v1.2d, v25.2d, v27.2d + trn1 v2.2d, v26.2d, v28.2d + trn2 v3.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v4.2d, v25.2d, v27.2d + trn2 v5.2d, v25.2d, v27.2d + trn1 v6.2d, v26.2d, v28.2d + trn2 v7.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v8.2d, v25.2d, v27.2d + trn2 v9.2d, v25.2d, v27.2d + trn1 v10.2d, v26.2d, v28.2d + trn2 v11.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v12.2d, v25.2d, v27.2d + trn2 v13.2d, v25.2d, v27.2d + trn1 v14.2d, v26.2d, v28.2d + trn2 v15.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v16.2d, v25.2d, v27.2d + trn2 v17.2d, v25.2d, v27.2d + trn1 v18.2d, v26.2d, v28.2d + trn2 v19.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v20.2d, v25.2d, v27.2d + trn2 v21.2d, v25.2d, v27.2d + trn1 v22.2d, v26.2d, v28.2d + trn2 v23.2d, v26.2d, v28.2d + ldr d25, [x0] + ldr d27, [x4] + trn1 v24.2d, v25.2d, v27.2d + sub x0, x0, #0xc0 + add x0, x0, #0x190 + ldp x1, x6, [x0] + ldp x11, x16, [x0, #0x10] + ldp x21, x2, [x0, #0x20] + ldp x7, x12, [x0, #0x30] + ldp x17, x22, [x0, #0x40] + ldp x3, x8, [x0, #0x50] + ldp x13, x28, [x0, #0x60] + ldp x23, x4, [x0, #0x70] + ldp x9, x14, [x0, #0x80] + ldp x19, x24, [x0, #0x90] + ldp x5, x10, [x0, #0xa0] + ldp x15, x20, [x0, #0xb0] + ldr x25, [x0, #0xc0] + sub x0, x0, #0x190 + +Lmld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_initial: + eor x30, x24, x25 + eor x27, x9, x10 + eor3 v30.16b, v0.16b, v5.16b, v10.16b + eor v30.16b, v30.16b, v15.16b + eor x0, x30, x21 + eor x26, x27, x6 + eor v30.16b, v30.16b, v20.16b + eor x27, x26, x7 + eor x29, x0, x22 + eor3 v29.16b, v1.16b, v6.16b, v11.16b + eor x26, x29, x23 + eor x29, x4, x5 + eor v29.16b, v29.16b, v16.16b + eor x30, x29, x1 + eor x0, x27, x8 + eor v29.16b, v29.16b, v21.16b + eor x29, x30, x2 + eor x30, x19, x20 + eor3 v28.16b, v2.16b, v7.16b, v12.16b + eor x30, x30, x16 + eor x27, x26, x0, ror #63 + eor v28.16b, v28.16b, v17.16b + eor x4, x4, x27 + eor x30, x30, x17 + eor v28.16b, v28.16b, v22.16b + eor x30, x30, x28 + eor x29, x29, x3 + eor3 v27.16b, v3.16b, v8.16b, v13.16b + eor x0, x0, x30, ror #63 + eor x30, x30, x29, ror #63 + eor v27.16b, v27.16b, v18.16b + eor x22, x22, x30 + eor v27.16b, v27.16b, v23.16b + eor x23, x23, x30 + str x23, [sp, #0xd0] + eor3 v26.16b, v4.16b, v9.16b, v14.16b + eor x23, x14, x15 + eor x14, x14, x0 + eor v26.16b, v26.16b, v19.16b + eor x23, x23, x11 + eor x15, x15, x0 + eor v26.16b, v26.16b, v24.16b + eor x1, x1, x27 + eor x23, x23, x12 + rax1 v25.2d, v30.2d, v28.2d + eor x23, x23, x13 + eor x11, x11, x0 + add v31.2d, v26.2d, v26.2d + eor x29, x29, x23, ror #63 + eor x23, x23, x26, ror #63 + sri v31.2d, v26.2d, #0x3f + eor x26, x13, x0 + eor x13, x28, x23 + eor v28.16b, v31.16b, v28.16b + eor x28, x24, x30 + eor x24, x16, x23 + rax1 v26.2d, v26.2d, v29.2d + eor x16, x21, x30 + eor x21, x25, x30 + add v31.2d, v27.2d, v27.2d + eor x30, x19, x23 + sri v31.2d, v27.2d, #0x3f + eor x19, x20, x23 + eor x20, x17, x23 + eor v29.16b, v31.16b, v29.16b + eor x17, x12, x0 + eor x0, x2, x27 + rax1 v27.2d, v27.2d, v30.2d + eor x2, x6, x29 + eor x6, x8, x29 + eor v30.16b, v0.16b, v26.16b + bic x8, x28, x13, ror #47 + eor x12, x3, x27 + eor v31.16b, v2.16b, v29.16b + bic x3, x13, x17, ror #19 + eor x5, x5, x27 + shl v0.2d, v31.2d, #0x3e + ldr x27, [sp, #0xd0] + bic x25, x17, x2, ror #5 + sri v0.2d, v31.2d, #0x2 + eor x9, x9, x29 + eor x23, x25, x5, ror #52 + xar v2.2d, v12.2d, v29.2d, #0x15 + eor x3, x3, x2, ror #24 + eor x8, x8, x17, ror #2 + eor v31.16b, v13.16b, v28.16b + eor x17, x10, x29 + bic x25, x12, x22, ror #47 + shl v12.2d, v31.2d, #0x19 + eor x29, x7, x29 + bic x10, x4, x27, ror #2 + sri v12.2d, v31.2d, #0x27 + bic x7, x5, x28, ror #10 + xar v13.2d, v19.2d, v27.2d, #0x38 + eor x10, x10, x20, ror #50 + eor x13, x7, x13, ror #57 + eor v31.16b, v23.16b, v28.16b + bic x7, x2, x5, ror #47 + eor x2, x25, x24, ror #39 + shl v19.2d, v31.2d, #0x38 + bic x25, x20, x11, ror #57 + bic x5, x17, x4, ror #25 + sri v19.2d, v31.2d, #0x8 + eor x25, x25, x17, ror #53 + bic x17, x11, x17, ror #60 + xar v23.2d, v15.2d, v26.2d, #0x17 + eor x28, x7, x28, ror #57 + bic x7, x9, x12, ror #42 + eor v31.16b, v1.16b, v25.16b + eor x7, x7, x22, ror #25 + bic x22, x22, x24, ror #56 + shl v15.2d, v31.2d, #0x1 + bic x24, x24, x15, ror #31 + eor x22, x22, x15, ror #23 + sri v15.2d, v31.2d, #0x3f + bic x20, x27, x20, ror #48 + bic x15, x15, x9, ror #16 + xar v1.2d, v8.2d, v28.2d, #0x9 + eor x12, x15, x12, ror #58 + eor x15, x5, x27, ror #27 + eor v31.16b, v16.16b, v25.16b + eor x5, x20, x11, ror #41 + shl v8.2d, v31.2d, #0x2d + ldr x11, [sp, #0x8] + eor x20, x17, x4, ror #21 + sri v8.2d, v31.2d, #0x13 + eor x17, x24, x9, ror #47 + mov x24, #0x1 // =1 + xar v16.2d, v7.2d, v29.2d, #0x3a + bic x9, x0, x16, ror #9 + str x24, [sp, #0x18] + eor v31.16b, v10.16b, v26.16b + bic x24, x29, x1, ror #44 + bic x27, x1, x21, ror #50 + shl v7.2d, v31.2d, #0x3 + bic x4, x26, x29, ror #63 + eor x1, x1, x4, ror #21 + sri v7.2d, v31.2d, #0x3d + ldr x11, [x11] + bic x4, x21, x30, ror #57 + xar v10.2d, v3.2d, v28.2d, #0x24 + eor x21, x24, x21, ror #30 + eor x24, x9, x19, ror #44 + eor v31.16b, v18.16b, v28.16b + bic x9, x14, x6, ror #5 + eor x9, x9, x0, ror #43 + shl v3.2d, v31.2d, #0x15 + bic x0, x6, x0, ror #38 + eor x1, x1, x11 + sri v3.2d, v31.2d, #0x2b + eor x11, x4, x26, ror #35 + eor x4, x0, x16, ror #47 + xar v18.2d, v17.2d, v29.2d, #0x31 + bic x0, x16, x19, ror #35 + eor v31.16b, v11.16b, v25.16b + eor x16, x27, x30, ror #43 + bic x27, x30, x26, ror #42 + shl v17.2d, v31.2d, #0xa + bic x26, x19, x14, ror #41 + eor x19, x0, x14, ror #12 + sri v17.2d, v31.2d, #0x36 + eor x14, x26, x6, ror #46 + eor x6, x27, x29, ror #41 + xar v11.2d, v9.2d, v27.2d, #0x2c + eor x0, x15, x11, ror #52 + eor x0, x0, x13, ror #48 + eor v31.16b, v22.16b, v29.16b + eor x26, x8, x9, ror #57 + eor x27, x0, x14, ror #10 + shl v9.2d, v31.2d, #0x3d + eor x29, x16, x28, ror #63 + eor x26, x26, x6, ror #51 + sri v9.2d, v31.2d, #0x3 + eor x30, x23, x22, ror #50 + eor x0, x26, x10, ror #31 + xar v22.2d, v14.2d, v27.2d, #0x19 + eor x29, x29, x19, ror #37 + eor x27, x27, x12, ror #5 + eor v31.16b, v20.16b, v26.16b + eor x30, x30, x24, ror #34 + eor x0, x0, x7, ror #27 + shl v14.2d, v31.2d, #0x12 + eor x26, x30, x21, ror #26 + sri v14.2d, v31.2d, #0x2e + eor x26, x26, x25, ror #15 + ror x30, x27, #0x3e + xar v20.2d, v4.2d, v27.2d, #0x25 + eor x30, x30, x26, ror #57 + ror x26, x26, #0x3a + eor v31.16b, v24.16b, v27.16b + eor x16, x30, x16 + eor x28, x30, x28, ror #63 + shl v4.2d, v31.2d, #0xe + str x28, [sp, #0xd0] + eor x29, x29, x17, ror #36 + sri v4.2d, v31.2d, #0x32 + eor x28, x1, x2, ror #61 + eor x19, x30, x19, ror #37 + xar v24.2d, v21.2d, v25.2d, #0x3e + eor x29, x29, x20, ror #2 + eor x28, x28, x4, ror #54 + eor v31.16b, v5.16b, v26.16b + eor x26, x26, x0, ror #55 + eor x28, x28, x3, ror #39 + shl v21.2d, v31.2d, #0x24 + eor x28, x28, x5, ror #25 + ror x0, x0, #0x38 + sri v21.2d, v31.2d, #0x1c + eor x0, x0, x29, ror #63 + eor x27, x28, x27, ror #61 + xar v27.2d, v6.2d, v25.2d, #0x14 + eor x13, x0, x13, ror #46 + eor x28, x29, x28, ror #63 + bic v31.16b, v7.16b, v11.16b + eor x29, x30, x20, ror #2 + eor v5.16b, v31.16b, v10.16b + eor x20, x26, x3, ror #39 + eor x11, x0, x11, ror #50 + bcax v6.16b, v11.16b, v8.16b, v7.16b + eor x25, x28, x25, ror #9 + eor x3, x28, x21, ror #20 + bic v31.16b, v9.16b, v8.16b + eor x21, x26, x1 + eor x9, x27, x9, ror #49 + eor v7.16b, v31.16b, v7.16b + eor x24, x28, x24, ror #28 + eor x1, x30, x17, ror #36 + bcax v8.16b, v8.16b, v10.16b, v9.16b + eor x14, x0, x14, ror #8 + eor x22, x28, x22, ror #44 + bic v31.16b, v11.16b, v10.16b + eor x8, x27, x8, ror #56 + eor x17, x27, x7, ror #19 + eor v9.16b, v31.16b, v9.16b + eor x15, x0, x15, ror #62 + bic x7, x20, x22, ror #47 + bcax v10.16b, v15.16b, v12.16b, v16.16b + eor x4, x26, x4, ror #54 + eor x0, x0, x12, ror #3 + bic v31.16b, v13.16b, v12.16b + eor x28, x28, x23, ror #58 + eor x23, x26, x2, ror #61 + eor v11.16b, v31.16b, v16.16b + eor x26, x26, x5, ror #25 + bcax v12.16b, v12.16b, v14.16b, v13.16b + eor x2, x7, x16, ror #39 + bic x7, x9, x20, ror #42 + bic v31.16b, v15.16b, v14.16b + bic x30, x15, x9, ror #16 + eor x7, x7, x22, ror #25 + eor v13.16b, v31.16b, v13.16b + eor x12, x30, x20, ror #58 + bic x20, x22, x16, ror #56 + bic v31.16b, v16.16b, v15.16b + eor x30, x27, x6, ror #43 + eor x22, x20, x15, ror #23 + eor v14.16b, v31.16b, v14.16b + bic x6, x19, x13, ror #42 + eor x6, x6, x17, ror #41 + bcax v15.16b, v20.16b, v17.16b, v21.16b + bic x5, x13, x17, ror #63 + eor x5, x21, x5, ror #21 + bic v31.16b, v18.16b, v17.16b + bic x17, x17, x21, ror #44 + eor x27, x27, x10, ror #23 + eor v16.16b, v31.16b, v21.16b + bic x21, x21, x25, ror #50 + bic x20, x27, x4, ror #25 + bcax v17.16b, v17.16b, v19.16b, v18.16b + bic x10, x16, x15, ror #31 + eor x16, x21, x19, ror #43 + bic v31.16b, v20.16b, v19.16b + eor x21, x17, x25, ror #30 + bic x19, x25, x19, ror #57 + eor v18.16b, v31.16b, v18.16b + ldr x25, [sp, #0x18] + bcax v19.16b, v19.16b, v21.16b, v20.16b + eor x17, x10, x9, ror #47 + ldr x9, [sp, #0x8] + bic v31.16b, v22.16b, v1.16b + eor x15, x20, x28, ror #27 + bic x20, x4, x28, ror #2 + eor v20.16b, v31.16b, v0.16b + eor x10, x20, x1, ror #50 + bic x20, x11, x27, ror #60 + bcax v21.16b, v1.16b, v23.16b, v22.16b + eor x20, x20, x4, ror #21 + bic x4, x28, x1, ror #48 + bic v31.16b, v24.16b, v23.16b + bic x1, x1, x11, ror #57 + ldr x28, [x9, x25, lsl #3] + eor v22.16b, v31.16b, v22.16b + ldr x9, [sp, #0xd0] + add x25, x25, #0x1 + bcax v23.16b, v23.16b, v0.16b, v24.16b + str x25, [sp, #0x18] + cmp x25, #0x17 + bic v31.16b, v1.16b, v0.16b + eor x25, x1, x27, ror #53 + bic x27, x30, x26, ror #47 + eor v24.16b, v31.16b, v24.16b + eor x1, x5, x28 + eor x5, x4, x11, ror #41 + bcax v0.16b, v30.16b, v2.16b, v27.16b + eor x11, x19, x13, ror #35 + bic v31.16b, v3.16b, v2.16b + bic x13, x26, x24, ror #10 + eor x28, x27, x24, ror #57 + eor v1.16b, v31.16b, v27.16b + bic x27, x24, x9, ror #47 + bic x19, x23, x3, ror #9 + bcax v2.16b, v2.16b, v4.16b, v3.16b + bic x4, x29, x14, ror #41 + eor x24, x19, x29, ror #44 + bic v31.16b, v30.16b, v4.16b + bic x29, x3, x29, ror #35 + eor x13, x13, x9, ror #57 + eor v3.16b, v31.16b, v3.16b + eor x19, x29, x14, ror #12 + bic x29, x9, x0, ror #19 + bcax v4.16b, v4.16b, v27.16b, v30.16b + bic x14, x14, x8, ror #5 + eor x9, x14, x23, ror #43 + eor x14, x4, x8, ror #46 + bic x23, x8, x23, ror #38 + eor x8, x27, x0, ror #2 + eor x4, x23, x3, ror #47 + bic x3, x0, x30, ror #5 + eor x23, x3, x26, ror #52 + eor x3, x29, x30, ror #24 + ldr x30, [sp, #0x10] + ld1r { v28.2d }, [x30], #8 + str x30, [sp, #0x10] + eor v0.16b, v0.16b, v28.16b + +Lmld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_loop: + eor x0, x15, x11, ror #52 + eor x0, x0, x13, ror #48 + eor3 v30.16b, v0.16b, v5.16b, v10.16b + eor v30.16b, v30.16b, v15.16b + eor x26, x8, x9, ror #57 + eor x27, x0, x14, ror #10 + eor v30.16b, v30.16b, v20.16b + eor x29, x16, x28, ror #63 + eor x26, x26, x6, ror #51 + eor3 v29.16b, v1.16b, v6.16b, v11.16b + eor x30, x23, x22, ror #50 + eor x0, x26, x10, ror #31 + eor v29.16b, v29.16b, v16.16b + eor x29, x29, x19, ror #37 + eor x27, x27, x12, ror #5 + eor v29.16b, v29.16b, v21.16b + eor x30, x30, x24, ror #34 + eor x0, x0, x7, ror #27 + eor3 v28.16b, v2.16b, v7.16b, v12.16b + eor x26, x30, x21, ror #26 + eor x26, x26, x25, ror #15 + eor v28.16b, v28.16b, v17.16b + ror x30, x27, #0x3e + eor x30, x30, x26, ror #57 + eor v28.16b, v28.16b, v22.16b + ror x26, x26, #0x3a + eor x16, x30, x16 + eor3 v27.16b, v3.16b, v8.16b, v13.16b + eor x28, x30, x28, ror #63 + str x28, [sp, #0xd0] + eor v27.16b, v27.16b, v18.16b + eor x29, x29, x17, ror #36 + eor x28, x1, x2, ror #61 + eor v27.16b, v27.16b, v23.16b + eor x19, x30, x19, ror #37 + eor x29, x29, x20, ror #2 + eor3 v26.16b, v4.16b, v9.16b, v14.16b + eor x28, x28, x4, ror #54 + eor x26, x26, x0, ror #55 + eor v26.16b, v26.16b, v19.16b + eor x28, x28, x3, ror #39 + eor x28, x28, x5, ror #25 + eor v26.16b, v26.16b, v24.16b + ror x0, x0, #0x38 + eor x0, x0, x29, ror #63 + rax1 v25.2d, v30.2d, v28.2d + eor x27, x28, x27, ror #61 + eor x13, x0, x13, ror #46 + add v31.2d, v26.2d, v26.2d + eor x28, x29, x28, ror #63 + eor x29, x30, x20, ror #2 + sri v31.2d, v26.2d, #0x3f + eor x20, x26, x3, ror #39 + eor x11, x0, x11, ror #50 + eor v28.16b, v31.16b, v28.16b + eor x25, x28, x25, ror #9 + eor x3, x28, x21, ror #20 + rax1 v26.2d, v26.2d, v29.2d + eor x21, x26, x1 + add v31.2d, v27.2d, v27.2d + eor x9, x27, x9, ror #49 + eor x24, x28, x24, ror #28 + sri v31.2d, v27.2d, #0x3f + eor x1, x30, x17, ror #36 + eor x14, x0, x14, ror #8 + eor v29.16b, v31.16b, v29.16b + eor x22, x28, x22, ror #44 + eor x8, x27, x8, ror #56 + rax1 v27.2d, v27.2d, v30.2d + eor x17, x27, x7, ror #19 + eor x15, x0, x15, ror #62 + eor v30.16b, v0.16b, v26.16b + bic x7, x20, x22, ror #47 + eor x4, x26, x4, ror #54 + eor v31.16b, v2.16b, v29.16b + eor x0, x0, x12, ror #3 + eor x28, x28, x23, ror #58 + shl v0.2d, v31.2d, #0x3e + eor x23, x26, x2, ror #61 + eor x26, x26, x5, ror #25 + sri v0.2d, v31.2d, #0x2 + eor x2, x7, x16, ror #39 + bic x7, x9, x20, ror #42 + xar v2.2d, v12.2d, v29.2d, #0x15 + bic x30, x15, x9, ror #16 + eor x7, x7, x22, ror #25 + eor v31.16b, v13.16b, v28.16b + eor x12, x30, x20, ror #58 + bic x20, x22, x16, ror #56 + shl v12.2d, v31.2d, #0x19 + eor x30, x27, x6, ror #43 + eor x22, x20, x15, ror #23 + sri v12.2d, v31.2d, #0x27 + bic x6, x19, x13, ror #42 + eor x6, x6, x17, ror #41 + xar v13.2d, v19.2d, v27.2d, #0x38 + bic x5, x13, x17, ror #63 + eor x5, x21, x5, ror #21 + eor v31.16b, v23.16b, v28.16b + bic x17, x17, x21, ror #44 + eor x27, x27, x10, ror #23 + shl v19.2d, v31.2d, #0x38 + bic x21, x21, x25, ror #50 + bic x20, x27, x4, ror #25 + sri v19.2d, v31.2d, #0x8 + bic x10, x16, x15, ror #31 + eor x16, x21, x19, ror #43 + xar v23.2d, v15.2d, v26.2d, #0x17 + eor x21, x17, x25, ror #30 + bic x19, x25, x19, ror #57 + eor v31.16b, v1.16b, v25.16b + ldr x25, [sp, #0x18] + eor x17, x10, x9, ror #47 + shl v15.2d, v31.2d, #0x1 + ldr x9, [sp, #0x8] + sri v15.2d, v31.2d, #0x3f + eor x15, x20, x28, ror #27 + bic x20, x4, x28, ror #2 + xar v1.2d, v8.2d, v28.2d, #0x9 + eor x10, x20, x1, ror #50 + bic x20, x11, x27, ror #60 + eor v31.16b, v16.16b, v25.16b + eor x20, x20, x4, ror #21 + bic x4, x28, x1, ror #48 + shl v8.2d, v31.2d, #0x2d + bic x1, x1, x11, ror #57 + ldr x28, [x9, x25, lsl #3] + sri v8.2d, v31.2d, #0x13 + ldr x9, [sp, #0xd0] + add x25, x25, #0x1 + xar v16.2d, v7.2d, v29.2d, #0x3a + str x25, [sp, #0x18] + cmp x25, #0x17 + eor v31.16b, v10.16b, v26.16b + eor x25, x1, x27, ror #53 + bic x27, x30, x26, ror #47 + shl v7.2d, v31.2d, #0x3 + eor x1, x5, x28 + eor x5, x4, x11, ror #41 + sri v7.2d, v31.2d, #0x3d + eor x11, x19, x13, ror #35 + bic x13, x26, x24, ror #10 + xar v10.2d, v3.2d, v28.2d, #0x24 + eor x28, x27, x24, ror #57 + bic x27, x24, x9, ror #47 + eor v31.16b, v18.16b, v28.16b + bic x19, x23, x3, ror #9 + bic x4, x29, x14, ror #41 + shl v3.2d, v31.2d, #0x15 + eor x24, x19, x29, ror #44 + bic x29, x3, x29, ror #35 + sri v3.2d, v31.2d, #0x2b + eor x13, x13, x9, ror #57 + eor x19, x29, x14, ror #12 + xar v18.2d, v17.2d, v29.2d, #0x31 + bic x29, x9, x0, ror #19 + bic x14, x14, x8, ror #5 + eor v31.16b, v11.16b, v25.16b + eor x9, x14, x23, ror #43 + eor x14, x4, x8, ror #46 + shl v17.2d, v31.2d, #0xa + bic x23, x8, x23, ror #38 + eor x8, x27, x0, ror #2 + sri v17.2d, v31.2d, #0x36 + eor x4, x23, x3, ror #47 + bic x3, x0, x30, ror #5 + xar v11.2d, v9.2d, v27.2d, #0x2c + eor x23, x3, x26, ror #52 + eor x3, x29, x30, ror #24 + eor v31.16b, v22.16b, v29.16b + eor x0, x15, x11, ror #52 + shl v9.2d, v31.2d, #0x3d + eor x0, x0, x13, ror #48 + eor x26, x8, x9, ror #57 + sri v9.2d, v31.2d, #0x3 + eor x27, x0, x14, ror #10 + eor x29, x16, x28, ror #63 + xar v22.2d, v14.2d, v27.2d, #0x19 + eor x26, x26, x6, ror #51 + eor x30, x23, x22, ror #50 + eor v31.16b, v20.16b, v26.16b + eor x0, x26, x10, ror #31 + eor x29, x29, x19, ror #37 + shl v14.2d, v31.2d, #0x12 + eor x27, x27, x12, ror #5 + eor x30, x30, x24, ror #34 + sri v14.2d, v31.2d, #0x2e + eor x0, x0, x7, ror #27 + eor x26, x30, x21, ror #26 + xar v20.2d, v4.2d, v27.2d, #0x25 + eor x26, x26, x25, ror #15 + ror x30, x27, #0x3e + eor v31.16b, v24.16b, v27.16b + eor x30, x30, x26, ror #57 + ror x26, x26, #0x3a + shl v4.2d, v31.2d, #0xe + eor x16, x30, x16 + eor x28, x30, x28, ror #63 + sri v4.2d, v31.2d, #0x32 + str x28, [sp, #0xd0] + eor x29, x29, x17, ror #36 + xar v24.2d, v21.2d, v25.2d, #0x3e + eor x28, x1, x2, ror #61 + eor x19, x30, x19, ror #37 + eor v31.16b, v5.16b, v26.16b + eor x29, x29, x20, ror #2 + eor x28, x28, x4, ror #54 + shl v21.2d, v31.2d, #0x24 + eor x26, x26, x0, ror #55 + eor x28, x28, x3, ror #39 + sri v21.2d, v31.2d, #0x1c + eor x28, x28, x5, ror #25 + ror x0, x0, #0x38 + xar v27.2d, v6.2d, v25.2d, #0x14 + eor x0, x0, x29, ror #63 + eor x27, x28, x27, ror #61 + bic v31.16b, v7.16b, v11.16b + eor x13, x0, x13, ror #46 + eor x28, x29, x28, ror #63 + eor v5.16b, v31.16b, v10.16b + eor x29, x30, x20, ror #2 + eor x20, x26, x3, ror #39 + bcax v6.16b, v11.16b, v8.16b, v7.16b + eor x11, x0, x11, ror #50 + eor x25, x28, x25, ror #9 + bic v31.16b, v9.16b, v8.16b + eor x3, x28, x21, ror #20 + eor v7.16b, v31.16b, v7.16b + eor x21, x26, x1 + eor x9, x27, x9, ror #49 + bcax v8.16b, v8.16b, v10.16b, v9.16b + eor x24, x28, x24, ror #28 + eor x1, x30, x17, ror #36 + bic v31.16b, v11.16b, v10.16b + eor x14, x0, x14, ror #8 + eor x22, x28, x22, ror #44 + eor v9.16b, v31.16b, v9.16b + eor x8, x27, x8, ror #56 + eor x17, x27, x7, ror #19 + bcax v10.16b, v15.16b, v12.16b, v16.16b + eor x15, x0, x15, ror #62 + bic x7, x20, x22, ror #47 + bic v31.16b, v13.16b, v12.16b + eor x4, x26, x4, ror #54 + eor x0, x0, x12, ror #3 + eor v11.16b, v31.16b, v16.16b + eor x28, x28, x23, ror #58 + eor x23, x26, x2, ror #61 + bcax v12.16b, v12.16b, v14.16b, v13.16b + eor x26, x26, x5, ror #25 + eor x2, x7, x16, ror #39 + bic v31.16b, v15.16b, v14.16b + bic x7, x9, x20, ror #42 + bic x30, x15, x9, ror #16 + eor v13.16b, v31.16b, v13.16b + eor x7, x7, x22, ror #25 + eor x12, x30, x20, ror #58 + bic v31.16b, v16.16b, v15.16b + bic x20, x22, x16, ror #56 + eor x30, x27, x6, ror #43 + eor v14.16b, v31.16b, v14.16b + eor x22, x20, x15, ror #23 + bic x6, x19, x13, ror #42 + bcax v15.16b, v20.16b, v17.16b, v21.16b + eor x6, x6, x17, ror #41 + bic x5, x13, x17, ror #63 + bic v31.16b, v18.16b, v17.16b + eor x5, x21, x5, ror #21 + bic x17, x17, x21, ror #44 + eor v16.16b, v31.16b, v21.16b + eor x27, x27, x10, ror #23 + bic x21, x21, x25, ror #50 + bcax v17.16b, v17.16b, v19.16b, v18.16b + bic x20, x27, x4, ror #25 + bic x10, x16, x15, ror #31 + bic v31.16b, v20.16b, v19.16b + eor x16, x21, x19, ror #43 + eor x21, x17, x25, ror #30 + eor v18.16b, v31.16b, v18.16b + bic x19, x25, x19, ror #57 + ldr x25, [sp, #0x18] + bcax v19.16b, v19.16b, v21.16b, v20.16b + eor x17, x10, x9, ror #47 + bic v31.16b, v22.16b, v1.16b + ldr x9, [sp, #0x8] + eor x15, x20, x28, ror #27 + eor v20.16b, v31.16b, v0.16b + bic x20, x4, x28, ror #2 + eor x10, x20, x1, ror #50 + bcax v21.16b, v1.16b, v23.16b, v22.16b + bic x20, x11, x27, ror #60 + eor x20, x20, x4, ror #21 + bic v31.16b, v24.16b, v23.16b + bic x4, x28, x1, ror #48 + bic x1, x1, x11, ror #57 + eor v22.16b, v31.16b, v22.16b + ldr x28, [x9, x25, lsl #3] + ldr x9, [sp, #0xd0] + bcax v23.16b, v23.16b, v0.16b, v24.16b + add x25, x25, #0x1 + str x25, [sp, #0x18] + bic v31.16b, v1.16b, v0.16b + cmp x25, #0x17 + eor x25, x1, x27, ror #53 + eor v24.16b, v31.16b, v24.16b + bic x27, x30, x26, ror #47 + eor x1, x5, x28 + bcax v0.16b, v30.16b, v2.16b, v27.16b + eor x5, x4, x11, ror #41 + eor x11, x19, x13, ror #35 + bic v31.16b, v3.16b, v2.16b + bic x13, x26, x24, ror #10 + eor x28, x27, x24, ror #57 + eor v1.16b, v31.16b, v27.16b + bic x27, x24, x9, ror #47 + bic x19, x23, x3, ror #9 + bcax v2.16b, v2.16b, v4.16b, v3.16b + bic x4, x29, x14, ror #41 + eor x24, x19, x29, ror #44 + bic v31.16b, v30.16b, v4.16b + bic x29, x3, x29, ror #35 + eor x13, x13, x9, ror #57 + eor v3.16b, v31.16b, v3.16b + eor x19, x29, x14, ror #12 + bic x29, x9, x0, ror #19 + bcax v4.16b, v4.16b, v27.16b, v30.16b + bic x14, x14, x8, ror #5 + eor x9, x14, x23, ror #43 + eor x14, x4, x8, ror #46 + bic x23, x8, x23, ror #38 + eor x8, x27, x0, ror #2 + eor x4, x23, x3, ror #47 + bic x3, x0, x30, ror #5 + eor x23, x3, x26, ror #52 + eor x3, x29, x30, ror #24 + ldr x30, [sp, #0x10] + ld1r { v28.2d }, [x30], #8 + str x30, [sp, #0x10] + eor v0.16b, v0.16b, v28.16b + +Lmld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_loop_end: + b.le Lmld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_loop + ror x2, x2, #0x3d + ror x3, x3, #0x27 + ror x4, x4, #0x36 + ror x5, x5, #0x19 + ror x6, x6, #0x2b + ror x7, x7, #0x13 + ror x8, x8, #0x38 + ror x9, x9, #0x31 + ror x10, x10, #0x17 + ror x11, x11, #0x32 + ror x12, x12, #0x3 + ror x13, x13, #0x2e + ror x14, x14, #0x8 + ror x15, x15, #0x3e + ror x17, x17, #0x24 + ror x28, x28, #0x3f + ror x19, x19, #0x25 + ror x20, x20, #0x2 + ror x21, x21, #0x14 + ror x22, x22, #0x2c + ror x23, x23, #0x3a + ror x24, x24, #0x1c + ror x25, x25, #0x9 + ldr x30, [sp, #0x20] + cmp x30, #0x1 + b.eq Lmld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_done + mov x30, #0x1 // =1 + str x30, [sp, #0x20] + ldr x0, [sp] + add x0, x0, #0x190 + stp x1, x6, [x0] + stp x11, x16, [x0, #0x10] + stp x21, x2, [x0, #0x20] + stp x7, x12, [x0, #0x30] + stp x17, x22, [x0, #0x40] + stp x3, x8, [x0, #0x50] + stp x13, x28, [x0, #0x60] + stp x23, x4, [x0, #0x70] + stp x9, x14, [x0, #0x80] + stp x19, x24, [x0, #0x90] + stp x5, x10, [x0, #0xa0] + stp x15, x20, [x0, #0xb0] + str x25, [x0, #0xc0] + sub x0, x0, #0x190 + add x0, x0, #0x258 + ldp x1, x6, [x0] + ldp x11, x16, [x0, #0x10] + ldp x21, x2, [x0, #0x20] + ldp x7, x12, [x0, #0x30] + ldp x17, x22, [x0, #0x40] + ldp x3, x8, [x0, #0x50] + ldp x13, x28, [x0, #0x60] + ldp x23, x4, [x0, #0x70] + ldp x9, x14, [x0, #0x80] + ldp x19, x24, [x0, #0x90] + ldp x5, x10, [x0, #0xa0] + ldp x15, x20, [x0, #0xb0] + ldr x25, [x0, #0xc0] + sub x0, x0, #0x258 + b Lmld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_initial + +Lmld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_done: + ldr x0, [sp] + add x0, x0, #0x258 + stp x1, x6, [x0] + stp x11, x16, [x0, #0x10] + stp x21, x2, [x0, #0x20] + stp x7, x12, [x0, #0x30] + stp x17, x22, [x0, #0x40] + stp x3, x8, [x0, #0x50] + stp x13, x28, [x0, #0x60] + stp x23, x4, [x0, #0x70] + stp x9, x14, [x0, #0x80] + stp x19, x24, [x0, #0x90] + stp x5, x10, [x0, #0xa0] + stp x15, x20, [x0, #0xb0] + str x25, [x0, #0xc0] + sub x0, x0, #0x258 + add x4, x0, #0xc8 + trn1 v25.2d, v0.2d, v1.2d + trn1 v26.2d, v2.2d, v3.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v0.2d, v1.2d + trn2 v28.2d, v2.2d, v3.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v4.2d, v5.2d + trn1 v26.2d, v6.2d, v7.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v4.2d, v5.2d + trn2 v28.2d, v6.2d, v7.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v8.2d, v9.2d + trn1 v26.2d, v10.2d, v11.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v8.2d, v9.2d + trn2 v28.2d, v10.2d, v11.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v12.2d, v13.2d + trn1 v26.2d, v14.2d, v15.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v12.2d, v13.2d + trn2 v28.2d, v14.2d, v15.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v16.2d, v17.2d + trn1 v26.2d, v18.2d, v19.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v16.2d, v17.2d + trn2 v28.2d, v18.2d, v19.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v20.2d, v21.2d + trn1 v26.2d, v22.2d, v23.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v20.2d, v21.2d + trn2 v28.2d, v22.2d, v23.2d + st1 { v27.2d, v28.2d }, [x4], #32 + str d24, [x0] + trn2 v25.2d, v24.2d, v24.2d + str d25, [x4] + ldp d8, d9, [sp, #0x90] + .cfi_restore d8 + .cfi_restore d9 + ldp d10, d11, [sp, #0xa0] + .cfi_restore d10 + .cfi_restore d11 + ldp d12, d13, [sp, #0xb0] + .cfi_restore d12 + .cfi_restore d13 + ldp d14, d15, [sp, #0xc0] + .cfi_restore d14 + .cfi_restore d15 + ldp x19, x20, [sp, #0x30] + .cfi_restore x19 + .cfi_restore x20 + ldp x21, x22, [sp, #0x40] + .cfi_restore x21 + .cfi_restore x22 + ldp x23, x24, [sp, #0x50] + .cfi_restore x23 + .cfi_restore x24 + ldp x25, x26, [sp, #0x60] + .cfi_restore x25 + .cfi_restore x26 + ldp x27, x28, [sp, #0x70] + .cfi_restore x27 + .cfi_restore x28 + ldp x29, x30, [sp, #0x80] + .cfi_restore x29 + .cfi_restore x30 + add sp, sp, #0xe0 + .cfi_adjust_cfa_offset -0xe0 + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm) + +#endif /* __ARM_FEATURE_SHA3 */ + +#endif /* MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/fips202/native/aarch64/src/keccakf1600_round_constants.c b/cbits/mldsa/src/fips202/native/aarch64/src/keccakf1600_round_constants.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/aarch64/src/keccakf1600_round_constants.c @@ -0,0 +1,47 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mldsa-native repository. + * Do not modify it directly. + */ + +#include "../../../../common.h" + +#if (defined(MLD_FIPS202_AARCH64_NEED_X1_SCALAR) || \ + defined(MLD_FIPS202_AARCH64_NEED_X1_V84A) || \ + defined(MLD_FIPS202_AARCH64_NEED_X2_V84A) || \ + defined(MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID) || \ + defined(MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID)) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +#include "fips202_native_aarch64.h" + +MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint64_t + mld_keccakf1600_round_constants[24] = { + 0x0000000000000001, 0x0000000000008082, 0x800000000000808a, + 0x8000000080008000, 0x000000000000808b, 0x0000000080000001, + 0x8000000080008081, 0x8000000000008009, 0x000000000000008a, + 0x0000000000000088, 0x0000000080008009, 0x000000008000000a, + 0x000000008000808b, 0x800000000000008b, 0x8000000000008089, + 0x8000000000008003, 0x8000000000008002, 0x8000000000000080, + 0x000000000000800a, 0x800000008000000a, 0x8000000080008081, + 0x8000000000008080, 0x0000000080000001, 0x8000000080008008, +}; + +#else /* (MLD_FIPS202_AARCH64_NEED_X1_SCALAR || \ + MLD_FIPS202_AARCH64_NEED_X1_V84A || MLD_FIPS202_AARCH64_NEED_X2_V84A \ + || MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID || \ + MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID) && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +MLD_EMPTY_CU(fips202_aarch64_round_constants) + +#endif /* !((MLD_FIPS202_AARCH64_NEED_X1_SCALAR || \ + MLD_FIPS202_AARCH64_NEED_X1_V84A || MLD_FIPS202_AARCH64_NEED_X2_V84A \ + || MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID || \ + MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID) && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mldsa/src/fips202/native/aarch64/x1_scalar.h b/cbits/mldsa/src/fips202/native/aarch64/x1_scalar.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/aarch64/x1_scalar.h @@ -0,0 +1,27 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_FIPS202_NATIVE_AARCH64_X1_SCALAR_H +#define MLD_FIPS202_NATIVE_AARCH64_X1_SCALAR_H + +/* Part of backend API */ +#define MLD_USE_NATIVE_FIPS202_X1 +/* Guard for assembly file */ +#define MLD_FIPS202_AARCH64_NEED_X1_SCALAR + +#if !defined(__ASSEMBLER__) +#include "../api.h" +#include "src/fips202_native_aarch64.h" +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_keccak_f1600_x1_native(uint64_t *state) +{ + mld_keccak_f1600_x1_scalar_aarch64_asm(state, + mld_keccakf1600_round_constants); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLD_FIPS202_NATIVE_AARCH64_X1_SCALAR_H */ diff --git a/cbits/mldsa/src/fips202/native/aarch64/x1_v84a.h b/cbits/mldsa/src/fips202/native/aarch64/x1_v84a.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/aarch64/x1_v84a.h @@ -0,0 +1,36 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_FIPS202_NATIVE_AARCH64_X1_V84A_H +#define MLD_FIPS202_NATIVE_AARCH64_X1_V84A_H + +#if !defined(__ARM_FEATURE_SHA3) +#error This backend can only be used if SHA3 extensions are available. +#endif + +/* Part of backend API */ +#define MLD_USE_NATIVE_FIPS202_X1 +/* Guard for assembly file */ +#define MLD_FIPS202_AARCH64_NEED_X1_V84A + +#if !defined(__ASSEMBLER__) +#include "../api.h" +#include "src/fips202_native_aarch64.h" +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_keccak_f1600_x1_native(uint64_t *state) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON) || + !mld_sys_check_capability(MLD_SYS_CAP_AARCH64_SHA3)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + + mld_keccak_f1600_x1_v84a_aarch64_asm(state, mld_keccakf1600_round_constants); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLD_FIPS202_NATIVE_AARCH64_X1_V84A_H */ diff --git a/cbits/mldsa/src/fips202/native/aarch64/x2_v84a.h b/cbits/mldsa/src/fips202/native/aarch64/x2_v84a.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/aarch64/x2_v84a.h @@ -0,0 +1,40 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_FIPS202_NATIVE_AARCH64_X2_V84A_H +#define MLD_FIPS202_NATIVE_AARCH64_X2_V84A_H + +#if !defined(__ARM_FEATURE_SHA3) +#error This backend can only be used if SHA3 extensions are available. +#endif + +/* Part of backend API */ +#define MLD_USE_NATIVE_FIPS202_X4 +/* Guard for assembly file */ +#define MLD_FIPS202_AARCH64_NEED_X2_V84A + +#if !defined(__ASSEMBLER__) +#include "../api.h" +#include "src/fips202_native_aarch64.h" + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_keccak_f1600_x4_native(uint64_t *state) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON) || + !mld_sys_check_capability(MLD_SYS_CAP_AARCH64_SHA3)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + + mld_keccak_f1600_x2_v84a_aarch64_asm(state + 0 * 25, + mld_keccakf1600_round_constants); + mld_keccak_f1600_x2_v84a_aarch64_asm(state + 2 * 25, + mld_keccakf1600_round_constants); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLD_FIPS202_NATIVE_AARCH64_X2_V84A_H */ diff --git a/cbits/mldsa/src/fips202/native/aarch64/x4_v8a_scalar.h b/cbits/mldsa/src/fips202/native/aarch64/x4_v8a_scalar.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/aarch64/x4_v8a_scalar.h @@ -0,0 +1,32 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H +#define MLD_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H + +/* Part of backend API */ +#define MLD_USE_NATIVE_FIPS202_X4 +/* Guard for assembly file */ +#define MLD_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID + +#if !defined(__ASSEMBLER__) +#include "../api.h" +#include "src/fips202_native_aarch64.h" +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_keccak_f1600_x4_native(uint64_t *state) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + + mld_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm( + state, mld_keccakf1600_round_constants); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLD_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H */ diff --git a/cbits/mldsa/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h b/cbits/mldsa/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h @@ -0,0 +1,37 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H +#define MLD_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H + +#if !defined(__ARM_FEATURE_SHA3) +#error This backend can only be used if SHA3 extensions are available. +#endif + +/* Part of backend API */ +#define MLD_USE_NATIVE_FIPS202_X4 +/* Guard for assembly file */ +#define MLD_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID + +#if !defined(__ASSEMBLER__) +#include "../api.h" +#include "src/fips202_native_aarch64.h" +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_keccak_f1600_x4_native(uint64_t *state) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON) || + !mld_sys_check_capability(MLD_SYS_CAP_AARCH64_SHA3)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + + mld_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm( + state, mld_keccakf1600_round_constants); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLD_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H */ diff --git a/cbits/mldsa/src/fips202/native/api.h b/cbits/mldsa/src/fips202/native/api.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/api.h @@ -0,0 +1,129 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_FIPS202_NATIVE_API_H +#define MLD_FIPS202_NATIVE_API_H +/* + * FIPS-202 native interface + * + * This header is primarily for documentation purposes. + * It should not be included by backend implementations. + */ + +#include "../../cbmc.h" +#include "../../common.h" + +/* Backends must return MLD_NATIVE_FUNC_SUCCESS upon success. */ +#define MLD_NATIVE_FUNC_SUCCESS (0) +/* Backends may return MLD_NATIVE_FUNC_FALLBACK to signal to the frontend that + * the target/parameters are unsupported; typically, this would be because of + * dependencies on CPU features not detected on the host CPU. In this case, + * the frontend falls back to the default C implementation. + * + * IMPORTANT: Backend implementations must ensure that the decision of whether + * to fallback (return MLD_NATIVE_FUNC_FALLBACK) or not must never depend on + * the input data itself. Fallback decisions may only depend on system + * capabilities (e.g., CPU features) and, where present, length information. + * This requirement applies to all backend functions to maintain constant-time + * properties. + */ +#define MLD_NATIVE_FUNC_FALLBACK (-1) + +/* + * This is the C<->native interface allowing for the drop-in + * of custom Keccak-F1600 implementations. + * + * A _backend_ is a specific implementation of parts of this interface. + * + * You can replace 1-fold or 4-fold batched Keccak-F1600. + * To enable, set MLD_USE_NATIVE_FIPS202_X1 or MLD_USE_NATIVE_FIPS202_X4 + * in your backend, and define the inline wrappers mld_keccak_f1600_x1_native() + * and/or mld_keccak_f1600_x4_native(), respectively, to forward to your + * implementation. + */ + +#if defined(MLD_USE_NATIVE_FIPS202_X1) +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_keccak_f1600_x1_native(uint64_t *state) +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 1)) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 1)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged_u64(state, 25 * 1)) +); +#endif /* MLD_USE_NATIVE_FIPS202_X1 */ +#if defined(MLD_USE_NATIVE_FIPS202_X4) +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_keccak_f1600_x4_native(uint64_t *state) +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4)) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged_u64(state, 25 * 4)) +); +#endif /* MLD_USE_NATIVE_FIPS202_X4 */ + +/* + * Native x4 XOR bytes and extract bytes interface. + * + * These functions allow backends to provide optimized implementations for + * XORing input data into the state and extracting output data from the state. + * This is particularly useful for backends that use a different internal state + * representation (e.g., bit-interleaved), as conversion can happen during + * XOR/extract rather than before/after each permutation. + * + * NOTE: We assume that the custom representation of the zero state is the + * all-zero state. + * + * MLD_USE_NATIVE_FIPS202_X4_XOR_BYTES: Backend provides native XOR bytes + * MLD_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES: Backend provides native extract + * bytes + */ + +#if defined(MLD_USE_NATIVE_FIPS202_X4_XOR_BYTES) +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_keccakf1600_xor_bytes_x4_native( + uint64_t *state, const unsigned char *data0, const unsigned char *data1, + const unsigned char *data2, const unsigned char *data3, unsigned offset, + unsigned length) +__contract__( + requires(0 <= offset && offset <= 25 * sizeof(uint64_t) && + 0 <= length && length <= 25 * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4)) + requires(memory_no_alias(data0, length)) + requires((data0 == data1 && + data0 == data2 && + data0 == data3) || + (memory_no_alias(data1, length) && + memory_no_alias(data2, length) && + memory_no_alias(data3, length))) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged_u64(state, 25 * 4))); +#endif /* MLD_USE_NATIVE_FIPS202_X4_XOR_BYTES */ + +#if defined(MLD_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES) +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_keccakf1600_extract_bytes_x4_native( + uint64_t *state, unsigned char *data0, unsigned char *data1, + unsigned char *data2, unsigned char *data3, unsigned offset, + unsigned length) +__contract__( + requires(0 <= offset && offset <= 25 * sizeof(uint64_t) && + 0 <= length && length <= 25 * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4)) + requires(memory_no_alias(data0, length)) + requires(memory_no_alias(data1, length)) + requires(memory_no_alias(data2, length)) + requires(memory_no_alias(data3, length)) + assigns(memory_slice(data0, length)) + assigns(memory_slice(data1, length)) + assigns(memory_slice(data2, length)) + assigns(memory_slice(data3, length)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS)); +#endif /* MLD_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES */ + +#endif /* !MLD_FIPS202_NATIVE_API_H */ diff --git a/cbits/mldsa/src/fips202/native/auto.h b/cbits/mldsa/src/fips202/native/auto.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/auto.h @@ -0,0 +1,35 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_FIPS202_NATIVE_AUTO_H +#define MLD_FIPS202_NATIVE_AUTO_H + +/* + * Default FIPS202 backend + */ +#include "../../sys.h" + +#if defined(MLD_SYS_AARCH64) +#include "aarch64/auto.h" +#endif + +/* The x86_64 backend requires toolchain support for the SysV ABI */ +#if defined(MLD_SYS_X86_64_AVX2) && defined(MLD_SYSV_ABI_SUPPORTED) && \ + (!defined(MLD_CONFIG_NO_KEYPAIR_API) || \ + !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_REDUCE_RAM)) && \ + !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) +#include "x86_64/keccak_f1600_x4_avx2.h" +#endif /* MLD_SYS_X86_64_AVX2 && MLD_SYSV_ABI_SUPPORTED && \ + (!MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API || \ + !MLD_CONFIG_REDUCE_RAM) && !MLD_CONFIG_SERIAL_FIPS202_ONLY */ + +/* We do not yet include the FIPS202 backend for Armv8.1-M+MVE by default + * as it is still experimental and undergoing review. */ +/* #if defined(MLD_SYS_ARMV81M_MVE) */ +/* #include "armv81m/mve.h" */ +/* #endif */ + +#endif /* !MLD_FIPS202_NATIVE_AUTO_H */ diff --git a/cbits/mldsa/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h b/cbits/mldsa/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h @@ -0,0 +1,34 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H +#define MLD_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H + +#include "../../../common.h" + +#define MLD_FIPS202_X86_64_NEED_X4_AVX2 + +/* Part of backend API */ +#define MLD_USE_NATIVE_FIPS202_X4 + +#if !defined(__ASSEMBLER__) +#include "../api.h" +#include "src/fips202_native_x86_64.h" +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_keccak_f1600_x4_native(uint64_t *state) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + + mld_keccak_f1600_x4_avx2_asm(state, mld_keccakf1600_round_constants, + mld_keccak_rho8, mld_keccak_rho56); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLD_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H */ diff --git a/cbits/mldsa/src/fips202/native/x86_64/src/fips202_native_x86_64.h b/cbits/mldsa/src/fips202/native/x86_64/src/fips202_native_x86_64.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/x86_64/src/fips202_native_x86_64.h @@ -0,0 +1,45 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H +#define MLD_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H + +#include "../../../../cbmc.h" +#include "../../../../common.h" + +/* TODO: Reconsider whether this check is needed -- x86_64 is always + * little-endian, so the backend selection already implies this. */ +#ifndef MLD_SYS_LITTLE_ENDIAN +#error Expecting a little-endian platform +#endif + +#define mld_keccakf1600_round_constants \ + MLD_NAMESPACE(keccakf1600_round_constants) +MLD_INTERNAL_DATA_DECLARATION const uint64_t + mld_keccakf1600_round_constants[24]; + +#define mld_keccak_rho8 MLD_NAMESPACE(keccak_rho8) +MLD_INTERNAL_DATA_DECLARATION const uint64_t mld_keccak_rho8[4]; + +#define mld_keccak_rho56 MLD_NAMESPACE(keccak_rho56) +MLD_INTERNAL_DATA_DECLARATION const uint64_t mld_keccak_rho56[4]; + +#define mld_keccak_f1600_x4_avx2_asm MLD_NAMESPACE(keccak_f1600_x4_avx2_asm) +MLD_SYSV_ABI +void mld_keccak_f1600_x4_avx2_asm(uint64_t states[100], const uint64_t rc[24], + const uint64_t rho8[4], + const uint64_t rho56[4]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/keccak_f1600_x4_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(states, sizeof(uint64_t) * 25 * 4)) + requires(rc == mld_keccakf1600_round_constants) + requires(rho8 == mld_keccak_rho8) + requires(rho56 == mld_keccak_rho56) + assigns(memory_slice(states, sizeof(uint64_t) * 25 * 4)) +); + +#endif /* !MLD_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H */ diff --git a/cbits/mldsa/src/fips202/native/x86_64/src/keccak_f1600_x4_avx2_asm.S b/cbits/mldsa/src/fips202/native/x86_64/src/keccak_f1600_x4_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/x86_64/src/keccak_f1600_x4_avx2_asm.S @@ -0,0 +1,488 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +/*yaml + Name: keccak_f1600_x4_avx2_asm + Description: x86_64 AVX2 Keccak-f[1600] permutation for four sequential states + Signature: void mld_keccak_f1600_x4_avx2_asm(uint64_t states[100], const uint64_t rc[24], const uint64_t rho8[4], const uint64_t rho56[4]) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 800 + permissions: read/write + c_parameter: uint64_t states[100] + description: Four sequential Keccak states (4 x 25 x uint64_t) + rsi: + type: buffer + size_bytes: 192 + permissions: read-only + c_parameter: const uint64_t rc[24] + description: Round constants (24 x uint64_t) + rdx: + type: buffer + size_bytes: 32 + permissions: read-only + c_parameter: const uint64_t rho8[4] + description: Rotation constant rho8 (4 x uint64_t) + rcx: + type: buffer + size_bytes: 32 + permissions: read-only + c_parameter: const uint64_t rho56[4] + description: Rotation constant rho56 (4 x uint64_t) +*/ + +#include "../../../../common.h" + +#if defined(MLD_FIPS202_X86_64_NEED_X4_AVX2) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/fips202/x86_64/src/keccak_f1600_x4_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(keccak_f1600_x4_avx2_asm) +MLD_ASM_FN_SYMBOL(keccak_f1600_x4_avx2_asm) + + .cfi_startproc + movq %rsp, %r11 + .cfi_def_cfa_register %r11 + andq $-0x20, %rsp + subq $0x300, %rsp # imm = 0x300 + vmovdqu (%rdi), %ymm0 + vmovdqu 0xc8(%rdi), %ymm3 + vmovdqu 0x190(%rdi), %ymm1 + vmovdqu 0x258(%rdi), %ymm4 + vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3] + vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2] + vperm2i128 $0x20, %ymm3, %ymm2, %ymm7 # ymm7 = ymm2[0,1],ymm3[0,1] + vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3] + vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3] + vmovdqu 0x278(%rdi), %ymm4 + vmovdqu %ymm3, 0x40(%rsp) + vperm2i128 $0x31, %ymm1, %ymm0, %ymm3 # ymm3 = ymm0[2,3],ymm1[2,3] + vmovdqu %ymm7, (%rsp) + vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1] + vmovdqu 0x20(%rdi), %ymm0 + vmovdqu 0x1b0(%rdi), %ymm1 + vmovdqu %ymm3, 0x60(%rsp) + vmovdqu 0xe8(%rdi), %ymm3 + vmovdqu %ymm7, 0x20(%rsp) + vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3] + vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2] + vperm2i128 $0x20, %ymm3, %ymm2, %ymm7 # ymm7 = ymm2[0,1],ymm3[0,1] + vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3] + vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3] + vmovdqu 0x298(%rdi), %ymm4 + vperm2i128 $0x31, %ymm1, %ymm0, %ymm14 # ymm14 = ymm0[2,3],ymm1[2,3] + vmovdqu %ymm7, 0x80(%rsp) + vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1] + vmovdqu 0x40(%rdi), %ymm0 + vmovdqu 0x1d0(%rdi), %ymm1 + vmovdqu %ymm3, 0xc0(%rsp) + vmovdqu 0x108(%rdi), %ymm3 + vmovdqu %ymm14, %ymm10 + vmovdqu %ymm7, 0xa0(%rsp) + vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3] + vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3] + vperm2i128 $0x20, %ymm3, %ymm2, %ymm11 # ymm11 = ymm2[0,1],ymm3[0,1] + vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3] + vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1] + vmovdqu %ymm3, 0x100(%rsp) + vperm2i128 $0x31, %ymm1, %ymm0, %ymm8 # ymm8 = ymm0[2,3],ymm1[2,3] + vmovdqu 0x128(%rdi), %ymm3 + vmovdqu 0x60(%rdi), %ymm0 + vmovdqu 0x1f0(%rdi), %ymm1 + vmovdqu %ymm7, 0xe0(%rsp) + vmovdqu %ymm11, %ymm14 + vmovdqu 0x2b8(%rdi), %ymm4 + vmovdqu 0x2f8(%rdi), %ymm5 + vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3] + vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3] + vmovdqu 0x2d8(%rdi), %ymm4 + vperm2i128 $0x20, %ymm3, %ymm2, %ymm15 # ymm15 = ymm2[0,1],ymm3[0,1] + vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3] + vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1] + vperm2i128 $0x31, %ymm1, %ymm0, %ymm9 # ymm9 = ymm0[2,3],ymm1[2,3] + vmovdqu %ymm3, 0x140(%rsp) + vmovdqu 0x80(%rdi), %ymm0 + vmovdqu 0x148(%rdi), %ymm3 + vmovdqu 0x210(%rdi), %ymm1 + vmovdqu %ymm7, 0x120(%rsp) + vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3] + vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3] + vperm2i128 $0x20, %ymm3, %ymm2, %ymm7 # ymm7 = ymm2[0,1],ymm3[0,1] + vperm2i128 $0x31, %ymm3, %ymm2, %ymm13 # ymm13 = ymm2[2,3],ymm3[2,3] + vperm2i128 $0x31, %ymm1, %ymm0, %ymm3 # ymm3 = ymm0[2,3],ymm1[2,3] + vmovdqu %ymm7, 0x160(%rsp) + vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1] + vmovdqu 0xa0(%rdi), %ymm0 + vmovdqu 0x230(%rdi), %ymm1 + vmovdqu %ymm3, 0x1a0(%rsp) + vmovdqu 0x168(%rdi), %ymm3 + vpunpcklqdq %ymm5, %ymm1, %ymm4 # ymm4 = ymm1[0],ymm5[0],ymm1[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm5[1],ymm1[3],ymm5[3] + vmovdqu %ymm7, 0x180(%rsp) + vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3] + vperm2i128 $0x20, %ymm4, %ymm2, %ymm12 # ymm12 = ymm2[0,1],ymm4[0,1] + vperm2i128 $0x20, %ymm1, %ymm0, %ymm3 # ymm3 = ymm0[0,1],ymm1[0,1] + vperm2i128 $0x31, %ymm4, %ymm2, %ymm7 # ymm7 = ymm2[2,3],ymm4[2,3] + vperm2i128 $0x31, %ymm1, %ymm0, %ymm4 # ymm4 = ymm0[2,3],ymm1[2,3] + vmovq 0x250(%rdi), %xmm0 + vmovq 0xc0(%rdi), %xmm1 + vmovdqu %ymm12, 0x1c0(%rsp) + vmovdqu %ymm4, 0x1e0(%rsp) + vpinsrq $0x1, 0x318(%rdi), %xmm0, %xmm0 + vpinsrq $0x1, 0x188(%rdi), %xmm1, %xmm1 + vinserti128 $0x1, %xmm0, %ymm1, %ymm2 + movq $0x0, %r10 + +LLmld_keccak_f1600_x4_avx2_asm: + vmovdqu 0xa0(%rsp), %ymm4 + vpxor 0x1c0(%rsp), %ymm9, %ymm0 + vmovdqu %ymm9, 0x200(%rsp) + vmovdqu %ymm10, %ymm9 + vmovdqu 0xc0(%rsp), %ymm11 + vmovdqu 0x160(%rsp), %ymm12 + vmovdqu %ymm3, 0x240(%rsp) + vpxor 0x100(%rsp), %ymm4, %ymm1 + vmovdqu 0x40(%rsp), %ymm10 + vmovdqu %ymm4, 0x220(%rsp) + vpxor %ymm3, %ymm12, %ymm12 + vmovdqu 0x20(%rsp), %ymm6 + vmovdqu 0x140(%rsp), %ymm4 + vmovdqu %ymm14, 0x2a0(%rsp) + vpxor %ymm1, %ymm0, %ymm0 + vpxor %ymm8, %ymm11, %ymm1 + vpxor 0x180(%rsp), %ymm7, %ymm11 + vmovdqu %ymm10, 0x280(%rsp) + vpxor %ymm1, %ymm12, %ymm12 + vpxor %ymm15, %ymm9, %ymm1 + vmovdqu 0xe0(%rsp), %ymm3 + vmovdqu %ymm8, 0x260(%rsp) + vpxor %ymm1, %ymm11, %ymm11 + vpxor 0x120(%rsp), %ymm14, %ymm1 + vpxor %ymm6, %ymm12, %ymm12 + vmovdqu 0x60(%rsp), %ymm8 + vpxor %ymm10, %ymm11, %ymm11 + vpxor 0x1e0(%rsp), %ymm13, %ymm10 + vpxor %ymm4, %ymm3, %ymm3 + vmovdqu %ymm4, 0x2c0(%rsp) + vpsrlq $0x3f, %ymm12, %ymm4 + vpsrlq $0x3f, %ymm11, %ymm5 + vpxor (%rsp), %ymm0, %ymm0 + vpxor %ymm1, %ymm10, %ymm10 + vmovdqu 0x80(%rsp), %ymm1 + vpxor %ymm8, %ymm10, %ymm10 + vmovdqu %ymm1, %ymm14 + vpxor 0x1a0(%rsp), %ymm2, %ymm1 + vmovdqu %ymm14, 0x2e0(%rsp) + vpxor %ymm3, %ymm1, %ymm1 + vpsllq $0x1, %ymm12, %ymm3 + vpor %ymm4, %ymm3, %ymm3 + vpsllq $0x1, %ymm11, %ymm4 + vpxor %ymm14, %ymm1, %ymm1 + vpor %ymm5, %ymm4, %ymm4 + vpsrlq $0x3f, %ymm10, %ymm14 + vpxor %ymm1, %ymm3, %ymm3 + vpsllq $0x1, %ymm10, %ymm5 + vpxor %ymm0, %ymm4, %ymm4 + vpor %ymm14, %ymm5, %ymm5 + vpxor %ymm6, %ymm4, %ymm6 + vpxor %ymm12, %ymm5, %ymm5 + vpsrlq $0x3f, %ymm1, %ymm12 + vpsllq $0x1, %ymm1, %ymm1 + vpxor %ymm7, %ymm5, %ymm7 + vpxor %ymm9, %ymm5, %ymm9 + vpor %ymm12, %ymm1, %ymm1 + vpxor (%rsp), %ymm3, %ymm12 + vpxor %ymm11, %ymm1, %ymm1 + vpsrlq $0x3f, %ymm0, %ymm11 + vpsllq $0x1, %ymm0, %ymm0 + vpxor %ymm13, %ymm1, %ymm13 + vpxor %ymm8, %ymm1, %ymm8 + vpor %ymm11, %ymm0, %ymm0 + vpxor %ymm10, %ymm0, %ymm0 + vpxor 0xc0(%rsp), %ymm4, %ymm10 + vpxor %ymm2, %ymm0, %ymm2 + vpsrlq $0x14, %ymm10, %ymm11 + vpsllq $0x2c, %ymm10, %ymm10 + vpor %ymm11, %ymm10, %ymm10 + vpxor %ymm15, %ymm5, %ymm11 + vpbroadcastq (%rsi), %ymm15 + vpsrlq $0x15, %ymm11, %ymm14 + vpsllq $0x2b, %ymm11, %ymm11 + vpor %ymm14, %ymm11, %ymm11 + vpandn %ymm11, %ymm10, %ymm14 + vpxor %ymm15, %ymm14, %ymm14 + vpxor %ymm12, %ymm14, %ymm15 + vpsrlq $0x2b, %ymm13, %ymm14 + vpsllq $0x15, %ymm13, %ymm13 + vmovdqu %ymm15, (%rsp) + vpor %ymm14, %ymm13, %ymm13 + vpandn %ymm13, %ymm11, %ymm14 + vpxor %ymm10, %ymm14, %ymm15 + vpsrlq $0x32, %ymm2, %ymm14 + vpsllq $0xe, %ymm2, %ymm2 + vmovdqu %ymm15, 0x20(%rsp) + vpor %ymm14, %ymm2, %ymm2 + vpandn %ymm2, %ymm13, %ymm14 + vpxor %ymm11, %ymm14, %ymm11 + vmovdqu %ymm11, 0x40(%rsp) + vpandn %ymm12, %ymm2, %ymm11 + vpandn %ymm10, %ymm12, %ymm12 + vpxor %ymm13, %ymm11, %ymm11 + vmovdqu %ymm11, 0x60(%rsp) + vpxor %ymm2, %ymm12, %ymm11 + vpsrlq $0x24, %ymm8, %ymm2 + vpsllq $0x1c, %ymm8, %ymm8 + vmovdqu %ymm11, 0x80(%rsp) + vpor %ymm2, %ymm8, %ymm8 + vpxor 0xe0(%rsp), %ymm0, %ymm2 + vpsrlq $0x2c, %ymm2, %ymm10 + vpsllq $0x14, %ymm2, %ymm2 + vpor %ymm10, %ymm2, %ymm2 + vpxor 0x100(%rsp), %ymm3, %ymm10 + vpsrlq $0x3d, %ymm10, %ymm11 + vpsllq $0x3, %ymm10, %ymm10 + vpor %ymm11, %ymm10, %ymm10 + vpandn %ymm10, %ymm2, %ymm11 + vpxor %ymm8, %ymm11, %ymm11 + vmovdqu %ymm11, 0xa0(%rsp) + vpxor 0x160(%rsp), %ymm4, %ymm11 + vpsrlq $0x13, %ymm11, %ymm12 + vpsllq $0x2d, %ymm11, %ymm11 + vpor %ymm12, %ymm11, %ymm11 + vpandn %ymm11, %ymm10, %ymm12 + vpxor %ymm2, %ymm12, %ymm12 + vmovdqu %ymm12, 0xc0(%rsp) + vpsrlq $0x3, %ymm7, %ymm12 + vpsllq $0x3d, %ymm7, %ymm7 + vpor %ymm12, %ymm7, %ymm7 + vpandn %ymm7, %ymm11, %ymm12 + vpxor %ymm10, %ymm12, %ymm10 + vpandn %ymm8, %ymm7, %ymm12 + vpandn %ymm2, %ymm8, %ymm8 + vpsrlq $0x3f, %ymm6, %ymm2 + vpsllq $0x1, %ymm6, %ymm6 + vpxor %ymm11, %ymm12, %ymm14 + vpor %ymm2, %ymm6, %ymm6 + vpsrlq $0x3a, %ymm9, %ymm2 + vpxor %ymm7, %ymm8, %ymm12 + vpsllq $0x6, %ymm9, %ymm9 + vmovdqu %ymm12, 0xe0(%rsp) + vpxor 0x1a0(%rsp), %ymm0, %ymm7 + vpor %ymm2, %ymm9, %ymm9 + vpxor 0x120(%rsp), %ymm1, %ymm2 + vpshufb (%rdx), %ymm7, %ymm7 + vpsrlq $0x27, %ymm2, %ymm11 + vpsllq $0x19, %ymm2, %ymm2 + vpor %ymm2, %ymm11, %ymm11 + vpandn %ymm11, %ymm9, %ymm2 + vpandn %ymm7, %ymm11, %ymm8 + vpxor %ymm6, %ymm2, %ymm12 + vpxor 0x1c0(%rsp), %ymm3, %ymm2 + vpxor %ymm9, %ymm8, %ymm8 + vmovdqu %ymm12, 0x100(%rsp) + vpsrlq $0x2e, %ymm2, %ymm12 + vpsllq $0x12, %ymm2, %ymm2 + vpor %ymm2, %ymm12, %ymm2 + vpandn %ymm2, %ymm7, %ymm12 + vpxor %ymm11, %ymm12, %ymm15 + vpandn %ymm6, %ymm2, %ymm11 + vpandn %ymm9, %ymm6, %ymm6 + vpxor %ymm7, %ymm11, %ymm12 + vmovdqu %ymm12, 0x120(%rsp) + vpxor %ymm2, %ymm6, %ymm12 + vpxor 0x2e0(%rsp), %ymm0, %ymm6 + vpxor 0x2c0(%rsp), %ymm0, %ymm0 + vmovdqu %ymm12, 0x140(%rsp) + vpsrlq $0x25, %ymm6, %ymm2 + vpsllq $0x1b, %ymm6, %ymm6 + vpor %ymm6, %ymm2, %ymm2 + vpxor 0x220(%rsp), %ymm3, %ymm6 + vpxor 0x200(%rsp), %ymm3, %ymm3 + vpsrlq $0x1c, %ymm6, %ymm7 + vpsllq $0x24, %ymm6, %ymm6 + vpor %ymm6, %ymm7, %ymm7 + vpxor 0x260(%rsp), %ymm4, %ymm6 + vpxor 0x240(%rsp), %ymm4, %ymm4 + vpsrlq $0x36, %ymm6, %ymm12 + vpsllq $0xa, %ymm6, %ymm6 + vpor %ymm6, %ymm12, %ymm12 + vpxor 0x180(%rsp), %ymm5, %ymm6 + vpxor 0x280(%rsp), %ymm5, %ymm5 + vpandn %ymm12, %ymm7, %ymm9 + vpsrlq $0x31, %ymm6, %ymm11 + vpsllq $0xf, %ymm6, %ymm6 + vpxor %ymm2, %ymm9, %ymm9 + vpor %ymm6, %ymm11, %ymm11 + vpandn %ymm11, %ymm12, %ymm6 + vpxor %ymm7, %ymm6, %ymm6 + vmovdqu %ymm6, 0x160(%rsp) + vpxor 0x1e0(%rsp), %ymm1, %ymm6 + vpxor 0x2a0(%rsp), %ymm1, %ymm1 + vpshufb (%rcx), %ymm6, %ymm6 + vpandn %ymm6, %ymm11, %ymm13 + vpxor %ymm12, %ymm13, %ymm13 + vmovdqu %ymm13, 0x180(%rsp) + vpandn %ymm2, %ymm6, %ymm13 + vpandn %ymm7, %ymm2, %ymm2 + vpxor %ymm6, %ymm2, %ymm2 + vpsrlq $0x3e, %ymm4, %ymm6 + vpxor %ymm11, %ymm13, %ymm13 + vmovdqu %ymm2, 0x1a0(%rsp) + vpsrlq $0x2, %ymm5, %ymm2 + vpsllq $0x3e, %ymm5, %ymm5 + vpor %ymm5, %ymm2, %ymm2 + vpsrlq $0x9, %ymm1, %ymm5 + vpsllq $0x37, %ymm1, %ymm1 + vpsllq $0x2, %ymm4, %ymm4 + vpor %ymm1, %ymm5, %ymm1 + vpsrlq $0x19, %ymm0, %ymm5 + vpor %ymm4, %ymm6, %ymm4 + vpsllq $0x27, %ymm0, %ymm0 + vpor %ymm0, %ymm5, %ymm5 + vpandn %ymm5, %ymm1, %ymm0 + vpxor %ymm2, %ymm0, %ymm0 + vmovdqu %ymm0, 0x1c0(%rsp) + vpsrlq $0x17, %ymm3, %ymm0 + vpsllq $0x29, %ymm3, %ymm3 + vpor %ymm3, %ymm0, %ymm0 + vpandn %ymm4, %ymm0, %ymm7 + vpandn %ymm0, %ymm5, %ymm3 + vpxor %ymm5, %ymm7, %ymm7 + vpandn %ymm2, %ymm4, %ymm5 + vpandn %ymm1, %ymm2, %ymm2 + vpxor %ymm0, %ymm5, %ymm5 + vpxor %ymm1, %ymm3, %ymm3 + vpxor %ymm4, %ymm2, %ymm2 + vmovdqu %ymm5, 0x1e0(%rsp) + addq $0x8, %rsi + addq $0x1, %r10 + cmpq $0x18, %r10 + jne LLmld_keccak_f1600_x4_avx2_asm + vmovdqu (%rsp), %ymm4 + vmovdqu 0x40(%rsp), %ymm5 + vmovdqu 0x20(%rsp), %ymm0 + vmovdqu 0x60(%rsp), %ymm1 + vmovdqu 0x1c0(%rsp), %ymm12 + vmovdqu %ymm2, 0x1c0(%rsp) + vpunpcklqdq %ymm0, %ymm4, %ymm2 # ymm2 = ymm4[0],ymm0[0],ymm4[2],ymm0[2] + vpunpckhqdq %ymm0, %ymm4, %ymm0 # ymm0 = ymm4[1],ymm0[1],ymm4[3],ymm0[3] + vpunpcklqdq %ymm1, %ymm5, %ymm4 # ymm4 = ymm5[0],ymm1[0],ymm5[2],ymm1[2] + vpunpckhqdq %ymm1, %ymm5, %ymm1 # ymm1 = ymm5[1],ymm1[1],ymm5[3],ymm1[3] + vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1] + vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3] + vmovdqu 0x80(%rsp), %ymm4 + vperm2i128 $0x20, %ymm1, %ymm0, %ymm5 # ymm5 = ymm0[0,1],ymm1[0,1] + vperm2i128 $0x31, %ymm1, %ymm0, %ymm0 # ymm0 = ymm0[2,3],ymm1[2,3] + vmovdqu %ymm6, (%rdi) + vmovdqu %ymm5, 0xc8(%rdi) + vmovdqu %ymm2, 0x190(%rdi) + vmovdqu %ymm0, 0x258(%rdi) + vmovdqu 0xa0(%rsp), %ymm0 + vpunpcklqdq %ymm0, %ymm4, %ymm2 # ymm2 = ymm4[0],ymm0[0],ymm4[2],ymm0[2] + vpunpckhqdq %ymm0, %ymm4, %ymm1 # ymm1 = ymm4[1],ymm0[1],ymm4[3],ymm0[3] + vmovdqu 0xc0(%rsp), %ymm0 + vpunpcklqdq %ymm10, %ymm0, %ymm4 # ymm4 = ymm0[0],ymm10[0],ymm0[2],ymm10[2] + vpunpckhqdq %ymm10, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm10[1],ymm0[3],ymm10[3] + vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1] + vperm2i128 $0x20, %ymm0, %ymm1, %ymm5 # ymm5 = ymm1[0,1],ymm0[0,1] + vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3] + vmovdqu 0xe0(%rsp), %ymm4 + vperm2i128 $0x31, %ymm0, %ymm1, %ymm1 # ymm1 = ymm1[2,3],ymm0[2,3] + vmovdqu 0x100(%rsp), %ymm0 + vmovdqu %ymm2, 0x1b0(%rdi) + vmovdqu %ymm1, 0x278(%rdi) + vpunpcklqdq %ymm4, %ymm14, %ymm2 # ymm2 = ymm14[0],ymm4[0],ymm14[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm14, %ymm1 # ymm1 = ymm14[1],ymm4[1],ymm14[3],ymm4[3] + vpunpcklqdq %ymm8, %ymm0, %ymm4 # ymm4 = ymm0[0],ymm8[0],ymm0[2],ymm8[2] + vpunpckhqdq %ymm8, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm8[1],ymm0[3],ymm8[3] + vmovdqu %ymm6, 0x20(%rdi) + vmovdqu %ymm5, 0xe8(%rdi) + vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1] + vperm2i128 $0x20, %ymm0, %ymm1, %ymm5 # ymm5 = ymm1[0,1],ymm0[0,1] + vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3] + vperm2i128 $0x31, %ymm0, %ymm1, %ymm1 # ymm1 = ymm1[2,3],ymm0[2,3] + vmovdqu 0x120(%rsp), %ymm4 + vmovdqu 0x140(%rsp), %ymm0 + vmovdqu %ymm2, 0x1d0(%rdi) + vmovdqu %ymm1, 0x298(%rdi) + vpunpcklqdq %ymm4, %ymm15, %ymm2 # ymm2 = ymm15[0],ymm4[0],ymm15[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm15, %ymm1 # ymm1 = ymm15[1],ymm4[1],ymm15[3],ymm4[3] + vpunpcklqdq %ymm9, %ymm0, %ymm4 # ymm4 = ymm0[0],ymm9[0],ymm0[2],ymm9[2] + vmovdqu %ymm5, 0x108(%rdi) + vpunpckhqdq %ymm9, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm9[1],ymm0[3],ymm9[3] + vmovdqu %ymm6, 0x40(%rdi) + vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1] + vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3] + vperm2i128 $0x20, %ymm0, %ymm1, %ymm5 # ymm5 = ymm1[0,1],ymm0[0,1] + vmovdqu 0x160(%rsp), %ymm4 + vperm2i128 $0x31, %ymm0, %ymm1, %ymm1 # ymm1 = ymm1[2,3],ymm0[2,3] + vmovdqu 0x180(%rsp), %ymm0 + vmovdqu %ymm5, 0x128(%rdi) + vmovdqu 0x1a0(%rsp), %ymm5 + vmovdqu %ymm2, 0x1f0(%rdi) + vpunpcklqdq %ymm0, %ymm4, %ymm2 # ymm2 = ymm4[0],ymm0[0],ymm4[2],ymm0[2] + vpunpckhqdq %ymm0, %ymm4, %ymm0 # ymm0 = ymm4[1],ymm0[1],ymm4[3],ymm0[3] + vpunpcklqdq %ymm5, %ymm13, %ymm4 # ymm4 = ymm13[0],ymm5[0],ymm13[2],ymm5[2] + vmovdqu %ymm6, 0x60(%rdi) + vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1] + vmovdqu %ymm1, 0x2b8(%rdi) + vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3] + vpunpckhqdq %ymm5, %ymm13, %ymm1 # ymm1 = ymm13[1],ymm5[1],ymm13[3],ymm5[3] + vmovdqu %ymm6, 0x80(%rdi) + vmovdqu 0x1e0(%rsp), %ymm4 + vperm2i128 $0x20, %ymm1, %ymm0, %ymm5 # ymm5 = ymm0[0,1],ymm1[0,1] + vperm2i128 $0x31, %ymm1, %ymm0, %ymm0 # ymm0 = ymm0[2,3],ymm1[2,3] + vmovdqu %ymm2, 0x210(%rdi) + vpunpcklqdq %ymm3, %ymm12, %ymm2 # ymm2 = ymm12[0],ymm3[0],ymm12[2],ymm3[2] + vmovdqu %ymm0, 0x2d8(%rdi) + vpunpckhqdq %ymm3, %ymm12, %ymm0 # ymm0 = ymm12[1],ymm3[1],ymm12[3],ymm3[3] + vpunpcklqdq %ymm4, %ymm7, %ymm3 # ymm3 = ymm7[0],ymm4[0],ymm7[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm7, %ymm1 # ymm1 = ymm7[1],ymm4[1],ymm7[3],ymm4[3] + vmovdqu %ymm5, 0x148(%rdi) + vperm2i128 $0x20, %ymm3, %ymm2, %ymm5 # ymm5 = ymm2[0,1],ymm3[0,1] + vperm2i128 $0x31, %ymm3, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm3[2,3] + vmovdqu 0x1c0(%rsp), %ymm3 + vperm2i128 $0x20, %ymm1, %ymm0, %ymm4 # ymm4 = ymm0[0,1],ymm1[0,1] + vperm2i128 $0x31, %ymm1, %ymm0, %ymm0 # ymm0 = ymm0[2,3],ymm1[2,3] + vmovdqu %ymm5, 0xa0(%rdi) + vextracti128 $0x1, %ymm3, %xmm15 + vmovdqu %ymm4, 0x168(%rdi) + vmovdqu %ymm2, 0x230(%rdi) + vmovdqu %ymm0, 0x2f8(%rdi) + vmovq %xmm3, 0xc0(%rdi) + vmovhpd %xmm3, 0x188(%rdi) + vmovq %xmm15, 0x250(%rdi) + vmovhpd %xmm15, 0x318(%rdi) + movq %r11, %rsp + .cfi_def_cfa_register %rsp + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(keccak_f1600_x4_avx2_asm) + +#endif /* MLD_FIPS202_X86_64_NEED_X4_AVX2 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/fips202/native/x86_64/src/keccakf1600_constants.c b/cbits/mldsa/src/fips202/native/x86_64/src/keccakf1600_constants.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/fips202/native/x86_64/src/keccakf1600_constants.c @@ -0,0 +1,52 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mldsa-native repository. + * Do not modify it directly. + */ + +#include "../../../../common.h" +#if defined(MLD_FIPS202_X86_64_NEED_X4_AVX2) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +#include + +#include "fips202_native_x86_64.h" + +MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint64_t + mld_keccakf1600_round_constants[24] = { + 0x0000000000000001, 0x0000000000008082, 0x800000000000808a, + 0x8000000080008000, 0x000000000000808b, 0x0000000080000001, + 0x8000000080008081, 0x8000000000008009, 0x000000000000008a, + 0x0000000000000088, 0x0000000080008009, 0x000000008000000a, + 0x000000008000808b, 0x800000000000008b, 0x8000000000008089, + 0x8000000000008003, 0x8000000000008002, 0x8000000000000080, + 0x000000000000800a, 0x800000008000000a, 0x8000000080008081, + 0x8000000000008080, 0x0000000080000001, 0x8000000080008008, +}; + +MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint64_t mld_keccak_rho8[4] = { + 0x0605040302010007, + 0x0e0d0c0b0a09080f, + 0x1615141312111017, + 0x1e1d1c1b1a19181f, +}; + +MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint64_t mld_keccak_rho56[4] = { + 0x0007060504030201, + 0x080f0e0d0c0b0a09, + 0x1017161514131211, + 0x181f1e1d1c1b1a19, +}; + +#else /* MLD_FIPS202_X86_64_NEED_X4_AVX2 && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +MLD_EMPTY_CU(fips202_x86_64_constants) + +#endif /* !(MLD_FIPS202_X86_64_NEED_X4_AVX2 && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mldsa/src/native/aarch64/meta.h b/cbits/mldsa/src/native/aarch64/meta.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/meta.h @@ -0,0 +1,314 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_NATIVE_AARCH64_META_H +#define MLD_NATIVE_AARCH64_META_H + +/* Set of primitives that this backend replaces */ +#define MLD_USE_NATIVE_NTT +#define MLD_USE_NATIVE_INTT +#define MLD_USE_NATIVE_REJ_UNIFORM +#define MLD_USE_NATIVE_REJ_UNIFORM_ETA2 +#define MLD_USE_NATIVE_REJ_UNIFORM_ETA4 +#define MLD_USE_NATIVE_POLY_DECOMPOSE_32 +#define MLD_USE_NATIVE_POLY_DECOMPOSE_88 +#define MLD_USE_NATIVE_POLY_CADDQ +#define MLD_USE_NATIVE_POLY_USE_HINT_32 +#define MLD_USE_NATIVE_POLY_USE_HINT_88 +#define MLD_USE_NATIVE_POLY_CHKNORM +#define MLD_USE_NATIVE_POLYZ_UNPACK_17 +#define MLD_USE_NATIVE_POLYZ_UNPACK_19 +#define MLD_USE_NATIVE_POINTWISE_MONTGOMERY +#define MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4 +#define MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5 +#define MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7 + +/* Identifier for this backend so that source and assembly files + * in the build can be appropriately guarded. */ +#define MLD_ARITH_BACKEND_AARCH64 + + +#if !defined(__ASSEMBLER__) +#include "../api.h" +#include "src/arith_native_aarch64.h" + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_ntt_native(int32_t data[MLDSA_N]) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_ntt_aarch64_asm(data, mld_aarch64_ntt_zetas_layer123456, + mld_aarch64_ntt_zetas_layer78); + return MLD_NATIVE_FUNC_SUCCESS; +} + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_intt_native(int32_t data[MLDSA_N]) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_intt_aarch64_asm(data, mld_aarch64_intt_zetas_layer78, + mld_aarch64_intt_zetas_layer123456); + return MLD_NATIVE_FUNC_SUCCESS; +} + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_rej_uniform_native(int32_t *r, unsigned len, + const uint8_t *buf, + unsigned buflen) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON) || len != MLDSA_N || + buflen % 24 != 0) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + + /* Safety: outlen is at most MLDSA_N, hence, this cast is safe. */ + return (int)mld_rej_uniform_aarch64_asm(r, buf, buflen, + mld_rej_uniform_table); +} + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 2 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_rej_uniform_eta2_native(int32_t *r, unsigned len, + const uint8_t *buf, + unsigned buflen) +{ + uint64_t outlen; + /* AArch64 implementation assumes specific buffer lengths */ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON) || len != MLDSA_N || + buflen != MLD_AARCH64_REJ_UNIFORM_ETA2_BUFLEN) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + /* Constant time: Inputs and outputs to this function are secret. + * It is safe to leak which coefficients are accepted/rejected. + * The assembly implementation must not leak any other information about the + * accepted coefficients. Constant-time testing cannot cover this, and we + * hence have to manually verify the assembly. + * We declassify prior the input data and mark the outputs as secret. + */ + MLD_CT_TESTING_DECLASSIFY(buf, buflen); + outlen = mld_rej_uniform_eta2_aarch64_asm(r, buf, buflen, + mld_rej_uniform_eta_table); + MLD_CT_TESTING_SECRET(r, sizeof(int32_t) * outlen); + /* Safety: outlen is at most MLDSA_N and, hence, this cast is safe. */ + return (int)outlen; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 2 */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 4 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_rej_uniform_eta4_native(int32_t *r, unsigned len, + const uint8_t *buf, + unsigned buflen) +{ + uint64_t outlen; + /* AArch64 implementation assumes specific buffer lengths */ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON) || len != MLDSA_N || + buflen != MLD_AARCH64_REJ_UNIFORM_ETA4_BUFLEN) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + /* Constant time: Inputs and outputs to this function are secret. + * It is safe to leak which coefficients are accepted/rejected. + * The assembly implementation must not leak any other information about the + * accepted coefficients. Constant-time testing cannot cover this, and we + * hence have to manually verify the assembly. + * We declassify prior the input data and mark the outputs as secret. + */ + MLD_CT_TESTING_DECLASSIFY(buf, buflen); + outlen = mld_rej_uniform_eta4_aarch64_asm(r, buf, buflen, + mld_rej_uniform_eta_table); + MLD_CT_TESTING_SECRET(r, sizeof(int32_t) * outlen); + /* Safety: outlen is at most MLDSA_N and, hence, this cast is safe. */ + return (int)outlen; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 4 */ +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_decompose_32_native(int32_t *a1, int32_t *a0) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_poly_decompose_32_aarch64_asm(a1, a0); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87 */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_decompose_88_native(int32_t *a1, int32_t *a0) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_poly_decompose_88_aarch64_asm(a1, a0); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \ + */ +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_caddq_native(int32_t a[MLDSA_N]) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_poly_caddq_aarch64_asm(a); + return MLD_NATIVE_FUNC_SUCCESS; +} + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_use_hint_32_native(int32_t *a, const int32_t *h) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_poly_use_hint_32_aarch64_asm(a, h); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87 */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_use_hint_88_native(int32_t *a, const int32_t *h) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_poly_use_hint_88_aarch64_asm(a, h); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \ + */ +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_chknorm_native(const int32_t *a, int32_t B) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + return mld_poly_chknorm_aarch64_asm(a, B); +} + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_polyz_unpack_17_native(int32_t *r, const uint8_t *buf) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_polyz_unpack_17_aarch64_asm(r, buf, mld_polyz_unpack_17_indices); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \ + */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_polyz_unpack_19_native(int32_t *r, const uint8_t *buf) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_polyz_unpack_19_aarch64_asm(r, buf, mld_polyz_unpack_19_indices); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87 */ +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \ + defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_pointwise_montgomery_native( + int32_t a[MLDSA_N], const int32_t b[MLDSA_N]) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_poly_pointwise_montgomery_aarch64_asm(a, b); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API || \ + MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 4 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l4_native( + int32_t w[MLDSA_N], const int32_t u[4][MLDSA_N], + const int32_t v[4][MLDSA_N]) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm(w, u, v); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 4 */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 5 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l5_native( + int32_t w[MLDSA_N], const int32_t u[5][MLDSA_N], + const int32_t v[5][MLDSA_N]) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm(w, u, v); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 5 */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 7 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l7_native( + int32_t w[MLDSA_N], const int32_t u[7][MLDSA_N], + const int32_t v[7][MLDSA_N]) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_AARCH64_NEON)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm(w, u, v); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 7 */ + +#endif /* !__ASSEMBLER__ */ +#endif /* !MLD_NATIVE_AARCH64_META_H */ diff --git a/cbits/mldsa/src/native/aarch64/src/aarch64_zetas.c b/cbits/mldsa/src/native/aarch64/src/aarch64_zetas.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/aarch64_zetas.c @@ -0,0 +1,248 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mldsa-native repository. + * Do not modify it directly. + */ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_AARCH64) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +#include "arith_native_aarch64.h" + +/* + * Table of zeta values used in the AArch64 forward NTT + * See autogen for details. + */ +MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const int32_t + mld_aarch64_ntt_zetas_layer123456[144] = { + -3572223, -915382907, 3765607, 964937599, 3761513, 963888510, + -3201494, -820383522, -2883726, -738955404, -3145678, -806080660, + -3201430, -820367122, 0, 0, -601683, -154181397, + -3370349, -863652652, -4063053, -1041158200, 3602218, 923069133, + 3182878, 815613168, 2740543, 702264730, -3586446, -919027554, + 0, 0, 3542485, 907762539, 2663378, 682491182, + -1674615, -429120452, -3110818, -797147778, 2101410, 538486762, + 3704823, 949361686, 1159875, 297218217, 0, 0, + 2682288, 687336873, -3524442, -903139016, -434125, -111244624, + 394148, 101000509, 928749, 237992130, 1095468, 280713909, + -3506380, -898510625, 0, 0, 2129892, 545785280, + 676590, 173376332, -1335936, -342333886, 2071829, 530906624, + -4018989, -1029866791, 3241972, 830756018, 2156050, 552488273, + 0, 0, 3764867, 964747974, -3227876, -827143915, + 1714295, 439288460, 3415069, 875112161, 1759347, 450833045, + -817536, -209493775, -3574466, -915957677, 0, 0, + -1005239, -257592709, 2453983, 628833668, 1460718, 374309300, + 3756790, 962678241, -1935799, -496048908, -1716988, -439978542, + -3950053, -1012201926, 0, 0, 557458, 142848732, + -642628, -164673562, -3585098, -918682129, -2897314, -742437332, + 3192354, 818041395, 556856, 142694469, 3870317, 991769559, + 0, 0, -1221177, -312926867, 2815639, 721508096, + 2283733, 585207070, 2917338, 747568486, 1853806, 475038184, + 3345963, 857403734, 1858416, 476219497, 0, 0, +}; + +MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const int32_t + mld_aarch64_ntt_zetas_layer78[384] = { + 3073009, 1277625, -2635473, 3852015, 787459213, + 327391679, -675340520, 987079667, 1753, -2659525, + 2660408, -59148, 449207, -681503850, 681730119, + -15156688, -1935420, -1455890, -1780227, 2772600, + -495951789, -373072124, -456183549, 710479343, 4183372, + -3222807, -3121440, -274060, 1071989969, -825844983, + -799869667, -70227934, 1182243, 636927, -3956745, + -3284915, 302950022, 163212680, -1013916752, -841760171, + 87208, -3965306, -2296397, -3716946, 22347069, + -1016110510, -588452222, -952468207, 2508980, 2028118, + 1937570, -3815725, 642926661, 519705671, 496502727, + -977780347, -27812, 1009365, -1979497, -3956944, + -7126831, 258649997, -507246529, -1013967746, 822541, + -2454145, 1596822, -3759465, 210776307, -628875181, + 409185979, -963363710, 2811291, -2983781, -1109516, + 4158088, 720393920, -764594519, -284313712, 1065510939, + -1685153, 2678278, -3551006, -250446, -431820817, + 686309310, -909946047, -64176841, -3410568, -3768948, + 635956, -2455377, -873958779, -965793731, 162963861, + -629190881, 1528066, 482649, 1148858, -2962264, + 391567239, 123678909, 294395108, -759080783, -4146264, + 2192938, 2387513, -268456, -1062481036, 561940831, + 611800717, -68791907, -1772588, -1727088, -3611750, + -3180456, -454226054, -442566669, -925511710, -814992530, + -565603, 169688, 2462444, -3334383, -144935890, + 43482586, 631001801, -854436357, 3747250, 1239911, + 3195676, 1254190, 960233614, 317727459, 818892658, + 321386456, 2296099, -3838479, 2642980, -12417, + 588375860, -983611064, 677264190, -3181859, -4166425, + -3488383, 1987814, -3197248, -1067647297, -893898890, + 509377762, -819295484, 2998219, -89301, -1354892, + -1310261, 768294260, -22883400, -347191365, -335754661, + 141835, 2513018, 613238, -2218467, 36345249, + 643961400, 157142369, -568482643, 1736313, 235407, + -3250154, 3258457, 444930577, 60323094, -832852657, + 834980303, -458740, 4040196, 2039144, -818761, + -117552223, 1035301089, 522531086, -209807681, -1921994, + -3472069, -1879878, -2178965, -492511373, -889718424, + -481719139, -558360247, -2579253, 1787943, -2391089, + -2254727, -660934133, 458160776, -612717067, -577774276, + -1623354, -2374402, 586241, 527981, -415984810, + -608441020, 150224382, 135295244, 2105286, -2033807, + -1179613, -2743411, 539479988, -521163479, -302276083, + -702999655, 3482206, -4182915, -1300016, -2362063, + 892316032, -1071872863, -333129378, -605279149, -1476985, + 2491325, 507927, -724804, -378477722, 638402564, + 130156402, -185731180, 1994046, -1393159, -1187885, + -1834526, 510974714, -356997292, -304395785, -470097680, + -1317678, 2461387, 3035980, 621164, -337655269, + 630730945, 777970524, 159173408, -3033742, 2647994, + -2612853, 749577, -777397036, 678549029, -669544140, + 192079267, -338420, 3009748, 4148469, -4022750, + -86720197, 771248568, 1063046068, -1030830548, 3901472, + -1226661, 2925816, 3374250, 999753034, -314332144, + 749740976, 864652284, 3980599, -1615530, 1665318, + 1163598, 1020029345, -413979908, 426738094, 298172236, + 2569011, 1723229, 2028038, -3369273, 658309618, + 441577800, 519685171, -863376927, 1356448, -2775755, + 2683270, -2778788, 347590090, -711287812, 687588511, + -712065019, 3994671, -1370517, 3363542, 545376, + 1023635298, -351195274, 861908357, 139752717, -11879, + 3020393, 214880, -770441, -3043996, 773976352, + 55063046, -197425671, -3467665, 2312838, -653275, + -459163, -888589898, 592665232, -167401858, -117660617, + 3105558, 508145, 860144, 140244, 795799901, + 130212265, 220412084, 35937555, -1103344, -553718, + 3430436, -1514152, -282732136, -141890356, 879049958, + -388001774, 348812, -327848, 1011223, -2354215, + 89383150, -84011120, 259126110, -603268097, -2185084, + 2358373, -3014420, 2926054, -559928242, 604333585, + -772445769, 749801963, 3123762, -2193087, -1716814, + -392707, 800464680, -561979013, -439933955, -100631253, + -3818627, -1922253, -2236726, 1744507, -978523985, + -492577742, -573161516, 447030292, -303005, -3974485, + 1900052, 1054478, -77645096, -1018462631, 486888731, + 270210213, 3531229, -3773731, -781875, -731434, + 904878186, -967019376, -200355636, -187430119, +}; + +MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const int32_t + mld_aarch64_intt_zetas_layer78[384] = { + -1744507, 2236726, 1922253, 3818627, -447030292, + 573161516, 492577742, 978523985, 731434, 781875, + 3773731, -3531229, 187430119, 200355636, 967019376, + -904878186, -1054478, -1900052, 3974485, 303005, + -270210213, -486888731, 1018462631, 77645096, 2354215, + -1011223, 327848, -348812, 603268097, -259126110, + 84011120, -89383150, 392707, 1716814, 2193087, + -3123762, 100631253, 439933955, 561979013, -800464680, + -2926054, 3014420, -2358373, 2185084, -749801963, + 772445769, -604333585, 559928242, 459163, 653275, + -2312838, 3467665, 117660617, 167401858, -592665232, + 888589898, 1514152, -3430436, 553718, 1103344, + 388001774, -879049958, 141890356, 282732136, -140244, + -860144, -508145, -3105558, -35937555, -220412084, + -130212265, -795799901, 2778788, -2683270, 2775755, + -1356448, 712065019, -687588511, 711287812, -347590090, + 770441, -214880, -3020393, 11879, 197425671, + -55063046, -773976352, 3043996, -545376, -3363542, + 1370517, -3994671, -139752717, -861908357, 351195274, + -1023635298, -3374250, -2925816, 1226661, -3901472, + -864652284, -749740976, 314332144, -999753034, 3369273, + -2028038, -1723229, -2569011, 863376927, -519685171, + -441577800, -658309618, -1163598, -1665318, 1615530, + -3980599, -298172236, -426738094, 413979908, -1020029345, + -621164, -3035980, -2461387, 1317678, -159173408, + -777970524, -630730945, 337655269, 4022750, -4148469, + -3009748, 338420, 1030830548, -1063046068, -771248568, + 86720197, -749577, 2612853, -2647994, 3033742, + -192079267, 669544140, -678549029, 777397036, 2362063, + 1300016, 4182915, -3482206, 605279149, 333129378, + 1071872863, -892316032, 1834526, 1187885, 1393159, + -1994046, 470097680, 304395785, 356997292, -510974714, + 724804, -507927, -2491325, 1476985, 185731180, + -130156402, -638402564, 378477722, 2254727, 2391089, + -1787943, 2579253, 577774276, 612717067, -458160776, + 660934133, 2743411, 1179613, 2033807, -2105286, + 702999655, 302276083, 521163479, -539479988, -527981, + -586241, 2374402, 1623354, -135295244, -150224382, + 608441020, 415984810, -3258457, 3250154, -235407, + -1736313, -834980303, 832852657, -60323094, -444930577, + 2178965, 1879878, 3472069, 1921994, 558360247, + 481719139, 889718424, 492511373, 818761, -2039144, + -4040196, 458740, 209807681, -522531086, -1035301089, + 117552223, 3197248, -1987814, 3488383, 4166425, + 819295484, -509377762, 893898890, 1067647297, 2218467, + -613238, -2513018, -141835, 568482643, -157142369, + -643961400, -36345249, 1310261, 1354892, 89301, + -2998219, 335754661, 347191365, 22883400, -768294260, + 3334383, -2462444, -169688, 565603, 854436357, + -631001801, -43482586, 144935890, 12417, -2642980, + 3838479, -2296099, 3181859, -677264190, 983611064, + -588375860, -1254190, -3195676, -1239911, -3747250, + -321386456, -818892658, -317727459, -960233614, 2962264, + -1148858, -482649, -1528066, 759080783, -294395108, + -123678909, -391567239, 3180456, 3611750, 1727088, + 1772588, 814992530, 925511710, 442566669, 454226054, + 268456, -2387513, -2192938, 4146264, 68791907, + -611800717, -561940831, 1062481036, -4158088, 1109516, + 2983781, -2811291, -1065510939, 284313712, 764594519, + -720393920, 2455377, -635956, 3768948, 3410568, + 629190881, -162963861, 965793731, 873958779, 250446, + 3551006, -2678278, 1685153, 64176841, 909946047, + -686309310, 431820817, 3815725, -1937570, -2028118, + -2508980, 977780347, -496502727, -519705671, -642926661, + 3759465, -1596822, 2454145, -822541, 963363710, + -409185979, 628875181, -210776307, 3956944, 1979497, + -1009365, 27812, 1013967746, 507246529, -258649997, + 7126831, 274060, 3121440, 3222807, -4183372, + 70227934, 799869667, 825844983, -1071989969, 3716946, + 2296397, 3965306, -87208, 952468207, 588452222, + 1016110510, -22347069, 3284915, 3956745, -636927, + -1182243, 841760171, 1013916752, -163212680, -302950022, + -3852015, 2635473, -1277625, -3073009, -987079667, + 675340520, -327391679, -787459213, -2772600, 1780227, + 1455890, 1935420, -710479343, 456183549, 373072124, + 495951789, 59148, -2660408, 2659525, -1753, + 15156688, -681730119, 681503850, -449207, +}; + +MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const int32_t + mld_aarch64_intt_zetas_layer123456[160] = { + -2283733, -585207070, 0, 0, -1858416, -476219497, + -3345963, -857403734, -2815639, -721508096, 0, 0, + -1853806, -475038184, -2917338, -747568486, 3585098, 918682129, + 0, 0, -3870317, -991769559, -556856, -142694469, + 642628, 164673562, 0, 0, -3192354, -818041395, + 2897314, 742437332, -1460718, -374309300, 0, 0, + 3950053, 1012201926, 1716988, 439978542, -2453983, -628833668, + 0, 0, 1935799, 496048908, -3756790, -962678241, + -1714295, -439288460, 0, 0, 3574466, 915957677, + 817536, 209493775, 3227876, 827143915, 0, 0, + -1759347, -450833045, -3415069, -875112161, 1335936, 342333886, + 0, 0, -2156050, -552488273, -3241972, -830756018, + -676590, -173376332, 0, 0, 4018989, 1029866791, + -2071829, -530906624, 434125, 111244624, 0, 0, + 3506380, 898510625, -1095468, -280713909, 3524442, 903139016, + 0, 0, -928749, -237992130, -394148, -101000509, + 1674615, 429120452, 0, 0, -1159875, -297218217, + -3704823, -949361686, -2663378, -682491182, 0, 0, + -2101410, -538486762, 3110818, 797147778, 4063053, 1041158200, + 0, 0, 3586446, 919027554, -2740543, -702264730, + 3370349, 863652652, 0, 0, -3182878, -815613168, + -3602218, -923069133, -294725, -75523344, -3761513, -963888510, + -3765607, -964937599, 3201430, 820367122, 3145678, 806080660, + 2883726, 738955404, 3201494, 820383522, 1221177, 312926867, + -557458, -142848732, 1005239, 257592709, -3764867, -964747974, + -2129892, -545785280, -2682288, -687336873, -3542485, -907762539, + 601683, 154181397, 0, 0, +}; + +#else /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +MLD_EMPTY_CU(aarch64_zetas) + +#endif /* !(MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mldsa/src/native/aarch64/src/arith_native_aarch64.h b/cbits/mldsa/src/native/aarch64/src/arith_native_aarch64.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/arith_native_aarch64.h @@ -0,0 +1,367 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H +#define MLD_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H + +#include "../../../cbmc.h" +#include "../../../common.h" + +#define mld_aarch64_ntt_zetas_layer123456 \ + MLD_NAMESPACE(aarch64_ntt_zetas_layer123456) +#define mld_aarch64_ntt_zetas_layer78 MLD_NAMESPACE(aarch64_ntt_zetas_layer78) + +#define mld_aarch64_intt_zetas_layer78 MLD_NAMESPACE(aarch64_intt_zetas_layer78) +#define mld_aarch64_intt_zetas_layer123456 \ + MLD_NAMESPACE(aarch64_intt_zetas_layer123456) + +MLD_INTERNAL_DATA_DECLARATION const int32_t + mld_aarch64_ntt_zetas_layer123456[144]; +MLD_INTERNAL_DATA_DECLARATION const int32_t mld_aarch64_ntt_zetas_layer78[384]; + +MLD_INTERNAL_DATA_DECLARATION const int32_t mld_aarch64_intt_zetas_layer78[384]; +MLD_INTERNAL_DATA_DECLARATION const int32_t + mld_aarch64_intt_zetas_layer123456[160]; + +#define mld_rej_uniform_table MLD_NAMESPACE(rej_uniform_table) +MLD_INTERNAL_DATA_DECLARATION const uint8_t mld_rej_uniform_table[256]; +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +#define mld_rej_uniform_eta_table MLD_NAMESPACE(rej_uniform_eta_table) +MLD_INTERNAL_DATA_DECLARATION const uint8_t mld_rej_uniform_eta_table[4096]; +#endif + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44 +#define mld_polyz_unpack_17_indices MLD_NAMESPACE(polyz_unpack_17_indices) +MLD_INTERNAL_DATA_DECLARATION const uint8_t mld_polyz_unpack_17_indices[64]; +#endif +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) +#define mld_polyz_unpack_19_indices MLD_NAMESPACE(polyz_unpack_19_indices) +MLD_INTERNAL_DATA_DECLARATION const uint8_t mld_polyz_unpack_19_indices[64]; +#endif +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + + +/* + * Sampling 256 coefficients mod 15 using rejection sampling from 4 bits. + * Expected number of required bytes: (256 * (16/15))/2 = 136.5 bytes. + * We sample 1 block (=136 bytes) of SHAKE256_RATE output initially. + * Sampling 2 blocks initially results in slightly worse performance. + */ +#define MLD_AARCH64_REJ_UNIFORM_ETA2_BUFLEN (1 * 136) +/* + * Sampling 256 coefficients mod 9 using rejection sampling from 4 bits. + * Expected number of required bytes: (256 * (16/9))/2 = 227.5 bytes. + * We sample 2 blocks (=272 bytes) of SHAKE256_RATE output initially. + */ +#define MLD_AARCH64_REJ_UNIFORM_ETA4_BUFLEN (2 * 136) + +#define mld_ntt_aarch64_asm MLD_NAMESPACE(ntt_aarch64_asm) +void mld_ntt_aarch64_asm(int32_t r[MLDSA_N], const int32_t zetas_l123456[144], + const int32_t zetas_l78[384]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mldsa_ntt_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + /* check-magic: 8380417 == MLDSA_Q */ + requires(array_abs_bound(r, 0, MLDSA_N, 8380417)) + requires(zetas_l123456 == mld_aarch64_ntt_zetas_layer123456) + requires(zetas_l78 == mld_aarch64_ntt_zetas_layer78) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + /* check-magic: off */ + ensures(array_abs_bound(r, 0, MLDSA_N, 75423753)) + /* check-magic: on */ +); + +#define mld_intt_aarch64_asm MLD_NAMESPACE(intt_aarch64_asm) +void mld_intt_aarch64_asm(int32_t r[MLDSA_N], const int32_t zetas_l78[384], + const int32_t zetas_l123456[160]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mldsa_intt_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(array_abs_bound(r, 0, MLDSA_N, 8380417)) + requires(zetas_l78 == mld_aarch64_intt_zetas_layer78) + requires(zetas_l123456 == mld_aarch64_intt_zetas_layer123456) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + /* check-magic: off */ + ensures(array_abs_bound(r, 0, MLDSA_N, 8380417)) + /* check-magic: on */ +); + +#define mld_rej_uniform_aarch64_asm MLD_NAMESPACE(rej_uniform_aarch64_asm) +MLD_MUST_CHECK_RETURN_VALUE +uint64_t mld_rej_uniform_aarch64_asm(int32_t r[MLDSA_N], const uint8_t *buf, + unsigned buflen, const uint8_t table[256]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mldsa_rej_uniform_aarch64_asm.ml. */ +__contract__( + requires(buflen % 24 == 0) + requires(memory_no_alias(buf, buflen)) + requires(table == mld_rej_uniform_table) + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(return_value <= MLDSA_N) + ensures(array_bound(r, 0, (unsigned) return_value, 0, MLDSA_Q)) +); + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +#define mld_rej_uniform_eta2_aarch64_asm \ + MLD_NAMESPACE(rej_uniform_eta2_aarch64_asm) +MLD_MUST_CHECK_RETURN_VALUE +uint64_t mld_rej_uniform_eta2_aarch64_asm(int32_t r[MLDSA_N], + const uint8_t *buf, unsigned buflen, + const uint8_t table[4096]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mldsa_rej_uniform_eta2_aarch64_asm.ml */ +__contract__( + requires(buflen % 8 == 0) + requires(buflen >= 8) + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(buf, buflen)) + requires(table == mld_rej_uniform_eta_table) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(return_value <= MLDSA_N) + /* check-magic: 3 == 2 + 1 (asm is eta=2-specific) */ + ensures(array_abs_bound(r, 0, return_value, 3)) +); + +#define mld_rej_uniform_eta4_aarch64_asm \ + MLD_NAMESPACE(rej_uniform_eta4_aarch64_asm) +MLD_MUST_CHECK_RETURN_VALUE +uint64_t mld_rej_uniform_eta4_aarch64_asm(int32_t r[MLDSA_N], + const uint8_t *buf, unsigned buflen, + const uint8_t table[4096]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mldsa_rej_uniform_eta4_aarch64_asm.ml */ +__contract__( + requires(buflen % 8 == 0) + requires(buflen >= 8) + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(buf, buflen)) + requires(table == mld_rej_uniform_eta_table) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(return_value <= MLDSA_N) + /* check-magic: 5 == 4 + 1 (asm is eta=4-specific) */ + ensures(array_abs_bound(r, 0, return_value, 5)) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +#define mld_poly_decompose_32_aarch64_asm \ + MLD_NAMESPACE(poly_decompose_32_aarch64_asm) +void mld_poly_decompose_32_aarch64_asm(int32_t a1[MLDSA_N], int32_t a0[MLDSA_N]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mldsa_poly_decompose_32_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(a1, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a0, sizeof(int32_t) * MLDSA_N)) + requires(array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q)) + assigns(memory_slice(a1, sizeof(int32_t) * MLDSA_N)) + assigns(memory_slice(a0, sizeof(int32_t) * MLDSA_N)) + /* check-magic: 16 == (MLDSA_Q - 1) / (2 * ((MLDSA_Q - 1) / 32)) */ + ensures(array_bound(a1, 0, MLDSA_N, 0, 16)) + /* check-magic: 261889 == (MLDSA_Q - 1) / 32 + 1 */ + ensures(array_abs_bound(a0, 0, MLDSA_N, 261889)) +); + +#define mld_poly_decompose_88_aarch64_asm \ + MLD_NAMESPACE(poly_decompose_88_aarch64_asm) +void mld_poly_decompose_88_aarch64_asm(int32_t a1[MLDSA_N], int32_t a0[MLDSA_N]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mldsa_poly_decompose_88_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(a1, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a0, sizeof(int32_t) * MLDSA_N)) + requires(array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q)) + assigns(memory_slice(a1, sizeof(int32_t) * MLDSA_N)) + assigns(memory_slice(a0, sizeof(int32_t) * MLDSA_N)) + /* check-magic: 44 == (MLDSA_Q - 1) / (2 * ((MLDSA_Q - 1) / 88)) */ + ensures(array_bound(a1, 0, MLDSA_N, 0, 44)) + /* check-magic: 95233 == (MLDSA_Q - 1) / 88 + 1 */ + ensures(array_abs_bound(a0, 0, MLDSA_N, 95233)) +); +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#define mld_poly_caddq_aarch64_asm MLD_NAMESPACE(poly_caddq_aarch64_asm) +void mld_poly_caddq_aarch64_asm(int32_t a[MLDSA_N]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mldsa_poly_caddq_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N)) + requires(array_abs_bound(a, 0, MLDSA_N, MLDSA_Q)) + assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N)) + ensures(array_bound(a, 0, MLDSA_N, 0, MLDSA_Q)) +); + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +#define mld_poly_use_hint_32_aarch64_asm \ + MLD_NAMESPACE(poly_use_hint_32_aarch64_asm) +void mld_poly_use_hint_32_aarch64_asm(int32_t a[MLDSA_N], + const int32_t h[MLDSA_N]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mldsa_poly_use_hint_32_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(h, sizeof(int32_t) * MLDSA_N)) + requires(array_bound(a, 0, MLDSA_N, 0, MLDSA_Q)) + requires(array_bound(h, 0, MLDSA_N, 0, 2)) + assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N)) + ensures(array_bound(a, 0, MLDSA_N, 0, 16)) +); + +#define mld_poly_use_hint_88_aarch64_asm \ + MLD_NAMESPACE(poly_use_hint_88_aarch64_asm) +void mld_poly_use_hint_88_aarch64_asm(int32_t a[MLDSA_N], + const int32_t h[MLDSA_N]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mldsa_poly_use_hint_88_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(h, sizeof(int32_t) * MLDSA_N)) + requires(array_bound(a, 0, MLDSA_N, 0, MLDSA_Q)) + requires(array_bound(h, 0, MLDSA_N, 0, 2)) + assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N)) + ensures(array_bound(a, 0, MLDSA_N, 0, 44)) +); +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#define mld_poly_chknorm_aarch64_asm MLD_NAMESPACE(poly_chknorm_aarch64_asm) +MLD_MUST_CHECK_RETURN_VALUE +int mld_poly_chknorm_aarch64_asm(const int32_t a[MLDSA_N], int32_t B) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mldsa_poly_chknorm_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N)) + /* HOL Light precondition: abs(ival(x i)) < 2^31, i.e., a[i] != INT32_MIN */ + requires(forall(k0, 0, MLDSA_N, a[k0] > INT32_MIN)) + ensures(return_value == 0 || return_value == 1) + ensures((return_value == 0) == array_abs_bound(a, 0, MLDSA_N, B)) +); + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44 +#define mld_polyz_unpack_17_aarch64_asm \ + MLD_NAMESPACE(polyz_unpack_17_aarch64_asm) +void mld_polyz_unpack_17_aarch64_asm(int32_t r[MLDSA_N], const uint8_t buf[576], + const uint8_t indices[64]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mldsa_polyz_unpack_17_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(buf, 576)) + requires(indices == mld_polyz_unpack_17_indices) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(array_bound(r, 0, MLDSA_N, -((1 << 17) - 1), (1 << 17) + 1)) +); +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \ + */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) +#define mld_polyz_unpack_19_aarch64_asm \ + MLD_NAMESPACE(polyz_unpack_19_aarch64_asm) +void mld_polyz_unpack_19_aarch64_asm(int32_t r[MLDSA_N], const uint8_t buf[640], + const uint8_t indices[64]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mldsa_polyz_unpack_19_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(buf, 640)) + requires(indices == mld_polyz_unpack_19_indices) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(array_bound(r, 0, MLDSA_N, -((1 << 19) - 1), (1 << 19) + 1)) +); +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87 */ +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \ + defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +#define mld_poly_pointwise_montgomery_aarch64_asm \ + MLD_NAMESPACE(poly_pointwise_montgomery_aarch64_asm) +void mld_poly_pointwise_montgomery_aarch64_asm(int32_t a[MLDSA_N], + const int32_t b[MLDSA_N]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mldsa_pointwise_montgomery_aarch64_asm.ml + */ +__contract__( + requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(b, sizeof(int32_t) * MLDSA_N)) + /* Input bound MLD_NTT_BOUND = 9 * MLD_FQMUL_BOUND, the guaranteed bound of + * any forward NTT implementation. Hardcoded here to keep this header free + * of poly.h. */ + /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */ + requires(array_abs_bound(a, 0, MLDSA_N, 94279698)) + requires(array_abs_bound(b, 0, MLDSA_N, 94279698)) + assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N)) + ensures(array_abs_bound(a, 0, MLDSA_N, 8380417)) +); +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API || \ + MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +#define mld_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm \ + MLD_NAMESPACE(polyvecl_pointwise_acc_montgomery_l4_aarch64_asm) +void mld_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm( + int32_t r[MLDSA_N], const int32_t a[4][MLDSA_N], + const int32_t b[4][MLDSA_N]) +/* This must be kept in sync with the HOL-Light specification + * in + * proofs/hol_light/aarch64/proofs/mldsa_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm.ml + */ +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a, sizeof(int32_t) * 4 * MLDSA_N)) + requires(memory_no_alias(b, sizeof(int32_t) * 4 * MLDSA_N)) + requires(forall(l0, 0, 4, array_abs_bound(a[l0], 0, MLDSA_N, 8380417))) + /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */ + requires(forall(l1, 0, 4, array_abs_bound(b[l1], 0, MLDSA_N, 94279698))) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(array_abs_bound(r, 0, MLDSA_N, 8380417)) +); + +#define mld_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm \ + MLD_NAMESPACE(polyvecl_pointwise_acc_montgomery_l5_aarch64_asm) +void mld_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm( + int32_t r[MLDSA_N], const int32_t a[5][MLDSA_N], + const int32_t b[5][MLDSA_N]) +/* This must be kept in sync with the HOL-Light specification + * in + * proofs/hol_light/aarch64/proofs/mldsa_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm.ml + */ +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a, sizeof(int32_t) * 5 * MLDSA_N)) + requires(memory_no_alias(b, sizeof(int32_t) * 5 * MLDSA_N)) + requires(forall(l0, 0, 5, array_abs_bound(a[l0], 0, MLDSA_N, 8380417))) + /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */ + requires(forall(l1, 0, 5, array_abs_bound(b[l1], 0, MLDSA_N, 94279698))) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(array_abs_bound(r, 0, MLDSA_N, 8380417)) +); + +#define mld_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm \ + MLD_NAMESPACE(polyvecl_pointwise_acc_montgomery_l7_aarch64_asm) +void mld_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm( + int32_t r[MLDSA_N], const int32_t a[7][MLDSA_N], + const int32_t b[7][MLDSA_N]) +/* This must be kept in sync with the HOL-Light specification + * in + * proofs/hol_light/aarch64/proofs/mldsa_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm.ml + */ +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a, sizeof(int32_t) * 7 * MLDSA_N)) + requires(memory_no_alias(b, sizeof(int32_t) * 7 * MLDSA_N)) + requires(forall(l0, 0, 7, array_abs_bound(a[l0], 0, MLDSA_N, 8380417))) + /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */ + requires(forall(l1, 0, 7, array_abs_bound(b[l1], 0, MLDSA_N, 94279698))) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(array_abs_bound(r, 0, MLDSA_N, 8380417)) +); + +#endif /* !MLD_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H */ diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_intt_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_intt_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_intt_aarch64_asm.S @@ -0,0 +1,786 @@ +/* Copyright (c) 2022 Arm Limited + * Copyright (c) 2022 Hanno Becker + * Copyright (c) 2023 Amin Abdulrahman, Matthias Kannwischer + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [NeonNTT] + * Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1 + * Becker, Hwang, Kannwischer, Yang, Yang + * https://eprint.iacr.org/2021/986 + * + * - [NeonNTT_Autoformalised] + * Neon NTT - (Auto)formalised + * Hanno Becker + * https://eprint.iacr.org/2026/1223 + * + * - [SLOTHY_Paper] + * Fast and Clean: Auditable high-performance assembly via constraint solving + * Abdulrahman, Becker, Kannwischer, Klein + * https://eprint.iacr.org/2022/1303 + */ + +/* AArch64 ML-DSA inverse NTT following @[NeonNTT], @[SLOTHY_Paper], and @[NeonNTT_Autoformalised] */ + +/*yaml + Name: intt_aarch64_asm + Description: AArch64 ML-DSA inverse NTT + Signature: void mld_intt_aarch64_asm(int32_t r[256], const int32_t zetas_l78[384], const int32_t zetas_l123456[160]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t r[256] + description: Input/output polynomial (256 x int32_t) + x1: + type: buffer + size_bytes: 1536 + permissions: read-only + c_parameter: const int32_t zetas_l78[384] + description: Twiddle factors for layers 7-8 (384 x int32_t) + x2: + type: buffer + size_bytes: 640 + permissions: read-only + c_parameter: const int32_t zetas_l123456[160] + description: Twiddle factors for layers 1-6 (160 x int32_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_intt_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(intt_aarch64_asm) +MLD_ASM_FN_SYMBOL(intt_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x40 + .cfi_adjust_cfa_offset 0x40 + stp d8, d9, [sp] + .cfi_rel_offset d8, 0x0 + .cfi_rel_offset d9, 0x8 + stp d10, d11, [sp, #0x10] + .cfi_rel_offset d10, 0x10 + .cfi_rel_offset d11, 0x18 + stp d12, d13, [sp, #0x20] + .cfi_rel_offset d12, 0x20 + .cfi_rel_offset d13, 0x28 + stp d14, d15, [sp, #0x30] + .cfi_rel_offset d14, 0x30 + .cfi_rel_offset d15, 0x38 + mov w5, #0xe001 // =57345 + movk w5, #0x7f, lsl #16 + dup v31.4s, w5 + mov x3, x0 + mov x4, #0x10 // =16 + ldr q27, [x3, #0x10] + ldr q18, [x3] + ldr q3, [x3, #0x20] + ldr q13, [x3, #0x30] + ldr q2, [x1, #0x30] + ldr q8, [x3, #0x70] + ldr q21, [x3, #0x60] + trn1 v10.4s, v18.4s, v27.4s + trn2 v23.4s, v18.4s, v27.4s + trn1 v6.4s, v3.4s, v13.4s + trn2 v18.4s, v3.4s, v13.4s + ldr q12, [x1, #0x50] + ldr q17, [x1, #0x40] + trn2 v29.2d, v10.2d, v6.2d + trn2 v14.2d, v23.2d, v18.2d + trn1 v10.2d, v10.2d, v6.2d + trn1 v26.2d, v23.2d, v18.2d + sub v3.4s, v29.4s, v14.4s + trn1 v13.4s, v21.4s, v8.4s + ldr q6, [x1, #0x10] + add v24.4s, v10.4s, v26.4s + sub v30.4s, v10.4s, v26.4s + sqrdmulh v12.4s, v3.4s, v12.4s + ldr q9, [x1, #0x20] + sqrdmulh v5.4s, v30.4s, v2.4s + mul v4.4s, v3.4s, v17.4s + ldr q3, [x3, #0x50] + add v10.4s, v29.4s, v14.4s + ldr q26, [x3, #0x40] + mul v15.4s, v30.4s, v9.4s + trn2 v30.4s, v21.4s, v8.4s + sub v21.4s, v24.4s, v10.4s + add v29.4s, v24.4s, v10.4s + mls v15.4s, v5.4s, v31.s[0] + ldr q16, [x1], #0x60 + trn2 v10.4s, v26.4s, v3.4s + mls v4.4s, v12.4s, v31.s[0] + trn1 v3.4s, v26.4s, v3.4s + ldr q0, [x1, #0x50] + trn2 v25.2d, v10.2d, v30.2d + sqrdmulh v12.4s, v21.4s, v6.4s + trn2 v1.2d, v3.2d, v13.2d + mul v21.4s, v21.4s, v16.4s + sub v23.4s, v1.4s, v25.4s + sub v2.4s, v15.4s, v4.4s + add v20.4s, v15.4s, v4.4s + sqrdmulh v4.4s, v23.4s, v0.4s + trn1 v7.2d, v3.2d, v13.2d + sqrdmulh v3.4s, v2.4s, v6.4s + trn1 v15.2d, v10.2d, v30.2d + mls v21.4s, v12.4s, v31.s[0] + ldr q12, [x1, #0x30] + sub v13.4s, v7.4s, v15.4s + mul v10.4s, v2.4s, v16.4s + ldr q16, [x1, #0x40] + ldr q5, [x1, #0x20] + mls v10.4s, v3.4s, v31.s[0] + trn2 v11.4s, v29.4s, v20.4s + sqrdmulh v2.4s, v13.4s, v12.4s + ldr d9, [x2], #0x20 + mul v17.4s, v13.4s, v5.4s + trn1 v24.4s, v29.4s, v20.4s + trn1 v12.4s, v21.4s, v10.4s + trn2 v10.4s, v21.4s, v10.4s + mul v22.4s, v23.4s, v16.4s + ldur q26, [x2, #-0x10] + trn1 v13.2d, v24.2d, v12.2d + trn1 v3.2d, v11.2d, v10.2d + mls v17.4s, v2.4s, v31.s[0] + mls v22.4s, v4.4s, v31.s[0] + sub v23.4s, v13.4s, v3.4s + trn2 v10.2d, v11.2d, v10.2d + sqrdmulh v0.4s, v23.4s, v26.s[1] + trn2 v21.2d, v24.2d, v12.2d + mul v29.4s, v23.4s, v26.s[0] + sub v23.4s, v21.4s, v10.4s + sqrdmulh v8.4s, v23.4s, v26.s[3] + add v6.4s, v1.4s, v25.4s + add v11.4s, v21.4s, v10.4s + mls v29.4s, v0.4s, v31.s[0] + add v30.4s, v13.4s, v3.4s + ldr q2, [x1, #0x10] + mul v3.4s, v23.4s, v26.s[2] + add v12.4s, v7.4s, v15.4s + mls v3.4s, v8.4s, v31.s[0] + sub v13.4s, v30.4s, v11.4s + sub v21.4s, v12.4s, v6.4s + add v26.4s, v17.4s, v22.4s + sqrdmulh v28.4s, v13.4s, v9.s[1] + mul v18.4s, v13.4s, v9.s[0] + sub v8.4s, v29.4s, v3.4s + sqrdmulh v24.4s, v21.4s, v2.4s + add v15.4s, v30.4s, v11.4s + add v10.4s, v29.4s, v3.4s + add v14.4s, v12.4s, v6.4s + sqrdmulh v16.4s, v8.4s, v9.s[1] + sub v3.4s, v17.4s, v22.4s + mul v8.4s, v8.4s, v9.s[0] + ldr q17, [x1], #0x60 + sub x4, x4, #0x2 + +Lmld_intt_layer5678_start: + ldr d4, [x2], #0x20 + mul v0.4s, v21.4s, v17.4s + ldr q7, [x3, #0xa0] + ldr q27, [x3, #0xb0] + mls v8.4s, v16.4s, v31.s[0] + ldr q29, [x3, #0x80] + trn1 v9.4s, v14.4s, v26.4s + ldr q25, [x3, #0x90] + mls v0.4s, v24.4s, v31.s[0] + str q15, [x3], #0x40 + trn1 v1.4s, v7.4s, v27.4s + ldr q15, [x1, #0x50] + trn2 v12.4s, v7.4s, v27.4s + sqrdmulh v16.4s, v3.4s, v2.4s + trn1 v21.4s, v29.4s, v25.4s + stur q8, [x3, #-0x10] + trn2 v19.4s, v29.4s, v25.4s + mls v18.4s, v28.4s, v31.s[0] + ldr q22, [x1, #0x30] + trn2 v13.2d, v21.2d, v1.2d + trn2 v5.2d, v19.2d, v12.2d + mul v23.4s, v3.4s, v17.4s + trn1 v17.2d, v21.2d, v1.2d + ldr q29, [x1, #0x40] + mls v23.4s, v16.4s, v31.s[0] + sub v30.4s, v13.4s, v5.4s + trn1 v24.2d, v19.2d, v12.2d + stur q10, [x3, #-0x30] + sqrdmulh v16.4s, v30.4s, v15.4s + ldr q10, [x1, #0x20] + trn2 v12.4s, v14.4s, v26.4s + stur q18, [x3, #-0x20] + mul v11.4s, v30.4s, v29.4s + sub v8.4s, v17.4s, v24.4s + trn1 v20.4s, v0.4s, v23.4s + ldr q2, [x1, #0x10] + trn2 v26.4s, v0.4s, v23.4s + sqrdmulh v19.4s, v8.4s, v22.4s + ldur q14, [x2, #-0x10] + trn1 v21.2d, v9.2d, v20.2d + mul v8.4s, v8.4s, v10.4s + trn1 v6.2d, v12.2d, v26.2d + trn2 v0.2d, v9.2d, v20.2d + mls v11.4s, v16.4s, v31.s[0] + sub v7.4s, v21.4s, v6.4s + trn2 v28.2d, v12.2d, v26.2d + add v10.4s, v21.4s, v6.4s + mul v18.4s, v7.4s, v14.s[0] + add v20.4s, v13.4s, v5.4s + sqrdmulh v30.4s, v7.4s, v14.s[1] + sub v21.4s, v0.4s, v28.4s + add v13.4s, v0.4s, v28.4s + add v9.4s, v17.4s, v24.4s + sqrdmulh v1.4s, v21.4s, v14.s[3] + ldr q17, [x1], #0x60 + mul v14.4s, v21.4s, v14.s[2] + sub v21.4s, v9.4s, v20.4s + mls v18.4s, v30.4s, v31.s[0] + mls v14.4s, v1.4s, v31.s[0] + mls v8.4s, v19.4s, v31.s[0] + sub v3.4s, v10.4s, v13.4s + add v15.4s, v10.4s, v13.4s + sqrdmulh v28.4s, v3.4s, v4.s[1] + add v10.4s, v18.4s, v14.4s + sub v19.4s, v18.4s, v14.4s + mul v18.4s, v3.4s, v4.s[0] + sub v3.4s, v8.4s, v11.4s + add v26.4s, v8.4s, v11.4s + sqrdmulh v16.4s, v19.4s, v4.s[1] + add v14.4s, v9.4s, v20.4s + mul v8.4s, v19.4s, v4.s[0] + sqrdmulh v24.4s, v21.4s, v2.4s + subs x4, x4, #0x1 + cbnz x4, Lmld_intt_layer5678_start + sqrdmulh v19.4s, v3.4s, v2.4s + trn2 v0.4s, v14.4s, v26.4s + str q15, [x3], #0x40 + trn1 v30.4s, v14.4s, v26.4s + mul v26.4s, v21.4s, v17.4s + stur q10, [x3, #-0x30] + ldr d29, [x2], #0x20 + mls v26.4s, v24.4s, v31.s[0] + mul v10.4s, v3.4s, v17.4s + ldur q14, [x2, #-0x10] + mls v10.4s, v19.4s, v31.s[0] + mls v8.4s, v16.4s, v31.s[0] + trn2 v22.4s, v26.4s, v10.4s + trn1 v25.4s, v26.4s, v10.4s + trn1 v1.2d, v30.2d, v25.2d + trn1 v2.2d, v0.2d, v22.2d + trn2 v13.2d, v30.2d, v25.2d + trn2 v7.2d, v0.2d, v22.2d + mls v18.4s, v28.4s, v31.s[0] + sub v22.4s, v1.4s, v2.4s + add v3.4s, v13.4s, v7.4s + sqrdmulh v27.4s, v22.4s, v14.s[1] + sub v4.4s, v13.4s, v7.4s + stur q8, [x3, #-0x10] + sqrdmulh v0.4s, v4.4s, v14.s[3] + add v16.4s, v1.4s, v2.4s + stur q18, [x3, #-0x20] + mul v23.4s, v4.4s, v14.s[2] + add v4.4s, v16.4s, v3.4s + sub v17.4s, v16.4s, v3.4s + mul v14.4s, v22.4s, v14.s[0] + str q4, [x3], #0x40 + mls v23.4s, v0.4s, v31.s[0] + mls v14.4s, v27.4s, v31.s[0] + mul v11.4s, v17.4s, v29.s[0] + sqrdmulh v10.4s, v17.4s, v29.s[1] + sub v0.4s, v14.4s, v23.4s + add v6.4s, v14.4s, v23.4s + sqrdmulh v12.4s, v0.4s, v29.s[1] + mul v0.4s, v0.4s, v29.s[0] + stur q6, [x3, #-0x30] + mls v11.4s, v10.4s, v31.s[0] + mls v0.4s, v12.4s, v31.s[0] + stur q11, [x3, #-0x20] + stur q0, [x3, #-0x10] + mov w5, #0x3ffe // =16382 + dup v29.4s, w5 + mov w5, #0xe03 // =3587 + movk w5, #0x40, lsl #16 + dup v30.4s, w5 + mov x4, #0x4 // =4 + ldr q0, [x2], #0x80 + ldur q1, [x2, #-0x70] + ldur q2, [x2, #-0x60] + ldur q3, [x2, #-0x50] + ldur q4, [x2, #-0x40] + ldur q5, [x2, #-0x30] + ldur q6, [x2, #-0x20] + ldur q7, [x2, #-0x10] + ldr q8, [x0, #0xc0] + ldr q27, [x0, #0x80] + ldr q20, [x0, #0x1c0] + ldr q23, [x0, #0x180] + ldr q24, [x0, #0x3c0] + ldr q28, [x0, #0x40] + ldr q25, [x0, #0x340] + ldr q10, [x0, #0x380] + sub v15.4s, v27.4s, v8.4s + ldr q26, [x0] + ldr q18, [x0, #0x300] + add v9.4s, v23.4s, v20.4s + mul v11.4s, v15.4s, v4.s[0] + sub v19.4s, v23.4s, v20.4s + sub v22.4s, v10.4s, v24.4s + ldr q13, [x0, #0x240] + sqrdmulh v12.4s, v15.4s, v4.s[1] + sub v14.4s, v26.4s, v28.4s + ldr q17, [x0, #0x200] + add v23.4s, v18.4s, v25.4s + sqrdmulh v15.4s, v14.4s, v3.s[3] + sub v18.4s, v18.4s, v25.4s + mul v25.4s, v14.4s, v3.s[2] + sub v16.4s, v17.4s, v13.4s + mls v11.4s, v12.4s, v31.s[0] + add v21.4s, v10.4s, v24.4s + mls v25.4s, v15.4s, v31.s[0] + mul v12.4s, v16.4s, v5.s[2] + sub v10.4s, v23.4s, v21.4s + sqrdmulh v14.4s, v10.4s, v3.s[1] + add v23.4s, v23.4s, v21.4s + add v27.4s, v27.4s, v8.4s + mul v15.4s, v10.4s, v3.s[0] + add v24.4s, v25.4s, v11.4s + sub v10.4s, v25.4s, v11.4s + ldr q25, [x0, #0x100] + sqrdmulh v8.4s, v18.4s, v6.s[3] + add v21.4s, v26.4s, v28.4s + ldr q26, [x0, #0x140] + sqrdmulh v20.4s, v16.4s, v5.s[3] + sqrdmulh v28.4s, v22.4s, v7.s[1] + add v11.4s, v25.4s, v26.4s + mls v15.4s, v14.4s, v31.s[0] + add v17.4s, v17.4s, v13.4s + mul v13.4s, v22.4s, v7.s[0] + ldr q14, [x0, #0x280] + ldr q22, [x0, #0x2c0] + mls v13.4s, v28.4s, v31.s[0] + sub v16.4s, v25.4s, v26.4s + sub v28.4s, v21.4s, v27.4s + mls v12.4s, v20.4s, v31.s[0] + add v20.4s, v11.4s, v9.4s + add v26.4s, v14.4s, v22.4s + sub v14.4s, v14.4s, v22.4s + sqrdmulh v22.4s, v28.4s, v1.s[3] + sub v9.4s, v11.4s, v9.4s + mul v25.4s, v14.4s, v6.s[0] + sub v11.4s, v17.4s, v26.4s + add v17.4s, v17.4s, v26.4s + add v26.4s, v21.4s, v27.4s + sqrdmulh v27.4s, v11.4s, v2.s[3] + mul v21.4s, v11.4s, v2.s[2] + mul v11.4s, v18.4s, v6.s[2] + mls v11.4s, v8.4s, v31.s[0] + sqrdmulh v8.4s, v14.4s, v6.s[1] + add v18.4s, v26.4s, v20.4s + mls v21.4s, v27.4s, v31.s[0] + sub v27.4s, v17.4s, v23.4s + mul v14.4s, v28.4s, v1.s[2] + sub v28.4s, v26.4s, v20.4s + add v23.4s, v17.4s, v23.4s + mls v25.4s, v8.4s, v31.s[0] + add v26.4s, v21.4s, v15.4s + sub v15.4s, v21.4s, v15.4s + mul v8.4s, v19.4s, v5.s[0] + mls v14.4s, v22.4s, v31.s[0] + sub v22.4s, v11.4s, v13.4s + sub v20.4s, v12.4s, v25.4s + add v21.4s, v12.4s, v25.4s + sqrdmulh v12.4s, v22.4s, v3.s[1] + mul v17.4s, v16.4s, v4.s[2] + sqrdmulh v25.4s, v16.4s, v4.s[3] + add v16.4s, v11.4s, v13.4s + mul v11.4s, v22.4s, v3.s[0] + sqrdmulh v19.4s, v19.4s, v5.s[1] + sqrdmulh v22.4s, v15.4s, v1.s[1] + sqrdmulh v13.4s, v20.4s, v2.s[3] + mul v20.4s, v20.4s, v2.s[2] + mls v11.4s, v12.4s, v31.s[0] + mls v20.4s, v13.4s, v31.s[0] + sub v13.4s, v18.4s, v23.4s + add v23.4s, v18.4s, v23.4s + mls v17.4s, v25.4s, v31.s[0] + mls v8.4s, v19.4s, v31.s[0] + sub v18.4s, v20.4s, v11.4s + add v20.4s, v20.4s, v11.4s + sqrdmulh v12.4s, v9.4s, v2.s[1] + mul v15.4s, v15.4s, v1.s[0] + sub v11.4s, v17.4s, v8.4s + mls v15.4s, v22.4s, v31.s[0] + add v19.4s, v17.4s, v8.4s + sqrdmulh v25.4s, v11.4s, v2.s[1] + add v17.4s, v24.4s, v19.4s + sub v24.4s, v24.4s, v19.4s + mul v19.4s, v11.4s, v2.s[0] + sqrdmulh v8.4s, v24.4s, v0.s[3] + mls v19.4s, v25.4s, v31.s[0] + mul v25.4s, v9.4s, v2.s[0] + sub x4, x4, #0x1 + +Lmld_intt_layer1234_start: + sub v22.4s, v21.4s, v16.4s + mls v25.4s, v12.4s, v31.s[0] + add v12.4s, v21.4s, v16.4s + mul v9.4s, v24.4s, v0.s[2] + mls v9.4s, v8.4s, v31.s[0] + sub v16.4s, v14.4s, v25.4s + add v14.4s, v14.4s, v25.4s + sqrdmulh v24.4s, v22.4s, v1.s[1] + sub v11.4s, v14.4s, v26.4s + sqrdmulh v21.4s, v16.4s, v0.s[3] + add v25.4s, v14.4s, v26.4s + mul v14.4s, v22.4s, v1.s[0] + mls v14.4s, v24.4s, v31.s[0] + mul v16.4s, v16.4s, v0.s[2] + mls v16.4s, v21.4s, v31.s[0] + add v24.4s, v9.4s, v14.4s + sqrdmulh v21.4s, v25.4s, v30.4s + sqrdmulh v26.4s, v11.4s, v0.s[1] + mul v8.4s, v25.4s, v29.4s + mls v8.4s, v21.4s, v31.s[0] + mul v25.4s, v24.4s, v29.4s + mul v22.4s, v11.4s, v0.s[0] + sub v11.4s, v17.4s, v12.4s + str q8, [x0, #0x80] + sub v8.4s, v9.4s, v14.4s + add v14.4s, v16.4s, v15.4s + sqrdmulh v9.4s, v11.4s, v0.s[1] + sub v21.4s, v16.4s, v15.4s + mls v22.4s, v26.4s, v31.s[0] + mul v15.4s, v11.4s, v0.s[0] + mls v15.4s, v9.4s, v31.s[0] + add v11.4s, v17.4s, v12.4s + mul v16.4s, v10.4s, v1.s[2] + str q22, [x0, #0x280] + sqrdmulh v26.4s, v10.4s, v1.s[3] + str q15, [x0, #0x240] + sqrdmulh v9.4s, v8.4s, v0.s[1] + mul v12.4s, v8.4s, v0.s[0] + mls v16.4s, v26.4s, v31.s[0] + sqrdmulh v10.4s, v27.4s, v1.s[1] + sqrdmulh v15.4s, v14.4s, v30.4s + add v22.4s, v16.4s, v19.4s + sub v19.4s, v16.4s, v19.4s + mls v12.4s, v9.4s, v31.s[0] + add v8.4s, v22.4s, v20.4s + sub v26.4s, v22.4s, v20.4s + mul v27.4s, v27.4s, v1.s[0] + sqrdmulh v17.4s, v11.4s, v30.4s + mls v27.4s, v10.4s, v31.s[0] + mul v20.4s, v11.4s, v29.4s + sqrdmulh v22.4s, v24.4s, v30.4s + mls v20.4s, v17.4s, v31.s[0] + mul v14.4s, v14.4s, v29.4s + sqrdmulh v16.4s, v23.4s, v30.4s + str q20, [x0, #0x40] + sqrdmulh v11.4s, v21.4s, v0.s[1] + sqrdmulh v20.4s, v28.4s, v0.s[3] + mul v17.4s, v23.4s, v29.4s + mul v23.4s, v28.4s, v0.s[2] + mls v17.4s, v16.4s, v31.s[0] + mul v10.4s, v21.4s, v0.s[0] + str q12, [x0, #0x340] + mls v25.4s, v22.4s, v31.s[0] + str q17, [x0], #0x10 + ldr q17, [x0, #0x2c0] + mls v14.4s, v15.4s, v31.s[0] + ldr q12, [x0, #0x200] + sqrdmulh v16.4s, v13.4s, v0.s[1] + str q25, [x0, #0x130] + mul v28.4s, v13.4s, v0.s[0] + ldr q13, [x0, #0x240] + str q14, [x0, #0x170] + mul v14.4s, v18.4s, v1.s[0] + mls v28.4s, v16.4s, v31.s[0] + ldr q21, [x0, #0xc0] + mls v10.4s, v11.4s, v31.s[0] + mls v23.4s, v20.4s, v31.s[0] + str q28, [x0, #0x1f0] + ldr q11, [x0, #0x300] + sqrdmulh v28.4s, v18.4s, v1.s[1] + str q10, [x0, #0x370] + mul v16.4s, v19.4s, v0.s[2] + sqrdmulh v19.4s, v19.4s, v0.s[3] + sqrdmulh v10.4s, v26.4s, v0.s[1] + add v18.4s, v12.4s, v13.4s + sub v12.4s, v12.4s, v13.4s + mls v16.4s, v19.4s, v31.s[0] + ldr q9, [x0, #0x280] + ldr q20, [x0, #0x380] + add v24.4s, v23.4s, v27.4s + sub v25.4s, v23.4s, v27.4s + mul v19.4s, v26.4s, v0.s[0] + mls v14.4s, v28.4s, v31.s[0] + add v13.4s, v9.4s, v17.4s + sub v28.4s, v18.4s, v13.4s + mls v19.4s, v10.4s, v31.s[0] + sqrdmulh v23.4s, v25.4s, v0.s[1] + mul v15.4s, v25.4s, v0.s[0] + str q19, [x0, #0x2b0] + ldr q19, [x0, #0x80] + sqrdmulh v22.4s, v8.4s, v30.4s + mls v15.4s, v23.4s, v31.s[0] + add v26.4s, v16.4s, v14.4s + mul v10.4s, v8.4s, v29.4s + add v8.4s, v18.4s, v13.4s + sqrdmulh v13.4s, v26.4s, v30.4s + str q15, [x0, #0x2f0] + sub v18.4s, v9.4s, v17.4s + mul v27.4s, v26.4s, v29.4s + sqrdmulh v23.4s, v12.4s, v5.s[3] + mls v27.4s, v13.4s, v31.s[0] + mul v25.4s, v18.4s, v6.s[0] + sqrdmulh v13.4s, v18.4s, v6.s[1] + str q27, [x0, #0x1b0] + sub v27.4s, v19.4s, v21.4s + add v21.4s, v19.4s, v21.4s + mul v18.4s, v12.4s, v5.s[2] + sub v9.4s, v16.4s, v14.4s + mul v26.4s, v24.4s, v29.4s + ldr q12, [x0, #0x3c0] + mls v25.4s, v13.4s, v31.s[0] + sub v16.4s, v20.4s, v12.4s + sqrdmulh v17.4s, v24.4s, v30.4s + mul v13.4s, v16.4s, v7.s[0] + mls v18.4s, v23.4s, v31.s[0] + mls v26.4s, v17.4s, v31.s[0] + sqrdmulh v17.4s, v9.4s, v0.s[1] + sqrdmulh v24.4s, v16.4s, v7.s[1] + ldr q16, [x0, #0x340] + str q26, [x0, #0xf0] + sub v15.4s, v18.4s, v25.4s + mul v23.4s, v9.4s, v0.s[0] + ldr q19, [x0, #0x180] + ldr q14, [x0, #0x1c0] + sqrdmulh v9.4s, v15.4s, v2.s[3] + add v20.4s, v20.4s, v12.4s + add v12.4s, v11.4s, v16.4s + mul v26.4s, v15.4s, v2.s[2] + sub v16.4s, v11.4s, v16.4s + sub v11.4s, v12.4s, v20.4s + mls v23.4s, v17.4s, v31.s[0] + add v20.4s, v12.4s, v20.4s + sub v12.4s, v19.4s, v14.4s + mul v17.4s, v27.4s, v4.s[0] + mls v10.4s, v22.4s, v31.s[0] + add v22.4s, v19.4s, v14.4s + ldr q14, [x0, #0x40] + str q23, [x0, #0x3b0] + sqrdmulh v23.4s, v27.4s, v4.s[1] + ldr q19, [x0, #0x100] + ldr q15, [x0, #0x140] + mls v26.4s, v9.4s, v31.s[0] + str q10, [x0, #0xb0] + sqrdmulh v10.4s, v11.4s, v3.s[1] + sub v27.4s, v8.4s, v20.4s + mls v13.4s, v24.4s, v31.s[0] + add v8.4s, v8.4s, v20.4s + sub v20.4s, v19.4s, v15.4s + sqrdmulh v9.4s, v16.4s, v6.s[3] + add v19.4s, v19.4s, v15.4s + ldr q15, [x0] + mul v24.4s, v20.4s, v4.s[2] + mls v17.4s, v23.4s, v31.s[0] + add v23.4s, v15.4s, v14.4s + sub v14.4s, v15.4s, v14.4s + mul v15.4s, v11.4s, v3.s[0] + sub v11.4s, v23.4s, v21.4s + mul v16.4s, v16.4s, v6.s[2] + add v23.4s, v23.4s, v21.4s + add v21.4s, v18.4s, v25.4s + sqrdmulh v25.4s, v20.4s, v4.s[3] + sqrdmulh v18.4s, v12.4s, v5.s[1] + sqrdmulh v20.4s, v14.4s, v3.s[3] + mul v12.4s, v12.4s, v5.s[0] + mls v16.4s, v9.4s, v31.s[0] + mls v12.4s, v18.4s, v31.s[0] + mls v24.4s, v25.4s, v31.s[0] + sub v25.4s, v16.4s, v13.4s + add v16.4s, v16.4s, v13.4s + mul v13.4s, v28.4s, v2.s[2] + sqrdmulh v9.4s, v25.4s, v3.s[1] + mul v18.4s, v25.4s, v3.s[0] + sqrdmulh v25.4s, v28.4s, v2.s[3] + mls v18.4s, v9.4s, v31.s[0] + mls v15.4s, v10.4s, v31.s[0] + sub v9.4s, v24.4s, v12.4s + mul v28.4s, v14.4s, v3.s[2] + add v12.4s, v24.4s, v12.4s + mls v28.4s, v20.4s, v31.s[0] + add v20.4s, v26.4s, v18.4s + add v24.4s, v19.4s, v22.4s + mls v13.4s, v25.4s, v31.s[0] + sub v18.4s, v26.4s, v18.4s + sqrdmulh v26.4s, v11.4s, v1.s[3] + add v25.4s, v23.4s, v24.4s + mul v14.4s, v11.4s, v1.s[2] + sub v10.4s, v28.4s, v17.4s + add v17.4s, v28.4s, v17.4s + sub v28.4s, v23.4s, v24.4s + sqrdmulh v11.4s, v9.4s, v2.s[1] + sub v22.4s, v19.4s, v22.4s + sub v24.4s, v17.4s, v12.4s + mls v14.4s, v26.4s, v31.s[0] + add v17.4s, v17.4s, v12.4s + sqrdmulh v12.4s, v22.4s, v2.s[1] + sub v23.4s, v13.4s, v15.4s + add v26.4s, v13.4s, v15.4s + mul v19.4s, v9.4s, v2.s[0] + sqrdmulh v9.4s, v23.4s, v1.s[1] + mul v15.4s, v23.4s, v1.s[0] + mls v19.4s, v11.4s, v31.s[0] + sub v13.4s, v25.4s, v8.4s + mls v15.4s, v9.4s, v31.s[0] + add v23.4s, v25.4s, v8.4s + sqrdmulh v8.4s, v24.4s, v0.s[3] + mul v25.4s, v22.4s, v2.s[0] + subs x4, x4, #0x1 + cbnz x4, Lmld_intt_layer1234_start + mul v22.4s, v24.4s, v0.s[2] + sqrdmulh v9.4s, v23.4s, v30.4s + mls v25.4s, v12.4s, v31.s[0] + mul v23.4s, v23.4s, v29.4s + mls v23.4s, v9.4s, v31.s[0] + mls v22.4s, v8.4s, v31.s[0] + sqrdmulh v11.4s, v10.4s, v1.s[3] + sub v9.4s, v14.4s, v25.4s + str q23, [x0], #0x10 + mul v23.4s, v10.4s, v1.s[2] + sqrdmulh v24.4s, v9.4s, v0.s[3] + mls v23.4s, v11.4s, v31.s[0] + mul v9.4s, v9.4s, v0.s[2] + mls v9.4s, v24.4s, v31.s[0] + sub v24.4s, v23.4s, v19.4s + add v19.4s, v23.4s, v19.4s + mul v8.4s, v28.4s, v0.s[2] + add v23.4s, v14.4s, v25.4s + mul v11.4s, v13.4s, v0.s[0] + sub v25.4s, v21.4s, v16.4s + sub v10.4s, v9.4s, v15.4s + add v15.4s, v9.4s, v15.4s + add v16.4s, v21.4s, v16.4s + sqrdmulh v28.4s, v28.4s, v0.s[3] + add v9.4s, v23.4s, v26.4s + add v14.4s, v17.4s, v16.4s + sqrdmulh v21.4s, v13.4s, v0.s[1] + sub v13.4s, v17.4s, v16.4s + mul v16.4s, v14.4s, v29.4s + sub v12.4s, v23.4s, v26.4s + mul v17.4s, v13.4s, v0.s[0] + mls v8.4s, v28.4s, v31.s[0] + sqrdmulh v28.4s, v14.4s, v30.4s + sub v14.4s, v19.4s, v20.4s + add v20.4s, v19.4s, v20.4s + sqrdmulh v19.4s, v13.4s, v0.s[1] + sqrdmulh v26.4s, v27.4s, v1.s[1] + mls v16.4s, v28.4s, v31.s[0] + sqrdmulh v28.4s, v12.4s, v0.s[1] + mul v27.4s, v27.4s, v1.s[0] + str q16, [x0, #0x30] + mul v12.4s, v12.4s, v0.s[0] + mls v12.4s, v28.4s, v31.s[0] + mls v27.4s, v26.4s, v31.s[0] + sqrdmulh v26.4s, v25.4s, v1.s[1] + str q12, [x0, #0x270] + sqrdmulh v12.4s, v14.4s, v0.s[1] + sub v13.4s, v8.4s, v27.4s + add v27.4s, v8.4s, v27.4s + mul v16.4s, v14.4s, v0.s[0] + mul v8.4s, v20.4s, v29.4s + mls v16.4s, v12.4s, v31.s[0] + mul v12.4s, v18.4s, v1.s[0] + sqrdmulh v14.4s, v20.4s, v30.4s + str q16, [x0, #0x2b0] + sqrdmulh v20.4s, v18.4s, v1.s[1] + mul v25.4s, v25.4s, v1.s[0] + mls v25.4s, v26.4s, v31.s[0] + mls v17.4s, v19.4s, v31.s[0] + mul v18.4s, v9.4s, v29.4s + add v28.4s, v22.4s, v25.4s + sqrdmulh v9.4s, v9.4s, v30.4s + sub v23.4s, v22.4s, v25.4s + str q17, [x0, #0x230] + mls v11.4s, v21.4s, v31.s[0] + mls v8.4s, v14.4s, v31.s[0] + sqrdmulh v21.4s, v28.4s, v30.4s + str q11, [x0, #0x1f0] + mul v17.4s, v28.4s, v29.4s + str q8, [x0, #0xb0] + mul v28.4s, v10.4s, v0.s[0] + mls v17.4s, v21.4s, v31.s[0] + sqrdmulh v21.4s, v10.4s, v0.s[1] + mul v8.4s, v15.4s, v29.4s + str q17, [x0, #0x130] + sqrdmulh v15.4s, v15.4s, v30.4s + mul v10.4s, v24.4s, v0.s[2] + sqrdmulh v24.4s, v24.4s, v0.s[3] + mls v12.4s, v20.4s, v31.s[0] + mls v8.4s, v15.4s, v31.s[0] + mls v10.4s, v24.4s, v31.s[0] + sqrdmulh v24.4s, v27.4s, v30.4s + str q8, [x0, #0x170] + mul v14.4s, v13.4s, v0.s[0] + sub v16.4s, v10.4s, v12.4s + sqrdmulh v17.4s, v13.4s, v0.s[1] + add v10.4s, v10.4s, v12.4s + mul v13.4s, v27.4s, v29.4s + sqrdmulh v20.4s, v10.4s, v30.4s + mls v14.4s, v17.4s, v31.s[0] + mls v13.4s, v24.4s, v31.s[0] + sqrdmulh v24.4s, v23.4s, v0.s[1] + str q14, [x0, #0x2f0] + sqrdmulh v8.4s, v16.4s, v0.s[1] + str q13, [x0, #0xf0] + mls v18.4s, v9.4s, v31.s[0] + mul v9.4s, v23.4s, v0.s[0] + mul v14.4s, v16.4s, v0.s[0] + str q18, [x0, #0x70] + mul v27.4s, v10.4s, v29.4s + mls v28.4s, v21.4s, v31.s[0] + mls v14.4s, v8.4s, v31.s[0] + mls v27.4s, v20.4s, v31.s[0] + str q28, [x0, #0x370] + mls v9.4s, v24.4s, v31.s[0] + str q14, [x0, #0x3b0] + str q27, [x0, #0x1b0] + str q9, [x0, #0x330] + ldp d8, d9, [sp] + .cfi_restore d8 + .cfi_restore d9 + ldp d10, d11, [sp, #0x10] + .cfi_restore d10 + .cfi_restore d11 + ldp d12, d13, [sp, #0x20] + .cfi_restore d12 + .cfi_restore d13 + ldp d14, d15, [sp, #0x30] + .cfi_restore d14 + .cfi_restore d15 + add sp, sp, #0x40 + .cfi_adjust_cfa_offset -0x40 + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(intt_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_ntt_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_ntt_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_ntt_aarch64_asm.S @@ -0,0 +1,686 @@ +/* Copyright (c) 2022 Arm Limited + * Copyright (c) 2022 Hanno Becker + * Copyright (c) 2023 Amin Abdulrahman, Matthias Kannwischer + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [NeonNTT] + * Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1 + * Becker, Hwang, Kannwischer, Yang, Yang + * https://eprint.iacr.org/2021/986 + * + * - [NeonNTT_Autoformalised] + * Neon NTT - (Auto)formalised + * Hanno Becker + * https://eprint.iacr.org/2026/1223 + * + * - [SLOTHY_Paper] + * Fast and Clean: Auditable high-performance assembly via constraint solving + * Abdulrahman, Becker, Kannwischer, Klein + * https://eprint.iacr.org/2022/1303 + */ + +/* AArch64 ML-DSA forward NTT following @[NeonNTT], @[SLOTHY_Paper], and @[NeonNTT_Autoformalised] */ + +/*yaml + Name: ntt_aarch64_asm + Description: AArch64 ML-DSA forward NTT + Signature: void mld_ntt_aarch64_asm(int32_t r[256], const int32_t zetas_l123456[144], const int32_t zetas_l78[384]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t r[256] + description: Input/output polynomial (256 x int32_t) + x1: + type: buffer + size_bytes: 576 + permissions: read-only + c_parameter: const int32_t zetas_l123456[144] + description: Twiddle factors for layers 1-6 (144 x int32_t) + x2: + type: buffer + size_bytes: 1536 + permissions: read-only + c_parameter: const int32_t zetas_l78[384] + description: Twiddle factors for layers 7-8 (384 x int32_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_ntt_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(ntt_aarch64_asm) +MLD_ASM_FN_SYMBOL(ntt_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x40 + .cfi_adjust_cfa_offset 0x40 + stp d8, d9, [sp] + .cfi_rel_offset d8, 0x0 + .cfi_rel_offset d9, 0x8 + stp d10, d11, [sp, #0x10] + .cfi_rel_offset d10, 0x10 + .cfi_rel_offset d11, 0x18 + stp d12, d13, [sp, #0x20] + .cfi_rel_offset d12, 0x20 + .cfi_rel_offset d13, 0x28 + stp d14, d15, [sp, #0x30] + .cfi_rel_offset d14, 0x30 + .cfi_rel_offset d15, 0x38 + mov w5, #0xe001 // =57345 + movk w5, #0x7f, lsl #16 + dup v7.4s, w5 + mov x3, x0 + mov x4, #0x8 // =8 + ldr q0, [x1], #0x40 + ldur q1, [x1, #-0x30] + ldur q2, [x1, #-0x20] + ldur q3, [x1, #-0x10] + ldr q23, [x0, #0x390] + ldr q13, [x0, #0x380] + ldr q22, [x0, #0x80] + ldr q26, [x0, #0x190] + ldr q8, [x0, #0x280] + ldr q6, [x0, #0x210] + mul v10.4s, v13.4s, v0.s[0] + sqrdmulh v13.4s, v13.4s, v0.s[1] + mul v12.4s, v8.4s, v0.s[0] + sqrdmulh v27.4s, v8.4s, v0.s[1] + mul v4.4s, v6.4s, v0.s[0] + mls v10.4s, v13.4s, v7.s[0] + ldr q13, [x0, #0x180] + sqrdmulh v14.4s, v23.4s, v0.s[1] + mls v12.4s, v27.4s, v7.s[0] + add v31.4s, v13.4s, v10.4s + sub v13.4s, v13.4s, v10.4s + mul v10.4s, v23.4s, v0.s[0] + sqrdmulh v8.4s, v13.4s, v1.s[1] + sub v18.4s, v22.4s, v12.4s + mls v10.4s, v14.4s, v7.s[0] + mul v13.4s, v13.4s, v1.s[0] + mls v13.4s, v8.4s, v7.s[0] + sub v29.4s, v26.4s, v10.4s + add v25.4s, v26.4s, v10.4s + mul v10.4s, v31.4s, v0.s[2] + mul v14.4s, v25.4s, v0.s[2] + add v17.4s, v18.4s, v13.4s + sub v15.4s, v18.4s, v13.4s + sqrdmulh v13.4s, v31.4s, v0.s[3] + sqrdmulh v20.4s, v15.4s, v3.s[1] + sqrdmulh v5.4s, v17.4s, v2.s[3] + mls v10.4s, v13.4s, v7.s[0] + ldr q13, [x0, #0x300] + mul v18.4s, v17.4s, v2.s[2] + add v31.4s, v22.4s, v12.4s + mul v23.4s, v15.4s, v3.s[0] + ldr q17, [x0, #0x90] + add v19.4s, v31.4s, v10.4s + sub v16.4s, v31.4s, v10.4s + mul v10.4s, v13.4s, v0.s[0] + sqrdmulh v13.4s, v13.4s, v0.s[1] + sqrdmulh v27.4s, v16.4s, v2.s[1] + mul v11.4s, v16.4s, v2.s[0] + mls v10.4s, v13.4s, v7.s[0] + ldr q13, [x0, #0x290] + ldr q22, [x0, #0x100] + mls v11.4s, v27.4s, v7.s[0] + sqrdmulh v15.4s, v13.4s, v0.s[1] + sub v12.4s, v22.4s, v10.4s + add v30.4s, v22.4s, v10.4s + mul v10.4s, v13.4s, v0.s[0] + ldr q28, [x0] + sqrdmulh v13.4s, v25.4s, v0.s[3] + sqrdmulh v27.4s, v30.4s, v0.s[3] + mls v10.4s, v15.4s, v7.s[0] + mls v14.4s, v13.4s, v7.s[0] + ldr q13, [x0, #0x200] + sqrdmulh v25.4s, v12.4s, v1.s[1] + add v24.4s, v17.4s, v10.4s + sub v21.4s, v17.4s, v10.4s + sqrdmulh v8.4s, v13.4s, v0.s[1] + sub v9.4s, v24.4s, v14.4s + mul v26.4s, v12.4s, v1.s[0] + mul v13.4s, v13.4s, v0.s[0] + mls v13.4s, v8.4s, v7.s[0] + mul v8.4s, v30.4s, v0.s[2] + mls v8.4s, v27.4s, v7.s[0] + add v16.4s, v28.4s, v13.4s + sub v10.4s, v28.4s, v13.4s + mls v26.4s, v25.4s, v7.s[0] + sqrdmulh v12.4s, v19.4s, v1.s[3] + sub v25.4s, v16.4s, v8.4s + mls v23.4s, v20.4s, v7.s[0] + sub v22.4s, v25.4s, v11.4s + sqrdmulh v20.4s, v9.4s, v2.s[1] + sub v15.4s, v10.4s, v26.4s + sub x4, x4, #0x2 + +Lmld_ntt_layer123_start: + add v31.4s, v10.4s, v26.4s + mul v17.4s, v19.4s, v1.s[2] + add v26.4s, v15.4s, v23.4s + ldr q30, [x0, #0x2a0] + sub v13.4s, v15.4s, v23.4s + mul v23.4s, v29.4s, v1.s[0] + add v25.4s, v25.4s, v11.4s + str q22, [x0, #0x180] + mul v11.4s, v9.4s, v2.s[0] + str q13, [x0, #0x380] + ldr q28, [x0, #0x10] + add v10.4s, v16.4s, v8.4s + mls v17.4s, v12.4s, v7.s[0] + ldr q13, [x0, #0x3a0] + str q26, [x0, #0x300] + sqrdmulh v27.4s, v30.4s, v0.s[1] + mls v18.4s, v5.4s, v7.s[0] + ldr q9, [x0, #0x1a0] + sub v16.4s, v10.4s, v17.4s + add v15.4s, v10.4s, v17.4s + sqrdmulh v10.4s, v6.4s, v0.s[1] + str q16, [x0, #0x80] + str q15, [x0], #0x10 + sqrdmulh v19.4s, v13.4s, v0.s[1] + sub v15.4s, v31.4s, v18.4s + mul v8.4s, v13.4s, v0.s[0] + add v26.4s, v31.4s, v18.4s + str q15, [x0, #0x270] + sqrdmulh v13.4s, v29.4s, v1.s[1] + str q26, [x0, #0x1f0] + mls v8.4s, v19.4s, v7.s[0] + mls v11.4s, v20.4s, v7.s[0] + mls v23.4s, v13.4s, v7.s[0] + add v22.4s, v9.4s, v8.4s + ldr q6, [x0, #0x210] + sub v29.4s, v9.4s, v8.4s + mul v17.4s, v30.4s, v0.s[0] + ldr q9, [x0, #0x300] + sqrdmulh v13.4s, v22.4s, v0.s[3] + add v18.4s, v21.4s, v23.4s + mls v4.4s, v10.4s, v7.s[0] + sub v31.4s, v21.4s, v23.4s + sqrdmulh v16.4s, v31.4s, v3.s[1] + add v19.4s, v24.4s, v14.4s + mul v14.4s, v22.4s, v0.s[2] + sub v10.4s, v28.4s, v4.4s + mls v14.4s, v13.4s, v7.s[0] + ldr q13, [x0, #0x100] + sqrdmulh v22.4s, v9.4s, v0.s[1] + mul v8.4s, v9.4s, v0.s[0] + mul v23.4s, v31.4s, v3.s[0] + mls v8.4s, v22.4s, v7.s[0] + mls v23.4s, v16.4s, v7.s[0] + add v16.4s, v28.4s, v4.4s + ldr q22, [x0, #0x90] + mul v4.4s, v6.4s, v0.s[0] + mls v17.4s, v27.4s, v7.s[0] + add v21.4s, v13.4s, v8.4s + sub v27.4s, v13.4s, v8.4s + sqrdmulh v31.4s, v21.4s, v0.s[3] + str q25, [x0, #0xf0] + mul v8.4s, v21.4s, v0.s[2] + add v24.4s, v22.4s, v17.4s + sub v21.4s, v22.4s, v17.4s + sqrdmulh v5.4s, v18.4s, v2.s[3] + mls v8.4s, v31.4s, v7.s[0] + sub v9.4s, v24.4s, v14.4s + sqrdmulh v20.4s, v27.4s, v1.s[1] + mul v26.4s, v27.4s, v1.s[0] + sub v25.4s, v16.4s, v8.4s + mul v18.4s, v18.4s, v2.s[2] + sub v22.4s, v25.4s, v11.4s + mls v26.4s, v20.4s, v7.s[0] + sqrdmulh v20.4s, v9.4s, v2.s[1] + sqrdmulh v12.4s, v19.4s, v1.s[3] + sub v15.4s, v10.4s, v26.4s + subs x4, x4, #0x1 + cbnz x4, Lmld_ntt_layer123_start + add v13.4s, v10.4s, v26.4s + mls v18.4s, v5.4s, v7.s[0] + str q22, [x0, #0x180] + add v27.4s, v16.4s, v8.4s + mul v22.4s, v19.4s, v1.s[2] + add v26.4s, v24.4s, v14.4s + ldr q31, [x0, #0x110] + sub v14.4s, v15.4s, v23.4s + add v17.4s, v15.4s, v23.4s + mls v22.4s, v12.4s, v7.s[0] + add v28.4s, v13.4s, v18.4s + str q14, [x0, #0x380] + sqrdmulh v24.4s, v6.4s, v0.s[1] + add v5.4s, v25.4s, v11.4s + sub v19.4s, v13.4s, v18.4s + str q17, [x0, #0x300] + str q5, [x0, #0x100] + mul v16.4s, v9.4s, v2.s[0] + ldr q18, [x0, #0x310] + str q19, [x0, #0x280] + mls v16.4s, v20.4s, v7.s[0] + str q28, [x0, #0x200] + add v13.4s, v27.4s, v22.4s + ldr q15, [x0, #0x10] + sub v10.4s, v27.4s, v22.4s + mls v4.4s, v24.4s, v7.s[0] + str q13, [x0], #0x10 + str q10, [x0, #0x70] + sqrdmulh v12.4s, v29.4s, v1.s[1] + mul v23.4s, v29.4s, v1.s[0] + mul v8.4s, v26.4s, v1.s[2] + add v20.4s, v15.4s, v4.4s + sub v6.4s, v15.4s, v4.4s + mls v23.4s, v12.4s, v7.s[0] + sqrdmulh v22.4s, v18.4s, v0.s[1] + mul v5.4s, v18.4s, v0.s[0] + sub v28.4s, v21.4s, v23.4s + sqrdmulh v10.4s, v26.4s, v1.s[3] + mls v5.4s, v22.4s, v7.s[0] + sqrdmulh v30.4s, v28.4s, v3.s[1] + add v4.4s, v21.4s, v23.4s + mls v8.4s, v10.4s, v7.s[0] + add v12.4s, v31.4s, v5.4s + sub v9.4s, v31.4s, v5.4s + sqrdmulh v25.4s, v4.4s, v2.s[3] + sqrdmulh v15.4s, v9.4s, v1.s[1] + sqrdmulh v31.4s, v12.4s, v0.s[3] + mul v18.4s, v12.4s, v0.s[2] + mul v11.4s, v9.4s, v1.s[0] + mls v18.4s, v31.4s, v7.s[0] + mul v29.4s, v4.4s, v2.s[2] + mls v29.4s, v25.4s, v7.s[0] + add v23.4s, v20.4s, v18.4s + mls v11.4s, v15.4s, v7.s[0] + sub v31.4s, v20.4s, v18.4s + add v17.4s, v23.4s, v8.4s + add v5.4s, v31.4s, v16.4s + mul v24.4s, v28.4s, v3.s[0] + str q17, [x0], #0x10 + sub v19.4s, v31.4s, v16.4s + mls v24.4s, v30.4s, v7.s[0] + str q5, [x0, #0xf0] + add v31.4s, v6.4s, v11.4s + sub v26.4s, v23.4s, v8.4s + str q19, [x0, #0x170] + add v4.4s, v31.4s, v29.4s + sub v13.4s, v6.4s, v11.4s + str q26, [x0, #0x70] + sub v11.4s, v31.4s, v29.4s + sub v22.4s, v13.4s, v24.4s + add v23.4s, v13.4s, v24.4s + str q4, [x0, #0x1f0] + str q11, [x0, #0x270] + str q23, [x0, #0x2f0] + str q22, [x0, #0x370] + mov x0, x3 + mov x4, #0x8 // =8 + ldr q9, [x0, #0x40] + ldr q23, [x1], #0x40 + ldr q21, [x2, #0x60] + ldr q1, [x0, #0x20] + ldur q14, [x1, #-0x30] + ldr q13, [x0] + ldr q11, [x2, #0x50] + sqrdmulh v16.4s, v9.4s, v23.s[1] + ldr q17, [x0, #0x50] + mul v15.4s, v9.4s, v23.s[0] + ldr q30, [x0, #0x70] + ldr q27, [x0, #0x60] + ldr q8, [x2, #0x30] + sqrdmulh v12.4s, v17.4s, v23.s[1] + ldr q6, [x0, #0x30] + mls v15.4s, v16.4s, v7.s[0] + sqrdmulh v18.4s, v27.4s, v23.s[1] + sqrdmulh v19.4s, v30.4s, v23.s[1] + add v5.4s, v13.4s, v15.4s + mul v25.4s, v27.4s, v23.s[0] + sub v26.4s, v13.4s, v15.4s + mls v25.4s, v18.4s, v7.s[0] + mul v10.4s, v17.4s, v23.s[0] + mls v10.4s, v12.4s, v7.s[0] + mul v4.4s, v30.4s, v23.s[0] + sub v22.4s, v1.4s, v25.4s + mls v4.4s, v19.4s, v7.s[0] + add v28.4s, v1.4s, v25.4s + sqrdmulh v19.4s, v28.4s, v23.s[3] + sqrdmulh v9.4s, v22.4s, v14.s[1] + add v2.4s, v6.4s, v4.4s + mul v0.4s, v28.4s, v23.s[2] + sqrdmulh v27.4s, v2.4s, v23.s[3] + sub v17.4s, v6.4s, v4.4s + mul v3.4s, v2.4s, v23.s[2] + sqrdmulh v20.4s, v17.4s, v14.s[1] + ldr q1, [x0, #0x10] + mls v3.4s, v27.4s, v7.s[0] + mls v0.4s, v19.4s, v7.s[0] + ldur q16, [x1, #-0x20] + add v31.4s, v1.4s, v10.4s + mul v30.4s, v17.4s, v14.s[0] + mls v30.4s, v20.4s, v7.s[0] + add v27.4s, v31.4s, v3.4s + sub v23.4s, v1.4s, v10.4s + sub v24.4s, v31.4s, v3.4s + sqrdmulh v4.4s, v27.4s, v14.s[3] + sqrdmulh v10.4s, v24.4s, v16.s[1] + mul v18.4s, v24.4s, v16.s[0] + add v15.4s, v23.4s, v30.4s + sub v23.4s, v23.4s, v30.4s + mul v29.4s, v27.4s, v14.s[2] + sub v2.4s, v5.4s, v0.4s + add v12.4s, v5.4s, v0.4s + mls v18.4s, v10.4s, v7.s[0] + ldur q3, [x1, #-0x10] + mls v29.4s, v4.4s, v7.s[0] + mul v4.4s, v22.4s, v14.s[0] + add v1.4s, v2.4s, v18.4s + sub v24.4s, v2.4s, v18.4s + mls v4.4s, v9.4s, v7.s[0] + ldr q20, [x2, #0x10] + add v25.4s, v12.4s, v29.4s + mul v9.4s, v23.4s, v3.s[0] + sub v5.4s, v12.4s, v29.4s + sqrdmulh v31.4s, v23.4s, v3.s[1] + trn2 v6.4s, v1.4s, v24.4s + trn2 v10.4s, v25.4s, v5.4s + sqrdmulh v13.4s, v15.4s, v16.s[3] + trn2 v30.2d, v10.2d, v6.2d + ldr q3, [x2], #0xc0 + mul v12.4s, v15.4s, v16.s[2] + trn1 v27.2d, v10.2d, v6.2d + mls v9.4s, v31.4s, v7.s[0] + trn1 v22.4s, v25.4s, v5.4s + sub v6.4s, v26.4s, v4.4s + mls v12.4s, v13.4s, v7.s[0] + trn1 v1.4s, v1.4s, v24.4s + add v13.4s, v26.4s, v4.4s + trn2 v10.2d, v22.2d, v1.2d + mul v28.4s, v30.4s, v3.4s + sub v31.4s, v6.4s, v9.4s + sub x4, x4, #0x1 + +Lmld_ntt_layer45678_start: + add v2.4s, v13.4s, v12.4s + sqrdmulh v5.4s, v30.4s, v20.4s + sub v25.4s, v13.4s, v12.4s + add v17.4s, v6.4s, v9.4s + mul v19.4s, v10.4s, v3.4s + trn2 v4.4s, v2.4s, v25.4s + ldur q24, [x2, #-0x50] + trn2 v29.4s, v17.4s, v31.4s + sqrdmulh v15.4s, v10.4s, v20.4s + mls v28.4s, v5.4s, v7.s[0] + trn2 v3.2d, v4.2d, v29.2d + sqrdmulh v12.4s, v3.4s, v24.4s + mul v16.4s, v3.4s, v21.4s + mls v19.4s, v15.4s, v7.s[0] + ldur q10, [x2, #-0xa0] + add v13.4s, v27.4s, v28.4s + mls v16.4s, v12.4s, v7.s[0] + sqrdmulh v9.4s, v13.4s, v8.4s + sub v30.4s, v27.4s, v28.4s + ldr q18, [x1], #0x40 + mul v8.4s, v13.4s, v10.4s + ldr q10, [x0, #0xd0] + sqrdmulh v14.4s, v30.4s, v11.4s + ldr q23, [x0, #0xe0] + sqrdmulh v13.4s, v10.4s, v18.s[1] + sqrdmulh v12.4s, v23.4s, v18.s[1] + ldur q6, [x2, #-0x80] + mul v3.4s, v10.4s, v18.s[0] + mls v3.4s, v13.4s, v7.s[0] + ldr q13, [x0, #0xf0] + trn1 v27.4s, v2.4s, v25.4s + mul v2.4s, v30.4s, v6.4s + trn1 v20.4s, v17.4s, v31.4s + trn1 v25.2d, v4.2d, v29.2d + sqrdmulh v10.4s, v13.4s, v18.s[1] + trn2 v5.2d, v27.2d, v20.2d + ldur q6, [x2, #-0x10] + mls v8.4s, v9.4s, v7.s[0] + sub v15.4s, v25.4s, v16.4s + sqrdmulh v31.4s, v5.4s, v24.4s + sqrdmulh v30.4s, v15.4s, v6.4s + ldur q9, [x2, #-0x30] + mul v4.4s, v5.4s, v21.4s + ldur q21, [x2, #-0x40] + ldur q6, [x2, #-0x20] + add v5.4s, v25.4s, v16.4s + mls v4.4s, v31.4s, v7.s[0] + mul v0.4s, v5.4s, v21.4s + mul v17.4s, v13.4s, v18.s[0] + mls v17.4s, v10.4s, v7.s[0] + ldr q28, [x0, #0xb0] + sqrdmulh v26.4s, v5.4s, v9.4s + mul v9.4s, v15.4s, v6.4s + trn1 v6.2d, v22.2d, v1.2d + mls v9.4s, v30.4s, v7.s[0] + add v25.4s, v28.4s, v17.4s + mls v2.4s, v14.4s, v7.s[0] + trn1 v5.2d, v27.2d, v20.2d + ldr q20, [x2, #0x10] + mul v29.4s, v25.4s, v18.s[2] + add v15.4s, v6.4s, v19.4s + ldr q30, [x0, #0xc0] + sub v19.4s, v6.4s, v19.4s + add v31.4s, v15.4s, v8.4s + mls v0.4s, v26.4s, v7.s[0] + ldur q14, [x1, #-0x30] + add v21.4s, v19.4s, v2.4s + sub v24.4s, v19.4s, v2.4s + sqrdmulh v27.4s, v25.4s, v18.s[3] + sub v26.4s, v15.4s, v8.4s + ldr q2, [x0, #0x90] + mul v16.4s, v30.4s, v18.s[0] + sub v25.4s, v28.4s, v17.4s + trn1 v11.4s, v31.4s, v26.4s + ldr q1, [x0, #0xa0] + trn1 v6.4s, v21.4s, v24.4s + sqrdmulh v13.4s, v25.4s, v14.s[1] + add v8.4s, v2.4s, v3.4s + trn2 v28.2d, v11.2d, v6.2d + sqrdmulh v19.4s, v30.4s, v18.s[1] + sub v10.4s, v5.4s, v4.4s + ldur q22, [x1, #-0x20] + str q28, [x0, #0x20] + mls v29.4s, v27.4s, v7.s[0] + add v15.4s, v10.4s, v9.4s + mul v25.4s, v25.4s, v14.s[0] + ldur q27, [x1, #-0x10] + trn2 v17.4s, v31.4s, v26.4s + trn2 v21.4s, v21.4s, v24.4s + mls v16.4s, v19.4s, v7.s[0] + sub v24.4s, v8.4s, v29.4s + sub v10.4s, v10.4s, v9.4s + mls v25.4s, v13.4s, v7.s[0] + trn1 v13.2d, v11.2d, v6.2d + ldr q28, [x0, #0x80] + sqrdmulh v30.4s, v24.4s, v22.s[1] + trn2 v19.2d, v17.2d, v21.2d + trn1 v6.2d, v17.2d, v21.2d + mul v31.4s, v23.4s, v18.s[0] + str q13, [x0], #0x80 + stur q6, [x0, #-0x70] + stur q19, [x0, #-0x50] + ldr q11, [x2, #0x50] + mls v31.4s, v12.4s, v7.s[0] + ldr q21, [x2, #0x60] + trn1 v9.4s, v15.4s, v10.4s + trn2 v6.4s, v15.4s, v10.4s + mul v24.4s, v24.4s, v22.s[0] + sub v10.4s, v2.4s, v3.4s + ldr q3, [x2], #0xc0 + mls v24.4s, v30.4s, v7.s[0] + add v26.4s, v8.4s, v29.4s + ldur q8, [x2, #-0x90] + add v17.4s, v5.4s, v4.4s + sqrdmulh v2.4s, v26.4s, v14.s[3] + sub v13.4s, v1.4s, v31.4s + add v30.4s, v1.4s, v31.4s + add v15.4s, v10.4s, v25.4s + sqrdmulh v19.4s, v13.4s, v14.s[1] + sub v25.4s, v10.4s, v25.4s + mul v29.4s, v13.4s, v14.s[0] + sub v5.4s, v28.4s, v16.4s + sqrdmulh v4.4s, v30.4s, v18.s[3] + sub v23.4s, v17.4s, v0.4s + add v31.4s, v17.4s, v0.4s + mul v18.4s, v30.4s, v18.s[2] + add v1.4s, v28.4s, v16.4s + trn2 v12.4s, v31.4s, v23.4s + mls v29.4s, v19.4s, v7.s[0] + trn1 v13.4s, v31.4s, v23.4s + trn2 v30.2d, v12.2d, v6.2d + mls v18.4s, v4.4s, v7.s[0] + trn2 v10.2d, v13.2d, v9.2d + trn1 v31.2d, v13.2d, v9.2d + mul v19.4s, v26.4s, v14.s[2] + trn1 v12.2d, v12.2d, v6.2d + sub v6.4s, v5.4s, v29.4s + mls v19.4s, v2.4s, v7.s[0] + add v13.4s, v5.4s, v29.4s + stur q10, [x0, #-0x20] + sub v10.4s, v1.4s, v18.4s + add v28.4s, v1.4s, v18.4s + sqrdmulh v5.4s, v25.4s, v27.s[1] + stur q31, [x0, #-0x40] + add v26.4s, v10.4s, v24.4s + sub v31.4s, v10.4s, v24.4s + mul v9.4s, v25.4s, v27.s[0] + stur q12, [x0, #-0x30] + sub v24.4s, v28.4s, v19.4s + sqrdmulh v10.4s, v15.4s, v22.s[3] + trn1 v1.4s, v26.4s, v31.4s + stur q30, [x0, #-0x10] + add v30.4s, v28.4s, v19.4s + mls v9.4s, v5.4s, v7.s[0] + trn2 v25.4s, v26.4s, v31.4s + trn2 v14.4s, v30.4s, v24.4s + mul v12.4s, v15.4s, v22.s[2] + trn1 v22.4s, v30.4s, v24.4s + trn1 v27.2d, v14.2d, v25.2d + mls v12.4s, v10.4s, v7.s[0] + trn2 v30.2d, v14.2d, v25.2d + sub v31.4s, v6.4s, v9.4s + trn2 v10.2d, v22.2d, v1.2d + mul v28.4s, v30.4s, v3.4s + subs x4, x4, #0x1 + cbnz x4, Lmld_ntt_layer45678_start + add v9.4s, v6.4s, v9.4s + sqrdmulh v6.4s, v30.4s, v20.4s + ldur q24, [x2, #-0xa0] + add v25.4s, v13.4s, v12.4s + sub v15.4s, v13.4s, v12.4s + mul v19.4s, v10.4s, v3.4s + trn2 v5.4s, v9.4s, v31.4s + sqrdmulh v3.4s, v10.4s, v20.4s + trn2 v10.4s, v25.4s, v15.4s + mls v28.4s, v6.4s, v7.s[0] + trn2 v13.2d, v10.2d, v5.2d + ldur q30, [x2, #-0x50] + mul v12.4s, v13.4s, v21.4s + mls v19.4s, v3.4s, v7.s[0] + add v20.4s, v27.4s, v28.4s + sqrdmulh v13.4s, v13.4s, v30.4s + sub v3.4s, v27.4s, v28.4s + mul v24.4s, v20.4s, v24.4s + sqrdmulh v6.4s, v3.4s, v11.4s + ldur q27, [x2, #-0x80] + mls v12.4s, v13.4s, v7.s[0] + trn1 v25.4s, v25.4s, v15.4s + mul v27.4s, v3.4s, v27.4s + trn1 v31.4s, v9.4s, v31.4s + trn1 v3.2d, v10.2d, v5.2d + ldur q13, [x2, #-0x30] + ldur q15, [x2, #-0x40] + sqrdmulh v9.4s, v20.4s, v8.4s + trn2 v20.2d, v25.2d, v31.2d + ldur q10, [x2, #-0x10] + mls v27.4s, v6.4s, v7.s[0] + add v5.4s, v3.4s, v12.4s + sub v6.4s, v3.4s, v12.4s + sqrdmulh v3.4s, v20.4s, v30.4s + trn1 v12.2d, v22.2d, v1.2d + sqrdmulh v10.4s, v6.4s, v10.4s + mls v24.4s, v9.4s, v7.s[0] + sub v9.4s, v12.4s, v19.4s + trn1 v25.2d, v25.2d, v31.2d + sqrdmulh v31.4s, v5.4s, v13.4s + add v30.4s, v9.4s, v27.4s + add v13.4s, v12.4s, v19.4s + mul v1.4s, v20.4s, v21.4s + ldur q12, [x2, #-0x20] + add v21.4s, v13.4s, v24.4s + sub v13.4s, v13.4s, v24.4s + mls v1.4s, v3.4s, v7.s[0] + sub v3.4s, v9.4s, v27.4s + mul v9.4s, v6.4s, v12.4s + trn2 v12.4s, v21.4s, v13.4s + trn1 v6.4s, v30.4s, v3.4s + trn2 v30.4s, v30.4s, v3.4s + mls v9.4s, v10.4s, v7.s[0] + trn1 v13.4s, v21.4s, v13.4s + mul v15.4s, v5.4s, v15.4s + sub v3.4s, v25.4s, v1.4s + add v5.4s, v25.4s, v1.4s + mls v15.4s, v31.4s, v7.s[0] + trn1 v21.2d, v13.2d, v6.2d + trn2 v6.2d, v13.2d, v6.2d + add v10.4s, v3.4s, v9.4s + sub v13.4s, v3.4s, v9.4s + str q21, [x0], #0x80 + trn1 v3.2d, v12.2d, v30.2d + trn2 v31.2d, v12.2d, v30.2d + trn1 v21.4s, v10.4s, v13.4s + sub v30.4s, v5.4s, v15.4s + add v12.4s, v5.4s, v15.4s + stur q3, [x0, #-0x70] + trn2 v13.4s, v10.4s, v13.4s + trn1 v19.4s, v12.4s, v30.4s + trn2 v12.4s, v12.4s, v30.4s + stur q6, [x0, #-0x60] + stur q31, [x0, #-0x50] + trn1 v10.2d, v19.2d, v21.2d + trn2 v3.2d, v19.2d, v21.2d + trn1 v21.2d, v12.2d, v13.2d + trn2 v13.2d, v12.2d, v13.2d + stur q10, [x0, #-0x40] + stur q3, [x0, #-0x20] + stur q13, [x0, #-0x10] + stur q21, [x0, #-0x30] + ldp d8, d9, [sp] + .cfi_restore d8 + .cfi_restore d9 + ldp d10, d11, [sp, #0x10] + .cfi_restore d10 + .cfi_restore d11 + ldp d12, d13, [sp, #0x20] + .cfi_restore d12 + .cfi_restore d13 + ldp d14, d15, [sp, #0x30] + .cfi_restore d14 + .cfi_restore d15 + add sp, sp, #0x40 + .cfi_adjust_cfa_offset -0x40 + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(ntt_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_pointwise_montgomery_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_pointwise_montgomery_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_pointwise_montgomery_aarch64_asm.S @@ -0,0 +1,106 @@ +/* Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/*yaml + Name: poly_pointwise_montgomery_aarch64_asm + Description: AArch64 pointwise Montgomery multiplication of two polynomials + Signature: void mld_poly_pointwise_montgomery_aarch64_asm(int32_t a[256], const int32_t b[256]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t a[256] + description: Input/output polynomial (256 x int32_t) + x1: + type: buffer + size_bytes: 1024 + permissions: read-only + c_parameter: const int32_t b[256] + description: Input polynomial (256 x int32_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_AARCH64) && \ + (!defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \ + defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_pointwise_montgomery_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(poly_pointwise_montgomery_aarch64_asm) +MLD_ASM_FN_SYMBOL(poly_pointwise_montgomery_aarch64_asm) + + .cfi_startproc + mov w3, #0xe001 // =57345 + movk w3, #0x7f, lsl #16 + dup v0.4s, w3 + mov w3, #0x2001 // =8193 + movk w3, #0x380, lsl #16 + dup v1.4s, w3 + mov x3, #0x40 // =64 + +Lmld_poly_pointwise_montgomery_loop_start: + ldr q16, [x0] + ldr q17, [x0, #0x10] + ldr q18, [x0, #0x20] + ldr q19, [x0, #0x30] + ldr q21, [x1, #0x10] + ldr q22, [x1, #0x20] + ldr q23, [x1, #0x30] + ldr q20, [x1], #0x40 + smull v24.2d, v16.2s, v20.2s + smull2 v25.2d, v16.4s, v20.4s + smull v26.2d, v17.2s, v21.2s + smull2 v27.2d, v17.4s, v21.4s + smull v28.2d, v18.2s, v22.2s + smull2 v29.2d, v18.4s, v22.4s + smull v30.2d, v19.2s, v23.2s + smull2 v31.2d, v19.4s, v23.4s + uzp1 v16.4s, v24.4s, v25.4s + mul v16.4s, v16.4s, v1.4s + smlsl v24.2d, v16.2s, v0.2s + smlsl2 v25.2d, v16.4s, v0.4s + uzp2 v16.4s, v24.4s, v25.4s + uzp1 v17.4s, v26.4s, v27.4s + mul v17.4s, v17.4s, v1.4s + smlsl v26.2d, v17.2s, v0.2s + smlsl2 v27.2d, v17.4s, v0.4s + uzp2 v17.4s, v26.4s, v27.4s + uzp1 v18.4s, v28.4s, v29.4s + mul v18.4s, v18.4s, v1.4s + smlsl v28.2d, v18.2s, v0.2s + smlsl2 v29.2d, v18.4s, v0.4s + uzp2 v18.4s, v28.4s, v29.4s + uzp1 v19.4s, v30.4s, v31.4s + mul v19.4s, v19.4s, v1.4s + smlsl v30.2d, v19.2s, v0.2s + smlsl2 v31.2d, v19.4s, v0.4s + uzp2 v19.4s, v30.4s, v31.4s + str q17, [x0, #0x10] + str q18, [x0, #0x20] + str q19, [x0, #0x30] + str q16, [x0], #0x40 + subs x3, x3, #0x4 + cbnz x3, Lmld_poly_pointwise_montgomery_loop_start + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(poly_pointwise_montgomery_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && (!MLD_CONFIG_NO_SIGN_API || \ + !MLD_CONFIG_NO_VERIFY_API || MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) \ + && !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_poly_caddq_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_poly_caddq_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_poly_caddq_aarch64_asm.S @@ -0,0 +1,69 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +/*yaml + Name: poly_caddq_aarch64_asm + Description: AArch64 conditional addition of q to each coefficient + Signature: void mld_poly_caddq_aarch64_asm(int32_t a[256]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t a[256] + description: Input/output polynomial (256 x int32_t) +*/ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_poly_caddq_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(poly_caddq_aarch64_asm) +MLD_ASM_FN_SYMBOL(poly_caddq_aarch64_asm) + + .cfi_startproc + mov w9, #0xe001 // =57345 + movk w9, #0x7f, lsl #16 + dup v4.4s, w9 + mov x1, #0x10 // =16 + +Lmld_poly_caddq_loop: + ldr q0, [x0] + ldr q1, [x0, #0x10] + ldr q2, [x0, #0x20] + ldr q3, [x0, #0x30] + ushr v5.4s, v0.4s, #0x1f + mla v0.4s, v5.4s, v4.4s + ushr v5.4s, v1.4s, #0x1f + mla v1.4s, v5.4s, v4.4s + ushr v5.4s, v2.4s, #0x1f + mla v2.4s, v5.4s, v4.4s + ushr v5.4s, v3.4s, #0x1f + mla v3.4s, v5.4s, v4.4s + str q1, [x0, #0x10] + str q2, [x0, #0x20] + str q3, [x0, #0x30] + str q0, [x0], #0x40 + subs x1, x1, #0x1 + b.ne Lmld_poly_caddq_loop + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(poly_caddq_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_poly_chknorm_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_poly_chknorm_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_poly_chknorm_aarch64_asm.S @@ -0,0 +1,76 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +/*yaml + Name: poly_chknorm_aarch64_asm + Description: AArch64 infinity-norm bound check on polynomial coefficients + Signature: int mld_poly_chknorm_aarch64_asm(const int32_t a[256], int32_t B) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: read-only + c_parameter: const int32_t a[256] + description: Input polynomial (256 x int32_t) + x1: + type: scalar + c_parameter: int32_t B + description: Norm bound + test_with: 131072 # representative non-negative bound (1 << 17) +*/ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_poly_chknorm_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(poly_chknorm_aarch64_asm) +MLD_ASM_FN_SYMBOL(poly_chknorm_aarch64_asm) + + .cfi_startproc + dup v20.4s, w1 + eor v21.16b, v21.16b, v21.16b + mov x2, #0x10 // =16 + +Lmld_poly_chknorm_loop: + ldr q1, [x0, #0x10] + ldr q2, [x0, #0x20] + ldr q3, [x0, #0x30] + ldr q0, [x0], #0x40 + abs v1.4s, v1.4s + cmge v1.4s, v1.4s, v20.4s + orr v21.16b, v21.16b, v1.16b + abs v2.4s, v2.4s + cmge v2.4s, v2.4s, v20.4s + orr v21.16b, v21.16b, v2.16b + abs v3.4s, v3.4s + cmge v3.4s, v3.4s, v20.4s + orr v21.16b, v21.16b, v3.16b + abs v0.4s, v0.4s + cmge v0.4s, v0.4s, v20.4s + orr v21.16b, v21.16b, v0.16b + subs x2, x2, #0x1 + b.ne Lmld_poly_chknorm_loop + umaxv s21, v21.4s + fmov w0, s21 + and w0, w0, #0x1 + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(poly_chknorm_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_poly_decompose_32_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_poly_decompose_32_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_poly_decompose_32_aarch64_asm.S @@ -0,0 +1,108 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +/*yaml + Name: poly_decompose_32_aarch64_asm + Description: AArch64 coefficient decomposition (alpha = (Q-1)/32) + Signature: void mld_poly_decompose_32_aarch64_asm(int32_t a1[256], int32_t a0[256]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t a1[256] + description: Output high-part polynomial (256 x int32_t) + x1: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t a0[256] + description: Input polynomial / output low-part (256 x int32_t) +*/ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_NO_SIGN_API) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_poly_decompose_32_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(poly_decompose_32_aarch64_asm) +MLD_ASM_FN_SYMBOL(poly_decompose_32_aarch64_asm) + + .cfi_startproc + mov w4, #0xe001 // =57345 + movk w4, #0x7f, lsl #16 + dup v20.4s, w4 + mov w5, #0xe100 // =57600 + movk w5, #0x7b, lsl #16 + dup v21.4s, w5 + mov w7, #0xfe00 // =65024 + movk w7, #0x7, lsl #16 + dup v22.4s, w7 + mov w11, #0x401 // =1025 + movk w11, #0x4010, lsl #16 + dup v23.4s, w11 + mov x3, #0x10 // =16 + +Lmld_poly_decompose_32_loop: + ldr q0, [x1] + ldr q1, [x1, #0x10] + ldr q2, [x1, #0x20] + ldr q3, [x1, #0x30] + sqdmulh v5.4s, v1.4s, v23.4s + srshr v5.4s, v5.4s, #0x12 + cmgt v24.4s, v1.4s, v21.4s + mls v1.4s, v5.4s, v22.4s + bic v5.16b, v5.16b, v24.16b + add v1.4s, v1.4s, v24.4s + sqdmulh v6.4s, v2.4s, v23.4s + srshr v6.4s, v6.4s, #0x12 + cmgt v24.4s, v2.4s, v21.4s + mls v2.4s, v6.4s, v22.4s + bic v6.16b, v6.16b, v24.16b + add v2.4s, v2.4s, v24.4s + sqdmulh v7.4s, v3.4s, v23.4s + srshr v7.4s, v7.4s, #0x12 + cmgt v24.4s, v3.4s, v21.4s + mls v3.4s, v7.4s, v22.4s + bic v7.16b, v7.16b, v24.16b + add v3.4s, v3.4s, v24.4s + sqdmulh v4.4s, v0.4s, v23.4s + srshr v4.4s, v4.4s, #0x12 + cmgt v24.4s, v0.4s, v21.4s + mls v0.4s, v4.4s, v22.4s + bic v4.16b, v4.16b, v24.16b + add v0.4s, v0.4s, v24.4s + str q5, [x0, #0x10] + str q6, [x0, #0x20] + str q7, [x0, #0x30] + str q4, [x0], #0x40 + str q1, [x1, #0x10] + str q2, [x1, #0x20] + str q3, [x1, #0x30] + str q0, [x1], #0x40 + subs x3, x3, #0x1 + b.ne Lmld_poly_decompose_32_loop + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(poly_decompose_32_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_SIGN_API && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_poly_decompose_88_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_poly_decompose_88_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_poly_decompose_88_aarch64_asm.S @@ -0,0 +1,108 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +/*yaml + Name: poly_decompose_88_aarch64_asm + Description: AArch64 coefficient decomposition (alpha = (Q-1)/88) + Signature: void mld_poly_decompose_88_aarch64_asm(int32_t a1[256], int32_t a0[256]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t a1[256] + description: Output high-part polynomial (256 x int32_t) + x1: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t a0[256] + description: Input polynomial / output low-part (256 x int32_t) +*/ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_NO_SIGN_API) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_poly_decompose_88_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(poly_decompose_88_aarch64_asm) +MLD_ASM_FN_SYMBOL(poly_decompose_88_aarch64_asm) + + .cfi_startproc + mov w4, #0xe001 // =57345 + movk w4, #0x7f, lsl #16 + dup v20.4s, w4 + mov w5, #0x6c00 // =27648 + movk w5, #0x7e, lsl #16 + dup v21.4s, w5 + mov w7, #0xe800 // =59392 + movk w7, #0x2, lsl #16 + dup v22.4s, w7 + mov w11, #0x581 // =1409 + movk w11, #0x5816, lsl #16 + dup v23.4s, w11 + mov x3, #0x10 // =16 + +Lmld_poly_decompose_88_loop: + ldr q0, [x1] + ldr q1, [x1, #0x10] + ldr q2, [x1, #0x20] + ldr q3, [x1, #0x30] + sqdmulh v5.4s, v1.4s, v23.4s + srshr v5.4s, v5.4s, #0x11 + cmgt v24.4s, v1.4s, v21.4s + mls v1.4s, v5.4s, v22.4s + bic v5.16b, v5.16b, v24.16b + add v1.4s, v1.4s, v24.4s + sqdmulh v6.4s, v2.4s, v23.4s + srshr v6.4s, v6.4s, #0x11 + cmgt v24.4s, v2.4s, v21.4s + mls v2.4s, v6.4s, v22.4s + bic v6.16b, v6.16b, v24.16b + add v2.4s, v2.4s, v24.4s + sqdmulh v7.4s, v3.4s, v23.4s + srshr v7.4s, v7.4s, #0x11 + cmgt v24.4s, v3.4s, v21.4s + mls v3.4s, v7.4s, v22.4s + bic v7.16b, v7.16b, v24.16b + add v3.4s, v3.4s, v24.4s + sqdmulh v4.4s, v0.4s, v23.4s + srshr v4.4s, v4.4s, #0x11 + cmgt v24.4s, v0.4s, v21.4s + mls v0.4s, v4.4s, v22.4s + bic v4.16b, v4.16b, v24.16b + add v0.4s, v0.4s, v24.4s + str q5, [x0, #0x10] + str q6, [x0, #0x20] + str q7, [x0, #0x30] + str q4, [x0], #0x40 + str q1, [x1, #0x10] + str q2, [x1, #0x20] + str q3, [x1, #0x30] + str q0, [x1], #0x40 + subs x3, x3, #0x1 + b.ne Lmld_poly_decompose_88_loop + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(poly_decompose_88_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_SIGN_API && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44) \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_poly_use_hint_32_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_poly_use_hint_32_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_poly_use_hint_32_aarch64_asm.S @@ -0,0 +1,125 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +/*yaml + Name: poly_use_hint_32_aarch64_asm + Description: AArch64 hint application (alpha = (Q-1)/32) + Signature: void mld_poly_use_hint_32_aarch64_asm(int32_t a[256], const int32_t h[256]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t a[256] + description: Input/output polynomial (256 x int32_t) + x1: + type: buffer + size_bytes: 1024 + permissions: read-only + c_parameter: const int32_t h[256] + description: Hint polynomial (256 x int32_t) +*/ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_NO_VERIFY_API) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_poly_use_hint_32_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(poly_use_hint_32_aarch64_asm) +MLD_ASM_FN_SYMBOL(poly_use_hint_32_aarch64_asm) + + .cfi_startproc + mov w4, #0xe001 // =57345 + movk w4, #0x7f, lsl #16 + dup v20.4s, w4 + mov w5, #0xe100 // =57600 + movk w5, #0x7b, lsl #16 + dup v21.4s, w5 + mov w7, #0xfe00 // =65024 + movk w7, #0x7, lsl #16 + dup v22.4s, w7 + mov w11, #0x401 // =1025 + movk w11, #0x4010, lsl #16 + dup v23.4s, w11 + movi v24.4s, #0xf + mov x3, #0x10 // =16 + +Lmld_poly_use_hint_32_loop: + ldr q1, [x0, #0x10] + ldr q2, [x0, #0x20] + ldr q3, [x0, #0x30] + ldr q0, [x0] + ldr q5, [x1, #0x10] + ldr q6, [x1, #0x20] + ldr q7, [x1, #0x30] + ldr q4, [x1], #0x40 + sqdmulh v17.4s, v1.4s, v23.4s + srshr v17.4s, v17.4s, #0x12 + cmgt v25.4s, v1.4s, v21.4s + mls v1.4s, v17.4s, v22.4s + bic v17.16b, v17.16b, v25.16b + add v1.4s, v1.4s, v25.4s + cmle v1.4s, v1.4s, #0 + orr v1.4s, #0x1 + mla v17.4s, v1.4s, v5.4s + and v17.16b, v17.16b, v24.16b + sqdmulh v18.4s, v2.4s, v23.4s + srshr v18.4s, v18.4s, #0x12 + cmgt v25.4s, v2.4s, v21.4s + mls v2.4s, v18.4s, v22.4s + bic v18.16b, v18.16b, v25.16b + add v2.4s, v2.4s, v25.4s + cmle v2.4s, v2.4s, #0 + orr v2.4s, #0x1 + mla v18.4s, v2.4s, v6.4s + and v18.16b, v18.16b, v24.16b + sqdmulh v19.4s, v3.4s, v23.4s + srshr v19.4s, v19.4s, #0x12 + cmgt v25.4s, v3.4s, v21.4s + mls v3.4s, v19.4s, v22.4s + bic v19.16b, v19.16b, v25.16b + add v3.4s, v3.4s, v25.4s + cmle v3.4s, v3.4s, #0 + orr v3.4s, #0x1 + mla v19.4s, v3.4s, v7.4s + and v19.16b, v19.16b, v24.16b + sqdmulh v16.4s, v0.4s, v23.4s + srshr v16.4s, v16.4s, #0x12 + cmgt v25.4s, v0.4s, v21.4s + mls v0.4s, v16.4s, v22.4s + bic v16.16b, v16.16b, v25.16b + add v0.4s, v0.4s, v25.4s + cmle v0.4s, v0.4s, #0 + orr v0.4s, #0x1 + mla v16.4s, v0.4s, v4.4s + and v16.16b, v16.16b, v24.16b + str q17, [x0, #0x10] + str q18, [x0, #0x20] + str q19, [x0, #0x30] + str q16, [x0], #0x40 + subs x3, x3, #0x1 + b.ne Lmld_poly_use_hint_32_loop + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(poly_use_hint_32_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_VERIFY_API && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_poly_use_hint_88_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_poly_use_hint_88_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_poly_use_hint_88_aarch64_asm.S @@ -0,0 +1,133 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +/*yaml + Name: poly_use_hint_88_aarch64_asm + Description: AArch64 hint application (alpha = (Q-1)/88) + Signature: void mld_poly_use_hint_88_aarch64_asm(int32_t a[256], const int32_t h[256]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t a[256] + description: Input/output polynomial (256 x int32_t) + x1: + type: buffer + size_bytes: 1024 + permissions: read-only + c_parameter: const int32_t h[256] + description: Hint polynomial (256 x int32_t) +*/ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_NO_VERIFY_API) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_poly_use_hint_88_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(poly_use_hint_88_aarch64_asm) +MLD_ASM_FN_SYMBOL(poly_use_hint_88_aarch64_asm) + + .cfi_startproc + mov w4, #0xe001 // =57345 + movk w4, #0x7f, lsl #16 + dup v20.4s, w4 + mov w5, #0x6c00 // =27648 + movk w5, #0x7e, lsl #16 + dup v21.4s, w5 + mov w7, #0xe800 // =59392 + movk w7, #0x2, lsl #16 + dup v22.4s, w7 + mov w11, #0x581 // =1409 + movk w11, #0x5816, lsl #16 + dup v23.4s, w11 + movi v24.4s, #0x2b + mov x3, #0x10 // =16 + +Lmld_poly_use_hint_88_loop: + ldr q1, [x0, #0x10] + ldr q2, [x0, #0x20] + ldr q3, [x0, #0x30] + ldr q0, [x0] + ldr q5, [x1, #0x10] + ldr q6, [x1, #0x20] + ldr q7, [x1, #0x30] + ldr q4, [x1], #0x40 + sqdmulh v17.4s, v1.4s, v23.4s + srshr v17.4s, v17.4s, #0x11 + cmgt v25.4s, v1.4s, v21.4s + mls v1.4s, v17.4s, v22.4s + bic v17.16b, v17.16b, v25.16b + add v1.4s, v1.4s, v25.4s + cmle v1.4s, v1.4s, #0 + orr v1.4s, #0x1 + mla v17.4s, v1.4s, v5.4s + cmgt v25.4s, v17.4s, v24.4s + bic v17.16b, v17.16b, v25.16b + umin v17.4s, v17.4s, v24.4s + sqdmulh v18.4s, v2.4s, v23.4s + srshr v18.4s, v18.4s, #0x11 + cmgt v25.4s, v2.4s, v21.4s + mls v2.4s, v18.4s, v22.4s + bic v18.16b, v18.16b, v25.16b + add v2.4s, v2.4s, v25.4s + cmle v2.4s, v2.4s, #0 + orr v2.4s, #0x1 + mla v18.4s, v2.4s, v6.4s + cmgt v25.4s, v18.4s, v24.4s + bic v18.16b, v18.16b, v25.16b + umin v18.4s, v18.4s, v24.4s + sqdmulh v19.4s, v3.4s, v23.4s + srshr v19.4s, v19.4s, #0x11 + cmgt v25.4s, v3.4s, v21.4s + mls v3.4s, v19.4s, v22.4s + bic v19.16b, v19.16b, v25.16b + add v3.4s, v3.4s, v25.4s + cmle v3.4s, v3.4s, #0 + orr v3.4s, #0x1 + mla v19.4s, v3.4s, v7.4s + cmgt v25.4s, v19.4s, v24.4s + bic v19.16b, v19.16b, v25.16b + umin v19.4s, v19.4s, v24.4s + sqdmulh v16.4s, v0.4s, v23.4s + srshr v16.4s, v16.4s, #0x11 + cmgt v25.4s, v0.4s, v21.4s + mls v0.4s, v16.4s, v22.4s + bic v16.16b, v16.16b, v25.16b + add v0.4s, v0.4s, v25.4s + cmle v0.4s, v0.4s, #0 + orr v0.4s, #0x1 + mla v16.4s, v0.4s, v4.4s + cmgt v25.4s, v16.4s, v24.4s + bic v16.16b, v16.16b, v25.16b + umin v16.4s, v16.4s, v24.4s + str q17, [x0, #0x10] + str q18, [x0, #0x20] + str q19, [x0, #0x30] + str q16, [x0], #0x40 + subs x3, x3, #0x1 + b.ne Lmld_poly_use_hint_88_loop + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(poly_use_hint_88_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_VERIFY_API && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44) \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm.S @@ -0,0 +1,157 @@ +/* Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/*yaml + Name: polyvecl_pointwise_acc_montgomery_l4_aarch64_asm + Description: AArch64 pointwise multiply-accumulate of length-4 polynomial vectors + Signature: void mld_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm(int32_t r[256], const int32_t a[4][256], const int32_t b[4][256]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t r[256] + description: Output polynomial (256 x int32_t) + x1: + type: buffer + size_bytes: 4096 + permissions: read-only + c_parameter: const int32_t a[4][256] + description: Input polynomial vector a (4 x 256 x int32_t) + x2: + type: buffer + size_bytes: 4096 + permissions: read-only + c_parameter: const int32_t b[4][256] + description: Input polynomial vector b (4 x 256 x int32_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 4) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_polyvecl_pointwise_acc_montgomery_l4_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(polyvecl_pointwise_acc_montgomery_l4_aarch64_asm) +MLD_ASM_FN_SYMBOL(polyvecl_pointwise_acc_montgomery_l4_aarch64_asm) + + .cfi_startproc + mov w3, #0xe001 // =57345 + movk w3, #0x7f, lsl #16 + dup v0.4s, w3 + mov w3, #0x2001 // =8193 + movk w3, #0x380, lsl #16 + dup v1.4s, w3 + mov x3, #0x40 // =64 + +Lmld_polyvecl_pointwise_acc_montgomery_l4_loop_start: + ldr q17, [x1, #0x10] + ldr q18, [x1, #0x20] + ldr q19, [x1, #0x30] + ldr q16, [x1], #0x40 + ldr q21, [x2, #0x10] + ldr q22, [x2, #0x20] + ldr q23, [x2, #0x30] + ldr q20, [x2], #0x40 + smull v24.2d, v16.2s, v20.2s + smull2 v25.2d, v16.4s, v20.4s + smull v26.2d, v17.2s, v21.2s + smull2 v27.2d, v17.4s, v21.4s + smull v28.2d, v18.2s, v22.2s + smull2 v29.2d, v18.4s, v22.4s + smull v30.2d, v19.2s, v23.2s + smull2 v31.2d, v19.4s, v23.4s + ldr q16, [x1, #0x3c0] + ldr q17, [x1, #0x3d0] + ldr q18, [x1, #0x3e0] + ldr q19, [x1, #0x3f0] + ldr q20, [x2, #0x3c0] + ldr q21, [x2, #0x3d0] + ldr q22, [x2, #0x3e0] + ldr q23, [x2, #0x3f0] + smlal v24.2d, v16.2s, v20.2s + smlal2 v25.2d, v16.4s, v20.4s + smlal v26.2d, v17.2s, v21.2s + smlal2 v27.2d, v17.4s, v21.4s + smlal v28.2d, v18.2s, v22.2s + smlal2 v29.2d, v18.4s, v22.4s + smlal v30.2d, v19.2s, v23.2s + smlal2 v31.2d, v19.4s, v23.4s + ldr q16, [x1, #0x7c0] + ldr q17, [x1, #0x7d0] + ldr q18, [x1, #0x7e0] + ldr q19, [x1, #0x7f0] + ldr q20, [x2, #0x7c0] + ldr q21, [x2, #0x7d0] + ldr q22, [x2, #0x7e0] + ldr q23, [x2, #0x7f0] + smlal v24.2d, v16.2s, v20.2s + smlal2 v25.2d, v16.4s, v20.4s + smlal v26.2d, v17.2s, v21.2s + smlal2 v27.2d, v17.4s, v21.4s + smlal v28.2d, v18.2s, v22.2s + smlal2 v29.2d, v18.4s, v22.4s + smlal v30.2d, v19.2s, v23.2s + smlal2 v31.2d, v19.4s, v23.4s + ldr q16, [x1, #0xbc0] + ldr q17, [x1, #0xbd0] + ldr q18, [x1, #0xbe0] + ldr q19, [x1, #0xbf0] + ldr q20, [x2, #0xbc0] + ldr q21, [x2, #0xbd0] + ldr q22, [x2, #0xbe0] + ldr q23, [x2, #0xbf0] + smlal v24.2d, v16.2s, v20.2s + smlal2 v25.2d, v16.4s, v20.4s + smlal v26.2d, v17.2s, v21.2s + smlal2 v27.2d, v17.4s, v21.4s + smlal v28.2d, v18.2s, v22.2s + smlal2 v29.2d, v18.4s, v22.4s + smlal v30.2d, v19.2s, v23.2s + smlal2 v31.2d, v19.4s, v23.4s + uzp1 v16.4s, v24.4s, v25.4s + mul v16.4s, v16.4s, v1.4s + smlsl v24.2d, v16.2s, v0.2s + smlsl2 v25.2d, v16.4s, v0.4s + uzp2 v16.4s, v24.4s, v25.4s + uzp1 v17.4s, v26.4s, v27.4s + mul v17.4s, v17.4s, v1.4s + smlsl v26.2d, v17.2s, v0.2s + smlsl2 v27.2d, v17.4s, v0.4s + uzp2 v17.4s, v26.4s, v27.4s + uzp1 v18.4s, v28.4s, v29.4s + mul v18.4s, v18.4s, v1.4s + smlsl v28.2d, v18.2s, v0.2s + smlsl2 v29.2d, v18.4s, v0.4s + uzp2 v18.4s, v28.4s, v29.4s + uzp1 v19.4s, v30.4s, v31.4s + mul v19.4s, v19.4s, v1.4s + smlsl v30.2d, v19.2s, v0.2s + smlsl2 v31.2d, v19.4s, v0.4s + uzp2 v19.4s, v30.4s, v31.4s + str q17, [x0, #0x10] + str q18, [x0, #0x20] + str q19, [x0, #0x30] + str q16, [x0], #0x40 + subs x3, x3, #0x4 + cbnz x3, Lmld_polyvecl_pointwise_acc_montgomery_l4_loop_start + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(polyvecl_pointwise_acc_montgomery_l4_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 4) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm.S @@ -0,0 +1,173 @@ +/* Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/*yaml + Name: polyvecl_pointwise_acc_montgomery_l5_aarch64_asm + Description: AArch64 pointwise multiply-accumulate of length-5 polynomial vectors + Signature: void mld_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm(int32_t r[256], const int32_t a[5][256], const int32_t b[5][256]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t r[256] + description: Output polynomial (256 x int32_t) + x1: + type: buffer + size_bytes: 5120 + permissions: read-only + c_parameter: const int32_t a[5][256] + description: Input polynomial vector a (5 x 256 x int32_t) + x2: + type: buffer + size_bytes: 5120 + permissions: read-only + c_parameter: const int32_t b[5][256] + description: Input polynomial vector b (5 x 256 x int32_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 5) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_polyvecl_pointwise_acc_montgomery_l5_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(polyvecl_pointwise_acc_montgomery_l5_aarch64_asm) +MLD_ASM_FN_SYMBOL(polyvecl_pointwise_acc_montgomery_l5_aarch64_asm) + + .cfi_startproc + mov w3, #0xe001 // =57345 + movk w3, #0x7f, lsl #16 + dup v0.4s, w3 + mov w3, #0x2001 // =8193 + movk w3, #0x380, lsl #16 + dup v1.4s, w3 + mov x3, #0x40 // =64 + +Lmld_polyvecl_pointwise_acc_montgomery_l5_loop_start: + ldr q17, [x1, #0x10] + ldr q18, [x1, #0x20] + ldr q19, [x1, #0x30] + ldr q16, [x1], #0x40 + ldr q21, [x2, #0x10] + ldr q22, [x2, #0x20] + ldr q23, [x2, #0x30] + ldr q20, [x2], #0x40 + smull v24.2d, v16.2s, v20.2s + smull2 v25.2d, v16.4s, v20.4s + smull v26.2d, v17.2s, v21.2s + smull2 v27.2d, v17.4s, v21.4s + smull v28.2d, v18.2s, v22.2s + smull2 v29.2d, v18.4s, v22.4s + smull v30.2d, v19.2s, v23.2s + smull2 v31.2d, v19.4s, v23.4s + ldr q16, [x1, #0x3c0] + ldr q17, [x1, #0x3d0] + ldr q18, [x1, #0x3e0] + ldr q19, [x1, #0x3f0] + ldr q20, [x2, #0x3c0] + ldr q21, [x2, #0x3d0] + ldr q22, [x2, #0x3e0] + ldr q23, [x2, #0x3f0] + smlal v24.2d, v16.2s, v20.2s + smlal2 v25.2d, v16.4s, v20.4s + smlal v26.2d, v17.2s, v21.2s + smlal2 v27.2d, v17.4s, v21.4s + smlal v28.2d, v18.2s, v22.2s + smlal2 v29.2d, v18.4s, v22.4s + smlal v30.2d, v19.2s, v23.2s + smlal2 v31.2d, v19.4s, v23.4s + ldr q16, [x1, #0x7c0] + ldr q17, [x1, #0x7d0] + ldr q18, [x1, #0x7e0] + ldr q19, [x1, #0x7f0] + ldr q20, [x2, #0x7c0] + ldr q21, [x2, #0x7d0] + ldr q22, [x2, #0x7e0] + ldr q23, [x2, #0x7f0] + smlal v24.2d, v16.2s, v20.2s + smlal2 v25.2d, v16.4s, v20.4s + smlal v26.2d, v17.2s, v21.2s + smlal2 v27.2d, v17.4s, v21.4s + smlal v28.2d, v18.2s, v22.2s + smlal2 v29.2d, v18.4s, v22.4s + smlal v30.2d, v19.2s, v23.2s + smlal2 v31.2d, v19.4s, v23.4s + ldr q16, [x1, #0xbc0] + ldr q17, [x1, #0xbd0] + ldr q18, [x1, #0xbe0] + ldr q19, [x1, #0xbf0] + ldr q20, [x2, #0xbc0] + ldr q21, [x2, #0xbd0] + ldr q22, [x2, #0xbe0] + ldr q23, [x2, #0xbf0] + smlal v24.2d, v16.2s, v20.2s + smlal2 v25.2d, v16.4s, v20.4s + smlal v26.2d, v17.2s, v21.2s + smlal2 v27.2d, v17.4s, v21.4s + smlal v28.2d, v18.2s, v22.2s + smlal2 v29.2d, v18.4s, v22.4s + smlal v30.2d, v19.2s, v23.2s + smlal2 v31.2d, v19.4s, v23.4s + ldr q16, [x1, #0xfc0] + ldr q17, [x1, #0xfd0] + ldr q18, [x1, #0xfe0] + ldr q19, [x1, #0xff0] + ldr q20, [x2, #0xfc0] + ldr q21, [x2, #0xfd0] + ldr q22, [x2, #0xfe0] + ldr q23, [x2, #0xff0] + smlal v24.2d, v16.2s, v20.2s + smlal2 v25.2d, v16.4s, v20.4s + smlal v26.2d, v17.2s, v21.2s + smlal2 v27.2d, v17.4s, v21.4s + smlal v28.2d, v18.2s, v22.2s + smlal2 v29.2d, v18.4s, v22.4s + smlal v30.2d, v19.2s, v23.2s + smlal2 v31.2d, v19.4s, v23.4s + uzp1 v16.4s, v24.4s, v25.4s + mul v16.4s, v16.4s, v1.4s + smlsl v24.2d, v16.2s, v0.2s + smlsl2 v25.2d, v16.4s, v0.4s + uzp2 v16.4s, v24.4s, v25.4s + uzp1 v17.4s, v26.4s, v27.4s + mul v17.4s, v17.4s, v1.4s + smlsl v26.2d, v17.2s, v0.2s + smlsl2 v27.2d, v17.4s, v0.4s + uzp2 v17.4s, v26.4s, v27.4s + uzp1 v18.4s, v28.4s, v29.4s + mul v18.4s, v18.4s, v1.4s + smlsl v28.2d, v18.2s, v0.2s + smlsl2 v29.2d, v18.4s, v0.4s + uzp2 v18.4s, v28.4s, v29.4s + uzp1 v19.4s, v30.4s, v31.4s + mul v19.4s, v19.4s, v1.4s + smlsl v30.2d, v19.2s, v0.2s + smlsl2 v31.2d, v19.4s, v0.4s + uzp2 v19.4s, v30.4s, v31.4s + str q17, [x0, #0x10] + str q18, [x0, #0x20] + str q19, [x0, #0x30] + str q16, [x0], #0x40 + subs x3, x3, #0x4 + cbnz x3, Lmld_polyvecl_pointwise_acc_montgomery_l5_loop_start + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(polyvecl_pointwise_acc_montgomery_l5_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 5) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm.S @@ -0,0 +1,205 @@ +/* Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/*yaml + Name: polyvecl_pointwise_acc_montgomery_l7_aarch64_asm + Description: AArch64 pointwise multiply-accumulate of length-7 polynomial vectors + Signature: void mld_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm(int32_t r[256], const int32_t a[7][256], const int32_t b[7][256]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t r[256] + description: Output polynomial (256 x int32_t) + x1: + type: buffer + size_bytes: 7168 + permissions: read-only + c_parameter: const int32_t a[7][256] + description: Input polynomial vector a (7 x 256 x int32_t) + x2: + type: buffer + size_bytes: 7168 + permissions: read-only + c_parameter: const int32_t b[7][256] + description: Input polynomial vector b (7 x 256 x int32_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_AARCH64) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 7) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_polyvecl_pointwise_acc_montgomery_l7_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(polyvecl_pointwise_acc_montgomery_l7_aarch64_asm) +MLD_ASM_FN_SYMBOL(polyvecl_pointwise_acc_montgomery_l7_aarch64_asm) + + .cfi_startproc + mov w3, #0xe001 // =57345 + movk w3, #0x7f, lsl #16 + dup v0.4s, w3 + mov w3, #0x2001 // =8193 + movk w3, #0x380, lsl #16 + dup v1.4s, w3 + mov x3, #0x40 // =64 + +Lmld_polyvecl_pointwise_acc_montgomery_l7_loop_start: + ldr q17, [x1, #0x10] + ldr q18, [x1, #0x20] + ldr q19, [x1, #0x30] + ldr q16, [x1], #0x40 + ldr q21, [x2, #0x10] + ldr q22, [x2, #0x20] + ldr q23, [x2, #0x30] + ldr q20, [x2], #0x40 + smull v24.2d, v16.2s, v20.2s + smull2 v25.2d, v16.4s, v20.4s + smull v26.2d, v17.2s, v21.2s + smull2 v27.2d, v17.4s, v21.4s + smull v28.2d, v18.2s, v22.2s + smull2 v29.2d, v18.4s, v22.4s + smull v30.2d, v19.2s, v23.2s + smull2 v31.2d, v19.4s, v23.4s + ldr q16, [x1, #0x3c0] + ldr q17, [x1, #0x3d0] + ldr q18, [x1, #0x3e0] + ldr q19, [x1, #0x3f0] + ldr q20, [x2, #0x3c0] + ldr q21, [x2, #0x3d0] + ldr q22, [x2, #0x3e0] + ldr q23, [x2, #0x3f0] + smlal v24.2d, v16.2s, v20.2s + smlal2 v25.2d, v16.4s, v20.4s + smlal v26.2d, v17.2s, v21.2s + smlal2 v27.2d, v17.4s, v21.4s + smlal v28.2d, v18.2s, v22.2s + smlal2 v29.2d, v18.4s, v22.4s + smlal v30.2d, v19.2s, v23.2s + smlal2 v31.2d, v19.4s, v23.4s + ldr q16, [x1, #0x7c0] + ldr q17, [x1, #0x7d0] + ldr q18, [x1, #0x7e0] + ldr q19, [x1, #0x7f0] + ldr q20, [x2, #0x7c0] + ldr q21, [x2, #0x7d0] + ldr q22, [x2, #0x7e0] + ldr q23, [x2, #0x7f0] + smlal v24.2d, v16.2s, v20.2s + smlal2 v25.2d, v16.4s, v20.4s + smlal v26.2d, v17.2s, v21.2s + smlal2 v27.2d, v17.4s, v21.4s + smlal v28.2d, v18.2s, v22.2s + smlal2 v29.2d, v18.4s, v22.4s + smlal v30.2d, v19.2s, v23.2s + smlal2 v31.2d, v19.4s, v23.4s + ldr q16, [x1, #0xbc0] + ldr q17, [x1, #0xbd0] + ldr q18, [x1, #0xbe0] + ldr q19, [x1, #0xbf0] + ldr q20, [x2, #0xbc0] + ldr q21, [x2, #0xbd0] + ldr q22, [x2, #0xbe0] + ldr q23, [x2, #0xbf0] + smlal v24.2d, v16.2s, v20.2s + smlal2 v25.2d, v16.4s, v20.4s + smlal v26.2d, v17.2s, v21.2s + smlal2 v27.2d, v17.4s, v21.4s + smlal v28.2d, v18.2s, v22.2s + smlal2 v29.2d, v18.4s, v22.4s + smlal v30.2d, v19.2s, v23.2s + smlal2 v31.2d, v19.4s, v23.4s + ldr q16, [x1, #0xfc0] + ldr q17, [x1, #0xfd0] + ldr q18, [x1, #0xfe0] + ldr q19, [x1, #0xff0] + ldr q20, [x2, #0xfc0] + ldr q21, [x2, #0xfd0] + ldr q22, [x2, #0xfe0] + ldr q23, [x2, #0xff0] + smlal v24.2d, v16.2s, v20.2s + smlal2 v25.2d, v16.4s, v20.4s + smlal v26.2d, v17.2s, v21.2s + smlal2 v27.2d, v17.4s, v21.4s + smlal v28.2d, v18.2s, v22.2s + smlal2 v29.2d, v18.4s, v22.4s + smlal v30.2d, v19.2s, v23.2s + smlal2 v31.2d, v19.4s, v23.4s + ldr q16, [x1, #0x13c0] + ldr q17, [x1, #0x13d0] + ldr q18, [x1, #0x13e0] + ldr q19, [x1, #0x13f0] + ldr q20, [x2, #0x13c0] + ldr q21, [x2, #0x13d0] + ldr q22, [x2, #0x13e0] + ldr q23, [x2, #0x13f0] + smlal v24.2d, v16.2s, v20.2s + smlal2 v25.2d, v16.4s, v20.4s + smlal v26.2d, v17.2s, v21.2s + smlal2 v27.2d, v17.4s, v21.4s + smlal v28.2d, v18.2s, v22.2s + smlal2 v29.2d, v18.4s, v22.4s + smlal v30.2d, v19.2s, v23.2s + smlal2 v31.2d, v19.4s, v23.4s + ldr q16, [x1, #0x17c0] + ldr q17, [x1, #0x17d0] + ldr q18, [x1, #0x17e0] + ldr q19, [x1, #0x17f0] + ldr q20, [x2, #0x17c0] + ldr q21, [x2, #0x17d0] + ldr q22, [x2, #0x17e0] + ldr q23, [x2, #0x17f0] + smlal v24.2d, v16.2s, v20.2s + smlal2 v25.2d, v16.4s, v20.4s + smlal v26.2d, v17.2s, v21.2s + smlal2 v27.2d, v17.4s, v21.4s + smlal v28.2d, v18.2s, v22.2s + smlal2 v29.2d, v18.4s, v22.4s + smlal v30.2d, v19.2s, v23.2s + smlal2 v31.2d, v19.4s, v23.4s + uzp1 v16.4s, v24.4s, v25.4s + mul v16.4s, v16.4s, v1.4s + smlsl v24.2d, v16.2s, v0.2s + smlsl2 v25.2d, v16.4s, v0.4s + uzp2 v16.4s, v24.4s, v25.4s + uzp1 v17.4s, v26.4s, v27.4s + mul v17.4s, v17.4s, v1.4s + smlsl v26.2d, v17.2s, v0.2s + smlsl2 v27.2d, v17.4s, v0.4s + uzp2 v17.4s, v26.4s, v27.4s + uzp1 v18.4s, v28.4s, v29.4s + mul v18.4s, v18.4s, v1.4s + smlsl v28.2d, v18.2s, v0.2s + smlsl2 v29.2d, v18.4s, v0.4s + uzp2 v18.4s, v28.4s, v29.4s + uzp1 v19.4s, v30.4s, v31.4s + mul v19.4s, v19.4s, v1.4s + smlsl v30.2d, v19.2s, v0.2s + smlsl2 v31.2d, v19.4s, v0.4s + uzp2 v19.4s, v30.4s, v31.4s + str q17, [x0, #0x10] + str q18, [x0, #0x20] + str q19, [x0, #0x30] + str q16, [x0], #0x40 + subs x3, x3, #0x4 + cbnz x3, Lmld_polyvecl_pointwise_acc_montgomery_l7_loop_start + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(polyvecl_pointwise_acc_montgomery_l7_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 7) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_polyz_unpack_17_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_polyz_unpack_17_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_polyz_unpack_17_aarch64_asm.S @@ -0,0 +1,103 @@ +/* + * Copyright (c) The mldsa-native project authors + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/*yaml + Name: polyz_unpack_17_aarch64_asm + Description: AArch64 unpacking of 17-bit packed coefficients + Signature: void mld_polyz_unpack_17_aarch64_asm(int32_t r[256], const uint8_t buf[576], const uint8_t indices[64]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t r[256] + description: Output polynomial (256 x int32_t) + x1: + type: buffer + size_bytes: 576 + permissions: read-only + c_parameter: const uint8_t buf[576] + description: Packed input bytes + x2: + type: buffer + size_bytes: 64 + permissions: read-only + c_parameter: const uint8_t indices[64] + description: Permutation index table (64 x uint8_t) +*/ + + #include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_AARCH64) && \ + (!defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_polyz_unpack_17_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(polyz_unpack_17_aarch64_asm) +MLD_ASM_FN_SYMBOL(polyz_unpack_17_aarch64_asm) + + .cfi_startproc + ldr q24, [x2] + ldr q25, [x2, #0x10] + ldr q26, [x2, #0x20] + ldr q27, [x2, #0x30] + mov x3, #0xfe00000000 // =1090921693184 + mov v28.d[0], x3 + mov x3, #0xfc // =252 + movk x3, #0xfa, lsl #32 + mov v28.d[1], x3 + movi v29.4s, #0x3, msl #16 + movi v30.4s, #0x2, lsl #16 + mov x9, #0x10 // =16 + +Lmld_polyz_unpack_17_loop: + ld1 { v0.16b, v1.16b }, [x1] + add x1, x1, #0x14 + ld1 { v2.16b }, [x1], #16 + tbl v4.16b, { v0.16b }, v24.16b + tbl v5.16b, { v0.16b, v1.16b }, v25.16b + tbl v6.16b, { v1.16b }, v26.16b + tbl v7.16b, { v1.16b, v2.16b }, v27.16b + ushl v4.4s, v4.4s, v28.4s + and v4.16b, v4.16b, v29.16b + sub v4.4s, v30.4s, v4.4s + ushl v5.4s, v5.4s, v28.4s + and v5.16b, v5.16b, v29.16b + sub v5.4s, v30.4s, v5.4s + ushl v6.4s, v6.4s, v28.4s + and v6.16b, v6.16b, v29.16b + sub v6.4s, v30.4s, v6.4s + ushl v7.4s, v7.4s, v28.4s + and v7.16b, v7.16b, v29.16b + sub v7.4s, v30.4s, v7.4s + str q5, [x0, #0x10] + str q6, [x0, #0x20] + str q7, [x0, #0x30] + str q4, [x0], #0x40 + subs x9, x9, #0x1 + b.ne Lmld_polyz_unpack_17_loop + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(polyz_unpack_17_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && (!MLD_CONFIG_NO_SIGN_API || \ + !MLD_CONFIG_NO_VERIFY_API) && !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44) \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_polyz_unpack_19_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_polyz_unpack_19_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_polyz_unpack_19_aarch64_asm.S @@ -0,0 +1,100 @@ +/* + * Copyright (c) The mldsa-native project authors + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/*yaml + Name: polyz_unpack_19_aarch64_asm + Description: AArch64 unpacking of 19-bit packed coefficients + Signature: void mld_polyz_unpack_19_aarch64_asm(int32_t r[256], const uint8_t buf[640], const uint8_t indices[64]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t r[256] + description: Output polynomial (256 x int32_t) + x1: + type: buffer + size_bytes: 640 + permissions: read-only + c_parameter: const uint8_t buf[640] + description: Packed input bytes + x2: + type: buffer + size_bytes: 64 + permissions: read-only + c_parameter: const uint8_t indices[64] + description: Permutation index table (64 x uint8_t) +*/ + + #include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_AARCH64) && \ + (!defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API)) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_polyz_unpack_19_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(polyz_unpack_19_aarch64_asm) +MLD_ASM_FN_SYMBOL(polyz_unpack_19_aarch64_asm) + + .cfi_startproc + ldr q24, [x2] + ldr q25, [x2, #0x10] + ldr q26, [x2, #0x20] + ldr q27, [x2, #0x30] + mov x3, #0xfc00000000 // =1082331758592 + dup v28.2d, x3 + movi v29.4s, #0xf, msl #16 + movi v30.4s, #0x8, lsl #16 + mov x9, #0x10 // =16 + +Lmld_polyz_unpack_19_loop: + ld1 { v0.16b, v1.16b }, [x1] + add x1, x1, #0x18 + ld1 { v2.16b }, [x1], #16 + tbl v4.16b, { v0.16b }, v24.16b + tbl v5.16b, { v0.16b, v1.16b }, v25.16b + tbl v6.16b, { v1.16b }, v26.16b + tbl v7.16b, { v1.16b, v2.16b }, v27.16b + ushl v4.4s, v4.4s, v28.4s + and v4.16b, v4.16b, v29.16b + sub v4.4s, v30.4s, v4.4s + ushl v5.4s, v5.4s, v28.4s + and v5.16b, v5.16b, v29.16b + sub v5.4s, v30.4s, v5.4s + ushl v6.4s, v6.4s, v28.4s + and v6.16b, v6.16b, v29.16b + sub v6.4s, v30.4s, v6.4s + ushl v7.4s, v7.4s, v28.4s + and v7.16b, v7.16b, v29.16b + sub v7.4s, v30.4s, v7.4s + str q5, [x0, #0x10] + str q6, [x0, #0x20] + str q7, [x0, #0x30] + str q4, [x0], #0x40 + subs x9, x9, #0x1 + b.ne Lmld_polyz_unpack_19_loop + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(polyz_unpack_19_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && (!MLD_CONFIG_NO_SIGN_API || \ + !MLD_CONFIG_NO_VERIFY_API) && !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_rej_uniform_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_rej_uniform_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_rej_uniform_aarch64_asm.S @@ -0,0 +1,222 @@ +/* + * Copyright (c) The mldsa-native project authors + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/*yaml + Name: rej_uniform_aarch64_asm + Description: AArch64 rejection sampling of uniform coefficients mod q + Signature: uint64_t mld_rej_uniform_aarch64_asm(int32_t r[256], const uint8_t *buf, unsigned buflen, const uint8_t table[256]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t r[256] + description: Output buffer (256 x int32_t) + x1: + type: buffer + size_bytes: x2 + permissions: read-only + c_parameter: const uint8_t *buf + description: Input buffer + x2: + type: scalar + c_parameter: unsigned buflen + description: Length of input buffer (must be a multiple of 24) + test_with: 840 # MLD_POLY_UNIFORM_NBLOCKS * SHAKE128_RATE = 5 * 168 + x3: + type: buffer + size_bytes: 256 + permissions: read-only + c_parameter: const uint8_t table[256] + description: Lookup table (256 x uint8_t) +*/ + + #include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_AARCH64) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_rej_uniform_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(rej_uniform_aarch64_asm) +MLD_ASM_FN_SYMBOL(rej_uniform_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x440 + .cfi_adjust_cfa_offset 0x440 + mov x7, #0x1 // =1 + movk x7, #0x2, lsl #32 + mov v31.d[0], x7 + mov x7, #0x4 // =4 + movk x7, #0x8, lsl #32 + mov v31.d[1], x7 + mov w7, #0xe001 // =57345 + movk w7, #0x7f, lsl #16 + dup v30.4s, w7 + mov x8, sp + mov x7, x8 + mov x11, #0x0 // =0 + eor v16.16b, v16.16b, v16.16b + +Lmld_rej_uniform_initial_zero: + str q16, [x7], #0x40 + stur q16, [x7, #-0x30] + stur q16, [x7, #-0x20] + stur q16, [x7, #-0x10] + add x11, x11, #0x10 + cmp x11, #0x100 + b.lt Lmld_rej_uniform_initial_zero + mov x7, x8 + mov x9, #0x0 // =0 + mov x4, #0x100 // =256 + cmp x2, #0x30 + b.lo Lmld_rej_uniform_loop48_end + +Lmld_rej_uniform_loop48: + cmp x9, x4 + b.hs Lmld_rej_uniform_memory_copy + sub x2, x2, #0x30 + ld3 { v0.16b, v1.16b, v2.16b }, [x1], #48 + movi v4.16b, #0x80 + bic v2.16b, v2.16b, v4.16b + zip1 v4.16b, v0.16b, v1.16b + zip2 v5.16b, v0.16b, v1.16b + ushll v6.8h, v2.8b, #0x0 + ushll2 v7.8h, v2.16b, #0x0 + zip1 v16.8h, v4.8h, v6.8h + zip2 v17.8h, v4.8h, v6.8h + zip1 v18.8h, v5.8h, v7.8h + zip2 v19.8h, v5.8h, v7.8h + cmhi v4.4s, v30.4s, v16.4s + cmhi v5.4s, v30.4s, v17.4s + cmhi v6.4s, v30.4s, v18.4s + cmhi v7.4s, v30.4s, v19.4s + and v4.16b, v4.16b, v31.16b + and v5.16b, v5.16b, v31.16b + and v6.16b, v6.16b, v31.16b + and v7.16b, v7.16b, v31.16b + uaddlv d20, v4.4s + uaddlv d21, v5.4s + uaddlv d22, v6.4s + uaddlv d23, v7.4s + fmov x12, d20 + fmov x13, d21 + fmov x14, d22 + fmov x15, d23 + ldr q24, [x3, x12, lsl #4] + ldr q25, [x3, x13, lsl #4] + ldr q26, [x3, x14, lsl #4] + ldr q27, [x3, x15, lsl #4] + cnt v4.16b, v4.16b + cnt v5.16b, v5.16b + cnt v6.16b, v6.16b + cnt v7.16b, v7.16b + uaddlv d20, v4.4s + uaddlv d21, v5.4s + uaddlv d22, v6.4s + uaddlv d23, v7.4s + fmov x12, d20 + fmov x13, d21 + fmov x14, d22 + fmov x15, d23 + tbl v16.16b, { v16.16b }, v24.16b + tbl v17.16b, { v17.16b }, v25.16b + tbl v18.16b, { v18.16b }, v26.16b + tbl v19.16b, { v19.16b }, v27.16b + st1 { v16.4s }, [x7] + add x7, x7, x12, lsl #2 + st1 { v17.4s }, [x7] + add x7, x7, x13, lsl #2 + st1 { v18.4s }, [x7] + add x7, x7, x14, lsl #2 + st1 { v19.4s }, [x7] + add x7, x7, x15, lsl #2 + add x12, x12, x13 + add x14, x14, x15 + add x9, x9, x12 + add x9, x9, x14 + cmp x2, #0x30 + b.hs Lmld_rej_uniform_loop48 + +Lmld_rej_uniform_loop48_end: + cmp x9, x4 + b.hs Lmld_rej_uniform_memory_copy + cmp x2, #0x18 + b.lo Lmld_rej_uniform_memory_copy + sub x2, x2, #0x18 + ld3 { v0.8b, v1.8b, v2.8b }, [x1], #24 + movi v4.16b, #0x80 + bic v2.16b, v2.16b, v4.16b + zip1 v4.16b, v0.16b, v1.16b + ushll v6.8h, v2.8b, #0x0 + zip1 v16.8h, v4.8h, v6.8h + zip2 v17.8h, v4.8h, v6.8h + cmhi v4.4s, v30.4s, v16.4s + cmhi v5.4s, v30.4s, v17.4s + and v4.16b, v4.16b, v31.16b + and v5.16b, v5.16b, v31.16b + uaddlv d20, v4.4s + uaddlv d21, v5.4s + fmov x12, d20 + fmov x13, d21 + ldr q24, [x3, x12, lsl #4] + ldr q25, [x3, x13, lsl #4] + cnt v4.16b, v4.16b + cnt v5.16b, v5.16b + uaddlv d20, v4.4s + uaddlv d21, v5.4s + fmov x12, d20 + fmov x13, d21 + tbl v16.16b, { v16.16b }, v24.16b + tbl v17.16b, { v17.16b }, v25.16b + st1 { v16.4s }, [x7] + add x7, x7, x12, lsl #2 + st1 { v17.4s }, [x7] + add x7, x7, x13, lsl #2 + add x9, x9, x12 + add x9, x9, x13 + +Lmld_rej_uniform_memory_copy: + cmp x9, x4 + csel x9, x9, x4, lo + mov x11, #0x0 // =0 + mov x7, x8 + +Lmld_rej_uniform_final_copy: + ldr q16, [x7], #0x40 + ldur q17, [x7, #-0x30] + ldur q18, [x7, #-0x20] + ldur q19, [x7, #-0x10] + str q16, [x0], #0x40 + stur q17, [x0, #-0x30] + stur q18, [x0, #-0x20] + stur q19, [x0, #-0x10] + add x11, x11, #0x10 + cmp x11, #0x100 + b.lt Lmld_rej_uniform_final_copy + mov x0, x9 + b Lmld_rej_uniform_return + +Lmld_rej_uniform_return: + add sp, sp, #0x440 + .cfi_adjust_cfa_offset -0x440 + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(rej_uniform_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_rej_uniform_eta2_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_rej_uniform_eta2_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_rej_uniform_eta2_aarch64_asm.S @@ -0,0 +1,170 @@ +/* + * Copyright (c) The mldsa-native project authors + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/*yaml + Name: rej_uniform_eta2_aarch64_asm + Description: AArch64 rejection sampling of eta=2 secret coefficients + Signature: uint64_t mld_rej_uniform_eta2_aarch64_asm(int32_t r[256], const uint8_t *buf, unsigned buflen, const uint8_t table[4096]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t r[256] + description: Output buffer (256 x int32_t) + x1: + type: buffer + size_bytes: x2 + permissions: read-only + c_parameter: const uint8_t *buf + description: Input buffer + x2: + type: scalar + c_parameter: unsigned buflen + description: Length of input buffer (must be a multiple of 8) + test_with: 136 # MLD_AARCH64_REJ_UNIFORM_ETA2_BUFLEN + x3: + type: buffer + size_bytes: 4096 + permissions: read-only + c_parameter: const uint8_t table[4096] + description: Lookup table (4096 x uint8_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_AARCH64) && \ + !defined(MLD_CONFIG_NO_KEYPAIR_API) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 2) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_rej_uniform_eta2_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(rej_uniform_eta2_aarch64_asm) +MLD_ASM_FN_SYMBOL(rej_uniform_eta2_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x240 + .cfi_adjust_cfa_offset 0x240 + mov x7, #0x1 // =1 + movk x7, #0x2, lsl #16 + movk x7, #0x4, lsl #32 + movk x7, #0x8, lsl #48 + mov v31.d[0], x7 + mov x7, #0x10 // =16 + movk x7, #0x20, lsl #16 + movk x7, #0x40, lsl #32 + movk x7, #0x80, lsl #48 + mov v31.d[1], x7 + movi v30.8h, #0xf + mov x8, sp + mov x7, x8 + mov x11, #0x0 // =0 + eor v16.16b, v16.16b, v16.16b + +Lmld_rej_uniform_eta2_initial_zero: + str q16, [x7], #0x40 + stur q16, [x7, #-0x30] + stur q16, [x7, #-0x20] + stur q16, [x7, #-0x10] + add x11, x11, #0x20 + cmp x11, #0x100 + b.lt Lmld_rej_uniform_eta2_initial_zero + mov x7, x8 + mov x9, #0x0 // =0 + mov x4, #0x100 // =256 + +Lmld_rej_uniform_eta2_loop8: + cmp x9, x4 + b.hs Lmld_rej_uniform_eta2_memory_copy + sub x2, x2, #0x8 + ld1 { v0.8b }, [x1], #8 + movi v26.8b, #0xf + and v27.8b, v0.8b, v26.8b + ushr v28.8b, v0.8b, #0x4 + zip1 v26.8b, v27.8b, v28.8b + zip2 v29.8b, v27.8b, v28.8b + ushll v16.8h, v26.8b, #0x0 + ushll v17.8h, v29.8b, #0x0 + cmhi v4.8h, v30.8h, v16.8h + cmhi v5.8h, v30.8h, v17.8h + and v4.16b, v4.16b, v31.16b + and v5.16b, v5.16b, v31.16b + uaddlv s20, v4.8h + uaddlv s21, v5.8h + fmov w12, s20 + fmov w13, s21 + ldr q24, [x3, x12, lsl #4] + ldr q25, [x3, x13, lsl #4] + cnt v4.16b, v4.16b + cnt v5.16b, v5.16b + uaddlv s20, v4.8h + uaddlv s21, v5.8h + fmov w12, s20 + fmov w13, s21 + tbl v16.16b, { v16.16b }, v24.16b + tbl v17.16b, { v17.16b }, v25.16b + st1 { v16.8h }, [x7] + add x7, x7, x12, lsl #1 + st1 { v17.8h }, [x7] + add x7, x7, x13, lsl #1 + add x12, x12, x13 + add x9, x9, x12 + cmp x2, #0x8 + b.hs Lmld_rej_uniform_eta2_loop8 + +Lmld_rej_uniform_eta2_memory_copy: + cmp x9, x4 + csel x9, x9, x4, lo + mov w7, #0x199a // =6554 + dup v26.8h, w7 + movi v27.8h, #0x5 + movi v7.8h, #0x2 + mov x11, #0x0 // =0 + mov x7, x8 + +Lmld_rej_uniform_eta2_final_copy: + ldr q16, [x7], #0x20 + ldur q18, [x7, #-0x10] + sqdmulh v28.8h, v16.8h, v26.8h + mls v16.8h, v28.8h, v27.8h + sqdmulh v28.8h, v18.8h, v26.8h + mls v18.8h, v28.8h, v27.8h + sub v16.8h, v7.8h, v16.8h + sub v18.8h, v7.8h, v18.8h + sshll2 v17.4s, v16.8h, #0x0 + sshll v16.4s, v16.4h, #0x0 + sshll2 v19.4s, v18.8h, #0x0 + sshll v18.4s, v18.4h, #0x0 + str q16, [x0], #0x40 + stur q17, [x0, #-0x30] + stur q18, [x0, #-0x20] + stur q19, [x0, #-0x10] + add x11, x11, #0x10 + cmp x11, #0x100 + b.lt Lmld_rej_uniform_eta2_final_copy + mov x0, x9 + add sp, sp, #0x240 + .cfi_adjust_cfa_offset -0x240 + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(rej_uniform_eta2_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_KEYPAIR_API && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 2) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/mldsa_rej_uniform_eta4_aarch64_asm.S b/cbits/mldsa/src/native/aarch64/src/mldsa_rej_uniform_eta4_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/mldsa_rej_uniform_eta4_aarch64_asm.S @@ -0,0 +1,163 @@ +/* + * Copyright (c) The mldsa-native project authors + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/*yaml + Name: rej_uniform_eta4_aarch64_asm + Description: AArch64 rejection sampling of eta=4 secret coefficients + Signature: uint64_t mld_rej_uniform_eta4_aarch64_asm(int32_t r[256], const uint8_t *buf, unsigned buflen, const uint8_t table[4096]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t r[256] + description: Output buffer (256 x int32_t) + x1: + type: buffer + size_bytes: x2 + permissions: read-only + c_parameter: const uint8_t *buf + description: Input buffer + x2: + type: scalar + c_parameter: unsigned buflen + description: Length of input buffer (must be a multiple of 8) + test_with: 272 # MLD_AARCH64_REJ_UNIFORM_ETA4_BUFLEN + x3: + type: buffer + size_bytes: 4096 + permissions: read-only + c_parameter: const uint8_t table[4096] + description: Lookup table (4096 x uint8_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_AARCH64) && \ + !defined(MLD_CONFIG_NO_KEYPAIR_API) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 4) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/aarch64_opt/src/mldsa_rej_uniform_eta4_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(rej_uniform_eta4_aarch64_asm) +MLD_ASM_FN_SYMBOL(rej_uniform_eta4_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x240 + .cfi_adjust_cfa_offset 0x240 + mov x7, #0x1 // =1 + movk x7, #0x2, lsl #16 + movk x7, #0x4, lsl #32 + movk x7, #0x8, lsl #48 + mov v31.d[0], x7 + mov x7, #0x10 // =16 + movk x7, #0x20, lsl #16 + movk x7, #0x40, lsl #32 + movk x7, #0x80, lsl #48 + mov v31.d[1], x7 + movi v30.8h, #0x9 + movi v7.8h, #0x4 + mov x8, sp + mov x7, x8 + mov x11, #0x0 // =0 + eor v16.16b, v16.16b, v16.16b + +Lmld_rej_uniform_eta4_initial_zero: + str q16, [x7], #0x40 + stur q16, [x7, #-0x30] + stur q16, [x7, #-0x20] + stur q16, [x7, #-0x10] + add x11, x11, #0x20 + cmp x11, #0x100 + b.lt Lmld_rej_uniform_eta4_initial_zero + mov x7, x8 + mov x9, #0x0 // =0 + mov x4, #0x100 // =256 + +Lmld_rej_uniform_eta4_loop8: + cmp x9, x4 + b.hs Lmld_rej_uniform_eta4_memory_copy + sub x2, x2, #0x8 + ld1 { v0.8b }, [x1], #8 + movi v26.8b, #0xf + and v27.8b, v0.8b, v26.8b + ushr v28.8b, v0.8b, #0x4 + zip1 v26.8b, v27.8b, v28.8b + zip2 v29.8b, v27.8b, v28.8b + ushll v16.8h, v26.8b, #0x0 + ushll v17.8h, v29.8b, #0x0 + cmhi v4.8h, v30.8h, v16.8h + cmhi v5.8h, v30.8h, v17.8h + and v4.16b, v4.16b, v31.16b + and v5.16b, v5.16b, v31.16b + uaddlv s20, v4.8h + uaddlv s21, v5.8h + fmov w12, s20 + fmov w13, s21 + ldr q24, [x3, x12, lsl #4] + ldr q25, [x3, x13, lsl #4] + cnt v4.16b, v4.16b + cnt v5.16b, v5.16b + uaddlv s20, v4.8h + uaddlv s21, v5.8h + fmov w12, s20 + fmov w13, s21 + tbl v16.16b, { v16.16b }, v24.16b + tbl v17.16b, { v17.16b }, v25.16b + st1 { v16.8h }, [x7] + add x7, x7, x12, lsl #1 + st1 { v17.8h }, [x7] + add x7, x7, x13, lsl #1 + add x12, x12, x13 + add x9, x9, x12 + cmp x2, #0x8 + b.hs Lmld_rej_uniform_eta4_loop8 + +Lmld_rej_uniform_eta4_memory_copy: + cmp x9, x4 + csel x9, x9, x4, lo + mov x11, #0x0 // =0 + mov x7, x8 + +Lmld_rej_uniform_eta4_final_copy: + ldr q16, [x7], #0x20 + ldur q18, [x7, #-0x10] + sub v16.8h, v7.8h, v16.8h + sub v18.8h, v7.8h, v18.8h + sshll2 v17.4s, v16.8h, #0x0 + sshll v16.4s, v16.4h, #0x0 + sshll2 v19.4s, v18.8h, #0x0 + sshll v18.4s, v18.4h, #0x0 + str q16, [x0], #0x40 + stur q17, [x0, #-0x30] + stur q18, [x0, #-0x20] + stur q19, [x0, #-0x10] + add x11, x11, #0x10 + cmp x11, #0x100 + b.lt Lmld_rej_uniform_eta4_final_copy + mov x0, x9 + add sp, sp, #0x240 + .cfi_adjust_cfa_offset -0x240 + ret + .cfi_endproc + +MLD_ASM_FN_SIZE(rej_uniform_eta4_aarch64_asm) + +#endif /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_KEYPAIR_API && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 4) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/aarch64/src/polyz_unpack_table.c b/cbits/mldsa/src/native/aarch64/src/polyz_unpack_table.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/polyz_unpack_table.c @@ -0,0 +1,52 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mldsa-native repository. + * Do not modify it directly. + */ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_AARCH64) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +#include "arith_native_aarch64.h" + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +/* Table of indices used for tbl instructions in polyz_unpack_{17,19}. + * See autogen for details. */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44 +MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint8_t + mld_polyz_unpack_17_indices[64] = { + 0, 1, 2, 255, 2, 3, 4, 255, 4, 5, 6, 255, 6, 7, 8, 255, + 9, 10, 11, 255, 11, 12, 13, 255, 13, 14, 15, 255, 15, 16, 17, 255, + 2, 3, 4, 255, 4, 5, 6, 255, 6, 7, 8, 255, 8, 9, 10, 255, + 11, 12, 13, 255, 13, 14, 15, 255, 15, 28, 29, 255, 29, 30, 31, 255, +}; +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \ + */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) +MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint8_t + mld_polyz_unpack_19_indices[64] = { + 0, 1, 2, 255, 2, 3, 4, 255, 5, 6, 7, 255, 7, 8, 9, 255, + 10, 11, 12, 255, 12, 13, 14, 255, 15, 16, 17, 255, 17, 18, 19, 255, + 4, 5, 6, 255, 6, 7, 8, 255, 9, 10, 11, 255, 11, 12, 13, 255, + 14, 15, 24, 255, 24, 25, 26, 255, 27, 28, 29, 255, 29, 30, 31, 255, +}; +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87 */ + +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#else /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +MLD_EMPTY_CU(aarch64_polyz_unpack_table) + +#endif /* !(MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mldsa/src/native/aarch64/src/rej_uniform_eta_table.c b/cbits/mldsa/src/native/aarch64/src/rej_uniform_eta_table.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/rej_uniform_eta_table.c @@ -0,0 +1,547 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mldsa-native repository. + * Do not modify it directly. + */ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_AARCH64) && \ + !defined(MLD_CONFIG_NO_KEYPAIR_API) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +#include "arith_native_aarch64.h" + +/* + * Lookup table used by 16-bit rejection sampling (rej_eta). + * Adapted from ML-KEM for ML-DSA eta rejection sampling. + * See autogen for details. + */ +MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint8_t + mld_rej_uniform_eta_table[4096] = { + 255, 255, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 0 */, + 0, 1, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 1 */, + 2, 3, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 2 */, + 0, 1, 2, 3, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 3 */, + 4, 5, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 4 */, + 0, 1, 4, 5, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 5 */, + 2, 3, 4, 5, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 6 */, + 0, 1, 2, 3, 4, 5, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 7 */, + 6, 7, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 8 */, + 0, 1, 6, 7, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 9 */, + 2, 3, 6, 7, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 10 */, + 0, 1, 2, 3, 6, 7, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 11 */, + 4, 5, 6, 7, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 12 */, + 0, 1, 4, 5, 6, 7, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 13 */, + 2, 3, 4, 5, 6, 7, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 14 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 255, 255, 255, 255, 255, 255, 255, 255 /* 15 */, + 8, 9, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 16 */, + 0, 1, 8, 9, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 17 */, + 2, 3, 8, 9, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 18 */, + 0, 1, 2, 3, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 19 */, + 4, 5, 8, 9, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 20 */, + 0, 1, 4, 5, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 21 */, + 2, 3, 4, 5, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 22 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 255, 255, 255, 255, 255, 255, 255, 255 /* 23 */, + 6, 7, 8, 9, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 24 */, + 0, 1, 6, 7, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 25 */, + 2, 3, 6, 7, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 26 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 255, 255, 255, 255, 255, 255, 255, 255 /* 27 */, + 4, 5, 6, 7, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 28 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 255, 255, 255, 255, 255, 255, 255, 255 /* 29 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 255, 255, 255, 255, 255, 255, 255, 255 /* 30 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 255, 255, 255, 255, 255, 255 /* 31 */, + 10, 11, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 32 */, + 0, 1, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 33 */, + 2, 3, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 34 */, + 0, 1, 2, 3, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 35 */, + 4, 5, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 36 */, + 0, 1, 4, 5, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 37 */, + 2, 3, 4, 5, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 38 */, + 0, 1, 2, 3, 4, 5, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 39 */, + 6, 7, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 40 */, + 0, 1, 6, 7, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 41 */, + 2, 3, 6, 7, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 42 */, + 0, 1, 2, 3, 6, 7, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 43 */, + 4, 5, 6, 7, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 44 */, + 0, 1, 4, 5, 6, 7, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 45 */, + 2, 3, 4, 5, 6, 7, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 46 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 10, 11, 255, 255, 255, 255, 255, 255 /* 47 */, + 8, 9, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 48 */, + 0, 1, 8, 9, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 49 */, + 2, 3, 8, 9, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 50 */, + 0, 1, 2, 3, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 51 */, + 4, 5, 8, 9, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 52 */, + 0, 1, 4, 5, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 53 */, + 2, 3, 4, 5, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 54 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 10, 11, 255, 255, 255, 255, 255, 255 /* 55 */, + 6, 7, 8, 9, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 56 */, + 0, 1, 6, 7, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 57 */, + 2, 3, 6, 7, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 58 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 10, 11, 255, 255, 255, 255, 255, 255 /* 59 */, + 4, 5, 6, 7, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 60 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 10, 11, 255, 255, 255, 255, 255, 255 /* 61 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 10, 11, 255, 255, 255, 255, 255, 255 /* 62 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 255, 255, 255, 255 /* 63 */, + 12, 13, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 64 */, + 0, 1, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 65 */, + 2, 3, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 66 */, + 0, 1, 2, 3, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 67 */, + 4, 5, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 68 */, + 0, 1, 4, 5, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 69 */, + 2, 3, 4, 5, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 70 */, + 0, 1, 2, 3, 4, 5, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 71 */, + 6, 7, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 72 */, + 0, 1, 6, 7, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 73 */, + 2, 3, 6, 7, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 74 */, + 0, 1, 2, 3, 6, 7, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 75 */, + 4, 5, 6, 7, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 76 */, + 0, 1, 4, 5, 6, 7, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 77 */, + 2, 3, 4, 5, 6, 7, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 78 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 12, 13, 255, 255, 255, 255, 255, 255 /* 79 */, + 8, 9, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 80 */, + 0, 1, 8, 9, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 81 */, + 2, 3, 8, 9, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 82 */, + 0, 1, 2, 3, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 83 */, + 4, 5, 8, 9, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 84 */, + 0, 1, 4, 5, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 85 */, + 2, 3, 4, 5, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 86 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 12, 13, 255, 255, 255, 255, 255, 255 /* 87 */, + 6, 7, 8, 9, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 88 */, + 0, 1, 6, 7, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 89 */, + 2, 3, 6, 7, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 90 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 12, 13, 255, 255, 255, 255, 255, 255 /* 91 */, + 4, 5, 6, 7, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 92 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 12, 13, 255, 255, 255, 255, 255, 255 /* 93 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 12, 13, 255, 255, 255, 255, 255, 255 /* 94 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 12, 13, 255, 255, 255, 255 /* 95 */, + 10, 11, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 96 */, + 0, 1, 10, 11, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 97 */, + 2, 3, 10, 11, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 98 */, + 0, 1, 2, 3, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 99 */, + 4, 5, 10, 11, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 100 */, + 0, 1, 4, 5, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 101 */, + 2, 3, 4, 5, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 102 */, + 0, 1, 2, 3, 4, 5, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 103 */, + 6, 7, 10, 11, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 104 */, + 0, 1, 6, 7, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 105 */, + 2, 3, 6, 7, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 106 */, + 0, 1, 2, 3, 6, 7, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 107 */, + 4, 5, 6, 7, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 108 */, + 0, 1, 4, 5, 6, 7, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 109 */, + 2, 3, 4, 5, 6, 7, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 110 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 10, 11, 12, 13, 255, 255, 255, 255 /* 111 */, + 8, 9, 10, 11, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 112 */, + 0, 1, 8, 9, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 113 */, + 2, 3, 8, 9, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 114 */, + 0, 1, 2, 3, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 115 */, + 4, 5, 8, 9, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 116 */, + 0, 1, 4, 5, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 117 */, + 2, 3, 4, 5, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 118 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 10, 11, 12, 13, 255, 255, 255, 255 /* 119 */, + 6, 7, 8, 9, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 120 */, + 0, 1, 6, 7, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 121 */, + 2, 3, 6, 7, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 122 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 10, 11, 12, 13, 255, 255, 255, 255 /* 123 */, + 4, 5, 6, 7, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 124 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 10, 11, 12, 13, 255, 255, 255, 255 /* 125 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 10, 11, 12, 13, 255, 255, 255, 255 /* 126 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 12, 13, 255, 255 /* 127 */, + 14, 15, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 128 */, + 0, 1, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 129 */, + 2, 3, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 130 */, + 0, 1, 2, 3, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 131 */, + 4, 5, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 132 */, + 0, 1, 4, 5, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 133 */, + 2, 3, 4, 5, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 134 */, + 0, 1, 2, 3, 4, 5, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 135 */, + 6, 7, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 136 */, + 0, 1, 6, 7, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 137 */, + 2, 3, 6, 7, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 138 */, + 0, 1, 2, 3, 6, 7, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 139 */, + 4, 5, 6, 7, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 140 */, + 0, 1, 4, 5, 6, 7, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 141 */, + 2, 3, 4, 5, 6, 7, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 142 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 14, 15, 255, 255, 255, 255, 255, 255 /* 143 */, + 8, 9, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 144 */, + 0, 1, 8, 9, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 145 */, + 2, 3, 8, 9, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 146 */, + 0, 1, 2, 3, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 147 */, + 4, 5, 8, 9, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 148 */, + 0, 1, 4, 5, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 149 */, + 2, 3, 4, 5, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 150 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 14, 15, 255, 255, 255, 255, 255, 255 /* 151 */, + 6, 7, 8, 9, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 152 */, + 0, 1, 6, 7, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 153 */, + 2, 3, 6, 7, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 154 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 14, 15, 255, 255, 255, 255, 255, 255 /* 155 */, + 4, 5, 6, 7, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 156 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 14, 15, 255, 255, 255, 255, 255, 255 /* 157 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 14, 15, 255, 255, 255, 255, 255, 255 /* 158 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 14, 15, 255, 255, 255, 255 /* 159 */, + 10, 11, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 160 */, + 0, 1, 10, 11, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 161 */, + 2, 3, 10, 11, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 162 */, + 0, 1, 2, 3, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 163 */, + 4, 5, 10, 11, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 164 */, + 0, 1, 4, 5, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 165 */, + 2, 3, 4, 5, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 166 */, + 0, 1, 2, 3, 4, 5, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 167 */, + 6, 7, 10, 11, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 168 */, + 0, 1, 6, 7, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 169 */, + 2, 3, 6, 7, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 170 */, + 0, 1, 2, 3, 6, 7, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 171 */, + 4, 5, 6, 7, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 172 */, + 0, 1, 4, 5, 6, 7, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 173 */, + 2, 3, 4, 5, 6, 7, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 174 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 10, 11, 14, 15, 255, 255, 255, 255 /* 175 */, + 8, 9, 10, 11, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 176 */, + 0, 1, 8, 9, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 177 */, + 2, 3, 8, 9, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 178 */, + 0, 1, 2, 3, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 179 */, + 4, 5, 8, 9, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 180 */, + 0, 1, 4, 5, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 181 */, + 2, 3, 4, 5, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 182 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 10, 11, 14, 15, 255, 255, 255, 255 /* 183 */, + 6, 7, 8, 9, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 184 */, + 0, 1, 6, 7, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 185 */, + 2, 3, 6, 7, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 186 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 10, 11, 14, 15, 255, 255, 255, 255 /* 187 */, + 4, 5, 6, 7, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 188 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 10, 11, 14, 15, 255, 255, 255, 255 /* 189 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 10, 11, 14, 15, 255, 255, 255, 255 /* 190 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 14, 15, 255, 255 /* 191 */, + 12, 13, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 192 */, + 0, 1, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 193 */, + 2, 3, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 194 */, + 0, 1, 2, 3, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 195 */, + 4, 5, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 196 */, + 0, 1, 4, 5, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 197 */, + 2, 3, 4, 5, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 198 */, + 0, 1, 2, 3, 4, 5, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 199 */, + 6, 7, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 200 */, + 0, 1, 6, 7, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 201 */, + 2, 3, 6, 7, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 202 */, + 0, 1, 2, 3, 6, 7, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 203 */, + 4, 5, 6, 7, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 204 */, + 0, 1, 4, 5, 6, 7, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 205 */, + 2, 3, 4, 5, 6, 7, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 206 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 12, 13, 14, 15, 255, 255, 255, 255 /* 207 */, + 8, 9, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 208 */, + 0, 1, 8, 9, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 209 */, + 2, 3, 8, 9, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 210 */, + 0, 1, 2, 3, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 211 */, + 4, 5, 8, 9, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 212 */, + 0, 1, 4, 5, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 213 */, + 2, 3, 4, 5, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 214 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 12, 13, 14, 15, 255, 255, 255, 255 /* 215 */, + 6, 7, 8, 9, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 216 */, + 0, 1, 6, 7, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 217 */, + 2, 3, 6, 7, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 218 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 12, 13, 14, 15, 255, 255, 255, 255 /* 219 */, + 4, 5, 6, 7, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 220 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 12, 13, 14, 15, 255, 255, 255, 255 /* 221 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 12, 13, 14, 15, 255, 255, 255, 255 /* 222 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 12, 13, 14, 15, 255, 255 /* 223 */, + 10, 11, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 224 */, + 0, 1, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 225 */, + 2, 3, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 226 */, + 0, 1, 2, 3, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 227 */, + 4, 5, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 228 */, + 0, 1, 4, 5, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 229 */, + 2, 3, 4, 5, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 230 */, + 0, 1, 2, 3, 4, 5, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 231 */, + 6, 7, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 232 */, + 0, 1, 6, 7, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 233 */, + 2, 3, 6, 7, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 234 */, + 0, 1, 2, 3, 6, 7, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 235 */, + 4, 5, 6, 7, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 236 */, + 0, 1, 4, 5, 6, 7, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 237 */, + 2, 3, 4, 5, 6, 7, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 238 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 10, 11, 12, 13, 14, 15, 255, 255 /* 239 */, + 8, 9, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 240 */, + 0, 1, 8, 9, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 241 */, + 2, 3, 8, 9, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 242 */, + 0, 1, 2, 3, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 243 */, + 4, 5, 8, 9, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 244 */, + 0, 1, 4, 5, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 245 */, + 2, 3, 4, 5, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 246 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 10, 11, 12, 13, 14, 15, 255, 255 /* 247 */, + 6, 7, 8, 9, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 248 */, + 0, 1, 6, 7, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 249 */, + 2, 3, 6, 7, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 250 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 10, 11, 12, 13, 14, 15, 255, 255 /* 251 */, + 4, 5, 6, 7, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 252 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 10, 11, 12, 13, 14, 15, 255, 255 /* 253 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 10, 11, 12, 13, 14, 15, 255, 255 /* 254 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 12, 13, 14, 15 /* 255 */, +}; + +#else /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_KEYPAIR_API && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +MLD_EMPTY_CU(aarch64_rej_uniform_eta_table) + +#endif /* !(MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_NO_KEYPAIR_API && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mldsa/src/native/aarch64/src/rej_uniform_table.c b/cbits/mldsa/src/native/aarch64/src/rej_uniform_table.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/aarch64/src/rej_uniform_table.c @@ -0,0 +1,63 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mldsa-native repository. + * Do not modify it directly. + */ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_AARCH64) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +#include "arith_native_aarch64.h" + +/* + * Lookup table used by rejection sampling of the public matrix. + * See autogen for details. + */ +MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint8_t + mld_rej_uniform_table[256] = { + 255, 255, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 0 */, + 0, 1, 2, 3, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 1 */, + 4, 5, 6, 7, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 2 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 255, 255, 255, 255, 255, 255, 255, 255 /* 3 */, + 8, 9, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 4 */, + 0, 1, 2, 3, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 5 */, + 4, 5, 6, 7, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 6 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 255, 255, 255, 255 /* 7 */, + 12, 13, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 8 */, + 0, 1, 2, 3, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 9 */, + 4, 5, 6, 7, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 10 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 12, 13, 14, 15, 255, 255, 255, 255 /* 11 */, + 8, 9, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 12 */, + 0, 1, 2, 3, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 13 */, + 4, 5, 6, 7, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 14 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 12, 13, 14, 15 /* 15 */, +}; + +#else /* MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +MLD_EMPTY_CU(aarch64_rej_uniform_table) + +#endif /* !(MLD_ARITH_BACKEND_AARCH64 && !MLD_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mldsa/src/native/api.h b/cbits/mldsa/src/native/api.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/api.h @@ -0,0 +1,617 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_NATIVE_API_H +#define MLD_NATIVE_API_H +/* + * Native arithmetic interface + * + * This header is primarily for documentation purposes. + * It should not be included by backend implementations. + * + * To ensure consistency with backends, the header will be + * included automatically after inclusion of the active + * backend, to ensure consistency of function signatures, + * and run sanity checks. + */ + +#include "../cbmc.h" +#include "../common.h" + +/* Backends must return MLD_NATIVE_FUNC_SUCCESS upon success. */ +#define MLD_NATIVE_FUNC_SUCCESS (0) +/* Backends may return MLD_NATIVE_FUNC_FALLBACK to signal to the frontend that + * the target/parameters are unsupported; typically, this would be because of + * dependencies on CPU features not detected on the host CPU. In this case, + * the frontend falls back to the default C implementation. + * + * IMPORTANT: Backend implementations must ensure that the decision of whether + * to fallback (return MLD_NATIVE_FUNC_FALLBACK) or not must never depend on + * the input data itself. Fallback decisions may only depend on system + * capabilities (e.g., CPU features) and, where present, length information. + * This requirement applies to all backend functions to maintain constant-time + * properties. + */ +#define MLD_NATIVE_FUNC_FALLBACK (-1) + +/* Absolute exclusive upper bound for the output of fqmul. + * + * NOTE: This is the same bound as in poly.h and has to be kept + * in sync. */ +#define MLD_FQMUL_BOUND ((5 * MLDSA_Q + 3) / 4) + +/* Bound on absolute value of coefficients after NTT. + * + * NOTE: This is the same bound as in poly.h and has to be kept + * in sync. */ +#define MLD_NTT_BOUND (9 * MLD_FQMUL_BOUND) + +/* Absolute exclusive upper bound for the output of the inverse NTT + * + * NOTE: This is the same bound as in poly.h and has to be kept + * in sync. */ +#define MLD_INTT_BOUND MLDSA_Q + +/* Absolute bound for range of mld_reduce32() + * + * NOTE: This is the same bound as in reduce.h and has to be kept + * in sync. */ +/* check-magic: 6283009 == (MLD_REDUCE32_DOMAIN_MAX - 255 * MLDSA_Q + 1) */ +#define MLD_REDUCE32_RANGE_MAX 6283009 +/* + * This is the C<->native interface allowing for the drop-in of + * native code for performance-critical arithmetic components of ML-DSA. + * + * A _backend_ is a specific implementation of (part of) this interface. + * + * To add a function to a backend, define MLD_USE_NATIVE_XXX and + * implement `static inline xxx(...)` in the profile header. + */ + +/* + * Those functions are meant to be trivial wrappers around the chosen native + * implementation. The are static inline to avoid unnecessary calls. + * The macro before each declaration controls whether a native + * implementation is present. + */ + +#if defined(MLD_USE_NATIVE_NTT) +/** + * Computes negacyclic number-theoretic transform (NTT) of a polynomial + * in place. + * + * The input polynomial is assumed to be in normal order. The output + * polynomial is in bitreversed order. + * + * @param[in,out] p Pointer to in/output polynomial. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_ntt_native(int32_t p[MLDSA_N]) +__contract__( + requires(memory_no_alias(p, sizeof(int32_t) * MLDSA_N)) + requires(array_abs_bound(p, 0, MLDSA_N, MLDSA_Q)) + assigns(memory_slice(p, sizeof(int32_t) * MLDSA_N)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(p, 0, MLDSA_N, MLD_NTT_BOUND)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_abs_bound(p, 0, MLDSA_N, MLDSA_Q)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(p, MLDSA_N)) +); +#endif /* MLD_USE_NATIVE_NTT */ + + +#if defined(MLD_USE_NATIVE_NTT_CUSTOM_ORDER) +/* + * This must only be set if NTT and INTT have native implementations + * that are adapted to the custom order. + */ +#if !defined(MLD_USE_NATIVE_NTT) || !defined(MLD_USE_NATIVE_INTT) +#error \ + "Invalid native profile: MLD_USE_NATIVE_NTT_CUSTOM_ORDER can only be \ +set if there are native implementations for NTT and INTT." +#endif + +/** + * When MLD_USE_NATIVE_NTT_CUSTOM_ORDER is defined, convert a polynomial in + * NTT domain from bitreversed order to the custom order output by the native + * NTT. + * + * This must only be defined if there is native code for both the NTT and + * INTT. + * + * @param[in,out] p Pointer to in/output polynomial. + */ +static MLD_INLINE void mld_poly_permute_bitrev_to_custom(int32_t p[MLDSA_N]) +__contract__( + /* We don't specify that this should be a permutation, but only + * that it does not change the bound established at the end of + * mld_polyvec_matrix_expand. + */ + requires(memory_no_alias(p, sizeof(int32_t) * MLDSA_N)) + requires(array_bound(p, 0, MLDSA_N, 0, MLDSA_Q)) + assigns(memory_slice(p, sizeof(int32_t) * MLDSA_N)) + ensures(array_bound(p, 0, MLDSA_N, 0, MLDSA_Q))); +#endif /* MLD_USE_NATIVE_NTT_CUSTOM_ORDER */ + + +#if defined(MLD_USE_NATIVE_INTT) +/** + * Computes inverse of negacyclic number-theoretic transform (NTT) of a + * polynomial in place. + * + * The input polynomial is in bitreversed order. The output polynomial is + * assumed to be in normal order. + * + * @param[in,out] p Pointer to in/output polynomial. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_intt_native(int32_t p[MLDSA_N]) +__contract__( + requires(memory_no_alias(p, sizeof(int32_t) * MLDSA_N)) + requires(array_abs_bound(p, 0, MLDSA_N, MLDSA_Q)) + assigns(memory_slice(p, sizeof(int32_t) * MLDSA_N)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(p, 0, MLDSA_N, MLD_INTT_BOUND)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_abs_bound(p, 0, MLDSA_N, MLDSA_Q)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(p, MLDSA_N)) +); +#endif /* MLD_USE_NATIVE_INTT */ + +#if defined(MLD_USE_NATIVE_REJ_UNIFORM) +/** + * Run rejection sampling on uniform random bytes to generate uniform random + * integers in [0, MLDSA_Q-1]. + * + * @param[out] r Pointer to output buffer. + * @param len Requested number of 32-bit integers (uniform mod + * MLDSA_Q). + * @param[in] buf Pointer to input buffer (assumed to be uniform random + * bytes). + * @param buflen Length of input buffer in bytes. + * + * @return - MLD_NATIVE_FUNC_FALLBACK if the native implementation does not + * support the input lengths. + * - Otherwise, the non-negative number of sampled 32-bit integers + * (at most len). + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_rej_uniform_native(int32_t *r, unsigned len, + const uint8_t *buf, + unsigned buflen) +__contract__( + requires(len <= MLDSA_N) + requires(buflen <= ( 5 * 168) && buflen % 3 == 0) + requires(memory_no_alias(r, sizeof(int32_t) * len)) + requires(memory_no_alias(buf, buflen)) + assigns(memory_slice(r, sizeof(int32_t) * len)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || (0 <= return_value && return_value <= len)) + ensures((return_value != MLD_NATIVE_FUNC_FALLBACK) ==> array_bound(r, 0, (unsigned) return_value, 0, MLDSA_Q)) +); +#endif /* MLD_USE_NATIVE_REJ_UNIFORM */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +#if defined(MLD_USE_NATIVE_REJ_UNIFORM_ETA2) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 2 +/** + * Run rejection sampling on uniform random bytes to generate uniform random + * integers in [-2, +2]. + * + * @param[out] r Pointer to output buffer. + * @param len Requested number of 32-bit integers (uniform in + * [-2, +2]). + * @param[in] buf Pointer to input buffer (assumed to be uniform random + * bytes). + * @param buflen Length of input buffer in bytes. + * + * @return - MLD_NATIVE_FUNC_FALLBACK if the native implementation does not + * support the input lengths. + * - Otherwise, the non-negative number of sampled 32-bit integers + * (at most len). + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_rej_uniform_eta2_native(int32_t *r, unsigned len, + const uint8_t *buf, + unsigned buflen) +__contract__( + requires(len <= MLDSA_N) + requires(buflen <= (2 * 136)) + requires(memory_no_alias(r, sizeof(int32_t) * len)) + requires(memory_no_alias(buf, buflen)) + assigns(memory_slice(r, sizeof(int32_t) * len)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || (0 <= return_value && return_value <= len)) + /* check-magic: 3 == 2 + 1 (decl gated on MLDSA_ETA == 2) */ + ensures((return_value != MLD_NATIVE_FUNC_FALLBACK) ==> (array_abs_bound(r, 0, return_value, 3))) +); +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 2 */ +#endif /* MLD_USE_NATIVE_REJ_UNIFORM_ETA2 */ + +#if defined(MLD_USE_NATIVE_REJ_UNIFORM_ETA4) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 4 +/** + * Run rejection sampling on uniform random bytes to generate uniform random + * integers in [-4, +4]. + * + * @param[out] r Pointer to output buffer. + * @param len Requested number of 32-bit integers (uniform in + * [-4, +4]). + * @param[in] buf Pointer to input buffer (assumed to be uniform random + * bytes). + * @param buflen Length of input buffer in bytes. + * + * @return - MLD_NATIVE_FUNC_FALLBACK if the native implementation does not + * support the input lengths. + * - Otherwise, the non-negative number of sampled 32-bit integers + * (at most len). + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_rej_uniform_eta4_native(int32_t *r, unsigned len, + const uint8_t *buf, + unsigned buflen) +__contract__( + requires(len <= MLDSA_N) + requires(buflen <= (2 * 136)) + requires(memory_no_alias(r, sizeof(int32_t) * len)) + requires(memory_no_alias(buf, buflen)) + assigns(memory_slice(r, sizeof(int32_t) * len)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || (0 <= return_value && return_value <= len)) + /* check-magic: 5 == 4 + 1 (decl gated on MLDSA_ETA == 4) */ + ensures((return_value != MLD_NATIVE_FUNC_FALLBACK) ==> (array_abs_bound(r, 0, return_value, 5))) +); +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 4 */ +#endif /* MLD_USE_NATIVE_REJ_UNIFORM_ETA4 */ +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +#if defined(MLD_USE_NATIVE_POLY_DECOMPOSE_32) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) +/** + * Native implementation of poly_decompose for GAMMA2 = (MLDSA_Q-1)/32. + * + * For all coefficients c of the input polynomial, compute high and low bits + * c0, c1 such c mod MLDSA_Q = c1*(2*GAMMA2) + c0 with + * -(2*GAMMA2)/2 < c0 <= (2*GAMMA2)/2 except c1 = (MLDSA_Q-1)/(2*GAMMA2) where + * we set c1 = 0 and -(2*GAMMA2)/2 <= c0 = c mod MLDSA_Q - MLDSA_Q < 0. + * Assumes coefficients to be standard representatives. + * + * @param[out] a1 Output polynomial with coefficients c1. + * @param[in,out] a0 Input/output polynomial. Output has coefficients c0. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_decompose_32_native(int32_t *a1, int32_t *a0) +__contract__( + requires(memory_no_alias(a1, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a0, sizeof(int32_t) * MLDSA_N)) + requires(array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q)) + assigns(memory_slice(a1, sizeof(int32_t) * MLDSA_N)) + assigns(memory_slice(a0, sizeof(int32_t) * MLDSA_N)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_bound(a1, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2))) + ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(a0, 0, MLDSA_N, MLDSA_GAMMA2+1)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(a0, MLDSA_N)) +); +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87 */ +#endif /* MLD_USE_NATIVE_POLY_DECOMPOSE_32 */ + +#if defined(MLD_USE_NATIVE_POLY_DECOMPOSE_88) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44 +/** + * Native implementation of poly_decompose for GAMMA2 = (MLDSA_Q-1)/88. + * + * For all coefficients c of the input polynomial, compute high and low bits + * c0, c1 such c mod MLDSA_Q = c1*(2*GAMMA2) + c0 with + * -(2*GAMMA2)/2 < c0 <= (2*GAMMA2)/2 except c1 = (MLDSA_Q-1)/(2*GAMMA2) where + * we set c1 = 0 and -(2*GAMMA2)/2 <= c0 = c mod MLDSA_Q - MLDSA_Q < 0. + * Assumes coefficients to be standard representatives. + * + * @param[out] a1 Output polynomial with coefficients c1. + * @param[in,out] a0 Input/output polynomial. Output has coefficients c0. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_decompose_88_native(int32_t *a1, int32_t *a0) +__contract__( + requires(memory_no_alias(a1, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a0, sizeof(int32_t) * MLDSA_N)) + requires(array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q)) + assigns(memory_slice(a1, sizeof(int32_t) * MLDSA_N)) + assigns(memory_slice(a0, sizeof(int32_t) * MLDSA_N)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_bound(a1, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2))) + ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(a0, 0, MLDSA_N, MLDSA_GAMMA2+1)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(a0, MLDSA_N)) +); +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \ + */ +#endif /* MLD_USE_NATIVE_POLY_DECOMPOSE_88 */ +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if defined(MLD_USE_NATIVE_POLY_CADDQ) +/** + * For all coefficients of in/out polynomial add Q if coefficient is negative. + * + * @param[in,out] a Pointer to input/output polynomial. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_caddq_native(int32_t a[MLDSA_N]) +__contract__( + requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N)) + requires(array_abs_bound(a, 0, MLDSA_N, MLDSA_Q)) + assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_bound(a, 0, MLDSA_N, 0, MLDSA_Q)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_abs_bound(a, 0, MLDSA_N, MLDSA_Q)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(a, MLDSA_N)) +); +#endif /* MLD_USE_NATIVE_POLY_CADDQ */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +#if defined(MLD_USE_NATIVE_POLY_USE_HINT_32) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) +/** + * Native implementation of poly_use_hint for GAMMA2 = (MLDSA_Q-1)/32. + * + * Use hint h to correct the high bits of a in-place. + * + * @param[in,out] a Input/output polynomial. + * @param[in] h Hint polynomial. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_use_hint_32_native(int32_t *a, const int32_t *h) +__contract__( + requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(h, sizeof(int32_t) * MLDSA_N)) + requires(array_bound(a, 0, MLDSA_N, 0, MLDSA_Q)) + requires(array_bound(h, 0, MLDSA_N, 0, 2)) + assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_bound(a, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2))) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(a, MLDSA_N)) +); +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87 */ +#endif /* MLD_USE_NATIVE_POLY_USE_HINT_32 */ + +#if defined(MLD_USE_NATIVE_POLY_USE_HINT_88) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44 +/** + * Native implementation of poly_use_hint for GAMMA2 = (MLDSA_Q-1)/88. + * + * Use hint h to correct the high bits of a in-place. + * + * @param[in,out] a Input/output polynomial. + * @param[in] h Hint polynomial. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_use_hint_88_native(int32_t *a, const int32_t *h) +__contract__( + requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(h, sizeof(int32_t) * MLDSA_N)) + requires(array_bound(a, 0, MLDSA_N, 0, MLDSA_Q)) + requires(array_bound(h, 0, MLDSA_N, 0, 2)) + assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_bound(a, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2))) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(a, MLDSA_N)) +); +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \ + */ +#endif /* MLD_USE_NATIVE_POLY_USE_HINT_88 */ +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if defined(MLD_USE_NATIVE_POLY_CHKNORM) +/** + * Check infinity norm of polynomial against given bound. Assumes input + * coefficients were reduced by mld_reduce32(). + * + * @param[in] a Pointer to polynomial. + * @param B Norm bound, which must be in the range + * 0 .. MLDSA_Q - MLD_REDUCE32_RANGE_MAX inclusive. + * + * @return - MLD_NATIVE_FUNC_FALLBACK if the target CPU cannot support a + * native implementation of this function. + * - MLD_NATIVE_FUNC_SUCCESS if the infinity norm is strictly smaller + * than B. + * - 1 otherwise. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_chknorm_native(const int32_t *a, int32_t B) +__contract__( + requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N)) + requires(0 <= B && B <= MLDSA_Q - MLD_REDUCE32_RANGE_MAX) + requires(array_bound(a, 0, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == 0 || + return_value == 1) + ensures((return_value != MLD_NATIVE_FUNC_FALLBACK) ==> + ((return_value == 0) == array_abs_bound(a, 0, MLDSA_N, B))) +); +#endif /* MLD_USE_NATIVE_POLY_CHKNORM */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +#if defined(MLD_USE_NATIVE_POLYZ_UNPACK_17) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44 +/** + * Native implementation of polyz_unpack for GAMMA1 = 2^17. + * + * Unpack polynomial z with coefficients in + * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1]. + * + * @param[out] r Pointer to output polynomial. + * @param[in] a Byte array with bit-packed polynomial. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_polyz_unpack_17_native(int32_t *r, const uint8_t *a) +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a, MLDSA_POLYZ_PACKEDBYTES)) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_bound(r, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(r, MLDSA_N)) +); +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \ + */ +#endif /* MLD_USE_NATIVE_POLYZ_UNPACK_17 */ + +#if defined(MLD_USE_NATIVE_POLYZ_UNPACK_19) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) +/** + * Native implementation of polyz_unpack for GAMMA1 = 2^19. + * + * Unpack polynomial z with coefficients in + * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1]. + * + * @param[out] r Pointer to output polynomial. + * @param[in] a Byte array with bit-packed polynomial. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_polyz_unpack_19_native(int32_t *r, const uint8_t *a) +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a, MLDSA_POLYZ_PACKEDBYTES)) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_bound(r, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(r, MLDSA_N)) +); +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87 */ +#endif /* MLD_USE_NATIVE_POLYZ_UNPACK_19 */ +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \ + defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +#if defined(MLD_USE_NATIVE_POINTWISE_MONTGOMERY) +/** + * Pointwise multiplication of polynomials in NTT domain with Montgomery + * reduction. Destructive in the first argument. + * + * Computes a[i] = a[i] * b[i] * R^(-1) mod MLDSA_Q for all i, where R = 2^32. + * + * @param[in,out] a First input/output polynomial. + * @param[in] b Second input polynomial. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_pointwise_montgomery_native( + int32_t a[MLDSA_N], const int32_t b[MLDSA_N]) +__contract__( + requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(b, sizeof(int32_t) * MLDSA_N)) + requires(array_abs_bound(a, 0, MLDSA_N, MLD_NTT_BOUND)) + requires(array_abs_bound(b, 0, MLDSA_N, MLD_NTT_BOUND)) + assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(a, 0, MLDSA_N, MLDSA_Q)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_abs_bound(a, 0, MLDSA_N, MLD_NTT_BOUND)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_abs_bound(b, 0, MLDSA_N, MLD_NTT_BOUND)) +); +#endif /* MLD_USE_NATIVE_POINTWISE_MONTGOMERY */ +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API || \ + MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +#if defined(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 4 +/** + * Native implementation of polyvecl_pointwise_acc_montgomery for MLDSA_L = 4. + * + * Pointwise multiply vectors of polynomials of length MLDSA_L, multiply + * resulting vector by 2^{-32} and add (accumulate) polynomials in it. + * Input/output vectors are in NTT domain representation. + * + * @param[out] w Output polynomial. + * @param[in] u First input vector. + * @param[in] v Second input vector. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l4_native( + int32_t w[MLDSA_N], const int32_t u[4][MLDSA_N], + const int32_t v[4][MLDSA_N]) +__contract__( + requires(memory_no_alias(w, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(u, sizeof(int32_t) * 4 * MLDSA_N)) + requires(memory_no_alias(v, sizeof(int32_t) * 4 * MLDSA_N)) + requires(forall(l0, 0, 4, + array_bound(u[l0], 0, MLDSA_N, 0, MLDSA_Q))) + requires(forall(l1, 0, 4, + array_abs_bound(v[l1], 0, MLDSA_N, MLD_NTT_BOUND))) + assigns(memory_slice(w, sizeof(int32_t) * MLDSA_N)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(w, 0, MLDSA_N, MLDSA_Q)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(w, MLDSA_N)) +); +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 4 */ +#endif /* MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4 */ + +#if defined(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 5 +/** + * Native implementation of polyvecl_pointwise_acc_montgomery for MLDSA_L = 5. + * + * Pointwise multiply vectors of polynomials of length MLDSA_L, multiply + * resulting vector by 2^{-32} and add (accumulate) polynomials in it. + * Input/output vectors are in NTT domain representation. + * + * @param[out] w Output polynomial. + * @param[in] u First input vector. + * @param[in] v Second input vector. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l5_native( + int32_t w[MLDSA_N], const int32_t u[5][MLDSA_N], + const int32_t v[5][MLDSA_N]) +__contract__( + requires(memory_no_alias(w, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(u, sizeof(int32_t) * 5 * MLDSA_N)) + requires(memory_no_alias(v, sizeof(int32_t) * 5 * MLDSA_N)) + requires(forall(l0, 0, 5, + array_bound(u[l0], 0, MLDSA_N, 0, MLDSA_Q))) + requires(forall(l1, 0, 5, + array_abs_bound(v[l1], 0, MLDSA_N, MLD_NTT_BOUND))) + assigns(memory_slice(w, sizeof(int32_t) * MLDSA_N)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(w, 0, MLDSA_N, MLDSA_Q)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(w, MLDSA_N)) +); +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 5 */ +#endif /* MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5 */ + +#if defined(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 7 +/** + * Native implementation of polyvecl_pointwise_acc_montgomery for MLDSA_L = 7. + * + * Pointwise multiply vectors of polynomials of length MLDSA_L, multiply + * resulting vector by 2^{-32} and add (accumulate) polynomials in it. + * Input/output vectors are in NTT domain representation. + * + * @param[out] w Output polynomial. + * @param[in] u First input vector. + * @param[in] v Second input vector. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l7_native( + int32_t w[MLDSA_N], const int32_t u[7][MLDSA_N], + const int32_t v[7][MLDSA_N]) +__contract__( + requires(memory_no_alias(w, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(u, sizeof(int32_t) * 7 * MLDSA_N)) + requires(memory_no_alias(v, sizeof(int32_t) * 7 * MLDSA_N)) + requires(forall(l0, 0, 7, + array_bound(u[l0], 0, MLDSA_N, 0, MLDSA_Q))) + requires(forall(l1, 0, 7, + array_abs_bound(v[l1], 0, MLDSA_N, MLD_NTT_BOUND))) + assigns(memory_slice(w, sizeof(int32_t) * MLDSA_N)) + ensures(return_value == MLD_NATIVE_FUNC_FALLBACK || return_value == MLD_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLD_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(w, 0, MLDSA_N, MLDSA_Q)) + ensures((return_value == MLD_NATIVE_FUNC_FALLBACK) ==> array_unchanged(w, MLDSA_N)) +); +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 7 */ +#endif /* MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7 */ + +#endif /* !MLD_NATIVE_API_H */ diff --git a/cbits/mldsa/src/native/meta.h b/cbits/mldsa/src/native/meta.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/meta.h @@ -0,0 +1,24 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_NATIVE_META_H +#define MLD_NATIVE_META_H + +/* + * Default arithmetic backend + */ +#include "../sys.h" + +#ifdef MLD_SYS_AARCH64_NEON +#include "aarch64/meta.h" +#endif + +/* The x86_64 backend requires toolchain support for the SysV ABI */ +#if defined(MLD_SYS_X86_64_AVX2) && defined(MLD_SYSV_ABI_SUPPORTED) +#include "x86_64/meta.h" +#endif + +#endif /* !MLD_NATIVE_META_H */ diff --git a/cbits/mldsa/src/native/x86_64/meta.h b/cbits/mldsa/src/native/x86_64/meta.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/meta.h @@ -0,0 +1,323 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_NATIVE_X86_64_META_H +#define MLD_NATIVE_X86_64_META_H + +/* Identifier for this backend so that source and assembly files + * in the build can be appropriately guarded. */ +#define MLD_ARITH_BACKEND_X86_64_DEFAULT + +#define MLD_USE_NATIVE_NTT_CUSTOM_ORDER +#define MLD_USE_NATIVE_NTT +#define MLD_USE_NATIVE_INTT +#define MLD_USE_NATIVE_REJ_UNIFORM +#define MLD_USE_NATIVE_REJ_UNIFORM_ETA2 +#define MLD_USE_NATIVE_REJ_UNIFORM_ETA4 +#define MLD_USE_NATIVE_POLY_DECOMPOSE_32 +#define MLD_USE_NATIVE_POLY_DECOMPOSE_88 +#define MLD_USE_NATIVE_POLY_CADDQ +#define MLD_USE_NATIVE_POLY_USE_HINT_32 +#define MLD_USE_NATIVE_POLY_USE_HINT_88 +#define MLD_USE_NATIVE_POLY_CHKNORM +#define MLD_USE_NATIVE_POLYZ_UNPACK_17 +#define MLD_USE_NATIVE_POLYZ_UNPACK_19 +#define MLD_USE_NATIVE_POINTWISE_MONTGOMERY +#define MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4 +#define MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5 +#define MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7 + +#if !defined(__ASSEMBLER__) +#include +#include "../../common.h" +#include "../api.h" +#include "src/arith_native_x86_64.h" + +static MLD_INLINE void mld_poly_permute_bitrev_to_custom(int32_t data[MLDSA_N]) +{ + if (mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + mld_nttunpack_avx2_asm(data); + } +} + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_ntt_native(int32_t data[MLDSA_N]) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + + mld_ntt_avx2_asm(data, mld_qdata); + return MLD_NATIVE_FUNC_SUCCESS; +} + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_intt_native(int32_t data[MLDSA_N]) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_invntt_avx2_asm(data, mld_qdata); + return MLD_NATIVE_FUNC_SUCCESS; +} + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_rej_uniform_native(int32_t *r, unsigned len, + const uint8_t *buf, + unsigned buflen) +{ + /* AVX2 implementation assumes specific buffer lengths */ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2) || len != MLDSA_N || + buflen != MLD_AVX2_REJ_UNIFORM_BUFLEN) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + + /* Safety: outlen is at most MLDSA_N and, hence, this cast is safe. */ + return (int)mld_rej_uniform_avx2_asm(r, buf, mld_rej_uniform_table); +} + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 2 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_rej_uniform_eta2_native(int32_t *r, unsigned len, + const uint8_t *buf, + unsigned buflen) +{ + unsigned int outlen; + /* AVX2 implementation assumes specific buffer lengths */ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2) || len != MLDSA_N || + buflen != MLD_AVX2_REJ_UNIFORM_ETA2_BUFLEN) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + + /* Constant time: Inputs and outputs to this function are secret. + * It is safe to leak which coefficients are accepted/rejected. + * The assembly implementation must not leak any other information about the + * accepted coefficients. Constant-time testing cannot cover this, and we + * hence have to manually verify the assembly. + * We declassify prior the input data and mark the outputs as secret. + */ + MLD_CT_TESTING_DECLASSIFY(buf, buflen); + outlen = mld_rej_uniform_eta2_avx2_asm(r, buf, mld_rej_uniform_table); + MLD_CT_TESTING_SECRET(r, sizeof(int32_t) * outlen); + /* Safety: outlen is at most MLDSA_N and, hence, this cast is safe. */ + return (int)outlen; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 2 */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_ETA == 4 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_rej_uniform_eta4_native(int32_t *r, unsigned len, + const uint8_t *buf, + unsigned buflen) +{ + unsigned int outlen; + /* AVX2 implementation assumes specific buffer lengths */ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2) || len != MLDSA_N || + buflen != MLD_AVX2_REJ_UNIFORM_ETA4_BUFLEN) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + + /* Constant time: Inputs and outputs to this function are secret. + * It is safe to leak which coefficients are accepted/rejected. + * The assembly implementation must not leak any other information about the + * accepted coefficients. Constant-time testing cannot cover this, and we + * hence have to manually verify the assembly. + * We declassify prior the input data and mark the outputs as secret. + */ + MLD_CT_TESTING_DECLASSIFY(buf, buflen); + outlen = mld_rej_uniform_eta4_avx2_asm(r, buf, mld_rej_uniform_table); + MLD_CT_TESTING_SECRET(r, sizeof(int32_t) * outlen); + /* Safety: outlen is at most MLDSA_N and, hence, this cast is safe. */ + return (int)outlen; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_ETA == 4 */ +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_decompose_32_native(int32_t *a1, int32_t *a0) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_poly_decompose_32_avx2_asm(a1, a0); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87 */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_decompose_88_native(int32_t *a1, int32_t *a0) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_poly_decompose_88_avx2_asm(a1, a0); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \ + */ +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_caddq_native(int32_t a[MLDSA_N]) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_poly_caddq_avx2_asm(a); + return MLD_NATIVE_FUNC_SUCCESS; +} + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_use_hint_32_native(int32_t *a, const int32_t *h) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_poly_use_hint_32_avx2_asm(a, h); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87 */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_use_hint_88_native(int32_t *a, const int32_t *h) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_poly_use_hint_88_avx2_asm(a, h); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \ + */ +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_chknorm_native(const int32_t *a, int32_t B) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + return mld_poly_chknorm_avx2_asm(a, B); +} + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_polyz_unpack_17_native(int32_t *r, const uint8_t *a) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_polyz_unpack_17_avx2_asm(r, a); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \ + */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_polyz_unpack_19_native(int32_t *r, const uint8_t *a) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_polyz_unpack_19_avx2_asm(r, a); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87 */ +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \ + defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_poly_pointwise_montgomery_native( + int32_t a[MLDSA_N], const int32_t b[MLDSA_N]) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_pointwise_avx2_asm(a, b, mld_qdata); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API || \ + MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 4 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l4_native( + int32_t w[MLDSA_N], const int32_t u[4][MLDSA_N], + const int32_t v[4][MLDSA_N]) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_pointwise_acc_l4_avx2_asm(w, u, v, mld_qdata); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 4 */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 5 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l5_native( + int32_t w[MLDSA_N], const int32_t u[5][MLDSA_N], + const int32_t v[5][MLDSA_N]) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_pointwise_acc_l5_avx2_asm(w, u, v, mld_qdata); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 5 */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 7 +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_polyvecl_pointwise_acc_montgomery_l7_native( + int32_t w[MLDSA_N], const int32_t u[7][MLDSA_N], + const int32_t v[7][MLDSA_N]) +{ + if (!mld_sys_check_capability(MLD_SYS_CAP_X86_64_AVX2)) + { + return MLD_NATIVE_FUNC_FALLBACK; + } + mld_pointwise_acc_l7_avx2_asm(w, u, v, mld_qdata); + return MLD_NATIVE_FUNC_SUCCESS; +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 7 */ + +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLD_NATIVE_X86_64_META_H */ diff --git a/cbits/mldsa/src/native/x86_64/src/arith_native_x86_64.h b/cbits/mldsa/src/native/x86_64/src/arith_native_x86_64.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/arith_native_x86_64.h @@ -0,0 +1,330 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLD_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H +#define MLD_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H +#include "../../../common.h" + +#include "consts.h" + +#define MLD_AVX2_REJ_UNIFORM_BUFLEN \ + (5 * 168) /* REJ_UNIFORM_NBLOCKS * SHAKE128_RATE */ + + +/* + * Sampling 256 coefficients mod 15 using rejection sampling from 4 bits. + * Expected number of required bytes: (256 * (16/15))/2 = 136.5 bytes. + * We sample 1 block (=136 bytes) of SHAKE256_RATE output initially. + * Sampling 2 blocks initially results in slightly worse performance. + */ +#define MLD_AVX2_REJ_UNIFORM_ETA2_BUFLEN (1 * 136) + +/* + * Sampling 256 coefficients mod 9 using rejection sampling from 4 bits. + * Expected number of required bytes: (256 * (16/9))/2 = 227.5 bytes. + * We sample 2 blocks (=272 bytes) of SHAKE256_RATE output initially. + */ +#define MLD_AVX2_REJ_UNIFORM_ETA4_BUFLEN (2 * 136) + +#define mld_rej_uniform_table MLD_NAMESPACE(mld_rej_uniform_table) +MLD_INTERNAL_DATA_DECLARATION const uint8_t mld_rej_uniform_table[256][8]; + +#define mld_ntt_avx2_asm MLD_NAMESPACE(ntt_avx2_asm) +MLD_SYSV_ABI +void mld_ntt_avx2_asm(int32_t *r, const int32_t *qdata) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_ntt_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + /* check-magic: 8380417 == MLDSA_Q */ + requires(array_abs_bound(r, 0, MLDSA_N, 8380417)) + requires(qdata == mld_qdata) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + /* check-magic: off */ + ensures(array_abs_bound(r, 0, MLDSA_N, 42035262)) + /* check-magic: on */ +); + +#define mld_invntt_avx2_asm MLD_NAMESPACE(invntt_avx2_asm) +MLD_SYSV_ABI +void mld_invntt_avx2_asm(int32_t *r, const int32_t *qdata) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_intt_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(array_abs_bound(r, 0, MLDSA_N, 8380417)) + requires(qdata == mld_qdata) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + /* check-magic: off */ + ensures(array_abs_bound(r, 0, MLDSA_N, 6285313)) + /* check-magic: on */ +); + +#define mld_nttunpack_avx2_asm MLD_NAMESPACE(nttunpack_avx2_asm) +MLD_SYSV_ABI +void mld_nttunpack_avx2_asm(int32_t *r) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_nttunpack_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(array_abs_bound(r, 0, MLDSA_N, 8380417)) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + /* Output is a permutation of input: every output coefficient + * is some input coefficient */ + ensures(forall(i, 0, MLDSA_N, exists(j, 0, MLDSA_N, + r[i] == old(*(int32_t (*)[MLDSA_N])r)[j]))) +); + +#define mld_rej_uniform_avx2_asm MLD_NAMESPACE(rej_uniform_avx2_asm) +/* This contract must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_rej_uniform_avx2_asm.ml */ +MLD_MUST_CHECK_RETURN_VALUE MLD_SYSV_ABI +unsigned mld_rej_uniform_avx2_asm( + int32_t *r, const uint8_t buf[MLD_AVX2_REJ_UNIFORM_BUFLEN], + const uint8_t table[256][8]) +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(buf, 840)) + requires(table == mld_rej_uniform_table) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(return_value <= MLDSA_N) + ensures(array_bound(r, 0, return_value, 0, MLDSA_Q)) +); + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +#define mld_rej_uniform_eta2_avx2_asm MLD_NAMESPACE(rej_uniform_eta2_avx2_asm) +/* This contract must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_rej_uniform_eta2_avx2_asm.ml */ +MLD_MUST_CHECK_RETURN_VALUE MLD_SYSV_ABI +unsigned mld_rej_uniform_eta2_avx2_asm( + int32_t *r, const uint8_t buf[MLD_AVX2_REJ_UNIFORM_ETA2_BUFLEN], + const uint8_t table[256][8]) +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(buf, MLD_AVX2_REJ_UNIFORM_ETA2_BUFLEN)) + requires(table == mld_rej_uniform_table) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(return_value <= MLDSA_N) + ensures(array_abs_bound(r, 0, return_value, 3)) +); + +#define mld_rej_uniform_eta4_avx2_asm MLD_NAMESPACE(rej_uniform_eta4_avx2_asm) +/* This contract must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_rej_uniform_eta4_avx2_asm.ml */ +MLD_MUST_CHECK_RETURN_VALUE MLD_SYSV_ABI +unsigned mld_rej_uniform_eta4_avx2_asm( + int32_t *r, const uint8_t buf[MLD_AVX2_REJ_UNIFORM_ETA4_BUFLEN], + const uint8_t table[256][8]) +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(buf, MLD_AVX2_REJ_UNIFORM_ETA4_BUFLEN)) + requires(table == mld_rej_uniform_table) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(return_value <= MLDSA_N) + ensures(array_abs_bound(r, 0, return_value, 5)) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +#define mld_poly_decompose_32_avx2_asm MLD_NAMESPACE(poly_decompose_32_avx2_asm) +MLD_SYSV_ABI +void mld_poly_decompose_32_avx2_asm(int32_t *a1, int32_t *a0) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_poly_decompose_32_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(a1, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a0, sizeof(int32_t) * MLDSA_N)) + requires(array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q)) + assigns(memory_slice(a1, sizeof(int32_t) * MLDSA_N)) + assigns(memory_slice(a0, sizeof(int32_t) * MLDSA_N)) + /* check-magic: 16 == (MLDSA_Q - 1) / (2 * ((MLDSA_Q - 1) / 32)) */ + ensures(array_bound(a1, 0, MLDSA_N, 0, 16)) + /* check-magic: 261889 == (MLDSA_Q - 1) / 32 + 1 */ + ensures(array_abs_bound(a0, 0, MLDSA_N, 261889)) +); + +#define mld_poly_decompose_88_avx2_asm MLD_NAMESPACE(poly_decompose_88_avx2_asm) +MLD_SYSV_ABI +void mld_poly_decompose_88_avx2_asm(int32_t *a1, int32_t *a0) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_poly_decompose_88_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(a1, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a0, sizeof(int32_t) * MLDSA_N)) + requires(array_bound(a0, 0, MLDSA_N, 0, MLDSA_Q)) + assigns(memory_slice(a1, sizeof(int32_t) * MLDSA_N)) + assigns(memory_slice(a0, sizeof(int32_t) * MLDSA_N)) + /* check-magic: 44 == (MLDSA_Q - 1) / (2 * ((MLDSA_Q - 1) / 88)) */ + ensures(array_bound(a1, 0, MLDSA_N, 0, 44)) + /* check-magic: 95233 == (MLDSA_Q - 1) / 88 + 1 */ + ensures(array_abs_bound(a0, 0, MLDSA_N, 95233)) +); +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#define mld_poly_caddq_avx2_asm MLD_NAMESPACE(poly_caddq_avx2_asm) +MLD_SYSV_ABI +void mld_poly_caddq_avx2_asm(int32_t *r) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_poly_caddq_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(array_abs_bound(r, 0, MLDSA_N, MLDSA_Q)) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(array_bound(r, 0, MLDSA_N, 0, MLDSA_Q)) +); + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +#define mld_poly_use_hint_32_avx2_asm MLD_NAMESPACE(poly_use_hint_32_avx2_asm) +MLD_SYSV_ABI +void mld_poly_use_hint_32_avx2_asm(int32_t *a, const int32_t *h) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_poly_use_hint_32_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(h, sizeof(int32_t) * MLDSA_N)) + requires(array_bound(a, 0, MLDSA_N, 0, MLDSA_Q)) + requires(array_bound(h, 0, MLDSA_N, 0, 2)) + assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N)) + /* check-magic: 16 == (MLDSA_Q - 1) / (2 * ((MLDSA_Q - 1) / 32)) */ + ensures(array_bound(a, 0, MLDSA_N, 0, 16)) +); + +#define mld_poly_use_hint_88_avx2_asm MLD_NAMESPACE(poly_use_hint_88_avx2_asm) +MLD_SYSV_ABI +void mld_poly_use_hint_88_avx2_asm(int32_t *a, const int32_t *h) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_poly_use_hint_88_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(h, sizeof(int32_t) * MLDSA_N)) + requires(array_bound(a, 0, MLDSA_N, 0, MLDSA_Q)) + requires(array_bound(h, 0, MLDSA_N, 0, 2)) + assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N)) + /* check-magic: 44 == (MLDSA_Q - 1) / (2 * ((MLDSA_Q - 1) / 88)) */ + ensures(array_bound(a, 0, MLDSA_N, 0, 44)) +); +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#define mld_poly_chknorm_avx2_asm MLD_NAMESPACE(poly_chknorm_avx2_asm) +MLD_MUST_CHECK_RETURN_VALUE MLD_SYSV_ABI +int mld_poly_chknorm_avx2_asm(const int32_t *a, int32_t B) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_poly_chknorm_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N)) + /* HOL Light precondition: abs(ival(x i)) < 2^31, i.e., a[i] != INT32_MIN */ + requires(forall(k0, 0, MLDSA_N, a[k0] > INT32_MIN)) + /* HOL Light precondition: 0 <= ival bound (asm computes B-1 internally) */ + requires(B >= 0) + ensures(return_value == 0 || return_value == 1) + ensures((return_value == 0) == array_abs_bound(a, 0, MLDSA_N, B)) +); + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +#define mld_polyz_unpack_17_avx2_asm MLD_NAMESPACE(polyz_unpack_17_avx2_asm) +MLD_SYSV_ABI +void mld_polyz_unpack_17_avx2_asm(int32_t *r, const uint8_t *a) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_polyz_unpack_17_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a, 576)) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(array_bound(r, 0, MLDSA_N, -((1 << 17) - 1), (1 << 17) + 1)) +); + +#define mld_polyz_unpack_19_avx2_asm MLD_NAMESPACE(polyz_unpack_19_avx2_asm) +MLD_SYSV_ABI +void mld_polyz_unpack_19_avx2_asm(int32_t *r, const uint8_t *a) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_polyz_unpack_19_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a, 640)) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(array_bound(r, 0, MLDSA_N, -((1 << 19) - 1), (1 << 19) + 1)) +); +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#define mld_pointwise_avx2_asm MLD_NAMESPACE(pointwise_avx2_asm) +MLD_SYSV_ABI +void mld_pointwise_avx2_asm(int32_t *a, const int32_t *b, const int32_t *qdata) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_pointwise_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(a, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(b, sizeof(int32_t) * MLDSA_N)) + /* Input bound MLD_NTT_BOUND = 9 * MLD_FQMUL_BOUND, the guaranteed bound of + * any forward NTT implementation. Hardcoded here to keep this header free + * of poly.h. */ + /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */ + requires(array_abs_bound(a, 0, MLDSA_N, 94279698)) + requires(array_abs_bound(b, 0, MLDSA_N, 94279698)) + requires(qdata == mld_qdata) + assigns(memory_slice(a, sizeof(int32_t) * MLDSA_N)) + ensures(array_abs_bound(a, 0, MLDSA_N, 8380417)) +); + +#define mld_pointwise_acc_l4_avx2_asm MLD_NAMESPACE(pointwise_acc_l4_avx2_asm) +MLD_SYSV_ABI +void mld_pointwise_acc_l4_avx2_asm(int32_t c[MLDSA_N], + const int32_t a[4][MLDSA_N], + const int32_t b[4][MLDSA_N], + const int32_t *qdata) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_pointwise_acc_l4_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(c, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a, sizeof(int32_t) * 4 * MLDSA_N)) + requires(memory_no_alias(b, sizeof(int32_t) * 4 * MLDSA_N)) + requires(forall(l0, 0, 4, array_abs_bound(a[l0], 0, MLDSA_N, 8380417))) + /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */ + requires(forall(l1, 0, 4, array_abs_bound(b[l1], 0, MLDSA_N, 94279698))) + requires(qdata == mld_qdata) + assigns(memory_slice(c, sizeof(int32_t) * MLDSA_N)) + ensures(array_abs_bound(c, 0, MLDSA_N, 8380417)) +); + +#define mld_pointwise_acc_l5_avx2_asm MLD_NAMESPACE(pointwise_acc_l5_avx2_asm) +MLD_SYSV_ABI +void mld_pointwise_acc_l5_avx2_asm(int32_t c[MLDSA_N], + const int32_t a[5][MLDSA_N], + const int32_t b[5][MLDSA_N], + const int32_t *qdata) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_pointwise_acc_l5_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(c, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a, sizeof(int32_t) * 5 * MLDSA_N)) + requires(memory_no_alias(b, sizeof(int32_t) * 5 * MLDSA_N)) + requires(forall(l0, 0, 5, array_abs_bound(a[l0], 0, MLDSA_N, 8380417))) + /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */ + requires(forall(l1, 0, 5, array_abs_bound(b[l1], 0, MLDSA_N, 94279698))) + requires(qdata == mld_qdata) + assigns(memory_slice(c, sizeof(int32_t) * MLDSA_N)) + ensures(array_abs_bound(c, 0, MLDSA_N, 8380417)) +); + +#define mld_pointwise_acc_l7_avx2_asm MLD_NAMESPACE(pointwise_acc_l7_avx2_asm) +MLD_SYSV_ABI +void mld_pointwise_acc_l7_avx2_asm(int32_t c[MLDSA_N], + const int32_t a[7][MLDSA_N], + const int32_t b[7][MLDSA_N], + const int32_t *qdata) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mldsa_pointwise_acc_l7_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(c, sizeof(int32_t) * MLDSA_N)) + requires(memory_no_alias(a, sizeof(int32_t) * 7 * MLDSA_N)) + requires(memory_no_alias(b, sizeof(int32_t) * 7 * MLDSA_N)) + requires(forall(l0, 0, 7, array_abs_bound(a[l0], 0, MLDSA_N, 8380417))) + /* check-magic: 94279698 == 9 * ((5 * MLDSA_Q + 3) / 4) */ + requires(forall(l1, 0, 7, array_abs_bound(b[l1], 0, MLDSA_N, 94279698))) + requires(qdata == mld_qdata) + assigns(memory_slice(c, sizeof(int32_t) * MLDSA_N)) + ensures(array_abs_bound(c, 0, MLDSA_N, 8380417)) +); + +#endif /* !MLD_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H */ diff --git a/cbits/mldsa/src/native/x86_64/src/consts.c b/cbits/mldsa/src/native/x86_64/src/consts.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/consts.c @@ -0,0 +1,157 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mldsa-native repository. + * Do not modify it directly. + */ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +#include "consts.h" + +/* + * Table of zeta values used in the AVX2 forward and inverse NTT + * See autogen for details. + */ +MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const int32_t mld_qdata[624] = { + 8380417, 8380417, 8380417, 8380417, 8380417, + 8380417, 8380417, 8380417, 58728449, 58728449, + 58728449, 58728449, 58728449, 58728449, 58728449, + 58728449, -8395782, -8395782, -8395782, -8395782, + -8395782, -8395782, -8395782, -8395782, 41978, + 41978, 41978, 41978, 41978, 41978, + 41978, 41978, -151046689, 1830765815, -1929875198, + -1927777021, 1640767044, 1477910808, 1612161320, 1640734244, + 308362795, 308362795, 308362795, 308362795, -1815525077, + -1815525077, -1815525077, -1815525077, -1374673747, -1374673747, + -1374673747, -1374673747, -1091570561, -1091570561, -1091570561, + -1091570561, -1929495947, -1929495947, -1929495947, -1929495947, + 515185417, 515185417, 515185417, 515185417, -285697463, + -285697463, -285697463, -285697463, 625853735, 625853735, + 625853735, 625853735, 1727305304, 1727305304, 2082316400, + 2082316400, -1364982364, -1364982364, 858240904, 858240904, + 1806278032, 1806278032, 222489248, 222489248, -346752664, + -346752664, 684667771, 684667771, 1654287830, 1654287830, + -878576921, -878576921, -1257667337, -1257667337, -748618600, + -748618600, 329347125, 329347125, 1837364258, 1837364258, + -1443016191, -1443016191, -1170414139, -1170414139, -1846138265, + -1631226336, -1404529459, 1838055109, 1594295555, -1076973524, + -1898723372, -594436433, -202001019, -475984260, -561427818, + 1797021249, -1061813248, 2059733581, -1661512036, -1104976547, + -1750224323, -901666090, 418987550, 1831915353, -1925356481, + 992097815, 879957084, 2024403852, 1484874664, -1636082790, + -285388938, -1983539117, -1495136972, -950076368, -1714807468, + -952438995, -1574918427, 1350681039, -2143979939, 1599739335, + -1285853323, -993005454, -1440787840, 568627424, -783134478, + -588790216, 289871779, -1262003603, 2135294594, -1018755525, + -889861155, 1665705315, 1321868265, 1225434135, -1784632064, + 666258756, 675310538, -1555941048, -1999506068, -1499481951, + -695180180, -1375177022, 1777179795, 334803717, -178766299, + -518252220, 1957047970, 1146323031, -654783359, -1974159335, + 1651689966, 140455867, -1039411342, 1955560694, 1529189038, + -2131021878, -247357819, 1518161567, -86965173, 1708872713, + 1787797779, 1638590967, -120646188, -1669960606, -916321552, + 1155548552, 2143745726, 1210558298, -1261461890, -318346816, + 628664287, -1729304568, 1422575624, 1424130038, -1185330464, + 235321234, 168022240, 1206536194, 985155484, -894060583, + -898413, -1363460238, -605900043, 2027833504, 14253662, + 1014493059, 863641633, 1819892093, 2124962073, -1223601433, + -1920467227, -1637785316, -1536588520, 694382729, 235104446, + -1045062172, 831969619, -300448763, 756955444, -260312805, + 1554794072, 1339088280, -2040058690, -853476187, -2047270596, + -1723816713, -1591599803, -440824168, 1119856484, 1544891539, + 155290192, -973777462, 991903578, 912367099, -44694137, + 1176904444, -421552614, -818371958, 1747917558, -325927722, + 908452108, 1851023419, -1176751719, -1354528380, -72690498, + -314284737, 985022747, 963438279, -1078959975, 604552167, + -1021949428, 608791570, 173440395, -2126092136, -1316619236, + -1039370342, 6087993, -110126092, 565464272, -1758099917, + -1600929361, 879867909, -1809756372, 400711272, 1363007700, + 30313375, -326425360, 1683520342, -517299994, 2027935492, + -1372618620, 128353682, -1123881663, 137583815, -635454918, + -642772911, 45766801, 671509323, -2070602178, 419615363, + 1216882040, -270590488, -1276805128, 371462360, -1357098057, + -384158533, 827959816, -596344473, 702390549, -279505433, + -260424530, -71875110, -1208667171, -1499603926, 2036925262, + -540420426, 746144248, -1420958686, 2032221021, 1904936414, + 1257750362, 1926727420, 1931587462, 1258381762, 885133339, + 1629985060, 1967222129, 6363718, -1287922800, 1136965286, + 1779436847, 1116720494, 1042326957, 1405999311, 713994583, + 940195359, -1542497137, 2061661095, -883155599, 1726753853, + -1547952704, 394851342, 283780712, 776003547, 1123958025, + 201262505, 1934038751, 374860238, -3975713, 25847, + -2608894, -518909, 237124, -777960, -876248, + 466468, 1826347, 1826347, 1826347, 1826347, + 2353451, 2353451, 2353451, 2353451, -359251, + -359251, -359251, -359251, -2091905, -2091905, + -2091905, -2091905, 3119733, 3119733, 3119733, + 3119733, -2884855, -2884855, -2884855, -2884855, + 3111497, 3111497, 3111497, 3111497, 2680103, + 2680103, 2680103, 2680103, 2725464, 2725464, + 1024112, 1024112, -1079900, -1079900, 3585928, + 3585928, -549488, -549488, -1119584, -1119584, + 2619752, 2619752, -2108549, -2108549, -2118186, + -2118186, -3859737, -3859737, -1399561, -1399561, + -3277672, -3277672, 1757237, 1757237, -19422, + -19422, 4010497, 4010497, 280005, 280005, + 2706023, 95776, 3077325, 3530437, -1661693, + -3592148, -2537516, 3915439, -3861115, -3043716, + 3574422, -2867647, 3539968, -300467, 2348700, + -539299, -1699267, -1643818, 3505694, -3821735, + 3507263, -2140649, -1600420, 3699596, 811944, + 531354, 954230, 3881043, 3900724, -2556880, + 2071892, -2797779, -3930395, -3677745, -1452451, + 2176455, -1257611, -4083598, -3190144, -3632928, + 3412210, 2147896, -2967645, -411027, -671102, + -22981, -381987, 1852771, -3343383, 508951, + 44288, 904516, -3724342, 1653064, 2389356, + 759969, 189548, 3159746, -2409325, 1315589, + 1285669, -812732, -3019102, -3628969, -1528703, + -3041255, 3475950, -1585221, 1939314, -1000202, + -3157330, 126922, -983419, 2715295, -3693493, + -2477047, -1228525, -1308169, 1349076, -1430430, + 264944, 3097992, -1100098, 3958618, -8578, + -3249728, -210977, -1316856, -3553272, -1851402, + -177440, 1341330, -1584928, -1439742, -3881060, + 3839961, 2091667, -3342478, 266997, -3520352, + 900702, 495491, -655327, -3556995, 342297, + 3437287, 2842341, 4055324, -3767016, -2994039, + -1333058, -451100, -1279661, 1500165, -542412, + -2584293, -2013608, 1957272, -3183426, 810149, + -3038916, 2213111, -426683, -1667432, -2939036, + 183443, -554416, 3937738, 3407706, 2244091, + 2434439, -3759364, 1859098, -1613174, -3122442, + -525098, 286988, -3342277, 2691481, 1247620, + 1250494, 1869119, 1237275, 1312455, 1917081, + 777191, -2831860, -3724270, 2432395, 3369112, + 162844, 1652634, 3523897, -975884, 1723600, + -1104333, -2235985, -976891, 3919660, 1400424, + 2316500, -2446433, -1235728, -1197226, 909542, + -43260, 2031748, -768622, -2437823, 1735879, + -2590150, 2486353, 2635921, 1903435, -3318210, + 3306115, -2546312, 2235880, -1671176, 594136, + 2454455, 185531, 1616392, -3694233, 3866901, + 1717735, -1803090, -260646, -420899, 1612842, + -48306, -846154, 3817976, -3562462, 3513181, + -3193378, 819034, -522500, 3207046, -3595838, + 4108315, 203044, 1265009, 1595974, -3548272, + -1050970, -1430225, -1962642, -1374803, 3406031, + -1846953, -3776993, -164721, -1207385, 3014001, + -1799107, 269760, 472078, 1910376, -3833893, + -2286327, -3545687, -1362209, 1976782, +}; + +#else /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +MLD_EMPTY_CU(avx2_consts) + +#endif /* !(MLD_ARITH_BACKEND_X86_64_DEFAULT && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mldsa/src/native/x86_64/src/consts.h b/cbits/mldsa/src/native/x86_64/src/consts.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/consts.h @@ -0,0 +1,27 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mldsa-native repository. + * Do not modify it directly. + */ + +#ifndef MLD_NATIVE_X86_64_SRC_CONSTS_H +#define MLD_NATIVE_X86_64_SRC_CONSTS_H +#include "../../../common.h" +#define MLD_AVX2_BACKEND_DATA_OFFSET_8XQ 0 +#define MLD_AVX2_BACKEND_DATA_OFFSET_8XQINV 8 +#define MLD_AVX2_BACKEND_DATA_OFFSET_8XDIV_QINV 16 +#define MLD_AVX2_BACKEND_DATA_OFFSET_8XDIV 24 +#define MLD_AVX2_BACKEND_DATA_OFFSET_ZETAS_QINV 32 +#define MLD_AVX2_BACKEND_DATA_OFFSET_ZETAS 328 + +#ifndef __ASSEMBLER__ +#define mld_qdata MLD_NAMESPACE(qdata) +MLD_INTERNAL_DATA_DECLARATION const int32_t mld_qdata[624]; +#endif + +#endif /* !MLD_NATIVE_X86_64_SRC_CONSTS_H */ diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_intt_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_intt_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_intt_avx2_asm.S @@ -0,0 +1,2333 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + + /* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + +/*yaml + Name: invntt_avx2_asm + Description: x86_64 AVX2 inverse NTT + Signature: void mld_invntt_avx2_asm(int32_t *r, const int32_t *qdata) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t *r + description: Input/output polynomial (256 x int32_t) + rsi: + type: buffer + size_bytes: 2496 + permissions: read-only + c_parameter: const int32_t *qdata + description: Precomputed constants (624 x int32_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_intt_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(invntt_avx2_asm) +MLD_ASM_FN_SYMBOL(invntt_avx2_asm) + + .cfi_startproc + vmovdqa (%rsi), %ymm0 + vmovdqa (%rdi), %ymm4 + vmovdqa 0x20(%rdi), %ymm5 + vmovdqa 0x40(%rdi), %ymm6 + vmovdqa 0x60(%rdi), %ymm7 + vmovdqa 0x80(%rdi), %ymm8 + vmovdqa 0xa0(%rdi), %ymm9 + vmovdqa 0xc0(%rdi), %ymm10 + vmovdqa 0xe0(%rdi), %ymm11 + vpermq $0x1b, 0x500(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x9a0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm4, %ymm5, %ymm12 + vpaddd %ymm5, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm5 # ymm5 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm5, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm5, %ymm5 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm5, %ymm5 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7] + vpermq $0x1b, 0x480(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x920(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm6, %ymm7, %ymm12 + vpaddd %ymm7, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpermq $0x1b, 0x400(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x8a0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm8, %ymm9, %ymm12 + vpaddd %ymm9, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpermq $0x1b, 0x380(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x820(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm10, %ymm11, %ymm12 + vpaddd %ymm11, %ymm10, %ymm10 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpermq $0x1b, 0x300(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x7a0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm4, %ymm6, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm6 # ymm6 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm6, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm6, %ymm6 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm6, %ymm6 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7] + vpsubd %ymm5, %ymm7, %ymm12 + vpaddd %ymm7, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpermq $0x1b, 0x280(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x720(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm8, %ymm10, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm9, %ymm11, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpermq $0x1b, 0x200(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x6a0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm4, %ymm8, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm5, %ymm9, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpsubd %ymm6, %ymm10, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm7, %ymm11, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vmovsldup %ymm5, %ymm3 # ymm3 = ymm5[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7] + vpsrlq $0x20, %ymm4, %ymm4 + vpblendd $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7] + vmovsldup %ymm7, %ymm4 # ymm4 = ymm7[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7] + vpsrlq $0x20, %ymm6, %ymm6 + vpblendd $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7] + vmovsldup %ymm9, %ymm6 # ymm6 = ymm9[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7] + vpsrlq $0x20, %ymm8, %ymm8 + vpblendd $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7] + vmovsldup %ymm11, %ymm8 # ymm8 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm8, %ymm10, %ymm8 # ymm8 = ymm10[0],ymm8[1],ymm10[2],ymm8[3],ymm10[4],ymm8[5],ymm10[6],ymm8[7] + vpsrlq $0x20, %ymm10, %ymm10 + vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7] + vpermq $0x1b, 0x180(%rsi), %ymm1 # ymm1 = mem[3,2,1,0] + vpermq $0x1b, 0x620(%rsi), %ymm2 # ymm2 = mem[3,2,1,0] + vpsubd %ymm3, %ymm5, %ymm12 + vpaddd %ymm5, %ymm3, %ymm3 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm5 # ymm5 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm5, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm5, %ymm5 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7] + vpsubd %ymm4, %ymm7, %ymm12 + vpaddd %ymm7, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpsubd %ymm6, %ymm9, %ymm12 + vpaddd %ymm6, %ymm9, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpsubd %ymm8, %ymm11, %ymm12 + vpaddd %ymm11, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpunpcklqdq %ymm4, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm4[0],ymm3[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[1],ymm4[1],ymm3[3],ymm4[3] + vpunpcklqdq %ymm8, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm8[0],ymm6[2],ymm8[2] + vpunpckhqdq %ymm8, %ymm6, %ymm8 # ymm8 = ymm6[1],ymm8[1],ymm6[3],ymm8[3] + vpunpcklqdq %ymm7, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm7[0],ymm5[2],ymm7[2] + vpunpckhqdq %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[1],ymm7[1],ymm5[3],ymm7[3] + vpunpcklqdq %ymm11, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm11[0],ymm9[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3] + vpermq $0x1b, 0x100(%rsi), %ymm1 # ymm1 = mem[3,2,1,0] + vpermq $0x1b, 0x5a0(%rsi), %ymm2 # ymm2 = mem[3,2,1,0] + vpsubd %ymm10, %ymm4, %ymm12 + vpaddd %ymm4, %ymm10, %ymm10 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm4 # ymm4 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm4, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm4, %ymm4 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm4, %ymm12, %ymm4 # ymm4 = ymm12[0],ymm4[1],ymm12[2],ymm4[3],ymm12[4],ymm4[5],ymm12[6],ymm4[7] + vpsubd %ymm3, %ymm8, %ymm12 + vpaddd %ymm3, %ymm8, %ymm3 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm6, %ymm7, %ymm12 + vpaddd %ymm7, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpsubd %ymm5, %ymm11, %ymm12 + vpaddd %ymm5, %ymm11, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vperm2i128 $0x20, %ymm3, %ymm10, %ymm9 # ymm9 = ymm10[0,1],ymm3[0,1] + vperm2i128 $0x31, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[2,3],ymm3[2,3] + vperm2i128 $0x20, %ymm5, %ymm6, %ymm10 # ymm10 = ymm6[0,1],ymm5[0,1] + vperm2i128 $0x31, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[2,3],ymm5[2,3] + vperm2i128 $0x20, %ymm8, %ymm4, %ymm6 # ymm6 = ymm4[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm11, %ymm7, %ymm4 # ymm4 = ymm7[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3] + vpbroadcastd 0x9c(%rsi), %ymm1 + vpbroadcastd 0x53c(%rsi), %ymm2 + vpsubd %ymm9, %ymm3, %ymm12 + vpaddd %ymm3, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm3 # ymm3 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm3, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm3, %ymm3 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm3, %ymm3 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm3, %ymm12, %ymm3 # ymm3 = ymm12[0],ymm3[1],ymm12[2],ymm3[3],ymm12[4],ymm3[5],ymm12[6],ymm3[7] + vpsubd %ymm10, %ymm5, %ymm12 + vpaddd %ymm5, %ymm10, %ymm10 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm5 # ymm5 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm5, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm5, %ymm5 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7] + vpsubd %ymm6, %ymm8, %ymm12 + vpaddd %ymm6, %ymm8, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm4, %ymm11, %ymm12 + vpaddd %ymm4, %ymm11, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vmovdqa %ymm9, (%rdi) + vmovdqa %ymm10, 0x20(%rdi) + vmovdqa %ymm6, 0x40(%rdi) + vmovdqa %ymm4, 0x60(%rdi) + vmovdqa %ymm3, 0x80(%rdi) + vmovdqa %ymm5, 0xa0(%rdi) + vmovdqa %ymm8, 0xc0(%rdi) + vmovdqa %ymm11, 0xe0(%rdi) + vmovdqa 0x100(%rdi), %ymm4 + vmovdqa 0x120(%rdi), %ymm5 + vmovdqa 0x140(%rdi), %ymm6 + vmovdqa 0x160(%rdi), %ymm7 + vmovdqa 0x180(%rdi), %ymm8 + vmovdqa 0x1a0(%rdi), %ymm9 + vmovdqa 0x1c0(%rdi), %ymm10 + vmovdqa 0x1e0(%rdi), %ymm11 + vpermq $0x1b, 0x4e0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x980(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm4, %ymm5, %ymm12 + vpaddd %ymm5, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm5 # ymm5 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm5, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm5, %ymm5 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm5, %ymm5 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7] + vpermq $0x1b, 0x460(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x900(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm6, %ymm7, %ymm12 + vpaddd %ymm7, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpermq $0x1b, 0x3e0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x880(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm8, %ymm9, %ymm12 + vpaddd %ymm9, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpermq $0x1b, 0x360(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x800(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm10, %ymm11, %ymm12 + vpaddd %ymm11, %ymm10, %ymm10 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpermq $0x1b, 0x2e0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x780(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm4, %ymm6, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm6 # ymm6 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm6, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm6, %ymm6 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm6, %ymm6 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7] + vpsubd %ymm5, %ymm7, %ymm12 + vpaddd %ymm7, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpermq $0x1b, 0x260(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x700(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm8, %ymm10, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm9, %ymm11, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpermq $0x1b, 0x1e0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x680(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm4, %ymm8, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm5, %ymm9, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpsubd %ymm6, %ymm10, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm7, %ymm11, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vmovsldup %ymm5, %ymm3 # ymm3 = ymm5[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7] + vpsrlq $0x20, %ymm4, %ymm4 + vpblendd $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7] + vmovsldup %ymm7, %ymm4 # ymm4 = ymm7[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7] + vpsrlq $0x20, %ymm6, %ymm6 + vpblendd $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7] + vmovsldup %ymm9, %ymm6 # ymm6 = ymm9[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7] + vpsrlq $0x20, %ymm8, %ymm8 + vpblendd $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7] + vmovsldup %ymm11, %ymm8 # ymm8 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm8, %ymm10, %ymm8 # ymm8 = ymm10[0],ymm8[1],ymm10[2],ymm8[3],ymm10[4],ymm8[5],ymm10[6],ymm8[7] + vpsrlq $0x20, %ymm10, %ymm10 + vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7] + vpermq $0x1b, 0x160(%rsi), %ymm1 # ymm1 = mem[3,2,1,0] + vpermq $0x1b, 0x600(%rsi), %ymm2 # ymm2 = mem[3,2,1,0] + vpsubd %ymm3, %ymm5, %ymm12 + vpaddd %ymm5, %ymm3, %ymm3 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm5 # ymm5 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm5, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm5, %ymm5 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7] + vpsubd %ymm4, %ymm7, %ymm12 + vpaddd %ymm7, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpsubd %ymm6, %ymm9, %ymm12 + vpaddd %ymm6, %ymm9, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpsubd %ymm8, %ymm11, %ymm12 + vpaddd %ymm11, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpunpcklqdq %ymm4, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm4[0],ymm3[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[1],ymm4[1],ymm3[3],ymm4[3] + vpunpcklqdq %ymm8, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm8[0],ymm6[2],ymm8[2] + vpunpckhqdq %ymm8, %ymm6, %ymm8 # ymm8 = ymm6[1],ymm8[1],ymm6[3],ymm8[3] + vpunpcklqdq %ymm7, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm7[0],ymm5[2],ymm7[2] + vpunpckhqdq %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[1],ymm7[1],ymm5[3],ymm7[3] + vpunpcklqdq %ymm11, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm11[0],ymm9[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3] + vpermq $0x1b, 0xe0(%rsi), %ymm1 # ymm1 = mem[3,2,1,0] + vpermq $0x1b, 0x580(%rsi), %ymm2 # ymm2 = mem[3,2,1,0] + vpsubd %ymm10, %ymm4, %ymm12 + vpaddd %ymm4, %ymm10, %ymm10 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm4 # ymm4 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm4, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm4, %ymm4 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm4, %ymm12, %ymm4 # ymm4 = ymm12[0],ymm4[1],ymm12[2],ymm4[3],ymm12[4],ymm4[5],ymm12[6],ymm4[7] + vpsubd %ymm3, %ymm8, %ymm12 + vpaddd %ymm3, %ymm8, %ymm3 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm6, %ymm7, %ymm12 + vpaddd %ymm7, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpsubd %ymm5, %ymm11, %ymm12 + vpaddd %ymm5, %ymm11, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vperm2i128 $0x20, %ymm3, %ymm10, %ymm9 # ymm9 = ymm10[0,1],ymm3[0,1] + vperm2i128 $0x31, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[2,3],ymm3[2,3] + vperm2i128 $0x20, %ymm5, %ymm6, %ymm10 # ymm10 = ymm6[0,1],ymm5[0,1] + vperm2i128 $0x31, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[2,3],ymm5[2,3] + vperm2i128 $0x20, %ymm8, %ymm4, %ymm6 # ymm6 = ymm4[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm11, %ymm7, %ymm4 # ymm4 = ymm7[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3] + vpbroadcastd 0x98(%rsi), %ymm1 + vpbroadcastd 0x538(%rsi), %ymm2 + vpsubd %ymm9, %ymm3, %ymm12 + vpaddd %ymm3, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm3 # ymm3 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm3, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm3, %ymm3 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm3, %ymm3 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm3, %ymm12, %ymm3 # ymm3 = ymm12[0],ymm3[1],ymm12[2],ymm3[3],ymm12[4],ymm3[5],ymm12[6],ymm3[7] + vpsubd %ymm10, %ymm5, %ymm12 + vpaddd %ymm5, %ymm10, %ymm10 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm5 # ymm5 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm5, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm5, %ymm5 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7] + vpsubd %ymm6, %ymm8, %ymm12 + vpaddd %ymm6, %ymm8, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm4, %ymm11, %ymm12 + vpaddd %ymm4, %ymm11, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vmovdqa %ymm9, 0x100(%rdi) + vmovdqa %ymm10, 0x120(%rdi) + vmovdqa %ymm6, 0x140(%rdi) + vmovdqa %ymm4, 0x160(%rdi) + vmovdqa %ymm3, 0x180(%rdi) + vmovdqa %ymm5, 0x1a0(%rdi) + vmovdqa %ymm8, 0x1c0(%rdi) + vmovdqa %ymm11, 0x1e0(%rdi) + vmovdqa 0x200(%rdi), %ymm4 + vmovdqa 0x220(%rdi), %ymm5 + vmovdqa 0x240(%rdi), %ymm6 + vmovdqa 0x260(%rdi), %ymm7 + vmovdqa 0x280(%rdi), %ymm8 + vmovdqa 0x2a0(%rdi), %ymm9 + vmovdqa 0x2c0(%rdi), %ymm10 + vmovdqa 0x2e0(%rdi), %ymm11 + vpermq $0x1b, 0x4c0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x960(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm4, %ymm5, %ymm12 + vpaddd %ymm5, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm5 # ymm5 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm5, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm5, %ymm5 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm5, %ymm5 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7] + vpermq $0x1b, 0x440(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x8e0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm6, %ymm7, %ymm12 + vpaddd %ymm7, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpermq $0x1b, 0x3c0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x860(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm8, %ymm9, %ymm12 + vpaddd %ymm9, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpermq $0x1b, 0x340(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x7e0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm10, %ymm11, %ymm12 + vpaddd %ymm11, %ymm10, %ymm10 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpermq $0x1b, 0x2c0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x760(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm4, %ymm6, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm6 # ymm6 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm6, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm6, %ymm6 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm6, %ymm6 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7] + vpsubd %ymm5, %ymm7, %ymm12 + vpaddd %ymm7, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpermq $0x1b, 0x240(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x6e0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm8, %ymm10, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm9, %ymm11, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpermq $0x1b, 0x1c0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x660(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm4, %ymm8, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm5, %ymm9, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpsubd %ymm6, %ymm10, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm7, %ymm11, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vmovsldup %ymm5, %ymm3 # ymm3 = ymm5[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7] + vpsrlq $0x20, %ymm4, %ymm4 + vpblendd $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7] + vmovsldup %ymm7, %ymm4 # ymm4 = ymm7[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7] + vpsrlq $0x20, %ymm6, %ymm6 + vpblendd $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7] + vmovsldup %ymm9, %ymm6 # ymm6 = ymm9[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7] + vpsrlq $0x20, %ymm8, %ymm8 + vpblendd $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7] + vmovsldup %ymm11, %ymm8 # ymm8 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm8, %ymm10, %ymm8 # ymm8 = ymm10[0],ymm8[1],ymm10[2],ymm8[3],ymm10[4],ymm8[5],ymm10[6],ymm8[7] + vpsrlq $0x20, %ymm10, %ymm10 + vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7] + vpermq $0x1b, 0x140(%rsi), %ymm1 # ymm1 = mem[3,2,1,0] + vpermq $0x1b, 0x5e0(%rsi), %ymm2 # ymm2 = mem[3,2,1,0] + vpsubd %ymm3, %ymm5, %ymm12 + vpaddd %ymm5, %ymm3, %ymm3 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm5 # ymm5 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm5, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm5, %ymm5 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7] + vpsubd %ymm4, %ymm7, %ymm12 + vpaddd %ymm7, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpsubd %ymm6, %ymm9, %ymm12 + vpaddd %ymm6, %ymm9, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpsubd %ymm8, %ymm11, %ymm12 + vpaddd %ymm11, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpunpcklqdq %ymm4, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm4[0],ymm3[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[1],ymm4[1],ymm3[3],ymm4[3] + vpunpcklqdq %ymm8, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm8[0],ymm6[2],ymm8[2] + vpunpckhqdq %ymm8, %ymm6, %ymm8 # ymm8 = ymm6[1],ymm8[1],ymm6[3],ymm8[3] + vpunpcklqdq %ymm7, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm7[0],ymm5[2],ymm7[2] + vpunpckhqdq %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[1],ymm7[1],ymm5[3],ymm7[3] + vpunpcklqdq %ymm11, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm11[0],ymm9[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3] + vpermq $0x1b, 0xc0(%rsi), %ymm1 # ymm1 = mem[3,2,1,0] + vpermq $0x1b, 0x560(%rsi), %ymm2 # ymm2 = mem[3,2,1,0] + vpsubd %ymm10, %ymm4, %ymm12 + vpaddd %ymm4, %ymm10, %ymm10 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm4 # ymm4 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm4, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm4, %ymm4 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm4, %ymm12, %ymm4 # ymm4 = ymm12[0],ymm4[1],ymm12[2],ymm4[3],ymm12[4],ymm4[5],ymm12[6],ymm4[7] + vpsubd %ymm3, %ymm8, %ymm12 + vpaddd %ymm3, %ymm8, %ymm3 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm6, %ymm7, %ymm12 + vpaddd %ymm7, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpsubd %ymm5, %ymm11, %ymm12 + vpaddd %ymm5, %ymm11, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vperm2i128 $0x20, %ymm3, %ymm10, %ymm9 # ymm9 = ymm10[0,1],ymm3[0,1] + vperm2i128 $0x31, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[2,3],ymm3[2,3] + vperm2i128 $0x20, %ymm5, %ymm6, %ymm10 # ymm10 = ymm6[0,1],ymm5[0,1] + vperm2i128 $0x31, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[2,3],ymm5[2,3] + vperm2i128 $0x20, %ymm8, %ymm4, %ymm6 # ymm6 = ymm4[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm11, %ymm7, %ymm4 # ymm4 = ymm7[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3] + vpbroadcastd 0x94(%rsi), %ymm1 + vpbroadcastd 0x534(%rsi), %ymm2 + vpsubd %ymm9, %ymm3, %ymm12 + vpaddd %ymm3, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm3 # ymm3 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm3, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm3, %ymm3 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm3, %ymm3 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm3, %ymm12, %ymm3 # ymm3 = ymm12[0],ymm3[1],ymm12[2],ymm3[3],ymm12[4],ymm3[5],ymm12[6],ymm3[7] + vpsubd %ymm10, %ymm5, %ymm12 + vpaddd %ymm5, %ymm10, %ymm10 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm5 # ymm5 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm5, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm5, %ymm5 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7] + vpsubd %ymm6, %ymm8, %ymm12 + vpaddd %ymm6, %ymm8, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm4, %ymm11, %ymm12 + vpaddd %ymm4, %ymm11, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vmovdqa %ymm9, 0x200(%rdi) + vmovdqa %ymm10, 0x220(%rdi) + vmovdqa %ymm6, 0x240(%rdi) + vmovdqa %ymm4, 0x260(%rdi) + vmovdqa %ymm3, 0x280(%rdi) + vmovdqa %ymm5, 0x2a0(%rdi) + vmovdqa %ymm8, 0x2c0(%rdi) + vmovdqa %ymm11, 0x2e0(%rdi) + vmovdqa 0x300(%rdi), %ymm4 + vmovdqa 0x320(%rdi), %ymm5 + vmovdqa 0x340(%rdi), %ymm6 + vmovdqa 0x360(%rdi), %ymm7 + vmovdqa 0x380(%rdi), %ymm8 + vmovdqa 0x3a0(%rdi), %ymm9 + vmovdqa 0x3c0(%rdi), %ymm10 + vmovdqa 0x3e0(%rdi), %ymm11 + vpermq $0x1b, 0x4a0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x940(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm4, %ymm5, %ymm12 + vpaddd %ymm5, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm5 # ymm5 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm5, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm5, %ymm5 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm5, %ymm5 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7] + vpermq $0x1b, 0x420(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x8c0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm6, %ymm7, %ymm12 + vpaddd %ymm7, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpermq $0x1b, 0x3a0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x840(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm8, %ymm9, %ymm12 + vpaddd %ymm9, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpermq $0x1b, 0x320(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x7c0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm10, %ymm11, %ymm12 + vpaddd %ymm11, %ymm10, %ymm10 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpermq $0x1b, 0x2a0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x740(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm4, %ymm6, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm6 # ymm6 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm6, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm6, %ymm6 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm6, %ymm6 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7] + vpsubd %ymm5, %ymm7, %ymm12 + vpaddd %ymm7, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpermq $0x1b, 0x220(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x6c0(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm8, %ymm10, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm9, %ymm11, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpermq $0x1b, 0x1a0(%rsi), %ymm3 # ymm3 = mem[3,2,1,0] + vpermq $0x1b, 0x640(%rsi), %ymm15 # ymm15 = mem[3,2,1,0] + vmovshdup %ymm3, %ymm1 # ymm1 = ymm3[1,1,3,3,5,5,7,7] + vmovshdup %ymm15, %ymm2 # ymm2 = ymm15[1,1,3,3,5,5,7,7] + vpsubd %ymm4, %ymm8, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm5, %ymm9, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpsubd %ymm6, %ymm10, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm7, %ymm11, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm3, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm15, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vmovsldup %ymm5, %ymm3 # ymm3 = ymm5[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7] + vpsrlq $0x20, %ymm4, %ymm4 + vpblendd $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7] + vmovsldup %ymm7, %ymm4 # ymm4 = ymm7[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7] + vpsrlq $0x20, %ymm6, %ymm6 + vpblendd $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7] + vmovsldup %ymm9, %ymm6 # ymm6 = ymm9[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7] + vpsrlq $0x20, %ymm8, %ymm8 + vpblendd $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7] + vmovsldup %ymm11, %ymm8 # ymm8 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm8, %ymm10, %ymm8 # ymm8 = ymm10[0],ymm8[1],ymm10[2],ymm8[3],ymm10[4],ymm8[5],ymm10[6],ymm8[7] + vpsrlq $0x20, %ymm10, %ymm10 + vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7] + vpermq $0x1b, 0x120(%rsi), %ymm1 # ymm1 = mem[3,2,1,0] + vpermq $0x1b, 0x5c0(%rsi), %ymm2 # ymm2 = mem[3,2,1,0] + vpsubd %ymm3, %ymm5, %ymm12 + vpaddd %ymm5, %ymm3, %ymm3 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm5 # ymm5 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm5, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm5, %ymm5 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7] + vpsubd %ymm4, %ymm7, %ymm12 + vpaddd %ymm7, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpsubd %ymm6, %ymm9, %ymm12 + vpaddd %ymm6, %ymm9, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpsubd %ymm8, %ymm11, %ymm12 + vpaddd %ymm11, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpunpcklqdq %ymm4, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm4[0],ymm3[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[1],ymm4[1],ymm3[3],ymm4[3] + vpunpcklqdq %ymm8, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm8[0],ymm6[2],ymm8[2] + vpunpckhqdq %ymm8, %ymm6, %ymm8 # ymm8 = ymm6[1],ymm8[1],ymm6[3],ymm8[3] + vpunpcklqdq %ymm7, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm7[0],ymm5[2],ymm7[2] + vpunpckhqdq %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[1],ymm7[1],ymm5[3],ymm7[3] + vpunpcklqdq %ymm11, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm11[0],ymm9[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3] + vpermq $0x1b, 0xa0(%rsi), %ymm1 # ymm1 = mem[3,2,1,0] + vpermq $0x1b, 0x540(%rsi), %ymm2 # ymm2 = mem[3,2,1,0] + vpsubd %ymm10, %ymm4, %ymm12 + vpaddd %ymm4, %ymm10, %ymm10 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm4 # ymm4 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm4, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm4, %ymm4 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm4, %ymm12, %ymm4 # ymm4 = ymm12[0],ymm4[1],ymm12[2],ymm4[3],ymm12[4],ymm4[5],ymm12[6],ymm4[7] + vpsubd %ymm3, %ymm8, %ymm12 + vpaddd %ymm3, %ymm8, %ymm3 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm6, %ymm7, %ymm12 + vpaddd %ymm7, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpsubd %ymm5, %ymm11, %ymm12 + vpaddd %ymm5, %ymm11, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vperm2i128 $0x20, %ymm3, %ymm10, %ymm9 # ymm9 = ymm10[0,1],ymm3[0,1] + vperm2i128 $0x31, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[2,3],ymm3[2,3] + vperm2i128 $0x20, %ymm5, %ymm6, %ymm10 # ymm10 = ymm6[0,1],ymm5[0,1] + vperm2i128 $0x31, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[2,3],ymm5[2,3] + vperm2i128 $0x20, %ymm8, %ymm4, %ymm6 # ymm6 = ymm4[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm11, %ymm7, %ymm4 # ymm4 = ymm7[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3] + vpbroadcastd 0x90(%rsi), %ymm1 + vpbroadcastd 0x530(%rsi), %ymm2 + vpsubd %ymm9, %ymm3, %ymm12 + vpaddd %ymm3, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm3 # ymm3 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm3, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm3, %ymm3 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm3, %ymm3 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm3, %ymm12, %ymm3 # ymm3 = ymm12[0],ymm3[1],ymm12[2],ymm3[3],ymm12[4],ymm3[5],ymm12[6],ymm3[7] + vpsubd %ymm10, %ymm5, %ymm12 + vpaddd %ymm5, %ymm10, %ymm10 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm5 # ymm5 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm5, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm5, %ymm5 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm5, %ymm12, %ymm5 # ymm5 = ymm12[0],ymm5[1],ymm12[2],ymm5[3],ymm12[4],ymm5[5],ymm12[6],ymm5[7] + vpsubd %ymm6, %ymm8, %ymm12 + vpaddd %ymm6, %ymm8, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm4, %ymm11, %ymm12 + vpaddd %ymm4, %ymm11, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vmovdqa %ymm9, 0x300(%rdi) + vmovdqa %ymm10, 0x320(%rdi) + vmovdqa %ymm6, 0x340(%rdi) + vmovdqa %ymm4, 0x360(%rdi) + vmovdqa %ymm3, 0x380(%rdi) + vmovdqa %ymm5, 0x3a0(%rdi) + vmovdqa %ymm8, 0x3c0(%rdi) + vmovdqa %ymm11, 0x3e0(%rdi) + vmovdqa (%rdi), %ymm4 + vmovdqa 0x80(%rdi), %ymm5 + vmovdqa 0x100(%rdi), %ymm6 + vmovdqa 0x180(%rdi), %ymm7 + vmovdqa 0x200(%rdi), %ymm8 + vmovdqa 0x280(%rdi), %ymm9 + vmovdqa 0x300(%rdi), %ymm10 + vmovdqa 0x380(%rdi), %ymm11 + vpbroadcastd 0x8c(%rsi), %ymm1 + vpbroadcastd 0x52c(%rsi), %ymm2 + vpsubd %ymm4, %ymm6, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm6 # ymm6 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm6, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm6, %ymm6 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7] + vpsubd %ymm5, %ymm7, %ymm12 + vpaddd %ymm7, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpbroadcastd 0x88(%rsi), %ymm1 + vpbroadcastd 0x528(%rsi), %ymm2 + vpsubd %ymm8, %ymm10, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm9, %ymm11, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpbroadcastd 0x80(%rsi), %ymm1 + vpbroadcastd 0x520(%rsi), %ymm2 + vpsubd %ymm4, %ymm8, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm5, %ymm9, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpsubd %ymm6, %ymm10, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm7, %ymm11, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vmovdqa %ymm8, 0x200(%rdi) + vmovdqa %ymm9, 0x280(%rdi) + vmovdqa %ymm10, 0x300(%rdi) + vmovdqa %ymm11, 0x380(%rdi) + vmovdqa 0x40(%rsi), %ymm1 + vmovdqa 0x60(%rsi), %ymm2 + vpmuldq %ymm1, %ymm4, %ymm12 + vpmuldq %ymm1, %ymm5, %ymm13 + vmovshdup %ymm4, %ymm8 # ymm8 = ymm4[1,1,3,3,5,5,7,7] + vmovshdup %ymm5, %ymm9 # ymm9 = ymm5[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm1, %ymm9, %ymm15 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpmuldq %ymm0, %ymm15, %ymm15 + vpsubd %ymm12, %ymm4, %ymm4 + vpsubd %ymm13, %ymm5, %ymm5 + vpsubd %ymm14, %ymm8, %ymm8 + vpsubd %ymm15, %ymm9, %ymm9 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vmovshdup %ymm5, %ymm5 # ymm5 = ymm5[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm8[1],ymm4[2],ymm8[3],ymm4[4],ymm8[5],ymm4[6],ymm8[7] + vpblendd $0xaa, %ymm9, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm9[1],ymm5[2],ymm9[3],ymm5[4],ymm9[5],ymm5[6],ymm9[7] + vpmuldq %ymm1, %ymm6, %ymm12 + vpmuldq %ymm1, %ymm7, %ymm13 + vmovshdup %ymm6, %ymm8 # ymm8 = ymm6[1,1,3,3,5,5,7,7] + vmovshdup %ymm7, %ymm9 # ymm9 = ymm7[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm1, %ymm9, %ymm15 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpmuldq %ymm0, %ymm15, %ymm15 + vpsubd %ymm12, %ymm6, %ymm6 + vpsubd %ymm13, %ymm7, %ymm7 + vpsubd %ymm14, %ymm8, %ymm8 + vpsubd %ymm15, %ymm9, %ymm9 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vmovshdup %ymm7, %ymm7 # ymm7 = ymm7[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm8[1],ymm6[2],ymm8[3],ymm6[4],ymm8[5],ymm6[6],ymm8[7] + vpblendd $0xaa, %ymm9, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vmovdqa %ymm4, (%rdi) + vmovdqa %ymm5, 0x80(%rdi) + vmovdqa %ymm6, 0x100(%rdi) + vmovdqa %ymm7, 0x180(%rdi) + vmovdqa 0x20(%rdi), %ymm4 + vmovdqa 0xa0(%rdi), %ymm5 + vmovdqa 0x120(%rdi), %ymm6 + vmovdqa 0x1a0(%rdi), %ymm7 + vmovdqa 0x220(%rdi), %ymm8 + vmovdqa 0x2a0(%rdi), %ymm9 + vmovdqa 0x320(%rdi), %ymm10 + vmovdqa 0x3a0(%rdi), %ymm11 + vpbroadcastd 0x8c(%rsi), %ymm1 + vpbroadcastd 0x52c(%rsi), %ymm2 + vpsubd %ymm4, %ymm6, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm6 # ymm6 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm6, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm6, %ymm6 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7] + vpsubd %ymm5, %ymm7, %ymm12 + vpaddd %ymm7, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpbroadcastd 0x88(%rsi), %ymm1 + vpbroadcastd 0x528(%rsi), %ymm2 + vpsubd %ymm8, %ymm10, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm9, %ymm11, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpbroadcastd 0x80(%rsi), %ymm1 + vpbroadcastd 0x520(%rsi), %ymm2 + vpsubd %ymm4, %ymm8, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm5, %ymm9, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpsubd %ymm6, %ymm10, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm7, %ymm11, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vmovdqa %ymm8, 0x220(%rdi) + vmovdqa %ymm9, 0x2a0(%rdi) + vmovdqa %ymm10, 0x320(%rdi) + vmovdqa %ymm11, 0x3a0(%rdi) + vmovdqa 0x40(%rsi), %ymm1 + vmovdqa 0x60(%rsi), %ymm2 + vpmuldq %ymm1, %ymm4, %ymm12 + vpmuldq %ymm1, %ymm5, %ymm13 + vmovshdup %ymm4, %ymm8 # ymm8 = ymm4[1,1,3,3,5,5,7,7] + vmovshdup %ymm5, %ymm9 # ymm9 = ymm5[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm1, %ymm9, %ymm15 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpmuldq %ymm0, %ymm15, %ymm15 + vpsubd %ymm12, %ymm4, %ymm4 + vpsubd %ymm13, %ymm5, %ymm5 + vpsubd %ymm14, %ymm8, %ymm8 + vpsubd %ymm15, %ymm9, %ymm9 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vmovshdup %ymm5, %ymm5 # ymm5 = ymm5[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm8[1],ymm4[2],ymm8[3],ymm4[4],ymm8[5],ymm4[6],ymm8[7] + vpblendd $0xaa, %ymm9, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm9[1],ymm5[2],ymm9[3],ymm5[4],ymm9[5],ymm5[6],ymm9[7] + vpmuldq %ymm1, %ymm6, %ymm12 + vpmuldq %ymm1, %ymm7, %ymm13 + vmovshdup %ymm6, %ymm8 # ymm8 = ymm6[1,1,3,3,5,5,7,7] + vmovshdup %ymm7, %ymm9 # ymm9 = ymm7[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm1, %ymm9, %ymm15 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpmuldq %ymm0, %ymm15, %ymm15 + vpsubd %ymm12, %ymm6, %ymm6 + vpsubd %ymm13, %ymm7, %ymm7 + vpsubd %ymm14, %ymm8, %ymm8 + vpsubd %ymm15, %ymm9, %ymm9 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vmovshdup %ymm7, %ymm7 # ymm7 = ymm7[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm8[1],ymm6[2],ymm8[3],ymm6[4],ymm8[5],ymm6[6],ymm8[7] + vpblendd $0xaa, %ymm9, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vmovdqa %ymm4, 0x20(%rdi) + vmovdqa %ymm5, 0xa0(%rdi) + vmovdqa %ymm6, 0x120(%rdi) + vmovdqa %ymm7, 0x1a0(%rdi) + vmovdqa 0x40(%rdi), %ymm4 + vmovdqa 0xc0(%rdi), %ymm5 + vmovdqa 0x140(%rdi), %ymm6 + vmovdqa 0x1c0(%rdi), %ymm7 + vmovdqa 0x240(%rdi), %ymm8 + vmovdqa 0x2c0(%rdi), %ymm9 + vmovdqa 0x340(%rdi), %ymm10 + vmovdqa 0x3c0(%rdi), %ymm11 + vpbroadcastd 0x8c(%rsi), %ymm1 + vpbroadcastd 0x52c(%rsi), %ymm2 + vpsubd %ymm4, %ymm6, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm6 # ymm6 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm6, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm6, %ymm6 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7] + vpsubd %ymm5, %ymm7, %ymm12 + vpaddd %ymm7, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpbroadcastd 0x88(%rsi), %ymm1 + vpbroadcastd 0x528(%rsi), %ymm2 + vpsubd %ymm8, %ymm10, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm9, %ymm11, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpbroadcastd 0x80(%rsi), %ymm1 + vpbroadcastd 0x520(%rsi), %ymm2 + vpsubd %ymm4, %ymm8, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm5, %ymm9, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpsubd %ymm6, %ymm10, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm7, %ymm11, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vmovdqa %ymm8, 0x240(%rdi) + vmovdqa %ymm9, 0x2c0(%rdi) + vmovdqa %ymm10, 0x340(%rdi) + vmovdqa %ymm11, 0x3c0(%rdi) + vmovdqa 0x40(%rsi), %ymm1 + vmovdqa 0x60(%rsi), %ymm2 + vpmuldq %ymm1, %ymm4, %ymm12 + vpmuldq %ymm1, %ymm5, %ymm13 + vmovshdup %ymm4, %ymm8 # ymm8 = ymm4[1,1,3,3,5,5,7,7] + vmovshdup %ymm5, %ymm9 # ymm9 = ymm5[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm1, %ymm9, %ymm15 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpmuldq %ymm0, %ymm15, %ymm15 + vpsubd %ymm12, %ymm4, %ymm4 + vpsubd %ymm13, %ymm5, %ymm5 + vpsubd %ymm14, %ymm8, %ymm8 + vpsubd %ymm15, %ymm9, %ymm9 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vmovshdup %ymm5, %ymm5 # ymm5 = ymm5[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm8[1],ymm4[2],ymm8[3],ymm4[4],ymm8[5],ymm4[6],ymm8[7] + vpblendd $0xaa, %ymm9, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm9[1],ymm5[2],ymm9[3],ymm5[4],ymm9[5],ymm5[6],ymm9[7] + vpmuldq %ymm1, %ymm6, %ymm12 + vpmuldq %ymm1, %ymm7, %ymm13 + vmovshdup %ymm6, %ymm8 # ymm8 = ymm6[1,1,3,3,5,5,7,7] + vmovshdup %ymm7, %ymm9 # ymm9 = ymm7[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm1, %ymm9, %ymm15 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpmuldq %ymm0, %ymm15, %ymm15 + vpsubd %ymm12, %ymm6, %ymm6 + vpsubd %ymm13, %ymm7, %ymm7 + vpsubd %ymm14, %ymm8, %ymm8 + vpsubd %ymm15, %ymm9, %ymm9 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vmovshdup %ymm7, %ymm7 # ymm7 = ymm7[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm8[1],ymm6[2],ymm8[3],ymm6[4],ymm8[5],ymm6[6],ymm8[7] + vpblendd $0xaa, %ymm9, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vmovdqa %ymm4, 0x40(%rdi) + vmovdqa %ymm5, 0xc0(%rdi) + vmovdqa %ymm6, 0x140(%rdi) + vmovdqa %ymm7, 0x1c0(%rdi) + vmovdqa 0x60(%rdi), %ymm4 + vmovdqa 0xe0(%rdi), %ymm5 + vmovdqa 0x160(%rdi), %ymm6 + vmovdqa 0x1e0(%rdi), %ymm7 + vmovdqa 0x260(%rdi), %ymm8 + vmovdqa 0x2e0(%rdi), %ymm9 + vmovdqa 0x360(%rdi), %ymm10 + vmovdqa 0x3e0(%rdi), %ymm11 + vpbroadcastd 0x8c(%rsi), %ymm1 + vpbroadcastd 0x52c(%rsi), %ymm2 + vpsubd %ymm4, %ymm6, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm6 # ymm6 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm6, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm6, %ymm6 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm6, %ymm12, %ymm6 # ymm6 = ymm12[0],ymm6[1],ymm12[2],ymm6[3],ymm12[4],ymm6[5],ymm12[6],ymm6[7] + vpsubd %ymm5, %ymm7, %ymm12 + vpaddd %ymm7, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm7 # ymm7 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm7, %ymm7 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm7, %ymm12, %ymm7 # ymm7 = ymm12[0],ymm7[1],ymm12[2],ymm7[3],ymm12[4],ymm7[5],ymm12[6],ymm7[7] + vpbroadcastd 0x88(%rsi), %ymm1 + vpbroadcastd 0x528(%rsi), %ymm2 + vpsubd %ymm8, %ymm10, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm9, %ymm11, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vpbroadcastd 0x80(%rsi), %ymm1 + vpbroadcastd 0x520(%rsi), %ymm2 + vpsubd %ymm4, %ymm8, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm8 # ymm8 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm8, %ymm8 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm12, %ymm8 # ymm8 = ymm12[0],ymm8[1],ymm12[2],ymm8[3],ymm12[4],ymm8[5],ymm12[6],ymm8[7] + vpsubd %ymm5, %ymm9, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm9 # ymm9 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm9, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm9, %ymm9 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm9, %ymm12, %ymm9 # ymm9 = ymm12[0],ymm9[1],ymm12[2],ymm9[3],ymm12[4],ymm9[5],ymm12[6],ymm9[7] + vpsubd %ymm6, %ymm10, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm10 # ymm10 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm10, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm10, %ymm10 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm10, %ymm12, %ymm10 # ymm10 = ymm12[0],ymm10[1],ymm12[2],ymm10[3],ymm12[4],ymm10[5],ymm12[6],ymm10[7] + vpsubd %ymm7, %ymm11, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vpmuldq %ymm1, %ymm12, %ymm13 + vmovshdup %ymm12, %ymm11 # ymm11 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm14 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpsubd %ymm13, %ymm12, %ymm12 + vpsubd %ymm14, %ymm11, %ymm11 + vmovshdup %ymm12, %ymm12 # ymm12 = ymm12[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm11, %ymm12, %ymm11 # ymm11 = ymm12[0],ymm11[1],ymm12[2],ymm11[3],ymm12[4],ymm11[5],ymm12[6],ymm11[7] + vmovdqa %ymm8, 0x260(%rdi) + vmovdqa %ymm9, 0x2e0(%rdi) + vmovdqa %ymm10, 0x360(%rdi) + vmovdqa %ymm11, 0x3e0(%rdi) + vmovdqa 0x40(%rsi), %ymm1 + vmovdqa 0x60(%rsi), %ymm2 + vpmuldq %ymm1, %ymm4, %ymm12 + vpmuldq %ymm1, %ymm5, %ymm13 + vmovshdup %ymm4, %ymm8 # ymm8 = ymm4[1,1,3,3,5,5,7,7] + vmovshdup %ymm5, %ymm9 # ymm9 = ymm5[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm1, %ymm9, %ymm15 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpmuldq %ymm0, %ymm15, %ymm15 + vpsubd %ymm12, %ymm4, %ymm4 + vpsubd %ymm13, %ymm5, %ymm5 + vpsubd %ymm14, %ymm8, %ymm8 + vpsubd %ymm15, %ymm9, %ymm9 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vmovshdup %ymm5, %ymm5 # ymm5 = ymm5[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm8[1],ymm4[2],ymm8[3],ymm4[4],ymm8[5],ymm4[6],ymm8[7] + vpblendd $0xaa, %ymm9, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm9[1],ymm5[2],ymm9[3],ymm5[4],ymm9[5],ymm5[6],ymm9[7] + vpmuldq %ymm1, %ymm6, %ymm12 + vpmuldq %ymm1, %ymm7, %ymm13 + vmovshdup %ymm6, %ymm8 # ymm8 = ymm6[1,1,3,3,5,5,7,7] + vmovshdup %ymm7, %ymm9 # ymm9 = ymm7[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm14 + vpmuldq %ymm1, %ymm9, %ymm15 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm0, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vpmuldq %ymm0, %ymm15, %ymm15 + vpsubd %ymm12, %ymm6, %ymm6 + vpsubd %ymm13, %ymm7, %ymm7 + vpsubd %ymm14, %ymm8, %ymm8 + vpsubd %ymm15, %ymm9, %ymm9 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vmovshdup %ymm7, %ymm7 # ymm7 = ymm7[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm8, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm8[1],ymm6[2],ymm8[3],ymm6[4],ymm8[5],ymm6[6],ymm8[7] + vpblendd $0xaa, %ymm9, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vmovdqa %ymm4, 0x60(%rdi) + vmovdqa %ymm5, 0xe0(%rdi) + vmovdqa %ymm6, 0x160(%rdi) + vmovdqa %ymm7, 0x1e0(%rdi) + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(invntt_avx2_asm) + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_ntt_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_ntt_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_ntt_avx2_asm.S @@ -0,0 +1,2405 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + + /* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + +/*yaml + Name: ntt_avx2_asm + Description: x86_64 AVX2 forward NTT + Signature: void mld_ntt_avx2_asm(int32_t *r, const int32_t *qdata) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t *r + description: Input/output polynomial (256 x int32_t) + rsi: + type: buffer + size_bytes: 2496 + permissions: read-only + c_parameter: const int32_t *qdata + description: Precomputed constants (624 x int32_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_ntt_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(ntt_avx2_asm) +MLD_ASM_FN_SYMBOL(ntt_avx2_asm) + + .cfi_startproc + vmovdqa (%rsi), %ymm0 + vpbroadcastd 0x84(%rsi), %ymm1 + vpbroadcastd 0x524(%rsi), %ymm2 + vmovdqa (%rdi), %ymm4 + vmovdqa 0x80(%rdi), %ymm5 + vmovdqa 0x100(%rdi), %ymm6 + vmovdqa 0x180(%rdi), %ymm7 + vmovdqa 0x200(%rdi), %ymm8 + vmovdqa 0x280(%rdi), %ymm9 + vmovdqa 0x300(%rdi), %ymm10 + vmovdqa 0x380(%rdi), %ymm11 + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm4, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm9, %ymm13 + vmovshdup %ymm9, %ymm12 # ymm12 = ymm9[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm9, %ymm9 # ymm9 = ymm9[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7] + vpsubd %ymm9, %ymm5, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm9 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm6, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm7, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm7, %ymm7 + vpbroadcastd 0x88(%rsi), %ymm1 + vpbroadcastd 0x528(%rsi), %ymm2 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm4, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm7, %ymm13 + vmovshdup %ymm7, %ymm12 # ymm12 = ymm7[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm7, %ymm7 # ymm7 = ymm7[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7] + vpsubd %ymm7, %ymm5, %ymm12 + vpaddd %ymm7, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm7 + vpsubd %ymm13, %ymm5, %ymm5 + vpbroadcastd 0x8c(%rsi), %ymm1 + vpbroadcastd 0x52c(%rsi), %ymm2 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm8, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm9, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm9, %ymm9 + vmovdqa %ymm4, (%rdi) + vmovdqa %ymm5, 0x80(%rdi) + vmovdqa %ymm6, 0x100(%rdi) + vmovdqa %ymm7, 0x180(%rdi) + vmovdqa %ymm8, 0x200(%rdi) + vmovdqa %ymm9, 0x280(%rdi) + vmovdqa %ymm10, 0x300(%rdi) + vmovdqa %ymm11, 0x380(%rdi) + vpbroadcastd 0x84(%rsi), %ymm1 + vpbroadcastd 0x524(%rsi), %ymm2 + vmovdqa 0x20(%rdi), %ymm4 + vmovdqa 0xa0(%rdi), %ymm5 + vmovdqa 0x120(%rdi), %ymm6 + vmovdqa 0x1a0(%rdi), %ymm7 + vmovdqa 0x220(%rdi), %ymm8 + vmovdqa 0x2a0(%rdi), %ymm9 + vmovdqa 0x320(%rdi), %ymm10 + vmovdqa 0x3a0(%rdi), %ymm11 + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm4, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm9, %ymm13 + vmovshdup %ymm9, %ymm12 # ymm12 = ymm9[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm9, %ymm9 # ymm9 = ymm9[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7] + vpsubd %ymm9, %ymm5, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm9 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm6, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm7, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm7, %ymm7 + vpbroadcastd 0x88(%rsi), %ymm1 + vpbroadcastd 0x528(%rsi), %ymm2 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm4, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm7, %ymm13 + vmovshdup %ymm7, %ymm12 # ymm12 = ymm7[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm7, %ymm7 # ymm7 = ymm7[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7] + vpsubd %ymm7, %ymm5, %ymm12 + vpaddd %ymm7, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm7 + vpsubd %ymm13, %ymm5, %ymm5 + vpbroadcastd 0x8c(%rsi), %ymm1 + vpbroadcastd 0x52c(%rsi), %ymm2 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm8, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm9, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm9, %ymm9 + vmovdqa %ymm4, 0x20(%rdi) + vmovdqa %ymm5, 0xa0(%rdi) + vmovdqa %ymm6, 0x120(%rdi) + vmovdqa %ymm7, 0x1a0(%rdi) + vmovdqa %ymm8, 0x220(%rdi) + vmovdqa %ymm9, 0x2a0(%rdi) + vmovdqa %ymm10, 0x320(%rdi) + vmovdqa %ymm11, 0x3a0(%rdi) + vpbroadcastd 0x84(%rsi), %ymm1 + vpbroadcastd 0x524(%rsi), %ymm2 + vmovdqa 0x40(%rdi), %ymm4 + vmovdqa 0xc0(%rdi), %ymm5 + vmovdqa 0x140(%rdi), %ymm6 + vmovdqa 0x1c0(%rdi), %ymm7 + vmovdqa 0x240(%rdi), %ymm8 + vmovdqa 0x2c0(%rdi), %ymm9 + vmovdqa 0x340(%rdi), %ymm10 + vmovdqa 0x3c0(%rdi), %ymm11 + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm4, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm9, %ymm13 + vmovshdup %ymm9, %ymm12 # ymm12 = ymm9[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm9, %ymm9 # ymm9 = ymm9[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7] + vpsubd %ymm9, %ymm5, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm9 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm6, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm7, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm7, %ymm7 + vpbroadcastd 0x88(%rsi), %ymm1 + vpbroadcastd 0x528(%rsi), %ymm2 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm4, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm7, %ymm13 + vmovshdup %ymm7, %ymm12 # ymm12 = ymm7[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm7, %ymm7 # ymm7 = ymm7[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7] + vpsubd %ymm7, %ymm5, %ymm12 + vpaddd %ymm7, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm7 + vpsubd %ymm13, %ymm5, %ymm5 + vpbroadcastd 0x8c(%rsi), %ymm1 + vpbroadcastd 0x52c(%rsi), %ymm2 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm8, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm9, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm9, %ymm9 + vmovdqa %ymm4, 0x40(%rdi) + vmovdqa %ymm5, 0xc0(%rdi) + vmovdqa %ymm6, 0x140(%rdi) + vmovdqa %ymm7, 0x1c0(%rdi) + vmovdqa %ymm8, 0x240(%rdi) + vmovdqa %ymm9, 0x2c0(%rdi) + vmovdqa %ymm10, 0x340(%rdi) + vmovdqa %ymm11, 0x3c0(%rdi) + vpbroadcastd 0x84(%rsi), %ymm1 + vpbroadcastd 0x524(%rsi), %ymm2 + vmovdqa 0x60(%rdi), %ymm4 + vmovdqa 0xe0(%rdi), %ymm5 + vmovdqa 0x160(%rdi), %ymm6 + vmovdqa 0x1e0(%rdi), %ymm7 + vmovdqa 0x260(%rdi), %ymm8 + vmovdqa 0x2e0(%rdi), %ymm9 + vmovdqa 0x360(%rdi), %ymm10 + vmovdqa 0x3e0(%rdi), %ymm11 + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm4, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm9, %ymm13 + vmovshdup %ymm9, %ymm12 # ymm12 = ymm9[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm9, %ymm9 # ymm9 = ymm9[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7] + vpsubd %ymm9, %ymm5, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm9 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm6, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm7, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm7, %ymm7 + vpbroadcastd 0x88(%rsi), %ymm1 + vpbroadcastd 0x528(%rsi), %ymm2 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm4, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm7, %ymm13 + vmovshdup %ymm7, %ymm12 # ymm12 = ymm7[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm7, %ymm7 # ymm7 = ymm7[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7] + vpsubd %ymm7, %ymm5, %ymm12 + vpaddd %ymm7, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm7 + vpsubd %ymm13, %ymm5, %ymm5 + vpbroadcastd 0x8c(%rsi), %ymm1 + vpbroadcastd 0x52c(%rsi), %ymm2 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm8, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm9, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm9, %ymm9 + vmovdqa %ymm4, 0x60(%rdi) + vmovdqa %ymm5, 0xe0(%rdi) + vmovdqa %ymm6, 0x160(%rdi) + vmovdqa %ymm7, 0x1e0(%rdi) + vmovdqa %ymm8, 0x260(%rdi) + vmovdqa %ymm9, 0x2e0(%rdi) + vmovdqa %ymm10, 0x360(%rdi) + vmovdqa %ymm11, 0x3e0(%rdi) + vmovdqa (%rdi), %ymm4 + vmovdqa 0x20(%rdi), %ymm5 + vmovdqa 0x40(%rdi), %ymm6 + vmovdqa 0x60(%rdi), %ymm7 + vmovdqa 0x80(%rdi), %ymm8 + vmovdqa 0xa0(%rdi), %ymm9 + vmovdqa 0xc0(%rdi), %ymm10 + vmovdqa 0xe0(%rdi), %ymm11 + vpbroadcastd 0x90(%rsi), %ymm1 + vpbroadcastd 0x530(%rsi), %ymm2 + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm4, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm9, %ymm13 + vmovshdup %ymm9, %ymm12 # ymm12 = ymm9[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm9, %ymm9 # ymm9 = ymm9[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7] + vpsubd %ymm9, %ymm5, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm9 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm6, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm7, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm7, %ymm7 + vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3] + vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1] + vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3] + vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3] + vmovdqa 0xa0(%rsi), %ymm1 + vmovdqa 0x540(%rsi), %ymm2 + vpmuldq %ymm1, %ymm5, %ymm13 + vmovshdup %ymm5, %ymm12 # ymm12 = ymm5[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm5, %ymm5 # ymm5 = ymm5[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7] + vpsubd %ymm5, %ymm3, %ymm12 + vpaddd %ymm5, %ymm3, %ymm3 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm5 + vpsubd %ymm13, %ymm3, %ymm3 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm8, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm4, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm9, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm9, %ymm9 + vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3] + vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2] + vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3] + vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2] + vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3] + vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3] + vmovdqa 0x120(%rsi), %ymm1 + vmovdqa 0x5c0(%rsi), %ymm2 + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm7, %ymm12 + vpaddd %ymm7, %ymm8, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm7, %ymm7 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm5, %ymm12 + vpaddd %ymm6, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm4, %ymm13 + vmovshdup %ymm4, %ymm12 # ymm12 = ymm4[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7] + vpsubd %ymm4, %ymm3, %ymm12 + vpaddd %ymm4, %ymm3, %ymm3 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm4 + vpsubd %ymm13, %ymm3, %ymm3 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm10, %ymm12 + vpaddd %ymm11, %ymm10, %ymm10 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm10, %ymm10 + vmovsldup %ymm8, %ymm9 # ymm9 = ymm8[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vpsrlq $0x20, %ymm7, %ymm7 + vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7] + vmovsldup %ymm6, %ymm7 # ymm7 = ymm6[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7] + vpsrlq $0x20, %ymm5, %ymm5 + vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7] + vmovsldup %ymm4, %ymm5 # ymm5 = ymm4[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7] + vmovsldup %ymm11, %ymm3 # ymm3 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7] + vpsrlq $0x20, %ymm10, %ymm10 + vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7] + vmovdqa 0x1a0(%rsi), %ymm1 + vmovdqa 0x640(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm5, %ymm13 + vmovshdup %ymm5, %ymm12 # ymm12 = ymm5[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm5, %ymm5 # ymm5 = ymm5[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7] + vpsubd %ymm5, %ymm9, %ymm12 + vpaddd %ymm5, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm5 + vpsubd %ymm13, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm4, %ymm13 + vmovshdup %ymm4, %ymm12 # ymm12 = ymm4[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7] + vpsubd %ymm4, %ymm8, %ymm12 + vpaddd %ymm4, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm4 + vpsubd %ymm13, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm3, %ymm13 + vmovshdup %ymm3, %ymm12 # ymm12 = ymm3[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm3, %ymm3 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm3, %ymm3 # ymm3 = ymm3[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7] + vpsubd %ymm3, %ymm7, %ymm12 + vpaddd %ymm3, %ymm7, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm3 + vpsubd %ymm13, %ymm7, %ymm7 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm6, %ymm12 + vpaddd %ymm6, %ymm11, %ymm6 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm6, %ymm6 + vmovdqa 0x220(%rsi), %ymm1 + vmovdqa 0x6c0(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm13 + vmovshdup %ymm7, %ymm12 # ymm12 = ymm7[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm7, %ymm7 # ymm7 = ymm7[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7] + vpsubd %ymm7, %ymm9, %ymm12 + vpaddd %ymm7, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm7 + vpsubd %ymm13, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm8, %ymm12 + vpaddd %ymm6, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm8, %ymm8 + vmovdqa 0x2a0(%rsi), %ymm1 + vmovdqa 0x740(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm3, %ymm13 + vmovshdup %ymm3, %ymm12 # ymm12 = ymm3[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm3, %ymm3 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm3, %ymm3 # ymm3 = ymm3[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7] + vpsubd %ymm3, %ymm5, %ymm12 + vpaddd %ymm3, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm3 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm4, %ymm12 + vpaddd %ymm4, %ymm11, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm4, %ymm4 + vmovdqa 0x320(%rsi), %ymm1 + vmovdqa 0x7c0(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm9, %ymm12 + vpaddd %ymm8, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm9, %ymm9 + vmovdqa 0x3a0(%rsi), %ymm1 + vmovdqa 0x840(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm7, %ymm12 + vpaddd %ymm6, %ymm7, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm7, %ymm7 + vmovdqa 0x420(%rsi), %ymm1 + vmovdqa 0x8c0(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm4, %ymm13 + vmovshdup %ymm4, %ymm12 # ymm12 = ymm4[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7] + vpsubd %ymm4, %ymm5, %ymm12 + vpaddd %ymm4, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm4 + vpsubd %ymm13, %ymm5, %ymm5 + vmovdqa 0x4a0(%rsi), %ymm1 + vmovdqa 0x940(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm3, %ymm12 + vpaddd %ymm3, %ymm11, %ymm3 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm3, %ymm3 + vmovdqa %ymm9, (%rdi) + vmovdqa %ymm8, 0x20(%rdi) + vmovdqa %ymm7, 0x40(%rdi) + vmovdqa %ymm6, 0x60(%rdi) + vmovdqa %ymm5, 0x80(%rdi) + vmovdqa %ymm4, 0xa0(%rdi) + vmovdqa %ymm3, 0xc0(%rdi) + vmovdqa %ymm11, 0xe0(%rdi) + vmovdqa 0x100(%rdi), %ymm4 + vmovdqa 0x120(%rdi), %ymm5 + vmovdqa 0x140(%rdi), %ymm6 + vmovdqa 0x160(%rdi), %ymm7 + vmovdqa 0x180(%rdi), %ymm8 + vmovdqa 0x1a0(%rdi), %ymm9 + vmovdqa 0x1c0(%rdi), %ymm10 + vmovdqa 0x1e0(%rdi), %ymm11 + vpbroadcastd 0x94(%rsi), %ymm1 + vpbroadcastd 0x534(%rsi), %ymm2 + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm4, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm9, %ymm13 + vmovshdup %ymm9, %ymm12 # ymm12 = ymm9[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm9, %ymm9 # ymm9 = ymm9[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7] + vpsubd %ymm9, %ymm5, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm9 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm6, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm7, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm7, %ymm7 + vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3] + vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1] + vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3] + vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3] + vmovdqa 0xc0(%rsi), %ymm1 + vmovdqa 0x560(%rsi), %ymm2 + vpmuldq %ymm1, %ymm5, %ymm13 + vmovshdup %ymm5, %ymm12 # ymm12 = ymm5[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm5, %ymm5 # ymm5 = ymm5[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7] + vpsubd %ymm5, %ymm3, %ymm12 + vpaddd %ymm5, %ymm3, %ymm3 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm5 + vpsubd %ymm13, %ymm3, %ymm3 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm8, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm4, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm9, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm9, %ymm9 + vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3] + vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2] + vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3] + vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2] + vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3] + vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3] + vmovdqa 0x140(%rsi), %ymm1 + vmovdqa 0x5e0(%rsi), %ymm2 + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm7, %ymm12 + vpaddd %ymm7, %ymm8, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm7, %ymm7 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm5, %ymm12 + vpaddd %ymm6, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm4, %ymm13 + vmovshdup %ymm4, %ymm12 # ymm12 = ymm4[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7] + vpsubd %ymm4, %ymm3, %ymm12 + vpaddd %ymm4, %ymm3, %ymm3 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm4 + vpsubd %ymm13, %ymm3, %ymm3 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm10, %ymm12 + vpaddd %ymm11, %ymm10, %ymm10 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm10, %ymm10 + vmovsldup %ymm8, %ymm9 # ymm9 = ymm8[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vpsrlq $0x20, %ymm7, %ymm7 + vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7] + vmovsldup %ymm6, %ymm7 # ymm7 = ymm6[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7] + vpsrlq $0x20, %ymm5, %ymm5 + vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7] + vmovsldup %ymm4, %ymm5 # ymm5 = ymm4[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7] + vmovsldup %ymm11, %ymm3 # ymm3 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7] + vpsrlq $0x20, %ymm10, %ymm10 + vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7] + vmovdqa 0x1c0(%rsi), %ymm1 + vmovdqa 0x660(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm5, %ymm13 + vmovshdup %ymm5, %ymm12 # ymm12 = ymm5[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm5, %ymm5 # ymm5 = ymm5[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7] + vpsubd %ymm5, %ymm9, %ymm12 + vpaddd %ymm5, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm5 + vpsubd %ymm13, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm4, %ymm13 + vmovshdup %ymm4, %ymm12 # ymm12 = ymm4[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7] + vpsubd %ymm4, %ymm8, %ymm12 + vpaddd %ymm4, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm4 + vpsubd %ymm13, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm3, %ymm13 + vmovshdup %ymm3, %ymm12 # ymm12 = ymm3[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm3, %ymm3 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm3, %ymm3 # ymm3 = ymm3[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7] + vpsubd %ymm3, %ymm7, %ymm12 + vpaddd %ymm3, %ymm7, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm3 + vpsubd %ymm13, %ymm7, %ymm7 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm6, %ymm12 + vpaddd %ymm6, %ymm11, %ymm6 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm6, %ymm6 + vmovdqa 0x240(%rsi), %ymm1 + vmovdqa 0x6e0(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm13 + vmovshdup %ymm7, %ymm12 # ymm12 = ymm7[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm7, %ymm7 # ymm7 = ymm7[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7] + vpsubd %ymm7, %ymm9, %ymm12 + vpaddd %ymm7, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm7 + vpsubd %ymm13, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm8, %ymm12 + vpaddd %ymm6, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm8, %ymm8 + vmovdqa 0x2c0(%rsi), %ymm1 + vmovdqa 0x760(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm3, %ymm13 + vmovshdup %ymm3, %ymm12 # ymm12 = ymm3[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm3, %ymm3 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm3, %ymm3 # ymm3 = ymm3[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7] + vpsubd %ymm3, %ymm5, %ymm12 + vpaddd %ymm3, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm3 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm4, %ymm12 + vpaddd %ymm4, %ymm11, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm4, %ymm4 + vmovdqa 0x340(%rsi), %ymm1 + vmovdqa 0x7e0(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm9, %ymm12 + vpaddd %ymm8, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm9, %ymm9 + vmovdqa 0x3c0(%rsi), %ymm1 + vmovdqa 0x860(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm7, %ymm12 + vpaddd %ymm6, %ymm7, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm7, %ymm7 + vmovdqa 0x440(%rsi), %ymm1 + vmovdqa 0x8e0(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm4, %ymm13 + vmovshdup %ymm4, %ymm12 # ymm12 = ymm4[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7] + vpsubd %ymm4, %ymm5, %ymm12 + vpaddd %ymm4, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm4 + vpsubd %ymm13, %ymm5, %ymm5 + vmovdqa 0x4c0(%rsi), %ymm1 + vmovdqa 0x960(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm3, %ymm12 + vpaddd %ymm3, %ymm11, %ymm3 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm3, %ymm3 + vmovdqa %ymm9, 0x100(%rdi) + vmovdqa %ymm8, 0x120(%rdi) + vmovdqa %ymm7, 0x140(%rdi) + vmovdqa %ymm6, 0x160(%rdi) + vmovdqa %ymm5, 0x180(%rdi) + vmovdqa %ymm4, 0x1a0(%rdi) + vmovdqa %ymm3, 0x1c0(%rdi) + vmovdqa %ymm11, 0x1e0(%rdi) + vmovdqa 0x200(%rdi), %ymm4 + vmovdqa 0x220(%rdi), %ymm5 + vmovdqa 0x240(%rdi), %ymm6 + vmovdqa 0x260(%rdi), %ymm7 + vmovdqa 0x280(%rdi), %ymm8 + vmovdqa 0x2a0(%rdi), %ymm9 + vmovdqa 0x2c0(%rdi), %ymm10 + vmovdqa 0x2e0(%rdi), %ymm11 + vpbroadcastd 0x98(%rsi), %ymm1 + vpbroadcastd 0x538(%rsi), %ymm2 + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm4, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm9, %ymm13 + vmovshdup %ymm9, %ymm12 # ymm12 = ymm9[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm9, %ymm9 # ymm9 = ymm9[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7] + vpsubd %ymm9, %ymm5, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm9 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm6, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm7, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm7, %ymm7 + vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3] + vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1] + vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3] + vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3] + vmovdqa 0xe0(%rsi), %ymm1 + vmovdqa 0x580(%rsi), %ymm2 + vpmuldq %ymm1, %ymm5, %ymm13 + vmovshdup %ymm5, %ymm12 # ymm12 = ymm5[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm5, %ymm5 # ymm5 = ymm5[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7] + vpsubd %ymm5, %ymm3, %ymm12 + vpaddd %ymm5, %ymm3, %ymm3 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm5 + vpsubd %ymm13, %ymm3, %ymm3 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm8, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm4, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm9, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm9, %ymm9 + vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3] + vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2] + vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3] + vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2] + vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3] + vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3] + vmovdqa 0x160(%rsi), %ymm1 + vmovdqa 0x600(%rsi), %ymm2 + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm7, %ymm12 + vpaddd %ymm7, %ymm8, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm7, %ymm7 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm5, %ymm12 + vpaddd %ymm6, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm4, %ymm13 + vmovshdup %ymm4, %ymm12 # ymm12 = ymm4[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7] + vpsubd %ymm4, %ymm3, %ymm12 + vpaddd %ymm4, %ymm3, %ymm3 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm4 + vpsubd %ymm13, %ymm3, %ymm3 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm10, %ymm12 + vpaddd %ymm11, %ymm10, %ymm10 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm10, %ymm10 + vmovsldup %ymm8, %ymm9 # ymm9 = ymm8[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vpsrlq $0x20, %ymm7, %ymm7 + vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7] + vmovsldup %ymm6, %ymm7 # ymm7 = ymm6[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7] + vpsrlq $0x20, %ymm5, %ymm5 + vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7] + vmovsldup %ymm4, %ymm5 # ymm5 = ymm4[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7] + vmovsldup %ymm11, %ymm3 # ymm3 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7] + vpsrlq $0x20, %ymm10, %ymm10 + vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7] + vmovdqa 0x1e0(%rsi), %ymm1 + vmovdqa 0x680(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm5, %ymm13 + vmovshdup %ymm5, %ymm12 # ymm12 = ymm5[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm5, %ymm5 # ymm5 = ymm5[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7] + vpsubd %ymm5, %ymm9, %ymm12 + vpaddd %ymm5, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm5 + vpsubd %ymm13, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm4, %ymm13 + vmovshdup %ymm4, %ymm12 # ymm12 = ymm4[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7] + vpsubd %ymm4, %ymm8, %ymm12 + vpaddd %ymm4, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm4 + vpsubd %ymm13, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm3, %ymm13 + vmovshdup %ymm3, %ymm12 # ymm12 = ymm3[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm3, %ymm3 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm3, %ymm3 # ymm3 = ymm3[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7] + vpsubd %ymm3, %ymm7, %ymm12 + vpaddd %ymm3, %ymm7, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm3 + vpsubd %ymm13, %ymm7, %ymm7 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm6, %ymm12 + vpaddd %ymm6, %ymm11, %ymm6 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm6, %ymm6 + vmovdqa 0x260(%rsi), %ymm1 + vmovdqa 0x700(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm13 + vmovshdup %ymm7, %ymm12 # ymm12 = ymm7[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm7, %ymm7 # ymm7 = ymm7[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7] + vpsubd %ymm7, %ymm9, %ymm12 + vpaddd %ymm7, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm7 + vpsubd %ymm13, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm8, %ymm12 + vpaddd %ymm6, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm8, %ymm8 + vmovdqa 0x2e0(%rsi), %ymm1 + vmovdqa 0x780(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm3, %ymm13 + vmovshdup %ymm3, %ymm12 # ymm12 = ymm3[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm3, %ymm3 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm3, %ymm3 # ymm3 = ymm3[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7] + vpsubd %ymm3, %ymm5, %ymm12 + vpaddd %ymm3, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm3 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm4, %ymm12 + vpaddd %ymm4, %ymm11, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm4, %ymm4 + vmovdqa 0x360(%rsi), %ymm1 + vmovdqa 0x800(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm9, %ymm12 + vpaddd %ymm8, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm9, %ymm9 + vmovdqa 0x3e0(%rsi), %ymm1 + vmovdqa 0x880(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm7, %ymm12 + vpaddd %ymm6, %ymm7, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm7, %ymm7 + vmovdqa 0x460(%rsi), %ymm1 + vmovdqa 0x900(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm4, %ymm13 + vmovshdup %ymm4, %ymm12 # ymm12 = ymm4[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7] + vpsubd %ymm4, %ymm5, %ymm12 + vpaddd %ymm4, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm4 + vpsubd %ymm13, %ymm5, %ymm5 + vmovdqa 0x4e0(%rsi), %ymm1 + vmovdqa 0x980(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm3, %ymm12 + vpaddd %ymm3, %ymm11, %ymm3 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm3, %ymm3 + vmovdqa %ymm9, 0x200(%rdi) + vmovdqa %ymm8, 0x220(%rdi) + vmovdqa %ymm7, 0x240(%rdi) + vmovdqa %ymm6, 0x260(%rdi) + vmovdqa %ymm5, 0x280(%rdi) + vmovdqa %ymm4, 0x2a0(%rdi) + vmovdqa %ymm3, 0x2c0(%rdi) + vmovdqa %ymm11, 0x2e0(%rdi) + vmovdqa 0x300(%rdi), %ymm4 + vmovdqa 0x320(%rdi), %ymm5 + vmovdqa 0x340(%rdi), %ymm6 + vmovdqa 0x360(%rdi), %ymm7 + vmovdqa 0x380(%rdi), %ymm8 + vmovdqa 0x3a0(%rdi), %ymm9 + vmovdqa 0x3c0(%rdi), %ymm10 + vmovdqa 0x3e0(%rdi), %ymm11 + vpbroadcastd 0x9c(%rsi), %ymm1 + vpbroadcastd 0x53c(%rsi), %ymm2 + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm4, %ymm12 + vpaddd %ymm4, %ymm8, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm9, %ymm13 + vmovshdup %ymm9, %ymm12 # ymm12 = ymm9[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm9, %ymm9 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm9, %ymm9 # ymm9 = ymm9[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm9, %ymm9 # ymm9 = ymm9[0],ymm12[1],ymm9[2],ymm12[3],ymm9[4],ymm12[5],ymm9[6],ymm12[7] + vpsubd %ymm9, %ymm5, %ymm12 + vpaddd %ymm5, %ymm9, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm9 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm6, %ymm12 + vpaddd %ymm6, %ymm10, %ymm6 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm6, %ymm6 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm7, %ymm12 + vpaddd %ymm7, %ymm11, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm7, %ymm7 + vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3] + vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1] + vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3] + vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3] + vmovdqa 0x100(%rsi), %ymm1 + vmovdqa 0x5a0(%rsi), %ymm2 + vpmuldq %ymm1, %ymm5, %ymm13 + vmovshdup %ymm5, %ymm12 # ymm12 = ymm5[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm5, %ymm5 # ymm5 = ymm5[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7] + vpsubd %ymm5, %ymm3, %ymm12 + vpaddd %ymm5, %ymm3, %ymm3 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm5 + vpsubd %ymm13, %ymm3, %ymm3 + vpmuldq %ymm1, %ymm10, %ymm13 + vmovshdup %ymm10, %ymm12 # ymm12 = ymm10[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm10, %ymm10 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm10, %ymm10 # ymm10 = ymm10[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm10, %ymm10 # ymm10 = ymm10[0],ymm12[1],ymm10[2],ymm12[3],ymm10[4],ymm12[5],ymm10[6],ymm12[7] + vpsubd %ymm10, %ymm8, %ymm12 + vpaddd %ymm10, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm10 + vpsubd %ymm13, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm4, %ymm12 + vpaddd %ymm6, %ymm4, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm4, %ymm4 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm9, %ymm12 + vpaddd %ymm11, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm9, %ymm9 + vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3] + vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2] + vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3] + vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2] + vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3] + vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3] + vmovdqa 0x180(%rsi), %ymm1 + vmovdqa 0x620(%rsi), %ymm2 + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm7, %ymm12 + vpaddd %ymm7, %ymm8, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm7, %ymm7 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm5, %ymm12 + vpaddd %ymm6, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm4, %ymm13 + vmovshdup %ymm4, %ymm12 # ymm12 = ymm4[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7] + vpsubd %ymm4, %ymm3, %ymm12 + vpaddd %ymm4, %ymm3, %ymm3 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm4 + vpsubd %ymm13, %ymm3, %ymm3 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm2, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm10, %ymm12 + vpaddd %ymm11, %ymm10, %ymm10 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm10, %ymm10 + vmovsldup %ymm8, %ymm9 # ymm9 = ymm8[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vpsrlq $0x20, %ymm7, %ymm7 + vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7] + vmovsldup %ymm6, %ymm7 # ymm7 = ymm6[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7] + vpsrlq $0x20, %ymm5, %ymm5 + vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7] + vmovsldup %ymm4, %ymm5 # ymm5 = ymm4[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7] + vmovsldup %ymm11, %ymm3 # ymm3 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7] + vpsrlq $0x20, %ymm10, %ymm10 + vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7] + vmovdqa 0x200(%rsi), %ymm1 + vmovdqa 0x6a0(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm5, %ymm13 + vmovshdup %ymm5, %ymm12 # ymm12 = ymm5[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm5, %ymm5 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm5, %ymm5 # ymm5 = ymm5[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm5, %ymm5 # ymm5 = ymm5[0],ymm12[1],ymm5[2],ymm12[3],ymm5[4],ymm12[5],ymm5[6],ymm12[7] + vpsubd %ymm5, %ymm9, %ymm12 + vpaddd %ymm5, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm5 + vpsubd %ymm13, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm4, %ymm13 + vmovshdup %ymm4, %ymm12 # ymm12 = ymm4[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7] + vpsubd %ymm4, %ymm8, %ymm12 + vpaddd %ymm4, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm4 + vpsubd %ymm13, %ymm8, %ymm8 + vpmuldq %ymm1, %ymm3, %ymm13 + vmovshdup %ymm3, %ymm12 # ymm12 = ymm3[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm3, %ymm3 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm3, %ymm3 # ymm3 = ymm3[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7] + vpsubd %ymm3, %ymm7, %ymm12 + vpaddd %ymm3, %ymm7, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm3 + vpsubd %ymm13, %ymm7, %ymm7 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm6, %ymm12 + vpaddd %ymm6, %ymm11, %ymm6 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm6, %ymm6 + vmovdqa 0x280(%rsi), %ymm1 + vmovdqa 0x720(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm7, %ymm13 + vmovshdup %ymm7, %ymm12 # ymm12 = ymm7[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm7, %ymm7 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm7, %ymm7 # ymm7 = ymm7[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm7, %ymm7 # ymm7 = ymm7[0],ymm12[1],ymm7[2],ymm12[3],ymm7[4],ymm12[5],ymm7[6],ymm12[7] + vpsubd %ymm7, %ymm9, %ymm12 + vpaddd %ymm7, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm7 + vpsubd %ymm13, %ymm9, %ymm9 + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm8, %ymm12 + vpaddd %ymm6, %ymm8, %ymm8 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm8, %ymm8 + vmovdqa 0x300(%rsi), %ymm1 + vmovdqa 0x7a0(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm3, %ymm13 + vmovshdup %ymm3, %ymm12 # ymm12 = ymm3[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm3, %ymm3 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm3, %ymm3 # ymm3 = ymm3[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm3, %ymm3 # ymm3 = ymm3[0],ymm12[1],ymm3[2],ymm12[3],ymm3[4],ymm12[5],ymm3[6],ymm12[7] + vpsubd %ymm3, %ymm5, %ymm12 + vpaddd %ymm3, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm3 + vpsubd %ymm13, %ymm5, %ymm5 + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm4, %ymm12 + vpaddd %ymm4, %ymm11, %ymm4 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm4, %ymm4 + vmovdqa 0x380(%rsi), %ymm1 + vmovdqa 0x820(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm8, %ymm13 + vmovshdup %ymm8, %ymm12 # ymm12 = ymm8[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm8, %ymm8 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm8, %ymm8 # ymm8 = ymm8[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm8, %ymm8 # ymm8 = ymm8[0],ymm12[1],ymm8[2],ymm12[3],ymm8[4],ymm12[5],ymm8[6],ymm12[7] + vpsubd %ymm8, %ymm9, %ymm12 + vpaddd %ymm8, %ymm9, %ymm9 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm8 + vpsubd %ymm13, %ymm9, %ymm9 + vmovdqa 0x400(%rsi), %ymm1 + vmovdqa 0x8a0(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm6, %ymm13 + vmovshdup %ymm6, %ymm12 # ymm12 = ymm6[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm6, %ymm6 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm12[1],ymm6[2],ymm12[3],ymm6[4],ymm12[5],ymm6[6],ymm12[7] + vpsubd %ymm6, %ymm7, %ymm12 + vpaddd %ymm6, %ymm7, %ymm7 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm6 + vpsubd %ymm13, %ymm7, %ymm7 + vmovdqa 0x480(%rsi), %ymm1 + vmovdqa 0x920(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm4, %ymm13 + vmovshdup %ymm4, %ymm12 # ymm12 = ymm4[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm4, %ymm4 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm12[1],ymm4[2],ymm12[3],ymm4[4],ymm12[5],ymm4[6],ymm12[7] + vpsubd %ymm4, %ymm5, %ymm12 + vpaddd %ymm4, %ymm5, %ymm5 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm4 + vpsubd %ymm13, %ymm5, %ymm5 + vmovdqa 0x500(%rsi), %ymm1 + vmovdqa 0x9a0(%rsi), %ymm2 + vpsrlq $0x20, %ymm1, %ymm10 + vmovshdup %ymm2, %ymm15 # ymm15 = ymm2[1,1,3,3,5,5,7,7] + vpmuldq %ymm1, %ymm11, %ymm13 + vmovshdup %ymm11, %ymm12 # ymm12 = ymm11[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm12, %ymm14 + vpmuldq %ymm2, %ymm11, %ymm11 + vpmuldq %ymm15, %ymm12, %ymm12 + vpmuldq %ymm0, %ymm13, %ymm13 + vpmuldq %ymm0, %ymm14, %ymm14 + vmovshdup %ymm11, %ymm11 # ymm11 = ymm11[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm12, %ymm11, %ymm11 # ymm11 = ymm11[0],ymm12[1],ymm11[2],ymm12[3],ymm11[4],ymm12[5],ymm11[6],ymm12[7] + vpsubd %ymm11, %ymm3, %ymm12 + vpaddd %ymm3, %ymm11, %ymm3 + vmovshdup %ymm13, %ymm13 # ymm13 = ymm13[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm14, %ymm13, %ymm13 # ymm13 = ymm13[0],ymm14[1],ymm13[2],ymm14[3],ymm13[4],ymm14[5],ymm13[6],ymm14[7] + vpaddd %ymm13, %ymm12, %ymm11 + vpsubd %ymm13, %ymm3, %ymm3 + vmovdqa %ymm9, 0x300(%rdi) + vmovdqa %ymm8, 0x320(%rdi) + vmovdqa %ymm7, 0x340(%rdi) + vmovdqa %ymm6, 0x360(%rdi) + vmovdqa %ymm5, 0x380(%rdi) + vmovdqa %ymm4, 0x3a0(%rdi) + vmovdqa %ymm3, 0x3c0(%rdi) + vmovdqa %ymm11, 0x3e0(%rdi) + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(ntt_avx2_asm) + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_nttunpack_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_nttunpack_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_nttunpack_avx2_asm.S @@ -0,0 +1,254 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + +/*yaml + Name: nttunpack_avx2_asm + Description: x86_64 AVX2 NTT coefficient unpacking/permutation + Signature: void mld_nttunpack_avx2_asm(int32_t *r) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t *r + description: Input/output polynomial (256 x int32_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_nttunpack_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(nttunpack_avx2_asm) +MLD_ASM_FN_SYMBOL(nttunpack_avx2_asm) + + .cfi_startproc + vmovdqa (%rdi), %ymm4 + vmovdqa 0x20(%rdi), %ymm5 + vmovdqa 0x40(%rdi), %ymm6 + vmovdqa 0x60(%rdi), %ymm7 + vmovdqa 0x80(%rdi), %ymm8 + vmovdqa 0xa0(%rdi), %ymm9 + vmovdqa 0xc0(%rdi), %ymm10 + vmovdqa 0xe0(%rdi), %ymm11 + vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3] + vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1] + vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3] + vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3] + vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3] + vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2] + vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3] + vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2] + vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3] + vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3] + vmovsldup %ymm8, %ymm9 # ymm9 = ymm8[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vpsrlq $0x20, %ymm7, %ymm7 + vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7] + vmovsldup %ymm6, %ymm7 # ymm7 = ymm6[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7] + vpsrlq $0x20, %ymm5, %ymm5 + vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7] + vmovsldup %ymm4, %ymm5 # ymm5 = ymm4[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7] + vmovsldup %ymm11, %ymm3 # ymm3 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7] + vpsrlq $0x20, %ymm10, %ymm10 + vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7] + vmovdqa %ymm9, (%rdi) + vmovdqa %ymm8, 0x20(%rdi) + vmovdqa %ymm7, 0x40(%rdi) + vmovdqa %ymm6, 0x60(%rdi) + vmovdqa %ymm5, 0x80(%rdi) + vmovdqa %ymm4, 0xa0(%rdi) + vmovdqa %ymm3, 0xc0(%rdi) + vmovdqa %ymm11, 0xe0(%rdi) + vmovdqa 0x100(%rdi), %ymm4 + vmovdqa 0x120(%rdi), %ymm5 + vmovdqa 0x140(%rdi), %ymm6 + vmovdqa 0x160(%rdi), %ymm7 + vmovdqa 0x180(%rdi), %ymm8 + vmovdqa 0x1a0(%rdi), %ymm9 + vmovdqa 0x1c0(%rdi), %ymm10 + vmovdqa 0x1e0(%rdi), %ymm11 + vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3] + vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1] + vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3] + vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3] + vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3] + vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2] + vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3] + vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2] + vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3] + vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3] + vmovsldup %ymm8, %ymm9 # ymm9 = ymm8[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vpsrlq $0x20, %ymm7, %ymm7 + vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7] + vmovsldup %ymm6, %ymm7 # ymm7 = ymm6[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7] + vpsrlq $0x20, %ymm5, %ymm5 + vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7] + vmovsldup %ymm4, %ymm5 # ymm5 = ymm4[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7] + vmovsldup %ymm11, %ymm3 # ymm3 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7] + vpsrlq $0x20, %ymm10, %ymm10 + vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7] + vmovdqa %ymm9, 0x100(%rdi) + vmovdqa %ymm8, 0x120(%rdi) + vmovdqa %ymm7, 0x140(%rdi) + vmovdqa %ymm6, 0x160(%rdi) + vmovdqa %ymm5, 0x180(%rdi) + vmovdqa %ymm4, 0x1a0(%rdi) + vmovdqa %ymm3, 0x1c0(%rdi) + vmovdqa %ymm11, 0x1e0(%rdi) + vmovdqa 0x200(%rdi), %ymm4 + vmovdqa 0x220(%rdi), %ymm5 + vmovdqa 0x240(%rdi), %ymm6 + vmovdqa 0x260(%rdi), %ymm7 + vmovdqa 0x280(%rdi), %ymm8 + vmovdqa 0x2a0(%rdi), %ymm9 + vmovdqa 0x2c0(%rdi), %ymm10 + vmovdqa 0x2e0(%rdi), %ymm11 + vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3] + vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1] + vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3] + vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3] + vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3] + vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2] + vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3] + vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2] + vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3] + vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3] + vmovsldup %ymm8, %ymm9 # ymm9 = ymm8[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vpsrlq $0x20, %ymm7, %ymm7 + vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7] + vmovsldup %ymm6, %ymm7 # ymm7 = ymm6[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7] + vpsrlq $0x20, %ymm5, %ymm5 + vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7] + vmovsldup %ymm4, %ymm5 # ymm5 = ymm4[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7] + vmovsldup %ymm11, %ymm3 # ymm3 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7] + vpsrlq $0x20, %ymm10, %ymm10 + vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7] + vmovdqa %ymm9, 0x200(%rdi) + vmovdqa %ymm8, 0x220(%rdi) + vmovdqa %ymm7, 0x240(%rdi) + vmovdqa %ymm6, 0x260(%rdi) + vmovdqa %ymm5, 0x280(%rdi) + vmovdqa %ymm4, 0x2a0(%rdi) + vmovdqa %ymm3, 0x2c0(%rdi) + vmovdqa %ymm11, 0x2e0(%rdi) + vmovdqa 0x300(%rdi), %ymm4 + vmovdqa 0x320(%rdi), %ymm5 + vmovdqa 0x340(%rdi), %ymm6 + vmovdqa 0x360(%rdi), %ymm7 + vmovdqa 0x380(%rdi), %ymm8 + vmovdqa 0x3a0(%rdi), %ymm9 + vmovdqa 0x3c0(%rdi), %ymm10 + vmovdqa 0x3e0(%rdi), %ymm11 + vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3] + vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1] + vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3] + vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3] + vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3] + vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2] + vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3] + vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2] + vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3] + vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3] + vmovsldup %ymm8, %ymm9 # ymm9 = ymm8[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vpsrlq $0x20, %ymm7, %ymm7 + vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7] + vmovsldup %ymm6, %ymm7 # ymm7 = ymm6[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7] + vpsrlq $0x20, %ymm5, %ymm5 + vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7] + vmovsldup %ymm4, %ymm5 # ymm5 = ymm4[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7] + vmovsldup %ymm11, %ymm3 # ymm3 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7] + vpsrlq $0x20, %ymm10, %ymm10 + vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7] + vmovdqa %ymm9, 0x300(%rdi) + vmovdqa %ymm8, 0x320(%rdi) + vmovdqa %ymm7, 0x340(%rdi) + vmovdqa %ymm6, 0x360(%rdi) + vmovdqa %ymm5, 0x380(%rdi) + vmovdqa %ymm4, 0x3a0(%rdi) + vmovdqa %ymm3, 0x3c0(%rdi) + vmovdqa %ymm11, 0x3e0(%rdi) + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(nttunpack_avx2_asm) + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_acc_l4_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_acc_l4_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_acc_l4_avx2_asm.S @@ -0,0 +1,173 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + +/*yaml + Name: pointwise_acc_l4_avx2_asm + Description: x86_64 AVX2 pointwise multiply-accumulate of length-4 polynomial vectors + Signature: void mld_pointwise_acc_l4_avx2_asm(int32_t *c, const int32_t a[4][256], const int32_t b[4][256], const int32_t *qdata) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t *c + description: Output polynomial (256 x int32_t) + rsi: + type: buffer + size_bytes: 4096 + permissions: read-only + c_parameter: const int32_t a[4][256] + description: Input polynomial vector a (4 x 256 x int32_t) + rdx: + type: buffer + size_bytes: 4096 + permissions: read-only + c_parameter: const int32_t b[4][256] + description: Input polynomial vector b (4 x 256 x int32_t) + rcx: + type: buffer + size_bytes: 2496 + permissions: read-only + c_parameter: const int32_t *qdata + description: Precomputed constants (624 x int32_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 4) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_pointwise_acc_l4_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(pointwise_acc_l4_avx2_asm) +MLD_ASM_FN_SYMBOL(pointwise_acc_l4_avx2_asm) + + .cfi_startproc + vmovdqa 0x20(%rcx), %ymm0 + vmovdqa (%rcx), %ymm1 + xorl %eax, %eax + +Lmld_pointwise_acc_l4_avx2_looptop2: + vmovdqa (%rsi), %ymm6 + vmovdqa 0x20(%rsi), %ymm8 + vmovdqa (%rdx), %ymm10 + vmovdqa 0x20(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vmovdqa %ymm6, %ymm2 + vmovdqa %ymm7, %ymm3 + vmovdqa %ymm8, %ymm4 + vmovdqa %ymm9, %ymm5 + vmovdqa 0x400(%rsi), %ymm6 + vmovdqa 0x420(%rsi), %ymm8 + vmovdqa 0x400(%rdx), %ymm10 + vmovdqa 0x420(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vpaddq %ymm2, %ymm6, %ymm2 + vpaddq %ymm3, %ymm7, %ymm3 + vpaddq %ymm4, %ymm8, %ymm4 + vpaddq %ymm5, %ymm9, %ymm5 + vmovdqa 0x800(%rsi), %ymm6 + vmovdqa 0x820(%rsi), %ymm8 + vmovdqa 0x800(%rdx), %ymm10 + vmovdqa 0x820(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vpaddq %ymm2, %ymm6, %ymm2 + vpaddq %ymm3, %ymm7, %ymm3 + vpaddq %ymm4, %ymm8, %ymm4 + vpaddq %ymm5, %ymm9, %ymm5 + vmovdqa 0xc00(%rsi), %ymm6 + vmovdqa 0xc20(%rsi), %ymm8 + vmovdqa 0xc00(%rdx), %ymm10 + vmovdqa 0xc20(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vpaddq %ymm2, %ymm6, %ymm2 + vpaddq %ymm3, %ymm7, %ymm3 + vpaddq %ymm4, %ymm8, %ymm4 + vpaddq %ymm5, %ymm9, %ymm5 + vpmuldq %ymm2, %ymm0, %ymm6 + vpmuldq %ymm3, %ymm0, %ymm7 + vpmuldq %ymm4, %ymm0, %ymm8 + vpmuldq %ymm5, %ymm0, %ymm9 + vpmuldq %ymm6, %ymm1, %ymm6 + vpmuldq %ymm7, %ymm1, %ymm7 + vpmuldq %ymm8, %ymm1, %ymm8 + vpmuldq %ymm9, %ymm1, %ymm9 + vpsubq %ymm6, %ymm2, %ymm2 + vpsubq %ymm7, %ymm3, %ymm3 + vpsubq %ymm8, %ymm4, %ymm4 + vpsubq %ymm9, %ymm5, %ymm5 + vpsrlq $0x20, %ymm2, %ymm2 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm3, %ymm2, %ymm2 # ymm2 = ymm2[0],ymm3[1],ymm2[2],ymm3[3],ymm2[4],ymm3[5],ymm2[6],ymm3[7] + vpblendd $0xaa, %ymm5, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7] + vmovdqa %ymm2, (%rdi) + vmovdqa %ymm4, 0x20(%rdi) + addq $0x40, %rsi + addq $0x40, %rdx + addq $0x40, %rdi + addl $0x1, %eax + cmpl $0x10, %eax + jb Lmld_pointwise_acc_l4_avx2_looptop2 + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(pointwise_acc_l4_avx2_asm) + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + && (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 4) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_acc_l5_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_acc_l5_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_acc_l5_avx2_asm.S @@ -0,0 +1,189 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + +/*yaml + Name: pointwise_acc_l5_avx2_asm + Description: x86_64 AVX2 pointwise multiply-accumulate of length-5 polynomial vectors + Signature: void mld_pointwise_acc_l5_avx2_asm(int32_t *c, const int32_t a[5][256], const int32_t b[5][256], const int32_t *qdata) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t *c + description: Output polynomial (256 x int32_t) + rsi: + type: buffer + size_bytes: 5120 + permissions: read-only + c_parameter: const int32_t a[5][256] + description: Input polynomial vector a (5 x 256 x int32_t) + rdx: + type: buffer + size_bytes: 5120 + permissions: read-only + c_parameter: const int32_t b[5][256] + description: Input polynomial vector b (5 x 256 x int32_t) + rcx: + type: buffer + size_bytes: 2496 + permissions: read-only + c_parameter: const int32_t *qdata + description: Precomputed constants (624 x int32_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 5) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_pointwise_acc_l5_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(pointwise_acc_l5_avx2_asm) +MLD_ASM_FN_SYMBOL(pointwise_acc_l5_avx2_asm) + + .cfi_startproc + vmovdqa 0x20(%rcx), %ymm0 + vmovdqa (%rcx), %ymm1 + xorl %eax, %eax + +Lmld_pointwise_acc_l5_avx2_looptop2: + vmovdqa (%rsi), %ymm6 + vmovdqa 0x20(%rsi), %ymm8 + vmovdqa (%rdx), %ymm10 + vmovdqa 0x20(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vmovdqa %ymm6, %ymm2 + vmovdqa %ymm7, %ymm3 + vmovdqa %ymm8, %ymm4 + vmovdqa %ymm9, %ymm5 + vmovdqa 0x400(%rsi), %ymm6 + vmovdqa 0x420(%rsi), %ymm8 + vmovdqa 0x400(%rdx), %ymm10 + vmovdqa 0x420(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vpaddq %ymm2, %ymm6, %ymm2 + vpaddq %ymm3, %ymm7, %ymm3 + vpaddq %ymm4, %ymm8, %ymm4 + vpaddq %ymm5, %ymm9, %ymm5 + vmovdqa 0x800(%rsi), %ymm6 + vmovdqa 0x820(%rsi), %ymm8 + vmovdqa 0x800(%rdx), %ymm10 + vmovdqa 0x820(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vpaddq %ymm2, %ymm6, %ymm2 + vpaddq %ymm3, %ymm7, %ymm3 + vpaddq %ymm4, %ymm8, %ymm4 + vpaddq %ymm5, %ymm9, %ymm5 + vmovdqa 0xc00(%rsi), %ymm6 + vmovdqa 0xc20(%rsi), %ymm8 + vmovdqa 0xc00(%rdx), %ymm10 + vmovdqa 0xc20(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vpaddq %ymm2, %ymm6, %ymm2 + vpaddq %ymm3, %ymm7, %ymm3 + vpaddq %ymm4, %ymm8, %ymm4 + vpaddq %ymm5, %ymm9, %ymm5 + vmovdqa 0x1000(%rsi), %ymm6 + vmovdqa 0x1020(%rsi), %ymm8 + vmovdqa 0x1000(%rdx), %ymm10 + vmovdqa 0x1020(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vpaddq %ymm2, %ymm6, %ymm2 + vpaddq %ymm3, %ymm7, %ymm3 + vpaddq %ymm4, %ymm8, %ymm4 + vpaddq %ymm5, %ymm9, %ymm5 + vpmuldq %ymm2, %ymm0, %ymm6 + vpmuldq %ymm3, %ymm0, %ymm7 + vpmuldq %ymm4, %ymm0, %ymm8 + vpmuldq %ymm5, %ymm0, %ymm9 + vpmuldq %ymm6, %ymm1, %ymm6 + vpmuldq %ymm7, %ymm1, %ymm7 + vpmuldq %ymm8, %ymm1, %ymm8 + vpmuldq %ymm9, %ymm1, %ymm9 + vpsubq %ymm6, %ymm2, %ymm2 + vpsubq %ymm7, %ymm3, %ymm3 + vpsubq %ymm8, %ymm4, %ymm4 + vpsubq %ymm9, %ymm5, %ymm5 + vpsrlq $0x20, %ymm2, %ymm2 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm3, %ymm2, %ymm2 # ymm2 = ymm2[0],ymm3[1],ymm2[2],ymm3[3],ymm2[4],ymm3[5],ymm2[6],ymm3[7] + vpblendd $0xaa, %ymm5, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7] + vmovdqa %ymm2, (%rdi) + vmovdqa %ymm4, 0x20(%rdi) + addq $0x40, %rsi + addq $0x40, %rdx + addq $0x40, %rdi + addl $0x1, %eax + cmpl $0x10, %eax + jb Lmld_pointwise_acc_l5_avx2_looptop2 + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(pointwise_acc_l5_avx2_asm) + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + && (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 5) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_acc_l7_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_acc_l7_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_acc_l7_avx2_asm.S @@ -0,0 +1,221 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + +/*yaml + Name: pointwise_acc_l7_avx2_asm + Description: x86_64 AVX2 pointwise multiply-accumulate of length-7 polynomial vectors + Signature: void mld_pointwise_acc_l7_avx2_asm(int32_t *c, const int32_t a[7][256], const int32_t b[7][256], const int32_t *qdata) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t *c + description: Output polynomial (256 x int32_t) + rsi: + type: buffer + size_bytes: 7168 + permissions: read-only + c_parameter: const int32_t a[7][256] + description: Input polynomial vector a (7 x 256 x int32_t) + rdx: + type: buffer + size_bytes: 7168 + permissions: read-only + c_parameter: const int32_t b[7][256] + description: Input polynomial vector b (7 x 256 x int32_t) + rcx: + type: buffer + size_bytes: 2496 + permissions: read-only + c_parameter: const int32_t *qdata + description: Precomputed constants (624 x int32_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLDSA_L == 7) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_pointwise_acc_l7_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(pointwise_acc_l7_avx2_asm) +MLD_ASM_FN_SYMBOL(pointwise_acc_l7_avx2_asm) + + .cfi_startproc + vmovdqa 0x20(%rcx), %ymm0 + vmovdqa (%rcx), %ymm1 + xorl %eax, %eax + +Lmld_pointwise_acc_l7_avx2_looptop2: + vmovdqa (%rsi), %ymm6 + vmovdqa 0x20(%rsi), %ymm8 + vmovdqa (%rdx), %ymm10 + vmovdqa 0x20(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vmovdqa %ymm6, %ymm2 + vmovdqa %ymm7, %ymm3 + vmovdqa %ymm8, %ymm4 + vmovdqa %ymm9, %ymm5 + vmovdqa 0x400(%rsi), %ymm6 + vmovdqa 0x420(%rsi), %ymm8 + vmovdqa 0x400(%rdx), %ymm10 + vmovdqa 0x420(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vpaddq %ymm2, %ymm6, %ymm2 + vpaddq %ymm3, %ymm7, %ymm3 + vpaddq %ymm4, %ymm8, %ymm4 + vpaddq %ymm5, %ymm9, %ymm5 + vmovdqa 0x800(%rsi), %ymm6 + vmovdqa 0x820(%rsi), %ymm8 + vmovdqa 0x800(%rdx), %ymm10 + vmovdqa 0x820(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vpaddq %ymm2, %ymm6, %ymm2 + vpaddq %ymm3, %ymm7, %ymm3 + vpaddq %ymm4, %ymm8, %ymm4 + vpaddq %ymm5, %ymm9, %ymm5 + vmovdqa 0xc00(%rsi), %ymm6 + vmovdqa 0xc20(%rsi), %ymm8 + vmovdqa 0xc00(%rdx), %ymm10 + vmovdqa 0xc20(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vpaddq %ymm2, %ymm6, %ymm2 + vpaddq %ymm3, %ymm7, %ymm3 + vpaddq %ymm4, %ymm8, %ymm4 + vpaddq %ymm5, %ymm9, %ymm5 + vmovdqa 0x1000(%rsi), %ymm6 + vmovdqa 0x1020(%rsi), %ymm8 + vmovdqa 0x1000(%rdx), %ymm10 + vmovdqa 0x1020(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vpaddq %ymm2, %ymm6, %ymm2 + vpaddq %ymm3, %ymm7, %ymm3 + vpaddq %ymm4, %ymm8, %ymm4 + vpaddq %ymm5, %ymm9, %ymm5 + vmovdqa 0x1400(%rsi), %ymm6 + vmovdqa 0x1420(%rsi), %ymm8 + vmovdqa 0x1400(%rdx), %ymm10 + vmovdqa 0x1420(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vpaddq %ymm2, %ymm6, %ymm2 + vpaddq %ymm3, %ymm7, %ymm3 + vpaddq %ymm4, %ymm8, %ymm4 + vpaddq %ymm5, %ymm9, %ymm5 + vmovdqa 0x1800(%rsi), %ymm6 + vmovdqa 0x1820(%rsi), %ymm8 + vmovdqa 0x1800(%rdx), %ymm10 + vmovdqa 0x1820(%rdx), %ymm12 + vpsrlq $0x20, %ymm6, %ymm7 + vpsrlq $0x20, %ymm8, %ymm9 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm6, %ymm6 + vpmuldq %ymm11, %ymm7, %ymm7 + vpmuldq %ymm12, %ymm8, %ymm8 + vpmuldq %ymm13, %ymm9, %ymm9 + vpaddq %ymm2, %ymm6, %ymm2 + vpaddq %ymm3, %ymm7, %ymm3 + vpaddq %ymm4, %ymm8, %ymm4 + vpaddq %ymm5, %ymm9, %ymm5 + vpmuldq %ymm2, %ymm0, %ymm6 + vpmuldq %ymm3, %ymm0, %ymm7 + vpmuldq %ymm4, %ymm0, %ymm8 + vpmuldq %ymm5, %ymm0, %ymm9 + vpmuldq %ymm6, %ymm1, %ymm6 + vpmuldq %ymm7, %ymm1, %ymm7 + vpmuldq %ymm8, %ymm1, %ymm8 + vpmuldq %ymm9, %ymm1, %ymm9 + vpsubq %ymm6, %ymm2, %ymm2 + vpsubq %ymm7, %ymm3, %ymm3 + vpsubq %ymm8, %ymm4, %ymm4 + vpsubq %ymm9, %ymm5, %ymm5 + vpsrlq $0x20, %ymm2, %ymm2 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm3, %ymm2, %ymm2 # ymm2 = ymm2[0],ymm3[1],ymm2[2],ymm3[3],ymm2[4],ymm3[5],ymm2[6],ymm3[7] + vpblendd $0xaa, %ymm5, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7] + vmovdqa %ymm2, (%rdi) + vmovdqa %ymm4, 0x20(%rdi) + addq $0x40, %rsi + addq $0x40, %rdx + addq $0x40, %rdi + addl $0x1, %eax + cmpl $0x10, %eax + jb Lmld_pointwise_acc_l7_avx2_looptop2 + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(pointwise_acc_l7_avx2_asm) + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + && (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLDSA_L == 7) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_pointwise_avx2_asm.S @@ -0,0 +1,158 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + +/*yaml + Name: pointwise_avx2_asm + Description: x86_64 AVX2 pointwise Montgomery multiplication + Signature: void mld_pointwise_avx2_asm(int32_t *a, const int32_t *b, const int32_t *qdata) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t *a + description: Input/output polynomial (256 x int32_t) + rsi: + type: buffer + size_bytes: 1024 + permissions: read-only + c_parameter: const int32_t *b + description: Input polynomial (256 x int32_t) + rdx: + type: buffer + size_bytes: 2496 + permissions: read-only + c_parameter: const int32_t *qdata + description: Precomputed constants (624 x int32_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_pointwise_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(pointwise_avx2_asm) +MLD_ASM_FN_SYMBOL(pointwise_avx2_asm) + + .cfi_startproc + vmovdqa 0x20(%rdx), %ymm0 + vmovdqa (%rdx), %ymm1 + xorl %eax, %eax + +Lmld_pointwise_avx2_looptop1: + vmovdqa (%rdi), %ymm2 + vmovdqa 0x20(%rdi), %ymm4 + vmovdqa 0x40(%rdi), %ymm6 + vmovdqa (%rsi), %ymm10 + vmovdqa 0x20(%rsi), %ymm12 + vmovdqa 0x40(%rsi), %ymm14 + vpsrlq $0x20, %ymm2, %ymm3 + vpsrlq $0x20, %ymm4, %ymm5 + vmovshdup %ymm6, %ymm7 # ymm7 = ymm6[1,1,3,3,5,5,7,7] + vpsrlq $0x20, %ymm10, %ymm11 + vpsrlq $0x20, %ymm12, %ymm13 + vmovshdup %ymm14, %ymm15 # ymm15 = ymm14[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm2, %ymm2 + vpmuldq %ymm11, %ymm3, %ymm3 + vpmuldq %ymm12, %ymm4, %ymm4 + vpmuldq %ymm13, %ymm5, %ymm5 + vpmuldq %ymm14, %ymm6, %ymm6 + vpmuldq %ymm15, %ymm7, %ymm7 + vpmuldq %ymm2, %ymm0, %ymm10 + vpmuldq %ymm3, %ymm0, %ymm11 + vpmuldq %ymm4, %ymm0, %ymm12 + vpmuldq %ymm5, %ymm0, %ymm13 + vpmuldq %ymm6, %ymm0, %ymm14 + vpmuldq %ymm7, %ymm0, %ymm15 + vpmuldq %ymm10, %ymm1, %ymm10 + vpmuldq %ymm11, %ymm1, %ymm11 + vpmuldq %ymm12, %ymm1, %ymm12 + vpmuldq %ymm13, %ymm1, %ymm13 + vpmuldq %ymm14, %ymm1, %ymm14 + vpmuldq %ymm15, %ymm1, %ymm15 + vpsubq %ymm10, %ymm2, %ymm2 + vpsubq %ymm11, %ymm3, %ymm3 + vpsubq %ymm12, %ymm4, %ymm4 + vpsubq %ymm13, %ymm5, %ymm5 + vpsubq %ymm14, %ymm6, %ymm6 + vpsubq %ymm15, %ymm7, %ymm7 + vpsrlq $0x20, %ymm2, %ymm2 + vpsrlq $0x20, %ymm4, %ymm4 + vmovshdup %ymm6, %ymm6 # ymm6 = ymm6[1,1,3,3,5,5,7,7] + vpblendd $0xaa, %ymm3, %ymm2, %ymm2 # ymm2 = ymm2[0],ymm3[1],ymm2[2],ymm3[3],ymm2[4],ymm3[5],ymm2[6],ymm3[7] + vpblendd $0xaa, %ymm5, %ymm4, %ymm4 # ymm4 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7] + vpblendd $0xaa, %ymm7, %ymm6, %ymm6 # ymm6 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7] + vmovdqa %ymm2, (%rdi) + vmovdqa %ymm4, 0x20(%rdi) + vmovdqa %ymm6, 0x40(%rdi) + addq $0x60, %rdi + addq $0x60, %rsi + addl $0x1, %eax + cmpl $0xa, %eax + jb Lmld_pointwise_avx2_looptop1 + vmovdqa (%rdi), %ymm2 + vmovdqa 0x20(%rdi), %ymm4 + vmovdqa (%rsi), %ymm10 + vmovdqa 0x20(%rsi), %ymm12 + vpsrlq $0x20, %ymm2, %ymm3 + vpsrlq $0x20, %ymm4, %ymm5 + vmovshdup %ymm10, %ymm11 # ymm11 = ymm10[1,1,3,3,5,5,7,7] + vmovshdup %ymm12, %ymm13 # ymm13 = ymm12[1,1,3,3,5,5,7,7] + vpmuldq %ymm10, %ymm2, %ymm2 + vpmuldq %ymm11, %ymm3, %ymm3 + vpmuldq %ymm12, %ymm4, %ymm4 + vpmuldq %ymm13, %ymm5, %ymm5 + vpmuldq %ymm2, %ymm0, %ymm10 + vpmuldq %ymm3, %ymm0, %ymm11 + vpmuldq %ymm4, %ymm0, %ymm12 + vpmuldq %ymm5, %ymm0, %ymm13 + vpmuldq %ymm10, %ymm1, %ymm10 + vpmuldq %ymm11, %ymm1, %ymm11 + vpmuldq %ymm12, %ymm1, %ymm12 + vpmuldq %ymm13, %ymm1, %ymm13 + vpsubq %ymm10, %ymm2, %ymm2 + vpsubq %ymm11, %ymm3, %ymm3 + vpsubq %ymm12, %ymm4, %ymm4 + vpsubq %ymm13, %ymm5, %ymm5 + vpsrlq $0x20, %ymm2, %ymm2 + vmovshdup %ymm4, %ymm4 # ymm4 = ymm4[1,1,3,3,5,5,7,7] + vpblendd $0x55, %ymm2, %ymm3, %ymm2 # ymm2 = ymm2[0],ymm3[1],ymm2[2],ymm3[3],ymm2[4],ymm3[5],ymm2[6],ymm3[7] + vpblendd $0x55, %ymm4, %ymm5, %ymm4 # ymm4 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7] + vmovdqa %ymm2, (%rdi) + vmovdqa %ymm4, 0x20(%rdi) + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(pointwise_avx2_asm) + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_poly_caddq_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_poly_caddq_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_poly_caddq_avx2_asm.S @@ -0,0 +1,199 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + + +/*yaml + Name: poly_caddq_avx2_asm + Description: x86_64 AVX2 conditional addition of q to each coefficient. + For all coefficients of the in/out polynomial, add Q if the coefficient + is negative. + Signature: void mld_poly_caddq_avx2_asm(int32_t *r) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t *r + description: Input/output polynomial (256 x int32_t) +*/ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_poly_caddq_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(poly_caddq_avx2_asm) +MLD_ASM_FN_SYMBOL(poly_caddq_avx2_asm) + + .cfi_startproc + vpxor %xmm2, %xmm2, %xmm2 + movl $0x7fe001, %eax # imm = 0x7FE001 + vmovd %eax, %xmm1 + vpbroadcastd %xmm1, %ymm1 + vpcmpgtd (%rdi), %ymm2, %ymm0 + vpand %ymm1, %ymm0, %ymm0 + vpaddd (%rdi), %ymm0, %ymm0 + vmovdqa %ymm0, (%rdi) + vpcmpgtd 0x20(%rdi), %ymm2, %ymm3 + vpand %ymm1, %ymm3, %ymm3 + vpaddd 0x20(%rdi), %ymm3, %ymm3 + vmovdqa %ymm3, 0x20(%rdi) + vpcmpgtd 0x40(%rdi), %ymm2, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpaddd 0x40(%rdi), %ymm4, %ymm4 + vmovdqa %ymm4, 0x40(%rdi) + vpcmpgtd 0x60(%rdi), %ymm2, %ymm5 + vpand %ymm1, %ymm5, %ymm5 + vpaddd 0x60(%rdi), %ymm5, %ymm5 + vmovdqa %ymm5, 0x60(%rdi) + vpcmpgtd 0x80(%rdi), %ymm2, %ymm0 + vpand %ymm1, %ymm0, %ymm0 + vpaddd 0x80(%rdi), %ymm0, %ymm0 + vmovdqa %ymm0, 0x80(%rdi) + vpcmpgtd 0xa0(%rdi), %ymm2, %ymm3 + vpand %ymm1, %ymm3, %ymm3 + vpaddd 0xa0(%rdi), %ymm3, %ymm3 + vmovdqa %ymm3, 0xa0(%rdi) + vpcmpgtd 0xc0(%rdi), %ymm2, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpaddd 0xc0(%rdi), %ymm4, %ymm4 + vmovdqa %ymm4, 0xc0(%rdi) + vpcmpgtd 0xe0(%rdi), %ymm2, %ymm5 + vpand %ymm1, %ymm5, %ymm5 + vpaddd 0xe0(%rdi), %ymm5, %ymm5 + vmovdqa %ymm5, 0xe0(%rdi) + vpcmpgtd 0x100(%rdi), %ymm2, %ymm0 + vpand %ymm1, %ymm0, %ymm0 + vpaddd 0x100(%rdi), %ymm0, %ymm0 + vmovdqa %ymm0, 0x100(%rdi) + vpcmpgtd 0x120(%rdi), %ymm2, %ymm3 + vpand %ymm1, %ymm3, %ymm3 + vpaddd 0x120(%rdi), %ymm3, %ymm3 + vmovdqa %ymm3, 0x120(%rdi) + vpcmpgtd 0x140(%rdi), %ymm2, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpaddd 0x140(%rdi), %ymm4, %ymm4 + vmovdqa %ymm4, 0x140(%rdi) + vpcmpgtd 0x160(%rdi), %ymm2, %ymm5 + vpand %ymm1, %ymm5, %ymm5 + vpaddd 0x160(%rdi), %ymm5, %ymm5 + vmovdqa %ymm5, 0x160(%rdi) + vpcmpgtd 0x180(%rdi), %ymm2, %ymm0 + vpand %ymm1, %ymm0, %ymm0 + vpaddd 0x180(%rdi), %ymm0, %ymm0 + vmovdqa %ymm0, 0x180(%rdi) + vpcmpgtd 0x1a0(%rdi), %ymm2, %ymm3 + vpand %ymm1, %ymm3, %ymm3 + vpaddd 0x1a0(%rdi), %ymm3, %ymm3 + vmovdqa %ymm3, 0x1a0(%rdi) + vpcmpgtd 0x1c0(%rdi), %ymm2, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpaddd 0x1c0(%rdi), %ymm4, %ymm4 + vmovdqa %ymm4, 0x1c0(%rdi) + vpcmpgtd 0x1e0(%rdi), %ymm2, %ymm5 + vpand %ymm1, %ymm5, %ymm5 + vpaddd 0x1e0(%rdi), %ymm5, %ymm5 + vmovdqa %ymm5, 0x1e0(%rdi) + vpcmpgtd 0x200(%rdi), %ymm2, %ymm0 + vpand %ymm1, %ymm0, %ymm0 + vpaddd 0x200(%rdi), %ymm0, %ymm0 + vmovdqa %ymm0, 0x200(%rdi) + vpcmpgtd 0x220(%rdi), %ymm2, %ymm3 + vpand %ymm1, %ymm3, %ymm3 + vpaddd 0x220(%rdi), %ymm3, %ymm3 + vmovdqa %ymm3, 0x220(%rdi) + vpcmpgtd 0x240(%rdi), %ymm2, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpaddd 0x240(%rdi), %ymm4, %ymm4 + vmovdqa %ymm4, 0x240(%rdi) + vpcmpgtd 0x260(%rdi), %ymm2, %ymm5 + vpand %ymm1, %ymm5, %ymm5 + vpaddd 0x260(%rdi), %ymm5, %ymm5 + vmovdqa %ymm5, 0x260(%rdi) + vpcmpgtd 0x280(%rdi), %ymm2, %ymm0 + vpand %ymm1, %ymm0, %ymm0 + vpaddd 0x280(%rdi), %ymm0, %ymm0 + vmovdqa %ymm0, 0x280(%rdi) + vpcmpgtd 0x2a0(%rdi), %ymm2, %ymm3 + vpand %ymm1, %ymm3, %ymm3 + vpaddd 0x2a0(%rdi), %ymm3, %ymm3 + vmovdqa %ymm3, 0x2a0(%rdi) + vpcmpgtd 0x2c0(%rdi), %ymm2, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpaddd 0x2c0(%rdi), %ymm4, %ymm4 + vmovdqa %ymm4, 0x2c0(%rdi) + vpcmpgtd 0x2e0(%rdi), %ymm2, %ymm5 + vpand %ymm1, %ymm5, %ymm5 + vpaddd 0x2e0(%rdi), %ymm5, %ymm5 + vmovdqa %ymm5, 0x2e0(%rdi) + vpcmpgtd 0x300(%rdi), %ymm2, %ymm0 + vpand %ymm1, %ymm0, %ymm0 + vpaddd 0x300(%rdi), %ymm0, %ymm0 + vmovdqa %ymm0, 0x300(%rdi) + vpcmpgtd 0x320(%rdi), %ymm2, %ymm3 + vpand %ymm1, %ymm3, %ymm3 + vpaddd 0x320(%rdi), %ymm3, %ymm3 + vmovdqa %ymm3, 0x320(%rdi) + vpcmpgtd 0x340(%rdi), %ymm2, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpaddd 0x340(%rdi), %ymm4, %ymm4 + vmovdqa %ymm4, 0x340(%rdi) + vpcmpgtd 0x360(%rdi), %ymm2, %ymm5 + vpand %ymm1, %ymm5, %ymm5 + vpaddd 0x360(%rdi), %ymm5, %ymm5 + vmovdqa %ymm5, 0x360(%rdi) + vpcmpgtd 0x380(%rdi), %ymm2, %ymm0 + vpand %ymm1, %ymm0, %ymm0 + vpaddd 0x380(%rdi), %ymm0, %ymm0 + vmovdqa %ymm0, 0x380(%rdi) + vpcmpgtd 0x3a0(%rdi), %ymm2, %ymm3 + vpand %ymm1, %ymm3, %ymm3 + vpaddd 0x3a0(%rdi), %ymm3, %ymm3 + vmovdqa %ymm3, 0x3a0(%rdi) + vpcmpgtd 0x3c0(%rdi), %ymm2, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpaddd 0x3c0(%rdi), %ymm4, %ymm4 + vmovdqa %ymm4, 0x3c0(%rdi) + vpcmpgtd 0x3e0(%rdi), %ymm2, %ymm5 + vpand %ymm1, %ymm5, %ymm5 + vpaddd 0x3e0(%rdi), %ymm5, %ymm5 + vmovdqa %ymm5, 0x3e0(%rdi) + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(poly_caddq_avx2_asm) + + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_poly_chknorm_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_poly_chknorm_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_poly_chknorm_avx2_asm.S @@ -0,0 +1,176 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + + +/*yaml + Name: poly_chknorm_avx2_asm + Description: x86_64 AVX2 infinity-norm bound check on polynomial coefficients. + Check the infinity norm of the polynomial against the given bound B. + Returns 0 if the norm is strictly smaller than B; otherwise returns 1 + (i.e. returns 1 if any |coefficient| >= B, 0 otherwise). + Signature: int mld_poly_chknorm_avx2_asm(const int32_t *a, int32_t B) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: read-only + c_parameter: const int32_t *a + description: Input polynomial (256 x int32_t) + rsi: + type: scalar + c_parameter: int32_t B + description: Norm bound (must be non-negative) + test_with: 131072 # representative non-negative bound (1 << 17) +*/ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_poly_chknorm_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(poly_chknorm_avx2_asm) +MLD_ASM_FN_SYMBOL(poly_chknorm_avx2_asm) + + .cfi_startproc + subl $0x1, %esi + vpxor %xmm1, %xmm1, %xmm1 + vmovd %esi, %xmm2 + vpbroadcastd %xmm2, %ymm2 + vpabsd (%rdi), %ymm0 + vpcmpgtd %ymm2, %ymm0, %ymm0 + vpor %ymm0, %ymm1, %ymm1 + vpabsd 0x20(%rdi), %ymm3 + vpcmpgtd %ymm2, %ymm3, %ymm3 + vpor %ymm3, %ymm1, %ymm1 + vpabsd 0x40(%rdi), %ymm4 + vpcmpgtd %ymm2, %ymm4, %ymm4 + vpor %ymm4, %ymm1, %ymm1 + vpabsd 0x60(%rdi), %ymm5 + vpcmpgtd %ymm2, %ymm5, %ymm5 + vpor %ymm5, %ymm1, %ymm1 + vpabsd 0x80(%rdi), %ymm0 + vpcmpgtd %ymm2, %ymm0, %ymm0 + vpor %ymm0, %ymm1, %ymm1 + vpabsd 0xa0(%rdi), %ymm3 + vpcmpgtd %ymm2, %ymm3, %ymm3 + vpor %ymm3, %ymm1, %ymm1 + vpabsd 0xc0(%rdi), %ymm4 + vpcmpgtd %ymm2, %ymm4, %ymm4 + vpor %ymm4, %ymm1, %ymm1 + vpabsd 0xe0(%rdi), %ymm5 + vpcmpgtd %ymm2, %ymm5, %ymm5 + vpor %ymm5, %ymm1, %ymm1 + vpabsd 0x100(%rdi), %ymm0 + vpcmpgtd %ymm2, %ymm0, %ymm0 + vpor %ymm0, %ymm1, %ymm1 + vpabsd 0x120(%rdi), %ymm3 + vpcmpgtd %ymm2, %ymm3, %ymm3 + vpor %ymm3, %ymm1, %ymm1 + vpabsd 0x140(%rdi), %ymm4 + vpcmpgtd %ymm2, %ymm4, %ymm4 + vpor %ymm4, %ymm1, %ymm1 + vpabsd 0x160(%rdi), %ymm5 + vpcmpgtd %ymm2, %ymm5, %ymm5 + vpor %ymm5, %ymm1, %ymm1 + vpabsd 0x180(%rdi), %ymm0 + vpcmpgtd %ymm2, %ymm0, %ymm0 + vpor %ymm0, %ymm1, %ymm1 + vpabsd 0x1a0(%rdi), %ymm3 + vpcmpgtd %ymm2, %ymm3, %ymm3 + vpor %ymm3, %ymm1, %ymm1 + vpabsd 0x1c0(%rdi), %ymm4 + vpcmpgtd %ymm2, %ymm4, %ymm4 + vpor %ymm4, %ymm1, %ymm1 + vpabsd 0x1e0(%rdi), %ymm5 + vpcmpgtd %ymm2, %ymm5, %ymm5 + vpor %ymm5, %ymm1, %ymm1 + vpabsd 0x200(%rdi), %ymm0 + vpcmpgtd %ymm2, %ymm0, %ymm0 + vpor %ymm0, %ymm1, %ymm1 + vpabsd 0x220(%rdi), %ymm3 + vpcmpgtd %ymm2, %ymm3, %ymm3 + vpor %ymm3, %ymm1, %ymm1 + vpabsd 0x240(%rdi), %ymm4 + vpcmpgtd %ymm2, %ymm4, %ymm4 + vpor %ymm4, %ymm1, %ymm1 + vpabsd 0x260(%rdi), %ymm5 + vpcmpgtd %ymm2, %ymm5, %ymm5 + vpor %ymm5, %ymm1, %ymm1 + vpabsd 0x280(%rdi), %ymm0 + vpcmpgtd %ymm2, %ymm0, %ymm0 + vpor %ymm0, %ymm1, %ymm1 + vpabsd 0x2a0(%rdi), %ymm3 + vpcmpgtd %ymm2, %ymm3, %ymm3 + vpor %ymm3, %ymm1, %ymm1 + vpabsd 0x2c0(%rdi), %ymm4 + vpcmpgtd %ymm2, %ymm4, %ymm4 + vpor %ymm4, %ymm1, %ymm1 + vpabsd 0x2e0(%rdi), %ymm5 + vpcmpgtd %ymm2, %ymm5, %ymm5 + vpor %ymm5, %ymm1, %ymm1 + vpabsd 0x300(%rdi), %ymm0 + vpcmpgtd %ymm2, %ymm0, %ymm0 + vpor %ymm0, %ymm1, %ymm1 + vpabsd 0x320(%rdi), %ymm3 + vpcmpgtd %ymm2, %ymm3, %ymm3 + vpor %ymm3, %ymm1, %ymm1 + vpabsd 0x340(%rdi), %ymm4 + vpcmpgtd %ymm2, %ymm4, %ymm4 + vpor %ymm4, %ymm1, %ymm1 + vpabsd 0x360(%rdi), %ymm5 + vpcmpgtd %ymm2, %ymm5, %ymm5 + vpor %ymm5, %ymm1, %ymm1 + vpabsd 0x380(%rdi), %ymm0 + vpcmpgtd %ymm2, %ymm0, %ymm0 + vpor %ymm0, %ymm1, %ymm1 + vpabsd 0x3a0(%rdi), %ymm3 + vpcmpgtd %ymm2, %ymm3, %ymm3 + vpor %ymm3, %ymm1, %ymm1 + vpabsd 0x3c0(%rdi), %ymm4 + vpcmpgtd %ymm2, %ymm4, %ymm4 + vpor %ymm4, %ymm1, %ymm1 + vpabsd 0x3e0(%rdi), %ymm5 + vpcmpgtd %ymm2, %ymm5, %ymm5 + vpor %ymm5, %ymm1, %ymm1 + xorl %eax, %eax + vptest %ymm1, %ymm1 + setne %al + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(poly_chknorm_avx2_asm) + + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_poly_decompose_32_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_poly_decompose_32_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_poly_decompose_32_avx2_asm.S @@ -0,0 +1,490 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + * + * The algorithm for Decompose(r) (more specifically the handling for the + * wrap-around cases) is modified. See the AVX2 intrinsics version + * (poly_decompose_32_avx2.c, predecessor of this file) for a more detailed + * comparison. + */ + + +/*yaml + Name: poly_decompose_32_avx2_asm + Description: x86_64 AVX2 coefficient decomposition (alpha = 2*(Q-1)/32). + For all coefficients c of the input polynomial, compute high and low bits + c0, c1 such that c = c1*ALPHA + c0 with -ALPHA/2 < c0 <= ALPHA/2, except if + c1 = (Q-1)/ALPHA where we set c1 = 0 and -ALPHA/2 <= c0 = c - Q < 0. + Assumes coefficients to be standard (unsigned canonical) representatives, + i.e. 0 <= c < Q. For ML-DSA-65 / ML-DSA-87 (gamma2 = (Q-1)/32, + alpha = 2*gamma2). + Signature: void mld_poly_decompose_32_avx2_asm(int32_t *a1, int32_t *a0) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t *a1 + description: Output high-part polynomial (256 x int32_t) + rsi: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t *a0 + description: Input polynomial / output low-part (256 x int32_t) +*/ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_NO_SIGN_API) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)) + + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_poly_decompose_32_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(poly_decompose_32_avx2_asm) +MLD_ASM_FN_SYMBOL(poly_decompose_32_avx2_asm) + + .cfi_startproc + movl $0x7f, %eax + vmovd %eax, %xmm10 + vpbroadcastd %xmm10, %ymm10 + movl $0x401, %eax # imm = 0x401 + vmovd %eax, %xmm11 + vpbroadcastd %xmm11, %ymm11 + movl $0x200, %eax # imm = 0x200 + vmovd %eax, %xmm12 + vpbroadcastd %xmm12, %ymm12 + movl $0x7be100, %eax # imm = 0x7BE100 + vmovd %eax, %xmm13 + vpbroadcastd %xmm13, %ymm13 + movl $0x7fe00, %eax # imm = 0x7FE00 + vmovd %eax, %xmm14 + vpbroadcastd %xmm14, %ymm14 + vmovdqa (%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, (%rdi) + vmovdqa %ymm2, (%rsi) + vmovdqa 0x20(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x20(%rdi) + vmovdqa %ymm2, 0x20(%rsi) + vmovdqa 0x40(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x40(%rdi) + vmovdqa %ymm2, 0x40(%rsi) + vmovdqa 0x60(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x60(%rdi) + vmovdqa %ymm2, 0x60(%rsi) + vmovdqa 0x80(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x80(%rdi) + vmovdqa %ymm2, 0x80(%rsi) + vmovdqa 0xa0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0xa0(%rdi) + vmovdqa %ymm2, 0xa0(%rsi) + vmovdqa 0xc0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0xc0(%rdi) + vmovdqa %ymm2, 0xc0(%rsi) + vmovdqa 0xe0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0xe0(%rdi) + vmovdqa %ymm2, 0xe0(%rsi) + vmovdqa 0x100(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x100(%rdi) + vmovdqa %ymm2, 0x100(%rsi) + vmovdqa 0x120(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x120(%rdi) + vmovdqa %ymm2, 0x120(%rsi) + vmovdqa 0x140(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x140(%rdi) + vmovdqa %ymm2, 0x140(%rsi) + vmovdqa 0x160(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x160(%rdi) + vmovdqa %ymm2, 0x160(%rsi) + vmovdqa 0x180(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x180(%rdi) + vmovdqa %ymm2, 0x180(%rsi) + vmovdqa 0x1a0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x1a0(%rdi) + vmovdqa %ymm2, 0x1a0(%rsi) + vmovdqa 0x1c0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x1c0(%rdi) + vmovdqa %ymm2, 0x1c0(%rsi) + vmovdqa 0x1e0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x1e0(%rdi) + vmovdqa %ymm2, 0x1e0(%rsi) + vmovdqa 0x200(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x200(%rdi) + vmovdqa %ymm2, 0x200(%rsi) + vmovdqa 0x220(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x220(%rdi) + vmovdqa %ymm2, 0x220(%rsi) + vmovdqa 0x240(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x240(%rdi) + vmovdqa %ymm2, 0x240(%rsi) + vmovdqa 0x260(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x260(%rdi) + vmovdqa %ymm2, 0x260(%rsi) + vmovdqa 0x280(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x280(%rdi) + vmovdqa %ymm2, 0x280(%rsi) + vmovdqa 0x2a0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x2a0(%rdi) + vmovdqa %ymm2, 0x2a0(%rsi) + vmovdqa 0x2c0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x2c0(%rdi) + vmovdqa %ymm2, 0x2c0(%rsi) + vmovdqa 0x2e0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x2e0(%rdi) + vmovdqa %ymm2, 0x2e0(%rsi) + vmovdqa 0x300(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x300(%rdi) + vmovdqa %ymm2, 0x300(%rsi) + vmovdqa 0x320(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x320(%rdi) + vmovdqa %ymm2, 0x320(%rsi) + vmovdqa 0x340(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x340(%rdi) + vmovdqa %ymm2, 0x340(%rsi) + vmovdqa 0x360(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x360(%rdi) + vmovdqa %ymm2, 0x360(%rsi) + vmovdqa 0x380(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x380(%rdi) + vmovdqa %ymm2, 0x380(%rsi) + vmovdqa 0x3a0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x3a0(%rdi) + vmovdqa %ymm2, 0x3a0(%rsi) + vmovdqa 0x3c0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x3c0(%rdi) + vmovdqa %ymm2, 0x3c0(%rsi) + vmovdqa 0x3e0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x3e0(%rdi) + vmovdqa %ymm2, 0x3e0(%rsi) + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(poly_decompose_32_avx2_asm) + + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_NO_SIGN_API && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_poly_decompose_88_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_poly_decompose_88_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_poly_decompose_88_avx2_asm.S @@ -0,0 +1,489 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + * + * The algorithm for Decompose(r) (more specifically the handling for the + * wrap-around cases) is modified. See the AVX2 intrinsics version + * (poly_decompose_88_avx2.c, predecessor of this file) for a more detailed + * comparison. + */ + + +/*yaml + Name: poly_decompose_88_avx2_asm + Description: x86_64 AVX2 coefficient decomposition (alpha = 2*(Q-1)/88). + For all coefficients c of the input polynomial, compute high and low bits + c0, c1 such that c = c1*ALPHA + c0 with -ALPHA/2 < c0 <= ALPHA/2, except if + c1 = (Q-1)/ALPHA where we set c1 = 0 and -ALPHA/2 <= c0 = c - Q < 0. + Assumes coefficients to be standard (unsigned canonical) representatives, + i.e. 0 <= c < Q. For ML-DSA-44 (gamma2 = (Q-1)/88, alpha = 2*gamma2). + Signature: void mld_poly_decompose_88_avx2_asm(int32_t *a1, int32_t *a0) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t *a1 + description: Output high-part polynomial (256 x int32_t) + rsi: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t *a0 + description: Input polynomial / output low-part (256 x int32_t) +*/ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_NO_SIGN_API) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + MLD_CONFIG_PARAMETER_SET == 44) + + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_poly_decompose_88_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(poly_decompose_88_avx2_asm) +MLD_ASM_FN_SYMBOL(poly_decompose_88_avx2_asm) + + .cfi_startproc + movl $0x7f, %eax + vmovd %eax, %xmm10 + vpbroadcastd %xmm10, %ymm10 + movl $0x2c0b, %eax # imm = 0x2C0B + vmovd %eax, %xmm11 + vpbroadcastd %xmm11, %ymm11 + movl $0x80, %eax + vmovd %eax, %xmm12 + vpbroadcastd %xmm12, %ymm12 + movl $0x7e6c00, %eax # imm = 0x7E6C00 + vmovd %eax, %xmm13 + vpbroadcastd %xmm13, %ymm13 + movl $0x2e800, %eax # imm = 0x2E800 + vmovd %eax, %xmm14 + vpbroadcastd %xmm14, %ymm14 + vmovdqa (%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, (%rdi) + vmovdqa %ymm2, (%rsi) + vmovdqa 0x20(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x20(%rdi) + vmovdqa %ymm2, 0x20(%rsi) + vmovdqa 0x40(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x40(%rdi) + vmovdqa %ymm2, 0x40(%rsi) + vmovdqa 0x60(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x60(%rdi) + vmovdqa %ymm2, 0x60(%rsi) + vmovdqa 0x80(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x80(%rdi) + vmovdqa %ymm2, 0x80(%rsi) + vmovdqa 0xa0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0xa0(%rdi) + vmovdqa %ymm2, 0xa0(%rsi) + vmovdqa 0xc0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0xc0(%rdi) + vmovdqa %ymm2, 0xc0(%rsi) + vmovdqa 0xe0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0xe0(%rdi) + vmovdqa %ymm2, 0xe0(%rsi) + vmovdqa 0x100(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x100(%rdi) + vmovdqa %ymm2, 0x100(%rsi) + vmovdqa 0x120(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x120(%rdi) + vmovdqa %ymm2, 0x120(%rsi) + vmovdqa 0x140(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x140(%rdi) + vmovdqa %ymm2, 0x140(%rsi) + vmovdqa 0x160(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x160(%rdi) + vmovdqa %ymm2, 0x160(%rsi) + vmovdqa 0x180(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x180(%rdi) + vmovdqa %ymm2, 0x180(%rsi) + vmovdqa 0x1a0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x1a0(%rdi) + vmovdqa %ymm2, 0x1a0(%rsi) + vmovdqa 0x1c0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x1c0(%rdi) + vmovdqa %ymm2, 0x1c0(%rsi) + vmovdqa 0x1e0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x1e0(%rdi) + vmovdqa %ymm2, 0x1e0(%rsi) + vmovdqa 0x200(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x200(%rdi) + vmovdqa %ymm2, 0x200(%rsi) + vmovdqa 0x220(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x220(%rdi) + vmovdqa %ymm2, 0x220(%rsi) + vmovdqa 0x240(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x240(%rdi) + vmovdqa %ymm2, 0x240(%rsi) + vmovdqa 0x260(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x260(%rdi) + vmovdqa %ymm2, 0x260(%rsi) + vmovdqa 0x280(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x280(%rdi) + vmovdqa %ymm2, 0x280(%rsi) + vmovdqa 0x2a0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x2a0(%rdi) + vmovdqa %ymm2, 0x2a0(%rsi) + vmovdqa 0x2c0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x2c0(%rdi) + vmovdqa %ymm2, 0x2c0(%rsi) + vmovdqa 0x2e0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x2e0(%rdi) + vmovdqa %ymm2, 0x2e0(%rsi) + vmovdqa 0x300(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x300(%rdi) + vmovdqa %ymm2, 0x300(%rsi) + vmovdqa 0x320(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x320(%rdi) + vmovdqa %ymm2, 0x320(%rsi) + vmovdqa 0x340(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x340(%rdi) + vmovdqa %ymm2, 0x340(%rsi) + vmovdqa 0x360(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x360(%rdi) + vmovdqa %ymm2, 0x360(%rsi) + vmovdqa 0x380(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x380(%rdi) + vmovdqa %ymm2, 0x380(%rsi) + vmovdqa 0x3a0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x3a0(%rdi) + vmovdqa %ymm2, 0x3a0(%rsi) + vmovdqa 0x3c0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x3c0(%rdi) + vmovdqa %ymm2, 0x3c0(%rsi) + vmovdqa 0x3e0(%rsi), %ymm0 + vpaddd %ymm10, %ymm0, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm11, %ymm1, %ymm1 + vpmulhrsw %ymm12, %ymm1, %ymm1 + vpcmpgtd %ymm13, %ymm0, %ymm3 + vpmulld %ymm14, %ymm1, %ymm2 + vpsubd %ymm2, %ymm0, %ymm2 + vpandn %ymm1, %ymm3, %ymm1 + vpaddd %ymm3, %ymm2, %ymm2 + vmovdqa %ymm1, 0x3e0(%rdi) + vmovdqa %ymm2, 0x3e0(%rsi) + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(poly_decompose_88_avx2_asm) + + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_NO_SIGN_API && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44) \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_poly_use_hint_32_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_poly_use_hint_32_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_poly_use_hint_32_avx2_asm.S @@ -0,0 +1,123 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + + +/*yaml + Name: poly_use_hint_32_avx2_asm + Description: x86_64 AVX2 hint application (alpha = (Q-1)/32). + Use the hint polynomial h to correct the high bits of the polynomial a, + in place. Variant for parameter sets ML-DSA-65 and ML-DSA-87 + (GAMMA2 = (Q-1)/32). + Signature: void mld_poly_use_hint_32_avx2_asm(int32_t *a, const int32_t *h) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t *a + description: Input/output polynomial (256 x int32_t) + rsi: + type: buffer + size_bytes: 1024 + permissions: read-only + c_parameter: const int32_t *h + description: Hint polynomial (256 x int32_t) +*/ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_NO_VERIFY_API) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)) + + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_poly_use_hint_32_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(poly_use_hint_32_avx2_asm) +MLD_ASM_FN_SYMBOL(poly_use_hint_32_avx2_asm) + + .cfi_startproc + movl $0x7f, %ecx + movl $0x401, %r8d # imm = 0x401 + vmovd %r8d, %xmm8 + vpbroadcastd %xmm8, %ymm8 + xorl %eax, %eax + vpxor %xmm6, %xmm6, %xmm6 + vmovd %ecx, %xmm5 + movl $0x7be100, %ecx # imm = 0x7BE100 + movl $0x200, %r9d # imm = 0x200 + vmovd %r9d, %xmm7 + vpbroadcastd %xmm7, %ymm7 + vmovd %ecx, %xmm4 + movl $0xf, %ecx + vpbroadcastd %xmm5, %ymm5 + vmovd %ecx, %xmm3 + vpbroadcastd %xmm4, %ymm4 + vpbroadcastd %xmm3, %ymm3 + +Lmld_poly_use_hint_32_avx2_asm_loop: + vmovdqa (%rdi), %ymm0 + vmovdqa (%rsi), %ymm2 + vpaddd %ymm0, %ymm5, %ymm1 + vpsrld $0x7, %ymm1, %ymm1 + vpmulhuw %ymm8, %ymm1, %ymm1 + vpmulhrsw %ymm7, %ymm1, %ymm1 + vpcmpgtd %ymm4, %ymm0, %ymm11 + vpandn %ymm1, %ymm11, %ymm9 + vpslld $0xa, %ymm1, %ymm10 + vpsubd %ymm1, %ymm10, %ymm1 + vpslld $0x9, %ymm1, %ymm1 + vpsubd %ymm1, %ymm0, %ymm0 + vpaddd %ymm11, %ymm0, %ymm0 + vpcmpgtd %ymm6, %ymm0, %ymm0 + vpandn %ymm2, %ymm0, %ymm0 + vpslld $0x1, %ymm0, %ymm0 + vpsubd %ymm0, %ymm2, %ymm2 + vpaddd %ymm9, %ymm2, %ymm2 + vpand %ymm3, %ymm2, %ymm2 + vmovdqa %ymm2, (%rdi) + addq $0x20, %rdi + addq $0x20, %rsi + addq $0x20, %rax + cmpq $0x400, %rax # imm = 0x400 + jne Lmld_poly_use_hint_32_avx2_asm_loop + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(poly_use_hint_32_avx2_asm) + + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_NO_VERIFY_API && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_poly_use_hint_88_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_poly_use_hint_88_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_poly_use_hint_88_avx2_asm.S @@ -0,0 +1,125 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + + +/*yaml + Name: poly_use_hint_88_avx2_asm + Description: x86_64 AVX2 hint application (alpha = (Q-1)/88). + Use the hint polynomial h to correct the high bits of the polynomial a, + in place. Variant for parameter set ML-DSA-44 (GAMMA2 = (Q-1)/88). + Signature: void mld_poly_use_hint_88_avx2_asm(int32_t *a, const int32_t *h) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: read/write + c_parameter: int32_t *a + description: Input/output polynomial (256 x int32_t) + rsi: + type: buffer + size_bytes: 1024 + permissions: read-only + c_parameter: const int32_t *h + description: Hint polynomial (256 x int32_t) +*/ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_NO_VERIFY_API) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + MLD_CONFIG_PARAMETER_SET == 44) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_poly_use_hint_88_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(poly_use_hint_88_avx2_asm) +MLD_ASM_FN_SYMBOL(poly_use_hint_88_avx2_asm) + + .cfi_startproc + movl $0x7f, %ecx + xorl %eax, %eax + vpxor %xmm5, %xmm5, %xmm5 + movl $0x2c0b, %r8d # imm = 0x2C0B + vmovd %r8d, %xmm8 + vpbroadcastd %xmm8, %ymm8 + vmovd %ecx, %xmm4 + movl $0x7e6c00, %ecx # imm = 0x7E6C00 + movl $0x80, %r9d + vmovd %r9d, %xmm7 + vpbroadcastd %xmm7, %ymm7 + movl $0x2b, %r10d + vmovd %r10d, %xmm6 + vpbroadcastd %xmm6, %ymm6 + vmovd %ecx, %xmm3 + vpbroadcastd %xmm4, %ymm4 + vpbroadcastd %xmm3, %ymm3 + +Lmld_poly_use_hint_88_avx2_asm_loop: + vmovdqa (%rdi), %ymm0 + vmovdqa (%rsi), %ymm1 + vpaddd %ymm0, %ymm4, %ymm10 + vpsrld $0x7, %ymm10, %ymm10 + vpmulhuw %ymm8, %ymm10, %ymm10 + vpmulhrsw %ymm7, %ymm10, %ymm10 + vpcmpgtd %ymm3, %ymm0, %ymm11 + vpandn %ymm10, %ymm11, %ymm9 + vpslld $0x1, %ymm10, %ymm12 + vpaddd %ymm10, %ymm12, %ymm12 + vpslld $0x5, %ymm12, %ymm10 + vpsubd %ymm12, %ymm10, %ymm10 + vpslld $0xb, %ymm10, %ymm10 + vpsubd %ymm10, %ymm0, %ymm0 + vpaddd %ymm11, %ymm0, %ymm0 + vpcmpgtd %ymm5, %ymm0, %ymm0 + vpandn %ymm1, %ymm0, %ymm0 + vpslld $0x1, %ymm0, %ymm0 + vpsubd %ymm0, %ymm1, %ymm0 + vpaddd %ymm9, %ymm0, %ymm0 + vpblendvb %ymm0, %ymm6, %ymm0, %ymm0 + vpcmpgtd %ymm6, %ymm0, %ymm1 + vpandn %ymm0, %ymm1, %ymm0 + vmovdqa %ymm0, (%rdi) + addq $0x20, %rdi + addq $0x20, %rsi + addq $0x20, %rax + cmpq $0x400, %rax # imm = 0x400 + jne Lmld_poly_use_hint_88_avx2_asm_loop + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(poly_use_hint_88_avx2_asm) + + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_NO_VERIFY_API && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44) \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_polyz_unpack_17_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_polyz_unpack_17_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_polyz_unpack_17_avx2_asm.S @@ -0,0 +1,355 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + + +/*yaml + Name: polyz_unpack_17_avx2_asm + Description: x86_64 AVX2 unpacking of 17-bit packed coefficients. + Unpack polynomial z with 18-bit packed coefficients (GAMMA1 = 2^17), + mapping packed [0, 2^18-1] to signed [-(2^17-1), 2^17] via GAMMA1 - x. + Signature: void mld_polyz_unpack_17_avx2_asm(int32_t *r, const uint8_t *a) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t *r + description: Output polynomial (256 x int32_t) + rsi: + type: buffer + size_bytes: 576 + permissions: read-only + c_parameter: const uint8_t *a + description: Packed input bytes +*/ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + (!defined(MLD_CONFIG_NO_SIGN_API) || \ + !defined(MLD_CONFIG_NO_VERIFY_API)) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + MLD_CONFIG_PARAMETER_SET == 44) + + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_polyz_unpack_17_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(polyz_unpack_17_avx2_asm) +MLD_ASM_FN_SYMBOL(polyz_unpack_17_avx2_asm) + + .cfi_startproc + movabsq $-0xfbfcfd00fdff00, %rax # imm = 0xFF040302FF020100 + vmovq %rax, %xmm1 + movabsq $-0xf7f8f900f9fafc, %rax # imm = 0xFF080706FF060504 + vpinsrq $0x1, %rax, %xmm1, %xmm1 + movabsq $-0xe4e5e600e6e7e9, %rax # imm = 0xFF1B1A19FF191817 + vmovq %rax, %xmm5 + movabsq $-0xe0e1e200e2e3e5, %rax # imm = 0xFF1F1E1DFF1D1C1B + vpinsrq $0x1, %rax, %xmm5, %xmm5 + vinserti128 $0x1, %xmm5, %ymm1, %ymm1 + movabsq $0x200000000, %rax # imm = 0x200000000 + vmovq %rax, %xmm2 + movabsq $0x600000004, %rax # imm = 0x600000004 + vpinsrq $0x1, %rax, %xmm2, %xmm2 + vinserti128 $0x1, %xmm2, %ymm2, %ymm2 + movl $0x3ffff, %eax # imm = 0x3FFFF + vmovd %eax, %xmm3 + vpbroadcastd %xmm3, %ymm3 + movl $0x20000, %eax # imm = 0x20000 + vmovd %eax, %xmm4 + vpbroadcastd %xmm4, %ymm4 + vmovdqu (%rsi), %xmm0 + vmovdqu 0x2(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, (%rdi) + vmovdqu 0x12(%rsi), %xmm0 + vmovdqu 0x14(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x20(%rdi) + vmovdqu 0x24(%rsi), %xmm0 + vmovdqu 0x26(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x40(%rdi) + vmovdqu 0x36(%rsi), %xmm0 + vmovdqu 0x38(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x60(%rdi) + vmovdqu 0x48(%rsi), %xmm0 + vmovdqu 0x4a(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x80(%rdi) + vmovdqu 0x5a(%rsi), %xmm0 + vmovdqu 0x5c(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0xa0(%rdi) + vmovdqu 0x6c(%rsi), %xmm0 + vmovdqu 0x6e(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0xc0(%rdi) + vmovdqu 0x7e(%rsi), %xmm0 + vmovdqu 0x80(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0xe0(%rdi) + vmovdqu 0x90(%rsi), %xmm0 + vmovdqu 0x92(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x100(%rdi) + vmovdqu 0xa2(%rsi), %xmm0 + vmovdqu 0xa4(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x120(%rdi) + vmovdqu 0xb4(%rsi), %xmm0 + vmovdqu 0xb6(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x140(%rdi) + vmovdqu 0xc6(%rsi), %xmm0 + vmovdqu 0xc8(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x160(%rdi) + vmovdqu 0xd8(%rsi), %xmm0 + vmovdqu 0xda(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x180(%rdi) + vmovdqu 0xea(%rsi), %xmm0 + vmovdqu 0xec(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x1a0(%rdi) + vmovdqu 0xfc(%rsi), %xmm0 + vmovdqu 0xfe(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x1c0(%rdi) + vmovdqu 0x10e(%rsi), %xmm0 + vmovdqu 0x110(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x1e0(%rdi) + vmovdqu 0x120(%rsi), %xmm0 + vmovdqu 0x122(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x200(%rdi) + vmovdqu 0x132(%rsi), %xmm0 + vmovdqu 0x134(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x220(%rdi) + vmovdqu 0x144(%rsi), %xmm0 + vmovdqu 0x146(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x240(%rdi) + vmovdqu 0x156(%rsi), %xmm0 + vmovdqu 0x158(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x260(%rdi) + vmovdqu 0x168(%rsi), %xmm0 + vmovdqu 0x16a(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x280(%rdi) + vmovdqu 0x17a(%rsi), %xmm0 + vmovdqu 0x17c(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x2a0(%rdi) + vmovdqu 0x18c(%rsi), %xmm0 + vmovdqu 0x18e(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x2c0(%rdi) + vmovdqu 0x19e(%rsi), %xmm0 + vmovdqu 0x1a0(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x2e0(%rdi) + vmovdqu 0x1b0(%rsi), %xmm0 + vmovdqu 0x1b2(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x300(%rdi) + vmovdqu 0x1c2(%rsi), %xmm0 + vmovdqu 0x1c4(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x320(%rdi) + vmovdqu 0x1d4(%rsi), %xmm0 + vmovdqu 0x1d6(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x340(%rdi) + vmovdqu 0x1e6(%rsi), %xmm0 + vmovdqu 0x1e8(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x360(%rdi) + vmovdqu 0x1f8(%rsi), %xmm0 + vmovdqu 0x1fa(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x380(%rdi) + vmovdqu 0x20a(%rsi), %xmm0 + vmovdqu 0x20c(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x3a0(%rdi) + vmovdqu 0x21c(%rsi), %xmm0 + vmovdqu 0x21e(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x3c0(%rdi) + vmovdqu 0x22e(%rsi), %xmm0 + vmovdqu 0x230(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x3e0(%rdi) + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(polyz_unpack_17_avx2_asm) + + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && (!MLD_CONFIG_NO_SIGN_API || \ + !MLD_CONFIG_NO_VERIFY_API) && !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44) \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_polyz_unpack_19_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_polyz_unpack_19_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_polyz_unpack_19_avx2_asm.S @@ -0,0 +1,355 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + + +/*yaml + Name: polyz_unpack_19_avx2_asm + Description: x86_64 AVX2 unpacking of 19-bit packed coefficients. + Unpack polynomial z with 20-bit packed coefficients (GAMMA1 = 2^19), + mapping packed [0, 2^20-1] to signed [-(2^19-1), 2^19] via GAMMA1 - x. + Signature: void mld_polyz_unpack_19_avx2_asm(int32_t *r, const uint8_t *a) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t *r + description: Output polynomial (256 x int32_t) + rsi: + type: buffer + size_bytes: 640 + permissions: read-only + c_parameter: const uint8_t *a + description: Packed input bytes +*/ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + (!defined(MLD_CONFIG_NO_SIGN_API) || \ + !defined(MLD_CONFIG_NO_VERIFY_API)) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87)) + + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_polyz_unpack_19_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(polyz_unpack_19_avx2_asm) +MLD_ASM_FN_SYMBOL(polyz_unpack_19_avx2_asm) + + .cfi_startproc + movabsq $-0xfbfcfd00fdff00, %rax # imm = 0xFF040302FF020100 + vmovq %rax, %xmm1 + movabsq $-0xf6f7f800f8f9fb, %rax # imm = 0xFF090807FF070605 + vpinsrq $0x1, %rax, %xmm1, %xmm1 + movabsq $-0xe5e6e700e7e8ea, %rax # imm = 0xFF1A1918FF181716 + vmovq %rax, %xmm5 + movabsq $-0xe0e1e200e2e3e5, %rax # imm = 0xFF1F1E1DFF1D1C1B + vpinsrq $0x1, %rax, %xmm5, %xmm5 + vinserti128 $0x1, %xmm5, %ymm1, %ymm1 + movabsq $0x400000000, %rax # imm = 0x400000000 + vmovq %rax, %xmm2 + movabsq $0x400000000, %rax # imm = 0x400000000 + vpinsrq $0x1, %rax, %xmm2, %xmm2 + vinserti128 $0x1, %xmm2, %ymm2, %ymm2 + movl $0xfffff, %eax # imm = 0xFFFFF + vmovd %eax, %xmm3 + vpbroadcastd %xmm3, %ymm3 + movl $0x80000, %eax # imm = 0x80000 + vmovd %eax, %xmm4 + vpbroadcastd %xmm4, %ymm4 + vmovdqu (%rsi), %xmm0 + vmovdqu 0x4(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, (%rdi) + vmovdqu 0x14(%rsi), %xmm0 + vmovdqu 0x18(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x20(%rdi) + vmovdqu 0x28(%rsi), %xmm0 + vmovdqu 0x2c(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x40(%rdi) + vmovdqu 0x3c(%rsi), %xmm0 + vmovdqu 0x40(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x60(%rdi) + vmovdqu 0x50(%rsi), %xmm0 + vmovdqu 0x54(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x80(%rdi) + vmovdqu 0x64(%rsi), %xmm0 + vmovdqu 0x68(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0xa0(%rdi) + vmovdqu 0x78(%rsi), %xmm0 + vmovdqu 0x7c(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0xc0(%rdi) + vmovdqu 0x8c(%rsi), %xmm0 + vmovdqu 0x90(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0xe0(%rdi) + vmovdqu 0xa0(%rsi), %xmm0 + vmovdqu 0xa4(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x100(%rdi) + vmovdqu 0xb4(%rsi), %xmm0 + vmovdqu 0xb8(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x120(%rdi) + vmovdqu 0xc8(%rsi), %xmm0 + vmovdqu 0xcc(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x140(%rdi) + vmovdqu 0xdc(%rsi), %xmm0 + vmovdqu 0xe0(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x160(%rdi) + vmovdqu 0xf0(%rsi), %xmm0 + vmovdqu 0xf4(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x180(%rdi) + vmovdqu 0x104(%rsi), %xmm0 + vmovdqu 0x108(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x1a0(%rdi) + vmovdqu 0x118(%rsi), %xmm0 + vmovdqu 0x11c(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x1c0(%rdi) + vmovdqu 0x12c(%rsi), %xmm0 + vmovdqu 0x130(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x1e0(%rdi) + vmovdqu 0x140(%rsi), %xmm0 + vmovdqu 0x144(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x200(%rdi) + vmovdqu 0x154(%rsi), %xmm0 + vmovdqu 0x158(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x220(%rdi) + vmovdqu 0x168(%rsi), %xmm0 + vmovdqu 0x16c(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x240(%rdi) + vmovdqu 0x17c(%rsi), %xmm0 + vmovdqu 0x180(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x260(%rdi) + vmovdqu 0x190(%rsi), %xmm0 + vmovdqu 0x194(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x280(%rdi) + vmovdqu 0x1a4(%rsi), %xmm0 + vmovdqu 0x1a8(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x2a0(%rdi) + vmovdqu 0x1b8(%rsi), %xmm0 + vmovdqu 0x1bc(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x2c0(%rdi) + vmovdqu 0x1cc(%rsi), %xmm0 + vmovdqu 0x1d0(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x2e0(%rdi) + vmovdqu 0x1e0(%rsi), %xmm0 + vmovdqu 0x1e4(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x300(%rdi) + vmovdqu 0x1f4(%rsi), %xmm0 + vmovdqu 0x1f8(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x320(%rdi) + vmovdqu 0x208(%rsi), %xmm0 + vmovdqu 0x20c(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x340(%rdi) + vmovdqu 0x21c(%rsi), %xmm0 + vmovdqu 0x220(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x360(%rdi) + vmovdqu 0x230(%rsi), %xmm0 + vmovdqu 0x234(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x380(%rdi) + vmovdqu 0x244(%rsi), %xmm0 + vmovdqu 0x248(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x3a0(%rdi) + vmovdqu 0x258(%rsi), %xmm0 + vmovdqu 0x25c(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x3c0(%rdi) + vmovdqu 0x26c(%rsi), %xmm0 + vmovdqu 0x270(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm0, %ymm0 + vpshufb %ymm1, %ymm0, %ymm0 + vpsrlvd %ymm2, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubd %ymm0, %ymm4, %ymm0 + vmovdqa %ymm0, 0x3e0(%rdi) + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(polyz_unpack_19_avx2_asm) + + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && (!MLD_CONFIG_NO_SIGN_API || \ + !MLD_CONFIG_NO_VERIFY_API) && !MLD_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_rej_uniform_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_rej_uniform_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_rej_uniform_avx2_asm.S @@ -0,0 +1,132 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + +/*yaml + Name: rej_uniform_avx2_asm + Description: x86_64 AVX2 rejection sampling of uniform coefficients mod q. + Extract 23-bit values from the input byte buffer and accept those that are + < MLDSA_Q, writing accepted coefficients to the output buffer. Returns the + number of valid coefficients written (at most 256). + Signature: unsigned mld_rej_uniform_avx2_asm(int32_t *r, const uint8_t buf[840], const uint8_t table[256][8]) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t *r + description: Output buffer (256 x int32_t) + rsi: + type: buffer + size_bytes: 840 + permissions: read-only + c_parameter: const uint8_t buf[840] + description: Input buffer (MLD_POLY_UNIFORM_NBLOCKS * SHAKE128_RATE = 5 * 168) + rdx: + type: buffer + size_bytes: 2048 + permissions: read-only + c_parameter: const uint8_t table[256][8] + description: Lookup table (256 x 8 x uint8_t) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_rej_uniform_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(rej_uniform_avx2_asm) +MLD_ASM_FN_SYMBOL(rej_uniform_avx2_asm) + + .cfi_startproc + movabsq $-0xfafbfc00fdff00, %r10 # imm = 0xFF050403FF020100 + vmovq %r10, %xmm0 + movabsq $-0xf4f5f600f7f8fa, %r10 # imm = 0xFF0B0A09FF080706 + vpinsrq $0x1, %r10, %xmm0, %xmm0 + movabsq $-0xf6f7f800f9fafc, %r10 # imm = 0xFF090807FF060504 + vmovq %r10, %xmm3 + movabsq $-0xf0f1f200f3f4f6, %r10 # imm = 0xFF0F0E0DFF0C0B0A + vpinsrq $0x1, %r10, %xmm3, %xmm3 + vinserti128 $0x1, %xmm3, %ymm0, %ymm0 + movl $0x7fffff, %r8d # imm = 0x7FFFFF + vmovd %r8d, %xmm1 + vpbroadcastd %xmm1, %ymm1 + movl $0x7fe001, %r8d # imm = 0x7FE001 + vmovd %r8d, %xmm2 + vpbroadcastd %xmm2, %ymm2 + xorl %eax, %eax + xorl %ecx, %ecx + +Lmld_rej_uniform_avx2_asm_loop: + cmpl $0xf8, %eax + ja Lmld_rej_uniform_avx2_asm_scalar + cmpl $0x328, %ecx # imm = 0x328 + ja Lmld_rej_uniform_avx2_asm_scalar + vmovdqu (%rsi,%rcx), %ymm3 + addl $0x18, %ecx + vpermq $0x94, %ymm3, %ymm3 # ymm3 = ymm3[0,1,1,2] + vpshufb %ymm0, %ymm3, %ymm3 + vpand %ymm1, %ymm3, %ymm3 + vpsubd %ymm2, %ymm3, %ymm4 + vmovmskps %ymm4, %r8d + popcntl %r8d, %r9d + vmovq (%rdx,%r8,8), %xmm4 + vpmovzxbd %xmm4, %ymm4 # ymm4 = xmm4[0],zero,zero,zero,xmm4[1],zero,zero,zero,xmm4[2],zero,zero,zero,xmm4[3],zero,zero,zero,xmm4[4],zero,zero,zero,xmm4[5],zero,zero,zero,xmm4[6],zero,zero,zero,xmm4[7],zero,zero,zero + vpermd %ymm3, %ymm4, %ymm3 + vmovdqu %ymm3, (%rdi,%rax,4) + addl %r9d, %eax + jmp Lmld_rej_uniform_avx2_asm_loop + +Lmld_rej_uniform_avx2_asm_scalar: + cmpl $0x100, %eax # imm = 0x100 + jae Lmld_rej_uniform_avx2_asm_done + cmpl $0x345, %ecx # imm = 0x345 + ja Lmld_rej_uniform_avx2_asm_done + movzwl (%rsi,%rcx), %r8d + movzbl 0x2(%rsi,%rcx), %r9d + shll $0x10, %r9d + orl %r9d, %r8d + andl $0x7fffff, %r8d # imm = 0x7FFFFF + addl $0x3, %ecx + cmpl $0x7fe001, %r8d # imm = 0x7FE001 + jae Lmld_rej_uniform_avx2_asm_scalar + movl %r8d, (%rdi,%rax,4) + addl $0x1, %eax + jmp Lmld_rej_uniform_avx2_asm_scalar + +Lmld_rej_uniform_avx2_asm_done: + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(rej_uniform_avx2_asm) + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_rej_uniform_eta2_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_rej_uniform_eta2_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_rej_uniform_eta2_avx2_asm.S @@ -0,0 +1,205 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + +/*yaml + Name: rej_uniform_eta2_avx2_asm + Description: x86_64 AVX2 rejection sampling of eta=2 secret coefficients. + Extracts 4-bit nibbles from the input byte buffer, accepts those < 15, + and maps an accepted nibble n to the coefficient 2 - (n mod 5) via a + centered modulo-5 reduction, producing values in [-2, 2]. + Signature: unsigned mld_rej_uniform_eta2_avx2_asm(int32_t *r, const uint8_t *buf, const uint8_t table[256][8]) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t *r + description: Output coefficient buffer (256 x int32_t) + rsi: + type: buffer + size_bytes: 136 + permissions: read-only + c_parameter: const uint8_t *buf + description: Input byte buffer (MLD_AVX2_REJ_UNIFORM_ETA2_BUFLEN = 136) + rdx: + type: buffer + size_bytes: 2048 + permissions: read-only + c_parameter: const uint8_t *table + description: Lookup table (256 x 8 uint8_t = mld_rej_uniform_table) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_rej_uniform_eta2_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(rej_uniform_eta2_avx2_asm) +MLD_ASM_FN_SYMBOL(rej_uniform_eta2_avx2_asm) + + .cfi_startproc + movl $0xf0f0f0f, %r8d # imm = 0xF0F0F0F + vmovd %r8d, %xmm3 + vpbroadcastd %xmm3, %ymm3 + movl $0x2020202, %r8d # imm = 0x2020202 + vmovd %r8d, %xmm4 + vpbroadcastd %xmm4, %ymm4 + movl $0xf0f0f0f, %r8d # imm = 0xF0F0F0F + vmovd %r8d, %xmm5 + vpbroadcastd %xmm5, %ymm5 + movl $0xffffe660, %r8d # imm = 0xFFFFE660 + vpinsrw $0x0, %r8d, %xmm6, %xmm6 + vpbroadcastw %xmm6, %ymm6 + movl $0x5, %r8d + vpinsrw $0x0, %r8d, %xmm7, %xmm7 + vpbroadcastw %xmm7, %ymm7 + xorl %eax, %eax + xorl %ecx, %ecx + +Lmld_rej_uniform_eta2_avx2_asm_loop: + cmpl $0xf8, %eax + ja Lmld_rej_uniform_eta2_avx2_asm_scalar + cmpl $0x78, %ecx + ja Lmld_rej_uniform_eta2_avx2_asm_scalar + vpmovzxbw (%rsi,%rcx), %ymm0 + vpsllw $0x4, %ymm0, %ymm1 + vpor %ymm1, %ymm0, %ymm0 + vpand %ymm3, %ymm0, %ymm0 + vpsubb %ymm5, %ymm0, %ymm1 + vpsubb %ymm0, %ymm4, %ymm0 + vpmovmskb %ymm1, %r8d + vextracti128 $0x0, %ymm0, %xmm8 + movzbl %r8b, %r10d + vmovq (%rdx,%r10,8), %xmm9 + vpshufb %xmm9, %xmm8, %xmm9 + vpmovsxbd %xmm9, %ymm1 + vpmulhrsw %ymm6, %ymm1, %ymm2 + vpmullw %ymm7, %ymm2, %ymm2 + vpaddd %ymm2, %ymm1, %ymm1 + vmovdqu %ymm1, (%rdi,%rax,4) + popcntl %r10d, %r9d + addl %r9d, %eax + shrl $0x8, %r8d + addl $0x4, %ecx + cmpl $0xf8, %eax + ja Lmld_rej_uniform_eta2_avx2_asm_scalar + vpsrldq $0x8, %xmm8, %xmm8 # xmm8 = xmm8[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero + movzbl %r8b, %r10d + vmovq (%rdx,%r10,8), %xmm9 + vpshufb %xmm9, %xmm8, %xmm9 + vpmovsxbd %xmm9, %ymm1 + vpmulhrsw %ymm6, %ymm1, %ymm2 + vpmullw %ymm7, %ymm2, %ymm2 + vpaddd %ymm2, %ymm1, %ymm1 + vmovdqu %ymm1, (%rdi,%rax,4) + popcntl %r10d, %r9d + addl %r9d, %eax + shrl $0x8, %r8d + addl $0x4, %ecx + cmpl $0xf8, %eax + ja Lmld_rej_uniform_eta2_avx2_asm_scalar + vextracti128 $0x1, %ymm0, %xmm8 + movzbl %r8b, %r10d + vmovq (%rdx,%r10,8), %xmm9 + vpshufb %xmm9, %xmm8, %xmm9 + vpmovsxbd %xmm9, %ymm1 + vpmulhrsw %ymm6, %ymm1, %ymm2 + vpmullw %ymm7, %ymm2, %ymm2 + vpaddd %ymm2, %ymm1, %ymm1 + vmovdqu %ymm1, (%rdi,%rax,4) + popcntl %r10d, %r9d + addl %r9d, %eax + shrl $0x8, %r8d + addl $0x4, %ecx + cmpl $0xf8, %eax + ja Lmld_rej_uniform_eta2_avx2_asm_scalar + vpsrldq $0x8, %xmm8, %xmm8 # xmm8 = xmm8[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero + movzbl %r8b, %r10d + vmovq (%rdx,%r10,8), %xmm9 + vpshufb %xmm9, %xmm8, %xmm9 + vpmovsxbd %xmm9, %ymm1 + vpmulhrsw %ymm6, %ymm1, %ymm2 + vpmullw %ymm7, %ymm2, %ymm2 + vpaddd %ymm2, %ymm1, %ymm1 + vmovdqu %ymm1, (%rdi,%rax,4) + popcntl %r10d, %r9d + addl %r9d, %eax + addl $0x4, %ecx + jmp Lmld_rej_uniform_eta2_avx2_asm_loop + +Lmld_rej_uniform_eta2_avx2_asm_scalar: + cmpl $0x100, %eax # imm = 0x100 + jae Lmld_rej_uniform_eta2_avx2_asm_done + cmpl $0x88, %ecx + jae Lmld_rej_uniform_eta2_avx2_asm_done + movzbl (%rsi,%rcx), %r11d + incl %ecx + movl %r11d, %r10d + andl $0xf, %r10d + cmpl $0xf, %r10d + jae Lmld_rej_uniform_eta2_avx2_asm_high_nibble + movl %r10d, %r11d + imull $0xcd, %r11d, %r11d + shrl $0xa, %r11d + imull $0x5, %r11d, %r11d + subl %r11d, %r10d + movl $0x2, %r11d + subl %r10d, %r11d + movl %r11d, (%rdi,%rax,4) + incl %eax + cmpl $0x100, %eax # imm = 0x100 + jae Lmld_rej_uniform_eta2_avx2_asm_done + +Lmld_rej_uniform_eta2_avx2_asm_high_nibble: + movzbl -0x1(%rsi,%rcx), %r11d + shrl $0x4, %r11d + andl $0xf, %r11d + cmpl $0xf, %r11d + jae Lmld_rej_uniform_eta2_avx2_asm_scalar + movl %r11d, %r10d + imull $0xcd, %r10d, %r10d + shrl $0xa, %r10d + imull $0x5, %r10d, %r10d + subl %r10d, %r11d + movl $0x2, %r10d + subl %r11d, %r10d + movl %r10d, (%rdi,%rax,4) + incl %eax + jmp Lmld_rej_uniform_eta2_avx2_asm_scalar + +Lmld_rej_uniform_eta2_avx2_asm_done: + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(rej_uniform_eta2_avx2_asm) + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/mldsa_rej_uniform_eta4_avx2_asm.S b/cbits/mldsa/src/native/x86_64/src/mldsa_rej_uniform_eta4_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/mldsa_rej_uniform_eta4_avx2_asm.S @@ -0,0 +1,176 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Dilithium optimized AVX2 implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Dilithium implementation @[REF_AVX2]. + */ + +/*yaml + Name: rej_uniform_eta4_avx2_asm + Description: x86_64 AVX2 rejection sampling of eta=4 secret coefficients. + Extracts 4-bit nibbles from the input byte buffer, accepts those < 9, + and maps an accepted nibble n to the coefficient 4 - n, producing + values in [-4, 4]. + Signature: unsigned mld_rej_uniform_eta4_avx2_asm(int32_t *r, const uint8_t *buf, const uint8_t table[256][8]) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 1024 + permissions: write-only + c_parameter: int32_t *r + description: Output coefficient buffer (256 x int32_t) + rsi: + type: buffer + size_bytes: 272 + permissions: read-only + c_parameter: const uint8_t *buf + description: Input byte buffer (MLD_AVX2_REJ_UNIFORM_ETA4_BUFLEN = 272) + rdx: + type: buffer + size_bytes: 2048 + permissions: read-only + c_parameter: const uint8_t *table + description: Lookup table (256 x 8 uint8_t = mld_rej_uniform_table) +*/ + +#include "../../../common.h" +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mldsa-native source file + * dev/x86_64/src/mldsa_rej_uniform_eta4_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLD_ASM_NAMESPACE(rej_uniform_eta4_avx2_asm) +MLD_ASM_FN_SYMBOL(rej_uniform_eta4_avx2_asm) + + .cfi_startproc + movl $0xf0f0f0f, %r8d # imm = 0xF0F0F0F + vmovd %r8d, %xmm2 + vpbroadcastd %xmm2, %ymm2 + movl $0x4040404, %r8d # imm = 0x4040404 + vmovd %r8d, %xmm3 + vpbroadcastd %xmm3, %ymm3 + movl $0x9090909, %r8d # imm = 0x9090909 + vmovd %r8d, %xmm4 + vpbroadcastd %xmm4, %ymm4 + xorl %eax, %eax + xorl %ecx, %ecx + +Lmld_rej_uniform_eta4_avx2_asm_loop: + cmpl $0xf8, %eax + ja Lmld_rej_uniform_eta4_avx2_asm_scalar + cmpl $0x100, %ecx # imm = 0x100 + ja Lmld_rej_uniform_eta4_avx2_asm_scalar + vpmovzxbw (%rsi,%rcx), %ymm0 + vpsllw $0x4, %ymm0, %ymm1 + vpor %ymm1, %ymm0, %ymm0 + vpand %ymm2, %ymm0, %ymm0 + vpsubb %ymm4, %ymm0, %ymm1 + vpsubb %ymm0, %ymm3, %ymm0 + vpmovmskb %ymm1, %r8d + vextracti128 $0x0, %ymm0, %xmm5 + movzbl %r8b, %r10d + vmovq (%rdx,%r10,8), %xmm6 + vpshufb %xmm6, %xmm5, %xmm6 + vpmovsxbd %xmm6, %ymm1 + vmovdqu %ymm1, (%rdi,%rax,4) + popcntl %r10d, %r9d + addl %r9d, %eax + shrl $0x8, %r8d + addl $0x4, %ecx + cmpl $0xf8, %eax + ja Lmld_rej_uniform_eta4_avx2_asm_scalar + vpsrldq $0x8, %xmm5, %xmm5 # xmm5 = xmm5[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero + movzbl %r8b, %r10d + vmovq (%rdx,%r10,8), %xmm6 + vpshufb %xmm6, %xmm5, %xmm6 + vpmovsxbd %xmm6, %ymm1 + vmovdqu %ymm1, (%rdi,%rax,4) + popcntl %r10d, %r9d + addl %r9d, %eax + shrl $0x8, %r8d + addl $0x4, %ecx + cmpl $0xf8, %eax + ja Lmld_rej_uniform_eta4_avx2_asm_scalar + vextracti128 $0x1, %ymm0, %xmm5 + movzbl %r8b, %r10d + vmovq (%rdx,%r10,8), %xmm6 + vpshufb %xmm6, %xmm5, %xmm6 + vpmovsxbd %xmm6, %ymm1 + vmovdqu %ymm1, (%rdi,%rax,4) + popcntl %r10d, %r9d + addl %r9d, %eax + shrl $0x8, %r8d + addl $0x4, %ecx + cmpl $0xf8, %eax + ja Lmld_rej_uniform_eta4_avx2_asm_scalar + vpsrldq $0x8, %xmm5, %xmm5 # xmm5 = xmm5[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero + movzbl %r8b, %r10d + vmovq (%rdx,%r10,8), %xmm6 + vpshufb %xmm6, %xmm5, %xmm6 + vpmovsxbd %xmm6, %ymm1 + vmovdqu %ymm1, (%rdi,%rax,4) + popcntl %r10d, %r9d + addl %r9d, %eax + addl $0x4, %ecx + jmp Lmld_rej_uniform_eta4_avx2_asm_loop + +Lmld_rej_uniform_eta4_avx2_asm_scalar: + cmpl $0x100, %eax # imm = 0x100 + jae Lmld_rej_uniform_eta4_avx2_asm_done + cmpl $0x110, %ecx # imm = 0x110 + jae Lmld_rej_uniform_eta4_avx2_asm_done + movzbl (%rsi,%rcx), %r11d + incl %ecx + movl %r11d, %r10d + andl $0xf, %r10d + cmpl $0x9, %r10d + jae Lmld_rej_uniform_eta4_avx2_asm_high_nibble + movl $0x4, %r9d + subl %r10d, %r9d + movl %r9d, (%rdi,%rax,4) + incl %eax + cmpl $0x100, %eax # imm = 0x100 + jae Lmld_rej_uniform_eta4_avx2_asm_done + +Lmld_rej_uniform_eta4_avx2_asm_high_nibble: + shrl $0x4, %r11d + andl $0xf, %r11d + cmpl $0x9, %r11d + jae Lmld_rej_uniform_eta4_avx2_asm_scalar + movl $0x4, %r10d + subl %r11d, %r10d + movl %r10d, (%rdi,%rax,4) + incl %eax + jmp Lmld_rej_uniform_eta4_avx2_asm_scalar + +Lmld_rej_uniform_eta4_avx2_asm_done: + retq + .cfi_endproc + +MLD_ASM_FN_SIZE(rej_uniform_eta4_avx2_asm) + +#endif /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mldsa/src/native/x86_64/src/rej_uniform_table.c b/cbits/mldsa/src/native/x86_64/src/rej_uniform_table.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/native/x86_64/src/rej_uniform_table.c @@ -0,0 +1,161 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mldsa-native repository. + * Do not modify it directly. + */ + +#include "../../../common.h" + +#if defined(MLD_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) + +#include "arith_native_x86_64.h" + +/* + * Lookup table used by rejection sampling. + * See autogen for details. + */ +MLD_ALIGN MLD_INTERNAL_DATA_DEFINITION const uint8_t + mld_rej_uniform_table[256][8] = { + {0, 0, 0, 0, 0, 0, 0, 0}, {0, 0, 0, 0, 0, 0, 0, 0}, + {1, 0, 0, 0, 0, 0, 0, 0}, {0, 1, 0, 0, 0, 0, 0, 0}, + {2, 0, 0, 0, 0, 0, 0, 0}, {0, 2, 0, 0, 0, 0, 0, 0}, + {1, 2, 0, 0, 0, 0, 0, 0}, {0, 1, 2, 0, 0, 0, 0, 0}, + {3, 0, 0, 0, 0, 0, 0, 0}, {0, 3, 0, 0, 0, 0, 0, 0}, + {1, 3, 0, 0, 0, 0, 0, 0}, {0, 1, 3, 0, 0, 0, 0, 0}, + {2, 3, 0, 0, 0, 0, 0, 0}, {0, 2, 3, 0, 0, 0, 0, 0}, + {1, 2, 3, 0, 0, 0, 0, 0}, {0, 1, 2, 3, 0, 0, 0, 0}, + {4, 0, 0, 0, 0, 0, 0, 0}, {0, 4, 0, 0, 0, 0, 0, 0}, + {1, 4, 0, 0, 0, 0, 0, 0}, {0, 1, 4, 0, 0, 0, 0, 0}, + {2, 4, 0, 0, 0, 0, 0, 0}, {0, 2, 4, 0, 0, 0, 0, 0}, + {1, 2, 4, 0, 0, 0, 0, 0}, {0, 1, 2, 4, 0, 0, 0, 0}, + {3, 4, 0, 0, 0, 0, 0, 0}, {0, 3, 4, 0, 0, 0, 0, 0}, + {1, 3, 4, 0, 0, 0, 0, 0}, {0, 1, 3, 4, 0, 0, 0, 0}, + {2, 3, 4, 0, 0, 0, 0, 0}, {0, 2, 3, 4, 0, 0, 0, 0}, + {1, 2, 3, 4, 0, 0, 0, 0}, {0, 1, 2, 3, 4, 0, 0, 0}, + {5, 0, 0, 0, 0, 0, 0, 0}, {0, 5, 0, 0, 0, 0, 0, 0}, + {1, 5, 0, 0, 0, 0, 0, 0}, {0, 1, 5, 0, 0, 0, 0, 0}, + {2, 5, 0, 0, 0, 0, 0, 0}, {0, 2, 5, 0, 0, 0, 0, 0}, + {1, 2, 5, 0, 0, 0, 0, 0}, {0, 1, 2, 5, 0, 0, 0, 0}, + {3, 5, 0, 0, 0, 0, 0, 0}, {0, 3, 5, 0, 0, 0, 0, 0}, + {1, 3, 5, 0, 0, 0, 0, 0}, {0, 1, 3, 5, 0, 0, 0, 0}, + {2, 3, 5, 0, 0, 0, 0, 0}, {0, 2, 3, 5, 0, 0, 0, 0}, + {1, 2, 3, 5, 0, 0, 0, 0}, {0, 1, 2, 3, 5, 0, 0, 0}, + {4, 5, 0, 0, 0, 0, 0, 0}, {0, 4, 5, 0, 0, 0, 0, 0}, + {1, 4, 5, 0, 0, 0, 0, 0}, {0, 1, 4, 5, 0, 0, 0, 0}, + {2, 4, 5, 0, 0, 0, 0, 0}, {0, 2, 4, 5, 0, 0, 0, 0}, + {1, 2, 4, 5, 0, 0, 0, 0}, {0, 1, 2, 4, 5, 0, 0, 0}, + {3, 4, 5, 0, 0, 0, 0, 0}, {0, 3, 4, 5, 0, 0, 0, 0}, + {1, 3, 4, 5, 0, 0, 0, 0}, {0, 1, 3, 4, 5, 0, 0, 0}, + {2, 3, 4, 5, 0, 0, 0, 0}, {0, 2, 3, 4, 5, 0, 0, 0}, + {1, 2, 3, 4, 5, 0, 0, 0}, {0, 1, 2, 3, 4, 5, 0, 0}, + {6, 0, 0, 0, 0, 0, 0, 0}, {0, 6, 0, 0, 0, 0, 0, 0}, + {1, 6, 0, 0, 0, 0, 0, 0}, {0, 1, 6, 0, 0, 0, 0, 0}, + {2, 6, 0, 0, 0, 0, 0, 0}, {0, 2, 6, 0, 0, 0, 0, 0}, + {1, 2, 6, 0, 0, 0, 0, 0}, {0, 1, 2, 6, 0, 0, 0, 0}, + {3, 6, 0, 0, 0, 0, 0, 0}, {0, 3, 6, 0, 0, 0, 0, 0}, + {1, 3, 6, 0, 0, 0, 0, 0}, {0, 1, 3, 6, 0, 0, 0, 0}, + {2, 3, 6, 0, 0, 0, 0, 0}, {0, 2, 3, 6, 0, 0, 0, 0}, + {1, 2, 3, 6, 0, 0, 0, 0}, {0, 1, 2, 3, 6, 0, 0, 0}, + {4, 6, 0, 0, 0, 0, 0, 0}, {0, 4, 6, 0, 0, 0, 0, 0}, + {1, 4, 6, 0, 0, 0, 0, 0}, {0, 1, 4, 6, 0, 0, 0, 0}, + {2, 4, 6, 0, 0, 0, 0, 0}, {0, 2, 4, 6, 0, 0, 0, 0}, + {1, 2, 4, 6, 0, 0, 0, 0}, {0, 1, 2, 4, 6, 0, 0, 0}, + {3, 4, 6, 0, 0, 0, 0, 0}, {0, 3, 4, 6, 0, 0, 0, 0}, + {1, 3, 4, 6, 0, 0, 0, 0}, {0, 1, 3, 4, 6, 0, 0, 0}, + {2, 3, 4, 6, 0, 0, 0, 0}, {0, 2, 3, 4, 6, 0, 0, 0}, + {1, 2, 3, 4, 6, 0, 0, 0}, {0, 1, 2, 3, 4, 6, 0, 0}, + {5, 6, 0, 0, 0, 0, 0, 0}, {0, 5, 6, 0, 0, 0, 0, 0}, + {1, 5, 6, 0, 0, 0, 0, 0}, {0, 1, 5, 6, 0, 0, 0, 0}, + {2, 5, 6, 0, 0, 0, 0, 0}, {0, 2, 5, 6, 0, 0, 0, 0}, + {1, 2, 5, 6, 0, 0, 0, 0}, {0, 1, 2, 5, 6, 0, 0, 0}, + {3, 5, 6, 0, 0, 0, 0, 0}, {0, 3, 5, 6, 0, 0, 0, 0}, + {1, 3, 5, 6, 0, 0, 0, 0}, {0, 1, 3, 5, 6, 0, 0, 0}, + {2, 3, 5, 6, 0, 0, 0, 0}, {0, 2, 3, 5, 6, 0, 0, 0}, + {1, 2, 3, 5, 6, 0, 0, 0}, {0, 1, 2, 3, 5, 6, 0, 0}, + {4, 5, 6, 0, 0, 0, 0, 0}, {0, 4, 5, 6, 0, 0, 0, 0}, + {1, 4, 5, 6, 0, 0, 0, 0}, {0, 1, 4, 5, 6, 0, 0, 0}, + {2, 4, 5, 6, 0, 0, 0, 0}, {0, 2, 4, 5, 6, 0, 0, 0}, + {1, 2, 4, 5, 6, 0, 0, 0}, {0, 1, 2, 4, 5, 6, 0, 0}, + {3, 4, 5, 6, 0, 0, 0, 0}, {0, 3, 4, 5, 6, 0, 0, 0}, + {1, 3, 4, 5, 6, 0, 0, 0}, {0, 1, 3, 4, 5, 6, 0, 0}, + {2, 3, 4, 5, 6, 0, 0, 0}, {0, 2, 3, 4, 5, 6, 0, 0}, + {1, 2, 3, 4, 5, 6, 0, 0}, {0, 1, 2, 3, 4, 5, 6, 0}, + {7, 0, 0, 0, 0, 0, 0, 0}, {0, 7, 0, 0, 0, 0, 0, 0}, + {1, 7, 0, 0, 0, 0, 0, 0}, {0, 1, 7, 0, 0, 0, 0, 0}, + {2, 7, 0, 0, 0, 0, 0, 0}, {0, 2, 7, 0, 0, 0, 0, 0}, + {1, 2, 7, 0, 0, 0, 0, 0}, {0, 1, 2, 7, 0, 0, 0, 0}, + {3, 7, 0, 0, 0, 0, 0, 0}, {0, 3, 7, 0, 0, 0, 0, 0}, + {1, 3, 7, 0, 0, 0, 0, 0}, {0, 1, 3, 7, 0, 0, 0, 0}, + {2, 3, 7, 0, 0, 0, 0, 0}, {0, 2, 3, 7, 0, 0, 0, 0}, + {1, 2, 3, 7, 0, 0, 0, 0}, {0, 1, 2, 3, 7, 0, 0, 0}, + {4, 7, 0, 0, 0, 0, 0, 0}, {0, 4, 7, 0, 0, 0, 0, 0}, + {1, 4, 7, 0, 0, 0, 0, 0}, {0, 1, 4, 7, 0, 0, 0, 0}, + {2, 4, 7, 0, 0, 0, 0, 0}, {0, 2, 4, 7, 0, 0, 0, 0}, + {1, 2, 4, 7, 0, 0, 0, 0}, {0, 1, 2, 4, 7, 0, 0, 0}, + {3, 4, 7, 0, 0, 0, 0, 0}, {0, 3, 4, 7, 0, 0, 0, 0}, + {1, 3, 4, 7, 0, 0, 0, 0}, {0, 1, 3, 4, 7, 0, 0, 0}, + {2, 3, 4, 7, 0, 0, 0, 0}, {0, 2, 3, 4, 7, 0, 0, 0}, + {1, 2, 3, 4, 7, 0, 0, 0}, {0, 1, 2, 3, 4, 7, 0, 0}, + {5, 7, 0, 0, 0, 0, 0, 0}, {0, 5, 7, 0, 0, 0, 0, 0}, + {1, 5, 7, 0, 0, 0, 0, 0}, {0, 1, 5, 7, 0, 0, 0, 0}, + {2, 5, 7, 0, 0, 0, 0, 0}, {0, 2, 5, 7, 0, 0, 0, 0}, + {1, 2, 5, 7, 0, 0, 0, 0}, {0, 1, 2, 5, 7, 0, 0, 0}, + {3, 5, 7, 0, 0, 0, 0, 0}, {0, 3, 5, 7, 0, 0, 0, 0}, + {1, 3, 5, 7, 0, 0, 0, 0}, {0, 1, 3, 5, 7, 0, 0, 0}, + {2, 3, 5, 7, 0, 0, 0, 0}, {0, 2, 3, 5, 7, 0, 0, 0}, + {1, 2, 3, 5, 7, 0, 0, 0}, {0, 1, 2, 3, 5, 7, 0, 0}, + {4, 5, 7, 0, 0, 0, 0, 0}, {0, 4, 5, 7, 0, 0, 0, 0}, + {1, 4, 5, 7, 0, 0, 0, 0}, {0, 1, 4, 5, 7, 0, 0, 0}, + {2, 4, 5, 7, 0, 0, 0, 0}, {0, 2, 4, 5, 7, 0, 0, 0}, + {1, 2, 4, 5, 7, 0, 0, 0}, {0, 1, 2, 4, 5, 7, 0, 0}, + {3, 4, 5, 7, 0, 0, 0, 0}, {0, 3, 4, 5, 7, 0, 0, 0}, + {1, 3, 4, 5, 7, 0, 0, 0}, {0, 1, 3, 4, 5, 7, 0, 0}, + {2, 3, 4, 5, 7, 0, 0, 0}, {0, 2, 3, 4, 5, 7, 0, 0}, + {1, 2, 3, 4, 5, 7, 0, 0}, {0, 1, 2, 3, 4, 5, 7, 0}, + {6, 7, 0, 0, 0, 0, 0, 0}, {0, 6, 7, 0, 0, 0, 0, 0}, + {1, 6, 7, 0, 0, 0, 0, 0}, {0, 1, 6, 7, 0, 0, 0, 0}, + {2, 6, 7, 0, 0, 0, 0, 0}, {0, 2, 6, 7, 0, 0, 0, 0}, + {1, 2, 6, 7, 0, 0, 0, 0}, {0, 1, 2, 6, 7, 0, 0, 0}, + {3, 6, 7, 0, 0, 0, 0, 0}, {0, 3, 6, 7, 0, 0, 0, 0}, + {1, 3, 6, 7, 0, 0, 0, 0}, {0, 1, 3, 6, 7, 0, 0, 0}, + {2, 3, 6, 7, 0, 0, 0, 0}, {0, 2, 3, 6, 7, 0, 0, 0}, + {1, 2, 3, 6, 7, 0, 0, 0}, {0, 1, 2, 3, 6, 7, 0, 0}, + {4, 6, 7, 0, 0, 0, 0, 0}, {0, 4, 6, 7, 0, 0, 0, 0}, + {1, 4, 6, 7, 0, 0, 0, 0}, {0, 1, 4, 6, 7, 0, 0, 0}, + {2, 4, 6, 7, 0, 0, 0, 0}, {0, 2, 4, 6, 7, 0, 0, 0}, + {1, 2, 4, 6, 7, 0, 0, 0}, {0, 1, 2, 4, 6, 7, 0, 0}, + {3, 4, 6, 7, 0, 0, 0, 0}, {0, 3, 4, 6, 7, 0, 0, 0}, + {1, 3, 4, 6, 7, 0, 0, 0}, {0, 1, 3, 4, 6, 7, 0, 0}, + {2, 3, 4, 6, 7, 0, 0, 0}, {0, 2, 3, 4, 6, 7, 0, 0}, + {1, 2, 3, 4, 6, 7, 0, 0}, {0, 1, 2, 3, 4, 6, 7, 0}, + {5, 6, 7, 0, 0, 0, 0, 0}, {0, 5, 6, 7, 0, 0, 0, 0}, + {1, 5, 6, 7, 0, 0, 0, 0}, {0, 1, 5, 6, 7, 0, 0, 0}, + {2, 5, 6, 7, 0, 0, 0, 0}, {0, 2, 5, 6, 7, 0, 0, 0}, + {1, 2, 5, 6, 7, 0, 0, 0}, {0, 1, 2, 5, 6, 7, 0, 0}, + {3, 5, 6, 7, 0, 0, 0, 0}, {0, 3, 5, 6, 7, 0, 0, 0}, + {1, 3, 5, 6, 7, 0, 0, 0}, {0, 1, 3, 5, 6, 7, 0, 0}, + {2, 3, 5, 6, 7, 0, 0, 0}, {0, 2, 3, 5, 6, 7, 0, 0}, + {1, 2, 3, 5, 6, 7, 0, 0}, {0, 1, 2, 3, 5, 6, 7, 0}, + {4, 5, 6, 7, 0, 0, 0, 0}, {0, 4, 5, 6, 7, 0, 0, 0}, + {1, 4, 5, 6, 7, 0, 0, 0}, {0, 1, 4, 5, 6, 7, 0, 0}, + {2, 4, 5, 6, 7, 0, 0, 0}, {0, 2, 4, 5, 6, 7, 0, 0}, + {1, 2, 4, 5, 6, 7, 0, 0}, {0, 1, 2, 4, 5, 6, 7, 0}, + {3, 4, 5, 6, 7, 0, 0, 0}, {0, 3, 4, 5, 6, 7, 0, 0}, + {1, 3, 4, 5, 6, 7, 0, 0}, {0, 1, 3, 4, 5, 6, 7, 0}, + {2, 3, 4, 5, 6, 7, 0, 0}, {0, 2, 3, 4, 5, 6, 7, 0}, + {1, 2, 3, 4, 5, 6, 7, 0}, {0, 1, 2, 3, 4, 5, 6, 7}, +}; + +#else /* MLD_ARITH_BACKEND_X86_64_DEFAULT && !MLD_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +MLD_EMPTY_CU(avx2_rej_uniform_table) + +#endif /* !(MLD_ARITH_BACKEND_X86_64_DEFAULT && \ + !MLD_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mldsa/src/packing.c b/cbits/mldsa/src/packing.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/packing.c @@ -0,0 +1,213 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#include + +#include "common.h" +#include "packing.h" +#include "poly.h" +#include "polyvec.h" +#include "rounding.h" + +/* Parameter set namespacing + * This is to facilitate building multiple instances + * of mldsa-native (e.g. with varying parameter sets) + * within a single compilation unit. */ +/* End of parameter set namespacing */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +MLD_INTERNAL_API +void mld_unpack_pk_t1(mld_poly *t1, + const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + unsigned int i) +{ + mld_polyt1_unpack(t1, pk + MLDSA_PK_T1_OFFSET + i * MLDSA_POLYT1_PACKEDBYTES); +} +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +MLD_INTERNAL_API +void mld_pack_sk_s1(uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + const mld_polyvecl *s1) +{ + mld_polyvecl_pack_eta(sk + MLDSA_SK_S1_OFFSET, s1); +} + +MLD_INTERNAL_API +void mld_pack_sk_rho_key_tr_s2(uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + const uint8_t rho[MLDSA_SEEDBYTES], + const uint8_t tr[MLDSA_TRBYTES], + const uint8_t key[MLDSA_SEEDBYTES], + const mld_polyveck *s2) +{ + mld_memcpy(sk + MLDSA_SK_RHO_OFFSET, rho, MLDSA_SEEDBYTES); + mld_memcpy(sk + MLDSA_SK_KEY_OFFSET, key, MLDSA_SEEDBYTES); + mld_memcpy(sk + MLDSA_SK_TR_OFFSET, tr, MLDSA_TRBYTES); + /* s1 already packed via mld_pack_sk_s1 */ + mld_polyveck_pack_eta(sk + MLDSA_SK_S2_OFFSET, s2); + /* t0 already packed via mld_compute_pack_t0_t1 */ +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +MLD_INTERNAL_API +void mld_unpack_sk(uint8_t rho[MLDSA_SEEDBYTES], uint8_t tr[MLDSA_TRBYTES], + uint8_t key[MLDSA_SEEDBYTES], mld_sk_t0hat *t0, + mld_sk_s1hat *s1, mld_sk_s2hat *s2, + const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES]) +{ + mld_memcpy(rho, sk + MLDSA_SK_RHO_OFFSET, MLDSA_SEEDBYTES); + mld_memcpy(key, sk + MLDSA_SK_KEY_OFFSET, MLDSA_SEEDBYTES); + mld_memcpy(tr, sk + MLDSA_SK_TR_OFFSET, MLDSA_TRBYTES); + mld_unpack_sk_s1hat(s1, sk + MLDSA_SK_S1_OFFSET); + mld_unpack_sk_s2hat(s2, sk + MLDSA_SK_S2_OFFSET); + mld_unpack_sk_t0hat(t0, sk + MLDSA_SK_T0_OFFSET); +} + +MLD_INTERNAL_API +void mld_pack_sig_c(uint8_t sig[MLDSA_CRYPTO_BYTES], + const uint8_t c[MLDSA_CTILDEBYTES]) +{ + mld_memcpy(sig, c, MLDSA_CTILDEBYTES); +} + +MLD_INTERNAL_API +int mld_pack_sig_h(uint8_t sig[MLDSA_CRYPTO_BYTES], const mld_polyveck *w0, + const mld_polyveck *w1) +{ + unsigned int j, k, n; + + /* The hint section of sig[] is MLDSA_POLYVECH_PACKEDBYTES long, where + * MLDSA_POLYVECH_PACKEDBYTES = MLDSA_OMEGA + MLDSA_K. + * + * The first OMEGA bytes record the index numbers of the coefficients + * that are not equal to 0. + * + * The final K bytes record a running tally of the number of hints + * coming from each of the K polynomials. */ + uint8_t *sig_h = sig + MLDSA_SIG_H_OFFSET; + + mld_memset(sig_h, 0, MLDSA_POLYVECH_PACKEDBYTES); + n = 0; + + /* For each coefficient of each polynomial, compute its hint bit and, if + * non-zero, record the index in the hint section of sig. If recording the + * hint would overflow the OMEGA-sized index array, abort early and return + * MLD_ERR_FAIL. The caller is expected to reject the signature in that case. + * + * Constant time: At this point w0/w1 are public (see comment in sign.c + * before the call), so a data-dependent early return is fine. */ + for (k = 0; k < MLDSA_K; k++) + __loop__( + assigns(k, j, n, memory_slice(sig_h, MLDSA_POLYVECH_PACKEDBYTES)) + invariant(k <= MLDSA_K && n <= MLDSA_OMEGA) + decreases(MLDSA_K - k) + ) + { + for (j = 0; j < MLDSA_N; j++) + __loop__( + assigns(j, n, memory_slice(sig_h, MLDSA_POLYVECH_PACKEDBYTES)) + invariant(j <= MLDSA_N && n <= MLDSA_OMEGA) + decreases(MLDSA_N - j) + ) + { + const unsigned int hint_bit = + mld_make_hint(w0->vec[k].coeffs[j], w1->vec[k].coeffs[j]); + if (hint_bit) + { + if (n == MLDSA_OMEGA) + { + return MLD_ERR_FAIL; + } + /* Safety: branch above ensures n < MLDSA_OMEGA so n is a valid index + * into the OMEGA-sized index array; j < MLDSA_N <= 256 fits in + * uint8_t. */ + sig_h[n] = (uint8_t)j; + n++; + } + } + /* Record the running tally into the correct slot for this polynomial. + * Safety: k < MLDSA_K, so MLDSA_OMEGA + k is a valid index into the + * K-byte tally tail; n <= MLDSA_OMEGA fits in uint8_t. */ + sig_h[MLDSA_OMEGA + k] = (uint8_t)n; + } + return 0; +} + +MLD_INTERNAL_API +void mld_pack_sig_z(uint8_t sig[MLDSA_CRYPTO_BYTES], const mld_poly *zi, + unsigned i) +{ + mld_polyz_pack(sig + MLDSA_SIG_Z_OFFSET + i * MLDSA_POLYZ_PACKEDBYTES, zi); +} +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +MLD_INTERNAL_API +int mld_sig_unpack_hints(mld_poly *h, const uint8_t sig[MLDSA_CRYPTO_BYTES], + unsigned int i) +{ + const uint8_t *packed_hints = sig + MLDSA_SIG_H_OFFSET; + const unsigned int old_hint_count = + (i == 0) ? 0 : packed_hints[MLDSA_OMEGA + i - 1]; + const unsigned int new_hint_count = packed_hints[MLDSA_OMEGA + i]; + unsigned int j; + + if (new_hint_count < old_hint_count || new_hint_count > MLDSA_OMEGA) + { + return MLD_ERR_FAIL; + } + + mld_memset(h, 0, sizeof(mld_poly)); + + for (j = old_hint_count; j < new_hint_count; ++j) + __loop__( + invariant(j >= old_hint_count && j <= new_hint_count && + new_hint_count <= MLDSA_OMEGA) + invariant(array_bound(h->coeffs, 0, MLDSA_N, 0, 2)) + invariant(forall(p, 0, MLDSA_N, + (h->coeffs[p] == 1) == + exists(hj, old_hint_count, j, packed_hints[hj] == p))) + decreases(new_hint_count - j) + ) + { + if (j > old_hint_count && packed_hints[j] <= packed_hints[j - 1]) + { + return MLD_ERR_FAIL; + } + /* Safety: packed_hints[j] is uint8_t (<= 255) and MLDSA_N == 256. */ + h->coeffs[packed_hints[j]] = 1; + } + + /* On the last row, also verify that the trailing index slots are zero. */ + if (i == MLDSA_K - 1) + { + for (j = new_hint_count; j < MLDSA_OMEGA; ++j) + __loop__( + invariant(j <= MLDSA_OMEGA) + decreases(MLDSA_OMEGA - j) + ) + { + if (packed_hints[j] != 0) + { + return MLD_ERR_FAIL; + } + } + } + + /* On success, h->coeffs[p] is 1 exactly for the hint indices decoded for this + * row, i.e. packed_hints[old_hint_count, new_hint_count). Asserted here + * rather than posted as a contract to not unnecessarily increase proof + * complexity at callers that do not need the functional description. */ + cassert(forall( + p, 0, MLDSA_N, + (h->coeffs[p] == 1) == + exists(hj, old_hint_count, new_hint_count, packed_hints[hj] == p))); + + return 0; +} +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. + * Don't modify by hand -- this is auto-generated by scripts/autogen. */ diff --git a/cbits/mldsa/src/packing.h b/cbits/mldsa/src/packing.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/packing.h @@ -0,0 +1,277 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + */ + +#ifndef MLD_PACKING_H +#define MLD_PACKING_H + +#include "polyvec.h" +#include "polyvec_lazy.h" + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +#define mld_pack_sk_s1 MLD_NAMESPACE_KL(pack_sk_s1) +/** + * Bit-pack the s1 component into the secret key. + * + * @spec{Partially implements @[FIPS204, Algorithm 24, skEncode] (s1 + * component).} + * + * @param[out] sk Output byte array. + * @param[in] s1 Pointer to vector s1. + */ +MLD_INTERNAL_API +void mld_pack_sk_s1(uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + const mld_polyvecl *s1) +__contract__( + requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) + requires(memory_no_alias(s1, sizeof(mld_polyvecl))) + requires(forall(k1, 0, MLDSA_L, + array_abs_bound(s1->vec[k1].coeffs, 0, MLDSA_N, MLDSA_ETA + 1))) + assigns(memory_slice(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) +); + +#define mld_pack_sk_rho_key_tr_s2 MLD_NAMESPACE_KL(pack_sk_rho_key_tr_s2) +/** + * Bit-pack rho, key, tr, s2 into the secret key. + * + * s1 must already be packed via mld_pack_sk_s1, and t0 via + * mld_compute_pack_t0_t1. + * + * @spec{Partially implements @[FIPS204, Algorithm 24, skEncode] (rho, key, tr, + * s2 components).} + * + * @param[out] sk Output byte array. + * @param[in] rho Byte array containing rho. + * @param[in] tr Byte array containing tr. + * @param[in] key Byte array containing key. + * @param[in] s2 Pointer to vector s2. + */ +MLD_INTERNAL_API +void mld_pack_sk_rho_key_tr_s2(uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + const uint8_t rho[MLDSA_SEEDBYTES], + const uint8_t tr[MLDSA_TRBYTES], + const uint8_t key[MLDSA_SEEDBYTES], + const mld_polyveck *s2) +__contract__( + requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) + requires(memory_no_alias(rho, MLDSA_SEEDBYTES)) + requires(memory_no_alias(tr, MLDSA_TRBYTES)) + requires(memory_no_alias(key, MLDSA_SEEDBYTES)) + requires(memory_no_alias(s2, sizeof(mld_polyveck))) + requires(forall(k2, 0, MLDSA_K, + array_abs_bound(s2->vec[k2].coeffs, 0, MLDSA_N, MLDSA_ETA + 1))) + assigns(memory_slice(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + + +#if !defined(MLD_CONFIG_NO_SIGN_API) +#define mld_pack_sig_c MLD_NAMESPACE_KL(pack_sig_c) +/** + * Bit-pack challenge c into sig = (c, z, h). + * + * @spec{Partially implements @[FIPS204, Algorithm 26, sigEncode] (c + * component).} + * + * @param[out] sig Output byte array. + * @param[in] c Pointer to challenge hash. + */ +MLD_INTERNAL_API +void mld_pack_sig_c(uint8_t sig[MLDSA_CRYPTO_BYTES], + const uint8_t c[MLDSA_CTILDEBYTES]) +__contract__( + requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(c, MLDSA_CTILDEBYTES)) + assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES)) +); + +#define mld_pack_sig_h MLD_NAMESPACE_KL(pack_sig_h) +/** + * Compute hints from (w0, w1) and pack them into the hint section of sig. + * + * @spec{Combines the hint computation @[FIPS204, Algorithm 39, MakeHint] with + * the packing @[FIPS204, Algorithm 20, HintBitPack] (the h component of + * @[FIPS204, Algorithm 26, sigEncode]): it computes the hint vector h from + * (w0, w1) and packs it, rather than receiving a ready-made h as HintBitPack + * does. The hints are computed via mld_make_hint (rounding.h), a specialized + * MakeHint valid only for the values arising during signing; see the block + * comment in mld_attempt_signature_generation (sign.c).} + * + * @param[in,out] sig Byte array containing signature. + * @param[in] w0 Pointer to low part of input vector. + * @param[in] w1 Pointer to high part of input vector. + * + * @retval 0 Success. + * @retval MLD_ERR_FAIL The total number of hints exceeds MLDSA_OMEGA. In this + * case the hint section of sig is left in a + * partially-written state and the caller must reject the + * signature. + */ +MLD_INTERNAL_API +MLD_MUST_CHECK_RETURN_VALUE +int mld_pack_sig_h(uint8_t sig[MLDSA_CRYPTO_BYTES], const mld_polyveck *w0, + const mld_polyveck *w1) +__contract__( + requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(w0, sizeof(mld_polyveck))) + requires(memory_no_alias(w1, sizeof(mld_polyveck))) + assigns(memory_slice(sig + MLDSA_SIG_H_OFFSET, MLDSA_POLYVECH_PACKEDBYTES)) + ensures(return_value == 0 || return_value == MLD_ERR_FAIL) +); + +#define mld_pack_sig_z MLD_NAMESPACE_KL(pack_sig_z) +/** + * Bit-pack single polynomial of z component of sig = (c, z, h). + * + * The c and h components are packed separately using mld_pack_sig_c and + * mld_pack_sig_h. + * + * @spec{Partially implements @[FIPS204, Algorithm 26, sigEncode] (one + * polynomial of the z component).} + * + * @param[in,out] sig Output byte array. + * @param[in] zi Pointer to a single polynomial in z. + * @param i Index of zi in vector z. + */ +MLD_INTERNAL_API +void mld_pack_sig_z(uint8_t sig[MLDSA_CRYPTO_BYTES], const mld_poly *zi, + unsigned i) +__contract__( + requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(zi, sizeof(mld_poly))) + requires(i < MLDSA_L) + requires(array_bound(zi->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) + assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES)) +); +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +#define mld_unpack_pk_t1 MLD_NAMESPACE_KL(unpack_pk_t1) +/** + * Unpack a single polynomial of the t1 component of a public key + * pk = (rho, t1). + * + * @spec{Partially implements @[FIPS204, Algorithm 23, pkDecode] (one polynomial + * of t1).} + * + * @param[out] t1 Pointer to output polynomial t1[i]. + * @param[in] pk Byte array containing bit-packed pk. + * @param i Row index, must be < MLDSA_K. + */ +MLD_INTERNAL_API +void mld_unpack_pk_t1(mld_poly *t1, + const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + unsigned int i) +__contract__( + requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES)) + requires(memory_no_alias(t1, sizeof(mld_poly))) + requires(i < MLDSA_K) + assigns(memory_slice(t1, sizeof(mld_poly))) + ensures(array_bound(t1->coeffs, 0, MLDSA_N, 0, 1 << 10)) +); +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +#define mld_unpack_sk MLD_NAMESPACE_KL(unpack_sk) +/** + * Unpack secret key sk = (rho, tr, key, t0, s1, s2). + * + * NOTE: In REDUCE_RAM mode, s1/s2/t0 borrow from sk rather than copying. + * + * @spec{Implements @[FIPS204, Algorithm 25, skDecode].} + * + * @param[out] rho Output byte array for rho. + * @param[out] tr Output byte array for tr. + * @param[out] key Output byte array for key. + * @param[out] t0 Pointer to output vector t0. + * @param[out] s1 Pointer to output vector s1. + * @param[out] s2 Pointer to output vector s2. + * @param[in] sk Byte array containing bit-packed sk. + */ +MLD_INTERNAL_API +void mld_unpack_sk(uint8_t rho[MLDSA_SEEDBYTES], uint8_t tr[MLDSA_TRBYTES], + uint8_t key[MLDSA_SEEDBYTES], mld_sk_t0hat *t0, + mld_sk_s1hat *s1, mld_sk_s2hat *s2, + const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES]) +__contract__( + requires(memory_no_alias(rho, MLDSA_SEEDBYTES)) + requires(memory_no_alias(tr, MLDSA_TRBYTES)) + requires(memory_no_alias(key, MLDSA_SEEDBYTES)) + requires(memory_no_alias(t0, sizeof(mld_sk_t0hat))) + requires(memory_no_alias(s1, sizeof(mld_sk_s1hat))) + requires(memory_no_alias(s2, sizeof(mld_sk_s2hat))) + requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) + assigns(memory_slice(rho, MLDSA_SEEDBYTES)) + assigns(memory_slice(tr, MLDSA_TRBYTES)) + assigns(memory_slice(key, MLDSA_SEEDBYTES)) + assigns(memory_slice(t0, sizeof(mld_sk_t0hat))) + assigns(memory_slice(s1, sizeof(mld_sk_s1hat))) + assigns(memory_slice(s2, sizeof(mld_sk_s2hat))) + MLD_IF_NOT_REDUCE_RAM( + ensures(forall(k0, 0, MLDSA_K, + array_abs_bound(t0->vec.vec[k0].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + ensures(forall(k1, 0, MLDSA_L, + array_abs_bound(s1->vec.vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + ensures(forall(k2, 0, MLDSA_K, + array_abs_bound(s2->vec.vec[k2].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + ) + MLD_IF_REDUCE_RAM( + ensures(s1->packed == old(sk) + MLDSA_SK_S1_OFFSET) + ensures(s2->packed == old(sk) + MLDSA_SK_S2_OFFSET) + ensures(t0->packed == old(sk) + MLDSA_SK_T0_OFFSET) + ) +); +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +#define mld_sig_unpack_hints MLD_NAMESPACE_KL(sig_unpack_hints) +/** + * Decode and validate a single row of the hint vector h from a signature + * buffer. + * + * The hint encoding is shared across all rows (a count array followed by a + * single index list), so this function performs the validation relevant to + * row i: + * - the i'th hint count is non-decreasing and bounded by MLDSA_OMEGA; + * - the indices for row i are strictly ascending; + * - on i == MLDSA_K - 1, the trailing index slots are zero. + * + * Callers must invoke this for every i in [0, 1, .., MLDSA_K - 1]; if any + * call returns MLD_ERR_FAIL the encoding is malformed and the signature must + * be rejected. + * + * @spec{Implements @[FIPS204, Algorithm 21, HintBitUnpack] (one row; part of + * @[FIPS204, Algorithm 27, sigDecode]).} + * + * @param[out] h Pointer to output polynomial h[i]. + * @param[in] sig Signature buffer. + * @param i Row index, must be < MLDSA_K. + * + * @retval 0 Hints were decoded successfully. + * @retval MLD_ERR_FAIL Hints are malformed. + */ +MLD_INTERNAL_API +MLD_MUST_CHECK_RETURN_VALUE +int mld_sig_unpack_hints(mld_poly *h, const uint8_t sig[MLDSA_CRYPTO_BYTES], + unsigned int i) +__contract__( + requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(h, sizeof(mld_poly))) + requires(i < MLDSA_K) + assigns(memory_slice(h, sizeof(mld_poly))) + ensures(return_value == 0 || return_value == MLD_ERR_FAIL) + ensures(return_value == 0 ==> array_bound(h->coeffs, 0, MLDSA_N, 0, 2)) +); +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#endif /* !MLD_PACKING_H */ diff --git a/cbits/mldsa/src/params.h b/cbits/mldsa/src/params.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/params.h @@ -0,0 +1,153 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLD_PARAMS_H +#define MLD_PARAMS_H + +#define MLDSA_SEEDBYTES 32 +#define MLDSA_CRHBYTES 64 +#define MLDSA_TRBYTES 64 +#define MLDSA_RNDBYTES 32 +#define MLDSA_N 256 +#define MLDSA_Q 8380417 +#define MLDSA_Q_HALF ((MLDSA_Q + 1) / 2) +#define MLDSA_D 13 + +#define MLDSA_GAMMA2_88 ((MLDSA_Q - 1) / 88) +#define MLDSA_GAMMA2_32 ((MLDSA_Q - 1) / 32) +#define MLDSA_POLYW1_PACKEDBYTES_88 192 +#define MLDSA_POLYW1_PACKEDBYTES_32 128 + +#if MLD_CONFIG_PARAMETER_SET == 44 + +#define MLDSA_K 4 +#define MLDSA_L 4 +#define MLDSA_ETA 2 +#define MLDSA_TAU 39 +#define MLDSA_BETA 78 +#define MLDSA_GAMMA1 ((int32_t)1 << 17) +#define MLDSA_GAMMA2 MLDSA_GAMMA2_88 +#define MLDSA_OMEGA 80 +#define MLDSA_CTILDEBYTES 32 +#define MLDSA_POLYZ_PACKEDBYTES 576 +#define MLDSA_POLYW1_PACKEDBYTES MLDSA_POLYW1_PACKEDBYTES_88 +#define MLDSA_POLYETA_PACKEDBYTES 96 + +#elif MLD_CONFIG_PARAMETER_SET == 65 + +#define MLDSA_K 6 +#define MLDSA_L 5 +#define MLDSA_ETA 4 +#define MLDSA_TAU 49 +#define MLDSA_BETA 196 +#define MLDSA_GAMMA1 ((int32_t)1 << 19) +#define MLDSA_GAMMA2 MLDSA_GAMMA2_32 +#define MLDSA_OMEGA 55 +#define MLDSA_CTILDEBYTES 48 +#define MLDSA_POLYZ_PACKEDBYTES 640 +#define MLDSA_POLYW1_PACKEDBYTES MLDSA_POLYW1_PACKEDBYTES_32 +#define MLDSA_POLYETA_PACKEDBYTES 128 + +#elif MLD_CONFIG_PARAMETER_SET == 87 + +#define MLDSA_K 8 +#define MLDSA_L 7 +#define MLDSA_ETA 2 +#define MLDSA_TAU 60 +#define MLDSA_BETA 120 +#define MLDSA_GAMMA1 ((int32_t)1 << 19) +#define MLDSA_GAMMA2 MLDSA_GAMMA2_32 +#define MLDSA_OMEGA 75 +#define MLDSA_CTILDEBYTES 64 +#define MLDSA_POLYZ_PACKEDBYTES 640 +#define MLDSA_POLYW1_PACKEDBYTES MLDSA_POLYW1_PACKEDBYTES_32 +#define MLDSA_POLYETA_PACKEDBYTES 96 + +#endif /* MLD_CONFIG_PARAMETER_SET == 87 */ + +#define MLDSA_POLYT1_PACKEDBYTES 320 +#define MLDSA_POLYT0_PACKEDBYTES 416 +#define MLDSA_POLYVECH_PACKEDBYTES (MLDSA_OMEGA + MLDSA_K) + +/* Sampling y from counter kappa uses nonces kappa, ..., kappa+L-1, which fit in + * uint16_t iff kappa <= UINT16_MAX - MLDSA_L. With kappa = attempt*MLDSA_L this + * bounds the number of signing attempts by MLD_MAX_KAPPA / MLDSA_L; see + * MLD_MAX_SIGNING_ATTEMPTS in sign.c. */ +#define MLD_MAX_KAPPA (UINT16_MAX - MLDSA_L) + +/* Layout of the packed public key pk[MLDSA_CRYPTO_PUBLICKEYBYTES] = (rho, t1): + * + * +-------------+--------------------------+ + * | rho | t1 | + * +-------------+--------------------------+ + * | SEEDBYTES | K * POLYT1_PACKEDBYTES | + * +-------------+--------------------------+ + */ +#define MLDSA_PK_RHO_OFFSET 0 +#define MLDSA_PK_RHO_BYTES MLDSA_SEEDBYTES + +#define MLDSA_PK_T1_OFFSET (MLDSA_PK_RHO_OFFSET + MLDSA_PK_RHO_BYTES) +#define MLDSA_PK_T1_BYTES (MLDSA_K * MLDSA_POLYT1_PACKEDBYTES) + +#define MLDSA_PK_END (MLDSA_PK_T1_OFFSET + MLDSA_PK_T1_BYTES) + +#define MLDSA_CRYPTO_PUBLICKEYBYTES MLDSA_PK_END + +/* Layout of the packed secret key + * sk[MLDSA_CRYPTO_SECRETKEYBYTES] = (rho, key, tr, s1, s2, t0): + * + * +-----------+-----------+-----------+-----------+-----------+-----------+ + * | rho | key | tr | s1 | s2 | t0 | + * +-----------+-----------+-----------+-----------+-----------+-----------+ + * | SEEDBYTES | SEEDBYTES | TRBYTES | L * | K * | K * | + * | | | | POLYETA_ | POLYETA_ | POLYT0_ | + * | | | | PACKED- | PACKED- | PACKED- | + * | | | | BYTES | BYTES | BYTES | + * +-----------+-----------+-----------+-----------+-----------+-----------+ + */ +#define MLDSA_SK_RHO_OFFSET 0 +#define MLDSA_SK_RHO_BYTES MLDSA_SEEDBYTES + +#define MLDSA_SK_KEY_OFFSET (MLDSA_SK_RHO_OFFSET + MLDSA_SK_RHO_BYTES) +#define MLDSA_SK_KEY_BYTES MLDSA_SEEDBYTES + +#define MLDSA_SK_TR_OFFSET (MLDSA_SK_KEY_OFFSET + MLDSA_SK_KEY_BYTES) +#define MLDSA_SK_TR_BYTES MLDSA_TRBYTES + +#define MLDSA_SK_S1_OFFSET (MLDSA_SK_TR_OFFSET + MLDSA_SK_TR_BYTES) +#define MLDSA_SK_S1_BYTES (MLDSA_L * MLDSA_POLYETA_PACKEDBYTES) + +#define MLDSA_SK_S2_OFFSET (MLDSA_SK_S1_OFFSET + MLDSA_SK_S1_BYTES) +#define MLDSA_SK_S2_BYTES (MLDSA_K * MLDSA_POLYETA_PACKEDBYTES) + +#define MLDSA_SK_T0_OFFSET (MLDSA_SK_S2_OFFSET + MLDSA_SK_S2_BYTES) +#define MLDSA_SK_T0_BYTES (MLDSA_K * MLDSA_POLYT0_PACKEDBYTES) + +#define MLDSA_SK_END (MLDSA_SK_T0_OFFSET + MLDSA_SK_T0_BYTES) + +#define MLDSA_CRYPTO_SECRETKEYBYTES MLDSA_SK_END + +/* Layout of the packed signature sig[MLDSA_CRYPTO_BYTES] = (c, z, h): + * + * +----------------+-------------------+----------------------+ + * | c (challenge) | z | h (hints) | + * +----------------+-------------------+----------------------+ + * | CTILDEBYTES | L * | POLYVECH_PACKEDBYTES | + * | | POLYZ_PACKEDBYTES | (= OMEGA + K) | + * +----------------+-------------------+----------------------+ + */ +#define MLDSA_SIG_C_OFFSET 0 +#define MLDSA_SIG_C_BYTES MLDSA_CTILDEBYTES + +#define MLDSA_SIG_Z_OFFSET (MLDSA_SIG_C_OFFSET + MLDSA_SIG_C_BYTES) +#define MLDSA_SIG_Z_BYTES (MLDSA_L * MLDSA_POLYZ_PACKEDBYTES) + +#define MLDSA_SIG_H_OFFSET (MLDSA_SIG_Z_OFFSET + MLDSA_SIG_Z_BYTES) +#define MLDSA_SIG_H_BYTES MLDSA_POLYVECH_PACKEDBYTES + +#define MLDSA_SIG_END (MLDSA_SIG_H_OFFSET + MLDSA_SIG_H_BYTES) + +#define MLDSA_CRYPTO_BYTES MLDSA_SIG_END + +#endif /* !MLD_PARAMS_H */ diff --git a/cbits/mldsa/src/poly.c b/cbits/mldsa/src/poly.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/poly.c @@ -0,0 +1,1066 @@ +/* + * Copyright (c) The mldsa-native project authors + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + * + * - [REF] + * CRYSTALS-Dilithium reference implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/ref + */ + +#include "poly.h" + +#include "common.h" +#include "ct.h" +#include "debug.h" +#include "reduce.h" +#include "rounding.h" +#include "symmetric.h" + +#if !defined(MLD_CONFIG_MULTILEVEL_NO_SHARED) +#include "zetas.inc" + +MLD_INTERNAL_API +void mld_poly_reduce(mld_poly *a) +{ + unsigned int i; + mld_assert_bound(a->coeffs, MLDSA_N, INT32_MIN, MLD_REDUCE32_DOMAIN_MAX); + + for (i = 0; i < MLDSA_N; ++i) + __loop__( + invariant(i <= MLDSA_N) + invariant(forall(k0, i, MLDSA_N, a->coeffs[k0] == loop_entry(*a).coeffs[k0])) + invariant(array_bound(a->coeffs, 0, i, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX)) + decreases(MLDSA_N - i)) + { + a->coeffs[i] = mld_reduce32(a->coeffs[i]); + } + + mld_assert_bound(a->coeffs, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, + MLD_REDUCE32_RANGE_MAX); +} + +MLD_STATIC_TESTABLE void mld_poly_caddq_c(mld_poly *a) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q)) + assigns(memory_slice(a, sizeof(mld_poly))) + ensures(array_bound(a->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) +) +{ + unsigned int i; + mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q); + + for (i = 0; i < MLDSA_N; ++i) + __loop__( + invariant(i <= MLDSA_N) + invariant(forall(k0, i, MLDSA_N, a->coeffs[k0] == loop_entry(*a).coeffs[k0])) + invariant(array_bound(a->coeffs, 0, i, 0, MLDSA_Q)) + decreases(MLDSA_N - i) + ) + { + a->coeffs[i] = mld_caddq(a->coeffs[i]); + } + + mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q); +} + +MLD_INTERNAL_API +void mld_poly_caddq(mld_poly *a) +{ +#if defined(MLD_USE_NATIVE_POLY_CADDQ) + int ret; + mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q); + ret = mld_poly_caddq_native(a->coeffs); + if (ret == MLD_NATIVE_FUNC_SUCCESS) + { + mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q); + return; + } +#endif /* MLD_USE_NATIVE_POLY_CADDQ */ + mld_poly_caddq_c(a); +} + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_SIGN_API) || \ + defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +/* Reference: We use destructive version (output=first input) to avoid + * reasoning about aliasing in the CBMC specification */ +MLD_INTERNAL_API +void mld_poly_add(mld_poly *r, const mld_poly *b) +{ + unsigned int i; + for (i = 0; i < MLDSA_N; ++i) + __loop__( + assigns(i, memory_slice(r, sizeof(mld_poly))) + invariant(i <= MLDSA_N) + invariant(forall(k0, i, MLDSA_N, r->coeffs[k0] == loop_entry(*r).coeffs[k0])) + invariant(forall(k1, 0, i, r->coeffs[k1] == loop_entry(*r).coeffs[k1] + b->coeffs[k1])) + invariant(forall(k2, 0, i, r->coeffs[k2] < MLD_REDUCE32_DOMAIN_MAX)) + invariant(forall(k2, 0, i, r->coeffs[k2] >= INT32_MIN)) + decreases(MLDSA_N - i) + ) + { + r->coeffs[i] = r->coeffs[i] + b->coeffs[i]; + } +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API || \ + MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +/* Reference: We use destructive version (output=first input) to avoid + * reasoning about aliasing in the CBMC specification */ +MLD_INTERNAL_API +void mld_poly_sub(mld_poly *r, const mld_poly *b) +{ + unsigned int i; + mld_assert_abs_bound(b->coeffs, MLDSA_N, MLDSA_Q); + mld_assert_abs_bound(r->coeffs, MLDSA_N, MLDSA_Q); + + for (i = 0; i < MLDSA_N; ++i) + __loop__( + invariant(i <= MLDSA_N) + invariant(array_bound(r->coeffs, 0, i, INT32_MIN, MLD_REDUCE32_DOMAIN_MAX)) + invariant(forall(k0, i, MLDSA_N, r->coeffs[k0] == loop_entry(*r).coeffs[k0])) + decreases(MLDSA_N - i) + ) + { + r->coeffs[i] = r->coeffs[i] - b->coeffs[i]; + } + + mld_assert_bound(r->coeffs, MLDSA_N, INT32_MIN, MLD_REDUCE32_DOMAIN_MAX); +} +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +MLD_INTERNAL_API +void mld_poly_shiftl(mld_poly *a) +{ + unsigned int i; + mld_assert_bound(a->coeffs, MLDSA_N, 0, 1 << 10); + + for (i = 0; i < MLDSA_N; i++) + __loop__( + invariant(i <= MLDSA_N) + invariant(array_bound(a->coeffs, 0, i, 0, MLDSA_Q)) + invariant(forall(k0, i, MLDSA_N, a->coeffs[k0] == loop_entry(*a).coeffs[k0])) + decreases(MLDSA_N - i)) + { + /* Reference: uses a left shift by MLDSA_D which is undefined behaviour in + * C90/C99 + */ + a->coeffs[i] *= (1 << MLDSA_D); + } + mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q); +} +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +static MLD_INLINE int32_t mld_fqmul(int32_t a, int32_t b) +__contract__( + requires(b > -MLDSA_Q_HALF && b < MLDSA_Q_HALF) + ensures(return_value > -MLD_FQMUL_BOUND && return_value < MLD_FQMUL_BOUND) +) +{ + /* Bounds: We argue in mld_montgomery_reduce() that the result + * of Montgomery reduction is < MLDSA_Q if the input is smaller + * than 2^31 * MLDSA_Q in absolute value. Indeed, we have: + * + * |a * b| = |a| * |b| + * < 2^31 * MLDSA_Q_HALF + * < 2^31 * MLDSA_Q + * + * So the output is < MLDSA_Q < MLD_FQMUL_BOUND. + */ + return mld_montgomery_reduce((int64_t)a * (int64_t)b); +} + +/* mld_ntt_butterfly_block() + * + * Computes a block CT butterflies with a fixed twiddle factor, + * using Montgomery multiplication. + * + * Parameters: + * - r: Pointer to base of polynomial (_not_ the base of butterfly block) + * - zeta: Twiddle factor to use for the butterfly. This must be in + * Montgomery form and signed canonical. + * - start: Offset to the beginning of the butterfly block + * - len: Index difference between coefficients subject to a butterfly + * - bound: Ghost variable describing coefficient bound: Prior to `start`, + * coefficients must be bound by `bound + MLDSA_Q`. Post `start`, + * they must be bound by `bound`. + * When this function returns, output coefficients in the index range + * [start, start+2*len) have bound bumped to `bound + MLDSA_Q`. + * Example: + * - start=8, len=4 + * This would compute the following four butterflies + * 8 -- 12 + * 9 -- 13 + * 10 -- 14 + * 11 -- 15 + * - start=4, len=2 + * This would compute the following two butterflies + * 4 -- 6 + * 5 -- 7 + */ + +/* Reference: Embedded in `ntt()` in the reference implementation @[REF]. */ +static MLD_INLINE void mld_ntt_butterfly_block(int32_t r[MLDSA_N], + const int32_t zeta, + const unsigned start, + const unsigned len, + const uint32_t bound) +__contract__( + requires(start < MLDSA_N) + requires(1 <= len && len <= MLDSA_N / 2 && start + 2 * len <= MLDSA_N) + requires(0 <= bound && bound < INT32_MAX - MLD_FQMUL_BOUND) + requires(-MLDSA_Q_HALF < zeta && zeta < MLDSA_Q_HALF) + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(array_abs_bound(r, 0, start, bound + MLD_FQMUL_BOUND)) + requires(array_abs_bound(r, start, MLDSA_N, bound)) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(array_abs_bound(r, 0, start + 2*len, bound + MLD_FQMUL_BOUND)) + ensures(array_abs_bound(r, start + 2 * len, MLDSA_N, bound))) +{ + /* `bound` is a ghost variable only needed in the CBMC specification */ + unsigned j; + ((void)bound); + for (j = start; j < start + len; j++) + __loop__( + invariant(start <= j && j <= start + len) + /* + * Coefficients are updated in strided pairs, so the bounds for the + * intermediate states alternate twice between the old and new bound + */ + invariant(array_abs_bound(r, 0, j, bound + MLD_FQMUL_BOUND)) + invariant(array_abs_bound(r, j, start + len, bound)) + invariant(array_abs_bound(r, start + len, j + len, bound + MLD_FQMUL_BOUND)) + invariant(array_abs_bound(r, j + len, MLDSA_N, bound)) + decreases(start + len - j)) + { + int32_t t; + t = mld_fqmul(r[j + len], zeta); + r[j + len] = r[j] - t; + r[j] = r[j] + t; + } +} + +/* mld_ntt_layer() + * + * Compute one layer of forward NTT + * + * Parameters: + * - r: Pointer to base of polynomial + * - layer: Indicates which layer is being applied. + */ + +/* Reference: Embedded in `ntt()` in the reference implementation @[REF]. */ +static MLD_INLINE void mld_ntt_layer(int32_t r[MLDSA_N], const unsigned layer) +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(1 <= layer && layer <= 8) + requires(array_abs_bound(r, 0, MLDSA_N, layer * MLD_FQMUL_BOUND)) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(array_abs_bound(r, 0, MLDSA_N, (layer + 1) * MLD_FQMUL_BOUND))) +{ + unsigned start, k, len; + /* Twiddle factors for layer n are at indices 2^(n-1)..2^n-1. */ + k = 1u << (layer - 1); + len = (unsigned)MLDSA_N >> layer; + for (start = 0; start < MLDSA_N; start += 2 * len) + __loop__( + invariant(start < MLDSA_N + 2 * len) + invariant(k <= MLDSA_N) + invariant(2 * len * k == start + MLDSA_N) + invariant(array_abs_bound(r, 0, start, layer * MLD_FQMUL_BOUND + MLD_FQMUL_BOUND)) + invariant(array_abs_bound(r, start, MLDSA_N, layer * MLD_FQMUL_BOUND)) + decreases(MLDSA_N - start)) + { + int32_t zeta = mld_zetas[k++]; + mld_ntt_butterfly_block(r, zeta, start, len, layer * MLD_FQMUL_BOUND); + } +} + +MLD_STATIC_TESTABLE void mld_poly_ntt_c(mld_poly *a) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q)) + assigns(memory_slice(a, sizeof(mld_poly))) + ensures(array_abs_bound(a->coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) +) +{ + unsigned int layer; + int32_t *r; + + + mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q); + r = a->coeffs; + + for (layer = 1; layer < 9; layer++) + __loop__( + invariant(1 <= layer && layer <= 9) + invariant(array_abs_bound(r, 0, MLDSA_N, layer * MLD_FQMUL_BOUND)) + decreases(9 - layer) + ) + { + mld_ntt_layer(r, layer); + } + + mld_assert_abs_bound(a->coeffs, MLDSA_N, MLD_NTT_BOUND); +} + +MLD_INTERNAL_API +void mld_poly_ntt(mld_poly *a) +{ +#if defined(MLD_USE_NATIVE_NTT) + int ret; + mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q); + ret = mld_ntt_native(a->coeffs); + if (ret == MLD_NATIVE_FUNC_SUCCESS) + { + mld_assert_abs_bound(a->coeffs, MLDSA_N, MLD_NTT_BOUND); + return; + } +#endif /* MLD_USE_NATIVE_NTT */ + mld_poly_ntt_c(a); +} + +/** + * Scale a field element by mont/256, i.e., perform Montgomery multiplication + * by mont^2/256. + * + * Input is expected to have absolute value smaller than 256 * MLDSA_Q. Output + * has absolute value smaller than MLD_INTT_BOUND. + * + * @param a Field element to be scaled. + */ +static MLD_INLINE int32_t mld_fqscale(int32_t a) +__contract__( + requires(a > -256*MLDSA_Q && a < 256*MLDSA_Q) + ensures(return_value > -MLD_INTT_BOUND && return_value < MLD_INTT_BOUND) +) +{ + /* check-magic: 41978 == pow(2,64-8,MLDSA_Q) */ + const int32_t f = 41978; + /* Bounds: MLD_INTT_BOUND is MLDSA_Q, so the bounds reasoning is just + * a special case of that in mld_fqmul(). */ + return mld_montgomery_reduce((int64_t)a * f); +} + +/* Reference: Embedded into `invntt_tomont()` in the reference implementation + * @[REF] */ +static MLD_INLINE void mld_invntt_layer(int32_t r[MLDSA_N], unsigned layer) +__contract__( + requires(memory_no_alias(r, sizeof(int32_t) * MLDSA_N)) + requires(1 <= layer && layer <= 8) + requires(array_abs_bound(r, 0, MLDSA_N, (MLDSA_N >> layer) * MLDSA_Q)) + assigns(memory_slice(r, sizeof(int32_t) * MLDSA_N)) + ensures(array_abs_bound(r, 0, MLDSA_N, (MLDSA_N >> (layer - 1)) * MLDSA_Q))) +{ + unsigned start, k, len; + len = (unsigned)MLDSA_N >> layer; + k = (1u << layer) - 1; + for (start = 0; start < MLDSA_N; start += 2 * len) + __loop__( + invariant(start <= MLDSA_N && k <= 255) + invariant(2 * len * k + start == 2 * MLDSA_N - 2 * len) + invariant(array_abs_bound(r, 0, start, (MLDSA_N >> (layer - 1)) * MLDSA_Q)) + invariant(array_abs_bound(r, start, MLDSA_N, (MLDSA_N >> layer) * MLDSA_Q)) + decreases(MLDSA_N - start)) + { + unsigned j; + int32_t zeta = -mld_zetas[k--]; + + /* The bound `(MLDSA_N >> (layer - 1)) * MLDSA_Q` is loose enough to + * cover both the input bound `(MLDSA_N >> layer) * MLDSA_Q` + * (for layers >= 1) and the fqmul output bound `MLD_FQMUL_BOUND` + * (which is < 2 * MLDSA_Q <= (MLDSA_N >> (layer - 1)) * MLDSA_Q). */ + for (j = start; j < start + len; j++) + __loop__( + invariant(start <= j && j <= start + len) + invariant(array_abs_bound(r, 0, start, (MLDSA_N >> (layer - 1)) * MLDSA_Q)) + invariant(array_abs_bound(r, start, j, (MLDSA_N >> (layer - 1)) * MLDSA_Q)) + invariant(array_abs_bound(r, j, start + len, (MLDSA_N >> layer) * MLDSA_Q)) + invariant(array_abs_bound(r, start + len, j + len, (MLDSA_N >> (layer - 1)) * MLDSA_Q)) + invariant(array_abs_bound(r, j + len, MLDSA_N, (MLDSA_N >> layer) * MLDSA_Q)) + decreases(start + len - j)) + { + int32_t t = r[j]; + r[j] = t + r[j + len]; + r[j + len] = t - r[j + len]; + r[j + len] = mld_fqmul(r[j + len], zeta); + } + } +} + +MLD_STATIC_TESTABLE void mld_poly_invntt_tomont_c(mld_poly *a) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q)) + assigns(memory_slice(a, sizeof(mld_poly))) + ensures(array_abs_bound(a->coeffs, 0, MLDSA_N, MLD_INTT_BOUND)) +) +{ + unsigned int layer, j; + int32_t *r; + + mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q); + + r = a->coeffs; + for (layer = 8; layer >= 1; layer--) + __loop__( + invariant(layer <= 8) + /* Absolute bounds increase from 1Q before layer 8 */ + /* up to 256Q after layer 1 */ + invariant(array_abs_bound(r, 0, MLDSA_N, (MLDSA_N >> layer) * MLDSA_Q)) + decreases(layer)) + { + mld_invntt_layer(r, layer); + } + + /* Coefficient bounds are now at 256Q. We now scale by mont / 256, + * i.e., compute the Montgomery multiplication by mont^2 / 256. + * mont corrects the mont^-1 factor introduced in the basemul. + * 1/256 performs that scaling of the inverse NTT. + * The reduced value is bounded by MLD_INTT_BOUND in absolute + * value.*/ + for (j = 0; j < MLDSA_N; ++j) + __loop__( + invariant(j <= MLDSA_N) + invariant(array_abs_bound(r, 0, j, MLD_INTT_BOUND)) + invariant(array_abs_bound(r, j, MLDSA_N, MLDSA_N * MLDSA_Q)) + decreases(MLDSA_N - j) + ) + { + r[j] = mld_fqscale(r[j]); + } + + mld_assert_abs_bound(a->coeffs, MLDSA_N, MLD_INTT_BOUND); +} + + +MLD_INTERNAL_API +void mld_poly_invntt_tomont(mld_poly *a) +{ +#if defined(MLD_USE_NATIVE_INTT) + int ret; + mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q); + ret = mld_intt_native(a->coeffs); + if (ret == MLD_NATIVE_FUNC_SUCCESS) + { + mld_assert_abs_bound(a->coeffs, MLDSA_N, MLD_INTT_BOUND); + return; + } +#endif /* MLD_USE_NATIVE_INTT */ + mld_poly_invntt_tomont_c(a); +} + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \ + defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +MLD_STATIC_TESTABLE void mld_poly_pointwise_montgomery_c(mld_poly *a, + const mld_poly *b) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(memory_no_alias(b, sizeof(mld_poly))) + requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) + requires(array_abs_bound(b->coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) + assigns(memory_slice(a, sizeof(mld_poly))) + ensures(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q)) +) +{ + unsigned int i; + mld_assert_abs_bound(a->coeffs, MLDSA_N, MLD_NTT_BOUND); + mld_assert_abs_bound(b->coeffs, MLDSA_N, MLD_NTT_BOUND); + + for (i = 0; i < MLDSA_N; ++i) + __loop__( + invariant(i <= MLDSA_N) + invariant(array_abs_bound(a->coeffs, 0, i, MLDSA_Q)) + invariant(array_abs_bound(a->coeffs, i, MLDSA_N, MLD_NTT_BOUND)) + decreases(MLDSA_N - i) + ) + { + a->coeffs[i] = mld_montgomery_reduce((int64_t)a->coeffs[i] * b->coeffs[i]); + } + mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q); +} + +MLD_INTERNAL_API +void mld_poly_pointwise_montgomery(mld_poly *a, const mld_poly *b) +{ +#if defined(MLD_USE_NATIVE_POINTWISE_MONTGOMERY) + int ret; + mld_assert_abs_bound(a->coeffs, MLDSA_N, MLD_NTT_BOUND); + mld_assert_abs_bound(b->coeffs, MLDSA_N, MLD_NTT_BOUND); + ret = mld_poly_pointwise_montgomery_native(a->coeffs, b->coeffs); + if (ret == MLD_NATIVE_FUNC_SUCCESS) + { + mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_Q); + return; + } +#endif /* MLD_USE_NATIVE_POINTWISE_MONTGOMERY */ + mld_poly_pointwise_montgomery_c(a, b); +} +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API || \ + MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +MLD_INTERNAL_API +void mld_poly_power2round(mld_poly *a1, mld_poly *a0, const mld_poly *a) +{ + unsigned int i; + mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q); + + for (i = 0; i < MLDSA_N; ++i) + __loop__( + assigns(i, memory_slice(a0, sizeof(mld_poly)), memory_slice(a1, sizeof(mld_poly))) + invariant(i <= MLDSA_N) + invariant(forall(k0, i, MLDSA_N, a->coeffs[k0] == loop_entry(*a).coeffs[k0])) + invariant(array_bound(a0->coeffs, 0, i, -(MLD_2_POW_D/2)+1, (MLD_2_POW_D/2)+1)) + invariant(array_bound(a1->coeffs, 0, i, 0, ((MLDSA_Q - 1) / MLD_2_POW_D) + 1)) + decreases(MLDSA_N - i) + ) + { + mld_power2round(&a0->coeffs[i], &a1->coeffs[i], a->coeffs[i]); + } + + mld_assert_bound(a0->coeffs, MLDSA_N, -(MLD_2_POW_D / 2) + 1, + (MLD_2_POW_D / 2) + 1); + mld_assert_bound(a1->coeffs, MLDSA_N, 0, ((MLDSA_Q - 1) / MLD_2_POW_D) + 1); +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#ifndef MLD_POLY_UNIFORM_NBLOCKS +#define MLD_POLY_UNIFORM_NBLOCKS \ + ((768 + MLD_STREAM128_BLOCKBYTES - 1) / MLD_STREAM128_BLOCKBYTES) +#endif +/* Reference: `mld_rej_uniform()` in the reference implementation @[REF]. + * - Our signature differs from the reference implementation + * in that it adds the offset and always expects the base of the + * target buffer. This avoids shifting the buffer base in the + * caller, which appears tricky to reason about. */ +MLD_STATIC_TESTABLE unsigned int mld_rej_uniform_c(int32_t *a, + unsigned int target, + unsigned int offset, + const uint8_t *buf, + unsigned int buflen) +__contract__( + requires(offset <= target && target <= MLDSA_N) + requires(buflen <= (MLD_POLY_UNIFORM_NBLOCKS * MLD_STREAM128_BLOCKBYTES) && buflen % 3 == 0) + requires(memory_no_alias(a, sizeof(int32_t) * target)) + requires(memory_no_alias(buf, buflen)) + requires(array_bound(a, 0, offset, 0, MLDSA_Q)) + assigns(memory_slice(a, sizeof(int32_t) * target)) + ensures(offset <= return_value && return_value <= target) + ensures(array_bound(a, 0, return_value, 0, MLDSA_Q)) +) +{ + unsigned int ctr, pos; + uint32_t t; + mld_assert_bound(a, offset, 0, MLDSA_Q); + + ctr = offset; + pos = 0; + /* pos + 3 cannot overflow due to the assumption + buflen <= (MLD_POLY_UNIFORM_NBLOCKS * MLD_STREAM128_BLOCKBYTES) */ + while (ctr < target && pos + 3 <= buflen) + __loop__( + invariant(offset <= ctr && ctr <= target && pos <= buflen) + invariant(array_bound(a, 0, ctr, 0, MLDSA_Q)) + decreases(buflen - pos)) + { + t = buf[pos++]; + t |= (uint32_t)buf[pos++] << 8; + t |= (uint32_t)buf[pos++] << 16; + t &= 0x7FFFFF; + + if (t < MLDSA_Q) + { + a[ctr++] = (int32_t)t; + } + } + + mld_assert_bound(a, ctr, 0, MLDSA_Q); + + return ctr; +} +/** + * Sample uniformly random coefficients in [0, MLDSA_Q-1] by performing + * rejection sampling on an array of random bytes. + * + * @param[out] a Pointer to output array (allocated). + * @param target Requested number of coefficients to sample. + * @param offset Number of coefficients already sampled. + * @param[in] buf Array of random bytes to sample from. + * @param buflen Length of array of random bytes (must be multiple of 3). + * + * @return Number of sampled coefficients. Can be smaller than len if not + * enough random bytes were given. + */ + +/* Reference: `mld_rej_uniform()` in the reference implementation @[REF]. + * - Our signature differs from the reference implementation + * in that it adds the offset and always expects the base of the + * target buffer. This avoids shifting the buffer base in the + * caller, which appears tricky to reason about. */ +static unsigned int mld_rej_uniform(int32_t *a, unsigned int target, + unsigned int offset, const uint8_t *buf, + unsigned int buflen) +__contract__( + requires(offset <= target && target <= MLDSA_N) + requires(buflen <= (MLD_POLY_UNIFORM_NBLOCKS * MLD_STREAM128_BLOCKBYTES) && buflen % 3 == 0) + requires(memory_no_alias(a, sizeof(int32_t) * target)) + requires(memory_no_alias(buf, buflen)) + requires(array_bound(a, 0, offset, 0, MLDSA_Q)) + assigns(memory_slice(a, sizeof(int32_t) * target)) + ensures(offset <= return_value && return_value <= target) + ensures(array_bound(a, 0, return_value, 0, MLDSA_Q)) +) +{ +#if defined(MLD_USE_NATIVE_REJ_UNIFORM) + int ret; + mld_assert_bound(a, offset, 0, MLDSA_Q); + if (offset == 0) + { + ret = mld_rej_uniform_native(a, target, buf, buflen); + if (ret != MLD_NATIVE_FUNC_FALLBACK) + { + unsigned res = (unsigned)ret; + mld_assert_bound(a, res, 0, MLDSA_Q); + return res; + } + } +#endif /* MLD_USE_NATIVE_REJ_UNIFORM */ + + return mld_rej_uniform_c(a, target, offset, buf, buflen); +} + +/* Reference: poly_uniform() in the reference implementation @[REF]. + * - Simplified from reference by removing buffer tail handling + * since buflen % 3 = 0 always holds true (MLD_STREAM128_BLOCKBYTES + * = 168). + * - Modified rej_uniform interface to track offset directly. + * - Pass nonce packed in the extended seed array instead of a third + * argument. + * */ +MLD_INTERNAL_API +void mld_poly_uniform(mld_poly *a, const uint8_t seed[MLDSA_SEEDBYTES + 2]) +{ + unsigned int ctr; + unsigned int buflen = MLD_POLY_UNIFORM_NBLOCKS * MLD_STREAM128_BLOCKBYTES; + MLD_ALIGN uint8_t buf[MLD_POLY_UNIFORM_NBLOCKS * MLD_STREAM128_BLOCKBYTES]; + mld_xof128_ctx state; + + mld_xof128_init(&state); + mld_xof128_absorb_once(&state, seed, MLDSA_SEEDBYTES + 2); + mld_xof128_squeezeblocks(buf, MLD_POLY_UNIFORM_NBLOCKS, &state); + + ctr = mld_rej_uniform(a->coeffs, MLDSA_N, 0, buf, buflen); + buflen = MLD_STREAM128_BLOCKBYTES; + while (ctr < MLDSA_N) + __loop__( + assigns(ctr, state, memory_slice(a, sizeof(mld_poly)), object_whole(buf)) + invariant(ctr <= MLDSA_N) + invariant(array_bound(a->coeffs, 0, ctr, 0, MLDSA_Q)) + invariant(state.pos <= SHAKE128_RATE) + ) + { + mld_xof128_squeezeblocks(buf, 1, &state); + ctr = mld_rej_uniform(a->coeffs, MLDSA_N, ctr, buf, buflen); + } + mld_xof128_release(&state); + mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q); + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(buf, sizeof(buf)); +} + +#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)) +MLD_INTERNAL_API +void mld_poly_uniform_4x(mld_poly *vec0, mld_poly *vec1, mld_poly *vec2, + mld_poly *vec3, + uint8_t seed[4][MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2)]) +{ + /* Temporary buffers for XOF output before rejection sampling */ + MLD_ALIGN uint8_t + buf[4][MLD_ALIGN_UP(MLD_POLY_UNIFORM_NBLOCKS * MLD_STREAM128_BLOCKBYTES)]; + + /* Tracks the number of coefficients we have already sampled */ + unsigned ctr[4]; + mld_xof128_x4_ctx state; + unsigned buflen; + + mld_xof128_x4_init(&state); + mld_xof128_x4_absorb(&state, seed, MLDSA_SEEDBYTES + 2); + + /* + * Initially, squeeze heuristic number of MLD_POLY_UNIFORM_NBLOCKS. + * This should generate the matrix entries with high probability. + */ + + mld_xof128_x4_squeezeblocks(buf, MLD_POLY_UNIFORM_NBLOCKS, &state); + buflen = MLD_POLY_UNIFORM_NBLOCKS * MLD_STREAM128_BLOCKBYTES; + ctr[0] = mld_rej_uniform(vec0->coeffs, MLDSA_N, 0, buf[0], buflen); + ctr[1] = mld_rej_uniform(vec1->coeffs, MLDSA_N, 0, buf[1], buflen); + ctr[2] = mld_rej_uniform(vec2->coeffs, MLDSA_N, 0, buf[2], buflen); + ctr[3] = mld_rej_uniform(vec3->coeffs, MLDSA_N, 0, buf[3], buflen); + + /* + * So long as not all matrix entries have been generated, squeeze + * one more block a time until we're done. + */ + buflen = MLD_STREAM128_BLOCKBYTES; + while (ctr[0] < MLDSA_N || ctr[1] < MLDSA_N || ctr[2] < MLDSA_N || + ctr[3] < MLDSA_N) + __loop__( + assigns(ctr, state, object_whole(buf), + memory_slice(vec0, sizeof(mld_poly)), memory_slice(vec1, sizeof(mld_poly)), + memory_slice(vec2, sizeof(mld_poly)), memory_slice(vec3, sizeof(mld_poly))) + invariant(ctr[0] <= MLDSA_N && ctr[1] <= MLDSA_N) + invariant(ctr[2] <= MLDSA_N && ctr[3] <= MLDSA_N) + invariant(array_bound(vec0->coeffs, 0, ctr[0], 0, MLDSA_Q)) + invariant(array_bound(vec1->coeffs, 0, ctr[1], 0, MLDSA_Q)) + invariant(array_bound(vec2->coeffs, 0, ctr[2], 0, MLDSA_Q)) + invariant(array_bound(vec3->coeffs, 0, ctr[3], 0, MLDSA_Q))) + { + mld_xof128_x4_squeezeblocks(buf, 1, &state); + ctr[0] = mld_rej_uniform(vec0->coeffs, MLDSA_N, ctr[0], buf[0], buflen); + ctr[1] = mld_rej_uniform(vec1->coeffs, MLDSA_N, ctr[1], buf[1], buflen); + ctr[2] = mld_rej_uniform(vec2->coeffs, MLDSA_N, ctr[2], buf[2], buflen); + ctr[3] = mld_rej_uniform(vec3->coeffs, MLDSA_N, ctr[3], buf[3], buflen); + } + mld_xof128_x4_release(&state); + + mld_assert_bound(vec0->coeffs, MLDSA_N, 0, MLDSA_Q); + mld_assert_bound(vec1->coeffs, MLDSA_N, 0, MLDSA_Q); + mld_assert_bound(vec2->coeffs, MLDSA_N, 0, MLDSA_Q); + mld_assert_bound(vec3->coeffs, MLDSA_N, 0, MLDSA_Q); + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(buf, sizeof(buf)); +} + +#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY && (!MLD_CONFIG_REDUCE_RAM || \ + MLD_UNIT_TEST) */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +MLD_INTERNAL_API +void mld_polyt1_pack(uint8_t r[MLDSA_POLYT1_PACKEDBYTES], const mld_poly *a) +{ + unsigned int i; + mld_assert_bound(a->coeffs, MLDSA_N, 0, 1 << 10); + + for (i = 0; i < MLDSA_N / 4; ++i) + __loop__( + invariant(i <= MLDSA_N/4) + decreases(MLDSA_N / 4 - i)) + { + r[5 * i + 0] = (uint8_t)((a->coeffs[4 * i + 0] >> 0) & 0xFF); + r[5 * i + 1] = + (uint8_t)(((a->coeffs[4 * i + 0] >> 8) | (a->coeffs[4 * i + 1] << 2)) & + 0xFF); + r[5 * i + 2] = + (uint8_t)(((a->coeffs[4 * i + 1] >> 6) | (a->coeffs[4 * i + 2] << 4)) & + 0xFF); + r[5 * i + 3] = + (uint8_t)(((a->coeffs[4 * i + 2] >> 4) | (a->coeffs[4 * i + 3] << 6)) & + 0xFF); + r[5 * i + 4] = (uint8_t)((a->coeffs[4 * i + 3] >> 2) & 0xFF); + } +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +MLD_INTERNAL_API +void mld_polyt1_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYT1_PACKEDBYTES]) +{ + unsigned int i; + + for (i = 0; i < MLDSA_N / 4; ++i) + __loop__( + invariant(i <= MLDSA_N/4) + invariant(array_bound(r->coeffs, 0, i*4, 0, 1 << 10)) + decreases(MLDSA_N / 4 - i)) + { + r->coeffs[4 * i + 0] = + ((a[5 * i + 0] >> 0) | ((int32_t)a[5 * i + 1] << 8)) & 0x3FF; + r->coeffs[4 * i + 1] = + ((a[5 * i + 1] >> 2) | ((int32_t)a[5 * i + 2] << 6)) & 0x3FF; + r->coeffs[4 * i + 2] = + ((a[5 * i + 2] >> 4) | ((int32_t)a[5 * i + 3] << 4)) & 0x3FF; + r->coeffs[4 * i + 3] = + ((a[5 * i + 3] >> 6) | ((int32_t)a[5 * i + 4] << 2)) & 0x3FF; + } + + mld_assert_bound(r->coeffs, MLDSA_N, 0, 1 << 10); +} +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +MLD_INTERNAL_API +void mld_polyt0_pack(uint8_t r[MLDSA_POLYT0_PACKEDBYTES], const mld_poly *a) +{ + unsigned int i; + uint32_t t[8]; + + mld_assert_bound(a->coeffs, MLDSA_N, -(1 << (MLDSA_D - 1)) + 1, + (1 << (MLDSA_D - 1)) + 1); + + for (i = 0; i < MLDSA_N / 8; ++i) + __loop__( + invariant(i <= MLDSA_N/8) + decreases(MLDSA_N / 8 - i)) + { + /* Safety: a->coeffs[i] <= (1 << (MLDSA_D - 1) as they are output of + * power2round, hence, these casts are safe. */ + t[0] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 0]); + t[1] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 1]); + t[2] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 2]); + t[3] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 3]); + t[4] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 4]); + t[5] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 5]); + t[6] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 6]); + t[7] = (uint32_t)((1 << (MLDSA_D - 1)) - a->coeffs[8 * i + 7]); + + r[13 * i + 0] = (uint8_t)((t[0]) & 0xFF); + r[13 * i + 1] = (uint8_t)((t[0] >> 8) & 0xFF); + r[13 * i + 1] |= (uint8_t)((t[1] << 5) & 0xFF); + r[13 * i + 2] = (uint8_t)((t[1] >> 3) & 0xFF); + r[13 * i + 3] = (uint8_t)((t[1] >> 11) & 0xFF); + r[13 * i + 3] |= (uint8_t)((t[2] << 2) & 0xFF); + r[13 * i + 4] = (uint8_t)((t[2] >> 6) & 0xFF); + r[13 * i + 4] |= (uint8_t)((t[3] << 7) & 0xFF); + r[13 * i + 5] = (uint8_t)((t[3] >> 1) & 0xFF); + r[13 * i + 6] = (uint8_t)((t[3] >> 9) & 0xFF); + r[13 * i + 6] |= (uint8_t)((t[4] << 4) & 0xFF); + r[13 * i + 7] = (uint8_t)((t[4] >> 4) & 0xFF); + r[13 * i + 8] = (uint8_t)((t[4] >> 12) & 0xFF); + r[13 * i + 8] |= (uint8_t)((t[5] << 1) & 0xFF); + r[13 * i + 9] = (uint8_t)((t[5] >> 7) & 0xFF); + r[13 * i + 9] |= (uint8_t)((t[6] << 6) & 0xFF); + r[13 * i + 10] = (uint8_t)((t[6] >> 2) & 0xFF); + r[13 * i + 11] = (uint8_t)((t[6] >> 10) & 0xFF); + r[13 * i + 11] |= (uint8_t)((t[7] << 3) & 0xFF); + r[13 * i + 12] = (uint8_t)((t[7] >> 5) & 0xFF); + } +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST) +MLD_INTERNAL_API +void mld_polyt0_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYT0_PACKEDBYTES]) +{ + unsigned int i; + + for (i = 0; i < MLDSA_N / 8; ++i) + __loop__( + invariant(i <= MLDSA_N/8) + invariant(array_bound(r->coeffs, 0, i*8, -(1<<(MLDSA_D-1)) + 1, (1<<(MLDSA_D-1)) + 1)) + decreases(MLDSA_N / 8 - i)) + { + r->coeffs[8 * i + 0] = a[13 * i + 0]; + r->coeffs[8 * i + 0] |= (int32_t)a[13 * i + 1] << 8; + r->coeffs[8 * i + 0] &= 0x1FFF; + + r->coeffs[8 * i + 1] = a[13 * i + 1] >> 5; + r->coeffs[8 * i + 1] |= (int32_t)a[13 * i + 2] << 3; + r->coeffs[8 * i + 1] |= (int32_t)a[13 * i + 3] << 11; + r->coeffs[8 * i + 1] &= 0x1FFF; + + r->coeffs[8 * i + 2] = a[13 * i + 3] >> 2; + r->coeffs[8 * i + 2] |= (int32_t)a[13 * i + 4] << 6; + r->coeffs[8 * i + 2] &= 0x1FFF; + + r->coeffs[8 * i + 3] = a[13 * i + 4] >> 7; + r->coeffs[8 * i + 3] |= (int32_t)a[13 * i + 5] << 1; + r->coeffs[8 * i + 3] |= (int32_t)a[13 * i + 6] << 9; + r->coeffs[8 * i + 3] &= 0x1FFF; + + r->coeffs[8 * i + 4] = a[13 * i + 6] >> 4; + r->coeffs[8 * i + 4] |= (int32_t)a[13 * i + 7] << 4; + r->coeffs[8 * i + 4] |= (int32_t)a[13 * i + 8] << 12; + r->coeffs[8 * i + 4] &= 0x1FFF; + + r->coeffs[8 * i + 5] = a[13 * i + 8] >> 1; + r->coeffs[8 * i + 5] |= (int32_t)a[13 * i + 9] << 7; + r->coeffs[8 * i + 5] &= 0x1FFF; + + r->coeffs[8 * i + 6] = a[13 * i + 9] >> 6; + r->coeffs[8 * i + 6] |= (int32_t)a[13 * i + 10] << 2; + r->coeffs[8 * i + 6] |= (int32_t)a[13 * i + 11] << 10; + r->coeffs[8 * i + 6] &= 0x1FFF; + + r->coeffs[8 * i + 7] = a[13 * i + 11] >> 3; + r->coeffs[8 * i + 7] |= (int32_t)a[13 * i + 12] << 5; + r->coeffs[8 * i + 7] &= 0x1FFF; + + r->coeffs[8 * i + 0] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 0]; + r->coeffs[8 * i + 1] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 1]; + r->coeffs[8 * i + 2] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 2]; + r->coeffs[8 * i + 3] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 3]; + r->coeffs[8 * i + 4] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 4]; + r->coeffs[8 * i + 5] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 5]; + r->coeffs[8 * i + 6] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 6]; + r->coeffs[8 * i + 7] = (1 << (MLDSA_D - 1)) - r->coeffs[8 * i + 7]; + } + + mld_assert_bound(r->coeffs, MLDSA_N, -(1 << (MLDSA_D - 1)) + 1, + (1 << (MLDSA_D - 1)) + 1); +} +#endif /* !MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST */ + +MLD_STATIC_TESTABLE uint32_t mld_poly_chknorm_c(const mld_poly *a, int32_t B) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(0 <= B && B <= MLDSA_Q - MLD_REDUCE32_RANGE_MAX) + requires(array_bound(a->coeffs, 0, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX)) + ensures(return_value == 0 || return_value == 0xFFFFFFFF) + ensures((return_value == 0) == array_abs_bound(a->coeffs, 0, MLDSA_N, B)) +) +{ + unsigned int i; + uint32_t t = 0; + mld_assert_bound(a->coeffs, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, + MLD_REDUCE32_RANGE_MAX); + for (i = 0; i < MLDSA_N; ++i) + __loop__( + invariant(i <= MLDSA_N) + invariant(t == 0 || t == 0xFFFFFFFF) + invariant((t == 0) == array_abs_bound(a->coeffs, 0, i, B)) + decreases(MLDSA_N - i) + ) + { + /* + * Since we know that -MLD_REDUCE32_RANGE_MAX <= a < MLD_REDUCE32_RANGE_MAX, + * and B <= MLDSA_Q - MLD_REDUCE32_RANGE_MAX, to check if + * -B < (a mod± MLDSA_Q) < B, it suffices to check if -B < a < B. + * + * We prove this to be true using the following CBMC assertions. + * a ==> b expressed as !a || b to also allow run-time assertion. + */ + mld_assert(a->coeffs[i] < B || a->coeffs[i] - MLDSA_Q <= -B); + mld_assert(a->coeffs[i] > -B || a->coeffs[i] + MLDSA_Q >= B); + + /* Reference: Leaks which coefficient violates the bound via a conditional. + * We are more conservative to reduce the number of declassifications in + * constant-time testing. + */ + + /* if (abs(a[i]) >= B) */ + t |= mld_ct_cmask_neg_i32(B - 1 - mld_ct_abs_i32(a->coeffs[i])); + } + + return t; +} + +/* Reference: explicitly checks the bound B to be <= (MLDSA_Q - 1) / 8). + * This is unnecessary as it's always a compile-time constant. + * We instead model it as a precondition. + * Checking the bound is performed using a conditional arguing + * that it is okay to leak which coefficient violates the bound (while the + * coefficient itself must remain secret). + * We instead perform everything in constant-time. + * Also it is sufficient to check that it is smaller than + * MLDSA_Q - MLD_REDUCE32_RANGE_MAX > (MLDSA_Q - 1) / 8). + */ +MLD_INTERNAL_API +uint32_t mld_poly_chknorm(const mld_poly *a, int32_t B) +{ +#if defined(MLD_USE_NATIVE_POLY_CHKNORM) + int ret; + int success; + mld_assert_bound(a->coeffs, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, + MLD_REDUCE32_RANGE_MAX); + /* The native backend returns 0 if all coefficients are within the bound, + * 1 if at least one coefficient exceeds the bound, and + * -1 (MLD_NATIVE_FUNC_FALLBACK) if the platform does not have the + * required capabilities to run the native function. + */ + ret = mld_poly_chknorm_native(a->coeffs, B); + + success = (ret != MLD_NATIVE_FUNC_FALLBACK); + /* Constant-time: It would be fine to leak the return value of chknorm + * entirely (as it is fine to leak if any coefficient exceeded the bound or + * not). However, it is cleaner to perform declassification in sign.c. + * Hence, here we only declassify if the native function returned + * MLD_NATIVE_FUNC_FALLBACK or not (which solely depends on system + * capabilities). + */ + MLD_CT_TESTING_DECLASSIFY(&success, sizeof(int)); + if (success) + { + /* Convert 0 / 1 to 0 / 0xFFFFFFFF here */ + return mld_ct_cmask_nonzero_u32((uint32_t)ret); + } +#endif /* MLD_USE_NATIVE_POLY_CHKNORM */ + return mld_poly_chknorm_c(a, B); +} + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + MLD_CONFIG_PARAMETER_SET == 44 +MLD_INTERNAL_API +void mld_polyw1_pack_88(uint8_t r[MLDSA_POLYW1_PACKEDBYTES_88], + const mld_poly *a) +{ + unsigned int i; + + mld_assert_bound(a->coeffs, MLDSA_N, 0, + (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2_88)); + + for (i = 0; i < MLDSA_N / 4; ++i) + __loop__( + invariant(i <= MLDSA_N/4) + decreases(MLDSA_N / 4 - i)) + { + r[3 * i + 0] = (uint8_t)((a->coeffs[4 * i + 0]) & 0xFF); + r[3 * i + 0] |= (uint8_t)((a->coeffs[4 * i + 1] << 6) & 0xFF); + r[3 * i + 1] = (uint8_t)((a->coeffs[4 * i + 1] >> 2) & 0xFF); + r[3 * i + 1] |= (uint8_t)((a->coeffs[4 * i + 2] << 4) & 0xFF); + r[3 * i + 2] = (uint8_t)((a->coeffs[4 * i + 2] >> 4) & 0xFF); + r[3 * i + 2] |= (uint8_t)((a->coeffs[4 * i + 3] << 2) & 0xFF); + } +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \ + */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) +MLD_INTERNAL_API +void mld_polyw1_pack_32(uint8_t r[MLDSA_POLYW1_PACKEDBYTES_32], + const mld_poly *a) +{ + unsigned int i; + + mld_assert_bound(a->coeffs, MLDSA_N, 0, + (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2_32)); + + for (i = 0; i < MLDSA_N / 2; ++i) + __loop__( + invariant(i <= MLDSA_N/2) + decreases(MLDSA_N / 2 - i)) + { + r[i] = + (uint8_t)((a->coeffs[2 * i + 0] | (a->coeffs[2 * i + 1] << 4)) & 0xFF); + } +} +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87 */ +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#else /* !MLD_CONFIG_MULTILEVEL_NO_SHARED */ +MLD_EMPTY_CU(mld_poly) +#endif /* MLD_CONFIG_MULTILEVEL_NO_SHARED */ + +/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. + * Don't modify by hand -- this is auto-generated by scripts/autogen. */ +#undef MLD_POLY_UNIFORM_NBLOCKS diff --git a/cbits/mldsa/src/poly.h b/cbits/mldsa/src/poly.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/poly.h @@ -0,0 +1,464 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + */ + +#ifndef MLD_POLY_H +#define MLD_POLY_H + +#include "cbmc.h" +#include "common.h" +#include "reduce.h" +#include "rounding.h" + +/* Absolute exclusive upper bound for the output of fqmul */ +#define MLD_FQMUL_BOUND ((5 * MLDSA_Q + 3) / 4) +/* Absolute exclusive upper bound for the output of the forward NTT */ +#define MLD_NTT_BOUND (9 * MLD_FQMUL_BOUND) +/* Absolute exclusive upper bound for the output of the inverse NTT*/ +#define MLD_INTT_BOUND MLDSA_Q + +/** + * Element of R_q = Z_q[X]/(X^n + 1). Represents polynomial + * coeffs[0] + X*coeffs[1] + X^2*coeffs[2] + ... + X^{n-1}*coeffs[n-1]. + */ +typedef struct +{ + int32_t coeffs[MLDSA_N]; /**< Polynomial coefficients. */ +} MLD_ALIGN mld_poly; + +#define mld_poly_reduce MLD_NAMESPACE(poly_reduce) +/** + * In-place reduction of all coefficients of polynomial to representative in + * [-MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX]. + * + * @param[in,out] a Pointer to input/output polynomial. + */ +MLD_INTERNAL_API +void mld_poly_reduce(mld_poly *a) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(array_bound(a->coeffs, 0, MLDSA_N, INT32_MIN, MLD_REDUCE32_DOMAIN_MAX)) + assigns(memory_slice(a, sizeof(mld_poly))) + ensures(array_bound(a->coeffs, 0, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX)) +); + +#define mld_poly_caddq MLD_NAMESPACE(poly_caddq) +/** + * For all coefficients of in/out polynomial add MLDSA_Q if coefficient is + * negative. + * + * @param[in,out] a Pointer to input/output polynomial. + */ +MLD_INTERNAL_API +void mld_poly_caddq(mld_poly *a) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q)) + assigns(memory_slice(a, sizeof(mld_poly))) + ensures(array_bound(a->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) +); + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_SIGN_API) || \ + defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +#define mld_poly_add MLD_NAMESPACE(poly_add) +/** + * Add polynomials. No modular reduction is performed. + * + * @spec{Implements @[FIPS204, Algorithm 44, AddNTT] (coefficientwise + * polynomial addition; also used for addition in the normal domain).} + * + * @param[in,out] r Pointer to input-output polynomial to be added to. + * @param[in] b Pointer to input polynomial that should be added to r. + * Must be disjoint from r. + */ + +/* + * NOTE: The reference implementation uses a 3-argument poly_add. + * We specialize to the accumulator form to avoid reasoning about aliasing. + */ +MLD_INTERNAL_API +void mld_poly_add(mld_poly *r, const mld_poly *b) +__contract__( + requires(memory_no_alias(b, sizeof(mld_poly))) + requires(memory_no_alias(r, sizeof(mld_poly))) + requires(forall(k0, 0, MLDSA_N, (int64_t) r->coeffs[k0] + b->coeffs[k0] < MLD_REDUCE32_DOMAIN_MAX)) + requires(forall(k1, 0, MLDSA_N, (int64_t) r->coeffs[k1] + b->coeffs[k1] >= INT32_MIN)) + assigns(memory_slice(r, sizeof(mld_poly))) + ensures(forall(k2, 0, MLDSA_N, r->coeffs[k2] == old(*r).coeffs[k2] + b->coeffs[k2])) + ensures(forall(k3, 0, MLDSA_N, r->coeffs[k3] < MLD_REDUCE32_DOMAIN_MAX)) + ensures(forall(k4, 0, MLDSA_N, r->coeffs[k4] >= INT32_MIN)) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API || \ + MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +#define mld_poly_sub MLD_NAMESPACE(poly_sub) +/** + * Subtract polynomials. No modular reduction is performed. + * + * @param[in,out] r Pointer to input-output polynomial. + * @param[in] b Pointer to input polynomial that should be subtracted from + * r. Must be disjoint from r. + */ +/* + * NOTE: The reference implementation uses a 3-argument poly_sub. + * We specialize to the accumulator form to avoid reasoning about aliasing. + */ +MLD_INTERNAL_API +void mld_poly_sub(mld_poly *r, const mld_poly *b) +__contract__( + requires(memory_no_alias(b, sizeof(mld_poly))) + requires(memory_no_alias(r, sizeof(mld_poly))) + requires(array_abs_bound(r->coeffs, 0, MLDSA_N, MLDSA_Q)) + requires(array_abs_bound(b->coeffs, 0, MLDSA_N, MLDSA_Q)) + assigns(memory_slice(r, sizeof(mld_poly))) + ensures(array_bound(r->coeffs, 0, MLDSA_N, INT32_MIN, MLD_REDUCE32_DOMAIN_MAX)) +); +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +#define mld_poly_shiftl MLD_NAMESPACE(poly_shiftl) +/** + * Multiply polynomial by 2^MLDSA_D without modular reduction. Assumes input + * coefficients to be less than 2^{31-MLDSA_D} in absolute value. + * + * @param[in,out] a Pointer to input/output polynomial. + */ +MLD_INTERNAL_API +void mld_poly_shiftl(mld_poly *a) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(array_bound(a->coeffs, 0, MLDSA_N, 0, 1 << 10)) + assigns(memory_slice(a, sizeof(mld_poly))) + ensures(array_bound(a->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) +); +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#define mld_poly_ntt MLD_NAMESPACE(poly_ntt) +/** + * In-place forward NTT. Output coefficients are bounded by MLD_NTT_BOUND in + * absolute value. + * + * @spec{Implements @[FIPS204, Algorithm 41, NTT].} + * + * @param[in,out] a Pointer to input/output polynomial. + */ +MLD_INTERNAL_API +void mld_poly_ntt(mld_poly *a) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q)) + assigns(memory_slice(a, sizeof(mld_poly))) + ensures(array_abs_bound(a->coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) +); + + +#define mld_poly_invntt_tomont MLD_NAMESPACE(poly_invntt_tomont) +/** + * In-place inverse NTT. + * + * Input coefficients need to be less than MLDSA_Q in absolute value and + * output coefficients are bounded by MLD_INTT_BOUND. + * + * @spec{Implements @[FIPS204, Algorithm 42, NTT^{-1}] up to scaling: + * The input is scaled by 2^{-32} as a result of the Montgomery base + * multiplication. The output is in normal domain. In other words, this + * function implements `NTT^{-1} o mult(2^32)`.} + * + * @param[in,out] a Pointer to input/output polynomial. + */ +MLD_INTERNAL_API +void mld_poly_invntt_tomont(mld_poly *a) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q)) + assigns(memory_slice(a, sizeof(mld_poly))) + ensures(array_abs_bound(a->coeffs, 0, MLDSA_N, MLD_INTT_BOUND)) +); + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \ + defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +#define mld_poly_pointwise_montgomery MLD_NAMESPACE(poly_pointwise_montgomery) +/** + * Pointwise multiplication of polynomials. Destructive in the first argument. + * + * @spec{Implements @[FIPS204, Algorithm 45, MultiplyNTT], up to scaling: The + * input is in normal domain, the output is scaled by 2^{-32} as a result of + * the use of Montgomery multiplication. In other words, this function + * implements `mult(2^{-32}) o MultiplyNTT`.} + * + * @param[in,out] a Pointer to first input/output polynomial. On entry, holds + * the first multiplicand; on exit, holds the product + * a * b * 2^{-32}. + * @param[in] b Pointer to second input polynomial. + */ +MLD_INTERNAL_API +void mld_poly_pointwise_montgomery(mld_poly *a, const mld_poly *b) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(memory_no_alias(b, sizeof(mld_poly))) + requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) + requires(array_abs_bound(b->coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) + assigns(memory_slice(a, sizeof(mld_poly))) + ensures(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_Q)) +); +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API || \ + MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +#define mld_poly_power2round MLD_NAMESPACE(poly_power2round) +/** + * For all coefficients c of the input polynomial, compute c0, c1 such that + * c mod MLDSA_Q = c1*2^MLDSA_D + c0 with -2^{MLDSA_D-1} < c0 <= 2^{MLDSA_D-1}. + * Assumes coefficients to be standard representatives. + * + * @param[out] a1 Pointer to output polynomial with coefficients c1. + * @param[out] a0 Pointer to output polynomial with coefficients c0; may alias + * the input polynomial a. + * @param[in] a Pointer to input polynomial. + */ +MLD_INTERNAL_API +void mld_poly_power2round(mld_poly *a1, mld_poly *a0, const mld_poly *a) +__contract__( + requires(memory_no_alias(a0, sizeof(mld_poly))) + requires(memory_no_alias(a1, sizeof(mld_poly))) + /* The implementation does not require a0 == a, but the single call site + * aliases them and asserting equality simplifies the proof. */ + requires(a0 == a) + requires(array_bound(a->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) + assigns(memory_slice(a1, sizeof(mld_poly))) + assigns(memory_slice(a0, sizeof(mld_poly))) + ensures(array_bound(a0->coeffs, 0, MLDSA_N, -(MLD_2_POW_D/2)+1, (MLD_2_POW_D/2)+1)) + ensures(array_bound(a1->coeffs, 0, MLDSA_N, 0, ((MLDSA_Q - 1) / MLD_2_POW_D) + 1)) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#define mld_poly_uniform MLD_NAMESPACE(poly_uniform) +/** + * Sample polynomial with uniformly random coefficients in [0, MLDSA_Q-1] by + * performing rejection sampling on the output stream of SHAKE128(seed|nonce). + * + * @spec{Implements @[FIPS204, Algorithm 30, RejNTTPoly].} + * + * @param[out] a Pointer to output polynomial. + * @param[in] seed Byte array with seed of length MLDSA_SEEDBYTES and the + * packed 2-byte nonce. + */ +MLD_INTERNAL_API +void mld_poly_uniform(mld_poly *a, const uint8_t seed[MLDSA_SEEDBYTES + 2]) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(memory_no_alias(seed, MLDSA_SEEDBYTES + 2)) + assigns(memory_slice(a, sizeof(mld_poly))) + ensures(array_bound(a->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) +); + +#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)) +#define mld_poly_uniform_4x MLD_NAMESPACE(poly_uniform_4x) +/** + * Generate four polynomials using rejection sampling on (pseudo-)uniformly + * random bytes sampled from a seed. + * + * @spec{Implements @[FIPS204, Algorithm 30, RejNTTPoly] (four-way batched).} + * + * @param[out] vec0 Pointer to first polynomial to be sampled. + * @param[out] vec1 Pointer to second polynomial to be sampled. + * @param[out] vec2 Pointer to third polynomial to be sampled. + * @param[out] vec3 Pointer to fourth polynomial to be sampled. + * @param[in] seed Pointer to consecutive array of seed buffers of size + * MLDSA_SEEDBYTES + 2 each, plus padding for alignment. + */ +MLD_INTERNAL_API +void mld_poly_uniform_4x(mld_poly *vec0, mld_poly *vec1, mld_poly *vec2, + mld_poly *vec3, + uint8_t seed[4][MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2)]) +__contract__( + requires(memory_no_alias(vec0, sizeof(mld_poly))) + requires(memory_no_alias(vec1, sizeof(mld_poly))) + requires(memory_no_alias(vec2, sizeof(mld_poly))) + requires(memory_no_alias(vec3, sizeof(mld_poly))) + requires(memory_no_alias(seed, 4 * MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2))) + assigns(memory_slice(vec0, sizeof(mld_poly))) + assigns(memory_slice(vec1, sizeof(mld_poly))) + assigns(memory_slice(vec2, sizeof(mld_poly))) + assigns(memory_slice(vec3, sizeof(mld_poly))) + ensures(array_bound(vec0->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) + ensures(array_bound(vec1->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) + ensures(array_bound(vec2->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) + ensures(array_bound(vec3->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) +); +#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY && (!MLD_CONFIG_REDUCE_RAM || \ + MLD_UNIT_TEST) */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +#define mld_polyt1_pack MLD_NAMESPACE(polyt1_pack) +/** + * Bit-pack polynomial t1 with coefficients fitting in 10 bits. Input + * coefficients are assumed to be standard representatives. + * + * @spec{Implements @[FIPS204, Algorithm 16, SimpleBitPack].} + * + * @param[out] r Pointer to output byte array with at least + * MLDSA_POLYT1_PACKEDBYTES bytes. + * @param[in] a Pointer to input polynomial. + */ +MLD_INTERNAL_API +void mld_polyt1_pack(uint8_t r[MLDSA_POLYT1_PACKEDBYTES], const mld_poly *a) +__contract__( + requires(memory_no_alias(r, MLDSA_POLYT1_PACKEDBYTES)) + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(array_bound(a->coeffs, 0, MLDSA_N, 0, 1 << 10)) + assigns(memory_slice(r, MLDSA_POLYT1_PACKEDBYTES)) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +#define mld_polyt1_unpack MLD_NAMESPACE(polyt1_unpack) +/** + * Unpack polynomial t1 with 10-bit coefficients. Output coefficients are + * standard representatives. + * + * @spec{Implements @[FIPS204, Algorithm 18, SimpleBitUnpack].} + * + * @param[out] r Pointer to output polynomial. + * @param[in] a Byte array with bit-packed polynomial. + */ +MLD_INTERNAL_API +void mld_polyt1_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYT1_PACKEDBYTES]) +__contract__( + requires(memory_no_alias(r, sizeof(mld_poly))) + requires(memory_no_alias(a, MLDSA_POLYT1_PACKEDBYTES)) + assigns(memory_slice(r, sizeof(mld_poly))) + ensures(array_bound(r->coeffs, 0, MLDSA_N, 0, 1 << 10)) +); +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +#define mld_polyt0_pack MLD_NAMESPACE(polyt0_pack) +/** + * Bit-pack polynomial t0 with coefficients in ]-2^{MLDSA_D-1}, 2^{MLDSA_D-1}]. + * + * @spec{Implements @[FIPS204, Algorithm 17, BitPack].} + * + * @param[out] r Pointer to output byte array with at least + * MLDSA_POLYT0_PACKEDBYTES bytes. + * @param[in] a Pointer to input polynomial. + */ +MLD_INTERNAL_API +void mld_polyt0_pack(uint8_t r[MLDSA_POLYT0_PACKEDBYTES], const mld_poly *a) +__contract__( + requires(memory_no_alias(r, MLDSA_POLYT0_PACKEDBYTES)) + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(array_bound(a->coeffs, 0, MLDSA_N, -(1<<(MLDSA_D-1)) + 1, (1<<(MLDSA_D-1)) + 1)) + assigns(memory_slice(r, MLDSA_POLYT0_PACKEDBYTES)) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST) +#define mld_polyt0_unpack MLD_NAMESPACE(polyt0_unpack) +/** + * Unpack polynomial t0 with coefficients in ]-2^{MLDSA_D-1}, 2^{MLDSA_D-1}]. + * + * @spec{Implements @[FIPS204, Algorithm 19, BitUnpack].} + * + * @param[out] r Pointer to output polynomial. + * @param[in] a Byte array with bit-packed polynomial. + */ +MLD_INTERNAL_API +void mld_polyt0_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYT0_PACKEDBYTES]) +__contract__( + requires(memory_no_alias(r, sizeof(mld_poly))) + requires(memory_no_alias(a, MLDSA_POLYT0_PACKEDBYTES)) + assigns(memory_slice(r, sizeof(mld_poly))) + ensures(array_bound(r->coeffs, 0, MLDSA_N, -(1<<(MLDSA_D-1)) + 1, (1<<(MLDSA_D-1)) + 1)) +); +#endif /* !MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST */ + +#define mld_poly_chknorm MLD_NAMESPACE(poly_chknorm) +/** + * Check infinity norm of polynomial against given bound. Assumes input + * coefficients were reduced by mld_reduce32(). + * + * @spec{@[FIPS204] defines the infinity norm via signed canonical reduction + * (mod± MLDSA_Q) prior to applying the bounds check. However, + * `-B < (a mod± MLDSA_Q) < B` is equivalent to `-B < a < B` under the + * assumption that `B <= MLDSA_Q - MLD_REDUCE32_RANGE_MAX` (cf. the assertion in + * the code). Hence, this contract and implementation are correct without + * reduction.} + * + * @param[in] a Pointer to polynomial. + * @param B Norm bound. + * + * @return 0 if norm is strictly smaller than + * B <= (MLDSA_Q - MLD_REDUCE32_RANGE_MAX) and 0xFFFFFFFF otherwise. + */ +MLD_INTERNAL_API +MLD_MUST_CHECK_RETURN_VALUE +uint32_t mld_poly_chknorm(const mld_poly *a, int32_t B) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(0 <= B && B <= MLDSA_Q - MLD_REDUCE32_RANGE_MAX) + requires(array_bound(a->coeffs, 0, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX)) + ensures(return_value == 0 || return_value == 0xFFFFFFFF) + ensures((return_value == 0) == array_abs_bound(a->coeffs, 0, MLDSA_N, B)) +); + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || MLD_CONFIG_PARAMETER_SET == 44 +#define mld_polyw1_pack_88 MLD_NAMESPACE(polyw1_pack_88) +/** + * Bit-pack polynomial w1, using 6 bits per coefficient. + * This is the variant for parameter sets with MLDSA_GAMMA2 = (MLDSA_Q-1)/88 + * (ML-DSA-44), for which w1 coefficients lie in [0, 43]. + * + * @param[out] r Pointer to output byte array (MLDSA_POLYW1_PACKEDBYTES_88). + * @param[in] a Pointer to input polynomial. + */ +MLD_INTERNAL_API +void mld_polyw1_pack_88(uint8_t r[MLDSA_POLYW1_PACKEDBYTES_88], + const mld_poly *a) +__contract__( + requires(memory_no_alias(r, MLDSA_POLYW1_PACKEDBYTES_88)) + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(array_bound(a->coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2_88))) + assigns(memory_slice(r, MLDSA_POLYW1_PACKEDBYTES_88)) +); +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 44 \ + */ + +#if defined(MLD_CONFIG_MULTILEVEL_WITH_SHARED) || \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) +#define mld_polyw1_pack_32 MLD_NAMESPACE(polyw1_pack_32) +/** + * Bit-pack polynomial w1, using 4 bits per coefficient. + * This is the variant for parameter sets with MLDSA_GAMMA2 = (MLDSA_Q-1)/32 + * (ML-DSA-65 and ML-DSA-87), for which w1 coefficients lie in [0, 15]. + * + * @param[out] r Pointer to output byte array (MLDSA_POLYW1_PACKEDBYTES_32). + * @param[in] a Pointer to input polynomial. + */ +MLD_INTERNAL_API +void mld_polyw1_pack_32(uint8_t r[MLDSA_POLYW1_PACKEDBYTES_32], + const mld_poly *a) +__contract__( + requires(memory_no_alias(r, MLDSA_POLYW1_PACKEDBYTES_32)) + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(array_bound(a->coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2_32))) + assigns(memory_slice(r, MLDSA_POLYW1_PACKEDBYTES_32)) +); +#endif /* MLD_CONFIG_MULTILEVEL_WITH_SHARED || MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87 */ +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#endif /* !MLD_POLY_H */ diff --git a/cbits/mldsa/src/poly_kl.c b/cbits/mldsa/src/poly_kl.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/poly_kl.c @@ -0,0 +1,910 @@ +/* + * Copyright (c) The mldsa-native project authors + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + * + * - [REF] + * CRYSTALS-Dilithium reference implementation + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/dilithium/tree/master/ref + */ + +#include "poly_kl.h" + +#include "ct.h" +#include "debug.h" +#include "rounding.h" +#include "symmetric.h" + +/* Parameter set namespacing + * This is to facilitate building multiple instances + * of mldsa-native (e.g. with varying parameter sets) + * within a single compilation unit. */ +#define mld_rej_eta MLD_ADD_PARAM_SET(mld_rej_eta) +#define mld_rej_eta_c MLD_ADD_PARAM_SET(mld_rej_eta_c) +#define mld_poly_decompose_c MLD_ADD_PARAM_SET(mld_poly_decompose_c) +#define mld_poly_use_hint_c MLD_ADD_PARAM_SET(mld_poly_use_hint_c) +#define mld_polyz_unpack_c MLD_ADD_PARAM_SET(mld_polyz_unpack_c) +/* End of parameter set namespacing */ + + +#if !defined(MLD_CONFIG_NO_SIGN_API) +MLD_STATIC_TESTABLE +void mld_poly_decompose_c(mld_poly *a1, mld_poly *a0) +__contract__( + requires(memory_no_alias(a1, sizeof(mld_poly))) + requires(memory_no_alias(a0, sizeof(mld_poly))) + requires(array_bound(a0->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) + assigns(memory_slice(a1, sizeof(mld_poly))) + assigns(memory_slice(a0, sizeof(mld_poly))) + ensures(array_bound(a1->coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2))) + ensures(array_abs_bound(a0->coeffs, 0, MLDSA_N, MLDSA_GAMMA2+1)) +) +{ + unsigned int i; + mld_assert_bound(a0->coeffs, MLDSA_N, 0, MLDSA_Q); + for (i = 0; i < MLDSA_N; ++i) + __loop__( + assigns(i, memory_slice(a0, sizeof(mld_poly)), memory_slice(a1, sizeof(mld_poly))) + invariant(i <= MLDSA_N) + invariant(array_bound(a0->coeffs, i, MLDSA_N, 0, MLDSA_Q)) + invariant(array_bound(a1->coeffs, 0, i, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2))) + invariant(array_abs_bound(a0->coeffs, 0, i, MLDSA_GAMMA2+1)) + decreases(MLDSA_N - i) + ) + { + mld_decompose(&a0->coeffs[i], &a1->coeffs[i], a0->coeffs[i]); + } + + mld_assert_abs_bound(a0->coeffs, MLDSA_N, MLDSA_GAMMA2 + 1); + mld_assert_bound(a1->coeffs, MLDSA_N, 0, (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2)); +} + +MLD_INTERNAL_API +void mld_poly_decompose(mld_poly *a1, mld_poly *a0) +{ +#if defined(MLD_USE_NATIVE_POLY_DECOMPOSE_88) && MLD_CONFIG_PARAMETER_SET == 44 + int ret; + mld_assert_bound(a0->coeffs, MLDSA_N, 0, MLDSA_Q); + ret = mld_poly_decompose_88_native(a1->coeffs, a0->coeffs); + if (ret == MLD_NATIVE_FUNC_SUCCESS) + { + mld_assert_abs_bound(a0->coeffs, MLDSA_N, MLDSA_GAMMA2 + 1); + mld_assert_bound(a1->coeffs, MLDSA_N, 0, + (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2)); + return; + } +#elif defined(MLD_USE_NATIVE_POLY_DECOMPOSE_32) && \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) + int ret; + mld_assert_bound(a0->coeffs, MLDSA_N, 0, MLDSA_Q); + ret = mld_poly_decompose_32_native(a1->coeffs, a0->coeffs); + if (ret == MLD_NATIVE_FUNC_SUCCESS) + { + mld_assert_abs_bound(a0->coeffs, MLDSA_N, MLDSA_GAMMA2 + 1); + mld_assert_bound(a1->coeffs, MLDSA_N, 0, + (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2)); + return; + } +#endif /* !(MLD_USE_NATIVE_POLY_DECOMPOSE_88 && MLD_CONFIG_PARAMETER_SET == \ + 44) && MLD_USE_NATIVE_POLY_DECOMPOSE_32 && (MLD_CONFIG_PARAMETER_SET \ + == 65 || MLD_CONFIG_PARAMETER_SET == 87) */ + mld_poly_decompose_c(a1, a0); +} + +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +MLD_STATIC_TESTABLE void mld_poly_use_hint_c(mld_poly *a, const mld_poly *h) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(memory_no_alias(h, sizeof(mld_poly))) + requires(array_bound(a->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) + requires(array_bound(h->coeffs, 0, MLDSA_N, 0, 2)) + assigns(memory_slice(a, sizeof(mld_poly))) + ensures(array_bound(a->coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2))) +) +{ + unsigned int i; + mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q); + mld_assert_bound(h->coeffs, MLDSA_N, 0, 2); + + for (i = 0; i < MLDSA_N; ++i) + __loop__( + invariant(i <= MLDSA_N) + invariant(array_bound(a->coeffs, 0, i, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2))) + invariant(array_bound(a->coeffs, i, MLDSA_N, 0, MLDSA_Q)) + decreases(MLDSA_N - i) + ) + { + a->coeffs[i] = mld_use_hint(a->coeffs[i], h->coeffs[i]); + } + mld_assert_bound(a->coeffs, MLDSA_N, 0, (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2)); +} + +MLD_INTERNAL_API +void mld_poly_use_hint(mld_poly *a, const mld_poly *h) +{ +#if defined(MLD_USE_NATIVE_POLY_USE_HINT_88) && MLD_CONFIG_PARAMETER_SET == 44 + int ret; + mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q); + mld_assert_bound(h->coeffs, MLDSA_N, 0, 2); + ret = mld_poly_use_hint_88_native(a->coeffs, h->coeffs); + if (ret == MLD_NATIVE_FUNC_SUCCESS) + { + mld_assert_bound(a->coeffs, MLDSA_N, 0, (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2)); + return; + } +#elif defined(MLD_USE_NATIVE_POLY_USE_HINT_32) && \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) + int ret; + mld_assert_bound(a->coeffs, MLDSA_N, 0, MLDSA_Q); + mld_assert_bound(h->coeffs, MLDSA_N, 0, 2); + ret = mld_poly_use_hint_32_native(a->coeffs, h->coeffs); + if (ret == MLD_NATIVE_FUNC_SUCCESS) + { + mld_assert_bound(a->coeffs, MLDSA_N, 0, (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2)); + return; + } +#endif /* !(MLD_USE_NATIVE_POLY_USE_HINT_88 && MLD_CONFIG_PARAMETER_SET == 44) \ + && MLD_USE_NATIVE_POLY_USE_HINT_32 && (MLD_CONFIG_PARAMETER_SET == \ + 65 || MLD_CONFIG_PARAMETER_SET == 87) */ + mld_poly_use_hint_c(a, h); +} +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +/** + * Sample uniformly random coefficients in [-MLDSA_ETA, MLDSA_ETA] by + * performing rejection sampling on an array of random bytes. + * + * @param[out] a Pointer to output array (allocated). + * @param target Requested number of coefficients to sample. + * @param offset Number of coefficients already sampled. + * @param[in] buf Array of random bytes to sample from. + * @param buflen Length of array of random bytes. + * + * @return Number of sampled coefficients. Can be smaller than target if not + * enough random bytes were given. + */ + +/* Reference: `mld_rej_eta()` in the reference implementation @[REF]. + * - Our signature differs from the reference implementation + * in that it adds the offset and always expects the base of the + * target buffer. This avoids shifting the buffer base in the + * caller, which appears tricky to reason about. */ +#if MLDSA_ETA == 2 +/* + * Sampling 256 coefficients mod 15 using rejection sampling from 4 bits. + * Expected number of required bytes: (256 * (16/15))/2 = 136.5 bytes. + * We sample 1 block (=136 bytes) of SHAKE256_RATE output initially. + * Sampling 2 blocks initially results in slightly worse performance. + */ +#define MLD_POLY_UNIFORM_ETA_NBLOCKS 1 +#elif MLDSA_ETA == 4 +/* + * Sampling 256 coefficients mod 9 using rejection sampling from 4 bits. + * Expected number of required bytes: (256 * (16/9))/2 = 227.5 bytes. + * We sample 2 blocks (=272 bytes) of SHAKE256_RATE output initially. + */ +#define MLD_POLY_UNIFORM_ETA_NBLOCKS 2 +#else /* MLDSA_ETA == 4 */ +#error "Invalid value of MLDSA_ETA" +#endif /* MLDSA_ETA != 2 && MLDSA_ETA != 4 */ + +MLD_STATIC_TESTABLE unsigned int mld_rej_eta_c(int32_t *a, unsigned int target, + unsigned int offset, + const uint8_t *buf, + unsigned int buflen) +__contract__( + requires(offset <= target && target <= MLDSA_N) + requires(buflen <= (MLD_POLY_UNIFORM_ETA_NBLOCKS * MLD_STREAM256_BLOCKBYTES)) + requires(memory_no_alias(a, sizeof(int32_t) * target)) + requires(memory_no_alias(buf, buflen)) + requires(array_abs_bound(a, 0, offset, MLDSA_ETA + 1)) + assigns(memory_slice(a, sizeof(int32_t) * target)) + ensures(offset <= return_value && return_value <= target) + ensures(array_abs_bound(a, 0, return_value, MLDSA_ETA + 1)) +) +{ + unsigned int ctr, pos; + int t_valid; + uint32_t t0, t1; + mld_assert_abs_bound(a, offset, MLDSA_ETA + 1); + ctr = offset; + pos = 0; + while (ctr < target && pos < buflen) + __loop__( + invariant(offset <= ctr && ctr <= target && pos <= buflen) + invariant(array_abs_bound(a, 0, ctr, MLDSA_ETA + 1)) + decreases(buflen - pos) + ) + { + t0 = buf[pos] & 0x0F; + t1 = buf[pos++] >> 4; + + /* Constant time: The inputs and outputs to the rejection sampling are + * secret. However, it is fine to leak which coefficients have been + * rejected. For constant-time testing, we declassify the result of + * the comparison. + */ +#if MLDSA_ETA == 2 + t_valid = t0 < 15; + MLD_CT_TESTING_DECLASSIFY(&t_valid, sizeof(int)); + if (t_valid) /* t0 < 15 */ + { + t0 = t0 - (205 * t0 >> 10) * 5; + a[ctr++] = 2 - (int32_t)t0; + } + t_valid = t1 < 15; + MLD_CT_TESTING_DECLASSIFY(&t_valid, sizeof(int)); + if (t_valid && ctr < target) /* t1 < 15 */ + { + t1 = t1 - (205 * t1 >> 10) * 5; + a[ctr++] = 2 - (int32_t)t1; + } +#elif MLDSA_ETA == 4 + t_valid = t0 < 9; + MLD_CT_TESTING_DECLASSIFY(&t_valid, sizeof(int)); + if (t_valid) /* t0 < 9 */ + { + a[ctr++] = 4 - (int32_t)t0; + } + t_valid = t1 < 9; /* t1 < 9 */ + MLD_CT_TESTING_DECLASSIFY(&t_valid, sizeof(int)); + if (t_valid && ctr < target) + { + a[ctr++] = 4 - (int32_t)t1; + } +#else /* MLDSA_ETA == 4 */ +#error "Invalid value of MLDSA_ETA" +#endif /* MLDSA_ETA != 2 && MLDSA_ETA != 4 */ + } + + mld_assert_abs_bound(a, ctr, MLDSA_ETA + 1); + + return ctr; +} + +static unsigned int mld_rej_eta(int32_t *a, unsigned int target, + unsigned int offset, const uint8_t *buf, + unsigned int buflen) +__contract__( + requires(offset <= target && target <= MLDSA_N) + requires(buflen <= (MLD_POLY_UNIFORM_ETA_NBLOCKS * MLD_STREAM256_BLOCKBYTES)) + requires(memory_no_alias(a, sizeof(int32_t) * target)) + requires(memory_no_alias(buf, buflen)) + requires(array_abs_bound(a, 0, offset, MLDSA_ETA + 1)) + assigns(memory_slice(a, sizeof(int32_t) * target)) + ensures(offset <= return_value && return_value <= target) + ensures(array_abs_bound(a, 0, return_value, MLDSA_ETA + 1)) +) +{ +#if MLDSA_ETA == 2 && defined(MLD_USE_NATIVE_REJ_UNIFORM_ETA2) + int ret; + mld_assert_abs_bound(a, offset, MLDSA_ETA + 1); + if (offset == 0) + { + ret = mld_rej_uniform_eta2_native(a, target, buf, buflen); + if (ret != MLD_NATIVE_FUNC_FALLBACK) + { + unsigned res = (unsigned)ret; + mld_assert_abs_bound(a, res, MLDSA_ETA + 1); + return res; + } + } +#elif MLDSA_ETA == 4 && defined(MLD_USE_NATIVE_REJ_UNIFORM_ETA4) + int ret; + mld_assert_abs_bound(a, offset, MLDSA_ETA + 1); + if (offset == 0) + { + ret = mld_rej_uniform_eta4_native(a, target, buf, buflen); + if (ret != MLD_NATIVE_FUNC_FALLBACK) + { + unsigned res = (unsigned)ret; + mld_assert_abs_bound(a, res, MLDSA_ETA + 1); + return res; + } + } +#endif /* !(MLDSA_ETA == 2 && MLD_USE_NATIVE_REJ_UNIFORM_ETA2) && MLDSA_ETA == \ + 4 && MLD_USE_NATIVE_REJ_UNIFORM_ETA4 */ + + return mld_rej_eta_c(a, target, offset, buf, buflen); +} + +#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) +MLD_INTERNAL_API +void mld_poly_uniform_eta_4x(mld_poly *r0, mld_poly *r1, mld_poly *r2, + mld_poly *r3, const uint8_t seed[MLDSA_CRHBYTES], + uint8_t nonce0, uint8_t nonce1, uint8_t nonce2, + uint8_t nonce3) +{ + /* Temporary buffers for XOF output before rejection sampling */ + MLD_ALIGN uint8_t buf[4][MLD_ALIGN_UP(MLD_POLY_UNIFORM_ETA_NBLOCKS * + MLD_STREAM256_BLOCKBYTES)]; + + MLD_ALIGN uint8_t extseed[4][MLD_ALIGN_UP(MLDSA_CRHBYTES + 2)]; + + /* Tracks the number of coefficients we have already sampled */ + unsigned ctr[4]; + mld_xof256_x4_ctx state; + unsigned buflen; + + mld_memcpy(extseed[0], seed, MLDSA_CRHBYTES); + mld_memcpy(extseed[1], seed, MLDSA_CRHBYTES); + mld_memcpy(extseed[2], seed, MLDSA_CRHBYTES); + mld_memcpy(extseed[3], seed, MLDSA_CRHBYTES); + extseed[0][MLDSA_CRHBYTES] = nonce0; + extseed[1][MLDSA_CRHBYTES] = nonce1; + extseed[2][MLDSA_CRHBYTES] = nonce2; + extseed[3][MLDSA_CRHBYTES] = nonce3; + extseed[0][MLDSA_CRHBYTES + 1] = 0; + extseed[1][MLDSA_CRHBYTES + 1] = 0; + extseed[2][MLDSA_CRHBYTES + 1] = 0; + extseed[3][MLDSA_CRHBYTES + 1] = 0; + + mld_xof256_x4_init(&state); + mld_xof256_x4_absorb(&state, extseed, MLDSA_CRHBYTES + 2); + + /* + * Initially, squeeze heuristic number of MLD_POLY_UNIFORM_ETA_NBLOCKS. + * This should generate the coefficients with high probability. + */ + mld_xof256_x4_squeezeblocks(buf, MLD_POLY_UNIFORM_ETA_NBLOCKS, &state); + buflen = MLD_POLY_UNIFORM_ETA_NBLOCKS * MLD_STREAM256_BLOCKBYTES; + + ctr[0] = mld_rej_eta(r0->coeffs, MLDSA_N, 0, buf[0], buflen); + ctr[1] = mld_rej_eta(r1->coeffs, MLDSA_N, 0, buf[1], buflen); + ctr[2] = mld_rej_eta(r2->coeffs, MLDSA_N, 0, buf[2], buflen); + ctr[3] = mld_rej_eta(r3->coeffs, MLDSA_N, 0, buf[3], buflen); + + /* + * So long as not all entries have been generated, squeeze + * one more block at a time until we're done. + */ + buflen = MLD_STREAM256_BLOCKBYTES; + while (ctr[0] < MLDSA_N || ctr[1] < MLDSA_N || ctr[2] < MLDSA_N || + ctr[3] < MLDSA_N) + __loop__( + assigns(ctr, state, memory_slice(r0, sizeof(mld_poly)), + memory_slice(r1, sizeof(mld_poly)), memory_slice(r2, sizeof(mld_poly)), + memory_slice(r3, sizeof(mld_poly)), object_whole(buf[0]), + object_whole(buf[1]), object_whole(buf[2]), + object_whole(buf[3])) + invariant(ctr[0] <= MLDSA_N && ctr[1] <= MLDSA_N) + invariant(ctr[2] <= MLDSA_N && ctr[3] <= MLDSA_N) + invariant(array_abs_bound(r0->coeffs, 0, ctr[0], MLDSA_ETA + 1)) + invariant(array_abs_bound(r1->coeffs, 0, ctr[1], MLDSA_ETA + 1)) + invariant(array_abs_bound(r2->coeffs, 0, ctr[2], MLDSA_ETA + 1)) + invariant(array_abs_bound(r3->coeffs, 0, ctr[3], MLDSA_ETA + 1))) + { + mld_xof256_x4_squeezeblocks(buf, 1, &state); + ctr[0] = mld_rej_eta(r0->coeffs, MLDSA_N, ctr[0], buf[0], buflen); + ctr[1] = mld_rej_eta(r1->coeffs, MLDSA_N, ctr[1], buf[1], buflen); + ctr[2] = mld_rej_eta(r2->coeffs, MLDSA_N, ctr[2], buf[2], buflen); + ctr[3] = mld_rej_eta(r3->coeffs, MLDSA_N, ctr[3], buf[3], buflen); + } + + mld_xof256_x4_release(&state); + + mld_assert_abs_bound(r0->coeffs, MLDSA_N, MLDSA_ETA + 1); + mld_assert_abs_bound(r1->coeffs, MLDSA_N, MLDSA_ETA + 1); + mld_assert_abs_bound(r2->coeffs, MLDSA_N, MLDSA_ETA + 1); + mld_assert_abs_bound(r3->coeffs, MLDSA_N, MLDSA_ETA + 1); + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(buf, sizeof(buf)); + mld_zeroize(extseed, sizeof(extseed)); +} +#else /* !MLD_CONFIG_SERIAL_FIPS202_ONLY */ + +MLD_INTERNAL_API +void mld_poly_uniform_eta(mld_poly *r, const uint8_t seed[MLDSA_CRHBYTES], + uint8_t nonce) +{ + /* Temporary buffer for XOF output before rejection sampling */ + MLD_ALIGN uint8_t + buf[MLD_POLY_UNIFORM_ETA_NBLOCKS * MLD_STREAM256_BLOCKBYTES]; + MLD_ALIGN uint8_t extseed[MLDSA_CRHBYTES + 2]; + + /* Tracks the number of coefficients we have already sampled */ + unsigned ctr; + mld_xof256_ctx state; + unsigned buflen; + + mld_memcpy(extseed, seed, MLDSA_CRHBYTES); + extseed[MLDSA_CRHBYTES] = nonce; + extseed[MLDSA_CRHBYTES + 1] = 0; + + mld_xof256_init(&state); + mld_xof256_absorb_once(&state, extseed, MLDSA_CRHBYTES + 2); + + /* + * Initially, squeeze heuristic number of MLD_POLY_UNIFORM_ETA_NBLOCKS. + * This should generate the coefficients with high probability. + */ + mld_xof256_squeezeblocks(buf, MLD_POLY_UNIFORM_ETA_NBLOCKS, &state); + buflen = MLD_POLY_UNIFORM_ETA_NBLOCKS * MLD_STREAM256_BLOCKBYTES; + + ctr = mld_rej_eta(r->coeffs, MLDSA_N, 0, buf, buflen); + + /* + * So long as not all entries have been generated, squeeze + * one more block at a time until we're done. + */ + buflen = MLD_STREAM256_BLOCKBYTES; + while (ctr < MLDSA_N) + __loop__( + assigns(ctr, object_whole(&state), + object_whole(buf), memory_slice(r, sizeof(mld_poly))) + invariant(ctr <= MLDSA_N) + invariant(state.pos <= SHAKE256_RATE) + invariant(array_abs_bound(r->coeffs, 0, ctr, MLDSA_ETA + 1))) + { + mld_xof256_squeezeblocks(buf, 1, &state); + ctr = mld_rej_eta(r->coeffs, MLDSA_N, ctr, buf, buflen); + } + + mld_xof256_release(&state); + + mld_assert_abs_bound(r->coeffs, MLDSA_N, MLDSA_ETA + 1); + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(buf, sizeof(buf)); + mld_zeroize(extseed, sizeof(extseed)); +} +#endif /* MLD_CONFIG_SERIAL_FIPS202_ONLY */ +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +#define MLD_POLY_UNIFORM_GAMMA1_NBLOCKS \ + ((MLDSA_POLYZ_PACKEDBYTES + MLD_STREAM256_BLOCKBYTES - 1) / \ + MLD_STREAM256_BLOCKBYTES) + +#if MLD_CONFIG_PARAMETER_SET == 65 || \ + defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) || \ + defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +MLD_INTERNAL_API +void mld_poly_uniform_gamma1(mld_poly *a, const uint8_t seed[MLDSA_CRHBYTES], + uint16_t nonce) +{ + MLD_ALIGN uint8_t + buf[MLD_POLY_UNIFORM_GAMMA1_NBLOCKS * MLD_STREAM256_BLOCKBYTES]; + MLD_ALIGN uint8_t extseed[MLDSA_CRHBYTES + 2]; + mld_xof256_ctx state; + + mld_memcpy(extseed, seed, MLDSA_CRHBYTES); + extseed[MLDSA_CRHBYTES] = (uint8_t)(nonce & 0xFF); + extseed[MLDSA_CRHBYTES + 1] = (uint8_t)(nonce >> 8); + + mld_xof256_init(&state); + mld_xof256_absorb_once(&state, extseed, MLDSA_CRHBYTES + 2); + + mld_xof256_squeezeblocks(buf, MLD_POLY_UNIFORM_GAMMA1_NBLOCKS, &state); + mld_polyz_unpack(a, buf); + + mld_xof256_release(&state); + + mld_assert_bound(a->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1); + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(buf, sizeof(buf)); + mld_zeroize(extseed, sizeof(extseed)); +} +#endif /* MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_SERIAL_FIPS202_ONLY || \ + MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + + +#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)) +MLD_INTERNAL_API +void mld_poly_uniform_gamma1_4x(mld_poly *r0, mld_poly *r1, mld_poly *r2, + mld_poly *r3, + const uint8_t seed[MLDSA_CRHBYTES], + uint16_t nonce0, uint16_t nonce1, + uint16_t nonce2, uint16_t nonce3) +{ + /* Temporary buffers for XOF output before rejection sampling */ + MLD_ALIGN uint8_t buf[4][MLD_ALIGN_UP(MLD_POLY_UNIFORM_GAMMA1_NBLOCKS * + MLD_STREAM256_BLOCKBYTES)]; + + MLD_ALIGN uint8_t extseed[4][MLD_ALIGN_UP(MLDSA_CRHBYTES + 2)]; + + /* Tracks the number of coefficients we have already sampled */ + mld_xof256_x4_ctx state; + + mld_memcpy(extseed[0], seed, MLDSA_CRHBYTES); + mld_memcpy(extseed[1], seed, MLDSA_CRHBYTES); + mld_memcpy(extseed[2], seed, MLDSA_CRHBYTES); + mld_memcpy(extseed[3], seed, MLDSA_CRHBYTES); + extseed[0][MLDSA_CRHBYTES] = (uint8_t)(nonce0 & 0xFF); + extseed[1][MLDSA_CRHBYTES] = (uint8_t)(nonce1 & 0xFF); + extseed[2][MLDSA_CRHBYTES] = (uint8_t)(nonce2 & 0xFF); + extseed[3][MLDSA_CRHBYTES] = (uint8_t)(nonce3 & 0xFF); + extseed[0][MLDSA_CRHBYTES + 1] = (uint8_t)(nonce0 >> 8); + extseed[1][MLDSA_CRHBYTES + 1] = (uint8_t)(nonce1 >> 8); + extseed[2][MLDSA_CRHBYTES + 1] = (uint8_t)(nonce2 >> 8); + extseed[3][MLDSA_CRHBYTES + 1] = (uint8_t)(nonce3 >> 8); + + mld_xof256_x4_init(&state); + mld_xof256_x4_absorb(&state, extseed, MLDSA_CRHBYTES + 2); + mld_xof256_x4_squeezeblocks(buf, MLD_POLY_UNIFORM_GAMMA1_NBLOCKS, &state); + + mld_polyz_unpack(r0, buf[0]); + mld_polyz_unpack(r1, buf[1]); + mld_polyz_unpack(r2, buf[2]); + mld_polyz_unpack(r3, buf[3]); + mld_xof256_x4_release(&state); + + mld_assert_bound(r0->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1); + mld_assert_bound(r1->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1); + mld_assert_bound(r2->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1); + mld_assert_bound(r3->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1); + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(buf, sizeof(buf)); + mld_zeroize(extseed, sizeof(extseed)); +} +#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY && (!MLD_CONFIG_REDUCE_RAM || \ + MLD_UNIT_TEST) */ +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +MLD_INTERNAL_API +void mld_poly_challenge(mld_poly *c, const uint8_t seed[MLDSA_CTILDEBYTES]) +{ + unsigned int i, j, pos; + uint64_t signs; + uint64_t offset; + MLD_ALIGN uint8_t buf[SHAKE256_RATE]; + mld_shake256ctx state; + + mld_shake256_init(&state); + mld_shake256_absorb(&state, seed, MLDSA_CTILDEBYTES); + mld_shake256_finalize(&state); + mld_shake256_squeeze(buf, SHAKE256_RATE, &state); + + /* Convert the first 8 bytes of buf[] into an unsigned 64-bit value. */ + /* Each bit of that dictates the sign of the resulting challenge value */ + signs = 0; + for (i = 0; i < 8; ++i) + __loop__( + assigns(i, signs) + invariant(i <= 8) + decreases(8 - i) + ) + { + signs |= (uint64_t)buf[i] << 8 * i; + } + pos = 8; + + mld_memset(c, 0, sizeof(mld_poly)); + + for (i = MLDSA_N - MLDSA_TAU; i < MLDSA_N; ++i) + __loop__( + assigns(i, j, object_whole(buf), state, pos, memory_slice(c, sizeof(mld_poly)), signs) + invariant(i >= MLDSA_N - MLDSA_TAU) + invariant(i <= MLDSA_N) + invariant(pos >= 1) + invariant(pos <= SHAKE256_RATE) + invariant(array_bound(c->coeffs, 0, MLDSA_N, -1, 2)) + invariant(state.pos <= SHAKE256_RATE) + decreases(MLDSA_N - i) + ) + { + /* This loop terminates only probabilistically, hence no decreases + * clause. */ + do + __loop__( + assigns(j, object_whole(buf), state, pos) + invariant(state.pos <= SHAKE256_RATE) + ) + { + if (pos >= SHAKE256_RATE) + { + mld_shake256_squeeze(buf, SHAKE256_RATE, &state); + pos = 0; + } + j = buf[pos++]; + } while (j > i); + + c->coeffs[i] = c->coeffs[j]; + + /* Reference: Compute coefficient value here in two steps to */ + /* avoid mixing unsigned and signed arithmetic with implicit */ + /* conversions, and so that CBMC can keep track of ranges */ + /* to complete type-safety proof here. */ + + /* The least-significant bit of signs tells us if we want -1 or +1 */ + offset = 2 * (signs & 1); + + /* offset has value 0 or 2 here, so (1 - (int32_t) offset) has + * value -1 or +1 */ + c->coeffs[j] = 1 - (int32_t)offset; + + /* Move to the next bit of signs for next time */ + signs >>= 1; + } + + mld_assert_bound(c->coeffs, MLDSA_N, -1, 2); + mld_shake256_release(&state); + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(buf, sizeof(buf)); + mld_zeroize(&signs, sizeof(signs)); +} +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +MLD_INTERNAL_API +void mld_polyeta_pack(uint8_t r[MLDSA_POLYETA_PACKEDBYTES], const mld_poly *a) +{ + unsigned int i; + uint8_t t[8]; + + mld_assert_abs_bound(a->coeffs, MLDSA_N, MLDSA_ETA + 1); + +#if MLDSA_ETA == 2 + for (i = 0; i < MLDSA_N / 8; ++i) + __loop__( + invariant(i <= MLDSA_N/8) + decreases(MLDSA_N / 8 - i)) + { + /* The casts are safe since we assume that the coefficients + * of a are <= MLDSA_ETA in absolute value. */ + t[0] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 0]); + t[1] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 1]); + t[2] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 2]); + t[3] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 3]); + t[4] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 4]); + t[5] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 5]); + t[6] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 6]); + t[7] = (uint8_t)(MLDSA_ETA - a->coeffs[8 * i + 7]); + + r[3 * i + 0] = (uint8_t)(((t[0] >> 0) | (t[1] << 3) | (t[2] << 6)) & 0xFF); + r[3 * i + 1] = + (uint8_t)(((t[2] >> 2) | (t[3] << 1) | (t[4] << 4) | (t[5] << 7)) & + 0xFF); + r[3 * i + 2] = (uint8_t)(((t[5] >> 1) | (t[6] << 2) | (t[7] << 5)) & 0xFF); + } +#elif MLDSA_ETA == 4 + for (i = 0; i < MLDSA_N / 2; ++i) + __loop__( + invariant(i <= MLDSA_N/2) + decreases(MLDSA_N / 2 - i)) + { + /* The casts are safe since we assume that the coefficients + * of a are <= MLDSA_ETA in absolute value. */ + t[0] = (uint8_t)(MLDSA_ETA - a->coeffs[2 * i + 0]); + t[1] = (uint8_t)(MLDSA_ETA - a->coeffs[2 * i + 1]); + r[i] = (uint8_t)(t[0] | (t[1] << 4)); + } +#else /* MLDSA_ETA == 4 */ +#error "Invalid value of MLDSA_ETA" +#endif /* MLDSA_ETA != 2 && MLDSA_ETA != 4 */ +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_SIGN_API) +MLD_INTERNAL_API +void mld_polyeta_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYETA_PACKEDBYTES]) +{ + unsigned int i; + +#if MLDSA_ETA == 2 + for (i = 0; i < MLDSA_N / 8; ++i) + __loop__( + invariant(i <= MLDSA_N/8) + invariant(array_bound(r->coeffs, 0, i*8, -5, MLDSA_ETA + 1)) + decreases(MLDSA_N / 8 - i)) + { + r->coeffs[8 * i + 0] = (a[3 * i + 0] >> 0) & 7; + r->coeffs[8 * i + 1] = (a[3 * i + 0] >> 3) & 7; + r->coeffs[8 * i + 2] = ((a[3 * i + 0] >> 6) | (a[3 * i + 1] << 2)) & 7; + r->coeffs[8 * i + 3] = (a[3 * i + 1] >> 1) & 7; + r->coeffs[8 * i + 4] = (a[3 * i + 1] >> 4) & 7; + r->coeffs[8 * i + 5] = ((a[3 * i + 1] >> 7) | (a[3 * i + 2] << 1)) & 7; + r->coeffs[8 * i + 6] = (a[3 * i + 2] >> 2) & 7; + r->coeffs[8 * i + 7] = (a[3 * i + 2] >> 5) & 7; + + r->coeffs[8 * i + 0] = MLDSA_ETA - r->coeffs[8 * i + 0]; + r->coeffs[8 * i + 1] = MLDSA_ETA - r->coeffs[8 * i + 1]; + r->coeffs[8 * i + 2] = MLDSA_ETA - r->coeffs[8 * i + 2]; + r->coeffs[8 * i + 3] = MLDSA_ETA - r->coeffs[8 * i + 3]; + r->coeffs[8 * i + 4] = MLDSA_ETA - r->coeffs[8 * i + 4]; + r->coeffs[8 * i + 5] = MLDSA_ETA - r->coeffs[8 * i + 5]; + r->coeffs[8 * i + 6] = MLDSA_ETA - r->coeffs[8 * i + 6]; + r->coeffs[8 * i + 7] = MLDSA_ETA - r->coeffs[8 * i + 7]; + } +#elif MLDSA_ETA == 4 + for (i = 0; i < MLDSA_N / 2; ++i) + __loop__( + invariant(i <= MLDSA_N/2) + invariant(array_bound(r->coeffs, 0, i*2, -11, MLDSA_ETA + 1)) + decreases(MLDSA_N / 2 - i)) + { + r->coeffs[2 * i + 0] = a[i] & 0x0F; + r->coeffs[2 * i + 1] = a[i] >> 4; + r->coeffs[2 * i + 0] = MLDSA_ETA - r->coeffs[2 * i + 0]; + r->coeffs[2 * i + 1] = MLDSA_ETA - r->coeffs[2 * i + 1]; + } +#else /* MLDSA_ETA == 4 */ +#error "Invalid value of MLDSA_ETA" +#endif /* MLDSA_ETA != 2 && MLDSA_ETA != 4 */ + + mld_assert_bound(r->coeffs, MLDSA_N, MLD_POLYETA_UNPACK_LOWER_BOUND, + MLDSA_ETA + 1); +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +MLD_INTERNAL_API +void mld_polyz_pack(uint8_t r[MLDSA_POLYZ_PACKEDBYTES], const mld_poly *a) +{ + unsigned int i; + uint32_t t[4]; + + mld_assert_bound(a->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1); + +#if MLD_CONFIG_PARAMETER_SET == 44 + for (i = 0; i < MLDSA_N / 4; ++i) + __loop__( + invariant(i <= MLDSA_N/4) + decreases(MLDSA_N / 4 - i)) + { + /* Safety: a->coeffs[i] <= MLDSA_GAMMA1, hence, these casts are safe. */ + t[0] = (uint32_t)(MLDSA_GAMMA1 - a->coeffs[4 * i + 0]); + t[1] = (uint32_t)(MLDSA_GAMMA1 - a->coeffs[4 * i + 1]); + t[2] = (uint32_t)(MLDSA_GAMMA1 - a->coeffs[4 * i + 2]); + t[3] = (uint32_t)(MLDSA_GAMMA1 - a->coeffs[4 * i + 3]); + + r[9 * i + 0] = (uint8_t)((t[0]) & 0xFF); + r[9 * i + 1] = (uint8_t)((t[0] >> 8) & 0xFF); + r[9 * i + 2] = (uint8_t)((t[0] >> 16) & 0xFF); + r[9 * i + 2] |= (uint8_t)((t[1] << 2) & 0xFF); + r[9 * i + 3] = (uint8_t)((t[1] >> 6) & 0xFF); + r[9 * i + 4] = (uint8_t)((t[1] >> 14) & 0xFF); + r[9 * i + 4] |= (uint8_t)((t[2] << 4) & 0xFF); + r[9 * i + 5] = (uint8_t)((t[2] >> 4) & 0xFF); + r[9 * i + 6] = (uint8_t)((t[2] >> 12) & 0xFF); + r[9 * i + 6] |= (uint8_t)((t[3] << 6) & 0xFF); + r[9 * i + 7] = (uint8_t)((t[3] >> 2) & 0xFF); + r[9 * i + 8] = (uint8_t)((t[3] >> 10) & 0xFF); + } +#else /* MLD_CONFIG_PARAMETER_SET == 44 */ + for (i = 0; i < MLDSA_N / 2; ++i) + __loop__( + invariant(i <= MLDSA_N/2) + decreases(MLDSA_N / 2 - i)) + { + /* Safety: a->coeffs[i] <= MLDSA_GAMMA1, hence, these casts are safe. */ + t[0] = (uint32_t)(MLDSA_GAMMA1 - a->coeffs[2 * i + 0]); + t[1] = (uint32_t)(MLDSA_GAMMA1 - a->coeffs[2 * i + 1]); + + r[5 * i + 0] = (uint8_t)((t[0]) & 0xFF); + r[5 * i + 1] = (uint8_t)((t[0] >> 8) & 0xFF); + r[5 * i + 2] = (uint8_t)((t[0] >> 16) & 0xFF); + r[5 * i + 2] |= (uint8_t)((t[1] << 4) & 0xFF); + r[5 * i + 3] = (uint8_t)((t[1] >> 4) & 0xFF); + r[5 * i + 4] = (uint8_t)((t[1] >> 12) & 0xFF); + } +#endif /* MLD_CONFIG_PARAMETER_SET != 44 */ +} +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +MLD_STATIC_TESTABLE void mld_polyz_unpack_c( + mld_poly *r, const uint8_t a[MLDSA_POLYZ_PACKEDBYTES]) +__contract__( + requires(memory_no_alias(r, sizeof(mld_poly))) + requires(memory_no_alias(a, MLDSA_POLYZ_PACKEDBYTES)) + assigns(memory_slice(r, sizeof(mld_poly))) + ensures(array_bound(r->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) +) +{ + unsigned int i; +#if MLD_CONFIG_PARAMETER_SET == 44 + for (i = 0; i < MLDSA_N / 4; ++i) + __loop__( + invariant(i <= MLDSA_N/4) + invariant(array_bound(r->coeffs, 0, i*4, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) + decreases(MLDSA_N / 4 - i)) + { + r->coeffs[4 * i + 0] = a[9 * i + 0]; + r->coeffs[4 * i + 0] |= (int32_t)a[9 * i + 1] << 8; + r->coeffs[4 * i + 0] |= (int32_t)a[9 * i + 2] << 16; + r->coeffs[4 * i + 0] &= 0x3FFFF; + + r->coeffs[4 * i + 1] = a[9 * i + 2] >> 2; + r->coeffs[4 * i + 1] |= (int32_t)a[9 * i + 3] << 6; + r->coeffs[4 * i + 1] |= (int32_t)a[9 * i + 4] << 14; + r->coeffs[4 * i + 1] &= 0x3FFFF; + + r->coeffs[4 * i + 2] = a[9 * i + 4] >> 4; + r->coeffs[4 * i + 2] |= (int32_t)a[9 * i + 5] << 4; + r->coeffs[4 * i + 2] |= (int32_t)a[9 * i + 6] << 12; + r->coeffs[4 * i + 2] &= 0x3FFFF; + + r->coeffs[4 * i + 3] = a[9 * i + 6] >> 6; + r->coeffs[4 * i + 3] |= (int32_t)a[9 * i + 7] << 2; + r->coeffs[4 * i + 3] |= (int32_t)a[9 * i + 8] << 10; + r->coeffs[4 * i + 3] &= 0x3FFFF; + + r->coeffs[4 * i + 0] = MLDSA_GAMMA1 - r->coeffs[4 * i + 0]; + r->coeffs[4 * i + 1] = MLDSA_GAMMA1 - r->coeffs[4 * i + 1]; + r->coeffs[4 * i + 2] = MLDSA_GAMMA1 - r->coeffs[4 * i + 2]; + r->coeffs[4 * i + 3] = MLDSA_GAMMA1 - r->coeffs[4 * i + 3]; + } +#else /* MLD_CONFIG_PARAMETER_SET == 44 */ + for (i = 0; i < MLDSA_N / 2; ++i) + __loop__( + invariant(i <= MLDSA_N/2) + invariant(array_bound(r->coeffs, 0, i*2, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) + decreases(MLDSA_N / 2 - i)) + { + r->coeffs[2 * i + 0] = a[5 * i + 0]; + r->coeffs[2 * i + 0] |= (int32_t)a[5 * i + 1] << 8; + r->coeffs[2 * i + 0] |= (int32_t)a[5 * i + 2] << 16; + r->coeffs[2 * i + 0] &= 0xFFFFF; + + r->coeffs[2 * i + 1] = a[5 * i + 2] >> 4; + r->coeffs[2 * i + 1] |= (int32_t)a[5 * i + 3] << 4; + r->coeffs[2 * i + 1] |= (int32_t)a[5 * i + 4] << 12; + /* r->coeffs[2*i+1] &= 0xFFFFF; */ /* No effect, since we're anyway at 20 + bits */ + + r->coeffs[2 * i + 0] = MLDSA_GAMMA1 - r->coeffs[2 * i + 0]; + r->coeffs[2 * i + 1] = MLDSA_GAMMA1 - r->coeffs[2 * i + 1]; + } +#endif /* MLD_CONFIG_PARAMETER_SET != 44 */ + mld_assert_bound(r->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1); +} + +MLD_INTERNAL_API +void mld_polyz_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYZ_PACKEDBYTES]) +{ +#if defined(MLD_USE_NATIVE_POLYZ_UNPACK_17) && MLD_CONFIG_PARAMETER_SET == 44 + int ret; + ret = mld_polyz_unpack_17_native(r->coeffs, a); + if (ret == MLD_NATIVE_FUNC_SUCCESS) + { + mld_assert_bound(r->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1); + return; + } +#elif defined(MLD_USE_NATIVE_POLYZ_UNPACK_19) && \ + (MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_PARAMETER_SET == 87) + int ret; + ret = mld_polyz_unpack_19_native(r->coeffs, a); + if (ret == MLD_NATIVE_FUNC_SUCCESS) + { + mld_assert_bound(r->coeffs, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1); + return; + } +#endif /* !(MLD_USE_NATIVE_POLYZ_UNPACK_17 && MLD_CONFIG_PARAMETER_SET == 44) \ + && MLD_USE_NATIVE_POLYZ_UNPACK_19 && (MLD_CONFIG_PARAMETER_SET == 65 \ + || MLD_CONFIG_PARAMETER_SET == 87) */ + + mld_polyz_unpack_c(r, a); +} +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. */ + +/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. + * Don't modify by hand -- this is auto-generated by scripts/autogen. */ +#undef mld_rej_eta +#undef mld_rej_eta_c +#undef mld_poly_decompose_c +#undef mld_poly_use_hint_c +#undef mld_polyz_unpack_c +#undef MLD_POLY_UNIFORM_ETA_NBLOCKS +#undef MLD_POLY_UNIFORM_GAMMA1_NBLOCKS diff --git a/cbits/mldsa/src/poly_kl.h b/cbits/mldsa/src/poly_kl.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/poly_kl.h @@ -0,0 +1,367 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + */ + +#ifndef MLD_POLY_KL_H +#define MLD_POLY_KL_H + +#include "cbmc.h" +#include "common.h" +#include "poly.h" + +#if !defined(MLD_CONFIG_NO_SIGN_API) +#define mld_poly_decompose MLD_NAMESPACE_KL(poly_decompose) +/** + * For all coefficients c of the input polynomial, compute high and low bits + * c0, c1 such c mod MLDSA_Q = c1*ALPHA + c0 with -ALPHA/2 < c0 <= ALPHA/2 + * except c1 = (MLDSA_Q-1)/ALPHA where we set c1 = 0 and + * -ALPHA/2 <= c0 = c mod MLDSA_Q - MLDSA_Q < 0. Assumes coefficients to be + * standard representatives. + * + * @reference{The reference implementation has the input polynomial as a + * separate argument that may be aliased with either of the outputs. Removing + * the aliasing eases CBMC proofs.} + * + * @param[out] a1 Pointer to output polynomial with coefficients c1. + * @param[in,out] a0 Pointer to input/output polynomial. Output polynomial has + * coefficients c0. + */ +MLD_INTERNAL_API +void mld_poly_decompose(mld_poly *a1, mld_poly *a0) +__contract__( + requires(memory_no_alias(a1, sizeof(mld_poly))) + requires(memory_no_alias(a0, sizeof(mld_poly))) + requires(array_bound(a0->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) + assigns(memory_slice(a1, sizeof(mld_poly))) + assigns(memory_slice(a0, sizeof(mld_poly))) + ensures(array_bound(a1->coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2))) + ensures(array_abs_bound(a0->coeffs, 0, MLDSA_N, MLDSA_GAMMA2+1)) +); + +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +#define mld_poly_use_hint MLD_NAMESPACE_KL(poly_use_hint) +/** + * Use hint polynomial h to correct the high bits of a in-place. + * + * @param[in,out] a Input/output polynomial. + * @param[in] h Hint polynomial. + */ +MLD_INTERNAL_API +void mld_poly_use_hint(mld_poly *a, const mld_poly *h) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(memory_no_alias(h, sizeof(mld_poly))) + requires(array_bound(a->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) + requires(array_bound(h->coeffs, 0, MLDSA_N, 0, 2)) + assigns(memory_slice(a, sizeof(mld_poly))) + ensures(array_bound(a->coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2))) +); +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) +#define mld_poly_uniform_eta_4x MLD_NAMESPACE_KL(poly_uniform_eta_4x) +/** + * Sample four polynomials with uniformly random coefficients in + * [-MLDSA_ETA, MLDSA_ETA] by performing rejection sampling on the output + * stream from SHAKE256(seed|nonce_i). + * + * @spec{Implements @[FIPS204, Algorithm 31, RejBoundedPoly] (four-way + * batched).} + * + * @param[out] r0 Pointer to first output polynomial. + * @param[out] r1 Pointer to second output polynomial. + * @param[out] r2 Pointer to third output polynomial. + * @param[out] r3 Pointer to fourth output polynomial. + * @param[in] seed Byte array with seed of length MLDSA_CRHBYTES. + * @param nonce0 First nonce. + * @param nonce1 Second nonce. + * @param nonce2 Third nonce. + * @param nonce3 Fourth nonce. + */ +MLD_INTERNAL_API +void mld_poly_uniform_eta_4x(mld_poly *r0, mld_poly *r1, mld_poly *r2, + mld_poly *r3, const uint8_t seed[MLDSA_CRHBYTES], + uint8_t nonce0, uint8_t nonce1, uint8_t nonce2, + uint8_t nonce3) +__contract__( + requires(memory_no_alias(r0, sizeof(mld_poly))) + requires(memory_no_alias(r1, sizeof(mld_poly))) + requires(memory_no_alias(r2, sizeof(mld_poly))) + requires(memory_no_alias(r3, sizeof(mld_poly))) + requires(memory_no_alias(seed, MLDSA_CRHBYTES)) + assigns(memory_slice(r0, sizeof(mld_poly))) + assigns(memory_slice(r1, sizeof(mld_poly))) + assigns(memory_slice(r2, sizeof(mld_poly))) + assigns(memory_slice(r3, sizeof(mld_poly))) + ensures(array_abs_bound(r0->coeffs, 0, MLDSA_N, MLDSA_ETA + 1)) + ensures(array_abs_bound(r1->coeffs, 0, MLDSA_N, MLDSA_ETA + 1)) + ensures(array_abs_bound(r2->coeffs, 0, MLDSA_N, MLDSA_ETA + 1)) + ensures(array_abs_bound(r3->coeffs, 0, MLDSA_N, MLDSA_ETA + 1)) +); +#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY */ + +#if defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) +#define mld_poly_uniform_eta MLD_NAMESPACE_KL(poly_uniform_eta) +/** + * Sample polynomial with uniformly random coefficients in + * [-MLDSA_ETA, MLDSA_ETA] by performing rejection sampling on the output + * stream from SHAKE256(seed|nonce). + * + * @spec{Implements @[FIPS204, Algorithm 31, RejBoundedPoly].} + * + * @param[out] r Pointer to output polynomial. + * @param[in] seed Byte array with seed of length MLDSA_CRHBYTES. + * @param nonce Nonce. + */ +MLD_INTERNAL_API +void mld_poly_uniform_eta(mld_poly *r, const uint8_t seed[MLDSA_CRHBYTES], + uint8_t nonce) +__contract__( + requires(memory_no_alias(r, sizeof(mld_poly))) + requires(memory_no_alias(seed, MLDSA_CRHBYTES)) + assigns(memory_slice(r, sizeof(mld_poly))) + ensures(array_abs_bound(r->coeffs, 0, MLDSA_N, MLDSA_ETA + 1)) +); +#endif /* MLD_CONFIG_SERIAL_FIPS202_ONLY */ +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +#if MLD_CONFIG_PARAMETER_SET == 65 || \ + defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) || \ + defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +#define mld_poly_uniform_gamma1 MLD_NAMESPACE_KL(poly_uniform_gamma1) +/** + * Sample polynomial with uniformly random coefficients in + * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1] by unpacking output stream of + * SHAKE256(seed|nonce). + * + * @spec{Partially implements @[FIPS204, Algorithm 34, ExpandMask] (one + * polynomial, i.e. the loop body of lines 3-5).} + * + * @param[out] a Pointer to output polynomial. + * @param[in] seed Byte array with seed of length MLDSA_CRHBYTES. + * @param nonce 16-bit nonce. + */ +MLD_INTERNAL_API +void mld_poly_uniform_gamma1(mld_poly *a, const uint8_t seed[MLDSA_CRHBYTES], + uint16_t nonce) +__contract__( + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(memory_no_alias(seed, MLDSA_CRHBYTES)) + assigns(memory_slice(a, sizeof(mld_poly))) + ensures(array_bound(a->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) +); +#endif /* MLD_CONFIG_PARAMETER_SET == 65 || MLD_CONFIG_SERIAL_FIPS202_ONLY || \ + MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)) +#define mld_poly_uniform_gamma1_4x MLD_NAMESPACE_KL(poly_uniform_gamma1_4x) +/** + * Sample four polynomials with uniformly random coefficients in + * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1] by unpacking output streams of + * SHAKE256(seed|nonce_i). + * + * @spec{Partially implements @[FIPS204, Algorithm 34, ExpandMask] (four-way + * batched, i.e. four iterations of the loop body of lines 3-5).} + * + * @param[out] r0 Pointer to first output polynomial. + * @param[out] r1 Pointer to second output polynomial. + * @param[out] r2 Pointer to third output polynomial. + * @param[out] r3 Pointer to fourth output polynomial. + * @param[in] seed Byte array with seed of length MLDSA_CRHBYTES. + * @param nonce0 First 16-bit nonce. + * @param nonce1 Second 16-bit nonce. + * @param nonce2 Third 16-bit nonce. + * @param nonce3 Fourth 16-bit nonce. + */ +MLD_INTERNAL_API +void mld_poly_uniform_gamma1_4x(mld_poly *r0, mld_poly *r1, mld_poly *r2, + mld_poly *r3, + const uint8_t seed[MLDSA_CRHBYTES], + uint16_t nonce0, uint16_t nonce1, + uint16_t nonce2, uint16_t nonce3) +__contract__( + requires(memory_no_alias(r0, sizeof(mld_poly))) + requires(memory_no_alias(r1, sizeof(mld_poly))) + requires(memory_no_alias(r2, sizeof(mld_poly))) + requires(memory_no_alias(r3, sizeof(mld_poly))) + requires(memory_no_alias(seed, MLDSA_CRHBYTES)) + assigns(memory_slice(r0, sizeof(mld_poly))) + assigns(memory_slice(r1, sizeof(mld_poly))) + assigns(memory_slice(r2, sizeof(mld_poly))) + assigns(memory_slice(r3, sizeof(mld_poly))) + ensures(array_bound(r0->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) + ensures(array_bound(r1->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) + ensures(array_bound(r2->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) + ensures(array_bound(r3->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) +); +#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY && (!MLD_CONFIG_REDUCE_RAM || \ + MLD_UNIT_TEST) */ +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +#define mld_poly_challenge MLD_NAMESPACE_KL(poly_challenge) +/** + * Samples polynomial with MLDSA_TAU nonzero coefficients in {-1, 1} using the + * output stream of SHAKE256(seed). + * + * @spec{Implements @[FIPS204, Algorithm 29, SampleInBall].} + * + * @param[out] c Pointer to output polynomial. + * @param[in] seed Byte array containing seed of length MLDSA_CTILDEBYTES. + */ +MLD_INTERNAL_API +void mld_poly_challenge(mld_poly *c, const uint8_t seed[MLDSA_CTILDEBYTES]) +__contract__( + requires(memory_no_alias(c, sizeof(mld_poly))) + requires(memory_no_alias(seed, MLDSA_CTILDEBYTES)) + assigns(memory_slice(c, sizeof(mld_poly))) + /* All coefficients of c are -1, 0 or +1 */ + ensures(array_bound(c->coeffs, 0, MLDSA_N, -1, 2)) +); +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +#define mld_polyeta_pack MLD_NAMESPACE_KL(polyeta_pack) +/** + * Bit-pack polynomial with coefficients in [-MLDSA_ETA, MLDSA_ETA]. + * + * @spec{Implements @[FIPS204, Algorithm 17, BitPack].} + * + * @param[out] r Pointer to output byte array with at least + * MLDSA_POLYETA_PACKEDBYTES bytes. + * @param[in] a Pointer to input polynomial. + */ +MLD_INTERNAL_API +void mld_polyeta_pack(uint8_t r[MLDSA_POLYETA_PACKEDBYTES], const mld_poly *a) +__contract__( + requires(memory_no_alias(r, MLDSA_POLYETA_PACKEDBYTES)) + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(array_abs_bound(a->coeffs, 0, MLDSA_N, MLDSA_ETA + 1)) + assigns(memory_slice(r, MLDSA_POLYETA_PACKEDBYTES)) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_SIGN_API) +/* + * polyeta_unpack produces coefficients in [-MLDSA_ETA, MLDSA_ETA] for + * well-formed inputs (i.e., those produced by polyeta_pack). + * However, when passed an arbitrary byte array, it may produce smaller values, + * i.e., values in [MLD_POLYETA_UNPACK_LOWER_BOUND, MLDSA_ETA]. + * Even though this should never happen, we use use the bound for arbitrary + * inputs in the CBMC proofs. + */ +#if MLDSA_ETA == 2 +#define MLD_POLYETA_UNPACK_LOWER_BOUND (-5) +#elif MLDSA_ETA == 4 +#define MLD_POLYETA_UNPACK_LOWER_BOUND (-11) +#else +#error "Invalid value of MLDSA_ETA" +#endif + +#define mld_polyeta_unpack MLD_NAMESPACE_KL(polyeta_unpack) +/** + * Unpack polynomial with coefficients in [-MLDSA_ETA, MLDSA_ETA]. + * + * @spec{Implements @[FIPS204, Algorithm 19, BitUnpack].} + * + * @param[out] r Pointer to output polynomial. + * @param[in] a Byte array with bit-packed polynomial. + */ +MLD_INTERNAL_API +void mld_polyeta_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYETA_PACKEDBYTES]) +__contract__( + requires(memory_no_alias(r, sizeof(mld_poly))) + requires(memory_no_alias(a, MLDSA_POLYETA_PACKEDBYTES)) + assigns(memory_slice(r, sizeof(mld_poly))) + ensures(array_bound(r->coeffs, 0, MLDSA_N, MLD_POLYETA_UNPACK_LOWER_BOUND, MLDSA_ETA + 1)) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +#define mld_polyz_pack MLD_NAMESPACE_KL(polyz_pack) +/** + * Bit-pack polynomial with coefficients in + * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1]. + * + * @spec{Implements @[FIPS204, Algorithm 17, BitPack].} + * + * @param[out] r Pointer to output byte array with at least + * MLDSA_POLYZ_PACKEDBYTES bytes. + * @param[in] a Pointer to input polynomial. + */ +MLD_INTERNAL_API +void mld_polyz_pack(uint8_t r[MLDSA_POLYZ_PACKEDBYTES], const mld_poly *a) +__contract__( + requires(memory_no_alias(r, MLDSA_POLYZ_PACKEDBYTES)) + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(array_bound(a->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) + assigns(memory_slice(r, MLDSA_POLYZ_PACKEDBYTES)) +); +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +#define mld_polyz_unpack MLD_NAMESPACE_KL(polyz_unpack) +/** + * Unpack polynomial z with coefficients in + * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1]. + * + * @spec{Implements @[FIPS204, Algorithm 19, BitUnpack].} + * + * @param[out] r Pointer to output polynomial. + * @param[in] a Byte array with bit-packed polynomial. + */ +MLD_INTERNAL_API +void mld_polyz_unpack(mld_poly *r, const uint8_t a[MLDSA_POLYZ_PACKEDBYTES]) +__contract__( + requires(memory_no_alias(r, sizeof(mld_poly))) + requires(memory_no_alias(a, MLDSA_POLYZ_PACKEDBYTES)) + assigns(memory_slice(r, sizeof(mld_poly))) + ensures(array_bound(r->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) +); + +#define mld_polyw1_pack MLD_NAMESPACE_KL(polyw1_pack) +/** + * Bit-pack polynomial w1. Input coefficients must be in + * [0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)), i.e. [0, 43] for ML-DSA-44 and [0, 15] + * for ML-DSA-65/87. Dispatches to the value-specialized variant for the + * selected parameter set. + * + * @spec{Implements @[FIPS204, Algorithm 16, SimpleBitPack].} + * + * @param[out] r Pointer to output byte array with at least + * MLDSA_POLYW1_PACKEDBYTES bytes. + * @param[in] a Pointer to input polynomial. + */ +static MLD_INLINE void mld_polyw1_pack(uint8_t r[MLDSA_POLYW1_PACKEDBYTES], + const mld_poly *a) +__contract__( + requires(memory_no_alias(r, MLDSA_POLYW1_PACKEDBYTES)) + requires(memory_no_alias(a, sizeof(mld_poly))) + requires(array_bound(a->coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2))) + assigns(memory_slice(r, MLDSA_POLYW1_PACKEDBYTES)) +) +{ +#if MLD_CONFIG_PARAMETER_SET == 44 + mld_polyw1_pack_88(r, a); +#else + mld_polyw1_pack_32(r, a); +#endif +} +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#endif /* !MLD_POLY_KL_H */ diff --git a/cbits/mldsa/src/polyvec.c b/cbits/mldsa/src/polyvec.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/polyvec.c @@ -0,0 +1,509 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + */ + +#include "polyvec.h" + +#include "debug.h" +#include "polyvec_lazy.h" + +/* This namespacing is not done at the top to avoid a naming conflict + * with native backends, which are currently not yet namespaced. */ +#define mld_polyvecl_pointwise_acc_montgomery_c \ + MLD_ADD_PARAM_SET(mld_polyvecl_pointwise_acc_montgomery_c) + +/**************************************************************/ +/************ Vectors of polynomials of length MLDSA_L **************/ +/**************************************************************/ +#if !defined(MLD_CONFIG_NO_SIGN_API) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)) +MLD_INTERNAL_API +void mld_polyvecl_uniform_gamma1(mld_polyvecl *v, + const uint8_t seed[MLDSA_CRHBYTES], + uint16_t kappa) +{ +#if defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) + int i; +#endif + + /* The caller passes the base counter kappa; component i is sampled from + * kappa + i. Safety: kappa <= MLD_MAX_KAPPA and i < MLDSA_L, so the + * casts below are safe. See MLD_MAX_KAPPA comment in params.h. */ +#if defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) + for (i = 0; i < MLDSA_L; i++) + { + mld_poly_uniform_gamma1(&v->vec[i], seed, (uint16_t)(kappa + i)); + } +#else /* MLD_CONFIG_SERIAL_FIPS202_ONLY */ +#if MLDSA_L == 4 + mld_poly_uniform_gamma1_4x(&v->vec[0], &v->vec[1], &v->vec[2], &v->vec[3], + seed, kappa, (uint16_t)(kappa + 1), + (uint16_t)(kappa + 2), (uint16_t)(kappa + 3)); +#elif MLDSA_L == 5 + mld_poly_uniform_gamma1_4x(&v->vec[0], &v->vec[1], &v->vec[2], &v->vec[3], + seed, kappa, (uint16_t)(kappa + 1), + (uint16_t)(kappa + 2), (uint16_t)(kappa + 3)); + mld_poly_uniform_gamma1(&v->vec[4], seed, (uint16_t)(kappa + 4)); +#elif MLDSA_L == 7 + mld_poly_uniform_gamma1_4x(&v->vec[0], &v->vec[1], &v->vec[2], + &v->vec[3 /* irrelevant */], seed, kappa, + (uint16_t)(kappa + 1), (uint16_t)(kappa + 2), + 0xFF /* irrelevant */); + mld_poly_uniform_gamma1_4x(&v->vec[3], &v->vec[4], &v->vec[5], &v->vec[6], + seed, (uint16_t)(kappa + 3), (uint16_t)(kappa + 4), + (uint16_t)(kappa + 5), (uint16_t)(kappa + 6)); +#endif /* MLDSA_L == 7 */ +#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY */ + + mld_assert_bound_2d(v->vec, MLDSA_L, MLDSA_N, -(MLDSA_GAMMA1 - 1), + MLDSA_GAMMA1 + 1); +} +#endif /* !MLD_CONFIG_NO_SIGN_API && (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) \ + */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || \ + !defined(MLD_CONFIG_NO_VERIFY_API) || \ + (!defined(MLD_CONFIG_NO_SIGN_API) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))) +MLD_INTERNAL_API +void mld_polyvecl_ntt(mld_polyvecl *v) +{ + unsigned int i; + mld_assert_abs_bound_2d(v->vec, MLDSA_L, MLDSA_N, MLDSA_Q); + + for (i = 0; i < MLDSA_L; ++i) + __loop__( + assigns(i, memory_slice(v, sizeof(mld_polyvecl))) + invariant(i <= MLDSA_L) + invariant(forall(k0, i, MLDSA_L, forall(k1, 0, MLDSA_N, v->vec[k0].coeffs[k1] == loop_entry(*v).vec[k0].coeffs[k1]))) + invariant(forall(k1, 0, i, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + decreases(MLDSA_L - i)) + { + mld_poly_ntt(&v->vec[i]); + } + + mld_assert_abs_bound_2d(v->vec, MLDSA_L, MLDSA_N, MLD_NTT_BOUND); +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_VERIFY_API || \ + (!MLD_CONFIG_NO_SIGN_API && (!MLD_CONFIG_REDUCE_RAM || \ + MLD_UNIT_TEST)) */ + +#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +MLD_STATIC_TESTABLE void mld_polyvecl_pointwise_acc_montgomery_c( + mld_poly *w, const mld_polyvecl *u, const mld_polyvecl *v) +__contract__( + requires(memory_no_alias(w, sizeof(mld_poly))) + requires(memory_no_alias(u, sizeof(mld_polyvecl))) + requires(memory_no_alias(v, sizeof(mld_polyvecl))) + requires(forall(l0, 0, MLDSA_L, + array_bound(u->vec[l0].coeffs, 0, MLDSA_N, 0, MLDSA_Q))) + requires(forall(l1, 0, MLDSA_L, + array_abs_bound(v->vec[l1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + assigns(memory_slice(w, sizeof(mld_poly))) + ensures(array_abs_bound(w->coeffs, 0, MLDSA_N, MLDSA_Q)) +) +{ + unsigned int i, j; + mld_assert_bound_2d(u->vec, MLDSA_L, MLDSA_N, 0, MLDSA_Q); + mld_assert_abs_bound_2d(v->vec, MLDSA_L, MLDSA_N, MLD_NTT_BOUND); + for (i = 0; i < MLDSA_N; i++) + __loop__( + assigns(i, j, memory_slice(w, sizeof(mld_poly))) + invariant(i <= MLDSA_N) + invariant(array_abs_bound(w->coeffs, 0, i, MLDSA_Q)) + decreases(MLDSA_N - i) + ) + { + int64_t t = 0; + int32_t r; + for (j = 0; j < MLDSA_L; j++) + __loop__( + assigns(j, t) + invariant(j <= MLDSA_L) + invariant(t >= -(int64_t)j*(MLDSA_Q - 1)*(MLD_NTT_BOUND - 1)) + invariant(t <= (int64_t)j*(MLDSA_Q - 1)*(MLD_NTT_BOUND - 1)) + decreases(MLDSA_L - j) + ) + { + t += (int64_t)u->vec[j].coeffs[i] * v->vec[j].coeffs[i]; + } + + r = mld_montgomery_reduce(t); + w->coeffs[i] = r; + } + + mld_assert_abs_bound(w->coeffs, MLDSA_N, MLDSA_Q); +} + +MLD_INTERNAL_API +void mld_polyvecl_pointwise_acc_montgomery(mld_poly *w, const mld_polyvecl *u, + const mld_polyvecl *v) +{ +#if defined(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4) && \ + MLD_CONFIG_PARAMETER_SET == 44 + int ret; + mld_assert_bound_2d(u->vec, MLDSA_L, MLDSA_N, 0, MLDSA_Q); + mld_assert_abs_bound_2d(v->vec, MLDSA_L, MLDSA_N, MLD_NTT_BOUND); + ret = mld_polyvecl_pointwise_acc_montgomery_l4_native( + w->coeffs, (const int32_t (*)[MLDSA_N])u->vec, + (const int32_t (*)[MLDSA_N])v->vec); + if (ret == MLD_NATIVE_FUNC_SUCCESS) + { + mld_assert_abs_bound(w->coeffs, MLDSA_N, MLDSA_Q); + return; + } +#elif defined(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5) && \ + MLD_CONFIG_PARAMETER_SET == 65 + int ret; + mld_assert_bound_2d(u->vec, MLDSA_L, MLDSA_N, 0, MLDSA_Q); + mld_assert_abs_bound_2d(v->vec, MLDSA_L, MLDSA_N, MLD_NTT_BOUND); + ret = mld_polyvecl_pointwise_acc_montgomery_l5_native( + w->coeffs, (const int32_t (*)[MLDSA_N])u->vec, + (const int32_t (*)[MLDSA_N])v->vec); + if (ret == MLD_NATIVE_FUNC_SUCCESS) + { + mld_assert_abs_bound(w->coeffs, MLDSA_N, MLDSA_Q); + return; + } +#elif defined(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7) && \ + MLD_CONFIG_PARAMETER_SET == 87 + int ret; + mld_assert_bound_2d(u->vec, MLDSA_L, MLDSA_N, 0, MLDSA_Q); + mld_assert_abs_bound_2d(v->vec, MLDSA_L, MLDSA_N, MLD_NTT_BOUND); + ret = mld_polyvecl_pointwise_acc_montgomery_l7_native( + w->coeffs, (const int32_t (*)[MLDSA_N])u->vec, + (const int32_t (*)[MLDSA_N])v->vec); + if (ret == MLD_NATIVE_FUNC_SUCCESS) + { + mld_assert_abs_bound(w->coeffs, MLDSA_N, MLDSA_Q); + return; + } +#endif /* !(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L4 && \ + MLD_CONFIG_PARAMETER_SET == 44) && \ + !(MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L5 && \ + MLD_CONFIG_PARAMETER_SET == 65) && \ + MLD_USE_NATIVE_POLYVECL_POINTWISE_ACC_MONTGOMERY_L7 && \ + MLD_CONFIG_PARAMETER_SET == 87 */ + /* The first input is bounded by [0, MLDSA_Q-1] inclusive. + * The second input is bounded by [-(MLD_NTT_BOUND-1), MLD_NTT_BOUND-1]. + * Hence, we can safely accumulate in 64-bits without intermediate reductions + * as MLDSA_L * (MLD_NTT_BOUND-1) * (MLDSA_Q-1) < INT64_MAX. + * + * The worst case is ML-DSA-87: 7 * (MLD_NTT_BOUND-1) * (MLDSA_Q-1) < 2**53 + * (and likewise for negative values). + */ + mld_polyvecl_pointwise_acc_montgomery_c(w, u, v); +} +#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_VERIFY_API) || \ + defined(MLD_UNIT_TEST) +MLD_INTERNAL_API +uint32_t mld_polyvecl_chknorm(const mld_polyvecl *v, int32_t bound) +{ + unsigned int i; + uint32_t t = 0; + mld_assert_bound_2d(v->vec, MLDSA_L, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, + MLD_REDUCE32_RANGE_MAX); + + for (i = 0; i < MLDSA_L; ++i) + __loop__( + invariant(i <= MLDSA_L) + invariant(t == 0 || t == 0xFFFFFFFF) + invariant((t == 0) == forall(k1, 0, i, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, bound))) + decreases(MLDSA_L - i) + ) + { + /* Reference: Leaks which polynomial violates the bound via a conditional. + * We are more conservative to reduce the number of declassifications in + * constant-time testing. + */ + t |= mld_poly_chknorm(&v->vec[i], bound); + } + return t; +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_VERIFY_API || \ + MLD_UNIT_TEST */ + +/**************************************************************/ +/************ Vectors of polynomials of length MLDSA_K **************/ +/**************************************************************/ +#if (!defined(MLD_CONFIG_NO_SIGN_API) && \ + defined(MLD_CONFIG_REDUCE_RAM)) || \ + defined(MLD_UNIT_TEST) +MLD_INTERNAL_API +void mld_polyveck_reduce(mld_polyveck *v) +{ + unsigned int i; + mld_assert_bound_2d(v->vec, MLDSA_K, MLDSA_N, INT32_MIN, + MLD_REDUCE32_DOMAIN_MAX); + + for (i = 0; i < MLDSA_K; ++i) + __loop__( + assigns(i, memory_slice(v, sizeof(mld_polyveck))) + invariant(i <= MLDSA_K) + invariant(forall(k0, i, MLDSA_K, forall(k1, 0, MLDSA_N, v->vec[k0].coeffs[k1] == loop_entry(*v).vec[k0].coeffs[k1]))) + invariant(forall(k2, 0, i, + array_bound(v->vec[k2].coeffs, 0, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX))) + decreases(MLDSA_K - i) + ) + { + mld_poly_reduce(&v->vec[i]); + } + + mld_assert_bound_2d(v->vec, MLDSA_K, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, + MLD_REDUCE32_RANGE_MAX); +} +#endif /* (!MLD_CONFIG_NO_SIGN_API && MLD_CONFIG_REDUCE_RAM) || MLD_UNIT_TEST \ + */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST) +MLD_INTERNAL_API +void mld_polyveck_caddq(mld_polyveck *v) +{ + unsigned int i; + mld_assert_abs_bound_2d(v->vec, MLDSA_K, MLDSA_N, MLDSA_Q); + + for (i = 0; i < MLDSA_K; ++i) + __loop__( + assigns(i, memory_slice(v, sizeof(mld_polyveck))) + invariant(i <= MLDSA_K) + invariant(forall(k0, i, MLDSA_K, forall(k1, 0, MLDSA_N, v->vec[k0].coeffs[k1] == loop_entry(*v).vec[k0].coeffs[k1]))) + invariant(forall(k1, 0, i, array_bound(v->vec[k1].coeffs, 0, MLDSA_N, 0, MLDSA_Q))) + decreases(MLDSA_K - i)) + { + mld_poly_caddq(&v->vec[i]); + } + + mld_assert_bound_2d(v->vec, MLDSA_K, MLDSA_N, 0, MLDSA_Q); +} +#endif /* !MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST */ + +#if (!defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST)) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)) +MLD_INTERNAL_API +void mld_polyveck_ntt(mld_polyveck *v) +{ + unsigned int i; + mld_assert_abs_bound_2d(v->vec, MLDSA_K, MLDSA_N, MLDSA_Q); + + for (i = 0; i < MLDSA_K; ++i) + __loop__( + assigns(i, memory_slice(v, sizeof(mld_polyveck))) + invariant(i <= MLDSA_K) + invariant(forall(k0, i, MLDSA_K, forall(k1, 0, MLDSA_N, v->vec[k0].coeffs[k1] == loop_entry(*v).vec[k0].coeffs[k1]))) + invariant(forall(k1, 0, i, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + decreases(MLDSA_K - i)) + { + mld_poly_ntt(&v->vec[i]); + } + mld_assert_abs_bound_2d(v->vec, MLDSA_K, MLDSA_N, MLD_NTT_BOUND); +} +#endif /* (!MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST) && \ + (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST) +MLD_INTERNAL_API +void mld_polyveck_invntt_tomont(mld_polyveck *v) +{ + unsigned int i; + mld_assert_abs_bound_2d(v->vec, MLDSA_K, MLDSA_N, MLDSA_Q); + + for (i = 0; i < MLDSA_K; ++i) + __loop__( + assigns(i, memory_slice(v, sizeof(mld_polyveck))) + invariant(i <= MLDSA_K) + invariant(forall(k0, i, MLDSA_K, forall(k1, 0, MLDSA_N, v->vec[k0].coeffs[k1] == loop_entry(*v).vec[k0].coeffs[k1]))) + invariant(forall(k1, 0, i, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, MLD_INTT_BOUND))) + decreases(MLDSA_K - i)) + { + mld_poly_invntt_tomont(&v->vec[i]); + } + + mld_assert_abs_bound_2d(v->vec, MLDSA_K, MLDSA_N, MLD_INTT_BOUND); +} +#endif /* !MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +MLD_INTERNAL_API +uint32_t mld_polyveck_chknorm(const mld_polyveck *v, int32_t bound) +{ + unsigned int i; + uint32_t t = 0; + mld_assert_bound_2d(v->vec, MLDSA_K, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, + MLD_REDUCE32_RANGE_MAX); + + for (i = 0; i < MLDSA_K; ++i) + __loop__( + invariant(i <= MLDSA_K) + invariant(t == 0 || t == 0xFFFFFFFF) + invariant((t == 0) == forall(k1, 0, i, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, bound))) + decreases(MLDSA_K - i) + ) + { + /* Reference: Leaks which polynomial violates the bound via a conditional. + * We are more conservative to reduce the number of declassifications in + * constant-time testing. + */ + t |= mld_poly_chknorm(&v->vec[i], bound); + } + + return t; +} + +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +MLD_INTERNAL_API +void mld_polyveck_decompose(mld_polyveck *v1, mld_polyveck *v0) +{ + unsigned int i; + mld_assert_bound_2d(v0->vec, MLDSA_K, MLDSA_N, 0, MLDSA_Q); + + for (i = 0; i < MLDSA_K; ++i) + __loop__( + assigns(i, memory_slice(v0, sizeof(mld_polyveck)), memory_slice(v1, sizeof(mld_polyveck))) + invariant(i <= MLDSA_K) + invariant(forall(k1, 0, i, + array_bound(v1->vec[k1].coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)))) + invariant(forall(k2, 0, i, + array_abs_bound(v0->vec[k2].coeffs, 0, MLDSA_N, MLDSA_GAMMA2+1))) + invariant(forall(k3, i, MLDSA_K, + array_bound(v0->vec[k3].coeffs, 0, MLDSA_N, 0, MLDSA_Q))) + decreases(MLDSA_K - i) + ) + { + mld_poly_decompose(&v1->vec[i], &v0->vec[i]); + } + + mld_assert_bound_2d(v1->vec, MLDSA_K, MLDSA_N, 0, + (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2)); + mld_assert_abs_bound_2d(v0->vec, MLDSA_K, MLDSA_N, MLDSA_GAMMA2 + 1); +} +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +MLD_INTERNAL_API +void mld_polyveck_pack_w1(uint8_t r[MLDSA_K * MLDSA_POLYW1_PACKEDBYTES], + const mld_polyveck *w1) +{ + unsigned int i; + mld_assert_bound_2d(w1->vec, MLDSA_K, MLDSA_N, 0, + (MLDSA_Q - 1) / (2 * MLDSA_GAMMA2)); + + for (i = 0; i < MLDSA_K; ++i) + __loop__( + assigns(i, memory_slice(r, MLDSA_K * MLDSA_POLYW1_PACKEDBYTES)) + invariant(i <= MLDSA_K) + decreases(MLDSA_K - i) + ) + { + mld_polyw1_pack(&r[i * MLDSA_POLYW1_PACKEDBYTES], &w1->vec[i]); + } +} +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +MLD_INTERNAL_API +void mld_polyveck_pack_eta(uint8_t r[MLDSA_K * MLDSA_POLYETA_PACKEDBYTES], + const mld_polyveck *p) +{ + unsigned int i; + mld_assert_abs_bound_2d(p->vec, MLDSA_K, MLDSA_N, MLDSA_ETA + 1); + for (i = 0; i < MLDSA_K; ++i) + __loop__( + assigns(i, memory_slice(r, MLDSA_K * MLDSA_POLYETA_PACKEDBYTES)) + invariant(i <= MLDSA_K) + decreases(MLDSA_K - i) + ) + { + mld_polyeta_pack(&r[i * MLDSA_POLYETA_PACKEDBYTES], &p->vec[i]); + } +} + +MLD_INTERNAL_API +void mld_polyvecl_pack_eta(uint8_t r[MLDSA_L * MLDSA_POLYETA_PACKEDBYTES], + const mld_polyvecl *p) +{ + unsigned int i; + mld_assert_abs_bound_2d(p->vec, MLDSA_L, MLDSA_N, MLDSA_ETA + 1); + for (i = 0; i < MLDSA_L; ++i) + __loop__( + assigns(i, memory_slice(r, MLDSA_L * MLDSA_POLYETA_PACKEDBYTES)) + invariant(i <= MLDSA_L) + decreases(MLDSA_L - i) + ) + { + mld_polyeta_pack(&r[i * MLDSA_POLYETA_PACKEDBYTES], &p->vec[i]); + } +} + +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || \ + (!defined(MLD_CONFIG_NO_SIGN_API) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))) +MLD_INTERNAL_API +void mld_polyvecl_unpack_eta( + mld_polyvecl *p, const uint8_t r[MLDSA_L * MLDSA_POLYETA_PACKEDBYTES]) +{ + unsigned int i; + for (i = 0; i < MLDSA_L; ++i) + { + mld_polyeta_unpack(&p->vec[i], r + i * MLDSA_POLYETA_PACKEDBYTES); + } + + mld_assert_bound_2d(p->vec, MLDSA_L, MLDSA_N, MLD_POLYETA_UNPACK_LOWER_BOUND, + MLDSA_ETA + 1); +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || (!MLD_CONFIG_NO_SIGN_API && \ + (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST)) */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +MLD_INTERNAL_API +void mld_polyvecl_unpack_z(mld_polyvecl *z, + const uint8_t r[MLDSA_L * MLDSA_POLYZ_PACKEDBYTES]) +{ + unsigned int i; + for (i = 0; i < MLDSA_L; ++i) + { + mld_polyz_unpack(&z->vec[i], r + i * MLDSA_POLYZ_PACKEDBYTES); + } + + mld_assert_bound_2d(z->vec, MLDSA_L, MLDSA_N, -(MLDSA_GAMMA1 - 1), + MLDSA_GAMMA1 + 1); +} +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || \ + (!defined(MLD_CONFIG_NO_SIGN_API) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))) +MLD_INTERNAL_API +void mld_polyveck_unpack_eta( + mld_polyveck *p, const uint8_t r[MLDSA_K * MLDSA_POLYETA_PACKEDBYTES]) +{ + unsigned int i; + for (i = 0; i < MLDSA_K; ++i) + { + mld_polyeta_unpack(&p->vec[i], r + i * MLDSA_POLYETA_PACKEDBYTES); + } + + mld_assert_bound_2d(p->vec, MLDSA_K, MLDSA_N, MLD_POLYETA_UNPACK_LOWER_BOUND, + MLDSA_ETA + 1); +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || (!MLD_CONFIG_NO_SIGN_API && \ + (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST)) */ + +/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. + * Don't modify by hand -- this is auto-generated by scripts/autogen. */ +#undef mld_polyvecl_pointwise_acc_montgomery_c diff --git a/cbits/mldsa/src/polyvec.h b/cbits/mldsa/src/polyvec.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/polyvec.h @@ -0,0 +1,435 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + */ + +#ifndef MLD_POLYVEC_H +#define MLD_POLYVEC_H + +#include "cbmc.h" +#include "common.h" +#include "poly.h" +#include "poly_kl.h" + +/* Parameter set namespacing + * This is to facilitate building multiple instances + * of mldsa-native (e.g. with varying parameter sets) + * within a single compilation unit. */ +#define mld_polyvecl MLD_ADD_PARAM_SET(mld_polyvecl) +#define mld_polyveck MLD_ADD_PARAM_SET(mld_polyveck) +/* End of parameter set namespacing */ + +/** Vector of MLDSA_L polynomials. */ +typedef struct +{ + mld_poly vec[MLDSA_L]; /**< Component polynomials. */ +} mld_polyvecl; + + +#if !defined(MLD_CONFIG_NO_SIGN_API) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)) +#define mld_polyvecl_uniform_gamma1 MLD_NAMESPACE_KL(polyvecl_uniform_gamma1) +/** + * Sample vector of polynomials with uniformly random coefficients in + * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1] by unpacking output stream of + * SHAKE256(seed|kappa+i) for component i. + * + * @spec{Implements @[FIPS204, Algorithm 34, ExpandMask].} + * + * @param[out] v Pointer to output vector. + * @param[in] seed Byte array with seed of length MLDSA_CRHBYTES. + * @param kappa Base counter; component i uses kappa + i. + */ +MLD_INTERNAL_API +void mld_polyvecl_uniform_gamma1(mld_polyvecl *v, + const uint8_t seed[MLDSA_CRHBYTES], + uint16_t kappa) +__contract__( + requires(memory_no_alias(v, sizeof(mld_polyvecl))) + requires(memory_no_alias(seed, MLDSA_CRHBYTES)) + requires(kappa <= MLD_MAX_KAPPA) + assigns(memory_slice(v, sizeof(mld_polyvecl))) + ensures(forall(k0, 0, MLDSA_L, + array_bound(v->vec[k0].coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))) +); +#endif /* !MLD_CONFIG_NO_SIGN_API && (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) \ + */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || \ + !defined(MLD_CONFIG_NO_VERIFY_API) || \ + (!defined(MLD_CONFIG_NO_SIGN_API) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))) +#define mld_polyvecl_ntt MLD_NAMESPACE_KL(polyvecl_ntt) +/** + * Forward NTT of all polynomials in vector of length MLDSA_L. Output + * coefficients are bounded by MLD_NTT_BOUND in absolute value. + * + * @param[in,out] v Pointer to input/output vector. + */ +MLD_INTERNAL_API +void mld_polyvecl_ntt(mld_polyvecl *v) +__contract__( + requires(memory_no_alias(v, sizeof(mld_polyvecl))) + requires(forall(k0, 0, MLDSA_L, array_abs_bound(v->vec[k0].coeffs, 0, MLDSA_N, MLDSA_Q))) + assigns(memory_slice(v, sizeof(mld_polyvecl))) + ensures(forall(k1, 0, MLDSA_L, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_VERIFY_API || \ + (!MLD_CONFIG_NO_SIGN_API && (!MLD_CONFIG_REDUCE_RAM || \ + MLD_UNIT_TEST)) */ + +#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +#define mld_polyvecl_pointwise_acc_montgomery \ + MLD_NAMESPACE_KL(polyvecl_pointwise_acc_montgomery) +/** + * Pointwise multiply vectors of polynomials of length MLDSA_L, multiply + * resulting vector by 2^{-32} and add (accumulate) polynomials in it. + * Input/output vectors are in NTT domain representation. + * + * The first input "u" must be the output of polyvec_matrix_expand() and so + * have coefficients in [0, MLDSA_Q-1] inclusive. + * + * The second input "v" is assumed to be output of an NTT, and hence must have + * coefficients bounded by [-(MLD_NTT_BOUND-1), MLD_NTT_BOUND-1] inclusive. + * + * @spec{Partially implements @[FIPS204, Algorithm 48, MatrixVectorNTT] + * (one output polynomial; multiply-accumulate of two NTT-domain vectors).} + * + * @param[out] w Output polynomial. + * @param[in] u Pointer to first input vector. + * @param[in] v Pointer to second input vector. + */ +MLD_INTERNAL_API +void mld_polyvecl_pointwise_acc_montgomery(mld_poly *w, const mld_polyvecl *u, + const mld_polyvecl *v) +__contract__( + requires(memory_no_alias(w, sizeof(mld_poly))) + requires(memory_no_alias(u, sizeof(mld_polyvecl))) + requires(memory_no_alias(v, sizeof(mld_polyvecl))) + requires(forall(l0, 0, MLDSA_L, + array_bound(u->vec[l0].coeffs, 0, MLDSA_N, 0, MLDSA_Q))) + requires(forall(l1, 0, MLDSA_L, + array_abs_bound(v->vec[l1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + assigns(memory_slice(w, sizeof(mld_poly))) + ensures(array_abs_bound(w->coeffs, 0, MLDSA_N, MLDSA_Q)) +); +#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +#define mld_polyvecl_chknorm MLD_NAMESPACE_KL(polyvecl_chknorm) +/** + * Check infinity norm of polynomials in vector of length MLDSA_L. Assumes + * input mld_polyvecl to be reduced by polyvecl_reduce(). + * + * @param[in] v Pointer to vector. + * @param B Norm bound. + * + * @return 0 if norm of all polynomials is strictly smaller than + * B <= (MLDSA_Q-1)/8 and 0xFFFFFFFF otherwise. + */ +MLD_INTERNAL_API +MLD_MUST_CHECK_RETURN_VALUE +uint32_t mld_polyvecl_chknorm(const mld_polyvecl *v, int32_t B) +__contract__( + requires(memory_no_alias(v, sizeof(mld_polyvecl))) + requires(0 <= B && B <= (MLDSA_Q - 1) / 8) + requires(forall(k0, 0, MLDSA_L, + array_bound(v->vec[k0].coeffs, 0, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX))) + ensures(return_value == 0 || return_value == 0xFFFFFFFF) + ensures((return_value == 0) == forall(k1, 0, MLDSA_L, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, B))) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_VERIFY_API */ + +/** Vector of MLDSA_K polynomials. */ +typedef struct +{ + mld_poly vec[MLDSA_K]; /**< Component polynomials. */ +} mld_polyveck; + +#if (!defined(MLD_CONFIG_NO_SIGN_API) && defined(MLD_CONFIG_REDUCE_RAM)) || \ + defined(MLD_UNIT_TEST) +#define mld_polyveck_reduce MLD_NAMESPACE_KL(polyveck_reduce) +/** + * Reduce coefficients of polynomials in vector of length MLDSA_K to + * representatives in [-MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX]. + * + * @param[in,out] v Pointer to input/output vector. + */ +MLD_INTERNAL_API +void mld_polyveck_reduce(mld_polyveck *v) +__contract__( + requires(memory_no_alias(v, sizeof(mld_polyveck))) + requires(forall(k0, 0, MLDSA_K, + array_bound(v->vec[k0].coeffs, 0, MLDSA_N, INT32_MIN, MLD_REDUCE32_DOMAIN_MAX))) + assigns(memory_slice(v, sizeof(mld_polyveck))) + ensures(forall(k1, 0, MLDSA_K, + array_bound(v->vec[k1].coeffs, 0, MLDSA_N, -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX))) +); +#endif /* (!MLD_CONFIG_NO_SIGN_API && MLD_CONFIG_REDUCE_RAM) || MLD_UNIT_TEST \ + */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST) +#define mld_polyveck_caddq MLD_NAMESPACE_KL(polyveck_caddq) +/** + * For all coefficients of polynomials in vector of length MLDSA_K add MLDSA_Q + * if coefficient is negative. + * + * @param[in,out] v Pointer to input/output vector. + */ +MLD_INTERNAL_API +void mld_polyveck_caddq(mld_polyveck *v) +__contract__( + requires(memory_no_alias(v, sizeof(mld_polyveck))) + requires(forall(k0, 0, MLDSA_K, + array_abs_bound(v->vec[k0].coeffs, 0, MLDSA_N, MLDSA_Q))) + assigns(memory_slice(v, sizeof(mld_polyveck))) + ensures(forall(k1, 0, MLDSA_K, + array_bound(v->vec[k1].coeffs, 0, MLDSA_N, 0, MLDSA_Q))) +); +#endif /* !MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST */ + +#if (!defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST)) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)) +#define mld_polyveck_ntt MLD_NAMESPACE_KL(polyveck_ntt) +/** + * Forward NTT of all polynomials in vector of length MLDSA_K. Output + * coefficients are bounded by MLD_NTT_BOUND in absolute value. + * + * @param[in,out] v Pointer to input/output vector. + */ +MLD_INTERNAL_API +void mld_polyveck_ntt(mld_polyveck *v) +__contract__( + requires(memory_no_alias(v, sizeof(mld_polyveck))) + requires(forall(k0, 0, MLDSA_K, array_abs_bound(v->vec[k0].coeffs, 0, MLDSA_N, MLDSA_Q))) + assigns(memory_slice(v, sizeof(mld_polyveck))) + ensures(forall(k1, 0, MLDSA_K, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) +); +#endif /* (!MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST) && \ + (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST) +#define mld_polyveck_invntt_tomont MLD_NAMESPACE_KL(polyveck_invntt_tomont) +/** + * Inverse NTT and multiplication by 2^{32} of polynomials in vector of + * length MLDSA_K. + * + * Input coefficients need to be less than MLDSA_Q, and output coefficients + * are bounded by MLD_INTT_BOUND. + * + * @param[in,out] v Pointer to input/output vector. + */ +MLD_INTERNAL_API +void mld_polyveck_invntt_tomont(mld_polyveck *v) +__contract__( + requires(memory_no_alias(v, sizeof(mld_polyveck))) + requires(forall(k0, 0, MLDSA_K, array_abs_bound(v->vec[k0].coeffs, 0, MLDSA_N, MLDSA_Q))) + assigns(memory_slice(v, sizeof(mld_polyveck))) + ensures(forall(k1, 0, MLDSA_K, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, MLD_INTT_BOUND))) +); +#endif /* !MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +#define mld_polyveck_chknorm MLD_NAMESPACE_KL(polyveck_chknorm) +/** + * Check infinity norm of polynomials in vector of length MLDSA_K. Assumes + * input mld_polyveck to be reduced by polyveck_reduce(). + * + * @param[in] v Pointer to vector. + * @param B Norm bound. + * + * @return 0 if norm of all polynomials are strictly smaller than + * B <= (MLDSA_Q-1)/8 and 0xFFFFFFFF otherwise. + */ +MLD_INTERNAL_API +MLD_MUST_CHECK_RETURN_VALUE +uint32_t mld_polyveck_chknorm(const mld_polyveck *v, int32_t B) +__contract__( + requires(memory_no_alias(v, sizeof(mld_polyveck))) + requires(0 <= B && B <= (MLDSA_Q - 1) / 8) + requires(forall(k0, 0, MLDSA_K, + array_bound(v->vec[k0].coeffs, 0, MLDSA_N, + -MLD_REDUCE32_RANGE_MAX, MLD_REDUCE32_RANGE_MAX))) + ensures(return_value == 0 || return_value == 0xFFFFFFFF) + ensures((return_value == 0) == forall(k1, 0, MLDSA_K, array_abs_bound(v->vec[k1].coeffs, 0, MLDSA_N, B))) +); + +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +#define mld_polyveck_decompose MLD_NAMESPACE_KL(polyveck_decompose) +/** + * For all coefficients a of polynomials in vector of length MLDSA_K, compute + * high and low bits a0, a1 such a mod^+ MLDSA_Q = a1*ALPHA + a0 with + * -ALPHA/2 < a0 <= ALPHA/2 except a1 = (MLDSA_Q-1)/ALPHA where we set + * a1 = 0 and -ALPHA/2 <= a0 = a mod MLDSA_Q - MLDSA_Q < 0. Assumes + * coefficients to be standard representatives. + * + * @reference{The reference implementation has the input polynomial as a + * separate argument that may be aliased with either of the outputs. Removing + * the aliasing eases CBMC proofs.} + * + * @param[out] v1 Pointer to output vector of polynomials with + * coefficients a1. + * @param[in,out] v0 Pointer to input/output vector of polynomials. Output + * polynomial has coefficients a0. + */ +MLD_INTERNAL_API +void mld_polyveck_decompose(mld_polyveck *v1, mld_polyveck *v0) +__contract__( + requires(memory_no_alias(v1, sizeof(mld_polyveck))) + requires(memory_no_alias(v0, sizeof(mld_polyveck))) + requires(forall(k0, 0, MLDSA_K, + array_bound(v0->vec[k0].coeffs, 0, MLDSA_N, 0, MLDSA_Q))) + assigns(memory_slice(v1, sizeof(mld_polyveck))) + assigns(memory_slice(v0, sizeof(mld_polyveck))) + ensures(forall(k1, 0, MLDSA_K, + array_bound(v1->vec[k1].coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)))) + ensures(forall(k2, 0, MLDSA_K, + array_abs_bound(v0->vec[k2].coeffs, 0, MLDSA_N, MLDSA_GAMMA2+1))) +); +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +#define mld_polyveck_pack_w1 MLD_NAMESPACE_KL(polyveck_pack_w1) +/** + * Bit-pack polynomial vector w1 with coefficients in [0, 15] or [0, 43]. Input + * coefficients are assumed to be standard representatives. + * + * @spec{Implements @[FIPS204, Algorithm 28, w1Encode].} + * + * @param[out] r Pointer to output byte array with at least + * MLDSA_K * MLDSA_POLYW1_PACKEDBYTES bytes. + * @param[in] w1 Pointer to input polynomial vector. + */ +MLD_INTERNAL_API +void mld_polyveck_pack_w1(uint8_t r[MLDSA_K * MLDSA_POLYW1_PACKEDBYTES], + const mld_polyveck *w1) +__contract__( + requires(memory_no_alias(r, MLDSA_K * MLDSA_POLYW1_PACKEDBYTES)) + requires(memory_no_alias(w1, sizeof(mld_polyveck))) + requires(forall(k1, 0, MLDSA_K, + array_bound(w1->vec[k1].coeffs, 0, MLDSA_N, 0, (MLDSA_Q-1)/(2*MLDSA_GAMMA2)))) + assigns(memory_slice(r, MLDSA_K * MLDSA_POLYW1_PACKEDBYTES)) +); +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +#define mld_polyveck_pack_eta MLD_NAMESPACE_KL(polyveck_pack_eta) +/** + * Bit-pack polynomial vector with coefficients in [-MLDSA_ETA, MLDSA_ETA]. + * + * @param[out] r Pointer to output byte array with + * MLDSA_K * MLDSA_POLYETA_PACKEDBYTES bytes. + * @param[in] p Pointer to input polynomial vector. + */ +MLD_INTERNAL_API +void mld_polyveck_pack_eta(uint8_t r[MLDSA_K * MLDSA_POLYETA_PACKEDBYTES], + const mld_polyveck *p) +__contract__( + requires(memory_no_alias(r, MLDSA_K * MLDSA_POLYETA_PACKEDBYTES)) + requires(memory_no_alias(p, sizeof(mld_polyveck))) + requires(forall(k1, 0, MLDSA_K, + array_abs_bound(p->vec[k1].coeffs, 0, MLDSA_N, MLDSA_ETA + 1))) + assigns(memory_slice(r, MLDSA_K * MLDSA_POLYETA_PACKEDBYTES)) +); + +#define mld_polyvecl_pack_eta MLD_NAMESPACE_KL(polyvecl_pack_eta) +/** + * Bit-pack polynomial vector with coefficients in [-MLDSA_ETA, MLDSA_ETA]. + * + * @param[out] r Pointer to output byte array with + * MLDSA_L * MLDSA_POLYETA_PACKEDBYTES bytes. + * @param[in] p Pointer to input polynomial vector. + */ +MLD_INTERNAL_API +void mld_polyvecl_pack_eta(uint8_t r[MLDSA_L * MLDSA_POLYETA_PACKEDBYTES], + const mld_polyvecl *p) +__contract__( + requires(memory_no_alias(r, MLDSA_L * MLDSA_POLYETA_PACKEDBYTES)) + requires(memory_no_alias(p, sizeof(mld_polyvecl))) + requires(forall(k1, 0, MLDSA_L, + array_abs_bound(p->vec[k1].coeffs, 0, MLDSA_N, MLDSA_ETA + 1))) + assigns(memory_slice(r, MLDSA_L * MLDSA_POLYETA_PACKEDBYTES)) +); + +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || \ + (!defined(MLD_CONFIG_NO_SIGN_API) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))) +#define mld_polyvecl_unpack_eta MLD_NAMESPACE_KL(polyvecl_unpack_eta) +/** + * Unpack polynomial vector with coefficients in [-MLDSA_ETA, MLDSA_ETA]. + * + * @param[out] p Pointer to output polynomial vector. + * @param[in] r Input byte array with bit-packed polynomial vector. + */ +MLD_INTERNAL_API +void mld_polyvecl_unpack_eta( + mld_polyvecl *p, const uint8_t r[MLDSA_L * MLDSA_POLYETA_PACKEDBYTES]) +__contract__( + requires(memory_no_alias(r, MLDSA_L * MLDSA_POLYETA_PACKEDBYTES)) + requires(memory_no_alias(p, sizeof(mld_polyvecl))) + assigns(memory_slice(p, sizeof(mld_polyvecl))) + ensures(forall(k1, 0, MLDSA_L, + array_bound(p->vec[k1].coeffs, 0, MLDSA_N, MLD_POLYETA_UNPACK_LOWER_BOUND, MLDSA_ETA + 1))) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || (!MLD_CONFIG_NO_SIGN_API && \ + (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST)) */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +#define mld_polyvecl_unpack_z MLD_NAMESPACE_KL(polyvecl_unpack_z) +/** + * Unpack polynomial vector with coefficients in + * [-(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1]. + * + * @param[out] z Pointer to output polynomial vector. + * @param[in] r Input byte array with bit-packed polynomial vector. + */ +MLD_INTERNAL_API +void mld_polyvecl_unpack_z(mld_polyvecl *z, + const uint8_t r[MLDSA_L * MLDSA_POLYZ_PACKEDBYTES]) +__contract__( + requires(memory_no_alias(r, MLDSA_L * MLDSA_POLYZ_PACKEDBYTES)) + requires(memory_no_alias(z, sizeof(mld_polyvecl))) + assigns(memory_slice(z, sizeof(mld_polyvecl))) + ensures(forall(k1, 0, MLDSA_L, + array_bound(z->vec[k1].coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))) +); +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || \ + (!defined(MLD_CONFIG_NO_SIGN_API) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST))) +#define mld_polyveck_unpack_eta MLD_NAMESPACE_KL(polyveck_unpack_eta) +/** + * Unpack polynomial vector with coefficients in [-MLDSA_ETA, MLDSA_ETA]. + * + * @param[out] p Pointer to output polynomial vector. + * @param[in] r Input byte array with bit-packed polynomial vector. + */ +MLD_INTERNAL_API +void mld_polyveck_unpack_eta( + mld_polyveck *p, const uint8_t r[MLDSA_K * MLDSA_POLYETA_PACKEDBYTES]) +__contract__( + requires(memory_no_alias(r, MLDSA_K * MLDSA_POLYETA_PACKEDBYTES)) + requires(memory_no_alias(p, sizeof(mld_polyveck))) + assigns(memory_slice(p, sizeof(mld_polyveck))) + ensures(forall(k1, 0, MLDSA_K, + array_bound(p->vec[k1].coeffs, 0, MLDSA_N, MLD_POLYETA_UNPACK_LOWER_BOUND, MLDSA_ETA + 1))) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || (!MLD_CONFIG_NO_SIGN_API && \ + (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST)) */ + + +#endif /* !MLD_POLYVEC_H */ diff --git a/cbits/mldsa/src/polyvec_lazy.c b/cbits/mldsa/src/polyvec_lazy.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/polyvec_lazy.c @@ -0,0 +1,311 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + */ + +#include "polyvec_lazy.h" + +#include "debug.h" + +/* This namespacing is not done at the top to avoid a naming conflict + * with native backends, which are currently not yet namespaced. */ +#define mld_polymat_expand_entry MLD_ADD_PARAM_SET(mld_polymat_expand_entry) + +/** + * Sample a single matrix entry A[k][l] of ExpandA(rho) by rejection sampling + * from SHAKE128(rho|l|k), and apply the custom-order permutation when a + * native NTT backend is in use. + * + * The caller is expected to have copied rho into the first MLDSA_SEEDBYTES + * of seed_ext. This function writes the domain-separation bytes + * seed_ext[SEEDBYTES..+2] = {l, k} before sampling. + * + * @spec{Partially implements @[FIPS204, Algorithm 32, ExpandA] (samples one + * matrix entry via @[FIPS204, Algorithm 30, RejNTTPoly]).} + * + * @param[out] p Pointer to output polynomial. + * @param[in,out] seed_ext Seed buffer pre-filled with rho in the first + * MLDSA_SEEDBYTES; the final two bytes are + * overwritten. + * @param l Column index (inner, aka nonce low byte). + * @param k Row index (outer, aka nonce high byte). + */ +static MLD_INLINE void mld_polymat_expand_entry( + mld_poly *p, uint8_t seed_ext[MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2)], uint8_t l, + uint8_t k) +__contract__( + requires(memory_no_alias(p, sizeof(mld_poly))) + requires(memory_no_alias(seed_ext, MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2))) + assigns(memory_slice(p, sizeof(mld_poly))) + assigns(memory_slice(seed_ext, MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2))) + ensures(array_bound(p->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) +) +{ + seed_ext[MLDSA_SEEDBYTES + 0] = l; + seed_ext[MLDSA_SEEDBYTES + 1] = k; + mld_poly_uniform(p, seed_ext); + mld_poly_permute_bitrev_to_custom_optional(p); +} + +#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) + +MLD_INTERNAL_API +void mld_polyvec_matrix_expand_eager(mld_polymat_eager *mat, + const uint8_t rho[MLDSA_SEEDBYTES]) +{ + unsigned int i, j; + MLD_ALIGN uint8_t seed_ext[4][MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2)]; + + for (j = 0; j < 4; j++) + __loop__( + assigns(j, object_whole(seed_ext)) + invariant(j <= 4) + decreases(4 - j) + ) + { + mld_memcpy(seed_ext[j], rho, MLDSA_SEEDBYTES); + } + +#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) + /* Sample 4 matrix entries a time. */ + for (i = 0; i < (MLDSA_K * MLDSA_L / 4) * 4; i += 4) + __loop__( + assigns(i, j, object_whole(seed_ext), memory_slice(mat, sizeof(mld_polymat_eager))) + invariant(i <= (MLDSA_K * MLDSA_L / 4) * 4 && i % 4 == 0) + /* vectors 0 .. i / MLDSA_L are completely sampled */ + invariant(forall(k1, 0, i / MLDSA_L, forall(l1, 0, MLDSA_L, + array_bound(mat->vec[k1].vec[l1].coeffs, 0, MLDSA_N, 0, MLDSA_Q)))) + /* last vector is sampled up to i % MLDSA_L */ + invariant(forall(k2, i / MLDSA_L, i / MLDSA_L + 1, forall(l2, 0, i % MLDSA_L, + array_bound(mat->vec[k2].vec[l2].coeffs, 0, MLDSA_N, 0, MLDSA_Q)))) + decreases((MLDSA_K * MLDSA_L / 4) * 4 - i) + ) + { + for (j = 0; j < 4; j++) + __loop__( + assigns(j, object_whole(seed_ext)) + invariant(j <= 4) + decreases(4 - j) + ) + { + uint8_t x = (uint8_t)((i + j) / MLDSA_L); + uint8_t y = (uint8_t)((i + j) % MLDSA_L); + + seed_ext[j][MLDSA_SEEDBYTES + 0] = y; + seed_ext[j][MLDSA_SEEDBYTES + 1] = x; + } + + mld_poly_uniform_4x(&mat->vec[i / MLDSA_L].vec[i % MLDSA_L], + &mat->vec[(i + 1) / MLDSA_L].vec[(i + 1) % MLDSA_L], + &mat->vec[(i + 2) / MLDSA_L].vec[(i + 2) % MLDSA_L], + &mat->vec[(i + 3) / MLDSA_L].vec[(i + 3) % MLDSA_L], + seed_ext); + mld_poly_permute_bitrev_to_custom_optional( + &mat->vec[i / MLDSA_L].vec[i % MLDSA_L]); + mld_poly_permute_bitrev_to_custom_optional( + &mat->vec[(i + 1) / MLDSA_L].vec[(i + 1) % MLDSA_L]); + mld_poly_permute_bitrev_to_custom_optional( + &mat->vec[(i + 2) / MLDSA_L].vec[(i + 2) % MLDSA_L]); + mld_poly_permute_bitrev_to_custom_optional( + &mat->vec[(i + 3) / MLDSA_L].vec[(i + 3) % MLDSA_L]); + } +#else /* !MLD_CONFIG_SERIAL_FIPS202_ONLY */ + i = 0; +#endif /* MLD_CONFIG_SERIAL_FIPS202_ONLY */ + + /* Entries omitted by the batch-sampling are sampled individually. */ + while (i < MLDSA_K * MLDSA_L) + __loop__( + assigns(i, object_whole(seed_ext), memory_slice(mat, sizeof(mld_polymat_eager))) + invariant(i <= MLDSA_K * MLDSA_L) + /* vectors 0 .. i / MLDSA_L are completely sampled */ + invariant(forall(k1, 0, i / MLDSA_L, forall(l1, 0, MLDSA_L, + array_bound(mat->vec[k1].vec[l1].coeffs, 0, MLDSA_N, 0, MLDSA_Q)))) + /* last vector is sampled up to i % MLDSA_L */ + invariant(forall(k2, i / MLDSA_L, i / MLDSA_L + 1, forall(l2, 0, i % MLDSA_L, + array_bound(mat->vec[k2].vec[l2].coeffs, 0, MLDSA_N, 0, MLDSA_Q)))) + decreases(MLDSA_K * MLDSA_L - i) + ) + { + uint8_t x = (uint8_t)(i / MLDSA_L); + uint8_t y = (uint8_t)(i % MLDSA_L); + mld_polymat_expand_entry(&mat->vec[x].vec[y], seed_ext[0], y, x); + i++; + } + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(seed_ext, sizeof(seed_ext)); +} + +MLD_INTERNAL_API +void mld_polyvec_matrix_pointwise_montgomery_row_eager(mld_poly *t_row, + mld_polymat_eager *mat, + const mld_polyvecl *v, + unsigned int i) +{ + mld_polyvecl_pointwise_acc_montgomery(t_row, &mat->vec[i], v); +} + +#if !defined(MLD_CONFIG_NO_SIGN_API) +MLD_INTERNAL_API +void mld_polyvec_matrix_pointwise_montgomery_yvec_eager(mld_polyveck *w, + mld_polymat_eager *mat, + const mld_yvec_eager *y, + mld_polyvecl *scratch) +{ + unsigned int i; + *scratch = y->vec; + mld_polyvecl_ntt(scratch); + + for (i = 0; i < MLDSA_K; ++i) + __loop__( + assigns(i, memory_slice(w, sizeof(mld_polyveck))) + invariant(i <= MLDSA_K) + invariant(forall(k0, 0, i, + array_abs_bound(w->vec[k0].coeffs, 0, MLDSA_N, MLDSA_Q))) + decreases(MLDSA_K - i) + ) + { + mld_polyvec_matrix_pointwise_montgomery_row_eager(&w->vec[i], mat, scratch, + i); + } + + mld_polyveck_invntt_tomont(w); +} +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +#if defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) + +MLD_INTERNAL_API +void mld_polyvec_matrix_expand_lazy(mld_polymat_lazy *mat, + const uint8_t rho[MLDSA_SEEDBYTES]) +{ + mld_memcpy(mat->rho, rho, MLDSA_SEEDBYTES); +} + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +MLD_INTERNAL_API +void mld_polyvec_matrix_pointwise_montgomery_row_lazy(mld_poly *t_row, + mld_polymat_lazy *mat, + const mld_polyvecl *v, + unsigned int i) +{ + unsigned int l; + MLD_ALIGN uint8_t seed_ext[MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2)]; + mld_memcpy(seed_ext, mat->rho, MLDSA_SEEDBYTES); + + mld_polymat_expand_entry(t_row, seed_ext, 0, (uint8_t)i); + mld_poly_pointwise_montgomery(t_row, &v->vec[0]); + + for (l = 1; l < MLDSA_L; ++l) + __loop__( + assigns(l, object_whole(seed_ext), + memory_slice(t_row, sizeof(mld_poly)), + memory_slice(mat, sizeof(mld_polymat_lazy))) + invariant(l >= 1 && l <= MLDSA_L) + invariant(array_abs_bound(t_row->coeffs, 0, MLDSA_N, l * MLDSA_Q)) + decreases(MLDSA_L - l) + ) + { + mld_polymat_expand_entry(&mat->cur, seed_ext, (uint8_t)l, (uint8_t)i); + mld_poly_pointwise_montgomery(&mat->cur, &v->vec[l]); + mld_poly_add(t_row, &mat->cur); + } + mld_poly_reduce(t_row); + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(seed_ext, sizeof(seed_ext)); +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +MLD_INTERNAL_API +void mld_polyvec_matrix_pointwise_montgomery_yvec_lazy(mld_polyveck *w, + mld_polymat_lazy *mat, + const mld_yvec_lazy *y, + mld_polyvecl *scratch) +{ + unsigned int k, l; + MLD_ALIGN uint8_t seed_ext[MLD_ALIGN_UP(MLDSA_SEEDBYTES + 2)]; + /* Only the first poly of the polyvecl scratch is used. The polyvecl type + * matches the eager variant for API uniformity; in REDUCE_RAM mode the + * polyvecl storage is provided "for free" by the caller's polyveck/polyvecl + * union. */ + mld_poly *y_ntt = &scratch->vec[0]; + + mld_memcpy(seed_ext, mat->rho, MLDSA_SEEDBYTES); + + /* Column-by-column: sample y[l], NTT, accumulate column l of A into w. */ + for (l = 0; l < MLDSA_L; l++) + __loop__( + assigns(k, l, object_whole(seed_ext), + memory_slice(w, sizeof(mld_polyveck)), + memory_slice(mat, sizeof(mld_polymat_lazy)), + memory_slice(scratch, sizeof(mld_polyvecl))) + invariant(l <= MLDSA_L) + invariant(l == 0 || + forall(k0, 0, MLDSA_K, + array_abs_bound(w->vec[k0].coeffs, 0, MLDSA_N, + (int)l * MLDSA_Q))) + decreases(MLDSA_L - l) + ) + { + mld_yvec_get_poly_lazy(y_ntt, y, l); + mld_poly_ntt(y_ntt); + for (k = 0; k < MLDSA_K; k++) + __loop__( + assigns(k, object_whole(seed_ext), + memory_slice(w, sizeof(mld_polyveck)), + memory_slice(mat, sizeof(mld_polymat_lazy))) + invariant(k <= MLDSA_K) + invariant(l != 0 || + forall(k1, 0, k, + array_abs_bound(w->vec[k1].coeffs, 0, MLDSA_N, MLDSA_Q))) + invariant(l == 0 || + forall(k2, 0, k, + array_abs_bound(w->vec[k2].coeffs, 0, MLDSA_N, + ((int)l + 1) * MLDSA_Q))) + invariant(l == 0 || + forall(k3, k, MLDSA_K, + array_abs_bound(w->vec[k3].coeffs, 0, MLDSA_N, + (int)l * MLDSA_Q))) + decreases(MLDSA_K - k) + ) + { + if (l == 0) + { + mld_polymat_expand_entry(&w->vec[k], seed_ext, 0, (uint8_t)k); + mld_poly_pointwise_montgomery(&w->vec[k], y_ntt); + } + else + { + mld_polymat_expand_entry(&mat->cur, seed_ext, (uint8_t)l, (uint8_t)k); + mld_poly_pointwise_montgomery(&mat->cur, y_ntt); + mld_poly_add(&w->vec[k], &mat->cur); + } + } + } + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(seed_ext, sizeof(seed_ext)); + mld_polyveck_reduce(w); + mld_polyveck_invntt_tomont(w); +} +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#endif /* MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. + * Don't modify by hand -- this is auto-generated by scripts/autogen. */ +#undef mld_polymat_expand_entry diff --git a/cbits/mldsa/src/polyvec_lazy.h b/cbits/mldsa/src/polyvec_lazy.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/polyvec_lazy.h @@ -0,0 +1,652 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + */ + +/* + * Eager and lazy variants of polynomial vector types. + * + * In eager mode, full vectors are precomputed and stored in memory. + * In lazy mode, data is stored in packed form and expanded on demand, + * trading computation for reduced memory usage. + * + * MLD_CONFIG_REDUCE_RAM selects which variant is used. + */ + +#ifndef MLD_POLYVEC_LAZY_H +#define MLD_POLYVEC_LAZY_H + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_SIGN_API) || \ + !defined(MLD_CONFIG_NO_VERIFY_API) + +#include "poly.h" +#include "poly_kl.h" +#include "polyvec.h" + +/* Parameter set namespacing */ +#define mld_sk_s1hat_eager MLD_ADD_PARAM_SET(mld_sk_s1hat_eager) +#define mld_sk_s1hat_lazy MLD_ADD_PARAM_SET(mld_sk_s1hat_lazy) +#define mld_sk_s1hat MLD_ADD_PARAM_SET(mld_sk_s1hat) +#define mld_unpack_sk_s1hat_eager MLD_ADD_PARAM_SET(mld_unpack_sk_s1hat_eager) +#define mld_unpack_sk_s1hat_lazy MLD_ADD_PARAM_SET(mld_unpack_sk_s1hat_lazy) +#define mld_sk_s1hat_get_poly_eager \ + MLD_ADD_PARAM_SET(mld_sk_s1hat_get_poly_eager) +#define mld_sk_s1hat_get_poly_lazy MLD_ADD_PARAM_SET(mld_sk_s1hat_get_poly_lazy) +#define mld_sk_s2hat_eager MLD_ADD_PARAM_SET(mld_sk_s2hat_eager) +#define mld_sk_s2hat_lazy MLD_ADD_PARAM_SET(mld_sk_s2hat_lazy) +#define mld_sk_s2hat MLD_ADD_PARAM_SET(mld_sk_s2hat) +#define mld_unpack_sk_s2hat_eager MLD_ADD_PARAM_SET(mld_unpack_sk_s2hat_eager) +#define mld_unpack_sk_s2hat_lazy MLD_ADD_PARAM_SET(mld_unpack_sk_s2hat_lazy) +#define mld_sk_s2hat_get_poly_eager \ + MLD_ADD_PARAM_SET(mld_sk_s2hat_get_poly_eager) +#define mld_sk_s2hat_get_poly_lazy MLD_ADD_PARAM_SET(mld_sk_s2hat_get_poly_lazy) +#define mld_sk_t0hat_eager MLD_ADD_PARAM_SET(mld_sk_t0hat_eager) +#define mld_sk_t0hat_lazy MLD_ADD_PARAM_SET(mld_sk_t0hat_lazy) +#define mld_sk_t0hat MLD_ADD_PARAM_SET(mld_sk_t0hat) +#define mld_unpack_sk_t0hat_eager MLD_ADD_PARAM_SET(mld_unpack_sk_t0hat_eager) +#define mld_unpack_sk_t0hat_lazy MLD_ADD_PARAM_SET(mld_unpack_sk_t0hat_lazy) +#define mld_sk_t0hat_get_poly_eager \ + MLD_ADD_PARAM_SET(mld_sk_t0hat_get_poly_eager) +#define mld_sk_t0hat_get_poly_lazy MLD_ADD_PARAM_SET(mld_sk_t0hat_get_poly_lazy) +#define mld_polymat MLD_ADD_PARAM_SET(mld_polymat) +#define mld_polymat_eager MLD_ADD_PARAM_SET(mld_polymat_eager) +#define mld_polymat_lazy MLD_ADD_PARAM_SET(mld_polymat_lazy) +#define mld_poly_permute_bitrev_to_custom_optional \ + MLD_ADD_PARAM_SET(mld_poly_permute_bitrev_to_custom_optional) +#define mld_polyvec_matrix_expand_eager \ + MLD_NAMESPACE_KL(polyvec_matrix_expand_eager) +#define mld_polyvec_matrix_expand_lazy \ + MLD_NAMESPACE_KL(polyvec_matrix_expand_lazy) +#define mld_polyvec_matrix_pointwise_montgomery \ + MLD_NAMESPACE_KL(polyvec_matrix_pointwise_montgomery) +#define mld_polyvec_matrix_pointwise_montgomery_row_eager \ + MLD_NAMESPACE_KL(polyvec_matrix_pointwise_montgomery_row_eager) +#define mld_polyvec_matrix_pointwise_montgomery_row_lazy \ + MLD_NAMESPACE_KL(polyvec_matrix_pointwise_montgomery_row_lazy) +#define mld_polyvec_matrix_pointwise_montgomery_yvec_eager \ + MLD_NAMESPACE_KL(polyvec_matrix_pointwise_montgomery_yvec_eager) +#define mld_polyvec_matrix_pointwise_montgomery_yvec_lazy \ + MLD_NAMESPACE_KL(polyvec_matrix_pointwise_montgomery_yvec_lazy) +#define mld_yvec_eager MLD_ADD_PARAM_SET(mld_yvec_eager) +#define mld_yvec_lazy MLD_ADD_PARAM_SET(mld_yvec_lazy) +#define mld_yvec MLD_ADD_PARAM_SET(mld_yvec) +#define mld_yvec_init_eager MLD_ADD_PARAM_SET(mld_yvec_init_eager) +#define mld_yvec_init_lazy MLD_ADD_PARAM_SET(mld_yvec_init_lazy) +#define mld_yvec_get_poly_eager MLD_ADD_PARAM_SET(mld_yvec_get_poly_eager) +#define mld_yvec_get_poly_lazy MLD_ADD_PARAM_SET(mld_yvec_get_poly_lazy) +/* End of parameter set namespacing */ + +/** Eager s1hat: precomputed s1 vector in NTT domain. */ +typedef struct +{ + mld_polyvecl vec; /**< s1 vector in NTT domain. */ +} mld_sk_s1hat_eager; + +/** Eager s2hat: precomputed s2 vector in NTT domain. */ +typedef struct +{ + mld_polyveck vec; /**< s2 vector in NTT domain. */ +} mld_sk_s2hat_eager; + +/** Eager t0hat: precomputed t0 vector in NTT domain. */ +typedef struct +{ + mld_polyveck vec; /**< t0 vector in NTT domain. */ +} mld_sk_t0hat_eager; + +/** Lazy s1hat: borrow packed s1, unpack and convert to NTT domain on demand. */ +typedef struct +{ + const uint8_t *packed; /**< Pointer to packed s1 in the secret key. */ +} mld_sk_s1hat_lazy; + +/** Lazy s2hat: borrow packed s2, unpack and convert to NTT domain on demand. */ +typedef struct +{ + const uint8_t *packed; /**< Pointer to packed s2 in the secret key. */ +} mld_sk_s2hat_lazy; + +/** Lazy t0hat: borrow packed t0, unpack and convert to NTT domain on demand. */ +typedef struct +{ + const uint8_t *packed; /**< Pointer to packed t0 in the secret key. */ +} mld_sk_t0hat_lazy; + +/** Eager yvec: precomputed and stored full signing masking vector y. */ +typedef struct +{ + mld_polyvecl vec; /**< Masking vector y. */ +} mld_yvec_eager; + +/** Lazy yvec: store seed and base counter kappa, regenerate y[i] on demand. */ +typedef struct +{ + const uint8_t *rhoprime; /**< Pointer to seed used to derive y. */ + uint16_t kappa; /**< Base counter; component i uses kappa + i. */ +} mld_yvec_lazy; + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_SIGN_API) +/* s1vec */ + +#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +static MLD_INLINE void mld_unpack_sk_s1hat_eager( + mld_sk_s1hat_eager *s1, + const uint8_t packed_s1[MLDSA_L * MLDSA_POLYETA_PACKEDBYTES]) +__contract__( + requires(memory_no_alias(s1, sizeof(mld_sk_s1hat_eager))) + requires(memory_no_alias(packed_s1, MLDSA_L * MLDSA_POLYETA_PACKEDBYTES)) + assigns(memory_slice(s1, sizeof(mld_sk_s1hat_eager))) + ensures(forall(k1, 0, MLDSA_L, + array_abs_bound(s1->vec.vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) +) +{ + mld_polyvecl_unpack_eta(&s1->vec, packed_s1); + mld_polyvecl_ntt(&s1->vec); +} + +#if !defined(MLD_CONFIG_NO_SIGN_API) +static MLD_INLINE void mld_sk_s1hat_get_poly_eager(mld_poly *buf, + const mld_sk_s1hat_eager *s1, + unsigned int i) +__contract__( + requires(memory_no_alias(buf, sizeof(mld_poly))) + requires(memory_no_alias(s1, sizeof(mld_sk_s1hat_eager))) + requires(i < MLDSA_L) + requires(array_abs_bound(s1->vec.vec[i].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) + assigns(memory_slice(buf, sizeof(mld_poly))) + ensures(array_abs_bound(buf->coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) +) { *buf = s1->vec.vec[i]; } +#endif /* !MLD_CONFIG_NO_SIGN_API */ +#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ +#if defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +static MLD_INLINE void mld_unpack_sk_s1hat_lazy( + mld_sk_s1hat_lazy *s1, + const uint8_t packed_s1[MLDSA_L * MLDSA_POLYETA_PACKEDBYTES]) +__contract__( + requires(memory_no_alias(s1, sizeof(mld_sk_s1hat_lazy))) + assigns(memory_slice(s1, sizeof(mld_sk_s1hat_lazy))) + ensures(s1->packed == old(packed_s1)) +) { s1->packed = packed_s1; } + +#if !defined(MLD_CONFIG_NO_SIGN_API) +static MLD_INLINE void mld_sk_s1hat_get_poly_lazy(mld_poly *buf, + const mld_sk_s1hat_lazy *s1, + unsigned int i) +__contract__( + requires(memory_no_alias(buf, sizeof(mld_poly))) + requires(memory_no_alias(s1, sizeof(mld_sk_s1hat_lazy))) + requires(i < MLDSA_L) + requires(memory_no_alias(s1->packed, MLDSA_L * MLDSA_POLYETA_PACKEDBYTES)) + assigns(memory_slice(buf, sizeof(mld_poly))) + ensures(array_abs_bound(buf->coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) +) +{ + mld_polyeta_unpack(buf, s1->packed + i * MLDSA_POLYETA_PACKEDBYTES); + mld_poly_ntt(buf); +} +#endif /* !MLD_CONFIG_NO_SIGN_API */ +#endif /* MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +/* s2vec */ + +#if (!defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST)) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)) +static MLD_INLINE void mld_unpack_sk_s2hat_eager( + mld_sk_s2hat_eager *s2, + const uint8_t packed_s2[MLDSA_K * MLDSA_POLYETA_PACKEDBYTES]) +__contract__( + requires(memory_no_alias(s2, sizeof(mld_sk_s2hat_eager))) + requires(memory_no_alias(packed_s2, MLDSA_K * MLDSA_POLYETA_PACKEDBYTES)) + assigns(memory_slice(s2, sizeof(mld_sk_s2hat_eager))) + ensures(forall(k1, 0, MLDSA_K, + array_abs_bound(s2->vec.vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) +) +{ + mld_polyveck_unpack_eta(&s2->vec, packed_s2); + mld_polyveck_ntt(&s2->vec); +} + +#if !defined(MLD_CONFIG_NO_SIGN_API) +static MLD_INLINE void mld_sk_s2hat_get_poly_eager(mld_poly *buf, + const mld_sk_s2hat_eager *s2, + unsigned int i) +__contract__( + requires(memory_no_alias(buf, sizeof(mld_poly))) + requires(memory_no_alias(s2, sizeof(mld_sk_s2hat_eager))) + requires(i < MLDSA_K) + requires(array_abs_bound(s2->vec.vec[i].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) + assigns(memory_slice(buf, sizeof(mld_poly))) + ensures(array_abs_bound(buf->coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) +) { *buf = s2->vec.vec[i]; } +#endif /* !MLD_CONFIG_NO_SIGN_API */ +#endif /* (!MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST) && \ + (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) */ +#if defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +static MLD_INLINE void mld_unpack_sk_s2hat_lazy( + mld_sk_s2hat_lazy *s2, + const uint8_t packed_s2[MLDSA_K * MLDSA_POLYETA_PACKEDBYTES]) +__contract__( + requires(memory_no_alias(s2, sizeof(mld_sk_s2hat_lazy))) + assigns(memory_slice(s2, sizeof(mld_sk_s2hat_lazy))) + ensures(s2->packed == old(packed_s2)) +) { s2->packed = packed_s2; } + +#if !defined(MLD_CONFIG_NO_SIGN_API) +static MLD_INLINE void mld_sk_s2hat_get_poly_lazy(mld_poly *buf, + const mld_sk_s2hat_lazy *s2, + unsigned int i) +__contract__( + requires(memory_no_alias(buf, sizeof(mld_poly))) + requires(memory_no_alias(s2, sizeof(mld_sk_s2hat_lazy))) + requires(i < MLDSA_K) + requires(memory_no_alias(s2->packed, MLDSA_K * MLDSA_POLYETA_PACKEDBYTES)) + assigns(memory_slice(buf, sizeof(mld_poly))) + ensures(array_abs_bound(buf->coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) +) +{ + mld_polyeta_unpack(buf, s2->packed + i * MLDSA_POLYETA_PACKEDBYTES); + mld_poly_ntt(buf); +} +#endif /* !MLD_CONFIG_NO_SIGN_API */ +#endif /* MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +/* t0vec */ + +#if (!defined(MLD_CONFIG_NO_SIGN_API) || defined(MLD_UNIT_TEST)) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)) +static MLD_INLINE void mld_unpack_sk_t0hat_eager( + mld_sk_t0hat_eager *t0, + const uint8_t packed_t0[MLDSA_K * MLDSA_POLYT0_PACKEDBYTES]) +__contract__( + requires(memory_no_alias(t0, sizeof(mld_sk_t0hat_eager))) + requires(memory_no_alias(packed_t0, MLDSA_K * MLDSA_POLYT0_PACKEDBYTES)) + assigns(memory_slice(t0, sizeof(mld_sk_t0hat_eager))) + ensures(forall(k1, 0, MLDSA_K, + array_abs_bound(t0->vec.vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) +) +{ + unsigned int i; + for (i = 0; i < MLDSA_K; ++i) + __loop__( + assigns(i, memory_slice(t0, sizeof(mld_sk_t0hat_eager))) + invariant(i <= MLDSA_K) + invariant(forall(k0, 0, i, + array_bound(t0->vec.vec[k0].coeffs, 0, MLDSA_N, + -(1 << (MLDSA_D - 1)) + 1, (1 << (MLDSA_D - 1)) + 1))) + decreases(MLDSA_K - i) + ) + { + mld_polyt0_unpack(&t0->vec.vec[i], + packed_t0 + i * MLDSA_POLYT0_PACKEDBYTES); + } + mld_polyveck_ntt(&t0->vec); +} + +#if !defined(MLD_CONFIG_NO_SIGN_API) +static MLD_INLINE void mld_sk_t0hat_get_poly_eager(mld_poly *buf, + const mld_sk_t0hat_eager *t0, + unsigned int i) +__contract__( + requires(memory_no_alias(buf, sizeof(mld_poly))) + requires(memory_no_alias(t0, sizeof(mld_sk_t0hat_eager))) + requires(i < MLDSA_K) + requires(array_abs_bound(t0->vec.vec[i].coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) + assigns(memory_slice(buf, sizeof(mld_poly))) + ensures(array_abs_bound(buf->coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) +) { *buf = t0->vec.vec[i]; } +#endif /* !MLD_CONFIG_NO_SIGN_API */ +#endif /* (!MLD_CONFIG_NO_SIGN_API || MLD_UNIT_TEST) && \ + (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) */ +#if defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +static MLD_INLINE void mld_unpack_sk_t0hat_lazy( + mld_sk_t0hat_lazy *t0, + const uint8_t packed_t0[MLDSA_K * MLDSA_POLYT0_PACKEDBYTES]) +__contract__( + requires(memory_no_alias(t0, sizeof(mld_sk_t0hat_lazy))) + assigns(memory_slice(t0, sizeof(mld_sk_t0hat_lazy))) + ensures(t0->packed == old(packed_t0)) +) { t0->packed = packed_t0; } + +#if !defined(MLD_CONFIG_NO_SIGN_API) +static MLD_INLINE void mld_sk_t0hat_get_poly_lazy(mld_poly *buf, + const mld_sk_t0hat_lazy *t0, + unsigned int i) +__contract__( + requires(memory_no_alias(buf, sizeof(mld_poly))) + requires(memory_no_alias(t0, sizeof(mld_sk_t0hat_lazy))) + requires(i < MLDSA_K) + requires(memory_no_alias(t0->packed, MLDSA_K * MLDSA_POLYT0_PACKEDBYTES)) + assigns(memory_slice(buf, sizeof(mld_poly))) + ensures(array_abs_bound(buf->coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) +) +{ + mld_polyt0_unpack(buf, t0->packed + i * MLDSA_POLYT0_PACKEDBYTES); + mld_poly_ntt(buf); +} +#endif /* !MLD_CONFIG_NO_SIGN_API */ +#endif /* MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API */ + +/* yvec */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) && \ + (!defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)) +static MLD_INLINE void mld_yvec_init_eager( + mld_yvec_eager *y, const uint8_t rhoprime[MLDSA_CRHBYTES], uint16_t kappa) +__contract__( + requires(memory_no_alias(y, sizeof(mld_yvec_eager))) + requires(memory_no_alias(rhoprime, MLDSA_CRHBYTES)) + requires(kappa <= MLD_MAX_KAPPA) + assigns(memory_slice(y, sizeof(mld_yvec_eager))) + ensures(forall(k1, 0, MLDSA_L, + array_bound(y->vec.vec[k1].coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))) +) +{ + mld_polyvecl_uniform_gamma1(&y->vec, rhoprime, kappa); +} + +static MLD_INLINE void mld_yvec_get_poly_eager(mld_poly *buf, + const mld_yvec_eager *y, + unsigned int i) +__contract__( + requires(memory_no_alias(buf, sizeof(mld_poly))) + requires(memory_no_alias(y, sizeof(mld_yvec_eager))) + requires(i < MLDSA_L) + requires(array_bound(y->vec.vec[i].coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) + assigns(memory_slice(buf, sizeof(mld_poly))) + ensures(array_bound(buf->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) +) { *buf = y->vec.vec[i]; } +#endif /* !MLD_CONFIG_NO_SIGN_API && (!MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) \ + */ +#if !defined(MLD_CONFIG_NO_SIGN_API) && \ + (defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST)) +static MLD_INLINE void mld_yvec_init_lazy( + mld_yvec_lazy *y, const uint8_t rhoprime[MLDSA_CRHBYTES], uint16_t kappa) +__contract__( + requires(memory_no_alias(y, sizeof(mld_yvec_lazy))) + assigns(memory_slice(y, sizeof(mld_yvec_lazy))) + ensures(y->rhoprime == old(rhoprime)) + ensures(y->kappa == old(kappa)) +) +{ + y->rhoprime = rhoprime; + y->kappa = kappa; +} + +static MLD_INLINE void mld_yvec_get_poly_lazy(mld_poly *buf, + const mld_yvec_lazy *y, + unsigned int i) +__contract__( + requires(memory_no_alias(buf, sizeof(mld_poly))) + requires(memory_no_alias(y, sizeof(mld_yvec_lazy))) + requires(i < MLDSA_L) + requires(memory_no_alias(y->rhoprime, MLDSA_CRHBYTES)) + requires(y->kappa <= MLD_MAX_KAPPA) + assigns(memory_slice(buf, sizeof(mld_poly))) + ensures(array_bound(buf->coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1)) +) +{ + /* Safety: y->kappa <= MLD_MAX_KAPPA and i < MLDSA_L, so y->kappa + i + * fits in uint16_t. See MLD_MAX_KAPPA comment in params.h. */ + mld_poly_uniform_gamma1(buf, y->rhoprime, (uint16_t)(y->kappa + i)); +} +#endif /* !MLD_CONFIG_NO_SIGN_API && (MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST) \ + */ + +/* polymat */ + +#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +/** Eager polymat: precomputed and stored full MLDSA_K x MLDSA_L matrix. */ +typedef struct +{ + mld_polyvecl vec[MLDSA_K]; /**< Rows of the matrix. */ +} mld_polymat_eager; +#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +/** Lazy polymat: store seed, sample elements A[k][l] on demand. */ +typedef struct +{ + mld_poly cur; /**< On-demand sampled matrix element A[k][l]. */ + uint8_t rho[MLDSA_SEEDBYTES]; /**< Public seed used to expand A. */ +} mld_polymat_lazy; + +static MLD_INLINE void mld_poly_permute_bitrev_to_custom_optional(mld_poly *p) +__contract__( + /* We don't specify that this is a permutation, only that it preserves + * the bounds. + * When the native NTT backend does not use the custom order, this is a no-op. */ + requires(memory_no_alias(p, sizeof(mld_poly))) + requires(array_bound(p->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) + assigns(memory_slice(p, sizeof(mld_poly))) + ensures(array_bound(p->coeffs, 0, MLDSA_N, 0, MLDSA_Q)) +) +{ +#if defined(MLD_USE_NATIVE_NTT_CUSTOM_ORDER) + mld_poly_permute_bitrev_to_custom(p->coeffs); +#else + (void)p; +#endif +} + +#if !defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +/** + * Generates matrix A with uniformly random coefficients a_{i,j} by performing + * rejection sampling on the output stream of SHAKE128(rho|j|i). + * + * @spec{Implements @[FIPS204, Algorithm 32, ExpandA].} + * + * @param[out] mat Pointer to output matrix. + * @param[in] rho Byte array containing seed rho. + */ +MLD_INTERNAL_API +void mld_polyvec_matrix_expand_eager(mld_polymat_eager *mat, + const uint8_t rho[MLDSA_SEEDBYTES]) +__contract__( + requires(memory_no_alias(mat, sizeof(mld_polymat_eager))) + requires(memory_no_alias(rho, MLDSA_SEEDBYTES)) + assigns(memory_slice(mat, sizeof(mld_polymat_eager))) + ensures(forall(k1, 0, MLDSA_K, forall(l1, 0, MLDSA_L, + array_bound(mat->vec[k1].vec[l1].coeffs, 0, MLDSA_N, 0, MLDSA_Q)))) +); + +/** + * Compute row i of matrix-vector multiplication in NTT domain with pointwise + * multiplication and multiplication by 2^{-32}. + * + * Input matrix and vector must be in NTT domain representation. Output + * coefficients are bounded by MLDSA_Q in absolute value. + * + * @param[out] t_row Pointer to output row polynomial. + * @param[in] mat Pointer to input matrix. + * @param[in] v Pointer to input vector v. + * @param i Row index, 0 <= i < MLDSA_K. + */ +MLD_INTERNAL_API +void mld_polyvec_matrix_pointwise_montgomery_row_eager(mld_poly *t_row, + mld_polymat_eager *mat, + const mld_polyvecl *v, + unsigned int i) +__contract__( + requires(memory_no_alias(t_row, sizeof(mld_poly))) + requires(memory_no_alias(mat, sizeof(mld_polymat_eager))) + requires(memory_no_alias(v, sizeof(mld_polyvecl))) + requires(i < MLDSA_K) + requires(forall(l1, 0, MLDSA_L, + array_bound(mat->vec[i].vec[l1].coeffs, 0, MLDSA_N, 0, MLDSA_Q))) + requires(forall(l2, 0, MLDSA_L, + array_abs_bound(v->vec[l2].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + assigns(memory_slice(t_row, sizeof(mld_poly))) + ensures(array_abs_bound(t_row->coeffs, 0, MLDSA_N, MLDSA_Q)) +); + +#if !defined(MLD_CONFIG_NO_SIGN_API) +/** + * Compute w = invNTT(A * NTT(y)) for the signing y vector. + * + * The eager variant copies y into the scratch polyvecl, NTTs it in place, + * calls the standard matrix-vector multiply, and finally inverse-NTTs the + * result into w. + * + * @param[out] w Pointer to output vector. + * @param[in] mat Pointer to input matrix. + * @param[in] y Pointer to (non-NTT) y vector. + * @param[out] scratch Scratch polyvecl for NTT'd copy of y. + */ +MLD_INTERNAL_API +void mld_polyvec_matrix_pointwise_montgomery_yvec_eager(mld_polyveck *w, + mld_polymat_eager *mat, + const mld_yvec_eager *y, + mld_polyvecl *scratch) +__contract__( + requires(memory_no_alias(w, sizeof(mld_polyveck))) + requires(memory_no_alias(mat, sizeof(mld_polymat_eager))) + requires(memory_no_alias(y, sizeof(mld_yvec_eager))) + requires(memory_no_alias(scratch, sizeof(mld_polyvecl))) + requires(forall(k1, 0, MLDSA_K, forall(l1, 0, MLDSA_L, + array_bound(mat->vec[k1].vec[l1].coeffs, 0, MLDSA_N, 0, MLDSA_Q)))) + requires(forall(l2, 0, MLDSA_L, + array_bound(y->vec.vec[l2].coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))) + assigns(memory_slice(w, sizeof(mld_polyveck))) + assigns(memory_slice(scratch, sizeof(mld_polyvecl))) + ensures(forall(k0, 0, MLDSA_K, + array_abs_bound(w->vec[k0].coeffs, 0, MLDSA_N, MLD_INTT_BOUND))) +); +#endif /* !MLD_CONFIG_NO_SIGN_API */ +#endif /* !MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +#if defined(MLD_CONFIG_REDUCE_RAM) || defined(MLD_UNIT_TEST) +MLD_INTERNAL_API +void mld_polyvec_matrix_expand_lazy(mld_polymat_lazy *mat, + const uint8_t rho[MLDSA_SEEDBYTES]) +__contract__( + requires(memory_no_alias(mat, sizeof(mld_polymat_lazy))) + requires(memory_no_alias(rho, MLDSA_SEEDBYTES)) + assigns(memory_slice(mat, sizeof(mld_polymat_lazy))) +); + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +/** + * Compute row i of matrix-vector multiplication in NTT domain with pointwise + * multiplication and multiplication by 2^{-32}. + * + * Input vector must be in NTT domain representation; the matrix entries are + * sampled on demand from the seed stored in mat->rho, using mat->cur as + * scratch. Output coefficients are bounded by MLDSA_Q in absolute value. + * + * @param[out] t_row Pointer to output row polynomial. + * @param[in,out] mat Pointer to input matrix (seed + scratch). + * @param[in] v Pointer to input vector v. + * @param i Row index, 0 <= i < MLDSA_K. + */ +MLD_INTERNAL_API +void mld_polyvec_matrix_pointwise_montgomery_row_lazy(mld_poly *t_row, + mld_polymat_lazy *mat, + const mld_polyvecl *v, + unsigned int i) +__contract__( + requires(memory_no_alias(t_row, sizeof(mld_poly))) + requires(memory_no_alias(mat, sizeof(mld_polymat_lazy))) + requires(memory_no_alias(v, sizeof(mld_polyvecl))) + requires(i < MLDSA_K) + requires(forall(l1, 0, MLDSA_L, + array_abs_bound(v->vec[l1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + assigns(memory_slice(t_row, sizeof(mld_poly))) + assigns(memory_slice(mat, sizeof(mld_polymat_lazy))) + ensures(array_abs_bound(t_row->coeffs, 0, MLDSA_N, MLDSA_Q)) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +/** + * Compute w = invNTT(A * NTT(y)) for the signing y vector. + * + * The lazy variant samples one column of y at a time, NTTs it into + * &scratch->vec[0], and accumulates the matrix-vector product + * column-by-column with on-demand sampling of A[k][l]. Only the first poly of + * the polyvecl scratch is used; the polyvecl type is shared with the eager + * variant for API uniformity (the storage is provided "for free" by the + * caller's polyveck/polyvecl union in REDUCE_RAM mode). + * + * @param[out] w Pointer to output vector. + * @param[in,out] mat Pointer to input matrix. + * @param[in] y Pointer to y seed/kappa. + * @param[out] scratch Scratch (only &scratch->vec[0] used). + */ +MLD_INTERNAL_API +void mld_polyvec_matrix_pointwise_montgomery_yvec_lazy(mld_polyveck *w, + mld_polymat_lazy *mat, + const mld_yvec_lazy *y, + mld_polyvecl *scratch) +__contract__( + requires(memory_no_alias(w, sizeof(mld_polyveck))) + requires(memory_no_alias(mat, sizeof(mld_polymat_lazy))) + requires(memory_no_alias(y, sizeof(mld_yvec_lazy))) + requires(memory_no_alias(scratch, sizeof(mld_polyvecl))) + requires(memory_no_alias(y->rhoprime, MLDSA_CRHBYTES)) + requires(y->kappa <= MLD_MAX_KAPPA) + assigns(memory_slice(w, sizeof(mld_polyveck))) + assigns(memory_slice(mat, sizeof(mld_polymat_lazy))) + assigns(memory_slice(scratch, sizeof(mld_polyvecl))) + ensures(forall(k0, 0, MLDSA_K, + array_abs_bound(w->vec[k0].coeffs, 0, MLDSA_N, MLD_INTT_BOUND))) +); +#endif /* !MLD_CONFIG_NO_SIGN_API */ +#endif /* MLD_CONFIG_REDUCE_RAM || MLD_UNIT_TEST */ + +/* Dispatch: typedef and define based on MLD_CONFIG_REDUCE_RAM */ +#if defined(MLD_CONFIG_REDUCE_RAM) +typedef mld_sk_s1hat_lazy mld_sk_s1hat; +typedef mld_sk_s2hat_lazy mld_sk_s2hat; +typedef mld_sk_t0hat_lazy mld_sk_t0hat; +typedef mld_polymat_lazy mld_polymat; +typedef mld_yvec_lazy mld_yvec; +#define mld_unpack_sk_s1hat mld_unpack_sk_s1hat_lazy +#define mld_unpack_sk_s2hat mld_unpack_sk_s2hat_lazy +#define mld_unpack_sk_t0hat mld_unpack_sk_t0hat_lazy +#if !defined(MLD_CONFIG_NO_SIGN_API) +#define mld_sk_s1hat_get_poly mld_sk_s1hat_get_poly_lazy +#define mld_sk_s2hat_get_poly mld_sk_s2hat_get_poly_lazy +#define mld_sk_t0hat_get_poly mld_sk_t0hat_get_poly_lazy +#endif +#define mld_polyvec_matrix_expand mld_polyvec_matrix_expand_lazy +#define mld_polyvec_matrix_pointwise_montgomery_row \ + mld_polyvec_matrix_pointwise_montgomery_row_lazy +#define mld_yvec_init mld_yvec_init_lazy +#define mld_yvec_get_poly mld_yvec_get_poly_lazy +#define mld_polyvec_matrix_pointwise_montgomery_yvec \ + mld_polyvec_matrix_pointwise_montgomery_yvec_lazy +#else /* MLD_CONFIG_REDUCE_RAM */ +typedef mld_sk_s1hat_eager mld_sk_s1hat; +typedef mld_sk_s2hat_eager mld_sk_s2hat; +typedef mld_sk_t0hat_eager mld_sk_t0hat; +typedef mld_polymat_eager mld_polymat; +typedef mld_yvec_eager mld_yvec; +#define mld_unpack_sk_s1hat mld_unpack_sk_s1hat_eager +#define mld_unpack_sk_s2hat mld_unpack_sk_s2hat_eager +#define mld_unpack_sk_t0hat mld_unpack_sk_t0hat_eager +#if !defined(MLD_CONFIG_NO_SIGN_API) +#define mld_sk_s2hat_get_poly mld_sk_s2hat_get_poly_eager +#define mld_sk_s1hat_get_poly mld_sk_s1hat_get_poly_eager +#define mld_sk_t0hat_get_poly mld_sk_t0hat_get_poly_eager +#endif +#define mld_polyvec_matrix_expand mld_polyvec_matrix_expand_eager +#define mld_polyvec_matrix_pointwise_montgomery_row \ + mld_polyvec_matrix_pointwise_montgomery_row_eager +#define mld_yvec_init mld_yvec_init_eager +#define mld_yvec_get_poly mld_yvec_get_poly_eager +#define mld_polyvec_matrix_pointwise_montgomery_yvec \ + mld_polyvec_matrix_pointwise_montgomery_yvec_eager +#endif /* !MLD_CONFIG_REDUCE_RAM */ + +#endif /* !MLD_CONFIG_NO_KEYPAIR_API || !MLD_CONFIG_NO_SIGN_API || \ + !MLD_CONFIG_NO_VERIFY_API */ +#endif /* !MLD_POLYVEC_LAZY_H */ diff --git a/cbits/mldsa/src/randombytes.h b/cbits/mldsa/src/randombytes.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/randombytes.h @@ -0,0 +1,26 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLD_RANDOMBYTES_H +#define MLD_RANDOMBYTES_H + +#include + +#include "cbmc.h" +#include "common.h" + +#if !defined(MLD_CONFIG_NO_RANDOMIZED_API) +#if !defined(MLD_CONFIG_CUSTOM_RANDOMBYTES) +MLD_MUST_CHECK_RETURN_VALUE +int randombytes(uint8_t *out, size_t outlen); + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_randombytes(uint8_t *out, size_t outlen) +__contract__( + requires(memory_no_alias(out, outlen)) + assigns(memory_slice(out, outlen)) +) { return randombytes(out, outlen); } +#endif /* !MLD_CONFIG_CUSTOM_RANDOMBYTES */ +#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */ +#endif /* !MLD_RANDOMBYTES_H */ diff --git a/cbits/mldsa/src/reduce.h b/cbits/mldsa/src/reduce.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/reduce.h @@ -0,0 +1,144 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + */ + +#ifndef MLD_REDUCE_H +#define MLD_REDUCE_H + +#include "cbmc.h" +#include "common.h" +#include "ct.h" +#include "debug.h" + +/* check-magic: -4186625 == pow(2,32,MLDSA_Q) */ +#define MLD_MONT (-4186625) + +/* Upper bound for domain of mld_reduce32() */ +#define MLD_REDUCE32_DOMAIN_MAX (INT32_MAX - ((int32_t)1 << 22)) + +/* Absolute bound for range of mld_reduce32() */ +/* check-magic: 6283009 == (MLD_REDUCE32_DOMAIN_MAX - 255 * MLDSA_Q + 1) */ +#define MLD_REDUCE32_RANGE_MAX 6283009 + +/** + * Generic Montgomery reduction; given a 64-bit integer a, computes a 32-bit + * integer congruent to a * R^-1 mod MLDSA_Q, where R=2^32. + * + * @spec{Implements @[FIPS204, Algorithm 49, MontgomeryReduce].} + * + * @param a Input integer to be reduced, of absolute value smaller or equal + * to INT64_MAX - 2^31 * MLDSA_Q. + * + * @return Integer congruent to a * R^-1 modulo MLDSA_Q, with absolute value + * <= |a| / 2^32 + MLDSA_Q / 2. + * In particular, if |a| < 2^31 * MLDSA_Q, the absolute value of the + * return value is < MLDSA_Q. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int32_t mld_montgomery_reduce(int64_t a) +__contract__( + /* We don't attempt to express an input-dependent output bound + * as the post-condition here, as all call-sites satisfy the + * absolute input bound 2^31 * MLDSA_Q and higher-level + * reasoning can be conducted using |return_value| < MLDSA_Q. */ + requires(a > -(((int64_t)1 << 31) * MLDSA_Q) && + a < (((int64_t)1 << 31) * MLDSA_Q)) + ensures(return_value > -MLDSA_Q && return_value < MLDSA_Q) +) +{ + /* check-magic: 58728449 == unsigned_mod(pow(MLDSA_Q, -1, 2^32), 2^32) */ + const uint64_t QINV = 58728449; + + /* Compute a*q^{-1} mod 2^32 in unsigned representatives */ + const uint32_t a_reduced = mld_cast_int64_to_uint32(a); + const uint32_t a_inverted = (a_reduced * QINV) & UINT32_MAX; + + /* Lift to signed canonical representative mod 2^32. */ + const int32_t t = mld_cast_uint32_to_int32(a_inverted); + + int64_t r; + + mld_assert(a < +(INT64_MAX - (((int64_t)1 << 31) * MLDSA_Q)) && + a > -(INT64_MAX - (((int64_t)1 << 31) * MLDSA_Q))); + + r = a - (int64_t)t * MLDSA_Q; + + /* + * PORTABILITY: Right-shift on a signed integer is, strictly-speaking, + * implementation-defined for negative left argument. Here, + * we assume it's sign-preserving "arithmetic" shift right. (C99 6.5.7 (5)) + */ + r = r >> 32; + + /* Bounds: + * + * By construction of the Montgomery multiplication, by the time we + * compute r >> 32, r is divisible by 2^32, and hence + * + * |r >> 32| = |r| / 2^32 + * <= |a| / 2^32 + MLDSA_Q / 2 + * + * (In general, we would only have |x >> n| <= ceil(|x| / 2^n)). + * + * In particular, if |a| < 2^31 * MLDSA_Q, then |return_value| < MLDSA_Q. + */ + return (int32_t)r; +} + +/** + * For finite field element a with a <= 2^{31} - 2^{22} - 1, compute + * r congruent to a (mod MLDSA_Q) such that + * -MLD_REDUCE32_RANGE_MAX <= r < MLD_REDUCE32_RANGE_MAX. + * + * @param a Finite field element. + * + * @return r. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int32_t mld_reduce32(int32_t a) +__contract__( + requires(a <= MLD_REDUCE32_DOMAIN_MAX) + ensures(return_value >= -MLD_REDUCE32_RANGE_MAX) + ensures(return_value < MLD_REDUCE32_RANGE_MAX) +) +{ + int32_t t; + + t = (a + ((int32_t)1 << 22)) >> 23; + t = a - t * MLDSA_Q; + mld_assert((t - a) % MLDSA_Q == 0); + return t; +} + +/** + * Add MLDSA_Q if input coefficient is negative. + * + * @param a Finite field element. + * + * @return r. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int32_t mld_caddq(int32_t a) +__contract__( + requires(a > -MLDSA_Q) + requires(a < MLDSA_Q) + ensures(return_value >= 0) + ensures(return_value < MLDSA_Q) + ensures(return_value == ((a >= 0) ? a : (a + MLDSA_Q))) +) +{ + return mld_ct_sel_int32(a + MLDSA_Q, a, mld_ct_cmask_neg_i32(a)); +} + + +#endif /* !MLD_REDUCE_H */ diff --git a/cbits/mldsa/src/rounding.h b/cbits/mldsa/src/rounding.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/rounding.h @@ -0,0 +1,265 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + */ + +#ifndef MLD_ROUNDING_H +#define MLD_ROUNDING_H + +#include "cbmc.h" +#include "common.h" +#include "ct.h" +#include "debug.h" + +/* Parameter set namespacing + * This is to facilitate building multiple instances + * of mldsa-native (e.g. with varying parameter sets) + * within a single compilation unit. */ +#define mld_power2round MLD_ADD_PARAM_SET(mld_power2round) +#define mld_decompose MLD_ADD_PARAM_SET(mld_decompose) +#define mld_make_hint MLD_ADD_PARAM_SET(mld_make_hint) +#define mld_use_hint MLD_ADD_PARAM_SET(mld_use_hint) +/* End of parameter set namespacing */ + +#define MLD_2_POW_D (1 << MLDSA_D) + +/** + * For finite field element a, compute a0, a1 such that + * a mod^+ MLDSA_Q = a1*2^MLDSA_D + a0 with + * -2^{MLDSA_D-1} < a0 <= 2^{MLDSA_D-1}. Assumes a to be standard + * representative. + * + * @spec{Implements @[FIPS204, Algorithm 35, Power2Round].} + * + * @reference{In the reference implementation, a1 is passed as a return value + * instead.} + * + * @param[out] a0 Pointer to output element a0. + * @param[out] a1 Pointer to output element a1. + * @param a Input element. + */ +static MLD_INLINE void mld_power2round(int32_t *a0, int32_t *a1, int32_t a) +__contract__( + requires(memory_no_alias(a0, sizeof(int32_t))) + requires(memory_no_alias(a1, sizeof(int32_t))) + requires(a >= 0 && a < MLDSA_Q) + assigns(memory_slice(a0, sizeof(int32_t))) + assigns(memory_slice(a1, sizeof(int32_t))) + ensures(*a0 > -(MLD_2_POW_D/2) && *a0 <= (MLD_2_POW_D/2)) + ensures(*a1 >= 0 && *a1 <= (MLDSA_Q - 1) / MLD_2_POW_D) + ensures((*a1 * MLD_2_POW_D + *a0 - a) % MLDSA_Q == 0) +) +{ + *a1 = (a + (1 << (MLDSA_D - 1)) - 1) >> MLDSA_D; + *a0 = a - (*a1 << MLDSA_D); +} + +/** + * For finite field element a, compute high and low bits a0, a1 such that + * a mod^+ MLDSA_Q = a1 * 2 * MLDSA_GAMMA2 + a0 with + * -MLDSA_GAMMA2 < a0 <= MLDSA_GAMMA2 except if + * a1 = (MLDSA_Q-1)/(MLDSA_GAMMA2*2) where we set a1 = 0 and + * -MLDSA_GAMMA2 <= a0 = a mod^+ MLDSA_Q - MLDSA_Q < 0. Assumes a to be + * standard representative. + * + * @spec{Implements @[FIPS204, Algorithm 36, Decompose].} + * + * @reference{In the reference implementation, a1 is passed as a return value + * instead.} + * + * @param[out] a0 Pointer to output element a0. + * @param[out] a1 Pointer to output element a1. + * @param a Input element. + */ +static MLD_INLINE void mld_decompose(int32_t *a0, int32_t *a1, int32_t a) +__contract__( + requires(memory_no_alias(a0, sizeof(int32_t))) + requires(memory_no_alias(a1, sizeof(int32_t))) + requires(a >= 0 && a < MLDSA_Q) + assigns(memory_slice(a0, sizeof(int32_t))) + assigns(memory_slice(a1, sizeof(int32_t))) + /* a0 = -MLDSA_GAMMA2 occurs exactly when a = MLDSA_Q - MLDSA_GAMMA2: the + * border case of Decompose where a1 = (MLDSA_Q-1)/(2*MLDSA_GAMMA2) is + * wrapped to 0 and a0 = a - MLDSA_Q (@[FIPS204, Algorithm 36, Decompose]) */ + ensures(*a0 >= -MLDSA_GAMMA2 && *a0 <= MLDSA_GAMMA2) + ensures(*a1 >= 0 && *a1 < (MLDSA_Q-1)/(2*MLDSA_GAMMA2)) + ensures((*a1 * 2 * MLDSA_GAMMA2 + *a0 - a) % MLDSA_Q == 0) +) +{ + /* + * The goal is to compute f1 = round-(f / (2*GAMMA2)), which can be computed + * alternatively as round-(f / (128B)) = round-(ceil(f / 128) / B) where + * B = 2*GAMMA2 / 128. Here round-() denotes "round half down". + * + * The equality round-(f / (128B)) = round-(ceil(f / 128) / B) can deduced + * as follows. Since changing f to align-up(f, 128) can move f onto but not + * across a rounding boundary for division by 128*B (note that we need B to be + * even for this to work), and round- rounds down on the boundary, we have + * + * round-(f / (128B)) = round-(align-up(f, 128) / (128B)) + * = round-((align-up(f, 128) / 128) / B) + * = round-(ceil(f / 128) / B). + */ + *a1 = (a + 127) >> 7; + /* We know a >= 0 and a < MLDSA_Q, so... */ + /* check-magic: 65472 == round((MLDSA_Q-1)/128) */ + mld_assert(*a1 >= 0 && *a1 <= 65472); + +#if MLD_CONFIG_PARAMETER_SET == 44 + /* check-magic: 1488 == 2 * intdiv(intdiv(MLDSA_Q - 1, 88), 128) */ + /* check-magic: 11275 == floor(2**24 / 1488) */ + /* check-magic: 1560281088 == 1 / (1 / 1488 - 11275 / 2**24) */ + /* + * Compute f1 = round-(f1' / B) ≈ round(f1' * 11275 / 2^24). This is exact for + * 0 <= f1' < 2^16. + * + * To see this, consider the (signed) error f1' * (1 / B - 11275 / 2^24) + * between f1' / B and the (under-)approximation f1' * 11275 / 2^24. Because + * eps := 1 / B - 11275 / 2^24 is 1 / 1560281088 ≈ 2^(-30.54) < 2^(-30), we + * have 0 <= f1' * eps < 2^16 * 2^(-30) = 1 / 2^14 < 1 / 2^11 < 1 / B (note + * that f1' is non-negative). + * + * On the other hand, 1 / B is the spacing between the integral multiples + * of 1 / B, which includes all rounding boundaries n + 0.5 (since B is even). + * Hence, if f1' / B is not of the form n + 0.5, then it is at least 1 / B + * away from the nearest rounding boundary, so moving from f1' / B to + * f1' * 11275 / 2^24 does not affect the rounding result, no matter the type + * of rounding used in either side. In particular, we have round-(f1' / B) = + * round(f1' * 11275 / 2^24) as claimed. + * + * As for the remaining case where f1' / B _is_ of the form n + 0.5, because + * f1' * 11275 / 2^24 is slightly but strictly below f1' / B = n + 0.5 (note + * that f1' and thus the error f1' * eps cannot be 0 here), it is always + * rounded down to n. More precisely, we have round-(f1' / B) = + * round(f1' * 11275 / 2^24), where the round-down on the LHS is essential, + * and on the RHS the type of rounding again does not matter. This concludes + * the proof. + * + * See proofs/isabelle/compress for a formalization of the above argument. + */ + *a1 = (*a1 * 11275 + ((int32_t)1 << 23)) >> 24; + mld_assert(*a1 >= 0 && *a1 <= 44); + + *a1 = mld_ct_sel_int32(0, *a1, mld_ct_cmask_neg_i32(43 - *a1)); + mld_assert(*a1 >= 0 && *a1 <= 43); +#else /* MLD_CONFIG_PARAMETER_SET == 44 */ + /* check-magic: 4092 == 2 * intdiv(intdiv(MLDSA_Q - 1, 32), 128) */ + /* check-magic: 1025 == floor(2**22 / 4092) */ + /* check-magic: 4290772992 == 1 / (1 / 4092 - 1025 / 2**22) */ + /* + * Compute f1 = round-(f1' / B) ≈ round(f1' * 1025 / 2^22). This is exact for + * 0 <= f1' < 2^16. Following the same argument above, it suffices to show + * that f1' * eps < 1 / B, where eps := 1 / B - 1025 / 2^22. Indeed, we have + * eps = 1 / 4290772992 ≈ 2^(-31.99) < 2^(-31), therefore f1' * eps < + * 2^16 * 2^(-31) = 1 / 2^15 < 1 / 2^12 < 1 / B. + */ + *a1 = (*a1 * 1025 + ((int32_t)1 << 21)) >> 22; + mld_assert(*a1 >= 0 && *a1 <= 16); + + *a1 &= 15; + mld_assert(*a1 >= 0 && *a1 <= 15); + +#endif /* MLD_CONFIG_PARAMETER_SET != 44 */ + + *a0 = a - *a1 * 2 * MLDSA_GAMMA2; + *a0 = mld_ct_sel_int32(*a0 - MLDSA_Q, *a0, + mld_ct_cmask_neg_i32((MLDSA_Q - 1) / 2 - *a0)); +} + +/** + * Decide a single hint bit from the low part a0 and high part a1 of a + * coefficient: return 1 unless a0 lies in the range (-GAMMA2, GAMMA2] that + * LowBits would produce, with the boundary value -GAMMA2 also admitted when + * a1 == 0 (the Decompose border case). + * + * @note This is not a line-for-line implementation of FIPS 204's MakeHint(z, r) + * (@[FIPS204, Algorithm 39, MakeHint]), which takes two ring elements and + * returns [[HighBits(r) != HighBits(r + z)]]. Instead, it takes the already + * decomposed low/high parts (a0, a1) of a coefficient and decides the hint bit + * from them directly. As explained in the block comment of + * mld_attempt_signature_generation (sign.c), for the specific values that arise + * during signing -- a0 = w0 - cs2 + ct0 and a1 = w1 = HighBits(w) -- this is + * equivalent to the spec's MakeHint(-ct0, w - cs2 + ct0) coefficient-wise. + * Because it consumes (a0, a1) rather than (z, r), it relies on the caller + * having computed a compatible decomposition. + * + * @param a0 Low bits of input element. + * @param a1 High bits of input element. + * + * @return 1 if overflow, 0 otherwise. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE unsigned int mld_make_hint(int32_t a0, int32_t a1) +__contract__( + ensures(return_value >= 0 && return_value <= 1) + ensures(return_value == (a0 > MLDSA_GAMMA2 || a0 < -MLDSA_GAMMA2 || + (a0 == -MLDSA_GAMMA2 && a1 != 0))) +) +{ + if (a0 > MLDSA_GAMMA2 || a0 < -MLDSA_GAMMA2 || + (a0 == -MLDSA_GAMMA2 && a1 != 0)) + { + return 1; + } + + return 0; +} + +/** + * Correct high bits according to hint. + * + * @spec{Implements @[FIPS204, Algorithm 40, UseHint].} + * + * @param a Input element. + * @param hint Hint bit. + * + * @return Corrected high bits. + */ +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int32_t mld_use_hint(int32_t a, int32_t hint) +__contract__( + requires(hint >= 0 && hint <= 1) + requires(a >= 0 && a < MLDSA_Q) + ensures(return_value >= 0 && return_value < (MLDSA_Q-1)/(2*MLDSA_GAMMA2)) +) +{ + int32_t a0, a1; + + mld_decompose(&a0, &a1, a); + if (hint == 0) + { + return a1; + } + +#if MLD_CONFIG_PARAMETER_SET == 44 + if (a0 > 0) + { + return (a1 == 43) ? 0 : a1 + 1; + } + else + { + return (a1 == 0) ? 43 : a1 - 1; + } +#else /* MLD_CONFIG_PARAMETER_SET == 44 */ + if (a0 > 0) + { + return (a1 + 1) & 15; + } + else + { + return (a1 - 1) & 15; + } +#endif /* MLD_CONFIG_PARAMETER_SET != 44 */ +} + + +#endif /* !MLD_ROUNDING_H */ diff --git a/cbits/mldsa/src/sign.c b/cbits/mldsa/src/sign.c new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/sign.c @@ -0,0 +1,1720 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS140_3_IG] + * Implementation Guidance for FIPS 140-3 and the Cryptographic Module + * Validation Program + * National Institute of Standards and Technology + * https://csrc.nist.gov/projects/cryptographic-module-validation-program/fips-140-3-ig-announcements + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + * + * - [FIPS204_UPDATES] + * FIPS 204 Potential Updates (Errata) + * National Institute of Standards and Technology + * https://csrc.nist.gov/files/pubs/fips/204/final/docs/fips-204-potential-updates.xlsx + * + * - [Round3_Spec] + * CRYSTALS-Dilithium Algorithm Specifications and Supporting Documentation + * (Version 3.1) + * Bai, Ducas, Kiltz, Lepoint, Lyubashevsky, Schwabe, Seiler, Stehlé + * https://pq-crystals.org/dilithium/data/dilithium-specification-round3-20210208.pdf + */ + +#include "sign.h" + +#include "cbmc.h" +#include "ct.h" +#include "debug.h" +#include "packing.h" +#include "poly.h" +#include "poly_kl.h" +#include "polyvec.h" +#include "randombytes.h" +#include "symmetric.h" + +/* Parameter set namespacing + * This is to facilitate building multiple instances + * of mldsa-native (e.g. with varying parameter sets) + * within a single compilation unit. */ +#define mld_check_pct MLD_ADD_PARAM_SET(mld_check_pct) MLD_CONTEXT_PARAMETERS_2 +#define mld_sample_s1_s2 MLD_ADD_PARAM_SET(mld_sample_s1_s2) +#define mld_validate_hash_length MLD_ADD_PARAM_SET(mld_validate_hash_length) +#define mld_get_hash_oid MLD_ADD_PARAM_SET(mld_get_hash_oid) +#define mld_H MLD_ADD_PARAM_SET(mld_H) +#define mld_compute_pack_z MLD_ADD_PARAM_SET(mld_compute_pack_z) +#define mld_attempt_signature_generation \ + MLD_ADD_PARAM_SET(mld_attempt_signature_generation) MLD_CONTEXT_PARAMETERS_8 +#define mld_compute_pack_t0_t1 \ + MLD_ADD_PARAM_SET(mld_compute_pack_t0_t1) MLD_CONTEXT_PARAMETERS_5 +#define mld_get_max_signing_attempts \ + MLD_ADD_PARAM_SET(mld_get_max_signing_attempts) + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +static int mld_check_pct(uint8_t const pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + uint8_t const sk[MLDSA_CRYPTO_SECRETKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES)) + requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) + ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY || + return_value == MLD_ERR_RNG_FAIL || + return_value == MLD_ERR_SIGNING_PAUSED || + return_value == MLD_ERR_PCT_FAIL) +); + +#if defined(MLD_CONFIG_KEYGEN_PCT) +/** + * Pair-wise Consistency Test (PCT) for DSA keypairs. + * + * @[FIPS140_3_IG] TE10.35.02 + * (https://csrc.nist.gov/csrc/media/Projects/cryptographic-module-validation-program/documents/fips%20140-3/FIPS%20140-3%20IG.pdf). + * + * Validates that a generated public/private key pair can correctly sign and + * verify data. Performs signature generation using the private key (sk), + * followed by signature verification using the public key (pk). + * + * @note @[FIPS204] requires that public/private key pairs are to be used + * only for the calculation and/or verification of digital signatures. + * + * @param[in] pk Public key. + * @param[in] sk Secret key. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was used and an + * allocation via MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_RNG_FAIL Random number generation failed. + * @retval MLD_ERR_SIGNING_PAUSED The PCT's signing step was paused by a + * MLD_CONFIG_SIGN_HOOK_ATTEMPT hook. + * @retval MLD_ERR_PCT_FAIL The consistency check failed. + */ +static int mld_check_pct(uint8_t const pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + uint8_t const sk[MLDSA_CRYPTO_SECRETKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + MLD_ALIGN uint8_t message[1] = {0}; + int ret; + MLD_ALLOC(signature, uint8_t, MLDSA_CRYPTO_BYTES, context); + MLD_ALLOC(pk_test, uint8_t, MLDSA_CRYPTO_PUBLICKEYBYTES, context); + + if (signature == NULL || pk_test == NULL) + { + ret = MLD_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + /* Copy public key for testing */ + mld_memcpy(pk_test, pk, MLDSA_CRYPTO_PUBLICKEYBYTES); + + /* Sign a test message using the original secret key */ + ret = mld_sign_signature(signature, message, sizeof(message), NULL, 0, sk, + context); + if (ret != 0) + { + goto cleanup; + } + +#if defined(MLD_CONFIG_KEYGEN_PCT_BREAKAGE_TEST) + /* Deliberately break public key for testing purposes */ + if (mld_break_pct()) + { + pk_test[0] = ~pk_test[0]; + } +#endif /* MLD_CONFIG_KEYGEN_PCT_BREAKAGE_TEST */ + + /* Verify the signature using the (potentially corrupted) public key. */ + ret = mld_sign_verify(signature, message, sizeof(message), NULL, 0, pk_test, + context); + if (ret != 0) + { + goto cleanup; + } + +cleanup: + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + MLD_FREE(pk_test, uint8_t, MLDSA_CRYPTO_PUBLICKEYBYTES, context); + MLD_FREE(signature, uint8_t, MLDSA_CRYPTO_BYTES, context); + + /* A failed signing operation or an invalid signature hint at a faulty + * implementation and map to a dedicated error code for PCT failure. */ + if (ret == MLD_ERR_INVALID_SIGNATURE || + ret == MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED) + { + ret = MLD_ERR_PCT_FAIL; + } + + /* Other error codes, e.g. platform failures like out of memory or + * randomness failure, are passed on unmodified. */ + + return ret; +} +#else /* MLD_CONFIG_KEYGEN_PCT */ +static int mld_check_pct(uint8_t const pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + uint8_t const sk[MLDSA_CRYPTO_SECRETKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + /* Skip PCT */ + ((void)pk); + ((void)sk); + MLD_CONTEXT_UNUSED(context); + return 0; +} +#endif /* !MLD_CONFIG_KEYGEN_PCT */ + +/** + * Sample the short secret vectors s1 (length MLDSA_L) and s2 (length MLDSA_K) + * with coefficients in [-MLDSA_ETA, MLDSA_ETA] from the seed. + * + * @spec{Implements @[FIPS204, Algorithm 33, ExpandS].} + * + * @param[out] s1 Output vector s1. + * @param[out] s2 Output vector s2. + * @param[in] seed Byte array with seed of length MLDSA_CRHBYTES. + */ +static void mld_sample_s1_s2(mld_polyvecl *s1, mld_polyveck *s2, + const uint8_t seed[MLDSA_CRHBYTES]) +__contract__( + requires(memory_no_alias(s1, sizeof(mld_polyvecl))) + requires(memory_no_alias(s2, sizeof(mld_polyveck))) + requires(memory_no_alias(seed, MLDSA_CRHBYTES)) + assigns(object_whole(s1), object_whole(s2)) + ensures(forall(l0, 0, MLDSA_L, array_abs_bound(s1->vec[l0].coeffs, 0, MLDSA_N, MLDSA_ETA + 1))) + ensures(forall(k0, 0, MLDSA_K, array_abs_bound(s2->vec[k0].coeffs, 0, MLDSA_N, MLDSA_ETA + 1))) +) +{ +/* Sample short vectors s1 and s2 */ +#if defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) + int i; + uint16_t nonce = 0; + /* Safety: The nonces are at most 14 (MLDSA_L + MLDSA_K - 1), and, hence, the + * casts are safe. */ + for (i = 0; i < MLDSA_L; i++) + { + mld_poly_uniform_eta(&s1->vec[i], seed, (uint8_t)(nonce + i)); + } + for (i = 0; i < MLDSA_K; i++) + { + mld_poly_uniform_eta(&s2->vec[i], seed, (uint8_t)(nonce + MLDSA_L + i)); + } +#else /* MLD_CONFIG_SERIAL_FIPS202_ONLY */ +#if MLD_CONFIG_PARAMETER_SET == 44 + mld_poly_uniform_eta_4x(&s1->vec[0], &s1->vec[1], &s1->vec[2], &s1->vec[3], + seed, 0, 1, 2, 3); + mld_poly_uniform_eta_4x(&s2->vec[0], &s2->vec[1], &s2->vec[2], &s2->vec[3], + seed, 4, 5, 6, 7); +#elif MLD_CONFIG_PARAMETER_SET == 65 + mld_poly_uniform_eta_4x(&s1->vec[0], &s1->vec[1], &s1->vec[2], &s1->vec[3], + seed, 0, 1, 2, 3); + mld_poly_uniform_eta_4x(&s1->vec[4], &s2->vec[0], &s2->vec[1], + &s2->vec[2] /* irrelevant */, seed, 4, 5, 6, + 0xFF /* irrelevant */); + mld_poly_uniform_eta_4x(&s2->vec[2], &s2->vec[3], &s2->vec[4], &s2->vec[5], + seed, 7, 8, 9, 10); +#elif MLD_CONFIG_PARAMETER_SET == 87 + mld_poly_uniform_eta_4x(&s1->vec[0], &s1->vec[1], &s1->vec[2], &s1->vec[3], + seed, 0, 1, 2, 3); + mld_poly_uniform_eta_4x(&s1->vec[4], &s1->vec[5], &s1->vec[6], + &s2->vec[0] /* irrelevant */, seed, 4, 5, 6, + 0xFF /* irrelevant */); + mld_poly_uniform_eta_4x(&s2->vec[0], &s2->vec[1], &s2->vec[2], &s2->vec[3], + seed, 7, 8, 9, 10); + mld_poly_uniform_eta_4x(&s2->vec[4], &s2->vec[5], &s2->vec[6], &s2->vec[7], + seed, 11, 12, 13, 14); +#endif /* MLD_CONFIG_PARAMETER_SET == 87 */ +#endif /* !MLD_CONFIG_SERIAL_FIPS202_ONLY */ +} + +/** + * Compute t = A*s1hat + s2 row by row, decompose each row into t0[k] and + * t1[k] via power2round, and bit-pack t1[k] into pk_t1 and t0[k] into the + * t0_packed buffer. Used by both keygen and pk_from_sk. + * + * @spec{Partially implements @[FIPS204, Algorithm 22, pkEncode] (t1) and + * @[FIPS204, Algorithm 24, skEncode] (t0).} + * + * @param[out] pk_t1 Output buffer for packed t1 (size + * MLDSA_K * MLDSA_POLYT1_PACKEDBYTES; i.e. the t1 + * region of pk). + * @param[out] t0_packed Output buffer for packed t0 (size + * MLDSA_K * MLDSA_POLYT0_PACKEDBYTES). + * @param[in] s1hat s1 in NTT domain. + * @param[in] s2 s2. + * @param[in] rho Byte array containing seed rho. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @return - 0: Success. + * - MLD_ERR_OUT_OF_MEMORY: If MLD_CONFIG_CUSTOM_ALLOC_FREE is used and + * an allocation via MLD_CUSTOM_ALLOC returned NULL. + */ +MLD_MUST_CHECK_RETURN_VALUE +static int mld_compute_pack_t0_t1( + uint8_t pk_t1[MLDSA_K * MLDSA_POLYT1_PACKEDBYTES], + uint8_t t0_packed[MLDSA_K * MLDSA_POLYT0_PACKEDBYTES], + const mld_polyvecl *s1hat, const mld_polyveck *s2, + const uint8_t rho[MLDSA_SEEDBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(pk_t1, MLDSA_K * MLDSA_POLYT1_PACKEDBYTES)) + requires(memory_no_alias(t0_packed, MLDSA_K * MLDSA_POLYT0_PACKEDBYTES)) + requires(memory_no_alias(s1hat, sizeof(mld_polyvecl))) + requires(memory_no_alias(s2, sizeof(mld_polyveck))) + requires(memory_no_alias(rho, MLDSA_SEEDBYTES)) + requires(forall(l1, 0, MLDSA_L, + array_abs_bound(s1hat->vec[l1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + requires(forall(k2, 0, MLDSA_K, + array_bound(s2->vec[k2].coeffs, 0, MLDSA_N, + MLD_POLYETA_UNPACK_LOWER_BOUND, MLDSA_ETA + 1))) + assigns(memory_slice(pk_t1, MLDSA_K * MLDSA_POLYT1_PACKEDBYTES)) + assigns(memory_slice(t0_packed, MLDSA_K * MLDSA_POLYT0_PACKEDBYTES)) + ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY)) +{ + unsigned int k; + int ret; + MLD_ALLOC(mat, mld_polymat, 1, context); + MLD_ALLOC(t0k, mld_poly, 1, context); + MLD_ALLOC(t1k, mld_poly, 1, context); + + if (mat == NULL || t0k == NULL || t1k == NULL) + { + ret = MLD_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + /* Expand matrix */ + mld_polyvec_matrix_expand(mat, rho); + + for (k = 0; k < MLDSA_K; k++) + __loop__( + assigns(k, memory_slice(pk_t1, MLDSA_K * MLDSA_POLYT1_PACKEDBYTES), + memory_slice(t0_packed, MLDSA_K * MLDSA_POLYT0_PACKEDBYTES), + memory_slice(t0k, sizeof(mld_poly)), + memory_slice(t1k, sizeof(mld_poly)) + MLD_IF_REDUCE_RAM(, memory_slice(mat, sizeof(mld_polymat)))) + invariant(k <= MLDSA_K) + decreases(MLDSA_K - k) + ) + { + /* t0k = (A * s1hat)_k in NTT domain */ + mld_polyvec_matrix_pointwise_montgomery_row(t0k, mat, s1hat, k); + + /* t0k = invNTT(t0k) */ + mld_poly_invntt_tomont(t0k); + + /* t0k += s2[k] */ + mld_poly_add(t0k, &s2->vec[k]); + + /* Reference: The following reduction is not present in the reference + * implementation. Omitting this reduction requires the output + * of the invntt to be small enough such that the addition of + * s2 does not result in absolute values >= MLDSA_Q. While our + * C, x86_64, and AArch64 invntt implementations produce small + * enough values for this to work out, it complicates the + * bounds reasoning. We instead add an additional reduction, + * and can consequently, relax the bounds requirements for the + * invntt. + */ + mld_poly_reduce(t0k); + + /* Decompose into t1[k] and t0[k] (in place into t0k). */ + mld_poly_caddq(t0k); + mld_poly_power2round(t1k, t0k, t0k); + + /* Pack t1[k] into pk and t0[k] into the t0 output buffer. */ + mld_polyt1_pack(pk_t1 + k * MLDSA_POLYT1_PACKEDBYTES, t1k); + mld_polyt0_pack(t0_packed + k * MLDSA_POLYT0_PACKEDBYTES, t0k); + } + + ret = 0; +cleanup: + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + MLD_FREE(t1k, mld_poly, 1, context); + MLD_FREE(t0k, mld_poly, 1, context); + MLD_FREE(mat, mld_polymat, 1, context); + return ret; +} + +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_keypair_internal(uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + const uint8_t seed[MLDSA_SEEDBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret; + const uint8_t *rho, *rhoprime, *key; + + MLD_ALLOC(seedbuf, uint8_t, 2 * MLDSA_SEEDBYTES + MLDSA_CRHBYTES, context); + MLD_ALLOC(inbuf, uint8_t, MLDSA_SEEDBYTES + 2, context); + MLD_ALLOC(tr, uint8_t, MLDSA_TRBYTES, context); + MLD_ALLOC(s1, mld_polyvecl, 1, context); + MLD_ALLOC(s2, mld_polyveck, 1, context); + + if (seedbuf == NULL || inbuf == NULL || tr == NULL || s1 == NULL || + s2 == NULL) + { + ret = MLD_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + /* Get randomness for rho, rhoprime and key */ + mld_memcpy(inbuf, seed, MLDSA_SEEDBYTES); + inbuf[MLDSA_SEEDBYTES + 0] = MLDSA_K; + inbuf[MLDSA_SEEDBYTES + 1] = MLDSA_L; + mld_shake256(seedbuf, 2 * MLDSA_SEEDBYTES + MLDSA_CRHBYTES, inbuf, + MLDSA_SEEDBYTES + 2); + rho = seedbuf; + rhoprime = rho + MLDSA_SEEDBYTES; + key = rhoprime + MLDSA_CRHBYTES; + + /* Constant time: rho is part of the public key and, hence, public. */ + MLD_CT_TESTING_DECLASSIFY(rho, MLDSA_SEEDBYTES); + + /* Sample s1 and s2 */ + mld_sample_s1_s2(s1, s2, rhoprime); + + /* Pack s1 into sk before NTT */ + mld_pack_sk_s1(sk, s1); + + /* NTT s1 in place to use as s1hat */ + mld_polyvecl_ntt(s1); + + /* Pack rho into pk */ + mld_memcpy(pk + MLDSA_PK_RHO_OFFSET, rho, MLDSA_SEEDBYTES); + + /* Compute t = A*s1hat + s2 row by row, decompose into t1/t0, and pack + * t1 into pk and t0 directly into the t0 region of sk. */ + ret = mld_compute_pack_t0_t1(pk + MLDSA_PK_T1_OFFSET, sk + MLDSA_SK_T0_OFFSET, + s1, s2, rho, context); + if (ret != 0) + { + goto cleanup; + } + + /* Compute tr = H(pk) */ + mld_shake256(tr, MLDSA_TRBYTES, pk, MLDSA_CRYPTO_PUBLICKEYBYTES); + + /* Pack remaining secret key components (s1 and t0 already packed) */ + mld_pack_sk_rho_key_tr_s2(sk, rho, tr, key, s2); + + /* Constant time: pk is the public key, inherently public data */ + MLD_CT_TESTING_DECLASSIFY(pk, MLDSA_CRYPTO_PUBLICKEYBYTES); + +cleanup: + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + MLD_FREE(s2, mld_polyveck, 1, context); + MLD_FREE(s1, mld_polyvecl, 1, context); + MLD_FREE(tr, uint8_t, MLDSA_TRBYTES, context); + MLD_FREE(inbuf, uint8_t, MLDSA_SEEDBYTES + 2, context); + MLD_FREE(seedbuf, uint8_t, 2 * MLDSA_SEEDBYTES + MLDSA_CRHBYTES, context); + + /* Pairwise Consistency Test (PCT) @[FIPS140_3_IG, p.87] */ + /* Do this after freeing all temporaries. */ + if (ret == 0) + { + ret = mld_check_pct(pk, sk, context); + } + + if (ret != 0) + { + /* Clear caller outputs on failure. */ + mld_zeroize(pk, MLDSA_CRYPTO_PUBLICKEYBYTES); + mld_zeroize(sk, MLDSA_CRYPTO_SECRETKEYBYTES); + } + + return ret; +} + +#if !defined(MLD_CONFIG_CORE_API_ONLY) +#if !defined(MLD_CONFIG_NO_RANDOMIZED_API) +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_keypair(uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret; + MLD_ALLOC(seed, uint8_t, MLDSA_SEEDBYTES, context); + + if (seed == NULL) + { + ret = MLD_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + if (mld_randombytes(seed, MLDSA_SEEDBYTES) != 0) + { + ret = MLD_ERR_RNG_FAIL; + goto cleanup; + } + MLD_CT_TESTING_SECRET(seed, MLDSA_SEEDBYTES); + ret = mld_sign_keypair_internal(pk, sk, seed, context); + +cleanup: + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + MLD_FREE(seed, uint8_t, MLDSA_SEEDBYTES, context); + return ret; +} +#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */ +#endif /* !MLD_CONFIG_CORE_API_ONLY */ +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +/** + * Abstracts application of SHAKE256 to one, two or three blocks of data, + * yielding a user-requested size of output. + * + * @param[out] out Pointer to output. + * @param outlen Requested output length in bytes. + * @param[in] in1 Pointer to input block 1. Must NOT be NULL. + * @param in1len Length of input in1 in bytes. + * @param[in] in2 Pointer to input block 2. May be NULL if in2len == 0, + * in which case this block is ignored. + * @param in2len Length of input in2 in bytes. + * @param[in] in3 Pointer to input block 3. May be NULL if in3len == 0, + * in which case this block is ignored. + * @param in3len Length of input in3 in bytes. + */ +static void mld_H(uint8_t *out, size_t outlen, const uint8_t *in1, + size_t in1len, const uint8_t *in2, size_t in2len, + const uint8_t *in3, size_t in3len) +__contract__( + requires(in1len <= MLD_MAX_BUFFER_SIZE) + requires(in2len <= MLD_MAX_BUFFER_SIZE) + requires(in3len <= MLD_MAX_BUFFER_SIZE) + requires(outlen <= 8 * SHAKE256_RATE /* somewhat arbitrary bound */) + requires(memory_no_alias(in1, in1len)) + requires(in2len == 0 || memory_no_alias(in2, in2len)) + requires(in3len == 0 || memory_no_alias(in3, in3len)) + requires(memory_no_alias(out, outlen)) + assigns(memory_slice(out, outlen)) +) +{ + mld_shake256ctx state; + mld_shake256_init(&state); + mld_shake256_absorb(&state, in1, in1len); + if (in2len != 0) + { + mld_shake256_absorb(&state, in2, in2len); + } + if (in3len != 0) + { + mld_shake256_absorb(&state, in3, in3len); + } + mld_shake256_finalize(&state); + mld_shake256_squeeze(out, outlen, &state); + mld_shake256_release(&state); + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(&state, sizeof(state)); +} +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +/* MLD_MAX_KAPPA (see params.h) bounds the rejection-sampling counter kappa; + * MLD_MAX_SIGNING_ATTEMPTS below turns that into a bound on attempts. */ + +/** + * Compute z = y + s1*c, check that z has coefficients smaller than + * MLDSA_GAMMA1 - MLDSA_BETA, and pack z into the signature buffer. + * + * @reference{This function is inlined into mld_sign_signature in the + * reference implementation.} + * + * @param[in,out] sig Output signature. + * @param[in] cp Challenge polynomial. + * @param[in] s1hat Secret vector s1 in NTT domain. + * @param[in] y Masking vector y (or seed in REDUCE_RAM mode). + * @param[out] z Scratch polynomial for z computation. + * @param[out] tmp Scratch polynomial. + * + * @return - 0: Success (z has coefficients smaller than + * MLDSA_GAMMA1 - MLDSA_BETA). + * - MLD_ERR_FAIL: z rejected (norm check failed). + * - MLD_ERR_OUT_OF_MEMORY: If MLD_CONFIG_CUSTOM_ALLOC_FREE is used and + * an allocation via MLD_CUSTOM_ALLOC returned NULL. + */ +MLD_MUST_CHECK_RETURN_VALUE +static int mld_compute_pack_z(uint8_t sig[MLDSA_CRYPTO_BYTES], + const mld_poly *cp, const mld_sk_s1hat *s1hat, + const mld_yvec *y, mld_poly *z, mld_poly *tmp) +__contract__( + requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(cp, sizeof(mld_poly))) + requires(memory_no_alias(s1hat, sizeof(mld_sk_s1hat))) + requires(memory_no_alias(y, sizeof(mld_yvec))) + requires(memory_no_alias(z, sizeof(mld_poly))) + requires(memory_no_alias(tmp, sizeof(mld_poly))) + requires(array_abs_bound(cp->coeffs, 0, MLDSA_N, MLD_NTT_BOUND)) + MLD_IF_NOT_REDUCE_RAM( + requires(forall(k0, 0, MLDSA_L, + array_bound(y->vec.vec[k0].coeffs, 0, MLDSA_N, -(MLDSA_GAMMA1 - 1), MLDSA_GAMMA1 + 1))) + requires(forall(k1, 0, MLDSA_L, array_abs_bound(s1hat->vec.vec[k1].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + ) + MLD_IF_REDUCE_RAM( + requires(memory_no_alias(s1hat->packed, MLDSA_L * MLDSA_POLYETA_PACKEDBYTES)) + requires(memory_no_alias(y->rhoprime, MLDSA_CRHBYTES)) + requires(y->kappa <= MLD_MAX_KAPPA) + ) + assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES)) + assigns(memory_slice(z, sizeof(mld_poly))) + assigns(memory_slice(tmp, sizeof(mld_poly))) + ensures(return_value == 0 || return_value == MLD_ERR_FAIL || + return_value == MLD_ERR_OUT_OF_MEMORY) +) +{ + unsigned int i; + uint32_t z_invalid; + for (i = 0; i < MLDSA_L; i++) + __loop__( + assigns(i, memory_slice(z, sizeof(mld_poly)), + memory_slice(tmp, sizeof(mld_poly)), + memory_slice(sig, MLDSA_CRYPTO_BYTES)) + invariant(i <= MLDSA_L) + decreases(MLDSA_L - i) + ) + { + mld_sk_s1hat_get_poly(z, s1hat, i); + mld_poly_pointwise_montgomery(z, cp); + mld_poly_invntt_tomont(z); + mld_yvec_get_poly(tmp, y, i); + mld_poly_add(z, tmp); + mld_poly_reduce(z); + + z_invalid = mld_poly_chknorm(z, MLDSA_GAMMA1 - MLDSA_BETA); + /* Constant time: It is fine (and prohibitively expensive to avoid) + * to leak the result of the norm check and which polynomial in z caused a + * rejection. It would even be okay to leak which coefficient led to + * rejection as the candidate signature will be discarded anyway. + * See Section 5.5 of @[Round3_Spec]. */ + MLD_CT_TESTING_DECLASSIFY(&z_invalid, sizeof(uint32_t)); + if (z_invalid) + { + return MLD_ERR_FAIL; /* reject */ + } + /* If z is valid, then its coefficients are bounded by + * MLDSA_GAMMA1 - MLDSA_BETA. This will be needed below + * to prove the pre-condition of pack_sig_z() */ + mld_assert_abs_bound(z, MLDSA_N, (MLDSA_GAMMA1 - MLDSA_BETA)); + + /* After the norm check, the distribution of each coefficient of z is + * independent of the secret key and it can, hence, be considered + * public. It is, hence, okay to immediately pack it into the user-provided + * signature buffer. */ + mld_pack_sig_z(sig, z, i); + } + return 0; +} + +/* Effective bound on signing attempts: the configured bound + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS (see mldsa_native_config.h) if set, otherwise + * the hard type-safety bound MLD_MAX_KAPPA / MLDSA_L (see MLD_MAX_KAPPA in + * params.h). */ +#if defined(MLD_CONFIG_MAX_SIGNING_ATTEMPTS) + +#if !defined(MLD_ALLOW_NONCOMPLIANT_SIGNING_BOUND) && \ + MLD_CONFIG_MAX_SIGNING_ATTEMPTS < 821 +#error Bad configuration: MLD_CONFIG_MAX_SIGNING_ATTEMPTS must be >= 821 for FIPS 204 compliance @[FIPS204, Appendix C] @[FIPS204_UPDATES] +#endif + +#if MLD_CONFIG_MAX_SIGNING_ATTEMPTS < 1 +#error Bad configuration: MLD_CONFIG_MAX_SIGNING_ATTEMPTS must be >= 1 +#endif + +#if MLD_CONFIG_MAX_SIGNING_ATTEMPTS > MLD_MAX_KAPPA / MLDSA_L +#error Bad configuration: MLD_CONFIG_MAX_SIGNING_ATTEMPTS exceeds the maximum allowed value. +#endif + +#define MLD_MAX_SIGNING_ATTEMPTS MLD_CONFIG_MAX_SIGNING_ATTEMPTS +#else /* MLD_CONFIG_MAX_SIGNING_ATTEMPTS */ +#define MLD_MAX_SIGNING_ATTEMPTS (MLD_MAX_KAPPA / MLDSA_L) +#endif /* !MLD_CONFIG_MAX_SIGNING_ATTEMPTS */ + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE uint16_t mld_get_max_signing_attempts(void) +__contract__( + ensures(return_value >= 1) + ensures(return_value <= MLD_MAX_KAPPA / MLDSA_L) +) +{ + /* cassert(0) ensures CBMC uses the contract rather than inlining the body, + * keeping proofs agnostic of the configured value. */ + cassert(0); + return MLD_MAX_SIGNING_ATTEMPTS; +} + +/** + * Attempt to generate a single signature: one iteration of the + * ML-DSA.Sign_internal rejection-sampling loop. + * + * @spec{Implements one iteration of the rejection-sampling loop body of + * @[FIPS204, Algorithm 7, ML-DSA.Sign_internal] (lines 11-30) plus, on success, + * the sigEncode step (line 33). The per-signature setup (Algorithm 7 lines 1-7: + * skDecode, NTT of s1/s2/t0, ExpandA, and computation of mu and rhoprime) and + * the loop itself (lines 8-10, 31-32) live in the caller + * mld_sign_signature_internal; kappa is this iteration's counter, used to + * sample y.} + * + * @reference{This code differs from the reference implementation in that it + * factors out the core signature generation step into a distinct function + * here in order to improve efficiency of CBMC proof.} + * + * @param[out] sig Pointer to output signature. + * @param[in] mu Pointer to message or hash of exactly MLDSA_CRHBYTES + * bytes. + * @param[in] rhoprime Pointer to randomness seed. + * @param kappa Counter for this iteration (= attempt*MLDSA_L). + * @param[in] mat Expanded matrix. + * @param[in] s1hat Secret vector s1 in NTT domain. + * @param[in] s2hat Secret vector s2 in NTT domain. + * @param[in] t0hat Vector t0 in NTT domain. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @return - 0: Signature generation succeeded. + * - MLD_ERR_FAIL: Signature rejected (norm check failed). + * - MLD_ERR_OUT_OF_MEMORY: If MLD_CONFIG_CUSTOM_ALLOC_FREE is used and + * an allocation via MLD_CUSTOM_ALLOC returned NULL. + */ +MLD_MUST_CHECK_RETURN_VALUE +/* NOLINTNEXTLINE(readability-function-cognitive-complexity) */ +static int mld_attempt_signature_generation( + uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *mu, + const uint8_t rhoprime[MLDSA_CRHBYTES], uint16_t kappa, mld_polymat *mat, + const mld_sk_s1hat *s1hat, const mld_sk_s2hat *s2hat, + const mld_sk_t0hat *t0hat, MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(mu, MLDSA_CRHBYTES)) + requires(memory_no_alias(rhoprime, MLDSA_CRHBYTES)) + requires(memory_no_alias(mat, sizeof(mld_polymat))) + requires(memory_no_alias(s1hat, sizeof(mld_sk_s1hat))) + requires(memory_no_alias(s2hat, sizeof(mld_sk_s2hat))) + requires(memory_no_alias(t0hat, sizeof(mld_sk_t0hat))) + requires(kappa <= MLD_MAX_KAPPA) + MLD_IF_NOT_REDUCE_RAM( + requires(forall(k1, 0, MLDSA_K, forall(l1, 0, MLDSA_L, + array_bound(mat->vec[k1].vec[l1].coeffs, 0, MLDSA_N, 0, MLDSA_Q)))) + requires(forall(k2, 0, MLDSA_K, array_abs_bound(t0hat->vec.vec[k2].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + requires(forall(k3, 0, MLDSA_L, array_abs_bound(s1hat->vec.vec[k3].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + requires(forall(k4, 0, MLDSA_K, array_abs_bound(s2hat->vec.vec[k4].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + ) + MLD_IF_REDUCE_RAM( + requires(memory_no_alias(s1hat->packed, MLDSA_L * MLDSA_POLYETA_PACKEDBYTES)) + requires(memory_no_alias(s2hat->packed, MLDSA_K * MLDSA_POLYETA_PACKEDBYTES)) + requires(memory_no_alias(t0hat->packed, MLDSA_K * MLDSA_POLYT0_PACKEDBYTES)) + ) + assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES)) + MLD_IF_REDUCE_RAM( + assigns(memory_slice(mat, sizeof(mld_polymat))) + ) + ensures(return_value == 0 || return_value == MLD_ERR_FAIL || + return_value == MLD_ERR_OUT_OF_MEMORY) +) +{ + unsigned int k; + uint32_t w0_invalid, h_invalid; + int ret; + + typedef union + { + mld_polyveck w1; + mld_polyvecl tmp; + } w1tmp_u; + mld_polyveck *w1; + mld_polyvecl *tmp; + + MLD_ALLOC(challenge_bytes, uint8_t, MLDSA_CTILDEBYTES, context); + MLD_ALLOC(y, mld_yvec, 1, context); + MLD_ALLOC(z, mld_poly, 1, context); + MLD_ALLOC(w1tmp, w1tmp_u, 1, context); + MLD_ALLOC(w0, mld_polyveck, 1, context); + MLD_ALLOC(cp, mld_poly, 1, context); + MLD_ALLOC(t, mld_poly, 1, context); + + if (challenge_bytes == NULL || y == NULL || z == NULL || w1tmp == NULL || + w0 == NULL || cp == NULL || t == NULL) + { + ret = MLD_ERR_OUT_OF_MEMORY; + goto cleanup; + } + w1 = &w1tmp->w1; + tmp = &w1tmp->tmp; + + /* @[FIPS204, Algorithm 7, line 11] y <- ExpandMask(rhoprime, kappa). */ + mld_yvec_init(y, rhoprime, kappa); + + /* @[FIPS204, Algorithm 7, line 12] w <- invNTT(A_hat o NTT(y)). This call + * performs the whole line: it NTTs y, accumulates the pointwise product with + * A_hat, and applies the inverse NTT. In REDUCE_RAM mode the y sampling is + * fused into the same pass. */ + mld_polyvec_matrix_pointwise_montgomery_yvec(w0, mat, y, tmp); + + /* @[FIPS204, Algorithm 7, line 13] w1 <- HighBits(w), here together with the + * low part: Decompose yields w = 2*GAMMA2*w1 + w0, keeping both w1 and w0 + * (w0 is reused below in the line-21/26 alternative, see further down). */ + mld_polyveck_caddq(w0); + mld_polyveck_decompose(w1, w0); + + /* @[FIPS204, Algorithm 7, line 15] ctilde <- H(mu || w1Encode(w1), lambda/4). + * w1Encode(w1) is packed into the w1 region of sig (mld_polyveck_pack_w1), + * then absorbed by H together with mu. */ + mld_polyveck_pack_w1(sig, w1); + + mld_H(challenge_bytes, MLDSA_CTILDEBYTES, mu, MLDSA_CRHBYTES, sig, + MLDSA_K * MLDSA_POLYW1_PACKEDBYTES, NULL, 0); + /* Constant time: Leaking challenge_bytes does not reveal any information + * about the secret key as H() is modelled as random oracle. + * This also applies to challenges for rejected signatures. + * See Section 5.5 of @[Round3_Spec]. */ + MLD_CT_TESTING_DECLASSIFY(challenge_bytes, MLDSA_CTILDEBYTES); + /* @[FIPS204, Algorithm 7, line 16] c <- SampleInBall(ctilde) and + * @[FIPS204, Algorithm 7, line 17] c_hat <- NTT(c). */ + mld_poly_challenge(cp, challenge_bytes); + mld_poly_ntt(cp); + + /* @[FIPS204, Algorithm 7, lines 18+20] cs1 <- invNTT(c_hat o s1_hat) and + * z <- y + cs1, followed by the line-23 norm check ||z||_inf >= GAMMA1 - + * BETA. mld_compute_pack_z fuses all three per polynomial and, on success, + * packs z into sig; it returns MLD_ERR_FAIL if the norm check rejects z. */ + ret = mld_compute_pack_z(sig, cp, s1hat, y, t, z); + if (ret != 0) + { + goto cleanup; + } + + /* The remaining steps realize @[FIPS204, Algorithm 7, lines 21-28] (the + * low-bits norm check and the hint h) via the faster alternative formulation + * of @[Round3_Spec, Section 5.1]. @[FIPS204] explicitly permits this: the + * note accompanying Algorithm 7 states that the validity checks on z and the + * computation of h may instead be implemented "as described in Section 5.1 of + * [6]", and that reference is @[Round3_Spec, Section 5.1]. + * + * The loop below builds w0 - cs2 + ct0 in place in w0; w1 is unmodified, and + * is HighBits(w) from line 13. Those are the inputs to the streamlined + * computation of MakeHint explained below. + * + * Low-bits norm check: + * @[FIPS204, Algorithm 7, line 21] computes r0 = LowBits(w - cs2) and line + * 23 rejects when ||r0||_inf >= GAMMA2 - BETA. By @[Round3_Spec, Section + * 5.1] (Lemma 3), this line-23 check on r0 = LowBits(w - cs2) is implied by + * ||w0 - cs2||_inf < GAMMA2 - BETA, where w0 is the low part of w. In our + * context, w0 already holds the low part of w from the line-13 Decompose; + * after subtracting cs2 from it in place, the mld_poly_chknorm(w0, GAMMA2 - + * BETA) call below is exactly that check. + * + * Hint: + * @[FIPS204, Algorithm 7, line 26] sets h = MakeHint(-ct0, w - cs2 + ct0), + * and line 28 rejects when ||ct0||_inf >= GAMMA2 or h has more than OMEGA + * nonzero coefficients. @[Round3_Spec, Section 5.1] provides the following + * alternative description for MakeHint(-ct0, w - cs2 + ct0): a hint bit is + * zero exactly when the coefficient of w0 - cs2 + ct0 lies in + * (-GAMMA2, GAMMA2], or equals -GAMMA2 while the matching w1 coefficient is + * zero (the Decompose border case), and is set otherwise. This equivalence + * is precisely what mld_pack_sig_h -> mld_make_hint compute from w0 + * (= w0 - cs2 + ct0) and w1. The line-28 ||ct0||_inf >= GAMMA2 check is the + * mld_poly_chknorm(z, GAMMA2) call on ct0 below; the weight bound is + * enforced by mld_pack_sig_h. + * + * Building w0 per-component and checking norms incrementally also avoids + * allocating a full polyveck for h. */ + for (k = 0; k < MLDSA_K; k++) + __loop__( + assigns(k, + object_whole(z), + object_whole(w0)) + invariant(k <= MLDSA_K) + invariant(forall(k0, k, MLDSA_K, + array_abs_bound(w0->vec[k0].coeffs, 0, MLDSA_N, MLDSA_GAMMA2 + 1))) + decreases(MLDSA_K - k) + ) + { + /* @[FIPS204, Algorithm 7, line 19] cs2[k] <- invNTT(c_hat o s2_hat)[k], + * then subtract from w0[k] to form (w0 - cs2)[k]. */ + mld_sk_s2hat_get_poly(z, s2hat, k); + mld_poly_pointwise_montgomery(z, cp); + mld_poly_invntt_tomont(z); + + mld_poly_sub(&w0->vec[k], z); + mld_poly_reduce(&w0->vec[k]); + + /* Low-bits norm check (see block comment above): the line-23 check on + * r0 = LowBits(w - cs2) holds via ||w0 - cs2||_inf < GAMMA2 - BETA. */ + w0_invalid = mld_poly_chknorm(&w0->vec[k], MLDSA_GAMMA2 - MLDSA_BETA); + /* Constant time: w0_invalid may be leaked - see comment for z_invalid. */ + MLD_CT_TESTING_DECLASSIFY(&w0_invalid, sizeof(uint32_t)); + if (w0_invalid) + { + ret = MLD_ERR_FAIL; /* reject */ + goto cleanup; + } + + /* @[FIPS204, Algorithm 7, line 25] ct0[k] <- invNTT(c_hat o t0_hat)[k]. */ + mld_sk_t0hat_get_poly(z, t0hat, k); + mld_poly_pointwise_montgomery(z, cp); + mld_poly_invntt_tomont(z); + mld_poly_reduce(z); + + /* @[FIPS204, Algorithm 7, line 28] reject when ||ct0||_inf >= GAMMA2 (the + * second part, the OMEGA weight bound, is enforced by mld_pack_sig_h). */ + h_invalid = mld_poly_chknorm(z, MLDSA_GAMMA2); + /* Constant time: h_invalid may be leaked - see comment for z_invalid. */ + MLD_CT_TESTING_DECLASSIFY(&h_invalid, sizeof(uint32_t)); + if (h_invalid) + { + ret = MLD_ERR_FAIL; /* reject */ + goto cleanup; + } + + /* Add ct0[k] to (w0 - cs2)[k], leaving (w0 - cs2 + ct0)[k] in w0[k] -- the + * MakeHint input prepared for mld_pack_sig_h (see block comment above). */ + mld_poly_add(&w0->vec[k], z); + } + + /* Constant time: At this point all norm checks have passed and we, hence, + * know that the signature does not leak any secret information. + * Consequently, any value that can be computed from the signature and public + * key is considered public. + * w0 and w1 are public as they can be computed from Az - ct = \alpha w1 + w0. + * h=c*t0 is public as both c and t0 are considered public. + * While t0 is not part of the public key, it can be reconstructed from + * a small number of signatures and need not be regarded as secret + * (see @[FIPS204, Section 6.1]). + */ + MLD_CT_TESTING_DECLASSIFY(w0, sizeof(*w0)); + MLD_CT_TESTING_DECLASSIFY(w1, sizeof(*w1)); + + /* @[FIPS204, Algorithm 7, line 33] sigEncode(ctilde, z mod+/- q, h) is split + * across three calls: z was already packed by mld_compute_pack_z, this call + * packs ctilde, and mld_pack_sig_h below packs the hint h. */ + mld_pack_sig_c(sig, challenge_bytes); + + /* @[FIPS204, Algorithm 7, line 26] h <- MakeHint(-ct0, w - cs2 + ct0), + * computed from (w0 = w0 - cs2 + ct0, w1) as described in the block comment + * above, and packed as the h component of the line-33 sigEncode. Returns + * MLD_ERR_FAIL if h would exceed OMEGA nonzero coefficients (the remaining + * part of the line-28 check), in which case we reject. */ + ret = mld_pack_sig_h(sig, w0, w1); + if (ret != 0) + { + goto cleanup; + } + + /* Constant time: At this point it is clear that the signature is valid - it + * can, hence, be considered public. */ + MLD_CT_TESTING_DECLASSIFY(sig, MLDSA_CRYPTO_BYTES); + ret = 0; /* success */ + +cleanup: + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + MLD_FREE(t, mld_poly, 1, context); + MLD_FREE(cp, mld_poly, 1, context); + MLD_FREE(w0, mld_polyveck, 1, context); + MLD_FREE(w1tmp, w1tmp_u, 1, context); + MLD_FREE(z, mld_poly, 1, context); + MLD_FREE(y, mld_yvec, 1, context); + MLD_FREE(challenge_bytes, uint8_t, MLDSA_CTILDEBYTES, context); + + return ret; +} +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_signature_internal(uint8_t sig[MLDSA_CRYPTO_BYTES], + const uint8_t *m, size_t mlen, + const uint8_t *pre, size_t prelen, + const uint8_t rnd[MLDSA_RNDBYTES], + const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + int externalmu, + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret; + uint8_t *rho, *tr, *key, *mu, *rhoprime; + uint16_t attempt; + const uint16_t max_signing_attempts = mld_get_max_signing_attempts(); + MLD_ALLOC(seedbuf, uint8_t, + 2 * MLDSA_SEEDBYTES + MLDSA_TRBYTES + 2 * MLDSA_CRHBYTES, context); + MLD_ALLOC(mat, mld_polymat, 1, context); + MLD_ALLOC(s1hat, mld_sk_s1hat, 1, context); + MLD_ALLOC(t0hat, mld_sk_t0hat, 1, context); + MLD_ALLOC(s2hat, mld_sk_s2hat, 1, context); + + if (seedbuf == NULL || mat == NULL || s1hat == NULL || t0hat == NULL || + s2hat == NULL) + { + ret = MLD_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + /* If a resume hook is configured via MLD_CONFIG_SIGN_HOOK_RESUME, it provides + * the attempt to resume from after an earlier pause. Otherwise, we start at + * 0. Clamp to max_signing_attempts. */ + attempt = mld_sign_resume(context); + if (attempt > max_signing_attempts) + { + attempt = max_signing_attempts; + } + + rho = seedbuf; + tr = rho + MLDSA_SEEDBYTES; + key = tr + MLDSA_TRBYTES; + mu = key + MLDSA_SEEDBYTES; + rhoprime = mu + MLDSA_CRHBYTES; + /* @[FIPS204, Algorithm 7, line 1] (rho, K, tr, s1, s2, t0) <- skDecode(sk) + * and @[FIPS204, Algorithm 7, lines 2-4] s1_hat/s2_hat/t0_hat <- NTT(...): + * mld_unpack_sk returns s1hat, s2hat, t0hat already in NTT domain. The spec's + * private random seed K is held in the local variable key. */ + mld_unpack_sk(rho, tr, key, t0hat, s1hat, s2hat, sk); + + if (!externalmu) + { + /* @[FIPS204, Algorithm 7, line 6] mu <- H(BytesToBits(tr) || M', 64). */ + mld_H(mu, MLDSA_CRHBYTES, tr, MLDSA_TRBYTES, pre, prelen, m, mlen); + } + else + { + /* mu has been provided directly (external-mu variant; line 6 done by the + * caller in a separate cryptographic module). */ + mld_memcpy(mu, m, MLDSA_CRHBYTES); + } + + /* @[FIPS204, Algorithm 7, line 7] rhoprime <- H(K || rnd || mu, 64). */ + mld_H(rhoprime, MLDSA_CRHBYTES, key, MLDSA_SEEDBYTES, rnd, MLDSA_RNDBYTES, mu, + MLDSA_CRHBYTES); + + /* Constant time: rho is part of the public key and, hence, public. */ + MLD_CT_TESTING_DECLASSIFY(rho, MLDSA_SEEDBYTES); + /* @[FIPS204, Algorithm 7, line 5] A_hat <- ExpandA(rho). */ + mld_polyvec_matrix_expand(mat, rho); + + /* @[FIPS204, Algorithm 7, lines 8-10 and 31-32] the rejection-sampling loop, + * tracked by attempt (kappa = attempt*MLDSA_L). Each iteration's body (lines + * 11-30) plus, on success, the line-33 sigEncode are performed by + * mld_attempt_signature_generation. */ + + /* Reference: the reference loops unboundedly; we instead iterate over the + * bounded range [0, max_signing_attempts) for predictable termination. + * A success or fatal error exits via goto cleanup; running to completion + * means every attempt was rejected; with a FIPS compliant choice of + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS, this should never happen. */ + for (; attempt < max_signing_attempts; attempt++) + __loop__( + MLD_IF_NOT_REDUCE_RAM( + assigns(attempt, ret, memory_slice(sig, MLDSA_CRYPTO_BYTES)) + ) + MLD_IF_REDUCE_RAM( + assigns(attempt, ret, memory_slice(sig, MLDSA_CRYPTO_BYTES), + memory_slice(mat, sizeof(mld_polymat))) + ) + invariant(attempt <= max_signing_attempts) + + /* t0, s1, s2, and mat are initialized above and are NOT changed by this */ + /* loop. We can therefore re-assert their bounds here as part of the */ + /* loop invariant. This makes proof noticeably faster with CBMC */ + MLD_IF_NOT_REDUCE_RAM( + invariant(forall(k1, 0, MLDSA_K, forall(l1, 0, MLDSA_L, + array_bound(mat->vec[k1].vec[l1].coeffs, 0, MLDSA_N, 0, MLDSA_Q)))) + invariant(forall(k2, 0, MLDSA_K, array_abs_bound(t0hat->vec.vec[k2].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + invariant(forall(k3, 0, MLDSA_L, array_abs_bound(s1hat->vec.vec[k3].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + invariant(forall(k4, 0, MLDSA_K, array_abs_bound(s2hat->vec.vec[k4].coeffs, 0, MLDSA_N, MLD_NTT_BOUND))) + ) + decreases(max_signing_attempts - attempt) + ) + { + /* Safety: attempt < max_signing_attempts <= MLD_MAX_KAPPA / MLDSA_L, so + * kappa <= MLD_MAX_KAPPA and the cast is safe. */ + const uint16_t kappa = (uint16_t)(attempt * MLDSA_L); + + /* Query configurable signing hook whether signing should be paused. + * This is skipped by default and only used if the user sets the + * configuration option MLD_CONFIG_SIGN_HOOK_ATTEMPT. */ + if (mld_sign_attempt(attempt, context) != 0) + { + ret = MLD_ERR_SIGNING_PAUSED; + goto cleanup; + } + + ret = mld_attempt_signature_generation(sig, mu, rhoprime, kappa, mat, s1hat, + s2hat, t0hat, context); + + /* Decide whether to keep trying based on the return value: + * - ret == 0: a valid signature was produced; we are done. + * - ret == MLD_ERR_FAIL: this candidate was rejected by one of the norm + * or hint checks. We continue the loop and try again with the next + * nonce. + * - any other value (e.g. MLD_ERR_OUT_OF_MEMORY): an unrecoverable error + * occurred, so we propagate it to the caller. */ + if (ret == 0) + { + /* Signing succeeded: record the attempt that succeeded. No-op in the + * default build. */ + mld_sign_finish(attempt, context); + goto cleanup; + } + if (ret != MLD_ERR_FAIL) + { + goto cleanup; + } + } + + /* Loop ran to completion: all attempts rejected, budget exhausted. + * This should never happen with a FIPS compliant choice of + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS. */ + ret = MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED; + +cleanup: + + if (ret != 0) + { + /* To be on the safe-side, we zeroize the signature buffer. */ + mld_zeroize(sig, MLDSA_CRYPTO_BYTES); + } + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + MLD_FREE(s2hat, mld_sk_s2hat, 1, context); + MLD_FREE(t0hat, mld_sk_t0hat, 1, context); + MLD_FREE(s1hat, mld_sk_s1hat, 1, context); + MLD_FREE(mat, mld_polymat, 1, context); + MLD_FREE(seedbuf, uint8_t, + 2 * MLDSA_SEEDBYTES + MLDSA_TRBYTES + 2 * MLDSA_CRHBYTES, context); + return ret; +} + +#if !defined(MLD_CONFIG_CORE_API_ONLY) +#if !defined(MLD_CONFIG_NO_RANDOMIZED_API) +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_signature(uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m, + size_t mlen, const uint8_t *ctx, size_t ctxlen, + const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + size_t pre_len; + int ret; + MLD_ALLOC(pre, uint8_t, MLD_DOMAIN_SEPARATION_MAX_BYTES, context); + MLD_ALLOC(rnd, uint8_t, MLDSA_RNDBYTES, context); + + if (pre == NULL || rnd == NULL) + { + ret = MLD_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + /* Prepare domain separation prefix for pure ML-DSA */ + pre_len = mld_prepare_domain_separation_prefix(pre, NULL, 0, ctx, ctxlen, + MLD_PREHASH_NONE); + if (pre_len == 0) + { + ret = MLD_ERR_INVALID_ARG; + goto cleanup; + } + + /* Randomized variant of ML-DSA. If you need the deterministic variant, + * call mld_sign_signature_internal directly with all-zero rnd. */ + if (mld_randombytes(rnd, MLDSA_RNDBYTES) != 0) + { + ret = MLD_ERR_RNG_FAIL; + goto cleanup; + } + MLD_CT_TESTING_SECRET(rnd, MLDSA_RNDBYTES); + + ret = mld_sign_signature_internal(sig, m, mlen, pre, pre_len, rnd, sk, 0, + context); + +cleanup: + if (ret != 0) + { + /* To be on the safe-side, make sure sig has a well-defined value, even in + * the case of error. + * + * If we come from mld_sign_signature_internal, this is redundant, but the + * error case should not be the norm, and the added cost of the zeroization + * insignificant. */ + mld_zeroize(sig, MLDSA_CRYPTO_BYTES); + } + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + MLD_FREE(rnd, uint8_t, MLDSA_RNDBYTES, context); + MLD_FREE(pre, uint8_t, MLD_DOMAIN_SEPARATION_MAX_BYTES, context); + + return ret; +} +#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */ + +#if !defined(MLD_CONFIG_NO_RANDOMIZED_API) +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_signature_extmu(uint8_t sig[MLDSA_CRYPTO_BYTES], + const uint8_t mu[MLDSA_CRHBYTES], + const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret; + MLD_ALLOC(rnd, uint8_t, MLDSA_RNDBYTES, context); + + if (rnd == NULL) + { + ret = MLD_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + /* Randomized variant of ML-DSA. If you need the deterministic variant, + * call mld_sign_signature_internal directly with all-zero rnd. */ + if (mld_randombytes(rnd, MLDSA_RNDBYTES) != 0) + { + ret = MLD_ERR_RNG_FAIL; + goto cleanup; + } + MLD_CT_TESTING_SECRET(rnd, MLDSA_RNDBYTES); + + ret = mld_sign_signature_internal(sig, mu, MLDSA_CRHBYTES, NULL, 0, rnd, sk, + 1, context); + +cleanup: + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + MLD_FREE(rnd, uint8_t, MLDSA_RNDBYTES, context); + + return ret; +} +#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */ +#endif /* !MLD_CONFIG_CORE_API_ONLY */ +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_verify_internal(const uint8_t sig[MLDSA_CRYPTO_BYTES], + const uint8_t *m, size_t mlen, const uint8_t *pre, + size_t prelen, + const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + int externalmu, + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret, cmp; + unsigned int i; + + MLD_ALLOC(buf, uint8_t, (MLDSA_K * MLDSA_POLYW1_PACKEDBYTES), context); + MLD_ALLOC(mu, uint8_t, MLDSA_CRHBYTES, context); + MLD_ALLOC(c, uint8_t, MLDSA_CTILDEBYTES, context); + MLD_ALLOC(c2, uint8_t, MLDSA_CTILDEBYTES, context); + MLD_ALLOC(z, mld_polyvecl, 1, context); + MLD_ALLOC(cp, mld_poly, 1, context); + MLD_ALLOC(mat, mld_polymat, 1, context); + MLD_ALLOC(w1, mld_poly, 1, context); + MLD_ALLOC(tmp, mld_poly, 1, context); + + if (buf == NULL || mu == NULL || c == NULL || c2 == NULL || z == NULL || + cp == NULL || mat == NULL || w1 == NULL || tmp == NULL) + { + ret = MLD_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + mld_memcpy(c, sig, MLDSA_CTILDEBYTES); + mld_polyvecl_unpack_z(z, sig + MLDSA_SIG_Z_OFFSET); + + /* mld_polyvecl_chknorm signals failure through a single non-zero error code + * that's not yet aligned with MLD_ERR_XXX. A norm-check failure here means + * the signature is invalid, so map it to MLD_ERR_INVALID_SIGNATURE. */ + if (mld_polyvecl_chknorm(z, MLDSA_GAMMA1 - MLDSA_BETA)) + { + ret = MLD_ERR_INVALID_SIGNATURE; + goto cleanup; + } + + if (!externalmu) + { + /* Compute CRH(H(rho, t1), pre, msg) */ + MLD_ALIGN uint8_t hpk[MLDSA_CRHBYTES]; + mld_H(hpk, MLDSA_TRBYTES, pk, MLDSA_CRYPTO_PUBLICKEYBYTES, NULL, 0, NULL, + 0); + mld_H(mu, MLDSA_CRHBYTES, hpk, MLDSA_TRBYTES, pre, prelen, m, mlen); + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(hpk, sizeof(hpk)); + } + else + { + /* mu has been provided directly */ + mld_memcpy(mu, m, MLDSA_CRHBYTES); + } + + /* Matrix-vector multiplication and per-row reconstruction of w1. */ + mld_polyvecl_ntt(z); + mld_polyvec_matrix_expand(mat, pk); + mld_poly_challenge(cp, c); + mld_poly_ntt(cp); + + for (i = 0; i < MLDSA_K; ++i) + __loop__( + assigns(MLD_IF_REDUCE_RAM(memory_slice(mat, sizeof(mld_polymat)),) + i, ret, + memory_slice(w1, sizeof(mld_poly)), + memory_slice(tmp, sizeof(mld_poly)), + memory_slice(buf, MLDSA_K * MLDSA_POLYW1_PACKEDBYTES) + ) + invariant(i <= MLDSA_K) + decreases(MLDSA_K - i) + ) + { + /* w1 = (A * z)_i in NTT domain */ + mld_polyvec_matrix_pointwise_montgomery_row(w1, mat, z, i); + + /* tmp = c * t1_i * 2^d in NTT domain */ + mld_unpack_pk_t1(tmp, pk, i); + mld_poly_shiftl(tmp); + mld_poly_ntt(tmp); + mld_poly_pointwise_montgomery(tmp, cp); + + /* w1 = invNTT(w1 - c * t1_i * 2^d) */ + mld_poly_sub(w1, tmp); + mld_poly_reduce(w1); + mld_poly_invntt_tomont(w1); + mld_poly_caddq(w1); + + /* tmp = h_i (decoded and validated from signature). A non-zero return + * means the hint encoding is malformed, i.e. the signature is invalid. */ + if (mld_sig_unpack_hints(tmp, sig, i) != 0) + { + ret = MLD_ERR_INVALID_SIGNATURE; + goto cleanup; + } + + /* w1 = use_hint(w1, tmp), then pack into buf[i] */ + mld_poly_use_hint(w1, tmp); + mld_polyw1_pack(buf + i * MLDSA_POLYW1_PACKEDBYTES, w1); + } + + /* Call random oracle and verify challenge */ + mld_H(c2, MLDSA_CTILDEBYTES, mu, MLDSA_CRHBYTES, buf, + MLDSA_K * MLDSA_POLYW1_PACKEDBYTES, NULL, 0); + + cmp = mld_ct_memcmp(c, c2, MLDSA_CTILDEBYTES); + + /* Declassify the result of the verification. */ + MLD_CT_TESTING_DECLASSIFY(&cmp, sizeof(cmp)); + + ret = cmp == 0 ? 0 : MLD_ERR_INVALID_SIGNATURE; + +cleanup: + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + MLD_FREE(tmp, mld_poly, 1, context); + MLD_FREE(w1, mld_poly, 1, context); + MLD_FREE(mat, mld_polymat, 1, context); + MLD_FREE(cp, mld_poly, 1, context); + MLD_FREE(z, mld_polyvecl, 1, context); + MLD_FREE(c2, uint8_t, MLDSA_CTILDEBYTES, context); + MLD_FREE(c, uint8_t, MLDSA_CTILDEBYTES, context); + MLD_FREE(mu, uint8_t, MLDSA_CRHBYTES, context); + MLD_FREE(buf, uint8_t, (MLDSA_K * MLDSA_POLYW1_PACKEDBYTES), context); + return ret; +} + +#if !defined(MLD_CONFIG_CORE_API_ONLY) +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_verify(const uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m, + size_t mlen, const uint8_t *ctx, size_t ctxlen, + const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + MLD_ALIGN uint8_t pre[MLD_DOMAIN_SEPARATION_MAX_BYTES]; + size_t pre_len; + int ret; + + pre_len = mld_prepare_domain_separation_prefix(pre, NULL, 0, ctx, ctxlen, + MLD_PREHASH_NONE); + if (pre_len == 0) + { + ret = MLD_ERR_INVALID_ARG; + goto cleanup; + } + + ret = mld_sign_verify_internal(sig, m, mlen, pre, pre_len, pk, 0, context); + +cleanup: + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(pre, sizeof(pre)); + + return ret; +} + +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_verify_extmu(const uint8_t sig[MLDSA_CRYPTO_BYTES], + const uint8_t mu[MLDSA_CRHBYTES], + const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + return mld_sign_verify_internal(sig, mu, MLDSA_CRHBYTES, NULL, 0, pk, 1, + context); +} +#endif /* !MLD_CONFIG_CORE_API_ONLY */ +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_CORE_API_ONLY) +#if !defined(MLD_CONFIG_NO_SIGN_API) +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_signature_pre_hash_internal( + uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *ph, size_t phlen, + const uint8_t *ctx, size_t ctxlen, const uint8_t rnd[MLDSA_RNDBYTES], + const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], int hashalg, + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + MLD_ALIGN uint8_t pre[MLD_DOMAIN_SEPARATION_MAX_BYTES]; + size_t pre_len; + int ret; + + if (hashalg == MLD_PREHASH_NONE) + { + ret = MLD_ERR_INVALID_ARG; + goto cleanup; + } + + pre_len = mld_prepare_domain_separation_prefix(pre, ph, phlen, ctx, ctxlen, + hashalg); + if (pre_len == 0) + { + ret = MLD_ERR_INVALID_ARG; + goto cleanup; + } + + ret = mld_sign_signature_internal(sig, pre, pre_len, NULL, 0, rnd, sk, 0, + context); +cleanup: + if (ret != 0) + { + /* To be on the safe-side, make sure sig has a well-defined value, even in + * the case of error. + * + * If we come from mld_sign_signature_internal, this is redundant, but the + * error case should not be the norm, and the added cost of the zeroization + * insignificant. */ + mld_zeroize(sig, MLDSA_CRYPTO_BYTES); + } + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(pre, sizeof(pre)); + return ret; +} +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_verify_pre_hash_internal( + const uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *ph, size_t phlen, + const uint8_t *ctx, size_t ctxlen, + const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], int hashalg, + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + MLD_ALIGN uint8_t pre[MLD_DOMAIN_SEPARATION_MAX_BYTES]; + size_t pre_len; + int ret; + + if (hashalg == MLD_PREHASH_NONE) + { + ret = MLD_ERR_INVALID_ARG; + goto cleanup; + } + + pre_len = mld_prepare_domain_separation_prefix(pre, ph, phlen, ctx, ctxlen, + hashalg); + if (pre_len == 0) + { + ret = MLD_ERR_INVALID_ARG; + goto cleanup; + } + + ret = mld_sign_verify_internal(sig, pre, pre_len, NULL, 0, pk, 0, context); + +cleanup: + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(pre, sizeof(pre)); + return ret; +} +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_signature_pre_hash_shake256( + uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m, size_t mlen, + const uint8_t *ctx, size_t ctxlen, const uint8_t rnd[MLDSA_RNDBYTES], + const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + MLD_ALIGN uint8_t ph[64]; + int ret; + mld_shake256(ph, sizeof(ph), m, mlen); + ret = mld_sign_signature_pre_hash_internal(sig, ph, sizeof(ph), ctx, ctxlen, + rnd, sk, MLD_PREHASH_SHAKE_256, + context); + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(ph, sizeof(ph)); + return ret; +} +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_verify_pre_hash_shake256( + const uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m, size_t mlen, + const uint8_t *ctx, size_t ctxlen, + const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + MLD_ALIGN uint8_t ph[64]; + int ret; + mld_shake256(ph, sizeof(ph), m, mlen); + ret = mld_sign_verify_pre_hash_internal(sig, ph, sizeof(ph), ctx, ctxlen, pk, + MLD_PREHASH_SHAKE_256, context); + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + mld_zeroize(ph, sizeof(ph)); + return ret; +} +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +#define MLD_PRE_HASH_OID_LEN 11 + +/** + * Return the OID of a given SHA-2/SHA-3 hash function. + * + * @param[out] oid Pointer to output OID. + * @param hashalg Hash algorithm constant (MLD_PREHASH_*). + */ +static void mld_get_hash_oid(uint8_t oid[MLD_PRE_HASH_OID_LEN], int hashalg) +{ + unsigned int i; + static const struct + { + int alg; + uint8_t oid[MLD_PRE_HASH_OID_LEN]; + } oid_map[] = { + {MLD_PREHASH_SHA2_224, + {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x04}}, + {MLD_PREHASH_SHA2_256, + {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x01}}, + {MLD_PREHASH_SHA2_384, + {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x02}}, + {MLD_PREHASH_SHA2_512, + {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x03}}, + {MLD_PREHASH_SHA2_512_224, + {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x05}}, + {MLD_PREHASH_SHA2_512_256, + {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x06}}, + {MLD_PREHASH_SHA3_224, + {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x07}}, + {MLD_PREHASH_SHA3_256, + {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x08}}, + {MLD_PREHASH_SHA3_384, + {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x09}}, + {MLD_PREHASH_SHA3_512, + {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x0A}}, + {MLD_PREHASH_SHAKE_128, + {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x0B}}, + {MLD_PREHASH_SHAKE_256, + {0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x0C}}}; + + for (i = 0; i < sizeof(oid_map) / sizeof(oid_map[0]); i++) + __loop__( + invariant(i <= sizeof(oid_map) / sizeof(oid_map[0])) + decreases(sizeof(oid_map) / sizeof(oid_map[0]) - i) + ) + { + if (oid_map[i].alg == hashalg) + { + mld_memcpy(oid, oid_map[i].oid, MLD_PRE_HASH_OID_LEN); + return; + } + } +} + +static int mld_validate_hash_length(int hashalg, size_t len) +{ + switch (hashalg) + { + case MLD_PREHASH_SHA2_224: + return (len == 224 / 8) ? 0 : -1; + case MLD_PREHASH_SHA2_256: + return (len == 256 / 8) ? 0 : -1; + case MLD_PREHASH_SHA2_384: + return (len == 384 / 8) ? 0 : -1; + case MLD_PREHASH_SHA2_512: + return (len == 512 / 8) ? 0 : -1; + case MLD_PREHASH_SHA2_512_224: + return (len == 224 / 8) ? 0 : -1; + case MLD_PREHASH_SHA2_512_256: + return (len == 256 / 8) ? 0 : -1; + case MLD_PREHASH_SHA3_224: + return (len == 224 / 8) ? 0 : -1; + case MLD_PREHASH_SHA3_256: + return (len == 256 / 8) ? 0 : -1; + case MLD_PREHASH_SHA3_384: + return (len == 384 / 8) ? 0 : -1; + case MLD_PREHASH_SHA3_512: + return (len == 512 / 8) ? 0 : -1; + case MLD_PREHASH_SHAKE_128: + return (len == 256 / 8) ? 0 : -1; + case MLD_PREHASH_SHAKE_256: + return (len == 512 / 8) ? 0 : -1; + default: + return -1; + } +} + +MLD_EXTERNAL_API +size_t mld_prepare_domain_separation_prefix( + uint8_t prefix[MLD_DOMAIN_SEPARATION_MAX_BYTES], const uint8_t *ph, + size_t phlen, const uint8_t *ctx, size_t ctxlen, int hashalg) +{ + if (ctxlen > 255) + { + return 0; + } + + if (hashalg != MLD_PREHASH_NONE) + { + if (ph == NULL || mld_validate_hash_length(hashalg, phlen) != 0) + { + return 0; + } + } + + /* Common prefix: 0x00/0x01 || ctxlen || ctx */ + prefix[0] = (hashalg == MLD_PREHASH_NONE) ? 0 : 1; + prefix[1] = (uint8_t)ctxlen; + if (ctxlen > 0) + { + mld_memcpy(prefix + 2, ctx, ctxlen); + } + + if (hashalg == MLD_PREHASH_NONE) + { + return 2 + ctxlen; + } + + /* HashML-DSA: append oid || ph */ + mld_get_hash_oid(prefix + 2 + ctxlen, hashalg); + mld_memcpy(prefix + 2 + ctxlen + MLD_PRE_HASH_OID_LEN, ph, phlen); + return 2 + ctxlen + MLD_PRE_HASH_OID_LEN + phlen; +} +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +MLD_EXTERNAL_API +int mld_sign_pk_from_sk(uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + uint8_t check, cmp0, cmp1, chk1, chk2; + int ret; + MLD_ALLOC(rho, uint8_t, MLDSA_SEEDBYTES, context); + MLD_ALLOC(tr, uint8_t, MLDSA_TRBYTES, context); + MLD_ALLOC(tr_computed, uint8_t, MLDSA_TRBYTES, context); + MLD_ALLOC(key, uint8_t, MLDSA_SEEDBYTES, context); + MLD_ALLOC(s1, mld_polyvecl, 1, context); + MLD_ALLOC(s2, mld_polyveck, 1, context); + MLD_ALLOC(t0_packed, uint8_t, MLDSA_K *MLDSA_POLYT0_PACKEDBYTES, context); + + if (rho == NULL || tr == NULL || tr_computed == NULL || key == NULL || + s1 == NULL || s2 == NULL || t0_packed == NULL) + { + ret = MLD_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + /* Inline unpack_sk: mld_unpack_sk uses lazy types for s1/s2/t0 which + * we cannot use here. t0 stays in packed form -- we compare it against + * the recomputed value below. */ + mld_memcpy(rho, sk + MLDSA_SK_RHO_OFFSET, MLDSA_SEEDBYTES); + mld_memcpy(key, sk + MLDSA_SK_KEY_OFFSET, MLDSA_SEEDBYTES); + mld_memcpy(tr, sk + MLDSA_SK_TR_OFFSET, MLDSA_TRBYTES); + mld_polyvecl_unpack_eta(s1, sk + MLDSA_SK_S1_OFFSET); + mld_polyveck_unpack_eta(s2, sk + MLDSA_SK_S2_OFFSET); + + /* Validate s1 and s2 coefficients are within [-MLDSA_ETA, MLDSA_ETA] */ + chk1 = mld_polyvecl_chknorm(s1, MLDSA_ETA + 1) & 0xFF; + chk2 = mld_polyveck_chknorm(s2, MLDSA_ETA + 1) & 0xFF; + + /* NTT s1 in place to use as s1hat */ + mld_polyvecl_ntt(s1); + + /* Pack rho into pk */ + mld_memcpy(pk + MLDSA_PK_RHO_OFFSET, rho, MLDSA_SEEDBYTES); + + /* Recompute t row by row, decompose, and pack t1 into pk and t0 into + * t0_packed. */ + ret = mld_compute_pack_t0_t1(pk + MLDSA_PK_T1_OFFSET, t0_packed, s1, s2, rho, + context); + if (ret != 0) + { + goto cleanup; + } + + /* Compare recomputed packed t0 against the t0 region of sk. */ + cmp0 = mld_ct_memcmp(t0_packed, sk + MLDSA_SK_T0_OFFSET, + MLDSA_K * MLDSA_POLYT0_PACKEDBYTES); + + /* Compute tr_computed = H(pk) and compare to the stored tr */ + mld_shake256(tr_computed, MLDSA_TRBYTES, pk, MLDSA_CRYPTO_PUBLICKEYBYTES); + cmp1 = mld_ct_memcmp((const uint8_t *)tr, (const uint8_t *)tr_computed, + MLDSA_TRBYTES); + check = mld_value_barrier_u8(cmp0 | cmp1 | chk1 | chk2); + + /* Declassify the final result of the validity check. */ + MLD_CT_TESTING_DECLASSIFY(&check, sizeof(check)); + ret = (check != 0) ? MLD_ERR_INVALID_KEY : 0; + +cleanup: + + if (ret != 0) + { + mld_zeroize(pk, MLDSA_CRYPTO_PUBLICKEYBYTES); + } + + /* Constant time: pk is either the valid public key or zeroed on error */ + MLD_CT_TESTING_DECLASSIFY(pk, MLDSA_CRYPTO_PUBLICKEYBYTES); + + /* @[FIPS204, Section 3.6.3] Destruction of intermediate values. */ + MLD_FREE(t0_packed, uint8_t, MLDSA_K *MLDSA_POLYT0_PACKEDBYTES, context); + MLD_FREE(s2, mld_polyveck, 1, context); + MLD_FREE(s1, mld_polyvecl, 1, context); + MLD_FREE(key, uint8_t, MLDSA_SEEDBYTES, context); + MLD_FREE(tr_computed, uint8_t, MLDSA_TRBYTES, context); + MLD_FREE(tr, uint8_t, MLDSA_TRBYTES, context); + MLD_FREE(rho, uint8_t, MLDSA_SEEDBYTES, context); + + return ret; +} +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ +#endif /* !MLD_CONFIG_CORE_API_ONLY */ + +/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. + * Don't modify by hand -- this is auto-generated by scripts/autogen. */ +#undef mld_check_pct +#undef mld_sample_s1_s2 +#undef mld_validate_hash_length +#undef mld_get_hash_oid +#undef mld_H +#undef mld_compute_pack_z +#undef mld_attempt_signature_generation +#undef mld_compute_pack_t0_t1 +#undef mld_get_max_signing_attempts +#undef MLD_MAX_SIGNING_ATTEMPTS +#undef MLD_PRE_HASH_OID_LEN diff --git a/cbits/mldsa/src/sign.h b/cbits/mldsa/src/sign.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/sign.h @@ -0,0 +1,850 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS204] + * FIPS 204 Module-Lattice-Based Digital Signature Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/204/final + */ + +#ifndef MLD_SIGN_H +#define MLD_SIGN_H + +#include +#include "cbmc.h" +#include "common.h" +#include "poly.h" +#include "polyvec.h" +#include "sys.h" + +#if defined(MLD_CHECK_APIS) +/* Include to ensure consistency between internal sign.h + * and external mldsa_native.h. */ +#include "mldsa_native.h" + +#if MLDSA_CRYPTO_SECRETKEYBYTES != \ + MLDSA_SECRETKEYBYTES(MLD_CONFIG_PARAMETER_SET) +#error Mismatch for SECRETKEYBYTES between sign.h and mldsa_native.h +#endif + +#if MLDSA_CRYPTO_PUBLICKEYBYTES != \ + MLDSA_PUBLICKEYBYTES(MLD_CONFIG_PARAMETER_SET) +#error Mismatch for PUBLICKEYBYTES between sign.h and mldsa_native.h +#endif + +#if MLDSA_CRYPTO_BYTES != MLDSA_BYTES(MLD_CONFIG_PARAMETER_SET) +#error Mismatch for BYTES between sign.h and mldsa_native.h +#endif + +#endif /* MLD_CHECK_APIS */ + +#define mld_sign_keypair_internal \ + MLD_NAMESPACE_KL(keypair_internal) MLD_CONTEXT_PARAMETERS_3 +#define mld_sign_keypair MLD_NAMESPACE_KL(keypair) MLD_CONTEXT_PARAMETERS_2 +#define mld_sign_signature_internal \ + MLD_NAMESPACE_KL(signature_internal) MLD_CONTEXT_PARAMETERS_8 +#define mld_sign_signature MLD_NAMESPACE_KL(signature) MLD_CONTEXT_PARAMETERS_6 +#define mld_sign_signature_extmu \ + MLD_NAMESPACE_KL(signature_extmu) MLD_CONTEXT_PARAMETERS_3 +#define mld_sign_verify_internal \ + MLD_NAMESPACE_KL(verify_internal) MLD_CONTEXT_PARAMETERS_7 +#define mld_sign_verify MLD_NAMESPACE_KL(verify) MLD_CONTEXT_PARAMETERS_6 +#define mld_sign_verify_extmu \ + MLD_NAMESPACE_KL(verify_extmu) MLD_CONTEXT_PARAMETERS_3 +#define mld_sign_signature_pre_hash_internal \ + MLD_NAMESPACE_KL(signature_pre_hash_internal) MLD_CONTEXT_PARAMETERS_8 +#define mld_sign_verify_pre_hash_internal \ + MLD_NAMESPACE_KL(verify_pre_hash_internal) MLD_CONTEXT_PARAMETERS_7 +#define mld_sign_signature_pre_hash_shake256 \ + MLD_NAMESPACE_KL(signature_pre_hash_shake256) MLD_CONTEXT_PARAMETERS_7 +#define mld_sign_verify_pre_hash_shake256 \ + MLD_NAMESPACE_KL(verify_pre_hash_shake256) MLD_CONTEXT_PARAMETERS_6 +#define mld_prepare_domain_separation_prefix \ + MLD_NAMESPACE_KL(prepare_domain_separation_prefix) +#define mld_sign_pk_from_sk \ + MLD_NAMESPACE_KL(pk_from_sk) MLD_CONTEXT_PARAMETERS_2 + +/* Hash algorithm constants for domain separation */ +#define MLD_PREHASH_NONE 0 +#define MLD_PREHASH_SHA2_224 1 +#define MLD_PREHASH_SHA2_256 2 +#define MLD_PREHASH_SHA2_384 3 +#define MLD_PREHASH_SHA2_512 4 +#define MLD_PREHASH_SHA2_512_224 5 +#define MLD_PREHASH_SHA2_512_256 6 +#define MLD_PREHASH_SHA3_224 7 +#define MLD_PREHASH_SHA3_256 8 +#define MLD_PREHASH_SHA3_384 9 +#define MLD_PREHASH_SHA3_512 10 +#define MLD_PREHASH_SHAKE_128 11 +#define MLD_PREHASH_SHAKE_256 12 + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +/** + * Generate a public-private key pair from a seed. + * + * When MLD_CONFIG_KEYGEN_PCT is set, performs a Pairwise Consistency Test + * (PCT) as required by FIPS 140-3 IG. + * + * @spec{Implements @[FIPS204, Algorithm 6, ML-DSA.KeyGen_internal].} + * + * @param[out] pk Output public key. + * @param[out] sk Output private key. + * @param[in] seed Input random seed. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_RNG_FAIL Random number generation failed + * during the PCT. Only possible when + * MLD_CONFIG_KEYGEN_PCT is enabled. + * @retval MLD_ERR_SIGNING_PAUSED The PCT's signing step was paused by + * a MLD_CONFIG_SIGN_HOOK_ATTEMPT hook. + * This should currently never happen: + * signing hooks require + * MLD_CONFIG_NO_RANDOMIZED_API, which + * is incompatible with + * MLD_CONFIG_KEYGEN_PCT, so the two + * cannot be enabled simultaneously. + * @retval MLD_ERR_PCT_FAIL MLD_CONFIG_KEYGEN_PCT is enabled and + * the PCT check failed. + */ +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_keypair_internal(uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + const uint8_t seed[MLDSA_SEEDBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES)) + requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) + requires(memory_no_alias(seed, MLDSA_SEEDBYTES)) + assigns(object_whole(pk)) + assigns(object_whole(sk)) + ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY || + return_value == MLD_ERR_RNG_FAIL || + return_value == MLD_ERR_SIGNING_PAUSED || + return_value == MLD_ERR_PCT_FAIL) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(pk, MLDSA_CRYPTO_PUBLICKEYBYTES)) + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) +); + +#if !defined(MLD_CONFIG_CORE_API_ONLY) +#if !defined(MLD_CONFIG_NO_RANDOMIZED_API) +/** + * Generate a public-private key pair. + * + * When MLD_CONFIG_KEYGEN_PCT is set, performs a Pairwise Consistency Test + * (PCT) as required by FIPS 140-3 IG. + * + * @spec{Implements @[FIPS204, Algorithm 1, ML-DSA.KeyGen].} + * + * @param[out] pk Output public key. + * @param[out] sk Output private key. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_RNG_FAIL Random number generation failed. + * @retval MLD_ERR_SIGNING_PAUSED The PCT's signing step was paused by + * a MLD_CONFIG_SIGN_HOOK_ATTEMPT hook. + * This should currently never happen: + * signing hooks require + * MLD_CONFIG_NO_RANDOMIZED_API, which + * is incompatible with + * MLD_CONFIG_KEYGEN_PCT, so the two + * cannot be enabled simultaneously. + * @retval MLD_ERR_PCT_FAIL MLD_CONFIG_KEYGEN_PCT is enabled and + * the PCT check failed. + */ +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_keypair(uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES)) + requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) + assigns(object_whole(pk)) + assigns(object_whole(sk)) + ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY || + return_value == MLD_ERR_RNG_FAIL || + return_value == MLD_ERR_SIGNING_PAUSED || + return_value == MLD_ERR_PCT_FAIL) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(pk, MLDSA_CRYPTO_PUBLICKEYBYTES)) + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) +); +#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */ +#endif /* !MLD_CONFIG_CORE_API_ONLY */ +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +/** + * Compute signature using a caller-supplied random seed and prefix. + * + * On error (non-zero return value), the signature buffer sig is zeroized. + * + * @spec{Implements @[FIPS204, Algorithm 7, ML-DSA.Sign_internal].} + * + * @warning This function does not perform secret key validation. + * Callers importing serialized keys can use mld_sign_pk_from_sk + * to validate them before signing. + * + * @param[out] sig Pointer to buffer to hold the generated signature of + * MLDSA_CRYPTO_BYTES bytes. + * @param[in] m Pointer to message to be signed (when + * externalmu == 0), or to a precomputed + * message representative mu (when externalmu != 0). + * @param mlen Length of m. Must equal MLDSA_CRHBYTES when + * externalmu != 0. + * @param[in] pre Pointer to prefix string. Ignored when + * externalmu != 0. + * @param prelen Length of prefix string. Ignored when + * externalmu != 0. + * @param[in] rnd Random seed. + * @param[in] sk Bit-packed secret key; assumed to be valid. + * @param externalmu 0: m/mlen is the raw message; mu = H(tr, pre, m) is + * computed internally. + * non-zero: m points to a precomputed mu of + * MLDSA_CRHBYTES bytes; pre/prelen unused. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS + * iterations. + * @retval MLD_ERR_SIGNING_PAUSED A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook + * paused signing; re-invoke to resume. + */ +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_signature_internal(uint8_t sig[MLDSA_CRYPTO_BYTES], + const uint8_t *m, size_t mlen, + const uint8_t *pre, size_t prelen, + const uint8_t rnd[MLDSA_RNDBYTES], + const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + int externalmu, + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(mlen <= MLD_MAX_BUFFER_SIZE) + requires(prelen <= MLD_MAX_BUFFER_SIZE) + requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(m, mlen)) + requires(memory_no_alias(rnd, MLDSA_RNDBYTES)) + requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) + requires((externalmu == 0) ==> ((prelen == 0) || memory_no_alias(pre, prelen))) + requires((externalmu != 0) ==> (mlen == MLDSA_CRHBYTES)) + assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES)) + ensures(return_value == 0 || + return_value == MLD_ERR_OUT_OF_MEMORY || + return_value == MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED || + return_value == MLD_ERR_SIGNING_PAUSED) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(sig, MLDSA_CRYPTO_BYTES))); + +#if !defined(MLD_CONFIG_CORE_API_ONLY) +#if !defined(MLD_CONFIG_NO_RANDOMIZED_API) +/** + * Compute signature. This function implements the randomized variant of + * ML-DSA. If you require the deterministic variant, use + * mld_sign_signature_internal directly. + * + * @spec{Implements @[FIPS204, Algorithm 2, ML-DSA.Sign].} + * + * @warning This function does not perform secret key validation. + * Callers importing serialized keys can use mld_sign_pk_from_sk + * to validate them before signing. + * + * @param[out] sig Output signature. + * @param[in] m Pointer to message to be signed. + * @param mlen Length of message. + * @param[in] ctx Pointer to context string. May be NULL if ctxlen == 0. + * @param ctxlen Length of context string. Should be <= 255. + * @param[in] sk Bit-packed secret key; assumed to be valid. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_RNG_FAIL Random number generation failed. + * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS + * iterations. + * @retval MLD_ERR_SIGNING_PAUSED A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook + * paused signing; re-invoke to resume. + * @retval MLD_ERR_INVALID_ARG The context string exceeded 255 + * bytes. + */ +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_signature(uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m, + size_t mlen, const uint8_t *ctx, size_t ctxlen, + const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(mlen <= MLD_MAX_BUFFER_SIZE) + requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(m, mlen)) + requires(ctxlen <= MLD_MAX_BUFFER_SIZE) + requires(ctxlen == 0 || memory_no_alias(ctx, ctxlen)) + requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) + assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES)) + ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY || + return_value == MLD_ERR_RNG_FAIL || + return_value == MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED || + return_value == MLD_ERR_SIGNING_PAUSED || + return_value == MLD_ERR_INVALID_ARG) + ensures((return_value == MLD_ERR_INVALID_ARG) ==> (ctxlen > 255)) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(sig, MLDSA_CRYPTO_BYTES)) +); + +/** + * Compute signature in "external mu" mode: the caller has already computed + * the message representative mu = SHAKE256(tr || M', 64), where + * tr = SHAKE256(pk, 64) and M' is the FIPS 204 formatted message (e.g. + * 0x00 || ctxlen || ctx || msg for pure ML-DSA). This is the randomized + * variant; for the deterministic variant, use mld_sign_signature_internal + * directly with externalmu set to non-zero and an all-zero rnd. + * + * @spec{Implements @[FIPS204, Algorithm 2, ML-DSA.Sign external mu variant].} + * + * @warning This function does not perform secret key validation. + * Callers importing serialized keys can use mld_sign_pk_from_sk + * to validate them before signing. + * + * @param[out] sig Output signature. + * @param[in] mu Precomputed message representative. + * @param[in] sk Bit-packed secret key; assumed to be valid. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_RNG_FAIL Random number generation failed. + * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS + * iterations. + * @retval MLD_ERR_SIGNING_PAUSED A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook + * paused signing; re-invoke to resume. + */ +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_signature_extmu(uint8_t sig[MLDSA_CRYPTO_BYTES], + const uint8_t mu[MLDSA_CRHBYTES], + const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(mu, MLDSA_CRHBYTES)) + requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) + assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES)) + ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY || + return_value == MLD_ERR_RNG_FAIL || + return_value == MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED || + return_value == MLD_ERR_SIGNING_PAUSED) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(sig, MLDSA_CRYPTO_BYTES)) +); + +#endif /* !MLD_CONFIG_NO_RANDOMIZED_API */ +#endif /* !MLD_CONFIG_CORE_API_ONLY */ +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +/** + * Verify signature. + * + * @spec{Implements @[FIPS204, Algorithm 8, ML-DSA.Verify_internal].} + * + * @param[in] sig Pointer to input signature of + * MLDSA_CRYPTO_BYTES bytes. + * @param[in] m Pointer to message (when externalmu == 0), or to a + * precomputed message representative mu (when + * externalmu != 0). + * @param mlen Length of m. Must equal MLDSA_CRHBYTES when + * externalmu != 0. + * @param[in] pre Pointer to prefix string. Ignored when externalmu != 0. + * @param prelen Length of prefix string. Ignored when externalmu != 0. + * @param[in] pk Bit-packed public key. + * @param externalmu 0: m/mlen is the raw message; mu = H(H(pk), pre, m) is + * computed internally. + * non-zero: m points to a precomputed mu of + * MLDSA_CRHBYTES bytes; pre/prelen unused. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed. + */ +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_verify_internal(const uint8_t sig[MLDSA_CRYPTO_BYTES], + const uint8_t *m, size_t mlen, const uint8_t *pre, + size_t prelen, + const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + int externalmu, + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(prelen <= MLD_MAX_BUFFER_SIZE) + requires(mlen <= MLD_MAX_BUFFER_SIZE) + requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(m, mlen)) + requires((externalmu == 0) ==> ((prelen == 0) || memory_no_alias(pre, prelen))) + requires((externalmu != 0) ==> (mlen == MLDSA_CRHBYTES)) + requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES)) + ensures(return_value == 0 || return_value == MLD_ERR_INVALID_SIGNATURE || return_value == MLD_ERR_OUT_OF_MEMORY) +); + +#if !defined(MLD_CONFIG_CORE_API_ONLY) +/** + * Verify signature. + * + * @spec{Implements @[FIPS204, Algorithm 3, ML-DSA.Verify].} + * + * @param[in] sig Pointer to input signature. + * @param[in] m Pointer to message. + * @param mlen Length of message. + * @param[in] ctx Pointer to context string. May be NULL if ctxlen == 0. + * @param ctxlen Length of context string. + * @param[in] pk Bit-packed public key. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed. + * @retval MLD_ERR_INVALID_ARG The context string exceeded 255 bytes. + */ +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_verify(const uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m, + size_t mlen, const uint8_t *ctx, size_t ctxlen, + const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(mlen <= MLD_MAX_BUFFER_SIZE) + requires(ctxlen <= MLD_MAX_BUFFER_SIZE) + requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(m, mlen)) + requires(ctxlen == 0 || memory_no_alias(ctx, ctxlen)) + requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES)) + ensures(return_value == 0 || return_value == MLD_ERR_INVALID_SIGNATURE || return_value == MLD_ERR_INVALID_ARG || return_value == MLD_ERR_OUT_OF_MEMORY) + ensures((return_value == MLD_ERR_INVALID_ARG) ==> (ctxlen > 255)) +); + +/** + * Verify signature in "external mu" mode: the caller has already computed + * the message representative mu = SHAKE256(tr || M', 64), where + * tr = SHAKE256(pk, 64) and M' is the FIPS 204 formatted message (e.g. + * 0x00 || ctxlen || ctx || msg for pure ML-DSA). The same mu must have been + * used at signing time. + * + * @spec{Implements @[FIPS204, Algorithm 3, ML-DSA.Verify external mu variant].} + * + * @param[in] sig Pointer to input signature. + * @param[in] mu Precomputed message representative. + * @param[in] pk Bit-packed public key. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed. + */ +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_verify_extmu(const uint8_t sig[MLDSA_CRYPTO_BYTES], + const uint8_t mu[MLDSA_CRHBYTES], + const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(mu, MLDSA_CRHBYTES)) + requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES)) + ensures(return_value == 0 || return_value == MLD_ERR_INVALID_SIGNATURE || return_value == MLD_ERR_OUT_OF_MEMORY) +); + +#endif /* !MLD_CONFIG_CORE_API_ONLY */ +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_CORE_API_ONLY) +#if !defined(MLD_CONFIG_NO_SIGN_API) +/** + * Compute signature with pre-hashed message. + * + * @spec{Implements @[FIPS204, Algorithm 4, HashML-DSA.Sign].} + * + * Supported hash algorithm constants: + * MLD_PREHASH_SHA2_224, MLD_PREHASH_SHA2_256, MLD_PREHASH_SHA2_384, + * MLD_PREHASH_SHA2_512, MLD_PREHASH_SHA2_512_224, MLD_PREHASH_SHA2_512_256, + * MLD_PREHASH_SHA3_224, MLD_PREHASH_SHA3_256, MLD_PREHASH_SHA3_384, + * MLD_PREHASH_SHA3_512, MLD_PREHASH_SHAKE_128, MLD_PREHASH_SHAKE_256. + * + * MLD_PREHASH_NONE is rejected by this API. + * + * @warning This is an unstable API that may change in the future. If you need + * a stable API use mld_sign_signature_pre_hash_shake256. + * + * @warning This function does not perform secret key validation. + * Callers importing serialized keys can use mld_sign_pk_from_sk + * to validate them before signing. + * + * @param[out] sig Output signature. + * @param[in] ph Pointer to pre-hashed message. + * @param phlen Length of pre-hashed message. Must match the output + * length of hashalg (the digest size for SHA-2/SHA-3, + * 32 bytes for MLD_PREHASH_SHAKE_128, 64 bytes for + * MLD_PREHASH_SHAKE_256). + * @param[in] ctx Pointer to context string. May be NULL if ctxlen == 0. + * @param ctxlen Length of context string. + * @param[in] rnd Random seed. + * @param[in] sk Bit-packed secret key; assumed to be valid. + * @param hashalg Hash algorithm constant (one of MLD_PREHASH_*). + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS + * iterations. + * @retval MLD_ERR_SIGNING_PAUSED A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook + * paused signing; re-invoke to resume. + * @retval MLD_ERR_INVALID_ARG The pre-hash algorithm was + * MLD_PREHASH_NONE or unsupported, + * phlen did not match the output + * length of hashalg, or the context + * string exceeded 255 bytes. + */ +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_signature_pre_hash_internal( + uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *ph, size_t phlen, + const uint8_t *ctx, size_t ctxlen, const uint8_t rnd[MLDSA_RNDBYTES], + const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], int hashalg, + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(ctxlen <= MLD_MAX_BUFFER_SIZE) + requires(phlen <= MLD_MAX_BUFFER_SIZE) + requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(ph, phlen)) + requires(ctxlen == 0 || memory_no_alias(ctx, ctxlen)) + requires(memory_no_alias(rnd, MLDSA_RNDBYTES)) + requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) + assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES)) + ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY || return_value == MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED || return_value == MLD_ERR_SIGNING_PAUSED || return_value == MLD_ERR_INVALID_ARG) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(sig, MLDSA_CRYPTO_BYTES)) +); +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +/** + * Verify signature with pre-hashed message. + * + * @spec{Implements @[FIPS204, Algorithm 5, HashML-DSA.Verify].} + * + * Supported hash algorithm constants: + * MLD_PREHASH_SHA2_224, MLD_PREHASH_SHA2_256, MLD_PREHASH_SHA2_384, + * MLD_PREHASH_SHA2_512, MLD_PREHASH_SHA2_512_224, MLD_PREHASH_SHA2_512_256, + * MLD_PREHASH_SHA3_224, MLD_PREHASH_SHA3_256, MLD_PREHASH_SHA3_384, + * MLD_PREHASH_SHA3_512, MLD_PREHASH_SHAKE_128, MLD_PREHASH_SHAKE_256. + * + * MLD_PREHASH_NONE is rejected by this API. + * + * @warning This is an unstable API that may change in the future. If you need + * a stable API use mld_sign_verify_pre_hash_shake256. + * + * @param[in] sig Pointer to input signature. + * @param[in] ph Pointer to pre-hashed message. + * @param phlen Length of pre-hashed message. Must match the output + * length of hashalg (the digest size for SHA-2/SHA-3, + * 32 bytes for MLD_PREHASH_SHAKE_128, 64 bytes for + * MLD_PREHASH_SHAKE_256). + * @param[in] ctx Pointer to context string. May be NULL if ctxlen == 0. + * @param ctxlen Length of context string. + * @param[in] pk Bit-packed public key. + * @param hashalg Hash algorithm constant (one of MLD_PREHASH_*). + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed. + * @retval MLD_ERR_INVALID_ARG The pre-hash algorithm was + * MLD_PREHASH_NONE or unsupported, phlen + * did not match the output length of + * hashalg, or the context string exceeded + * 255 bytes. + */ +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_verify_pre_hash_internal( + const uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *ph, size_t phlen, + const uint8_t *ctx, size_t ctxlen, + const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], int hashalg, + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(phlen <= MLD_MAX_BUFFER_SIZE) + requires(ctxlen <= MLD_MAX_BUFFER_SIZE - 77) + requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(ph, phlen)) + requires(ctxlen == 0 || memory_no_alias(ctx, ctxlen)) + requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES)) + ensures(return_value == 0 || return_value == MLD_ERR_INVALID_SIGNATURE || return_value == MLD_ERR_INVALID_ARG || return_value == MLD_ERR_OUT_OF_MEMORY) +); +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) +/** + * Compute signature with pre-hashed message using SHAKE256. This function + * computes the SHAKE256 hash of the message internally. + * + * @spec{Implements @[FIPS204, Algorithm 4, HashML-DSA.Sign] with SHAKE256 as + * the pre-hash.} + * + * @warning This function does not perform secret key validation. + * Callers importing serialized keys can use mld_sign_pk_from_sk + * to validate them before signing. + * + * @param[out] sig Output signature. + * @param[in] m Pointer to message to be hashed and signed. + * @param mlen Length of message. + * @param[in] ctx Pointer to context string. May be NULL if ctxlen == 0. + * @param ctxlen Length of context string. + * @param[in] rnd Random seed. + * @param[in] sk Bit-packed secret key; assumed to be valid. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED The rejection-sampling loop exceeded + * MLD_CONFIG_MAX_SIGNING_ATTEMPTS + * iterations. + * @retval MLD_ERR_SIGNING_PAUSED A MLD_CONFIG_SIGN_HOOK_ATTEMPT hook + * paused signing; re-invoke to resume. + * @retval MLD_ERR_INVALID_ARG The context string exceeded 255 + * bytes. + */ +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_signature_pre_hash_shake256( + uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m, size_t mlen, + const uint8_t *ctx, size_t ctxlen, const uint8_t rnd[MLDSA_RNDBYTES], + const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(mlen <= MLD_MAX_BUFFER_SIZE) + requires(ctxlen <= MLD_MAX_BUFFER_SIZE) + requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(m, mlen)) + requires(ctxlen == 0 || memory_no_alias(ctx, ctxlen)) + requires(memory_no_alias(rnd, MLDSA_RNDBYTES)) + requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) + assigns(memory_slice(sig, MLDSA_CRYPTO_BYTES)) + ensures(return_value == 0 || return_value == MLD_ERR_OUT_OF_MEMORY || return_value == MLD_ERR_SIGN_ATTEMPTS_EXHAUSTED || return_value == MLD_ERR_SIGNING_PAUSED || return_value == MLD_ERR_INVALID_ARG) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(sig, MLDSA_CRYPTO_BYTES)) +); +#endif /* !MLD_CONFIG_NO_SIGN_API */ + +#if !defined(MLD_CONFIG_NO_VERIFY_API) +/** + * Verify signature with pre-hashed message using SHAKE256. This function + * computes the SHAKE256 hash of the message internally. + * + * @spec{Implements @[FIPS204, Algorithm 5, HashML-DSA.Verify] with SHAKE256 as + * the pre-hash.} + * + * @param[in] sig Pointer to input signature. + * @param[in] m Pointer to message to be hashed and verified. + * @param mlen Length of message. + * @param[in] ctx Pointer to context string. May be NULL if ctxlen == 0. + * @param ctxlen Length of context string. + * @param[in] pk Bit-packed public key. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was + * used and an allocation via + * MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_INVALID_SIGNATURE Signature verification failed. + * @retval MLD_ERR_INVALID_ARG The context string exceeded 255 bytes. + */ +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_verify_pre_hash_shake256( + const uint8_t sig[MLDSA_CRYPTO_BYTES], const uint8_t *m, size_t mlen, + const uint8_t *ctx, size_t ctxlen, + const uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(mlen <= MLD_MAX_BUFFER_SIZE) + requires(ctxlen <= MLD_MAX_BUFFER_SIZE - 77) + requires(memory_no_alias(sig, MLDSA_CRYPTO_BYTES)) + requires(memory_no_alias(m, mlen)) + requires(ctxlen == 0 || memory_no_alias(ctx, ctxlen)) + requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES)) + ensures(return_value == 0 || return_value == MLD_ERR_INVALID_SIGNATURE || return_value == MLD_ERR_INVALID_ARG || return_value == MLD_ERR_OUT_OF_MEMORY) +); +#endif /* !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_SIGN_API) || !defined(MLD_CONFIG_NO_VERIFY_API) +/* Maximum formatted domain separation message length: + * - Pure ML-DSA: 0x00 || ctxlen || ctx (max 255) + * - HashML-DSA: 0x01 || ctxlen || ctx (max 255) || oid (11) || ph (max 64) */ +#define MLD_DOMAIN_SEPARATION_MAX_BYTES (2 + 255 + 11 + 64) + +/** + * Prepare domain separation prefix for ML-DSA signing. + * + * For pure ML-DSA (hashalg == MLD_PREHASH_NONE): + * Format: 0x00 || ctxlen (1 byte) || ctx. + * + * For HashML-DSA (hashalg != MLD_PREHASH_NONE): + * Format: 0x01 || ctxlen (1 byte) || ctx || oid (11 bytes) || ph. + * + * This function is useful for building incremental signing APIs. + * + * @spec{For HashML-DSA (hashalg != MLD_PREHASH_NONE), implements + * @[FIPS204, Algorithm 4, line 23]. For Pure ML-DSA + * (hashalg == MLD_PREHASH_NONE), implements + * ``` + * M' <- BytesToBits(IntegerToBytes(0, 1) + * || IntegerToBytes(|ctx|, 1) + * || ctx + * ``` + * which is part of @[FIPS204, Algorithm 2, ML-DSA.Sign, line 10] and + * @[FIPS204, Algorithm 3, ML-DSA.Verify, line 5].} + * + * @param[out] prefix Output domain separation prefix buffer. + * @param[in] ph Pointer to pre-hashed message (ignored for pure + * ML-DSA). + * @param phlen Length of pre-hashed message; must match the output + * length of hashalg (ignored for pure ML-DSA). + * @param[in] ctx Pointer to context string. May be NULL if ctxlen == 0. + * @param ctxlen Length of context string. + * @param hashalg Hash algorithm constant (MLD_PREHASH_NONE for pure + * ML-DSA, or MLD_PREHASH_* for HashML-DSA). + * + * @return The total length of the formatted prefix, or 0 on error. + * Errors are: + * - The context string exceeded 255 bytes. + * - For HashML-DSA: hashalg was unsupported, ph was NULL, or phlen + * did not match the output length of hashalg. + */ +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +size_t mld_prepare_domain_separation_prefix( + uint8_t prefix[MLD_DOMAIN_SEPARATION_MAX_BYTES], const uint8_t *ph, + size_t phlen, const uint8_t *ctx, size_t ctxlen, int hashalg) +__contract__( + requires(ctxlen <= 255) + requires(phlen <= MLD_MAX_BUFFER_SIZE) + requires(ctxlen == 0 || memory_no_alias(ctx, ctxlen)) + requires(hashalg == MLD_PREHASH_NONE || memory_no_alias(ph, phlen)) + requires(memory_no_alias(prefix, MLD_DOMAIN_SEPARATION_MAX_BYTES)) + assigns(memory_slice(prefix, MLD_DOMAIN_SEPARATION_MAX_BYTES)) + ensures(return_value <= MLD_DOMAIN_SEPARATION_MAX_BYTES) +); +#endif /* !MLD_CONFIG_NO_SIGN_API || !MLD_CONFIG_NO_VERIFY_API */ + +#if !defined(MLD_CONFIG_NO_KEYPAIR_API) +/** + * Perform basic validity checks on secret key, and derive public key. + * + * Referring to the decoding of the secret key `sk=(rho, K, tr, s1, s2, t0)` + * (cf. @[FIPS204, Algorithm 25, skDecode]), the following checks are + * performed: + * - Check that s1 and s2 have coefficients in [-MLDSA_ETA, MLDSA_ETA]. + * - Check that t0 and tr stored in sk match recomputed values. + * + * @note This function leaks whether the secret key is valid or invalid + * through its return value and timing. + * + * @param[out] pk Output public key. + * @param[in] sk Input secret key. + * @param context Application context. Only present when + * MLD_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLD_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLD_ERR_OUT_OF_MEMORY MLD_CONFIG_CUSTOM_ALLOC_FREE was used and an + * allocation via MLD_CUSTOM_ALLOC returned NULL. + * @retval MLD_ERR_INVALID_KEY Secret key validation failed. + */ +MLD_MUST_CHECK_RETURN_VALUE +MLD_EXTERNAL_API +int mld_sign_pk_from_sk(uint8_t pk[MLDSA_CRYPTO_PUBLICKEYBYTES], + const uint8_t sk[MLDSA_CRYPTO_SECRETKEYBYTES], + MLD_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(pk, MLDSA_CRYPTO_PUBLICKEYBYTES)) + requires(memory_no_alias(sk, MLDSA_CRYPTO_SECRETKEYBYTES)) + assigns(memory_slice(pk, MLDSA_CRYPTO_PUBLICKEYBYTES)) + ensures(return_value == 0 || return_value == MLD_ERR_INVALID_KEY || return_value == MLD_ERR_OUT_OF_MEMORY) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(pk, MLDSA_CRYPTO_PUBLICKEYBYTES)) +); +#endif /* !MLD_CONFIG_NO_KEYPAIR_API */ +#endif /* !MLD_CONFIG_CORE_API_ONLY */ + +#endif /* !MLD_SIGN_H */ diff --git a/cbits/mldsa/src/symmetric.h b/cbits/mldsa/src/symmetric.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/symmetric.h @@ -0,0 +1,68 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLD_SYMMETRIC_H +#define MLD_SYMMETRIC_H + +#include "cbmc.h" +#include "common.h" + +#include MLD_FIPS202_HEADER_FILE +#if !defined(MLD_CONFIG_SERIAL_FIPS202_ONLY) +#include MLD_FIPS202X4_HEADER_FILE +#endif + +#define MLD_STREAM128_BLOCKBYTES SHAKE128_RATE +#define MLD_STREAM256_BLOCKBYTES SHAKE256_RATE + +#define mld_xof256_ctx mld_shake256ctx +#define mld_xof256_init(CTX) mld_shake256_init(CTX) + +#define mld_xof256_absorb_once(CTX, IN, INBYTES) \ + do \ + { \ + mld_shake256_absorb(CTX, IN, INBYTES); \ + mld_shake256_finalize(CTX); \ + } while (0) + + +#define mld_xof256_release(CTX) mld_shake256_release(CTX) +#define mld_xof256_squeezeblocks(OUT, OUTBLOCKS, STATE) \ + mld_shake256_squeeze(OUT, (OUTBLOCKS) * SHAKE256_RATE, STATE) + +#define mld_xof128_ctx mld_shake128ctx +#define mld_xof128_init(CTX) mld_shake128_init(CTX) + +#define mld_xof128_absorb_once(CTX, IN, INBYTES) \ + do \ + { \ + mld_shake128_absorb(CTX, IN, INBYTES); \ + mld_shake128_finalize(CTX); \ + } while (0) + +#define mld_xof128_release(CTX) mld_shake128_release(CTX) +#define mld_xof128_squeezeblocks(OUT, OUTBLOCKS, STATE) \ + mld_shake128_squeeze(OUT, (OUTBLOCKS) * SHAKE128_RATE, STATE) + +#define mld_xof256_x4_ctx mld_shake256x4ctx +#define mld_xof256_x4_init(CTX) mld_shake256x4_init((CTX)) +#define mld_xof256_x4_absorb(CTX, IN, INBYTES) \ + mld_shake256x4_absorb_once((CTX), (IN)[0], (IN)[1], (IN)[2], (IN)[3], \ + (INBYTES)) +#define mld_xof256_x4_squeezeblocks(BUF, NBLOCKS, CTX) \ + mld_shake256x4_squeezeblocks((BUF)[0], (BUF)[1], (BUF)[2], (BUF)[3], \ + (NBLOCKS), (CTX)) +#define mld_xof256_x4_release(CTX) mld_shake256x4_release((CTX)) + +#define mld_xof128_x4_ctx mld_shake128x4ctx +#define mld_xof128_x4_init(CTX) mld_shake128x4_init((CTX)) +#define mld_xof128_x4_absorb(CTX, IN, INBYTES) \ + mld_shake128x4_absorb_once((CTX), (IN)[0], (IN)[1], (IN)[2], (IN)[3], \ + (INBYTES)) +#define mld_xof128_x4_squeezeblocks(BUF, NBLOCKS, CTX) \ + mld_shake128x4_squeezeblocks((BUF)[0], (BUF)[1], (BUF)[2], (BUF)[3], \ + (NBLOCKS), (CTX)) +#define mld_xof128_x4_release(CTX) mld_shake128x4_release((CTX)) + +#endif /* !MLD_SYMMETRIC_H */ diff --git a/cbits/mldsa/src/sys.h b/cbits/mldsa/src/sys.h new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/sys.h @@ -0,0 +1,327 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLD_SYS_H +#define MLD_SYS_H + +#if !defined(MLD_CONFIG_NO_ASM) && (defined(__GNUC__) || defined(__clang__)) +#define MLD_HAVE_INLINE_ASM +#endif + +/* Try to find endianness, if not forced through CFLAGS already */ +#if !defined(MLD_SYS_LITTLE_ENDIAN) && !defined(MLD_SYS_BIG_ENDIAN) +#if defined(__BYTE_ORDER__) +#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ +#define MLD_SYS_LITTLE_ENDIAN +#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ +#define MLD_SYS_BIG_ENDIAN +#else +#error "__BYTE_ORDER__ defined, but don't recognize value." +#endif +#endif /* __BYTE_ORDER__ */ + +/* MSVC does not define __BYTE_ORDER__. However, MSVC only supports + * little endian x86, x86_64, and AArch64. It is, hence, safe to assume + * little endian. */ +#if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_AMD64) || \ + defined(_M_IX86) || defined(_M_ARM64)) +#define MLD_SYS_LITTLE_ENDIAN +#endif + +#endif /* !MLD_SYS_LITTLE_ENDIAN && !MLD_SYS_BIG_ENDIAN */ + +/* Check if we're running on an AArch64 little endian system. _M_ARM64 is set by + * MSVC. */ +#if defined(__AARCH64EL__) || defined(_M_ARM64) +#define MLD_SYS_AARCH64 +#endif + +/* Check if the AArch64 compilation target supports NEON (Advanced SIMD). + * + * Some compilers also define __ARM_NEON__, but __ARM_NEON is the most reliable + * signal. Specifically, clang on Apple appears to keep __ARM_NEON__ set even if + * -march=armv8-a+nosimd is set. + * + * gcc 4.8 -- the first gcc version introducing Neon support -- sets neither + * __ARM_NEON nor __ARM_NEON__; in fact, there is no preprocessor signal that + * Neon is enabled. If you use gcc 4.8 and need Neon, you should set + * MLD_SYS_AARCH64_NEON manually. gcc 4.9 onwards do set __ARM_NEON. + */ +#if defined(MLD_SYS_AARCH64) && defined(__ARM_NEON) +#define MLD_SYS_AARCH64_NEON +#endif + +/* Check if we're running on an AArch64 big endian system. */ +#if defined(__AARCH64EB__) +#define MLD_SYS_AARCH64_EB +#endif + +/* Check if we're running on an Armv8.1-M system with MVE */ +#if defined(__ARM_ARCH_8_1M_MAIN__) || defined(__ARM_FEATURE_MVE) +#define MLD_SYS_ARMV81M_MVE +#endif + +/* Check if we're running on an x86_64 system. */ +#if defined(__x86_64__) || defined(_M_X64) || defined(_M_AMD64) +#define MLD_SYS_X86_64 +#if defined(__AVX2__) +#define MLD_SYS_X86_64_AVX2 +#endif +#endif /* __x86_64__ || _M_X64 || _M_AMD64 */ + +#if defined(MLD_SYS_LITTLE_ENDIAN) && defined(__powerpc64__) +#define MLD_SYS_PPC64LE +#endif + +#if defined(__riscv) && defined(__riscv_xlen) && __riscv_xlen == 64 +#define MLD_SYS_RISCV64 +#endif + +#if defined(MLD_SYS_RISCV64) && defined(__riscv_vector) && \ + defined(__riscv_v_intrinsic) +#define MLD_SYS_RISCV64_RVV +#endif + +#if defined(__riscv) && defined(__riscv_xlen) && __riscv_xlen == 32 +#define MLD_SYS_RISCV32 +#endif + +#if defined(_WIN64) || defined(_WIN32) +#define MLD_SYS_WINDOWS +#endif + +#if defined(__linux__) +#define MLD_SYS_LINUX +#endif + +#if defined(__APPLE__) +#define MLD_SYS_APPLE +#endif + +/* If MLD_FORCE_AARCH64 is set, assert that we're indeed on an AArch64 system. + */ +#if defined(MLD_FORCE_AARCH64) && !defined(MLD_SYS_AARCH64) +#error "MLD_FORCE_AARCH64 is set, but we don't seem to be on an AArch64 system." +#endif + +/* If MLD_FORCE_AARCH64_EB is set, assert that we're indeed on a big endian + * AArch64 system. */ +#if defined(MLD_FORCE_AARCH64_EB) && !defined(MLD_SYS_AARCH64_EB) +#error \ + "MLD_FORCE_AARCH64_EB is set, but we don't seem to be on an AArch64 system." +#endif + +/* If MLD_FORCE_X86_64 is set, assert that we're indeed on an X86_64 system. */ +#if defined(MLD_FORCE_X86_64) && !defined(MLD_SYS_X86_64) +#error "MLD_FORCE_X86_64 is set, but we don't seem to be on an X86_64 system." +#endif + +#if defined(MLD_FORCE_PPC64LE) && !defined(MLD_SYS_PPC64LE) +#error "MLD_FORCE_PPC64LE is set, but we don't seem to be on a PPC64LE system." +#endif + +#if defined(MLD_FORCE_RISCV64) && !defined(MLD_SYS_RISCV64) +#error "MLD_FORCE_RISCV64 is set, but we don't seem to be on a RISCV64 system." +#endif + +#if defined(MLD_FORCE_RISCV32) && !defined(MLD_SYS_RISCV32) +#error "MLD_FORCE_RISCV32 is set, but we don't seem to be on a RISCV32 system." +#endif + +/* + * MLD_INLINE: Hint for inlining. + * - MSVC: __inline + * - C99+: inline + * - GCC/Clang C90: __attribute__((unused)) to silence warnings + * - Other C90: empty + */ +#if !defined(MLD_INLINE) +#if defined(_MSC_VER) +#define MLD_INLINE __inline +#elif defined(inline) || \ + (defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L) +#define MLD_INLINE inline +#elif defined(__GNUC__) || defined(__clang__) +#define MLD_INLINE __attribute__((unused)) +#else +#define MLD_INLINE +#endif +#endif /* !MLD_INLINE */ + +/* + * MLD_ALWAYS_INLINE: Force inlining. + * - MSVC: __forceinline + * - GCC/Clang C99+: MLD_INLINE __attribute__((always_inline)) + * - Other: MLD_INLINE (no forced inlining) + */ +#if !defined(MLD_ALWAYS_INLINE) +#if defined(_MSC_VER) +#define MLD_ALWAYS_INLINE __forceinline +#elif (defined(__GNUC__) || defined(__clang__)) && \ + (defined(inline) || \ + (defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L)) +#define MLD_ALWAYS_INLINE MLD_INLINE __attribute__((always_inline)) +#else +#define MLD_ALWAYS_INLINE MLD_INLINE +#endif +#endif /* !MLD_ALWAYS_INLINE */ + +/* + * MLD_NOINLINE: Prevent inlining. + * - MSVC: __declspec(noinline) + * - GCC/Clang: __attribute__((noinline)) + * - Other: empty + */ +#if !defined(MLD_NOINLINE) +#if defined(_MSC_VER) +#define MLD_NOINLINE __declspec(noinline) +#elif defined(__GNUC__) || defined(__clang__) +#define MLD_NOINLINE __attribute__((noinline)) +#else +#define MLD_NOINLINE +#endif +#endif /* !MLD_NOINLINE */ + +#ifndef MLD_STATIC_TESTABLE +#define MLD_STATIC_TESTABLE static +#endif + +/* + * C90 does not have the restrict compiler directive yet. + * We don't use it in C90 builds. + */ +#if !defined(restrict) +#if defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L +#define MLD_RESTRICT restrict +#else +#define MLD_RESTRICT +#endif + +#else /* !restrict */ + +#define MLD_RESTRICT restrict +#endif /* restrict */ + +#define MLD_DEFAULT_ALIGN 32 +#define MLD_ALIGN_UP(N) \ + ((((N) + (MLD_DEFAULT_ALIGN - 1)) / MLD_DEFAULT_ALIGN) * MLD_DEFAULT_ALIGN) +#if defined(__GNUC__) +#define MLD_ALIGN __attribute__((aligned(MLD_DEFAULT_ALIGN))) +#elif defined(_MSC_VER) +#define MLD_ALIGN __declspec(align(MLD_DEFAULT_ALIGN)) +#else +#define MLD_ALIGN /* No known support for alignment constraints */ +#endif + + +/* New X86_64 CPUs support control-flow protection using the CET instructions. + * When enabled (through -fcf-protection=), all compilation units (including + * empty ones) need to support CET for this to work. + * For assembly, this means that source files need to signal support for + * CET by setting the appropriate note.gnu.property section. + * This can be achieved by including the header in all assembly file. + * This file also provides the _CET_ENDBR macro which needs to be placed at + * every potential target of an indirect branch. + * If CET is enabled _CET_ENDBR maps to the endbr64 instruction, otherwise + * it is empty. + * In case the compiler does not support CET (e.g., +#define MLD_CET_ENDBR _CET_ENDBR +#else +#define MLD_CET_ENDBR +#endif +#endif /* MLD_SYS_X86_64 */ + +#if defined(MLD_CONFIG_CT_TESTING_ENABLED) && !defined(__ASSEMBLER__) +#include +#define MLD_CT_TESTING_SECRET(ptr, len) \ + VALGRIND_MAKE_MEM_UNDEFINED((ptr), (len)) +#define MLD_CT_TESTING_DECLASSIFY(ptr, len) \ + VALGRIND_MAKE_MEM_DEFINED((ptr), (len)) +#else /* MLD_CONFIG_CT_TESTING_ENABLED && !__ASSEMBLER__ */ +#define MLD_CT_TESTING_SECRET(ptr, len) \ + do \ + { \ + } while (0) +#define MLD_CT_TESTING_DECLASSIFY(ptr, len) \ + do \ + { \ + } while (0) +#endif /* !(MLD_CONFIG_CT_TESTING_ENABLED && !__ASSEMBLER__) */ + +#if defined(__GNUC__) || defined(__clang__) +#define MLD_MUST_CHECK_RETURN_VALUE __attribute__((warn_unused_result)) +#else +#define MLD_MUST_CHECK_RETURN_VALUE +#endif + +/* The x86_64 assembly backend uses the SysV calling convention. On Windows, + * where the Microsoft x64 calling convention is the default, it can still be + * used with compilers that allow choosing the calling convention per + * function: GCC and Clang support __attribute__((sysv_abi)), which makes + * calls to the annotated function follow the SysV calling convention. + * + * MLD_SYSV_ABI_SUPPORTED signals that the toolchain can call SysV assembly + * routines; the x86_64 assembly backend is only enabled if it is defined. + * MLD_SYSV_ABI is the attribute carried by declarations of x86_64 assembly + * routines. Both macros can be set externally for toolchains offering an + * equivalent mechanism that is not recognized here. */ +#if defined(MLD_SYS_X86_64) && !defined(MLD_SYSV_ABI_SUPPORTED) +#if !defined(MLD_SYS_WINDOWS) || defined(__GNUC__) || defined(__clang__) +#define MLD_SYSV_ABI_SUPPORTED +#endif +#endif + +#if !defined(MLD_SYSV_ABI) +#if defined(MLD_SYS_WINDOWS) && defined(MLD_SYSV_ABI_SUPPORTED) +#define MLD_SYSV_ABI __attribute__((sysv_abi)) +#else +#define MLD_SYSV_ABI +#endif +#endif /* !MLD_SYSV_ABI */ + +#if !defined(__ASSEMBLER__) +/* System capability enumeration */ +typedef enum +{ + /* x86_64 */ + MLD_SYS_CAP_X86_64_AVX2, + /* AArch64 */ + MLD_SYS_CAP_AARCH64_NEON, + MLD_SYS_CAP_AARCH64_SHA3, + /* Armv8.1-M */ + MLD_SYS_CAP_ARMV81M_MVE +} mld_sys_cap; + +#if !defined(MLD_CONFIG_CUSTOM_CAPABILITY_FUNC) +#include "cbmc.h" + +MLD_MUST_CHECK_RETURN_VALUE +static MLD_INLINE int mld_sys_check_capability(mld_sys_cap cap) +__contract__( + ensures(return_value == 0 || return_value == 1) +) +{ + /* By default, we rely on compile-time feature detection/specification: + * If a feature is enabled at compile-time, we assume it is supported by + * the host that the resulting library/binary will be built on. + * If this assumption is not true, you MUST overwrite this function. + * See the documentation of MLD_CONFIG_CUSTOM_CAPABILITY_FUNC in + * mldsa_native_config.h for more information. */ + (void)cap; + return 1; +} +#endif /* !MLD_CONFIG_CUSTOM_CAPABILITY_FUNC */ +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLD_SYS_H */ diff --git a/cbits/mldsa/src/zetas.inc b/cbits/mldsa/src/zetas.inc new file mode 100644 --- /dev/null +++ b/cbits/mldsa/src/zetas.inc @@ -0,0 +1,55 @@ +/* + * Copyright (c) The mldsa-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mldsa-native repository. + * Do not modify it directly. + */ + + +/* + * Table of zeta values used in the reference NTT and inverse NTT. + * See autogen for details. + */ +static const int32_t mld_zetas[MLDSA_N] = { + 0, 25847, -2608894, -518909, 237124, -777960, -876248, + 466468, 1826347, 2353451, -359251, -2091905, 3119733, -2884855, + 3111497, 2680103, 2725464, 1024112, -1079900, 3585928, -549488, + -1119584, 2619752, -2108549, -2118186, -3859737, -1399561, -3277672, + 1757237, -19422, 4010497, 280005, 2706023, 95776, 3077325, + 3530437, -1661693, -3592148, -2537516, 3915439, -3861115, -3043716, + 3574422, -2867647, 3539968, -300467, 2348700, -539299, -1699267, + -1643818, 3505694, -3821735, 3507263, -2140649, -1600420, 3699596, + 811944, 531354, 954230, 3881043, 3900724, -2556880, 2071892, + -2797779, -3930395, -1528703, -3677745, -3041255, -1452451, 3475950, + 2176455, -1585221, -1257611, 1939314, -4083598, -1000202, -3190144, + -3157330, -3632928, 126922, 3412210, -983419, 2147896, 2715295, + -2967645, -3693493, -411027, -2477047, -671102, -1228525, -22981, + -1308169, -381987, 1349076, 1852771, -1430430, -3343383, 264944, + 508951, 3097992, 44288, -1100098, 904516, 3958618, -3724342, + -8578, 1653064, -3249728, 2389356, -210977, 759969, -1316856, + 189548, -3553272, 3159746, -1851402, -2409325, -177440, 1315589, + 1341330, 1285669, -1584928, -812732, -1439742, -3019102, -3881060, + -3628969, 3839961, 2091667, 3407706, 2316500, 3817976, -3342478, + 2244091, -2446433, -3562462, 266997, 2434439, -1235728, 3513181, + -3520352, -3759364, -1197226, -3193378, 900702, 1859098, 909542, + 819034, 495491, -1613174, -43260, -522500, -655327, -3122442, + 2031748, 3207046, -3556995, -525098, -768622, -3595838, 342297, + 286988, -2437823, 4108315, 3437287, -3342277, 1735879, 203044, + 2842341, 2691481, -2590150, 1265009, 4055324, 1247620, 2486353, + 1595974, -3767016, 1250494, 2635921, -3548272, -2994039, 1869119, + 1903435, -1050970, -1333058, 1237275, -3318210, -1430225, -451100, + 1312455, 3306115, -1962642, -1279661, 1917081, -2546312, -1374803, + 1500165, 777191, 2235880, 3406031, -542412, -2831860, -1671176, + -1846953, -2584293, -3724270, 594136, -3776993, -2013608, 2432395, + 2454455, -164721, 1957272, 3369112, 185531, -1207385, -3183426, + 162844, 1616392, 3014001, 810149, 1652634, -3694233, -1799107, + -3038916, 3523897, 3866901, 269760, 2213111, -975884, 1717735, + 472078, -426683, 1723600, -1803090, 1910376, -1667432, -1104333, + -260646, -3833893, -2939036, -2235985, -420899, -2286327, 183443, + -976891, 1612842, -3545687, -554416, 3919660, -48306, -1362209, + 3937738, 1400424, -846154, 1976782, +}; diff --git a/cbits/mlkem/COMMIT b/cbits/mlkem/COMMIT new file mode 100644 --- /dev/null +++ b/cbits/mlkem/COMMIT @@ -0,0 +1,2 @@ +d1b2fe782888bdb761a50336012923180be7f502 +v2.0.0 diff --git a/cbits/mlkem/LICENSE b/cbits/mlkem/LICENSE new file mode 100644 --- /dev/null +++ b/cbits/mlkem/LICENSE @@ -0,0 +1,312 @@ +mlkem-native is a fork of the public domain Kyber reference +implementation, available on https://github.com/pq-crystals/kyber. + +All source code in mlkem/* and dev/* is licensed under your choice +of the Apache-2.0 license OR the ISC license OR the MIT license. +These licenses are reproduced at the bottom of this file. +The copyright holders are indicated at the top of each file. + +Files outside the library itself may carry different terms. Every file +states its own SPDX-License-Identifier, which determines the terms that +apply to it. In particular: + +The code in test/notrandombytes/*, and its copies in +examples/*/test_only_rng/* and scripts/notrandombytes, is derived from +https://cr.yp.to/papers.html#surf and licensed under +LicenseRef-PD-hp OR CC0-1.0 OR 0BSD OR MIT-0 OR MIT. +It is only used for testing purposes. + +The benchmarking code in test/hal/* carries the +MIT license. It is only used for testing purposes. + +The tiny_sha3 code in +examples/bring_your_own_fips202/custom_fips202/tiny_sha3/* and +examples/custom_backend/mlkem_native/src/fips202/native/custom/src/* +carries the MIT license. It is only used to demonstrate custom +FIPS-202 implementations. + +The proofs in proofs/* are in part derived from Amazon Web Services +verification infrastructure. Individual files carry, among others, +Apache-2.0 OR ISC OR MIT-0, MIT-0, and MIT-0 AND Apache-2.0. None of the +proofs are part of the library. + +Documentation is licensed under CC-BY-4.0. + +``` +Copyright (c) The mlkem-native project authors +Copyright (c) 2020 Dougall Johnson +Copyright (c) 2022 Arm Limited +SPDX-License-Identifier: MIT + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +``` + +The tiny_sha3 implementation in examples/custom_backend/mlkem_native/src/fips202/native/custom/src/ +carries the MIT license. It is only used for testing purposes. + +``` +SPDX-License-Identifier: MIT + +19-Nov-11 Markku-Juhani O. Saarinen +``` + +ISC license +----------- + +Copyright + +Permission to use, copy, modify, and/or distribute this software for any purpose +with or without fee is hereby granted, provided that the above copyright notice +and this permission notice appear in all copies. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH +REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND +FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, +INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS +OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER +TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF +THIS SOFTWARE. + + +MIT license +----------- + +Copyright + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the “Software”), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +Apache-2.0 license +------------------ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/cbits/mlkem/README.md b/cbits/mlkem/README.md new file mode 100644 --- /dev/null +++ b/cbits/mlkem/README.md @@ -0,0 +1,59 @@ +# mlkem-native + +ML-KEM (FIPS 203) from the PQ Code Package, vendored here and built for all +three parameter sets. + +## Where it comes from + +. `COMMIT` holds the +revision this tree is at; `import.sh` puts it there and is how the tree is +refreshed. Nothing here is edited by hand -- every choice crypton makes is +made in `crypton_mlkem.h`, `crypton_mlkem.c` and `crypton.cabal`, so that a +re-import is a straight overwrite. + +## Licence + +`Apache-2.0 OR ISC OR MIT`, the same three-way form as `cbits/s2n` and by +some of the same authors. crypton takes it under ISC, which is already in +the package's `license:` field. `LICENSE` is the upstream file and is +listed in `license-files:`. + +## What is taken and what is not + +The whole of `mlkem/`, less the backends for architectures crypton does not +build for: `src/native/riscv64`, `src/native/ppc64le` and the 32-bit +`src/fips202/native/armv81m`. Every reference to those is behind an +`MLK_SYS_` guard that cannot be true on the architectures crypton does +build for, so dropping them changes no build and keeps a few dozen files of +unreachable assembly out of the release tarball. To take one back, delete +its line from `import.sh` and add its directory to `extra-source-files:`. + +## How it is built + +Upstream builds for one parameter set at a time. `crypton_mlkem.c` +includes the amalgamation once per set -- the level-independent half kept by +exactly one of them -- which is how a single crypton offers ML-KEM-512, 768 +and 1024. `crypton_mlkem_asm.S` does the same for the assembly, which is +level-independent and so is included once. + +This is the shape `cbits/aes/armv8.c` already uses for the three AES key +sizes, and it has the same hazard: **cabal does not know that the wrapper +depends on the tree it includes.** After changing anything under +`cbits/mlkem`, touch `crypton_mlkem.c`, or the build keeps the object it +already has and the change is not tested. + +The hand-written backends are selected by `CRYPTON_MLKEM_NATIVE_BACKEND`, +which `crypton.cabal` defines on x86-64 and AArch64 other than Windows -- +the same exclusion, and for the same reasons, as `cbits/s2n`. Everywhere +else the portable C is built, which is the same code and passes the same +tests. + +There is no randomised API: `MLK_CONFIG_NO_RANDOMIZED_API` is set, no +`randombytes()` is needed, and randomness is drawn in Haskell through +`MonadRandom` as it is for every other key crypton generates. + +The symbols are `crypton_mlkem512_*`, `crypton_mlkem768_*` and +`crypton_mlkem1024_*` rather than upstream's defaults, so that an +application linking another copy of mlkem-native -- through some other +library, or its own -- does not present the linker with two sets of +functions answering to one set of names. diff --git a/cbits/mlkem/crypton_mlkem.c b/cbits/mlkem/crypton_mlkem.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/crypton_mlkem.c @@ -0,0 +1,31 @@ +/* + * All three ML-KEM parameter sets in one translation unit. + * + * mlkem-native is built for one parameter set at a time; a build wanting + * several includes the amalgamation once per set, with the level-independent + * half kept by exactly one of them. This is the same shape as + * cbits/aes/armv8.c, which includes cbits/aes/armv8_impl.c three times for + * the three AES key sizes. + * + * NOTE, as there: cabal does not know that this file depends on the tree it + * includes. After changing anything under cbits/mlkem, touch this file, or + * the build will quietly keep the object it already has. + */ +#include "crypton_mlkem.h" + +#define MLK_CONFIG_MULTILEVEL_WITH_SHARED 1 +#define MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS +#define MLK_CONFIG_PARAMETER_SET 512 +#include "mlkem_native.c" +#undef MLK_CONFIG_MULTILEVEL_WITH_SHARED +#undef MLK_CONFIG_PARAMETER_SET + +#define MLK_CONFIG_MULTILEVEL_NO_SHARED +#define MLK_CONFIG_PARAMETER_SET 768 +#include "mlkem_native.c" +#undef MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS +#undef MLK_CONFIG_PARAMETER_SET + +#define MLK_CONFIG_PARAMETER_SET 1024 +#include "mlkem_native.c" +#undef MLK_CONFIG_PARAMETER_SET diff --git a/cbits/mlkem/crypton_mlkem.h b/cbits/mlkem/crypton_mlkem.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/crypton_mlkem.h @@ -0,0 +1,79 @@ +/* + * What crypton asks of the vendored mlkem-native, in one place. The tree + * under cbits/mlkem is upstream's and is overwritten by import.sh, so every + * choice crypton makes is made here instead of by editing it. + * + * Included first by both crypton_mlkem.c and crypton_mlkem_asm.S, so it must + * hold nothing the assembler cannot read: directives, and C behind + * __ASSEMBLER__. + */ +#ifndef CRYPTON_MLKEM_H +#define CRYPTON_MLKEM_H + +/* + * The symbols are crypton's own, not the default PQCP_MLKEM_NATIVE_*. An + * application is free to link another copy of mlkem-native -- through some + * other library, or its own -- and two copies answering to one set of names + * is a problem the linker resolves silently and in nobody's favour. With + * MLK_CONFIG_MULTILEVEL_BUILD the level is appended, so the entry points + * are crypton_mlkem512_*, crypton_mlkem768_* and crypton_mlkem1024_*. + */ +#define MLK_CONFIG_NAMESPACE_PREFIX crypton_mlkem +#define MLK_CONFIG_MULTILEVEL_BUILD + +/* + * No randomised API, so no randombytes() to provide. Randomness is drawn + * in Haskell through MonadRandom, the way every other key in crypton is + * generated, and the deterministic entry points are what the FFI calls. + * That also keeps the C free of any opinion about where entropy comes from. + */ +#define MLK_CONFIG_NO_RANDOMIZED_API + +/* + * The hand-written backends, where crypton.cabal says the architecture has + * them. Without this the portable C is built, which is correct everywhere + * and is what every other architecture gets. + */ +#ifdef CRYPTON_MLKEM_NATIVE_BACKEND +#define MLK_CONFIG_USE_NATIVE_BACKEND_ARITH +#define MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 + +/* + * Which of the backend's instructions this processor actually has. + * + * Without this both packages decide at compile time: the backend is built + * because the architecture has the instructions in principle, and is then + * called with no check. Upstream says so plainly -- if the machine that + * runs the binary may differ from the one that built it, this function + * MUST be provided -- and crypton is a library whose builds are copied + * between machines and shipped by distributions. Without it a binary + * built on a host with AVX2 dies with SIGILL on one without (#316). + * + * The dispatch sites are upstream's; each falls back to the portable C + * when this answers no. The answers come from crypton's own detection, + * the same that AES-NI, PCLMUL and the s2n-bignum variants use. + */ +#define MLK_CONFIG_CUSTOM_CAPABILITY_FUNC +#ifndef __ASSEMBLER__ +#include "src/sys.h" +#include "crypton_cpu.h" + +static MLK_INLINE int mlk_sys_check_capability(mlk_sys_cap cap) +{ +#if defined(__x86_64__) + if (cap == MLK_SYS_CAP_X86_64_AVX2) + return (crypton_x86_simd_features() & CRYPTON_X86_AVX2) != 0; +#elif defined(__aarch64__) + /* NEON is not optional on AArch64; the Keccak instructions are. */ + if (cap == MLK_SYS_CAP_AARCH64_NEON) + return 1; + if (cap == MLK_SYS_CAP_AARCH64_SHA3) + return (crypton_arm_features() & CRYPTON_ARM_SHA3) != 0; +#endif + (void)cap; + return 0; +} +#endif /* !__ASSEMBLER__ */ +#endif + +#endif /* CRYPTON_MLKEM_H */ diff --git a/cbits/mlkem/crypton_mlkem_asm.S b/cbits/mlkem/crypton_mlkem_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/crypton_mlkem_asm.S @@ -0,0 +1,14 @@ +/* + * The assembly half, which is level-independent: it is included once, with + * the shared directives kept, and covers all three parameter sets. See the + * comment at the top of mlkem_native_asm.S. + * + * Built only where crypton.cabal defines CRYPTON_MLKEM_NATIVE_BACKEND; on + * every other architecture this file is not in asm-sources at all. + */ +#include "crypton_mlkem.h" + +#define MLK_CONFIG_MULTILEVEL_WITH_SHARED 1 +#define MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS +#define MLK_CONFIG_PARAMETER_SET 768 +#include "mlkem_native_asm.S" diff --git a/cbits/mlkem/import.sh b/cbits/mlkem/import.sh new file mode 100644 --- /dev/null +++ b/cbits/mlkem/import.sh @@ -0,0 +1,42 @@ +#!/bin/sh +# Re-import the vendored parts of the PQ Code Package's mlkem-native. +# +# The files are kept unmodified. Everything crypton decides -- which +# parameter sets exist, what the symbols are called, that there is no +# randomised API -- is decided in cbits/mlkem/crypton_mlkem.c and in +# crypton.cabal, not by editing anything here. Run this from cbits/mlkem: +# +# ./import.sh [tag-or-commit] +# +# and commit the result together with the COMMIT line it writes, so that the +# tree always says which upstream revision it holds. +set -eu + +REPO=https://github.com/pq-code-package/mlkem-native +REV=${1:-v2.0.0} +HERE=$(cd "$(dirname "$0")" && pwd) +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT + +git clone -q "$REPO" "$TMP/u" +git -C "$TMP/u" checkout -q "$REV" + +rm -rf "$HERE/src" +cp "$TMP/u/mlkem/mlkem_native.c" "$HERE/" +cp "$TMP/u/mlkem/mlkem_native.h" "$HERE/" +cp "$TMP/u/mlkem/mlkem_native_asm.S" "$HERE/" +cp "$TMP/u/mlkem/mlkem_native_config.h" "$HERE/" +cp -R "$TMP/u/mlkem/src" "$HERE/src" +cp "$TMP/u/LICENSE" "$HERE/LICENSE" + +# The backends for architectures crypton does not build for. Every +# reference to them is behind an MLK_SYS_ guard that cannot be true on the +# two it does, so dropping them changes no build and keeps a few dozen files +# of unreachable assembly out of the release tarball. If crypton ever +# wants one of them, delete its line here rather than patching anything. +rm -rf "$HERE/src/native/riscv64" "$HERE/src/native/ppc64le" +rm -rf "$HERE/src/fips202/native/armv81m" + +git -C "$TMP/u" rev-parse HEAD > "$HERE/COMMIT" +git -C "$TMP/u" describe --tags --exact-match 2>/dev/null >> "$HERE/COMMIT" || true +echo "imported $(head -1 "$HERE/COMMIT")" diff --git a/cbits/mlkem/mlkem_native.c b/cbits/mlkem/mlkem_native.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/mlkem_native.c @@ -0,0 +1,692 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mlkem-native repository. + * Do not modify it directly. + */ + +/****************************************************************************** + * + * Single compilation unit (SCU) for fixed-level build of mlkem-native + * + * This compilation unit bundles together all source files for a build + * of mlkem-native for a fixed security level (MLKEM-512/768/1024). + * + * # API + * + * The API exposed by this file is described in mlkem_native.h. + * + * # Multi-level build + * + * If you want an SCU build of mlkem-native with support for multiple security + * levels, you need to include this file multiple times, and set + * MLK_CONFIG_MULTILEVEL_WITH_SHARED and MLK_CONFIG_MULTILEVEL_NO_SHARED + * appropriately. This is exemplified in examples/monolithic_build_multilevel + * and examples/monolithic_build_multilevel_native. + * + * # Configuration + * + * The following options from the mlkem-native configuration are relevant: + * + * - MLK_CONFIG_FIPS202_CUSTOM_HEADER + * Set this option if you use a custom FIPS202 implementation. + * + * - MLK_CONFIG_USE_NATIVE_BACKEND_ARITH + * Set this option if you want to include the native arithmetic backends + * in your build. + * + * - MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 + * Set this option if you want to include the native FIPS202 backends + * in your build. + * + * - MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS + * Set this option if you want to keep the directives defined in + * level-independent headers. This is needed for a multi-level build. + */ + +/* If parts of the mlkem-native source tree are not used, + * consider reducing this header via `unifdef`. + * + * Example: + * ```bash + * unifdef -UMLK_CONFIG_USE_NATIVE_BACKEND_ARITH mlkem_native.c + * ``` + */ + +#include "src/common.h" + +#include "src/compress.c" +#include "src/debug.c" +#include "src/indcpa.c" +#include "src/kem.c" +#include "src/poly.c" +#include "src/poly_k.c" +#include "src/sampling.c" +#include "src/verify.c" + +#if !defined(MLK_CONFIG_FIPS202_CUSTOM_HEADER) +#include "src/fips202/fips202.c" +#include "src/fips202/fips202x4.c" +#include "src/fips202/keccakf1600.c" +#endif + +#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH) +#if defined(MLK_SYS_AARCH64) +#include "src/native/aarch64/src/aarch64_zetas.c" +#include "src/native/aarch64/src/rej_uniform_table.c" +#endif +#if defined(MLK_SYS_X86_64) +#include "src/native/x86_64/src/compress_consts.c" +#include "src/native/x86_64/src/consts.c" +#include "src/native/x86_64/src/rej_uniform_table.c" +#endif +#if defined(MLK_SYS_RISCV64) +#include "src/native/riscv64/src/rv64v_debug.c" +#include "src/native/riscv64/src/rv64v_poly.c" +#endif +#if defined(MLK_SYS_PPC64LE) +#include "src/native/ppc64le/src/consts.c" +#endif +#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_ARITH */ + +#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202) +#if defined(MLK_SYS_AARCH64) +#include "src/fips202/native/aarch64/src/keccakf1600_round_constants.c" +#endif +#if defined(MLK_SYS_X86_64) +#include "src/fips202/native/x86_64/src/keccakf1600_constants.c" +#endif +#if defined(MLK_SYS_ARMV81M_MVE) +#include "src/fips202/native/armv81m/src/keccak_f1600_x4_mve.c" +#include "src/fips202/native/armv81m/src/keccakf1600_round_constants.c" +#endif +#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 */ + +/* Macro #undef's + * + * The following undefines macros from headers + * included by the source files imported above. + * + * This is to allow building and linking multiple builds + * of mlkem-native for varying parameter sets through concatenation + * of this file, as if the files had been compiled separately. + * If this is not relevant to you, you may remove the following. + */ + +/* + * Undefine macros from MLK_CONFIG_PARAMETER_SET-specific files + */ +/* mlkem/mlkem_native.h */ +#undef MLKEM1024_BYTES +#undef MLKEM1024_CIPHERTEXTBYTES +#undef MLKEM1024_PUBLICKEYBYTES +#undef MLKEM1024_SECRETKEYBYTES +#undef MLKEM1024_SYMBYTES +#undef MLKEM512_BYTES +#undef MLKEM512_CIPHERTEXTBYTES +#undef MLKEM512_PUBLICKEYBYTES +#undef MLKEM512_SECRETKEYBYTES +#undef MLKEM512_SYMBYTES +#undef MLKEM768_BYTES +#undef MLKEM768_CIPHERTEXTBYTES +#undef MLKEM768_PUBLICKEYBYTES +#undef MLKEM768_SECRETKEYBYTES +#undef MLKEM768_SYMBYTES +#undef MLKEM_BYTES +#undef MLKEM_CIPHERTEXTBYTES +#undef MLKEM_CIPHERTEXTBYTES_ +#undef MLKEM_PUBLICKEYBYTES +#undef MLKEM_PUBLICKEYBYTES_ +#undef MLKEM_SECRETKEYBYTES +#undef MLKEM_SECRETKEYBYTES_ +#undef MLKEM_SYMBYTES +#undef MLK_API_CONCAT +#undef MLK_API_CONCAT_ +#undef MLK_API_CONCAT_UNDERSCORE +#undef MLK_API_MUST_CHECK_RETURN_VALUE +#undef MLK_API_NAMESPACE +#undef MLK_API_NAMESPACE_PREFIX +#undef MLK_API_QUALIFIER +#undef MLK_ERR_FAIL +#undef MLK_ERR_INVALID_PK +#undef MLK_ERR_INVALID_SK +#undef MLK_ERR_OUT_OF_MEMORY +#undef MLK_ERR_PCT_FAIL +#undef MLK_ERR_RNG_FAIL +#undef MLK_H +#undef MLK_MAX3_ +#undef MLK_TOTAL_ALLOC_1024 +#undef MLK_TOTAL_ALLOC_1024_DECAPS +#undef MLK_TOTAL_ALLOC_1024_ENCAPS +#undef MLK_TOTAL_ALLOC_1024_KEYPAIR +#undef MLK_TOTAL_ALLOC_1024_KEYPAIR_NO_PCT +#undef MLK_TOTAL_ALLOC_1024_KEYPAIR_PCT +#undef MLK_TOTAL_ALLOC_512 +#undef MLK_TOTAL_ALLOC_512_DECAPS +#undef MLK_TOTAL_ALLOC_512_ENCAPS +#undef MLK_TOTAL_ALLOC_512_KEYPAIR +#undef MLK_TOTAL_ALLOC_512_KEYPAIR_NO_PCT +#undef MLK_TOTAL_ALLOC_512_KEYPAIR_PCT +#undef MLK_TOTAL_ALLOC_768 +#undef MLK_TOTAL_ALLOC_768_DECAPS +#undef MLK_TOTAL_ALLOC_768_ENCAPS +#undef MLK_TOTAL_ALLOC_768_KEYPAIR +#undef MLK_TOTAL_ALLOC_768_KEYPAIR_NO_PCT +#undef MLK_TOTAL_ALLOC_768_KEYPAIR_PCT +/* mlkem/src/common.h */ +#undef MLK_ADD_PARAM_SET +#undef MLK_ALLOC +#undef MLK_APPLY +#undef MLK_ASM_FN_SIZE +#undef MLK_ASM_FN_SYMBOL +#undef MLK_ASM_NAMESPACE +#undef MLK_BUILD_INTERNAL +#undef MLK_COMMON_H +#undef MLK_CONCAT +#undef MLK_CONCAT_ +#undef MLK_EMPTY_CU +#undef MLK_ERR_FAIL +#undef MLK_ERR_INVALID_PK +#undef MLK_ERR_INVALID_SK +#undef MLK_ERR_OUT_OF_MEMORY +#undef MLK_ERR_PCT_FAIL +#undef MLK_ERR_RNG_FAIL +#undef MLK_EXTERNAL_API +#undef MLK_FIPS202X4_HEADER_FILE +#undef MLK_FIPS202_HEADER_FILE +#undef MLK_FREE +#undef MLK_INTERNAL_API +#undef MLK_INTERNAL_DATA_DECLARATION +#undef MLK_INTERNAL_DATA_DEFINITION +#undef MLK_NAMESPACE +#undef MLK_NAMESPACE_K +#undef MLK_NAMESPACE_PREFIX +#undef MLK_NAMESPACE_PREFIX_K +#undef mlk_memcpy +#undef mlk_memset +/* mlkem/src/indcpa.h */ +#undef MLK_INDCPA_H +#undef mlk_gen_matrix +#undef mlk_indcpa_dec +#undef mlk_indcpa_enc +#undef mlk_indcpa_keypair_derand +/* mlkem/src/kem.h */ +#undef MLK_KEM_H +#undef mlk_kem_check_pk +#undef mlk_kem_check_sk +#undef mlk_kem_dec +#undef mlk_kem_enc +#undef mlk_kem_enc_derand +#undef mlk_kem_keypair +#undef mlk_kem_keypair_derand +/* mlkem/src/params.h */ +#undef MLKEM_DU +#undef MLKEM_DV +#undef MLKEM_ETA1 +#undef MLKEM_ETA2 +#undef MLKEM_INDCCA_CIPHERTEXTBYTES +#undef MLKEM_INDCCA_PUBLICKEYBYTES +#undef MLKEM_INDCCA_SECRETKEYBYTES +#undef MLKEM_INDCPA_BYTES +#undef MLKEM_INDCPA_MSGBYTES +#undef MLKEM_INDCPA_PUBLICKEYBYTES +#undef MLKEM_INDCPA_SECRETKEYBYTES +#undef MLKEM_K +#undef MLKEM_N +#undef MLKEM_POLYBYTES +#undef MLKEM_POLYCOMPRESSEDBYTES_D10 +#undef MLKEM_POLYCOMPRESSEDBYTES_D11 +#undef MLKEM_POLYCOMPRESSEDBYTES_D4 +#undef MLKEM_POLYCOMPRESSEDBYTES_D5 +#undef MLKEM_POLYCOMPRESSEDBYTES_DU +#undef MLKEM_POLYCOMPRESSEDBYTES_DV +#undef MLKEM_POLYVECBYTES +#undef MLKEM_POLYVECCOMPRESSEDBYTES_DU +#undef MLKEM_Q +#undef MLKEM_Q_HALF +#undef MLKEM_SSBYTES +#undef MLKEM_SYMBYTES +#undef MLKEM_UINT12_LIMIT +#undef MLK_PARAMS_H +/* mlkem/src/poly_k.h */ +#undef MLK_POLY_K_H +#undef mlk_poly_compress_du +#undef mlk_poly_compress_dv +#undef mlk_poly_decompress_du +#undef mlk_poly_decompress_dv +#undef mlk_poly_getnoise_eta1122_4x +#undef mlk_poly_getnoise_eta1_4x +#undef mlk_poly_getnoise_eta2 +#undef mlk_poly_getnoise_eta2_4x +#undef mlk_polymat +#undef mlk_polyvec +#undef mlk_polyvec_add +#undef mlk_polyvec_basemul_acc_montgomery_cached +#undef mlk_polyvec_compress_du +#undef mlk_polyvec_decompress_du +#undef mlk_polyvec_frombytes +#undef mlk_polyvec_invntt_tomont +#undef mlk_polyvec_mulcache +#undef mlk_polyvec_mulcache_compute +#undef mlk_polyvec_ntt +#undef mlk_polyvec_reduce +#undef mlk_polyvec_tobytes +#undef mlk_polyvec_tomont + +#if !defined(MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS) +/* + * Undefine macros from MLK_CONFIG_PARAMETER_SET-generic files + */ +/* mlkem/src/compress.h */ +#undef MLK_COMPRESS_H +#undef mlk_poly_compress_d10 +#undef mlk_poly_compress_d11 +#undef mlk_poly_compress_d4 +#undef mlk_poly_compress_d5 +#undef mlk_poly_decompress_d10 +#undef mlk_poly_decompress_d11 +#undef mlk_poly_decompress_d4 +#undef mlk_poly_decompress_d5 +#undef mlk_poly_frombytes +#undef mlk_poly_frommsg +#undef mlk_poly_tobytes +#undef mlk_poly_tomsg +/* mlkem/src/context.h */ +#undef MLK_CONTEXT_H +#undef MLK_CONTEXT_PARAMETERS_0 +#undef MLK_CONTEXT_PARAMETERS_1 +#undef MLK_CONTEXT_PARAMETERS_2 +#undef MLK_CONTEXT_PARAMETERS_3 +#undef MLK_CONTEXT_PARAMETERS_4 +#undef MLK_CONTEXT_UNUSED +/* mlkem/src/debug.h */ +#undef MLK_DEBUG_H +#undef mlk_assert +#undef mlk_assert_abs_bound +#undef mlk_assert_abs_bound_2d +#undef mlk_assert_bound +#undef mlk_assert_bound_2d +#undef mlk_debug_check_assert +#undef mlk_debug_check_bounds +/* mlkem/src/poly.h */ +#undef MLK_INVNTT_BOUND +#undef MLK_NTT_BOUND +#undef MLK_POLY_H +#undef mlk_poly_add +#undef mlk_poly_invntt_tomont +#undef mlk_poly_mulcache_compute +#undef mlk_poly_ntt +#undef mlk_poly_reduce +#undef mlk_poly_sub +#undef mlk_poly_tomont +/* mlkem/src/randombytes.h */ +#undef MLK_RANDOMBYTES_H +/* mlkem/src/sampling.h */ +#undef MLK_SAMPLING_H +#undef mlk_poly_cbd2 +#undef mlk_poly_cbd3 +#undef mlk_poly_rej_uniform +#undef mlk_poly_rej_uniform_x4 +/* mlkem/src/symmetric.h */ +#undef MLK_SYMMETRIC_H +#undef MLK_XOF_RATE +#undef mlk_hash_g +#undef mlk_hash_h +#undef mlk_hash_j +#undef mlk_prf_eta +#undef mlk_prf_eta1 +#undef mlk_prf_eta1_x4 +#undef mlk_prf_eta2 +#undef mlk_xof_absorb +#undef mlk_xof_ctx +#undef mlk_xof_init +#undef mlk_xof_release +#undef mlk_xof_squeezeblocks +#undef mlk_xof_x4_absorb +#undef mlk_xof_x4_ctx +#undef mlk_xof_x4_init +#undef mlk_xof_x4_release +#undef mlk_xof_x4_squeezeblocks +/* mlkem/src/sys.h */ +#undef MLK_ALIGN +#undef MLK_ALIGN_UP +#undef MLK_ALWAYS_INLINE +#undef MLK_CET_ENDBR +#undef MLK_CT_TESTING_DECLASSIFY +#undef MLK_CT_TESTING_SECRET +#undef MLK_DEFAULT_ALIGN +#undef MLK_HAVE_INLINE_ASM +#undef MLK_INLINE +#undef MLK_MUST_CHECK_RETURN_VALUE +#undef MLK_NOINLINE +#undef MLK_RESTRICT +#undef MLK_STATIC_TESTABLE +#undef MLK_SYSV_ABI +#undef MLK_SYSV_ABI_SUPPORTED +#undef MLK_SYS_AARCH64 +#undef MLK_SYS_AARCH64_EB +#undef MLK_SYS_AARCH64_NEON +#undef MLK_SYS_APPLE +#undef MLK_SYS_ARMV81M_MVE +#undef MLK_SYS_BIG_ENDIAN +#undef MLK_SYS_H +#undef MLK_SYS_LINUX +#undef MLK_SYS_LITTLE_ENDIAN +#undef MLK_SYS_PPC64LE +#undef MLK_SYS_RISCV32 +#undef MLK_SYS_RISCV64 +#undef MLK_SYS_RISCV64_RVV +#undef MLK_SYS_WINDOWS +#undef MLK_SYS_X86_64 +#undef MLK_SYS_X86_64_AVX2 +/* mlkem/src/verify.h */ +#undef MLK_USE_ASM_VALUE_BARRIER +#undef MLK_VERIFY_H +#undef mlk_ct_opt_blocker_u64 +/* mlkem/src/cbmc.h */ +#undef MLK_CBMC_H +#undef __contract__ +#undef __loop__ + +#if !defined(MLK_CONFIG_FIPS202_CUSTOM_HEADER) +/* + * Undefine macros from FIPS-202 files + */ +/* mlkem/src/fips202/fips202.h */ +#undef FIPS202_X4_DEFAULT_IMPLEMENTATION +#undef MLK_FIPS202_FIPS202_H +#undef SHA3_256_HASHBYTES +#undef SHA3_256_RATE +#undef SHA3_384_RATE +#undef SHA3_512_HASHBYTES +#undef SHA3_512_RATE +#undef SHAKE128_RATE +#undef SHAKE256_RATE +#undef mlk_sha3_256 +#undef mlk_sha3_512 +#undef mlk_shake128_absorb_once +#undef mlk_shake128_init +#undef mlk_shake128_release +#undef mlk_shake128_squeezeblocks +#undef mlk_shake256 +/* mlkem/src/fips202/fips202x4.h */ +#undef MLK_FIPS202_FIPS202X4_H +#undef mlk_shake128x4_absorb_once +#undef mlk_shake128x4_init +#undef mlk_shake128x4_release +#undef mlk_shake128x4_squeezeblocks +#undef mlk_shake256x4 +/* mlkem/src/fips202/keccakf1600.h */ +#undef MLK_FIPS202_KECCAKF1600_H +#undef MLK_KECCAK_LANES +#undef MLK_KECCAK_WAY +#undef mlk_keccakf1600_extract_bytes +#undef mlk_keccakf1600_permute +#undef mlk_keccakf1600_xor_bytes +#undef mlk_keccakf1600x4_extract_bytes +#undef mlk_keccakf1600x4_permute +#undef mlk_keccakf1600x4_xor_bytes +#endif /* !MLK_CONFIG_FIPS202_CUSTOM_HEADER */ + +#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202) +/* mlkem/src/fips202/native/api.h */ +#undef MLK_FIPS202_NATIVE_API_H +#undef MLK_NATIVE_FUNC_FALLBACK +#undef MLK_NATIVE_FUNC_SUCCESS +/* mlkem/src/fips202/native/auto.h */ +#undef MLK_FIPS202_NATIVE_AUTO_H +#if defined(MLK_SYS_AARCH64) +/* + * Undefine macros from native code (FIPS202, AArch64) + */ +/* mlkem/src/fips202/native/aarch64/auto.h */ +#undef MLK_FIPS202_NATIVE_AARCH64_AUTO_H +/* mlkem/src/fips202/native/aarch64/src/fips202_native_aarch64.h */ +#undef MLK_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H +#undef mlk_keccak_f1600_x1_scalar_aarch64_asm +#undef mlk_keccak_f1600_x1_v84a_aarch64_asm +#undef mlk_keccak_f1600_x2_v84a_aarch64_asm +#undef mlk_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm +#undef mlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm +#undef mlk_keccakf1600_round_constants +/* mlkem/src/fips202/native/aarch64/x1_scalar.h */ +#undef MLK_FIPS202_AARCH64_NEED_X1_SCALAR +#undef MLK_FIPS202_NATIVE_AARCH64_X1_SCALAR_H +#undef MLK_USE_NATIVE_FIPS202_X1 +/* mlkem/src/fips202/native/aarch64/x1_v84a.h */ +#undef MLK_FIPS202_AARCH64_NEED_X1_V84A +#undef MLK_FIPS202_NATIVE_AARCH64_X1_V84A_H +#undef MLK_USE_NATIVE_FIPS202_X1 +/* mlkem/src/fips202/native/aarch64/x2_v84a.h */ +#undef MLK_FIPS202_AARCH64_NEED_X2_V84A +#undef MLK_FIPS202_NATIVE_AARCH64_X2_V84A_H +#undef MLK_USE_NATIVE_FIPS202_X4 +/* mlkem/src/fips202/native/aarch64/x4_v8a_scalar.h */ +#undef MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID +#undef MLK_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H +#undef MLK_USE_NATIVE_FIPS202_X4 +/* mlkem/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h */ +#undef MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID +#undef MLK_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H +#undef MLK_USE_NATIVE_FIPS202_X4 +#endif /* MLK_SYS_AARCH64 */ +#if defined(MLK_SYS_X86_64) +/* + * Undefine macros from native code (FIPS202, x86_64) + */ +/* mlkem/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h */ +#undef MLK_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H +#undef MLK_FIPS202_X86_64_NEED_X4_AVX2 +#undef MLK_USE_NATIVE_FIPS202_X4 +/* mlkem/src/fips202/native/x86_64/src/fips202_native_x86_64.h */ +#undef MLK_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H +#undef mlk_keccak_f1600_x4_avx2_asm +#undef mlk_keccak_rho56 +#undef mlk_keccak_rho8 +#undef mlk_keccakf1600_round_constants +#endif /* MLK_SYS_X86_64 */ +#if defined(MLK_SYS_ARMV81M_MVE) +/* + * Undefine macros from native code (FIPS202, Armv8.1-M) + */ +/* mlkem/src/fips202/native/armv81m/mve.h */ +#undef MLK_FIPS202_ARMV81M_NEED_X4 +#undef MLK_FIPS202_NATIVE_ARMV81M +#undef MLK_FIPS202_NATIVE_ARMV81M_MVE_H +#undef MLK_USE_NATIVE_FIPS202_X4 +#undef MLK_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES +#undef MLK_USE_NATIVE_FIPS202_X4_XOR_BYTES +#undef mlk_keccak_f1600_x4_native_impl +/* mlkem/src/fips202/native/armv81m/src/fips202_native_armv81m.h */ +#undef MLK_FIPS202_NATIVE_ARMV81M_SRC_FIPS202_NATIVE_ARMV81M_H +#undef mlk_keccak_f1600_x4_mve_asm +#undef mlk_keccak_f1600_x4_state_extract_bytes_asm +#undef mlk_keccak_f1600_x4_state_xor_bytes_asm +#undef mlk_keccakf1600_round_constants +#endif /* MLK_SYS_ARMV81M_MVE */ +#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 */ +#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH) +/* mlkem/src/native/api.h */ +#undef MLK_INVNTT_BOUND +#undef MLK_NATIVE_API_H +#undef MLK_NATIVE_FUNC_FALLBACK +#undef MLK_NATIVE_FUNC_SUCCESS +#undef MLK_NTT_BOUND +/* mlkem/src/native/meta.h */ +#undef MLK_NATIVE_META_H +#if defined(MLK_SYS_AARCH64) +/* + * Undefine macros from native code (Arith, AArch64) + */ +/* mlkem/src/native/aarch64/meta.h */ +#undef MLK_ARITH_BACKEND_AARCH64 +#undef MLK_NATIVE_AARCH64_META_H +#undef MLK_USE_NATIVE_INTT +#undef MLK_USE_NATIVE_NTT +#undef MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED +#undef MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE +#undef MLK_USE_NATIVE_POLY_REDUCE +#undef MLK_USE_NATIVE_POLY_TOBYTES +#undef MLK_USE_NATIVE_POLY_TOMONT +#undef MLK_USE_NATIVE_REJ_UNIFORM +/* mlkem/src/native/aarch64/src/arith_native_aarch64.h */ +#undef MLK_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H +#undef mlk_aarch64_invntt_zetas_layer12345 +#undef mlk_aarch64_invntt_zetas_layer67 +#undef mlk_aarch64_ntt_zetas_layer12345 +#undef mlk_aarch64_ntt_zetas_layer67 +#undef mlk_aarch64_zetas_mulcache_native +#undef mlk_aarch64_zetas_mulcache_twisted_native +#undef mlk_intt_aarch64_asm +#undef mlk_ntt_aarch64_asm +#undef mlk_poly_mulcache_compute_aarch64_asm +#undef mlk_poly_reduce_aarch64_asm +#undef mlk_poly_tobytes_aarch64_asm +#undef mlk_poly_tomont_aarch64_asm +#undef mlk_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm +#undef mlk_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm +#undef mlk_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm +#undef mlk_rej_uniform_aarch64_asm +#undef mlk_rej_uniform_table +#endif /* MLK_SYS_AARCH64 */ +#if defined(MLK_SYS_X86_64) +/* + * Undefine macros from native code (Arith, X86_64) + */ +/* mlkem/src/native/x86_64/meta.h */ +#undef MLK_ARITH_BACKEND_X86_64_DEFAULT +#undef MLK_NATIVE_X86_64_META_H +#undef MLK_USE_NATIVE_INTT +#undef MLK_USE_NATIVE_NTT +#undef MLK_USE_NATIVE_NTT_CUSTOM_ORDER +#undef MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED +#undef MLK_USE_NATIVE_POLY_COMPRESS_D10 +#undef MLK_USE_NATIVE_POLY_COMPRESS_D11 +#undef MLK_USE_NATIVE_POLY_COMPRESS_D4 +#undef MLK_USE_NATIVE_POLY_COMPRESS_D5 +#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D10 +#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D11 +#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D4 +#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D5 +#undef MLK_USE_NATIVE_POLY_FROMBYTES +#undef MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE +#undef MLK_USE_NATIVE_POLY_REDUCE +#undef MLK_USE_NATIVE_POLY_TOBYTES +#undef MLK_USE_NATIVE_POLY_TOMONT +#undef MLK_USE_NATIVE_REJ_UNIFORM +/* mlkem/src/native/x86_64/src/arith_native_x86_64.h */ +#undef MLK_AVX2_REJ_UNIFORM_BUFLEN +#undef MLK_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H +#undef mlk_invntt_avx2_asm +#undef mlk_ntt_avx2_asm +#undef mlk_nttfrombytes_avx2_asm +#undef mlk_ntttobytes_avx2_asm +#undef mlk_nttunpack_avx2_asm +#undef mlk_poly_compress_d10_avx2_asm +#undef mlk_poly_compress_d11_avx2_asm +#undef mlk_poly_compress_d4_avx2_asm +#undef mlk_poly_compress_d5_avx2_asm +#undef mlk_poly_decompress_d10_avx2_asm +#undef mlk_poly_decompress_d11_avx2_asm +#undef mlk_poly_decompress_d4_avx2_asm +#undef mlk_poly_decompress_d5_avx2_asm +#undef mlk_poly_mulcache_compute_avx2_asm +#undef mlk_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm +#undef mlk_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm +#undef mlk_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm +#undef mlk_reduce_avx2_asm +#undef mlk_rej_uniform_avx2_asm +#undef mlk_rej_uniform_table +#undef mlk_tomont_avx2_asm +/* mlkem/src/native/x86_64/src/compress_consts.h */ +#undef MLK_NATIVE_X86_64_SRC_COMPRESS_CONSTS_H +#undef mlk_compress_d10_data +#undef mlk_compress_d11_data +#undef mlk_compress_d4_data +#undef mlk_compress_d5_data +#undef mlk_decompress_d10_data +#undef mlk_decompress_d11_data +#undef mlk_decompress_d4_data +#undef mlk_decompress_d5_data +/* mlkem/src/native/x86_64/src/consts.h */ +#undef MLK_AVX2_BACKEND_DATA_OFFSET_MULCACHE_TWIDDLES +#undef MLK_AVX2_BACKEND_DATA_OFFSET_REVIDXB +#undef MLK_AVX2_BACKEND_DATA_OFFSET_REVIDXD +#undef MLK_AVX2_BACKEND_DATA_OFFSET_ZETAS_EXP +#undef MLK_NATIVE_X86_64_SRC_CONSTS_H +#undef mlk_qdata +#endif /* MLK_SYS_X86_64 */ +#if defined(MLK_SYS_RISCV64) +/* + * Undefine macros from native code (Arith, RISC-V 64) + */ +/* mlkem/src/native/riscv64/meta.h */ +#undef MLK_ARITH_BACKEND_RISCV64 +#undef MLK_NATIVE_RISCV64_META_H +#undef MLK_USE_NATIVE_INTT +#undef MLK_USE_NATIVE_NTT +#undef MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED +#undef MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE +#undef MLK_USE_NATIVE_POLY_REDUCE +#undef MLK_USE_NATIVE_POLY_TOMONT +#undef MLK_USE_NATIVE_REJ_UNIFORM +/* mlkem/src/native/riscv64/src/arith_native_riscv64.h */ +#undef MLK_NATIVE_RISCV64_SRC_ARITH_NATIVE_RISCV64_H +#undef mlk_rv64v_poly_add +#undef mlk_rv64v_poly_basemul_mont_add_k2 +#undef mlk_rv64v_poly_basemul_mont_add_k3 +#undef mlk_rv64v_poly_basemul_mont_add_k4 +#undef mlk_rv64v_poly_invntt_tomont +#undef mlk_rv64v_poly_ntt +#undef mlk_rv64v_poly_reduce +#undef mlk_rv64v_poly_sub +#undef mlk_rv64v_poly_tomont +#undef mlk_rv64v_rej_uniform +/* mlkem/src/native/riscv64/src/rv64v_debug.h */ +#undef MLK_NATIVE_RISCV64_SRC_RV64V_DEBUG_H +#undef mlk_assert_abs_bound_int16m1 +#undef mlk_assert_abs_bound_int16m2 +#undef mlk_assert_bound_int16m1 +#undef mlk_assert_bound_int16m2 +#undef mlk_debug_check_bounds_int16m1 +#undef mlk_debug_check_bounds_int16m2 +#endif /* MLK_SYS_RISCV64 */ +#if defined(MLK_SYS_PPC64LE) +/* + * Undefine macros from native code (Arith, PPC64LE) + */ +/* mlkem/src/native/ppc64le/meta.h */ +#undef MLK_ARITH_BACKEND_NAME +#undef MLK_ARITH_BACKEND_PPC64LE_DEFAULT +#undef MLK_NATIVE_PPC64LE_META_H +#undef MLK_USE_NATIVE_INTT +#undef MLK_USE_NATIVE_NTT +#undef MLK_USE_NATIVE_POLY_REDUCE +#undef MLK_USE_NATIVE_POLY_TOMONT +/* mlkem/src/native/ppc64le/src/arith_native_ppc64le.h */ +#undef MLK_NATIVE_PPC64LE_SRC_ARITH_NATIVE_PPC64LE_H +#undef mlk_intt_ppc_asm +#undef mlk_ntt_ppc_asm +#undef mlk_poly_tomont_ppc_asm +#undef mlk_reduce_ppc_asm +/* mlkem/src/native/ppc64le/src/consts.h */ +#undef MLK_NATIVE_PPC64LE_SRC_CONSTS_H +#undef MLK_PPC_C20159_OFFSET +#undef MLK_PPC_NQ_OFFSET +#undef MLK_PPC_N_INV_OFFSET +#undef MLK_PPC_N_INV_TW_OFFSET +#undef MLK_PPC_Q_OFFSET +#undef MLK_PPC_TOMONT_OFFSET +#undef MLK_PPC_TOMONT_TW_OFFSET +#undef MLK_PPC_ZETA_INTT_OFFSET +#undef MLK_PPC_ZETA_INTT_TW_OFFSET +#undef MLK_PPC_ZETA_NTT_OFFSET +#undef MLK_PPC_ZETA_NTT_TW_OFFSET +#undef mlk_ppc_qdata +#endif /* MLK_SYS_PPC64LE */ +#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_ARITH */ +#endif /* !MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS */ diff --git a/cbits/mlkem/mlkem_native.h b/cbits/mlkem/mlkem_native.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/mlkem_native.h @@ -0,0 +1,464 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS140_3_IG] + * Implementation Guidance for FIPS 140-3 and the Cryptographic Module + * Validation Program + * National Institute of Standards and Technology + * https://csrc.nist.gov/projects/cryptographic-module-validation-program/fips-140-3-ig-announcements + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + */ + +#ifndef MLK_H +#define MLK_H + +/* + * Public API for mlkem-native. + * + * This header defines the public API of a single build of mlkem-native. + * + * Make sure the configuration file is in the include path + * (this is "mlkem_native_config.h" by default, or MLK_CONFIG_FILE if defined). + * + * # API conventions + * + * Conventions shared by all functions below (return values, pointer validity, + * output buffers on error) are documented in API-CONVENTIONS.md. + * + * # Multi-level builds + * + * This header specifies a build of mlkem-native for a fixed security level. + * If you need multiple security levels, leave the security level unspecified + * in the configuration file and include this header multiple times, setting + * MLK_CONFIG_PARAMETER_SET accordingly for each, and #undef'ing the MLK_H + * guard to allow multiple inclusions. + * + * In this case, the configuration file must also set + * MLK_CONFIG_MULTILEVEL_BUILD. Without it, the parameter set is not appended + * to the namespace prefix and all inclusions declare the same symbol names. + */ + +/******************************* Key sizes ************************************/ + +/* Sizes of cryptographic material, per parameter set */ +/* See mlkem/src/params.h for the arithmetic expressions giving rise to these */ +/* check-magic: off */ +#define MLKEM512_SECRETKEYBYTES 1632 +#define MLKEM512_PUBLICKEYBYTES 800 +#define MLKEM512_CIPHERTEXTBYTES 768 + +#define MLKEM768_SECRETKEYBYTES 2400 +#define MLKEM768_PUBLICKEYBYTES 1184 +#define MLKEM768_CIPHERTEXTBYTES 1088 + +#define MLKEM1024_SECRETKEYBYTES 3168 +#define MLKEM1024_PUBLICKEYBYTES 1568 +#define MLKEM1024_CIPHERTEXTBYTES 1568 +/* check-magic: on */ + +/* Size of randomness coins in bytes (level-independent) */ +#define MLKEM_SYMBYTES 32 +#define MLKEM512_SYMBYTES MLKEM_SYMBYTES +#define MLKEM768_SYMBYTES MLKEM_SYMBYTES +#define MLKEM1024_SYMBYTES MLKEM_SYMBYTES +/* Size of shared secret in bytes (level-independent) */ +#define MLKEM_BYTES 32 +#define MLKEM512_BYTES MLKEM_BYTES +#define MLKEM768_BYTES MLKEM_BYTES +#define MLKEM1024_BYTES MLKEM_BYTES + +/* Sizes of cryptographic material, as a function of LVL=512,768,1024 */ +#define MLKEM_SECRETKEYBYTES_(LVL) MLKEM##LVL##_SECRETKEYBYTES +#define MLKEM_PUBLICKEYBYTES_(LVL) MLKEM##LVL##_PUBLICKEYBYTES +#define MLKEM_CIPHERTEXTBYTES_(LVL) MLKEM##LVL##_CIPHERTEXTBYTES +#define MLKEM_SECRETKEYBYTES(LVL) MLKEM_SECRETKEYBYTES_(LVL) +#define MLKEM_PUBLICKEYBYTES(LVL) MLKEM_PUBLICKEYBYTES_(LVL) +#define MLKEM_CIPHERTEXTBYTES(LVL) MLKEM_CIPHERTEXTBYTES_(LVL) + +/****************************** Error codes ***********************************/ + +/* Generic failure condition. Currently not returned by any function; + * reserved for failures that no more specific code covers. */ +#define MLK_ERR_FAIL (-1) +/* An allocation failed. This can only happen if MLK_CONFIG_CUSTOM_ALLOC_FREE + * is defined and the provided MLK_CUSTOM_ALLOC can fail. */ +#define MLK_ERR_OUT_OF_MEMORY (-2) +/* An RNG failure occurred. Might be due to insufficient entropy or + * system misconfiguration. */ +#define MLK_ERR_RNG_FAIL (-3) +/* Public key validation failed: the @[FIPS203, Section 7.2, 'modulus check'] + * found a coefficient outside [0,q-1]. Returned by check_pk and by the + * encapsulation API. */ +#define MLK_ERR_INVALID_PK (-4) +/* Secret key validation failed: the @[FIPS203, Section 7.3, 'hash check'] + * found the embedded public key hash inconsistent. Returned by check_sk and + * by the decapsulation API. */ +#define MLK_ERR_INVALID_SK (-5) +/* The 'Pairwise Consistency Test' @[FIPS140_3_IG, p.87] and + * @[FIPS203, Section 7.1, Pairwise Consistency] failed. Only possible when + * MLK_CONFIG_KEYGEN_PCT is enabled; signals that the freshly generated key + * pair failed its encaps/decaps self-test. */ +#define MLK_ERR_PCT_FAIL (-6) + +/********************* Namespacing and Qualifiers *****************************/ + +#define MLK_API_CONCAT_(x, y) x##y +#define MLK_API_CONCAT(x, y) MLK_API_CONCAT_(x, y) +#define MLK_API_CONCAT_UNDERSCORE(x, y) MLK_API_CONCAT(MLK_API_CONCAT(x, _), y) + +/* You need to make sure the config file is in the include path. */ +#if defined(MLK_CONFIG_FILE) +#include MLK_CONFIG_FILE +#else +#include "mlkem_native_config.h" +#endif + +/* Namespace prefix for the public API symbols. For multi-level builds, the + * parameter set is appended to disambiguate the security levels. */ +#if defined(MLK_CONFIG_MULTILEVEL_BUILD) +#define MLK_API_NAMESPACE_PREFIX \ + MLK_API_CONCAT(MLK_CONFIG_NAMESPACE_PREFIX, MLK_CONFIG_PARAMETER_SET) +#else +#define MLK_API_NAMESPACE_PREFIX MLK_CONFIG_NAMESPACE_PREFIX +#endif + +#define MLK_API_NAMESPACE(sym) \ + MLK_API_CONCAT_UNDERSCORE(MLK_API_NAMESPACE_PREFIX, sym) + +#if defined(__GNUC__) || defined(__clang__) +#define MLK_API_MUST_CHECK_RETURN_VALUE __attribute__((warn_unused_result)) +#else +#define MLK_API_MUST_CHECK_RETURN_VALUE +#endif + +#if defined(MLK_CONFIG_EXTERNAL_API_QUALIFIER) +#define MLK_API_QUALIFIER MLK_CONFIG_EXTERNAL_API_QUALIFIER +#else +#define MLK_API_QUALIFIER +#endif + +/****************************** Function API **********************************/ + +#if !defined(MLK_CONFIG_CONSTANTS_ONLY) + +#include + +#ifdef __cplusplus +extern "C" +{ +#endif + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) +/** + * Generate a public/private keypair for the ML-KEM key encapsulation mechanism. + * + * @spec{Implements @[FIPS203, Algorithm 16, ML-KEM.KeyGen_Internal].} + * + * @param[out] pk Output public key, an array of + * MLKEM{512,768,1024}_PUBLICKEYBYTES bytes. + * @param[out] sk Output private key, an array of + * MLKEM{512,768,1024}_SECRETKEYBYTES bytes. + * @param[in] coins Input randomness, an array of 2*MLKEM_SYMBYTES uniformly + * random bytes. + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_PCT_FAIL MLK_CONFIG_KEYGEN_PCT enabled and PCT failed. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + * @retval MLK_ERR_RNG_FAIL MLK_CONFIG_KEYGEN_PCT enabled and random + * number generation failed within the PCT. + */ +MLK_API_QUALIFIER +MLK_API_MUST_CHECK_RETURN_VALUE +int MLK_API_NAMESPACE(keypair_derand)( + uint8_t pk[MLKEM_PUBLICKEYBYTES(MLK_CONFIG_PARAMETER_SET)], + uint8_t sk[MLKEM_SECRETKEYBYTES(MLK_CONFIG_PARAMETER_SET)], + const uint8_t coins[2 * MLKEM_SYMBYTES] +#ifdef MLK_CONFIG_CONTEXT_PARAMETER + , + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); + + +#if !defined(MLK_CONFIG_NO_RANDOMIZED_API) +/** + * Generate a public/private keypair for the ML-KEM key encapsulation mechanism. + * + * @spec{Implements @[FIPS203, Algorithm 19, ML-KEM.KeyGen].} + * + * @param[out] pk Output public key, an array of + * MLKEM{512,768,1024}_PUBLICKEYBYTES bytes. + * @param[out] sk Output private key, an array of + * MLKEM{512,768,1024}_SECRETKEYBYTES bytes. + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_PCT_FAIL MLK_CONFIG_KEYGEN_PCT enabled and PCT failed. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + * @retval MLK_ERR_RNG_FAIL Random number generation failed. + */ +MLK_API_QUALIFIER +MLK_API_MUST_CHECK_RETURN_VALUE +int MLK_API_NAMESPACE(keypair)( + uint8_t pk[MLKEM_PUBLICKEYBYTES(MLK_CONFIG_PARAMETER_SET)], + uint8_t sk[MLKEM_SECRETKEYBYTES(MLK_CONFIG_PARAMETER_SET)] +#ifdef MLK_CONFIG_CONTEXT_PARAMETER + , + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); +#endif /* !MLK_CONFIG_NO_RANDOMIZED_API */ +#endif /* !MLK_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) +/** + * Generate ciphertext and shared secret for a given public key. + * + * @spec{Implements @[FIPS203, Algorithm 17, ML-KEM.Encaps_Internal].} + * + * @param[out] ct Output ciphertext, an array of + * MLKEM{512,768,1024}_CIPHERTEXTBYTES bytes. + * @param[out] ss Output shared secret, an array of MLKEM_BYTES bytes. + * @param[in] pk Input public key, an array of + * MLKEM{512,768,1024}_PUBLICKEYBYTES bytes. + * @param[in] coins Input randomness, an array of MLKEM_SYMBYTES bytes. + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_INVALID_PK The 'modulus check' @[FIPS203, Section 7.2] + * for the public key failed. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + */ +MLK_API_QUALIFIER +MLK_API_MUST_CHECK_RETURN_VALUE +int MLK_API_NAMESPACE(enc_derand)( + uint8_t ct[MLKEM_CIPHERTEXTBYTES(MLK_CONFIG_PARAMETER_SET)], + uint8_t ss[MLKEM_BYTES], + const uint8_t pk[MLKEM_PUBLICKEYBYTES(MLK_CONFIG_PARAMETER_SET)], + const uint8_t coins[MLKEM_SYMBYTES] +#ifdef MLK_CONFIG_CONTEXT_PARAMETER + , + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); + +#if !defined(MLK_CONFIG_NO_RANDOMIZED_API) +/** + * Generate ciphertext and shared secret for a given public key. + * + * @spec{Implements @[FIPS203, Algorithm 20, ML-KEM.Encaps].} + * + * @param[out] ct Output ciphertext, an array of + * MLKEM{512,768,1024}_CIPHERTEXTBYTES bytes. + * @param[out] ss Output shared secret, an array of MLKEM_BYTES bytes. + * @param[in] pk Input public key, an array of + * MLKEM{512,768,1024}_PUBLICKEYBYTES bytes. + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_INVALID_PK The 'modulus check' @[FIPS203, Section 7.2] + * for the public key failed. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + * @retval MLK_ERR_RNG_FAIL Random number generation failed. + */ +MLK_API_QUALIFIER +MLK_API_MUST_CHECK_RETURN_VALUE +int MLK_API_NAMESPACE(enc)( + uint8_t ct[MLKEM_CIPHERTEXTBYTES(MLK_CONFIG_PARAMETER_SET)], + uint8_t ss[MLKEM_BYTES], + const uint8_t pk[MLKEM_PUBLICKEYBYTES(MLK_CONFIG_PARAMETER_SET)] +#ifdef MLK_CONFIG_CONTEXT_PARAMETER + , + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); +#endif /* !MLK_CONFIG_NO_RANDOMIZED_API */ +#endif /* !MLK_CONFIG_NO_ENCAPS_API */ + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +/** + * Generate shared secret for a given ciphertext and private key. + * + * @spec{Implements @[FIPS203, Algorithm 21, ML-KEM.Decaps].} + * + * @param[out] ss Output shared secret, an array of MLKEM_BYTES bytes. + * @param[in] ct Input ciphertext, an array of + * MLKEM{512,768,1024}_CIPHERTEXTBYTES bytes. + * @param[in] sk Input private key, an array of + * MLKEM{512,768,1024}_SECRETKEYBYTES bytes. + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_INVALID_SK The 'hash check' @[FIPS203, Section 7.3] + * for the secret key failed. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + */ +MLK_API_QUALIFIER +MLK_API_MUST_CHECK_RETURN_VALUE +int MLK_API_NAMESPACE(dec)( + uint8_t ss[MLKEM_BYTES], + const uint8_t ct[MLKEM_CIPHERTEXTBYTES(MLK_CONFIG_PARAMETER_SET)], + const uint8_t sk[MLKEM_SECRETKEYBYTES(MLK_CONFIG_PARAMETER_SET)] +#ifdef MLK_CONFIG_CONTEXT_PARAMETER + , + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + + +/** + * Implements modulus check mandated by FIPS 203, i.e., ensures that + * coefficients are in [0,q-1]. + * + * @spec{Implements @[FIPS203, Section 7.2, 'modulus check'].} + * + * @param[in] pk Input public key, an array of + * MLKEM{512,768,1024}_PUBLICKEYBYTES bytes. + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_INVALID_PK Modulus check failed. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + */ +#if !defined(MLK_CONFIG_NO_ENCAPS_API) +MLK_API_QUALIFIER +MLK_API_MUST_CHECK_RETURN_VALUE +int MLK_API_NAMESPACE(check_pk)( + const uint8_t pk[MLKEM_PUBLICKEYBYTES(MLK_CONFIG_PARAMETER_SET)] +#ifdef MLK_CONFIG_CONTEXT_PARAMETER + , + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); +#endif /* !MLK_CONFIG_NO_ENCAPS_API */ + +/** + * Implements public key hash check mandated by FIPS 203, i.e., ensures that + * sk[768𝑘+32 ∶ 768𝑘+64] = H(pk) = H(sk[384𝑘 : 768𝑘+32]). + * + * @spec{Implements @[FIPS203, Section 7.3, 'hash check'].} + * + * @param[in] sk Input private key, an array of + * MLKEM{512,768,1024}_SECRETKEYBYTES bytes. + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_INVALID_SK Public key hash check failed. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + */ +#if !defined(MLK_CONFIG_NO_DECAPS_API) +MLK_API_QUALIFIER +MLK_API_MUST_CHECK_RETURN_VALUE +int MLK_API_NAMESPACE(check_sk)( + const uint8_t sk[MLKEM_SECRETKEYBYTES(MLK_CONFIG_PARAMETER_SET)] +#ifdef MLK_CONFIG_CONTEXT_PARAMETER + , + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context +#endif +); +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +#ifdef __cplusplus +} +#endif + +#undef MLK_API_NAMESPACE_PREFIX + +#endif /* !MLK_CONFIG_CONSTANTS_ONLY */ + + +/***************************** Memory Usage **********************************/ + +/* + * By default mlkem-native performs all memory allocations on the stack. + * Alternatively, mlkem-native supports custom allocation of large structures + * through the `MLK_CONFIG_CUSTOM_ALLOC_FREE` configuration option. + * See mlkem_native_config.h for details. + * + * `MLK_TOTAL_ALLOC_{512,768,1024}_{KEYPAIR,ENCAPS,DECAPS}` indicates the + * maximum (accumulative) allocation via MLK_ALLOC for each parameter set and + * operation. Note that some stack allocation remains even when using custom + * allocators, so these values are lower than total stack usage with the default + * stack-only allocation. + * + * These constants may be used to implement custom allocations using a + * fixed-sized buffer and a simple allocator (e.g., bump allocator). + */ +/* check-magic: off */ +#define MLK_TOTAL_ALLOC_512_KEYPAIR_NO_PCT 5824 +#define MLK_TOTAL_ALLOC_512_KEYPAIR_PCT 10048 +#define MLK_TOTAL_ALLOC_512_ENCAPS 8384 +#define MLK_TOTAL_ALLOC_512_DECAPS 9152 +#define MLK_TOTAL_ALLOC_768_KEYPAIR_NO_PCT 10176 +#define MLK_TOTAL_ALLOC_768_KEYPAIR_PCT 15552 +#define MLK_TOTAL_ALLOC_768_ENCAPS 13248 +#define MLK_TOTAL_ALLOC_768_DECAPS 14336 +#define MLK_TOTAL_ALLOC_1024_KEYPAIR_NO_PCT 15552 +#define MLK_TOTAL_ALLOC_1024_KEYPAIR_PCT 22400 +#define MLK_TOTAL_ALLOC_1024_ENCAPS 19136 +#define MLK_TOTAL_ALLOC_1024_DECAPS 20704 +/* check-magic: on */ + +/* + * MLK_TOTAL_ALLOC_*_KEYPAIR adapts based on MLK_CONFIG_KEYGEN_PCT. + */ +#if defined(MLK_CONFIG_KEYGEN_PCT) +#define MLK_TOTAL_ALLOC_512_KEYPAIR MLK_TOTAL_ALLOC_512_KEYPAIR_PCT +#define MLK_TOTAL_ALLOC_768_KEYPAIR MLK_TOTAL_ALLOC_768_KEYPAIR_PCT +#define MLK_TOTAL_ALLOC_1024_KEYPAIR MLK_TOTAL_ALLOC_1024_KEYPAIR_PCT +#else +#define MLK_TOTAL_ALLOC_512_KEYPAIR MLK_TOTAL_ALLOC_512_KEYPAIR_NO_PCT +#define MLK_TOTAL_ALLOC_768_KEYPAIR MLK_TOTAL_ALLOC_768_KEYPAIR_NO_PCT +#define MLK_TOTAL_ALLOC_1024_KEYPAIR MLK_TOTAL_ALLOC_1024_KEYPAIR_NO_PCT +#endif + +#define MLK_MAX3_(a, b, c) \ + ((a) > (b) ? ((a) > (c) ? (a) : (c)) : ((b) > (c) ? (b) : (c))) + +/* + * `MLK_TOTAL_ALLOC_{512,768,1024}` is the maximum across all operations for + * each parameter set. + */ +#define MLK_TOTAL_ALLOC_512 \ + MLK_MAX3_(MLK_TOTAL_ALLOC_512_KEYPAIR, MLK_TOTAL_ALLOC_512_ENCAPS, \ + MLK_TOTAL_ALLOC_512_DECAPS) +#define MLK_TOTAL_ALLOC_768 \ + MLK_MAX3_(MLK_TOTAL_ALLOC_768_KEYPAIR, MLK_TOTAL_ALLOC_768_ENCAPS, \ + MLK_TOTAL_ALLOC_768_DECAPS) +#define MLK_TOTAL_ALLOC_1024 \ + MLK_MAX3_(MLK_TOTAL_ALLOC_1024_KEYPAIR, MLK_TOTAL_ALLOC_1024_ENCAPS, \ + MLK_TOTAL_ALLOC_1024_DECAPS) + +#endif /* !MLK_H */ diff --git a/cbits/mlkem/mlkem_native_asm.S b/cbits/mlkem/mlkem_native_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/mlkem_native_asm.S @@ -0,0 +1,716 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mlkem-native repository. + * Do not modify it directly. + */ + +/****************************************************************************** + * + * Single assembly unit for fixed-level build of mlkem-native + * + * This assembly unit bundles together all assembly files for a build + * of mlkem-native for a fixed security level (MLKEM-512/768/1024). + * + * # Multi-level build + * + * If you want an SCU build of mlkem-native with support for multiple security + * levels, you should include this file once with + * MLK_CONFIG_MULTILEVEL_WITH_SHARED set. + * + * (You could also follow the same pattern as for mlkem_native.c + * and include it for every level, setting MLK_CONFIG_MULTILEVEL_NO_SHARED + * for all but one. For builds with MLK_CONFIG_MULTILEVEL_NO_SHARED, this + * file will then be ignored.) + * + * # Configuration + * + * The following options from the mlkem-native configuration are relevant: + * + * - MLK_CONFIG_FIPS202_CUSTOM_HEADER + * Set this option if you use a custom FIPS202 implementation. + * + * - MLK_CONFIG_USE_NATIVE_BACKEND_ARITH + * Set this option if you want to include the native arithmetic backends + * in your build. + * + * - MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 + * Set this option if you want to include the native FIPS202 backends + * in your build. + * + * - MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS + * Set this option if you want to keep the directives defined in + * level-independent headers. This is needed for a multi-level build. + */ + +/* If parts of the mlkem-native source tree are not used, + * consider reducing this header via `unifdef`. + * + * Example: + * ```bash + * unifdef -UMLK_CONFIG_USE_NATIVE_BACKEND_ARITH mlkem_native_asm.S + * ``` + */ + +#include "src/common.h" + +#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH) +#if defined(MLK_SYS_AARCH64) +#include "src/native/aarch64/src/mlkem_intt_aarch64_asm.S" +#include "src/native/aarch64/src/mlkem_ntt_aarch64_asm.S" +#include "src/native/aarch64/src/mlkem_poly_mulcache_compute_aarch64_asm.S" +#include "src/native/aarch64/src/mlkem_poly_reduce_aarch64_asm.S" +#include "src/native/aarch64/src/mlkem_poly_tobytes_aarch64_asm.S" +#include "src/native/aarch64/src/mlkem_poly_tomont_aarch64_asm.S" +#include "src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm.S" +#include "src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm.S" +#include "src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm.S" +#include "src/native/aarch64/src/mlkem_rej_uniform_aarch64_asm.S" +#endif /* MLK_SYS_AARCH64 */ +#if defined(MLK_SYS_X86_64) +#include "src/native/x86_64/src/mlkem_intt_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_ntt_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_nttfrombytes_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_ntttobytes_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_nttunpack_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_poly_compress_d10_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_poly_compress_d11_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_poly_compress_d4_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_poly_compress_d5_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_poly_decompress_d10_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_poly_decompress_d11_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_poly_decompress_d4_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_poly_decompress_d5_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_poly_mulcache_compute_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_reduce_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_rej_uniform_avx2_asm.S" +#include "src/native/x86_64/src/mlkem_tomont_avx2_asm.S" +#endif /* MLK_SYS_X86_64 */ +#if defined(MLK_SYS_RISCV64) +#endif +#if defined(MLK_SYS_PPC64LE) +#include "src/native/ppc64le/src/mlkem_intt_ppc_asm.S" +#include "src/native/ppc64le/src/mlkem_ntt_ppc_asm.S" +#include "src/native/ppc64le/src/mlkem_poly_tomont_ppc_asm.S" +#include "src/native/ppc64le/src/mlkem_reduce_ppc_asm.S" +#endif /* MLK_SYS_PPC64LE */ +#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_ARITH */ + +#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202) +#if defined(MLK_SYS_AARCH64) +#include "src/fips202/native/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S" +#include "src/fips202/native/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S" +#include "src/fips202/native/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S" +#include "src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S" +#include "src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S" +#endif /* MLK_SYS_AARCH64 */ +#if defined(MLK_SYS_X86_64) +#include "src/fips202/native/x86_64/src/keccak_f1600_x4_avx2_asm.S" +#endif +#if defined(MLK_SYS_ARMV81M_MVE) +#include "src/fips202/native/armv81m/src/keccak_f1600_x4_mve.S" +#include "src/fips202/native/armv81m/src/keccak_f1600_x4_state_extract_bytes_mve.S" +#include "src/fips202/native/armv81m/src/keccak_f1600_x4_state_xor_bytes_mve.S" +#endif +#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 */ + + +/* Macro #undef's + * + * The following undefines macros from headers + * included by the source files imported above. + * + * This is to allow building and linking multiple builds + * of mlkem-native for varying parameter sets through concatenation + * of this file, as if the files had been compiled separately. + * If this is not relevant to you, you may remove the following. + * + * NOTE: This is not needed for the assembly SCU since, at present, + * there is no need to include it multiple times. + * We keep it for uniformity with mlkem_native.c only. + * + * NOTE: To avoid having to distinguish between which headers are included + * from the assembly files, we #undef the same set of directives + * as in mlkem_native.c + */ + +/* + * Undefine macros from MLK_CONFIG_PARAMETER_SET-specific files + */ +/* mlkem/mlkem_native.h */ +#undef MLKEM1024_BYTES +#undef MLKEM1024_CIPHERTEXTBYTES +#undef MLKEM1024_PUBLICKEYBYTES +#undef MLKEM1024_SECRETKEYBYTES +#undef MLKEM1024_SYMBYTES +#undef MLKEM512_BYTES +#undef MLKEM512_CIPHERTEXTBYTES +#undef MLKEM512_PUBLICKEYBYTES +#undef MLKEM512_SECRETKEYBYTES +#undef MLKEM512_SYMBYTES +#undef MLKEM768_BYTES +#undef MLKEM768_CIPHERTEXTBYTES +#undef MLKEM768_PUBLICKEYBYTES +#undef MLKEM768_SECRETKEYBYTES +#undef MLKEM768_SYMBYTES +#undef MLKEM_BYTES +#undef MLKEM_CIPHERTEXTBYTES +#undef MLKEM_CIPHERTEXTBYTES_ +#undef MLKEM_PUBLICKEYBYTES +#undef MLKEM_PUBLICKEYBYTES_ +#undef MLKEM_SECRETKEYBYTES +#undef MLKEM_SECRETKEYBYTES_ +#undef MLKEM_SYMBYTES +#undef MLK_API_CONCAT +#undef MLK_API_CONCAT_ +#undef MLK_API_CONCAT_UNDERSCORE +#undef MLK_API_MUST_CHECK_RETURN_VALUE +#undef MLK_API_NAMESPACE +#undef MLK_API_NAMESPACE_PREFIX +#undef MLK_API_QUALIFIER +#undef MLK_ERR_FAIL +#undef MLK_ERR_INVALID_PK +#undef MLK_ERR_INVALID_SK +#undef MLK_ERR_OUT_OF_MEMORY +#undef MLK_ERR_PCT_FAIL +#undef MLK_ERR_RNG_FAIL +#undef MLK_H +#undef MLK_MAX3_ +#undef MLK_TOTAL_ALLOC_1024 +#undef MLK_TOTAL_ALLOC_1024_DECAPS +#undef MLK_TOTAL_ALLOC_1024_ENCAPS +#undef MLK_TOTAL_ALLOC_1024_KEYPAIR +#undef MLK_TOTAL_ALLOC_1024_KEYPAIR_NO_PCT +#undef MLK_TOTAL_ALLOC_1024_KEYPAIR_PCT +#undef MLK_TOTAL_ALLOC_512 +#undef MLK_TOTAL_ALLOC_512_DECAPS +#undef MLK_TOTAL_ALLOC_512_ENCAPS +#undef MLK_TOTAL_ALLOC_512_KEYPAIR +#undef MLK_TOTAL_ALLOC_512_KEYPAIR_NO_PCT +#undef MLK_TOTAL_ALLOC_512_KEYPAIR_PCT +#undef MLK_TOTAL_ALLOC_768 +#undef MLK_TOTAL_ALLOC_768_DECAPS +#undef MLK_TOTAL_ALLOC_768_ENCAPS +#undef MLK_TOTAL_ALLOC_768_KEYPAIR +#undef MLK_TOTAL_ALLOC_768_KEYPAIR_NO_PCT +#undef MLK_TOTAL_ALLOC_768_KEYPAIR_PCT +/* mlkem/src/common.h */ +#undef MLK_ADD_PARAM_SET +#undef MLK_ALLOC +#undef MLK_APPLY +#undef MLK_ASM_FN_SIZE +#undef MLK_ASM_FN_SYMBOL +#undef MLK_ASM_NAMESPACE +#undef MLK_BUILD_INTERNAL +#undef MLK_COMMON_H +#undef MLK_CONCAT +#undef MLK_CONCAT_ +#undef MLK_EMPTY_CU +#undef MLK_ERR_FAIL +#undef MLK_ERR_INVALID_PK +#undef MLK_ERR_INVALID_SK +#undef MLK_ERR_OUT_OF_MEMORY +#undef MLK_ERR_PCT_FAIL +#undef MLK_ERR_RNG_FAIL +#undef MLK_EXTERNAL_API +#undef MLK_FIPS202X4_HEADER_FILE +#undef MLK_FIPS202_HEADER_FILE +#undef MLK_FREE +#undef MLK_INTERNAL_API +#undef MLK_INTERNAL_DATA_DECLARATION +#undef MLK_INTERNAL_DATA_DEFINITION +#undef MLK_NAMESPACE +#undef MLK_NAMESPACE_K +#undef MLK_NAMESPACE_PREFIX +#undef MLK_NAMESPACE_PREFIX_K +#undef mlk_memcpy +#undef mlk_memset +/* mlkem/src/indcpa.h */ +#undef MLK_INDCPA_H +#undef mlk_gen_matrix +#undef mlk_indcpa_dec +#undef mlk_indcpa_enc +#undef mlk_indcpa_keypair_derand +/* mlkem/src/kem.h */ +#undef MLK_KEM_H +#undef mlk_kem_check_pk +#undef mlk_kem_check_sk +#undef mlk_kem_dec +#undef mlk_kem_enc +#undef mlk_kem_enc_derand +#undef mlk_kem_keypair +#undef mlk_kem_keypair_derand +/* mlkem/src/params.h */ +#undef MLKEM_DU +#undef MLKEM_DV +#undef MLKEM_ETA1 +#undef MLKEM_ETA2 +#undef MLKEM_INDCCA_CIPHERTEXTBYTES +#undef MLKEM_INDCCA_PUBLICKEYBYTES +#undef MLKEM_INDCCA_SECRETKEYBYTES +#undef MLKEM_INDCPA_BYTES +#undef MLKEM_INDCPA_MSGBYTES +#undef MLKEM_INDCPA_PUBLICKEYBYTES +#undef MLKEM_INDCPA_SECRETKEYBYTES +#undef MLKEM_K +#undef MLKEM_N +#undef MLKEM_POLYBYTES +#undef MLKEM_POLYCOMPRESSEDBYTES_D10 +#undef MLKEM_POLYCOMPRESSEDBYTES_D11 +#undef MLKEM_POLYCOMPRESSEDBYTES_D4 +#undef MLKEM_POLYCOMPRESSEDBYTES_D5 +#undef MLKEM_POLYCOMPRESSEDBYTES_DU +#undef MLKEM_POLYCOMPRESSEDBYTES_DV +#undef MLKEM_POLYVECBYTES +#undef MLKEM_POLYVECCOMPRESSEDBYTES_DU +#undef MLKEM_Q +#undef MLKEM_Q_HALF +#undef MLKEM_SSBYTES +#undef MLKEM_SYMBYTES +#undef MLKEM_UINT12_LIMIT +#undef MLK_PARAMS_H +/* mlkem/src/poly_k.h */ +#undef MLK_POLY_K_H +#undef mlk_poly_compress_du +#undef mlk_poly_compress_dv +#undef mlk_poly_decompress_du +#undef mlk_poly_decompress_dv +#undef mlk_poly_getnoise_eta1122_4x +#undef mlk_poly_getnoise_eta1_4x +#undef mlk_poly_getnoise_eta2 +#undef mlk_poly_getnoise_eta2_4x +#undef mlk_polymat +#undef mlk_polyvec +#undef mlk_polyvec_add +#undef mlk_polyvec_basemul_acc_montgomery_cached +#undef mlk_polyvec_compress_du +#undef mlk_polyvec_decompress_du +#undef mlk_polyvec_frombytes +#undef mlk_polyvec_invntt_tomont +#undef mlk_polyvec_mulcache +#undef mlk_polyvec_mulcache_compute +#undef mlk_polyvec_ntt +#undef mlk_polyvec_reduce +#undef mlk_polyvec_tobytes +#undef mlk_polyvec_tomont + +#if !defined(MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS) +/* + * Undefine macros from MLK_CONFIG_PARAMETER_SET-generic files + */ +/* mlkem/src/compress.h */ +#undef MLK_COMPRESS_H +#undef mlk_poly_compress_d10 +#undef mlk_poly_compress_d11 +#undef mlk_poly_compress_d4 +#undef mlk_poly_compress_d5 +#undef mlk_poly_decompress_d10 +#undef mlk_poly_decompress_d11 +#undef mlk_poly_decompress_d4 +#undef mlk_poly_decompress_d5 +#undef mlk_poly_frombytes +#undef mlk_poly_frommsg +#undef mlk_poly_tobytes +#undef mlk_poly_tomsg +/* mlkem/src/context.h */ +#undef MLK_CONTEXT_H +#undef MLK_CONTEXT_PARAMETERS_0 +#undef MLK_CONTEXT_PARAMETERS_1 +#undef MLK_CONTEXT_PARAMETERS_2 +#undef MLK_CONTEXT_PARAMETERS_3 +#undef MLK_CONTEXT_PARAMETERS_4 +#undef MLK_CONTEXT_UNUSED +/* mlkem/src/debug.h */ +#undef MLK_DEBUG_H +#undef mlk_assert +#undef mlk_assert_abs_bound +#undef mlk_assert_abs_bound_2d +#undef mlk_assert_bound +#undef mlk_assert_bound_2d +#undef mlk_debug_check_assert +#undef mlk_debug_check_bounds +/* mlkem/src/poly.h */ +#undef MLK_INVNTT_BOUND +#undef MLK_NTT_BOUND +#undef MLK_POLY_H +#undef mlk_poly_add +#undef mlk_poly_invntt_tomont +#undef mlk_poly_mulcache_compute +#undef mlk_poly_ntt +#undef mlk_poly_reduce +#undef mlk_poly_sub +#undef mlk_poly_tomont +/* mlkem/src/randombytes.h */ +#undef MLK_RANDOMBYTES_H +/* mlkem/src/sampling.h */ +#undef MLK_SAMPLING_H +#undef mlk_poly_cbd2 +#undef mlk_poly_cbd3 +#undef mlk_poly_rej_uniform +#undef mlk_poly_rej_uniform_x4 +/* mlkem/src/symmetric.h */ +#undef MLK_SYMMETRIC_H +#undef MLK_XOF_RATE +#undef mlk_hash_g +#undef mlk_hash_h +#undef mlk_hash_j +#undef mlk_prf_eta +#undef mlk_prf_eta1 +#undef mlk_prf_eta1_x4 +#undef mlk_prf_eta2 +#undef mlk_xof_absorb +#undef mlk_xof_ctx +#undef mlk_xof_init +#undef mlk_xof_release +#undef mlk_xof_squeezeblocks +#undef mlk_xof_x4_absorb +#undef mlk_xof_x4_ctx +#undef mlk_xof_x4_init +#undef mlk_xof_x4_release +#undef mlk_xof_x4_squeezeblocks +/* mlkem/src/sys.h */ +#undef MLK_ALIGN +#undef MLK_ALIGN_UP +#undef MLK_ALWAYS_INLINE +#undef MLK_CET_ENDBR +#undef MLK_CT_TESTING_DECLASSIFY +#undef MLK_CT_TESTING_SECRET +#undef MLK_DEFAULT_ALIGN +#undef MLK_HAVE_INLINE_ASM +#undef MLK_INLINE +#undef MLK_MUST_CHECK_RETURN_VALUE +#undef MLK_NOINLINE +#undef MLK_RESTRICT +#undef MLK_STATIC_TESTABLE +#undef MLK_SYSV_ABI +#undef MLK_SYSV_ABI_SUPPORTED +#undef MLK_SYS_AARCH64 +#undef MLK_SYS_AARCH64_EB +#undef MLK_SYS_AARCH64_NEON +#undef MLK_SYS_APPLE +#undef MLK_SYS_ARMV81M_MVE +#undef MLK_SYS_BIG_ENDIAN +#undef MLK_SYS_H +#undef MLK_SYS_LINUX +#undef MLK_SYS_LITTLE_ENDIAN +#undef MLK_SYS_PPC64LE +#undef MLK_SYS_RISCV32 +#undef MLK_SYS_RISCV64 +#undef MLK_SYS_RISCV64_RVV +#undef MLK_SYS_WINDOWS +#undef MLK_SYS_X86_64 +#undef MLK_SYS_X86_64_AVX2 +/* mlkem/src/verify.h */ +#undef MLK_USE_ASM_VALUE_BARRIER +#undef MLK_VERIFY_H +#undef mlk_ct_opt_blocker_u64 +/* mlkem/src/cbmc.h */ +#undef MLK_CBMC_H +#undef __contract__ +#undef __loop__ + +#if !defined(MLK_CONFIG_FIPS202_CUSTOM_HEADER) +/* + * Undefine macros from FIPS-202 files + */ +/* mlkem/src/fips202/fips202.h */ +#undef FIPS202_X4_DEFAULT_IMPLEMENTATION +#undef MLK_FIPS202_FIPS202_H +#undef SHA3_256_HASHBYTES +#undef SHA3_256_RATE +#undef SHA3_384_RATE +#undef SHA3_512_HASHBYTES +#undef SHA3_512_RATE +#undef SHAKE128_RATE +#undef SHAKE256_RATE +#undef mlk_sha3_256 +#undef mlk_sha3_512 +#undef mlk_shake128_absorb_once +#undef mlk_shake128_init +#undef mlk_shake128_release +#undef mlk_shake128_squeezeblocks +#undef mlk_shake256 +/* mlkem/src/fips202/fips202x4.h */ +#undef MLK_FIPS202_FIPS202X4_H +#undef mlk_shake128x4_absorb_once +#undef mlk_shake128x4_init +#undef mlk_shake128x4_release +#undef mlk_shake128x4_squeezeblocks +#undef mlk_shake256x4 +/* mlkem/src/fips202/keccakf1600.h */ +#undef MLK_FIPS202_KECCAKF1600_H +#undef MLK_KECCAK_LANES +#undef MLK_KECCAK_WAY +#undef mlk_keccakf1600_extract_bytes +#undef mlk_keccakf1600_permute +#undef mlk_keccakf1600_xor_bytes +#undef mlk_keccakf1600x4_extract_bytes +#undef mlk_keccakf1600x4_permute +#undef mlk_keccakf1600x4_xor_bytes +#endif /* !MLK_CONFIG_FIPS202_CUSTOM_HEADER */ + +#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202) +/* mlkem/src/fips202/native/api.h */ +#undef MLK_FIPS202_NATIVE_API_H +#undef MLK_NATIVE_FUNC_FALLBACK +#undef MLK_NATIVE_FUNC_SUCCESS +/* mlkem/src/fips202/native/auto.h */ +#undef MLK_FIPS202_NATIVE_AUTO_H +#if defined(MLK_SYS_AARCH64) +/* + * Undefine macros from native code (FIPS202, AArch64) + */ +/* mlkem/src/fips202/native/aarch64/auto.h */ +#undef MLK_FIPS202_NATIVE_AARCH64_AUTO_H +/* mlkem/src/fips202/native/aarch64/src/fips202_native_aarch64.h */ +#undef MLK_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H +#undef mlk_keccak_f1600_x1_scalar_aarch64_asm +#undef mlk_keccak_f1600_x1_v84a_aarch64_asm +#undef mlk_keccak_f1600_x2_v84a_aarch64_asm +#undef mlk_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm +#undef mlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm +#undef mlk_keccakf1600_round_constants +/* mlkem/src/fips202/native/aarch64/x1_scalar.h */ +#undef MLK_FIPS202_AARCH64_NEED_X1_SCALAR +#undef MLK_FIPS202_NATIVE_AARCH64_X1_SCALAR_H +#undef MLK_USE_NATIVE_FIPS202_X1 +/* mlkem/src/fips202/native/aarch64/x1_v84a.h */ +#undef MLK_FIPS202_AARCH64_NEED_X1_V84A +#undef MLK_FIPS202_NATIVE_AARCH64_X1_V84A_H +#undef MLK_USE_NATIVE_FIPS202_X1 +/* mlkem/src/fips202/native/aarch64/x2_v84a.h */ +#undef MLK_FIPS202_AARCH64_NEED_X2_V84A +#undef MLK_FIPS202_NATIVE_AARCH64_X2_V84A_H +#undef MLK_USE_NATIVE_FIPS202_X4 +/* mlkem/src/fips202/native/aarch64/x4_v8a_scalar.h */ +#undef MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID +#undef MLK_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H +#undef MLK_USE_NATIVE_FIPS202_X4 +/* mlkem/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h */ +#undef MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID +#undef MLK_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H +#undef MLK_USE_NATIVE_FIPS202_X4 +#endif /* MLK_SYS_AARCH64 */ +#if defined(MLK_SYS_X86_64) +/* + * Undefine macros from native code (FIPS202, x86_64) + */ +/* mlkem/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h */ +#undef MLK_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H +#undef MLK_FIPS202_X86_64_NEED_X4_AVX2 +#undef MLK_USE_NATIVE_FIPS202_X4 +/* mlkem/src/fips202/native/x86_64/src/fips202_native_x86_64.h */ +#undef MLK_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H +#undef mlk_keccak_f1600_x4_avx2_asm +#undef mlk_keccak_rho56 +#undef mlk_keccak_rho8 +#undef mlk_keccakf1600_round_constants +#endif /* MLK_SYS_X86_64 */ +#if defined(MLK_SYS_ARMV81M_MVE) +/* + * Undefine macros from native code (FIPS202, Armv8.1-M) + */ +/* mlkem/src/fips202/native/armv81m/mve.h */ +#undef MLK_FIPS202_ARMV81M_NEED_X4 +#undef MLK_FIPS202_NATIVE_ARMV81M +#undef MLK_FIPS202_NATIVE_ARMV81M_MVE_H +#undef MLK_USE_NATIVE_FIPS202_X4 +#undef MLK_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES +#undef MLK_USE_NATIVE_FIPS202_X4_XOR_BYTES +#undef mlk_keccak_f1600_x4_native_impl +/* mlkem/src/fips202/native/armv81m/src/fips202_native_armv81m.h */ +#undef MLK_FIPS202_NATIVE_ARMV81M_SRC_FIPS202_NATIVE_ARMV81M_H +#undef mlk_keccak_f1600_x4_mve_asm +#undef mlk_keccak_f1600_x4_state_extract_bytes_asm +#undef mlk_keccak_f1600_x4_state_xor_bytes_asm +#undef mlk_keccakf1600_round_constants +#endif /* MLK_SYS_ARMV81M_MVE */ +#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 */ +#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH) +/* mlkem/src/native/api.h */ +#undef MLK_INVNTT_BOUND +#undef MLK_NATIVE_API_H +#undef MLK_NATIVE_FUNC_FALLBACK +#undef MLK_NATIVE_FUNC_SUCCESS +#undef MLK_NTT_BOUND +/* mlkem/src/native/meta.h */ +#undef MLK_NATIVE_META_H +#if defined(MLK_SYS_AARCH64) +/* + * Undefine macros from native code (Arith, AArch64) + */ +/* mlkem/src/native/aarch64/meta.h */ +#undef MLK_ARITH_BACKEND_AARCH64 +#undef MLK_NATIVE_AARCH64_META_H +#undef MLK_USE_NATIVE_INTT +#undef MLK_USE_NATIVE_NTT +#undef MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED +#undef MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE +#undef MLK_USE_NATIVE_POLY_REDUCE +#undef MLK_USE_NATIVE_POLY_TOBYTES +#undef MLK_USE_NATIVE_POLY_TOMONT +#undef MLK_USE_NATIVE_REJ_UNIFORM +/* mlkem/src/native/aarch64/src/arith_native_aarch64.h */ +#undef MLK_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H +#undef mlk_aarch64_invntt_zetas_layer12345 +#undef mlk_aarch64_invntt_zetas_layer67 +#undef mlk_aarch64_ntt_zetas_layer12345 +#undef mlk_aarch64_ntt_zetas_layer67 +#undef mlk_aarch64_zetas_mulcache_native +#undef mlk_aarch64_zetas_mulcache_twisted_native +#undef mlk_intt_aarch64_asm +#undef mlk_ntt_aarch64_asm +#undef mlk_poly_mulcache_compute_aarch64_asm +#undef mlk_poly_reduce_aarch64_asm +#undef mlk_poly_tobytes_aarch64_asm +#undef mlk_poly_tomont_aarch64_asm +#undef mlk_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm +#undef mlk_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm +#undef mlk_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm +#undef mlk_rej_uniform_aarch64_asm +#undef mlk_rej_uniform_table +#endif /* MLK_SYS_AARCH64 */ +#if defined(MLK_SYS_X86_64) +/* + * Undefine macros from native code (Arith, X86_64) + */ +/* mlkem/src/native/x86_64/meta.h */ +#undef MLK_ARITH_BACKEND_X86_64_DEFAULT +#undef MLK_NATIVE_X86_64_META_H +#undef MLK_USE_NATIVE_INTT +#undef MLK_USE_NATIVE_NTT +#undef MLK_USE_NATIVE_NTT_CUSTOM_ORDER +#undef MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED +#undef MLK_USE_NATIVE_POLY_COMPRESS_D10 +#undef MLK_USE_NATIVE_POLY_COMPRESS_D11 +#undef MLK_USE_NATIVE_POLY_COMPRESS_D4 +#undef MLK_USE_NATIVE_POLY_COMPRESS_D5 +#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D10 +#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D11 +#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D4 +#undef MLK_USE_NATIVE_POLY_DECOMPRESS_D5 +#undef MLK_USE_NATIVE_POLY_FROMBYTES +#undef MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE +#undef MLK_USE_NATIVE_POLY_REDUCE +#undef MLK_USE_NATIVE_POLY_TOBYTES +#undef MLK_USE_NATIVE_POLY_TOMONT +#undef MLK_USE_NATIVE_REJ_UNIFORM +/* mlkem/src/native/x86_64/src/arith_native_x86_64.h */ +#undef MLK_AVX2_REJ_UNIFORM_BUFLEN +#undef MLK_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H +#undef mlk_invntt_avx2_asm +#undef mlk_ntt_avx2_asm +#undef mlk_nttfrombytes_avx2_asm +#undef mlk_ntttobytes_avx2_asm +#undef mlk_nttunpack_avx2_asm +#undef mlk_poly_compress_d10_avx2_asm +#undef mlk_poly_compress_d11_avx2_asm +#undef mlk_poly_compress_d4_avx2_asm +#undef mlk_poly_compress_d5_avx2_asm +#undef mlk_poly_decompress_d10_avx2_asm +#undef mlk_poly_decompress_d11_avx2_asm +#undef mlk_poly_decompress_d4_avx2_asm +#undef mlk_poly_decompress_d5_avx2_asm +#undef mlk_poly_mulcache_compute_avx2_asm +#undef mlk_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm +#undef mlk_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm +#undef mlk_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm +#undef mlk_reduce_avx2_asm +#undef mlk_rej_uniform_avx2_asm +#undef mlk_rej_uniform_table +#undef mlk_tomont_avx2_asm +/* mlkem/src/native/x86_64/src/compress_consts.h */ +#undef MLK_NATIVE_X86_64_SRC_COMPRESS_CONSTS_H +#undef mlk_compress_d10_data +#undef mlk_compress_d11_data +#undef mlk_compress_d4_data +#undef mlk_compress_d5_data +#undef mlk_decompress_d10_data +#undef mlk_decompress_d11_data +#undef mlk_decompress_d4_data +#undef mlk_decompress_d5_data +/* mlkem/src/native/x86_64/src/consts.h */ +#undef MLK_AVX2_BACKEND_DATA_OFFSET_MULCACHE_TWIDDLES +#undef MLK_AVX2_BACKEND_DATA_OFFSET_REVIDXB +#undef MLK_AVX2_BACKEND_DATA_OFFSET_REVIDXD +#undef MLK_AVX2_BACKEND_DATA_OFFSET_ZETAS_EXP +#undef MLK_NATIVE_X86_64_SRC_CONSTS_H +#undef mlk_qdata +#endif /* MLK_SYS_X86_64 */ +#if defined(MLK_SYS_RISCV64) +/* + * Undefine macros from native code (Arith, RISC-V 64) + */ +/* mlkem/src/native/riscv64/meta.h */ +#undef MLK_ARITH_BACKEND_RISCV64 +#undef MLK_NATIVE_RISCV64_META_H +#undef MLK_USE_NATIVE_INTT +#undef MLK_USE_NATIVE_NTT +#undef MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED +#undef MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE +#undef MLK_USE_NATIVE_POLY_REDUCE +#undef MLK_USE_NATIVE_POLY_TOMONT +#undef MLK_USE_NATIVE_REJ_UNIFORM +/* mlkem/src/native/riscv64/src/arith_native_riscv64.h */ +#undef MLK_NATIVE_RISCV64_SRC_ARITH_NATIVE_RISCV64_H +#undef mlk_rv64v_poly_add +#undef mlk_rv64v_poly_basemul_mont_add_k2 +#undef mlk_rv64v_poly_basemul_mont_add_k3 +#undef mlk_rv64v_poly_basemul_mont_add_k4 +#undef mlk_rv64v_poly_invntt_tomont +#undef mlk_rv64v_poly_ntt +#undef mlk_rv64v_poly_reduce +#undef mlk_rv64v_poly_sub +#undef mlk_rv64v_poly_tomont +#undef mlk_rv64v_rej_uniform +/* mlkem/src/native/riscv64/src/rv64v_debug.h */ +#undef MLK_NATIVE_RISCV64_SRC_RV64V_DEBUG_H +#undef mlk_assert_abs_bound_int16m1 +#undef mlk_assert_abs_bound_int16m2 +#undef mlk_assert_bound_int16m1 +#undef mlk_assert_bound_int16m2 +#undef mlk_debug_check_bounds_int16m1 +#undef mlk_debug_check_bounds_int16m2 +#endif /* MLK_SYS_RISCV64 */ +#if defined(MLK_SYS_PPC64LE) +/* + * Undefine macros from native code (Arith, PPC64LE) + */ +/* mlkem/src/native/ppc64le/meta.h */ +#undef MLK_ARITH_BACKEND_NAME +#undef MLK_ARITH_BACKEND_PPC64LE_DEFAULT +#undef MLK_NATIVE_PPC64LE_META_H +#undef MLK_USE_NATIVE_INTT +#undef MLK_USE_NATIVE_NTT +#undef MLK_USE_NATIVE_POLY_REDUCE +#undef MLK_USE_NATIVE_POLY_TOMONT +/* mlkem/src/native/ppc64le/src/arith_native_ppc64le.h */ +#undef MLK_NATIVE_PPC64LE_SRC_ARITH_NATIVE_PPC64LE_H +#undef mlk_intt_ppc_asm +#undef mlk_ntt_ppc_asm +#undef mlk_poly_tomont_ppc_asm +#undef mlk_reduce_ppc_asm +/* mlkem/src/native/ppc64le/src/consts.h */ +#undef MLK_NATIVE_PPC64LE_SRC_CONSTS_H +#undef MLK_PPC_C20159_OFFSET +#undef MLK_PPC_NQ_OFFSET +#undef MLK_PPC_N_INV_OFFSET +#undef MLK_PPC_N_INV_TW_OFFSET +#undef MLK_PPC_Q_OFFSET +#undef MLK_PPC_TOMONT_OFFSET +#undef MLK_PPC_TOMONT_TW_OFFSET +#undef MLK_PPC_ZETA_INTT_OFFSET +#undef MLK_PPC_ZETA_INTT_TW_OFFSET +#undef MLK_PPC_ZETA_NTT_OFFSET +#undef MLK_PPC_ZETA_NTT_TW_OFFSET +#undef mlk_ppc_qdata +#endif /* MLK_SYS_PPC64LE */ +#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_ARITH */ +#endif /* !MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS */ diff --git a/cbits/mlkem/mlkem_native_config.h b/cbits/mlkem/mlkem_native_config.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/mlkem_native_config.h @@ -0,0 +1,683 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS140_3_IG] + * Implementation Guidance for FIPS 140-3 and the Cryptographic Module + * Validation Program + * National Institute of Standards and Technology + * https://csrc.nist.gov/projects/cryptographic-module-validation-program/fips-140-3-ig-announcements + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + */ + +#ifndef MLK_CONFIG_H +#define MLK_CONFIG_H + +/** + * Specifies the parameter set for ML-KEM: + * - MLK_CONFIG_PARAMETER_SET=512 corresponds to ML-KEM-512 + * - MLK_CONFIG_PARAMETER_SET=768 corresponds to ML-KEM-768 + * - MLK_CONFIG_PARAMETER_SET=1024 corresponds to ML-KEM-1024 + * + * If you want to support multiple parameter sets, build the library multiple + * times and set MLK_CONFIG_MULTILEVEL_BUILD. See MLK_CONFIG_MULTILEVEL_BUILD + * for how to do this while minimizing code duplication. + * + * This can also be set using CFLAGS. + */ +#ifndef MLK_CONFIG_PARAMETER_SET +#define MLK_CONFIG_PARAMETER_SET \ + 768 /* Change this for different security strengths */ +#endif + +/** + * MLK_CONFIG_FILE + * + * If defined, this is a header that will be included instead of the default + * configuration file mlkem/mlkem_native_config.h. + * + * When you need to build mlkem-native in multiple configurations, using + * varying MLK_CONFIG_FILE can be more convenient than configuring everything + * through CFLAGS. + * + * To use, MLK_CONFIG_FILE _must_ be defined prior to the inclusion of any + * mlkem-native headers. For example, it can be set by passing + * `-DMLK_CONFIG_FILE="..."` on the command line. + */ +/* #define MLK_CONFIG_FILE "mlkem_native_config.h" */ + +/** + * The prefix to use to namespace global symbols from mlkem/. + * + * In a multi-level build, level-dependent symbols will additionally be + * prefixed with the parameter set (512/768/1024). + * + * This can also be set using CFLAGS. + */ +#if !defined(MLK_CONFIG_NAMESPACE_PREFIX) +#define MLK_CONFIG_NAMESPACE_PREFIX MLK_DEFAULT_NAMESPACE_PREFIX +#endif + +/** + * MLK_CONFIG_MULTILEVEL_BUILD + * + * Set this if the build is part of a multi-level build supporting multiple + * parameter sets. + * + * If you need only a single parameter set, keep this unset. + * + * To build mlkem-native with support for all parameter sets, build it three + * times -- once per parameter set -- and set the option + * MLK_CONFIG_MULTILEVEL_WITH_SHARED for exactly one of them, and + * MLK_CONFIG_MULTILEVEL_NO_SHARED for the others. + * MLK_CONFIG_MULTILEVEL_BUILD should be set for all of them. + * + * See examples/multilevel_build for an example. + * + * This can also be set using CFLAGS. + */ +/* #define MLK_CONFIG_MULTILEVEL_BUILD */ + +/** + * MLK_CONFIG_EXTERNAL_API_QUALIFIER + * + * If set, this option provides an additional function qualifier to be added + * to declarations of mlkem-native's public API. + * + * The primary use case for this option are single-CU builds where the public + * API exposed by mlkem-native is wrapped by another API in the consuming + * application. In this case, even mlkem-native's public API can be marked + * `static`. + */ +/* #define MLK_CONFIG_EXTERNAL_API_QUALIFIER */ + +/** + * MLK_CONFIG_NO_KEYPAIR_API + * + * By default, mlkem-native includes support for generating key pairs. + * If you don't need this, set MLK_CONFIG_NO_KEYPAIR_API to exclude + * keypair and keypair_derand, and all internal + * APIs only needed by those functions. + */ +/* #define MLK_CONFIG_NO_KEYPAIR_API */ + +/** + * MLK_CONFIG_NO_ENCAPS_API + * + * By default, mlkem-native includes support for encapsulation. If you + * don't need this, set MLK_CONFIG_NO_ENCAPS_API to exclude + * enc, enc_derand, check_pk, and + * all internal APIs only needed by those functions. + * + * @note Setting this option is incompatible with MLK_CONFIG_KEYGEN_PCT + * as the current PCT implementation requires enc(). + */ +/* #define MLK_CONFIG_NO_ENCAPS_API */ + +/** + * MLK_CONFIG_NO_DECAPS_API + * + * By default, mlkem-native includes support for decapsulation. If you + * don't need this, set MLK_CONFIG_NO_DECAPS_API to exclude + * dec, check_sk, and all internal APIs only + * needed by those functions. + * + * @note Setting this option is incompatible with MLK_CONFIG_KEYGEN_PCT + * as the current PCT implementation requires dec(). + */ +/* #define MLK_CONFIG_NO_DECAPS_API */ + +/** + * MLK_CONFIG_NO_RANDOMIZED_API + * + * If this option is set, mlkem-native will be built without the randomized + * API functions (keypair and enc). This allows users + * to build mlkem-native without providing a randombytes() implementation + * if they only need the deterministic API (keypair_derand, + * enc_derand, dec). + * + * @note This option is incompatible with MLK_CONFIG_KEYGEN_PCT as the + * current PCT implementation requires enc(). + */ +/* #define MLK_CONFIG_NO_RANDOMIZED_API */ + +/** + * MLK_CONFIG_CONSTANTS_ONLY + * + * If you only need the size constants (MLKEM_PUBLICKEYBYTES, etc.) but no + * function declarations, set MLK_CONFIG_CONSTANTS_ONLY. + * + * This only affects the public header mlkem_native.h, not the + * implementation. + */ +/* #define MLK_CONFIG_CONSTANTS_ONLY */ + +/****************************************************************************** + * + * Build-only configuration options + * + * The remaining configurations are build-options only. + * They do not affect the API described in mlkem_native.h. + * + *****************************************************************************/ + +#if defined(MLK_BUILD_INTERNAL) +/** + * MLK_CONFIG_MULTILEVEL_WITH_SHARED + * + * This is for multi-level builds of mlkem-native only. If you need only a + * single parameter set, keep this unset. + * + * If this is set, all MLK_CONFIG_PARAMETER_SET-independent code will be + * included in the build, including code needed only for other parameter + * sets. + * + * Example: mlk_poly_cbd3 is only needed for MLK_CONFIG_PARAMETER_SET == 512. + * Yet, if this option is set for a build with + * MLK_CONFIG_PARAMETER_SET == 768/1024, it would be included. + * + * To build mlkem-native with support for all parameter sets, build it three + * times -- once per parameter set -- and set the option + * MLK_CONFIG_MULTILEVEL_WITH_SHARED for exactly one of them, and + * MLK_CONFIG_MULTILEVEL_NO_SHARED for the others. + * MLK_CONFIG_MULTILEVEL_BUILD should be set for all of them. + * + * See examples/multilevel_build for an example. + * + * This can also be set using CFLAGS. + */ +/* #define MLK_CONFIG_MULTILEVEL_WITH_SHARED */ + +/** + * MLK_CONFIG_MULTILEVEL_NO_SHARED + * + * This is for multi-level builds of mlkem-native only. If you need only a + * single parameter set, keep this unset. + * + * If this is set, no MLK_CONFIG_PARAMETER_SET-independent code will be + * included in the build. + * + * To build mlkem-native with support for all parameter sets, build it three + * times -- once per parameter set -- and set the option + * MLK_CONFIG_MULTILEVEL_WITH_SHARED for exactly one of them, and + * MLK_CONFIG_MULTILEVEL_NO_SHARED for the others. + * MLK_CONFIG_MULTILEVEL_BUILD should be set for all of them. + * + * See examples/multilevel_build for an example. + * + * This can also be set using CFLAGS. + */ +/* #define MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +/** + * MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS + * + * This is only relevant for single compilation unit (SCU) builds of + * mlkem-native. In this case, it determines whether directives defined in + * parameter-set-independent headers should be #undef'ined or not at the + * end of the SCU file. This is needed in multilevel builds. + * + * See examples/multilevel_build_native for an example. + * + * This can also be set using CFLAGS. + */ +/* #define MLK_CONFIG_MONOBUILD_KEEP_SHARED_HEADERS */ + +/** + * MLK_CONFIG_USE_NATIVE_BACKEND_ARITH + * + * Determines whether a native arithmetic backend should be used. + * + * The arithmetic backend covers performance-critical functions such as the + * number-theoretic transform (NTT). + * + * If this option is unset, the C backend will be used. + * + * If this option is set, the arithmetic backend to be used is determined + * by MLK_CONFIG_ARITH_BACKEND_FILE: if the latter is unset, the default + * backend for the target architecture will be used. If set, it must be the + * name of a backend metadata file. + * + * This can also be set using CFLAGS. + */ +#if !defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH) +/* #define MLK_CONFIG_USE_NATIVE_BACKEND_ARITH */ +#endif + +/** + * MLK_CONFIG_ARITH_BACKEND_FILE + * + * The arithmetic backend to use. + * + * If MLK_CONFIG_USE_NATIVE_BACKEND_ARITH is unset, this option is ignored. + * + * If MLK_CONFIG_USE_NATIVE_BACKEND_ARITH is set, this option must either + * be undefined or the filename of an arithmetic backend. If unset, the + * default backend will be used. + * + * This can be set using CFLAGS. + */ +#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH) && \ + !defined(MLK_CONFIG_ARITH_BACKEND_FILE) +#define MLK_CONFIG_ARITH_BACKEND_FILE "native/meta.h" +#endif + +/** + * MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 + * + * Determines whether a native FIPS202 backend should be used. + * + * The FIPS202 backend covers 1x/2x/4x-fold Keccak-f1600, which is the + * performance bottleneck of SHA3 and SHAKE. + * + * If this option is unset, the C backend will be used. + * + * If this option is set, the FIPS202 backend to be used is determined by + * MLK_CONFIG_FIPS202_BACKEND_FILE: if the latter is unset, the default + * backend for the target architecture will be used. If set, it must be + * the name of a backend metadata file. + * + * This can also be set using CFLAGS. + */ +#if !defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202) +/* #define MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 */ +#endif + +/** + * MLK_CONFIG_FIPS202_BACKEND_FILE + * + * The FIPS-202 backend to use. + * + * If MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 is set, this option must either + * be undefined or the filename of a FIPS202 backend. If unset, the default + * backend will be used. + * + * This can be set using CFLAGS. + */ +#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202) && \ + !defined(MLK_CONFIG_FIPS202_BACKEND_FILE) +#define MLK_CONFIG_FIPS202_BACKEND_FILE "fips202/native/auto.h" +#endif + +/** + * MLK_CONFIG_FIPS202_CUSTOM_HEADER + * + * Custom header to use for FIPS-202. + * + * This should only be set if you intend to use a custom FIPS-202 + * implementation, different from the one shipped with mlkem-native. + * + * If set, it must be the name of a file serving as the replacement for + * mlkem/src/fips202/fips202.h, and exposing the same API (see FIPS202.md). + */ +/* #define MLK_CONFIG_FIPS202_CUSTOM_HEADER "SOME_FILE.h" */ + +/** + * MLK_CONFIG_FIPS202X4_CUSTOM_HEADER + * + * Custom header to use for FIPS-202-X4. + * + * This should only be set if you intend to use a custom FIPS-202 + * implementation, different from the one shipped with mlkem-native. + * + * If set, it must be the name of a file serving as the replacement for + * mlkem/src/fips202/fips202x4.h, and exposing the same API (see FIPS202.md). + */ +/* #define MLK_CONFIG_FIPS202X4_CUSTOM_HEADER "SOME_FILE.h" */ + +/** + * MLK_CONFIG_CUSTOM_ZEROIZE + * + * In compliance with @[FIPS203, Section 3.3], mlkem-native zeroizes + * intermediate buffers before returning from function calls. By default, + * those buffers are allocated from the stack; if MLK_CONFIG_CUSTOM_ALLOC_FREE + * is set, they are (mostly -- few exceptions remain at present) allocated from + * the configured custom allocator. + * + * mlkem-native also zeroizes caller-owned output buffers as needed to uphold + * the API convention that outputs be either unmodified or zeroized upon + * failure. + * + * Set this option and define `mlk_zeroize` if you want to use a custom + * method to zeroize intermediate and output buffers. + * + * The default implementation uses SecureZeroMemory on Windows and a + * memset + compiler barrier otherwise. If neither of those is available on + * the target platform, compilation will fail, and you will need to use + * MLK_CONFIG_CUSTOM_ZEROIZE to provide a custom implementation of + * `mlk_zeroize()`. + * + * @warning + * The zeroization conducted by mlkem-native reduces the likelihood of data + * leaking on the stack or custom allocators, but it does not eliminate it. + * For example, the C standard makes no guarantee about where a compiler + * allocates local structures and whether/where it makes copies of them. + * Also, in addition to entire structures, there may also be potentially + * exploitable leakage of individual values on the stack. If you need + * bullet-proof zeroization of the stack, you need to consider additional + * measures instead of what this feature provides. In this case, you can + * set mlk_zeroize to a no-op. Note that in this case you are also responsible + * for zeroizing output buffers upon failure. + */ +/* #define MLK_CONFIG_CUSTOM_ZEROIZE + #if !defined(__ASSEMBLER__) + #include + #include "src/sys.h" + static MLK_INLINE void mlk_zeroize(void *ptr, size_t len) + { + ... your implementation ... + } + #endif +*/ + +/** + * MLK_CONFIG_CUSTOM_RANDOMBYTES + * + * mlkem-native does not provide a secure randombytes implementation. Such + * an implementation has to be provided by the consumer. + * + * If this option is not set, mlkem-native expects a function + * int randombytes(uint8_t *out, size_t outlen). It is expected to return + * zero on success, and non-zero on failure. In case of failure, the + * top-level APIs will return an MLK_ERR_RNG_FAIL error code. + * + * Set this option and define `mlk_randombytes` (with the same signature + * and behaviour) if you want to use a custom method to sample randombytes + * with a different name or signature. + */ +/* #define MLK_CONFIG_CUSTOM_RANDOMBYTES + #if !defined(__ASSEMBLER__) + #include + #include "src/sys.h" + static MLK_INLINE int mlk_randombytes(uint8_t *ptr, size_t len) + { + ... your implementation ... + return 0; + } + #endif +*/ + +/** + * MLK_CONFIG_CUSTOM_CAPABILITY_FUNC + * + * mlkem-native backends may rely on specific hardware features. Those + * backends will only be included in an mlkem-native build if support for + * the respective features is enabled at compile-time. However, when + * building for a heterogeneous set of CPUs to run the resulting + * binary/library on, feature detection at _runtime_ is needed to decide + * whether a backend can be used or not. + * + * Set this option and define `mlk_sys_check_capability` if you want to + * use a custom method to dispatch between implementations. + * + * If this option is not set, mlkem-native uses compile-time feature + * detection only to decide which backend to use. + * + * If you compile mlkem-native on a system with different capabilities + * than the system that the resulting binary/library will be run on, you + * must use this option. + */ +/* #define MLK_CONFIG_CUSTOM_CAPABILITY_FUNC + static MLK_INLINE int mlk_sys_check_capability(mlk_sys_cap cap) + __contract__( + ensures(return_value == 0 || return_value == 1) + ) + { + ... your implementation ... + } +*/ + +/** + * MLK_CONFIG_CUSTOM_ALLOC_FREE [EXPERIMENTAL] + * + * Set this option and define `MLK_CUSTOM_ALLOC` and `MLK_CUSTOM_FREE` if + * you want to use custom allocation for large local structures or buffers. + * + * By default, all buffers/structures are allocated on the stack. If this + * option is set, most of them will be allocated via MLK_CUSTOM_ALLOC. + * + * Parameters to MLK_CUSTOM_ALLOC: + * - T* v: Target pointer to declare. + * - T: Type of structure to be allocated. + * - N: Number of elements to be allocated. + * + * Parameters to MLK_CUSTOM_FREE: + * - T* v: Target pointer to free. May be NULL. + * - T: Type of structure to be freed. + * - N: Number of elements to be freed. + * + * @warning This option is experimental. Its scope, configuration and + * function/macro signatures may change at any time. We expect a + * stable API in a future version. + * + * @note Even if this option is set, some allocations further down the call + * stack will still be made from the stack, consuming up to 3KB of + * stack space. Those will likely be added to the scope of this + * option in the future. + * + * @note MLK_CUSTOM_ALLOC need not guarantee a successful allocation nor + * include error handling. Upon failure, the target pointer should + * simply be set to NULL. The calling code will handle this case and + * invoke MLK_CUSTOM_FREE. + */ +/* #define MLK_CONFIG_CUSTOM_ALLOC_FREE + #if !defined(__ASSEMBLER__) + #include + #define MLK_CUSTOM_ALLOC(v, T, N) \ + T* (v) = (T *)aligned_alloc(MLK_DEFAULT_ALIGN, \ + MLK_ALIGN_UP(sizeof(T) * (N))) + #define MLK_CUSTOM_FREE(v, T, N) free(v) + #endif +*/ + +/** + * MLK_CONFIG_CUSTOM_MEMCPY + * + * Set this option and define `mlk_memcpy` if you want to use a custom + * method to copy memory instead of the standard library memcpy function. + * + * The custom implementation must have the same signature and behavior as + * the standard memcpy function: + * void *mlk_memcpy(void *dest, const void *src, size_t n) + */ +/* #define MLK_CONFIG_CUSTOM_MEMCPY + #if !defined(__ASSEMBLER__) + #include + #include "src/sys.h" + static MLK_INLINE void *mlk_memcpy(void *dest, const void *src, size_t n) + { + ... your implementation ... + } + #endif +*/ + +/** + * MLK_CONFIG_CUSTOM_MEMSET + * + * Set this option and define `mlk_memset` if you want to use a custom + * method to set memory instead of the standard library memset function. + * + * The custom implementation must have the same signature and behavior as + * the standard memset function: + * void *mlk_memset(void *s, int c, size_t n) + */ +/* #define MLK_CONFIG_CUSTOM_MEMSET + #if !defined(__ASSEMBLER__) + #include + #include "src/sys.h" + static MLK_INLINE void *mlk_memset(void *s, int c, size_t n) + { + ... your implementation ... + } + #endif +*/ + +/** + * MLK_CONFIG_INTERNAL_API_QUALIFIER + * + * If set, this option provides an additional qualifier to be added to + * declarations of internal API functions and data. + * + * The primary use case for this option are single-CU builds, in which case + * this option can be set to `static`. + */ +/* #define MLK_CONFIG_INTERNAL_API_QUALIFIER */ + +/** + * MLK_CONFIG_CT_TESTING_ENABLED + * + * If set, mlkem-native annotates data as secret/public using valgrind's + * annotations VALGRIND_MAKE_MEM_UNDEFINED and VALGRIND_MAKE_MEM_DEFINED, + * enabling various checks for secret-dependent control flow or + * variable-time execution (depending on the exact version of valgrind + * installed). + */ +/* #define MLK_CONFIG_CT_TESTING_ENABLED */ + +/** + * MLK_CONFIG_NO_ASM + * + * If this option is set, mlkem-native will be built without use of native + * code or inline assembly. + * + * By default, inline assembly is used to implement value barriers. Without + * inline assembly, mlkem-native will use a global volatile 'opt blocker' + * instead; see verify.h. + * + * Inline assembly is also used to implement a secure zeroization function + * on non-Windows platforms. If this option is set and the target platform + * is not Windows, you MUST set MLK_CONFIG_CUSTOM_ZEROIZE and provide a + * custom zeroization function. + * + * If this option is set, MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 and + * MLK_CONFIG_USE_NATIVE_BACKEND_ARITH will be ignored, and no native + * backends will be used. + */ +/* #define MLK_CONFIG_NO_ASM */ + +/** + * MLK_CONFIG_NO_ASM_VALUE_BARRIER + * + * If this option is set, mlkem-native will be built without use of native + * code or inline assembly for value barriers. + * + * By default, inline assembly (if available) is used to implement value + * barriers. Without inline assembly, mlkem-native will use a global + * volatile 'opt blocker' instead; see verify.h. + */ +/* #define MLK_CONFIG_NO_ASM_VALUE_BARRIER */ + +/** + * MLK_CONFIG_KEYGEN_PCT + * + * Compliance with @[FIPS140_3_IG, p.87] requires a Pairwise Consistency + * Test (PCT) to be carried out on a freshly generated keypair before it + * can be exported. + * + * Set this option if such a check should be implemented. In this case, + * keypair_derand and keypair will return + * MLK_ERR_PCT_FAIL if the PCT failed. + * + * @note This feature will drastically lower the performance of key + * generation. + */ +/* #define MLK_CONFIG_KEYGEN_PCT */ + +/** + * MLK_CONFIG_KEYGEN_PCT_BREAKAGE_TEST + * + * If this option is set, the user must provide a runtime function + * `static inline int mlk_break_pct() { ... }` to indicate whether the PCT + * should be made to fail. + * + * This option only has an effect if MLK_CONFIG_KEYGEN_PCT is set. + */ +/* #define MLK_CONFIG_KEYGEN_PCT_BREAKAGE_TEST + #if !defined(__ASSEMBLER__) + #include "src/sys.h" + static MLK_INLINE int mlk_break_pct(void) + { + ... return 0/1 depending on whether PCT should be broken ... + } + #endif +*/ + +/** + * MLK_CONFIG_SERIAL_FIPS202_ONLY + * + * Set this to use a FIPS202 implementation with global state that supports + * only one active Keccak computation at a time (e.g. some hardware + * accelerators). + * + * If this option is set, batched Keccak operations are disabled for + * rejection sampling during matrix generation. Instead, matrix entries + * will be generated one at a time. + * + * This allows offloading Keccak computations to a hardware accelerator + * that holds only a single Keccak state locally, rather than requiring + * support for batched (4x) Keccak states. + * + * @note Depending on the target CPU, disabling batched Keccak may reduce + * performance when using software FIPS202 implementations. Only + * enable this when you have to. + */ +/* #define MLK_CONFIG_SERIAL_FIPS202_ONLY */ + +/** + * MLK_CONFIG_CONTEXT_PARAMETER + * + * Set this to add a context parameter that is provided to public API + * functions and is then available in custom callbacks. + * + * The type of the context parameter is configured via + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + */ +/* #define MLK_CONFIG_CONTEXT_PARAMETER */ + +/** + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE + * + * Set this to define the type for the context parameter used by + * MLK_CONFIG_CONTEXT_PARAMETER. + * + * This is only relevant if MLK_CONFIG_CONTEXT_PARAMETER is set. + */ +/* #define MLK_CONFIG_CONTEXT_PARAMETER_TYPE void* */ + +/************************* Config internals ********************************/ + +#endif /* MLK_BUILD_INTERNAL */ + +/* Default namespace + * + * Don't change this. If you need a different namespace, re-define + * MLK_CONFIG_NAMESPACE_PREFIX above instead, and remove the following. + * + * The default MLKEM namespace is + * + * PQCP_MLKEM_NATIVE_MLKEM_ + * + * e.g., PQCP_MLKEM_NATIVE_MLKEM512_ + */ + +#if defined(MLK_CONFIG_MULTILEVEL_BUILD) +/* In a multi-level build the parameter set is appended by the namespacing + * machinery, so the default prefix must not embed it. */ +#define MLK_DEFAULT_NAMESPACE_PREFIX PQCP_MLKEM_NATIVE_MLKEM +#elif MLK_CONFIG_PARAMETER_SET == 512 +#define MLK_DEFAULT_NAMESPACE_PREFIX PQCP_MLKEM_NATIVE_MLKEM512 +#elif MLK_CONFIG_PARAMETER_SET == 768 +#define MLK_DEFAULT_NAMESPACE_PREFIX PQCP_MLKEM_NATIVE_MLKEM768 +#elif MLK_CONFIG_PARAMETER_SET == 1024 +#define MLK_DEFAULT_NAMESPACE_PREFIX PQCP_MLKEM_NATIVE_MLKEM1024 +#endif + +#endif /* !MLK_CONFIG_H */ diff --git a/cbits/mlkem/src/cbmc.h b/cbits/mlkem/src/cbmc.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/cbmc.h @@ -0,0 +1,222 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLK_CBMC_H +#define MLK_CBMC_H +/*************************************************** + * Basic replacements for __CPROVER_XXX contracts + ***************************************************/ +/* + * The `__contract__` / `__loop__` annotation macros use a + * leading-double-underscore spelling in line with other CBMC macros. + * clang-tidy flags these as reserved identifiers; we suppress the diagnostic + * at each definition site (NOLINT) rather than disabling the check globally, + * so it stays active for the rest of the tree. + */ +#ifndef CBMC + +/* clang-format off */ +#define __contract__(x) /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */ +#define __loop__(x) /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */ +/* clang-format on */ + +#else /* !CBMC */ + + +/* clang-format off */ +#define __contract__(x) x /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */ +#define __loop__(x) x /* NOLINT(bugprone-reserved-identifier,cert-dcl37-c,cert-dcl51-cpp) */ +/* clang-format on */ + +/* https://diffblue.github.io/cbmc/contracts-assigns.html */ +#define assigns(...) __CPROVER_assigns(__VA_ARGS__) + +/* https://diffblue.github.io/cbmc/contracts-requires-ensures.html */ +#define requires(...) __CPROVER_requires(__VA_ARGS__) +#define ensures(...) __CPROVER_ensures(__VA_ARGS__) +/* https://diffblue.github.io/cbmc/contracts-loops.html */ +#define invariant(...) __CPROVER_loop_invariant(__VA_ARGS__) +#define decreases(...) __CPROVER_decreases(__VA_ARGS__) +/* cassert to avoid confusion with in-built assert */ +#define cassert(x) __CPROVER_assert(x, "cbmc assertion failed") +#define assume(...) __CPROVER_assume(__VA_ARGS__) + +/*************************************************** + * Macros for "expression" forms that may appear + * _inside_ top-level contracts. + ***************************************************/ + +/* + * function return value - useful inside ensures + * https://diffblue.github.io/cbmc/contracts-functions.html + */ +#define return_value (__CPROVER_return_value) + +/* + * assigns l-value targets + * https://diffblue.github.io/cbmc/contracts-assigns.html + */ +#define object_whole(...) __CPROVER_object_whole(__VA_ARGS__) +#define memory_slice(...) __CPROVER_object_upto(__VA_ARGS__) + +/* + * Pointer-related predicates + * https://diffblue.github.io/cbmc/contracts-memory-predicates.html + */ +#define memory_no_alias(...) __CPROVER_is_fresh(__VA_ARGS__) +#define readable(...) __CPROVER_r_ok(__VA_ARGS__) +#define writeable(...) __CPROVER_w_ok(__VA_ARGS__) + +/* Maximum supported buffer size + * + * Larger buffers may be supported, but due to internal modeling constraints + * in CBMC, the proofs of memory- and type-safety won't be able to run. + * + * If you find yourself in need for a buffer size larger than this, + * please contact the maintainers, so we can prioritize work to relax + * this somewhat artificial bound. + */ +#define MLK_MAX_BUFFER_SIZE (SIZE_MAX >> 12) + +/* + * History variables + * https://diffblue.github.io/cbmc/contracts-history-variables.html + */ +#define old(...) __CPROVER_old(__VA_ARGS__) +#define loop_entry(...) __CPROVER_loop_entry(__VA_ARGS__) + +/* + * Quantifiers + * Note that the range on qvar is _exclusive_ between qvar_lb .. qvar_ub + * https://diffblue.github.io/cbmc/contracts-quantifiers.html + * + * The quantified variable is declared as uint32_t, so these macros + * quantify only over indices in [0, UINT32_MAX). Bounds larger than + * UINT32_MAX (4 GiB) are NOT supported: the explicit (uint32_t) casts + * on the bounds will trigger CBMC's conversion check if a wider bound + * (e.g. a size_t > UINT32_MAX) is passed. + * + * Quantifying over size_t (64-bit) was found to blow up SMT proof + * times, so we deliberately keep the index width at 32 bits. Callers + * dealing with size_t-typed buffers must add an explicit + * requires(len <= UINT32_MAX) + * precondition. + */ + +/* + * Prevent clang-format from corrupting CBMC's special ==> operator + */ +/* clang-format off */ +#define forall(qvar, qvar_lb, qvar_ub, predicate) \ + __CPROVER_forall \ + { \ + uint32_t qvar; \ + ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> (predicate) \ + } + +#define exists(qvar, qvar_lb, qvar_ub, predicate) \ + __CPROVER_exists \ + { \ + uint32_t qvar; \ + ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) && (predicate) \ + } +/* clang-format on */ + +/*************************************************** + * Convenience macros for common contract patterns + ***************************************************/ + +/* + * Boolean-value predidate that asserts that "all values of array_var are in + * range value_lb (inclusive) .. value_ub (exclusive)" + * Example: + * array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q) + * expands to + * __CPROVER_forall { int k; (0 <= k && k <= MLKEM_N-1) ==> ( + * 0 <= a->coeffs[k]) && a->coeffs[k] < MLKEM_Q)) } + */ + +/* + * Prevent clang-format from corrupting CBMC's special ==> operator + */ +/* clang-format off */ +#define CBMC_CONCAT_(left, right) left##right +#define CBMC_CONCAT(left, right) CBMC_CONCAT_(left, right) + +#define array_bound_core(qvar, qvar_lb, qvar_ub, array_var, \ + value_lb, value_ub) \ + __CPROVER_forall \ + { \ + uint32_t qvar; \ + ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> \ + (((int)(value_lb) <= ((array_var)[(qvar)])) && \ + (((array_var)[(qvar)]) < (int)(value_ub))) \ + } + +#define array_bound(array_var, qvar_lb, qvar_ub, value_lb, value_ub) \ + array_bound_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), (qvar_lb), \ + (qvar_ub), (array_var), (value_lb), (value_ub)) + +#define array_unchanged_core(qvar, qvar_lb, qvar_ub, array_var) \ + __CPROVER_forall \ + { \ + uint32_t qvar; \ + ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> \ + ((array_var)[(qvar)]) == (old(* (int16_t (*)[(qvar_ub)])(array_var)))[(qvar)] \ + } + +#define array_unchanged(array_var, N) \ + array_unchanged_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), 0, (N), (array_var)) + +#define array_unchanged_u64_core(qvar, qvar_lb, qvar_ub, array_var) \ + __CPROVER_forall \ + { \ + uint32_t qvar; \ + ((uint32_t) (qvar_lb) <= (qvar) && (qvar) < (uint32_t) (qvar_ub)) ==> \ + ((array_var)[(qvar)]) == (old(* (uint64_t (*)[(qvar_ub)])(array_var)))[(qvar)] \ + } + +#define array_unchanged_u64(array_var, N) \ + array_unchanged_u64_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), 0, (N), (array_var)) + +#define array_unchanged_u8_core(qvar, array_var, N) \ + forall(qvar, 0, (N), \ + ((array_var)[(qvar)]) == (old(* (uint8_t (*)[(N)])(array_var)))[(qvar)]) + +#define array_unchanged_u8(array_var, N) \ + array_unchanged_u8_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), (array_var), (N)) + +#define array_zeroized_u8_core(qvar, array_var, N) \ + forall(qvar, 0, (N), ((array_var)[(qvar)]) == 0) + +#define array_zeroized_u8(array_var, N) \ + array_zeroized_u8_core(CBMC_CONCAT(_cbmc_idx, __COUNTER__), (array_var), (N)) +/* clang-format on */ + +/* + * Output-buffer discipline on failure, as documented in API-CONVENTIONS.md: + * when a function fails, each caller-owned output buffer is left either + * fully unchanged or fully zeroized -- never holding partially computed or + * stale data that could be mistaken for a valid result. + * + * Note the disjunction is over the buffer as a whole: it is not enough for + * each byte to be individually either unchanged or zero. + */ +#define array_unchanged_or_zeroized_u8(array_var, N) \ + (array_unchanged_u8((array_var), (N)) || array_zeroized_u8((array_var), (N))) + +/* Wrapper around array_bound operating on absolute values. + * + * The absolute value bound `k` is exclusive. + * + * Note that since the lower bound in array_bound is inclusive, we have to + * raise it by 1 here. + */ +#define array_abs_bound(arr, lb, ub, k) \ + array_bound((arr), (lb), (ub), -((int)(k)) + 1, (k)) + +#endif /* CBMC */ + +#endif /* !MLK_CBMC_H */ diff --git a/cbits/mlkem/src/common.h b/cbits/mlkem/src/common.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/common.h @@ -0,0 +1,296 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS140_3_IG] + * Implementation Guidance for FIPS 140-3 and the Cryptographic Module + * Validation Program + * National Institute of Standards and Technology + * https://csrc.nist.gov/projects/cryptographic-module-validation-program/fips-140-3-ig-announcements + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + */ + +#ifndef MLK_COMMON_H +#define MLK_COMMON_H + +#ifndef __ASSEMBLER__ +#include +#endif + +#define MLK_BUILD_INTERNAL + +#if defined(MLK_CONFIG_FILE) +#include MLK_CONFIG_FILE +#else +#include "mlkem_native_config.h" +#endif + +#include "params.h" +#include "sys.h" + +/* Internal and public API have external linkage by default, but + * this can be overwritten by the user, e.g. for single-CU builds. */ +#if !defined(MLK_CONFIG_INTERNAL_API_QUALIFIER) +#define MLK_INTERNAL_API +#define MLK_INTERNAL_DATA_DECLARATION extern +#define MLK_INTERNAL_DATA_DEFINITION +#else +#define MLK_INTERNAL_API MLK_CONFIG_INTERNAL_API_QUALIFIER +#define MLK_INTERNAL_DATA_DECLARATION MLK_CONFIG_INTERNAL_API_QUALIFIER +#define MLK_INTERNAL_DATA_DEFINITION MLK_CONFIG_INTERNAL_API_QUALIFIER +#endif + +#if !defined(MLK_CONFIG_EXTERNAL_API_QUALIFIER) +#define MLK_EXTERNAL_API +#else +#define MLK_EXTERNAL_API MLK_CONFIG_EXTERNAL_API_QUALIFIER +#endif + +#define MLK_CONCAT_(x1, x2) x1##x2 +#define MLK_CONCAT(x1, x2) MLK_CONCAT_(x1, x2) + +#if (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || \ + defined(MLK_CONFIG_MULTILEVEL_NO_SHARED)) +#define MLK_ADD_PARAM_SET(s) MLK_CONCAT(s, MLK_CONFIG_PARAMETER_SET) +#else +#define MLK_ADD_PARAM_SET(s) s +#endif + +#define MLK_NAMESPACE_PREFIX MLK_CONCAT(MLK_CONFIG_NAMESPACE_PREFIX, _) +#define MLK_NAMESPACE_PREFIX_K \ + MLK_CONCAT(MLK_ADD_PARAM_SET(MLK_CONFIG_NAMESPACE_PREFIX), _) + +/* Functions are prefixed by MLK_CONFIG_NAMESPACE_PREFIX. + * + * If multiple parameter sets are used, functions depending on the parameter + * set are additionally prefixed with 512/768/1024. See mlkem_native_config.h. + * + * Example: If MLK_CONFIG_NAMESPACE_PREFIX is mlkem, then + * MLK_NAMESPACE_K(enc) becomes mlkem512_enc/mlkem768_enc/mlkem1024_enc. + */ +#define MLK_NAMESPACE(s) MLK_CONCAT(MLK_NAMESPACE_PREFIX, s) +#define MLK_NAMESPACE_K(s) MLK_CONCAT(MLK_NAMESPACE_PREFIX_K, s) + +/* On Apple platforms, we need to emit leading underscore + * in front of assembly symbols. We thus introduce a separate + * namespace wrapper for ASM symbols. */ +#if !defined(__APPLE__) +#define MLK_ASM_NAMESPACE(sym) MLK_NAMESPACE(sym) +#else +#define MLK_ASM_NAMESPACE(sym) MLK_CONCAT(_, MLK_NAMESPACE(sym)) +#endif + +/* + * On X86_64 if control-flow protections (CET) are enabled (through + * -fcf-protection=), we add an endbr64 instruction at every global function + * label. See sys.h for more details + */ +#if defined(MLK_SYS_X86_64) +#define MLK_ASM_FN_SYMBOL(sym) MLK_ASM_NAMESPACE(sym) : MLK_CET_ENDBR +#elif defined(MLK_SYS_ARMV81M_MVE) +/* clang-format off */ +#define MLK_ASM_FN_SYMBOL(sym) \ + .type MLK_ASM_NAMESPACE(sym), %function; \ + MLK_ASM_NAMESPACE(sym) : +/* clang-format on */ +#else /* !MLK_SYS_X86_64 && MLK_SYS_ARMV81M_MVE */ +#define MLK_ASM_FN_SYMBOL(sym) MLK_ASM_NAMESPACE(sym) : +#endif /* !MLK_SYS_X86_64 && !MLK_SYS_ARMV81M_MVE */ + +/* + * Output the size of an assembly function. + */ +#if defined(__ELF__) +#define MLK_ASM_FN_SIZE(sym) \ + .size MLK_ASM_NAMESPACE(sym), .- MLK_ASM_NAMESPACE(sym) +#else +#define MLK_ASM_FN_SIZE(sym) +#endif + +/* We aim to simplify the user's life by supporting builds where + * all source files are included, even those that are not needed. + * Those files are appropriately guarded and will be empty when unneeded. + * The following is to avoid compilers complaining about this. */ +#define MLK_EMPTY_CU(s) extern int MLK_NAMESPACE_K(empty_cu_##s); + +/* MLK_CONFIG_NO_ASM takes precedence over MLK_USE_NATIVE_XXX */ +#if defined(MLK_CONFIG_NO_ASM) +#undef MLK_CONFIG_USE_NATIVE_BACKEND_ARITH +#undef MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 +#endif + +#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH) && \ + !defined(MLK_CONFIG_ARITH_BACKEND_FILE) +#error Bad configuration: MLK_CONFIG_USE_NATIVE_BACKEND_ARITH is set, but MLK_CONFIG_ARITH_BACKEND_FILE is not. +#endif + +#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202) && \ + !defined(MLK_CONFIG_FIPS202_BACKEND_FILE) +#error Bad configuration: MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 is set, but MLK_CONFIG_FIPS202_BACKEND_FILE is not. +#endif + +#if defined(MLK_CONFIG_NO_RANDOMIZED_API) && defined(MLK_CONFIG_KEYGEN_PCT) +#error Bad configuration: MLK_CONFIG_NO_RANDOMIZED_API is incompatible with MLK_CONFIG_KEYGEN_PCT as the current PCT implementation requires enc() +#endif + +#if defined(MLK_CONFIG_NO_ENCAPS_API) && defined(MLK_CONFIG_KEYGEN_PCT) +#error Bad configuration: MLK_CONFIG_NO_ENCAPS_API is incompatible with MLK_CONFIG_KEYGEN_PCT as the current PCT implementation requires enc() +#endif + +#if defined(MLK_CONFIG_NO_DECAPS_API) && defined(MLK_CONFIG_KEYGEN_PCT) +#error Bad configuration: MLK_CONFIG_NO_DECAPS_API is incompatible with MLK_CONFIG_KEYGEN_PCT as the current PCT implementation requires dec() +#endif + +#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_ARITH) +#include MLK_CONFIG_ARITH_BACKEND_FILE +/* Include to enforce consistency of API and implementation, + * and conduct sanity checks on the backend. + * + * Keep this _after_ the inclusion of the backend; otherwise, + * the sanity checks won't have an effect. */ +#if defined(MLK_CHECK_APIS) && !defined(__ASSEMBLER__) +#include "native/api.h" +#endif +#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_ARITH */ + +#if defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202) +#include MLK_CONFIG_FIPS202_BACKEND_FILE +/* Include to enforce consistency of API and implementation, + * and conduct sanity checks on the backend. + * + * Keep this _after_ the inclusion of the backend; otherwise, + * the sanity checks won't have an effect. */ +#if defined(MLK_CHECK_APIS) && !defined(__ASSEMBLER__) +#include "fips202/native/api.h" +#endif +#endif /* MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 */ + +#if !defined(MLK_CONFIG_FIPS202_CUSTOM_HEADER) +#define MLK_FIPS202_HEADER_FILE "fips202/fips202.h" +#else +#define MLK_FIPS202_HEADER_FILE MLK_CONFIG_FIPS202_CUSTOM_HEADER +#endif + +#if !defined(MLK_CONFIG_FIPS202X4_CUSTOM_HEADER) +#define MLK_FIPS202X4_HEADER_FILE "fips202/fips202x4.h" +#else +#define MLK_FIPS202X4_HEADER_FILE MLK_CONFIG_FIPS202X4_CUSTOM_HEADER +#endif + +/* Standard library function replacements */ +#if !defined(__ASSEMBLER__) +#if !defined(MLK_CONFIG_CUSTOM_MEMCPY) +#include +#define mlk_memcpy memcpy +#endif + +#if !defined(MLK_CONFIG_CUSTOM_MEMSET) +#include +#define mlk_memset memset +#endif + + +/* Allocation macros for large local structures + * + * MLK_ALLOC(v, T, N) declares T *v and attempts to point it to an T[N] + * MLK_FREE(v, T, N) zeroizes and frees the allocation + * + * Default implementation uses stack allocation. + * Can be overridden by setting the config option MLK_CONFIG_CUSTOM_ALLOC_FREE + * and defining MLK_CUSTOM_ALLOC and MLK_CUSTOM_FREE. + */ +#if defined(MLK_CONFIG_CUSTOM_ALLOC_FREE) != \ + (defined(MLK_CUSTOM_ALLOC) && defined(MLK_CUSTOM_FREE)) +#error Bad configuration: MLK_CONFIG_CUSTOM_ALLOC_FREE must be set together with MLK_CUSTOM_ALLOC and MLK_CUSTOM_FREE +#endif + +/* Context-parameter machinery (MLK_CONTEXT_PARAMETERS_n and related config + * checks). Kept in a separate, level-generic header for readability; included + * here so it is available to the allocation macros below and to all consumers + * of common.h. */ +#include "context.h" + +#if !defined(MLK_CONFIG_CUSTOM_ALLOC_FREE) +/* Default: stack allocation */ + +/* This is a declaration macro, not an expression macro: T is a type and v is + * a declarator, neither of which can be wrapped in parentheses. The + * bugprone-macro-parentheses diagnostic is therefore a false positive here. */ +#define MLK_ALLOC(v, T, N, context) \ + MLK_ALIGN T mlk_alloc_##v[N]; \ + T *v = mlk_alloc_##v /* NOLINT(bugprone-macro-parentheses) */ + +/* The MLK_FREE macro body references mlk_zeroize(), which is declared in + * verify.h. We deliberately do NOT include verify.h here: doing so would + * create a circular dependency (verify.h includes common.h), and common.h + * itself never calls mlk_zeroize() -- only the macro expansion does. Each + * translation unit that uses MLK_FREE therefore includes verify.h directly. */ +#define MLK_FREE(v, T, N, context) \ + do \ + { \ + MLK_CONTEXT_UNUSED(context); \ + mlk_zeroize(mlk_alloc_##v, sizeof(mlk_alloc_##v)); \ + (v) = NULL; \ + } while (0) + +#else /* !MLK_CONFIG_CUSTOM_ALLOC_FREE */ + +/* Custom allocation */ + +/* + * The indirection here is necessary to use MLK_CONTEXT_PARAMETERS_3 here. + */ +#define MLK_APPLY(f, args) f args + +#define MLK_ALLOC(v, T, N, context) \ + MLK_APPLY(MLK_CUSTOM_ALLOC, MLK_CONTEXT_PARAMETERS_3(v, T, N, context)) + +#define MLK_FREE(v, T, N, context) \ + do \ + { \ + if (v != NULL) \ + { \ + mlk_zeroize(v, sizeof(T) * (N)); \ + MLK_APPLY(MLK_CUSTOM_FREE, MLK_CONTEXT_PARAMETERS_3(v, T, N, context)); \ + v = NULL; \ + } \ + } while (0) + +#endif /* MLK_CONFIG_CUSTOM_ALLOC_FREE */ + +/****************************** Error codes ***********************************/ + +/* Generic failure condition. Currently not returned by any function; + * reserved for failures that no more specific code covers. */ +#define MLK_ERR_FAIL (-1) +/* An allocation failed. This can only happen if MLK_CONFIG_CUSTOM_ALLOC_FREE + * is defined and the provided MLK_CUSTOM_ALLOC can fail. */ +#define MLK_ERR_OUT_OF_MEMORY (-2) +/* An RNG failure occurred. Might be due to insufficient entropy or + * system misconfiguration. */ +#define MLK_ERR_RNG_FAIL (-3) +/* Public key validation failed: the @[FIPS203, Section 7.2, 'modulus check'] + * found a coefficient outside [0,q-1]. Returned by check_pk and by the + * encapsulation API. */ +#define MLK_ERR_INVALID_PK (-4) +/* Secret key validation failed: the @[FIPS203, Section 7.3, 'hash check'] + * found the embedded public key hash inconsistent. Returned by check_sk and + * by the decapsulation API. */ +#define MLK_ERR_INVALID_SK (-5) +/* The 'Pairwise Consistency Test' @[FIPS140_3_IG, p.87] and + * @[FIPS203, Section 7.1, Pairwise Consistency] failed. Only possible when + * MLK_CONFIG_KEYGEN_PCT is enabled; signals that the freshly generated key + * pair failed its encaps/decaps self-test. */ +#define MLK_ERR_PCT_FAIL (-6) + +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLK_COMMON_H */ diff --git a/cbits/mlkem/src/compress.c b/cbits/mlkem/src/compress.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/compress.c @@ -0,0 +1,763 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + * + * - [REF] + * CRYSTALS-Kyber C reference implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/ref + */ + +#include "common.h" +#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + + +#include "cbmc.h" +#include "compress.h" +#include "debug.h" +#include "verify.h" + +#if (!defined(MLK_CONFIG_NO_ENCAPS_API) || \ + !defined(MLK_CONFIG_NO_DECAPS_API)) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 || \ + MLKEM_K == 3) +/* Reference: `poly_compress()` in the reference implementation @[REF], + * for ML-KEM-{512,768}. + * - In contrast to the reference implementation, we assume + * unsigned canonical coefficients here. + * The reference implementation works with coefficients + * in the range [-(MLKEM_Q-1), MLKEM_Q-1]. */ +MLK_STATIC_TESTABLE void mlk_poly_compress_d4_c( + uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D4], const mlk_poly *a) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D4)) + requires(memory_no_alias(a, sizeof(mlk_poly))) + requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D4)) +) +{ + unsigned i; + mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q); + + for (i = 0; i < MLKEM_N / 8; i++) + __loop__(invariant(i <= MLKEM_N / 8) + decreases(MLKEM_N / 8 - i)) + { + unsigned j; + uint8_t t[8] = {0}; + for (j = 0; j < 8; j++) + __loop__( + invariant(i <= MLKEM_N / 8 && j <= 8) + invariant(array_bound(t, 0, j, 0, 16)) + decreases(8 - j)) + { + t[j] = mlk_scalar_compress_d4(a->coeffs[8 * i + j]); + } + + /* All t[i] are 4-bit wide, so the truncations don't alter the value. */ + r[i * 4] = (uint8_t)(t[0] | (t[1] << 4)); + r[i * 4 + 1] = (uint8_t)(t[2] | (t[3] << 4)); + r[i * 4 + 2] = (uint8_t)(t[4] | (t[5] << 4)); + r[i * 4 + 3] = (uint8_t)(t[6] | (t[7] << 4)); + } +} + +MLK_INTERNAL_API +void mlk_poly_compress_d4(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D4], + const mlk_poly *a) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D4)) + requires(memory_no_alias(a, sizeof(mlk_poly))) + requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D4)) +) +{ +#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D4) + int ret; + mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q); + ret = mlk_poly_compress_d4_native(r, a->coeffs); + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + return; + } +#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D4 */ + + mlk_poly_compress_d4_c(r, a); +} + +/* Reference: Embedded into `polyvec_compress()` in the + * reference implementation, for ML-KEM-{512,768}. + * - In contrast to the reference implementation, we assume + * unsigned canonical coefficients here. + * The reference implementation works with coefficients + * in the range [-(MLKEM_Q-1), MLKEM_Q-1]. */ +MLK_STATIC_TESTABLE void mlk_poly_compress_d10_c( + uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D10], const mlk_poly *a) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D10)) + requires(memory_no_alias(a, sizeof(mlk_poly))) + requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D10)) +) +{ + unsigned j; + mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q); + for (j = 0; j < MLKEM_N / 4; j++) + __loop__(invariant(j <= MLKEM_N / 4) + decreases(MLKEM_N / 4 - j)) + { + unsigned k; + uint16_t t[4]; + for (k = 0; k < 4; k++) + __loop__( + invariant(k <= 4) + invariant(forall(r, 0, k, t[r] < (1u << 10))) + decreases(4 - k)) + { + t[k] = mlk_scalar_compress_d10(a->coeffs[4 * j + k]); + } + + /* + * Make all implicit truncation explicit. No data is being + * truncated for the LHS's since each t[i] is 10-bit in size. + */ + r[5 * j + 0] = (uint8_t)((t[0] >> 0) & 0xFF); + r[5 * j + 1] = (uint8_t)((t[0] >> 8) | ((t[1] << 2) & 0xFF)); + r[5 * j + 2] = (uint8_t)((t[1] >> 6) | ((t[2] << 4) & 0xFF)); + r[5 * j + 3] = (uint8_t)((t[2] >> 4) | ((t[3] << 6) & 0xFF)); + r[5 * j + 4] = (uint8_t)(t[3] >> 2); + } +} + +MLK_INTERNAL_API +void mlk_poly_compress_d10(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D10], + const mlk_poly *a) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D10)) + requires(memory_no_alias(a, sizeof(mlk_poly))) + requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D10)) +) +{ +#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D10) + int ret; + mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q); + ret = mlk_poly_compress_d10_native(r, a->coeffs); + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + return; + } +#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D10 */ + + mlk_poly_compress_d10_c(r, a); +} + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +/* Reference: `poly_decompress()` in the reference implementation @[REF], + * for ML-KEM-{512,768}. */ +MLK_STATIC_TESTABLE void mlk_poly_decompress_d4_c( + mlk_poly *r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D4]) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D4)) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) +) +{ + unsigned i; + for (i = 0; i < MLKEM_N / 2; i++) + __loop__( + invariant(i <= MLKEM_N / 2) + invariant(array_bound(r->coeffs, 0, 2 * i, 0, MLKEM_Q)) + decreases(MLKEM_N / 2 - i)) + { + r->coeffs[2 * i + 0] = mlk_scalar_decompress_d4((a[i] >> 0) & 0xF); + r->coeffs[2 * i + 1] = mlk_scalar_decompress_d4((a[i] >> 4) & 0xF); + } + + mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q); +} + +MLK_INTERNAL_API +void mlk_poly_decompress_d4(mlk_poly *r, + const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D4]) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D4)) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) +) +{ +#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D4) + int ret; + ret = mlk_poly_decompress_d4_native(r->coeffs, a); + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q); + return; + } +#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D4 */ + + mlk_poly_decompress_d4_c(r, a); +} + +/* Reference: Embedded into `polyvec_decompress()` in the + * reference implementation, for ML-KEM-{512,768}. */ +MLK_STATIC_TESTABLE void mlk_poly_decompress_d10_c( + mlk_poly *r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D10]) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D10)) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) +) +{ + unsigned j; + for (j = 0; j < MLKEM_N / 4; j++) + __loop__( + invariant(j <= MLKEM_N / 4) + invariant(array_bound(r->coeffs, 0, 4 * j, 0, MLKEM_Q)) + decreases(MLKEM_N / 4 - j)) + { + unsigned k; + uint16_t t[4]; + uint8_t const *base = &a[5 * j]; + + t[0] = 0x3FF & ((base[0] >> 0) | ((uint16_t)base[1] << 8)); + t[1] = 0x3FF & ((base[1] >> 2) | ((uint16_t)base[2] << 6)); + t[2] = 0x3FF & ((base[2] >> 4) | ((uint16_t)base[3] << 4)); + t[3] = 0x3FF & ((base[3] >> 6) | ((uint16_t)base[4] << 2)); + + for (k = 0; k < 4; k++) + __loop__( + invariant(k <= 4) + invariant(array_bound(r->coeffs, 0, 4 * j + k, 0, MLKEM_Q)) + decreases(4 - k)) + { + r->coeffs[4 * j + k] = mlk_scalar_decompress_d10(t[k]); + } + } + + mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q); +} + +MLK_INTERNAL_API +void mlk_poly_decompress_d10(mlk_poly *r, + const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D10]) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D10)) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) +) +{ +#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D10) + int ret; + ret = mlk_poly_decompress_d10_native(r->coeffs, a); + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q); + return; + } +#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D10 */ + + mlk_poly_decompress_d10_c(r, a); +} +#endif /* !MLK_CONFIG_NO_DECAPS_API */ +#endif /* (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) && \ + (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 || MLKEM_K == 3) \ + */ + +#if (!defined(MLK_CONFIG_NO_ENCAPS_API) || \ + !defined(MLK_CONFIG_NO_DECAPS_API)) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4) +/* Reference: `poly_compress()` in the reference implementation @[REF], + * for ML-KEM-1024. + * - In contrast to the reference implementation, we assume + * unsigned canonical coefficients here. + * The reference implementation works with coefficients + * in the range [-(MLKEM_Q-1), MLKEM_Q-1]. */ +MLK_STATIC_TESTABLE void mlk_poly_compress_d5_c( + uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D5], const mlk_poly *a) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D5)) + requires(memory_no_alias(a, sizeof(mlk_poly))) + requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D5)) +) +{ + unsigned i; + mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q); + + for (i = 0; i < MLKEM_N / 8; i++) + __loop__(invariant(i <= MLKEM_N / 8) + decreases(MLKEM_N / 8 - i)) + { + unsigned j; + uint8_t t[8] = {0}; + for (j = 0; j < 8; j++) + __loop__( + invariant(i <= MLKEM_N / 8 && j <= 8) + invariant(array_bound(t, 0, j, 0, 32)) + decreases(8 - j)) + { + t[j] = mlk_scalar_compress_d5(a->coeffs[8 * i + j]); + } + + r[i * 5] = (uint8_t)(0xFF & ((t[0] >> 0) | (t[1] << 5))); + r[i * 5 + 1] = (uint8_t)(0xFF & ((t[1] >> 3) | (t[2] << 2) | (t[3] << 7))); + r[i * 5 + 2] = (uint8_t)(0xFF & ((t[3] >> 1) | (t[4] << 4))); + r[i * 5 + 3] = (uint8_t)(0xFF & ((t[4] >> 4) | (t[5] << 1) | (t[6] << 6))); + r[i * 5 + 4] = (uint8_t)(0xFF & ((t[6] >> 2) | (t[7] << 3))); + } +} + +MLK_INTERNAL_API +void mlk_poly_compress_d5(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D5], + const mlk_poly *a) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D5)) + requires(memory_no_alias(a, sizeof(mlk_poly))) + requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D5)) +) +{ +#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D5) + int ret; + mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q); + ret = mlk_poly_compress_d5_native(r, a->coeffs); + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + return; + } +#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D5 */ + + mlk_poly_compress_d5_c(r, a); +} + +/* Reference: Embedded into `polyvec_compress()` in the + * reference implementation, for ML-KEM-1024. + * - In contrast to the reference implementation, we assume + * unsigned canonical coefficients here. + * The reference implementation works with coefficients + * in the range [-(MLKEM_Q-1), MLKEM_Q-1]. */ +MLK_STATIC_TESTABLE void mlk_poly_compress_d11_c( + uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D11], const mlk_poly *a) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D11)) + requires(memory_no_alias(a, sizeof(mlk_poly))) + requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D11)) +) +{ + unsigned j; + mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q); + + for (j = 0; j < MLKEM_N / 8; j++) + __loop__(invariant(j <= MLKEM_N / 8) + decreases(MLKEM_N / 8 - j)) + { + unsigned k; + uint16_t t[8]; + for (k = 0; k < 8; k++) + __loop__( + invariant(k <= 8) + invariant(forall(r, 0, k, t[r] < (1u << 11))) + decreases(8 - k)) + { + t[k] = mlk_scalar_compress_d11(a->coeffs[8 * j + k]); + } + + /* + * Make all implicit truncation explicit. No data is being + * truncated for the LHS's since each t[i] is 11-bit in size. + */ + r[11 * j + 0] = (uint8_t)((t[0] >> 0) & 0xFF); + r[11 * j + 1] = (uint8_t)((t[0] >> 8) | ((t[1] << 3) & 0xFF)); + r[11 * j + 2] = (uint8_t)((t[1] >> 5) | ((t[2] << 6) & 0xFF)); + r[11 * j + 3] = (uint8_t)((t[2] >> 2) & 0xFF); + r[11 * j + 4] = (uint8_t)((t[2] >> 10) | ((t[3] << 1) & 0xFF)); + r[11 * j + 5] = (uint8_t)((t[3] >> 7) | ((t[4] << 4) & 0xFF)); + r[11 * j + 6] = (uint8_t)((t[4] >> 4) | ((t[5] << 7) & 0xFF)); + r[11 * j + 7] = (uint8_t)((t[5] >> 1) & 0xFF); + r[11 * j + 8] = (uint8_t)((t[5] >> 9) | ((t[6] << 2) & 0xFF)); + r[11 * j + 9] = (uint8_t)((t[6] >> 6) | ((t[7] << 5) & 0xFF)); + r[11 * j + 10] = (uint8_t)(t[7] >> 3); + } +} + +MLK_INTERNAL_API +void mlk_poly_compress_d11(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D11], + const mlk_poly *a) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D11)) + requires(memory_no_alias(a, sizeof(mlk_poly))) + requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D11)) +) +{ +#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D11) + int ret; + mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q); + ret = mlk_poly_compress_d11_native(r, a->coeffs); + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + return; + } +#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D11 */ + + mlk_poly_compress_d11_c(r, a); +} + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +/* Reference: `poly_decompress()` in the reference implementation @[REF], + * for ML-KEM-1024. */ +MLK_STATIC_TESTABLE void mlk_poly_decompress_d5_c( + mlk_poly *r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D5]) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D5)) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) +) +{ + unsigned i; + for (i = 0; i < MLKEM_N / 8; i++) + __loop__( + invariant(i <= MLKEM_N / 8) + invariant(array_bound(r->coeffs, 0, 8 * i, 0, MLKEM_Q)) + decreases(MLKEM_N / 8 - i)) + { + unsigned j; + uint8_t t[8]; + const unsigned offset = i * 5; + /* + * Explicitly truncate to avoid warning about + * implicit truncation in CBMC and unwind loop for ease + * of proof. + */ + + /* + * Decompress 5 8-bit bytes (so 40 bits) into + * 8 5-bit values stored in t[] + */ + t[0] = 0x1F & (a[offset + 0] >> 0); + t[1] = 0x1F & ((a[offset + 0] >> 5) | (a[offset + 1] << 3)); + t[2] = 0x1F & (a[offset + 1] >> 2); + t[3] = 0x1F & ((a[offset + 1] >> 7) | (a[offset + 2] << 1)); + t[4] = 0x1F & ((a[offset + 2] >> 4) | (a[offset + 3] << 4)); + t[5] = 0x1F & (a[offset + 3] >> 1); + t[6] = 0x1F & ((a[offset + 3] >> 6) | (a[offset + 4] << 2)); + t[7] = 0x1F & (a[offset + 4] >> 3); + + /* and copy to the correct slice in r[] */ + for (j = 0; j < 8; j++) + __loop__( + invariant(j <= 8 && i <= MLKEM_N / 8) + invariant(array_bound(r->coeffs, 0, 8 * i + j, 0, MLKEM_Q)) + decreases(8 - j)) + { + r->coeffs[8 * i + j] = mlk_scalar_decompress_d5(t[j]); + } + } + + mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q); +} + +MLK_INTERNAL_API +void mlk_poly_decompress_d5(mlk_poly *r, + const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D5]) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D5)) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) +) +{ +#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D5) + int ret; + ret = mlk_poly_decompress_d5_native(r->coeffs, a); + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q); + return; + } +#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D5 */ + + mlk_poly_decompress_d5_c(r, a); +} + +/* Reference: Embedded into `polyvec_decompress()` in the + * reference implementation, for ML-KEM-1024. */ +MLK_STATIC_TESTABLE void mlk_poly_decompress_d11_c( + mlk_poly *r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D11]) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D11)) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) +) +{ + unsigned j; + for (j = 0; j < MLKEM_N / 8; j++) + __loop__( + invariant(j <= MLKEM_N / 8) + invariant(array_bound(r->coeffs, 0, 8 * j, 0, MLKEM_Q)) + decreases(MLKEM_N / 8 - j)) + { + unsigned k; + uint16_t t[8]; + uint8_t const *base = &a[11 * j]; + t[0] = 0x7FF & ((base[0] >> 0) | ((uint16_t)base[1] << 8)); + t[1] = 0x7FF & ((base[1] >> 3) | ((uint16_t)base[2] << 5)); + t[2] = 0x7FF & ((base[2] >> 6) | ((uint16_t)base[3] << 2) | + ((uint16_t)base[4] << 10)); + t[3] = 0x7FF & ((base[4] >> 1) | ((uint16_t)base[5] << 7)); + t[4] = 0x7FF & ((base[5] >> 4) | ((uint16_t)base[6] << 4)); + t[5] = 0x7FF & ((base[6] >> 7) | ((uint16_t)base[7] << 1) | + ((uint16_t)base[8] << 9)); + t[6] = 0x7FF & ((base[8] >> 2) | ((uint16_t)base[9] << 6)); + t[7] = 0x7FF & ((base[9] >> 5) | ((uint16_t)base[10] << 3)); + + for (k = 0; k < 8; k++) + __loop__( + invariant(k <= 8) + invariant(array_bound(r->coeffs, 0, 8 * j + k, 0, MLKEM_Q)) + decreases(8 - k)) + { + r->coeffs[8 * j + k] = mlk_scalar_decompress_d11(t[k]); + } + } + + mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q); +} + +MLK_INTERNAL_API +void mlk_poly_decompress_d11(mlk_poly *r, + const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D11]) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D11)) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) +) +{ +#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D11) + int ret; + ret = mlk_poly_decompress_d11_native(r->coeffs, a); + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q); + return; + } +#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D11 */ + + mlk_poly_decompress_d11_c(r, a); +} + +#endif /* !MLK_CONFIG_NO_DECAPS_API */ +#endif /* (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) && \ + (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4) */ + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) || !defined(MLK_CONFIG_NO_ENCAPS_API) +/* Reference: `poly_tobytes()` in the reference implementation @[REF]. + * - In contrast to the reference implementation, we assume + * unsigned canonical coefficients here. + * The reference implementation works with coefficients + * in the range [-(MLKEM_Q-1), MLKEM_Q-1]. */ +MLK_STATIC_TESTABLE void mlk_poly_tobytes_c(uint8_t r[MLKEM_POLYBYTES], + const mlk_poly *a) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYBYTES)) + requires(memory_no_alias(a, sizeof(mlk_poly))) + requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYBYTES)) +) +{ + unsigned i; + mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q); + + for (i = 0; i < MLKEM_N / 2; i++) + __loop__(invariant(i <= MLKEM_N / 2) + decreases(MLKEM_N / 2 - i)) + { + /* The conversion to uint16_t is safe since we assume that + * the coefficients of `a` are non-negative. */ + const uint16_t t0 = (uint16_t)a->coeffs[2 * i]; + const uint16_t t1 = (uint16_t)a->coeffs[2 * i + 1]; + /* + * t0 and t1 are both < MLKEM_Q, so contain at most 12 bits each of + * significant data, so these can be packed into 24 bits or exactly + * 3 bytes, as follows. + */ + + /* Least significant bits 0 - 7 of t0. */ + r[3 * i + 0] = (uint8_t)(t0 & 0xFF); + + /* + * Most significant bits 8 - 11 of t0 become the least significant + * nibble of the second byte. The least significant 4 bits + * of t1 become the upper nibble of the second byte. + * + * The conversion to uint8_t does not alter the value. + */ + r[3 * i + 1] = (uint8_t)((t0 >> 8) | ((t1 << 4) & 0xF0)); + + /* Bits 4 - 11 of t1 become the third byte. The conversion to uint8_t + * does not alter the value because t1 is 12-bit wide. */ + r[3 * i + 2] = (uint8_t)(t1 >> 4); + } +} + +MLK_INTERNAL_API +void mlk_poly_tobytes(uint8_t r[MLKEM_POLYBYTES], const mlk_poly *a) +{ +#if defined(MLK_USE_NATIVE_POLY_TOBYTES) + int ret; + mlk_assert_bound(a, MLKEM_N, 0, MLKEM_Q); + ret = mlk_poly_tobytes_native(r, a->coeffs); + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + return; + } +#endif /* MLK_USE_NATIVE_POLY_TOBYTES */ + + mlk_poly_tobytes_c(r, a); +} +#endif /* !MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API */ + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +/* Reference: `poly_frombytes()` in the reference implementation @[REF]. */ +MLK_STATIC_TESTABLE void mlk_poly_frombytes_c(mlk_poly *r, + const uint8_t a[MLKEM_POLYBYTES]) +__contract__( + requires(memory_no_alias(a, MLKEM_POLYBYTES)) + requires(memory_no_alias(r, sizeof(mlk_poly))) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT)) +) +{ + unsigned i; + for (i = 0; i < MLKEM_N / 2; i++) + __loop__( + invariant(i <= MLKEM_N / 2) + invariant(array_bound(r->coeffs, 0, 2 * i, 0, MLKEM_UINT12_LIMIT)) + decreases(MLKEM_N / 2 - i)) + { + const uint8_t t0 = a[3 * i + 0]; + const uint8_t t1 = a[3 * i + 1]; + const uint8_t t2 = a[3 * i + 2]; + /* Safety: + * - The explicit cast to uint16_t ensures that << 8 does + * not signed-overflow even on a 16-bit system. + * - The cast to int16_t is safe due to the explicit 0xFFF truncation. + */ + r->coeffs[2 * i + 0] = (int16_t)(t0 | (((uint16_t)t1 << 8) & 0xFFF)); + r->coeffs[2 * i + 1] = (int16_t)((t1 >> 4) | (t2 << 4)); + } + + /* Note that the coefficients are not canonical */ + mlk_assert_bound(r, MLKEM_N, 0, MLKEM_UINT12_LIMIT); +} + +MLK_INTERNAL_API +void mlk_poly_frombytes(mlk_poly *r, const uint8_t a[MLKEM_POLYBYTES]) +{ +#if defined(MLK_USE_NATIVE_POLY_FROMBYTES) + int ret; + ret = mlk_poly_frombytes_native(r->coeffs, a); + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + return; + } +#endif /* MLK_USE_NATIVE_POLY_FROMBYTES */ + + mlk_poly_frombytes_c(r, a); +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +/* Reference: `poly_frommsg()` in the reference implementation @[REF]. + * - We use a value barrier around the bit-selection mask to + * reduce the risk of compiler-introduced branches. + * The reference implementation contains the expression + * `(msg[i] >> j) & 1` which the compiler can reason must + * be either 0 or 1. */ +MLK_INTERNAL_API +void mlk_poly_frommsg(mlk_poly *r, const uint8_t msg[MLKEM_INDCPA_MSGBYTES]) +{ + unsigned i; +#if (MLKEM_INDCPA_MSGBYTES != MLKEM_N / 8) +#error "MLKEM_INDCPA_MSGBYTES must be equal to MLKEM_N/8 bytes!" +#endif + + for (i = 0; i < MLKEM_N / 8; i++) + __loop__( + invariant(i <= MLKEM_N / 8) + invariant(array_bound(r->coeffs, 0, 8 * i, 0, MLKEM_Q)) + decreases(MLKEM_N / 8 - i)) + { + unsigned j; + for (j = 0; j < 8; j++) + __loop__( + invariant(i < MLKEM_N / 8 && j <= 8) + invariant(array_bound(r->coeffs, 0, 8 * i + j, 0, MLKEM_Q)) + decreases(8 - j)) + { + /* mlk_ct_sel_int16(MLKEM_Q_HALF, 0, b) is `Decompress_1(b != 0)` + * as per @[FIPS203, Eq (4.8)]. */ + + /* Prevent the compiler from recognizing this as a bit selection */ + uint8_t mask = mlk_value_barrier_u8((uint8_t)(1u << j)); + r->coeffs[8 * i + j] = mlk_ct_sel_int16(MLKEM_Q_HALF, 0, msg[i] & mask); + } + } + mlk_assert_abs_bound(r, MLKEM_N, MLKEM_Q); +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +/* Reference: `poly_tomsg()` in the reference implementation @[REF]. + * - In contrast to the reference implementation, we assume + * unsigned canonical coefficients here. + * The reference implementation works with coefficients + * in the range [-(MLKEM_Q-1), MLKEM_Q-1]. + */ +MLK_INTERNAL_API +void mlk_poly_tomsg(uint8_t msg[MLKEM_INDCPA_MSGBYTES], const mlk_poly *r) +{ + unsigned i; + mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q); + + for (i = 0; i < MLKEM_N / 8; i++) + __loop__(invariant(i <= MLKEM_N / 8) + decreases(MLKEM_N / 8 - i)) + { + unsigned j; + msg[i] = 0; + for (j = 0; j < 8; j++) + __loop__( + invariant(i <= MLKEM_N / 8 && j <= 8) + decreases(8 - j)) + { + uint32_t t = mlk_scalar_compress_d1(r->coeffs[8 * i + j]); + msg[i] |= (uint8_t)(t << j); + } + } +} +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +#else /* !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +MLK_EMPTY_CU(compress) + +#endif /* MLK_CONFIG_MULTILEVEL_NO_SHARED */ diff --git a/cbits/mlkem/src/compress.h b/cbits/mlkem/src/compress.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/compress.h @@ -0,0 +1,613 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + * + * - [REF] + * CRYSTALS-Kyber C reference implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/ref + */ + +#ifndef MLK_COMPRESS_H +#define MLK_COMPRESS_H + + +#include "cbmc.h" +#include "common.h" +#include "debug.h" +#include "poly.h" +#include "verify.h" + +/* + * The multiplication in this routine will exceed UINT32_MAX + * and wrap around for large values of u. This is expected and required. + */ +#ifdef CBMC +#pragma CPROVER check push +#pragma CPROVER check disable "unsigned-overflow" +#endif + +/** + * Compute round(u * 2 / MLKEM_Q). + * + * @spec{Compress_1 from @[FIPS203, Eq (4.7)].} + * + * @reference{Part of poly_tomsg() in the reference implementation @[REF].} + * + * @param u Unsigned canonical modulus modulo MLKEM_Q to be compressed. + * + * @return Compressed value. + */ +static MLK_INLINE uint8_t mlk_scalar_compress_d1(int16_t u) +__contract__( + requires(0 <= u && u <= MLKEM_Q - 1) + ensures(return_value < 2) + ensures(return_value == (((uint32_t)u * 2 + MLKEM_Q / 2) / MLKEM_Q) % 2) ) +{ + /* Compute as follows: + * ``` + * round(u * 2 / MLKEM_Q) + * = round(u * 2 * (2^31 / MLKEM_Q) / 2^31) + * ~= round(u * 2 * round(2^31 / MLKEM_Q) / 2^31) + * ``` + */ + /* check-magic: 1290168 == 2*round(2^31 / MLKEM_Q) */ + uint32_t d0 = (uint32_t)u * 1290168; + /* Unsigned shifting by 31 positions leaves only the top bit. */ + return (uint8_t)((d0 + ((uint32_t)1u << 30)) >> 31); +} +#ifdef CBMC +#pragma CPROVER check pop +#endif + +/* + * The multiplication in this routine will exceed UINT32_MAX + * and wrap around for large values of u. This is expected and required. + */ +#ifdef CBMC +#pragma CPROVER check push +#pragma CPROVER check disable "unsigned-overflow" +#endif + +/** + * Compute round(u * 16 / MLKEM_Q) % 16. + * + * @spec{Compress_4 from @[FIPS203, Eq (4.7)].} + * + * @reference{Embedded into `poly_compress()` in the reference + * implementation @[REF].} + * + * @param u Unsigned canonical modulus modulo MLKEM_Q to be compressed. + * + * @return Compressed value. + */ +static MLK_INLINE uint8_t mlk_scalar_compress_d4(int16_t u) +__contract__( + requires(0 <= u && u <= MLKEM_Q - 1) + ensures(return_value < 16) + ensures(return_value == (((uint32_t)u * 16 + MLKEM_Q / 2) / MLKEM_Q) % 16)) +{ + /* Compute as follows: + * ``` + * round(u * 16 / MLKEM_Q) + * = round(u * 16 * (2^28 / MLKEM_Q) / 2^28) + * ~= round(u * 16 * round(2^28 / MLKEM_Q) / 2^28) + * ``` + */ + /* check-magic: 1290160 == 16 * round(2^28 / MLKEM_Q) */ + uint32_t d0 = (uint32_t)u * 1290160; + /* The return value is < 16, so not altered by the conversion to uint8_t. */ + return (uint8_t)((d0 + ((uint32_t)1u << 27)) >> 28); /* round(d0/2^28) */ +} +#ifdef CBMC +#pragma CPROVER check pop +#endif + +/** + * Compute round(u * MLKEM_Q / 16). + * + * @spec{Decompress_4 from @[FIPS203, Eq (4.8)].} + * + * @reference{Embedded into `poly_decompress()` in the reference + * implementation @[REF].} + * + * @param u Unsigned canonical modulus modulo 16 to be decompressed. + * + * @return Decompressed value. + */ +static MLK_INLINE int16_t mlk_scalar_decompress_d4(uint8_t u) +__contract__( + requires(0 <= u && u < 16) + ensures(return_value <= (MLKEM_Q - 1)) +) +{ + /* The return value is in 0..MLKEM_Q-1, hence not altered by the + * conversion to int16_t. */ + return (int16_t)((((uint32_t)u * MLKEM_Q) + 8) >> 4); +} + +/* + * The multiplication in this routine will exceed UINT32_MAX + * and wrap around for large values of u. This is expected and required. + */ +#ifdef CBMC +#pragma CPROVER check push +#pragma CPROVER check disable "unsigned-overflow" +#endif + +/** + * Compute round(u * 32 / MLKEM_Q) % 32. + * + * @spec{Compress_5 from @[FIPS203, Eq (4.7)].} + * + * @reference{Embedded into `poly_compress()` in the reference + * implementation @[REF].} + * + * @param u Unsigned canonical modulus modulo MLKEM_Q to be compressed. + * + * @return Compressed value. + */ +static MLK_INLINE uint8_t mlk_scalar_compress_d5(int16_t u) +__contract__( + requires(0 <= u && u <= MLKEM_Q - 1) + ensures(return_value < 32) + ensures(return_value == (((uint32_t)u * 32 + MLKEM_Q / 2) / MLKEM_Q) % 32) ) +{ + /* Compute as follows: + * ``` + * round(u * 32 / MLKEM_Q) + * = round(u * 32 * (2^27 / MLKEM_Q) / 2^27) + * ~= round(u * 32 * round(2^27 / MLKEM_Q) / 2^27) + * ``` + */ + /* check-magic: 1290176 == 2^5 * round(2^27 / MLKEM_Q) */ + uint32_t d0 = (uint32_t)u * 1290176; + /* The return value is < 32, so not altered by the conversion to uint8_t. */ + return (uint8_t)((d0 + ((uint32_t)1u << 26)) >> 27); /* round(d0/2^27) */ +} +#ifdef CBMC +#pragma CPROVER check pop +#endif + +/** + * Compute round(u * MLKEM_Q / 32). + * + * @spec{Decompress_5 from @[FIPS203, Eq (4.8)].} + * + * @reference{Embedded into `poly_decompress()` in the reference + * implementation @[REF].} + * + * @param u Unsigned canonical modulus modulo 32 to be decompressed. + * + * @return Decompressed value. + */ +static MLK_INLINE int16_t mlk_scalar_decompress_d5(uint8_t u) +__contract__( + requires(0 <= u && u < 32) + ensures(0 <= return_value && return_value <= MLKEM_Q - 1) +) +{ + /* The return value is in 0..MLKEM_Q-1, hence not altered by the + * conversion to int16_t. */ + return (int16_t)((((uint32_t)u * MLKEM_Q) + 16) >> 5); +} + +/* + * The multiplication in this routine will exceed UINT32_MAX + * and wrap around for large values of u. This is expected and required. + */ +#ifdef CBMC +#pragma CPROVER check push +#pragma CPROVER check disable "unsigned-overflow" +#endif + +/** + * Compute round(u * 2**10 / MLKEM_Q) % 2**10. + * + * @spec{Compress_10 from @[FIPS203, Eq (4.7)].} + * + * @reference{Embedded into `polyvec_compress()` in the reference + * implementation @[REF].} + * + * @param u Unsigned canonical modulus modulo MLKEM_Q to be compressed. + * + * @return Compressed value. + */ +static MLK_INLINE uint16_t mlk_scalar_compress_d10(int16_t u) +__contract__( + requires(0 <= u && u <= MLKEM_Q - 1) + ensures(return_value < (1u << 10)) + ensures(return_value == (((uint32_t)u * (1u << 10) + MLKEM_Q / 2) / MLKEM_Q) % (1 << 10))) +{ + /* Compute as follows: + * ``` + * round(u * 1024 / MLKEM_Q) + * = round(u * 1024 * (2^33 / MLKEM_Q) / 2^33) + * ~= round(u * 1024 * round(2^33 / MLKEM_Q) / 2^33) + * ``` + */ + /* check-magic: 2642263040 == 2^10 * round(2^33 / MLKEM_Q) */ + uint64_t d0 = (uint64_t)u * 2642263040; + d0 = (d0 + ((uint64_t)1u << 32)) >> 33; /* round(d0/2^33) */ + return (d0 & 0x3FF); +} +#ifdef CBMC +#pragma CPROVER check pop +#endif + +/** + * Compute round(u * MLKEM_Q / 1024). + * + * @spec{Decompress_10 from @[FIPS203, Eq (4.8)].} + * + * @reference{Embedded into `polyvec_decompress()` in the reference + * implementation @[REF].} + * + * @param u Unsigned canonical modulus modulo 1024 to be decompressed. + * + * @return Decompressed value. + */ +static MLK_INLINE int16_t mlk_scalar_decompress_d10(uint16_t u) +__contract__( + requires(0 <= u && u < 1024) + ensures(0 <= return_value && return_value <= (MLKEM_Q - 1)) +) +{ + /* The return value is in 0..MLKEM_Q-1, hence not altered by the + * conversion to int16_t. */ + return (int16_t)((((uint32_t)u * MLKEM_Q) + 512) >> 10); +} + +/* + * The multiplication in this routine will exceed UINT32_MAX + * and wrap around for large values of u. This is expected and required. + */ +#ifdef CBMC +#pragma CPROVER check push +#pragma CPROVER check disable "unsigned-overflow" +#endif + +/** + * Compute round(u * 2**11 / MLKEM_Q) % 2**11. + * + * @spec{Compress_11 from @[FIPS203, Eq (4.7)].} + * + * @reference{Embedded into `polyvec_compress()` in the reference + * implementation @[REF].} + * + * @param u Unsigned canonical modulus modulo MLKEM_Q to be compressed. + * + * @return Compressed value. + */ +static MLK_INLINE uint16_t mlk_scalar_compress_d11(int16_t u) +__contract__( + requires(0 <= u && u <= MLKEM_Q - 1) + ensures(return_value < (1u << 11)) + ensures(return_value == (((uint32_t)u * (1u << 11) + MLKEM_Q / 2) / MLKEM_Q) % (1 << 11))) +{ + /* Compute as follows: + * ``` + * round(u * 2048 / MLKEM_Q) + * = round(u * 2048 * (2^33 / MLKEM_Q) / 2^33) + * ~= round(u * 2048 * round(2^33 / MLKEM_Q) / 2^33) + * ``` + */ + /* check-magic: 5284526080 == 2^11 * round(2^33 / MLKEM_Q) */ + uint64_t d0 = (uint64_t)u * 5284526080; + d0 = (d0 + ((uint64_t)1u << 32)) >> 33; /* round(d0/2^33) */ + return (d0 & 0x7FF); +} +#ifdef CBMC +#pragma CPROVER check pop +#endif + +/** + * Compute round(u * MLKEM_Q / 2048). + * + * @spec{Decompress_11 from @[FIPS203, Eq (4.8)].} + * + * @reference{Embedded into `polyvec_decompress()` in the reference + * implementation @[REF].} + * + * @param u Unsigned canonical modulus modulo 2048 to be decompressed. + * + * @return Decompressed value. + */ +static MLK_INLINE int16_t mlk_scalar_decompress_d11(uint16_t u) +__contract__( + requires(0 <= u && u < 2048) + ensures(0 <= return_value && return_value <= (MLKEM_Q - 1)) +) +{ + /* The return value is in 0..MLKEM_Q-1, hence not altered by the + * conversion to int16_t. */ + return (int16_t)((((uint32_t)u * MLKEM_Q) + 1024) >> 11); +} + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || (MLKEM_K == 2 || MLKEM_K == 3) +#define mlk_poly_compress_d4 MLK_NAMESPACE(poly_compress_d4) +/** + * Compression (4 bits) and subsequent serialization of a polynomial. + * + * @spec{Implements `ByteEncode_4 (Compress_4 (a))`: ByteEncode_d + * @[FIPS203, Algorithm 5], Compress_d @[FIPS203, Eq (4.7)], extended to + * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays]. + * `ByteEncode_{d_v} (Compress_{d_v} (v))` appears in @[FIPS203, Algorithm + * 14 (K-PKE.Encrypt), L23], where `d_v=4` for ML-KEM-{512,768} @[FIPS203, + * Table 2].} + * + * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D4 + * bytes). + * @param[in] a Input polynomial. Coefficients must be unsigned canonical, + * i.e. in [0,1,..,MLKEM_Q-1]. + */ +MLK_INTERNAL_API +void mlk_poly_compress_d4(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D4], + const mlk_poly *a); + +#define mlk_poly_compress_d10 MLK_NAMESPACE(poly_compress_d10) +/** + * Compression (10 bits) and subsequent serialization of a polynomial. + * + * @spec{Implements `ByteEncode_10 (Compress_10 (a))`: ByteEncode_d + * @[FIPS203, Algorithm 5], Compress_d @[FIPS203, Eq (4.7)], extended to + * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays]. + * `ByteEncode_{d_u} (Compress_{d_u} (u))` appears in @[FIPS203, Algorithm + * 14 (K-PKE.Encrypt), L22], where `d_u=10` for ML-KEM-{512,768} @[FIPS203, + * Table 2].} + * + * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D10 + * bytes). + * @param[in] a Input polynomial. Coefficients must be unsigned canonical, + * i.e. in [0,1,..,MLKEM_Q-1]. + */ +MLK_INTERNAL_API +void mlk_poly_compress_d10(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D10], + const mlk_poly *a); + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_poly_decompress_d4 MLK_NAMESPACE(poly_decompress_d4) +/** + * De-serialization and subsequent decompression (4 bits) of a polynomial; + * approximate inverse of mlk_poly_compress_d4. + * + * Upon return, the coefficients of the output polynomial are + * unsigned-canonical (non-negative and smaller than MLKEM_Q). + * + * @spec{Implements `Decompress_4 (ByteDecode_4 (a))`: ByteDecode_d + * @[FIPS203, Algorithm 6], Decompress_d @[FIPS203, Eq (4.8)], extended to + * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays]. + * `Decompress_{d_v} (ByteDecode_{d_v} (v))` appears in @[FIPS203, Algorithm + * 15 (K-PKE.Decrypt), L4], where `d_v=4` for ML-KEM-{512,768} @[FIPS203, + * Table 2].} + * + * @param[out] r Output polynomial. + * @param[in] a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D4 + * bytes). + */ +MLK_INTERNAL_API +void mlk_poly_decompress_d4(mlk_poly *r, + const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D4]); + +#define mlk_poly_decompress_d10 MLK_NAMESPACE(poly_decompress_d10) +/** + * De-serialization and subsequent decompression (10 bits) of a polynomial; + * approximate inverse of mlk_poly_compress_d10. + * + * Upon return, the coefficients of the output polynomial are + * unsigned-canonical (non-negative and smaller than MLKEM_Q). + * + * @spec{Implements `Decompress_10 (ByteDecode_10 (a))`: ByteDecode_d + * @[FIPS203, Algorithm 6], Decompress_d @[FIPS203, Eq (4.8)], extended to + * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays]. + * `Decompress_{d_u} (ByteDecode_{d_u} (u))` appears in @[FIPS203, Algorithm + * 15 (K-PKE.Decrypt), L3], where `d_u=10` for ML-KEM-{512,768} @[FIPS203, + * Table 2].} + * + * @param[out] r Output polynomial. + * @param[in] a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D10 + * bytes). + */ +MLK_INTERNAL_API +void mlk_poly_decompress_d10(mlk_poly *r, + const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D10]); +#endif /* !MLK_CONFIG_NO_DECAPS_API */ +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 || MLKEM_K == 3 */ + +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4 +#define mlk_poly_compress_d5 MLK_NAMESPACE(poly_compress_d5) +/** + * Compression (5 bits) and subsequent serialization of a polynomial. + * + * @spec{Implements `ByteEncode_5 (Compress_5 (a))`: ByteEncode_d + * @[FIPS203, Algorithm 5], Compress_d @[FIPS203, Eq (4.7)], extended to + * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays]. + * `ByteEncode_{d_v} (Compress_{d_v} (v))` appears in @[FIPS203, Algorithm + * 14 (K-PKE.Encrypt), L23], where `d_v=5` for ML-KEM-1024 @[FIPS203, + * Table 2].} + * + * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D5 + * bytes). + * @param[in] a Input polynomial. Coefficients must be unsigned canonical, + * i.e. in [0,1,..,MLKEM_Q-1]. + */ +MLK_INTERNAL_API +void mlk_poly_compress_d5(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D5], + const mlk_poly *a); + +#define mlk_poly_compress_d11 MLK_NAMESPACE(poly_compress_d11) +/** + * Compression (11 bits) and subsequent serialization of a polynomial. + * + * @spec{`ByteEncode_11 (Compress_11 (a))`: ByteEncode_d @[FIPS203, + * Algorithm 5], Compress_d @[FIPS203, Eq (4.7)], extended to vectors as + * per @[FIPS203, 2.4.8 Applying Algorithms to Arrays]. + * `ByteEncode_{d_u} (Compress_{d_u} (u))` appears in @[FIPS203, Algorithm + * 14 (K-PKE.Encrypt), L22], where `d_u=11` for ML-KEM-1024 @[FIPS203, + * Table 2].} + * + * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D11 + * bytes). + * @param[in] a Input polynomial. Coefficients must be unsigned canonical, + * i.e. in [0,1,..,MLKEM_Q-1]. + */ +MLK_INTERNAL_API +void mlk_poly_compress_d11(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D11], + const mlk_poly *a); + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_poly_decompress_d5 MLK_NAMESPACE(poly_decompress_d5) +/** + * De-serialization and subsequent decompression (5 bits) of a polynomial; + * approximate inverse of mlk_poly_compress_d5. + * + * Upon return, the coefficients of the output polynomial are + * unsigned-canonical (non-negative and smaller than MLKEM_Q). + * + * @spec{Implements `Decompress_5 (ByteDecode_5 (a))`: ByteDecode_d + * @[FIPS203, Algorithm 6], Decompress_d @[FIPS203, Eq (4.8)], extended to + * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays]. + * `Decompress_{d_v} (ByteDecode_{d_v} (v))` appears in @[FIPS203, Algorithm + * 15 (K-PKE.Decrypt), L4], where `d_v=5` for ML-KEM-1024 @[FIPS203, + * Table 2].} + * + * @param[out] r Output polynomial. + * @param[in] a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D5 + * bytes). + */ +MLK_INTERNAL_API +void mlk_poly_decompress_d5(mlk_poly *r, + const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D5]); + +#define mlk_poly_decompress_d11 MLK_NAMESPACE(poly_decompress_d11) +/** + * De-serialization and subsequent decompression (11 bits) of a polynomial; + * approximate inverse of mlk_poly_compress_d11. + * + * Upon return, the coefficients of the output polynomial are + * unsigned-canonical (non-negative and smaller than MLKEM_Q). + * + * @spec{Implements `Decompress_11 (ByteDecode_11 (a))`: ByteDecode_d + * @[FIPS203, Algorithm 6], Decompress_d @[FIPS203, Eq (4.8)], extended to + * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays]. + * `Decompress_{d_u} (ByteDecode_{d_u} (u))` appears in @[FIPS203, Algorithm + * 15 (K-PKE.Decrypt), L3], where `d_u=11` for ML-KEM-1024 @[FIPS203, + * Table 2].} + * + * @param[out] r Output polynomial. + * @param[in] a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D11 + * bytes). + */ +MLK_INTERNAL_API +void mlk_poly_decompress_d11(mlk_poly *r, + const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D11]); +#endif /* !MLK_CONFIG_NO_DECAPS_API */ +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4 */ +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) || !defined(MLK_CONFIG_NO_ENCAPS_API) +#define mlk_poly_tobytes MLK_NAMESPACE(poly_tobytes) +/** + * Serialization of a polynomial. Signed coefficients are converted to + * unsigned form before serialization. + * + * @spec{Implements ByteEncode_12 @[FIPS203, Algorithm 5]. Extended to + * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays].} + * + * @param[out] r Output byte array (of MLKEM_POLYBYTES bytes). + * @param[in] a Input polynomial, with each coefficient in the range + * [0,1,..,MLKEM_Q-1]. + */ +MLK_INTERNAL_API +void mlk_poly_tobytes(uint8_t r[MLKEM_POLYBYTES], const mlk_poly *a) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYBYTES)) + requires(memory_no_alias(a, sizeof(mlk_poly))) + requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYBYTES)) +); +#endif /* !MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API */ + + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_poly_frombytes MLK_NAMESPACE(poly_frombytes) +/** + * De-serialization of a polynomial. + * + * @spec{Implements ByteDecode_12 @[FIPS203, Algorithm 6]. Extended to + * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays].} + * + * @param[out] r Output polynomial, with each coefficient unsigned and in + * the range 0..4095. + * @param[in] a Input byte array (of MLKEM_POLYBYTES bytes). + */ +MLK_INTERNAL_API +void mlk_poly_frombytes(mlk_poly *r, const uint8_t a[MLKEM_POLYBYTES]) +__contract__( + requires(memory_no_alias(a, MLKEM_POLYBYTES)) + requires(memory_no_alias(r, sizeof(mlk_poly))) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT)) +); +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_poly_frommsg MLK_NAMESPACE(poly_frommsg) +/** + * Convert a 32-byte message to a polynomial. + * + * @spec{Implements `Decompress_1 (ByteDecode_1 (a))`: ByteDecode_d + * @[FIPS203, Algorithm 6], Decompress_d @[FIPS203, Eq (4.8)], extended to + * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays]. + * `Decompress_1 (ByteDecode_1 (w))` appears in @[FIPS203, Algorithm 15 + * (K-PKE.Encrypt), L20].} + * + * @param[out] r Output polynomial. + * @param[in] msg Input message. + */ +MLK_INTERNAL_API +void mlk_poly_frommsg(mlk_poly *r, const uint8_t msg[MLKEM_INDCPA_MSGBYTES]) +__contract__( + requires(memory_no_alias(msg, MLKEM_INDCPA_MSGBYTES)) + requires(memory_no_alias(r, sizeof(mlk_poly))) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) +); +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_poly_tomsg MLK_NAMESPACE(poly_tomsg) +/** + * Convert a polynomial to a 32-byte message. + * + * @spec{Implements `ByteEncode_1 (Compress_1 (a))`: ByteEncode_d + * @[FIPS203, Algorithm 5], Compress_d @[FIPS203, Eq (4.7)], extended to + * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays]. + * `ByteEncode_1 (Compress_1 (w))` appears in @[FIPS203, Algorithm 14 + * (K-PKE.Decrypt), L7].} + * + * @param[out] msg Output message. + * @param[in] r Input polynomial. Coefficients must be unsigned canonical. + */ +MLK_INTERNAL_API +void mlk_poly_tomsg(uint8_t msg[MLKEM_INDCPA_MSGBYTES], const mlk_poly *r) +__contract__( + requires(memory_no_alias(msg, MLKEM_INDCPA_MSGBYTES)) + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(msg, MLKEM_INDCPA_MSGBYTES)) +); +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +#endif /* !MLK_COMPRESS_H */ diff --git a/cbits/mlkem/src/context.h b/cbits/mlkem/src/context.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/context.h @@ -0,0 +1,51 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLK_CONTEXT_H +#define MLK_CONTEXT_H + +/* This header is included by common.h once the configuration has been pulled + * in; it is not meant to be included directly. */ + +/* + * If the integration wants to provide a context parameter for use in + * platform-specific hooks, then it should define this parameter. + * + * The MLK_CONTEXT_PARAMETERS_n macros are intended to be used with macros + * defining the function names and expand to either pass or discard the context + * argument as required by the current build. If there is no context parameter + * requested then these are removed from the prototypes and from all calls. + */ +#ifdef MLK_CONFIG_CONTEXT_PARAMETER +#define MLK_CONTEXT_PARAMETERS_0(context) (context) +#define MLK_CONTEXT_PARAMETERS_1(arg0, context) (arg0, context) +#define MLK_CONTEXT_PARAMETERS_2(arg0, arg1, context) (arg0, arg1, context) +#define MLK_CONTEXT_PARAMETERS_3(arg0, arg1, arg2, context) \ + (arg0, arg1, arg2, context) +#define MLK_CONTEXT_PARAMETERS_4(arg0, arg1, arg2, arg3, context) \ + (arg0, arg1, arg2, arg3, context) +#else /* MLK_CONFIG_CONTEXT_PARAMETER */ +#define MLK_CONTEXT_PARAMETERS_0(context) () +#define MLK_CONTEXT_PARAMETERS_1(arg0, context) (arg0) +#define MLK_CONTEXT_PARAMETERS_2(arg0, arg1, context) (arg0, arg1) +#define MLK_CONTEXT_PARAMETERS_3(arg0, arg1, arg2, context) (arg0, arg1, arg2) +#define MLK_CONTEXT_PARAMETERS_4(arg0, arg1, arg2, arg3, context) \ + (arg0, arg1, arg2, arg3) +#endif /* !MLK_CONFIG_CONTEXT_PARAMETER */ + +/* Consume a context parameter carried only for the integration's benefit, + * avoiding -Wunused-parameter; expands to nothing when no context is + * configured. */ +#if defined(MLK_CONFIG_CONTEXT_PARAMETER) +#define MLK_CONTEXT_UNUSED(context) ((void)(context)) +#else +#define MLK_CONTEXT_UNUSED(context) ((void)0) +#endif + +#if defined(MLK_CONFIG_CONTEXT_PARAMETER_TYPE) != \ + defined(MLK_CONFIG_CONTEXT_PARAMETER) +#error MLK_CONFIG_CONTEXT_PARAMETER_TYPE must be defined if and only if MLK_CONFIG_CONTEXT_PARAMETER is defined +#endif + +#endif /* !MLK_CONTEXT_H */ diff --git a/cbits/mlkem/src/debug.c b/cbits/mlkem/src/debug.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/debug.c @@ -0,0 +1,64 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* NOTE: You can remove this file unless you compile with MLKEM_DEBUG. */ + +#include "common.h" + +#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && defined(MLKEM_DEBUG) + + +#include +#include +#include "debug.h" + +#define MLK_DEBUG_ERROR_HEADER "[ERROR:%s:%04d] " + +void mlk_debug_check_assert(const char *file, int line, const int val) +{ + if (val == 0) + { + fprintf(stderr, MLK_DEBUG_ERROR_HEADER "Assertion failed (value %d)\n", + file, line, val); + exit(1); + } +} + +void mlk_debug_check_bounds(const char *file, int line, const int16_t *ptr, + unsigned len, int lower_bound_exclusive, + int upper_bound_exclusive) +{ + int err = 0; + unsigned i; + for (i = 0; i < len; i++) + { + int16_t val = ptr[i]; + if (!(val > lower_bound_exclusive && val < upper_bound_exclusive)) + { + fprintf( + stderr, + MLK_DEBUG_ERROR_HEADER + "Bounds assertion failed: Index %u, value %d out of bounds (%d,%d)\n", + file, line, i, (int)val, lower_bound_exclusive, + upper_bound_exclusive); + err = 1; + } + } + + if (err == 1) + { + exit(1); + } +} + +#else /* !MLK_CONFIG_MULTILEVEL_NO_SHARED && MLKEM_DEBUG */ + +MLK_EMPTY_CU(debug) + +#endif /* !(!MLK_CONFIG_MULTILEVEL_NO_SHARED && MLKEM_DEBUG) */ + +/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. + * Don't modify by hand -- this is auto-generated by scripts/autogen. */ +#undef MLK_DEBUG_ERROR_HEADER diff --git a/cbits/mlkem/src/debug.h b/cbits/mlkem/src/debug.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/debug.h @@ -0,0 +1,121 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLK_DEBUG_H +#define MLK_DEBUG_H +#include "common.h" + +#if defined(MLKEM_DEBUG) + +/** + * Check debug assertion. + * + * Prints an error message to stderr and calls exit(1) on failure. + * + * @param[in] file Filename. + * @param line Line number. + * @param val Value asserted to be non-zero. + */ +#define mlk_debug_check_assert MLK_NAMESPACE(mlkem_debug_assert) +void mlk_debug_check_assert(const char *file, int line, const int val); + +/** + * Check whether values in an array of int16_t are within specified bounds. + * + * Prints an error message to stderr and calls exit(1) on failure. + * + * @param[in] file Filename. + * @param line Line number. + * @param[in] ptr Base of array to be checked. + * @param len Number of int16_t in @p ptr. + * @param lower_bound_exclusive Exclusive lower bound. + * @param upper_bound_exclusive Exclusive upper bound. + */ +#define mlk_debug_check_bounds MLK_NAMESPACE(mlkem_debug_check_bounds) +void mlk_debug_check_bounds(const char *file, int line, const int16_t *ptr, + unsigned len, int lower_bound_exclusive, + int upper_bound_exclusive); + +/* Check assertion, calling exit() upon failure + * + * val: Value that's asserted to be non-zero + */ +#define mlk_assert(val) mlk_debug_check_assert(__FILE__, __LINE__, (val)) + +/* Check bounds in array of int16_t's + * ptr: Base of int16_t array; will be explicitly cast to int16_t*, + * so you may pass a byte-compatible type such as mlk_poly or mlk_polyvec. + * len: Number of int16_t in array + * value_lb: Inclusive lower value bound + * value_ub: Exclusive upper value bound */ +#define mlk_assert_bound(ptr, len, value_lb, value_ub) \ + mlk_debug_check_bounds(__FILE__, __LINE__, (const int16_t *)(ptr), (len), \ + (value_lb) - 1, (value_ub)) + +/* Check absolute bounds in array of int16_t's + * ptr: Base of array, expression of type int16_t* + * len: Number of int16_t in array + * value_abs_bd: Exclusive absolute upper bound */ +#define mlk_assert_abs_bound(ptr, len, value_abs_bd) \ + mlk_assert_bound((ptr), (len), (-(value_abs_bd) + 1), (value_abs_bd)) + +/* Version of bounds assertions for 2-dimensional arrays */ +#define mlk_assert_bound_2d(ptr, len0, len1, value_lb, value_ub) \ + mlk_assert_bound((ptr), ((len0) * (len1)), (value_lb), (value_ub)) + +#define mlk_assert_abs_bound_2d(ptr, len0, len1, value_abs_bd) \ + mlk_assert_abs_bound((ptr), ((len0) * (len1)), (value_abs_bd)) + +/* When running CBMC, convert debug assertions into proof obligations */ +#elif defined(CBMC) +#include "cbmc.h" + +#define mlk_assert(val) cassert(val) + +#define mlk_assert_bound(ptr, len, value_lb, value_ub) \ + cassert(array_bound(((int16_t *)(ptr)), 0, (len), (value_lb), (value_ub))) + +#define mlk_assert_abs_bound(ptr, len, value_abs_bd) \ + cassert(array_abs_bound(((int16_t *)(ptr)), 0, (len), (value_abs_bd))) + +/* Because of https://github.com/diffblue/cbmc/issues/8570, we can't + * just use a single flattened array_bound(...) here. */ +#define mlk_assert_bound_2d(ptr, M, N, value_lb, value_ub) \ + cassert(forall(kN, 0, (M), \ + array_bound(&((int16_t (*)[(N)])(ptr))[kN][0], 0, (N), \ + (value_lb), (value_ub)))) + +#define mlk_assert_abs_bound_2d(ptr, M, N, value_abs_bd) \ + cassert(forall(kN, 0, (M), \ + array_abs_bound(&((int16_t (*)[(N)])(ptr))[kN][0], 0, (N), \ + (value_abs_bd)))) + +#else /* !MLKEM_DEBUG && CBMC */ + +#define mlk_assert(val) \ + do \ + { \ + } while (0) +#define mlk_assert_bound(ptr, len, value_lb, value_ub) \ + do \ + { \ + } while (0) +#define mlk_assert_abs_bound(ptr, len, value_abs_bd) \ + do \ + { \ + } while (0) + +#define mlk_assert_bound_2d(ptr, len0, len1, value_lb, value_ub) \ + do \ + { \ + } while (0) + +#define mlk_assert_abs_bound_2d(ptr, len0, len1, value_abs_bd) \ + do \ + { \ + } while (0) + + +#endif /* !MLKEM_DEBUG && !CBMC */ +#endif /* !MLK_DEBUG_H */ diff --git a/cbits/mlkem/src/fips202/fips202.c b/cbits/mlkem/src/fips202/fips202.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/fips202.c @@ -0,0 +1,249 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + * + * - [mupq] + * Common files for pqm4, pqm3, pqriscv + * Kannwischer, Petri, Rijneveld, Schwabe, Stoffelen + * https://github.com/mupq/mupq + * + * - [supercop] + * SUPERCOP benchmarking framework + * Daniel J. Bernstein + * http://bench.cr.yp.to/supercop.html + * + * - [tweetfips] + * 'tweetfips202' FIPS202 implementation + * Van Assche, Bernstein, Schwabe + * https://keccak.team/2015/tweetfips202.html + */ + +/* Based on the CC0 implementation from @[mupq] and the public domain + * implementation @[supercop, crypto_hash/keccakc512/simple/] + * by Ronny Van Keer, and the public domain @[tweetfips] implementation. */ + +#include "../common.h" +#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + + +#include "../verify.h" +#include "fips202.h" +#include "keccakf1600.h" + +/** + * Absorb step of Keccak; non-incremental, starts by zeroeing the state. + * + * @warning Must only be called once. + * + * @param[out] s Pointer to (uninitialized) output Keccak state. + * @param r Rate in bytes (e.g., 168 for SHAKE128). + * @param[in] m Input to be absorbed into @p s. + * @param mlen Length of input in bytes. + * @param p Domain-separation byte for different Keccak-derived + * functions. + */ +static void mlk_keccak_absorb_once(uint64_t *s, unsigned r, const uint8_t *m, + size_t mlen, uint8_t p) +__contract__( + requires(mlen <= MLK_MAX_BUFFER_SIZE) + requires(r > 0) + requires(r <= sizeof(uint64_t) * MLK_KECCAK_LANES) + requires(memory_no_alias(s, sizeof(uint64_t) * MLK_KECCAK_LANES)) + requires(memory_no_alias(m, mlen)) + assigns(memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES))) +{ + /* Initialize state */ + size_t i; + for (i = 0; i < 25; ++i) + __loop__(invariant(i <= 25) + decreases(25 - i)) + { + s[i] = 0; + } + + while (mlen >= r) + __loop__( + assigns(mlen, m, memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES)) + invariant(mlen <= loop_entry(mlen)) + invariant(m == loop_entry(m) + (loop_entry(mlen) - mlen)) + decreases(mlen)) + { + mlk_keccakf1600_xor_bytes(s, m, 0, r); + mlk_keccakf1600_permute(s); + mlen -= r; + m += r; + } + + /* At this point, mlen < r, so the truncations to unsigned are safe below. */ + + if (mlen > 0) + { + mlk_keccakf1600_xor_bytes(s, m, 0, (unsigned int)mlen); + } + + if (mlen == r - 1) + { + p |= 128; + mlk_keccakf1600_xor_bytes(s, &p, (unsigned int)mlen, 1); + } + else + { + mlk_keccakf1600_xor_bytes(s, &p, (unsigned int)mlen, 1); + p = 128; + mlk_keccakf1600_xor_bytes(s, &p, r - 1, 1); + } +} + +/** + * Block-level Keccak squeeze. + * + * @param[out] h Output bytes. + * @param nblocks Number of blocks to be squeezed. + * @param[in,out] s Input/output state. + * @param r Rate in bytes (e.g., 168 for SHAKE128). + */ +static void mlk_keccak_squeezeblocks(uint8_t *h, size_t nblocks, uint64_t *s, + unsigned r) +__contract__( + requires(r <= sizeof(uint64_t) * MLK_KECCAK_LANES) + requires(nblocks <= 8 /* somewhat arbitrary bound */) + requires(memory_no_alias(s, sizeof(uint64_t) * MLK_KECCAK_LANES)) + requires(memory_no_alias(h, nblocks * r)) + assigns(memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES)) + assigns(memory_slice(h, nblocks * r))) +{ + while (nblocks > 0) + __loop__( + assigns(h, nblocks, + memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES), + memory_slice(h, nblocks * r)) + invariant(nblocks <= loop_entry(nblocks) && + h == loop_entry(h) + r * (loop_entry(nblocks) - nblocks)) + decreases(nblocks)) + { + mlk_keccakf1600_permute(s); + mlk_keccakf1600_extract_bytes(s, h, 0, r); + h += r; + nblocks--; + } +} + +/** + * Keccak squeeze; can be called on byte-level. + * + * @warning Must only be called once. + * + * @param[out] h Output bytes. + * @param outlen Number of bytes to be squeezed. + * @param[in,out] s Keccak state. + * @param r Rate in bytes (e.g., 168 for SHAKE128). + */ +static void mlk_keccak_squeeze_once(uint8_t *h, size_t outlen, uint64_t *s, + unsigned r) +__contract__( + requires(outlen <= MLK_MAX_BUFFER_SIZE) + requires(r > 0) + requires(r <= sizeof(uint64_t) * MLK_KECCAK_LANES) + requires(memory_no_alias(s, sizeof(uint64_t) * MLK_KECCAK_LANES)) + requires(memory_no_alias(h, outlen)) + assigns(memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES)) + assigns(memory_slice(h, outlen))) +{ + size_t len; + while (outlen > 0) + __loop__( + assigns(len, h, outlen, + memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES), + memory_slice(h, outlen)) + invariant(outlen <= loop_entry(outlen) && + h == loop_entry(h) + (loop_entry(outlen) - outlen)) + decreases(outlen)) + { + mlk_keccakf1600_permute(s); + + if (outlen < r) + { + len = outlen; + } + else + { + len = r; + } + mlk_keccakf1600_extract_bytes(s, h, 0, (unsigned int)len); + h += len; + outlen -= len; + } +} + +void mlk_shake128_absorb_once(mlk_shake128ctx *state, const uint8_t *input, + size_t inlen) +{ + mlk_keccak_absorb_once(state->ctx, SHAKE128_RATE, input, inlen, 0x1F); +} + +void mlk_shake128_squeezeblocks(uint8_t *output, size_t nblocks, + mlk_shake128ctx *state) +{ + mlk_keccak_squeezeblocks(output, nblocks, state->ctx, SHAKE128_RATE); +} + +void mlk_shake128_init(mlk_shake128ctx *state) { (void)state; } +void mlk_shake128_release(mlk_shake128ctx *state) +{ + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + mlk_zeroize(state, sizeof(mlk_shake128ctx)); +} + +typedef mlk_shake128ctx mlk_shake256ctx; +void mlk_shake256(uint8_t *output, size_t outlen, const uint8_t *input, + size_t inlen) +{ + mlk_shake256ctx state; + /* Absorb input */ + mlk_keccak_absorb_once(state.ctx, SHAKE256_RATE, input, inlen, 0x1F); + /* Squeeze output */ + mlk_keccak_squeeze_once(output, outlen, state.ctx, SHAKE256_RATE); + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + mlk_zeroize(&state, sizeof(state)); +} + +void mlk_sha3_256(uint8_t *output, const uint8_t *input, size_t inlen) +{ + uint64_t ctx[25]; + /* Absorb input */ + mlk_keccak_absorb_once(ctx, SHA3_256_RATE, input, inlen, 0x06); + /* Squeeze output */ + mlk_keccak_squeeze_once(output, 32, ctx, SHA3_256_RATE); + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + mlk_zeroize(ctx, sizeof(ctx)); +} + +void mlk_sha3_512(uint8_t *output, const uint8_t *input, size_t inlen) +{ + uint64_t ctx[25]; + /* Absorb input */ + mlk_keccak_absorb_once(ctx, SHA3_512_RATE, input, inlen, 0x06); + /* Squeeze output */ + mlk_keccak_squeeze_once(output, 64, ctx, SHA3_512_RATE); + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + mlk_zeroize(ctx, sizeof(ctx)); +} + +#else /* !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +MLK_EMPTY_CU(fips202) + +#endif /* MLK_CONFIG_MULTILEVEL_NO_SHARED */ diff --git a/cbits/mlkem/src/fips202/fips202.h b/cbits/mlkem/src/fips202/fips202.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/fips202.h @@ -0,0 +1,144 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLK_FIPS202_FIPS202_H +#define MLK_FIPS202_FIPS202_H + +#include "../cbmc.h" +#include "../common.h" + +#define SHAKE128_RATE 168 +#define SHAKE256_RATE 136 +#define SHA3_256_RATE 136 +#define SHA3_384_RATE 104 +#define SHA3_512_RATE 72 + +/** Context for the non-incremental SHAKE128 API. */ +typedef struct +{ + uint64_t ctx[25]; /**< Keccak state. */ +} MLK_ALIGN mlk_shake128ctx; + +#define mlk_shake128_absorb_once MLK_NAMESPACE(shake128_absorb_once) +/** + * One-shot absorb step of the SHAKE128 XOF. + * + * For call-sites (in mlkem-native): + * - This function MUST ONLY be called straight after mlk_shake128_init(). + * - This function MUST ONLY be called once. + * + * Consequently, for providers of custom FIPS202 code to be used with + * mlkem-native: + * - You may assume that the input context is freshly initialized via + * mlk_shake128_init(). + * - You may assume that this function is called exactly once. + * + * @param[in,out] state SHAKE128 context. + * @param[in] input Input to be absorbed into the state. + * @param inlen Length of input in bytes. + */ +void mlk_shake128_absorb_once(mlk_shake128ctx *state, const uint8_t *input, + size_t inlen) +__contract__( + requires(inlen <= MLK_MAX_BUFFER_SIZE) + requires(memory_no_alias(state, sizeof(mlk_shake128ctx))) + requires(memory_no_alias(input, inlen)) + assigns(memory_slice(state, sizeof(mlk_shake128ctx))) +); + +#define mlk_shake128_squeezeblocks MLK_NAMESPACE(shake128_squeezeblocks) +/** + * Squeeze step of SHAKE128 XOF. Squeezes full blocks of SHAKE128_RATE bytes + * each. Modifies the state. Can be called multiple times to keep squeezing, + * i.e., is incremental. + * + * @param[out] output Output blocks. + * @param nblocks Number of blocks to be squeezed (written to output). + * @param[in,out] state Keccak state. + */ +void mlk_shake128_squeezeblocks(uint8_t *output, size_t nblocks, + mlk_shake128ctx *state) +__contract__( + requires(nblocks <= 8 /* somewhat arbitrary bound */) + requires(memory_no_alias(state, sizeof(mlk_shake128ctx))) + requires(memory_no_alias(output, nblocks * SHAKE128_RATE)) + assigns(memory_slice(output, nblocks * SHAKE128_RATE), memory_slice(state, sizeof(mlk_shake128ctx))) +); + +#define mlk_shake128_init MLK_NAMESPACE(shake128_init) +void mlk_shake128_init(mlk_shake128ctx *state); + +#define mlk_shake128_release MLK_NAMESPACE(shake128_release) +void mlk_shake128_release(mlk_shake128ctx *state); + +/* One-stop SHAKE256 call. Aliasing between input and + * output is not permitted */ +#define mlk_shake256 MLK_NAMESPACE(shake256) +/** + * SHAKE256 XOF with non-incremental API. + * + * @param[out] output Output buffer. + * @param outlen Requested output length in bytes. + * @param[in] input Input buffer. + * @param inlen Length of input in bytes. + */ +void mlk_shake256(uint8_t *output, size_t outlen, const uint8_t *input, + size_t inlen) +__contract__( + requires(inlen <= MLK_MAX_BUFFER_SIZE) + requires(outlen <= MLK_MAX_BUFFER_SIZE) + requires(memory_no_alias(input, inlen)) + requires(memory_no_alias(output, outlen)) + assigns(memory_slice(output, outlen)) +); + +/* One-stop SHA3_256 call. Aliasing between input and + * output is not permitted */ +#define SHA3_256_HASHBYTES 32 +#define mlk_sha3_256 MLK_NAMESPACE(sha3_256) +/** + * SHA3-256 with non-incremental API. + * + * @param[out] output Output buffer. + * @param[in] input Input buffer. + * @param inlen Length of input in bytes. + */ +void mlk_sha3_256(uint8_t *output, const uint8_t *input, size_t inlen) +__contract__( + requires(inlen <= MLK_MAX_BUFFER_SIZE) + requires(memory_no_alias(input, inlen)) + requires(memory_no_alias(output, SHA3_256_HASHBYTES)) + assigns(memory_slice(output, SHA3_256_HASHBYTES)) +); + +/* One-stop SHA3_512 call. Aliasing between input and + * output is not permitted */ +#define SHA3_512_HASHBYTES 64 +#define mlk_sha3_512 MLK_NAMESPACE(sha3_512) +/** + * SHA3-512 with non-incremental API. + * + * @param[out] output Output buffer. + * @param[in] input Input buffer. + * @param inlen Length of input in bytes. + */ +void mlk_sha3_512(uint8_t *output, const uint8_t *input, size_t inlen) +__contract__( + requires(inlen <= MLK_MAX_BUFFER_SIZE) + requires(memory_no_alias(input, inlen)) + requires(memory_no_alias(output, SHA3_512_HASHBYTES)) + assigns(memory_slice(output, SHA3_512_HASHBYTES)) +); + +#if !defined(MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202) || \ + !defined(MLK_USE_NATIVE_FIPS202_X4) +/* If you provide your own FIPS-202 implementation where the x4- + * Keccak-f1600-x4 implementation falls back to 4-fold Keccak-f1600, + * set this to gain a small speedup. */ +#define FIPS202_X4_DEFAULT_IMPLEMENTATION +#endif /* !MLK_CONFIG_USE_NATIVE_BACKEND_FIPS202 || !MLK_USE_NATIVE_FIPS202_X4 \ + */ + + +#endif /* !MLK_FIPS202_FIPS202_H */ diff --git a/cbits/mlkem/src/fips202/fips202x4.c b/cbits/mlkem/src/fips202/fips202x4.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/fips202x4.c @@ -0,0 +1,207 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + */ + +#include "../common.h" +#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +#include "../verify.h" +#include "fips202.h" +#include "fips202x4.h" +#include "keccakf1600.h" + +typedef mlk_shake128x4ctx mlk_shake256x4_ctx; + +static void mlk_keccak_absorb_once_x4(uint64_t *s, unsigned r, + const uint8_t *in0, const uint8_t *in1, + const uint8_t *in2, const uint8_t *in3, + size_t inlen, uint8_t p) +__contract__( + requires(inlen <= MLK_MAX_BUFFER_SIZE) + requires(memory_no_alias(s, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY)) + requires(r > 0) + requires(r <= sizeof(uint64_t) * MLK_KECCAK_LANES) + requires(memory_no_alias(in0, inlen)) + requires(memory_no_alias(in1, inlen)) + requires(memory_no_alias(in2, inlen)) + requires(memory_no_alias(in3, inlen)) + assigns(memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY))) +{ + while (inlen >= r) + __loop__( + assigns(inlen, in0, in1, in2, in3, memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY)) + invariant(inlen <= loop_entry(inlen)) + invariant(in0 == loop_entry(in0) + (loop_entry(inlen) - inlen)) + invariant(in1 == loop_entry(in1) + (loop_entry(inlen) - inlen)) + invariant(in2 == loop_entry(in2) + (loop_entry(inlen) - inlen)) + invariant(in3 == loop_entry(in3) + (loop_entry(inlen) - inlen)) + decreases(inlen)) + { + mlk_keccakf1600x4_xor_bytes(s, in0, in1, in2, in3, 0, r); + mlk_keccakf1600x4_permute(s); + + in0 += r; + in1 += r; + in2 += r; + in3 += r; + inlen -= r; + } + + /* At this point, inlen < r, so the truncations to unsigned are safe below. */ + + if (inlen > 0) + { + mlk_keccakf1600x4_xor_bytes(s, in0, in1, in2, in3, 0, (unsigned int)inlen); + } + + if (inlen == r - 1) + { + p |= 128; + mlk_keccakf1600x4_xor_bytes(s, &p, &p, &p, &p, (unsigned int)inlen, 1); + } + else + { + mlk_keccakf1600x4_xor_bytes(s, &p, &p, &p, &p, (unsigned int)inlen, 1); + p = 128; + mlk_keccakf1600x4_xor_bytes(s, &p, &p, &p, &p, r - 1, 1); + } +} + +static void mlk_keccak_squeezeblocks_x4(uint8_t *out0, uint8_t *out1, + uint8_t *out2, uint8_t *out3, + size_t nblocks, uint64_t *s, unsigned r) +__contract__( + requires(r <= sizeof(uint64_t) * MLK_KECCAK_LANES) + requires(r == SHAKE128_RATE || r == SHAKE256_RATE) + requires(nblocks <= (MLK_MAX_BUFFER_SIZE / SHAKE256_RATE)) + requires(memory_no_alias(s, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY)) + requires(memory_no_alias(out0, nblocks * r)) + requires(memory_no_alias(out1, nblocks * r)) + requires(memory_no_alias(out2, nblocks * r)) + requires(memory_no_alias(out3, nblocks * r)) + assigns(memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY)) + assigns(memory_slice(out0, nblocks * r)) + assigns(memory_slice(out1, nblocks * r)) + assigns(memory_slice(out2, nblocks * r)) + assigns(memory_slice(out3, nblocks * r))) +{ + size_t current_offset = 0; + while (nblocks > 0) + __loop__( + assigns(nblocks, current_offset, + memory_slice(s, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY), + memory_slice(out0, nblocks * r), + memory_slice(out1, nblocks * r), + memory_slice(out2, nblocks * r), + memory_slice(out3, nblocks * r)) + invariant(nblocks <= loop_entry(nblocks)) + invariant(current_offset == (loop_entry(nblocks) - nblocks) * r) + decreases(nblocks)) + { + mlk_keccakf1600x4_permute(s); + mlk_keccakf1600x4_extract_bytes( + s, &out0[current_offset], &out1[current_offset], &out2[current_offset], + &out3[current_offset], 0, r); + current_offset += r; + nblocks--; + } +} + +void mlk_shake128x4_absorb_once(mlk_shake128x4ctx *state, const uint8_t *in0, + const uint8_t *in1, const uint8_t *in2, + const uint8_t *in3, size_t inlen) +{ + mlk_memset(state, 0, sizeof(mlk_shake128x4ctx)); + mlk_keccak_absorb_once_x4(state->ctx, SHAKE128_RATE, in0, in1, in2, in3, + inlen, 0x1F); +} + +void mlk_shake128x4_squeezeblocks(uint8_t *out0, uint8_t *out1, uint8_t *out2, + uint8_t *out3, size_t nblocks, + mlk_shake128x4ctx *state) +{ + mlk_keccak_squeezeblocks_x4(out0, out1, out2, out3, nblocks, state->ctx, + SHAKE128_RATE); +} + +void mlk_shake128x4_init(mlk_shake128x4ctx *state) { (void)state; } +void mlk_shake128x4_release(mlk_shake128x4ctx *state) +{ + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + mlk_zeroize(state, sizeof(mlk_shake128x4ctx)); +} + +static void mlk_shake256x4_absorb_once(mlk_shake256x4_ctx *state, + const uint8_t *in0, const uint8_t *in1, + const uint8_t *in2, const uint8_t *in3, + size_t inlen) +{ + mlk_memset(state, 0, sizeof(mlk_shake128x4ctx)); + mlk_keccak_absorb_once_x4(state->ctx, SHAKE256_RATE, in0, in1, in2, in3, + inlen, 0x1F); +} + +static void mlk_shake256x4_squeezeblocks(uint8_t *out0, uint8_t *out1, + uint8_t *out2, uint8_t *out3, + size_t nblocks, + mlk_shake256x4_ctx *state) +{ + mlk_keccak_squeezeblocks_x4(out0, out1, out2, out3, nblocks, state->ctx, + SHAKE256_RATE); +} + +void mlk_shake256x4(uint8_t *out0, uint8_t *out1, uint8_t *out2, uint8_t *out3, + size_t outlen, const uint8_t *in0, const uint8_t *in1, + const uint8_t *in2, const uint8_t *in3, size_t inlen) +{ + mlk_shake256x4_ctx statex; + size_t nblocks = outlen / SHAKE256_RATE; + uint8_t tmp0[SHAKE256_RATE]; + uint8_t tmp1[SHAKE256_RATE]; + uint8_t tmp2[SHAKE256_RATE]; + uint8_t tmp3[SHAKE256_RATE]; + + mlk_shake256x4_absorb_once(&statex, in0, in1, in2, in3, inlen); + mlk_shake256x4_squeezeblocks(out0, out1, out2, out3, nblocks, &statex); + + out0 += nblocks * SHAKE256_RATE; + out1 += nblocks * SHAKE256_RATE; + out2 += nblocks * SHAKE256_RATE; + out3 += nblocks * SHAKE256_RATE; + + outlen -= nblocks * SHAKE256_RATE; + + if (outlen) + { + mlk_shake256x4_squeezeblocks(tmp0, tmp1, tmp2, tmp3, 1, &statex); + mlk_memcpy(out0, tmp0, outlen); + mlk_memcpy(out1, tmp1, outlen); + mlk_memcpy(out2, tmp2, outlen); + mlk_memcpy(out3, tmp3, outlen); + } + + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + mlk_zeroize(&statex, sizeof(statex)); + mlk_zeroize(tmp0, sizeof(tmp0)); + mlk_zeroize(tmp1, sizeof(tmp1)); + mlk_zeroize(tmp2, sizeof(tmp2)); + mlk_zeroize(tmp3, sizeof(tmp3)); +} + +#else /* !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +MLK_EMPTY_CU(fips202x4) + +#endif /* MLK_CONFIG_MULTILEVEL_NO_SHARED */ diff --git a/cbits/mlkem/src/fips202/fips202x4.h b/cbits/mlkem/src/fips202/fips202x4.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/fips202x4.h @@ -0,0 +1,81 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLK_FIPS202_FIPS202X4_H +#define MLK_FIPS202_FIPS202X4_H + + +#include "../cbmc.h" +#include "../common.h" + +#include "fips202.h" +#include "keccakf1600.h" + +/** Context for the non-incremental 4-way SHAKE128 API. */ +typedef struct +{ + uint64_t ctx[MLK_KECCAK_LANES * + MLK_KECCAK_WAY]; /**< 4-way Keccak state, stored sequentially. */ +} MLK_ALIGN mlk_shake128x4ctx; + +#define mlk_shake128x4_absorb_once MLK_NAMESPACE(shake128x4_absorb_once) +void mlk_shake128x4_absorb_once(mlk_shake128x4ctx *state, const uint8_t *in0, + const uint8_t *in1, const uint8_t *in2, + const uint8_t *in3, size_t inlen) +__contract__( + requires(inlen <= MLK_MAX_BUFFER_SIZE) + requires(memory_no_alias(state, sizeof(mlk_shake128x4ctx))) + requires(memory_no_alias(in0, inlen)) + requires(memory_no_alias(in1, inlen)) + requires(memory_no_alias(in2, inlen)) + requires(memory_no_alias(in3, inlen)) + assigns(memory_slice(state, sizeof(mlk_shake128x4ctx))) +); + +#define mlk_shake128x4_squeezeblocks MLK_NAMESPACE(shake128x4_squeezeblocks) +void mlk_shake128x4_squeezeblocks(uint8_t *out0, uint8_t *out1, uint8_t *out2, + uint8_t *out3, size_t nblocks, + mlk_shake128x4ctx *state) +__contract__( + requires(nblocks <= 8 /* somewhat arbitrary bound */) + requires(memory_no_alias(state, sizeof(mlk_shake128x4ctx))) + requires(memory_no_alias(out0, nblocks * SHAKE128_RATE)) + requires(memory_no_alias(out1, nblocks * SHAKE128_RATE)) + requires(memory_no_alias(out2, nblocks * SHAKE128_RATE)) + requires(memory_no_alias(out3, nblocks * SHAKE128_RATE)) + assigns(memory_slice(out0, nblocks * SHAKE128_RATE), + memory_slice(out1, nblocks * SHAKE128_RATE), + memory_slice(out2, nblocks * SHAKE128_RATE), + memory_slice(out3, nblocks * SHAKE128_RATE), + memory_slice(state, sizeof(mlk_shake128x4ctx))) +); + +#define mlk_shake128x4_init MLK_NAMESPACE(shake128x4_init) +void mlk_shake128x4_init(mlk_shake128x4ctx *state); + +#define mlk_shake128x4_release MLK_NAMESPACE(shake128x4_release) +void mlk_shake128x4_release(mlk_shake128x4ctx *state); + +#define mlk_shake256x4 MLK_NAMESPACE(shake256x4) +void mlk_shake256x4(uint8_t *out0, uint8_t *out1, uint8_t *out2, uint8_t *out3, + size_t outlen, const uint8_t *in0, const uint8_t *in1, + const uint8_t *in2, const uint8_t *in3, size_t inlen) +__contract__( + requires(inlen <= MLK_MAX_BUFFER_SIZE) + requires(outlen <= MLK_MAX_BUFFER_SIZE) + requires(memory_no_alias(in0, inlen)) + requires(memory_no_alias(in1, inlen)) + requires(memory_no_alias(in2, inlen)) + requires(memory_no_alias(in3, inlen)) + requires(memory_no_alias(out0, outlen)) + requires(memory_no_alias(out1, outlen)) + requires(memory_no_alias(out2, outlen)) + requires(memory_no_alias(out3, outlen)) + assigns(memory_slice(out0, outlen)) + assigns(memory_slice(out1, outlen)) + assigns(memory_slice(out2, outlen)) + assigns(memory_slice(out3, outlen)) +); + +#endif /* !MLK_FIPS202_FIPS202X4_H */ diff --git a/cbits/mlkem/src/fips202/keccakf1600.c b/cbits/mlkem/src/fips202/keccakf1600.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/keccakf1600.c @@ -0,0 +1,499 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [mupq] + * Common files for pqm4, pqm3, pqriscv + * Kannwischer, Petri, Rijneveld, Schwabe, Stoffelen + * https://github.com/mupq/mupq + * + * - [supercop] + * SUPERCOP benchmarking framework + * Daniel J. Bernstein + * http://bench.cr.yp.to/supercop.html + * + * - [tweetfips] + * 'tweetfips202' FIPS202 implementation + * Van Assche, Bernstein, Schwabe + * https://keccak.team/2015/tweetfips202.html + */ + +/* Based on the CC0 implementation from @[mupq] and the public domain + * implementation @[supercop, crypto_hash/keccakc512/simple/] + * by Ronny Van Keer, and the public domain @[tweetfips] implementation. */ + + +#include "keccakf1600.h" +#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +#define MLK_KECCAK_NROUNDS 24 +#define MLK_KECCAK_ROL(a, offset) (((a) << (offset)) ^ ((a) >> (64 - (offset)))) + +void mlk_keccakf1600_extract_bytes(uint64_t *state, unsigned char *data, + unsigned offset, unsigned length) +{ + unsigned i; +#if defined(MLK_SYS_LITTLE_ENDIAN) + uint8_t *state_ptr = (uint8_t *)state + offset; + for (i = 0; i < length; i++) + __loop__(invariant(i <= length) + decreases(length - i)) + { + data[i] = state_ptr[i]; + } +#else /* MLK_SYS_LITTLE_ENDIAN */ + /* Portable version */ + for (i = 0; i < length; i++) + __loop__(invariant(i <= length) + decreases(length - i)) + { + data[i] = (state[(offset + i) >> 3] >> (8 * ((offset + i) & 0x07))) & 0xFF; + } +#endif /* !MLK_SYS_LITTLE_ENDIAN */ +} + +void mlk_keccakf1600_xor_bytes(uint64_t *state, const unsigned char *data, + unsigned offset, unsigned length) +{ + unsigned i; +#if defined(MLK_SYS_LITTLE_ENDIAN) + uint8_t *state_ptr = (uint8_t *)state + offset; + for (i = 0; i < length; i++) + __loop__(invariant(i <= length) + decreases(length - i)) + { + state_ptr[i] ^= data[i]; + } +#else /* MLK_SYS_LITTLE_ENDIAN */ + /* Portable version */ + for (i = 0; i < length; i++) + __loop__(invariant(i <= length) + decreases(length - i)) + { + state[(offset + i) >> 3] ^= (uint64_t)data[i] + << (8 * ((offset + i) & 0x07)); + } +#endif /* !MLK_SYS_LITTLE_ENDIAN */ +} + +static void mlk_keccakf1600x4_extract_bytes_c(uint64_t *state, + unsigned char *data0, + unsigned char *data1, + unsigned char *data2, + unsigned char *data3, + unsigned offset, unsigned length) +__contract__( + requires(0 <= offset && offset <= MLK_KECCAK_LANES * sizeof(uint64_t) && + 0 <= length && length <= MLK_KECCAK_LANES * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY)) + requires(memory_no_alias(data0, length)) + requires(memory_no_alias(data1, length)) + requires(memory_no_alias(data2, length)) + requires(memory_no_alias(data3, length)) + assigns(memory_slice(data0, length)) + assigns(memory_slice(data1, length)) + assigns(memory_slice(data2, length)) + assigns(memory_slice(data3, length)) +) +{ + mlk_keccakf1600_extract_bytes(state + MLK_KECCAK_LANES * 0, data0, offset, + length); + mlk_keccakf1600_extract_bytes(state + MLK_KECCAK_LANES * 1, data1, offset, + length); + mlk_keccakf1600_extract_bytes(state + MLK_KECCAK_LANES * 2, data2, offset, + length); + mlk_keccakf1600_extract_bytes(state + MLK_KECCAK_LANES * 3, data3, offset, + length); +} + +void mlk_keccakf1600x4_extract_bytes(uint64_t *state, unsigned char *data0, + unsigned char *data1, unsigned char *data2, + unsigned char *data3, unsigned offset, + unsigned length) +{ +#if defined(MLK_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES) + if (mlk_keccakf1600_extract_bytes_x4_native(state, data0, data1, data2, data3, + offset, length) == + MLK_NATIVE_FUNC_SUCCESS) + { + return; + } +#endif /* MLK_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES */ + mlk_keccakf1600x4_extract_bytes_c(state, data0, data1, data2, data3, offset, + length); +} + +static void mlk_keccakf1600x4_xor_bytes_c(uint64_t *state, + const unsigned char *data0, + const unsigned char *data1, + const unsigned char *data2, + const unsigned char *data3, + unsigned offset, unsigned length) +__contract__( + requires(0 <= offset && offset <= MLK_KECCAK_LANES * sizeof(uint64_t) && + 0 <= length && length <= MLK_KECCAK_LANES * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY)) + requires(memory_no_alias(data0, length)) + /* Case 1: all input buffers are distinct; Case 2: All input buffers are the same */ + requires((data0 == data1 && + data0 == data2 && + data0 == data3) || + (memory_no_alias(data1, length) && + memory_no_alias(data2, length) && + memory_no_alias(data3, length))) + assigns(memory_slice(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY)) +) +{ + mlk_keccakf1600_xor_bytes(state + MLK_KECCAK_LANES * 0, data0, offset, + length); + mlk_keccakf1600_xor_bytes(state + MLK_KECCAK_LANES * 1, data1, offset, + length); + mlk_keccakf1600_xor_bytes(state + MLK_KECCAK_LANES * 2, data2, offset, + length); + mlk_keccakf1600_xor_bytes(state + MLK_KECCAK_LANES * 3, data3, offset, + length); +} + +void mlk_keccakf1600x4_xor_bytes(uint64_t *state, const unsigned char *data0, + const unsigned char *data1, + const unsigned char *data2, + const unsigned char *data3, unsigned offset, + unsigned length) +{ +#if defined(MLK_USE_NATIVE_FIPS202_X4_XOR_BYTES) + if (mlk_keccakf1600_xor_bytes_x4_native(state, data0, data1, data2, data3, + offset, + length) == MLK_NATIVE_FUNC_SUCCESS) + { + return; + } +#endif /* MLK_USE_NATIVE_FIPS202_X4_XOR_BYTES */ + mlk_keccakf1600x4_xor_bytes_c(state, data0, data1, data2, data3, offset, + length); +} + +void mlk_keccakf1600x4_permute(uint64_t *state) +{ +#if defined(MLK_USE_NATIVE_FIPS202_X4) + if (mlk_keccak_f1600_x4_native(state) == MLK_NATIVE_FUNC_SUCCESS) + { + return; + } +#endif /* MLK_USE_NATIVE_FIPS202_X4 */ + mlk_keccakf1600_permute(state + MLK_KECCAK_LANES * 0); + mlk_keccakf1600_permute(state + MLK_KECCAK_LANES * 1); + mlk_keccakf1600_permute(state + MLK_KECCAK_LANES * 2); + mlk_keccakf1600_permute(state + MLK_KECCAK_LANES * 3); +} + +static const uint64_t mlk_KeccakF_RoundConstants[MLK_KECCAK_NROUNDS] = { + (uint64_t)0x0000000000000001ULL, (uint64_t)0x0000000000008082ULL, + (uint64_t)0x800000000000808aULL, (uint64_t)0x8000000080008000ULL, + (uint64_t)0x000000000000808bULL, (uint64_t)0x0000000080000001ULL, + (uint64_t)0x8000000080008081ULL, (uint64_t)0x8000000000008009ULL, + (uint64_t)0x000000000000008aULL, (uint64_t)0x0000000000000088ULL, + (uint64_t)0x0000000080008009ULL, (uint64_t)0x000000008000000aULL, + (uint64_t)0x000000008000808bULL, (uint64_t)0x800000000000008bULL, + (uint64_t)0x8000000000008089ULL, (uint64_t)0x8000000000008003ULL, + (uint64_t)0x8000000000008002ULL, (uint64_t)0x8000000000000080ULL, + (uint64_t)0x000000000000800aULL, (uint64_t)0x800000008000000aULL, + (uint64_t)0x8000000080008081ULL, (uint64_t)0x8000000000008080ULL, + (uint64_t)0x0000000080000001ULL, (uint64_t)0x8000000080008008ULL}; + +MLK_STATIC_TESTABLE +void mlk_keccakf1600_permute_c(uint64_t *state) +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES)) + assigns(memory_slice(state, sizeof(uint64_t) * MLK_KECCAK_LANES)) +) +{ + unsigned round; + + uint64_t Aba, Abe, Abi, Abo, Abu; + uint64_t Aga, Age, Agi, Ago, Agu; + uint64_t Aka, Ake, Aki, Ako, Aku; + uint64_t Ama, Ame, Ami, Amo, Amu; + uint64_t Asa, Ase, Asi, Aso, Asu; + uint64_t BCa, BCe, BCi, BCo, BCu; + uint64_t Da, De, Di, Do, Du; + uint64_t Eba, Ebe, Ebi, Ebo, Ebu; + uint64_t Ega, Ege, Egi, Ego, Egu; + uint64_t Eka, Eke, Eki, Eko, Eku; + uint64_t Ema, Eme, Emi, Emo, Emu; + uint64_t Esa, Ese, Esi, Eso, Esu; + + /* copyFromState(A, state) */ + Aba = state[0]; + Abe = state[1]; + Abi = state[2]; + Abo = state[3]; + Abu = state[4]; + Aga = state[5]; + Age = state[6]; + Agi = state[7]; + Ago = state[8]; + Agu = state[9]; + Aka = state[10]; + Ake = state[11]; + Aki = state[12]; + Ako = state[13]; + Aku = state[14]; + Ama = state[15]; + Ame = state[16]; + Ami = state[17]; + Amo = state[18]; + Amu = state[19]; + Asa = state[20]; + Ase = state[21]; + Asi = state[22]; + Aso = state[23]; + Asu = state[24]; + + for (round = 0; round < MLK_KECCAK_NROUNDS; round += 2) + __loop__(invariant(round <= MLK_KECCAK_NROUNDS && round % 2 == 0) + decreases(MLK_KECCAK_NROUNDS - round)) + { + /* prepareTheta */ + BCa = Aba ^ Aga ^ Aka ^ Ama ^ Asa; + BCe = Abe ^ Age ^ Ake ^ Ame ^ Ase; + BCi = Abi ^ Agi ^ Aki ^ Ami ^ Asi; + BCo = Abo ^ Ago ^ Ako ^ Amo ^ Aso; + BCu = Abu ^ Agu ^ Aku ^ Amu ^ Asu; + + /* thetaRhoPiChiIotaPrepareTheta(round, A, E) */ + Da = BCu ^ MLK_KECCAK_ROL(BCe, 1); + De = BCa ^ MLK_KECCAK_ROL(BCi, 1); + Di = BCe ^ MLK_KECCAK_ROL(BCo, 1); + Do = BCi ^ MLK_KECCAK_ROL(BCu, 1); + Du = BCo ^ MLK_KECCAK_ROL(BCa, 1); + + Aba ^= Da; + BCa = Aba; + Age ^= De; + BCe = MLK_KECCAK_ROL(Age, 44); + Aki ^= Di; + BCi = MLK_KECCAK_ROL(Aki, 43); + Amo ^= Do; + BCo = MLK_KECCAK_ROL(Amo, 21); + Asu ^= Du; + BCu = MLK_KECCAK_ROL(Asu, 14); + Eba = BCa ^ ((~BCe) & BCi); + Eba ^= (uint64_t)mlk_KeccakF_RoundConstants[round]; + Ebe = BCe ^ ((~BCi) & BCo); + Ebi = BCi ^ ((~BCo) & BCu); + Ebo = BCo ^ ((~BCu) & BCa); + Ebu = BCu ^ ((~BCa) & BCe); + + Abo ^= Do; + BCa = MLK_KECCAK_ROL(Abo, 28); + Agu ^= Du; + BCe = MLK_KECCAK_ROL(Agu, 20); + Aka ^= Da; + BCi = MLK_KECCAK_ROL(Aka, 3); + Ame ^= De; + BCo = MLK_KECCAK_ROL(Ame, 45); + Asi ^= Di; + BCu = MLK_KECCAK_ROL(Asi, 61); + Ega = BCa ^ ((~BCe) & BCi); + Ege = BCe ^ ((~BCi) & BCo); + Egi = BCi ^ ((~BCo) & BCu); + Ego = BCo ^ ((~BCu) & BCa); + Egu = BCu ^ ((~BCa) & BCe); + + Abe ^= De; + BCa = MLK_KECCAK_ROL(Abe, 1); + Agi ^= Di; + BCe = MLK_KECCAK_ROL(Agi, 6); + Ako ^= Do; + BCi = MLK_KECCAK_ROL(Ako, 25); + Amu ^= Du; + BCo = MLK_KECCAK_ROL(Amu, 8); + Asa ^= Da; + BCu = MLK_KECCAK_ROL(Asa, 18); + Eka = BCa ^ ((~BCe) & BCi); + Eke = BCe ^ ((~BCi) & BCo); + Eki = BCi ^ ((~BCo) & BCu); + Eko = BCo ^ ((~BCu) & BCa); + Eku = BCu ^ ((~BCa) & BCe); + + Abu ^= Du; + BCa = MLK_KECCAK_ROL(Abu, 27); + Aga ^= Da; + BCe = MLK_KECCAK_ROL(Aga, 36); + Ake ^= De; + BCi = MLK_KECCAK_ROL(Ake, 10); + Ami ^= Di; + BCo = MLK_KECCAK_ROL(Ami, 15); + Aso ^= Do; + BCu = MLK_KECCAK_ROL(Aso, 56); + Ema = BCa ^ ((~BCe) & BCi); + Eme = BCe ^ ((~BCi) & BCo); + Emi = BCi ^ ((~BCo) & BCu); + Emo = BCo ^ ((~BCu) & BCa); + Emu = BCu ^ ((~BCa) & BCe); + + Abi ^= Di; + BCa = MLK_KECCAK_ROL(Abi, 62); + Ago ^= Do; + BCe = MLK_KECCAK_ROL(Ago, 55); + Aku ^= Du; + BCi = MLK_KECCAK_ROL(Aku, 39); + Ama ^= Da; + BCo = MLK_KECCAK_ROL(Ama, 41); + Ase ^= De; + BCu = MLK_KECCAK_ROL(Ase, 2); + Esa = BCa ^ ((~BCe) & BCi); + Ese = BCe ^ ((~BCi) & BCo); + Esi = BCi ^ ((~BCo) & BCu); + Eso = BCo ^ ((~BCu) & BCa); + Esu = BCu ^ ((~BCa) & BCe); + + /* prepareTheta */ + BCa = Eba ^ Ega ^ Eka ^ Ema ^ Esa; + BCe = Ebe ^ Ege ^ Eke ^ Eme ^ Ese; + BCi = Ebi ^ Egi ^ Eki ^ Emi ^ Esi; + BCo = Ebo ^ Ego ^ Eko ^ Emo ^ Eso; + BCu = Ebu ^ Egu ^ Eku ^ Emu ^ Esu; + + /* thetaRhoPiChiIotaPrepareTheta(round+1, E, A) */ + Da = BCu ^ MLK_KECCAK_ROL(BCe, 1); + De = BCa ^ MLK_KECCAK_ROL(BCi, 1); + Di = BCe ^ MLK_KECCAK_ROL(BCo, 1); + Do = BCi ^ MLK_KECCAK_ROL(BCu, 1); + Du = BCo ^ MLK_KECCAK_ROL(BCa, 1); + + Eba ^= Da; + BCa = Eba; + Ege ^= De; + BCe = MLK_KECCAK_ROL(Ege, 44); + Eki ^= Di; + BCi = MLK_KECCAK_ROL(Eki, 43); + Emo ^= Do; + BCo = MLK_KECCAK_ROL(Emo, 21); + Esu ^= Du; + BCu = MLK_KECCAK_ROL(Esu, 14); + Aba = BCa ^ ((~BCe) & BCi); + Aba ^= (uint64_t)mlk_KeccakF_RoundConstants[round + 1]; + Abe = BCe ^ ((~BCi) & BCo); + Abi = BCi ^ ((~BCo) & BCu); + Abo = BCo ^ ((~BCu) & BCa); + Abu = BCu ^ ((~BCa) & BCe); + + Ebo ^= Do; + BCa = MLK_KECCAK_ROL(Ebo, 28); + Egu ^= Du; + BCe = MLK_KECCAK_ROL(Egu, 20); + Eka ^= Da; + BCi = MLK_KECCAK_ROL(Eka, 3); + Eme ^= De; + BCo = MLK_KECCAK_ROL(Eme, 45); + Esi ^= Di; + BCu = MLK_KECCAK_ROL(Esi, 61); + Aga = BCa ^ ((~BCe) & BCi); + Age = BCe ^ ((~BCi) & BCo); + Agi = BCi ^ ((~BCo) & BCu); + Ago = BCo ^ ((~BCu) & BCa); + Agu = BCu ^ ((~BCa) & BCe); + + Ebe ^= De; + BCa = MLK_KECCAK_ROL(Ebe, 1); + Egi ^= Di; + BCe = MLK_KECCAK_ROL(Egi, 6); + Eko ^= Do; + BCi = MLK_KECCAK_ROL(Eko, 25); + Emu ^= Du; + BCo = MLK_KECCAK_ROL(Emu, 8); + Esa ^= Da; + BCu = MLK_KECCAK_ROL(Esa, 18); + Aka = BCa ^ ((~BCe) & BCi); + Ake = BCe ^ ((~BCi) & BCo); + Aki = BCi ^ ((~BCo) & BCu); + Ako = BCo ^ ((~BCu) & BCa); + Aku = BCu ^ ((~BCa) & BCe); + + Ebu ^= Du; + BCa = MLK_KECCAK_ROL(Ebu, 27); + Ega ^= Da; + BCe = MLK_KECCAK_ROL(Ega, 36); + Eke ^= De; + BCi = MLK_KECCAK_ROL(Eke, 10); + Emi ^= Di; + BCo = MLK_KECCAK_ROL(Emi, 15); + Eso ^= Do; + BCu = MLK_KECCAK_ROL(Eso, 56); + Ama = BCa ^ ((~BCe) & BCi); + Ame = BCe ^ ((~BCi) & BCo); + Ami = BCi ^ ((~BCo) & BCu); + Amo = BCo ^ ((~BCu) & BCa); + Amu = BCu ^ ((~BCa) & BCe); + + Ebi ^= Di; + BCa = MLK_KECCAK_ROL(Ebi, 62); + Ego ^= Do; + BCe = MLK_KECCAK_ROL(Ego, 55); + Eku ^= Du; + BCi = MLK_KECCAK_ROL(Eku, 39); + Ema ^= Da; + BCo = MLK_KECCAK_ROL(Ema, 41); + Ese ^= De; + BCu = MLK_KECCAK_ROL(Ese, 2); + Asa = BCa ^ ((~BCe) & BCi); + Ase = BCe ^ ((~BCi) & BCo); + Asi = BCi ^ ((~BCo) & BCu); + Aso = BCo ^ ((~BCu) & BCa); + Asu = BCu ^ ((~BCa) & BCe); + } + + /* copyToState(state, A) */ + state[0] = Aba; + state[1] = Abe; + state[2] = Abi; + state[3] = Abo; + state[4] = Abu; + state[5] = Aga; + state[6] = Age; + state[7] = Agi; + state[8] = Ago; + state[9] = Agu; + state[10] = Aka; + state[11] = Ake; + state[12] = Aki; + state[13] = Ako; + state[14] = Aku; + state[15] = Ama; + state[16] = Ame; + state[17] = Ami; + state[18] = Amo; + state[19] = Amu; + state[20] = Asa; + state[21] = Ase; + state[22] = Asi; + state[23] = Aso; + state[24] = Asu; +} + +void mlk_keccakf1600_permute(uint64_t *state) +{ +#if defined(MLK_USE_NATIVE_FIPS202_X1) + if (mlk_keccak_f1600_x1_native(state) == MLK_NATIVE_FUNC_SUCCESS) + { + return; + } +#endif /* MLK_USE_NATIVE_FIPS202_X1 */ + mlk_keccakf1600_permute_c(state); +} + +#else /* !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +MLK_EMPTY_CU(keccakf1600) + +#endif /* MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. + * Don't modify by hand -- this is auto-generated by scripts/autogen. */ +#undef MLK_KECCAK_NROUNDS +#undef MLK_KECCAK_ROL diff --git a/cbits/mlkem/src/fips202/keccakf1600.h b/cbits/mlkem/src/fips202/keccakf1600.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/keccakf1600.h @@ -0,0 +1,98 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLK_FIPS202_KECCAKF1600_H +#define MLK_FIPS202_KECCAKF1600_H +#include "../cbmc.h" +#include "../common.h" + +#define MLK_KECCAK_LANES 25 +#define MLK_KECCAK_WAY 4 + +/* + * WARNING: + * The contents of this structure, including the placement + * and interleaving of Keccak lanes, are IMPLEMENTATION-DEFINED. + * The struct is only exposed here to allow its construction on the stack. + */ + +#define mlk_keccakf1600_extract_bytes MLK_NAMESPACE(keccakf1600_extract_bytes) +void mlk_keccakf1600_extract_bytes(uint64_t *state, unsigned char *data, + unsigned offset, unsigned length) +__contract__( + requires(0 <= offset && offset <= MLK_KECCAK_LANES * sizeof(uint64_t) && + 0 <= length && length <= MLK_KECCAK_LANES * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES)) + requires(memory_no_alias(data, length)) + assigns(memory_slice(data, length)) +); + +#define mlk_keccakf1600_xor_bytes MLK_NAMESPACE(keccakf1600_xor_bytes) +void mlk_keccakf1600_xor_bytes(uint64_t *state, const unsigned char *data, + unsigned offset, unsigned length) +__contract__( + requires(0 <= offset && offset <= MLK_KECCAK_LANES * sizeof(uint64_t) && + 0 <= length && length <= MLK_KECCAK_LANES * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES)) + requires(memory_no_alias(data, length)) + assigns(memory_slice(state, sizeof(uint64_t) * MLK_KECCAK_LANES)) +); + +#define mlk_keccakf1600x4_extract_bytes \ + MLK_NAMESPACE(keccakf1600x4_extract_bytes) +void mlk_keccakf1600x4_extract_bytes(uint64_t *state, unsigned char *data0, + unsigned char *data1, unsigned char *data2, + unsigned char *data3, unsigned offset, + unsigned length) +__contract__( + requires(0 <= offset && offset <= MLK_KECCAK_LANES * sizeof(uint64_t) && + 0 <= length && length <= MLK_KECCAK_LANES * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY)) + requires(memory_no_alias(data0, length)) + requires(memory_no_alias(data1, length)) + requires(memory_no_alias(data2, length)) + requires(memory_no_alias(data3, length)) + assigns(memory_slice(data0, length)) + assigns(memory_slice(data1, length)) + assigns(memory_slice(data2, length)) + assigns(memory_slice(data3, length)) +); + +#define mlk_keccakf1600x4_xor_bytes MLK_NAMESPACE(keccakf1600x4_xor_bytes) +void mlk_keccakf1600x4_xor_bytes(uint64_t *state, const unsigned char *data0, + const unsigned char *data1, + const unsigned char *data2, + const unsigned char *data3, unsigned offset, + unsigned length) +__contract__( + requires(0 <= offset && offset <= MLK_KECCAK_LANES * sizeof(uint64_t) && + 0 <= length && length <= MLK_KECCAK_LANES * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY)) + requires(memory_no_alias(data0, length)) + /* Case 1: all input buffers are distinct; Case 2: All input buffers are the same */ + requires((data0 == data1 && + data0 == data2 && + data0 == data3) || + (memory_no_alias(data1, length) && + memory_no_alias(data2, length) && + memory_no_alias(data3, length))) + assigns(memory_slice(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY)) +); + + +#define mlk_keccakf1600x4_permute MLK_NAMESPACE(keccakf1600x4_permute) +void mlk_keccakf1600x4_permute(uint64_t *state) +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY)) + assigns(memory_slice(state, sizeof(uint64_t) * MLK_KECCAK_LANES * MLK_KECCAK_WAY)) +); + +#define mlk_keccakf1600_permute MLK_NAMESPACE(keccakf1600_permute) +void mlk_keccakf1600_permute(uint64_t *state) +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * MLK_KECCAK_LANES)) + assigns(memory_slice(state, sizeof(uint64_t) * MLK_KECCAK_LANES)) +); + +#endif /* !MLK_FIPS202_KECCAKF1600_H */ diff --git a/cbits/mlkem/src/fips202/native/aarch64/auto.h b/cbits/mlkem/src/fips202/native/aarch64/auto.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/aarch64/auto.h @@ -0,0 +1,78 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [HYBRID] + * Hybrid scalar/vector implementations of Keccak and SPHINCS+ on AArch64 + * Becker, Kannwischer + * https://eprint.iacr.org/2022/1243 + */ + +#ifndef MLK_FIPS202_NATIVE_AARCH64_AUTO_H +#define MLK_FIPS202_NATIVE_AARCH64_AUTO_H +/* Default FIPS202 assembly profile for AArch64 systems */ + +/* + * Default logic to decide which implementation to use. + * + */ + +/* + * Keccak-f1600 + * + * - On Arm-based Apple CPUs, or if MLK_SYS_AARCH64_FAST_SHA3 is set, + * we pick a pure Neon implementation. + * - Otherwise, unless MLK_SYS_AARCH64_SLOW_BARREL_SHIFTER is set, + * we use lazy-rotation scalar assembly from @[HYBRID]. + * - Otherwise, if MLK_SYS_AARCH64_SLOW_BARREL_SHIFTER is set, we + * fall back to the standard C implementation. + */ +#if defined(__ARM_FEATURE_SHA3) && \ + (defined(__APPLE__) || defined(MLK_SYS_AARCH64_FAST_SHA3)) +#include "x1_v84a.h" +#elif !defined(MLK_SYS_AARCH64_SLOW_BARREL_SHIFTER) +#include "x1_scalar.h" +#endif + +/* Batched, SIMD-based Keccak-f1600 implementations. */ +#if defined(MLK_SYS_AARCH64_NEON) + +/* + * Keccak-f1600x2/x4 + * + * The optimal implementation is highly CPU-specific; see @[HYBRID]. + * + * For now, if v8.4-A is not implemented, we fall back to Keccak-f1600. + * If v8.4-A is implemented and we are on an Apple CPU or + * MLK_SYS_AARCH64_FAST_SHA3 is set, we use a plain Neon-based + * implementation. + * Otherwise, if v8.4-A is implemented, we use a scalar/Neon/Neon hybrid. + * The reason for this distinction is that Apple CPUs (and CPUs flagged with + * MLK_SYS_AARCH64_FAST_SHA3) implement the SHA3 instructions on all SIMD + * units, while Arm CPUs prior to Cortex-X4 don't, and ordinary Neon + * instructions are still needed. + */ +#if defined(__ARM_FEATURE_SHA3) +/* + * For Apple-M cores (and CPUs flagged with MLK_SYS_AARCH64_FAST_SHA3), we + * use a plain implementation leveraging SHA3 instructions only. + */ +#if defined(__APPLE__) || defined(MLK_SYS_AARCH64_FAST_SHA3) +#include "x2_v84a.h" +#else +#include "x4_v8a_v84a_scalar.h" +#endif + +#else /* __ARM_FEATURE_SHA3 */ + +#include "x4_v8a_scalar.h" + +#endif /* !__ARM_FEATURE_SHA3 */ + +#endif /* MLK_SYS_AARCH64_NEON */ + +#endif /* !MLK_FIPS202_NATIVE_AARCH64_AUTO_H */ diff --git a/cbits/mlkem/src/fips202/native/aarch64/src/fips202_native_aarch64.h b/cbits/mlkem/src/fips202/native/aarch64/src/fips202_native_aarch64.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/aarch64/src/fips202_native_aarch64.h @@ -0,0 +1,80 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLK_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H +#define MLK_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H + +#include "../../../../cbmc.h" +#include "../../../../common.h" + +#define mlk_keccakf1600_round_constants \ + MLK_NAMESPACE(keccakf1600_round_constants) +MLK_INTERNAL_DATA_DECLARATION const uint64_t + mlk_keccakf1600_round_constants[24]; + +#define mlk_keccak_f1600_x1_scalar_aarch64_asm \ + MLK_NAMESPACE(keccak_f1600_x1_scalar_aarch64_asm) +void mlk_keccak_f1600_x1_scalar_aarch64_asm(uint64_t state[25], + const uint64_t rc[24]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/keccak_f1600_x1_scalar_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 1)) + requires(rc == mlk_keccakf1600_round_constants) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 1)) +); + +#define mlk_keccak_f1600_x1_v84a_aarch64_asm \ + MLK_NAMESPACE(keccak_f1600_x1_v84a_aarch64_asm) +void mlk_keccak_f1600_x1_v84a_aarch64_asm(uint64_t state[25], + const uint64_t rc[24]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/keccak_f1600_x1_v84a_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 1)) + requires(rc == mlk_keccakf1600_round_constants) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 1)) +); + +#define mlk_keccak_f1600_x2_v84a_aarch64_asm \ + MLK_NAMESPACE(keccak_f1600_x2_v84a_aarch64_asm) +void mlk_keccak_f1600_x2_v84a_aarch64_asm(uint64_t state[50], + const uint64_t rc[24]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/keccak_f1600_x2_v84a_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 2)) + requires(rc == mlk_keccakf1600_round_constants) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 2)) +); + +#define mlk_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm \ + MLK_NAMESPACE(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm) +void mlk_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm(uint64_t state[100], + const uint64_t rc[24]) +/* This must be kept in sync with the HOL-Light specification + * in + * proofs/hol_light/aarch64/proofs/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.ml + */ +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4)) + requires(rc == mlk_keccakf1600_round_constants) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4)) +); + +#define mlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm \ + MLK_NAMESPACE(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm) +void mlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm( + uint64_t state[100], const uint64_t rc[24]) +/* This must be kept in sync with the HOL-Light specification + * in + * proofs/hol_light/aarch64/proofs/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.ml + */ +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4)) + requires(rc == mlk_keccakf1600_round_constants) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4)) +); + +#endif /* !MLK_FIPS202_NATIVE_AARCH64_SRC_FIPS202_NATIVE_AARCH64_H */ diff --git a/cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S b/cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S @@ -0,0 +1,377 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) 2021-2022 Arm Limited + * Copyright (c) 2022 Matthias Kannwischer + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +// Author: Hanno Becker +// Author: Matthias Kannwischer + +/*yaml + Name: keccak_f1600_x1_scalar_aarch64_asm + Description: AArch64 scalar implementation of Keccak-f[1600] permutation for single state + Signature: void mlk_keccak_f1600_x1_scalar_aarch64_asm(uint64_t state[25], const uint64_t rc[24]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + x0: + type: buffer + size_bytes: 200 + permissions: read/write + c_parameter: uint64_t state[25] + description: Keccak state (25 x uint64_t) + x1: + type: buffer + size_bytes: 192 + permissions: read-only + c_parameter: uint64_t const *rc + description: Round constants (24 x uint64_t) + Stack: + bytes: 128 + description: register preservation and temporary storage +*/ + +#include "../../../../common.h" +#if defined(MLK_FIPS202_AARCH64_NEED_X1_SCALAR) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/fips202/aarch64/src/keccak_f1600_x1_scalar_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(keccak_f1600_x1_scalar_aarch64_asm) +MLK_ASM_FN_SYMBOL(keccak_f1600_x1_scalar_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x80 + .cfi_adjust_cfa_offset 0x80 + stp x19, x20, [sp, #0x20] + .cfi_rel_offset x19, 0x20 + .cfi_rel_offset x20, 0x28 + stp x21, x22, [sp, #0x30] + .cfi_rel_offset x21, 0x30 + .cfi_rel_offset x22, 0x38 + stp x23, x24, [sp, #0x40] + .cfi_rel_offset x23, 0x40 + .cfi_rel_offset x24, 0x48 + stp x25, x26, [sp, #0x50] + .cfi_rel_offset x25, 0x50 + .cfi_rel_offset x26, 0x58 + stp x27, x28, [sp, #0x60] + .cfi_rel_offset x27, 0x60 + .cfi_rel_offset x28, 0x68 + stp x29, x30, [sp, #0x70] + .cfi_rel_offset x29, 0x70 + .cfi_rel_offset x30, 0x78 + +Lmlk_keccak_f1600_x1_scalar_initial: + mov x26, x1 + str x1, [sp, #0x8] + ldp x1, x6, [x0] + ldp x11, x16, [x0, #0x10] + ldp x21, x2, [x0, #0x20] + ldp x7, x12, [x0, #0x30] + ldp x17, x22, [x0, #0x40] + ldp x3, x8, [x0, #0x50] + ldp x13, x28, [x0, #0x60] + ldp x23, x4, [x0, #0x70] + ldp x9, x14, [x0, #0x80] + ldp x19, x24, [x0, #0x90] + ldp x5, x10, [x0, #0xa0] + ldp x15, x20, [x0, #0xb0] + ldr x25, [x0, #0xc0] + str x0, [sp] + eor x30, x24, x25 + eor x27, x9, x10 + eor x0, x30, x21 + eor x26, x27, x6 + eor x27, x26, x7 + eor x29, x0, x22 + eor x26, x29, x23 + eor x29, x4, x5 + eor x30, x29, x1 + eor x0, x27, x8 + eor x29, x30, x2 + eor x30, x19, x20 + eor x30, x30, x16 + eor x27, x26, x0, ror #63 + eor x4, x4, x27 + eor x30, x30, x17 + eor x30, x30, x28 + eor x29, x29, x3 + eor x0, x0, x30, ror #63 + eor x30, x30, x29, ror #63 + eor x22, x22, x30 + eor x23, x23, x30 + str x23, [sp, #0x18] + eor x23, x14, x15 + eor x14, x14, x0 + eor x23, x23, x11 + eor x15, x15, x0 + eor x1, x1, x27 + eor x23, x23, x12 + eor x23, x23, x13 + eor x11, x11, x0 + eor x29, x29, x23, ror #63 + eor x23, x23, x26, ror #63 + eor x26, x13, x0 + eor x13, x28, x23 + eor x28, x24, x30 + eor x24, x16, x23 + eor x16, x21, x30 + eor x21, x25, x30 + eor x30, x19, x23 + eor x19, x20, x23 + eor x20, x17, x23 + eor x17, x12, x0 + eor x0, x2, x27 + eor x2, x6, x29 + eor x6, x8, x29 + bic x8, x28, x13, ror #47 + eor x12, x3, x27 + bic x3, x13, x17, ror #19 + eor x5, x5, x27 + ldr x27, [sp, #0x18] + bic x25, x17, x2, ror #5 + eor x9, x9, x29 + eor x23, x25, x5, ror #52 + eor x3, x3, x2, ror #24 + eor x8, x8, x17, ror #2 + eor x17, x10, x29 + bic x25, x12, x22, ror #47 + eor x29, x7, x29 + bic x10, x4, x27, ror #2 + bic x7, x5, x28, ror #10 + eor x10, x10, x20, ror #50 + eor x13, x7, x13, ror #57 + bic x7, x2, x5, ror #47 + eor x2, x25, x24, ror #39 + bic x25, x20, x11, ror #57 + bic x5, x17, x4, ror #25 + eor x25, x25, x17, ror #53 + bic x17, x11, x17, ror #60 + eor x28, x7, x28, ror #57 + bic x7, x9, x12, ror #42 + eor x7, x7, x22, ror #25 + bic x22, x22, x24, ror #56 + bic x24, x24, x15, ror #31 + eor x22, x22, x15, ror #23 + bic x20, x27, x20, ror #48 + bic x15, x15, x9, ror #16 + eor x12, x15, x12, ror #58 + eor x15, x5, x27, ror #27 + eor x5, x20, x11, ror #41 + ldr x11, [sp, #0x8] + eor x20, x17, x4, ror #21 + eor x17, x24, x9, ror #47 + mov x24, #0x1 // =1 + bic x9, x0, x16, ror #9 + str x24, [sp, #0x10] + bic x24, x29, x1, ror #44 + bic x27, x1, x21, ror #50 + bic x4, x26, x29, ror #63 + eor x1, x1, x4, ror #21 + ldr x11, [x11] + bic x4, x21, x30, ror #57 + eor x21, x24, x21, ror #30 + eor x24, x9, x19, ror #44 + bic x9, x14, x6, ror #5 + eor x9, x9, x0, ror #43 + bic x0, x6, x0, ror #38 + eor x1, x1, x11 + eor x11, x4, x26, ror #35 + eor x4, x0, x16, ror #47 + bic x0, x16, x19, ror #35 + eor x16, x27, x30, ror #43 + bic x27, x30, x26, ror #42 + bic x26, x19, x14, ror #41 + eor x19, x0, x14, ror #12 + eor x14, x26, x6, ror #46 + eor x6, x27, x29, ror #41 + +Lmlk_keccak_f1600_x1_scalar_loop: + eor x0, x15, x11, ror #52 + eor x0, x0, x13, ror #48 + eor x26, x8, x9, ror #57 + eor x27, x0, x14, ror #10 + eor x29, x16, x28, ror #63 + eor x26, x26, x6, ror #51 + eor x30, x23, x22, ror #50 + eor x0, x26, x10, ror #31 + eor x29, x29, x19, ror #37 + eor x27, x27, x12, ror #5 + eor x30, x30, x24, ror #34 + eor x0, x0, x7, ror #27 + eor x26, x30, x21, ror #26 + eor x26, x26, x25, ror #15 + ror x30, x27, #0x3e + eor x30, x30, x26, ror #57 + ror x26, x26, #0x3a + eor x16, x30, x16 + eor x28, x30, x28, ror #63 + str x28, [sp, #0x18] + eor x29, x29, x17, ror #36 + eor x28, x1, x2, ror #61 + eor x19, x30, x19, ror #37 + eor x29, x29, x20, ror #2 + eor x28, x28, x4, ror #54 + eor x26, x26, x0, ror #55 + eor x28, x28, x3, ror #39 + eor x28, x28, x5, ror #25 + ror x0, x0, #0x38 + eor x0, x0, x29, ror #63 + eor x27, x28, x27, ror #61 + eor x13, x0, x13, ror #46 + eor x28, x29, x28, ror #63 + eor x29, x30, x20, ror #2 + eor x20, x26, x3, ror #39 + eor x11, x0, x11, ror #50 + eor x25, x28, x25, ror #9 + eor x3, x28, x21, ror #20 + eor x21, x26, x1 + eor x9, x27, x9, ror #49 + eor x24, x28, x24, ror #28 + eor x1, x30, x17, ror #36 + eor x14, x0, x14, ror #8 + eor x22, x28, x22, ror #44 + eor x8, x27, x8, ror #56 + eor x17, x27, x7, ror #19 + eor x15, x0, x15, ror #62 + bic x7, x20, x22, ror #47 + eor x4, x26, x4, ror #54 + eor x0, x0, x12, ror #3 + eor x28, x28, x23, ror #58 + eor x23, x26, x2, ror #61 + eor x26, x26, x5, ror #25 + eor x2, x7, x16, ror #39 + bic x7, x9, x20, ror #42 + bic x30, x15, x9, ror #16 + eor x7, x7, x22, ror #25 + eor x12, x30, x20, ror #58 + bic x20, x22, x16, ror #56 + eor x30, x27, x6, ror #43 + eor x22, x20, x15, ror #23 + bic x6, x19, x13, ror #42 + eor x6, x6, x17, ror #41 + bic x5, x13, x17, ror #63 + eor x5, x21, x5, ror #21 + bic x17, x17, x21, ror #44 + eor x27, x27, x10, ror #23 + bic x21, x21, x25, ror #50 + bic x20, x27, x4, ror #25 + bic x10, x16, x15, ror #31 + eor x16, x21, x19, ror #43 + eor x21, x17, x25, ror #30 + bic x19, x25, x19, ror #57 + ldr x25, [sp, #0x10] + eor x17, x10, x9, ror #47 + ldr x9, [sp, #0x8] + eor x15, x20, x28, ror #27 + bic x20, x4, x28, ror #2 + eor x10, x20, x1, ror #50 + bic x20, x11, x27, ror #60 + eor x20, x20, x4, ror #21 + bic x4, x28, x1, ror #48 + bic x1, x1, x11, ror #57 + ldr x28, [x9, x25, lsl #3] + ldr x9, [sp, #0x18] + add x25, x25, #0x1 + str x25, [sp, #0x10] + cmp x25, #0x17 + eor x25, x1, x27, ror #53 + bic x27, x30, x26, ror #47 + eor x1, x5, x28 + eor x5, x4, x11, ror #41 + eor x11, x19, x13, ror #35 + bic x13, x26, x24, ror #10 + eor x28, x27, x24, ror #57 + bic x27, x24, x9, ror #47 + bic x19, x23, x3, ror #9 + bic x4, x29, x14, ror #41 + eor x24, x19, x29, ror #44 + bic x29, x3, x29, ror #35 + eor x13, x13, x9, ror #57 + eor x19, x29, x14, ror #12 + bic x29, x9, x0, ror #19 + bic x14, x14, x8, ror #5 + eor x9, x14, x23, ror #43 + eor x14, x4, x8, ror #46 + bic x23, x8, x23, ror #38 + eor x8, x27, x0, ror #2 + eor x4, x23, x3, ror #47 + bic x3, x0, x30, ror #5 + eor x23, x3, x26, ror #52 + eor x3, x29, x30, ror #24 + b.le Lmlk_keccak_f1600_x1_scalar_loop + ror x6, x6, #0x2b + ror x11, x11, #0x32 + ror x21, x21, #0x14 + ror x2, x2, #0x3d + ror x7, x7, #0x13 + ror x12, x12, #0x3 + ror x17, x17, #0x24 + ror x22, x22, #0x2c + ror x3, x3, #0x27 + ror x8, x8, #0x38 + ror x13, x13, #0x2e + ror x28, x28, #0x3f + ror x23, x23, #0x3a + ror x4, x4, #0x36 + ror x9, x9, #0x31 + ror x14, x14, #0x8 + ror x19, x19, #0x25 + ror x24, x24, #0x1c + ror x5, x5, #0x19 + ror x10, x10, #0x17 + ror x15, x15, #0x3e + ror x20, x20, #0x2 + ror x25, x25, #0x9 + ldr x0, [sp] + stp x1, x6, [x0] + stp x11, x16, [x0, #0x10] + stp x21, x2, [x0, #0x20] + stp x7, x12, [x0, #0x30] + stp x17, x22, [x0, #0x40] + stp x3, x8, [x0, #0x50] + stp x13, x28, [x0, #0x60] + stp x23, x4, [x0, #0x70] + stp x9, x14, [x0, #0x80] + stp x19, x24, [x0, #0x90] + stp x5, x10, [x0, #0xa0] + stp x15, x20, [x0, #0xb0] + str x25, [x0, #0xc0] + ldp x19, x20, [sp, #0x20] + .cfi_restore x19 + .cfi_restore x20 + ldp x21, x22, [sp, #0x30] + .cfi_restore x21 + .cfi_restore x22 + ldp x23, x24, [sp, #0x40] + .cfi_restore x23 + .cfi_restore x24 + ldp x25, x26, [sp, #0x50] + .cfi_restore x25 + .cfi_restore x26 + ldp x27, x28, [sp, #0x60] + .cfi_restore x27 + .cfi_restore x28 + ldp x29, x30, [sp, #0x70] + .cfi_restore x29 + .cfi_restore x30 + add sp, sp, #0x80 + .cfi_adjust_cfa_offset -0x80 + ret + .cfi_endproc + +MLK_ASM_FN_SIZE(keccak_f1600_x1_scalar_aarch64_asm) + +#endif /* MLK_FIPS202_AARCH64_NEED_X1_SCALAR && \ + !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S b/cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S @@ -0,0 +1,206 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) 2021-2022 Arm Limited + * Copyright (c) 2022 Matthias Kannwischer + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [HYBRID] + * Hybrid scalar/vector implementations of Keccak and SPHINCS+ on AArch64 + * Becker, Kannwischer + * https://eprint.iacr.org/2022/1243 + */ + +/*yaml + Name: keccak_f1600_x1_v84a_aarch64_asm + Description: AArch64 ARMv8.4-A implementation of Keccak-f[1600] permutation for single state + Signature: void mlk_keccak_f1600_x1_v84a_aarch64_asm(uint64_t state[25], const uint64_t rc[24]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON, SHA3] + x0: + type: buffer + size_bytes: 200 + permissions: read/write + c_parameter: uint64_t state[25] + description: Keccak state (25 x uint64_t) + x1: + type: buffer + size_bytes: 192 + permissions: read-only + c_parameter: const uint64_t rc[24] + description: Round constants (24 x uint64_t) + Stack: + bytes: 64 + description: register preservation +*/ + +// +// Author: Hanno Becker +// Author: Matthias Kannwischer +// +// This implementation is essentially from the paper @[HYBRID]. +// The only difference is interleaving/deinterleaving of Keccak state +// during load and store, so that the caller need not do this. +// + +#include "../../../../common.h" +#if defined(MLK_FIPS202_AARCH64_NEED_X1_V84A) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +#if defined(__ARM_FEATURE_SHA3) + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/fips202/aarch64/src/keccak_f1600_x1_v84a_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(keccak_f1600_x1_v84a_aarch64_asm) +MLK_ASM_FN_SYMBOL(keccak_f1600_x1_v84a_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x40 + .cfi_adjust_cfa_offset 0x40 + stp d8, d9, [sp] + .cfi_rel_offset d8, 0x0 + .cfi_rel_offset d9, 0x8 + stp d10, d11, [sp, #0x10] + .cfi_rel_offset d10, 0x10 + .cfi_rel_offset d11, 0x18 + stp d12, d13, [sp, #0x20] + .cfi_rel_offset d12, 0x20 + .cfi_rel_offset d13, 0x28 + stp d14, d15, [sp, #0x30] + .cfi_rel_offset d14, 0x30 + .cfi_rel_offset d15, 0x38 + ldp d0, d1, [x0] + ldp d2, d3, [x0, #0x10] + ldp d4, d5, [x0, #0x20] + ldp d6, d7, [x0, #0x30] + ldp d8, d9, [x0, #0x40] + ldp d10, d11, [x0, #0x50] + ldp d12, d13, [x0, #0x60] + ldp d14, d15, [x0, #0x70] + ldp d16, d17, [x0, #0x80] + ldp d18, d19, [x0, #0x90] + ldp d20, d21, [x0, #0xa0] + ldp d22, d23, [x0, #0xb0] + ldr d24, [x0, #0xc0] + mov x2, #0x18 // =24 + +Lmlk_keccak_f1600_x1_v84a_loop: + eor3 v30.16b, v0.16b, v5.16b, v10.16b + eor3 v29.16b, v1.16b, v6.16b, v11.16b + eor3 v28.16b, v2.16b, v7.16b, v12.16b + eor3 v27.16b, v3.16b, v8.16b, v13.16b + eor3 v26.16b, v4.16b, v9.16b, v14.16b + eor3 v30.16b, v30.16b, v15.16b, v20.16b + eor3 v29.16b, v29.16b, v16.16b, v21.16b + eor3 v28.16b, v28.16b, v17.16b, v22.16b + eor3 v27.16b, v27.16b, v18.16b, v23.16b + eor3 v26.16b, v26.16b, v19.16b, v24.16b + rax1 v25.2d, v30.2d, v28.2d + rax1 v28.2d, v28.2d, v26.2d + rax1 v26.2d, v26.2d, v29.2d + rax1 v29.2d, v29.2d, v27.2d + rax1 v27.2d, v27.2d, v30.2d + eor v30.16b, v0.16b, v26.16b + xar v0.2d, v2.2d, v29.2d, #0x2 + xar v2.2d, v12.2d, v29.2d, #0x15 + xar v12.2d, v13.2d, v28.2d, #0x27 + xar v13.2d, v19.2d, v27.2d, #0x38 + xar v19.2d, v23.2d, v28.2d, #0x8 + xar v23.2d, v15.2d, v26.2d, #0x17 + xar v15.2d, v1.2d, v25.2d, #0x3f + xar v1.2d, v8.2d, v28.2d, #0x9 + xar v8.2d, v16.2d, v25.2d, #0x13 + xar v16.2d, v7.2d, v29.2d, #0x3a + xar v7.2d, v10.2d, v26.2d, #0x3d + xar v10.2d, v3.2d, v28.2d, #0x24 + xar v3.2d, v18.2d, v28.2d, #0x2b + xar v18.2d, v17.2d, v29.2d, #0x31 + xar v17.2d, v11.2d, v25.2d, #0x36 + xar v11.2d, v9.2d, v27.2d, #0x2c + xar v9.2d, v22.2d, v29.2d, #0x3 + xar v22.2d, v14.2d, v27.2d, #0x19 + xar v14.2d, v20.2d, v26.2d, #0x2e + xar v20.2d, v4.2d, v27.2d, #0x25 + xar v4.2d, v24.2d, v27.2d, #0x32 + xar v24.2d, v21.2d, v25.2d, #0x3e + xar v21.2d, v5.2d, v26.2d, #0x1c + xar v27.2d, v6.2d, v25.2d, #0x14 + ld1r { v31.2d }, [x1], #8 + bcax v5.16b, v10.16b, v7.16b, v11.16b + bcax v6.16b, v11.16b, v8.16b, v7.16b + bcax v7.16b, v7.16b, v9.16b, v8.16b + bcax v8.16b, v8.16b, v10.16b, v9.16b + bcax v9.16b, v9.16b, v11.16b, v10.16b + bcax v10.16b, v15.16b, v12.16b, v16.16b + bcax v11.16b, v16.16b, v13.16b, v12.16b + bcax v12.16b, v12.16b, v14.16b, v13.16b + bcax v13.16b, v13.16b, v15.16b, v14.16b + bcax v14.16b, v14.16b, v16.16b, v15.16b + bcax v15.16b, v20.16b, v17.16b, v21.16b + bcax v16.16b, v21.16b, v18.16b, v17.16b + bcax v17.16b, v17.16b, v19.16b, v18.16b + bcax v18.16b, v18.16b, v20.16b, v19.16b + bcax v19.16b, v19.16b, v21.16b, v20.16b + bcax v20.16b, v0.16b, v22.16b, v1.16b + bcax v21.16b, v1.16b, v23.16b, v22.16b + bcax v22.16b, v22.16b, v24.16b, v23.16b + bcax v23.16b, v23.16b, v0.16b, v24.16b + bcax v24.16b, v24.16b, v1.16b, v0.16b + bcax v0.16b, v30.16b, v2.16b, v27.16b + bcax v1.16b, v27.16b, v3.16b, v2.16b + bcax v2.16b, v2.16b, v4.16b, v3.16b + bcax v3.16b, v3.16b, v30.16b, v4.16b + bcax v4.16b, v4.16b, v27.16b, v30.16b + eor v0.16b, v0.16b, v31.16b + sub x2, x2, #0x1 + cbnz x2, Lmlk_keccak_f1600_x1_v84a_loop + stp d0, d1, [x0] + stp d2, d3, [x0, #0x10] + stp d4, d5, [x0, #0x20] + stp d6, d7, [x0, #0x30] + stp d8, d9, [x0, #0x40] + stp d10, d11, [x0, #0x50] + stp d12, d13, [x0, #0x60] + stp d14, d15, [x0, #0x70] + stp d16, d17, [x0, #0x80] + stp d18, d19, [x0, #0x90] + stp d20, d21, [x0, #0xa0] + stp d22, d23, [x0, #0xb0] + str d24, [x0, #0xc0] + ldp d8, d9, [sp] + .cfi_restore d8 + .cfi_restore d9 + ldp d10, d11, [sp, #0x10] + .cfi_restore d10 + .cfi_restore d11 + ldp d12, d13, [sp, #0x20] + .cfi_restore d12 + .cfi_restore d13 + ldp d14, d15, [sp, #0x30] + .cfi_restore d14 + .cfi_restore d15 + add sp, sp, #0x40 + .cfi_adjust_cfa_offset -0x40 + ret + .cfi_endproc + +MLK_ASM_FN_SIZE(keccak_f1600_x1_v84a_aarch64_asm) + +#endif /* __ARM_FEATURE_SHA3 */ + +#endif /* MLK_FIPS202_AARCH64_NEED_X1_V84A && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S b/cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S @@ -0,0 +1,261 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) 2021-2022 Arm Limited + * Copyright (c) 2022 Matthias Kannwischer + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [HYBRID] + * Hybrid scalar/vector implementations of Keccak and SPHINCS+ on AArch64 + * Becker, Kannwischer + * https://eprint.iacr.org/2022/1243 + */ + +/*yaml + Name: keccak_f1600_x2_v84a_aarch64_asm + Description: AArch64 ARMv8.4-A implementation of Keccak-f[1600] permutation for two sequential states + Signature: void mlk_keccak_f1600_x2_v84a_aarch64_asm(uint64_t state[50], const uint64_t rc[24]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON, SHA3] + x0: + type: buffer + size_bytes: 400 + permissions: read/write + c_parameter: uint64_t state[50] + description: Two sequential Keccak states (state0[25], state1[25]) + x1: + type: buffer + size_bytes: 192 + permissions: read-only + c_parameter: const uint64_t rc[24] + description: Round constants (24 x uint64_t) + Stack: + bytes: 64 + description: register preservation +*/ + +// +// Author: Hanno Becker +// Author: Matthias Kannwischer +// +// This implementation is essentially from the paper @[HYBRID]. +// The only difference is interleaving/deinterleaving of Keccak state +// during load and store, so that the caller need not do this. +// + +#include "../../../../common.h" +#if defined(MLK_FIPS202_AARCH64_NEED_X2_V84A) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +#if defined(__ARM_FEATURE_SHA3) + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/fips202/aarch64/src/keccak_f1600_x2_v84a_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(keccak_f1600_x2_v84a_aarch64_asm) +MLK_ASM_FN_SYMBOL(keccak_f1600_x2_v84a_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x40 + .cfi_adjust_cfa_offset 0x40 + stp d8, d9, [sp] + .cfi_rel_offset d8, 0x0 + .cfi_rel_offset d9, 0x8 + stp d10, d11, [sp, #0x10] + .cfi_rel_offset d10, 0x10 + .cfi_rel_offset d11, 0x18 + stp d12, d13, [sp, #0x20] + .cfi_rel_offset d12, 0x20 + .cfi_rel_offset d13, 0x28 + stp d14, d15, [sp, #0x30] + .cfi_rel_offset d14, 0x30 + .cfi_rel_offset d15, 0x38 + add x2, x0, #0xc8 + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x2], #32 + trn1 v0.2d, v25.2d, v27.2d + trn2 v1.2d, v25.2d, v27.2d + trn1 v2.2d, v26.2d, v28.2d + trn2 v3.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x2], #32 + trn1 v4.2d, v25.2d, v27.2d + trn2 v5.2d, v25.2d, v27.2d + trn1 v6.2d, v26.2d, v28.2d + trn2 v7.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x2], #32 + trn1 v8.2d, v25.2d, v27.2d + trn2 v9.2d, v25.2d, v27.2d + trn1 v10.2d, v26.2d, v28.2d + trn2 v11.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x2], #32 + trn1 v12.2d, v25.2d, v27.2d + trn2 v13.2d, v25.2d, v27.2d + trn1 v14.2d, v26.2d, v28.2d + trn2 v15.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x2], #32 + trn1 v16.2d, v25.2d, v27.2d + trn2 v17.2d, v25.2d, v27.2d + trn1 v18.2d, v26.2d, v28.2d + trn2 v19.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x2], #32 + trn1 v20.2d, v25.2d, v27.2d + trn2 v21.2d, v25.2d, v27.2d + trn1 v22.2d, v26.2d, v28.2d + trn2 v23.2d, v26.2d, v28.2d + ldr d25, [x0] + ldr d27, [x2] + trn1 v24.2d, v25.2d, v27.2d + mov x2, #0x18 // =24 + +Lmlk_keccak_f1600_x2_v84a_loop: + eor3 v30.16b, v0.16b, v5.16b, v10.16b + eor3 v29.16b, v1.16b, v6.16b, v11.16b + eor3 v28.16b, v2.16b, v7.16b, v12.16b + eor3 v27.16b, v3.16b, v8.16b, v13.16b + eor3 v26.16b, v4.16b, v9.16b, v14.16b + eor3 v30.16b, v30.16b, v15.16b, v20.16b + eor3 v29.16b, v29.16b, v16.16b, v21.16b + eor3 v28.16b, v28.16b, v17.16b, v22.16b + eor3 v27.16b, v27.16b, v18.16b, v23.16b + eor3 v26.16b, v26.16b, v19.16b, v24.16b + rax1 v25.2d, v30.2d, v28.2d + rax1 v28.2d, v28.2d, v26.2d + rax1 v26.2d, v26.2d, v29.2d + rax1 v29.2d, v29.2d, v27.2d + rax1 v27.2d, v27.2d, v30.2d + eor v30.16b, v0.16b, v26.16b + xar v0.2d, v2.2d, v29.2d, #0x2 + xar v2.2d, v12.2d, v29.2d, #0x15 + xar v12.2d, v13.2d, v28.2d, #0x27 + xar v13.2d, v19.2d, v27.2d, #0x38 + xar v19.2d, v23.2d, v28.2d, #0x8 + xar v23.2d, v15.2d, v26.2d, #0x17 + xar v15.2d, v1.2d, v25.2d, #0x3f + xar v1.2d, v8.2d, v28.2d, #0x9 + xar v8.2d, v16.2d, v25.2d, #0x13 + xar v16.2d, v7.2d, v29.2d, #0x3a + xar v7.2d, v10.2d, v26.2d, #0x3d + xar v10.2d, v3.2d, v28.2d, #0x24 + xar v3.2d, v18.2d, v28.2d, #0x2b + xar v18.2d, v17.2d, v29.2d, #0x31 + xar v17.2d, v11.2d, v25.2d, #0x36 + xar v11.2d, v9.2d, v27.2d, #0x2c + xar v9.2d, v22.2d, v29.2d, #0x3 + xar v22.2d, v14.2d, v27.2d, #0x19 + xar v14.2d, v20.2d, v26.2d, #0x2e + xar v20.2d, v4.2d, v27.2d, #0x25 + xar v4.2d, v24.2d, v27.2d, #0x32 + xar v24.2d, v21.2d, v25.2d, #0x3e + xar v21.2d, v5.2d, v26.2d, #0x1c + xar v27.2d, v6.2d, v25.2d, #0x14 + ld1r { v31.2d }, [x1], #8 + bcax v5.16b, v10.16b, v7.16b, v11.16b + bcax v6.16b, v11.16b, v8.16b, v7.16b + bcax v7.16b, v7.16b, v9.16b, v8.16b + bcax v8.16b, v8.16b, v10.16b, v9.16b + bcax v9.16b, v9.16b, v11.16b, v10.16b + bcax v10.16b, v15.16b, v12.16b, v16.16b + bcax v11.16b, v16.16b, v13.16b, v12.16b + bcax v12.16b, v12.16b, v14.16b, v13.16b + bcax v13.16b, v13.16b, v15.16b, v14.16b + bcax v14.16b, v14.16b, v16.16b, v15.16b + bcax v15.16b, v20.16b, v17.16b, v21.16b + bcax v16.16b, v21.16b, v18.16b, v17.16b + bcax v17.16b, v17.16b, v19.16b, v18.16b + bcax v18.16b, v18.16b, v20.16b, v19.16b + bcax v19.16b, v19.16b, v21.16b, v20.16b + bcax v20.16b, v0.16b, v22.16b, v1.16b + bcax v21.16b, v1.16b, v23.16b, v22.16b + bcax v22.16b, v22.16b, v24.16b, v23.16b + bcax v23.16b, v23.16b, v0.16b, v24.16b + bcax v24.16b, v24.16b, v1.16b, v0.16b + bcax v0.16b, v30.16b, v2.16b, v27.16b + bcax v1.16b, v27.16b, v3.16b, v2.16b + bcax v2.16b, v2.16b, v4.16b, v3.16b + bcax v3.16b, v3.16b, v30.16b, v4.16b + bcax v4.16b, v4.16b, v27.16b, v30.16b + eor v0.16b, v0.16b, v31.16b + sub x2, x2, #0x1 + cbnz x2, Lmlk_keccak_f1600_x2_v84a_loop + sub x0, x0, #0xc0 + add x2, x0, #0xc8 + trn1 v25.2d, v0.2d, v1.2d + trn1 v26.2d, v2.2d, v3.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v0.2d, v1.2d + trn2 v28.2d, v2.2d, v3.2d + st1 { v27.2d, v28.2d }, [x2], #32 + trn1 v25.2d, v4.2d, v5.2d + trn1 v26.2d, v6.2d, v7.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v4.2d, v5.2d + trn2 v28.2d, v6.2d, v7.2d + st1 { v27.2d, v28.2d }, [x2], #32 + trn1 v25.2d, v8.2d, v9.2d + trn1 v26.2d, v10.2d, v11.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v8.2d, v9.2d + trn2 v28.2d, v10.2d, v11.2d + st1 { v27.2d, v28.2d }, [x2], #32 + trn1 v25.2d, v12.2d, v13.2d + trn1 v26.2d, v14.2d, v15.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v12.2d, v13.2d + trn2 v28.2d, v14.2d, v15.2d + st1 { v27.2d, v28.2d }, [x2], #32 + trn1 v25.2d, v16.2d, v17.2d + trn1 v26.2d, v18.2d, v19.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v16.2d, v17.2d + trn2 v28.2d, v18.2d, v19.2d + st1 { v27.2d, v28.2d }, [x2], #32 + trn1 v25.2d, v20.2d, v21.2d + trn1 v26.2d, v22.2d, v23.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v20.2d, v21.2d + trn2 v28.2d, v22.2d, v23.2d + st1 { v27.2d, v28.2d }, [x2], #32 + str d24, [x0] + trn2 v25.2d, v24.2d, v24.2d + str d25, [x2] + ldp d8, d9, [sp] + .cfi_restore d8 + .cfi_restore d9 + ldp d10, d11, [sp, #0x10] + .cfi_restore d10 + .cfi_restore d11 + ldp d12, d13, [sp, #0x20] + .cfi_restore d12 + .cfi_restore d13 + ldp d14, d15, [sp, #0x30] + .cfi_restore d14 + .cfi_restore d15 + add sp, sp, #0x40 + .cfi_adjust_cfa_offset -0x40 + ret + .cfi_endproc + +MLK_ASM_FN_SIZE(keccak_f1600_x2_v84a_aarch64_asm) + +#endif /* __ARM_FEATURE_SHA3 */ + +#endif /* MLK_FIPS202_AARCH64_NEED_X2_V84A && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S b/cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S @@ -0,0 +1,1079 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) 2021-2022 Arm Limited + * Copyright (c) 2022 Matthias Kannwischer + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +// Author: Hanno Becker +// Author: Matthias Kannwischer + +/*yaml + Name: keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm + Description: AArch64 hybrid scalar/vector implementation of Keccak-f[1600] permutation for four sequential states + Signature: void mlk_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm(uint64_t state[100], const uint64_t rc[24]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 800 + permissions: read/write + c_parameter: uint64_t state[100] + description: Four sequential Keccak states (state0[25], state1[25], state2[25], state3[25]) + x1: + type: buffer + size_bytes: 192 + permissions: read-only + c_parameter: const uint64_t rc[24] + description: Round constants (24 x uint64_t) + Stack: + bytes: 224 + description: register preservation and temporary storage +*/ + +#include "../../../../common.h" +#if defined(MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/fips202/aarch64/src/keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm) +MLK_ASM_FN_SYMBOL(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0xe0 + .cfi_adjust_cfa_offset 0xe0 + stp x19, x20, [sp, #0x30] + .cfi_rel_offset x19, 0x30 + .cfi_rel_offset x20, 0x38 + stp x21, x22, [sp, #0x40] + .cfi_rel_offset x21, 0x40 + .cfi_rel_offset x22, 0x48 + stp x23, x24, [sp, #0x50] + .cfi_rel_offset x23, 0x50 + .cfi_rel_offset x24, 0x58 + stp x25, x26, [sp, #0x60] + .cfi_rel_offset x25, 0x60 + .cfi_rel_offset x26, 0x68 + stp x27, x28, [sp, #0x70] + .cfi_rel_offset x27, 0x70 + .cfi_rel_offset x28, 0x78 + stp x29, x30, [sp, #0x80] + .cfi_rel_offset x29, 0x80 + .cfi_rel_offset x30, 0x88 + stp d8, d9, [sp, #0x90] + .cfi_rel_offset d8, 0x90 + .cfi_rel_offset d9, 0x98 + stp d10, d11, [sp, #0xa0] + .cfi_rel_offset d10, 0xa0 + .cfi_rel_offset d11, 0xa8 + stp d12, d13, [sp, #0xb0] + .cfi_rel_offset d12, 0xb0 + .cfi_rel_offset d13, 0xb8 + stp d14, d15, [sp, #0xc0] + .cfi_rel_offset d14, 0xc0 + .cfi_rel_offset d15, 0xc8 + mov x29, x1 + mov x30, #0x0 // =0 + str x30, [sp, #0x20] + str x29, [sp, #0x8] + str x29, [sp, #0x10] + str x0, [sp] + add x4, x0, #0xc8 + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v0.2d, v25.2d, v27.2d + trn2 v1.2d, v25.2d, v27.2d + trn1 v2.2d, v26.2d, v28.2d + trn2 v3.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v4.2d, v25.2d, v27.2d + trn2 v5.2d, v25.2d, v27.2d + trn1 v6.2d, v26.2d, v28.2d + trn2 v7.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v8.2d, v25.2d, v27.2d + trn2 v9.2d, v25.2d, v27.2d + trn1 v10.2d, v26.2d, v28.2d + trn2 v11.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v12.2d, v25.2d, v27.2d + trn2 v13.2d, v25.2d, v27.2d + trn1 v14.2d, v26.2d, v28.2d + trn2 v15.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v16.2d, v25.2d, v27.2d + trn2 v17.2d, v25.2d, v27.2d + trn1 v18.2d, v26.2d, v28.2d + trn2 v19.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v20.2d, v25.2d, v27.2d + trn2 v21.2d, v25.2d, v27.2d + trn1 v22.2d, v26.2d, v28.2d + trn2 v23.2d, v26.2d, v28.2d + ldr d25, [x0] + ldr d27, [x4] + trn1 v24.2d, v25.2d, v27.2d + sub x0, x0, #0xc0 + add x0, x0, #0x190 + ldp x1, x6, [x0] + ldp x11, x16, [x0, #0x10] + ldp x21, x2, [x0, #0x20] + ldp x7, x12, [x0, #0x30] + ldp x17, x22, [x0, #0x40] + ldp x3, x8, [x0, #0x50] + ldp x13, x28, [x0, #0x60] + ldp x23, x4, [x0, #0x70] + ldp x9, x14, [x0, #0x80] + ldp x19, x24, [x0, #0x90] + ldp x5, x10, [x0, #0xa0] + ldp x15, x20, [x0, #0xb0] + ldr x25, [x0, #0xc0] + sub x0, x0, #0x190 + +Lmlk_keccak_f1600_x4_v8a_scalar_hybrid_initial: + eor x30, x24, x25 + eor x27, x9, x10 + eor v30.16b, v0.16b, v5.16b + eor v30.16b, v30.16b, v10.16b + eor x0, x30, x21 + eor v30.16b, v30.16b, v15.16b + eor x26, x27, x6 + eor x27, x26, x7 + eor v30.16b, v30.16b, v20.16b + eor x29, x0, x22 + eor v29.16b, v1.16b, v6.16b + eor x26, x29, x23 + eor v29.16b, v29.16b, v11.16b + eor x29, x4, x5 + eor x30, x29, x1 + eor v29.16b, v29.16b, v16.16b + eor x0, x27, x8 + eor v29.16b, v29.16b, v21.16b + eor x29, x30, x2 + eor v28.16b, v2.16b, v7.16b + eor x30, x19, x20 + eor x30, x30, x16 + eor v28.16b, v28.16b, v12.16b + eor x27, x26, x0, ror #63 + eor v28.16b, v28.16b, v17.16b + eor x4, x4, x27 + eor v28.16b, v28.16b, v22.16b + eor x30, x30, x17 + eor x30, x30, x28 + eor v27.16b, v3.16b, v8.16b + eor x29, x29, x3 + eor v27.16b, v27.16b, v13.16b + eor x0, x0, x30, ror #63 + eor v27.16b, v27.16b, v18.16b + eor x30, x30, x29, ror #63 + eor x22, x22, x30 + eor v27.16b, v27.16b, v23.16b + eor x23, x23, x30 + eor v26.16b, v4.16b, v9.16b + str x23, [sp, #0xd0] + eor v26.16b, v26.16b, v14.16b + eor x23, x14, x15 + eor x14, x14, x0 + eor v26.16b, v26.16b, v19.16b + eor x23, x23, x11 + eor v26.16b, v26.16b, v24.16b + eor x15, x15, x0 + eor x1, x1, x27 + add v31.2d, v28.2d, v28.2d + eor x23, x23, x12 + sri v31.2d, v28.2d, #0x3f + eor x23, x23, x13 + eor v25.16b, v31.16b, v30.16b + eor x11, x11, x0 + eor x29, x29, x23, ror #63 + add v31.2d, v26.2d, v26.2d + eor x23, x23, x26, ror #63 + sri v31.2d, v26.2d, #0x3f + eor x26, x13, x0 + eor v28.16b, v31.16b, v28.16b + eor x13, x28, x23 + eor x28, x24, x30 + add v31.2d, v29.2d, v29.2d + eor x24, x16, x23 + sri v31.2d, v29.2d, #0x3f + eor x16, x21, x30 + eor v26.16b, v31.16b, v26.16b + eor x21, x25, x30 + eor x30, x19, x23 + add v31.2d, v27.2d, v27.2d + eor x19, x20, x23 + sri v31.2d, v27.2d, #0x3f + eor x20, x17, x23 + eor v29.16b, v31.16b, v29.16b + eor x17, x12, x0 + eor x0, x2, x27 + add v31.2d, v30.2d, v30.2d + eor x2, x6, x29 + sri v31.2d, v30.2d, #0x3f + eor x6, x8, x29 + eor v27.16b, v31.16b, v27.16b + bic x8, x28, x13, ror #47 + eor x12, x3, x27 + eor v30.16b, v0.16b, v26.16b + bic x3, x13, x17, ror #19 + eor v31.16b, v2.16b, v29.16b + eor x5, x5, x27 + ldr x27, [sp, #0xd0] + shl v0.2d, v31.2d, #0x3e + bic x25, x17, x2, ror #5 + sri v0.2d, v31.2d, #0x2 + eor x9, x9, x29 + eor v31.16b, v12.16b, v29.16b + eor x23, x25, x5, ror #52 + eor x3, x3, x2, ror #24 + shl v2.2d, v31.2d, #0x2b + eor x8, x8, x17, ror #2 + sri v2.2d, v31.2d, #0x15 + eor x17, x10, x29 + eor v31.16b, v13.16b, v28.16b + bic x25, x12, x22, ror #47 + eor x29, x7, x29 + shl v12.2d, v31.2d, #0x19 + bic x10, x4, x27, ror #2 + sri v12.2d, v31.2d, #0x27 + bic x7, x5, x28, ror #10 + eor v31.16b, v19.16b, v27.16b + eor x10, x10, x20, ror #50 + eor x13, x7, x13, ror #57 + shl v13.2d, v31.2d, #0x8 + bic x7, x2, x5, ror #47 + sri v13.2d, v31.2d, #0x38 + eor x2, x25, x24, ror #39 + eor v31.16b, v23.16b, v28.16b + bic x25, x20, x11, ror #57 + bic x5, x17, x4, ror #25 + shl v19.2d, v31.2d, #0x38 + eor x25, x25, x17, ror #53 + sri v19.2d, v31.2d, #0x8 + bic x17, x11, x17, ror #60 + eor v31.16b, v15.16b, v26.16b + eor x28, x7, x28, ror #57 + bic x7, x9, x12, ror #42 + shl v23.2d, v31.2d, #0x29 + eor x7, x7, x22, ror #25 + sri v23.2d, v31.2d, #0x17 + bic x22, x22, x24, ror #56 + bic x24, x24, x15, ror #31 + eor v31.16b, v1.16b, v25.16b + eor x22, x22, x15, ror #23 + shl v15.2d, v31.2d, #0x1 + bic x20, x27, x20, ror #48 + sri v15.2d, v31.2d, #0x3f + bic x15, x15, x9, ror #16 + eor x12, x15, x12, ror #58 + eor v31.16b, v8.16b, v28.16b + eor x15, x5, x27, ror #27 + shl v1.2d, v31.2d, #0x37 + eor x5, x20, x11, ror #41 + sri v1.2d, v31.2d, #0x9 + ldr x11, [sp, #0x8] + eor x20, x17, x4, ror #21 + eor v31.16b, v16.16b, v25.16b + eor x17, x24, x9, ror #47 + shl v8.2d, v31.2d, #0x2d + mov x24, #0x1 // =1 + sri v8.2d, v31.2d, #0x13 + bic x9, x0, x16, ror #9 + str x24, [sp, #0x18] + eor v31.16b, v7.16b, v29.16b + bic x24, x29, x1, ror #44 + shl v16.2d, v31.2d, #0x6 + bic x27, x1, x21, ror #50 + sri v16.2d, v31.2d, #0x3a + bic x4, x26, x29, ror #63 + eor x1, x1, x4, ror #21 + eor v31.16b, v10.16b, v26.16b + ldr x11, [x11] + shl v7.2d, v31.2d, #0x3 + bic x4, x21, x30, ror #57 + sri v7.2d, v31.2d, #0x3d + eor x21, x24, x21, ror #30 + eor x24, x9, x19, ror #44 + eor v31.16b, v3.16b, v28.16b + bic x9, x14, x6, ror #5 + shl v10.2d, v31.2d, #0x1c + eor x9, x9, x0, ror #43 + sri v10.2d, v31.2d, #0x24 + bic x0, x6, x0, ror #38 + eor x1, x1, x11 + eor v31.16b, v18.16b, v28.16b + eor x11, x4, x26, ror #35 + shl v3.2d, v31.2d, #0x15 + eor x4, x0, x16, ror #47 + bic x0, x16, x19, ror #35 + sri v3.2d, v31.2d, #0x2b + eor x16, x27, x30, ror #43 + eor v31.16b, v17.16b, v29.16b + bic x27, x30, x26, ror #42 + shl v18.2d, v31.2d, #0xf + bic x26, x19, x14, ror #41 + eor x19, x0, x14, ror #12 + sri v18.2d, v31.2d, #0x31 + eor x14, x26, x6, ror #46 + eor v31.16b, v11.16b, v25.16b + eor x6, x27, x29, ror #41 + shl v17.2d, v31.2d, #0xa + eor x0, x15, x11, ror #52 + eor x0, x0, x13, ror #48 + sri v17.2d, v31.2d, #0x36 + eor x26, x8, x9, ror #57 + eor v31.16b, v9.16b, v27.16b + eor x27, x0, x14, ror #10 + shl v11.2d, v31.2d, #0x14 + eor x29, x16, x28, ror #63 + eor x26, x26, x6, ror #51 + sri v11.2d, v31.2d, #0x2c + eor x30, x23, x22, ror #50 + eor v31.16b, v22.16b, v29.16b + eor x0, x26, x10, ror #31 + shl v9.2d, v31.2d, #0x3d + eor x29, x29, x19, ror #37 + eor x27, x27, x12, ror #5 + sri v9.2d, v31.2d, #0x3 + eor x30, x30, x24, ror #34 + eor v31.16b, v14.16b, v27.16b + eor x0, x0, x7, ror #27 + shl v22.2d, v31.2d, #0x27 + eor x26, x30, x21, ror #26 + eor x26, x26, x25, ror #15 + sri v22.2d, v31.2d, #0x19 + ror x30, x27, #0x3e + eor v31.16b, v20.16b, v26.16b + eor x30, x30, x26, ror #57 + ror x26, x26, #0x3a + shl v14.2d, v31.2d, #0x12 + eor x16, x30, x16 + sri v14.2d, v31.2d, #0x2e + eor x28, x30, x28, ror #63 + eor v31.16b, v4.16b, v27.16b + str x28, [sp, #0xd0] + eor x29, x29, x17, ror #36 + shl v20.2d, v31.2d, #0x1b + eor x28, x1, x2, ror #61 + sri v20.2d, v31.2d, #0x25 + eor x19, x30, x19, ror #37 + eor v31.16b, v24.16b, v27.16b + eor x29, x29, x20, ror #2 + eor x28, x28, x4, ror #54 + shl v4.2d, v31.2d, #0xe + eor x26, x26, x0, ror #55 + sri v4.2d, v31.2d, #0x32 + eor x28, x28, x3, ror #39 + eor v31.16b, v21.16b, v25.16b + eor x28, x28, x5, ror #25 + ror x0, x0, #0x38 + shl v24.2d, v31.2d, #0x2 + eor x0, x0, x29, ror #63 + sri v24.2d, v31.2d, #0x3e + eor x27, x28, x27, ror #61 + eor v31.16b, v5.16b, v26.16b + eor x13, x0, x13, ror #46 + eor x28, x29, x28, ror #63 + shl v21.2d, v31.2d, #0x24 + eor x29, x30, x20, ror #2 + sri v21.2d, v31.2d, #0x1c + eor x20, x26, x3, ror #39 + eor v31.16b, v6.16b, v25.16b + eor x11, x0, x11, ror #50 + eor x25, x28, x25, ror #9 + shl v27.2d, v31.2d, #0x2c + eor x3, x28, x21, ror #20 + sri v27.2d, v31.2d, #0x14 + eor x21, x26, x1 + eor x9, x27, x9, ror #49 + bic v31.16b, v7.16b, v11.16b + eor x24, x28, x24, ror #28 + eor v5.16b, v31.16b, v10.16b + eor x1, x30, x17, ror #36 + bic v31.16b, v8.16b, v7.16b + eor x14, x0, x14, ror #8 + eor x22, x28, x22, ror #44 + eor v6.16b, v31.16b, v11.16b + eor x8, x27, x8, ror #56 + bic v31.16b, v9.16b, v8.16b + eor x17, x27, x7, ror #19 + eor v7.16b, v31.16b, v7.16b + eor x15, x0, x15, ror #62 + bic x7, x20, x22, ror #47 + bic v31.16b, v10.16b, v9.16b + eor x4, x26, x4, ror #54 + eor v8.16b, v31.16b, v8.16b + eor x0, x0, x12, ror #3 + bic v31.16b, v11.16b, v10.16b + eor x28, x28, x23, ror #58 + eor x23, x26, x2, ror #61 + eor v9.16b, v31.16b, v9.16b + eor x26, x26, x5, ror #25 + bic v31.16b, v12.16b, v16.16b + eor x2, x7, x16, ror #39 + eor v10.16b, v31.16b, v15.16b + bic x7, x9, x20, ror #42 + bic x30, x15, x9, ror #16 + bic v31.16b, v13.16b, v12.16b + eor x7, x7, x22, ror #25 + eor v11.16b, v31.16b, v16.16b + eor x12, x30, x20, ror #58 + bic v31.16b, v14.16b, v13.16b + bic x20, x22, x16, ror #56 + eor x30, x27, x6, ror #43 + eor v12.16b, v31.16b, v12.16b + eor x22, x20, x15, ror #23 + bic v31.16b, v15.16b, v14.16b + bic x6, x19, x13, ror #42 + eor v13.16b, v31.16b, v13.16b + eor x6, x6, x17, ror #41 + bic x5, x13, x17, ror #63 + bic v31.16b, v16.16b, v15.16b + eor x5, x21, x5, ror #21 + eor v14.16b, v31.16b, v14.16b + bic x17, x17, x21, ror #44 + eor x27, x27, x10, ror #23 + bic v31.16b, v17.16b, v21.16b + bic x21, x21, x25, ror #50 + eor v15.16b, v31.16b, v20.16b + bic x20, x27, x4, ror #25 + bic v31.16b, v18.16b, v17.16b + bic x10, x16, x15, ror #31 + eor x16, x21, x19, ror #43 + eor v16.16b, v31.16b, v21.16b + eor x21, x17, x25, ror #30 + bic v31.16b, v19.16b, v18.16b + bic x19, x25, x19, ror #57 + eor v17.16b, v31.16b, v17.16b + ldr x25, [sp, #0x18] + eor x17, x10, x9, ror #47 + bic v31.16b, v20.16b, v19.16b + ldr x9, [sp, #0x8] + eor v18.16b, v31.16b, v18.16b + eor x15, x20, x28, ror #27 + bic v31.16b, v21.16b, v20.16b + bic x20, x4, x28, ror #2 + eor x10, x20, x1, ror #50 + eor v19.16b, v31.16b, v19.16b + bic x20, x11, x27, ror #60 + bic v31.16b, v22.16b, v1.16b + eor x20, x20, x4, ror #21 + eor v20.16b, v31.16b, v0.16b + bic x4, x28, x1, ror #48 + bic x1, x1, x11, ror #57 + bic v31.16b, v23.16b, v22.16b + ldr x28, [x9, x25, lsl #3] + eor v21.16b, v31.16b, v1.16b + ldr x9, [sp, #0xd0] + bic v31.16b, v24.16b, v23.16b + add x25, x25, #0x1 + str x25, [sp, #0x18] + eor v22.16b, v31.16b, v22.16b + cmp x25, #0x17 + bic v31.16b, v0.16b, v24.16b + eor x25, x1, x27, ror #53 + bic x27, x30, x26, ror #47 + eor v23.16b, v31.16b, v23.16b + eor x1, x5, x28 + bic v31.16b, v1.16b, v0.16b + eor x5, x4, x11, ror #41 + eor v24.16b, v31.16b, v24.16b + eor x11, x19, x13, ror #35 + bic x13, x26, x24, ror #10 + bic v31.16b, v2.16b, v27.16b + eor x28, x27, x24, ror #57 + eor v0.16b, v31.16b, v30.16b + bic x27, x24, x9, ror #47 + bic v31.16b, v3.16b, v2.16b + bic x19, x23, x3, ror #9 + bic x4, x29, x14, ror #41 + eor v1.16b, v31.16b, v27.16b + eor x24, x19, x29, ror #44 + bic v31.16b, v4.16b, v3.16b + bic x29, x3, x29, ror #35 + eor v2.16b, v31.16b, v2.16b + eor x13, x13, x9, ror #57 + eor x19, x29, x14, ror #12 + bic v31.16b, v30.16b, v4.16b + bic x29, x9, x0, ror #19 + eor v3.16b, v31.16b, v3.16b + bic x14, x14, x8, ror #5 + bic v31.16b, v27.16b, v30.16b + eor x9, x14, x23, ror #43 + eor x14, x4, x8, ror #46 + eor v4.16b, v31.16b, v4.16b + bic x23, x8, x23, ror #38 + eor x8, x27, x0, ror #2 + eor x4, x23, x3, ror #47 + bic x3, x0, x30, ror #5 + eor x23, x3, x26, ror #52 + eor x3, x29, x30, ror #24 + ldr x30, [sp, #0x10] + ld1r { v28.2d }, [x30], #8 + str x30, [sp, #0x10] + eor v0.16b, v0.16b, v28.16b + +Lmlk_keccak_f1600_x4_v8a_scalar_hybrid_loop: + eor x0, x15, x11, ror #52 + eor x0, x0, x13, ror #48 + eor v30.16b, v0.16b, v5.16b + eor v30.16b, v30.16b, v10.16b + eor x26, x8, x9, ror #57 + eor v30.16b, v30.16b, v15.16b + eor x27, x0, x14, ror #10 + eor x29, x16, x28, ror #63 + eor v30.16b, v30.16b, v20.16b + eor x26, x26, x6, ror #51 + eor v29.16b, v1.16b, v6.16b + eor x30, x23, x22, ror #50 + eor v29.16b, v29.16b, v11.16b + eor x0, x26, x10, ror #31 + eor x29, x29, x19, ror #37 + eor v29.16b, v29.16b, v16.16b + eor x27, x27, x12, ror #5 + eor v29.16b, v29.16b, v21.16b + eor x30, x30, x24, ror #34 + eor x0, x0, x7, ror #27 + eor v28.16b, v2.16b, v7.16b + eor x26, x30, x21, ror #26 + eor v28.16b, v28.16b, v12.16b + eor x26, x26, x25, ror #15 + eor v28.16b, v28.16b, v17.16b + ror x30, x27, #0x3e + eor x30, x30, x26, ror #57 + eor v28.16b, v28.16b, v22.16b + ror x26, x26, #0x3a + eor v27.16b, v3.16b, v8.16b + eor x16, x30, x16 + eor v27.16b, v27.16b, v13.16b + eor x28, x30, x28, ror #63 + str x28, [sp, #0xd0] + eor v27.16b, v27.16b, v18.16b + eor x29, x29, x17, ror #36 + eor v27.16b, v27.16b, v23.16b + eor x28, x1, x2, ror #61 + eor x19, x30, x19, ror #37 + eor v26.16b, v4.16b, v9.16b + eor x29, x29, x20, ror #2 + eor v26.16b, v26.16b, v14.16b + eor x28, x28, x4, ror #54 + eor v26.16b, v26.16b, v19.16b + eor x26, x26, x0, ror #55 + eor x28, x28, x3, ror #39 + eor v26.16b, v26.16b, v24.16b + eor x28, x28, x5, ror #25 + add v31.2d, v28.2d, v28.2d + ror x0, x0, #0x38 + eor x0, x0, x29, ror #63 + sri v31.2d, v28.2d, #0x3f + eor x27, x28, x27, ror #61 + eor v25.16b, v31.16b, v30.16b + eor x13, x0, x13, ror #46 + add v31.2d, v26.2d, v26.2d + eor x28, x29, x28, ror #63 + eor x29, x30, x20, ror #2 + sri v31.2d, v26.2d, #0x3f + eor x20, x26, x3, ror #39 + eor v28.16b, v31.16b, v28.16b + eor x11, x0, x11, ror #50 + add v31.2d, v29.2d, v29.2d + eor x25, x28, x25, ror #9 + eor x3, x28, x21, ror #20 + sri v31.2d, v29.2d, #0x3f + eor x21, x26, x1 + eor v26.16b, v31.16b, v26.16b + eor x9, x27, x9, ror #49 + eor x24, x28, x24, ror #28 + add v31.2d, v27.2d, v27.2d + eor x1, x30, x17, ror #36 + sri v31.2d, v27.2d, #0x3f + eor x14, x0, x14, ror #8 + eor v29.16b, v31.16b, v29.16b + eor x22, x28, x22, ror #44 + eor x8, x27, x8, ror #56 + add v31.2d, v30.2d, v30.2d + eor x17, x27, x7, ror #19 + sri v31.2d, v30.2d, #0x3f + eor x15, x0, x15, ror #62 + bic x7, x20, x22, ror #47 + eor v27.16b, v31.16b, v27.16b + eor x4, x26, x4, ror #54 + eor v30.16b, v0.16b, v26.16b + eor x0, x0, x12, ror #3 + eor v31.16b, v2.16b, v29.16b + eor x28, x28, x23, ror #58 + eor x23, x26, x2, ror #61 + shl v0.2d, v31.2d, #0x3e + eor x26, x26, x5, ror #25 + sri v0.2d, v31.2d, #0x2 + eor x2, x7, x16, ror #39 + eor v31.16b, v12.16b, v29.16b + bic x7, x9, x20, ror #42 + bic x30, x15, x9, ror #16 + shl v2.2d, v31.2d, #0x2b + eor x7, x7, x22, ror #25 + sri v2.2d, v31.2d, #0x15 + eor x12, x30, x20, ror #58 + bic x20, x22, x16, ror #56 + eor v31.16b, v13.16b, v28.16b + eor x30, x27, x6, ror #43 + shl v12.2d, v31.2d, #0x19 + eor x22, x20, x15, ror #23 + sri v12.2d, v31.2d, #0x27 + bic x6, x19, x13, ror #42 + eor x6, x6, x17, ror #41 + eor v31.16b, v19.16b, v27.16b + bic x5, x13, x17, ror #63 + shl v13.2d, v31.2d, #0x8 + eor x5, x21, x5, ror #21 + sri v13.2d, v31.2d, #0x38 + bic x17, x17, x21, ror #44 + eor x27, x27, x10, ror #23 + eor v31.16b, v23.16b, v28.16b + bic x21, x21, x25, ror #50 + shl v19.2d, v31.2d, #0x38 + bic x20, x27, x4, ror #25 + bic x10, x16, x15, ror #31 + sri v19.2d, v31.2d, #0x8 + eor x16, x21, x19, ror #43 + eor v31.16b, v15.16b, v26.16b + eor x21, x17, x25, ror #30 + shl v23.2d, v31.2d, #0x29 + bic x19, x25, x19, ror #57 + ldr x25, [sp, #0x18] + sri v23.2d, v31.2d, #0x17 + eor x17, x10, x9, ror #47 + eor v31.16b, v1.16b, v25.16b + ldr x9, [sp, #0x8] + eor x15, x20, x28, ror #27 + shl v15.2d, v31.2d, #0x1 + bic x20, x4, x28, ror #2 + sri v15.2d, v31.2d, #0x3f + eor x10, x20, x1, ror #50 + eor v31.16b, v8.16b, v28.16b + bic x20, x11, x27, ror #60 + eor x20, x20, x4, ror #21 + shl v1.2d, v31.2d, #0x37 + bic x4, x28, x1, ror #48 + sri v1.2d, v31.2d, #0x9 + bic x1, x1, x11, ror #57 + eor v31.16b, v16.16b, v25.16b + ldr x28, [x9, x25, lsl #3] + ldr x9, [sp, #0xd0] + shl v8.2d, v31.2d, #0x2d + add x25, x25, #0x1 + sri v8.2d, v31.2d, #0x13 + str x25, [sp, #0x18] + cmp x25, #0x17 + eor v31.16b, v7.16b, v29.16b + eor x25, x1, x27, ror #53 + shl v16.2d, v31.2d, #0x6 + bic x27, x30, x26, ror #47 + sri v16.2d, v31.2d, #0x3a + eor x1, x5, x28 + eor x5, x4, x11, ror #41 + eor v31.16b, v10.16b, v26.16b + eor x11, x19, x13, ror #35 + shl v7.2d, v31.2d, #0x3 + bic x13, x26, x24, ror #10 + eor x28, x27, x24, ror #57 + sri v7.2d, v31.2d, #0x3d + bic x27, x24, x9, ror #47 + eor v31.16b, v3.16b, v28.16b + bic x19, x23, x3, ror #9 + shl v10.2d, v31.2d, #0x1c + bic x4, x29, x14, ror #41 + eor x24, x19, x29, ror #44 + sri v10.2d, v31.2d, #0x24 + bic x29, x3, x29, ror #35 + eor v31.16b, v18.16b, v28.16b + eor x13, x13, x9, ror #57 + shl v3.2d, v31.2d, #0x15 + eor x19, x29, x14, ror #12 + bic x29, x9, x0, ror #19 + sri v3.2d, v31.2d, #0x2b + bic x14, x14, x8, ror #5 + eor v31.16b, v17.16b, v29.16b + eor x9, x14, x23, ror #43 + eor x14, x4, x8, ror #46 + shl v18.2d, v31.2d, #0xf + bic x23, x8, x23, ror #38 + sri v18.2d, v31.2d, #0x31 + eor x8, x27, x0, ror #2 + eor v31.16b, v11.16b, v25.16b + eor x4, x23, x3, ror #47 + bic x3, x0, x30, ror #5 + shl v17.2d, v31.2d, #0xa + eor x23, x3, x26, ror #52 + sri v17.2d, v31.2d, #0x36 + eor x3, x29, x30, ror #24 + eor x0, x15, x11, ror #52 + eor v31.16b, v9.16b, v27.16b + eor x0, x0, x13, ror #48 + shl v11.2d, v31.2d, #0x14 + eor x26, x8, x9, ror #57 + sri v11.2d, v31.2d, #0x2c + eor x27, x0, x14, ror #10 + eor x29, x16, x28, ror #63 + eor v31.16b, v22.16b, v29.16b + eor x26, x26, x6, ror #51 + shl v9.2d, v31.2d, #0x3d + eor x30, x23, x22, ror #50 + sri v9.2d, v31.2d, #0x3 + eor x0, x26, x10, ror #31 + eor x29, x29, x19, ror #37 + eor v31.16b, v14.16b, v27.16b + eor x27, x27, x12, ror #5 + shl v22.2d, v31.2d, #0x27 + eor x30, x30, x24, ror #34 + eor x0, x0, x7, ror #27 + sri v22.2d, v31.2d, #0x19 + eor x26, x30, x21, ror #26 + eor v31.16b, v20.16b, v26.16b + eor x26, x26, x25, ror #15 + shl v14.2d, v31.2d, #0x12 + ror x30, x27, #0x3e + eor x30, x30, x26, ror #57 + sri v14.2d, v31.2d, #0x2e + ror x26, x26, #0x3a + eor v31.16b, v4.16b, v27.16b + eor x16, x30, x16 + shl v20.2d, v31.2d, #0x1b + eor x28, x30, x28, ror #63 + str x28, [sp, #0xd0] + sri v20.2d, v31.2d, #0x25 + eor x29, x29, x17, ror #36 + eor v31.16b, v24.16b, v27.16b + eor x28, x1, x2, ror #61 + eor x19, x30, x19, ror #37 + shl v4.2d, v31.2d, #0xe + eor x29, x29, x20, ror #2 + sri v4.2d, v31.2d, #0x32 + eor x28, x28, x4, ror #54 + eor v31.16b, v21.16b, v25.16b + eor x26, x26, x0, ror #55 + eor x28, x28, x3, ror #39 + shl v24.2d, v31.2d, #0x2 + eor x28, x28, x5, ror #25 + sri v24.2d, v31.2d, #0x3e + ror x0, x0, #0x38 + eor x0, x0, x29, ror #63 + eor v31.16b, v5.16b, v26.16b + eor x27, x28, x27, ror #61 + shl v21.2d, v31.2d, #0x24 + eor x13, x0, x13, ror #46 + sri v21.2d, v31.2d, #0x1c + eor x28, x29, x28, ror #63 + eor x29, x30, x20, ror #2 + eor v31.16b, v6.16b, v25.16b + eor x20, x26, x3, ror #39 + shl v27.2d, v31.2d, #0x2c + eor x11, x0, x11, ror #50 + sri v27.2d, v31.2d, #0x14 + eor x25, x28, x25, ror #9 + eor x3, x28, x21, ror #20 + bic v31.16b, v7.16b, v11.16b + eor x21, x26, x1 + eor v5.16b, v31.16b, v10.16b + eor x9, x27, x9, ror #49 + eor x24, x28, x24, ror #28 + bic v31.16b, v8.16b, v7.16b + eor x1, x30, x17, ror #36 + eor v6.16b, v31.16b, v11.16b + eor x14, x0, x14, ror #8 + bic v31.16b, v9.16b, v8.16b + eor x22, x28, x22, ror #44 + eor x8, x27, x8, ror #56 + eor v7.16b, v31.16b, v7.16b + eor x17, x27, x7, ror #19 + bic v31.16b, v10.16b, v9.16b + eor x15, x0, x15, ror #62 + bic x7, x20, x22, ror #47 + eor v8.16b, v31.16b, v8.16b + eor x4, x26, x4, ror #54 + bic v31.16b, v11.16b, v10.16b + eor x0, x0, x12, ror #3 + eor v9.16b, v31.16b, v9.16b + eor x28, x28, x23, ror #58 + eor x23, x26, x2, ror #61 + bic v31.16b, v12.16b, v16.16b + eor x26, x26, x5, ror #25 + eor v10.16b, v31.16b, v15.16b + eor x2, x7, x16, ror #39 + bic v31.16b, v13.16b, v12.16b + bic x7, x9, x20, ror #42 + bic x30, x15, x9, ror #16 + eor v11.16b, v31.16b, v16.16b + eor x7, x7, x22, ror #25 + bic v31.16b, v14.16b, v13.16b + eor x12, x30, x20, ror #58 + bic x20, x22, x16, ror #56 + eor v12.16b, v31.16b, v12.16b + eor x30, x27, x6, ror #43 + bic v31.16b, v15.16b, v14.16b + eor x22, x20, x15, ror #23 + eor v13.16b, v31.16b, v13.16b + bic x6, x19, x13, ror #42 + eor x6, x6, x17, ror #41 + bic v31.16b, v16.16b, v15.16b + bic x5, x13, x17, ror #63 + eor v14.16b, v31.16b, v14.16b + eor x5, x21, x5, ror #21 + bic v31.16b, v17.16b, v21.16b + bic x17, x17, x21, ror #44 + eor x27, x27, x10, ror #23 + eor v15.16b, v31.16b, v20.16b + bic x21, x21, x25, ror #50 + bic v31.16b, v18.16b, v17.16b + bic x20, x27, x4, ror #25 + bic x10, x16, x15, ror #31 + eor v16.16b, v31.16b, v21.16b + eor x16, x21, x19, ror #43 + bic v31.16b, v19.16b, v18.16b + eor x21, x17, x25, ror #30 + eor v17.16b, v31.16b, v17.16b + bic x19, x25, x19, ror #57 + ldr x25, [sp, #0x18] + bic v31.16b, v20.16b, v19.16b + eor x17, x10, x9, ror #47 + eor v18.16b, v31.16b, v18.16b + ldr x9, [sp, #0x8] + eor x15, x20, x28, ror #27 + bic v31.16b, v21.16b, v20.16b + bic x20, x4, x28, ror #2 + eor v19.16b, v31.16b, v19.16b + eor x10, x20, x1, ror #50 + bic v31.16b, v22.16b, v1.16b + bic x20, x11, x27, ror #60 + eor x20, x20, x4, ror #21 + eor v20.16b, v31.16b, v0.16b + bic x4, x28, x1, ror #48 + bic v31.16b, v23.16b, v22.16b + bic x1, x1, x11, ror #57 + eor v21.16b, v31.16b, v1.16b + ldr x28, [x9, x25, lsl #3] + ldr x9, [sp, #0xd0] + bic v31.16b, v24.16b, v23.16b + add x25, x25, #0x1 + eor v22.16b, v31.16b, v22.16b + str x25, [sp, #0x18] + cmp x25, #0x17 + bic v31.16b, v0.16b, v24.16b + eor x25, x1, x27, ror #53 + eor v23.16b, v31.16b, v23.16b + bic x27, x30, x26, ror #47 + bic v31.16b, v1.16b, v0.16b + eor x1, x5, x28 + eor x5, x4, x11, ror #41 + eor v24.16b, v31.16b, v24.16b + eor x11, x19, x13, ror #35 + bic v31.16b, v2.16b, v27.16b + bic x13, x26, x24, ror #10 + eor x28, x27, x24, ror #57 + eor v0.16b, v31.16b, v30.16b + bic x27, x24, x9, ror #47 + bic v31.16b, v3.16b, v2.16b + bic x19, x23, x3, ror #9 + eor v1.16b, v31.16b, v27.16b + bic x4, x29, x14, ror #41 + eor x24, x19, x29, ror #44 + bic v31.16b, v4.16b, v3.16b + bic x29, x3, x29, ror #35 + eor v2.16b, v31.16b, v2.16b + eor x13, x13, x9, ror #57 + bic v31.16b, v30.16b, v4.16b + eor x19, x29, x14, ror #12 + bic x29, x9, x0, ror #19 + eor v3.16b, v31.16b, v3.16b + bic x14, x14, x8, ror #5 + bic v31.16b, v27.16b, v30.16b + eor x9, x14, x23, ror #43 + eor x14, x4, x8, ror #46 + eor v4.16b, v31.16b, v4.16b + bic x23, x8, x23, ror #38 + eor x8, x27, x0, ror #2 + eor x4, x23, x3, ror #47 + bic x3, x0, x30, ror #5 + eor x23, x3, x26, ror #52 + eor x3, x29, x30, ror #24 + ldr x30, [sp, #0x10] + ld1r { v28.2d }, [x30], #8 + str x30, [sp, #0x10] + eor v0.16b, v0.16b, v28.16b + +Lmlk_keccak_f1600_x4_v8a_scalar_hybrid_loop_end: + b.le Lmlk_keccak_f1600_x4_v8a_scalar_hybrid_loop + ror x2, x2, #0x3d + ror x3, x3, #0x27 + ror x4, x4, #0x36 + ror x5, x5, #0x19 + ror x6, x6, #0x2b + ror x7, x7, #0x13 + ror x8, x8, #0x38 + ror x9, x9, #0x31 + ror x10, x10, #0x17 + ror x11, x11, #0x32 + ror x12, x12, #0x3 + ror x13, x13, #0x2e + ror x14, x14, #0x8 + ror x15, x15, #0x3e + ror x17, x17, #0x24 + ror x28, x28, #0x3f + ror x19, x19, #0x25 + ror x20, x20, #0x2 + ror x21, x21, #0x14 + ror x22, x22, #0x2c + ror x23, x23, #0x3a + ror x24, x24, #0x1c + ror x25, x25, #0x9 + ldr x30, [sp, #0x20] + cmp x30, #0x1 + b.eq Lmlk_keccak_f1600_x4_v8a_scalar_hybrid_done + mov x30, #0x1 // =1 + str x30, [sp, #0x20] + ldr x0, [sp] + add x0, x0, #0x190 + stp x1, x6, [x0] + stp x11, x16, [x0, #0x10] + stp x21, x2, [x0, #0x20] + stp x7, x12, [x0, #0x30] + stp x17, x22, [x0, #0x40] + stp x3, x8, [x0, #0x50] + stp x13, x28, [x0, #0x60] + stp x23, x4, [x0, #0x70] + stp x9, x14, [x0, #0x80] + stp x19, x24, [x0, #0x90] + stp x5, x10, [x0, #0xa0] + stp x15, x20, [x0, #0xb0] + str x25, [x0, #0xc0] + sub x0, x0, #0x190 + add x0, x0, #0x258 + ldp x1, x6, [x0] + ldp x11, x16, [x0, #0x10] + ldp x21, x2, [x0, #0x20] + ldp x7, x12, [x0, #0x30] + ldp x17, x22, [x0, #0x40] + ldp x3, x8, [x0, #0x50] + ldp x13, x28, [x0, #0x60] + ldp x23, x4, [x0, #0x70] + ldp x9, x14, [x0, #0x80] + ldp x19, x24, [x0, #0x90] + ldp x5, x10, [x0, #0xa0] + ldp x15, x20, [x0, #0xb0] + ldr x25, [x0, #0xc0] + sub x0, x0, #0x258 + b Lmlk_keccak_f1600_x4_v8a_scalar_hybrid_initial + +Lmlk_keccak_f1600_x4_v8a_scalar_hybrid_done: + ldr x0, [sp] + add x0, x0, #0x258 + stp x1, x6, [x0] + stp x11, x16, [x0, #0x10] + stp x21, x2, [x0, #0x20] + stp x7, x12, [x0, #0x30] + stp x17, x22, [x0, #0x40] + stp x3, x8, [x0, #0x50] + stp x13, x28, [x0, #0x60] + stp x23, x4, [x0, #0x70] + stp x9, x14, [x0, #0x80] + stp x19, x24, [x0, #0x90] + stp x5, x10, [x0, #0xa0] + stp x15, x20, [x0, #0xb0] + str x25, [x0, #0xc0] + sub x0, x0, #0x258 + add x4, x0, #0xc8 + trn1 v25.2d, v0.2d, v1.2d + trn1 v26.2d, v2.2d, v3.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v0.2d, v1.2d + trn2 v28.2d, v2.2d, v3.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v4.2d, v5.2d + trn1 v26.2d, v6.2d, v7.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v4.2d, v5.2d + trn2 v28.2d, v6.2d, v7.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v8.2d, v9.2d + trn1 v26.2d, v10.2d, v11.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v8.2d, v9.2d + trn2 v28.2d, v10.2d, v11.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v12.2d, v13.2d + trn1 v26.2d, v14.2d, v15.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v12.2d, v13.2d + trn2 v28.2d, v14.2d, v15.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v16.2d, v17.2d + trn1 v26.2d, v18.2d, v19.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v16.2d, v17.2d + trn2 v28.2d, v18.2d, v19.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v20.2d, v21.2d + trn1 v26.2d, v22.2d, v23.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v20.2d, v21.2d + trn2 v28.2d, v22.2d, v23.2d + st1 { v27.2d, v28.2d }, [x4], #32 + str d24, [x0] + trn2 v25.2d, v24.2d, v24.2d + str d25, [x4] + ldp d8, d9, [sp, #0x90] + .cfi_restore d8 + .cfi_restore d9 + ldp d10, d11, [sp, #0xa0] + .cfi_restore d10 + .cfi_restore d11 + ldp d12, d13, [sp, #0xb0] + .cfi_restore d12 + .cfi_restore d13 + ldp d14, d15, [sp, #0xc0] + .cfi_restore d14 + .cfi_restore d15 + ldp x19, x20, [sp, #0x30] + .cfi_restore x19 + .cfi_restore x20 + ldp x21, x22, [sp, #0x40] + .cfi_restore x21 + .cfi_restore x22 + ldp x23, x24, [sp, #0x50] + .cfi_restore x23 + .cfi_restore x24 + ldp x25, x26, [sp, #0x60] + .cfi_restore x25 + .cfi_restore x26 + ldp x27, x28, [sp, #0x70] + .cfi_restore x27 + .cfi_restore x28 + ldp x29, x30, [sp, #0x80] + .cfi_restore x29 + .cfi_restore x30 + add sp, sp, #0xe0 + .cfi_adjust_cfa_offset -0xe0 + ret + .cfi_endproc + +MLK_ASM_FN_SIZE(keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm) + +#endif /* MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID && \ + !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S b/cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S @@ -0,0 +1,989 @@ +/* + * Copyright (c) The mlkem-native project authors + * Copyright (c) 2021-2022 Arm Limited + * Copyright (c) 2022 Matthias Kannwischer + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +// Author: Hanno Becker +// Author: Matthias Kannwischer + +/*yaml + Name: keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm + Description: AArch64 hybrid scalar/vector implementation of Keccak-f[1600] permutation for four sequential states with ARMv8.4-A optimizations + Signature: void mlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm(uint64_t state[100], const uint64_t rc[24]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON, SHA3] + x0: + type: buffer + size_bytes: 800 + permissions: read/write + c_parameter: uint64_t state[100] + description: Four sequential Keccak states (state0[25], state1[25], state2[25], state3[25]) + x1: + type: buffer + size_bytes: 192 + permissions: read-only + c_parameter: const uint64_t rc[24] + description: Round constants (24 x uint64_t) + Stack: + bytes: 224 + description: register preservation and temporary storage +*/ + +#include "../../../../common.h" +#if defined(MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +#if defined(__ARM_FEATURE_SHA3) + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/fips202/aarch64/src/keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm) +MLK_ASM_FN_SYMBOL(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0xe0 + .cfi_adjust_cfa_offset 0xe0 + stp x19, x20, [sp, #0x30] + .cfi_rel_offset x19, 0x30 + .cfi_rel_offset x20, 0x38 + stp x21, x22, [sp, #0x40] + .cfi_rel_offset x21, 0x40 + .cfi_rel_offset x22, 0x48 + stp x23, x24, [sp, #0x50] + .cfi_rel_offset x23, 0x50 + .cfi_rel_offset x24, 0x58 + stp x25, x26, [sp, #0x60] + .cfi_rel_offset x25, 0x60 + .cfi_rel_offset x26, 0x68 + stp x27, x28, [sp, #0x70] + .cfi_rel_offset x27, 0x70 + .cfi_rel_offset x28, 0x78 + stp x29, x30, [sp, #0x80] + .cfi_rel_offset x29, 0x80 + .cfi_rel_offset x30, 0x88 + stp d8, d9, [sp, #0x90] + .cfi_rel_offset d8, 0x90 + .cfi_rel_offset d9, 0x98 + stp d10, d11, [sp, #0xa0] + .cfi_rel_offset d10, 0xa0 + .cfi_rel_offset d11, 0xa8 + stp d12, d13, [sp, #0xb0] + .cfi_rel_offset d12, 0xb0 + .cfi_rel_offset d13, 0xb8 + stp d14, d15, [sp, #0xc0] + .cfi_rel_offset d14, 0xc0 + .cfi_rel_offset d15, 0xc8 + mov x29, x1 + mov x30, #0x0 // =0 + str x30, [sp, #0x20] + str x29, [sp, #0x8] + str x29, [sp, #0x10] + str x0, [sp] + add x4, x0, #0xc8 + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v0.2d, v25.2d, v27.2d + trn2 v1.2d, v25.2d, v27.2d + trn1 v2.2d, v26.2d, v28.2d + trn2 v3.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v4.2d, v25.2d, v27.2d + trn2 v5.2d, v25.2d, v27.2d + trn1 v6.2d, v26.2d, v28.2d + trn2 v7.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v8.2d, v25.2d, v27.2d + trn2 v9.2d, v25.2d, v27.2d + trn1 v10.2d, v26.2d, v28.2d + trn2 v11.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v12.2d, v25.2d, v27.2d + trn2 v13.2d, v25.2d, v27.2d + trn1 v14.2d, v26.2d, v28.2d + trn2 v15.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v16.2d, v25.2d, v27.2d + trn2 v17.2d, v25.2d, v27.2d + trn1 v18.2d, v26.2d, v28.2d + trn2 v19.2d, v26.2d, v28.2d + ldp q25, q26, [x0], #0x20 + ld1 { v27.2d, v28.2d }, [x4], #32 + trn1 v20.2d, v25.2d, v27.2d + trn2 v21.2d, v25.2d, v27.2d + trn1 v22.2d, v26.2d, v28.2d + trn2 v23.2d, v26.2d, v28.2d + ldr d25, [x0] + ldr d27, [x4] + trn1 v24.2d, v25.2d, v27.2d + sub x0, x0, #0xc0 + add x0, x0, #0x190 + ldp x1, x6, [x0] + ldp x11, x16, [x0, #0x10] + ldp x21, x2, [x0, #0x20] + ldp x7, x12, [x0, #0x30] + ldp x17, x22, [x0, #0x40] + ldp x3, x8, [x0, #0x50] + ldp x13, x28, [x0, #0x60] + ldp x23, x4, [x0, #0x70] + ldp x9, x14, [x0, #0x80] + ldp x19, x24, [x0, #0x90] + ldp x5, x10, [x0, #0xa0] + ldp x15, x20, [x0, #0xb0] + ldr x25, [x0, #0xc0] + sub x0, x0, #0x190 + +Lmlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_initial: + eor x30, x24, x25 + eor x27, x9, x10 + eor3 v30.16b, v0.16b, v5.16b, v10.16b + eor v30.16b, v30.16b, v15.16b + eor x0, x30, x21 + eor x26, x27, x6 + eor v30.16b, v30.16b, v20.16b + eor x27, x26, x7 + eor x29, x0, x22 + eor3 v29.16b, v1.16b, v6.16b, v11.16b + eor x26, x29, x23 + eor x29, x4, x5 + eor v29.16b, v29.16b, v16.16b + eor x30, x29, x1 + eor x0, x27, x8 + eor v29.16b, v29.16b, v21.16b + eor x29, x30, x2 + eor x30, x19, x20 + eor3 v28.16b, v2.16b, v7.16b, v12.16b + eor x30, x30, x16 + eor x27, x26, x0, ror #63 + eor v28.16b, v28.16b, v17.16b + eor x4, x4, x27 + eor x30, x30, x17 + eor v28.16b, v28.16b, v22.16b + eor x30, x30, x28 + eor x29, x29, x3 + eor3 v27.16b, v3.16b, v8.16b, v13.16b + eor x0, x0, x30, ror #63 + eor x30, x30, x29, ror #63 + eor v27.16b, v27.16b, v18.16b + eor x22, x22, x30 + eor v27.16b, v27.16b, v23.16b + eor x23, x23, x30 + str x23, [sp, #0xd0] + eor3 v26.16b, v4.16b, v9.16b, v14.16b + eor x23, x14, x15 + eor x14, x14, x0 + eor v26.16b, v26.16b, v19.16b + eor x23, x23, x11 + eor x15, x15, x0 + eor v26.16b, v26.16b, v24.16b + eor x1, x1, x27 + eor x23, x23, x12 + rax1 v25.2d, v30.2d, v28.2d + eor x23, x23, x13 + eor x11, x11, x0 + add v31.2d, v26.2d, v26.2d + eor x29, x29, x23, ror #63 + eor x23, x23, x26, ror #63 + sri v31.2d, v26.2d, #0x3f + eor x26, x13, x0 + eor x13, x28, x23 + eor v28.16b, v31.16b, v28.16b + eor x28, x24, x30 + eor x24, x16, x23 + rax1 v26.2d, v26.2d, v29.2d + eor x16, x21, x30 + eor x21, x25, x30 + add v31.2d, v27.2d, v27.2d + eor x30, x19, x23 + sri v31.2d, v27.2d, #0x3f + eor x19, x20, x23 + eor x20, x17, x23 + eor v29.16b, v31.16b, v29.16b + eor x17, x12, x0 + eor x0, x2, x27 + rax1 v27.2d, v27.2d, v30.2d + eor x2, x6, x29 + eor x6, x8, x29 + eor v30.16b, v0.16b, v26.16b + bic x8, x28, x13, ror #47 + eor x12, x3, x27 + eor v31.16b, v2.16b, v29.16b + bic x3, x13, x17, ror #19 + eor x5, x5, x27 + shl v0.2d, v31.2d, #0x3e + ldr x27, [sp, #0xd0] + bic x25, x17, x2, ror #5 + sri v0.2d, v31.2d, #0x2 + eor x9, x9, x29 + eor x23, x25, x5, ror #52 + xar v2.2d, v12.2d, v29.2d, #0x15 + eor x3, x3, x2, ror #24 + eor x8, x8, x17, ror #2 + eor v31.16b, v13.16b, v28.16b + eor x17, x10, x29 + bic x25, x12, x22, ror #47 + shl v12.2d, v31.2d, #0x19 + eor x29, x7, x29 + bic x10, x4, x27, ror #2 + sri v12.2d, v31.2d, #0x27 + bic x7, x5, x28, ror #10 + xar v13.2d, v19.2d, v27.2d, #0x38 + eor x10, x10, x20, ror #50 + eor x13, x7, x13, ror #57 + eor v31.16b, v23.16b, v28.16b + bic x7, x2, x5, ror #47 + eor x2, x25, x24, ror #39 + shl v19.2d, v31.2d, #0x38 + bic x25, x20, x11, ror #57 + bic x5, x17, x4, ror #25 + sri v19.2d, v31.2d, #0x8 + eor x25, x25, x17, ror #53 + bic x17, x11, x17, ror #60 + xar v23.2d, v15.2d, v26.2d, #0x17 + eor x28, x7, x28, ror #57 + bic x7, x9, x12, ror #42 + eor v31.16b, v1.16b, v25.16b + eor x7, x7, x22, ror #25 + bic x22, x22, x24, ror #56 + shl v15.2d, v31.2d, #0x1 + bic x24, x24, x15, ror #31 + eor x22, x22, x15, ror #23 + sri v15.2d, v31.2d, #0x3f + bic x20, x27, x20, ror #48 + bic x15, x15, x9, ror #16 + xar v1.2d, v8.2d, v28.2d, #0x9 + eor x12, x15, x12, ror #58 + eor x15, x5, x27, ror #27 + eor v31.16b, v16.16b, v25.16b + eor x5, x20, x11, ror #41 + shl v8.2d, v31.2d, #0x2d + ldr x11, [sp, #0x8] + eor x20, x17, x4, ror #21 + sri v8.2d, v31.2d, #0x13 + eor x17, x24, x9, ror #47 + mov x24, #0x1 // =1 + xar v16.2d, v7.2d, v29.2d, #0x3a + bic x9, x0, x16, ror #9 + str x24, [sp, #0x18] + eor v31.16b, v10.16b, v26.16b + bic x24, x29, x1, ror #44 + bic x27, x1, x21, ror #50 + shl v7.2d, v31.2d, #0x3 + bic x4, x26, x29, ror #63 + eor x1, x1, x4, ror #21 + sri v7.2d, v31.2d, #0x3d + ldr x11, [x11] + bic x4, x21, x30, ror #57 + xar v10.2d, v3.2d, v28.2d, #0x24 + eor x21, x24, x21, ror #30 + eor x24, x9, x19, ror #44 + eor v31.16b, v18.16b, v28.16b + bic x9, x14, x6, ror #5 + eor x9, x9, x0, ror #43 + shl v3.2d, v31.2d, #0x15 + bic x0, x6, x0, ror #38 + eor x1, x1, x11 + sri v3.2d, v31.2d, #0x2b + eor x11, x4, x26, ror #35 + eor x4, x0, x16, ror #47 + xar v18.2d, v17.2d, v29.2d, #0x31 + bic x0, x16, x19, ror #35 + eor v31.16b, v11.16b, v25.16b + eor x16, x27, x30, ror #43 + bic x27, x30, x26, ror #42 + shl v17.2d, v31.2d, #0xa + bic x26, x19, x14, ror #41 + eor x19, x0, x14, ror #12 + sri v17.2d, v31.2d, #0x36 + eor x14, x26, x6, ror #46 + eor x6, x27, x29, ror #41 + xar v11.2d, v9.2d, v27.2d, #0x2c + eor x0, x15, x11, ror #52 + eor x0, x0, x13, ror #48 + eor v31.16b, v22.16b, v29.16b + eor x26, x8, x9, ror #57 + eor x27, x0, x14, ror #10 + shl v9.2d, v31.2d, #0x3d + eor x29, x16, x28, ror #63 + eor x26, x26, x6, ror #51 + sri v9.2d, v31.2d, #0x3 + eor x30, x23, x22, ror #50 + eor x0, x26, x10, ror #31 + xar v22.2d, v14.2d, v27.2d, #0x19 + eor x29, x29, x19, ror #37 + eor x27, x27, x12, ror #5 + eor v31.16b, v20.16b, v26.16b + eor x30, x30, x24, ror #34 + eor x0, x0, x7, ror #27 + shl v14.2d, v31.2d, #0x12 + eor x26, x30, x21, ror #26 + sri v14.2d, v31.2d, #0x2e + eor x26, x26, x25, ror #15 + ror x30, x27, #0x3e + xar v20.2d, v4.2d, v27.2d, #0x25 + eor x30, x30, x26, ror #57 + ror x26, x26, #0x3a + eor v31.16b, v24.16b, v27.16b + eor x16, x30, x16 + eor x28, x30, x28, ror #63 + shl v4.2d, v31.2d, #0xe + str x28, [sp, #0xd0] + eor x29, x29, x17, ror #36 + sri v4.2d, v31.2d, #0x32 + eor x28, x1, x2, ror #61 + eor x19, x30, x19, ror #37 + xar v24.2d, v21.2d, v25.2d, #0x3e + eor x29, x29, x20, ror #2 + eor x28, x28, x4, ror #54 + eor v31.16b, v5.16b, v26.16b + eor x26, x26, x0, ror #55 + eor x28, x28, x3, ror #39 + shl v21.2d, v31.2d, #0x24 + eor x28, x28, x5, ror #25 + ror x0, x0, #0x38 + sri v21.2d, v31.2d, #0x1c + eor x0, x0, x29, ror #63 + eor x27, x28, x27, ror #61 + xar v27.2d, v6.2d, v25.2d, #0x14 + eor x13, x0, x13, ror #46 + eor x28, x29, x28, ror #63 + bic v31.16b, v7.16b, v11.16b + eor x29, x30, x20, ror #2 + eor v5.16b, v31.16b, v10.16b + eor x20, x26, x3, ror #39 + eor x11, x0, x11, ror #50 + bcax v6.16b, v11.16b, v8.16b, v7.16b + eor x25, x28, x25, ror #9 + eor x3, x28, x21, ror #20 + bic v31.16b, v9.16b, v8.16b + eor x21, x26, x1 + eor x9, x27, x9, ror #49 + eor v7.16b, v31.16b, v7.16b + eor x24, x28, x24, ror #28 + eor x1, x30, x17, ror #36 + bcax v8.16b, v8.16b, v10.16b, v9.16b + eor x14, x0, x14, ror #8 + eor x22, x28, x22, ror #44 + bic v31.16b, v11.16b, v10.16b + eor x8, x27, x8, ror #56 + eor x17, x27, x7, ror #19 + eor v9.16b, v31.16b, v9.16b + eor x15, x0, x15, ror #62 + bic x7, x20, x22, ror #47 + bcax v10.16b, v15.16b, v12.16b, v16.16b + eor x4, x26, x4, ror #54 + eor x0, x0, x12, ror #3 + bic v31.16b, v13.16b, v12.16b + eor x28, x28, x23, ror #58 + eor x23, x26, x2, ror #61 + eor v11.16b, v31.16b, v16.16b + eor x26, x26, x5, ror #25 + bcax v12.16b, v12.16b, v14.16b, v13.16b + eor x2, x7, x16, ror #39 + bic x7, x9, x20, ror #42 + bic v31.16b, v15.16b, v14.16b + bic x30, x15, x9, ror #16 + eor x7, x7, x22, ror #25 + eor v13.16b, v31.16b, v13.16b + eor x12, x30, x20, ror #58 + bic x20, x22, x16, ror #56 + bic v31.16b, v16.16b, v15.16b + eor x30, x27, x6, ror #43 + eor x22, x20, x15, ror #23 + eor v14.16b, v31.16b, v14.16b + bic x6, x19, x13, ror #42 + eor x6, x6, x17, ror #41 + bcax v15.16b, v20.16b, v17.16b, v21.16b + bic x5, x13, x17, ror #63 + eor x5, x21, x5, ror #21 + bic v31.16b, v18.16b, v17.16b + bic x17, x17, x21, ror #44 + eor x27, x27, x10, ror #23 + eor v16.16b, v31.16b, v21.16b + bic x21, x21, x25, ror #50 + bic x20, x27, x4, ror #25 + bcax v17.16b, v17.16b, v19.16b, v18.16b + bic x10, x16, x15, ror #31 + eor x16, x21, x19, ror #43 + bic v31.16b, v20.16b, v19.16b + eor x21, x17, x25, ror #30 + bic x19, x25, x19, ror #57 + eor v18.16b, v31.16b, v18.16b + ldr x25, [sp, #0x18] + bcax v19.16b, v19.16b, v21.16b, v20.16b + eor x17, x10, x9, ror #47 + ldr x9, [sp, #0x8] + bic v31.16b, v22.16b, v1.16b + eor x15, x20, x28, ror #27 + bic x20, x4, x28, ror #2 + eor v20.16b, v31.16b, v0.16b + eor x10, x20, x1, ror #50 + bic x20, x11, x27, ror #60 + bcax v21.16b, v1.16b, v23.16b, v22.16b + eor x20, x20, x4, ror #21 + bic x4, x28, x1, ror #48 + bic v31.16b, v24.16b, v23.16b + bic x1, x1, x11, ror #57 + ldr x28, [x9, x25, lsl #3] + eor v22.16b, v31.16b, v22.16b + ldr x9, [sp, #0xd0] + add x25, x25, #0x1 + bcax v23.16b, v23.16b, v0.16b, v24.16b + str x25, [sp, #0x18] + cmp x25, #0x17 + bic v31.16b, v1.16b, v0.16b + eor x25, x1, x27, ror #53 + bic x27, x30, x26, ror #47 + eor v24.16b, v31.16b, v24.16b + eor x1, x5, x28 + eor x5, x4, x11, ror #41 + bcax v0.16b, v30.16b, v2.16b, v27.16b + eor x11, x19, x13, ror #35 + bic v31.16b, v3.16b, v2.16b + bic x13, x26, x24, ror #10 + eor x28, x27, x24, ror #57 + eor v1.16b, v31.16b, v27.16b + bic x27, x24, x9, ror #47 + bic x19, x23, x3, ror #9 + bcax v2.16b, v2.16b, v4.16b, v3.16b + bic x4, x29, x14, ror #41 + eor x24, x19, x29, ror #44 + bic v31.16b, v30.16b, v4.16b + bic x29, x3, x29, ror #35 + eor x13, x13, x9, ror #57 + eor v3.16b, v31.16b, v3.16b + eor x19, x29, x14, ror #12 + bic x29, x9, x0, ror #19 + bcax v4.16b, v4.16b, v27.16b, v30.16b + bic x14, x14, x8, ror #5 + eor x9, x14, x23, ror #43 + eor x14, x4, x8, ror #46 + bic x23, x8, x23, ror #38 + eor x8, x27, x0, ror #2 + eor x4, x23, x3, ror #47 + bic x3, x0, x30, ror #5 + eor x23, x3, x26, ror #52 + eor x3, x29, x30, ror #24 + ldr x30, [sp, #0x10] + ld1r { v28.2d }, [x30], #8 + str x30, [sp, #0x10] + eor v0.16b, v0.16b, v28.16b + +Lmlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_loop: + eor x0, x15, x11, ror #52 + eor x0, x0, x13, ror #48 + eor3 v30.16b, v0.16b, v5.16b, v10.16b + eor v30.16b, v30.16b, v15.16b + eor x26, x8, x9, ror #57 + eor x27, x0, x14, ror #10 + eor v30.16b, v30.16b, v20.16b + eor x29, x16, x28, ror #63 + eor x26, x26, x6, ror #51 + eor3 v29.16b, v1.16b, v6.16b, v11.16b + eor x30, x23, x22, ror #50 + eor x0, x26, x10, ror #31 + eor v29.16b, v29.16b, v16.16b + eor x29, x29, x19, ror #37 + eor x27, x27, x12, ror #5 + eor v29.16b, v29.16b, v21.16b + eor x30, x30, x24, ror #34 + eor x0, x0, x7, ror #27 + eor3 v28.16b, v2.16b, v7.16b, v12.16b + eor x26, x30, x21, ror #26 + eor x26, x26, x25, ror #15 + eor v28.16b, v28.16b, v17.16b + ror x30, x27, #0x3e + eor x30, x30, x26, ror #57 + eor v28.16b, v28.16b, v22.16b + ror x26, x26, #0x3a + eor x16, x30, x16 + eor3 v27.16b, v3.16b, v8.16b, v13.16b + eor x28, x30, x28, ror #63 + str x28, [sp, #0xd0] + eor v27.16b, v27.16b, v18.16b + eor x29, x29, x17, ror #36 + eor x28, x1, x2, ror #61 + eor v27.16b, v27.16b, v23.16b + eor x19, x30, x19, ror #37 + eor x29, x29, x20, ror #2 + eor3 v26.16b, v4.16b, v9.16b, v14.16b + eor x28, x28, x4, ror #54 + eor x26, x26, x0, ror #55 + eor v26.16b, v26.16b, v19.16b + eor x28, x28, x3, ror #39 + eor x28, x28, x5, ror #25 + eor v26.16b, v26.16b, v24.16b + ror x0, x0, #0x38 + eor x0, x0, x29, ror #63 + rax1 v25.2d, v30.2d, v28.2d + eor x27, x28, x27, ror #61 + eor x13, x0, x13, ror #46 + add v31.2d, v26.2d, v26.2d + eor x28, x29, x28, ror #63 + eor x29, x30, x20, ror #2 + sri v31.2d, v26.2d, #0x3f + eor x20, x26, x3, ror #39 + eor x11, x0, x11, ror #50 + eor v28.16b, v31.16b, v28.16b + eor x25, x28, x25, ror #9 + eor x3, x28, x21, ror #20 + rax1 v26.2d, v26.2d, v29.2d + eor x21, x26, x1 + add v31.2d, v27.2d, v27.2d + eor x9, x27, x9, ror #49 + eor x24, x28, x24, ror #28 + sri v31.2d, v27.2d, #0x3f + eor x1, x30, x17, ror #36 + eor x14, x0, x14, ror #8 + eor v29.16b, v31.16b, v29.16b + eor x22, x28, x22, ror #44 + eor x8, x27, x8, ror #56 + rax1 v27.2d, v27.2d, v30.2d + eor x17, x27, x7, ror #19 + eor x15, x0, x15, ror #62 + eor v30.16b, v0.16b, v26.16b + bic x7, x20, x22, ror #47 + eor x4, x26, x4, ror #54 + eor v31.16b, v2.16b, v29.16b + eor x0, x0, x12, ror #3 + eor x28, x28, x23, ror #58 + shl v0.2d, v31.2d, #0x3e + eor x23, x26, x2, ror #61 + eor x26, x26, x5, ror #25 + sri v0.2d, v31.2d, #0x2 + eor x2, x7, x16, ror #39 + bic x7, x9, x20, ror #42 + xar v2.2d, v12.2d, v29.2d, #0x15 + bic x30, x15, x9, ror #16 + eor x7, x7, x22, ror #25 + eor v31.16b, v13.16b, v28.16b + eor x12, x30, x20, ror #58 + bic x20, x22, x16, ror #56 + shl v12.2d, v31.2d, #0x19 + eor x30, x27, x6, ror #43 + eor x22, x20, x15, ror #23 + sri v12.2d, v31.2d, #0x27 + bic x6, x19, x13, ror #42 + eor x6, x6, x17, ror #41 + xar v13.2d, v19.2d, v27.2d, #0x38 + bic x5, x13, x17, ror #63 + eor x5, x21, x5, ror #21 + eor v31.16b, v23.16b, v28.16b + bic x17, x17, x21, ror #44 + eor x27, x27, x10, ror #23 + shl v19.2d, v31.2d, #0x38 + bic x21, x21, x25, ror #50 + bic x20, x27, x4, ror #25 + sri v19.2d, v31.2d, #0x8 + bic x10, x16, x15, ror #31 + eor x16, x21, x19, ror #43 + xar v23.2d, v15.2d, v26.2d, #0x17 + eor x21, x17, x25, ror #30 + bic x19, x25, x19, ror #57 + eor v31.16b, v1.16b, v25.16b + ldr x25, [sp, #0x18] + eor x17, x10, x9, ror #47 + shl v15.2d, v31.2d, #0x1 + ldr x9, [sp, #0x8] + sri v15.2d, v31.2d, #0x3f + eor x15, x20, x28, ror #27 + bic x20, x4, x28, ror #2 + xar v1.2d, v8.2d, v28.2d, #0x9 + eor x10, x20, x1, ror #50 + bic x20, x11, x27, ror #60 + eor v31.16b, v16.16b, v25.16b + eor x20, x20, x4, ror #21 + bic x4, x28, x1, ror #48 + shl v8.2d, v31.2d, #0x2d + bic x1, x1, x11, ror #57 + ldr x28, [x9, x25, lsl #3] + sri v8.2d, v31.2d, #0x13 + ldr x9, [sp, #0xd0] + add x25, x25, #0x1 + xar v16.2d, v7.2d, v29.2d, #0x3a + str x25, [sp, #0x18] + cmp x25, #0x17 + eor v31.16b, v10.16b, v26.16b + eor x25, x1, x27, ror #53 + bic x27, x30, x26, ror #47 + shl v7.2d, v31.2d, #0x3 + eor x1, x5, x28 + eor x5, x4, x11, ror #41 + sri v7.2d, v31.2d, #0x3d + eor x11, x19, x13, ror #35 + bic x13, x26, x24, ror #10 + xar v10.2d, v3.2d, v28.2d, #0x24 + eor x28, x27, x24, ror #57 + bic x27, x24, x9, ror #47 + eor v31.16b, v18.16b, v28.16b + bic x19, x23, x3, ror #9 + bic x4, x29, x14, ror #41 + shl v3.2d, v31.2d, #0x15 + eor x24, x19, x29, ror #44 + bic x29, x3, x29, ror #35 + sri v3.2d, v31.2d, #0x2b + eor x13, x13, x9, ror #57 + eor x19, x29, x14, ror #12 + xar v18.2d, v17.2d, v29.2d, #0x31 + bic x29, x9, x0, ror #19 + bic x14, x14, x8, ror #5 + eor v31.16b, v11.16b, v25.16b + eor x9, x14, x23, ror #43 + eor x14, x4, x8, ror #46 + shl v17.2d, v31.2d, #0xa + bic x23, x8, x23, ror #38 + eor x8, x27, x0, ror #2 + sri v17.2d, v31.2d, #0x36 + eor x4, x23, x3, ror #47 + bic x3, x0, x30, ror #5 + xar v11.2d, v9.2d, v27.2d, #0x2c + eor x23, x3, x26, ror #52 + eor x3, x29, x30, ror #24 + eor v31.16b, v22.16b, v29.16b + eor x0, x15, x11, ror #52 + shl v9.2d, v31.2d, #0x3d + eor x0, x0, x13, ror #48 + eor x26, x8, x9, ror #57 + sri v9.2d, v31.2d, #0x3 + eor x27, x0, x14, ror #10 + eor x29, x16, x28, ror #63 + xar v22.2d, v14.2d, v27.2d, #0x19 + eor x26, x26, x6, ror #51 + eor x30, x23, x22, ror #50 + eor v31.16b, v20.16b, v26.16b + eor x0, x26, x10, ror #31 + eor x29, x29, x19, ror #37 + shl v14.2d, v31.2d, #0x12 + eor x27, x27, x12, ror #5 + eor x30, x30, x24, ror #34 + sri v14.2d, v31.2d, #0x2e + eor x0, x0, x7, ror #27 + eor x26, x30, x21, ror #26 + xar v20.2d, v4.2d, v27.2d, #0x25 + eor x26, x26, x25, ror #15 + ror x30, x27, #0x3e + eor v31.16b, v24.16b, v27.16b + eor x30, x30, x26, ror #57 + ror x26, x26, #0x3a + shl v4.2d, v31.2d, #0xe + eor x16, x30, x16 + eor x28, x30, x28, ror #63 + sri v4.2d, v31.2d, #0x32 + str x28, [sp, #0xd0] + eor x29, x29, x17, ror #36 + xar v24.2d, v21.2d, v25.2d, #0x3e + eor x28, x1, x2, ror #61 + eor x19, x30, x19, ror #37 + eor v31.16b, v5.16b, v26.16b + eor x29, x29, x20, ror #2 + eor x28, x28, x4, ror #54 + shl v21.2d, v31.2d, #0x24 + eor x26, x26, x0, ror #55 + eor x28, x28, x3, ror #39 + sri v21.2d, v31.2d, #0x1c + eor x28, x28, x5, ror #25 + ror x0, x0, #0x38 + xar v27.2d, v6.2d, v25.2d, #0x14 + eor x0, x0, x29, ror #63 + eor x27, x28, x27, ror #61 + bic v31.16b, v7.16b, v11.16b + eor x13, x0, x13, ror #46 + eor x28, x29, x28, ror #63 + eor v5.16b, v31.16b, v10.16b + eor x29, x30, x20, ror #2 + eor x20, x26, x3, ror #39 + bcax v6.16b, v11.16b, v8.16b, v7.16b + eor x11, x0, x11, ror #50 + eor x25, x28, x25, ror #9 + bic v31.16b, v9.16b, v8.16b + eor x3, x28, x21, ror #20 + eor v7.16b, v31.16b, v7.16b + eor x21, x26, x1 + eor x9, x27, x9, ror #49 + bcax v8.16b, v8.16b, v10.16b, v9.16b + eor x24, x28, x24, ror #28 + eor x1, x30, x17, ror #36 + bic v31.16b, v11.16b, v10.16b + eor x14, x0, x14, ror #8 + eor x22, x28, x22, ror #44 + eor v9.16b, v31.16b, v9.16b + eor x8, x27, x8, ror #56 + eor x17, x27, x7, ror #19 + bcax v10.16b, v15.16b, v12.16b, v16.16b + eor x15, x0, x15, ror #62 + bic x7, x20, x22, ror #47 + bic v31.16b, v13.16b, v12.16b + eor x4, x26, x4, ror #54 + eor x0, x0, x12, ror #3 + eor v11.16b, v31.16b, v16.16b + eor x28, x28, x23, ror #58 + eor x23, x26, x2, ror #61 + bcax v12.16b, v12.16b, v14.16b, v13.16b + eor x26, x26, x5, ror #25 + eor x2, x7, x16, ror #39 + bic v31.16b, v15.16b, v14.16b + bic x7, x9, x20, ror #42 + bic x30, x15, x9, ror #16 + eor v13.16b, v31.16b, v13.16b + eor x7, x7, x22, ror #25 + eor x12, x30, x20, ror #58 + bic v31.16b, v16.16b, v15.16b + bic x20, x22, x16, ror #56 + eor x30, x27, x6, ror #43 + eor v14.16b, v31.16b, v14.16b + eor x22, x20, x15, ror #23 + bic x6, x19, x13, ror #42 + bcax v15.16b, v20.16b, v17.16b, v21.16b + eor x6, x6, x17, ror #41 + bic x5, x13, x17, ror #63 + bic v31.16b, v18.16b, v17.16b + eor x5, x21, x5, ror #21 + bic x17, x17, x21, ror #44 + eor v16.16b, v31.16b, v21.16b + eor x27, x27, x10, ror #23 + bic x21, x21, x25, ror #50 + bcax v17.16b, v17.16b, v19.16b, v18.16b + bic x20, x27, x4, ror #25 + bic x10, x16, x15, ror #31 + bic v31.16b, v20.16b, v19.16b + eor x16, x21, x19, ror #43 + eor x21, x17, x25, ror #30 + eor v18.16b, v31.16b, v18.16b + bic x19, x25, x19, ror #57 + ldr x25, [sp, #0x18] + bcax v19.16b, v19.16b, v21.16b, v20.16b + eor x17, x10, x9, ror #47 + bic v31.16b, v22.16b, v1.16b + ldr x9, [sp, #0x8] + eor x15, x20, x28, ror #27 + eor v20.16b, v31.16b, v0.16b + bic x20, x4, x28, ror #2 + eor x10, x20, x1, ror #50 + bcax v21.16b, v1.16b, v23.16b, v22.16b + bic x20, x11, x27, ror #60 + eor x20, x20, x4, ror #21 + bic v31.16b, v24.16b, v23.16b + bic x4, x28, x1, ror #48 + bic x1, x1, x11, ror #57 + eor v22.16b, v31.16b, v22.16b + ldr x28, [x9, x25, lsl #3] + ldr x9, [sp, #0xd0] + bcax v23.16b, v23.16b, v0.16b, v24.16b + add x25, x25, #0x1 + str x25, [sp, #0x18] + bic v31.16b, v1.16b, v0.16b + cmp x25, #0x17 + eor x25, x1, x27, ror #53 + eor v24.16b, v31.16b, v24.16b + bic x27, x30, x26, ror #47 + eor x1, x5, x28 + bcax v0.16b, v30.16b, v2.16b, v27.16b + eor x5, x4, x11, ror #41 + eor x11, x19, x13, ror #35 + bic v31.16b, v3.16b, v2.16b + bic x13, x26, x24, ror #10 + eor x28, x27, x24, ror #57 + eor v1.16b, v31.16b, v27.16b + bic x27, x24, x9, ror #47 + bic x19, x23, x3, ror #9 + bcax v2.16b, v2.16b, v4.16b, v3.16b + bic x4, x29, x14, ror #41 + eor x24, x19, x29, ror #44 + bic v31.16b, v30.16b, v4.16b + bic x29, x3, x29, ror #35 + eor x13, x13, x9, ror #57 + eor v3.16b, v31.16b, v3.16b + eor x19, x29, x14, ror #12 + bic x29, x9, x0, ror #19 + bcax v4.16b, v4.16b, v27.16b, v30.16b + bic x14, x14, x8, ror #5 + eor x9, x14, x23, ror #43 + eor x14, x4, x8, ror #46 + bic x23, x8, x23, ror #38 + eor x8, x27, x0, ror #2 + eor x4, x23, x3, ror #47 + bic x3, x0, x30, ror #5 + eor x23, x3, x26, ror #52 + eor x3, x29, x30, ror #24 + ldr x30, [sp, #0x10] + ld1r { v28.2d }, [x30], #8 + str x30, [sp, #0x10] + eor v0.16b, v0.16b, v28.16b + +Lmlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_loop_end: + b.le Lmlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_loop + ror x2, x2, #0x3d + ror x3, x3, #0x27 + ror x4, x4, #0x36 + ror x5, x5, #0x19 + ror x6, x6, #0x2b + ror x7, x7, #0x13 + ror x8, x8, #0x38 + ror x9, x9, #0x31 + ror x10, x10, #0x17 + ror x11, x11, #0x32 + ror x12, x12, #0x3 + ror x13, x13, #0x2e + ror x14, x14, #0x8 + ror x15, x15, #0x3e + ror x17, x17, #0x24 + ror x28, x28, #0x3f + ror x19, x19, #0x25 + ror x20, x20, #0x2 + ror x21, x21, #0x14 + ror x22, x22, #0x2c + ror x23, x23, #0x3a + ror x24, x24, #0x1c + ror x25, x25, #0x9 + ldr x30, [sp, #0x20] + cmp x30, #0x1 + b.eq Lmlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_done + mov x30, #0x1 // =1 + str x30, [sp, #0x20] + ldr x0, [sp] + add x0, x0, #0x190 + stp x1, x6, [x0] + stp x11, x16, [x0, #0x10] + stp x21, x2, [x0, #0x20] + stp x7, x12, [x0, #0x30] + stp x17, x22, [x0, #0x40] + stp x3, x8, [x0, #0x50] + stp x13, x28, [x0, #0x60] + stp x23, x4, [x0, #0x70] + stp x9, x14, [x0, #0x80] + stp x19, x24, [x0, #0x90] + stp x5, x10, [x0, #0xa0] + stp x15, x20, [x0, #0xb0] + str x25, [x0, #0xc0] + sub x0, x0, #0x190 + add x0, x0, #0x258 + ldp x1, x6, [x0] + ldp x11, x16, [x0, #0x10] + ldp x21, x2, [x0, #0x20] + ldp x7, x12, [x0, #0x30] + ldp x17, x22, [x0, #0x40] + ldp x3, x8, [x0, #0x50] + ldp x13, x28, [x0, #0x60] + ldp x23, x4, [x0, #0x70] + ldp x9, x14, [x0, #0x80] + ldp x19, x24, [x0, #0x90] + ldp x5, x10, [x0, #0xa0] + ldp x15, x20, [x0, #0xb0] + ldr x25, [x0, #0xc0] + sub x0, x0, #0x258 + b Lmlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_initial + +Lmlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_done: + ldr x0, [sp] + add x0, x0, #0x258 + stp x1, x6, [x0] + stp x11, x16, [x0, #0x10] + stp x21, x2, [x0, #0x20] + stp x7, x12, [x0, #0x30] + stp x17, x22, [x0, #0x40] + stp x3, x8, [x0, #0x50] + stp x13, x28, [x0, #0x60] + stp x23, x4, [x0, #0x70] + stp x9, x14, [x0, #0x80] + stp x19, x24, [x0, #0x90] + stp x5, x10, [x0, #0xa0] + stp x15, x20, [x0, #0xb0] + str x25, [x0, #0xc0] + sub x0, x0, #0x258 + add x4, x0, #0xc8 + trn1 v25.2d, v0.2d, v1.2d + trn1 v26.2d, v2.2d, v3.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v0.2d, v1.2d + trn2 v28.2d, v2.2d, v3.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v4.2d, v5.2d + trn1 v26.2d, v6.2d, v7.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v4.2d, v5.2d + trn2 v28.2d, v6.2d, v7.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v8.2d, v9.2d + trn1 v26.2d, v10.2d, v11.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v8.2d, v9.2d + trn2 v28.2d, v10.2d, v11.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v12.2d, v13.2d + trn1 v26.2d, v14.2d, v15.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v12.2d, v13.2d + trn2 v28.2d, v14.2d, v15.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v16.2d, v17.2d + trn1 v26.2d, v18.2d, v19.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v16.2d, v17.2d + trn2 v28.2d, v18.2d, v19.2d + st1 { v27.2d, v28.2d }, [x4], #32 + trn1 v25.2d, v20.2d, v21.2d + trn1 v26.2d, v22.2d, v23.2d + stp q25, q26, [x0], #0x20 + trn2 v27.2d, v20.2d, v21.2d + trn2 v28.2d, v22.2d, v23.2d + st1 { v27.2d, v28.2d }, [x4], #32 + str d24, [x0] + trn2 v25.2d, v24.2d, v24.2d + str d25, [x4] + ldp d8, d9, [sp, #0x90] + .cfi_restore d8 + .cfi_restore d9 + ldp d10, d11, [sp, #0xa0] + .cfi_restore d10 + .cfi_restore d11 + ldp d12, d13, [sp, #0xb0] + .cfi_restore d12 + .cfi_restore d13 + ldp d14, d15, [sp, #0xc0] + .cfi_restore d14 + .cfi_restore d15 + ldp x19, x20, [sp, #0x30] + .cfi_restore x19 + .cfi_restore x20 + ldp x21, x22, [sp, #0x40] + .cfi_restore x21 + .cfi_restore x22 + ldp x23, x24, [sp, #0x50] + .cfi_restore x23 + .cfi_restore x24 + ldp x25, x26, [sp, #0x60] + .cfi_restore x25 + .cfi_restore x26 + ldp x27, x28, [sp, #0x70] + .cfi_restore x27 + .cfi_restore x28 + ldp x29, x30, [sp, #0x80] + .cfi_restore x29 + .cfi_restore x30 + add sp, sp, #0xe0 + .cfi_adjust_cfa_offset -0xe0 + ret + .cfi_endproc + +MLK_ASM_FN_SIZE(keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm) + +#endif /* __ARM_FEATURE_SHA3 */ + +#endif /* MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID && \ + !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/fips202/native/aarch64/src/keccakf1600_round_constants.c b/cbits/mlkem/src/fips202/native/aarch64/src/keccakf1600_round_constants.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/aarch64/src/keccakf1600_round_constants.c @@ -0,0 +1,47 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mlkem-native repository. + * Do not modify it directly. + */ + +#include "../../../../common.h" + +#if (defined(MLK_FIPS202_AARCH64_NEED_X1_SCALAR) || \ + defined(MLK_FIPS202_AARCH64_NEED_X1_V84A) || \ + defined(MLK_FIPS202_AARCH64_NEED_X2_V84A) || \ + defined(MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID) || \ + defined(MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID)) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +#include "fips202_native_aarch64.h" + +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint64_t + mlk_keccakf1600_round_constants[24] = { + 0x0000000000000001, 0x0000000000008082, 0x800000000000808a, + 0x8000000080008000, 0x000000000000808b, 0x0000000080000001, + 0x8000000080008081, 0x8000000000008009, 0x000000000000008a, + 0x0000000000000088, 0x0000000080008009, 0x000000008000000a, + 0x000000008000808b, 0x800000000000008b, 0x8000000000008089, + 0x8000000000008003, 0x8000000000008002, 0x8000000000000080, + 0x000000000000800a, 0x800000008000000a, 0x8000000080008081, + 0x8000000000008080, 0x0000000080000001, 0x8000000080008008, +}; + +#else /* (MLK_FIPS202_AARCH64_NEED_X1_SCALAR || \ + MLK_FIPS202_AARCH64_NEED_X1_V84A || MLK_FIPS202_AARCH64_NEED_X2_V84A \ + || MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID || \ + MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID) && \ + !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +MLK_EMPTY_CU(fips202_aarch64_round_constants) + +#endif /* !((MLK_FIPS202_AARCH64_NEED_X1_SCALAR || \ + MLK_FIPS202_AARCH64_NEED_X1_V84A || MLK_FIPS202_AARCH64_NEED_X2_V84A \ + || MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID || \ + MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID) && \ + !MLK_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mlkem/src/fips202/native/aarch64/x1_scalar.h b/cbits/mlkem/src/fips202/native/aarch64/x1_scalar.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/aarch64/x1_scalar.h @@ -0,0 +1,26 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLK_FIPS202_NATIVE_AARCH64_X1_SCALAR_H +#define MLK_FIPS202_NATIVE_AARCH64_X1_SCALAR_H + +/* Part of backend API */ +#define MLK_USE_NATIVE_FIPS202_X1 +/* Guard for assembly file */ +#define MLK_FIPS202_AARCH64_NEED_X1_SCALAR + +#if !defined(__ASSEMBLER__) +#include "../api.h" +#include "src/fips202_native_aarch64.h" +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_keccak_f1600_x1_native(uint64_t *state) +{ + mlk_keccak_f1600_x1_scalar_aarch64_asm(state, + mlk_keccakf1600_round_constants); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLK_FIPS202_NATIVE_AARCH64_X1_SCALAR_H */ diff --git a/cbits/mlkem/src/fips202/native/aarch64/x1_v84a.h b/cbits/mlkem/src/fips202/native/aarch64/x1_v84a.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/aarch64/x1_v84a.h @@ -0,0 +1,35 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLK_FIPS202_NATIVE_AARCH64_X1_V84A_H +#define MLK_FIPS202_NATIVE_AARCH64_X1_V84A_H + +#if !defined(__ARM_FEATURE_SHA3) +#error This backend can only be used if SHA3 extensions are available. +#endif + +/* Part of backend API */ +#define MLK_USE_NATIVE_FIPS202_X1 +/* Guard for assembly file */ +#define MLK_FIPS202_AARCH64_NEED_X1_V84A + +#if !defined(__ASSEMBLER__) +#include "../api.h" +#include "src/fips202_native_aarch64.h" +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_keccak_f1600_x1_native(uint64_t *state) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON) || + !mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_SHA3)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_keccak_f1600_x1_v84a_aarch64_asm(state, mlk_keccakf1600_round_constants); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLK_FIPS202_NATIVE_AARCH64_X1_V84A_H */ diff --git a/cbits/mlkem/src/fips202/native/aarch64/x2_v84a.h b/cbits/mlkem/src/fips202/native/aarch64/x2_v84a.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/aarch64/x2_v84a.h @@ -0,0 +1,38 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLK_FIPS202_NATIVE_AARCH64_X2_V84A_H +#define MLK_FIPS202_NATIVE_AARCH64_X2_V84A_H + +#if !defined(__ARM_FEATURE_SHA3) +#error This backend can only be used if SHA3 extensions are available. +#endif + +/* Part of backend API */ +#define MLK_USE_NATIVE_FIPS202_X4 +/* Guard for assembly file */ +#define MLK_FIPS202_AARCH64_NEED_X2_V84A + +#if !defined(__ASSEMBLER__) +#include "../api.h" +#include "src/fips202_native_aarch64.h" +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_keccak_f1600_x4_native(uint64_t *state) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON) || + !mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_SHA3)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_keccak_f1600_x2_v84a_aarch64_asm(state + 0 * 25, + mlk_keccakf1600_round_constants); + mlk_keccak_f1600_x2_v84a_aarch64_asm(state + 2 * 25, + mlk_keccakf1600_round_constants); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLK_FIPS202_NATIVE_AARCH64_X2_V84A_H */ diff --git a/cbits/mlkem/src/fips202/native/aarch64/x4_v8a_scalar.h b/cbits/mlkem/src/fips202/native/aarch64/x4_v8a_scalar.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/aarch64/x4_v8a_scalar.h @@ -0,0 +1,31 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLK_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H +#define MLK_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H + +/* Part of backend API */ +#define MLK_USE_NATIVE_FIPS202_X4 +/* Guard for assembly file */ +#define MLK_FIPS202_AARCH64_NEED_X4_V8A_SCALAR_HYBRID + +#if !defined(__ASSEMBLER__) +#include "../api.h" +#include "src/fips202_native_aarch64.h" +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_keccak_f1600_x4_native(uint64_t *state) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_keccak_f1600_x4_v8a_scalar_hybrid_aarch64_asm( + state, mlk_keccakf1600_round_constants); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLK_FIPS202_NATIVE_AARCH64_X4_V8A_SCALAR_H */ diff --git a/cbits/mlkem/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h b/cbits/mlkem/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/aarch64/x4_v8a_v84a_scalar.h @@ -0,0 +1,36 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLK_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H +#define MLK_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H + +#if !defined(__ARM_FEATURE_SHA3) +#error This backend can only be used if SHA3 extensions are available. +#endif + +/* Part of backend API */ +#define MLK_USE_NATIVE_FIPS202_X4 +/* Guard for assembly file */ +#define MLK_FIPS202_AARCH64_NEED_X4_V8A_V84A_SCALAR_HYBRID + +#if !defined(__ASSEMBLER__) +#include "../api.h" +#include "src/fips202_native_aarch64.h" +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_keccak_f1600_x4_native(uint64_t *state) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON) || + !mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_SHA3)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_keccak_f1600_x4_v8a_v84a_scalar_hybrid_aarch64_asm( + state, mlk_keccakf1600_round_constants); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLK_FIPS202_NATIVE_AARCH64_X4_V8A_V84A_SCALAR_H */ diff --git a/cbits/mlkem/src/fips202/native/api.h b/cbits/mlkem/src/fips202/native/api.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/api.h @@ -0,0 +1,117 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLK_FIPS202_NATIVE_API_H +#define MLK_FIPS202_NATIVE_API_H +/* + * FIPS-202 native interface + * + * This header is primarily for documentation purposes. + * It should not be included by backend implementations. + */ + +#include "../../cbmc.h" + +/* Backends must return MLK_NATIVE_FUNC_SUCCESS upon success. */ +#define MLK_NATIVE_FUNC_SUCCESS (0) +/* Backends may return MLK_NATIVE_FUNC_FALLBACK to signal to the frontend that + * the target/parameters are unsupported; typically, this would be because of + * dependencies on CPU features not detected on the host CPU. In this case, + * the frontend falls back to the default C implementation. */ +#define MLK_NATIVE_FUNC_FALLBACK (-1) + +/* + * This is the C<->native interface allowing for the drop-in + * of custom Keccak-F1600 implementations. + * + * A _backend_ is a specific implementation of parts of this interface. + * + * You can replace 1-fold or 4-fold batched Keccak-F1600. + * To enable, set MLK_USE_NATIVE_FIPS202_X1 or MLK_USE_NATIVE_FIPS202_X4 + * in your backend, and define the inline wrappers mlk_keccak_f1600_x1_native() + * and/or mlk_keccak_f1600_x4_native(), respectively, to forward to your + * implementation. + */ + +#if defined(MLK_USE_NATIVE_FIPS202_X1) +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_keccak_f1600_x1_native(uint64_t *state) +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 1)) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 1)) + ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_unchanged_u64(state, 25 * 1))); +#endif /* MLK_USE_NATIVE_FIPS202_X1 */ +#if defined(MLK_USE_NATIVE_FIPS202_X4) +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_keccak_f1600_x4_native(uint64_t *state) +__contract__( + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4)) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4)) + ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_unchanged_u64(state, 25 * 4))); +#endif /* MLK_USE_NATIVE_FIPS202_X4 */ + +/* + * Native x4 XOR bytes and extract bytes interface. + * + * These functions allow backends to provide optimized implementations for + * XORing input data into the state and extracting output data from the state. + * This is particularly useful for backends that use a different internal state + * representation (e.g., bit-interleaved), as conversion can happen during + * XOR/extract rather than before/after each permutation. + * + * NOTE: We assume that the custom representation of the zero state is the + * all-zero state. + * + * MLK_USE_NATIVE_FIPS202_X4_XOR_BYTES: Backend provides native XOR bytes + * MLK_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES: Backend provides native extract + * bytes + */ + +#if defined(MLK_USE_NATIVE_FIPS202_X4_XOR_BYTES) +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_keccakf1600_xor_bytes_x4_native( + uint64_t *state, const unsigned char *data0, const unsigned char *data1, + const unsigned char *data2, const unsigned char *data3, unsigned offset, + unsigned length) +__contract__( + requires(0 <= offset && offset <= 25 * sizeof(uint64_t) && + 0 <= length && length <= 25 * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4)) + requires(memory_no_alias(data0, length)) + requires((data0 == data1 && + data0 == data2 && + data0 == data3) || + (memory_no_alias(data1, length) && + memory_no_alias(data2, length) && + memory_no_alias(data3, length))) + assigns(memory_slice(state, sizeof(uint64_t) * 25 * 4)) + ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_unchanged_u64(state, 25 * 4))); +#endif /* MLK_USE_NATIVE_FIPS202_X4_XOR_BYTES */ + +#if defined(MLK_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES) +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_keccakf1600_extract_bytes_x4_native( + uint64_t *state, unsigned char *data0, unsigned char *data1, + unsigned char *data2, unsigned char *data3, unsigned offset, + unsigned length) +__contract__( + requires(0 <= offset && offset <= 25 * sizeof(uint64_t) && + 0 <= length && length <= 25 * sizeof(uint64_t) - offset) + requires(memory_no_alias(state, sizeof(uint64_t) * 25 * 4)) + requires(memory_no_alias(data0, length)) + requires(memory_no_alias(data1, length)) + requires(memory_no_alias(data2, length)) + requires(memory_no_alias(data3, length)) + assigns(memory_slice(data0, length)) + assigns(memory_slice(data1, length)) + assigns(memory_slice(data2, length)) + assigns(memory_slice(data3, length)) + ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS)); +#endif /* MLK_USE_NATIVE_FIPS202_X4_EXTRACT_BYTES */ + +#endif /* !MLK_FIPS202_NATIVE_API_H */ diff --git a/cbits/mlkem/src/fips202/native/auto.h b/cbits/mlkem/src/fips202/native/auto.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/auto.h @@ -0,0 +1,29 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLK_FIPS202_NATIVE_AUTO_H +#define MLK_FIPS202_NATIVE_AUTO_H + +/* + * Default FIPS202 backend + */ +#include "../../sys.h" + +#if defined(MLK_SYS_AARCH64) +#include "aarch64/auto.h" +#endif + +/* The x86_64 backend requires toolchain support for the SysV ABI */ +#if defined(MLK_SYS_X86_64_AVX2) && defined(MLK_SYSV_ABI_SUPPORTED) +#include "x86_64/keccak_f1600_x4_avx2.h" +#endif + +/* We do not yet include the FIPS202 backend for Armv8.1-M+MVE by default + * as it is still experimental and undergoing review. */ +/* #if defined(MLK_SYS_ARMV81M_MVE) */ +/* #include "armv81m/mve.h" */ +/* #endif */ + +#endif /* !MLK_FIPS202_NATIVE_AUTO_H */ diff --git a/cbits/mlkem/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h b/cbits/mlkem/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/x86_64/keccak_f1600_x4_avx2.h @@ -0,0 +1,33 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLK_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H +#define MLK_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H + +#include "../../../common.h" + +#define MLK_FIPS202_X86_64_NEED_X4_AVX2 + +/* Part of backend API */ +#define MLK_USE_NATIVE_FIPS202_X4 + +#if !defined(__ASSEMBLER__) +#include "../api.h" +#include "src/fips202_native_x86_64.h" +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_keccak_f1600_x4_native(uint64_t *state) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_keccak_f1600_x4_avx2_asm(state, mlk_keccakf1600_round_constants, + mlk_keccak_rho8, mlk_keccak_rho56); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLK_FIPS202_NATIVE_X86_64_KECCAK_F1600_X4_AVX2_H */ diff --git a/cbits/mlkem/src/fips202/native/x86_64/src/fips202_native_x86_64.h b/cbits/mlkem/src/fips202/native/x86_64/src/fips202_native_x86_64.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/x86_64/src/fips202_native_x86_64.h @@ -0,0 +1,44 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLK_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H +#define MLK_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H + +#include "../../../../cbmc.h" +#include "../../../../common.h" + +/* TODO: Reconsider whether this check is needed -- x86_64 is always + * little-endian, so the backend selection already implies this. */ +#ifndef MLK_SYS_LITTLE_ENDIAN +#error Expecting a little-endian platform +#endif + +#define mlk_keccakf1600_round_constants \ + MLK_NAMESPACE(keccakf1600_round_constants) +MLK_INTERNAL_DATA_DECLARATION const uint64_t + mlk_keccakf1600_round_constants[24]; + +#define mlk_keccak_rho8 MLK_NAMESPACE(keccak_rho8) +MLK_INTERNAL_DATA_DECLARATION const uint64_t mlk_keccak_rho8[4]; + +#define mlk_keccak_rho56 MLK_NAMESPACE(keccak_rho56) +MLK_INTERNAL_DATA_DECLARATION const uint64_t mlk_keccak_rho56[4]; + +#define mlk_keccak_f1600_x4_avx2_asm MLK_NAMESPACE(keccak_f1600_x4_avx2_asm) +MLK_SYSV_ABI +void mlk_keccak_f1600_x4_avx2_asm(uint64_t states[100], const uint64_t rc[24], + const uint64_t rho8[4], + const uint64_t rho56[4]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/keccak_f1600_x4_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(states, sizeof(uint64_t) * 25 * 4)) + requires(rc == mlk_keccakf1600_round_constants) + requires(rho8 == mlk_keccak_rho8) + requires(rho56 == mlk_keccak_rho56) + assigns(memory_slice(states, sizeof(uint64_t) * 25 * 4)) +); + +#endif /* !MLK_FIPS202_NATIVE_X86_64_SRC_FIPS202_NATIVE_X86_64_H */ diff --git a/cbits/mlkem/src/fips202/native/x86_64/src/keccak_f1600_x4_avx2_asm.S b/cbits/mlkem/src/fips202/native/x86_64/src/keccak_f1600_x4_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/x86_64/src/keccak_f1600_x4_avx2_asm.S @@ -0,0 +1,487 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#include "../../../../common.h" + +#if defined(MLK_FIPS202_X86_64_NEED_X4_AVX2) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) +/*yaml + Name: keccak_f1600_x4_avx2_asm + Description: x86_64 AVX2 Keccak-f[1600] permutation for four sequential states + Signature: void mlk_keccak_f1600_x4_avx2_asm(uint64_t states[100], const uint64_t rc[24], const uint64_t rho8[4], const uint64_t rho56[4]) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 800 + permissions: read/write + c_parameter: uint64_t states[100] + description: Four sequential Keccak states (4 x 25 x uint64_t) + rsi: + type: buffer + size_bytes: 192 + permissions: read-only + c_parameter: const uint64_t rc[24] + description: Round constants (24 x uint64_t) + rdx: + type: buffer + size_bytes: 32 + permissions: read-only + c_parameter: const uint64_t rho8[4] + description: Rotation constant rho8 (4 x uint64_t) + rcx: + type: buffer + size_bytes: 32 + permissions: read-only + c_parameter: const uint64_t rho56[4] + description: Rotation constant rho56 (4 x uint64_t) +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/fips202/x86_64/src/keccak_f1600_x4_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(keccak_f1600_x4_avx2_asm) +MLK_ASM_FN_SYMBOL(keccak_f1600_x4_avx2_asm) + + .cfi_startproc + movq %rsp, %r11 + .cfi_def_cfa_register %r11 + andq $-0x20, %rsp + subq $0x300, %rsp # imm = 0x300 + vmovdqu (%rdi), %ymm0 + vmovdqu 0xc8(%rdi), %ymm3 + vmovdqu 0x190(%rdi), %ymm1 + vmovdqu 0x258(%rdi), %ymm4 + vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3] + vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2] + vperm2i128 $0x20, %ymm3, %ymm2, %ymm7 # ymm7 = ymm2[0,1],ymm3[0,1] + vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3] + vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3] + vmovdqu 0x278(%rdi), %ymm4 + vmovdqu %ymm3, 0x40(%rsp) + vperm2i128 $0x31, %ymm1, %ymm0, %ymm3 # ymm3 = ymm0[2,3],ymm1[2,3] + vmovdqu %ymm7, (%rsp) + vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1] + vmovdqu 0x20(%rdi), %ymm0 + vmovdqu 0x1b0(%rdi), %ymm1 + vmovdqu %ymm3, 0x60(%rsp) + vmovdqu 0xe8(%rdi), %ymm3 + vmovdqu %ymm7, 0x20(%rsp) + vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3] + vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2] + vperm2i128 $0x20, %ymm3, %ymm2, %ymm7 # ymm7 = ymm2[0,1],ymm3[0,1] + vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3] + vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3] + vmovdqu 0x298(%rdi), %ymm4 + vperm2i128 $0x31, %ymm1, %ymm0, %ymm14 # ymm14 = ymm0[2,3],ymm1[2,3] + vmovdqu %ymm7, 0x80(%rsp) + vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1] + vmovdqu 0x40(%rdi), %ymm0 + vmovdqu 0x1d0(%rdi), %ymm1 + vmovdqu %ymm3, 0xc0(%rsp) + vmovdqu 0x108(%rdi), %ymm3 + vmovdqu %ymm14, %ymm10 + vmovdqu %ymm7, 0xa0(%rsp) + vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3] + vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3] + vperm2i128 $0x20, %ymm3, %ymm2, %ymm11 # ymm11 = ymm2[0,1],ymm3[0,1] + vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3] + vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1] + vmovdqu %ymm3, 0x100(%rsp) + vperm2i128 $0x31, %ymm1, %ymm0, %ymm8 # ymm8 = ymm0[2,3],ymm1[2,3] + vmovdqu 0x128(%rdi), %ymm3 + vmovdqu 0x60(%rdi), %ymm0 + vmovdqu 0x1f0(%rdi), %ymm1 + vmovdqu %ymm7, 0xe0(%rsp) + vmovdqu %ymm11, %ymm14 + vmovdqu 0x2b8(%rdi), %ymm4 + vmovdqu 0x2f8(%rdi), %ymm5 + vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3] + vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3] + vmovdqu 0x2d8(%rdi), %ymm4 + vperm2i128 $0x20, %ymm3, %ymm2, %ymm15 # ymm15 = ymm2[0,1],ymm3[0,1] + vperm2i128 $0x31, %ymm3, %ymm2, %ymm3 # ymm3 = ymm2[2,3],ymm3[2,3] + vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1] + vperm2i128 $0x31, %ymm1, %ymm0, %ymm9 # ymm9 = ymm0[2,3],ymm1[2,3] + vmovdqu %ymm3, 0x140(%rsp) + vmovdqu 0x80(%rdi), %ymm0 + vmovdqu 0x148(%rdi), %ymm3 + vmovdqu 0x210(%rdi), %ymm1 + vmovdqu %ymm7, 0x120(%rsp) + vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3] + vpunpcklqdq %ymm4, %ymm1, %ymm3 # ymm3 = ymm1[0],ymm4[0],ymm1[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm4[1],ymm1[3],ymm4[3] + vperm2i128 $0x20, %ymm3, %ymm2, %ymm7 # ymm7 = ymm2[0,1],ymm3[0,1] + vperm2i128 $0x31, %ymm3, %ymm2, %ymm13 # ymm13 = ymm2[2,3],ymm3[2,3] + vperm2i128 $0x31, %ymm1, %ymm0, %ymm3 # ymm3 = ymm0[2,3],ymm1[2,3] + vmovdqu %ymm7, 0x160(%rsp) + vperm2i128 $0x20, %ymm1, %ymm0, %ymm7 # ymm7 = ymm0[0,1],ymm1[0,1] + vmovdqu 0xa0(%rdi), %ymm0 + vmovdqu 0x230(%rdi), %ymm1 + vmovdqu %ymm3, 0x1a0(%rsp) + vmovdqu 0x168(%rdi), %ymm3 + vpunpcklqdq %ymm5, %ymm1, %ymm4 # ymm4 = ymm1[0],ymm5[0],ymm1[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm1, %ymm1 # ymm1 = ymm1[1],ymm5[1],ymm1[3],ymm5[3] + vmovdqu %ymm7, 0x180(%rsp) + vpunpcklqdq %ymm3, %ymm0, %ymm2 # ymm2 = ymm0[0],ymm3[0],ymm0[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm3[1],ymm0[3],ymm3[3] + vperm2i128 $0x20, %ymm4, %ymm2, %ymm12 # ymm12 = ymm2[0,1],ymm4[0,1] + vperm2i128 $0x20, %ymm1, %ymm0, %ymm3 # ymm3 = ymm0[0,1],ymm1[0,1] + vperm2i128 $0x31, %ymm4, %ymm2, %ymm7 # ymm7 = ymm2[2,3],ymm4[2,3] + vperm2i128 $0x31, %ymm1, %ymm0, %ymm4 # ymm4 = ymm0[2,3],ymm1[2,3] + vmovq 0x250(%rdi), %xmm0 + vmovq 0xc0(%rdi), %xmm1 + vmovdqu %ymm12, 0x1c0(%rsp) + vmovdqu %ymm4, 0x1e0(%rsp) + vpinsrq $0x1, 0x318(%rdi), %xmm0, %xmm0 + vpinsrq $0x1, 0x188(%rdi), %xmm1, %xmm1 + vinserti128 $0x1, %xmm0, %ymm1, %ymm2 + movq $0x0, %r10 + +LLmlk_keccak_f1600_x4_avx2: + vmovdqu 0xa0(%rsp), %ymm4 + vpxor 0x1c0(%rsp), %ymm9, %ymm0 + vmovdqu %ymm9, 0x200(%rsp) + vmovdqu %ymm10, %ymm9 + vmovdqu 0xc0(%rsp), %ymm11 + vmovdqu 0x160(%rsp), %ymm12 + vmovdqu %ymm3, 0x240(%rsp) + vpxor 0x100(%rsp), %ymm4, %ymm1 + vmovdqu 0x40(%rsp), %ymm10 + vmovdqu %ymm4, 0x220(%rsp) + vpxor %ymm3, %ymm12, %ymm12 + vmovdqu 0x20(%rsp), %ymm6 + vmovdqu 0x140(%rsp), %ymm4 + vmovdqu %ymm14, 0x2a0(%rsp) + vpxor %ymm1, %ymm0, %ymm0 + vpxor %ymm8, %ymm11, %ymm1 + vpxor 0x180(%rsp), %ymm7, %ymm11 + vmovdqu %ymm10, 0x280(%rsp) + vpxor %ymm1, %ymm12, %ymm12 + vpxor %ymm15, %ymm9, %ymm1 + vmovdqu 0xe0(%rsp), %ymm3 + vmovdqu %ymm8, 0x260(%rsp) + vpxor %ymm1, %ymm11, %ymm11 + vpxor 0x120(%rsp), %ymm14, %ymm1 + vpxor %ymm6, %ymm12, %ymm12 + vmovdqu 0x60(%rsp), %ymm8 + vpxor %ymm10, %ymm11, %ymm11 + vpxor 0x1e0(%rsp), %ymm13, %ymm10 + vpxor %ymm4, %ymm3, %ymm3 + vmovdqu %ymm4, 0x2c0(%rsp) + vpsrlq $0x3f, %ymm12, %ymm4 + vpsrlq $0x3f, %ymm11, %ymm5 + vpxor (%rsp), %ymm0, %ymm0 + vpxor %ymm1, %ymm10, %ymm10 + vmovdqu 0x80(%rsp), %ymm1 + vpxor %ymm8, %ymm10, %ymm10 + vmovdqu %ymm1, %ymm14 + vpxor 0x1a0(%rsp), %ymm2, %ymm1 + vmovdqu %ymm14, 0x2e0(%rsp) + vpxor %ymm3, %ymm1, %ymm1 + vpsllq $0x1, %ymm12, %ymm3 + vpor %ymm4, %ymm3, %ymm3 + vpsllq $0x1, %ymm11, %ymm4 + vpxor %ymm14, %ymm1, %ymm1 + vpor %ymm5, %ymm4, %ymm4 + vpsrlq $0x3f, %ymm10, %ymm14 + vpxor %ymm1, %ymm3, %ymm3 + vpsllq $0x1, %ymm10, %ymm5 + vpxor %ymm0, %ymm4, %ymm4 + vpor %ymm14, %ymm5, %ymm5 + vpxor %ymm6, %ymm4, %ymm6 + vpxor %ymm12, %ymm5, %ymm5 + vpsrlq $0x3f, %ymm1, %ymm12 + vpsllq $0x1, %ymm1, %ymm1 + vpxor %ymm7, %ymm5, %ymm7 + vpxor %ymm9, %ymm5, %ymm9 + vpor %ymm12, %ymm1, %ymm1 + vpxor (%rsp), %ymm3, %ymm12 + vpxor %ymm11, %ymm1, %ymm1 + vpsrlq $0x3f, %ymm0, %ymm11 + vpsllq $0x1, %ymm0, %ymm0 + vpxor %ymm13, %ymm1, %ymm13 + vpxor %ymm8, %ymm1, %ymm8 + vpor %ymm11, %ymm0, %ymm0 + vpxor %ymm10, %ymm0, %ymm0 + vpxor 0xc0(%rsp), %ymm4, %ymm10 + vpxor %ymm2, %ymm0, %ymm2 + vpsrlq $0x14, %ymm10, %ymm11 + vpsllq $0x2c, %ymm10, %ymm10 + vpor %ymm11, %ymm10, %ymm10 + vpxor %ymm15, %ymm5, %ymm11 + vpbroadcastq (%rsi), %ymm15 + vpsrlq $0x15, %ymm11, %ymm14 + vpsllq $0x2b, %ymm11, %ymm11 + vpor %ymm14, %ymm11, %ymm11 + vpandn %ymm11, %ymm10, %ymm14 + vpxor %ymm15, %ymm14, %ymm14 + vpxor %ymm12, %ymm14, %ymm15 + vpsrlq $0x2b, %ymm13, %ymm14 + vpsllq $0x15, %ymm13, %ymm13 + vmovdqu %ymm15, (%rsp) + vpor %ymm14, %ymm13, %ymm13 + vpandn %ymm13, %ymm11, %ymm14 + vpxor %ymm10, %ymm14, %ymm15 + vpsrlq $0x32, %ymm2, %ymm14 + vpsllq $0xe, %ymm2, %ymm2 + vmovdqu %ymm15, 0x20(%rsp) + vpor %ymm14, %ymm2, %ymm2 + vpandn %ymm2, %ymm13, %ymm14 + vpxor %ymm11, %ymm14, %ymm11 + vmovdqu %ymm11, 0x40(%rsp) + vpandn %ymm12, %ymm2, %ymm11 + vpandn %ymm10, %ymm12, %ymm12 + vpxor %ymm13, %ymm11, %ymm11 + vmovdqu %ymm11, 0x60(%rsp) + vpxor %ymm2, %ymm12, %ymm11 + vpsrlq $0x24, %ymm8, %ymm2 + vpsllq $0x1c, %ymm8, %ymm8 + vmovdqu %ymm11, 0x80(%rsp) + vpor %ymm2, %ymm8, %ymm8 + vpxor 0xe0(%rsp), %ymm0, %ymm2 + vpsrlq $0x2c, %ymm2, %ymm10 + vpsllq $0x14, %ymm2, %ymm2 + vpor %ymm10, %ymm2, %ymm2 + vpxor 0x100(%rsp), %ymm3, %ymm10 + vpsrlq $0x3d, %ymm10, %ymm11 + vpsllq $0x3, %ymm10, %ymm10 + vpor %ymm11, %ymm10, %ymm10 + vpandn %ymm10, %ymm2, %ymm11 + vpxor %ymm8, %ymm11, %ymm11 + vmovdqu %ymm11, 0xa0(%rsp) + vpxor 0x160(%rsp), %ymm4, %ymm11 + vpsrlq $0x13, %ymm11, %ymm12 + vpsllq $0x2d, %ymm11, %ymm11 + vpor %ymm12, %ymm11, %ymm11 + vpandn %ymm11, %ymm10, %ymm12 + vpxor %ymm2, %ymm12, %ymm12 + vmovdqu %ymm12, 0xc0(%rsp) + vpsrlq $0x3, %ymm7, %ymm12 + vpsllq $0x3d, %ymm7, %ymm7 + vpor %ymm12, %ymm7, %ymm7 + vpandn %ymm7, %ymm11, %ymm12 + vpxor %ymm10, %ymm12, %ymm10 + vpandn %ymm8, %ymm7, %ymm12 + vpandn %ymm2, %ymm8, %ymm8 + vpsrlq $0x3f, %ymm6, %ymm2 + vpsllq $0x1, %ymm6, %ymm6 + vpxor %ymm11, %ymm12, %ymm14 + vpor %ymm2, %ymm6, %ymm6 + vpsrlq $0x3a, %ymm9, %ymm2 + vpxor %ymm7, %ymm8, %ymm12 + vpsllq $0x6, %ymm9, %ymm9 + vmovdqu %ymm12, 0xe0(%rsp) + vpxor 0x1a0(%rsp), %ymm0, %ymm7 + vpor %ymm2, %ymm9, %ymm9 + vpxor 0x120(%rsp), %ymm1, %ymm2 + vpshufb (%rdx), %ymm7, %ymm7 + vpsrlq $0x27, %ymm2, %ymm11 + vpsllq $0x19, %ymm2, %ymm2 + vpor %ymm2, %ymm11, %ymm11 + vpandn %ymm11, %ymm9, %ymm2 + vpandn %ymm7, %ymm11, %ymm8 + vpxor %ymm6, %ymm2, %ymm12 + vpxor 0x1c0(%rsp), %ymm3, %ymm2 + vpxor %ymm9, %ymm8, %ymm8 + vmovdqu %ymm12, 0x100(%rsp) + vpsrlq $0x2e, %ymm2, %ymm12 + vpsllq $0x12, %ymm2, %ymm2 + vpor %ymm2, %ymm12, %ymm2 + vpandn %ymm2, %ymm7, %ymm12 + vpxor %ymm11, %ymm12, %ymm15 + vpandn %ymm6, %ymm2, %ymm11 + vpandn %ymm9, %ymm6, %ymm6 + vpxor %ymm7, %ymm11, %ymm12 + vmovdqu %ymm12, 0x120(%rsp) + vpxor %ymm2, %ymm6, %ymm12 + vpxor 0x2e0(%rsp), %ymm0, %ymm6 + vpxor 0x2c0(%rsp), %ymm0, %ymm0 + vmovdqu %ymm12, 0x140(%rsp) + vpsrlq $0x25, %ymm6, %ymm2 + vpsllq $0x1b, %ymm6, %ymm6 + vpor %ymm6, %ymm2, %ymm2 + vpxor 0x220(%rsp), %ymm3, %ymm6 + vpxor 0x200(%rsp), %ymm3, %ymm3 + vpsrlq $0x1c, %ymm6, %ymm7 + vpsllq $0x24, %ymm6, %ymm6 + vpor %ymm6, %ymm7, %ymm7 + vpxor 0x260(%rsp), %ymm4, %ymm6 + vpxor 0x240(%rsp), %ymm4, %ymm4 + vpsrlq $0x36, %ymm6, %ymm12 + vpsllq $0xa, %ymm6, %ymm6 + vpor %ymm6, %ymm12, %ymm12 + vpxor 0x180(%rsp), %ymm5, %ymm6 + vpxor 0x280(%rsp), %ymm5, %ymm5 + vpandn %ymm12, %ymm7, %ymm9 + vpsrlq $0x31, %ymm6, %ymm11 + vpsllq $0xf, %ymm6, %ymm6 + vpxor %ymm2, %ymm9, %ymm9 + vpor %ymm6, %ymm11, %ymm11 + vpandn %ymm11, %ymm12, %ymm6 + vpxor %ymm7, %ymm6, %ymm6 + vmovdqu %ymm6, 0x160(%rsp) + vpxor 0x1e0(%rsp), %ymm1, %ymm6 + vpxor 0x2a0(%rsp), %ymm1, %ymm1 + vpshufb (%rcx), %ymm6, %ymm6 + vpandn %ymm6, %ymm11, %ymm13 + vpxor %ymm12, %ymm13, %ymm13 + vmovdqu %ymm13, 0x180(%rsp) + vpandn %ymm2, %ymm6, %ymm13 + vpandn %ymm7, %ymm2, %ymm2 + vpxor %ymm6, %ymm2, %ymm2 + vpsrlq $0x3e, %ymm4, %ymm6 + vpxor %ymm11, %ymm13, %ymm13 + vmovdqu %ymm2, 0x1a0(%rsp) + vpsrlq $0x2, %ymm5, %ymm2 + vpsllq $0x3e, %ymm5, %ymm5 + vpor %ymm5, %ymm2, %ymm2 + vpsrlq $0x9, %ymm1, %ymm5 + vpsllq $0x37, %ymm1, %ymm1 + vpsllq $0x2, %ymm4, %ymm4 + vpor %ymm1, %ymm5, %ymm1 + vpsrlq $0x19, %ymm0, %ymm5 + vpor %ymm4, %ymm6, %ymm4 + vpsllq $0x27, %ymm0, %ymm0 + vpor %ymm0, %ymm5, %ymm5 + vpandn %ymm5, %ymm1, %ymm0 + vpxor %ymm2, %ymm0, %ymm0 + vmovdqu %ymm0, 0x1c0(%rsp) + vpsrlq $0x17, %ymm3, %ymm0 + vpsllq $0x29, %ymm3, %ymm3 + vpor %ymm3, %ymm0, %ymm0 + vpandn %ymm4, %ymm0, %ymm7 + vpandn %ymm0, %ymm5, %ymm3 + vpxor %ymm5, %ymm7, %ymm7 + vpandn %ymm2, %ymm4, %ymm5 + vpandn %ymm1, %ymm2, %ymm2 + vpxor %ymm0, %ymm5, %ymm5 + vpxor %ymm1, %ymm3, %ymm3 + vpxor %ymm4, %ymm2, %ymm2 + vmovdqu %ymm5, 0x1e0(%rsp) + addq $0x8, %rsi + addq $0x1, %r10 + cmpq $0x18, %r10 + jne LLmlk_keccak_f1600_x4_avx2 + vmovdqu (%rsp), %ymm4 + vmovdqu 0x40(%rsp), %ymm5 + vmovdqu 0x20(%rsp), %ymm0 + vmovdqu 0x60(%rsp), %ymm1 + vmovdqu 0x1c0(%rsp), %ymm12 + vmovdqu %ymm2, 0x1c0(%rsp) + vpunpcklqdq %ymm0, %ymm4, %ymm2 # ymm2 = ymm4[0],ymm0[0],ymm4[2],ymm0[2] + vpunpckhqdq %ymm0, %ymm4, %ymm0 # ymm0 = ymm4[1],ymm0[1],ymm4[3],ymm0[3] + vpunpcklqdq %ymm1, %ymm5, %ymm4 # ymm4 = ymm5[0],ymm1[0],ymm5[2],ymm1[2] + vpunpckhqdq %ymm1, %ymm5, %ymm1 # ymm1 = ymm5[1],ymm1[1],ymm5[3],ymm1[3] + vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1] + vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3] + vmovdqu 0x80(%rsp), %ymm4 + vperm2i128 $0x20, %ymm1, %ymm0, %ymm5 # ymm5 = ymm0[0,1],ymm1[0,1] + vperm2i128 $0x31, %ymm1, %ymm0, %ymm0 # ymm0 = ymm0[2,3],ymm1[2,3] + vmovdqu %ymm6, (%rdi) + vmovdqu %ymm5, 0xc8(%rdi) + vmovdqu %ymm2, 0x190(%rdi) + vmovdqu %ymm0, 0x258(%rdi) + vmovdqu 0xa0(%rsp), %ymm0 + vpunpcklqdq %ymm0, %ymm4, %ymm2 # ymm2 = ymm4[0],ymm0[0],ymm4[2],ymm0[2] + vpunpckhqdq %ymm0, %ymm4, %ymm1 # ymm1 = ymm4[1],ymm0[1],ymm4[3],ymm0[3] + vmovdqu 0xc0(%rsp), %ymm0 + vpunpcklqdq %ymm10, %ymm0, %ymm4 # ymm4 = ymm0[0],ymm10[0],ymm0[2],ymm10[2] + vpunpckhqdq %ymm10, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm10[1],ymm0[3],ymm10[3] + vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1] + vperm2i128 $0x20, %ymm0, %ymm1, %ymm5 # ymm5 = ymm1[0,1],ymm0[0,1] + vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3] + vmovdqu 0xe0(%rsp), %ymm4 + vperm2i128 $0x31, %ymm0, %ymm1, %ymm1 # ymm1 = ymm1[2,3],ymm0[2,3] + vmovdqu 0x100(%rsp), %ymm0 + vmovdqu %ymm2, 0x1b0(%rdi) + vmovdqu %ymm1, 0x278(%rdi) + vpunpcklqdq %ymm4, %ymm14, %ymm2 # ymm2 = ymm14[0],ymm4[0],ymm14[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm14, %ymm1 # ymm1 = ymm14[1],ymm4[1],ymm14[3],ymm4[3] + vpunpcklqdq %ymm8, %ymm0, %ymm4 # ymm4 = ymm0[0],ymm8[0],ymm0[2],ymm8[2] + vpunpckhqdq %ymm8, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm8[1],ymm0[3],ymm8[3] + vmovdqu %ymm6, 0x20(%rdi) + vmovdqu %ymm5, 0xe8(%rdi) + vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1] + vperm2i128 $0x20, %ymm0, %ymm1, %ymm5 # ymm5 = ymm1[0,1],ymm0[0,1] + vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3] + vperm2i128 $0x31, %ymm0, %ymm1, %ymm1 # ymm1 = ymm1[2,3],ymm0[2,3] + vmovdqu 0x120(%rsp), %ymm4 + vmovdqu 0x140(%rsp), %ymm0 + vmovdqu %ymm2, 0x1d0(%rdi) + vmovdqu %ymm1, 0x298(%rdi) + vpunpcklqdq %ymm4, %ymm15, %ymm2 # ymm2 = ymm15[0],ymm4[0],ymm15[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm15, %ymm1 # ymm1 = ymm15[1],ymm4[1],ymm15[3],ymm4[3] + vpunpcklqdq %ymm9, %ymm0, %ymm4 # ymm4 = ymm0[0],ymm9[0],ymm0[2],ymm9[2] + vmovdqu %ymm5, 0x108(%rdi) + vpunpckhqdq %ymm9, %ymm0, %ymm0 # ymm0 = ymm0[1],ymm9[1],ymm0[3],ymm9[3] + vmovdqu %ymm6, 0x40(%rdi) + vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1] + vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3] + vperm2i128 $0x20, %ymm0, %ymm1, %ymm5 # ymm5 = ymm1[0,1],ymm0[0,1] + vmovdqu 0x160(%rsp), %ymm4 + vperm2i128 $0x31, %ymm0, %ymm1, %ymm1 # ymm1 = ymm1[2,3],ymm0[2,3] + vmovdqu 0x180(%rsp), %ymm0 + vmovdqu %ymm5, 0x128(%rdi) + vmovdqu 0x1a0(%rsp), %ymm5 + vmovdqu %ymm2, 0x1f0(%rdi) + vpunpcklqdq %ymm0, %ymm4, %ymm2 # ymm2 = ymm4[0],ymm0[0],ymm4[2],ymm0[2] + vpunpckhqdq %ymm0, %ymm4, %ymm0 # ymm0 = ymm4[1],ymm0[1],ymm4[3],ymm0[3] + vpunpcklqdq %ymm5, %ymm13, %ymm4 # ymm4 = ymm13[0],ymm5[0],ymm13[2],ymm5[2] + vmovdqu %ymm6, 0x60(%rdi) + vperm2i128 $0x20, %ymm4, %ymm2, %ymm6 # ymm6 = ymm2[0,1],ymm4[0,1] + vmovdqu %ymm1, 0x2b8(%rdi) + vperm2i128 $0x31, %ymm4, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm4[2,3] + vpunpckhqdq %ymm5, %ymm13, %ymm1 # ymm1 = ymm13[1],ymm5[1],ymm13[3],ymm5[3] + vmovdqu %ymm6, 0x80(%rdi) + vmovdqu 0x1e0(%rsp), %ymm4 + vperm2i128 $0x20, %ymm1, %ymm0, %ymm5 # ymm5 = ymm0[0,1],ymm1[0,1] + vperm2i128 $0x31, %ymm1, %ymm0, %ymm0 # ymm0 = ymm0[2,3],ymm1[2,3] + vmovdqu %ymm2, 0x210(%rdi) + vpunpcklqdq %ymm3, %ymm12, %ymm2 # ymm2 = ymm12[0],ymm3[0],ymm12[2],ymm3[2] + vmovdqu %ymm0, 0x2d8(%rdi) + vpunpckhqdq %ymm3, %ymm12, %ymm0 # ymm0 = ymm12[1],ymm3[1],ymm12[3],ymm3[3] + vpunpcklqdq %ymm4, %ymm7, %ymm3 # ymm3 = ymm7[0],ymm4[0],ymm7[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm7, %ymm1 # ymm1 = ymm7[1],ymm4[1],ymm7[3],ymm4[3] + vmovdqu %ymm5, 0x148(%rdi) + vperm2i128 $0x20, %ymm3, %ymm2, %ymm5 # ymm5 = ymm2[0,1],ymm3[0,1] + vperm2i128 $0x31, %ymm3, %ymm2, %ymm2 # ymm2 = ymm2[2,3],ymm3[2,3] + vmovdqu 0x1c0(%rsp), %ymm3 + vperm2i128 $0x20, %ymm1, %ymm0, %ymm4 # ymm4 = ymm0[0,1],ymm1[0,1] + vperm2i128 $0x31, %ymm1, %ymm0, %ymm0 # ymm0 = ymm0[2,3],ymm1[2,3] + vmovdqu %ymm5, 0xa0(%rdi) + vextracti128 $0x1, %ymm3, %xmm15 + vmovdqu %ymm4, 0x168(%rdi) + vmovdqu %ymm2, 0x230(%rdi) + vmovdqu %ymm0, 0x2f8(%rdi) + vmovq %xmm3, 0xc0(%rdi) + vmovhpd %xmm3, 0x188(%rdi) + vmovq %xmm15, 0x250(%rdi) + vmovhpd %xmm15, 0x318(%rdi) + movq %r11, %rsp + .cfi_def_cfa_register %rsp + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(keccak_f1600_x4_avx2_asm) + +#endif /* MLK_FIPS202_X86_64_NEED_X4_AVX2 && !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/fips202/native/x86_64/src/keccakf1600_constants.c b/cbits/mlkem/src/fips202/native/x86_64/src/keccakf1600_constants.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/fips202/native/x86_64/src/keccakf1600_constants.c @@ -0,0 +1,52 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mlkem-native repository. + * Do not modify it directly. + */ + +#include "../../../../common.h" +#if defined(MLK_FIPS202_X86_64_NEED_X4_AVX2) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +#include + +#include "fips202_native_x86_64.h" + +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint64_t + mlk_keccakf1600_round_constants[24] = { + 0x0000000000000001, 0x0000000000008082, 0x800000000000808a, + 0x8000000080008000, 0x000000000000808b, 0x0000000080000001, + 0x8000000080008081, 0x8000000000008009, 0x000000000000008a, + 0x0000000000000088, 0x0000000080008009, 0x000000008000000a, + 0x000000008000808b, 0x800000000000008b, 0x8000000000008089, + 0x8000000000008003, 0x8000000000008002, 0x8000000000000080, + 0x000000000000800a, 0x800000008000000a, 0x8000000080008081, + 0x8000000000008080, 0x0000000080000001, 0x8000000080008008, +}; + +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint64_t mlk_keccak_rho8[4] = { + 0x0605040302010007, + 0x0e0d0c0b0a09080f, + 0x1615141312111017, + 0x1e1d1c1b1a19181f, +}; + +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint64_t mlk_keccak_rho56[4] = { + 0x0007060504030201, + 0x080f0e0d0c0b0a09, + 0x1017161514131211, + 0x181f1e1d1c1b1a19, +}; + +#else /* MLK_FIPS202_X86_64_NEED_X4_AVX2 && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +MLK_EMPTY_CU(fips202_x86_64_constants) + +#endif /* !(MLK_FIPS202_X86_64_NEED_X4_AVX2 && \ + !MLK_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mlkem/src/indcpa.c b/cbits/mlkem/src/indcpa.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/indcpa.c @@ -0,0 +1,675 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + * + * - [REF] + * CRYSTALS-Kyber C reference implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/ref + */ + +#include "indcpa.h" + +#include "debug.h" +#include "randombytes.h" +#include "sampling.h" +#include "symmetric.h" +#include "verify.h" + +/* Parameter set namespacing + * This is to facilitate building multiple instances + * of mlkem-native (e.g. with varying parameter sets) + * within a single compilation unit. */ +#define mlk_pack_pk MLK_ADD_PARAM_SET(mlk_pack_pk) +#define mlk_unpack_pk MLK_ADD_PARAM_SET(mlk_unpack_pk) +#define mlk_pack_sk MLK_ADD_PARAM_SET(mlk_pack_sk) +#define mlk_unpack_sk MLK_ADD_PARAM_SET(mlk_unpack_sk) +#define mlk_pack_ciphertext MLK_ADD_PARAM_SET(mlk_pack_ciphertext) +#define mlk_unpack_ciphertext MLK_ADD_PARAM_SET(mlk_unpack_ciphertext) +#define mlk_matvec_mul MLK_ADD_PARAM_SET(mlk_matvec_mul) +#define mlk_polyvec_permute_bitrev_to_custom \ + MLK_ADD_PARAM_SET(mlk_polyvec_permute_bitrev_to_custom) +#define mlk_polymat_permute_bitrev_to_custom \ + MLK_ADD_PARAM_SET(mlk_polymat_permute_bitrev_to_custom) +#define mlk_keypair_getnoise_eta1 MLK_ADD_PARAM_SET(mlk_keypair_getnoise_eta1) +#define mlk_enc_getnoise_eta1_eta2 MLK_ADD_PARAM_SET(mlk_enc_getnoise_eta1_eta2) +/* End of parameter set namespacing */ + +/** + * Serialize the public key as the concatenation of the serialized vector of + * polynomials pk and the public seed used to generate the matrix A. + * + * @spec{Implements @[FIPS203, Algorithm 13 (K-PKE.KeyGen), L19].} + * + * @param[out] r Output serialized public key. + * @param[in] pk Input public-key polyvec. Must have coefficients within + * [0,..,MLKEM_Q-1]. + * @param[in] seed Input public seed. + */ +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) +static void mlk_pack_pk(uint8_t r[MLKEM_INDCPA_PUBLICKEYBYTES], + const mlk_polyvec *pk, + const uint8_t seed[MLKEM_SYMBYTES]) +{ + mlk_assert_bound_2d(pk->vec, MLKEM_K, MLKEM_N, 0, MLKEM_Q); + mlk_polyvec_tobytes(r, pk); + mlk_memcpy(r + MLKEM_POLYVECBYTES, seed, MLKEM_SYMBYTES); +} +#endif /* !MLK_CONFIG_NO_KEYPAIR_API */ + +/** + * De-serialize public key from a byte array; approximate inverse of + * mlk_pack_pk. + * + * @spec{Implements @[FIPS203, Algorithm 14 (K-PKE.Encrypt), L2-3].} + * + * @param[out] pk Output public-key polynomial vector. Coefficients + * will be normalized to [0,1,..,MLKEM_Q-1]. + * @param[out] seed Output seed to generate matrix A. + * @param[in] packedpk Input serialized public key. + */ +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +static void mlk_unpack_pk(mlk_polyvec *pk, uint8_t seed[MLKEM_SYMBYTES], + const uint8_t packedpk[MLKEM_INDCPA_PUBLICKEYBYTES]) +{ + mlk_polyvec_frombytes(pk, packedpk); + mlk_memcpy(seed, packedpk + MLKEM_POLYVECBYTES, MLKEM_SYMBYTES); + + /* NOTE: If a modulus check was conducted on the PK, we know at this + * point that the coefficients of `pk` are unsigned canonical. The + * specifications and proofs, however, do _not_ assume this, and instead + * work with the easily provable bound by MLKEM_UINT12_LIMIT. */ +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +/** + * Serialize the secret key. + * + * @spec{Implements @[FIPS203, Algorithm 13 (K-PKE.KeyGen), L20].} + * + * @param[out] r Output serialized secret key. + * @param[in] sk Input vector of polynomials (secret key). + */ +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) +static void mlk_pack_sk(uint8_t r[MLKEM_INDCPA_SECRETKEYBYTES], + const mlk_polyvec *sk) +{ + mlk_assert_bound_2d(sk->vec, MLKEM_K, MLKEM_N, 0, MLKEM_Q); + mlk_polyvec_tobytes(r, sk); +} +#endif /* !MLK_CONFIG_NO_KEYPAIR_API */ + +/** + * De-serialize the secret key; inverse of mlk_pack_sk. + * + * @spec{Implements @[FIPS203, Algorithm 15 (K-PKE.Decrypt), L5].} + * + * @param[out] sk Output vector of polynomials (secret key). + * @param[in] packedsk Input serialized secret key. + */ +#if !defined(MLK_CONFIG_NO_DECAPS_API) +static void mlk_unpack_sk(mlk_polyvec *sk, + const uint8_t packedsk[MLKEM_INDCPA_SECRETKEYBYTES]) +{ + mlk_polyvec_frombytes(sk, packedsk); +} +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +/** + * Serialize the ciphertext as the concatenation of the compressed and + * serialized vector of polynomials b and the compressed and serialized + * polynomial v. + * + * @spec{Implements @[FIPS203, Algorithm 14 (K-PKE.Encrypt), L22-23].} + * + * @param[out] r Output serialized ciphertext. + * @param[in] b Input vector of polynomials b. + * @param[in] v Input polynomial v. + */ +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +static void mlk_pack_ciphertext(uint8_t r[MLKEM_INDCPA_BYTES], + const mlk_polyvec *b, mlk_poly *v) +{ + mlk_polyvec_compress_du(r, b); + mlk_poly_compress_dv(r + MLKEM_POLYVECCOMPRESSEDBYTES_DU, v); +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +/** + * De-serialize and decompress ciphertext from a byte array; approximate + * inverse of mlk_pack_ciphertext. + * + * @spec{Implements @[FIPS203, Algorithm 15 (K-PKE.Decrypt), L1-4].} + * + * @param[out] b Output vector of polynomials b. + * @param[out] v Output polynomial v. + * @param[in] c Input serialized ciphertext. + */ +#if !defined(MLK_CONFIG_NO_DECAPS_API) +static void mlk_unpack_ciphertext(mlk_polyvec *b, mlk_poly *v, + const uint8_t c[MLKEM_INDCPA_BYTES]) +{ + mlk_polyvec_decompress_du(b, c); + mlk_poly_decompress_dv(v, c + MLKEM_POLYVECCOMPRESSEDBYTES_DU); +} +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +/* Helper function to ensure that the polynomial entries in the output + * of gen_matrix use the standard (bitreversed) ordering of coefficients. + * No-op unless a native backend with a custom ordering is used. + * + * We don't inline this into gen_matrix to avoid having to split the CBMC + * proof for gen_matrix based on MLK_USE_NATIVE_NTT_CUSTOM_ORDER. */ +static void mlk_polyvec_permute_bitrev_to_custom(mlk_polyvec *v) +__contract__( + /* We don't specify that this should be a permutation, but only + * that it does not change the bound established at the end of mlk_gen_matrix. */ + requires(memory_no_alias(v, sizeof(mlk_polyvec))) + requires(forall(x, 0, MLKEM_K, + array_bound(v->vec[x].coeffs, 0, MLKEM_N, 0, MLKEM_Q))) + assigns(memory_slice(v, sizeof(mlk_polyvec))) + ensures(forall(x, 0, MLKEM_K, + array_bound(v->vec[x].coeffs, 0, MLKEM_N, 0, MLKEM_Q)))) +{ +#if defined(MLK_USE_NATIVE_NTT_CUSTOM_ORDER) + unsigned i; + for (i = 0; i < MLKEM_K; i++) + __loop__( + assigns(i, memory_slice(v, sizeof(mlk_polyvec))) + invariant(i <= MLKEM_K) + invariant(forall(x, 0, MLKEM_K, + array_bound(v->vec[x].coeffs, 0, MLKEM_N, 0, MLKEM_Q))) + decreases(MLKEM_K - i)) + { + mlk_poly_permute_bitrev_to_custom(v->vec[i].coeffs); + } +#else /* MLK_USE_NATIVE_NTT_CUSTOM_ORDER */ + /* Nothing to do */ + (void)v; +#endif /* !MLK_USE_NATIVE_NTT_CUSTOM_ORDER */ +} + +static void mlk_polymat_permute_bitrev_to_custom(mlk_polymat *a) +__contract__( + /* We don't specify that this should be a permutation, but only + * that it does not change the bound established at the end of mlk_gen_matrix. */ + requires(memory_no_alias(a, sizeof(mlk_polymat))) + requires(forall(x, 0, MLKEM_K, forall(y, 0, MLKEM_K, + array_bound(a->vec[x].vec[y].coeffs, 0, MLKEM_N, 0, MLKEM_Q)))) + assigns(memory_slice(a, sizeof(mlk_polymat))) + ensures(forall(x, 0, MLKEM_K, forall(y, 0, MLKEM_K, + array_bound(a->vec[x].vec[y].coeffs, 0, MLKEM_N, 0, MLKEM_Q))))) +{ + unsigned i; + for (i = 0; i < MLKEM_K; i++) + __loop__( + assigns(i, memory_slice(a, sizeof(mlk_polymat))) + invariant(i <= MLKEM_K) + invariant(forall(x, 0, MLKEM_K, forall(y, 0, MLKEM_K, + array_bound(a->vec[x].vec[y].coeffs, 0, MLKEM_N, 0, MLKEM_Q)))) + decreases(MLKEM_K - i)) + { + mlk_polyvec_permute_bitrev_to_custom(&a->vec[i]); + } +} + +/* Reference: `gen_matrix()` in the reference implementation @[REF]. + * - We use a special subroutine to generate 4 polynomials + * at a time, to be able to leverage batched Keccak-f1600 + * implementations. The reference implementation generates + * one matrix entry a time. + * + * Not static for benchmarking */ +MLK_INTERNAL_API +void mlk_gen_matrix(mlk_polymat *a, const uint8_t seed[MLKEM_SYMBYTES], + int transposed) +{ + unsigned i, j; + MLK_ALIGN uint8_t seed_ext[4][MLK_ALIGN_UP(MLKEM_SYMBYTES + 2)]; + + for (j = 0; j < 4; j++) + { + mlk_memcpy(seed_ext[j], seed, MLKEM_SYMBYTES); + } + +#if !defined(MLK_CONFIG_SERIAL_FIPS202_ONLY) + /* Sample 4 matrix entries a time. */ + for (i = 0; i < (MLKEM_K * MLKEM_K / 4) * 4; i += 4) + { + for (j = 0; j < 4; j++) + { + uint8_t x, y; + /* MLKEM_K <= 4, so the values fit in uint8_t. */ + x = (uint8_t)((i + j) / MLKEM_K); + y = (uint8_t)((i + j) % MLKEM_K); + if (transposed) + { + seed_ext[j][MLKEM_SYMBYTES + 0] = x; + seed_ext[j][MLKEM_SYMBYTES + 1] = y; + } + else + { + seed_ext[j][MLKEM_SYMBYTES + 0] = y; + seed_ext[j][MLKEM_SYMBYTES + 1] = x; + } + } + + mlk_poly_rej_uniform_x4(&a->vec[i / MLKEM_K].vec[i % MLKEM_K], + &a->vec[(i + 1) / MLKEM_K].vec[(i + 1) % MLKEM_K], + &a->vec[(i + 2) / MLKEM_K].vec[(i + 2) % MLKEM_K], + &a->vec[(i + 3) / MLKEM_K].vec[(i + 3) % MLKEM_K], + seed_ext); + } +#else /* !MLK_CONFIG_SERIAL_FIPS202_ONLY */ + /* When using serial FIPS202, sample all entries individually. */ + i = 0; +#endif /* MLK_CONFIG_SERIAL_FIPS202_ONLY */ + + /* For MLKEM_K == 3, sample the last entry individually. + * When MLK_CONFIG_SERIAL_FIPS202_ONLY is set, sample all entries + * individually. */ + for (; i < MLKEM_K * MLKEM_K; i++) + { + uint8_t x, y; + /* MLKEM_K <= 4, so the values fit in uint8_t. */ + x = (uint8_t)(i / MLKEM_K); + y = (uint8_t)(i % MLKEM_K); + + if (transposed) + { + seed_ext[0][MLKEM_SYMBYTES + 0] = x; + seed_ext[0][MLKEM_SYMBYTES + 1] = y; + } + else + { + seed_ext[0][MLKEM_SYMBYTES + 0] = y; + seed_ext[0][MLKEM_SYMBYTES + 1] = x; + } + + mlk_poly_rej_uniform(&a->vec[i / MLKEM_K].vec[i % MLKEM_K], seed_ext[0]); + } + + mlk_assert(i == MLKEM_K * MLKEM_K); + + /* + * The public matrix is generated in NTT domain. If the native backend + * uses a custom order in NTT domain, permute A accordingly. + */ + mlk_polymat_permute_bitrev_to_custom(a); + + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + mlk_zeroize(seed_ext, sizeof(seed_ext)); +} + +/** + * Compute matrix-vector product in NTT domain, via Montgomery multiplication. + * + * @spec{Implements @[FIPS203, Section 2.4.7, Eq (2.12), (2.13)].} + * + * @param[out] out Output polynomial vector. + * @param[in] a Input matrix. Must be in NTT domain and have coefficients + * of absolute value < 4096. + * @param[in] v Input polynomial vector. Must be in NTT domain. + * @param[in] vc Mulcache for @p v, computed via + * mlk_polyvec_mulcache_compute(). + */ +static void mlk_matvec_mul(mlk_polyvec *out, const mlk_polymat *a, + const mlk_polyvec *v, const mlk_polyvec_mulcache *vc) +__contract__( + requires(memory_no_alias(out, sizeof(mlk_polyvec))) + requires(memory_no_alias(a, sizeof(mlk_polymat))) + requires(memory_no_alias(v, sizeof(mlk_polyvec))) + requires(memory_no_alias(vc, sizeof(mlk_polyvec_mulcache))) + requires(forall(k0, 0, MLKEM_K, + forall(k1, 0, MLKEM_K, + array_bound(a->vec[k0].vec[k1].coeffs, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT)))) + assigns(memory_slice(out, sizeof(mlk_polyvec)))) +{ + unsigned i; + for (i = 0; i < MLKEM_K; i++) + __loop__( + assigns(i, memory_slice(out, sizeof(mlk_polyvec))) + invariant(i <= MLKEM_K) + decreases(MLKEM_K - i)) + { + mlk_polyvec_basemul_acc_montgomery_cached(&out->vec[i], &a->vec[i], v, vc); + } +} + +/** + * Compute and fill the pv and e polyvec structures needed by + * mlk_keypair_derand(). Uses x4-batched versions of `poly_getnoise` to + * leverage batched Keccak-f1600. + * + * @spec{Implements @[FIPS203, Algorithm 13 (K-PKE.KeyGen)] steps 8-15.} + * + * @param[out] pv Output polynomial vector. + * @param[out] e Output polynomial vector. + * @param[in] seed Seed bytes for sampling. + */ +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) +static void mlk_keypair_getnoise_eta1(mlk_polyvec *pv, mlk_polyvec *e, + const uint8_t seed[MLKEM_SYMBYTES]) +__contract__( + requires(memory_no_alias(pv, sizeof(mlk_polyvec))) + requires(memory_no_alias(e, sizeof(mlk_polyvec))) + requires(memory_no_alias(seed, MLKEM_SYMBYTES)) + assigns(memory_slice(pv, sizeof(mlk_polyvec))) + assigns(memory_slice(e, sizeof(mlk_polyvec))) + ensures(forall(k0, 0, MLKEM_K, array_abs_bound(pv->vec[k0].coeffs, 0, MLKEM_N, MLKEM_ETA1 + 1))) + ensures(forall(k1, 0, MLKEM_K, array_abs_bound(e->vec[k1].coeffs, 0, MLKEM_N, MLKEM_ETA1 + 1))) +) +{ +#if MLKEM_K == 2 + mlk_poly_getnoise_eta1_4x(&pv->vec[0], &pv->vec[1], /* Fill elements of pv */ + &e->vec[0], &e->vec[1], /* and two elements of e */ + seed, 0, 1, 2, 3); +#elif MLKEM_K == 3 + /* + * Only the first three output buffers are needed, so we pass NULL as + * the fourth parameter, and 0xFF as its dummy nonce. + */ + mlk_poly_getnoise_eta1_4x(&pv->vec[0], &pv->vec[1], &pv->vec[2], NULL, seed, + 0, 1, 2, 0xFF); + /* Same here */ + mlk_poly_getnoise_eta1_4x(&e->vec[0], &e->vec[1], &e->vec[2], NULL, seed, 3, + 4, 5, 0xFF); +#elif MLKEM_K == 4 + mlk_poly_getnoise_eta1_4x(&pv->vec[0], &pv->vec[1], &pv->vec[2], &pv->vec[3], + seed, 0, 1, 2, 3); + mlk_poly_getnoise_eta1_4x(&e->vec[0], &e->vec[1], &e->vec[2], &e->vec[3], + seed, 4, 5, 6, 7); +#endif /* MLKEM_K == 4 */ +} +#endif /* !MLK_CONFIG_NO_KEYPAIR_API */ + +/** + * Compute and fill the sp, ep, and epp polynomial structures needed by + * mlk_indcpa_enc(). Uses x4-batched versions of `poly_getnoise` to leverage + * batched Keccak-f1600. + * + * @spec{Implements @[FIPS203, Algorithm 14 (K-PKE.Encrypt)] steps 9-16.} + * + * @param[out] sp Output polynomial vector. + * @param[out] ep Output polynomial vector. + * @param[out] epp Output polynomial. + * @param[in] coins Seed bytes for sampling. + */ +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +static void mlk_enc_getnoise_eta1_eta2(mlk_polyvec *sp, mlk_polyvec *ep, + mlk_poly *epp, + const uint8_t coins[MLKEM_SYMBYTES]) +__contract__( + requires(memory_no_alias(sp, sizeof(mlk_polyvec))) + requires(memory_no_alias(ep, sizeof(mlk_polyvec))) + requires(memory_no_alias(epp, sizeof(mlk_poly))) + requires(memory_no_alias(coins, MLKEM_SYMBYTES)) + assigns(memory_slice(sp, sizeof(mlk_polyvec))) + assigns(memory_slice(ep, sizeof(mlk_polyvec))) + assigns(memory_slice(epp, sizeof(mlk_poly))) + ensures(forall(k0, 0, MLKEM_K, array_abs_bound(sp->vec[k0].coeffs, 0, MLKEM_N, MLKEM_ETA1 + 1))) + ensures(forall(k1, 0, MLKEM_K, array_abs_bound(ep->vec[k1].coeffs, 0, MLKEM_N, MLKEM_ETA2 + 1))) + ensures(array_abs_bound(epp->coeffs, 0, MLKEM_N, MLKEM_ETA2 + 1)) +) +{ +#if MLKEM_K == 2 + mlk_poly_getnoise_eta1122_4x(&sp->vec[0], &sp->vec[1], &ep->vec[0], + &ep->vec[1], coins, 0, 1, 2, 3); + mlk_poly_getnoise_eta2(epp, coins, 4); +#elif MLKEM_K == 3 + /* + * In this call, only the first three output buffers are needed, + * so we pass NULL as the fourth parameter, and 0xFF as its dummy nonce. + */ + mlk_poly_getnoise_eta1_4x(&sp->vec[0], &sp->vec[1], &sp->vec[2], NULL, coins, + 0, 1, 2, 0xFF /* irrelevant */); + /* The fourth output buffer in this call _is_ used. */ + mlk_poly_getnoise_eta2_4x(&ep->vec[0], &ep->vec[1], &ep->vec[2], epp, coins, + 3, 4, 5, 6); +#elif MLKEM_K == 4 + mlk_poly_getnoise_eta1_4x(&sp->vec[0], &sp->vec[1], &sp->vec[2], &sp->vec[3], + coins, 0, 1, 2, 3); + mlk_poly_getnoise_eta2_4x(&ep->vec[0], &ep->vec[1], &ep->vec[2], &ep->vec[3], + coins, 4, 5, 6, 7); + mlk_poly_getnoise_eta2(epp, coins, 8); +#endif /* MLKEM_K == 4 */ +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + + +/* Reference: `indcpa_keypair_derand()` in the reference implementation @[REF]. + * - We use a different implementation of `gen_matrix()` which + * uses x4-batched Keccak-f1600 (see `mlk_gen_matrix()` above). + * - We use a mulcache to speed up matrix-vector multiplication. + * - We include buffer zeroization. + */ +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) +MLK_INTERNAL_API +int mlk_indcpa_keypair_derand(uint8_t pk[MLKEM_INDCPA_PUBLICKEYBYTES], + uint8_t sk[MLKEM_INDCPA_SECRETKEYBYTES], + const uint8_t coins[MLKEM_SYMBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret = 0; + const uint8_t *publicseed; + const uint8_t *noiseseed; + MLK_ALLOC(buf, uint8_t, 2 * MLKEM_SYMBYTES, context); + MLK_ALLOC(coins_with_domain_separator, uint8_t, MLKEM_SYMBYTES + 1, context); + MLK_ALLOC(a, mlk_polymat, 1, context); + MLK_ALLOC(e, mlk_polyvec, 1, context); + MLK_ALLOC(pkpv, mlk_polyvec, 1, context); + MLK_ALLOC(skpv, mlk_polyvec, 1, context); + MLK_ALLOC(skpv_cache, mlk_polyvec_mulcache, 1, context); + + if (buf == NULL || coins_with_domain_separator == NULL || a == NULL || + e == NULL || pkpv == NULL || skpv == NULL || skpv_cache == NULL) + { + ret = MLK_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + publicseed = buf; + noiseseed = buf + MLKEM_SYMBYTES; + + /* Concatenate coins with MLKEM_K for domain separation of security levels */ + mlk_memcpy(coins_with_domain_separator, coins, MLKEM_SYMBYTES); + coins_with_domain_separator[MLKEM_SYMBYTES] = MLKEM_K; + + mlk_hash_g(buf, coins_with_domain_separator, MLKEM_SYMBYTES + 1); + + /* + * Declassify the public seed. + * Required to use it in conditional-branches in rejection sampling. + * This is needed because all output of randombytes is marked as secret + * (=undefined) + */ + MLK_CT_TESTING_DECLASSIFY(publicseed, MLKEM_SYMBYTES); + + mlk_gen_matrix(a, publicseed, 0 /* no transpose */); + + mlk_keypair_getnoise_eta1(skpv, e, noiseseed); + + mlk_polyvec_ntt(skpv); + mlk_polyvec_ntt(e); + + mlk_polyvec_mulcache_compute(skpv_cache, skpv); + mlk_matvec_mul(pkpv, a, skpv, skpv_cache); + mlk_polyvec_tomont(pkpv); + + mlk_polyvec_add(pkpv, e); + mlk_polyvec_reduce(pkpv); + mlk_polyvec_reduce(skpv); + + mlk_pack_sk(sk, skpv); + mlk_pack_pk(pk, pkpv, publicseed); + +cleanup: + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + MLK_FREE(skpv_cache, mlk_polyvec_mulcache, 1, context); + MLK_FREE(skpv, mlk_polyvec, 1, context); + MLK_FREE(pkpv, mlk_polyvec, 1, context); + MLK_FREE(e, mlk_polyvec, 1, context); + MLK_FREE(a, mlk_polymat, 1, context); + MLK_FREE(coins_with_domain_separator, uint8_t, MLKEM_SYMBYTES + 1, context); + MLK_FREE(buf, uint8_t, 2 * MLKEM_SYMBYTES, context); + return ret; +} +#endif /* !MLK_CONFIG_NO_KEYPAIR_API */ + +/* Reference: `indcpa_enc()` in the reference implementation @[REF]. + * - We use x4-batched versions of `poly_getnoise` to leverage + * batched x4-batched Keccak-f1600. + * - We use a different implementation of `gen_matrix()` which + * uses x4-batched Keccak-f1600 (see `mlk_gen_matrix()` above). + * - We use a mulcache to speed up matrix-vector multiplication. + * - We include buffer zeroization. + */ +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +MLK_INTERNAL_API +int mlk_indcpa_enc(uint8_t c[MLKEM_INDCPA_BYTES], + const uint8_t m[MLKEM_INDCPA_MSGBYTES], + const uint8_t pk[MLKEM_INDCPA_PUBLICKEYBYTES], + const uint8_t coins[MLKEM_SYMBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret = 0; + MLK_ALLOC(seed, uint8_t, MLKEM_SYMBYTES, context); + MLK_ALLOC(at, mlk_polymat, 1, context); + MLK_ALLOC(sp, mlk_polyvec, 1, context); + MLK_ALLOC(pkpv, mlk_polyvec, 1, context); + MLK_ALLOC(ep, mlk_polyvec, 1, context); + MLK_ALLOC(b, mlk_polyvec, 1, context); + MLK_ALLOC(v, mlk_poly, 1, context); + MLK_ALLOC(k, mlk_poly, 1, context); + MLK_ALLOC(epp, mlk_poly, 1, context); + MLK_ALLOC(sp_cache, mlk_polyvec_mulcache, 1, context); + + if (seed == NULL || at == NULL || sp == NULL || pkpv == NULL || ep == NULL || + b == NULL || v == NULL || k == NULL || epp == NULL || sp_cache == NULL) + { + ret = MLK_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + mlk_unpack_pk(pkpv, seed, pk); + mlk_poly_frommsg(k, m); + + /* + * Declassify the public seed. + * Required to use it in conditional-branches in rejection sampling. + * This is needed because in re-encryption the publicseed originated from sk + * which is marked undefined. + */ + MLK_CT_TESTING_DECLASSIFY(seed, MLKEM_SYMBYTES); + + mlk_gen_matrix(at, seed, 1 /* transpose */); + + mlk_enc_getnoise_eta1_eta2(sp, ep, epp, coins); + + mlk_polyvec_ntt(sp); + + mlk_polyvec_mulcache_compute(sp_cache, sp); + mlk_matvec_mul(b, at, sp, sp_cache); + mlk_polyvec_basemul_acc_montgomery_cached(v, pkpv, sp, sp_cache); + + mlk_polyvec_invntt_tomont(b); + mlk_poly_invntt_tomont(v); + + mlk_polyvec_add(b, ep); + mlk_poly_add(v, epp); + mlk_poly_add(v, k); + + mlk_polyvec_reduce(b); + mlk_poly_reduce(v); + + mlk_pack_ciphertext(c, b, v); + +cleanup: + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + MLK_FREE(sp_cache, mlk_polyvec_mulcache, 1, context); + MLK_FREE(epp, mlk_poly, 1, context); + MLK_FREE(k, mlk_poly, 1, context); + MLK_FREE(v, mlk_poly, 1, context); + MLK_FREE(b, mlk_polyvec, 1, context); + MLK_FREE(ep, mlk_polyvec, 1, context); + MLK_FREE(pkpv, mlk_polyvec, 1, context); + MLK_FREE(sp, mlk_polyvec, 1, context); + MLK_FREE(at, mlk_polymat, 1, context); + MLK_FREE(seed, uint8_t, MLKEM_SYMBYTES, context); + return ret; +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +/* Reference: `indcpa_dec()` in the reference implementation @[REF]. + * - We use a mulcache for the scalar product. + * - We include buffer zeroization. */ +#if !defined(MLK_CONFIG_NO_DECAPS_API) +MLK_INTERNAL_API +int mlk_indcpa_dec(uint8_t m[MLKEM_INDCPA_MSGBYTES], + const uint8_t c[MLKEM_INDCPA_BYTES], + const uint8_t sk[MLKEM_INDCPA_SECRETKEYBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret = 0; + MLK_ALLOC(b, mlk_polyvec, 1, context); + MLK_ALLOC(skpv, mlk_polyvec, 1, context); + MLK_ALLOC(v, mlk_poly, 1, context); + MLK_ALLOC(sb, mlk_poly, 1, context); + MLK_ALLOC(b_cache, mlk_polyvec_mulcache, 1, context); + + if (b == NULL || skpv == NULL || v == NULL || sb == NULL || b_cache == NULL) + { + ret = MLK_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + mlk_unpack_ciphertext(b, v, c); + mlk_unpack_sk(skpv, sk); + + mlk_polyvec_ntt(b); + mlk_polyvec_mulcache_compute(b_cache, b); + mlk_polyvec_basemul_acc_montgomery_cached(sb, skpv, b, b_cache); + mlk_poly_invntt_tomont(sb); + + mlk_poly_sub(v, sb); + mlk_poly_reduce(v); + + mlk_poly_tomsg(m, v); + +cleanup: + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + MLK_FREE(b_cache, mlk_polyvec_mulcache, 1, context); + MLK_FREE(sb, mlk_poly, 1, context); + MLK_FREE(v, mlk_poly, 1, context); + MLK_FREE(skpv, mlk_polyvec, 1, context); + MLK_FREE(b, mlk_polyvec, 1, context); + return ret; +} +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. + * Don't modify by hand -- this is auto-generated by scripts/autogen. */ +#undef mlk_pack_pk +#undef mlk_unpack_pk +#undef mlk_pack_sk +#undef mlk_unpack_sk +#undef mlk_pack_ciphertext +#undef mlk_unpack_ciphertext +#undef mlk_matvec_mul +#undef mlk_polyvec_permute_bitrev_to_custom +#undef mlk_polymat_permute_bitrev_to_custom +#undef mlk_keypair_getnoise_eta1 +#undef mlk_enc_getnoise_eta1_eta2 diff --git a/cbits/mlkem/src/indcpa.h b/cbits/mlkem/src/indcpa.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/indcpa.h @@ -0,0 +1,172 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + */ + +#ifndef MLK_INDCPA_H +#define MLK_INDCPA_H + +#include "cbmc.h" +#include "common.h" +#include "poly_k.h" + +#define mlk_gen_matrix MLK_NAMESPACE_K(gen_matrix) +/** + * Deterministically generate matrix A (or the transpose of A) from a seed. + * Entries of the matrix are polynomials that look uniformly random. + * Performs rejection sampling on the output of an XOF. + * + * @spec{Implements @[FIPS203, Algorithm 13 (K-PKE.KeyGen), L3-7] and + * @[FIPS203, Algorithm 14 (K-PKE.Encrypt), L4-8]. The @p transposed + * parameter only affects internal presentation.} + * + * @param[out] a Output matrix A. + * @param[in] seed Input seed. + * @param transposed Boolean deciding whether A or A^T is generated. + */ +MLK_INTERNAL_API +void mlk_gen_matrix(mlk_polymat *a, const uint8_t seed[MLKEM_SYMBYTES], + int transposed) +__contract__( + requires(memory_no_alias(a, sizeof(mlk_polymat))) + requires(memory_no_alias(seed, MLKEM_SYMBYTES)) + requires(transposed == 0 || transposed == 1) + assigns(memory_slice(a, sizeof(mlk_polymat))) + ensures(forall(x, 0, MLKEM_K, forall(y, 0, MLKEM_K, + array_bound(a->vec[x].vec[y].coeffs, 0, MLKEM_N, 0, MLKEM_Q)))) +); + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) +#define mlk_indcpa_keypair_derand \ + MLK_NAMESPACE_K(indcpa_keypair_derand) MLK_CONTEXT_PARAMETERS_3 +/** + * Generate public and private key for the CPA-secure public-key encryption + * scheme underlying ML-KEM. + * + * @spec{Implements @[FIPS203, Algorithm 13 (K-PKE.KeyGen)].} + * + * @param[out] pk Output public key + * (length MLKEM_INDCPA_PUBLICKEYBYTES bytes). + * @param[out] sk Output private key + * (length MLKEM_INDCPA_SECRETKEYBYTES bytes). + * @param[in] coins Input randomness (length MLKEM_SYMBYTES bytes). + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + */ +MLK_INTERNAL_API +MLK_MUST_CHECK_RETURN_VALUE +int mlk_indcpa_keypair_derand(uint8_t pk[MLKEM_INDCPA_PUBLICKEYBYTES], + uint8_t sk[MLKEM_INDCPA_SECRETKEYBYTES], + const uint8_t coins[MLKEM_SYMBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(pk, MLKEM_INDCPA_PUBLICKEYBYTES)) + requires(memory_no_alias(sk, MLKEM_INDCPA_SECRETKEYBYTES)) + requires(memory_no_alias(coins, MLKEM_SYMBYTES)) + assigns(memory_slice(pk, MLKEM_INDCPA_PUBLICKEYBYTES)) + assigns(memory_slice(sk, MLKEM_INDCPA_SECRETKEYBYTES)) + ensures(return_value == 0 || return_value == MLK_ERR_OUT_OF_MEMORY) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(pk, MLKEM_INDCPA_PUBLICKEYBYTES)) + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(sk, MLKEM_INDCPA_SECRETKEYBYTES)) +); +#endif /* !MLK_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_indcpa_enc MLK_NAMESPACE_K(indcpa_enc) MLK_CONTEXT_PARAMETERS_4 +/** + * Encryption function of the CPA-secure public-key encryption scheme + * underlying ML-KEM. + * + * @spec{Implements @[FIPS203, Algorithm 14 (K-PKE.Encrypt)].} + * + * @param[out] c Output ciphertext (length MLKEM_INDCPA_BYTES bytes). + * @param[in] m Input message (length MLKEM_INDCPA_MSGBYTES bytes). + * @param[in] pk Input public key + * (length MLKEM_INDCPA_PUBLICKEYBYTES bytes). + * @param[in] coins Input random coins used as seed (length MLKEM_SYMBYTES + * bytes) to deterministically generate all randomness. + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + */ +MLK_INTERNAL_API +MLK_MUST_CHECK_RETURN_VALUE +int mlk_indcpa_enc(uint8_t c[MLKEM_INDCPA_BYTES], + const uint8_t m[MLKEM_INDCPA_MSGBYTES], + const uint8_t pk[MLKEM_INDCPA_PUBLICKEYBYTES], + const uint8_t coins[MLKEM_SYMBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(c, MLKEM_INDCPA_BYTES)) + requires(memory_no_alias(m, MLKEM_INDCPA_MSGBYTES)) + requires(memory_no_alias(pk, MLKEM_INDCPA_PUBLICKEYBYTES)) + requires(memory_no_alias(coins, MLKEM_SYMBYTES)) + assigns(memory_slice(c, MLKEM_INDCPA_BYTES)) + ensures(return_value == 0 || return_value == MLK_ERR_OUT_OF_MEMORY) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(c, MLKEM_INDCPA_BYTES)) +); +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_indcpa_dec MLK_NAMESPACE_K(indcpa_dec) MLK_CONTEXT_PARAMETERS_3 +/** + * Decryption function of the CPA-secure public-key encryption scheme + * underlying ML-KEM. + * + * @spec{Implements @[FIPS203, Algorithm 15 (K-PKE.Decrypt)].} + * + * @param[out] m Output decrypted message + * (length MLKEM_INDCPA_MSGBYTES bytes). + * @param[in] c Input ciphertext (length MLKEM_INDCPA_BYTES bytes). + * @param[in] sk Input secret key + * (length MLKEM_INDCPA_SECRETKEYBYTES bytes). + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + */ +MLK_INTERNAL_API +MLK_MUST_CHECK_RETURN_VALUE +int mlk_indcpa_dec(uint8_t m[MLKEM_INDCPA_MSGBYTES], + const uint8_t c[MLKEM_INDCPA_BYTES], + const uint8_t sk[MLKEM_INDCPA_SECRETKEYBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(c, MLKEM_INDCPA_BYTES)) + requires(memory_no_alias(m, MLKEM_INDCPA_MSGBYTES)) + requires(memory_no_alias(sk, MLKEM_INDCPA_SECRETKEYBYTES)) + assigns(memory_slice(m, MLKEM_INDCPA_MSGBYTES)) + ensures(return_value == 0 || return_value == MLK_ERR_OUT_OF_MEMORY) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(m, MLKEM_INDCPA_MSGBYTES)) +); +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +#endif /* !MLK_INDCPA_H */ diff --git a/cbits/mlkem/src/kem.c b/cbits/mlkem/src/kem.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/kem.c @@ -0,0 +1,474 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS140_3_IG] + * Implementation Guidance for FIPS 140-3 and the Cryptographic Module + * Validation Program + * National Institute of Standards and Technology + * https://csrc.nist.gov/projects/cryptographic-module-validation-program/fips-140-3-ig-announcements + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + * + * - [REF] + * CRYSTALS-Kyber C reference implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/ref + */ + +#include "kem.h" + +#include "indcpa.h" +#include "randombytes.h" +#include "symmetric.h" +#include "verify.h" + +/* Parameter set namespacing + * This is to facilitate building multiple instances + * of mlkem-native (e.g. with varying security levels) + * within a single compilation unit. */ +#define mlk_check_pct MLK_ADD_PARAM_SET(mlk_check_pct) MLK_CONTEXT_PARAMETERS_2 +/* End of parameter set namespacing */ + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) +/* Reference: Not implemented in the reference implementation @[REF]. */ +MLK_EXTERNAL_API +MLK_MUST_CHECK_RETURN_VALUE +int mlk_kem_check_pk(const uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret = 0; + MLK_ALLOC(p, mlk_polyvec, 1, context); + MLK_ALLOC(p_reencoded, uint8_t, MLKEM_POLYVECBYTES, context); + + if (p == NULL || p_reencoded == NULL) + { + ret = MLK_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + mlk_polyvec_frombytes(p, pk); + mlk_polyvec_reduce(p); + mlk_polyvec_tobytes(p_reencoded, p); + + /* We use a constant-time memcmp here to avoid having to + * declassify the PK before the PCT has succeeded. */ + ret = mlk_ct_memcmp(pk, p_reencoded, MLKEM_POLYVECBYTES) ? MLK_ERR_INVALID_PK + : 0; + +cleanup: + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + MLK_FREE(p_reencoded, uint8_t, MLKEM_POLYVECBYTES, context); + MLK_FREE(p, mlk_polyvec, 1, context); + return ret; +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API */ + + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +/* Reference: Not implemented in the reference implementation @[REF]. */ +MLK_EXTERNAL_API +MLK_MUST_CHECK_RETURN_VALUE +int mlk_kem_check_sk(const uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret = 0; + MLK_ALLOC(test, uint8_t, MLKEM_SYMBYTES, context); + + if (test == NULL) + { + ret = MLK_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + /* + * The parts of `sk` being hashed and compared here are public, so + * no public information is leaked through the runtime or the return value + * of this function. + */ + + /* Declassify the public part of the secret key */ + MLK_CT_TESTING_DECLASSIFY(sk + MLKEM_INDCPA_SECRETKEYBYTES, + MLKEM_INDCCA_PUBLICKEYBYTES); + MLK_CT_TESTING_DECLASSIFY( + sk + MLKEM_INDCCA_SECRETKEYBYTES - 2 * MLKEM_SYMBYTES, MLKEM_SYMBYTES); + + mlk_hash_h(test, sk + MLKEM_INDCPA_SECRETKEYBYTES, + MLKEM_INDCCA_PUBLICKEYBYTES); + /* This doesn't have to be a constant-time memcmp, but it's the only place + * in the library where a normal memcmp would be used otherwise, so for sake + * of minimizing stdlib dependency, we use our constant-time one anyway. */ + ret = mlk_ct_memcmp(sk + MLKEM_INDCCA_SECRETKEYBYTES - 2 * MLKEM_SYMBYTES, + test, MLKEM_SYMBYTES) + ? MLK_ERR_INVALID_SK + : 0; + +cleanup: + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + MLK_FREE(test, uint8_t, MLKEM_SYMBYTES, context); + return ret; +} +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) +MLK_MUST_CHECK_RETURN_VALUE +static int mlk_check_pct(uint8_t const pk[MLKEM_INDCCA_PUBLICKEYBYTES], + uint8_t const sk[MLKEM_INDCCA_SECRETKEYBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(pk, MLKEM_INDCCA_PUBLICKEYBYTES)) + requires(memory_no_alias(sk, MLKEM_INDCCA_SECRETKEYBYTES)) + ensures(return_value == 0 || return_value == MLK_ERR_OUT_OF_MEMORY || + return_value == MLK_ERR_RNG_FAIL || + return_value == MLK_ERR_PCT_FAIL) +); + +#if defined(MLK_CONFIG_KEYGEN_PCT) +/* Specification: + * Partially implements 'Pairwise Consistency Test' @[FIPS140_3_IG, p.87] and + * @[FIPS203, Section 7.1, Pairwise Consistency]. */ + +/* Reference: Not implemented in the reference implementation @[REF]. + * + * @retval 0 Success. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + * @retval MLK_ERR_RNG_FAIL Random number generation failed. + * @retval MLK_ERR_PCT_FAIL The consistency check failed. */ +MLK_MUST_CHECK_RETURN_VALUE +static int mlk_check_pct(uint8_t const pk[MLKEM_INDCCA_PUBLICKEYBYTES], + uint8_t const sk[MLKEM_INDCCA_SECRETKEYBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret = 0; + MLK_ALLOC(ct, uint8_t, MLKEM_INDCCA_CIPHERTEXTBYTES, context); + MLK_ALLOC(ss_enc, uint8_t, MLKEM_SSBYTES, context); + MLK_ALLOC(ss_dec, uint8_t, MLKEM_SSBYTES, context); + + if (ct == NULL || ss_enc == NULL || ss_dec == NULL) + { + ret = MLK_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + ret = mlk_kem_enc(ct, ss_enc, pk, context); + if (ret != 0) + { + goto cleanup; + } + + ret = mlk_kem_dec(ss_dec, ct, sk, context); + if (ret != 0) + { + goto cleanup; + } + +#if defined(MLK_CONFIG_KEYGEN_PCT_BREAKAGE_TEST) + /* Deliberately break PCT for testing purposes */ + if (mlk_break_pct()) + { + ss_enc[0] = ~ss_enc[0]; + } +#endif /* MLK_CONFIG_KEYGEN_PCT_BREAKAGE_TEST */ + + ret = mlk_ct_memcmp(ss_enc, ss_dec, MLKEM_SSBYTES); + /* The result of the PCT is public. */ + MLK_CT_TESTING_DECLASSIFY(&ret, sizeof(ret)); + + if (ret != 0) + { + ret = MLK_ERR_PCT_FAIL; + } + +cleanup: + + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + MLK_FREE(ss_dec, uint8_t, MLKEM_SSBYTES, context); + MLK_FREE(ss_enc, uint8_t, MLKEM_SSBYTES, context); + MLK_FREE(ct, uint8_t, MLKEM_INDCCA_CIPHERTEXTBYTES, context); + + /* The key pair being tested was just generated by this library, so a key + * check rejecting it hints at a faulty implementation rather than at bad + * input. Report it as a PCT failure. */ + if (ret == MLK_ERR_INVALID_PK || ret == MLK_ERR_INVALID_SK) + { + ret = MLK_ERR_PCT_FAIL; + } + + /* Other error codes, e.g. platform failures like out of memory or + * randomness failure, are passed on unmodified. */ + + return ret; +} +#else /* MLK_CONFIG_KEYGEN_PCT */ +MLK_MUST_CHECK_RETURN_VALUE +static int mlk_check_pct(uint8_t const pk[MLKEM_INDCCA_PUBLICKEYBYTES], + uint8_t const sk[MLKEM_INDCCA_SECRETKEYBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + /* Skip PCT */ + ((void)pk); + ((void)sk); + MLK_CONTEXT_UNUSED(context); + return 0; +} +#endif /* !MLK_CONFIG_KEYGEN_PCT */ + +/* Reference: `crypto_kem_keypair_derand()` in the reference implementation + * @[REF]. + * - We optionally include PCT which is not present in + * the reference code. */ +MLK_EXTERNAL_API +MLK_MUST_CHECK_RETURN_VALUE +int mlk_kem_keypair_derand(uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES], + uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES], + const uint8_t coins[2 * MLKEM_SYMBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret; + + ret = mlk_indcpa_keypair_derand(pk, sk, coins, context); + if (ret != 0) + { + goto cleanup; + } + + mlk_memcpy(sk + MLKEM_INDCPA_SECRETKEYBYTES, pk, MLKEM_INDCCA_PUBLICKEYBYTES); + mlk_hash_h(sk + MLKEM_INDCCA_SECRETKEYBYTES - 2 * MLKEM_SYMBYTES, pk, + MLKEM_INDCCA_PUBLICKEYBYTES); + /* Value z for pseudo-random output on reject */ + mlk_memcpy(sk + MLKEM_INDCCA_SECRETKEYBYTES - MLKEM_SYMBYTES, + coins + MLKEM_SYMBYTES, MLKEM_SYMBYTES); + + /* Declassify public key */ + MLK_CT_TESTING_DECLASSIFY(pk, MLKEM_INDCCA_PUBLICKEYBYTES); + + /* Pairwise Consistency Test (PCT) @[FIPS140_3_IG, p.87] */ + ret = mlk_check_pct(pk, sk, context); + if (ret != 0) + { + goto cleanup; + } + +cleanup: + if (ret != 0) + { + mlk_zeroize(pk, MLKEM_INDCCA_PUBLICKEYBYTES); + mlk_zeroize(sk, MLKEM_INDCCA_SECRETKEYBYTES); + } + + return ret; +} + +#if !defined(MLK_CONFIG_NO_RANDOMIZED_API) +/* Reference: `crypto_kem_keypair()` in the reference implementation @[REF] + * - We zeroize the stack buffer */ +MLK_EXTERNAL_API +int mlk_kem_keypair(uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES], + uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret = 0; + MLK_ALLOC(coins, uint8_t, 2 * MLKEM_SYMBYTES, context); + + if (coins == NULL) + { + ret = MLK_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + /* Acquire necessary randomness, and mark it as secret. */ + if (mlk_randombytes(coins, 2 * MLKEM_SYMBYTES) != 0) + { + ret = MLK_ERR_RNG_FAIL; + goto cleanup; + } + + MLK_CT_TESTING_SECRET(coins, 2 * MLKEM_SYMBYTES); + + ret = mlk_kem_keypair_derand(pk, sk, coins, context); + +cleanup: + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + MLK_FREE(coins, uint8_t, 2 * MLKEM_SYMBYTES, context); + return ret; +} +#endif /* !MLK_CONFIG_NO_RANDOMIZED_API */ +#endif /* !MLK_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) +/* Reference: `crypto_kem_enc_derand()` in the reference implementation @[REF] + * - We include public key check + * - We include stack buffer zeroization */ +MLK_EXTERNAL_API +int mlk_kem_enc_derand(uint8_t ct[MLKEM_INDCCA_CIPHERTEXTBYTES], + uint8_t ss[MLKEM_SSBYTES], + const uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES], + const uint8_t coins[MLKEM_SYMBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret = 0; + MLK_ALLOC(buf, uint8_t, 2 * MLKEM_SYMBYTES, context); + MLK_ALLOC(kr, uint8_t, 2 * MLKEM_SYMBYTES, context); + + if (buf == NULL || kr == NULL) + { + ret = MLK_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + /* Specification: Implements @[FIPS203, Section 7.2, Modulus check] */ + ret = mlk_kem_check_pk(pk, context); + if (ret != 0) + { + goto cleanup; + } + + mlk_memcpy(buf, coins, MLKEM_SYMBYTES); + + /* Multitarget countermeasure for coins + contributory KEM */ + mlk_hash_h(buf + MLKEM_SYMBYTES, pk, MLKEM_INDCCA_PUBLICKEYBYTES); + mlk_hash_g(kr, buf, 2 * MLKEM_SYMBYTES); + + /* coins are in kr+MLKEM_SYMBYTES */ + ret = mlk_indcpa_enc(ct, buf, pk, kr + MLKEM_SYMBYTES, context); + if (ret != 0) + { + goto cleanup; + } + + mlk_memcpy(ss, kr, MLKEM_SYMBYTES); + +cleanup: + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + MLK_FREE(kr, uint8_t, 2 * MLKEM_SYMBYTES, context); + MLK_FREE(buf, uint8_t, 2 * MLKEM_SYMBYTES, context); + return ret; +} + +#if !defined(MLK_CONFIG_NO_RANDOMIZED_API) +/* Reference: `crypto_kem_enc()` in the reference implementation @[REF] + * - We include stack buffer zeroization */ +MLK_EXTERNAL_API +int mlk_kem_enc(uint8_t ct[MLKEM_INDCCA_CIPHERTEXTBYTES], + uint8_t ss[MLKEM_SSBYTES], + const uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret = 0; + MLK_ALLOC(coins, uint8_t, MLKEM_SYMBYTES, context); + + if (coins == NULL) + { + ret = MLK_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + if (mlk_randombytes(coins, MLKEM_SYMBYTES) != 0) + { + ret = MLK_ERR_RNG_FAIL; + goto cleanup; + } + + MLK_CT_TESTING_SECRET(coins, MLKEM_SYMBYTES); + + ret = mlk_kem_enc_derand(ct, ss, pk, coins, context); + +cleanup: + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + MLK_FREE(coins, uint8_t, MLKEM_SYMBYTES, context); + return ret; +} +#endif /* !MLK_CONFIG_NO_RANDOMIZED_API */ +#endif /* !MLK_CONFIG_NO_ENCAPS_API */ + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +/* Reference: `crypto_kem_dec()` in the reference implementation @[REF] + * - We include secret key check + * - We include stack buffer zeroization */ +MLK_EXTERNAL_API +int mlk_kem_dec(uint8_t ss[MLKEM_SSBYTES], + const uint8_t ct[MLKEM_INDCCA_CIPHERTEXTBYTES], + const uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +{ + int ret = 0; + uint8_t fail; + const uint8_t *pk = sk + MLKEM_INDCPA_SECRETKEYBYTES; + MLK_ALLOC(buf, uint8_t, 2 * MLKEM_SYMBYTES, context); + MLK_ALLOC(kr, uint8_t, 2 * MLKEM_SYMBYTES, context); + MLK_ALLOC(tmp, uint8_t, MLKEM_SYMBYTES + MLKEM_INDCCA_CIPHERTEXTBYTES, + context); + + if (buf == NULL || kr == NULL || tmp == NULL) + { + ret = MLK_ERR_OUT_OF_MEMORY; + goto cleanup; + } + + /* Specification: Implements @[FIPS203, Section 7.3, Hash check] */ + ret = mlk_kem_check_sk(sk, context); + if (ret != 0) + { + goto cleanup; + } + + ret = mlk_indcpa_dec(buf, ct, sk, context); + if (ret != 0) + { + goto cleanup; + } + + /* Multitarget countermeasure for coins + contributory KEM */ + mlk_memcpy(buf + MLKEM_SYMBYTES, + sk + MLKEM_INDCCA_SECRETKEYBYTES - 2 * MLKEM_SYMBYTES, + MLKEM_SYMBYTES); + mlk_hash_g(kr, buf, 2 * MLKEM_SYMBYTES); + + /* Recompute and compare ciphertext */ + /* coins are in kr+MLKEM_SYMBYTES */ + ret = mlk_indcpa_enc(tmp, buf, pk, kr + MLKEM_SYMBYTES, context); + if (ret != 0) + { + goto cleanup; + } + + fail = mlk_ct_memcmp(ct, tmp, MLKEM_INDCCA_CIPHERTEXTBYTES); + + /* Compute rejection key */ + mlk_memcpy(tmp, sk + MLKEM_INDCCA_SECRETKEYBYTES - MLKEM_SYMBYTES, + MLKEM_SYMBYTES); + mlk_memcpy(tmp + MLKEM_SYMBYTES, ct, MLKEM_INDCCA_CIPHERTEXTBYTES); + mlk_hash_j(ss, tmp, MLKEM_SYMBYTES + MLKEM_INDCCA_CIPHERTEXTBYTES); + + /* Copy true key to return buffer if fail is 0 */ + mlk_ct_cmov_zero(ss, kr, MLKEM_SYMBYTES, fail); + +cleanup: + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + MLK_FREE(tmp, uint8_t, MLKEM_SYMBYTES + MLKEM_INDCCA_CIPHERTEXTBYTES, + context); + MLK_FREE(kr, uint8_t, 2 * MLKEM_SYMBYTES, context); + MLK_FREE(buf, uint8_t, 2 * MLKEM_SYMBYTES, context); + + return ret; +} +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. + * Don't modify by hand -- this is auto-generated by scripts/autogen. */ +#undef mlk_check_pct diff --git a/cbits/mlkem/src/kem.h b/cbits/mlkem/src/kem.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/kem.h @@ -0,0 +1,353 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + * + * - [REF] + * CRYSTALS-Kyber C reference implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/ref + */ + +#ifndef MLK_KEM_H +#define MLK_KEM_H + +#include "cbmc.h" +#include "common.h" +#include "sys.h" + +#if defined(MLK_CHECK_APIS) +/* Include to ensure consistency between internal kem.h + * and external mlkem_native.h. */ +#include "mlkem_native.h" + +#if MLKEM_INDCCA_SECRETKEYBYTES != \ + MLKEM_SECRETKEYBYTES(MLK_CONFIG_PARAMETER_SET) +#error Mismatch for SECRETKEYBYTES between kem.h and mlkem_native.h +#endif + +#if MLKEM_INDCCA_PUBLICKEYBYTES != \ + MLKEM_PUBLICKEYBYTES(MLK_CONFIG_PARAMETER_SET) +#error Mismatch for PUBLICKEYBYTES between kem.h and mlkem_native.h +#endif + +#if MLKEM_INDCCA_CIPHERTEXTBYTES != \ + MLKEM_CIPHERTEXTBYTES(MLK_CONFIG_PARAMETER_SET) +#error Mismatch for CIPHERTEXTBYTES between kem.h and mlkem_native.h +#endif + +#endif /* MLK_CHECK_APIS */ + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) +#define mlk_kem_keypair_derand \ + MLK_NAMESPACE_K(keypair_derand) MLK_CONTEXT_PARAMETERS_3 +#if !defined(MLK_CONFIG_NO_RANDOMIZED_API) +#define mlk_kem_keypair MLK_NAMESPACE_K(keypair) MLK_CONTEXT_PARAMETERS_2 +#endif +#endif /* !MLK_CONFIG_NO_KEYPAIR_API */ +#if !defined(MLK_CONFIG_NO_ENCAPS_API) +#define mlk_kem_enc_derand MLK_NAMESPACE_K(enc_derand) MLK_CONTEXT_PARAMETERS_4 +#if !defined(MLK_CONFIG_NO_RANDOMIZED_API) +#define mlk_kem_enc MLK_NAMESPACE_K(enc) MLK_CONTEXT_PARAMETERS_3 +#endif +#define mlk_kem_check_pk MLK_NAMESPACE_K(check_pk) MLK_CONTEXT_PARAMETERS_1 +#endif /* !MLK_CONFIG_NO_ENCAPS_API */ +#if !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_kem_dec MLK_NAMESPACE_K(dec) MLK_CONTEXT_PARAMETERS_3 +#define mlk_kem_check_sk MLK_NAMESPACE_K(check_sk) MLK_CONTEXT_PARAMETERS_1 +#endif + +/** + * Implements modulus check mandated by FIPS 203, i.e., ensures that + * coefficients are in [0,q-1]. + * + * @spec{Implements @[FIPS203, Section 7.2, 'modulus check'].} + * + * @reference{Not implemented in the reference implementation @[REF].} + * + * @param[in] pk Input public key (an already allocated array of + * MLKEM_INDCCA_PUBLICKEYBYTES bytes). + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_INVALID_PK Modulus check failed. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + */ +#if !defined(MLK_CONFIG_NO_ENCAPS_API) +MLK_EXTERNAL_API +MLK_MUST_CHECK_RETURN_VALUE +int mlk_kem_check_pk(const uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(pk, MLKEM_INDCCA_PUBLICKEYBYTES)) + ensures(return_value == 0 || return_value == MLK_ERR_INVALID_PK || + return_value == MLK_ERR_OUT_OF_MEMORY) +); +#endif /* !MLK_CONFIG_NO_ENCAPS_API */ + + +/** + * Implements public key hash check mandated by FIPS 203, i.e., ensures that + * sk[768𝑘+32 ∶ 768𝑘+64] = H(pk) = H(sk[384𝑘 : 768𝑘+32]). + * + * @spec{Implements @[FIPS203, Section 7.3, 'hash check'].} + * + * @reference{Not implemented in the reference implementation @[REF].} + * + * @param[in] sk Input private key (an already allocated array of + * MLKEM_INDCCA_SECRETKEYBYTES bytes). + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_INVALID_SK Public key hash check failed. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + */ +#if !defined(MLK_CONFIG_NO_DECAPS_API) +MLK_EXTERNAL_API +MLK_MUST_CHECK_RETURN_VALUE +int mlk_kem_check_sk(const uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(sk, MLKEM_INDCCA_SECRETKEYBYTES)) + ensures(return_value == 0 || return_value == MLK_ERR_INVALID_SK || + return_value == MLK_ERR_OUT_OF_MEMORY) +); +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) +/** + * Generate a public/private keypair for the ML-KEM key encapsulation mechanism. + * + * @spec{Implements @[FIPS203, Algorithm 16, ML-KEM.KeyGen_Internal].} + * + * @param[out] pk Output public key (an already allocated array of + * MLKEM_INDCCA_PUBLICKEYBYTES bytes). + * @param[out] sk Output private key (an already allocated array of + * MLKEM_INDCCA_SECRETKEYBYTES bytes). + * @param[in] coins Input randomness (an already allocated array filled + * with 2*MLKEM_SYMBYTES random bytes). + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_PCT_FAIL MLK_CONFIG_KEYGEN_PCT enabled and PCT failed. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + * @retval MLK_ERR_RNG_FAIL MLK_CONFIG_KEYGEN_PCT enabled and random + * number generation failed within the PCT. + */ +MLK_EXTERNAL_API +MLK_MUST_CHECK_RETURN_VALUE +int mlk_kem_keypair_derand(uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES], + uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES], + const uint8_t coins[2 * MLKEM_SYMBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(pk, MLKEM_INDCCA_PUBLICKEYBYTES)) + requires(memory_no_alias(sk, MLKEM_INDCCA_SECRETKEYBYTES)) + requires(memory_no_alias(coins, 2 * MLKEM_SYMBYTES)) + assigns(memory_slice(pk, MLKEM_INDCCA_PUBLICKEYBYTES)) + assigns(memory_slice(sk, MLKEM_INDCCA_SECRETKEYBYTES)) + ensures(return_value == 0 || return_value == MLK_ERR_PCT_FAIL || + return_value == MLK_ERR_OUT_OF_MEMORY || + return_value == MLK_ERR_RNG_FAIL) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(pk, MLKEM_INDCCA_PUBLICKEYBYTES)) + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(sk, MLKEM_INDCCA_SECRETKEYBYTES)) +); + +#if !defined(MLK_CONFIG_NO_RANDOMIZED_API) +/** + * Generate a public/private keypair for the ML-KEM key encapsulation mechanism. + * + * @spec{Implements @[FIPS203, Algorithm 19, ML-KEM.KeyGen].} + * + * @param[out] pk Output public key (an already allocated array of + * MLKEM_INDCCA_PUBLICKEYBYTES bytes). + * @param[out] sk Output private key (an already allocated array of + * MLKEM_INDCCA_SECRETKEYBYTES bytes). + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + * @retval MLK_ERR_RNG_FAIL Random number generation failed. + * @retval MLK_ERR_PCT_FAIL MLK_CONFIG_KEYGEN_PCT enabled and PCT failed. + */ +MLK_EXTERNAL_API +MLK_MUST_CHECK_RETURN_VALUE +int mlk_kem_keypair(uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES], + uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(pk, MLKEM_INDCCA_PUBLICKEYBYTES)) + requires(memory_no_alias(sk, MLKEM_INDCCA_SECRETKEYBYTES)) + assigns(memory_slice(pk, MLKEM_INDCCA_PUBLICKEYBYTES)) + assigns(memory_slice(sk, MLKEM_INDCCA_SECRETKEYBYTES)) + ensures(return_value == 0 || return_value == MLK_ERR_PCT_FAIL || + return_value == MLK_ERR_OUT_OF_MEMORY || + return_value == MLK_ERR_RNG_FAIL) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(pk, MLKEM_INDCCA_PUBLICKEYBYTES)) + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(sk, MLKEM_INDCCA_SECRETKEYBYTES)) +); +#endif /* !MLK_CONFIG_NO_RANDOMIZED_API */ +#endif /* !MLK_CONFIG_NO_KEYPAIR_API */ + +/** + * Generate ciphertext and shared secret for a given public key. + * + * @spec{Implements @[FIPS203, Algorithm 17, ML-KEM.Encaps_Internal].} + * + * @param[out] ct Output ciphertext (an already allocated array of + * MLKEM_INDCCA_CIPHERTEXTBYTES bytes). + * @param[out] ss Output shared secret (an already allocated array of + * MLKEM_SSBYTES bytes). + * @param[in] pk Input public key (an already allocated array of + * MLKEM_INDCCA_PUBLICKEYBYTES bytes). + * @param[in] coins Input randomness (an already allocated array filled + * with MLKEM_SYMBYTES random bytes). + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_INVALID_PK The 'modulus check' @[FIPS203, Section 7.2] + * for the public key failed. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + */ +#if !defined(MLK_CONFIG_NO_ENCAPS_API) +MLK_EXTERNAL_API +MLK_MUST_CHECK_RETURN_VALUE +int mlk_kem_enc_derand(uint8_t ct[MLKEM_INDCCA_CIPHERTEXTBYTES], + uint8_t ss[MLKEM_SSBYTES], + const uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES], + const uint8_t coins[MLKEM_SYMBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(ct, MLKEM_INDCCA_CIPHERTEXTBYTES)) + requires(memory_no_alias(ss, MLKEM_SSBYTES)) + requires(memory_no_alias(pk, MLKEM_INDCCA_PUBLICKEYBYTES)) + requires(memory_no_alias(coins, MLKEM_SYMBYTES)) + assigns(memory_slice(ct, MLKEM_INDCCA_CIPHERTEXTBYTES)) + assigns(memory_slice(ss, MLKEM_SSBYTES)) + ensures(return_value == 0 || return_value == MLK_ERR_INVALID_PK || + return_value == MLK_ERR_OUT_OF_MEMORY) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(ct, MLKEM_INDCCA_CIPHERTEXTBYTES)) + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(ss, MLKEM_SSBYTES)) +); + +#if !defined(MLK_CONFIG_NO_RANDOMIZED_API) +/** + * Generate ciphertext and shared secret for a given public key. + * + * @spec{Implements @[FIPS203, Algorithm 20, ML-KEM.Encaps].} + * + * @param[out] ct Output ciphertext (an already allocated array of + * MLKEM_INDCCA_CIPHERTEXTBYTES bytes). + * @param[out] ss Output shared secret (an already allocated array of + * MLKEM_SSBYTES bytes). + * @param[in] pk Input public key (an already allocated array of + * MLKEM_INDCCA_PUBLICKEYBYTES bytes). + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + * @retval MLK_ERR_RNG_FAIL Random number generation failed. + * @retval MLK_ERR_INVALID_PK The 'modulus check' @[FIPS203, Section 7.2] + * for the public key failed. + */ +MLK_EXTERNAL_API +MLK_MUST_CHECK_RETURN_VALUE +int mlk_kem_enc(uint8_t ct[MLKEM_INDCCA_CIPHERTEXTBYTES], + uint8_t ss[MLKEM_SSBYTES], + const uint8_t pk[MLKEM_INDCCA_PUBLICKEYBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(ct, MLKEM_INDCCA_CIPHERTEXTBYTES)) + requires(memory_no_alias(ss, MLKEM_SSBYTES)) + requires(memory_no_alias(pk, MLKEM_INDCCA_PUBLICKEYBYTES)) + assigns(memory_slice(ct, MLKEM_INDCCA_CIPHERTEXTBYTES)) + assigns(memory_slice(ss, MLKEM_SSBYTES)) + ensures(return_value == 0 || return_value == MLK_ERR_INVALID_PK || + return_value == MLK_ERR_OUT_OF_MEMORY || + return_value == MLK_ERR_RNG_FAIL) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(ct, MLKEM_INDCCA_CIPHERTEXTBYTES)) + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(ss, MLKEM_SSBYTES)) +); +#endif /* !MLK_CONFIG_NO_RANDOMIZED_API */ +#endif /* !MLK_CONFIG_NO_ENCAPS_API */ + +/** + * Generate shared secret for a given ciphertext and private key. + * + * @spec{Implements @[FIPS203, Algorithm 21, ML-KEM.Decaps].} + * + * @param[out] ss Output shared secret (an already allocated array of + * MLKEM_SSBYTES bytes). + * @param[in] ct Input ciphertext (an already allocated array of + * MLKEM_INDCCA_CIPHERTEXTBYTES bytes). + * @param[in] sk Input private key (an already allocated array of + * MLKEM_INDCCA_SECRETKEYBYTES bytes). + * @param context Application context. Only present when + * MLK_CONFIG_CONTEXT_PARAMETER is defined; type set by + * MLK_CONFIG_CONTEXT_PARAMETER_TYPE. + * + * @retval 0 Success. + * @retval MLK_ERR_INVALID_SK The 'hash check' @[FIPS203, Section 7.3] + * for the secret key failed. + * @retval MLK_ERR_OUT_OF_MEMORY MLK_CONFIG_CUSTOM_ALLOC_FREE was used and + * MLK_CUSTOM_ALLOC returned NULL. + */ +#if !defined(MLK_CONFIG_NO_DECAPS_API) +MLK_EXTERNAL_API +MLK_MUST_CHECK_RETURN_VALUE +int mlk_kem_dec(uint8_t ss[MLKEM_SSBYTES], + const uint8_t ct[MLKEM_INDCCA_CIPHERTEXTBYTES], + const uint8_t sk[MLKEM_INDCCA_SECRETKEYBYTES], + MLK_CONFIG_CONTEXT_PARAMETER_TYPE context) +__contract__( + requires(memory_no_alias(ss, MLKEM_SSBYTES)) + requires(memory_no_alias(ct, MLKEM_INDCCA_CIPHERTEXTBYTES)) + requires(memory_no_alias(sk, MLKEM_INDCCA_SECRETKEYBYTES)) + assigns(memory_slice(ss, MLKEM_SSBYTES)) + ensures(return_value == 0 || return_value == MLK_ERR_INVALID_SK || + return_value == MLK_ERR_OUT_OF_MEMORY) + /* Output buffers on error, per API-CONVENTIONS.md */ + ensures(return_value != 0 ==> + array_unchanged_or_zeroized_u8(ss, MLKEM_SSBYTES)) +); +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +#endif /* !MLK_KEM_H */ diff --git a/cbits/mlkem/src/native/aarch64/README.md b/cbits/mlkem/src/native/aarch64/README.md new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/aarch64/README.md @@ -0,0 +1,16 @@ +[//]: # (SPDX-License-Identifier: CC-BY-4.0) + +# AArch64 backend (little endian) + +This directory contains a native backend for little endian AArch64 systems. It is derived from [^NeonNTT] [^SLOTHY_Paper]. + +The code in this directory is auto-generated from the 'clean' assembly in [dev/aarch64_clean](../../../../dev/aarch64_clean) +in a two-step fashion: First, it is superoptimized using the [SLOTHY](https://github.com/slothy-optimizer/slothy) superoptimizer, +giving the assembly in [dev/aarch64_opt](../../../../dev/aarch64_opt). Then, it is stripped of remaining register aliases, macros +and most preprocessor directives by [`scripts/simpasm`](../../../../scripts/simpasm). + +If you want to understand how the assembly works, and/or make changes to it, consult [dev/](../../../../dev). + + +[^NeonNTT]: Becker, Hwang, Kannwischer, Yang, Yang: Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1, [https://eprint.iacr.org/2021/986](https://eprint.iacr.org/2021/986) +[^SLOTHY_Paper]: Abdulrahman, Becker, Kannwischer, Klein: Fast and Clean: Auditable high-performance assembly via constraint solving, [https://eprint.iacr.org/2022/1303](https://eprint.iacr.org/2022/1303) diff --git a/cbits/mlkem/src/native/aarch64/meta.h b/cbits/mlkem/src/native/aarch64/meta.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/aarch64/meta.h @@ -0,0 +1,166 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLK_NATIVE_AARCH64_META_H +#define MLK_NATIVE_AARCH64_META_H + +/* Set of primitives that this backend replaces */ +#define MLK_USE_NATIVE_NTT +#define MLK_USE_NATIVE_INTT +#define MLK_USE_NATIVE_POLY_REDUCE +#define MLK_USE_NATIVE_POLY_TOMONT +#define MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE +#define MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED +#define MLK_USE_NATIVE_POLY_TOBYTES +#define MLK_USE_NATIVE_REJ_UNIFORM + +/* Identifier for this backend so that source and assembly files + * in the build can be appropriately guarded. */ +#define MLK_ARITH_BACKEND_AARCH64 + + +#if !defined(__ASSEMBLER__) +#include "../api.h" +#include "src/arith_native_aarch64.h" + +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_ntt_native(int16_t data[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + mlk_ntt_aarch64_asm(data, mlk_aarch64_ntt_zetas_layer12345, + mlk_aarch64_ntt_zetas_layer67); + return MLK_NATIVE_FUNC_SUCCESS; +} + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_intt_native(int16_t data[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + mlk_intt_aarch64_asm(data, mlk_aarch64_invntt_zetas_layer12345, + mlk_aarch64_invntt_zetas_layer67); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_reduce_native(int16_t data[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + mlk_poly_reduce_aarch64_asm(data); + return MLK_NATIVE_FUNC_SUCCESS; +} + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_tomont_native(int16_t data[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + mlk_poly_tomont_aarch64_asm(data); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* !MLK_CONFIG_NO_KEYPAIR_API */ + +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_mulcache_compute_native(int16_t x[MLKEM_N / 2], + const int16_t y[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + mlk_poly_mulcache_compute_aarch64_asm( + x, y, mlk_aarch64_zetas_mulcache_native, + mlk_aarch64_zetas_mulcache_twisted_native); + return MLK_NATIVE_FUNC_SUCCESS; +} + +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k2_native( + int16_t r[MLKEM_N], const int16_t a[2 * MLKEM_N], + const int16_t b[2 * MLKEM_N], const int16_t b_cache[2 * (MLKEM_N / 2)]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + mlk_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm(r, a, b, b_cache); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 */ + +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 3 +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k3_native( + int16_t r[MLKEM_N], const int16_t a[3 * MLKEM_N], + const int16_t b[3 * MLKEM_N], const int16_t b_cache[3 * (MLKEM_N / 2)]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + mlk_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm(r, a, b, b_cache); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 3 */ + +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4 +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k4_native( + int16_t r[MLKEM_N], const int16_t a[4 * MLKEM_N], + const int16_t b[4 * MLKEM_N], const int16_t b_cache[4 * (MLKEM_N / 2)]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + mlk_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm(r, a, b, b_cache); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4 */ + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) || !defined(MLK_CONFIG_NO_ENCAPS_API) +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_tobytes_native(uint8_t r[MLKEM_POLYBYTES], + const int16_t a[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + mlk_poly_tobytes_aarch64_asm(r, a); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* !MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API */ + +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_rej_uniform_native(int16_t *r, unsigned len, + const uint8_t *buf, + unsigned buflen) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_AARCH64_NEON) || len != MLKEM_N || + buflen % 24 != 0) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + return (int)mlk_rej_uniform_aarch64_asm(r, buf, buflen, + mlk_rej_uniform_table); +} +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLK_NATIVE_AARCH64_META_H */ diff --git a/cbits/mlkem/src/native/aarch64/src/aarch64_zetas.c b/cbits/mlkem/src/native/aarch64/src/aarch64_zetas.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/aarch64/src/aarch64_zetas.c @@ -0,0 +1,184 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mlkem-native repository. + * Do not modify it directly. + */ + +#include "../../../common.h" + +#if defined(MLK_ARITH_BACKEND_AARCH64) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +#include "arith_native_aarch64.h" + +/* + * Table of zeta values used in the AArch64 forward NTT + * See autogen for details. + */ +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const int16_t + mlk_aarch64_ntt_zetas_layer12345[80] = { + -1600, -15749, -749, -7373, -40, -394, -687, -6762, 630, 6201, + -1432, -14095, 848, 8347, 0, 0, 1062, 10453, 296, 2914, + -882, -8682, 0, 0, -1410, -13879, 1339, 13180, 1476, 14529, + 0, 0, 193, 1900, -283, -2786, 56, 551, 0, 0, + 797, 7845, -1089, -10719, 1333, 13121, 0, 0, -543, -5345, + 1426, 14036, -1235, -12156, 0, 0, -69, -679, 535, 5266, + -447, -4400, 0, 0, 569, 5601, -936, -9213, -450, -4429, + 0, 0, -1583, -15582, -1355, -13338, 821, 8081, 0, 0, +}; + +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const int16_t + mlk_aarch64_ntt_zetas_layer67[384] = { + 289, 289, 331, 331, -76, -76, -1573, -1573, 2845, + 2845, 3258, 3258, -748, -748, -15483, -15483, 17, 17, + 583, 583, 1637, 1637, -1041, -1041, 167, 167, 5739, + 5739, 16113, 16113, -10247, -10247, -568, -568, -680, -680, + 723, 723, 1100, 1100, -5591, -5591, -6693, -6693, 7117, + 7117, 10828, 10828, 1197, 1197, -1025, -1025, -1052, -1052, + -1274, -1274, 11782, 11782, -10089, -10089, -10355, -10355, -12540, + -12540, 1409, 1409, -48, -48, 756, 756, -314, -314, + 13869, 13869, -472, -472, 7441, 7441, -3091, -3091, -667, + -667, 233, 233, -1173, -1173, -279, -279, -6565, -6565, + 2293, 2293, -11546, -11546, -2746, -2746, 650, 650, -1352, + -1352, -816, -816, 632, 632, 6398, 6398, -13308, -13308, + -8032, -8032, 6221, 6221, -1626, -1626, -540, -540, -1482, + -1482, 1461, 1461, -16005, -16005, -5315, -5315, -14588, -14588, + 14381, 14381, 1651, 1651, -1540, -1540, 952, 952, -642, + -642, 16251, 16251, -15159, -15159, 9371, 9371, -6319, -6319, + -464, -464, 33, 33, 1320, 1320, -1414, -1414, -4567, + -4567, 325, 325, 12993, 12993, -13918, -13918, 939, 939, + -892, -892, 733, 733, 268, 268, 9243, 9243, -8780, + -8780, 7215, 7215, 2638, 2638, -1021, -1021, -941, -941, + -992, -992, 641, 641, -10050, -10050, -9262, -9262, -9764, + -9764, 6309, 6309, -1010, -1010, 1435, 1435, 807, 807, + 452, 452, -9942, -9942, 14125, 14125, 7943, 7943, 4449, + 4449, 1584, 1584, -1292, -1292, 375, 375, -1239, -1239, + 15592, 15592, -12717, -12717, 3691, 3691, -12196, -12196, -1031, + -1031, -109, -109, -780, -780, 1645, 1645, -10148, -10148, + -1073, -1073, -7678, -7678, 16192, 16192, 1438, 1438, -461, + -461, 1534, 1534, -927, -927, 14155, 14155, -4538, -4538, + 15099, 15099, -9125, -9125, 1063, 1063, -556, -556, -1230, + -1230, -863, -863, 10463, 10463, -5473, -5473, -12107, -12107, + -8495, -8495, 319, 319, 757, 757, 561, 561, -735, + -735, 3140, 3140, 7451, 7451, 5522, 5522, -7235, -7235, + -682, -682, -712, -712, 1481, 1481, 648, 648, -6713, + -6713, -7008, -7008, 14578, 14578, 6378, 6378, -525, -525, + 403, 403, 1143, 1143, -554, -554, -5168, -5168, 3967, + 3967, 11251, 11251, -5453, -5453, 1092, 1092, 1026, 1026, + -1179, -1179, 886, 886, 10749, 10749, 10099, 10099, -11605, + -11605, 8721, 8721, -855, -855, -219, -219, 1227, 1227, + 910, 910, -8416, -8416, -2156, -2156, 12078, 12078, 8957, + 8957, -1607, -1607, -1455, -1455, -1219, -1219, 885, 885, + -15818, -15818, -14322, -14322, -11999, -11999, 8711, 8711, 1212, + 1212, 1029, 1029, -394, -394, -1175, -1175, 11930, 11930, + 10129, 10129, -3878, -3878, -11566, -11566, +}; + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const int16_t + mlk_aarch64_invntt_zetas_layer12345[80] = { + 1583, 15582, -821, -8081, 1355, 13338, 0, 0, -569, + -5601, 450, 4429, 936, 9213, 0, 0, 69, 679, + 447, 4400, -535, -5266, 0, 0, 543, 5345, 1235, + 12156, -1426, -14036, 0, 0, -797, -7845, -1333, -13121, + 1089, 10719, 0, 0, -193, -1900, -56, -551, 283, + 2786, 0, 0, 1410, 13879, -1476, -14529, -1339, -13180, + 0, 0, -1062, -10453, 882, 8682, -296, -2914, 0, + 0, 1600, 15749, 40, 394, 749, 7373, -848, -8347, + 1432, 14095, -630, -6201, 687, 6762, 0, 0, +}; + +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const int16_t + mlk_aarch64_invntt_zetas_layer67[384] = { + -910, -910, -1227, -1227, 219, 219, 855, 855, -8957, + -8957, -12078, -12078, 2156, 2156, 8416, 8416, 1175, 1175, + 394, 394, -1029, -1029, -1212, -1212, 11566, 11566, 3878, + 3878, -10129, -10129, -11930, -11930, -885, -885, 1219, 1219, + 1455, 1455, 1607, 1607, -8711, -8711, 11999, 11999, 14322, + 14322, 15818, 15818, -648, -648, -1481, -1481, 712, 712, + 682, 682, -6378, -6378, -14578, -14578, 7008, 7008, 6713, + 6713, -886, -886, 1179, 1179, -1026, -1026, -1092, -1092, + -8721, -8721, 11605, 11605, -10099, -10099, -10749, -10749, 554, + 554, -1143, -1143, -403, -403, 525, 525, 5453, 5453, + -11251, -11251, -3967, -3967, 5168, 5168, 927, 927, -1534, + -1534, 461, 461, -1438, -1438, 9125, 9125, -15099, -15099, + 4538, 4538, -14155, -14155, 735, 735, -561, -561, -757, + -757, -319, -319, 7235, 7235, -5522, -5522, -7451, -7451, + -3140, -3140, 863, 863, 1230, 1230, 556, 556, -1063, + -1063, 8495, 8495, 12107, 12107, 5473, 5473, -10463, -10463, + -452, -452, -807, -807, -1435, -1435, 1010, 1010, -4449, + -4449, -7943, -7943, -14125, -14125, 9942, 9942, -1645, -1645, + 780, 780, 109, 109, 1031, 1031, -16192, -16192, 7678, + 7678, 1073, 1073, 10148, 10148, 1239, 1239, -375, -375, + 1292, 1292, -1584, -1584, 12196, 12196, -3691, -3691, 12717, + 12717, -15592, -15592, 1414, 1414, -1320, -1320, -33, -33, + 464, 464, 13918, 13918, -12993, -12993, -325, -325, 4567, + 4567, -641, -641, 992, 992, 941, 941, 1021, 1021, + -6309, -6309, 9764, 9764, 9262, 9262, 10050, 10050, -268, + -268, -733, -733, 892, 892, -939, -939, -2638, -2638, + -7215, -7215, 8780, 8780, -9243, -9243, -632, -632, 816, + 816, 1352, 1352, -650, -650, -6221, -6221, 8032, 8032, + 13308, 13308, -6398, -6398, 642, 642, -952, -952, 1540, + 1540, -1651, -1651, 6319, 6319, -9371, -9371, 15159, 15159, + -16251, -16251, -1461, -1461, 1482, 1482, 540, 540, 1626, + 1626, -14381, -14381, 14588, 14588, 5315, 5315, 16005, 16005, + 1274, 1274, 1052, 1052, 1025, 1025, -1197, -1197, 12540, + 12540, 10355, 10355, 10089, 10089, -11782, -11782, 279, 279, + 1173, 1173, -233, -233, 667, 667, 2746, 2746, 11546, + 11546, -2293, -2293, 6565, 6565, 314, 314, -756, -756, + 48, 48, -1409, -1409, 3091, 3091, -7441, -7441, 472, + 472, -13869, -13869, 1573, 1573, 76, 76, -331, -331, + -289, -289, 15483, 15483, 748, 748, -3258, -3258, -2845, + -2845, -1100, -1100, -723, -723, 680, 680, 568, 568, + -10828, -10828, -7117, -7117, 6693, 6693, 5591, 5591, 1041, + 1041, -1637, -1637, -583, -583, -17, -17, 10247, 10247, + -16113, -16113, -5739, -5739, -167, -167, +}; +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const int16_t + mlk_aarch64_zetas_mulcache_native[128] = { + 17, -17, -568, 568, 583, -583, -680, 680, 1637, -1637, + 723, -723, -1041, 1041, 1100, -1100, 1409, -1409, -667, 667, + -48, 48, 233, -233, 756, -756, -1173, 1173, -314, 314, + -279, 279, -1626, 1626, 1651, -1651, -540, 540, -1540, 1540, + -1482, 1482, 952, -952, 1461, -1461, -642, 642, 939, -939, + -1021, 1021, -892, 892, -941, 941, 733, -733, -992, 992, + 268, -268, 641, -641, 1584, -1584, -1031, 1031, -1292, 1292, + -109, 109, 375, -375, -780, 780, -1239, 1239, 1645, -1645, + 1063, -1063, 319, -319, -556, 556, 757, -757, -1230, 1230, + 561, -561, -863, 863, -735, 735, -525, 525, 1092, -1092, + 403, -403, 1026, -1026, 1143, -1143, -1179, 1179, -554, 554, + 886, -886, -1607, 1607, 1212, -1212, -1455, 1455, 1029, -1029, + -1219, 1219, -394, 394, 885, -885, -1175, 1175, +}; + +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const int16_t + mlk_aarch64_zetas_mulcache_twisted_native[128] = { + 167, -167, -5591, 5591, 5739, -5739, -6693, 6693, 16113, + -16113, 7117, -7117, -10247, 10247, 10828, -10828, 13869, -13869, + -6565, 6565, -472, 472, 2293, -2293, 7441, -7441, -11546, + 11546, -3091, 3091, -2746, 2746, -16005, 16005, 16251, -16251, + -5315, 5315, -15159, 15159, -14588, 14588, 9371, -9371, 14381, + -14381, -6319, 6319, 9243, -9243, -10050, 10050, -8780, 8780, + -9262, 9262, 7215, -7215, -9764, 9764, 2638, -2638, 6309, + -6309, 15592, -15592, -10148, 10148, -12717, 12717, -1073, 1073, + 3691, -3691, -7678, 7678, -12196, 12196, 16192, -16192, 10463, + -10463, 3140, -3140, -5473, 5473, 7451, -7451, -12107, 12107, + 5522, -5522, -8495, 8495, -7235, 7235, -5168, 5168, 10749, + -10749, 3967, -3967, 10099, -10099, 11251, -11251, -11605, 11605, + -5453, 5453, 8721, -8721, -15818, 15818, 11930, -11930, -14322, + 14322, 10129, -10129, -11999, 11999, -3878, 3878, 8711, -8711, + -11566, 11566, +}; + +#else /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +MLK_EMPTY_CU(aarch64_zetas) + +#endif /* !(MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mlkem/src/native/aarch64/src/arith_native_aarch64.h b/cbits/mlkem/src/native/aarch64/src/arith_native_aarch64.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/aarch64/src/arith_native_aarch64.h @@ -0,0 +1,184 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLK_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H +#define MLK_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H + +#include "../../../cbmc.h" +#include "../../../common.h" + +#define mlk_aarch64_ntt_zetas_layer12345 \ + MLK_NAMESPACE(aarch64_ntt_zetas_layer12345) +#define mlk_aarch64_ntt_zetas_layer67 MLK_NAMESPACE(aarch64_ntt_zetas_layer67) +#define mlk_aarch64_invntt_zetas_layer12345 \ + MLK_NAMESPACE(aarch64_invntt_zetas_layer12345) +#define mlk_aarch64_invntt_zetas_layer67 \ + MLK_NAMESPACE(aarch64_invntt_zetas_layer67) +#define mlk_aarch64_zetas_mulcache_native \ + MLK_NAMESPACE(aarch64_zetas_mulcache_native) +#define mlk_aarch64_zetas_mulcache_twisted_native \ + MLK_NAMESPACE(aarch64_zetas_mulcache_twisted_native) +#define mlk_rej_uniform_table MLK_NAMESPACE(rej_uniform_table) + +MLK_INTERNAL_DATA_DECLARATION const int16_t + mlk_aarch64_ntt_zetas_layer12345[80]; +MLK_INTERNAL_DATA_DECLARATION const int16_t mlk_aarch64_ntt_zetas_layer67[384]; +MLK_INTERNAL_DATA_DECLARATION const int16_t + mlk_aarch64_invntt_zetas_layer12345[80]; +MLK_INTERNAL_DATA_DECLARATION const int16_t + mlk_aarch64_invntt_zetas_layer67[384]; +MLK_INTERNAL_DATA_DECLARATION const int16_t + mlk_aarch64_zetas_mulcache_native[128]; +MLK_INTERNAL_DATA_DECLARATION const int16_t + mlk_aarch64_zetas_mulcache_twisted_native[128]; +MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_rej_uniform_table[4096]; + +#define mlk_ntt_aarch64_asm MLK_NAMESPACE(ntt_aarch64_asm) +void mlk_ntt_aarch64_asm(int16_t p[256], const int16_t twiddles12345[80], + const int16_t twiddles56[384]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mlkem_ntt_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N)) + requires(array_abs_bound(p, 0, MLKEM_N, 8192)) + requires(twiddles12345 == mlk_aarch64_ntt_zetas_layer12345) + requires(twiddles56 == mlk_aarch64_ntt_zetas_layer67) + assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N)) + /* check-magic: off */ + ensures(array_abs_bound(p, 0, MLKEM_N, 23595)) + /* check-magic: on */ +); + +#define mlk_intt_aarch64_asm MLK_NAMESPACE(intt_aarch64_asm) +void mlk_intt_aarch64_asm(int16_t p[256], const int16_t twiddles12345[80], + const int16_t twiddles56[384]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mlkem_intt_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N)) + requires(twiddles12345 == mlk_aarch64_invntt_zetas_layer12345) + requires(twiddles56 == mlk_aarch64_invntt_zetas_layer67) + assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N)) + /* check-magic: off */ + ensures(array_abs_bound(p, 0, MLKEM_N, 26625)) + /* check-magic: on */ +); + +#define mlk_poly_reduce_aarch64_asm MLK_NAMESPACE(poly_reduce_aarch64_asm) +void mlk_poly_reduce_aarch64_asm(int16_t p[256]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mlkem_poly_reduce_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N)) + assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N)) + ensures(array_bound(p, 0, MLKEM_N, 0, MLKEM_Q)) +); + +#define mlk_poly_tomont_aarch64_asm MLK_NAMESPACE(poly_tomont_aarch64_asm) +void mlk_poly_tomont_aarch64_asm(int16_t p[256]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mlkem_poly_tomont_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N)) + assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N)) + ensures(array_abs_bound(p, 0, MLKEM_N, MLKEM_Q)) +); + +#define mlk_poly_mulcache_compute_aarch64_asm \ + MLK_NAMESPACE(poly_mulcache_compute_aarch64_asm) +void mlk_poly_mulcache_compute_aarch64_asm(int16_t cache[128], + const int16_t mlk_poly[256], + const int16_t zetas[128], + const int16_t zetas_twisted[128]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mlkem_poly_mulcache_compute_aarch64_asm.ml + */ +__contract__( + requires(memory_no_alias(cache, sizeof(int16_t) * (MLKEM_N / 2))) + requires(memory_no_alias(mlk_poly, sizeof(int16_t) * MLKEM_N)) + requires(zetas == mlk_aarch64_zetas_mulcache_native) + requires(zetas_twisted == mlk_aarch64_zetas_mulcache_twisted_native) + assigns(memory_slice(cache, sizeof(int16_t) * (MLKEM_N / 2))) + ensures(array_abs_bound(cache, 0, MLKEM_N/2, MLKEM_Q)) +); + +#define mlk_poly_tobytes_aarch64_asm MLK_NAMESPACE(poly_tobytes_aarch64_asm) +void mlk_poly_tobytes_aarch64_asm(uint8_t r[384], const int16_t a[256]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mlkem_poly_tobytes_aarch64_asm.ml */ +__contract__( + requires(memory_no_alias(r, MLKEM_POLYBYTES)) + requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N)) + requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT)) + assigns(memory_slice(r, MLKEM_POLYBYTES)) +); + +#define mlk_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm \ + MLK_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm) +void mlk_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm( + int16_t r[256], const int16_t a[512], const int16_t b[512], + const int16_t b_cache[256]) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/aarch64/proofs/mlkem_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, sizeof(int16_t) * 2 * MLKEM_N)) + requires(memory_no_alias(b, sizeof(int16_t) * 2 * MLKEM_N)) + requires(memory_no_alias(b_cache, sizeof(int16_t) * 2 * (MLKEM_N / 2))) + requires(array_abs_bound(a, 0, 2 * MLKEM_N, MLKEM_UINT12_LIMIT + 1)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) +); + +#define mlk_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm \ + MLK_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm) +void mlk_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm( + int16_t r[256], const int16_t a[768], const int16_t b[768], + const int16_t b_cache[384]) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/aarch64/proofs/mlkem_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, sizeof(int16_t) * 3 * MLKEM_N)) + requires(memory_no_alias(b, sizeof(int16_t) * 3 * MLKEM_N)) + requires(memory_no_alias(b_cache, sizeof(int16_t) * 3 * (MLKEM_N / 2))) + requires(array_abs_bound(a, 0, 3 * MLKEM_N, MLKEM_UINT12_LIMIT + 1)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) +); + +#define mlk_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm \ + MLK_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm) +void mlk_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm( + int16_t r[256], const int16_t a[1024], const int16_t b[1024], + const int16_t b_cache[512]) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/aarch64/proofs/mlkem_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, sizeof(int16_t) * 4 * MLKEM_N)) + requires(memory_no_alias(b, sizeof(int16_t) * 4 * MLKEM_N)) + requires(memory_no_alias(b_cache, sizeof(int16_t) * 4 * (MLKEM_N / 2))) + requires(array_abs_bound(a, 0, 4 * MLKEM_N, MLKEM_UINT12_LIMIT + 1)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) +); + +#define mlk_rej_uniform_aarch64_asm MLK_NAMESPACE(rej_uniform_aarch64_asm) +MLK_MUST_CHECK_RETURN_VALUE +uint64_t mlk_rej_uniform_aarch64_asm(int16_t r[256], const uint8_t *buf, + unsigned buflen, const uint8_t table[4096]) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/aarch64/proofs/mlkem_rej_uniform_aarch64_asm.ml. */ +__contract__( + requires(buflen % 24 == 0) + requires(memory_no_alias(buf, buflen)) + requires(table == mlk_rej_uniform_table) + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(return_value <= MLKEM_N) + ensures(array_bound(r, 0, (unsigned) return_value, 0, MLKEM_Q)) +); + +#endif /* !MLK_NATIVE_AARCH64_SRC_ARITH_NATIVE_AARCH64_H */ diff --git a/cbits/mlkem/src/native/aarch64/src/mlkem_intt_aarch64_asm.S b/cbits/mlkem/src/native/aarch64/src/mlkem_intt_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/aarch64/src/mlkem_intt_aarch64_asm.S @@ -0,0 +1,635 @@ +/* Copyright (c) 2022 Arm Limited + * Copyright (c) 2022 Hanno Becker + * Copyright (c) 2023 Amin Abdulrahman, Matthias Kannwischer + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [NeonNTT] + * Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1 + * Becker, Hwang, Kannwischer, Yang, Yang + * https://eprint.iacr.org/2021/986 + * + * - [SLOTHY_Paper] + * Fast and Clean: Auditable high-performance assembly via constraint solving + * Abdulrahman, Becker, Kannwischer, Klein + * https://eprint.iacr.org/2022/1303 + */ + +/*yaml + Name: intt_aarch64_asm + Description: AArch64 ML-KEM inverse NTT following @[NeonNTT] and @[SLOTHY_Paper] + Signature: void mlk_intt_aarch64_asm(int16_t p[256], const int16_t twiddles12345[80], const int16_t twiddles56[384]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 512 + permissions: read/write + c_parameter: int16_t p[256] + description: Input/output polynomial + x1: + type: buffer + size_bytes: 160 + permissions: read-only + c_parameter: const int16_t twiddles12345[80] + description: Twiddle factors for layers 1-5 + x2: + type: buffer + size_bytes: 768 + permissions: read-only + c_parameter: const int16_t twiddles56[384] + description: Twiddle factors for layers 6-7 + Stack: + bytes: 64 + description: saving callee-saved Neon registers +*/ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_AARCH64) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + (!defined(MLK_CONFIG_NO_ENCAPS_API) || \ + !defined(MLK_CONFIG_NO_DECAPS_API)) + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/aarch64_opt/src/mlkem_intt_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(intt_aarch64_asm) +MLK_ASM_FN_SYMBOL(intt_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x40 + .cfi_adjust_cfa_offset 0x40 + stp d8, d9, [sp] + .cfi_rel_offset d8, 0x0 + .cfi_rel_offset d9, 0x8 + stp d10, d11, [sp, #0x10] + .cfi_rel_offset d10, 0x10 + .cfi_rel_offset d11, 0x18 + stp d12, d13, [sp, #0x20] + .cfi_rel_offset d12, 0x20 + .cfi_rel_offset d13, 0x28 + stp d14, d15, [sp, #0x30] + .cfi_rel_offset d14, 0x30 + .cfi_rel_offset d15, 0x38 + mov w5, #0xd01 // =3329 + mov v7.h[0], w5 + mov w5, #0x4ebf // =20159 + mov v7.h[1], w5 + mov w5, #0x200 // =512 + dup v29.8h, w5 + mov w5, #0x13b0 // =5040 + dup v30.8h, w5 + mov x3, x0 + mov x4, #0x8 // =8 + ldr q13, [x3, #0x20] + ldr q8, [x3, #0x30] + ldr q6, [x3] + ldr q16, [x3, #0x10] + ldr q4, [x3, #0x50] + ldr q11, [x3, #0x40] + ldr q3, [x3, #0x70] + trn1 v23.4s, v13.4s, v8.4s + ldr q0, [x3, #0x60] + trn2 v19.4s, v6.4s, v16.4s + trn2 v21.4s, v13.4s, v8.4s + trn1 v6.4s, v6.4s, v16.4s + ldr q24, [x2, #0x20] + trn1 v10.2d, v19.2d, v21.2d + ldr q16, [x2], #0x60 + trn1 v5.2d, v6.2d, v23.2d + trn1 v28.4s, v0.4s, v3.4s + trn2 v18.2d, v6.2d, v23.2d + mul v31.8h, v10.8h, v29.8h + trn2 v13.4s, v0.4s, v3.4s + ldur q14, [x2, #-0x50] + sqrdmulh v26.8h, v18.8h, v30.8h + ldur q20, [x2, #-0x20] + mul v17.8h, v18.8h, v29.8h + trn2 v18.2d, v19.2d, v21.2d + mul v9.8h, v18.8h, v29.8h + trn1 v12.4s, v11.4s, v4.4s + sqrdmulh v22.8h, v18.8h, v30.8h + sqrdmulh v3.8h, v10.8h, v30.8h + sqrdmulh v25.8h, v5.8h, v30.8h + mls v9.8h, v22.8h, v7.h[0] + mls v17.8h, v26.8h, v7.h[0] + trn2 v26.4s, v11.4s, v4.4s + mul v8.8h, v5.8h, v29.8h + trn1 v10.2d, v26.2d, v13.2d + ldur q11, [x2, #-0x10] + mls v31.8h, v3.8h, v7.h[0] + trn1 v6.2d, v12.2d, v28.2d + trn2 v3.2d, v26.2d, v13.2d + ldur q4, [x2, #-0x30] + mls v8.8h, v25.8h, v7.h[0] + sub v19.8h, v17.8h, v9.8h + trn2 v13.2d, v12.2d, v28.2d + sqrdmulh v1.8h, v3.8h, v30.8h + add v9.8h, v17.8h, v9.8h + mul v18.8h, v19.8h, v20.8h + add v28.8h, v8.8h, v31.8h + sqrdmulh v20.8h, v19.8h, v11.8h + sub v12.8h, v28.8h, v9.8h + sub v23.8h, v8.8h, v31.8h + sqrdmulh v11.8h, v13.8h, v30.8h + sqrdmulh v5.8h, v23.8h, v4.8h + mul v0.8h, v23.8h, v24.8h + mul v2.8h, v13.8h, v29.8h + mls v0.8h, v5.8h, v7.h[0] + add v24.8h, v28.8h, v9.8h + mls v18.8h, v20.8h, v7.h[0] + sqrdmulh v15.8h, v6.8h, v30.8h + sqrdmulh v25.8h, v12.8h, v14.8h + mul v21.8h, v12.8h, v16.8h + sub v23.8h, v0.8h, v18.8h + sqrdmulh v8.8h, v23.8h, v14.8h + mul v23.8h, v23.8h, v16.8h + mls v21.8h, v25.8h, v7.h[0] + mls v23.8h, v8.8h, v7.h[0] + mul v14.8h, v3.8h, v29.8h + add v3.8h, v0.8h, v18.8h + trn2 v4.4s, v24.4s, v3.4s + mls v14.8h, v1.8h, v7.h[0] + trn1 v9.4s, v24.4s, v3.4s + trn2 v12.4s, v21.4s, v23.4s + mls v2.8h, v11.8h, v7.h[0] + trn1 v28.4s, v21.4s, v23.4s + ldr q11, [x1], #0x10 + mul v31.8h, v10.8h, v29.8h + trn1 v25.2d, v4.2d, v12.2d + trn1 v20.2d, v9.2d, v28.2d + ldr q23, [x2, #0x50] + trn2 v13.2d, v4.2d, v12.2d + sqrdmulh v21.8h, v10.8h, v30.8h + trn2 v4.2d, v9.2d, v28.2d + ldr q9, [x2, #0x40] + mul v27.8h, v6.8h, v29.8h + add v26.8h, v20.8h, v25.8h + sub v3.8h, v2.8h, v14.8h + sqdmulh v12.8h, v26.8h, v7.h[1] + add v5.8h, v4.8h, v13.8h + sub v8.8h, v4.8h, v13.8h + add v10.8h, v2.8h, v14.8h + sqdmulh v6.8h, v5.8h, v7.h[1] + ldr q2, [x2, #0x10] + mls v27.8h, v15.8h, v7.h[0] + ldr q15, [x2, #0x20] + srshr v12.8h, v12.8h, #0xb + mls v31.8h, v21.8h, v7.h[0] + srshr v6.8h, v6.8h, #0xb + sqrdmulh v23.8h, v3.8h, v23.8h + mls v26.8h, v12.8h, v7.h[0] + add v21.8h, v27.8h, v31.8h + mls v5.8h, v6.8h, v7.h[0] + sub v6.8h, v27.8h, v31.8h + sub v14.8h, v21.8h, v10.8h + ldr q27, [x2], #0x60 + mul v3.8h, v3.8h, v9.8h + mls v3.8h, v23.8h, v7.h[0] + ldur q13, [x2, #-0x30] + sub v12.8h, v26.8h, v5.8h + add v5.8h, v26.8h, v5.8h + sqrdmulh v31.8h, v8.8h, v11.h[5] + sqrdmulh v19.8h, v12.8h, v11.h[1] + mul v24.8h, v12.8h, v11.h[0] + sqrdmulh v13.8h, v6.8h, v13.8h + mls v24.8h, v19.8h, v7.h[0] + sub x4, x4, #0x2 + +Lmlk_intt_layer4567_start: + add v16.8h, v21.8h, v10.8h + mul v18.8h, v6.8h, v15.8h + sub v19.8h, v20.8h, v25.8h + ldr q21, [x3, #0xa0] + str q5, [x3], #0x40 + mls v18.8h, v13.8h, v7.h[0] + sqrdmulh v15.8h, v14.8h, v2.8h + ldr q10, [x3, #0x50] + ldr q12, [x3, #0x40] + stur q24, [x3, #-0x20] + mul v5.8h, v8.8h, v11.h[4] + sub v0.8h, v18.8h, v3.8h + ldr q24, [x3, #0x70] + mls v5.8h, v31.8h, v7.h[0] + ldr q26, [x2, #0x50] + trn2 v1.4s, v12.4s, v10.4s + add v6.8h, v18.8h, v3.8h + sqrdmulh v20.8h, v0.8h, v2.8h + trn1 v13.4s, v12.4s, v10.4s + trn1 v18.4s, v16.4s, v6.4s + mul v22.8h, v0.8h, v27.8h + trn1 v17.4s, v21.4s, v24.4s + sqrdmulh v0.8h, v19.8h, v11.h[3] + trn1 v25.2d, v13.2d, v17.2d + mls v22.8h, v20.8h, v7.h[0] + trn2 v21.4s, v21.4s, v24.4s + mul v24.8h, v25.8h, v29.8h + trn2 v28.2d, v13.2d, v17.2d + sqrdmulh v4.8h, v25.8h, v30.8h + trn2 v3.2d, v1.2d, v21.2d + mul v17.8h, v28.8h, v29.8h + sqrdmulh v31.8h, v28.8h, v30.8h + ldr q2, [x2, #0x10] + mls v24.8h, v4.8h, v7.h[0] + mul v4.8h, v19.8h, v11.h[2] + ldr q19, [x2, #0x40] + mls v4.8h, v0.8h, v7.h[0] + mul v0.8h, v14.8h, v27.8h + mls v0.8h, v15.8h, v7.h[0] + sub v8.8h, v4.8h, v5.8h + mul v12.8h, v3.8h, v29.8h + mul v23.8h, v8.8h, v11.h[0] + trn2 v28.4s, v16.4s, v6.4s + sqrdmulh v10.8h, v8.8h, v11.h[1] + trn1 v9.4s, v0.4s, v22.4s + trn2 v22.4s, v0.4s, v22.4s + ldr q11, [x1], #0x10 + mls v17.8h, v31.8h, v7.h[0] + trn1 v20.2d, v18.2d, v9.2d + trn2 v14.2d, v18.2d, v9.2d + ldr q15, [x2, #0x20] + trn1 v6.2d, v1.2d, v21.2d + sqrdmulh v9.8h, v3.8h, v30.8h + trn1 v25.2d, v28.2d, v22.2d + trn2 v16.2d, v28.2d, v22.2d + mls v23.8h, v10.8h, v7.h[0] + add v1.8h, v20.8h, v25.8h + sqrdmulh v21.8h, v6.8h, v30.8h + add v8.8h, v14.8h, v16.8h + ldr q27, [x2], #0x60 + sqdmulh v28.8h, v8.8h, v7.h[1] + mls v12.8h, v9.8h, v7.h[0] + sqdmulh v31.8h, v1.8h, v7.h[1] + mul v0.8h, v6.8h, v29.8h + sub v10.8h, v17.8h, v12.8h + mls v0.8h, v21.8h, v7.h[0] + srshr v21.8h, v28.8h, #0xb + srshr v13.8h, v31.8h, #0xb + sqrdmulh v22.8h, v10.8h, v26.8h + mls v8.8h, v21.8h, v7.h[0] + mls v1.8h, v13.8h, v7.h[0] + add v21.8h, v24.8h, v0.8h + stur q23, [x3, #-0x10] + sub v6.8h, v24.8h, v0.8h + mul v3.8h, v10.8h, v19.8h + add v0.8h, v4.8h, v5.8h + sqdmulh v13.8h, v0.8h, v7.h[1] + ldur q10, [x2, #-0x30] + add v5.8h, v1.8h, v8.8h + mls v3.8h, v22.8h, v7.h[0] + sub v8.8h, v1.8h, v8.8h + mul v24.8h, v8.8h, v11.h[0] + sqrdmulh v8.8h, v8.8h, v11.h[1] + srshr v1.8h, v13.8h, #0xb + sqrdmulh v13.8h, v6.8h, v10.8h + mls v0.8h, v1.8h, v7.h[0] + add v10.8h, v17.8h, v12.8h + mls v24.8h, v8.8h, v7.h[0] + sub v8.8h, v14.8h, v16.8h + sqrdmulh v31.8h, v8.8h, v11.h[5] + sub v14.8h, v21.8h, v10.8h + stur q0, [x3, #-0x30] + sub x4, x4, #0x1 + cbnz x4, Lmlk_intt_layer4567_start + mul v15.8h, v6.8h, v15.8h + sub v22.8h, v20.8h, v25.8h + add v4.8h, v21.8h, v10.8h + str q24, [x3, #0x20] + mls v15.8h, v13.8h, v7.h[0] + str q5, [x3], #0x40 + ldr q9, [x1], #0x10 + sqrdmulh v28.8h, v14.8h, v2.8h + mul v16.8h, v14.8h, v27.8h + sub v18.8h, v15.8h, v3.8h + add v15.8h, v15.8h, v3.8h + sqrdmulh v0.8h, v18.8h, v2.8h + trn2 v24.4s, v4.4s, v15.4s + trn1 v2.4s, v4.4s, v15.4s + mul v18.8h, v18.8h, v27.8h + mls v16.8h, v28.8h, v7.h[0] + mls v18.8h, v0.8h, v7.h[0] + mul v23.8h, v8.8h, v11.h[4] + sqrdmulh v12.8h, v22.8h, v11.h[3] + trn1 v17.4s, v16.4s, v18.4s + trn2 v4.4s, v16.4s, v18.4s + mls v23.8h, v31.8h, v7.h[0] + trn2 v3.2d, v2.2d, v17.2d + trn2 v6.2d, v24.2d, v4.2d + mul v26.8h, v22.8h, v11.h[2] + trn1 v28.2d, v2.2d, v17.2d + mls v26.8h, v12.8h, v7.h[0] + add v25.8h, v3.8h, v6.8h + sub v18.8h, v3.8h, v6.8h + trn1 v24.2d, v24.2d, v4.2d + sqdmulh v1.8h, v25.8h, v7.h[1] + sub v27.8h, v28.8h, v24.8h + sqrdmulh v2.8h, v18.8h, v9.h[5] + add v28.8h, v28.8h, v24.8h + mul v24.8h, v27.8h, v9.h[2] + sqdmulh v12.8h, v28.8h, v7.h[1] + mul v20.8h, v18.8h, v9.h[4] + mls v20.8h, v2.8h, v7.h[0] + srshr v1.8h, v1.8h, #0xb + sqrdmulh v19.8h, v27.8h, v9.h[3] + srshr v15.8h, v12.8h, #0xb + mls v25.8h, v1.8h, v7.h[0] + add v8.8h, v26.8h, v23.8h + sub v4.8h, v26.8h, v23.8h + mls v28.8h, v15.8h, v7.h[0] + mls v24.8h, v19.8h, v7.h[0] + mul v2.8h, v4.8h, v11.h[0] + sub v19.8h, v28.8h, v25.8h + sqrdmulh v15.8h, v4.8h, v11.h[1] + add v25.8h, v28.8h, v25.8h + sub v10.8h, v24.8h, v20.8h + str q25, [x3], #0x40 + sqrdmulh v22.8h, v19.8h, v9.h[1] + add v28.8h, v24.8h, v20.8h + sqrdmulh v25.8h, v10.8h, v9.h[1] + mul v27.8h, v19.8h, v9.h[0] + mul v26.8h, v10.8h, v9.h[0] + sqdmulh v20.8h, v28.8h, v7.h[1] + sqdmulh v16.8h, v8.8h, v7.h[1] + mls v26.8h, v25.8h, v7.h[0] + mls v2.8h, v15.8h, v7.h[0] + srshr v15.8h, v20.8h, #0xb + srshr v1.8h, v16.8h, #0xb + mls v27.8h, v22.8h, v7.h[0] + mls v28.8h, v15.8h, v7.h[0] + mls v8.8h, v1.8h, v7.h[0] + stur q27, [x3, #-0x20] + stur q2, [x3, #-0x50] + stur q28, [x3, #-0x30] + stur q26, [x3, #-0x10] + stur q8, [x3, #-0x70] + mov x4, #0x4 // =4 + ldr q0, [x1], #0x20 + ldur q1, [x1, #-0x10] + ldr q26, [x0] + ldr q13, [x0, #0x40] + ldr q28, [x0, #0xc0] + ldr q2, [x0, #0x140] + ldr q6, [x0, #0x80] + ldr q9, [x0, #0x100] + ldr q29, [x0, #0x1c0] + ldr q23, [x0, #0x180] + sub v17.8h, v26.8h, v13.8h + add v4.8h, v26.8h, v13.8h + ldr q25, [x0, #0xd0] + ldr q24, [x0, #0x50] + add v5.8h, v6.8h, v28.8h + mul v19.8h, v17.8h, v0.h[6] + sub v10.8h, v6.8h, v28.8h + ldr q30, [x0, #0x150] + sqrdmulh v12.8h, v17.8h, v0.h[7] + add v17.8h, v9.8h, v2.8h + sub v28.8h, v9.8h, v2.8h + ldr q2, [x0, #0x90] + sub v26.8h, v23.8h, v29.8h + sqrdmulh v31.8h, v10.8h, v1.h[1] + add v22.8h, v23.8h, v29.8h + ldr q3, [x0, #0x110] + sqrdmulh v9.8h, v28.8h, v1.h[3] + sub v20.8h, v4.8h, v5.8h + sub v27.8h, v17.8h, v22.8h + ldr q29, [x0, #0x10] + add v16.8h, v4.8h, v5.8h + sqrdmulh v4.8h, v26.8h, v1.h[5] + add v6.8h, v17.8h, v22.8h + ldr q22, [x0, #0x1d0] + mul v8.8h, v28.8h, v1.h[2] + sub v21.8h, v2.8h, v25.8h + sub v5.8h, v16.8h, v6.8h + mul v17.8h, v26.8h, v1.h[4] + mul v26.8h, v10.8h, v1.h[0] + mls v26.8h, v31.8h, v7.h[0] + mls v17.8h, v4.8h, v7.h[0] + mls v19.8h, v12.8h, v7.h[0] + mls v8.8h, v9.8h, v7.h[0] + sqrdmulh v10.8h, v27.8h, v0.h[5] + sub v12.8h, v19.8h, v26.8h + add v9.8h, v19.8h, v26.8h + sqrdmulh v26.8h, v20.8h, v0.h[3] + sub v11.8h, v8.8h, v17.8h + add v14.8h, v8.8h, v17.8h + sqrdmulh v13.8h, v12.8h, v0.h[3] + add v23.8h, v9.8h, v14.8h + sqrdmulh v28.8h, v11.8h, v0.h[5] + sub v19.8h, v9.8h, v14.8h + mul v17.8h, v27.8h, v0.h[4] + str q23, [x0, #0x40] + mul v14.8h, v20.8h, v0.h[2] + mul v8.8h, v11.8h, v0.h[4] + mul v4.8h, v12.8h, v0.h[2] + mls v14.8h, v26.8h, v7.h[0] + mls v17.8h, v10.8h, v7.h[0] + mls v8.8h, v28.8h, v7.h[0] + mls v4.8h, v13.8h, v7.h[0] + sub v10.8h, v14.8h, v17.8h + add v20.8h, v14.8h, v17.8h + sqrdmulh v28.8h, v5.8h, v0.h[1] + mul v18.8h, v5.8h, v0.h[0] + str q20, [x0, #0x80] + sub v13.8h, v4.8h, v8.8h + mul v23.8h, v10.8h, v0.h[0] + mul v17.8h, v19.8h, v0.h[0] + sqrdmulh v9.8h, v13.8h, v0.h[1] + mls v18.8h, v28.8h, v7.h[0] + sqrdmulh v10.8h, v10.8h, v0.h[1] + sub x4, x4, #0x2 + +Lmlk_intt_layer123_start: + sub v12.8h, v3.8h, v30.8h + mul v11.8h, v21.8h, v1.h[0] + add v28.8h, v4.8h, v8.8h + ldr q20, [x0, #0x190] + add v27.8h, v16.8h, v6.8h + sqrdmulh v8.8h, v12.8h, v1.h[3] + add v16.8h, v29.8h, v24.8h + str q28, [x0, #0xc0] + mls v23.8h, v10.8h, v7.h[0] + str q27, [x0], #0x10 + add v15.8h, v20.8h, v22.8h + str q18, [x0, #0xf0] + mul v14.8h, v13.8h, v0.h[0] + add v2.8h, v2.8h, v25.8h + sub v26.8h, v20.8h, v22.8h + mul v4.8h, v12.8h, v1.h[2] + sub v5.8h, v16.8h, v2.8h + str q23, [x0, #0x170] + add v20.8h, v3.8h, v30.8h + sqrdmulh v27.8h, v26.8h, v1.h[5] + add v16.8h, v16.8h, v2.8h + mul v18.8h, v26.8h, v1.h[4] + sub v31.8h, v20.8h, v15.8h + mls v4.8h, v8.8h, v7.h[0] + sub v28.8h, v29.8h, v24.8h + mls v18.8h, v27.8h, v7.h[0] + ldr q22, [x0, #0x1d0] + mul v26.8h, v28.8h, v0.h[6] + mul v2.8h, v5.8h, v0.h[2] + sub v12.8h, v4.8h, v18.8h + sqrdmulh v24.8h, v28.8h, v0.h[7] + mls v14.8h, v9.8h, v7.h[0] + sqrdmulh v10.8h, v12.8h, v0.h[5] + mls v26.8h, v24.8h, v7.h[0] + ldr q24, [x0, #0x50] + mul v8.8h, v12.8h, v0.h[4] + str q14, [x0, #0x1b0] + add v28.8h, v4.8h, v18.8h + sqrdmulh v5.8h, v5.8h, v0.h[3] + add v6.8h, v20.8h, v15.8h + sqrdmulh v3.8h, v19.8h, v0.h[1] + sub v13.8h, v16.8h, v6.8h + sqrdmulh v12.8h, v21.8h, v1.h[1] + sqrdmulh v21.8h, v13.8h, v0.h[1] + sqrdmulh v27.8h, v31.8h, v0.h[5] + ldr q25, [x0, #0xd0] + mls v11.8h, v12.8h, v7.h[0] + mul v23.8h, v31.8h, v0.h[4] + mul v18.8h, v13.8h, v0.h[0] + add v30.8h, v26.8h, v11.8h + sub v13.8h, v26.8h, v11.8h + mls v23.8h, v27.8h, v7.h[0] + add v12.8h, v30.8h, v28.8h + sub v19.8h, v30.8h, v28.8h + mls v2.8h, v5.8h, v7.h[0] + str q12, [x0, #0x40] + sqrdmulh v26.8h, v13.8h, v0.h[3] + mls v8.8h, v10.8h, v7.h[0] + ldr q30, [x0, #0x150] + sub v20.8h, v2.8h, v23.8h + mul v4.8h, v13.8h, v0.h[2] + add v13.8h, v2.8h, v23.8h + mls v4.8h, v26.8h, v7.h[0] + ldr q2, [x0, #0x90] + mul v23.8h, v20.8h, v0.h[0] + ldr q29, [x0, #0x10] + sqrdmulh v10.8h, v20.8h, v0.h[1] + str q13, [x0, #0x80] + sub v13.8h, v4.8h, v8.8h + mls v17.8h, v3.8h, v7.h[0] + ldr q3, [x0, #0x110] + mls v18.8h, v21.8h, v7.h[0] + sub v21.8h, v2.8h, v25.8h + sqrdmulh v9.8h, v13.8h, v0.h[1] + str q17, [x0, #0x130] + mul v17.8h, v19.8h, v0.h[0] + sub x4, x4, #0x1 + cbnz x4, Lmlk_intt_layer123_start + mls v23.8h, v10.8h, v7.h[0] + ldr q11, [x0, #0x190] + str q18, [x0, #0x100] + add v27.8h, v3.8h, v30.8h + mul v13.8h, v13.8h, v0.h[0] + sub v5.8h, v29.8h, v24.8h + add v14.8h, v16.8h, v6.8h + mls v13.8h, v9.8h, v7.h[0] + add v10.8h, v11.8h, v22.8h + str q23, [x0, #0x180] + sub v20.8h, v11.8h, v22.8h + sub v23.8h, v27.8h, v10.8h + sqrdmulh v16.8h, v21.8h, v1.h[1] + sqrdmulh v31.8h, v23.8h, v0.h[5] + str q13, [x0, #0x1c0] + add v13.8h, v4.8h, v8.8h + mul v18.8h, v21.8h, v1.h[0] + str q13, [x0, #0xc0] + sqrdmulh v13.8h, v19.8h, v0.h[1] + sqrdmulh v28.8h, v20.8h, v1.h[5] + str q14, [x0], #0x10 + mul v4.8h, v20.8h, v1.h[4] + mls v17.8h, v13.8h, v7.h[0] + sub v13.8h, v3.8h, v30.8h + sqrdmulh v8.8h, v13.8h, v1.h[3] + mul v12.8h, v13.8h, v1.h[2] + mls v4.8h, v28.8h, v7.h[0] + mls v12.8h, v8.8h, v7.h[0] + mls v18.8h, v16.8h, v7.h[0] + str q17, [x0, #0x130] + sqrdmulh v15.8h, v5.8h, v0.h[7] + add v11.8h, v27.8h, v10.8h + mul v16.8h, v5.8h, v0.h[6] + sub v8.8h, v12.8h, v4.8h + sqrdmulh v28.8h, v8.8h, v0.h[5] + add v13.8h, v2.8h, v25.8h + mls v16.8h, v15.8h, v7.h[0] + add v26.8h, v12.8h, v4.8h + mul v8.8h, v8.8h, v0.h[4] + add v4.8h, v29.8h, v24.8h + mls v8.8h, v28.8h, v7.h[0] + sub v20.8h, v4.8h, v13.8h + add v14.8h, v4.8h, v13.8h + add v12.8h, v16.8h, v18.8h + sqrdmulh v22.8h, v20.8h, v0.h[3] + add v27.8h, v14.8h, v11.8h + sub v13.8h, v16.8h, v18.8h + mul v4.8h, v20.8h, v0.h[2] + str q27, [x0], #0x10 + sub v24.8h, v12.8h, v26.8h + sqrdmulh v3.8h, v13.8h, v0.h[3] + mul v13.8h, v13.8h, v0.h[2] + sqrdmulh v27.8h, v24.8h, v0.h[1] + mls v13.8h, v3.8h, v7.h[0] + mul v9.8h, v24.8h, v0.h[0] + mls v9.8h, v27.8h, v7.h[0] + add v30.8h, v13.8h, v8.8h + sub v13.8h, v13.8h, v8.8h + mls v4.8h, v22.8h, v7.h[0] + str q30, [x0, #0xb0] + sqrdmulh v16.8h, v13.8h, v0.h[1] + str q9, [x0, #0x130] + mul v9.8h, v13.8h, v0.h[0] + add v13.8h, v12.8h, v26.8h + str q13, [x0, #0x30] + mul v13.8h, v23.8h, v0.h[4] + sub v23.8h, v14.8h, v11.8h + mls v13.8h, v31.8h, v7.h[0] + mls v9.8h, v16.8h, v7.h[0] + mul v30.8h, v23.8h, v0.h[0] + sub v24.8h, v4.8h, v13.8h + add v13.8h, v4.8h, v13.8h + sqrdmulh v23.8h, v23.8h, v0.h[1] + str q9, [x0, #0x1b0] + str q13, [x0, #0x70] + sqrdmulh v13.8h, v24.8h, v0.h[1] + mul v21.8h, v24.8h, v0.h[0] + mls v30.8h, v23.8h, v7.h[0] + mls v21.8h, v13.8h, v7.h[0] + str q30, [x0, #0xf0] + str q21, [x0, #0x170] + ldp d8, d9, [sp] + .cfi_restore d8 + .cfi_restore d9 + ldp d10, d11, [sp, #0x10] + .cfi_restore d10 + .cfi_restore d11 + ldp d12, d13, [sp, #0x20] + .cfi_restore d12 + .cfi_restore d13 + ldp d14, d15, [sp, #0x30] + .cfi_restore d14 + .cfi_restore d15 + add sp, sp, #0x40 + .cfi_adjust_cfa_offset -0x40 + ret + .cfi_endproc + +MLK_ASM_FN_SIZE(intt_aarch64_asm) + +#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED && \ + (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/aarch64/src/mlkem_ntt_aarch64_asm.S b/cbits/mlkem/src/native/aarch64/src/mlkem_ntt_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/aarch64/src/mlkem_ntt_aarch64_asm.S @@ -0,0 +1,565 @@ +/* Copyright (c) 2022 Arm Limited + * Copyright (c) 2022 Hanno Becker + * Copyright (c) 2023 Amin Abdulrahman, Matthias Kannwischer + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [NeonNTT] + * Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1 + * Becker, Hwang, Kannwischer, Yang, Yang + * https://eprint.iacr.org/2021/986 + * + * - [SLOTHY_Paper] + * Fast and Clean: Auditable high-performance assembly via constraint solving + * Abdulrahman, Becker, Kannwischer, Klein + * https://eprint.iacr.org/2022/1303 + */ + +/*yaml + Name: ntt_aarch64_asm + Description: AArch64 ML-KEM forward NTT following @[NeonNTT] and @[SLOTHY_Paper] + Signature: void mlk_ntt_aarch64_asm(int16_t p[256], const int16_t twiddles12345[80], const int16_t twiddles56[384]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 512 + permissions: read/write + c_parameter: int16_t p[256] + description: Input/output polynomial + x1: + type: buffer + size_bytes: 160 + permissions: read-only + c_parameter: const int16_t twiddles12345[80] + description: Twiddle factors for layers 1-5 + x2: + type: buffer + size_bytes: 768 + permissions: read-only + c_parameter: const int16_t twiddles56[384] + description: Twiddle factors for layers 6-7 + Stack: + bytes: 64 + description: saving callee-saved Neon registers +*/ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_AARCH64) && !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/aarch64_opt/src/mlkem_ntt_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(ntt_aarch64_asm) +MLK_ASM_FN_SYMBOL(ntt_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x40 + .cfi_adjust_cfa_offset 0x40 + stp d8, d9, [sp] + .cfi_rel_offset d8, 0x0 + .cfi_rel_offset d9, 0x8 + stp d10, d11, [sp, #0x10] + .cfi_rel_offset d10, 0x10 + .cfi_rel_offset d11, 0x18 + stp d12, d13, [sp, #0x20] + .cfi_rel_offset d12, 0x20 + .cfi_rel_offset d13, 0x28 + stp d14, d15, [sp, #0x30] + .cfi_rel_offset d14, 0x30 + .cfi_rel_offset d15, 0x38 + mov w5, #0xd01 // =3329 + mov v7.h[0], w5 + mov w5, #0x4ebf // =20159 + mov v7.h[1], w5 + mov x3, x0 + mov x4, #0x4 // =4 + ldr q0, [x1], #0x20 + ldur q1, [x1, #-0x10] + ldr q21, [x0, #0x40] + ldr q5, [x0, #0x1c0] + ldr q30, [x0, #0x110] + ldr q24, [x0, #0x140] + ldr q12, [x0, #0x80] + sqrdmulh v9.8h, v5.8h, v0.h[1] + mul v23.8h, v5.8h, v0.h[0] + sqrdmulh v17.8h, v24.8h, v0.h[1] + ldr q13, [x0, #0xc0] + mls v23.8h, v9.8h, v7.h[0] + mul v8.8h, v24.8h, v0.h[0] + mls v8.8h, v17.8h, v7.h[0] + add v9.8h, v13.8h, v23.8h + sub v10.8h, v13.8h, v23.8h + mul v11.8h, v30.8h, v0.h[0] + ldr q13, [x0, #0x180] + sqrdmulh v28.8h, v9.8h, v0.h[3] + sub v29.8h, v21.8h, v8.8h + mul v26.8h, v9.8h, v0.h[2] + add v8.8h, v21.8h, v8.8h + mul v2.8h, v13.8h, v0.h[0] + mls v26.8h, v28.8h, v7.h[0] + mul v28.8h, v10.8h, v0.h[4] + sqrdmulh v23.8h, v10.8h, v0.h[5] + add v22.8h, v8.8h, v26.8h + sqrdmulh v10.8h, v13.8h, v0.h[1] + sqrdmulh v21.8h, v22.8h, v0.h[7] + ldr q13, [x0, #0x100] + mul v16.8h, v22.8h, v0.h[6] + mls v28.8h, v23.8h, v7.h[0] + mls v2.8h, v10.8h, v7.h[0] + sqrdmulh v23.8h, v13.8h, v0.h[1] + sub v10.8h, v29.8h, v28.8h + add v17.8h, v29.8h, v28.8h + mls v16.8h, v21.8h, v7.h[0] + sub v18.8h, v12.8h, v2.8h + ldr q29, [x0] + sqrdmulh v14.8h, v17.8h, v1.h[3] + add v22.8h, v12.8h, v2.8h + sqrdmulh v9.8h, v18.8h, v0.h[5] + mul v21.8h, v13.8h, v0.h[0] + ldr q13, [x0, #0x150] + mul v5.8h, v18.8h, v0.h[4] + mls v5.8h, v9.8h, v7.h[0] + mul v18.8h, v13.8h, v0.h[0] + mls v21.8h, v23.8h, v7.h[0] + sqrdmulh v2.8h, v13.8h, v0.h[1] + mul v13.8h, v17.8h, v1.h[2] + sub v4.8h, v29.8h, v21.8h + mls v13.8h, v14.8h, v7.h[0] + add v25.8h, v29.8h, v21.8h + add v6.8h, v4.8h, v5.8h + sqrdmulh v15.8h, v22.8h, v0.h[3] + sub v21.8h, v4.8h, v5.8h + sub v5.8h, v8.8h, v26.8h + mul v23.8h, v22.8h, v0.h[2] + add v28.8h, v6.8h, v13.8h + sub v13.8h, v6.8h, v13.8h + mul v4.8h, v5.8h, v1.h[0] + sub x4, x4, #0x2 + +Lmlk_ntt_layer123_start: + mls v23.8h, v15.8h, v7.h[0] + ldr q6, [x0, #0x190] + ldr q15, [x0, #0x90] + ldr q19, [x0, #0x10] + mul v22.8h, v10.8h, v1.h[4] + ldr q24, [x0, #0x50] + str q13, [x0, #0x140] + sqrdmulh v13.8h, v6.8h, v0.h[1] + sub v20.8h, v25.8h, v23.8h + sqrdmulh v3.8h, v30.8h, v0.h[1] + str q28, [x0, #0x100] + ldr q30, [x0, #0x120] + mul v8.8h, v6.8h, v0.h[0] + sqrdmulh v27.8h, v10.8h, v1.h[5] + mls v11.8h, v3.8h, v7.h[0] + mls v18.8h, v2.8h, v7.h[0] + ldr q31, [x0, #0x160] + sqrdmulh v10.8h, v5.8h, v1.h[1] + mls v8.8h, v13.8h, v7.h[0] + ldr q13, [x0, #0x1d0] + sub v14.8h, v24.8h, v18.8h + add v9.8h, v24.8h, v18.8h + sqrdmulh v2.8h, v31.8h, v0.h[1] + mls v4.8h, v10.8h, v7.h[0] + add v10.8h, v25.8h, v23.8h + sub v24.8h, v19.8h, v11.8h + add v25.8h, v19.8h, v11.8h + sqrdmulh v28.8h, v13.8h, v0.h[1] + mul v11.8h, v30.8h, v0.h[0] + mul v17.8h, v13.8h, v0.h[0] + sub v13.8h, v10.8h, v16.8h + sub v6.8h, v15.8h, v8.8h + mls v17.8h, v28.8h, v7.h[0] + str q13, [x0, #0x40] + mls v22.8h, v27.8h, v7.h[0] + ldr q13, [x0, #0xd0] + add v26.8h, v20.8h, v4.8h + mul v18.8h, v31.8h, v0.h[0] + add v27.8h, v10.8h, v16.8h + str q26, [x0, #0x80] + sqrdmulh v31.8h, v6.8h, v0.h[5] + add v3.8h, v21.8h, v22.8h + str q27, [x0], #0x10 + mul v26.8h, v6.8h, v0.h[4] + add v6.8h, v13.8h, v17.8h + sub v5.8h, v13.8h, v17.8h + str q3, [x0, #0x170] + sub v17.8h, v21.8h, v22.8h + sqrdmulh v10.8h, v6.8h, v0.h[3] + sub v13.8h, v20.8h, v4.8h + add v20.8h, v15.8h, v8.8h + sqrdmulh v12.8h, v5.8h, v0.h[5] + str q13, [x0, #0xb0] + mul v8.8h, v6.8h, v0.h[2] + str q17, [x0, #0x1b0] + mls v8.8h, v10.8h, v7.h[0] + mul v29.8h, v5.8h, v0.h[4] + mls v29.8h, v12.8h, v7.h[0] + sub v5.8h, v9.8h, v8.8h + add v3.8h, v9.8h, v8.8h + sqrdmulh v15.8h, v20.8h, v0.h[3] + mul v4.8h, v5.8h, v1.h[0] + add v6.8h, v14.8h, v29.8h + sqrdmulh v9.8h, v3.8h, v0.h[7] + sqrdmulh v12.8h, v6.8h, v1.h[3] + sub v10.8h, v14.8h, v29.8h + mul v23.8h, v6.8h, v1.h[2] + mls v26.8h, v31.8h, v7.h[0] + mls v23.8h, v12.8h, v7.h[0] + mul v16.8h, v3.8h, v0.h[6] + add v13.8h, v24.8h, v26.8h + sub v21.8h, v24.8h, v26.8h + mls v16.8h, v9.8h, v7.h[0] + add v28.8h, v13.8h, v23.8h + sub v13.8h, v13.8h, v23.8h + mul v23.8h, v20.8h, v0.h[2] + sub x4, x4, #0x1 + cbnz x4, Lmlk_ntt_layer123_start + sqrdmulh v3.8h, v5.8h, v1.h[1] + mls v23.8h, v15.8h, v7.h[0] + ldr q5, [x0, #0x190] + mul v29.8h, v10.8h, v1.h[4] + mls v4.8h, v3.8h, v7.h[0] + sub v19.8h, v25.8h, v23.8h + sqrdmulh v31.8h, v5.8h, v0.h[1] + sqrdmulh v6.8h, v30.8h, v0.h[1] + sub v3.8h, v19.8h, v4.8h + mul v5.8h, v5.8h, v0.h[0] + str q3, [x0, #0xc0] + sqrdmulh v12.8h, v10.8h, v1.h[5] + mls v18.8h, v2.8h, v7.h[0] + ldr q3, [x0, #0x1d0] + mls v5.8h, v31.8h, v7.h[0] + sqrdmulh v10.8h, v3.8h, v0.h[1] + mls v11.8h, v6.8h, v7.h[0] + ldr q31, [x0, #0x90] + mul v30.8h, v3.8h, v0.h[0] + mls v30.8h, v10.8h, v7.h[0] + sub v10.8h, v31.8h, v5.8h + mls v29.8h, v12.8h, v7.h[0] + ldr q6, [x0, #0xd0] + sqrdmulh v15.8h, v10.8h, v0.h[5] + mul v17.8h, v10.8h, v0.h[4] + add v10.8h, v6.8h, v30.8h + sub v6.8h, v6.8h, v30.8h + sqrdmulh v12.8h, v10.8h, v0.h[3] + sub v27.8h, v21.8h, v29.8h + sqrdmulh v3.8h, v6.8h, v0.h[5] + mul v10.8h, v10.8h, v0.h[2] + ldr q20, [x0, #0x50] + mls v10.8h, v12.8h, v7.h[0] + mul v2.8h, v6.8h, v0.h[4] + add v6.8h, v20.8h, v18.8h + add v5.8h, v31.8h, v5.8h + mls v2.8h, v3.8h, v7.h[0] + sub v31.8h, v6.8h, v10.8h + sqrdmulh v12.8h, v5.8h, v0.h[3] + sub v22.8h, v20.8h, v18.8h + add v6.8h, v6.8h, v10.8h + mul v20.8h, v31.8h, v1.h[0] + add v30.8h, v22.8h, v2.8h + sqrdmulh v3.8h, v6.8h, v0.h[7] + sqrdmulh v10.8h, v30.8h, v1.h[3] + mul v9.8h, v30.8h, v1.h[2] + ldr q30, [x0, #0x10] + mls v17.8h, v15.8h, v7.h[0] + mls v9.8h, v10.8h, v7.h[0] + mul v15.8h, v6.8h, v0.h[6] + add v24.8h, v30.8h, v11.8h + sub v10.8h, v22.8h, v2.8h + mls v15.8h, v3.8h, v7.h[0] + add v6.8h, v19.8h, v4.8h + add v22.8h, v25.8h, v23.8h + sqrdmulh v3.8h, v10.8h, v1.h[5] + str q13, [x0, #0x140] + sub v19.8h, v30.8h, v11.8h + add v25.8h, v22.8h, v16.8h + mul v5.8h, v5.8h, v0.h[2] + sub v13.8h, v22.8h, v16.8h + str q28, [x0, #0x100] + mls v5.8h, v12.8h, v7.h[0] + str q13, [x0, #0x40] + str q6, [x0, #0x80] + add v21.8h, v21.8h, v29.8h + sqrdmulh v13.8h, v31.8h, v1.h[1] + str q25, [x0], #0x10 + add v12.8h, v19.8h, v17.8h + sub v31.8h, v19.8h, v17.8h + mul v30.8h, v10.8h, v1.h[4] + str q21, [x0, #0x170] + add v21.8h, v24.8h, v5.8h + add v6.8h, v12.8h, v9.8h + mls v30.8h, v3.8h, v7.h[0] + str q27, [x0, #0x1b0] + sub v10.8h, v21.8h, v15.8h + sub v12.8h, v12.8h, v9.8h + mls v20.8h, v13.8h, v7.h[0] + str q6, [x0, #0x100] + str q10, [x0, #0x40] + sub v13.8h, v24.8h, v5.8h + add v3.8h, v21.8h, v15.8h + str q12, [x0, #0x140] + sub v10.8h, v31.8h, v30.8h + add v21.8h, v31.8h, v30.8h + str q3, [x0], #0x10 + add v12.8h, v13.8h, v20.8h + sub v13.8h, v13.8h, v20.8h + str q21, [x0, #0x170] + str q10, [x0, #0x1b0] + str q12, [x0, #0x70] + str q13, [x0, #0xb0] + mov x0, x3 + mov x4, #0x8 // =8 + ldr q2, [x0, #0x20] + ldr q13, [x1], #0x10 + ldr q30, [x0, #0x30] + ldr q25, [x2, #0x40] + ldr q5, [x0] + ldr q18, [x0, #0x60] + ldr q12, [x0, #0x70] + sqrdmulh v17.8h, v2.8h, v13.h[1] + ldr q4, [x1], #0x10 + ldr q23, [x0, #0x10] + sqrdmulh v21.8h, v30.8h, v13.h[1] + ldr q24, [x2, #0x20] + ldr q9, [x2], #0x60 + mul v10.8h, v30.8h, v13.h[0] + mul v11.8h, v2.8h, v13.h[0] + mls v10.8h, v21.8h, v7.h[0] + sqrdmulh v29.8h, v12.8h, v4.h[1] + mul v1.8h, v12.8h, v4.h[0] + add v21.8h, v23.8h, v10.8h + sub v10.8h, v23.8h, v10.8h + mul v8.8h, v18.8h, v4.h[0] + sqrdmulh v23.8h, v21.8h, v13.h[3] + mul v2.8h, v21.8h, v13.h[2] + mls v1.8h, v29.8h, v7.h[0] + mls v2.8h, v23.8h, v7.h[0] + ldur q15, [x2, #-0x50] + sqrdmulh v0.8h, v10.8h, v13.h[5] + mls v11.8h, v17.8h, v7.h[0] + ldr q29, [x0, #0x50] + mul v23.8h, v10.8h, v13.h[4] + mls v23.8h, v0.8h, v7.h[0] + sub v16.8h, v29.8h, v1.8h + add v3.8h, v5.8h, v11.8h + sub v31.8h, v5.8h, v11.8h + sqrdmulh v22.8h, v16.8h, v4.h[5] + add v30.8h, v3.8h, v2.8h + sub v0.8h, v3.8h, v2.8h + sqrdmulh v28.8h, v18.8h, v4.h[1] + add v21.8h, v31.8h, v23.8h + sub v19.8h, v31.8h, v23.8h + mul v26.8h, v16.8h, v4.h[4] + trn2 v3.4s, v30.4s, v0.4s + ldur q23, [x2, #-0x10] + trn2 v18.4s, v21.4s, v19.4s + mls v26.8h, v22.8h, v7.h[0] + trn1 v13.4s, v30.4s, v0.4s + mls v8.8h, v28.8h, v7.h[0] + trn2 v31.2d, v3.2d, v18.2d + trn1 v11.4s, v21.4s, v19.4s + add v27.8h, v29.8h, v1.8h + sqrdmulh v6.8h, v31.8h, v15.8h + trn1 v2.2d, v13.2d, v11.2d + trn2 v13.2d, v13.2d, v11.2d + mul v1.8h, v31.8h, v9.8h + ldr q11, [x0, #0x40] + sqrdmulh v29.8h, v13.8h, v15.8h + mls v1.8h, v6.8h, v7.h[0] + trn1 v6.2d, v3.2d, v18.2d + mul v17.8h, v13.8h, v9.8h + sub v13.8h, v11.8h, v8.8h + sqrdmulh v10.8h, v27.8h, v4.h[3] + sub v12.8h, v13.8h, v26.8h + sub v18.8h, v6.8h, v1.8h + mls v17.8h, v29.8h, v7.h[0] + add v30.8h, v6.8h, v1.8h + add v6.8h, v13.8h, v26.8h + ldur q13, [x2, #-0x30] + sqrdmulh v16.8h, v18.8h, v23.8h + trn1 v28.4s, v6.4s, v12.4s + mul v23.8h, v18.8h, v25.8h + ldr q25, [x2, #0x10] + add v20.8h, v2.8h, v17.8h + mul v0.8h, v30.8h, v24.8h + sqrdmulh v29.8h, v30.8h, v13.8h + sub v30.8h, v2.8h, v17.8h + mls v23.8h, v16.8h, v7.h[0] + sub x4, x4, #0x2 + +Lmlk_ntt_layer4567_start: + ldr q19, [x2, #0x50] + sub v31.8h, v30.8h, v23.8h + mls v0.8h, v29.8h, v7.h[0] + add v16.8h, v11.8h, v8.8h + ldr q18, [x0, #0xa0] + trn2 v14.4s, v6.4s, v12.4s + mul v26.8h, v27.8h, v4.h[2] + ldr q4, [x1], #0x10 + ldr q24, [x2, #0x40] + ldr q21, [x0, #0xb0] + mls v26.8h, v10.8h, v7.h[0] + add v23.8h, v30.8h, v23.8h + sub v15.8h, v20.8h, v0.8h + ldr q9, [x0, #0x90] + add v10.8h, v20.8h, v0.8h + mul v8.8h, v18.8h, v4.h[0] + ldr q1, [x2], #0x60 + trn1 v27.4s, v23.4s, v31.4s + sqrdmulh v12.8h, v18.8h, v4.h[1] + trn1 v5.4s, v10.4s, v15.4s + sub v30.8h, v16.8h, v26.8h + trn2 v13.2d, v5.2d, v27.2d + sqrdmulh v2.8h, v21.8h, v4.h[1] + add v29.8h, v16.8h, v26.8h + mul v0.8h, v21.8h, v4.h[0] + str q13, [x0, #0x20] + trn1 v11.4s, v29.4s, v30.4s + mls v8.8h, v12.8h, v7.h[0] + trn2 v26.4s, v29.4s, v30.4s + trn2 v6.2d, v11.2d, v28.2d + mls v0.8h, v2.8h, v7.h[0] + trn2 v16.2d, v26.2d, v14.2d + trn1 v26.2d, v26.2d, v14.2d + trn1 v20.2d, v5.2d, v27.2d + sqrdmulh v29.8h, v6.8h, v25.8h + trn2 v15.4s, v10.4s, v15.4s + sqrdmulh v13.8h, v16.8h, v25.8h + str q20, [x0], #0x40 + sub v30.8h, v9.8h, v0.8h + add v27.8h, v9.8h, v0.8h + mul v17.8h, v6.8h, v1.8h + sqrdmulh v22.8h, v30.8h, v4.h[5] + mul v18.8h, v16.8h, v1.8h + mls v18.8h, v13.8h, v7.h[0] + mul v2.8h, v30.8h, v4.h[4] + mls v2.8h, v22.8h, v7.h[0] + trn2 v22.4s, v23.4s, v31.4s + sub v3.8h, v26.8h, v18.8h + ldur q25, [x2, #-0x30] + mls v17.8h, v29.8h, v7.h[0] + trn2 v31.2d, v15.2d, v22.2d + trn1 v20.2d, v15.2d, v22.2d + add v16.8h, v26.8h, v18.8h + sqrdmulh v26.8h, v3.8h, v19.8h + trn1 v21.2d, v11.2d, v28.2d + ldr q11, [x0, #0x40] + sqrdmulh v29.8h, v16.8h, v25.8h + stur q20, [x0, #-0x30] + add v20.8h, v21.8h, v17.8h + stur q31, [x0, #-0x10] + mul v23.8h, v3.8h, v24.8h + ldr q25, [x2, #0x10] + sub v13.8h, v11.8h, v8.8h + mls v23.8h, v26.8h, v7.h[0] + ldur q1, [x2, #-0x40] + sub v12.8h, v13.8h, v2.8h + add v6.8h, v13.8h, v2.8h + sqrdmulh v10.8h, v27.8h, v4.h[3] + sub v30.8h, v21.8h, v17.8h + mul v0.8h, v16.8h, v1.8h + trn1 v28.4s, v6.4s, v12.4s + sub x4, x4, #0x1 + cbnz x4, Lmlk_ntt_layer4567_start + add v22.8h, v11.8h, v8.8h + mul v27.8h, v27.8h, v4.h[2] + trn2 v17.4s, v6.4s, v12.4s + ldr q15, [x2], #0x60 + mls v27.8h, v10.8h, v7.h[0] + add v4.8h, v30.8h, v23.8h + sub v18.8h, v30.8h, v23.8h + ldur q6, [x2, #-0x30] + mls v0.8h, v29.8h, v7.h[0] + ldur q12, [x2, #-0x40] + ldur q24, [x2, #-0x20] + ldur q2, [x2, #-0x10] + trn1 v9.4s, v4.4s, v18.4s + add v10.8h, v22.8h, v27.8h + sub v13.8h, v22.8h, v27.8h + sub v1.8h, v20.8h, v0.8h + trn2 v21.4s, v10.4s, v13.4s + add v27.8h, v20.8h, v0.8h + trn2 v3.2d, v21.2d, v17.2d + trn1 v13.4s, v10.4s, v13.4s + trn1 v31.4s, v27.4s, v1.4s + sqrdmulh v10.8h, v3.8h, v25.8h + trn2 v5.2d, v13.2d, v28.2d + trn1 v13.2d, v13.2d, v28.2d + trn1 v21.2d, v21.2d, v17.2d + sqrdmulh v17.8h, v5.8h, v25.8h + trn2 v30.2d, v31.2d, v9.2d + mul v25.8h, v3.8h, v15.8h + str q30, [x0, #0x20] + trn2 v30.4s, v4.4s, v18.4s + mls v25.8h, v10.8h, v7.h[0] + trn2 v3.4s, v27.4s, v1.4s + mul v20.8h, v5.8h, v15.8h + trn2 v10.2d, v3.2d, v30.2d + mls v20.8h, v17.8h, v7.h[0] + str q10, [x0, #0x30] + sub v18.8h, v21.8h, v25.8h + add v10.8h, v21.8h, v25.8h + trn1 v3.2d, v3.2d, v30.2d + sqrdmulh v30.8h, v18.8h, v2.8h + mul v12.8h, v10.8h, v12.8h + sqrdmulh v6.8h, v10.8h, v6.8h + str q3, [x0, #0x10] + add v21.8h, v13.8h, v20.8h + mul v10.8h, v18.8h, v24.8h + sub v13.8h, v13.8h, v20.8h + mls v10.8h, v30.8h, v7.h[0] + mls v12.8h, v6.8h, v7.h[0] + trn1 v30.2d, v31.2d, v9.2d + sub v3.8h, v13.8h, v10.8h + add v6.8h, v13.8h, v10.8h + add v10.8h, v21.8h, v12.8h + sub v21.8h, v21.8h, v12.8h + trn2 v13.4s, v6.4s, v3.4s + trn1 v12.4s, v10.4s, v21.4s + trn2 v21.4s, v10.4s, v21.4s + trn1 v3.4s, v6.4s, v3.4s + str q30, [x0], #0x40 + trn2 v10.2d, v21.2d, v13.2d + trn1 v13.2d, v21.2d, v13.2d + trn2 v21.2d, v12.2d, v3.2d + trn1 v3.2d, v12.2d, v3.2d + str q10, [x0, #0x30] + str q13, [x0, #0x10] + str q3, [x0], #0x40 + stur q21, [x0, #-0x20] + ldp d8, d9, [sp] + .cfi_restore d8 + .cfi_restore d9 + ldp d10, d11, [sp, #0x10] + .cfi_restore d10 + .cfi_restore d11 + ldp d12, d13, [sp, #0x20] + .cfi_restore d12 + .cfi_restore d13 + ldp d14, d15, [sp, #0x30] + .cfi_restore d14 + .cfi_restore d15 + add sp, sp, #0x40 + .cfi_adjust_cfa_offset -0x40 + ret + .cfi_endproc + +MLK_ASM_FN_SIZE(ntt_aarch64_asm) + +#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/aarch64/src/mlkem_poly_mulcache_compute_aarch64_asm.S b/cbits/mlkem/src/native/aarch64/src/mlkem_poly_mulcache_compute_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/aarch64/src/mlkem_poly_mulcache_compute_aarch64_asm.S @@ -0,0 +1,130 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/*yaml + Name: poly_mulcache_compute_aarch64_asm + Description: Compute multiplication cache for polynomial + Signature: void mlk_poly_mulcache_compute_aarch64_asm(int16_t cache[128], const int16_t mlk_poly[256], const int16_t zetas[128], const int16_t zetas_twisted[128]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 256 + permissions: write-only + c_parameter: int16_t cache[128] + description: Output cache + x1: + type: buffer + size_bytes: 512 + permissions: read-only + c_parameter: const int16_t mlk_poly[256] + description: Input polynomial + x2: + type: buffer + size_bytes: 256 + permissions: read-only + c_parameter: const int16_t zetas[128] + description: Zeta values + x3: + type: buffer + size_bytes: 256 + permissions: read-only + c_parameter: const int16_t zetas_twisted[128] + description: Twisted zeta values + Stack: + bytes: 0 +*/ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_AARCH64) && !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/aarch64_opt/src/mlkem_poly_mulcache_compute_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(poly_mulcache_compute_aarch64_asm) +MLK_ASM_FN_SYMBOL(poly_mulcache_compute_aarch64_asm) + + .cfi_startproc + mov w5, #0xd01 // =3329 + dup v6.8h, w5 + mov w5, #0x4ebf // =20159 + dup v7.8h, w5 + mov x4, #0x10 // =16 + ldr q0, [x1], #0x20 + ldur q2, [x1, #-0x10] + ldr q19, [x1], #0x20 + ldr q29, [x3], #0x10 + ldur q16, [x1, #-0x10] + ldr q18, [x2], #0x10 + ldr q26, [x1], #0x20 + ldr q25, [x2], #0x10 + uzp2 v5.8h, v0.8h, v2.8h + ldr q28, [x3], #0x10 + ldur q7, [x1, #-0x10] + ldr q2, [x1], #0x20 + uzp2 v27.8h, v19.8h, v16.8h + sqrdmulh v16.8h, v5.8h, v29.8h + ldr q17, [x3], #0x10 + ldr q19, [x3], #0x10 + mul v5.8h, v5.8h, v18.8h + uzp2 v29.8h, v26.8h, v7.8h + mul v26.8h, v27.8h, v25.8h + sqrdmulh v4.8h, v27.8h, v28.8h + mls v5.8h, v16.8h, v6.h[0] + lsr x4, x4, #1 + sub x4, x4, #0x2 + +Lmlk_poly_mulcache_compute_loop_start: + str q5, [x0], #0x10 + sqrdmulh v22.8h, v29.8h, v17.8h + ldr q28, [x2], #0x10 + ldur q24, [x1, #-0x10] + ldr q0, [x1], #0x20 + mls v26.8h, v4.8h, v6.h[0] + ldur q16, [x1, #-0x10] + ldr q17, [x3], #0x10 + mul v5.8h, v29.8h, v28.8h + uzp2 v23.8h, v2.8h, v24.8h + ldr q18, [x2], #0x10 + mls v5.8h, v22.8h, v6.h[0] + uzp2 v29.8h, v0.8h, v16.8h + sqrdmulh v4.8h, v23.8h, v19.8h + ldr q2, [x1], #0x20 + ldr q19, [x3], #0x10 + str q26, [x0], #0x10 + mul v26.8h, v23.8h, v18.8h + subs x4, x4, #0x1 + cbnz x4, Lmlk_poly_mulcache_compute_loop_start + mls v26.8h, v4.8h, v6.h[0] + str q5, [x0], #0x10 + ldr q5, [x2], #0x10 + ldur q4, [x1, #-0x10] + sqrdmulh v16.8h, v29.8h, v17.8h + ldr q0, [x2], #0x10 + mul v29.8h, v29.8h, v5.8h + uzp2 v18.8h, v2.8h, v4.8h + str q26, [x0], #0x10 + sqrdmulh v17.8h, v18.8h, v19.8h + mls v29.8h, v16.8h, v6.h[0] + mul v26.8h, v18.8h, v0.8h + mls v26.8h, v17.8h, v6.h[0] + str q29, [x0], #0x10 + str q26, [x0], #0x10 + ret + .cfi_endproc + +MLK_ASM_FN_SIZE(poly_mulcache_compute_aarch64_asm) + +#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/aarch64/src/mlkem_poly_reduce_aarch64_asm.S b/cbits/mlkem/src/native/aarch64/src/mlkem_poly_reduce_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/aarch64/src/mlkem_poly_reduce_aarch64_asm.S @@ -0,0 +1,153 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/*yaml + Name: poly_reduce_aarch64_asm + Description: Barrett reduction of polynomial coefficients + Signature: void mlk_poly_reduce_aarch64_asm(int16_t p[256]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 512 + permissions: read/write + c_parameter: int16_t p[256] + description: Input/output polynomial + Stack: + bytes: 0 +*/ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_AARCH64) && !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/aarch64_opt/src/mlkem_poly_reduce_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(poly_reduce_aarch64_asm) +MLK_ASM_FN_SYMBOL(poly_reduce_aarch64_asm) + + .cfi_startproc + mov w2, #0xd01 // =3329 + dup v3.8h, w2 + mov w2, #0x4ebf // =20159 + dup v4.8h, w2 + mov x1, #0x8 // =8 + ldr q21, [x0], #0x40 + ldur q18, [x0, #-0x20] + ldur q0, [x0, #-0x30] + ldur q5, [x0, #-0x10] + ldr q26, [x0], #0x40 + sqdmulh v17.8h, v21.8h, v4.h[0] + sqdmulh v27.8h, v18.8h, v4.h[0] + sqdmulh v22.8h, v0.8h, v4.h[0] + srshr v17.8h, v17.8h, #0xb + sqdmulh v23.8h, v5.8h, v4.h[0] + srshr v29.8h, v27.8h, #0xb + mls v21.8h, v17.8h, v3.h[0] + srshr v17.8h, v22.8h, #0xb + mls v18.8h, v29.8h, v3.h[0] + srshr v22.8h, v23.8h, #0xb + mls v0.8h, v17.8h, v3.h[0] + sshr v2.8h, v21.8h, #0xf + mls v5.8h, v22.8h, v3.h[0] + sshr v29.8h, v18.8h, #0xf + and v19.16b, v3.16b, v2.16b + sqdmulh v2.8h, v26.8h, v4.h[0] + sshr v31.8h, v0.8h, #0xf + add v17.8h, v21.8h, v19.8h + and v21.16b, v3.16b, v29.16b + and v31.16b, v3.16b, v31.16b + sub x1, x1, #0x2 + +Lmlk_poly_reduce_loop_start: + add v21.8h, v18.8h, v21.8h + ldur q18, [x0, #-0x20] + add v25.8h, v0.8h, v31.8h + ldur q0, [x0, #-0x30] + stur q21, [x0, #-0x60] + sshr v28.8h, v5.8h, #0xf + stur q17, [x0, #-0x80] + srshr v23.8h, v2.8h, #0xb + sqdmulh v30.8h, v18.8h, v4.h[0] + stur q25, [x0, #-0x70] + and v22.16b, v3.16b, v28.16b + sqdmulh v7.8h, v0.8h, v4.h[0] + add v16.8h, v5.8h, v22.8h + ldur q5, [x0, #-0x10] + mls v26.8h, v23.8h, v3.h[0] + stur q16, [x0, #-0x50] + srshr v6.8h, v30.8h, #0xb + srshr v1.8h, v7.8h, #0xb + sqdmulh v19.8h, v5.8h, v4.h[0] + mls v18.8h, v6.8h, v3.h[0] + sshr v24.8h, v26.8h, #0xf + mls v0.8h, v1.8h, v3.h[0] + and v27.16b, v3.16b, v24.16b + srshr v29.8h, v19.8h, #0xb + add v17.8h, v26.8h, v27.8h + ldr q26, [x0], #0x40 + sshr v1.8h, v18.8h, #0xf + mls v5.8h, v29.8h, v3.h[0] + sshr v20.8h, v0.8h, #0xf + and v21.16b, v3.16b, v1.16b + and v31.16b, v3.16b, v20.16b + sqdmulh v2.8h, v26.8h, v4.h[0] + subs x1, x1, #0x1 + cbnz x1, Lmlk_poly_reduce_loop_start + add v28.8h, v0.8h, v31.8h + ldur q29, [x0, #-0x10] + add v21.8h, v18.8h, v21.8h + srshr v18.8h, v2.8h, #0xb + sshr v2.8h, v5.8h, #0xf + ldur q16, [x0, #-0x20] + stur q17, [x0, #-0x80] + ldur q0, [x0, #-0x30] + and v2.16b, v3.16b, v2.16b + sqdmulh v24.8h, v29.8h, v4.h[0] + stur q28, [x0, #-0x70] + stur q21, [x0, #-0x60] + add v31.8h, v5.8h, v2.8h + sqdmulh v6.8h, v16.8h, v4.h[0] + stur q31, [x0, #-0x50] + sqdmulh v17.8h, v0.8h, v4.h[0] + srshr v22.8h, v24.8h, #0xb + mls v26.8h, v18.8h, v3.h[0] + srshr v31.8h, v6.8h, #0xb + mls v29.8h, v22.8h, v3.h[0] + srshr v19.8h, v17.8h, #0xb + mls v16.8h, v31.8h, v3.h[0] + sshr v7.8h, v26.8h, #0xf + mls v0.8h, v19.8h, v3.h[0] + and v5.16b, v3.16b, v7.16b + sshr v22.8h, v29.8h, #0xf + add v27.8h, v26.8h, v5.8h + and v26.16b, v3.16b, v22.16b + sshr v20.8h, v16.8h, #0xf + stur q27, [x0, #-0x40] + and v2.16b, v3.16b, v20.16b + sshr v23.8h, v0.8h, #0xf + add v18.8h, v29.8h, v26.8h + add v31.8h, v16.8h, v2.8h + and v29.16b, v3.16b, v23.16b + stur q18, [x0, #-0x10] + add v25.8h, v0.8h, v29.8h + stur q31, [x0, #-0x20] + stur q25, [x0, #-0x30] + ret + .cfi_endproc + +MLK_ASM_FN_SIZE(poly_reduce_aarch64_asm) + +#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/aarch64/src/mlkem_poly_tobytes_aarch64_asm.S b/cbits/mlkem/src/native/aarch64/src/mlkem_poly_tobytes_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/aarch64/src/mlkem_poly_tobytes_aarch64_asm.S @@ -0,0 +1,124 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/*yaml + Name: poly_tobytes_aarch64_asm + Description: Convert polynomial to byte representation + Signature: void mlk_poly_tobytes_aarch64_asm(uint8_t r[384], const int16_t a[256]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 384 + permissions: write-only + c_parameter: uint8_t r[384] + description: Output byte array + x1: + type: buffer + size_bytes: 512 + permissions: read-only + c_parameter: const int16_t a[256] + description: Input polynomial + Stack: + bytes: 0 +*/ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_AARCH64) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + (!defined(MLK_CONFIG_NO_KEYPAIR_API) || \ + !defined(MLK_CONFIG_NO_ENCAPS_API)) + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/aarch64_opt/src/mlkem_poly_tobytes_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(poly_tobytes_aarch64_asm) +MLK_ASM_FN_SYMBOL(poly_tobytes_aarch64_asm) + + .cfi_startproc + mov x2, #0x10 // =16 + ldr q5, [x1, #0x10] + ldr q3, [x1], #0x20 + ldr q29, [x1], #0x20 + ldur q2, [x1, #-0x10] + ldr q27, [x1, #0x10] + ldr q23, [x1, #0x30] + ldr q17, [x1], #0x20 + ldr q16, [x1], #0x20 + uzp2 v26.8h, v3.8h, v5.8h + uzp1 v19.8h, v3.8h, v5.8h + uzp2 v0.8h, v29.8h, v2.8h + uzp1 v1.8h, v29.8h, v2.8h + xtn v5.8b, v26.8h + shrn v3.8b, v19.8h, #0x8 + shrn v4.8b, v26.8h, #0x4 + xtn v18.8b, v0.8h + shrn v30.8b, v0.8h, #0x4 + xtn v28.8b, v1.8h + shrn v29.8b, v1.8h, #0x8 + sli v3.8b, v5.8b, #0x4 + xtn v2.8b, v19.8h + sli v29.8b, v18.8b, #0x4 + lsr x2, x2, #1 + sub x2, x2, #0x2 + +Lmlk_poly_tobytes_loop_start: + uzp1 v25.8h, v17.8h, v27.8h + uzp2 v31.8h, v17.8h, v27.8h + uzp1 v24.8h, v16.8h, v23.8h + uzp2 v6.8h, v16.8h, v23.8h + st3 { v2.8b, v3.8b, v4.8b }, [x0], #24 + shrn v3.8b, v25.8h, #0x8 + ldr q17, [x1], #0x20 + shrn v4.8b, v31.8h, #0x4 + xtn v21.8b, v6.8h + ldr q23, [x1, #0x10] + st3 { v28.8b, v29.8b, v30.8b }, [x0], #24 + shrn v29.8b, v24.8h, #0x8 + ldur q27, [x1, #-0x10] + xtn v20.8b, v31.8h + ldr q16, [x1], #0x20 + sli v29.8b, v21.8b, #0x4 + xtn v2.8b, v25.8h + sli v3.8b, v20.8b, #0x4 + xtn v28.8b, v24.8h + shrn v30.8b, v6.8h, #0x4 + subs x2, x2, #0x1 + cbnz x2, Lmlk_poly_tobytes_loop_start + uzp2 v7.8h, v17.8h, v27.8h + uzp1 v25.8h, v17.8h, v27.8h + uzp2 v0.8h, v16.8h, v23.8h + st3 { v2.8b, v3.8b, v4.8b }, [x0], #24 + st3 { v28.8b, v29.8b, v30.8b }, [x0], #24 + shrn v21.8b, v25.8h, #0x8 + uzp1 v2.8h, v16.8h, v23.8h + shrn v22.8b, v7.8h, #0x4 + shrn v4.8b, v0.8h, #0x4 + xtn v28.8b, v7.8h + xtn v27.8b, v0.8h + shrn v3.8b, v2.8h, #0x8 + sli v21.8b, v28.8b, #0x4 + xtn v2.8b, v2.8h + sli v3.8b, v27.8b, #0x4 + xtn v20.8b, v25.8h + st3 { v20.8b, v21.8b, v22.8b }, [x0], #24 + st3 { v2.8b, v3.8b, v4.8b }, [x0], #24 + ret + .cfi_endproc + +MLK_ASM_FN_SIZE(poly_tobytes_aarch64_asm) + +#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED && \ + (!MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/aarch64/src/mlkem_poly_tomont_aarch64_asm.S b/cbits/mlkem/src/native/aarch64/src/mlkem_poly_tomont_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/aarch64/src/mlkem_poly_tomont_aarch64_asm.S @@ -0,0 +1,102 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/*yaml + Name: poly_tomont_aarch64_asm + Description: Convert polynomial to Montgomery domain + Signature: void mlk_poly_tomont_aarch64_asm(int16_t p[256]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 512 + permissions: read/write + c_parameter: int16_t p[256] + description: Input/output polynomial + Stack: + bytes: 0 +*/ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_AARCH64) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + !defined(MLK_CONFIG_NO_KEYPAIR_API) + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/aarch64_opt/src/mlkem_poly_tomont_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(poly_tomont_aarch64_asm) +MLK_ASM_FN_SYMBOL(poly_tomont_aarch64_asm) + + .cfi_startproc + mov w2, #0xd01 // =3329 + dup v4.8h, w2 + mov w2, #-0x414 // =-1044 + dup v2.8h, w2 + mov w2, #-0x2824 // =-10276 + dup v3.8h, w2 + mov x1, #0x8 // =8 + ldr q18, [x0, #0x20] + ldr q0, [x0, #0x10] + ldr q16, [x0], #0x40 + sqrdmulh v23.8h, v0.8h, v3.8h + mul v26.8h, v0.8h, v2.8h + sqrdmulh v19.8h, v16.8h, v3.8h + mls v26.8h, v23.8h, v4.h[0] + mul v29.8h, v16.8h, v2.8h + ldur q16, [x0, #-0x10] + mls v29.8h, v19.8h, v4.h[0] + stur q26, [x0, #-0x30] + sqrdmulh v26.8h, v18.8h, v3.8h + mul v18.8h, v18.8h, v2.8h + stur q29, [x0, #-0x40] + sqrdmulh v29.8h, v16.8h, v3.8h + mls v18.8h, v26.8h, v4.h[0] + sub x1, x1, #0x1 + +Lmlk_poly_tomont_loop: + ldr q19, [x0, #0x10] + mul v26.8h, v16.8h, v2.8h + ldr q23, [x0, #0x20] + ldr q17, [x0], #0x40 + mls v26.8h, v29.8h, v4.h[0] + ldur q16, [x0, #-0x10] + sqrdmulh v28.8h, v19.8h, v3.8h + stur q18, [x0, #-0x60] + mul v0.8h, v19.8h, v2.8h + stur q26, [x0, #-0x50] + sqrdmulh v24.8h, v23.8h, v3.8h + mul v18.8h, v23.8h, v2.8h + sqrdmulh v22.8h, v17.8h, v3.8h + mul v26.8h, v17.8h, v2.8h + mls v0.8h, v28.8h, v4.h[0] + mls v26.8h, v22.8h, v4.h[0] + sqrdmulh v29.8h, v16.8h, v3.8h + stur q0, [x0, #-0x30] + mls v18.8h, v24.8h, v4.h[0] + stur q26, [x0, #-0x40] + sub x1, x1, #0x1 + cbnz x1, Lmlk_poly_tomont_loop + mul v16.8h, v16.8h, v2.8h + stur q18, [x0, #-0x20] + mls v16.8h, v29.8h, v4.h[0] + stur q16, [x0, #-0x10] + ret + .cfi_endproc + +MLK_ASM_FN_SIZE(poly_tomont_aarch64_asm) + +#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED && \ + !MLK_CONFIG_NO_KEYPAIR_API */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm.S b/cbits/mlkem/src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm.S @@ -0,0 +1,264 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [NeonNTT] + * Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1 + * Becker, Hwang, Kannwischer, Yang, Yang + * https://eprint.iacr.org/2021/986 + */ + +/*yaml + Name: polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm + Description: Re-implementation of asymmetric base multiplication following @[NeonNTT] for k=2 + Signature: void mlk_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm(int16_t r[256], const int16_t a[512], const int16_t b[512], const int16_t b_cache[256]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 512 + permissions: write-only + c_parameter: int16_t r[256] + description: Output polynomial + x1: + type: buffer + size_bytes: 1024 + permissions: read-only + c_parameter: const int16_t a[512] + description: Input polynomial vector a + x2: + type: buffer + size_bytes: 1024 + permissions: read-only + c_parameter: const int16_t b[512] + description: Input polynomial vector b + x3: + type: buffer + size_bytes: 512 + permissions: read-only + c_parameter: const int16_t b_cache[256] + description: Cached values for b + Stack: + bytes: 64 + description: saving callee-saved Neon registers +*/ + +/* Re-implementation of asymmetric base multiplication following @[NeonNTT] */ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_AARCH64) && !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2) + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/aarch64_opt/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm) +MLK_ASM_FN_SYMBOL(polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x40 + .cfi_adjust_cfa_offset 0x40 + stp d8, d9, [sp] + .cfi_rel_offset d8, 0x0 + .cfi_rel_offset d9, 0x8 + stp d10, d11, [sp, #0x10] + .cfi_rel_offset d10, 0x10 + .cfi_rel_offset d11, 0x18 + stp d12, d13, [sp, #0x20] + .cfi_rel_offset d12, 0x20 + .cfi_rel_offset d13, 0x28 + stp d14, d15, [sp, #0x30] + .cfi_rel_offset d14, 0x30 + .cfi_rel_offset d15, 0x38 + mov w14, #0xd01 // =3329 + dup v0.8h, w14 + mov w14, #0xcff // =3327 + dup v2.8h, w14 + add x4, x1, #0x200 + add x5, x2, #0x200 + add x6, x3, #0x100 + mov x13, #0x10 // =16 + ldr q12, [x1], #0x20 + ldur q9, [x1, #-0x10] + ldr q22, [x2], #0x20 + ldur q30, [x2, #-0x10] + ldr q6, [x5], #0x20 + ldr q7, [x4, #0x10] + ldr q8, [x4], #0x20 + ldur q23, [x5, #-0x10] + uzp1 v16.8h, v12.8h, v9.8h + uzp2 v14.8h, v12.8h, v9.8h + uzp2 v13.8h, v22.8h, v30.8h + uzp1 v18.8h, v22.8h, v30.8h + ld1 { v27.8h }, [x3], #16 + ld1 { v17.8h }, [x6], #16 + smull2 v4.4s, v16.8h, v18.8h + ldr q31, [x1, #0x10] + smull v19.4s, v16.4h, v13.4h + ldr q24, [x1], #0x20 + smlal v19.4s, v14.4h, v18.4h + ldr q22, [x2], #0x20 + smlal2 v4.4s, v14.8h, v27.8h + uzp2 v5.8h, v6.8h, v23.8h + smull2 v29.4s, v16.8h, v13.8h + uzp2 v26.8h, v8.8h, v7.8h + smlal2 v29.4s, v14.8h, v18.8h + uzp1 v30.8h, v24.8h, v31.8h + uzp1 v8.8h, v8.8h, v7.8h + smull v11.4s, v16.4h, v18.4h + smlal v11.4s, v14.4h, v27.4h + ldur q1, [x2, #-0x10] + uzp1 v28.8h, v6.8h, v23.8h + smlal2 v29.4s, v8.8h, v5.8h + ldr q25, [x5], #0x20 + smlal v19.4s, v8.4h, v5.4h + ldr q3, [x4, #0x10] + smlal2 v29.4s, v26.8h, v28.8h + uzp1 v27.8h, v22.8h, v1.8h + smlal v19.4s, v26.4h, v28.4h + ldr q12, [x4], #0x20 + smlal2 v4.4s, v8.8h, v28.8h + ldur q21, [x5, #-0x10] + smlal2 v4.4s, v26.8h, v17.8h + smlal v11.4s, v8.4h, v28.4h + ld1 { v15.8h }, [x6], #16 + smlal v11.4s, v26.4h, v17.4h + ld1 { v20.8h }, [x3], #16 + uzp1 v28.8h, v19.8h, v29.8h + smull2 v23.4s, v30.8h, v27.8h + smull v26.4s, v30.4h, v27.4h + uzp2 v16.8h, v22.8h, v1.8h + mul v28.8h, v28.8h, v2.8h + uzp1 v10.8h, v11.8h, v4.8h + smull2 v8.4s, v30.8h, v16.8h + mul v13.8h, v10.8h, v2.8h + smlal v19.4s, v28.4h, v0.4h + smlal2 v29.4s, v28.8h, v0.8h + smull v18.4s, v30.4h, v16.4h + uzp1 v30.8h, v25.8h, v21.8h + smlal v11.4s, v13.4h, v0.4h + uzp2 v6.8h, v24.8h, v31.8h + uzp1 v16.8h, v12.8h, v3.8h + smlal2 v4.4s, v13.8h, v0.8h + uzp2 v17.8h, v25.8h, v21.8h + smlal2 v8.4s, v6.8h, v27.8h + uzp2 v12.8h, v12.8h, v3.8h + smlal v18.4s, v6.4h, v27.4h + uzp2 v9.8h, v19.8h, v29.8h + smlal2 v8.4s, v16.8h, v17.8h + smlal2 v8.4s, v12.8h, v30.8h + uzp2 v19.8h, v11.8h, v4.8h + sub x13, x13, #0x2 + +Lmlk_polyvec_basemul_acc_montgomery_cached_k2_loop_start: + smlal v18.4s, v16.4h, v17.4h + ldr q7, [x4], #0x20 + ldr q10, [x2, #0x10] + smlal v18.4s, v12.4h, v30.4h + smlal2 v23.4s, v6.8h, v20.8h + ldr q14, [x2], #0x20 + smlal2 v23.4s, v16.8h, v30.8h + zip1 v25.8h, v19.8h, v9.8h + zip2 v3.8h, v19.8h, v9.8h + smlal2 v23.4s, v12.8h, v15.8h + smlal v26.4s, v6.4h, v20.4h + uzp1 v5.8h, v18.8h, v8.8h + uzp2 v21.8h, v14.8h, v10.8h + smlal v26.4s, v16.4h, v30.4h + str q25, [x0], #0x20 + mul v29.8h, v5.8h, v2.8h + uzp1 v24.8h, v14.8h, v10.8h + stur q3, [x0, #-0x10] + smlal v26.4s, v12.4h, v15.4h + ld1 { v15.8h }, [x6], #16 + ldr q28, [x1, #0x10] + ldr q11, [x1], #0x20 + ldr q13, [x5], #0x20 + ldur q27, [x4, #-0x10] + smlal2 v8.4s, v29.8h, v0.8h + ldur q22, [x5, #-0x10] + smlal v18.4s, v29.4h, v0.4h + uzp1 v4.8h, v26.8h, v23.8h + uzp1 v1.8h, v11.8h, v28.8h + uzp2 v6.8h, v11.8h, v28.8h + uzp1 v16.8h, v7.8h, v27.8h + mul v31.8h, v4.8h, v2.8h + uzp2 v17.8h, v13.8h, v22.8h + ld1 { v20.8h }, [x3], #16 + uzp2 v9.8h, v18.8h, v8.8h + smull2 v8.4s, v1.8h, v21.8h + uzp1 v30.8h, v13.8h, v22.8h + smlal2 v8.4s, v6.8h, v24.8h + smlal2 v8.4s, v16.8h, v17.8h + uzp2 v12.8h, v7.8h, v27.8h + smlal v26.4s, v31.4h, v0.4h + smlal2 v23.4s, v31.8h, v0.8h + smull v18.4s, v1.4h, v21.4h + smlal v18.4s, v6.4h, v24.4h + smlal2 v8.4s, v12.8h, v30.8h + uzp2 v19.8h, v26.8h, v23.8h + smull2 v23.4s, v1.8h, v24.8h + smull v26.4s, v1.4h, v24.4h + subs x13, x13, #0x1 + cbnz x13, Lmlk_polyvec_basemul_acc_montgomery_cached_k2_loop_start + smlal v26.4s, v6.4h, v20.4h + smlal2 v23.4s, v6.8h, v20.8h + smlal v26.4s, v16.4h, v30.4h + smlal2 v23.4s, v16.8h, v30.8h + smlal v26.4s, v12.4h, v15.4h + smlal2 v23.4s, v12.8h, v15.8h + smlal v18.4s, v16.4h, v17.4h + smlal v18.4s, v12.4h, v30.4h + zip1 v12.8h, v19.8h, v9.8h + str q12, [x0], #0x20 + uzp1 v12.8h, v26.8h, v23.8h + mul v6.8h, v12.8h, v2.8h + uzp1 v12.8h, v18.8h, v8.8h + mul v12.8h, v12.8h, v2.8h + smlal v26.4s, v6.4h, v0.4h + smlal2 v23.4s, v6.8h, v0.8h + smlal2 v8.4s, v12.8h, v0.8h + smlal v18.4s, v12.4h, v0.4h + zip2 v12.8h, v19.8h, v9.8h + uzp2 v6.8h, v26.8h, v23.8h + stur q12, [x0, #-0x10] + uzp2 v12.8h, v18.8h, v8.8h + zip2 v1.8h, v6.8h, v12.8h + zip1 v12.8h, v6.8h, v12.8h + str q1, [x0, #0x10] + str q12, [x0], #0x20 + ldp d8, d9, [sp] + .cfi_restore d8 + .cfi_restore d9 + ldp d10, d11, [sp, #0x10] + .cfi_restore d10 + .cfi_restore d11 + ldp d12, d13, [sp, #0x20] + .cfi_restore d12 + .cfi_restore d13 + ldp d14, d15, [sp, #0x30] + .cfi_restore d14 + .cfi_restore d15 + add sp, sp, #0x40 + .cfi_adjust_cfa_offset -0x40 + ret + .cfi_endproc + +MLK_ASM_FN_SIZE(polyvec_basemul_acc_montgomery_cached_k2_aarch64_asm) + + +#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm.S b/cbits/mlkem/src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm.S @@ -0,0 +1,317 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [NeonNTT] + * Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1 + * Becker, Hwang, Kannwischer, Yang, Yang + * https://eprint.iacr.org/2021/986 + */ + +/*yaml + Name: polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm + Description: Re-implementation of asymmetric base multiplication following @[NeonNTT] for k=3 + Signature: void mlk_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm(int16_t r[256], const int16_t a[768], const int16_t b[768], const int16_t b_cache[384]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 512 + permissions: write-only + c_parameter: int16_t r[256] + description: Output polynomial + x1: + type: buffer + size_bytes: 1536 + permissions: read-only + c_parameter: const int16_t a[768] + description: Input polynomial vector a + x2: + type: buffer + size_bytes: 1536 + permissions: read-only + c_parameter: const int16_t b[768] + description: Input polynomial vector b + x3: + type: buffer + size_bytes: 768 + permissions: read-only + c_parameter: const int16_t b_cache[384] + description: Cached values for b + Stack: + bytes: 64 + description: saving callee-saved Neon registers +*/ + +/* Re-implementation of asymmetric base multiplication following @[NeonNTT] */ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_AARCH64) && !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 3) + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/aarch64_opt/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm) +MLK_ASM_FN_SYMBOL(polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x40 + .cfi_adjust_cfa_offset 0x40 + stp d8, d9, [sp] + .cfi_rel_offset d8, 0x0 + .cfi_rel_offset d9, 0x8 + stp d10, d11, [sp, #0x10] + .cfi_rel_offset d10, 0x10 + .cfi_rel_offset d11, 0x18 + stp d12, d13, [sp, #0x20] + .cfi_rel_offset d12, 0x20 + .cfi_rel_offset d13, 0x28 + stp d14, d15, [sp, #0x30] + .cfi_rel_offset d14, 0x30 + .cfi_rel_offset d15, 0x38 + mov w14, #0xd01 // =3329 + dup v0.8h, w14 + mov w14, #0xcff // =3327 + dup v2.8h, w14 + add x4, x1, #0x200 + add x5, x2, #0x200 + add x6, x3, #0x100 + add x7, x1, #0x400 + add x8, x2, #0x400 + add x9, x3, #0x200 + mov x13, #0x10 // =16 + ldr q6, [x7], #0x20 + ldr q19, [x2, #0x10] + ldr q23, [x1], #0x20 + ldur q14, [x1, #-0x10] + ldr q17, [x2], #0x20 + ldr q11, [x4, #0x10] + ldur q28, [x7, #-0x10] + ld1 { v30.8h }, [x3], #16 + ldr q26, [x4], #0x20 + ldr q16, [x8, #0x10] + uzp1 v8.8h, v23.8h, v14.8h + ldr q22, [x5, #0x10] + ldr q18, [x5], #0x20 + uzp1 v20.8h, v17.8h, v19.8h + uzp2 v24.8h, v23.8h, v14.8h + ldr q31, [x8], #0x20 + smull2 v4.4s, v8.8h, v20.8h + uzp1 v25.8h, v26.8h, v11.8h + smull v13.4s, v8.4h, v20.4h + ld1 { v23.8h }, [x6], #16 + uzp1 v1.8h, v18.8h, v22.8h + smlal v13.4s, v24.4h, v30.4h + smlal2 v4.4s, v24.8h, v30.8h + uzp2 v5.8h, v26.8h, v11.8h + smlal2 v4.4s, v25.8h, v1.8h + uzp1 v29.8h, v6.8h, v28.8h + smlal2 v4.4s, v5.8h, v23.8h + ld1 { v7.8h }, [x9], #16 + smlal v13.4s, v25.4h, v1.4h + uzp2 v17.8h, v17.8h, v19.8h + uzp1 v27.8h, v31.8h, v16.8h + smlal v13.4s, v5.4h, v23.4h + uzp2 v22.8h, v18.8h, v22.8h + smull v18.4s, v8.4h, v17.4h + uzp2 v28.8h, v6.8h, v28.8h + smlal v13.4s, v29.4h, v27.4h + smlal2 v4.4s, v29.8h, v27.8h + uzp2 v26.8h, v31.8h, v16.8h + smlal2 v4.4s, v28.8h, v7.8h + ldr q3, [x7, #0x10] + smlal v13.4s, v28.4h, v7.4h + ldr q7, [x1], #0x20 + smlal v18.4s, v24.4h, v20.4h + ldr q15, [x2], #0x20 + smlal v18.4s, v25.4h, v22.4h + smull2 v8.4s, v8.8h, v17.8h + ldur q17, [x1, #-0x10] + uzp1 v23.8h, v13.8h, v4.8h + smlal v18.4s, v5.4h, v1.4h + smlal2 v8.4s, v24.8h, v20.8h + ld1 { v16.8h }, [x3], #16 + mul v23.8h, v23.8h, v2.8h + ldr q19, [x5, #0x10] + ldr q14, [x4, #0x10] + ldr q11, [x4], #0x20 + ldur q20, [x2, #-0x10] + smlal2 v8.4s, v25.8h, v22.8h + smlal2 v8.4s, v5.8h, v1.8h + ldr q22, [x5], #0x20 + uzp1 v1.8h, v7.8h, v17.8h + smlal v18.4s, v29.4h, v26.4h + smlal v13.4s, v23.4h, v0.4h + uzp2 v31.8h, v11.8h, v14.8h + uzp1 v21.8h, v15.8h, v20.8h + smlal2 v4.4s, v23.8h, v0.8h + ld1 { v9.8h }, [x6], #16 + smlal v18.4s, v28.4h, v27.4h + smlal2 v8.4s, v29.8h, v26.8h + ldr q25, [x7], #0x20 + smull v26.4s, v1.4h, v21.4h + uzp1 v24.8h, v22.8h, v19.8h + smlal2 v8.4s, v28.8h, v27.8h + uzp2 v28.8h, v7.8h, v17.8h + uzp1 v29.8h, v11.8h, v14.8h + smull2 v23.4s, v1.8h, v21.8h + ldr q27, [x8], #0x20 + smlal2 v23.4s, v28.8h, v16.8h + ldur q11, [x8, #-0x10] + smlal2 v23.4s, v29.8h, v24.8h + uzp2 v7.8h, v13.8h, v4.8h + uzp2 v19.8h, v22.8h, v19.8h + ld1 { v4.8h }, [x9], #16 + smlal2 v23.4s, v31.8h, v9.8h + uzp1 v13.8h, v25.8h, v3.8h + uzp1 v14.8h, v18.8h, v8.8h + smlal v26.4s, v28.4h, v16.4h + uzp2 v17.8h, v27.8h, v11.8h + uzp2 v20.8h, v15.8h, v20.8h + mul v14.8h, v14.8h, v2.8h + sub x13, x13, #0x2 + +Lmlk_polyvec_basemul_acc_montgomery_cached_k3_loop_start: + uzp1 v6.8h, v27.8h, v11.8h + smlal v26.4s, v29.4h, v24.4h + uzp2 v16.8h, v25.8h, v3.8h + smlal v26.4s, v31.4h, v9.4h + ldr q3, [x7, #0x10] + smlal v26.4s, v13.4h, v6.4h + smlal2 v8.4s, v14.8h, v0.8h + ldr q27, [x8], #0x20 + smlal v18.4s, v14.4h, v0.4h + ldr q25, [x7], #0x20 + smlal2 v23.4s, v13.8h, v6.8h + ldr q11, [x1], #0x20 + smlal2 v23.4s, v16.8h, v4.8h + smlal v26.4s, v16.4h, v4.4h + ldur q22, [x1, #-0x10] + uzp2 v30.8h, v18.8h, v8.8h + smull v18.4s, v1.4h, v20.4h + smlal v18.4s, v28.4h, v21.4h + ldr q14, [x2], #0x20 + smlal v18.4s, v29.4h, v19.4h + zip1 v5.8h, v7.8h, v30.8h + uzp1 v4.8h, v26.8h, v23.8h + smull2 v8.4s, v1.8h, v20.8h + zip2 v10.8h, v7.8h, v30.8h + smlal v18.4s, v31.4h, v24.4h + mul v12.8h, v4.8h, v2.8h + ldr q4, [x5, #0x10] + ldr q20, [x4, #0x10] + ldr q1, [x4], #0x20 + ldur q30, [x2, #-0x10] + smlal2 v8.4s, v28.8h, v21.8h + smlal2 v8.4s, v29.8h, v19.8h + ldr q19, [x5], #0x20 + smlal2 v8.4s, v31.8h, v24.8h + ld1 { v15.8h }, [x3], #16 + uzp2 v31.8h, v1.8h, v20.8h + smlal v26.4s, v12.4h, v0.4h + smlal2 v23.4s, v12.8h, v0.8h + uzp1 v21.8h, v14.8h, v30.8h + uzp1 v29.8h, v1.8h, v20.8h + uzp1 v1.8h, v11.8h, v22.8h + smlal2 v8.4s, v13.8h, v17.8h + ld1 { v9.8h }, [x6], #16 + smlal v18.4s, v13.4h, v17.4h + uzp1 v24.8h, v19.8h, v4.8h + uzp2 v7.8h, v26.8h, v23.8h + smull v26.4s, v1.4h, v21.4h + smlal v18.4s, v16.4h, v6.4h + uzp2 v19.8h, v19.8h, v4.8h + smlal2 v8.4s, v16.8h, v6.8h + uzp2 v28.8h, v11.8h, v22.8h + smull2 v23.4s, v1.8h, v21.8h + uzp1 v13.8h, v25.8h, v3.8h + smlal2 v23.4s, v28.8h, v15.8h + ldur q11, [x8, #-0x10] + smlal2 v23.4s, v29.8h, v24.8h + ld1 { v4.8h }, [x9], #16 + smlal2 v23.4s, v31.8h, v9.8h + uzp1 v12.8h, v18.8h, v8.8h + uzp2 v20.8h, v14.8h, v30.8h + smlal v26.4s, v28.4h, v15.4h + str q5, [x0], #0x20 + mul v14.8h, v12.8h, v2.8h + stur q10, [x0, #-0x10] + uzp2 v17.8h, v27.8h, v11.8h + subs x13, x13, #0x1 + cbnz x13, Lmlk_polyvec_basemul_acc_montgomery_cached_k3_loop_start + uzp2 v3.8h, v25.8h, v3.8h + smull2 v16.4s, v1.8h, v20.8h + smull v25.4s, v1.4h, v20.4h + uzp1 v22.8h, v27.8h, v11.8h + smlal2 v16.4s, v28.8h, v21.8h + smlal v25.4s, v28.4h, v21.4h + smlal2 v16.4s, v29.8h, v19.8h + smlal v25.4s, v29.4h, v19.4h + smlal2 v16.4s, v31.8h, v24.8h + smlal v25.4s, v31.4h, v24.4h + smlal v25.4s, v13.4h, v17.4h + smlal2 v16.4s, v13.8h, v17.8h + smlal2 v16.4s, v3.8h, v22.8h + smlal v25.4s, v3.4h, v22.4h + smlal2 v23.4s, v13.8h, v22.8h + smlal v26.4s, v29.4h, v24.4h + smlal v26.4s, v31.4h, v9.4h + smlal v26.4s, v13.4h, v22.4h + uzp1 v10.8h, v25.8h, v16.8h + smlal2 v23.4s, v3.8h, v4.8h + smlal v26.4s, v3.4h, v4.4h + mul v13.8h, v10.8h, v2.8h + smlal v18.4s, v14.4h, v0.4h + smlal2 v8.4s, v14.8h, v0.8h + uzp1 v3.8h, v26.8h, v23.8h + mul v24.8h, v3.8h, v2.8h + uzp2 v17.8h, v18.8h, v8.8h + smlal v25.4s, v13.4h, v0.4h + smlal2 v16.4s, v13.8h, v0.8h + zip1 v21.8h, v7.8h, v17.8h + zip2 v20.8h, v7.8h, v17.8h + smlal2 v23.4s, v24.8h, v0.8h + str q21, [x0], #0x20 + smlal v26.4s, v24.4h, v0.4h + uzp2 v13.8h, v25.8h, v16.8h + stur q20, [x0, #-0x10] + uzp2 v23.8h, v26.8h, v23.8h + zip1 v18.8h, v23.8h, v13.8h + zip2 v13.8h, v23.8h, v13.8h + str q18, [x0], #0x20 + stur q13, [x0, #-0x10] + ldp d8, d9, [sp] + .cfi_restore d8 + .cfi_restore d9 + ldp d10, d11, [sp, #0x10] + .cfi_restore d10 + .cfi_restore d11 + ldp d12, d13, [sp, #0x20] + .cfi_restore d12 + .cfi_restore d13 + ldp d14, d15, [sp, #0x30] + .cfi_restore d14 + .cfi_restore d15 + add sp, sp, #0x40 + .cfi_adjust_cfa_offset -0x40 + ret + .cfi_endproc + +MLK_ASM_FN_SIZE(polyvec_basemul_acc_montgomery_cached_k3_aarch64_asm) + +#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 3) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm.S b/cbits/mlkem/src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/aarch64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm.S @@ -0,0 +1,371 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [NeonNTT] + * Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1 + * Becker, Hwang, Kannwischer, Yang, Yang + * https://eprint.iacr.org/2021/986 + */ + +/*yaml + Name: polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm + Description: Re-implementation of asymmetric base multiplication following @[NeonNTT] for k=4 + Signature: void mlk_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm(int16_t r[256], const int16_t a[1024], const int16_t b[1024], const int16_t b_cache[512]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 512 + permissions: write-only + c_parameter: int16_t r[256] + description: Output polynomial + x1: + type: buffer + size_bytes: 2048 + permissions: read-only + c_parameter: const int16_t a[1024] + description: Input polynomial vector a + x2: + type: buffer + size_bytes: 2048 + permissions: read-only + c_parameter: const int16_t b[1024] + description: Input polynomial vector b + x3: + type: buffer + size_bytes: 1024 + permissions: read-only + c_parameter: const int16_t b_cache[512] + description: Cached values for b + Stack: + bytes: 64 + description: saving callee-saved Neon registers +*/ + +/* Re-implementation of asymmetric base multiplication following @[NeonNTT] */ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_AARCH64) && !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4) + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/aarch64_opt/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm) +MLK_ASM_FN_SYMBOL(polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x40 + .cfi_adjust_cfa_offset 0x40 + stp d8, d9, [sp] + .cfi_rel_offset d8, 0x0 + .cfi_rel_offset d9, 0x8 + stp d10, d11, [sp, #0x10] + .cfi_rel_offset d10, 0x10 + .cfi_rel_offset d11, 0x18 + stp d12, d13, [sp, #0x20] + .cfi_rel_offset d12, 0x20 + .cfi_rel_offset d13, 0x28 + stp d14, d15, [sp, #0x30] + .cfi_rel_offset d14, 0x30 + .cfi_rel_offset d15, 0x38 + mov w14, #0xd01 // =3329 + dup v0.8h, w14 + mov w14, #0xcff // =3327 + dup v2.8h, w14 + add x4, x1, #0x200 + add x5, x2, #0x200 + add x6, x3, #0x100 + add x7, x1, #0x400 + add x8, x2, #0x400 + add x9, x3, #0x200 + add x10, x1, #0x600 + add x11, x2, #0x600 + add x12, x3, #0x300 + mov x13, #0x10 // =16 + ldr q28, [x1], #0x20 + ldur q5, [x1, #-0x10] + ldr q31, [x2], #0x20 + ldur q27, [x2, #-0x10] + ldr q7, [x5], #0x20 + ldr q10, [x4], #0x20 + ldur q18, [x5, #-0x10] + ldur q9, [x4, #-0x10] + uzp1 v11.8h, v28.8h, v5.8h + uzp2 v19.8h, v28.8h, v5.8h + uzp2 v4.8h, v31.8h, v27.8h + uzp1 v1.8h, v31.8h, v27.8h + ldr q29, [x7], #0x20 + ldr q28, [x8, #0x10] + uzp1 v24.8h, v10.8h, v9.8h + uzp1 v17.8h, v7.8h, v18.8h + uzp2 v7.8h, v7.8h, v18.8h + ldr q21, [x8], #0x20 + uzp2 v27.8h, v10.8h, v9.8h + ldur q6, [x7, #-0x10] + smull v18.4s, v11.4h, v4.4h + ld1 { v9.8h }, [x3], #16 + smull2 v8.4s, v11.8h, v4.8h + ldr q16, [x11], #0x20 + smlal2 v8.4s, v19.8h, v1.8h + ldur q14, [x11, #-0x10] + smlal v18.4s, v19.4h, v1.4h + uzp1 v10.8h, v21.8h, v28.8h + smlal v18.4s, v24.4h, v7.4h + ldr q4, [x10], #0x20 + smlal2 v8.4s, v24.8h, v7.8h + ld1 { v12.8h }, [x6], #16 + smull2 v23.4s, v11.8h, v1.8h + uzp2 v13.8h, v29.8h, v6.8h + smull v26.4s, v11.4h, v1.4h + uzp1 v29.8h, v29.8h, v6.8h + smlal v26.4s, v19.4h, v9.4h + ldur q15, [x10, #-0x10] + smlal2 v23.4s, v19.8h, v9.8h + uzp2 v9.8h, v21.8h, v28.8h + smlal v18.4s, v27.4h, v17.4h + uzp2 v6.8h, v16.8h, v14.8h + uzp1 v21.8h, v16.8h, v14.8h + smlal2 v8.4s, v27.8h, v17.8h + smlal2 v8.4s, v29.8h, v9.8h + uzp1 v30.8h, v4.8h, v15.8h + uzp2 v16.8h, v4.8h, v15.8h + smlal v18.4s, v29.4h, v9.4h + smlal2 v8.4s, v13.8h, v10.8h + ld1 { v15.8h }, [x9], #16 + smlal v18.4s, v13.4h, v10.4h + ldr q11, [x4], #0x20 + smlal v18.4s, v30.4h, v6.4h + ldr q7, [x2], #0x20 + smlal2 v8.4s, v30.8h, v6.8h + ld1 { v9.8h }, [x12], #16 + smlal2 v23.4s, v24.8h, v17.8h + ldur q4, [x2, #-0x10] + smlal v26.4s, v24.4h, v17.4h + ldur q25, [x4, #-0x10] + smlal2 v8.4s, v16.8h, v21.8h + ldr q5, [x5], #0x20 + smlal v18.4s, v16.4h, v21.4h + ldur q22, [x5, #-0x10] + smlal v26.4s, v27.4h, v12.4h + ldr q19, [x1, #0x10] + smlal v26.4s, v29.4h, v10.4h + ld1 { v20.8h }, [x3], #16 + smlal v26.4s, v13.4h, v15.4h + uzp1 v24.8h, v7.8h, v4.8h + smlal2 v23.4s, v27.8h, v12.8h + uzp1 v28.8h, v18.8h, v8.8h + smlal v26.4s, v30.4h, v21.4h + uzp2 v27.8h, v11.8h, v25.8h + smlal2 v23.4s, v29.8h, v10.8h + uzp2 v31.8h, v7.8h, v4.8h + smlal2 v23.4s, v13.8h, v15.8h + uzp1 v14.8h, v5.8h, v22.8h + uzp1 v17.8h, v11.8h, v25.8h + smlal v26.4s, v16.4h, v9.4h + mul v29.8h, v28.8h, v2.8h + sub x13, x13, #0x2 + +Lmlk_polyvec_basemul_acc_montgomery_cached_k4_loop_start: + smlal2 v23.4s, v30.8h, v21.8h + ldr q11, [x1], #0x20 + uzp2 v15.8h, v5.8h, v22.8h + smlal v18.4s, v29.4h, v0.4h + ldr q12, [x7], #0x20 + smlal2 v8.4s, v29.8h, v0.8h + ldur q3, [x7, #-0x10] + ldr q21, [x8], #0x20 + uzp1 v29.8h, v11.8h, v19.8h + ldur q13, [x8, #-0x10] + uzp2 v5.8h, v11.8h, v19.8h + smlal2 v23.4s, v16.8h, v9.8h + uzp2 v28.8h, v18.8h, v8.8h + smull2 v8.4s, v29.8h, v31.8h + smlal2 v8.4s, v5.8h, v24.8h + uzp1 v7.8h, v12.8h, v3.8h + smlal2 v8.4s, v17.8h, v15.8h + uzp2 v11.8h, v21.8h, v13.8h + uzp1 v4.8h, v26.8h, v23.8h + smlal2 v8.4s, v27.8h, v14.8h + smlal2 v8.4s, v7.8h, v11.8h + mul v6.8h, v4.8h, v2.8h + ldr q19, [x11], #0x20 + uzp2 v25.8h, v12.8h, v3.8h + ldr q12, [x10], #0x20 + smull v18.4s, v29.4h, v31.4h + ldur q3, [x10, #-0x10] + smlal v18.4s, v5.4h, v24.4h + uzp1 v4.8h, v21.8h, v13.8h + smlal v18.4s, v17.4h, v15.4h + ldur q13, [x11, #-0x10] + ld1 { v1.8h }, [x6], #16 + smlal v26.4s, v6.4h, v0.4h + smlal2 v23.4s, v6.8h, v0.8h + ld1 { v10.8h }, [x9], #16 + smlal v18.4s, v27.4h, v14.4h + uzp1 v30.8h, v12.8h, v3.8h + smlal2 v8.4s, v25.8h, v4.8h + uzp2 v31.8h, v19.8h, v13.8h + smlal v18.4s, v7.4h, v11.4h + ld1 { v9.8h }, [x12], #16 + smlal v18.4s, v25.4h, v4.4h + uzp1 v21.8h, v19.8h, v13.8h + uzp2 v16.8h, v12.8h, v3.8h + smlal v18.4s, v30.4h, v31.4h + smlal2 v8.4s, v30.8h, v31.8h + uzp2 v31.8h, v26.8h, v23.8h + smlal2 v8.4s, v16.8h, v21.8h + smlal v18.4s, v16.4h, v21.4h + zip1 v15.8h, v31.8h, v28.8h + ldr q19, [x1, #0x10] + smull2 v23.4s, v29.8h, v24.8h + smull v26.4s, v29.4h, v24.4h + ldr q3, [x2, #0x10] + smlal v26.4s, v5.4h, v20.4h + ldr q11, [x2], #0x20 + uzp1 v6.8h, v18.8h, v8.8h + smlal v26.4s, v17.4h, v14.4h + smlal v26.4s, v27.4h, v1.4h + zip2 v13.8h, v31.8h, v28.8h + smlal v26.4s, v7.4h, v4.4h + str q15, [x0], #0x20 + smlal v26.4s, v25.4h, v10.4h + stur q13, [x0, #-0x10] + mul v29.8h, v6.8h, v2.8h + uzp1 v24.8h, v11.8h, v3.8h + uzp2 v31.8h, v11.8h, v3.8h + ldr q11, [x4], #0x20 + smlal2 v23.4s, v5.8h, v20.8h + ldur q28, [x4, #-0x10] + smlal2 v23.4s, v17.8h, v14.8h + ldr q5, [x5], #0x20 + smlal2 v23.4s, v27.8h, v1.8h + ldur q22, [x5, #-0x10] + smlal v26.4s, v30.4h, v21.4h + ld1 { v20.8h }, [x3], #16 + smlal v26.4s, v16.4h, v9.4h + uzp1 v17.8h, v11.8h, v28.8h + smlal2 v23.4s, v7.8h, v4.8h + uzp2 v27.8h, v11.8h, v28.8h + smlal2 v23.4s, v25.8h, v10.8h + uzp1 v14.8h, v5.8h, v22.8h + subs x13, x13, #0x1 + cbnz x13, Lmlk_polyvec_basemul_acc_montgomery_cached_k4_loop_start + smlal v18.4s, v29.4h, v0.4h + ldr q11, [x1], #0x20 + uzp2 v28.8h, v5.8h, v22.8h + smlal2 v23.4s, v30.8h, v21.8h + smlal2 v8.4s, v29.8h, v0.8h + ldr q15, [x8, #0x10] + smlal2 v23.4s, v16.8h, v9.8h + ldr q21, [x8], #0x20 + uzp1 v22.8h, v11.8h, v19.8h + uzp2 v12.8h, v11.8h, v19.8h + ldr q1, [x7, #0x10] + ld1 { v6.8h }, [x6], #16 + uzp2 v3.8h, v18.8h, v8.8h + smull v9.4s, v22.4h, v31.4h + smull2 v18.4s, v22.8h, v31.8h + ldr q16, [x7], #0x20 + smull v19.4s, v22.4h, v24.4h + uzp1 v30.8h, v21.8h, v15.8h + uzp2 v25.8h, v21.8h, v15.8h + smull2 v8.4s, v22.8h, v24.8h + smlal v19.4s, v12.4h, v20.4h + ldr q13, [x10, #0x10] + smlal2 v8.4s, v12.8h, v20.8h + uzp1 v29.8h, v16.8h, v1.8h + smlal2 v18.4s, v12.8h, v24.8h + ldr q5, [x10], #0x20 + smlal v9.4s, v12.4h, v24.4h + ldr q4, [x11], #0x20 + smlal v9.4s, v17.4h, v28.4h + ldur q22, [x11, #-0x10] + smlal2 v18.4s, v17.8h, v28.8h + uzp2 v16.8h, v16.8h, v1.8h + smlal v19.4s, v17.4h, v14.4h + ld1 { v28.8h }, [x9], #16 + smlal2 v8.4s, v17.8h, v14.8h + uzp1 v7.8h, v5.8h, v13.8h + smlal v9.4s, v27.4h, v14.4h + uzp1 v17.8h, v4.8h, v22.8h + smlal2 v18.4s, v27.8h, v14.8h + uzp2 v12.8h, v5.8h, v13.8h + uzp2 v21.8h, v4.8h, v22.8h + smlal v19.4s, v27.4h, v6.4h + smlal2 v8.4s, v27.8h, v6.8h + ld1 { v15.8h }, [x12], #16 + smlal v19.4s, v29.4h, v30.4h + uzp1 v20.8h, v26.8h, v23.8h + smlal v9.4s, v29.4h, v25.4h + smlal2 v18.4s, v29.8h, v25.8h + smlal2 v8.4s, v29.8h, v30.8h + smlal v19.4s, v16.4h, v28.4h + smlal2 v8.4s, v16.8h, v28.8h + smlal2 v18.4s, v16.8h, v30.8h + smlal v9.4s, v16.4h, v30.4h + smlal v9.4s, v7.4h, v21.4h + smlal2 v18.4s, v7.8h, v21.8h + smlal2 v8.4s, v7.8h, v17.8h + smlal v19.4s, v7.4h, v17.4h + smlal v19.4s, v12.4h, v15.4h + smlal2 v8.4s, v12.8h, v15.8h + smlal2 v18.4s, v12.8h, v17.8h + smlal v9.4s, v12.4h, v17.4h + mul v6.8h, v20.8h, v2.8h + uzp1 v4.8h, v19.8h, v8.8h + mul v17.8h, v4.8h, v2.8h + uzp1 v12.8h, v9.8h, v18.8h + smlal v26.4s, v6.4h, v0.4h + mul v21.8h, v12.8h, v2.8h + smlal2 v23.4s, v6.8h, v0.8h + smlal2 v8.4s, v17.8h, v0.8h + smlal v19.4s, v17.4h, v0.4h + smlal2 v18.4s, v21.8h, v0.8h + uzp2 v23.8h, v26.8h, v23.8h + smlal v9.4s, v21.4h, v0.4h + zip2 v12.8h, v23.8h, v3.8h + zip1 v22.8h, v23.8h, v3.8h + uzp2 v14.8h, v19.8h, v8.8h + uzp2 v18.8h, v9.8h, v18.8h + str q12, [x0, #0x10] + str q22, [x0], #0x20 + zip2 v24.8h, v14.8h, v18.8h + zip1 v21.8h, v14.8h, v18.8h + str q24, [x0, #0x10] + str q21, [x0], #0x20 + ldp d8, d9, [sp] + .cfi_restore d8 + .cfi_restore d9 + ldp d10, d11, [sp, #0x10] + .cfi_restore d10 + .cfi_restore d11 + ldp d12, d13, [sp, #0x20] + .cfi_restore d12 + .cfi_restore d13 + ldp d14, d15, [sp, #0x30] + .cfi_restore d14 + .cfi_restore d15 + add sp, sp, #0x40 + .cfi_adjust_cfa_offset -0x40 + ret + .cfi_endproc + +MLK_ASM_FN_SIZE(polyvec_basemul_acc_montgomery_cached_k4_aarch64_asm) + +#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/aarch64/src/mlkem_rej_uniform_aarch64_asm.S b/cbits/mlkem/src/native/aarch64/src/mlkem_rej_uniform_aarch64_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/aarch64/src/mlkem_rej_uniform_aarch64_asm.S @@ -0,0 +1,226 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/*yaml + Name: rej_uniform_aarch64_asm + Description: Run rejection sampling on uniform random bytes to generate uniform random integers mod q + Signature: uint64_t mlk_rej_uniform_aarch64_asm(int16_t r[256], const uint8_t *buf, unsigned buflen, const uint8_t table[4096]) + ABI: + Architecture: aarch64 + CallingConvention: AAPCS64 + Features: [NEON] + x0: + type: buffer + size_bytes: 512 + permissions: write-only + c_parameter: int16_t r[256] + description: Output buffer + x1: + type: buffer + size_bytes: x2 + permissions: read-only + c_parameter: const uint8_t *buf + description: Input buffer + x2: + type: scalar + c_parameter: unsigned buflen + description: Length of input buffer (must be multiple of 24) + test_with: 504 # MLKEM_GEN_MATRIX_NBLOCKS * MLK_XOF_RATE + x3: + type: buffer + size_bytes: 4096 + permissions: read-only + c_parameter: const uint8_t table[4096] + description: Lookup table + Stack: + bytes: 576 + description: register preservation and temporary storage +*/ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_AARCH64) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/aarch64_opt/src/mlkem_rej_uniform_aarch64_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(rej_uniform_aarch64_asm) +MLK_ASM_FN_SYMBOL(rej_uniform_aarch64_asm) + + .cfi_startproc + sub sp, sp, #0x240 + .cfi_adjust_cfa_offset 0x240 + mov x7, #0x1 // =1 + movk x7, #0x2, lsl #16 + movk x7, #0x4, lsl #32 + movk x7, #0x8, lsl #48 + mov v31.d[0], x7 + mov x7, #0x10 // =16 + movk x7, #0x20, lsl #16 + movk x7, #0x40, lsl #32 + movk x7, #0x80, lsl #48 + mov v31.d[1], x7 + mov w11, #0xd01 // =3329 + dup v30.8h, w11 + mov x8, sp + mov x7, x8 + mov x11, #0x0 // =0 + eor v16.16b, v16.16b, v16.16b + +Lmlk_rej_uniform_initial_zero: + str q16, [x7], #0x40 + stur q16, [x7, #-0x30] + stur q16, [x7, #-0x20] + stur q16, [x7, #-0x10] + add x11, x11, #0x20 + cmp x11, #0x100 + b.lt Lmlk_rej_uniform_initial_zero + mov x7, x8 + mov x9, #0x0 // =0 + mov x4, #0x100 // =256 + cmp x2, #0x30 + b.lo Lmlk_rej_uniform_loop48_end + +Lmlk_rej_uniform_loop48: + cmp x9, x4 + b.hs Lmlk_rej_uniform_memory_copy + sub x2, x2, #0x30 + ld3 { v0.16b, v1.16b, v2.16b }, [x1], #48 + zip1 v4.16b, v0.16b, v1.16b + zip2 v5.16b, v0.16b, v1.16b + zip1 v6.16b, v1.16b, v2.16b + zip2 v7.16b, v1.16b, v2.16b + bic v4.8h, #0xf0, lsl #8 + bic v5.8h, #0xf0, lsl #8 + ushr v6.8h, v6.8h, #0x4 + ushr v7.8h, v7.8h, #0x4 + zip1 v16.8h, v4.8h, v6.8h + zip2 v17.8h, v4.8h, v6.8h + zip1 v18.8h, v5.8h, v7.8h + zip2 v19.8h, v5.8h, v7.8h + cmhi v4.8h, v30.8h, v16.8h + cmhi v5.8h, v30.8h, v17.8h + cmhi v6.8h, v30.8h, v18.8h + cmhi v7.8h, v30.8h, v19.8h + and v4.16b, v4.16b, v31.16b + and v5.16b, v5.16b, v31.16b + and v6.16b, v6.16b, v31.16b + and v7.16b, v7.16b, v31.16b + uaddlv s20, v4.8h + uaddlv s21, v5.8h + uaddlv s22, v6.8h + uaddlv s23, v7.8h + fmov w12, s20 + fmov w13, s21 + fmov w14, s22 + fmov w15, s23 + ldr q24, [x3, x12, lsl #4] + ldr q25, [x3, x13, lsl #4] + ldr q26, [x3, x14, lsl #4] + ldr q27, [x3, x15, lsl #4] + cnt v4.16b, v4.16b + cnt v5.16b, v5.16b + cnt v6.16b, v6.16b + cnt v7.16b, v7.16b + uaddlv s20, v4.8h + uaddlv s21, v5.8h + uaddlv s22, v6.8h + uaddlv s23, v7.8h + fmov w12, s20 + fmov w13, s21 + fmov w14, s22 + fmov w15, s23 + tbl v16.16b, { v16.16b }, v24.16b + tbl v17.16b, { v17.16b }, v25.16b + tbl v18.16b, { v18.16b }, v26.16b + tbl v19.16b, { v19.16b }, v27.16b + st1 { v16.8h }, [x7] + add x7, x7, x12, lsl #1 + st1 { v17.8h }, [x7] + add x7, x7, x13, lsl #1 + st1 { v18.8h }, [x7] + add x7, x7, x14, lsl #1 + st1 { v19.8h }, [x7] + add x7, x7, x15, lsl #1 + add x12, x12, x13 + add x14, x14, x15 + add x9, x9, x12 + add x9, x9, x14 + cmp x2, #0x30 + b.hs Lmlk_rej_uniform_loop48 + +Lmlk_rej_uniform_loop48_end: + cmp x9, x4 + b.hs Lmlk_rej_uniform_memory_copy + cmp x2, #0x18 + b.lo Lmlk_rej_uniform_memory_copy + ld3 { v0.8b, v1.8b, v2.8b }, [x1], #24 + zip1 v4.16b, v0.16b, v1.16b + zip1 v5.16b, v1.16b, v2.16b + bic v4.8h, #0xf0, lsl #8 + ushr v5.8h, v5.8h, #0x4 + zip1 v16.8h, v4.8h, v5.8h + zip2 v17.8h, v4.8h, v5.8h + cmhi v4.8h, v30.8h, v16.8h + cmhi v5.8h, v30.8h, v17.8h + and v4.16b, v4.16b, v31.16b + and v5.16b, v5.16b, v31.16b + uaddlv s20, v4.8h + uaddlv s21, v5.8h + fmov w12, s20 + fmov w13, s21 + ldr q24, [x3, x12, lsl #4] + ldr q25, [x3, x13, lsl #4] + cnt v4.16b, v4.16b + cnt v5.16b, v5.16b + uaddlv s20, v4.8h + uaddlv s21, v5.8h + fmov w12, s20 + fmov w13, s21 + tbl v16.16b, { v16.16b }, v24.16b + tbl v17.16b, { v17.16b }, v25.16b + st1 { v16.8h }, [x7] + add x7, x7, x12, lsl #1 + st1 { v17.8h }, [x7] + add x9, x9, x12 + add x9, x9, x13 + +Lmlk_rej_uniform_memory_copy: + cmp x9, x4 + csel x9, x9, x4, lo + mov x11, #0x0 // =0 + mov x7, x8 + +Lmlk_rej_uniform_final_copy: + ldr q16, [x7], #0x40 + ldur q17, [x7, #-0x30] + ldur q18, [x7, #-0x20] + ldur q19, [x7, #-0x10] + str q16, [x0], #0x40 + stur q17, [x0, #-0x30] + stur q18, [x0, #-0x20] + stur q19, [x0, #-0x10] + add x11, x11, #0x20 + cmp x11, #0x100 + b.lt Lmlk_rej_uniform_final_copy + mov x0, x9 + +Lmlk_rej_uniform_return: + add sp, sp, #0x240 + .cfi_adjust_cfa_offset -0x240 + ret + .cfi_endproc + +MLK_ASM_FN_SIZE(rej_uniform_aarch64_asm) + +#endif /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/aarch64/src/rej_uniform_table.c b/cbits/mlkem/src/native/aarch64/src/rej_uniform_table.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/aarch64/src/rej_uniform_table.c @@ -0,0 +1,543 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mlkem-native repository. + * Do not modify it directly. + */ + +#include "../../../common.h" + +#if defined(MLK_ARITH_BACKEND_AARCH64) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +#include "arith_native_aarch64.h" + +/* + * Lookup table used by rejection sampling of the public matrix. + * See autogen for details. + */ +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t + mlk_rej_uniform_table[4096] = { + 255, 255, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 0 */, + 0, 1, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 1 */, + 2, 3, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 2 */, + 0, 1, 2, 3, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 3 */, + 4, 5, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 4 */, + 0, 1, 4, 5, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 5 */, + 2, 3, 4, 5, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 6 */, + 0, 1, 2, 3, 4, 5, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 7 */, + 6, 7, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 8 */, + 0, 1, 6, 7, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 9 */, + 2, 3, 6, 7, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 10 */, + 0, 1, 2, 3, 6, 7, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 11 */, + 4, 5, 6, 7, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 12 */, + 0, 1, 4, 5, 6, 7, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 13 */, + 2, 3, 4, 5, 6, 7, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 14 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 255, 255, 255, 255, 255, 255, 255, 255 /* 15 */, + 8, 9, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 16 */, + 0, 1, 8, 9, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 17 */, + 2, 3, 8, 9, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 18 */, + 0, 1, 2, 3, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 19 */, + 4, 5, 8, 9, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 20 */, + 0, 1, 4, 5, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 21 */, + 2, 3, 4, 5, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 22 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 255, 255, 255, 255, 255, 255, 255, 255 /* 23 */, + 6, 7, 8, 9, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 24 */, + 0, 1, 6, 7, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 25 */, + 2, 3, 6, 7, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 26 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 255, 255, 255, 255, 255, 255, 255, 255 /* 27 */, + 4, 5, 6, 7, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 28 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 255, 255, 255, 255, 255, 255, 255, 255 /* 29 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 255, 255, 255, 255, 255, 255, 255, 255 /* 30 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 255, 255, 255, 255, 255, 255 /* 31 */, + 10, 11, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 32 */, + 0, 1, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 33 */, + 2, 3, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 34 */, + 0, 1, 2, 3, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 35 */, + 4, 5, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 36 */, + 0, 1, 4, 5, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 37 */, + 2, 3, 4, 5, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 38 */, + 0, 1, 2, 3, 4, 5, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 39 */, + 6, 7, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 40 */, + 0, 1, 6, 7, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 41 */, + 2, 3, 6, 7, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 42 */, + 0, 1, 2, 3, 6, 7, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 43 */, + 4, 5, 6, 7, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 44 */, + 0, 1, 4, 5, 6, 7, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 45 */, + 2, 3, 4, 5, 6, 7, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 46 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 10, 11, 255, 255, 255, 255, 255, 255 /* 47 */, + 8, 9, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 48 */, + 0, 1, 8, 9, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 49 */, + 2, 3, 8, 9, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 50 */, + 0, 1, 2, 3, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 51 */, + 4, 5, 8, 9, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 52 */, + 0, 1, 4, 5, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 53 */, + 2, 3, 4, 5, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 54 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 10, 11, 255, 255, 255, 255, 255, 255 /* 55 */, + 6, 7, 8, 9, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 56 */, + 0, 1, 6, 7, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 57 */, + 2, 3, 6, 7, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 58 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 10, 11, 255, 255, 255, 255, 255, 255 /* 59 */, + 4, 5, 6, 7, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 60 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 10, 11, 255, 255, 255, 255, 255, 255 /* 61 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 10, 11, 255, 255, 255, 255, 255, 255 /* 62 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 255, 255, 255, 255 /* 63 */, + 12, 13, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 64 */, + 0, 1, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 65 */, + 2, 3, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 66 */, + 0, 1, 2, 3, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 67 */, + 4, 5, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 68 */, + 0, 1, 4, 5, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 69 */, + 2, 3, 4, 5, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 70 */, + 0, 1, 2, 3, 4, 5, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 71 */, + 6, 7, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 72 */, + 0, 1, 6, 7, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 73 */, + 2, 3, 6, 7, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 74 */, + 0, 1, 2, 3, 6, 7, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 75 */, + 4, 5, 6, 7, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 76 */, + 0, 1, 4, 5, 6, 7, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 77 */, + 2, 3, 4, 5, 6, 7, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 78 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 12, 13, 255, 255, 255, 255, 255, 255 /* 79 */, + 8, 9, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 80 */, + 0, 1, 8, 9, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 81 */, + 2, 3, 8, 9, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 82 */, + 0, 1, 2, 3, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 83 */, + 4, 5, 8, 9, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 84 */, + 0, 1, 4, 5, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 85 */, + 2, 3, 4, 5, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 86 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 12, 13, 255, 255, 255, 255, 255, 255 /* 87 */, + 6, 7, 8, 9, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 88 */, + 0, 1, 6, 7, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 89 */, + 2, 3, 6, 7, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 90 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 12, 13, 255, 255, 255, 255, 255, 255 /* 91 */, + 4, 5, 6, 7, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 92 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 12, 13, 255, 255, 255, 255, 255, 255 /* 93 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 12, 13, 255, 255, 255, 255, 255, 255 /* 94 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 12, 13, 255, 255, 255, 255 /* 95 */, + 10, 11, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 96 */, + 0, 1, 10, 11, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 97 */, + 2, 3, 10, 11, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 98 */, + 0, 1, 2, 3, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 99 */, + 4, 5, 10, 11, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 100 */, + 0, 1, 4, 5, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 101 */, + 2, 3, 4, 5, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 102 */, + 0, 1, 2, 3, 4, 5, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 103 */, + 6, 7, 10, 11, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 104 */, + 0, 1, 6, 7, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 105 */, + 2, 3, 6, 7, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 106 */, + 0, 1, 2, 3, 6, 7, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 107 */, + 4, 5, 6, 7, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 108 */, + 0, 1, 4, 5, 6, 7, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 109 */, + 2, 3, 4, 5, 6, 7, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 110 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 10, 11, 12, 13, 255, 255, 255, 255 /* 111 */, + 8, 9, 10, 11, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 112 */, + 0, 1, 8, 9, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 113 */, + 2, 3, 8, 9, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 114 */, + 0, 1, 2, 3, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 115 */, + 4, 5, 8, 9, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 116 */, + 0, 1, 4, 5, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 117 */, + 2, 3, 4, 5, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 118 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 10, 11, 12, 13, 255, 255, 255, 255 /* 119 */, + 6, 7, 8, 9, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 120 */, + 0, 1, 6, 7, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 121 */, + 2, 3, 6, 7, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 122 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 10, 11, 12, 13, 255, 255, 255, 255 /* 123 */, + 4, 5, 6, 7, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 124 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 10, 11, 12, 13, 255, 255, 255, 255 /* 125 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 10, 11, 12, 13, 255, 255, 255, 255 /* 126 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 12, 13, 255, 255 /* 127 */, + 14, 15, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 128 */, + 0, 1, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 129 */, + 2, 3, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 130 */, + 0, 1, 2, 3, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 131 */, + 4, 5, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 132 */, + 0, 1, 4, 5, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 133 */, + 2, 3, 4, 5, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 134 */, + 0, 1, 2, 3, 4, 5, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 135 */, + 6, 7, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 136 */, + 0, 1, 6, 7, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 137 */, + 2, 3, 6, 7, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 138 */, + 0, 1, 2, 3, 6, 7, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 139 */, + 4, 5, 6, 7, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 140 */, + 0, 1, 4, 5, 6, 7, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 141 */, + 2, 3, 4, 5, 6, 7, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 142 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 14, 15, 255, 255, 255, 255, 255, 255 /* 143 */, + 8, 9, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 144 */, + 0, 1, 8, 9, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 145 */, + 2, 3, 8, 9, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 146 */, + 0, 1, 2, 3, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 147 */, + 4, 5, 8, 9, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 148 */, + 0, 1, 4, 5, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 149 */, + 2, 3, 4, 5, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 150 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 14, 15, 255, 255, 255, 255, 255, 255 /* 151 */, + 6, 7, 8, 9, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 152 */, + 0, 1, 6, 7, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 153 */, + 2, 3, 6, 7, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 154 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 14, 15, 255, 255, 255, 255, 255, 255 /* 155 */, + 4, 5, 6, 7, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 156 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 14, 15, 255, 255, 255, 255, 255, 255 /* 157 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 14, 15, 255, 255, 255, 255, 255, 255 /* 158 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 14, 15, 255, 255, 255, 255 /* 159 */, + 10, 11, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 160 */, + 0, 1, 10, 11, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 161 */, + 2, 3, 10, 11, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 162 */, + 0, 1, 2, 3, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 163 */, + 4, 5, 10, 11, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 164 */, + 0, 1, 4, 5, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 165 */, + 2, 3, 4, 5, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 166 */, + 0, 1, 2, 3, 4, 5, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 167 */, + 6, 7, 10, 11, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 168 */, + 0, 1, 6, 7, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 169 */, + 2, 3, 6, 7, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 170 */, + 0, 1, 2, 3, 6, 7, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 171 */, + 4, 5, 6, 7, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 172 */, + 0, 1, 4, 5, 6, 7, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 173 */, + 2, 3, 4, 5, 6, 7, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 174 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 10, 11, 14, 15, 255, 255, 255, 255 /* 175 */, + 8, 9, 10, 11, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 176 */, + 0, 1, 8, 9, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 177 */, + 2, 3, 8, 9, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 178 */, + 0, 1, 2, 3, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 179 */, + 4, 5, 8, 9, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 180 */, + 0, 1, 4, 5, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 181 */, + 2, 3, 4, 5, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 182 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 10, 11, 14, 15, 255, 255, 255, 255 /* 183 */, + 6, 7, 8, 9, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 184 */, + 0, 1, 6, 7, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 185 */, + 2, 3, 6, 7, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 186 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 10, 11, 14, 15, 255, 255, 255, 255 /* 187 */, + 4, 5, 6, 7, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 188 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 10, 11, 14, 15, 255, 255, 255, 255 /* 189 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 10, 11, 14, 15, 255, 255, 255, 255 /* 190 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 14, 15, 255, 255 /* 191 */, + 12, 13, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 192 */, + 0, 1, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 193 */, + 2, 3, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 194 */, + 0, 1, 2, 3, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 195 */, + 4, 5, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 196 */, + 0, 1, 4, 5, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 197 */, + 2, 3, 4, 5, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 198 */, + 0, 1, 2, 3, 4, 5, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 199 */, + 6, 7, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 200 */, + 0, 1, 6, 7, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 201 */, + 2, 3, 6, 7, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 202 */, + 0, 1, 2, 3, 6, 7, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 203 */, + 4, 5, 6, 7, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 204 */, + 0, 1, 4, 5, 6, 7, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 205 */, + 2, 3, 4, 5, 6, 7, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 206 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 12, 13, 14, 15, 255, 255, 255, 255 /* 207 */, + 8, 9, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 208 */, + 0, 1, 8, 9, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 209 */, + 2, 3, 8, 9, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 210 */, + 0, 1, 2, 3, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 211 */, + 4, 5, 8, 9, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 212 */, + 0, 1, 4, 5, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 213 */, + 2, 3, 4, 5, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 214 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 12, 13, 14, 15, 255, 255, 255, 255 /* 215 */, + 6, 7, 8, 9, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 216 */, + 0, 1, 6, 7, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 217 */, + 2, 3, 6, 7, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 218 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 12, 13, 14, 15, 255, 255, 255, 255 /* 219 */, + 4, 5, 6, 7, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 220 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 12, 13, 14, 15, 255, 255, 255, 255 /* 221 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 12, 13, 14, 15, 255, 255, 255, 255 /* 222 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 12, 13, 14, 15, 255, 255 /* 223 */, + 10, 11, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 224 */, + 0, 1, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 225 */, + 2, 3, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 226 */, + 0, 1, 2, 3, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 227 */, + 4, 5, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 228 */, + 0, 1, 4, 5, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 229 */, + 2, 3, 4, 5, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 230 */, + 0, 1, 2, 3, 4, 5, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 231 */, + 6, 7, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 232 */, + 0, 1, 6, 7, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 233 */, + 2, 3, 6, 7, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 234 */, + 0, 1, 2, 3, 6, 7, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 235 */, + 4, 5, 6, 7, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 236 */, + 0, 1, 4, 5, 6, 7, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 237 */, + 2, 3, 4, 5, 6, 7, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 238 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 10, 11, 12, 13, 14, 15, 255, 255 /* 239 */, + 8, 9, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 240 */, + 0, 1, 8, 9, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 241 */, + 2, 3, 8, 9, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 242 */, + 0, 1, 2, 3, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 243 */, + 4, 5, 8, 9, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 244 */, + 0, 1, 4, 5, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 245 */, + 2, 3, 4, 5, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 246 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 10, 11, 12, 13, 14, 15, 255, 255 /* 247 */, + 6, 7, 8, 9, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 248 */, + 0, 1, 6, 7, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 249 */, + 2, 3, 6, 7, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 250 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 10, 11, 12, 13, 14, 15, 255, 255 /* 251 */, + 4, 5, 6, 7, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 252 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 10, 11, 12, 13, 14, 15, 255, 255 /* 253 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 10, 11, 12, 13, 14, 15, 255, 255 /* 254 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 12, 13, 14, 15 /* 255 */, +}; + +#else /* MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +MLK_EMPTY_CU(aarch64_rej_uniform_table) + +#endif /* !(MLK_ARITH_BACKEND_AARCH64 && !MLK_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mlkem/src/native/api.h b/cbits/mlkem/src/native/api.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/api.h @@ -0,0 +1,651 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + */ + +#ifndef MLK_NATIVE_API_H +#define MLK_NATIVE_API_H +/* + * Native arithmetic interface + * + * This header is primarily for documentation purposes. + * It should not be included by backend implementations. + * + * To ensure consistency with backends, the header will be + * included automatically after inclusion of the active + * backend, to ensure consistency of function signatures, + * and run sanity checks. + */ + +#include "../cbmc.h" +#include "../common.h" + +/* Backends must return MLK_NATIVE_FUNC_SUCCESS upon success. */ +#define MLK_NATIVE_FUNC_SUCCESS (0) +/* Backends may return MLK_NATIVE_FUNC_FALLBACK to signal to the frontend that + * the target/parameters are unsupported; typically, this would be because of + * dependencies on CPU features not detected on the host CPU. In this case, + * the frontend falls back to the default C implementation. */ +#define MLK_NATIVE_FUNC_FALLBACK (-1) + + +/* Absolute exclusive upper bound for the output of the inverse NTT + * + * NOTE: This is the same bound as in poly.h and has to be kept + * in sync. */ +#define MLK_INVNTT_BOUND (8 * MLKEM_Q) + +/* Absolute exclusive upper bound for the output of the forward NTT + * + * NOTE: This is the same bound as in poly.h and has to be kept + * in sync. */ +#define MLK_NTT_BOUND (8 * MLKEM_Q) + +/* + * This is the C<->native interface allowing for the drop-in of + * native code for performance critical arithmetic components of ML-KEM. + * + * A _backend_ is a specific implementation of (part of) this interface. + * + * To add a function to a backend, define MLK_USE_NATIVE_XXX and + * implement `static inline xxx(...)` in the profile header. + * + * The only exception is MLK_USE_NATIVE_NTT_CUSTOM_ORDER. This option can + * be set if there are native implementations for all of NTT, invNTT, and + * base multiplication, and allows the native implementation to use a + * custom order of polynomial coefficients in NTT domain -- the use of such + * custom order is not an implementation-detail since the public matrix + * is generated in NTT domain. In this case, a permutation function + * mlk_poly_permute_bitrev_to_custom() needs to be provided that permutes + * polynomials in NTT domain from bitreversed to the custom order. + */ + +/* + * Those functions are meant to be trivial wrappers around the chosen native + * implementation. The are static inline to avoid unnecessary calls. + * The macro before each declaration controls whether a native + * implementation is present. + */ + +#if defined(MLK_USE_NATIVE_NTT) +/** + * Compute the negacyclic number-theoretic transform (NTT) of a polynomial + * in place. + * + * The input polynomial is assumed to be in normal order. The output + * polynomial is in bitreversed order, or of a custom order if + * MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set. See the documentation of + * MLK_USE_NATIVE_NTT_CUSTOM_ORDER for more information. + * + * @param[in,out] p Input/output polynomial. + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_ntt_native(int16_t p[MLKEM_N]) +__contract__( + requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N)) + requires(array_abs_bound(p, 0, MLKEM_N, MLKEM_Q)) + assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N)) + ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(p, 0, MLKEM_N, MLK_NTT_BOUND)) + ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_abs_bound(p, 0, MLKEM_N, MLKEM_Q)) + ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_unchanged(p, MLKEM_N)) +); +#endif /* MLK_USE_NATIVE_NTT */ + +#if defined(MLK_USE_NATIVE_NTT_CUSTOM_ORDER) +/* + * This must only be set if NTT, invNTT, basemul, mulcache, and + * to/from byte stream conversions all have native implementations + * that are adapted to the custom order. + */ +#if !defined(MLK_USE_NATIVE_NTT) || !defined(MLK_USE_NATIVE_INTT) || \ + !defined(MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE) || \ + !defined(MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED) || \ + !defined(MLK_USE_NATIVE_POLY_TOBYTES) || \ + !defined(MLK_USE_NATIVE_POLY_FROMBYTES) +#error \ + "Invalid native profile: MLK_USE_NATIVE_NTT_CUSTOM_ORDER can only be \ +set if there are native implementations for NTT, invNTT, mulcache, basemul, \ +and to/from bytes conversions." +#endif /* !MLK_USE_NATIVE_NTT || !MLK_USE_NATIVE_INTT || \ + !MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE || \ + !MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED || \ + !MLK_USE_NATIVE_POLY_TOBYTES || !MLK_USE_NATIVE_POLY_FROMBYTES */ + +/** + * When MLK_USE_NATIVE_NTT_CUSTOM_ORDER is defined, convert a polynomial in + * NTT domain from bitreversed order to the custom order output by the + * native NTT. + * + * This must only be defined if there is native code for all of (a) NTT, + * (b) invNTT, (c) basemul, (d) mulcache. + * + * @param[in,out] p Input/output polynomial. + */ +static MLK_INLINE void mlk_poly_permute_bitrev_to_custom(int16_t p[MLKEM_N]) +__contract__( + /* We don't specify that this should be a permutation, but only + * that it does not change the bound established at the end of mlk_gen_matrix. */ + requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N)) + requires(array_bound(p, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N)) + ensures(array_bound(p, 0, MLKEM_N, 0, MLKEM_Q))); +#endif /* MLK_USE_NATIVE_NTT_CUSTOM_ORDER */ + +#if defined(MLK_USE_NATIVE_INTT) && \ + (!defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)) +/** + * Compute the inverse negacyclic number-theoretic transform (NTT) of a + * polynomial in place. + * + * The input polynomial is in bitreversed order, or of a custom order if + * MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set. See the documentation of + * MLK_USE_NATIVE_NTT_CUSTOM_ORDER for more information. The output + * polynomial is assumed to be in normal order. + * + * @param[in,out] p Input/output polynomial. + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_intt_native(int16_t p[MLKEM_N]) +__contract__( + requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N)) + assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N)) + ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(p, 0, MLKEM_N, MLK_INVNTT_BOUND)) + ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_unchanged(p, MLKEM_N)) +); +#endif /* MLK_USE_NATIVE_INTT && (!MLK_CONFIG_NO_ENCAPS_API || \ + !MLK_CONFIG_NO_DECAPS_API) */ + +#if defined(MLK_USE_NATIVE_POLY_REDUCE) +/** + * Apply modular reduction to all coefficients of a polynomial, mapping them + * to unsigned canonical representatives in [0,..,MLKEM_Q-1]. + * + * @param[in,out] p Input/output polynomial. + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_reduce_native(int16_t p[MLKEM_N]) +__contract__( + requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N)) + assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N)) + ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_bound(p, 0, MLKEM_N, 0, MLKEM_Q)) + ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_unchanged(p, MLKEM_N)) +); +#endif /* MLK_USE_NATIVE_POLY_REDUCE */ + +#if defined(MLK_USE_NATIVE_POLY_TOMONT) && !defined(MLK_CONFIG_NO_KEYPAIR_API) +/** + * In-place conversion of all coefficients of a polynomial from the normal + * domain to the Montgomery domain. + * + * @param[in,out] p Input/output polynomial. + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_tomont_native(int16_t p[MLKEM_N]) +__contract__( + requires(memory_no_alias(p, sizeof(int16_t) * MLKEM_N)) + assigns(memory_slice(p, sizeof(int16_t) * MLKEM_N)) + ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(p, 0, MLKEM_N, MLKEM_Q)) + ensures((return_value == MLK_NATIVE_FUNC_FALLBACK) ==> array_unchanged(p, MLKEM_N)) +); +#endif /* MLK_USE_NATIVE_POLY_TOMONT && !MLK_CONFIG_NO_KEYPAIR_API */ + +#if defined(MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE) +/** + * Compute multiplication cache for a polynomial in NTT domain. + * + * The purpose of the multiplication cache is to cache repeated computations + * required during a base multiplication of polynomials in NTT domain. The + * structure of the multiplication-cache is implementation defined. + * + * @param[out] cache Multiplication cache. + * @param[in] mlk_poly Input polynomial. Must be in NTT domain and in + * bitreversed order, or of a custom order if + * MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set. See the + * documentation of MLK_USE_NATIVE_NTT_CUSTOM_ORDER for + * more information. + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_mulcache_compute_native( + int16_t cache[MLKEM_N / 2], const int16_t mlk_poly[MLKEM_N]) +__contract__( + requires(memory_no_alias(cache, sizeof(int16_t) * (MLKEM_N / 2))) + requires(memory_no_alias(mlk_poly, sizeof(int16_t) * MLKEM_N)) + assigns(memory_slice(cache, sizeof(int16_t) * (MLKEM_N / 2))) + ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS) + ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_abs_bound(cache, 0, MLKEM_N/2, MLKEM_Q)) +); +#endif /* MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE */ + +#if defined(MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED) +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 +/** + * Compute scalar product of length-2 polynomial vectors in NTT domain. + * + * @param[out] r Result of the scalar product. Again in NTT domain, of + * the same ordering as @p a and @p b. + * @param[in] a First polynomial vector operand. Must be in NTT + * domain and in bitreversed order, or of a custom order + * if MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set. See the + * documentation of MLK_USE_NATIVE_NTT_CUSTOM_ORDER for + * more information. + * @param[in] b Second polynomial vector operand. As for @p a. + * @param[in] b_cache Multiplication-cache for @p b. + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k2_native( + int16_t r[MLKEM_N], const int16_t a[2 * MLKEM_N], + const int16_t b[2 * MLKEM_N], const int16_t b_cache[2 * (MLKEM_N / 2)]) +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, sizeof(int16_t) * 2 * MLKEM_N)) + requires(memory_no_alias(b, sizeof(int16_t) * 2 * MLKEM_N)) + requires(memory_no_alias(b_cache, sizeof(int16_t) * 2 * (MLKEM_N / 2))) + requires(array_bound(a, 0, 2 * MLKEM_N, 0, MLKEM_UINT12_LIMIT)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS) +); +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 */ + +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 3 +/** + * Compute scalar product of length-3 polynomial vectors in NTT domain. + * + * @param[out] r Result of the scalar product. Again in NTT domain, of + * the same ordering as @p a and @p b. + * @param[in] a First polynomial vector operand. Must be in NTT + * domain and in bitreversed order, or of a custom order + * if MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set. See the + * documentation of MLK_USE_NATIVE_NTT_CUSTOM_ORDER for + * more information. + * @param[in] b Second polynomial vector operand. As for @p a. + * @param[in] b_cache Multiplication-cache for @p b. + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k3_native( + int16_t r[MLKEM_N], const int16_t a[3 * MLKEM_N], + const int16_t b[3 * MLKEM_N], const int16_t b_cache[3 * (MLKEM_N / 2)]) +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, sizeof(int16_t) * 3 * MLKEM_N)) + requires(memory_no_alias(b, sizeof(int16_t) * 3 * MLKEM_N)) + requires(memory_no_alias(b_cache, sizeof(int16_t) * 3 * (MLKEM_N / 2))) + requires(array_bound(a, 0, 3 * MLKEM_N, 0, MLKEM_UINT12_LIMIT)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS) +); +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 3 */ + +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4 +/** + * Compute scalar product of length-4 polynomial vectors in NTT domain. + * + * @param[out] r Result of the scalar product. Again in NTT domain, of + * the same ordering as @p a and @p b. + * @param[in] a First polynomial vector operand. Must be in NTT + * domain and in bitreversed order, or of a custom order + * if MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set. See the + * documentation of MLK_USE_NATIVE_NTT_CUSTOM_ORDER for + * more information. + * @param[in] b Second polynomial vector operand. As for @p a. + * @param[in] b_cache Multiplication-cache for @p b. + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k4_native( + int16_t r[MLKEM_N], const int16_t a[4 * MLKEM_N], + const int16_t b[4 * MLKEM_N], const int16_t b_cache[4 * (MLKEM_N / 2)]) +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, sizeof(int16_t) * 4 * MLKEM_N)) + requires(memory_no_alias(b, sizeof(int16_t) * 4 * MLKEM_N)) + requires(memory_no_alias(b_cache, sizeof(int16_t) * 4 * (MLKEM_N / 2))) + requires(array_bound(a, 0, 4 * MLKEM_N, 0, MLKEM_UINT12_LIMIT)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(return_value == MLK_NATIVE_FUNC_FALLBACK || return_value == MLK_NATIVE_FUNC_SUCCESS) +); +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4 */ +#endif /* MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED */ + +#if defined(MLK_USE_NATIVE_POLY_TOBYTES) && \ + (!defined(MLK_CONFIG_NO_KEYPAIR_API) || \ + !defined(MLK_CONFIG_NO_ENCAPS_API)) +/** + * Serialization of a polynomial with unsigned canonical coefficients. + * + * @spec{Implements ByteEncode_12 from @[FIPS203, Algorithm 5].} + * + * @param[out] r Output byte array (of MLKEM_POLYBYTES bytes). + * @param[in] a Input polynomial, with each coefficient in [0,..,MLKEM_Q-1]. + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_tobytes_native(uint8_t r[MLKEM_POLYBYTES], + const int16_t a[MLKEM_N]) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYBYTES)) + requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N)) + requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYBYTES)) + ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK) +); +#endif /* MLK_USE_NATIVE_POLY_TOBYTES && (!MLK_CONFIG_NO_KEYPAIR_API || \ + !MLK_CONFIG_NO_ENCAPS_API) */ + +#if defined(MLK_USE_NATIVE_POLY_FROMBYTES) && \ + (!defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)) +/** + * Deserialization of a polynomial. + * + * @spec{Implements ByteDecode_12 from @[FIPS203, Algorithm 6].} + * + * @param[out] a Output polynomial in NTT domain. + * @param[in] r Input byte array (of MLKEM_POLYBYTES bytes). + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_frombytes_native( + int16_t a[MLKEM_N], const uint8_t r[MLKEM_POLYBYTES]) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYBYTES)) + requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N)) + assigns(memory_slice(a, sizeof(int16_t) * MLKEM_N)) + ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK) + ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_bound(a, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT)) +); +#endif /* MLK_USE_NATIVE_POLY_FROMBYTES && (!MLK_CONFIG_NO_ENCAPS_API || \ + !MLK_CONFIG_NO_DECAPS_API) */ + +#if defined(MLK_USE_NATIVE_REJ_UNIFORM) +/** + * Run rejection sampling on uniformly random bytes to generate uniformly random + * integers mod MLKEM_Q, represented in [0,..,MLKEM_Q-1]. + * + * @param[out] r Output buffer. + * @param len Requested number of 16-bit integers (uniform mod MLKEM_Q). + * @param[in] buf Input buffer (assumed to be uniform random bytes). + * @param buflen Length of input buffer in bytes. + * + * @retval MLK_NATIVE_FUNC_FALLBACK Native implementation does not support + * the input lengths. + * @retval other Non-negative number of sampled 16-bit + * integers (at most @p len). + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_rej_uniform_native(int16_t *r, unsigned len, + const uint8_t *buf, + unsigned buflen) +__contract__( + requires(len <= 4096 && buflen <= 4096 && buflen % 3 == 0) + requires(memory_no_alias(r, sizeof(int16_t) * len)) + requires(memory_no_alias(buf, buflen)) + assigns(memory_slice(r, sizeof(int16_t) * len)) + ensures(return_value != MLK_NATIVE_FUNC_FALLBACK + ==> (0 <= return_value && return_value <= len)) + ensures(return_value != MLK_NATIVE_FUNC_FALLBACK + ==> array_bound(r, 0, (unsigned) return_value, 0, MLKEM_Q)) +); +#endif /* MLK_USE_NATIVE_REJ_UNIFORM */ + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || (MLKEM_K == 2 || MLKEM_K == 3) +#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D4) +/** + * Compression (4 bits) and subsequent serialization of a polynomial. + * + * @spec{Compress_4 from @[FIPS203, Eq (4.7)].} + * + * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D4 + * bytes). + * @param[in] a Input polynomial. Coefficients must be unsigned canonical, + * i.e. in [0,1,..,MLKEM_Q-1]. + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_compress_d4_native( + uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D4], const int16_t a[MLKEM_N]) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D4)) + requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N)) + requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D4)) + ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK)); +#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D4 */ + +#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D10) +/** + * Compression (10 bits) and subsequent serialization of a polynomial. + * + * @spec{Compress_10 from @[FIPS203, Eq (4.7)].} + * + * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D10 + * bytes). + * @param[in] a Input polynomial. Coefficients must be unsigned canonical, + * i.e. in [0,1,..,MLKEM_Q-1]. + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_compress_d10_native( + uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D10], const int16_t a[MLKEM_N]) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D10)) + requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N)) + requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D10)) + ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK)); +#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D10 */ + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D4) +/** + * De-serialization and subsequent decompression (4 bits) of a polynomial; + * approximate inverse of mlk_poly_compress_d4. + * + * @spec{Decompress_4 from @[FIPS203, Eq (4.8)].} + * + * Upon return, the coefficients of the output polynomial are + * unsigned-canonical (non-negative and smaller than MLKEM_Q). + * + * @param[out] r Output polynomial. + * @param[in] a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D4 + * bytes). + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_decompress_d4_native( + int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D4]) +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D4)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK) + ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_bound(r, 0, MLKEM_N, 0, MLKEM_Q))); +#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D4 */ + +#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D10) +/** + * De-serialization and subsequent decompression (10 bits) of a polynomial; + * approximate inverse of mlk_poly_compress_d10. + * + * @spec{Decompress_10 from @[FIPS203, Eq (4.8)].} + * + * Upon return, the coefficients of the output polynomial are + * unsigned-canonical (non-negative and smaller than MLKEM_Q). + * + * @param[out] r Output polynomial. + * @param[in] a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D10 + * bytes). + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_decompress_d10_native( + int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D10]) +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D10)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK) + ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_bound(r, 0, MLKEM_N, 0, MLKEM_Q))); +#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D10 */ +#endif /* !MLK_CONFIG_NO_DECAPS_API */ +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 || MLKEM_K == 3 */ + +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4 +#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D5) +/** + * Compression (5 bits) and subsequent serialization of a polynomial. + * + * @spec{Compress_5 from @[FIPS203, Eq (4.7)].} + * + * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D5 + * bytes). + * @param[in] a Input polynomial. Coefficients must be unsigned canonical, + * i.e. in [0,1,..,MLKEM_Q-1]. + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_compress_d5_native( + uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D5], const int16_t a[MLKEM_N]) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D5)) + requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N)) + requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D5)) + ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK)); +#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D5 */ + +#if defined(MLK_USE_NATIVE_POLY_COMPRESS_D11) +/** + * Compression (11 bits) and subsequent serialization of a polynomial. + * + * @spec{Compress_11 from @[FIPS203, Eq (4.7)].} + * + * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D11 + * bytes). + * @param[in] a Input polynomial. Coefficients must be unsigned canonical, + * i.e. in [0,1,..,MLKEM_Q-1]. + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_compress_d11_native( + uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D11], const int16_t a[MLKEM_N]) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D11)) + requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N)) + requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D11)) + ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK)); +#endif /* MLK_USE_NATIVE_POLY_COMPRESS_D11 */ + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D5) +/** + * De-serialization and subsequent decompression (5 bits) of a polynomial; + * approximate inverse of mlk_poly_compress_d5. + * + * @spec{Decompress_5 from @[FIPS203, Eq (4.8)].} + * + * Upon return, the coefficients of the output polynomial are + * unsigned-canonical (non-negative and smaller than MLKEM_Q). + * + * @param[out] r Output polynomial. + * @param[in] a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D5 + * bytes). + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_decompress_d5_native( + int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D5]) +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D5)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK) + ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_bound(r, 0, MLKEM_N, 0, MLKEM_Q))); +#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D5 */ + +#if defined(MLK_USE_NATIVE_POLY_DECOMPRESS_D11) +/** + * De-serialization and subsequent decompression (11 bits) of a polynomial; + * approximate inverse of mlk_poly_compress_d11. + * + * @spec{Decompress_11 from @[FIPS203, Eq (4.8)].} + * + * Upon return, the coefficients of the output polynomial are + * unsigned-canonical (non-negative and smaller than MLKEM_Q). + * + * @param[out] r Output polynomial. + * @param[in] a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_D11 + * bytes). + * + * @retval MLK_NATIVE_FUNC_SUCCESS Operation succeeded. + * @retval MLK_NATIVE_FUNC_FALLBACK Backend declined; caller should fall back. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_decompress_d11_native( + int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D11]) +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D11)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(return_value == MLK_NATIVE_FUNC_SUCCESS || return_value == MLK_NATIVE_FUNC_FALLBACK) + ensures((return_value == MLK_NATIVE_FUNC_SUCCESS) ==> array_bound(r, 0, MLKEM_N, 0, MLKEM_Q))); +#endif /* MLK_USE_NATIVE_POLY_DECOMPRESS_D11 */ +#endif /* !MLK_CONFIG_NO_DECAPS_API */ +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4 */ +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +#endif /* !MLK_NATIVE_API_H */ diff --git a/cbits/mlkem/src/native/meta.h b/cbits/mlkem/src/native/meta.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/meta.h @@ -0,0 +1,30 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLK_NATIVE_META_H +#define MLK_NATIVE_META_H + +/* + * Default arithmetic backend + */ +#include "../sys.h" + +#ifdef MLK_SYS_AARCH64_NEON +#include "aarch64/meta.h" +#endif + +/* The x86_64 backend requires toolchain support for the SysV ABI */ +#if defined(MLK_SYS_X86_64_AVX2) && defined(MLK_SYSV_ABI_SUPPORTED) +#include "x86_64/meta.h" +#endif + +#if defined(MLK_SYS_RISCV64_RVV) +#include "riscv64/meta.h" +#endif + +#ifdef MLK_SYS_PPC64LE +#include "ppc64le/meta.h" +#endif + +#endif /* !MLK_NATIVE_META_H */ diff --git a/cbits/mlkem/src/native/x86_64/README.md b/cbits/mlkem/src/native/x86_64/README.md new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/README.md @@ -0,0 +1,4 @@ +[//]: # (SPDX-License-Identifier: CC-BY-4.0) + +This directory contains the native x86_64 arithmetic backend for ML-KEM provided by the official [AVX2 +implementation](https://github.com/pq-crystals/kyber/tree/main/avx2) of the Kyber team. diff --git a/cbits/mlkem/src/native/x86_64/meta.h b/cbits/mlkem/src/native/x86_64/meta.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/meta.h @@ -0,0 +1,324 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#ifndef MLK_NATIVE_X86_64_META_H +#define MLK_NATIVE_X86_64_META_H + +/* Identifier for this backend so that source and assembly files + * in the build can be appropriately guarded. */ +#define MLK_ARITH_BACKEND_X86_64_DEFAULT + +#define MLK_USE_NATIVE_NTT_CUSTOM_ORDER +#define MLK_USE_NATIVE_REJ_UNIFORM +#define MLK_USE_NATIVE_NTT +#define MLK_USE_NATIVE_INTT +#define MLK_USE_NATIVE_POLY_REDUCE +#define MLK_USE_NATIVE_POLY_TOMONT +#define MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED +#define MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE +#define MLK_USE_NATIVE_POLY_TOBYTES +#define MLK_USE_NATIVE_POLY_FROMBYTES +#define MLK_USE_NATIVE_POLY_COMPRESS_D4 +#define MLK_USE_NATIVE_POLY_COMPRESS_D5 +#define MLK_USE_NATIVE_POLY_COMPRESS_D10 +#define MLK_USE_NATIVE_POLY_COMPRESS_D11 +#define MLK_USE_NATIVE_POLY_DECOMPRESS_D4 +#define MLK_USE_NATIVE_POLY_DECOMPRESS_D5 +#define MLK_USE_NATIVE_POLY_DECOMPRESS_D10 +#define MLK_USE_NATIVE_POLY_DECOMPRESS_D11 + +#if !defined(__ASSEMBLER__) +#include "../../common.h" +#include "../api.h" +#include "src/arith_native_x86_64.h" +#include "src/compress_consts.h" + +static MLK_INLINE void mlk_poly_permute_bitrev_to_custom(int16_t data[MLKEM_N]) +{ + if (mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + mlk_nttunpack_avx2_asm(data); + } +} + +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_rej_uniform_native(int16_t *r, unsigned len, + const uint8_t *buf, + unsigned buflen) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2) || len != MLKEM_N || + buflen % 12 != 0) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + return (int)mlk_rej_uniform_avx2_asm(r, buf, buflen, mlk_rej_uniform_table); +} + +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_ntt_native(int16_t data[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_ntt_avx2_asm(data, mlk_qdata); + return MLK_NATIVE_FUNC_SUCCESS; +} + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_intt_native(int16_t data[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_invntt_avx2_asm(data, mlk_qdata); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_reduce_native(int16_t data[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_reduce_avx2_asm(data); + return MLK_NATIVE_FUNC_SUCCESS; +} + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_tomont_native(int16_t data[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_tomont_avx2_asm(data); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* !MLK_CONFIG_NO_KEYPAIR_API */ + +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_mulcache_compute_native(int16_t x[MLKEM_N / 2], + const int16_t y[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_poly_mulcache_compute_avx2_asm(x, y, mlk_qdata); + return MLK_NATIVE_FUNC_SUCCESS; +} + +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k2_native( + int16_t r[MLKEM_N], const int16_t a[2 * MLKEM_N], + const int16_t b[2 * MLKEM_N], const int16_t b_cache[2 * (MLKEM_N / 2)]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm(r, a, b, b_cache); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 */ + +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 3 +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k3_native( + int16_t r[MLKEM_N], const int16_t a[3 * MLKEM_N], + const int16_t b[3 * MLKEM_N], const int16_t b_cache[3 * (MLKEM_N / 2)]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm(r, a, b, b_cache); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 3 */ + +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4 +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_polyvec_basemul_acc_montgomery_cached_k4_native( + int16_t r[MLKEM_N], const int16_t a[4 * MLKEM_N], + const int16_t b[4 * MLKEM_N], const int16_t b_cache[4 * (MLKEM_N / 2)]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm(r, a, b, b_cache); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4 */ + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) || !defined(MLK_CONFIG_NO_ENCAPS_API) +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_tobytes_native(uint8_t r[MLKEM_POLYBYTES], + const int16_t a[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_ntttobytes_avx2_asm(r, a); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* !MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API */ + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_frombytes_native( + int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYBYTES]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_nttfrombytes_avx2_asm(r, a); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +#if (!defined(MLK_CONFIG_NO_ENCAPS_API) || \ + !defined(MLK_CONFIG_NO_DECAPS_API)) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 || \ + MLKEM_K == 3) +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_compress_d4_native( + uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D4], const int16_t a[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_poly_compress_d4_avx2_asm(r, a, mlk_compress_d4_data); + return MLK_NATIVE_FUNC_SUCCESS; +} + +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_compress_d10_native( + uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D10], const int16_t a[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_poly_compress_d10_avx2_asm(r, a, mlk_compress_d10_data); + return MLK_NATIVE_FUNC_SUCCESS; +} + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_decompress_d4_native( + int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D4]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_poly_decompress_d4_avx2_asm(r, a, mlk_decompress_d4_data); + return MLK_NATIVE_FUNC_SUCCESS; +} + +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_decompress_d10_native( + int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D10]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_poly_decompress_d10_avx2_asm(r, a, mlk_decompress_d10_data); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* !MLK_CONFIG_NO_DECAPS_API */ +#endif /* (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) && \ + (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 || MLKEM_K == 3) \ + */ + +#if (!defined(MLK_CONFIG_NO_ENCAPS_API) || \ + !defined(MLK_CONFIG_NO_DECAPS_API)) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4) +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_compress_d5_native( + uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D5], const int16_t a[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_poly_compress_d5_avx2_asm(r, a, mlk_compress_d5_data); + return MLK_NATIVE_FUNC_SUCCESS; +} + +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_compress_d11_native( + uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D11], const int16_t a[MLKEM_N]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_poly_compress_d11_avx2_asm(r, a, mlk_compress_d11_data); + return MLK_NATIVE_FUNC_SUCCESS; +} + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_decompress_d5_native( + int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D5]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_poly_decompress_d5_avx2_asm(r, a, mlk_decompress_d5_data); + return MLK_NATIVE_FUNC_SUCCESS; +} + +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_poly_decompress_d11_native( + int16_t r[MLKEM_N], const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D11]) +{ + if (!mlk_sys_check_capability(MLK_SYS_CAP_X86_64_AVX2)) + { + return MLK_NATIVE_FUNC_FALLBACK; + } + + mlk_poly_decompress_d11_avx2_asm(r, a, mlk_decompress_d11_data); + return MLK_NATIVE_FUNC_SUCCESS; +} +#endif /* !MLK_CONFIG_NO_DECAPS_API */ +#endif /* (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) && \ + (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4) */ + +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLK_NATIVE_X86_64_META_H */ diff --git a/cbits/mlkem/src/native/x86_64/src/arith_native_x86_64.h b/cbits/mlkem/src/native/x86_64/src/arith_native_x86_64.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/arith_native_x86_64.h @@ -0,0 +1,327 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLK_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H +#define MLK_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H + +#include "../../../common.h" + +#include +#include "compress_consts.h" +#include "consts.h" + +#define MLK_AVX2_REJ_UNIFORM_BUFLEN \ + (3 * 168) /* REJ_UNIFORM_NBLOCKS * SHAKE128_RATE */ + +#define mlk_rej_uniform_table MLK_NAMESPACE(rej_uniform_table) +MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_rej_uniform_table[4096]; + +#define mlk_rej_uniform_avx2_asm MLK_NAMESPACE(rej_uniform_avx2_asm) +MLK_MUST_CHECK_RETURN_VALUE MLK_SYSV_ABI +uint64_t mlk_rej_uniform_avx2_asm(int16_t *r, const uint8_t *buf, + unsigned buflen, const uint8_t *table) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mlkem_rej_uniform_avx2_asm.ml. */ +__contract__( + requires(buflen % 12 == 0) + requires(memory_no_alias(buf, buflen)) + requires(table == mlk_rej_uniform_table) + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(return_value <= MLKEM_N) + ensures(array_bound(r, 0, (unsigned) return_value, 0, MLKEM_Q)) +); + +#define mlk_ntt_avx2_asm MLK_NAMESPACE(ntt_avx2_asm) +MLK_SYSV_ABI +void mlk_ntt_avx2_asm(int16_t *r, const int16_t *qdata) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mlkem_ntt_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(array_abs_bound(r, 0, MLKEM_N, 8192)) + requires(qdata == mlk_qdata) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + /* check-magic: off */ + ensures(array_abs_bound(r, 0, MLKEM_N, 23595)) + /* check-magic: on */ +); + +#define mlk_invntt_avx2_asm MLK_NAMESPACE(invntt_avx2_asm) +MLK_SYSV_ABI +void mlk_invntt_avx2_asm(int16_t *r, const int16_t *qdata) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mlkem_intt_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(qdata == mlk_qdata) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + /* check-magic: off */ + ensures(array_abs_bound(r, 0, MLKEM_N, 26632)) + /* check-magic: on */ +); + +#define mlk_nttunpack_avx2_asm MLK_NAMESPACE(nttunpack_avx2_asm) +MLK_SYSV_ABI +void mlk_nttunpack_avx2_asm(int16_t *r) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mlkem_nttunpack_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(array_bound(r, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + /* Output is a permutation of input: every output coefficient + * is some input coefficient */ + ensures(forall(i, 0, MLKEM_N, exists(j, 0, MLKEM_N, + r[i] == old(*(int16_t (*)[MLKEM_N])r)[j]))) +); + +#define mlk_reduce_avx2_asm MLK_NAMESPACE(reduce_avx2_asm) +MLK_SYSV_ABI +void mlk_reduce_avx2_asm(int16_t *r) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mlkem_reduce_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(array_bound(r, 0, MLKEM_N, 0, MLKEM_Q)) +); + +#define mlk_poly_mulcache_compute_avx2_asm \ + MLK_NAMESPACE(poly_mulcache_compute_avx2_asm) +MLK_SYSV_ABI +void mlk_poly_mulcache_compute_avx2_asm(int16_t *out, const int16_t *in, + const int16_t *qdata) +/* This must be kept in sync with the HOL-Light specification + * in proofs/hol_light/x86_64/proofs/mlkem_poly_mulcache_compute_avx2_asm.ml */ +__contract__( + requires(memory_no_alias(out, sizeof(int16_t) * (MLKEM_N / 2))) + requires(memory_no_alias(in, sizeof(int16_t) * MLKEM_N)) + requires(qdata == mlk_qdata) + assigns(memory_slice(out, sizeof(int16_t) * (MLKEM_N / 2))) + ensures(array_abs_bound(out, 0, MLKEM_N/2, MLKEM_Q)) +); + +#define mlk_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm \ + MLK_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k2_avx2_asm) +MLK_SYSV_ABI +void mlk_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm( + int16_t *r, const int16_t *a, const int16_t *b, const int16_t *b_cache) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/x86_64/proofs/mlkem_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, sizeof(int16_t) * 2 * MLKEM_N)) + requires(memory_no_alias(b, sizeof(int16_t) * 2 * MLKEM_N)) + requires(memory_no_alias(b_cache, sizeof(int16_t) * 2 * (MLKEM_N / 2))) + requires(array_abs_bound(a, 0, 2 * MLKEM_N, MLKEM_UINT12_LIMIT + 1)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) +); + +#define mlk_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm \ + MLK_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k3_avx2_asm) +MLK_SYSV_ABI +void mlk_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm( + int16_t *r, const int16_t *a, const int16_t *b, const int16_t *b_cache) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/x86_64/proofs/mlkem_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, sizeof(int16_t) * 3 * MLKEM_N)) + requires(memory_no_alias(b, sizeof(int16_t) * 3 * MLKEM_N)) + requires(memory_no_alias(b_cache, sizeof(int16_t) * 3 * (MLKEM_N / 2))) + requires(array_abs_bound(a, 0, 3 * MLKEM_N, MLKEM_UINT12_LIMIT + 1)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) +); + +#define mlk_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm \ + MLK_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k4_avx2_asm) +MLK_SYSV_ABI +void mlk_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm( + int16_t *r, const int16_t *a, const int16_t *b, const int16_t *b_cache) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/x86_64/proofs/mlkem_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, sizeof(int16_t) * 4 * MLKEM_N)) + requires(memory_no_alias(b, sizeof(int16_t) * 4 * MLKEM_N)) + requires(memory_no_alias(b_cache, sizeof(int16_t) * 4 * (MLKEM_N / 2))) + requires(array_abs_bound(a, 0, 4 * MLKEM_N, MLKEM_UINT12_LIMIT + 1)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) +); + +#define mlk_ntttobytes_avx2_asm MLK_NAMESPACE(ntttobytes_avx2_asm) +MLK_SYSV_ABI +void mlk_ntttobytes_avx2_asm(uint8_t *r, const int16_t *a) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/x86_64/proofs/mlkem_ntttobytes_avx2_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, MLKEM_POLYBYTES)) + requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N)) + requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYBYTES)) +); + +#define mlk_nttfrombytes_avx2_asm MLK_NAMESPACE(nttfrombytes_avx2_asm) +MLK_SYSV_ABI +void mlk_nttfrombytes_avx2_asm(int16_t *r, const uint8_t *a) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/x86_64/proofs/mlkem_nttfrombytes_avx2_asm.ml. + */ +__contract__( + requires(memory_no_alias(a, MLKEM_POLYBYTES)) + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(array_bound(r, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT)) +); + +#define mlk_tomont_avx2_asm MLK_NAMESPACE(tomont_avx2_asm) +MLK_SYSV_ABI +void mlk_tomont_avx2_asm(int16_t *r) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/x86_64/proofs/mlkem_tomont_avx2_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(array_abs_bound(r, 0, MLKEM_N, MLKEM_Q)) +); + +#define mlk_poly_compress_d4_avx2_asm MLK_NAMESPACE(poly_compress_d4_avx2_asm) +MLK_SYSV_ABI +void mlk_poly_compress_d4_avx2_asm(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D4], + const int16_t *MLK_RESTRICT a, + const uint8_t *data) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/x86_64/proofs/mlkem_poly_compress_d4_avx2_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D4)) + requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N)) + requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q)) + requires(data == mlk_compress_d4_data) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D4)) +); + +#define mlk_poly_decompress_d4_avx2_asm \ + MLK_NAMESPACE(poly_decompress_d4_avx2_asm) +MLK_SYSV_ABI +void mlk_poly_decompress_d4_avx2_asm( + int16_t *MLK_RESTRICT r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D4], + const uint8_t *data) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/x86_64/proofs/mlkem_poly_decompress_d4_avx2_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D4)) + requires(data == mlk_decompress_d4_data) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(array_bound(r, 0, MLKEM_N, 0, MLKEM_Q)) +); + +#define mlk_poly_compress_d10_avx2_asm MLK_NAMESPACE(poly_compress_d10_avx2_asm) +MLK_SYSV_ABI +void mlk_poly_compress_d10_avx2_asm(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D10], + const int16_t *MLK_RESTRICT a, + const uint8_t *data) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/x86_64/proofs/mlkem_poly_compress_d10_avx2_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D10)) + requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N)) + requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q)) + requires(data == mlk_compress_d10_data) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D10)) +); + +#define mlk_poly_decompress_d10_avx2_asm \ + MLK_NAMESPACE(poly_decompress_d10_avx2_asm) +MLK_SYSV_ABI +void mlk_poly_decompress_d10_avx2_asm( + int16_t *MLK_RESTRICT r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D10], + const uint8_t *data) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/x86_64/proofs/mlkem_poly_decompress_d10_avx2_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D10)) + requires(data == mlk_decompress_d10_data) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(array_bound(r, 0, MLKEM_N, 0, MLKEM_Q)) +); + +#define mlk_poly_compress_d5_avx2_asm MLK_NAMESPACE(poly_compress_d5_avx2_asm) +MLK_SYSV_ABI +void mlk_poly_compress_d5_avx2_asm(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D5], + const int16_t *MLK_RESTRICT a, + const uint8_t *data) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/x86_64/proofs/mlkem_poly_compress_d5_avx2_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D5)) + requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N)) + requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q)) + requires(data == mlk_compress_d5_data) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D5)) +); + +#define mlk_poly_decompress_d5_avx2_asm \ + MLK_NAMESPACE(poly_decompress_d5_avx2_asm) +MLK_SYSV_ABI +void mlk_poly_decompress_d5_avx2_asm( + int16_t *MLK_RESTRICT r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D5], + const uint8_t *data) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/x86_64/proofs/mlkem_poly_decompress_d5_avx2_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D5)) + requires(data == mlk_decompress_d5_data) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(array_bound(r, 0, MLKEM_N, 0, MLKEM_Q)) +); + +#define mlk_poly_compress_d11_avx2_asm MLK_NAMESPACE(poly_compress_d11_avx2_asm) +MLK_SYSV_ABI +void mlk_poly_compress_d11_avx2_asm(uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_D11], + const int16_t *MLK_RESTRICT a, + const uint8_t *data) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/x86_64/proofs/mlkem_poly_compress_d11_avx2_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_D11)) + requires(memory_no_alias(a, sizeof(int16_t) * MLKEM_N)) + requires(array_bound(a, 0, MLKEM_N, 0, MLKEM_Q)) + requires(data == mlk_compress_d11_data) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_D11)) +); + +#define mlk_poly_decompress_d11_avx2_asm \ + MLK_NAMESPACE(poly_decompress_d11_avx2_asm) +MLK_SYSV_ABI +void mlk_poly_decompress_d11_avx2_asm( + int16_t *MLK_RESTRICT r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_D11], + const uint8_t *data) +/* This must be kept in sync with the HOL-Light specification in + * proofs/hol_light/x86_64/proofs/mlkem_poly_decompress_d11_avx2_asm.ml. + */ +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_D11)) + requires(data == mlk_decompress_d11_data) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(array_bound(r, 0, MLKEM_N, 0, MLKEM_Q)) +); + +#endif /* !MLK_NATIVE_X86_64_SRC_ARITH_NATIVE_X86_64_H */ diff --git a/cbits/mlkem/src/native/x86_64/src/compress_consts.c b/cbits/mlkem/src/native/x86_64/src/compress_consts.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/compress_consts.c @@ -0,0 +1,115 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mlkem-native repository. + * Do not modify it directly. + */ + +#include "../../../common.h" + +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + (!defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)) + +#include "compress_consts.h" + +#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 || \ + MLKEM_K == 3) + +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t + mlk_compress_d4_data[32] = { + 0, 0, 0, 0, 4, 0, 0, 0, 1, 0, 0, 0, 5, 0, 0, 0, + 2, 0, 0, 0, 6, 0, 0, 0, 3, 0, 0, 0, 7, 0, 0, 0, /* permdidx */ +}; + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t + mlk_decompress_d4_data[32] = { + 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, + 4, 4, 4, 4, 5, 5, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7, /* shufbidx */ +}; +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t + mlk_compress_d10_data[32] = { + 0, 1, 2, 3, 4, 8, 9, 10, 11, 12, 255, + 255, 255, 255, 255, 255, 9, 10, 11, 12, 255, 255, + 255, 255, 255, 255, 0, 1, 2, 3, 4, 8, /* shufbidx */ +}; + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t + mlk_decompress_d10_data[32] = { + 0, 1, 1, 2, 2, 3, 3, 4, 5, 6, 6, 7, 7, 8, 8, 9, + 2, 3, 3, 4, 4, 5, 5, 6, 7, 8, 8, 9, 9, 10, 10, 11, /* shufbidx */ +}; +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +#endif /* !MLK_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 || MLKEM_K == 3) \ + */ + +#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4) + +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t + mlk_compress_d5_data[32] = { + 0, 1, 2, 3, 4, 255, 255, 255, 255, 255, 8, + 9, 10, 11, 12, 255, 9, 10, 11, 12, 255, 0, + 1, 2, 3, 4, 255, 255, 255, 255, 255, 8, /* shufbidx */ +}; + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +/* shufbidx[0:32], mask[32:64], shift[64:96] */ +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t + mlk_decompress_d5_data[96] = { + 0, 0, 0, 1, 1, 1, 1, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5, + 5, 6, 6, 6, 6, 7, 7, 8, 8, 8, 8, 9, 9, 9, /* shufbidx */ + 31, 0, 224, 3, 124, 0, 128, 15, 240, 1, 62, 0, 192, 7, 248, 0, 31, 0, + 224, 3, 124, 0, 128, 15, 240, 1, 62, 0, 192, 7, 248, 0, /* mask */ + 0, 4, 32, 0, 0, 1, 8, 0, 64, 0, 0, 2, 16, 0, 128, 0, 0, 4, + 32, 0, 0, 1, 8, 0, 64, 0, 0, 2, 16, 0, 128, 0, /* shift */ +}; +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +/* srlvqidx[0:32], shufbidx[32:64] */ +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t + mlk_compress_d11_data[64] = + { + 10, 0, 0, 0, 0, 0, 0, 0, 30, 0, 0, + 0, 0, 0, 0, 0, 10, 0, 0, 0, 0, 0, + 0, 0, 30, 0, 0, 0, 0, 0, 0, 0, /* srlvqidx */ + 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, + 255, 255, 255, 255, 255, 5, 6, 7, 8, 9, 10, + 255, 255, 255, 255, 0, 0, 1, 2, 3, 4, /* shufbidx */ +}; + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +/* shufbidx[0:32], srlvdidx[32:64], srlvqidx[64:96], shift[96:128] */ +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t + mlk_decompress_d11_data[128] = { + 0, 1, 1, 2, 2, 3, 4, 5, 5, 6, 6, 7, 8, 9, 9, 10, + 3, 4, 4, 5, 5, 6, 7, 8, 8, 9, 9, 10, 11, 12, 12, 13, /* shufbidx */ + 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* srlvdidx */ + 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, /* srlvqidx */ + 32, 0, 4, 0, 1, 0, 32, 0, 8, 0, 1, 0, 32, 0, 4, 0, + 32, 0, 4, 0, 1, 0, 32, 0, 8, 0, 1, 0, 32, 0, 4, 0, /* shift */ +}; +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +#endif /* !MLK_CONFIG_MULTILEVEL_NO_SHARED && \ + (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4) */ + +#else /* MLK_ARITH_BACKEND_X86_64_DEFAULT && (!MLK_CONFIG_NO_ENCAPS_API || \ + !MLK_CONFIG_NO_DECAPS_API) */ + +MLK_EMPTY_CU(avx2_compress_consts) + +#endif /* !(MLK_ARITH_BACKEND_X86_64_DEFAULT && (!MLK_CONFIG_NO_ENCAPS_API || \ + !MLK_CONFIG_NO_DECAPS_API)) */ diff --git a/cbits/mlkem/src/native/x86_64/src/compress_consts.h b/cbits/mlkem/src/native/x86_64/src/compress_consts.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/compress_consts.h @@ -0,0 +1,53 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mlkem-native repository. + * Do not modify it directly. + */ + +#ifndef MLK_NATIVE_X86_64_SRC_COMPRESS_CONSTS_H +#define MLK_NATIVE_X86_64_SRC_COMPRESS_CONSTS_H + +#include "../../../common.h" + +#ifndef __ASSEMBLER__ + +#define mlk_compress_d4_data MLK_NAMESPACE(compress_d4_data) +MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_compress_d4_data[32]; + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_decompress_d4_data MLK_NAMESPACE(decompress_d4_data) +MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_decompress_d4_data[32]; +#endif + +#define mlk_compress_d10_data MLK_NAMESPACE(compress_d10_data) +MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_compress_d10_data[32]; + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_decompress_d10_data MLK_NAMESPACE(decompress_d10_data) +MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_decompress_d10_data[32]; +#endif + +#define mlk_compress_d5_data MLK_NAMESPACE(compress_d5_data) +MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_compress_d5_data[32]; + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_decompress_d5_data MLK_NAMESPACE(decompress_d5_data) +MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_decompress_d5_data[96]; +#endif + +#define mlk_compress_d11_data MLK_NAMESPACE(compress_d11_data) +MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_compress_d11_data[64]; + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_decompress_d11_data MLK_NAMESPACE(decompress_d11_data) +MLK_INTERNAL_DATA_DECLARATION const uint8_t mlk_decompress_d11_data[128]; +#endif + +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLK_NATIVE_X86_64_SRC_COMPRESS_CONSTS_H */ diff --git a/cbits/mlkem/src/native/x86_64/src/consts.c b/cbits/mlkem/src/native/x86_64/src/consts.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/consts.c @@ -0,0 +1,102 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mlkem-native repository. + * Do not modify it directly. + */ + +#include "../../../common.h" + +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +#include "consts.h" + +/* + * Table of zeta values used in the AVX2 NTTs + * See autogen for details. + */ +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const int16_t mlk_qdata[624] = { + 3854, 3340, 2826, 2312, 1798, 1284, 770, 256, 3854, + 3340, 2826, 2312, 1798, 1284, 770, 256, 7, 0, + 6, 0, 5, 0, 4, 0, 3, 0, 2, + 0, 1, 0, 0, 0, 31498, 31498, 31498, 31498, + -758, -758, -758, -758, 0, 0, 0, 0, 0, + 0, 0, 0, 14745, 14745, 14745, 14745, 14745, 14745, + 14745, 14745, 14745, 14745, 14745, 14745, 14745, 14745, 14745, + 14745, -359, -359, -359, -359, -359, -359, -359, -359, + -359, -359, -359, -359, -359, -359, -359, -359, 13525, + 13525, 13525, 13525, 13525, 13525, 13525, 13525, -12402, -12402, + -12402, -12402, -12402, -12402, -12402, -12402, 1493, 1493, 1493, + 1493, 1493, 1493, 1493, 1493, 1422, 1422, 1422, 1422, + 1422, 1422, 1422, 1422, -20907, -20907, -20907, -20907, 27758, + 27758, 27758, 27758, -3799, -3799, -3799, -3799, -15690, -15690, + -15690, -15690, -171, -171, -171, -171, 622, 622, 622, + 622, 1577, 1577, 1577, 1577, 182, 182, 182, 182, + -5827, -5827, 17363, 17363, -26360, -26360, -29057, -29057, 5571, + 5571, -1102, -1102, 21438, 21438, -26242, -26242, 573, 573, + -1325, -1325, 264, 264, 383, 383, -829, -829, 1458, + 1458, -1602, -1602, -130, -130, -5689, -6516, 1496, 30967, + -23565, 20179, 20710, 25080, -12796, 26616, 16064, -12442, 9134, + -650, -25986, 27837, 1223, 652, -552, 1015, -1293, 1491, + -282, -1544, 516, -8, -320, -666, -1618, -1162, 126, + 1469, -335, -11477, -32227, 20494, -27738, 945, -14883, 6182, + 32010, 10631, 29175, -28762, -18486, 17560, -14430, -5276, -1103, + 555, -1251, 1550, 422, 177, -291, 1574, -246, 1159, + -777, -602, -1590, -872, 418, -156, 11182, 13387, -14233, + -21655, 13131, -4587, 23092, 5493, -32502, 30317, -18741, 12639, + 20100, 18525, 19529, -12619, 430, 843, 871, 105, 587, + -235, -460, 1653, 778, -147, 1483, 1119, 644, 349, + 329, -75, 787, 787, 787, 787, 787, 787, 787, + 787, 787, 787, 787, 787, 787, 787, 787, 787, + -1517, -1517, -1517, -1517, -1517, -1517, -1517, -1517, -1517, + -1517, -1517, -1517, -1517, -1517, -1517, -1517, 28191, 28191, + 28191, 28191, 28191, 28191, 28191, 28191, -16694, -16694, -16694, + -16694, -16694, -16694, -16694, -16694, 287, 287, 287, 287, + 287, 287, 287, 287, 202, 202, 202, 202, 202, + 202, 202, 202, 10690, 10690, 10690, 10690, 1358, 1358, + 1358, 1358, -11202, -11202, -11202, -11202, 31164, 31164, 31164, + 31164, 962, 962, 962, 962, -1202, -1202, -1202, -1202, + -1474, -1474, -1474, -1474, 1468, 1468, 1468, 1468, -28073, + -28073, 24313, 24313, -10532, -10532, 8800, 8800, 18426, 18426, + 8859, 8859, 26675, 26675, -16163, -16163, -681, -681, 1017, + 1017, 732, 732, 608, 608, -1542, -1542, 411, 411, + -205, -205, -1571, -1571, 19883, -28250, -15887, -8898, -28309, + 9075, -30199, 18249, 13426, 14017, -29156, -12757, 16832, 4311, + -24155, -17915, -853, -90, -271, 830, 107, -1421, -247, + -951, -398, 961, -1508, -725, 448, -1065, 677, -1275, + -31183, 25435, -7382, 24391, -20927, 10946, 24214, 16989, 10335, + -7934, -22502, 10906, 31636, 28644, 23998, -17422, 817, 603, + 1322, -1465, -1215, 1218, -874, -1187, -1185, -1278, -1510, + -870, -108, 996, 958, 1522, 20297, 2146, 15355, -32384, + -6280, -14903, -11044, 14469, -21498, -20198, 23210, -17442, -23860, + -20257, 7756, 23132, 1097, 610, -1285, 384, -136, -1335, + 220, -1659, -1530, 794, -854, 478, -308, 991, -1460, + 1628, -1103, 555, -1251, 1550, 422, 177, -291, 1574, + -246, 1159, -777, -602, -1590, -872, 418, -156, 430, + 843, 871, 105, 587, -235, -460, 1653, 778, -147, + 1483, 1119, 644, 349, 329, -75, 817, 603, 1322, + -1465, -1215, 1218, -874, -1187, -1185, -1278, -1510, -870, + -108, 996, 958, 1522, 1097, 610, -1285, 384, -136, + -1335, 220, -1659, -1530, 794, -854, 478, -308, 991, + -1460, 1628, -335, -11477, -32227, 20494, -27738, 945, -14883, + 6182, 32010, 10631, 29175, -28762, -18486, 17560, -14430, -5276, + 11182, 13387, -14233, -21655, 13131, -4587, 23092, 5493, -32502, + 30317, -18741, 12639, 20100, 18525, 19529, -12619, -31183, 25435, + -7382, 24391, -20927, 10946, 24214, 16989, 10335, -7934, -22502, + 10906, 31636, 28644, 23998, -17422, 20297, 2146, 15355, -32384, + -6280, -14903, -11044, 14469, -21498, -20198, 23210, -17442, -23860, + -20257, 7756, 23132, +}; + +#else /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +MLK_EMPTY_CU(avx2_consts) + +#endif /* !(MLK_ARITH_BACKEND_X86_64_DEFAULT && \ + !MLK_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mlkem/src/native/x86_64/src/consts.h b/cbits/mlkem/src/native/x86_64/src/consts.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/consts.h @@ -0,0 +1,25 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mlkem-native repository. + * Do not modify it directly. + */ + +#ifndef MLK_NATIVE_X86_64_SRC_CONSTS_H +#define MLK_NATIVE_X86_64_SRC_CONSTS_H +#include "../../../common.h" +#define MLK_AVX2_BACKEND_DATA_OFFSET_REVIDXB 0 +#define MLK_AVX2_BACKEND_DATA_OFFSET_REVIDXD 16 +#define MLK_AVX2_BACKEND_DATA_OFFSET_ZETAS_EXP 32 +#define MLK_AVX2_BACKEND_DATA_OFFSET_MULCACHE_TWIDDLES 496 + +#ifndef __ASSEMBLER__ +#define mlk_qdata MLK_NAMESPACE(qdata) +MLK_INTERNAL_DATA_DECLARATION const int16_t mlk_qdata[624]; +#endif + +#endif /* !MLK_NATIVE_X86_64_SRC_CONSTS_H */ diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_intt_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_intt_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_intt_avx2_asm.S @@ -0,0 +1,743 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [AVX2_NTT] + * Faster AVX2 optimized NTT multiplication for Ring-LWE lattice cryptography. + * Gregor Seiler + * https://eprint.iacr.org/2018/039 + * + * - [REF_AVX2] + * CRYSTALS-Kyber optimized AVX2 implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Kyber implementation @[REF_AVX2]. + * + * The core ideas behind the implementation are described in @[AVX2_NTT]. + * + * Changes: + * - Different placement of modular reductions to simplify + * reasoning of non-overflow + */ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + (!defined(MLK_CONFIG_NO_ENCAPS_API) || \ + !defined(MLK_CONFIG_NO_DECAPS_API)) +/*yaml + Name: invntt_avx2_asm + Description: x86_64 AVX2 inverse NTT + Signature: void mlk_invntt_avx2_asm(int16_t *r, const int16_t *qdata) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 512 + permissions: read/write + c_parameter: int16_t *r + description: Input/output polynomial (256 x int16_t) + rsi: + type: buffer + size_bytes: 1248 + permissions: read-only + c_parameter: const int16_t *qdata + description: Precomputed constants (624 x int16_t) +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_intt_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(invntt_avx2_asm) +MLK_ASM_FN_SYMBOL(invntt_avx2_asm) + + .cfi_startproc + movl $0xd010d01, %eax # imm = 0xD010D01 + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + movl $0xd8a1d8a1, %eax # imm = 0xD8A1D8A1 + vmovd %eax, %xmm2 + vpbroadcastd %xmm2, %ymm2 + movl $0x5a105a1, %eax # imm = 0x5A105A1 + vmovd %eax, %xmm3 + vpbroadcastd %xmm3, %ymm3 + vmovdqa (%rdi), %ymm4 + vmovdqa 0x40(%rdi), %ymm6 + vmovdqa 0x20(%rdi), %ymm5 + vmovdqa 0x60(%rdi), %ymm7 + vpmullw %ymm2, %ymm4, %ymm12 + vpmulhw %ymm3, %ymm4, %ymm4 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm4, %ymm4 + vpmullw %ymm2, %ymm6, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm6 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm6, %ymm6 + vpmullw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm5, %ymm5 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm5, %ymm5 + vpmullw %ymm2, %ymm7, %ymm12 + vpmulhw %ymm3, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm7, %ymm7 + vmovdqa 0x80(%rdi), %ymm8 + vmovdqa 0xc0(%rdi), %ymm10 + vmovdqa 0xa0(%rdi), %ymm9 + vmovdqa 0xe0(%rdi), %ymm11 + vpmullw %ymm2, %ymm8, %ymm12 + vpmulhw %ymm3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm8, %ymm8 + vpmullw %ymm2, %ymm10, %ymm12 + vpmulhw %ymm3, %ymm10, %ymm10 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm10, %ymm10 + vpmullw %ymm2, %ymm9, %ymm12 + vpmulhw %ymm3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm9, %ymm9 + vpmullw %ymm2, %ymm11, %ymm12 + vpmulhw %ymm3, %ymm11, %ymm11 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm11, %ymm11 + vpermq $0x4e, 0x3a0(%rsi), %ymm15 # ymm15 = mem[2,3,0,1] + vpermq $0x4e, 0x360(%rsi), %ymm1 # ymm1 = mem[2,3,0,1] + vpermq $0x4e, 0x3c0(%rsi), %ymm2 # ymm2 = mem[2,3,0,1] + vpermq $0x4e, 0x380(%rsi), %ymm3 # ymm3 = mem[2,3,0,1] + vmovdqa (%rsi), %ymm12 + vpshufb %ymm12, %ymm15, %ymm15 + vpshufb %ymm12, %ymm1, %ymm1 + vpshufb %ymm12, %ymm2, %ymm2 + vpshufb %ymm12, %ymm3, %ymm3 + vpsubw %ymm4, %ymm6, %ymm12 + vpaddw %ymm6, %ymm4, %ymm4 + vpsubw %ymm5, %ymm7, %ymm13 + vpmullw %ymm15, %ymm12, %ymm6 + vpaddw %ymm7, %ymm5, %ymm5 + vpsubw %ymm8, %ymm10, %ymm14 + vpmullw %ymm15, %ymm13, %ymm7 + vpaddw %ymm10, %ymm8, %ymm8 + vpsubw %ymm9, %ymm11, %ymm15 + vpmullw %ymm1, %ymm14, %ymm10 + vpaddw %ymm11, %ymm9, %ymm9 + vpmullw %ymm1, %ymm15, %ymm11 + vpmulhw %ymm2, %ymm12, %ymm12 + vpmulhw %ymm2, %ymm13, %ymm13 + vpmulhw %ymm3, %ymm14, %ymm14 + vpmulhw %ymm3, %ymm15, %ymm15 + vpmulhw %ymm0, %ymm6, %ymm6 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm10, %ymm10 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm6, %ymm12, %ymm6 + vpsubw %ymm7, %ymm13, %ymm7 + vpsubw %ymm10, %ymm14, %ymm10 + vpsubw %ymm11, %ymm15, %ymm11 + vpermq $0x4e, 0x320(%rsi), %ymm2 # ymm2 = mem[2,3,0,1] + vpermq $0x4e, 0x340(%rsi), %ymm3 # ymm3 = mem[2,3,0,1] + vmovdqa (%rsi), %ymm1 + vpshufb %ymm1, %ymm2, %ymm2 + vpshufb %ymm1, %ymm3, %ymm3 + vpsubw %ymm4, %ymm8, %ymm12 + vpaddw %ymm8, %ymm4, %ymm4 + vpsubw %ymm5, %ymm9, %ymm13 + vpmullw %ymm2, %ymm12, %ymm8 + vpaddw %ymm9, %ymm5, %ymm5 + vpsubw %ymm6, %ymm10, %ymm14 + vpmullw %ymm2, %ymm13, %ymm9 + vpaddw %ymm10, %ymm6, %ymm6 + vpsubw %ymm7, %ymm11, %ymm15 + vpmullw %ymm2, %ymm14, %ymm10 + vpaddw %ymm11, %ymm7, %ymm7 + vpmullw %ymm2, %ymm15, %ymm11 + vpmulhw %ymm3, %ymm12, %ymm12 + vpmulhw %ymm3, %ymm13, %ymm13 + vpmulhw %ymm3, %ymm14, %ymm14 + vpmulhw %ymm3, %ymm15, %ymm15 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm10, %ymm10 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm8, %ymm12, %ymm8 + vpsubw %ymm9, %ymm13, %ymm9 + vpsubw %ymm10, %ymm14, %ymm10 + vpsubw %ymm11, %ymm15, %ymm11 + vpslld $0x10, %ymm5, %ymm3 + vpblendw $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7],ymm4[8],ymm3[9],ymm4[10],ymm3[11],ymm4[12],ymm3[13],ymm4[14],ymm3[15] + vpsrld $0x10, %ymm4, %ymm4 + vpblendw $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7],ymm4[8],ymm5[9],ymm4[10],ymm5[11],ymm4[12],ymm5[13],ymm4[14],ymm5[15] + vpslld $0x10, %ymm7, %ymm4 + vpblendw $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7],ymm6[8],ymm4[9],ymm6[10],ymm4[11],ymm6[12],ymm4[13],ymm6[14],ymm4[15] + vpsrld $0x10, %ymm6, %ymm6 + vpblendw $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7],ymm6[8],ymm7[9],ymm6[10],ymm7[11],ymm6[12],ymm7[13],ymm6[14],ymm7[15] + vpslld $0x10, %ymm9, %ymm6 + vpblendw $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7],ymm8[8],ymm6[9],ymm8[10],ymm6[11],ymm8[12],ymm6[13],ymm8[14],ymm6[15] + vpsrld $0x10, %ymm8, %ymm8 + vpblendw $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7],ymm8[8],ymm9[9],ymm8[10],ymm9[11],ymm8[12],ymm9[13],ymm8[14],ymm9[15] + vpslld $0x10, %ymm11, %ymm8 + vpblendw $0xaa, %ymm8, %ymm10, %ymm8 # ymm8 = ymm10[0],ymm8[1],ymm10[2],ymm8[3],ymm10[4],ymm8[5],ymm10[6],ymm8[7],ymm10[8],ymm8[9],ymm10[10],ymm8[11],ymm10[12],ymm8[13],ymm10[14],ymm8[15] + vpsrld $0x10, %ymm10, %ymm10 + vpblendw $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7],ymm10[8],ymm11[9],ymm10[10],ymm11[11],ymm10[12],ymm11[13],ymm10[14],ymm11[15] + vmovdqa 0x20(%rsi), %ymm12 + vpermd 0x2e0(%rsi), %ymm12, %ymm2 + vpermd 0x300(%rsi), %ymm12, %ymm10 + vpsubw %ymm3, %ymm5, %ymm12 + vpaddw %ymm5, %ymm3, %ymm3 + vpsubw %ymm4, %ymm7, %ymm13 + vpmullw %ymm2, %ymm12, %ymm5 + vpaddw %ymm7, %ymm4, %ymm4 + vpsubw %ymm6, %ymm9, %ymm14 + vpmullw %ymm2, %ymm13, %ymm7 + vpaddw %ymm9, %ymm6, %ymm6 + vpsubw %ymm8, %ymm11, %ymm15 + vpmullw %ymm2, %ymm14, %ymm9 + vpaddw %ymm11, %ymm8, %ymm8 + vpmullw %ymm2, %ymm15, %ymm11 + vpmulhw %ymm10, %ymm12, %ymm12 + vpmulhw %ymm10, %ymm13, %ymm13 + vpmulhw %ymm10, %ymm14, %ymm14 + vpmulhw %ymm10, %ymm15, %ymm15 + vpmulhw %ymm0, %ymm5, %ymm5 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm5, %ymm12, %ymm5 + vpsubw %ymm7, %ymm13, %ymm7 + vpsubw %ymm9, %ymm14, %ymm9 + vpsubw %ymm11, %ymm15, %ymm11 + movl $0x4ebf4ebf, %eax # imm = 0x4EBF4EBF + vmovd %eax, %xmm1 + vpbroadcastd %xmm1, %ymm1 + vpmulhw %ymm1, %ymm3, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm3, %ymm3 + vmovsldup %ymm4, %ymm10 # ymm10 = ymm4[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm10, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm10[1],ymm3[2],ymm10[3],ymm3[4],ymm10[5],ymm3[6],ymm10[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7] + vmovsldup %ymm8, %ymm3 # ymm3 = ymm8[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm3[1],ymm6[2],ymm3[3],ymm6[4],ymm3[5],ymm6[6],ymm3[7] + vpsrlq $0x20, %ymm6, %ymm6 + vpblendd $0xaa, %ymm8, %ymm6, %ymm8 # ymm8 = ymm6[0],ymm8[1],ymm6[2],ymm8[3],ymm6[4],ymm8[5],ymm6[6],ymm8[7] + vmovsldup %ymm7, %ymm6 # ymm6 = ymm7[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7] + vpsrlq $0x20, %ymm5, %ymm5 + vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7] + vmovsldup %ymm11, %ymm5 # ymm5 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm5, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm5[1],ymm9[2],ymm5[3],ymm9[4],ymm5[5],ymm9[6],ymm5[7] + vpsrlq $0x20, %ymm9, %ymm9 + vpblendd $0xaa, %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[0],ymm11[1],ymm9[2],ymm11[3],ymm9[4],ymm11[5],ymm9[6],ymm11[7] + vpermq $0x1b, 0x2a0(%rsi), %ymm2 # ymm2 = mem[3,2,1,0] + vpermq $0x1b, 0x2c0(%rsi), %ymm9 # ymm9 = mem[3,2,1,0] + vpsubw %ymm10, %ymm4, %ymm12 + vpaddw %ymm4, %ymm10, %ymm10 + vpsubw %ymm3, %ymm8, %ymm13 + vpmullw %ymm2, %ymm12, %ymm4 + vpaddw %ymm8, %ymm3, %ymm3 + vpsubw %ymm6, %ymm7, %ymm14 + vpmullw %ymm2, %ymm13, %ymm8 + vpaddw %ymm7, %ymm6, %ymm6 + vpsubw %ymm5, %ymm11, %ymm15 + vpmullw %ymm2, %ymm14, %ymm7 + vpaddw %ymm11, %ymm5, %ymm5 + vpmullw %ymm2, %ymm15, %ymm11 + vpmulhw %ymm9, %ymm12, %ymm12 + vpmulhw %ymm9, %ymm13, %ymm13 + vpmulhw %ymm9, %ymm14, %ymm14 + vpmulhw %ymm9, %ymm15, %ymm15 + vpmulhw %ymm0, %ymm4, %ymm4 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm4, %ymm12, %ymm4 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm7, %ymm14, %ymm7 + vpsubw %ymm11, %ymm15, %ymm11 + vpmulhw %ymm1, %ymm10, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm10, %ymm10 + vpunpcklqdq %ymm3, %ymm10, %ymm9 # ymm9 = ymm10[0],ymm3[0],ymm10[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[1],ymm3[1],ymm10[3],ymm3[3] + vpunpcklqdq %ymm5, %ymm6, %ymm10 # ymm10 = ymm6[0],ymm5[0],ymm6[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[1],ymm5[1],ymm6[3],ymm5[3] + vpunpcklqdq %ymm8, %ymm4, %ymm6 # ymm6 = ymm4[0],ymm8[0],ymm4[2],ymm8[2] + vpunpckhqdq %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[1],ymm8[1],ymm4[3],ymm8[3] + vpunpcklqdq %ymm11, %ymm7, %ymm4 # ymm4 = ymm7[0],ymm11[0],ymm7[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[1],ymm11[1],ymm7[3],ymm11[3] + vpermq $0x4e, 0x260(%rsi), %ymm2 # ymm2 = mem[2,3,0,1] + vpermq $0x4e, 0x280(%rsi), %ymm7 # ymm7 = mem[2,3,0,1] + vpsubw %ymm9, %ymm3, %ymm12 + vpaddw %ymm3, %ymm9, %ymm9 + vpsubw %ymm10, %ymm5, %ymm13 + vpmullw %ymm2, %ymm12, %ymm3 + vpaddw %ymm5, %ymm10, %ymm10 + vpsubw %ymm6, %ymm8, %ymm14 + vpmullw %ymm2, %ymm13, %ymm5 + vpaddw %ymm8, %ymm6, %ymm6 + vpsubw %ymm4, %ymm11, %ymm15 + vpmullw %ymm2, %ymm14, %ymm8 + vpaddw %ymm11, %ymm4, %ymm4 + vpmullw %ymm2, %ymm15, %ymm11 + vpmulhw %ymm7, %ymm12, %ymm12 + vpmulhw %ymm7, %ymm13, %ymm13 + vpmulhw %ymm7, %ymm14, %ymm14 + vpmulhw %ymm7, %ymm15, %ymm15 + vpmulhw %ymm0, %ymm3, %ymm3 + vpmulhw %ymm0, %ymm5, %ymm5 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm3, %ymm12, %ymm3 + vpsubw %ymm5, %ymm13, %ymm5 + vpsubw %ymm8, %ymm14, %ymm8 + vpsubw %ymm11, %ymm15, %ymm11 + vpmulhw %ymm1, %ymm9, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm9, %ymm9 + vperm2i128 $0x20, %ymm10, %ymm9, %ymm7 # ymm7 = ymm9[0,1],ymm10[0,1] + vperm2i128 $0x31, %ymm10, %ymm9, %ymm10 # ymm10 = ymm9[2,3],ymm10[2,3] + vperm2i128 $0x20, %ymm4, %ymm6, %ymm9 # ymm9 = ymm6[0,1],ymm4[0,1] + vperm2i128 $0x31, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[2,3],ymm4[2,3] + vperm2i128 $0x20, %ymm5, %ymm3, %ymm6 # ymm6 = ymm3[0,1],ymm5[0,1] + vperm2i128 $0x31, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[2,3],ymm5[2,3] + vperm2i128 $0x20, %ymm11, %ymm8, %ymm3 # ymm3 = ymm8[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm8, %ymm11 # ymm11 = ymm8[2,3],ymm11[2,3] + vmovdqa 0x220(%rsi), %ymm2 + vmovdqa 0x240(%rsi), %ymm8 + vpsubw %ymm7, %ymm10, %ymm12 + vpaddw %ymm10, %ymm7, %ymm7 + vpsubw %ymm9, %ymm4, %ymm13 + vpmullw %ymm2, %ymm12, %ymm10 + vpaddw %ymm4, %ymm9, %ymm9 + vpsubw %ymm6, %ymm5, %ymm14 + vpmullw %ymm2, %ymm13, %ymm4 + vpaddw %ymm5, %ymm6, %ymm6 + vpsubw %ymm3, %ymm11, %ymm15 + vpmullw %ymm2, %ymm14, %ymm5 + vpaddw %ymm11, %ymm3, %ymm3 + vpmullw %ymm2, %ymm15, %ymm11 + vpmulhw %ymm8, %ymm12, %ymm12 + vpmulhw %ymm8, %ymm13, %ymm13 + vpmulhw %ymm8, %ymm14, %ymm14 + vpmulhw %ymm8, %ymm15, %ymm15 + vpmulhw %ymm0, %ymm10, %ymm10 + vpmulhw %ymm0, %ymm4, %ymm4 + vpmulhw %ymm0, %ymm5, %ymm5 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm10, %ymm12, %ymm10 + vpsubw %ymm4, %ymm13, %ymm4 + vpsubw %ymm5, %ymm14, %ymm5 + vpsubw %ymm11, %ymm15, %ymm11 + vpmulhw %ymm1, %ymm7, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm7, %ymm7 + vmovdqa %ymm7, (%rdi) + vmovdqa %ymm9, 0x20(%rdi) + vmovdqa %ymm6, 0x40(%rdi) + vmovdqa %ymm3, 0x60(%rdi) + vmovdqa %ymm10, 0x80(%rdi) + vmovdqa %ymm4, 0xa0(%rdi) + vmovdqa %ymm5, 0xc0(%rdi) + vmovdqa %ymm11, 0xe0(%rdi) + movl $0xd8a1d8a1, %eax # imm = 0xD8A1D8A1 + vmovd %eax, %xmm2 + vpbroadcastd %xmm2, %ymm2 + movl $0x5a105a1, %eax # imm = 0x5A105A1 + vmovd %eax, %xmm3 + vpbroadcastd %xmm3, %ymm3 + vmovdqa 0x100(%rdi), %ymm4 + vmovdqa 0x140(%rdi), %ymm6 + vmovdqa 0x120(%rdi), %ymm5 + vmovdqa 0x160(%rdi), %ymm7 + vpmullw %ymm2, %ymm4, %ymm12 + vpmulhw %ymm3, %ymm4, %ymm4 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm4, %ymm4 + vpmullw %ymm2, %ymm6, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm6 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm6, %ymm6 + vpmullw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm5, %ymm5 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm5, %ymm5 + vpmullw %ymm2, %ymm7, %ymm12 + vpmulhw %ymm3, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm7, %ymm7 + vmovdqa 0x180(%rdi), %ymm8 + vmovdqa 0x1c0(%rdi), %ymm10 + vmovdqa 0x1a0(%rdi), %ymm9 + vmovdqa 0x1e0(%rdi), %ymm11 + vpmullw %ymm2, %ymm8, %ymm12 + vpmulhw %ymm3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm8, %ymm8 + vpmullw %ymm2, %ymm10, %ymm12 + vpmulhw %ymm3, %ymm10, %ymm10 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm10, %ymm10 + vpmullw %ymm2, %ymm9, %ymm12 + vpmulhw %ymm3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm9, %ymm9 + vpmullw %ymm2, %ymm11, %ymm12 + vpmulhw %ymm3, %ymm11, %ymm11 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm11, %ymm11 + vpermq $0x4e, 0x1e0(%rsi), %ymm15 # ymm15 = mem[2,3,0,1] + vpermq $0x4e, 0x1a0(%rsi), %ymm1 # ymm1 = mem[2,3,0,1] + vpermq $0x4e, 0x200(%rsi), %ymm2 # ymm2 = mem[2,3,0,1] + vpermq $0x4e, 0x1c0(%rsi), %ymm3 # ymm3 = mem[2,3,0,1] + vmovdqa (%rsi), %ymm12 + vpshufb %ymm12, %ymm15, %ymm15 + vpshufb %ymm12, %ymm1, %ymm1 + vpshufb %ymm12, %ymm2, %ymm2 + vpshufb %ymm12, %ymm3, %ymm3 + vpsubw %ymm4, %ymm6, %ymm12 + vpaddw %ymm6, %ymm4, %ymm4 + vpsubw %ymm5, %ymm7, %ymm13 + vpmullw %ymm15, %ymm12, %ymm6 + vpaddw %ymm7, %ymm5, %ymm5 + vpsubw %ymm8, %ymm10, %ymm14 + vpmullw %ymm15, %ymm13, %ymm7 + vpaddw %ymm10, %ymm8, %ymm8 + vpsubw %ymm9, %ymm11, %ymm15 + vpmullw %ymm1, %ymm14, %ymm10 + vpaddw %ymm11, %ymm9, %ymm9 + vpmullw %ymm1, %ymm15, %ymm11 + vpmulhw %ymm2, %ymm12, %ymm12 + vpmulhw %ymm2, %ymm13, %ymm13 + vpmulhw %ymm3, %ymm14, %ymm14 + vpmulhw %ymm3, %ymm15, %ymm15 + vpmulhw %ymm0, %ymm6, %ymm6 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm10, %ymm10 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm6, %ymm12, %ymm6 + vpsubw %ymm7, %ymm13, %ymm7 + vpsubw %ymm10, %ymm14, %ymm10 + vpsubw %ymm11, %ymm15, %ymm11 + vpermq $0x4e, 0x160(%rsi), %ymm2 # ymm2 = mem[2,3,0,1] + vpermq $0x4e, 0x180(%rsi), %ymm3 # ymm3 = mem[2,3,0,1] + vmovdqa (%rsi), %ymm1 + vpshufb %ymm1, %ymm2, %ymm2 + vpshufb %ymm1, %ymm3, %ymm3 + vpsubw %ymm4, %ymm8, %ymm12 + vpaddw %ymm8, %ymm4, %ymm4 + vpsubw %ymm5, %ymm9, %ymm13 + vpmullw %ymm2, %ymm12, %ymm8 + vpaddw %ymm9, %ymm5, %ymm5 + vpsubw %ymm6, %ymm10, %ymm14 + vpmullw %ymm2, %ymm13, %ymm9 + vpaddw %ymm10, %ymm6, %ymm6 + vpsubw %ymm7, %ymm11, %ymm15 + vpmullw %ymm2, %ymm14, %ymm10 + vpaddw %ymm11, %ymm7, %ymm7 + vpmullw %ymm2, %ymm15, %ymm11 + vpmulhw %ymm3, %ymm12, %ymm12 + vpmulhw %ymm3, %ymm13, %ymm13 + vpmulhw %ymm3, %ymm14, %ymm14 + vpmulhw %ymm3, %ymm15, %ymm15 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm10, %ymm10 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm8, %ymm12, %ymm8 + vpsubw %ymm9, %ymm13, %ymm9 + vpsubw %ymm10, %ymm14, %ymm10 + vpsubw %ymm11, %ymm15, %ymm11 + vpslld $0x10, %ymm5, %ymm3 + vpblendw $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7],ymm4[8],ymm3[9],ymm4[10],ymm3[11],ymm4[12],ymm3[13],ymm4[14],ymm3[15] + vpsrld $0x10, %ymm4, %ymm4 + vpblendw $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7],ymm4[8],ymm5[9],ymm4[10],ymm5[11],ymm4[12],ymm5[13],ymm4[14],ymm5[15] + vpslld $0x10, %ymm7, %ymm4 + vpblendw $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7],ymm6[8],ymm4[9],ymm6[10],ymm4[11],ymm6[12],ymm4[13],ymm6[14],ymm4[15] + vpsrld $0x10, %ymm6, %ymm6 + vpblendw $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7],ymm6[8],ymm7[9],ymm6[10],ymm7[11],ymm6[12],ymm7[13],ymm6[14],ymm7[15] + vpslld $0x10, %ymm9, %ymm6 + vpblendw $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7],ymm8[8],ymm6[9],ymm8[10],ymm6[11],ymm8[12],ymm6[13],ymm8[14],ymm6[15] + vpsrld $0x10, %ymm8, %ymm8 + vpblendw $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7],ymm8[8],ymm9[9],ymm8[10],ymm9[11],ymm8[12],ymm9[13],ymm8[14],ymm9[15] + vpslld $0x10, %ymm11, %ymm8 + vpblendw $0xaa, %ymm8, %ymm10, %ymm8 # ymm8 = ymm10[0],ymm8[1],ymm10[2],ymm8[3],ymm10[4],ymm8[5],ymm10[6],ymm8[7],ymm10[8],ymm8[9],ymm10[10],ymm8[11],ymm10[12],ymm8[13],ymm10[14],ymm8[15] + vpsrld $0x10, %ymm10, %ymm10 + vpblendw $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7],ymm10[8],ymm11[9],ymm10[10],ymm11[11],ymm10[12],ymm11[13],ymm10[14],ymm11[15] + vmovdqa 0x20(%rsi), %ymm12 + vpermd 0x120(%rsi), %ymm12, %ymm2 + vpermd 0x140(%rsi), %ymm12, %ymm10 + vpsubw %ymm3, %ymm5, %ymm12 + vpaddw %ymm5, %ymm3, %ymm3 + vpsubw %ymm4, %ymm7, %ymm13 + vpmullw %ymm2, %ymm12, %ymm5 + vpaddw %ymm7, %ymm4, %ymm4 + vpsubw %ymm6, %ymm9, %ymm14 + vpmullw %ymm2, %ymm13, %ymm7 + vpaddw %ymm9, %ymm6, %ymm6 + vpsubw %ymm8, %ymm11, %ymm15 + vpmullw %ymm2, %ymm14, %ymm9 + vpaddw %ymm11, %ymm8, %ymm8 + vpmullw %ymm2, %ymm15, %ymm11 + vpmulhw %ymm10, %ymm12, %ymm12 + vpmulhw %ymm10, %ymm13, %ymm13 + vpmulhw %ymm10, %ymm14, %ymm14 + vpmulhw %ymm10, %ymm15, %ymm15 + vpmulhw %ymm0, %ymm5, %ymm5 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm5, %ymm12, %ymm5 + vpsubw %ymm7, %ymm13, %ymm7 + vpsubw %ymm9, %ymm14, %ymm9 + vpsubw %ymm11, %ymm15, %ymm11 + movl $0x4ebf4ebf, %eax # imm = 0x4EBF4EBF + vmovd %eax, %xmm1 + vpbroadcastd %xmm1, %ymm1 + vpmulhw %ymm1, %ymm3, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm3, %ymm3 + vmovsldup %ymm4, %ymm10 # ymm10 = ymm4[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm10, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm10[1],ymm3[2],ymm10[3],ymm3[4],ymm10[5],ymm3[6],ymm10[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7] + vmovsldup %ymm8, %ymm3 # ymm3 = ymm8[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm3[1],ymm6[2],ymm3[3],ymm6[4],ymm3[5],ymm6[6],ymm3[7] + vpsrlq $0x20, %ymm6, %ymm6 + vpblendd $0xaa, %ymm8, %ymm6, %ymm8 # ymm8 = ymm6[0],ymm8[1],ymm6[2],ymm8[3],ymm6[4],ymm8[5],ymm6[6],ymm8[7] + vmovsldup %ymm7, %ymm6 # ymm6 = ymm7[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7] + vpsrlq $0x20, %ymm5, %ymm5 + vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7] + vmovsldup %ymm11, %ymm5 # ymm5 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm5, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm5[1],ymm9[2],ymm5[3],ymm9[4],ymm5[5],ymm9[6],ymm5[7] + vpsrlq $0x20, %ymm9, %ymm9 + vpblendd $0xaa, %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[0],ymm11[1],ymm9[2],ymm11[3],ymm9[4],ymm11[5],ymm9[6],ymm11[7] + vpermq $0x1b, 0xe0(%rsi), %ymm2 # ymm2 = mem[3,2,1,0] + vpermq $0x1b, 0x100(%rsi), %ymm9 # ymm9 = mem[3,2,1,0] + vpsubw %ymm10, %ymm4, %ymm12 + vpaddw %ymm4, %ymm10, %ymm10 + vpsubw %ymm3, %ymm8, %ymm13 + vpmullw %ymm2, %ymm12, %ymm4 + vpaddw %ymm8, %ymm3, %ymm3 + vpsubw %ymm6, %ymm7, %ymm14 + vpmullw %ymm2, %ymm13, %ymm8 + vpaddw %ymm7, %ymm6, %ymm6 + vpsubw %ymm5, %ymm11, %ymm15 + vpmullw %ymm2, %ymm14, %ymm7 + vpaddw %ymm11, %ymm5, %ymm5 + vpmullw %ymm2, %ymm15, %ymm11 + vpmulhw %ymm9, %ymm12, %ymm12 + vpmulhw %ymm9, %ymm13, %ymm13 + vpmulhw %ymm9, %ymm14, %ymm14 + vpmulhw %ymm9, %ymm15, %ymm15 + vpmulhw %ymm0, %ymm4, %ymm4 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm4, %ymm12, %ymm4 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm7, %ymm14, %ymm7 + vpsubw %ymm11, %ymm15, %ymm11 + vpmulhw %ymm1, %ymm10, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm10, %ymm10 + vpunpcklqdq %ymm3, %ymm10, %ymm9 # ymm9 = ymm10[0],ymm3[0],ymm10[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[1],ymm3[1],ymm10[3],ymm3[3] + vpunpcklqdq %ymm5, %ymm6, %ymm10 # ymm10 = ymm6[0],ymm5[0],ymm6[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[1],ymm5[1],ymm6[3],ymm5[3] + vpunpcklqdq %ymm8, %ymm4, %ymm6 # ymm6 = ymm4[0],ymm8[0],ymm4[2],ymm8[2] + vpunpckhqdq %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[1],ymm8[1],ymm4[3],ymm8[3] + vpunpcklqdq %ymm11, %ymm7, %ymm4 # ymm4 = ymm7[0],ymm11[0],ymm7[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[1],ymm11[1],ymm7[3],ymm11[3] + vpermq $0x4e, 0xa0(%rsi), %ymm2 # ymm2 = mem[2,3,0,1] + vpermq $0x4e, 0xc0(%rsi), %ymm7 # ymm7 = mem[2,3,0,1] + vpsubw %ymm9, %ymm3, %ymm12 + vpaddw %ymm3, %ymm9, %ymm9 + vpsubw %ymm10, %ymm5, %ymm13 + vpmullw %ymm2, %ymm12, %ymm3 + vpaddw %ymm5, %ymm10, %ymm10 + vpsubw %ymm6, %ymm8, %ymm14 + vpmullw %ymm2, %ymm13, %ymm5 + vpaddw %ymm8, %ymm6, %ymm6 + vpsubw %ymm4, %ymm11, %ymm15 + vpmullw %ymm2, %ymm14, %ymm8 + vpaddw %ymm11, %ymm4, %ymm4 + vpmullw %ymm2, %ymm15, %ymm11 + vpmulhw %ymm7, %ymm12, %ymm12 + vpmulhw %ymm7, %ymm13, %ymm13 + vpmulhw %ymm7, %ymm14, %ymm14 + vpmulhw %ymm7, %ymm15, %ymm15 + vpmulhw %ymm0, %ymm3, %ymm3 + vpmulhw %ymm0, %ymm5, %ymm5 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm3, %ymm12, %ymm3 + vpsubw %ymm5, %ymm13, %ymm5 + vpsubw %ymm8, %ymm14, %ymm8 + vpsubw %ymm11, %ymm15, %ymm11 + vpmulhw %ymm1, %ymm9, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm9, %ymm9 + vperm2i128 $0x20, %ymm10, %ymm9, %ymm7 # ymm7 = ymm9[0,1],ymm10[0,1] + vperm2i128 $0x31, %ymm10, %ymm9, %ymm10 # ymm10 = ymm9[2,3],ymm10[2,3] + vperm2i128 $0x20, %ymm4, %ymm6, %ymm9 # ymm9 = ymm6[0,1],ymm4[0,1] + vperm2i128 $0x31, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[2,3],ymm4[2,3] + vperm2i128 $0x20, %ymm5, %ymm3, %ymm6 # ymm6 = ymm3[0,1],ymm5[0,1] + vperm2i128 $0x31, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[2,3],ymm5[2,3] + vperm2i128 $0x20, %ymm11, %ymm8, %ymm3 # ymm3 = ymm8[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm8, %ymm11 # ymm11 = ymm8[2,3],ymm11[2,3] + vmovdqa 0x60(%rsi), %ymm2 + vmovdqa 0x80(%rsi), %ymm8 + vpsubw %ymm7, %ymm10, %ymm12 + vpaddw %ymm10, %ymm7, %ymm7 + vpsubw %ymm9, %ymm4, %ymm13 + vpmullw %ymm2, %ymm12, %ymm10 + vpaddw %ymm4, %ymm9, %ymm9 + vpsubw %ymm6, %ymm5, %ymm14 + vpmullw %ymm2, %ymm13, %ymm4 + vpaddw %ymm5, %ymm6, %ymm6 + vpsubw %ymm3, %ymm11, %ymm15 + vpmullw %ymm2, %ymm14, %ymm5 + vpaddw %ymm11, %ymm3, %ymm3 + vpmullw %ymm2, %ymm15, %ymm11 + vpmulhw %ymm8, %ymm12, %ymm12 + vpmulhw %ymm8, %ymm13, %ymm13 + vpmulhw %ymm8, %ymm14, %ymm14 + vpmulhw %ymm8, %ymm15, %ymm15 + vpmulhw %ymm0, %ymm10, %ymm10 + vpmulhw %ymm0, %ymm4, %ymm4 + vpmulhw %ymm0, %ymm5, %ymm5 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm10, %ymm12, %ymm10 + vpsubw %ymm4, %ymm13, %ymm4 + vpsubw %ymm5, %ymm14, %ymm5 + vpsubw %ymm11, %ymm15, %ymm11 + vpmulhw %ymm1, %ymm7, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm7, %ymm7 + vmovdqa %ymm7, 0x100(%rdi) + vmovdqa %ymm9, 0x120(%rdi) + vmovdqa %ymm6, 0x140(%rdi) + vmovdqa %ymm3, 0x160(%rdi) + vmovdqa %ymm10, 0x180(%rdi) + vmovdqa %ymm4, 0x1a0(%rdi) + vmovdqa %ymm5, 0x1c0(%rdi) + vmovdqa %ymm11, 0x1e0(%rdi) + vmovdqa (%rdi), %ymm4 + vmovdqa 0x100(%rdi), %ymm8 + vmovdqa 0x20(%rdi), %ymm5 + vmovdqa 0x120(%rdi), %ymm9 + vpbroadcastq 0x40(%rsi), %ymm2 + vmovdqa 0x40(%rdi), %ymm6 + vmovdqa 0x140(%rdi), %ymm10 + vmovdqa 0x60(%rdi), %ymm7 + vmovdqa 0x160(%rdi), %ymm11 + vpbroadcastq 0x48(%rsi), %ymm3 + vpsubw %ymm4, %ymm8, %ymm12 + vpaddw %ymm8, %ymm4, %ymm4 + vpsubw %ymm5, %ymm9, %ymm13 + vpmullw %ymm2, %ymm12, %ymm8 + vpaddw %ymm9, %ymm5, %ymm5 + vpsubw %ymm6, %ymm10, %ymm14 + vpmullw %ymm2, %ymm13, %ymm9 + vpaddw %ymm10, %ymm6, %ymm6 + vpsubw %ymm7, %ymm11, %ymm15 + vpmullw %ymm2, %ymm14, %ymm10 + vpaddw %ymm11, %ymm7, %ymm7 + vpmullw %ymm2, %ymm15, %ymm11 + vpmulhw %ymm3, %ymm12, %ymm12 + vpmulhw %ymm3, %ymm13, %ymm13 + vpmulhw %ymm3, %ymm14, %ymm14 + vpmulhw %ymm3, %ymm15, %ymm15 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm10, %ymm10 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm8, %ymm12, %ymm8 + vpsubw %ymm9, %ymm13, %ymm9 + vpsubw %ymm10, %ymm14, %ymm10 + vpsubw %ymm11, %ymm15, %ymm11 + vmovdqa %ymm4, (%rdi) + vmovdqa %ymm5, 0x20(%rdi) + vmovdqa %ymm6, 0x40(%rdi) + vmovdqa %ymm7, 0x60(%rdi) + vmovdqa %ymm8, 0x100(%rdi) + vmovdqa %ymm9, 0x120(%rdi) + vmovdqa %ymm10, 0x140(%rdi) + vmovdqa %ymm11, 0x160(%rdi) + vmovdqa 0x80(%rdi), %ymm4 + vmovdqa 0x180(%rdi), %ymm8 + vmovdqa 0xa0(%rdi), %ymm5 + vmovdqa 0x1a0(%rdi), %ymm9 + vpbroadcastq 0x40(%rsi), %ymm2 + vmovdqa 0xc0(%rdi), %ymm6 + vmovdqa 0x1c0(%rdi), %ymm10 + vmovdqa 0xe0(%rdi), %ymm7 + vmovdqa 0x1e0(%rdi), %ymm11 + vpbroadcastq 0x48(%rsi), %ymm3 + vpsubw %ymm4, %ymm8, %ymm12 + vpaddw %ymm8, %ymm4, %ymm4 + vpsubw %ymm5, %ymm9, %ymm13 + vpmullw %ymm2, %ymm12, %ymm8 + vpaddw %ymm9, %ymm5, %ymm5 + vpsubw %ymm6, %ymm10, %ymm14 + vpmullw %ymm2, %ymm13, %ymm9 + vpaddw %ymm10, %ymm6, %ymm6 + vpsubw %ymm7, %ymm11, %ymm15 + vpmullw %ymm2, %ymm14, %ymm10 + vpaddw %ymm11, %ymm7, %ymm7 + vpmullw %ymm2, %ymm15, %ymm11 + vpmulhw %ymm3, %ymm12, %ymm12 + vpmulhw %ymm3, %ymm13, %ymm13 + vpmulhw %ymm3, %ymm14, %ymm14 + vpmulhw %ymm3, %ymm15, %ymm15 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm10, %ymm10 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm8, %ymm12, %ymm8 + vpsubw %ymm9, %ymm13, %ymm9 + vpsubw %ymm10, %ymm14, %ymm10 + vpsubw %ymm11, %ymm15, %ymm11 + vmovdqa %ymm4, 0x80(%rdi) + vmovdqa %ymm5, 0xa0(%rdi) + vmovdqa %ymm6, 0xc0(%rdi) + vmovdqa %ymm7, 0xe0(%rdi) + vmovdqa %ymm8, 0x180(%rdi) + vmovdqa %ymm9, 0x1a0(%rdi) + vmovdqa %ymm10, 0x1c0(%rdi) + vmovdqa %ymm11, 0x1e0(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(invntt_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + && (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_ntt_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_ntt_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_ntt_avx2_asm.S @@ -0,0 +1,661 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [AVX2_NTT] + * Faster AVX2 optimized NTT multiplication for Ring-LWE lattice cryptography. + * Gregor Seiler + * https://eprint.iacr.org/2018/039 + * + * - [REF_AVX2] + * CRYSTALS-Kyber optimized AVX2 implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Kyber implementation @[REF_AVX2]. + * + * The core ideas behind the implementation are described in @[AVX2_NTT]. + */ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) +/*yaml + Name: ntt_avx2_asm + Description: x86_64 AVX2 forward NTT + Signature: void mlk_ntt_avx2_asm(int16_t *r, const int16_t *qdata) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 512 + permissions: read/write + c_parameter: int16_t *r + description: Input/output polynomial (256 x int16_t) + rsi: + type: buffer + size_bytes: 1248 + permissions: read-only + c_parameter: const int16_t *qdata + description: Precomputed constants (624 x int16_t) +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_ntt_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(ntt_avx2_asm) +MLK_ASM_FN_SYMBOL(ntt_avx2_asm) + + .cfi_startproc + movl $0xd010d01, %eax # imm = 0xD010D01 + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + vpbroadcastq 0x40(%rsi), %ymm15 + vmovdqa 0x100(%rdi), %ymm8 + vmovdqa 0x120(%rdi), %ymm9 + vmovdqa 0x140(%rdi), %ymm10 + vmovdqa 0x160(%rdi), %ymm11 + vpbroadcastq 0x48(%rsi), %ymm2 + vpmullw %ymm15, %ymm8, %ymm12 + vpmullw %ymm15, %ymm9, %ymm13 + vpmullw %ymm15, %ymm10, %ymm14 + vpmullw %ymm15, %ymm11, %ymm15 + vpmulhw %ymm2, %ymm8, %ymm8 + vpmulhw %ymm2, %ymm9, %ymm9 + vpmulhw %ymm2, %ymm10, %ymm10 + vpmulhw %ymm2, %ymm11, %ymm11 + vmovdqa (%rdi), %ymm4 + vmovdqa 0x20(%rdi), %ymm5 + vmovdqa 0x40(%rdi), %ymm6 + vmovdqa 0x60(%rdi), %ymm7 + vpmulhw %ymm0, %ymm12, %ymm12 + vpmulhw %ymm0, %ymm13, %ymm13 + vpmulhw %ymm0, %ymm14, %ymm14 + vpmulhw %ymm0, %ymm15, %ymm15 + vpaddw %ymm8, %ymm4, %ymm3 + vpsubw %ymm8, %ymm4, %ymm8 + vpaddw %ymm9, %ymm5, %ymm4 + vpsubw %ymm9, %ymm5, %ymm9 + vpaddw %ymm10, %ymm6, %ymm5 + vpsubw %ymm10, %ymm6, %ymm10 + vpaddw %ymm11, %ymm7, %ymm6 + vpsubw %ymm11, %ymm7, %ymm11 + vpsubw %ymm12, %ymm3, %ymm3 + vpaddw %ymm12, %ymm8, %ymm8 + vpsubw %ymm13, %ymm4, %ymm4 + vpaddw %ymm13, %ymm9, %ymm9 + vpsubw %ymm14, %ymm5, %ymm5 + vpaddw %ymm14, %ymm10, %ymm10 + vpsubw %ymm15, %ymm6, %ymm6 + vpaddw %ymm15, %ymm11, %ymm11 + vmovdqa %ymm3, (%rdi) + vmovdqa %ymm4, 0x20(%rdi) + vmovdqa %ymm5, 0x40(%rdi) + vmovdqa %ymm6, 0x60(%rdi) + vmovdqa %ymm8, 0x100(%rdi) + vmovdqa %ymm9, 0x120(%rdi) + vmovdqa %ymm10, 0x140(%rdi) + vmovdqa %ymm11, 0x160(%rdi) + vpbroadcastq 0x40(%rsi), %ymm15 + vmovdqa 0x180(%rdi), %ymm8 + vmovdqa 0x1a0(%rdi), %ymm9 + vmovdqa 0x1c0(%rdi), %ymm10 + vmovdqa 0x1e0(%rdi), %ymm11 + vpbroadcastq 0x48(%rsi), %ymm2 + vpmullw %ymm15, %ymm8, %ymm12 + vpmullw %ymm15, %ymm9, %ymm13 + vpmullw %ymm15, %ymm10, %ymm14 + vpmullw %ymm15, %ymm11, %ymm15 + vpmulhw %ymm2, %ymm8, %ymm8 + vpmulhw %ymm2, %ymm9, %ymm9 + vpmulhw %ymm2, %ymm10, %ymm10 + vpmulhw %ymm2, %ymm11, %ymm11 + vmovdqa 0x80(%rdi), %ymm4 + vmovdqa 0xa0(%rdi), %ymm5 + vmovdqa 0xc0(%rdi), %ymm6 + vmovdqa 0xe0(%rdi), %ymm7 + vpmulhw %ymm0, %ymm12, %ymm12 + vpmulhw %ymm0, %ymm13, %ymm13 + vpmulhw %ymm0, %ymm14, %ymm14 + vpmulhw %ymm0, %ymm15, %ymm15 + vpaddw %ymm8, %ymm4, %ymm3 + vpsubw %ymm8, %ymm4, %ymm8 + vpaddw %ymm9, %ymm5, %ymm4 + vpsubw %ymm9, %ymm5, %ymm9 + vpaddw %ymm10, %ymm6, %ymm5 + vpsubw %ymm10, %ymm6, %ymm10 + vpaddw %ymm11, %ymm7, %ymm6 + vpsubw %ymm11, %ymm7, %ymm11 + vpsubw %ymm12, %ymm3, %ymm3 + vpaddw %ymm12, %ymm8, %ymm8 + vpsubw %ymm13, %ymm4, %ymm4 + vpaddw %ymm13, %ymm9, %ymm9 + vpsubw %ymm14, %ymm5, %ymm5 + vpaddw %ymm14, %ymm10, %ymm10 + vpsubw %ymm15, %ymm6, %ymm6 + vpaddw %ymm15, %ymm11, %ymm11 + vmovdqa %ymm3, 0x80(%rdi) + vmovdqa %ymm4, 0xa0(%rdi) + vmovdqa %ymm5, 0xc0(%rdi) + vmovdqa %ymm6, 0xe0(%rdi) + vmovdqa %ymm8, 0x180(%rdi) + vmovdqa %ymm9, 0x1a0(%rdi) + vmovdqa %ymm10, 0x1c0(%rdi) + vmovdqa %ymm11, 0x1e0(%rdi) + vmovdqa 0x60(%rsi), %ymm15 + vmovdqa 0x80(%rdi), %ymm8 + vmovdqa 0xa0(%rdi), %ymm9 + vmovdqa 0xc0(%rdi), %ymm10 + vmovdqa 0xe0(%rdi), %ymm11 + vmovdqa 0x80(%rsi), %ymm2 + vpmullw %ymm15, %ymm8, %ymm12 + vpmullw %ymm15, %ymm9, %ymm13 + vpmullw %ymm15, %ymm10, %ymm14 + vpmullw %ymm15, %ymm11, %ymm15 + vpmulhw %ymm2, %ymm8, %ymm8 + vpmulhw %ymm2, %ymm9, %ymm9 + vpmulhw %ymm2, %ymm10, %ymm10 + vpmulhw %ymm2, %ymm11, %ymm11 + vmovdqa (%rdi), %ymm4 + vmovdqa 0x20(%rdi), %ymm5 + vmovdqa 0x40(%rdi), %ymm6 + vmovdqa 0x60(%rdi), %ymm7 + vpmulhw %ymm0, %ymm12, %ymm12 + vpmulhw %ymm0, %ymm13, %ymm13 + vpmulhw %ymm0, %ymm14, %ymm14 + vpmulhw %ymm0, %ymm15, %ymm15 + vpaddw %ymm8, %ymm4, %ymm3 + vpsubw %ymm8, %ymm4, %ymm8 + vpaddw %ymm9, %ymm5, %ymm4 + vpsubw %ymm9, %ymm5, %ymm9 + vpaddw %ymm10, %ymm6, %ymm5 + vpsubw %ymm10, %ymm6, %ymm10 + vpaddw %ymm11, %ymm7, %ymm6 + vpsubw %ymm11, %ymm7, %ymm11 + vpsubw %ymm12, %ymm3, %ymm3 + vpaddw %ymm12, %ymm8, %ymm8 + vpsubw %ymm13, %ymm4, %ymm4 + vpaddw %ymm13, %ymm9, %ymm9 + vpsubw %ymm14, %ymm5, %ymm5 + vpaddw %ymm14, %ymm10, %ymm10 + vpsubw %ymm15, %ymm6, %ymm6 + vpaddw %ymm15, %ymm11, %ymm11 + vperm2i128 $0x20, %ymm10, %ymm5, %ymm7 # ymm7 = ymm5[0,1],ymm10[0,1] + vperm2i128 $0x31, %ymm10, %ymm5, %ymm10 # ymm10 = ymm5[2,3],ymm10[2,3] + vperm2i128 $0x20, %ymm11, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm6, %ymm11 # ymm11 = ymm6[2,3],ymm11[2,3] + vmovdqa 0xa0(%rsi), %ymm15 + vmovdqa 0xc0(%rsi), %ymm2 + vpmullw %ymm15, %ymm7, %ymm12 + vpmullw %ymm15, %ymm10, %ymm13 + vpmullw %ymm15, %ymm5, %ymm14 + vpmullw %ymm15, %ymm11, %ymm15 + vpmulhw %ymm2, %ymm7, %ymm7 + vpmulhw %ymm2, %ymm10, %ymm10 + vpmulhw %ymm2, %ymm5, %ymm5 + vpmulhw %ymm2, %ymm11, %ymm11 + vperm2i128 $0x20, %ymm8, %ymm3, %ymm6 # ymm6 = ymm3[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm9, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm4, %ymm9 # ymm9 = ymm4[2,3],ymm9[2,3] + vpmulhw %ymm0, %ymm12, %ymm12 + vpmulhw %ymm0, %ymm13, %ymm13 + vpmulhw %ymm0, %ymm14, %ymm14 + vpmulhw %ymm0, %ymm15, %ymm15 + vpaddw %ymm7, %ymm6, %ymm4 + vpsubw %ymm7, %ymm6, %ymm7 + vpaddw %ymm10, %ymm8, %ymm6 + vpsubw %ymm10, %ymm8, %ymm10 + vpaddw %ymm5, %ymm3, %ymm8 + vpsubw %ymm5, %ymm3, %ymm5 + vpaddw %ymm11, %ymm9, %ymm3 + vpsubw %ymm11, %ymm9, %ymm11 + vpsubw %ymm12, %ymm4, %ymm4 + vpaddw %ymm12, %ymm7, %ymm7 + vpsubw %ymm13, %ymm6, %ymm6 + vpaddw %ymm13, %ymm10, %ymm10 + vpsubw %ymm14, %ymm8, %ymm8 + vpaddw %ymm14, %ymm5, %ymm5 + vpsubw %ymm15, %ymm3, %ymm3 + vpaddw %ymm15, %ymm11, %ymm11 + vpunpcklqdq %ymm5, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm5[0],ymm8[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm8, %ymm5 # ymm5 = ymm8[1],ymm5[1],ymm8[3],ymm5[3] + vpunpcklqdq %ymm11, %ymm3, %ymm8 # ymm8 = ymm3[0],ymm11[0],ymm3[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm3, %ymm11 # ymm11 = ymm3[1],ymm11[1],ymm3[3],ymm11[3] + vmovdqa 0xe0(%rsi), %ymm15 + vmovdqa 0x100(%rsi), %ymm2 + vpmullw %ymm15, %ymm9, %ymm12 + vpmullw %ymm15, %ymm5, %ymm13 + vpmullw %ymm15, %ymm8, %ymm14 + vpmullw %ymm15, %ymm11, %ymm15 + vpmulhw %ymm2, %ymm9, %ymm9 + vpmulhw %ymm2, %ymm5, %ymm5 + vpmulhw %ymm2, %ymm8, %ymm8 + vpmulhw %ymm2, %ymm11, %ymm11 + vpunpcklqdq %ymm7, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm7[0],ymm4[2],ymm7[2] + vpunpckhqdq %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[1],ymm7[1],ymm4[3],ymm7[3] + vpunpcklqdq %ymm10, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm10[0],ymm6[2],ymm10[2] + vpunpckhqdq %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[1],ymm10[1],ymm6[3],ymm10[3] + vpmulhw %ymm0, %ymm12, %ymm12 + vpmulhw %ymm0, %ymm13, %ymm13 + vpmulhw %ymm0, %ymm14, %ymm14 + vpmulhw %ymm0, %ymm15, %ymm15 + vpaddw %ymm9, %ymm3, %ymm6 + vpsubw %ymm9, %ymm3, %ymm9 + vpaddw %ymm5, %ymm7, %ymm3 + vpsubw %ymm5, %ymm7, %ymm5 + vpaddw %ymm8, %ymm4, %ymm7 + vpsubw %ymm8, %ymm4, %ymm8 + vpaddw %ymm11, %ymm10, %ymm4 + vpsubw %ymm11, %ymm10, %ymm11 + vpsubw %ymm12, %ymm6, %ymm6 + vpaddw %ymm12, %ymm9, %ymm9 + vpsubw %ymm13, %ymm3, %ymm3 + vpaddw %ymm13, %ymm5, %ymm5 + vpsubw %ymm14, %ymm7, %ymm7 + vpaddw %ymm14, %ymm8, %ymm8 + vpsubw %ymm15, %ymm4, %ymm4 + vpaddw %ymm15, %ymm11, %ymm11 + vmovsldup %ymm8, %ymm10 # ymm10 = ymm8[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm10, %ymm7, %ymm10 # ymm10 = ymm7[0],ymm10[1],ymm7[2],ymm10[3],ymm7[4],ymm10[5],ymm7[6],ymm10[7] + vpsrlq $0x20, %ymm7, %ymm7 + vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7] + vmovsldup %ymm11, %ymm7 # ymm7 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[0],ymm7[1],ymm4[2],ymm7[3],ymm4[4],ymm7[5],ymm4[6],ymm7[7] + vpsrlq $0x20, %ymm4, %ymm4 + vpblendd $0xaa, %ymm11, %ymm4, %ymm11 # ymm11 = ymm4[0],ymm11[1],ymm4[2],ymm11[3],ymm4[4],ymm11[5],ymm4[6],ymm11[7] + vmovdqa 0x120(%rsi), %ymm15 + vmovdqa 0x140(%rsi), %ymm2 + vpmullw %ymm15, %ymm10, %ymm12 + vpmullw %ymm15, %ymm8, %ymm13 + vpmullw %ymm15, %ymm7, %ymm14 + vpmullw %ymm15, %ymm11, %ymm15 + vpmulhw %ymm2, %ymm10, %ymm10 + vpmulhw %ymm2, %ymm8, %ymm8 + vpmulhw %ymm2, %ymm7, %ymm7 + vpmulhw %ymm2, %ymm11, %ymm11 + vmovsldup %ymm9, %ymm4 # ymm4 = ymm9[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7] + vpsrlq $0x20, %ymm6, %ymm6 + vpblendd $0xaa, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[0],ymm9[1],ymm6[2],ymm9[3],ymm6[4],ymm9[5],ymm6[6],ymm9[7] + vmovsldup %ymm5, %ymm6 # ymm6 = ymm5[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm6, %ymm3, %ymm6 # ymm6 = ymm3[0],ymm6[1],ymm3[2],ymm6[3],ymm3[4],ymm6[5],ymm3[6],ymm6[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7] + vpmulhw %ymm0, %ymm12, %ymm12 + vpmulhw %ymm0, %ymm13, %ymm13 + vpmulhw %ymm0, %ymm14, %ymm14 + vpmulhw %ymm0, %ymm15, %ymm15 + vpaddw %ymm10, %ymm4, %ymm3 + vpsubw %ymm10, %ymm4, %ymm10 + vpaddw %ymm8, %ymm9, %ymm4 + vpsubw %ymm8, %ymm9, %ymm8 + vpaddw %ymm7, %ymm6, %ymm9 + vpsubw %ymm7, %ymm6, %ymm7 + vpaddw %ymm11, %ymm5, %ymm6 + vpsubw %ymm11, %ymm5, %ymm11 + vpsubw %ymm12, %ymm3, %ymm3 + vpaddw %ymm12, %ymm10, %ymm10 + vpsubw %ymm13, %ymm4, %ymm4 + vpaddw %ymm13, %ymm8, %ymm8 + vpsubw %ymm14, %ymm9, %ymm9 + vpaddw %ymm14, %ymm7, %ymm7 + vpsubw %ymm15, %ymm6, %ymm6 + vpaddw %ymm15, %ymm11, %ymm11 + vpslld $0x10, %ymm7, %ymm5 + vpblendw $0xaa, %ymm5, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm5[1],ymm9[2],ymm5[3],ymm9[4],ymm5[5],ymm9[6],ymm5[7],ymm9[8],ymm5[9],ymm9[10],ymm5[11],ymm9[12],ymm5[13],ymm9[14],ymm5[15] + vpsrld $0x10, %ymm9, %ymm9 + vpblendw $0xaa, %ymm7, %ymm9, %ymm7 # ymm7 = ymm9[0],ymm7[1],ymm9[2],ymm7[3],ymm9[4],ymm7[5],ymm9[6],ymm7[7],ymm9[8],ymm7[9],ymm9[10],ymm7[11],ymm9[12],ymm7[13],ymm9[14],ymm7[15] + vpslld $0x10, %ymm11, %ymm9 + vpblendw $0xaa, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[0],ymm9[1],ymm6[2],ymm9[3],ymm6[4],ymm9[5],ymm6[6],ymm9[7],ymm6[8],ymm9[9],ymm6[10],ymm9[11],ymm6[12],ymm9[13],ymm6[14],ymm9[15] + vpsrld $0x10, %ymm6, %ymm6 + vpblendw $0xaa, %ymm11, %ymm6, %ymm11 # ymm11 = ymm6[0],ymm11[1],ymm6[2],ymm11[3],ymm6[4],ymm11[5],ymm6[6],ymm11[7],ymm6[8],ymm11[9],ymm6[10],ymm11[11],ymm6[12],ymm11[13],ymm6[14],ymm11[15] + vmovdqa 0x160(%rsi), %ymm15 + vmovdqa 0x180(%rsi), %ymm2 + vpmullw %ymm15, %ymm5, %ymm12 + vpmullw %ymm15, %ymm7, %ymm13 + vpmullw %ymm15, %ymm9, %ymm14 + vpmullw %ymm15, %ymm11, %ymm15 + vpmulhw %ymm2, %ymm5, %ymm5 + vpmulhw %ymm2, %ymm7, %ymm7 + vpmulhw %ymm2, %ymm9, %ymm9 + vpmulhw %ymm2, %ymm11, %ymm11 + vpslld $0x10, %ymm10, %ymm6 + vpblendw $0xaa, %ymm6, %ymm3, %ymm6 # ymm6 = ymm3[0],ymm6[1],ymm3[2],ymm6[3],ymm3[4],ymm6[5],ymm3[6],ymm6[7],ymm3[8],ymm6[9],ymm3[10],ymm6[11],ymm3[12],ymm6[13],ymm3[14],ymm6[15] + vpsrld $0x10, %ymm3, %ymm3 + vpblendw $0xaa, %ymm10, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm10[1],ymm3[2],ymm10[3],ymm3[4],ymm10[5],ymm3[6],ymm10[7],ymm3[8],ymm10[9],ymm3[10],ymm10[11],ymm3[12],ymm10[13],ymm3[14],ymm10[15] + vpslld $0x10, %ymm8, %ymm3 + vpblendw $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7],ymm4[8],ymm3[9],ymm4[10],ymm3[11],ymm4[12],ymm3[13],ymm4[14],ymm3[15] + vpsrld $0x10, %ymm4, %ymm4 + vpblendw $0xaa, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm8[1],ymm4[2],ymm8[3],ymm4[4],ymm8[5],ymm4[6],ymm8[7],ymm4[8],ymm8[9],ymm4[10],ymm8[11],ymm4[12],ymm8[13],ymm4[14],ymm8[15] + vpmulhw %ymm0, %ymm12, %ymm12 + vpmulhw %ymm0, %ymm13, %ymm13 + vpmulhw %ymm0, %ymm14, %ymm14 + vpmulhw %ymm0, %ymm15, %ymm15 + vpaddw %ymm5, %ymm6, %ymm4 + vpsubw %ymm5, %ymm6, %ymm5 + vpaddw %ymm7, %ymm10, %ymm6 + vpsubw %ymm7, %ymm10, %ymm7 + vpaddw %ymm9, %ymm3, %ymm10 + vpsubw %ymm9, %ymm3, %ymm9 + vpaddw %ymm11, %ymm8, %ymm3 + vpsubw %ymm11, %ymm8, %ymm11 + vpsubw %ymm12, %ymm4, %ymm4 + vpaddw %ymm12, %ymm5, %ymm5 + vpsubw %ymm13, %ymm6, %ymm6 + vpaddw %ymm13, %ymm7, %ymm7 + vpsubw %ymm14, %ymm10, %ymm10 + vpaddw %ymm14, %ymm9, %ymm9 + vpsubw %ymm15, %ymm3, %ymm3 + vpaddw %ymm15, %ymm11, %ymm11 + vmovdqa 0x1a0(%rsi), %ymm14 + vmovdqa 0x1e0(%rsi), %ymm15 + vmovdqa 0x1c0(%rsi), %ymm8 + vmovdqa 0x200(%rsi), %ymm2 + vpmullw %ymm14, %ymm10, %ymm12 + vpmullw %ymm14, %ymm3, %ymm13 + vpmullw %ymm15, %ymm9, %ymm14 + vpmullw %ymm15, %ymm11, %ymm15 + vpmulhw %ymm8, %ymm10, %ymm10 + vpmulhw %ymm8, %ymm3, %ymm3 + vpmulhw %ymm2, %ymm9, %ymm9 + vpmulhw %ymm2, %ymm11, %ymm11 + vpmulhw %ymm0, %ymm12, %ymm12 + vpmulhw %ymm0, %ymm13, %ymm13 + vpmulhw %ymm0, %ymm14, %ymm14 + vpmulhw %ymm0, %ymm15, %ymm15 + vpaddw %ymm10, %ymm4, %ymm8 + vpsubw %ymm10, %ymm4, %ymm10 + vpaddw %ymm3, %ymm6, %ymm4 + vpsubw %ymm3, %ymm6, %ymm3 + vpaddw %ymm9, %ymm5, %ymm6 + vpsubw %ymm9, %ymm5, %ymm9 + vpaddw %ymm11, %ymm7, %ymm5 + vpsubw %ymm11, %ymm7, %ymm11 + vpsubw %ymm12, %ymm8, %ymm8 + vpaddw %ymm12, %ymm10, %ymm10 + vpsubw %ymm13, %ymm4, %ymm4 + vpaddw %ymm13, %ymm3, %ymm3 + vpsubw %ymm14, %ymm6, %ymm6 + vpaddw %ymm14, %ymm9, %ymm9 + vpsubw %ymm15, %ymm5, %ymm5 + vpaddw %ymm15, %ymm11, %ymm11 + vmovdqa %ymm8, (%rdi) + vmovdqa %ymm4, 0x20(%rdi) + vmovdqa %ymm10, 0x40(%rdi) + vmovdqa %ymm3, 0x60(%rdi) + vmovdqa %ymm6, 0x80(%rdi) + vmovdqa %ymm5, 0xa0(%rdi) + vmovdqa %ymm9, 0xc0(%rdi) + vmovdqa %ymm11, 0xe0(%rdi) + vmovdqa 0x220(%rsi), %ymm15 + vmovdqa 0x180(%rdi), %ymm8 + vmovdqa 0x1a0(%rdi), %ymm9 + vmovdqa 0x1c0(%rdi), %ymm10 + vmovdqa 0x1e0(%rdi), %ymm11 + vmovdqa 0x240(%rsi), %ymm2 + vpmullw %ymm15, %ymm8, %ymm12 + vpmullw %ymm15, %ymm9, %ymm13 + vpmullw %ymm15, %ymm10, %ymm14 + vpmullw %ymm15, %ymm11, %ymm15 + vpmulhw %ymm2, %ymm8, %ymm8 + vpmulhw %ymm2, %ymm9, %ymm9 + vpmulhw %ymm2, %ymm10, %ymm10 + vpmulhw %ymm2, %ymm11, %ymm11 + vmovdqa 0x100(%rdi), %ymm4 + vmovdqa 0x120(%rdi), %ymm5 + vmovdqa 0x140(%rdi), %ymm6 + vmovdqa 0x160(%rdi), %ymm7 + vpmulhw %ymm0, %ymm12, %ymm12 + vpmulhw %ymm0, %ymm13, %ymm13 + vpmulhw %ymm0, %ymm14, %ymm14 + vpmulhw %ymm0, %ymm15, %ymm15 + vpaddw %ymm8, %ymm4, %ymm3 + vpsubw %ymm8, %ymm4, %ymm8 + vpaddw %ymm9, %ymm5, %ymm4 + vpsubw %ymm9, %ymm5, %ymm9 + vpaddw %ymm10, %ymm6, %ymm5 + vpsubw %ymm10, %ymm6, %ymm10 + vpaddw %ymm11, %ymm7, %ymm6 + vpsubw %ymm11, %ymm7, %ymm11 + vpsubw %ymm12, %ymm3, %ymm3 + vpaddw %ymm12, %ymm8, %ymm8 + vpsubw %ymm13, %ymm4, %ymm4 + vpaddw %ymm13, %ymm9, %ymm9 + vpsubw %ymm14, %ymm5, %ymm5 + vpaddw %ymm14, %ymm10, %ymm10 + vpsubw %ymm15, %ymm6, %ymm6 + vpaddw %ymm15, %ymm11, %ymm11 + vperm2i128 $0x20, %ymm10, %ymm5, %ymm7 # ymm7 = ymm5[0,1],ymm10[0,1] + vperm2i128 $0x31, %ymm10, %ymm5, %ymm10 # ymm10 = ymm5[2,3],ymm10[2,3] + vperm2i128 $0x20, %ymm11, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm6, %ymm11 # ymm11 = ymm6[2,3],ymm11[2,3] + vmovdqa 0x260(%rsi), %ymm15 + vmovdqa 0x280(%rsi), %ymm2 + vpmullw %ymm15, %ymm7, %ymm12 + vpmullw %ymm15, %ymm10, %ymm13 + vpmullw %ymm15, %ymm5, %ymm14 + vpmullw %ymm15, %ymm11, %ymm15 + vpmulhw %ymm2, %ymm7, %ymm7 + vpmulhw %ymm2, %ymm10, %ymm10 + vpmulhw %ymm2, %ymm5, %ymm5 + vpmulhw %ymm2, %ymm11, %ymm11 + vperm2i128 $0x20, %ymm8, %ymm3, %ymm6 # ymm6 = ymm3[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm9, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm4, %ymm9 # ymm9 = ymm4[2,3],ymm9[2,3] + vpmulhw %ymm0, %ymm12, %ymm12 + vpmulhw %ymm0, %ymm13, %ymm13 + vpmulhw %ymm0, %ymm14, %ymm14 + vpmulhw %ymm0, %ymm15, %ymm15 + vpaddw %ymm7, %ymm6, %ymm4 + vpsubw %ymm7, %ymm6, %ymm7 + vpaddw %ymm10, %ymm8, %ymm6 + vpsubw %ymm10, %ymm8, %ymm10 + vpaddw %ymm5, %ymm3, %ymm8 + vpsubw %ymm5, %ymm3, %ymm5 + vpaddw %ymm11, %ymm9, %ymm3 + vpsubw %ymm11, %ymm9, %ymm11 + vpsubw %ymm12, %ymm4, %ymm4 + vpaddw %ymm12, %ymm7, %ymm7 + vpsubw %ymm13, %ymm6, %ymm6 + vpaddw %ymm13, %ymm10, %ymm10 + vpsubw %ymm14, %ymm8, %ymm8 + vpaddw %ymm14, %ymm5, %ymm5 + vpsubw %ymm15, %ymm3, %ymm3 + vpaddw %ymm15, %ymm11, %ymm11 + vpunpcklqdq %ymm5, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm5[0],ymm8[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm8, %ymm5 # ymm5 = ymm8[1],ymm5[1],ymm8[3],ymm5[3] + vpunpcklqdq %ymm11, %ymm3, %ymm8 # ymm8 = ymm3[0],ymm11[0],ymm3[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm3, %ymm11 # ymm11 = ymm3[1],ymm11[1],ymm3[3],ymm11[3] + vmovdqa 0x2a0(%rsi), %ymm15 + vmovdqa 0x2c0(%rsi), %ymm2 + vpmullw %ymm15, %ymm9, %ymm12 + vpmullw %ymm15, %ymm5, %ymm13 + vpmullw %ymm15, %ymm8, %ymm14 + vpmullw %ymm15, %ymm11, %ymm15 + vpmulhw %ymm2, %ymm9, %ymm9 + vpmulhw %ymm2, %ymm5, %ymm5 + vpmulhw %ymm2, %ymm8, %ymm8 + vpmulhw %ymm2, %ymm11, %ymm11 + vpunpcklqdq %ymm7, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm7[0],ymm4[2],ymm7[2] + vpunpckhqdq %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[1],ymm7[1],ymm4[3],ymm7[3] + vpunpcklqdq %ymm10, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm10[0],ymm6[2],ymm10[2] + vpunpckhqdq %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[1],ymm10[1],ymm6[3],ymm10[3] + vpmulhw %ymm0, %ymm12, %ymm12 + vpmulhw %ymm0, %ymm13, %ymm13 + vpmulhw %ymm0, %ymm14, %ymm14 + vpmulhw %ymm0, %ymm15, %ymm15 + vpaddw %ymm9, %ymm3, %ymm6 + vpsubw %ymm9, %ymm3, %ymm9 + vpaddw %ymm5, %ymm7, %ymm3 + vpsubw %ymm5, %ymm7, %ymm5 + vpaddw %ymm8, %ymm4, %ymm7 + vpsubw %ymm8, %ymm4, %ymm8 + vpaddw %ymm11, %ymm10, %ymm4 + vpsubw %ymm11, %ymm10, %ymm11 + vpsubw %ymm12, %ymm6, %ymm6 + vpaddw %ymm12, %ymm9, %ymm9 + vpsubw %ymm13, %ymm3, %ymm3 + vpaddw %ymm13, %ymm5, %ymm5 + vpsubw %ymm14, %ymm7, %ymm7 + vpaddw %ymm14, %ymm8, %ymm8 + vpsubw %ymm15, %ymm4, %ymm4 + vpaddw %ymm15, %ymm11, %ymm11 + vmovsldup %ymm8, %ymm10 # ymm10 = ymm8[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm10, %ymm7, %ymm10 # ymm10 = ymm7[0],ymm10[1],ymm7[2],ymm10[3],ymm7[4],ymm10[5],ymm7[6],ymm10[7] + vpsrlq $0x20, %ymm7, %ymm7 + vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7] + vmovsldup %ymm11, %ymm7 # ymm7 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[0],ymm7[1],ymm4[2],ymm7[3],ymm4[4],ymm7[5],ymm4[6],ymm7[7] + vpsrlq $0x20, %ymm4, %ymm4 + vpblendd $0xaa, %ymm11, %ymm4, %ymm11 # ymm11 = ymm4[0],ymm11[1],ymm4[2],ymm11[3],ymm4[4],ymm11[5],ymm4[6],ymm11[7] + vmovdqa 0x2e0(%rsi), %ymm15 + vmovdqa 0x300(%rsi), %ymm2 + vpmullw %ymm15, %ymm10, %ymm12 + vpmullw %ymm15, %ymm8, %ymm13 + vpmullw %ymm15, %ymm7, %ymm14 + vpmullw %ymm15, %ymm11, %ymm15 + vpmulhw %ymm2, %ymm10, %ymm10 + vpmulhw %ymm2, %ymm8, %ymm8 + vpmulhw %ymm2, %ymm7, %ymm7 + vpmulhw %ymm2, %ymm11, %ymm11 + vmovsldup %ymm9, %ymm4 # ymm4 = ymm9[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7] + vpsrlq $0x20, %ymm6, %ymm6 + vpblendd $0xaa, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[0],ymm9[1],ymm6[2],ymm9[3],ymm6[4],ymm9[5],ymm6[6],ymm9[7] + vmovsldup %ymm5, %ymm6 # ymm6 = ymm5[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm6, %ymm3, %ymm6 # ymm6 = ymm3[0],ymm6[1],ymm3[2],ymm6[3],ymm3[4],ymm6[5],ymm3[6],ymm6[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7] + vpmulhw %ymm0, %ymm12, %ymm12 + vpmulhw %ymm0, %ymm13, %ymm13 + vpmulhw %ymm0, %ymm14, %ymm14 + vpmulhw %ymm0, %ymm15, %ymm15 + vpaddw %ymm10, %ymm4, %ymm3 + vpsubw %ymm10, %ymm4, %ymm10 + vpaddw %ymm8, %ymm9, %ymm4 + vpsubw %ymm8, %ymm9, %ymm8 + vpaddw %ymm7, %ymm6, %ymm9 + vpsubw %ymm7, %ymm6, %ymm7 + vpaddw %ymm11, %ymm5, %ymm6 + vpsubw %ymm11, %ymm5, %ymm11 + vpsubw %ymm12, %ymm3, %ymm3 + vpaddw %ymm12, %ymm10, %ymm10 + vpsubw %ymm13, %ymm4, %ymm4 + vpaddw %ymm13, %ymm8, %ymm8 + vpsubw %ymm14, %ymm9, %ymm9 + vpaddw %ymm14, %ymm7, %ymm7 + vpsubw %ymm15, %ymm6, %ymm6 + vpaddw %ymm15, %ymm11, %ymm11 + vpslld $0x10, %ymm7, %ymm5 + vpblendw $0xaa, %ymm5, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm5[1],ymm9[2],ymm5[3],ymm9[4],ymm5[5],ymm9[6],ymm5[7],ymm9[8],ymm5[9],ymm9[10],ymm5[11],ymm9[12],ymm5[13],ymm9[14],ymm5[15] + vpsrld $0x10, %ymm9, %ymm9 + vpblendw $0xaa, %ymm7, %ymm9, %ymm7 # ymm7 = ymm9[0],ymm7[1],ymm9[2],ymm7[3],ymm9[4],ymm7[5],ymm9[6],ymm7[7],ymm9[8],ymm7[9],ymm9[10],ymm7[11],ymm9[12],ymm7[13],ymm9[14],ymm7[15] + vpslld $0x10, %ymm11, %ymm9 + vpblendw $0xaa, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[0],ymm9[1],ymm6[2],ymm9[3],ymm6[4],ymm9[5],ymm6[6],ymm9[7],ymm6[8],ymm9[9],ymm6[10],ymm9[11],ymm6[12],ymm9[13],ymm6[14],ymm9[15] + vpsrld $0x10, %ymm6, %ymm6 + vpblendw $0xaa, %ymm11, %ymm6, %ymm11 # ymm11 = ymm6[0],ymm11[1],ymm6[2],ymm11[3],ymm6[4],ymm11[5],ymm6[6],ymm11[7],ymm6[8],ymm11[9],ymm6[10],ymm11[11],ymm6[12],ymm11[13],ymm6[14],ymm11[15] + vmovdqa 0x320(%rsi), %ymm15 + vmovdqa 0x340(%rsi), %ymm2 + vpmullw %ymm15, %ymm5, %ymm12 + vpmullw %ymm15, %ymm7, %ymm13 + vpmullw %ymm15, %ymm9, %ymm14 + vpmullw %ymm15, %ymm11, %ymm15 + vpmulhw %ymm2, %ymm5, %ymm5 + vpmulhw %ymm2, %ymm7, %ymm7 + vpmulhw %ymm2, %ymm9, %ymm9 + vpmulhw %ymm2, %ymm11, %ymm11 + vpslld $0x10, %ymm10, %ymm6 + vpblendw $0xaa, %ymm6, %ymm3, %ymm6 # ymm6 = ymm3[0],ymm6[1],ymm3[2],ymm6[3],ymm3[4],ymm6[5],ymm3[6],ymm6[7],ymm3[8],ymm6[9],ymm3[10],ymm6[11],ymm3[12],ymm6[13],ymm3[14],ymm6[15] + vpsrld $0x10, %ymm3, %ymm3 + vpblendw $0xaa, %ymm10, %ymm3, %ymm10 # ymm10 = ymm3[0],ymm10[1],ymm3[2],ymm10[3],ymm3[4],ymm10[5],ymm3[6],ymm10[7],ymm3[8],ymm10[9],ymm3[10],ymm10[11],ymm3[12],ymm10[13],ymm3[14],ymm10[15] + vpslld $0x10, %ymm8, %ymm3 + vpblendw $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7],ymm4[8],ymm3[9],ymm4[10],ymm3[11],ymm4[12],ymm3[13],ymm4[14],ymm3[15] + vpsrld $0x10, %ymm4, %ymm4 + vpblendw $0xaa, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm8[1],ymm4[2],ymm8[3],ymm4[4],ymm8[5],ymm4[6],ymm8[7],ymm4[8],ymm8[9],ymm4[10],ymm8[11],ymm4[12],ymm8[13],ymm4[14],ymm8[15] + vpmulhw %ymm0, %ymm12, %ymm12 + vpmulhw %ymm0, %ymm13, %ymm13 + vpmulhw %ymm0, %ymm14, %ymm14 + vpmulhw %ymm0, %ymm15, %ymm15 + vpaddw %ymm5, %ymm6, %ymm4 + vpsubw %ymm5, %ymm6, %ymm5 + vpaddw %ymm7, %ymm10, %ymm6 + vpsubw %ymm7, %ymm10, %ymm7 + vpaddw %ymm9, %ymm3, %ymm10 + vpsubw %ymm9, %ymm3, %ymm9 + vpaddw %ymm11, %ymm8, %ymm3 + vpsubw %ymm11, %ymm8, %ymm11 + vpsubw %ymm12, %ymm4, %ymm4 + vpaddw %ymm12, %ymm5, %ymm5 + vpsubw %ymm13, %ymm6, %ymm6 + vpaddw %ymm13, %ymm7, %ymm7 + vpsubw %ymm14, %ymm10, %ymm10 + vpaddw %ymm14, %ymm9, %ymm9 + vpsubw %ymm15, %ymm3, %ymm3 + vpaddw %ymm15, %ymm11, %ymm11 + vmovdqa 0x360(%rsi), %ymm14 + vmovdqa 0x3a0(%rsi), %ymm15 + vmovdqa 0x380(%rsi), %ymm8 + vmovdqa 0x3c0(%rsi), %ymm2 + vpmullw %ymm14, %ymm10, %ymm12 + vpmullw %ymm14, %ymm3, %ymm13 + vpmullw %ymm15, %ymm9, %ymm14 + vpmullw %ymm15, %ymm11, %ymm15 + vpmulhw %ymm8, %ymm10, %ymm10 + vpmulhw %ymm8, %ymm3, %ymm3 + vpmulhw %ymm2, %ymm9, %ymm9 + vpmulhw %ymm2, %ymm11, %ymm11 + vpmulhw %ymm0, %ymm12, %ymm12 + vpmulhw %ymm0, %ymm13, %ymm13 + vpmulhw %ymm0, %ymm14, %ymm14 + vpmulhw %ymm0, %ymm15, %ymm15 + vpaddw %ymm10, %ymm4, %ymm8 + vpsubw %ymm10, %ymm4, %ymm10 + vpaddw %ymm3, %ymm6, %ymm4 + vpsubw %ymm3, %ymm6, %ymm3 + vpaddw %ymm9, %ymm5, %ymm6 + vpsubw %ymm9, %ymm5, %ymm9 + vpaddw %ymm11, %ymm7, %ymm5 + vpsubw %ymm11, %ymm7, %ymm11 + vpsubw %ymm12, %ymm8, %ymm8 + vpaddw %ymm12, %ymm10, %ymm10 + vpsubw %ymm13, %ymm4, %ymm4 + vpaddw %ymm13, %ymm3, %ymm3 + vpsubw %ymm14, %ymm6, %ymm6 + vpaddw %ymm14, %ymm9, %ymm9 + vpsubw %ymm15, %ymm5, %ymm5 + vpaddw %ymm15, %ymm11, %ymm11 + vmovdqa %ymm8, 0x100(%rdi) + vmovdqa %ymm4, 0x120(%rdi) + vmovdqa %ymm10, 0x140(%rdi) + vmovdqa %ymm3, 0x160(%rdi) + vmovdqa %ymm6, 0x180(%rdi) + vmovdqa %ymm5, 0x1a0(%rdi) + vmovdqa %ymm9, 0x1c0(%rdi) + vmovdqa %ymm11, 0x1e0(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(ntt_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_nttfrombytes_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_nttfrombytes_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_nttfrombytes_avx2_asm.S @@ -0,0 +1,217 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Kyber optimized AVX2 implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Kyber implementation @[REF_AVX2]. + */ + +#include "../../../common.h" + +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + (!defined(MLK_CONFIG_NO_ENCAPS_API) || \ + !defined(MLK_CONFIG_NO_DECAPS_API)) +/*yaml + Name: nttfrombytes_avx2_asm + Description: x86_64 AVX2 polynomial deserialization in NTT domain + Signature: void mlk_nttfrombytes_avx2_asm(int16_t *r, const uint8_t *a) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 512 + permissions: write-only + c_parameter: int16_t *r + description: Output polynomial (256 x int16_t) + rsi: + type: buffer + size_bytes: 384 + permissions: read-only + c_parameter: const uint8_t *a + description: Input byte array (MLKEM_POLYBYTES = 384) +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_nttfrombytes_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(nttfrombytes_avx2_asm) +MLK_ASM_FN_SYMBOL(nttfrombytes_avx2_asm) + + .cfi_startproc + movl $0xfff0fff, %eax # imm = 0xFFF0FFF + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + vmovdqu (%rsi), %ymm4 + vmovdqu 0x20(%rsi), %ymm5 + vmovdqu 0x40(%rsi), %ymm6 + vmovdqu 0x60(%rsi), %ymm7 + vmovdqu 0x80(%rsi), %ymm8 + vmovdqu 0xa0(%rsi), %ymm9 + vperm2i128 $0x20, %ymm7, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm7[0,1] + vperm2i128 $0x31, %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[2,3],ymm7[2,3] + vperm2i128 $0x20, %ymm8, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm5, %ymm8 # ymm8 = ymm5[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm9, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[2,3],ymm9[2,3] + vpunpcklqdq %ymm8, %ymm3, %ymm6 # ymm6 = ymm3[0],ymm8[0],ymm3[2],ymm8[2] + vpunpckhqdq %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[1],ymm8[1],ymm3[3],ymm8[3] + vpunpcklqdq %ymm5, %ymm7, %ymm3 # ymm3 = ymm7[0],ymm5[0],ymm7[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm7, %ymm5 # ymm5 = ymm7[1],ymm5[1],ymm7[3],ymm5[3] + vpunpcklqdq %ymm9, %ymm4, %ymm7 # ymm7 = ymm4[0],ymm9[0],ymm4[2],ymm9[2] + vpunpckhqdq %ymm9, %ymm4, %ymm9 # ymm9 = ymm4[1],ymm9[1],ymm4[3],ymm9[3] + vmovsldup %ymm5, %ymm4 # ymm4 = ymm5[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7] + vpsrlq $0x20, %ymm6, %ymm6 + vpblendd $0xaa, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[0],ymm5[1],ymm6[2],ymm5[3],ymm6[4],ymm5[5],ymm6[6],ymm5[7] + vmovsldup %ymm7, %ymm6 # ymm6 = ymm7[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7] + vpsrlq $0x20, %ymm8, %ymm8 + vpblendd $0xaa, %ymm7, %ymm8, %ymm7 # ymm7 = ymm8[0],ymm7[1],ymm8[2],ymm7[3],ymm8[4],ymm7[5],ymm8[6],ymm7[7] + vmovsldup %ymm9, %ymm8 # ymm8 = ymm9[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[0],ymm8[1],ymm3[2],ymm8[3],ymm3[4],ymm8[5],ymm3[6],ymm8[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm9, %ymm3, %ymm9 # ymm9 = ymm3[0],ymm9[1],ymm3[2],ymm9[3],ymm3[4],ymm9[5],ymm3[6],ymm9[7] + vpslld $0x10, %ymm7, %ymm10 + vpblendw $0xaa, %ymm10, %ymm4, %ymm10 # ymm10 = ymm4[0],ymm10[1],ymm4[2],ymm10[3],ymm4[4],ymm10[5],ymm4[6],ymm10[7],ymm4[8],ymm10[9],ymm4[10],ymm10[11],ymm4[12],ymm10[13],ymm4[14],ymm10[15] + vpsrld $0x10, %ymm4, %ymm4 + vpblendw $0xaa, %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[0],ymm7[1],ymm4[2],ymm7[3],ymm4[4],ymm7[5],ymm4[6],ymm7[7],ymm4[8],ymm7[9],ymm4[10],ymm7[11],ymm4[12],ymm7[13],ymm4[14],ymm7[15] + vpslld $0x10, %ymm8, %ymm4 + vpblendw $0xaa, %ymm4, %ymm5, %ymm4 # ymm4 = ymm5[0],ymm4[1],ymm5[2],ymm4[3],ymm5[4],ymm4[5],ymm5[6],ymm4[7],ymm5[8],ymm4[9],ymm5[10],ymm4[11],ymm5[12],ymm4[13],ymm5[14],ymm4[15] + vpsrld $0x10, %ymm5, %ymm5 + vpblendw $0xaa, %ymm8, %ymm5, %ymm8 # ymm8 = ymm5[0],ymm8[1],ymm5[2],ymm8[3],ymm5[4],ymm8[5],ymm5[6],ymm8[7],ymm5[8],ymm8[9],ymm5[10],ymm8[11],ymm5[12],ymm8[13],ymm5[14],ymm8[15] + vpslld $0x10, %ymm9, %ymm5 + vpblendw $0xaa, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[0],ymm5[1],ymm6[2],ymm5[3],ymm6[4],ymm5[5],ymm6[6],ymm5[7],ymm6[8],ymm5[9],ymm6[10],ymm5[11],ymm6[12],ymm5[13],ymm6[14],ymm5[15] + vpsrld $0x10, %ymm6, %ymm6 + vpblendw $0xaa, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[0],ymm9[1],ymm6[2],ymm9[3],ymm6[4],ymm9[5],ymm6[6],ymm9[7],ymm6[8],ymm9[9],ymm6[10],ymm9[11],ymm6[12],ymm9[13],ymm6[14],ymm9[15] + vpsrlw $0xc, %ymm10, %ymm11 + vpsllw $0x4, %ymm7, %ymm12 + vpor %ymm11, %ymm12, %ymm11 + vpand %ymm0, %ymm10, %ymm10 + vpand %ymm0, %ymm11, %ymm11 + vpsrlw $0x8, %ymm7, %ymm12 + vpsllw $0x8, %ymm4, %ymm13 + vpor %ymm12, %ymm13, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpsrlw $0x4, %ymm4, %ymm13 + vpand %ymm0, %ymm13, %ymm13 + vpsrlw $0xc, %ymm8, %ymm14 + vpsllw $0x4, %ymm5, %ymm15 + vpor %ymm14, %ymm15, %ymm14 + vpand %ymm0, %ymm8, %ymm8 + vpand %ymm0, %ymm14, %ymm14 + vpsrlw $0x8, %ymm5, %ymm15 + vpsllw $0x8, %ymm9, %ymm1 + vpor %ymm15, %ymm1, %ymm15 + vpand %ymm0, %ymm15, %ymm15 + vpsrlw $0x4, %ymm9, %ymm1 + vpand %ymm0, %ymm1, %ymm1 + vmovdqa %ymm10, (%rdi) + vmovdqa %ymm11, 0x20(%rdi) + vmovdqa %ymm12, 0x40(%rdi) + vmovdqa %ymm13, 0x60(%rdi) + vmovdqa %ymm8, 0x80(%rdi) + vmovdqa %ymm14, 0xa0(%rdi) + vmovdqa %ymm15, 0xc0(%rdi) + vmovdqa %ymm1, 0xe0(%rdi) + vmovdqu 0xc0(%rsi), %ymm4 + vmovdqu 0xe0(%rsi), %ymm5 + vmovdqu 0x100(%rsi), %ymm6 + vmovdqu 0x120(%rsi), %ymm7 + vmovdqu 0x140(%rsi), %ymm8 + vmovdqu 0x160(%rsi), %ymm9 + vperm2i128 $0x20, %ymm7, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm7[0,1] + vperm2i128 $0x31, %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[2,3],ymm7[2,3] + vperm2i128 $0x20, %ymm8, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm5, %ymm8 # ymm8 = ymm5[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm9, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[2,3],ymm9[2,3] + vpunpcklqdq %ymm8, %ymm3, %ymm6 # ymm6 = ymm3[0],ymm8[0],ymm3[2],ymm8[2] + vpunpckhqdq %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[1],ymm8[1],ymm3[3],ymm8[3] + vpunpcklqdq %ymm5, %ymm7, %ymm3 # ymm3 = ymm7[0],ymm5[0],ymm7[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm7, %ymm5 # ymm5 = ymm7[1],ymm5[1],ymm7[3],ymm5[3] + vpunpcklqdq %ymm9, %ymm4, %ymm7 # ymm7 = ymm4[0],ymm9[0],ymm4[2],ymm9[2] + vpunpckhqdq %ymm9, %ymm4, %ymm9 # ymm9 = ymm4[1],ymm9[1],ymm4[3],ymm9[3] + vmovsldup %ymm5, %ymm4 # ymm4 = ymm5[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7] + vpsrlq $0x20, %ymm6, %ymm6 + vpblendd $0xaa, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[0],ymm5[1],ymm6[2],ymm5[3],ymm6[4],ymm5[5],ymm6[6],ymm5[7] + vmovsldup %ymm7, %ymm6 # ymm6 = ymm7[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7] + vpsrlq $0x20, %ymm8, %ymm8 + vpblendd $0xaa, %ymm7, %ymm8, %ymm7 # ymm7 = ymm8[0],ymm7[1],ymm8[2],ymm7[3],ymm8[4],ymm7[5],ymm8[6],ymm7[7] + vmovsldup %ymm9, %ymm8 # ymm8 = ymm9[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[0],ymm8[1],ymm3[2],ymm8[3],ymm3[4],ymm8[5],ymm3[6],ymm8[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm9, %ymm3, %ymm9 # ymm9 = ymm3[0],ymm9[1],ymm3[2],ymm9[3],ymm3[4],ymm9[5],ymm3[6],ymm9[7] + vpslld $0x10, %ymm7, %ymm10 + vpblendw $0xaa, %ymm10, %ymm4, %ymm10 # ymm10 = ymm4[0],ymm10[1],ymm4[2],ymm10[3],ymm4[4],ymm10[5],ymm4[6],ymm10[7],ymm4[8],ymm10[9],ymm4[10],ymm10[11],ymm4[12],ymm10[13],ymm4[14],ymm10[15] + vpsrld $0x10, %ymm4, %ymm4 + vpblendw $0xaa, %ymm7, %ymm4, %ymm7 # ymm7 = ymm4[0],ymm7[1],ymm4[2],ymm7[3],ymm4[4],ymm7[5],ymm4[6],ymm7[7],ymm4[8],ymm7[9],ymm4[10],ymm7[11],ymm4[12],ymm7[13],ymm4[14],ymm7[15] + vpslld $0x10, %ymm8, %ymm4 + vpblendw $0xaa, %ymm4, %ymm5, %ymm4 # ymm4 = ymm5[0],ymm4[1],ymm5[2],ymm4[3],ymm5[4],ymm4[5],ymm5[6],ymm4[7],ymm5[8],ymm4[9],ymm5[10],ymm4[11],ymm5[12],ymm4[13],ymm5[14],ymm4[15] + vpsrld $0x10, %ymm5, %ymm5 + vpblendw $0xaa, %ymm8, %ymm5, %ymm8 # ymm8 = ymm5[0],ymm8[1],ymm5[2],ymm8[3],ymm5[4],ymm8[5],ymm5[6],ymm8[7],ymm5[8],ymm8[9],ymm5[10],ymm8[11],ymm5[12],ymm8[13],ymm5[14],ymm8[15] + vpslld $0x10, %ymm9, %ymm5 + vpblendw $0xaa, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[0],ymm5[1],ymm6[2],ymm5[3],ymm6[4],ymm5[5],ymm6[6],ymm5[7],ymm6[8],ymm5[9],ymm6[10],ymm5[11],ymm6[12],ymm5[13],ymm6[14],ymm5[15] + vpsrld $0x10, %ymm6, %ymm6 + vpblendw $0xaa, %ymm9, %ymm6, %ymm9 # ymm9 = ymm6[0],ymm9[1],ymm6[2],ymm9[3],ymm6[4],ymm9[5],ymm6[6],ymm9[7],ymm6[8],ymm9[9],ymm6[10],ymm9[11],ymm6[12],ymm9[13],ymm6[14],ymm9[15] + vpsrlw $0xc, %ymm10, %ymm11 + vpsllw $0x4, %ymm7, %ymm12 + vpor %ymm11, %ymm12, %ymm11 + vpand %ymm0, %ymm10, %ymm10 + vpand %ymm0, %ymm11, %ymm11 + vpsrlw $0x8, %ymm7, %ymm12 + vpsllw $0x8, %ymm4, %ymm13 + vpor %ymm12, %ymm13, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpsrlw $0x4, %ymm4, %ymm13 + vpand %ymm0, %ymm13, %ymm13 + vpsrlw $0xc, %ymm8, %ymm14 + vpsllw $0x4, %ymm5, %ymm15 + vpor %ymm14, %ymm15, %ymm14 + vpand %ymm0, %ymm8, %ymm8 + vpand %ymm0, %ymm14, %ymm14 + vpsrlw $0x8, %ymm5, %ymm15 + vpsllw $0x8, %ymm9, %ymm1 + vpor %ymm15, %ymm1, %ymm15 + vpand %ymm0, %ymm15, %ymm15 + vpsrlw $0x4, %ymm9, %ymm1 + vpand %ymm0, %ymm1, %ymm1 + vmovdqa %ymm10, 0x100(%rdi) + vmovdqa %ymm11, 0x120(%rdi) + vmovdqa %ymm12, 0x140(%rdi) + vmovdqa %ymm13, 0x160(%rdi) + vmovdqa %ymm8, 0x180(%rdi) + vmovdqa %ymm14, 0x1a0(%rdi) + vmovdqa %ymm15, 0x1c0(%rdi) + vmovdqa %ymm1, 0x1e0(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(nttfrombytes_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + && (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_ntttobytes_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_ntttobytes_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_ntttobytes_avx2_asm.S @@ -0,0 +1,205 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Kyber optimized AVX2 implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Kyber implementation @[REF_AVX2]. + */ + +#include "../../../common.h" + +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + (!defined(MLK_CONFIG_NO_KEYPAIR_API) || \ + !defined(MLK_CONFIG_NO_ENCAPS_API)) +/*yaml + Name: ntttobytes_avx2_asm + Description: x86_64 AVX2 polynomial serialization in NTT domain + Signature: void mlk_ntttobytes_avx2_asm(uint8_t *r, const int16_t *a) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 384 + permissions: write-only + c_parameter: uint8_t *r + description: Output byte array (MLKEM_POLYBYTES = 384) + rsi: + type: buffer + size_bytes: 512 + permissions: read-only + c_parameter: const int16_t *a + description: Input polynomial (256 x int16_t) +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_ntttobytes_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(ntttobytes_avx2_asm) +MLK_ASM_FN_SYMBOL(ntttobytes_avx2_asm) + + .cfi_startproc + movl $0xd010d01, %eax # imm = 0xD010D01 + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + vmovdqa (%rsi), %ymm5 + vmovdqa 0x20(%rsi), %ymm6 + vmovdqa 0x40(%rsi), %ymm7 + vmovdqa 0x60(%rsi), %ymm8 + vmovdqa 0x80(%rsi), %ymm9 + vmovdqa 0xa0(%rsi), %ymm10 + vmovdqa 0xc0(%rsi), %ymm11 + vmovdqa 0xe0(%rsi), %ymm12 + vpsllw $0xc, %ymm6, %ymm4 + vpor %ymm4, %ymm5, %ymm4 + vpsrlw $0x4, %ymm6, %ymm5 + vpsllw $0x8, %ymm7, %ymm6 + vpor %ymm5, %ymm6, %ymm5 + vpsrlw $0x8, %ymm7, %ymm6 + vpsllw $0x4, %ymm8, %ymm7 + vpor %ymm6, %ymm7, %ymm6 + vpsllw $0xc, %ymm10, %ymm7 + vpor %ymm7, %ymm9, %ymm7 + vpsrlw $0x4, %ymm10, %ymm8 + vpsllw $0x8, %ymm11, %ymm9 + vpor %ymm8, %ymm9, %ymm8 + vpsrlw $0x8, %ymm11, %ymm9 + vpsllw $0x4, %ymm12, %ymm10 + vpor %ymm9, %ymm10, %ymm9 + vpslld $0x10, %ymm5, %ymm3 + vpblendw $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7],ymm4[8],ymm3[9],ymm4[10],ymm3[11],ymm4[12],ymm3[13],ymm4[14],ymm3[15] + vpsrld $0x10, %ymm4, %ymm4 + vpblendw $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7],ymm4[8],ymm5[9],ymm4[10],ymm5[11],ymm4[12],ymm5[13],ymm4[14],ymm5[15] + vpslld $0x10, %ymm7, %ymm4 + vpblendw $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7],ymm6[8],ymm4[9],ymm6[10],ymm4[11],ymm6[12],ymm4[13],ymm6[14],ymm4[15] + vpsrld $0x10, %ymm6, %ymm6 + vpblendw $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7],ymm6[8],ymm7[9],ymm6[10],ymm7[11],ymm6[12],ymm7[13],ymm6[14],ymm7[15] + vpslld $0x10, %ymm9, %ymm6 + vpblendw $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7],ymm8[8],ymm6[9],ymm8[10],ymm6[11],ymm8[12],ymm6[13],ymm8[14],ymm6[15] + vpsrld $0x10, %ymm8, %ymm8 + vpblendw $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7],ymm8[8],ymm9[9],ymm8[10],ymm9[11],ymm8[12],ymm9[13],ymm8[14],ymm9[15] + vmovsldup %ymm4, %ymm8 # ymm8 = ymm4[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[0],ymm8[1],ymm3[2],ymm8[3],ymm3[4],ymm8[5],ymm3[6],ymm8[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7] + vmovsldup %ymm5, %ymm3 # ymm3 = ymm5[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm3[1],ymm6[2],ymm3[3],ymm6[4],ymm3[5],ymm6[6],ymm3[7] + vpsrlq $0x20, %ymm6, %ymm6 + vpblendd $0xaa, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[0],ymm5[1],ymm6[2],ymm5[3],ymm6[4],ymm5[5],ymm6[6],ymm5[7] + vmovsldup %ymm9, %ymm6 # ymm6 = ymm9[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm6, %ymm7, %ymm6 # ymm6 = ymm7[0],ymm6[1],ymm7[2],ymm6[3],ymm7[4],ymm6[5],ymm7[6],ymm6[7] + vpsrlq $0x20, %ymm7, %ymm7 + vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vpunpcklqdq %ymm3, %ymm8, %ymm7 # ymm7 = ymm8[0],ymm3[0],ymm8[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm8, %ymm3 # ymm3 = ymm8[1],ymm3[1],ymm8[3],ymm3[3] + vpunpcklqdq %ymm4, %ymm6, %ymm8 # ymm8 = ymm6[0],ymm4[0],ymm6[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[1],ymm4[1],ymm6[3],ymm4[3] + vpunpcklqdq %ymm9, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm9[0],ymm5[2],ymm9[2] + vpunpckhqdq %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[1],ymm9[1],ymm5[3],ymm9[3] + vperm2i128 $0x20, %ymm8, %ymm7, %ymm5 # ymm5 = ymm7[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm3, %ymm6, %ymm7 # ymm7 = ymm6[0,1],ymm3[0,1] + vperm2i128 $0x31, %ymm3, %ymm6, %ymm3 # ymm3 = ymm6[2,3],ymm3[2,3] + vperm2i128 $0x20, %ymm9, %ymm4, %ymm6 # ymm6 = ymm4[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm4, %ymm9 # ymm9 = ymm4[2,3],ymm9[2,3] + vmovdqu %ymm5, (%rdi) + vmovdqu %ymm7, 0x20(%rdi) + vmovdqu %ymm6, 0x40(%rdi) + vmovdqu %ymm8, 0x60(%rdi) + vmovdqu %ymm3, 0x80(%rdi) + vmovdqu %ymm9, 0xa0(%rdi) + vmovdqa 0x100(%rsi), %ymm5 + vmovdqa 0x120(%rsi), %ymm6 + vmovdqa 0x140(%rsi), %ymm7 + vmovdqa 0x160(%rsi), %ymm8 + vmovdqa 0x180(%rsi), %ymm9 + vmovdqa 0x1a0(%rsi), %ymm10 + vmovdqa 0x1c0(%rsi), %ymm11 + vmovdqa 0x1e0(%rsi), %ymm12 + vpsllw $0xc, %ymm6, %ymm4 + vpor %ymm4, %ymm5, %ymm4 + vpsrlw $0x4, %ymm6, %ymm5 + vpsllw $0x8, %ymm7, %ymm6 + vpor %ymm5, %ymm6, %ymm5 + vpsrlw $0x8, %ymm7, %ymm6 + vpsllw $0x4, %ymm8, %ymm7 + vpor %ymm6, %ymm7, %ymm6 + vpsllw $0xc, %ymm10, %ymm7 + vpor %ymm7, %ymm9, %ymm7 + vpsrlw $0x4, %ymm10, %ymm8 + vpsllw $0x8, %ymm11, %ymm9 + vpor %ymm8, %ymm9, %ymm8 + vpsrlw $0x8, %ymm11, %ymm9 + vpsllw $0x4, %ymm12, %ymm10 + vpor %ymm9, %ymm10, %ymm9 + vpslld $0x10, %ymm5, %ymm3 + vpblendw $0xaa, %ymm3, %ymm4, %ymm3 # ymm3 = ymm4[0],ymm3[1],ymm4[2],ymm3[3],ymm4[4],ymm3[5],ymm4[6],ymm3[7],ymm4[8],ymm3[9],ymm4[10],ymm3[11],ymm4[12],ymm3[13],ymm4[14],ymm3[15] + vpsrld $0x10, %ymm4, %ymm4 + vpblendw $0xaa, %ymm5, %ymm4, %ymm5 # ymm5 = ymm4[0],ymm5[1],ymm4[2],ymm5[3],ymm4[4],ymm5[5],ymm4[6],ymm5[7],ymm4[8],ymm5[9],ymm4[10],ymm5[11],ymm4[12],ymm5[13],ymm4[14],ymm5[15] + vpslld $0x10, %ymm7, %ymm4 + vpblendw $0xaa, %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[0],ymm4[1],ymm6[2],ymm4[3],ymm6[4],ymm4[5],ymm6[6],ymm4[7],ymm6[8],ymm4[9],ymm6[10],ymm4[11],ymm6[12],ymm4[13],ymm6[14],ymm4[15] + vpsrld $0x10, %ymm6, %ymm6 + vpblendw $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7],ymm6[8],ymm7[9],ymm6[10],ymm7[11],ymm6[12],ymm7[13],ymm6[14],ymm7[15] + vpslld $0x10, %ymm9, %ymm6 + vpblendw $0xaa, %ymm6, %ymm8, %ymm6 # ymm6 = ymm8[0],ymm6[1],ymm8[2],ymm6[3],ymm8[4],ymm6[5],ymm8[6],ymm6[7],ymm8[8],ymm6[9],ymm8[10],ymm6[11],ymm8[12],ymm6[13],ymm8[14],ymm6[15] + vpsrld $0x10, %ymm8, %ymm8 + vpblendw $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7],ymm8[8],ymm9[9],ymm8[10],ymm9[11],ymm8[12],ymm9[13],ymm8[14],ymm9[15] + vmovsldup %ymm4, %ymm8 # ymm8 = ymm4[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm8, %ymm3, %ymm8 # ymm8 = ymm3[0],ymm8[1],ymm3[2],ymm8[3],ymm3[4],ymm8[5],ymm3[6],ymm8[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7] + vmovsldup %ymm5, %ymm3 # ymm3 = ymm5[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm6, %ymm3 # ymm3 = ymm6[0],ymm3[1],ymm6[2],ymm3[3],ymm6[4],ymm3[5],ymm6[6],ymm3[7] + vpsrlq $0x20, %ymm6, %ymm6 + vpblendd $0xaa, %ymm5, %ymm6, %ymm5 # ymm5 = ymm6[0],ymm5[1],ymm6[2],ymm5[3],ymm6[4],ymm5[5],ymm6[6],ymm5[7] + vmovsldup %ymm9, %ymm6 # ymm6 = ymm9[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm6, %ymm7, %ymm6 # ymm6 = ymm7[0],ymm6[1],ymm7[2],ymm6[3],ymm7[4],ymm6[5],ymm7[6],ymm6[7] + vpsrlq $0x20, %ymm7, %ymm7 + vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vpunpcklqdq %ymm3, %ymm8, %ymm7 # ymm7 = ymm8[0],ymm3[0],ymm8[2],ymm3[2] + vpunpckhqdq %ymm3, %ymm8, %ymm3 # ymm3 = ymm8[1],ymm3[1],ymm8[3],ymm3[3] + vpunpcklqdq %ymm4, %ymm6, %ymm8 # ymm8 = ymm6[0],ymm4[0],ymm6[2],ymm4[2] + vpunpckhqdq %ymm4, %ymm6, %ymm4 # ymm4 = ymm6[1],ymm4[1],ymm6[3],ymm4[3] + vpunpcklqdq %ymm9, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm9[0],ymm5[2],ymm9[2] + vpunpckhqdq %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[1],ymm9[1],ymm5[3],ymm9[3] + vperm2i128 $0x20, %ymm8, %ymm7, %ymm5 # ymm5 = ymm7[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm3, %ymm6, %ymm7 # ymm7 = ymm6[0,1],ymm3[0,1] + vperm2i128 $0x31, %ymm3, %ymm6, %ymm3 # ymm3 = ymm6[2,3],ymm3[2,3] + vperm2i128 $0x20, %ymm9, %ymm4, %ymm6 # ymm6 = ymm4[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm4, %ymm9 # ymm9 = ymm4[2,3],ymm9[2,3] + vmovdqu %ymm5, 0xc0(%rdi) + vmovdqu %ymm7, 0xe0(%rdi) + vmovdqu %ymm6, 0x100(%rdi) + vmovdqu %ymm8, 0x120(%rdi) + vmovdqu %ymm3, 0x140(%rdi) + vmovdqu %ymm9, 0x160(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(ntttobytes_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + && (!MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_nttunpack_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_nttunpack_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_nttunpack_avx2_asm.S @@ -0,0 +1,190 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Kyber optimized AVX2 implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Kyber implementation @[REF_AVX2]. + */ + +#include "../../../common.h" + +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) +/*yaml + Name: nttunpack_avx2_asm + Description: x86_64 AVX2 NTT unpack from custom to bitreversed order + Signature: void mlk_nttunpack_avx2_asm(int16_t *r) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 512 + permissions: read/write + c_parameter: int16_t *r + description: Input/output polynomial (256 x int16_t) +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_nttunpack_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(nttunpack_avx2_asm) +MLK_ASM_FN_SYMBOL(nttunpack_avx2_asm) + + .cfi_startproc + vmovdqa (%rdi), %ymm4 + vmovdqa 0x20(%rdi), %ymm5 + vmovdqa 0x40(%rdi), %ymm6 + vmovdqa 0x60(%rdi), %ymm7 + vmovdqa 0x80(%rdi), %ymm8 + vmovdqa 0xa0(%rdi), %ymm9 + vmovdqa 0xc0(%rdi), %ymm10 + vmovdqa 0xe0(%rdi), %ymm11 + vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3] + vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1] + vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3] + vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3] + vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3] + vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2] + vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3] + vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2] + vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3] + vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3] + vmovsldup %ymm8, %ymm9 # ymm9 = ymm8[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vpsrlq $0x20, %ymm7, %ymm7 + vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7] + vmovsldup %ymm6, %ymm7 # ymm7 = ymm6[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7] + vpsrlq $0x20, %ymm5, %ymm5 + vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7] + vmovsldup %ymm4, %ymm5 # ymm5 = ymm4[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7] + vmovsldup %ymm11, %ymm3 # ymm3 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7] + vpsrlq $0x20, %ymm10, %ymm10 + vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7] + vpslld $0x10, %ymm5, %ymm10 + vpblendw $0xaa, %ymm10, %ymm9, %ymm10 # ymm10 = ymm9[0],ymm10[1],ymm9[2],ymm10[3],ymm9[4],ymm10[5],ymm9[6],ymm10[7],ymm9[8],ymm10[9],ymm9[10],ymm10[11],ymm9[12],ymm10[13],ymm9[14],ymm10[15] + vpsrld $0x10, %ymm9, %ymm9 + vpblendw $0xaa, %ymm5, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm5[1],ymm9[2],ymm5[3],ymm9[4],ymm5[5],ymm9[6],ymm5[7],ymm9[8],ymm5[9],ymm9[10],ymm5[11],ymm9[12],ymm5[13],ymm9[14],ymm5[15] + vpslld $0x10, %ymm4, %ymm9 + vpblendw $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7],ymm8[8],ymm9[9],ymm8[10],ymm9[11],ymm8[12],ymm9[13],ymm8[14],ymm9[15] + vpsrld $0x10, %ymm8, %ymm8 + vpblendw $0xaa, %ymm4, %ymm8, %ymm4 # ymm4 = ymm8[0],ymm4[1],ymm8[2],ymm4[3],ymm8[4],ymm4[5],ymm8[6],ymm4[7],ymm8[8],ymm4[9],ymm8[10],ymm4[11],ymm8[12],ymm4[13],ymm8[14],ymm4[15] + vpslld $0x10, %ymm3, %ymm8 + vpblendw $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7],ymm7[8],ymm8[9],ymm7[10],ymm8[11],ymm7[12],ymm8[13],ymm7[14],ymm8[15] + vpsrld $0x10, %ymm7, %ymm7 + vpblendw $0xaa, %ymm3, %ymm7, %ymm3 # ymm3 = ymm7[0],ymm3[1],ymm7[2],ymm3[3],ymm7[4],ymm3[5],ymm7[6],ymm3[7],ymm7[8],ymm3[9],ymm7[10],ymm3[11],ymm7[12],ymm3[13],ymm7[14],ymm3[15] + vpslld $0x10, %ymm11, %ymm7 + vpblendw $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7],ymm6[8],ymm7[9],ymm6[10],ymm7[11],ymm6[12],ymm7[13],ymm6[14],ymm7[15] + vpsrld $0x10, %ymm6, %ymm6 + vpblendw $0xaa, %ymm11, %ymm6, %ymm11 # ymm11 = ymm6[0],ymm11[1],ymm6[2],ymm11[3],ymm6[4],ymm11[5],ymm6[6],ymm11[7],ymm6[8],ymm11[9],ymm6[10],ymm11[11],ymm6[12],ymm11[13],ymm6[14],ymm11[15] + vmovdqa %ymm10, (%rdi) + vmovdqa %ymm5, 0x20(%rdi) + vmovdqa %ymm9, 0x40(%rdi) + vmovdqa %ymm4, 0x60(%rdi) + vmovdqa %ymm8, 0x80(%rdi) + vmovdqa %ymm3, 0xa0(%rdi) + vmovdqa %ymm7, 0xc0(%rdi) + vmovdqa %ymm11, 0xe0(%rdi) + vmovdqa 0x100(%rdi), %ymm4 + vmovdqa 0x120(%rdi), %ymm5 + vmovdqa 0x140(%rdi), %ymm6 + vmovdqa 0x160(%rdi), %ymm7 + vmovdqa 0x180(%rdi), %ymm8 + vmovdqa 0x1a0(%rdi), %ymm9 + vmovdqa 0x1c0(%rdi), %ymm10 + vmovdqa 0x1e0(%rdi), %ymm11 + vperm2i128 $0x20, %ymm8, %ymm4, %ymm3 # ymm3 = ymm4[0,1],ymm8[0,1] + vperm2i128 $0x31, %ymm8, %ymm4, %ymm8 # ymm8 = ymm4[2,3],ymm8[2,3] + vperm2i128 $0x20, %ymm9, %ymm5, %ymm4 # ymm4 = ymm5[0,1],ymm9[0,1] + vperm2i128 $0x31, %ymm9, %ymm5, %ymm9 # ymm9 = ymm5[2,3],ymm9[2,3] + vperm2i128 $0x20, %ymm10, %ymm6, %ymm5 # ymm5 = ymm6[0,1],ymm10[0,1] + vperm2i128 $0x31, %ymm10, %ymm6, %ymm10 # ymm10 = ymm6[2,3],ymm10[2,3] + vperm2i128 $0x20, %ymm11, %ymm7, %ymm6 # ymm6 = ymm7[0,1],ymm11[0,1] + vperm2i128 $0x31, %ymm11, %ymm7, %ymm11 # ymm11 = ymm7[2,3],ymm11[2,3] + vpunpcklqdq %ymm5, %ymm3, %ymm7 # ymm7 = ymm3[0],ymm5[0],ymm3[2],ymm5[2] + vpunpckhqdq %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[1],ymm5[1],ymm3[3],ymm5[3] + vpunpcklqdq %ymm10, %ymm8, %ymm3 # ymm3 = ymm8[0],ymm10[0],ymm8[2],ymm10[2] + vpunpckhqdq %ymm10, %ymm8, %ymm10 # ymm10 = ymm8[1],ymm10[1],ymm8[3],ymm10[3] + vpunpcklqdq %ymm6, %ymm4, %ymm8 # ymm8 = ymm4[0],ymm6[0],ymm4[2],ymm6[2] + vpunpckhqdq %ymm6, %ymm4, %ymm6 # ymm6 = ymm4[1],ymm6[1],ymm4[3],ymm6[3] + vpunpcklqdq %ymm11, %ymm9, %ymm4 # ymm4 = ymm9[0],ymm11[0],ymm9[2],ymm11[2] + vpunpckhqdq %ymm11, %ymm9, %ymm11 # ymm11 = ymm9[1],ymm11[1],ymm9[3],ymm11[3] + vmovsldup %ymm8, %ymm9 # ymm9 = ymm8[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm9, %ymm7, %ymm9 # ymm9 = ymm7[0],ymm9[1],ymm7[2],ymm9[3],ymm7[4],ymm9[5],ymm7[6],ymm9[7] + vpsrlq $0x20, %ymm7, %ymm7 + vpblendd $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7] + vmovsldup %ymm6, %ymm7 # ymm7 = ymm6[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm7, %ymm5, %ymm7 # ymm7 = ymm5[0],ymm7[1],ymm5[2],ymm7[3],ymm5[4],ymm7[5],ymm5[6],ymm7[7] + vpsrlq $0x20, %ymm5, %ymm5 + vpblendd $0xaa, %ymm6, %ymm5, %ymm6 # ymm6 = ymm5[0],ymm6[1],ymm5[2],ymm6[3],ymm5[4],ymm6[5],ymm5[6],ymm6[7] + vmovsldup %ymm4, %ymm5 # ymm5 = ymm4[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm5, %ymm3, %ymm5 # ymm5 = ymm3[0],ymm5[1],ymm3[2],ymm5[3],ymm3[4],ymm5[5],ymm3[6],ymm5[7] + vpsrlq $0x20, %ymm3, %ymm3 + vpblendd $0xaa, %ymm4, %ymm3, %ymm4 # ymm4 = ymm3[0],ymm4[1],ymm3[2],ymm4[3],ymm3[4],ymm4[5],ymm3[6],ymm4[7] + vmovsldup %ymm11, %ymm3 # ymm3 = ymm11[0,0,2,2,4,4,6,6] + vpblendd $0xaa, %ymm3, %ymm10, %ymm3 # ymm3 = ymm10[0],ymm3[1],ymm10[2],ymm3[3],ymm10[4],ymm3[5],ymm10[6],ymm3[7] + vpsrlq $0x20, %ymm10, %ymm10 + vpblendd $0xaa, %ymm11, %ymm10, %ymm11 # ymm11 = ymm10[0],ymm11[1],ymm10[2],ymm11[3],ymm10[4],ymm11[5],ymm10[6],ymm11[7] + vpslld $0x10, %ymm5, %ymm10 + vpblendw $0xaa, %ymm10, %ymm9, %ymm10 # ymm10 = ymm9[0],ymm10[1],ymm9[2],ymm10[3],ymm9[4],ymm10[5],ymm9[6],ymm10[7],ymm9[8],ymm10[9],ymm9[10],ymm10[11],ymm9[12],ymm10[13],ymm9[14],ymm10[15] + vpsrld $0x10, %ymm9, %ymm9 + vpblendw $0xaa, %ymm5, %ymm9, %ymm5 # ymm5 = ymm9[0],ymm5[1],ymm9[2],ymm5[3],ymm9[4],ymm5[5],ymm9[6],ymm5[7],ymm9[8],ymm5[9],ymm9[10],ymm5[11],ymm9[12],ymm5[13],ymm9[14],ymm5[15] + vpslld $0x10, %ymm4, %ymm9 + vpblendw $0xaa, %ymm9, %ymm8, %ymm9 # ymm9 = ymm8[0],ymm9[1],ymm8[2],ymm9[3],ymm8[4],ymm9[5],ymm8[6],ymm9[7],ymm8[8],ymm9[9],ymm8[10],ymm9[11],ymm8[12],ymm9[13],ymm8[14],ymm9[15] + vpsrld $0x10, %ymm8, %ymm8 + vpblendw $0xaa, %ymm4, %ymm8, %ymm4 # ymm4 = ymm8[0],ymm4[1],ymm8[2],ymm4[3],ymm8[4],ymm4[5],ymm8[6],ymm4[7],ymm8[8],ymm4[9],ymm8[10],ymm4[11],ymm8[12],ymm4[13],ymm8[14],ymm4[15] + vpslld $0x10, %ymm3, %ymm8 + vpblendw $0xaa, %ymm8, %ymm7, %ymm8 # ymm8 = ymm7[0],ymm8[1],ymm7[2],ymm8[3],ymm7[4],ymm8[5],ymm7[6],ymm8[7],ymm7[8],ymm8[9],ymm7[10],ymm8[11],ymm7[12],ymm8[13],ymm7[14],ymm8[15] + vpsrld $0x10, %ymm7, %ymm7 + vpblendw $0xaa, %ymm3, %ymm7, %ymm3 # ymm3 = ymm7[0],ymm3[1],ymm7[2],ymm3[3],ymm7[4],ymm3[5],ymm7[6],ymm3[7],ymm7[8],ymm3[9],ymm7[10],ymm3[11],ymm7[12],ymm3[13],ymm7[14],ymm3[15] + vpslld $0x10, %ymm11, %ymm7 + vpblendw $0xaa, %ymm7, %ymm6, %ymm7 # ymm7 = ymm6[0],ymm7[1],ymm6[2],ymm7[3],ymm6[4],ymm7[5],ymm6[6],ymm7[7],ymm6[8],ymm7[9],ymm6[10],ymm7[11],ymm6[12],ymm7[13],ymm6[14],ymm7[15] + vpsrld $0x10, %ymm6, %ymm6 + vpblendw $0xaa, %ymm11, %ymm6, %ymm11 # ymm11 = ymm6[0],ymm11[1],ymm6[2],ymm11[3],ymm6[4],ymm11[5],ymm6[6],ymm11[7],ymm6[8],ymm11[9],ymm6[10],ymm11[11],ymm6[12],ymm11[13],ymm6[14],ymm11[15] + vmovdqa %ymm10, 0x100(%rdi) + vmovdqa %ymm5, 0x120(%rdi) + vmovdqa %ymm9, 0x140(%rdi) + vmovdqa %ymm4, 0x160(%rdi) + vmovdqa %ymm8, 0x180(%rdi) + vmovdqa %ymm3, 0x1a0(%rdi) + vmovdqa %ymm7, 0x1c0(%rdi) + vmovdqa %ymm11, 0x1e0(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(nttunpack_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d10_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d10_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d10_avx2_asm.S @@ -0,0 +1,413 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Kyber optimized AVX2 implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Kyber implementation @[REF_AVX2]. + */ + +/************************************************* + * Name: mlk_poly_compress_d10_avx2_asm + * + * Description: Compression of a polynomial to 10 bits per coefficient. + * + * Arguments: - uint8_t *r: pointer to output byte array + * (of length MLKEM_POLYCOMPRESSEDBYTES_D10) + * - const int16_t *a: pointer to input polynomial + * - const uint8_t *data: pointer to shufbidx constant + **************************************************/ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + (!defined(MLK_CONFIG_NO_ENCAPS_API) || \ + !defined(MLK_CONFIG_NO_DECAPS_API)) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 || MLKEM_K == 3) +/*yaml + Name: poly_compress_d10_avx2_asm + Description: x86_64 AVX2 polynomial compression (d=10) + Signature: void mlk_poly_compress_d10_avx2_asm(uint8_t *r, const int16_t *a, const uint8_t *data) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 320 + permissions: write-only + c_parameter: uint8_t *r + description: Output compressed polynomial + rsi: + type: buffer + size_bytes: 512 + permissions: read-only + c_parameter: const int16_t *a + description: Input polynomial (256 x int16_t) + rdx: + type: buffer + size_bytes: 32 + permissions: read-only + c_parameter: const uint8_t *data + description: Precomputed compression constants +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_poly_compress_d10_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(poly_compress_d10_avx2_asm) +MLK_ASM_FN_SYMBOL(poly_compress_d10_avx2_asm) + + .cfi_startproc + movl $0x4ebf4ebf, %eax # imm = 0x4EBF4EBF + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + vpsllw $0x3, %ymm0, %ymm1 + movl $0xf000f, %eax # imm = 0xF000F + vmovd %eax, %xmm2 + vpbroadcastd %xmm2, %ymm2 + movl $0x10001000, %eax # imm = 0x10001000 + vmovd %eax, %xmm3 + vpbroadcastd %xmm3, %ymm3 + movl $0x3ff03ff, %eax # imm = 0x3FF03FF + vmovd %eax, %xmm4 + vpbroadcastd %xmm4, %ymm4 + movabsq $0x400000104000001, %rax # imm = 0x400000104000001 + vmovq %rax, %xmm5 + vpbroadcastq %xmm5, %ymm5 + movl $0xc, %eax + vmovq %rax, %xmm6 + vpbroadcastq %xmm6, %ymm6 + vmovdqa (%rdx), %ymm7 + vmovdqa (%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, (%rdi) + vmovd %xmm9, 0x10(%rdi) + vmovdqa 0x20(%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, 0x14(%rdi) + vmovd %xmm9, 0x24(%rdi) + vmovdqa 0x40(%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, 0x28(%rdi) + vmovd %xmm9, 0x38(%rdi) + vmovdqa 0x60(%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, 0x3c(%rdi) + vmovd %xmm9, 0x4c(%rdi) + vmovdqa 0x80(%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, 0x50(%rdi) + vmovd %xmm9, 0x60(%rdi) + vmovdqa 0xa0(%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, 0x64(%rdi) + vmovd %xmm9, 0x74(%rdi) + vmovdqa 0xc0(%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, 0x78(%rdi) + vmovd %xmm9, 0x88(%rdi) + vmovdqa 0xe0(%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, 0x8c(%rdi) + vmovd %xmm9, 0x9c(%rdi) + vmovdqa 0x100(%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, 0xa0(%rdi) + vmovd %xmm9, 0xb0(%rdi) + vmovdqa 0x120(%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, 0xb4(%rdi) + vmovd %xmm9, 0xc4(%rdi) + vmovdqa 0x140(%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, 0xc8(%rdi) + vmovd %xmm9, 0xd8(%rdi) + vmovdqa 0x160(%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, 0xdc(%rdi) + vmovd %xmm9, 0xec(%rdi) + vmovdqa 0x180(%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, 0xf0(%rdi) + vmovd %xmm9, 0x100(%rdi) + vmovdqa 0x1a0(%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, 0x104(%rdi) + vmovd %xmm9, 0x114(%rdi) + vmovdqa 0x1c0(%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, 0x118(%rdi) + vmovd %xmm9, 0x128(%rdi) + vmovdqa 0x1e0(%rsi), %ymm8 + vpmullw %ymm1, %ymm8, %ymm9 + vpaddw %ymm2, %ymm8, %ymm10 + vpsllw $0x3, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm8, %ymm8 + vpsubw %ymm10, %ymm9, %ymm10 + vpandn %ymm10, %ymm9, %ymm9 + vpsrlw $0xf, %ymm9, %ymm9 + vpsubw %ymm9, %ymm8, %ymm8 + vpmulhrsw %ymm3, %ymm8, %ymm8 + vpand %ymm4, %ymm8, %ymm8 + vpmaddwd %ymm5, %ymm8, %ymm8 + vpsllvd %ymm6, %ymm8, %ymm8 + vpsrlq $0xc, %ymm8, %ymm8 + vpshufb %ymm7, %ymm8, %ymm8 + vextracti128 $0x1, %ymm8, %xmm9 + vpblendw $0xe0, %xmm9, %xmm8, %xmm8 # xmm8 = xmm8[0,1,2,3,4],xmm9[5,6,7] + vmovdqu %xmm8, 0x12c(%rdi) + vmovd %xmm9, 0x13c(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(poly_compress_d10_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + && (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) && \ + (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 || MLKEM_K == 3) \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d11_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d11_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d11_avx2_asm.S @@ -0,0 +1,479 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Kyber optimized AVX2 implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Kyber implementation @[REF_AVX2]. + */ + +/************************************************* + * Name: mlk_poly_compress_d11_avx2_asm + * + * Description: Compression of a polynomial to 11 bits per coefficient. + * + * Arguments: - uint8_t *r: pointer to output byte array + * (of length MLKEM_POLYCOMPRESSEDBYTES_D11) + * - const int16_t *a: pointer to input polynomial + * - const uint8_t *data: pointer to constants + * (srlvqidx[0:32], shufbidx[32:64]) + **************************************************/ + +#include "../../../common.h" + +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + (!defined(MLK_CONFIG_NO_ENCAPS_API) || \ + !defined(MLK_CONFIG_NO_DECAPS_API)) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4) +/*yaml + Name: poly_compress_d11_avx2_asm + Description: x86_64 AVX2 polynomial compression (d=11) + Signature: void mlk_poly_compress_d11_avx2_asm(uint8_t *r, const int16_t *a, const uint8_t *data) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 352 + permissions: write-only + c_parameter: uint8_t *r + description: Output compressed polynomial + rsi: + type: buffer + size_bytes: 512 + permissions: read-only + c_parameter: const int16_t *a + description: Input polynomial (256 x int16_t) + rdx: + type: buffer + size_bytes: 64 + permissions: read-only + c_parameter: const uint8_t *data + description: Precomputed compression constants +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_poly_compress_d11_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(poly_compress_d11_avx2_asm) +MLK_ASM_FN_SYMBOL(poly_compress_d11_avx2_asm) + + .cfi_startproc + movl $0x4ebf4ebf, %eax # imm = 0x4EBF4EBF + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + vpsllw $0x3, %ymm0, %ymm1 + movl $0x240024, %eax # imm = 0x240024 + vmovd %eax, %xmm2 + vpbroadcastd %xmm2, %ymm2 + movl $0x20002000, %eax # imm = 0x20002000 + vmovd %eax, %xmm3 + vpbroadcastd %xmm3, %ymm3 + movl $0x7ff07ff, %eax # imm = 0x7FF07FF + vmovd %eax, %xmm4 + vpbroadcastd %xmm4, %ymm4 + movabsq $0x800000108000001, %rax # imm = 0x800000108000001 + vmovq %rax, %xmm5 + vpbroadcastq %xmm5, %ymm5 + movl $0xa, %eax + vmovq %rax, %xmm6 + vpbroadcastq %xmm6, %ymm6 + vmovdqa (%rdx), %ymm7 + vmovdqa 0x20(%rdx), %ymm8 + vmovdqa (%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, (%rdi) + vmovd %xmm10, 0x10(%rdi) + vpextrw $0x2, %xmm10, 0x14(%rdi) + vmovdqa 0x20(%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, 0x16(%rdi) + vmovd %xmm10, 0x26(%rdi) + vpextrw $0x2, %xmm10, 0x2a(%rdi) + vmovdqa 0x40(%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, 0x2c(%rdi) + vmovd %xmm10, 0x3c(%rdi) + vpextrw $0x2, %xmm10, 0x40(%rdi) + vmovdqa 0x60(%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, 0x42(%rdi) + vmovd %xmm10, 0x52(%rdi) + vpextrw $0x2, %xmm10, 0x56(%rdi) + vmovdqa 0x80(%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, 0x58(%rdi) + vmovd %xmm10, 0x68(%rdi) + vpextrw $0x2, %xmm10, 0x6c(%rdi) + vmovdqa 0xa0(%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, 0x6e(%rdi) + vmovd %xmm10, 0x7e(%rdi) + vpextrw $0x2, %xmm10, 0x82(%rdi) + vmovdqa 0xc0(%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, 0x84(%rdi) + vmovd %xmm10, 0x94(%rdi) + vpextrw $0x2, %xmm10, 0x98(%rdi) + vmovdqa 0xe0(%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, 0x9a(%rdi) + vmovd %xmm10, 0xaa(%rdi) + vpextrw $0x2, %xmm10, 0xae(%rdi) + vmovdqa 0x100(%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, 0xb0(%rdi) + vmovd %xmm10, 0xc0(%rdi) + vpextrw $0x2, %xmm10, 0xc4(%rdi) + vmovdqa 0x120(%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, 0xc6(%rdi) + vmovd %xmm10, 0xd6(%rdi) + vpextrw $0x2, %xmm10, 0xda(%rdi) + vmovdqa 0x140(%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, 0xdc(%rdi) + vmovd %xmm10, 0xec(%rdi) + vpextrw $0x2, %xmm10, 0xf0(%rdi) + vmovdqa 0x160(%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, 0xf2(%rdi) + vmovd %xmm10, 0x102(%rdi) + vpextrw $0x2, %xmm10, 0x106(%rdi) + vmovdqa 0x180(%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, 0x108(%rdi) + vmovd %xmm10, 0x118(%rdi) + vpextrw $0x2, %xmm10, 0x11c(%rdi) + vmovdqa 0x1a0(%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, 0x11e(%rdi) + vmovd %xmm10, 0x12e(%rdi) + vpextrw $0x2, %xmm10, 0x132(%rdi) + vmovdqa 0x1c0(%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, 0x134(%rdi) + vmovd %xmm10, 0x144(%rdi) + vpextrw $0x2, %xmm10, 0x148(%rdi) + vmovdqa 0x1e0(%rsi), %ymm9 + vpmullw %ymm1, %ymm9, %ymm10 + vpaddw %ymm2, %ymm9, %ymm11 + vpsllw $0x3, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm9, %ymm9 + vpsubw %ymm11, %ymm10, %ymm11 + vpandn %ymm11, %ymm10, %ymm10 + vpsrlw $0xf, %ymm10, %ymm10 + vpsubw %ymm10, %ymm9, %ymm9 + vpmulhrsw %ymm3, %ymm9, %ymm9 + vpand %ymm4, %ymm9, %ymm9 + vpmaddwd %ymm5, %ymm9, %ymm9 + vpsllvd %ymm6, %ymm9, %ymm9 + vpsrldq $0x8, %ymm9, %ymm10 # ymm10 = ymm9[8,9,10,11,12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,ymm9[24,25,26,27,28,29,30,31],zero,zero,zero,zero,zero,zero,zero,zero + vpsrlvq %ymm7, %ymm9, %ymm9 + vpsllq $0x22, %ymm10, %ymm10 + vpaddq %ymm10, %ymm9, %ymm9 + vpshufb %ymm8, %ymm9, %ymm9 + vextracti128 $0x1, %ymm9, %xmm10 + vpblendvb %xmm8, %xmm10, %xmm9, %xmm9 + vmovdqu %xmm9, 0x14a(%rdi) + vmovd %xmm10, 0x15a(%rdi) + vpextrw $0x2, %xmm10, 0x15e(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(poly_compress_d11_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + && (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) && \ + (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d4_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d4_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d4_avx2_asm.S @@ -0,0 +1,194 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Kyber optimized AVX2 implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Kyber implementation @[REF_AVX2]. + */ + +/************************************************* + * Name: mlk_poly_compress_d4_avx2_asm + * + * Description: Compression of a polynomial to 4 bits per coefficient. + * + * Arguments: - uint8_t *r: pointer to output byte array + * (of length MLKEM_POLYCOMPRESSEDBYTES_D4) + * - const int16_t *a: pointer to input polynomial + * - const uint8_t *data: pointer to permdidx constant + **************************************************/ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + (!defined(MLK_CONFIG_NO_ENCAPS_API) || \ + !defined(MLK_CONFIG_NO_DECAPS_API)) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 || MLKEM_K == 3) +/*yaml + Name: poly_compress_d4_avx2_asm + Description: x86_64 AVX2 polynomial compression (d=4) + Signature: void mlk_poly_compress_d4_avx2_asm(uint8_t *r, const int16_t *a, const uint8_t *data) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 128 + permissions: write-only + c_parameter: uint8_t *r + description: Output compressed polynomial + rsi: + type: buffer + size_bytes: 512 + permissions: read-only + c_parameter: const int16_t *a + description: Input polynomial (256 x int16_t) + rdx: + type: buffer + size_bytes: 32 + permissions: read-only + c_parameter: const uint8_t *data + description: Precomputed compression constants +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_poly_compress_d4_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(poly_compress_d4_avx2_asm) +MLK_ASM_FN_SYMBOL(poly_compress_d4_avx2_asm) + + .cfi_startproc + movl $0x4ebf4ebf, %eax # imm = 0x4EBF4EBF + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + movl $0x2000200, %eax # imm = 0x2000200 + vmovd %eax, %xmm1 + vpbroadcastd %xmm1, %ymm1 + movl $0xf000f, %eax # imm = 0xF000F + vmovd %eax, %xmm2 + vpbroadcastd %xmm2, %ymm2 + movl $0x10011001, %eax # imm = 0x10011001 + vmovd %eax, %xmm3 + vpbroadcastd %xmm3, %ymm3 + vmovdqa (%rdx), %ymm4 + vmovdqa (%rsi), %ymm5 + vmovdqa 0x20(%rsi), %ymm6 + vmovdqa 0x40(%rsi), %ymm7 + vmovdqa 0x60(%rsi), %ymm8 + vpmulhw %ymm0, %ymm5, %ymm5 + vpmulhw %ymm0, %ymm6, %ymm6 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhrsw %ymm1, %ymm5, %ymm5 + vpmulhrsw %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm1, %ymm7, %ymm7 + vpmulhrsw %ymm1, %ymm8, %ymm8 + vpand %ymm2, %ymm5, %ymm5 + vpand %ymm2, %ymm6, %ymm6 + vpand %ymm2, %ymm7, %ymm7 + vpand %ymm2, %ymm8, %ymm8 + vpackuswb %ymm6, %ymm5, %ymm5 + vpackuswb %ymm8, %ymm7, %ymm7 + vpmaddubsw %ymm3, %ymm5, %ymm5 + vpmaddubsw %ymm3, %ymm7, %ymm7 + vpackuswb %ymm7, %ymm5, %ymm5 + vpermd %ymm5, %ymm4, %ymm5 + vmovdqu %ymm5, (%rdi) + vmovdqa 0x80(%rsi), %ymm5 + vmovdqa 0xa0(%rsi), %ymm6 + vmovdqa 0xc0(%rsi), %ymm7 + vmovdqa 0xe0(%rsi), %ymm8 + vpmulhw %ymm0, %ymm5, %ymm5 + vpmulhw %ymm0, %ymm6, %ymm6 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhrsw %ymm1, %ymm5, %ymm5 + vpmulhrsw %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm1, %ymm7, %ymm7 + vpmulhrsw %ymm1, %ymm8, %ymm8 + vpand %ymm2, %ymm5, %ymm5 + vpand %ymm2, %ymm6, %ymm6 + vpand %ymm2, %ymm7, %ymm7 + vpand %ymm2, %ymm8, %ymm8 + vpackuswb %ymm6, %ymm5, %ymm5 + vpackuswb %ymm8, %ymm7, %ymm7 + vpmaddubsw %ymm3, %ymm5, %ymm5 + vpmaddubsw %ymm3, %ymm7, %ymm7 + vpackuswb %ymm7, %ymm5, %ymm5 + vpermd %ymm5, %ymm4, %ymm5 + vmovdqu %ymm5, 0x20(%rdi) + vmovdqa 0x100(%rsi), %ymm5 + vmovdqa 0x120(%rsi), %ymm6 + vmovdqa 0x140(%rsi), %ymm7 + vmovdqa 0x160(%rsi), %ymm8 + vpmulhw %ymm0, %ymm5, %ymm5 + vpmulhw %ymm0, %ymm6, %ymm6 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhrsw %ymm1, %ymm5, %ymm5 + vpmulhrsw %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm1, %ymm7, %ymm7 + vpmulhrsw %ymm1, %ymm8, %ymm8 + vpand %ymm2, %ymm5, %ymm5 + vpand %ymm2, %ymm6, %ymm6 + vpand %ymm2, %ymm7, %ymm7 + vpand %ymm2, %ymm8, %ymm8 + vpackuswb %ymm6, %ymm5, %ymm5 + vpackuswb %ymm8, %ymm7, %ymm7 + vpmaddubsw %ymm3, %ymm5, %ymm5 + vpmaddubsw %ymm3, %ymm7, %ymm7 + vpackuswb %ymm7, %ymm5, %ymm5 + vpermd %ymm5, %ymm4, %ymm5 + vmovdqu %ymm5, 0x40(%rdi) + vmovdqa 0x180(%rsi), %ymm5 + vmovdqa 0x1a0(%rsi), %ymm6 + vmovdqa 0x1c0(%rsi), %ymm7 + vmovdqa 0x1e0(%rsi), %ymm8 + vpmulhw %ymm0, %ymm5, %ymm5 + vpmulhw %ymm0, %ymm6, %ymm6 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhrsw %ymm1, %ymm5, %ymm5 + vpmulhrsw %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm1, %ymm7, %ymm7 + vpmulhrsw %ymm1, %ymm8, %ymm8 + vpand %ymm2, %ymm5, %ymm5 + vpand %ymm2, %ymm6, %ymm6 + vpand %ymm2, %ymm7, %ymm7 + vpand %ymm2, %ymm8, %ymm8 + vpackuswb %ymm6, %ymm5, %ymm5 + vpackuswb %ymm8, %ymm7, %ymm7 + vpmaddubsw %ymm3, %ymm5, %ymm5 + vpmaddubsw %ymm3, %ymm7, %ymm7 + vpackuswb %ymm7, %ymm5, %ymm5 + vpermd %ymm5, %ymm4, %ymm5 + vmovdqu %ymm5, 0x60(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(poly_compress_d4_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + && (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) && \ + (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2 || MLKEM_K == 3) \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d5_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d5_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_compress_d5_avx2_asm.S @@ -0,0 +1,251 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Kyber optimized AVX2 implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Kyber implementation @[REF_AVX2]. + */ + +/************************************************* + * Name: mlk_poly_compress_d5_avx2_asm + * + * Description: Compression of a polynomial to 5 bits per coefficient. + * + * Arguments: - uint8_t *r: pointer to output byte array + * (of length MLKEM_POLYCOMPRESSEDBYTES_D5) + * - const int16_t *a: pointer to input polynomial + * - const uint8_t *data: pointer to shufbidx constant + **************************************************/ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + (!defined(MLK_CONFIG_NO_ENCAPS_API) || \ + !defined(MLK_CONFIG_NO_DECAPS_API)) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4) +/*yaml + Name: poly_compress_d5_avx2_asm + Description: x86_64 AVX2 polynomial compression (d=5) + Signature: void mlk_poly_compress_d5_avx2_asm(uint8_t *r, const int16_t *a, const uint8_t *data) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 160 + permissions: write-only + c_parameter: uint8_t *r + description: Output compressed polynomial + rsi: + type: buffer + size_bytes: 512 + permissions: read-only + c_parameter: const int16_t *a + description: Input polynomial (256 x int16_t) + rdx: + type: buffer + size_bytes: 32 + permissions: read-only + c_parameter: const uint8_t *data + description: Precomputed compression constants +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_poly_compress_d5_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(poly_compress_d5_avx2_asm) +MLK_ASM_FN_SYMBOL(poly_compress_d5_avx2_asm) + + .cfi_startproc + movl $0x4ebf4ebf, %eax # imm = 0x4EBF4EBF + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + movl $0x4000400, %eax # imm = 0x4000400 + vmovd %eax, %xmm1 + vpbroadcastd %xmm1, %ymm1 + movl $0x1f001f, %eax # imm = 0x1F001F + vmovd %eax, %xmm2 + vpbroadcastd %xmm2, %ymm2 + movl $0x20012001, %eax # imm = 0x20012001 + vmovd %eax, %xmm3 + vpbroadcastd %xmm3, %ymm3 + movl $0x4000001, %eax # imm = 0x4000001 + vmovd %eax, %xmm4 + vpbroadcastd %xmm4, %ymm4 + movl $0xc, %eax + vmovq %rax, %xmm5 + vpbroadcastq %xmm5, %ymm5 + vmovdqa (%rdx), %ymm6 + vmovdqa (%rsi), %ymm7 + vmovdqa 0x20(%rsi), %ymm8 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhrsw %ymm1, %ymm7, %ymm7 + vpmulhrsw %ymm1, %ymm8, %ymm8 + vpand %ymm2, %ymm7, %ymm7 + vpand %ymm2, %ymm8, %ymm8 + vpackuswb %ymm8, %ymm7, %ymm7 + vpmaddubsw %ymm3, %ymm7, %ymm7 + vpmaddwd %ymm4, %ymm7, %ymm7 + vpsllvd %ymm5, %ymm7, %ymm7 + vpsrlvq %ymm5, %ymm7, %ymm7 + vpshufb %ymm6, %ymm7, %ymm7 + vextracti128 $0x1, %ymm7, %xmm8 + vpblendvb %xmm6, %xmm8, %xmm7, %xmm7 + vmovdqu %xmm7, (%rdi) + vmovd %xmm8, 0x10(%rdi) + vmovdqa 0x40(%rsi), %ymm7 + vmovdqa 0x60(%rsi), %ymm8 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhrsw %ymm1, %ymm7, %ymm7 + vpmulhrsw %ymm1, %ymm8, %ymm8 + vpand %ymm2, %ymm7, %ymm7 + vpand %ymm2, %ymm8, %ymm8 + vpackuswb %ymm8, %ymm7, %ymm7 + vpmaddubsw %ymm3, %ymm7, %ymm7 + vpmaddwd %ymm4, %ymm7, %ymm7 + vpsllvd %ymm5, %ymm7, %ymm7 + vpsrlvq %ymm5, %ymm7, %ymm7 + vpshufb %ymm6, %ymm7, %ymm7 + vextracti128 $0x1, %ymm7, %xmm8 + vpblendvb %xmm6, %xmm8, %xmm7, %xmm7 + vmovdqu %xmm7, 0x14(%rdi) + vmovd %xmm8, 0x24(%rdi) + vmovdqa 0x80(%rsi), %ymm7 + vmovdqa 0xa0(%rsi), %ymm8 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhrsw %ymm1, %ymm7, %ymm7 + vpmulhrsw %ymm1, %ymm8, %ymm8 + vpand %ymm2, %ymm7, %ymm7 + vpand %ymm2, %ymm8, %ymm8 + vpackuswb %ymm8, %ymm7, %ymm7 + vpmaddubsw %ymm3, %ymm7, %ymm7 + vpmaddwd %ymm4, %ymm7, %ymm7 + vpsllvd %ymm5, %ymm7, %ymm7 + vpsrlvq %ymm5, %ymm7, %ymm7 + vpshufb %ymm6, %ymm7, %ymm7 + vextracti128 $0x1, %ymm7, %xmm8 + vpblendvb %xmm6, %xmm8, %xmm7, %xmm7 + vmovdqu %xmm7, 0x28(%rdi) + vmovd %xmm8, 0x38(%rdi) + vmovdqa 0xc0(%rsi), %ymm7 + vmovdqa 0xe0(%rsi), %ymm8 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhrsw %ymm1, %ymm7, %ymm7 + vpmulhrsw %ymm1, %ymm8, %ymm8 + vpand %ymm2, %ymm7, %ymm7 + vpand %ymm2, %ymm8, %ymm8 + vpackuswb %ymm8, %ymm7, %ymm7 + vpmaddubsw %ymm3, %ymm7, %ymm7 + vpmaddwd %ymm4, %ymm7, %ymm7 + vpsllvd %ymm5, %ymm7, %ymm7 + vpsrlvq %ymm5, %ymm7, %ymm7 + vpshufb %ymm6, %ymm7, %ymm7 + vextracti128 $0x1, %ymm7, %xmm8 + vpblendvb %xmm6, %xmm8, %xmm7, %xmm7 + vmovdqu %xmm7, 0x3c(%rdi) + vmovd %xmm8, 0x4c(%rdi) + vmovdqa 0x100(%rsi), %ymm7 + vmovdqa 0x120(%rsi), %ymm8 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhrsw %ymm1, %ymm7, %ymm7 + vpmulhrsw %ymm1, %ymm8, %ymm8 + vpand %ymm2, %ymm7, %ymm7 + vpand %ymm2, %ymm8, %ymm8 + vpackuswb %ymm8, %ymm7, %ymm7 + vpmaddubsw %ymm3, %ymm7, %ymm7 + vpmaddwd %ymm4, %ymm7, %ymm7 + vpsllvd %ymm5, %ymm7, %ymm7 + vpsrlvq %ymm5, %ymm7, %ymm7 + vpshufb %ymm6, %ymm7, %ymm7 + vextracti128 $0x1, %ymm7, %xmm8 + vpblendvb %xmm6, %xmm8, %xmm7, %xmm7 + vmovdqu %xmm7, 0x50(%rdi) + vmovd %xmm8, 0x60(%rdi) + vmovdqa 0x140(%rsi), %ymm7 + vmovdqa 0x160(%rsi), %ymm8 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhrsw %ymm1, %ymm7, %ymm7 + vpmulhrsw %ymm1, %ymm8, %ymm8 + vpand %ymm2, %ymm7, %ymm7 + vpand %ymm2, %ymm8, %ymm8 + vpackuswb %ymm8, %ymm7, %ymm7 + vpmaddubsw %ymm3, %ymm7, %ymm7 + vpmaddwd %ymm4, %ymm7, %ymm7 + vpsllvd %ymm5, %ymm7, %ymm7 + vpsrlvq %ymm5, %ymm7, %ymm7 + vpshufb %ymm6, %ymm7, %ymm7 + vextracti128 $0x1, %ymm7, %xmm8 + vpblendvb %xmm6, %xmm8, %xmm7, %xmm7 + vmovdqu %xmm7, 0x64(%rdi) + vmovd %xmm8, 0x74(%rdi) + vmovdqa 0x180(%rsi), %ymm7 + vmovdqa 0x1a0(%rsi), %ymm8 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhrsw %ymm1, %ymm7, %ymm7 + vpmulhrsw %ymm1, %ymm8, %ymm8 + vpand %ymm2, %ymm7, %ymm7 + vpand %ymm2, %ymm8, %ymm8 + vpackuswb %ymm8, %ymm7, %ymm7 + vpmaddubsw %ymm3, %ymm7, %ymm7 + vpmaddwd %ymm4, %ymm7, %ymm7 + vpsllvd %ymm5, %ymm7, %ymm7 + vpsrlvq %ymm5, %ymm7, %ymm7 + vpshufb %ymm6, %ymm7, %ymm7 + vextracti128 $0x1, %ymm7, %xmm8 + vpblendvb %xmm6, %xmm8, %xmm7, %xmm7 + vmovdqu %xmm7, 0x78(%rdi) + vmovd %xmm8, 0x88(%rdi) + vmovdqa 0x1c0(%rsi), %ymm7 + vmovdqa 0x1e0(%rsi), %ymm8 + vpmulhw %ymm0, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm8, %ymm8 + vpmulhrsw %ymm1, %ymm7, %ymm7 + vpmulhrsw %ymm1, %ymm8, %ymm8 + vpand %ymm2, %ymm7, %ymm7 + vpand %ymm2, %ymm8, %ymm8 + vpackuswb %ymm8, %ymm7, %ymm7 + vpmaddubsw %ymm3, %ymm7, %ymm7 + vpmaddwd %ymm4, %ymm7, %ymm7 + vpsllvd %ymm5, %ymm7, %ymm7 + vpsrlvq %ymm5, %ymm7, %ymm7 + vpshufb %ymm6, %ymm7, %ymm7 + vextracti128 $0x1, %ymm7, %xmm8 + vpblendvb %xmm6, %xmm8, %xmm7, %xmm7 + vmovdqu %xmm7, 0x8c(%rdi) + vmovd %xmm8, 0x9c(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(poly_compress_d5_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + && (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API) && \ + (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d10_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d10_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d10_avx2_asm.S @@ -0,0 +1,257 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Kyber optimized AVX2 implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Kyber implementation @[REF_AVX2]. + */ + +/************************************************* + * Name: mlk_poly_decompress_d10_avx2_asm + * + * Description: Decompression of a polynomial from 10 bits per coefficient. + * + * Arguments: - int16_t *r: pointer to output polynomial + * - const uint8_t *a: pointer to input byte array + * (of length MLKEM_POLYCOMPRESSEDBYTES_D10) + * - const uint8_t *data: pointer to shufbidx constant + **************************************************/ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + !defined(MLK_CONFIG_NO_DECAPS_API) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 || MLKEM_K == 3) +/*yaml + Name: poly_decompress_d10_avx2_asm + Description: x86_64 AVX2 polynomial decompression (d=10) + Signature: void mlk_poly_decompress_d10_avx2_asm(int16_t *r, const uint8_t *a, const uint8_t *data) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 512 + permissions: write-only + c_parameter: int16_t *r + description: Output polynomial (256 x int16_t) + rsi: + type: buffer + size_bytes: 320 + permissions: read-only + c_parameter: const uint8_t *a + description: Input compressed polynomial + rdx: + type: buffer + size_bytes: 32 + permissions: read-only + c_parameter: const uint8_t *data + description: Precomputed decompression constants +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_poly_decompress_d10_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(poly_decompress_d10_avx2_asm) +MLK_ASM_FN_SYMBOL(poly_decompress_d10_avx2_asm) + + .cfi_startproc + movl $0xd013404, %eax # imm = 0xD013404 + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + movl $0x4, %eax + vmovq %rax, %xmm1 + vpbroadcastq %xmm1, %ymm1 + movl $0x7fe01ff8, %eax # imm = 0x7FE01FF8 + vmovd %eax, %xmm2 + vpbroadcastd %xmm2, %ymm2 + vmovdqa (%rdx), %ymm3 + vmovdqu (%rsi), %xmm4 + vmovd 0x10(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, (%rdi) + vmovdqu 0x14(%rsi), %xmm4 + vmovd 0x24(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x20(%rdi) + vmovdqu 0x28(%rsi), %xmm4 + vmovd 0x38(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x40(%rdi) + vmovdqu 0x3c(%rsi), %xmm4 + vmovd 0x4c(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x60(%rdi) + vmovdqu 0x50(%rsi), %xmm4 + vmovd 0x60(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x80(%rdi) + vmovdqu 0x64(%rsi), %xmm4 + vmovd 0x74(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0xa0(%rdi) + vmovdqu 0x78(%rsi), %xmm4 + vmovd 0x88(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0xc0(%rdi) + vmovdqu 0x8c(%rsi), %xmm4 + vmovd 0x9c(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0xe0(%rdi) + vmovdqu 0xa0(%rsi), %xmm4 + vmovd 0xb0(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x100(%rdi) + vmovdqu 0xb4(%rsi), %xmm4 + vmovd 0xc4(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x120(%rdi) + vmovdqu 0xc8(%rsi), %xmm4 + vmovd 0xd8(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x140(%rdi) + vmovdqu 0xdc(%rsi), %xmm4 + vmovd 0xec(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x160(%rdi) + vmovdqu 0xf0(%rsi), %xmm4 + vmovd 0x100(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x180(%rdi) + vmovdqu 0x104(%rsi), %xmm4 + vmovd 0x114(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x1a0(%rdi) + vmovdqu 0x118(%rsi), %xmm4 + vmovd 0x128(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x1c0(%rdi) + vmovdqu 0x12c(%rsi), %xmm4 + vmovd 0x13c(%rsi), %xmm5 + vinserti128 $0x1, %xmm5, %ymm4, %ymm4 + vpermq $0x94, %ymm4, %ymm4 # ymm4 = ymm4[0,1,1,2] + vpshufb %ymm3, %ymm4, %ymm4 + vpsllvd %ymm1, %ymm4, %ymm4 + vpsrlw $0x1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x1e0(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(poly_decompress_d10_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + && !MLK_CONFIG_NO_DECAPS_API && (MLK_CONFIG_MULTILEVEL_WITH_SHARED || \ + MLKEM_K == 2 || MLKEM_K == 3) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d11_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d11_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d11_avx2_asm.S @@ -0,0 +1,307 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Kyber optimized AVX2 implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Kyber implementation @[REF_AVX2]. + */ + +/************************************************* + * Name: mlk_poly_decompress_d11_avx2_asm + * + * Description: Decompression of a polynomial from 11 bits per coefficient. + * + * Arguments: - int16_t *r: pointer to output polynomial + * - const uint8_t *a: pointer to input byte array + * (of length MLKEM_POLYCOMPRESSEDBYTES_D11) + * - const uint8_t *data: pointer to constants + * (shufbidx[0:32], srlvdidx[32:64], + * srlvqidx[64:96], shift[96:128]) + **************************************************/ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + !defined(MLK_CONFIG_NO_DECAPS_API) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4) +/*yaml + Name: poly_decompress_d11_avx2_asm + Description: x86_64 AVX2 polynomial decompression (d=11) + Signature: void mlk_poly_decompress_d11_avx2_asm(int16_t *r, const uint8_t *a, const uint8_t *data) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 512 + permissions: write-only + c_parameter: int16_t *r + description: Output polynomial (256 x int16_t) + rsi: + type: buffer + size_bytes: 352 + permissions: read-only + c_parameter: const uint8_t *a + description: Input compressed polynomial + rdx: + type: buffer + size_bytes: 128 + permissions: read-only + c_parameter: const uint8_t *data + description: Precomputed decompression constants +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_poly_decompress_d11_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(poly_decompress_d11_avx2_asm) +MLK_ASM_FN_SYMBOL(poly_decompress_d11_avx2_asm) + + .cfi_startproc + movl $0xd010d01, %eax # imm = 0xD010D01 + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + movl $0x7ff07ff0, %eax # imm = 0x7FF07FF0 + vmovd %eax, %xmm1 + vpbroadcastd %xmm1, %ymm1 + vmovdqa (%rdx), %ymm2 + vmovdqa 0x20(%rdx), %ymm3 + vmovdqa 0x40(%rdx), %ymm4 + vmovdqa 0x60(%rdx), %ymm5 + vmovdqu (%rsi), %xmm6 + vmovd 0x10(%rsi), %xmm7 + vpinsrw $0x2, 0x14(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, (%rdi) + vmovdqu 0x16(%rsi), %xmm6 + vmovd 0x26(%rsi), %xmm7 + vpinsrw $0x2, 0x2a(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, 0x20(%rdi) + vmovdqu 0x2c(%rsi), %xmm6 + vmovd 0x3c(%rsi), %xmm7 + vpinsrw $0x2, 0x40(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, 0x40(%rdi) + vmovdqu 0x42(%rsi), %xmm6 + vmovd 0x52(%rsi), %xmm7 + vpinsrw $0x2, 0x56(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, 0x60(%rdi) + vmovdqu 0x58(%rsi), %xmm6 + vmovd 0x68(%rsi), %xmm7 + vpinsrw $0x2, 0x6c(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, 0x80(%rdi) + vmovdqu 0x6e(%rsi), %xmm6 + vmovd 0x7e(%rsi), %xmm7 + vpinsrw $0x2, 0x82(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, 0xa0(%rdi) + vmovdqu 0x84(%rsi), %xmm6 + vmovd 0x94(%rsi), %xmm7 + vpinsrw $0x2, 0x98(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, 0xc0(%rdi) + vmovdqu 0x9a(%rsi), %xmm6 + vmovd 0xaa(%rsi), %xmm7 + vpinsrw $0x2, 0xae(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, 0xe0(%rdi) + vmovdqu 0xb0(%rsi), %xmm6 + vmovd 0xc0(%rsi), %xmm7 + vpinsrw $0x2, 0xc4(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, 0x100(%rdi) + vmovdqu 0xc6(%rsi), %xmm6 + vmovd 0xd6(%rsi), %xmm7 + vpinsrw $0x2, 0xda(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, 0x120(%rdi) + vmovdqu 0xdc(%rsi), %xmm6 + vmovd 0xec(%rsi), %xmm7 + vpinsrw $0x2, 0xf0(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, 0x140(%rdi) + vmovdqu 0xf2(%rsi), %xmm6 + vmovd 0x102(%rsi), %xmm7 + vpinsrw $0x2, 0x106(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, 0x160(%rdi) + vmovdqu 0x108(%rsi), %xmm6 + vmovd 0x118(%rsi), %xmm7 + vpinsrw $0x2, 0x11c(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, 0x180(%rdi) + vmovdqu 0x11e(%rsi), %xmm6 + vmovd 0x12e(%rsi), %xmm7 + vpinsrw $0x2, 0x132(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, 0x1a0(%rdi) + vmovdqu 0x134(%rsi), %xmm6 + vmovd 0x144(%rsi), %xmm7 + vpinsrw $0x2, 0x148(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, 0x1c0(%rdi) + vmovdqu 0x14a(%rsi), %xmm6 + vmovd 0x15a(%rsi), %xmm7 + vpinsrw $0x2, 0x15e(%rsi), %xmm7, %xmm7 + vinserti128 $0x1, %xmm7, %ymm6, %ymm6 + vpermq $0x94, %ymm6, %ymm6 # ymm6 = ymm6[0,1,1,2] + vpshufb %ymm2, %ymm6, %ymm6 + vpsrlvd %ymm3, %ymm6, %ymm6 + vpsrlvq %ymm4, %ymm6, %ymm6 + vpmullw %ymm5, %ymm6, %ymm6 + vpsrlw $0x1, %ymm6, %ymm6 + vpand %ymm1, %ymm6, %ymm6 + vpmulhrsw %ymm0, %ymm6, %ymm6 + vmovdqu %ymm6, 0x1e0(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(poly_decompress_d11_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + && !MLK_CONFIG_NO_DECAPS_API && (MLK_CONFIG_MULTILEVEL_WITH_SHARED || \ + MLKEM_K == 4) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d4_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d4_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d4_avx2_asm.S @@ -0,0 +1,209 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Kyber optimized AVX2 implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Kyber implementation @[REF_AVX2]. + */ + +/************************************************* + * Name: mlk_poly_decompress_d4_avx2_asm + * + * Description: Decompression of a polynomial from 4 bits per coefficient. + * + * Arguments: - int16_t *r: pointer to output polynomial + * - const uint8_t *a: pointer to input byte array + * (of length MLKEM_POLYCOMPRESSEDBYTES_D4) + * - const int8_t *data: pointer to shufbidx constant + **************************************************/ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + !defined(MLK_CONFIG_NO_DECAPS_API) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2 || MLKEM_K == 3) +/*yaml + Name: poly_decompress_d4_avx2_asm + Description: x86_64 AVX2 polynomial decompression (d=4) + Signature: void mlk_poly_decompress_d4_avx2_asm(int16_t *r, const uint8_t *a, const uint8_t *data) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 512 + permissions: write-only + c_parameter: int16_t *r + description: Output polynomial (256 x int16_t) + rsi: + type: buffer + size_bytes: 128 + permissions: read-only + c_parameter: const uint8_t *a + description: Input compressed polynomial + rdx: + type: buffer + size_bytes: 32 + permissions: read-only + c_parameter: const uint8_t *data + description: Precomputed decompression constants +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_poly_decompress_d4_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(poly_decompress_d4_avx2_asm) +MLK_ASM_FN_SYMBOL(poly_decompress_d4_avx2_asm) + + .cfi_startproc + movl $0xd010d01, %eax # imm = 0xD010D01 + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + movl $0xf0000f, %eax # imm = 0xF0000F + vmovd %eax, %xmm1 + vpbroadcastd %xmm1, %ymm1 + movl $0x800800, %eax # imm = 0x800800 + vmovd %eax, %xmm2 + vpbroadcastd %xmm2, %ymm2 + vmovdqa (%rdx), %ymm3 + vmovq (%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, (%rdi) + vmovq 0x8(%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x20(%rdi) + vmovq 0x10(%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x40(%rdi) + vmovq 0x18(%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x60(%rdi) + vmovq 0x20(%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x80(%rdi) + vmovq 0x28(%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0xa0(%rdi) + vmovq 0x30(%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0xc0(%rdi) + vmovq 0x38(%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0xe0(%rdi) + vmovq 0x40(%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x100(%rdi) + vmovq 0x48(%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x120(%rdi) + vmovq 0x50(%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x140(%rdi) + vmovq 0x58(%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x160(%rdi) + vmovq 0x60(%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x180(%rdi) + vmovq 0x68(%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x1a0(%rdi) + vmovq 0x70(%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x1c0(%rdi) + vmovq 0x78(%rsi), %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm3, %ymm4, %ymm4 + vpand %ymm1, %ymm4, %ymm4 + vpmullw %ymm2, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x1e0(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(poly_decompress_d4_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + && !MLK_CONFIG_NO_DECAPS_API && (MLK_CONFIG_MULTILEVEL_WITH_SHARED || \ + MLKEM_K == 2 || MLKEM_K == 3) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d5_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d5_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_decompress_d5_avx2_asm.S @@ -0,0 +1,222 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Kyber optimized AVX2 implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Kyber implementation @[REF_AVX2]. + */ + +/************************************************* + * Name: mlk_poly_decompress_d5_avx2_asm + * + * Description: Decompression of a polynomial from 5 bits per coefficient. + * + * Arguments: - int16_t *r: pointer to output polynomial + * - const uint8_t *a: pointer to input byte array + * (of length MLKEM_POLYCOMPRESSEDBYTES_D5) + * - const uint8_t *data: pointer to constants + * (shufbidx[0:32], mask[32:64], shift[64:96]) + **************************************************/ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + !defined(MLK_CONFIG_NO_DECAPS_API) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4) +/*yaml + Name: poly_decompress_d5_avx2_asm + Description: x86_64 AVX2 polynomial decompression (d=5) + Signature: void mlk_poly_decompress_d5_avx2_asm(int16_t *r, const uint8_t *a, const uint8_t *data) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 512 + permissions: write-only + c_parameter: int16_t *r + description: Output polynomial (256 x int16_t) + rsi: + type: buffer + size_bytes: 160 + permissions: read-only + c_parameter: const uint8_t *a + description: Input compressed polynomial + rdx: + type: buffer + size_bytes: 96 + permissions: read-only + c_parameter: const uint8_t *data + description: Precomputed decompression constants +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_poly_decompress_d5_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(poly_decompress_d5_avx2_asm) +MLK_ASM_FN_SYMBOL(poly_decompress_d5_avx2_asm) + + .cfi_startproc + movl $0xd010d01, %eax # imm = 0xD010D01 + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + vmovdqa (%rdx), %ymm1 + vmovdqa 0x20(%rdx), %ymm2 + vmovdqa 0x40(%rdx), %ymm3 + vmovq (%rsi), %xmm4 + vpinsrw $0x4, 0x8(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, (%rdi) + vmovq 0xa(%rsi), %xmm4 + vpinsrw $0x4, 0x12(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x20(%rdi) + vmovq 0x14(%rsi), %xmm4 + vpinsrw $0x4, 0x1c(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x40(%rdi) + vmovq 0x1e(%rsi), %xmm4 + vpinsrw $0x4, 0x26(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x60(%rdi) + vmovq 0x28(%rsi), %xmm4 + vpinsrw $0x4, 0x30(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x80(%rdi) + vmovq 0x32(%rsi), %xmm4 + vpinsrw $0x4, 0x3a(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0xa0(%rdi) + vmovq 0x3c(%rsi), %xmm4 + vpinsrw $0x4, 0x44(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0xc0(%rdi) + vmovq 0x46(%rsi), %xmm4 + vpinsrw $0x4, 0x4e(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0xe0(%rdi) + vmovq 0x50(%rsi), %xmm4 + vpinsrw $0x4, 0x58(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x100(%rdi) + vmovq 0x5a(%rsi), %xmm4 + vpinsrw $0x4, 0x62(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x120(%rdi) + vmovq 0x64(%rsi), %xmm4 + vpinsrw $0x4, 0x6c(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x140(%rdi) + vmovq 0x6e(%rsi), %xmm4 + vpinsrw $0x4, 0x76(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x160(%rdi) + vmovq 0x78(%rsi), %xmm4 + vpinsrw $0x4, 0x80(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x180(%rdi) + vmovq 0x82(%rsi), %xmm4 + vpinsrw $0x4, 0x8a(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x1a0(%rdi) + vmovq 0x8c(%rsi), %xmm4 + vpinsrw $0x4, 0x94(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x1c0(%rdi) + vmovq 0x96(%rsi), %xmm4 + vpinsrw $0x4, 0x9e(%rsi), %xmm4, %xmm4 + vinserti128 $0x1, %xmm4, %ymm4, %ymm4 + vpshufb %ymm1, %ymm4, %ymm4 + vpand %ymm2, %ymm4, %ymm4 + vpmullw %ymm3, %ymm4, %ymm4 + vpmulhrsw %ymm0, %ymm4, %ymm4 + vmovdqu %ymm4, 0x1e0(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(poly_decompress_d5_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + && !MLK_CONFIG_NO_DECAPS_API && (MLK_CONFIG_MULTILEVEL_WITH_SHARED || \ + MLKEM_K == 4) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_poly_mulcache_compute_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_mulcache_compute_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_poly_mulcache_compute_avx2_asm.S @@ -0,0 +1,118 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) +/*yaml + Name: poly_mulcache_compute_avx2_asm + Description: x86_64 AVX2 mulcache computation + Signature: void mlk_poly_mulcache_compute_avx2_asm(int16_t *out, const int16_t *in, const int16_t *qdata) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 256 + permissions: write-only + c_parameter: int16_t *out + description: Output mulcache (128 x int16_t) + rsi: + type: buffer + size_bytes: 512 + permissions: read-only + c_parameter: const int16_t *in + description: Input polynomial (256 x int16_t) + rdx: + type: buffer + size_bytes: 1248 + permissions: read-only + c_parameter: const int16_t *qdata + description: Precomputed constants (624 x int16_t) +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_poly_mulcache_compute_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(poly_mulcache_compute_avx2_asm) +MLK_ASM_FN_SYMBOL(poly_mulcache_compute_avx2_asm) + + .cfi_startproc + movl $0xd010d01, %eax # imm = 0xD010D01 + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + vmovdqa 0x20(%rsi), %ymm2 + vmovdqa 0x60(%rsi), %ymm3 + vmovdqa 0x3e0(%rdx), %ymm4 + vmovdqa 0x460(%rdx), %ymm1 + vpmullw %ymm2, %ymm1, %ymm5 + vpmullw %ymm3, %ymm1, %ymm6 + vpmulhw %ymm2, %ymm4, %ymm7 + vpmulhw %ymm3, %ymm4, %ymm8 + vpmulhw %ymm5, %ymm0, %ymm9 + vpmulhw %ymm6, %ymm0, %ymm10 + vpsubw %ymm9, %ymm7, %ymm7 + vpsubw %ymm10, %ymm8, %ymm8 + vmovdqa %ymm7, (%rdi) + vmovdqa %ymm8, 0x20(%rdi) + vmovdqa 0xa0(%rsi), %ymm2 + vmovdqa 0xe0(%rsi), %ymm3 + vmovdqa 0x400(%rdx), %ymm4 + vmovdqa 0x480(%rdx), %ymm1 + vpmullw %ymm2, %ymm1, %ymm5 + vpmullw %ymm3, %ymm1, %ymm6 + vpmulhw %ymm2, %ymm4, %ymm7 + vpmulhw %ymm3, %ymm4, %ymm8 + vpmulhw %ymm5, %ymm0, %ymm9 + vpmulhw %ymm6, %ymm0, %ymm10 + vpsubw %ymm9, %ymm7, %ymm7 + vpsubw %ymm10, %ymm8, %ymm8 + vmovdqa %ymm7, 0x40(%rdi) + vmovdqa %ymm8, 0x60(%rdi) + vmovdqa 0x120(%rsi), %ymm2 + vmovdqa 0x160(%rsi), %ymm3 + vmovdqa 0x420(%rdx), %ymm4 + vmovdqa 0x4a0(%rdx), %ymm1 + vpmullw %ymm2, %ymm1, %ymm5 + vpmullw %ymm3, %ymm1, %ymm6 + vpmulhw %ymm2, %ymm4, %ymm7 + vpmulhw %ymm3, %ymm4, %ymm8 + vpmulhw %ymm5, %ymm0, %ymm9 + vpmulhw %ymm6, %ymm0, %ymm10 + vpsubw %ymm9, %ymm7, %ymm7 + vpsubw %ymm10, %ymm8, %ymm8 + vmovdqa %ymm7, 0x80(%rdi) + vmovdqa %ymm8, 0xa0(%rdi) + vmovdqa 0x1a0(%rsi), %ymm2 + vmovdqa 0x1e0(%rsi), %ymm3 + vmovdqa 0x440(%rdx), %ymm4 + vmovdqa 0x4c0(%rdx), %ymm1 + vpmullw %ymm2, %ymm1, %ymm5 + vpmullw %ymm3, %ymm1, %ymm6 + vpmulhw %ymm2, %ymm4, %ymm7 + vpmulhw %ymm3, %ymm4, %ymm8 + vpmulhw %ymm5, %ymm0, %ymm9 + vpmulhw %ymm6, %ymm0, %ymm10 + vpsubw %ymm9, %ymm7, %ymm7 + vpsubw %ymm10, %ymm8, %ymm8 + vmovdqa %ymm7, 0xc0(%rdi) + vmovdqa %ymm8, 0xe0(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(poly_mulcache_compute_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm.S @@ -0,0 +1,536 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 2) +/*yaml + Name: polyvec_basemul_acc_montgomery_cached_k2_avx2_asm + Description: x86_64 AVX2 base multiplication with accumulation (k=2) + Signature: void mlk_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm(int16_t *r, const int16_t *a, const int16_t *b, const int16_t *b_cache) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 512 + permissions: read/write + c_parameter: int16_t *r + description: Output polynomial (256 x int16_t) + rsi: + type: buffer + size_bytes: 1024 + permissions: read-only + c_parameter: const int16_t *a + description: Input polyvec a (2 x 256 x int16_t) + rdx: + type: buffer + size_bytes: 1024 + permissions: read-only + c_parameter: const int16_t *b + description: Input polyvec b (2 x 256 x int16_t) + rcx: + type: buffer + size_bytes: 512 + permissions: read-only + c_parameter: const int16_t *b_cache + description: Mulcache for b (2 x 128 x int16_t) +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k2_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k2_avx2_asm) +MLK_ASM_FN_SYMBOL(polyvec_basemul_acc_montgomery_cached_k2_avx2_asm) + + .cfi_startproc + movl $0xd010d01, %eax # imm = 0xD010D01 + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + movl $0xf301f301, %eax # imm = 0xF301F301 + vmovd %eax, %xmm1 + vpbroadcastd %xmm1, %ymm1 + vmovdqa (%rsi), %ymm2 + vmovdqa 0x20(%rsi), %ymm3 + vmovdqa (%rdx), %ymm4 + vmovdqa 0x20(%rdx), %ymm5 + vmovdqa (%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, (%rdi) + vmovdqa %ymm9, 0x20(%rdi) + vmovdqa 0x40(%rsi), %ymm2 + vmovdqa 0x60(%rsi), %ymm3 + vmovdqa 0x40(%rdx), %ymm4 + vmovdqa 0x60(%rdx), %ymm5 + vmovdqa 0x20(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x40(%rdi) + vmovdqa %ymm9, 0x60(%rdi) + vmovdqa 0x80(%rsi), %ymm2 + vmovdqa 0xa0(%rsi), %ymm3 + vmovdqa 0x80(%rdx), %ymm4 + vmovdqa 0xa0(%rdx), %ymm5 + vmovdqa 0x40(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x80(%rdi) + vmovdqa %ymm9, 0xa0(%rdi) + vmovdqa 0xc0(%rsi), %ymm2 + vmovdqa 0xe0(%rsi), %ymm3 + vmovdqa 0xc0(%rdx), %ymm4 + vmovdqa 0xe0(%rdx), %ymm5 + vmovdqa 0x60(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0xc0(%rdi) + vmovdqa %ymm9, 0xe0(%rdi) + vmovdqa 0x100(%rsi), %ymm2 + vmovdqa 0x120(%rsi), %ymm3 + vmovdqa 0x100(%rdx), %ymm4 + vmovdqa 0x120(%rdx), %ymm5 + vmovdqa 0x80(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x100(%rdi) + vmovdqa %ymm9, 0x120(%rdi) + vmovdqa 0x140(%rsi), %ymm2 + vmovdqa 0x160(%rsi), %ymm3 + vmovdqa 0x140(%rdx), %ymm4 + vmovdqa 0x160(%rdx), %ymm5 + vmovdqa 0xa0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x140(%rdi) + vmovdqa %ymm9, 0x160(%rdi) + vmovdqa 0x180(%rsi), %ymm2 + vmovdqa 0x1a0(%rsi), %ymm3 + vmovdqa 0x180(%rdx), %ymm4 + vmovdqa 0x1a0(%rdx), %ymm5 + vmovdqa 0xc0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x180(%rdi) + vmovdqa %ymm9, 0x1a0(%rdi) + vmovdqa 0x1c0(%rsi), %ymm2 + vmovdqa 0x1e0(%rsi), %ymm3 + vmovdqa 0x1c0(%rdx), %ymm4 + vmovdqa 0x1e0(%rdx), %ymm5 + vmovdqa 0xe0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x1c0(%rdi) + vmovdqa %ymm9, 0x1e0(%rdi) + vmovdqa 0x200(%rsi), %ymm2 + vmovdqa 0x220(%rsi), %ymm3 + vmovdqa 0x200(%rdx), %ymm4 + vmovdqa 0x220(%rdx), %ymm5 + vmovdqa 0x100(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa (%rdi), %ymm8 + vmovdqa 0x20(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, (%rdi) + vmovdqa %ymm9, 0x20(%rdi) + vmovdqa 0x240(%rsi), %ymm2 + vmovdqa 0x260(%rsi), %ymm3 + vmovdqa 0x240(%rdx), %ymm4 + vmovdqa 0x260(%rdx), %ymm5 + vmovdqa 0x120(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x40(%rdi), %ymm8 + vmovdqa 0x60(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x40(%rdi) + vmovdqa %ymm9, 0x60(%rdi) + vmovdqa 0x280(%rsi), %ymm2 + vmovdqa 0x2a0(%rsi), %ymm3 + vmovdqa 0x280(%rdx), %ymm4 + vmovdqa 0x2a0(%rdx), %ymm5 + vmovdqa 0x140(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x80(%rdi), %ymm8 + vmovdqa 0xa0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x80(%rdi) + vmovdqa %ymm9, 0xa0(%rdi) + vmovdqa 0x2c0(%rsi), %ymm2 + vmovdqa 0x2e0(%rsi), %ymm3 + vmovdqa 0x2c0(%rdx), %ymm4 + vmovdqa 0x2e0(%rdx), %ymm5 + vmovdqa 0x160(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0xc0(%rdi), %ymm8 + vmovdqa 0xe0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0xc0(%rdi) + vmovdqa %ymm9, 0xe0(%rdi) + vmovdqa 0x300(%rsi), %ymm2 + vmovdqa 0x320(%rsi), %ymm3 + vmovdqa 0x300(%rdx), %ymm4 + vmovdqa 0x320(%rdx), %ymm5 + vmovdqa 0x180(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x100(%rdi), %ymm8 + vmovdqa 0x120(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x100(%rdi) + vmovdqa %ymm9, 0x120(%rdi) + vmovdqa 0x340(%rsi), %ymm2 + vmovdqa 0x360(%rsi), %ymm3 + vmovdqa 0x340(%rdx), %ymm4 + vmovdqa 0x360(%rdx), %ymm5 + vmovdqa 0x1a0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x140(%rdi), %ymm8 + vmovdqa 0x160(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x140(%rdi) + vmovdqa %ymm9, 0x160(%rdi) + vmovdqa 0x380(%rsi), %ymm2 + vmovdqa 0x3a0(%rsi), %ymm3 + vmovdqa 0x380(%rdx), %ymm4 + vmovdqa 0x3a0(%rdx), %ymm5 + vmovdqa 0x1c0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x180(%rdi), %ymm8 + vmovdqa 0x1a0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x180(%rdi) + vmovdqa %ymm9, 0x1a0(%rdi) + vmovdqa 0x3c0(%rsi), %ymm2 + vmovdqa 0x3e0(%rsi), %ymm3 + vmovdqa 0x3c0(%rdx), %ymm4 + vmovdqa 0x3e0(%rdx), %ymm5 + vmovdqa 0x1e0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x1c0(%rdi), %ymm8 + vmovdqa 0x1e0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x1c0(%rdi) + vmovdqa %ymm9, 0x1e0(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(polyvec_basemul_acc_montgomery_cached_k2_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + && (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 2) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm.S @@ -0,0 +1,784 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 3) +/*yaml + Name: polyvec_basemul_acc_montgomery_cached_k3_avx2_asm + Description: x86_64 AVX2 base multiplication with accumulation (k=3) + Signature: void mlk_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm(int16_t *r, const int16_t *a, const int16_t *b, const int16_t *b_cache) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 512 + permissions: read/write + c_parameter: int16_t *r + description: Output polynomial (256 x int16_t) + rsi: + type: buffer + size_bytes: 1536 + permissions: read-only + c_parameter: const int16_t *a + description: Input polyvec a (3 x 256 x int16_t) + rdx: + type: buffer + size_bytes: 1536 + permissions: read-only + c_parameter: const int16_t *b + description: Input polyvec b (3 x 256 x int16_t) + rcx: + type: buffer + size_bytes: 768 + permissions: read-only + c_parameter: const int16_t *b_cache + description: Mulcache for b (3 x 128 x int16_t) +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k3_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k3_avx2_asm) +MLK_ASM_FN_SYMBOL(polyvec_basemul_acc_montgomery_cached_k3_avx2_asm) + + .cfi_startproc + movl $0xd010d01, %eax # imm = 0xD010D01 + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + movl $0xf301f301, %eax # imm = 0xF301F301 + vmovd %eax, %xmm1 + vpbroadcastd %xmm1, %ymm1 + vmovdqa (%rsi), %ymm2 + vmovdqa 0x20(%rsi), %ymm3 + vmovdqa (%rdx), %ymm4 + vmovdqa 0x20(%rdx), %ymm5 + vmovdqa (%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, (%rdi) + vmovdqa %ymm9, 0x20(%rdi) + vmovdqa 0x40(%rsi), %ymm2 + vmovdqa 0x60(%rsi), %ymm3 + vmovdqa 0x40(%rdx), %ymm4 + vmovdqa 0x60(%rdx), %ymm5 + vmovdqa 0x20(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x40(%rdi) + vmovdqa %ymm9, 0x60(%rdi) + vmovdqa 0x80(%rsi), %ymm2 + vmovdqa 0xa0(%rsi), %ymm3 + vmovdqa 0x80(%rdx), %ymm4 + vmovdqa 0xa0(%rdx), %ymm5 + vmovdqa 0x40(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x80(%rdi) + vmovdqa %ymm9, 0xa0(%rdi) + vmovdqa 0xc0(%rsi), %ymm2 + vmovdqa 0xe0(%rsi), %ymm3 + vmovdqa 0xc0(%rdx), %ymm4 + vmovdqa 0xe0(%rdx), %ymm5 + vmovdqa 0x60(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0xc0(%rdi) + vmovdqa %ymm9, 0xe0(%rdi) + vmovdqa 0x100(%rsi), %ymm2 + vmovdqa 0x120(%rsi), %ymm3 + vmovdqa 0x100(%rdx), %ymm4 + vmovdqa 0x120(%rdx), %ymm5 + vmovdqa 0x80(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x100(%rdi) + vmovdqa %ymm9, 0x120(%rdi) + vmovdqa 0x140(%rsi), %ymm2 + vmovdqa 0x160(%rsi), %ymm3 + vmovdqa 0x140(%rdx), %ymm4 + vmovdqa 0x160(%rdx), %ymm5 + vmovdqa 0xa0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x140(%rdi) + vmovdqa %ymm9, 0x160(%rdi) + vmovdqa 0x180(%rsi), %ymm2 + vmovdqa 0x1a0(%rsi), %ymm3 + vmovdqa 0x180(%rdx), %ymm4 + vmovdqa 0x1a0(%rdx), %ymm5 + vmovdqa 0xc0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x180(%rdi) + vmovdqa %ymm9, 0x1a0(%rdi) + vmovdqa 0x1c0(%rsi), %ymm2 + vmovdqa 0x1e0(%rsi), %ymm3 + vmovdqa 0x1c0(%rdx), %ymm4 + vmovdqa 0x1e0(%rdx), %ymm5 + vmovdqa 0xe0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x1c0(%rdi) + vmovdqa %ymm9, 0x1e0(%rdi) + vmovdqa 0x200(%rsi), %ymm2 + vmovdqa 0x220(%rsi), %ymm3 + vmovdqa 0x200(%rdx), %ymm4 + vmovdqa 0x220(%rdx), %ymm5 + vmovdqa 0x100(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa (%rdi), %ymm8 + vmovdqa 0x20(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, (%rdi) + vmovdqa %ymm9, 0x20(%rdi) + vmovdqa 0x240(%rsi), %ymm2 + vmovdqa 0x260(%rsi), %ymm3 + vmovdqa 0x240(%rdx), %ymm4 + vmovdqa 0x260(%rdx), %ymm5 + vmovdqa 0x120(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x40(%rdi), %ymm8 + vmovdqa 0x60(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x40(%rdi) + vmovdqa %ymm9, 0x60(%rdi) + vmovdqa 0x280(%rsi), %ymm2 + vmovdqa 0x2a0(%rsi), %ymm3 + vmovdqa 0x280(%rdx), %ymm4 + vmovdqa 0x2a0(%rdx), %ymm5 + vmovdqa 0x140(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x80(%rdi), %ymm8 + vmovdqa 0xa0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x80(%rdi) + vmovdqa %ymm9, 0xa0(%rdi) + vmovdqa 0x2c0(%rsi), %ymm2 + vmovdqa 0x2e0(%rsi), %ymm3 + vmovdqa 0x2c0(%rdx), %ymm4 + vmovdqa 0x2e0(%rdx), %ymm5 + vmovdqa 0x160(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0xc0(%rdi), %ymm8 + vmovdqa 0xe0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0xc0(%rdi) + vmovdqa %ymm9, 0xe0(%rdi) + vmovdqa 0x300(%rsi), %ymm2 + vmovdqa 0x320(%rsi), %ymm3 + vmovdqa 0x300(%rdx), %ymm4 + vmovdqa 0x320(%rdx), %ymm5 + vmovdqa 0x180(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x100(%rdi), %ymm8 + vmovdqa 0x120(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x100(%rdi) + vmovdqa %ymm9, 0x120(%rdi) + vmovdqa 0x340(%rsi), %ymm2 + vmovdqa 0x360(%rsi), %ymm3 + vmovdqa 0x340(%rdx), %ymm4 + vmovdqa 0x360(%rdx), %ymm5 + vmovdqa 0x1a0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x140(%rdi), %ymm8 + vmovdqa 0x160(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x140(%rdi) + vmovdqa %ymm9, 0x160(%rdi) + vmovdqa 0x380(%rsi), %ymm2 + vmovdqa 0x3a0(%rsi), %ymm3 + vmovdqa 0x380(%rdx), %ymm4 + vmovdqa 0x3a0(%rdx), %ymm5 + vmovdqa 0x1c0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x180(%rdi), %ymm8 + vmovdqa 0x1a0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x180(%rdi) + vmovdqa %ymm9, 0x1a0(%rdi) + vmovdqa 0x3c0(%rsi), %ymm2 + vmovdqa 0x3e0(%rsi), %ymm3 + vmovdqa 0x3c0(%rdx), %ymm4 + vmovdqa 0x3e0(%rdx), %ymm5 + vmovdqa 0x1e0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x1c0(%rdi), %ymm8 + vmovdqa 0x1e0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x1c0(%rdi) + vmovdqa %ymm9, 0x1e0(%rdi) + vmovdqa 0x400(%rsi), %ymm2 + vmovdqa 0x420(%rsi), %ymm3 + vmovdqa 0x400(%rdx), %ymm4 + vmovdqa 0x420(%rdx), %ymm5 + vmovdqa 0x200(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa (%rdi), %ymm8 + vmovdqa 0x20(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, (%rdi) + vmovdqa %ymm9, 0x20(%rdi) + vmovdqa 0x440(%rsi), %ymm2 + vmovdqa 0x460(%rsi), %ymm3 + vmovdqa 0x440(%rdx), %ymm4 + vmovdqa 0x460(%rdx), %ymm5 + vmovdqa 0x220(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x40(%rdi), %ymm8 + vmovdqa 0x60(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x40(%rdi) + vmovdqa %ymm9, 0x60(%rdi) + vmovdqa 0x480(%rsi), %ymm2 + vmovdqa 0x4a0(%rsi), %ymm3 + vmovdqa 0x480(%rdx), %ymm4 + vmovdqa 0x4a0(%rdx), %ymm5 + vmovdqa 0x240(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x80(%rdi), %ymm8 + vmovdqa 0xa0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x80(%rdi) + vmovdqa %ymm9, 0xa0(%rdi) + vmovdqa 0x4c0(%rsi), %ymm2 + vmovdqa 0x4e0(%rsi), %ymm3 + vmovdqa 0x4c0(%rdx), %ymm4 + vmovdqa 0x4e0(%rdx), %ymm5 + vmovdqa 0x260(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0xc0(%rdi), %ymm8 + vmovdqa 0xe0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0xc0(%rdi) + vmovdqa %ymm9, 0xe0(%rdi) + vmovdqa 0x500(%rsi), %ymm2 + vmovdqa 0x520(%rsi), %ymm3 + vmovdqa 0x500(%rdx), %ymm4 + vmovdqa 0x520(%rdx), %ymm5 + vmovdqa 0x280(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x100(%rdi), %ymm8 + vmovdqa 0x120(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x100(%rdi) + vmovdqa %ymm9, 0x120(%rdi) + vmovdqa 0x540(%rsi), %ymm2 + vmovdqa 0x560(%rsi), %ymm3 + vmovdqa 0x540(%rdx), %ymm4 + vmovdqa 0x560(%rdx), %ymm5 + vmovdqa 0x2a0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x140(%rdi), %ymm8 + vmovdqa 0x160(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x140(%rdi) + vmovdqa %ymm9, 0x160(%rdi) + vmovdqa 0x580(%rsi), %ymm2 + vmovdqa 0x5a0(%rsi), %ymm3 + vmovdqa 0x580(%rdx), %ymm4 + vmovdqa 0x5a0(%rdx), %ymm5 + vmovdqa 0x2c0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x180(%rdi), %ymm8 + vmovdqa 0x1a0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x180(%rdi) + vmovdqa %ymm9, 0x1a0(%rdi) + vmovdqa 0x5c0(%rsi), %ymm2 + vmovdqa 0x5e0(%rsi), %ymm3 + vmovdqa 0x5c0(%rdx), %ymm4 + vmovdqa 0x5e0(%rdx), %ymm5 + vmovdqa 0x2e0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x1c0(%rdi), %ymm8 + vmovdqa 0x1e0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x1c0(%rdi) + vmovdqa %ymm9, 0x1e0(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(polyvec_basemul_acc_montgomery_cached_k3_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + && (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 3) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm.S @@ -0,0 +1,1032 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + (defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_K == 4) +/*yaml + Name: polyvec_basemul_acc_montgomery_cached_k4_avx2_asm + Description: x86_64 AVX2 base multiplication with accumulation (k=4) + Signature: void mlk_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm(int16_t *r, const int16_t *a, const int16_t *b, const int16_t *b_cache) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 512 + permissions: read/write + c_parameter: int16_t *r + description: Output polynomial (256 x int16_t) + rsi: + type: buffer + size_bytes: 2048 + permissions: read-only + c_parameter: const int16_t *a + description: Input polyvec a (4 x 256 x int16_t) + rdx: + type: buffer + size_bytes: 2048 + permissions: read-only + c_parameter: const int16_t *b + description: Input polyvec b (4 x 256 x int16_t) + rcx: + type: buffer + size_bytes: 1024 + permissions: read-only + c_parameter: const int16_t *b_cache + description: Mulcache for b (4 x 128 x int16_t) +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_polyvec_basemul_acc_montgomery_cached_k4_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(polyvec_basemul_acc_montgomery_cached_k4_avx2_asm) +MLK_ASM_FN_SYMBOL(polyvec_basemul_acc_montgomery_cached_k4_avx2_asm) + + .cfi_startproc + movl $0xd010d01, %eax # imm = 0xD010D01 + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + movl $0xf301f301, %eax # imm = 0xF301F301 + vmovd %eax, %xmm1 + vpbroadcastd %xmm1, %ymm1 + vmovdqa (%rsi), %ymm2 + vmovdqa 0x20(%rsi), %ymm3 + vmovdqa (%rdx), %ymm4 + vmovdqa 0x20(%rdx), %ymm5 + vmovdqa (%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, (%rdi) + vmovdqa %ymm9, 0x20(%rdi) + vmovdqa 0x40(%rsi), %ymm2 + vmovdqa 0x60(%rsi), %ymm3 + vmovdqa 0x40(%rdx), %ymm4 + vmovdqa 0x60(%rdx), %ymm5 + vmovdqa 0x20(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x40(%rdi) + vmovdqa %ymm9, 0x60(%rdi) + vmovdqa 0x80(%rsi), %ymm2 + vmovdqa 0xa0(%rsi), %ymm3 + vmovdqa 0x80(%rdx), %ymm4 + vmovdqa 0xa0(%rdx), %ymm5 + vmovdqa 0x40(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x80(%rdi) + vmovdqa %ymm9, 0xa0(%rdi) + vmovdqa 0xc0(%rsi), %ymm2 + vmovdqa 0xe0(%rsi), %ymm3 + vmovdqa 0xc0(%rdx), %ymm4 + vmovdqa 0xe0(%rdx), %ymm5 + vmovdqa 0x60(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0xc0(%rdi) + vmovdqa %ymm9, 0xe0(%rdi) + vmovdqa 0x100(%rsi), %ymm2 + vmovdqa 0x120(%rsi), %ymm3 + vmovdqa 0x100(%rdx), %ymm4 + vmovdqa 0x120(%rdx), %ymm5 + vmovdqa 0x80(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x100(%rdi) + vmovdqa %ymm9, 0x120(%rdi) + vmovdqa 0x140(%rsi), %ymm2 + vmovdqa 0x160(%rsi), %ymm3 + vmovdqa 0x140(%rdx), %ymm4 + vmovdqa 0x160(%rdx), %ymm5 + vmovdqa 0xa0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x140(%rdi) + vmovdqa %ymm9, 0x160(%rdi) + vmovdqa 0x180(%rsi), %ymm2 + vmovdqa 0x1a0(%rsi), %ymm3 + vmovdqa 0x180(%rdx), %ymm4 + vmovdqa 0x1a0(%rdx), %ymm5 + vmovdqa 0xc0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x180(%rdi) + vmovdqa %ymm9, 0x1a0(%rdi) + vmovdqa 0x1c0(%rsi), %ymm2 + vmovdqa 0x1e0(%rsi), %ymm3 + vmovdqa 0x1c0(%rdx), %ymm4 + vmovdqa 0x1e0(%rdx), %ymm5 + vmovdqa 0xe0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x1c0(%rdi) + vmovdqa %ymm9, 0x1e0(%rdi) + vmovdqa 0x200(%rsi), %ymm2 + vmovdqa 0x220(%rsi), %ymm3 + vmovdqa 0x200(%rdx), %ymm4 + vmovdqa 0x220(%rdx), %ymm5 + vmovdqa 0x100(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa (%rdi), %ymm8 + vmovdqa 0x20(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, (%rdi) + vmovdqa %ymm9, 0x20(%rdi) + vmovdqa 0x240(%rsi), %ymm2 + vmovdqa 0x260(%rsi), %ymm3 + vmovdqa 0x240(%rdx), %ymm4 + vmovdqa 0x260(%rdx), %ymm5 + vmovdqa 0x120(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x40(%rdi), %ymm8 + vmovdqa 0x60(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x40(%rdi) + vmovdqa %ymm9, 0x60(%rdi) + vmovdqa 0x280(%rsi), %ymm2 + vmovdqa 0x2a0(%rsi), %ymm3 + vmovdqa 0x280(%rdx), %ymm4 + vmovdqa 0x2a0(%rdx), %ymm5 + vmovdqa 0x140(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x80(%rdi), %ymm8 + vmovdqa 0xa0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x80(%rdi) + vmovdqa %ymm9, 0xa0(%rdi) + vmovdqa 0x2c0(%rsi), %ymm2 + vmovdqa 0x2e0(%rsi), %ymm3 + vmovdqa 0x2c0(%rdx), %ymm4 + vmovdqa 0x2e0(%rdx), %ymm5 + vmovdqa 0x160(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0xc0(%rdi), %ymm8 + vmovdqa 0xe0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0xc0(%rdi) + vmovdqa %ymm9, 0xe0(%rdi) + vmovdqa 0x300(%rsi), %ymm2 + vmovdqa 0x320(%rsi), %ymm3 + vmovdqa 0x300(%rdx), %ymm4 + vmovdqa 0x320(%rdx), %ymm5 + vmovdqa 0x180(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x100(%rdi), %ymm8 + vmovdqa 0x120(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x100(%rdi) + vmovdqa %ymm9, 0x120(%rdi) + vmovdqa 0x340(%rsi), %ymm2 + vmovdqa 0x360(%rsi), %ymm3 + vmovdqa 0x340(%rdx), %ymm4 + vmovdqa 0x360(%rdx), %ymm5 + vmovdqa 0x1a0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x140(%rdi), %ymm8 + vmovdqa 0x160(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x140(%rdi) + vmovdqa %ymm9, 0x160(%rdi) + vmovdqa 0x380(%rsi), %ymm2 + vmovdqa 0x3a0(%rsi), %ymm3 + vmovdqa 0x380(%rdx), %ymm4 + vmovdqa 0x3a0(%rdx), %ymm5 + vmovdqa 0x1c0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x180(%rdi), %ymm8 + vmovdqa 0x1a0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x180(%rdi) + vmovdqa %ymm9, 0x1a0(%rdi) + vmovdqa 0x3c0(%rsi), %ymm2 + vmovdqa 0x3e0(%rsi), %ymm3 + vmovdqa 0x3c0(%rdx), %ymm4 + vmovdqa 0x3e0(%rdx), %ymm5 + vmovdqa 0x1e0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x1c0(%rdi), %ymm8 + vmovdqa 0x1e0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x1c0(%rdi) + vmovdqa %ymm9, 0x1e0(%rdi) + vmovdqa 0x400(%rsi), %ymm2 + vmovdqa 0x420(%rsi), %ymm3 + vmovdqa 0x400(%rdx), %ymm4 + vmovdqa 0x420(%rdx), %ymm5 + vmovdqa 0x200(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa (%rdi), %ymm8 + vmovdqa 0x20(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, (%rdi) + vmovdqa %ymm9, 0x20(%rdi) + vmovdqa 0x440(%rsi), %ymm2 + vmovdqa 0x460(%rsi), %ymm3 + vmovdqa 0x440(%rdx), %ymm4 + vmovdqa 0x460(%rdx), %ymm5 + vmovdqa 0x220(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x40(%rdi), %ymm8 + vmovdqa 0x60(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x40(%rdi) + vmovdqa %ymm9, 0x60(%rdi) + vmovdqa 0x480(%rsi), %ymm2 + vmovdqa 0x4a0(%rsi), %ymm3 + vmovdqa 0x480(%rdx), %ymm4 + vmovdqa 0x4a0(%rdx), %ymm5 + vmovdqa 0x240(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x80(%rdi), %ymm8 + vmovdqa 0xa0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x80(%rdi) + vmovdqa %ymm9, 0xa0(%rdi) + vmovdqa 0x4c0(%rsi), %ymm2 + vmovdqa 0x4e0(%rsi), %ymm3 + vmovdqa 0x4c0(%rdx), %ymm4 + vmovdqa 0x4e0(%rdx), %ymm5 + vmovdqa 0x260(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0xc0(%rdi), %ymm8 + vmovdqa 0xe0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0xc0(%rdi) + vmovdqa %ymm9, 0xe0(%rdi) + vmovdqa 0x500(%rsi), %ymm2 + vmovdqa 0x520(%rsi), %ymm3 + vmovdqa 0x500(%rdx), %ymm4 + vmovdqa 0x520(%rdx), %ymm5 + vmovdqa 0x280(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x100(%rdi), %ymm8 + vmovdqa 0x120(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x100(%rdi) + vmovdqa %ymm9, 0x120(%rdi) + vmovdqa 0x540(%rsi), %ymm2 + vmovdqa 0x560(%rsi), %ymm3 + vmovdqa 0x540(%rdx), %ymm4 + vmovdqa 0x560(%rdx), %ymm5 + vmovdqa 0x2a0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x140(%rdi), %ymm8 + vmovdqa 0x160(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x140(%rdi) + vmovdqa %ymm9, 0x160(%rdi) + vmovdqa 0x580(%rsi), %ymm2 + vmovdqa 0x5a0(%rsi), %ymm3 + vmovdqa 0x580(%rdx), %ymm4 + vmovdqa 0x5a0(%rdx), %ymm5 + vmovdqa 0x2c0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x180(%rdi), %ymm8 + vmovdqa 0x1a0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x180(%rdi) + vmovdqa %ymm9, 0x1a0(%rdi) + vmovdqa 0x5c0(%rsi), %ymm2 + vmovdqa 0x5e0(%rsi), %ymm3 + vmovdqa 0x5c0(%rdx), %ymm4 + vmovdqa 0x5e0(%rdx), %ymm5 + vmovdqa 0x2e0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x1c0(%rdi), %ymm8 + vmovdqa 0x1e0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x1c0(%rdi) + vmovdqa %ymm9, 0x1e0(%rdi) + vmovdqa 0x600(%rsi), %ymm2 + vmovdqa 0x620(%rsi), %ymm3 + vmovdqa 0x600(%rdx), %ymm4 + vmovdqa 0x620(%rdx), %ymm5 + vmovdqa 0x300(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa (%rdi), %ymm8 + vmovdqa 0x20(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, (%rdi) + vmovdqa %ymm9, 0x20(%rdi) + vmovdqa 0x640(%rsi), %ymm2 + vmovdqa 0x660(%rsi), %ymm3 + vmovdqa 0x640(%rdx), %ymm4 + vmovdqa 0x660(%rdx), %ymm5 + vmovdqa 0x320(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x40(%rdi), %ymm8 + vmovdqa 0x60(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x40(%rdi) + vmovdqa %ymm9, 0x60(%rdi) + vmovdqa 0x680(%rsi), %ymm2 + vmovdqa 0x6a0(%rsi), %ymm3 + vmovdqa 0x680(%rdx), %ymm4 + vmovdqa 0x6a0(%rdx), %ymm5 + vmovdqa 0x340(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x80(%rdi), %ymm8 + vmovdqa 0xa0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x80(%rdi) + vmovdqa %ymm9, 0xa0(%rdi) + vmovdqa 0x6c0(%rsi), %ymm2 + vmovdqa 0x6e0(%rsi), %ymm3 + vmovdqa 0x6c0(%rdx), %ymm4 + vmovdqa 0x6e0(%rdx), %ymm5 + vmovdqa 0x360(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0xc0(%rdi), %ymm8 + vmovdqa 0xe0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0xc0(%rdi) + vmovdqa %ymm9, 0xe0(%rdi) + vmovdqa 0x700(%rsi), %ymm2 + vmovdqa 0x720(%rsi), %ymm3 + vmovdqa 0x700(%rdx), %ymm4 + vmovdqa 0x720(%rdx), %ymm5 + vmovdqa 0x380(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x100(%rdi), %ymm8 + vmovdqa 0x120(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x100(%rdi) + vmovdqa %ymm9, 0x120(%rdi) + vmovdqa 0x740(%rsi), %ymm2 + vmovdqa 0x760(%rsi), %ymm3 + vmovdqa 0x740(%rdx), %ymm4 + vmovdqa 0x760(%rdx), %ymm5 + vmovdqa 0x3a0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x140(%rdi), %ymm8 + vmovdqa 0x160(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x140(%rdi) + vmovdqa %ymm9, 0x160(%rdi) + vmovdqa 0x780(%rsi), %ymm2 + vmovdqa 0x7a0(%rsi), %ymm3 + vmovdqa 0x780(%rdx), %ymm4 + vmovdqa 0x7a0(%rdx), %ymm5 + vmovdqa 0x3c0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm8, %ymm13, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x180(%rdi), %ymm8 + vmovdqa 0x1a0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x180(%rdi) + vmovdqa %ymm9, 0x1a0(%rdi) + vmovdqa 0x7c0(%rsi), %ymm2 + vmovdqa 0x7e0(%rsi), %ymm3 + vmovdqa 0x7c0(%rdx), %ymm4 + vmovdqa 0x7e0(%rdx), %ymm5 + vmovdqa 0x3e0(%rcx), %ymm6 + vpmullw %ymm2, %ymm1, %ymm13 + vpmullw %ymm3, %ymm1, %ymm14 + vpmullw %ymm13, %ymm4, %ymm7 + vpmullw %ymm13, %ymm5, %ymm9 + vpmullw %ymm14, %ymm6, %ymm8 + vpmullw %ymm14, %ymm4, %ymm10 + vpmulhw %ymm7, %ymm0, %ymm7 + vpmulhw %ymm9, %ymm0, %ymm9 + vpmulhw %ymm8, %ymm0, %ymm8 + vpmulhw %ymm10, %ymm0, %ymm10 + vpmulhw %ymm2, %ymm4, %ymm11 + vpmulhw %ymm2, %ymm5, %ymm12 + vpmulhw %ymm3, %ymm6, %ymm13 + vpmulhw %ymm3, %ymm4, %ymm14 + vpsubw %ymm7, %ymm11, %ymm7 + vpsubw %ymm9, %ymm12, %ymm9 + vpsubw %ymm13, %ymm8, %ymm8 + vpsubw %ymm10, %ymm14, %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa 0x1c0(%rdi), %ymm8 + vmovdqa 0x1e0(%rdi), %ymm10 + vpaddw %ymm7, %ymm8, %ymm7 + vpaddw %ymm9, %ymm10, %ymm9 + vmovdqa %ymm7, 0x1c0(%rdi) + vmovdqa %ymm9, 0x1e0(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(polyvec_basemul_acc_montgomery_cached_k4_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + && (MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_K == 4) */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_reduce_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_reduce_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_reduce_avx2_asm.S @@ -0,0 +1,234 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Kyber optimized AVX2 implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/avx2 + */ + +/* + * This file is derived from the public domain + * AVX2 Kyber implementation @[REF_AVX2]. + * + * Changes: + * - Add call to csub in reduce128_avx to produce outputs + * in [0,1,...,q-1] rather than [0,1,...,q], matching the + * semantics of mlk_poly_reduce(), + * - Use a macro instead of a local function call. + */ + +#include "../../../common.h" + +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) +/*yaml + Name: reduce_avx2_asm + Description: x86_64 AVX2 modular reduction + Signature: void mlk_reduce_avx2_asm(int16_t *r) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 512 + permissions: read/write + c_parameter: int16_t *r + description: Input/output polynomial (256 x int16_t) +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_reduce_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(reduce_avx2_asm) +MLK_ASM_FN_SYMBOL(reduce_avx2_asm) + + .cfi_startproc + movl $0xd010d01, %eax # imm = 0xD010D01 + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + movl $0x4ebf4ebf, %eax # imm = 0x4EBF4EBF + vmovd %eax, %xmm1 + vpbroadcastd %xmm1, %ymm1 + vmovdqa (%rdi), %ymm2 + vmovdqa 0x20(%rdi), %ymm3 + vmovdqa 0x40(%rdi), %ymm4 + vmovdqa 0x60(%rdi), %ymm5 + vmovdqa 0x80(%rdi), %ymm6 + vmovdqa 0xa0(%rdi), %ymm7 + vmovdqa 0xc0(%rdi), %ymm8 + vmovdqa 0xe0(%rdi), %ymm9 + vpmulhw %ymm1, %ymm2, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm2, %ymm2 + vpmulhw %ymm1, %ymm3, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm3, %ymm3 + vpmulhw %ymm1, %ymm4, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm4, %ymm4 + vpmulhw %ymm1, %ymm5, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm5, %ymm5 + vpmulhw %ymm1, %ymm6, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm6, %ymm6 + vpmulhw %ymm1, %ymm7, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm7, %ymm7 + vpmulhw %ymm1, %ymm8, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm8, %ymm8 + vpmulhw %ymm1, %ymm9, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm9, %ymm9 + vpsubw %ymm0, %ymm2, %ymm2 + vpsraw $0xf, %ymm2, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm2, %ymm2 + vpsubw %ymm0, %ymm3, %ymm3 + vpsraw $0xf, %ymm3, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm3, %ymm3 + vpsubw %ymm0, %ymm4, %ymm4 + vpsraw $0xf, %ymm4, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm4, %ymm4 + vpsubw %ymm0, %ymm5, %ymm5 + vpsraw $0xf, %ymm5, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm5, %ymm5 + vpsubw %ymm0, %ymm6, %ymm6 + vpsraw $0xf, %ymm6, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm6, %ymm6 + vpsubw %ymm0, %ymm7, %ymm7 + vpsraw $0xf, %ymm7, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm7, %ymm7 + vpsubw %ymm0, %ymm8, %ymm8 + vpsraw $0xf, %ymm8, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm8, %ymm8 + vpsubw %ymm0, %ymm9, %ymm9 + vpsraw $0xf, %ymm9, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm9, %ymm9 + vmovdqa %ymm2, (%rdi) + vmovdqa %ymm3, 0x20(%rdi) + vmovdqa %ymm4, 0x40(%rdi) + vmovdqa %ymm5, 0x60(%rdi) + vmovdqa %ymm6, 0x80(%rdi) + vmovdqa %ymm7, 0xa0(%rdi) + vmovdqa %ymm8, 0xc0(%rdi) + vmovdqa %ymm9, 0xe0(%rdi) + vmovdqa 0x100(%rdi), %ymm2 + vmovdqa 0x120(%rdi), %ymm3 + vmovdqa 0x140(%rdi), %ymm4 + vmovdqa 0x160(%rdi), %ymm5 + vmovdqa 0x180(%rdi), %ymm6 + vmovdqa 0x1a0(%rdi), %ymm7 + vmovdqa 0x1c0(%rdi), %ymm8 + vmovdqa 0x1e0(%rdi), %ymm9 + vpmulhw %ymm1, %ymm2, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm2, %ymm2 + vpmulhw %ymm1, %ymm3, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm3, %ymm3 + vpmulhw %ymm1, %ymm4, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm4, %ymm4 + vpmulhw %ymm1, %ymm5, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm5, %ymm5 + vpmulhw %ymm1, %ymm6, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm6, %ymm6 + vpmulhw %ymm1, %ymm7, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm7, %ymm7 + vpmulhw %ymm1, %ymm8, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm8, %ymm8 + vpmulhw %ymm1, %ymm9, %ymm12 + vpsraw $0xa, %ymm12, %ymm12 + vpmullw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm9, %ymm9 + vpsubw %ymm0, %ymm2, %ymm2 + vpsraw $0xf, %ymm2, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm2, %ymm2 + vpsubw %ymm0, %ymm3, %ymm3 + vpsraw $0xf, %ymm3, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm3, %ymm3 + vpsubw %ymm0, %ymm4, %ymm4 + vpsraw $0xf, %ymm4, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm4, %ymm4 + vpsubw %ymm0, %ymm5, %ymm5 + vpsraw $0xf, %ymm5, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm5, %ymm5 + vpsubw %ymm0, %ymm6, %ymm6 + vpsraw $0xf, %ymm6, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm6, %ymm6 + vpsubw %ymm0, %ymm7, %ymm7 + vpsraw $0xf, %ymm7, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm7, %ymm7 + vpsubw %ymm0, %ymm8, %ymm8 + vpsraw $0xf, %ymm8, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm8, %ymm8 + vpsubw %ymm0, %ymm9, %ymm9 + vpsraw $0xf, %ymm9, %ymm12 + vpand %ymm0, %ymm12, %ymm12 + vpaddw %ymm12, %ymm9, %ymm9 + vmovdqa %ymm2, 0x100(%rdi) + vmovdqa %ymm3, 0x120(%rdi) + vmovdqa %ymm4, 0x140(%rdi) + vmovdqa %ymm5, 0x160(%rdi) + vmovdqa %ymm6, 0x180(%rdi) + vmovdqa %ymm7, 0x1a0(%rdi) + vmovdqa %ymm8, 0x1c0(%rdi) + vmovdqa %ymm9, 0x1e0(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(reduce_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_rej_uniform_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_rej_uniform_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_rej_uniform_avx2_asm.S @@ -0,0 +1,136 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/************************************************* + * Name: mlk_rej_uniform_avx2_asm + * + * Description: Run rejection sampling on uniform random bytes to generate + * uniform random integers mod q + * + * Arguments: - int16_t *r: pointer to output buffer of MLKEM_N + * 16-bit coefficients. + * - const uint8_t *buf: pointer to input buffer + * (assumed to be uniform random bytes) + * - unsigned buflen: length of input buffer in bytes. + * Must be a multiple of 12. + * + * Returns number of sampled 16-bit integers (at most MLKEM_N). + **************************************************/ +#include "../../../common.h" + +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) +/*yaml + Name: rej_uniform_avx2_asm + Description: x86_64 AVX2 rejection sampling + Signature: uint64_t mlk_rej_uniform_avx2_asm(int16_t *r, const uint8_t *buf, unsigned buflen, const uint8_t *table) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 512 + permissions: write-only + c_parameter: int16_t *r + description: Output buffer (256 x int16_t) + rsi: + type: buffer + size_bytes: rdx + permissions: read-only + c_parameter: const uint8_t *buf + description: Input buffer + rdx: + type: scalar + c_parameter: unsigned buflen + description: Length of input buffer (must be multiple of 12) + test_with: 504 # MLKEM_GEN_MATRIX_NBLOCKS * MLK_XOF_RATE + rcx: + type: buffer + size_bytes: 4096 + permissions: read-only + c_parameter: const uint8_t *table + description: Lookup table +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_rej_uniform_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(rej_uniform_avx2_asm) +MLK_ASM_FN_SYMBOL(rej_uniform_avx2_asm) + + .cfi_startproc + subq $0x210, %rsp # imm = 0x210 + .cfi_adjust_cfa_offset 0x210 + xorl %eax, %eax + testq %rdx, %rdx + je Lmlk_rej_uniform_asm_end + movabsq $0xd010d010d010d01, %rax # imm = 0xD010D010D010D01 + movq %rax, %xmm0 + pinsrq $0x1, %rax, %xmm0 + movabsq $0xfff0fff0fff0fff, %rax # imm = 0xFFF0FFF0FFF0FFF + movq %rax, %xmm5 + pinsrq $0x1, %rax, %xmm5 + movabsq $0x504040302010100, %rax # imm = 0x504040302010100 + movq %rax, %xmm4 + movabsq $0xb0a0a0908070706, %rax # imm = 0xB0A0A0908070706 + pinsrq $0x1, %rax, %xmm4 + movq $0x0, %rax + movq $0x0, %r8 + movq $0x5555, %r9 # imm = 0x5555 + +Lmlk_rej_uniform_asm_loop_start: + movq (%rsi,%r8), %xmm2 + pinsrd $0x2, 0x8(%rsi,%r8), %xmm2 + pshufb %xmm4, %xmm2 + movdqa %xmm2, %xmm3 + psrlw $0x4, %xmm3 + pblendw $0xaa, %xmm3, %xmm2 # xmm2 = xmm2[0],xmm3[1],xmm2[2],xmm3[3],xmm2[4],xmm3[5],xmm2[6],xmm3[7] + pand %xmm5, %xmm2 + movdqa %xmm0, %xmm1 + pcmpgtw %xmm2, %xmm1 + pmovmskb %xmm1, %r11d + pextq %r9, %r11, %r11 + movq %r11, %r10 + shlq $0x4, %r10 + movdqu (%rcx,%r10), %xmm3 + pshufb %xmm3, %xmm2 + movdqu %xmm2, (%rsp,%rax,2) + popcntq %r11, %r11 + addq %r11, %rax + cmpq $0x100, %rax # imm = 0x100 + jae Lmlk_rej_uniform_asm_final_copy + addq $0xc, %r8 + cmpq %r8, %rdx + ja Lmlk_rej_uniform_asm_loop_start + +Lmlk_rej_uniform_asm_final_copy: + movq $0x100, %rcx # imm = 0x100 + cmpq $0x100, %rax # imm = 0x100 + cmovaq %rcx, %rax + movq %rsp, %rsi + movq %rax, %rcx + shlq %rcx + rep movsb (%rsi), %es:(%rdi) + +Lmlk_rej_uniform_asm_end: + addq $0x210, %rsp # imm = 0x210 + .cfi_adjust_cfa_offset -0x210 + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(rej_uniform_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/mlkem_tomont_avx2_asm.S b/cbits/mlkem/src/native/x86_64/src/mlkem_tomont_avx2_asm.S new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/mlkem_tomont_avx2_asm.S @@ -0,0 +1,172 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [REF_AVX2] + * CRYSTALS-Kyber optimized AVX2 implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/avx2 + */ + +/* + * Implementation from Kyber reference repository @[REF_AVX2] + * + * Changes: + * - Add call to csub in reduce128_avx to produce outputs + * in [0,1,...,q-1] rather than [0,1,...,q], matching the + * semantics of mlk_poly_reduce(), + * - Use a macro instead of a local function call. + */ + +#include "../../../common.h" +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) && \ + !defined(MLK_CONFIG_NO_KEYPAIR_API) +/*yaml + Name: tomont_avx2_asm + Description: x86_64 AVX2 Montgomery conversion + Signature: void mlk_tomont_avx2_asm(int16_t *r) + ABI: + Architecture: x86_64 + CallingConvention: SysV + Features: [AVX2] + rdi: + type: buffer + size_bytes: 512 + permissions: read/write + c_parameter: int16_t *r + description: Input/output polynomial (256 x int16_t) +*/ + + +/* + * WARNING: This file is auto-derived from the mlkem-native source file + * dev/x86_64/src/mlkem_tomont_avx2_asm.S using scripts/simpasm. Do not modify it directly. + */ + +.text +.balign 4 +.global MLK_ASM_NAMESPACE(tomont_avx2_asm) +MLK_ASM_FN_SYMBOL(tomont_avx2_asm) + + .cfi_startproc + movl $0xd010d01, %eax # imm = 0xD010D01 + vmovd %eax, %xmm0 + vpbroadcastd %xmm0, %ymm0 + movl $0x50495049, %eax # imm = 0x50495049 + vmovd %eax, %xmm1 + vpbroadcastd %xmm1, %ymm1 + movl $0x5490549, %eax # imm = 0x5490549 + vmovd %eax, %xmm2 + vpbroadcastd %xmm2, %ymm2 + vmovdqa (%rdi), %ymm3 + vmovdqa 0x20(%rdi), %ymm4 + vmovdqa 0x40(%rdi), %ymm5 + vmovdqa 0x60(%rdi), %ymm6 + vmovdqa 0x80(%rdi), %ymm7 + vmovdqa 0xa0(%rdi), %ymm8 + vmovdqa 0xc0(%rdi), %ymm9 + vmovdqa 0xe0(%rdi), %ymm10 + vpmullw %ymm1, %ymm3, %ymm11 + vpmulhw %ymm2, %ymm3, %ymm3 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm11, %ymm3, %ymm3 + vpmullw %ymm1, %ymm4, %ymm12 + vpmulhw %ymm2, %ymm4, %ymm4 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm4, %ymm4 + vpmullw %ymm1, %ymm5, %ymm13 + vpmulhw %ymm2, %ymm5, %ymm5 + vpmulhw %ymm0, %ymm13, %ymm13 + vpsubw %ymm13, %ymm5, %ymm5 + vpmullw %ymm1, %ymm6, %ymm14 + vpmulhw %ymm2, %ymm6, %ymm6 + vpmulhw %ymm0, %ymm14, %ymm14 + vpsubw %ymm14, %ymm6, %ymm6 + vpmullw %ymm1, %ymm7, %ymm15 + vpmulhw %ymm2, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm15, %ymm15 + vpsubw %ymm15, %ymm7, %ymm7 + vpmullw %ymm1, %ymm8, %ymm11 + vpmulhw %ymm2, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm11, %ymm8, %ymm8 + vpmullw %ymm1, %ymm9, %ymm12 + vpmulhw %ymm2, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm9, %ymm9 + vpmullw %ymm1, %ymm10, %ymm13 + vpmulhw %ymm2, %ymm10, %ymm10 + vpmulhw %ymm0, %ymm13, %ymm13 + vpsubw %ymm13, %ymm10, %ymm10 + vmovdqa %ymm3, (%rdi) + vmovdqa %ymm4, 0x20(%rdi) + vmovdqa %ymm5, 0x40(%rdi) + vmovdqa %ymm6, 0x60(%rdi) + vmovdqa %ymm7, 0x80(%rdi) + vmovdqa %ymm8, 0xa0(%rdi) + vmovdqa %ymm9, 0xc0(%rdi) + vmovdqa %ymm10, 0xe0(%rdi) + vmovdqa 0x100(%rdi), %ymm3 + vmovdqa 0x120(%rdi), %ymm4 + vmovdqa 0x140(%rdi), %ymm5 + vmovdqa 0x160(%rdi), %ymm6 + vmovdqa 0x180(%rdi), %ymm7 + vmovdqa 0x1a0(%rdi), %ymm8 + vmovdqa 0x1c0(%rdi), %ymm9 + vmovdqa 0x1e0(%rdi), %ymm10 + vpmullw %ymm1, %ymm3, %ymm11 + vpmulhw %ymm2, %ymm3, %ymm3 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm11, %ymm3, %ymm3 + vpmullw %ymm1, %ymm4, %ymm12 + vpmulhw %ymm2, %ymm4, %ymm4 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm4, %ymm4 + vpmullw %ymm1, %ymm5, %ymm13 + vpmulhw %ymm2, %ymm5, %ymm5 + vpmulhw %ymm0, %ymm13, %ymm13 + vpsubw %ymm13, %ymm5, %ymm5 + vpmullw %ymm1, %ymm6, %ymm14 + vpmulhw %ymm2, %ymm6, %ymm6 + vpmulhw %ymm0, %ymm14, %ymm14 + vpsubw %ymm14, %ymm6, %ymm6 + vpmullw %ymm1, %ymm7, %ymm15 + vpmulhw %ymm2, %ymm7, %ymm7 + vpmulhw %ymm0, %ymm15, %ymm15 + vpsubw %ymm15, %ymm7, %ymm7 + vpmullw %ymm1, %ymm8, %ymm11 + vpmulhw %ymm2, %ymm8, %ymm8 + vpmulhw %ymm0, %ymm11, %ymm11 + vpsubw %ymm11, %ymm8, %ymm8 + vpmullw %ymm1, %ymm9, %ymm12 + vpmulhw %ymm2, %ymm9, %ymm9 + vpmulhw %ymm0, %ymm12, %ymm12 + vpsubw %ymm12, %ymm9, %ymm9 + vpmullw %ymm1, %ymm10, %ymm13 + vpmulhw %ymm2, %ymm10, %ymm10 + vpmulhw %ymm0, %ymm13, %ymm13 + vpsubw %ymm13, %ymm10, %ymm10 + vmovdqa %ymm3, 0x100(%rdi) + vmovdqa %ymm4, 0x120(%rdi) + vmovdqa %ymm5, 0x140(%rdi) + vmovdqa %ymm6, 0x160(%rdi) + vmovdqa %ymm7, 0x180(%rdi) + vmovdqa %ymm8, 0x1a0(%rdi) + vmovdqa %ymm9, 0x1c0(%rdi) + vmovdqa %ymm10, 0x1e0(%rdi) + retq + .cfi_endproc + +MLK_ASM_FN_SIZE(tomont_avx2_asm) + +#endif /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + && !MLK_CONFIG_NO_KEYPAIR_API */ + +#if defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/mlkem/src/native/x86_64/src/rej_uniform_table.c b/cbits/mlkem/src/native/x86_64/src/rej_uniform_table.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/native/x86_64/src/rej_uniform_table.c @@ -0,0 +1,545 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mlkem-native repository. + * Do not modify it directly. + */ + +#include "../../../common.h" + +#if defined(MLK_ARITH_BACKEND_X86_64_DEFAULT) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +#include "arith_native_x86_64.h" + +/* + * Lookup table used by rejection sampling of the public matrix. + * See autogen for details. + */ +MLK_ALIGN MLK_INTERNAL_DATA_DEFINITION const uint8_t + mlk_rej_uniform_table[4096] = { + 255, 255, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 0 */, + 0, 1, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 1 */, + 2, 3, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 2 */, + 0, 1, 2, 3, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 3 */, + 4, 5, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 4 */, + 0, 1, 4, 5, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 5 */, + 2, 3, 4, 5, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 6 */, + 0, 1, 2, 3, 4, 5, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 7 */, + 6, 7, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 8 */, + 0, 1, 6, 7, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 9 */, + 2, 3, 6, 7, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 10 */, + 0, 1, 2, 3, 6, 7, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 11 */, + 4, 5, 6, 7, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 12 */, + 0, 1, 4, 5, 6, 7, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 13 */, + 2, 3, 4, 5, 6, 7, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 14 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 255, 255, 255, 255, 255, 255, 255, 255 /* 15 */, + 8, 9, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 16 */, + 0, 1, 8, 9, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 17 */, + 2, 3, 8, 9, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 18 */, + 0, 1, 2, 3, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 19 */, + 4, 5, 8, 9, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 20 */, + 0, 1, 4, 5, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 21 */, + 2, 3, 4, 5, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 22 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 255, 255, 255, 255, 255, 255, 255, 255 /* 23 */, + 6, 7, 8, 9, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 24 */, + 0, 1, 6, 7, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 25 */, + 2, 3, 6, 7, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 26 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 255, 255, 255, 255, 255, 255, 255, 255 /* 27 */, + 4, 5, 6, 7, 8, 9, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 28 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 255, 255, 255, 255, 255, 255, 255, 255 /* 29 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 255, 255, 255, 255, 255, 255, 255, 255 /* 30 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 255, 255, 255, 255, 255, 255 /* 31 */, + 10, 11, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 32 */, + 0, 1, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 33 */, + 2, 3, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 34 */, + 0, 1, 2, 3, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 35 */, + 4, 5, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 36 */, + 0, 1, 4, 5, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 37 */, + 2, 3, 4, 5, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 38 */, + 0, 1, 2, 3, 4, 5, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 39 */, + 6, 7, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 40 */, + 0, 1, 6, 7, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 41 */, + 2, 3, 6, 7, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 42 */, + 0, 1, 2, 3, 6, 7, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 43 */, + 4, 5, 6, 7, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 44 */, + 0, 1, 4, 5, 6, 7, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 45 */, + 2, 3, 4, 5, 6, 7, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 46 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 10, 11, 255, 255, 255, 255, 255, 255 /* 47 */, + 8, 9, 10, 11, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 48 */, + 0, 1, 8, 9, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 49 */, + 2, 3, 8, 9, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 50 */, + 0, 1, 2, 3, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 51 */, + 4, 5, 8, 9, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 52 */, + 0, 1, 4, 5, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 53 */, + 2, 3, 4, 5, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 54 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 10, 11, 255, 255, 255, 255, 255, 255 /* 55 */, + 6, 7, 8, 9, 10, 11, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 56 */, + 0, 1, 6, 7, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 57 */, + 2, 3, 6, 7, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 58 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 10, 11, 255, 255, 255, 255, 255, 255 /* 59 */, + 4, 5, 6, 7, 8, 9, 10, 11, + 255, 255, 255, 255, 255, 255, 255, 255 /* 60 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 10, 11, 255, 255, 255, 255, 255, 255 /* 61 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 10, 11, 255, 255, 255, 255, 255, 255 /* 62 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 255, 255, 255, 255 /* 63 */, + 12, 13, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 64 */, + 0, 1, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 65 */, + 2, 3, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 66 */, + 0, 1, 2, 3, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 67 */, + 4, 5, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 68 */, + 0, 1, 4, 5, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 69 */, + 2, 3, 4, 5, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 70 */, + 0, 1, 2, 3, 4, 5, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 71 */, + 6, 7, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 72 */, + 0, 1, 6, 7, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 73 */, + 2, 3, 6, 7, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 74 */, + 0, 1, 2, 3, 6, 7, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 75 */, + 4, 5, 6, 7, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 76 */, + 0, 1, 4, 5, 6, 7, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 77 */, + 2, 3, 4, 5, 6, 7, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 78 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 12, 13, 255, 255, 255, 255, 255, 255 /* 79 */, + 8, 9, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 80 */, + 0, 1, 8, 9, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 81 */, + 2, 3, 8, 9, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 82 */, + 0, 1, 2, 3, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 83 */, + 4, 5, 8, 9, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 84 */, + 0, 1, 4, 5, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 85 */, + 2, 3, 4, 5, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 86 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 12, 13, 255, 255, 255, 255, 255, 255 /* 87 */, + 6, 7, 8, 9, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 88 */, + 0, 1, 6, 7, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 89 */, + 2, 3, 6, 7, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 90 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 12, 13, 255, 255, 255, 255, 255, 255 /* 91 */, + 4, 5, 6, 7, 8, 9, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 92 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 12, 13, 255, 255, 255, 255, 255, 255 /* 93 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 12, 13, 255, 255, 255, 255, 255, 255 /* 94 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 12, 13, 255, 255, 255, 255 /* 95 */, + 10, 11, 12, 13, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 96 */, + 0, 1, 10, 11, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 97 */, + 2, 3, 10, 11, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 98 */, + 0, 1, 2, 3, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 99 */, + 4, 5, 10, 11, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 100 */, + 0, 1, 4, 5, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 101 */, + 2, 3, 4, 5, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 102 */, + 0, 1, 2, 3, 4, 5, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 103 */, + 6, 7, 10, 11, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 104 */, + 0, 1, 6, 7, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 105 */, + 2, 3, 6, 7, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 106 */, + 0, 1, 2, 3, 6, 7, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 107 */, + 4, 5, 6, 7, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 108 */, + 0, 1, 4, 5, 6, 7, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 109 */, + 2, 3, 4, 5, 6, 7, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 110 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 10, 11, 12, 13, 255, 255, 255, 255 /* 111 */, + 8, 9, 10, 11, 12, 13, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 112 */, + 0, 1, 8, 9, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 113 */, + 2, 3, 8, 9, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 114 */, + 0, 1, 2, 3, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 115 */, + 4, 5, 8, 9, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 116 */, + 0, 1, 4, 5, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 117 */, + 2, 3, 4, 5, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 118 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 10, 11, 12, 13, 255, 255, 255, 255 /* 119 */, + 6, 7, 8, 9, 10, 11, 12, 13, + 255, 255, 255, 255, 255, 255, 255, 255 /* 120 */, + 0, 1, 6, 7, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 121 */, + 2, 3, 6, 7, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 122 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 10, 11, 12, 13, 255, 255, 255, 255 /* 123 */, + 4, 5, 6, 7, 8, 9, 10, 11, + 12, 13, 255, 255, 255, 255, 255, 255 /* 124 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 10, 11, 12, 13, 255, 255, 255, 255 /* 125 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 10, 11, 12, 13, 255, 255, 255, 255 /* 126 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 12, 13, 255, 255 /* 127 */, + 14, 15, 255, 255, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 128 */, + 0, 1, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 129 */, + 2, 3, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 130 */, + 0, 1, 2, 3, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 131 */, + 4, 5, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 132 */, + 0, 1, 4, 5, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 133 */, + 2, 3, 4, 5, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 134 */, + 0, 1, 2, 3, 4, 5, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 135 */, + 6, 7, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 136 */, + 0, 1, 6, 7, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 137 */, + 2, 3, 6, 7, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 138 */, + 0, 1, 2, 3, 6, 7, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 139 */, + 4, 5, 6, 7, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 140 */, + 0, 1, 4, 5, 6, 7, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 141 */, + 2, 3, 4, 5, 6, 7, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 142 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 14, 15, 255, 255, 255, 255, 255, 255 /* 143 */, + 8, 9, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 144 */, + 0, 1, 8, 9, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 145 */, + 2, 3, 8, 9, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 146 */, + 0, 1, 2, 3, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 147 */, + 4, 5, 8, 9, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 148 */, + 0, 1, 4, 5, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 149 */, + 2, 3, 4, 5, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 150 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 14, 15, 255, 255, 255, 255, 255, 255 /* 151 */, + 6, 7, 8, 9, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 152 */, + 0, 1, 6, 7, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 153 */, + 2, 3, 6, 7, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 154 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 14, 15, 255, 255, 255, 255, 255, 255 /* 155 */, + 4, 5, 6, 7, 8, 9, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 156 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 14, 15, 255, 255, 255, 255, 255, 255 /* 157 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 14, 15, 255, 255, 255, 255, 255, 255 /* 158 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 14, 15, 255, 255, 255, 255 /* 159 */, + 10, 11, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 160 */, + 0, 1, 10, 11, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 161 */, + 2, 3, 10, 11, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 162 */, + 0, 1, 2, 3, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 163 */, + 4, 5, 10, 11, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 164 */, + 0, 1, 4, 5, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 165 */, + 2, 3, 4, 5, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 166 */, + 0, 1, 2, 3, 4, 5, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 167 */, + 6, 7, 10, 11, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 168 */, + 0, 1, 6, 7, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 169 */, + 2, 3, 6, 7, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 170 */, + 0, 1, 2, 3, 6, 7, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 171 */, + 4, 5, 6, 7, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 172 */, + 0, 1, 4, 5, 6, 7, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 173 */, + 2, 3, 4, 5, 6, 7, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 174 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 10, 11, 14, 15, 255, 255, 255, 255 /* 175 */, + 8, 9, 10, 11, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 176 */, + 0, 1, 8, 9, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 177 */, + 2, 3, 8, 9, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 178 */, + 0, 1, 2, 3, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 179 */, + 4, 5, 8, 9, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 180 */, + 0, 1, 4, 5, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 181 */, + 2, 3, 4, 5, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 182 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 10, 11, 14, 15, 255, 255, 255, 255 /* 183 */, + 6, 7, 8, 9, 10, 11, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 184 */, + 0, 1, 6, 7, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 185 */, + 2, 3, 6, 7, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 186 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 10, 11, 14, 15, 255, 255, 255, 255 /* 187 */, + 4, 5, 6, 7, 8, 9, 10, 11, + 14, 15, 255, 255, 255, 255, 255, 255 /* 188 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 10, 11, 14, 15, 255, 255, 255, 255 /* 189 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 10, 11, 14, 15, 255, 255, 255, 255 /* 190 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 14, 15, 255, 255 /* 191 */, + 12, 13, 14, 15, 255, 255, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 192 */, + 0, 1, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 193 */, + 2, 3, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 194 */, + 0, 1, 2, 3, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 195 */, + 4, 5, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 196 */, + 0, 1, 4, 5, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 197 */, + 2, 3, 4, 5, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 198 */, + 0, 1, 2, 3, 4, 5, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 199 */, + 6, 7, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 200 */, + 0, 1, 6, 7, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 201 */, + 2, 3, 6, 7, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 202 */, + 0, 1, 2, 3, 6, 7, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 203 */, + 4, 5, 6, 7, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 204 */, + 0, 1, 4, 5, 6, 7, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 205 */, + 2, 3, 4, 5, 6, 7, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 206 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 12, 13, 14, 15, 255, 255, 255, 255 /* 207 */, + 8, 9, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 208 */, + 0, 1, 8, 9, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 209 */, + 2, 3, 8, 9, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 210 */, + 0, 1, 2, 3, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 211 */, + 4, 5, 8, 9, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 212 */, + 0, 1, 4, 5, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 213 */, + 2, 3, 4, 5, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 214 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 12, 13, 14, 15, 255, 255, 255, 255 /* 215 */, + 6, 7, 8, 9, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 216 */, + 0, 1, 6, 7, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 217 */, + 2, 3, 6, 7, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 218 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 12, 13, 14, 15, 255, 255, 255, 255 /* 219 */, + 4, 5, 6, 7, 8, 9, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 220 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 12, 13, 14, 15, 255, 255, 255, 255 /* 221 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 12, 13, 14, 15, 255, 255, 255, 255 /* 222 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 12, 13, 14, 15, 255, 255 /* 223 */, + 10, 11, 12, 13, 14, 15, 255, 255, + 255, 255, 255, 255, 255, 255, 255, 255 /* 224 */, + 0, 1, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 225 */, + 2, 3, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 226 */, + 0, 1, 2, 3, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 227 */, + 4, 5, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 228 */, + 0, 1, 4, 5, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 229 */, + 2, 3, 4, 5, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 230 */, + 0, 1, 2, 3, 4, 5, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 231 */, + 6, 7, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 232 */, + 0, 1, 6, 7, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 233 */, + 2, 3, 6, 7, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 234 */, + 0, 1, 2, 3, 6, 7, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 235 */, + 4, 5, 6, 7, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 236 */, + 0, 1, 4, 5, 6, 7, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 237 */, + 2, 3, 4, 5, 6, 7, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 238 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 10, 11, 12, 13, 14, 15, 255, 255 /* 239 */, + 8, 9, 10, 11, 12, 13, 14, 15, + 255, 255, 255, 255, 255, 255, 255, 255 /* 240 */, + 0, 1, 8, 9, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 241 */, + 2, 3, 8, 9, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 242 */, + 0, 1, 2, 3, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 243 */, + 4, 5, 8, 9, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 244 */, + 0, 1, 4, 5, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 245 */, + 2, 3, 4, 5, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 246 */, + 0, 1, 2, 3, 4, 5, 8, 9, + 10, 11, 12, 13, 14, 15, 255, 255 /* 247 */, + 6, 7, 8, 9, 10, 11, 12, 13, + 14, 15, 255, 255, 255, 255, 255, 255 /* 248 */, + 0, 1, 6, 7, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 249 */, + 2, 3, 6, 7, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 250 */, + 0, 1, 2, 3, 6, 7, 8, 9, + 10, 11, 12, 13, 14, 15, 255, 255 /* 251 */, + 4, 5, 6, 7, 8, 9, 10, 11, + 12, 13, 14, 15, 255, 255, 255, 255 /* 252 */, + 0, 1, 4, 5, 6, 7, 8, 9, + 10, 11, 12, 13, 14, 15, 255, 255 /* 253 */, + 2, 3, 4, 5, 6, 7, 8, 9, + 10, 11, 12, 13, 14, 15, 255, 255 /* 254 */, + 0, 1, 2, 3, 4, 5, 6, 7, + 8, 9, 10, 11, 12, 13, 14, 15 /* 255 */, +}; + +#else /* MLK_ARITH_BACKEND_X86_64_DEFAULT && !MLK_CONFIG_MULTILEVEL_NO_SHARED \ + */ + +MLK_EMPTY_CU(avx2_rej_uniform_table) + +#endif /* !(MLK_ARITH_BACKEND_X86_64_DEFAULT && \ + !MLK_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mlkem/src/params.h b/cbits/mlkem/src/params.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/params.h @@ -0,0 +1,76 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLK_PARAMS_H +#define MLK_PARAMS_H + +#if !defined(MLK_CONFIG_PARAMETER_SET) +#error MLK_CONFIG_PARAMETER_SET is not defined +#endif + +#if MLK_CONFIG_PARAMETER_SET == 512 +#define MLKEM_K 2 +#elif MLK_CONFIG_PARAMETER_SET == 768 +#define MLKEM_K 3 +#elif MLK_CONFIG_PARAMETER_SET == 1024 +#define MLKEM_K 4 +#else +#error Invalid value for MLK_CONFIG_PARAMETER_SET. Must be 512, 768, or 1024. +#endif + +#define MLKEM_N 256 +#define MLKEM_Q 3329 +#define MLKEM_Q_HALF ((MLKEM_Q + 1) / 2) /* 1665 */ +#define MLKEM_UINT12_LIMIT 4096 + +#define MLKEM_SYMBYTES 32 /* size in bytes of hashes, and seeds */ +#define MLKEM_SSBYTES 32 /* size in bytes of shared key */ + +#define MLKEM_POLYBYTES 384 +#define MLKEM_POLYVECBYTES (MLKEM_K * MLKEM_POLYBYTES) + +#define MLKEM_POLYCOMPRESSEDBYTES_D4 128 +#define MLKEM_POLYCOMPRESSEDBYTES_D5 160 +#define MLKEM_POLYCOMPRESSEDBYTES_D10 320 +#define MLKEM_POLYCOMPRESSEDBYTES_D11 352 + +#if MLKEM_K == 2 +#define MLKEM_ETA1 3 +#define MLKEM_DU 10 +#define MLKEM_DV 4 +#define MLKEM_POLYCOMPRESSEDBYTES_DV MLKEM_POLYCOMPRESSEDBYTES_D4 +#define MLKEM_POLYCOMPRESSEDBYTES_DU MLKEM_POLYCOMPRESSEDBYTES_D10 +#define MLKEM_POLYVECCOMPRESSEDBYTES_DU (MLKEM_K * MLKEM_POLYCOMPRESSEDBYTES_DU) +#elif MLKEM_K == 3 +#define MLKEM_ETA1 2 +#define MLKEM_DU 10 +#define MLKEM_DV 4 +#define MLKEM_POLYCOMPRESSEDBYTES_DV MLKEM_POLYCOMPRESSEDBYTES_D4 +#define MLKEM_POLYCOMPRESSEDBYTES_DU MLKEM_POLYCOMPRESSEDBYTES_D10 +#define MLKEM_POLYVECCOMPRESSEDBYTES_DU (MLKEM_K * MLKEM_POLYCOMPRESSEDBYTES_DU) +#elif MLKEM_K == 4 +#define MLKEM_ETA1 2 +#define MLKEM_DU 11 +#define MLKEM_DV 5 +#define MLKEM_POLYCOMPRESSEDBYTES_DV MLKEM_POLYCOMPRESSEDBYTES_D5 +#define MLKEM_POLYCOMPRESSEDBYTES_DU MLKEM_POLYCOMPRESSEDBYTES_D11 +#define MLKEM_POLYVECCOMPRESSEDBYTES_DU (MLKEM_K * MLKEM_POLYCOMPRESSEDBYTES_DU) +#endif /* MLKEM_K == 4 */ + +#define MLKEM_ETA2 2 + +#define MLKEM_INDCPA_MSGBYTES (MLKEM_SYMBYTES) +#define MLKEM_INDCPA_PUBLICKEYBYTES (MLKEM_POLYVECBYTES + MLKEM_SYMBYTES) +#define MLKEM_INDCPA_SECRETKEYBYTES (MLKEM_POLYVECBYTES) +#define MLKEM_INDCPA_BYTES \ + (MLKEM_POLYVECCOMPRESSEDBYTES_DU + MLKEM_POLYCOMPRESSEDBYTES_DV) + +#define MLKEM_INDCCA_PUBLICKEYBYTES (MLKEM_INDCPA_PUBLICKEYBYTES) +/* 32 bytes of additional space to save H(pk) */ +#define MLKEM_INDCCA_SECRETKEYBYTES \ + (MLKEM_INDCPA_SECRETKEYBYTES + MLKEM_INDCPA_PUBLICKEYBYTES + \ + 2 * MLKEM_SYMBYTES) +#define MLKEM_INDCCA_CIPHERTEXTBYTES (MLKEM_INDCPA_BYTES) + +#endif /* !MLK_PARAMS_H */ diff --git a/cbits/mlkem/src/poly.c b/cbits/mlkem/src/poly.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/poly.c @@ -0,0 +1,581 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [NeonNTT] + * Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1 + * Becker, Hwang, Kannwischer, Yang, Yang + * https://eprint.iacr.org/2021/986 + * + * - [REF] + * CRYSTALS-Kyber C reference implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/ref + */ + +#include "common.h" +#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + + +#include "cbmc.h" +#include "debug.h" +#include "poly.h" +#include "sampling.h" +#include "symmetric.h" +#include "verify.h" + +/** + * Montgomery multiplication modulo MLKEM_Q. + * + * @reference{`fqmul()` in the reference implementation @[REF].} + * + * @param a First factor. Can be any int16_t. + * @param b Second factor. Must be signed canonical + * (abs value < (MLKEM_Q+1)/2). + * + * @return 16-bit integer congruent to a*b*R^{-1} mod MLKEM_Q, and + * smaller than MLKEM_Q in absolute value. + */ +static MLK_INLINE int16_t mlk_fqmul(int16_t a, int16_t b) +__contract__( + requires(b > -MLKEM_Q_HALF && b < MLKEM_Q_HALF) + ensures(return_value > -MLKEM_Q && return_value < MLKEM_Q) +) +{ + int16_t res; + mlk_assert_abs_bound(&b, 1, MLKEM_Q_HALF); + + res = mlk_montgomery_reduce((int32_t)a * (int32_t)b); + /* Bounds: + * |res| <= ceil(|a| * |b| / 2^16) + (MLKEM_Q + 1) / 2 + * <= ceil(2^15 * ((MLKEM_Q - 1)/2) / 2^16) + (MLKEM_Q + 1) / 2 + * <= ceil((MLKEM_Q - 1) / 4) + (MLKEM_Q + 1) / 2 + * < MLKEM_Q + */ + + mlk_assert_abs_bound(&res, 1, MLKEM_Q); + return res; +} + +/** + * Barrett reduction; given a 16-bit integer a, computes the centered + * representative congruent to a mod MLKEM_Q in [-(MLKEM_Q-1)/2, (MLKEM_Q-1)/2]. + * + * @reference{`barrett_reduce()` in the reference implementation @[REF].} + * + * @param a Input integer to be reduced. + * + * @return Integer in [-(MLKEM_Q-1)/2, (MLKEM_Q-1)/2] congruent to @p a modulo + * MLKEM_Q. + */ +static MLK_INLINE int16_t mlk_barrett_reduce(int16_t a) +__contract__( + ensures(return_value > -MLKEM_Q_HALF && return_value < MLKEM_Q_HALF) +) +{ + /* Barrett reduction approximates + * ``` + * round(a/MLKEM_Q) + * = round(a*(2^N/MLKEM_Q))/2^N) + * ~= round(a*round(2^N/MLKEM_Q)/2^N) + * ``` + * Here, we pick N=26. + */ + const int32_t magic = 20159; /* check-magic: 20159 == round(2^26 / MLKEM_Q) */ + + /* + * PORTABILITY: Right-shift on a signed integer is + * implementation-defined for negative left argument. + * Here, we assume it's sign-preserving "arithmetic" shift right. + * See (C99 6.5.7 (5)) + */ + const int32_t t = (magic * a + ((int32_t)1 << 25)) >> 26; + + /* + * t is in -10 .. +10, so we need 32-bit math to + * evaluate t * MLKEM_Q and the subsequent subtraction + */ + int16_t res = (int16_t)(a - t * MLKEM_Q); + + mlk_assert_abs_bound(&res, 1, MLKEM_Q_HALF); + return res; +} + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) +/* Reference: `poly_tomont()` in the reference implementation @[REF]. */ +MLK_STATIC_TESTABLE void mlk_poly_tomont_c(mlk_poly *r) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_abs_bound(r->coeffs, 0, MLKEM_N, MLKEM_Q)) +) +{ + unsigned i; + const int16_t f = 1353; /* check-magic: 1353 == signed_mod(2^32, MLKEM_Q) */ + for (i = 0; i < MLKEM_N; i++) + __loop__( + invariant(i <= MLKEM_N) + invariant(array_abs_bound(r->coeffs, 0, i, MLKEM_Q)) + decreases(MLKEM_N - i)) + { + r->coeffs[i] = mlk_fqmul(r->coeffs[i], f); + } + + mlk_assert_abs_bound(r, MLKEM_N, MLKEM_Q); +} + +MLK_INTERNAL_API +void mlk_poly_tomont(mlk_poly *r) +{ +#if defined(MLK_USE_NATIVE_POLY_TOMONT) + int ret; + ret = mlk_poly_tomont_native(r->coeffs); + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + mlk_assert_abs_bound(r, MLKEM_N, MLKEM_Q); + return; + } +#endif /* MLK_USE_NATIVE_POLY_TOMONT */ + + mlk_poly_tomont_c(r); +} +#endif /* !MLK_CONFIG_NO_KEYPAIR_API */ + +/** + * Constant-time conversion of signed representatives modulo MLKEM_Q within + * range [-(MLKEM_Q-1), MLKEM_Q-1] into unsigned representatives within + * range [0, MLKEM_Q-1]. + * + * @reference{Not present in the reference implementation @[REF]. Used here + * to implement different semantics of `poly_reduce()`; see below. In the + * reference implementation @[REF] this logic is part of all compression + * functions (see `compress.c`).} + * + * @param c Signed coefficient to be converted. + * + * @return Unsigned representative in [0, MLKEM_Q). + */ +static MLK_INLINE int16_t mlk_scalar_signed_to_unsigned_q(int16_t c) +__contract__( + requires(c > -MLKEM_Q && c < MLKEM_Q) + ensures(return_value >= 0 && return_value < MLKEM_Q) + ensures(return_value == (int32_t)c + (((int32_t)c < 0) * MLKEM_Q))) +{ + mlk_assert_abs_bound(&c, 1, MLKEM_Q); + + /* Add MLKEM_Q if c is negative, but in constant time. + * + * Note that c + MLKEM_Q does not overflow in int16_t, + * so the cast to uint16_t is safe. */ + c = mlk_ct_sel_int16((int16_t)(c + MLKEM_Q), c, mlk_ct_cmask_neg_i16(c)); + + mlk_assert_bound(&c, 1, 0, MLKEM_Q); + return c; +} + +/* Reference: `poly_reduce()` in the reference implementation @[REF] + * - We use _unsigned_ canonical outputs, while the reference + * implementation uses _signed_ canonical outputs. + * Accordingly, we need a conditional addition of MLKEM_Q + * here to go from signed to unsigned representatives. + * This conditional addition is then dropped from all + * polynomial compression functions instead (see `compress.c`). */ +MLK_STATIC_TESTABLE void mlk_poly_reduce_c(mlk_poly *r) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) +) +{ + unsigned i; + + for (i = 0; i < MLKEM_N; i++) + __loop__( + invariant(i <= MLKEM_N) + invariant(array_bound(r->coeffs, 0, i, 0, MLKEM_Q)) + decreases(MLKEM_N - i)) + { + /* Barrett reduction, giving signed canonical representative */ + int16_t t = mlk_barrett_reduce(r->coeffs[i]); + /* Conditional addition to get unsigned canonical representative */ + r->coeffs[i] = mlk_scalar_signed_to_unsigned_q(t); + } + + mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q); +} + +MLK_INTERNAL_API +void mlk_poly_reduce(mlk_poly *r) +{ +#if defined(MLK_USE_NATIVE_POLY_REDUCE) + int ret; + ret = mlk_poly_reduce_native(r->coeffs); + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + mlk_assert_bound(r, MLKEM_N, 0, MLKEM_Q); + return; + } +#endif /* MLK_USE_NATIVE_POLY_REDUCE */ + + mlk_poly_reduce_c(r); +} + +/* Reference: `poly_add()` in the reference implementation @[REF]. + * - We use destructive version (output=first input) to avoid + * reasoning about aliasing in the CBMC specification */ +MLK_INTERNAL_API +void mlk_poly_add(mlk_poly *r, const mlk_poly *b) +{ + unsigned i; + for (i = 0; i < MLKEM_N; i++) + __loop__( + invariant(i <= MLKEM_N) + invariant(forall(k0, i, MLKEM_N, r->coeffs[k0] == loop_entry(*r).coeffs[k0])) + invariant(forall(k1, 0, i, r->coeffs[k1] == loop_entry(*r).coeffs[k1] + b->coeffs[k1])) + decreases(MLKEM_N - i)) + { + /* The preconditions imply that the addition stays within int16_t. */ + r->coeffs[i] = (int16_t)(r->coeffs[i] + b->coeffs[i]); + } +} + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +/* Reference: `poly_sub()` in the reference implementation @[REF]. + * - We use destructive version (output=first input) to avoid + * reasoning about aliasing in the CBMC specification */ +MLK_INTERNAL_API +void mlk_poly_sub(mlk_poly *r, const mlk_poly *b) +{ + unsigned i; + for (i = 0; i < MLKEM_N; i++) + __loop__( + invariant(i <= MLKEM_N) + invariant(forall(k0, i, MLKEM_N, r->coeffs[k0] == loop_entry(*r).coeffs[k0])) + invariant(forall(k1, 0, i, r->coeffs[k1] == loop_entry(*r).coeffs[k1] - b->coeffs[k1])) + decreases(MLKEM_N - i)) + { + /* The preconditions imply that the subtraction stays within int16_t. */ + r->coeffs[i] = (int16_t)(r->coeffs[i] - b->coeffs[i]); + } +} +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +#include "zetas.inc" + +/* Reference: Does not exist in the reference implementation @[REF]. + * - The reference implementation does not use a + * multiplication cache ('mulcache'). This idea originates + * from @[NeonNTT] and is used at the C level here. */ +MLK_STATIC_TESTABLE void mlk_poly_mulcache_compute_c(mlk_poly_mulcache *x, + const mlk_poly *a) +__contract__( + requires(memory_no_alias(x, sizeof(mlk_poly_mulcache))) + requires(memory_no_alias(a, sizeof(mlk_poly))) + assigns(memory_slice(x, sizeof(mlk_poly_mulcache))) +) +{ + unsigned i; + for (i = 0; i < MLKEM_N / 4; i++) + __loop__( + invariant(i <= MLKEM_N / 4) + invariant(array_abs_bound(x->coeffs, 0, 2 * i, MLKEM_Q)) + decreases(MLKEM_N / 4 - i)) + { + x->coeffs[2 * i + 0] = mlk_fqmul(a->coeffs[4 * i + 1], mlk_zetas[64 + i]); + /* The values in zeta table are <= MLKEM_Q in absolute value, + * so the negation in int16_t is safe. */ + x->coeffs[2 * i + 1] = + mlk_fqmul(a->coeffs[4 * i + 3], (int16_t)(-mlk_zetas[64 + i])); + } + + /* + * This bound is true for the C implementation, but not needed + * in the higher level bounds reasoning. It is thus omitted + * from the spec to not unnecessarily constrain native + * implementations, but checked here nonetheless. + */ + mlk_assert_abs_bound(x, MLKEM_N / 2, MLKEM_Q); +} + +MLK_INTERNAL_API +void mlk_poly_mulcache_compute(mlk_poly_mulcache *x, const mlk_poly *a) +{ +#if defined(MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE) + int ret; + ret = mlk_poly_mulcache_compute_native(x->coeffs, a->coeffs); + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + return; + } +#endif /* MLK_USE_NATIVE_POLY_MULCACHE_COMPUTE */ + + mlk_poly_mulcache_compute_c(x, a); +} + +/* + * Computes a block CT butterflies with a fixed twiddle factor, + * using Montgomery multiplication. + * Parameters: + * - r: Pointer to base of polynomial (_not_ the base of butterfly block) + * - root: Twiddle factor to use for the butterfly. This must be in + * Montgomery form and signed canonical. + * - start: Offset to the beginning of the butterfly block + * - len: Index difference between coefficients subject to a butterfly + * - bound: Ghost variable describing coefficient bound: Prior to `start`, + * coefficients must be bound by `bound + MLKEM_Q`. Post `start`, + * they must be bound by `bound`. + * When this function returns, output coefficients in the index range + * [start, start+2*len) have bound bumped to `bound + MLKEM_Q`. + * Example: + * - start=8, len=4 + * This would compute the following four butterflies + * 8 -- 12 + * 9 -- 13 + * 10 -- 14 + * 11 -- 15 + * - start=4, len=2 + * This would compute the following two butterflies + * 4 -- 6 + * 5 -- 7 + */ + +/* Reference: Embedded in `ntt()` in the reference implementation @[REF]. */ +static void mlk_ntt_butterfly_block(int16_t r[MLKEM_N], int16_t zeta, + unsigned start, unsigned len, + unsigned bound) +__contract__( + requires(start < MLKEM_N) + requires(1 <= len && len <= MLKEM_N / 2 && start + 2 * len <= MLKEM_N) + requires(0 <= bound && bound < INT16_MAX - MLKEM_Q) + requires(-MLKEM_Q_HALF < zeta && zeta < MLKEM_Q_HALF) + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(array_abs_bound(r, 0, start, bound + MLKEM_Q)) + requires(array_abs_bound(r, start, MLKEM_N, bound)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(array_abs_bound(r, 0, start + 2*len, bound + MLKEM_Q)) + ensures(array_abs_bound(r, start + 2 * len, MLKEM_N, bound))) +{ + /* `bound` is a ghost variable only needed in the CBMC specification */ + unsigned j; + ((void)bound); + for (j = start; j < start + len; j++) + __loop__( + invariant(start <= j && j <= start + len) + /* + * Coefficients are updated in strided pairs, so the bounds for the + * intermediate states alternate twice between the old and new bound + */ + invariant(array_abs_bound(r, 0, j, bound + MLKEM_Q)) + invariant(array_abs_bound(r, j, start + len, bound)) + invariant(array_abs_bound(r, start + len, j + len, bound + MLKEM_Q)) + invariant(array_abs_bound(r, j + len, MLKEM_N, bound)) + decreases(start + len - j)) + { + int16_t t; + t = mlk_fqmul(r[j + len], zeta); + /* The precondition implies that the arithmetic does not overflow. */ + r[j + len] = (int16_t)(r[j] - t); + r[j] = (int16_t)(r[j] + t); + } +} + +/* + * Compute one layer of forward NTT + * Parameters: + * - r: Pointer to base of polynomial + * - layer: Variable indicating which layer is being applied. + */ + +/* Reference: Embedded in `ntt()` in the reference implementation @[REF]. */ +static void mlk_ntt_layer(int16_t r[MLKEM_N], unsigned layer) +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(1 <= layer && layer <= 7) + requires(array_abs_bound(r, 0, MLKEM_N, layer * MLKEM_Q)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(array_abs_bound(r, 0, MLKEM_N, (layer + 1) * MLKEM_Q))) +{ + unsigned start, k, len; + /* Twiddle factors for layer n are at indices 2^(n-1)..2^n-1. */ + k = 1u << (layer - 1); + len = (unsigned)MLKEM_N >> layer; + for (start = 0; start < MLKEM_N; start += 2 * len) + __loop__( + invariant(start < MLKEM_N + 2 * len) + invariant(k <= MLKEM_N / 2 && 2 * len * k == start + MLKEM_N) + invariant(array_abs_bound(r, 0, start, layer * MLKEM_Q + MLKEM_Q)) + invariant(array_abs_bound(r, start, MLKEM_N, layer * MLKEM_Q)) + decreases(MLKEM_N - start)) + { + int16_t zeta = mlk_zetas[k++]; + mlk_ntt_butterfly_block(r, zeta, start, len, layer * MLKEM_Q); + } +} + +/* + * Compute full forward NTT + * NOTE: This particular implementation satisfies a much tighter + * bound on the output coefficients (5*q) than the contractual one (8*q), + * but this is not needed in the calling code. Should we change the + * base multiplication strategy to require smaller NTT output bounds, + * the proof may need strengthening. + */ + +/* Reference: `ntt()` in the reference implementation @[REF]. + * - Iterate over `layer` instead of `len` in the outer loop + * to simplify computation of zeta index. */ +MLK_STATIC_TESTABLE void mlk_poly_ntt_c(mlk_poly *p) +__contract__( + requires(memory_no_alias(p, sizeof(mlk_poly))) + requires(array_abs_bound(p->coeffs, 0, MLKEM_N, MLKEM_Q)) + assigns(memory_slice(p, sizeof(mlk_poly))) + ensures(array_abs_bound(p->coeffs, 0, MLKEM_N, MLK_NTT_BOUND)) +) +{ + unsigned layer; + int16_t *r; + + mlk_assert_abs_bound(p, MLKEM_N, MLKEM_Q); + + r = p->coeffs; + + for (layer = 1; layer <= 7; layer++) + __loop__( + invariant(1 <= layer && layer <= 8) + invariant(array_abs_bound(r, 0, MLKEM_N, layer * MLKEM_Q)) + decreases(8 - layer)) + { + mlk_ntt_layer(r, layer); + } + + /* Check the stronger bound */ + mlk_assert_abs_bound(p, MLKEM_N, MLK_NTT_BOUND); +} + +MLK_INTERNAL_API +void mlk_poly_ntt(mlk_poly *r) +{ +#if defined(MLK_USE_NATIVE_NTT) + int ret; + mlk_assert_abs_bound(r, MLKEM_N, MLKEM_Q); + ret = mlk_ntt_native(r->coeffs); + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + mlk_assert_abs_bound(r, MLKEM_N, MLK_NTT_BOUND); + return; + } +#endif /* MLK_USE_NATIVE_NTT */ + + mlk_poly_ntt_c(r); +} + + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +/* Compute one layer of inverse NTT */ + +/* Reference: Embedded into `invntt()` in the reference implementation @[REF] */ +static void mlk_invntt_layer(int16_t *r, unsigned layer) +__contract__( + requires(memory_no_alias(r, sizeof(int16_t) * MLKEM_N)) + requires(1 <= layer && layer <= 7) + requires(array_abs_bound(r, 0, MLKEM_N, MLKEM_Q)) + assigns(memory_slice(r, sizeof(int16_t) * MLKEM_N)) + ensures(array_abs_bound(r, 0, MLKEM_N, MLKEM_Q))) +{ + unsigned start, k, len; + len = (unsigned)MLKEM_N >> layer; + k = (1u << layer) - 1; + for (start = 0; start < MLKEM_N; start += 2 * len) + __loop__( + invariant(array_abs_bound(r, 0, MLKEM_N, MLKEM_Q)) + invariant(start <= MLKEM_N && k <= 127) + /* Normalised form of k == MLKEM_N / len - 1 - start / (2 * len) */ + invariant(2 * len * k + start == 2 * MLKEM_N - 2 * len) + decreases(MLKEM_N - start)) + { + unsigned j; + int16_t zeta = mlk_zetas[k--]; + for (j = start; j < start + len; j++) + __loop__( + invariant(start <= j && j <= start + len) + invariant(start <= MLKEM_N && k <= 127) + invariant(array_abs_bound(r, 0, MLKEM_N, MLKEM_Q)) + decreases(start + len - j)) + { + int16_t t = r[j]; + /* The preconditions imply that the arithmetic does not overflow. */ + r[j] = mlk_barrett_reduce((int16_t)(t + r[j + len])); + r[j + len] = (int16_t)(r[j + len] - t); + r[j + len] = mlk_fqmul(r[j + len], zeta); + } + } +} + +/* Reference: `invntt()` in the reference implementation @[REF] + * - We normalize at the beginning of the inverse NTT, + * while the reference implementation normalizes at + * the end. This allows us to drop a call to `poly_reduce()` + * from the base multiplication. */ +MLK_STATIC_TESTABLE void mlk_poly_invntt_tomont_c(mlk_poly *p) +__contract__( + requires(memory_no_alias(p, sizeof(mlk_poly))) + assigns(memory_slice(p, sizeof(mlk_poly))) + ensures(array_abs_bound(p->coeffs, 0, MLKEM_N, MLK_INVNTT_BOUND)) +) +{ + unsigned j, layer; + const int16_t f = 1441; /* check-magic: 1441 == pow(2,32 - 7,MLKEM_Q) */ + int16_t *r = p->coeffs; + + /* + * Scale input polynomial to account for Montgomery factor + * and NTT twist. This also brings coefficients down to + * absolute value < MLKEM_Q. + */ + for (j = 0; j < MLKEM_N; j++) + __loop__( + invariant(j <= MLKEM_N) + invariant(array_abs_bound(r, 0, j, MLKEM_Q)) + decreases(MLKEM_N - j)) + { + r[j] = mlk_fqmul(r[j], f); + } + + /* Run the invNTT layers */ + for (layer = 7; layer > 0; layer--) + __loop__( + invariant(0 <= layer && layer < 8) + invariant(array_abs_bound(r, 0, MLKEM_N, MLKEM_Q)) + decreases(layer)) + { + mlk_invntt_layer(r, layer); + } + + mlk_assert_abs_bound(p, MLKEM_N, MLK_INVNTT_BOUND); +} + +MLK_INTERNAL_API +void mlk_poly_invntt_tomont(mlk_poly *r) +{ +#if defined(MLK_USE_NATIVE_INTT) + int ret; + ret = mlk_intt_native(r->coeffs); + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + mlk_assert_abs_bound(r, MLKEM_N, MLK_INVNTT_BOUND); + return; + } +#endif /* MLK_USE_NATIVE_INTT */ + + mlk_poly_invntt_tomont_c(r); +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ +#else /* !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +MLK_EMPTY_CU(mlk_poly) + +#endif /* MLK_CONFIG_MULTILEVEL_NO_SHARED */ diff --git a/cbits/mlkem/src/poly.h b/cbits/mlkem/src/poly.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/poly.h @@ -0,0 +1,296 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + */ + +#ifndef MLK_POLY_H +#define MLK_POLY_H + + +#include "cbmc.h" +#include "common.h" +#include "debug.h" +#include "verify.h" + +/* Absolute exclusive upper bound for the output of the inverse NTT */ +#define MLK_INVNTT_BOUND (8 * MLKEM_Q) + +/* Absolute exclusive upper bound for the output of the forward NTT */ +#define MLK_NTT_BOUND (8 * MLKEM_Q) + +/** + * Element of R_q = Z_q[X]/(X^n + 1). Represents polynomial + * coeffs[0] + X*coeffs[1] + X^2*coeffs[2] + ... + X^{n-1}*coeffs[n-1]. + */ +typedef struct +{ + int16_t coeffs[MLKEM_N]; /**< Polynomial coefficients. */ +} MLK_ALIGN mlk_poly; + +/** + * INTERNAL representation of precomputed data speeding up + * the base multiplication of two polynomials in NTT domain. + */ +typedef struct +{ + int16_t coeffs[MLKEM_N >> 1]; /**< Cached coefficients. */ +} MLK_ALIGN mlk_poly_mulcache; + +/** + * Generic Montgomery reduction; given a 32-bit integer a, computes a 16-bit + * integer congruent to a * R^-1 mod MLKEM_Q, where R=2^16. + * + * @param a Input integer to be reduced, of absolute value smaller or equal + * to INT32_MAX - 2^15 * MLKEM_Q. + * + * @return Integer congruent to a * R^-1 modulo MLKEM_Q, with absolute value + * <= ceil(|a| / 2^16) + (MLKEM_Q + 1)/2. + */ +static MLK_ALWAYS_INLINE int16_t mlk_montgomery_reduce(int32_t a) +__contract__( + requires(a < +(INT32_MAX - (((int32_t)1 << 15) * MLKEM_Q)) && + a > -(INT32_MAX - (((int32_t)1 << 15) * MLKEM_Q))) + /* We don't attempt to express an input-dependent output bound + * as the post-condition here. There are two call-sites for this + * function: + * - The base multiplication: Here, we need no output bound. + * - mlk_fqmul: Here, we inline this function and prove another spec + * for mlk_fqmul which does have a post-condition bound. */ +) +{ + /* check-magic: 62209 == unsigned_mod(pow(MLKEM_Q, -1, 2^16), 2^16) */ + const uint32_t QINV = 62209; + + /* Compute a*q^{-1} mod 2^16 in unsigned representatives. */ + const uint16_t a_reduced = mlk_cast_int32_to_uint16(a); + const uint16_t a_inverted = (a_reduced * QINV) & UINT16_MAX; + + /* Lift to signed canonical representative mod 2^16. */ + const int16_t t = mlk_cast_uint16_to_int16(a_inverted); + + int32_t r; + + mlk_assert(a < +(INT32_MAX - (((int32_t)1 << 15) * MLKEM_Q)) && + a > -(INT32_MAX - (((int32_t)1 << 15) * MLKEM_Q))); + + r = a - ((int32_t)t * MLKEM_Q); + + /* + * PORTABILITY: Right-shift on a signed integer is, strictly-speaking, + * implementation-defined for negative left argument. Here, + * we assume it's sign-preserving "arithmetic" shift right. (C99 6.5.7 (5)) + */ + r = r >> 16; + /* Bounds: |r >> 16| <= ceil(|r| / 2^16) + * <= ceil(|a| / 2^16 + MLKEM_Q / 2) + * <= ceil(|a| / 2^16) + (MLKEM_Q + 1) / 2 + * + * (Note that |a >> n| = ceil(|a| / 2^16) for negative a) + */ + return (int16_t)r; +} + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) +#define mlk_poly_tomont MLK_NAMESPACE(poly_tomont) +/** + * In-place conversion of all coefficients of a polynomial from the normal + * domain to the Montgomery domain. + * + * Bounds: output < MLKEM_Q in absolute value. + * + * @spec{Internal normalization required in `mlk_indcpa_keypair_derand` as + * part of matrix-vector multiplication @[FIPS203, Algorithm 13, K-PKE.KeyGen, + * L18].} + * + * @param[in,out] r Input/output polynomial. + */ +MLK_INTERNAL_API +void mlk_poly_tomont(mlk_poly *r) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_abs_bound(r->coeffs, 0, MLKEM_N, MLKEM_Q)) +); +#endif /* !MLK_CONFIG_NO_KEYPAIR_API */ + +#define mlk_poly_mulcache_compute MLK_NAMESPACE(poly_mulcache_compute) +/** + * Compute the mulcache for a polynomial in NTT domain. + * + * The mulcache of a degree-2 polynomial b := b0 + b1*X in Fq[X]/(X^2-zeta) + * is the value b1*zeta, needed when computing products of b in + * Fq[X]/(X^2-zeta). + * + * The mulcache of a polynomial in NTT domain -- which is a 128-tuple of + * degree-2 polynomials in Fq[X]/(X^2-zeta), for varying zeta, is the + * 128-tuple of mulcaches of those polynomials. + * + * @spec{Caches `b_1 * \gamma` in @[FIPS203, Algorithm 12, BaseCaseMultiply, + * L1].} + * + * @param[out] x Mulcache to be populated. + * @param[in] a Input polynomial. + */ +/* + * NOTE: The default C implementation of this function populates + * the mulcache with values in (-q,q), but this is not needed for the + * higher level safety proofs, and thus not part of the spec. + */ +MLK_INTERNAL_API +void mlk_poly_mulcache_compute(mlk_poly_mulcache *x, const mlk_poly *a) +__contract__( + requires(memory_no_alias(x, sizeof(mlk_poly_mulcache))) + requires(memory_no_alias(a, sizeof(mlk_poly))) + assigns(memory_slice(x, sizeof(mlk_poly_mulcache))) +); + +#define mlk_poly_reduce MLK_NAMESPACE(poly_reduce) +/** + * Convert a polynomial to unsigned canonical representatives. + * + * The input coefficients can be arbitrary integers in int16_t. The output + * coefficients are in [0,1,..,MLKEM_Q-1]. + * + * @spec{Normalizes on unsigned canonical representatives ahead of calling + * @[FIPS203, Compress_d, Eq (4.7)]. This is not made explicit in FIPS 203.} + * + * @param[in,out] r Input/output polynomial. + */ +/* + * NOTE: The semantics of mlk_poly_reduce() is different in + * the reference implementation, which requires + * signed canonical output data. Unsigned canonical + * outputs are better suited to the only remaining + * use of mlk_poly_reduce() in the context of (de)serialization. + */ +MLK_INTERNAL_API +void mlk_poly_reduce(mlk_poly *r) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) +); + +#define mlk_poly_add MLK_NAMESPACE(poly_add) +/** + * Add two polynomials in place. + * + * The coefficients of @p r and @p b must be such that the addition does not + * overflow. Otherwise, the behaviour of this function is undefined. + * + * @spec{@[FIPS203, 2.4.5, Arithmetic With Polynomials and NTT + * Representations]. Used in @[FIPS203, Algorithm 14 (K-PKE.Encrypt), L21].} + * + * @param[in,out] r Input-output polynomial to be added to. + * @param[in] b Input polynomial that should be added to @p r. Must be + * disjoint from @p r. + */ +/* + * NOTE: The reference implementation uses a 3-argument mlk_poly_add. + * We specialize to the accumulator form to avoid reasoning about aliasing. + */ +MLK_INTERNAL_API +void mlk_poly_add(mlk_poly *r, const mlk_poly *b) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(memory_no_alias(b, sizeof(mlk_poly))) + requires(forall(k0, 0, MLKEM_N, (int32_t) r->coeffs[k0] + b->coeffs[k0] <= INT16_MAX)) + requires(forall(k1, 0, MLKEM_N, (int32_t) r->coeffs[k1] + b->coeffs[k1] >= INT16_MIN)) + ensures(forall(k, 0, MLKEM_N, r->coeffs[k] == old(*r).coeffs[k] + b->coeffs[k])) + assigns(memory_slice(r, sizeof(mlk_poly))) +); + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_poly_sub MLK_NAMESPACE(poly_sub) +/** + * Subtract two polynomials; no modular reduction is performed. + * + * @spec{@[FIPS203, 2.4.5, Arithmetic With Polynomials and NTT + * Representations]. Used in @[FIPS203, Algorithm 15, K-PKE.Decrypt, L6].} + * + * @param[in,out] r Input-output polynomial to be subtracted from. + * @param[in] b Second input polynomial. + */ +/* + * NOTE: The reference implementation uses a 3-argument mlk_poly_sub. + * We specialize to the accumulator form to avoid reasoning about aliasing. + */ +MLK_INTERNAL_API +void mlk_poly_sub(mlk_poly *r, const mlk_poly *b) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(memory_no_alias(b, sizeof(mlk_poly))) + requires(forall(k0, 0, MLKEM_N, (int32_t) r->coeffs[k0] - b->coeffs[k0] <= INT16_MAX)) + requires(forall(k1, 0, MLKEM_N, (int32_t) r->coeffs[k1] - b->coeffs[k1] >= INT16_MIN)) + ensures(forall(k, 0, MLKEM_N, r->coeffs[k] == old(*r).coeffs[k] - b->coeffs[k])) + assigns(memory_slice(r, sizeof(mlk_poly))) +); +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +#define mlk_poly_ntt MLK_NAMESPACE(poly_ntt) +/** + * Compute the negacyclic number-theoretic transform (NTT) of a polynomial + * in place. + * + * The input is assumed to be in normal order and coefficient-wise bound by + * MLKEM_Q in absolute value. + * + * The output polynomial is in bitreversed order, or of a custom order if + * MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set, and coefficient-wise bound + * by MLK_NTT_BOUND in absolute value. + * + * (NOTE: Sometimes the input to the NTT is actually smaller, which gives + * better bounds.) + * + * @spec{Implements @[FIPS203, Algorithm 9, NTT].} + * + * @param[in,out] r Input/output polynomial. + */ +MLK_INTERNAL_API +void mlk_poly_ntt(mlk_poly *r) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(array_abs_bound(r->coeffs, 0, MLKEM_N, MLKEM_Q)) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_abs_bound(r->coeffs, 0, MLKEM_N, MLK_NTT_BOUND)) +); + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_poly_invntt_tomont MLK_NAMESPACE(poly_invntt_tomont) +/** + * Compute the inverse negacyclic number-theoretic transform (NTT) of a + * polynomial in place; input assumed to be in bitreversed order, output in + * normal order. + * + * The input is assumed to be in bitreversed order, or of a custom order if + * MLK_USE_NATIVE_NTT_CUSTOM_ORDER is set, and can have arbitrary + * coefficients in int16_t. + * + * The output polynomial is in normal order, and coefficient-wise bound by + * MLK_INVNTT_BOUND in absolute value. + * + * @spec{Implements composition of @[FIPS203, Algorithm 10, NTT^{-1}] and + * elementwise modular multiplication with a suitable Montgomery factor + * introduced during the base multiplication.} + * + * @param[in,out] r Input/output polynomial. + */ +MLK_INTERNAL_API +void mlk_poly_invntt_tomont(mlk_poly *r) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_abs_bound(r->coeffs, 0, MLKEM_N, MLK_INVNTT_BOUND)) +); +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +#endif /* !MLK_POLY_H */ diff --git a/cbits/mlkem/src/poly_k.c b/cbits/mlkem/src/poly_k.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/poly_k.c @@ -0,0 +1,522 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + * + * - [NeonNTT] + * Neon NTT: Faster Dilithium, Kyber, and Saber on Cortex-A72 and Apple M1 + * Becker, Hwang, Kannwischer, Yang, Yang + * https://eprint.iacr.org/2021/986 + * + * - [REF] + * CRYSTALS-Kyber C reference implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/ref + */ + +#include "poly_k.h" + +#include "debug.h" +#include "sampling.h" +#include "symmetric.h" +#include "verify.h" + +/* Parameter set namespacing + * This is to facilitate building multiple instances + * of mlkem-native (e.g. with varying parameter sets) + * within a single compilation unit. */ +#define mlk_poly_cbd_eta1 MLK_ADD_PARAM_SET(mlk_poly_cbd_eta1) +#define mlk_poly_cbd_eta2 MLK_ADD_PARAM_SET(mlk_poly_cbd_eta2) +#define mlk_polyvec_basemul_acc_montgomery_cached_c \ + MLK_ADD_PARAM_SET(mlk_polyvec_basemul_acc_montgomery_cached_c) +/* End of parameter set namespacing */ + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +/* Reference: `polyvec_compress()` in the reference implementation @[REF] + * - In contrast to the reference implementation, we assume + * unsigned canonical coefficients here. + * The reference implementation works with coefficients + * in the range [-(MLKEM_Q-1), MLKEM_Q-1]. */ +MLK_INTERNAL_API +void mlk_polyvec_compress_du(uint8_t r[MLKEM_POLYVECCOMPRESSEDBYTES_DU], + const mlk_polyvec *a) +{ + unsigned i; + mlk_assert_bound_2d(a->vec, MLKEM_K, MLKEM_N, 0, MLKEM_Q); + + for (i = 0; i < MLKEM_K; i++) + { + mlk_poly_compress_du(r + i * MLKEM_POLYCOMPRESSEDBYTES_DU, &a->vec[i]); + } +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +/* Reference: `polyvec_decompress()` in the reference implementation @[REF]. */ +MLK_INTERNAL_API +void mlk_polyvec_decompress_du(mlk_polyvec *r, + const uint8_t a[MLKEM_POLYVECCOMPRESSEDBYTES_DU]) +{ + unsigned i; + for (i = 0; i < MLKEM_K; i++) + { + mlk_poly_decompress_du(&r->vec[i], a + i * MLKEM_POLYCOMPRESSEDBYTES_DU); + } + + mlk_assert_bound_2d(r->vec, MLKEM_K, MLKEM_N, 0, MLKEM_Q); +} +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) || !defined(MLK_CONFIG_NO_ENCAPS_API) +/* Reference: `polyvec_tobytes()` in the reference implementation @[REF]. + * - In contrast to the reference implementation, we assume + * unsigned canonical coefficients here. + * The reference implementation works with coefficients + * in the range [-(MLKEM_Q-1), MLKEM_Q-1]. */ +MLK_INTERNAL_API +void mlk_polyvec_tobytes(uint8_t r[MLKEM_POLYVECBYTES], const mlk_polyvec *a) +{ + unsigned i; + mlk_assert_bound_2d(a->vec, MLKEM_K, MLKEM_N, 0, MLKEM_Q); + + for (i = 0; i < MLKEM_K; i++) + __loop__( + assigns(i, memory_slice(r, MLKEM_POLYVECBYTES)) + invariant(i <= MLKEM_K) + decreases(MLKEM_K - i) + ) + { + mlk_poly_tobytes(&r[i * MLKEM_POLYBYTES], &a->vec[i]); + } +} +#endif /* !MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API */ + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +/* Reference: `polyvec_frombytes()` in the reference implementation @[REF]. */ +MLK_INTERNAL_API +void mlk_polyvec_frombytes(mlk_polyvec *r, const uint8_t a[MLKEM_POLYVECBYTES]) +{ + unsigned i; + for (i = 0; i < MLKEM_K; i++) + { + mlk_poly_frombytes(&r->vec[i], a + i * MLKEM_POLYBYTES); + } + + mlk_assert_bound_2d(r->vec, MLKEM_K, MLKEM_N, 0, MLKEM_UINT12_LIMIT); +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +/* Reference: `polyvec_ntt()` in the reference implementation @[REF]. */ +MLK_INTERNAL_API +void mlk_polyvec_ntt(mlk_polyvec *r) +{ + unsigned i; + for (i = 0; i < MLKEM_K; i++) + { + mlk_poly_ntt(&r->vec[i]); + } + + mlk_assert_abs_bound_2d(r->vec, MLKEM_K, MLKEM_N, MLK_NTT_BOUND); +} + +/* Reference: `polyvec_invntt_tomont()` in the reference implementation @[REF]. + * - We normalize at the beginning of the inverse NTT, + * while the reference implementation normalizes at + * the end. This allows us to drop a call to `poly_reduce()` + * from the base multiplication. */ +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +MLK_INTERNAL_API +void mlk_polyvec_invntt_tomont(mlk_polyvec *r) +{ + unsigned i; + for (i = 0; i < MLKEM_K; i++) + { + mlk_poly_invntt_tomont(&r->vec[i]); + } + + mlk_assert_abs_bound_2d(r->vec, MLKEM_K, MLKEM_N, MLK_INVNTT_BOUND); +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +/* Reference: `polyvec_basemul_acc_montgomery()` in the + * reference implementation @[REF]. + * - We use a multiplication cache ('mulcache') here + * which is not present in the reference implementation @[REF]. + * This idea originates from @[NeonNTT] and is used + * at the C level here. + * - We compute the coefficients of the scalar product in 32-bit + * coefficients and perform only a single modular reduction + * at the end. The reference implementation uses 2 * MLKEM_K + * more modular reductions since it reduces after every modular + * multiplication. */ +MLK_STATIC_TESTABLE void mlk_polyvec_basemul_acc_montgomery_cached_c( + mlk_poly *r, const mlk_polyvec *a, const mlk_polyvec *b, + const mlk_polyvec_mulcache *b_cache) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(memory_no_alias(a, sizeof(mlk_polyvec))) + requires(memory_no_alias(b, sizeof(mlk_polyvec))) + requires(memory_no_alias(b_cache, sizeof(mlk_polyvec_mulcache))) + requires(forall(k1, 0, MLKEM_K, + array_bound(a->vec[k1].coeffs, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT))) + assigns(memory_slice(r, sizeof(mlk_poly))) +) +{ + unsigned i; + mlk_assert_bound_2d(a->vec, MLKEM_K, MLKEM_N, 0, MLKEM_UINT12_LIMIT); + + for (i = 0; i < MLKEM_N / 2; i++) + __loop__(invariant(i <= MLKEM_N / 2) + decreases(MLKEM_N / 2 - i)) + { + unsigned k; + int32_t t[2] = {0}; + for (k = 0; k < MLKEM_K; k++) + __loop__( + invariant(k <= MLKEM_K && + t[0] <= (int32_t) k * 2 * MLKEM_UINT12_LIMIT * 32768 && + t[0] >= - ((int32_t) k * 2 * MLKEM_UINT12_LIMIT * 32768) && + t[1] <= ((int32_t) k * 2 * MLKEM_UINT12_LIMIT * 32768) && + t[1] >= - ((int32_t) k * 2 * MLKEM_UINT12_LIMIT * 32768)) + decreases(MLKEM_K - k)) + { + t[0] += (int32_t)a->vec[k].coeffs[2 * i + 1] * b_cache->vec[k].coeffs[i]; + t[0] += (int32_t)a->vec[k].coeffs[2 * i] * b->vec[k].coeffs[2 * i]; + t[1] += (int32_t)a->vec[k].coeffs[2 * i] * b->vec[k].coeffs[2 * i + 1]; + t[1] += (int32_t)a->vec[k].coeffs[2 * i + 1] * b->vec[k].coeffs[2 * i]; + } + r->coeffs[2 * i + 0] = mlk_montgomery_reduce(t[0]); + r->coeffs[2 * i + 1] = mlk_montgomery_reduce(t[1]); + } +} + +MLK_INTERNAL_API +void mlk_polyvec_basemul_acc_montgomery_cached( + mlk_poly *r, const mlk_polyvec *a, const mlk_polyvec *b, + const mlk_polyvec_mulcache *b_cache) +{ +#if defined(MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED) + { + int ret; + mlk_assert_bound_2d(a->vec, MLKEM_K, MLKEM_N, 0, MLKEM_UINT12_LIMIT); +#if MLKEM_K == 2 + ret = mlk_polyvec_basemul_acc_montgomery_cached_k2_native( + r->coeffs, (const int16_t *)a, (const int16_t *)b, + (const int16_t *)b_cache); +#elif MLKEM_K == 3 + ret = mlk_polyvec_basemul_acc_montgomery_cached_k3_native( + r->coeffs, (const int16_t *)a, (const int16_t *)b, + (const int16_t *)b_cache); +#elif MLKEM_K == 4 + ret = mlk_polyvec_basemul_acc_montgomery_cached_k4_native( + r->coeffs, (const int16_t *)a, (const int16_t *)b, + (const int16_t *)b_cache); +#endif + if (ret == MLK_NATIVE_FUNC_SUCCESS) + { + return; + } + } +#endif /* MLK_USE_NATIVE_POLYVEC_BASEMUL_ACC_MONTGOMERY_CACHED */ + + mlk_polyvec_basemul_acc_montgomery_cached_c(r, a, b, b_cache); +} + +/* Reference: Does not exist in the reference implementation @[REF]. + * - The reference implementation does not use a + * multiplication cache ('mulcache'). This idea originates + * from @[NeonNTT] and is used at the C level here. */ +MLK_INTERNAL_API +void mlk_polyvec_mulcache_compute(mlk_polyvec_mulcache *x, const mlk_polyvec *a) +{ + unsigned i; + for (i = 0; i < MLKEM_K; i++) + { + mlk_poly_mulcache_compute(&x->vec[i], &a->vec[i]); + } +} + +/* Reference: `polyvec_reduce()` in the reference implementation @[REF]. + * - We use _unsigned_ canonical outputs, while the reference + * implementation uses _signed_ canonical outputs. + * Accordingly, we need a conditional addition of MLKEM_Q + * here to go from signed to unsigned representatives. + * This conditional addition is then dropped from all + * polynomial compression functions instead (see `compress.c`). */ +MLK_INTERNAL_API +void mlk_polyvec_reduce(mlk_polyvec *r) +{ + unsigned i; + for (i = 0; i < MLKEM_K; i++) + { + mlk_poly_reduce(&r->vec[i]); + } + + mlk_assert_bound_2d(r->vec, MLKEM_K, MLKEM_N, 0, MLKEM_Q); +} + +/* Reference: `polyvec_add()` in the reference implementation @[REF]. + * - We use destructive version (output=first input) to avoid + * reasoning about aliasing in the CBMC specification */ +MLK_INTERNAL_API +void mlk_polyvec_add(mlk_polyvec *r, const mlk_polyvec *b) +{ + unsigned i; + for (i = 0; i < MLKEM_K; i++) + __loop__( + assigns(i, memory_slice(r, sizeof(mlk_polyvec))) + invariant(i <= MLKEM_K) + invariant(forall(j0, i, MLKEM_K, + forall(k0, 0, MLKEM_N, + ((int32_t)r->vec[j0].coeffs[k0] + b->vec[j0].coeffs[k0] <= INT16_MAX) && + ((int32_t)r->vec[j0].coeffs[k0] + b->vec[j0].coeffs[k0] >= INT16_MIN)))) + invariant(forall(j2, 0, i, + forall(k2, 0, MLKEM_N, + (r->vec[j2].coeffs[k2] <= INT16_MAX) && + (r->vec[j2].coeffs[k2] >= INT16_MIN)))) + decreases(MLKEM_K - i) + ) + { + mlk_poly_add(&r->vec[i], &b->vec[i]); + } +} + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) +/* Reference: `polyvec_tomont()` in the reference implementation @[REF]. */ +MLK_INTERNAL_API +void mlk_polyvec_tomont(mlk_polyvec *r) +{ + unsigned i; + for (i = 0; i < MLKEM_K; i++) + { + mlk_poly_tomont(&r->vec[i]); + } + + mlk_assert_abs_bound_2d(r->vec, MLKEM_K, MLKEM_N, MLKEM_Q); +} +#endif /* !MLK_CONFIG_NO_KEYPAIR_API */ + + +/** + * Given an array of uniformly random bytes, compute a polynomial with + * coefficients distributed according to a centered binomial distribution + * with parameter MLKEM_ETA1. + * + * @spec{Implements @[FIPS203, Algorithm 8, SamplePolyCBD_eta1], where eta1 + * is specified per parameter set in @[FIPS203, Table 2] and represented as + * MLKEM_ETA1 here.} + * + * @reference{`poly_cbd_eta1` in the reference implementation @[REF].} + * + * @param[out] r Output polynomial. + * @param[in] buf Input byte array. + */ +static MLK_INLINE void mlk_poly_cbd_eta1( + mlk_poly *r, const uint8_t buf[MLKEM_ETA1 * MLKEM_N / 4]) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(memory_no_alias(buf, MLKEM_ETA1 * MLKEM_N / 4)) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_abs_bound(r->coeffs, 0, MLKEM_N, MLKEM_ETA1 + 1)) +) +{ +#if MLKEM_ETA1 == 2 + mlk_poly_cbd2(r, buf); +#elif MLKEM_ETA1 == 3 + mlk_poly_cbd3(r, buf); +#else +#error "Invalid value of MLKEM_ETA1" +#endif +} + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) || \ + (MLKEM_ETA1 == MLKEM_ETA2 && (!defined(MLK_CONFIG_NO_ENCAPS_API) || \ + !defined(MLK_CONFIG_NO_DECAPS_API))) +/* Reference: Does not exist in the reference implementation @[REF]. + * - This implements a x4-batched version of `poly_getnoise_eta1()` + * from the reference implementation, to leverage + * batched Keccak-f1600.*/ +MLK_INTERNAL_API +void mlk_poly_getnoise_eta1_4x(mlk_poly *r0, mlk_poly *r1, mlk_poly *r2, + mlk_poly *r3, const uint8_t seed[MLKEM_SYMBYTES], + uint8_t nonce0, uint8_t nonce1, uint8_t nonce2, + uint8_t nonce3) +{ + MLK_ALIGN uint8_t buf[4][MLK_ALIGN_UP(MLKEM_ETA1 * MLKEM_N / 4)]; + MLK_ALIGN uint8_t extkey[4][MLK_ALIGN_UP(MLKEM_SYMBYTES + 1)]; + mlk_memcpy(extkey[0], seed, MLKEM_SYMBYTES); + mlk_memcpy(extkey[1], seed, MLKEM_SYMBYTES); + mlk_memcpy(extkey[2], seed, MLKEM_SYMBYTES); + mlk_memcpy(extkey[3], seed, MLKEM_SYMBYTES); + extkey[0][MLKEM_SYMBYTES] = nonce0; + extkey[1][MLKEM_SYMBYTES] = nonce1; + extkey[2][MLKEM_SYMBYTES] = nonce2; + extkey[3][MLKEM_SYMBYTES] = nonce3; + +#if !defined(FIPS202_X4_DEFAULT_IMPLEMENTATION) && \ + !defined(MLK_CONFIG_SERIAL_FIPS202_ONLY) + mlk_prf_eta1_x4(buf, extkey); +#else + mlk_prf_eta1(buf[0], extkey[0]); + mlk_prf_eta1(buf[1], extkey[1]); + mlk_prf_eta1(buf[2], extkey[2]); + if (r3 != NULL) + { + mlk_prf_eta1(buf[3], extkey[3]); + } +#endif /* !(!FIPS202_X4_DEFAULT_IMPLEMENTATION && \ + !MLK_CONFIG_SERIAL_FIPS202_ONLY) */ + + mlk_poly_cbd_eta1(r0, buf[0]); + mlk_poly_cbd_eta1(r1, buf[1]); + mlk_poly_cbd_eta1(r2, buf[2]); + if (r3 != NULL) + { + mlk_poly_cbd_eta1(r3, buf[3]); + mlk_assert_abs_bound(r3, MLKEM_N, MLKEM_ETA1 + 1); + } + + mlk_assert_abs_bound(r0, MLKEM_N, MLKEM_ETA1 + 1); + mlk_assert_abs_bound(r1, MLKEM_N, MLKEM_ETA1 + 1); + mlk_assert_abs_bound(r2, MLKEM_N, MLKEM_ETA1 + 1); + + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + mlk_zeroize(buf, sizeof(buf)); + mlk_zeroize(extkey, sizeof(extkey)); +} +#endif /* !MLK_CONFIG_NO_KEYPAIR_API || (MLKEM_ETA1 == MLKEM_ETA2 && \ + (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API)) */ + +#if (MLKEM_K == 2 || MLKEM_K == 4) && \ + (!defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)) +/** + * Given an array of uniformly random bytes, compute a polynomial with + * coefficients distributed according to a centered binomial distribution + * with parameter MLKEM_ETA2. + * + * @spec{Implements @[FIPS203, Algorithm 8, SamplePolyCBD_eta2], where eta2 + * is specified per parameter set in @[FIPS203, Table 2] and represented as + * MLKEM_ETA2 here.} + * + * @reference{`poly_cbd_eta2` in the reference implementation @[REF].} + * + * @param[out] r Output polynomial. + * @param[in] buf Input byte array. + */ +static MLK_INLINE void mlk_poly_cbd_eta2( + mlk_poly *r, const uint8_t buf[MLKEM_ETA2 * MLKEM_N / 4]) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(memory_no_alias(buf, MLKEM_ETA2 * MLKEM_N / 4)) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_abs_bound(r->coeffs, 0, MLKEM_N, MLKEM_ETA2 + 1))) +{ +#if MLKEM_ETA2 == 2 + mlk_poly_cbd2(r, buf); +#else +#error "Invalid value of MLKEM_ETA2" +#endif +} + +/* Reference: `poly_getnoise_eta2()` in the reference implementation @[REF]. + * - We include buffer zeroization. */ +MLK_INTERNAL_API +void mlk_poly_getnoise_eta2(mlk_poly *r, const uint8_t seed[MLKEM_SYMBYTES], + uint8_t nonce) +{ + MLK_ALIGN uint8_t buf[MLKEM_ETA2 * MLKEM_N / 4]; + MLK_ALIGN uint8_t extkey[MLKEM_SYMBYTES + 1]; + + mlk_memcpy(extkey, seed, MLKEM_SYMBYTES); + extkey[MLKEM_SYMBYTES] = nonce; + mlk_prf_eta2(buf, extkey); + + mlk_poly_cbd_eta2(r, buf); + + mlk_assert_abs_bound(r, MLKEM_N, MLKEM_ETA2 + 1); + + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + mlk_zeroize(buf, sizeof(buf)); + mlk_zeroize(extkey, sizeof(extkey)); +} +#endif /* (MLKEM_K == 2 || MLKEM_K == 4) && (!MLK_CONFIG_NO_ENCAPS_API || \ + !MLK_CONFIG_NO_DECAPS_API) */ + +#if MLKEM_K == 2 && \ + (!defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)) +/* Reference: Does not exist in the reference implementation @[REF]. + * - This implements a x4-batched version of `poly_getnoise_eta1()` + * and `poly_getnoise_eta2()` from the reference implementation, + * leveraging batched Keccak-f1600. + * - If a x4-batched Keccak-f1600 is available, we squeeze + * more random data than needed for the eta2 calls, to be + * be able to use a x4-batched Keccak-f1600. */ +MLK_INTERNAL_API +void mlk_poly_getnoise_eta1122_4x(mlk_poly *r0, mlk_poly *r1, mlk_poly *r2, + mlk_poly *r3, + const uint8_t seed[MLKEM_SYMBYTES], + uint8_t nonce0, uint8_t nonce1, + uint8_t nonce2, uint8_t nonce3) +{ +#if MLKEM_ETA2 >= MLKEM_ETA1 +#error mlk_poly_getnoise_eta1122_4x assumes MLKEM_ETA1 > MLKEM_ETA2 +#endif + MLK_ALIGN uint8_t buf[4][MLK_ALIGN_UP(MLKEM_ETA1 * MLKEM_N / 4)]; + MLK_ALIGN uint8_t extkey[4][MLK_ALIGN_UP(MLKEM_SYMBYTES + 1)]; + + mlk_memcpy(extkey[0], seed, MLKEM_SYMBYTES); + mlk_memcpy(extkey[1], seed, MLKEM_SYMBYTES); + mlk_memcpy(extkey[2], seed, MLKEM_SYMBYTES); + mlk_memcpy(extkey[3], seed, MLKEM_SYMBYTES); + extkey[0][MLKEM_SYMBYTES] = nonce0; + extkey[1][MLKEM_SYMBYTES] = nonce1; + extkey[2][MLKEM_SYMBYTES] = nonce2; + extkey[3][MLKEM_SYMBYTES] = nonce3; + + /* On systems with fast batched Keccak, we use 4-fold batched PRF, + * even though that means generating more random data in buf[2] and buf[3] + * than necessary. */ +#if !defined(FIPS202_X4_DEFAULT_IMPLEMENTATION) && \ + !defined(MLK_CONFIG_SERIAL_FIPS202_ONLY) + mlk_prf_eta1_x4(buf, extkey); +#else + mlk_prf_eta1(buf[0], extkey[0]); + mlk_prf_eta1(buf[1], extkey[1]); + mlk_prf_eta2(buf[2], extkey[2]); + mlk_prf_eta2(buf[3], extkey[3]); +#endif /* !(!FIPS202_X4_DEFAULT_IMPLEMENTATION && \ + !MLK_CONFIG_SERIAL_FIPS202_ONLY) */ + + mlk_poly_cbd_eta1(r0, buf[0]); + mlk_poly_cbd_eta1(r1, buf[1]); + mlk_poly_cbd_eta2(r2, buf[2]); + mlk_poly_cbd_eta2(r3, buf[3]); + + mlk_assert_abs_bound(r0, MLKEM_N, MLKEM_ETA1 + 1); + mlk_assert_abs_bound(r1, MLKEM_N, MLKEM_ETA1 + 1); + mlk_assert_abs_bound(r2, MLKEM_N, MLKEM_ETA2 + 1); + mlk_assert_abs_bound(r3, MLKEM_N, MLKEM_ETA2 + 1); + + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + mlk_zeroize(buf, sizeof(buf)); + mlk_zeroize(extkey, sizeof(extkey)); +} +#endif /* MLKEM_K == 2 && (!MLK_CONFIG_NO_ENCAPS_API || \ + !MLK_CONFIG_NO_DECAPS_API) */ + +/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. + * Don't modify by hand -- this is auto-generated by scripts/autogen. */ +#undef mlk_poly_cbd_eta1 +#undef mlk_poly_cbd_eta2 +#undef mlk_polyvec_basemul_acc_montgomery_cached_c diff --git a/cbits/mlkem/src/poly_k.h b/cbits/mlkem/src/poly_k.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/poly_k.h @@ -0,0 +1,622 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + */ + +#ifndef MLK_POLY_K_H +#define MLK_POLY_K_H + +#include "common.h" +#include "compress.h" +#include "poly.h" + +/* Parameter set namespacing + * This is to facilitate building multiple instances + * of mlkem-native (e.g. with varying parameter sets) + * within a single compilation unit. */ +#define mlk_polyvec MLK_ADD_PARAM_SET(mlk_polyvec) +#define mlk_polymat MLK_ADD_PARAM_SET(mlk_polymat) +#define mlk_polyvec_mulcache MLK_ADD_PARAM_SET(mlk_polyvec_mulcache) +/* End of parameter set namespacing */ + +/** Vector of MLKEM_K polynomials. */ +typedef struct +{ + mlk_poly vec[MLKEM_K]; /**< Component polynomials. */ +} MLK_ALIGN mlk_polyvec; + +/** MLKEM_K x MLKEM_K matrix of polynomials. */ +typedef struct +{ + mlk_polyvec vec[MLKEM_K]; /**< Rows of the matrix. */ +} MLK_ALIGN mlk_polymat; + +/** Vector of MLKEM_K mlk_poly_mulcache entries. */ +typedef struct +{ + mlk_poly_mulcache vec[MLKEM_K]; /**< Per-component caches. */ +} MLK_ALIGN mlk_polyvec_mulcache; + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_poly_compress_du MLK_NAMESPACE_K(poly_compress_du) +/** + * Compression (du bits) and subsequent serialization of a polynomial. + * + * @spec{Implements `ByteEncode_{d_u} (Compress_{d_u} (u))` in @[FIPS203, + * Algorithm 14 (K-PKE.Encrypt), L22], with level-specific d_u defined in + * @[FIPS203, Table 2], and given by MLKEM_DU here.} + * + * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_DU + * bytes). + * @param[in] a Input polynomial. Coefficients must be unsigned canonical, + * i.e. in [0,1,..,MLKEM_Q-1]. + */ +static MLK_INLINE void mlk_poly_compress_du( + uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_DU], const mlk_poly *a) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_DU)) + requires(memory_no_alias(a, sizeof(mlk_poly))) + requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_DU))) +{ +#if MLKEM_DU == 10 + mlk_poly_compress_d10(r, a); +#elif MLKEM_DU == 11 + mlk_poly_compress_d11(r, a); +#else +#error "Invalid value of MLKEM_DU" +#endif +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_poly_decompress_du MLK_NAMESPACE_K(poly_decompress_du) +/** + * De-serialization and subsequent decompression (du bits) of a polynomial; + * approximate inverse of mlk_poly_compress_du. + * + * Upon return, the coefficients of the output polynomial are + * unsigned-canonical (non-negative and smaller than MLKEM_Q). + * + * @spec{Implements `Decompress_{d_u} (ByteDecode_{d_u} (u))` in @[FIPS203, + * Algorithm 15 (K-PKE.Decrypt), L3], with level-specific d_u defined in + * @[FIPS203, Table 2], and given by MLKEM_DU here.} + * + * @param[out] r Output polynomial. + * @param[in] a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_DU + * bytes). + */ +static MLK_INLINE void mlk_poly_decompress_du( + mlk_poly *r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_DU]) +__contract__( + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_DU)) + requires(memory_no_alias(r, sizeof(mlk_poly))) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q))) +{ +#if MLKEM_DU == 10 + mlk_poly_decompress_d10(r, a); +#elif MLKEM_DU == 11 + mlk_poly_decompress_d11(r, a); +#else +#error "Invalid value of MLKEM_DU" +#endif +} +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_poly_compress_dv MLK_NAMESPACE_K(poly_compress_dv) +/** + * Compression (dv bits) and subsequent serialization of a polynomial. + * + * @spec{Implements `ByteEncode_{d_v} (Compress_{d_v} (v))` in @[FIPS203, + * Algorithm 14 (K-PKE.Encrypt), L23], with level-specific d_v defined in + * @[FIPS203, Table 2], and given by MLKEM_DV here.} + * + * @param[out] r Output byte array (of length MLKEM_POLYCOMPRESSEDBYTES_DV + * bytes). + * @param[in] a Input polynomial. Coefficients must be unsigned canonical, + * i.e. in [0,1,..,MLKEM_Q-1]. + */ +static MLK_INLINE void mlk_poly_compress_dv( + uint8_t r[MLKEM_POLYCOMPRESSEDBYTES_DV], const mlk_poly *a) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYCOMPRESSEDBYTES_DV)) + requires(memory_no_alias(a, sizeof(mlk_poly))) + requires(array_bound(a->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + assigns(memory_slice(r, MLKEM_POLYCOMPRESSEDBYTES_DV))) +{ +#if MLKEM_DV == 4 + mlk_poly_compress_d4(r, a); +#elif MLKEM_DV == 5 + mlk_poly_compress_d5(r, a); +#else +#error "Invalid value of MLKEM_DV" +#endif +} +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_poly_decompress_dv MLK_NAMESPACE_K(poly_decompress_dv) +/** + * De-serialization and subsequent decompression (dv bits) of a polynomial; + * approximate inverse of mlk_poly_compress_dv. + * + * Upon return, the coefficients of the output polynomial are + * unsigned-canonical (non-negative and smaller than MLKEM_Q). + * + * @spec{Implements `Decompress_{d_v} (ByteDecode_{d_v} (v))` in @[FIPS203, + * Algorithm 15 (K-PKE.Decrypt), L4], with level-specific d_v defined in + * @[FIPS203, Table 2], and given by MLKEM_DV here.} + * + * @param[out] r Output polynomial. + * @param[in] a Input byte array (of length MLKEM_POLYCOMPRESSEDBYTES_DV + * bytes). + */ +static MLK_INLINE void mlk_poly_decompress_dv( + mlk_poly *r, const uint8_t a[MLKEM_POLYCOMPRESSEDBYTES_DV]) +__contract__( + requires(memory_no_alias(a, MLKEM_POLYCOMPRESSEDBYTES_DV)) + requires(memory_no_alias(r, sizeof(mlk_poly))) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_bound(r->coeffs, 0, MLKEM_N, 0, MLKEM_Q))) +{ +#if MLKEM_DV == 4 + mlk_poly_decompress_d4(r, a); +#elif MLKEM_DV == 5 + mlk_poly_decompress_d5(r, a); +#else +#error "Invalid value of MLKEM_DV" +#endif +} +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_polyvec_compress_du MLK_NAMESPACE_K(polyvec_compress_du) +/** + * Compress and serialize a vector of polynomials. + * + * @spec{Implements `ByteEncode_{d_u} (Compress_{d_u} (u))` in @[FIPS203, + * Algorithm 14 (K-PKE.Encrypt), L22], with level-specific d_u defined in + * @[FIPS203, Table 2], and given by MLKEM_DU here.} + * + * @param[out] r Output byte array (needs space for + * MLKEM_POLYVECCOMPRESSEDBYTES_DU bytes). + * @param[in] a Input vector of polynomials. Coefficients must be unsigned + * canonical, i.e. in [0,1,..,MLKEM_Q-1]. + */ +MLK_INTERNAL_API +void mlk_polyvec_compress_du(uint8_t r[MLKEM_POLYVECCOMPRESSEDBYTES_DU], + const mlk_polyvec *a) +__contract__( + requires(memory_no_alias(r, MLKEM_POLYVECCOMPRESSEDBYTES_DU)) + requires(memory_no_alias(a, sizeof(mlk_polyvec))) + requires(forall(k0, 0, MLKEM_K, + array_bound(a->vec[k0].coeffs, 0, MLKEM_N, 0, MLKEM_Q))) + assigns(memory_slice(r, MLKEM_POLYVECCOMPRESSEDBYTES_DU)) +); +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +#if !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_polyvec_decompress_du MLK_NAMESPACE_K(polyvec_decompress_du) +/** + * De-serialize and decompress a vector of polynomials; approximate inverse + * of mlk_polyvec_compress_du. + * + * @spec{Implements `Decompress_{d_u} (ByteDecode_{d_u} (u))` in @[FIPS203, + * Algorithm 15 (K-PKE.Decrypt), L3], with level-specific d_u defined in + * @[FIPS203, Table 2], and given by MLKEM_DU here.} + * + * @param[out] r Output vector of polynomials. Coefficients are normalized + * to [0,1,..,MLKEM_Q-1]. + * @param[in] a Input byte array (of length MLKEM_POLYVECCOMPRESSEDBYTES_DU + * bytes). + */ +MLK_INTERNAL_API +void mlk_polyvec_decompress_du(mlk_polyvec *r, + const uint8_t a[MLKEM_POLYVECCOMPRESSEDBYTES_DU]) +__contract__( + requires(memory_no_alias(a, MLKEM_POLYVECCOMPRESSEDBYTES_DU)) + requires(memory_no_alias(r, sizeof(mlk_polyvec))) + assigns(memory_slice(r, sizeof(mlk_polyvec))) + ensures(forall(k0, 0, MLKEM_K, + array_bound(r->vec[k0].coeffs, 0, MLKEM_N, 0, MLKEM_Q))) +); +#endif /* !MLK_CONFIG_NO_DECAPS_API */ + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) || !defined(MLK_CONFIG_NO_ENCAPS_API) +#define mlk_polyvec_tobytes MLK_NAMESPACE_K(polyvec_tobytes) +/** + * Serialize a vector of polynomials. + * + * @spec{Implements ByteEncode_12 @[FIPS203, Algorithm 5]. Extended to + * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays] and + * @[FIPS203, 2.4.6, Matrices and Vectors].} + * + * @param[out] r Output byte array (needs space for MLKEM_POLYVECBYTES bytes). + * @param[in] a Input vector of polynomials. Each polynomial must have + * coefficients in [0,1,..,MLKEM_Q-1]. + */ +MLK_INTERNAL_API +void mlk_polyvec_tobytes(uint8_t r[MLKEM_POLYVECBYTES], const mlk_polyvec *a) +__contract__( + requires(memory_no_alias(a, sizeof(mlk_polyvec))) + requires(memory_no_alias(r, MLKEM_POLYVECBYTES)) + requires(forall(k0, 0, MLKEM_K, + array_bound(a->vec[k0].coeffs, 0, MLKEM_N, 0, MLKEM_Q))) + assigns(memory_slice(r, MLKEM_POLYVECBYTES)) +); +#endif /* !MLK_CONFIG_NO_KEYPAIR_API || !MLK_CONFIG_NO_ENCAPS_API */ + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_polyvec_frombytes MLK_NAMESPACE_K(polyvec_frombytes) +/** + * De-serialize a vector of polynomials; inverse of mlk_polyvec_tobytes. + * + * @spec{Implements ByteDecode_12 @[FIPS203, Algorithm 6]. Extended to + * vectors as per @[FIPS203, 2.4.8 Applying Algorithms to Arrays] and + * @[FIPS203, 2.4.6, Matrices and Vectors].} + * + * @param[out] r Output vector of polynomials. Coefficients will be + * normalized in [0,1,..,4095]. + * @param[in] a Input byte array (of length MLKEM_POLYVECBYTES bytes). + */ +MLK_INTERNAL_API +void mlk_polyvec_frombytes(mlk_polyvec *r, const uint8_t a[MLKEM_POLYVECBYTES]) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_polyvec))) + requires(memory_no_alias(a, MLKEM_POLYVECBYTES)) + assigns(memory_slice(r, sizeof(mlk_polyvec))) + ensures(forall(k0, 0, MLKEM_K, + array_bound(r->vec[k0].coeffs, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT))) +); +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +#define mlk_polyvec_ntt MLK_NAMESPACE_K(polyvec_ntt) +/** + * Apply forward NTT to all elements of a vector of polynomials. + * + * The input is assumed to be in normal order and coefficient-wise bound by + * MLKEM_Q in absolute value. + * + * The output polynomial is in bitreversed order, and coefficient-wise bound + * by MLK_NTT_BOUND in absolute value. + * + * @spec{Implements @[FIPS203, Algorithm 9, NTT]. Extended to vectors as per + * @[FIPS203, 2.4.6, Matrices and Vectors].} + * + * @param[in,out] r Input/output vector of polynomials. + */ +MLK_INTERNAL_API +void mlk_polyvec_ntt(mlk_polyvec *r) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_polyvec))) + requires(forall(j, 0, MLKEM_K, + array_abs_bound(r->vec[j].coeffs, 0, MLKEM_N, MLKEM_Q))) + assigns(memory_slice(r, sizeof(mlk_polyvec))) + ensures(forall(j, 0, MLKEM_K, + array_abs_bound(r->vec[j].coeffs, 0, MLKEM_N, MLK_NTT_BOUND))) +); + +#if !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_polyvec_invntt_tomont MLK_NAMESPACE_K(polyvec_invntt_tomont) +/** + * Apply inverse NTT to all elements of a vector of polynomials and multiply + * by Montgomery factor 2^16. + * + * The input is assumed to be in bitreversed order, and can have arbitrary + * coefficients in int16_t. + * + * The output polynomial is in normal order, and coefficient-wise bound by + * MLK_INVNTT_BOUND in absolute value. + * + * @spec{Implements @[FIPS203, Algorithm 10, NTT^{-1}]. Extended to vectors + * as per @[FIPS203, 2.4.6, Matrices and Vectors].} + * + * @param[in,out] r Input/output vector of polynomials. + */ +MLK_INTERNAL_API +void mlk_polyvec_invntt_tomont(mlk_polyvec *r) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_polyvec))) + assigns(memory_slice(r, sizeof(mlk_polyvec))) + ensures(forall(j, 0, MLKEM_K, + array_abs_bound(r->vec[j].coeffs, 0, MLKEM_N, MLK_INVNTT_BOUND))) +); +#endif /* !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +#define mlk_polyvec_basemul_acc_montgomery_cached \ + MLK_NAMESPACE_K(polyvec_basemul_acc_montgomery_cached) +/** + * Scalar product of two vectors of polynomials in NTT domain, using + * mulcache for the second operand. + * + * Bounds: every coefficient of @p a is assumed to be in [0,1,..,4095]. No + * bounds guarantees for the coefficients in the result. + * + * @spec{Implements @[FIPS203, Section 2.4.7, Eq (2.14)], @[FIPS203, + * Algorithm 11, MultiplyNTTs], and @[FIPS203, Algorithm 12, + * BaseCaseMultiply].} + * + * @param[out] r Output polynomial. + * @param[in] a First input polynomial vector. + * @param[in] b Second input polynomial vector. + * @param[in] b_cache Mulcache for the second input polynomial vector. Can + * be computed via mlk_polyvec_mulcache_compute(). + */ +MLK_INTERNAL_API +void mlk_polyvec_basemul_acc_montgomery_cached( + mlk_poly *r, const mlk_polyvec *a, const mlk_polyvec *b, + const mlk_polyvec_mulcache *b_cache) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(memory_no_alias(a, sizeof(mlk_polyvec))) + requires(memory_no_alias(b, sizeof(mlk_polyvec))) + requires(memory_no_alias(b_cache, sizeof(mlk_polyvec_mulcache))) + requires(forall(k1, 0, MLKEM_K, + array_bound(a->vec[k1].coeffs, 0, MLKEM_N, 0, MLKEM_UINT12_LIMIT))) + assigns(memory_slice(r, sizeof(mlk_poly))) +); + +#define mlk_polyvec_mulcache_compute MLK_NAMESPACE_K(polyvec_mulcache_compute) +/** + * Compute the mulcache for a vector of polynomials in NTT domain. + * + * The mulcache of a degree-2 polynomial b := b0 + b1*X in Fq[X]/(X^2-zeta) + * is the value b1*zeta, needed when computing products of b in + * Fq[X]/(X^2-zeta). + * + * The mulcache of a polynomial in NTT domain -- which is a 128-tuple of + * degree-2 polynomials in Fq[X]/(X^2-zeta), for varying zeta, is the + * 128-tuple of mulcaches of those polynomials. + * + * The mulcache of a vector of polynomials is the vector of mulcaches of + * its entries. + * + * @spec{Caches `b_1 * \gamma` in @[FIPS203, Algorithm 12, BaseCaseMultiply, + * L1].} + * + * @param[out] x Mulcache to be populated. + * @param[in] a Input polynomial vector. + */ +/* + * NOTE: The default C implementation of this function populates + * the mulcache with values in (-q,q), but this is not needed for the + * higher level safety proofs, and thus not part of the spec. + */ +MLK_INTERNAL_API +void mlk_polyvec_mulcache_compute(mlk_polyvec_mulcache *x, const mlk_polyvec *a) +__contract__( + requires(memory_no_alias(x, sizeof(mlk_polyvec_mulcache))) + requires(memory_no_alias(a, sizeof(mlk_polyvec))) + assigns(memory_slice(x, sizeof(mlk_polyvec_mulcache))) +); + +#define mlk_polyvec_reduce MLK_NAMESPACE_K(polyvec_reduce) +/** + * Apply Barrett reduction to each coefficient of each element of a vector + * of polynomials. For details of the Barrett reduction see comments in + * poly.c. + * + * @spec{Normalizes on unsigned canonical representatives ahead of calling + * @[FIPS203, Compress_d, Eq (4.7)]. This is not made explicit in FIPS 203.} + * + * @param[in,out] r Input/output polynomial vector. + */ +/* + * NOTE: The semantics of mlk_polyvec_reduce() is different in + * the reference implementation, which requires + * signed canonical output data. Unsigned canonical + * outputs are better suited to the only remaining + * use of mlk_poly_reduce() in the context of (de)serialization. + */ +MLK_INTERNAL_API +void mlk_polyvec_reduce(mlk_polyvec *r) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_polyvec))) + assigns(memory_slice(r, sizeof(mlk_polyvec))) + ensures(forall(k0, 0, MLKEM_K, + array_bound(r->vec[k0].coeffs, 0, MLKEM_N, 0, MLKEM_Q))) +); + +#define mlk_polyvec_add MLK_NAMESPACE_K(polyvec_add) +/** + * Add vectors of polynomials. + * + * The coefficients of @p r and @p b must be such that the addition does + * not overflow. Otherwise, the behaviour of this function is undefined. + * + * The coefficients returned in @p *r are in int16_t which is sufficient to + * prove type-safety of calling units. Therefore, no stronger ensures clause + * is required on this function. + * + * @spec{@[FIPS203, 2.4.5, Arithmetic With Polynomials and NTT + * Representations]. Used in @[FIPS203, Algorithm 14 (K-PKE.Encrypt), L19].} + * + * @param[in,out] r Input-output vector of polynomials to be added to. + * @param[in] b Second input vector of polynomials. + */ +MLK_INTERNAL_API +void mlk_polyvec_add(mlk_polyvec *r, const mlk_polyvec *b) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_polyvec))) + requires(memory_no_alias(b, sizeof(mlk_polyvec))) + requires(forall(j0, 0, MLKEM_K, + forall(k0, 0, MLKEM_N, + (int32_t)r->vec[j0].coeffs[k0] + b->vec[j0].coeffs[k0] <= INT16_MAX))) + requires(forall(j1, 0, MLKEM_K, + forall(k1, 0, MLKEM_N, + (int32_t)r->vec[j1].coeffs[k1] + b->vec[j1].coeffs[k1] >= INT16_MIN))) + assigns(memory_slice(r, sizeof(mlk_polyvec))) +); + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) +#define mlk_polyvec_tomont MLK_NAMESPACE_K(polyvec_tomont) +/** + * In-place conversion of all coefficients of a polynomial vector from the + * normal domain to the Montgomery domain. + * + * Bounds: output < MLKEM_Q in absolute value. + * + * @spec{Internal normalization required in `mlk_indcpa_keypair_derand` as + * part of matrix-vector multiplication @[FIPS203, Algorithm 13, K-PKE.KeyGen, + * L18].} + * + * @param[in,out] r Input/output polynomial vector. + */ +MLK_INTERNAL_API +void mlk_polyvec_tomont(mlk_polyvec *r) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_polyvec))) + assigns(memory_slice(r, sizeof(mlk_polyvec))) + ensures(forall(j, 0, MLKEM_K, + array_abs_bound(r->vec[j].coeffs, 0, MLKEM_N, MLKEM_Q))) +); +#endif /* !MLK_CONFIG_NO_KEYPAIR_API */ + +#if !defined(MLK_CONFIG_NO_KEYPAIR_API) || \ + (MLKEM_ETA1 == MLKEM_ETA2 && (!defined(MLK_CONFIG_NO_ENCAPS_API) || \ + !defined(MLK_CONFIG_NO_DECAPS_API))) +#define mlk_poly_getnoise_eta1_4x MLK_NAMESPACE_K(poly_getnoise_eta1_4x) +/** + * Batch sample four polynomials deterministically from a seed and nonces, + * with output polynomials close to centered binomial distribution with + * parameter MLKEM_ETA1. + * + * @spec{Implements 4x `SamplePolyCBD_{eta1} (PRF_{eta1} (sigma, N))`: + * @[FIPS203, Algorithm 8, SamplePolyCBD_eta] and @[FIPS203, Eq (4.3), + * PRF_eta]. `SamplePolyCBD_{eta1} (PRF_{eta1} (sigma, N))` appears in + * @[FIPS203, Algorithm 13, K-PKE.KeyGen, L{9, 13}] and @[FIPS203, + * Algorithm 14, K-PKE.Encrypt, L10].} + * + * @param[out] r0 Output polynomial. + * @param[out] r1 Output polynomial. + * @param[out] r2 Output polynomial. + * @param[out] r3 Output polynomial. May be NULL. + * @param[in] seed Input seed (of length MLKEM_SYMBYTES bytes). + * @param nonce0 One-byte input nonce. + * @param nonce1 One-byte input nonce. + * @param nonce2 One-byte input nonce. + * @param nonce3 One-byte input nonce. + */ +MLK_INTERNAL_API +void mlk_poly_getnoise_eta1_4x(mlk_poly *r0, mlk_poly *r1, mlk_poly *r2, + mlk_poly *r3, const uint8_t seed[MLKEM_SYMBYTES], + uint8_t nonce0, uint8_t nonce1, uint8_t nonce2, + uint8_t nonce3) +__contract__( + requires(memory_no_alias(seed, MLKEM_SYMBYTES)) + requires(memory_no_alias(r0, sizeof(mlk_poly))) + requires(memory_no_alias(r1, sizeof(mlk_poly))) + requires(memory_no_alias(r2, sizeof(mlk_poly))) + requires(r3 == NULL || memory_no_alias(r3, sizeof(mlk_poly))) + assigns(memory_slice(r0, sizeof(mlk_poly))) + assigns(memory_slice(r1, sizeof(mlk_poly))) + assigns(memory_slice(r2, sizeof(mlk_poly))) + assigns(r3 != NULL: memory_slice(r3, sizeof(mlk_poly))) + ensures(array_abs_bound(r0->coeffs,0, MLKEM_N, MLKEM_ETA1 + 1)) + ensures(array_abs_bound(r1->coeffs,0, MLKEM_N, MLKEM_ETA1 + 1)) + ensures(array_abs_bound(r2->coeffs,0, MLKEM_N, MLKEM_ETA1 + 1)) + ensures(r3 != NULL ==> array_abs_bound(r3->coeffs,0, MLKEM_N, MLKEM_ETA1 + 1)) +); + +#if MLKEM_ETA1 == MLKEM_ETA2 +/* + * We only require mlk_poly_getnoise_eta2_4x for ml-kem-768 and ml-kem-1024 + * where MLKEM_ETA2 = MLKEM_ETA1 = 2. + * For ml-kem-512, mlk_poly_getnoise_eta1122_4x is used instead. + */ +#define mlk_poly_getnoise_eta2_4x mlk_poly_getnoise_eta1_4x +#endif /* MLKEM_ETA1 == MLKEM_ETA2 */ +#endif /* !MLK_CONFIG_NO_KEYPAIR_API || (MLKEM_ETA1 == MLKEM_ETA2 && \ + (!MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API)) */ + +#if (MLKEM_K == 2 || MLKEM_K == 4) && \ + (!defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)) +#define mlk_poly_getnoise_eta2 MLK_NAMESPACE_K(poly_getnoise_eta2) +/** + * Sample a polynomial deterministically from a seed and a nonce, with + * output polynomial close to centered binomial distribution with parameter + * MLKEM_ETA2. + * + * @spec{Implements `SamplePolyCBD_{eta2} (PRF_{eta2} (sigma, N))`: + * @[FIPS203, Algorithm 8, SamplePolyCBD_eta] and @[FIPS203, Eq (4.3), + * PRF_eta]. `SamplePolyCBD_{eta2} (PRF_{eta2} (sigma, N))` appears in + * @[FIPS203, Algorithm 14, K-PKE.Encrypt, L14].} + * + * @param[out] r Output polynomial. + * @param[in] seed Input seed (of length MLKEM_SYMBYTES bytes). + * @param nonce One-byte input nonce. + */ +MLK_INTERNAL_API +void mlk_poly_getnoise_eta2(mlk_poly *r, const uint8_t seed[MLKEM_SYMBYTES], + uint8_t nonce) +__contract__( + requires(memory_no_alias(r, sizeof(mlk_poly))) + requires(memory_no_alias(seed, MLKEM_SYMBYTES)) + assigns(memory_slice(r, sizeof(mlk_poly))) + ensures(array_abs_bound(r->coeffs, 0, MLKEM_N, MLKEM_ETA2 + 1)) +); +#endif /* (MLKEM_K == 2 || MLKEM_K == 4) && (!MLK_CONFIG_NO_ENCAPS_API || \ + !MLK_CONFIG_NO_DECAPS_API) */ + +#if MLKEM_K == 2 && \ + (!defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API)) +#define mlk_poly_getnoise_eta1122_4x MLK_NAMESPACE_K(poly_getnoise_eta1122_4x) +/** + * Batch sample four polynomials deterministically from a seed and nonces, + * with output polynomials close to centered binomial distribution with + * parameter MLKEM_ETA1 and MLKEM_ETA2. + * + * @spec{Implements two instances each of + * `SamplePolyCBD_{eta1} (PRF_{eta1} (sigma, N))` and + * `SamplePolyCBD_{eta2} (PRF_{eta2} (sigma, N))`: + * @[FIPS203, Algorithm 8, SamplePolyCBD_eta] and @[FIPS203, Eq (4.3), + * PRF_eta]. `SamplePolyCBD_{eta2} (PRF_{eta2} (sigma, N))` appears in + * @[FIPS203, Algorithm 14, K-PKE.Encrypt, L14].} + * + * @param[out] r0 Output polynomial. + * @param[out] r1 Output polynomial. + * @param[out] r2 Output polynomial. + * @param[out] r3 Output polynomial. + * @param[in] seed Input seed (of length MLKEM_SYMBYTES bytes). + * @param nonce0 One-byte input nonce. + * @param nonce1 One-byte input nonce. + * @param nonce2 One-byte input nonce. + * @param nonce3 One-byte input nonce. + */ +MLK_INTERNAL_API +void mlk_poly_getnoise_eta1122_4x(mlk_poly *r0, mlk_poly *r1, mlk_poly *r2, + mlk_poly *r3, + const uint8_t seed[MLKEM_SYMBYTES], + uint8_t nonce0, uint8_t nonce1, + uint8_t nonce2, uint8_t nonce3) +__contract__( + requires(memory_no_alias(r0, sizeof(mlk_poly))) + requires(memory_no_alias(r1, sizeof(mlk_poly))) + requires(memory_no_alias(r2, sizeof(mlk_poly))) + requires(memory_no_alias(r3, sizeof(mlk_poly))) + requires(memory_no_alias(seed, MLKEM_SYMBYTES)) + assigns(memory_slice(r0, sizeof(mlk_poly))) + assigns(memory_slice(r1, sizeof(mlk_poly))) + assigns(memory_slice(r2, sizeof(mlk_poly))) + assigns(memory_slice(r3, sizeof(mlk_poly))) + ensures(array_abs_bound(r0->coeffs,0, MLKEM_N, MLKEM_ETA1 + 1) + && array_abs_bound(r1->coeffs,0, MLKEM_N, MLKEM_ETA1 + 1) + && array_abs_bound(r2->coeffs,0, MLKEM_N, MLKEM_ETA2 + 1) + && array_abs_bound(r3->coeffs,0, MLKEM_N, MLKEM_ETA2 + 1)) +); +#endif /* MLKEM_K == 2 && (!MLK_CONFIG_NO_ENCAPS_API || \ + !MLK_CONFIG_NO_DECAPS_API) */ + +#endif /* !MLK_POLY_K_H */ diff --git a/cbits/mlkem/src/randombytes.h b/cbits/mlkem/src/randombytes.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/randombytes.h @@ -0,0 +1,52 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLK_RANDOMBYTES_H +#define MLK_RANDOMBYTES_H + + +#include "cbmc.h" +#include "common.h" + +#if !defined(MLK_CONFIG_NO_RANDOMIZED_API) +#if !defined(MLK_CONFIG_CUSTOM_RANDOMBYTES) +/** + * Fill a buffer with cryptographically secure random bytes. + * + * mlkem-native does not provide an implementation of this function. + * It must be provided by the consumer. + * + * To use a custom random byte source with a different name or signature, + * set MLK_CONFIG_CUSTOM_RANDOMBYTES and define mlk_randombytes directly. + * + * @param[out] out Output buffer. + * @param outlen Number of random bytes to write. + * + * @retval 0 Success. + * @retval other Failure; top-level APIs propagate this as MLK_ERR_RNG_FAIL. + */ +int randombytes(uint8_t *out, size_t outlen); + +/** + * Internal wrapper around randombytes(). + * + * Fills a buffer with cryptographically secure random bytes. + * + * This function can be replaced by setting MLK_CONFIG_CUSTOM_RANDOMBYTES + * and defining mlk_randombytes directly. + * + * @param[out] out Output buffer. + * @param outlen Number of random bytes to write. + * + * @retval 0 Success. + * @retval other Failure; top-level APIs propagate this as MLK_ERR_RNG_FAIL. + */ +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_randombytes(uint8_t *out, size_t outlen) +__contract__( + requires(memory_no_alias(out, outlen)) + assigns(memory_slice(out, outlen))) { return randombytes(out, outlen); } +#endif /* !MLK_CONFIG_CUSTOM_RANDOMBYTES */ +#endif /* !MLK_CONFIG_NO_RANDOMIZED_API */ +#endif /* !MLK_RANDOMBYTES_H */ diff --git a/cbits/mlkem/src/sampling.c b/cbits/mlkem/src/sampling.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/sampling.c @@ -0,0 +1,371 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + * + * - [REF] + * CRYSTALS-Kyber C reference implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/ref + */ + +#include "common.h" +#if !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) + +#include "debug.h" +#include "sampling.h" +#include "symmetric.h" +#include "verify.h" + +/* Reference: `rej_uniform()` in the reference implementation @[REF]. + * - Our signature differs from the reference implementation + * in that it adds the offset and always expects the base of the + * target buffer. This avoids shifting the buffer base in the + * caller, which appears tricky to reason about. */ +MLK_STATIC_TESTABLE unsigned mlk_rej_uniform_c(int16_t *r, unsigned target, + unsigned offset, + const uint8_t *buf, + unsigned buflen) +__contract__( + requires(offset <= target && target <= 4096 && buflen <= 4096 && buflen % 3 == 0) + requires(memory_no_alias(r, sizeof(int16_t) * target)) + requires(memory_no_alias(buf, buflen)) + requires(array_bound(r, 0, offset, 0, MLKEM_Q)) + assigns(memory_slice(r, sizeof(int16_t) * target)) + ensures(offset <= return_value && return_value <= target) + ensures(array_bound(r, 0, return_value, 0, MLKEM_Q))) +{ + unsigned ctr, pos; + int16_t val0, val1; + + mlk_assert_bound(r, offset, 0, MLKEM_Q); + + ctr = offset; + pos = 0; + /* pos + 3 cannot overflow due to the assumption buflen <= 4096 */ + while (ctr < target && pos + 3 <= buflen) + __loop__( + invariant(offset <= ctr && ctr <= target && pos <= buflen) + invariant(array_bound(r, 0, ctr, 0, MLKEM_Q)) + decreases(buflen - pos)) + { + /* Safety: + * - The explicit cast to uint16_t ensures that << 8 does + * not signed-overflow even on a 16-bit system. + * - The conversion to int16_t is safe due to the explicit 0xFFF + * truncation. + */ + val0 = (int16_t)(((buf[pos + 0] >> 0) | ((uint16_t)buf[pos + 1] << 8)) & + 0xFFF); + val1 = (int16_t)(((buf[pos + 1] >> 4) | (buf[pos + 2] << 4)) & 0xFFF); + pos += 3; + + if (val0 < MLKEM_Q) + { + r[ctr++] = val0; + } + if (ctr < target && val1 < MLKEM_Q) + { + r[ctr++] = val1; + } + } + + mlk_assert_bound(r, ctr, 0, MLKEM_Q); + return ctr; +} + +/** + * Run rejection sampling on uniform random bytes to generate uniform random + * integers mod MLKEM_Q. + * + * @reference{`rej_uniform()` in the reference implementation @[REF]. Our + * signature differs from the reference in that it adds the offset and always + * expects the base of the target buffer; this avoids shifting the buffer + * base in the caller, which is tricky to reason about. Has an optional + * fallback to a native implementation.} + * + * @param[out] r Output buffer. + * @param target Requested number of 16-bit integers (uniform mod MLKEM_Q). + * Must be <= 4096. + * @param offset Number of 16-bit integers that have already been + * sampled. Must be <= @p target. + * @param[in] buf Input buffer (assumed to be uniform random bytes). + * @param buflen Length of input buffer in bytes. Must be <= 4096 and a + * multiple of 3. + * + * @note Strictly speaking, only a few values of @p buflen near UINT_MAX need + * excluding. The limit of 4096 is somewhat arbitrary but sufficient + * for all uses of this function. Similarly, the actual limit for + * @p target is UINT_MAX/2. + * + * @return New offset of sampled 16-bit integers, at most @p target and at + * least the initial @p offset. If the new offset is strictly less + * than @p target, the entire input buffer is guaranteed to have been + * consumed; otherwise no information is provided on how many bytes + * of the input buffer have been consumed. + */ +static unsigned mlk_rej_uniform(int16_t *r, unsigned target, unsigned offset, + const uint8_t *buf, unsigned buflen) +__contract__( + requires(offset <= target && target <= 4096 && buflen <= 4096 && buflen % 3 == 0) + requires(memory_no_alias(r, sizeof(int16_t) * target)) + requires(memory_no_alias(buf, buflen)) + requires(array_bound(r, 0, offset, 0, MLKEM_Q)) + assigns(memory_slice(r, sizeof(int16_t) * target)) + ensures(offset <= return_value && return_value <= target) + ensures(array_bound(r, 0, return_value, 0, MLKEM_Q)) +) +{ +#if defined(MLK_USE_NATIVE_REJ_UNIFORM) + if (offset == 0) + { + int ret; + ret = mlk_rej_uniform_native(r, target, buf, buflen); + if (ret != MLK_NATIVE_FUNC_FALLBACK) + { + unsigned res = (unsigned)ret; + mlk_assert_bound(r, res, 0, MLKEM_Q); + return res; + } + } +#endif /* MLK_USE_NATIVE_REJ_UNIFORM */ + + return mlk_rej_uniform_c(r, target, offset, buf, buflen); +} + +#ifndef MLKEM_GEN_MATRIX_NBLOCKS +#define MLKEM_GEN_MATRIX_NBLOCKS \ + ((12 * MLKEM_N / 8 * ((uint32_t)1 << 12) / MLKEM_Q + MLK_XOF_RATE) / \ + MLK_XOF_RATE) +#endif + +#if !defined(MLK_CONFIG_SERIAL_FIPS202_ONLY) +/* Reference: Does not exist in the reference implementation @[REF]. + * - x4-batched version of `rej_uniform()` from the + * reference implementation, leveraging x4-batched Keccak-f1600. */ +MLK_INTERNAL_API +void mlk_poly_rej_uniform_x4(mlk_poly *vec0, mlk_poly *vec1, mlk_poly *vec2, + mlk_poly *vec3, + uint8_t seed[4][MLK_ALIGN_UP(MLKEM_SYMBYTES + 2)]) +{ + /* Temporary buffers for XOF output before rejection sampling */ + MLK_ALIGN uint8_t + buf[4][MLK_ALIGN_UP(MLKEM_GEN_MATRIX_NBLOCKS * MLK_XOF_RATE)]; + + /* Tracks the number of coefficients we have already sampled */ + unsigned ctr[4]; + mlk_xof_x4_ctx statex; + unsigned buflen; + + mlk_xof_x4_init(&statex); + mlk_xof_x4_absorb(&statex, seed, MLKEM_SYMBYTES + 2); + + /* + * Initially, squeeze heuristic number of MLKEM_GEN_MATRIX_NBLOCKS. + * This should generate the matrix entries with high probability. + */ + mlk_xof_x4_squeezeblocks(buf, MLKEM_GEN_MATRIX_NBLOCKS, &statex); + buflen = MLKEM_GEN_MATRIX_NBLOCKS * MLK_XOF_RATE; + ctr[0] = mlk_rej_uniform(vec0->coeffs, MLKEM_N, 0, buf[0], buflen); + ctr[1] = mlk_rej_uniform(vec1->coeffs, MLKEM_N, 0, buf[1], buflen); + ctr[2] = mlk_rej_uniform(vec2->coeffs, MLKEM_N, 0, buf[2], buflen); + ctr[3] = mlk_rej_uniform(vec3->coeffs, MLKEM_N, 0, buf[3], buflen); + + /* + * So long as not all matrix entries have been generated, squeeze + * one more block a time until we're done. + */ + buflen = MLK_XOF_RATE; + while (ctr[0] < MLKEM_N || ctr[1] < MLKEM_N || ctr[2] < MLKEM_N || + ctr[3] < MLKEM_N) + __loop__( + assigns(ctr, statex, + memory_slice(vec0, sizeof(mlk_poly)), + memory_slice(vec1, sizeof(mlk_poly)), + memory_slice(vec2, sizeof(mlk_poly)), + memory_slice(vec3, sizeof(mlk_poly)), + object_whole(buf)) + invariant(ctr[0] <= MLKEM_N && ctr[1] <= MLKEM_N) + invariant(ctr[2] <= MLKEM_N && ctr[3] <= MLKEM_N) + invariant(array_bound(vec0->coeffs, 0, ctr[0], 0, MLKEM_Q)) + invariant(array_bound(vec1->coeffs, 0, ctr[1], 0, MLKEM_Q)) + invariant(array_bound(vec2->coeffs, 0, ctr[2], 0, MLKEM_Q)) + invariant(array_bound(vec3->coeffs, 0, ctr[3], 0, MLKEM_Q))) + { + mlk_xof_x4_squeezeblocks(buf, 1, &statex); + ctr[0] = mlk_rej_uniform(vec0->coeffs, MLKEM_N, ctr[0], buf[0], buflen); + ctr[1] = mlk_rej_uniform(vec1->coeffs, MLKEM_N, ctr[1], buf[1], buflen); + ctr[2] = mlk_rej_uniform(vec2->coeffs, MLKEM_N, ctr[2], buf[2], buflen); + ctr[3] = mlk_rej_uniform(vec3->coeffs, MLKEM_N, ctr[3], buf[3], buflen); + } + + mlk_xof_x4_release(&statex); + + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + mlk_zeroize(buf, sizeof(buf)); +} +#endif /* !MLK_CONFIG_SERIAL_FIPS202_ONLY */ + +MLK_INTERNAL_API +void mlk_poly_rej_uniform(mlk_poly *entry, uint8_t seed[MLKEM_SYMBYTES + 2]) +{ + mlk_xof_ctx state; + MLK_ALIGN uint8_t buf[MLKEM_GEN_MATRIX_NBLOCKS * MLK_XOF_RATE]; + unsigned ctr, buflen; + + mlk_xof_init(&state); + mlk_xof_absorb(&state, seed, MLKEM_SYMBYTES + 2); + + /* Initially, squeeze + sample heuristic number of MLKEM_GEN_MATRIX_NBLOCKS. + */ + /* This should generate the matrix entry with high probability. */ + mlk_xof_squeezeblocks(buf, MLKEM_GEN_MATRIX_NBLOCKS, &state); + buflen = MLKEM_GEN_MATRIX_NBLOCKS * MLK_XOF_RATE; + ctr = mlk_rej_uniform(entry->coeffs, MLKEM_N, 0, buf, buflen); + + /* Squeeze + sample one more block a time until we're done */ + buflen = MLK_XOF_RATE; + while (ctr < MLKEM_N) + __loop__( + assigns(ctr, state, memory_slice(entry, sizeof(mlk_poly)), object_whole(buf)) + invariant(ctr <= MLKEM_N) + invariant(array_bound(entry->coeffs, 0, ctr, 0, MLKEM_Q))) + { + mlk_xof_squeezeblocks(buf, 1, &state); + ctr = mlk_rej_uniform(entry->coeffs, MLKEM_N, ctr, buf, buflen); + } + + mlk_xof_release(&state); + + /* Specification: Partially implements + * @[FIPS203, Section 3.3, Destruction of intermediate values] */ + mlk_zeroize(buf, sizeof(buf)); +} + +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_ETA1 == 2 || \ + !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +/** + * Load 4 bytes into a 32-bit integer in little-endian order. + * + * @reference{`load32_littleendian()` in the reference implementation @[REF].} + * + * @param[in] x Input byte array. + * + * @return 32-bit unsigned integer loaded from @p x. + */ +static uint32_t mlk_load32_littleendian(const uint8_t x[4]) +{ + uint32_t r; + r = (uint32_t)x[0]; + r |= (uint32_t)x[1] << 8; + r |= (uint32_t)x[2] << 16; + r |= (uint32_t)x[3] << 24; + return r; +} + +/* Reference: `cbd2()` in the reference implementation @[REF]. */ +MLK_INTERNAL_API +void mlk_poly_cbd2(mlk_poly *r, const uint8_t buf[2 * MLKEM_N / 4]) +{ + unsigned i; + for (i = 0; i < MLKEM_N / 8; i++) + __loop__( + invariant(i <= MLKEM_N / 8) + invariant(array_abs_bound(r->coeffs, 0, 8 * i, 3)) + decreases(MLKEM_N / 8 - i)) + { + unsigned j; + uint32_t t = mlk_load32_littleendian(buf + 4 * i); + uint32_t d = t & 0x55555555; + d += (t >> 1) & 0x55555555; + + for (j = 0; j < 8; j++) + __loop__( + invariant(i <= MLKEM_N / 8 && j <= 8) + invariant(array_abs_bound(r->coeffs, 0, 8 * i + j, 3)) + decreases(8 - j)) + { + /* Safety: The & 0x3 masks each value to 2 bits (range [0, 3]), so the + * truncation and subsequent subtraction in int16_t is lossless. */ + const int16_t a = (int16_t)((d >> (4 * j + 0)) & 0x3); + const int16_t b = (int16_t)((d >> (4 * j + 2)) & 0x3); + r->coeffs[8 * i + j] = (int16_t)(a - b); + } + } +} +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_ETA1 == 2 || \ + !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_ETA1 == 3 +/** + * Load 3 bytes into a 32-bit integer in little-endian order. + * + * This function is only needed for ML-KEM-512. + * + * @reference{`load24_littleendian()` in the reference implementation @[REF].} + * + * @param[in] x Input byte array. + * + * @return 32-bit unsigned integer loaded from @p x (most significant byte + * is zero). + */ +static uint32_t mlk_load24_littleendian(const uint8_t x[3]) +{ + uint32_t r; + r = (uint32_t)x[0]; + r |= (uint32_t)x[1] << 8; + r |= (uint32_t)x[2] << 16; + return r; +} + +/* Reference: `cbd3()` in the reference implementation @[REF]. */ +MLK_INTERNAL_API +void mlk_poly_cbd3(mlk_poly *r, const uint8_t buf[3 * MLKEM_N / 4]) +{ + unsigned i; + for (i = 0; i < MLKEM_N / 4; i++) + __loop__( + invariant(i <= MLKEM_N / 4) + invariant(array_abs_bound(r->coeffs, 0, 4 * i, 4)) + decreases(MLKEM_N / 4 - i)) + { + unsigned j; + const uint32_t t = mlk_load24_littleendian(buf + 3 * i); + uint32_t d = t & 0x00249249; + d += (t >> 1) & 0x00249249; + d += (t >> 2) & 0x00249249; + + for (j = 0; j < 4; j++) + __loop__( + invariant(i <= MLKEM_N / 4 && j <= 4) + invariant(array_abs_bound(r->coeffs, 0, 4 * i + j, 4)) + decreases(4 - j)) + { + /* Safety: The & 0x7 masks each value to 3 bits (range [0, 7]), so the + * truncation and subsequent subtraction in int16_t is lossless. */ + const int16_t a = (int16_t)((d >> (6 * j + 0)) & 0x7); + const int16_t b = (int16_t)((d >> (6 * j + 3)) & 0x7); + r->coeffs[4 * i + j] = (int16_t)(a - b); + } + } +} +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_ETA1 == 3 */ + +#else /* !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +MLK_EMPTY_CU(sampling) + +#endif /* MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +/* To facilitate single-compilation-unit (SCU) builds, undefine all macros. + * Don't modify by hand -- this is auto-generated by scripts/autogen. */ +#undef MLKEM_GEN_MATRIX_NBLOCKS diff --git a/cbits/mlkem/src/sampling.h b/cbits/mlkem/src/sampling.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/sampling.h @@ -0,0 +1,111 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + */ + +#ifndef MLK_SAMPLING_H +#define MLK_SAMPLING_H + +#include "cbmc.h" +#include "common.h" +#include "poly.h" + +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_ETA1 == 2 || \ + !defined(MLK_CONFIG_NO_ENCAPS_API) || !defined(MLK_CONFIG_NO_DECAPS_API) +#define mlk_poly_cbd2 MLK_NAMESPACE(poly_cbd2) +/** + * Given an array of uniformly random bytes, compute a polynomial with + * coefficients distributed according to a centered binomial distribution + * with parameter eta=2. + * + * @spec{Implements @[FIPS203, Algorithm 8, SamplePolyCBD_2].} + * + * @param[out] r Output polynomial. + * @param[in] buf Input byte array. + */ +MLK_INTERNAL_API +void mlk_poly_cbd2(mlk_poly *r, const uint8_t buf[2 * MLKEM_N / 4]); +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_ETA1 == 2 || \ + !MLK_CONFIG_NO_ENCAPS_API || !MLK_CONFIG_NO_DECAPS_API */ + +#if defined(MLK_CONFIG_MULTILEVEL_WITH_SHARED) || MLKEM_ETA1 == 3 +#define mlk_poly_cbd3 MLK_NAMESPACE(poly_cbd3) +/** + * Given an array of uniformly random bytes, compute a polynomial with + * coefficients distributed according to a centered binomial distribution + * with parameter eta=3. + * + * This function is only needed for ML-KEM-512. + * + * @spec{Implements @[FIPS203, Algorithm 8, SamplePolyCBD_3].} + * + * @param[out] r Output polynomial. + * @param[in] buf Input byte array. + */ +MLK_INTERNAL_API +void mlk_poly_cbd3(mlk_poly *r, const uint8_t buf[3 * MLKEM_N / 4]); +#endif /* MLK_CONFIG_MULTILEVEL_WITH_SHARED || MLKEM_ETA1 == 3 */ + +#if !defined(MLK_CONFIG_SERIAL_FIPS202_ONLY) +#define mlk_poly_rej_uniform_x4 MLK_NAMESPACE(poly_rej_uniform_x4) +/** + * Generate four polynomials using rejection sampling on (pseudo-)uniformly + * random bytes sampled from a seed. + * + * @spec{Implements @[FIPS203, Algorithm 7, SampleNTT].} + * + * @param[out] vec0 Polynomial to be sampled. + * @param[out] vec1 Polynomial to be sampled. + * @param[out] vec2 Polynomial to be sampled. + * @param[out] vec3 Polynomial to be sampled. + * @param[in] seed Consecutive array of 4 seed buffers of size + * MLKEM_SYMBYTES + 2 each, plus padding for alignment. + */ +MLK_INTERNAL_API +void mlk_poly_rej_uniform_x4(mlk_poly *vec0, mlk_poly *vec1, mlk_poly *vec2, + mlk_poly *vec3, + uint8_t seed[4][MLK_ALIGN_UP(MLKEM_SYMBYTES + 2)]) +__contract__( + requires(memory_no_alias(vec0, sizeof(mlk_poly))) + requires(memory_no_alias(vec1, sizeof(mlk_poly))) + requires(memory_no_alias(vec2, sizeof(mlk_poly))) + requires(memory_no_alias(vec3, sizeof(mlk_poly))) + requires(memory_no_alias(seed, 4 * MLK_ALIGN_UP(MLKEM_SYMBYTES + 2))) + assigns(memory_slice(vec0, sizeof(mlk_poly))) + assigns(memory_slice(vec1, sizeof(mlk_poly))) + assigns(memory_slice(vec2, sizeof(mlk_poly))) + assigns(memory_slice(vec3, sizeof(mlk_poly))) + ensures(array_bound(vec0->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + ensures(array_bound(vec1->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + ensures(array_bound(vec2->coeffs, 0, MLKEM_N, 0, MLKEM_Q)) + ensures(array_bound(vec3->coeffs, 0, MLKEM_N, 0, MLKEM_Q))); +#endif /* !MLK_CONFIG_SERIAL_FIPS202_ONLY */ + +#define mlk_poly_rej_uniform MLK_NAMESPACE(poly_rej_uniform) +/** + * Generate a polynomial using rejection sampling on (pseudo-)uniformly + * random bytes sampled from a seed. + * + * @spec{Implements @[FIPS203, Algorithm 7, SampleNTT].} + * + * @param[out] entry Polynomial to be sampled. + * @param[in] seed Seed buffer of size MLKEM_SYMBYTES + 2. + */ +MLK_INTERNAL_API +void mlk_poly_rej_uniform(mlk_poly *entry, uint8_t seed[MLKEM_SYMBYTES + 2]) +__contract__( + requires(memory_no_alias(entry, sizeof(mlk_poly))) + requires(memory_no_alias(seed, MLKEM_SYMBYTES + 2)) + assigns(memory_slice(entry, sizeof(mlk_poly))) + ensures(array_bound(entry->coeffs, 0, MLKEM_N, 0, MLKEM_Q))); + +#endif /* !MLK_SAMPLING_H */ diff --git a/cbits/mlkem/src/symmetric.h b/cbits/mlkem/src/symmetric.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/symmetric.h @@ -0,0 +1,70 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + */ + +#ifndef MLK_SYMMETRIC_H +#define MLK_SYMMETRIC_H + + +#include "cbmc.h" +#include "common.h" +#include MLK_FIPS202_HEADER_FILE +#if !defined(MLK_CONFIG_SERIAL_FIPS202_ONLY) +#include MLK_FIPS202X4_HEADER_FILE +#endif + +/* Macros denoting FIPS 203 specific Hash functions */ + +/* Hash function H, @[FIPS203, Section 4.1, Eq (4.4)] */ +#define mlk_hash_h(OUT, IN, INBYTES) mlk_sha3_256(OUT, IN, INBYTES) + +/* Hash function G, @[FIPS203, Section 4.1, Eq (4.5)] */ +#define mlk_hash_g(OUT, IN, INBYTES) mlk_sha3_512(OUT, IN, INBYTES) + +/* Hash function J, @[FIPS203, Section 4.1, Eq (4.4)] */ +#define mlk_hash_j(OUT, IN, INBYTES) \ + mlk_shake256(OUT, MLKEM_SYMBYTES, IN, INBYTES) + +/* PRF function, @[FIPS203, Section 4.1, Eq (4.3)] + * Referring to (eq 4.3), `OUT` is assumed to contain `s || b`. */ +#define mlk_prf_eta(ETA, OUT, IN) \ + mlk_shake256(OUT, (ETA) * MLKEM_N / 4, IN, MLKEM_SYMBYTES + 1) +#define mlk_prf_eta1(OUT, IN) mlk_prf_eta(MLKEM_ETA1, OUT, IN) +#define mlk_prf_eta2(OUT, IN) mlk_prf_eta(MLKEM_ETA2, OUT, IN) +#define mlk_prf_eta1_x4(OUT, IN) \ + mlk_shake256x4((OUT)[0], (OUT)[1], (OUT)[2], (OUT)[3], \ + (MLKEM_ETA1 * MLKEM_N / 4), (IN)[0], (IN)[1], (IN)[2], \ + (IN)[3], MLKEM_SYMBYTES + 1) + +/* XOF function, FIPS 203 4.1 */ +#define mlk_xof_ctx mlk_shake128ctx +#define mlk_xof_x4_ctx mlk_shake128x4ctx +#define mlk_xof_init(CTX) mlk_shake128_init((CTX)) +#define mlk_xof_absorb(CTX, IN, INBYTES) \ + mlk_shake128_absorb_once((CTX), (IN), (INBYTES)) +#define mlk_xof_squeezeblocks(BUF, NBLOCKS, CTX) \ + mlk_shake128_squeezeblocks((BUF), (NBLOCKS), (CTX)) +#define mlk_xof_release(CTX) mlk_shake128_release((CTX)) + +#define mlk_xof_x4_init(CTX) mlk_shake128x4_init((CTX)) +#define mlk_xof_x4_absorb(CTX, IN, INBYTES) \ + mlk_shake128x4_absorb_once((CTX), (IN)[0], (IN)[1], (IN)[2], (IN)[3], \ + (INBYTES)) +#define mlk_xof_x4_squeezeblocks(BUF, NBLOCKS, CTX) \ + mlk_shake128x4_squeezeblocks((BUF)[0], (BUF)[1], (BUF)[2], (BUF)[3], \ + (NBLOCKS), (CTX)) +#define mlk_xof_x4_release(CTX) mlk_shake128x4_release((CTX)) + +#define MLK_XOF_RATE SHAKE128_RATE + +#endif /* !MLK_SYMMETRIC_H */ diff --git a/cbits/mlkem/src/sys.h b/cbits/mlkem/src/sys.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/sys.h @@ -0,0 +1,320 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#ifndef MLK_SYS_H +#define MLK_SYS_H + +#if !defined(MLK_CONFIG_NO_ASM) && (defined(__GNUC__) || defined(__clang__)) +#define MLK_HAVE_INLINE_ASM +#endif + +/* Try to find endianness, if not forced through CFLAGS already */ +#if !defined(MLK_SYS_LITTLE_ENDIAN) && !defined(MLK_SYS_BIG_ENDIAN) +#if defined(__BYTE_ORDER__) +#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ +#define MLK_SYS_LITTLE_ENDIAN +#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ +#define MLK_SYS_BIG_ENDIAN +#else +#error "__BYTE_ORDER__ defined, but don't recognize value." +#endif +#endif /* __BYTE_ORDER__ */ + +/* MSVC does not define __BYTE_ORDER__. However, MSVC only supports + * little endian x86, x86_64, and AArch64. It is, hence, safe to assume + * little endian. */ +#if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_AMD64) || \ + defined(_M_IX86) || defined(_M_ARM64)) +#define MLK_SYS_LITTLE_ENDIAN +#endif + +#endif /* !MLK_SYS_LITTLE_ENDIAN && !MLK_SYS_BIG_ENDIAN */ + +/* Check if we're running on an AArch64 little endian system. _M_ARM64 is set by + * MSVC. */ +#if defined(__AARCH64EL__) || defined(_M_ARM64) +#define MLK_SYS_AARCH64 +#endif + +/* Check if the AArch64 compilation target supports NEON (Advanced SIMD). + * + * Some compilers also define __ARM_NEON__, but __ARM_NEON is the most reliable + * signal. Specifically, clang on Apple appears to keep __ARM_NEON__ set even if + * -march=armv8-a+nosimd is set. + * + * gcc 4.8 -- the first gcc version introducing Neon support -- sets neither + * __ARM_NEON nor __ARM_NEON__; in fact, there is no preprocessor signal that + * Neon is enabled. If you use gcc 4.8, you should set __ARM_NEON manually. + * gcc 4.9 onwards do set __ARM_NEON. + */ +#if defined(MLK_SYS_AARCH64) && defined(__ARM_NEON) +#define MLK_SYS_AARCH64_NEON +#endif + +/* Check if we're running on an AArch64 big endian system. */ +#if defined(__AARCH64EB__) +#define MLK_SYS_AARCH64_EB +#endif + +/* Check if we're running on an Armv8.1-M system with MVE */ +#if defined(__ARM_ARCH_8_1M_MAIN__) || defined(__ARM_FEATURE_MVE) +#define MLK_SYS_ARMV81M_MVE +#endif + +/* Check if we're running on an x86_64 system. */ +#if defined(__x86_64__) || defined(_M_X64) || defined(_M_AMD64) +#define MLK_SYS_X86_64 +#if defined(__AVX2__) +#define MLK_SYS_X86_64_AVX2 +#endif +#endif /* __x86_64__ || _M_X64 || _M_AMD64 */ + +#if defined(MLK_SYS_LITTLE_ENDIAN) && defined(__powerpc64__) +#define MLK_SYS_PPC64LE +#endif + +#if defined(__riscv) && defined(__riscv_xlen) && __riscv_xlen == 64 +#define MLK_SYS_RISCV64 +#endif + +#if defined(MLK_SYS_RISCV64) && defined(__riscv_vector) && \ + defined(__riscv_v_intrinsic) +#define MLK_SYS_RISCV64_RVV +#endif + +#if defined(__riscv) && defined(__riscv_xlen) && __riscv_xlen == 32 +#define MLK_SYS_RISCV32 +#endif + +#if defined(_WIN32) +#define MLK_SYS_WINDOWS +#endif + +#if defined(__linux__) +#define MLK_SYS_LINUX +#endif + +#if defined(__APPLE__) +#define MLK_SYS_APPLE +#endif + +#if defined(MLK_FORCE_AARCH64) && !defined(MLK_SYS_AARCH64) +#error "MLK_FORCE_AARCH64 is set, but we don't seem to be on an AArch64 system." +#endif + +#if defined(MLK_FORCE_AARCH64_EB) && !defined(MLK_SYS_AARCH64_EB) +#error \ + "MLK_FORCE_AARCH64_EB is set, but we don't seem to be on an AArch64 system." +#endif + +#if defined(MLK_FORCE_X86_64) && !defined(MLK_SYS_X86_64) +#error "MLK_FORCE_X86_64 is set, but we don't seem to be on an X86_64 system." +#endif + +#if defined(MLK_FORCE_PPC64LE) && !defined(MLK_SYS_PPC64LE) +#error "MLK_FORCE_PPC64LE is set, but we don't seem to be on a PPC64LE system." +#endif + +#if defined(MLK_FORCE_RISCV64) && !defined(MLK_SYS_RISCV64) +#error "MLK_FORCE_RISCV64 is set, but we don't seem to be on a RISCV64 system." +#endif + +#if defined(MLK_FORCE_RISCV32) && !defined(MLK_SYS_RISCV32) +#error "MLK_FORCE_RISCV32 is set, but we don't seem to be on a RISCV32 system." +#endif + +/* + * MLK_INLINE: Hint for inlining. + * - MSVC: __inline + * - C99+: inline + * - GCC/Clang C90: __attribute__((unused)) to silence warnings + * - Other C90: empty + */ +#if !defined(MLK_INLINE) +#if defined(_MSC_VER) +#define MLK_INLINE __inline +#elif defined(inline) || \ + (defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L) +#define MLK_INLINE inline +#elif defined(__GNUC__) || defined(__clang__) +#define MLK_INLINE __attribute__((unused)) +#else +#define MLK_INLINE +#endif +#endif /* !MLK_INLINE */ + +/* + * MLK_ALWAYS_INLINE: Force inlining. + * - MSVC: __forceinline + * - GCC/Clang C99+: MLK_INLINE __attribute__((always_inline)) + * - Other: MLK_INLINE (no forced inlining) + */ +#if !defined(MLK_ALWAYS_INLINE) +#if defined(_MSC_VER) +#define MLK_ALWAYS_INLINE __forceinline +#elif (defined(__GNUC__) || defined(__clang__)) && \ + (defined(inline) || \ + (defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L)) +#define MLK_ALWAYS_INLINE MLK_INLINE __attribute__((always_inline)) +#else +#define MLK_ALWAYS_INLINE MLK_INLINE +#endif +#endif /* !MLK_ALWAYS_INLINE */ + +/* + * MLK_NOINLINE: Prevent inlining. + * - MSVC: __declspec(noinline) + * - GCC/Clang: __attribute__((noinline)) + * - Other: empty + */ +#if !defined(MLK_NOINLINE) +#if defined(_MSC_VER) +#define MLK_NOINLINE __declspec(noinline) +#elif defined(__GNUC__) || defined(__clang__) +#define MLK_NOINLINE __attribute__((noinline)) +#else +#define MLK_NOINLINE +#endif +#endif /* !MLK_NOINLINE */ + +#ifndef MLK_STATIC_TESTABLE +#define MLK_STATIC_TESTABLE static +#endif + +/* + * C90 does not have the restrict compiler directive yet. + * We don't use it in C90 builds. + */ +#if !defined(restrict) +#if defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L +#define MLK_RESTRICT restrict +#else +#define MLK_RESTRICT +#endif + +#else /* !restrict */ + +#define MLK_RESTRICT restrict +#endif /* restrict */ + +#define MLK_DEFAULT_ALIGN 32 +#define MLK_ALIGN_UP(N) \ + ((((N) + (MLK_DEFAULT_ALIGN - 1)) / MLK_DEFAULT_ALIGN) * MLK_DEFAULT_ALIGN) +#if defined(__GNUC__) +#define MLK_ALIGN __attribute__((aligned(MLK_DEFAULT_ALIGN))) +#elif defined(_MSC_VER) +#define MLK_ALIGN __declspec(align(MLK_DEFAULT_ALIGN)) +#else +#define MLK_ALIGN /* No known support for alignment constraints */ +#endif + + +/* New X86_64 CPUs support control-flow protection using the CET instructions. + * When enabled (through -fcf-protection=), all compilation units (including + * empty ones) need to support CET for this to work. + * For assembly, this means that source files need to signal support for + * CET by setting the appropriate note.gnu.property section. + * This can be achieved by including the header in all assembly file. + * This file also provides the _CET_ENDBR macro which needs to be placed at + * every potential target of an indirect branch. + * If CET is enabled _CET_ENDBR maps to the endbr64 instruction, otherwise + * it is empty. + * In case the compiler does not support CET (e.g., +#define MLK_CET_ENDBR _CET_ENDBR +#else +#define MLK_CET_ENDBR +#endif +#endif /* MLK_SYS_X86_64 */ + +#if defined(MLK_CONFIG_CT_TESTING_ENABLED) && !defined(__ASSEMBLER__) +#include +#define MLK_CT_TESTING_SECRET(ptr, len) \ + VALGRIND_MAKE_MEM_UNDEFINED((ptr), (len)) +#define MLK_CT_TESTING_DECLASSIFY(ptr, len) \ + VALGRIND_MAKE_MEM_DEFINED((ptr), (len)) +#else /* MLK_CONFIG_CT_TESTING_ENABLED && !__ASSEMBLER__ */ +#define MLK_CT_TESTING_SECRET(ptr, len) \ + do \ + { \ + } while (0) +#define MLK_CT_TESTING_DECLASSIFY(ptr, len) \ + do \ + { \ + } while (0) +#endif /* !(MLK_CONFIG_CT_TESTING_ENABLED && !__ASSEMBLER__) */ + +#if defined(__GNUC__) || defined(__clang__) +#define MLK_MUST_CHECK_RETURN_VALUE __attribute__((warn_unused_result)) +#else +#define MLK_MUST_CHECK_RETURN_VALUE +#endif + +/* The x86_64 assembly backend uses the SysV calling convention. On Windows, + * where the Microsoft x64 calling convention is the default, it can still be + * used with compilers that allow choosing the calling convention per + * function: GCC and Clang support __attribute__((sysv_abi)), which makes + * calls to the annotated function follow the SysV calling convention. + * + * MLK_SYSV_ABI_SUPPORTED signals that the toolchain can call SysV assembly + * routines; the x86_64 assembly backend is only enabled if it is defined. + * MLK_SYSV_ABI is the attribute carried by declarations of x86_64 assembly + * routines. Both macros can be set externally for toolchains offering an + * equivalent mechanism that is not recognized here. */ +#if defined(MLK_SYS_X86_64) && !defined(MLK_SYSV_ABI_SUPPORTED) +#if !defined(MLK_SYS_WINDOWS) || defined(__GNUC__) || defined(__clang__) +#define MLK_SYSV_ABI_SUPPORTED +#endif +#endif + +#if !defined(MLK_SYSV_ABI) +#if defined(MLK_SYS_WINDOWS) && defined(MLK_SYSV_ABI_SUPPORTED) +#define MLK_SYSV_ABI __attribute__((sysv_abi)) +#else +#define MLK_SYSV_ABI +#endif +#endif /* !MLK_SYSV_ABI */ + +#if !defined(__ASSEMBLER__) +/* System capability enumeration */ +typedef enum +{ + /* x86_64 */ + MLK_SYS_CAP_X86_64_AVX2, + /* AArch64 */ + MLK_SYS_CAP_AARCH64_NEON, + MLK_SYS_CAP_AARCH64_SHA3, + /* Armv8.1-M */ + MLK_SYS_CAP_ARMV81M_MVE +} mlk_sys_cap; + +#if !defined(MLK_CONFIG_CUSTOM_CAPABILITY_FUNC) +#include "cbmc.h" + +MLK_MUST_CHECK_RETURN_VALUE +static MLK_INLINE int mlk_sys_check_capability(mlk_sys_cap cap) +__contract__( + ensures(return_value == 0 || return_value == 1) +) +{ + /* By default, we rely on compile-time feature detection/specification: + * If a feature is enabled at compile-time, we assume it is supported by + * the host that the resulting library/binary will be built on. + * If this assumption is not true, you MUST overwrite this function. + * See the documentation of MLK_CONFIG_CUSTOM_CAPABILITY_FUNC in + * mlkem_native_config.h for more information. */ + (void)cap; + return 1; +} +#endif /* !MLK_CONFIG_CUSTOM_CAPABILITY_FUNC */ +#endif /* !__ASSEMBLER__ */ + +#endif /* !MLK_SYS_H */ diff --git a/cbits/mlkem/src/verify.c b/cbits/mlkem/src/verify.c new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/verify.c @@ -0,0 +1,20 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ +#include "verify.h" + +#if !defined(MLK_USE_ASM_VALUE_BARRIER) && \ + !defined(MLK_CONFIG_MULTILEVEL_NO_SHARED) +/* + * Masking value used in constant-time functions from + * verify.h to block the compiler's range analysis and + * thereby reduce the risk of compiler-introduced branches. + */ +volatile uint64_t mlk_ct_opt_blocker_u64 = 0; + +#else /* !MLK_USE_ASM_VALUE_BARRIER && !MLK_CONFIG_MULTILEVEL_NO_SHARED */ + +MLK_EMPTY_CU(verify) + +#endif /* !(!MLK_USE_ASM_VALUE_BARRIER && !MLK_CONFIG_MULTILEVEL_NO_SHARED) */ diff --git a/cbits/mlkem/src/verify.h b/cbits/mlkem/src/verify.h new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/verify.h @@ -0,0 +1,447 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* References + * ========== + * + * - [FIPS203] + * FIPS 203 Module-Lattice-Based Key-Encapsulation Mechanism Standard + * National Institute of Standards and Technology + * https://csrc.nist.gov/pubs/fips/203/final + * + * - [REF] + * CRYSTALS-Kyber C reference implementation + * Bos, Ducas, Kiltz, Lepoint, Lyubashevsky, Schanck, Schwabe, Seiler, Stehlé + * https://github.com/pq-crystals/kyber/tree/main/ref + * + * - [libmceliece] + * libmceliece implementation of Classic McEliece + * Bernstein, Chou + * https://lib.mceliece.org/ + * + * - [optblocker] + * PQC forum post on opt-blockers using volatile globals + * Daniel J. Bernstein + * https://groups.google.com/a/list.nist.gov/g/pqc-forum/c/hqbtIGFKIpU/m/H14H0wOlBgAJ + */ + +#ifndef MLK_VERIFY_H +#define MLK_VERIFY_H + + +#include "cbmc.h" +#include "common.h" + +/* Constant-time comparisons and conditional operations + + We reduce the risk for compilation into variable-time code + through the use of 'value barriers'. + + Functionally, a value barrier is a no-op. To the compiler, however, + it constitutes an arbitrary modification of its input, and therefore + harden's value propagation and range analysis. + + We consider two approaches to implement a value barrier: + - An empty inline asm block which marks the target value as clobbered. + - XOR'ing with the value of a volatile global that's set to 0; + see @[optblocker] for a discussion of this idea, and + @[libmceliece, inttypes/crypto_intN.h] for an implementation. + + The first approach is cheap because it only prevents the compiler + from reasoning about the value of the variable past the barrier, + but does not directly generate additional instructions. + + The second approach generates redundant loads and XOR operations + and therefore comes at a higher runtime cost. However, it appears + more robust towards optimization, as compilers should never drop + a volatile load. + + We use the empty-ASM value barrier for GCC and clang, and fall + back to the global volatile barrier otherwise. + + The global value barrier can be forced by setting + MLK_CONFIG_NO_ASM_VALUE_BARRIER. + +*/ + +#if defined(MLK_HAVE_INLINE_ASM) && !defined(MLK_CONFIG_NO_ASM_VALUE_BARRIER) +#define MLK_USE_ASM_VALUE_BARRIER +#endif + +#if !defined(MLK_USE_ASM_VALUE_BARRIER) + +/* + * Declaration of global volatile that the global value barrier + * is loading from and masking with. + */ +#define mlk_ct_opt_blocker_u64 MLK_NAMESPACE(ct_opt_blocker_u64) +extern volatile uint64_t mlk_ct_opt_blocker_u64; + +/* Helper functions for obtaining global masks of various sizes */ + +/* This contract is not proved but treated as an axiom. + * + * Its validity relies on the assumption that the global opt-blocker + * constant mlk_ct_opt_blocker_u64 is not modified. + */ +static MLK_INLINE uint64_t mlk_ct_get_optblocker_u64(void) +__contract__(ensures(return_value == 0)) { return mlk_ct_opt_blocker_u64; } + +static MLK_INLINE uint8_t mlk_ct_get_optblocker_u8(void) +__contract__(ensures(return_value == 0)) { return (uint8_t)mlk_ct_get_optblocker_u64(); } + +static MLK_INLINE uint32_t mlk_ct_get_optblocker_u32(void) +__contract__(ensures(return_value == 0)) { return (uint32_t)mlk_ct_get_optblocker_u64(); } + +static MLK_INLINE int32_t mlk_ct_get_optblocker_i32(void) +__contract__(ensures(return_value == 0)) { return (int32_t)mlk_ct_get_optblocker_u64(); } + +/* Opt-blocker based implementation of value barriers */ +static MLK_INLINE uint32_t mlk_value_barrier_u32(uint32_t b) +__contract__(ensures(return_value == b)) { return (b ^ mlk_ct_get_optblocker_u32()); } + +static MLK_INLINE int32_t mlk_value_barrier_i32(int32_t b) +__contract__(ensures(return_value == b)) { return (b ^ mlk_ct_get_optblocker_i32()); } + +static MLK_INLINE uint8_t mlk_value_barrier_u8(uint8_t b) +__contract__(ensures(return_value == b)) { return (b ^ mlk_ct_get_optblocker_u8()); } + +#else /* !MLK_USE_ASM_VALUE_BARRIER */ + +static MLK_INLINE uint32_t mlk_value_barrier_u32(uint32_t b) +__contract__(ensures(return_value == b)) +{ + __asm__ volatile("" : "+r"(b)); + return b; +} + +static MLK_INLINE int32_t mlk_value_barrier_i32(int32_t b) +__contract__(ensures(return_value == b)) +{ + __asm__ volatile("" : "+r"(b)); + return b; +} + +static MLK_INLINE uint8_t mlk_value_barrier_u8(uint8_t b) +__contract__(ensures(return_value == b)) +{ + __asm__ volatile("" : "+r"(b)); + return b; +} + +#endif /* MLK_USE_ASM_VALUE_BARRIER */ + +#ifdef CBMC +#pragma CPROVER check push +#pragma CPROVER check disable "conversion" +#endif +/** + * Cast uint16 value to int16. + * + * @param x Input value. + * + * @return For uint16_t x, the unique y in int16_t so that x == y mod 2^16. + * Concretely: + * - x < 32768: returns x + * - x >= 32768: returns x - 65536 + */ +static MLK_ALWAYS_INLINE int16_t mlk_cast_uint16_to_int16(uint16_t x) +{ + /* + * PORTABILITY: This relies on uint16_t -> int16_t + * being implemented as the inverse of int16_t -> uint16_t, + * which is implementation-defined (C99 6.3.1.3 (3)) + * CBMC (correctly) fails to prove this conversion is OK, + * so we have to suppress that check here + */ + return (int16_t)x; +} +#ifdef CBMC +#pragma CPROVER check pop +#endif + +/** + * Cast int32 value to uint16 as per C standard. + * + * @param x Input value. + * + * @return For int32_t x, the unique y in uint16_t so that x == y mod 2^16. + */ +static MLK_ALWAYS_INLINE uint16_t mlk_cast_int32_to_uint16(int32_t x) +{ + return (uint16_t)(x & (int32_t)UINT16_MAX); +} + +/** + * Cast int16 value to uint16 as per C standard. + * + * @param x Input value. + * + * @return For int16_t x, the unique y in uint16_t so that x == y mod 2^16. + */ +static MLK_ALWAYS_INLINE uint16_t mlk_cast_int16_to_uint16(int32_t x) +{ + return mlk_cast_int32_to_uint16(x); +} + +/** + * Return 0 if input is non-negative, and -1 otherwise. + * + * @reference{Embedded in the polynomial compression function in the + * reference implementation @[REF]. Used as part of signed->unsigned + * conversion for modular representatives to detect whether the input is + * negative. This happens in `mlk_poly_reduce()` here, and as part of + * polynomial compression functions in the reference implementation. See + * `mlk_poly_reduce()`. We use value barriers to reduce the risk of + * compiler-introduced branches.} + * + * @param x Value to be converted into a mask. + * + * @return Mask value (0 or 0xFFFF). + */ +static MLK_INLINE uint16_t mlk_ct_cmask_neg_i16(int16_t x) +__contract__(ensures(return_value == ((x < 0) ? 0xFFFF : 0))) +{ + int32_t tmp = mlk_value_barrier_i32((int32_t)x); + /* + * PORTABILITY: Right-shift on a signed integer is + * implementation-defined for negative left argument. + * Here, we assume it's sign-preserving "arithmetic" shift right. + * See (C99 6.5.7 (5)) + */ + tmp >>= 16; + return mlk_cast_int32_to_uint16(tmp); +} + +/** + * Return 0 if input is zero, and -1 otherwise. + * + * @reference{Embedded in `cmov_int16()` in the reference implementation + * @[REF]. Uses a value barrier and shift instead of `b = -b` to convert + * condition into mask.} + * + * @param x Value to be converted into a mask. + * + * @return Mask value (0 or 0xFFFF). + */ +static MLK_INLINE uint16_t mlk_ct_cmask_nonzero_u16(uint16_t x) +__contract__(ensures(return_value == ((x == 0) ? 0 : 0xFFFF))) +{ + int32_t tmp = mlk_value_barrier_i32(-((int32_t)x)); + /* + * PORTABILITY: Right-shift on a signed integer is + * implementation-defined for negative left argument. + * Here, we assume it's sign-preserving "arithmetic" shift right. + * See (C99 6.5.7 (5)) + */ + tmp >>= 16; + return mlk_cast_int32_to_uint16(tmp); +} + +/** + * Return 0 if input is zero, and -1 otherwise. + * + * @reference{Embedded in `verify()` and `cmov()` in the reference + * implementation @[REF]. We include a value barrier not present in the + * reference implementation, to prevent the compiler from realizing that + * this function returns a mask.} + * + * @param x Value to be converted into a mask. + * + * @return Mask value (0 or 0xFF). + */ +static MLK_INLINE uint8_t mlk_ct_cmask_nonzero_u8(uint8_t x) +__contract__(ensures(return_value == ((x == 0) ? 0 : 0xFF))) +{ + uint16_t mask = mlk_ct_cmask_nonzero_u16((uint16_t)x); + return (uint8_t)(mask & 0xFF); +} + +/** + * Functionally equivalent to cond ? a : b, but implemented with guards + * against compiler-introduced branches. + * + * @spec{With `a = MLKEM_Q_HALF` and `b=0`, this essentially implements + * `Decompress_1` @[FIPS203, Eq (4.8)] in `mlk_poly_frommsg()`. With + * `a = x + MLKEM_Q`, `b = x`, and `cond` indicating whether `x` is negative, + * implements signed->unsigned conversion of modular representatives. + * Questions of representation are not considered in the specification + * @[FIPS203, Section 2.4.1, "The pseudocode is agnostic regarding how an + * integer modulo 𝑚 is represented in actual implementations"].} + * + * @reference{Embedded in the polynomial compression function in the + * reference implementation @[REF]. Used as part of signed->unsigned + * conversion for modular representatives. This happens in `mlk_poly_reduce()` + * here, and as part of polynomial compression functions in @[REF]. See + * `mlk_poly_reduce()`. Barrier to reduce the risk of compiler-introduced + * branches. For `a = MLKEM_Q_HALF` and `b=0`, also embedded in + * `poly_frommsg()` from the reference implementation, which uses + * `cmov_int16()` instead.} + * + * @param a First alternative. + * @param b Second alternative. + * @param cond Condition variable. + * + * @return @p a if @p cond != 0, else @p b. + */ +static MLK_INLINE int16_t mlk_ct_sel_int16(int16_t a, int16_t b, uint16_t cond) +__contract__(ensures(return_value == (cond ? a : b))) +{ + uint16_t au = mlk_cast_int16_to_uint16(a); + uint16_t bu = mlk_cast_int16_to_uint16(b); + uint16_t res = bu ^ (mlk_ct_cmask_nonzero_u16(cond) & (au ^ bu)); + return mlk_cast_uint16_to_int16(res); +} + +/** + * Functionally equivalent to cond ? a : b, but implemented with guards + * against compiler-introduced branches. + * + * @reference{Embedded into `cmov()` in the reference implementation @[REF]. + * Uses a value barrier to get mask from condition value.} + * + * @param a First alternative. + * @param b Second alternative. + * @param cond Condition variable. + * + * @return @p a if @p cond != 0, else @p b. + */ +static MLK_INLINE uint8_t mlk_ct_sel_uint8(uint8_t a, uint8_t b, uint8_t cond) +__contract__(ensures(return_value == (cond ? a : b))) +{ + return b ^ (mlk_ct_cmask_nonzero_u8(cond) & (a ^ b)); +} + +/** + * Compare two arrays for equality in constant time. + * + * @spec{Used to securely compute conditional move in @[FIPS203, Algorithm + * 18 (ML-KEM.Decaps_Internal, L9-11].} + * + * @reference{`cmov()` in the reference implementation @[REF]. We return + * `uint8_t`, not `int`. We use an additional XOR-accumulator in the + * comparison loop which prevents early abort if the OR-accumulator is 0xFF. + * We use a value barrier to convert the OR-accumulator into a mask; the + * reference implementation uses a shift which the compiler can argue to + * result in either 0 or 0xFF..FF.} + * + * @param[in] a First byte array. + * @param[in] b Second byte array. + * @param len Length of the byte arrays, upper-bounded to UINT16_MAX to + * control proof complexity only. + * + * @retval 0 The byte arrays are equal. + * @retval 0xFF The byte arrays are not equal. + */ +static MLK_INLINE uint8_t mlk_ct_memcmp(const uint8_t *a, const uint8_t *b, + const size_t len) +__contract__( + requires(len <= UINT16_MAX) + requires(memory_no_alias(a, len)) + requires(memory_no_alias(b, len)) + ensures((return_value == 0) || (return_value == 0xFF)) + ensures((return_value == 0) == forall(i, 0, len, (a[i] == b[i])))) +{ + uint8_t r = 0, s = 0; + unsigned i; + + for (i = 0; i < len; i++) + __loop__( + invariant(i <= len) + invariant((r == 0) == (forall(k, 0, i, (a[k] == b[k])))) + decreases(len - i)) + { + r |= a[i] ^ b[i]; + /* s is useless, but prevents the loop from being aborted once r=0xff. */ + s ^= a[i] ^ b[i]; + } + + /* + * - Convert r into a mask; this may not be necessary, but is an additional + * safeguard + * towards leaking information about a and b. + * - XOR twice with s, separated by a value barrier, to prevent the compile + * from dropping the s computation in the loop. + */ + return (mlk_value_barrier_u8(mlk_ct_cmask_nonzero_u8(r) ^ s) ^ s); +} + +/** + * Copy len bytes from x to r if b is zero; don't modify x if b is non-zero. + * Assumes two's complement representation of negative integers. Runs in + * constant time. + * + * @spec{Used to securely compute conditional move in @[FIPS203, Algorithm + * 18 (ML-KEM.Decaps_Internal, L9-11].} + * + * @reference{`cmov()` in the reference implementation @[REF]. We move if + * condition value is `0`, not `1`. We use `mlk_ct_sel_uint8` for + * constant-time selection.} + * + * @param[out] r Output byte array. + * @param[in] x Input byte array. + * @param len Number of bytes to be copied. + * @param b Condition value. + */ +static MLK_INLINE void mlk_ct_cmov_zero(uint8_t *r, const uint8_t *x, + size_t len, uint8_t b) +__contract__( + requires(len <= UINT32_MAX) + requires(memory_no_alias(r, len)) + requires(memory_no_alias(x, len)) + assigns(memory_slice(r, len)) + ensures(forall(i, 0, len, (r[i] == (b == 0 ? x[i] : old(r)[i]))))) +{ + size_t i; + for (i = 0; i < len; i++) + __loop__( + invariant(i <= len) + invariant(forall(k, 0, i, r[k] == (b == 0 ? x[k] : loop_entry(r)[k]))) + decreases(len - i)) + { + r[i] = mlk_ct_sel_uint8(r[i], x[i], b); + } +} + +/** + * Force-zeroize a buffer. + * + * @spec{Used to implement @[FIPS203, Section 3.3, Destruction of + * intermediate values].} + * + * @reference{Not present in the reference implementation @[REF].} + * + * @param[out] ptr Buffer to be zeroed. + * @param len Number of bytes to be zeroed. + */ +#if !defined(MLK_CONFIG_CUSTOM_ZEROIZE) +#if defined(MLK_SYS_WINDOWS) +#include +#elif !defined(MLK_HAVE_INLINE_ASM) +#error No plausibly-secure implementation of mlk_zeroize available. Please provide your own using MLK_CONFIG_CUSTOM_ZEROIZE. +#endif + +static MLK_INLINE void mlk_zeroize(void *ptr, size_t len) +__contract__( + requires(len <= UINT32_MAX) + requires(memory_no_alias(ptr, len)) + assigns(memory_slice(ptr, len)) + ensures(array_zeroized_u8((uint8_t *)ptr, len))) +{ +#if defined(MLK_SYS_WINDOWS) + SecureZeroMemory(ptr, len); +#else + mlk_memset(ptr, 0, len); + /* This follows OpenSSL and seems sufficient to prevent the compiler + * from optimizing away the memset. + * + * If there was a reliable way to detect availability of memset_s(), + * that would be preferred. */ + __asm__ volatile("" : : "r"(ptr) : "memory"); +#endif /* !MLK_SYS_WINDOWS */ +} +#endif /* !MLK_CONFIG_CUSTOM_ZEROIZE */ + +#endif /* !MLK_VERIFY_H */ diff --git a/cbits/mlkem/src/zetas.inc b/cbits/mlkem/src/zetas.inc new file mode 100644 --- /dev/null +++ b/cbits/mlkem/src/zetas.inc @@ -0,0 +1,30 @@ +/* + * Copyright (c) The mlkem-native project authors + * SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT + */ + +/* + * WARNING: This file is auto-generated from scripts/autogen + * in the mlkem-native repository. + * Do not modify it directly. + */ + + +/* + * Table of zeta values used in the reference NTT and inverse NTT. + * See autogen for details. + */ +static MLK_ALIGN const int16_t mlk_zetas[128] = { + -1044, -758, -359, -1517, 1493, 1422, 287, 202, -171, 622, 1577, + 182, 962, -1202, -1474, 1468, 573, -1325, 264, 383, -829, 1458, + -1602, -130, -681, 1017, 732, 608, -1542, 411, -205, -1571, 1223, + 652, -552, 1015, -1293, 1491, -282, -1544, 516, -8, -320, -666, + -1618, -1162, 126, 1469, -853, -90, -271, 830, 107, -1421, -247, + -951, -398, 961, -1508, -725, 448, -1065, 677, -1275, -1103, 430, + 555, 843, -1251, 871, 1550, 105, 422, 587, 177, -235, -291, + -460, 1574, 1653, -246, 778, 1159, -147, -777, 1483, -602, 1119, + -1590, 644, -872, 349, 418, 329, -156, -75, 817, 1097, 603, + 610, 1322, -1285, -1465, 384, -1215, -136, 1218, -1335, -874, 220, + -1187, -1659, -1185, -1530, -1278, 794, -1510, -854, -870, 478, -108, + -308, 996, 991, 958, -1460, 1522, 1628, +}; diff --git a/cbits/p256/gen_base_table.py b/cbits/p256/gen_base_table.py new file mode 100644 --- /dev/null +++ b/cbits/p256/gen_base_table.py @@ -0,0 +1,126 @@ +#!/usr/bin/env python3 +"""Build the two tables of multiples of the base point that crypton reads. + + ./gen_base_table.py 5 p256_base_table.c # for p256_scalarmulbase + ./gen_base_table.py odd 7 p256_wnaf_table.c # for p256_verify.c + +Its own words for the layout: for each i, j with blocksize*i <= 256 and +1 <= j <= B, where B = 2^(blocksize-1), the multiple 2^(blocksize*i) * j * P +goes at tab + 64*(B*i + (j-1)), as a Montgomery-affine pair, four +little-endian 64-bit words each. + +Five is the blocksize crypton ships: on an Apple M4 it is 6.40 microseconds +against 7.15 for four and 6.10 for six, and the table is 52 KiB against 33 +and 88. Nothing outside this file knows the number -- it is written into +the generated file and read from there. + +The curve constants come from `openssl ecparam`, not from memory, and the +generated table is checked against crypton's own base-point multiplication +by the test suite, so a mistake here does not pass quietly. +""" +import subprocess, re, sys + +def curve(): + out = subprocess.run(["openssl","ecparam","-name","prime256v1", + "-param_enc","explicit","-text","-noout"], + capture_output=True, text=True).stdout + f, cur = {}, None + for line in out.splitlines(): + m = re.match(r'^(Prime|A|B|Generator \(uncompressed\)|Order):?\s*$', line.strip()) + if m: + cur = m.group(1); f[cur] = ""; continue + if cur and re.match(r'^\s+[0-9a-f]{2}[:0-9a-f]*:?\s*$', line): + f[cur] += line.strip() + elif cur and line and not line.startswith(' '): + cur = None + def num(s): + return int.from_bytes(bytes(int(v,16) for v in s.strip(':').split(':') if v), 'big') + g = bytes(int(v,16) for v in f['Generator (uncompressed)'].strip(':').split(':') if v) + assert g[0] == 4 and len(g) == 65 + return (num(f['Prime']), num(f['A']), num(f['B']), + int.from_bytes(g[1:33],'big'), int.from_bytes(g[33:],'big'), num(f['Order'])) + +P, A, B, GX, GY, N = curve() +assert (GY*GY - GX**3 - A*GX - B) % P == 0, "the generator is not on the curve" + +def add(p, q): + if p is None: return q + if q is None: return p + (x1,y1),(x2,y2) = p,q + if x1 == x2: + if (y1 + y2) % P == 0: return None + l = (3*x1*x1 + A) * pow(2*y1, -1, P) % P + else: + l = (y2-y1) * pow(x2-x1, -1, P) % P + x3 = (l*l - x1 - x2) % P + return (x3, (l*(x1-x3) - y1) % P) + +R = 1 << 256 +def mont(v): return v * R % P +def words(v): return [(v >> (64*k)) & 0xFFFFFFFFFFFFFFFF for k in range(4)] + +def table(blocksize): + Bn = 1 << (blocksize - 1) + blocks = 256 // blocksize + 1 + out = [] + base = (GX, GY) # 2^(blocksize*i) * P + for i in range(blocks): + acc = None + for j in range(1, Bn + 1): + acc = add(acc, base) # j * base + out.append(acc) + for _ in range(blocksize): + base = add(base, base) + return blocks, Bn, out + +def odd_table(width): + """The odd multiples 1P, 3P, ..., (2^(width-1)-1)P, which is what the + windowed form in cbits/p256/p256_verify.c reads for the base point.""" + n = 1 << (width - 2) + twice = add((GX, GY), (GX, GY)) + out, acc = [], (GX, GY) + for _ in range(n): + out.append(acc) + acc = add(acc, twice) + return out + +def emit_odd(width, path): + pts = odd_table(width) + with open(path, 'w') as fh: + fh.write("/* Generated by cbits/p256/gen_base_table.py; do not hand-edit.\n" + " * The odd multiples of the base point, in Montgomery-affine\n" + " * form, which cbits/p256/p256_verify.c reads. A verification\n" + " * is public, so this table is walked in variable time. */\n") + fh.write("#include \n\n") + fh.write(f"const uint64_t crypton_p256_wnaf_width = {width};\n\n") + fh.write(f"const uint64_t crypton_p256_wnaf_table[{len(pts)*8}] = {{\n") + for (x, y) in pts: + ws = words(mont(x)) + words(mont(y)) + fh.write("\t" + ",".join(f"0x{w:016x}ULL" for w in ws) + ",\n") + fh.write("};\n") + return len(pts), len(pts) * 64 + +def emit(blocksize, path): + blocks, Bn, pts = table(blocksize) + with open(path, 'w') as fh: + fh.write("/* Generated by cbits/p256/gen_base_table.py; do not hand-edit.\n" + " * The multiples of the base point that s2n-bignum's\n" + " * p256_scalarmulbase reads, in Montgomery-affine form. */\n") + fh.write("#include \n\n") + fh.write(f"const uint64_t crypton_p256_s2n_base_blocksize = {blocksize};\n\n") + fh.write(f"const uint64_t crypton_p256_s2n_base_table[{len(pts)*8}] = {{\n") + for (x, y) in pts: + ws = words(mont(x)) + words(mont(y)) + fh.write("\t" + ",".join(f"0x{w:016x}ULL" for w in ws) + ",\n") + fh.write("};\n") + return blocks, Bn, len(pts)*64 + +if __name__ == "__main__": + if sys.argv[1] == "odd": + w = int(sys.argv[2]); path = sys.argv[3] + n, size = emit_odd(w, path) + print(f"width {w}: {n} odd multiples = {size} bytes") + else: + b = int(sys.argv[1]); path = sys.argv[2] + blocks, Bn, size = emit(b, path) + print(f"blocksize {b}: {blocks} blocks x {Bn} = {size} bytes") diff --git a/cbits/p256/p256.c b/cbits/p256/p256.c --- a/cbits/p256/p256.c +++ b/cbits/p256/p256.c @@ -39,6 +39,15 @@ #include "p256/p256.h" +#ifdef CRYPTON_S2N_BIGNUM +extern void bignum_modinv(uint64_t k, uint64_t *z, const uint64_t *a, + const uint64_t *b, uint64_t *t); +/* the digits are handed over as they lie */ +#if P256_BITSPERDIGIT != 64 || P256_NDIGITS != 4 +#error "CRYPTON_S2N_BIGNUM wants the 64-bit crypton_p256_int" +#endif +#endif + const crypton_p256_int crypton_SECP256r1_n = // curve order {{P256_LITERAL(0xfc632551, 0xf3b9cac2), P256_LITERAL(0xa7179e84, 0xbce6faad), P256_LITERAL(-1, -1), P256_LITERAL(0, -1)}}; @@ -214,7 +223,11 @@ n %= P256_BITSPERDIGIT; for (i = P256_NDIGITS - 1; i > 0; --i) { crypton_p256_digit accu = (P256_DIGIT(a, i) << n); - accu |= (P256_DIGIT(a, i - 1) >> (P256_BITSPERDIGIT - n)); + /* n is zero as often as it is anything else, and a digit shifted by its + * own width is undefined: x86 takes the count modulo the width and hands + * back the whole digit, ARM hands back nothing. Two shifts that are each + * inside the width say the intended nothing on both. */ + accu |= (P256_DIGIT(a, i - 1) >> (P256_BITSPERDIGIT - 1 - n) >> 1); P256_DIGIT(b, i) = accu; } P256_DIGIT(b, i) = (P256_DIGIT(a, i) << n); @@ -230,7 +243,8 @@ n %= P256_BITSPERDIGIT; for (i = 0; i < P256_NDIGITS - 1; ++i) { crypton_p256_digit accu = (P256_DIGIT(a, i) >> n); - accu |= (P256_DIGIT(a, i + 1) << (P256_BITSPERDIGIT - n)); + /* the same full-width shift as in crypton_p256_shl above */ + accu |= (P256_DIGIT(a, i + 1) << (P256_BITSPERDIGIT - 1 - n) << 1); P256_DIGIT(b, i) = accu; } P256_DIGIT(b, i) = (P256_DIGIT(a, i) >> n); @@ -244,8 +258,11 @@ accu |= (P256_DIGIT(a, i + 1) << (P256_BITSPERDIGIT - 1)); P256_DIGIT(b, i) = accu; } + /* The shift is unsigned: highbit is a carry, zero or one, and one shifted + * into the sign bit of the signed digit is an overflow the standard leaves + * undefined. The value put into b is the same either way. */ P256_DIGIT(b, i) = (P256_DIGIT(a, i) >> 1) | - (((crypton_p256_sdigit) highbit) << (P256_BITSPERDIGIT - 1)); + (((crypton_p256_digit) highbit) << (P256_BITSPERDIGIT - 1)); } // Return -1, 0, 1 for a < b, a == b or a > b respectively. @@ -312,6 +329,22 @@ crypton_p256_int U = *MOD; crypton_p256_int V = *a; + /* Zero has no inverse, and the loop below never finds that out: V stays + even forever, so it is halved forever, and the only break is in the + branch both U and V have to be odd to reach. The other input without an + inverse is MOD itself -- 2*MOD does not fit in 256 bits, so there is no + third -- and that one already leaves here as zero, which is also what + Crypto.PubKey.ECC.P256's scalarInvSafe answers for both. Answer the + same for zero rather than not answering. + + Reachable: Crypto.PubKey.ECC.P256 exports scalarInv, and scalarFromBinary + accepts any 256 bits. A hang inside a foreign call cannot be interrupted + by System.Timeout either. */ + if (crypton_p256_is_zero(a)) { + crypton_p256_clear(b); + return; + } + for (;;) { if (crypton_p256_is_even(&U)) { crypton_p256_shr1(&U, 0, &U); @@ -480,6 +513,16 @@ // b = 1/a mod n, using Fermat's little theorem. void crypton_p256e_scalar_invert(const crypton_p256_int* a, crypton_p256_int* b) { +#ifdef CRYPTON_S2N_BIGNUM + /* The assembly, which takes a fixed number of division steps instead: 0.80 + * microseconds against 6.02 on an Apple M4. It answers zero where a has no + * inverse, which is what the chain below does as well, and the caller reads + * a zero as "no inverse". */ + uint64_t t[12]; + + bignum_modinv(4, P256_DIGITS(b), P256_DIGITS(a), + P256_DIGITS(&crypton_SECP256r1_n), t); +#else crypton_p256_int _1, _10, _11, _101, _111, _1010, _1111; crypton_p256_int _10101, _101010, _101111, x6, x8, x16, x32; int i; @@ -534,4 +577,5 @@ // Demontgomerize crypton_p256e_montmul(b, &crypton_SECP256r1_one, b); +#endif } diff --git a/cbits/p256/p256_base_table.c b/cbits/p256/p256_base_table.c new file mode 100644 --- /dev/null +++ b/cbits/p256/p256_base_table.c @@ -0,0 +1,841 @@ +/* Generated by cbits/p256/gen_base_table.py; do not hand-edit. + * The multiples of the base point that s2n-bignum's + * p256_scalarmulbase reads, in Montgomery-affine form. */ +#include + +const uint64_t crypton_p256_s2n_base_blocksize = 5; + +const uint64_t crypton_p256_s2n_base_table[6656] = { + 0x79e730d418a9143cULL,0x75ba95fc5fedb601ULL,0x79fb732b77622510ULL,0x18905f76a53755c6ULL,0xddf25357ce95560aULL,0x8b4ab8e4ba19e45cULL,0xd2e88688dd21f325ULL,0x8571ff1825885d85ULL, + 0x850046d410ddd64dULL,0xaa6ae3c1a433827dULL,0x732205038d1490d9ULL,0xf6bb32e43dcf3a3bULL,0x2f3648d361bee1a5ULL,0x152cd7cbeb236ff8ULL,0x19a8fb0e92042dbeULL,0x78c577510a5b8a3bULL, + 0xffac3f904eebc127ULL,0xb027f84a087d81fbULL,0x66ad77dd87cbbc98ULL,0x26936a3fb6ff747eULL,0xb04c5c1fc983a7ebULL,0x583e47ad0861fe1aULL,0x788208311a2ee98eULL,0xd5f06a29e587cc07ULL, + 0x74b0b50d46918dccULL,0x4650a6edc623c173ULL,0x0cdaacace8100af2ULL,0x577362f541b0176bULL,0x2d96f24ce4cbaba6ULL,0x17628471fad6f447ULL,0x6b6c36dee5ddd22eULL,0x84b14c394c5ab863ULL, + 0xbe1b8aaec45c61f5ULL,0x90ec649a94b9537dULL,0x941cb5aad076c20cULL,0xc9079605890523c8ULL,0xeb309b4ae7ba4f10ULL,0x73c568efe5eb882bULL,0x3540a9877e7a1f68ULL,0x73a076bb2dd1e916ULL, + 0x403947373e77664aULL,0x55ae744f346cee3eULL,0xd50a961a5b17a3adULL,0x13074b5954213673ULL,0x93d36220d377e44bULL,0x299c2b53adff14b5ULL,0xf424d44cef639f11ULL,0xa4c9916d4a07f75fULL, + 0x0746354ea0173b4fULL,0x2bd20213d23c00f7ULL,0xf43eaab50c23bb08ULL,0x13ba5119c3123e03ULL,0x2847d0303f5b9d4dULL,0x6742f2f25da67bddULL,0xef933bdc77c94195ULL,0xeaedd9156e240867ULL, + 0x27f14cd19499a78fULL,0x462ab5c56f9b3455ULL,0x8f90f02af02cfc6bULL,0xb763891eb265230dULL,0xf59da3a9532d4977ULL,0x21e3327dcf9eba15ULL,0x123c7b84be60bbf0ULL,0x56ec12f27706df76ULL, + 0x75c96e8f264e20e8ULL,0xabe6bfed59a7a841ULL,0x2cc09c0444c8eb00ULL,0xe05b3080f0c4e16bULL,0x1eb7777aa45f3314ULL,0x56af7bedce5d45e3ULL,0x2b6e019a88b12f1aULL,0x086659cdfd835f9bULL, + 0x2c18dbd19dc21ec8ULL,0x98f9868a0fcf8139ULL,0x737d2cd648250b49ULL,0xcc61c94724b3428fULL,0x0c2b407880dd9e76ULL,0xc43a8991383fbe08ULL,0x5f7d2d65779be5d2ULL,0x78719a54eb3b4ab5ULL, + 0xea7d260a6245e404ULL,0x9de407956e7fdfe0ULL,0x1ff3a4158dac1ab5ULL,0x3e7090f1649c9073ULL,0x1a7685612b944e88ULL,0x250f939ee57f61c8ULL,0x0c0daa891ead643dULL,0x68930023e125b88eULL, + 0x04b71aa7d2697768ULL,0xabdedef5ca345a33ULL,0x2409d29dee37385eULL,0x4ee1df77cb83e156ULL,0x0cac12d91cbb5b43ULL,0x170ed2f6ca895637ULL,0x28228cfa8ade6d66ULL,0x7ff57c9553238acaULL, + 0xccc425634b2ed709ULL,0x0e356769856fd30dULL,0xbcbcd43f559e9811ULL,0x738477ac5395b759ULL,0x35752b90c00ee17fULL,0x68748390742ed2e3ULL,0x7cd06422bd1f5bc1ULL,0xfbc08769c9e7b797ULL, + 0xa242a35bb0cf664aULL,0x126e48f77f9707e3ULL,0x1717bf54c6832660ULL,0xfaae7332fd12c72eULL,0x27b52db7995d586bULL,0xbe29569e832237c2ULL,0xe8e4193e2a65e7dbULL,0x152706dc2eaa1bbbULL, + 0x72bcd8b7bc60055bULL,0x03cc23ee56e27e4bULL,0xee337424e4819370ULL,0xe2aa0e430ad3da09ULL,0x40b8524f6383c45dULL,0xd766355442a41b25ULL,0x64efa6de778a4797ULL,0x2042170a7079adf4ULL, + 0x808b0b650bc6fb80ULL,0x5882e0753ffe2e6bULL,0xd5ef2f7c2c83f549ULL,0x54d63c809103b723ULL,0xf2f11bd652a23f9bULL,0x3670c3194b0b6587ULL,0x55c4623bb1580e9eULL,0x64edf7b201efe220ULL, + 0xd8c5fccfc5e3a3d8ULL,0xbefd904c4079dfbfULL,0xbc6d6a58fead0197ULL,0x39227077695532a4ULL,0x09e23e6ddbef42f5ULL,0x7e449b64480a9908ULL,0x7b969c1aad9a2e40ULL,0x6231d7929591c2a4ULL, + 0xdb6d96f305968b80ULL,0x380a0913089f73b9ULL,0x7da70b83c2c61e01ULL,0x95fb8394569b38c7ULL,0x9a3c651280edfe2fULL,0x8f726bb98faeaf82ULL,0x8010a4a078424bf8ULL,0x296720440e844970ULL, + 0x802b8d2333e12b70ULL,0x6d490a4b19dd329bULL,0x14f356cc6abc354dULL,0x11eddf7fd0a0da0dULL,0x1e208328d87fd1d8ULL,0xfd2f4f8cfd025813ULL,0x03b48cc47c29bca2ULL,0x3f2a78b3241a2b71ULL, + 0x63c5cb817a2ad62aULL,0x7ef2b6b9ac62ff54ULL,0x3749bba4b3ad9db5ULL,0xad311f2c46d5a617ULL,0xb77a8087c2ff3b6dULL,0xb46feaf3367834ffULL,0xf8aa266d75d6b138ULL,0xfa38d320ec008188ULL, + 0xc04afa1ae3451a09ULL,0x7cc69103bc117423ULL,0x876be3aa51cf56eeULL,0xe7577d57ad844a25ULL,0x266fed8cdb77f341ULL,0xcfa258dc23ae4a2aULL,0x53a7a98cda782760ULL,0x04b48868ceaf7d4aULL, + 0xc0f2affc4e6916c6ULL,0x6fb94957811842daULL,0x6034bcb624b4d157ULL,0xde2efdc7992efb90ULL,0xd66f7eceac793c87ULL,0x02f026267dc6fdcdULL,0x90d3235c9aa1c501ULL,0xf6e494962b4666f0ULL, + 0x082736d19c0859c3ULL,0x89ea5516b269386aULL,0xf25071871aa87b33ULL,0xe9d82f5f704e8236ULL,0x7834612442e855f5ULL,0x209f50fe395e00d8ULL,0xcd9e03aae6e7e62bULL,0xb4b4959e5e5be37bULL, + 0x486d8ffa696946fcULL,0x50fbc6d8b9cba56dULL,0x7e3d423e90f35a15ULL,0x7c3da195c0dd962cULL,0xe673fdb03cfd5d8bULL,0x0704b7c2889dfca5ULL,0xf6ce581ff52305aaULL,0x399d49eb914d5e53ULL, + 0x7966afbb10e6d950ULL,0x37e4a4c4e2bf970aULL,0x23d0c8559d54ca2aULL,0x13d62865fee39a10ULL,0x15f53c38d3bd15e9ULL,0x014b8bed84a80bccULL,0x10674c77bfd8f608ULL,0x4dfab986c93fbfefULL, + 0x24c97c367b92d453ULL,0xd2c271a2249a26c0ULL,0x60eb4b2b89d14a39ULL,0x1198de20432e8005ULL,0x9eabea75799b80d9ULL,0xab6e0c2b8f826ae5ULL,0xca004eedd10061ecULL,0x31a9f439e99c4fd8ULL, + 0x734c8b75b5498a7cULL,0xaeccad8a29f64c2dULL,0x95dd54fa295b1677ULL,0x383902a0b4c54968ULL,0x78cb276feb9d33a9ULL,0x00aedca1af552869ULL,0xa01d14594c5c1630ULL,0xfeba17067cf7d50dULL, + 0x380a496d6ec293cdULL,0x733dbda78e7051f5ULL,0x037e388db849140aULL,0xee4b32b05946dbf6ULL,0xb1c4fda9cae368d1ULL,0x5001a7b0fdb0b2f3ULL,0x6df593742e3ac46eULL,0x4af675f239b3e656ULL, + 0x1a1fffdcc01b0a46ULL,0x07ad675f83f843c2ULL,0xbcec2d076738e81aULL,0x910aec75aa8b7da8ULL,0x13b4d740a4509ba7ULL,0x057010734c7b8216ULL,0x3d75c8f71591f1e5ULL,0x134c1b6f0dfe1d90ULL, + 0xb20e7c44d67826ebULL,0x1212d3cfac379637ULL,0x614f67877de03a5dULL,0x7538a2fc802baa26ULL,0x133c37a19d252415ULL,0x7db390506eb4b587ULL,0x5d40d7574c49d1d9ULL,0xf1126b99a801c4baULL, + 0x39a7aeb9c56729fcULL,0x21a59448e7a8bd85ULL,0xd7c6da5d049d10c6ULL,0x93a4c4a8c5197afbULL,0xd185539c25933861ULL,0x4994bcba27494d9dULL,0xef033de14ce7bfa9ULL,0x22e9b000321f9236ULL, + 0x44e3811039949296ULL,0x5b63827b361db1b5ULL,0x3e5323ed206eaff5ULL,0x942370d2c21f4290ULL,0xf2caaf2ee0d985a1ULL,0x192cc64b7239846dULL,0x7c0b8f47ae6312f8ULL,0x7dc61f9196620108ULL, + 0x4f7081e144cc3addULL,0xd5ffa1d687be82cfULL,0x89890b6c0edd6472ULL,0xada26e1a3ed17863ULL,0x276f271563483caaULL,0xe6924cd92f6077fdULL,0x05a7fe980a466e3cULL,0xf1c794b0b1902d1fULL, + 0x3d2b24b9eb7926b8ULL,0xbff88cb3cdbe5509ULL,0xd0f399afe4dd640bULL,0x3c5fe1302f76ed45ULL,0x6f3562f43764fb3dULL,0x7b5af3183151b62dULL,0xd5bd0bc7d79ce5f3ULL,0xfdaf6b20ec66890fULL, + 0x32027fe891e5d7d3ULL,0xf14b7d1773a07678ULL,0xf88497b3c0dfdd61ULL,0xf7c2eec02a8c4f48ULL,0xaa5573f43756e621ULL,0xc013a2401825b948ULL,0x1c03b34563878572ULL,0xa0472bea653a4184ULL, + 0x6772b0e5ab4b35a2ULL,0x1d8b6001f5eeaacfULL,0x728f7ce4795b9580ULL,0x4a20ed2a41fb81daULL,0x9f685cd44fec01e6ULL,0x3ed7ddcca7ff50adULL,0x460fd2640c2d97fdULL,0x3a241426eb82f4f9ULL, + 0x29ae2cf983dfedc9ULL,0xf84371348d87631aULL,0xaf5717117429c8d2ULL,0x18d15867146d9272ULL,0x83053ecf69769bb7ULL,0xc55eb856c479ab82ULL,0x5ef7791c21b0f4b2ULL,0xaa5956ba3d491525ULL, + 0x84cfbfa1c5c5ea50ULL,0xd3baf14c67960681ULL,0x263984030dd50942ULL,0xe4b7839c4716a663ULL,0xd5f1f794e7de6dc0ULL,0x5cd0f4d4622aa7ceULL,0x5295f3f159acfeecULL,0x8d933552953e0607ULL, + 0xe652533b3cef0d7dULL,0xd94f7b182bbb4381ULL,0x838752be0e80f500ULL,0x8e6e24889e9c9bfbULL,0xc975169716caca6aULL,0x866c49d838531ad9ULL,0xc917e2397151ade1ULL,0x2d016ec16037c407ULL, + 0x80009862d5d721d5ULL,0x0c3357a35bd3a182ULL,0x27f3a83b7aa2cda4ULL,0xb58ae74ef6f83085ULL,0x2a911a812e6dad6bULL,0xde286051f43d6c5bULL,0x4bdccc41f996c4d8ULL,0xe7312ec00ae1e24eULL, + 0x903f6e3960e913afULL,0xb2b58bee98bf140dULL,0x9deff025354890b8ULL,0x155810068d2e924eULL,0xb5755db493c95e5bULL,0x3fac42f0dae20eb8ULL,0x9377c8c109b6d8e0ULL,0xa43e2b46ab47ceffULL, + 0x5f57b2fbfacfa459ULL,0x874b1498c1b5aa6bULL,0xb9e89acac4db2092ULL,0x1362bf8ddf4381daULL,0x25d76830b76328a0ULL,0x38188b7098572ae4ULL,0xb43e941429132f7dULL,0x7895a29f22dd42c9ULL, + 0xcbde78dd5e22cbb2ULL,0xf449c85b76bb4391ULL,0x4289f357b6a4273bULL,0x9fce23fd48e84a19ULL,0xcfc32730939eb3b4ULL,0x8b3d982c16c32280ULL,0x5ac234bad5f1346cULL,0x781954b470769fc9ULL, + 0x6faf68feaae6ee70ULL,0x78f4cc155602b0c9ULL,0x7e3321a86e94052aULL,0x2fb3a0d6734d5d80ULL,0xf3b98f3bb25a43baULL,0x30bf803119ee2951ULL,0x7ffee43321b0612aULL,0x12f775e42eb821d0ULL, + 0x4fdff805f57209b5ULL,0x9bd65ac3f952ac8dULL,0x02a3abd3c7969a6fULL,0x1359927ef523775fULL,0xe09b463f88d2e861ULL,0x661d2199623287c3ULL,0x821e64495a70eb7aULL,0x0afbbb1dd67dc684ULL, + 0x7418e3d3acff89f9ULL,0x227f16aed852251fULL,0xdd5bc6e4eb84658bULL,0xf066b9c8f90a9f7eULL,0xc2369071800a7f87ULL,0x383ddc0d5a72862aULL,0x5b48465d8a776da5ULL,0x3d82f64f5e2d8318ULL, + 0x5852104b87453b28ULL,0x073e8128b387344dULL,0x300e78e4817cfc08ULL,0x3a82ed4799362088ULL,0xe222304c88de46a4ULL,0x666c94fd57fadf4aULL,0x40b2d08ea0c8e108ULL,0x4b2955b909e050faULL, + 0xf8d112e76e6485b3ULL,0x4d3e24db771c52f8ULL,0x48e3ee41684a2f6dULL,0x7161957d21d95551ULL,0x19631283cdb12a6cULL,0xbf3fa8822e50e164ULL,0xf6254b633166cc73ULL,0x3aefa7aeaee8cc38ULL, + 0x79b0fe623b36f9fdULL,0x26543b23fde19fc0ULL,0x136e64a0958482efULL,0x23f637719b095825ULL,0x14cfd596b6a1142eULL,0x5ea6aac6335aac0bULL,0x86a0e8bdf3081dd5ULL,0x5fb89d79003dc12aULL, + 0x0f0165fce3779ee3ULL,0xe00e7f9dbd495d9eULL,0x1fa4efa220284e7aULL,0x4564bade47ac6219ULL,0x90e6312ac4708e8eULL,0x4f5725fba71e9adfULL,0xe95f55ae3d684b9fULL,0x47f7ccb11e94b415ULL, + 0xda3a77e5522e6b69ULL,0x69c908c3bbcd6c18ULL,0x1f1b9e48d924fd56ULL,0x37c64e36aa4bb3f7ULL,0x5a4fdbdfee478d7dULL,0xba75c8bc0193f7a0ULL,0x84bc1e8456cd16dfULL,0x1fb08f0846fad151ULL, + 0x3617890361a341c1ULL,0x3604dc600cfd6142ULL,0x022295eb8533316cULL,0x3dbde4ac44af2922ULL,0x898afc5d1c7eef69ULL,0x58896805d14f4fa1ULL,0x05002160203c21caULL,0x6f0d1f3040ef730bULL, + 0x48201b4b12cfe297ULL,0x3eee129c292f74e5ULL,0xe1fe114ec9e874e8ULL,0x899b055c92c5fc41ULL,0x4e477a643a39c8cfULL,0x82f09efe78963cc9ULL,0x6fd3fd8fd333f863ULL,0x85132b2adc949c63ULL, + 0xbd9b8b1dbe7a2af3ULL,0xec51caa94fb74a72ULL,0xb9937a4b63879697ULL,0x7c9a9d20ec2687d5ULL,0x1773e44f6ef5f014ULL,0x8abcf412e90c6900ULL,0x387bd0228142161eULL,0x50393755fcb6ff2aULL, + 0x766e072232398baaULL,0x205fee425cfca031ULL,0xa49f53417a029cf2ULL,0xa88c68b84023890dULL,0xbc2750417337aaa8ULL,0x9ed364ad0eb384f4ULL,0xe0816f8529aba92fULL,0x2e9e194104e38a88ULL, + 0xfabf770977f7195aULL,0x8ec86167adeb838fULL,0xea1285a8bb4f012dULL,0xd68835039a3eab3fULL,0xee5d24f8309004c2ULL,0xa96e4b7613ffe95eULL,0x0cdffe12bd223ea4ULL,0x8f5c2ee5b6739a53ULL, + 0xecace1dda1887395ULL,0x40960f36932a65deULL,0x9611ff5c3aa95529ULL,0xc58215b07c1e5a36ULL,0xd48c9b58f0e1a524ULL,0xb406856bf590dfb8ULL,0xc7605e049cd95662ULL,0x0dd036eea33ecf82ULL, + 0x3d61333959145a65ULL,0xcd9bc368fa406337ULL,0x82d11be32d8a52a0ULL,0xf6877b2797a1c590ULL,0x837a819bf5cbdb25ULL,0x2a4fd1d8de090249ULL,0x622a7de774990e5fULL,0x840fa5a07945511bULL, + 0xfe2893277946d3f9ULL,0xe132bd2407472273ULL,0xeeeb510c1eb6ae86ULL,0x777708c5f0595067ULL,0x18e2c8cd1297029eULL,0x2c61095cbbf9305eULL,0xe466c2586b85d6d9ULL,0x8ac06c36da1ea530ULL, + 0xe58e90b36b0cf82eULL,0x6438d2462615b5e7ULL,0x07b1f8fc669c145aULL,0xb0d8b2da36f1e1cbULL,0x54d5dadbd9184c4dULL,0x3dbb18d5f93d9976ULL,0x0a3e0f56d1147d47ULL,0x2afa8c8da0a48609ULL, + 0x871239ad653ae326ULL,0x14bcf72aa74cbb43ULL,0x8737650e20d4c083ULL,0x3df86536110ed4afULL,0xd2d86fe7b53ca555ULL,0x688cb00dabd5d538ULL,0xcf81bda31ad38468ULL,0x7ccfe3ccf01167b6ULL, + 0x26e08c07e3533d77ULL,0xd7222e6a2e341c99ULL,0x9d60ec3d8d2dc4edULL,0xbdfe0d8f7c476cf8ULL,0x1fe59ab61d056605ULL,0xa9ea9df686a8551fULL,0x8489941e47fb8d8cULL,0xfeb874eb4a7f1b10ULL, + 0x61fc181060a71676ULL,0xe852d1a8f66a8ad1ULL,0x172bbd656417231eULL,0x0d6de7bd3babb11fULL,0x6fde6f88c8e347f8ULL,0x1c5875479bd99cc3ULL,0x78e54ed034076950ULL,0x97f0f334796e83baULL, + 0xed406aa9bd763802ULL,0xc21486a065303da1ULL,0x61ae291ec7e62ec4ULL,0x622a0492df99333eULL,0x7fd80c9dbb7a8ee0ULL,0xdc2ed3bc6c01aedbULL,0x35c35a1208be74ecULL,0xd540cb1a469f671fULL, + 0xa7a8746a584c5e20ULL,0x267e4ea1b9dc7035ULL,0x593a15cfb9548c9bULL,0x5e6e21354bd012f3ULL,0xdf31cc6a8c8f936eULL,0x8af84d04b5c241dcULL,0x63990a6f345efb86ULL,0x6fef4e61b9b962cbULL, + 0xf6368f0925722608ULL,0x131260db131cf5c6ULL,0x40eb353bfab4f7acULL,0x85c7888037eee829ULL,0x4c1581ffc3bdf24eULL,0x5bff75cbf5c3c5a8ULL,0x35e8c83fa14e6f40ULL,0xb81d1c0f0295e0caULL, + 0xf2efe23d442a8ad1ULL,0xc3816a7d06b9c164ULL,0xa9df2d8bdc0aa5e5ULL,0x191ae46f120a8e65ULL,0x83667f8700611c5bULL,0x83171ed7ff109948ULL,0x33a2ecf8ca695952ULL,0xfa4a73eef48d1a13ULL, + 0xfcde7cc8f43a730fULL,0xe89b6f3c33ab590eULL,0xc823f529ad03240bULL,0x82b79afe98bea5dbULL,0x568f2856962fe5deULL,0x0c590adb60c591f3ULL,0x1fc74a144a28a858ULL,0x3b662498b3203f4cULL, + 0x48fc4ed082dd1b6aULL,0x5783a13867b703afULL,0x2463cb9a005d6aaaULL,0xd31ec55c706ecd43ULL,0x9f8ed33f8e9a7641ULL,0x625453ed098d9e7aULL,0xa3beade4ec887493ULL,0x442b80505a795566ULL, + 0x91e3cf0d6c39765aULL,0xa2db3acdac3cca0bULL,0x288f2f08cb953b50ULL,0x2414582ccf43cf1aULL,0x8dec8bbc60eee9a8ULL,0x54c79f02729aa042ULL,0xd81cd5ec6532f5d5ULL,0xa672303acf82e15fULL, + 0x46df582d3bfab839ULL,0x92474e042f8adadeULL,0x36a7766a147a1bc3ULL,0xb6940f540dc0f979ULL,0x44738ef2f2759f25ULL,0x9dd95789a719f4c6ULL,0x2859b7f40750c345ULL,0x5e788bf2b22180d5ULL, + 0x376aafa8719c0563ULL,0xcd8ad2dcbc5fc79fULL,0x303fdb9fcb750cd3ULL,0x14ff052f4418b08eULL,0xf75084cf3e2d6520ULL,0x7ebdf0f8144ed509ULL,0xf43bf0f2d3f25b98ULL,0x86ad71cfa354d837ULL, + 0xa839c9fdfd67ca25ULL,0x023e626860f2015cULL,0x2414a7930e7b2a65ULL,0x92dbe372b13edcbbULL,0xf64981ee64c2200fULL,0x94fb9cdf8446f2f3ULL,0x01411a6a3f1367bbULL,0x7985c1915a1e8331ULL, + 0xb827fe9226f43572ULL,0xdfd3ab5b5d824758ULL,0x315dd23a539094c1ULL,0x85c0e37a66623d68ULL,0x575c79727be19ae0ULL,0x616a3396df0d36b5ULL,0xa1ebb3c826b1ff7eULL,0x635b9485140ad453ULL, + 0xc8123c6037e2efeaULL,0x8d49b502034a96f6ULL,0x466a346b973e4a95ULL,0xf176b5bab7de00ffULL,0x1c58fa3b82dfa945ULL,0x2eb27a9609e429aeULL,0x57c67a67a12b187cULL,0xb155ba82e2298bbaULL, + 0x92bf3cdada430c0bULL,0x4702850e3a96dac6ULL,0xc91cf0a515ac326aULL,0x95de4f49ab8c25e4ULL,0xb01bad09e265c17cULL,0x24e45464087b3881ULL,0xd43e583ce1fac5caULL,0xe17cb3186ead97a6ULL, + 0xf1a542073d99bcfaULL,0x59db703ce8becf6dULL,0x2e455142d2459569ULL,0xb0ee5143a901b910ULL,0xfc05d451e26d994fULL,0x7a6062b41360caafULL,0xdf1ded5f4fa639b1ULL,0xaf930348d335b8b0ULL, + 0x6cc3924374dcec46ULL,0x33cfc02d54c2b73fULL,0x82917844f26cd99cULL,0x8819dd95d1773f89ULL,0x09572aa60871f427ULL,0x8e0cf365f6f01c34ULL,0x7fa52988bff1f5afULL,0x4eb357eae75e8e50ULL, + 0x3d8f248a21fd0861ULL,0xade3bd649bd5a4b6ULL,0xcb56c953c2e2a6bfULL,0x699cd2b5287d6c5fULL,0xdebce1be47d05e8fULL,0x1a4fbb13a8f53732ULL,0x97163beaa5852b08ULL,0x92c49e6ceec6987aULL, + 0xd9d0c8c4868af75dULL,0xd7325cff45c8c7eaULL,0xab471996cc81ecb0ULL,0xff5d55f3611824edULL,0xbe3145411977a0eeULL,0x5085c4c5722038c6ULL,0x2d5335bff94bb495ULL,0x894ad8a6c8e2a082ULL, + 0x540234b22c11bb37ULL,0x2d0366dded4c74a3ULL,0xf9a968daeec5f25dULL,0x3660106867b63142ULL,0x07cd6d2c68d7b6d4ULL,0xa8f74f090c842942ULL,0xe27514047768b1eeULL,0x4b5f7e89fe62aee4ULL, + 0xf2369f0b879fbbedULL,0x0ff0ae86da9d1869ULL,0x5251d75956766f45ULL,0x4984d8c02be8d0fcULL,0x7ecc95a6d21008f0ULL,0x29bd54a03a1a1c49ULL,0xab9828c5d26c50f3ULL,0x32c0087c51d0d251ULL, + 0x47feeb6662b5f3afULL,0xcefab5610abb3734ULL,0x449de60e19f35cb1ULL,0x39f8db14157f0eb9ULL,0xffaecc5b3c61bfd6ULL,0xa5a4d41d41216703ULL,0x7f8fabed224e1cc2ULL,0x0d5a8186871ad953ULL, + 0x190d8ea601799a52ULL,0xa20cec41b86d2952ULL,0x3062ffb27fff2a7cULL,0x741b32e579f19d37ULL,0xf80d81814eb57d47ULL,0x7a2d0ed416aef06bULL,0x09735fb01cecb588ULL,0x1641caaac6061f5bULL, + 0xae2ad171656e8c3aULL,0xc0e2a4631acd0705ULL,0x006f6a8aa0b6055cULL,0xaf4513d72b65a26eULL,0x3f549e14d616d5bcULL,0x64ee395571253b1fULL,0xe8b10bc1b8ce243aULL,0xbcbeace5913a4e77ULL, + 0x68d32256f779d6a7ULL,0x20423b4d19e7284eULL,0xde19aa1b38f3b153ULL,0x73d0b6c28444f703ULL,0x666161489c64e6a3ULL,0x99587c3737256224ULL,0x6f5277fe61331563ULL,0x7174ad4aa656502eULL, + 0xec1ffc920133918dULL,0x15d9bf5eb2c9ef97ULL,0x7885b542fc6cbb9aULL,0x8abe64535720cf5dULL,0x4e715dced4ec68abULL,0x57a67614279c24a3ULL,0x788ed52a31bb61cfULL,0xaba82444f437a003ULL, + 0x7f99824f20151427ULL,0x206828b692430206ULL,0xaa9097d7e1112357ULL,0xacf9a2f209e414ecULL,0xdbdac9da27915356ULL,0x7e0734b7001efee3ULL,0x54fab5bbd2b288e2ULL,0x4c630fc4f62dd09cULL, + 0x6d883e66bd70a5e1ULL,0x9bd884c1d05de713ULL,0x48d9d445d4d8487dULL,0x8440cd8b0eeae405ULL,0xa3cd0f293d26f83eULL,0x3fd9453022a3c5e4ULL,0x63a078663ffa2ca7ULL,0xbfeadd2900d4a097ULL, + 0x13aea559d68432baULL,0x940043fbd33915dbULL,0x476c7c94e2e8da08ULL,0x6b1630c4443065bbULL,0xd6d8546d19f06c75ULL,0x7e1e98c22eb35abdULL,0x751c9310f0157d8eULL,0xdef84327f6fa6075ULL, + 0x765b993c3a7a4e0eULL,0xfb8f4aef9d79d314ULL,0x5ea2c50e2a19eb24ULL,0x925016e5b891bc8cULL,0x8f70afdc3ad1ece7ULL,0x0abba84cf08fabe2ULL,0x0a9922c815f6132bULL,0x6c0213735d076d77ULL, + 0x00af2615a4453961ULL,0x1705494b993d112cULL,0x0032e12aa1cdd652ULL,0x0ebfa612046d9cb3ULL,0x9f03a9f31b63728cULL,0x8f3618a2de022b05ULL,0xd5b24d903e77c5c9ULL,0x837838a372acb77cULL, + 0xfe65cbd98fceb047ULL,0xde872ef53f8b11a1ULL,0x9aba0d3d8fec802fULL,0x139f1d329a9f381dULL,0x0721aed9c587958dULL,0x066a015cb9f6a7daULL,0x059ec4e3ed5d9d06ULL,0x144285716cbaca1fULL, + 0x66fe0fffe298cdf5ULL,0x3f61bea47b2e51b6ULL,0x7d372117bad3afa4ULL,0x6521a09cef656e2fULL,0xb3b8c966e8a58fe7ULL,0x25203a115a47ebc7ULL,0xfe81588d5c4be573ULL,0x6132e2f31f49a03cULL, + 0xec1ab0130f8c2c99ULL,0x60e8968ff17725c2ULL,0xe1a4a593940a980bULL,0x15ed15b020e9ccb1ULL,0x77d754cc64a00becULL,0x90d09c3341687382ULL,0x294fe02dc31fb651ULL,0x8372cd1a1c94ae53ULL, + 0x8537107a1ac2703bULL,0xb49258d86bc857b5ULL,0x57df14debcdaccd1ULL,0x24ab68d7c4ae8529ULL,0x7ed8b5d4734e59d0ULL,0x5f8740c8c495cc80ULL,0x84aedd5a291db9b3ULL,0x80b360f84fb995beULL, + 0x55d5c68da61a76faULL,0x598b441dca1554dcULL,0xd39923b9773b279cULL,0x33331d3c36bf9efcULL,0x2d4c848e298de399ULL,0xcfdb8e77a1a27f56ULL,0x94c855ea57b8ab70ULL,0xdcdb9dae6f7879baULL, + 0x4c4f07367f636a38ULL,0x9f943fb70e76d5cbULL,0xb03510baa8b68b8bULL,0xc246780a9ed07a1fULL,0x3c0514156d549fc2ULL,0xc2953f31607781caULL,0x955e2c69d8d95413ULL,0xb300fadc7bd282e3ULL, + 0xa14b1163c27901b4ULL,0xfd9236e0899b8bf3ULL,0x42b091eccbc6da0aULL,0xbb1dac6f5ad1d297ULL,0x80e61d53a91cf76eULL,0x4110a412d31f1ee7ULL,0x2d87c3ba13efcf77ULL,0x1f374bb4df450d76ULL, + 0x202886024147519aULL,0xd0981eac26b372f0ULL,0xa9d4a7caa785ebc8ULL,0xd953c50ddbdf58e9ULL,0x9d6361ccfd590f8fULL,0x72e9626b44e6c917ULL,0x7fd9611022eb64cfULL,0x863ebb7e9eb288f3ULL, + 0x15e10a0a097e4403ULL,0xcb3d0a8619854665ULL,0x88d8e211d67d4826ULL,0xb39af66e0b9d2839ULL,0xa5f94588bd475ca8ULL,0xe06b7966c077b80bULL,0xfedb1485da27c26cULL,0xd290d33afe0fd5e0ULL, + 0x686202f31306583dULL,0x05b10da0437c622eULL,0xbf9aaa0f076a7bc8ULL,0x25e94efb8f8f4e43ULL,0x8a35c9b7fa3dc26dULL,0xe0e5fb9396ff03c5ULL,0xa77e3843ebc394ceULL,0xcede65958361de60ULL, + 0xbf9075090c22b540ULL,0x2cde42aab7c267d4ULL,0xba18f9ed5ab0d693ULL,0x3ba62aa66e4660d9ULL,0xb24bf97bab9ea96aULL,0x5d039642e3b60e32ULL,0x4e6a45067c4d9bd5ULL,0x666c5b9e7ed4a6a4ULL, + 0x877b7cf5678a31b0ULL,0xd50301ae3998b620ULL,0x734257c5c00fb396ULL,0xf9fb18a004e672a6ULL,0xff8bd8ebe8758851ULL,0x1e64e4c65d99ba44ULL,0x4b8eaedf7dfd93b7ULL,0xba2f2a9804e76b8cULL, + 0xae02a2f09b56ddbdULL,0x1339b5ac8a2f1cf3ULL,0xf2b569c7839dff0dULL,0xb0b9e864fee9a43dULL,0x4ff8ca4177bb064eULL,0x145a2812fd249f63ULL,0x3ab7beacf86f689aULL,0x9bafec2701d35f5eULL, + 0xbba23fddae57c7b7ULL,0x345342f21b932522ULL,0xfd9c80fe556d4aa3ULL,0xa03907ba6525bb61ULL,0x38b010e1ff218933ULL,0xc066b654aa52117bULL,0x8e14192094f2e6eaULL,0x66a27dca0d32f2b2ULL, + 0xb2e6b121fbabbe92ULL,0x281850fbe1330076ULL,0x093581ec97890015ULL,0x69b1dded75ff77f5ULL,0x7cf0b18fab105105ULL,0x953ced31a89ccfefULL,0x3151f85feb914009ULL,0x3c9f1b8788ed48adULL, + 0xa18f07e0e90fb21eULL,0x00fd2b80bba7fca1ULL,0x20387f2795cd67b5ULL,0x5b89a4e7d39707f7ULL,0x8f83ad3f894407ceULL,0xa0025b946c226132ULL,0xc79563c7f906c13bULL,0x5f548f314e7bb025ULL, + 0xff7589494fde0c1fULL,0xbf8a1abee5b6ec20ULL,0x702278fb87e1db6cULL,0xc447ad7a35ed658fULL,0x48d4aa3803d0ccf2ULL,0x80acb338819a7c03ULL,0x9bc7c89e6e17ceccULL,0x46736b8b03be1d82ULL, + 0x2126f742d43b5eaaULL,0x054a0766dfa59b85ULL,0x9d0d5e36126bfd45ULL,0xa1f8fbd7384f8a8fULL,0x317680f5d563fcccULL,0x48ca5055f280a928ULL,0xe00b81b227b578cfULL,0x10aad9182994a514ULL, + 0xe63f301f358bcdc0ULL,0x07689e990a9d47f8ULL,0x1f689e2f4f43d43aULL,0x4d542a1690920904ULL,0xaea293d59ca0a707ULL,0xd061fe458ac68065ULL,0x1033bf1b0090008cULL,0x29749558c08a6db6ULL, + 0x0ee6d3a7c35d8794ULL,0x042e65580356bae5ULL,0x9f59698d643322fdULL,0x9379ae1550a61967ULL,0x64b9ae62fcc9981eULL,0xaed3d6316d2934c6ULL,0x2454b3025e4e65ebULL,0xab09f647f9950428ULL, + 0xb2083a1222248accULL,0x1f6ec0ef3264e366ULL,0x5659b7045afdee28ULL,0x7a823a40e6430bb5ULL,0x24592a04e1900a79ULL,0xcde09d4ac9ee6576ULL,0x52b6463f4b5ea54aULL,0x1efe9ed3d3ca65a7ULL, + 0xe27a6dbe305406ddULL,0x8eb7dc7fdd5d1957ULL,0xf54a6876387d4d8fULL,0x9c479409c7762de4ULL,0xbe4d5b5d99b30778ULL,0x25380c566e793682ULL,0x602d37f3dac740e3ULL,0x140deabe1566e4aeULL, + 0x4481d067afd32acfULL,0xd8f0fccae1f71ccfULL,0xd208dd0cb596f2daULL,0xd049d7309aad93f9ULL,0xc79f263d42ab580eULL,0x09411bb123f707b4ULL,0x8cfde1ff835e0edaULL,0x7270749090f03402ULL, + 0xeaee6126c49a861eULL,0x024f3b65e14f0d06ULL,0x51a3f1e8c69bfc17ULL,0xc3c3a8e9a7686381ULL,0x3400752cb103d4c8ULL,0x02bc46139218b36bULL,0xc67f75eb7651504aULL,0xd6848b56d02aebfaULL, + 0xbd9802e6c30fa92bULL,0x5a70d96d9a552784ULL,0x9085c4ea3f83169bULL,0xfa9423bb06908228ULL,0x2ffebe12fe97a5b9ULL,0x85da604971b99118ULL,0x9cbc2f7f63178846ULL,0xfd96bc709153218eULL, + 0x958381db1782269bULL,0xae34bf792597e550ULL,0xbb5c60645f385153ULL,0x6f0e96afe3088048ULL,0xbf6a021577884456ULL,0xb3b5688c69310ea7ULL,0x17c9429504fad2deULL,0xe020f0e517896d4dULL, + 0x730ba0ab0976505fULL,0x567f6813095e2ec5ULL,0x470620106331ab71ULL,0x72cfa97741d22b9fULL,0x33e55ead8a2373daULL,0xa8d0d5f47ba45a68ULL,0xba1d8f9c03029d15ULL,0x8f34f1ccfc55b9f3ULL, + 0xcca4428dbbe5a1a9ULL,0x8187fd5f3126bd67ULL,0x0036973a48105826ULL,0xa39b6663b8bd61a0ULL,0x6d42deef2d65a808ULL,0x4969044f94636b19ULL,0xf611ee47dd5d564cULL,0x7b2f3a49d2873077ULL, + 0x94157d45300eb294ULL,0x2b2a656e169c1494ULL,0xc000dd76d3a47aa9ULL,0xa2864e4fa6243ea4ULL,0x82716c47db89842eULL,0x12dfd7d761479fb7ULL,0x3b9a2c56e0b2f6dcULL,0x46be862ad7f85d67ULL, + 0x03b0d8dd0f82b214ULL,0x460c34f9f103cbc6ULL,0xf32e5c0318d79e19ULL,0x8b8888baa84117f8ULL,0x8f3c37dcc0722677ULL,0x10d21be91c1c0f27ULL,0xd47c8468e0f7a0c6ULL,0x9bf02213adecc0e0ULL, + 0x0baa7d1242b48b99ULL,0x1bcb665d48424096ULL,0x8b847cd6ebfb5cfbULL,0x87c2ae569ad4d10dULL,0xf1cbb1220de36726ULL,0xe7043c683fdfbd21ULL,0x4bd0826a4e79d460ULL,0x11f5e5984bd1a2cbULL, + 0x97554160b7fe7b6eULL,0x7d16189a400a3fb2ULL,0xd73e9beae328ca1eULL,0x0dd04b97e793d8ccULL,0xa9c83c9b506db8ccULL,0x5cd47aaecf38814cULL,0x26fc430db64b45e6ULL,0x079b5499d818ea84ULL, + 0xebb01102c1c24a3bULL,0xca24e5681c161c1aULL,0x103eea6936f00a4aULL,0x9ad76ee876176c7bULL,0x97451fc2538e0ff7ULL,0x94f898096604b3b0ULL,0x6311436e3249cfd7ULL,0x27b4a7bd41224f69ULL, + 0x03b5d21ae0ac2941ULL,0x279b0254c2d31937ULL,0x3307c052cac992d0ULL,0x6aa7cb92efa8b1f3ULL,0x5a1825800d37c7a5ULL,0x13380c37342d5422ULL,0x92ac2d66d5d2ef92ULL,0x035a70c9030c63c6ULL, + 0xc16025dd4ce4f152ULL,0x1f419a71f9df7c06ULL,0x6d5b221491e4bb14ULL,0xfc43c6cc839fb4ceULL,0x49f06591925d6b2dULL,0x4b37d9d362186598ULL,0x8c54a971d01b1629ULL,0xe1a9c29f51d50e05ULL, + 0x5109b78571ba1861ULL,0x48b22d5cd0c8f93dULL,0xe8fa84a78633bb93ULL,0x53fba6ba5aebbd08ULL,0x7ff27df3e5eea7d8ULL,0x521c879668ca7158ULL,0xb9d5133bce6f1a05ULL,0x2d50cd53fd0ebee4ULL, + 0x889f6d65533ef217ULL,0x7158c7e4c3ca2e87ULL,0xfb670dfbdc2b4167ULL,0x75910a01844c257fULL,0xf336bf07cf88577dULL,0x22245250e45e2aceULL,0x2ed92e8d7ca23d85ULL,0x29f8be4c2b812f58ULL, + 0xfbb9b2452133ffd9ULL,0x39a8b2f1830f1a20ULL,0x484bc97dd5a1f52aULL,0xd6aebf56a40eddf8ULL,0x32257acb76ccdac6ULL,0xaf4d36ec1586ff27ULL,0x8eaa8863f8de7dd1ULL,0x0045d5cf88647c16ULL, + 0xc51e414351facc61ULL,0xbaf2647de68a25bcULL,0x8f5271a00ff872edULL,0x8f32ef993d2d9659ULL,0xca12488c7593cbd4ULL,0xed266c5d02b82fabULL,0x0a2f78ad14eb3f16ULL,0xc34049484d47afe3ULL, + 0xa6f3d574c005979dULL,0xc2072b426a40e350ULL,0xfca5c1568de2ecf9ULL,0xa8c8bf5ba515344eULL,0x97aee555114df14aULL,0xd4374a4dfdc5ec6bULL,0x754cc28f2ca85418ULL,0x71cb9e27d3c41f78ULL, + 0x09c1670209470496ULL,0xa489a5edebd23815ULL,0xc4dde4648edd4398ULL,0x3ca7b94a80111696ULL,0x3c385d682ad636a4ULL,0x6702702508dc5f1eULL,0x0c1965deafa21943ULL,0x18666e16610be69eULL, + 0x6792fd350369c8e1ULL,0x9271aa62b9dc843bULL,0x8711a4b14d02e2abULL,0x02b2a3e27ee1a383ULL,0xb226e35f0e2b379bULL,0x3d3de39cd652ab25ULL,0xaca6d4c93b560106ULL,0xeced0cf4c95bd877ULL, + 0x45beb4ca2a604b3bULL,0x56f651843a616762ULL,0xf52f5a70978b806eULL,0x7aa3978711dc4480ULL,0xe13fac2a0e01fabcULL,0x7c6ee8a5237d99f9ULL,0x251384ee05211ffeULL,0x4ff6976d1bc9d3ebULL, + 0x8910507903605c39ULL,0xf0843d9ea142c96cULL,0xf374493416923684ULL,0x732caa2ffa0a2893ULL,0xb2e8c27061160170ULL,0xc32788cc437fbaa3ULL,0x39cd818ea6eda3acULL,0xe2e942399e2b2e07ULL, + 0xdde0492316e043a2ULL,0x98a452611dd3d209ULL,0xeaf9f61bd431ebe8ULL,0x00919f4dbaf56abdULL,0xe42417db6d8774b1ULL,0x5fc5279c58e0e309ULL,0x64aa40613adf81eaULL,0xef419edabc627c7fULL, + 0x3919759239ef620fULL,0x9d47284074fa29c4ULL,0x4e428fa39d416d83ULL,0xd1a7c25129f30269ULL,0x46076e1cd746218fULL,0xf3ad6ee8110d967eULL,0xfbb5f434a00ae61fULL,0x3cd2c01980d4c929ULL, + 0xfa24d0537a4af00fULL,0x3f938926ca294614ULL,0x0d700c183982182eULL,0x801334434cc59947ULL,0xf0397106ec87c925ULL,0x62bd59fc0ed6665cULL,0xe8414348c7cca8b5ULL,0x574c76209f9f0a30ULL, + 0x6967d39b0260e52aULL,0xd42585cc90653325ULL,0x0d9bd60521ca7954ULL,0x4fa2087781ed57b3ULL,0x60c1eff8e34a0bbeULL,0x56b0040c84f6ef64ULL,0x28be2b24b1af8483ULL,0xb2278163f5531614ULL, + 0x95be42e2bb8b6a07ULL,0x64be74eeca23f86aULL,0xa73d74fd154ce470ULL,0x1c2d2857d8dc076aULL,0xb1fa1c575a887868ULL,0x38df8e0b3de64818ULL,0xd88e52f9c34e8967ULL,0x274b4f018b4cc76cULL, + 0x59c81ec2c11c208bULL,0x240207da4e3a7234ULL,0x4957eb80f4a089d8ULL,0xc86960bc2137a072ULL,0xde1154fd7f1a932cULL,0xb517fe74d3f2fa17ULL,0x2d5d94557111c0ebULL,0x2aa3378980929d36ULL, + 0x3f5c05b4f8b7559dULL,0x0be4c7acfae29200ULL,0xdd6d3ef756532accULL,0xf6c3ed87eea7a285ULL,0xe463b0a8f46ec59bULL,0x531d9b14ecea6c83ULL,0x3d6bdbafc2dc836bULL,0x3ee501e92ab27f0bULL, + 0x8df275455922ac1cULL,0xa7b3ef5ca52b3f63ULL,0x8e77b21471de57c4ULL,0x31682c10834c008bULL,0xc76824f04bd55d31ULL,0xb6d1c08617b61c71ULL,0x31db0903c2a5089dULL,0x9c092172184e5d3fULL, + 0x5ace5035ea6c3997ULL,0x54259aaac2610befULL,0xef18bb3f3c80dd39ULL,0x6910b95b5fc3fa39ULL,0xfce2f51043e09aeeULL,0xced56c9fa7675665ULL,0x10e265acd872db61ULL,0x6982812eae9fce69ULL, + 0xb8fa3d931341ed7aULL,0x4223272ca7b59d49ULL,0x3dcb194783b8c4a4ULL,0x4e413c01ed1302e4ULL,0x6d999127e17e44ceULL,0xee86bf7533b3adfbULL,0xf6902fe625aa96caULL,0xb73540e4e5aae47dULL, + 0xcc50ef6c872b4a60ULL,0xab2a34a44613521bULL,0x39c5c190983e15d1ULL,0x61dde5df59905512ULL,0xe417f6219f2275f3ULL,0x0750c8b6451d894bULL,0x75b04ab978b0bdaaULL,0x3bfd9fd4458589bdULL, + 0xaafcbfabaf95894cULL,0x7b9bdc07276b2241ULL,0xeaf983625bdda48bULL,0x5977faf2a3fcb4dfULL,0xbed042ef052c4b5bULL,0x9fe87f71067591f0ULL,0xc89c73ca22f24ec7ULL,0x7d37fa9ee64a9f1bULL, + 0xbd78148045ee2f40ULL,0x75e354af416b60cfULL,0xde0b58a18d49a8c4ULL,0xe40e94e2fa359536ULL,0xbd4fa59f62accd76ULL,0x05cf466a8c762837ULL,0xb5abda99448c277bULL,0x5a9e01bf48b13740ULL, + 0xb51e55e6f2606a82ULL,0xe25f706190f2fb57ULL,0xacef6c2ab1a4e37cULL,0x864e359d5dcf2706ULL,0x479e6b187ce57316ULL,0x2cab25003a96b23dULL,0xed4898628ef16df7ULL,0x2056538cef3758b5ULL, + 0xef69a0c3d41d3bd3ULL,0xb533b8c907a26bdeULL,0xe2801d97db2edf9fULL,0xdc4a8269e1877af0ULL,0x6c1c58513d590dbeULL,0x84632f6bee4e9357ULL,0xd36d36b779b33374ULL,0xb46833e39bbca2e6ULL, + 0xcc7a64880a750c0fULL,0x39bacfe34e548e83ULL,0x3d418c760c110f05ULL,0x3e4daa4cb1f11588ULL,0x2733e7b55ffc69ffULL,0x46f147bc92053127ULL,0x885b2434d722df94ULL,0x6a444f65e6fc6b7cULL, + 0x008be4bb346ac9cbULL,0x2e6cb02429811bb2ULL,0xa6ccc747f41540c1ULL,0xf119334efbf6de47ULL,0x4e6bffe9cc97fe6eULL,0x7f4b578bf3d82883ULL,0x459db722753dd1c7ULL,0x05843cd82066b495ULL, + 0x7cb88cd5dbf2af33ULL,0xded9566eaf6b0eabULL,0xd1caf5488e40d844ULL,0x3ae59bd67ce1f67fULL,0xacf4ad33dd82429eULL,0xcc7ebd2bbc4467c7ULL,0xf4f6e95de783d011ULL,0x5197e39116e92db7ULL, + 0xd4f2031885e7a4afULL,0x291bf9a0f7a6878dULL,0xadd8b6eebd051913ULL,0x174f5124f36be034ULL,0x2ac3364527f60189ULL,0xd8902eed8b37ec73ULL,0x546166a11cfdb42eULL,0x31c4e3b807076421ULL, + 0x7de29dfdbc6ebdfbULL,0xa1ee450589de549bULL,0xfd7181aa3ef26a64ULL,0x003179eedf2f3980ULL,0x1bce4d30fd71bc78ULL,0xeb842b14be86a583ULL,0xe00125dc5eb85711ULL,0x9f586983ff11f405ULL, + 0x5b5a089eb833eccbULL,0xc1b3077a7bc51c79ULL,0xe581157f9fbe0e93ULL,0xab487d695b29b172ULL,0xc72551082222f24bULL,0x338cd22cf6dd35dfULL,0x92010e6f5fc24afeULL,0xf8298f15681d46edULL, + 0x2ce7f723042389edULL,0xdf7de0d5a54b1972ULL,0xb8ea2e142c251d75ULL,0xd22a4d37f4e8e8c8ULL,0xe99958566c29c8bcULL,0x9fefdbfe1d1e201eULL,0xb97d946edac885c9ULL,0xf6ea9767d38eac70ULL, + 0x9aaca8e974e75a2eULL,0xcda6fc1eadac968fULL,0x3f6651bf46306befULL,0xc7445e4ecb2ed7f9ULL,0x584a12d8d1571ac1ULL,0x684846c4daf3a679ULL,0xcfc622a9863fbc43ULL,0x2f9e101dea6814f2ULL, + 0x7a1a465ac3f16ea8ULL,0x115a461db2f1d11cULL,0x4767dd956c68a172ULL,0x3392f2ebd13a4698ULL,0xc7a99ccde526cdc7ULL,0x8e537fdc22292b81ULL,0x76d8cf69a6d39198ULL,0xffc5ff432446852dULL, + 0x97b14f7ea90567e6ULL,0x513257b7b6ae5cb7ULL,0x85454a3c9f10903dULL,0xd8d2c9ad69bc3724ULL,0x38da93246b29cb44ULL,0xb540a21d77c8cbacULL,0x9bbfe43501918e42ULL,0xfffa707a56c3614eULL, + 0x56c2e05b1cb76219ULL,0x0ec0bf9171567e7eULL,0xe7076f8661c4c910ULL,0xd67b085bbabc04d9ULL,0x9fb904595e93a96aULL,0x7526c1eafbdc249aULL,0x0d44d367ecdd0bb7ULL,0x953999179dc0d695ULL, + 0x2ccbc583a4c506ecULL,0x957ed188d1acfe97ULL,0x8baed83312f1aea2ULL,0xef2a6cb48325362dULL,0x130dde428e195c43ULL,0xc842025a0e6050c6ULL,0x2da972a708686a5dULL,0xb52999a1e508b4a8ULL, + 0x83f49167ceca9754ULL,0x426d2cf64b7939a0ULL,0x2555e355723fd0bfULL,0xa96e6d06c4f144e2ULL,0x4768a8dd87880e61ULL,0x15543815e508e4d5ULL,0x09d7e772b1b65e15ULL,0x63439dd6ac302fa0ULL, + 0xb11df8e1698e04ccULL,0x877be203169005c8ULL,0x32749e8c4f3c6179ULL,0x2dbc9d0a7853fc05ULL,0x187d4f939454d937ULL,0xe682ce9db4800e1bULL,0xa9129ad8165e68e8ULL,0x0fe29735be7f785bULL, + 0x31019ccf3a4434b4ULL,0xa34581111a7954dcULL,0xa9dac80de34972a7ULL,0xb043d05474f6b8ddULL,0x021c319e11137b1aULL,0x00a754ceed5cc03fULL,0x0aa2c794cbea5ad4ULL,0x093e67f470c015b6ULL, + 0x20ac0351d598d710ULL,0x272c4166cb3a4da4ULL,0xdb82fe1aca71de1fULL,0x746e79f2d8f54b0fULL,0x6e7fc7364b573e9bULL,0x75d03f46fd4b5040ULL,0x5c1cc36d0b98d87bULL,0x513ba3f11f472da1ULL, + 0x859d3145983c38b5ULL,0xb14f176c637abc8bULL,0x2793fb9dcaff7be6ULL,0xebe5a55f35a66a5aULL,0x7cec1dcd9f87dc59ULL,0x7c595cd3fbdbf560ULL,0x5b543b2226eb3257ULL,0x69080646c4c935fdULL, + 0xdca3b70678a6513bULL,0x92ea4a2a9edb1943ULL,0x02642216db6e2dd8ULL,0x9b45d0b49fd57894ULL,0x114e70dbc69d11aeULL,0x1477dd194c57595fULL,0xbc2208b4ec77c272ULL,0x95c5b4d7db68f59cULL, + 0xfe541fa47ea67c77ULL,0x952bd2afe3ea810cULL,0x791fef568d01d374ULL,0xa3a1c6210f11336eULL,0x5ad0d5a9c7ec6d79ULL,0xff7038af3225c342ULL,0x003c6689bc69601bULL,0x25059bc745e8747dULL, + 0x9a75c80676cb2566ULL,0x8f76acb1b24892d9ULL,0x7ae7b9cc1f08fe45ULL,0x19ef73296a4907d8ULL,0x2db4ab715f228bf0ULL,0xf3cdea39817032d7ULL,0x0b1f482edcabe3c0ULL,0x3baf76b4bb86325cULL, + 0x6aac688eadd70482ULL,0x708de92a7b4a4e8aULL,0x75b6dd73758a6eefULL,0xea4bf352725b3c43ULL,0x10041f2c87912868ULL,0xb1b1be95ef09297aULL,0x19ae23c5a9f3860aULL,0xc4f0f839515dcf4bULL, + 0xc71e27bf8538a5c6ULL,0x195c63dd89abff17ULL,0xfd3152851b71e3daULL,0x9cbdfda7fa680fa0ULL,0x9db876ca849d7eabULL,0xebe2764b3c273271ULL,0x663357e3f208dceaULL,0x8c5bd833565b1b70ULL, + 0x75900d7c2fa4f126ULL,0x08a3b8655c99a232ULL,0x2478b6bfdb25e0c3ULL,0x482cc2c271db2edfULL,0x37df7e645f321bb8ULL,0x8a93821b9a8005b4ULL,0x3fa2f10ccc8c1958ULL,0x0d3322182c269d0aULL, + 0xba5514df3fd165e8ULL,0x499fd6a9061f8811ULL,0x72cd1fe0bfef9f00ULL,0x120a4bb979ad7e8aULL,0xf2ffd0955f4a5ac5ULL,0xcfd174f195a7a2f0ULL,0xd42301ba9d17baf1ULL,0xd2fa487a77f22089ULL, + 0xb93452381d531696ULL,0x57201c0088cdde69ULL,0xdde922519a86afc7ULL,0xe3043895bd35cea8ULL,0x7608c1e18555970dULL,0x8267dfa92535935eULL,0xd4c60a57322ea38bULL,0xe0bf7977804ef8b5ULL, + 0x6233ea68c094dbb5ULL,0xb77d062ed968d410ULL,0x3e719bbc58b3002dULL,0x68e7dd3d3dc49d58ULL,0x8d825740013a5e58ULL,0x213117473c9e3c1bULL,0x0cb0a2a77c99b6abULL,0x5c48a3b3c2f888f2ULL, + 0xc7913e91991724f3ULL,0x5eda799c39cbd686ULL,0xddb595c763d4fc1eULL,0x6b63b80bac4fed54ULL,0x6ea0fc697e5fb516ULL,0x737708bad0f1c964ULL,0x9628745f11a92ca5ULL,0x61f379589a86967aULL, + 0x5320fd610979e6b3ULL,0x1d0bd3e593d40723ULL,0x0ac006fcaa80baccULL,0xb1d9c60ed2002515ULL,0x610e7ed0af780b92ULL,0xf5bf446e80a9ce31ULL,0x441c011d3c20b54bULL,0x77f76da1191596c3ULL, + 0x9af39b2caa665072ULL,0x78322fa4efd324efULL,0x3d153394c327bd31ULL,0x81d5f2713129dab0ULL,0xc72e0c42f48027f5ULL,0xaa40cdbc8536e717ULL,0xf45a657a2d369d0fULL,0xb03bbfc4ea7f74e6ULL, + 0x7a04d5429b6a42eaULL,0xfa597e853daf41b9ULL,0x4c58ec27726b0b89ULL,0xed8eb16a030d43deULL,0x65e1e5e1ec9dcf57ULL,0xb7a770c1697cff81ULL,0x1e6d918f9f6e2b22ULL,0x7c277a9ac64e82b7ULL, + 0x46a8c4180d738dedULL,0x6f1a5bb0e0de5729ULL,0xf10230b98ba81675ULL,0x32c6f30c112b33d4ULL,0x7559129dd8fffb62ULL,0x6a281b47b459bf05ULL,0x77c1bd3afa3b6776ULL,0x0709b3807829973aULL, + 0x0875e0c1da36cb47ULL,0xfdf5b7cb1e0210f0ULL,0x7e0c7e4d3a3787c8ULL,0xf043f5262b1c741fULL,0x76df1b006d74d72dULL,0x0514df7338b45ba9ULL,0xaecf7c3e0a6b797aULL,0x5e30b285ddeaac39ULL, + 0x8c26b232a3326505ULL,0x38d69272ee1d41bfULL,0x0459453effe32afaULL,0xce8143ad7cb3ea87ULL,0x932ec1fa7e6ab666ULL,0x6cd2d23022286264ULL,0x459a46fe6736f8edULL,0x50bf0d009eca85bbULL, + 0x15c78f7d605238a5ULL,0xe9d87842448496abULL,0xcfcf75d0d210acc1ULL,0x2948f2295c8c14e2ULL,0xe81fd76de8daf0cbULL,0xd02d11e4a03be800ULL,0x0c4df3518778d30cULL,0x3482bc96965139cbULL, + 0x0b825852877a21ecULL,0x300414a70f537a94ULL,0x3f1cba4021a9a6a2ULL,0x50824eee76943c00ULL,0xa0dbfcecf83cba5dULL,0xf953814893b4f3c0ULL,0x6174416248f24dd7ULL,0x5322d64de4fb09ddULL, + 0x56b6cf278a448bbcULL,0x0ca898dfc85251daULL,0x9082cad836e79b24ULL,0x2e7b9ed31a8e51a7ULL,0xdc7d318a43e1c802ULL,0x4750e523cbf8689dULL,0x9887a072f0071b1aULL,0x52090f87814bfdc1ULL, + 0x574473843d9325f3ULL,0xa9bef2d0f371cb84ULL,0x77d2188ba61e36c5ULL,0xbbd6a7d7c602df72ULL,0xba3aa9028f61bc0bULL,0xf49085ed6ed0b6a1ULL,0x8bc625d6ae6e8298ULL,0x832b0b1da2e9c01dULL, + 0x5bebf2196196bc6eULL,0x0e66736bef097efdULL,0x1128f3b8ea87293aULL,0x2998e4056addfcdeULL,0x55cc31b85d16c961ULL,0x87a434e12418f056ULL,0x2e94aaccdc9fe819ULL,0x94a486c1a56490c6ULL, + 0xa337c447f1f0ced1ULL,0x800cc7939492dd2bULL,0x4b93151dbea08efaULL,0x820cf3f8de0a741eULL,0xff1982dc1c0f7d13ULL,0xef92196084dde6caULL,0x1ad7d97245f96ee3ULL,0x319c8dbe29dea0c7ULL, + 0x1ff2385bde259ec8ULL,0xf6b0836a30f67b0dULL,0x04cc65b006661cffULL,0x467e6358d4230f5cULL,0xce468c802dbed3d3ULL,0x7b984262f1920da6ULL,0x34d257421537479fULL,0x5c8aa88c87bb8de1ULL, + 0xd3ea38717b82b99bULL,0x75922d4d470eb624ULL,0x8f66ec543b95d466ULL,0x66e673ccbee1e346ULL,0x6afe67c4b5f2b89aULL,0x3de9c1e6290e5cd3ULL,0x8c278bb6310a2adaULL,0x420fa3840bdb323bULL, + 0x646f96796424c49bULL,0xf888dfe867c241c9ULL,0xe12d4b9324f68b49ULL,0x9a6b62d8a571df20ULL,0x81b4b26d179483cbULL,0x666f96329511fae2ULL,0xd281b3e4d53aa51fULL,0x7f96a7657f3dbd16ULL, + 0x8553d37c051af62bULL,0xe9a998eb0bf94496ULL,0xe0844f9fb0d59aa1ULL,0x983fd558e6afb813ULL,0x9670c0ca65d69804ULL,0x732b22de6ea5ff2dULL,0xd7640ba95fd8623bULL,0x9f619163a6351782ULL, + 0xf167b4e0bdefdd4fULL,0x69958465f366e401ULL,0x5aa368aba73bbec0ULL,0x121487097b240c21ULL,0x378c323318969006ULL,0xcb4d73cee1fe53d1ULL,0x5f50a80e130c4361ULL,0xd67f59517ef5212bULL, + 0x332f81088cad38c0ULL,0x471b7e906bd68ae2ULL,0x56ac3fb20d8e27a3ULL,0xb54660db136b4b0dULL,0x123a1e11a6fd8de4ULL,0x44dbffeaa37799efULL,0x4540b977ce6ac17cULL,0x495173a8af60acefULL, + 0xeb4437434573eab0ULL,0x11570dfbd1ac6031ULL,0xf7d9b45b44dd9afdULL,0xb8066add22067231ULL,0x15f92ad8f8a3f0b4ULL,0x9e0e4899e0ace2a2ULL,0xbdcd0aadfab38b80ULL,0x46506ae917020052ULL, + 0x429a69f78fca399dULL,0xfe9e27d20207bb63ULL,0xec655ed68788f582ULL,0xa426d748adb75f6eULL,0x18695c02ca81c66dULL,0x84fb8d27a531d425ULL,0x3a3a8956deff48baULL,0xaf1d0d56766d2247ULL, + 0x5a059565352c4b5cULL,0x49261531590bc3e2ULL,0x809f7521f66f9f5fULL,0x2baef6bfc70a4a9bULL,0xe7e6fa6509ed3561ULL,0x11370233984b230cULL,0x2151659bd04cdc69ULL,0xbdb83c63f007d416ULL, + 0x9ebb284d391c2a82ULL,0xbcdd4863158308e8ULL,0x006f16ec83f1edcaULL,0xa13e2c37695dc6c8ULL,0x2ab756f04a057a87ULL,0xa8765500a6b48f98ULL,0x4252face68651c44ULL,0xa52b540be1765e02ULL, + 0xcb35a1a85ca37ff0ULL,0xe1a04f1ccd2f1c8fULL,0x238816ce15a26112ULL,0xe206a111095b177eULL,0x3c10b6048a424149ULL,0xc6a3f56774752cfbULL,0xbf16a37a47f1dbb8ULL,0x7c372f9ad31a3dfbULL, + 0x122d05b5c20b4d2aULL,0xff659cf50c662a67ULL,0xed57c128e8ffc9e9ULL,0x0fbb15859e987683ULL,0xadd70df247319a2bULL,0x4b98baba374be470ULL,0xf03d747356a7b307ULL,0x342e696e3efebf30ULL, + 0xf84b48f7864ac537ULL,0x04713409a6940d3dULL,0x014db22d6174c7aeULL,0xc73a1c438c213034ULL,0x18ac4ea5ffdd93ecULL,0x724fc7576102783eULL,0x9fe13fcc91c3e83fULL,0x92a8c2c8f08f0bf5ULL, + 0x7f2e22fd8f67f6deULL,0xab018833d8693177ULL,0x266d1db6863eca95ULL,0x6bb7732b31b5ef2bULL,0x4fa927c6915f80ceULL,0x6fa1d6d25f90efd8ULL,0x7bd75de8456b48adULL,0xd2cb507a845b6429ULL, + 0xa72cf82ae255d7ecULL,0x52025c23a460e204ULL,0x10ae542d7d5b0a44ULL,0xa85143109305aedaULL,0x958315f5a14bbfe8ULL,0x3f361826385365feULL,0xc2b3a36b66d95040ULL,0x12c7b3347cf4eda2ULL, + 0xa545b4d1e744119dULL,0x4c93b169829a71e7ULL,0x2dbb908c6117fcbfULL,0x4dc97320b35a3d85ULL,0x94c04f856bf88105ULL,0x452e1bce1b51bc1fULL,0x0c41ff50b3013af3ULL,0xf07af445224d7e24ULL, + 0xbdb9e57ca3d24f6aULL,0x8a8246d7f345a763ULL,0x73bd2a6d98cfbb5fULL,0x1dd8e85e86ed04dbULL,0x76f2da42c01f420bULL,0x7ef0547364407bc7ULL,0x7e98ba7faff548f5ULL,0x6b7afbeefd30b64aULL, + 0x4f922fc516a0d2bbULL,0x0d5cc16c1a623499ULL,0x9241cf3a57c62c8bULL,0x2f5e6961fd1b667fULL,0x5c15c70bf5a01797ULL,0x3d20b44d60956192ULL,0x04911b37071fdb52ULL,0xf648f9168d6f0f7bULL, + 0x027cc8b8fac61d9aULL,0x7d25e062e3c6fe8aULL,0xe08805bfe5bff503ULL,0x13271e6c6ff632f7ULL,0x55dca6c0232f76a5ULL,0x8957c32d701ef426ULL,0xee728bcba10a5178ULL,0x5ea60411b62c5173ULL, + 0x9ad5462bb4d8bc50ULL,0x181c0b16a9195770ULL,0xebd4fe1c78412a68ULL,0xae0341bcc0dff48cULL,0xb6bc45cf7003e866ULL,0xf11a6dea8a24a41bULL,0x5407151ad04c24c2ULL,0x62c9d27dda5b7b68ULL, + 0x32865719a8afd30bULL,0x867983288a826dceULL,0xdf04e891c4a8fbe0ULL,0xbb6b6e1bebf56ad3ULL,0x0a695b11471f1ff0ULL,0xd76c3389be15baf0ULL,0x018edb95be96c43eULL,0xf2beaaf490794158ULL, + 0x0a50b12e523b8bf6ULL,0x8009eb5b8f910c1bULL,0xf535af824a167588ULL,0x0f835f9cfb2a2abdULL,0xf59b29312afceb62ULL,0xc797df2a169d383fULL,0xeb3f5fb066ac02b0ULL,0x029d4c6fdaa2d0caULL, + 0x87a7ebd1e0a1b12aULL,0x1e4ef88d770ba95fULL,0x8c33345cdc2ae9cbULL,0xcecf127601cc8403ULL,0x687c012e1b39b80fULL,0xfd90d0ad35c33ba4ULL,0xa3ef5a675c9661c2ULL,0x368fc88ee017429eULL, + 0xb82226052b7ce542ULL,0xe6d4ce997472bde1ULL,0x53e16ebe09d2f4daULL,0x180ff42e53b92b2eULL,0xc59bcc022c34a1c6ULL,0x3803d6f9422c46c2ULL,0x18aff74f5c14a8a2ULL,0x55aebf8010a08b28ULL, + 0xb956970e2fdd23ccULL,0xb80288bc5682e971ULL,0xe6e6d91e9ae86ebcULL,0x0564c83f8c9f1939ULL,0x551932a239560368ULL,0xe893752b049c28e2ULL,0x0b03cee5a6a158c3ULL,0xe12d656b04964263ULL, + 0x58af2010f5b343bcULL,0x0f2e400af2f142feULL,0x3483bfdea85f4bdfULL,0xf0b1d09303bfeaa9ULL,0x2ea01b95c7081603ULL,0xe943e4c93dba1097ULL,0x47be92adb438f3a6ULL,0x00bb7742e5bf6636ULL, + 0x4ed714576be5f7deULL,0xd93006f8c2263c9eULL,0xe073694ccacacb36ULL,0x2ff7a5b43ae118abULL,0x3cce53f1cd871236ULL,0xf156a39dc2aa6d52ULL,0x9cc5f271b198d76dULL,0xbc615b6f81383d39ULL, + 0x137a4fb486df2a61ULL,0xa1ed9c07ecf7b4a2ULL,0xb2e460e27bd042ffULL,0xb7f5e2fa5f62f5ecULL,0x7aa6ec6bcc2423b7ULL,0x75ce0a7fba63eea7ULL,0x67a45fb1f250a6e1ULL,0x93bc919ce53cdc9fULL, + 0x67930af231f63950ULL,0xa77797c114caa2c9ULL,0x526e80ee27ac7e62ULL,0xe1e6e62658b28aecULL,0x636178b0b3c9fef0ULL,0xaf7752e06d5f90beULL,0x94ecaf18eece51cfULL,0x2864d0edca806e1fULL, + 0xec2fccaaddce3345ULL,0x2a6811b7012a4350ULL,0x96760ff1ac598bdcULL,0x054d652ad1bf4128ULL,0x0a1151d492a21005ULL,0xad7f397133110fdfULL,0x8c95928c1960100fULL,0x6c91c8257bf03362ULL, + 0x17785b7799eb6df0ULL,0x26c3cc517386b779ULL,0x345ed9886417a48eULL,0xe990b4e407d6ef31ULL,0x0f456b7e2586abbaULL,0x239ca6a559c96e9aULL,0xe327459ce2eb4206ULL,0x3a4c3313a002b90aULL, + 0x19e6125dec3f1decULL,0x07b1f040911178daULL,0xd93ededa904a6738ULL,0x55187a5a0bebedcdULL,0xf7d04722eb329d41ULL,0xf449099ef170b391ULL,0xfd317a69ca99f828ULL,0x50c3db2b34a4976dULL, + 0x3806b69b92222f1fULL,0x5a2459ca6cf7ae70ULL,0x6789f69ca85217eeULL,0x5f232b5ee3dc85acULL,0x660e3ec548e9e516ULL,0x124b4e473197eb31ULL,0x10a0cb13aafcca23ULL,0x7bd63ba48213224fULL, + 0xb674481b7bfe7178ULL,0x4e1debae65405868ULL,0x061b2821c48c867dULL,0x69c15b35513b30eaULL,0x3b4a166636871088ULL,0xe5e29f5d1220b1ffULL,0x4b82bb35233d9f4dULL,0x4e07633318cdc675ULL, + 0x0d53f5c7a3e6fcedULL,0xe8cbbdd5f45fbdebULL,0xf85c01df13339a70ULL,0x0ff71880142ceb81ULL,0x4c4e8774bd70437aULL,0x5fb32891ba0bda6aULL,0x1cdbebd2f18bd26eULL,0x2f9526f103a9d522ULL, + 0x40ce305192c4d684ULL,0x8b04d7257612efcdULL,0xb9dcda366f9cae20ULL,0x0edc4d24f058856cULL,0x64f2e6bf85427900ULL,0x3de81295dc09dfeaULL,0xd41b4487379bf26cULL,0x50b62c6d6df135a9ULL, + 0xd4f8e3b4c72dfe67ULL,0xc416b0f690e19fdfULL,0x18b9098d4c13bd35ULL,0xac11118a15b8cb9eULL,0xf598a318f0062841ULL,0xbfe0602f89f356f4ULL,0x7ae3637e30177a0cULL,0x3409774761136537ULL, + 0x0db2fb5ed005832aULL,0x5f5efd3b91042e4fULL,0x8c4ffdc6ed70f8caULL,0xe4645d0bb52da9ccULL,0x9596f58bc9001d1fULL,0x52c8f0bc4e117205ULL,0xfd4aa0d2e398a084ULL,0x815bfe3a104f49deULL, + 0x97e5443f23885e5fULL,0xf72f8f99e8433aabULL,0xbd00b154e4d4e604ULL,0xd0b35e6ae5e173ffULL,0x57b2a0489164722dULL,0x3e3c665b88761ec8ULL,0x6bdd13973da83832ULL,0x3c8b1a1e73dafe3bULL, + 0x4497ace654317cacULL,0xbe600ab9521771b3ULL,0xb42e409eb0dfe8b8ULL,0x386a67d73942310fULL,0x25548d8d4431cc28ULL,0xa7cff142985dc524ULL,0x4d60f5a193c4be32ULL,0x83ebd5c8d071c6e1ULL, + 0xba3a80a7b1fd2b0bULL,0x9b3ad3965bec33e8ULL,0xb3868d6179743fb3ULL,0xcfd169fcfdb462faULL,0xd3b499d79ce0a6afULL,0x55dc1cf1e42d3ff8ULL,0x04fb9e6cc6c3e1b2ULL,0x47e6961d6f69a474ULL, + 0x54eb3acce548b37bULL,0xb38e754284d40549ULL,0x8c3daa517b341b4fULL,0x2f6928ec690bf7faULL,0x0496b32386ce6c41ULL,0x01be1c5510adadcdULL,0xc04e67e74bb5faf9ULL,0x3cbaf678e15c9985ULL, + 0x8cd1214550ca4247ULL,0xba1aa47ae7dd30aaULL,0x2f81ddf1e58fee24ULL,0x03452936eec9b0e8ULL,0x8bdc3b81243aea96ULL,0x9a2919af15c3d0e5ULL,0x9ea640ec10948361ULL,0x5ac86d5b6e0bcccfULL, + 0xf892d918c36cf440ULL,0xaed3e837c939719cULL,0xb07b08d2c0218b64ULL,0x6f1bcbbace9790ddULL,0x4a84d6ed60919b8eULL,0xd89007918ac1f9ebULL,0xf84941aa0dd5daefULL,0xb22fe40a67fd62c5ULL, + 0x97e15ba2157f2db3ULL,0xbda2fc8f8e28ca9cULL,0x5d050da437b9f454ULL,0x3d57eb572379d72eULL,0xe9b5eba2fb5ee997ULL,0x01648ca2e11538caULL,0x32bb76f6f6327974ULL,0x338f14b8ff3f4bb7ULL, + 0x524d226ad7ab9a2dULL,0x9c00090d7dfae958ULL,0x0ba5f5398751d8c2ULL,0x8afcbcdd3ab8262dULL,0x57392729e99d043bULL,0xef51263baebc943aULL,0x9feace9320862935ULL,0x639efc03b06c817bULL, + 0x1fe054b366b4be7aULL,0x3f25a9de84a37a1eULL,0xf39ef1ad78d75cd9ULL,0xd7b58f495062c1b5ULL,0x6f74f9a9ff563436ULL,0xf718ff29e8af51e7ULL,0x5234d31315e97fecULL,0xb6a8e2b1292f1c0aULL, + 0xa7f53aa8327720c1ULL,0x956ca322ba092cc8ULL,0x8f03d64a28746c4dULL,0x51fe178266d0d392ULL,0xd19b34db3c832c80ULL,0x60dccc5c6da2e3b4ULL,0x245dd62e0a104cccULL,0xa7ab1de1620b21fdULL, + 0xb293ae0b3893d123ULL,0xf7b75783b15ee71cULL,0x5aa3c61442a9468bULL,0xd686123cdb15d744ULL,0x8c616891a7ab4116ULL,0x6fcd72c8a4e6a459ULL,0xac21911077e5fad7ULL,0xfb6a20e7704fa46bULL, + 0xe839be7d341d81dcULL,0xcddb688932148379ULL,0xda6211a1f7026eadULL,0xf3b2575ff4d1cc5eULL,0x40cfc8f6a7a73ae6ULL,0x83879a5e61d5b483ULL,0xc5acb1ed41a50ebcULL,0x59a60cc83c07d8faULL, + 0x439530b665c7322dULL,0xcf12cc01b3c1b3fbULL,0xc70b01860172f685ULL,0xb915ee221b58391dULL,0x9afdf03ba317db24ULL,0x87dec65917b8ffc4ULL,0x7f46597be4d3d050ULL,0x80a1c1ed006500e7ULL, + 0x3e22a7b397acf4ecULL,0x0426c4005ea8b640ULL,0x5e3295a64e969285ULL,0x22aabc59a6a45670ULL,0xb929714c5f5942bcULL,0x9a6168bdfa3182edULL,0x2216a665104152baULL,0x46908d03b6926368ULL, + 0x52cb8ac6c2babcc1ULL,0x4748d448fe81ae8dULL,0x5844f03f80f1a711ULL,0x3db784b2f8df4ac4ULL,0xad918f122df1fe36ULL,0xe40f25b9f33cc7c0ULL,0x4700d0e73b5e5555ULL,0x5c28fa08b03326f9ULL, + 0xa9f5d8745a1251fbULL,0x967747a8c72725c7ULL,0x195c33e531ffe89eULL,0x609d210fe964935eULL,0xcafd6ca82fe12227ULL,0xaf9b5b960426469dULL,0x2e9ee04c5693183cULL,0x1084a333c8146fefULL, + 0x56dab1c8321a518cULL,0xfd4439a68bce226fULL,0xe0b30d194facb9faULL,0xb5052f307583571bULL,0x1442641012afd476ULL,0xd02e417203fe624aULL,0xfc394f65531c92e6ULL,0x16d4bf5ad4bc0b52ULL, + 0xce06b88210395755ULL,0x117ce6345ec1df80ULL,0xfefae513eff55e96ULL,0xcf36cba6fd7fed1eULL,0x7340eca9a40ebf88ULL,0xe6ec1bcfb3d37e12ULL,0xca51b64e86bbf9ffULL,0x4e0dbb588b40e05eULL, + 0x120ad0bf5f8b8a84ULL,0xbfa00f36866f3edeULL,0xa8c6064ec30ebc2cULL,0xc39e40b823001e3bULL,0x9614f7cde7b1cbabULL,0xcd4420c704a56284ULL,0x8446a8f316857a19ULL,0x519b93b1e6f706cbULL, + 0x96649933aed1d1f7ULL,0x566eaff350563090ULL,0x345057f0ad2e39cfULL,0x148ff65b1f832124ULL,0x042e89d4cf94cf0dULL,0x319bec84520c58b3ULL,0x2a2676265361aa0dULL,0xc86fa3028fbc87adULL, + 0x3805e5b80863b664ULL,0x8be7ac6b3cabdb53ULL,0x9f7d70505dfeff91ULL,0x7dea8bd095896206ULL,0x28005a3b410e3c4eULL,0x24a4f0e9bc603ebfULL,0xcc4fd5ae4aec15d2ULL,0x4dc253f80f96641dULL, + 0x359d7b9c7ea2ee34ULL,0x3fd0d94c09cc3a71ULL,0xbb53c31c3a1ea37aULL,0x533425facf818c87ULL,0x7cd199c3810156e0ULL,0x0ea020e430c16448ULL,0xe557ba094a642542ULL,0xe657e7e79465f5eaULL, + 0xea723ad1fbc82439ULL,0xc726868bf896e9fdULL,0xd97c913c511c33ffULL,0x8a3fa8e2e2114231ULL,0x2a6c0e5608445b3eULL,0x2c4cd884f8d098d3ULL,0x7bf51faf2fea77d4ULL,0x709f208a7b1c4f71ULL, + 0xfc83d2ab5c8b06d5ULL,0xb1a785a2fe4eac46ULL,0xb99315bc846f7779ULL,0xcf31d816ef9ea505ULL,0x2391fe6a15d7dc85ULL,0x2f132b04b4016b33ULL,0x29547fe3181cb4c7ULL,0xdb66d8a6650155a1ULL, + 0x5b9e4843f45aac50ULL,0xc31e042e91eaaad0ULL,0x6e8c0b345a54eea0ULL,0xf0437b94962c7a57ULL,0xe4531ce8fe1d348bULL,0xe1489378e3786432ULL,0xd5e19c4a3f510d38ULL,0x4df3f016ce348b00ULL, + 0x59cd0e8b593d070fULL,0x437575165255625dULL,0x551fdda75b7a0399ULL,0x7bb6e6b02dec1eebULL,0x729bb662334c0922ULL,0x3df631df0cf41b79ULL,0x01abf3c578f32402ULL,0xfcb4666c9cd33c88ULL, + 0xb805b445735e843cULL,0x2a8e890d97379134ULL,0xebc7c10c52ab9f87ULL,0xcbb5e1ecb80a92b6ULL,0xd6ada2d9dc2c4efeULL,0xfccf504eae8cc7bdULL,0x650115acb2418a74ULL,0x8dd90e06c52bd80cULL, + 0x6b66d7e1adc1696fULL,0x98ebe5930acd72d0ULL,0x65f24550cc1b7435ULL,0xce231393b4b9a5ecULL,0x234a22d4db067df9ULL,0x98dda095caff9b00ULL,0x1bbc75a06100c9c1ULL,0x1560a9c8939cf695ULL, + 0xe4050f1cf1c367caULL,0x9bc85a9bc90fbc7dULL,0xa373c4a2e1a11032ULL,0xb64232b7ad0393a9ULL,0xf5577eb0167dad29ULL,0x1604f30194b78ab2ULL,0x0baa94afe829348bULL,0x77fbd8dd41654342ULL, + 0x31f14802fcf0a7fdULL,0x42fd07895488b01eULL,0x71d78d6d9952b498ULL,0x8eb572d907ac5201ULL,0xe0a2a44c4d194a88ULL,0xd2b63fd9ba017e66ULL,0x78efc6c8f888aefcULL,0xb76f6bda4a881a11ULL, + 0xa2f7932c68af43eeULL,0x5502468e703d00bdULL,0xe5dc978f2fb061f5ULL,0xc9a1904a28c815adULL,0xd3af538d470c56a4ULL,0x159abc5f193d8cedULL,0x2a37245f20108ef3ULL,0xfa17081e223f7178ULL, + 0x1fe2a9b2b4b4b67cULL,0xc1d10df0e8020604ULL,0x9d64abfcbc8058d8ULL,0x8943b9b2712a0fbbULL,0x90eed9143b3def04ULL,0x85ab3aa24ce775ffULL,0x605fd4ca7bbc9040ULL,0x8b34a564e2c75dfbULL, + 0x5c18acf88e2f7d90ULL,0xfdbf33d777be32cdULL,0x0a085cd7d2eb5ee9ULL,0x2d702cfbb3201115ULL,0xb6e0ebdb85c88ce8ULL,0x23a3ce3c1e01d617ULL,0x3041618e567333acULL,0x9dd0fd8f157edb6bULL, + 0xb2b2610798fa7aaaULL,0x41209ee4f073aa4eULL,0xf1570359f2d6b19bULL,0xcbe6868cfc577cafULL,0x186c4bdc32c04dd3ULL,0xa6c35faecfeee397ULL,0xb4a1b312f086c0cfULL,0xe0a5ccc6d9461fe2ULL, + 0x516ff3a36fa6110cULL,0x74fb1eb1fb93561fULL,0x6c0c90478457522bULL,0xcfd321046bb8bdc6ULL,0x2d6884a2cc80ad57ULL,0x7c27fc3586a9b637ULL,0x3461baedadf4e8cdULL,0x1d56251a617242f0ULL, + 0xb84011a9431dd80eULL,0xeb7c7cca73306cd9ULL,0x20fadd29d1b3b730ULL,0x83858b5bfe37b3d3ULL,0xbf4cd193b6251d5cULL,0x1cca1fd31352d952ULL,0xc66157a490fbc051ULL,0x7990a63889b98636ULL, + 0x892c81a321175ec1ULL,0x9159a505ee018109ULL,0xc70130532d8be316ULL,0x76060c21426fa2e5ULL,0x074d2dfc6b6f0f22ULL,0x9725fc64ca01a671ULL,0x3f6679b92770bd8eULL,0x8fe6604fd7c9b3feULL, + 0xce711154b6e00a84ULL,0xd9fe7e4224890e60ULL,0xd10bc6c34560988fULL,0xbdc2ef526859b004ULL,0xdcf0d868d5c890eeULL,0x893115e6119c47dcULL,0xe97966fbee714567ULL,0x117813355c85aa53ULL, + 0x71d530cc73204349ULL,0xc9df473d94a0679cULL,0xc572f0014261e031ULL,0x9786b71f22f135feULL,0xed6505fa6b64e56fULL,0xe2fb48e905219c46ULL,0x0dbec45bedf53d71ULL,0xd7d782f2c589f406ULL, + 0x350fb66b73ed0966ULL,0x968e4b082886032bULL,0x5a16ed6e88390493ULL,0x0a83ce84a121edbeULL,0x7c86fc10a3d9cda0ULL,0x5a40a6d595ce67fbULL,0x6cb8cda7f937dbf6ULL,0x95b44768651f6283ULL, + 0x06513c8a446cd7f4ULL,0x158c423b906d52a6ULL,0x71503261c423866cULL,0x4b96f57093c148eeULL,0x5daf9cc7239a8523ULL,0x611b597695ac4b8bULL,0xde3981db724bf7f6ULL,0x7e7d0f7867afc443ULL, + 0x984f101ed6fc3837ULL,0x340bf99f5e1b3a09ULL,0xbb96036f06942626ULL,0x7bc878ab0c7da618ULL,0xb37416441c6fb035ULL,0xc65bd5aea182fe9fULL,0x1b9c2fb86cc7a67aULL,0x8d1b19af5ce68d7dULL, + 0x3d1ab80c8ce59954ULL,0x742c5a9478222ac0ULL,0x3ddacbf894f878ddULL,0xfc085117e7d54a99ULL,0xfb0f1dfa21e38ec2ULL,0x1c7b59cb16f4ff7fULL,0x988752397ea888feULL,0x705d270cb10dc889ULL, + 0xe5aa692a87dec0e1ULL,0x010ded8df7b39d00ULL,0x7b1b80c854cfa0b5ULL,0x66beb876a0f8ea28ULL,0x50d7f5313476cd0eULL,0xa63d0e65b08d3949ULL,0x1a09eea953479fc6ULL,0x82ae9891f499e742ULL, + 0xab58b9105ca7d866ULL,0x582967e23adb3b34ULL,0x89ae4447cceac0bcULL,0x919c667c7bf56af5ULL,0x9aec17b160f5dcd7ULL,0xec697b9fddcaadbcULL,0x0b98f341463467f5ULL,0xb187f1f7a967132fULL, + 0xeb5ddcb6ec7fae9fULL,0x995f2714efb66e5aULL,0xdee95d8e69445d52ULL,0x1b6c2d4609e27620ULL,0x32621c318129d716ULL,0xb03909f10958c1aaULL,0x8c468ef91af4af63ULL,0x162c429ffba5cdf6ULL, + 0xe8cb5eef9c053df7ULL,0x8de25b37b300ea6fULL,0xdb03fa92c849cffbULL,0x242e43a7e84169bbULL,0xe4fa51f4dd6f958eULL,0x6925a77ff4445a8dULL,0xe6e72a50e90d8949ULL,0xc66648e32b1f6390ULL, + 0x6c3b96f31711ebecULL,0x2da40f1fce98fdc4ULL,0xb99774d357b4411fULL,0x87c8bdf415b65bb6ULL,0xda3a89e3c2eef12dULL,0xde95bb9b3c7471f3ULL,0x600f225bd812c594ULL,0x54907c5d2b75a56bULL, + 0x699e4d2945c1dd53ULL,0xcadc5898231debb5ULL,0xdf49fcc7a77f00e0ULL,0x93057bbfa73e5a0eULL,0x2f8b7ecd027a4cd1ULL,0x114734b3c614011aULL,0xe7a01db767677c68ULL,0x89d9be5e7e273f4fULL, + 0xe0deee5931fba239ULL,0xf47424d398bd91d1ULL,0x0f8886f4071a3c1dULL,0x3f7d41e8a819233bULL,0x708623c2cf6eb998ULL,0x86bb49af609a287fULL,0x942bb24963c90762ULL,0x0ef6eea555a9654bULL, + 0x4add4a2e649d4e57ULL,0xcd53a2b01917526eULL,0xc526233020b44ac4ULL,0x4028746abaa2c31dULL,0x5131839064291d4cULL,0xbf48f151ee5ad909ULL,0xcce57f597b185681ULL,0x7c3ac1b04854d442ULL, + 0xa80d1db6f79588c0ULL,0xfa52fc69b55768ccULL,0x0b4df1ae7f54438aULL,0x0cadd1a7f9b46a4fULL,0xb40ea6b31803dd6fULL,0x488e4fa555eaae35ULL,0x9f047d55382e4e16ULL,0xc9b5b7e02f6e0c98ULL, + 0xc19972d0b611c24bULL,0x1d468e6560a8f351ULL,0xeb7580697bcf6421ULL,0xec9dd0ee88fbc491ULL,0x5b59d2bf956c2e32ULL,0x73dc6864dcddf94eULL,0xfd5e2321bcee7665ULL,0xa7b4f8ef5e9a06c4ULL, + 0x03cc8f17cf41c6e8ULL,0xf1f03c2a037b925cULL,0xc39c19cc66d2427cULL,0x823d24ba7b6c18e4ULL,0x32ef9013901f0b4fULL,0x684360f1f8941c2eULL,0x0ebaff522c28092eULL,0x7891e4e3256c932fULL, + 0x174e8f82d8d38a9bULL,0x2e97c600e7de1391ULL,0xc5709850a1c175ddULL,0x969041a032ae5035ULL,0xcbfd533b76a2086bULL,0xd6bba71bd7c2e8feULL,0xb2d58ee6099dfb67ULL,0x3a8b342d064a85d9ULL, + 0xf83cbf0502f40d9aULL,0x4681c4682c318a4dULL,0x985756180e9c2674ULL,0xbe79d0461847092eULL,0xaf1e480a78bd01e0ULL,0x6dd359e472a51db9ULL,0x62ce3821e3afbab6ULL,0xc5cee5b617733199ULL, + 0x671ed8fc3b922bf8ULL,0xe4d8c0a04c29b133ULL,0x87eb12393b6e99c4ULL,0xaff3974c8793bebaULL,0x037494052c18df9bULL,0xc5c3a29391007139ULL,0x6a77234fe37a0b95ULL,0x02c29a21b661c96bULL, + 0x5a52fe2e34d74e31ULL,0xa352c3103bf79ab6ULL,0x97ff6c5aabfeeb8fULL,0xbfbe8feff5c97305ULL,0xd6081ce6a7904608ULL,0x1f812f3ac4fca249ULL,0x9b24bc9ab9e5e200ULL,0x91022c6738012ee8ULL, + 0x184de7d7cc5f4394ULL,0xb5551b5c4536e142ULL,0x2e89b212d34aa60aULL,0x14a96feaf50051d5ULL,0x4e21ef740d12bb0bULL,0xc522f02060b9677eULL,0x8b12e4672df7731dULL,0x39f803827b326d31ULL, + 0xc12738b67c4a658aULL,0xb3c4763940e72182ULL,0x3b77be468798e44fULL,0xdc047df217a7f85fULL,0x2439d4c55e59d92dULL,0xcedca475e8e64d8dULL,0xa724cd0d87ca9b16ULL,0x35e4fd59a5540dfeULL, + 0x9894344f3a29467aULL,0xde81e949c51eba6dULL,0xdaea066ba5e5c2f2ULL,0x3fc8a61408c8c7b3ULL,0x7adff88f06d0de9fULL,0xbbc11cf53b75ce0aULL,0x9fbb7accfbbc87d5ULL,0xa1458e267badfde2ULL, + 0x1cb43668e039c256ULL,0x5f26fb8b7c17fd5dULL,0xeee426af79aa062bULL,0x072002d0d78fbf04ULL,0x4c9ca237e84fb7e3ULL,0xb401d8a10c82133dULL,0xaaa525926d7e4181ULL,0xe943083373dbb152ULL, + 0x2f5fad1e6ee7c983ULL,0xeb0f9d7cb41328f5ULL,0x9ba68b441d78d5f7ULL,0xa06b3b9e35bc726fULL,0xa2550255593e1ff1ULL,0x552dd43ddfbec115ULL,0x2c48a7abbba8f046ULL,0xd4fc56a3ad0bf133ULL, + 0xf92dda31be24319aULL,0x03f7d28be095a8e7ULL,0xa52fe84098782185ULL,0x276ddafe29c24dbcULL,0x80cd54961d7a64ebULL,0xe43608897f1dbe42ULL,0x2f81a8778438d2d5ULL,0x7e4d52a885169036ULL, + 0x62f21cefba04b5d1ULL,0x9a442707224c7352ULL,0xbf07966c33c6171fULL,0xdb7ba8911e0816b0ULL,0x306fea59033745a9ULL,0x2aacf7e0c0a78f67ULL,0xb5aa3883ad251bf9ULL,0x345aede926bd7086ULL, + 0x19e3d5b11d59715dULL,0xc7eaa762d788983eULL,0xe5a730b0abf1f248ULL,0xfbab8084fae3fd83ULL,0x65e50d2153765b2fULL,0xbdd4e083fa127f3dULL,0x9cf3c074397b1b10ULL,0x59f8090cb1b59fd3ULL, + 0xdc1de17d98119f10ULL,0x74353c5d488c36a6ULL,0x14aaf33a3d8e23dfULL,0x31e075c078baf593ULL,0x0f7ca03a46d1ca3cULL,0x99c5e3ac47b660c7ULL,0x70d0241388fe2e59ULL,0x2e9a6be12a7ec005ULL, + 0x7b15fd9d615faa8fULL,0x8fa1eb40968554edULL,0x7bb4447e7aa44882ULL,0x2bb2d0d1029fff32ULL,0x075e2a646caa6d2fULL,0x8eb879de22e7351bULL,0xbcd5624e9a506c62ULL,0x218eaef0a87e24dcULL, + 0xac449695241fbd6fULL,0x67c9b170081c1223ULL,0x16868f21b56aac6fULL,0x34bd8fa3f8bcb721ULL,0x06b6bd33b6691c76ULL,0x6c924766381a7973ULL,0x6a12444ca54078dbULL,0xd02e91a96d1051ccULL, + 0x37e5684744ddfa35ULL,0x9ccfc5c5dab3f747ULL,0x9ac1df3f1ee96cf4ULL,0x0c0571a13b480b8fULL,0x2fbeb3d54b3a7b3cULL,0x35c036695dcdbb99ULL,0x52a0f5dcb2415b3aULL,0xd57759b44413ed9aULL, + 0x077379c00b33d3f8ULL,0x421883c67064e409ULL,0x2d0873d76c29c8f6ULL,0xbfa433a3d274c0c8ULL,0x56dc778f23a5891eULL,0xd663bf6535e2de04ULL,0x488fdb485db517ceULL,0x00bba55e19b226c2ULL, + 0x1fe647d83d30a2c5ULL,0x0857f77ef78a81dcULL,0x11d5a334131a4a9bULL,0xc0a94af929d393f5ULL,0xbc3a5c0bdaa6ec1aULL,0xba9fe49388d2d7edULL,0xbb4335b4bb614797ULL,0x991c4d6872f83533ULL, + 0xedbbeee78a058fb6ULL,0xb9d19ddcfb09121aULL,0xa41bb45bd34dddceULL,0x2dbc80b900964bc4ULL,0x4ed9137d1d6cb654ULL,0x1b9016db483d01c5ULL,0x5fc501bc6528e22eULL,0xb2d2f8816cad646bULL, + 0x53258c28d2f01cb3ULL,0x93d6eaa3d75db0b1ULL,0x419a2b0de87d0db4ULL,0xa1e48f03d8fe8493ULL,0xf747faf6c508b23aULL,0xf137571a35d53549ULL,0x9f5e58e2fcf9b838ULL,0xc7186ceea7fd3cf5ULL, + 0x5e76fb2f286bad39ULL,0xbad9efe39dcad1e2ULL,0x60e75190edc7e904ULL,0x6a6f063e0fecb5a5ULL,0x5150ed85aed8acc3ULL,0xb56ccfbc6d20af6cULL,0x7e0d1e982c69dbfaULL,0xabf5628a7c7e10a9ULL, + 0x77b868cee978a1d3ULL,0xe3a68b337ab92d04ULL,0x5102979487a5b862ULL,0x5f0606c33a61d41dULL,0x2814be276f9326f1ULL,0x2f521c14c6fe3c2eULL,0x17464d7dacdf7351ULL,0x10f5f9d3777f7e44ULL, + 0xb06b1244c5f95cd8ULL,0xda8c8af0f4ab95f4ULL,0x1bae59c2b9e5836dULL,0x07d51e7e3acffffcULL,0x01e15e6ac2ccbcdaULL,0x3bc1923f8528c3e0ULL,0x43324577a49fead4ULL,0x61a1b8842aa7a711ULL, + 0x4fe7ee31b0e63d34ULL,0xf4600572a9e54fabULL,0xc0493334d5e7b5a4ULL,0x8589fb9206d54831ULL,0xaa70f5cc6583553aULL,0x0879094ae25649e5ULL,0xcc90450710044652ULL,0xebb0696d02541c4fULL, + 0x172a5247d95ea168ULL,0x1758fada2970764aULL,0xac803a511d978169ULL,0x299cfe2ede77e01bULL,0x652a1e17b0a98927ULL,0x2e26e1d120014495ULL,0x7ae0af9f7175b56aULL,0xc2e22a80d64b9f95ULL, + 0x758c1a3ea2dee7a6ULL,0xdcde2f3c734b2284ULL,0xaba445d24eaba6adULL,0x35aaf66876cee0a7ULL,0x7e0b04a9e5aa049aULL,0xe74083ad91103e84ULL,0xbeb183ce40afecc3ULL,0x6b89de9fea043f7aULL, + 0xafbfb1eda4f7e665ULL,0x403cce6aa23df8e7ULL,0xb49cc83f1312c2f4ULL,0xe1cc2366771a9c34ULL,0x7ab6a6c0db92faacULL,0xacd15e0dec3befe1ULL,0x7f8ed988583a0f36ULL,0x1821de7705f9be09ULL, + 0xb99f0e0399375235ULL,0x7614c847b9917970ULL,0xfec93ce9524ec067ULL,0xe40e7bf89b122520ULL,0xb5670631ee4c4774ULL,0x6f03847a3b04914cULL,0xc96e9429dc9dd226ULL,0x43489b6c8c57c1f8ULL, + 0x7b9722a5c5f26464ULL,0xca4c3dfba442809aULL,0x7d10986723644810ULL,0x9e4951723c924f82ULL,0xef4a6968a2c5bc14ULL,0x750eac4f68de6b7aULL,0x4e01884d52a2cbb5ULL,0xac40830af4a5f446ULL, + 0x0e299d23fe67ba66ULL,0x9145076093cf2f34ULL,0xf45b5ea997fcf913ULL,0x5be008438bd7dddaULL,0x358c3e05d53ff04dULL,0xbf7ccdc35de91ef7ULL,0xad684dbfb69ec1a0ULL,0x367e7cf2801fd997ULL, + 0x8c54f1076103adf8ULL,0x2d813d6cbe8e9810ULL,0xb1466fa85fbd3c9bULL,0x68c65d2240e1ca76ULL,0xb81baa40255f9164ULL,0x5b34c3eed1a864b2ULL,0x3602209b122ca141ULL,0xe7d7248e885badebULL, + 0x46ffd227cc2338fbULL,0x89ff6fa990e26153ULL,0xbe570779331a0076ULL,0x43d241c506e1f3afULL,0xfdcdb97dde9b62a3ULL,0x6a06e984a0ae30eaULL,0xc9bf16804fbddf7dULL,0x170471a2d36163c4ULL, + 0xfd23e207a6469d43ULL,0xcb9f5f112f753a85ULL,0xde2625d4fbb5ca72ULL,0x82e4e54ab7b1c78bULL,0x2cd0ca5378b9e814ULL,0xfcd44051125b817aULL,0xb68f719f30cfd965ULL,0x31644719d848a974ULL, + 0xff5ba8ae3113655eULL,0xfa2c6e2b57b83180ULL,0x1c48271977e0eabeULL,0xf9f3c555337fea97ULL,0x340f7022a42581cbULL,0xe1de0bc218f710e3ULL,0xee640adef62e5aa8ULL,0x16b2389149428940ULL, + 0x5045738f25f653f5ULL,0xe42b8cb83764f635ULL,0xb4f89406dc11ffc3ULL,0x99593144b6b3e4aaULL,0x81c849f3c9740052ULL,0x2c9cf4c155ffc48bULL,0x6299e52177f67a49ULL,0x5869f6e3c00c6c62ULL, + 0x361619e455950cc3ULL,0xc71d665c56b66bb8ULL,0xea034b34afac6d84ULL,0xa987f832e5e4c7e3ULL,0xa07427727a79a6a7ULL,0x56e5d017e26d6c23ULL,0x7e50b97638167e10ULL,0xaa6c81efe88aa84eULL, + 0xb84186f75fe01576ULL,0x446e276cdea51395ULL,0xf3c5ef8105f3f8d4ULL,0x3d7f7df674f7f142ULL,0x7c69b565f9ef1656ULL,0x87efa4247c414ef6ULL,0xeb7e620d3d292060ULL,0x50b1de346eec21aeULL, + 0x0ca1f3b7b0dc8595ULL,0x27de46089f1d9f2eULL,0x1af3bf39badd82a7ULL,0x79356a7965862448ULL,0xc0602345f5f9a052ULL,0x1a8b0f89139a42f9ULL,0xb53eee42844d40fcULL,0x93b0bfe54e5b6368ULL, + 0x0f893a5dc8de610bULL,0xe8c515fb67e223ceULL,0x7774bfa64ead6dc5ULL,0x89d20f95925c728fULL,0x7a1e0966098583ceULL,0xa2eedb9493f2a7d7ULL,0x1b2820974c304d4aULL,0x0842e3dac077282dULL, + 0xa1010e9d74cd06ffULL,0x9c17c7dfaca3eeacULL,0x74c86cd38063aa2bULL,0x8595c4b3734614ffULL,0xa3de00ca990f62ccULL,0xd9bed213ca0c3be5ULL,0x7886078adf8ce9f5ULL,0xddb27ce35cd44444ULL, + 0x5a3097befc15aa1eULL,0x40d12548b54b0745ULL,0x5bad4706519a5f12ULL,0xed03f717a439dee6ULL,0x0794bb6c4a02c499ULL,0xf725083dcffe71d2ULL,0x2cad75190f3adcafULL,0x7f68ea1c43729310ULL, + 0x9c7c581d26ee8382ULL,0xcf17dcc5359d638eULL,0xee8273abb728ae3dULL,0x1d112926f821f047ULL,0x1149847750491a74ULL,0x687fa761fde0dfb9ULL,0x2c2580227ea435abULL,0x6b8bdb9491ce7e3fULL, + 0x9c806d8af7f91d0fULL,0x3b61b0f1a82a5728ULL,0x4640032d94d76754ULL,0x273eb5de47d834c6ULL,0x2988abf77b4e4d53ULL,0xb7ce66bfde401777ULL,0x9fba6b32715071b3ULL,0x82413c24ad3a1a98ULL, + 0x75537b7e3cc8ac85ULL,0x8d725f57dd02753bULL,0xfd05ff64b737df2fULL,0x55fe8712f6d2531dULL,0x57ce04a96ab6b01cULL,0x69a02a897cd93724ULL,0x4f82ac35cf86699bULL,0x8242d3ad9cb4b232ULL, + 0x69c435269be47be0ULL,0x323b7dd8cb28fea1ULL,0xfa5538ba3a6c67e5ULL,0xef921d701d378e46ULL,0xf92961fc3c4b880eULL,0x3f6f914e98940a67ULL,0xa990eb0afef0ff39ULL,0xa6c2920ff0eeff9cULL, + 0xb23a03a553fb2b56ULL,0x6ce141e74e057f78ULL,0x796525c389e490d9ULL,0x0bc95725a31a7e75ULL,0x1ec567911220fd06ULL,0x716e3a3c408b0bd6ULL,0x31cd6bf7e8ebeba9ULL,0xa7326ca6bee6b670ULL, + 0x70b63d32343bf1a9ULL,0x8fd3bd2837d1a6b1ULL,0x0454879c316865b4ULL,0xee959ff6c458efa2ULL,0x0461dcf89706dc3fULL,0x737db0e2164e4b2eULL,0x092626802f8843c8ULL,0x54498bbc7745e6f6ULL, + 0x5341352b5acf6e10ULL,0xc50343fdafe652c3ULL,0x4af3792d18577a7fULL,0xe1a4c617af16823dULL,0x9b26d0cd33425d0aULL,0x306399ed9b7bc47fULL,0x2a792f33706bb20bULL,0x3121961498111055ULL, + 0x4c1f428cd5f30851ULL,0x94dfed272a4f6630ULL,0x4df53772fc5d48a4ULL,0xdd2d5a2f933260ceULL,0x574115bdd44cc7a5ULL,0x4ba6b20dbd12533aULL,0x30e93cb8243057c9ULL,0x794c486a14de320eULL, + 0x6095355699f241d7ULL,0xee4adbd7001a349dULL,0x0b35bf6aaa89e491ULL,0x7f0076f4136f7546ULL,0xd19a18ba9264da3dULL,0x6eb2d2cd62a7a28bULL,0xcdba941f8761c971ULL,0x1550518ba3be4a5dULL, + 0xc232d97302f1cd1eULL,0xce87eacb1dd212a4ULL,0x6e4c8c73e69802f7ULL,0x12ef02901fffddbdULL,0x941ec74e1bcea6e2ULL,0xd0b540243cb92cbbULL,0x809fb9d47e8f9d05ULL,0x3bf16159f2992aaeULL, + 0xb2497007eafbb1e1ULL,0xd75c9ce6e75b7a93ULL,0x3558352defb68d78ULL,0xa2f26699223f6396ULL,0xeb911ecfe469b17aULL,0x62545779e72d3ec2ULL,0x8ea47de782cb113fULL,0xebe4b0864e1fa98dULL, + 0xbdb8e675b055cb40ULL,0x898f8e7b977b5167ULL,0xecc65651b82fb863ULL,0x565448146d88f01fULL,0xb0928e95263a75a9ULL,0xcfb6836f1a22fcdaULL,0x651d14db3f3bd37cULL,0x1d3837fbb6ad4664ULL, + 0x20d3c982cf7d62d2ULL,0x1f36e29d23ba8150ULL,0x48ae0bf092763f9eULL,0x7a527e6b1d3a7007ULL,0xb4a89097581a85e3ULL,0x1f1a520fdc158be5ULL,0xf98db37d167d726eULL,0x8802786e1113e862ULL, + 0xefb2149e36f09ab0ULL,0x03f163ca4a10bb5bULL,0xd029704506e20998ULL,0x56f0af001b5a3babULL,0x7af4cfec70880e0dULL,0x7332a66fbe3d913fULL,0x32e6c84a7eceb4bdULL,0xedc4a79a9c228f55ULL, + 0xc37c7dd0c55c4496ULL,0xa6a9635725bbabd2ULL,0x5b7e63f2add7f363ULL,0x9dce37822e73f1dfULL,0xe1e5a16ab2b91f71ULL,0xe44898235ba0163cULL,0xf2759c32f6e515adULL,0xa5e2f1f88615eecfULL, + 0x74519be7abded551ULL,0x03d358b8c8b74410ULL,0x4d00b10b0e10d9a9ULL,0x6392b0b128da52b7ULL,0x6744a2980b75c904ULL,0xc305b0aea8f7f96cULL,0x042e421d182cf932ULL,0xf6fc5d509e4636caULL, + 0x795847c9d64cc78cULL,0x6c50621b9b6cb27bULL,0x07099bf8df8022abULL,0x48f862ebc04eda1dULL,0xd12732ede1603c16ULL,0x19a80e0f5c9a9450ULL,0xe2257f54b429b4fcULL,0x66d3b2c645460515ULL, + 0x6ca4f87e822e37beULL,0x73f237b4253bda4eULL,0xf747f3a241190aebULL,0xf06fa36f804cf284ULL,0x0a6bbb6efc621c12ULL,0x5d624b6440b80ec6ULL,0x4b0724257ba556f3ULL,0x7fa0c3543e2d20a8ULL, + 0xe921fa31e3229d41ULL,0xa929c65294531bd4ULL,0x84156027a6d38209ULL,0xf3d69f736bdb97bdULL,0x8906d19a16833631ULL,0x68a34c2e03d51be3ULL,0xcb59583b0e511cd8ULL,0x99ce6bfdfdc132a8ULL, + 0x3facdaaaffcdb463ULL,0x658bbc1a34a38b08ULL,0x12a801f8f1a9078dULL,0x1567bcf96ab855deULL,0xe08498e03572359bULL,0xcf0353e58659e68bULL,0xbb86e9c87d23807cULL,0xbc08728d2198e8a2ULL, + 0x8de2b7bc453cadd6ULL,0x203900a7bc0bc1f8ULL,0xbcd86e47a6abd3afULL,0x911cac128502effbULL,0x2d550242ec965469ULL,0x0e9f769229e0017eULL,0x633f078f65979885ULL,0xfb87d4494cf751efULL, + 0xe1790e4bfc25419aULL,0x364672034bff3cfdULL,0xc8db638625b6e83fULL,0x6cc69f236cad6fd2ULL,0x0219e45a6bc68bb9ULL,0xe43d79b6297f7334ULL,0x7d445368465dc97cULL,0x4b9eea322a0b949aULL, + 0x1b96c6ba6102d021ULL,0xeaafac782f4461eaULL,0xd4b85c41c49f19a8ULL,0x275c28e4cf538875ULL,0x35451a9ddd2e54e0ULL,0x6991adb50605618bULL,0x5b8b4bcd7b36cd24ULL,0x372a4f8c56f37216ULL, + 0xc890bd73a6a5da60ULL,0x6f083da0dc4c9ff0ULL,0xf4e14d94f0536e57ULL,0xf9ee1edaaaec8243ULL,0x571241ec8bdcf8e7ULL,0xa5db82710b041e26ULL,0x9a0b9a99e3fff040ULL,0xcaaf21dd7c271202ULL, + 0xb4e2b2e14f0dd2e8ULL,0xe77e7c4f0a377ac7ULL,0x69202c3f0d7a2198ULL,0xf759b7ff28200eb8ULL,0xc87526eddcfe314eULL,0xeb84c52453d5cf99ULL,0xb1b52ace515138b6ULL,0x5aa7ff8c23fca3f4ULL, + 0xff0b13c3b9791a26ULL,0x960022dacdd58b16ULL,0xdbd55c9257aad2deULL,0x3baaaaa3f30fe619ULL,0x9a4b23460d881efdULL,0x506416c046325e2aULL,0x91381e76035c18d4ULL,0xb3bb68bef27817b0ULL, + 0x15bfb8bf5116f937ULL,0x7c64a586c1268943ULL,0x71e25cc38419a2c8ULL,0x9fd6b0c48335f463ULL,0x4bf0ba3ce8ee0e0eULL,0x6f6fba60298c21faULL,0x57d57b39ae66bee0ULL,0x292d513022672544ULL, + 0xf451105dbab093b3ULL,0x012f59b902839986ULL,0x8a9158023474a89cULL,0x048c919c2de03e97ULL,0xc476a2b591071cd5ULL,0x791ed89a034970a5ULL,0x89bd9042e1b7994bULL,0x8eaf5179a1057ffdULL, + 0x6066e2a2d551ee10ULL,0x87a8f1d8727e09a6ULL,0x00d08bab2c01148dULL,0x6da8e4f1424f33feULL,0x466d17f0cf9a4e71ULL,0xff5020103bf5cb19ULL,0xdccf97d8d062ecc0ULL,0x80c0d9af81d80ac4ULL, + 0x98857ceb1bf4581cULL,0xe635e186aca7b166ULL,0x278ddd22659722acULL,0xa0903c4c1db68007ULL,0x366e458948f21402ULL,0x31b49c14b96abda2ULL,0x329c4b09e0403190ULL,0x97197ca3d29f43feULL, + 0x03e2de1cf3480d4aULL,0xf0d8edc7bc8acf1aULL,0xf23e330368295a9cULL,0xfadd5f68c546a97dULL,0x895597ad96f8acb1ULL,0xbddd49d5671bdae2ULL,0x16fcd52821dd43f4ULL,0xa5a454126619141aULL, + 0xfa32c79f439f29cfULL,0x7bf321c04dd82a3bULL,0xff127f54eaa2c1b1ULL,0xa8365f2df35e9618ULL,0x852d29024d0ef8ddULL,0x395ce2c159228c4aULL,0xb69f44e8215afed3ULL,0x16c1f898b27458e3ULL, + 0x8ce9b6bfc360e25aULL,0xe6425195075a1a78ULL,0x9dc756a8481732f4ULL,0x83c0440f5432b57aULL,0xc670b3f1d720281fULL,0x2205910ed135e051ULL,0xded14b0edb052be7ULL,0x697b3d27c568ea39ULL, + 0xdef29005dc453233ULL,0xc208c47a2fac4bcfULL,0x6057a3feac3d55acULL,0x1723725d902c1207ULL,0x9c31c62733eb0fecULL,0x4913ccdbbb3c63bbULL,0x113e542b07305838ULL,0x9d48e72a310c2d97ULL, + 0x54424ec4d9bbeb3dULL,0x34ceafe3d7b2921cULL,0x5e68022e296d37c5ULL,0xa28e0a2b359f16b0ULL,0xfdd82dd9bc3f9d73ULL,0x6939a8f9baf3e1ffULL,0x55cff45c31736dc4ULL,0x910f56427892e8e7ULL, + 0xde09b349dfe39260ULL,0x984612f773549093ULL,0x7cede28167853b02ULL,0x7d809bb429d17703ULL,0x450b6bfb2756c4f0ULL,0xc59ff9ba93f02b80ULL,0x3e69545720ad9561ULL,0x0c7cf0be3331e2c7ULL, + 0x2e599b9afb3ff9edULL,0x28c2e0ab17f6515cULL,0x1cbee4fd474da449ULL,0x071279a44f364452ULL,0x97abff6601fbe855ULL,0x3ee394e85fda51c4ULL,0x190385f667597c0bULL,0x6e9fccc6a27ee34bULL, + 0xced2d419362fb228ULL,0x894637c206aa0be4ULL,0x7a4fc55eb294b197ULL,0xd9cbac1cfd4ca1e8ULL,0x068b74800ccdb6ceULL,0xca4c556580dfaa49ULL,0xe835382176033b78ULL,0x35db7525c5ce98a7ULL, + 0x8cf5927274fd1997ULL,0x987d2031ffadeb58ULL,0x5647c6c3d4db260cULL,0xf08bb13b985543beULL,0x566eeb1056fad695ULL,0x39e17dde68334cc7ULL,0x8a97b3bb7ddd1db2ULL,0xf91199d8c1c5a300ULL, + 0x402fa437bdc47fe4ULL,0x35bd25234f3751f0ULL,0x8c2281b7cf57d485ULL,0x50083ea58f607cb6ULL,0x6f41e480df6d730dULL,0x1164e47e91bb06a2ULL,0xd9040c22391fb48aULL,0xe12df251da23dda7ULL, + 0x0b89de9314092ebbULL,0xf17256bd428e240cULL,0xcf89a7f393d2f064ULL,0x4f57841ee1ed3b14ULL,0x4ee14405e708d855ULL,0x856aae7203f1c3d0ULL,0xc8e5424fbdd7eed5ULL,0x3333e4ef73ab4270ULL, + 0xf9cf2b5148f835b1ULL,0x4a70faf32adaaba8ULL,0xa0d2e24fed7beadeULL,0x2b1e4e6715f04032ULL,0xeadbc0b91fb3ceeeULL,0xb817b0863a54b0b4ULL,0x14a787257fd4a188ULL,0xff13304623482e60ULL, + 0x312012ec4efe42dcULL,0x7251b2c2c664b2bcULL,0x996c2e6b798f9cb9ULL,0x3543376a3b8f3465ULL,0x337ae8416fbfb6a9ULL,0xe0893a840cb91a03ULL,0x53744e9d7b02d855ULL,0x7e673186206d473fULL, + 0x9509996f3ef143dbULL,0x9cf1e82e399378bcULL,0xb59cd09b0b8ccc89ULL,0x64b4a2fc52cda6beULL,0xd3bda8b31bcd55dfULL,0xfec4e87d2507cb37ULL,0x3d48a85eb610e49bULL,0x02fbe1bd1f9cc445ULL, + 0x3bc77adedda492f8ULL,0xc11a3aea78297205ULL,0x5e89a3e734931b4cULL,0x17512e2e9f5694bbULL,0x5dc349f3177bf8b6ULL,0x232ea4ba08c7ff3eULL,0x9c4f9d16f511145dULL,0xccf109a333b379c3ULL, + 0x60e2857080eb24a9ULL,0x7bedfb4d488e0cfdULL,0x721ebbd7c259cdb8ULL,0x0b0da855bc6390a9ULL,0x2b4d04dbde314c70ULL,0xcdbf1fbc6c32e846ULL,0x33833eabb162fc9eULL,0x9939b48bb0dd3ab7ULL, + 0x96892c1f711b0eb9ULL,0xb905f2c8780ab954ULL,0xace26309a20792dbULL,0xec8ac9b30684e126ULL,0x486ad8b6b40a2447ULL,0x60121fc19fe3fb24ULL,0x5626fccf1a8e3b3fULL,0x4e5686226ad1f394ULL, + 0x5cfbbb22236f4a98ULL,0x0b0c59e9066800bbULL,0x4ac69a8f5a9a7774ULL,0x2b33f804d6bec948ULL,0xb372929532e6c466ULL,0x68956d0f4e599c73ULL,0xa47a249f155c31ccULL,0x24d80f0de1ce284eULL, + 0x5a4b46c64a8a3d62ULL,0x8469c4d0247743d2ULL,0x2bb3a13d88f7e433ULL,0x62b23a1001be5849ULL,0xe83596b4a63d1a4cULL,0x454e7fea7d183f3eULL,0x643fce6117afb01cULL,0x4e65e5e61c4c3638ULL, + 0xb7e830e3dc09508bULL,0xfaf6d2cf74317655ULL,0x72606cebdf690355ULL,0x48bb92b3d0c3ded6ULL,0x65b754845c7cf892ULL,0xf6cd7ac9d5d5f01fULL,0xc2c30a5996401d69ULL,0x91268650ed921878ULL, + 0xe5db77176add8545ULL,0x1b71cb6672c49b66ULL,0xd856073968421d77ULL,0x03840fe883e3afeaULL,0xb391dad51ec69977ULL,0xae243fb9307f6726ULL,0xc88ac87be8ca160cULL,0x5174cced4ce355f4ULL, + 0x752067f8ff81578eULL,0x786221509045447dULL,0xc0c22fcf0505aa6fULL,0x1030f0a66bed1c77ULL,0x31f29f151f0bd739ULL,0x2d7989c7e6debe85ULL,0x5c070e728e677e98ULL,0x0a817bd306e81fd5ULL, + 0xc1e17eb6cbc613e5ULL,0x33131d55497ea61cULL,0x2f69d39eaf7eded5ULL,0x73c2f434de6af11bULL,0x4ca52493a4a375faULL,0x5f06787cb833c5c2ULL,0x814e091f3e6e71cfULL,0x76451f578b746666ULL, + 0x01c7e082e1539388ULL,0xfd286f30759a9c6bULL,0x94581041176bacfbULL,0xe580f07c3bc3de53ULL,0xdcdb6f75884d772fULL,0xd75c3bb840bb9d4fULL,0xdc6c2e4edb083011ULL,0xe18789a2cd9c298dULL, + 0x5ee6ab8495fe1347ULL,0xab0f6c396f24503cULL,0x807e3ffb4486dd6bULL,0xf00b6c748002fef5ULL,0x48bff9a6a7862999ULL,0x85e5a06cbed89e26ULL,0x86d311af3d8419ebULL,0x24f3ad7834733f16ULL, + 0x5e20551311820d44ULL,0x0c651bcc86c4473aULL,0x1d230c2b9bd80eefULL,0x042c4a1207515be5ULL,0x42517ca0bfe9e284ULL,0xe8f605782369827aULL,0x184f04f01638699dULL,0x174618edf2bc6d05ULL, + 0x5e3e03fc6c68d687ULL,0x3e732c3d1ff052c7ULL,0xf2d0efa66ed16e7aULL,0x63d92b26b65bb746ULL,0xffcd82badd44867cULL,0xa71b4a9ef8c081b8ULL,0x6c1676a7736c8785ULL,0xbe2c06169d8932d0ULL, + 0x7bfa1a4b9eca1c9fULL,0x960bc1dc8ce5e535ULL,0xe267d9f317eec30aULL,0x06fb89ef6c257d38ULL,0x2328999ad364a26dULL,0x69b794cb26eaab58ULL,0xad28ab1fb85ba596ULL,0x05dcbff356d0aa94ULL, + 0x53376d282bcffbc4ULL,0x708817a706eadb7aULL,0x6ff50e05cd35ae69ULL,0x63b5fb7574bc7fdeULL,0x71c9e953e7fe08c4ULL,0xb4d8bfd4f583ca18ULL,0xde8d788245e81c5cULL,0xa5f5e93ce0474138ULL, + 0x426c160f293c9f31ULL,0x8eb56333e864d7a7ULL,0xbe1164023ebbba30ULL,0x64c2bae32fd5a302ULL,0x800601e1265aff7bULL,0x52d8a88066fd4b14ULL,0x5aba20e746075a9cULL,0xdaa32bf87e1234c6ULL, + 0x80f9bdef694db7e0ULL,0xedca8787b9fcddc6ULL,0x51981c3403b8dce1ULL,0x4274dcf170e10ba1ULL,0xf72743b86def6d1aULL,0xd25b1670ebdb1866ULL,0xc4491e8c050c6f58ULL,0x2be2b2ab87fbd7f5ULL, + 0x3e0e5c9dd111f8ecULL,0xbcc33f8db7c4e760ULL,0x702f9a91bd392a51ULL,0x7da4a795c132e92dULL,0x1a0b0ae30bb1151bULL,0x54febac802e32251ULL,0xea3a5082694e9e78ULL,0xe58ffec1e4fe40b8ULL, + 0xfbb8349d29c4120bULL,0x9f94391fc0d0d915ULL,0xc4074fa75410ba51ULL,0xa66adbf6150a5911ULL,0xc164543c34bfca38ULL,0xe0f27560b9e1ccfcULL,0x99da0f53e820219cULL,0xe8234498c6b4997aULL, + 0x7b23c513516e19e4ULL,0x56e2e847c5c4d593ULL,0x9f727d735ce71ef6ULL,0x5b6304a6f79a44c5ULL,0x6638a7363ab7e433ULL,0x1adea470fe742f83ULL,0xe054b8545b7fc19fULL,0xf935381aba1d0698ULL, + 0xb5504f9d918e4936ULL,0x65035ef6b2513982ULL,0x0553a0c26f4d9cb9ULL,0x6cb10d56bea85509ULL,0x48d957b7a242da11ULL,0x16a4d3dd672b7268ULL,0x3d7e637c8502a96bULL,0x27c7032b730d463bULL, + 0x55366b7d5846426fULL,0xe7d09e89247d441dULL,0x510b404d736fbf48ULL,0x7fa003d0e784bd7dULL,0x25f7614f17fd9596ULL,0x49e0e0a135cb98dbULL,0x2c65957b2e83a76aULL,0x5d40da8dcddbe0f8ULL, + 0x37f68bb4a595939dULL,0x0355647928740217ULL,0x8e740e7c84ad7612ULL,0xd89bc8439044695fULL,0xf7f3da5d85a9184dULL,0x562563bb9fc0b074ULL,0x06d2e6aaf88a888eULL,0x612d8643161fbe7cULL, + 0x9fb3bba354530bb2ULL,0xbde3ef77cb0869eaULL,0x89bc90460b431163ULL,0x4d03d7d2e4819a35ULL,0x33ae4f9e43b6a782ULL,0x216db3079c88a686ULL,0x91dd88e000ffedd9ULL,0xb280da9f12bd4840ULL, + 0x458f86913e538cd7ULL,0xa7001f6c8e08ad53ULL,0x52b8c6e6bf5d15ffULL,0x548234a4011215ddULL,0xff5a9d2d3d5b4045ULL,0xb0ffeeb64a904190ULL,0x55a3aca448607f8bULL,0x8cbd665c30a0672aULL, + 0xa37f3573f37f5937ULL,0xeb0f6c7dd1e4fca5ULL,0x2965a554ac8ab0fcULL,0x17fbf56c274676acULL,0x2e2f6bd9acf7d720ULL,0x41fc8f8810224766ULL,0x517a14b385d53befULL,0xdae327a57d76a7d1ULL, + 0x515d5c891f5f82dcULL,0x9a7f67d76361079eULL,0xa8da81e311a35330ULL,0xe44990c44b18be1bULL,0xc7d5ed95af103e59ULL,0xece8aba78dac9261ULL,0xbe82b0999394b8d3ULL,0x6830f09a16adfe83ULL, + 0x43c41ac194d7d9b1ULL,0x5bafdd82c82e7f17ULL,0xdf0614c15fda0fcaULL,0x74b043a7a8ae37adULL,0x3ba6afa19e71734cULL,0x15d5437e9c450f2eULL,0x4a5883fe67e242b1ULL,0x5143bdc22c1953c2ULL, + 0xa2a9ce7c6b53f5f9ULL,0x642465951b176d99ULL,0xb1298d36b95c081bULL,0x53505bb81d9a9ee6ULL,0x3f6f9e61f2ba70b0ULL,0xd07e16c98afad453ULL,0x9f1694bbe7eb4a6aULL,0xdfebced93cb0bc8eULL, + 0xc676d7f2b1f3390bULL,0x9f7a1b8ca5b61272ULL,0x4ebebfc9c2e127a9ULL,0x4602500c5dd997bfULL,0x7f09771c4711230fULL,0x058eb37c020f09c1ULL,0xab693d4bfee5e38bULL,0x9289eb1f4653cbc0ULL, + 0xa44d2b391770f5a7ULL,0xe4d4d7910e44eb82ULL,0x42e69d1e3f69712aULL,0xbf11c4d6ac6a820eULL,0xb5e7f3e542c4224cULL,0xd6b4e81c449d941cULL,0x5d72bd165450e878ULL,0x6a61e28aee25ac54ULL, + 0x54da9dc7ab952578ULL,0xb5423df226e84d0bULL,0xa8b64eeb9b872042ULL,0xac2057825990f6dfULL,0x4ff696eb21f4c77aULL,0x1a79c3e4aab273afULL,0x29bc922e9436b3f1ULL,0xff807ef8d6d9a27aULL, + 0xc7f3a8f833f6746cULL,0x21e46f65fea990caULL,0x915fd5c5caddb0a9ULL,0xbd41f01678614555ULL,0x346f4434426ffb58ULL,0x8055943614dbc204ULL,0xf3dd20fe5a969b7fULL,0x9d59e956e899a39aULL, + 0x3c2f0ba9b733aa5fULL,0xdece47cbf05af235ULL,0xf8e3f715a2ac82a5ULL,0xc97ba6412203f18aULL,0xc3af550409c11060ULL,0x56ea2c0546af512dULL,0xfac28daff3f28146ULL,0x87fab43a959ef494ULL, + 0x09891641d4c5105fULL,0x1ae80f8e6d7fbd65ULL,0x9d67225fbee6bdb0ULL,0x3b433b597fc4d860ULL,0x44e66db693e85638ULL,0xf7b59252e3e9862fULL,0xdb785157665c32ecULL,0x702fefd7ae362f50ULL, + 0xfe756a5c97290293ULL,0xbf04a19cd388acbfULL,0xfbbbb9cf5e916bdaULL,0xf489527391f93becULL,0xdee07ec32a5923d7ULL,0xc7bc949bfde0c370ULL,0xbd5121750419d8fcULL,0x54f5d4763fdcc93fULL, + 0x3754475d0fefb0c3ULL,0xd48fb56b46d7c35dULL,0xa070b633363798a4ULL,0xae89f3d28fdb98e6ULL,0x970b89c86363d14cULL,0x8981752167abd27dULL,0x9bf7d47444d5a021ULL,0xb3083bafcac72aeeULL, + 0x0acda2ffcc5e62e9ULL,0x3b8b7d755edb02a4ULL,0xa700741c66120c76ULL,0xf77e847f7d974064ULL,0x0d310678a5e3d464ULL,0xde68b1f346bf35a3ULL,0xcae83028d32f9043ULL,0x724e4717517cb0cbULL, + 0x389741debe949a44ULL,0x638e9388546a4fa5ULL,0x3fe6419ca0047bdcULL,0x7047f648aaea57caULL,0x54e48a9041fbab17ULL,0xda8e0b28576bdba2ULL,0xe807eebcc72afddcULL,0x07d3336df42577bfULL, + 0xfae8563b4f3011dbULL,0xda33776536610c03ULL,0xad4f6f2a6381af0bULL,0xc277984cd95378ecULL,0x5ab0a10c5751d2b2ULL,0x70a18bb97a4bf3f4ULL,0x38d07ad4eae3caf3ULL,0xe8ef552fec430361ULL, + 0x62a8c244bfe20925ULL,0x91c19ac38fdce867ULL,0x5a96a5d5dd387063ULL,0x61d587d421d324f6ULL,0xe87673a2a37173eaULL,0x2384800853778b65ULL,0x10f8441e05bab43eULL,0xfa11fe124621efbeULL, + 0xedd0389a8391d54bULL,0xdac74c08c8afe546ULL,0x8525a4ad5be60bbdULL,0x2419ac9690ad7b87ULL,0x078a3a0277eee51bULL,0xe86ecf367239768dULL,0xcb0a259d8fd48035ULL,0x94db43cb9d29ca5bULL, + 0x047b772e81685d7bULL,0x23f27d81bf34a976ULL,0xc27608e2915f48efULL,0x3b0b43faa521d5c3ULL,0x7613fb2663ca7284ULL,0x7f5729b41d4db837ULL,0x87b14898583b526bULL,0x00b732a6bbadd3d1ULL, + 0x4fffa25b90f8550aULL,0x254db3d31c8dae9dULL,0x58cef963c1fbb232ULL,0xad1cb481a4bfdf0fULL,0x84d26ea1958773c2ULL,0x58622664010114f1ULL,0xeadb3a87f051e67eULL,0xf3185722b69e45c6ULL, + 0x8e02f4262048e396ULL,0x436b50b6383d9de4ULL,0xf78d3481471e85adULL,0x8b01ea6ad005c8d6ULL,0xd3c7afee97015c07ULL,0x46cdf1a94e3ba2aeULL,0x7a42e50183d3a1d2ULL,0xd54b5268b541dff4ULL, + 0xe120c4f948a48e22ULL,0xf24977eed0335a96ULL,0x9af3442336151c7bULL,0xe2815a7fd36648d8ULL,0xb4d2deba66e0a6b2ULL,0x783a84cf817515e8ULL,0x78424c0bff3ff24aULL,0x12dd1bb638e1c5d9ULL, + 0x3f24cf304e23e9bcULL,0x4387f816126e3624ULL,0x26a46a033b0b6d61ULL,0xaf1bc8458b2d777cULL,0x25c401ba527de79cULL,0x0e1346d44261bbb6ULL,0x4b96c44b287b4bc7ULL,0x658493c75254562fULL, + 0xfcb0e9d8fdd41d98ULL,0x8be980c1bcc7c7fbULL,0xa72e86506f8fa4d9ULL,0x356b14ad748cb88dULL,0xfd9fccefea6178f2ULL,0xc84a620d78bb0e35ULL,0xbc75367c62f391a9ULL,0x6a83a5c623100c05ULL, + 0x23f949feb8a24a20ULL,0x17ebfed1f52ca53fULL,0x9b691bbebcfb4853ULL,0x5617ff6b6278a05dULL,0x241b34c5e3c99ebdULL,0xfc64242e1784156aULL,0x4206482f695d67dfULL,0xb967ce0eee27c011ULL, + 0xb4480f0441c23fa3ULL,0xb4712eb0c1989a2eULL,0x3ccbba0f93a29ca7ULL,0x6e205c14d619428cULL,0x90db7957b3641686ULL,0x0432691d45ac8b4eULL,0x07a759acf64e0350ULL,0x0514d89c9c972517ULL, + 0xe3b22c6bc4fe3c39ULL,0xba4a81536c7bebdfULL,0xf23ab6b725693459ULL,0x53bc377014922b11ULL,0x4645c8ab5afc60dbULL,0xaa02235520b9f2a3ULL,0x52a2954cce0fc507ULL,0x8c2731bb7ce1c2e7ULL, + 0x5066efb6d9790ed6ULL,0xa77a0cbca6aa793bULL,0x1a915f3c223e042eULL,0x1c5def0469c5874bULL,0x0e83007873b6c1daULL,0x55cf85d2fcd8557aULL,0x0f7c7c760460f3b1ULL,0x87052acb46e58063ULL, + 0x6a7091c2e48fb889ULL,0x26882c137b8a9d06ULL,0xa24986631b82a0e2ULL,0x844ed7363518152dULL,0x282f476fd86e27c7ULL,0xa04edaca04afefdcULL,0x8b256ebc6119e34dULL,0x56a413e90787d78bULL, + 0x16eab6a20d645fd6ULL,0x632cbd8df61d3148ULL,0xcc1bf7cf62079ae9ULL,0x257ee5c7f33eccbbULL,0xbf6b34a81680ac73ULL,0xaa084e8872c77aa0ULL,0x7b5a864e05a0a1d1ULL,0x0641f6db359a1b16ULL, + 0xf01d095dc8385050ULL,0x0d54a5d5df4b441cULL,0x2a37ccb40927706aULL,0xdf008f5445d7eb7eULL,0x74eb34f35bf716c7ULL,0x57a65b58641bd6caULL,0xef345e4835e6fa02ULL,0x191f913b88342a09ULL, + 0x1554d46da670ff1dULL,0x24833d88cb97a1ccULL,0x8fa6ab3cded97493ULL,0x215e037189926498ULL,0x549bd592e56d74ffULL,0x58a8caf543b5e1ecULL,0x3c6087a323e93cb9ULL,0x8b0549875648b83cULL, + 0x82ee061d5a74be50ULL,0xe41781c4dea16ff5ULL,0xe0b0c81e99bfc8a2ULL,0x624f4d690b547e2dULL,0x3a83545dbdcc9ae4ULL,0x2573dbb6409b1e8eULL,0x482960c4a6c93539ULL,0xf01059ad5ae18798ULL, + 0xc431a238013ff83bULL,0x7c0018b2fad69d08ULL,0x99aeb52a4c9589eaULL,0x121f41ab9b1cf19fULL,0x0cfbbcbaef0f5958ULL,0x8deb3aeb7be8fbdcULL,0x12b954081f15aa31ULL,0x5acc09b34c0c06fdULL, + 0x775cbfa86d518ffbULL,0xdecee1f6930f124bULL,0x9a402804f5e81d0fULL,0x0e8225c52a0eeb2fULL,0x884a5d39fee9e867ULL,0x9540428ffb505454ULL,0xb2bf2e20107a70d1ULL,0xd9917c3ba010b2aaULL, + 0xa98f42fa3d843d53ULL,0x33777cc613ef927aULL,0xc440cdbecb84ca74ULL,0x8c22f9631dc7c5ddULL,0x4bc82b70c8d94708ULL,0x7e0b43fcc814364fULL,0x286d4e2486f59b7eULL,0x1abc895e4d6bf4c4ULL, + 0x38151e274d559d96ULL,0x4f18c0d3b8db6c01ULL,0x49a3aa836f9921afULL,0xdbeab27b8c046029ULL,0x242b9eaa7040bf3bULL,0x39c479e51614b091ULL,0x338ede2b0e4baf5dULL,0x5bb192b7f0a53945ULL, + 0x896d572337e440d7ULL,0x685c5fd9ade23f68ULL,0xb5b1a26dc2c64918ULL,0xb9390e30dad6580cULL,0x87911c4e7dee5b9bULL,0xb90c5053deb04f6eULL,0x37b942a18f065aa6ULL,0x34acdf2a1ca0928dULL, + 0x733b64d39de40ca3ULL,0x1d4b6d6fd2f3857eULL,0xbe2be8e9b2ed92f7ULL,0x64ca7047b77da248ULL,0xc65dae9b8da99315ULL,0x9c1451750fc698a4ULL,0x8a296b94ff958c27ULL,0x38684e0843950097ULL, + 0x7872e34b3390ff23ULL,0x968ce4abde7d18efULL,0x9b4a745e627fe7b1ULL,0x9607b0a0caff3e2aULL,0x1b05818eeb40e3a5ULL,0x6ac62204c0fa8d7aULL,0xb5b9058571ed4809ULL,0xb2432ef0f7cb65f2ULL, + 0x715c9f973112795fULL,0xe8244437984e6ee1ULL,0x55cb4858ecb66bcdULL,0x7c136735abaffbeeULL,0x546615955dbec38eULL,0x51c0782c388ad153ULL,0x9ba4c53ac6e0952fULL,0x27e6782a1b21dfa8ULL, + 0xfeb09740e2c2bf15ULL,0x627a2205a9e99704ULL,0xec8d73d0c2fbc565ULL,0x223eed8fc20c8de8ULL,0x1ee32583a8363b49ULL,0x1a0b6cb9c9c2b0a6ULL,0x49f7c3d290dbc85cULL,0xa8dfbb971ef4c1acULL, + 0xc16c236e846e364fULL,0x7f33527cdea50ca0ULL,0xc48107750926b86dULL,0x6c2a36090598e70cULL,0xa6755e52f024e924ULL,0xe0fa07a49db4afcaULL,0x15c3ce7d66831790ULL,0x5b4ef350a6cbb0d6ULL, + 0x42806b2da6dc1d29ULL,0xd3030009f871e144ULL,0xa1feb333aaf49276ULL,0xb5583b9ec70bc04bULL,0x1db0be7895695f20ULL,0xfc84181189d012b5ULL,0x6409f27205f61643ULL,0x40d34174d5883128ULL, + 0x05214c050f15dde9ULL,0xa47a76a80d5f2b82ULL,0xbb254d3062e82b62ULL,0x11a05fe03ec955eeULL,0x7eaff46e9d529b36ULL,0x55ab13018f9e3df6ULL,0xc463e37199317698ULL,0xfd251438ccda47adULL, + 0x8c3c669c72ba075bULL,0x89f78b55ba469015ULL,0x5706aade3e9f8ba8ULL,0x6d8bd565b32d7ed7ULL,0x25f4e63b805f08d6ULL,0x7f48200dc3bcc1b5ULL,0x4e801968b025d847ULL,0x74afac0487cbe0a8ULL, + 0xe2a37598a9d82abfULL,0x5f188ccbe6c170f5ULL,0x816822005066b087ULL,0xda22c212c7155adaULL,0x151e5d3afbddb479ULL,0x4b606b846d715b99ULL,0x4a73b54bf997cb2eULL,0x9a1bfe433ecd8b66ULL, + 0x79732522cccc18adULL,0xaadf3f8df1a6e027ULL,0xf7382c9317c2354dULL,0x5ce1680cd818b689ULL,0x359ebbfcd9ecbee9ULL,0x4330689c1cae62acULL,0xb55ce5b4c51ac38aULL,0x7921dfeafe238ee8ULL, + 0xe13122f3dbfb894eULL,0xbe9b79f6ce274b18ULL,0x85a49de5ca58aadfULL,0x2495775811487351ULL,0x111def61bb939099ULL,0x1d6a974a26d13694ULL,0x4474b4ced3fc253bULL,0x3a1485e64c5db15eULL, + 0x65994ddb0f5f27caULL,0xe85461fba80d59ffULL,0xff05481a66601023ULL,0xc665427afc9ebbfbULL,0xb0571a697587fd52ULL,0x935289f88d49efceULL,0x61becc60ea420688ULL,0xb22639d913a786afULL, + 0x5afddab61430c9abULL,0x0bdd41d32238e997ULL,0xf0947430418042aeULL,0x71f9addacdddc4cbULL,0x7090c016c52dd907ULL,0xd9bdf44d29e2047fULL,0xe6f1fe801b1011a6ULL,0xb63accbcd9acdc78ULL, + 0x264c76680448087cULL,0xac30903f71432daeULL,0x3851b26600f9bf47ULL,0x400ed3116cdd6d03ULL,0x045e79fef8fd2424ULL,0xfdfd974afa6da98bULL,0x45c9f6410c1e673aULL,0x76f2e7335b2c5168ULL, + 0x7817acab4baef62eULL,0x9f5a2202a85b91e8ULL,0x9666ebe66ce57610ULL,0x32ad31f3f73bfe03ULL,0x628330a425bcf4d6ULL,0xea950593515056e6ULL,0x59811c89e1332156ULL,0xc89cf1fe8c11b2d7ULL, + 0x889e5acbc46d7ce1ULL,0x9a515bb78b085877ULL,0xfac1a03d0b7a5050ULL,0x7d3e738af2926035ULL,0x861cc2ce2a6cb0ebULL,0x6f2e29558f7adc79ULL,0x61c4d45133016376ULL,0xd9fd2c805ad59090ULL, + 0x0ad7337ac0b7eff3ULL,0x8552225ec5e48b3cULL,0xe6f78b0c73f13a5fULL,0x5e70062e82349cbeULL,0x6b8d5048e7073969ULL,0x392d2a29c33cb3d2ULL,0xee4f727c4ecaa20fULL,0xa068c99e2ccde707ULL, + 0x1888d65861a023efULL,0x1d72aab4b9e5246eULL,0xa9a26348e5563ec0ULL,0xa0971963c3439a43ULL,0x567dd54badb9b5b7ULL,0x73fac1a1c45a524bULL,0x8fe97ef7fe38e608ULL,0x608748d23f384f48ULL, + 0xebde86ec1ed66f18ULL,0x225d906bd61fce43ULL,0x5cab07d6e8bed74dULL,0x16e4617f27855ab7ULL,0x6568aaddb2fbc3ddULL,0xedb5484f8aeddf5bULL,0x878f20e86dcf2fadULL,0x3516497c615f5699ULL, + 0xef0a3fecfa181e69ULL,0x9ea02f8130d69a98ULL,0xb2e9cf8e66eab95dULL,0x520f2beb24720021ULL,0x621c540a1df84361ULL,0x1203772171fa6d5dULL,0x6e3c7b510ff5f6ffULL,0x817a069babb2bef3ULL, + 0x83572fb6b294cda6ULL,0x6ce9bf75b9039f34ULL,0x20e012f0095cbb21ULL,0xa0aecc1bd063f0daULL,0x57c21c3af02909e5ULL,0xc7d59ecf48ce9cdcULL,0x2732b8448ae336f8ULL,0x056e37233f4f85f4ULL, + 0x8a10b53189e800caULL,0x50fe0c17145208fdULL,0x9e43c0d3b714ba37ULL,0x427d200e34189accULL,0x05dee24fe616e2c0ULL,0x9c25f4c8ee1854c1ULL,0x4d3222a58f342a73ULL,0x0807804fa027c952ULL, + 0xc222653a4f0d56f3ULL,0x961e4047ca28b805ULL,0x2c03f8b04a73434bULL,0x4c966787ab712a19ULL,0xcc196c42864fee42ULL,0xc1be93da5b0ece5cULL,0xa87d9f22c131c159ULL,0x2bb6d593dce45655ULL, + 0x22c49ec9b809b7ceULL,0x8a41486be2c72c2cULL,0x813b9420fea0bf36ULL,0xb3d36ee9a66dac69ULL,0x6fddc08a328cc987ULL,0x0a3bcd2c3a326461ULL,0x7103c49dd810dbbaULL,0xf9d81a284b78a4c4ULL, + 0x3de865ade4d55941ULL,0xdedafa5e30384087ULL,0x6f414abb4ef18b9bULL,0x9ee9ea42faee5268ULL,0x260faa1637a55a4aULL,0xeb19a514015f93b9ULL,0x51d7ebd29e9c3598ULL,0x523fc56d1932178eULL, + 0x501d070cb98fe684ULL,0xd60fbe9a124a1458ULL,0xa45761c892bc6b3fULL,0xf5384858fe6f27cbULL,0x4b0271f7b59e763bULL,0x3d4606a95b5a8e5eULL,0x1eda5d9b05a48292ULL,0xda7731d0e6fec446ULL, + 0xa3e3369390d45871ULL,0xe976404006166d8dULL,0xb5c3368289a90403ULL,0x4bd1798372f1d637ULL,0xa616679ed5d2c53aULL,0x5ec4bcd8fdcf3b87ULL,0xae6d7613b66a694eULL,0x7460fc76e3fc27e5ULL, + 0x70469b8295caabeeULL,0xde024ca5889501e3ULL,0x6bdadc06076ed265ULL,0x0cb1236b5a0ef8b2ULL,0x4065ddbf0972ebf9ULL,0xf1dd387522aca432ULL,0xa88b97cf744aff76ULL,0xd1359afdfe8e3d24ULL, + 0x52a3ba2b91502cf3ULL,0x2c3832a8084db75dULL,0x04a12dddde30b1c9ULL,0x7802eabce31fd60cULL,0x33707327a37fddabULL,0x65d6f2abfaafa973ULL,0x3525c5b811e6f91aULL,0x76aeb0c95f46530bULL, + 0xe8815ff62f93a675ULL,0xa6ec968405f48679ULL,0x6dcbb556358ae884ULL,0x0af61472e19e3873ULL,0x72334372a5f696beULL,0xc65e57ea6f22fb70ULL,0x268da30c946cea90ULL,0x136a8a8765681b2aULL, + 0xad5e81dc0f9f44d4ULL,0xf09a69602c46585aULL,0xd1649164c447d1b1ULL,0x3b4b36c8879dc8b1ULL,0x20d4177b3b6b234cULL,0x096a25051730d9d0ULL,0x0611b9b8ef80531dULL,0xba904b3b64bb495dULL, + 0x1192d9d493a3147aULL,0x9f30a5dc9a565545ULL,0x90b1f9cb6ef07212ULL,0x299585460d87fc13ULL,0xd3323effc17db9baULL,0xcb18548ccb1644a8ULL,0x18a306d44f49ffbcULL,0x28d658f14c2e8684ULL, + 0x44ba60cda99f8c71ULL,0x67b7abdb4bf742ffULL,0x66310f9c914b3f99ULL,0xae430a32f412c161ULL,0x1e6776d388ace52fULL,0x4bc0fa2452d7067dULL,0x03c286aa8f07cd1bULL,0x4cb8f38ca985b2c1ULL, + 0x83ccbe808c3bff36ULL,0x005a0bd25263e575ULL,0x460d7dda259bdcd1ULL,0x4a1c5642fa5cab6bULL,0x2b7bdbb99fe4fc88ULL,0x09418e28cc97bbb5ULL,0xd8274fb4a12321aeULL,0xb137007d5c87b64eULL, + 0x80531fe1c63c4962ULL,0x50541e89981fdb25ULL,0xdc1291a1fd4c2b6bULL,0xc0693a17a6df4fcaULL,0xb2c4604e0117f203ULL,0x245f19630a99b8d0ULL,0xaedc20aac6212c44ULL,0xb1ed4e56520f52a8ULL, + 0xb5560fb6700a1acdULL,0xe823fd73fd999681ULL,0xda915d1f6cb4e1baULL,0x0d0301186ebe00a3ULL,0x744fb0c989fca8cdULL,0x970d01dbf9da0e0bULL,0x0ad8c5647931d76fULL,0xb15737bff659b96aULL, + 0xa12b384ece53c2d0ULL,0x779d897d5e4606daULL,0xa53e47b073ec12b0ULL,0x462dbbba5756f1adULL,0x69fe09f2cafe37b6ULL,0x273d1ebfecce2e17ULL,0x8ac1d5383cf607fdULL,0x8035f7ff12e10c25ULL, + 0xca442d5a2093c22aULL,0xebd0bd31d5703aedULL,0x308f2afd653287b6ULL,0x9bb88bac0d1bc8baULL,0xfbaf853875c1e3b2ULL,0xbd2ac950ca11447cULL,0x286d816cea5c4c8dULL,0xdc3aa80028dc3208ULL, + 0x854d34c77e6c5520ULL,0xc27df9efdcb9ea58ULL,0x405f2369d686666dULL,0x29d1febf0417aa85ULL,0x9846819e93470afeULL,0x3e6a9669e2a27f9eULL,0x24d008a2e31e6504ULL,0xdba7cecf9cb7680aULL, + 0x26a43e41d07fa53dULL,0x3154a78a74e35bc5ULL,0x7b768924e0da2f8cULL,0xba964a2b23613f9aULL,0x5a548d35ba1d16c4ULL,0x2e1bfed1fb54d057ULL,0xff992136bc640205ULL,0xf39cb9148156df29ULL, + 0xc913e64699b444adULL,0xddfce99dc40504c5ULL,0x58482a99d42e53dbULL,0x9aaf2c25d1aff537ULL,0xee90f7962664cf67ULL,0x74ab5c99f1393e2bULL,0xfad0faeae6225bb0ULL,0xc355648c2d63dd6cULL, + 0xe4e31d271d91cf9dULL,0xcb35d4fdb377b20aULL,0x74de1e45055e1327ULL,0x3298e31b28703e75ULL,0x55087237de013339ULL,0x32cbf30123d101c6ULL,0xc70dba22e8aab0dcULL,0x4a52623d3d155bb9ULL, + 0xecaff541338d6e43ULL,0x56f7dd734541d5ccULL,0xb5d426de96bc88caULL,0x48d94f6b9ed3a2c3ULL,0x6354a3bb2ef8279cULL,0xd575465b0b1867f2ULL,0xef99b0ff95225151ULL,0xf3e19d88f94500d8ULL, + 0xdbf435acc85dcf57ULL,0x61745658c88f5415ULL,0x26367e9a17c55807ULL,0x22d077a5ca90c56fULL,0xfbf72258a2e04e76ULL,0xba965d3e6e06e405ULL,0x5724d06fe3e6f954ULL,0x3e47d47581251a74ULL, + 0xb8ba0151e0fb82f7ULL,0x0d160726d5668ac2ULL,0x622ba25814d711b0ULL,0x6addf5577f3fe2f2ULL,0x2b831e1c6b9c9435ULL,0xce3a060ab73826bdULL,0x93fa11c11c240f89ULL,0x4f9cc8d8956e303aULL, + 0x4b9331f6641f82c9ULL,0xd97c7c54dffec756ULL,0xf5ee6d1f1a9158abULL,0x054493a385c3da7fULL,0xa57a05f5eb7d96dfULL,0xa3afd447e4473a39ULL,0x42a4d9c488e16d55ULL,0x83e144f5f5f876aeULL, + 0x92a83268e32dd620ULL,0x913ec99f627849a2ULL,0xedd8fdfa2c378882ULL,0xaf96f33eee6f8cfeULL,0xc06737e5dc3fa8a5ULL,0x236bb531b0b03a1dULL,0x33e59f2989f037b0ULL,0x13f9b5a7d9a12a53ULL, + 0x50d8ae9559029aa6ULL,0xd74e292c5a4db2edULL,0x0b9c3355848f373dULL,0xec018db6ac45ab38ULL,0x1f44690269cc53a8ULL,0x8c4b628d1a879864ULL,0x1c743d284b13475eULL,0xbf4a933873de19f6ULL, + 0x4a8a4f47f0cefa69ULL,0xdc8e4cbaa4546866ULL,0x359ba69b23f603c1ULL,0xdab4d601187b7ac5ULL,0xa6ca4337c1ebc8d9ULL,0x9fa6585452b4074bULL,0x1a4b4f81902fb733ULL,0xd2bb5d7aa525deaaULL, + 0xeb2e92d5f81f9567ULL,0x54cb95ea4d698470ULL,0x5f2acb28e04c81ebULL,0x1c1ebfc4f8ceec64ULL,0x8f799fac06e07423ULL,0x72225f9937fa0c85ULL,0xb0cd861634f4db44ULL,0x5ec36159752c9091ULL, + 0x0d0df6ce51efb310ULL,0xcb5b2eb4958df5beULL,0xd6459e2936158e59ULL,0x82aae2b91466e336ULL,0xfb658a39411aa636ULL,0x7152ecc5d4c0a933ULL,0xf10c758a49f026b7ULL,0xf4837f97cb09311fULL, + 0x994f523a626332d5ULL,0x7bc388335561bb44ULL,0x005ed4b03d845ea2ULL,0xd39d3ee1c2a1f08aULL,0x6561fdd3e7676b0dULL,0x620e35fffb706017ULL,0x36ce424ff264f9a8ULL,0xc4c3419fda2681f7ULL, + 0x00f831769bb81648ULL,0xd69eb485653120d0ULL,0xd17d75f44ccabc62ULL,0x34a07f82b749fcb1ULL,0x2c3af787bbfb5554ULL,0xb06ed4d062e283f8ULL,0x5722889fa19213a0ULL,0x162b085edcf3c7b4ULL, + 0x36d90ddaeb300f7aULL,0x9dcf7dfcedb5e801ULL,0x645cb26874d5244cULL,0xa127ee79348e3aa2ULL,0x488acc53575f1dbbULL,0x95037e8580e6161eULL,0x57e59283292650d0ULL,0xabe67d9914938216ULL, + 0x32670d2f7189e71fULL,0xc64387485ecf91e7ULL,0x15758e57db757a21ULL,0x427d09f8290a9ce5ULL,0x846a308f38384a7aULL,0xaac3acb4b0732b99ULL,0x9e94100917845819ULL,0x95cba111a7ce5e03ULL, + 0xeb81aa377378058eULL,0x41c746a104411154ULL,0xa10c73bcfb828ac7ULL,0x6439be919d972b29ULL,0x4bf3b4b043a2fbadULL,0x39e6dadf82b5e840ULL,0x4f7164086397bd4cULL,0x0f7de5687f1eeccbULL, + 0xdb332a73f37ec3c3ULL,0xc65259bddd59eba0ULL,0x2291709cdb4d3257ULL,0x9a793b25bd389390ULL,0xf39fe34be43756f0ULL,0x2f76bdce9afb56c9ULL,0x9f37867a61208b27ULL,0xea1d4307089972c3ULL, + 0xd0a744878a429f4fULL,0x0649712bdb516609ULL,0xb826ba57e769b5dfULL,0x82335df21fc7aaf2ULL,0x2389f0675c93d995ULL,0x59ac367a68677be6ULL,0xa77985ff21d9951bULL,0x038956fb85011cceULL, + 0x97b7851aaaca5e9bULL,0x518aa52156713b97ULL,0x3357e8c7150a61f6ULL,0x7842e7e2ec2c2b69ULL,0x8dffaf656868a548ULL,0xd963bd82e068fc81ULL,0x64da5c8b65917733ULL,0x927090ff7b247328ULL, + 0xd6ffdb942f6d0d97ULL,0x05c3ee41443b9373ULL,0xb2e541ebffd36db6ULL,0xb7415a96ce1dcc3eULL,0xe383682e163aa2f6ULL,0x46febdd42f3af218ULL,0x90a0507ebafdbadbULL,0x4ca8ab4dce52e21aULL, + 0xa3f0832e1b7cfb73ULL,0x7a8afe523ec354c9ULL,0xae91c97e378eadcaULL,0x7449c599ac3b32bbULL,0xa619c3710b1c4655ULL,0x692e4c6af79da87eULL,0xff3f5d86de38d96aULL,0xc5320f421c08c0ecULL, + 0x92a6f2bc8fec5decULL,0xa71383ff84d6786cULL,0x87588c06dbffa084ULL,0x0d85f5ca6857e715ULL,0xe87311b3b6c774d4ULL,0x672357c84c3521a8ULL,0xe5fe74615b29fe0fULL,0x02bc51105b7158cdULL, + 0x37a01e48a105fc8eULL,0x769d754a289ba48cULL,0xc08c6fe1d51c2180ULL,0xb032dd33b7bd1387ULL,0x953826db020b0aa6ULL,0x05137e800664c73cULL,0xc66302c4660cf95dULL,0x99004e11b2cef28aULL, + 0x824f5b284f973536ULL,0xb43e299ed35b04eaULL,0xc72c88f74da03089ULL,0x8269d57a45a2e42cULL,0x7c1e63fc6607b38eULL,0xe89e2aaf29390b0cULL,0xc7c740da1bee2869ULL,0x8556f6fcaf3fb974ULL, + 0x1a0a3995a4b6bed1ULL,0x2dab579597095c54ULL,0x06c6a1ff2aa73ce9ULL,0xadd0a54b4de438a7ULL,0x160b6b1afca906cdULL,0x25fc601629de10ddULL,0x348e9c99d3633da3ULL,0x1fe3f746158a4d5aULL, + 0xc253edc88be85c1dULL,0xdd3d0e483ca09cb6ULL,0xb997f6879ae3055aULL,0x0c929ad007431dbfULL,0xcef1621584d2db42ULL,0xb50df3ef078828cdULL,0x4589da9d6dbd4b66ULL,0xbc4fd2e3d99c2b04ULL, + 0x214bc9a7d298c241ULL,0xe3b697ba56807cfdULL,0xef1c78024564eadbULL,0xdde8cdcfb48149c5ULL,0x946bf0a75a4d2604ULL,0x27154d7f6c1538afULL,0x95cc9230de5b1fccULL,0xd88519e966864f82ULL, + 0xb828dd1a7cb1282cULL,0xa08d7626be46973aULL,0x6baf8d40e708d6b2ULL,0x72571fa14daeb3f3ULL,0x85b1732ff22dfd98ULL,0x87ab01a70087108dULL,0xaaaafea85988207aULL,0xccc832f869f00755ULL, + 0x488f1185ca8d9d1aULL,0xadf2c77dd987ded2ULL,0x5f3039f060c46124ULL,0xe5d70b7571e095f4ULL,0x82d586506260e70fULL,0x39d75ea7f750d105ULL,0x8cf3d0b175bac364ULL,0xf3a7564d21d01329ULL, + 0x94ab4700ec3128c2ULL,0x6c76d8628e383f49ULL,0xdc36b150c03024ebULL,0xfb43947753daac69ULL,0xfc68764a8dc79623ULL,0x5b86995db440fbb2ULL,0xd66879bfccc5ee0dULL,0x0522894295aa8bd3ULL, + 0xb24aa43e3fcd3efcULL,0xdd26c034b8088e9aULL,0xa5ef4dc9bd3d46eaULL,0xa2f99d588a4c6a6fULL,0xddabd3552f1da46cULL,0x72c3f8ce1afacdd1ULL,0xd90c4eee92d40578ULL,0xd28bb41fca623b94ULL, + 0x5e7c3becee8314f3ULL,0x1c068aeddbea298fULL,0x08d381f17c80acecULL,0x03b56be8e330495bULL,0xaeffb8f29222882dULL,0x95ff38f6c4af8bf7ULL,0x50e32d351fc57d8cULL,0x6635be5217b444f0ULL, + 0x242792d2e7417ce1ULL,0xff42bc71970ee7f5ULL,0x1ff4dc6d5c67a41eULL,0x77709b7b20882a58ULL,0x3554731dbe217f2cULL,0x2af2a8cd5bb72177ULL,0x58eee769591dd059ULL,0xbb2930c94bba6477ULL, + 0x174a9126cecdaa7aULL,0xfc8c7e0e0b13247bULL,0x29c110d23484c1c4ULL,0xf8eb8757831dfc3bULL,0x022f0212c0067452ULL,0x3f6f69ee7b9b926cULL,0x09032da0ef42daf4ULL,0x79f00ade83f80de4ULL, + 0x1e6adddaf176f2c0ULL,0x01ca4604e2572658ULL,0x0a404ded85342ffbULL,0x8cf60f96441838d6ULL,0x9bbc691cc9071c4aULL,0xfd58874434442803ULL,0x97101c85809c0d81ULL,0xa7fb754c8c456f7fULL, + 0xf8559ff4c1e99d81ULL,0x08e1a7d6a3c617c0ULL,0xb398fd43248c6ba7ULL,0x6ffedd91d1283794ULL,0x8a6a59d2d629d208ULL,0xa9d141d53490530eULL,0x42f6fc1838505989ULL,0x09bf250d479d94eeULL, + 0x6af7a1d5af71013fULL,0xe68216e50bedc946ULL,0xf4cba30bd27370a0ULL,0x7981afbf870421ccULL,0x02496a679449f0e1ULL,0x86cfc4be0a47edaeULL,0x3073c936b1feca22ULL,0xf569461203f8f8fbULL, + 0xec14f9e12cb7191eULL,0x78ea1bd8e5b08ea6ULL,0x3c65aa9b46332bb9ULL,0x84cc22b3bf80ce25ULL,0x0098e9e9d49d5bf1ULL,0xcd4ec1c619087da4ULL,0x3c9d07c5aef6e357ULL,0x839a02689f8f64b8ULL, + 0xbcadd6715bde48f8ULL,0xc97038732189bc7dULL,0x5d45299ec709ee8aULL,0xd1287ee2845aaff8ULL,0x7d1f8874db1dbf1fULL,0xea46588b990c88d6ULL,0x60ba649a84368313ULL,0xd5fdcbce60d543aeULL, + 0xf795643037577dd8ULL,0x83b82af429c5fe88ULL,0x9c1bea26cdbdc132ULL,0x589fa0869c04339eULL,0x033e9538b13799dfULL,0x85fa8b21d295d034ULL,0xdf17f73fbd9ddccaULL,0xf32bd122ddb66334ULL, + 0xcf3de9959890272dULL,0x75f3432a3e713a10ULL,0x5e13479fe28227b8ULL,0xb8561ea9fefacdc8ULL,0xa6a297a08332aafdULL,0x9b0d8bb573809b62ULL,0xd2fa1cfd0c63036fULL,0x7a16eb55bd64bda8ULL, + 0x4cc34ec13cf48283ULL,0xb09daa259c8a705eULL,0xd1e9d0d05b7d4f84ULL,0x4df6ef64db38929dULL,0xe16b0763aa21ba46ULL,0xc6b1d178a293f8fbULL,0x0ff5b602d520aabfULL,0x94d671bdc339397aULL, + 0xf7e48e8a2ac13e27ULL,0x4494f6df4eb1a9f5ULL,0xedbf84eb981f0a62ULL,0x49badc32536438f0ULL,0x50bea541004f7571ULL,0xbac67d10df1c94eeULL,0x253d73a1b727bc31ULL,0xb3d01cf230686e28ULL, + 0xd433e50f6d3549cfULL,0x6f33696ffacd665eULL,0x695bfdacce11fcb4ULL,0x810ee252af7c9860ULL,0x65450fe17159bb2cULL,0xf7dfbebe758b357bULL,0x2b057e74d69fea72ULL,0xd485717a92731745ULL, + 0x896c42e8ee36860cULL,0xdaf04dfd4113c22dULL,0x1adbb7b744104213ULL,0xe5fd5fa11fd394eaULL,0x68235d941a4e0551ULL,0x6772cfbe18d10151ULL,0x276071e309984523ULL,0xe4e879de5a56ba98ULL, + 0x6c8d0aa9b898fd52ULL,0x2fb38a57be9af1a7ULL,0xe1f2b9a93b4f03f8ULL,0x2b1aad44c3f0cc6fULL,0x58b5332e7cf2c084ULL,0x1c57d96f0367d26dULL,0x2297eabdfa6e4a8dULL,0x65a947ee4a0e2b6aULL, + 0xaaafafb0285b9491ULL,0x01a0be881e4c705eULL,0xff1d4f5d2ad9caabULL,0x6e349a4ac37a233fULL,0xcf1c12464a1c6a16ULL,0xd99e6b6629383260ULL,0xea3d43665f6d5471ULL,0x36974d04ff8cc89bULL, + 0xf535b616fdd5b854ULL,0x592549c85728719fULL,0xe231468606921cadULL,0x98c8ce34311b1ef8ULL,0x28b937e7e9090b36ULL,0x67fc3ab90bf7bbb7ULL,0x12337097a9d87974ULL,0x3e5adca1f970e3feULL, + 0xc26c49a1cfe89d80ULL,0xb42c026dda9c8371ULL,0xca6c013adad066d2ULL,0xfb8f722856a4f3eeULL,0x08b579ecd850935bULL,0x34c1a74cd631e1b3ULL,0xcb5fe596ac198534ULL,0x39ff21f6e1f24f25ULL, + 0xcdcc68a7b3f85ff0ULL,0xacd21cdd1a888044ULL,0xb6719b2e05dbe894ULL,0xfae1d3d88b8260d4ULL,0xedfedece8a1c5d92ULL,0xbca01a94dc52077eULL,0xc085549c16dd13edULL,0xdc5c3bae495ebaadULL, + 0x27f29e148f929057ULL,0x7a64ae06c0c853dfULL,0x256cd18358e9c5ceULL,0x9d9cce82ded092a5ULL,0xcc6e59796e93b7c7ULL,0xe1e4709231bb9e27ULL,0xb70b3083aa9e29a0ULL,0xbf181a753785e644ULL, + 0xcc17063fbe7b643aULL,0x7872e1c846085760ULL,0x86b0fffbb4214c9eULL,0xb18bbc0e72bf3638ULL,0x8b17de0c722591c9ULL,0x1edeab1948c29e0cULL,0x9fbfd98ef4304f20ULL,0x2d1dbb6b9c77ffb6ULL, + 0xf53f2c658ead09f7ULL,0x1335e1d59780d14dULL,0x69cc20e0cd1b66bcULL,0x9b670a37bbe0bfc8ULL,0xce53dc8128efbeedULL,0x0c74e77c8326a6e5ULL,0x3604e0d2b88e9a63ULL,0xbab38fca13dc2248ULL, + 0x255616d3c7141771ULL,0xa86691ab2f226b66ULL,0xda19fea4b3ca63a9ULL,0xfc05dc42ae672f2bULL,0xa9c6e786718ba28fULL,0x07b7995b9c66b984ULL,0x0f434f551b3702f2ULL,0xd6f6212fda84eeffULL, + 0x8ed6e8c85c0a3f1eULL,0xbcad24927c87c37fULL,0xfdfb62bb9ee3b78dULL,0xeba8e477cbceba46ULL,0x37d38cb0eeaede4bULL,0x0bc498e87976deb6ULL,0xb2944c046b6147fbULL,0x8b123f35f71f9609ULL, + 0x4b0e7987b5b41d78ULL,0xea7df9074bf0c4f8ULL,0xb4d03560fab80ecdULL,0x6cf306f6fb1db7e5ULL,0x0d59fb5689fd4773ULL,0xab254f4000f9be33ULL,0x18a09a9277352da4ULL,0xf81862f5641ea3efULL, + 0xa155dcc7de79dc24ULL,0xf1168a32558f69cdULL,0xbac215950d1850dfULL,0x15c8295bb204c848ULL,0xf661aa367d8184ffULL,0xc396228e30447bdbULL,0x11cd5143bde4a59eULL,0xe3a26e3b6beab5e6ULL, + 0xb59b01579f759d01ULL,0xa2923d2f7eae4fdeULL,0x18327757690ba8c0ULL,0x4bf7e38b44f51443ULL,0xb6812563b413fc26ULL,0xedb7d36379e53b36ULL,0x4fa585c4c389f66dULL,0x8e1adc3154bd3416ULL, + 0xd3b3a13f1402b9d0ULL,0x573441c32c7bc863ULL,0x4b301ec4578c3e6eULL,0xc26fc9c40adaf57eULL,0x96e71bfd7493cea3ULL,0xd05d4b3f1af81456ULL,0xdaca2a8a6a8c608fULL,0x53ef07f60725b276ULL, + 0xa6b5c9d646ac49d2ULL,0x42c77c0b83137aa9ULL,0x24d000fc68225a38ULL,0x0f63cfc82fe1e907ULL,0x22d1b01bc6441f95ULL,0x7d38f719ec8e448fULL,0x9b33fa5f787fb1baULL,0x94dcfda1190158dfULL, + 0x211cde10296c36efULL,0x7ee8967282c4da77ULL,0xb617d270a57836daULL,0xf0cd9c319cb7560bULL,0x01fdcbf7e455fe90ULL,0x3fb53cbb7e7334f3ULL,0x781e2ea44e7de4ecULL,0x8adab3ad0b384fd0ULL, + 0x01778a2b599ff0f9ULL,0x68a923d78104fc6bULL,0x5bfa44dfda694ff3ULL,0x4f7199dbf7667f12ULL,0xc06d8ff6e46f2a79ULL,0x08b5deade9f8131dULL,0x02519a59abb4ce7cULL,0xc4f710bcb42aec3eULL, + 0x3014368b4ed80940ULL,0x67e6d0567a6fceddULL,0x7c208c49ca97579fULL,0xfe3d7a81a23597f6ULL,0x5e2032027e096ae2ULL,0xb1f3e1e724b39366ULL,0x26da26f32fdcdffcULL,0x79422f1d6097be83ULL, + 0x50549c748c878145ULL,0x67f14edf39c63565ULL,0x22ddf78c9bcf2d5eULL,0xffaa842f68201d10ULL,0x47d94a9dd1b2de28ULL,0xc09c4be8054be414ULL,0xac80e178ca82755bULL,0xe3251d105697c3bdULL, + 0x2bbe09d35001417bULL,0x795e84ee5962ed5eULL,0x5b79d1ca279f46c3ULL,0x1f7f8a3b83836a2eULL,0x692200b14a64dc32ULL,0xc84243350f84f739ULL,0xf110da07cc9155c0ULL,0xee8fbe61594b0507ULL, + 0x2f6a5391035703ccULL,0x9899bf6a40c7e24dULL,0xc3f7f248bbfcbb9aULL,0xf65027ded555875bULL,0xa7a16b69ffff3b37ULL,0x67b6eb54145b4431ULL,0x19d7e1d249afd679ULL,0xbd819bab110fccdfULL, + 0x263a2cfb9db3b381ULL,0x9c3a2deed4df0a4bULL,0x728d06e97d04e61fULL,0x8b1adfbc42449325ULL,0x6ec1d9397e053a1bULL,0xee2be5c766daf707ULL,0x80ba1e14810ac7abULL,0xdd2ae778f530f174ULL, + 0x3e708e703b9f0426ULL,0xe5b02fb60c84f17cULL,0x2f4ff35be3b70a0bULL,0x781b3c5f9b15565dULL,0xe76c636a6e124c3aULL,0xbde81eba8b496784ULL,0xa412f8e2443f0370ULL,0x15d42362999be45dULL, + 0x0f503ae2a4af76c1ULL,0x550e66dc08276fe7ULL,0x11e0c1fcbf3a33c6ULL,0x42be231006629f85ULL,0xedf7743e516ace49ULL,0xce436668436a2262ULL,0xe1ac7036446ca192ULL,0x73631cb5476fe13eULL, + 0xd47f82d4a160bcfaULL,0x258fb075b8ceb1f2ULL,0x4f818e8fdf5a8d25ULL,0x6685475e6fd31c9dULL,0xcef6385ae1e9b13fULL,0xe0a42594f0508bdbULL,0x5ad7ae16aef1f90dULL,0x45a155ef63f8a81eULL, + 0x8ca407c28034b95eULL,0xc93eb97617bdc560ULL,0x4ec24e8d339807e8ULL,0x91b734d6dd64a4ebULL,0xd0fece398f668b26ULL,0x4822cc4b141823d5ULL,0xb953bc32f09e4e00ULL,0xca0a7c6006d861aeULL, + 0x4c74d4470f33e712ULL,0x3cec1e0625a87cb0ULL,0x5cec0610e5962db4ULL,0xd971af1571a256aaULL,0xa044c983a2ef4ac9ULL,0xcaa1da63d74e9d00ULL,0xfb972d834673d881ULL,0x50747c5a03a26c8bULL, + 0x04349982a3e25566ULL,0xeef9075e18e1b896ULL,0x4c7bead092b2d24bULL,0xd99f72fb0a21ba55ULL,0xb93e09315005e541ULL,0x2a7a98389ece3205ULL,0xeb388ed11462f2f6ULL,0xb488b15a2e3460a6ULL, + 0x43f6cd67969d56afULL,0x9e0d872cdfc58a8bULL,0x401c1509a4e70377ULL,0x103d1a1308ad646cULL,0x078ee37e9d062427ULL,0x4e69c5acb9bef78cULL,0x521ec00136e66142ULL,0x8de1ecb2a634cd82ULL, + 0x0435d97a205b9d8bULL,0x6eb8f064056756d4ULL,0xd5e88a8bb6f8210eULL,0x070ef12dec9fd9eaULL,0x4d8495053bcc876aULL,0x12a75338a7404ce3ULL,0xd22b49e1b8a1db5eULL,0xec1f205114bfa5adULL, + 0x43ed81b5c4e83d33ULL,0xd9f358795efd488bULL,0x164a620f9deb4d0fULL,0xc6927bdbac6a7394ULL,0x45c28df79f9e0f03ULL,0x2868661efcd7e1a9ULL,0x7cf4e8d0ffa348f1ULL,0x6bd4c284398538e0ULL, + 0x56036e8c06d75fc1ULL,0x2dcf7bb73249a89fULL,0x81dd1d3de245e7ddULL,0xf578dc4bebd6e2a7ULL,0x4c028903df2ce7a0ULL,0xaee362889c39afacULL,0xdc847c31146404abULL,0x6304c0d8a4e97818ULL, + 0xfb6836c327d02dccULL,0x5ad009827a68bcc2ULL,0x1b24b44c005e912dULL,0xcc83d20f811fdcfeULL,0x36527ec1666fba0cULL,0x6994819714754635ULL,0xfcdcb1a8556da9c2ULL,0xa593426781a732b2ULL, + 0xe4ac8b33070d3aabULL,0x2643672b9a2cd5e5ULL,0x52eff79b1cfc9173ULL,0x665ca49b90a7c13fULL,0x5a8dda59b3efb998ULL,0x8a5b922d052f1341ULL,0xae9ebbab3cf9a530ULL,0x35986e7bf56da4d7ULL, + 0x63ee4cbd8088b454ULL,0xdb7f32f79a9e0c8aULL,0xb377d4186b2447cbULL,0xe3e982aad370219bULL,0x06ccc1e4c2a2a593ULL,0x72c368650773f24fULL,0xa13b4da795859423ULL,0x8bbf1d3375040c8fULL, + 0x03c187d0ad886aacULL,0x5c16878ab771b645ULL,0xb07dfc6fc74045abULL,0x2c6360bf7800caedULL,0x24295bb5b9c972a3ULL,0xc9e6f88e7c9a6dbaULL,0x90ffbf2492a79aa6ULL,0xde29d50a41c26ac2ULL, + 0xb1f0fb68d84d835dULL,0xc90caf39861dc1e6ULL,0x12e5b0467594f8d7ULL,0x26897ae265012b92ULL,0xbcf68a08a4d6755dULL,0x403ee41c0991fbdaULL,0x733e343e3bbf17e8ULL,0xd2c7980d679b3d65ULL, + 0x534acf4fda79e5acULL,0x68b83b3a8630215fULL,0x5c748b2ed085756eULL,0xb0317258e5d37cb2ULL,0x6735841ac5ccc2c4ULL,0x7d7dc96b3d9d5069ULL,0xa147e410fd1754bdULL,0x65296e94d399ddd5ULL, + 0x1e71c9a1deb8568bULL,0xa35daea080fb3d32ULL,0xe8b6f2662cf8fb81ULL,0x6d51afe89490696aULL,0x81beac6e51803a19ULL,0xe3d24b7f86219080ULL,0x727cfd9ddf6f463cULL,0x8c6865ca72284ee8ULL, + 0xe00df169d23233f3ULL,0x3e32279677cb637fULL,0x1f897c0e1da0cf6cULL,0xa651f5d831d6bbddULL,0xdd61af191a230c76ULL,0xbd527272cdaa5e4aULL,0xca753636d0abcd7eULL,0x78bdd37c370bd8dcULL, + 0xcddb27c17078c432ULL,0xe1961b9cb77fedb7ULL,0x1edc2f5cc2290570ULL,0x2c3fefca19cbd886ULL,0xcf880a36c2af389aULL,0x96c610fdbda71ceaULL,0xf03977a932aa8463ULL,0x8eb7763f8586d90aULL, + 0x831ab3edf0290a8fULL,0xcae81966cb47c387ULL,0xaad7dece184efb4fULL,0xdcfc53b34749110eULL,0x6698f23c4cb632f9ULL,0xc42a1ad6b91f8067ULL,0xb116a81d6284180aULL,0xebedf5f8e901326fULL, + 0x91633f0ab2cf8940ULL,0x72b0b1786f948f51ULL,0x2d28dc30782653c8ULL,0x88829849db903a05ULL,0xb8095d0c6a19d2bbULL,0x4b9e7f0c86f782cbULL,0x7af739882d907064ULL,0xd12be0fe8b32643cULL, + 0x9561f28b638a7e81ULL,0x54155cdf5980ddc3ULL,0xb2db4a96d26f247aULL,0x9d774e4e4787d100ULL,0x1a9e6e2e078637d2ULL,0x1c363e2d5e0ae06aULL,0x7493483ee9cfa354ULL,0x76843cb37f74b98dULL, + 0x0491f1bc789a283bULL,0x72d3ac3d880836f4ULL,0xaa1c5ea388e5402dULL,0x1b192421d5cc473dULL,0x5c0b99989dc84cacULL,0xb0a8482d9c6e75b8ULL,0x639961d03a191ce2ULL,0xda3bc8656d837930ULL, + 0xd7e0c4cdb30cfb3aULL,0x6d09b8c16c9db4c8ULL,0x40ba1a4207c8d9dfULL,0x6fd495f71c52c66dULL,0xfb0e169f275264daULL,0x80c2b746e57d8362ULL,0xedd987f749ad7222ULL,0xfdc229af4398ec7bULL, + 0xb0d1ed8452666a58ULL,0x4bcb6e00e6a9c3c2ULL,0x3c57411c26906408ULL,0xcfc2075513556400ULL,0xa08b1c505294dba3ULL,0xa30ba2868b7dd31eULL,0xd70ba90e991eca74ULL,0x094e142ce762c2b9ULL, + 0xb81d783e979f3925ULL,0x1efd130aaf4c89a7ULL,0x525c2144fd1bf7faULL,0x4b2969041b265a9eULL,0xed8e9634b9db65b6ULL,0x35c82e3203599d8aULL,0xdaa7a54f403563f3ULL,0x9df088ad022c38abULL, + 0xe5cfb066bb3fd30aULL,0x429169daeff0354eULL,0x809cf8523524e36cULL,0x136f4fb30155be1dULL,0x4826af011fbba712ULL,0x6ef0f0b4506ba1a1ULL,0xd9928b3177aea73eULL,0xe2bf6af25eaa244eULL, + 0x8d084f124237b64bULL,0x688ebe99e3ecfd07ULL,0x57b8a70cf6845dd8ULL,0x808fc59c5da4a325ULL,0xa9032b2ba3585862ULL,0xb66825d5edf29386ULL,0xb5a5a8db431ec29bULL,0xbb143a983a1e8dc8ULL, + 0x35ee94ce12ae381bULL,0x3a7f176c86ccda90ULL,0xc63a657e4606eacaULL,0x9ae5a38043cd04dfULL,0x9bec8d15ed251b46ULL,0x1f5d6d30caca5e64ULL,0x347b3b359ff20f07ULL,0x4d65f034f7e4b286ULL, + 0x9e93ba24f111661eULL,0xedced484b105eb04ULL,0x96dc9ba1f424b578ULL,0xbf8f66b7e83e9069ULL,0x872d4df4d7ed8216ULL,0xbf07f3778e2cbecfULL,0x4281d89998e73754ULL,0xfec85fbb8aab8708ULL, + 0x9a3c0deea5ba5b0bULL,0xe6a116ce42d05299ULL,0xae9775fee9b02d42ULL,0x72b05200a1545cb6ULL,0xbc506f7d31a3b4eaULL,0xe58930788bbd9b32ULL,0xc8bc5f37e4b12a97ULL,0x6b000c064a73b671ULL, + 0x13b5bf22765fa7d0ULL,0x59805bf01d6a5370ULL,0x67a5e29d4280db98ULL,0x4f53916f776b1ce3ULL,0x714ff61f33ddf626ULL,0x4206238ea085d103ULL,0x1c50d4b7e5809ee3ULL,0x999f450d85f8eb1dULL, + 0x658a6051e4c79e9bULL,0x1394cb73c66a9feaULL,0x27f31ed5c6be7b23ULL,0xf4c88f365aa6f8feULL,0x0fb0721f4aaa499eULL,0x68b3a7d5e3fb2a6bULL,0xa788097d3a92851dULL,0x060e7f8ae96f4913ULL, + 0x82eebe731a3a93bcULL,0x42bbf465a21adc1aULL,0xc10b6fa4ef030efdULL,0x247aa4c787b097bbULL,0x8b8dc632f60c77daULL,0x6ffbc26ac223523eULL,0xa4f6ff11344579cfULL,0x5825653c980250f6ULL, + 0xb2dd097ebc1aa2b9ULL,0x0788939337a0333aULL,0x1cf55e7137a0db38ULL,0x2648487f792c1613ULL,0xdad013363fcef261ULL,0x6239c81d0eabf129ULL,0x8ee761de9d276be2ULL,0x406a7a341eda6ad3ULL, + 0x4bf367ba4a493b31ULL,0x54f20a529bf7f026ULL,0xb696e0629795914bULL,0xcddab96d8bf236acULL,0x4ff2c70aed25ea13ULL,0xfa1d09eb81cbbbe7ULL,0x88fc8c87468544c5ULL,0x847a670d696b3317ULL, + 0xf133421e64bcb626ULL,0xaea638c826dee0b5ULL,0xd6e7680bb310346cULL,0xe06f4097d5d4ced3ULL,0x099614527512a30bULL,0xf3d867fde589a59aULL,0x2e73254f52d0c180ULL,0x9063d8a3333c74acULL, + 0xeda6c595d314e7bcULL,0x2ee7464b467899edULL,0x1cef423c0a1ed5d3ULL,0x217e76ea69cc7613ULL,0x27ccce1fe7cda917ULL,0x12d8016b8a893f16ULL,0xbcd6de849fc74f6bULL,0xfa5817e2f3144e61ULL, + 0x1f3541640821ee4cULL,0x1583eab40bc61992ULL,0x7490caf61d72879fULL,0x998ad9f3f76ae7b2ULL,0x1e181950a41157f7ULL,0xa9d7e1e6e8da3a7eULL,0x963784eb8426b95fULL,0x0ee4ed6e542e2a10ULL, + 0xb79d4cc5ac751e7bULL,0x93f96472fd4211bdULL,0x8c72d3d2c8de4fc6ULL,0x7b69cbf5df44f064ULL,0x3da90ca2f4bf94e1ULL,0x1a5325f8f12894e2ULL,0x0a437f6c7917d60bULL,0x9be7048696c9cb5dULL, + 0x949c9976e1337c26ULL,0x6faadebdd73d68e5ULL,0x9e158614f1b768d9ULL,0x22dfa5579cc4f069ULL,0xccd6da17be93c6d6ULL,0x24866c61a504f5b9ULL,0x2121353c8d694da1ULL,0x1c6ca5800140b8c6ULL, + 0xf1604a7dd4b79bb8ULL,0xaee806fb52c878c8ULL,0x34144f118d47b8e8ULL,0x72edf52b949f9054ULL,0xebfca84e2127015aULL,0x9051d0c09cb7cef3ULL,0x86e8fe58296deec8ULL,0x33b2818841010d74ULL, + 0xbd5660ed9aed9f40ULL,0x70ca6ad1532a8c99ULL,0xc4978bfb95c371eaULL,0xe5464d0d7003109dULL,0x1af32fdfd9e535efULL,0xabf57ea798c9185bULL,0xed7a741712b42488ULL,0x8e0296a7e97286faULL, + 0x01079383171b445fULL,0x9bcf21e38131ad4cULL,0x8cdfe205c93987e8ULL,0xe63f4152c92e8c8fULL,0x729462a930add43dULL,0x62ebb143c980f05aULL,0x4f3954e53b06e968ULL,0xfe1d75ad242cf6b1ULL, + 0x8b57416e1f017d5eULL,0x375333967674e99bULL,0x6e6d94c0e8f488a0ULL,0xb93a787adc16f95eULL,0xc3ac51a2dcc99cccULL,0xc134b4139aa47c1dULL,0xf28fcdafafdfd8d5ULL,0x0d57bd8e10b831edULL, + 0x9276fbccf0bcfc46ULL,0x3a822aceb5cffee6ULL,0x328ed2fec75d915bULL,0xa145c113c359476cULL,0xf61a81538be17bcdULL,0x01e867c3aa6c3d8fULL,0x5634e15d6516c82fULL,0xc1437bd26948b9b0ULL, + 0xd2fcd2006c19d4c7ULL,0xa0f3c437e1b1e976ULL,0xf0545ff694f237e8ULL,0xdd10ec3fc0bf8bb1ULL,0x4f89696cac7cd3e1ULL,0xed3714ec5f24bfe6ULL,0x363eb1d85faf7706ULL,0xfcbd604dc027cc32ULL, + 0x5f95c6c7af8685c8ULL,0xd4c1c8ce2f8f01aaULL,0xc44bbe322574692aULL,0xb8003478d4a4a068ULL,0x7c8fc6e52eca3cdbULL,0xea1db16bec04d399ULL,0xb05bc82e8f2bc5cfULL,0x763d517ff44793d2ULL, + 0x16ce8eddc355363bULL,0x4af2f70ff8820d6eULL,0xcb7ed4d27661a508ULL,0x41d3444edd195472ULL,0x17fea2b438da9649ULL,0x9bf69356aeb4a200ULL,0xa13b5f916ab19c3dULL,0xc0519c14dc9360a6ULL, + 0xc2571ae92e42e171ULL,0xcb31ab63ed41ccf9ULL,0x37f3c576b5c8854fULL,0x66e5191bc62392a1ULL,0x71565a1c6cd5683bULL,0x484b0283606fe689ULL,0xf3a25d6767e2fda6ULL,0x87ba21de8a65c0a4ULL, + 0xde74e49ca70684d1ULL,0x3ae8766133e80c3dULL,0x5984a2a916a5c34dULL,0x09a83eccb8298c35ULL,0x9a19867caa4ca4c0ULL,0x02085610b375b8ffULL,0xf296328bf70396dcULL,0x9c9ddc4cde6fae63ULL, + 0x4451c1b808bd98d0ULL,0x644b1cd46575f240ULL,0x6907eb337375d270ULL,0x56c8bebdfa2286bdULL,0xc713d2acc4632b46ULL,0x17da427aafd60242ULL,0x313065b7c95c7546ULL,0xf8239898bf17a3deULL, + 0x94683d260b083b6eULL,0x0a3752eb06f6a54dULL,0x48bedc23752074ddULL,0x637622fc3e822593ULL,0xea0005136be55d3bULL,0x9f5e12f4324d006dULL,0x529486a964fc0270ULL,0x09ba0d0c923399e6ULL, + 0x363858473a977080ULL,0x4cf8e1b80c6a6ab6ULL,0x919a5c6c0482261eULL,0x517a9ad0e5ce4806ULL,0x2792d40c056aa7aaULL,0x4c7c6adae56c61b0ULL,0xf19cb178a4b19e0cULL,0x046d5c4fe4ba267fULL, + 0xd3e926ab121550b3ULL,0xe4975e4ac147ce84ULL,0x7a8be0f95eff722aULL,0x71e4702c6fd4f2a0ULL,0x13b92acf3cb7b280ULL,0xc588716d28272d73ULL,0x862c7bf3daa9fe5cULL,0x78c008f2e2a79e42ULL, + 0xf3b7963f4c830320ULL,0x842c7aa0903203e3ULL,0xaf22ca0ae7327afbULL,0x38e13092967609b6ULL,0x73b8fb62757558f1ULL,0x3cc3e831f7eca8c1ULL,0xe4174474f6331627ULL,0xa77989cac3c40234ULL, + 0xae8317f4b0166f7aULL,0xfbd3e3f7ceec74e6ULL,0xfdb516ace0874bfdULL,0x3d846019c681f3a3ULL,0x0b12ee5c7c1620b0ULL,0xba68b4dd2b63c501ULL,0xac03cd326668c51eULL,0x2a6279f74e0bcb5bULL, + 0xfd8e139f8f5fcda8ULL,0xf3e558c4bdee5bfdULL,0xd76cbaf4e33f9f77ULL,0x3a4c97a471771969ULL,0xda27e84bf6dce6a7ULL,0xff373d9613e6c2d1ULL,0xf115193cd759a6e9ULL,0x3f9b702563d2262cULL, + 0x12536fea87baa627ULL,0x58c1fec1f72aa680ULL,0x6c29b637601e5dc9ULL,0x9e3c3c1cde9e01b9ULL,0xefc8127b2bcfe0b0ULL,0x351071022a12f50dULL,0x6ccd6cb14879b397ULL,0xf792f804f8a82f21ULL, + 0x8c3184911a335cc8ULL,0x563459ba6a5913e4ULL,0x1b920d61c7b32919ULL,0x805ab8b6a02425adULL,0x2ac512da8d006086ULL,0x6ca4846abcf5c0fdULL,0xafea51d8ac2138d7ULL,0xcb647545344cd443ULL, + 0xa3f4f521e447f2c4ULL,0x81b8da7a604291f0ULL,0xd680bc467d5926deULL,0x84f21fd534a1202fULL,0x1d1e31814e9df3d8ULL,0x1ca4861a39ab8d34ULL,0x809ddeec5b19aa4aULL,0x59f72f7e4d329366ULL, + 0x9f1b2466cdedca85ULL,0x140bb7101a09538cULL,0xac8ae8515e11115dULL,0x0d63ff676f03f59eULL,0x755e55517d234afbULL,0x61c2db4e7e208fc1ULL,0xaa9859cef28a4b5dULL,0xbdd6d4fc34af030fULL, + 0x3f39e67f906151e5ULL,0xcea27f5f55e10649ULL,0xdca1d4e1c17cf7b7ULL,0x0c326d122fe2362dULL,0x05f7ac337dd35df3ULL,0x0c3b7639c396dbdfULL,0x0912f5ac03b7db1cULL,0x9dea4b705c9ed4a9ULL, + 0x511053e453544774ULL,0x834d0ecc3adba2bcULL,0x4215d7f7bae371f5ULL,0xfcfd57bf6c8663bcULL,0xded2383dd6901b1dULL,0x3b49fbb4b5587dc3ULL,0xfd44a08d07625f62ULL,0x3ee4d65b9de9b762ULL, + 0x3e56fe5bdf53aad0ULL,0x51314de5e4604a67ULL,0x386ad98607a261a0ULL,0x8b7e021217afcc91ULL,0xcbf411273b72aec5ULL,0x13c85d05c4f9f509ULL,0xeda56845b6484b57ULL,0x13cb1642b3d0995bULL, + 0x5a994b6e717815deULL,0xd995c7a0a7e131d1ULL,0xc8b46df226c023aaULL,0x8cfd094d702afcedULL,0x7bc743cdced6a886ULL,0xb7d70ec41fcabe75ULL,0x2a6c9e47ddac9390ULL,0x720694259310aa90ULL, + 0xa607b3263e8f793cULL,0xa541166ecde3b289ULL,0xf44ff2924a915b21ULL,0x68bea906b58ecda6ULL,0xd85b37b440292897ULL,0xd2a508ae4b768423ULL,0xd10bb79da413bbbeULL,0x0262f481061491f4ULL, + 0xd17e80f55464d0ebULL,0x89d3e1a767a613a7ULL,0x77791260c8c97dadULL,0xec2ff21fe4f0cbe7ULL,0xed984ac2e9e6bc10ULL,0xe3c53de877cba305ULL,0x9bbaf9b283624fdcULL,0x5e9451cd1485c0ecULL, + 0xfdfc9f63b7abad11ULL,0xc7ec3a9263a46189ULL,0x49ebee42f67037b7ULL,0x8247f504cdac1710ULL,0xfc518f8d397583e5ULL,0xe7d24de70c3f8c2eULL,0x354832669c5edcb1ULL,0x94bba483f8c2cefcULL, + 0x1f13756db1761ec8ULL,0xe53c8b98a4b97e55ULL,0xb2aee3f84096cc28ULL,0x48c361a0920f1a8dULL,0xa98b672d8c31190aULL,0x7bc1e7d1001855d4ULL,0x242cfb07bf3f4b2aULL,0x9bf44a3f32a28bc4ULL, + 0xeba8976299da550cULL,0xb2c1781a16baa042ULL,0x3068b082788c2f9dULL,0xc0fa414594869a9eULL,0x73bd9e39d50b693fULL,0xb79e2a9c988e2c5eULL,0xf1cb8de40f8f9f62ULL,0x415b04ee1ea50c7bULL, + 0x64e5137d0d63d1faULL,0x658fc05202a9d89fULL,0x4889487450436309ULL,0xe9ae30f8d598da61ULL,0x2ed710d1818baf91ULL,0xe27e9e068b6a0c20ULL,0x1e28dcfb1c1a6b44ULL,0x883acb64d6ac57dcULL, + 0x8735728dc2c6ff70ULL,0x79d6122fc5dc2235ULL,0x23f5d00319e277f9ULL,0x7ee84e25dded8cc7ULL,0x91a8afb063cd880aULL,0x3f3ea7c63574af60ULL,0x0cfcdc8402de7f42ULL,0x62d0792fb31aa152ULL, + 0xb02c83f9dec31a21ULL,0x988c8b236ad9d573ULL,0x53e983aea57be365ULL,0xe968734d646f834eULL,0x9137ea8f5da6309bULL,0x10f3a624c1f1ce16ULL,0x782a9ea2ca440921ULL,0xdf94739e5b46f1b5ULL, + 0x1df165a434a35ea8ULL,0x3418e0f74d4412f6ULL,0x5af1f8af518836c3ULL,0x42ceef4d130e1965ULL,0x5560ca0b543a1957ULL,0xc33761e5886cb123ULL,0x66624b1ffe98ed30ULL,0xf772f4bf1090997dULL, + 0x56f8410ef4f8b16aULL,0x97241afec47b266aULL,0x0a406b8e6d9c87c1ULL,0x803f3e02cd42ab1bULL,0x7f0309a804dbec69ULL,0xa83b85f73bbad05fULL,0xc6097273ad8e197fULL,0xc097440e5067adc1ULL, + 0xc507b6dd418e7dddULL,0x39888d93472f19d6ULL,0x7eae26be0c27eb4dULL,0x17b53ed3fbabb884ULL,0xfc27021b2b01ae4fULL,0x88462e87cf488682ULL,0xbee096ec215e2d87ULL,0xeb2fea9ad242e29bULL, + 0xbbcc00c756b95bceULL,0x5ec03906616da680ULL,0x79162ee672214252ULL,0x43132b6386a892d2ULL,0x4bdd3ff22f3263bfULL,0xd5b3733c9cd0a142ULL,0x592eaa8244415ccbULL,0x663e89248d5474eaULL, + 0x0d38ab35ce7c42d4ULL,0x9fd493ef82feab10ULL,0x46056b6d82111b45ULL,0xda11dae173efc5c3ULL,0xdc7402785545a7fbULL,0xbdb2601c40d507e6ULL,0x121dfeeb7066fa58ULL,0x214369a839ae8c2aULL, + 0x3f747fa0b311898cULL,0xe2a272e4cd0eac65ULL,0x4bba5851f914d0bcULL,0x7a1a9660c4a43ee3ULL,0xe5a367cea1c8cde9ULL,0x9d958ba97271abe3ULL,0xf3ff7eb63d1615cdULL,0xa2280dcef5ae20b0ULL, + 0x8c0ed566d4312483ULL,0x5179a95d643e216fULL,0xcc185fec17044493ULL,0xb306333954991a21ULL,0xd801ecdb0081a726ULL,0x0149b0c64fa89bbbULL,0xafe9065a4391b6b9ULL,0xedc92786d633f3a3ULL, + 0xd6d9d9e37a6a308bULL,0x623758304c2767d3ULL,0x874a8bc6f38cbeb6ULL,0xd94d3f1accb6fd9eULL,0x92a9735bba21f248ULL,0x272ad0e56cd1efb0ULL,0x7437b69c05b03284ULL,0xe7f047026948c225ULL, + 0x9ae868a9e9adfe1cULL,0x3984403d314e39bbULL,0xb5875720f2fe378fULL,0x33f901e0ba44a628ULL,0xea1125fe3652438cULL,0xae9ec4e69dd1f20bULL,0x1e740d9ebebf7fbdULL,0x6dbd3ddc42dbe79cULL, + 0x266344a43794f8dcULL,0xdcca923a483c5c36ULL,0x2d6b6bbf3f9d10a0ULL,0xb320c5ca81d9bdf3ULL,0x620e28ff47b50a95ULL,0x933e3b01cef03371ULL,0xf081bf8599100153ULL,0x183be9a0c3a8c8d6ULL, + 0x8a9443d77613aa81ULL,0x8010080085fe6584ULL,0x70fc4dbc7fb10288ULL,0xf58280d3e86beee8ULL,0x14fdd82f7c978c38ULL,0xdf1204c10de44d7bULL,0xa08a1c844160252fULL,0x591554cac17646a5ULL, + 0x7ddc81ea77b46a08ULL,0xcf5a6477c7480699ULL,0x43a8cb346633f683ULL,0x1b867e6b92363c60ULL,0x439211141f60558eULL,0xcdbcdd632f41450eULL,0x7fc04601cc630e8bULL,0xea7c66d597038b43ULL, + 0x34fc8820fbeee3f9ULL,0x93e5349049091afdULL,0x764b9be59a31f35cULL,0x71f3786457e3d924ULL,0x02fb34e0943aa75eULL,0xa18c9c58ab8ff6e4ULL,0x080f31b133cf0d19ULL,0x5c9682db083518a7ULL, + 0xb6c185c341dca566ULL,0x7de7fedad8622aa3ULL,0x99e84d92901b6dfbULL,0x30a02b0e7c4ad288ULL,0xc7c81daa2fd3cf36ULL,0xd1319547df89e59fULL,0xb2be8184cd496733ULL,0xd5f449eb93d3412bULL, + 0xc492ec644cd8f64cULL,0x58a2d790279d7b51ULL,0x0ced1fc51fc75256ULL,0x3e658aed8f433017ULL,0x0b61942e05da59ebULL,0xba3d60a30ddc3722ULL,0x7c311cd1742e7f87ULL,0x6473ffeef6b01b6eULL, + 0x8303604f692ac542ULL,0xf079ffe1227b91d3ULL,0x19f63e6315aaf9bdULL,0xf99ee565f1f344fbULL,0x8a1d661fd6219199ULL,0x8c883bc6d48ce41cULL,0x1065118f3c74d904ULL,0x713889ee0faf8b1bULL, + 0xc035f697960eb8c7ULL,0xf1599f2ce2de04d3ULL,0x892450f8d2ad9228ULL,0x7d48129bb829c1abULL,0x24d785e13a50afc9ULL,0x2745ba2763a96ee0ULL,0x956534013bfb6d7bULL,0x536202671bad2a42ULL, + 0x972b3f8f81a1b3beULL,0x4f3ce145ce2764a0ULL,0xe2d0f1cc28c4f5f7ULL,0xdeee0c0dc7f3985bULL,0x7df4adc0d39e25c3ULL,0x40619820c467a080ULL,0x440ebc9361cf5a58ULL,0x527729a6422ad600ULL, + 0xa691398a4a9eb3f0ULL,0x56c1dbff3b99a48fULL,0x9a87e1b91b4b5b32ULL,0xad6396145378b5feULL,0x437a243ec26b5302ULL,0x0275878c3ccb4c10ULL,0x0e81e4a21de07015ULL,0x0c6265c9850df3c0ULL, + 0xca6c0937b1b76ba6ULL,0x1a2eab854d2026dcULL,0xb1715e1519d9ae0aULL,0xf1ad9199bac4a026ULL,0x35b3dfb807ea7b0eULL,0xedf5496f3ed9eb89ULL,0x8932e5ff2d6d08abULL,0xf314874e25bd2731ULL, + 0xc8327149a8c25ff6ULL,0x29bf2556782e6569ULL,0x9012f5c6cd68fc38ULL,0x3e67e8bd3b982ad5ULL,0x5e3a75386ecdca88ULL,0xf297eaa6c1753a04ULL,0x10121e5405db3256ULL,0xab9697d4f0851055ULL, + 0xefb26a753f73f449ULL,0x1d1c94f88d44fc79ULL,0x49f0fbc53bc0dc4dULL,0xb747ea0b3698a0d0ULL,0x5218c3fe228d291eULL,0x35b804b543c129d6ULL,0xfac859b8d1acc516ULL,0x6c10697d95d6e668ULL, + 0xe5d27171f6bdf1bfULL,0x0b77b876facb0d8fULL,0xda95471d8496a31bULL,0x46a50dbb3f16b103ULL,0x2a4f3f977b865bffULL,0x848195e66b1c198cULL,0x491ad08821702ea6ULL,0x3f20b43749035228ULL, + 0xc38e438f0876fd4eULL,0x45f0c30783d2f383ULL,0x203cc2ecb10934cbULL,0x6a8f24392c9d46eeULL,0xf16b431b65ccde7bULL,0x41e2cd1827e76a6fULL,0xb9c8cf8f4e3484d7ULL,0x64426efd8315244aULL, + 0xe6ec98093a69fc01ULL,0x7e20fecbfaa9dfc2ULL,0x5cfdbb07f56f2a55ULL,0xb1cd68680bbdbfdfULL,0x247b4995986eb9edULL,0x74785bf53dd0955eULL,0x88f74f61c0c7a201ULL,0x8861a15b5d01a80dULL, + 0x1c0a8e44fc94dea3ULL,0x34c8cdbfdad6a0b0ULL,0x919c384004113cefULL,0xfd32fba415490ffaULL,0x58d190f6795dcfb7ULL,0xfef01b0383588bafULL,0x9e6d1d63ca1fc1c0ULL,0x53173f96f0a41ac9ULL, + 0x54637e4182997cc1ULL,0x08c5a96ce3720c9cULL,0x78bce01c11de5d45ULL,0x49d623e50dfdd75aULL,0x8c72a4680fb2a3acULL,0xcc53bbff319c25afULL,0x198eba7978a92421ULL,0xcd61f28ba3bdecf3ULL, + 0x2b1d402aba16f73bULL,0x2fb310148cf9b9fcULL,0x2d51e60e446ef7bfULL,0xc731021bb91e1745ULL,0x9d3b47244fee99d4ULL,0x4bca48b6fac5c1eaULL,0x70f5f514bbea9af7ULL,0x751f55a5974c283aULL, + 0x23899fe8662595c2ULL,0x495d672711a80773ULL,0x86c971d2b0d1d43bULL,0xb518637c93b7a65fULL,0x30e453bad98c99ceULL,0xba6e0d4a14d39f5bULL,0xf7db02a6431ce415ULL,0xcd909c7cf6e1d823ULL, + 0x6e30251acb452fdbULL,0x31ee696550f30650ULL,0xb0b3e508933548d9ULL,0xb8949a4ff4b0ef5bULL,0x208b83263c88f3bdULL,0xab147c30db1d9989ULL,0xed6515fd44d4df03ULL,0x17a12f75e72eb0c5ULL, + 0x25914f7881fdad90ULL,0xcf638f560d2cf6abULL,0xb90bc03fcc054de5ULL,0x932811a718b06350ULL,0x2f00b3309bbd11ffULL,0x76108a6fb4044974ULL,0x801bb9e0a851d266ULL,0x0dd099bebf8990c1ULL, + 0x14c6dd8a58d6cd46ULL,0x9cb633b58e6634d2ULL,0xc1305047f81bc328ULL,0x12ede0e226a177e5ULL,0x332cca62065a6f4fULL,0xc3a47ecd67be487bULL,0x741eb1870f47ed1cULL,0x99e66e58e7598b14ULL, + 0xebd6a6777b0ac93dULL,0xa6e37b0d78f5e0d7ULL,0x2516c09676f5492bULL,0x1e4bf8889ac05f3aULL,0xcdb42ce04df0ba2bULL,0x935d5cfd5062341bULL,0x8a30333382acac20ULL,0x429438c45198b00eULL, + 0xfb2838be67e573e0ULL,0x05891db94084c44bULL,0x9131137396c1c2c5ULL,0x6aebfa3fd958444bULL,0xac9cdce9e56e55c1ULL,0x7148ced32caa46d0ULL,0x2e10c7efb61fe8ebULL,0x9fd835daff97cf4dULL, + 0x6c626f56c1770616ULL,0x5351909e09da9a2dULL,0xe58e6825a3730e45ULL,0x9d8c8bc003ef0a79ULL,0x543f78b6056becfdULL,0x33f13253a090b36dULL,0x82ad4997794432f9ULL,0x1386493c4721f502ULL, + 0x3794eefa5abea82aULL,0x8dc611b993fe62d4ULL,0x69f1af37281ef606ULL,0x6af546c839839e69ULL,0x625578c7c977ec23ULL,0xa8de294cbd5c0576ULL,0xe2ddaf0f7cd1a4c0ULL,0x8243fc704f95f4d4ULL, + 0xe566f400b008733aULL,0xcba0697d512e1f57ULL,0x9537c2b240509cd0ULL,0x5f989c6957353d8cULL,0x7dbec9724c3c2b2fULL,0x90e02fa8ff031fa8ULL,0xf4d15c53cfd5d11fULL,0xb3404fae48314dfcULL, + 0xa36da109081e9387ULL,0xfb9780d78c935828ULL,0xd5940332e540b015ULL,0xc9d7b51be0f466faULL,0xfaadcd41d6d9f671ULL,0xba6c1e28b1a2ac17ULL,0x066a7833ed201e5fULL,0x19d99719f90f462bULL, + 0xf02cc3a9f327a07fULL,0xefb27a9b4490937dULL,0x81451e96b1b3afa5ULL,0x67e24de891883be4ULL,0x1ad65d4770869e54ULL,0xd36291a464a3856aULL,0x070a1abf7132e880ULL,0x9511d0a30e28dfdfULL, + 0xfdeed650f8d1cac4ULL,0xeb99194b6d16bda5ULL,0xb53b19f71cabbe46ULL,0x5f45af5039b9276cULL,0xd0784c6126ee9d77ULL,0xf7a1558b0c02ca5dULL,0xb61d6c59f032e720ULL,0xae3ffb95470cf3f7ULL, + 0x9b185facc72a4be5ULL,0xf66de2364d848089ULL,0xba14d07c717afea9ULL,0x25bfbfc02d551c1cULL,0x2cef0ecd4cdf3d88ULL,0x8cee2aa3647f73c4ULL,0xc10a7d3d722d67f7ULL,0x090037a294564a21ULL, + 0xe6567987fa9ced27ULL,0x36b2d8842a9e5dbcULL,0xf4bdeec6380d8e8cULL,0xb5e6a6b0dbc300d0ULL,0x7ba7e9b9592bef36ULL,0x2b373c4fee81b749ULL,0x484b5e01f0ce596bULL,0x7cc51c62c0bf54ccULL, + 0x6ac07bb84f3815c4ULL,0xddb9f6241aa9017eULL,0x31e30228ca85720aULL,0xe59d63f57cb75838ULL,0x69e18e777baad2d0ULL,0x2cfdb784d42f5d73ULL,0x025dd53df5774983ULL,0x2f80e7cee042cd52ULL, + 0x4bf56bafec695bb0ULL,0x22da1ca8f13c78adULL,0x0f9c4b131182abb0ULL,0x02ea555aae7d249eULL,0x868583f25e05d9e3ULL,0xe09cdcdf70382afaULL,0xdf072ec787080408ULL,0x0a317847cbf75658ULL, + 0x43f18d7f4d6ee4abULL,0xd3ac8cde9570c3dcULL,0x527e49070b8c9b2aULL,0x716709a7c5a4c0f1ULL,0x930852b0916a26b1ULL,0x3cc17fcf4e071177ULL,0x34f5e3d459694868ULL,0xee0341aba28f655dULL, + 0xf431f462060b5f61ULL,0xa56f46b47bd057c2ULL,0x348dca6c47e1bf65ULL,0x9a38783e41bcf1ffULL,0x7a5d33a9da710718ULL,0x5a7799872e0aeaf6ULL,0xca87314d2d29d187ULL,0xfa0edc3ec687d733ULL, + 0x1c894849cb198ac7ULL,0xa884a93d0f264665ULL,0x2da964ef9b200678ULL,0x3c351b87009834e6ULL,0xafb2ef9fe2c4b44bULL,0x580f6c473326790cULL,0xb84805210b02264aULL,0x8ba6f9e242a194e2ULL, + 0x39d934abd3c095f1ULL,0x04b261bee4b76d71ULL,0x1d2e6970e73e6984ULL,0x879fb23b5e5fcb11ULL,0x11506c72dfd75490ULL,0x3a97d08561bcf1c1ULL,0x43201d82bf5e7007ULL,0x7f0ac52f798232a7ULL, + 0xb25101fb319d7682ULL,0xb02931290a982feeULL,0x51c1c9b90261b344ULL,0x0e008c5bbfd371faULL,0xd866dd1c0278ca33ULL,0x666f76a6e5aa53b1ULL,0xe5cfb7796013a2cfULL,0x1d3a1aada3521836ULL, + 0x76b4131a567193ecULL,0xaf3c305ae5f6e70bULL,0x9587bd39031eebddULL,0x5709def871bbe831ULL,0x570599830eb2b669ULL,0x4d80ce1b875b7029ULL,0x838a7da80364ac16ULL,0x2f431d23be1c83abULL, + 0xe781276638235d4eULL,0x1c62bd67496e3298ULL,0x8378660c3f175bc8ULL,0x4d04e18917afdd4dULL,0x32a8160185a8068cULL,0xdb58e4e192b29a85ULL,0xe8a65b86c70d8a3bULL,0x5f0e6f4e98a0403bULL, + 0x2c2492b73f894ae0ULL,0xf59df3e5b75f18ceULL,0x7cb740d28f53cad0ULL,0x3eb585fbc4f01294ULL,0x17da0c8632c7f717ULL,0xeb8c795baf943f4cULL,0x4ee23fb5f67c51d2ULL,0xef18757568889949ULL, + 0x3ea011a4e822f0d0ULL,0xbc647ad15a8704f8ULL,0xbb315b3550c6820fULL,0x863dec3db7e76becULL,0x01ff5d3af017bfc7ULL,0x20054439976b8229ULL,0x067fca370bbd0d3bULL,0xf63dde647f5e3d0fULL, + 0x75d9bc15adf7cccfULL,0x81a3e5d6dfa1e1b0ULL,0x8c39e444249bc17eULL,0xf37dccb28ea7fd43ULL,0xda654873907fba12ULL,0x35daa6da4a372904ULL,0x0564cfc66283a6c5ULL,0xd09fa4f64a9395bfULL, + 0xb510b3b56aa39dffULL,0x59b43da29f8e4d8cULL,0xa8ce31fd9e4c4b9fULL,0x0e20be26c1303c01ULL,0x18187182e8ee47c9ULL,0xd9687cdb7db98101ULL,0x7a520e4da1e14ff6ULL,0x429808ba8836d572ULL, + 0x174d46996d16768eULL,0x9fc4ff6a628bf217ULL,0x77705a94154e490dULL,0x9d96dd288d2d997aULL,0x77e2d9d8ce5d72c4ULL,0x9d06c5a4c11c714fULL,0x02aa513679e4a03eULL,0x1386b3c2030ff28bULL, + 0x26a42d69ea40dc3aULL,0xdc84ad22aecc018fULL,0x25c36c7b3270f04aULL,0x46ba6d4750fa72edULL,0x6c37d1c593e58a8eULL,0xa2394731120c088cULL,0xc3be4263cb6e86daULL,0x2c417d367126d038ULL, + 0xcca523bb440e2229ULL,0x324673a273ef4d04ULL,0xaf3adf343e11ec39ULL,0x6136d7f1dc5968d3ULL,0x7a7b2899b053a927ULL,0x3eaa2661ae067ecdULL,0x8549b9c802779cd9ULL,0x061d7940c53385eaULL, + 0xe07141fcaaa2902bULL,0x539ad799e4f69ad3ULL,0xa6453f94813f9ffdULL,0xc58d3c48375bc2f7ULL,0xb3326fad5dc64e96ULL,0x3aafcaa9b240e354ULL,0x1d1b0903aca1e7a9ULL,0x4ceb97671211b8a0ULL, + 0x1eb4de4687032d58ULL,0xc54f3d835e2c79e0ULL,0x07818df45d04ef23ULL,0x55faa9c8673d41b4ULL,0xced64f6f89b95355ULL,0x4860d2eab7415c84ULL,0x5fdb9bd2050ebad3ULL,0xdb53e0cc6685a5bfULL, + 0x919fca5fabfae1caULL,0x937afaac1a21459bULL,0x9e0ca91c1f66a4d2ULL,0x194cc7f323ec1331ULL,0xad25143a8aa11690ULL,0xbe40ad8d09b59e08ULL,0x37d60d9be750860aULL,0x6c53b008c6bf434cULL, + 0x832d7080eb6b242dULL,0xd30bd0233b71e246ULL,0x7027991bbe31139dULL,0x68797e91462e4e53ULL,0x423fe20a6b4e185aULL,0x82f2c67e42d9b707ULL,0x25c817684cf7811bULL,0xbd53005e045bb95dULL, + 0xe5f649be9d8e68fdULL,0xdb0f05331b044320ULL,0xf6fde9b3e0c33398ULL,0x92f4209b66c8cfaeULL,0xe9d1afcc1a739d4bULL,0x09aea75fa28ab8deULL,0x14375fb5eac6f1d0ULL,0x6420b560708f7aa5ULL, + 0x9eae499c6254dc41ULL,0x7e2939247a837e7eULL,0x74aec08c090524a7ULL,0xf82b92198d6f55f2ULL,0x493c962e1402cec5ULL,0x9f17ca17fa2f30e7ULL,0xbcd783e8e9b879cbULL,0xea3d8c145a6f145fULL, + 0xdede15e75e0dee6eULL,0x74f24872dc628aa2ULL,0xd3e9c4fe7861bb93ULL,0x56d4822a6187b2e0ULL,0xb66417cfc59826f9ULL,0xca2609692408169eULL,0xedf69d06c79ef885ULL,0x00031f8adc7d138fULL, + 0x103c46e60ebcf726ULL,0x4482b8316231470eULL,0x6f6dfaca487c2109ULL,0x2e0ace9762e666efULL,0x3246a9d31f8d1f42ULL,0x1b1e83f1574944d2ULL,0x13dfa63aa57f334bULL,0x0cf8daed9f025d81ULL, + 0x30d78ea800ee11c1ULL,0xeb053cd4b5e3dd75ULL,0x9b65b13ed58c43c5ULL,0xc3ad49bdbd151663ULL,0x99fd8e41b6427990ULL,0x12cf15bd707eae1eULL,0x29ad4f1b1aabb71eULL,0x5143e74d07545d0eULL, + 0x30266336c88bdee1ULL,0x25f293065876767cULL,0x9c078571c6731996ULL,0xc88690b2ed552951ULL,0x274f2c2d852705b4ULL,0xb0bf8d444e09552dULL,0x7628beeb986575d1ULL,0x407be2387f864651ULL, + 0x0e5e3049a639fc6bULL,0xe75c35d986003625ULL,0x0cf35bd85dcc1646ULL,0x8bcaced26c26273aULL,0xe22ecf1db5536742ULL,0x013dd8971a9e068bULL,0x17f411cb8a7909c5ULL,0x5757ac98861dd506ULL, + 0x85de1f0d1e935abbULL,0xdefd10b4154de37aULL,0xb8d9e392369cebb5ULL,0x54d5ef9b761324beULL,0x4d6341ba74f17e26ULL,0xc0a0e3c878c1dde4ULL,0xa6d7758187d918fdULL,0x6687601502ca3a13ULL, + 0xc7313e9cf36658f0ULL,0xc433ef1c71f8057eULL,0x853262461b6a835aULL,0xc8f053987c86394cULL,0xff398cdfe983c4a1ULL,0xbf5e816203b7b931ULL,0x93193c46b7b9045bULL,0x1e4ebf5da4a6e46bULL, + 0xf9942a6043a24fe7ULL,0x29c1191effb3492bULL,0x9f662449902fde05ULL,0xc792a7ac6713c32dULL,0x2fd88ad8b737982cULL,0x7e3a0319a21e60e3ULL,0x09b0de447383591aULL,0x6df141ee8310a456ULL, + 0xaec1a039e6d6f471ULL,0x14b2ba0f1198d12eULL,0xebc1a1603aeee5acULL,0x401f4836e0b964ceULL,0x2ee437964fd03f66ULL,0x3fdb4e49dd8f3f12ULL,0x6ef267f629380f18ULL,0x3e8e96708da64d16ULL, + 0xbc19180c207674f1ULL,0x112e09a733ae8fdbULL,0x996675546aaeb71eULL,0x79432af1e101b1c7ULL,0xd5eb558fde2ddec6ULL,0x81392d1f5357753fULL,0xa7a76b973ae1158aULL,0x416fbbff4a899991ULL, + 0x9e65fdfd0d4a9dcfULL,0x7bc29e48944ddf12ULL,0xbc1a92d93c856866ULL,0x273c69056e98dfe2ULL,0x69fce418cdfaa6b8ULL,0x606bd8235061c69fULL,0x42d495a06af75e27ULL,0x8ed3d5056d873a1fULL, + 0xaf5528416ab25b6aULL,0xc6c0ffc72b1a4523ULL,0xab18827b21c99e03ULL,0x060e86489034691bULL,0x5207f90f93c7f398ULL,0x9f4a96cb82f8d10bULL,0xdd71cd793ad0f9e3ULL,0x84f435d2fc3a54f5ULL, + 0x4b03c55b8e33787fULL,0xef42f975a6384673ULL,0xff7304f75051b9f0ULL,0x18aca1dc741c87c2ULL,0x56f120a72d4bfe80ULL,0xfd823b3d053e732cULL,0x11bccfe47537ca16ULL,0xdf6c9c741b5a996bULL, + 0xee7332c7904fc3faULL,0x14a23f45c7e3636aULL,0xc38659c3f091d9aaULL,0x4a995e5db12d8540ULL,0x20a53becf3a5598aULL,0x56534b17b1eaa995ULL,0x9ed3dca4bf04e03cULL,0x716c563ad8d56268ULL, + 0x5043dea7e0f222c2ULL,0x309d42ac72e65142ULL,0x94fe9ddd9216cd30ULL,0xd6539c7d0f87feecULL,0x03c5a57c432ac7d7ULL,0x72692cf0327fda10ULL,0xec28c85f280698deULL,0x2331fb467ec283b1ULL, + 0xa0158eeae457a477ULL,0xd19857dbee6ddc05ULL,0xb326522418c41671ULL,0x3ffdfc7e3c2c0d58ULL,0x3a3a525426ee7cdaULL,0x341b0869df02c3a8ULL,0xa023bf42723bbfc8ULL,0x3d15002a14452691ULL, + 0xc961b2f687500b96ULL,0x795510e72dcd9425ULL,0x0308172978615433ULL,0xe5d0145465445029ULL,0x5bd13302bf690cbeULL,0x44e48831731eca67ULL,0x73306bc72b8038a4ULL,0x351d151ebf57bf02ULL, + 0x5ef7324c85edfa30ULL,0x2597655487d4f3daULL,0x352f5bc0dcb50c86ULL,0x8f6927b04832a96cULL,0xd08ee1ba55f2f94cULL,0x6a996f99344b45faULL,0xe133cb8da8aa455dULL,0x5d0721ec758dc1f7ULL, + 0xf3d44e1f9a876441ULL,0x82bc0c14147a818dULL,0x33ddc1170603eddeULL,0x77163f2e0a25f260ULL,0xb54c02caa3bfaa53ULL,0xbd1b2502e2256982ULL,0xc4e1728c1a6f37dcULL,0xfe36c213814c94a5ULL, + 0xf3cae7e9262a3539ULL,0x78a49d1d6670d59eULL,0x37de0f63c1c5e1b9ULL,0x3072c30c69cb7c1cULL,0x1d278a5277c850e6ULL,0x84f15f8f1f6a3de6ULL,0x46a8bb45592ca7adULL,0x1912e3eee4d424b8ULL, + 0xc1ffe2d490e31734ULL,0x91b1f1267fca007cULL,0x5459b1d0ae3f77e8ULL,0x262b051d46425c88ULL,0xcf5c8f765c51e274ULL,0x997481e2304e6146ULL,0x6fc1198bd84046b5ULL,0x1cb0a6bbe7f7a6bdULL, + 0x6ba7a92079e5fb67ULL,0xe1331feb70aa725eULL,0x5080ccf57df5d837ULL,0xe4cae01d7ff72e21ULL,0xd9243ee60412a77dULL,0x06ff7cacdf449025ULL,0xbe75f7cd23ef5a31ULL,0xbc9578220ddef7a8ULL, + 0xc4737ad11b7f30b0ULL,0x525ab2c63629dcf9ULL,0x963f4cc1186ae160ULL,0x8507671373e6b6e0ULL,0xd9be3180f6998bcfULL,0x93d91da3b1c8d8d8ULL,0xf902ce661b8c0054ULL,0x47e7924d74a8a768ULL, + 0xdc988086365e668bULL,0xada8dcdaaabda5fbULL,0xbc146b4c255f1fbeULL,0x9cfcde29cf34cfc3ULL,0xacbb453e7e85d1e4ULL,0x9ca09679f92358b5ULL,0x15fc2d96240823ffULL,0x8d65adf70c11d11eULL, + 0x323cfd177e19a46fULL,0x0948a7a786161156ULL,0x50d06b977e7d3363ULL,0x41c47ec1a702579dULL,0x9455998e59e9260bULL,0xc865e44446c24260ULL,0x393021ec13bc3744ULL,0x4981994ecd92a14aULL, + 0x8cf7230cb0ce1c55ULL,0x5b534d050bbfb607ULL,0xee1ef1130e16363bULL,0x27e0aa7ab4999e82ULL,0xce1dac2d79362c41ULL,0x67920c9091bb6cb0ULL,0x1e648d632223df24ULL,0x0f7d9eefe32e8f28ULL, + 0x9766f264d66f51c0ULL,0x644317ab5d9ccea5ULL,0xa39b37bcd721e232ULL,0x98bdde0bb9daf737ULL,0xc2ecc758165166bdULL,0x0951a285a2802108ULL,0xe39fbf24997aa66fULL,0x1a2f6862db62da27ULL, + 0x775557f10296f4fdULL,0x1dca76a3ea51b436ULL,0xf3e98f60fb950805ULL,0x31ff32ea831cf7f1ULL,0x643e7bf18d2c714bULL,0x64b5c3392e9d2acaULL,0xa9fd9ccc6adc2d23ULL,0xfc2397eccc721b9bULL, + 0xab651fc764465367ULL,0xe43870152b098f57ULL,0x0a91d519d382376fULL,0xb5afdb0a53cad929ULL,0x457e1875138d5523ULL,0xa92becae1aecacfbULL,0x0762f6e811484f49ULL,0x114b5c86dda16c2bULL, + 0x6943f39afa833834ULL,0x22951722a6328562ULL,0x81d63dd54170fc10ULL,0x9f5fa58faecc2e6dULL,0xb66c8725e77d9a3bULL,0x11235cea6384ebe0ULL,0x06a8c1185845e24aULL,0x0137b286ebd093b1ULL, + 0xdb567d6ac42bd6d2ULL,0x6df86468bb1f96aeULL,0x0efe5b1a4843b28eULL,0x961bbb056379b240ULL,0xb6caf5f070a6a26bULL,0x70686c0d328e6e39ULL,0x80da06cf895fc8d3ULL,0x804d8810b363fdc9ULL, + 0x63b99ce74462007dULL,0xb8ab48a54cb5f5b7ULL,0x9ec673d2f55edde7ULL,0xd1567f748cfaefdaULL,0x46381b6b0887bcecULL,0x694497cee178f3c2ULL,0x5e6525e31e6266cbULL,0x5931de26697d6413ULL, + 0x14e49da11f17a34cULL,0x5420ab39235a1456ULL,0xb76372412f50363bULL,0x7b15d623c3fabb6eULL,0xa0ef40b1e274e49cULL,0x5cf5074496b1860aULL,0xd6583fbf66afe5a4ULL,0x44240510f47e3e9aULL, + 0x142b55021a93507aULL,0xb4cd11878d3c06cfULL,0xdf70e76a91ec3f40ULL,0x484e81ad4e7553c2ULL,0x830f87b5272e9d6eULL,0xea1c93e5c6ff514aULL,0x67cc2adcc4192a8eULL,0xc77e27e242f4535aULL, + 0xb5358b1e48ac2840ULL,0x18311294ecba9477ULL,0xda58f990a6946b43ULL,0x3098baf99ab41819ULL,0x66c4c1584198da52ULL,0xab4fc17c146bfd1bULL,0x2f0a4c3cbf36a908ULL,0x2ae9e34b58cf7838ULL, + 0x45eb40ec0ccced58ULL,0x25cd4b9c0da44f98ULL,0x43e06458871812c6ULL,0x99f80d5516cef651ULL,0x571340c9ce6dc153ULL,0x138d5117d8665521ULL,0xacdb45bc4e07014dULL,0x2f34bb3884b60b91ULL, + 0x417499e84a34f239ULL,0x15fdb83cb90402d5ULL,0xb75f46bf433aa832ULL,0xb61e15af63215db1ULL,0xaabe59d4a127f89aULL,0x5d541e0c07e816daULL,0xaaba0659a618b692ULL,0x5532773317266026ULL, + 0x8cda9cf2d0c05199ULL,0x502fbc22fae78454ULL,0xc0bda9dff572a182ULL,0x5f9b71b86158b372ULL,0xe0f33a592b82dd07ULL,0x763027359523032eULL,0x7fe1a721c4505a32ULL,0x7b6e3e82f796409fULL, + 0x023c155d3f6effc7ULL,0x1fbd69ff9c90f0c7ULL,0xe5d7da8abeec2c5dULL,0x8813872bd7e86273ULL,0x9f3bc2c655f5e228ULL,0x11482869b0923b41ULL,0x65d75c741aa307caULL,0xda92c2577f24eee5ULL, + 0x26357732edc665d1ULL,0x9fb5b731a939ef1bULL,0x7db720fb94968089ULL,0x36f75f2c33138c52ULL,0xf8b793ec48d3cb97ULL,0x8dff1d456d261726ULL,0xfb65791b885c4ffbULL,0xf7c79e2df1a3a870ULL, + 0x08dd1028754c92e1ULL,0xca90b57acf0fef34ULL,0x1a9b84ac8af55919ULL,0xaa95e0e1ed93686bULL,0x46737315167021a4ULL,0x6cb6a0da20d5ff98ULL,0xecc4801a1092e706ULL,0xedcab23a3c5e61a6ULL, + 0xb4c66356ea37ba9eULL,0x1adc84150afff55dULL,0xf0080ef9596cc862ULL,0x756c85b86d647ab6ULL,0xc9db94aa1db9c215ULL,0x2dd36db12013b1a5ULL,0xde6ac61c4286c903ULL,0x3fd32f88c76cf884ULL, + 0x7f1290fca06d107eULL,0x697261fdb7661137ULL,0x1bb5be4e947b4b38ULL,0xb49826b63bb79130ULL,0x019ddfe85ba8bffbULL,0xb1af79007e3fa8e4ULL,0x72e1bdf201bcfe7fULL,0x2ed3ca8fd1169aeaULL, + 0x3befda8cd745fff1ULL,0x70b9e9b669b9924eULL,0xa5df48cfd1511381ULL,0x84f93fe2d06bc535ULL,0xaa42c5a9b279a6c3ULL,0x651da6c4d8f96132ULL,0xb0368c8b01b6aea5ULL,0x64e44c47ac7862a2ULL, + 0xe17a9947d9de99a8ULL,0xc2e61b2dc93477bdULL,0x57f684d41d19e287ULL,0x843c2122fe358135ULL,0xe2d3e2e904f7e8abULL,0xbf93ffe9b5f27aeeULL,0x29830d1d7b1858c4ULL,0xa8f449648106adbfULL, + 0xe3417bc035d0b34aULL,0x440b386b8327c0a7ULL,0x8fb7262dac0362d1ULL,0x2c41114ce0cdf943ULL,0x2ba5cef1ad95a0b1ULL,0xc09b37a867d54362ULL,0x26d6cdd201e486c9ULL,0x20477abf42ff9297ULL, + 0xa004dcb3292a9287ULL,0xddc15cf677b092c7ULL,0x083a8464806c0605ULL,0x4a68df703db997b0ULL,0x9c134e4505bf7dd0ULL,0xa4e63d398ccf7f8cULL,0xa6e6517f41b5f8afULL,0xaa8b9342ad7bc1ccULL, + 0xc41764717af715d2ULL,0xe2f7f594d0134a96ULL,0x2c1873efa41ec956ULL,0xe4e7b4f677821304ULL,0xe5c8ff9788d5374aULL,0x2b915e6380823d5bULL,0xea6bc755b2ee8fe2ULL,0x6657624ce7112651ULL, + 0xd6f800e07442f1d5ULL,0x475607d166e0e3abULL,0x82807f16b7c64047ULL,0x8858e1e3a749883dULL,0x5859120b8231ee10ULL,0x1b80e7eb638a1eceULL,0xcb72525ac6aa73a4ULL,0xa7cdea3d844423acULL, + 0x57477b11e51732d2ULL,0xdfd6eb282538fc0eULL,0x5c43b0cc3b39eec5ULL,0x6af12778cb36cc57ULL,0x70b0852d06c425aeULL,0x6df92f8c5c221b9bULL,0x6c8d4f9ece826d9cULL,0xf59aba7bb49359c3ULL, + 0xd2eb2cf152bfda05ULL,0xe0e4c4e96197b98cULL,0x1d35076cf8a1726fULL,0x6c06085b2db11e3dULL,0x15c0c4d74463ba14ULL,0x9d292f830030238cULL,0x1311ee8b3727536dULL,0xfeea86efbeaedc1eULL, + 0xa7f96054afa05dd8ULL,0x26dfcf21fcaf119eULL,0xe20ef2e30564bb59ULL,0xef4dca5061cb02b8ULL,0xcda7838a65d30672ULL,0x8b08d534fd657e86ULL,0x4c5b439546d595c8ULL,0x39b58725425cb836ULL, + 0xfda853931a62cc26ULL,0x23c69b9650c0e052ULL,0xa227df15bfc633f3ULL,0x2ac788481bae7d48ULL,0x487878f9187d073dULL,0x6c2be919967f807dULL,0x765861d8336e6d8fULL,0x88b8974cce528a43ULL, + 0xc37e2c2e421d3aa4ULL,0xf926407ce84fa840ULL,0x18abc03d1454e41cULL,0x26605ecd3f7af644ULL,0x242341a6d6a5eabfULL,0x1edb84f4216b668eULL,0xd836edb804010102ULL,0x5b337ce7945e1d8cULL, + 0x666ba2dccc78cf66ULL,0xb30181746fdbff77ULL,0x8d4dd0db168d4668ULL,0x259455d01dab3a2aULL,0xf58564c5cde3acecULL,0x7714192513adb276ULL,0x527d725d8a303f65ULL,0x55deb6c9e6f38f7bULL, + 0x864d05d73272d838ULL,0xe22924f9fa6295c5ULL,0x8189593f6c2fda32ULL,0x330d7189b184b544ULL,0x79efa62cbde1f714ULL,0x35771c94e5cb1a63ULL,0x2f4826b8641c8332ULL,0x00a894fbc8cee854ULL, + 0xdcdacd0a1058a318ULL,0x369cf3f578053a9aULL,0xc6c3de5031c68de2ULL,0x4653a5763c4b6d9fULL,0x1688dd5aaa4e5c97ULL,0x5be80aa1b7ab3c74ULL,0x70cefe7cbc65c283ULL,0x57f95f1306867091ULL, + 0xc240b6de34eaacdaULL,0xd9e116e82ba0f1deULL,0xcbe45ec779438e55ULL,0x91787c9d96f752d7ULL,0x897f532bf129ac2fULL,0xd307b7c85a36e22cULL,0x91940675749fb8f3ULL,0xd14f95d0157fdb28ULL, + 0x1625360416df4285ULL,0xb0c9babbd0c56ae2ULL,0x73032b19cfc5cfc3ULL,0xe497e5c309752056ULL,0x12096bb4164bda96ULL,0x1ee42419a0b74da1ULL,0x8fc36243403826baULL,0x0c8f0069dc09e660ULL, + 0xc44b74a15b0ec6f5ULL,0x47989fe45289b2b8ULL,0x745f848458d6fc73ULL,0xec362a6ff61c70abULL,0x070c98a7b3a8ad41ULL,0x73a20fc07b63db51ULL,0xed2c2173f44c35f4ULL,0x8a56149d9acc9dcaULL, + 0xa395c36f35d33ae7ULL,0x200ea12350bb5a94ULL,0x20c789bd0bafe84bULL,0x243ef52d0919276aULL,0x3934c577e23ae233ULL,0xb93807afa460d1ecULL,0xb72a53b1f8fa76a4ULL,0xd8914cb0c3ca4491ULL, + 0x4c076b86d23ddc82ULL,0x03fd344c7e0143f0ULL,0xa95362ff317af2c5ULL,0x0add3db7e18b7a4fULL,0x9c673e3f8260e01bULL,0xfbeb49e554a1cc91ULL,0x91351bf292f2e433ULL,0xc755e7ec851141ebULL, + 0x2bf5db47f23206d5ULL,0x2f6d34201d260152ULL,0x17b876533f8ff89aULL,0x5157c30c378fa458ULL,0x7517c5c52d4fb936ULL,0xef22f7ace6518cdcULL,0xdeb483e6bf847a64ULL,0xf508455892e0fa89ULL, + 0xab9659d8df7304d4ULL,0xb71bcf1bff210e8eULL,0xa9a2438bd73fbd60ULL,0x4595cd1f5d11b4deULL,0x9c0d329a4835859dULL,0x4a0f0d2d7dbb6e56ULL,0xc6038e5edf928a4eULL,0xc94296218f5ad154ULL, + 0x08a33840a70c6ec4ULL,0x9e8819f7e0311195ULL,0xed209d96708ab202ULL,0x10d7c4e7ce943a27ULL,0x372fb317a29b49a1ULL,0x57a67fb346627d1fULL,0xf912561e7cdf39ecULL,0xfa3ce6f26f7c8f17ULL, + 0x91213462f23f2d92ULL,0x6cab71bd60b94078ULL,0x6bdd0a63176cde20ULL,0x54c9b20cee4d54bcULL,0x3cd2d8aa9f2ac02fULL,0x03f8e617206eedb0ULL,0xc7f68e1693086434ULL,0x831469c592dd3db9ULL, + 0xfe7d7465653f3c5fULL,0x283dd45ef040feb1ULL,0x91fe599bf3b7edfeULL,0x5ff039ad80379311ULL,0xbf76995b4e96fa49ULL,0x2640b6b2a3e25094ULL,0x8b096341c1c83f74ULL,0xd2bec884fa560ac3ULL, + 0x8521df248f981354ULL,0x587e23ec3588a259ULL,0xcbedf281d7a0992cULL,0x06930a5538961407ULL,0x09320debbe5bbe21ULL,0xa7ffa5b52491817fULL,0xe6c8b4d909065160ULL,0xac4f3992fff6d2a9ULL, + 0x2e4eb2a3cc53da66ULL,0x04708a71f17a9b4eULL,0xdbfdc7b20de05b2cULL,0x4cdc9aee4907a201ULL,0xe5cc8dfc6c475566ULL,0x2b83cbfb47be1691ULL,0x695833a74c05c3fdULL,0xee938a7243b0deb0ULL, + 0x7aa7a1583ae9c1bdULL,0xe0af6d98e37ce240ULL,0xe54342d928ab38b4ULL,0xe8b750070a1c98caULL,0xefce86afe02358f2ULL,0x31b8b856ea921228ULL,0x052a19120a1c67fcULL,0xb4069ea4e3aead59ULL, + 0x3b826205c6ffcfe6ULL,0x2eb31256aa39418aULL,0xc4b4a9e1c18521fbULL,0x48614dd8db610615ULL,0x04c362bbeea7475fULL,0x916ab969a8ead162ULL,0x6a7975bc0310a876ULL,0xea4e7d11ecc77ec4ULL, + 0x3232d6e27fa03cb3ULL,0xdb938e5b0fdd7d88ULL,0x04c1d2cd2ccbfc5dULL,0xd2f45c12af3a580fULL,0x592620b57883e614ULL,0x5fd27e68be7c5f26ULL,0x139e45a91567e1e3ULL,0x2cc71d2d44d8aaafULL, + 0x6ef493c706ba8e6fULL,0x05c07bf8123deffdULL,0xcebf9b4d9c2b5a4fULL,0xf958147e2d038e9cULL,0x5af5cee03e5561c4ULL,0x8d0b5a7a794a6afcULL,0x9de22df13772168bULL,0xe5d249c5d84097fdULL, + 0x4a9090cde36d0757ULL,0xf722d7b1d9a29382ULL,0xfb7fb04c04b48ddfULL,0x628ad2a7ebe16f43ULL,0xcd3fbfb520226040ULL,0x6c34ecb15104b6c4ULL,0x30c0754ec903c188ULL,0xec336b082d23cab0ULL, + 0x74c70b4295c69248ULL,0x8813259dfed90303ULL,0xa3e330684b8cc87aULL,0xe689371c111a7a95ULL,0x52bfbbf7fbbbc20bULL,0x73a8543d65f9a6e2ULL,0x67bb2fdd7e413e6dULL,0xa066b3efc5cf2032ULL, + 0x473d62a21e206ee5ULL,0xf1e274808c49a633ULL,0x87ab956ce9f6b2c3ULL,0x61830b4862b606eaULL,0x67cd6846e78e815fULL,0xfe40139f4c02082aULL,0x52bbbfcb952ec365ULL,0x74c116426b9836abULL, + 0xcf61b89c44f48971ULL,0xe2d700f76d660683ULL,0x72ef285cd1d431bdULL,0x0593b24e9bdebf4aULL,0x4084bc5b0561f8a1ULL,0x84ce74d0aa16f256ULL,0x9cbc79d309f6d277ULL,0xa94fe2fe4139bdfeULL, + 0x9f51439e558df019ULL,0x230da4baac712b27ULL,0x518919e355185a24ULL,0x4dcefcdd84b78f50ULL,0xa7d90fb2a47d4c5aULL,0x55ac9abfb30e009eULL,0xfd2fc35974eed273ULL,0xb72d824cdbea8fafULL, + 0x549db2b5ef7d9289ULL,0x2480d4a8197f015aULL,0x61d5590bc40493b6ULL,0x3a55b52e6f780331ULL,0x40eb8115309eadb0ULL,0xdea7de5a92e5c625ULL,0x64d631f0cc6a3d5aULL,0x9d5e9d7c93e8dd61ULL, + 0x196860411e84e0e5ULL,0xa5db84d3aea34c93ULL,0xf9d5bb197073a732ULL,0xb8d2fe566bcfd7c0ULL,0x45775f36f3eb82faULL,0x8cb20cccfdff8b58ULL,0x1659b65f8374c110ULL,0xb8b4a422330c789aULL, + 0xc925ff87aedbae9fULL,0x7daf0eb936880a54ULL,0x9284ddf59c4d0e71ULL,0x1581cf93316f8cf5ULL,0x3eeca8873ac1f452ULL,0xb417fce9fb6aeffeULL,0xa5918046eefb8dc3ULL,0x73d318ac02209400ULL, + 0xc4f4cda3af2ebc2fULL,0xa0af843dcb4efe24ULL,0x53b857c19ccd10b1ULL,0xddc9d1eb914d3e04ULL,0x7bdec8bb62771debULL,0x829277aa91c5aa81ULL,0x7af18dd6832391aeULL,0x1740f316c71a84caULL, + 0xdd4a12d8e12b31f8ULL,0x577e29bc177736e6ULL,0x2353722ba88935e8ULL,0xca1d3729015f286dULL,0x86c7b6a239a3e035ULL,0x6e5250bfd3b03a9fULL,0x79d98930fd0d536eULL,0x8c4cbbabfa0c3832ULL, + 0x2d500910cab91f1eULL,0xbedd9e444d1cd216ULL,0xd634b74fedd02252ULL,0xbd60f8e11258617aULL,0xd8c7537b9e05614aULL,0xfd26c766e7af5fc5ULL,0x0660b581582bd926ULL,0x87019244acf07fc8ULL, + 0xafa26ccfaf64ecb6ULL,0xe6054f974bd72775ULL,0xbbcbab5b140f695aULL,0xbc71b4a4e348efdeULL,0xfc2e52becc96d963ULL,0x150abf5f5e5d9018ULL,0xbd182fa604568771ULL,0x35b4c06170339f83ULL, + 0x8928e99aeeaf8c49ULL,0xee7aa73d6e24d728ULL,0x4c5007c2e72b156cULL,0x5fcf57c5ed408a1dULL,0x9f719e39b6057604ULL,0x7d343c01c2868bbfULL,0x2cca254b7e103e2dULL,0xe6eb38a9f131bea2ULL, + 0xc624a04e5f40ff52ULL,0x3611d7cffcd2914aULL,0xb7e8b42b1fd3bfd6ULL,0x6cde40fdfa85063aULL,0x7811c449178b7e5bULL,0x4cb609312972cc13ULL,0xf579125e33b46135ULL,0xca102ef788a4e56eULL, + 0x0ba4e3520a981b0dULL,0x1c354cb3bd1a41a4ULL,0x1aabaa3adf9fab9cULL,0x0701a7d153c418d5ULL,0xdd1a7cefdcf2b921ULL,0x6ceef0b3bcf48061ULL,0x1083b598de25cce6ULL,0x890a54c7e90a5e34ULL, + 0xc535956640ac1807ULL,0xd4c1e56684da0b1eULL,0xc4b33f97e0b82421ULL,0xd0bd23177b41be00ULL,0x53a4b42e147b72e1ULL,0xf8d39f5ff777104cULL,0x36e64e64d3fb530dULL,0xa7a6ba6756074fddULL, + 0x405718db4f6d01b1ULL,0xe73c6bc28f11e8a0ULL,0xac11bb8ca0591a3bULL,0x12d09a5a0acc4531ULL,0xcbf174eee7de13f4ULL,0x177e2be6044fd682ULL,0x65f574cb1c48af70ULL,0xce5966929961cb7cULL, + 0x989fe84ebc6fab9fULL,0xe70ce6b1c80f6474ULL,0xbe9ff3053b02a1bcULL,0x12ef486699c0afd3ULL,0x22d957f9e3411a26ULL,0x0b41d8817b485b98ULL,0x820e56d04ae20119ULL,0x81e3d01f328528e0ULL, + 0xc59eed6c048752a1ULL,0x41f2702ea01341b4ULL,0x6e35903b9dc6b092ULL,0x4291aba81f5b5b23ULL,0x8173aa70a653d61dULL,0xd1b648d44f2eb51eULL,0x31b7ce065ab93f8fULL,0xa55408ee99e2f4feULL, + 0x1fa4ed0985e34160ULL,0xa26d7dc37a03cde2ULL,0x1dac0848fa84df2dULL,0x5e9a28d61b697108ULL,0xa88004d914ea0ea1ULL,0xa8d154283ffe5520ULL,0x4f422dae639b139cULL,0xeedccc0dd4b5a861ULL, + 0xb33e624f8be762b4ULL,0x2a9ee4d1058e3413ULL,0x968e636967d805faULL,0x9848949b7db8bfd7ULL,0x5308d7e5d23a8417ULL,0x892f3b1df3e29da5ULL,0xc95c139e3dee471fULL,0x8631594dd757e089ULL, + 0x1083e2ea1f095615ULL,0x0a28ad7714e68c33ULL,0x6bfc02523d8818beULL,0xb585113af35850cdULL,0x7d935f0b30df8aa1ULL,0xaddda07c4ab7e3acULL,0x92c34299552f00cbULL,0xc33ed1de2909df6cULL, + 0x2dc40d483e07113cULL,0x6e4a5d397d8b63aeULL,0x5582a94b79684c2bULL,0x932b33d4622da26cULL,0xf534f6510dbbf08dULL,0x211d07c964c23a52ULL,0x0eeece0fee5bdc9bULL,0xdf178168f7015558ULL, + 0xabe7905a83cdd60eULL,0x50602fb5a1170184ULL,0x689886cdb023642aULL,0xd568d090a6e1fb00ULL,0x5b1922c70259217fULL,0x93831cd9c43141e4ULL,0xdfca35870c95f86eULL,0xdec2057a568ae828ULL, + 0x568f8925913cc16dULL,0x18bc5b6de1a26f5aULL,0xdfa413bef5f499aeULL,0xf8835decc3f0ae84ULL,0xb6e60bd865a40ab0ULL,0x65596439194b377eULL,0xbcd8562592084a69ULL,0x5ce433b94f23ede0ULL, + 0x860d523d42e06189ULL,0xbf0779414e3aff13ULL,0x0b616dcac1b20650ULL,0xe66dd6d12131300dULL,0xd4a0fd67ff99abdeULL,0xc9903550c7aac50dULL,0x022ecf8b7c46b2d7ULL,0x3333b1e83abf92afULL, + 0xc0da65e784d6365dULL,0xbcb7443f8f759fb8ULL,0x35c712b17ae81930ULL,0x80428dff4c6e08abULL,0xf19dafefa4faf843ULL,0xced8538dffa9855fULL,0x20ac409cbe3ac7ceULL,0x358c1fb6882da71eULL, + 0xefecdef7be42a582ULL,0xd3fc608065046be6ULL,0xc9af13c809e8dba9ULL,0x1e6c9847641491ffULL,0x3b574925d30c31f7ULL,0xb7eb72baac2a2122ULL,0x776a0dacef0859e7ULL,0x06fec31421900942ULL, + 0x324794b07e50122bULL,0xdd744f8b4af07ca5ULL,0x30a12f08d63fc97bULL,0x39650f1a76626d9dULL,0x101b47f71fa38477ULL,0x3d815f19d4dc124fULL,0x1569ae95b26eb58aULL,0xc3cde18895fb1887ULL, + 0x7ec62fbbf4737f21ULL,0xd8dba5ab6209f5acULL,0x24b5d7a9a5f9adbeULL,0x707d28f7a61dc768ULL,0x7711460bcaa999eaULL,0xba7b174d1c92e4ccULL,0x3c4bab6618d4bf2dULL,0xb8f0c980eb8bd279ULL, + 0x9d658932790691bfULL,0xed61058906b736aeULL,0x712c2f04c0d63b6eULL,0x5cf06fd5c63d488fULL,0x97363facd9588e41ULL,0x1f9bf7622b93257eULL,0xa9d1ffc4667acaceULL,0x1cf4a1aa0a061ecfULL, + 0x28d675b2c0519a23ULL,0x9ebf94fe4f6952e3ULL,0xf28bb767a2294a8aULL,0x85512b4dfe0af3f5ULL,0x18958ba899b16a0dULL,0x95c2430cba7548a7ULL,0xb30d1b10a16be615ULL,0xe3ebbb9785bfb74cULL, + 0x07b53f5eb2e63645ULL,0xbe57e54784c84232ULL,0xd779c2167214d5cfULL,0x617969cd029a3acaULL,0xd17668cd8a7017a0ULL,0x77b4d19abe9b7ee8ULL,0x58fd0e939c161776ULL,0xa8c4f4efd5968a72ULL, + 0x81eeb865d2fdca23ULL,0x5a15ee08cc8ef895ULL,0x768fa10a01905614ULL,0xeff5b8ef880ee19bULL,0xf0c0cabbcb1c8a0eULL,0x2e1ee9cdb8c838f9ULL,0x0587d8b88a4a14c0ULL,0xf6f278962ff698e5ULL, + 0x519d34b3bf44da80ULL,0x283834f95ab32e66ULL,0x6e6087976278a000ULL,0x1e62960e627312f6ULL,0x9b87b27be6901c55ULL,0x80e7853824fdbc1fULL,0xbbbc09512facc27dULL,0x06394239ac143b5aULL, + 0x9c4b646e9e2fce99ULL,0x68a210811e80857fULL,0x06d54e443643b52aULL,0xde8d6d630d8eb843ULL,0x7032156342146a0aULL,0x8ba826f25eaa3622ULL,0x227a58bd86138787ULL,0x43b6c03c10281d37ULL, + 0x02b37a952f41deffULL,0x0e44a59ae63b89b7ULL,0x673257dc143ff951ULL,0x19c02205d752baf4ULL,0x46c23069c4b7d692ULL,0x2e6392c3fd1502acULL,0x6057b1a21b220846ULL,0xe51ff9460c1b5b63ULL, + 0x6e85cb51566c5c43ULL,0xcff9c9193597f046ULL,0x9354e90c4994d94aULL,0xe0a393322147927dULL,0x8427fac10dc1eb2bULL,0x88cfd8c22ff319faULL,0xe2d4e68401965274ULL,0xfa2e067d67aaa746ULL, + 0xb6d92a7f3e5f9f11ULL,0x9afe153ad6cb3b8eULL,0x4d1a6dd7ddf800bdULL,0xf6c13cc0caf17e19ULL,0x15f6c58e325fc3eeULL,0x71095400a31dc3b2ULL,0x168e7c07afa3d3e7ULL,0x3f8417a194c7ae2dULL, + 0xec234772813b230dULL,0x634d0f5f17344427ULL,0x11548ab1d77fc56aULL,0x7fab1750ce06af77ULL,0xb62c10a74f7c4f83ULL,0xa7d2edc4220a67d9ULL,0x1c404170921209a0ULL,0x0b9815a0face59f0ULL, + 0x2842589b319540c3ULL,0x18490f59a283d6f8ULL,0xa2731f84daae9fcbULL,0x3db6d960c3683ba0ULL,0xc85c63bb14611069ULL,0xb19436af0788bf05ULL,0x905459df347460d2ULL,0x73f6e094e11a7db1ULL, + 0xdc7f938eb6357f37ULL,0xc5d00f792bd8aa62ULL,0xc878dcb92ca979fcULL,0x37e83ed9eb023a99ULL,0x6b23e2731560bf3dULL,0x1086e4591d0fae61ULL,0x782483169a9414bdULL,0x1b956bc0f0ea9ea1ULL, + 0x7b85bb91c31b9c38ULL,0x0c5aa90b48ef57b5ULL,0xdedeb169af3bab6fULL,0xe610ad732d373685ULL,0xf13870df02ba8e15ULL,0x0337edb68ca7f771ULL,0xe4acf747b62c036cULL,0xd921d576b6b94e81ULL, + 0xdbc864392c422f7aULL,0xfb635362ed348898ULL,0x83084668c45bfcd1ULL,0xc357c9e32b315e11ULL,0xb173b5405b2e5b8cULL,0x7e946931e102b9a4ULL,0x17c890eb7b0fb199ULL,0xec225a83d61b662bULL, + 0xf306a3c8ee3c76cbULL,0x3cf11623d32a1f6eULL,0xe6d5ab646863e956ULL,0x3b8a4cbe5c005c26ULL,0xdcd529a59ce6bb27ULL,0xc4afaa5204d4b16fULL,0xb0624a267923798dULL,0x85e56df66b307fabULL, + 0x0281893c2bf29698ULL,0x91fc19a4d7ce7603ULL,0x75a5dca3ad9a558fULL,0x40ceb3fa4d50bf77ULL,0x1baf6060bc9ba369ULL,0x927e1037597888c2ULL,0xd936bf1986a34c07ULL,0xd4cf10c1c34ae980ULL, + 0x3a3e5334859dd614ULL,0x9c475b5b18d0c8eeULL,0x63080d1f07cd51d5ULL,0xc9c0d0a6b88b4326ULL,0x1ac98691c234296fULL,0x2a0a83a494887fb6ULL,0x565114270cea9cf2ULL,0x5230a6e8a24802f5ULL, + 0xf7a2bf0f72e3d5c1ULL,0x377174464f21439eULL,0xfedcbf259ce30334ULL,0xe0030a787ce202f9ULL,0x6f2d9ebf1202e9caULL,0xe79dde6c75e6e591ULL,0xf52072aff1dac4f8ULL,0x6c8d087ebb9b404dULL, + 0xad0fc73dbce913afULL,0x909e587b458a07cbULL,0x1300da84d4f00c8aULL,0x425cd048b54466acULL,0xb59cb9be90e9d8bfULL,0x991616db3e431b0eULL,0xd3aa117a531aecffULL,0x91af92d359f4dc3bULL, + 0x9b1ec292e93fda29ULL,0x76bb6c17e97d91bcULL,0x7509d95faface1e6ULL,0x3653fe47be855ae3ULL,0x73180b280f680e75ULL,0x75eefd1beeb6c26cULL,0xa4cdf29fb66d4236ULL,0x2d70a9976b5821d8ULL, + 0x7a3ee20720445c36ULL,0x71d1ac8259877174ULL,0x0fc539f7949f73e9ULL,0xd05cf3d7982e3081ULL,0x8758e20b7b1c7129ULL,0xffadcc20569e61f2ULL,0xb05d3a2f59544c2dULL,0xbe16f5c19fff5e53ULL, + 0x73cf65b8aad58135ULL,0x622c2119037aa5beULL,0x79373b3f646fd6a0ULL,0x0e029db50d3978cfULL,0x8bdfc43794fba037ULL,0xaefbd687620797a6ULL,0x3fa5382bbd30d38eULL,0x7627cfbf585d7464ULL, + 0xb2330fef4e4ca463ULL,0xbcef72873566cc63ULL,0xd161d2cacf780900ULL,0x135dc5395b54827dULL,0x638f052e27bf1bc6ULL,0x10a224f007dfa06cULL,0xe973586d6d3321daULL,0x8b0c573826152c8fULL, + 0x9910ba6b23a5d896ULL,0x1fe19e357fe4364eULL,0x6e1da8c39a33c677ULL,0x15b4488b29fd9fd0ULL,0x1f4392541a1f22bfULL,0x920a8a70ab8163e8ULL,0x3fd1b24907e5658eULL,0xf2c4f79cb6ec839bULL, + 0x8b5c619c76497ee8ULL,0x5d2b0ac6c717370eULL,0x98204cb64fcf68e1ULL,0x0bdec21162bc6792ULL,0x6973ccefa63b1011ULL,0xf9e3fa97e0de1ac5ULL,0x5efb693e3d0e0c8bULL,0x037248e9d2d4fcb4ULL, + 0xd3694e2ab20364e4ULL,0x62699718e770b20dULL,0x6183291b6ed77d1cULL,0x69aada7f6d6180a5ULL,0x51f9054bf185509bULL,0xfd7e845678701077ULL,0xd6a2308a7fb96d8dULL,0xd53e48d228dc87eeULL, + 0x80802dc91ec34f9eULL,0xd8772d3533810603ULL,0x3f06d66c530cb4f3ULL,0x7be5ed0dc475c129ULL,0xcb9e3c1931e82b10ULL,0xc63d2857c9ff6b4cULL,0xb92118c692a1b45eULL,0x0aec44147285bbcaULL, + 0x37071afbe8316c45ULL,0x982be4fd46700b8bULL,0x8d5d177c64ff8578ULL,0x5ec40582c9a82fa7ULL,0x5518e37bcfa86678ULL,0x24e809f49f031284ULL,0x312f39604bbbb74cULL,0xad4b4f6fc0c14de6ULL, + 0xa3d6f4868eb6e843ULL,0x6415834bac4ab3abULL,0x028a81514f3cf2dcULL,0xf3b4962f5f3e6c3eULL,0x9119ae90987dd2f2ULL,0x437a6d8ae10bce55ULL,0xc31cdd6b9b149ed6ULL,0x1b77791d06871332ULL, + 0x9c34b650e16e05e9ULL,0x965a774094e74640ULL,0xa3fd22fbcea3f029ULL,0x1eb6a9688f95277cULL,0x2520a63d7bad84f6ULL,0xad917201f58f2feeULL,0xea92c1669b840d48ULL,0x12109c4aacef5cbdULL, + 0xfc189ae71e29a3efULL,0xcbe906f04c93302eULL,0xd0107914ceaae10eULL,0xb7a23f34b68e19f8ULL,0xe9d875c2efd2119dULL,0x03198c6efcadc9c8ULL,0x65591bf64da17113ULL,0x3cf0bbf83d443038ULL, + 0xab293027aad991c1ULL,0x598d0bf8849be4b7ULL,0x8c94a21ab972da90ULL,0xada4cfdd7ecfa840ULL,0x93d4b9c0fbcec63aULL,0x7ca617a203219a34ULL,0x900424eb6a652a55ULL,0xaf9346e9eb8562e0ULL, + 0xc3e04d7902381461ULL,0xb1643ab5911bc478ULL,0xc92becfa390b3ef2ULL,0x54476778acd2f1b6ULL,0x8daa0c4d66bf3aafULL,0x2bc1287b2c21c65aULL,0xee182910b5a13ac3ULL,0xbb04730090b0790aULL, + 0x83766947d17d4e0bULL,0xc5772beefdc3a47bULL,0x765a50db1a6fd0ffULL,0x17f904ba45b0995eULL,0xcee643832883487eULL,0xf56db7f3c270aaedULL,0x6738d94f46cb1fd9ULL,0xc8fa426a142fd4d5ULL, + 0xae485bb72b724759ULL,0x945353e1b2d4c63aULL,0x82159d07de7d6f2cULL,0x389caef34ec5b109ULL,0x4a8ebb53db65ef14ULL,0x2dc2cb7edd99de43ULL,0x816fa3ed83f2405fULL,0x73429bb9c14208a3ULL, + 0x08ed8febd56daf06ULL,0x8d98277b4a837f69ULL,0x9947c636a9b6e05aULL,0x58c8a77ac0d58abdULL,0xf45496a45f121e4fULL,0x16cd67c71076d3d3ULL,0xecbd1958e3fb0c5dULL,0xfbe185ec38e1eb47ULL, + 0xb7ef9760ff79d2eeULL,0xdd4d06aff39e7832ULL,0xfd025001b905b499ULL,0x98fe1c61f5b61d31ULL,0xa9f83980c5f12805ULL,0x376e3b783009cd9aULL,0xa322b09f514eb16dULL,0x08e213122c3832dfULL, + 0xaea68626dc4ad4f4ULL,0x5dc516824ddbc0b6ULL,0xa76697bd602e9065ULL,0xbeeb3ea58c37888eULL,0x1ec4a2f214569113ULL,0xe48b820ca35f4484ULL,0x9fb560949ae44df2ULL,0x6ca1346292cc09fdULL, + 0xb618d590b01e6e27ULL,0x047e2ccde180b2dcULL,0xd1b299b504aea4a9ULL,0x412c9e1e9fa403a4ULL,0x88d28a3679407552ULL,0x49c50136f332b8e3ULL,0x3a1b6fcce668de19ULL,0x178851bc75122b97ULL, + 0x197dd46d95a7b1a2ULL,0x9c4e7ad63c6341fbULL,0x426eca29484c2eceULL,0x9211e489de7f4f8aULL,0x14997f6ec78ef1f4ULL,0x2b2c091006574586ULL,0x17286a6e1c3eede8ULL,0x25f92e470f60e018ULL, + 0xb4e370af3aeac968ULL,0xe4f7fee9c4b63266ULL,0xb4acd4c2e3ac5664ULL,0xf8910bd2ceb38cbfULL,0x1c3ae50cc9c0726eULL,0x15309569d97b40bfULL,0x70884b7ffd5a5a1bULL,0x3890896aef8314cdULL, + 0x090d7d205ffe7b37ULL,0x3b7f3efb1747d2daULL,0xa2cb525fb54fc519ULL,0x6e220932f66a971eULL,0xddc160dfb486d440ULL,0x7fcfec463fe13465ULL,0x83da7e4e76e4c151ULL,0xd6fa48a1d8d302b5ULL, + 0x5ced3c9f82e4c634ULL,0x8efb83143a4464f8ULL,0xe706381b7a1dca25ULL,0x6cd15a3c5a2a412bULL,0x9347a8fdbfcd8fb5ULL,0x31db2eef6e54cd22ULL,0xc4aeb11ef8d8932fULL,0x11e7c1ed344411afULL, + 0xae4065ef12045cf9ULL,0x6fcb2caf9ccce8bdULL,0x1fa0ba4ef2cf6525ULL,0xf683125dcb72c312ULL,0xa01da4eae312410eULL,0x67e286776cd8e830ULL,0xabd9575298fb3f07ULL,0x05f11e11eef649a5ULL, + 0x996884f5903fa271ULL,0xe6da0fd2b9da921eULL,0xa6f2f2695db01e54ULL,0x1ee3e9bd6876214eULL,0xa26e181ce27a9497ULL,0x36d254e48e215e04ULL,0x42f32a6c252cabcaULL,0x9948148780b57614ULL, + 0xea961058fa28a7e0ULL,0xc726cf250bf5ec74ULL,0xe74d55c8db229666ULL,0x0bd9abbfa57f5799ULL,0x7479ef074dfc47b3ULL,0xd9c65fc30c52f91dULL,0x8e0283fe36a8bde2ULL,0xa32a8b5e7d4b7280ULL, + 0xab41b43a43228d83ULL,0x24ae1c304ad63f99ULL,0x8e525f1a46a51229ULL,0x14af860fcd26d2b4ULL,0xd6baef613f714aa1ULL,0xf51865adeb78795eULL,0xd3e21fcee6a9d694ULL,0x82ceb1dd8a37b527ULL, + 0x8605d27d48307682ULL,0x745aaba3e10566daULL,0xe57cae36bff2d7abULL,0x91332ba14b127823ULL,0xcb5c3638f3429f43ULL,0x43a21c4fec462929ULL,0xe9bc95352b18b7bdULL,0x8c2addf3e78cb0c6ULL, + 0x4a665bfd2f9fd51aULL,0x7f2f1fe2481b97f7ULL,0xcad05d69ad36ce50ULL,0x314fc2a4844f4dedULL,0xd5593d8cb55fc5c6ULL,0xe3510ce8bfb1e23dULL,0xf9b7be6937453cceULL,0xd3541b7969fae631ULL, + 0xb013cbcad7154cedULL,0x949b28573f52651aULL,0x03b41f6e9f5e642eULL,0xc3a3462986ed94a4ULL,0xf3c86cd6222b24dcULL,0x7578fe8a028c9c26ULL,0xaac7bfa12edad3b6ULL,0x4112c5d78847940aULL, + 0x99296525eca445dfULL,0xf1af24f22cdfa4c6ULL,0xf5b4eb61eba6d3bcULL,0x4560910c98972cc7ULL,0x54751c32093eaa32ULL,0x018313497d3c67bbULL,0x3bd90ce62d871110ULL,0x75fc863a538baa7eULL, + 0x8fe1f8b64ae3a278ULL,0x160c5306137cdf65ULL,0x22f029e733be0492ULL,0x79a680427a75cd82ULL,0x1d8c094a5b3f3adaULL,0x5d723bbe165c3250ULL,0x08b958ffa5792e22ULL,0x829fa986b11c1eaaULL, + 0x711b8a4176a9f05dULL,0x06ca4e4b9011d488ULL,0x543bc62ba248a65eULL,0x017535ffc9290894ULL,0x840b84ce406851d7ULL,0xafa3acdf90e960b4ULL,0xac3394af7128fd34ULL,0x54eb4d5b2ac0f92cULL, + 0x923ea73e4a14f836ULL,0xc8cc8c57f0946328ULL,0xce3f117fe917a4dfULL,0x6372f933f72ce929ULL,0x75000249c29ba567ULL,0xcbd437e68c829ccaULL,0x6c63aaabdd02ec7aULL,0xe9b2f90c7b42bd17ULL, + 0xdb09e87355dbd4b3ULL,0x1f8799286639bbb1ULL,0xb83e47e51c651962ULL,0xd4ef0fb6c43fb574ULL,0x27d3b9d8f1bfb12aULL,0x6ab877e86e5e8b72ULL,0x8eebdc9d157b9014ULL,0x4c2110053aa5cb64ULL, +}; diff --git a/cbits/p256/p256_ec.c b/cbits/p256/p256_ec.c --- a/cbits/p256/p256_ec.c +++ b/cbits/p256/p256_ec.c @@ -33,8 +33,18 @@ // See http://www.imperialviolet.org/2010/12/04/ecc.html ([1]) for background. #include "p256/p256_gf.h" +#include "crypton_bzero.h" +#ifdef CRYPTON_S2N_BIGNUM +#include "p256/p256_s2n.h" +#include "p256/p256_verify.h" +/* the memcpy below is the two representations being the same thing */ +#if P256_BITSPERDIGIT != 64 || P256_NDIGITS != 4 +#error "CRYPTON_S2N_BIGNUM wants the 64-bit crypton_p256_int" +#endif +#endif + /* Field element operations: */ /* felem_inv calculates |out| = |in|^{-1} @@ -43,69 +53,95 @@ * a^p = a (mod p) * a^{p-1} = 1 (mod p) * a^{p-2} = a^{-1} (mod p) - */ + * + * The exponent is built left to right from the shape of p - 2, which for + * this prime is + * + * ffffffff 00000001 00000000 00000000 00000000 ffffffff ffffffff fffffffd + * \__32 ones__/ \_31 zeros, one 1_/ \______ 96 zeros ______/ \_94 ones, 0, 1_/ + * + * A run of k zeros is k squarings; a run of k ones is k squarings and one + * multiplication by a^(2^k - 1), which is why the powers below are kept. The + * whole chain is 255 squarings, which is the least an exponent of 256 bits + * can be done in, and 13 multiplications. + * + * The chain this replaces built the low 94 ones in a second accumulator and + * multiplied the two at the end, which cost 32 squarings more than the 255. */ static void felem_inv(felem out, const felem in) { - felem ftmp, ftmp2; - /* each e_I will hold |in|^{2^I - 1} */ - felem e2, e4, e8, e16, e32, e64; + felem ftmp, x2, x4, x8, x16, x32; unsigned i; - felem_square(ftmp, in); /* 2^1 */ - felem_mul(ftmp, in, ftmp); /* 2^2 - 2^0 */ - felem_assign(e2, ftmp); - felem_square(ftmp, ftmp); /* 2^3 - 2^1 */ - felem_square(ftmp, ftmp); /* 2^4 - 2^2 */ - felem_mul(ftmp, ftmp, e2); /* 2^4 - 2^0 */ - felem_assign(e4, ftmp); - felem_square(ftmp, ftmp); /* 2^5 - 2^1 */ - felem_square(ftmp, ftmp); /* 2^6 - 2^2 */ - felem_square(ftmp, ftmp); /* 2^7 - 2^3 */ - felem_square(ftmp, ftmp); /* 2^8 - 2^4 */ - felem_mul(ftmp, ftmp, e4); /* 2^8 - 2^0 */ - felem_assign(e8, ftmp); + /* x{k} holds in^(2^k - 1), a run of k ones. */ + felem_square(ftmp, in); + felem_mul(x2, ftmp, in); /* 2^2 - 1 */ + + felem_square(ftmp, x2); + felem_square(ftmp, ftmp); + felem_mul(x4, ftmp, x2); /* 2^4 - 1 */ + + felem_assign(ftmp, x4); + for (i = 0; i < 4; i++) { + felem_square(ftmp, ftmp); + } + felem_mul(x8, ftmp, x4); /* 2^8 - 1 */ + + felem_assign(ftmp, x8); for (i = 0; i < 8; i++) { felem_square(ftmp, ftmp); - } /* 2^16 - 2^8 */ - felem_mul(ftmp, ftmp, e8); /* 2^16 - 2^0 */ - felem_assign(e16, ftmp); + } + felem_mul(x16, ftmp, x8); /* 2^16 - 1 */ + + felem_assign(ftmp, x16); for (i = 0; i < 16; i++) { felem_square(ftmp, ftmp); - } /* 2^32 - 2^16 */ - felem_mul(ftmp, ftmp, e16); /* 2^32 - 2^0 */ - felem_assign(e32, ftmp); + } + felem_mul(x32, ftmp, x16); /* 2^32 - 1 */ + + /* The top 32 ones. */ + felem_assign(ftmp, x32); + + /* 31 zeros and a one: the 00000001 word. */ for (i = 0; i < 32; i++) { felem_square(ftmp, ftmp); - } /* 2^64 - 2^32 */ - felem_assign(e64, ftmp); - felem_mul(ftmp, ftmp, in); /* 2^64 - 2^32 + 2^0 */ - for (i = 0; i < 192; i++) { + } + felem_mul(ftmp, ftmp, in); + + /* 96 zeros. */ + for (i = 0; i < 96; i++) { felem_square(ftmp, ftmp); - } /* 2^256 - 2^224 + 2^192 */ + } - felem_mul(ftmp2, e64, e32); /* 2^64 - 2^0 */ + /* 94 ones, as 32 + 32 + 16 + 8 + 4 + 2. */ + for (i = 0; i < 32; i++) { + felem_square(ftmp, ftmp); + } + felem_mul(ftmp, ftmp, x32); + for (i = 0; i < 32; i++) { + felem_square(ftmp, ftmp); + } + felem_mul(ftmp, ftmp, x32); for (i = 0; i < 16; i++) { - felem_square(ftmp2, ftmp2); - } /* 2^80 - 2^16 */ - felem_mul(ftmp2, ftmp2, e16); /* 2^80 - 2^0 */ + felem_square(ftmp, ftmp); + } + felem_mul(ftmp, ftmp, x16); for (i = 0; i < 8; i++) { - felem_square(ftmp2, ftmp2); - } /* 2^88 - 2^8 */ - felem_mul(ftmp2, ftmp2, e8); /* 2^88 - 2^0 */ + felem_square(ftmp, ftmp); + } + felem_mul(ftmp, ftmp, x8); for (i = 0; i < 4; i++) { - felem_square(ftmp2, ftmp2); - } /* 2^92 - 2^4 */ - felem_mul(ftmp2, ftmp2, e4); /* 2^92 - 2^0 */ - felem_square(ftmp2, ftmp2); /* 2^93 - 2^1 */ - felem_square(ftmp2, ftmp2); /* 2^94 - 2^2 */ - felem_mul(ftmp2, ftmp2, e2); /* 2^94 - 2^0 */ - felem_square(ftmp2, ftmp2); /* 2^95 - 2^1 */ - felem_square(ftmp2, ftmp2); /* 2^96 - 2^2 */ - felem_mul(ftmp2, ftmp2, in); /* 2^96 - 3 */ + felem_square(ftmp, ftmp); + } + felem_mul(ftmp, ftmp, x4); + felem_square(ftmp, ftmp); + felem_square(ftmp, ftmp); + felem_mul(ftmp, ftmp, x2); - felem_mul(out, ftmp2, ftmp); /* 2^256 - 2^224 + 2^192 + 2^96 - 3 */ + /* A zero and a one: the d of fffffffd. */ + felem_square(ftmp, ftmp); + felem_square(ftmp, ftmp); + felem_mul(out, ftmp, in); } - /* Group operations: * * Elements of the elliptic curve group are represented in Jacobian @@ -298,7 +334,8 @@ } /* select_affine_point sets {out_x,out_y} to the index'th entry of table. - * On entry: index < 16, table[0] must be zero. */ + * On entry: index < 16. Every entry is a point of its own -- the signed + * representation has no zero digit -- so the scan starts at zero. */ static void select_affine_point(felem out_x, felem out_y, const limb* table, limb index) { limb i, j; @@ -306,7 +343,7 @@ memset(out_x, 0, sizeof(felem)); memset(out_y, 0, sizeof(felem)); - for (i = 1; i < 16; i++) { + for (i = 0; i < 16; i++) { limb mask = i ^ index; mask |= mask >> 2; mask |= mask >> 1; @@ -321,95 +358,306 @@ } } -/* select_jacobian_point sets {out_x,out_y,out_z} to the index'th entry of - * table. On entry: index < 16, table[0] must be zero. */ -static void select_jacobian_point(felem out_x, felem out_y, felem out_z, - const limb* table, limb index) { - limb i, j; +/* words_are_zero returns 1 when |v| is zero and 0 otherwise, without a + * branch. */ +static u32 words_are_zero(u32 v) { + v |= v >> 16; + v |= v >> 8; + v |= v >> 4; + v |= v >> 2; + v |= v >> 1; + return (v & 1) ^ 1; +} - memset(out_x, 0, sizeof(felem)); - memset(out_y, 0, sizeof(felem)); - memset(out_z, 0, sizeof(felem)); +/* The comb: five teeth to a block, the teeth of a block 52 apart and the two + * blocks 26 from each other, so 26 steps cover all 260 bits between them. + * + * first block i, 52+i, 104+i, 156+i, 208+i + * second block 26+i, 78+i, 130+i, 182+i, 234+i + * + * Five teeth would want a table of 32, but the signed representation makes + * every digit +-1, so the thirty-two values come in pairs that differ by sign + * and sixteen entries serve: kPrecomputed holds the ones whose top tooth is + * positive and the other sign is a negated y. That is the whole of the gain + * over the four-tooth unsigned comb this replaces, which took 32 steps for + * the same 256 bits: 25 doublings and 52 mixed additions against 31 and 64. + */ +#define COMB_TEETH 5 +#define COMB_STEPS 26 +#define COMB_SPAN (2 * COMB_STEPS) /* 52, the distance between a block's teeth */ +#define COMB_WORDS 9 /* 260 bits of signs, with room to add into */ +#define COMB_BIT(t, q) (((t)[(q) >> 5] >> ((q) & 31)) & 1) - /* The implicit value at index 0 is all zero. We don't need to perform that - * iteration of the loop because we already set out_* to zero. */ - table += 3 * NLIMBS; +/* comb_recode writes into |out| the value whose bits are the signs of the + * scalar's all-bits-set representation: digit j is +1 where bit j of |out| is + * set and -1 where it is not. + * + * For an odd k below 2^260 there is exactly one such representation, and it + * is (k + 2^260 - 1) / 2 read as bits -- an addition and a shift, and that is + * all the recoding is. An even scalar has the order added to make it odd, + * which changes the scalar and not the point it selects. + * + * Constant time in the scalar: every branch below is on a loop counter. */ +static void comb_recode(u32 out[COMB_WORDS], + const crypton_p256_int* scalar) { + u32 k[COMB_WORDS], ord[COMB_WORDS]; + u64 carry; + int i; - // Hit all entries to obscure cache profiling. - for (i = 1; i < 16; i++) { - limb mask = i ^ index; - mask |= mask >> 2; - mask |= mask >> 1; - mask &= 1; - mask--; - for (j = 0; j < NLIMBS; j++, table++) { - out_x[j] |= *table & mask; - } - for (j = 0; j < NLIMBS; j++, table++) { - out_y[j] |= *table & mask; - } - for (j = 0; j < NLIMBS; j++, table++) { - out_z[j] |= *table & mask; + for (i = 0; i < COMB_WORDS; i++) { + k[i] = 0; + ord[i] = 0; + } + for (i = 0; i < 256; i += 32) { + k[i >> 5] = (u32)(P256_DIGIT(scalar, i / P256_BITSPERDIGIT) + >> (i % P256_BITSPERDIGIT)); + ord[i >> 5] = (u32)(P256_DIGIT(&crypton_SECP256r1_n, i / P256_BITSPERDIGIT) + >> (i % P256_BITSPERDIGIT)); + } + + /* an even scalar becomes odd by taking on the order */ + { + u32 addmask = (u32)0 - (u32)((k[0] & 1) ^ 1); + + carry = 0; + for (i = 0; i < COMB_WORDS; i++) { + u64 v = (u64)k[i] + (u64)(ord[i] & addmask) + carry; + k[i] = (u32)v; + carry = v >> 32; } } + + /* out = (k + 2^260 - 1) >> 1 */ + carry = 0; + for (i = 0; i < COMB_WORDS; i++) { + u64 v = (u64)k[i] + (u64)(i < 8 ? 0xffffffffu : 0xfu) + carry; + out[i] = (u32)v; + carry = v >> 32; + } + for (i = 0; i < COMB_WORDS - 1; i++) { + out[i] = (out[i] >> 1) | (out[i + 1] << 31); + } + out[COMB_WORDS - 1] >>= 1; + } +/* point_add_complete_mixed sets {x3,y3,z3} = {x1,y1,z1} + {x2,y2}, where the + * accumulator is in projective coordinates and the added point is affine. + * + * This is Renes-Costello-Batina algorithm 5, for a curve with a = -3. It is + * complete: it is right when the two points are the same, when either is the + * point at infinity, and when they are each other's negation, which is what + * point_add_mixed cannot say. It costs 11 multiplications and two by b, + * against point_add_mixed's 8 multiplications and 3 squarings. + * + * The table this comb reads is affine, and an affine point has Z = 1, so + * Z = Z^2 = Z^3 and the same three numbers are the point in projective + * coordinates and in Jacobian ones. That is what lets a comb built on + * Jacobian arithmetic step into this formula for one addition and back out. + */ +static void point_add_complete_mixed(felem x3, felem y3, felem z3, + const felem x1, const felem y1, + const felem z1, const felem x2, + const felem y2) { + felem t0, t1, t2, t3, t4, xx, yy, zz; + + felem_mul(t0, x1, x2); + felem_mul(t1, y1, y2); + felem_sum(t3, x2, y2); + felem_sum(t4, x1, y1); + felem_mul(t3, t3, t4); + felem_sum(t4, t0, t1); + felem_diff(t3, t3, t4); + felem_mul(t4, y2, z1); + felem_sum(t4, t4, y1); + felem_mul(yy, x2, z1); + felem_sum(yy, yy, x1); + felem_mul(zz, kB, z1); + felem_diff(xx, yy, zz); + felem_sum(zz, xx, xx); + felem_sum(xx, xx, zz); + felem_diff(zz, t1, xx); + felem_sum(xx, t1, xx); + felem_mul(yy, kB, yy); + felem_sum(t1, z1, z1); + felem_sum(t2, t1, z1); + felem_diff(yy, yy, t2); + felem_diff(yy, yy, t0); + felem_sum(t1, yy, yy); + felem_sum(yy, t1, yy); + felem_sum(t1, t0, t0); + felem_sum(t0, t1, t0); + felem_diff(t0, t0, t2); + felem_mul(t1, t4, yy); + felem_mul(t2, t0, yy); + felem_mul(yy, xx, zz); + felem_sum(y3, yy, t2); + felem_mul(xx, t3, xx); + felem_diff(x3, xx, t1); + felem_mul(zz, t4, zz); + felem_mul(t1, t3, t0); + felem_sum(z3, zz, t1); +} + +/* point_add_complete sets {x3,y3,z3} = {x1,y1,z1} + {x2,y2,z2}, both in + * projective coordinates. + * + * Renes-Costello-Batina algorithm 4, for a = -3, and complete for the same + * reasons as the mixed one above. It costs 12 multiplications and two by b, + * against point_add's 11 multiplications and 5 squarings. + */ +static void point_add_complete(felem x3, felem y3, felem z3, const felem x1, + const felem y1, const felem z1, const felem x2, + const felem y2, const felem z2) { + felem t0, t1, t2, t3, t4, xx, yy, zz; + + felem_mul(t0, x1, x2); + felem_mul(t1, y1, y2); + felem_mul(t2, z1, z2); + felem_sum(t3, x1, y1); + felem_sum(t4, x2, y2); + felem_mul(t3, t3, t4); + felem_sum(t4, t0, t1); + felem_diff(t3, t3, t4); + felem_sum(t4, y1, z1); + felem_sum(xx, y2, z2); + felem_mul(t4, t4, xx); + felem_sum(xx, t1, t2); + felem_diff(t4, t4, xx); + felem_sum(xx, x1, z1); + felem_sum(yy, x2, z2); + felem_mul(xx, xx, yy); + felem_sum(yy, t0, t2); + felem_diff(yy, xx, yy); + felem_mul(zz, kB, t2); + felem_diff(xx, yy, zz); + felem_sum(zz, xx, xx); + felem_sum(xx, xx, zz); + felem_diff(zz, t1, xx); + felem_sum(xx, t1, xx); + felem_mul(yy, kB, yy); + felem_sum(t1, t2, t2); + felem_sum(t2, t1, t2); + felem_diff(yy, yy, t2); + felem_diff(yy, yy, t0); + felem_sum(t1, yy, yy); + felem_sum(yy, t1, yy); + felem_sum(t1, t0, t0); + felem_sum(t0, t1, t0); + felem_diff(t0, t0, t2); + felem_mul(t1, t4, yy); + felem_mul(t2, t0, yy); + felem_mul(yy, xx, zz); + felem_sum(y3, yy, t2); + felem_mul(xx, t3, xx); + felem_diff(x3, xx, t1); + felem_mul(zz, t4, zz); + felem_mul(t1, t3, t0); + felem_sum(z3, zz, t1); +} + +/* jacobian_to_projective sets {x2,y2,z2} to the projective form of the + * Jacobian point {x1,y1,z1}: (X/Z^2, Y/Z^3) is (XZ : Y : Z^3). */ +static void jacobian_to_projective(felem x2, felem y2, felem z2, + const felem x1, const felem y1, + const felem z1) { + felem zz, zzz; + + felem_square(zz, z1); + felem_mul(zzz, zz, z1); + felem_mul(x2, x1, z1); + memcpy(y2, y1, sizeof(felem)); + memcpy(z2, zzz, sizeof(felem)); +} + +/* projective_to_jacobian is the other way: (X/Z) is (XZ : YZ^2 : Z). */ +static void projective_to_jacobian(felem x2, felem y2, felem z2, + const felem x1, const felem y1, + const felem z1) { + felem zz; + + felem_square(zz, z1); + felem_mul(x2, x1, z1); + felem_mul(y2, y1, zz); + memcpy(z2, z1, sizeof(felem)); +} + /* scalar_base_mult sets {nx,ny,nz} = scalar*G where scalar is a little-endian * number. Note that the value of scalar must be less than the order of the * group. */ static void scalar_base_mult(felem nx, felem ny, felem nz, const crypton_p256_int* scalar) { - int i, j; - limb n_is_infinity_mask = -1, p_is_noninfinite_mask, mask; - u32 table_offset; + u32 rec[COMB_WORDS]; + limb n_is_infinity_mask = -1, mask; + felem px, py, negy, tx, ty, tz, jx, jy, jz, cx, cy, cz; + int i, blk, j; - felem px, py; - felem tx, ty, tz; + comb_recode(rec, scalar); memset(nx, 0, sizeof(felem)); memset(ny, 0, sizeof(felem)); memset(nz, 0, sizeof(felem)); - /* The loop adds bits at positions 0, 64, 128 and 192, followed by - * positions 32,96,160 and 224 and does this 32 times. */ - for (i = 0; i < 32; i++) { - if (i) { + for (i = COMB_STEPS - 1; i >= 0; i--) { + if (i != COMB_STEPS - 1) { point_double(nx, ny, nz, nx, ny, nz); } - table_offset = 0; - for (j = 0; j <= 32; j += 32) { - char bit0 = crypton_p256_get_bit(scalar, 31 - i + j); - char bit1 = crypton_p256_get_bit(scalar, 95 - i + j); - char bit2 = crypton_p256_get_bit(scalar, 159 - i + j); - char bit3 = crypton_p256_get_bit(scalar, 223 - i + j); - limb index = bit0 | (bit1 << 1) | (bit2 << 2) | (bit3 << 3); - select_affine_point(px, py, kPrecomputed + table_offset, index); - table_offset += 30 * NLIMBS; + for (blk = 0; blk < 2; blk++) { + u32 base = (u32)(blk * COMB_STEPS + i); + u32 top = COMB_BIT(rec, base + (COMB_TEETH - 1) * COMB_SPAN); + limb index = 0; - /* Since scalar is less than the order of the group, we know that - * {nx,ny,nz} != {px,py,1}, unless both are zero, which we handle - * below. */ - point_add_mixed(tx, ty, tz, nx, ny, nz, px, py); - /* The result of point_add_mixed is incorrect if {nx,ny,nz} is zero - * (a.k.a. the point at infinity). We handle that situation by - * copying the point from the table. */ + for (j = 0; j < COMB_TEETH - 1; j++) { + /* a tooth agreeing with the top one is a set bit of the index */ + index |= (limb)(COMB_BIT(rec, base + (u32)j * COMB_SPAN) ^ top ^ 1) + << j; + } + + select_affine_point(px, py, kPrecomputed + blk * 16 * 2 * NLIMBS, index); + felem_diff(negy, kZero, py); + copy_conditional(py, negy, (limb)0 - (limb)(top ^ 1)); + + /* The last addition is the one that can be a point added to itself: + * entering it the accumulator is (k' - B)*G and what it adds is B*G, + * where B is the second block's digit at step zero, so the two meet + * when k' = 2B modulo the order. One scalar below the order does + * that, and point_add_mixed cannot answer it. + * + * So that one addition goes through the complete formula instead. + * The table is affine, so the point just selected is the same three + * numbers read as projective coordinates as read as Jacobian ones -- + * which is what lets a comb built on Jacobian arithmetic step into + * the formula for an addition and back out of it. The accumulator + * is not affine and is converted. + * + * Measured on an M4, thread CPU time, the whole base point + * multiplication is 17.34 us this way against 17.30 us with an extra + * doubling and a mask, which is inside the spread of either. */ + if (i == 0 && blk == 1) { + jacobian_to_projective(jx, jy, jz, nx, ny, nz); + point_add_complete_mixed(cx, cy, cz, jx, jy, jz, px, py); + projective_to_jacobian(tx, ty, tz, cx, cy, cz); + } else { + point_add_mixed(tx, ty, tz, nx, ny, nz, px, py); + } + + /* The accumulator is the infinity until the first of these, and + * point_add_mixed cannot start from it; the point itself is the sum. */ copy_conditional(nx, px, n_is_infinity_mask); copy_conditional(ny, py, n_is_infinity_mask); copy_conditional(nz, kOne, n_is_infinity_mask); - - /* Equally, the result is also wrong if the point from the table is - * zero, which happens when the index is zero. We handle that by - * only copying from {tx,ty,tz} to {nx,ny,nz} if index != 0. */ - p_is_noninfinite_mask = NON_ZERO_TO_ALL_ONES(index); - mask = p_is_noninfinite_mask & ~n_is_infinity_mask; + mask = ~n_is_infinity_mask; copy_conditional(nx, tx, mask); copy_conditional(ny, ty, mask); copy_conditional(nz, tz, mask); - /* If p was not zero, then n is now non-zero. */ - n_is_infinity_mask &= ~p_is_noninfinite_mask; + n_is_infinity_mask = 0; } } + + /* The recoded scalar is the private key in another representation, so it + * does not stay on the stack. */ + crypton_bzero(rec, sizeof(rec)); } /* point_to_affine converts a Jacobian point to an affine point. If the input @@ -424,67 +672,309 @@ felem_mul(y_out, ny, z_inv); } -/* scalar_base_mult sets {nx,ny,nz} = scalar*{x,y}. */ -static void scalar_mult(felem nx, felem ny, felem nz, const felem x, - const felem y, const crypton_p256_int* scalar) { - int i; - felem px, py, pz, tx, ty, tz; - felem precomp[16][3]; - limb n_is_infinity_mask, index, p_is_noninfinite_mask, mask; +/* point_add_mixed_pm sets {xp,yp,zp} = {x1,y1,z1} + {x2,y2} and + * {xm,ym,zm} = {x1,y1,z1} - {x2,y2}, where {x2,y2} is affine. + * + * Negating the second point changes the sign of s2 and so of r, and nothing + * else: z1z1, tmp, u2, z1z1z1, h, i, j, v, the output z and the product y1*j + * are common to the two. What the second point costs over the first is one + * squaring (r*r) and one multiplication (by r), rather than another eleven. + * + * The same restrictions as point_add_mixed: this does not handle P+P, + * infinity+P nor P+infinity. */ +static void point_add_mixed_pm(felem xp, felem yp, felem zp, + felem xm, felem ym, felem zm, + const felem x1, const felem y1, const felem z1, + const felem x2, const felem y2) { + felem z1z1, z1z1z1, s2, u2, h, i, j, r, rr, v, y1j, tmp; - /* We precompute 0,1,2,... times {x,y}. */ - memset(precomp, 0, sizeof(felem) * 3); - memcpy(&precomp[1][0], x, sizeof(felem)); - memcpy(&precomp[1][1], y, sizeof(felem)); - memcpy(&precomp[1][2], kOne, sizeof(felem)); + felem_square(z1z1, z1); + felem_sum(tmp, z1, z1); - for (i = 2; i < 16; i += 2) { - point_double(precomp[i][0], precomp[i][1], precomp[i][2], - precomp[i / 2][0], precomp[i / 2][1], precomp[i / 2][2]); + felem_mul(u2, x2, z1z1); + felem_mul(z1z1z1, z1, z1z1); + felem_mul(s2, y2, z1z1z1); + felem_diff(h, u2, x1); + felem_sum(i, h, h); + felem_square(i, i); + felem_mul(j, h, i); + felem_mul(v, x1, i); + felem_mul(y1j, y1, j); - point_add_mixed(precomp[i + 1][0], precomp[i + 1][1], precomp[i + 1][2], - precomp[i][0], precomp[i][1], precomp[i][2], x, y); + /* The two points share their z. */ + felem_mul(zp, tmp, h); + felem_assign(zm, zp); + + /* X + P */ + felem_diff(r, s2, y1); + felem_sum(r, r, r); + felem_square(rr, r); + felem_diff(xp, rr, j); + felem_diff(xp, xp, v); + felem_diff(xp, xp, v); + felem_diff(tmp, v, xp); + felem_mul(yp, tmp, r); + felem_diff(yp, yp, y1j); + felem_diff(yp, yp, y1j); + + /* X - P. Negating the point negates s2, so r becomes -q where + * q = 2*(s2 + y1). The square is the same either way, and the sign is + * carried into y by taking (xm - v) where the other took (v - xp): + * xm = q^2 - j - 2v + * ym = (v - xm)*(-q) - 2*y1*j = (xm - v)*q - 2*y1*j + * so no field negation is needed. */ + felem_sum(r, s2, y1); + felem_sum(r, r, r); + felem_square(rr, r); + felem_diff(xm, rr, j); + felem_diff(xm, xm, v); + felem_diff(xm, xm, v); + felem_diff(tmp, xm, v); + felem_mul(ym, tmp, r); + felem_diff(ym, ym, y1j); + felem_diff(ym, ym, y1j); +} + +/* select_jacobian_odd sets {out_x,out_y,out_z} to the index'th of the 16 + * entries of table, for index < 16. There is no implicit infinity at index + * zero, as the unsigned window this replaces had: every entry is a real + * point, which is what lets the signed representation below do without the + * infinity masks. */ +static void select_jacobian_odd(felem out_x, felem out_y, felem out_z, + const limb* table, limb index) { + limb i, j; + + memset(out_x, 0, sizeof(felem)); + memset(out_y, 0, sizeof(felem)); + memset(out_z, 0, sizeof(felem)); + + for (i = 0; i < 16; i++) { + limb mask = i ^ index; + mask |= mask >> 2; + mask |= mask >> 1; + mask &= 1; + mask--; + for (j = 0; j < NLIMBS; j++, table++) { + out_x[j] |= *table & mask; + } + for (j = 0; j < NLIMBS; j++, table++) { + out_y[j] |= *table & mask; + } + for (j = 0; j < NLIMBS; j++, table++) { + out_z[j] |= *table & mask; + } } +} - memset(nx, 0, sizeof(felem)); - memset(ny, 0, sizeof(felem)); - memset(nz, 0, sizeof(felem)); - n_is_infinity_mask = -1; +/* The scalar, recoded: 52 signed odd digits, each in {+-1,+-3,...,+-31}, so + * that scalar = sum d_i * 32^i. A digit is one byte: the low four bits are + * the table index (|d|-1)/2, and bit four is set when d is negative. One + * byte rather than a byte and a word because this is the private key in + * another form and has to be wiped afterwards. */ +#define SABS_DIGITS 52 +#define SABS_INDEX(b) ((limb)((b) & 15)) +#define SABS_NEGMASK(b) ((limb)0 - (limb)((b) >> 4)) +typedef struct { + u8 digit[SABS_DIGITS]; +} sabs_scalar; - /* We add in a window of four bits each iteration and do this 64 times. */ - for (i = 0; i < 256; i += 4) { - if (i) { - point_double(nx, ny, nz, nx, ny, nz); - point_double(nx, ny, nz, nx, ny, nz); - point_double(nx, ny, nz, nx, ny, nz); - point_double(nx, ny, nz, nx, ny, nz); + +/* sabs_recode writes the signed representation of |scalar| into |out|. + * + * The recoding is the regular one of Joye and Tunstall: take the low six bits, + * subtract 32, and carry the difference upwards. It needs an odd input, which + * is arranged by adding the group order to an even scalar -- that changes the + * scalar but not the point it selects, the order being the order. A zero + * scalar is replaced by one and the caller is told, since zero times a point + * is the infinity this code deliberately cannot represent. + * + * Constant time in the scalar: every branch below is on a loop counter. */ +static limb sabs_recode(sabs_scalar* out, + const crypton_p256_int* scalar) { + u32 k[9], n[9]; + u32 nonzero; + limb is_zero_mask; + int i, b; + + for (i = 0; i < 9; i++) { + k[i] = 0; + n[i] = 0; + } + /* A word at a time. Bit at a time would be 512 calls into another + * translation unit, which the compiler cannot inline away. */ + for (b = 0; b < 256; b += 32) { + k[b >> 5] = (u32)(P256_DIGIT(scalar, b / P256_BITSPERDIGIT) + >> (b % P256_BITSPERDIGIT)); + n[b >> 5] = (u32)(P256_DIGIT(&crypton_SECP256r1_n, b / P256_BITSPERDIGIT) + >> (b % P256_BITSPERDIGIT)); + } + + /* Replace a zero scalar by one, and report it. */ + nonzero = 0; + for (i = 0; i < 9; i++) { + nonzero |= k[i]; + } + { + u32 z = words_are_zero(nonzero); + k[0] |= z; + is_zero_mask = (limb)0 - (limb)z; + } + + /* An even scalar becomes odd by adding the order. The sum is below 2^257, + * which is why nine words and fifty-two digits are enough. */ + { + u32 addmask = (u32)0 - (u32)((k[0] & 1) ^ 1); + u64 carry = 0; + for (i = 0; i < 9; i++) { + u64 t = (u64)k[i] + (u64)(n[i] & addmask) + carry; + k[i] = (u32)t; + carry = t >> 32; } + } - index = (crypton_p256_get_bit(scalar, 255 - i - 0) << 3) | - (crypton_p256_get_bit(scalar, 255 - i - 1) << 2) | - (crypton_p256_get_bit(scalar, 255 - i - 2) << 1) | - crypton_p256_get_bit(scalar, 255 - i - 3); + for (i = 0; i < SABS_DIGITS - 1; i++) { + u32 r6 = k[0] & 63; /* odd, so never 32 */ + u32 hi = (r6 >> 5) & 1; /* 1 when the digit is positive */ + u32 wabs = ((r6 - 32) & (0u - hi)) | ((32 - r6) & (hi - 1)); + u32 mlo = 32u - r6; /* two's complement of the digit's negation */ + u32 ext = 0u - hi; /* its sign extension */ + u64 carry = 0; + int w; - /* See the comments in scalar_base_mult about handling infinities. */ - select_jacobian_point(px, py, pz, precomp[0][0], index); - point_add(tx, ty, tz, nx, ny, nz, px, py, pz); - copy_conditional(nx, px, n_is_infinity_mask); - copy_conditional(ny, py, n_is_infinity_mask); - copy_conditional(nz, pz, n_is_infinity_mask); + out->digit[i] = (u8)(((wabs - 1) >> 1) | ((hi ^ 1) << 4)); - p_is_noninfinite_mask = NON_ZERO_TO_ALL_ONES(index); - mask = p_is_noninfinite_mask & ~n_is_infinity_mask; + /* k -= digit, i.e. k += -digit, sign extended over the nine words. */ + for (w = 0; w < 9; w++) { + u64 t = (u64)k[w] + (u64)(w == 0 ? mlo : ext) + carry; + k[w] = (u32)t; + carry = t >> 32; + } + /* k >>= 5 */ + for (w = 0; w < 8; w++) { + k[w] = (k[w] >> 5) | (k[w + 1] << 27); + } + k[8] >>= 5; + } - copy_conditional(nx, tx, mask); - copy_conditional(ny, ty, mask); - copy_conditional(nz, tz, mask); - n_is_infinity_mask &= ~p_is_noninfinite_mask; + /* What is left is odd, positive and at most five: the scalar is below + * 2^257 and fifty-one digits have taken 255 bits off it, each leaving a + * remainder below one. */ + out->digit[SABS_DIGITS - 1] = (u8)((k[0] - 1) >> 1); + + return is_zero_mask; +} + +/* scalar_mult sets {nx,ny,nz} = scalar*{x,y}. + * + * A five-bit signed window. The scalar is recoded into 52 digits, every one + * of them odd and none of them zero, so the table holds only the odd + * multiples P, 3P, ..., 31P and a negative digit is served by negating y, + * which is free. Against the four-bit unsigned window this replaces, the + * main loop trades 252 doublings and 64 additions for 255 and 51, and -- + * because no digit is zero and no partial sum is the infinity -- it drops the + * masks that stood in for infinity on every iteration. + * + * The table is built so that each pair of neighbouring odd multiples comes + * out of one doubling and one shared addition: + * + * 2P = 2*P 3P = 2P + P + * 6P = 2*(3P) 5P = 6P - P, 7P = 6P + P + * 10P = 2*(5P) 9P = 10P - P, 11P = 10P + P + * ... + * 30P = 2*(15P) 29P = 30P - P, 31P = 30P + P + * + * which is eight doublings, one mixed addition and seven shared pairs. */ +static void scalar_mult(felem nx, felem ny, felem nz, const felem x, + const felem y, const crypton_p256_int* scalar) { + /* odd[k] is (2k+1)*P, for k in 0..15. */ + felem odd[16][3]; + felem dx, dy, dz, px, py, pz, negy, ddx, ddy, ddz, qx, qy, qz, cx, cy, cz; + sabs_scalar rec; + limb is_zero_mask; + int i, k; + + is_zero_mask = sabs_recode(&rec, scalar); + + felem_assign(odd[0][0], x); + felem_assign(odd[0][1], y); + memcpy(odd[0][2], kOne, sizeof(felem)); + + /* 3P = 2P + P */ + point_double(dx, dy, dz, x, y, kOne); + point_add_mixed(odd[1][0], odd[1][1], odd[1][2], dx, dy, dz, x, y); + + /* (4k+2)P from (2k+1)P, then (4k+1)P and (4k+3)P from it. */ + for (k = 1; k < 8; k++) { + point_double(dx, dy, dz, odd[k][0], odd[k][1], odd[k][2]); + point_add_mixed_pm(odd[2 * k + 1][0], odd[2 * k + 1][1], odd[2 * k + 1][2], + odd[2 * k][0], odd[2 * k][1], odd[2 * k][2], + dx, dy, dz, x, y); } + + /* The top digit initialises the accumulator; it is always positive. */ + select_jacobian_odd(nx, ny, nz, odd[0][0], + SABS_INDEX(rec.digit[SABS_DIGITS - 1])); + + for (i = SABS_DIGITS - 2; i >= 0; i--) { + point_double(nx, ny, nz, nx, ny, nz); + point_double(nx, ny, nz, nx, ny, nz); + point_double(nx, ny, nz, nx, ny, nz); + point_double(nx, ny, nz, nx, ny, nz); + point_double(nx, ny, nz, nx, ny, nz); + + select_jacobian_odd(px, py, pz, odd[0][0], SABS_INDEX(rec.digit[i])); + felem_diff(negy, kZero, py); + copy_conditional(py, negy, SABS_NEGMASK(rec.digit[i])); + + /* On the last step alone the accumulator can be the very point being + * added -- the accumulator is (k' - d0)*P and it adds d0*P, so the two + * meet when k' = 2*d0 modulo the order, which the scalar 30 does with a + * digit of 15 -- and point_add answers the infinity where the truth is + * twice that point. That one addition goes through the complete formula + * instead, with both points converted to projective coordinates and the + * answer converted back. One of fifty-one iterations pays for it, and + * it comes to a field multiplication less than the doubling and the mask + * it replaces. + * + * point_add finishes with z before it touches x, and with each of x and + * y before the next, so on every other step the accumulator can be its + * own output. */ + if (i == 0) { + jacobian_to_projective(ddx, ddy, ddz, nx, ny, nz); + jacobian_to_projective(qx, qy, qz, px, py, pz); + point_add_complete(cx, cy, cz, ddx, ddy, ddz, qx, qy, qz); + projective_to_jacobian(nx, ny, nz, cx, cy, cz); + } else { + point_add(nx, ny, nz, nx, ny, nz, px, py, pz); + } + } + + /* Zero was replaced by one on the way in; put the infinity back. All + * three coordinates, not just z: crypton_p256_points_mul_vartime reads the + * comment above it as saying the whole point is zero. */ + for (i = 0; i < NLIMBS; i++) { + nx[i] &= ~is_zero_mask; + ny[i] &= ~is_zero_mask; + nz[i] &= ~is_zero_mask; + } + + /* The recoded scalar is the private key in another representation, so it + * does not stay on the stack. */ + crypton_bzero(&rec, sizeof(rec)); } /* crypton_p256_base_point_mul sets {out_x,out_y} = nG, where n is < the * order of the group. */ void crypton_p256_base_point_mul(const crypton_p256_int* n, crypton_p256_int* out_x, crypton_p256_int* out_y) { +#ifdef CRYPTON_S2N_BIGNUM + /* The assembly, four times faster, and constant time as this has to be: + * it is how a public key is derived from a private one. */ + uint64_t res[8]; + + crypton_s2n_p256_scalarmulbase(res, P256_DIGITS(n)); + memcpy(P256_DIGITS(out_x), res, P256_NBYTES); + memcpy(P256_DIGITS(out_y), res + P256_NDIGITS, P256_NBYTES); +#else felem x, y, z; scalar_base_mult(x, y, z, n); @@ -496,8 +986,29 @@ from_montgomery(out_x, x_affine); from_montgomery(out_y, y_affine); } +#endif } +#ifdef CRYPTON_S2N_BIGNUM +/* An affine point from the assembly as the Jacobian triple the addition + * below wants, keeping this file's convention that the point at infinity is + * all three coordinates zero -- the assembly says it with (0, 0). */ +static void s2n_lift(felem x, felem y, felem z, const uint64_t res[8]) { + crypton_p256_int t; + uint64_t any = 0; + int i; + + for (i = 0; i < 2 * P256_NDIGITS; i++) + any |= res[i]; + + memcpy(P256_DIGITS(&t), res, P256_NBYTES); + to_montgomery(x, &t); + memcpy(P256_DIGITS(&t), res + P256_NDIGITS, P256_NBYTES); + to_montgomery(y, &t); + memcpy(z, any != 0 ? kOne : kZero, sizeof(felem)); +} +#endif + /* crypton_p256_points_mul_vartime sets {out_x,out_y} = n1*G + n2*{in_x,in_y}, where * n1 and n2 are < the order of the group. * @@ -516,10 +1027,48 @@ return; } +#ifdef CRYPTON_S2N_BIGNUM + { + /* Both scalars at once, in variable time, which is what the two + * multiplications below are not: they are constant time, and pay for it, + * over values that are all public here. It gives up on the one case the + * assembly's addition does not cover -- adding a point to itself -- and + * then the constant-time pair answers instead. */ + uint64_t jr[3 * P256_NDIGITS]; + + if (crypton_p256_verify_mul(jr, P256_DIGITS(n1), P256_DIGITS(n2), + P256_DIGITS(in_x), P256_DIGITS(in_y))) { + crypton_p256_int t; + + memcpy(P256_DIGITS(&t), jr, P256_NBYTES); + to_montgomery(x1, &t); + memcpy(P256_DIGITS(&t), jr + P256_NDIGITS, P256_NBYTES); + to_montgomery(y1, &t); + memcpy(P256_DIGITS(&t), jr + 2 * P256_NDIGITS, P256_NBYTES); + to_montgomery(z1, &t); + + point_to_affine(px, py, x1, y1, z1); + from_montgomery(out_x, px); + from_montgomery(out_y, py); + return; + } + } + { + uint64_t r1[8], r2[8], pt[2 * P256_NDIGITS]; + + memcpy(pt, P256_DIGITS(in_x), P256_NBYTES); + memcpy(pt + P256_NDIGITS, P256_DIGITS(in_y), P256_NBYTES); + crypton_s2n_p256_scalarmulbase(r1, P256_DIGITS(n1)); + crypton_s2n_p256_scalarmul(r2, P256_DIGITS(n2), pt); + s2n_lift(x1, y1, z1, r1); + s2n_lift(x2, y2, z2, r2); + } +#else to_montgomery(px, in_x); to_montgomery(py, in_y); scalar_base_mult(x1, y1, z1, n1); scalar_mult(x2, y2, z2, px, py, n2); +#endif if (crypton_p256_is_zero(n2) != 0) { /* If n2 == 0, then {x2,y2,z2} is zero and the result is just @@ -581,6 +1130,19 @@ void crypton_p256e_point_mul(const crypton_p256_int* n, const crypton_p256_int* in_x, const crypton_p256_int* in_y, crypton_p256_int* out_x, crypton_p256_int* out_y) { +#ifdef CRYPTON_S2N_BIGNUM + /* The vendored assembly, which answers the same thing two and a half to + * three times faster. Both sides are four little-endian 64-bit words of + * an ordinary affine coordinate, so the points cross as they are, and + * both give (0, 0) for the point at infinity. See cbits/s2n/README.md. */ + uint64_t point[8], res[8]; + + memcpy(point, P256_DIGITS(in_x), P256_NBYTES); + memcpy(point + P256_NDIGITS, P256_DIGITS(in_y), P256_NBYTES); + crypton_s2n_p256_scalarmul(res, P256_DIGITS(n), point); + memcpy(P256_DIGITS(out_x), res, P256_NBYTES); + memcpy(P256_DIGITS(out_y), res + P256_NDIGITS, P256_NBYTES); +#else felem x, y, z, px, py; to_montgomery(px, in_x); @@ -589,4 +1151,5 @@ point_to_affine(px, py, x, y, z); from_montgomery(out_x, px); from_montgomery(out_y, py); +#endif } diff --git a/cbits/p256/p256_s2n.c b/cbits/p256/p256_s2n.c new file mode 100644 --- /dev/null +++ b/cbits/p256/p256_s2n.c @@ -0,0 +1,61 @@ +/* + * Choosing between s2n-bignum's two forms of each routine. The reasoning + * is in cbits/s2n/README.md; in short, on ARM the choice is a + * microarchitecture one that no feature bit answers, and on x86-64 it is + * exactly a feature bit. + */ +#include "p256/p256_s2n.h" +#include "crypton_cpu.h" + +extern void p256_scalarmul(uint64_t res[8], const uint64_t scalar[4], + const uint64_t point[8]); +extern void p256_scalarmul_alt(uint64_t res[8], const uint64_t scalar[4], + const uint64_t point[8]); +extern void p256_scalarmulbase(uint64_t res[8], const uint64_t scalar[4], + uint64_t blocksize, const uint64_t *table); +extern void p256_scalarmulbase_alt(uint64_t res[8], const uint64_t scalar[4], + uint64_t blocksize, const uint64_t *table); + +extern const uint64_t crypton_p256_s2n_base_blocksize; +extern const uint64_t crypton_p256_s2n_base_table[]; + +void crypton_s2n_p256_scalarmul(uint64_t res[8], const uint64_t scalar[4], + const uint64_t point[8]) +{ +#if defined(__aarch64__) || defined(__arm64__) +#ifdef __APPLE__ + /* the _alt form is the one written for high multiplier throughput, + * and it is 30-40% faster on Apple silicon */ + p256_scalarmul_alt(res, scalar, point); +#else + p256_scalarmul(res, scalar, point); +#endif +#else + if (crypton_x86_simd_features() & CRYPTON_X86_ADX) + p256_scalarmul(res, scalar, point); + else + p256_scalarmul_alt(res, scalar, point); +#endif +} + +void crypton_s2n_p256_scalarmulbase(uint64_t res[8], const uint64_t scalar[4]) +{ +#if defined(__aarch64__) || defined(__arm64__) +#ifdef __APPLE__ + p256_scalarmulbase_alt(res, scalar, crypton_p256_s2n_base_blocksize, + crypton_p256_s2n_base_table); +#else + p256_scalarmulbase(res, scalar, crypton_p256_s2n_base_blocksize, + crypton_p256_s2n_base_table); +#endif +#else + if (crypton_x86_simd_features() & CRYPTON_X86_ADX) + p256_scalarmulbase(res, scalar, + crypton_p256_s2n_base_blocksize, + crypton_p256_s2n_base_table); + else + p256_scalarmulbase_alt(res, scalar, + crypton_p256_s2n_base_blocksize, + crypton_p256_s2n_base_table); +#endif +} diff --git a/cbits/p256/p256_verify.c b/cbits/p256/p256_verify.c new file mode 100644 --- /dev/null +++ b/cbits/p256/p256_verify.c @@ -0,0 +1,300 @@ +/* + * The double scalar multiplication ECDSA verification does, in variable + * time. + * + * Verification asks for u1*G + u2*Q, and crypton used to work that out as + * two separate constant-time multiplications and an addition. Nothing here + * is secret -- the message, the signature and the public key are all sent in + * the clear -- so the constant-time work is paid for nothing, and the two + * multiplications can share their doublings besides. Measured, those two + * were 84% of a verification. + * + * So this is the usual interleaved windowed form: both scalars in + * width-5 non-adjacent form, a table of odd multiples of each point, and one + * pass down the digits with a doubling at every step and an addition where a + * digit is not zero. It is s2n-bignum's Jacobian point arithmetic + * underneath, the same assembly the constant-time paths use. + * + * s2n-bignum's p256_montjadd is correct except when its two arguments are + * the same point -- that is the side condition its proof carries -- so every + * sum is checked for the sign of that, which is a zero z where neither + * argument had one. There the answer is given up on and the caller falls + * back to the constant-time pair, which has no such condition. With random + * inputs it does not happen; it is here because an attacker chooses the + * public key and the signature. + */ +#include + +#include "p256/p256.h" + +#ifdef CRYPTON_S2N_BIGNUM + +#include "crypton_cpu.h" +#include "p256/p256_verify.h" + +/* a Jacobian triple in the Montgomery domain, as the assembly keeps them */ +#define JAC 12 +#define AFF 8 + +extern void p256_montjadd(uint64_t p3[JAC], const uint64_t p1[JAC], + const uint64_t p2[JAC]); +extern void p256_montjadd_alt(uint64_t p3[JAC], const uint64_t p1[JAC], + const uint64_t p2[JAC]); +extern void p256_montjdouble(uint64_t p3[JAC], const uint64_t p1[JAC]); +extern void p256_montjdouble_alt(uint64_t p3[JAC], const uint64_t p1[JAC]); +extern void p256_montjmixadd(uint64_t p3[JAC], const uint64_t p1[JAC], + const uint64_t p2[AFF]); +extern void p256_montjmixadd_alt(uint64_t p3[JAC], const uint64_t p1[JAC], + const uint64_t p2[AFF]); + +/* the odd multiples of the base point, from cbits/p256/gen_base_table.py */ +extern const uint64_t crypton_p256_wnaf_width; +extern const uint64_t crypton_p256_wnaf_table[]; +extern void bignum_tomont_p256(uint64_t z[4], const uint64_t x[4]); +extern void bignum_demont_p256(uint64_t z[4], const uint64_t x[4]); +extern void bignum_neg_p256(uint64_t z[4], const uint64_t x[4]); +#if !defined(__aarch64__) && !defined(__arm64__) +extern void bignum_tomont_p256_alt(uint64_t z[4], const uint64_t x[4]); +extern void bignum_demont_p256_alt(uint64_t z[4], const uint64_t x[4]); +#endif + +/* The same question as everywhere else in cbits/s2n: a microarchitecture one + * on ARM that no feature bit answers, and exactly a feature bit on x86-64. */ +static int use_alt(void) +{ +#if defined(__aarch64__) || defined(__arm64__) +#ifdef __APPLE__ + return 1; +#else + return 0; +#endif +#else + return (crypton_x86_simd_features() & CRYPTON_X86_ADX) == 0; +#endif +} + +static void padd(uint64_t r[JAC], const uint64_t a[JAC], const uint64_t b[JAC], + int alt) +{ + if (alt) + p256_montjadd_alt(r, a, b); + else + p256_montjadd(r, a, b); +} + +static void pmixadd(uint64_t r[JAC], const uint64_t a[JAC], + const uint64_t b[AFF], int alt) +{ + if (alt) + p256_montjmixadd_alt(r, a, b); + else + p256_montjmixadd(r, a, b); +} + +static void pdouble(uint64_t r[JAC], const uint64_t a[JAC], int alt) +{ + if (alt) + p256_montjdouble_alt(r, a); + else + p256_montjdouble(r, a); +} + +static void tomont(uint64_t z[4], const uint64_t x[4]) +{ +#if defined(__aarch64__) || defined(__arm64__) + bignum_tomont_p256(z, x); +#else + if (use_alt()) + bignum_tomont_p256_alt(z, x); + else + bignum_tomont_p256(z, x); +#endif +} + +static void demont(uint64_t z[4], const uint64_t x[4]) +{ +#if defined(__aarch64__) || defined(__arm64__) + bignum_demont_p256(z, x); +#else + if (use_alt()) + bignum_demont_p256_alt(z, x); + else + bignum_demont_p256(z, x); +#endif +} + +static int is_infinity(const uint64_t p[JAC]) +{ + return (p[8] | p[9] | p[10] | p[11]) == 0; +} + +/* + * The base point's table is a constant, so its window is as wide as the + * table is worth carrying: seven bits, thirty-two odd multiples, two + * kilobytes, and one addition every eight digits. The public key's table + * has to be built for each verification, so five bits is the trade there -- + * eight entries, seven point operations to build, and one addition every + * six digits. + */ +#define WG 7 +#define TG (1 << (WG - 2)) +#define WQ 5 +#define TQ (1 << (WQ - 2)) +#define NAF_MAX 258 + +/* + * The width-W non-adjacent form of a scalar, one signed digit per bit + * position: odd or zero, and never two non-zero digits within W of each + * other. Returns how many digits were written. + * + * The scalar is public, so the loop may look at it. + */ +static int wnaf(int8_t out[NAF_MAX], const uint64_t in[4], int w) +{ + uint64_t k[5]; + int len = 0; + + memcpy(k, in, 32); + k[4] = 0; + + while (k[0] | k[1] | k[2] | k[3] | k[4]) { + int d = 0; + + if (k[0] & 1) { + d = (int) (k[0] & ((1u << w) - 1)); + if (d >= (1 << (w - 1))) + d -= 1 << w; + if (d > 0) { + uint64_t borrow = (uint64_t) d; + int i; + + for (i = 0; i < 5 && borrow; i++) { + uint64_t t = k[i]; + + k[i] = t - borrow; + borrow = (k[i] > t); + } + } else { + uint64_t carry = (uint64_t) (-d); + int i; + + for (i = 0; i < 5 && carry; i++) { + k[i] += carry; + carry = (k[i] < carry); + } + } + } + out[len++] = (int8_t) d; + + { /* k >>= 1 */ + int i; + + for (i = 0; i < 4; i++) + k[i] = (k[i] >> 1) | (k[i + 1] << 63); + k[4] >>= 1; + } + } + return len; +} + +/* P, 3P, 5P, ..., (2*TQ-1)P from a Jacobian P */ +static int build_table(uint64_t t[TQ][JAC], const uint64_t p[JAC], int alt) +{ + uint64_t twice[JAC]; + int i; + + memcpy(t[0], p, sizeof(uint64_t) * JAC); + pdouble(twice, p, alt); + for (i = 1; i < TQ; i++) { + padd(t[i], t[i - 1], twice, alt); + /* the table is built from a point and its double, which are + * never the same point unless the point has order two, and + * P-256 has none */ + if (is_infinity(t[i]) && !is_infinity(t[i - 1]) + && !is_infinity(twice)) + return 0; + } + return 1; +} + +/* the table entry for a digit, negated when the digit is */ +static void pick(uint64_t out[JAC], const uint64_t t[TQ][JAC], int digit) +{ + int idx = (digit > 0 ? digit : -digit) / 2; + + memcpy(out, t[idx], sizeof(uint64_t) * JAC); + if (digit < 0) + bignum_neg_p256(out + 4, out + 4); +} + +/* the same from the base point's affine table */ +static void pick_affine(uint64_t out[AFF], int digit) +{ + int idx = (digit > 0 ? digit : -digit) / 2; + + memcpy(out, crypton_p256_wnaf_table + (size_t) idx * AFF, + sizeof(uint64_t) * AFF); + if (digit < 0) + bignum_neg_p256(out + 4, out + 4); +} + +int crypton_p256_verify_mul(uint64_t out[JAC], const uint64_t n1[4], + const uint64_t n2[4], const uint64_t qx[4], + const uint64_t qy[4]) +{ + /* the Montgomery form of one, which is the z of an affine point */ + static const uint64_t mont_one[4] = { + 0x0000000000000001ULL, 0xffffffff00000000ULL, + 0xffffffffffffffffULL, 0x00000000fffffffeULL + }; + uint64_t tq[TQ][JAC]; + uint64_t q[JAC], acc[JAC], addend[JAC], aff[AFF], sum[JAC]; + int8_t naf1[NAF_MAX], naf2[NAF_MAX]; + int len1, len2, len, i; + /* asked once rather than at every point operation */ + const int alt = use_alt(); + + /* the generated table has to be the width this file walks it at */ + if (crypton_p256_wnaf_width != WG) + return 0; + + tomont(q, qx); + tomont(q + 4, qy); + memcpy(q + 8, mont_one, sizeof mont_one); + + if (!build_table(tq, q, alt)) + return 0; + + len1 = wnaf(naf1, n1, WG); + len2 = wnaf(naf2, n2, WQ); + len = len1 > len2 ? len1 : len2; + + memset(acc, 0, sizeof acc); + for (i = len - 1; i >= 0; i--) { + pdouble(acc, acc, alt); + + if (i < len1 && naf1[i] != 0) { + /* the base point's entries are affine, which is a + * cheaper addition and no table to build */ + pick_affine(aff, naf1[i]); + pmixadd(sum, acc, aff, alt); + if (is_infinity(sum) && !is_infinity(acc)) + return 0; + memcpy(acc, sum, sizeof acc); + } + if (i < len2 && naf2[i] != 0) { + pick(addend, tq, naf2[i]); + padd(sum, acc, addend, alt); + if (is_infinity(sum) && !is_infinity(acc)) + return 0; + memcpy(acc, sum, sizeof acc); + } + } + + demont(out, acc); + demont(out + 4, acc + 4); + demont(out + 8, acc + 8); + return 1; +} + +#endif diff --git a/cbits/p256/p256_verify.h b/cbits/p256/p256_verify.h new file mode 100644 --- /dev/null +++ b/cbits/p256/p256_verify.h @@ -0,0 +1,19 @@ +#ifndef CRYPTON_P256_VERIFY_H +#define CRYPTON_P256_VERIFY_H + +#include + +/* + * n1*G + n2*Q, in variable time, as a Jacobian triple in the plain domain. + * + * Returns 1 when the answer is in `out`, and 0 when the walk met the one + * case s2n-bignum's point addition does not cover -- adding a point to + * itself -- in which case the caller works the answer out the constant-time + * way instead. Nothing here is secret: it is all in the signature and the + * public key. + */ +int crypton_p256_verify_mul(uint64_t out[12], const uint64_t n1[4], + const uint64_t n2[4], const uint64_t qx[4], + const uint64_t qy[4]); + +#endif diff --git a/cbits/p256/p256_wnaf_table.c b/cbits/p256/p256_wnaf_table.c new file mode 100644 --- /dev/null +++ b/cbits/p256/p256_wnaf_table.c @@ -0,0 +1,42 @@ +/* Generated by cbits/p256/gen_base_table.py; do not hand-edit. + * The odd multiples of the base point, in Montgomery-affine + * form, which cbits/p256/p256_verify.c reads. A verification + * is public, so this table is walked in variable time. */ +#include + +const uint64_t crypton_p256_wnaf_width = 7; + +const uint64_t crypton_p256_wnaf_table[256] = { + 0x79e730d418a9143cULL,0x75ba95fc5fedb601ULL,0x79fb732b77622510ULL,0x18905f76a53755c6ULL,0xddf25357ce95560aULL,0x8b4ab8e4ba19e45cULL,0xd2e88688dd21f325ULL,0x8571ff1825885d85ULL, + 0xffac3f904eebc127ULL,0xb027f84a087d81fbULL,0x66ad77dd87cbbc98ULL,0x26936a3fb6ff747eULL,0xb04c5c1fc983a7ebULL,0x583e47ad0861fe1aULL,0x788208311a2ee98eULL,0xd5f06a29e587cc07ULL, + 0xbe1b8aaec45c61f5ULL,0x90ec649a94b9537dULL,0x941cb5aad076c20cULL,0xc9079605890523c8ULL,0xeb309b4ae7ba4f10ULL,0x73c568efe5eb882bULL,0x3540a9877e7a1f68ULL,0x73a076bb2dd1e916ULL, + 0x0746354ea0173b4fULL,0x2bd20213d23c00f7ULL,0xf43eaab50c23bb08ULL,0x13ba5119c3123e03ULL,0x2847d0303f5b9d4dULL,0x6742f2f25da67bddULL,0xef933bdc77c94195ULL,0xeaedd9156e240867ULL, + 0x75c96e8f264e20e8ULL,0xabe6bfed59a7a841ULL,0x2cc09c0444c8eb00ULL,0xe05b3080f0c4e16bULL,0x1eb7777aa45f3314ULL,0x56af7bedce5d45e3ULL,0x2b6e019a88b12f1aULL,0x086659cdfd835f9bULL, + 0xea7d260a6245e404ULL,0x9de407956e7fdfe0ULL,0x1ff3a4158dac1ab5ULL,0x3e7090f1649c9073ULL,0x1a7685612b944e88ULL,0x250f939ee57f61c8ULL,0x0c0daa891ead643dULL,0x68930023e125b88eULL, + 0xccc425634b2ed709ULL,0x0e356769856fd30dULL,0xbcbcd43f559e9811ULL,0x738477ac5395b759ULL,0x35752b90c00ee17fULL,0x68748390742ed2e3ULL,0x7cd06422bd1f5bc1ULL,0xfbc08769c9e7b797ULL, + 0x72bcd8b7bc60055bULL,0x03cc23ee56e27e4bULL,0xee337424e4819370ULL,0xe2aa0e430ad3da09ULL,0x40b8524f6383c45dULL,0xd766355442a41b25ULL,0x64efa6de778a4797ULL,0x2042170a7079adf4ULL, + 0x97091dcbd53c5c9dULL,0xf17624b6ac0a177bULL,0xb0f139752cfe2dffULL,0xc1a35c0a6c7a574eULL,0x227d314693e79987ULL,0x0575bf30e89cb80eULL,0x2f4e247f0d1883bbULL,0xebd512263274c3d0ULL, + 0xfea912baa5659ae8ULL,0x68363aba25e1a16eULL,0xb8842277752c41acULL,0xfe545c282897c3fcULL,0x2d36e9e7dc4c696bULL,0x5806244afba977c5ULL,0x85665e9be39508c1ULL,0xf720ee256d12597bULL, + 0x562e4cecc135b208ULL,0x74e1b2654783f47dULL,0x6d2a506c5a3f3b30ULL,0xecead9f4c16762fcULL,0xf29dd4b2e286e5b9ULL,0x1b0fadc083bb3c61ULL,0x7a75023e7fac29a4ULL,0xc086d5f1c9477fa3ULL, + 0xf4f876532de45068ULL,0x37c7a7e89e2e1f6eULL,0xd0825fa2a3584069ULL,0xaf2cea7c1727bf42ULL,0x0360a4fb9e4785a9ULL,0xe5fda49c27299f4aULL,0x48068e1371ac2f71ULL,0x83d0687b9077666fULL, + 0xa4a319acd837879fULL,0x6fc1b49eed6b67b0ULL,0xe395993332f1f3afULL,0x966742eb65432a2eULL,0x4b8dc9feb4966228ULL,0x96cc631243f43950ULL,0x12068859c9b731eeULL,0x7b948dc356f79968ULL, + 0x042c2af497e2feb4ULL,0xd36a42d7aebf7313ULL,0x49d2c9eb084ffdd7ULL,0x9f8aa54b2ef7c76aULL,0x9200b7ba09895e70ULL,0x3bd0c66fddb7fb58ULL,0x2d97d10878eb4cbbULL,0x2d431068d84bde31ULL, + 0x5e5db46acb66e132ULL,0xf1be963a0d925880ULL,0x944a70270317b9e2ULL,0xe266f95948603d48ULL,0x98db66735c208899ULL,0x90472447a2fb18a3ULL,0x8a966939777c619fULL,0x3798142a2a3be21bULL, + 0xe2f73c696755ff89ULL,0xdd3cf7e7473017e6ULL,0x8ef5689d3cf7600dULL,0x948dc4f8b1fc87b4ULL,0xd9e9fe814ea53299ULL,0x2d921ca298eb6028ULL,0xfaecedfd0c9803fcULL,0xf38ae8914d7b4745ULL, + 0x871514560f664534ULL,0x85ceae7c4b68f103ULL,0xac09c4ae65578ab9ULL,0x33ec6868f044b10cULL,0x6ac4832b3a8ec1f1ULL,0x5509d1285847d5efULL,0xf909604f763f1574ULL,0xb16c4303c32f63c4ULL, + 0xfd16847fdec67ef5ULL,0x742ee464233e76b7ULL,0x0b8e4134efc2b4c8ULL,0xca640b8642a3e521ULL,0x653a01908ceb6aa9ULL,0x313c300c547852d5ULL,0x24e4ab126b237af7ULL,0x2ba901628bb47af8ULL, + 0x00467bc58cce08b5ULL,0xb636458c7f178d55ULL,0xc5748baea677d806ULL,0x2763a387dfa394ebULL,0xa12b448a7d3cebb6ULL,0xe7adda3e6f20d850ULL,0xf63ebce51558462cULL,0x58b36143620088a8ULL, + 0xa9d89488a059c142ULL,0x6f5ae714ff0b9346ULL,0x068f237d16fb3664ULL,0x5853e4c4363186acULL,0xe2d87d2363c52f98ULL,0x2ec4a76681828876ULL,0x47b864fae14e7b1cULL,0x0c0bc0e569192408ULL, + 0x624d60492ed22e91ULL,0x6fdfe0b56f072822ULL,0xeeca111539ce2271ULL,0x98100a4fdb01614fULL,0xb6b0daa2a35c628fULL,0xb6f94d2ec87e9a47ULL,0xc67732591d57d9ceULL,0xf70bfeec03884a7bULL, + 0x4ff23ffd248a7d06ULL,0x80c5bfb4878873faULL,0xb7d9ad9005745981ULL,0x179c85db3db01994ULL,0xba41b06261a6966cULL,0x4d82d052eadce5a8ULL,0x9e91cd3ba5e6a318ULL,0x47795f4f95b2dda0ULL, + 0x1ee426ccd5cd79bfULL,0x0032940b946c6e18ULL,0x1b1e8ae057477f58ULL,0xe94f7d346d823278ULL,0xc747cb96782ba21aULL,0xc5254469f72b33a5ULL,0x772ef6dec7f80c81ULL,0xd73acbfe2cd9e6b5ULL, + 0x283c7513caa76097ULL,0x0a624fa936c83906ULL,0x6b20afec715af2c7ULL,0x4b969974eba78bfdULL,0x220755ccd921d60eULL,0x9b944e107baeca13ULL,0x04819d515ded93d4ULL,0x9bbff86e6dddfd27ULL, + 0x21950b421ff6acd3ULL,0xffe7048453dc6909ULL,0xff4cd0b228766127ULL,0xabdbe6084fb7db2bULL,0x837c92285e1109e8ULL,0x26147d27f4645b5aULL,0x4d78f592f7818ed8ULL,0xd394077ef247fa36ULL, + 0x508cec1c3b3f64c9ULL,0xe20bc0ba1e5edf3fULL,0xda1deb852f4318d4ULL,0xd20ebe0d5c3fa443ULL,0x370b4ea773241ea3ULL,0x61f1511c5e1a5f65ULL,0x99a5e23d82681c62ULL,0xd731e383a2f54c2dULL, + 0x97359638546c4d8dULL,0x5f9c3fc492f24679ULL,0x912e8beda8c8acd9ULL,0xec3a318d306634b0ULL,0x80167f41c31cb264ULL,0x3db82f6f522113f2ULL,0xb155bcd2dcafe197ULL,0xfba1da5943465283ULL, + 0x258bbbf9e7305683ULL,0x31eea5bf07ef5be6ULL,0x0deb0e4a46c814c1ULL,0x5cee8449a7b730ddULL,0xeab495c5a0182bdeULL,0xee759f879e27a6b4ULL,0xc2cf6a6880e518caULL,0x25e8013ff14cf3f4ULL, + 0x3ec832e77acaca28ULL,0x1bfeea57c7385b29ULL,0x068212e3fd1eaf38ULL,0xc13298306acf8cccULL,0xb909f2db2aac9e59ULL,0x5748060db661782aULL,0xc5ab2632c79b7a01ULL,0xda44c6c600017626ULL, + 0x69d44ed65c46aa8eULL,0x2100d5d3a8d063d1ULL,0xcb9727eaa2d17c36ULL,0x4c2bab1b8add53b7ULL,0xa084e90c15426704ULL,0x778afcd3a837ebeaULL,0x6651f7017ce477f8ULL,0xa062499846fb7a8bULL, + 0x3667eb1a7f4c04ccULL,0x59556621a9404f84ULL,0x71cdf6537eceb50aULL,0x994a44a69b8335faULL,0xd7faf819dbeb9b69ULL,0x473c5680eed4350dULL,0xb6658466da44bba2ULL,0x0d1bc780872bdbf3ULL, + 0xb8d3d9319ff91fe5ULL,0x039c4800f0518eedULL,0x95c376329182cb26ULL,0x0763a43482fc568dULL,0x707c04d5383e76baULL,0xac98b930824e8197ULL,0x92bf7c8f91230de0ULL,0x90876a0140959b70ULL, +}; diff --git a/cbits/s2n/COMMIT b/cbits/s2n/COMMIT new file mode 100644 --- /dev/null +++ b/cbits/s2n/COMMIT @@ -0,0 +1,1 @@ +62ec77dc6c2c8cc4c48c768f5f785240b55a47bf diff --git a/cbits/s2n/LICENSE b/cbits/s2n/LICENSE new file mode 100644 --- /dev/null +++ b/cbits/s2n/LICENSE @@ -0,0 +1,222 @@ +SPDX-License-Identifier: Apache-2.0 OR ISC or MIT-0 + + +Apache 2.0 license +------------------------------------- + + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + +ISC license +------------------------------------- + +Copyright Amazon.com, Inc. or its affiliates. + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted, provided that the above +copyright notice and this permission notice appear in all copies. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + + +MIT-0 license +------------------------------------- + +Copyright 2021-2024 Amazon.com, Inc. or its affiliates. + +Permission is hereby granted, free of charge, to any person obtaining a +copy of this software and associated documentation files (the "Software"), +to deal in the Software without restriction, including without limitation +the rights to use, copy, modify, merge, publish, distribute, sublicense, +and/or sell copies of the Software, and to permit persons to whom the +Software is furnished to do so. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. diff --git a/cbits/s2n/README.md b/cbits/s2n/README.md new file mode 100644 --- /dev/null +++ b/cbits/s2n/README.md @@ -0,0 +1,184 @@ +# s2n-bignum + +Assembly for the NIST prime curves from AWS's +[s2n-bignum](https://github.com/awslabs/s2n-bignum), vendored here. + +`COMMIT` holds the upstream revision these files came from, and `import.sh` +fetches them again. The files are unmodified: the choice between the two +variants of each routine is made in crypton's own C, not by editing theirs. + +## Why + +crypton's P-256 is portable C, and on the two architectures s2n-bignum +covers, hand-written assembly beats it by a lot. Measured against crypton's +own code in the same process, agreeing with it on every scalar tried: + +| | crypton | s2n-bignum | +| --- | ---: | ---: | +| Apple M4 | 52.9 us | **20.6** | +| x86-64 with ADX (EPYC 9V74) | 175.9 | **54.5** | +| x86-64, ADX not advertised | 156.1 | **59.4** | + +The third row is the `_alt` path, which is what crypton picks where `CPUID` +does not report ADX. It was measured on a KVM guest whose `CPUID` says so +while the host underneath runs ADX instructions anyway, so it says what the +two implementations cost relative to each other on that path, not what an +actual pre-Broadwell part would do. + +Each routine also carries a machine-checked proof in HOL-Light that it +computes what it says, and is written in a constant-time style. + +## Licence + +`Apache-2.0 OR ISC OR MIT-0`, one of which is on every file and all three in +`LICENSE`. crypton is BSD-3, so it takes these under **ISC** -- the MIT-0 +option would do as well, and the Apache one is the reason OpenSSL's and +BoringSSL's `ecp_nistz256`, which is Apache-2.0 only, cannot be used here. + +## Which variant + +Both forms of each routine are vendored, because which one is faster is not +the same question on the two architectures: + +* **On ARM**, `_alt` is written for parts with high multiplier throughput, + which is what Apple silicon is, and it wins there by 30-40%. The plain + form is for parts that pipeline `UMULH` less well. There is no feature bit + for this, so the choice is made at compile time on `__APPLE__`. +* **On x86-64**, the plain form uses `MULX`, `ADCX` and `ADOX`, and `_alt` is + the fallback for processors without them. That *is* a feature bit, so the + choice is made at run time, from `crypton_x86_simd_features()`. + +## RSA + +`crypton_powm.c`'s modular exponentiation also borrows from here, but only its +innermost step and only on x86-64: `bignum_kmul_16_32`, `bignum_ksqr_16_32`, +`bignum_kmul_32_64`, `bignum_ksqr_32_64` and `bignum_emontredc_8n` replace the +C's Montgomery multiplication and square, and at RSA-3072's 24 limbs the square +is built from `bignum_sqr_8_16` and `bignum_mul_8_16`. The window, the table and its masked +scan are crypton's throughout. Sixteen limbs is 1024 bits and thirty-two is +2048: the halves a CRT exponentiation works in for RSA-2048 and RSA-4096, and +the whole thing without CRT. The reduction wants ADX, so the choice is made at +run time like the other x86-64 ones. + +It is twice the C, because the C cannot form the two carry chains `ADCX` and +`ADOX` give. Measured on an EPYC 7763 at 1024 bits: + +| | C | s2n-bignum | +| --- | ---: | ---: | +| multiplication | 0.4832 us | **0.2479** | +| square | 0.3984 | **0.1994** | + +### The Montgomery constant + +`bignum_montifier` is vendored for **both** architectures. It is the one +routine here that is not about a particular curve or a particular size: every +modular exponentiation and every setup in `crypton_ecc.c` starts by computing +R^2 mod m, and this computes it. crypton's C climbs to it by a +binary ladder of Montgomery squarings -- log2 of the modulus' bits of them, +eleven at 2048 -- and the assembly is between two and three and a half times +faster, the gap being the carry chains again: + +| | C | s2n-bignum | +| --- | ---: | ---: | +| M4, 1024 bits | 1.853 us | **0.771** | +| M4, 2048 bits | 5.835 | **2.407** | +| an Intel x86-64, 1024 bits | 6.145 | **1.995** | +| an Intel x86-64, 2048 bits | 23.332 | **6.852** | + +A CRT signature computes it twice, so an RSA-2048 private operation on the +EPYC 7763 goes from 613.5 to 606.4 microseconds -- a little over one per +cent, which is all a constant computed twice out of some two and a half +thousand multiplications can be. It is worth much more than that to anything +whose exponent is short, which is why it is here: see `crypton_powm.c`. + +The two agree bit for bit; `cbits/tests/powm` requires it at six widths, with +moduli that fill their limbs and moduli that do not, and a build without the +assembly (`-DCRYPTON_NO_MONTIFIER`) is what it is checked against. + +**None of the five above is vendored for AArch64**, where they measure level +with the C. That took three attempts to establish, and the first two were +wrong in opposite directions: a reading when the x86-64 routines went in put +the C 5% ahead, and one on 2026-09-26 put the assembly 4% ahead. Both were +wall-clock times on a machine with other work on it, where a swing of that size +says nothing. Measured by thread CPU time, taking the best of twenty-five +batches and alternating the two binaries with the order swapped, one RSA-2048 +CRT private operation is 598.9 us through the C and 597.7 through the assembly: +two tenths of a per cent, which is nothing. The two agree on 200 random cases +at 1024 and 2048 bits, so both were computing the same thing. Five files for +nothing is not a trade, so the C stays. + +The gap to OpenSSL on Apple silicon -- about half its speed -- is assembly and +not the C, and no portable C closes it. BearSSL's `i62`, which is as far as +portable C is known to go -- 62-bit limbs in 64-bit words, so that a +multiply-accumulate fits an `__int128` with no carry chain at all, and a +five-bit window against crypton's four -- was built and measured the same way +on the same machine: 626.4 us against crypton's C at 668.0, a tie. + +`bignum_emontredc_8n_cdiff` was tried too and is not the answer either: it +wants a precomputation this test did not give it, and was slower besides. What +the window is worth, and why it is four and not five, is in +`cbits/crypton_powm.c` beside the constant. + +## Windows + +x86-64 Windows takes the assembly as every other x86-64 does. Two things +stood in the way, and neither is fixed in a vendored file: + +* **The ELF directives.** Each file ends its symbols with `.type` and + `.size`, which the COFF assemblers refuse -- mingw's gas says "junk at end + of line" after a warning, clang says "expected absolute expression". + There is nothing in a PE object for them to say, so on `_WIN32` they are + emptied, as upstream already does for Mach-O. +* **The calling convention.** Each file carries a prologue that moves + Microsoft's argument registers into System V's, under `WINDOWS_ABI`, which + upstream leaves to the build to define. On `_WIN32` it is defined. + +Both are done in `x86_att/_internal_s2n_bignum_x86_att.h`, which is crypton's +and not upstream's. Every vendored file includes that name in quotes, and a +quoted include looks first beside the file that asks for it, so this is the +one each gets; it hands on to upstream's in `include/` by a relative path. +**It must not be removed as an upstream file that went missing**: without it +Windows does not assemble. On Linux and macOS it changes nothing -- the +objects are byte for byte what they were without it. + +Not `#include_next`, which is the obvious way to hand on: clang, finding this +header beside the file that asked for it, starts that search from the top +and finds this header again, without end. + +Windows on ARM keeps the C. The AArch64 files have the same directives and +would want the same header, and there is no runner to check one. + +## What is not here + +s2n-bignum has only x86-64 and AArch64, so crypton's C stays and is what +every other architecture uses. + +Its `generic` directory holds a complete variable-size bignum library on both +architectures -- `bignum_montmul`, `bignum_montredc`, `bignum_emontredc`, +`bignum_modinv` and a whole `bignum_modexp` -- and only `bignum_montifier` is +taken from it. **The rest was measured on an M4 and is slower than what is +here**, which is why it stays out; they agree with crypton's answers in every +case first, so both were computing the same thing. + +| nanoseconds, AArch64 | C | s2n-bignum | +| --- | ---: | ---: | +| Montgomery multiply, 1024 bits | **171.7** | 227.9 | +| Montgomery multiply, 2048 bits | **649.6** | 870.3 | +| Montgomery square, 1024 bits | **157.3** | 202.6 | +| Montgomery square, 2048 bits | **539.2** | 870.8 | + +That is the inline assembly above doing its job: four limbs to an iteration +with two carry chains, where a size-parametric routine takes them one at a +time. `bignum_modexp`, which would have replaced `crypton_powm_sec` outright, +loses by more again -- 421.0 microseconds against 246.4 at 1024 bits and 3578 +against 1534 at 2048 -- because it has that multiplication inside it and no +window, so it pays about twice the multiplications as well. + +The README there says every function is written in a constant-time style, with +one named exception that is none of these, though it also says that unlike +functional correctness this is not machine-checked. + +`p384_montjscalarmul` and `p521_jscalarmul` +have no affine wrapper upstream; crypton's is in `cbits/p256/p256_s2n.c`'s +sibling for those curves. There is no fixed-base routine for P-384 or +P-521 at all, so signing on those curves keeps crypton's comb. diff --git a/cbits/s2n/arm/bignum_deamont_p384.S b/cbits/s2n/arm/bignum_deamont_p384.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_deamont_p384.S @@ -0,0 +1,151 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Convert from almost-Montgomery form, z := (x / 2^384) mod p_384 +// Input x[6]; output z[6] +// +// extern void bignum_deamont_p384(uint64_t z[static 6], +// const uint64_t x[static 6]); +// +// Convert a 6-digit bignum x out of its (optionally almost) Montgomery form, +// "almost" meaning any 6-digit input will work, with no range restriction. +// +// Standard ARM ABI: X0 = z, X1 = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_deamont_p384) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_deamont_p384) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_deamont_p384) + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_deamont_p384_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_deamont_p384_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_deamont_p384_alt) + .text + .balign 4 + +// --------------------------------------------------------------------------- +// Core one-step "short" Montgomery reduction macro. Takes input in +// [d5;d4;d3;d2;d1;d0] and returns result in [d6;d5;d4;d3;d2;d1], +// adding to the existing contents of [d5;d4;d3;d2;d1]. It is fine +// for d6 to be the same register as d0. +// +// We want to add (2^384 - 2^128 - 2^96 + 2^32 - 1) * w +// where w = [d0 + (d0<<32)] mod 2^64 +// --------------------------------------------------------------------------- + +#define montreds(d6,d5,d4,d3,d2,d1,d0, t3,t2,t1) \ +/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64 */ \ +/* Recycle d0 (which we know gets implicitly cancelled) to store it */ \ + lsl t1, d0, #32 __LF \ + add d0, t1, d0 __LF \ +/* Now let [t2;t1] = 2^64 * w - w + w_hi where w_hi = floor(w/2^32) */ \ +/* We need to subtract 2^32 * this, and we can ignore its lower 32 */ \ +/* bits since by design it will cancel anyway; we only need the w_hi */ \ +/* part to get the carry propagation going. */ \ + lsr t1, d0, #32 __LF \ + subs t1, t1, d0 __LF \ + sbc t2, d0, xzr __LF \ +/* Now select in t1 the field to subtract from d1 */ \ + extr t1, t2, t1, #32 __LF \ +/* And now get the terms to subtract from d2 and d3 */ \ + lsr t2, t2, #32 __LF \ + adds t2, t2, d0 __LF \ + adc t3, xzr, xzr __LF \ +/* Do the subtraction of that portion */ \ + subs d1, d1, t1 __LF \ + sbcs d2, d2, t2 __LF \ + sbcs d3, d3, t3 __LF \ + sbcs d4, d4, xzr __LF \ + sbcs d5, d5, xzr __LF \ +/* Now effectively add 2^384 * w by taking d0 as the input for last sbc */ \ + sbc d6, d0, xzr + +// Input parameters + +#define z x0 +#define x x1 + +// Rotating registers for the intermediate windows + +#define d0 x2 +#define d1 x3 +#define d2 x4 +#define d3 x5 +#define d4 x6 +#define d5 x7 + +// Other temporaries + +#define u x8 +#define v x9 +#define w x10 + +S2N_BN_SYMBOL(bignum_deamont_p384): + +S2N_BN_SYMBOL(bignum_deamont_p384_alt): + CFI_START + +// Set up an initial window with the input x and an extra leading zero + + ldp d0, d1, [x] + ldp d2, d3, [x, #16] + ldp d4, d5, [x, #32] + +// Systematically scroll left doing 1-step reductions + + montreds(d0,d5,d4,d3,d2,d1,d0, u,v,w) + + montreds(d1,d0,d5,d4,d3,d2,d1, u,v,w) + + montreds(d2,d1,d0,d5,d4,d3,d2, u,v,w) + + montreds(d3,d2,d1,d0,d5,d4,d3, u,v,w) + + montreds(d4,d3,d2,d1,d0,d5,d4, u,v,w) + + montreds(d5,d4,d3,d2,d1,d0,d5, u,v,w) + +// Now compare end result in [d5;d4;d3;d2;d1;d0] = dd with p_384 by *adding* +// 2^384 - p_384 = [0;0;0;w;v;u]. This will set CF if +// dd + (2^384 - p_384) >= 2^384, hence iff dd >= p_384 + + mov u, #0xffffffff00000001 + mov v, #0x00000000ffffffff + mov w, #0x0000000000000001 + + adds xzr, d0, u + adcs xzr, d1, v + adcs xzr, d2, w + adcs xzr, d3, xzr + adcs xzr, d4, xzr + adcs xzr, d5, xzr + +// Convert the condition dd >= p_384 into a bitmask in w and do a masked +// subtraction of p_384, via a masked addition of 2^384 - p_384: + + csetm w, cs + and u, u, w + adds d0, d0, u + and v, v, w + adcs d1, d1, v + and w, w, #1 + adcs d2, d2, w + adcs d3, d3, xzr + adcs d4, d4, xzr + adc d5, d5, xzr + +// Store it back + + stp d0, d1, [z] + stp d2, d3, [z, #16] + stp d4, d5, [z, #32] + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_deamont_p384) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/bignum_demont_p256.S b/cbits/s2n/arm/bignum_demont_p256.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_demont_p256.S @@ -0,0 +1,99 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Convert from Montgomery form z := (x / 2^256) mod p_256, assuming x reduced +// Input x[4]; output z[4] +// +// extern void bignum_demont_p256(uint64_t z[static 4], +// const uint64_t x[static 4]); +// +// This assumes the input is < p_256 for correctness. If this is not the case, +// use the variant "bignum_deamont_p256" instead. +// +// Standard ARM ABI: X0 = z, X1 = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_demont_p256) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_demont_p256) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_demont_p256) + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_demont_p256_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_demont_p256_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_demont_p256_alt) + .text + .balign 4 + +// --------------------------------------------------------------------------- +// Core one-step "short" Montgomery reduction macro. Takes input in +// [d3;d2;d1;d0] and returns result in [d4;d3;d2;d1], adding to the +// existing contents of [d3;d2;d1] and generating d4 from zero, re-using +// d0 as a temporary internally together with t0, t1 and t2. +// It is fine for d4 to be the same register as d0, and it often is. +// --------------------------------------------------------------------------- + +#define montreds(d4,d3,d2,d1,d0, t2,t1,t0) \ +/* Let w = d0, the original word we use as offset; d0 gets recycled */ \ +/* First let [t2;t1] = 2^32 * w */ \ +/* then let [d0;t0] = (2^64 - 2^32 + 1) * w (overwrite old d0) */ \ + lsl t1, d0, #32 __LF \ + subs t0, d0, t1 __LF \ + lsr t2, d0, #32 __LF \ + sbc d0, d0, t2 __LF \ +/* Hence [d4;..;d1] := [d3;d2;d1;0] + (2^256 - 2^224 + 2^192 + 2^96) * w */ \ + adds d1, d1, t1 __LF \ + adcs d2, d2, t2 __LF \ + adcs d3, d3, t0 __LF \ + adc d4, d0, xzr + +// Input parameters + +#define z x0 +#define x x1 + +// Rotating registers for the intermediate windows (with repetitions) + +#define d0 x2 +#define d1 x3 +#define d2 x4 +#define d3 x5 + +// Other temporaries + +#define u x6 +#define v x7 +#define w x8 + +S2N_BN_SYMBOL(bignum_demont_p256): + +S2N_BN_SYMBOL(bignum_demont_p256_alt): + CFI_START + +// Set up an initial window with the input x and an extra leading zero + + ldp d0, d1, [x] + ldp d2, d3, [x, #16] + +// Systematically scroll left doing 1-step reductions + + montreds(d0,d3,d2,d1,d0, u,v,w) + + montreds(d1,d0,d3,d2,d1, u,v,w) + + montreds(d2,d1,d0,d3,d2, u,v,w) + + montreds(d3,d2,d1,d0,d3, u,v,w) + +// Write back result + + stp d0, d1, [z] + stp d2, d3, [z, #16] + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_demont_p256) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/bignum_inv_p521.S b/cbits/s2n/arm/bignum_inv_p521.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_inv_p521.S @@ -0,0 +1,1701 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Modular inverse modulo p_521 = 2^521 - 1 +// Input x[9]; output z[9] +// +// extern void bignum_inv_p521(uint64_t z[static 9],const uint64_t x[static 9]); +// +// Assuming the 9-digit input x is coprime to p_521, i.e. is not divisible +// by it, returns z < p_521 such that x * z == 1 (mod p_521). Note that +// x does not need to be reduced modulo p_521, but the output always is. +// +// Standard ARM ABI: X0 = z, X1 = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_inv_p521) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_inv_p521) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_inv_p521) + + .text + .balign 4 + +// Size in bytes of a 64-bit word + +#define N 8 + +// Used for the return pointer + +#define res x20 + +// Loop counter and d = 2 * delta value for divstep + +#define i x21 +#define d x22 + +// Registers used for matrix element magnitudes and signs + +#define m00 x10 +#define m01 x11 +#define m10 x12 +#define m11 x13 +#define s00 x14 +#define s01 x15 +#define s10 x16 +#define s11 x17 + +// Initial carries for combinations + +#define car0 x9 +#define car1 x19 + +// Input and output, plain registers treated according to pattern + +#define reg0 x0, #0 +#define reg1 x1, #0 +#define reg2 x2, #0 +#define reg3 x3, #0 +#define reg4 x4, #0 + +#define x x1, #0 +#define z x0, #0 + +// Pointer-offset pairs for temporaries on stack + +#define f sp, #0 +#define g sp, #(9*N) +#define u sp, #(18*N) +#define v sp, #(27*N) + +// Total size to reserve on the stack + +#define NSPACE 36*N + +// Very similar to a subroutine call to the s2n-bignum word_divstep59. +// But different in register usage and returning the final matrix in +// registers as follows +// +// [ m00 m01] +// [ m10 m11] + +#define divstep59() \ + and x4, x2, #0xfffff __LF \ + orr x4, x4, #0xfffffe0000000000 __LF \ + and x5, x3, #0xfffff __LF \ + orr x5, x5, #0xc000000000000000 __LF \ + tst x5, #0x1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + asr x5, x5, #1 __LF \ + add x8, x4, #0x100, lsl #12 __LF \ + sbfx x8, x8, #21, #21 __LF \ + mov x11, #0x100000 __LF \ + add x11, x11, x11, lsl #21 __LF \ + add x9, x4, x11 __LF \ + asr x9, x9, #42 __LF \ + add x10, x5, #0x100, lsl #12 __LF \ + sbfx x10, x10, #21, #21 __LF \ + add x11, x5, x11 __LF \ + asr x11, x11, #42 __LF \ + mul x6, x8, x2 __LF \ + mul x7, x9, x3 __LF \ + mul x2, x10, x2 __LF \ + mul x3, x11, x3 __LF \ + add x4, x6, x7 __LF \ + add x5, x2, x3 __LF \ + asr x2, x4, #20 __LF \ + asr x3, x5, #20 __LF \ + and x4, x2, #0xfffff __LF \ + orr x4, x4, #0xfffffe0000000000 __LF \ + and x5, x3, #0xfffff __LF \ + orr x5, x5, #0xc000000000000000 __LF \ + tst x5, #0x1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + asr x5, x5, #1 __LF \ + add x12, x4, #0x100, lsl #12 __LF \ + sbfx x12, x12, #21, #21 __LF \ + mov x15, #0x100000 __LF \ + add x15, x15, x15, lsl #21 __LF \ + add x13, x4, x15 __LF \ + asr x13, x13, #42 __LF \ + add x14, x5, #0x100, lsl #12 __LF \ + sbfx x14, x14, #21, #21 __LF \ + add x15, x5, x15 __LF \ + asr x15, x15, #42 __LF \ + mul x6, x12, x2 __LF \ + mul x7, x13, x3 __LF \ + mul x2, x14, x2 __LF \ + mul x3, x15, x3 __LF \ + add x4, x6, x7 __LF \ + add x5, x2, x3 __LF \ + asr x2, x4, #20 __LF \ + asr x3, x5, #20 __LF \ + and x4, x2, #0xfffff __LF \ + orr x4, x4, #0xfffffe0000000000 __LF \ + and x5, x3, #0xfffff __LF \ + orr x5, x5, #0xc000000000000000 __LF \ + tst x5, #0x1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + mul x2, x12, x8 __LF \ + mul x3, x12, x9 __LF \ + mul x6, x14, x8 __LF \ + mul x7, x14, x9 __LF \ + madd x8, x13, x10, x2 __LF \ + madd x9, x13, x11, x3 __LF \ + madd x16, x15, x10, x6 __LF \ + madd x17, x15, x11, x7 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + asr x5, x5, #1 __LF \ + add x12, x4, #0x100, lsl #12 __LF \ + sbfx x12, x12, #22, #21 __LF \ + mov x15, #0x100000 __LF \ + add x15, x15, x15, lsl #21 __LF \ + add x13, x4, x15 __LF \ + asr x13, x13, #43 __LF \ + add x14, x5, #0x100, lsl #12 __LF \ + sbfx x14, x14, #22, #21 __LF \ + add x15, x5, x15 __LF \ + asr x15, x15, #43 __LF \ + mneg x2, x12, x8 __LF \ + mneg x3, x12, x9 __LF \ + mneg x4, x14, x8 __LF \ + mneg x5, x14, x9 __LF \ + msub m00, x13, x16, x2 __LF \ + msub m01, x13, x17, x3 __LF \ + msub m10, x15, x16, x4 __LF \ + msub m11, x15, x17, x5 + +// Loading large constants + +#define movbig(nn,n3,n2,n1,n0) \ + movz nn, n0 __LF \ + movk nn, n1, lsl #16 __LF \ + movk nn, n2, lsl #32 __LF \ + movk nn, n3, lsl #48 + +S2N_BN_SYMBOL(bignum_inv_p521): + CFI_START + +// Save registers and make room for temporaries + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_DEC_SP(NSPACE) + +// Save the return pointer for the end so we can overwrite x0 later + + mov res, x0 + +// Copy the prime p_521 = 2^521 - 1 into the f variable + + mov x10, #0xFFFFFFFFFFFFFFFF + stp x10, x10, [f] + stp x10, x10, [f+16] + stp x10, x10, [f+32] + stp x10, x10, [f+48] + mov x11, #0x1FF + str x11, [f+64] + +// Copy the input into the g variable, but reduce it strictly mod p_521 +// so that g <= f as assumed in the bound proof. This code fragment is +// very similar to bignum_mod_p521_9 complete with carry condensation. + + ldr x8, [x1, #64] + lsr x9, x8, #9 + + subs xzr, xzr, xzr + ldp x10, x11, [x1] + adcs xzr, x10, x9 + adcs xzr, x11, xzr + ldp x12, x13, [x1, #16] + and x7, x12, x13 + adcs xzr, x7, xzr + ldp x14, x15, [x1, #32] + and x7, x14, x15 + adcs xzr, x7, xzr + ldp x16, x17, [x1, #48] + and x7, x16, x17 + adcs xzr, x7, xzr + orr x7, x8, #~0x1FF + adcs x7, x7, xzr + + adcs x10, x10, x9 + adcs x11, x11, xzr + adcs x12, x12, xzr + adcs x13, x13, xzr + adcs x14, x14, xzr + adcs x15, x15, xzr + adcs x16, x16, xzr + adcs x17, x17, xzr + adc x8, x8, xzr + and x8, x8, #0x1FF + + stp x10, x11, [g] + stp x12, x13, [g+16] + stp x14, x15, [g+32] + stp x16, x17, [g+48] + str x8, [g+64] + +// Also maintain weakly reduced < 2*p_521 vector [u,v] such that +// [f,g] == x * 2^{1239-59*i} * [u,v] (mod p_521) +// starting with [p_521,x] == x * 2^{1239-59*0} * [0,2^-1239] (mod p_521) +// Note that because (2^{a+521} == 2^a) (mod p_521) we simply have +// (2^-1239 == 2^324) (mod p_521) so the constant initializer is simple. +// +// Based on the standard divstep bound, for inputs <= 2^b we need at least +// n >= (9437 * b + 1) / 4096. Since b is 521, that means 1201 iterations. +// Since we package divstep in multiples of 59 bits, we do 21 blocks of 59 +// making *1239* total. (With a bit more effort we could avoid the full 59 +// divsteps and use a shorter tail computation, but we keep it simple.) +// Hence, after the 21st iteration we have [f,g] == x * [u,v] and since +// |f| = 1 we get the modular inverse from u by flipping its sign with f. + + stp xzr, xzr, [u] + stp xzr, xzr, [u+16] + stp xzr, xzr, [u+32] + stp xzr, xzr, [u+48] + str xzr, [u+64] + + mov x10, #16 + stp xzr, xzr, [v] + stp xzr, xzr, [v+16] + stp xzr, x10, [v+32] + stp xzr, xzr, [v+48] + str xzr, [v+64] + +// Start of main loop. We jump into the middle so that the divstep +// portion is common to the special 21st iteration after a uniform +// first 20. + + mov i, #21 + mov d, #1 + b Lbignum_inv_p521_midloop + +Lbignum_inv_p521_loop: + +// Separate the matrix elements into sign-magnitude pairs + + cmp m00, xzr + csetm s00, mi + cneg m00, m00, mi + + cmp m01, xzr + csetm s01, mi + cneg m01, m01, mi + + cmp m10, xzr + csetm s10, mi + cneg m10, m10, mi + + cmp m11, xzr + csetm s11, mi + cneg m11, m11, mi + +// Adjust the initial values to allow for complement instead of negation +// This initial offset is the same for [f,g] and [u,v] compositions. +// Save it in stable registers for the [u,v] part and do [f,g] first. + + and x0, m00, s00 + and x1, m01, s01 + add car0, x0, x1 + + and x0, m10, s10 + and x1, m11, s11 + add car1, x0, x1 + +// Now the computation of the updated f and g values. This maintains a +// 2-word carry between stages so we can conveniently insert the shift +// right by 59 before storing back, and not overwrite digits we need +// again of the old f and g values. +// +// Digit 0 of [f,g] + + ldr x7, [f] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x4, car0, x0 + adc x2, xzr, x1 + ldr x8, [g] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x4, x4, x0 + adc x2, x2, x1 + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x5, car1, x0 + adc x3, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x5, x5, x0 + adc x3, x3, x1 + +// Digit 1 of [f,g] + + ldr x7, [f+N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [g+N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x2, x2, x0 + adc x6, x6, x1 + extr x4, x2, x4, #59 + str x4, [f] + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x3, x3, x0 + adc x4, x4, x1 + extr x5, x3, x5, #59 + str x5, [g] + +// Digit 2 of [f,g] + + ldr x7, [f+2*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [g+2*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x6, x6, x0 + adc x5, x5, x1 + extr x2, x6, x2, #59 + str x2, [f+N] + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x4, x4, x0 + adc x2, x2, x1 + extr x3, x4, x3, #59 + str x3, [g+N] + +// Digit 3 of [f,g] + + ldr x7, [f+3*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x5, x5, x0 + adc x3, xzr, x1 + ldr x8, [g+3*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x5, x6, #59 + str x6, [f+2*N] + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x2, x2, x0 + adc x6, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x2, x2, x0 + adc x6, x6, x1 + extr x4, x2, x4, #59 + str x4, [g+2*N] + +// Digit 4 of [f,g] + + ldr x7, [f+4*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x3, x3, x0 + adc x4, xzr, x1 + ldr x8, [g+4*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x3, x3, x0 + adc x4, x4, x1 + extr x5, x3, x5, #59 + str x5, [f+3*N] + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x6, x6, x0 + adc x5, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x6, x6, x0 + adc x5, x5, x1 + extr x2, x6, x2, #59 + str x2, [g+3*N] + +// Digit 5 of [f,g] + + ldr x7, [f+5*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x4, x4, x0 + adc x2, xzr, x1 + ldr x8, [g+5*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x4, x4, x0 + adc x2, x2, x1 + extr x3, x4, x3, #59 + str x3, [f+4*N] + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x5, x5, x0 + adc x3, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x5, x6, #59 + str x6, [g+4*N] + +// Digit 6 of [f,g] + + ldr x7, [f+6*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [g+6*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x2, x2, x0 + adc x6, x6, x1 + extr x4, x2, x4, #59 + str x4, [f+5*N] + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x3, x3, x0 + adc x4, x4, x1 + extr x5, x3, x5, #59 + str x5, [g+5*N] + +// Digit 7 of [f,g] + + ldr x7, [f+7*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [g+7*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x6, x6, x0 + adc x5, x5, x1 + extr x2, x6, x2, #59 + str x2, [f+6*N] + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x4, x4, x0 + adc x2, x2, x1 + extr x3, x4, x3, #59 + str x3, [g+6*N] + +// Digits 8 and 9 of [f,g] + + ldr x7, [f+8*N] + eor x1, x7, s00 + asr x3, x1, #63 + and x3, x3, m00 + neg x3, x3 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [g+8*N] + eor x1, x8, s01 + asr x0, x1, #63 + and x0, x0, m01 + sub x3, x3, x0 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x5, x6, #59 + str x6, [f+7*N] + extr x5, x3, x5, #59 + str x5, [f+8*N] + + eor x1, x7, s10 + asr x5, x1, #63 + and x5, x5, m10 + neg x5, x5 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, s11 + asr x0, x1, #63 + and x0, x0, m11 + sub x5, x5, x0 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x2, x2, x0 + adc x5, x5, x1 + extr x4, x2, x4, #59 + str x4, [g+7*N] + extr x2, x5, x2, #59 + str x2, [g+8*N] + +// Now the computation of the updated u and v values and their +// modular reductions. A very similar accumulation except that +// the top words of u and v are unsigned and we don't shift. +// +// Digit 0 of [u,v] + + ldr x7, [u] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x4, car0, x0 + adc x2, xzr, x1 + ldr x8, [v] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x4, x4, x0 + str x4, [u] + adc x2, x2, x1 + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x5, car1, x0 + adc x3, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x5, x5, x0 + str x5, [v] + adc x3, x3, x1 + +// Digit 1 of [u,v] + + ldr x7, [u+N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [v+N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x2, x2, x0 + str x2, [u+N] + adc x6, x6, x1 + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x3, x3, x0 + str x3, [v+N] + adc x4, x4, x1 + +// Digit 2 of [u,v] + + ldr x7, [u+2*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [v+2*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x6, x6, x0 + str x6, [u+2*N] + adc x5, x5, x1 + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x4, x4, x0 + str x4, [v+2*N] + adc x2, x2, x1 + +// Digit 3 of [u,v] + + ldr x7, [u+3*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x5, x5, x0 + adc x3, xzr, x1 + ldr x8, [v+3*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x5, x5, x0 + str x5, [u+3*N] + adc x3, x3, x1 + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x2, x2, x0 + adc x6, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x2, x2, x0 + str x2, [v+3*N] + adc x6, x6, x1 + +// Digit 4 of [u,v] + + ldr x7, [u+4*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x3, x3, x0 + adc x4, xzr, x1 + ldr x8, [v+4*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x3, x3, x0 + str x3, [u+4*N] + adc x4, x4, x1 + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x6, x6, x0 + adc x5, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x6, x6, x0 + str x6, [v+4*N] + adc x5, x5, x1 + +// Digit 5 of [u,v] + + ldr x7, [u+5*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x4, x4, x0 + adc x2, xzr, x1 + ldr x8, [v+5*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x4, x4, x0 + str x4, [u+5*N] + adc x2, x2, x1 + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x5, x5, x0 + adc x3, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x5, x5, x0 + str x5, [v+5*N] + adc x3, x3, x1 + +// Digit 6 of [u,v] + + ldr x7, [u+6*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [v+6*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x2, x2, x0 + str x2, [u+6*N] + adc x6, x6, x1 + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x3, x3, x0 + str x3, [v+6*N] + adc x4, x4, x1 + +// Digit 7 of [u,v] + + ldr x7, [u+7*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [v+7*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x6, x6, x0 + str x6, [u+7*N] + adc x5, x5, x1 + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x4, x4, x0 + str x4, [v+7*N] + adc x2, x2, x1 + +// Digits 8 and 9 of u (top is unsigned) + + ldr x7, [u+8*N] + eor x1, x7, s00 + and x3, s00, m00 + neg x3, x3 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [v+8*N] + eor x1, x8, s01 + and x0, s01, m01 + sub x3, x3, x0 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x5, x5, x0 + adc x3, x3, x1 + +// Modular reduction of u, reloading as needed from u[0],...,u[7],x5,x3 + + extr x6, x3, x5, #9 + ldp x0, x1, [u] + add x6, x6, x3, asr #63 + sub x5, x5, x6, lsl #9 + adds x0, x0, x6 + asr x6, x6, #63 + adcs x1, x1, x6 + stp x0, x1, [u] + ldp x0, x1, [u+16] + adcs x0, x0, x6 + adcs x1, x1, x6 + stp x0, x1, [u+16] + ldp x0, x1, [u+32] + adcs x0, x0, x6 + adcs x1, x1, x6 + stp x0, x1, [u+32] + ldp x0, x1, [u+48] + adcs x0, x0, x6 + adcs x1, x1, x6 + stp x0, x1, [u+48] + adc x5, x5, x6 + str x5, [u+64] + +// Digits 8 and 9 of v (top is unsigned) + + eor x1, x7, s10 + and x5, s10, m10 + neg x5, x5 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, s11 + and x0, s11, m11 + sub x5, x5, x0 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x2, x2, x0 + adc x5, x5, x1 + +// Modular reduction of v, reloading as needed from v[0],...,v[7],x2,x5 + + extr x6, x5, x2, #9 + ldp x0, x1, [v] + add x6, x6, x5, asr #63 + sub x2, x2, x6, lsl #9 + adds x0, x0, x6 + asr x6, x6, #63 + adcs x1, x1, x6 + stp x0, x1, [v] + ldp x0, x1, [v+16] + adcs x0, x0, x6 + adcs x1, x1, x6 + stp x0, x1, [v+16] + ldp x0, x1, [v+32] + adcs x0, x0, x6 + adcs x1, x1, x6 + stp x0, x1, [v+32] + ldp x0, x1, [v+48] + adcs x0, x0, x6 + adcs x1, x1, x6 + stp x0, x1, [v+48] + adc x2, x2, x6 + str x2, [v+64] + +Lbignum_inv_p521_midloop: + + mov x1, d + ldr x2, [f] + ldr x3, [g] + divstep59() + mov d, x1 + +// Next iteration + + subs i, i, #1 + bne Lbignum_inv_p521_loop + +// The 21st and last iteration does not need anything except the +// u value and the sign of f; the latter can be obtained from the +// lowest word of f. So it's done differently from the main loop. +// Find the sign of the new f. For this we just need one digit +// since we know (for in-scope cases) that f is either +1 or -1. +// We don't explicitly shift right by 59 either, but looking at +// bit 63 (or any bit >= 60) of the unshifted result is enough +// to distinguish -1 from +1; this is then made into a mask. + + ldr x0, [f] + ldr x1, [g] + mul x0, x0, m00 + madd x1, x1, m01, x0 + asr x0, x1, #63 + +// Now separate out the matrix into sign-magnitude pairs +// and adjust each one based on the sign of f. +// +// Note that at this point we expect |f|=1 and we got its +// sign above, so then since [f,0] == x * [u,v] (mod p_521) +// we want to flip the sign of u according to that of f. + + cmp m00, xzr + csetm s00, mi + cneg m00, m00, mi + eor s00, s00, x0 + + cmp m01, xzr + csetm s01, mi + cneg m01, m01, mi + eor s01, s01, x0 + + cmp m10, xzr + csetm s10, mi + cneg m10, m10, mi + eor s10, s10, x0 + + cmp m11, xzr + csetm s11, mi + cneg m11, m11, mi + eor s11, s11, x0 + +// Adjust the initial value to allow for complement instead of negation + + and x0, m00, s00 + and x1, m01, s01 + add car0, x0, x1 + +// Digit 0 of [u] + + ldr x7, [u] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x4, car0, x0 + adc x2, xzr, x1 + ldr x8, [v] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x4, x4, x0 + str x4, [u] + adc x2, x2, x1 + +// Digit 1 of [u] + + ldr x7, [u+N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [v+N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x2, x2, x0 + str x2, [u+N] + adc x6, x6, x1 + +// Digit 2 of [u] + + ldr x7, [u+2*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [v+2*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x6, x6, x0 + str x6, [u+2*N] + adc x5, x5, x1 + +// Digit 3 of [u] + + ldr x7, [u+3*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x5, x5, x0 + adc x3, xzr, x1 + ldr x8, [v+3*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x5, x5, x0 + str x5, [u+3*N] + adc x3, x3, x1 + +// Digit 4 of [u] + + ldr x7, [u+4*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x3, x3, x0 + adc x4, xzr, x1 + ldr x8, [v+4*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x3, x3, x0 + str x3, [u+4*N] + adc x4, x4, x1 + +// Digit 5 of [u] + + ldr x7, [u+5*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x4, x4, x0 + adc x2, xzr, x1 + ldr x8, [v+5*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x4, x4, x0 + str x4, [u+5*N] + adc x2, x2, x1 + +// Digit 6 of [u] + + ldr x7, [u+6*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [v+6*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x2, x2, x0 + str x2, [u+6*N] + adc x6, x6, x1 + +// Digit 7 of [u] + + ldr x7, [u+7*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [v+7*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x6, x6, x0 + str x6, [u+7*N] + adc x5, x5, x1 + +// Digits 8 and 9 of u (top is unsigned) + + ldr x7, [u+8*N] + eor x1, x7, s00 + and x3, s00, m00 + neg x3, x3 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [v+8*N] + eor x1, x8, s01 + and x0, s01, m01 + sub x3, x3, x0 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x5, x5, x0 + adc x3, x3, x1 + +// Modular reduction of u, reloading as needed from u[0],...,u[7],x5,x3 + + extr x6, x3, x5, #9 + ldp x10, x11, [u] + add x6, x6, x3, asr #63 + sub x5, x5, x6, lsl #9 + adds x10, x10, x6 + asr x6, x6, #63 + adcs x11, x11, x6 + ldp x12, x13, [u+16] + adcs x12, x12, x6 + adcs x13, x13, x6 + ldp x14, x15, [u+32] + adcs x14, x14, x6 + adcs x15, x15, x6 + ldp x16, x17, [u+48] + adcs x16, x16, x6 + adcs x17, x17, x6 + adc x19, x5, x6 + +// Further strict reduction ready for the output, which just means +// a conditional subtraction of p_521 + + subs x0, x10, #-1 + adcs x1, x11, xzr + adcs x2, x12, xzr + adcs x3, x13, xzr + adcs x4, x14, xzr + adcs x5, x15, xzr + adcs x6, x16, xzr + adcs x7, x17, xzr + mov x8, #0x1FF + sbcs x8, x19, x8 + + csel x0, x0, x10, cs + csel x1, x1, x11, cs + csel x2, x2, x12, cs + csel x3, x3, x13, cs + csel x4, x4, x14, cs + csel x5, x5, x15, cs + csel x6, x6, x16, cs + csel x7, x7, x17, cs + csel x8, x8, x19, cs + +// Store it back to the final output + + stp x0, x1, [res] + stp x2, x3, [res, #16] + stp x4, x5, [res, #32] + stp x6, x7, [res, #48] + str x8, [res, #64] + +// Restore stack and registers + + CFI_INC_SP(NSPACE) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_inv_p521) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/bignum_modinv.S b/cbits/s2n/arm/bignum_modinv.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_modinv.S @@ -0,0 +1,613 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Invert modulo m, z = (1/a) mod b, assuming b is an odd number > 1, coprime a +// Inputs a[k], b[k]; output z[k]; temporary buffer t[>=3*k] +// +// extern void bignum_modinv(uint64_t k, uint64_t *z, const uint64_t *a, +// const uint64_t *b, uint64_t *t); +// +// k-digit (digit=64 bits) "z := a^-1 mod b" (modular inverse of a modulo b) +// using t as a temporary buffer (t at least 3*k words = 24*k bytes), and +// assuming that a and b are coprime *and* that b is an odd number > 1. +// +// Standard ARM ABI: X0 = k, X1 = z, X2 = a, X3 = b, X4 = t +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_modinv) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_modinv) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_modinv) + .text + .balign 4 + +// We get CHUNKSIZE bits per outer iteration, 64 minus a few for proxy errors + +#define CHUNKSIZE 58 + +// Pervasive variables + +#define k x0 +#define z x1 +#define b x3 +#define w x4 + +// This one is recycled after initial copying in of a as outer loop counter + +#define a x2 +#define t x2 + +// Additional variables; later ones are currently rather high regs + +#define l x5 + +#define m x21 +#define n x22 + +// The matrix of update factors to apply to m and n +// Also used a couple of additional temporary variables for the swapping loop +// Also used as an extra down-counter in corrective negation loops + +#define m_m x6 +#define m_n x7 +#define n_m x8 +#define n_n x9 + +#define j x6 + +// General temporary variables and loop counters + +#define i x10 +#define t1 x11 +#define t2 x12 + +// High and low proxies for the inner loop +// Then re-used for high and carry words during actual cross-multiplications + +#define m_hi x13 +#define n_hi x14 +#define m_lo x15 +#define n_lo x16 + +#define h1 x13 +#define h2 x14 +#define l1 x15 +#define l2 x16 + +#define c1 x17 +#define c2 x19 + +// Negated modular inverse for Montgomery + +#define v x20 + +// Some more intuitive names for temp regs in initial word-level negmodinv. +// These just use t1 and t2 again, though carefully since t1 = initial b[0] + +#define one t2 +#define e1 t2 +#define e2 t1 +#define e4 t2 +#define e8 t1 + +S2N_BN_SYMBOL(bignum_modinv): + CFI_START + +// We make use of registers beyond the modifiable + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + +// If k = 0 then do nothing (this is out of scope anyway) + + cbz k, Lbignum_modinv_end + +// Set up the additional two buffers m and n beyond w in temp space + + lsl i, k, #3 + add m, w, i + add n, m, i + +// Initialize the main buffers with their starting values: +// m = a, n = b, w = b (to be tweaked to b - 1) and z = 0 + + mov i, xzr +Lbignum_modinv_copyloop: + ldr t1, [a, i, lsl #3] + ldr t2, [b, i, lsl #3] + str t1, [m, i, lsl #3] + str t2, [n, i, lsl #3] + str t2, [w, i, lsl #3] + str xzr, [z, i, lsl #3] + add i, i, #1 + cmp i, k + bcc Lbignum_modinv_copyloop + +// Tweak down w to b - 1 (this crude approach is safe as b needs to be odd +// for it to be in scope). We have then established the congruence invariant: +// +// a * w == -m (mod b) +// a * z == n (mod b) +// +// This, with the bound w <= b and z <= b, is maintained round the outer loop + + ldr t1, [w] + sub t2, t1, #1 + str t2, [w] + +// Compute v = negated modular inverse of b mod 2^64, reusing t1 from above +// This is used for Montgomery reduction operations each time round the loop + + lsl v, t1, #2 + sub v, t1, v + eor v, v, #2 + mov one, #1 + madd e1, t1, v, one + mul e2, e1, e1 + madd v, e1, v, v + mul e4, e2, e2 + madd v, e2, v, v + mul e8, e4, e4 + madd v, e4, v, v + madd v, e8, v, v + +// Set up the outer loop count of 128 * k +// The invariant is that m * n < 2^t at all times. + + lsl t, k, #7 + +// Start of the main outer loop iterated t / CHUNKSIZE times + +Lbignum_modinv_outerloop: + +// We need only bother with sharper l = min k (ceil(t/64)) digits +// for the computations on m and n (but we still need k for w and z). +// Either both m and n fit in l digits, or m has become zero and so +// nothing happens in the loop anyway and this makes no difference. + + add i, t, #63 + lsr l, i, #6 + cmp l, k + csel l, k, l, cs + +// Select upper and lower proxies for both m and n to drive the inner +// loop. The lower proxies are simply the lowest digits themselves, +// m_lo = m[0] and n_lo = n[0], while the upper proxies are bitfields +// of the two inputs selected so their top bit (63) aligns with the +// most significant bit of *either* of the two inputs. + + mov h1, xzr // Previous high and low for m + mov l1, xzr + mov h2, xzr // Previous high and low for n + mov l2, xzr + mov c2, xzr // Mask flag: previous word of one was nonzero + // and in this case h1 and h2 are those words + mov i, xzr +Lbignum_modinv_toploop: + ldr t1, [m, i, lsl #3] + ldr t2, [n, i, lsl #3] + orr c1, t1, t2 + cmp c1, xzr + and c1, c2, h1 + csel l1, c1, l1, ne + and c1, c2, h2 + csel l2, c1, l2, ne + csel h1, t1, h1, ne + csel h2, t2, h2, ne + csetm c2, ne + add i, i, #1 + cmp i, l + bcc Lbignum_modinv_toploop + + orr t1, h1, h2 + clz t2, t1 + negs c1, t2 + lsl h1, h1, t2 + csel l1, l1, xzr, ne + lsl h2, h2, t2 + csel l2, l2, xzr, ne + lsr l1, l1, c1 + lsr l2, l2, c1 + orr m_hi, h1, l1 + orr n_hi, h2, l2 + + ldr m_lo, [m] + ldr n_lo, [n] + +// Now the inner loop, with i as loop counter from CHUNKSIZE down. +// This records a matrix of updates to apply to the initial +// values of m and n with, at stage j: +// +// sgn * m' = (m_m * m - m_n * n) / 2^j +// -sgn * n' = (n_m * m - n_n * n) / 2^j +// +// where "sgn" is either +1 or -1, and we lose track of which except +// that both instance above are the same. This throwing away the sign +// costs nothing (since we have to correct in general anyway because +// of the proxied comparison) and makes things a bit simpler. But it +// is simply the parity of the number of times the first condition, +// used as the swapping criterion, fires in this loop. + + mov m_m, #1 + mov m_n, xzr + mov n_m, xzr + mov n_n, #1 + + mov i, #CHUNKSIZE + +// Conceptually in the inner loop we follow these steps: +// +// * If m_lo is odd and m_hi < n_hi, then swap the four pairs +// (m_hi,n_hi); (m_lo,n_lo); (m_m,n_m); (m_n,n_n) +// +// * Now, if m_lo is odd (old or new, doesn't matter as initial n_lo is odd) +// m_hi := m_hi - n_hi, m_lo := m_lo - n_lo +// m_m := m_m + n_m, m_n := m_n + n_n +// +// * Halve and double them +// m_hi := m_hi / 2, m_lo := m_lo / 2 +// n_m := n_m * 2, n_n := n_n * 2 +// +// The actual computation computes updates before actually swapping and +// then corrects as needed. It also maintains the invariant ~ZF <=> odd(m_lo), +// since it seems to reduce the dependent latency. Set that up first. + + ands xzr, m_lo, #1 + +Lbignum_modinv_innerloop: + +// At the start of the loop ~ZF <=> m_lo is odd; mask values accordingly +// Set the flags for m_hi - [~ZF] * n_hi so we know to flip things. + + csel t1, n_hi, xzr, ne + csel t2, n_lo, xzr, ne + csel c1, n_m, xzr, ne + csel c2, n_n, xzr, ne + ccmp m_hi, n_hi, #0x2, ne + +// Compute subtractive updates, trivial in the case ZF <=> even(m_lo). + + sub t1, m_hi, t1 + sub t2, m_lo, t2 + +// If the subtraction borrows, swap things appropriately, negating where +// we've already subtracted so things are as if we actually swapped first. + + csel n_hi, n_hi, m_hi, cs + cneg t1, t1, cc + csel n_lo, n_lo, m_lo, cs + cneg m_lo, t2, cc + csel n_m, n_m, m_m, cs + csel n_n, n_n, m_n, cs + +// Update and shift while setting oddness flag for next iteration +// We look at bit 1 of t2 (m_lo before possible negation), which is +// safe because it is even. + + ands xzr, t2, #2 + add m_m, m_m, c1 + add m_n, m_n, c2 + lsr m_hi, t1, #1 + lsr m_lo, m_lo, #1 + add n_m, n_m, n_m + add n_n, n_n, n_n + +// Next iteration; don't disturb the flags since they are used at entry + + sub i, i, #1 + cbnz i, Lbignum_modinv_innerloop + +// Apply the update to w and z, using addition in this case, and also take +// the chance to shift an additional 6 = 64-CHUNKSIZE bits to be ready for a +// Montgomery multiplication. Because we know that m_m + m_n <= 2^58 and +// w, z <= b < 2^{64k}, we know that both of these fit in k+1 words. +// We do this before the m-n update to allow us to play with c1 and c2 here. +// +// h1::w = 2^6 * (m_m * w + m_n * z) +// h2::z = 2^6 * (n_m * w + n_n * z) +// +// with c1 and c2 recording previous words for the shifting part + + mov h1, xzr + mov h2, xzr + mov c1, xzr + mov c2, xzr + + mov i, xzr +Lbignum_modinv_congloop: + ldr t1, [w, i, lsl #3] + ldr t2, [z, i, lsl #3] + + mul l1, m_m, t1 + mul l2, m_n, t2 + adds l1, l1, h1 + umulh h1, m_m, t1 + adc h1, h1, xzr + adds l1, l1, l2 + extr c1, l1, c1, #CHUNKSIZE + str c1, [w, i, lsl #3] + mov c1, l1 + umulh l1, m_n, t2 + adc h1, h1, l1 + + mul l1, n_m, t1 + mul l2, n_n, t2 + adds l1, l1, h2 + umulh h2, n_m, t1 + adc h2, h2, xzr + adds l1, l1, l2 + extr c2, l1, c2, #CHUNKSIZE + str c2, [z, i, lsl #3] + mov c2, l1 + umulh l1, n_n, t2 + adc h2, h2, l1 + + add i, i, #1 + cmp i, k + bcc Lbignum_modinv_congloop + + extr h1, h1, c1, #CHUNKSIZE + extr h2, h2, c2, #CHUNKSIZE + +// Do a Montgomery reduction of h1::w + + ldr t1, [w] + mul c1, t1, v + ldr t2, [b] + mul l1, c1, t2 + umulh l2, c1, t2 + adds t1, t1, l1 // Will be zero but want the carry + + mov i, #1 + sub t1, k, #1 + cbz t1, Lbignum_modinv_wmontend +Lbignum_modinv_wmontloop: + ldr t1, [b, i, lsl #3] + ldr t2, [w, i, lsl #3] + mul l1, c1, t1 + adcs t2, t2, l2 + umulh l2, c1, t1 + adc l2, l2, xzr + adds t2, t2, l1 + sub l1, i, #1 + str t2, [w, l1, lsl #3] + add i, i, #1 + sub t1, i, k + cbnz t1, Lbignum_modinv_wmontloop +Lbignum_modinv_wmontend: + adcs l2, l2, h1 + adc h1, xzr, xzr + sub l1, i, #1 + str l2, [w, l1, lsl #3] + + subs i, xzr, xzr +Lbignum_modinv_wcmploop: + ldr t1, [w, i, lsl #3] + ldr t2, [b, i, lsl #3] + sbcs xzr, t1, t2 + add i, i, #1 + sub t1, i, k + cbnz t1, Lbignum_modinv_wcmploop + + sbcs xzr, h1, xzr + csetm h1, cs + + subs i, xzr, xzr +Lbignum_modinv_wcorrloop: + ldr t1, [w, i, lsl #3] + ldr t2, [b, i, lsl #3] + and t2, t2, h1 + sbcs t1, t1, t2 + str t1, [w, i, lsl #3] + add i, i, #1 + sub t1, i, k + cbnz t1, Lbignum_modinv_wcorrloop + +// Do a Montgomery reduction of h2::z + + ldr t1, [z] + mul c1, t1, v + ldr t2, [b] + mul l1, c1, t2 + umulh l2, c1, t2 + adds t1, t1, l1 // Will be zero but want the carry + + mov i, #1 + sub t1, k, #1 + cbz t1, Lbignum_modinv_zmontend +Lbignum_modinv_zmontloop: + ldr t1, [b, i, lsl #3] + ldr t2, [z, i, lsl #3] + mul l1, c1, t1 + adcs t2, t2, l2 + umulh l2, c1, t1 + adc l2, l2, xzr + adds t2, t2, l1 + sub l1, i, #1 + str t2, [z, l1, lsl #3] + add i, i, #1 + sub t1, i, k + cbnz t1, Lbignum_modinv_zmontloop +Lbignum_modinv_zmontend: + adcs l2, l2, h2 + adc h2, xzr, xzr + sub l1, i, #1 + str l2, [z, l1, lsl #3] + + subs i, xzr, xzr +Lbignum_modinv_zcmploop: + ldr t1, [z, i, lsl #3] + ldr t2, [b, i, lsl #3] + sbcs xzr, t1, t2 + add i, i, #1 + sub t1, i, k + cbnz t1, Lbignum_modinv_zcmploop + + sbcs xzr, h2, xzr + csetm h2, cs + + subs i, xzr, xzr +Lbignum_modinv_zcorrloop: + ldr t1, [z, i, lsl #3] + ldr t2, [b, i, lsl #3] + and t2, t2, h2 + sbcs t1, t1, t2 + str t1, [z, i, lsl #3] + add i, i, #1 + sub t1, i, k + cbnz t1, Lbignum_modinv_zcorrloop + +// Now actually compute the updates to m and n corresponding to the matrix, +// and correct the signs if they have gone negative. First we compute the +// (k+1)-sized updates with the following invariant (here c1 and c2 are in +// fact carry bitmasks, either 0 or -1): +// +// c1::h1::m = m_m * m - m_n * n +// c2::h2::n = n_m * m - n_n * n + + mov h1, xzr + mov h2, xzr + mov c1, xzr + mov c2, xzr + mov i, xzr +Lbignum_modinv_crossloop: + ldr t1, [m, i, lsl #3] + ldr t2, [n, i, lsl #3] + + mul l1, m_m, t1 + mul l2, m_n, t2 + adds l1, l1, h1 + umulh h1, m_m, t1 + adc h1, h1, xzr + subs l1, l1, l2 + str l1, [m, i, lsl #3] + umulh l1, m_n, t2 + sub c1, l1, c1 + sbcs h1, h1, c1 + csetm c1, cc + + mul l1, n_m, t1 + mul l2, n_n, t2 + adds l1, l1, h2 + umulh h2, n_m, t1 + adc h2, h2, xzr + subs l1, l1, l2 + str l1, [n, i, lsl #3] + umulh l1, n_n, t2 + sub c2, l1, c2 + sbcs h2, h2, c2 + csetm c2, cc + + add i, i, #1 + cmp i, l + bcc Lbignum_modinv_crossloop + +// Write back m optionally negated and shifted right CHUNKSIZE bits + + adds xzr, c1, c1 + + ldr l1, [m] + mov i, xzr + sub j, l, #1 + cbz j, Lbignum_modinv_negskip1 + +Lbignum_modinv_negloop1: + add t1, i, #8 + ldr t2, [m, t1] + extr l1, t2, l1, #CHUNKSIZE + eor l1, l1, c1 + adcs l1, l1, xzr + str l1, [m, i] + mov l1, t2 + add i, i, #8 + sub j, j, #1 + cbnz j, Lbignum_modinv_negloop1 +Lbignum_modinv_negskip1: + extr l1, h1, l1, #CHUNKSIZE + eor l1, l1, c1 + adcs l1, l1, xzr + str l1, [m, i] + +// Write back n optionally negated and shifted right CHUNKSIZE bits + + adds xzr, c2, c2 + + ldr l1, [n] + mov i, xzr + sub j, l, #1 + cbz j, Lbignum_modinv_negskip2 +Lbignum_modinv_negloop2: + add t1, i, #8 + ldr t2, [n, t1] + extr l1, t2, l1, #CHUNKSIZE + eor l1, l1, c2 + adcs l1, l1, xzr + str l1, [n, i] + mov l1, t2 + add i, i, #8 + sub j, j, #1 + cbnz j, Lbignum_modinv_negloop2 +Lbignum_modinv_negskip2: + extr l1, h2, l1, #CHUNKSIZE + eor l1, l1, c2 + adcs l1, l1, xzr + str l1, [n, i] + +// Finally, use the signs c1 and c2 to do optional modular negations of +// w and z respectively, flipping c2 to make signs work. We don't make +// any checks for zero values, but we certainly retain w <= b and z <= b. +// This is enough for the Montgomery step in the next iteration to give +// strict reduction w < b amd z < b, and anyway when we terminate we +// could not have z = b since it violates the coprimality assumption for +// in-scope cases. + + mov i, xzr + adds xzr, c1, c1 +Lbignum_modinv_wfliploop: + ldr t1, [b, i, lsl #3] + ldr t2, [w, i, lsl #3] + and t1, t1, c1 + eor t2, t2, c1 + adcs t1, t1, t2 + str t1, [w, i, lsl #3] + add i, i, #1 + sub t1, i, k + cbnz t1, Lbignum_modinv_wfliploop + + mvn c2, c2 + + mov i, xzr + adds xzr, c2, c2 +Lbignum_modinv_zfliploop: + ldr t1, [b, i, lsl #3] + ldr t2, [z, i, lsl #3] + and t1, t1, c2 + eor t2, t2, c2 + adcs t1, t1, t2 + str t1, [z, i, lsl #3] + add i, i, #1 + sub t1, i, k + cbnz t1, Lbignum_modinv_zfliploop + +// End of main loop. We can stop if t' <= 0 since then m * n < 2^0, which +// since n is odd and m and n are coprime (in the in-scope cases) means +// m = 0, n = 1 and hence from the congruence invariant a * z == 1 (mod b). +// Moreover we do in fact need to maintain strictly t > 0 in the main loop, +// or the computation of the optimized digit bound l could collapse to 0. + + subs t, t, #CHUNKSIZE + bhi Lbignum_modinv_outerloop + +Lbignum_modinv_end: + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_modinv) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/bignum_montifier.S b/cbits/s2n/arm/bignum_montifier.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_montifier.S @@ -0,0 +1,462 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Compute "montification" constant z := 2^{128k} mod m +// Input m[k]; output z[k]; temporary buffer t[>=k] +// +// extern void bignum_montifier(uint64_t k, uint64_t *z, const uint64_t *m, +// uint64_t *t); +// +// The last argument points to a temporary buffer t that should have size >= k. +// This is called "montifier" because given any other k-digit number x, +// whether or not it's reduced modulo m, it can be mapped to its Montgomery +// representation (2^{64k} * x) mod m just by Montgomery multiplication by z. +// +// Standard ARM ABI: X0 = k, X1 = z, X2 = m, X3 = t +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montifier) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montifier) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montifier) + .text + .balign 4 + +#define k x0 +#define z x1 +#define m x2 +#define t x3 + +// Some variables +// Modular inverse w is aliased to i, but we never use them together + +#define i x4 +#define w x4 +#define j x5 +#define h x6 +#define a x7 +#define l x8 +#define c x9 +#define b x10 +#define d x11 + +// Some aliases for the values b and d + +#define r x10 +#define q x11 + + +S2N_BN_SYMBOL(bignum_montifier): + CFI_START + +// If k = 0 the whole operation is trivial + + cbz k, Lbignum_montifier_end + +// Copy the input m into the temporary buffer t. The temporary register +// c matters since we want it to hold the highest digit, ready for the +// normalization phase. + + mov i, xzr +Lbignum_montifier_copyinloop: + ldr c, [m, i, lsl #3] + str c, [t, i, lsl #3] + add i, i, #1 + cmp i, k + bcc Lbignum_montifier_copyinloop + +// Do a rather stupid but constant-time digit normalization, conditionally +// shifting left (k-1) times based on whether the top word is zero. +// With careful binary striding this could be O(k*log(k)) instead of O(k^2) +// while still retaining the constant-time style. +// The "cmp c, xzr" sets the zeroness predicate (ZF) for the entire inner loop + + subs i, k, #1 + beq Lbignum_montifier_normalized +Lbignum_montifier_normloop: + mov j, xzr + cmp c, xzr + mov a, xzr +Lbignum_montifier_shufloop: + mov c, a + ldr a, [t, j, lsl #3] + csel c, c, a, eq + str c, [t, j, lsl #3] + add j, j, #1 + sub d, j, k + cbnz d, Lbignum_montifier_shufloop + subs i, i, #1 + bne Lbignum_montifier_normloop + +// We now have the top digit nonzero, assuming the input was nonzero, +// and as per the invariant of the loop above, c holds that digit. So +// now just count c's leading zeros and shift t bitwise that many bits. + +Lbignum_montifier_normalized: + clz c, c + + mov b, xzr + mov i, xzr + ands xzr, c, #63 + csetm l, ne + neg d, c +Lbignum_montifier_bitloop: + ldr j, [t, i, lsl #3] + lsl a, j, c + orr a, a, b + lsr b, j, d + and b, b, l + str a, [t, i, lsl #3] + add i, i, #1 + cmp i, k + bcc Lbignum_montifier_bitloop + +// Let h be the high word of n, which in all the in-scope cases is >= 2^63. +// Now successively form q = 2^i div h and r = 2^i mod h as i goes from +// 64 to 126. We avoid just using division out of constant-time concerns +// (at the least we would need to fix up h = 0 for out-of-scope inputs) and +// don't bother with Newton-Raphson, since this stupid simple loop doesn't +// contribute much of the overall runtime at typical sizes. + + sub h, k, #1 + ldr h, [t, h, lsl #3] + mov q, #1 + neg r, h + mov i, #62 +Lbignum_montifier_estloop: + add q, q, q + mov a, h + sub a, a, r + cmp r, a // CF <=> r >= h - r <=> 2 * r >= h + csetm a, cs + sub q, q, a + add r, r, r + and a, a, h + sub r, r, a + subs i, i, #1 + bne Lbignum_montifier_estloop + +// Strictly speaking the above loop doesn't quite give the true remainder +// and quotient in the special case r = h = 2^63, so fix it up. We get +// q = 2^63 - 1 and r = 2^63 and really want q = 2^63 and r = 0. This is +// supererogatory, because the main property of q used below still holds +// in this case unless the initial m = 1, and then anyway the overall +// specification (congruence modulo m) holds degenerately. But it seems +// nicer to get a "true" quotient and remainder. + + cmp r, h + csinc q, q, q, ne + +// So now we have q and r with 2^126 = q * h + r (imagining r = 0 in the +// fixed-up case above: note that we never actually use the computed +// value of r below and so didn't adjust it). And we can assume the ranges +// q <= 2^63 and r < h < 2^64. +// +// The idea is to use q as a first quotient estimate for a remainder +// of 2^{p+62} mod n, where p = 64 * k. We have, splitting n into the +// high and low parts h and l: +// +// 2^{p+62} - q * n = 2^{p+62} - q * (2^{p-64} * h + l) +// = 2^{p+62} - (2^{p-64} * (q * h) + q * l) +// = 2^{p+62} - 2^{p-64} * (2^126 - r) - q * l +// = 2^{p-64} * r - q * l +// +// Note that 2^{p-64} * r < 2^{p-64} * h <= n +// and also q * l < 2^63 * 2^{p-64} = 2^{p-1} <= n +// so |diff| = |2^{p-64} * r - q * l| < n. +// +// If in fact diff >= 0 then it is already 2^{p+62} mod n. +// otherwise diff + n is the right answer. +// +// To (maybe?) make the computation slightly easier we actually flip +// the sign and compute d = q * n - 2^{p+62}. Then the answer is either +// -d (when negative) or n - d; in either case we effectively negate d. +// This negating tweak in fact spoils the result for cases where +// 2^{p+62} mod n = 0, when we get n instead. However the only case +// where this can happen is m = 1, when the whole spec holds trivially, +// and actually the remainder of the logic below works anyway since +// the latter part of the code only needs a congruence for the k-digit +// result, not strict modular reduction (the doublings will maintain +// the non-strict inequality). + + mov c, xzr + adds i, xzr, xzr +Lbignum_montifier_mulloop: + ldr a, [t, i, lsl #3] + mul l, q, a + adcs l, l, c + umulh c, q, a + str l, [z, i, lsl #3] + add i, i, #1 + sub a, i, k + cbnz a, Lbignum_montifier_mulloop + + adc c, c, xzr + mov a, #0x4000000000000000 + subs c, c, a + csetm q, cs + +// Now do [c] * n - d for our final answer + + subs i, xzr, xzr +Lbignum_montifier_remloop: + ldr a, [t, i, lsl #3] + ldr b, [z, i, lsl #3] + and a, a, q + sbcs a, a, b + str a, [z, i, lsl #3] + add i, i, #1 + sub a, i, k + cbnz a, Lbignum_montifier_remloop + +// Now still need to do a couple of modular doublings to get us all the +// way up to 2^{p+64} == r from the initial 2^{p+62} == r (mod n). + + mov c, xzr + subs j, xzr, xzr +Lbignum_montifier_dubloop1: + ldr a, [z, j, lsl #3] + extr c, a, c, #63 + ldr b, [t, j, lsl #3] + sbcs c, c, b + str c, [z, j, lsl #3] + mov c, a + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_dubloop1 + lsr c, c, #63 + sbc c, c, xzr + adds j, xzr, xzr +Lbignum_montifier_corrloop1: + ldr a, [z, j, lsl #3] + ldr b, [t, j, lsl #3] + and b, b, c + adcs a, a, b + str a, [z, j, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_corrloop1 + +// This is not exactly the same: we also copy output to t giving the +// initialization t_1 = r == 2^{p+64} mod n for the main loop next. + + mov c, xzr + subs j, xzr, xzr +Lbignum_montifier_dubloop2: + ldr a, [z, j, lsl #3] + extr c, a, c, #63 + ldr b, [t, j, lsl #3] + sbcs c, c, b + str c, [z, j, lsl #3] + mov c, a + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_dubloop2 + lsr c, c, #63 + sbc c, c, xzr + adds j, xzr, xzr +Lbignum_montifier_corrloop2: + ldr a, [z, j, lsl #3] + ldr b, [t, j, lsl #3] + and b, b, c + adcs a, a, b + str a, [z, j, lsl #3] + str a, [t, j, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_corrloop2 + +// We then successively generate (k+1)-digit values satisfying +// t_i == 2^{p+64*i} mod n, each of which is stored in h::t. Finish +// initialization by zeroing h initially + + mov h, xzr + +// Then if t_i = 2^{p} * h + l +// we have t_{i+1} == 2^64 * t_i +// = (2^{p+64} * h) + (2^64 * l) +// == r * h + l<<64 +// Do this 2*k more times so we end up == 2^{192*k+64}, one more than we want +// +// Writing B = 2^{64k}, the possible correction of adding r, which for +// a (k+1)-digit result is equivalent to subtracting q = 2^{64*(k+1)} - r +// would give the overall worst-case value minus q of +// [ B * (B^k - 1) + (B - 1) * r ] - [B^{k+1} - r] +// = B * (r - 1) < B^{k+1} so we keep inside k+1 digits as required. +// +// This implementation makes the shift implicit by starting b with the +// "previous" digit (initially 0) to offset things by 1. + + add i, k, k +Lbignum_montifier_modloop: + mov j, xzr + mov b, xzr + adds c, xzr, xzr +Lbignum_montifier_cmaloop: + ldr a, [z, j, lsl #3] + mul l, h, a + adcs b, b, c + umulh c, h, a + adc c, c, xzr + adds l, b, l + ldr b, [t, j, lsl #3] + str l, [t, j, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_cmaloop + + adcs h, b, c + + csetm l, cs + + adds j, xzr, xzr +Lbignum_montifier_oaloop: + ldr a, [t, j, lsl #3] + ldr b, [z, j, lsl #3] + and b, b, l + adcs a, a, b + str a, [t, j, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_oaloop + adc h, h, xzr + + subs i, i, #1 + bne Lbignum_montifier_modloop + +// Compute the negated modular inverse w (same register as i, not used again). + + ldr a, [m] + lsl w, a, #2 + sub w, a, w + eor w, w, #2 + mov l, #1 + madd c, a, w, l + mul b, c, c + madd w, c, w, w + mul c, b, b + madd w, b, w, w + mul b, c, c + madd w, c, w, w + madd w, b, w, w + +// Now do one almost-Montgomery reduction w.r.t. the original m +// which lops off one 2^64 from the congruence and, with the usual +// almost-Montgomery correction, gets us back inside k digits for +// the end result. + + ldr b, [t] + mul d, b, w + + mul l, d, a + umulh c, d, a + mov j, #1 + sub a, k, #1 + adds xzr, b, l + cbz a, Lbignum_montifier_amontend + +Lbignum_montifier_amontloop: + ldr a, [m, j, lsl #3] + ldr b, [t, j, lsl #3] + mul l, d, a + adcs b, b, c + umulh c, d, a + adc c, c, xzr + adds b, b, l + sub a, j, #1 + str b, [t, a, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_amontloop +Lbignum_montifier_amontend: + adcs h, h, c + csetm l, cs + sub a, k, #1 + str h, [t, a, lsl #3] + + subs j, xzr, xzr +Lbignum_montifier_osloop: + ldr a, [t, j, lsl #3] + ldr b, [m, j, lsl #3] + and b, b, l + sbcs a, a, b + str a, [z, j, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_osloop + +// So far, the code(basically a variant of bignum_amontifier) has produced +// a k-digit value z == 2^{192k} (mod m), not necessarily fully reduced mod m. +// We now do a short Montgomery reduction (similar to bignum_demont) so that +// we achieve full reduction mod m while lopping 2^{64k} off the congruence. +// We recycle h as the somewhat strangely-named outer loop counter. + + mov h, k + +Lbignum_montifier_montouterloop: + ldr b, [z] + mul d, b, w + ldr a, [m] + mul l, d, a + umulh c, d, a + mov j, #1 + sub a, k, #1 + adds xzr, b, l + cbz a, Lbignum_montifier_montend +Lbignum_montifier_montloop: + ldr a, [m, j, lsl #3] + ldr b, [z, j, lsl #3] + mul l, d, a + adcs b, b, c + umulh c, d, a + adc c, c, xzr + adds b, b, l + sub a, j, #1 + str b, [z, a, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_montloop +Lbignum_montifier_montend: + adc c, c, xzr + sub a, k, #1 + str c, [z, a, lsl #3] + + subs h, h, #1 + bne Lbignum_montifier_montouterloop + +// Now do a comparison of z with m to set a final correction mask +// indicating that z >= m and so we need to subtract m. + + subs j, xzr, xzr +Lbignum_montifier_cmploop: + ldr a, [z, j, lsl #3] + ldr b, [m, j, lsl #3] + sbcs xzr, a, b + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_cmploop + csetm h, cs + +// Now do a masked subtraction of m for the final reduced result. + + subs j, xzr, xzr +Lbignum_montifier_corrloop: + ldr a, [z, j, lsl #3] + ldr b, [m, j, lsl #3] + and b, b, h + sbcs a, a, b + str a, [z, j, lsl #3] + add j, j, #1 + sub a, j, k + cbnz a, Lbignum_montifier_corrloop + +Lbignum_montifier_end: + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_montifier) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/bignum_montinv_p384.S b/cbits/s2n/arm/bignum_montinv_p384.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_montinv_p384.S @@ -0,0 +1,1493 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Montgomery inverse modulo p_384 = 2^384 - 2^128 - 2^96 + 2^32 - 1 +// Input x[6]; output z[6] +// +// extern void bignum_montinv_p384(uint64_t z[static 6], +// const uint64_t x[static 6]); +// +// If the 6-digit input x is coprime to p_384, i.e. is not divisible +// by it, returns z < p_384 such that x * z == 2^768 (mod p_384). This +// is effectively "Montgomery inverse" because if we consider x and z as +// Montgomery forms of X and Z, i.e. x == 2^384 * X and z == 2^384 * Z +// (both mod p_384) then X * Z == 1 (mod p_384). That is, this function +// gives the analog of the modular inverse bignum_inv_p384 but with both +// input and output in the Montgomery domain. Note that x does not need +// to be reduced modulo p_384, but the output always is. If the input +// is divisible (i.e. is 0 or p_384), then there can be no solution to +// the congruence x * z == 2^768 (mod p_384), and z = 0 is returned. +// +// Standard ARM ABI: X0 = z, X1 = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montinv_p384) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montinv_p384) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montinv_p384) + + .text + .balign 4 + +// Size in bytes of a 64-bit word + +#define N 8 + +// Used for the return pointer + +#define res x20 + +// Loop counter and d = 2 * delta value for divstep + +#define i x21 +#define d x22 + +// Registers used for matrix element magnitudes and signs + +#define m00 x10 +#define m01 x11 +#define m10 x12 +#define m11 x13 +#define s00 x14 +#define s01 x15 +#define s10 x16 +#define s11 x17 + +// Initial carries for combinations + +#define car0 x9 +#define car1 x19 + +// Input and output, plain registers treated according to pattern + +#define reg0 x0, #0 +#define reg1 x1, #0 +#define reg2 x2, #0 +#define reg3 x3, #0 +#define reg4 x4, #0 + +#define x x1, #0 +#define z x0, #0 + +// Pointer-offset pairs for temporaries on stack +// The u and v variables are 6 words each as expected, but the f and g +// variables are 8 words each -- they need to have at least one extra +// word for a sign word, and to preserve alignment we "round up" to 8. +// In fact, we currently keep an extra word in u and v as well. + +#define f sp, #0 +#define g sp, #(8*N) +#define u sp, #(16*N) +#define v sp, #(24*N) + +// Total size to reserve on the stack + +#define NSPACE 32*N + +// --------------------------------------------------------------------------- +// Core signed almost-Montgomery reduction macro. Takes input in +// [d6;d5;d4;d3;d2;d1;d0] and returns result in [d6;d5d4;d3;d2;d1], adding +// to the existing [d6;d5;d4;d3;d2;d1], and re-using d0 as a temporary +// internally as well as t0, t1, t2. This is almost-Montgomery, i.e. the +// result fits in 6 digits but is not necessarily strictly reduced mod p_384. +// --------------------------------------------------------------------------- + +#define amontred(d6,d5,d4,d3,d2,d1,d0, t3,t2,t1) \ +/* We only know the input is -2^444 < x < 2^444. To do traditional */ \ +/* unsigned Montgomery reduction, start by adding 2^61 * p_384. */ \ + mov t1, #0xe000000000000000 __LF \ + adds d0, d0, t1 __LF \ + mov t2, #0x000000001fffffff __LF \ + adcs d1, d1, t2 __LF \ + mov t3, #0xffffffffe0000000 __LF \ + bic t3, t3, #0x2000000000000000 __LF \ + adcs d2, d2, t3 __LF \ + sbcs d3, d3, xzr __LF \ + sbcs d4, d4, xzr __LF \ + sbcs d5, d5, xzr __LF \ + mov t1, #0x1fffffffffffffff __LF \ + adc d6, d6, t1 __LF \ +/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64 */ \ +/* Store it back into d0 since we no longer need that digit. */ \ + add d0, d0, d0, lsl #32 __LF \ +/* Now let [t3;t2;t1;-] = (2^384 - p_384) * w */ \ +/* We know the lowest word will cancel d0 so we don't need it */ \ + mov t1, #0xffffffff00000001 __LF \ + umulh t1, t1, d0 __LF \ + mov t2, #0x00000000ffffffff __LF \ + mul t3, t2, d0 __LF \ + umulh t2, t2, d0 __LF \ + adds t1, t1, t3 __LF \ + adcs t2, t2, d0 __LF \ + cset t3, cs __LF \ +/* Now x + p_384 * w = (x + 2^384 * w) - (2^384 - p_384) * w */ \ +/* We catch the net top carry from add-subtract in the digit d0 */ \ + adds d6, d6, d0 __LF \ + cset d0, cs __LF \ + subs d1, d1, t1 __LF \ + sbcs d2, d2, t2 __LF \ + sbcs d3, d3, t3 __LF \ + sbcs d4, d4, xzr __LF \ + sbcs d5, d5, xzr __LF \ + sbcs d6, d6, xzr __LF \ + sbcs d0, d0, xzr __LF \ +/* Now if d0 is nonzero we subtract p_384 (almost-Montgomery) */ \ + neg d0, d0 __LF \ + and t1, d0, #0x00000000ffffffff __LF \ + and t2, d0, #0xffffffff00000000 __LF \ + and t3, d0, #0xfffffffffffffffe __LF \ + subs d1, d1, t1 __LF \ + sbcs d2, d2, t2 __LF \ + sbcs d3, d3, t3 __LF \ + sbcs d4, d4, d0 __LF \ + sbcs d5, d5, d0 __LF \ + sbc d6, d6, d0 + +// Very similar to a subroutine call to the s2n-bignum word_divstep59. +// But different in register usage and returning the final matrix in +// registers as follows +// +// [ m00 m01] +// [ m10 m11] + +#define divstep59() \ + and x4, x2, #0xfffff __LF \ + orr x4, x4, #0xfffffe0000000000 __LF \ + and x5, x3, #0xfffff __LF \ + orr x5, x5, #0xc000000000000000 __LF \ + tst x5, #0x1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + asr x5, x5, #1 __LF \ + add x8, x4, #0x100, lsl #12 __LF \ + sbfx x8, x8, #21, #21 __LF \ + mov x11, #0x100000 __LF \ + add x11, x11, x11, lsl #21 __LF \ + add x9, x4, x11 __LF \ + asr x9, x9, #42 __LF \ + add x10, x5, #0x100, lsl #12 __LF \ + sbfx x10, x10, #21, #21 __LF \ + add x11, x5, x11 __LF \ + asr x11, x11, #42 __LF \ + mul x6, x8, x2 __LF \ + mul x7, x9, x3 __LF \ + mul x2, x10, x2 __LF \ + mul x3, x11, x3 __LF \ + add x4, x6, x7 __LF \ + add x5, x2, x3 __LF \ + asr x2, x4, #20 __LF \ + asr x3, x5, #20 __LF \ + and x4, x2, #0xfffff __LF \ + orr x4, x4, #0xfffffe0000000000 __LF \ + and x5, x3, #0xfffff __LF \ + orr x5, x5, #0xc000000000000000 __LF \ + tst x5, #0x1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + asr x5, x5, #1 __LF \ + add x12, x4, #0x100, lsl #12 __LF \ + sbfx x12, x12, #21, #21 __LF \ + mov x15, #0x100000 __LF \ + add x15, x15, x15, lsl #21 __LF \ + add x13, x4, x15 __LF \ + asr x13, x13, #42 __LF \ + add x14, x5, #0x100, lsl #12 __LF \ + sbfx x14, x14, #21, #21 __LF \ + add x15, x5, x15 __LF \ + asr x15, x15, #42 __LF \ + mul x6, x12, x2 __LF \ + mul x7, x13, x3 __LF \ + mul x2, x14, x2 __LF \ + mul x3, x15, x3 __LF \ + add x4, x6, x7 __LF \ + add x5, x2, x3 __LF \ + asr x2, x4, #20 __LF \ + asr x3, x5, #20 __LF \ + and x4, x2, #0xfffff __LF \ + orr x4, x4, #0xfffffe0000000000 __LF \ + and x5, x3, #0xfffff __LF \ + orr x5, x5, #0xc000000000000000 __LF \ + tst x5, #0x1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + mul x2, x12, x8 __LF \ + mul x3, x12, x9 __LF \ + mul x6, x14, x8 __LF \ + mul x7, x14, x9 __LF \ + madd x8, x13, x10, x2 __LF \ + madd x9, x13, x11, x3 __LF \ + madd x16, x15, x10, x6 __LF \ + madd x17, x15, x11, x7 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + tst x5, #0x2 __LF \ + asr x5, x5, #1 __LF \ + csel x6, x4, xzr, ne __LF \ + ccmp x1, xzr, #0x8, ne __LF \ + cneg x1, x1, ge __LF \ + cneg x6, x6, ge __LF \ + csel x4, x5, x4, ge __LF \ + add x5, x5, x6 __LF \ + add x1, x1, #0x2 __LF \ + asr x5, x5, #1 __LF \ + add x12, x4, #0x100, lsl #12 __LF \ + sbfx x12, x12, #22, #21 __LF \ + mov x15, #0x100000 __LF \ + add x15, x15, x15, lsl #21 __LF \ + add x13, x4, x15 __LF \ + asr x13, x13, #43 __LF \ + add x14, x5, #0x100, lsl #12 __LF \ + sbfx x14, x14, #22, #21 __LF \ + add x15, x5, x15 __LF \ + asr x15, x15, #43 __LF \ + mneg x2, x12, x8 __LF \ + mneg x3, x12, x9 __LF \ + mneg x4, x14, x8 __LF \ + mneg x5, x14, x9 __LF \ + msub m00, x13, x16, x2 __LF \ + msub m01, x13, x17, x3 __LF \ + msub m10, x15, x16, x4 __LF \ + msub m11, x15, x17, x5 + +S2N_BN_SYMBOL(bignum_montinv_p384): + CFI_START + +// Save registers and make room for temporaries + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_DEC_SP(NSPACE) + +// Save the return pointer for the end so we can overwrite x0 later + + mov res, x0 + +// Copy the prime and input into the main f and g variables respectively. +// Make sure x is reduced so that g <= f as assumed in the bound proof. + + mov x10, #0x00000000ffffffff + mov x11, #0xffffffff00000000 + mov x12, #0xfffffffffffffffe + mov x15, #0xffffffffffffffff + stp x10, x11, [f] + stp x12, x15, [f+2*N] + stp x15, x15, [f+4*N] + str xzr, [f+6*N] + + ldp x2, x3, [x1] + subs x10, x2, x10 + sbcs x11, x3, x11 + ldp x4, x5, [x1, #(2*N)] + sbcs x12, x4, x12 + sbcs x13, x5, x15 + ldp x6, x7, [x1, #(4*N)] + sbcs x14, x6, x15 + sbcs x15, x7, x15 + + csel x2, x2, x10, cc + csel x3, x3, x11, cc + csel x4, x4, x12, cc + csel x5, x5, x13, cc + csel x6, x6, x14, cc + csel x7, x7, x15, cc + + stp x2, x3, [g] + stp x4, x5, [g+2*N] + stp x6, x7, [g+4*N] + str xzr, [g+6*N] + +// Also maintain reduced < 2^384 vector [u,v] such that +// [f,g] == x * 2^{5*i-843} * [u,v] (mod p_384) +// starting with [p_384,x] == x * 2^{5*0-843} * [0,2^843] (mod p_384) +// The weird-looking 5*i modifications come in because we are doing +// 64-bit word-sized Montgomery reductions at each stage, which is +// 5 bits more than the 59-bit requirement to keep things stable. +// After the 15th and last iteration and sign adjustment, when +// f == 1 for in-scope cases, we have x * 2^{75-843} * u == 1, i.e. +// x * u == 2^768 as required. + + stp xzr, xzr, [u] + stp xzr, xzr, [u+2*N] + stp xzr, xzr, [u+4*N] + +// The starting constant 2^843 mod p_384 is +// 0x0000000000000800:00001000000007ff:fffff00000000000 +// :00001000000007ff:fffff00000000800:0000000000000000 +// where colons separate 64-bit subwords, least significant at the right. +// Not all of these are single loads on ARM so this is a bit dynamic + + mov x12, #0xfffff00000000000 + orr x10, x12, #0x0000000000000800 + stp xzr, x10, [v] + mov x11, #0x00000000000007ff + orr x11, x11, #0x0000100000000000 + stp x11, x12, [v+2*N] + mov x12, #0x0000000000000800 + stp x11, x12, [v+4*N] + +// Start of main loop. We jump into the middle so that the divstep +// portion is common to the special fifteenth iteration after a uniform +// first 14. + + mov i, #15 + mov d, #1 + b Lbignum_montinv_p384_midloop + +Lbignum_montinv_p384_loop: + +// Separate the matrix elements into sign-magnitude pairs + + cmp m00, xzr + csetm s00, mi + cneg m00, m00, mi + + cmp m01, xzr + csetm s01, mi + cneg m01, m01, mi + + cmp m10, xzr + csetm s10, mi + cneg m10, m10, mi + + cmp m11, xzr + csetm s11, mi + cneg m11, m11, mi + +// Adjust the initial values to allow for complement instead of negation +// This initial offset is the same for [f,g] and [u,v] compositions. +// Save it in stable registers for the [u,v] part and do [f,g] first. + + and x0, m00, s00 + and x1, m01, s01 + add car0, x0, x1 + + and x0, m10, s10 + and x1, m11, s11 + add car1, x0, x1 + +// Now the computation of the updated f and g values. This maintains a +// 2-word carry between stages so we can conveniently insert the shift +// right by 59 before storing back, and not overwrite digits we need +// again of the old f and g values. +// +// Digit 0 of [f,g] + + ldr x7, [f] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x4, car0, x0 + adc x2, xzr, x1 + ldr x8, [g] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x4, x4, x0 + adc x2, x2, x1 + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x5, car1, x0 + adc x3, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x5, x5, x0 + adc x3, x3, x1 + +// Digit 1 of [f,g] + + ldr x7, [f+N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [g+N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x2, x2, x0 + adc x6, x6, x1 + extr x4, x2, x4, #59 + str x4, [f] + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x3, x3, x0 + adc x4, x4, x1 + extr x5, x3, x5, #59 + str x5, [g] + +// Digit 2 of [f,g] + + ldr x7, [f+2*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [g+2*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x6, x6, x0 + adc x5, x5, x1 + extr x2, x6, x2, #59 + str x2, [f+N] + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x4, x4, x0 + adc x2, x2, x1 + extr x3, x4, x3, #59 + str x3, [g+N] + +// Digit 3 of [f,g] + + ldr x7, [f+3*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x5, x5, x0 + adc x3, xzr, x1 + ldr x8, [g+3*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x5, x6, #59 + str x6, [f+2*N] + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x2, x2, x0 + adc x6, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x2, x2, x0 + adc x6, x6, x1 + extr x4, x2, x4, #59 + str x4, [g+2*N] + +// Digit 4 of [f,g] + + ldr x7, [f+4*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x3, x3, x0 + adc x4, xzr, x1 + ldr x8, [g+4*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x3, x3, x0 + adc x4, x4, x1 + extr x5, x3, x5, #59 + str x5, [f+3*N] + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x6, x6, x0 + adc x5, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x6, x6, x0 + adc x5, x5, x1 + extr x2, x6, x2, #59 + str x2, [g+3*N] + +// Digits 5 and 6 of [f,g] + + ldr x7, [f+5*N] + eor x1, x7, s00 + ldr x23, [f+6*N] + eor x2, x23, s00 + and x2, x2, m00 + neg x2, x2 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x4, x4, x0 + adc x2, x2, x1 + ldr x8, [g+5*N] + eor x1, x8, s01 + ldr x24, [g+6*N] + eor x0, x24, s01 + and x0, x0, m01 + sub x2, x2, x0 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x4, x4, x0 + adc x2, x2, x1 + extr x3, x4, x3, #59 + str x3, [f+4*N] + extr x4, x2, x4, #59 + str x4, [f+5*N] + asr x2, x2, #59 + str x2, [f+6*N] + + eor x1, x7, s10 + eor x4, x23, s10 + and x4, x4, m10 + neg x4, x4 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x5, x5, x0 + adc x4, x4, x1 + eor x1, x8, s11 + eor x0, x24, s11 + and x0, x0, m11 + sub x4, x4, x0 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x5, x5, x0 + adc x4, x4, x1 + extr x6, x5, x6, #59 + str x6, [g+4*N] + extr x5, x4, x5, #59 + str x5, [g+5*N] + asr x4, x4, #59 + str x4, [g+6*N] + +// Now the computation of the updated u and v values and their +// Montgomery reductions. A very similar accumulation except that +// the top words of u and v are unsigned and we don't shift. +// +// Digit 0 of [u,v] + + ldr x7, [u] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x4, car0, x0 + adc x2, xzr, x1 + ldr x8, [v] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x4, x4, x0 + str x4, [u] + adc x2, x2, x1 + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x5, car1, x0 + adc x3, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x5, x5, x0 + str x5, [v] + adc x3, x3, x1 + +// Digit 1 of [u,v] + + ldr x7, [u+N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [v+N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x2, x2, x0 + str x2, [u+N] + adc x6, x6, x1 + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x3, x3, x0 + str x3, [v+N] + adc x4, x4, x1 + +// Digit 2 of [u,v] + + ldr x7, [u+2*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [v+2*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x6, x6, x0 + str x6, [u+2*N] + adc x5, x5, x1 + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x4, x4, x0 + str x4, [v+2*N] + adc x2, x2, x1 + +// Digit 3 of [u,v] + + ldr x7, [u+3*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x5, x5, x0 + adc x3, xzr, x1 + ldr x8, [v+3*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x5, x5, x0 + str x5, [u+3*N] + adc x3, x3, x1 + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x2, x2, x0 + adc x6, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x2, x2, x0 + str x2, [v+3*N] + adc x6, x6, x1 + +// Digit 4 of [u,v] + + ldr x7, [u+4*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x3, x3, x0 + adc x4, xzr, x1 + ldr x8, [v+4*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x3, x3, x0 + str x3, [u+4*N] + adc x4, x4, x1 + + eor x1, x7, s10 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x6, x6, x0 + adc x5, xzr, x1 + eor x1, x8, s11 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x6, x6, x0 + str x6, [v+4*N] + adc x5, x5, x1 + +// Digits 5 and 6 of [u,v] (top is unsigned) + + ldr x7, [u+5*N] + eor x1, x7, s00 + and x2, s00, m00 + neg x2, x2 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x4, x4, x0 + adc x2, x2, x1 + ldr x8, [v+5*N] + eor x1, x8, s01 + and x0, s01, m01 + sub x2, x2, x0 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x4, x4, x0 + str x4, [u+5*N] + adc x2, x2, x1 + str x2, [u+6*N] + + eor x1, x7, s10 + and x4, s10, m10 + neg x4, x4 + mul x0, x1, m10 + umulh x1, x1, m10 + adds x5, x5, x0 + adc x4, x4, x1 + eor x1, x8, s11 + and x0, s11, m11 + sub x4, x4, x0 + mul x0, x1, m11 + umulh x1, x1, m11 + adds x5, x5, x0 + str x5, [v+5*N] + adc x4, x4, x1 + str x4, [v+6*N] + +// Montgomery reduction of u + + ldp x0, x1, [u] + ldp x2, x3, [u+16] + ldp x4, x5, [u+32] + ldr x6, [u+48] + amontred(x6,x5,x4,x3,x2,x1,x0, x9,x8,x7) + stp x1, x2, [u] + stp x3, x4, [u+16] + stp x5, x6, [u+32] + +// Montgomery reduction of v + + ldp x0, x1, [v] + ldp x2, x3, [v+16] + ldp x4, x5, [v+32] + ldr x6, [v+48] + amontred(x6,x5,x4,x3,x2,x1,x0, x9,x8,x7) + stp x1, x2, [v] + stp x3, x4, [v+16] + stp x5, x6, [v+32] + +Lbignum_montinv_p384_midloop: + + mov x1, d + ldr x2, [f] + ldr x3, [g] + divstep59() + mov d, x1 + +// Next iteration + + subs i, i, #1 + bne Lbignum_montinv_p384_loop + +// The 15th and last iteration does not need anything except the +// u value and the sign of f; the latter can be obtained from the +// lowest word of f. So it's done differently from the main loop. +// Find the sign of the new f. For this we just need one digit +// since we know (for in-scope cases) that f is either +1 or -1. +// We don't explicitly shift right by 59 either, but looking at +// bit 63 (or any bit >= 60) of the unshifted result is enough +// to distinguish -1 from +1; this is then made into a mask. + + ldr x0, [f] + ldr x1, [g] + mul x0, x0, m00 + madd x1, x1, m01, x0 + asr x0, x1, #63 + +// Now separate out the matrix into sign-magnitude pairs +// and adjust each one based on the sign of f. +// +// Note that at this point we expect |f|=1 and we got its +// sign above, so then since [f,0] == x * 2^{-768} [u,v] (mod p_384) +// we want to flip the sign of u according to that of f. + + cmp m00, xzr + csetm s00, mi + cneg m00, m00, mi + eor s00, s00, x0 + + cmp m01, xzr + csetm s01, mi + cneg m01, m01, mi + eor s01, s01, x0 + + cmp m10, xzr + csetm s10, mi + cneg m10, m10, mi + eor s10, s10, x0 + + cmp m11, xzr + csetm s11, mi + cneg m11, m11, mi + eor s11, s11, x0 + +// Adjust the initial value to allow for complement instead of negation + + and x0, m00, s00 + and x1, m01, s01 + add car0, x0, x1 + +// Digit 0 of [u] + + ldr x7, [u] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x4, car0, x0 + adc x2, xzr, x1 + ldr x8, [v] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x4, x4, x0 + str x4, [u] + adc x2, x2, x1 + +// Digit 1 of [u] + + ldr x7, [u+N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [v+N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x2, x2, x0 + str x2, [u+N] + adc x6, x6, x1 + +// Digit 2 of [u] + + ldr x7, [u+2*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [v+2*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x6, x6, x0 + str x6, [u+2*N] + adc x5, x5, x1 + +// Digit 3 of [u] + + ldr x7, [u+3*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x5, x5, x0 + adc x3, xzr, x1 + ldr x8, [v+3*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x5, x5, x0 + str x5, [u+3*N] + adc x3, x3, x1 + +// Digit 4 of [u] + + ldr x7, [u+4*N] + eor x1, x7, s00 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x3, x3, x0 + adc x4, xzr, x1 + ldr x8, [v+4*N] + eor x1, x8, s01 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x3, x3, x0 + str x3, [u+4*N] + adc x4, x4, x1 + +// Digits 5 and 6 of [u] (top is unsigned) + + ldr x7, [u+5*N] + eor x1, x7, s00 + and x2, s00, m00 + neg x2, x2 + mul x0, x1, m00 + umulh x1, x1, m00 + adds x4, x4, x0 + adc x2, x2, x1 + ldr x8, [v+5*N] + eor x1, x8, s01 + and x0, s01, m01 + sub x2, x2, x0 + mul x0, x1, m01 + umulh x1, x1, m01 + adds x4, x4, x0 + str x4, [u+5*N] + adc x2, x2, x1 + str x2, [u+6*N] + +// Montgomery reduction of u. This needs to be strict not "almost" +// so it is followed by an optional subtraction of p_384 + + ldp x10, x0, [u] + ldp x1, x2, [u+16] + ldp x3, x4, [u+32] + ldr x5, [u+48] + amontred(x5,x4,x3,x2,x1,x0,x10, x9,x8,x7) + + mov x10, #0x00000000ffffffff + subs x10, x0, x10 + mov x11, #0xffffffff00000000 + sbcs x11, x1, x11 + mov x12, #0xfffffffffffffffe + sbcs x12, x2, x12 + mov x15, #0xffffffffffffffff + sbcs x13, x3, x15 + sbcs x14, x4, x15 + sbcs x15, x5, x15 + + csel x0, x0, x10, cc + csel x1, x1, x11, cc + csel x2, x2, x12, cc + csel x3, x3, x13, cc + csel x4, x4, x14, cc + csel x5, x5, x15, cc + +// Store it back to the final output + + stp x0, x1, [res] + stp x2, x3, [res, #16] + stp x4, x5, [res, #32] + +// Restore stack and registers + + CFI_INC_SP(NSPACE) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_montinv_p384) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/bignum_montmul_p384.S b/cbits/s2n/arm/bignum_montmul_p384.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_montmul_p384.S @@ -0,0 +1,891 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Montgomery multiply, z := (x * y / 2^384) mod p_384 +// Inputs x[6], y[6]; output z[6] +// +// extern void bignum_montmul_p384(uint64_t z[static 6], +// const uint64_t x[static 6], +// const uint64_t y[static 6]); +// +// Does z := (2^{-384} * x * y) mod p_384, assuming that the inputs x and y +// satisfy x * y <= 2^384 * p_384 (in particular this is true if we are in +// the "usual" case x < p_384 and y < p_384). +// +// Standard ARM ABI: X0 = z, X1 = x, X2 = y +// ---------------------------------------------------------------------------- + +// bignum_montmul_p384 is functionally equivalent to +// unopt/bignum_montmul_p384_base. +// It is written in a way that +// 1. A subset of scalar multiplications in bignum_montmul_p384 are carefully +// chosen and vectorized +// 2. The vectorized assembly is rescheduled using the SLOTHY superoptimizer. +// https://github.com/slothy-optimizer/slothy +// +// The output program of step 1. is as follows: +// +// stp x19, x20, [sp, #-16]! +// stp x21, x22, [sp, #-16]! +// stp x23, x24, [sp, #-16]! +// ldp x3, x21, [x1] +// ldr q30, [x1] +// ldp x8, x24, [x1, #16] +// ldp x5, x10, [x1, #32] +// ldp x13, x23, [x2] +// ldr q19, [x2] +// ldp x6, x14, [x2, #16] +// ldp x15, x17, [x2, #32] +// ldr q1, [x1, #32] +// ldr q28, [x2, #32] +// uzp1 v5.4S, v19.4S, v30.4S +// rev64 v19.4S, v19.4S +// uzp1 v0.4S, v30.4S, v30.4S +// mul v21.4S, v19.4S, v30.4S +// uaddlp v19.2D, v21.4S +// shl v19.2D, v19.2D, #32 +// umlal v19.2D, v0.2S, v5.2S +// mov x12, v19.d[0] +// mov x16, v19.d[1] +// mul x20, x8, x6 +// umulh x4, x3, x13 +// umulh x1, x21, x23 +// umulh x2, x8, x6 +// adds x4, x4, x16 +// adcs x19, x1, x20 +// adc x20, x2, xzr +// adds x11, x4, x12 +// adcs x16, x19, x4 +// adcs x1, x20, x19 +// adc x2, x20, xzr +// adds x7, x16, x12 +// adcs x4, x1, x4 +// adcs x9, x2, x19 +// adc x19, x20, xzr +// subs x2, x3, x21 +// cneg x20, x2, cc +// csetm x16, cc +// subs x2, x23, x13 +// cneg x2, x2, cc +// mul x1, x20, x2 +// umulh x2, x20, x2 +// cinv x16, x16, cc +// eor x1, x1, x16 +// eor x2, x2, x16 +// cmn x16, #0x1 +// adcs x11, x11, x1 +// adcs x7, x7, x2 +// adcs x4, x4, x16 +// adcs x9, x9, x16 +// adc x19, x19, x16 +// subs x2, x3, x8 +// cneg x20, x2, cc +// csetm x16, cc +// subs x2, x6, x13 +// cneg x2, x2, cc +// mul x1, x20, x2 +// umulh x2, x20, x2 +// cinv x16, x16, cc +// eor x1, x1, x16 +// eor x2, x2, x16 +// cmn x16, #0x1 +// adcs x7, x7, x1 +// adcs x4, x4, x2 +// adcs x9, x9, x16 +// adc x19, x19, x16 +// subs x2, x21, x8 +// cneg x20, x2, cc +// csetm x16, cc +// subs x2, x6, x23 +// cneg x2, x2, cc +// mul x1, x20, x2 +// umulh x2, x20, x2 +// cinv x16, x16, cc +// eor x1, x1, x16 +// eor x2, x2, x16 +// cmn x16, #0x1 +// adcs x4, x4, x1 +// adcs x20, x9, x2 +// adc x16, x19, x16 +// lsl x2, x12, #32 +// add x19, x2, x12 +// lsr x2, x19, #32 +// subs x1, x2, x19 +// sbc x2, x19, xzr +// extr x1, x2, x1, #32 +// lsr x2, x2, #32 +// adds x12, x2, x19 +// adc x2, xzr, xzr +// subs x1, x11, x1 +// sbcs x7, x7, x12 +// sbcs x4, x4, x2 +// sbcs x20, x20, xzr +// sbcs x16, x16, xzr +// sbc x9, x19, xzr +// lsl x2, x1, #32 +// add x19, x2, x1 +// lsr x2, x19, #32 +// subs x1, x2, x19 +// sbc x2, x19, xzr +// extr x1, x2, x1, #32 +// lsr x2, x2, #32 +// adds x12, x2, x19 +// adc x2, xzr, xzr +// subs x1, x7, x1 +// sbcs x4, x4, x12 +// sbcs x20, x20, x2 +// sbcs x16, x16, xzr +// sbcs x7, x9, xzr +// sbc x9, x19, xzr +// lsl x2, x1, #32 +// add x19, x2, x1 +// lsr x2, x19, #32 +// subs x1, x2, x19 +// sbc x2, x19, xzr +// extr x12, x2, x1, #32 +// lsr x2, x2, #32 +// adds x1, x2, x19 +// adc x2, xzr, xzr +// subs x4, x4, x12 +// sbcs x20, x20, x1 +// sbcs x16, x16, x2 +// sbcs x12, x7, xzr +// sbcs x1, x9, xzr +// sbc x2, x19, xzr +// stp x4, x20, [x0] // @slothy:writes=buffer0 +// stp x16, x12, [x0, #16] // @slothy:writes=buffer16 +// stp x1, x2, [x0, #32] // @slothy:writes=buffer32 +// mul x22, x24, x14 +// movi v31.2D, #0x00000000ffffffff +// uzp2 v16.4S, v28.4S, v28.4S +// xtn v6.2S, v1.2D +// xtn v30.2S, v28.2D +// rev64 v28.4S, v28.4S +// umull v5.2D, v6.2S, v30.2S +// umull v0.2D, v6.2S, v16.2S +// uzp2 v19.4S, v1.4S, v1.4S +// mul v20.4S, v28.4S, v1.4S +// usra v0.2D, v5.2D, #32 +// umull v1.2D, v19.2S, v16.2S +// uaddlp v24.2D, v20.4S +// and v5.16B, v0.16B, v31.16B +// umlal v5.2D, v19.2S, v30.2S +// shl v19.2D, v24.2D, #32 +// usra v1.2D, v0.2D, #32 +// umlal v19.2D, v6.2S, v30.2S +// usra v1.2D, v5.2D, #32 +// mov x20, v19.d[0] +// mov x16, v19.d[1] +// umulh x12, x24, x14 +// mov x1, v1.d[0] +// mov x2, v1.d[1] +// adds x4, x12, x20 +// adcs x20, x1, x16 +// adc x16, x2, xzr +// adds x7, x4, x22 +// adcs x12, x20, x4 +// adcs x1, x16, x20 +// adc x2, x16, xzr +// adds x9, x12, x22 +// adcs x19, x1, x4 +// adcs x4, x2, x20 +// adc x20, x16, xzr +// subs x2, x24, x5 +// cneg x16, x2, cc +// csetm x12, cc +// subs x2, x15, x14 +// cneg x2, x2, cc +// mul x1, x16, x2 +// umulh x2, x16, x2 +// cinv x12, x12, cc +// eor x1, x1, x12 +// eor x2, x2, x12 +// cmn x12, #0x1 +// adcs x11, x7, x1 +// adcs x9, x9, x2 +// adcs x19, x19, x12 +// adcs x4, x4, x12 +// adc x20, x20, x12 +// subs x2, x24, x10 +// cneg x16, x2, cc +// csetm x12, cc +// subs x2, x17, x14 +// cneg x2, x2, cc +// mul x1, x16, x2 +// umulh x2, x16, x2 +// cinv x12, x12, cc +// eor x1, x1, x12 +// eor x2, x2, x12 +// cmn x12, #0x1 +// adcs x7, x9, x1 +// adcs x19, x19, x2 +// adcs x4, x4, x12 +// adc x20, x20, x12 +// subs x2, x5, x10 +// cneg x16, x2, cc +// csetm x12, cc +// subs x2, x17, x15 +// cneg x2, x2, cc +// mul x1, x16, x2 +// umulh x2, x16, x2 +// cinv x16, x12, cc +// eor x1, x1, x16 +// eor x2, x2, x16 +// cmn x16, #0x1 +// adcs x19, x19, x1 +// adcs x12, x4, x2 +// adc x1, x20, x16 +// subs x2, x24, x3 +// sbcs x24, x5, x21 +// sbcs x21, x10, x8 +// ngc x5, xzr +// cmn x5, #0x1 +// eor x2, x2, x5 +// adcs x4, x2, xzr +// eor x2, x24, x5 +// adcs x20, x2, xzr +// eor x2, x21, x5 +// adc x16, x2, xzr +// subs x2, x13, x14 +// sbcs x24, x23, x15 +// sbcs x8, x6, x17 +// ngc x21, xzr +// cmn x21, #0x1 +// eor x2, x2, x21 +// adcs x15, x2, xzr +// eor x2, x24, x21 +// adcs x14, x2, xzr +// eor x2, x8, x21 +// adc x6, x2, xzr +// eor x9, x5, x21 +// ldp x21, x2, [x0] // @slothy:reads=buffer0 +// adds x10, x22, x21 +// adcs x5, x11, x2 +// ldp x21, x2, [x0, #16] // @slothy:reads=buffer16 +// adcs x24, x7, x21 +// adcs x8, x19, x2 +// ldp x21, x2, [x0, #32] // @slothy:reads=buffer32 +// adcs x21, x12, x21 +// adcs x2, x1, x2 +// adc x19, xzr, xzr +// stp x10, x5, [x0] // @slothy:writes=buffer0 +// stp x24, x8, [x0, #16] // @slothy:writes=buffer16 +// stp x21, x2, [x0, #32] // @slothy:writes=buffer32 +// mul x12, x4, x15 +// mul x5, x20, x14 +// mul x24, x16, x6 +// umulh x8, x4, x15 +// umulh x21, x20, x14 +// umulh x2, x16, x6 +// adds x10, x8, x5 +// adcs x5, x21, x24 +// adc x24, x2, xzr +// adds x23, x10, x12 +// adcs x8, x5, x10 +// adcs x21, x24, x5 +// adc x2, x24, xzr +// adds x13, x8, x12 +// adcs x1, x21, x10 +// adcs x10, x2, x5 +// adc x5, x24, xzr +// subs x2, x4, x20 +// cneg x24, x2, cc +// csetm x8, cc +// subs x2, x14, x15 +// cneg x2, x2, cc +// mul x21, x24, x2 +// umulh x2, x24, x2 +// cinv x8, x8, cc +// eor x21, x21, x8 +// eor x2, x2, x8 +// cmn x8, #0x1 +// adcs x23, x23, x21 +// adcs x13, x13, x2 +// adcs x1, x1, x8 +// adcs x10, x10, x8 +// adc x5, x5, x8 +// subs x2, x4, x16 +// cneg x24, x2, cc +// csetm x8, cc +// subs x2, x6, x15 +// cneg x2, x2, cc +// mul x21, x24, x2 +// umulh x2, x24, x2 +// cinv x8, x8, cc +// eor x21, x21, x8 +// eor x2, x2, x8 +// cmn x8, #0x1 +// adcs x4, x13, x21 +// adcs x13, x1, x2 +// adcs x1, x10, x8 +// adc x10, x5, x8 +// subs x2, x20, x16 +// cneg x24, x2, cc +// csetm x8, cc +// subs x2, x6, x14 +// cneg x2, x2, cc +// mul x21, x24, x2 +// umulh x2, x24, x2 +// cinv x5, x8, cc +// eor x21, x21, x5 +// eor x2, x2, x5 +// cmn x5, #0x1 +// adcs x24, x13, x21 +// adcs x8, x1, x2 +// adc x21, x10, x5 +// ldp x20, x16, [x0] // @slothy:reads=buffer0 +// ldp x17, x15, [x0, #16] // @slothy:reads=buffer16 +// ldp x14, x6, [x0, #32] // @slothy:reads=buffer32 +// cmn x9, #0x1 +// eor x2, x12, x9 +// adcs x12, x2, x20 +// eor x2, x23, x9 +// adcs x23, x2, x16 +// eor x2, x4, x9 +// adcs x13, x2, x17 +// eor x2, x24, x9 +// adcs x10, x2, x15 +// eor x2, x8, x9 +// adcs x5, x2, x14 +// eor x2, x21, x9 +// adcs x24, x2, x6 +// adcs x1, x9, x19 +// adcs x8, x9, xzr +// adcs x21, x9, xzr +// adc x2, x9, xzr +// adds x10, x10, x20 +// adcs x5, x5, x16 +// adcs x24, x24, x17 +// adcs x17, x1, x15 +// adcs x15, x8, x14 +// adcs x14, x21, x6 +// adc x6, x2, x19 +// lsl x2, x12, #32 +// add x1, x2, x12 +// lsr x2, x1, #32 +// subs x21, x2, x1 +// sbc x2, x1, xzr +// extr x21, x2, x21, #32 +// lsr x2, x2, #32 +// adds x8, x2, x1 +// adc x2, xzr, xzr +// subs x21, x23, x21 +// sbcs x23, x13, x8 +// sbcs x10, x10, x2 +// sbcs x5, x5, xzr +// sbcs x24, x24, xzr +// sbc x13, x1, xzr +// lsl x2, x21, #32 +// add x1, x2, x21 +// lsr x2, x1, #32 +// subs x21, x2, x1 +// sbc x2, x1, xzr +// extr x21, x2, x21, #32 +// lsr x2, x2, #32 +// adds x8, x2, x1 +// adc x2, xzr, xzr +// subs x21, x23, x21 +// sbcs x10, x10, x8 +// sbcs x5, x5, x2 +// sbcs x24, x24, xzr +// sbcs x23, x13, xzr +// sbc x13, x1, xzr +// lsl x2, x21, #32 +// add x1, x2, x21 +// lsr x2, x1, #32 +// subs x21, x2, x1 +// sbc x2, x1, xzr +// extr x8, x2, x21, #32 +// lsr x2, x2, #32 +// adds x21, x2, x1 +// adc x2, xzr, xzr +// subs x10, x10, x8 +// sbcs x5, x5, x21 +// sbcs x24, x24, x2 +// sbcs x8, x23, xzr +// sbcs x21, x13, xzr +// sbc x2, x1, xzr +// adds x23, x17, x8 +// adcs x13, x15, x21 +// adcs x1, x14, x2 +// adc x2, x6, xzr +// add x8, x2, #0x1 +// lsl x2, x8, #32 +// subs x21, x8, x2 +// sbc x2, x2, xzr +// adds x10, x10, x21 +// adcs x5, x5, x2 +// adcs x24, x24, x8 +// adcs x8, x23, xzr +// adcs x21, x13, xzr +// adcs x13, x1, xzr +// csetm x1, cc +// mov x2, #0xffffffff +// and x2, x2, x1 +// adds x10, x10, x2 +// eor x2, x2, x1 +// adcs x5, x5, x2 +// mov x2, #0xfffffffffffffffe +// and x2, x2, x1 +// adcs x24, x24, x2 +// adcs x8, x8, x1 +// adcs x21, x21, x1 +// adc x2, x13, x1 +// stp x10, x5, [x0] // @slothy:writes=buffer0 +// stp x24, x8, [x0, #16] // @slothy:writes=buffer16 +// stp x21, x2, [x0, #32] // @slothy:writes=buffer32 +// ldp x23, x24, [sp], #16 +// ldp x21, x22, [sp], #16 +// ldp x19, x20, [sp], #16 +// ret +// +// The bash script used for step 2 is as follows: +// +// # Store the assembly instructions except the last 'ret' and +// # callee-register store/loads as, say, 'input.S'. +// export OUTPUTS="[hint_buffer0,hint_buffer16,hint_buffer32]" +// export RESERVED_REGS="[x18,x25,x26,x27,x28,x29,x30,sp,q8,q9,q10,q11,q12,q13,q14,q15,v8,v9,v10,v11,v12,v13,v14,v15]" +// /tools/external/slothy.sh input.S my_out_dir +// # my_out_dir/3.opt.s is the optimized assembly. Its output may differ +// # from this file since the sequence is non-deterministically chosen. +// # Please add 'ret' at the end of the output assembly. + + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montmul_p384) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montmul_p384) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montmul_p384) + .text + .balign 4 + +S2N_BN_SYMBOL(bignum_montmul_p384): + CFI_START + +// Save some registers + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + + ldr q3, [x1] + ldr q25, [x2] + ldp x13, x23, [x2] + ldp x3, x21, [x1] + rev64 v23.4S, v25.4S + uzp1 v17.4S, v25.4S, v3.4S + umulh x15, x3, x13 + mul v6.4S, v23.4S, v3.4S + uzp1 v3.4S, v3.4S, v3.4S + ldr q27, [x2, #32] + ldp x8, x24, [x1, #16] + subs x6, x3, x21 + ldr q0, [x1, #32] + movi v23.2D, #0x00000000ffffffff + csetm x10, cc + umulh x19, x21, x23 + rev64 v4.4S, v27.4S + uzp2 v25.4S, v27.4S, v27.4S + cneg x4, x6, cc + subs x7, x23, x13 + xtn v22.2S, v0.2D + xtn v24.2S, v27.2D + cneg x20, x7, cc + ldp x6, x14, [x2, #16] + mul v27.4S, v4.4S, v0.4S + uaddlp v20.2D, v6.4S + cinv x5, x10, cc + mul x16, x4, x20 + uzp2 v6.4S, v0.4S, v0.4S + umull v21.2D, v22.2S, v25.2S + shl v0.2D, v20.2D, #32 + umlal v0.2D, v3.2S, v17.2S + mul x22, x8, x6 + umull v1.2D, v6.2S, v25.2S + subs x12, x3, x8 + umull v20.2D, v22.2S, v24.2S + cneg x17, x12, cc + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc + usra v21.2D, v20.2D, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2D, v21.2D, #32 + adds x22, x15, x7 + and v26.16B, v21.16B, v23.16B + adcs x16, x12, x15 + uaddlp v25.2D, v27.4S + adcs x9, x19, x12 + umlal v26.2D, v6.2S, v24.2S + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2D, v25.2D, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc + cinv x10, x10, cc + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc + eor x19, x19, x10 + csetm x4, cc + subs x16, x6, x23 + cneg x16, x16, cc + umlal v27.2D, v22.2S, v24.2S + mul x15, x20, x16 + cinv x4, x4, cc + cmn x10, #0x1 + usra v1.2D, v26.2D, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [x2, #32] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [x1, #32] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x16, x4, x16 + mov x4, v27.d[0] + sbcs x11, x20, x11 + sbcs x20, x9, x12 + stp x16, x11, [x0] + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [x0, #16] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc + csetm x2, cc + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc + cneg x19, x19, cc + stp x9, x20, [x0, #32] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc + csetm x12, cc + subs x9, x17, x14 + cinv x12, x12, cc + cneg x9, x9, cc + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc + cneg x24, x10, cc + subs x10, x17, x15 + cinv x7, x7, cc + cneg x10, x10, cc + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + ldp x20, x9, [x0] + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [x0, #16] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [x0, #32] + adds x20, x22, x20 + mul x10, x13, x14 + adcs x11, x11, x9 + eor x9, x8, x21 + adcs x21, x19, x17 + stp x20, x11, [x0] + adcs x12, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + stp x21, x12, [x0, #16] + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [x0, #32] + cneg x3, x21, cc + csetm x24, cc + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc + csetm x16, cc + subs x21, x6, x15 + cneg x22, x21, cc + cinv x21, x24, cc + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc + csetm x24, cc + subs x20, x14, x15 + cinv x24, x24, cc + mul x22, x3, x22 + cneg x3, x20, cc + subs x13, x6, x14 + cneg x20, x13, cc + cinv x15, x16, cc + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [x0] + ldp x21, x12, [x0, #16] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [x0, #32] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x21 + eor x1, x22, x9 + adcs x24, x23, x12 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x21 + adcs x15, x17, x12 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc + and x11, x16, x23 + and x20, x14, x23 + adds x22, x6, x20 + eor x3, x20, x23 + adcs x5, x7, x3 + adcs x14, x24, x11 + stp x22, x5, [x0] + adcs x5, x13, x23 + adcs x21, x8, x23 + stp x14, x5, [x0, #16] + adc x12, x15, x23 + stp x21, x12, [x0, #32] + +// Restore registers and return + + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_montmul_p384) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/bignum_montmul_p384_alt.S b/cbits/s2n/arm/bignum_montmul_p384_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_montmul_p384_alt.S @@ -0,0 +1,345 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Montgomery multiply, z := (x * y / 2^384) mod p_384 +// Inputs x[6], y[6]; output z[6] +// +// extern void bignum_montmul_p384_alt(uint64_t z[static 6], +// const uint64_t x[static 6], +// const uint64_t y[static 6]); +// +// Does z := (2^{-384} * x * y) mod p_384, assuming that the inputs x and y +// satisfy x * y <= 2^384 * p_384 (in particular this is true if we are in +// the "usual" case x < p_384 and y < p_384). +// +// Standard ARM ABI: X0 = z, X1 = x, X2 = y +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montmul_p384_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montmul_p384_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montmul_p384_alt) + .text + .balign 4 + +// --------------------------------------------------------------------------- +// Core one-step "short" Montgomery reduction macro. Takes input in +// [d5;d4;d3;d2;d1;d0] and returns result in [d6;d5;d4;d3;d2;d1], +// adding to the existing contents of [d5;d4;d3;d2;d1]. It is fine +// for d6 to be the same register as d0. +// +// We want to add (2^384 - 2^128 - 2^96 + 2^32 - 1) * w +// where w = [d0 + (d0<<32)] mod 2^64 +// --------------------------------------------------------------------------- + +#define montreds(d6,d5,d4,d3,d2,d1,d0, t3,t2,t1) \ +/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64 */ \ +/* Store it in d6 to make the 2^384 * w contribution already */ \ + lsl t1, d0, #32 __LF \ + add d6, t1, d0 __LF \ +/* Now let [t3;t2;t1;-] = (2^384 - p_384) * w */ \ +/* We know the lowest word will cancel d0 so we don't need it */ \ + mov t1, #0xffffffff00000001 __LF \ + umulh t1, t1, d6 __LF \ + mov t2, #0x00000000ffffffff __LF \ + mul t3, t2, d6 __LF \ + umulh t2, t2, d6 __LF \ + adds t1, t1, t3 __LF \ + adcs t2, t2, d6 __LF \ + adc t3, xzr, xzr __LF \ +/* Now add it, by subtracting from 2^384 * w + x */ \ + subs d1, d1, t1 __LF \ + sbcs d2, d2, t2 __LF \ + sbcs d3, d3, t3 __LF \ + sbcs d4, d4, xzr __LF \ + sbcs d5, d5, xzr __LF \ + sbc d6, d6, xzr + + +#define z x0 +#define x x1 +#define y x2 + +// These are repeated mod 2 as we load pairs of inputs + +#define a0 x3 +#define a1 x4 +#define a2 x3 +#define a3 x4 +#define a4 x3 +#define a5 x4 + +#define b0 x5 +#define b1 x6 +#define b2 x7 +#define b3 x8 +#define b4 x9 +#define b5 x10 + +#define l x11 + +#define u0 x12 +#define u1 x13 +#define u2 x14 +#define u3 x15 +#define u4 x16 +#define u5 x17 +#define u6 x19 +#define u7 x20 +#define u8 x21 +#define u9 x22 +#define u10 x2 // same as y +#define u11 x1 // same as x +#define h b5 // same as b5 + +S2N_BN_SYMBOL(bignum_montmul_p384_alt): + CFI_START + +// Save more registers + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + +// Load operands and set up row 0 = [u6;...;u0] = a0 * [b5;...;b0] + + ldp a0, a1, [x] + ldp b0, b1, [y] + + mul u0, a0, b0 + umulh u1, a0, b0 + mul l, a0, b1 + umulh u2, a0, b1 + adds u1, u1, l + + ldp b2, b3, [y, #16] + + mul l, a0, b2 + umulh u3, a0, b2 + adcs u2, u2, l + + mul l, a0, b3 + umulh u4, a0, b3 + adcs u3, u3, l + + ldp b4, b5, [y, #32] + + mul l, a0, b4 + umulh u5, a0, b4 + adcs u4, u4, l + + mul l, a0, b5 + umulh u6, a0, b5 + adcs u5, u5, l + + adc u6, u6, xzr + +// Row 1 = [u7;...;u0] = [a1;a0] * [b5;...;b0] + + mul l, a1, b0 + adds u1, u1, l + mul l, a1, b1 + adcs u2, u2, l + mul l, a1, b2 + adcs u3, u3, l + mul l, a1, b3 + adcs u4, u4, l + mul l, a1, b4 + adcs u5, u5, l + mul l, a1, b5 + adcs u6, u6, l + cset u7, cs + + umulh l, a1, b0 + adds u2, u2, l + umulh l, a1, b1 + adcs u3, u3, l + umulh l, a1, b2 + adcs u4, u4, l + umulh l, a1, b3 + adcs u5, u5, l + umulh l, a1, b4 + adcs u6, u6, l + umulh l, a1, b5 + adc u7, u7, l + +// Row 2 = [u8;...;u0] = [a2;a1;a0] * [b5;...;b0] + + ldp a2, a3, [x, #16] + + mul l, a2, b0 + adds u2, u2, l + mul l, a2, b1 + adcs u3, u3, l + mul l, a2, b2 + adcs u4, u4, l + mul l, a2, b3 + adcs u5, u5, l + mul l, a2, b4 + adcs u6, u6, l + mul l, a2, b5 + adcs u7, u7, l + cset u8, cs + + umulh l, a2, b0 + adds u3, u3, l + umulh l, a2, b1 + adcs u4, u4, l + umulh l, a2, b2 + adcs u5, u5, l + umulh l, a2, b3 + adcs u6, u6, l + umulh l, a2, b4 + adcs u7, u7, l + umulh l, a2, b5 + adc u8, u8, l + +// Row 3 = [u9;...;u0] = [a3;a2;a1;a0] * [b5;...;b0] + + mul l, a3, b0 + adds u3, u3, l + mul l, a3, b1 + adcs u4, u4, l + mul l, a3, b2 + adcs u5, u5, l + mul l, a3, b3 + adcs u6, u6, l + mul l, a3, b4 + adcs u7, u7, l + mul l, a3, b5 + adcs u8, u8, l + cset u9, cs + + umulh l, a3, b0 + adds u4, u4, l + umulh l, a3, b1 + adcs u5, u5, l + umulh l, a3, b2 + adcs u6, u6, l + umulh l, a3, b3 + adcs u7, u7, l + umulh l, a3, b4 + adcs u8, u8, l + umulh l, a3, b5 + adc u9, u9, l + +// Row 4 = [u10;...;u0] = [a4;a3;a2;a1;a0] * [b5;...;b0] + + ldp a4, a5, [x, #32] + + mul l, a4, b0 + adds u4, u4, l + mul l, a4, b1 + adcs u5, u5, l + mul l, a4, b2 + adcs u6, u6, l + mul l, a4, b3 + adcs u7, u7, l + mul l, a4, b4 + adcs u8, u8, l + mul l, a4, b5 + adcs u9, u9, l + cset u10, cs + + umulh l, a4, b0 + adds u5, u5, l + umulh l, a4, b1 + adcs u6, u6, l + umulh l, a4, b2 + adcs u7, u7, l + umulh l, a4, b3 + adcs u8, u8, l + umulh l, a4, b4 + adcs u9, u9, l + umulh l, a4, b5 + adc u10, u10, l + +// Row 5 = [u11;...;u0] = [a5;a4;a3;a2;a1;a0] * [b5;...;b0] + + mul l, a5, b0 + adds u5, u5, l + mul l, a5, b1 + adcs u6, u6, l + mul l, a5, b2 + adcs u7, u7, l + mul l, a5, b3 + adcs u8, u8, l + mul l, a5, b4 + adcs u9, u9, l + mul l, a5, b5 + adcs u10, u10, l + cset u11, cs + + umulh l, a5, b0 + adds u6, u6, l + umulh l, a5, b1 + adcs u7, u7, l + umulh l, a5, b2 + adcs u8, u8, l + umulh l, a5, b3 + adcs u9, u9, l + umulh l, a5, b4 + adcs u10, u10, l + umulh l, a5, b5 + adc u11, u11, l + +// Montgomery rotate the low half + + montreds(u0,u5,u4,u3,u2,u1,u0, b0,b1,b2) + montreds(u1,u0,u5,u4,u3,u2,u1, b0,b1,b2) + montreds(u2,u1,u0,u5,u4,u3,u2, b0,b1,b2) + montreds(u3,u2,u1,u0,u5,u4,u3, b0,b1,b2) + montreds(u4,u3,u2,u1,u0,u5,u4, b0,b1,b2) + montreds(u5,u4,u3,u2,u1,u0,u5, b0,b1,b2) + +// Add up the high and low parts as [h; u5;u4;u3;u2;u1;u0] = z + + adds u0, u0, u6 + adcs u1, u1, u7 + adcs u2, u2, u8 + adcs u3, u3, u9 + adcs u4, u4, u10 + adcs u5, u5, u11 + adc h, xzr, xzr + +// Now add [h; u11;u10;u9;u8;u7;u6] = z + (2^384 - p_384) + + mov l, #0xffffffff00000001 + adds u6, u0, l + mov l, #0x00000000ffffffff + adcs u7, u1, l + mov l, #0x0000000000000001 + adcs u8, u2, l + adcs u9, u3, xzr + adcs u10, u4, xzr + adcs u11, u5, xzr + adcs h, h, xzr + +// Now z >= p_384 iff h is nonzero, so select accordingly + + csel u0, u0, u6, eq + csel u1, u1, u7, eq + csel u2, u2, u8, eq + csel u3, u3, u9, eq + csel u4, u4, u10, eq + csel u5, u5, u11, eq + +// Store back final result + + stp u0, u1, [z] + stp u2, u3, [z, #16] + stp u4, u5, [z, #32] + +// Restore registers + + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_montmul_p384_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/bignum_montsqr_p384.S b/cbits/s2n/arm/bignum_montsqr_p384.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_montsqr_p384.S @@ -0,0 +1,671 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Montgomery square, z := (x^2 / 2^384) mod p_384 +// Input x[6]; output z[6] +// +// extern void bignum_montsqr_p384(uint64_t z[static 6], +// const uint64_t x[static 6]); +// +// Does z := (x^2 / 2^384) mod p_384, assuming x^2 <= 2^384 * p_384, which is +// guaranteed in particular if x < p_384 initially (the "intended" case). +// +// Standard ARM ABI: X0 = z, X1 = x +// ---------------------------------------------------------------------------- + +// bignum_montsqr_p384 is functionally equivalent to +// unopt/bignum_montsqr_p384_base. +// It is written in a way that +// 1. A subset of scalar multiplications in bignum_montsqr_p384 are carefully +// chosen and vectorized +// 2. The vectorized assembly is rescheduled using the SLOTHY superoptimizer. +// https://github.com/slothy-optimizer/slothy +// +// The output program of step 1. is as follows: +// +// ldp x9, x2, [x1] +// ldr q18, [x1] +// ldr q19, [x1] +// ldp x4, x6, [x1, #16] +// ldp x5, x10, [x1, #32] +// ldr q21, [x1, #32] +// ldr q28, [x1, #32] +// mul x12, x9, x2 +// mul x1, x9, x4 +// mul x13, x2, x4 +// movi v0.2D, #0x00000000ffffffff +// uzp2 v5.4S, v19.4S, v19.4S +// xtn v25.2S, v18.2D +// xtn v4.2S, v19.2D +// rev64 v23.4S, v19.4S +// umull v20.2D, v25.2S, v4.2S +// umull v30.2D, v25.2S, v5.2S +// uzp2 v19.4S, v18.4S, v18.4S +// mul v22.4S, v23.4S, v18.4S +// usra v30.2D, v20.2D, #32 +// umull v18.2D, v19.2S, v5.2S +// uaddlp v22.2D, v22.4S +// and v20.16B, v30.16B, v0.16B +// umlal v20.2D, v19.2S, v4.2S +// shl v19.2D, v22.2D, #32 +// usra v18.2D, v30.2D, #32 +// umlal v19.2D, v25.2S, v4.2S +// usra v18.2D, v20.2D, #32 +// mov x7, v19.d[0] +// mov x17, v19.d[1] +// mul x16, x4, x4 +// umulh x3, x9, x2 +// adds x15, x1, x3 +// umulh x1, x9, x4 +// adcs x13, x13, x1 +// umulh x1, x2, x4 +// adcs x8, x1, xzr +// mov x11, v18.d[0] +// mov x14, v18.d[1] +// umulh x1, x4, x4 +// adds x3, x12, x12 +// adcs x15, x15, x15 +// adcs x13, x13, x13 +// adcs x12, x8, x8 +// adc x1, x1, xzr +// adds x11, x11, x3 +// adcs x3, x17, x15 +// adcs x17, x14, x13 +// adcs x15, x16, x12 +// adc x13, x1, xzr +// lsl x1, x7, #32 +// add x16, x1, x7 +// lsr x1, x16, #32 +// subs x12, x1, x16 +// sbc x1, x16, xzr +// extr x12, x1, x12, #32 +// lsr x1, x1, #32 +// adds x7, x1, x16 +// adc x1, xzr, xzr +// subs x12, x11, x12 +// sbcs x11, x3, x7 +// sbcs x17, x17, x1 +// sbcs x15, x15, xzr +// sbcs x13, x13, xzr +// sbc x3, x16, xzr +// lsl x1, x12, #32 +// add x16, x1, x12 +// lsr x1, x16, #32 +// subs x12, x1, x16 +// sbc x1, x16, xzr +// extr x12, x1, x12, #32 +// lsr x1, x1, #32 +// adds x7, x1, x16 +// adc x1, xzr, xzr +// subs x12, x11, x12 +// sbcs x17, x17, x7 +// sbcs x15, x15, x1 +// sbcs x13, x13, xzr +// sbcs x11, x3, xzr +// sbc x3, x16, xzr +// lsl x1, x12, #32 +// add x16, x1, x12 +// lsr x1, x16, #32 +// subs x12, x1, x16 +// sbc x1, x16, xzr +// extr x7, x1, x12, #32 +// lsr x1, x1, #32 +// adds x12, x1, x16 +// adc x1, xzr, xzr +// subs x17, x17, x7 +// sbcs x15, x15, x12 +// sbcs x13, x13, x1 +// sbcs x7, x11, xzr +// sbcs x12, x3, xzr +// sbc x1, x16, xzr +// stp x17, x15, [x0] // @slothy:writes=buffer0 +// stp x13, x7, [x0, #16] // @slothy:writes=buffer16 +// stp x12, x1, [x0, #32] // @slothy:writes=buffer32 +// mul x14, x9, x6 +// mul x15, x2, x5 +// mul x13, x4, x10 +// umulh x7, x9, x6 +// umulh x12, x2, x5 +// umulh x1, x4, x10 +// adds x15, x7, x15 +// adcs x16, x12, x13 +// adc x13, x1, xzr +// adds x11, x15, x14 +// adcs x7, x16, x15 +// adcs x12, x13, x16 +// adc x1, x13, xzr +// adds x17, x7, x14 +// adcs x15, x12, x15 +// adcs x3, x1, x16 +// adc x16, x13, xzr +// subs x1, x9, x2 +// cneg x13, x1, cc +// csetm x7, cc +// subs x1, x5, x6 +// cneg x1, x1, cc +// mul x12, x13, x1 +// umulh x1, x13, x1 +// cinv x7, x7, cc +// eor x12, x12, x7 +// eor x1, x1, x7 +// cmn x7, #0x1 +// adcs x11, x11, x12 +// adcs x17, x17, x1 +// adcs x15, x15, x7 +// adcs x3, x3, x7 +// adc x16, x16, x7 +// subs x9, x9, x4 +// cneg x13, x9, cc +// csetm x7, cc +// subs x1, x10, x6 +// cneg x1, x1, cc +// mul x12, x13, x1 +// umulh x1, x13, x1 +// cinv x7, x7, cc +// eor x12, x12, x7 +// eor x1, x1, x7 +// cmn x7, #0x1 +// adcs x17, x17, x12 +// adcs x15, x15, x1 +// adcs x13, x3, x7 +// adc x7, x16, x7 +// subs x2, x2, x4 +// cneg x12, x2, cc +// csetm x1, cc +// subs x2, x10, x5 +// cneg x2, x2, cc +// mul x4, x12, x2 +// umulh x2, x12, x2 +// cinv x1, x1, cc +// eor x4, x4, x1 +// eor x2, x2, x1 +// cmn x1, #0x1 +// adcs x12, x15, x4 +// adcs x4, x13, x2 +// adc x2, x7, x1 +// adds x1, x14, x14 +// adcs x16, x11, x11 +// adcs x17, x17, x17 +// adcs x15, x12, x12 +// adcs x13, x4, x4 +// adcs x7, x2, x2 +// adc x12, xzr, xzr +// ldp x4, x2, [x0] // @slothy:reads=buffer0 +// adds x1, x1, x4 +// adcs x16, x16, x2 +// ldp x4, x2, [x0, #16] // @slothy:reads=buffer16 +// adcs x17, x17, x4 +// adcs x15, x15, x2 +// ldp x4, x2, [x0, #32] // @slothy:reads=buffer32 +// adcs x13, x13, x4 +// adcs x7, x7, x2 +// adc x11, x12, xzr +// lsl x2, x1, #32 +// add x12, x2, x1 +// lsr x2, x12, #32 +// subs x4, x2, x12 +// sbc x2, x12, xzr +// extr x4, x2, x4, #32 +// lsr x2, x2, #32 +// adds x1, x2, x12 +// adc x2, xzr, xzr +// subs x4, x16, x4 +// sbcs x16, x17, x1 +// sbcs x17, x15, x2 +// sbcs x15, x13, xzr +// sbcs x13, x7, xzr +// sbc x7, x12, xzr +// lsl x2, x4, #32 +// add x12, x2, x4 +// lsr x2, x12, #32 +// subs x4, x2, x12 +// sbc x2, x12, xzr +// extr x4, x2, x4, #32 +// lsr x2, x2, #32 +// adds x1, x2, x12 +// adc x2, xzr, xzr +// subs x4, x16, x4 +// sbcs x16, x17, x1 +// sbcs x17, x15, x2 +// sbcs x15, x13, xzr +// sbcs x13, x7, xzr +// sbc x7, x12, xzr +// lsl x2, x4, #32 +// add x12, x2, x4 +// lsr x2, x12, #32 +// subs x4, x2, x12 +// sbc x2, x12, xzr +// extr x1, x2, x4, #32 +// lsr x2, x2, #32 +// adds x4, x2, x12 +// adc x2, xzr, xzr +// subs x3, x16, x1 +// sbcs x17, x17, x4 +// sbcs x15, x15, x2 +// sbcs x1, x13, xzr +// sbcs x4, x7, xzr +// sbc x2, x12, xzr +// adds x13, x11, x1 +// adcs x7, x4, xzr +// adcs x12, x2, xzr +// adcs x16, xzr, xzr +// mul x2, x6, x6 +// adds x3, x3, x2 +// xtn v30.2S, v28.2D +// shrn v26.2S, v28.2D, #32 +// umull v26.2D, v30.2S, v26.2S +// shl v19.2D, v26.2D, #33 +// umlal v19.2D, v30.2S, v30.2S +// mov x1, v19.d[0] +// mov x4, v19.d[1] +// umulh x2, x6, x6 +// adcs x17, x17, x2 +// umulh x2, x5, x5 +// adcs x15, x15, x1 +// adcs x13, x13, x2 +// umulh x2, x10, x10 +// adcs x7, x7, x4 +// adcs x12, x12, x2 +// adc x16, x16, xzr +// dup v28.2D, x6 +// movi v0.2D, #0x00000000ffffffff +// uzp2 v5.4S, v21.4S, v21.4S +// xtn v25.2S, v28.2D +// xtn v4.2S, v21.2D +// rev64 v19.4S, v21.4S +// umull v30.2D, v25.2S, v4.2S +// umull v23.2D, v25.2S, v5.2S +// uzp2 v20.4S, v28.4S, v28.4S +// mul v19.4S, v19.4S, v28.4S +// usra v23.2D, v30.2D, #32 +// umull v18.2D, v20.2S, v5.2S +// uaddlp v19.2D, v19.4S +// and v30.16B, v23.16B, v0.16B +// umlal v30.2D, v20.2S, v4.2S +// shl v19.2D, v19.2D, #32 +// usra v18.2D, v23.2D, #32 +// umlal v19.2D, v25.2S, v4.2S +// usra v18.2D, v30.2D, #32 +// mov x6, v19.d[0] +// mov x1, v19.d[1] +// mul x4, x5, x10 +// mov x2, v18.d[0] +// adds x1, x1, x2 +// mov x2, v18.d[1] +// adcs x4, x4, x2 +// umulh x5, x5, x10 +// adc x2, x5, xzr +// adds x5, x6, x6 +// adcs x6, x1, x1 +// adcs x1, x4, x4 +// adcs x4, x2, x2 +// adc x2, xzr, xzr +// adds x17, x17, x5 +// adcs x15, x15, x6 +// adcs x13, x13, x1 +// adcs x7, x7, x4 +// adcs x12, x12, x2 +// adc x2, x16, xzr +// mov x5, #0xffffffff00000001 +// mov x6, #0xffffffff +// mov x1, #0x1 +// cmn x3, x5 +// adcs xzr, x17, x6 +// adcs xzr, x15, x1 +// adcs xzr, x13, xzr +// adcs xzr, x7, xzr +// adcs xzr, x12, xzr +// adc x2, x2, xzr +// neg x4, x2 +// and x2, x5, x4 +// adds x10, x3, x2 +// and x2, x6, x4 +// adcs x5, x17, x2 +// and x2, x1, x4 +// adcs x6, x15, x2 +// adcs x1, x13, xzr +// adcs x4, x7, xzr +// adc x2, x12, xzr +// stp x10, x5, [x0] // @slothy:writes=buffer0 +// stp x6, x1, [x0, #16] // @slothy:writes=buffer16 +// stp x4, x2, [x0, #32] // @slothy:writes=buffer32 +// ret +// +// The bash script used for step 2 is as follows: +// +// # Store the assembly instructions except the last 'ret' as, say, 'input.S'. +// export OUTPUTS="[hint_buffer0,hint_buffer16,hint_buffer32]" +// export RESERVED_REGS="[x18,x19,x20,x21,x22,x23,x24,x25,x26,x27,x28,x29,x30,sp,q8,q9,q10,q11,q12,q13,q14,q15,v8,v9,v10,v11,v12,v13,v14,v15]" +// /tools/external/slothy.sh input.S my_out_dir +// # my_out_dir/3.opt.s is the optimized assembly. Its output may differ +// # from this file since the sequence is non-deterministically chosen. +// # Please add 'ret' at the end of the output assembly. + + + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montsqr_p384) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montsqr_p384) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montsqr_p384) + .text + .balign 4 + +S2N_BN_SYMBOL(bignum_montsqr_p384): + CFI_START + + ldr q1, [x1] + ldp x9, x2, [x1] + ldr q0, [x1] + ldp x4, x6, [x1, #16] + rev64 v21.4S, v1.4S + uzp2 v28.4S, v1.4S, v1.4S + umulh x7, x9, x2 + xtn v17.2S, v1.2D + mul v27.4S, v21.4S, v0.4S + ldr q20, [x1, #32] + xtn v30.2S, v0.2D + ldr q1, [x1, #32] + uzp2 v31.4S, v0.4S, v0.4S + ldp x5, x10, [x1, #32] + umulh x8, x9, x4 + uaddlp v3.2D, v27.4S + umull v16.2D, v30.2S, v17.2S + mul x16, x9, x4 + umull v27.2D, v30.2S, v28.2S + shrn v0.2S, v20.2D, #32 + xtn v7.2S, v20.2D + shl v20.2D, v3.2D, #32 + umull v3.2D, v31.2S, v28.2S + mul x3, x2, x4 + umlal v20.2D, v30.2S, v17.2S + umull v22.2D, v7.2S, v0.2S + usra v27.2D, v16.2D, #32 + umulh x11, x2, x4 + movi v21.2D, #0x00000000ffffffff + uzp2 v28.4S, v1.4S, v1.4S + adds x15, x16, x7 + and v5.16B, v27.16B, v21.16B + adcs x3, x3, x8 + usra v3.2D, v27.2D, #32 + dup v29.2D, x6 + adcs x16, x11, xzr + mov x14, v20.d[0] + umlal v5.2D, v31.2S, v17.2S + mul x8, x9, x2 + mov x7, v20.d[1] + shl v19.2D, v22.2D, #33 + xtn v25.2S, v29.2D + rev64 v31.4S, v1.4S + lsl x13, x14, #32 + uzp2 v6.4S, v29.4S, v29.4S + umlal v19.2D, v7.2S, v7.2S + usra v3.2D, v5.2D, #32 + adds x1, x8, x8 + umulh x8, x4, x4 + add x12, x13, x14 + mul v17.4S, v31.4S, v29.4S + xtn v4.2S, v1.2D + adcs x14, x15, x15 + lsr x13, x12, #32 + adcs x15, x3, x3 + umull v31.2D, v25.2S, v28.2S + adcs x11, x16, x16 + umull v21.2D, v25.2S, v4.2S + mov x17, v3.d[0] + umull v18.2D, v6.2S, v28.2S + adc x16, x8, xzr + uaddlp v16.2D, v17.4S + movi v1.2D, #0x00000000ffffffff + subs x13, x13, x12 + usra v31.2D, v21.2D, #32 + sbc x8, x12, xzr + adds x17, x17, x1 + mul x1, x4, x4 + shl v28.2D, v16.2D, #32 + mov x3, v3.d[1] + adcs x14, x7, x14 + extr x7, x8, x13, #32 + adcs x13, x3, x15 + and v3.16B, v31.16B, v1.16B + adcs x11, x1, x11 + lsr x1, x8, #32 + umlal v3.2D, v6.2S, v4.2S + usra v18.2D, v31.2D, #32 + adc x3, x16, xzr + adds x1, x1, x12 + umlal v28.2D, v25.2S, v4.2S + adc x16, xzr, xzr + subs x15, x17, x7 + sbcs x7, x14, x1 + lsl x1, x15, #32 + sbcs x16, x13, x16 + add x8, x1, x15 + usra v18.2D, v3.2D, #32 + sbcs x14, x11, xzr + lsr x1, x8, #32 + sbcs x17, x3, xzr + sbc x11, x12, xzr + subs x13, x1, x8 + umulh x12, x4, x10 + sbc x1, x8, xzr + extr x13, x1, x13, #32 + lsr x1, x1, #32 + adds x15, x1, x8 + adc x1, xzr, xzr + subs x7, x7, x13 + sbcs x13, x16, x15 + lsl x3, x7, #32 + umulh x16, x2, x5 + sbcs x15, x14, x1 + add x7, x3, x7 + sbcs x3, x17, xzr + lsr x1, x7, #32 + sbcs x14, x11, xzr + sbc x11, x8, xzr + subs x8, x1, x7 + sbc x1, x7, xzr + extr x8, x1, x8, #32 + lsr x1, x1, #32 + adds x1, x1, x7 + adc x17, xzr, xzr + subs x13, x13, x8 + umulh x8, x9, x6 + sbcs x1, x15, x1 + sbcs x15, x3, x17 + sbcs x3, x14, xzr + mul x17, x2, x5 + sbcs x11, x11, xzr + stp x13, x1, [x0] // @slothy:writes=buffer0 + sbc x14, x7, xzr + mul x7, x4, x10 + subs x1, x9, x2 + stp x15, x3, [x0, #16] // @slothy:writes=buffer16 + csetm x15, cc + cneg x1, x1, cc + stp x11, x14, [x0, #32] // @slothy:writes=buffer32 + mul x14, x9, x6 + adds x17, x8, x17 + adcs x7, x16, x7 + adc x13, x12, xzr + subs x12, x5, x6 + cneg x3, x12, cc + cinv x16, x15, cc + mul x8, x1, x3 + umulh x1, x1, x3 + eor x12, x8, x16 + adds x11, x17, x14 + adcs x3, x7, x17 + adcs x15, x13, x7 + adc x8, x13, xzr + adds x3, x3, x14 + adcs x15, x15, x17 + adcs x17, x8, x7 + eor x1, x1, x16 + adc x13, x13, xzr + subs x9, x9, x4 + csetm x8, cc + cneg x9, x9, cc + subs x4, x2, x4 + cneg x4, x4, cc + csetm x7, cc + subs x2, x10, x6 + cinv x8, x8, cc + cneg x2, x2, cc + cmn x16, #0x1 + adcs x11, x11, x12 + mul x12, x9, x2 + adcs x3, x3, x1 + adcs x15, x15, x16 + umulh x9, x9, x2 + adcs x17, x17, x16 + adc x13, x13, x16 + subs x1, x10, x5 + cinv x2, x7, cc + cneg x1, x1, cc + eor x9, x9, x8 + cmn x8, #0x1 + eor x7, x12, x8 + mul x12, x4, x1 + adcs x3, x3, x7 + adcs x7, x15, x9 + adcs x15, x17, x8 + ldp x9, x17, [x0, #16] // @slothy:reads=buffer16 + umulh x4, x4, x1 + adc x8, x13, x8 + cmn x2, #0x1 + eor x1, x12, x2 + adcs x1, x7, x1 + ldp x7, x16, [x0] // @slothy:reads=buffer0 + eor x12, x4, x2 + adcs x4, x15, x12 + ldp x15, x12, [x0, #32] // @slothy:reads=buffer32 + adc x8, x8, x2 + adds x13, x14, x14 + umulh x14, x5, x10 + adcs x2, x11, x11 + adcs x3, x3, x3 + adcs x1, x1, x1 + adcs x4, x4, x4 + adcs x11, x8, x8 + adc x8, xzr, xzr + adds x13, x13, x7 + adcs x2, x2, x16 + mul x16, x5, x10 + adcs x3, x3, x9 + adcs x1, x1, x17 + umulh x5, x5, x5 + lsl x9, x13, #32 + add x9, x9, x13 + adcs x4, x4, x15 + mov x13, v28.d[1] + adcs x15, x11, x12 + lsr x7, x9, #32 + adc x11, x8, xzr + subs x7, x7, x9 + umulh x10, x10, x10 + sbc x17, x9, xzr + extr x7, x17, x7, #32 + lsr x17, x17, #32 + adds x17, x17, x9 + adc x12, xzr, xzr + subs x8, x2, x7 + sbcs x17, x3, x17 + lsl x7, x8, #32 + sbcs x2, x1, x12 + add x3, x7, x8 + sbcs x12, x4, xzr + lsr x1, x3, #32 + sbcs x7, x15, xzr + sbc x15, x9, xzr + subs x1, x1, x3 + sbc x4, x3, xzr + lsr x9, x4, #32 + extr x8, x4, x1, #32 + adds x9, x9, x3 + adc x4, xzr, xzr + subs x1, x17, x8 + lsl x17, x1, #32 + sbcs x8, x2, x9 + sbcs x9, x12, x4 + add x17, x17, x1 + mov x1, v18.d[1] + lsr x2, x17, #32 + sbcs x7, x7, xzr + mov x12, v18.d[0] + sbcs x15, x15, xzr + sbc x3, x3, xzr + subs x4, x2, x17 + sbc x2, x17, xzr + adds x12, x13, x12 + adcs x16, x16, x1 + lsr x13, x2, #32 + extr x1, x2, x4, #32 + adc x2, x14, xzr + adds x4, x13, x17 + mul x13, x6, x6 + adc x14, xzr, xzr + subs x1, x8, x1 + sbcs x4, x9, x4 + mov x9, v28.d[0] + sbcs x7, x7, x14 + sbcs x8, x15, xzr + sbcs x3, x3, xzr + sbc x14, x17, xzr + adds x17, x9, x9 + adcs x12, x12, x12 + mov x15, v19.d[0] + adcs x9, x16, x16 + umulh x6, x6, x6 + adcs x16, x2, x2 + adc x2, xzr, xzr + adds x11, x11, x8 + adcs x3, x3, xzr + adcs x14, x14, xzr + adcs x8, xzr, xzr + adds x13, x1, x13 + mov x1, v19.d[1] + adcs x6, x4, x6 + mov x4, #0xffffffff + adcs x15, x7, x15 + adcs x7, x11, x5 + adcs x1, x3, x1 + adcs x14, x14, x10 + adc x11, x8, xzr + adds x6, x6, x17 + adcs x8, x15, x12 + adcs x3, x7, x9 + adcs x15, x1, x16 + mov x16, #0xffffffff00000001 + adcs x14, x14, x2 + mov x2, #0x1 + adc x17, x11, xzr + cmn x13, x16 + adcs xzr, x6, x4 + adcs xzr, x8, x2 + adcs xzr, x3, xzr + adcs xzr, x15, xzr + adcs xzr, x14, xzr + adc x1, x17, xzr + neg x9, x1 + and x1, x16, x9 + adds x11, x13, x1 + and x13, x4, x9 + adcs x5, x6, x13 + and x1, x2, x9 + adcs x7, x8, x1 + stp x11, x5, [x0] // @slothy:writes=buffer0 + adcs x11, x3, xzr + adcs x2, x15, xzr + stp x7, x11, [x0, #16] // @slothy:writes=buffer16 + adc x17, x14, xzr + stp x2, x17, [x0, #32] // depth 72 // @slothy:writes=buffer32 + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_montsqr_p384) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/bignum_montsqr_p384_alt.S b/cbits/s2n/arm/bignum_montsqr_p384_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_montsqr_p384_alt.S @@ -0,0 +1,273 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Montgomery square, z := (x^2 / 2^384) mod p_384 +// Input x[6]; output z[6] +// +// extern void bignum_montsqr_p384_alt(uint64_t z[static 6], +// const uint64_t x[static 6]); +// +// Does z := (x^2 / 2^384) mod p_384, assuming x^2 <= 2^384 * p_384, which is +// guaranteed in particular if x < p_384 initially (the "intended" case). +// +// Standard ARM ABI: X0 = z, X1 = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montsqr_p384_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montsqr_p384_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montsqr_p384_alt) + .text + .balign 4 + +// --------------------------------------------------------------------------- +// Core one-step "short" Montgomery reduction macro. Takes input in +// [d5;d4;d3;d2;d1;d0] and returns result in [d6;d5;d4;d3;d2;d1], +// adding to the existing contents of [d5;d4;d3;d2;d1]. It is fine +// for d6 to be the same register as d0. +// +// We want to add (2^384 - 2^128 - 2^96 + 2^32 - 1) * w +// where w = [d0 + (d0<<32)] mod 2^64 +// --------------------------------------------------------------------------- + +#define montreds(d6,d5,d4,d3,d2,d1,d0, t3,t2,t1) \ +/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64 */ \ +/* Store it in d6 to make the 2^384 * w contribution already */ \ + lsl t1, d0, #32 __LF \ + add d6, t1, d0 __LF \ +/* Now let [t3;t2;t1;-] = (2^384 - p_384) * w */ \ +/* We know the lowest word will cancel d0 so we don't need it */ \ + mov t1, #0xffffffff00000001 __LF \ + umulh t1, t1, d6 __LF \ + mov t2, #0x00000000ffffffff __LF \ + mul t3, t2, d6 __LF \ + umulh t2, t2, d6 __LF \ + adds t1, t1, t3 __LF \ + adcs t2, t2, d6 __LF \ + adc t3, xzr, xzr __LF \ +/* Now add it, by subtracting from 2^384 * w + x */ \ + subs d1, d1, t1 __LF \ + sbcs d2, d2, t2 __LF \ + sbcs d3, d3, t3 __LF \ + sbcs d4, d4, xzr __LF \ + sbcs d5, d5, xzr __LF \ + sbc d6, d6, xzr + +#define z x0 +#define x x1 + +#define a0 x2 +#define a1 x3 +#define a2 x4 +#define a3 x5 +#define a4 x6 +#define a5 x7 + +#define l x8 + +#define u0 x2 // The same as a0, which is safe +#define u1 x9 +#define u2 x10 +#define u3 x11 +#define u4 x12 +#define u5 x13 +#define u6 x14 +#define u7 x15 +#define u8 x16 +#define u9 x17 +#define u10 x19 +#define u11 x20 +#define h x6 // same as a4 + +S2N_BN_SYMBOL(bignum_montsqr_p384_alt): + CFI_START + +// It's convenient to have two more registers to play with + + CFI_PUSH2(x19,x20) + +// Load all the elements as [a5;a4;a3;a2;a1;a0], set up an initial +// window [u8;u7; u6;u5; u4;u3; u2;u1] = [34;05;03;01], and then +// chain in the addition of 02 + 12 + 13 + 14 + 15 to that window +// (no carry-out possible since we add it to the top of a product). + + ldp a0, a1, [x] + + mul u1, a0, a1 + umulh u2, a0, a1 + + ldp a2, a3, [x, #16] + + mul l, a0, a2 + adds u2, u2, l + + mul u3, a0, a3 + mul l, a1, a2 + adcs u3, u3, l + + umulh u4, a0, a3 + mul l, a1, a3 + adcs u4, u4, l + + ldp a4, a5, [x, #32] + + mul u5, a0, a5 + mul l, a1, a4 + adcs u5, u5, l + + umulh u6, a0, a5 + mul l, a1, a5 + adcs u6, u6, l + + mul u7, a3, a4 + adcs u7, u7, xzr + + umulh u8, a3, a4 + adc u8, u8, xzr + + umulh l, a0, a2 + adds u3, u3, l + umulh l, a1, a2 + adcs u4, u4, l + umulh l, a1, a3 + adcs u5, u5, l + umulh l, a1, a4 + adcs u6, u6, l + umulh l, a1, a5 + adcs u7, u7, l + adc u8, u8, xzr + +// Now chain in the 04 + 23 + 24 + 25 + 35 + 45 terms + + mul l, a0, a4 + adds u4, u4, l + mul l, a2, a3 + adcs u5, u5, l + mul l, a2, a4 + adcs u6, u6, l + mul l, a2, a5 + adcs u7, u7, l + mul l, a3, a5 + adcs u8, u8, l + mul u9, a4, a5 + adcs u9, u9, xzr + umulh u10, a4, a5 + adc u10, u10, xzr + + umulh l, a0, a4 + adds u5, u5, l + umulh l, a2, a3 + adcs u6, u6, l + umulh l, a2, a4 + adcs u7, u7, l + umulh l, a2, a5 + adcs u8, u8, l + umulh l, a3, a5 + adcs u9, u9, l + adc u10, u10, xzr + +// Double that, with u11 holding the top carry + + adds u1, u1, u1 + adcs u2, u2, u2 + adcs u3, u3, u3 + adcs u4, u4, u4 + adcs u5, u5, u5 + adcs u6, u6, u6 + adcs u7, u7, u7 + adcs u8, u8, u8 + adcs u9, u9, u9 + adcs u10, u10, u10 + cset u11, cs + +// Add the homogeneous terms 00 + 11 + 22 + 33 + 44 + 55 + + umulh l, a0, a0 + mul u0, a0, a0 + adds u1, u1, l + + mul l, a1, a1 + adcs u2, u2, l + umulh l, a1, a1 + adcs u3, u3, l + + mul l, a2, a2 + adcs u4, u4, l + umulh l, a2, a2 + adcs u5, u5, l + + mul l, a3, a3 + adcs u6, u6, l + umulh l, a3, a3 + adcs u7, u7, l + + mul l, a4, a4 + adcs u8, u8, l + umulh l, a4, a4 + adcs u9, u9, l + + mul l, a5, a5 + adcs u10, u10, l + umulh l, a5, a5 + adc u11, u11, l + +// Montgomery rotate the low half + + montreds(u0,u5,u4,u3,u2,u1,u0, a1,a2,a3) + montreds(u1,u0,u5,u4,u3,u2,u1, a1,a2,a3) + montreds(u2,u1,u0,u5,u4,u3,u2, a1,a2,a3) + montreds(u3,u2,u1,u0,u5,u4,u3, a1,a2,a3) + montreds(u4,u3,u2,u1,u0,u5,u4, a1,a2,a3) + montreds(u5,u4,u3,u2,u1,u0,u5, a1,a2,a3) + +// Add up the high and low parts as [h; u5;u4;u3;u2;u1;u0] = z + + adds u0, u0, u6 + adcs u1, u1, u7 + adcs u2, u2, u8 + adcs u3, u3, u9 + adcs u4, u4, u10 + adcs u5, u5, u11 + adc h, xzr, xzr + +// Now add [h; u11;u10;u9;u8;u7;u6] = z + (2^384 - p_384) + + mov l, #0xffffffff00000001 + adds u6, u0, l + mov l, #0x00000000ffffffff + adcs u7, u1, l + mov l, #0x0000000000000001 + adcs u8, u2, l + adcs u9, u3, xzr + adcs u10, u4, xzr + adcs u11, u5, xzr + adcs h, h, xzr + +// Now z >= p_384 iff h is nonzero, so select accordingly + + csel u0, u0, u6, eq + csel u1, u1, u7, eq + csel u2, u2, u8, eq + csel u3, u3, u9, eq + csel u4, u4, u10, eq + csel u5, u5, u11, eq + +// Store back final result + + stp u0, u1, [z] + stp u2, u3, [z, #16] + stp u4, u5, [z, #32] + +// Restore registers + + CFI_POP2(x19,x20) + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_montsqr_p384_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/bignum_mul_p521.S b/cbits/s2n/arm/bignum_mul_p521.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_mul_p521.S @@ -0,0 +1,1407 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Multiply modulo p_521, z := (x * y) mod p_521, assuming x and y reduced +// Inputs x[9], y[9]; output z[9] +// +// extern void bignum_mul_p521(uint64_t z[static 9], const uint64_t x[static 9], +// const uint64_t y[static 9]); +// +// Standard ARM ABI: X0 = z, X1 = x, X2 = y +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + +// bignum_mul_p521 is functionally equivalent to unopt/bignum_mul_p521_base. +// It is written in a way that +// 1. A subset of scalar multiplications in bignum_montmul_p384 are carefully +// chosen and vectorized +// 2. The vectorized assembly is rescheduled using the SLOTHY superoptimizer. +// https://github.com/slothy-optimizer/slothy +// +// The output program of step 1. is as follows: +// +// stp x19, x20, [sp, #-16]! +// stp x21, x22, [sp, #-16]! +// stp x23, x24, [sp, #-16]! +// stp x25, x26, [sp, #-16]! +// sub sp, sp, #80 +// ldp x15, x21, [x1] +// ldp x10, x17, [x1, #16] +// ldp x13, x16, [x2] +// ldr q18, [x1] +// ldr q28, [x2] +// ldp x5, x20, [x2, #16] +// movi v16.2D, #0x00000000ffffffff +// uzp2 v7.4S, v28.4S, v28.4S +// xtn v4.2S, v18.2D +// xtn v1.2S, v28.2D +// rev64 v27.4S, v28.4S +// umull v21.2D, v4.2S, v1.2S +// umull v28.2D, v4.2S, v7.2S +// uzp2 v5.4S, v18.4S, v18.4S +// mul v18.4S, v27.4S, v18.4S +// usra v28.2D, v21.2D, #32 +// umull v29.2D, v5.2S, v7.2S +// uaddlp v18.2D, v18.4S +// and v16.16B, v28.16B, v16.16B +// umlal v16.2D, v5.2S, v1.2S +// shl v18.2D, v18.2D, #32 +// usra v29.2D, v28.2D, #32 +// umlal v18.2D, v4.2S, v1.2S +// usra v29.2D, v16.2D, #32 +// mov x8, v18.d[0] +// mov x9, v18.d[1] +// mul x6, x10, x5 +// mul x19, x17, x20 +// mov x14, v29.d[0] +// adds x9, x9, x14 +// mov x14, v29.d[1] +// adcs x6, x6, x14 +// umulh x14, x10, x5 +// adcs x19, x19, x14 +// umulh x14, x17, x20 +// adc x14, x14, xzr +// adds x11, x9, x8 +// adcs x9, x6, x9 +// adcs x6, x19, x6 +// adcs x19, x14, x19 +// adc x14, xzr, x14 +// adds x3, x9, x8 +// adcs x24, x6, x11 +// adcs x9, x19, x9 +// adcs x6, x14, x6 +// adcs x19, xzr, x19 +// adc x14, xzr, x14 +// subs x4, x10, x17 +// cneg x4, x4, cc +// csetm x7, cc +// subs x23, x20, x5 +// cneg x23, x23, cc +// mul x22, x4, x23 +// umulh x4, x4, x23 +// cinv x7, x7, cc +// cmn x7, #0x1 +// eor x23, x22, x7 +// adcs x6, x6, x23 +// eor x4, x4, x7 +// adcs x19, x19, x4 +// adc x14, x14, x7 +// subs x4, x15, x21 +// cneg x4, x4, cc +// csetm x7, cc +// subs x23, x16, x13 +// cneg x23, x23, cc +// mul x22, x4, x23 +// umulh x4, x4, x23 +// cinv x7, x7, cc +// cmn x7, #0x1 +// eor x23, x22, x7 +// adcs x11, x11, x23 +// eor x4, x4, x7 +// adcs x3, x3, x4 +// adcs x24, x24, x7 +// adcs x9, x9, x7 +// adcs x6, x6, x7 +// adcs x19, x19, x7 +// adc x14, x14, x7 +// subs x4, x21, x17 +// cneg x4, x4, cc +// csetm x7, cc +// subs x23, x20, x16 +// cneg x23, x23, cc +// mul x22, x4, x23 +// umulh x4, x4, x23 +// cinv x7, x7, cc +// cmn x7, #0x1 +// eor x23, x22, x7 +// adcs x9, x9, x23 +// eor x4, x4, x7 +// adcs x6, x6, x4 +// adcs x19, x19, x7 +// adc x14, x14, x7 +// subs x4, x15, x10 +// cneg x4, x4, cc +// csetm x7, cc +// subs x23, x5, x13 +// cneg x23, x23, cc +// mul x22, x4, x23 +// umulh x4, x4, x23 +// cinv x7, x7, cc +// cmn x7, #0x1 +// eor x23, x22, x7 +// adcs x3, x3, x23 +// eor x4, x4, x7 +// adcs x24, x24, x4 +// adcs x9, x9, x7 +// adcs x6, x6, x7 +// adcs x19, x19, x7 +// adc x14, x14, x7 +// subs x17, x15, x17 +// cneg x17, x17, cc +// csetm x4, cc +// subs x13, x20, x13 +// cneg x13, x13, cc +// mul x20, x17, x13 +// umulh x17, x17, x13 +// cinv x13, x4, cc +// cmn x13, #0x1 +// eor x20, x20, x13 +// adcs x20, x24, x20 +// eor x17, x17, x13 +// adcs x17, x9, x17 +// adcs x9, x6, x13 +// adcs x6, x19, x13 +// adc x13, x14, x13 +// subs x21, x21, x10 +// cneg x21, x21, cc +// csetm x10, cc +// subs x16, x5, x16 +// cneg x16, x16, cc +// mul x5, x21, x16 +// umulh x21, x21, x16 +// cinv x10, x10, cc +// cmn x10, #0x1 +// eor x16, x5, x10 +// adcs x16, x20, x16 +// eor x21, x21, x10 +// adcs x21, x17, x21 +// adcs x17, x9, x10 +// adcs x5, x6, x10 +// adc x10, x13, x10 +// lsl x13, x8, #9 +// extr x20, x11, x8, #55 +// extr x8, x3, x11, #55 +// extr x9, x16, x3, #55 +// lsr x16, x16, #55 +// stp x21, x17, [sp] // @slothy:writes=stack0 +// stp x5, x10, [sp, #16] // @slothy:writes=stack16 +// stp x13, x20, [sp, #32] // @slothy:writes=stack32 +// stp x8, x9, [sp, #48] // @slothy:writes=stack48 +// str x16, [sp, #64] // @slothy:writes=stack64 +// ldp x21, x10, [x1, #32] +// ldp x17, x13, [x1, #48] +// ldp x16, x5, [x2, #32] +// ldr q18, [x1, #32] +// ldr q28, [x2, #32] +// ldp x20, x8, [x2, #48] +// movi v16.2D, #0x00000000ffffffff +// uzp2 v7.4S, v28.4S, v28.4S +// xtn v4.2S, v18.2D +// xtn v1.2S, v28.2D +// rev64 v28.4S, v28.4S +// umull v27.2D, v4.2S, v1.2S +// umull v29.2D, v4.2S, v7.2S +// uzp2 v21.4S, v18.4S, v18.4S +// mul v28.4S, v28.4S, v18.4S +// usra v29.2D, v27.2D, #32 +// umull v18.2D, v21.2S, v7.2S +// uaddlp v28.2D, v28.4S +// and v16.16B, v29.16B, v16.16B +// umlal v16.2D, v21.2S, v1.2S +// shl v28.2D, v28.2D, #32 +// usra v18.2D, v29.2D, #32 +// umlal v28.2D, v4.2S, v1.2S +// usra v18.2D, v16.2D, #32 +// mov x9, v28.d[0] +// mov x6, v28.d[1] +// mul x19, x17, x20 +// mul x14, x13, x8 +// mov x11, v18.d[0] +// adds x6, x6, x11 +// mov x11, v18.d[1] +// adcs x19, x19, x11 +// umulh x11, x17, x20 +// adcs x14, x14, x11 +// umulh x11, x13, x8 +// adc x11, x11, xzr +// adds x3, x6, x9 +// adcs x6, x19, x6 +// adcs x19, x14, x19 +// adcs x14, x11, x14 +// adc x11, xzr, x11 +// adds x24, x6, x9 +// adcs x4, x19, x3 +// adcs x6, x14, x6 +// adcs x19, x11, x19 +// adcs x14, xzr, x14 +// adc x11, xzr, x11 +// subs x7, x17, x13 +// cneg x7, x7, cc +// csetm x23, cc +// subs x22, x8, x20 +// cneg x22, x22, cc +// mul x12, x7, x22 +// umulh x7, x7, x22 +// cinv x23, x23, cc +// cmn x23, #0x1 +// eor x22, x12, x23 +// adcs x19, x19, x22 +// eor x7, x7, x23 +// adcs x14, x14, x7 +// adc x11, x11, x23 +// subs x7, x21, x10 +// cneg x7, x7, cc +// csetm x23, cc +// subs x22, x5, x16 +// cneg x22, x22, cc +// mul x12, x7, x22 +// umulh x7, x7, x22 +// cinv x23, x23, cc +// cmn x23, #0x1 +// eor x22, x12, x23 +// adcs x3, x3, x22 +// eor x7, x7, x23 +// adcs x24, x24, x7 +// adcs x4, x4, x23 +// adcs x6, x6, x23 +// adcs x19, x19, x23 +// adcs x14, x14, x23 +// adc x11, x11, x23 +// subs x7, x10, x13 +// cneg x7, x7, cc +// csetm x23, cc +// subs x22, x8, x5 +// cneg x22, x22, cc +// mul x12, x7, x22 +// umulh x7, x7, x22 +// cinv x23, x23, cc +// cmn x23, #0x1 +// eor x22, x12, x23 +// adcs x6, x6, x22 +// eor x7, x7, x23 +// adcs x19, x19, x7 +// adcs x14, x14, x23 +// adc x11, x11, x23 +// subs x7, x21, x17 +// cneg x7, x7, cc +// csetm x23, cc +// subs x22, x20, x16 +// cneg x22, x22, cc +// mul x12, x7, x22 +// umulh x7, x7, x22 +// cinv x23, x23, cc +// cmn x23, #0x1 +// eor x22, x12, x23 +// adcs x24, x24, x22 +// eor x7, x7, x23 +// adcs x4, x4, x7 +// adcs x6, x6, x23 +// adcs x19, x19, x23 +// adcs x14, x14, x23 +// adc x11, x11, x23 +// subs x7, x21, x13 +// cneg x7, x7, cc +// csetm x23, cc +// subs x22, x8, x16 +// cneg x22, x22, cc +// mul x12, x7, x22 +// umulh x7, x7, x22 +// cinv x23, x23, cc +// cmn x23, #0x1 +// eor x22, x12, x23 +// adcs x4, x4, x22 +// eor x7, x7, x23 +// adcs x6, x6, x7 +// adcs x19, x19, x23 +// adcs x14, x14, x23 +// adc x11, x11, x23 +// subs x7, x10, x17 +// cneg x7, x7, cc +// csetm x23, cc +// subs x22, x20, x5 +// cneg x22, x22, cc +// mul x12, x7, x22 +// umulh x7, x7, x22 +// cinv x23, x23, cc +// cmn x23, #0x1 +// eor x22, x12, x23 +// adcs x4, x4, x22 +// eor x7, x7, x23 +// adcs x6, x6, x7 +// adcs x19, x19, x23 +// adcs x14, x14, x23 +// adc x11, x11, x23 +// ldp x7, x23, [sp] // @slothy:reads=stack0 +// adds x9, x9, x7 +// adcs x3, x3, x23 +// stp x9, x3, [sp] // @slothy:writes=stack0 +// ldp x9, x3, [sp, #16] // @slothy:reads=stack16 +// adcs x9, x24, x9 +// adcs x3, x4, x3 +// stp x9, x3, [sp, #16] // @slothy:writes=stack16 +// ldp x9, x3, [sp, #32] // @slothy:reads=stack32 +// adcs x9, x6, x9 +// adcs x6, x19, x3 +// stp x9, x6, [sp, #32] // @slothy:writes=stack32 +// ldp x9, x6, [sp, #48] // @slothy:reads=stack48 +// adcs x9, x14, x9 +// adcs x6, x11, x6 +// stp x9, x6, [sp, #48] // @slothy:writes=stack48 +// ldr x9, [sp, #64] // @slothy:reads=stack64 +// adc x9, x9, xzr +// str x9, [sp, #64] // @slothy:writes=stack64 +// ldp x9, x6, [x1] +// subs x21, x21, x9 +// sbcs x10, x10, x6 +// ldp x9, x6, [x1, #16] +// sbcs x17, x17, x9 +// sbcs x13, x13, x6 +// csetm x9, cc +// ldp x6, x19, [x2] +// subs x16, x6, x16 +// sbcs x5, x19, x5 +// ldp x6, x19, [x2, #16] +// sbcs x20, x6, x20 +// sbcs x8, x19, x8 +// csetm x6, cc +// eor x21, x21, x9 +// subs x21, x21, x9 +// eor x10, x10, x9 +// sbcs x10, x10, x9 +// eor x17, x17, x9 +// sbcs x17, x17, x9 +// eor x13, x13, x9 +// sbc x13, x13, x9 +// eor x16, x16, x6 +// subs x16, x16, x6 +// eor x5, x5, x6 +// sbcs x5, x5, x6 +// eor x20, x20, x6 +// sbcs x20, x20, x6 +// eor x8, x8, x6 +// sbc x8, x8, x6 +// eor x9, x6, x9 +// mul x6, x21, x16 +// mul x19, x10, x5 +// mul x14, x17, x20 +// mul x11, x13, x8 +// umulh x3, x21, x16 +// adds x19, x19, x3 +// umulh x3, x10, x5 +// adcs x14, x14, x3 +// umulh x3, x17, x20 +// adcs x11, x11, x3 +// umulh x3, x13, x8 +// adc x3, x3, xzr +// adds x24, x19, x6 +// adcs x19, x14, x19 +// adcs x14, x11, x14 +// adcs x11, x3, x11 +// adc x3, xzr, x3 +// adds x4, x19, x6 +// adcs x7, x14, x24 +// adcs x19, x11, x19 +// adcs x14, x3, x14 +// adcs x11, xzr, x11 +// adc x3, xzr, x3 +// subs x23, x17, x13 +// cneg x23, x23, cc +// csetm x22, cc +// subs x12, x8, x20 +// cneg x12, x12, cc +// mul x15, x23, x12 +// umulh x23, x23, x12 +// cinv x22, x22, cc +// cmn x22, #0x1 +// eor x12, x15, x22 +// adcs x14, x14, x12 +// eor x23, x23, x22 +// adcs x11, x11, x23 +// adc x3, x3, x22 +// subs x23, x21, x10 +// cneg x23, x23, cc +// csetm x22, cc +// subs x12, x5, x16 +// cneg x12, x12, cc +// mul x15, x23, x12 +// umulh x23, x23, x12 +// cinv x22, x22, cc +// cmn x22, #0x1 +// eor x12, x15, x22 +// adcs x24, x24, x12 +// eor x23, x23, x22 +// adcs x4, x4, x23 +// adcs x7, x7, x22 +// adcs x19, x19, x22 +// adcs x14, x14, x22 +// adcs x11, x11, x22 +// adc x3, x3, x22 +// subs x23, x10, x13 +// cneg x23, x23, cc +// csetm x22, cc +// subs x12, x8, x5 +// cneg x12, x12, cc +// mul x15, x23, x12 +// umulh x23, x23, x12 +// cinv x22, x22, cc +// cmn x22, #0x1 +// eor x12, x15, x22 +// adcs x19, x19, x12 +// eor x23, x23, x22 +// adcs x14, x14, x23 +// adcs x11, x11, x22 +// adc x3, x3, x22 +// subs x23, x21, x17 +// cneg x23, x23, cc +// csetm x22, cc +// subs x12, x20, x16 +// cneg x12, x12, cc +// mul x15, x23, x12 +// umulh x23, x23, x12 +// cinv x22, x22, cc +// cmn x22, #0x1 +// eor x12, x15, x22 +// adcs x4, x4, x12 +// eor x23, x23, x22 +// adcs x7, x7, x23 +// adcs x19, x19, x22 +// adcs x14, x14, x22 +// adcs x11, x11, x22 +// adc x3, x3, x22 +// subs x21, x21, x13 +// cneg x21, x21, cc +// csetm x13, cc +// subs x16, x8, x16 +// cneg x16, x16, cc +// mul x8, x21, x16 +// umulh x21, x21, x16 +// cinv x13, x13, cc +// cmn x13, #0x1 +// eor x16, x8, x13 +// adcs x16, x7, x16 +// eor x21, x21, x13 +// adcs x21, x19, x21 +// adcs x8, x14, x13 +// adcs x19, x11, x13 +// adc x13, x3, x13 +// subs x10, x10, x17 +// cneg x10, x10, cc +// csetm x17, cc +// subs x5, x20, x5 +// cneg x5, x5, cc +// mul x20, x10, x5 +// umulh x10, x10, x5 +// cinv x17, x17, cc +// cmn x17, #0x1 +// eor x5, x20, x17 +// adcs x16, x16, x5 +// eor x10, x10, x17 +// adcs x21, x21, x10 +// adcs x10, x8, x17 +// adcs x5, x19, x17 +// adc x17, x13, x17 +// ldp x13, x20, [sp] // @slothy:reads=stack0 +// ldp x8, x19, [sp, #16] // @slothy:reads=stack16 +// eor x6, x6, x9 +// adds x6, x6, x13 +// eor x14, x24, x9 +// adcs x14, x14, x20 +// eor x11, x4, x9 +// adcs x11, x11, x8 +// eor x16, x16, x9 +// adcs x16, x16, x19 +// eor x21, x21, x9 +// ldp x3, x24, [sp, #32] // @slothy:reads=stack32 +// ldp x4, x7, [sp, #48] // @slothy:reads=stack48 +// ldr x23, [sp, #64] // @slothy:reads=stack64 +// adcs x21, x21, x3 +// eor x10, x10, x9 +// adcs x10, x10, x24 +// eor x5, x5, x9 +// adcs x5, x5, x4 +// eor x17, x17, x9 +// adcs x17, x17, x7 +// adc x22, x23, xzr +// adds x21, x21, x13 +// adcs x10, x10, x20 +// adcs x13, x5, x8 +// adcs x17, x17, x19 +// and x5, x9, #0x1ff +// lsl x20, x6, #9 +// orr x5, x20, x5 +// adcs x5, x3, x5 +// extr x20, x14, x6, #55 +// adcs x20, x24, x20 +// extr x8, x11, x14, #55 +// adcs x8, x4, x8 +// extr x9, x16, x11, #55 +// adcs x9, x7, x9 +// lsr x16, x16, #55 +// adc x16, x16, x23 +// ldr x6, [x2, #64] +// ldp x19, x14, [x1] +// and x11, x19, #0xfffffffffffff +// mul x11, x6, x11 +// ldr x3, [x1, #64] +// ldp x24, x4, [x2] +// and x7, x24, #0xfffffffffffff +// mul x7, x3, x7 +// add x11, x11, x7 +// extr x19, x14, x19, #52 +// and x19, x19, #0xfffffffffffff +// mul x19, x6, x19 +// extr x24, x4, x24, #52 +// and x24, x24, #0xfffffffffffff +// mul x24, x3, x24 +// add x19, x19, x24 +// lsr x24, x11, #52 +// add x19, x19, x24 +// lsl x11, x11, #12 +// extr x11, x19, x11, #12 +// adds x21, x21, x11 +// ldp x11, x24, [x1, #16] +// ldp x7, x23, [x2, #16] +// extr x14, x11, x14, #40 +// and x14, x14, #0xfffffffffffff +// mul x14, x6, x14 +// extr x4, x7, x4, #40 +// and x4, x4, #0xfffffffffffff +// mul x4, x3, x4 +// add x14, x14, x4 +// lsr x4, x19, #52 +// add x14, x14, x4 +// lsl x19, x19, #12 +// extr x19, x14, x19, #24 +// adcs x10, x10, x19 +// extr x19, x24, x11, #28 +// and x19, x19, #0xfffffffffffff +// mul x19, x6, x19 +// extr x11, x23, x7, #28 +// and x11, x11, #0xfffffffffffff +// mul x11, x3, x11 +// add x19, x19, x11 +// lsr x11, x14, #52 +// add x19, x19, x11 +// lsl x14, x14, #12 +// extr x14, x19, x14, #36 +// adcs x13, x13, x14 +// and x14, x10, x13 +// ldp x11, x4, [x1, #32] +// ldp x7, x12, [x2, #32] +// extr x24, x11, x24, #16 +// and x24, x24, #0xfffffffffffff +// mul x24, x6, x24 +// extr x23, x7, x23, #16 +// and x23, x23, #0xfffffffffffff +// mul x23, x3, x23 +// add x24, x24, x23 +// lsl x23, x22, #48 +// add x24, x24, x23 +// lsr x23, x19, #52 +// add x24, x24, x23 +// lsl x19, x19, #12 +// extr x19, x24, x19, #48 +// adcs x17, x17, x19 +// and x19, x14, x17 +// lsr x14, x11, #4 +// and x14, x14, #0xfffffffffffff +// mul x14, x6, x14 +// lsr x23, x7, #4 +// and x23, x23, #0xfffffffffffff +// mul x23, x3, x23 +// add x14, x14, x23 +// lsr x23, x24, #52 +// add x14, x14, x23 +// lsl x24, x24, #12 +// extr x24, x14, x24, #60 +// extr x11, x4, x11, #56 +// and x11, x11, #0xfffffffffffff +// mul x11, x6, x11 +// extr x7, x12, x7, #56 +// and x7, x7, #0xfffffffffffff +// mul x7, x3, x7 +// add x11, x11, x7 +// lsr x14, x14, #52 +// add x14, x11, x14 +// lsl x11, x24, #8 +// extr x11, x14, x11, #8 +// adcs x5, x5, x11 +// and x19, x19, x5 +// ldp x11, x24, [x1, #48] +// ldp x2, x7, [x2, #48] +// extr x4, x11, x4, #44 +// and x4, x4, #0xfffffffffffff +// mul x4, x6, x4 +// extr x23, x2, x12, #44 +// and x23, x23, #0xfffffffffffff +// mul x23, x3, x23 +// add x4, x4, x23 +// lsr x23, x14, #52 +// add x4, x4, x23 +// lsl x14, x14, #12 +// extr x14, x4, x14, #20 +// adcs x20, x20, x14 +// and x19, x19, x20 +// extr x14, x24, x11, #32 +// and x14, x14, #0xfffffffffffff +// mul x14, x6, x14 +// extr x2, x7, x2, #32 +// and x2, x2, #0xfffffffffffff +// mul x2, x3, x2 +// add x2, x14, x2 +// lsr x14, x4, #52 +// add x2, x2, x14 +// lsl x14, x4, #12 +// extr x14, x2, x14, #32 +// adcs x8, x8, x14 +// and x19, x19, x8 +// lsr x14, x24, #20 +// mul x14, x6, x14 +// lsr x11, x7, #20 +// mul x11, x3, x11 +// add x14, x14, x11 +// lsr x11, x2, #52 +// add x14, x14, x11 +// lsl x2, x2, #12 +// extr x2, x14, x2, #44 +// adcs x9, x9, x2 +// and x2, x19, x9 +// mul x6, x6, x3 +// lsr x19, x14, #44 +// add x6, x6, x19 +// adc x16, x16, x6 +// lsr x6, x16, #9 +// orr x16, x16, #0xfffffffffffffe00 +// cmp xzr, xzr +// adcs xzr, x21, x6 +// adcs xzr, x2, xzr +// adcs xzr, x16, xzr +// adcs x21, x21, x6 +// adcs x10, x10, xzr +// adcs x13, x13, xzr +// adcs x17, x17, xzr +// adcs x5, x5, xzr +// adcs x20, x20, xzr +// adcs x8, x8, xzr +// adcs x9, x9, xzr +// adc x16, x16, xzr +// and x2, x21, #0x1ff +// extr x21, x10, x21, #9 +// extr x10, x13, x10, #9 +// stp x21, x10, [x0] // @slothy:writes=buffer0 +// extr x21, x17, x13, #9 +// extr x10, x5, x17, #9 +// stp x21, x10, [x0, #16] // @slothy:writes=buffer16 +// extr x21, x20, x5, #9 +// extr x10, x8, x20, #9 +// stp x21, x10, [x0, #32] // @slothy:writes=buffer32 +// extr x21, x9, x8, #9 +// extr x10, x16, x9, #9 +// stp x21, x10, [x0, #48] // @slothy:writes=buffer48 +// str x2, [x0, #64] // @slothy:writes=buffer64 +// add sp, sp, #80 +// ldp x25, x26, [sp], #16 +// ldp x23, x24, [sp], #16 +// ldp x21, x22, [sp], #16 +// ldp x19, x20, [sp], #16 +// ret +// +// The bash script used for step 2 is as follows: +// +// # Store the assembly instructions except the last 'ret', +// # callee-register store/loads and add/sub sp #80 as, say, 'input.S'. +// export OUTPUTS="[hint_buffer0,hint_buffer16,hint_buffer32,hint_buffer48,hint_buffer64]" +// export RESERVED_REGS="[x18,x27,x28,x29,x30,sp,q8,q9,q10,q11,q12,q13,q14,q15,v8,v9,v10,v11,v12,v13,v14,v15]" +// /tools/external/slothy.sh input.S my_out_dir +// # my_out_dir/3.opt.s is the optimized assembly. Its output may differ +// # from this file since the sequence is non-deterministically chosen. +// # Please add 'ret' at the end of the output assembly. + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_mul_p521) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_mul_p521) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_mul_p521) + .text + .balign 4 + +S2N_BN_SYMBOL(bignum_mul_p521): + CFI_START + +// Save registers and make space for the temporary buffer + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x26) + CFI_DEC_SP(80) + + ldr q6, [x2] + ldp x10, x17, [x1, #16] + ldr q4, [x1] + ldr q16, [x2, #32] + ldp x5, x20, [x2, #16] + ldr q2, [x1, #32] + movi v31.2D, #0x00000000ffffffff + uzp2 v17.4S, v6.4S, v6.4S + rev64 v7.4S, v6.4S + ldp x15, x21, [x1] + xtn v25.2S, v6.2D + xtn v22.2S, v4.2D + subs x14, x10, x17 + mul v7.4S, v7.4S, v4.4S + csetm x8, cc + rev64 v3.4S, v16.4S + xtn v1.2S, v16.2D + ldp x13, x16, [x2] + mul x26, x10, x5 + uzp2 v16.4S, v16.4S, v16.4S + uaddlp v26.2D, v7.4S + cneg x4, x14, cc + subs x24, x15, x21 + xtn v5.2S, v2.2D + mul v28.4S, v3.4S, v2.4S + shl v26.2D, v26.2D, #32 + mul x22, x17, x20 + umull v20.2D, v22.2S, v25.2S + uzp2 v6.4S, v4.4S, v4.4S + umull v18.2D, v22.2S, v17.2S + uzp2 v4.4S, v2.4S, v2.4S + cneg x14, x24, cc + csetm x7, cc + umulh x11, x17, x20 + usra v18.2D, v20.2D, #32 + uaddlp v7.2D, v28.4S + subs x19, x16, x13 + umlal v26.2D, v22.2S, v25.2S + cneg x19, x19, cc + shl v28.2D, v7.2D, #32 + umull v7.2D, v5.2S, v1.2S + umull v30.2D, v5.2S, v16.2S + cinv x6, x7, cc + mul x25, x14, x19 + umlal v28.2D, v5.2S, v1.2S + umull v21.2D, v6.2S, v17.2S + umulh x14, x14, x19 + usra v30.2D, v7.2D, #32 + subs x9, x20, x5 + and v29.16B, v18.16B, v31.16B + cinv x23, x8, cc + mov x8, v26.d[1] + cneg x12, x9, cc + usra v21.2D, v18.2D, #32 + umlal v29.2D, v6.2S, v25.2S + mul x24, x4, x12 + umull v18.2D, v4.2S, v16.2S + movi v25.2D, #0x00000000ffffffff + eor x9, x14, x6 + and v7.16B, v30.16B, v25.16B + usra v21.2D, v29.2D, #32 + umulh x7, x10, x5 + usra v18.2D, v30.2D, #32 + umlal v7.2D, v4.2S, v1.2S + mov x19, v21.d[0] + umulh x3, x4, x12 + mov x14, v21.d[1] + usra v18.2D, v7.2D, #32 + adds x4, x8, x19 + mov x8, v26.d[0] + adcs x19, x26, x14 + adcs x14, x22, x7 + adc x12, x11, xzr + adds x11, x4, x8 + adcs x26, x19, x4 + adcs x22, x14, x19 + eor x4, x24, x23 + adcs x14, x12, x14 + eor x7, x25, x6 + adc x25, xzr, x12 + eor x19, x3, x23 + adds x3, x26, x8 + adcs x24, x22, x11 + adcs x12, x14, x26 + adcs x22, x25, x22 + adcs x26, xzr, x14 + adc x14, xzr, x25 + cmn x23, #0x1 + adcs x22, x22, x4 + adcs x19, x26, x19 + adc x25, x14, x23 + subs x14, x21, x17 + cneg x23, x14, cc + csetm x26, cc + subs x4, x20, x16 + cneg x14, x4, cc + cinv x4, x26, cc + cmn x6, #0x1 + adcs x11, x11, x7 + mul x7, x23, x14 + adcs x9, x3, x9 + adcs x26, x24, x6 + umulh x3, x23, x14 + adcs x14, x12, x6 + adcs x22, x22, x6 + adcs x12, x19, x6 + extr x24, x11, x8, #55 + adc x6, x25, x6 + subs x19, x15, x17 + csetm x17, cc + cneg x23, x19, cc + subs x19, x20, x13 + lsl x25, x8, #9 + eor x8, x7, x4 + cneg x20, x19, cc + umulh x7, x23, x20 + cinv x19, x17, cc + subs x17, x15, x10 + csetm x15, cc + stp x25, x24, [sp, #32] + cneg x24, x17, cc + mul x20, x23, x20 + subs x25, x5, x13 + cneg x13, x25, cc + cinv x15, x15, cc + mul x25, x24, x13 + subs x21, x21, x10 + csetm x23, cc + cneg x17, x21, cc + subs x21, x5, x16 + umulh x13, x24, x13 + cinv x10, x23, cc + cneg x23, x21, cc + cmn x4, #0x1 + adcs x14, x14, x8 + eor x21, x3, x4 + adcs x21, x22, x21 + eor x5, x20, x19 + adcs x24, x12, x4 + mul x12, x17, x23 + eor x8, x25, x15 + adc x25, x6, x4 + cmn x15, #0x1 + adcs x6, x9, x8 + ldp x20, x8, [x2, #48] + eor x9, x13, x15 + adcs x4, x26, x9 + umulh x26, x17, x23 + ldp x17, x13, [x1, #48] + adcs x9, x14, x15 + adcs x16, x21, x15 + adcs x14, x24, x15 + eor x21, x7, x19 + mul x23, x17, x20 + adc x24, x25, x15 + cmn x19, #0x1 + adcs x7, x4, x5 + adcs x9, x9, x21 + umulh x3, x13, x8 + adcs x16, x16, x19 + adcs x22, x14, x19 + eor x5, x12, x10 + adc x12, x24, x19 + cmn x10, #0x1 + adcs x19, x7, x5 + eor x14, x26, x10 + mov x7, v28.d[1] + adcs x24, x9, x14 + extr x4, x19, x6, #55 + umulh x15, x17, x20 + mov x14, v18.d[1] + lsr x9, x19, #55 + adcs x5, x16, x10 + mov x16, v18.d[0] + adcs x19, x22, x10 + str x9, [sp, #64] + extr x25, x6, x11, #55 + adc x21, x12, x10 + subs x26, x17, x13 + stp x25, x4, [sp, #48] + stp x19, x21, [sp, #16] + csetm x6, cc + cneg x4, x26, cc + mul x19, x13, x8 + subs x11, x8, x20 + stp x24, x5, [sp] + ldp x21, x10, [x1, #32] + cinv x12, x6, cc + cneg x6, x11, cc + mov x9, v28.d[0] + umulh x25, x4, x6 + adds x22, x7, x16 + ldp x16, x5, [x2, #32] + adcs x14, x23, x14 + adcs x11, x19, x15 + adc x24, x3, xzr + adds x3, x22, x9 + adcs x15, x14, x22 + mul x22, x4, x6 + adcs x6, x11, x14 + adcs x4, x24, x11 + eor x14, x25, x12 + adc x26, xzr, x24 + subs x7, x21, x10 + csetm x23, cc + cneg x19, x7, cc + subs x24, x5, x16 + cneg x11, x24, cc + cinv x7, x23, cc + adds x25, x15, x9 + eor x23, x22, x12 + adcs x22, x6, x3 + mul x24, x19, x11 + adcs x15, x4, x15 + adcs x6, x26, x6 + umulh x19, x19, x11 + adcs x11, xzr, x4 + adc x26, xzr, x26 + cmn x12, #0x1 + adcs x4, x6, x23 + eor x6, x24, x7 + adcs x14, x11, x14 + adc x26, x26, x12 + subs x11, x10, x13 + cneg x12, x11, cc + csetm x11, cc + eor x19, x19, x7 + subs x24, x8, x5 + cinv x11, x11, cc + cneg x24, x24, cc + cmn x7, #0x1 + adcs x3, x3, x6 + mul x23, x12, x24 + adcs x25, x25, x19 + adcs x6, x22, x7 + umulh x19, x12, x24 + adcs x22, x15, x7 + adcs x12, x4, x7 + eor x24, x23, x11 + adcs x4, x14, x7 + adc x26, x26, x7 + eor x19, x19, x11 + subs x14, x21, x17 + cneg x7, x14, cc + csetm x14, cc + subs x23, x20, x16 + cinv x14, x14, cc + cneg x23, x23, cc + cmn x11, #0x1 + adcs x22, x22, x24 + mul x24, x7, x23 + adcs x15, x12, x19 + adcs x4, x4, x11 + adc x19, x26, x11 + umulh x26, x7, x23 + subs x7, x21, x13 + eor x11, x24, x14 + cneg x23, x7, cc + csetm x12, cc + subs x7, x8, x16 + cneg x7, x7, cc + cinv x12, x12, cc + cmn x14, #0x1 + eor x26, x26, x14 + adcs x11, x25, x11 + mul x25, x23, x7 + adcs x26, x6, x26 + adcs x6, x22, x14 + adcs x24, x15, x14 + umulh x23, x23, x7 + adcs x4, x4, x14 + adc x22, x19, x14 + eor x14, x25, x12 + eor x7, x23, x12 + cmn x12, #0x1 + adcs x14, x26, x14 + ldp x19, x25, [x2] + ldp x15, x23, [x2, #16] + adcs x26, x6, x7 + adcs x24, x24, x12 + adcs x7, x4, x12 + adc x4, x22, x12 + subs x19, x19, x16 + ldp x16, x22, [x1] + sbcs x6, x25, x5 + ldp x12, x25, [x1, #16] + sbcs x15, x15, x20 + sbcs x8, x23, x8 + csetm x23, cc + subs x21, x21, x16 + eor x16, x19, x23 + sbcs x19, x10, x22 + eor x22, x6, x23 + eor x8, x8, x23 + sbcs x6, x17, x12 + sbcs x13, x13, x25 + csetm x12, cc + subs x10, x10, x17 + cneg x17, x10, cc + csetm x25, cc + subs x5, x20, x5 + eor x10, x19, x12 + cneg x19, x5, cc + eor x20, x15, x23 + eor x21, x21, x12 + cinv x15, x25, cc + mul x25, x17, x19 + subs x16, x16, x23 + sbcs x5, x22, x23 + eor x6, x6, x12 + sbcs x20, x20, x23 + eor x22, x13, x12 + sbc x8, x8, x23 + subs x21, x21, x12 + umulh x19, x17, x19 + sbcs x10, x10, x12 + sbcs x17, x6, x12 + eor x6, x19, x15 + eor x19, x25, x15 + umulh x25, x17, x20 + sbc x13, x22, x12 + cmn x15, #0x1 + adcs x22, x14, x19 + adcs x19, x26, x6 + ldp x6, x26, [sp] + adcs x14, x24, x15 + umulh x24, x21, x16 + adcs x7, x7, x15 + adc x15, x4, x15 + adds x4, x9, x6 + eor x9, x23, x12 + adcs x12, x3, x26 + stp x4, x12, [sp] + ldp x4, x26, [sp, #16] + umulh x12, x10, x5 + ldp x6, x23, [sp, #32] + adcs x3, x11, x4 + mul x4, x13, x8 + adcs x26, x22, x26 + ldp x22, x11, [sp, #48] + adcs x6, x19, x6 + stp x3, x26, [sp, #16] + mul x26, x10, x5 + adcs x14, x14, x23 + stp x6, x14, [sp, #32] + ldr x6, [sp, #64] + adcs x22, x7, x22 + adcs x14, x15, x11 + mul x11, x17, x20 + adc x19, x6, xzr + stp x22, x14, [sp, #48] + adds x14, x26, x24 + str x19, [sp, #64] + umulh x19, x13, x8 + adcs x7, x11, x12 + adcs x22, x4, x25 + mul x6, x21, x16 + adc x19, x19, xzr + subs x11, x17, x13 + cneg x12, x11, cc + csetm x11, cc + subs x24, x8, x20 + cinv x11, x11, cc + cneg x24, x24, cc + adds x4, x14, x6 + adcs x14, x7, x14 + mul x3, x12, x24 + adcs x7, x22, x7 + adcs x22, x19, x22 + umulh x12, x12, x24 + adc x24, xzr, x19 + adds x19, x14, x6 + eor x3, x3, x11 + adcs x26, x7, x4 + adcs x14, x22, x14 + adcs x25, x24, x7 + adcs x23, xzr, x22 + eor x7, x12, x11 + adc x12, xzr, x24 + subs x22, x21, x10 + cneg x24, x22, cc + csetm x22, cc + subs x15, x5, x16 + cinv x22, x22, cc + cneg x15, x15, cc + cmn x11, #0x1 + adcs x3, x25, x3 + mul x25, x24, x15 + adcs x23, x23, x7 + adc x11, x12, x11 + subs x7, x10, x13 + umulh x15, x24, x15 + cneg x12, x7, cc + csetm x7, cc + eor x24, x25, x22 + eor x25, x15, x22 + cmn x22, #0x1 + adcs x24, x4, x24 + adcs x19, x19, x25 + adcs x15, x26, x22 + adcs x4, x14, x22 + adcs x26, x3, x22 + adcs x25, x23, x22 + adc x23, x11, x22 + subs x14, x21, x17 + cneg x3, x14, cc + csetm x11, cc + subs x14, x8, x5 + cneg x14, x14, cc + cinv x7, x7, cc + subs x13, x21, x13 + cneg x21, x13, cc + csetm x13, cc + mul x22, x12, x14 + subs x8, x8, x16 + cinv x13, x13, cc + umulh x14, x12, x14 + cneg x12, x8, cc + subs x8, x20, x16 + cneg x8, x8, cc + cinv x16, x11, cc + eor x22, x22, x7 + cmn x7, #0x1 + eor x14, x14, x7 + adcs x4, x4, x22 + mul x11, x3, x8 + adcs x22, x26, x14 + adcs x14, x25, x7 + eor x25, x24, x9 + adc x26, x23, x7 + umulh x7, x3, x8 + subs x17, x10, x17 + cneg x24, x17, cc + eor x3, x11, x16 + csetm x11, cc + subs x20, x20, x5 + cneg x5, x20, cc + cinv x11, x11, cc + cmn x16, #0x1 + mul x17, x21, x12 + eor x8, x7, x16 + adcs x10, x19, x3 + and x19, x9, #0x1ff + adcs x20, x15, x8 + umulh x15, x21, x12 + eor x12, x10, x9 + eor x8, x6, x9 + adcs x6, x4, x16 + adcs x4, x22, x16 + adcs x21, x14, x16 + adc x7, x26, x16 + mul x10, x24, x5 + cmn x13, #0x1 + ldp x3, x14, [x1] + eor x17, x17, x13 + umulh x5, x24, x5 + adcs x20, x20, x17 + eor x17, x15, x13 + adcs x16, x6, x17 + eor x22, x10, x11 + adcs x23, x4, x13 + extr x10, x14, x3, #52 + and x26, x3, #0xfffffffffffff + adcs x24, x21, x13 + and x15, x10, #0xfffffffffffff + adc x6, x7, x13 + cmn x11, #0x1 + adcs x17, x20, x22 + eor x4, x5, x11 + ldp x21, x10, [sp] + adcs x7, x16, x4 + eor x16, x17, x9 + eor x13, x7, x9 + ldp x3, x17, [sp, #16] + adcs x7, x23, x11 + eor x23, x7, x9 + ldp x5, x22, [sp, #32] + adcs x7, x24, x11 + adc x24, x6, x11 + ldr x6, [x2, #64] + adds x20, x8, x21 + lsl x11, x20, #9 + eor x4, x7, x9 + orr x7, x11, x19 + eor x8, x24, x9 + adcs x11, x25, x10 + mul x26, x6, x26 + ldp x19, x24, [sp, #48] + adcs x12, x12, x3 + adcs x16, x16, x17 + adcs x9, x13, x5 + ldr x25, [sp, #64] + extr x20, x11, x20, #55 + adcs x13, x23, x22 + adcs x4, x4, x19 + extr x23, x12, x11, #55 + adcs x8, x8, x24 + adc x11, x25, xzr + adds x21, x9, x21 + extr x9, x16, x12, #55 + lsr x12, x16, #55 + adcs x10, x13, x10 + mul x15, x6, x15 + adcs x13, x4, x3 + ldp x16, x4, [x2] + ldr x3, [x1, #64] + adcs x17, x8, x17 + adcs x5, x5, x7 + adcs x20, x22, x20 + adcs x8, x19, x23 + and x22, x16, #0xfffffffffffff + ldp x19, x7, [x1, #16] + adcs x9, x24, x9 + extr x24, x4, x16, #52 + adc x16, x12, x25 + mul x22, x3, x22 + and x25, x24, #0xfffffffffffff + extr x14, x19, x14, #40 + and x12, x14, #0xfffffffffffff + extr x23, x7, x19, #28 + ldp x19, x24, [x2, #16] + mul x14, x3, x25 + and x23, x23, #0xfffffffffffff + add x22, x26, x22 + lsl x11, x11, #48 + lsr x26, x22, #52 + lsl x25, x22, #12 + mul x22, x6, x12 + extr x12, x19, x4, #40 + add x4, x15, x14 + mul x15, x6, x23 + add x4, x4, x26 + extr x23, x24, x19, #28 + ldp x14, x19, [x1, #32] + and x26, x12, #0xfffffffffffff + extr x12, x4, x25, #12 + and x25, x23, #0xfffffffffffff + adds x21, x21, x12 + mul x12, x3, x26 + extr x23, x14, x7, #16 + and x23, x23, #0xfffffffffffff + mul x7, x3, x25 + ldp x25, x26, [x2, #32] + add x12, x22, x12 + extr x22, x19, x14, #56 + mul x23, x6, x23 + lsr x14, x14, #4 + extr x24, x25, x24, #16 + add x7, x15, x7 + and x15, x24, #0xfffffffffffff + and x22, x22, #0xfffffffffffff + lsr x24, x4, #52 + mul x15, x3, x15 + and x14, x14, #0xfffffffffffff + add x12, x12, x24 + lsl x24, x4, #12 + lsr x4, x12, #52 + extr x24, x12, x24, #24 + adcs x10, x10, x24 + lsl x24, x12, #12 + add x12, x7, x4 + mul x22, x6, x22 + add x4, x23, x15 + extr x7, x12, x24, #36 + adcs x13, x13, x7 + lsl x15, x12, #12 + add x7, x4, x11 + lsr x24, x12, #52 + ldp x23, x11, [x2, #48] + add x4, x7, x24 + mul x12, x6, x14 + extr x7, x26, x25, #56 + extr x14, x4, x15, #48 + and x2, x7, #0xfffffffffffff + extr x24, x11, x23, #32 + ldp x15, x7, [x1, #48] + and x1, x24, #0xfffffffffffff + lsr x24, x4, #52 + mul x2, x3, x2 + extr x26, x23, x26, #44 + lsr x23, x25, #4 + and x23, x23, #0xfffffffffffff + and x25, x26, #0xfffffffffffff + extr x26, x7, x15, #32 + extr x19, x15, x19, #44 + mul x23, x3, x23 + and x15, x26, #0xfffffffffffff + lsl x26, x4, #12 + and x4, x19, #0xfffffffffffff + lsr x11, x11, #20 + mul x19, x6, x4 + adcs x17, x17, x14 + add x14, x22, x2 + add x22, x12, x23 + lsr x7, x7, #20 + add x22, x22, x24 + extr x2, x22, x26, #60 + mul x24, x3, x25 + lsr x22, x22, #52 + add x14, x14, x22 + lsl x22, x2, #8 + extr x22, x14, x22, #8 + lsl x2, x14, #12 + mul x1, x3, x1 + adcs x12, x5, x22 + mul x5, x6, x15 + and x26, x10, x13 + and x4, x26, x17 + add x23, x19, x24 + lsr x14, x14, #52 + mul x22, x3, x11 + add x11, x23, x14 + extr x25, x11, x2, #20 + lsl x19, x11, #12 + adcs x25, x20, x25 + and x14, x4, x12 + add x1, x5, x1 + and x14, x14, x25 + mul x15, x6, x7 + add x26, x15, x22 + mul x6, x6, x3 + lsr x22, x11, #52 + add x4, x1, x22 + lsr x1, x4, #52 + extr x3, x4, x19, #32 + lsl x15, x4, #12 + add x7, x26, x1 + adcs x23, x8, x3 + extr x20, x7, x15, #44 + and x3, x14, x23 + lsr x19, x7, #44 + adcs x7, x9, x20 + add x11, x6, x19 + adc x4, x16, x11 + lsr x14, x4, #9 + cmp xzr, xzr + and x15, x3, x7 + orr x3, x4, #0xfffffffffffffe00 + adcs xzr, x21, x14 + adcs xzr, x15, xzr + adcs xzr, x3, xzr + adcs x11, x21, x14 + and x14, x11, #0x1ff + adcs x1, x10, xzr + extr x10, x1, x11, #9 + str x14, [x0, #64] + adcs x14, x13, xzr + extr x11, x14, x1, #9 + adcs x1, x17, xzr + extr x4, x1, x14, #9 + stp x10, x11, [x0] + adcs x11, x12, xzr + extr x14, x11, x1, #9 + adcs x10, x25, xzr + extr x11, x10, x11, #9 + stp x4, x14, [x0, #16] + adcs x14, x23, xzr + extr x10, x14, x10, #9 + adcs x1, x7, xzr + stp x11, x10, [x0, #32] + extr x14, x1, x14, #9 + adc x10, x3, xzr + extr x26, x10, x1, #9 + stp x14, x26, [x0, #48] + +// Restore regs and return + + CFI_INC_SP(80) + CFI_POP2(x25,x26) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_mul_p521) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/bignum_mul_p521_alt.S b/cbits/s2n/arm/bignum_mul_p521_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_mul_p521_alt.S @@ -0,0 +1,538 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Multiply modulo p_521, z := (x * y) mod p_521, assuming x and y reduced +// Inputs x[9], y[9]; output z[9] +// +// extern void bignum_mul_p521_alt(uint64_t z[static 9], +// const uint64_t x[static 9], +// const uint64_t y[static 9]); +// +// Standard ARM ABI: X0 = z, X1 = x, X2 = y +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_mul_p521_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_mul_p521_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_mul_p521_alt) + .text + .balign 4 + +#define z x0 +#define x x1 +#define y x2 + +// These are repeated mod 2 as we load paris of inputs + +#define a0 x3 +#define a1 x4 +#define a2 x3 +#define a3 x4 +#define a4 x3 +#define a5 x4 +#define a6 x3 +#define a7 x4 +#define a8 x3 + +#define b0 x5 +#define b1 x6 +#define b2 x7 +#define b3 x8 +#define b4 x9 +#define b5 x10 +#define b6 x11 +#define b7 x12 +#define b8 x13 + +#define t x14 + +// These repeat mod 11 as we stash some intermediate results in the +// output buffer. + +#define u0 x15 +#define u1 x16 +#define u2 x17 +#define u3 x19 +#define u4 x20 +#define u5 x21 +#define u6 x22 +#define u7 x23 +#define u8 x24 +#define u9 x25 +#define u10 x26 +#define u11 x15 +#define u12 x16 +#define u13 x17 +#define u14 x19 +#define u15 x20 +#define u16 x21 + +S2N_BN_SYMBOL(bignum_mul_p521_alt): + CFI_START + +// Save more registers and make temporary space on stack + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x26) + CFI_DEC_SP(64) + +// Load operands and set up row 0 = [u9;...;u0] = a0 * [b8;...;b0] + + ldp a0, a1, [x] + ldp b0, b1, [y] + + mul u0, a0, b0 + umulh u1, a0, b0 + mul t, a0, b1 + umulh u2, a0, b1 + adds u1, u1, t + + ldp b2, b3, [y, #16] + + mul t, a0, b2 + umulh u3, a0, b2 + adcs u2, u2, t + + mul t, a0, b3 + umulh u4, a0, b3 + adcs u3, u3, t + + ldp b4, b5, [y, #32] + + mul t, a0, b4 + umulh u5, a0, b4 + adcs u4, u4, t + + mul t, a0, b5 + umulh u6, a0, b5 + adcs u5, u5, t + + ldp b6, b7, [y, #48] + + mul t, a0, b6 + umulh u7, a0, b6 + adcs u6, u6, t + + ldr b8, [y, #64] + + mul t, a0, b7 + umulh u8, a0, b7 + adcs u7, u7, t + + mul t, a0, b8 + umulh u9, a0, b8 + adcs u8, u8, t + + adc u9, u9, xzr + +// Row 1 = [u10;...;u0] = [a1;a0] * [b8;...;b0] + + mul t, a1, b0 + adds u1, u1, t + mul t, a1, b1 + adcs u2, u2, t + mul t, a1, b2 + adcs u3, u3, t + mul t, a1, b3 + adcs u4, u4, t + mul t, a1, b4 + adcs u5, u5, t + mul t, a1, b5 + adcs u6, u6, t + mul t, a1, b6 + adcs u7, u7, t + mul t, a1, b7 + adcs u8, u8, t + mul t, a1, b8 + adcs u9, u9, t + cset u10, cs + + umulh t, a1, b0 + adds u2, u2, t + umulh t, a1, b1 + adcs u3, u3, t + umulh t, a1, b2 + adcs u4, u4, t + umulh t, a1, b3 + adcs u5, u5, t + umulh t, a1, b4 + adcs u6, u6, t + umulh t, a1, b5 + adcs u7, u7, t + umulh t, a1, b6 + adcs u8, u8, t + umulh t, a1, b7 + adcs u9, u9, t + umulh t, a1, b8 + adc u10, u10, t + + stp u0, u1, [sp] + +// Row 2 = [u11;...;u0] = [a2;a1;a0] * [b8;...;b0] + + ldp a2, a3, [x, #16] + + mul t, a2, b0 + adds u2, u2, t + mul t, a2, b1 + adcs u3, u3, t + mul t, a2, b2 + adcs u4, u4, t + mul t, a2, b3 + adcs u5, u5, t + mul t, a2, b4 + adcs u6, u6, t + mul t, a2, b5 + adcs u7, u7, t + mul t, a2, b6 + adcs u8, u8, t + mul t, a2, b7 + adcs u9, u9, t + mul t, a2, b8 + adcs u10, u10, t + cset u11, cs + + umulh t, a2, b0 + adds u3, u3, t + umulh t, a2, b1 + adcs u4, u4, t + umulh t, a2, b2 + adcs u5, u5, t + umulh t, a2, b3 + adcs u6, u6, t + umulh t, a2, b4 + adcs u7, u7, t + umulh t, a2, b5 + adcs u8, u8, t + umulh t, a2, b6 + adcs u9, u9, t + umulh t, a2, b7 + adcs u10, u10, t + umulh t, a2, b8 + adc u11, u11, t + +// Row 3 = [u12;...;u0] = [a3;a2;a1;a0] * [b8;...;b0] + + mul t, a3, b0 + adds u3, u3, t + mul t, a3, b1 + adcs u4, u4, t + mul t, a3, b2 + adcs u5, u5, t + mul t, a3, b3 + adcs u6, u6, t + mul t, a3, b4 + adcs u7, u7, t + mul t, a3, b5 + adcs u8, u8, t + mul t, a3, b6 + adcs u9, u9, t + mul t, a3, b7 + adcs u10, u10, t + mul t, a3, b8 + adcs u11, u11, t + cset u12, cs + + umulh t, a3, b0 + adds u4, u4, t + umulh t, a3, b1 + adcs u5, u5, t + umulh t, a3, b2 + adcs u6, u6, t + umulh t, a3, b3 + adcs u7, u7, t + umulh t, a3, b4 + adcs u8, u8, t + umulh t, a3, b5 + adcs u9, u9, t + umulh t, a3, b6 + adcs u10, u10, t + umulh t, a3, b7 + adcs u11, u11, t + umulh t, a3, b8 + adc u12, u12, t + + stp u2, u3, [sp, #16] + +// Row 4 = [u13;...;u0] = [a4;a3;a2;a1;a0] * [b8;...;b0] + + ldp a4, a5, [x, #32] + + mul t, a4, b0 + adds u4, u4, t + mul t, a4, b1 + adcs u5, u5, t + mul t, a4, b2 + adcs u6, u6, t + mul t, a4, b3 + adcs u7, u7, t + mul t, a4, b4 + adcs u8, u8, t + mul t, a4, b5 + adcs u9, u9, t + mul t, a4, b6 + adcs u10, u10, t + mul t, a4, b7 + adcs u11, u11, t + mul t, a4, b8 + adcs u12, u12, t + cset u13, cs + + umulh t, a4, b0 + adds u5, u5, t + umulh t, a4, b1 + adcs u6, u6, t + umulh t, a4, b2 + adcs u7, u7, t + umulh t, a4, b3 + adcs u8, u8, t + umulh t, a4, b4 + adcs u9, u9, t + umulh t, a4, b5 + adcs u10, u10, t + umulh t, a4, b6 + adcs u11, u11, t + umulh t, a4, b7 + adcs u12, u12, t + umulh t, a4, b8 + adc u13, u13, t + +// Row 5 = [u14;...;u0] = [a5;a4;a3;a2;a1;a0] * [b8;...;b0] + + mul t, a5, b0 + adds u5, u5, t + mul t, a5, b1 + adcs u6, u6, t + mul t, a5, b2 + adcs u7, u7, t + mul t, a5, b3 + adcs u8, u8, t + mul t, a5, b4 + adcs u9, u9, t + mul t, a5, b5 + adcs u10, u10, t + mul t, a5, b6 + adcs u11, u11, t + mul t, a5, b7 + adcs u12, u12, t + mul t, a5, b8 + adcs u13, u13, t + cset u14, cs + + umulh t, a5, b0 + adds u6, u6, t + umulh t, a5, b1 + adcs u7, u7, t + umulh t, a5, b2 + adcs u8, u8, t + umulh t, a5, b3 + adcs u9, u9, t + umulh t, a5, b4 + adcs u10, u10, t + umulh t, a5, b5 + adcs u11, u11, t + umulh t, a5, b6 + adcs u12, u12, t + umulh t, a5, b7 + adcs u13, u13, t + umulh t, a5, b8 + adc u14, u14, t + + stp u4, u5, [sp, #32] + +// Row 6 = [u15;...;u0] = [a6;a5;a4;a3;a2;a1;a0] * [b8;...;b0] + + ldp a6, a7, [x, #48] + + mul t, a6, b0 + adds u6, u6, t + mul t, a6, b1 + adcs u7, u7, t + mul t, a6, b2 + adcs u8, u8, t + mul t, a6, b3 + adcs u9, u9, t + mul t, a6, b4 + adcs u10, u10, t + mul t, a6, b5 + adcs u11, u11, t + mul t, a6, b6 + adcs u12, u12, t + mul t, a6, b7 + adcs u13, u13, t + mul t, a6, b8 + adcs u14, u14, t + cset u15, cs + + umulh t, a6, b0 + adds u7, u7, t + umulh t, a6, b1 + adcs u8, u8, t + umulh t, a6, b2 + adcs u9, u9, t + umulh t, a6, b3 + adcs u10, u10, t + umulh t, a6, b4 + adcs u11, u11, t + umulh t, a6, b5 + adcs u12, u12, t + umulh t, a6, b6 + adcs u13, u13, t + umulh t, a6, b7 + adcs u14, u14, t + umulh t, a6, b8 + adc u15, u15, t + +// Row 7 = [u16;...;u0] = [a7;a6;a5;a4;a3;a2;a1;a0] * [b8;...;b0] + + mul t, a7, b0 + adds u7, u7, t + mul t, a7, b1 + adcs u8, u8, t + mul t, a7, b2 + adcs u9, u9, t + mul t, a7, b3 + adcs u10, u10, t + mul t, a7, b4 + adcs u11, u11, t + mul t, a7, b5 + adcs u12, u12, t + mul t, a7, b6 + adcs u13, u13, t + mul t, a7, b7 + adcs u14, u14, t + mul t, a7, b8 + adcs u15, u15, t + cset u16, cs + + umulh t, a7, b0 + adds u8, u8, t + umulh t, a7, b1 + adcs u9, u9, t + umulh t, a7, b2 + adcs u10, u10, t + umulh t, a7, b3 + adcs u11, u11, t + umulh t, a7, b4 + adcs u12, u12, t + umulh t, a7, b5 + adcs u13, u13, t + umulh t, a7, b6 + adcs u14, u14, t + umulh t, a7, b7 + adcs u15, u15, t + umulh t, a7, b8 + adc u16, u16, t + + stp u6, u7, [sp, #48] + +// Row 8 = [u16;...;u0] = [a8;a7;a6;a5;a4;a3;a2;a1;a0] * [b8;...;b0] + + ldr a8, [x, #64] + + mul t, a8, b0 + adds u8, u8, t + mul t, a8, b1 + adcs u9, u9, t + mul t, a8, b2 + adcs u10, u10, t + mul t, a8, b3 + adcs u11, u11, t + mul t, a8, b4 + adcs u12, u12, t + mul t, a8, b5 + adcs u13, u13, t + mul t, a8, b6 + adcs u14, u14, t + mul t, a8, b7 + adcs u15, u15, t + mul t, a8, b8 + adc u16, u16, t + + umulh t, a8, b0 + adds u9, u9, t + umulh t, a8, b1 + adcs u10, u10, t + umulh t, a8, b2 + adcs u11, u11, t + umulh t, a8, b3 + adcs u12, u12, t + umulh t, a8, b4 + adcs u13, u13, t + umulh t, a8, b5 + adcs u14, u14, t + umulh t, a8, b6 + adcs u15, u15, t + umulh t, a8, b7 + adc u16, u16, t + +// Now we have the full product, which we consider as +// 2^521 * h + l. Form h + l + 1 + + subs xzr, xzr, xzr + ldp b0, b1, [sp] + extr t, u9, u8, #9 + adcs b0, b0, t + extr t, u10, u9, #9 + adcs b1, b1, t + ldp b2, b3, [sp, #16] + extr t, u11, u10, #9 + adcs b2, b2, t + extr t, u12, u11, #9 + adcs b3, b3, t + ldp b4, b5, [sp, #32] + extr t, u13, u12, #9 + adcs b4, b4, t + extr t, u14, u13, #9 + adcs b5, b5, t + ldp b6, b7, [sp, #48] + extr t, u15, u14, #9 + adcs b6, b6, t + extr t, u16, u15, #9 + adcs b7, b7, t + orr b8, u8, #~0x1FF + lsr t, u16, #9 + adcs b8, b8, t + +// Now CF is set if h + l + 1 >= 2^521, which means it's already +// the answer, while if ~CF the answer is h + l so we should subtract +// 1 (all considered in 521 bits). Hence subtract ~CF and mask. + + sbcs b0, b0, xzr + sbcs b1, b1, xzr + sbcs b2, b2, xzr + sbcs b3, b3, xzr + sbcs b4, b4, xzr + sbcs b5, b5, xzr + sbcs b6, b6, xzr + sbcs b7, b7, xzr + sbc b8, b8, xzr + and b8, b8, #0x1FF + +// Store back digits of final result + + stp b0, b1, [z] + stp b2, b3, [z, #16] + stp b4, b5, [z, #32] + stp b6, b7, [z, #48] + str b8, [z, #64] + +// Restore registers + + CFI_INC_SP(64) + CFI_POP2(x25,x26) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_mul_p521_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/bignum_neg_p256.S b/cbits/s2n/arm/bignum_neg_p256.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_neg_p256.S @@ -0,0 +1,72 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Negate modulo p_256, z := (-x) mod p_256, assuming x reduced +// Input x[4]; output z[4] +// +// extern void bignum_neg_p256(uint64_t z[static 4], const uint64_t x[static 4]); +// +// Standard ARM ABI: X0 = z, X1 = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_neg_p256) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_neg_p256) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_neg_p256) + .text + .balign 4 + +#define z x0 +#define x x1 + +#define p x2 +#define t x3 + +#define d0 x4 +#define d1 x5 +#define d2 x6 +#define d3 x7 + + +S2N_BN_SYMBOL(bignum_neg_p256): + CFI_START + +// Load the 4 digits of x + + ldp d0, d1, [x] + ldp d2, d3, [x, #16] + +// Set a bitmask p for the input being nonzero, so that we avoid doing +// -0 = p_256 and hence maintain strict modular reduction + + orr t, d0, d1 + orr p, d2, d3 + orr p, p, t + cmp p, #0 + csetm p, ne + +// Mask the nontrivial words of p_256 = [n3;0;n1;-1] and subtract + + subs d0, p, d0 + and t, p, #0x00000000ffffffff + sbcs d1, t, d1 + sbcs d2, xzr, d2 + and t, p, #0xffffffff00000001 + sbc d3, t, d3 + +// Write back the result + + stp d0, d1, [z] + stp d2, d3, [z, #16] + +// Return + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_neg_p256) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/bignum_sqr_p521.S b/cbits/s2n/arm/bignum_sqr_p521.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_sqr_p521.S @@ -0,0 +1,1125 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Square modulo p_521, z := (x^2) mod p_521, assuming x reduced +// Input x[9]; output z[9] +// +// extern void bignum_sqr_p521(uint64_t z[static 9], const uint64_t x[static 9]); +// +// Standard ARM ABI: X0 = z, X1 = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + +// bignum_sqr_p521 is functionally equivalent to unopt/bignum_sqr_p521_base. +// It is written in a way that +// 1. A subset of scalar multiplications in bignum_montmul_p384 are carefully +// chosen and vectorized +// 2. The vectorized assembly is rescheduled using the SLOTHY superoptimizer. +// https://github.com/slothy-optimizer/slothy +// +// The output program of step 1. is as follows: +// +// stp x19, x20, [sp, #-16]! +// stp x21, x22, [sp, #-16]! +// stp x23, x24, [sp, #-16]! +// ldp x20, x19, [x1] +// ldr q23, [x1] +// ldr q1, [x1] +// ldr q16, [x1] +// ldp x14, x12, [x1, #16] +// ldr q28, [x1, #16] +// ldr q31, [x1, #16] +// ldp x9, x2, [x1, #32] +// ldr q29, [x1, #32] +// ldr q4, [x1, #32] +// ldr q5, [x1] +// ldr q2, [x1, #32] +// ldp x6, x13, [x1, #48] +// ldr q24, [x1, #48] +// ldr q27, [x1, #48] +// ldr q0, [x1, #16] +// ldr q30, [x1, #48] +// mul x17, x9, x6 +// mul x10, x2, x13 +// umulh x24, x9, x6 +// subs x4, x9, x2 +// cneg x4, x4, cc +// csetm x16, cc +// subs x3, x13, x6 +// cneg x23, x3, cc +// mul x3, x4, x23 +// umulh x4, x4, x23 +// cinv x22, x16, cc +// eor x23, x3, x22 +// eor x16, x4, x22 +// adds x3, x17, x24 +// adc x24, x24, xzr +// umulh x4, x2, x13 +// adds x3, x3, x10 +// adcs x24, x24, x4 +// adc x4, x4, xzr +// adds x24, x24, x10 +// adc x10, x4, xzr +// cmn x22, #0x1 +// adcs x4, x3, x23 +// adcs x24, x24, x16 +// adc x10, x10, x22 +// adds x8, x17, x17 +// adcs x22, x4, x4 +// adcs x5, x24, x24 +// adcs x11, x10, x10 +// adc x23, xzr, xzr +// movi v25.2D, #0xffffffff +// uzp2 v19.4S, v4.4S, v4.4S +// xtn v26.2S, v29.2D +// xtn v22.2S, v4.2D +// rev64 v4.4S, v4.4S +// umull v7.2D, v26.2S, v22.2S +// umull v21.2D, v26.2S, v19.2S +// uzp2 v17.4S, v29.4S, v29.4S +// mul v4.4S, v4.4S, v29.4S +// usra v21.2D, v7.2D, #32 +// umull v18.2D, v17.2S, v19.2S +// uaddlp v4.2D, v4.4S +// and v7.16B, v21.16B, v25.16B +// umlal v7.2D, v17.2S, v22.2S +// shl v4.2D, v4.2D, #32 +// usra v18.2D, v21.2D, #32 +// umlal v4.2D, v26.2S, v22.2S +// usra v18.2D, v7.2D, #32 +// mov x15, v4.d[0] +// mov x16, v4.d[1] +// mul x3, x9, x2 +// mov x10, v18.d[0] +// mov x17, v18.d[1] +// umulh x4, x9, x2 +// adds x24, x10, x3 +// adcs x10, x16, x4 +// adc x17, x17, xzr +// adds x7, x24, x3 +// adcs x10, x10, x4 +// adc x17, x17, xzr +// adds x8, x8, x10 +// adcs x22, x22, x17 +// adcs x21, x5, xzr +// adcs x5, x11, xzr +// adc x11, x23, xzr +// movi v25.2D, #0xffffffff +// uzp2 v19.4S, v27.4S, v27.4S +// xtn v26.2S, v24.2D +// xtn v22.2S, v27.2D +// rev64 v4.4S, v27.4S +// umull v7.2D, v26.2S, v22.2S +// umull v21.2D, v26.2S, v19.2S +// uzp2 v17.4S, v24.4S, v24.4S +// mul v4.4S, v4.4S, v24.4S +// usra v21.2D, v7.2D, #32 +// umull v18.2D, v17.2S, v19.2S +// uaddlp v4.2D, v4.4S +// and v7.16B, v21.16B, v25.16B +// umlal v7.2D, v17.2S, v22.2S +// shl v4.2D, v4.2D, #32 +// usra v18.2D, v21.2D, #32 +// umlal v4.2D, v26.2S, v22.2S +// usra v18.2D, v7.2D, #32 +// mov x23, v4.d[0] +// mov x16, v4.d[1] +// mul x3, x6, x13 +// mov x10, v18.d[0] +// mov x17, v18.d[1] +// umulh x4, x6, x13 +// adds x24, x10, x3 +// adcs x10, x16, x4 +// adc x17, x17, xzr +// adds x24, x24, x3 +// adcs x10, x10, x4 +// adc x17, x17, xzr +// adds x23, x23, x21 +// adcs x16, x24, x5 +// adcs x3, x10, x11 +// adc x21, x17, xzr +// ldr x17, [x1, #64] +// add x5, x17, x17 +// mul x11, x17, x17 +// and x17, x20, #0xfffffffffffff +// mul x4, x5, x17 +// extr x17, x19, x20, #52 +// and x17, x17, #0xfffffffffffff +// mul x10, x5, x17 +// lsr x17, x4, #52 +// add x24, x10, x17 +// lsl x17, x4, #12 +// extr x17, x24, x17, #12 +// adds x15, x15, x17 +// extr x17, x14, x19, #40 +// and x17, x17, #0xfffffffffffff +// mul x10, x5, x17 +// lsr x17, x24, #52 +// add x4, x10, x17 +// lsl x17, x24, #12 +// extr x17, x4, x17, #24 +// adcs x7, x7, x17 +// extr x17, x12, x14, #28 +// and x17, x17, #0xfffffffffffff +// mul x10, x5, x17 +// lsr x17, x4, #52 +// add x24, x10, x17 +// lsl x17, x4, #12 +// extr x17, x24, x17, #36 +// adcs x8, x8, x17 +// extr x17, x9, x12, #16 +// and x17, x17, #0xfffffffffffff +// mul x10, x5, x17 +// lsr x17, x24, #52 +// add x4, x10, x17 +// lsl x17, x24, #12 +// extr x17, x4, x17, #48 +// adcs x22, x22, x17 +// lsr x17, x9, #4 +// and x17, x17, #0xfffffffffffff +// mul x10, x5, x17 +// lsr x17, x4, #52 +// add x24, x10, x17 +// lsl x17, x4, #12 +// extr x4, x24, x17, #60 +// extr x17, x2, x9, #56 +// and x17, x17, #0xfffffffffffff +// mul x10, x5, x17 +// lsr x17, x24, #52 +// add x24, x10, x17 +// lsl x17, x4, #8 +// extr x17, x24, x17, #8 +// adcs x23, x23, x17 +// extr x17, x6, x2, #44 +// and x17, x17, #0xfffffffffffff +// mul x10, x5, x17 +// lsr x17, x24, #52 +// add x4, x10, x17 +// lsl x17, x24, #12 +// extr x17, x4, x17, #20 +// adcs x16, x16, x17 +// extr x17, x13, x6, #32 +// and x17, x17, #0xfffffffffffff +// mul x10, x5, x17 +// lsr x17, x4, #52 +// add x24, x10, x17 +// lsl x17, x4, #12 +// extr x17, x24, x17, #32 +// adcs x3, x3, x17 +// lsr x17, x13, #20 +// mul x10, x5, x17 +// lsr x17, x24, #52 +// add x10, x10, x17 +// lsl x17, x24, #12 +// extr x17, x10, x17, #44 +// adcs x4, x21, x17 +// lsr x17, x10, #44 +// adc x24, x11, x17 +// extr x10, x7, x15, #9 +// extr x17, x8, x7, #9 +// stp x10, x17, [x0] // @slothy:writes=buffer0 +// extr x10, x22, x8, #9 +// extr x17, x23, x22, #9 +// stp x10, x17, [x0, #16] // @slothy:writes=buffer16 +// extr x10, x16, x23, #9 +// extr x17, x3, x16, #9 +// stp x10, x17, [x0, #32] // @slothy:writes=buffer32 +// extr x10, x4, x3, #9 +// extr x17, x24, x4, #9 +// stp x10, x17, [x0, #48] // @slothy:writes=buffer48 +// and x10, x15, #0x1ff +// lsr x17, x24, #9 +// add x17, x10, x17 +// str x17, [x0, #64] // @slothy:writes=buffer64 +// uzp1 v17.4S, v28.4S, v23.4S +// rev64 v4.4S, v28.4S +// uzp1 v7.4S, v23.4S, v23.4S +// mul v4.4S, v4.4S, v23.4S +// uaddlp v4.2D, v4.4S +// shl v4.2D, v4.2D, #32 +// umlal v4.2D, v7.2S, v17.2S +// mov x8, v4.d[0] +// mov x22, v4.d[1] +// umulh x23, x20, x14 +// subs x17, x20, x19 +// cneg x4, x17, cc +// csetm x24, cc +// subs x17, x12, x14 +// cneg x17, x17, cc +// mul x10, x4, x17 +// umulh x17, x4, x17 +// cinv x16, x24, cc +// eor x3, x10, x16 +// eor x4, x17, x16 +// adds x24, x8, x23 +// adc x10, x23, xzr +// umulh x17, x19, x12 +// adds x24, x24, x22 +// adcs x10, x10, x17 +// adc x17, x17, xzr +// adds x10, x10, x22 +// adc x17, x17, xzr +// cmn x16, #0x1 +// adcs x24, x24, x3 +// adcs x10, x10, x4 +// adc x17, x17, x16 +// adds x15, x8, x8 +// adcs x7, x24, x24 +// adcs x8, x10, x10 +// adcs x22, x17, x17 +// adc x23, xzr, xzr +// movi v25.2D, #0xffffffff +// uzp2 v19.4S, v16.4S, v16.4S +// xtn v26.2S, v1.2D +// xtn v22.2S, v16.2D +// rev64 v4.4S, v16.4S +// umull v7.2D, v26.2S, v22.2S +// umull v21.2D, v26.2S, v19.2S +// uzp2 v17.4S, v1.4S, v1.4S +// mul v4.4S, v4.4S, v1.4S +// usra v21.2D, v7.2D, #32 +// umull v18.2D, v17.2S, v19.2S +// uaddlp v4.2D, v4.4S +// and v7.16B, v21.16B, v25.16B +// umlal v7.2D, v17.2S, v22.2S +// shl v4.2D, v4.2D, #32 +// usra v18.2D, v21.2D, #32 +// umlal v4.2D, v26.2S, v22.2S +// usra v18.2D, v7.2D, #32 +// mov x21, v4.d[0] +// mov x16, v4.d[1] +// mul x3, x20, x19 +// mov x10, v18.d[0] +// mov x17, v18.d[1] +// umulh x4, x20, x19 +// adds x24, x10, x3 +// adcs x10, x16, x4 +// adc x17, x17, xzr +// adds x5, x24, x3 +// adcs x10, x10, x4 +// adc x17, x17, xzr +// adds x11, x15, x10 +// adcs x15, x7, x17 +// adcs x7, x8, xzr +// adcs x8, x22, xzr +// adc x22, x23, xzr +// xtn v7.2S, v31.2D +// shrn v4.2S, v31.2D, #32 +// umull v4.2D, v7.2S, v4.2S +// shl v4.2D, v4.2D, #33 +// umlal v4.2D, v7.2S, v7.2S +// mov x23, v4.d[0] +// mov x16, v4.d[1] +// mul x3, x14, x12 +// umulh x10, x14, x14 +// umulh x17, x12, x12 +// umulh x4, x14, x12 +// adds x24, x10, x3 +// adcs x10, x16, x4 +// adc x17, x17, xzr +// adds x24, x24, x3 +// adcs x10, x10, x4 +// adc x17, x17, xzr +// adds x16, x23, x7 +// adcs x3, x24, x8 +// adcs x4, x10, x22 +// adc x24, x17, xzr +// ldp x10, x17, [x0] // @slothy:reads=buffer0 +// adds x10, x10, x21 +// adcs x17, x17, x5 +// stp x10, x17, [x0] // @slothy:writes=buffer0 +// ldp x10, x17, [x0, #16] // @slothy:reads=buffer16 +// adcs x10, x10, x11 +// adcs x17, x17, x15 +// stp x10, x17, [x0, #16] // @slothy:writes=buffer16 +// ldp x10, x17, [x0, #32] // @slothy:reads=buffer32 +// adcs x10, x10, x16 +// adcs x17, x17, x3 +// stp x10, x17, [x0, #32] // @slothy:writes=buffer32 +// ldp x10, x17, [x0, #48] // @slothy:reads=buffer48 +// adcs x10, x10, x4 +// adcs x17, x17, x24 +// stp x10, x17, [x0, #48] // @slothy:writes=buffer48 +// ldr x17, [x0, #64] // @slothy:reads=buffer64 +// adc x17, x17, xzr +// str x17, [x0, #64] // @slothy:writes=buffer64 +// movi v25.2D, #0xffffffff +// uzp2 v19.4S, v2.4S, v2.4S +// xtn v26.2S, v5.2D +// xtn v22.2S, v2.2D +// rev64 v4.4S, v2.4S +// umull v7.2D, v26.2S, v22.2S +// umull v21.2D, v26.2S, v19.2S +// uzp2 v17.4S, v5.4S, v5.4S +// mul v4.4S, v4.4S, v5.4S +// usra v21.2D, v7.2D, #32 +// umull v18.2D, v17.2S, v19.2S +// uaddlp v4.2D, v4.4S +// and v7.16B, v21.16B, v25.16B +// umlal v7.2D, v17.2S, v22.2S +// shl v4.2D, v4.2D, #32 +// usra v18.2D, v21.2D, #32 +// umlal v4.2D, v26.2S, v22.2S +// usra v18.2D, v7.2D, #32 +// mov x5, v4.d[0] +// mov x4, v4.d[1] +// movi v25.2D, #0xffffffff +// uzp2 v17.4S, v30.4S, v30.4S +// xtn v19.2S, v0.2D +// xtn v26.2S, v30.2D +// rev64 v4.4S, v30.4S +// umull v7.2D, v19.2S, v26.2S +// umull v22.2D, v19.2S, v17.2S +// uzp2 v21.4S, v0.4S, v0.4S +// mul v4.4S, v4.4S, v0.4S +// usra v22.2D, v7.2D, #32 +// umull v17.2D, v21.2S, v17.2S +// uaddlp v4.2D, v4.4S +// and v7.16B, v22.16B, v25.16B +// umlal v7.2D, v21.2S, v26.2S +// shl v4.2D, v4.2D, #32 +// usra v17.2D, v22.2D, #32 +// umlal v4.2D, v19.2S, v26.2S +// usra v17.2D, v7.2D, #32 +// mov x24, v4.d[0] +// mov x10, v4.d[1] +// mov x17, v18.d[0] +// adds x4, x4, x17 +// mov x17, v18.d[1] +// adcs x24, x24, x17 +// mov x17, v17.d[0] +// adcs x10, x10, x17 +// mov x17, v17.d[1] +// adc x17, x17, xzr +// adds x15, x4, x5 +// adcs x4, x24, x4 +// adcs x24, x10, x24 +// adcs x10, x17, x10 +// adc x17, xzr, x17 +// adds x7, x4, x5 +// adcs x8, x24, x15 +// adcs x22, x10, x4 +// adcs x23, x17, x24 +// adcs x16, xzr, x10 +// adc x3, xzr, x17 +// subs x17, x14, x12 +// cneg x24, x17, cc +// csetm x4, cc +// subs x17, x13, x6 +// cneg x10, x17, cc +// mul x17, x24, x10 +// umulh x24, x24, x10 +// cinv x10, x4, cc +// cmn x10, #0x1 +// eor x17, x17, x10 +// adcs x23, x23, x17 +// eor x17, x24, x10 +// adcs x16, x16, x17 +// adc x3, x3, x10 +// subs x17, x20, x19 +// cneg x24, x17, cc +// csetm x4, cc +// subs x17, x2, x9 +// cneg x10, x17, cc +// mul x17, x24, x10 +// umulh x24, x24, x10 +// cinv x10, x4, cc +// cmn x10, #0x1 +// eor x17, x17, x10 +// adcs x11, x15, x17 +// eor x17, x24, x10 +// adcs x15, x7, x17 +// adcs x7, x8, x10 +// adcs x22, x22, x10 +// adcs x23, x23, x10 +// adcs x16, x16, x10 +// adc x3, x3, x10 +// subs x17, x19, x12 +// cneg x24, x17, cc +// csetm x4, cc +// subs x17, x13, x2 +// cneg x10, x17, cc +// mul x17, x24, x10 +// umulh x24, x24, x10 +// cinv x10, x4, cc +// cmn x10, #0x1 +// eor x17, x17, x10 +// adcs x8, x22, x17 +// eor x17, x24, x10 +// adcs x23, x23, x17 +// adcs x16, x16, x10 +// adc x3, x3, x10 +// subs x17, x20, x14 +// cneg x24, x17, cc +// csetm x4, cc +// subs x17, x6, x9 +// cneg x10, x17, cc +// mul x17, x24, x10 +// umulh x24, x24, x10 +// cinv x10, x4, cc +// cmn x10, #0x1 +// eor x17, x17, x10 +// adcs x22, x15, x17 +// eor x17, x24, x10 +// adcs x4, x7, x17 +// adcs x24, x8, x10 +// adcs x23, x23, x10 +// adcs x16, x16, x10 +// adc x3, x3, x10 +// subs x12, x20, x12 +// cneg x10, x12, cc +// csetm x17, cc +// subs x12, x13, x9 +// cneg x9, x12, cc +// mul x12, x10, x9 +// umulh x13, x10, x9 +// cinv x9, x17, cc +// cmn x9, #0x1 +// eor x12, x12, x9 +// adcs x4, x4, x12 +// eor x12, x13, x9 +// adcs x24, x24, x12 +// adcs x10, x23, x9 +// adcs x17, x16, x9 +// adc x13, x3, x9 +// subs x19, x19, x14 +// cneg x12, x19, cc +// csetm x9, cc +// subs x6, x6, x2 +// cneg x14, x6, cc +// mul x19, x12, x14 +// umulh x12, x12, x14 +// cinv x14, x9, cc +// cmn x14, #0x1 +// eor x19, x19, x14 +// adcs x23, x4, x19 +// eor x19, x12, x14 +// adcs x16, x24, x19 +// adcs x6, x10, x14 +// adcs x2, x17, x14 +// adc x9, x13, x14 +// ldp x12, x14, [x0] // @slothy:reads=buffer0 +// extr x19, x6, x16, #8 +// adds x10, x19, x12 +// extr x19, x2, x6, #8 +// adcs x17, x19, x14 +// ldp x14, x12, [x0, #16] // @slothy:reads=buffer16 +// extr x19, x9, x2, #8 +// adcs x13, x19, x14 +// and x14, x17, x13 +// lsr x19, x9, #8 +// adcs x6, x19, x12 +// and x9, x14, x6 +// ldp x14, x12, [x0, #32] // @slothy:reads=buffer32 +// lsl x19, x5, #1 +// adcs x2, x19, x14 +// and x14, x9, x2 +// extr x19, x11, x5, #63 +// adcs x3, x19, x12 +// and x9, x14, x3 +// ldp x14, x12, [x0, #48] // @slothy:reads=buffer48 +// extr x19, x22, x11, #63 +// adcs x4, x19, x14 +// and x14, x9, x4 +// extr x19, x23, x22, #63 +// adcs x24, x19, x12 +// and x12, x14, x24 +// ldr x14, [x0, #64] // @slothy:reads=buffer64 +// extr x19, x16, x23, #63 +// and x19, x19, #0x1ff +// adc x19, x14, x19 +// lsr x14, x19, #9 +// orr x19, x19, #0xfffffffffffffe00 +// cmp xzr, xzr +// adcs xzr, x10, x14 +// adcs xzr, x12, xzr +// adcs xzr, x19, xzr +// adcs x10, x10, x14 +// adcs x17, x17, xzr +// adcs x13, x13, xzr +// adcs x6, x6, xzr +// adcs x2, x2, xzr +// adcs x9, x3, xzr +// adcs x12, x4, xzr +// adcs x14, x24, xzr +// adc x19, x19, xzr +// and x19, x19, #0x1ff +// stp x10, x17, [x0] // @slothy:writes=buffer0 +// stp x13, x6, [x0, #16] // @slothy:writes=buffer16 +// stp x2, x9, [x0, #32] // @slothy:writes=buffer32 +// stp x12, x14, [x0, #48] // @slothy:writes=buffer48 +// str x19, [x0, #64] // @slothy:writes=buffer64 +// ldp x23, x24, [sp], #16 +// ldp x21, x22, [sp], #16 +// ldp x19, x20, [sp], #16 +// ret +// +// The bash script used for step 2 is as follows: +// +// # Store the assembly instructions except the last 'ret', +// # callee-register store/loads as, say, 'input.S'. +// export OUTPUTS="[hint_buffer0,hint_buffer16,hint_buffer32,hint_buffer48,hint_buffer64]" +// export RESERVED_REGS="[x18,x25,x26,x27,x28,x29,x30,sp,q8,q9,q10,q11,q12,q13,q14,q15,v8,v9,v10,v11,v12,v13,v14,v15]" +// /tools/external/slothy.sh input.S my_out_dir +// # my_out_dir/3.opt.s is the optimized assembly. Its output may differ +// # from this file since the sequence is non-deterministically chosen. +// # Please add 'ret' at the end of the output assembly. + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_sqr_p521) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_sqr_p521) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_sqr_p521) + .text + .balign 4 + +S2N_BN_SYMBOL(bignum_sqr_p521): + CFI_START + +// Save registers + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + + ldr q23, [x1, #32] + ldp x9, x2, [x1, #32] + ldr q16, [x1, #32] + ldr q20, [x1, #48] + ldp x6, x13, [x1, #48] + rev64 v2.4S, v23.4S + mul x14, x9, x2 + ldr q31, [x1, #48] + subs x22, x9, x2 + uzp2 v26.4S, v23.4S, v23.4S + mul v30.4S, v2.4S, v16.4S + xtn v0.2S, v20.2D + csetm x12, cc + xtn v21.2S, v16.2D + xtn v23.2S, v23.2D + umulh x10, x9, x6 + rev64 v27.4S, v31.4S + umull v2.2D, v21.2S, v26.2S + cneg x23, x22, cc + uaddlp v25.2D, v30.4S + umull v18.2D, v21.2S, v23.2S + mul x22, x9, x6 + mul v6.4S, v27.4S, v20.4S + uzp2 v17.4S, v20.4S, v20.4S + shl v20.2D, v25.2D, #32 + uzp2 v27.4S, v31.4S, v31.4S + mul x16, x2, x13 + umlal v20.2D, v21.2S, v23.2S + usra v2.2D, v18.2D, #32 + adds x8, x22, x10 + umull v25.2D, v17.2S, v27.2S + xtn v31.2S, v31.2D + movi v1.2D, #0xffffffff + adc x3, x10, xzr + umulh x21, x2, x13 + uzp2 v21.4S, v16.4S, v16.4S + umull v18.2D, v0.2S, v27.2S + subs x19, x13, x6 + and v7.16B, v2.16B, v1.16B + umull v27.2D, v0.2S, v31.2S + cneg x20, x19, cc + movi v30.2D, #0xffffffff + umull v16.2D, v21.2S, v26.2S + umlal v7.2D, v21.2S, v23.2S + mul x19, x23, x20 + cinv x7, x12, cc + uaddlp v6.2D, v6.4S + eor x12, x19, x7 + adds x11, x8, x16 + umulh x10, x23, x20 + ldr q1, [x1] + usra v16.2D, v2.2D, #32 + adcs x19, x3, x21 + shl v2.2D, v6.2D, #32 + adc x20, x21, xzr + adds x17, x19, x16 + usra v18.2D, v27.2D, #32 + adc x19, x20, xzr + cmn x7, #0x1 + umlal v2.2D, v0.2S, v31.2S + umulh x16, x9, x2 + adcs x8, x11, x12 + usra v16.2D, v7.2D, #32 + ldr x12, [x1, #64] + eor x20, x10, x7 + umulh x10, x6, x13 + mov x23, v2.d[0] + mov x3, v2.d[1] + adcs x21, x17, x20 + usra v25.2D, v18.2D, #32 + and v23.16B, v18.16B, v30.16B + adc x7, x19, x7 + adds x22, x22, x22 + ldr q7, [x1, #16] + adcs x17, x8, x8 + umlal v23.2D, v17.2S, v31.2S + mov x19, v16.d[0] + mul x11, x12, x12 + ldr q4, [x1] + usra v25.2D, v23.2D, #32 + add x5, x12, x12 + adcs x15, x21, x21 + ldr q28, [x1] + mov x12, v20.d[1] + adcs x24, x7, x7 + mov x21, v16.d[1] + adc x4, xzr, xzr + adds x19, x19, x14 + ldr q18, [x1, #16] + xtn v26.2S, v1.2D + adcs x8, x12, x16 + adc x21, x21, xzr + adds x7, x19, x14 + xtn v23.2S, v7.2D + rev64 v21.4S, v28.4S + adcs x12, x8, x16 + ldp x20, x19, [x1] + mov x16, v25.d[1] + xtn v22.2S, v28.2D + adc x14, x21, xzr + adds x8, x22, x12 + uzp2 v24.4S, v28.4S, v28.4S + rev64 v28.4S, v18.4S + mul x12, x6, x13 + mul v16.4S, v21.4S, v1.4S + shrn v31.2S, v7.2D, #32 + adcs x22, x17, x14 + mov x14, v25.d[0] + and x21, x20, #0xfffffffffffff + umull v17.2D, v26.2S, v24.2S + ldr q2, [x1, #32] + adcs x17, x15, xzr + ldr q30, [x1, #48] + umull v7.2D, v26.2S, v22.2S + adcs x15, x24, xzr + ldr q0, [x1, #16] + movi v6.2D, #0xffffffff + adc x4, x4, xzr + adds x14, x14, x12 + uzp1 v27.4S, v18.4S, v4.4S + uzp2 v19.4S, v1.4S, v1.4S + adcs x24, x3, x10 + mul x3, x5, x21 + umull v29.2D, v23.2S, v31.2S + ldr q5, [x1] + adc x21, x16, xzr + adds x16, x14, x12 + extr x12, x19, x20, #52 + umull v18.2D, v19.2S, v24.2S + adcs x24, x24, x10 + and x10, x12, #0xfffffffffffff + ldp x14, x12, [x1, #16] + usra v17.2D, v7.2D, #32 + adc x21, x21, xzr + adds x23, x23, x17 + mul x17, x5, x10 + shl v21.2D, v29.2D, #33 + lsl x10, x3, #12 + lsr x1, x3, #52 + rev64 v29.4S, v2.4S + uaddlp v25.2D, v16.4S + add x17, x17, x1 + adcs x16, x16, x15 + extr x3, x14, x19, #40 + mov x15, v20.d[0] + extr x10, x17, x10, #12 + and x3, x3, #0xfffffffffffff + shl v3.2D, v25.2D, #32 + and v6.16B, v17.16B, v6.16B + mul x1, x5, x3 + usra v18.2D, v17.2D, #32 + adcs x3, x24, x4 + extr x4, x12, x14, #28 + umlal v6.2D, v19.2S, v22.2S + xtn v20.2S, v2.2D + umlal v3.2D, v26.2S, v22.2S + movi v26.2D, #0xffffffff + lsr x24, x17, #52 + and x4, x4, #0xfffffffffffff + uzp2 v19.4S, v2.4S, v2.4S + add x1, x1, x24 + mul x24, x5, x4 + lsl x4, x17, #12 + xtn v24.2S, v5.2D + extr x17, x1, x4, #24 + adc x21, x21, xzr + umlal v21.2D, v23.2S, v23.2S + adds x4, x15, x10 + lsl x10, x1, #12 + adcs x15, x7, x17 + mul v23.4S, v28.4S, v4.4S + and x7, x4, #0x1ff + lsr x17, x1, #52 + umulh x1, x19, x12 + uzp2 v17.4S, v5.4S, v5.4S + extr x4, x15, x4, #9 + add x24, x24, x17 + mul v29.4S, v29.4S, v5.4S + extr x17, x24, x10, #36 + extr x10, x9, x12, #16 + uzp1 v28.4S, v4.4S, v4.4S + adcs x17, x8, x17 + and x8, x10, #0xfffffffffffff + umull v16.2D, v24.2S, v20.2S + extr x10, x17, x15, #9 + mul x15, x5, x8 + stp x4, x10, [x0] + lsl x4, x24, #12 + lsr x8, x9, #4 + uaddlp v4.2D, v23.4S + and x8, x8, #0xfffffffffffff + umull v23.2D, v24.2S, v19.2S + mul x8, x5, x8 + extr x10, x2, x9, #56 + lsr x24, x24, #52 + and x10, x10, #0xfffffffffffff + add x15, x15, x24 + extr x4, x15, x4, #48 + mul x24, x5, x10 + lsr x10, x15, #52 + usra v23.2D, v16.2D, #32 + add x10, x8, x10 + shl v4.2D, v4.2D, #32 + adcs x22, x22, x4 + extr x4, x6, x2, #44 + lsl x15, x15, #12 + lsr x8, x10, #52 + extr x15, x10, x15, #60 + and x10, x4, #0xfffffffffffff + umlal v4.2D, v28.2S, v27.2S + add x8, x24, x8 + extr x4, x13, x6, #32 + mul x24, x5, x10 + uzp2 v16.4S, v30.4S, v30.4S + lsl x10, x15, #8 + rev64 v28.4S, v30.4S + and x15, x4, #0xfffffffffffff + extr x4, x8, x10, #8 + mul x10, x5, x15 + lsl x15, x8, #12 + adcs x23, x23, x4 + lsr x4, x8, #52 + lsr x8, x13, #20 + add x4, x24, x4 + mul x8, x5, x8 + lsr x24, x4, #52 + extr x15, x4, x15, #20 + lsl x4, x4, #12 + add x10, x10, x24 + adcs x15, x16, x15 + extr x4, x10, x4, #32 + umulh x5, x20, x14 + adcs x3, x3, x4 + usra v18.2D, v6.2D, #32 + lsl x16, x10, #12 + extr x24, x15, x23, #9 + lsr x10, x10, #52 + uzp2 v27.4S, v0.4S, v0.4S + add x8, x8, x10 + extr x10, x3, x15, #9 + extr x4, x22, x17, #9 + and v25.16B, v23.16B, v26.16B + lsr x17, x8, #44 + extr x15, x8, x16, #44 + extr x16, x23, x22, #9 + xtn v7.2S, v30.2D + mov x8, v4.d[0] + stp x24, x10, [x0, #32] + uaddlp v30.2D, v29.4S + stp x4, x16, [x0, #16] + umulh x24, x20, x19 + adcs x15, x21, x15 + adc x16, x11, x17 + subs x11, x20, x19 + xtn v5.2S, v0.2D + csetm x17, cc + extr x3, x15, x3, #9 + mov x22, v4.d[1] + cneg x21, x11, cc + subs x10, x12, x14 + mul v31.4S, v28.4S, v0.4S + cneg x10, x10, cc + cinv x11, x17, cc + shl v4.2D, v30.2D, #32 + umull v28.2D, v5.2S, v16.2S + extr x23, x16, x15, #9 + adds x4, x8, x5 + mul x17, x21, x10 + umull v22.2D, v5.2S, v7.2S + adc x15, x5, xzr + adds x4, x4, x22 + uaddlp v2.2D, v31.4S + lsr x5, x16, #9 + adcs x16, x15, x1 + mov x15, v18.d[0] + adc x1, x1, xzr + umulh x10, x21, x10 + adds x22, x16, x22 + umlal v4.2D, v24.2S, v20.2S + umull v30.2D, v27.2S, v16.2S + stp x3, x23, [x0, #48] + add x3, x7, x5 + adc x16, x1, xzr + usra v28.2D, v22.2D, #32 + mul x23, x20, x19 + eor x1, x17, x11 + cmn x11, #0x1 + mov x17, v18.d[1] + umull v18.2D, v17.2S, v19.2S + adcs x7, x4, x1 + eor x1, x10, x11 + umlal v25.2D, v17.2S, v20.2S + movi v16.2D, #0xffffffff + adcs x22, x22, x1 + usra v18.2D, v23.2D, #32 + umulh x4, x14, x14 + adc x1, x16, x11 + adds x10, x8, x8 + shl v23.2D, v2.2D, #32 + str x3, [x0, #64] + adcs x5, x7, x7 + and v16.16B, v28.16B, v16.16B + usra v30.2D, v28.2D, #32 + adcs x7, x22, x22 + mov x21, v3.d[1] + adcs x11, x1, x1 + umlal v16.2D, v27.2S, v7.2S + adc x22, xzr, xzr + adds x16, x15, x23 + mul x8, x14, x12 + umlal v23.2D, v5.2S, v7.2S + usra v18.2D, v25.2D, #32 + umulh x15, x14, x12 + adcs x21, x21, x24 + usra v30.2D, v16.2D, #32 + adc x1, x17, xzr + adds x3, x16, x23 + adcs x21, x21, x24 + adc x1, x1, xzr + adds x24, x10, x21 + umulh x21, x12, x12 + adcs x16, x5, x1 + adcs x10, x7, xzr + mov x17, v21.d[1] + adcs x23, x11, xzr + adc x5, x22, xzr + adds x1, x4, x8 + adcs x22, x17, x15 + ldp x17, x4, [x0] + mov x11, v21.d[0] + adc x21, x21, xzr + adds x1, x1, x8 + adcs x15, x22, x15 + adc x8, x21, xzr + adds x22, x11, x10 + mov x21, v3.d[0] + adcs x11, x1, x23 + ldp x1, x10, [x0, #16] + adcs x15, x15, x5 + adc x7, x8, xzr + adds x8, x17, x21 + mov x23, v4.d[1] + ldp x5, x21, [x0, #32] + adcs x17, x4, x3 + ldr x4, [x0, #64] + mov x3, v18.d[0] + adcs x24, x1, x24 + stp x8, x17, [x0] + adcs x17, x10, x16 + ldp x1, x16, [x0, #48] + adcs x5, x5, x22 + adcs x8, x21, x11 + stp x5, x8, [x0, #32] + adcs x1, x1, x15 + mov x15, v23.d[1] + adcs x21, x16, x7 + stp x1, x21, [x0, #48] + adc x10, x4, xzr + subs x7, x14, x12 + mov x16, v18.d[1] + cneg x5, x7, cc + csetm x4, cc + subs x11, x13, x6 + mov x8, v23.d[0] + cneg x7, x11, cc + cinv x21, x4, cc + mov x11, v30.d[0] + adds x4, x23, x3 + mul x22, x5, x7 + mov x23, v30.d[1] + adcs x8, x8, x16 + adcs x16, x15, x11 + adc x11, x23, xzr + umulh x3, x5, x7 + stp x24, x17, [x0, #16] + mov x5, v4.d[0] + subs x15, x20, x19 + cneg x7, x15, cc + str x10, [x0, #64] + csetm x1, cc + subs x24, x2, x9 + cneg x17, x24, cc + cinv x15, x1, cc + adds x23, x4, x5 + umulh x1, x7, x17 + adcs x24, x8, x4 + adcs x10, x16, x8 + eor x8, x22, x21 + adcs x16, x11, x16 + mul x22, x7, x17 + eor x17, x1, x15 + adc x1, xzr, x11 + adds x11, x24, x5 + eor x7, x3, x21 + adcs x3, x10, x23 + adcs x24, x16, x24 + adcs x4, x1, x10 + eor x10, x22, x15 + adcs x16, xzr, x16 + adc x1, xzr, x1 + cmn x21, #0x1 + adcs x8, x4, x8 + adcs x22, x16, x7 + adc x7, x1, x21 + subs x21, x19, x12 + csetm x4, cc + cneg x1, x21, cc + subs x21, x13, x2 + cinv x16, x4, cc + cneg x4, x21, cc + cmn x15, #0x1 + adcs x21, x23, x10 + mul x23, x1, x4 + adcs x11, x11, x17 + adcs x3, x3, x15 + umulh x1, x1, x4 + adcs x24, x24, x15 + adcs x8, x8, x15 + adcs x22, x22, x15 + eor x17, x23, x16 + adc x15, x7, x15 + subs x7, x20, x14 + cneg x7, x7, cc + csetm x4, cc + subs x10, x20, x12 + cneg x23, x10, cc + csetm x10, cc + subs x12, x6, x9 + cinv x20, x4, cc + cneg x12, x12, cc + cmn x16, #0x1 + eor x1, x1, x16 + adcs x17, x24, x17 + mul x4, x7, x12 + adcs x8, x8, x1 + umulh x1, x7, x12 + adcs x24, x22, x16 + adc x7, x15, x16 + subs x12, x13, x9 + cneg x12, x12, cc + cinv x13, x10, cc + subs x19, x19, x14 + mul x9, x23, x12 + cneg x19, x19, cc + csetm x10, cc + eor x16, x1, x20 + subs x22, x6, x2 + umulh x12, x23, x12 + eor x1, x4, x20 + cinv x4, x10, cc + cneg x22, x22, cc + cmn x20, #0x1 + adcs x15, x11, x1 + eor x6, x12, x13 + adcs x10, x3, x16 + adcs x17, x17, x20 + eor x23, x9, x13 + adcs x2, x8, x20 + mul x11, x19, x22 + adcs x24, x24, x20 + adc x7, x7, x20 + cmn x13, #0x1 + adcs x3, x10, x23 + umulh x22, x19, x22 + adcs x17, x17, x6 + eor x12, x22, x4 + extr x22, x15, x21, #63 + adcs x8, x2, x13 + extr x21, x21, x5, #63 + ldp x16, x23, [x0] + adcs x20, x24, x13 + eor x1, x11, x4 + adc x6, x7, x13 + cmn x4, #0x1 + ldp x2, x7, [x0, #16] + adcs x1, x3, x1 + extr x19, x1, x15, #63 + adcs x14, x17, x12 + extr x1, x14, x1, #63 + lsl x17, x5, #1 + adcs x8, x8, x4 + extr x12, x8, x14, #8 + ldp x15, x11, [x0, #32] + adcs x9, x20, x4 + adc x3, x6, x4 + adds x16, x12, x16 + extr x6, x9, x8, #8 + ldp x14, x12, [x0, #48] + extr x8, x3, x9, #8 + adcs x20, x6, x23 + ldr x24, [x0, #64] + lsr x6, x3, #8 + adcs x8, x8, x2 + and x2, x1, #0x1ff + and x1, x20, x8 + adcs x4, x6, x7 + adcs x3, x17, x15 + and x1, x1, x4 + adcs x9, x21, x11 + and x1, x1, x3 + adcs x6, x22, x14 + and x1, x1, x9 + and x21, x1, x6 + adcs x14, x19, x12 + adc x1, x24, x2 + cmp xzr, xzr + orr x12, x1, #0xfffffffffffffe00 + lsr x1, x1, #9 + adcs xzr, x16, x1 + and x21, x21, x14 + adcs xzr, x21, xzr + adcs xzr, x12, xzr + adcs x21, x16, x1 + adcs x1, x20, xzr + adcs x19, x8, xzr + stp x21, x1, [x0] + adcs x1, x4, xzr + adcs x21, x3, xzr + stp x19, x1, [x0, #16] + adcs x1, x9, xzr + stp x21, x1, [x0, #32] + adcs x21, x6, xzr + adcs x1, x14, xzr + stp x21, x1, [x0, #48] + adc x1, x12, xzr + and x1, x1, #0x1ff + str x1, [x0, #64] + +// Restore regs and return + + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_sqr_p521) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/bignum_sqr_p521_alt.S b/cbits/s2n/arm/bignum_sqr_p521_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_sqr_p521_alt.S @@ -0,0 +1,374 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Square modulo p_521, z := (x^2) mod p_521, assuming x reduced +// Input x[9]; output z[9] +// +// extern void bignum_sqr_p521_alt(uint64_t z[static 9], +// const uint64_t x[static 9]); +// +// Standard ARM ABI: X0 = z, X1 = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_sqr_p521_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_sqr_p521_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_sqr_p521_alt) + .text + .balign 4 + +#define z x0 +#define x x1 + +#define a0 x2 +#define a1 x3 +#define a2 x4 +#define a3 x5 +#define a4 x6 +#define a5 x7 +#define a6 x8 +#define a7 x9 +#define a8 x1 // Overwrites input argument at last load + +#define l x10 + +#define u0 x2 // The same as a0 +#define u1 x11 +#define u2 x12 +#define u3 x13 +#define u4 x14 +#define u5 x15 +#define u6 x16 +#define u7 x17 +#define u8 x19 +#define u9 x20 +#define u10 x21 +#define u11 x22 +#define u12 x23 +#define u13 x24 +#define u14 x25 +#define u15 x26 +#define u16 x4 // The same as a2 + +S2N_BN_SYMBOL(bignum_sqr_p521_alt): + CFI_START + +// It's convenient to have more registers to play with + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x26) + +// Load low 8 elements as [a7;a6;a5;a4;a3;a2;a1;a0], set up an initial +// window [u8;u7;u6;u5;u4;u3;u2;u1] = 10 + 20 + 30 + 40 + 50 + 60 + 70 + + ldp a0, a1, [x] + + mul u1, a0, a1 + umulh u2, a0, a1 + + ldp a2, a3, [x, #16] + + mul l, a0, a2 + umulh u3, a0, a2 + adds u2, u2, l + + ldp a4, a5, [x, #32] + + mul l, a0, a3 + umulh u4, a0, a3 + adcs u3, u3, l + + ldp a6, a7, [x, #48] + + mul l, a0, a4 + umulh u5, a0, a4 + adcs u4, u4, l + + mul l, a0, a5 + umulh u6, a0, a5 + adcs u5, u5, l + + mul l, a0, a6 + umulh u7, a0, a6 + adcs u6, u6, l + + mul l, a0, a7 + umulh u8, a0, a7 + adcs u7, u7, l + + adc u8, u8, xzr + +// Add in the next diagonal = 21 + 31 + 41 + 51 + 61 + 71 + 54 + + mul l, a1, a2 + adds u3, u3, l + mul l, a1, a3 + adcs u4, u4, l + mul l, a1, a4 + adcs u5, u5, l + mul l, a1, a5 + adcs u6, u6, l + mul l, a1, a6 + adcs u7, u7, l + mul l, a1, a7 + adcs u8, u8, l + cset u9, cs + + umulh l, a1, a2 + adds u4, u4, l + umulh l, a1, a3 + adcs u5, u5, l + umulh l, a1, a4 + adcs u6, u6, l + umulh l, a1, a5 + adcs u7, u7, l + umulh l, a1, a6 + adcs u8, u8, l + umulh l, a1, a7 + adc u9, u9, l + mul l, a4, a5 + umulh u10, a4, a5 + adds u9, u9, l + adc u10, u10, xzr + +// And the next one = 32 + 42 + 52 + 62 + 72 + 64 + 65 + + mul l, a2, a3 + adds u5, u5, l + mul l, a2, a4 + adcs u6, u6, l + mul l, a2, a5 + adcs u7, u7, l + mul l, a2, a6 + adcs u8, u8, l + mul l, a2, a7 + adcs u9, u9, l + mul l, a4, a6 + adcs u10, u10, l + cset u11, cs + + umulh l, a2, a3 + adds u6, u6, l + umulh l, a2, a4 + adcs u7, u7, l + umulh l, a2, a5 + adcs u8, u8, l + umulh l, a2, a6 + adcs u9, u9, l + umulh l, a2, a7 + adcs u10, u10, l + umulh l, a4, a6 + adc u11, u11, l + mul l, a5, a6 + umulh u12, a5, a6 + adds u11, u11, l + adc u12, u12, xzr + +// And the final one = 43 + 53 + 63 + 73 + 74 + 75 + 76 + + mul l, a3, a4 + adds u7, u7, l + mul l, a3, a5 + adcs u8, u8, l + mul l, a3, a6 + adcs u9, u9, l + mul l, a3, a7 + adcs u10, u10, l + mul l, a4, a7 + adcs u11, u11, l + mul l, a5, a7 + adcs u12, u12, l + cset u13, cs + + umulh l, a3, a4 + adds u8, u8, l + umulh l, a3, a5 + adcs u9, u9, l + umulh l, a3, a6 + adcs u10, u10, l + umulh l, a3, a7 + adcs u11, u11, l + umulh l, a4, a7 + adcs u12, u12, l + umulh l, a5, a7 + adc u13, u13, l + mul l, a6, a7 + umulh u14, a6, a7 + adds u13, u13, l + adc u14, u14, xzr + +// Double that, with u15 holding the top carry + + adds u1, u1, u1 + adcs u2, u2, u2 + adcs u3, u3, u3 + adcs u4, u4, u4 + adcs u5, u5, u5 + adcs u6, u6, u6 + adcs u7, u7, u7 + adcs u8, u8, u8 + adcs u9, u9, u9 + adcs u10, u10, u10 + adcs u11, u11, u11 + adcs u12, u12, u12 + adcs u13, u13, u13 + adcs u14, u14, u14 + cset u15, cs + +// Add the homogeneous terms 00 + 11 + 22 + 33 + 44 + 55 + 66 + 77 + + umulh l, a0, a0 + adds u1, u1, l + + mul l, a1, a1 + adcs u2, u2, l + umulh l, a1, a1 + adcs u3, u3, l + + mul l, a2, a2 + adcs u4, u4, l + umulh l, a2, a2 + adcs u5, u5, l + + mul l, a3, a3 + adcs u6, u6, l + umulh l, a3, a3 + adcs u7, u7, l + + mul l, a4, a4 + adcs u8, u8, l + umulh l, a4, a4 + adcs u9, u9, l + + mul l, a5, a5 + adcs u10, u10, l + umulh l, a5, a5 + adcs u11, u11, l + + mul l, a6, a6 + adcs u12, u12, l + umulh l, a6, a6 + adcs u13, u13, l + + mul l, a7, a7 + adcs u14, u14, l + umulh l, a7, a7 + adc u15, u15, l + +// Now load in the top digit a8, and immediately double the register + + ldr a8, [x, #64] + add a8, a8, a8 + +// Add (2 * a8) * [a7;...;a0] into the top of the buffer +// At the end of the first chain we form u16 = a8 ^ 2. +// This needs us to shift right the modified a8 again but it saves a +// register, and the overall performance impact seems slightly positive. + + mul l, a8, a0 + adds u8, u8, l + umulh l, a8, a0 + adcs u9, u9, l + mul l, a8, a2 + adcs u10, u10, l + umulh l, a8, a2 + adcs u11, u11, l + mul l, a8, a4 + adcs u12, u12, l + umulh l, a8, a4 + adcs u13, u13, l + mul l, a8, a6 + adcs u14, u14, l + umulh l, a8, a6 + adcs u15, u15, l + lsr u16, a8, #1 + mul u16, u16, u16 + adc u16, u16, xzr + + mul l, a8, a1 + adds u9, u9, l + umulh l, a8, a1 + adcs u10, u10, l + mul l, a8, a3 + adcs u11, u11, l + umulh l, a8, a3 + adcs u12, u12, l + mul l, a8, a5 + adcs u13, u13, l + umulh l, a8, a5 + adcs u14, u14, l + mul l, a8, a7 + adcs u15, u15, l + umulh l, a8, a7 + adc u16, u16, l + +// Finally squeeze in the lowest mul. This didn't need to be involved +// in the addition chains and moreover lets us re-use u0 == a0 + + mul u0, a0, a0 + +// Now we have the full product, which we consider as +// 2^521 * h + l. Form h + l + 1 + + subs xzr, xzr, xzr + extr l, u9, u8, #9 + adcs u0, u0, l + extr l, u10, u9, #9 + adcs u1, u1, l + extr l, u11, u10, #9 + adcs u2, u2, l + extr l, u12, u11, #9 + adcs u3, u3, l + extr l, u13, u12, #9 + adcs u4, u4, l + extr l, u14, u13, #9 + adcs u5, u5, l + extr l, u15, u14, #9 + adcs u6, u6, l + extr l, u16, u15, #9 + adcs u7, u7, l + orr u8, u8, #~0x1FF + lsr l, u16, #9 + adcs u8, u8, l + +// Now CF is set if h + l + 1 >= 2^521, which means it's already +// the answer, while if ~CF the answer is h + l so we should subtract +// 1 (all considered in 521 bits). Hence subtract ~CF and mask. + + sbcs u0, u0, xzr + sbcs u1, u1, xzr + sbcs u2, u2, xzr + sbcs u3, u3, xzr + sbcs u4, u4, xzr + sbcs u5, u5, xzr + sbcs u6, u6, xzr + sbcs u7, u7, xzr + sbc u8, u8, xzr + and u8, u8, #0x1FF + +// Store back digits of final result + + stp u0, u1, [z] + stp u2, u3, [z, #16] + stp u4, u5, [z, #32] + stp u6, u7, [z, #48] + str u8, [z, #64] + +// Restore registers and return + + CFI_POP2(x25,x26) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_sqr_p521_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/bignum_tomont_p256.S b/cbits/s2n/arm/bignum_tomont_p256.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_tomont_p256.S @@ -0,0 +1,122 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Convert to Montgomery form z := (2^256 * x) mod p_256 +// Input x[4]; output z[4] +// +// extern void bignum_tomont_p256(uint64_t z[static 4], +// const uint64_t x[static 4]); +// +// Standard ARM ABI: X0 = z, X1 = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_tomont_p256) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_tomont_p256) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_tomont_p256) + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_tomont_p256_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_tomont_p256_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_tomont_p256_alt) + .text + .balign 4 + +// ---------------------------------------------------------------------------- +// Core "x |-> (2^64 * x) mod p_256" macro, with x assumed to be < p_256. +// Input is in [d4;d3;d2;d1] and output in [d3;d2;d1;d0] +// using d4 as well as t1, t2, t3 as temporaries. +// ---------------------------------------------------------------------------- + +#define modstep_p256(d4, d3,d2,d1,d0, t1,t2,t3) \ +/* Writing the input as z = 2^256 * h + 2^192 * l + t = 2^192 * hl + t, */ \ +/* our quotient approximation is MIN ((hl + hl>>32 + 1)>>64) (2^64 - 1). */ \ + subs xzr, xzr, xzr __LF/* Set carry flag for +1 */ \ + extr t3, d4, d3, #32 __LF \ + adcs xzr, d3, t3 __LF \ + lsr t3, d4, #32 __LF \ + adcs t3, d4, t3 __LF \ + csetm d0, cs __LF \ + orr t3, t3, d0 __LF \ +/* First do [t2;t1] = 2^32 * q, which we use twice */ \ + lsl t1, t3, #32 __LF \ + lsr t2, t3, #32 __LF \ +/* Add 2^224 * q to sum */ \ + adds d3, d3, t1 __LF \ + adc d4, d4, t2 __LF \ +/* Accumulate [t2;t1;d0] = (2^96 - 1) * q */ \ + subs d0, xzr, t3 __LF \ + sbcs t1, t1, xzr __LF \ + sbc t2, t2, xzr __LF \ +/* Subtract (2^256 + 2^192 + 2^96 - 1) * q */ \ + subs d0, xzr, d0 __LF \ + sbcs d1, d1, t1 __LF \ + sbcs d2, d2, t2 __LF \ + sbcs d3, d3, t3 __LF \ + sbcs d4, d4, t3 __LF \ +/* Use top word as mask to correct */ \ + adds d0, d0, d4 __LF \ + mov t1, #0x00000000ffffffff __LF \ + and t1, t1, d4 __LF \ + adcs d1, d1, t1 __LF \ + adcs d2, d2, xzr __LF \ + mov t1, #0xffffffff00000001 __LF \ + and t1, t1, d4 __LF \ + adc d3, d3, t1 + +#define d0 x2 +#define d1 x3 +#define d2 x4 +#define d3 x5 +#define d4 x6 + +#define t0 x1 +#define t1 x7 +#define t2 x8 +#define t3 x9 + +S2N_BN_SYMBOL(bignum_tomont_p256): + +S2N_BN_SYMBOL(bignum_tomont_p256_alt): + CFI_START + +// Load the input + + ldp d0, d1, [x1] + ldp d2, d3, [x1, #16] + +// Do an initial reduction to make sure this is < p_256, using just +// a copy of the bignum_mod_p256_4 code. This is needed to set up the +// invariant "input < p_256" for the main modular reduction steps. + + mov t0, #0xffffffffffffffff + mov t1, #0x00000000ffffffff + mov t3, #0xffffffff00000001 + subs t0, d0, t0 + sbcs t1, d1, t1 + sbcs t2, d2, xzr + sbcs t3, d3, t3 + csel d0, d0, t0, cc + csel d1, d1, t1, cc + csel d2, d2, t2, cc + csel d3, d3, t3, cc + +// Successively multiply by 2^64 and reduce + + modstep_p256(d3,d2,d1,d0,d4, t1,t2,t3) + modstep_p256(d2,d1,d0,d4,d3, t1,t2,t3) + modstep_p256(d1,d0,d4,d3,d2, t1,t2,t3) + modstep_p256(d0,d4,d3,d2,d1, t1,t2,t3) + +// Store the result and return + + stp d1, d2, [x0] + stp d3, d4, [x0, #16] + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_tomont_p256) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/bignum_tomont_p384.S b/cbits/s2n/arm/bignum_tomont_p384.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/bignum_tomont_p384.S @@ -0,0 +1,138 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Convert to Montgomery form z := (2^384 * x) mod p_384 +// Input x[6]; output z[6] +// +// extern void bignum_tomont_p384(uint64_t z[static 6], +// const uint64_t x[static 6]); +// +// Standard ARM ABI: X0 = z, X1 = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_tomont_p384) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_tomont_p384) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_tomont_p384) + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_tomont_p384_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_tomont_p384_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_tomont_p384_alt) + .text + .balign 4 + +// ---------------------------------------------------------------------------- +// Core "x |-> (2^64 * x) mod p_384" macro, with x assumed to be < p_384. +// Input is in [d6;d5;d4;d3;d2;d1] and output in [d5;d4;d3;d2;d1;d0] +// using d6 as well as t1, t2, t3 as temporaries. +// ---------------------------------------------------------------------------- + +#define modstep_p384(d6,d5,d4,d3,d2,d1,d0, t1,t2,t3) \ +/* Initial quotient approximation q = min (h + 1) (2^64 - 1) */ \ + adds d6, d6, #1 __LF \ + csetm t3, cs __LF \ + add d6, d6, t3 __LF \ + orn t3, xzr, t3 __LF \ + sub t2, d6, #1 __LF \ + sub t1, xzr, d6 __LF \ +/* Correction term [d6;t2;t1;d0] = q * (2^384 - p_384) */ \ + lsl d0, t1, #32 __LF \ + extr t1, t2, t1, #32 __LF \ + lsr t2, t2, #32 __LF \ + adds d0, d0, d6 __LF \ + adcs t1, t1, xzr __LF \ + adcs t2, t2, d6 __LF \ + adc d6, xzr, xzr __LF \ +/* Addition to the initial value */ \ + adds d1, d1, t1 __LF \ + adcs d2, d2, t2 __LF \ + adcs d3, d3, d6 __LF \ + adcs d4, d4, xzr __LF \ + adcs d5, d5, xzr __LF \ + adc t3, t3, xzr __LF \ +/* Use net top of the 7-word answer in t3 for masked correction */ \ + mov t1, #0x00000000ffffffff __LF \ + and t1, t1, t3 __LF \ + adds d0, d0, t1 __LF \ + eor t1, t1, t3 __LF \ + adcs d1, d1, t1 __LF \ + mov t1, #0xfffffffffffffffe __LF \ + and t1, t1, t3 __LF \ + adcs d2, d2, t1 __LF \ + adcs d3, d3, t3 __LF \ + adcs d4, d4, t3 __LF \ + adc d5, d5, t3 + +S2N_BN_SYMBOL(bignum_tomont_p384): + +S2N_BN_SYMBOL(bignum_tomont_p384_alt): + CFI_START + +#define d0 x2 +#define d1 x3 +#define d2 x4 +#define d3 x5 +#define d4 x6 +#define d5 x7 +#define d6 x8 + +#define t1 x9 +#define t2 x10 +#define t3 x11 + +#define n0 x8 +#define n1 x9 +#define n2 x10 +#define n3 x11 +#define n4 x12 +#define n5 x1 + +// Load the inputs + + ldp d0, d1, [x1] + ldp d2, d3, [x1, #16] + ldp d4, d5, [x1, #32] + +// Do an initial reduction to make sure this is < p_384, using just +// a copy of the bignum_mod_p384_6 code. This is needed to set up the +// invariant "input < p_384" for the main modular reduction steps. + + mov n0, #0x00000000ffffffff + mov n1, #0xffffffff00000000 + mov n2, #0xfffffffffffffffe + subs n0, d0, n0 + sbcs n1, d1, n1 + sbcs n2, d2, n2 + adcs n3, d3, xzr + adcs n4, d4, xzr + adcs n5, d5, xzr + csel d0, d0, n0, cc + csel d1, d1, n1, cc + csel d2, d2, n2, cc + csel d3, d3, n3, cc + csel d4, d4, n4, cc + csel d5, d5, n5, cc + +// Successively multiply by 2^64 and reduce + + modstep_p384(d5,d4,d3,d2,d1,d0,d6, t1,t2,t3) + modstep_p384(d4,d3,d2,d1,d0,d6,d5, t1,t2,t3) + modstep_p384(d3,d2,d1,d0,d6,d5,d4, t1,t2,t3) + modstep_p384(d2,d1,d0,d6,d5,d4,d3, t1,t2,t3) + modstep_p384(d1,d0,d6,d5,d4,d3,d2, t1,t2,t3) + modstep_p384(d0,d6,d5,d4,d3,d2,d1, t1,t2,t3) + +// Store the result and return + + stp d1, d2, [x0] + stp d3, d4, [x0, #16] + stp d5, d6, [x0, #32] + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_tomont_p384) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/curve25519_x25519.S b/cbits/s2n/arm/curve25519_x25519.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/curve25519_x25519.S @@ -0,0 +1,2596 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ********************************************************************** +// This code is substantially derived from Emil Lenngren's implementation +// +// https://github.com/Emill/X25519-AArch64/blob/master/X25519_AArch64.pdf +// https://github.com/Emill/X25519-AArch64 +// +// and the SLOTHY-based re-engineering of that code by Abdulrahman, Becker, +// Kannwischer and Klein: +// +// https://eprint.iacr.org/2022/1303.pdf +// https://github.com/slothy-optimizer/slothy/tree/main/paper +// ********************************************************************** + +// ---------------------------------------------------------------------------- +// The x25519 function for curve25519 +// Inputs scalar[4], point[4]; output res[4] +// +// extern void curve25519_x25519 +// (uint64_t res[static 4],const uint64_t scalar[static 4], +// const uint64_t point[static 4]); +// +// Given a scalar n and the X coordinate of an input point P = (X,Y) on +// curve25519 (Y can live in any extension field of characteristic 2^255-19), +// this returns the X coordinate of n * P = (X, Y), or 0 when n * P is the +// point at infinity. Both n and X inputs are first slightly modified/mangled +// as specified in the relevant RFC (https://www.rfc-editor.org/rfc/rfc7748); +// in particular the lower three bits of n are set to zero. Does not implement +// the zero-check specified in Section 6.1. +// +// Standard ARM ABI: X0 = res, X1 = scalar, X2 = point +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(curve25519_x25519) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(curve25519_x25519) + S2N_BN_SYM_PRIVACY_DIRECTIVE(curve25519_x25519) + + .text + .balign 4 + +// Pointer-offset pairs for temporaries on stack + +#define scalar sp, #0 +#define pointx sp, #32 +#define mask1 sp, #72 +#define mask2 sp, #80 +#define tmpa sp, #88 +#define tmpb sp, #128 +#define xn sp, #128 +#define zn sp, #160 + +#define res sp, #192 +#define i sp, #200 +#define swap sp, #208 + +// Total size to reserve on the stack + +#define NSPACE 224 +#define regsave sp, #NSPACE + +S2N_BN_SYMBOL(curve25519_x25519): + CFI_START + +// Save registers and make additional room #NSPACE for temporaries. +// We only need to save the low 64-bits of the Q8...Q15 registers +// according to the ABI, so we use a save of the D8...D15 forms. + + CFI_DEC_SP(NSPACE+160) + CFI_STACKSAVE2(d8,d9,NSPACE+0) + CFI_STACKSAVE2(d10,d11,NSPACE+16) + CFI_STACKSAVE2(d12,d13,NSPACE+32) + CFI_STACKSAVE2(d14,d15,NSPACE+48) + CFI_STACKSAVE2(x19,x20,NSPACE+64) + CFI_STACKSAVE2(x21,x22,NSPACE+80) + CFI_STACKSAVE2(x23,x24,NSPACE+96) + CFI_STACKSAVE2(x25,x26,NSPACE+112) + CFI_STACKSAVE2(x27,x28,NSPACE+128) + CFI_STACKSAVE2(x29,x30,NSPACE+144) + +// Move the output pointer to a stable place + + str x0, [res] + +// Copy the scalar to the corresponding local variable while +// mangling it. In principle it becomes 01xxx...xxx000 where +// the xxx are the corresponding bits of the original input +// scalar. We actually don't bother forcing the MSB to zero, +// but rather start the main loop below at 254 instead of 255. + + ldp x10, x11, [x1] + bic x10, x10, #7 + stp x10, x11, [scalar] + ldp x12, x13, [x1, #16] + orr x13, x13, #0x4000000000000000 + stp x12, x13, [scalar+16] + +// Discard the MSB of the point X coordinate (this is in +// accordance with the RFC, mod 2^255, *not* 2^255-19). +// Then recode it into the unsaturated base 25.5 form. + + ldp x0, x1, [x2] + ldp x2, x3, [x2, #16] + + lsr x12, x0, #51 + lsr x17, x2, #51 + orr x12, x12, x1, lsl #13 + orr x17, x17, x3, lsl #13 + ubfx x8, x3, #12, #26 + ubfx x9, x3, #38, #25 + ubfx x11, x0, #26, #25 + ubfx x13, x1, #13, #25 + lsr x14, x1, #38 + ubfx x16, x2, #25, #26 + and x10, x0, #0x3ffffff + and x12, x12, #0x3ffffff + and x15, x2, #0x1ffffff + and x17, x17, #0x1ffffff + orr x10, x10, x11, lsl #32 + orr x11, x12, x13, lsl #32 + orr x12, x14, x15, lsl #32 + orr x13, x16, x17, lsl #32 + orr x14, x8, x9, lsl #32 + + stp x10, x11, [pointx+0] + stp x12, x13, [pointx+16] + str x14, [pointx+32] + +// Initialize (X2,Z2) = (1,0), the identity (projective point at infinity) + + mov x1, #1 + mov v0.d[0], x1 + mov v2.d[0], xzr + mov v4.d[0], xzr + mov v6.d[0], xzr + mov v8.d[0], xzr + + mov v1.d[0], xzr + mov v3.d[0], xzr + mov v5.d[0], xzr + mov v7.d[0], xzr + mov v9.d[0], xzr + +// Initialize (X3,Z3) = (X,1), projective representation of X + + mov v10.d[0], x10 + mov v12.d[0], x11 + mov v14.d[0], x12 + mov v16.d[0], x13 + mov v18.d[0], x14 + + mov v11.d[0], x1 + mov v13.d[0], xzr + mov v15.d[0], xzr + mov v17.d[0], xzr + mov v19.d[0], xzr + +// Set up some constants used repeatedly in the main loop: +// +// Q31 = 0x1300000013 (two 32-bit copies of 19) +// Q30 = 0x3ffffff0000000003ffffff (two 64-bit copies of 2^26-1) +// Q29 = mask1 = (0x07ffffc,0x07fffffe) +// Q28 = mask2 = (0x07ffffb4,0x07fffffe) + + mov w0, #19 + add x0, x0, x0, lsl #32 + mov v31.d[0], x0 + mov v31.d[1], xzr + + mov x0, #67108863 // #(1<<26)-1 + mov v30.d[0], x0 + mov v30.d[1], x0 + + mov x0, #0x07fffffe07fffffe + sub x1, x0, #74 // #0xfe-0xb4 + sub x0, x0, #2 + + stp x0, x1, [mask1] + ldp d29, d28, [mask1] + +// The main loop over (modified) bits from i = 254, ..., i = 0 (inclusive); +// we explicitly skip bit 255 because it should be forced to zero initially. +// This is a classic Montgomery ladder using a "swap" variable. +// It's assumed x0 = i at the start of the loop, but that is volatile and +// needs to be reloaded from memory at the end of the loop. + + str xzr, [swap] + mov x0, #254 + str x0, [i] + +Lcurve25519_x25519_scalarloop: + + lsr x1, x0, #6 + ldr x2, [sp, x1, lsl #3] // Exploiting scalar = sp exactly + lsr x2, x2, x0 + and x2, x2, #1 + + ldr x0, [swap] + cmp x0, x2 + str x2, [swap] + +// The following inner loop code is derived closely following Lenngren's +// implementation available at "https://github.com/Emill/X25519-AArch64". +// In particular, the basic dataflow and the organization between integer +// and SIMD units is identical, with only a few minor changes to some +// individual instructions (for miscellaneous reasons). The scheduling +// was redone from scratch by SLOTHY starting from the un-interleaved +// form in the SLOTHY work cited above, and using the same scripts. +// +// The intermediate value annotations were added to provide data that +// is used in the formal proof, indicating which lines assign specific +// digits of the various intermediate results (mainly of field +// operations, sometimes other transformations). The names used for +// the intermediate results are similar but not identical to those in +// the abstract Algorithm 1 description in Lenngren's paper. Almost +// all equations are to be interpreted as field operations, i.e. as +// arithmetic modulo 2^255-19, not simple numeric equalities. +// +// b = x2 - z2 +// d = x3 - z3 +// a = x2 + z2 +// c = x3 + z3 +// f = if flip then c else a +// g = if flip then d else b +// aa = f^2 +// bb = g^2 +// bbalt = bb (change of representation) +// e = aa - bb +// bce = bbalt + 121666 * e +// z4 = bce * e +// bc = b * c +// ad = a * d +// t1 = ad + bc +// t2 = ad - bc +// x5 = t1^2 +// t3 = t2^2 +// x4 = aa * bb +// z5 = x * t3 +// +// Then the main variables are updated for the next iteration as +// +// (x2',z2') = (x4,z4) +// (x3',z3') = (x5,z5) + + add v22.2s, v2.2s, v3.2s // ubignum_of_qreglist 1 // INTERMEDIATE a + sub v21.2s, v28.2s, v1.2s + add v25.2s, v0.2s, v1.2s // ubignum_of_qreglist 0 // INTERMEDIATE a + sub v24.2s, v29.2s, v3.2s + add v3.2s, v18.2s, v19.2s // ubignum_of_qreglist 4 // INTERMEDIATE c + add v0.2s, v0.2s, v21.2s // ubignum_of_qreglist 0 // INTERMEDIATE b + sub v20.2s, v29.2s, v15.2s + sub v1.2s, v29.2s, v5.2s + sub v26.2s, v28.2s, v11.2s + sub v21.2s, v29.2s, v19.2s + add v19.2s, v10.2s, v11.2s // ubignum_of_qreglist 0 // INTERMEDIATE c + add v11.2s, v14.2s, v20.2s // ubignum_of_qreglist 2 // INTERMEDIATE d + add v21.2s, v18.2s, v21.2s // ubignum_of_qreglist 4 // INTERMEDIATE d + sub v20.2s, v29.2s, v17.2s + add v18.2s, v2.2s, v24.2s // ubignum_of_qreglist 1 // INTERMEDIATE b + add v14.2s, v14.2s, v15.2s // ubignum_of_qreglist 2 // INTERMEDIATE c + add v15.2s, v16.2s, v17.2s // ubignum_of_qreglist 3 // INTERMEDIATE c + add v2.2s, v16.2s, v20.2s // ubignum_of_qreglist 3 // INTERMEDIATE d + add v24.2s, v12.2s, v13.2s // ubignum_of_qreglist 1 // INTERMEDIATE c + add v26.2s, v10.2s, v26.2s // ubignum_of_qreglist 0 // INTERMEDIATE d + sub v10.2s, v29.2s, v13.2s + sub v13.2s, v29.2s, v7.2s + add v23.2s, v6.2s, v7.2s // ubignum_of_qreglist 3 // INTERMEDIATE a + sub v7.2s, v29.2s, v9.2s + add v27.2s, v12.2s, v10.2s // ubignum_of_qreglist 1 // INTERMEDIATE d + fcsel d20, d22, d24, eq // ubignum_of_qreglist 1 // INTERMEDIATE f + add v28.2s, v4.2s, v5.2s // ubignum_of_qreglist 2 // INTERMEDIATE a + fcsel d12, d23, d15, eq // ubignum_of_qreglist 3 // INTERMEDIATE f + add v7.2s, v8.2s, v7.2s // ubignum_of_qreglist 4 // INTERMEDIATE b + fcsel d16, d25, d19, eq // ubignum_of_qreglist 0 // INTERMEDIATE f + mov x0, v20.d[0] + fcsel d5, d28, d14, eq // ubignum_of_qreglist 2 // INTERMEDIATE f + mov x21, v12.d[0] + fcsel d29, d7, d21, eq // ubignum_of_qreglist 4 // INTERMEDIATE g + mov x5, v16.d[0] + lsr x26, x0, #32 + add x29, x21, x21 + umull x15, w5, w29 + add v13.2s, v6.2s, v13.2s // ubignum_of_qreglist 3 // INTERMEDIATE b + add x12, x26, x26 + mov x30, v5.d[0] + fcsel d10, d18, d27, eq // ubignum_of_qreglist 1 // INTERMEDIATE g + lsr x11, x5, #32 + lsr x10, x30, #32 + trn2 v20.2s, v21.2s, v3.2s + add v9.2s, v8.2s, v9.2s // ubignum_of_qreglist 4 // INTERMEDIATE a + add x14, x11, x11 + trn2 v6.2s, v2.2s, v15.2s + trn1 v12.2s, v25.2s, v0.2s + add v1.2s, v4.2s, v1.2s // ubignum_of_qreglist 2 // INTERMEDIATE b + trn1 v16.2s, v23.2s, v13.2s + fcsel d8, d13, d2, eq // ubignum_of_qreglist 3 // INTERMEDIATE g + trn2 v17.2s, v27.2s, v24.2s + str d29, [tmpb+32] + add x17, x10, x10 + trn2 v4.2s, v28.2s, v1.2s + trn1 v5.2s, v28.2s, v1.2s + trn1 v28.2s, v2.2s, v15.2s + trn1 v2.2s, v22.2s, v18.2s + fcsel d29, d0, d26, eq // ubignum_of_qreglist 0 // INTERMEDIATE g + trn2 v15.2s, v22.2s, v18.2s + umull v22.2d, v12.2s, v20.2s + umull x22, w30, w17 + stp d29, d10, [tmpb+0] + trn2 v10.2s, v23.2s, v13.2s + trn2 v23.2s, v11.2s, v14.2s + trn1 v13.2s, v27.2s, v24.2s + fcsel d27, d1, d11, eq // ubignum_of_qreglist 2 // INTERMEDIATE g + trn1 v14.2s, v11.2s, v14.2s + umlal v22.2d, v2.2s, v6.2s + umull x25, w30, w30 + umlal v22.2d, v5.2s, v23.2s + add x3, x30, x30 + umlal v22.2d, v16.2s, v17.2s + add w30, w21, w21, lsl #1; + stp d27, d8, [tmpb+16] + add w30, w30, w21, lsl #4 + trn1 v11.2s, v26.2s, v19.2s + trn2 v8.2s, v26.2s, v19.2s + trn2 v19.2s, v25.2s, v0.2s + mul v29.2s, v20.2s, v31.2s + ldr x20, [tmpb+24] + umull v25.2d, v19.2s, v6.2s + add x1, x0, x0 + umull v27.2d, v19.2s, v23.2s + umull x9, w5, w1 + umull v0.2d, v12.2s, v23.2s + lsr x24, x20, #32 + mul v20.2s, v23.2s, v31.2s + lsr x16, x21, #32 + umlal v25.2d, v15.2s, v23.2s + umaddl x13, w11, w14, x9 + umlal v25.2d, v4.2s, v17.2s + umaddl x9, w14, w17, x15 + umull v24.2d, v12.2s, v6.2s + add w2, w16, w16, lsl #1; + fcsel d26, d9, d3, eq // ubignum_of_qreglist 4 // INTERMEDIATE f + add w2, w2, w16, lsl #4 + trn1 v18.2s, v21.2s, v3.2s + umull v3.2d, v19.2s, v29.2s + umull x28, w5, w3 + mul v1.2s, v6.2s, v31.2s + umull x8, w5, w5 + umlal v24.2d, v2.2s, v23.2s + umaddl x13, w21, w30, x13 + mul v23.2s, v17.2s, v31.2s + umaddl x27, w14, w12, x28 + trn2 v6.2s, v9.2s, v7.2s + mov x6, v26.d[0] + umlal v3.2d, v15.2s, v1.2s + add x16, x16, x16 + umlal v3.2d, v4.2s, v20.2s + lsr x4, x6, #32 + umlal v3.2d, v10.2s, v23.2s + add x7, x6, x6 + umull v26.2d, v19.2s, v8.2s + add x23, x4, x4 + umaddl x28, w5, w23, x22 + trn1 v7.2s, v9.2s, v7.2s + umlal v27.2d, v15.2s, v17.2s + add w15, w4, w4, lsl #1; + umlal v27.2d, v4.2s, v8.2s + add w15, w15, w4, lsl #4 + add w22, w10, w10, lsl #1; + umlal v24.2d, v5.2s, v17.2s + add w22, w22, w10, lsl #4 + umaddl x10, w11, w7, x28 + umlal v25.2d, v10.2s, v8.2s + umull x21, w5, w16 + umlal v25.2d, v6.2s, v29.2s + umaddl x23, w15, w23, x25 + umlal v27.2d, v10.2s, v29.2s + umull x19, w5, w12 + umlal v27.2d, v6.2s, v1.2s + umaddl x25, w11, w29, x21 + umlal v0.2d, v2.2s, v17.2s + umaddl x28, w0, w3, x9 + shl v21.2d, v25.2d, #1 + umaddl x4, w11, w1, x19 + umaddl x21, w2, w29, x4 + mul v25.2s, v8.2s, v31.2s + umlal v24.2d, v16.2s, v8.2s + umaddl x19, w0, w17, x25 + umlal v24.2d, v7.2s, v29.2s + umull x25, w5, w17 + umlal v24.2d, v19.2s, v28.2s + umaddl x4, w0, w16, x10 + umull v9.2d, v12.2s, v8.2s + umaddl x23, w5, w7, x23 + umlal v21.2d, v12.2s, v18.2s + add w10, w6, w6, lsl #1; + shl v27.2d, v27.2d, #1 + add w10, w10, w6, lsl #4 + umaddl x28, w26, w12, x28 + umlal v26.2d, v15.2s, v29.2s + umaddl x9, w14, w16, x23 + umlal v9.2d, v2.2s, v29.2s + umaddl x22, w22, w17, x8 + umlal v21.2d, v2.2s, v28.2s + umaddl x28, w6, w10, x28 + umaddl x27, w0, w0, x27 + add x8, x14, x14 + umlal v0.2d, v5.2s, v8.2s + umull x5, w5, w14 + umlal v9.2d, v5.2s, v1.2s + umaddl x14, w0, w29, x9 + umlal v26.2d, v4.2s, v1.2s + umaddl x6, w2, w16, x27 + umlal v22.2d, v7.2s, v8.2s + umaddl x5, w30, w17, x5 + umaddl x5, w2, w3, x5 + add x23, x17, x17 + umlal v27.2d, v12.2s, v28.2s + umaddl x13, w2, w23, x13 + umlal v26.2d, v10.2s, v20.2s + add x9, x12, x12 + umlal v9.2d, v16.2s, v20.2s + umaddl x27, w10, w29, x6 + umlal v0.2d, v16.2s, v29.2s + umaddl x6, w11, w3, x25 + umlal v22.2d, v19.2s, v18.2s + umaddl x19, w26, w3, x19 + mul v18.2s, v18.2s, v31.2s + umaddl x23, w15, w23, x27 + umlal v3.2d, v6.2s, v25.2s + umaddl x0, w0, w12, x6 + umlal v0.2d, v7.2s, v1.2s + add x11, x16, x16 + umlal v9.2d, v7.2s, v23.2s + umaddl x6, w12, w17, x14 + umlal v9.2d, v19.2s, v11.2s + umaddl x25, w26, w29, x4 + umlal v9.2d, v15.2s, v18.2s + umaddl x14, w10, w3, x13 + umull v25.2d, v12.2s, v17.2s + umaddl x27, w10, w16, x0 + umlal v26.2d, v6.2s, v23.2s + add x0, x25, x6, lsr #26 + mul v23.2s, v28.2s, v31.2s + umaddl x12, w10, w12, x5 + shl v3.2d, v3.2d, #1 + add x16, x22, x0, lsr #25 + umlal v21.2d, v5.2s, v14.2s + bic x22, x0, #0x1ffffff + umlal v3.2d, v12.2s, v11.2s + add x26, x16, x22, lsr #24 + umlal v3.2d, v2.2s, v18.2s + umaddl x16, w10, w17, x21 + umlal v3.2d, v5.2s, v23.2s + add x22, x26, x22, lsr #21 + umlal v9.2d, v4.2s, v23.2s + umaddl x5, w15, w29, x27 + umull v17.2d, v19.2s, v17.2s + umaddl x17, w30, w3, x22 + umlal v25.2d, v2.2s, v8.2s + umaddl x25, w15, w3, x16 + umlal v25.2d, v5.2s, v29.2s + umaddl x26, w15, w7, x19 + umlal v0.2d, v19.2s, v14.2s + umaddl x17, w2, w9, x17 + umlal v17.2d, v15.2s, v8.2s + ldr x19, [tmpb+0] + umlal v17.2d, v4.2s, v29.2s + ldr x7, [tmpb+8] + shl v29.2d, v26.2d, #1 + umaddl x13, w10, w1, x17 + umlal v0.2d, v15.2s, v13.2s + lsr x2, x19, #32 + umlal v29.2d, v12.2s, v13.2s + umaddl x27, w15, w1, x12 + umlal v29.2d, v2.2s, v11.2s + umaddl x30, w15, w8, x13 + umlal v29.2d, v5.2s, v18.2s + add x4, x7, x7 + umlal v29.2d, v16.2s, v23.2s + umaddl x29, w15, w9, x14 + umlal v0.2d, v4.2s, v11.2s + add x17, x27, x30, lsr #26 + umlal v0.2d, v10.2s, v18.2s + umaddl x16, w15, w11, x28 + umlal v0.2d, v6.2s, v23.2s + add x1, x29, x17, lsr #25 + umlal v25.2d, v16.2s, v1.2s + umull x11, w19, w4 + ldr x8, [tmpb+32] + mul v26.2s, v14.2s, v31.2s + umlal v17.2d, v10.2s, v1.2s + ldr x15, [tmpb+16] + umlal v17.2d, v6.2s, v20.2s + and x9, x30, #0x3ffffff + bfi x9, x17, #32, #25 // ubignum_of_preglist 0 // INTERMEDIATE aa + add x17, x2, x2 + lsr x10, x15, #32 + add x27, x25, x1, lsr #26 + umlal v25.2d, v7.2s, v20.2s + add x13, x10, x10 + umlal v25.2d, v19.2s, v13.2s + add x29, x23, x27, lsr #25 + umlal v25.2d, v15.2s, v11.2s + lsr x30, x8, #32 + umlal v25.2d, v4.2s, v18.2s + add x23, x5, x29, lsr #26 + umlal v25.2d, v10.2s, v23.2s + and x14, x29, #0x3ffffff + umlal v25.2d, v6.2s, v26.2s + add x5, x16, x23, lsr #25 + shl v8.2d, v17.2d, #1 + umaddl x12, w2, w17, x11 + and x29, x5, #0x3ffffff + umull x21, w19, w19 + umlal v29.2d, v7.2s, v26.2s + add w16, w10, w10, lsl #1; + umlal v3.2d, v16.2s, v26.2s + add w16, w16, w10, lsl #4 + bfi x14, x23, #32, #25 // ubignum_of_preglist 2 // INTERMEDIATE aa + add w10, w24, w24, lsl #1; + add x22, x26, x5, lsr #26 + add w10, w10, w24, lsl #4 + umlal v8.2d, v12.2s, v14.2s + umaddl x25, w16, w13, x21 + umlal v8.2d, v2.2s, v13.2s + bfi x29, x22, #32, #25 // ubignum_of_preglist 3 // INTERMEDIATE aa + umlal v8.2d, v5.2s, v11.2s + add x26, x24, x24 + umlal v8.2d, v16.2s, v18.2s + stp x14, x29, [tmpa+16] + umlal v8.2d, v7.2s, v23.2s + add w24, w30, w30, lsl #1; + usra v25.2d, v29.2d, #26 + add w24, w24, w30, lsl #4 + umull x29, w15, w15 + umlal v27.2d, v2.2s, v14.2s + umull x3, w15, w13 + umlal v27.2d, v5.2s, v13.2s + add x21, x20, x20 + umlal v24.2d, v15.2s, v14.2s + umull x5, w19, w21 + umlal v24.2d, v4.2s, v13.2s + and x11, x1, #0x3ffffff + usra v8.2d, v25.2d, #25 + and x1, x0, #0x1ffffff + umlal v27.2d, v16.2s, v11.2s + umaddl x23, w17, w13, x5 + umlal v27.2d, v7.2s, v18.2s + add x5, x30, x30 + usra v0.2d, v8.2d, #26 + add x0, x15, x15 + umlal v24.2d, v10.2s, v11.2s + umaddl x23, w7, w0, x23 + umlal v24.2d, v6.2s, v18.2s + lsr x30, x7, #32 + usra v27.2d, v0.2d, #25 + add x16, x30, x30 + and v20.16b, v8.16b, v30.16b // ubignum_of_hreglist 4 + ubignum_of_lreglist 4 // INTERMEDIATE H|L = bc|ad + umaddl x15, w30, w16, x23 + ushr v23.2d, v30.2d, #1 + add w23, w8, w8, lsl #1; + usra v24.2d, v27.2d, #26 + add w23, w23, w8, lsl #4 + umaddl x14, w19, w5, x3 + and v8.16b, v27.16b, v30.16b // ubignum_of_hreglist 6 + ubignum_of_lreglist 6 // INTERMEDIATE H|L = bc|ad + add x28, x8, x8 + and v27.16b, v0.16b, v23.16b // ubignum_of_hreglist 5 + ubignum_of_lreglist 5 // INTERMEDIATE H|L = bc|ad + umaddl x8, w8, w23, x15 + and v5.16b, v24.16b, v23.16b // ubignum_of_hreglist 7 + ubignum_of_lreglist 7 // INTERMEDIATE H|L = bc|ad + umaddl x3, w2, w28, x14 + umlal v22.2d, v15.2s, v28.2s + bfi x11, x27, #32, #25 // ubignum_of_preglist 1 // INTERMEDIATE aa + uzp1 v5.4s, v8.4s, v5.4s + umaddl x14, w24, w5, x29 + umaddl x5, w19, w28, x14 + ldr d18, [mask1] + mov v18.d[1], v18.d[0] + umaddl x15, w7, w26, x3 + mul v12.2s, v13.2s, v31.2s + umlal v21.2d, v16.2s, v13.2s + stp x9, x11, [tmpa+0] + umlal v21.2d, v7.2s, v11.2s + umaddl x29, w17, w26, x5 + umlal v22.2d, v4.2s, v14.2s + add w14, w20, w20, lsl #1; + umlal v22.2d, v10.2s, v13.2s + add w14, w14, w20, lsl #4 + umull x3, w19, w0 + umlal v22.2d, v6.2s, v11.2s + umaddl x29, w7, w21, x29 + usra v21.2d, v24.2d, #25 + umaddl x11, w20, w14, x12 + and v0.16b, v25.16b, v23.16b + umaddl x5, w30, w21, x15 + and v14.16b, v29.16b, v30.16b + umaddl x12, w16, w13, x29 + usra v22.2d, v21.2d, #26 + umaddl x29, w17, w16, x3 + umlal v3.2d, v7.2s, v12.2s + add x9, x26, x26 + and v1.16b, v21.16b, v30.16b // ubignum_of_hreglist 8 + ubignum_of_lreglist 8 // INTERMEDIATE H|L = bc|ad + add x27, x5, x12, lsr #26 + bic v8.16b, v22.16b, v23.16b + umaddl x29, w7, w7, x29 + and v17.16b, v22.16b, v23.16b // ubignum_of_hreglist 9 + ubignum_of_lreglist 9 // INTERMEDIATE H|L = bc|ad + add x5, x25, x27, lsr #25 + usra v3.2d, v8.2d, #25 + umaddl x25, w24, w9, x8 + umlal v9.2d, v10.2s, v26.2s + add x8, x13, x13 + trn1 v22.4s, v1.4s, v17.4s + umaddl x11, w10, w8, x11 + usra v3.2d, v8.2d, #24 + umull x20, w19, w16 + add v26.2s, v22.2s, v18.2s + ldr d28, [mask2] + umlal v9.2d, v6.2s, v12.2s + umaddl x3, w23, w0, x11 + usra v3.2d, v8.2d, #21 + umaddl x29, w10, w26, x29 + uzp1 v11.4s, v20.4s, v27.4s + umaddl x20, w2, w4, x20 + umaddl x9, w10, w21, x20 + mov v17.d[0], v22.d[1] + usra v9.2d, v3.2d, #26 + umull x15, w19, w13 + and v7.16b, v3.16b, v30.16b // ubignum_of_hreglist 0 + ubignum_of_lreglist 0 // INTERMEDIATE H|L = bc|ad + add x11, x16, x16 + uzp2 v1.4s, v11.4s, v5.4s + umaddl x20, w23, w13, x9 + and v8.16b, v9.16b, v23.16b // ubignum_of_hreglist 1 + ubignum_of_lreglist 1 // INTERMEDIATE H|L = bc|ad + umaddl x9, w2, w0, x15 + usra v14.2d, v9.2d, #25 + and x6, x6, #0x3ffffff + uzp1 v7.4s, v7.4s, v8.4s + umaddl x29, w23, w21, x29 + uzp1 v27.4s, v11.4s, v5.4s + umull x15, w19, w26 + usra v0.2d, v14.2d, #26 // ubignum_of_hreglist 3 + ubignum_of_lreglist 3 // INTERMEDIATE H|L = bc|ad + add x6, x6, x22, lsr #25 + and v3.16b, v14.16b, v30.16b // ubignum_of_hreglist 2 + ubignum_of_lreglist 2 // INTERMEDIATE H|L = bc|ad + bic x22, x27, #0x1ffffff + sub v2.2s, v26.2s, v17.2s + add v9.2s, v22.2s, v17.2s + uzp1 v14.4s, v3.4s, v0.4s + umaddl x2, w2, w21, x15 + add v5.4s, v27.4s, v18.4s + add x5, x5, x22, lsr #24 + zip1 v22.2s, v2.2s, v9.2s // ubignum_of_h32reglist 8 + ubignum_of_l32reglist 8 // INTERMEDIATE H|L = t1|t2 + mov v18.b[0], v28.b[0] + uzp1 v8.4s, v7.4s, v14.4s + add x22, x5, x22, lsr #21 + uzp2 v3.4s, v7.4s, v14.4s + umaddl x5, w7, w16, x9 + add v25.4s, v8.4s, v18.4s + umaddl x15, w14, w0, x22 + add v12.4s, v27.4s, v1.4s + add x9, x17, x17 + sub v14.4s, v5.4s, v1.4s + umull x19, w19, w17 + sub v18.4s, v25.4s, v3.4s + ldr x22, [tmpa+8] + add v20.4s, v8.4s, v3.4s + umaddl x15, w10, w11, x15 + zip1 v16.4s, v14.4s, v12.4s // ubignum_of_h32reglist 4 + ubignum_of_l32reglist 4 // INTERMEDIATE H|L = t1|t2 + umaddl x14, w14, w13, x19 + zip2 v14.4s, v14.4s, v12.4s // ubignum_of_h32reglist 6 + ubignum_of_l32reglist 6 // INTERMEDIATE H|L = t1|t2 + and x17, x27, #0x1ffffff + zip2 v0.4s, v18.4s, v20.4s // ubignum_of_h32reglist 2 + ubignum_of_l32reglist 2 // INTERMEDIATE H|L = t1|t2 + umaddl x15, w23, w4, x15 + zip1 v1.4s, v18.4s, v20.4s // ubignum_of_h32reglist 0 + ubignum_of_l32reglist 0 // INTERMEDIATE H|L = t1|t2 + umaddl x10, w10, w0, x14 + zip2 v5.2s, v2.2s, v9.2s // ubignum_of_h32reglist 9 + ubignum_of_l32reglist 9 // INTERMEDIATE H|L = t1|t2 + shl v24.2s, v0.2s, #1 + mov v19.d[0], v1.d[1] // ubignum_of_h32reglist 1 + ubignum_of_l32reglist 1 // INTERMEDIATE H|L = t1|t2 + shl v26.2s, v22.2s, #1 + shl v17.2s, v16.2s, #1 + mov v15.d[0], v0.d[1] // ubignum_of_h32reglist 3 + ubignum_of_l32reglist 3 // INTERMEDIATE H|L = t1|t2 + shl v7.2s, v5.2s, #1 + shl v18.2s, v19.2s, #1 + umull v11.2d, v1.2s, v24.2s + umaddl x19, w23, w16, x10 + umull v6.2d, v1.2s, v17.2s + umaddl x10, w7, w13, x2 + mov v4.d[0], v16.d[1] // ubignum_of_h32reglist 5 + ubignum_of_l32reglist 5 // INTERMEDIATE H|L = t1|t2 + mov v10.d[0], v14.d[1] // ubignum_of_h32reglist 7 + ubignum_of_l32reglist 7 // INTERMEDIATE H|L = t1|t2 + umull v9.2d, v1.2s, v26.2s + ldr x13, [tmpa+0] + shl v28.2s, v15.2s, #1 + shl v3.2s, v10.2s, #1 + ldr x14, [tmpa+16] + mul v12.2s, v10.2s, v31.2s + umull v25.2d, v1.2s, v7.2s + ldr x2, [tmpa+24] + umlal v6.2d, v18.2s, v28.2s + umaddl x27, w30, w0, x10 + umaddl x16, w24, w0, x20 + shl v13.2s, v14.2s, #1 + umaddl x5, w23, w26, x5 + mul v2.2s, v22.2s, v31.2s + umull v21.2d, v1.2s, v13.2s + umaddl x23, w24, w8, x29 + umlal v11.2d, v18.2s, v19.2s + mov x10, #0x07fffffe07fffffe + sub x10, x10, #2 + umaddl x26, w24, w21, x5 + mul v29.2s, v14.2s, v31.2s + umlal v25.2d, v19.2s, v26.2s + add x7, x1, x6, lsr #26 + mul v20.2s, v4.2s, v31.2s + and x6, x6, #0x3ffffff + shl v8.2s, v18.2s, #1 + shl v4.2s, v4.2s, #1 + umlal v11.2d, v29.2s, v14.2s + bfi x6, x7, #32, #26 // ubignum_of_preglist 4 // INTERMEDIATE aa + umlal v25.2d, v0.2s, v3.2s + umaddl x0, w24, w4, x19 + umlal v25.2d, v15.2s, v13.2s + str x6, [tmpa+32] + umlal v21.2d, v18.2s, v4.2s + umaddl x8, w24, w11, x3 + umlal v21.2d, v0.2s, v17.2s + ldr x30, [tmpa+32] + mul v14.2s, v5.2s, v31.2s + add x2, x2, x10 + shl v5.2s, v28.2s, #1 + shl v27.2s, v4.2s, #1 + umlal v6.2d, v0.2s, v0.2s + umaddl x11, w24, w9, x15 + umlal v6.2d, v12.2s, v3.2s + add x4, x30, x10 + umlal v11.2d, v14.2s, v5.2s + add x3, x22, x10 + umlal v11.2d, v2.2s, v17.2s + add x6, x0, x11, lsr #26 + umlal v11.2d, v12.2s, v27.2s + add x14, x14, x10 + umlal v6.2d, v14.2s, v27.2s + add x8, x8, x6, lsr #25 + umlal v6.2d, v2.2s, v13.2s + movk x10, #0xffb4 + umlal v25.2d, v16.2s, v4.2s + add x29, x16, x8, lsr #26 + umull v27.2d, v1.2s, v3.2s + and x11, x11, #0x3ffffff + umlal v9.2d, v18.2s, v3.2s + add x19, x13, x10 + umlal v9.2d, v0.2s, v13.2s + and x5, x8, #0x3ffffff + umlal v9.2d, v28.2s, v4.2s + bfi x11, x6, #32, #25 // ubignum_of_preglist 0 // INTERMEDIATE bb + umlal v9.2d, v16.2s, v16.2s + umaddl x30, w24, w28, x27 + umlal v9.2d, v14.2s, v7.2s + sub x13, x19, x11 + umull v10.2d, v1.2s, v18.2s + add x7, x23, x29, lsr #25 + umlal v21.2d, v28.2s, v15.2s + lsr x16, x13, #32 // ubignum_of_wreglist 1 + ubignum_of_wreglist 0 // INTERMEDIATE e + umlal v21.2d, v2.2s, v22.2s + add x0, x26, x7, lsr #26 + usra v25.2d, v9.2d, #26 + and x20, x7, #0x3ffffff + umull v22.2d, v1.2s, v1.2s + add x8, x25, x0, lsr #25 + umull v7.2d, v1.2s, v28.2s + and x1, x29, #0x1ffffff // ubignum_of_xreglist 3 // INTERMEDIATE bbalt + bic v18.16b, v25.16b, v23.16b + and x19, x8, #0x3ffffff + and v16.16b, v9.16b, v30.16b + and x7, x12, #0x3ffffff + usra v22.2d, v18.2d, #25 + add x10, x30, x8, lsr #26 + umlal v7.2d, v19.2s, v24.2s + bfi x5, x29, #32, #25 // ubignum_of_preglist 1 // INTERMEDIATE bb + and v9.16b, v25.16b, v23.16b + add x27, x7, x10, lsr #25 + usra v22.2d, v18.2d, #24 + mov x21, #60833 + lsl x21, x21, #1 + add x15, x17, x27, lsr #26 + shl v25.2s, v3.2s, #1 + umlal v7.2d, v14.2s, v17.2s + and x29, x27, #0x3ffffff + usra v22.2d, v18.2d, #21 + bfi x29, x15, #32, #26 // ubignum_of_preglist 4 // INTERMEDIATE bb // ***SOURCE*** ubignum_of_xreglist 9 // INTERMEDIATE bbalt + umlal v10.2d, v14.2s, v24.2s + and x17, x6, #0x1ffffff // ubignum_of_xreglist 1 // INTERMEDIATE bbalt + umlal v10.2d, v2.2s, v28.2s + sub x6, x3, x5 + umlal v10.2d, v12.2s, v17.2s + umaddl x25, w16, w21, x17 + umlal v10.2d, v29.2s, v4.2s + mov w12, w5 // ubignum_of_xreglist 2 // INTERMEDIATE bbalt + umlal v22.2d, v20.2s, v4.2s + lsr x26, x6, #32 // ubignum_of_wreglist 3 + ubignum_of_wreglist 2 // INTERMEDIATE e + umlal v22.2d, v14.2s, v8.2s + and x24, x0, #0x1ffffff // ubignum_of_xreglist 5 // INTERMEDIATE bbalt + umlal v22.2d, v2.2s, v24.2s + stp x11, x5, [tmpb+0] + umlal v22.2d, v12.2s, v5.2s + bfi x20, x0, #32, #25 // ubignum_of_preglist 2 // INTERMEDIATE bb + umlal v22.2d, v29.2s, v17.2s + umaddl x12, w6, w21, x12 + umull v18.2d, v1.2s, v4.2s + bfi x19, x10, #32, #25 // ubignum_of_preglist 3 // INTERMEDIATE bb + umlal v7.2d, v2.2s, v4.2s + sub x7, x14, x20 + umlal v27.2d, v19.2s, v13.2s + mov w8, w20 // ubignum_of_xreglist 4 // INTERMEDIATE bbalt + usra v10.2d, v22.2d, #26 + lsr x14, x7, #32 // ubignum_of_wreglist 5 + ubignum_of_wreglist 4 // INTERMEDIATE e + umlal v18.2d, v19.2s, v17.2s + and x28, x10, #0x1ffffff // ubignum_of_xreglist 7 // INTERMEDIATE bbalt + umlal v7.2d, v12.2s, v13.2s + sub x5, x2, x19 + usra v11.2d, v10.2d, #25 + mov w2, w19 // ubignum_of_xreglist 6 // INTERMEDIATE bbalt + umlal v27.2d, v0.2s, v4.2s + umlal v21.2d, v14.2s, v25.2s + sub x23, x4, x29 + usra v7.2d, v11.2d, #26 + mov w0, w29 // ubignum_of_xreglist 8 // INTERMEDIATE bbalt + umlal v18.2d, v0.2s, v28.2s + lsr x22, x23, #32 // ubignum_of_wreglist 9 + ubignum_of_wreglist 8 // INTERMEDIATE e + umlal v27.2d, v15.2s, v17.2s + str x29, [tmpb+32] + usra v6.2d, v7.2d, #25 + mov w17, w11 // ubignum_of_xreglist 0 // INTERMEDIATE bbalt + and v0.16b, v22.16b, v30.16b // ubignum_of_hreglist 0 + ubignum_of_lreglist 0 // INTERMEDIATE H|L = x5|t3 + umaddl x27, w26, w21, x1 + umlal v18.2d, v14.2s, v13.2s + umaddl x30, w23, w21, x0 + umlal v18.2d, v2.2s, v3.2s + lsr x10, x5, #32 // ubignum_of_wreglist 7 + ubignum_of_wreglist 6 // INTERMEDIATE e + and v4.16b, v6.16b, v30.16b // ubignum_of_hreglist 4 + ubignum_of_lreglist 4 // INTERMEDIATE H|L = x5|t3 + and v1.16b, v10.16b, v23.16b // ubignum_of_hreglist 1 + ubignum_of_lreglist 1 // INTERMEDIATE H|L = x5|t3 + umaddl x4, w14, w21, x24 + ldr x0, [tmpa+0] + mov v0.s[1], w0 + lsr x0, x0, #32 + mov v1.s[1], w0 + umaddl x9, w7, w21, x8 + usra v18.2d, v6.2d, #26 + umaddl x24, w10, w21, x28 + and v3.16b, v7.16b, v23.16b // ubignum_of_hreglist 3 + ubignum_of_lreglist 3 // INTERMEDIATE H|L = x5|t3 + umaddl x8, w22, w21, x15 + umlal v27.2d, v14.2s, v26.2s + umaddl x15, w13, w21, x17 + usra v21.2d, v18.2d, #25 + stp x20, x19, [tmpb+16] + and v2.16b, v11.16b, v30.16b // ubignum_of_hreglist 2 + ubignum_of_lreglist 2 // INTERMEDIATE H|L = x5|t3 + lsr x29, x8, #25 + ldr x3, [tmpb+0] + mov v10.s[1], w3 + lsr x3, x3, #32 + mov v11.s[1], w3 + add x17, x15, x29 + usra v27.2d, v21.2d, #26 + add x28, x17, x29, lsl #1 + and v6.16b, v21.16b, v30.16b // ubignum_of_hreglist 6 + ubignum_of_lreglist 6 // INTERMEDIATE H|L = x5|t3 + and x20, x8, #0x1ffffff + and v5.16b, v18.16b, v23.16b // ubignum_of_hreglist 5 + ubignum_of_lreglist 5 // INTERMEDIATE H|L = x5|t3 + add x17, x28, x29, lsl #4 + and v7.16b, v27.16b, v23.16b // ubignum_of_hreglist 7 + ubignum_of_lreglist 7 // INTERMEDIATE H|L = x5|t3 + ldr x3, [tmpb+8] + mov v22.s[1], w3 + lsr x3, x3, #32 + mov v23.s[1], w3 + add x29, x25, x17, lsr #26 + ldr x15, [pointx+0] + mov v10.s[0], w15 + lsr x15, x15, #32 + mov v11.s[0], w15 + and x11, x17, #0x3ffffff // ubignum_of_xreglist 0 // INTERMEDIATE bce + usra v16.2d, v27.2d, #25 + add x8, x12, x29, lsr #25 + ldr x3, [tmpb+16] + mov v14.s[1], w3 + lsr x3, x3, #32 + mov v15.s[1], w3 + and x12, x29, #0x1ffffff // ubignum_of_xreglist 1 // INTERMEDIATE bce + ldr x15, [pointx+8] + mov v22.s[0], w15 + lsr x15, x15, #32 + mov v23.s[0], w15 + add x28, x27, x8, lsr #26 + and v8.16b, v16.16b, v30.16b // ubignum_of_hreglist 8 + ubignum_of_lreglist 8 // INTERMEDIATE H|L = x5|t3 + umull x1, w12, w10 + ldr x3, [tmpb+24] + mov v17.s[1], w3 + lsr x3, x3, #32 + mov v18.s[1], w3 + add x25, x9, x28, lsr #25 + ldr x15, [pointx+16] + mov v14.s[0], w15 + lsr x15, x15, #32 + mov v15.s[0], w15 + umaddl x19, w5, w21, x2 + usra v9.2d, v16.2d, #26 // ubignum_of_hreglist 8 + ubignum_of_lreglist 8 // INTERMEDIATE H|L = x5|t3 + add x2, x4, x25, lsr #26 + ldr x3, [tmpb+32] + mov v24.s[1], w3 + lsr x3, x3, #32 + mov v25.s[1], w3 + umull x3, w12, w23 + ldr x15, [pointx+24] + mov v17.s[0], w15 + lsr x15, x15, #32 + mov v18.s[0], w15 + add x29, x19, x2, lsr #25 + umull v26.2d, v0.2s, v23.2s + and x21, x28, #0x1ffffff // ubignum_of_xreglist 3 // INTERMEDIATE bce + ldr x0, [tmpa+8] + mov v2.s[1], w0 + lsr x0, x0, #32 + mov v3.s[1], w0 + umaddl x27, w21, w5, x3 + ldr x15, [pointx+32] + mov v24.s[0], w15 + lsr x15, x15, #32 + mov v25.s[0], w15 + add x17, x24, x29, lsr #26 + umull v29.2d, v1.2s, v18.2s + and x15, x8, #0x3ffffff // ubignum_of_xreglist 2 // INTERMEDIATE bce + umull v20.2d, v0.2s, v15.2s + add x19, x30, x17, lsr #25 + and x3, x17, #0x1ffffff // ubignum_of_xreglist 7 // INTERMEDIATE bce + mul v12.2s, v25.2s, v31.2s + ldr x0, [tmpa+16] + mov v4.s[1], w0 + lsr x0, x0, #32 + mov v5.s[1], w0 + add x4, x20, x19, lsr #26 // ubignum_of_xreglist 9 // INTERMEDIATE bce + umlal v26.2d, v2.2s, v11.2s + add w28, w3, w3, lsl #1; + umlal v20.2d, v2.2s, v23.2s + add w28, w28, w3, lsl #4 + umull x8, w12, w5 + ldr x0, [tmpa+24] + mov v6.s[1], w0 + lsr x0, x0, #32 + mov v7.s[1], w0 + and x30, x25, #0x3ffffff // ubignum_of_xreglist 4 // INTERMEDIATE bce + mul v16.2s, v18.2s, v31.2s + add w17, w4, w4, lsl #1; + umull v21.2d, v1.2s, v15.2s + add w17, w17, w4, lsl #4 + umaddl x25, w21, w7, x8 + umlal v20.2d, v4.2s, v11.2s + add w8, w21, w21, lsl #1; + ldr x0, [tmpa+32] + add w8, w8, w21, lsl #4 + mov v8.s[1], w0 + lsr x0, x0, #32 + mov v9.s[1], w0 + and x2, x2, #0x1ffffff // ubignum_of_xreglist 5 // INTERMEDIATE bce + umlal v29.2d, v3.2s, v15.2s + umaddl x24, w2, w6, x25 + umull v13.2d, v0.2s, v25.2s + umaddl x25, w2, w7, x27 + umaddl x0, w3, w6, x25 + mul v19.2s, v15.2s, v31.2s + umull v27.2d, v0.2s, v18.2s + umaddl x20, w3, w13, x24 + umlal v20.2d, v6.2s, v12.2s + umaddl x24, w21, w14, x1 + umlal v13.2d, v2.2s, v18.2s + umaddl x9, w4, w13, x0 + umull v25.2d, v0.2s, v11.2s + umaddl x20, w17, w23, x20 + umlal v27.2d, v2.2s, v15.2s + umaddl x0, w2, w26, x24 + umull v28.2d, v1.2s, v11.2s + umull x24, w17, w5 + umlal v29.2d, v5.2s, v23.2s + umaddl x9, w11, w22, x9 + umlal v13.2d, v4.2s, v15.2s + umaddl x27, w3, w16, x0 + umlal v27.2d, v4.2s, v23.2s + umull x0, w17, w14 + umlal v27.2d, v6.2s, v11.2s + umull x4, w12, w14 + umlal v27.2d, v8.2s, v12.2s + umaddl x25, w11, w10, x20 + umlal v27.2d, v1.2s, v17.2s + umaddl x0, w28, w10, x0 + umlal v13.2d, v6.2s, v23.2s + umull x3, w17, w6 + umlal v13.2d, v8.2s, v11.2s + umaddl x1, w21, w26, x4 + umlal v20.2d, v8.2s, v16.2s + umaddl x4, w2, w13, x24 + umlal v28.2d, v3.2s, v12.2s + umaddl x20, w28, w7, x3 + umlal v29.2d, v7.2s, v11.2s + and x3, x19, #0x3ffffff // ubignum_of_xreglist 9 // INTERMEDIATE bce + umlal v29.2d, v9.2s, v12.2s + umaddl x19, w17, w22, x27 + add w27, w2, w2, lsl #1; + mul v18.2s, v24.2s, v31.2s + add w27, w27, w2, lsl #4 + umlal v21.2d, v3.2s, v23.2s + umull x24, w17, w7 + umlal v13.2d, v1.2s, v24.2s + add x19, x19, x19 + shl v29.2d, v29.2d, #1 + umaddl x1, w2, w16, x1 + umull v15.2d, v1.2s, v23.2s + umaddl x0, w27, w22, x0 + umlal v29.2d, v0.2s, v24.2s + umaddl x2, w28, w5, x24 + mul v24.2s, v23.2s, v31.2s + umaddl x4, w28, w23, x4 + umlal v21.2d, v5.2s, v11.2s + umaddl x24, w27, w5, x20 + umlal v20.2d, v1.2s, v14.2s + umaddl x20, w11, w23, x19 + umlal v26.2d, v4.2s, v12.2s + umaddl x19, w27, w23, x2 + umlal v26.2d, v6.2s, v16.2s + umaddl x2, w21, w6, x4 + umlal v29.2d, v2.2s, v17.2s + umaddl x24, w8, w23, x24 + umlal v15.2d, v3.2s, v11.2s + umaddl x0, w21, w16, x0 + umaddl x4, w21, w13, x19 + mul v23.2s, v11.2s, v31.2s + umlal v20.2d, v3.2s, v22.2s + umaddl x2, w12, w7, x2 + umlal v20.2d, v5.2s, v10.2s + umaddl x19, w12, w26, x0 + umlal v29.2d, v4.2s, v14.2s + umaddl x0, w12, w13, x24 + umlal v26.2d, v8.2s, v19.2s + umaddl x20, w15, w5, x20 + umlal v26.2d, v1.2s, v22.2s + umaddl x21, w15, w10, x9 + umlal v26.2d, v3.2s, v10.2s + and x9, x29, #0x3ffffff // ubignum_of_xreglist 6 // INTERMEDIATE bce + umlal v29.2d, v6.2s, v22.2s + umaddl x20, w30, w7, x20 + umaddl x1, w28, w22, x1 + add x24, x19, x19 + umull v11.2d, v1.2s, v12.2s + add w19, w3, w3, lsl #1; + umlal v26.2d, v5.2s, v18.2s + add w19, w19, w3, lsl #4 + umaddl x20, w9, w6, x20 + umlal v29.2d, v8.2s, v10.2s + add w29, w9, w9, lsl #1; + umlal v13.2d, v3.2s, v17.2s + add w29, w29, w9, lsl #4 + umaddl x2, w19, w10, x2 + umlal v11.2d, v3.2s, v16.2s + umaddl x21, w30, w14, x21 + umlal v11.2d, v5.2s, v19.2s + umaddl x20, w3, w13, x20 + umlal v11.2d, v7.2s, v24.2s + umaddl x2, w29, w22, x2 + umlal v11.2d, v9.2s, v23.2s + umaddl x21, w9, w26, x21 + ushr v23.2d, v30.2d, #1 + umaddl x1, w17, w10, x1 + umlal v13.2d, v5.2s, v14.2s + umaddl x24, w19, w5, x24 + umlal v27.2d, v3.2s, v14.2s + umaddl x21, w3, w16, x21 + shl v11.2d, v11.2d, #1 + add w3, w30, w30, lsl #1; + umlal v28.2d, v5.2s, v16.2s + add w3, w3, w30, lsl #4 + umaddl x24, w29, w23, x24 + umlal v28.2d, v7.2s, v19.2s + add x1, x1, x1 + umlal v28.2d, v9.2s, v24.2s + umaddl x1, w11, w5, x1 + umlal v15.2d, v5.2s, v12.2s + umaddl x24, w30, w13, x24 + umlal v15.2d, v7.2s, v16.2s + umaddl x25, w15, w14, x25 + umlal v15.2d, v9.2s, v19.2s + umaddl x1, w15, w7, x1 + shl v28.2d, v28.2d, #1 + umaddl x24, w15, w6, x24 + umlal v21.2d, v7.2s, v12.2s + umaddl x2, w30, w16, x2 + umlal v21.2d, v9.2s, v16.2s + umaddl x25, w30, w26, x25 + shl v15.2d, v15.2d, #1 + umaddl x30, w30, w6, x1 + umlal v28.2d, v0.2s, v22.2s + umaddl x1, w15, w26, x2 + umlal v28.2d, v2.2s, v10.2s + umaddl x2, w9, w16, x25 + shl v21.2d, v21.2d, #1 + umaddl x24, w11, w7, x24 + umlal v15.2d, v0.2s, v14.2s + umaddl x1, w11, w14, x1 + umlal v21.2d, v0.2s, v17.2s + umaddl x25, w9, w13, x30 + umlal v28.2d, v4.2s, v18.2s + umaddl x0, w19, w26, x0 + umlal v25.2d, v2.2s, v12.2s + add x1, x1, x24, lsr #26 + umlal v25.2d, v4.2s, v16.2s + umaddl x30, w19, w22, x2 + umlal v21.2d, v2.2s, v14.2s + umaddl x4, w12, w6, x4 + mul v14.2s, v14.2s, v31.2s + umaddl x25, w19, w23, x25 + and x2, x1, #0x1ffffff + mul v16.2s, v17.2s, v31.2s + umlal v25.2d, v6.2s, v19.2s + umaddl x9, w19, w14, x4 + umlal v13.2d, v7.2s, v22.2s + add x25, x25, x1, lsr #25 + umlal v21.2d, v4.2s, v22.2s + umaddl x0, w29, w14, x0 + umlal v26.2d, v7.2s, v16.2s + add x30, x30, x25, lsr #26 + umlal v26.2d, v9.2s, v14.2s + add w1, w15, w15, lsl #1; + umlal v28.2d, v6.2s, v16.2s + add w1, w1, w15, lsl #4 + add x4, x20, x30, lsr #25 + umlal v28.2d, v8.2s, v14.2s + and x25, x25, #0x3ffffff + umlal v15.2d, v2.2s, v22.2s + add x21, x21, x4, lsr #26 + umlal v11.2d, v0.2s, v10.2s + bfi x25, x30, #32, #25 // ubignum_of_preglist 3 // INTERMEDIATE z4 + umlal v11.2d, v2.2s, v18.2s + bic x30, x21, #0x3ffffff + usra v26.2d, v28.2d, #26 + lsr x20, x30, #26 + umlal v15.2d, v4.2s, v10.2s + add x20, x20, x30, lsr #25 + umlal v15.2d, v6.2s, v18.2s + umaddl x9, w29, w10, x9 + umlal v15.2d, v8.2s, v16.2s + add x30, x20, x30, lsr #22 + umlal v27.2d, v5.2s, v22.2s + umull x20, w17, w26 + umlal v20.2d, v7.2s, v18.2s + umaddl x30, w17, w16, x30 + umlal v20.2d, v9.2s, v16.2s + umaddl x17, w3, w10, x0 + usra v15.2d, v26.2d, #25 + umaddl x0, w28, w14, x20 + umlal v27.2d, v7.2s, v10.2s + umaddl x20, w28, w26, x30 + umlal v27.2d, v9.2s, v18.2s + add w28, w12, w12, lsl #1; + usra v20.2d, v15.2d, #26 + add w28, w28, w12, lsl #4 + umaddl x30, w27, w10, x0 + and v17.16b, v15.16b, v30.16b // ubignum_of_hreglist 4 + ubignum_of_lreglist 4 // INTERMEDIATE H|L = x4|z5 + umaddl x27, w27, w14, x20 + umaddl x0, w8, w10, x27 + mul v12.2s, v22.2s, v31.2s + and v15.16b, v20.16b, v23.16b // ubignum_of_hreglist 5 + ubignum_of_lreglist 5 // INTERMEDIATE H|L = x4|z5 + umaddl x14, w3, w22, x9 + umlal v21.2d, v6.2s, v10.2s + umaddl x27, w8, w22, x30 + trn1 v15.4s, v17.4s, v15.4s // FINAL z3 + umaddl x10, w28, w22, x0 + umlal v11.2d, v4.2s, v16.2s + umaddl x30, w15, w16, x14 + and v26.16b, v26.16b, v23.16b + umaddl x28, w12, w16, x27 + umlal v21.2d, v8.2s, v18.2s + add x10, x10, x10 + umlal v25.2d, v8.2s, v24.2s + umaddl x20, w19, w6, x10 + umlal v25.2d, v1.2s, v10.2s + add x28, x28, x28 + umlal v25.2d, v3.2s, v18.2s + umaddl x28, w19, w7, x28 + usra v21.2d, v20.2d, #25 + umaddl x0, w29, w7, x20 + umlal v11.2d, v6.2s, v14.2s + umaddl x10, w11, w26, x30 + umlal v13.2d, v9.2s, v10.2s + umaddl x19, w29, w5, x28 + usra v27.2d, v21.2d, #26 + umaddl x0, w3, w5, x0 + umlal v25.2d, v5.2s, v16.2s + umaddl x20, w1, w22, x17 + and v20.16b, v28.16b, v30.16b + umaddl x29, w3, w23, x19 + usra v29.2d, v27.2d, #25 + umaddl x3, w1, w23, x0 + and v27.16b, v27.16b, v23.16b // ubignum_of_hreglist 7 + ubignum_of_lreglist 7 // INTERMEDIATE H|L = x4|z5 + umlal v11.2d, v8.2s, v12.2s + umaddl x12, w15, w13, x29 + usra v13.2d, v29.2d, #26 + umaddl x7, w11, w13, x3 + trn1 v6.4s, v6.4s, v7.4s + umaddl x17, w11, w16, x20 + umlal v25.2d, v7.2s, v14.2s + and x23, x4, #0x3ffffff + bic v19.16b, v13.16b, v23.16b + umaddl x19, w11, w6, x12 + and v28.16b, v13.16b, v23.16b // ubignum_of_hreglist 9 + ubignum_of_lreglist 9 // INTERMEDIATE H|L = x4|z5 + add x3, x17, x7, lsr #26 + usra v11.2d, v19.2d, #25 + trn1 v2.4s, v2.4s, v3.4s + add x17, x19, x3, lsr #25 + and v13.16b, v21.16b, v30.16b // ubignum_of_hreglist 6 + ubignum_of_lreglist 6 // INTERMEDIATE H|L = x4|z5 + and x5, x7, #0x3ffffff + usra v11.2d, v19.2d, #24 + add x7, x10, x17, lsr #26 + trn1 v0.4s, v0.4s, v1.4s + and x19, x24, #0x3ffffff + and v21.16b, v29.16b, v30.16b // ubignum_of_hreglist 8 + ubignum_of_lreglist 8 // INTERMEDIATE H|L = x4|z5 + add x29, x19, x7, lsr #25 + usra v11.2d, v19.2d, #21 + bfi x5, x3, #32, #25 // ubignum_of_preglist 0 // INTERMEDIATE z4 + trn1 v17.4s, v13.4s, v27.4s // FINAL z3 + add x19, x2, x29, lsr #26 + trn1 v19.4s, v21.4s, v28.4s // FINAL z3 + and x3, x29, #0x3ffffff + mov v16.d[0], v6.d[1] // FINAL x3 + mov v6.d[0], v17.d[1] // FINAL x2 + trn1 v8.4s, v8.4s, v9.4s + bfi x3, x19, #32, #26 // ubignum_of_preglist 2 // INTERMEDIATE z4 + and v21.16b, v11.16b, v30.16b // ubignum_of_hreglist 0 + ubignum_of_lreglist 0 // INTERMEDIATE H|L = x4|z5 + bfi x23, x21, #32, #26 // ubignum_of_preglist 4 // INTERMEDIATE z4 + mov v18.d[0], v8.d[1] // FINAL x3 + mov v8.d[0], v19.d[1] // FINAL x2 + umlal v25.2d, v9.2s, v12.2s + mov v9.d[0], x23 // FINAL z2 + mov v7.d[0], x25 // FINAL z2 + ldr d29, [mask1] + mov v12.d[0], v2.d[1] // FINAL x3 + trn1 v4.4s, v4.4s, v5.4s + and x17, x17, #0x3ffffff + usra v25.2d, v11.2d, #26 + mov v10.d[0], v0.d[1] // FINAL x3 + mov v14.d[0], v4.d[1] // FINAL x3 + mov v4.d[0], v15.d[1] // FINAL x2 + usra v20.2d, v25.2d, #25 + and v27.16b, v25.16b, v23.16b // ubignum_of_hreglist 1 + ubignum_of_lreglist 1 // INTERMEDIATE H|L = x4|z5 + bfi x17, x7, #32, #25 // ubignum_of_preglist 1 // INTERMEDIATE z4 + mov v5.d[0], x3 + mov v1.d[0], x5 // FINAL z2 + usra v26.2d, v20.2d, #26 // ubignum_of_hreglist 3 + ubignum_of_lreglist 3 // INTERMEDIATE H|L = x4|z5 + and v28.16b, v20.16b, v30.16b // ubignum_of_hreglist 2 + ubignum_of_lreglist 2 // INTERMEDIATE H|L = x4|z5 + trn1 v11.4s, v21.4s, v27.4s // FINAL z3 + trn1 v13.4s, v28.4s, v26.4s // FINAL z3 + mov v0.d[0], v11.d[1] // FINAL x2 + mov v3.d[0], x17 // FINAL z2 + mov v2.d[0], v13.d[1] // FINAL x2 + ldr d28, [mask2] + + ldr x0, [i] + subs x0, x0, #1 + str x0, [i] + bcs Lcurve25519_x25519_scalarloop + +// Repack X2 into the saturated representation as 256-bit value xn. +// This does not fully normalize mod 2^255-19 but stays within 256 bits. + + mov w0, v0.s[0] + mov w1, v0.s[1] + mov w2, v2.s[0] + mov w3, v2.s[1] + mov w4, v4.s[0] + mov w5, v4.s[1] + mov w6, v6.s[0] + mov w7, v6.s[1] + mov w8, v8.s[0] + mov w9, v8.s[1] + + add x0, x0, x1, lsl #26 + add x1, x2, x3, lsl #26 + add x2, x4, x5, lsl #26 + add x3, x6, x7, lsl #26 + add x4, x8, x9, lsl #26 + + adds x0, x0, x1, lsl #51 + lsr x6, x1, #13 + lsl x7, x2, #38 + adcs x1, x6, x7 + lsr x8, x2, #26 + lsl x9, x3, #25 + adcs x2, x8, x9 + lsr x10, x3, #39 + lsl x11, x4, #12 + adc x3, x10, x11 + stp x0, x1, [xn] + stp x2, x3, [xn+16] + +// Repack Z2 into the saturated representation as 256-bit value zn. +// This does not fully normalize mod 2^255-19. However since Z2, +// unlike X2, was not repacked (within the last multiplication) in +// right-to-left order, its top digit can be any 26-bit value, on +// the face of it. To make sure we don't overflow 256 bits here +// we remove b = 25th bit of the 9th digit (now scaled by 2^230 +// giving bit 25 a final weighting of 2^255) and add 19 * b to +// to the bottom of the sum here to compensate mod 2^255-19. + + mov w0, v1.s[0] + mov w1, v1.s[1] + mov w2, v3.s[0] + mov w3, v3.s[1] + mov w4, v5.s[0] + mov w5, v5.s[1] + mov w6, v7.s[0] + mov w7, v7.s[1] + mov w8, v9.s[0] + mov w9, v9.s[1] + + mov w10, #19 + add x0, x0, x1, lsl #26 + tst x9, #0x2000000 + add x1, x2, x3, lsl #26 + csel x10, x10, xzr, ne + add x2, x4, x5, lsl #26 + and x9, x9, #0x1FFFFFF + add x3, x6, x7, lsl #26 + add x0, x0, x10 + add x4, x8, x9, lsl #26 + + adds x0, x0, x1, lsl #51 + lsr x6, x1, #13 + lsl x7, x2, #38 + adcs x1, x6, x7 + lsr x8, x2, #26 + lsl x9, x3, #25 + adcs x2, x8, x9 + lsr x10, x3, #39 + lsl x11, x4, #12 + adc x3, x10, x11 + stp x0, x1, [zn] + stp x2, x3, [zn+16] + +// Because the lowest bit (indeed, the three lowest bits) of the scalar +// were forced to zero, we know that the projective result of the scalar +// multiplication was in (X2,Z2) and is now (xn,zn) in saturated form. +// Prepare to call the modular inverse function to get zn' = 1/zn. + + add x0, zn + add x1, zn + +// Inline copy of bignum_inv_p25519, identical except for stripping out +// the prologue and epilogue saving and restoring registers and making +// and reclaiming room on the stack. For more details and explanations see +// "arm/curve25519/bignum_inv_p25519.S". Note that the stack it uses for +// its own temporaries is 128 bytes, so it has no effect on variables +// that are needed in the rest of our computation here: res, xn, and zn. + + mov x20, x0 + mov x10, #0xffffffffffffffed + mov x11, #0xffffffffffffffff + stp x10, x11, [sp] + mov x12, #0x7fffffffffffffff + stp x11, x12, [sp, #16] + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + mov x7, #0x13 + lsr x6, x5, #63 + madd x6, x7, x6, x7 + adds x2, x2, x6 + adcs x3, x3, xzr + adcs x4, x4, xzr + orr x5, x5, #0x8000000000000000 + adcs x5, x5, xzr + csel x6, x7, xzr, cc + subs x2, x2, x6 + sbcs x3, x3, xzr + sbcs x4, x4, xzr + sbc x5, x5, xzr + and x5, x5, #0x7fffffffffffffff + stp x2, x3, [sp, #32] + stp x4, x5, [sp, #48] + stp xzr, xzr, [sp, #64] + stp xzr, xzr, [sp, #80] + mov x10, #0x2099 + movk x10, #0x7502, lsl #16 + movk x10, #0x9e23, lsl #32 + movk x10, #0xa0f9, lsl #48 + mov x11, #0x2595 + movk x11, #0x1d13, lsl #16 + movk x11, #0x8f3f, lsl #32 + movk x11, #0xa8c6, lsl #48 + mov x12, #0x5242 + movk x12, #0x5ac, lsl #16 + movk x12, #0x8938, lsl #32 + movk x12, #0x6c6c, lsl #48 + mov x13, #0x615 + movk x13, #0x4177, lsl #16 + movk x13, #0x8b2, lsl #32 + movk x13, #0x2765, lsl #48 + stp x10, x11, [sp, #96] + stp x12, x13, [sp, #112] + mov x21, #0xa + mov x22, #0x1 + b Lcurve25519_x25519_invmidloop +Lcurve25519_x25519_invloop: + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + and x0, x12, x16 + and x1, x13, x17 + add x19, x0, x1 + ldr x7, [sp] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #32] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x7, [sp, #8] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #40] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + adc x6, x6, x1 + extr x4, x2, x4, #59 + str x4, [sp] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + adc x4, x4, x1 + extr x5, x3, x5, #59 + str x5, [sp, #32] + ldr x7, [sp, #16] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #48] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + adc x5, x5, x1 + extr x2, x6, x2, #59 + str x2, [sp, #8] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + adc x2, x2, x1 + extr x3, x4, x3, #59 + str x3, [sp, #40] + ldr x7, [sp, #24] + eor x1, x7, x14 + asr x3, x1, #63 + and x3, x3, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #56] + eor x1, x8, x15 + asr x0, x1, #63 + and x0, x0, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x5, x6, #59 + str x6, [sp, #16] + extr x5, x3, x5, #59 + str x5, [sp, #24] + eor x1, x7, x16 + asr x5, x1, #63 + and x5, x5, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + asr x0, x1, #63 + and x0, x0, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x4, x2, x4, #59 + str x4, [sp, #48] + extr x2, x5, x2, #59 + str x2, [sp, #56] + ldr x7, [sp, #64] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #96] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #64] + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + str x5, [sp, #96] + adc x3, x3, x1 + ldr x7, [sp, #72] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #104] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #72] + adc x6, x6, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + str x3, [sp, #104] + adc x4, x4, x1 + ldr x7, [sp, #80] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #112] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #80] + adc x5, x5, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + str x4, [sp, #112] + adc x2, x2, x1 + ldr x7, [sp, #88] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #120] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x3, x5, #63 + ldp x0, x1, [sp, #64] + add x6, x6, x3, asr #63 + mov x3, #0x13 + mul x4, x6, x3 + add x5, x5, x6, lsl #63 + smulh x3, x6, x3 + ldr x6, [sp, #80] + adds x0, x0, x4 + adcs x1, x1, x3 + asr x3, x3, #63 + adcs x6, x6, x3 + adc x5, x5, x3 + stp x0, x1, [sp, #64] + stp x6, x5, [sp, #80] + eor x1, x7, x16 + and x5, x16, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + and x0, x17, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x6, x5, x2, #63 + ldp x0, x1, [sp, #96] + add x6, x6, x5, asr #63 + mov x5, #0x13 + mul x4, x6, x5 + add x2, x2, x6, lsl #63 + smulh x5, x6, x5 + ldr x3, [sp, #112] + adds x0, x0, x4 + adcs x1, x1, x5 + asr x5, x5, #63 + adcs x3, x3, x5 + adc x2, x2, x5 + stp x0, x1, [sp, #96] + stp x3, x2, [sp, #112] +Lcurve25519_x25519_invmidloop: + mov x1, x22 + ldr x2, [sp] + ldr x3, [sp, #32] + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x8, x4, #0x100, lsl #12 + sbfx x8, x8, #21, #21 + mov x11, #0x100000 + add x11, x11, x11, lsl #21 + add x9, x4, x11 + asr x9, x9, #42 + add x10, x5, #0x100, lsl #12 + sbfx x10, x10, #21, #21 + add x11, x5, x11 + asr x11, x11, #42 + mul x6, x8, x2 + mul x7, x9, x3 + mul x2, x10, x2 + mul x3, x11, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #21, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #42 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #21, #21 + add x15, x5, x15 + asr x15, x15, #42 + mul x6, x12, x2 + mul x7, x13, x3 + mul x2, x14, x2 + mul x3, x15, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + mul x2, x12, x8 + mul x3, x12, x9 + mul x6, x14, x8 + mul x7, x14, x9 + madd x8, x13, x10, x2 + madd x9, x13, x11, x3 + madd x16, x15, x10, x6 + madd x17, x15, x11, x7 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #22, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #43 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #22, #21 + add x15, x5, x15 + asr x15, x15, #43 + mneg x2, x12, x8 + mneg x3, x12, x9 + mneg x4, x14, x8 + mneg x5, x14, x9 + msub x10, x13, x16, x2 + msub x11, x13, x17, x3 + msub x12, x15, x16, x4 + msub x13, x15, x17, x5 + mov x22, x1 + subs x21, x21, #0x1 + b.ne Lcurve25519_x25519_invloop + ldr x0, [sp] + ldr x1, [sp, #32] + mul x0, x0, x10 + madd x1, x1, x11, x0 + asr x0, x1, #63 + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + eor x14, x14, x0 + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + eor x15, x15, x0 + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + eor x16, x16, x0 + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + eor x17, x17, x0 + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + ldr x7, [sp, #64] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #96] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #64] + adc x2, x2, x1 + ldr x7, [sp, #72] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #104] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #72] + adc x6, x6, x1 + ldr x7, [sp, #80] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #112] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #80] + adc x5, x5, x1 + ldr x7, [sp, #88] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #120] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x3, x5, #63 + ldp x0, x1, [sp, #64] + tst x3, x3 + cinc x6, x6, pl + mov x3, #0x13 + mul x4, x6, x3 + add x5, x5, x6, lsl #63 + smulh x6, x6, x3 + ldr x2, [sp, #80] + adds x0, x0, x4 + adcs x1, x1, x6 + asr x6, x6, #63 + adcs x2, x2, x6 + adcs x5, x5, x6 + csel x3, x3, xzr, mi + subs x0, x0, x3 + sbcs x1, x1, xzr + sbcs x2, x2, xzr + sbc x5, x5, xzr + and x5, x5, #0x7fffffffffffffff + mov x4, x20 + stp x0, x1, [x4] + stp x2, x5, [x4, #16] + +// Now the result is xn * (1/zn), fully reduced modulo p. +// Note that in the degenerate case zn = 0 (mod p_25519), the +// modular inverse code above will produce 1/zn = 0, giving +// the correct overall X25519 result of zero for the point at +// infinity. The multiplication below is just an inlined +// version of bignum_mul_p25519 except for the detailed +// addressing of inputs and outputs + + ldr x17, [res] + + ldp x3, x4, [xn] + ldp x5, x6, [zn] + umull x7, w3, w5 + lsr x0, x3, #32 + umull x15, w0, w5 + lsr x16, x5, #32 + umull x8, w16, w0 + umull x16, w3, w16 + adds x7, x7, x15, lsl #32 + lsr x15, x15, #32 + adc x8, x8, x15 + adds x7, x7, x16, lsl #32 + lsr x16, x16, #32 + adc x8, x8, x16 + mul x9, x4, x6 + umulh x10, x4, x6 + subs x4, x4, x3 + cneg x4, x4, cc + csetm x16, cc + adds x9, x9, x8 + adc x10, x10, xzr + subs x3, x5, x6 + cneg x3, x3, cc + cinv x16, x16, cc + mul x15, x4, x3 + umulh x3, x4, x3 + adds x8, x7, x9 + adcs x9, x9, x10 + adc x10, x10, xzr + cmn x16, #0x1 + eor x15, x15, x16 + adcs x8, x15, x8 + eor x3, x3, x16 + adcs x9, x3, x9 + adc x10, x10, x16 + ldp x3, x4, [xn+16] + ldp x5, x6, [zn+16] + umull x11, w3, w5 + lsr x0, x3, #32 + umull x15, w0, w5 + lsr x16, x5, #32 + umull x12, w16, w0 + umull x16, w3, w16 + adds x11, x11, x15, lsl #32 + lsr x15, x15, #32 + adc x12, x12, x15 + adds x11, x11, x16, lsl #32 + lsr x16, x16, #32 + adc x12, x12, x16 + mul x13, x4, x6 + umulh x14, x4, x6 + subs x4, x4, x3 + cneg x4, x4, cc + csetm x16, cc + adds x13, x13, x12 + adc x14, x14, xzr + subs x3, x5, x6 + cneg x3, x3, cc + cinv x16, x16, cc + mul x15, x4, x3 + umulh x3, x4, x3 + adds x12, x11, x13 + adcs x13, x13, x14 + adc x14, x14, xzr + cmn x16, #0x1 + eor x15, x15, x16 + adcs x12, x15, x12 + eor x3, x3, x16 + adcs x13, x3, x13 + adc x14, x14, x16 + ldp x3, x4, [xn+16] + ldp x15, x16, [xn] + subs x3, x3, x15 + sbcs x4, x4, x16 + csetm x16, cc + ldp x15, x0, [zn] + subs x5, x15, x5 + sbcs x6, x0, x6 + csetm x0, cc + eor x3, x3, x16 + subs x3, x3, x16 + eor x4, x4, x16 + sbc x4, x4, x16 + eor x5, x5, x0 + subs x5, x5, x0 + eor x6, x6, x0 + sbc x6, x6, x0 + eor x16, x0, x16 + adds x11, x11, x9 + adcs x12, x12, x10 + adcs x13, x13, xzr + adc x14, x14, xzr + mul x2, x3, x5 + umulh x0, x3, x5 + mul x15, x4, x6 + umulh x1, x4, x6 + subs x4, x4, x3 + cneg x4, x4, cc + csetm x9, cc + adds x15, x15, x0 + adc x1, x1, xzr + subs x6, x5, x6 + cneg x6, x6, cc + cinv x9, x9, cc + mul x5, x4, x6 + umulh x6, x4, x6 + adds x0, x2, x15 + adcs x15, x15, x1 + adc x1, x1, xzr + cmn x9, #0x1 + eor x5, x5, x9 + adcs x0, x5, x0 + eor x6, x6, x9 + adcs x15, x6, x15 + adc x1, x1, x9 + adds x9, x11, x7 + adcs x10, x12, x8 + adcs x11, x13, x11 + adcs x12, x14, x12 + adcs x13, x13, xzr + adc x14, x14, xzr + cmn x16, #0x1 + eor x2, x2, x16 + adcs x9, x2, x9 + eor x0, x0, x16 + adcs x10, x0, x10 + eor x15, x15, x16 + adcs x11, x15, x11 + eor x1, x1, x16 + adcs x12, x1, x12 + adcs x13, x13, x16 + adc x14, x14, x16 + mov x3, #0x26 + umull x4, w11, w3 + add x4, x4, w7, uxtw + lsr x7, x7, #32 + lsr x11, x11, #32 + umaddl x11, w11, w3, x7 + mov x7, x4 + umull x4, w12, w3 + add x4, x4, w8, uxtw + lsr x8, x8, #32 + lsr x12, x12, #32 + umaddl x12, w12, w3, x8 + mov x8, x4 + umull x4, w13, w3 + add x4, x4, w9, uxtw + lsr x9, x9, #32 + lsr x13, x13, #32 + umaddl x13, w13, w3, x9 + mov x9, x4 + umull x4, w14, w3 + add x4, x4, w10, uxtw + lsr x10, x10, #32 + lsr x14, x14, #32 + umaddl x14, w14, w3, x10 + mov x10, x4 + lsr x0, x14, #31 + mov x5, #0x13 + umaddl x5, w5, w0, x5 + add x7, x7, x5 + adds x7, x7, x11, lsl #32 + extr x3, x12, x11, #32 + adcs x8, x8, x3 + extr x3, x13, x12, #32 + adcs x9, x9, x3 + extr x3, x14, x13, #32 + lsl x5, x0, #63 + eor x10, x10, x5 + adc x10, x10, x3 + mov x3, #0x13 + tst x10, #0x8000000000000000 + csel x3, x3, xzr, pl + subs x7, x7, x3 + sbcs x8, x8, xzr + sbcs x9, x9, xzr + sbc x10, x10, xzr + and x10, x10, #0x7fffffffffffffff + stp x7, x8, [x17] + stp x9, x10, [x17, #16] + +// Restore stack and registers (this will zero the tops of Q8...Q15). + + CFI_STACKLOAD2(d8,d9,NSPACE+0) + CFI_STACKLOAD2(d10,d11,NSPACE+16) + CFI_STACKLOAD2(d12,d13,NSPACE+32) + CFI_STACKLOAD2(d14,d15,NSPACE+48) + CFI_STACKLOAD2(x19,x20,NSPACE+64) + CFI_STACKLOAD2(x21,x22,NSPACE+80) + CFI_STACKLOAD2(x23,x24,NSPACE+96) + CFI_STACKLOAD2(x25,x26,NSPACE+112) + CFI_STACKLOAD2(x27,x28,NSPACE+128) + CFI_STACKLOAD2(x29,x30,NSPACE+144) + CFI_INC_SP((NSPACE+160)) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(curve25519_x25519) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/curve25519_x25519_alt.S b/cbits/s2n/arm/curve25519_x25519_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/curve25519_x25519_alt.S @@ -0,0 +1,1702 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// The x25519 function for curve25519 +// Inputs scalar[4], point[4]; output res[4] +// +// extern void curve25519_x25519_alt +// (uint64_t res[static 4],const uint64_t scalar[static 4], +// const uint64_t point[static 4]); +// +// Given a scalar n and the X coordinate of an input point P = (X,Y) on +// curve25519 (Y can live in any extension field of characteristic 2^255-19), +// this returns the X coordinate of n * P = (X, Y), or 0 when n * P is the +// point at infinity. Both n and X inputs are first slightly modified/mangled +// as specified in the relevant RFC (https://www.rfc-editor.org/rfc/rfc7748); +// in particular the lower three bits of n are set to zero. Does not implement +// the zero-check specified in Section 6.1. +// +// Standard ARM ABI: X0 = res, X1 = scalar, X2 = point +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(curve25519_x25519_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(curve25519_x25519_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(curve25519_x25519_alt) + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Stable homes for the input result argument during the whole body +// and other variables that are only needed prior to the modular inverse. + +#define res x23 +#define i x20 +#define swap x21 + +// Pointers to result x coord to be written + +#define resx res, #0 + +// Pointer-offset pairs for temporaries on stack with some aliasing. + +#define scalar sp, #(0*NUMSIZE) + +#define pointx sp, #(1*NUMSIZE) + +#define zm sp, #(2*NUMSIZE) +#define sm sp, #(2*NUMSIZE) +#define dpro sp, #(2*NUMSIZE) + +#define sn sp, #(3*NUMSIZE) + +#define dm sp, #(4*NUMSIZE) + +#define zn sp, #(5*NUMSIZE) +#define dn sp, #(5*NUMSIZE) +#define e sp, #(5*NUMSIZE) + +#define dmsn sp, #(6*NUMSIZE) +#define p sp, #(6*NUMSIZE) + +#define xm sp, #(7*NUMSIZE) +#define dnsm sp, #(7*NUMSIZE) +#define spro sp, #(7*NUMSIZE) + +#define d sp, #(8*NUMSIZE) + +#define xn sp, #(9*NUMSIZE) +#define s sp, #(9*NUMSIZE) + +// Total size to reserve on the stack + +#define NSPACE 10*NUMSIZE + +// Macro wrapping up the basic field operation bignum_mul_p25519_alt, only +// trivially different from a pure function call to that subroutine. + +#define mul_p25519(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x7, x8, [P2] __LF \ + mul x12, x3, x7 __LF \ + umulh x13, x3, x7 __LF \ + mul x11, x3, x8 __LF \ + umulh x14, x3, x8 __LF \ + adds x13, x13, x11 __LF \ + ldp x9, x10, [P2+16] __LF \ + mul x11, x3, x9 __LF \ + umulh x15, x3, x9 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x3, x10 __LF \ + umulh x16, x3, x10 __LF \ + adcs x15, x15, x11 __LF \ + adc x16, x16, xzr __LF \ + ldp x5, x6, [P1+16] __LF \ + mul x11, x4, x7 __LF \ + adds x13, x13, x11 __LF \ + mul x11, x4, x8 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x4, x9 __LF \ + adcs x15, x15, x11 __LF \ + mul x11, x4, x10 __LF \ + adcs x16, x16, x11 __LF \ + umulh x3, x4, x10 __LF \ + adc x3, x3, xzr __LF \ + umulh x11, x4, x7 __LF \ + adds x14, x14, x11 __LF \ + umulh x11, x4, x8 __LF \ + adcs x15, x15, x11 __LF \ + umulh x11, x4, x9 __LF \ + adcs x16, x16, x11 __LF \ + adc x3, x3, xzr __LF \ + mul x11, x5, x7 __LF \ + adds x14, x14, x11 __LF \ + mul x11, x5, x8 __LF \ + adcs x15, x15, x11 __LF \ + mul x11, x5, x9 __LF \ + adcs x16, x16, x11 __LF \ + mul x11, x5, x10 __LF \ + adcs x3, x3, x11 __LF \ + umulh x4, x5, x10 __LF \ + adc x4, x4, xzr __LF \ + umulh x11, x5, x7 __LF \ + adds x15, x15, x11 __LF \ + umulh x11, x5, x8 __LF \ + adcs x16, x16, x11 __LF \ + umulh x11, x5, x9 __LF \ + adcs x3, x3, x11 __LF \ + adc x4, x4, xzr __LF \ + mul x11, x6, x7 __LF \ + adds x15, x15, x11 __LF \ + mul x11, x6, x8 __LF \ + adcs x16, x16, x11 __LF \ + mul x11, x6, x9 __LF \ + adcs x3, x3, x11 __LF \ + mul x11, x6, x10 __LF \ + adcs x4, x4, x11 __LF \ + umulh x5, x6, x10 __LF \ + adc x5, x5, xzr __LF \ + umulh x11, x6, x7 __LF \ + adds x16, x16, x11 __LF \ + umulh x11, x6, x8 __LF \ + adcs x3, x3, x11 __LF \ + umulh x11, x6, x9 __LF \ + adcs x4, x4, x11 __LF \ + adc x5, x5, xzr __LF \ + mov x7, #0x26 __LF \ + mul x11, x7, x16 __LF \ + umulh x9, x7, x16 __LF \ + adds x12, x12, x11 __LF \ + mul x11, x7, x3 __LF \ + umulh x3, x7, x3 __LF \ + adcs x13, x13, x11 __LF \ + mul x11, x7, x4 __LF \ + umulh x4, x7, x4 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x7, x5 __LF \ + umulh x5, x7, x5 __LF \ + adcs x15, x15, x11 __LF \ + cset x16, cs __LF \ + adds x15, x15, x4 __LF \ + adc x16, x16, x5 __LF \ + cmn x15, x15 __LF \ + orr x15, x15, #0x8000000000000000 __LF \ + adc x8, x16, x16 __LF \ + mov x7, #0x13 __LF \ + madd x11, x7, x8, x7 __LF \ + adds x12, x12, x11 __LF \ + adcs x13, x13, x9 __LF \ + adcs x14, x14, x3 __LF \ + adcs x15, x15, xzr __LF \ + csel x7, x7, xzr, cc __LF \ + subs x12, x12, x7 __LF \ + sbcs x13, x13, xzr __LF \ + sbcs x14, x14, xzr __LF \ + sbc x15, x15, xzr __LF \ + and x15, x15, #0x7fffffffffffffff __LF \ + stp x12, x13, [P0] __LF \ + stp x14, x15, [P0+16] + +// A version of multiplication that only guarantees output < 2 * p_25519. +// This basically skips the +1 and final correction in quotient estimation. + +#define mul_4(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x7, x8, [P2] __LF \ + mul x12, x3, x7 __LF \ + umulh x13, x3, x7 __LF \ + mul x11, x3, x8 __LF \ + umulh x14, x3, x8 __LF \ + adds x13, x13, x11 __LF \ + ldp x9, x10, [P2+16] __LF \ + mul x11, x3, x9 __LF \ + umulh x15, x3, x9 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x3, x10 __LF \ + umulh x16, x3, x10 __LF \ + adcs x15, x15, x11 __LF \ + adc x16, x16, xzr __LF \ + ldp x5, x6, [P1+16] __LF \ + mul x11, x4, x7 __LF \ + adds x13, x13, x11 __LF \ + mul x11, x4, x8 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x4, x9 __LF \ + adcs x15, x15, x11 __LF \ + mul x11, x4, x10 __LF \ + adcs x16, x16, x11 __LF \ + umulh x3, x4, x10 __LF \ + adc x3, x3, xzr __LF \ + umulh x11, x4, x7 __LF \ + adds x14, x14, x11 __LF \ + umulh x11, x4, x8 __LF \ + adcs x15, x15, x11 __LF \ + umulh x11, x4, x9 __LF \ + adcs x16, x16, x11 __LF \ + adc x3, x3, xzr __LF \ + mul x11, x5, x7 __LF \ + adds x14, x14, x11 __LF \ + mul x11, x5, x8 __LF \ + adcs x15, x15, x11 __LF \ + mul x11, x5, x9 __LF \ + adcs x16, x16, x11 __LF \ + mul x11, x5, x10 __LF \ + adcs x3, x3, x11 __LF \ + umulh x4, x5, x10 __LF \ + adc x4, x4, xzr __LF \ + umulh x11, x5, x7 __LF \ + adds x15, x15, x11 __LF \ + umulh x11, x5, x8 __LF \ + adcs x16, x16, x11 __LF \ + umulh x11, x5, x9 __LF \ + adcs x3, x3, x11 __LF \ + adc x4, x4, xzr __LF \ + mul x11, x6, x7 __LF \ + adds x15, x15, x11 __LF \ + mul x11, x6, x8 __LF \ + adcs x16, x16, x11 __LF \ + mul x11, x6, x9 __LF \ + adcs x3, x3, x11 __LF \ + mul x11, x6, x10 __LF \ + adcs x4, x4, x11 __LF \ + umulh x5, x6, x10 __LF \ + adc x5, x5, xzr __LF \ + umulh x11, x6, x7 __LF \ + adds x16, x16, x11 __LF \ + umulh x11, x6, x8 __LF \ + adcs x3, x3, x11 __LF \ + umulh x11, x6, x9 __LF \ + adcs x4, x4, x11 __LF \ + adc x5, x5, xzr __LF \ + mov x7, #0x26 __LF \ + mul x11, x7, x16 __LF \ + umulh x9, x7, x16 __LF \ + adds x12, x12, x11 __LF \ + mul x11, x7, x3 __LF \ + umulh x3, x7, x3 __LF \ + adcs x13, x13, x11 __LF \ + mul x11, x7, x4 __LF \ + umulh x4, x7, x4 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x7, x5 __LF \ + umulh x5, x7, x5 __LF \ + adcs x15, x15, x11 __LF \ + cset x16, cs __LF \ + adds x15, x15, x4 __LF \ + adc x16, x16, x5 __LF \ + cmn x15, x15 __LF \ + bic x15, x15, #0x8000000000000000 __LF \ + adc x8, x16, x16 __LF \ + mov x7, #0x13 __LF \ + mul x11, x7, x8 __LF \ + adds x12, x12, x11 __LF \ + adcs x13, x13, x9 __LF \ + adcs x14, x14, x3 __LF \ + adc x15, x15, xzr __LF \ + stp x12, x13, [P0] __LF \ + stp x14, x15, [P0+16] + +// Squaring just giving a result < 2 * p_25519, which is done by +// basically skipping the +1 in the quotient estimate and the final +// optional correction. + +#define sqr_4(P0,P1) \ + ldp x2, x3, [P1] __LF \ + mul x9, x2, x3 __LF \ + umulh x10, x2, x3 __LF \ + ldp x4, x5, [P1+16] __LF \ + mul x11, x2, x5 __LF \ + umulh x12, x2, x5 __LF \ + mul x7, x2, x4 __LF \ + umulh x6, x2, x4 __LF \ + adds x10, x10, x7 __LF \ + adcs x11, x11, x6 __LF \ + mul x7, x3, x4 __LF \ + umulh x6, x3, x4 __LF \ + adc x6, x6, xzr __LF \ + adds x11, x11, x7 __LF \ + mul x13, x4, x5 __LF \ + umulh x14, x4, x5 __LF \ + adcs x12, x12, x6 __LF \ + mul x7, x3, x5 __LF \ + umulh x6, x3, x5 __LF \ + adc x6, x6, xzr __LF \ + adds x12, x12, x7 __LF \ + adcs x13, x13, x6 __LF \ + adc x14, x14, xzr __LF \ + adds x9, x9, x9 __LF \ + adcs x10, x10, x10 __LF \ + adcs x11, x11, x11 __LF \ + adcs x12, x12, x12 __LF \ + adcs x13, x13, x13 __LF \ + adcs x14, x14, x14 __LF \ + cset x6, cs __LF \ + umulh x7, x2, x2 __LF \ + mul x8, x2, x2 __LF \ + adds x9, x9, x7 __LF \ + mul x7, x3, x3 __LF \ + adcs x10, x10, x7 __LF \ + umulh x7, x3, x3 __LF \ + adcs x11, x11, x7 __LF \ + mul x7, x4, x4 __LF \ + adcs x12, x12, x7 __LF \ + umulh x7, x4, x4 __LF \ + adcs x13, x13, x7 __LF \ + mul x7, x5, x5 __LF \ + adcs x14, x14, x7 __LF \ + umulh x7, x5, x5 __LF \ + adc x6, x6, x7 __LF \ + mov x3, #0x26 __LF \ + mul x7, x3, x12 __LF \ + umulh x4, x3, x12 __LF \ + adds x8, x8, x7 __LF \ + mul x7, x3, x13 __LF \ + umulh x13, x3, x13 __LF \ + adcs x9, x9, x7 __LF \ + mul x7, x3, x14 __LF \ + umulh x14, x3, x14 __LF \ + adcs x10, x10, x7 __LF \ + mul x7, x3, x6 __LF \ + umulh x6, x3, x6 __LF \ + adcs x11, x11, x7 __LF \ + cset x12, cs __LF \ + adds x11, x11, x14 __LF \ + adc x12, x12, x6 __LF \ + cmn x11, x11 __LF \ + bic x11, x11, #0x8000000000000000 __LF \ + adc x2, x12, x12 __LF \ + mov x3, #0x13 __LF \ + mul x7, x3, x2 __LF \ + adds x8, x8, x7 __LF \ + adcs x9, x9, x4 __LF \ + adcs x10, x10, x13 __LF \ + adc x11, x11, xzr __LF \ + stp x8, x9, [P0] __LF \ + stp x10, x11, [P0+16] + +// Modular addition with double modulus 2 * p_25519 = 2^256 - 38. +// This only ensures that the result fits in 4 digits, not that it is reduced +// even w.r.t. double modulus. The result is always correct modulo provided +// the sum of the inputs is < 2^256 + 2^256 - 38, so in particular provided +// at least one of them is reduced double modulo. + +#define add_twice4(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x7, x8, [P2] __LF \ + adds x3, x3, x7 __LF \ + adcs x4, x4, x8 __LF \ + ldp x5, x6, [P1+16] __LF \ + ldp x7, x8, [P2+16] __LF \ + adcs x5, x5, x7 __LF \ + adcs x6, x6, x8 __LF \ + mov x9, #38 __LF \ + csel x9, x9, xzr, cs __LF \ + adds x3, x3, x9 __LF \ + adcs x4, x4, xzr __LF \ + adcs x5, x5, xzr __LF \ + adc x6, x6, xzr __LF \ + stp x3, x4, [P0] __LF \ + stp x5, x6, [P0+16] + +// Modular subtraction with double modulus 2 * p_25519 = 2^256 - 38 + +#define sub_twice4(p0,p1,p2) \ + ldp x5, x6, [p1] __LF \ + ldp x4, x3, [p2] __LF \ + subs x5, x5, x4 __LF \ + sbcs x6, x6, x3 __LF \ + ldp x7, x8, [p1+16] __LF \ + ldp x4, x3, [p2+16] __LF \ + sbcs x7, x7, x4 __LF \ + sbcs x8, x8, x3 __LF \ + mov x4, #38 __LF \ + csel x3, x4, xzr, lo __LF \ + subs x5, x5, x3 __LF \ + sbcs x6, x6, xzr __LF \ + sbcs x7, x7, xzr __LF \ + sbc x8, x8, xzr __LF \ + stp x5, x6, [p0] __LF \ + stp x7, x8, [p0+16] + +// Combined z = c * x + y with reduction only < 2 * p_25519 +// where c is initially in the X1 register. It is assumed +// that 19 * (c * x + y) < 2^60 * 2^256 so we don't need a +// high mul in the final part. + +#define cmadd_4(p0,p2,p3) \ + ldp x7, x8, [p2] __LF \ + ldp x9, x10, [p2+16] __LF \ + mul x3, x1, x7 __LF \ + mul x4, x1, x8 __LF \ + mul x5, x1, x9 __LF \ + mul x6, x1, x10 __LF \ + umulh x7, x1, x7 __LF \ + umulh x8, x1, x8 __LF \ + umulh x9, x1, x9 __LF \ + umulh x10, x1, x10 __LF \ + adds x4, x4, x7 __LF \ + adcs x5, x5, x8 __LF \ + adcs x6, x6, x9 __LF \ + adc x10, x10, xzr __LF \ + ldp x7, x8, [p3] __LF \ + adds x3, x3, x7 __LF \ + adcs x4, x4, x8 __LF \ + ldp x7, x8, [p3+16] __LF \ + adcs x5, x5, x7 __LF \ + adcs x6, x6, x8 __LF \ + adc x10, x10, xzr __LF \ + cmn x6, x6 __LF \ + bic x6, x6, #0x8000000000000000 __LF \ + adc x8, x10, x10 __LF \ + mov x9, #19 __LF \ + mul x7, x8, x9 __LF \ + adds x3, x3, x7 __LF \ + adcs x4, x4, xzr __LF \ + adcs x5, x5, xzr __LF \ + adc x6, x6, xzr __LF \ + stp x3, x4, [p0] __LF \ + stp x5, x6, [p0+16] + +// Multiplex: z := if NZ then x else y + +#define mux_4(p0,p1,p2) \ + ldp x0, x1, [p1] __LF \ + ldp x2, x3, [p2] __LF \ + csel x0, x0, x2, ne __LF \ + csel x1, x1, x3, ne __LF \ + stp x0, x1, [p0] __LF \ + ldp x0, x1, [p1+16] __LF \ + ldp x2, x3, [p2+16] __LF \ + csel x0, x0, x2, ne __LF \ + csel x1, x1, x3, ne __LF \ + stp x0, x1, [p0+16] + +S2N_BN_SYMBOL(curve25519_x25519_alt): + CFI_START + +// Save regs and make room for temporaries + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_DEC_SP(NSPACE) + +// Move the output pointer to a stable place + + mov res, x0 + +// Copy the inputs to the local variables with minimal mangling: +// +// - The scalar is in principle turned into 01xxx...xxx000 but +// in the structure below the special handling of these bits is +// explicit in the main computation; the scalar is just copied. +// +// - The point x coord is reduced mod 2^255 by masking off the +// top bit. In the main loop we only need reduction < 2 * p_25519. + + ldp x10, x11, [x1] + stp x10, x11, [scalar] + ldp x12, x13, [x1, #16] + stp x12, x13, [scalar+16] + + ldp x10, x11, [x2] + stp x10, x11, [pointx] + ldp x12, x13, [x2, #16] + and x13, x13, #0x7fffffffffffffff + stp x12, x13, [pointx+16] + +// Initialize with explicit doubling in order to handle set bit 254. +// Set swap = 1 and (xm,zm) = (x,1) then double as (xn,zn) = 2 * (x,1). +// We use the fact that the point x coordinate is still in registers. +// Since zm = 1 we could do the doubling with an operation count of +// 2 * S + M instead of 2 * S + 2 * M, but it doesn't seem worth +// the slight complication arising from a different linear combination. + + mov swap, #1 + stp x10, x11, [xm] + stp x12, x13, [xm+16] + stp swap, xzr, [zm] + stp xzr, xzr, [zm+16] + + sub_twice4(d,xm,zm) + add_twice4(s,xm,zm) + sqr_4(d,d) + sqr_4(s,s) + sub_twice4(p,s,d) + mov x1, 0xdb42 + orr x1, x1, 0x10000 + cmadd_4(e,p,d) + mul_4(xn,s,d) + mul_4(zn,p,e) + +// The main loop over unmodified bits from i = 253, ..., i = 3 (inclusive). +// This is a classic Montgomery ladder, with the main coordinates only +// reduced mod 2 * p_25519, some intermediate results even more loosely. + + mov i, #253 + +Lcurve25519_x25519_alt_scalarloop: + +// sm = xm + zm; sn = xn + zn; dm = xm - zm; dn = xn - zn + + sub_twice4(dm,xm,zm) + add_twice4(sn,xn,zn) + sub_twice4(dn,xn,zn) + add_twice4(sm,xm,zm) + +// ADDING: dmsn = dm * sn +// DOUBLING: mux d = xt - zt and s = xt + zt for appropriate choice of (xt,zt) + + mul_4(dmsn,sn,dm) + + lsr x0, i, #6 + ldr x2, [sp, x0, lsl #3] // Exploiting scalar = sp exactly + lsr x2, x2, i + and x2, x2, #1 + + cmp swap, x2 + mov swap, x2 + + mux_4(d,dm,dn) + mux_4(s,sm,sn) + +// ADDING: dnsm = sm * dn + + mul_4(dnsm,sm,dn) + +// DOUBLING: d = (xt - zt)^2 + + sqr_4(d,d) + +// ADDING: dpro = (dmsn - dnsm)^2, spro = (dmsn + dnsm)^2 +// DOUBLING: s = (xt + zt)^2 + + sub_twice4(dpro,dmsn,dnsm) + sqr_4(s,s) + add_twice4(spro,dmsn,dnsm) + sqr_4(dpro,dpro) + +// DOUBLING: p = 4 * xt * zt = s - d + + sub_twice4(p,s,d) + +// ADDING: xm' = (dmsn + dnsm)^2 + + sqr_4(xm,spro) + +// DOUBLING: e = 121666 * p + d + + mov x1, 0xdb42 + orr x1, x1, 0x10000 + cmadd_4(e,p,d) + +// DOUBLING: xn' = (xt + zt)^2 * (xt - zt)^2 = s * d + + mul_4(xn,s,d) + +// ADDING: zm' = x * (dmsn - dnsm)^2 + + mul_4(zm,dpro,pointx) + +// DOUBLING: zn' = (4 * xt * zt) * ((xt - zt)^2 + 121666 * (4 * xt * zt)) +// = p * (d + 121666 * p) + + mul_4(zn,p,e) + +// Loop down as far as 3 (inclusive) + + sub i, i, #1 + cmp i, #3 + bcs Lcurve25519_x25519_alt_scalarloop + +// Multiplex directly into (xn,zn) then do three pure doubling steps; +// this accounts for the implicit zeroing of the three lowest bits +// of the scalar. + + cmp swap, xzr + mux_4(xn,xm,xn) + mux_4(zn,zm,zn) + + sub_twice4(d,xn,zn) + add_twice4(s,xn,zn) + sqr_4(d,d) + sqr_4(s,s) + sub_twice4(p,s,d) + mov x1, 0xdb42 + orr x1, x1, 0x10000 + cmadd_4(e,p,d) + mul_4(xn,s,d) + mul_4(zn,p,e) + + sub_twice4(d,xn,zn) + add_twice4(s,xn,zn) + sqr_4(d,d) + sqr_4(s,s) + sub_twice4(p,s,d) + mov x1, 0xdb42 + orr x1, x1, 0x10000 + cmadd_4(e,p,d) + mul_4(xn,s,d) + mul_4(zn,p,e) + + sub_twice4(d,xn,zn) + add_twice4(s,xn,zn) + sqr_4(d,d) + sqr_4(s,s) + sub_twice4(p,s,d) + mov x1, 0xdb42 + orr x1, x1, 0x10000 + cmadd_4(e,p,d) + mul_4(xn,s,d) + mul_4(zn,p,e) + +// The projective result of the scalar multiplication is now (xn,zn). +// Prepare to call the modular inverse function to get zn' = 1/zn + + add x0, zn + add x1, zn + +// Inline copy of bignum_inv_p25519, identical except for stripping out +// the prologue and epilogue saving and restoring registers and making +// and reclaiming room on the stack. For more details and explanations see +// "arm/curve25519/bignum_inv_p25519.S". Note that the stack it uses for +// its own temporaries is 128 bytes, so it has no effect on variables +// that are needed in the rest of our computation here: res, xn and zn. + + mov x20, x0 + mov x10, #0xffffffffffffffed + mov x11, #0xffffffffffffffff + stp x10, x11, [sp] + mov x12, #0x7fffffffffffffff + stp x11, x12, [sp, #16] + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + mov x7, #0x13 + lsr x6, x5, #63 + madd x6, x7, x6, x7 + adds x2, x2, x6 + adcs x3, x3, xzr + adcs x4, x4, xzr + orr x5, x5, #0x8000000000000000 + adcs x5, x5, xzr + csel x6, x7, xzr, cc + subs x2, x2, x6 + sbcs x3, x3, xzr + sbcs x4, x4, xzr + sbc x5, x5, xzr + and x5, x5, #0x7fffffffffffffff + stp x2, x3, [sp, #32] + stp x4, x5, [sp, #48] + stp xzr, xzr, [sp, #64] + stp xzr, xzr, [sp, #80] + mov x10, #0x2099 + movk x10, #0x7502, lsl #16 + movk x10, #0x9e23, lsl #32 + movk x10, #0xa0f9, lsl #48 + mov x11, #0x2595 + movk x11, #0x1d13, lsl #16 + movk x11, #0x8f3f, lsl #32 + movk x11, #0xa8c6, lsl #48 + mov x12, #0x5242 + movk x12, #0x5ac, lsl #16 + movk x12, #0x8938, lsl #32 + movk x12, #0x6c6c, lsl #48 + mov x13, #0x615 + movk x13, #0x4177, lsl #16 + movk x13, #0x8b2, lsl #32 + movk x13, #0x2765, lsl #48 + stp x10, x11, [sp, #96] + stp x12, x13, [sp, #112] + mov x21, #0xa + mov x22, #0x1 + b Lcurve25519_x25519_alt_invmidloop +Lcurve25519_x25519_alt_invloop: + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + and x0, x12, x16 + and x1, x13, x17 + add x19, x0, x1 + ldr x7, [sp] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #32] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x7, [sp, #8] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #40] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + adc x6, x6, x1 + extr x4, x2, x4, #59 + str x4, [sp] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + adc x4, x4, x1 + extr x5, x3, x5, #59 + str x5, [sp, #32] + ldr x7, [sp, #16] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #48] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + adc x5, x5, x1 + extr x2, x6, x2, #59 + str x2, [sp, #8] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + adc x2, x2, x1 + extr x3, x4, x3, #59 + str x3, [sp, #40] + ldr x7, [sp, #24] + eor x1, x7, x14 + asr x3, x1, #63 + and x3, x3, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #56] + eor x1, x8, x15 + asr x0, x1, #63 + and x0, x0, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x5, x6, #59 + str x6, [sp, #16] + extr x5, x3, x5, #59 + str x5, [sp, #24] + eor x1, x7, x16 + asr x5, x1, #63 + and x5, x5, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + asr x0, x1, #63 + and x0, x0, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x4, x2, x4, #59 + str x4, [sp, #48] + extr x2, x5, x2, #59 + str x2, [sp, #56] + ldr x7, [sp, #64] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #96] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #64] + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + str x5, [sp, #96] + adc x3, x3, x1 + ldr x7, [sp, #72] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #104] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #72] + adc x6, x6, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + str x3, [sp, #104] + adc x4, x4, x1 + ldr x7, [sp, #80] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #112] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #80] + adc x5, x5, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + str x4, [sp, #112] + adc x2, x2, x1 + ldr x7, [sp, #88] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #120] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x3, x5, #63 + ldp x0, x1, [sp, #64] + add x6, x6, x3, asr #63 + mov x3, #0x13 + mul x4, x6, x3 + add x5, x5, x6, lsl #63 + smulh x3, x6, x3 + ldr x6, [sp, #80] + adds x0, x0, x4 + adcs x1, x1, x3 + asr x3, x3, #63 + adcs x6, x6, x3 + adc x5, x5, x3 + stp x0, x1, [sp, #64] + stp x6, x5, [sp, #80] + eor x1, x7, x16 + and x5, x16, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + and x0, x17, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x6, x5, x2, #63 + ldp x0, x1, [sp, #96] + add x6, x6, x5, asr #63 + mov x5, #0x13 + mul x4, x6, x5 + add x2, x2, x6, lsl #63 + smulh x5, x6, x5 + ldr x3, [sp, #112] + adds x0, x0, x4 + adcs x1, x1, x5 + asr x5, x5, #63 + adcs x3, x3, x5 + adc x2, x2, x5 + stp x0, x1, [sp, #96] + stp x3, x2, [sp, #112] +Lcurve25519_x25519_alt_invmidloop: + mov x1, x22 + ldr x2, [sp] + ldr x3, [sp, #32] + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x8, x4, #0x100, lsl #12 + sbfx x8, x8, #21, #21 + mov x11, #0x100000 + add x11, x11, x11, lsl #21 + add x9, x4, x11 + asr x9, x9, #42 + add x10, x5, #0x100, lsl #12 + sbfx x10, x10, #21, #21 + add x11, x5, x11 + asr x11, x11, #42 + mul x6, x8, x2 + mul x7, x9, x3 + mul x2, x10, x2 + mul x3, x11, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #21, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #42 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #21, #21 + add x15, x5, x15 + asr x15, x15, #42 + mul x6, x12, x2 + mul x7, x13, x3 + mul x2, x14, x2 + mul x3, x15, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + mul x2, x12, x8 + mul x3, x12, x9 + mul x6, x14, x8 + mul x7, x14, x9 + madd x8, x13, x10, x2 + madd x9, x13, x11, x3 + madd x16, x15, x10, x6 + madd x17, x15, x11, x7 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #22, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #43 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #22, #21 + add x15, x5, x15 + asr x15, x15, #43 + mneg x2, x12, x8 + mneg x3, x12, x9 + mneg x4, x14, x8 + mneg x5, x14, x9 + msub x10, x13, x16, x2 + msub x11, x13, x17, x3 + msub x12, x15, x16, x4 + msub x13, x15, x17, x5 + mov x22, x1 + subs x21, x21, #0x1 + b.ne Lcurve25519_x25519_alt_invloop + ldr x0, [sp] + ldr x1, [sp, #32] + mul x0, x0, x10 + madd x1, x1, x11, x0 + asr x0, x1, #63 + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + eor x14, x14, x0 + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + eor x15, x15, x0 + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + eor x16, x16, x0 + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + eor x17, x17, x0 + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + ldr x7, [sp, #64] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #96] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #64] + adc x2, x2, x1 + ldr x7, [sp, #72] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #104] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #72] + adc x6, x6, x1 + ldr x7, [sp, #80] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #112] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #80] + adc x5, x5, x1 + ldr x7, [sp, #88] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #120] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x3, x5, #63 + ldp x0, x1, [sp, #64] + tst x3, x3 + cinc x6, x6, pl + mov x3, #0x13 + mul x4, x6, x3 + add x5, x5, x6, lsl #63 + smulh x6, x6, x3 + ldr x2, [sp, #80] + adds x0, x0, x4 + adcs x1, x1, x6 + asr x6, x6, #63 + adcs x2, x2, x6 + adcs x5, x5, x6 + csel x3, x3, xzr, mi + subs x0, x0, x3 + sbcs x1, x1, xzr + sbcs x2, x2, xzr + sbc x5, x5, xzr + and x5, x5, #0x7fffffffffffffff + mov x4, x20 + stp x0, x1, [x4] + stp x2, x5, [x4, #16] + +// Now the result is xn * (1/zn), fully reduced modulo p. +// Note that in the degenerate case zn = 0 (mod p_25519), the +// modular inverse code above will produce 1/zn = 0, giving +// the correct overall X25519 result of zero for the point at +// infinity. + + mul_p25519(resx,xn,zn) + +// Restore stack and registers + + CFI_INC_SP(NSPACE) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(curve25519_x25519_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/curve25519_x25519base.S b/cbits/s2n/arm/curve25519_x25519base.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/curve25519_x25519base.S @@ -0,0 +1,9591 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// The x25519 function for curve25519 on base element 9 +// Input scalar[4]; output res[4] +// +// extern void curve25519_x25519base +// (uint64_t res[static 4],const uint64_t scalar[static 4]); +// +// Given a scalar n, returns the X coordinate of n * G where G = (9,...) is +// the standard generator. The scalar is first slightly modified/mangled +// as specified in the relevant RFC (https://www.rfc-editor.org/rfc/rfc7748). +// +// Standard ARM ABI: X0 = res, X1 = scalar +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(curve25519_x25519base) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(curve25519_x25519base) + S2N_BN_SYM_PRIVACY_DIRECTIVE(curve25519_x25519base) + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Stable home for the input result argument during the whole body + +#define res x23 + +// Other variables that are only needed prior to the modular inverse. + +#define tab x19 + +#define i x20 + +#define bias x21 + +#define bf x22 +#define ix x22 + +// Pointer-offset pairs for result and temporaries on stack with some aliasing. + +#define resx res, #(0*NUMSIZE) + +#define scalar sp, #(0*NUMSIZE) + +#define tabent sp, #(1*NUMSIZE) +#define ymx_2 sp, #(1*NUMSIZE) +#define xpy_2 sp, #(2*NUMSIZE) +#define kxy_2 sp, #(3*NUMSIZE) + +#define acc sp, #(4*NUMSIZE) +#define x_1 sp, #(4*NUMSIZE) +#define y_1 sp, #(5*NUMSIZE) +#define z_1 sp, #(6*NUMSIZE) +#define w_1 sp, #(7*NUMSIZE) +#define x_3 sp, #(4*NUMSIZE) +#define y_3 sp, #(5*NUMSIZE) +#define z_3 sp, #(6*NUMSIZE) +#define w_3 sp, #(7*NUMSIZE) + +#define tmpspace sp, #(8*NUMSIZE) +#define t0 sp, #(8*NUMSIZE) +#define t1 sp, #(9*NUMSIZE) +#define t2 sp, #(10*NUMSIZE) +#define t3 sp, #(11*NUMSIZE) +#define t4 sp, #(12*NUMSIZE) +#define t5 sp, #(13*NUMSIZE) + +// Total size to reserve on the stack + +#define NSPACE 14*NUMSIZE + +// Macro wrapping up the basic field operation bignum_mul_p25519, only +// trivially different from a pure function call to that subroutine. + +#define mul_p25519(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x5, x6, [P2] __LF \ + umull x7, w3, w5 __LF \ + lsr x0, x3, #32 __LF \ + umull x15, w0, w5 __LF \ + lsr x16, x5, #32 __LF \ + umull x8, w16, w0 __LF \ + umull x16, w3, w16 __LF \ + adds x7, x7, x15, lsl #32 __LF \ + lsr x15, x15, #32 __LF \ + adc x8, x8, x15 __LF \ + adds x7, x7, x16, lsl #32 __LF \ + lsr x16, x16, #32 __LF \ + adc x8, x8, x16 __LF \ + mul x9, x4, x6 __LF \ + umulh x10, x4, x6 __LF \ + subs x4, x4, x3 __LF \ + cneg x4, x4, cc __LF \ + csetm x16, cc __LF \ + adds x9, x9, x8 __LF \ + adc x10, x10, xzr __LF \ + subs x3, x5, x6 __LF \ + cneg x3, x3, cc __LF \ + cinv x16, x16, cc __LF \ + mul x15, x4, x3 __LF \ + umulh x3, x4, x3 __LF \ + adds x8, x7, x9 __LF \ + adcs x9, x9, x10 __LF \ + adc x10, x10, xzr __LF \ + cmn x16, #0x1 __LF \ + eor x15, x15, x16 __LF \ + adcs x8, x15, x8 __LF \ + eor x3, x3, x16 __LF \ + adcs x9, x3, x9 __LF \ + adc x10, x10, x16 __LF \ + ldp x3, x4, [P1+16] __LF \ + ldp x5, x6, [P2+16] __LF \ + umull x11, w3, w5 __LF \ + lsr x0, x3, #32 __LF \ + umull x15, w0, w5 __LF \ + lsr x16, x5, #32 __LF \ + umull x12, w16, w0 __LF \ + umull x16, w3, w16 __LF \ + adds x11, x11, x15, lsl #32 __LF \ + lsr x15, x15, #32 __LF \ + adc x12, x12, x15 __LF \ + adds x11, x11, x16, lsl #32 __LF \ + lsr x16, x16, #32 __LF \ + adc x12, x12, x16 __LF \ + mul x13, x4, x6 __LF \ + umulh x14, x4, x6 __LF \ + subs x4, x4, x3 __LF \ + cneg x4, x4, cc __LF \ + csetm x16, cc __LF \ + adds x13, x13, x12 __LF \ + adc x14, x14, xzr __LF \ + subs x3, x5, x6 __LF \ + cneg x3, x3, cc __LF \ + cinv x16, x16, cc __LF \ + mul x15, x4, x3 __LF \ + umulh x3, x4, x3 __LF \ + adds x12, x11, x13 __LF \ + adcs x13, x13, x14 __LF \ + adc x14, x14, xzr __LF \ + cmn x16, #0x1 __LF \ + eor x15, x15, x16 __LF \ + adcs x12, x15, x12 __LF \ + eor x3, x3, x16 __LF \ + adcs x13, x3, x13 __LF \ + adc x14, x14, x16 __LF \ + ldp x3, x4, [P1+16] __LF \ + ldp x15, x16, [P1] __LF \ + subs x3, x3, x15 __LF \ + sbcs x4, x4, x16 __LF \ + csetm x16, cc __LF \ + ldp x15, x0, [P2] __LF \ + subs x5, x15, x5 __LF \ + sbcs x6, x0, x6 __LF \ + csetm x0, cc __LF \ + eor x3, x3, x16 __LF \ + subs x3, x3, x16 __LF \ + eor x4, x4, x16 __LF \ + sbc x4, x4, x16 __LF \ + eor x5, x5, x0 __LF \ + subs x5, x5, x0 __LF \ + eor x6, x6, x0 __LF \ + sbc x6, x6, x0 __LF \ + eor x16, x0, x16 __LF \ + adds x11, x11, x9 __LF \ + adcs x12, x12, x10 __LF \ + adcs x13, x13, xzr __LF \ + adc x14, x14, xzr __LF \ + mul x2, x3, x5 __LF \ + umulh x0, x3, x5 __LF \ + mul x15, x4, x6 __LF \ + umulh x1, x4, x6 __LF \ + subs x4, x4, x3 __LF \ + cneg x4, x4, cc __LF \ + csetm x9, cc __LF \ + adds x15, x15, x0 __LF \ + adc x1, x1, xzr __LF \ + subs x6, x5, x6 __LF \ + cneg x6, x6, cc __LF \ + cinv x9, x9, cc __LF \ + mul x5, x4, x6 __LF \ + umulh x6, x4, x6 __LF \ + adds x0, x2, x15 __LF \ + adcs x15, x15, x1 __LF \ + adc x1, x1, xzr __LF \ + cmn x9, #0x1 __LF \ + eor x5, x5, x9 __LF \ + adcs x0, x5, x0 __LF \ + eor x6, x6, x9 __LF \ + adcs x15, x6, x15 __LF \ + adc x1, x1, x9 __LF \ + adds x9, x11, x7 __LF \ + adcs x10, x12, x8 __LF \ + adcs x11, x13, x11 __LF \ + adcs x12, x14, x12 __LF \ + adcs x13, x13, xzr __LF \ + adc x14, x14, xzr __LF \ + cmn x16, #0x1 __LF \ + eor x2, x2, x16 __LF \ + adcs x9, x2, x9 __LF \ + eor x0, x0, x16 __LF \ + adcs x10, x0, x10 __LF \ + eor x15, x15, x16 __LF \ + adcs x11, x15, x11 __LF \ + eor x1, x1, x16 __LF \ + adcs x12, x1, x12 __LF \ + adcs x13, x13, x16 __LF \ + adc x14, x14, x16 __LF \ + mov x3, #0x26 __LF \ + umull x4, w11, w3 __LF \ + add x4, x4, w7, uxtw __LF \ + lsr x7, x7, #32 __LF \ + lsr x11, x11, #32 __LF \ + umaddl x11, w11, w3, x7 __LF \ + mov x7, x4 __LF \ + umull x4, w12, w3 __LF \ + add x4, x4, w8, uxtw __LF \ + lsr x8, x8, #32 __LF \ + lsr x12, x12, #32 __LF \ + umaddl x12, w12, w3, x8 __LF \ + mov x8, x4 __LF \ + umull x4, w13, w3 __LF \ + add x4, x4, w9, uxtw __LF \ + lsr x9, x9, #32 __LF \ + lsr x13, x13, #32 __LF \ + umaddl x13, w13, w3, x9 __LF \ + mov x9, x4 __LF \ + umull x4, w14, w3 __LF \ + add x4, x4, w10, uxtw __LF \ + lsr x10, x10, #32 __LF \ + lsr x14, x14, #32 __LF \ + umaddl x14, w14, w3, x10 __LF \ + mov x10, x4 __LF \ + lsr x0, x14, #31 __LF \ + mov x5, #0x13 __LF \ + umaddl x5, w5, w0, x5 __LF \ + add x7, x7, x5 __LF \ + adds x7, x7, x11, lsl #32 __LF \ + extr x3, x12, x11, #32 __LF \ + adcs x8, x8, x3 __LF \ + extr x3, x13, x12, #32 __LF \ + adcs x9, x9, x3 __LF \ + extr x3, x14, x13, #32 __LF \ + lsl x5, x0, #63 __LF \ + eor x10, x10, x5 __LF \ + adc x10, x10, x3 __LF \ + mov x3, #0x13 __LF \ + tst x10, #0x8000000000000000 __LF \ + csel x3, x3, xzr, pl __LF \ + subs x7, x7, x3 __LF \ + sbcs x8, x8, xzr __LF \ + sbcs x9, x9, xzr __LF \ + sbc x10, x10, xzr __LF \ + and x10, x10, #0x7fffffffffffffff __LF \ + stp x7, x8, [P0] __LF \ + stp x9, x10, [P0+16] + +// A version of multiplication that only guarantees output < 2 * p_25519. +// This basically skips the +1 and final correction in quotient estimation. + +#define mul_4(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x5, x6, [P2] __LF \ + umull x7, w3, w5 __LF \ + lsr x0, x3, #32 __LF \ + umull x15, w0, w5 __LF \ + lsr x16, x5, #32 __LF \ + umull x8, w16, w0 __LF \ + umull x16, w3, w16 __LF \ + adds x7, x7, x15, lsl #32 __LF \ + lsr x15, x15, #32 __LF \ + adc x8, x8, x15 __LF \ + adds x7, x7, x16, lsl #32 __LF \ + lsr x16, x16, #32 __LF \ + adc x8, x8, x16 __LF \ + mul x9, x4, x6 __LF \ + umulh x10, x4, x6 __LF \ + subs x4, x4, x3 __LF \ + cneg x4, x4, cc __LF \ + csetm x16, cc __LF \ + adds x9, x9, x8 __LF \ + adc x10, x10, xzr __LF \ + subs x3, x5, x6 __LF \ + cneg x3, x3, cc __LF \ + cinv x16, x16, cc __LF \ + mul x15, x4, x3 __LF \ + umulh x3, x4, x3 __LF \ + adds x8, x7, x9 __LF \ + adcs x9, x9, x10 __LF \ + adc x10, x10, xzr __LF \ + cmn x16, #0x1 __LF \ + eor x15, x15, x16 __LF \ + adcs x8, x15, x8 __LF \ + eor x3, x3, x16 __LF \ + adcs x9, x3, x9 __LF \ + adc x10, x10, x16 __LF \ + ldp x3, x4, [P1+16] __LF \ + ldp x5, x6, [P2+16] __LF \ + umull x11, w3, w5 __LF \ + lsr x0, x3, #32 __LF \ + umull x15, w0, w5 __LF \ + lsr x16, x5, #32 __LF \ + umull x12, w16, w0 __LF \ + umull x16, w3, w16 __LF \ + adds x11, x11, x15, lsl #32 __LF \ + lsr x15, x15, #32 __LF \ + adc x12, x12, x15 __LF \ + adds x11, x11, x16, lsl #32 __LF \ + lsr x16, x16, #32 __LF \ + adc x12, x12, x16 __LF \ + mul x13, x4, x6 __LF \ + umulh x14, x4, x6 __LF \ + subs x4, x4, x3 __LF \ + cneg x4, x4, cc __LF \ + csetm x16, cc __LF \ + adds x13, x13, x12 __LF \ + adc x14, x14, xzr __LF \ + subs x3, x5, x6 __LF \ + cneg x3, x3, cc __LF \ + cinv x16, x16, cc __LF \ + mul x15, x4, x3 __LF \ + umulh x3, x4, x3 __LF \ + adds x12, x11, x13 __LF \ + adcs x13, x13, x14 __LF \ + adc x14, x14, xzr __LF \ + cmn x16, #0x1 __LF \ + eor x15, x15, x16 __LF \ + adcs x12, x15, x12 __LF \ + eor x3, x3, x16 __LF \ + adcs x13, x3, x13 __LF \ + adc x14, x14, x16 __LF \ + ldp x3, x4, [P1+16] __LF \ + ldp x15, x16, [P1] __LF \ + subs x3, x3, x15 __LF \ + sbcs x4, x4, x16 __LF \ + csetm x16, cc __LF \ + ldp x15, x0, [P2] __LF \ + subs x5, x15, x5 __LF \ + sbcs x6, x0, x6 __LF \ + csetm x0, cc __LF \ + eor x3, x3, x16 __LF \ + subs x3, x3, x16 __LF \ + eor x4, x4, x16 __LF \ + sbc x4, x4, x16 __LF \ + eor x5, x5, x0 __LF \ + subs x5, x5, x0 __LF \ + eor x6, x6, x0 __LF \ + sbc x6, x6, x0 __LF \ + eor x16, x0, x16 __LF \ + adds x11, x11, x9 __LF \ + adcs x12, x12, x10 __LF \ + adcs x13, x13, xzr __LF \ + adc x14, x14, xzr __LF \ + mul x2, x3, x5 __LF \ + umulh x0, x3, x5 __LF \ + mul x15, x4, x6 __LF \ + umulh x1, x4, x6 __LF \ + subs x4, x4, x3 __LF \ + cneg x4, x4, cc __LF \ + csetm x9, cc __LF \ + adds x15, x15, x0 __LF \ + adc x1, x1, xzr __LF \ + subs x6, x5, x6 __LF \ + cneg x6, x6, cc __LF \ + cinv x9, x9, cc __LF \ + mul x5, x4, x6 __LF \ + umulh x6, x4, x6 __LF \ + adds x0, x2, x15 __LF \ + adcs x15, x15, x1 __LF \ + adc x1, x1, xzr __LF \ + cmn x9, #0x1 __LF \ + eor x5, x5, x9 __LF \ + adcs x0, x5, x0 __LF \ + eor x6, x6, x9 __LF \ + adcs x15, x6, x15 __LF \ + adc x1, x1, x9 __LF \ + adds x9, x11, x7 __LF \ + adcs x10, x12, x8 __LF \ + adcs x11, x13, x11 __LF \ + adcs x12, x14, x12 __LF \ + adcs x13, x13, xzr __LF \ + adc x14, x14, xzr __LF \ + cmn x16, #0x1 __LF \ + eor x2, x2, x16 __LF \ + adcs x9, x2, x9 __LF \ + eor x0, x0, x16 __LF \ + adcs x10, x0, x10 __LF \ + eor x15, x15, x16 __LF \ + adcs x11, x15, x11 __LF \ + eor x1, x1, x16 __LF \ + adcs x12, x1, x12 __LF \ + adcs x13, x13, x16 __LF \ + adc x14, x14, x16 __LF \ + mov x3, #0x26 __LF \ + umull x4, w11, w3 __LF \ + add x4, x4, w7, uxtw __LF \ + lsr x7, x7, #32 __LF \ + lsr x11, x11, #32 __LF \ + umaddl x11, w11, w3, x7 __LF \ + mov x7, x4 __LF \ + umull x4, w12, w3 __LF \ + add x4, x4, w8, uxtw __LF \ + lsr x8, x8, #32 __LF \ + lsr x12, x12, #32 __LF \ + umaddl x12, w12, w3, x8 __LF \ + mov x8, x4 __LF \ + umull x4, w13, w3 __LF \ + add x4, x4, w9, uxtw __LF \ + lsr x9, x9, #32 __LF \ + lsr x13, x13, #32 __LF \ + umaddl x13, w13, w3, x9 __LF \ + mov x9, x4 __LF \ + umull x4, w14, w3 __LF \ + add x4, x4, w10, uxtw __LF \ + lsr x10, x10, #32 __LF \ + lsr x14, x14, #32 __LF \ + umaddl x14, w14, w3, x10 __LF \ + mov x10, x4 __LF \ + lsr x0, x14, #31 __LF \ + mov x5, #0x13 __LF \ + umull x5, w5, w0 __LF \ + add x7, x7, x5 __LF \ + adds x7, x7, x11, lsl #32 __LF \ + extr x3, x12, x11, #32 __LF \ + adcs x8, x8, x3 __LF \ + extr x3, x13, x12, #32 __LF \ + adcs x9, x9, x3 __LF \ + extr x3, x14, x13, #32 __LF \ + lsl x5, x0, #63 __LF \ + eor x10, x10, x5 __LF \ + adc x10, x10, x3 __LF \ + stp x7, x8, [P0] __LF \ + stp x9, x10, [P0+16] + +// Modular subtraction with double modulus 2 * p_25519 = 2^256 - 38 + +#define sub_twice4(P0,P1,P2) \ + ldp x5, x6, [P1] __LF \ + ldp x4, x3, [P2] __LF \ + subs x5, x5, x4 __LF \ + sbcs x6, x6, x3 __LF \ + ldp x7, x8, [P1+16] __LF \ + ldp x4, x3, [P2+16] __LF \ + sbcs x7, x7, x4 __LF \ + sbcs x8, x8, x3 __LF \ + mov x4, #38 __LF \ + csel x3, x4, xzr, lo __LF \ + subs x5, x5, x3 __LF \ + sbcs x6, x6, xzr __LF \ + sbcs x7, x7, xzr __LF \ + sbc x8, x8, xzr __LF \ + stp x5, x6, [P0] __LF \ + stp x7, x8, [P0+16] + +// Modular addition and doubling with double modulus 2 * p_25519 = 2^256 - 38. +// This only ensures that the result fits in 4 digits, not that it is reduced +// even w.r.t. double modulus. The result is always correct modulo provided +// the sum of the inputs is < 2^256 + 2^256 - 38, so in particular provided +// at least one of them is reduced double modulo. + +#define add_twice4(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x7, x8, [P2] __LF \ + adds x3, x3, x7 __LF \ + adcs x4, x4, x8 __LF \ + ldp x5, x6, [P1+16] __LF \ + ldp x7, x8, [P2+16] __LF \ + adcs x5, x5, x7 __LF \ + adcs x6, x6, x8 __LF \ + mov x9, #38 __LF \ + csel x9, x9, xzr, cs __LF \ + adds x3, x3, x9 __LF \ + adcs x4, x4, xzr __LF \ + adcs x5, x5, xzr __LF \ + adc x6, x6, xzr __LF \ + stp x3, x4, [P0] __LF \ + stp x5, x6, [P0+16] + +#define double_twice4(P0,P1) \ + ldp x3, x4, [P1] __LF \ + adds x3, x3, x3 __LF \ + adcs x4, x4, x4 __LF \ + ldp x5, x6, [P1+16] __LF \ + adcs x5, x5, x5 __LF \ + adcs x6, x6, x6 __LF \ + mov x9, #38 __LF \ + csel x9, x9, xzr, cs __LF \ + adds x3, x3, x9 __LF \ + adcs x4, x4, xzr __LF \ + adcs x5, x5, xzr __LF \ + adc x6, x6, xzr __LF \ + stp x3, x4, [P0] __LF \ + stp x5, x6, [P0+16] + +S2N_BN_SYMBOL(curve25519_x25519base): + CFI_START + +// Save regs and make room for temporaries + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_DEC_SP(NSPACE) + +// Move the output pointer to a stable place + + mov res, x0 + +// Copy the input scalar to its local variable while mangling it. +// In principle the mangling is into 01xxx...xxx000, but actually +// we only clear the top two bits so 00xxx...xxxxxx. The additional +// 2^254 * G is taken care of by the starting value for the addition +// chain below, while we never look at the three low bits at all. + + ldp x10, x11, [x1] + stp x10, x11, [scalar] + ldp x12, x13, [x1, #16] + bic x13, x13, #0xc000000000000000 + stp x12, x13, [scalar+16] + +// The main part of the computation is on the edwards25519 curve in +// extended-projective coordinates (X,Y,Z,T), representing a point +// (x,y) via x = X/Z, y = Y/Z and x * y = T/Z (so X * Y = T * Z). +// Only at the very end do we translate back to curve25519. So G +// below means the generator within edwards25519 corresponding to +// (9,...) for curve25519, via the standard isomorphism. +// +// Initialize accumulator "acc" to either (2^254 + 8) * G or just 2^254 * G +// depending on bit 3 of the scalar, the only nonzero bit of the bottom 4. +// Thus, we have effectively dealt with bits 0, 1, 2, 3, 254 and 255. + + ldr x0, [scalar] + ands xzr, x0, #8 + +#if defined(__ELF__) + adrp tab, S2N_BN_SYMBOL(curve25519_x25519base_constant) + add tab, tab, :lo12:S2N_BN_SYMBOL(curve25519_x25519base_constant) +#else + adrp tab, S2N_BN_SYMBOL(curve25519_x25519base_constant)@PAGE + add tab, tab, S2N_BN_SYMBOL(curve25519_x25519base_constant)@PAGEOFF +#endif + + ldp x0, x1, [tab] + ldp x2, x3, [tab, #96] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc] + + ldp x0, x1, [tab, #1*16] + ldp x2, x3, [tab, #96+1*16] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+1*16] + + ldp x0, x1, [tab, #2*16] + ldp x2, x3, [tab, #96+2*16] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+2*16] + + ldp x0, x1, [tab, #3*16] + ldp x2, x3, [tab, #96+3*16] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+3*16] + + mov x0, #1 + stp x0, xzr, [acc+4*16] + stp xzr, xzr, [acc+5*16] + + ldp x0, x1, [tab, #4*16] + ldp x2, x3, [tab, #96+4*16] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+6*16] + + ldp x0, x1, [tab, #5*16] + ldp x2, x3, [tab, #96+5*16] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+7*16] + +// The counter "i" tracks the bit position for which the scalar has +// already been absorbed, starting at 4 and going up in chunks of 4. +// +// The pointer "tab" points at the current block of the table for +// multiples (2^i * j) * G at the current bit position i; 1 <= j <= 8. +// +// The bias is always either 0 and 1 and needs to be added to the +// partially processed scalar implicitly. This is used to absorb 4 bits +// of scalar per iteration from 3-bit table indexing by exploiting +// negation: (16 * h + l) * G = (16 * (h + 1) - (16 - l)) * G is used +// when l >= 9. Note that we can't have any bias left over at the +// end because of the clearing of bit 255 of the scalar, meaning the +// l >= 9 case cannot arise on the last iteration. + + mov i, 4 + add tab, tab, #192 + mov bias, xzr + +// Start of the main loop, repeated 63 times for i = 4, 8, ..., 252 + +Lcurve25519_x25519base_scalarloop: + +// Look at the next 4-bit field "bf", adding the previous bias as well. +// Choose the table index "ix" as bf when bf <= 8 and 16 - bf for bf >= 9, +// setting the bias to 1 for the next iteration in the latter case. + + lsr x0, i, #6 + ldr x2, [sp, x0, lsl #3] // Exploiting scalar = sp exactly + lsr x2, x2, i + and x2, x2, #15 + add bf, x2, bias + + cmp bf, 9 + cset bias, cs + + mov x0, 16 + sub x0, x0, bf + cmp bias, xzr + csel ix, x0, bf, ne + +// Perform constant-time lookup in the table to get element number "ix". +// The table entry for the affine point (x,y) is actually a triple +// (y - x,x + y,2 * d * x * y) to precompute parts of the addition. +// Note that "ix" can be 0, so we set up the appropriate identity first. + + mov x0, #1 + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, #1 + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + mov x9, xzr + mov x10, xzr + mov x11, xzr + + cmp ix, #1 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #2 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #3 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #4 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #5 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #6 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #7 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #8 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + +// We now have the triple from the table in registers as follows +// +// [x3;x2;x1;x0] = y - x +// [x7;x6;x5;x4] = x + y +// [x11;x10;x9;x8] = 2 * d * x * y +// +// In case bias = 1 we need to negate this. For Edwards curves +// -(x,y) = (-x,y), i.e. we need to negate the x coordinate. +// In this processed encoding, that amounts to swapping the +// first two fields and negating the third. +// +// The optional negation here also pretends bias = 0 whenever +// ix = 0 so that it doesn't need to handle the case of zero +// inputs, since no non-trivial table entries are zero. Note +// that in the zero case the whole negation is trivial, and +// so indeed is the swapping. + + cmp bias, #0 + + csel x12, x0, x4, eq + csel x13, x1, x5, eq + csel x14, x2, x6, eq + csel x15, x3, x7, eq + stp x12, x13, [tabent] + stp x14, x15, [tabent+16] + + csel x12, x0, x4, ne + csel x13, x1, x5, ne + csel x14, x2, x6, ne + csel x15, x3, x7, ne + stp x12, x13, [tabent+32] + stp x14, x15, [tabent+48] + + mov x0, #-19 + subs x0, x0, x8 + mov x2, #-1 + sbcs x1, x2, x9 + sbcs x2, x2, x10 + mov x3, #0x7FFFFFFFFFFFFFFF + sbc x3, x3, x11 + + cmp ix, xzr + ccmp bias, xzr, #4, ne + + csel x0, x0, x8, ne + csel x1, x1, x9, ne + stp x0, x1, [tabent+64] + csel x2, x2, x10, ne + csel x3, x3, x11, ne + stp x2, x3, [tabent+80] + +// Extended-projective and precomputed mixed addition. +// This is effectively the same as calling the standalone +// function edwards25519_pepadd(acc,acc,tabent), but we +// only retain slightly weaker normalization < 2 * p_25519 +// throughout the inner loop, so the computation is +// slightly different, and faster overall. + + double_twice4(t0,z_1) + sub_twice4(t1,y_1,x_1) + add_twice4(t2,y_1,x_1) + mul_4(t3,w_1,kxy_2) + mul_4(t1,t1,ymx_2) + mul_4(t2,t2,xpy_2) + sub_twice4(t4,t0,t3) + add_twice4(t0,t0,t3) + sub_twice4(t5,t2,t1) + add_twice4(t1,t2,t1) + mul_4(z_3,t4,t0) + mul_4(x_3,t5,t4) + mul_4(y_3,t0,t1) + mul_4(w_3,t5,t1) + +// End of the main loop; move on by 4 bits. + + add i, i, 4 + cmp i, 256 + bcc Lcurve25519_x25519base_scalarloop + +// Now we need to translate from Edwards curve edwards25519 back +// to the Montgomery form curve25519. The mapping in the affine +// representations is +// +// (x,y) |-> ((1 + y) / (1 - y), c * (1 + y) / ((1 - y) * x)) +// +// For x25519, we only need the x coordinate, and we compute this as +// +// (1 + y) / (1 - y) = (x + x * y) / (x - x * y) +// = (X/Z + T/Z) / (X/Z - T/Z) +// = (X + T) / (X - T) +// = (X + T) * inverse(X - T) +// +// We could equally well use (Z + Y) / (Z - Y), but the above has the +// same cost, and it more explicitly forces zero output whenever X = 0, +// regardless of how the modular inverse behaves on zero inputs. In +// the present setting (base point 9, mangled scalar) that doesn't +// really matter anyway since X = 0 never arises, but it seems a +// little bit tidier. Note that both Edwards point (0,1) which maps to +// the Montgomery point at infinity, and Edwards (0,-1) which maps to +// Montgomery (0,0) [this is the 2-torsion point] are both by definition +// mapped to 0 by the X coordinate mapping used to define curve25519. +// +// First the addition and subtraction: + + add_twice4(t1,x_3,w_3) + sub_twice4(t2,x_3,w_3) + +// Prepare to call the modular inverse function to get t0 = 1/t2 +// Note that this works for the weakly normalized z_3 equally well. +// The non-coprime case z_3 == 0 (mod p_25519) cannot arise anyway. + + add x0, t0 + add x1, t2 + +// Inline copy of bignum_inv_p25519, identical except for stripping out +// the prologue and epilogue saving and restoring registers and making +// and reclaiming room on the stack. For more details and explanations see +// "arm/curve25519/bignum_inv_p25519.S". Note that the stack it uses for +// its own temporaries is 128 bytes, so it has no effect on variables +// that are needed in the rest of our computation here: res, t0, t1, t2. + + mov x20, x0 + mov x10, #0xffffffffffffffed + mov x11, #0xffffffffffffffff + stp x10, x11, [sp] + mov x12, #0x7fffffffffffffff + stp x11, x12, [sp, #16] + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + mov x7, #0x13 + lsr x6, x5, #63 + madd x6, x7, x6, x7 + adds x2, x2, x6 + adcs x3, x3, xzr + adcs x4, x4, xzr + orr x5, x5, #0x8000000000000000 + adcs x5, x5, xzr + csel x6, x7, xzr, cc + subs x2, x2, x6 + sbcs x3, x3, xzr + sbcs x4, x4, xzr + sbc x5, x5, xzr + and x5, x5, #0x7fffffffffffffff + stp x2, x3, [sp, #32] + stp x4, x5, [sp, #48] + stp xzr, xzr, [sp, #64] + stp xzr, xzr, [sp, #80] + mov x10, #0x2099 + movk x10, #0x7502, lsl #16 + movk x10, #0x9e23, lsl #32 + movk x10, #0xa0f9, lsl #48 + mov x11, #0x2595 + movk x11, #0x1d13, lsl #16 + movk x11, #0x8f3f, lsl #32 + movk x11, #0xa8c6, lsl #48 + mov x12, #0x5242 + movk x12, #0x5ac, lsl #16 + movk x12, #0x8938, lsl #32 + movk x12, #0x6c6c, lsl #48 + mov x13, #0x615 + movk x13, #0x4177, lsl #16 + movk x13, #0x8b2, lsl #32 + movk x13, #0x2765, lsl #48 + stp x10, x11, [sp, #96] + stp x12, x13, [sp, #112] + mov x21, #0xa + mov x22, #0x1 + b Lcurve25519_x25519base_invmidloop +Lcurve25519_x25519base_invloop: + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + and x0, x12, x16 + and x1, x13, x17 + add x19, x0, x1 + ldr x7, [sp] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #32] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x7, [sp, #8] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #40] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + adc x6, x6, x1 + extr x4, x2, x4, #59 + str x4, [sp] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + adc x4, x4, x1 + extr x5, x3, x5, #59 + str x5, [sp, #32] + ldr x7, [sp, #16] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #48] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + adc x5, x5, x1 + extr x2, x6, x2, #59 + str x2, [sp, #8] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + adc x2, x2, x1 + extr x3, x4, x3, #59 + str x3, [sp, #40] + ldr x7, [sp, #24] + eor x1, x7, x14 + asr x3, x1, #63 + and x3, x3, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #56] + eor x1, x8, x15 + asr x0, x1, #63 + and x0, x0, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x5, x6, #59 + str x6, [sp, #16] + extr x5, x3, x5, #59 + str x5, [sp, #24] + eor x1, x7, x16 + asr x5, x1, #63 + and x5, x5, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + asr x0, x1, #63 + and x0, x0, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x4, x2, x4, #59 + str x4, [sp, #48] + extr x2, x5, x2, #59 + str x2, [sp, #56] + ldr x7, [sp, #64] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #96] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #64] + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + str x5, [sp, #96] + adc x3, x3, x1 + ldr x7, [sp, #72] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #104] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #72] + adc x6, x6, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + str x3, [sp, #104] + adc x4, x4, x1 + ldr x7, [sp, #80] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #112] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #80] + adc x5, x5, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + str x4, [sp, #112] + adc x2, x2, x1 + ldr x7, [sp, #88] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #120] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x3, x5, #63 + ldp x0, x1, [sp, #64] + add x6, x6, x3, asr #63 + mov x3, #0x13 + mul x4, x6, x3 + add x5, x5, x6, lsl #63 + smulh x3, x6, x3 + ldr x6, [sp, #80] + adds x0, x0, x4 + adcs x1, x1, x3 + asr x3, x3, #63 + adcs x6, x6, x3 + adc x5, x5, x3 + stp x0, x1, [sp, #64] + stp x6, x5, [sp, #80] + eor x1, x7, x16 + and x5, x16, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + and x0, x17, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x6, x5, x2, #63 + ldp x0, x1, [sp, #96] + add x6, x6, x5, asr #63 + mov x5, #0x13 + mul x4, x6, x5 + add x2, x2, x6, lsl #63 + smulh x5, x6, x5 + ldr x3, [sp, #112] + adds x0, x0, x4 + adcs x1, x1, x5 + asr x5, x5, #63 + adcs x3, x3, x5 + adc x2, x2, x5 + stp x0, x1, [sp, #96] + stp x3, x2, [sp, #112] +Lcurve25519_x25519base_invmidloop: + mov x1, x22 + ldr x2, [sp] + ldr x3, [sp, #32] + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x8, x4, #0x100, lsl #12 + sbfx x8, x8, #21, #21 + mov x11, #0x100000 + add x11, x11, x11, lsl #21 + add x9, x4, x11 + asr x9, x9, #42 + add x10, x5, #0x100, lsl #12 + sbfx x10, x10, #21, #21 + add x11, x5, x11 + asr x11, x11, #42 + mul x6, x8, x2 + mul x7, x9, x3 + mul x2, x10, x2 + mul x3, x11, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #21, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #42 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #21, #21 + add x15, x5, x15 + asr x15, x15, #42 + mul x6, x12, x2 + mul x7, x13, x3 + mul x2, x14, x2 + mul x3, x15, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + mul x2, x12, x8 + mul x3, x12, x9 + mul x6, x14, x8 + mul x7, x14, x9 + madd x8, x13, x10, x2 + madd x9, x13, x11, x3 + madd x16, x15, x10, x6 + madd x17, x15, x11, x7 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #22, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #43 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #22, #21 + add x15, x5, x15 + asr x15, x15, #43 + mneg x2, x12, x8 + mneg x3, x12, x9 + mneg x4, x14, x8 + mneg x5, x14, x9 + msub x10, x13, x16, x2 + msub x11, x13, x17, x3 + msub x12, x15, x16, x4 + msub x13, x15, x17, x5 + mov x22, x1 + subs x21, x21, #0x1 + b.ne Lcurve25519_x25519base_invloop + ldr x0, [sp] + ldr x1, [sp, #32] + mul x0, x0, x10 + madd x1, x1, x11, x0 + asr x0, x1, #63 + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + eor x14, x14, x0 + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + eor x15, x15, x0 + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + eor x16, x16, x0 + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + eor x17, x17, x0 + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + ldr x7, [sp, #64] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #96] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #64] + adc x2, x2, x1 + ldr x7, [sp, #72] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #104] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #72] + adc x6, x6, x1 + ldr x7, [sp, #80] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #112] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #80] + adc x5, x5, x1 + ldr x7, [sp, #88] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #120] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x3, x5, #63 + ldp x0, x1, [sp, #64] + tst x3, x3 + cinc x6, x6, pl + mov x3, #0x13 + mul x4, x6, x3 + add x5, x5, x6, lsl #63 + smulh x6, x6, x3 + ldr x2, [sp, #80] + adds x0, x0, x4 + adcs x1, x1, x6 + asr x6, x6, #63 + adcs x2, x2, x6 + adcs x5, x5, x6 + csel x3, x3, xzr, mi + subs x0, x0, x3 + sbcs x1, x1, xzr + sbcs x2, x2, xzr + sbc x5, x5, xzr + and x5, x5, #0x7fffffffffffffff + mov x4, x20 + stp x0, x1, [x4] + stp x2, x5, [x4, #16] + +// The final result is (X + T) / (X - T) +// This is the only operation in the whole computation that +// fully reduces modulo p_25519 since now we want the canonical +// answer as output. + + mul_p25519(resx,t1,t0) + +// Restore stack and registers + + CFI_INC_SP(NSPACE) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(curve25519_x25519base) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif + +// **************************************************************************** +// The precomputed data (all read-only). +// **************************************************************************** + +#if defined(__ELF__) +.section .rodata +.type S2N_BN_SYMBOL(curve25519_x25519base_constant), %object +.size S2N_BN_SYMBOL(curve25519_x25519base_constant), 48576 +#elif defined(__APPLE__) +.const_data +#endif + +S2N_BN_SYMBOL(curve25519_x25519base_constant): + +// 2^254 * G and (2^254 + 8) * G in extended-projective coordinates +// but with Z = 1 assumed and hence left out, so they are (X,Y,T) only. + + .quad 0x251037f7cf4e861d + .quad 0x10ede0fb19fb128f + .quad 0x96c033b175f5e2c8 + .quad 0x055f070d6c15fb0d + .quad 0x7c52af2c97473e69 + .quad 0x022f82391bad8378 + .quad 0x9991e1b02adb476f + .quad 0x511144a03a99b855 + .quad 0x5fafc3b88ff2e4ae + .quad 0x855e4ff0de1230ff + .quad 0x72e302a348492870 + .quad 0x1253c19e53dbe1bc + + .quad 0x331d086e0d9abcaa + .quad 0x1e23c96d311a10c9 + .quad 0x96d0f95e58c13478 + .quad 0x2f72f7384fcfcc59 + .quad 0x39a6cd1cfd7d87c9 + .quad 0x9867a0abd8ae153a + .quad 0xa49d2a5f35986745 + .quad 0x57012940cdfe82e1 + .quad 0x5046a6532ec5544a + .quad 0x6d674004739ff6c9 + .quad 0x9bbaa44b234a70e3 + .quad 0x5e6d8901138cf386 + +// Precomputed table of multiples of generator for edwards25519 +// all in precomputed extended-projective (y-x,x+y,2*d*x*y) triples. + + // 2^4 * 1 * G + + .quad 0x7ec851ca553e2df3 + .quad 0xa71284cba64878b3 + .quad 0xe6b5e4193288d1e7 + .quad 0x4cf210ec5a9a8883 + .quad 0x322d04a52d9021f6 + .quad 0xb9c19f3375c6bf9c + .quad 0x587a3a4342d20b09 + .quad 0x143b1cf8aa64fe61 + .quad 0x9f867c7d968acaab + .quad 0x5f54258e27092729 + .quad 0xd0a7d34bea180975 + .quad 0x21b546a3374126e1 + + // 2^4 * 2 * G + + .quad 0xa94ff858a2888343 + .quad 0xce0ed4565313ed3c + .quad 0xf55c3dcfb5bf34fa + .quad 0x0a653ca5c9eab371 + .quad 0x490a7a45d185218f + .quad 0x9a15377846049335 + .quad 0x0060ea09cc31e1f6 + .quad 0x7e041577f86ee965 + .quad 0x66b2a496ce5b67f3 + .quad 0xff5492d8bd569796 + .quad 0x503cec294a592cd0 + .quad 0x566943650813acb2 + + // 2^4 * 3 * G + + .quad 0xb818db0c26620798 + .quad 0x5d5c31d9606e354a + .quad 0x0982fa4f00a8cdc7 + .quad 0x17e12bcd4653e2d4 + .quad 0x5672f9eb1dabb69d + .quad 0xba70b535afe853fc + .quad 0x47ac0f752796d66d + .quad 0x32a5351794117275 + .quad 0xd3a644a6df648437 + .quad 0x703b6559880fbfdd + .quad 0xcb852540ad3a1aa5 + .quad 0x0900b3f78e4c6468 + + // 2^4 * 4 * G + + .quad 0x0a851b9f679d651b + .quad 0xe108cb61033342f2 + .quad 0xd601f57fe88b30a3 + .quad 0x371f3acaed2dd714 + .quad 0xed280fbec816ad31 + .quad 0x52d9595bd8e6efe3 + .quad 0x0fe71772f6c623f5 + .quad 0x4314030b051e293c + .quad 0xd560005efbf0bcad + .quad 0x8eb70f2ed1870c5e + .quad 0x201f9033d084e6a0 + .quad 0x4c3a5ae1ce7b6670 + + // 2^4 * 5 * G + + .quad 0x4138a434dcb8fa95 + .quad 0x870cf67d6c96840b + .quad 0xde388574297be82c + .quad 0x7c814db27262a55a + .quad 0xbaf875e4c93da0dd + .quad 0xb93282a771b9294d + .quad 0x80d63fb7f4c6c460 + .quad 0x6de9c73dea66c181 + .quad 0x478904d5a04df8f2 + .quad 0xfafbae4ab10142d3 + .quad 0xf6c8ac63555d0998 + .quad 0x5aac4a412f90b104 + + // 2^4 * 6 * G + + .quad 0xc64f326b3ac92908 + .quad 0x5551b282e663e1e0 + .quad 0x476b35f54a1a4b83 + .quad 0x1b9da3fe189f68c2 + .quad 0x603a0d0abd7f5134 + .quad 0x8089c932e1d3ae46 + .quad 0xdf2591398798bd63 + .quad 0x1c145cd274ba0235 + .quad 0x32e8386475f3d743 + .quad 0x365b8baf6ae5d9ef + .quad 0x825238b6385b681e + .quad 0x234929c1167d65e1 + + // 2^4 * 7 * G + + .quad 0x984decaba077ade8 + .quad 0x383f77ad19eb389d + .quad 0xc7ec6b7e2954d794 + .quad 0x59c77b3aeb7c3a7a + .quad 0x48145cc21d099fcf + .quad 0x4535c192cc28d7e5 + .quad 0x80e7c1e548247e01 + .quad 0x4a5f28743b2973ee + .quad 0xd3add725225ccf62 + .quad 0x911a3381b2152c5d + .quad 0xd8b39fad5b08f87d + .quad 0x6f05606b4799fe3b + + // 2^4 * 8 * G + + .quad 0x9ffe9e92177ba962 + .quad 0x98aee71d0de5cae1 + .quad 0x3ff4ae942d831044 + .quad 0x714de12e58533ac8 + .quad 0x5b433149f91b6483 + .quad 0xadb5dc655a2cbf62 + .quad 0x87fa8412632827b3 + .quad 0x60895e91ab49f8d8 + .quad 0xe9ecf2ed0cf86c18 + .quad 0xb46d06120735dfd4 + .quad 0xbc9da09804b96be7 + .quad 0x73e2e62fd96dc26b + + // 2^8 * 1 * G + + .quad 0xed5b635449aa515e + .quad 0xa865c49f0bc6823a + .quad 0x850c1fe95b42d1c4 + .quad 0x30d76d6f03d315b9 + .quad 0x2eccdd0e632f9c1d + .quad 0x51d0b69676893115 + .quad 0x52dfb76ba8637a58 + .quad 0x6dd37d49a00eef39 + .quad 0x6c4444172106e4c7 + .quad 0xfb53d680928d7f69 + .quad 0xb4739ea4694d3f26 + .quad 0x10c697112e864bb0 + + // 2^8 * 2 * G + + .quad 0x6493c4277dbe5fde + .quad 0x265d4fad19ad7ea2 + .quad 0x0e00dfc846304590 + .quad 0x25e61cabed66fe09 + .quad 0x0ca62aa08358c805 + .quad 0x6a3d4ae37a204247 + .quad 0x7464d3a63b11eddc + .quad 0x03bf9baf550806ef + .quad 0x3f13e128cc586604 + .quad 0x6f5873ecb459747e + .quad 0xa0b63dedcc1268f5 + .quad 0x566d78634586e22c + + // 2^8 * 3 * G + + .quad 0x1637a49f9cc10834 + .quad 0xbc8e56d5a89bc451 + .quad 0x1cb5ec0f7f7fd2db + .quad 0x33975bca5ecc35d9 + .quad 0xa1054285c65a2fd0 + .quad 0x6c64112af31667c3 + .quad 0x680ae240731aee58 + .quad 0x14fba5f34793b22a + .quad 0x3cd746166985f7d4 + .quad 0x593e5e84c9c80057 + .quad 0x2fc3f2b67b61131e + .quad 0x14829cea83fc526c + + // 2^8 * 4 * G + + .quad 0xff437b8497dd95c2 + .quad 0x6c744e30aa4eb5a7 + .quad 0x9e0c5d613c85e88b + .quad 0x2fd9c71e5f758173 + .quad 0x21e70b2f4e71ecb8 + .quad 0xe656ddb940a477e3 + .quad 0xbf6556cece1d4f80 + .quad 0x05fc3bc4535d7b7e + .quad 0x24b8b3ae52afdedd + .quad 0x3495638ced3b30cf + .quad 0x33a4bc83a9be8195 + .quad 0x373767475c651f04 + + // 2^8 * 5 * G + + .quad 0x2fba99fd40d1add9 + .quad 0xb307166f96f4d027 + .quad 0x4363f05215f03bae + .quad 0x1fbea56c3b18f999 + .quad 0x634095cb14246590 + .quad 0xef12144016c15535 + .quad 0x9e38140c8910bc60 + .quad 0x6bf5905730907c8c + .quad 0x0fa778f1e1415b8a + .quad 0x06409ff7bac3a77e + .quad 0x6f52d7b89aa29a50 + .quad 0x02521cf67a635a56 + + // 2^8 * 6 * G + + .quad 0x513fee0b0a9d5294 + .quad 0x8f98e75c0fdf5a66 + .quad 0xd4618688bfe107ce + .quad 0x3fa00a7e71382ced + .quad 0xb1146720772f5ee4 + .quad 0xe8f894b196079ace + .quad 0x4af8224d00ac824a + .quad 0x001753d9f7cd6cc4 + .quad 0x3c69232d963ddb34 + .quad 0x1dde87dab4973858 + .quad 0xaad7d1f9a091f285 + .quad 0x12b5fe2fa048edb6 + + // 2^8 * 7 * G + + .quad 0x71f0fbc496fce34d + .quad 0x73b9826badf35bed + .quad 0xd2047261ff28c561 + .quad 0x749b76f96fb1206f + .quad 0xdf2b7c26ad6f1e92 + .quad 0x4b66d323504b8913 + .quad 0x8c409dc0751c8bc3 + .quad 0x6f7e93c20796c7b8 + .quad 0x1f5af604aea6ae05 + .quad 0xc12351f1bee49c99 + .quad 0x61a808b5eeff6b66 + .quad 0x0fcec10f01e02151 + + // 2^8 * 8 * G + + .quad 0x644d58a649fe1e44 + .quad 0x21fcaea231ad777e + .quad 0x02441c5a887fd0d2 + .quad 0x4901aa7183c511f3 + .quad 0x3df2d29dc4244e45 + .quad 0x2b020e7493d8de0a + .quad 0x6cc8067e820c214d + .quad 0x413779166feab90a + .quad 0x08b1b7548c1af8f0 + .quad 0xce0f7a7c246299b4 + .quad 0xf760b0f91e06d939 + .quad 0x41bb887b726d1213 + + // 2^12 * 1 * G + + .quad 0x9267806c567c49d8 + .quad 0x066d04ccca791e6a + .quad 0xa69f5645e3cc394b + .quad 0x5c95b686a0788cd2 + .quad 0x97d980e0aa39f7d2 + .quad 0x35d0384252c6b51c + .quad 0x7d43f49307cd55aa + .quad 0x56bd36cfb78ac362 + .quad 0x2ac519c10d14a954 + .quad 0xeaf474b494b5fa90 + .quad 0xe6af8382a9f87a5a + .quad 0x0dea6db1879be094 + + // 2^12 * 2 * G + + .quad 0xaa66bf547344e5ab + .quad 0xda1258888f1b4309 + .quad 0x5e87d2b3fd564b2f + .quad 0x5b2c78885483b1dd + .quad 0x15baeb74d6a8797a + .quad 0x7ef55cf1fac41732 + .quad 0x29001f5a3c8b05c5 + .quad 0x0ad7cc8752eaccfb + .quad 0x52151362793408cf + .quad 0xeb0f170319963d94 + .quad 0xa833b2fa883d9466 + .quad 0x093a7fa775003c78 + + // 2^12 * 3 * G + + .quad 0xe5107de63a16d7be + .quad 0xa377ffdc9af332cf + .quad 0x70d5bf18440b677f + .quad 0x6a252b19a4a31403 + .quad 0xb8e9604460a91286 + .quad 0x7f3fd8047778d3de + .quad 0x67d01e31bf8a5e2d + .quad 0x7b038a06c27b653e + .quad 0x9ed919d5d36990f3 + .quad 0x5213aebbdb4eb9f2 + .quad 0xc708ea054cb99135 + .quad 0x58ded57f72260e56 + + // 2^12 * 4 * G + + .quad 0x78e79dade9413d77 + .quad 0xf257f9d59729e67d + .quad 0x59db910ee37aa7e6 + .quad 0x6aa11b5bbb9e039c + .quad 0xda6d53265b0fd48b + .quad 0x8960823193bfa988 + .quad 0xd78ac93261d57e28 + .quad 0x79f2942d3a5c8143 + .quad 0x97da2f25b6c88de9 + .quad 0x251ba7eaacf20169 + .quad 0x09b44f87ef4eb4e4 + .quad 0x7d90ab1bbc6a7da5 + + // 2^12 * 5 * G + + .quad 0x9acca683a7016bfe + .quad 0x90505f4df2c50b6d + .quad 0x6b610d5fcce435aa + .quad 0x19a10d446198ff96 + .quad 0x1a07a3f496b3c397 + .quad 0x11ceaa188f4e2532 + .quad 0x7d9498d5a7751bf0 + .quad 0x19ed161f508dd8a0 + .quad 0x560a2cd687dce6ca + .quad 0x7f3568c48664cf4d + .quad 0x8741e95222803a38 + .quad 0x483bdab1595653fc + + // 2^12 * 6 * G + + .quad 0xfa780f148734fa49 + .quad 0x106f0b70360534e0 + .quad 0x2210776fe3e307bd + .quad 0x3286c109dde6a0fe + .quad 0xd6cf4d0ab4da80f6 + .quad 0x82483e45f8307fe0 + .quad 0x05005269ae6f9da4 + .quad 0x1c7052909cf7877a + .quad 0x32ee7de2874e98d4 + .quad 0x14c362e9b97e0c60 + .quad 0x5781dcde6a60a38a + .quad 0x217dd5eaaa7aa840 + + // 2^12 * 7 * G + + .quad 0x9db7c4d0248e1eb0 + .quad 0xe07697e14d74bf52 + .quad 0x1e6a9b173c562354 + .quad 0x7fa7c21f795a4965 + .quad 0x8bdf1fb9be8c0ec8 + .quad 0x00bae7f8e30a0282 + .quad 0x4963991dad6c4f6c + .quad 0x07058a6e5df6f60a + .quad 0xe9eb02c4db31f67f + .quad 0xed25fd8910bcfb2b + .quad 0x46c8131f5c5cddb4 + .quad 0x33b21c13a0cb9bce + + // 2^12 * 8 * G + + .quad 0x360692f8087d8e31 + .quad 0xf4dcc637d27163f7 + .quad 0x25a4e62065ea5963 + .quad 0x659bf72e5ac160d9 + .quad 0x9aafb9b05ee38c5b + .quad 0xbf9d2d4e071a13c7 + .quad 0x8eee6e6de933290a + .quad 0x1c3bab17ae109717 + .quad 0x1c9ab216c7cab7b0 + .quad 0x7d65d37407bbc3cc + .quad 0x52744750504a58d5 + .quad 0x09f2606b131a2990 + + // 2^16 * 1 * G + + .quad 0x40e87d44744346be + .quad 0x1d48dad415b52b25 + .quad 0x7c3a8a18a13b603e + .quad 0x4eb728c12fcdbdf7 + .quad 0x7e234c597c6691ae + .quad 0x64889d3d0a85b4c8 + .quad 0xdae2c90c354afae7 + .quad 0x0a871e070c6a9e1d + .quad 0x3301b5994bbc8989 + .quad 0x736bae3a5bdd4260 + .quad 0x0d61ade219d59e3c + .quad 0x3ee7300f2685d464 + + // 2^16 * 2 * G + + .quad 0xf5d255e49e7dd6b7 + .quad 0x8016115c610b1eac + .quad 0x3c99975d92e187ca + .quad 0x13815762979125c2 + .quad 0x43fa7947841e7518 + .quad 0xe5c6fa59639c46d7 + .quad 0xa1065e1de3052b74 + .quad 0x7d47c6a2cfb89030 + .quad 0x3fdad0148ef0d6e0 + .quad 0x9d3e749a91546f3c + .quad 0x71ec621026bb8157 + .quad 0x148cf58d34c9ec80 + + // 2^16 * 3 * G + + .quad 0x46a492f67934f027 + .quad 0x469984bef6840aa9 + .quad 0x5ca1bc2a89611854 + .quad 0x3ff2fa1ebd5dbbd4 + .quad 0xe2572f7d9ae4756d + .quad 0x56c345bb88f3487f + .quad 0x9fd10b6d6960a88d + .quad 0x278febad4eaea1b9 + .quad 0xb1aa681f8c933966 + .quad 0x8c21949c20290c98 + .quad 0x39115291219d3c52 + .quad 0x4104dd02fe9c677b + + // 2^16 * 4 * G + + .quad 0x72b2bf5e1124422a + .quad 0xa1fa0c3398a33ab5 + .quad 0x94cb6101fa52b666 + .quad 0x2c863b00afaf53d5 + .quad 0x81214e06db096ab8 + .quad 0x21a8b6c90ce44f35 + .quad 0x6524c12a409e2af5 + .quad 0x0165b5a48efca481 + .quad 0xf190a474a0846a76 + .quad 0x12eff984cd2f7cc0 + .quad 0x695e290658aa2b8f + .quad 0x591b67d9bffec8b8 + + // 2^16 * 5 * G + + .quad 0x312f0d1c80b49bfa + .quad 0x5979515eabf3ec8a + .quad 0x727033c09ef01c88 + .quad 0x3de02ec7ca8f7bcb + .quad 0x99b9b3719f18b55d + .quad 0xe465e5faa18c641e + .quad 0x61081136c29f05ed + .quad 0x489b4f867030128b + .quad 0xd232102d3aeb92ef + .quad 0xe16253b46116a861 + .quad 0x3d7eabe7190baa24 + .quad 0x49f5fbba496cbebf + + // 2^16 * 6 * G + + .quad 0x30949a108a5bcfd4 + .quad 0xdc40dd70bc6473eb + .quad 0x92c294c1307c0d1c + .quad 0x5604a86dcbfa6e74 + .quad 0x155d628c1e9c572e + .quad 0x8a4d86acc5884741 + .quad 0x91a352f6515763eb + .quad 0x06a1a6c28867515b + .quad 0x7288d1d47c1764b6 + .quad 0x72541140e0418b51 + .quad 0x9f031a6018acf6d1 + .quad 0x20989e89fe2742c6 + + // 2^16 * 7 * G + + .quad 0x499777fd3a2dcc7f + .quad 0x32857c2ca54fd892 + .quad 0xa279d864d207e3a0 + .quad 0x0403ed1d0ca67e29 + .quad 0x1674278b85eaec2e + .quad 0x5621dc077acb2bdf + .quad 0x640a4c1661cbf45a + .quad 0x730b9950f70595d3 + .quad 0xc94b2d35874ec552 + .quad 0xc5e6c8cf98246f8d + .quad 0xf7cb46fa16c035ce + .quad 0x5bd7454308303dcc + + // 2^16 * 8 * G + + .quad 0x7f9ad19528b24cc2 + .quad 0x7f6b54656335c181 + .quad 0x66b8b66e4fc07236 + .quad 0x133a78007380ad83 + .quad 0x85c4932115e7792a + .quad 0xc64c89a2bdcdddc9 + .quad 0x9d1e3da8ada3d762 + .quad 0x5bb7db123067f82c + .quad 0x0961f467c6ca62be + .quad 0x04ec21d6211952ee + .quad 0x182360779bd54770 + .quad 0x740dca6d58f0e0d2 + + // 2^20 * 1 * G + + .quad 0x50b70bf5d3f0af0b + .quad 0x4feaf48ae32e71f7 + .quad 0x60e84ed3a55bbd34 + .quad 0x00ed489b3f50d1ed + .quad 0x3906c72aed261ae5 + .quad 0x9ab68fd988e100f7 + .quad 0xf5e9059af3360197 + .quad 0x0e53dc78bf2b6d47 + .quad 0xb90829bf7971877a + .quad 0x5e4444636d17e631 + .quad 0x4d05c52e18276893 + .quad 0x27632d9a5a4a4af5 + + // 2^20 * 2 * G + + .quad 0xd11ff05154b260ce + .quad 0xd86dc38e72f95270 + .quad 0x601fcd0d267cc138 + .quad 0x2b67916429e90ccd + .quad 0xa98285d187eaffdb + .quad 0xa5b4fbbbd8d0a864 + .quad 0xb658f27f022663f7 + .quad 0x3bbc2b22d99ce282 + .quad 0xb917c952583c0a58 + .quad 0x653ff9b80fe4c6f3 + .quad 0x9b0da7d7bcdf3c0c + .quad 0x43a0eeb6ab54d60e + + // 2^20 * 3 * G + + .quad 0x396966a46d4a5487 + .quad 0xf811a18aac2bb3ba + .quad 0x66e4685b5628b26b + .quad 0x70a477029d929b92 + .quad 0x3ac6322357875fe8 + .quad 0xd9d4f4ecf5fbcb8f + .quad 0x8dee8493382bb620 + .quad 0x50c5eaa14c799fdc + .quad 0xdd0edc8bd6f2fb3c + .quad 0x54c63aa79cc7b7a0 + .quad 0xae0b032b2c8d9f1a + .quad 0x6f9ce107602967fb + + // 2^20 * 4 * G + + .quad 0xad1054b1cde1c22a + .quad 0xc4a8e90248eb32df + .quad 0x5f3e7b33accdc0ea + .quad 0x72364713fc79963e + .quad 0x139693063520e0b5 + .quad 0x437fcf7c88ea03fe + .quad 0xf7d4c40bd3c959bc + .quad 0x699154d1f893ded9 + .quad 0x315d5c75b4b27526 + .quad 0xcccb842d0236daa5 + .quad 0x22f0c8a3345fee8e + .quad 0x73975a617d39dbed + + // 2^20 * 5 * G + + .quad 0xe4024df96375da10 + .quad 0x78d3251a1830c870 + .quad 0x902b1948658cd91c + .quad 0x7e18b10b29b7438a + .quad 0x6f37f392f4433e46 + .quad 0x0e19b9a11f566b18 + .quad 0x220fb78a1fd1d662 + .quad 0x362a4258a381c94d + .quad 0x9071d9132b6beb2f + .quad 0x0f26e9ad28418247 + .quad 0xeab91ec9bdec925d + .quad 0x4be65bc8f48af2de + + // 2^20 * 6 * G + + .quad 0x78487feba36e7028 + .quad 0x5f3f13001dd8ce34 + .quad 0x934fb12d4b30c489 + .quad 0x056c244d397f0a2b + .quad 0x1d50fba257c26234 + .quad 0x7bd4823adeb0678b + .quad 0xc2b0dc6ea6538af5 + .quad 0x5665eec6351da73e + .quad 0xdb3ee00943bfb210 + .quad 0x4972018720800ac2 + .quad 0x26ab5d6173bd8667 + .quad 0x20b209c2ab204938 + + // 2^20 * 7 * G + + .quad 0x549e342ac07fb34b + .quad 0x02d8220821373d93 + .quad 0xbc262d70acd1f567 + .quad 0x7a92c9fdfbcac784 + .quad 0x1fcca94516bd3289 + .quad 0x448d65aa41420428 + .quad 0x59c3b7b216a55d62 + .quad 0x49992cc64e612cd8 + .quad 0x65bd1bea70f801de + .quad 0x1befb7c0fe49e28a + .quad 0xa86306cdb1b2ae4a + .quad 0x3b7ac0cd265c2a09 + + // 2^20 * 8 * G + + .quad 0x822bee438c01bcec + .quad 0x530cb525c0fbc73b + .quad 0x48519034c1953fe9 + .quad 0x265cc261e09a0f5b + .quad 0xf0d54e4f22ed39a7 + .quad 0xa2aae91e5608150a + .quad 0xf421b2e9eddae875 + .quad 0x31bc531d6b7de992 + .quad 0xdf3d134da980f971 + .quad 0x7a4fb8d1221a22a7 + .quad 0x3df7d42035aad6d8 + .quad 0x2a14edcc6a1a125e + + // 2^24 * 1 * G + + .quad 0xdf48ee0752cfce4e + .quad 0xc3fffaf306ec08b7 + .quad 0x05710b2ab95459c4 + .quad 0x161d25fa963ea38d + .quad 0x231a8c570478433c + .quad 0xb7b5270ec281439d + .quad 0xdbaa99eae3d9079f + .quad 0x2c03f5256c2b03d9 + .quad 0x790f18757b53a47d + .quad 0x307b0130cf0c5879 + .quad 0x31903d77257ef7f9 + .quad 0x699468bdbd96bbaf + + // 2^24 * 2 * G + + .quad 0xbd1f2f46f4dafecf + .quad 0x7cef0114a47fd6f7 + .quad 0xd31ffdda4a47b37f + .quad 0x525219a473905785 + .quad 0xd8dd3de66aa91948 + .quad 0x485064c22fc0d2cc + .quad 0x9b48246634fdea2f + .quad 0x293e1c4e6c4a2e3a + .quad 0x376e134b925112e1 + .quad 0x703778b5dca15da0 + .quad 0xb04589af461c3111 + .quad 0x5b605c447f032823 + + // 2^24 * 3 * G + + .quad 0xb965805920c47c89 + .quad 0xe7f0100c923b8fcc + .quad 0x0001256502e2ef77 + .quad 0x24a76dcea8aeb3ee + .quad 0x3be9fec6f0e7f04c + .quad 0x866a579e75e34962 + .quad 0x5542ef161e1de61a + .quad 0x2f12fef4cc5abdd5 + .quad 0x0a4522b2dfc0c740 + .quad 0x10d06e7f40c9a407 + .quad 0xc6cf144178cff668 + .quad 0x5e607b2518a43790 + + // 2^24 * 4 * G + + .quad 0x58b31d8f6cdf1818 + .quad 0x35cfa74fc36258a2 + .quad 0xe1b3ff4f66e61d6e + .quad 0x5067acab6ccdd5f7 + .quad 0xa02c431ca596cf14 + .quad 0xe3c42d40aed3e400 + .quad 0xd24526802e0f26db + .quad 0x201f33139e457068 + .quad 0xfd527f6b08039d51 + .quad 0x18b14964017c0006 + .quad 0xd5220eb02e25a4a8 + .quad 0x397cba8862460375 + + // 2^24 * 5 * G + + .quad 0x30c13093f05959b2 + .quad 0xe23aa18de9a97976 + .quad 0x222fd491721d5e26 + .quad 0x2339d320766e6c3a + .quad 0x7815c3fbc81379e7 + .quad 0xa6619420dde12af1 + .quad 0xffa9c0f885a8fdd5 + .quad 0x771b4022c1e1c252 + .quad 0xd87dd986513a2fa7 + .quad 0xf5ac9b71f9d4cf08 + .quad 0xd06bc31b1ea283b3 + .quad 0x331a189219971a76 + + // 2^24 * 6 * G + + .quad 0xf5166f45fb4f80c6 + .quad 0x9c36c7de61c775cf + .quad 0xe3d4e81b9041d91c + .quad 0x31167c6b83bdfe21 + .quad 0x26512f3a9d7572af + .quad 0x5bcbe28868074a9e + .quad 0x84edc1c11180f7c4 + .quad 0x1ac9619ff649a67b + .quad 0xf22b3842524b1068 + .quad 0x5068343bee9ce987 + .quad 0xfc9d71844a6250c8 + .quad 0x612436341f08b111 + + // 2^24 * 7 * G + + .quad 0xd99d41db874e898d + .quad 0x09fea5f16c07dc20 + .quad 0x793d2c67d00f9bbc + .quad 0x46ebe2309e5eff40 + .quad 0x8b6349e31a2d2638 + .quad 0x9ddfb7009bd3fd35 + .quad 0x7f8bf1b8a3a06ba4 + .quad 0x1522aa3178d90445 + .quad 0x2c382f5369614938 + .quad 0xdafe409ab72d6d10 + .quad 0xe8c83391b646f227 + .quad 0x45fe70f50524306c + + // 2^24 * 8 * G + + .quad 0xda4875a6960c0b8c + .quad 0x5b68d076ef0e2f20 + .quad 0x07fb51cf3d0b8fd4 + .quad 0x428d1623a0e392d4 + .quad 0x62f24920c8951491 + .quad 0x05f007c83f630ca2 + .quad 0x6fbb45d2f5c9d4b8 + .quad 0x16619f6db57a2245 + .quad 0x084f4a4401a308fd + .quad 0xa82219c376a5caac + .quad 0xdeb8de4643d1bc7d + .quad 0x1d81592d60bd38c6 + + // 2^28 * 1 * G + + .quad 0xd833d7beec2a4c38 + .quad 0x2c9162830acc20ed + .quad 0xe93a47aa92df7581 + .quad 0x702d67a3333c4a81 + .quad 0x3a4a369a2f89c8a1 + .quad 0x63137a1d7c8de80d + .quad 0xbcac008a78eda015 + .quad 0x2cb8b3a5b483b03f + .quad 0x36e417cbcb1b90a1 + .quad 0x33b3ddaa7f11794e + .quad 0x3f510808885bc607 + .quad 0x24141dc0e6a8020d + + // 2^28 * 2 * G + + .quad 0x59f73c773fefee9d + .quad 0xb3f1ef89c1cf989d + .quad 0xe35dfb42e02e545f + .quad 0x5766120b47a1b47c + .quad 0x91925dccbd83157d + .quad 0x3ca1205322cc8094 + .quad 0x28e57f183f90d6e4 + .quad 0x1a4714cede2e767b + .quad 0xdb20ba0fb8b6b7ff + .quad 0xb732c3b677511fa1 + .quad 0xa92b51c099f02d89 + .quad 0x4f3875ad489ca5f1 + + // 2^28 * 3 * G + + .quad 0xc7fc762f4932ab22 + .quad 0x7ac0edf72f4c3c1b + .quad 0x5f6b55aa9aa895e8 + .quad 0x3680274dad0a0081 + .quad 0x79ed13f6ee73eec0 + .quad 0xa5c6526d69110bb1 + .quad 0xe48928c38603860c + .quad 0x722a1446fd7059f5 + .quad 0xd0959fe9a8cf8819 + .quad 0xd0a995508475a99c + .quad 0x6eac173320b09cc5 + .quad 0x628ecf04331b1095 + + // 2^28 * 4 * G + + .quad 0x98bcb118a9d0ddbc + .quad 0xee449e3408b4802b + .quad 0x87089226b8a6b104 + .quad 0x685f349a45c7915d + .quad 0x9b41acf85c74ccf1 + .quad 0xb673318108265251 + .quad 0x99c92aed11adb147 + .quad 0x7a47d70d34ecb40f + .quad 0x60a0c4cbcc43a4f5 + .quad 0x775c66ca3677bea9 + .quad 0xa17aa1752ff8f5ed + .quad 0x11ded9020e01fdc0 + + // 2^28 * 5 * G + + .quad 0x890e7809caefe704 + .quad 0x8728296de30e8c6c + .quad 0x4c5cd2a392aeb1c9 + .quad 0x194263d15771531f + .quad 0x471f95b03bea93b7 + .quad 0x0552d7d43313abd3 + .quad 0xbd9370e2e17e3f7b + .quad 0x7b120f1db20e5bec + .quad 0x17d2fb3d86502d7a + .quad 0xb564d84450a69352 + .quad 0x7da962c8a60ed75d + .quad 0x00d0f85b318736aa + + // 2^28 * 6 * G + + .quad 0x978b142e777c84fd + .quad 0xf402644705a8c062 + .quad 0xa67ad51be7e612c7 + .quad 0x2f7b459698dd6a33 + .quad 0xa6753c1efd7621c1 + .quad 0x69c0b4a7445671f5 + .quad 0x971f527405b23c11 + .quad 0x387bc74851a8c7cd + .quad 0x81894b4d4a52a9a8 + .quad 0xadd93e12f6b8832f + .quad 0x184d8548b61bd638 + .quad 0x3f1c62dbd6c9f6cd + + // 2^28 * 7 * G + + .quad 0x2e8f1f0091910c1f + .quad 0xa4df4fe0bff2e12c + .quad 0x60c6560aee927438 + .quad 0x6338283facefc8fa + .quad 0x3fad3e40148f693d + .quad 0x052656e194eb9a72 + .quad 0x2f4dcbfd184f4e2f + .quad 0x406f8db1c482e18b + .quad 0x9e630d2c7f191ee4 + .quad 0x4fbf8301bc3ff670 + .quad 0x787d8e4e7afb73c4 + .quad 0x50d83d5be8f58fa5 + + // 2^28 * 8 * G + + .quad 0x85683916c11a1897 + .quad 0x2d69a4efe506d008 + .quad 0x39af1378f664bd01 + .quad 0x65942131361517c6 + .quad 0xc0accf90b4d3b66d + .quad 0xa7059de561732e60 + .quad 0x033d1f7870c6b0ba + .quad 0x584161cd26d946e4 + .quad 0xbbf2b1a072d27ca2 + .quad 0xbf393c59fbdec704 + .quad 0xe98dbbcee262b81e + .quad 0x02eebd0b3029b589 + + // 2^32 * 1 * G + + .quad 0x61368756a60dac5f + .quad 0x17e02f6aebabdc57 + .quad 0x7f193f2d4cce0f7d + .quad 0x20234a7789ecdcf0 + .quad 0x8765b69f7b85c5e8 + .quad 0x6ff0678bd168bab2 + .quad 0x3a70e77c1d330f9b + .quad 0x3a5f6d51b0af8e7c + .quad 0x76d20db67178b252 + .quad 0x071c34f9d51ed160 + .quad 0xf62a4a20b3e41170 + .quad 0x7cd682353cffe366 + + // 2^32 * 2 * G + + .quad 0x0be1a45bd887fab6 + .quad 0x2a846a32ba403b6e + .quad 0xd9921012e96e6000 + .quad 0x2838c8863bdc0943 + .quad 0xa665cd6068acf4f3 + .quad 0x42d92d183cd7e3d3 + .quad 0x5759389d336025d9 + .quad 0x3ef0253b2b2cd8ff + .quad 0xd16bb0cf4a465030 + .quad 0xfa496b4115c577ab + .quad 0x82cfae8af4ab419d + .quad 0x21dcb8a606a82812 + + // 2^32 * 3 * G + + .quad 0x5c6004468c9d9fc8 + .quad 0x2540096ed42aa3cb + .quad 0x125b4d4c12ee2f9c + .quad 0x0bc3d08194a31dab + .quad 0x9a8d00fabe7731ba + .quad 0x8203607e629e1889 + .quad 0xb2cc023743f3d97f + .quad 0x5d840dbf6c6f678b + .quad 0x706e380d309fe18b + .quad 0x6eb02da6b9e165c7 + .quad 0x57bbba997dae20ab + .quad 0x3a4276232ac196dd + + // 2^32 * 4 * G + + .quad 0x4b42432c8a7084fa + .quad 0x898a19e3dfb9e545 + .quad 0xbe9f00219c58e45d + .quad 0x1ff177cea16debd1 + .quad 0x3bf8c172db447ecb + .quad 0x5fcfc41fc6282dbd + .quad 0x80acffc075aa15fe + .quad 0x0770c9e824e1a9f9 + .quad 0xcf61d99a45b5b5fd + .quad 0x860984e91b3a7924 + .quad 0xe7300919303e3e89 + .quad 0x39f264fd41500b1e + + // 2^32 * 5 * G + + .quad 0xa7ad3417dbe7e29c + .quad 0xbd94376a2b9c139c + .quad 0xa0e91b8e93597ba9 + .quad 0x1712d73468889840 + .quad 0xd19b4aabfe097be1 + .quad 0xa46dfce1dfe01929 + .quad 0xc3c908942ca6f1ff + .quad 0x65c621272c35f14e + .quad 0xe72b89f8ce3193dd + .quad 0x4d103356a125c0bb + .quad 0x0419a93d2e1cfe83 + .quad 0x22f9800ab19ce272 + + // 2^32 * 6 * G + + .quad 0x605a368a3e9ef8cb + .quad 0xe3e9c022a5504715 + .quad 0x553d48b05f24248f + .quad 0x13f416cd647626e5 + .quad 0x42029fdd9a6efdac + .quad 0xb912cebe34a54941 + .quad 0x640f64b987bdf37b + .quad 0x4171a4d38598cab4 + .quad 0xfa2758aa99c94c8c + .quad 0x23006f6fb000b807 + .quad 0xfbd291ddadda5392 + .quad 0x508214fa574bd1ab + + // 2^32 * 7 * G + + .quad 0xc20269153ed6fe4b + .quad 0xa65a6739511d77c4 + .quad 0xcbde26462c14af94 + .quad 0x22f960ec6faba74b + .quad 0x461a15bb53d003d6 + .quad 0xb2102888bcf3c965 + .quad 0x27c576756c683a5a + .quad 0x3a7758a4c86cb447 + .quad 0x548111f693ae5076 + .quad 0x1dae21df1dfd54a6 + .quad 0x12248c90f3115e65 + .quad 0x5d9fd15f8de7f494 + + // 2^32 * 8 * G + + .quad 0x031408d36d63727f + .quad 0x6a379aefd7c7b533 + .quad 0xa9e18fc5ccaee24b + .quad 0x332f35914f8fbed3 + .quad 0x3f244d2aeed7521e + .quad 0x8e3a9028432e9615 + .quad 0xe164ba772e9c16d4 + .quad 0x3bc187fa47eb98d8 + .quad 0x6d470115ea86c20c + .quad 0x998ab7cb6c46d125 + .quad 0xd77832b53a660188 + .quad 0x450d81ce906fba03 + + // 2^36 * 1 * G + + .quad 0xf8ae4d2ad8453902 + .quad 0x7018058ee8db2d1d + .quad 0xaab3995fc7d2c11e + .quad 0x53b16d2324ccca79 + .quad 0x23264d66b2cae0b5 + .quad 0x7dbaed33ebca6576 + .quad 0x030ebed6f0d24ac8 + .quad 0x2a887f78f7635510 + .quad 0x2a23b9e75c012d4f + .quad 0x0c974651cae1f2ea + .quad 0x2fb63273675d70ca + .quad 0x0ba7250b864403f5 + + // 2^36 * 2 * G + + .quad 0xbb0d18fd029c6421 + .quad 0xbc2d142189298f02 + .quad 0x8347f8e68b250e96 + .quad 0x7b9f2fe8032d71c9 + .quad 0xdd63589386f86d9c + .quad 0x61699176e13a85a4 + .quad 0x2e5111954eaa7d57 + .quad 0x32c21b57fb60bdfb + .quad 0xd87823cd319e0780 + .quad 0xefc4cfc1897775c5 + .quad 0x4854fb129a0ab3f7 + .quad 0x12c49d417238c371 + + // 2^36 * 3 * G + + .quad 0x0950b533ffe83769 + .quad 0x21861c1d8e1d6bd1 + .quad 0xf022d8381302e510 + .quad 0x2509200c6391cab4 + .quad 0x09b3a01783799542 + .quad 0x626dd08faad5ee3f + .quad 0xba00bceeeb70149f + .quad 0x1421b246a0a444c9 + .quad 0x4aa43a8e8c24a7c7 + .quad 0x04c1f540d8f05ef5 + .quad 0xadba5e0c0b3eb9dc + .quad 0x2ab5504448a49ce3 + + // 2^36 * 4 * G + + .quad 0x2ed227266f0f5dec + .quad 0x9824ee415ed50824 + .quad 0x807bec7c9468d415 + .quad 0x7093bae1b521e23f + .quad 0xdc07ac631c5d3afa + .quad 0x58615171f9df8c6c + .quad 0x72a079d89d73e2b0 + .quad 0x7301f4ceb4eae15d + .quad 0x6409e759d6722c41 + .quad 0xa674e1cf72bf729b + .quad 0xbc0a24eb3c21e569 + .quad 0x390167d24ebacb23 + + // 2^36 * 5 * G + + .quad 0x27f58e3bba353f1c + .quad 0x4c47764dbf6a4361 + .quad 0xafbbc4e56e562650 + .quad 0x07db2ee6aae1a45d + .quad 0xd7bb054ba2f2120b + .quad 0xe2b9ceaeb10589b7 + .quad 0x3fe8bac8f3c0edbe + .quad 0x4cbd40767112cb69 + .quad 0x0b603cc029c58176 + .quad 0x5988e3825cb15d61 + .quad 0x2bb61413dcf0ad8d + .quad 0x7b8eec6c74183287 + + // 2^36 * 6 * G + + .quad 0xe4ca40782cd27cb0 + .quad 0xdaf9c323fbe967bd + .quad 0xb29bd34a8ad41e9e + .quad 0x72810497626ede4d + .quad 0x32fee570fc386b73 + .quad 0xda8b0141da3a8cc7 + .quad 0x975ffd0ac8968359 + .quad 0x6ee809a1b132a855 + .quad 0x9444bb31fcfd863a + .quad 0x2fe3690a3e4e48c5 + .quad 0xdc29c867d088fa25 + .quad 0x13bd1e38d173292e + + // 2^36 * 7 * G + + .quad 0xd32b4cd8696149b5 + .quad 0xe55937d781d8aab7 + .quad 0x0bcb2127ae122b94 + .quad 0x41e86fcfb14099b0 + .quad 0x223fb5cf1dfac521 + .quad 0x325c25316f554450 + .quad 0x030b98d7659177ac + .quad 0x1ed018b64f88a4bd + .quad 0x3630dfa1b802a6b0 + .quad 0x880f874742ad3bd5 + .quad 0x0af90d6ceec5a4d4 + .quad 0x746a247a37cdc5d9 + + // 2^36 * 8 * G + + .quad 0xd531b8bd2b7b9af6 + .quad 0x5005093537fc5b51 + .quad 0x232fcf25c593546d + .quad 0x20a365142bb40f49 + .quad 0x6eccd85278d941ed + .quad 0x2254ae83d22f7843 + .quad 0xc522d02e7bbfcdb7 + .quad 0x681e3351bff0e4e2 + .quad 0x8b64b59d83034f45 + .quad 0x2f8b71f21fa20efb + .quad 0x69249495ba6550e4 + .quad 0x539ef98e45d5472b + + // 2^40 * 1 * G + + .quad 0x6e7bb6a1a6205275 + .quad 0xaa4f21d7413c8e83 + .quad 0x6f56d155e88f5cb2 + .quad 0x2de25d4ba6345be1 + .quad 0xd074d8961cae743f + .quad 0xf86d18f5ee1c63ed + .quad 0x97bdc55be7f4ed29 + .quad 0x4cbad279663ab108 + .quad 0x80d19024a0d71fcd + .quad 0xc525c20afb288af8 + .quad 0xb1a3974b5f3a6419 + .quad 0x7d7fbcefe2007233 + + // 2^40 * 2 * G + + .quad 0xfaef1e6a266b2801 + .quad 0x866c68c4d5739f16 + .quad 0xf68a2fbc1b03762c + .quad 0x5975435e87b75a8d + .quad 0xcd7c5dc5f3c29094 + .quad 0xc781a29a2a9105ab + .quad 0x80c61d36421c3058 + .quad 0x4f9cd196dcd8d4d7 + .quad 0x199297d86a7b3768 + .quad 0xd0d058241ad17a63 + .quad 0xba029cad5c1c0c17 + .quad 0x7ccdd084387a0307 + + // 2^40 * 3 * G + + .quad 0xdca6422c6d260417 + .quad 0xae153d50948240bd + .quad 0xa9c0c1b4fb68c677 + .quad 0x428bd0ed61d0cf53 + .quad 0x9b0c84186760cc93 + .quad 0xcdae007a1ab32a99 + .quad 0xa88dec86620bda18 + .quad 0x3593ca848190ca44 + .quad 0x9213189a5e849aa7 + .quad 0xd4d8c33565d8facd + .quad 0x8c52545b53fdbbd1 + .quad 0x27398308da2d63e6 + + // 2^40 * 4 * G + + .quad 0x42c38d28435ed413 + .quad 0xbd50f3603278ccc9 + .quad 0xbb07ab1a79da03ef + .quad 0x269597aebe8c3355 + .quad 0xb9a10e4c0a702453 + .quad 0x0fa25866d57d1bde + .quad 0xffb9d9b5cd27daf7 + .quad 0x572c2945492c33fd + .quad 0xc77fc745d6cd30be + .quad 0xe4dfe8d3e3baaefb + .quad 0xa22c8830aa5dda0c + .quad 0x7f985498c05bca80 + + // 2^40 * 5 * G + + .quad 0x3849ce889f0be117 + .quad 0x8005ad1b7b54a288 + .quad 0x3da3c39f23fc921c + .quad 0x76c2ec470a31f304 + .quad 0xd35615520fbf6363 + .quad 0x08045a45cf4dfba6 + .quad 0xeec24fbc873fa0c2 + .quad 0x30f2653cd69b12e7 + .quad 0x8a08c938aac10c85 + .quad 0x46179b60db276bcb + .quad 0xa920c01e0e6fac70 + .quad 0x2f1273f1596473da + + // 2^40 * 6 * G + + .quad 0x4739fc7c8ae01e11 + .quad 0xfd5274904a6aab9f + .quad 0x41d98a8287728f2e + .quad 0x5d9e572ad85b69f2 + .quad 0x30488bd755a70bc0 + .quad 0x06d6b5a4f1d442e7 + .quad 0xead1a69ebc596162 + .quad 0x38ac1997edc5f784 + .quad 0x0666b517a751b13b + .quad 0x747d06867e9b858c + .quad 0xacacc011454dde49 + .quad 0x22dfcd9cbfe9e69c + + // 2^40 * 7 * G + + .quad 0x8ddbd2e0c30d0cd9 + .quad 0xad8e665facbb4333 + .quad 0x8f6b258c322a961f + .quad 0x6b2916c05448c1c7 + .quad 0x56ec59b4103be0a1 + .quad 0x2ee3baecd259f969 + .quad 0x797cb29413f5cd32 + .quad 0x0fe9877824cde472 + .quad 0x7edb34d10aba913b + .quad 0x4ea3cd822e6dac0e + .quad 0x66083dff6578f815 + .quad 0x4c303f307ff00a17 + + // 2^40 * 8 * G + + .quad 0xd30a3bd617b28c85 + .quad 0xc5d377b739773bea + .quad 0xc6c6e78c1e6a5cbf + .quad 0x0d61b8f78b2ab7c4 + .quad 0x29fc03580dd94500 + .quad 0xecd27aa46fbbec93 + .quad 0x130a155fc2e2a7f8 + .quad 0x416b151ab706a1d5 + .quad 0x56a8d7efe9c136b0 + .quad 0xbd07e5cd58e44b20 + .quad 0xafe62fda1b57e0ab + .quad 0x191a2af74277e8d2 + + // 2^44 * 1 * G + + .quad 0xd550095bab6f4985 + .quad 0x04f4cd5b4fbfaf1a + .quad 0x9d8e2ed12a0c7540 + .quad 0x2bc24e04b2212286 + .quad 0x09d4b60b2fe09a14 + .quad 0xc384f0afdbb1747e + .quad 0x58e2ea8978b5fd6e + .quad 0x519ef577b5e09b0a + .quad 0x1863d7d91124cca9 + .quad 0x7ac08145b88a708e + .quad 0x2bcd7309857031f5 + .quad 0x62337a6e8ab8fae5 + + // 2^44 * 2 * G + + .quad 0x4bcef17f06ffca16 + .quad 0xde06e1db692ae16a + .quad 0x0753702d614f42b0 + .quad 0x5f6041b45b9212d0 + .quad 0xd1ab324e1b3a1273 + .quad 0x18947cf181055340 + .quad 0x3b5d9567a98c196e + .quad 0x7fa00425802e1e68 + .quad 0x7d531574028c2705 + .quad 0x80317d69db0d75fe + .quad 0x30fface8ef8c8ddd + .quad 0x7e9de97bb6c3e998 + + // 2^44 * 3 * G + + .quad 0x1558967b9e6585a3 + .quad 0x97c99ce098e98b92 + .quad 0x10af149b6eb3adad + .quad 0x42181fe8f4d38cfa + .quad 0xf004be62a24d40dd + .quad 0xba0659910452d41f + .quad 0x81c45ee162a44234 + .quad 0x4cb829d8a22266ef + .quad 0x1dbcaa8407b86681 + .quad 0x081f001e8b26753b + .quad 0x3cd7ce6a84048e81 + .quad 0x78af11633f25f22c + + // 2^44 * 4 * G + + .quad 0x8416ebd40b50babc + .quad 0x1508722628208bee + .quad 0xa3148fafb9c1c36d + .quad 0x0d07daacd32d7d5d + .quad 0x3241c00e7d65318c + .quad 0xe6bee5dcd0e86de7 + .quad 0x118b2dc2fbc08c26 + .quad 0x680d04a7fc603dc3 + .quad 0xf9c2414a695aa3eb + .quad 0xdaa42c4c05a68f21 + .quad 0x7c6c23987f93963e + .quad 0x210e8cd30c3954e3 + + // 2^44 * 5 * G + + .quad 0xac4201f210a71c06 + .quad 0x6a65e0aef3bfb021 + .quad 0xbc42c35c393632f7 + .quad 0x56ea8db1865f0742 + .quad 0x2b50f16137fe6c26 + .quad 0xe102bcd856e404d8 + .quad 0x12b0f1414c561f6b + .quad 0x51b17bc8d028ec91 + .quad 0xfff5fb4bcf535119 + .quad 0xf4989d79df1108a0 + .quad 0xbdfcea659a3ba325 + .quad 0x18a11f1174d1a6f2 + + // 2^44 * 6 * G + + .quad 0x407375ab3f6bba29 + .quad 0x9ec3b6d8991e482e + .quad 0x99c80e82e55f92e9 + .quad 0x307c13b6fb0c0ae1 + .quad 0xfbd63cdad27a5f2c + .quad 0xf00fc4bc8aa106d7 + .quad 0x53fb5c1a8e64a430 + .quad 0x04eaabe50c1a2e85 + .quad 0x24751021cb8ab5e7 + .quad 0xfc2344495c5010eb + .quad 0x5f1e717b4e5610a1 + .quad 0x44da5f18c2710cd5 + + // 2^44 * 7 * G + + .quad 0x033cc55ff1b82eb5 + .quad 0xb15ae36d411cae52 + .quad 0xba40b6198ffbacd3 + .quad 0x768edce1532e861f + .quad 0x9156fe6b89d8eacc + .quad 0xe6b79451e23126a1 + .quad 0xbd7463d93944eb4e + .quad 0x726373f6767203ae + .quad 0xe305ca72eb7ef68a + .quad 0x662cf31f70eadb23 + .quad 0x18f026fdb4c45b68 + .quad 0x513b5384b5d2ecbd + + // 2^44 * 8 * G + + .quad 0x46d46280c729989e + .quad 0x4b93fbd05368a5dd + .quad 0x63df3f81d1765a89 + .quad 0x34cebd64b9a0a223 + .quad 0x5e2702878af34ceb + .quad 0x900b0409b946d6ae + .quad 0x6512ebf7dabd8512 + .quad 0x61d9b76988258f81 + .quad 0xa6c5a71349b7d94b + .quad 0xa3f3d15823eb9446 + .quad 0x0416fbd277484834 + .quad 0x69d45e6f2c70812f + + // 2^48 * 1 * G + + .quad 0xce16f74bc53c1431 + .quad 0x2b9725ce2072edde + .quad 0xb8b9c36fb5b23ee7 + .quad 0x7e2e0e450b5cc908 + .quad 0x9fe62b434f460efb + .quad 0xded303d4a63607d6 + .quad 0xf052210eb7a0da24 + .quad 0x237e7dbe00545b93 + .quad 0x013575ed6701b430 + .quad 0x231094e69f0bfd10 + .quad 0x75320f1583e47f22 + .quad 0x71afa699b11155e3 + + // 2^48 * 2 * G + + .quad 0x65ce6f9b3953b61d + .quad 0xc65839eaafa141e6 + .quad 0x0f435ffda9f759fe + .quad 0x021142e9c2b1c28e + .quad 0xea423c1c473b50d6 + .quad 0x51e87a1f3b38ef10 + .quad 0x9b84bf5fb2c9be95 + .quad 0x00731fbc78f89a1c + .quad 0xe430c71848f81880 + .quad 0xbf960c225ecec119 + .quad 0xb6dae0836bba15e3 + .quad 0x4c4d6f3347e15808 + + // 2^48 * 3 * G + + .quad 0x18f7eccfc17d1fc9 + .quad 0x6c75f5a651403c14 + .quad 0xdbde712bf7ee0cdf + .quad 0x193fddaaa7e47a22 + .quad 0x2f0cddfc988f1970 + .quad 0x6b916227b0b9f51b + .quad 0x6ec7b6c4779176be + .quad 0x38bf9500a88f9fa8 + .quad 0x1fd2c93c37e8876f + .quad 0xa2f61e5a18d1462c + .quad 0x5080f58239241276 + .quad 0x6a6fb99ebf0d4969 + + // 2^48 * 4 * G + + .quad 0x6a46c1bb560855eb + .quad 0x2416bb38f893f09d + .quad 0xd71d11378f71acc1 + .quad 0x75f76914a31896ea + .quad 0xeeb122b5b6e423c6 + .quad 0x939d7010f286ff8e + .quad 0x90a92a831dcf5d8c + .quad 0x136fda9f42c5eb10 + .quad 0xf94cdfb1a305bdd1 + .quad 0x0f364b9d9ff82c08 + .quad 0x2a87d8a5c3bb588a + .quad 0x022183510be8dcba + + // 2^48 * 5 * G + + .quad 0x4af766385ead2d14 + .quad 0xa08ed880ca7c5830 + .quad 0x0d13a6e610211e3d + .quad 0x6a071ce17b806c03 + .quad 0x9d5a710143307a7f + .quad 0xb063de9ec47da45f + .quad 0x22bbfe52be927ad3 + .quad 0x1387c441fd40426c + .quad 0xb5d3c3d187978af8 + .quad 0x722b5a3d7f0e4413 + .quad 0x0d7b4848bb477ca0 + .quad 0x3171b26aaf1edc92 + + // 2^48 * 6 * G + + .quad 0xa92f319097564ca8 + .quad 0xff7bb84c2275e119 + .quad 0x4f55fe37a4875150 + .quad 0x221fd4873cf0835a + .quad 0xa60db7d8b28a47d1 + .quad 0xa6bf14d61770a4f1 + .quad 0xd4a1f89353ddbd58 + .quad 0x6c514a63344243e9 + .quad 0x2322204f3a156341 + .quad 0xfb73e0e9ba0a032d + .quad 0xfce0dd4c410f030e + .quad 0x48daa596fb924aaa + + // 2^48 * 7 * G + + .quad 0x6eca8e665ca59cc7 + .quad 0xa847254b2e38aca0 + .quad 0x31afc708d21e17ce + .quad 0x676dd6fccad84af7 + .quad 0x14f61d5dc84c9793 + .quad 0x9941f9e3ef418206 + .quad 0xcdf5b88f346277ac + .quad 0x58c837fa0e8a79a9 + .quad 0x0cf9688596fc9058 + .quad 0x1ddcbbf37b56a01b + .quad 0xdcc2e77d4935d66a + .quad 0x1c4f73f2c6a57f0a + + // 2^48 * 8 * G + + .quad 0x0e7a4fbd305fa0bb + .quad 0x829d4ce054c663ad + .quad 0xf421c3832fe33848 + .quad 0x795ac80d1bf64c42 + .quad 0xb36e706efc7c3484 + .quad 0x73dfc9b4c3c1cf61 + .quad 0xeb1d79c9781cc7e5 + .quad 0x70459adb7daf675c + .quad 0x1b91db4991b42bb3 + .quad 0x572696234b02dcca + .quad 0x9fdf9ee51f8c78dc + .quad 0x5fe162848ce21fd3 + + // 2^52 * 1 * G + + .quad 0xe2790aae4d077c41 + .quad 0x8b938270db7469a3 + .quad 0x6eb632dc8abd16a2 + .quad 0x720814ecaa064b72 + .quad 0x315c29c795115389 + .quad 0xd7e0e507862f74ce + .quad 0x0c4a762185927432 + .quad 0x72de6c984a25a1e4 + .quad 0xae9ab553bf6aa310 + .quad 0x050a50a9806d6e1b + .quad 0x92bb7403adff5139 + .quad 0x0394d27645be618b + + // 2^52 * 2 * G + + .quad 0x4d572251857eedf4 + .quad 0xe3724edde19e93c5 + .quad 0x8a71420e0b797035 + .quad 0x3b3c833687abe743 + .quad 0xf5396425b23545a4 + .quad 0x15a7a27e98fbb296 + .quad 0xab6c52bc636fdd86 + .quad 0x79d995a8419334ee + .quad 0xcd8a8ea61195dd75 + .quad 0xa504d8a81dd9a82f + .quad 0x540dca81a35879b6 + .quad 0x60dd16a379c86a8a + + // 2^52 * 3 * G + + .quad 0x35a2c8487381e559 + .quad 0x596ffea6d78082cb + .quad 0xcb9771ebdba7b653 + .quad 0x5a08b5019b4da685 + .quad 0x3501d6f8153e47b8 + .quad 0xb7a9675414a2f60c + .quad 0x112ee8b6455d9523 + .quad 0x4e62a3c18112ea8a + .quad 0xc8d4ac04516ab786 + .quad 0x595af3215295b23d + .quad 0xd6edd234db0230c1 + .quad 0x0929efe8825b41cc + + // 2^52 * 4 * G + + .quad 0x5f0601d1cbd0f2d3 + .quad 0x736e412f6132bb7f + .quad 0x83604432238dde87 + .quad 0x1e3a5272f5c0753c + .quad 0x8b3172b7ad56651d + .quad 0x01581b7a3fabd717 + .quad 0x2dc94df6424df6e4 + .quad 0x30376e5d2c29284f + .quad 0xd2918da78159a59c + .quad 0x6bdc1cd93f0713f3 + .quad 0x565f7a934acd6590 + .quad 0x53daacec4cb4c128 + + // 2^52 * 5 * G + + .quad 0x4ca73bd79cc8a7d6 + .quad 0x4d4a738f47e9a9b2 + .quad 0xf4cbf12942f5fe00 + .quad 0x01a13ff9bdbf0752 + .quad 0x99852bc3852cfdb0 + .quad 0x2cc12e9559d6ed0b + .quad 0x70f9e2bf9b5ac27b + .quad 0x4f3b8c117959ae99 + .quad 0x55b6c9c82ff26412 + .quad 0x1ac4a8c91fb667a8 + .quad 0xd527bfcfeb778bf2 + .quad 0x303337da7012a3be + + // 2^52 * 6 * G + + .quad 0x955422228c1c9d7c + .quad 0x01fac1371a9b340f + .quad 0x7e8d9177925b48d7 + .quad 0x53f8ad5661b3e31b + .quad 0x976d3ccbfad2fdd1 + .quad 0xcb88839737a640a8 + .quad 0x2ff00c1d6734cb25 + .quad 0x269ff4dc789c2d2b + .quad 0x0c003fbdc08d678d + .quad 0x4d982fa37ead2b17 + .quad 0xc07e6bcdb2e582f1 + .quad 0x296c7291df412a44 + + // 2^52 * 7 * G + + .quad 0x7903de2b33daf397 + .quad 0xd0ff0619c9a624b3 + .quad 0x8a1d252b555b3e18 + .quad 0x2b6d581c52e0b7c0 + .quad 0xdfb23205dab8b59e + .quad 0x465aeaa0c8092250 + .quad 0xd133c1189a725d18 + .quad 0x2327370261f117d1 + .quad 0x3d0543d3623e7986 + .quad 0x679414c2c278a354 + .quad 0xae43f0cc726196f6 + .quad 0x7836c41f8245eaba + + // 2^52 * 8 * G + + .quad 0xe7a254db49e95a81 + .quad 0x5192d5d008b0ad73 + .quad 0x4d20e5b1d00afc07 + .quad 0x5d55f8012cf25f38 + .quad 0xca651e848011937c + .quad 0xc6b0c46e6ef41a28 + .quad 0xb7021ba75f3f8d52 + .quad 0x119dff99ead7b9fd + .quad 0x43eadfcbf4b31d4d + .quad 0xc6503f7411148892 + .quad 0xfeee68c5060d3b17 + .quad 0x329293b3dd4a0ac8 + + // 2^56 * 1 * G + + .quad 0x4e59214fe194961a + .quad 0x49be7dc70d71cd4f + .quad 0x9300cfd23b50f22d + .quad 0x4789d446fc917232 + .quad 0x2879852d5d7cb208 + .quad 0xb8dedd70687df2e7 + .quad 0xdc0bffab21687891 + .quad 0x2b44c043677daa35 + .quad 0x1a1c87ab074eb78e + .quad 0xfac6d18e99daf467 + .quad 0x3eacbbcd484f9067 + .quad 0x60c52eef2bb9a4e4 + + // 2^56 * 2 * G + + .quad 0x0b5d89bc3bfd8bf1 + .quad 0xb06b9237c9f3551a + .quad 0x0e4c16b0d53028f5 + .quad 0x10bc9c312ccfcaab + .quad 0x702bc5c27cae6d11 + .quad 0x44c7699b54a48cab + .quad 0xefbc4056ba492eb2 + .quad 0x70d77248d9b6676d + .quad 0xaa8ae84b3ec2a05b + .quad 0x98699ef4ed1781e0 + .quad 0x794513e4708e85d1 + .quad 0x63755bd3a976f413 + + // 2^56 * 3 * G + + .quad 0xb55fa03e2ad10853 + .quad 0x356f75909ee63569 + .quad 0x9ff9f1fdbe69b890 + .quad 0x0d8cc1c48bc16f84 + .quad 0x3dc7101897f1acb7 + .quad 0x5dda7d5ec165bbd8 + .quad 0x508e5b9c0fa1020f + .quad 0x2763751737c52a56 + .quad 0x029402d36eb419a9 + .quad 0xf0b44e7e77b460a5 + .quad 0xcfa86230d43c4956 + .quad 0x70c2dd8a7ad166e7 + + // 2^56 * 4 * G + + .quad 0x656194509f6fec0e + .quad 0xee2e7ea946c6518d + .quad 0x9733c1f367e09b5c + .quad 0x2e0fac6363948495 + .quad 0x91d4967db8ed7e13 + .quad 0x74252f0ad776817a + .quad 0xe40982e00d852564 + .quad 0x32b8613816a53ce5 + .quad 0x79e7f7bee448cd64 + .quad 0x6ac83a67087886d0 + .quad 0xf89fd4d9a0e4db2e + .quad 0x4179215c735a4f41 + + // 2^56 * 5 * G + + .quad 0x8c7094e7d7dced2a + .quad 0x97fb8ac347d39c70 + .quad 0xe13be033a906d902 + .quad 0x700344a30cd99d76 + .quad 0xe4ae33b9286bcd34 + .quad 0xb7ef7eb6559dd6dc + .quad 0x278b141fb3d38e1f + .quad 0x31fa85662241c286 + .quad 0xaf826c422e3622f4 + .quad 0xc12029879833502d + .quad 0x9bc1b7e12b389123 + .quad 0x24bb2312a9952489 + + // 2^56 * 6 * G + + .quad 0xb1a8ed1732de67c3 + .quad 0x3cb49418461b4948 + .quad 0x8ebd434376cfbcd2 + .quad 0x0fee3e871e188008 + .quad 0x41f80c2af5f85c6b + .quad 0x687284c304fa6794 + .quad 0x8945df99a3ba1bad + .quad 0x0d1d2af9ffeb5d16 + .quad 0xa9da8aa132621edf + .quad 0x30b822a159226579 + .quad 0x4004197ba79ac193 + .quad 0x16acd79718531d76 + + // 2^56 * 7 * G + + .quad 0x72df72af2d9b1d3d + .quad 0x63462a36a432245a + .quad 0x3ecea07916b39637 + .quad 0x123e0ef6b9302309 + .quad 0xc959c6c57887b6ad + .quad 0x94e19ead5f90feba + .quad 0x16e24e62a342f504 + .quad 0x164ed34b18161700 + .quad 0x487ed94c192fe69a + .quad 0x61ae2cea3a911513 + .quad 0x877bf6d3b9a4de27 + .quad 0x78da0fc61073f3eb + + // 2^56 * 8 * G + + .quad 0x5bf15d28e52bc66a + .quad 0x2c47e31870f01a8e + .quad 0x2419afbc06c28bdd + .quad 0x2d25deeb256b173a + .quad 0xa29f80f1680c3a94 + .quad 0x71f77e151ae9e7e6 + .quad 0x1100f15848017973 + .quad 0x054aa4b316b38ddd + .quad 0xdfc8468d19267cb8 + .quad 0x0b28789c66e54daf + .quad 0x2aeb1d2a666eec17 + .quad 0x134610a6ab7da760 + + // 2^60 * 1 * G + + .quad 0xcaf55ec27c59b23f + .quad 0x99aeed3e154d04f2 + .quad 0x68441d72e14141f4 + .quad 0x140345133932a0a2 + .quad 0xd91430e0dc028c3c + .quad 0x0eb955a85217c771 + .quad 0x4b09e1ed2c99a1fa + .quad 0x42881af2bd6a743c + .quad 0x7bfec69aab5cad3d + .quad 0xc23e8cd34cb2cfad + .quad 0x685dd14bfb37d6a2 + .quad 0x0ad6d64415677a18 + + // 2^60 * 2 * G + + .quad 0x781a439e417becb5 + .quad 0x4ac5938cd10e0266 + .quad 0x5da385110692ac24 + .quad 0x11b065a2ade31233 + .quad 0x7914892847927e9f + .quad 0x33dad6ef370aa877 + .quad 0x1f8f24fa11122703 + .quad 0x5265ac2f2adf9592 + .quad 0x405fdd309afcb346 + .quad 0xd9723d4428e63f54 + .quad 0x94c01df05f65aaae + .quad 0x43e4dc3ae14c0809 + + // 2^60 * 3 * G + + .quad 0xbc12c7f1a938a517 + .quad 0x473028ab3180b2e1 + .quad 0x3f78571efbcd254a + .quad 0x74e534426ff6f90f + .quad 0xea6f7ac3adc2c6a3 + .quad 0xd0e928f6e9717c94 + .quad 0xe2d379ead645eaf5 + .quad 0x46dd8785c51ffbbe + .quad 0x709801be375c8898 + .quad 0x4b06dab5e3fd8348 + .quad 0x75880ced27230714 + .quad 0x2b09468fdd2f4c42 + + // 2^60 * 4 * G + + .quad 0x97c749eeb701cb96 + .quad 0x83f438d4b6a369c3 + .quad 0x62962b8b9a402cd9 + .quad 0x6976c7509888df7b + .quad 0x5b97946582ffa02a + .quad 0xda096a51fea8f549 + .quad 0xa06351375f77af9b + .quad 0x1bcfde61201d1e76 + .quad 0x4a4a5490246a59a2 + .quad 0xd63ebddee87fdd90 + .quad 0xd9437c670d2371fa + .quad 0x69e87308d30f8ed6 + + // 2^60 * 5 * G + + .quad 0x435a8bb15656beb0 + .quad 0xf8fac9ba4f4d5bca + .quad 0xb9b278c41548c075 + .quad 0x3eb0ef76e892b622 + .quad 0x0f80bf028bc80303 + .quad 0x6aae16b37a18cefb + .quad 0xdd47ea47d72cd6a3 + .quad 0x61943588f4ed39aa + .quad 0xd26e5c3e91039f85 + .quad 0xc0e9e77df6f33aa9 + .quad 0xe8968c5570066a93 + .quad 0x3c34d1881faaaddd + + // 2^60 * 6 * G + + .quad 0x3f9d2b5ea09f9ec0 + .quad 0x1dab3b6fb623a890 + .quad 0xa09ba3ea72d926c4 + .quad 0x374193513fd8b36d + .quad 0xbd5b0b8f2fffe0d9 + .quad 0x6aa254103ed24fb9 + .quad 0x2ac7d7bcb26821c4 + .quad 0x605b394b60dca36a + .quad 0xb4e856e45a9d1ed2 + .quad 0xefe848766c97a9a2 + .quad 0xb104cf641e5eee7d + .quad 0x2f50b81c88a71c8f + + // 2^60 * 7 * G + + .quad 0x31723c61fc6811bb + .quad 0x9cb450486211800f + .quad 0x768933d347995753 + .quad 0x3491a53502752fcd + .quad 0x2b552ca0a7da522a + .quad 0x3230b336449b0250 + .quad 0xf2c4c5bca4b99fb9 + .quad 0x7b2c674958074a22 + .quad 0xd55165883ed28cdf + .quad 0x12d84fd2d362de39 + .quad 0x0a874ad3e3378e4f + .quad 0x000d2b1f7c763e74 + + // 2^60 * 8 * G + + .quad 0x3d420811d06d4a67 + .quad 0xbefc048590e0ffe3 + .quad 0xf870c6b7bd487bde + .quad 0x6e2a7316319afa28 + .quad 0x9624778c3e94a8ab + .quad 0x0ad6f3cee9a78bec + .quad 0x948ac7810d743c4f + .quad 0x76627935aaecfccc + .quad 0x56a8ac24d6d59a9f + .quad 0xc8db753e3096f006 + .quad 0x477f41e68f4c5299 + .quad 0x588d851cf6c86114 + + // 2^64 * 1 * G + + .quad 0x51138ec78df6b0fe + .quad 0x5397da89e575f51b + .quad 0x09207a1d717af1b9 + .quad 0x2102fdba2b20d650 + .quad 0xcd2a65e777d1f515 + .quad 0x548991878faa60f1 + .quad 0xb1b73bbcdabc06e5 + .quad 0x654878cba97cc9fb + .quad 0x969ee405055ce6a1 + .quad 0x36bca7681251ad29 + .quad 0x3a1af517aa7da415 + .quad 0x0ad725db29ecb2ba + + // 2^64 * 2 * G + + .quad 0xdc4267b1834e2457 + .quad 0xb67544b570ce1bc5 + .quad 0x1af07a0bf7d15ed7 + .quad 0x4aefcffb71a03650 + .quad 0xfec7bc0c9b056f85 + .quad 0x537d5268e7f5ffd7 + .quad 0x77afc6624312aefa + .quad 0x4f675f5302399fd9 + .quad 0xc32d36360415171e + .quad 0xcd2bef118998483b + .quad 0x870a6eadd0945110 + .quad 0x0bccbb72a2a86561 + + // 2^64 * 3 * G + + .quad 0x185e962feab1a9c8 + .quad 0x86e7e63565147dcd + .quad 0xb092e031bb5b6df2 + .quad 0x4024f0ab59d6b73e + .quad 0x186d5e4c50fe1296 + .quad 0xe0397b82fee89f7e + .quad 0x3bc7f6c5507031b0 + .quad 0x6678fd69108f37c2 + .quad 0x1586fa31636863c2 + .quad 0x07f68c48572d33f2 + .quad 0x4f73cc9f789eaefc + .quad 0x2d42e2108ead4701 + + // 2^64 * 4 * G + + .quad 0x97f5131594dfd29b + .quad 0x6155985d313f4c6a + .quad 0xeba13f0708455010 + .quad 0x676b2608b8d2d322 + .quad 0x21717b0d0f537593 + .quad 0x914e690b131e064c + .quad 0x1bb687ae752ae09f + .quad 0x420bf3a79b423c6e + .quad 0x8138ba651c5b2b47 + .quad 0x8671b6ec311b1b80 + .quad 0x7bff0cb1bc3135b0 + .quad 0x745d2ffa9c0cf1e0 + + // 2^64 * 5 * G + + .quad 0xbf525a1e2bc9c8bd + .quad 0xea5b260826479d81 + .quad 0xd511c70edf0155db + .quad 0x1ae23ceb960cf5d0 + .quad 0x6036df5721d34e6a + .quad 0xb1db8827997bb3d0 + .quad 0xd3c209c3c8756afa + .quad 0x06e15be54c1dc839 + .quad 0x5b725d871932994a + .quad 0x32351cb5ceb1dab0 + .quad 0x7dc41549dab7ca05 + .quad 0x58ded861278ec1f7 + + // 2^64 * 6 * G + + .quad 0xd8173793f266c55c + .quad 0xc8c976c5cc454e49 + .quad 0x5ce382f8bc26c3a8 + .quad 0x2ff39de85485f6f9 + .quad 0x2dfb5ba8b6c2c9a8 + .quad 0x48eeef8ef52c598c + .quad 0x33809107f12d1573 + .quad 0x08ba696b531d5bd8 + .quad 0x77ed3eeec3efc57a + .quad 0x04e05517d4ff4811 + .quad 0xea3d7a3ff1a671cb + .quad 0x120633b4947cfe54 + + // 2^64 * 7 * G + + .quad 0x0b94987891610042 + .quad 0x4ee7b13cecebfae8 + .quad 0x70be739594f0a4c0 + .quad 0x35d30a99b4d59185 + .quad 0x82bd31474912100a + .quad 0xde237b6d7e6fbe06 + .quad 0xe11e761911ea79c6 + .quad 0x07433be3cb393bde + .quad 0xff7944c05ce997f4 + .quad 0x575d3de4b05c51a3 + .quad 0x583381fd5a76847c + .quad 0x2d873ede7af6da9f + + // 2^64 * 8 * G + + .quad 0x157a316443373409 + .quad 0xfab8b7eef4aa81d9 + .quad 0xb093fee6f5a64806 + .quad 0x2e773654707fa7b6 + .quad 0xaa6202e14e5df981 + .quad 0xa20d59175015e1f5 + .quad 0x18a275d3bae21d6c + .quad 0x0543618a01600253 + .quad 0x0deabdf4974c23c1 + .quad 0xaa6f0a259dce4693 + .quad 0x04202cb8a29aba2c + .quad 0x4b1443362d07960d + + // 2^68 * 1 * G + + .quad 0x47b837f753242cec + .quad 0x256dc48cc04212f2 + .quad 0xe222fbfbe1d928c5 + .quad 0x48ea295bad8a2c07 + .quad 0x299b1c3f57c5715e + .quad 0x96cb929e6b686d90 + .quad 0x3004806447235ab3 + .quad 0x2c435c24a44d9fe1 + .quad 0x0607c97c80f8833f + .quad 0x0e851578ca25ec5b + .quad 0x54f7450b161ebb6f + .quad 0x7bcb4792a0def80e + + // 2^68 * 2 * G + + .quad 0x8487e3d02bc73659 + .quad 0x4baf8445059979df + .quad 0xd17c975adcad6fbf + .quad 0x57369f0bdefc96b6 + .quad 0x1cecd0a0045224c2 + .quad 0x757f1b1b69e53952 + .quad 0x775b7a925289f681 + .quad 0x1b6cc62016736148 + .quad 0xf1a9990175638698 + .quad 0x353dd1beeeaa60d3 + .quad 0x849471334c9ba488 + .quad 0x63fa6e6843ade311 + + // 2^68 * 3 * G + + .quad 0xd15c20536597c168 + .quad 0x9f73740098d28789 + .quad 0x18aee7f13257ba1f + .quad 0x3418bfda07346f14 + .quad 0x2195becdd24b5eb7 + .quad 0x5e41f18cc0cd44f9 + .quad 0xdf28074441ca9ede + .quad 0x07073b98f35b7d67 + .quad 0xd03c676c4ce530d4 + .quad 0x0b64c0473b5df9f4 + .quad 0x065cef8b19b3a31e + .quad 0x3084d661533102c9 + + // 2^68 * 4 * G + + .quad 0xe1f6b79ebf8469ad + .quad 0x15801004e2663135 + .quad 0x9a498330af74181b + .quad 0x3ba2504f049b673c + .quad 0x9a6ce876760321fd + .quad 0x7fe2b5109eb63ad8 + .quad 0x00e7d4ae8ac80592 + .quad 0x73d86b7abb6f723a + .quad 0x0b52b5606dba5ab6 + .quad 0xa9134f0fbbb1edab + .quad 0x30a9520d9b04a635 + .quad 0x6813b8f37973e5db + + // 2^68 * 5 * G + + .quad 0x9854b054334127c1 + .quad 0x105d047882fbff25 + .quad 0xdb49f7f944186f4f + .quad 0x1768e838bed0b900 + .quad 0xf194ca56f3157e29 + .quad 0x136d35705ef528a5 + .quad 0xdd4cef778b0599bc + .quad 0x7d5472af24f833ed + .quad 0xd0ef874daf33da47 + .quad 0x00d3be5db6e339f9 + .quad 0x3f2a8a2f9c9ceece + .quad 0x5d1aeb792352435a + + // 2^68 * 6 * G + + .quad 0xf59e6bb319cd63ca + .quad 0x670c159221d06839 + .quad 0xb06d565b2150cab6 + .quad 0x20fb199d104f12a3 + .quad 0x12c7bfaeb61ba775 + .quad 0xb84e621fe263bffd + .quad 0x0b47a5c35c840dcf + .quad 0x7e83be0bccaf8634 + .quad 0x61943dee6d99c120 + .quad 0x86101f2e460b9fe0 + .quad 0x6bb2f1518ee8598d + .quad 0x76b76289fcc475cc + + // 2^68 * 7 * G + + .quad 0x791b4cc1756286fa + .quad 0xdbced317d74a157c + .quad 0x7e732421ea72bde6 + .quad 0x01fe18491131c8e9 + .quad 0x4245f1a1522ec0b3 + .quad 0x558785b22a75656d + .quad 0x1d485a2548a1b3c0 + .quad 0x60959eccd58fe09f + .quad 0x3ebfeb7ba8ed7a09 + .quad 0x49fdc2bbe502789c + .quad 0x44ebce5d3c119428 + .quad 0x35e1eb55be947f4a + + // 2^68 * 8 * G + + .quad 0xdbdae701c5738dd3 + .quad 0xf9c6f635b26f1bee + .quad 0x61e96a8042f15ef4 + .quad 0x3aa1d11faf60a4d8 + .quad 0x14fd6dfa726ccc74 + .quad 0x3b084cfe2f53b965 + .quad 0xf33ae4f552a2c8b4 + .quad 0x59aab07a0d40166a + .quad 0x77bcec4c925eac25 + .quad 0x1848718460137738 + .quad 0x5b374337fea9f451 + .quad 0x1865e78ec8e6aa46 + + // 2^72 * 1 * G + + .quad 0xccc4b7c7b66e1f7a + .quad 0x44157e25f50c2f7e + .quad 0x3ef06dfc713eaf1c + .quad 0x582f446752da63f7 + .quad 0x967c54e91c529ccb + .quad 0x30f6269264c635fb + .quad 0x2747aff478121965 + .quad 0x17038418eaf66f5c + .quad 0xc6317bd320324ce4 + .quad 0xa81042e8a4488bc4 + .quad 0xb21ef18b4e5a1364 + .quad 0x0c2a1c4bcda28dc9 + + // 2^72 * 2 * G + + .quad 0xd24dc7d06f1f0447 + .quad 0xb2269e3edb87c059 + .quad 0xd15b0272fbb2d28f + .quad 0x7c558bd1c6f64877 + .quad 0xedc4814869bd6945 + .quad 0x0d6d907dbe1c8d22 + .quad 0xc63bd212d55cc5ab + .quad 0x5a6a9b30a314dc83 + .quad 0xd0ec1524d396463d + .quad 0x12bb628ac35a24f0 + .quad 0xa50c3a791cbc5fa4 + .quad 0x0404a5ca0afbafc3 + + // 2^72 * 3 * G + + .quad 0x8c1f40070aa743d6 + .quad 0xccbad0cb5b265ee8 + .quad 0x574b046b668fd2de + .quad 0x46395bfdcadd9633 + .quad 0x62bc9e1b2a416fd1 + .quad 0xb5c6f728e350598b + .quad 0x04343fd83d5d6967 + .quad 0x39527516e7f8ee98 + .quad 0x117fdb2d1a5d9a9c + .quad 0x9c7745bcd1005c2a + .quad 0xefd4bef154d56fea + .quad 0x76579a29e822d016 + + // 2^72 * 4 * G + + .quad 0x45b68e7e49c02a17 + .quad 0x23cd51a2bca9a37f + .quad 0x3ed65f11ec224c1b + .quad 0x43a384dc9e05bdb1 + .quad 0x333cb51352b434f2 + .quad 0xd832284993de80e1 + .quad 0xb5512887750d35ce + .quad 0x02c514bb2a2777c1 + .quad 0x684bd5da8bf1b645 + .quad 0xfb8bd37ef6b54b53 + .quad 0x313916d7a9b0d253 + .quad 0x1160920961548059 + + // 2^72 * 5 * G + + .quad 0xb44d166929dacfaa + .quad 0xda529f4c8413598f + .quad 0xe9ef63ca453d5559 + .quad 0x351e125bc5698e0b + .quad 0x7a385616369b4dcd + .quad 0x75c02ca7655c3563 + .quad 0x7dc21bf9d4f18021 + .quad 0x2f637d7491e6e042 + .quad 0xd4b49b461af67bbe + .quad 0xd603037ac8ab8961 + .quad 0x71dee19ff9a699fb + .quad 0x7f182d06e7ce2a9a + + // 2^72 * 6 * G + + .quad 0x7a7c8e64ab0168ec + .quad 0xcb5a4a5515edc543 + .quad 0x095519d347cd0eda + .quad 0x67d4ac8c343e93b0 + .quad 0x09454b728e217522 + .quad 0xaa58e8f4d484b8d8 + .quad 0xd358254d7f46903c + .quad 0x44acc043241c5217 + .quad 0x1c7d6bbb4f7a5777 + .quad 0x8b35fed4918313e1 + .quad 0x4adca1c6c96b4684 + .quad 0x556d1c8312ad71bd + + // 2^72 * 7 * G + + .quad 0x17ef40e30c8d3982 + .quad 0x31f7073e15a3fa34 + .quad 0x4f21f3cb0773646e + .quad 0x746c6c6d1d824eff + .quad 0x81f06756b11be821 + .quad 0x0faff82310a3f3dd + .quad 0xf8b2d0556a99465d + .quad 0x097abe38cc8c7f05 + .quad 0x0c49c9877ea52da4 + .quad 0x4c4369559bdc1d43 + .quad 0x022c3809f7ccebd2 + .quad 0x577e14a34bee84bd + + // 2^72 * 8 * G + + .quad 0xf0e268ac61a73b0a + .quad 0xf2fafa103791a5f5 + .quad 0xc1e13e826b6d00e9 + .quad 0x60fa7ee96fd78f42 + .quad 0x94fecebebd4dd72b + .quad 0xf46a4fda060f2211 + .quad 0x124a5977c0c8d1ff + .quad 0x705304b8fb009295 + .quad 0xb63d1d354d296ec6 + .quad 0xf3c3053e5fad31d8 + .quad 0x670b958cb4bd42ec + .quad 0x21398e0ca16353fd + + // 2^76 * 1 * G + + .quad 0x216ab2ca8da7d2ef + .quad 0x366ad9dd99f42827 + .quad 0xae64b9004fdd3c75 + .quad 0x403a395b53909e62 + .quad 0x86c5fc16861b7e9a + .quad 0xf6a330476a27c451 + .quad 0x01667267a1e93597 + .quad 0x05ffb9cd6082dfeb + .quad 0xa617fa9ff53f6139 + .quad 0x60f2b5e513e66cb6 + .quad 0xd7a8beefb3448aa4 + .quad 0x7a2932856f5ea192 + + // 2^76 * 2 * G + + .quad 0x0b39d761b02de888 + .quad 0x5f550e7ed2414e1f + .quad 0xa6bfa45822e1a940 + .quad 0x050a2f7dfd447b99 + .quad 0xb89c444879639302 + .quad 0x4ae4f19350c67f2c + .quad 0xf0b35da8c81af9c6 + .quad 0x39d0003546871017 + .quad 0x437c3b33a650db77 + .quad 0x6bafe81dbac52bb2 + .quad 0xfe99402d2db7d318 + .quad 0x2b5b7eec372ba6ce + + // 2^76 * 3 * G + + .quad 0xb3bc4bbd83f50eef + .quad 0x508f0c998c927866 + .quad 0x43e76587c8b7e66e + .quad 0x0f7655a3a47f98d9 + .quad 0xa694404d613ac8f4 + .quad 0x500c3c2bfa97e72c + .quad 0x874104d21fcec210 + .quad 0x1b205fb38604a8ee + .quad 0x55ecad37d24b133c + .quad 0x441e147d6038c90b + .quad 0x656683a1d62c6fee + .quad 0x0157d5dc87e0ecae + + // 2^76 * 4 * G + + .quad 0xf2a7af510354c13d + .quad 0xd7a0b145aa372b60 + .quad 0x2869b96a05a3d470 + .quad 0x6528e42d82460173 + .quad 0x95265514d71eb524 + .quad 0xe603d8815df14593 + .quad 0x147cdf410d4de6b7 + .quad 0x5293b1730437c850 + .quad 0x23d0e0814bccf226 + .quad 0x92c745cd8196fb93 + .quad 0x8b61796c59541e5b + .quad 0x40a44df0c021f978 + + // 2^76 * 5 * G + + .quad 0xdaa869894f20ea6a + .quad 0xea14a3d14c620618 + .quad 0x6001fccb090bf8be + .quad 0x35f4e822947e9cf0 + .quad 0x86c96e514bc5d095 + .quad 0xf20d4098fca6804a + .quad 0x27363d89c826ea5d + .quad 0x39ca36565719cacf + .quad 0x97506f2f6f87b75c + .quad 0xc624aea0034ae070 + .quad 0x1ec856e3aad34dd6 + .quad 0x055b0be0e440e58f + + // 2^76 * 6 * G + + .quad 0x6469a17d89735d12 + .quad 0xdb6f27d5e662b9f1 + .quad 0x9fcba3286a395681 + .quad 0x363b8004d269af25 + .quad 0x4d12a04b6ea33da2 + .quad 0x57cf4c15e36126dd + .quad 0x90ec9675ee44d967 + .quad 0x64ca348d2a985aac + .quad 0x99588e19e4c4912d + .quad 0xefcc3b4e1ca5ce6b + .quad 0x4522ea60fa5b98d5 + .quad 0x7064bbab1de4a819 + + // 2^76 * 7 * G + + .quad 0xb919e1515a770641 + .quad 0xa9a2e2c74e7f8039 + .quad 0x7527250b3df23109 + .quad 0x756a7330ac27b78b + .quad 0xa290c06142542129 + .quad 0xf2e2c2aebe8d5b90 + .quad 0xcf2458db76abfe1b + .quad 0x02157ade83d626bf + .quad 0x3e46972a1b9a038b + .quad 0x2e4ee66a7ee03fb4 + .quad 0x81a248776edbb4ca + .quad 0x1a944ee88ecd0563 + + // 2^76 * 8 * G + + .quad 0xd5a91d1151039372 + .quad 0x2ed377b799ca26de + .quad 0xa17202acfd366b6b + .quad 0x0730291bd6901995 + .quad 0xbb40a859182362d6 + .quad 0xb99f55778a4d1abb + .quad 0x8d18b427758559f6 + .quad 0x26c20fe74d26235a + .quad 0x648d1d9fe9cc22f5 + .quad 0x66bc561928dd577c + .quad 0x47d3ed21652439d1 + .quad 0x49d271acedaf8b49 + + // 2^80 * 1 * G + + .quad 0x89f5058a382b33f3 + .quad 0x5ae2ba0bad48c0b4 + .quad 0x8f93b503a53db36e + .quad 0x5aa3ed9d95a232e6 + .quad 0x2798aaf9b4b75601 + .quad 0x5eac72135c8dad72 + .quad 0xd2ceaa6161b7a023 + .quad 0x1bbfb284e98f7d4e + .quad 0x656777e9c7d96561 + .quad 0xcb2b125472c78036 + .quad 0x65053299d9506eee + .quad 0x4a07e14e5e8957cc + + // 2^80 * 2 * G + + .quad 0x4ee412cb980df999 + .quad 0xa315d76f3c6ec771 + .quad 0xbba5edde925c77fd + .quad 0x3f0bac391d313402 + .quad 0x240b58cdc477a49b + .quad 0xfd38dade6447f017 + .quad 0x19928d32a7c86aad + .quad 0x50af7aed84afa081 + .quad 0x6e4fde0115f65be5 + .quad 0x29982621216109b2 + .quad 0x780205810badd6d9 + .quad 0x1921a316baebd006 + + // 2^80 * 3 * G + + .quad 0x89422f7edfb870fc + .quad 0x2c296beb4f76b3bd + .quad 0x0738f1d436c24df7 + .quad 0x6458df41e273aeb0 + .quad 0xd75aad9ad9f3c18b + .quad 0x566a0eef60b1c19c + .quad 0x3e9a0bac255c0ed9 + .quad 0x7b049deca062c7f5 + .quad 0xdccbe37a35444483 + .quad 0x758879330fedbe93 + .quad 0x786004c312c5dd87 + .quad 0x6093dccbc2950e64 + + // 2^80 * 4 * G + + .quad 0x1ff39a8585e0706d + .quad 0x36d0a5d8b3e73933 + .quad 0x43b9f2e1718f453b + .quad 0x57d1ea084827a97c + .quad 0x6bdeeebe6084034b + .quad 0x3199c2b6780fb854 + .quad 0x973376abb62d0695 + .quad 0x6e3180c98b647d90 + .quad 0xee7ab6e7a128b071 + .quad 0xa4c1596d93a88baa + .quad 0xf7b4de82b2216130 + .quad 0x363e999ddd97bd18 + + // 2^80 * 5 * G + + .quad 0x96a843c135ee1fc4 + .quad 0x976eb35508e4c8cf + .quad 0xb42f6801b58cd330 + .quad 0x48ee9b78693a052b + .quad 0x2f1848dce24baec6 + .quad 0x769b7255babcaf60 + .quad 0x90cb3c6e3cefe931 + .quad 0x231f979bc6f9b355 + .quad 0x5c31de4bcc2af3c6 + .quad 0xb04bb030fe208d1f + .quad 0xb78d7009c14fb466 + .quad 0x079bfa9b08792413 + + // 2^80 * 6 * G + + .quad 0xe3903a51da300df4 + .quad 0x843964233da95ab0 + .quad 0xed3cf12d0b356480 + .quad 0x038c77f684817194 + .quad 0xf3c9ed80a2d54245 + .quad 0x0aa08b7877f63952 + .quad 0xd76dac63d1085475 + .quad 0x1ef4fb159470636b + .quad 0x854e5ee65b167bec + .quad 0x59590a4296d0cdc2 + .quad 0x72b2df3498102199 + .quad 0x575ee92a4a0bff56 + + // 2^80 * 7 * G + + .quad 0xd4c080908a182fcf + .quad 0x30e170c299489dbd + .quad 0x05babd5752f733de + .quad 0x43d4e7112cd3fd00 + .quad 0x5d46bc450aa4d801 + .quad 0xc3af1227a533b9d8 + .quad 0x389e3b262b8906c2 + .quad 0x200a1e7e382f581b + .quad 0x518db967eaf93ac5 + .quad 0x71bc989b056652c0 + .quad 0xfe2b85d9567197f5 + .quad 0x050eca52651e4e38 + + // 2^80 * 8 * G + + .quad 0xc3431ade453f0c9c + .quad 0xe9f5045eff703b9b + .quad 0xfcd97ac9ed847b3d + .quad 0x4b0ee6c21c58f4c6 + .quad 0x97ac397660e668ea + .quad 0x9b19bbfe153ab497 + .quad 0x4cb179b534eca79f + .quad 0x6151c09fa131ae57 + .quad 0x3af55c0dfdf05d96 + .quad 0xdd262ee02ab4ee7a + .quad 0x11b2bb8712171709 + .quad 0x1fef24fa800f030b + + // 2^84 * 1 * G + + .quad 0xb496123a6b6c6609 + .quad 0xa750fe8580ab5938 + .quad 0xf471bf39b7c27a5f + .quad 0x507903ce77ac193c + .quad 0xff91a66a90166220 + .quad 0xf22552ae5bf1e009 + .quad 0x7dff85d87f90df7c + .quad 0x4f620ffe0c736fb9 + .quad 0x62f90d65dfde3e34 + .quad 0xcf28c592b9fa5fad + .quad 0x99c86ef9c6164510 + .quad 0x25d448044a256c84 + + // 2^84 * 2 * G + + .quad 0xbd68230ec7e9b16f + .quad 0x0eb1b9c1c1c5795d + .quad 0x7943c8c495b6b1ff + .quad 0x2f9faf620bbacf5e + .quad 0x2c7c4415c9022b55 + .quad 0x56a0d241812eb1fe + .quad 0xf02ea1c9d7b65e0d + .quad 0x4180512fd5323b26 + .quad 0xa4ff3e698a48a5db + .quad 0xba6a3806bd95403b + .quad 0x9f7ce1af47d5b65d + .quad 0x15e087e55939d2fb + + // 2^84 * 3 * G + + .quad 0x12207543745c1496 + .quad 0xdaff3cfdda38610c + .quad 0xe4e797272c71c34f + .quad 0x39c07b1934bdede9 + .quad 0x8894186efb963f38 + .quad 0x48a00e80dc639bd5 + .quad 0xa4e8092be96c1c99 + .quad 0x5a097d54ca573661 + .quad 0x2d45892b17c9e755 + .quad 0xd033fd7289308df8 + .quad 0x6c2fe9d9525b8bd9 + .quad 0x2edbecf1c11cc079 + + // 2^84 * 4 * G + + .quad 0x1616a4e3c715a0d2 + .quad 0x53623cb0f8341d4d + .quad 0x96ef5329c7e899cb + .quad 0x3d4e8dbba668baa6 + .quad 0xee0f0fddd087a25f + .quad 0x9c7531555c3e34ee + .quad 0x660c572e8fab3ab5 + .quad 0x0854fc44544cd3b2 + .quad 0x61eba0c555edad19 + .quad 0x24b533fef0a83de6 + .quad 0x3b77042883baa5f8 + .quad 0x678f82b898a47e8d + + // 2^84 * 5 * G + + .quad 0xb1491d0bd6900c54 + .quad 0x3539722c9d132636 + .quad 0x4db928920b362bc9 + .quad 0x4d7cd1fea68b69df + .quad 0x1e09d94057775696 + .quad 0xeed1265c3cd951db + .quad 0xfa9dac2b20bce16f + .quad 0x0f7f76e0e8d089f4 + .quad 0x36d9ebc5d485b00c + .quad 0xa2596492e4adb365 + .quad 0xc1659480c2119ccd + .quad 0x45306349186e0d5f + + // 2^84 * 6 * G + + .quad 0x94ddd0c1a6cdff1d + .quad 0x55f6f115e84213ae + .quad 0x6c935f85992fcf6a + .quad 0x067ee0f54a37f16f + .quad 0x96a414ec2b072491 + .quad 0x1bb2218127a7b65b + .quad 0x6d2849596e8a4af0 + .quad 0x65f3b08ccd27765f + .quad 0xecb29fff199801f7 + .quad 0x9d361d1fa2a0f72f + .quad 0x25f11d2375fd2f49 + .quad 0x124cefe80fe10fe2 + + // 2^84 * 7 * G + + .quad 0x4c126cf9d18df255 + .quad 0xc1d471e9147a63b6 + .quad 0x2c6d3c73f3c93b5f + .quad 0x6be3a6a2e3ff86a2 + .quad 0x1518e85b31b16489 + .quad 0x8faadcb7db710bfb + .quad 0x39b0bdf4a14ae239 + .quad 0x05f4cbea503d20c1 + .quad 0xce040e9ec04145bc + .quad 0xc71ff4e208f6834c + .quad 0xbd546e8dab8847a3 + .quad 0x64666aa0a4d2aba5 + + // 2^84 * 8 * G + + .quad 0x6841435a7c06d912 + .quad 0xca123c21bb3f830b + .quad 0xd4b37b27b1cbe278 + .quad 0x1d753b84c76f5046 + .quad 0xb0c53bf73337e94c + .quad 0x7cb5697e11e14f15 + .quad 0x4b84abac1930c750 + .quad 0x28dd4abfe0640468 + .quad 0x7dc0b64c44cb9f44 + .quad 0x18a3e1ace3925dbf + .quad 0x7a3034862d0457c4 + .quad 0x4c498bf78a0c892e + + // 2^88 * 1 * G + + .quad 0x37d653fb1aa73196 + .quad 0x0f9495303fd76418 + .quad 0xad200b09fb3a17b2 + .quad 0x544d49292fc8613e + .quad 0x22d2aff530976b86 + .quad 0x8d90b806c2d24604 + .quad 0xdca1896c4de5bae5 + .quad 0x28005fe6c8340c17 + .quad 0x6aefba9f34528688 + .quad 0x5c1bff9425107da1 + .quad 0xf75bbbcd66d94b36 + .quad 0x72e472930f316dfa + + // 2^88 * 2 * G + + .quad 0x2695208c9781084f + .quad 0xb1502a0b23450ee1 + .quad 0xfd9daea603efde02 + .quad 0x5a9d2e8c2733a34c + .quad 0x07f3f635d32a7627 + .quad 0x7aaa4d865f6566f0 + .quad 0x3c85e79728d04450 + .quad 0x1fee7f000fe06438 + .quad 0x765305da03dbf7e5 + .quad 0xa4daf2491434cdbd + .quad 0x7b4ad5cdd24a88ec + .quad 0x00f94051ee040543 + + // 2^88 * 3 * G + + .quad 0x8d356b23c3d330b2 + .quad 0xf21c8b9bb0471b06 + .quad 0xb36c316c6e42b83c + .quad 0x07d79c7e8beab10d + .quad 0xd7ef93bb07af9753 + .quad 0x583ed0cf3db766a7 + .quad 0xce6998bf6e0b1ec5 + .quad 0x47b7ffd25dd40452 + .quad 0x87fbfb9cbc08dd12 + .quad 0x8a066b3ae1eec29b + .quad 0x0d57242bdb1fc1bf + .quad 0x1c3520a35ea64bb6 + + // 2^88 * 4 * G + + .quad 0x80d253a6bccba34a + .quad 0x3e61c3a13838219b + .quad 0x90c3b6019882e396 + .quad 0x1c3d05775d0ee66f + .quad 0xcda86f40216bc059 + .quad 0x1fbb231d12bcd87e + .quad 0xb4956a9e17c70990 + .quad 0x38750c3b66d12e55 + .quad 0x692ef1409422e51a + .quad 0xcbc0c73c2b5df671 + .quad 0x21014fe7744ce029 + .quad 0x0621e2c7d330487c + + // 2^88 * 5 * G + + .quad 0xaf9860cc8259838d + .quad 0x90ea48c1c69f9adc + .quad 0x6526483765581e30 + .quad 0x0007d6097bd3a5bc + .quad 0xb7ae1796b0dbf0f3 + .quad 0x54dfafb9e17ce196 + .quad 0x25923071e9aaa3b4 + .quad 0x5d8e589ca1002e9d + .quad 0xc0bf1d950842a94b + .quad 0xb2d3c363588f2e3e + .quad 0x0a961438bb51e2ef + .quad 0x1583d7783c1cbf86 + + // 2^88 * 6 * G + + .quad 0xeceea2ef5da27ae1 + .quad 0x597c3a1455670174 + .quad 0xc9a62a126609167a + .quad 0x252a5f2e81ed8f70 + .quad 0x90034704cc9d28c7 + .quad 0x1d1b679ef72cc58f + .quad 0x16e12b5fbe5b8726 + .quad 0x4958064e83c5580a + .quad 0x0d2894265066e80d + .quad 0xfcc3f785307c8c6b + .quad 0x1b53da780c1112fd + .quad 0x079c170bd843b388 + + // 2^88 * 7 * G + + .quad 0x0506ece464fa6fff + .quad 0xbee3431e6205e523 + .quad 0x3579422451b8ea42 + .quad 0x6dec05e34ac9fb00 + .quad 0xcdd6cd50c0d5d056 + .quad 0x9af7686dbb03573b + .quad 0x3ca6723ff3c3ef48 + .quad 0x6768c0d7317b8acc + .quad 0x94b625e5f155c1b3 + .quad 0x417bf3a7997b7b91 + .quad 0xc22cbddc6d6b2600 + .quad 0x51445e14ddcd52f4 + + // 2^88 * 8 * G + + .quad 0x57502b4b3b144951 + .quad 0x8e67ff6b444bbcb3 + .quad 0xb8bd6927166385db + .quad 0x13186f31e39295c8 + .quad 0x893147ab2bbea455 + .quad 0x8c53a24f92079129 + .quad 0x4b49f948be30f7a7 + .quad 0x12e990086e4fd43d + .quad 0xf10c96b37fdfbb2e + .quad 0x9f9a935e121ceaf9 + .quad 0xdf1136c43a5b983f + .quad 0x77b2e3f05d3e99af + + // 2^92 * 1 * G + + .quad 0xfd0d75879cf12657 + .quad 0xe82fef94e53a0e29 + .quad 0xcc34a7f05bbb4be7 + .quad 0x0b251172a50c38a2 + .quad 0x9532f48fcc5cd29b + .quad 0x2ba851bea3ce3671 + .quad 0x32dacaa051122941 + .quad 0x478d99d9350004f2 + .quad 0x1d5ad94890bb02c0 + .quad 0x50e208b10ec25115 + .quad 0xa26a22894ef21702 + .quad 0x4dc923343b524805 + + // 2^92 * 2 * G + + .quad 0xe3828c400f8086b6 + .quad 0x3f77e6f7979f0dc8 + .quad 0x7ef6de304df42cb4 + .quad 0x5265797cb6abd784 + .quad 0x3ad3e3ebf36c4975 + .quad 0xd75d25a537862125 + .quad 0xe873943da025a516 + .quad 0x6bbc7cb4c411c847 + .quad 0x3c6f9cd1d4a50d56 + .quad 0xb6244077c6feab7e + .quad 0x6ff9bf483580972e + .quad 0x00375883b332acfb + + // 2^92 * 3 * G + + .quad 0x0001b2cd28cb0940 + .quad 0x63fb51a06f1c24c9 + .quad 0xb5ad8691dcd5ca31 + .quad 0x67238dbd8c450660 + .quad 0xc98bec856c75c99c + .quad 0xe44184c000e33cf4 + .quad 0x0a676b9bba907634 + .quad 0x669e2cb571f379d7 + .quad 0xcb116b73a49bd308 + .quad 0x025aad6b2392729e + .quad 0xb4793efa3f55d9b1 + .quad 0x72a1056140678bb9 + + // 2^92 * 4 * G + + .quad 0xa2b6812b1cc9249d + .quad 0x62866eee21211f58 + .quad 0x2cb5c5b85df10ece + .quad 0x03a6b259e263ae00 + .quad 0x0d8d2909e2e505b6 + .quad 0x98ca78abc0291230 + .quad 0x77ef5569a9b12327 + .quad 0x7c77897b81439b47 + .quad 0xf1c1b5e2de331cb5 + .quad 0x5a9f5d8e15fca420 + .quad 0x9fa438f17bd932b1 + .quad 0x2a381bf01c6146e7 + + // 2^92 * 5 * G + + .quad 0xac9b9879cfc811c1 + .quad 0x8b7d29813756e567 + .quad 0x50da4e607c70edfc + .quad 0x5dbca62f884400b6 + .quad 0xf7c0be32b534166f + .quad 0x27e6ca6419cf70d4 + .quad 0x934df7d7a957a759 + .quad 0x5701461dabdec2aa + .quad 0x2c6747402c915c25 + .quad 0x1bdcd1a80b0d340a + .quad 0x5e5601bd07b43f5f + .quad 0x2555b4e05539a242 + + // 2^92 * 6 * G + + .quad 0x6fc09f5266ddd216 + .quad 0xdce560a7c8e37048 + .quad 0xec65939da2df62fd + .quad 0x7a869ae7e52ed192 + .quad 0x78409b1d87e463d4 + .quad 0xad4da95acdfb639d + .quad 0xec28773755259b9c + .quad 0x69c806e9c31230ab + .quad 0x7b48f57414bb3f22 + .quad 0x68c7cee4aedccc88 + .quad 0xed2f936179ed80be + .quad 0x25d70b885f77bc4b + + // 2^92 * 7 * G + + .quad 0x4151c3d9762bf4de + .quad 0x083f435f2745d82b + .quad 0x29775a2e0d23ddd5 + .quad 0x138e3a6269a5db24 + .quad 0x98459d29bb1ae4d4 + .quad 0x56b9c4c739f954ec + .quad 0x832743f6c29b4b3e + .quad 0x21ea8e2798b6878a + .quad 0x87bef4b46a5a7b9c + .quad 0xd2299d1b5fc1d062 + .quad 0x82409818dd321648 + .quad 0x5c5abeb1e5a2e03d + + // 2^92 * 8 * G + + .quad 0x14722af4b73c2ddb + .quad 0xbc470c5f5a05060d + .quad 0x00943eac2581b02e + .quad 0x0e434b3b1f499c8f + .quad 0x02cde6de1306a233 + .quad 0x7b5a52a2116f8ec7 + .quad 0xe1c681f4c1163b5b + .quad 0x241d350660d32643 + .quad 0x6be4404d0ebc52c7 + .quad 0xae46233bb1a791f5 + .quad 0x2aec170ed25db42b + .quad 0x1d8dfd966645d694 + + // 2^96 * 1 * G + + .quad 0x296fa9c59c2ec4de + .quad 0xbc8b61bf4f84f3cb + .quad 0x1c7706d917a8f908 + .quad 0x63b795fc7ad3255d + .quad 0xd598639c12ddb0a4 + .quad 0xa5d19f30c024866b + .quad 0xd17c2f0358fce460 + .quad 0x07a195152e095e8a + .quad 0xa8368f02389e5fc8 + .quad 0x90433b02cf8de43b + .quad 0xafa1fd5dc5412643 + .quad 0x3e8fe83d032f0137 + + // 2^96 * 2 * G + + .quad 0x2f8b15b90570a294 + .quad 0x94f2427067084549 + .quad 0xde1c5ae161bbfd84 + .quad 0x75ba3b797fac4007 + .quad 0x08704c8de8efd13c + .quad 0xdfc51a8e33e03731 + .quad 0xa59d5da51260cde3 + .quad 0x22d60899a6258c86 + .quad 0x6239dbc070cdd196 + .quad 0x60fe8a8b6c7d8a9a + .quad 0xb38847bceb401260 + .quad 0x0904d07b87779e5e + + // 2^96 * 3 * G + + .quad 0xb4ce1fd4ddba919c + .quad 0xcf31db3ec74c8daa + .quad 0x2c63cc63ad86cc51 + .quad 0x43e2143fbc1dde07 + .quad 0xf4322d6648f940b9 + .quad 0x06952f0cbd2d0c39 + .quad 0x167697ada081f931 + .quad 0x6240aacebaf72a6c + .quad 0xf834749c5ba295a0 + .quad 0xd6947c5bca37d25a + .quad 0x66f13ba7e7c9316a + .quad 0x56bdaf238db40cac + + // 2^96 * 4 * G + + .quad 0x362ab9e3f53533eb + .quad 0x338568d56eb93d40 + .quad 0x9e0e14521d5a5572 + .quad 0x1d24a86d83741318 + .quad 0x1310d36cc19d3bb2 + .quad 0x062a6bb7622386b9 + .quad 0x7c9b8591d7a14f5c + .quad 0x03aa31507e1e5754 + .quad 0xf4ec7648ffd4ce1f + .quad 0xe045eaf054ac8c1c + .quad 0x88d225821d09357c + .quad 0x43b261dc9aeb4859 + + // 2^96 * 5 * G + + .quad 0xe55b1e1988bb79bb + .quad 0xa09ed07dc17a359d + .quad 0xb02c2ee2603dea33 + .quad 0x326055cf5b276bc2 + .quad 0x19513d8b6c951364 + .quad 0x94fe7126000bf47b + .quad 0x028d10ddd54f9567 + .quad 0x02b4d5e242940964 + .quad 0xb4a155cb28d18df2 + .quad 0xeacc4646186ce508 + .quad 0xc49cf4936c824389 + .quad 0x27a6c809ae5d3410 + + // 2^96 * 6 * G + + .quad 0x8ba6ebcd1f0db188 + .quad 0x37d3d73a675a5be8 + .quad 0xf22edfa315f5585a + .quad 0x2cb67174ff60a17e + .quad 0xcd2c270ac43d6954 + .quad 0xdd4a3e576a66cab2 + .quad 0x79fa592469d7036c + .quad 0x221503603d8c2599 + .quad 0x59eecdf9390be1d0 + .quad 0xa9422044728ce3f1 + .quad 0x82891c667a94f0f4 + .quad 0x7b1df4b73890f436 + + // 2^96 * 7 * G + + .quad 0xe492f2e0b3b2a224 + .quad 0x7c6c9e062b551160 + .quad 0x15eb8fe20d7f7b0e + .quad 0x61fcef2658fc5992 + .quad 0x5f2e221807f8f58c + .quad 0xe3555c9fd49409d4 + .quad 0xb2aaa88d1fb6a630 + .quad 0x68698245d352e03d + .quad 0xdbb15d852a18187a + .quad 0xf3e4aad386ddacd7 + .quad 0x44bae2810ff6c482 + .quad 0x46cf4c473daf01cf + + // 2^96 * 8 * G + + .quad 0x426525ed9ec4e5f9 + .quad 0x0e5eda0116903303 + .quad 0x72b1a7f2cbe5cadc + .quad 0x29387bcd14eb5f40 + .quad 0x213c6ea7f1498140 + .quad 0x7c1e7ef8392b4854 + .quad 0x2488c38c5629ceba + .quad 0x1065aae50d8cc5bb + .quad 0x1c2c4525df200d57 + .quad 0x5c3b2dd6bfca674a + .quad 0x0a07e7b1e1834030 + .quad 0x69a198e64f1ce716 + + // 2^100 * 1 * G + + .quad 0x7afcd613efa9d697 + .quad 0x0cc45aa41c067959 + .quad 0xa56fe104c1fada96 + .quad 0x3a73b70472e40365 + .quad 0x7b26e56b9e2d4734 + .quad 0xc4c7132b81c61675 + .quad 0xef5c9525ec9cde7f + .quad 0x39c80b16e71743ad + .quad 0x0f196e0d1b826c68 + .quad 0xf71ff0e24960e3db + .quad 0x6113167023b7436c + .quad 0x0cf0ea5877da7282 + + // 2^100 * 2 * G + + .quad 0x196c80a4ddd4ccbd + .quad 0x22e6f55d95f2dd9d + .quad 0xc75e33c740d6c71b + .quad 0x7bb51279cb3c042f + .quad 0xe332ced43ba6945a + .quad 0xde0b1361e881c05d + .quad 0x1ad40f095e67ed3b + .quad 0x5da8acdab8c63d5d + .quad 0xc4b6664a3a70159f + .quad 0x76194f0f0a904e14 + .quad 0xa5614c39a4096c13 + .quad 0x6cd0ff50979feced + + // 2^100 * 3 * G + + .quad 0xc0e067e78f4428ac + .quad 0x14835ab0a61135e3 + .quad 0xf21d14f338062935 + .quad 0x6390a4c8df04849c + .quad 0x7fecfabdb04ba18e + .quad 0xd0fc7bfc3bddbcf7 + .quad 0xa41d486e057a131c + .quad 0x641a4391f2223a61 + .quad 0xc5c6b95aa606a8db + .quad 0x914b7f9eb06825f1 + .quad 0x2a731f6b44fc9eff + .quad 0x30ddf38562705cfc + + // 2^100 * 4 * G + + .quad 0x4e3dcbdad1bff7f9 + .quad 0xc9118e8220645717 + .quad 0xbacccebc0f189d56 + .quad 0x1b4822e9d4467668 + .quad 0x33bef2bd68bcd52c + .quad 0xc649dbb069482ef2 + .quad 0xb5b6ee0c41cb1aee + .quad 0x5c294d270212a7e5 + .quad 0xab360a7f25563781 + .quad 0x2512228a480f7958 + .quad 0xc75d05276114b4e3 + .quad 0x222d9625d976fe2a + + // 2^100 * 5 * G + + .quad 0x1c717f85b372ace1 + .quad 0x81930e694638bf18 + .quad 0x239cad056bc08b58 + .quad 0x0b34271c87f8fff4 + .quad 0x0f94be7e0a344f85 + .quad 0xeb2faa8c87f22c38 + .quad 0x9ce1e75e4ee16f0f + .quad 0x43e64e5418a08dea + .quad 0x8155e2521a35ce63 + .quad 0xbe100d4df912028e + .quad 0xbff80bf8a57ddcec + .quad 0x57342dc96d6bc6e4 + + // 2^100 * 6 * G + + .quad 0xefeef065c8ce5998 + .quad 0xbf029510b5cbeaa2 + .quad 0x8c64a10620b7c458 + .quad 0x35134fb231c24855 + .quad 0xf3c3bcb71e707bf6 + .quad 0x351d9b8c7291a762 + .quad 0x00502e6edad69a33 + .quad 0x522f521f1ec8807f + .quad 0x272c1f46f9a3902b + .quad 0xc91ba3b799657bcc + .quad 0xae614b304f8a1c0e + .quad 0x7afcaad70b99017b + + // 2^100 * 7 * G + + .quad 0xc25ded54a4b8be41 + .quad 0x902d13e11bb0e2dd + .quad 0x41f43233cde82ab2 + .quad 0x1085faa5c3aae7cb + .quad 0xa88141ecef842b6b + .quad 0x55e7b14797abe6c5 + .quad 0x8c748f9703784ffe + .quad 0x5b50a1f7afcd00b7 + .quad 0x9b840f66f1361315 + .quad 0x18462242701003e9 + .quad 0x65ed45fae4a25080 + .quad 0x0a2862393fda7320 + + // 2^100 * 8 * G + + .quad 0x46ab13c8347cbc9d + .quad 0x3849e8d499c12383 + .quad 0x4cea314087d64ac9 + .quad 0x1f354134b1a29ee7 + .quad 0x960e737b6ecb9d17 + .quad 0xfaf24948d67ceae1 + .quad 0x37e7a9b4d55e1b89 + .quad 0x5cb7173cb46c59eb + .quad 0x4a89e68b82b7abf0 + .quad 0xf41cd9279ba6b7b9 + .quad 0x16e6c210e18d876f + .quad 0x7cacdb0f7f1b09c6 + + // 2^104 * 1 * G + + .quad 0x9062b2e0d91a78bc + .quad 0x47c9889cc8509667 + .quad 0x9df54a66405070b8 + .quad 0x7369e6a92493a1bf + .quad 0xe1014434dcc5caed + .quad 0x47ed5d963c84fb33 + .quad 0x70019576ed86a0e7 + .quad 0x25b2697bd267f9e4 + .quad 0x9d673ffb13986864 + .quad 0x3ca5fbd9415dc7b8 + .quad 0xe04ecc3bdf273b5e + .quad 0x1420683db54e4cd2 + + // 2^104 * 2 * G + + .quad 0xb478bd1e249dd197 + .quad 0x620c35005e58c102 + .quad 0xfb02d32fccbaac5c + .quad 0x60b63bebf508a72d + .quad 0x34eebb6fc1cc5ad0 + .quad 0x6a1b0ce99646ac8b + .quad 0xd3b0da49a66bde53 + .quad 0x31e83b4161d081c1 + .quad 0x97e8c7129e062b4f + .quad 0x49e48f4f29320ad8 + .quad 0x5bece14b6f18683f + .quad 0x55cf1eb62d550317 + + // 2^104 * 3 * G + + .quad 0x5879101065c23d58 + .quad 0x8b9d086d5094819c + .quad 0xe2402fa912c55fa7 + .quad 0x669a6564570891d4 + .quad 0x3076b5e37df58c52 + .quad 0xd73ab9dde799cc36 + .quad 0xbd831ce34913ee20 + .quad 0x1a56fbaa62ba0133 + .quad 0x943e6b505c9dc9ec + .quad 0x302557bba77c371a + .quad 0x9873ae5641347651 + .quad 0x13c4836799c58a5c + + // 2^104 * 4 * G + + .quad 0x423a5d465ab3e1b9 + .quad 0xfc13c187c7f13f61 + .quad 0x19f83664ecb5b9b6 + .quad 0x66f80c93a637b607 + .quad 0xc4dcfb6a5d8bd080 + .quad 0xdeebc4ec571a4842 + .quad 0xd4b2e883b8e55365 + .quad 0x50bdc87dc8e5b827 + .quad 0x606d37836edfe111 + .quad 0x32353e15f011abd9 + .quad 0x64b03ac325b73b96 + .quad 0x1dd56444725fd5ae + + // 2^104 * 5 * G + + .quad 0x8fa47ff83362127d + .quad 0xbc9f6ac471cd7c15 + .quad 0x6e71454349220c8b + .quad 0x0e645912219f732e + .quad 0xc297e60008bac89a + .quad 0x7d4cea11eae1c3e0 + .quad 0xf3e38be19fe7977c + .quad 0x3a3a450f63a305cd + .quad 0x078f2f31d8394627 + .quad 0x389d3183de94a510 + .quad 0xd1e36c6d17996f80 + .quad 0x318c8d9393a9a87b + + // 2^104 * 6 * G + + .quad 0xf2745d032afffe19 + .quad 0x0c9f3c497f24db66 + .quad 0xbc98d3e3ba8598ef + .quad 0x224c7c679a1d5314 + .quad 0x5d669e29ab1dd398 + .quad 0xfc921658342d9e3b + .quad 0x55851dfdf35973cd + .quad 0x509a41c325950af6 + .quad 0xbdc06edca6f925e9 + .quad 0x793ef3f4641b1f33 + .quad 0x82ec12809d833e89 + .quad 0x05bff02328a11389 + + // 2^104 * 7 * G + + .quad 0x3632137023cae00b + .quad 0x544acf0ad1accf59 + .quad 0x96741049d21a1c88 + .quad 0x780b8cc3fa2a44a7 + .quad 0x6881a0dd0dc512e4 + .quad 0x4fe70dc844a5fafe + .quad 0x1f748e6b8f4a5240 + .quad 0x576277cdee01a3ea + .quad 0x1ef38abc234f305f + .quad 0x9a577fbd1405de08 + .quad 0x5e82a51434e62a0d + .quad 0x5ff418726271b7a1 + + // 2^104 * 8 * G + + .quad 0x398e080c1789db9d + .quad 0xa7602025f3e778f5 + .quad 0xfa98894c06bd035d + .quad 0x106a03dc25a966be + .quad 0xe5db47e813b69540 + .quad 0xf35d2a3b432610e1 + .quad 0xac1f26e938781276 + .quad 0x29d4db8ca0a0cb69 + .quad 0xd9ad0aaf333353d0 + .quad 0x38669da5acd309e5 + .quad 0x3c57658ac888f7f0 + .quad 0x4ab38a51052cbefa + + // 2^108 * 1 * G + + .quad 0xdfdacbee4324c0e9 + .quad 0x054442883f955bb7 + .quad 0xdef7aaa8ea31609f + .quad 0x68aee70642287cff + .quad 0xf68fe2e8809de054 + .quad 0xe3bc096a9c82bad1 + .quad 0x076353d40aadbf45 + .quad 0x7b9b1fb5dea1959e + .quad 0xf01cc8f17471cc0c + .quad 0x95242e37579082bb + .quad 0x27776093d3e46b5f + .quad 0x2d13d55a28bd85fb + + // 2^108 * 2 * G + + .quad 0xfac5d2065b35b8da + .quad 0xa8da8a9a85624bb7 + .quad 0xccd2ca913d21cd0f + .quad 0x6b8341ee8bf90d58 + .quad 0xbf019cce7aee7a52 + .quad 0xa8ded2b6e454ead3 + .quad 0x3c619f0b87a8bb19 + .quad 0x3619b5d7560916d8 + .quad 0x3579f26b0282c4b2 + .quad 0x64d592f24fafefae + .quad 0xb7cded7b28c8c7c0 + .quad 0x6a927b6b7173a8d7 + + // 2^108 * 3 * G + + .quad 0x1f6db24f986e4656 + .quad 0x1021c02ed1e9105b + .quad 0xf8ff3fff2cc0a375 + .quad 0x1d2a6bf8c6c82592 + .quad 0x8d7040863ece88eb + .quad 0xf0e307a980eec08c + .quad 0xac2250610d788fda + .quad 0x056d92a43a0d478d + .quad 0x1b05a196fc3da5a1 + .quad 0x77d7a8c243b59ed0 + .quad 0x06da3d6297d17918 + .quad 0x66fbb494f12353f7 + + // 2^108 * 4 * G + + .quad 0x751a50b9d85c0fb8 + .quad 0xd1afdc258bcf097b + .quad 0x2f16a6a38309a969 + .quad 0x14ddff9ee5b00659 + .quad 0xd6d70996f12309d6 + .quad 0xdbfb2385e9c3d539 + .quad 0x46d602b0f7552411 + .quad 0x270a0b0557843e0c + .quad 0x61ff0640a7862bcc + .quad 0x81cac09a5f11abfe + .quad 0x9047830455d12abb + .quad 0x19a4bde1945ae873 + + // 2^108 * 5 * G + + .quad 0x9b9f26f520a6200a + .quad 0x64804443cf13eaf8 + .quad 0x8a63673f8631edd3 + .quad 0x72bbbce11ed39dc1 + .quad 0x40c709dec076c49f + .quad 0x657bfaf27f3e53f6 + .quad 0x40662331eca042c4 + .quad 0x14b375487eb4df04 + .quad 0xae853c94ab66dc47 + .quad 0xeb62343edf762d6e + .quad 0xf08e0e186fb2f7d1 + .quad 0x4f0b1c02700ab37a + + // 2^108 * 6 * G + + .quad 0xe1706787d81951fa + .quad 0xa10a2c8eb290c77b + .quad 0xe7382fa03ed66773 + .quad 0x0a4d84710bcc4b54 + .quad 0x79fd21ccc1b2e23f + .quad 0x4ae7c281453df52a + .quad 0xc8172ec9d151486b + .quad 0x68abe9443e0a7534 + .quad 0xda12c6c407831dcb + .quad 0x0da230d74d5c510d + .quad 0x4ab1531e6bd404e1 + .quad 0x4106b166bcf440ef + + // 2^108 * 7 * G + + .quad 0x02e57a421cd23668 + .quad 0x4ad9fb5d0eaef6fd + .quad 0x954e6727b1244480 + .quad 0x7f792f9d2699f331 + .quad 0xa485ccd539e4ecf2 + .quad 0x5aa3f3ad0555bab5 + .quad 0x145e3439937df82d + .quad 0x1238b51e1214283f + .quad 0x0b886b925fd4d924 + .quad 0x60906f7a3626a80d + .quad 0xecd367b4b98abd12 + .quad 0x2876beb1def344cf + + // 2^108 * 8 * G + + .quad 0xdc84e93563144691 + .quad 0x632fe8a0d61f23f4 + .quad 0x4caa800612a9a8d5 + .quad 0x48f9dbfa0e9918d3 + .quad 0xd594b3333a8a85f8 + .quad 0x4ea37689e78d7d58 + .quad 0x73bf9f455e8e351f + .quad 0x5507d7d2bc41ebb4 + .quad 0x1ceb2903299572fc + .quad 0x7c8ccaa29502d0ee + .quad 0x91bfa43411cce67b + .quad 0x5784481964a831e7 + + // 2^112 * 1 * G + + .quad 0xda7c2b256768d593 + .quad 0x98c1c0574422ca13 + .quad 0xf1a80bd5ca0ace1d + .quad 0x29cdd1adc088a690 + .quad 0xd6cfd1ef5fddc09c + .quad 0xe82b3efdf7575dce + .quad 0x25d56b5d201634c2 + .quad 0x3041c6bb04ed2b9b + .quad 0x0ff2f2f9d956e148 + .quad 0xade797759f356b2e + .quad 0x1a4698bb5f6c025c + .quad 0x104bbd6814049a7b + + // 2^112 * 2 * G + + .quad 0x51f0fd3168f1ed67 + .quad 0x2c811dcdd86f3bc2 + .quad 0x44dc5c4304d2f2de + .quad 0x5be8cc57092a7149 + .quad 0xa95d9a5fd67ff163 + .quad 0xe92be69d4cc75681 + .quad 0xb7f8024cde20f257 + .quad 0x204f2a20fb072df5 + .quad 0xc8143b3d30ebb079 + .quad 0x7589155abd652e30 + .quad 0x653c3c318f6d5c31 + .quad 0x2570fb17c279161f + + // 2^112 * 3 * G + + .quad 0x3efa367f2cb61575 + .quad 0xf5f96f761cd6026c + .quad 0xe8c7142a65b52562 + .quad 0x3dcb65ea53030acd + .quad 0x192ea9550bb8245a + .quad 0xc8e6fba88f9050d1 + .quad 0x7986ea2d88a4c935 + .quad 0x241c5f91de018668 + .quad 0x28d8172940de6caa + .quad 0x8fbf2cf022d9733a + .quad 0x16d7fcdd235b01d1 + .quad 0x08420edd5fcdf0e5 + + // 2^112 * 4 * G + + .quad 0xcdff20ab8362fa4a + .quad 0x57e118d4e21a3e6e + .quad 0xe3179617fc39e62b + .quad 0x0d9a53efbc1769fd + .quad 0x0358c34e04f410ce + .quad 0xb6135b5a276e0685 + .quad 0x5d9670c7ebb91521 + .quad 0x04d654f321db889c + .quad 0x5e7dc116ddbdb5d5 + .quad 0x2954deb68da5dd2d + .quad 0x1cb608173334a292 + .quad 0x4a7a4f2618991ad7 + + // 2^112 * 5 * G + + .quad 0xf4a718025fb15f95 + .quad 0x3df65f346b5c1b8f + .quad 0xcdfcf08500e01112 + .quad 0x11b50c4cddd31848 + .quad 0x24c3b291af372a4b + .quad 0x93da8270718147f2 + .quad 0xdd84856486899ef2 + .quad 0x4a96314223e0ee33 + .quad 0xa6e8274408a4ffd6 + .quad 0x738e177e9c1576d9 + .quad 0x773348b63d02b3f2 + .quad 0x4f4bce4dce6bcc51 + + // 2^112 * 6 * G + + .quad 0xa71fce5ae2242584 + .quad 0x26ea725692f58a9e + .quad 0xd21a09d71cea3cf4 + .quad 0x73fcdd14b71c01e6 + .quad 0x30e2616ec49d0b6f + .quad 0xe456718fcaec2317 + .quad 0x48eb409bf26b4fa6 + .quad 0x3042cee561595f37 + .quad 0x427e7079449bac41 + .quad 0x855ae36dbce2310a + .quad 0x4cae76215f841a7c + .quad 0x389e740c9a9ce1d6 + + // 2^112 * 7 * G + + .quad 0x64fcb3ae34dcb9ce + .quad 0x97500323e348d0ad + .quad 0x45b3f07d62c6381b + .quad 0x61545379465a6788 + .quad 0xc9bd78f6570eac28 + .quad 0xe55b0b3227919ce1 + .quad 0x65fc3eaba19b91ed + .quad 0x25c425e5d6263690 + .quad 0x3f3e06a6f1d7de6e + .quad 0x3ef976278e062308 + .quad 0x8c14f6264e8a6c77 + .quad 0x6539a08915484759 + + // 2^112 * 8 * G + + .quad 0xe9d21f74c3d2f773 + .quad 0xc150544125c46845 + .quad 0x624e5ce8f9b99e33 + .quad 0x11c5e4aac5cd186c + .quad 0xddc4dbd414bb4a19 + .quad 0x19b2bc3c98424f8e + .quad 0x48a89fd736ca7169 + .quad 0x0f65320ef019bd90 + .quad 0xd486d1b1cafde0c6 + .quad 0x4f3fe6e3163b5181 + .quad 0x59a8af0dfaf2939a + .quad 0x4cabc7bdec33072a + + // 2^116 * 1 * G + + .quad 0x16faa8fb532f7428 + .quad 0xdbd42ea046a4e272 + .quad 0x5337653b8b9ea480 + .quad 0x4065947223973f03 + .quad 0xf7c0a19c1a54a044 + .quad 0x4a1c5e2477bd9fbb + .quad 0xa6e3ca115af22972 + .quad 0x1819bb953f2e9e0d + .quad 0x498fbb795e042e84 + .quad 0x7d0dd89a7698b714 + .quad 0x8bfb0ba427fe6295 + .quad 0x36ba82e721200524 + + // 2^116 * 2 * G + + .quad 0xd60ecbb74245ec41 + .quad 0xfd9be89e34348716 + .quad 0xc9240afee42284de + .quad 0x4472f648d0531db4 + .quad 0xc8d69d0a57274ed5 + .quad 0x45ba803260804b17 + .quad 0xdf3cda102255dfac + .quad 0x77d221232709b339 + .quad 0x498a6d7064ad94d8 + .quad 0xa5b5c8fd9af62263 + .quad 0x8ca8ed0545c141f4 + .quad 0x2c63bec3662d358c + + // 2^116 * 3 * G + + .quad 0x7fe60d8bea787955 + .quad 0xb9dc117eb5f401b7 + .quad 0x91c7c09a19355cce + .quad 0x22692ef59442bedf + .quad 0x9a518b3a8586f8bf + .quad 0x9ee71af6cbb196f0 + .quad 0xaa0625e6a2385cf2 + .quad 0x1deb2176ddd7c8d1 + .quad 0x8563d19a2066cf6c + .quad 0x401bfd8c4dcc7cd7 + .quad 0xd976a6becd0d8f62 + .quad 0x67cfd773a278b05e + + // 2^116 * 4 * G + + .quad 0x8dec31faef3ee475 + .quad 0x99dbff8a9e22fd92 + .quad 0x512d11594e26cab1 + .quad 0x0cde561eec4310b9 + .quad 0x2d5fa9855a4e586a + .quad 0x65f8f7a449beab7e + .quad 0xaa074dddf21d33d3 + .quad 0x185cba721bcb9dee + .quad 0x93869da3f4e3cb41 + .quad 0xbf0392f540f7977e + .quad 0x026204fcd0463b83 + .quad 0x3ec91a769eec6eed + + // 2^116 * 5 * G + + .quad 0x1e9df75bf78166ad + .quad 0x4dfda838eb0cd7af + .quad 0xba002ed8c1eaf988 + .quad 0x13fedb3e11f33cfc + .quad 0x0fad2fb7b0a3402f + .quad 0x46615ecbfb69f4a8 + .quad 0xf745bcc8c5f8eaa6 + .quad 0x7a5fa8794a94e896 + .quad 0x52958faa13cd67a1 + .quad 0x965ee0818bdbb517 + .quad 0x16e58daa2e8845b3 + .quad 0x357d397d5499da8f + + // 2^116 * 6 * G + + .quad 0x1ebfa05fb0bace6c + .quad 0xc934620c1caf9a1e + .quad 0xcc771cc41d82b61a + .quad 0x2d94a16aa5f74fec + .quad 0x481dacb4194bfbf8 + .quad 0x4d77e3f1bae58299 + .quad 0x1ef4612e7d1372a0 + .quad 0x3a8d867e70ff69e1 + .quad 0x6f58cd5d55aff958 + .quad 0xba3eaa5c75567721 + .quad 0x75c123999165227d + .quad 0x69be1343c2f2b35e + + // 2^116 * 7 * G + + .quad 0x0e091d5ee197c92a + .quad 0x4f51019f2945119f + .quad 0x143679b9f034e99c + .quad 0x7d88112e4d24c696 + .quad 0x82bbbdac684b8de3 + .quad 0xa2f4c7d03fca0718 + .quad 0x337f92fbe096aaa8 + .quad 0x200d4d8c63587376 + .quad 0x208aed4b4893b32b + .quad 0x3efbf23ebe59b964 + .quad 0xd762deb0dba5e507 + .quad 0x69607bd681bd9d94 + + // 2^116 * 8 * G + + .quad 0xf6be021068de1ce1 + .quad 0xe8d518e70edcbc1f + .quad 0xe3effdd01b5505a5 + .quad 0x35f63353d3ec3fd0 + .quad 0x3b7f3bd49323a902 + .quad 0x7c21b5566b2c6e53 + .quad 0xe5ba8ff53a7852a7 + .quad 0x28bc77a5838ece00 + .quad 0x63ba78a8e25d8036 + .quad 0x63651e0094333490 + .quad 0x48d82f20288ce532 + .quad 0x3a31abfa36b57524 + + // 2^120 * 1 * G + + .quad 0x239e9624089c0a2e + .quad 0xc748c4c03afe4738 + .quad 0x17dbed2a764fa12a + .quad 0x639b93f0321c8582 + .quad 0xc08f788f3f78d289 + .quad 0xfe30a72ca1404d9f + .quad 0xf2778bfccf65cc9d + .quad 0x7ee498165acb2021 + .quad 0x7bd508e39111a1c3 + .quad 0x2b2b90d480907489 + .quad 0xe7d2aec2ae72fd19 + .quad 0x0edf493c85b602a6 + + // 2^120 * 2 * G + + .quad 0xaecc8158599b5a68 + .quad 0xea574f0febade20e + .quad 0x4fe41d7422b67f07 + .quad 0x403b92e3019d4fb4 + .quad 0x6767c4d284764113 + .quad 0xa090403ff7f5f835 + .quad 0x1c8fcffacae6bede + .quad 0x04c00c54d1dfa369 + .quad 0x4dc22f818b465cf8 + .quad 0x71a0f35a1480eff8 + .quad 0xaee8bfad04c7d657 + .quad 0x355bb12ab26176f4 + + // 2^120 * 3 * G + + .quad 0xa71e64cc7493bbf4 + .quad 0xe5bd84d9eca3b0c3 + .quad 0x0a6bc50cfa05e785 + .quad 0x0f9b8132182ec312 + .quad 0xa301dac75a8c7318 + .quad 0xed90039db3ceaa11 + .quad 0x6f077cbf3bae3f2d + .quad 0x7518eaf8e052ad8e + .quad 0xa48859c41b7f6c32 + .quad 0x0f2d60bcf4383298 + .quad 0x1815a929c9b1d1d9 + .quad 0x47c3871bbb1755c4 + + // 2^120 * 4 * G + + .quad 0x5144539771ec4f48 + .quad 0xf805b17dc98c5d6e + .quad 0xf762c11a47c3c66b + .quad 0x00b89b85764699dc + .quad 0xfbe65d50c85066b0 + .quad 0x62ecc4b0b3a299b0 + .quad 0xe53754ea441ae8e0 + .quad 0x08fea02ce8d48d5f + .quad 0x824ddd7668deead0 + .quad 0xc86445204b685d23 + .quad 0xb514cfcd5d89d665 + .quad 0x473829a74f75d537 + + // 2^120 * 5 * G + + .quad 0x82d2da754679c418 + .quad 0xe63bd7d8b2618df0 + .quad 0x355eef24ac47eb0a + .quad 0x2078684c4833c6b4 + .quad 0x23d9533aad3902c9 + .quad 0x64c2ddceef03588f + .quad 0x15257390cfe12fb4 + .quad 0x6c668b4d44e4d390 + .quad 0x3b48cf217a78820c + .quad 0xf76a0ab281273e97 + .quad 0xa96c65a78c8eed7b + .quad 0x7411a6054f8a433f + + // 2^120 * 6 * G + + .quad 0x4d659d32b99dc86d + .quad 0x044cdc75603af115 + .quad 0xb34c712cdcc2e488 + .quad 0x7c136574fb8134ff + .quad 0x579ae53d18b175b4 + .quad 0x68713159f392a102 + .quad 0x8455ecba1eef35f5 + .quad 0x1ec9a872458c398f + .quad 0xb8e6a4d400a2509b + .quad 0x9b81d7020bc882b4 + .quad 0x57e7cc9bf1957561 + .quad 0x3add88a5c7cd6460 + + // 2^120 * 7 * G + + .quad 0xab895770b635dcf2 + .quad 0x02dfef6cf66c1fbc + .quad 0x85530268beb6d187 + .quad 0x249929fccc879e74 + .quad 0x85c298d459393046 + .quad 0x8f7e35985ff659ec + .quad 0x1d2ca22af2f66e3a + .quad 0x61ba1131a406a720 + .quad 0xa3d0a0f116959029 + .quad 0x023b6b6cba7ebd89 + .quad 0x7bf15a3e26783307 + .quad 0x5620310cbbd8ece7 + + // 2^120 * 8 * G + + .quad 0x528993434934d643 + .quad 0xb9dbf806a51222f5 + .quad 0x8f6d878fc3f41c22 + .quad 0x37676a2a4d9d9730 + .quad 0x6646b5f477e285d6 + .quad 0x40e8ff676c8f6193 + .quad 0xa6ec7311abb594dd + .quad 0x7ec846f3658cec4d + .quad 0x9b5e8f3f1da22ec7 + .quad 0x130f1d776c01cd13 + .quad 0x214c8fcfa2989fb8 + .quad 0x6daaf723399b9dd5 + + // 2^124 * 1 * G + + .quad 0x591e4a5610628564 + .quad 0x2a4bb87ca8b4df34 + .quad 0xde2a2572e7a38e43 + .quad 0x3cbdabd9fee5046e + .quad 0x81aebbdd2cd13070 + .quad 0x962e4325f85a0e9e + .quad 0xde9391aacadffecb + .quad 0x53177fda52c230e6 + .quad 0xa7bc970650b9de79 + .quad 0x3d12a7fbc301b59b + .quad 0x02652e68d36ae38c + .quad 0x79d739835a6199dc + + // 2^124 * 2 * G + + .quad 0xd9354df64131c1bd + .quad 0x758094a186ec5822 + .quad 0x4464ee12e459f3c2 + .quad 0x6c11fce4cb133282 + .quad 0x21c9d9920d591737 + .quad 0x9bea41d2e9b46cd6 + .quad 0xe20e84200d89bfca + .quad 0x79d99f946eae5ff8 + .quad 0xf17b483568673205 + .quad 0x387deae83caad96c + .quad 0x61b471fd56ffe386 + .quad 0x31741195b745a599 + + // 2^124 * 3 * G + + .quad 0xe8d10190b77a360b + .quad 0x99b983209995e702 + .quad 0xbd4fdff8fa0247aa + .quad 0x2772e344e0d36a87 + .quad 0x17f8ba683b02a047 + .quad 0x50212096feefb6c8 + .quad 0x70139be21556cbe2 + .quad 0x203e44a11d98915b + .quad 0xd6863eba37b9e39f + .quad 0x105bc169723b5a23 + .quad 0x104f6459a65c0762 + .quad 0x567951295b4d38d4 + + // 2^124 * 4 * G + + .quad 0x535fd60613037524 + .quad 0xe210adf6b0fbc26a + .quad 0xac8d0a9b23e990ae + .quad 0x47204d08d72fdbf9 + .quad 0x07242eb30d4b497f + .quad 0x1ef96306b9bccc87 + .quad 0x37950934d8116f45 + .quad 0x05468d6201405b04 + .quad 0x00f565a9f93267de + .quad 0xcecfd78dc0d58e8a + .quad 0xa215e2dcf318e28e + .quad 0x4599ee919b633352 + + // 2^124 * 5 * G + + .quad 0xd3c220ca70e0e76b + .quad 0xb12bea58ea9f3094 + .quad 0x294ddec8c3271282 + .quad 0x0c3539e1a1d1d028 + .quad 0xac746d6b861ae579 + .quad 0x31ab0650f6aea9dc + .quad 0x241d661140256d4c + .quad 0x2f485e853d21a5de + .quad 0x329744839c0833f3 + .quad 0x6fe6257fd2abc484 + .quad 0x5327d1814b358817 + .quad 0x65712585893fe9bc + + // 2^124 * 6 * G + + .quad 0x9c102fb732a61161 + .quad 0xe48e10dd34d520a8 + .quad 0x365c63546f9a9176 + .quad 0x32f6fe4c046f6006 + .quad 0x81c29f1bd708ee3f + .quad 0xddcb5a05ae6407d0 + .quad 0x97aec1d7d2a3eba7 + .quad 0x1590521a91d50831 + .quad 0x40a3a11ec7910acc + .quad 0x9013dff8f16d27ae + .quad 0x1a9720d8abb195d4 + .quad 0x1bb9fe452ea98463 + + // 2^124 * 7 * G + + .quad 0xe9d1d950b3d54f9e + .quad 0x2d5f9cbee00d33c1 + .quad 0x51c2c656a04fc6ac + .quad 0x65c091ee3c1cbcc9 + .quad 0xcf5e6c95cc36747c + .quad 0x294201536b0bc30d + .quad 0x453ac67cee797af0 + .quad 0x5eae6ab32a8bb3c9 + .quad 0x7083661114f118ea + .quad 0x2b37b87b94349cad + .quad 0x7273f51cb4e99f40 + .quad 0x78a2a95823d75698 + + // 2^124 * 8 * G + + .quad 0xa2b072e95c8c2ace + .quad 0x69cffc96651e9c4b + .quad 0x44328ef842e7b42b + .quad 0x5dd996c122aadeb3 + .quad 0xb4f23c425ef83207 + .quad 0xabf894d3c9a934b5 + .quad 0xd0708c1339fd87f7 + .quad 0x1876789117166130 + .quad 0x925b5ef0670c507c + .quad 0x819bc842b93c33bf + .quad 0x10792e9a70dd003f + .quad 0x59ad4b7a6e28dc74 + + // 2^128 * 1 * G + + .quad 0x5f3a7562eb3dbe47 + .quad 0xf7ea38548ebda0b8 + .quad 0x00c3e53145747299 + .quad 0x1304e9e71627d551 + .quad 0x583b04bfacad8ea2 + .quad 0x29b743e8148be884 + .quad 0x2b1e583b0810c5db + .quad 0x2b5449e58eb3bbaa + .quad 0x789814d26adc9cfe + .quad 0x3c1bab3f8b48dd0b + .quad 0xda0fe1fff979c60a + .quad 0x4468de2d7c2dd693 + + // 2^128 * 2 * G + + .quad 0x51bb355e9419469e + .quad 0x33e6dc4c23ddc754 + .quad 0x93a5b6d6447f9962 + .quad 0x6cce7c6ffb44bd63 + .quad 0x4b9ad8c6f86307ce + .quad 0x21113531435d0c28 + .quad 0xd4a866c5657a772c + .quad 0x5da6427e63247352 + .quad 0x1a94c688deac22ca + .quad 0xb9066ef7bbae1ff8 + .quad 0x88ad8c388d59580f + .quad 0x58f29abfe79f2ca8 + + // 2^128 * 3 * G + + .quad 0xe90ecfab8de73e68 + .quad 0x54036f9f377e76a5 + .quad 0xf0495b0bbe015982 + .quad 0x577629c4a7f41e36 + .quad 0x4b5a64bf710ecdf6 + .quad 0xb14ce538462c293c + .quad 0x3643d056d50b3ab9 + .quad 0x6af93724185b4870 + .quad 0x3220024509c6a888 + .quad 0xd2e036134b558973 + .quad 0x83e236233c33289f + .quad 0x701f25bb0caec18f + + // 2^128 * 4 * G + + .quad 0xc3a8b0f8e4616ced + .quad 0xf700660e9e25a87d + .quad 0x61e3061ff4bca59c + .quad 0x2e0c92bfbdc40be9 + .quad 0x9d18f6d97cbec113 + .quad 0x844a06e674bfdbe4 + .quad 0x20f5b522ac4e60d6 + .quad 0x720a5bc050955e51 + .quad 0x0c3f09439b805a35 + .quad 0xe84e8b376242abfc + .quad 0x691417f35c229346 + .quad 0x0e9b9cbb144ef0ec + + // 2^128 * 5 * G + + .quad 0xfbbad48ffb5720ad + .quad 0xee81916bdbf90d0e + .quad 0xd4813152635543bf + .quad 0x221104eb3f337bd8 + .quad 0x8dee9bd55db1beee + .quad 0xc9c3ab370a723fb9 + .quad 0x44a8f1bf1c68d791 + .quad 0x366d44191cfd3cde + .quad 0x9e3c1743f2bc8c14 + .quad 0x2eda26fcb5856c3b + .quad 0xccb82f0e68a7fb97 + .quad 0x4167a4e6bc593244 + + // 2^128 * 6 * G + + .quad 0x643b9d2876f62700 + .quad 0x5d1d9d400e7668eb + .quad 0x1b4b430321fc0684 + .quad 0x7938bb7e2255246a + .quad 0xc2be2665f8ce8fee + .quad 0xe967ff14e880d62c + .quad 0xf12e6e7e2f364eee + .quad 0x34b33370cb7ed2f6 + .quad 0xcdc591ee8681d6cc + .quad 0xce02109ced85a753 + .quad 0xed7485c158808883 + .quad 0x1176fc6e2dfe65e4 + + // 2^128 * 7 * G + + .quad 0xb4af6cd05b9c619b + .quad 0x2ddfc9f4b2a58480 + .quad 0x3d4fa502ebe94dc4 + .quad 0x08fc3a4c677d5f34 + .quad 0xdb90e28949770eb8 + .quad 0x98fbcc2aacf440a3 + .quad 0x21354ffeded7879b + .quad 0x1f6a3e54f26906b6 + .quad 0x60a4c199d30734ea + .quad 0x40c085b631165cd6 + .quad 0xe2333e23f7598295 + .quad 0x4f2fad0116b900d1 + + // 2^128 * 8 * G + + .quad 0x44beb24194ae4e54 + .quad 0x5f541c511857ef6c + .quad 0xa61e6b2d368d0498 + .quad 0x445484a4972ef7ab + .quad 0x962cd91db73bb638 + .quad 0xe60577aafc129c08 + .quad 0x6f619b39f3b61689 + .quad 0x3451995f2944ee81 + .quad 0x9152fcd09fea7d7c + .quad 0x4a816c94b0935cf6 + .quad 0x258e9aaa47285c40 + .quad 0x10b89ca6042893b7 + + // 2^132 * 1 * G + + .quad 0x9b2a426e3b646025 + .quad 0x32127190385ce4cf + .quad 0xa25cffc2dd6dea45 + .quad 0x06409010bea8de75 + .quad 0xd67cded679d34aa0 + .quad 0xcc0b9ec0cc4db39f + .quad 0xa535a456e35d190f + .quad 0x2e05d9eaf61f6fef + .quad 0xc447901ad61beb59 + .quad 0x661f19bce5dc880a + .quad 0x24685482b7ca6827 + .quad 0x293c778cefe07f26 + + // 2^132 * 2 * G + + .quad 0x86809e7007069096 + .quad 0xaad75b15e4e50189 + .quad 0x07f35715a21a0147 + .quad 0x0487f3f112815d5e + .quad 0x16c795d6a11ff200 + .quad 0xcb70d0e2b15815c9 + .quad 0x89f293209b5395b5 + .quad 0x50b8c2d031e47b4f + .quad 0x48350c08068a4962 + .quad 0x6ffdd05351092c9a + .quad 0x17af4f4aaf6fc8dd + .quad 0x4b0553b53cdba58b + + // 2^132 * 3 * G + + .quad 0x9c65fcbe1b32ff79 + .quad 0xeb75ea9f03b50f9b + .quad 0xfced2a6c6c07e606 + .quad 0x35106cd551717908 + .quad 0xbf05211b27c152d4 + .quad 0x5ec26849bd1af639 + .quad 0x5e0b2caa8e6fab98 + .quad 0x054c8bdd50bd0840 + .quad 0x38a0b12f1dcf073d + .quad 0x4b60a8a3b7f6a276 + .quad 0xfed5ac25d3404f9a + .quad 0x72e82d5e5505c229 + + // 2^132 * 4 * G + + .quad 0x6b0b697ff0d844c8 + .quad 0xbb12f85cd979cb49 + .quad 0xd2a541c6c1da0f1f + .quad 0x7b7c242958ce7211 + .quad 0x00d9cdfd69771d02 + .quad 0x410276cd6cfbf17e + .quad 0x4c45306c1cb12ec7 + .quad 0x2857bf1627500861 + .quad 0x9f21903f0101689e + .quad 0xd779dfd3bf861005 + .quad 0xa122ee5f3deb0f1b + .quad 0x510df84b485a00d4 + + // 2^132 * 5 * G + + .quad 0xa54133bb9277a1fa + .quad 0x74ec3b6263991237 + .quad 0x1a3c54dc35d2f15a + .quad 0x2d347144e482ba3a + .quad 0x24b3c887c70ac15e + .quad 0xb0f3a557fb81b732 + .quad 0x9b2cde2fe578cc1b + .quad 0x4cf7ed0703b54f8e + .quad 0x6bd47c6598fbee0f + .quad 0x9e4733e2ab55be2d + .quad 0x1093f624127610c5 + .quad 0x4e05e26ad0a1eaa4 + + // 2^132 * 6 * G + + .quad 0xda9b6b624b531f20 + .quad 0x429a760e77509abb + .quad 0xdbe9f522e823cb80 + .quad 0x618f1856880c8f82 + .quad 0x1833c773e18fe6c0 + .quad 0xe3c4711ad3c87265 + .quad 0x3bfd3c4f0116b283 + .quad 0x1955875eb4cd4db8 + .quad 0x6da6de8f0e399799 + .quad 0x7ad61aa440fda178 + .quad 0xb32cd8105e3563dd + .quad 0x15f6beae2ae340ae + + // 2^132 * 7 * G + + .quad 0x862bcb0c31ec3a62 + .quad 0x810e2b451138f3c2 + .quad 0x788ec4b839dac2a4 + .quad 0x28f76867ae2a9281 + .quad 0xba9a0f7b9245e215 + .quad 0xf368612dd98c0dbb + .quad 0x2e84e4cbf220b020 + .quad 0x6ba92fe962d90eda + .quad 0x3e4df9655884e2aa + .quad 0xbd62fbdbdbd465a5 + .quad 0xd7596caa0de9e524 + .quad 0x6e8042ccb2b1b3d7 + + // 2^132 * 8 * G + + .quad 0xf10d3c29ce28ca6e + .quad 0xbad34540fcb6093d + .quad 0xe7426ed7a2ea2d3f + .quad 0x08af9d4e4ff298b9 + .quad 0x1530653616521f7e + .quad 0x660d06b896203dba + .quad 0x2d3989bc545f0879 + .quad 0x4b5303af78ebd7b0 + .quad 0x72f8a6c3bebcbde8 + .quad 0x4f0fca4adc3a8e89 + .quad 0x6fa9d4e8c7bfdf7a + .quad 0x0dcf2d679b624eb7 + + // 2^136 * 1 * G + + .quad 0x3d5947499718289c + .quad 0x12ebf8c524533f26 + .quad 0x0262bfcb14c3ef15 + .quad 0x20b878d577b7518e + .quad 0x753941be5a45f06e + .quad 0xd07caeed6d9c5f65 + .quad 0x11776b9c72ff51b6 + .quad 0x17d2d1d9ef0d4da9 + .quad 0x27f2af18073f3e6a + .quad 0xfd3fe519d7521069 + .quad 0x22e3b72c3ca60022 + .quad 0x72214f63cc65c6a7 + + // 2^136 * 2 * G + + .quad 0xb4e37f405307a693 + .quad 0xaba714d72f336795 + .quad 0xd6fbd0a773761099 + .quad 0x5fdf48c58171cbc9 + .quad 0x1d9db7b9f43b29c9 + .quad 0xd605824a4f518f75 + .quad 0xf2c072bd312f9dc4 + .quad 0x1f24ac855a1545b0 + .quad 0x24d608328e9505aa + .quad 0x4748c1d10c1420ee + .quad 0xc7ffe45c06fb25a2 + .quad 0x00ba739e2ae395e6 + + // 2^136 * 3 * G + + .quad 0x592e98de5c8790d6 + .quad 0xe5bfb7d345c2a2df + .quad 0x115a3b60f9b49922 + .quad 0x03283a3e67ad78f3 + .quad 0xae4426f5ea88bb26 + .quad 0x360679d984973bfb + .quad 0x5c9f030c26694e50 + .quad 0x72297de7d518d226 + .quad 0x48241dc7be0cb939 + .quad 0x32f19b4d8b633080 + .quad 0xd3dfc90d02289308 + .quad 0x05e1296846271945 + + // 2^136 * 4 * G + + .quad 0xba82eeb32d9c495a + .quad 0xceefc8fcf12bb97c + .quad 0xb02dabae93b5d1e0 + .quad 0x39c00c9c13698d9b + .quad 0xadbfbbc8242c4550 + .quad 0xbcc80cecd03081d9 + .quad 0x843566a6f5c8df92 + .quad 0x78cf25d38258ce4c + .quad 0x15ae6b8e31489d68 + .quad 0xaa851cab9c2bf087 + .quad 0xc9a75a97f04efa05 + .quad 0x006b52076b3ff832 + + // 2^136 * 5 * G + + .quad 0x29e0cfe19d95781c + .quad 0xb681df18966310e2 + .quad 0x57df39d370516b39 + .quad 0x4d57e3443bc76122 + .quad 0xf5cb7e16b9ce082d + .quad 0x3407f14c417abc29 + .quad 0xd4b36bce2bf4a7ab + .quad 0x7de2e9561a9f75ce + .quad 0xde70d4f4b6a55ecb + .quad 0x4801527f5d85db99 + .quad 0xdbc9c440d3ee9a81 + .quad 0x6b2a90af1a6029ed + + // 2^136 * 6 * G + + .quad 0x6923f4fc9ae61e97 + .quad 0x5735281de03f5fd1 + .quad 0xa764ae43e6edd12d + .quad 0x5fd8f4e9d12d3e4a + .quad 0x77ebf3245bb2d80a + .quad 0xd8301b472fb9079b + .quad 0xc647e6f24cee7333 + .quad 0x465812c8276c2109 + .quad 0x4d43beb22a1062d9 + .quad 0x7065fb753831dc16 + .quad 0x180d4a7bde2968d7 + .quad 0x05b32c2b1cb16790 + + // 2^136 * 7 * G + + .quad 0xc8c05eccd24da8fd + .quad 0xa1cf1aac05dfef83 + .quad 0xdbbeeff27df9cd61 + .quad 0x3b5556a37b471e99 + .quad 0xf7fca42c7ad58195 + .quad 0x3214286e4333f3cc + .quad 0xb6c29d0d340b979d + .quad 0x31771a48567307e1 + .quad 0x32b0c524e14dd482 + .quad 0xedb351541a2ba4b6 + .quad 0xa3d16048282b5af3 + .quad 0x4fc079d27a7336eb + + // 2^136 * 8 * G + + .quad 0x51c938b089bf2f7f + .quad 0x2497bd6502dfe9a7 + .quad 0xffffc09c7880e453 + .quad 0x124567cecaf98e92 + .quad 0xdc348b440c86c50d + .quad 0x1337cbc9cc94e651 + .quad 0x6422f74d643e3cb9 + .quad 0x241170c2bae3cd08 + .quad 0x3ff9ab860ac473b4 + .quad 0xf0911dee0113e435 + .quad 0x4ae75060ebc6c4af + .quad 0x3f8612966c87000d + + // 2^140 * 1 * G + + .quad 0x0c9c5303f7957be4 + .quad 0xa3c31a20e085c145 + .quad 0xb0721d71d0850050 + .quad 0x0aba390eab0bf2da + .quad 0x529fdffe638c7bf3 + .quad 0xdf2b9e60388b4995 + .quad 0xe027b34f1bad0249 + .quad 0x7bc92fc9b9fa74ed + .quad 0x9f97ef2e801ad9f9 + .quad 0x83697d5479afda3a + .quad 0xe906b3ffbd596b50 + .quad 0x02672b37dd3fb8e0 + + // 2^140 * 2 * G + + .quad 0x48b2ca8b260885e4 + .quad 0xa4286bec82b34c1c + .quad 0x937e1a2617f58f74 + .quad 0x741d1fcbab2ca2a5 + .quad 0xee9ba729398ca7f5 + .quad 0xeb9ca6257a4849db + .quad 0x29eb29ce7ec544e1 + .quad 0x232ca21ef736e2c8 + .quad 0xbf61423d253fcb17 + .quad 0x08803ceafa39eb14 + .quad 0xf18602df9851c7af + .quad 0x0400f3a049e3414b + + // 2^140 * 3 * G + + .quad 0xabce0476ba61c55b + .quad 0x36a3d6d7c4d39716 + .quad 0x6eb259d5e8d82d09 + .quad 0x0c9176e984d756fb + .quad 0x2efba412a06e7b06 + .quad 0x146785452c8d2560 + .quad 0xdf9713ebd67a91c7 + .quad 0x32830ac7157eadf3 + .quad 0x0e782a7ab73769e8 + .quad 0x04a05d7875b18e2c + .quad 0x29525226ebcceae1 + .quad 0x0d794f8383eba820 + + // 2^140 * 4 * G + + .quad 0xff35f5cb9e1516f4 + .quad 0xee805bcf648aae45 + .quad 0xf0d73c2bb93a9ef3 + .quad 0x097b0bf22092a6c2 + .quad 0x7be44ce7a7a2e1ac + .quad 0x411fd93efad1b8b7 + .quad 0x1734a1d70d5f7c9b + .quad 0x0d6592233127db16 + .quad 0xc48bab1521a9d733 + .quad 0xa6c2eaead61abb25 + .quad 0x625c6c1cc6cb4305 + .quad 0x7fc90fea93eb3a67 + + // 2^140 * 5 * G + + .quad 0x0408f1fe1f5c5926 + .quad 0x1a8f2f5e3b258bf4 + .quad 0x40a951a2fdc71669 + .quad 0x6598ee93c98b577e + .quad 0xc527deb59c7cb23d + .quad 0x955391695328404e + .quad 0xd64392817ccf2c7a + .quad 0x6ce97dabf7d8fa11 + .quad 0x25b5a8e50ef7c48f + .quad 0xeb6034116f2ce532 + .quad 0xc5e75173e53de537 + .quad 0x73119fa08c12bb03 + + // 2^140 * 6 * G + + .quad 0xed30129453f1a4cb + .quad 0xbce621c9c8f53787 + .quad 0xfacb2b1338bee7b9 + .quad 0x3025798a9ea8428c + .quad 0x7845b94d21f4774d + .quad 0xbf62f16c7897b727 + .quad 0x671857c03c56522b + .quad 0x3cd6a85295621212 + .quad 0x3fecde923aeca999 + .quad 0xbdaa5b0062e8c12f + .quad 0x67b99dfc96988ade + .quad 0x3f52c02852661036 + + // 2^140 * 7 * G + + .quad 0xffeaa48e2a1351c6 + .quad 0x28624754fa7f53d7 + .quad 0x0b5ba9e57582ddf1 + .quad 0x60c0104ba696ac59 + .quad 0x9258bf99eec416c6 + .quad 0xac8a5017a9d2f671 + .quad 0x629549ab16dea4ab + .quad 0x05d0e85c99091569 + .quad 0x051de020de9cbe97 + .quad 0xfa07fc56b50bcf74 + .quad 0x378cec9f0f11df65 + .quad 0x36853c69ab96de4d + + // 2^140 * 8 * G + + .quad 0x36d9b8de78f39b2d + .quad 0x7f42ed71a847b9ec + .quad 0x241cd1d679bd3fde + .quad 0x6a704fec92fbce6b + .quad 0x4433c0b0fac5e7be + .quad 0x724bae854c08dcbe + .quad 0xf1f24cc446978f9b + .quad 0x4a0aff6d62825fc8 + .quad 0xe917fb9e61095301 + .quad 0xc102df9402a092f8 + .quad 0xbf09e2f5fa66190b + .quad 0x681109bee0dcfe37 + + // 2^144 * 1 * G + + .quad 0x559a0cc9782a0dde + .quad 0x551dcdb2ea718385 + .quad 0x7f62865b31ef238c + .quad 0x504aa7767973613d + .quad 0x9c18fcfa36048d13 + .quad 0x29159db373899ddd + .quad 0xdc9f350b9f92d0aa + .quad 0x26f57eee878a19d4 + .quad 0x0cab2cd55687efb1 + .quad 0x5180d162247af17b + .quad 0x85c15a344f5a2467 + .quad 0x4041943d9dba3069 + + // 2^144 * 2 * G + + .quad 0xc3c0eeba43ebcc96 + .quad 0x8d749c9c26ea9caf + .quad 0xd9fa95ee1c77ccc6 + .quad 0x1420a1d97684340f + .quad 0x4b217743a26caadd + .quad 0x47a6b424648ab7ce + .quad 0xcb1d4f7a03fbc9e3 + .quad 0x12d931429800d019 + .quad 0x00c67799d337594f + .quad 0x5e3c5140b23aa47b + .quad 0x44182854e35ff395 + .quad 0x1b4f92314359a012 + + // 2^144 * 3 * G + + .quad 0x3e5c109d89150951 + .quad 0x39cefa912de9696a + .quad 0x20eae43f975f3020 + .quad 0x239b572a7f132dae + .quad 0x33cf3030a49866b1 + .quad 0x251f73d2215f4859 + .quad 0xab82aa4051def4f6 + .quad 0x5ff191d56f9a23f6 + .quad 0x819ed433ac2d9068 + .quad 0x2883ab795fc98523 + .quad 0xef4572805593eb3d + .quad 0x020c526a758f36cb + + // 2^144 * 4 * G + + .quad 0x779834f89ed8dbbc + .quad 0xc8f2aaf9dc7ca46c + .quad 0xa9524cdca3e1b074 + .quad 0x02aacc4615313877 + .quad 0xe931ef59f042cc89 + .quad 0x2c589c9d8e124bb6 + .quad 0xadc8e18aaec75997 + .quad 0x452cfe0a5602c50c + .quad 0x86a0f7a0647877df + .quad 0xbbc464270e607c9f + .quad 0xab17ea25f1fb11c9 + .quad 0x4cfb7d7b304b877b + + // 2^144 * 5 * G + + .quad 0x72b43d6cb89b75fe + .quad 0x54c694d99c6adc80 + .quad 0xb8c3aa373ee34c9f + .quad 0x14b4622b39075364 + .quad 0xe28699c29789ef12 + .quad 0x2b6ecd71df57190d + .quad 0xc343c857ecc970d0 + .quad 0x5b1d4cbc434d3ac5 + .quad 0xb6fb2615cc0a9f26 + .quad 0x3a4f0e2bb88dcce5 + .quad 0x1301498b3369a705 + .quad 0x2f98f71258592dd1 + + // 2^144 * 6 * G + + .quad 0x0c94a74cb50f9e56 + .quad 0x5b1ff4a98e8e1320 + .quad 0x9a2acc2182300f67 + .quad 0x3a6ae249d806aaf9 + .quad 0x2e12ae444f54a701 + .quad 0xfcfe3ef0a9cbd7de + .quad 0xcebf890d75835de0 + .quad 0x1d8062e9e7614554 + .quad 0x657ada85a9907c5a + .quad 0x1a0ea8b591b90f62 + .quad 0x8d0e1dfbdf34b4e9 + .quad 0x298b8ce8aef25ff3 + + // 2^144 * 7 * G + + .quad 0x2a927953eff70cb2 + .quad 0x4b89c92a79157076 + .quad 0x9418457a30a7cf6a + .quad 0x34b8a8404d5ce485 + .quad 0x837a72ea0a2165de + .quad 0x3fab07b40bcf79f6 + .quad 0x521636c77738ae70 + .quad 0x6ba6271803a7d7dc + .quad 0xc26eecb583693335 + .quad 0xd5a813df63b5fefd + .quad 0xa293aa9aa4b22573 + .quad 0x71d62bdd465e1c6a + + // 2^144 * 8 * G + + .quad 0x6533cc28d378df80 + .quad 0xf6db43790a0fa4b4 + .quad 0xe3645ff9f701da5a + .quad 0x74d5f317f3172ba4 + .quad 0xcd2db5dab1f75ef5 + .quad 0xd77f95cf16b065f5 + .quad 0x14571fea3f49f085 + .quad 0x1c333621262b2b3d + .quad 0xa86fe55467d9ca81 + .quad 0x398b7c752b298c37 + .quad 0xda6d0892e3ac623b + .quad 0x4aebcc4547e9d98c + + // 2^148 * 1 * G + + .quad 0x53175a7205d21a77 + .quad 0xb0c04422d3b934d4 + .quad 0xadd9f24bdd5deadc + .quad 0x074f46e69f10ff8c + .quad 0x0de9b204a059a445 + .quad 0xe15cb4aa4b17ad0f + .quad 0xe1bbec521f79c557 + .quad 0x2633f1b9d071081b + .quad 0xc1fb4177018b9910 + .quad 0xa6ea20dc6c0fe140 + .quad 0xd661f3e74354c6ff + .quad 0x5ecb72e6f1a3407a + + // 2^148 * 2 * G + + .quad 0xa515a31b2259fb4e + .quad 0x0960f3972bcac52f + .quad 0xedb52fec8d3454cb + .quad 0x382e2720c476c019 + .quad 0xfeeae106e8e86997 + .quad 0x9863337f98d09383 + .quad 0x9470480eaa06ebef + .quad 0x038b6898d4c5c2d0 + .quad 0xf391c51d8ace50a6 + .quad 0x3142d0b9ae2d2948 + .quad 0xdb4d5a1a7f24ca80 + .quad 0x21aeba8b59250ea8 + + // 2^148 * 3 * G + + .quad 0x24f13b34cf405530 + .quad 0x3c44ea4a43088af7 + .quad 0x5dd5c5170006a482 + .quad 0x118eb8f8890b086d + .quad 0x53853600f0087f23 + .quad 0x4c461879da7d5784 + .quad 0x6af303deb41f6860 + .quad 0x0a3c16c5c27c18ed + .quad 0x17e49c17cc947f3d + .quad 0xccc6eda6aac1d27b + .quad 0xdf6092ceb0f08e56 + .quad 0x4909b3e22c67c36b + + // 2^148 * 4 * G + + .quad 0x9c9c85ea63fe2e89 + .quad 0xbe1baf910e9412ec + .quad 0x8f7baa8a86fbfe7b + .quad 0x0fb17f9fef968b6c + .quad 0x59a16676706ff64e + .quad 0x10b953dd0d86a53d + .quad 0x5848e1e6ce5c0b96 + .quad 0x2d8b78e712780c68 + .quad 0x79d5c62eafc3902b + .quad 0x773a215289e80728 + .quad 0xc38ae640e10120b9 + .quad 0x09ae23717b2b1a6d + + // 2^148 * 5 * G + + .quad 0xbb6a192a4e4d083c + .quad 0x34ace0630029e192 + .quad 0x98245a59aafabaeb + .quad 0x6d9c8a9ada97faac + .quad 0x10ab8fa1ad32b1d0 + .quad 0xe9aced1be2778b24 + .quad 0xa8856bc0373de90f + .quad 0x66f35ddddda53996 + .quad 0xd27d9afb24997323 + .quad 0x1bb7e07ef6f01d2e + .quad 0x2ba7472df52ecc7f + .quad 0x03019b4f646f9dc8 + + // 2^148 * 6 * G + + .quad 0x04a186b5565345cd + .quad 0xeee76610bcc4116a + .quad 0x689c73b478fb2a45 + .quad 0x387dcbff65697512 + .quad 0xaf09b214e6b3dc6b + .quad 0x3f7573b5ad7d2f65 + .quad 0xd019d988100a23b0 + .quad 0x392b63a58b5c35f7 + .quad 0x4093addc9c07c205 + .quad 0xc565be15f532c37e + .quad 0x63dbecfd1583402a + .quad 0x61722b4aef2e032e + + // 2^148 * 7 * G + + .quad 0x0012aafeecbd47af + .quad 0x55a266fb1cd46309 + .quad 0xf203eb680967c72c + .quad 0x39633944ca3c1429 + .quad 0xd6b07a5581cb0e3c + .quad 0x290ff006d9444969 + .quad 0x08680b6a16dcda1f + .quad 0x5568d2b75a06de59 + .quad 0x8d0cb88c1b37cfe1 + .quad 0x05b6a5a3053818f3 + .quad 0xf2e9bc04b787d959 + .quad 0x6beba1249add7f64 + + // 2^148 * 8 * G + + .quad 0x1d06005ca5b1b143 + .quad 0x6d4c6bb87fd1cda2 + .quad 0x6ef5967653fcffe7 + .quad 0x097c29e8c1ce1ea5 + .quad 0x5c3cecb943f5a53b + .quad 0x9cc9a61d06c08df2 + .quad 0xcfba639a85895447 + .quad 0x5a845ae80df09fd5 + .quad 0x4ce97dbe5deb94ca + .quad 0x38d0a4388c709c48 + .quad 0xc43eced4a169d097 + .quad 0x0a1249fff7e587c3 + + // 2^152 * 1 * G + + .quad 0x12f0071b276d01c9 + .quad 0xe7b8bac586c48c70 + .quad 0x5308129b71d6fba9 + .quad 0x5d88fbf95a3db792 + .quad 0x0b408d9e7354b610 + .quad 0x806b32535ba85b6e + .quad 0xdbe63a034a58a207 + .quad 0x173bd9ddc9a1df2c + .quad 0x2b500f1efe5872df + .quad 0x58d6582ed43918c1 + .quad 0xe6ed278ec9673ae0 + .quad 0x06e1cd13b19ea319 + + // 2^152 * 2 * G + + .quad 0x40d0ad516f166f23 + .quad 0x118e32931fab6abe + .quad 0x3fe35e14a04d088e + .quad 0x3080603526e16266 + .quad 0x472baf629e5b0353 + .quad 0x3baa0b90278d0447 + .quad 0x0c785f469643bf27 + .quad 0x7f3a6a1a8d837b13 + .quad 0xf7e644395d3d800b + .quad 0x95a8d555c901edf6 + .quad 0x68cd7830592c6339 + .quad 0x30d0fded2e51307e + + // 2^152 * 3 * G + + .quad 0xe0594d1af21233b3 + .quad 0x1bdbe78ef0cc4d9c + .quad 0x6965187f8f499a77 + .quad 0x0a9214202c099868 + .quad 0x9cb4971e68b84750 + .quad 0xa09572296664bbcf + .quad 0x5c8de72672fa412b + .quad 0x4615084351c589d9 + .quad 0xbc9019c0aeb9a02e + .quad 0x55c7110d16034cae + .quad 0x0e6df501659932ec + .quad 0x3bca0d2895ca5dfe + + // 2^152 * 4 * G + + .quad 0x40f031bc3c5d62a4 + .quad 0x19fc8b3ecff07a60 + .quad 0x98183da2130fb545 + .quad 0x5631deddae8f13cd + .quad 0x9c688eb69ecc01bf + .quad 0xf0bc83ada644896f + .quad 0xca2d955f5f7a9fe2 + .quad 0x4ea8b4038df28241 + .quad 0x2aed460af1cad202 + .quad 0x46305305a48cee83 + .quad 0x9121774549f11a5f + .quad 0x24ce0930542ca463 + + // 2^152 * 5 * G + + .quad 0x1fe890f5fd06c106 + .quad 0xb5c468355d8810f2 + .quad 0x827808fe6e8caf3e + .quad 0x41d4e3c28a06d74b + .quad 0x3fcfa155fdf30b85 + .quad 0xd2f7168e36372ea4 + .quad 0xb2e064de6492f844 + .quad 0x549928a7324f4280 + .quad 0xf26e32a763ee1a2e + .quad 0xae91e4b7d25ffdea + .quad 0xbc3bd33bd17f4d69 + .quad 0x491b66dec0dcff6a + + // 2^152 * 6 * G + + .quad 0x98f5b13dc7ea32a7 + .quad 0xe3d5f8cc7e16db98 + .quad 0xac0abf52cbf8d947 + .quad 0x08f338d0c85ee4ac + .quad 0x75f04a8ed0da64a1 + .quad 0xed222caf67e2284b + .quad 0x8234a3791f7b7ba4 + .quad 0x4cf6b8b0b7018b67 + .quad 0xc383a821991a73bd + .quad 0xab27bc01df320c7a + .quad 0xc13d331b84777063 + .quad 0x530d4a82eb078a99 + + // 2^152 * 7 * G + + .quad 0x004c3630e1f94825 + .quad 0x7e2d78268cab535a + .quad 0xc7482323cc84ff8b + .quad 0x65ea753f101770b9 + .quad 0x6d6973456c9abf9e + .quad 0x257fb2fc4900a880 + .quad 0x2bacf412c8cfb850 + .quad 0x0db3e7e00cbfbd5b + .quad 0x3d66fc3ee2096363 + .quad 0x81d62c7f61b5cb6b + .quad 0x0fbe044213443b1a + .quad 0x02a4ec1921e1a1db + + // 2^152 * 8 * G + + .quad 0x5ce6259a3b24b8a2 + .quad 0xb8577acc45afa0b8 + .quad 0xcccbe6e88ba07037 + .quad 0x3d143c51127809bf + .quad 0xf5c86162f1cf795f + .quad 0x118c861926ee57f2 + .quad 0x172124851c063578 + .quad 0x36d12b5dec067fcf + .quad 0x126d279179154557 + .quad 0xd5e48f5cfc783a0a + .quad 0x36bdb6e8df179bac + .quad 0x2ef517885ba82859 + + // 2^156 * 1 * G + + .quad 0x88bd438cd11e0d4a + .quad 0x30cb610d43ccf308 + .quad 0xe09a0e3791937bcc + .quad 0x4559135b25b1720c + .quad 0x1ea436837c6da1e9 + .quad 0xf9c189af1fb9bdbe + .quad 0x303001fcce5dd155 + .quad 0x28a7c99ebc57be52 + .quad 0xb8fd9399e8d19e9d + .quad 0x908191cb962423ff + .quad 0xb2b948d747c742a3 + .quad 0x37f33226d7fb44c4 + + // 2^156 * 2 * G + + .quad 0x0dae8767b55f6e08 + .quad 0x4a43b3b35b203a02 + .quad 0xe3725a6e80af8c79 + .quad 0x0f7a7fd1705fa7a3 + .quad 0x33912553c821b11d + .quad 0x66ed42c241e301df + .quad 0x066fcc11104222fd + .quad 0x307a3b41c192168f + .quad 0x8eeb5d076eb55ce0 + .quad 0x2fc536bfaa0d925a + .quad 0xbe81830fdcb6c6e8 + .quad 0x556c7045827baf52 + + // 2^156 * 3 * G + + .quad 0x8e2b517302e9d8b7 + .quad 0xe3e52269248714e8 + .quad 0xbd4fbd774ca960b5 + .quad 0x6f4b4199c5ecada9 + .quad 0xb94b90022bf44406 + .quad 0xabd4237eff90b534 + .quad 0x7600a960faf86d3a + .quad 0x2f45abdac2322ee3 + .quad 0x61af4912c8ef8a6a + .quad 0xe58fa4fe43fb6e5e + .quad 0xb5afcc5d6fd427cf + .quad 0x6a5393281e1e11eb + + // 2^156 * 4 * G + + .quad 0xf3da5139a5d1ee89 + .quad 0x8145457cff936988 + .quad 0x3f622fed00e188c4 + .quad 0x0f513815db8b5a3d + .quad 0x0fff04fe149443cf + .quad 0x53cac6d9865cddd7 + .quad 0x31385b03531ed1b7 + .quad 0x5846a27cacd1039d + .quad 0x4ff5cdac1eb08717 + .quad 0x67e8b29590f2e9bc + .quad 0x44093b5e237afa99 + .quad 0x0d414bed8708b8b2 + + // 2^156 * 5 * G + + .quad 0xcfb68265fd0e75f6 + .quad 0xe45b3e28bb90e707 + .quad 0x7242a8de9ff92c7a + .quad 0x685b3201933202dd + .quad 0x81886a92294ac9e8 + .quad 0x23162b45d55547be + .quad 0x94cfbc4403715983 + .quad 0x50eb8fdb134bc401 + .quad 0xc0b73ec6d6b330cd + .quad 0x84e44807132faff1 + .quad 0x732b7352c4a5dee1 + .quad 0x5d7c7cf1aa7cd2d2 + + // 2^156 * 6 * G + + .quad 0xaf3b46bf7a4aafa2 + .quad 0xb78705ec4d40d411 + .quad 0x114f0c6aca7c15e3 + .quad 0x3f364faaa9489d4d + .quad 0x33d1013e9b73a562 + .quad 0x925cef5748ec26e1 + .quad 0xa7fce614dd468058 + .quad 0x78b0fad41e9aa438 + .quad 0xbf56a431ed05b488 + .quad 0xa533e66c9c495c7e + .quad 0xe8652baf87f3651a + .quad 0x0241800059d66c33 + + // 2^156 * 7 * G + + .quad 0xceb077fea37a5be4 + .quad 0xdb642f02e5a5eeb7 + .quad 0xc2e6d0c5471270b8 + .quad 0x4771b65538e4529c + .quad 0x28350c7dcf38ea01 + .quad 0x7c6cdbc0b2917ab6 + .quad 0xace7cfbe857082f7 + .quad 0x4d2845aba2d9a1e0 + .quad 0xbb537fe0447070de + .quad 0xcba744436dd557df + .quad 0xd3b5a3473600dbcb + .quad 0x4aeabbe6f9ffd7f8 + + // 2^156 * 8 * G + + .quad 0x4630119e40d8f78c + .quad 0xa01a9bc53c710e11 + .quad 0x486d2b258910dd79 + .quad 0x1e6c47b3db0324e5 + .quad 0x6a2134bcc4a9c8f2 + .quad 0xfbf8fd1c8ace2e37 + .quad 0x000ae3049911a0ba + .quad 0x046e3a616bc89b9e + .quad 0x14e65442f03906be + .quad 0x4a019d54e362be2a + .quad 0x68ccdfec8dc230c7 + .quad 0x7cfb7e3faf6b861c + + // 2^160 * 1 * G + + .quad 0x4637974e8c58aedc + .quad 0xb9ef22fbabf041a4 + .quad 0xe185d956e980718a + .quad 0x2f1b78fab143a8a6 + .quad 0x96eebffb305b2f51 + .quad 0xd3f938ad889596b8 + .quad 0xf0f52dc746d5dd25 + .quad 0x57968290bb3a0095 + .quad 0xf71ab8430a20e101 + .quad 0xf393658d24f0ec47 + .quad 0xcf7509a86ee2eed1 + .quad 0x7dc43e35dc2aa3e1 + + // 2^160 * 2 * G + + .quad 0x85966665887dd9c3 + .quad 0xc90f9b314bb05355 + .quad 0xc6e08df8ef2079b1 + .quad 0x7ef72016758cc12f + .quad 0x5a782a5c273e9718 + .quad 0x3576c6995e4efd94 + .quad 0x0f2ed8051f237d3e + .quad 0x044fb81d82d50a99 + .quad 0xc1df18c5a907e3d9 + .quad 0x57b3371dce4c6359 + .quad 0xca704534b201bb49 + .quad 0x7f79823f9c30dd2e + + // 2^160 * 3 * G + + .quad 0x8334d239a3b513e8 + .quad 0xc13670d4b91fa8d8 + .quad 0x12b54136f590bd33 + .quad 0x0a4e0373d784d9b4 + .quad 0x6a9c1ff068f587ba + .quad 0x0827894e0050c8de + .quad 0x3cbf99557ded5be7 + .quad 0x64a9b0431c06d6f0 + .quad 0x2eb3d6a15b7d2919 + .quad 0xb0b4f6a0d53a8235 + .quad 0x7156ce4389a45d47 + .quad 0x071a7d0ace18346c + + // 2^160 * 4 * G + + .quad 0xd3072daac887ba0b + .quad 0x01262905bfa562ee + .quad 0xcf543002c0ef768b + .quad 0x2c3bcc7146ea7e9c + .quad 0xcc0c355220e14431 + .quad 0x0d65950709b15141 + .quad 0x9af5621b209d5f36 + .quad 0x7c69bcf7617755d3 + .quad 0x07f0d7eb04e8295f + .quad 0x10db18252f50f37d + .quad 0xe951a9a3171798d7 + .quad 0x6f5a9a7322aca51d + + // 2^160 * 5 * G + + .quad 0x8ba1000c2f41c6c5 + .quad 0xc49f79c10cfefb9b + .quad 0x4efa47703cc51c9f + .quad 0x494e21a2e147afca + .quad 0xe729d4eba3d944be + .quad 0x8d9e09408078af9e + .quad 0x4525567a47869c03 + .quad 0x02ab9680ee8d3b24 + .quad 0xefa48a85dde50d9a + .quad 0x219a224e0fb9a249 + .quad 0xfa091f1dd91ef6d9 + .quad 0x6b5d76cbea46bb34 + + // 2^160 * 6 * G + + .quad 0x8857556cec0cd994 + .quad 0x6472dc6f5cd01dba + .quad 0xaf0169148f42b477 + .quad 0x0ae333f685277354 + .quad 0xe0f941171e782522 + .quad 0xf1e6ae74036936d3 + .quad 0x408b3ea2d0fcc746 + .quad 0x16fb869c03dd313e + .quad 0x288e199733b60962 + .quad 0x24fc72b4d8abe133 + .quad 0x4811f7ed0991d03e + .quad 0x3f81e38b8f70d075 + + // 2^160 * 7 * G + + .quad 0x7f910fcc7ed9affe + .quad 0x545cb8a12465874b + .quad 0xa8397ed24b0c4704 + .quad 0x50510fc104f50993 + .quad 0x0adb7f355f17c824 + .quad 0x74b923c3d74299a4 + .quad 0xd57c3e8bcbf8eaf7 + .quad 0x0ad3e2d34cdedc3d + .quad 0x6f0c0fc5336e249d + .quad 0x745ede19c331cfd9 + .quad 0xf2d6fd0009eefe1c + .quad 0x127c158bf0fa1ebe + + // 2^160 * 8 * G + + .quad 0xf6197c422e9879a2 + .quad 0xa44addd452ca3647 + .quad 0x9b413fc14b4eaccb + .quad 0x354ef87d07ef4f68 + .quad 0xdea28fc4ae51b974 + .quad 0x1d9973d3744dfe96 + .quad 0x6240680b873848a8 + .quad 0x4ed82479d167df95 + .quad 0xfee3b52260c5d975 + .quad 0x50352efceb41b0b8 + .quad 0x8808ac30a9f6653c + .quad 0x302d92d20539236d + + // 2^164 * 1 * G + + .quad 0x4c59023fcb3efb7c + .quad 0x6c2fcb99c63c2a94 + .quad 0xba4190e2c3c7e084 + .quad 0x0e545daea51874d9 + .quad 0x957b8b8b0df53c30 + .quad 0x2a1c770a8e60f098 + .quad 0xbbc7a670345796de + .quad 0x22a48f9a90c99bc9 + .quad 0x6b7dc0dc8d3fac58 + .quad 0x5497cd6ce6e42bfd + .quad 0x542f7d1bf400d305 + .quad 0x4159f47f048d9136 + + // 2^164 * 2 * G + + .quad 0x20ad660839e31e32 + .quad 0xf81e1bd58405be50 + .quad 0xf8064056f4dabc69 + .quad 0x14d23dd4ce71b975 + .quad 0x748515a8bbd24839 + .quad 0x77128347afb02b55 + .quad 0x50ba2ac649a2a17f + .quad 0x060525513ad730f1 + .quad 0xf2398e098aa27f82 + .quad 0x6d7982bb89a1b024 + .quad 0xfa694084214dd24c + .quad 0x71ab966fa32301c3 + + // 2^164 * 3 * G + + .quad 0x2dcbd8e34ded02fc + .quad 0x1151f3ec596f22aa + .quad 0xbca255434e0328da + .quad 0x35768fbe92411b22 + .quad 0xb1088a0702809955 + .quad 0x43b273ea0b43c391 + .quad 0xca9b67aefe0686ed + .quad 0x605eecbf8335f4ed + .quad 0x83200a656c340431 + .quad 0x9fcd71678ee59c2f + .quad 0x75d4613f71300f8a + .quad 0x7a912faf60f542f9 + + // 2^164 * 4 * G + + .quad 0xb204585e5edc1a43 + .quad 0x9f0e16ee5897c73c + .quad 0x5b82c0ae4e70483c + .quad 0x624a170e2bddf9be + .quad 0x253f4f8dfa2d5597 + .quad 0x25e49c405477130c + .quad 0x00c052e5996b1102 + .quad 0x33cb966e33bb6c4a + .quad 0x597028047f116909 + .quad 0x828ac41c1e564467 + .quad 0x70417dbde6217387 + .quad 0x721627aefbac4384 + + // 2^164 * 5 * G + + .quad 0x97d03bc38736add5 + .quad 0x2f1422afc532b130 + .quad 0x3aa68a057101bbc4 + .quad 0x4c946cf7e74f9fa7 + .quad 0xfd3097bc410b2f22 + .quad 0xf1a05da7b5cfa844 + .quad 0x61289a1def57ca74 + .quad 0x245ea199bb821902 + .quad 0xaedca66978d477f8 + .quad 0x1898ba3c29117fe1 + .quad 0xcf73f983720cbd58 + .quad 0x67da12e6b8b56351 + + // 2^164 * 6 * G + + .quad 0x7067e187b4bd6e07 + .quad 0x6e8f0203c7d1fe74 + .quad 0x93c6aa2f38c85a30 + .quad 0x76297d1f3d75a78a + .quad 0x2b7ef3d38ec8308c + .quad 0x828fd7ec71eb94ab + .quad 0x807c3b36c5062abd + .quad 0x0cb64cb831a94141 + .quad 0x3030fc33534c6378 + .quad 0xb9635c5ce541e861 + .quad 0x15d9a9bed9b2c728 + .quad 0x49233ea3f3775dcb + + // 2^164 * 7 * G + + .quad 0x629398fa8dbffc3a + .quad 0xe12fe52dd54db455 + .quad 0xf3be11dfdaf25295 + .quad 0x628b140dce5e7b51 + .quad 0x7b3985fe1c9f249b + .quad 0x4fd6b2d5a1233293 + .quad 0xceb345941adf4d62 + .quad 0x6987ff6f542de50c + .quad 0x47e241428f83753c + .quad 0x6317bebc866af997 + .quad 0xdabb5b433d1a9829 + .quad 0x074d8d245287fb2d + + // 2^164 * 8 * G + + .quad 0x8337d9cd440bfc31 + .quad 0x729d2ca1af318fd7 + .quad 0xa040a4a4772c2070 + .quad 0x46002ef03a7349be + .quad 0x481875c6c0e31488 + .quad 0x219429b2e22034b4 + .quad 0x7223c98a31283b65 + .quad 0x3420d60b342277f9 + .quad 0xfaa23adeaffe65f7 + .quad 0x78261ed45be0764c + .quad 0x441c0a1e2f164403 + .quad 0x5aea8e567a87d395 + + // 2^168 * 1 * G + + .quad 0x7813c1a2bca4283d + .quad 0xed62f091a1863dd9 + .quad 0xaec7bcb8c268fa86 + .quad 0x10e5d3b76f1cae4c + .quad 0x2dbc6fb6e4e0f177 + .quad 0x04e1bf29a4bd6a93 + .quad 0x5e1966d4787af6e8 + .quad 0x0edc5f5eb426d060 + .quad 0x5453bfd653da8e67 + .quad 0xe9dc1eec24a9f641 + .quad 0xbf87263b03578a23 + .quad 0x45b46c51361cba72 + + // 2^168 * 2 * G + + .quad 0xa9402abf314f7fa1 + .quad 0xe257f1dc8e8cf450 + .quad 0x1dbbd54b23a8be84 + .quad 0x2177bfa36dcb713b + .quad 0xce9d4ddd8a7fe3e4 + .quad 0xab13645676620e30 + .quad 0x4b594f7bb30e9958 + .quad 0x5c1c0aef321229df + .quad 0x37081bbcfa79db8f + .quad 0x6048811ec25f59b3 + .quad 0x087a76659c832487 + .quad 0x4ae619387d8ab5bb + + // 2^168 * 3 * G + + .quad 0x8ddbf6aa5344a32e + .quad 0x7d88eab4b41b4078 + .quad 0x5eb0eb974a130d60 + .quad 0x1a00d91b17bf3e03 + .quad 0x61117e44985bfb83 + .quad 0xfce0462a71963136 + .quad 0x83ac3448d425904b + .quad 0x75685abe5ba43d64 + .quad 0x6e960933eb61f2b2 + .quad 0x543d0fa8c9ff4952 + .quad 0xdf7275107af66569 + .quad 0x135529b623b0e6aa + + // 2^168 * 4 * G + + .quad 0x18f0dbd7add1d518 + .quad 0x979f7888cfc11f11 + .quad 0x8732e1f07114759b + .quad 0x79b5b81a65ca3a01 + .quad 0xf5c716bce22e83fe + .quad 0xb42beb19e80985c1 + .quad 0xec9da63714254aae + .quad 0x5972ea051590a613 + .quad 0x0fd4ac20dc8f7811 + .quad 0x9a9ad294ac4d4fa8 + .quad 0xc01b2d64b3360434 + .quad 0x4f7e9c95905f3bdb + + // 2^168 * 5 * G + + .quad 0x62674bbc5781302e + .quad 0xd8520f3989addc0f + .quad 0x8c2999ae53fbd9c6 + .quad 0x31993ad92e638e4c + .quad 0x71c8443d355299fe + .quad 0x8bcd3b1cdbebead7 + .quad 0x8092499ef1a49466 + .quad 0x1942eec4a144adc8 + .quad 0x7dac5319ae234992 + .quad 0x2c1b3d910cea3e92 + .quad 0x553ce494253c1122 + .quad 0x2a0a65314ef9ca75 + + // 2^168 * 6 * G + + .quad 0x2db7937ff7f927c2 + .quad 0xdb741f0617d0a635 + .quad 0x5982f3a21155af76 + .quad 0x4cf6e218647c2ded + .quad 0xcf361acd3c1c793a + .quad 0x2f9ebcac5a35bc3b + .quad 0x60e860e9a8cda6ab + .quad 0x055dc39b6dea1a13 + .quad 0xb119227cc28d5bb6 + .quad 0x07e24ebc774dffab + .quad 0xa83c78cee4a32c89 + .quad 0x121a307710aa24b6 + + // 2^168 * 7 * G + + .quad 0xe4db5d5e9f034a97 + .quad 0xe153fc093034bc2d + .quad 0x460546919551d3b1 + .quad 0x333fc76c7a40e52d + .quad 0xd659713ec77483c9 + .quad 0x88bfe077b82b96af + .quad 0x289e28231097bcd3 + .quad 0x527bb94a6ced3a9b + .quad 0x563d992a995b482e + .quad 0x3405d07c6e383801 + .quad 0x485035de2f64d8e5 + .quad 0x6b89069b20a7a9f7 + + // 2^168 * 8 * G + + .quad 0x812aa0416270220d + .quad 0x995a89faf9245b4e + .quad 0xffadc4ce5072ef05 + .quad 0x23bc2103aa73eb73 + .quad 0x4082fa8cb5c7db77 + .quad 0x068686f8c734c155 + .quad 0x29e6c8d9f6e7a57e + .quad 0x0473d308a7639bcf + .quad 0xcaee792603589e05 + .quad 0x2b4b421246dcc492 + .quad 0x02a1ef74e601a94f + .quad 0x102f73bfde04341a + + // 2^172 * 1 * G + + .quad 0xb5a2d50c7ec20d3e + .quad 0xc64bdd6ea0c97263 + .quad 0x56e89052c1ff734d + .quad 0x4929c6f72b2ffaba + .quad 0x358ecba293a36247 + .quad 0xaf8f9862b268fd65 + .quad 0x412f7e9968a01c89 + .quad 0x5786f312cd754524 + .quad 0x337788ffca14032c + .quad 0xf3921028447f1ee3 + .quad 0x8b14071f231bccad + .quad 0x4c817b4bf2344783 + + // 2^172 * 2 * G + + .quad 0x0ff853852871b96e + .quad 0xe13e9fab60c3f1bb + .quad 0xeefd595325344402 + .quad 0x0a37c37075b7744b + .quad 0x413ba057a40b4484 + .quad 0xba4c2e1a4f5f6a43 + .quad 0x614ba0a5aee1d61c + .quad 0x78a1531a8b05dc53 + .quad 0x6cbdf1703ad0562b + .quad 0x8ecf4830c92521a3 + .quad 0xdaebd303fd8424e7 + .quad 0x72ad82a42e5ec56f + + // 2^172 * 3 * G + + .quad 0x3f9e8e35bafb65f6 + .quad 0x39d69ec8f27293a1 + .quad 0x6cb8cd958cf6a3d0 + .quad 0x1734778173adae6d + .quad 0xc368939167024bc3 + .quad 0x8e69d16d49502fda + .quad 0xfcf2ec3ce45f4b29 + .quad 0x065f669ea3b4cbc4 + .quad 0x8a00aec75532db4d + .quad 0xb869a4e443e31bb1 + .quad 0x4a0f8552d3a7f515 + .quad 0x19adeb7c303d7c08 + + // 2^172 * 4 * G + + .quad 0xc720cb6153ead9a3 + .quad 0x55b2c97f512b636e + .quad 0xb1e35b5fd40290b1 + .quad 0x2fd9ccf13b530ee2 + .quad 0x9d05ba7d43c31794 + .quad 0x2470c8ff93322526 + .quad 0x8323dec816197438 + .quad 0x2852709881569b53 + .quad 0x07bd475b47f796b8 + .quad 0xd2c7b013542c8f54 + .quad 0x2dbd23f43b24f87e + .quad 0x6551afd77b0901d6 + + // 2^172 * 5 * G + + .quad 0x4546baaf54aac27f + .quad 0xf6f66fecb2a45a28 + .quad 0x582d1b5b562bcfe8 + .quad 0x44b123f3920f785f + .quad 0x68a24ce3a1d5c9ac + .quad 0xbb77a33d10ff6461 + .quad 0x0f86ce4425d3166e + .quad 0x56507c0950b9623b + .quad 0x1206f0b7d1713e63 + .quad 0x353fe3d915bafc74 + .quad 0x194ceb970ad9d94d + .quad 0x62fadd7cf9d03ad3 + + // 2^172 * 6 * G + + .quad 0xc6b5967b5598a074 + .quad 0x5efe91ce8e493e25 + .quad 0xd4b72c4549280888 + .quad 0x20ef1149a26740c2 + .quad 0x3cd7bc61e7ce4594 + .quad 0xcd6b35a9b7dd267e + .quad 0xa080abc84366ef27 + .quad 0x6ec7c46f59c79711 + .quad 0x2f07ad636f09a8a2 + .quad 0x8697e6ce24205e7d + .quad 0xc0aefc05ee35a139 + .quad 0x15e80958b5f9d897 + + // 2^172 * 7 * G + + .quad 0x25a5ef7d0c3e235b + .quad 0x6c39c17fbe134ee7 + .quad 0xc774e1342dc5c327 + .quad 0x021354b892021f39 + .quad 0x4dd1ed355bb061c4 + .quad 0x42dc0cef941c0700 + .quad 0x61305dc1fd86340e + .quad 0x56b2cc930e55a443 + .quad 0x1df79da6a6bfc5a2 + .quad 0x02f3a2749fde4369 + .quad 0xb323d9f2cda390a7 + .quad 0x7be0847b8774d363 + + // 2^172 * 8 * G + + .quad 0x8c99cc5a8b3f55c3 + .quad 0x0611d7253fded2a0 + .quad 0xed2995ff36b70a36 + .quad 0x1f699a54d78a2619 + .quad 0x1466f5af5307fa11 + .quad 0x817fcc7ded6c0af2 + .quad 0x0a6de44ec3a4a3fb + .quad 0x74071475bc927d0b + .quad 0xe77292f373e7ea8a + .quad 0x296537d2cb045a31 + .quad 0x1bd0653ed3274fde + .quad 0x2f9a2c4476bd2966 + + // 2^176 * 1 * G + + .quad 0xeb18b9ab7f5745c6 + .quad 0x023a8aee5787c690 + .quad 0xb72712da2df7afa9 + .quad 0x36597d25ea5c013d + .quad 0xa2b4dae0b5511c9a + .quad 0x7ac860292bffff06 + .quad 0x981f375df5504234 + .quad 0x3f6bd725da4ea12d + .quad 0x734d8d7b106058ac + .quad 0xd940579e6fc6905f + .quad 0x6466f8f99202932d + .quad 0x7b7ecc19da60d6d0 + + // 2^176 * 2 * G + + .quad 0x78c2373c695c690d + .quad 0xdd252e660642906e + .quad 0x951d44444ae12bd2 + .quad 0x4235ad7601743956 + .quad 0x6dae4a51a77cfa9b + .quad 0x82263654e7a38650 + .quad 0x09bbffcd8f2d82db + .quad 0x03bedc661bf5caba + .quad 0x6258cb0d078975f5 + .quad 0x492942549189f298 + .quad 0xa0cab423e2e36ee4 + .quad 0x0e7ce2b0cdf066a1 + + // 2^176 * 3 * G + + .quad 0xc494643ac48c85a3 + .quad 0xfd361df43c6139ad + .quad 0x09db17dd3ae94d48 + .quad 0x666e0a5d8fb4674a + .quad 0xfea6fedfd94b70f9 + .quad 0xf130c051c1fcba2d + .quad 0x4882d47e7f2fab89 + .quad 0x615256138aeceeb5 + .quad 0x2abbf64e4870cb0d + .quad 0xcd65bcf0aa458b6b + .quad 0x9abe4eba75e8985d + .quad 0x7f0bc810d514dee4 + + // 2^176 * 4 * G + + .quad 0xb9006ba426f4136f + .quad 0x8d67369e57e03035 + .quad 0xcbc8dfd94f463c28 + .quad 0x0d1f8dbcf8eedbf5 + .quad 0x83ac9dad737213a0 + .quad 0x9ff6f8ba2ef72e98 + .quad 0x311e2edd43ec6957 + .quad 0x1d3a907ddec5ab75 + .quad 0xba1693313ed081dc + .quad 0x29329fad851b3480 + .quad 0x0128013c030321cb + .quad 0x00011b44a31bfde3 + + // 2^176 * 5 * G + + .quad 0x3fdfa06c3fc66c0c + .quad 0x5d40e38e4dd60dd2 + .quad 0x7ae38b38268e4d71 + .quad 0x3ac48d916e8357e1 + .quad 0x16561f696a0aa75c + .quad 0xc1bf725c5852bd6a + .quad 0x11a8dd7f9a7966ad + .quad 0x63d988a2d2851026 + .quad 0x00120753afbd232e + .quad 0xe92bceb8fdd8f683 + .quad 0xf81669b384e72b91 + .quad 0x33fad52b2368a066 + + // 2^176 * 6 * G + + .quad 0x540649c6c5e41e16 + .quad 0x0af86430333f7735 + .quad 0xb2acfcd2f305e746 + .quad 0x16c0f429a256dca7 + .quad 0x8d2cc8d0c422cfe8 + .quad 0x072b4f7b05a13acb + .quad 0xa3feb6e6ecf6a56f + .quad 0x3cc355ccb90a71e2 + .quad 0xe9b69443903e9131 + .quad 0xb8a494cb7a5637ce + .quad 0xc87cd1a4baba9244 + .quad 0x631eaf426bae7568 + + // 2^176 * 7 * G + + .quad 0xb3e90410da66fe9f + .quad 0x85dd4b526c16e5a6 + .quad 0xbc3d97611ef9bf83 + .quad 0x5599648b1ea919b5 + .quad 0x47d975b9a3700de8 + .quad 0x7280c5fbe2f80552 + .quad 0x53658f2732e45de1 + .quad 0x431f2c7f665f80b5 + .quad 0xd6026344858f7b19 + .quad 0x14ab352fa1ea514a + .quad 0x8900441a2090a9d7 + .quad 0x7b04715f91253b26 + + // 2^176 * 8 * G + + .quad 0x83edbd28acf6ae43 + .quad 0x86357c8b7d5c7ab4 + .quad 0xc0404769b7eb2c44 + .quad 0x59b37bf5c2f6583f + .quad 0xb376c280c4e6bac6 + .quad 0x970ed3dd6d1d9b0b + .quad 0xb09a9558450bf944 + .quad 0x48d0acfa57cde223 + .quad 0xb60f26e47dabe671 + .quad 0xf1d1a197622f3a37 + .quad 0x4208ce7ee9960394 + .quad 0x16234191336d3bdb + + // 2^180 * 1 * G + + .quad 0xf19aeac733a63aef + .quad 0x2c7fba5d4442454e + .quad 0x5da87aa04795e441 + .quad 0x413051e1a4e0b0f5 + .quad 0x852dd1fd3d578bbe + .quad 0x2b65ce72c3286108 + .quad 0x658c07f4eace2273 + .quad 0x0933f804ec38ab40 + .quad 0xa7ab69798d496476 + .quad 0x8121aadefcb5abc8 + .quad 0xa5dc12ef7b539472 + .quad 0x07fd47065e45351a + + // 2^180 * 2 * G + + .quad 0xc8583c3d258d2bcd + .quad 0x17029a4daf60b73f + .quad 0xfa0fc9d6416a3781 + .quad 0x1c1e5fba38b3fb23 + .quad 0x304211559ae8e7c3 + .quad 0xf281b229944882a5 + .quad 0x8a13ac2e378250e4 + .quad 0x014afa0954ba48f4 + .quad 0xcb3197001bb3666c + .quad 0x330060524bffecb9 + .quad 0x293711991a88233c + .quad 0x291884363d4ed364 + + // 2^180 * 3 * G + + .quad 0x033c6805dc4babfa + .quad 0x2c15bf5e5596ecc1 + .quad 0x1bc70624b59b1d3b + .quad 0x3ede9850a19f0ec5 + .quad 0xfb9d37c3bc1ab6eb + .quad 0x02be14534d57a240 + .quad 0xf4d73415f8a5e1f6 + .quad 0x5964f4300ccc8188 + .quad 0xe44a23152d096800 + .quad 0x5c08c55970866996 + .quad 0xdf2db60a46affb6e + .quad 0x579155c1f856fd89 + + // 2^180 * 4 * G + + .quad 0x96324edd12e0c9ef + .quad 0x468b878df2420297 + .quad 0x199a3776a4f573be + .quad 0x1e7fbcf18e91e92a + .quad 0xb5f16b630817e7a6 + .quad 0x808c69233c351026 + .quad 0x324a983b54cef201 + .quad 0x53c092084a485345 + .quad 0xd2d41481f1cbafbf + .quad 0x231d2db6716174e5 + .quad 0x0b7d7656e2a55c98 + .quad 0x3e955cd82aa495f6 + + // 2^180 * 5 * G + + .quad 0xe48f535e3ed15433 + .quad 0xd075692a0d7270a3 + .quad 0x40fbd21daade6387 + .quad 0x14264887cf4495f5 + .quad 0xab39f3ef61bb3a3f + .quad 0x8eb400652eb9193e + .quad 0xb5de6ecc38c11f74 + .quad 0x654d7e9626f3c49f + .quad 0xe564cfdd5c7d2ceb + .quad 0x82eeafded737ccb9 + .quad 0x6107db62d1f9b0ab + .quad 0x0b6baac3b4358dbb + + // 2^180 * 6 * G + + .quad 0x7ae62bcb8622fe98 + .quad 0x47762256ceb891af + .quad 0x1a5a92bcf2e406b4 + .quad 0x7d29401784e41501 + .quad 0x204abad63700a93b + .quad 0xbe0023d3da779373 + .quad 0xd85f0346633ab709 + .quad 0x00496dc490820412 + .quad 0x1c74b88dc27e6360 + .quad 0x074854268d14850c + .quad 0xa145fb7b3e0dcb30 + .quad 0x10843f1b43803b23 + + // 2^180 * 7 * G + + .quad 0xc5f90455376276dd + .quad 0xce59158dd7645cd9 + .quad 0x92f65d511d366b39 + .quad 0x11574b6e526996c4 + .quad 0xd56f672de324689b + .quad 0xd1da8aedb394a981 + .quad 0xdd7b58fe9168cfed + .quad 0x7ce246cd4d56c1e8 + .quad 0xb8f4308e7f80be53 + .quad 0x5f3cb8cb34a9d397 + .quad 0x18a961bd33cc2b2c + .quad 0x710045fb3a9af671 + + // 2^180 * 8 * G + + .quad 0x73f93d36101b95eb + .quad 0xfaef33794f6f4486 + .quad 0x5651735f8f15e562 + .quad 0x7fa3f19058b40da1 + .quad 0xa03fc862059d699e + .quad 0x2370cfa19a619e69 + .quad 0xc4fe3b122f823deb + .quad 0x1d1b056fa7f0844e + .quad 0x1bc64631e56bf61f + .quad 0xd379ab106e5382a3 + .quad 0x4d58c57e0540168d + .quad 0x566256628442d8e4 + + // 2^184 * 1 * G + + .quad 0xb9e499def6267ff6 + .quad 0x7772ca7b742c0843 + .quad 0x23a0153fe9a4f2b1 + .quad 0x2cdfdfecd5d05006 + .quad 0xdd499cd61ff38640 + .quad 0x29cd9bc3063625a0 + .quad 0x51e2d8023dd73dc3 + .quad 0x4a25707a203b9231 + .quad 0x2ab7668a53f6ed6a + .quad 0x304242581dd170a1 + .quad 0x4000144c3ae20161 + .quad 0x5721896d248e49fc + + // 2^184 * 2 * G + + .quad 0x0b6e5517fd181bae + .quad 0x9022629f2bb963b4 + .quad 0x5509bce932064625 + .quad 0x578edd74f63c13da + .quad 0x285d5091a1d0da4e + .quad 0x4baa6fa7b5fe3e08 + .quad 0x63e5177ce19393b3 + .quad 0x03c935afc4b030fd + .quad 0x997276c6492b0c3d + .quad 0x47ccc2c4dfe205fc + .quad 0xdcd29b84dd623a3c + .quad 0x3ec2ab590288c7a2 + + // 2^184 * 3 * G + + .quad 0xa1a0d27be4d87bb9 + .quad 0xa98b4deb61391aed + .quad 0x99a0ddd073cb9b83 + .quad 0x2dd5c25a200fcace + .quad 0xa7213a09ae32d1cb + .quad 0x0f2b87df40f5c2d5 + .quad 0x0baea4c6e81eab29 + .quad 0x0e1bf66c6adbac5e + .quad 0xe2abd5e9792c887e + .quad 0x1a020018cb926d5d + .quad 0xbfba69cdbaae5f1e + .quad 0x730548b35ae88f5f + + // 2^184 * 4 * G + + .quad 0xc43551a3cba8b8ee + .quad 0x65a26f1db2115f16 + .quad 0x760f4f52ab8c3850 + .quad 0x3043443b411db8ca + .quad 0x805b094ba1d6e334 + .quad 0xbf3ef17709353f19 + .quad 0x423f06cb0622702b + .quad 0x585a2277d87845dd + .quad 0xa18a5f8233d48962 + .quad 0x6698c4b5ec78257f + .quad 0xa78e6fa5373e41ff + .quad 0x7656278950ef981f + + // 2^184 * 5 * G + + .quad 0x38c3cf59d51fc8c0 + .quad 0x9bedd2fd0506b6f2 + .quad 0x26bf109fab570e8f + .quad 0x3f4160a8c1b846a6 + .quad 0xe17073a3ea86cf9d + .quad 0x3a8cfbb707155fdc + .quad 0x4853e7fc31838a8e + .quad 0x28bbf484b613f616 + .quad 0xf2612f5c6f136c7c + .quad 0xafead107f6dd11be + .quad 0x527e9ad213de6f33 + .quad 0x1e79cb358188f75d + + // 2^184 * 6 * G + + .quad 0x013436c3eef7e3f1 + .quad 0x828b6a7ffe9e10f8 + .quad 0x7ff908e5bcf9defc + .quad 0x65d7951b3a3b3831 + .quad 0x77e953d8f5e08181 + .quad 0x84a50c44299dded9 + .quad 0xdc6c2d0c864525e5 + .quad 0x478ab52d39d1f2f4 + .quad 0x66a6a4d39252d159 + .quad 0xe5dde1bc871ac807 + .quad 0xb82c6b40a6c1c96f + .quad 0x16d87a411a212214 + + // 2^184 * 7 * G + + .quad 0xb3bd7e5a42066215 + .quad 0x879be3cd0c5a24c1 + .quad 0x57c05db1d6f994b7 + .quad 0x28f87c8165f38ca6 + .quad 0xfba4d5e2d54e0583 + .quad 0xe21fafd72ebd99fa + .quad 0x497ac2736ee9778f + .quad 0x1f990b577a5a6dde + .quad 0xa3344ead1be8f7d6 + .quad 0x7d1e50ebacea798f + .quad 0x77c6569e520de052 + .quad 0x45882fe1534d6d3e + + // 2^184 * 8 * G + + .quad 0x6669345d757983d6 + .quad 0x62b6ed1117aa11a6 + .quad 0x7ddd1857985e128f + .quad 0x688fe5b8f626f6dd + .quad 0xd8ac9929943c6fe4 + .quad 0xb5f9f161a38392a2 + .quad 0x2699db13bec89af3 + .quad 0x7dcf843ce405f074 + .quad 0x6c90d6484a4732c0 + .quad 0xd52143fdca563299 + .quad 0xb3be28c3915dc6e1 + .quad 0x6739687e7327191b + + // 2^188 * 1 * G + + .quad 0x9f65c5ea200814cf + .quad 0x840536e169a31740 + .quad 0x8b0ed13925c8b4ad + .quad 0x0080dbafe936361d + .quad 0x8ce5aad0c9cb971f + .quad 0x1156aaa99fd54a29 + .quad 0x41f7247015af9b78 + .quad 0x1fe8cca8420f49aa + .quad 0x72a1848f3c0cc82a + .quad 0x38c560c2877c9e54 + .quad 0x5004e228ce554140 + .quad 0x042418a103429d71 + + // 2^188 * 2 * G + + .quad 0x899dea51abf3ff5f + .quad 0x9b93a8672fc2d8ba + .quad 0x2c38cb97be6ebd5c + .quad 0x114d578497263b5d + .quad 0x58e84c6f20816247 + .quad 0x8db2b2b6e36fd793 + .quad 0x977182561d484d85 + .quad 0x0822024f8632abd7 + .quad 0xb301bb7c6b1beca3 + .quad 0x55393f6dc6eb1375 + .quad 0x910d281097b6e4eb + .quad 0x1ad4548d9d479ea3 + + // 2^188 * 3 * G + + .quad 0xcd5a7da0389a48fd + .quad 0xb38fa4aa9a78371e + .quad 0xc6d9761b2cdb8e6c + .quad 0x35cf51dbc97e1443 + .quad 0xa06fe66d0fe9fed3 + .quad 0xa8733a401c587909 + .quad 0x30d14d800df98953 + .quad 0x41ce5876c7b30258 + .quad 0x59ac3bc5d670c022 + .quad 0xeae67c109b119406 + .quad 0x9798bdf0b3782fda + .quad 0x651e3201fd074092 + + // 2^188 * 4 * G + + .quad 0xd63d8483ef30c5cf + .quad 0x4cd4b4962361cc0c + .quad 0xee90e500a48426ac + .quad 0x0af51d7d18c14eeb + .quad 0xa57ba4a01efcae9e + .quad 0x769f4beedc308a94 + .quad 0xd1f10eeb3603cb2e + .quad 0x4099ce5e7e441278 + .quad 0x1ac98e4f8a5121e9 + .quad 0x7dae9544dbfa2fe0 + .quad 0x8320aa0dd6430df9 + .quad 0x667282652c4a2fb5 + + // 2^188 * 5 * G + + .quad 0x874621f4d86bc9ab + .quad 0xb54c7bbe56fe6fea + .quad 0x077a24257fadc22c + .quad 0x1ab53be419b90d39 + .quad 0xada8b6e02946db23 + .quad 0x1c0ce51a7b253ab7 + .quad 0x8448c85a66dd485b + .quad 0x7f1fc025d0675adf + .quad 0xd8ee1b18319ea6aa + .quad 0x004d88083a21f0da + .quad 0x3bd6aa1d883a4f4b + .quad 0x4db9a3a6dfd9fd14 + + // 2^188 * 6 * G + + .quad 0x8ce7b23bb99c0755 + .quad 0x35c5d6edc4f50f7a + .quad 0x7e1e2ed2ed9b50c3 + .quad 0x36305f16e8934da1 + .quad 0xd95b00bbcbb77c68 + .quad 0xddbc846a91f17849 + .quad 0x7cf700aebe28d9b3 + .quad 0x5ce1285c85d31f3e + .quad 0x31b6972d98b0bde8 + .quad 0x7d920706aca6de5b + .quad 0xe67310f8908a659f + .quad 0x50fac2a6efdf0235 + + // 2^188 * 7 * G + + .quad 0xf3d3a9f35b880f5a + .quad 0xedec050cdb03e7c2 + .quad 0xa896981ff9f0b1a2 + .quad 0x49a4ae2bac5e34a4 + .quad 0x295b1c86f6f449bc + .quad 0x51b2e84a1f0ab4dd + .quad 0xc001cb30aa8e551d + .quad 0x6a28d35944f43662 + .quad 0x28bb12ee04a740e0 + .quad 0x14313bbd9bce8174 + .quad 0x72f5b5e4e8c10c40 + .quad 0x7cbfb19936adcd5b + + // 2^188 * 8 * G + + .quad 0xa311ddc26b89792d + .quad 0x1b30b4c6da512664 + .quad 0x0ca77b4ccf150859 + .quad 0x1de443df1b009408 + .quad 0x8e793a7acc36e6e0 + .quad 0xf9fab7a37d586eed + .quad 0x3a4f9692bae1f4e4 + .quad 0x1c14b03eff5f447e + .quad 0x19647bd114a85291 + .quad 0x57b76cb21034d3af + .quad 0x6329db440f9d6dfa + .quad 0x5ef43e586a571493 + + // 2^192 * 1 * G + + .quad 0xef782014385675a6 + .quad 0xa2649f30aafda9e8 + .quad 0x4cd1eb505cdfa8cb + .quad 0x46115aba1d4dc0b3 + .quad 0xa66dcc9dc80c1ac0 + .quad 0x97a05cf41b38a436 + .quad 0xa7ebf3be95dbd7c6 + .quad 0x7da0b8f68d7e7dab + .quad 0xd40f1953c3b5da76 + .quad 0x1dac6f7321119e9b + .quad 0x03cc6021feb25960 + .quad 0x5a5f887e83674b4b + + // 2^192 * 2 * G + + .quad 0x8f6301cf70a13d11 + .quad 0xcfceb815350dd0c4 + .quad 0xf70297d4a4bca47e + .quad 0x3669b656e44d1434 + .quad 0x9e9628d3a0a643b9 + .quad 0xb5c3cb00e6c32064 + .quad 0x9b5302897c2dec32 + .quad 0x43e37ae2d5d1c70c + .quad 0x387e3f06eda6e133 + .quad 0x67301d5199a13ac0 + .quad 0xbd5ad8f836263811 + .quad 0x6a21e6cd4fd5e9be + + // 2^192 * 3 * G + + .quad 0xf1c6170a3046e65f + .quad 0x58712a2a00d23524 + .quad 0x69dbbd3c8c82b755 + .quad 0x586bf9f1a195ff57 + .quad 0xef4129126699b2e3 + .quad 0x71d30847708d1301 + .quad 0x325432d01182b0bd + .quad 0x45371b07001e8b36 + .quad 0xa6db088d5ef8790b + .quad 0x5278f0dc610937e5 + .quad 0xac0349d261a16eb8 + .quad 0x0eafb03790e52179 + + // 2^192 * 4 * G + + .quad 0x960555c13748042f + .quad 0x219a41e6820baa11 + .quad 0x1c81f73873486d0c + .quad 0x309acc675a02c661 + .quad 0x5140805e0f75ae1d + .quad 0xec02fbe32662cc30 + .quad 0x2cebdf1eea92396d + .quad 0x44ae3344c5435bb3 + .quad 0x9cf289b9bba543ee + .quad 0xf3760e9d5ac97142 + .quad 0x1d82e5c64f9360aa + .quad 0x62d5221b7f94678f + + // 2^192 * 5 * G + + .quad 0x524c299c18d0936d + .quad 0xc86bb56c8a0c1a0c + .quad 0xa375052edb4a8631 + .quad 0x5c0efde4bc754562 + .quad 0x7585d4263af77a3c + .quad 0xdfae7b11fee9144d + .quad 0xa506708059f7193d + .quad 0x14f29a5383922037 + .quad 0xdf717edc25b2d7f5 + .quad 0x21f970db99b53040 + .quad 0xda9234b7c3ed4c62 + .quad 0x5e72365c7bee093e + + // 2^192 * 6 * G + + .quad 0x575bfc074571217f + .quad 0x3779675d0694d95b + .quad 0x9a0a37bbf4191e33 + .quad 0x77f1104c47b4eabc + .quad 0x7d9339062f08b33e + .quad 0x5b9659e5df9f32be + .quad 0xacff3dad1f9ebdfd + .quad 0x70b20555cb7349b7 + .quad 0xbe5113c555112c4c + .quad 0x6688423a9a881fcd + .quad 0x446677855e503b47 + .quad 0x0e34398f4a06404a + + // 2^192 * 7 * G + + .quad 0xb67d22d93ecebde8 + .quad 0x09b3e84127822f07 + .quad 0x743fa61fb05b6d8d + .quad 0x5e5405368a362372 + .quad 0x18930b093e4b1928 + .quad 0x7de3e10e73f3f640 + .quad 0xf43217da73395d6f + .quad 0x6f8aded6ca379c3e + .quad 0xe340123dfdb7b29a + .quad 0x487b97e1a21ab291 + .quad 0xf9967d02fde6949e + .quad 0x780de72ec8d3de97 + + // 2^192 * 8 * G + + .quad 0x0ae28545089ae7bc + .quad 0x388ddecf1c7f4d06 + .quad 0x38ac15510a4811b8 + .quad 0x0eb28bf671928ce4 + .quad 0x671feaf300f42772 + .quad 0x8f72eb2a2a8c41aa + .quad 0x29a17fd797373292 + .quad 0x1defc6ad32b587a6 + .quad 0xaf5bbe1aef5195a7 + .quad 0x148c1277917b15ed + .quad 0x2991f7fb7ae5da2e + .quad 0x467d201bf8dd2867 + + // 2^196 * 1 * G + + .quad 0x7906ee72f7bd2e6b + .quad 0x05d270d6109abf4e + .quad 0x8d5cfe45b941a8a4 + .quad 0x44c218671c974287 + .quad 0x745f9d56296bc318 + .quad 0x993580d4d8152e65 + .quad 0xb0e5b13f5839e9ce + .quad 0x51fc2b28d43921c0 + .quad 0x1b8fd11795e2a98c + .quad 0x1c4e5ee12b6b6291 + .quad 0x5b30e7107424b572 + .quad 0x6e6b9de84c4f4ac6 + + // 2^196 * 2 * G + + .quad 0xdff25fce4b1de151 + .quad 0xd841c0c7e11c4025 + .quad 0x2554b3c854749c87 + .quad 0x2d292459908e0df9 + .quad 0x6b7c5f10f80cb088 + .quad 0x736b54dc56e42151 + .quad 0xc2b620a5c6ef99c4 + .quad 0x5f4c802cc3a06f42 + .quad 0x9b65c8f17d0752da + .quad 0x881ce338c77ee800 + .quad 0xc3b514f05b62f9e3 + .quad 0x66ed5dd5bec10d48 + + // 2^196 * 3 * G + + .quad 0x7d38a1c20bb2089d + .quad 0x808334e196ccd412 + .quad 0xc4a70b8c6c97d313 + .quad 0x2eacf8bc03007f20 + .quad 0xf0adf3c9cbca047d + .quad 0x81c3b2cbf4552f6b + .quad 0xcfda112d44735f93 + .quad 0x1f23a0c77e20048c + .quad 0xf235467be5bc1570 + .quad 0x03d2d9020dbab38c + .quad 0x27529aa2fcf9e09e + .quad 0x0840bef29d34bc50 + + // 2^196 * 4 * G + + .quad 0x796dfb35dc10b287 + .quad 0x27176bcd5c7ff29d + .quad 0x7f3d43e8c7b24905 + .quad 0x0304f5a191c54276 + .quad 0xcd54e06b7f37e4eb + .quad 0x8cc15f87f5e96cca + .quad 0xb8248bb0d3597dce + .quad 0x246affa06074400c + .quad 0x37d88e68fbe45321 + .quad 0x86097548c0d75032 + .quad 0x4e9b13ef894a0d35 + .quad 0x25a83cac5753d325 + + // 2^196 * 5 * G + + .quad 0x10222f48eed8165e + .quad 0x623fc1234b8bcf3a + .quad 0x1e145c09c221e8f0 + .quad 0x7ccfa59fca782630 + .quad 0x9f0f66293952b6e2 + .quad 0x33db5e0e0934267b + .quad 0xff45252bd609fedc + .quad 0x06be10f5c506e0c9 + .quad 0x1a9615a9b62a345f + .quad 0x22050c564a52fecc + .quad 0xa7a2788528bc0dfe + .quad 0x5e82770a1a1ee71d + + // 2^196 * 6 * G + + .quad 0x35425183ad896a5c + .quad 0xe8673afbe78d52f6 + .quad 0x2c66f25f92a35f64 + .quad 0x09d04f3b3b86b102 + .quad 0xe802e80a42339c74 + .quad 0x34175166a7fffae5 + .quad 0x34865d1f1c408cae + .quad 0x2cca982c605bc5ee + .quad 0xfd2d5d35197dbe6e + .quad 0x207c2eea8be4ffa3 + .quad 0x2613d8db325ae918 + .quad 0x7a325d1727741d3e + + // 2^196 * 7 * G + + .quad 0xd036b9bbd16dfde2 + .quad 0xa2055757c497a829 + .quad 0x8e6cc966a7f12667 + .quad 0x4d3b1a791239c180 + .quad 0xecd27d017e2a076a + .quad 0xd788689f1636495e + .quad 0x52a61af0919233e5 + .quad 0x2a479df17bb1ae64 + .quad 0x9e5eee8e33db2710 + .quad 0x189854ded6c43ca5 + .quad 0xa41c22c592718138 + .quad 0x27ad5538a43a5e9b + + // 2^196 * 8 * G + + .quad 0x2746dd4b15350d61 + .quad 0xd03fcbc8ee9521b7 + .quad 0xe86e365a138672ca + .quad 0x510e987f7e7d89e2 + .quad 0xcb5a7d638e47077c + .quad 0x8db7536120a1c059 + .quad 0x549e1e4d8bedfdcc + .quad 0x080153b7503b179d + .quad 0xdda69d930a3ed3e3 + .quad 0x3d386ef1cd60a722 + .quad 0xc817ad58bdaa4ee6 + .quad 0x23be8d554fe7372a + + // 2^200 * 1 * G + + .quad 0x95fe919a74ef4fad + .quad 0x3a827becf6a308a2 + .quad 0x964e01d309a47b01 + .quad 0x71c43c4f5ba3c797 + .quad 0xbc1ef4bd567ae7a9 + .quad 0x3f624cb2d64498bd + .quad 0xe41064d22c1f4ec8 + .quad 0x2ef9c5a5ba384001 + .quad 0xb6fd6df6fa9e74cd + .quad 0xf18278bce4af267a + .quad 0x8255b3d0f1ef990e + .quad 0x5a758ca390c5f293 + + // 2^200 * 2 * G + + .quad 0xa2b72710d9462495 + .quad 0x3aa8c6d2d57d5003 + .quad 0xe3d400bfa0b487ca + .quad 0x2dbae244b3eb72ec + .quad 0x8ce0918b1d61dc94 + .quad 0x8ded36469a813066 + .quad 0xd4e6a829afe8aad3 + .quad 0x0a738027f639d43f + .quad 0x980f4a2f57ffe1cc + .quad 0x00670d0de1839843 + .quad 0x105c3f4a49fb15fd + .quad 0x2698ca635126a69c + + // 2^200 * 3 * G + + .quad 0xe765318832b0ba78 + .quad 0x381831f7925cff8b + .quad 0x08a81b91a0291fcc + .quad 0x1fb43dcc49caeb07 + .quad 0x2e3d702f5e3dd90e + .quad 0x9e3f0918e4d25386 + .quad 0x5e773ef6024da96a + .quad 0x3c004b0c4afa3332 + .quad 0x9aa946ac06f4b82b + .quad 0x1ca284a5a806c4f3 + .quad 0x3ed3265fc6cd4787 + .quad 0x6b43fd01cd1fd217 + + // 2^200 * 4 * G + + .quad 0xc7a75d4b4697c544 + .quad 0x15fdf848df0fffbf + .quad 0x2868b9ebaa46785a + .quad 0x5a68d7105b52f714 + .quad 0xb5c742583e760ef3 + .quad 0x75dc52b9ee0ab990 + .quad 0xbf1427c2072b923f + .quad 0x73420b2d6ff0d9f0 + .quad 0xaf2cf6cb9e851e06 + .quad 0x8f593913c62238c4 + .quad 0xda8ab89699fbf373 + .quad 0x3db5632fea34bc9e + + // 2^200 * 5 * G + + .quad 0xf46eee2bf75dd9d8 + .quad 0x0d17b1f6396759a5 + .quad 0x1bf2d131499e7273 + .quad 0x04321adf49d75f13 + .quad 0x2e4990b1829825d5 + .quad 0xedeaeb873e9a8991 + .quad 0xeef03d394c704af8 + .quad 0x59197ea495df2b0e + .quad 0x04e16019e4e55aae + .quad 0xe77b437a7e2f92e9 + .quad 0xc7ce2dc16f159aa4 + .quad 0x45eafdc1f4d70cc0 + + // 2^200 * 6 * G + + .quad 0x698401858045d72b + .quad 0x4c22faa2cf2f0651 + .quad 0x941a36656b222dc6 + .quad 0x5a5eebc80362dade + .quad 0xb60e4624cfccb1ed + .quad 0x59dbc292bd5c0395 + .quad 0x31a09d1ddc0481c9 + .quad 0x3f73ceea5d56d940 + .quad 0xb7a7bfd10a4e8dc6 + .quad 0xbe57007e44c9b339 + .quad 0x60c1207f1557aefa + .quad 0x26058891266218db + + // 2^200 * 7 * G + + .quad 0x59f704a68360ff04 + .quad 0xc3d93fde7661e6f4 + .quad 0x831b2a7312873551 + .quad 0x54ad0c2e4e615d57 + .quad 0x4c818e3cc676e542 + .quad 0x5e422c9303ceccad + .quad 0xec07cccab4129f08 + .quad 0x0dedfa10b24443b8 + .quad 0xee3b67d5b82b522a + .quad 0x36f163469fa5c1eb + .quad 0xa5b4d2f26ec19fd3 + .quad 0x62ecb2baa77a9408 + + // 2^200 * 8 * G + + .quad 0xe5ed795261152b3d + .quad 0x4962357d0eddd7d1 + .quad 0x7482c8d0b96b4c71 + .quad 0x2e59f919a966d8be + .quad 0x92072836afb62874 + .quad 0x5fcd5e8579e104a5 + .quad 0x5aad01adc630a14a + .quad 0x61913d5075663f98 + .quad 0x0dc62d361a3231da + .quad 0xfa47583294200270 + .quad 0x02d801513f9594ce + .quad 0x3ddbc2a131c05d5c + + // 2^204 * 1 * G + + .quad 0x3f50a50a4ffb81ef + .quad 0xb1e035093bf420bf + .quad 0x9baa8e1cc6aa2cd0 + .quad 0x32239861fa237a40 + .quad 0xfb735ac2004a35d1 + .quad 0x31de0f433a6607c3 + .quad 0x7b8591bfc528d599 + .quad 0x55be9a25f5bb050c + .quad 0x0d005acd33db3dbf + .quad 0x0111b37c80ac35e2 + .quad 0x4892d66c6f88ebeb + .quad 0x770eadb16508fbcd + + // 2^204 * 2 * G + + .quad 0x8451f9e05e4e89dd + .quad 0xc06302ffbc793937 + .quad 0x5d22749556a6495c + .quad 0x09a6755ca05603fb + .quad 0xf1d3b681a05071b9 + .quad 0x2207659a3592ff3a + .quad 0x5f0169297881e40e + .quad 0x16bedd0e86ba374e + .quad 0x5ecccc4f2c2737b5 + .quad 0x43b79e0c2dccb703 + .quad 0x33e008bc4ec43df3 + .quad 0x06c1b840f07566c0 + + // 2^204 * 3 * G + + .quad 0x7688a5c6a388f877 + .quad 0x02a96c14deb2b6ac + .quad 0x64c9f3431b8c2af8 + .quad 0x3628435554a1eed6 + .quad 0x69ee9e7f9b02805c + .quad 0xcbff828a547d1640 + .quad 0x3d93a869b2430968 + .quad 0x46b7b8cd3fe26972 + .quad 0xe9812086fe7eebe0 + .quad 0x4cba6be72f515437 + .quad 0x1d04168b516efae9 + .quad 0x5ea1391043982cb9 + + // 2^204 * 4 * G + + .quad 0x49125c9cf4702ee1 + .quad 0x4520b71f8b25b32d + .quad 0x33193026501fef7e + .quad 0x656d8997c8d2eb2b + .quad 0x6f2b3be4d5d3b002 + .quad 0xafec33d96a09c880 + .quad 0x035f73a4a8bcc4cc + .quad 0x22c5b9284662198b + .quad 0xcb58c8fe433d8939 + .quad 0x89a0cb2e6a8d7e50 + .quad 0x79ca955309fbbe5a + .quad 0x0c626616cd7fc106 + + // 2^204 * 5 * G + + .quad 0x1ffeb80a4879b61f + .quad 0x6396726e4ada21ed + .quad 0x33c7b093368025ba + .quad 0x471aa0c6f3c31788 + .quad 0x8fdfc379fbf454b1 + .quad 0x45a5a970f1a4b771 + .quad 0xac921ef7bad35915 + .quad 0x42d088dca81c2192 + .quad 0x8fda0f37a0165199 + .quad 0x0adadb77c8a0e343 + .quad 0x20fbfdfcc875e820 + .quad 0x1cf2bea80c2206e7 + + // 2^204 * 6 * G + + .quad 0xc2ddf1deb36202ac + .quad 0x92a5fe09d2e27aa5 + .quad 0x7d1648f6fc09f1d3 + .quad 0x74c2cc0513bc4959 + .quad 0x982d6e1a02c0412f + .quad 0x90fa4c83db58e8fe + .quad 0x01c2f5bcdcb18bc0 + .quad 0x686e0c90216abc66 + .quad 0x1fadbadba54395a7 + .quad 0xb41a02a0ae0da66a + .quad 0xbf19f598bba37c07 + .quad 0x6a12b8acde48430d + + // 2^204 * 7 * G + + .quad 0xf8daea1f39d495d9 + .quad 0x592c190e525f1dfc + .quad 0xdb8cbd04c9991d1b + .quad 0x11f7fda3d88f0cb7 + .quad 0x793bdd801aaeeb5f + .quad 0x00a2a0aac1518871 + .quad 0xe8a373a31f2136b4 + .quad 0x48aab888fc91ef19 + .quad 0x041f7e925830f40e + .quad 0x002d6ca979661c06 + .quad 0x86dc9ff92b046a2e + .quad 0x760360928b0493d1 + + // 2^204 * 8 * G + + .quad 0x21bb41c6120cf9c6 + .quad 0xeab2aa12decda59b + .quad 0xc1a72d020aa48b34 + .quad 0x215d4d27e87d3b68 + .quad 0xb43108e5695a0b05 + .quad 0x6cb00ee8ad37a38b + .quad 0x5edad6eea3537381 + .quad 0x3f2602d4b6dc3224 + .quad 0xc8b247b65bcaf19c + .quad 0x49779dc3b1b2c652 + .quad 0x89a180bbd5ece2e2 + .quad 0x13f098a3cec8e039 + + // 2^208 * 1 * G + + .quad 0x9adc0ff9ce5ec54b + .quad 0x039c2a6b8c2f130d + .quad 0x028007c7f0f89515 + .quad 0x78968314ac04b36b + .quad 0xf3aa57a22796bb14 + .quad 0x883abab79b07da21 + .quad 0xe54be21831a0391c + .quad 0x5ee7fb38d83205f9 + .quad 0x538dfdcb41446a8e + .quad 0xa5acfda9434937f9 + .quad 0x46af908d263c8c78 + .quad 0x61d0633c9bca0d09 + + // 2^208 * 2 * G + + .quad 0x63744935ffdb2566 + .quad 0xc5bd6b89780b68bb + .quad 0x6f1b3280553eec03 + .quad 0x6e965fd847aed7f5 + .quad 0xada328bcf8fc73df + .quad 0xee84695da6f037fc + .quad 0x637fb4db38c2a909 + .quad 0x5b23ac2df8067bdc + .quad 0x9ad2b953ee80527b + .quad 0xe88f19aafade6d8d + .quad 0x0e711704150e82cf + .quad 0x79b9bbb9dd95dedc + + // 2^208 * 3 * G + + .quad 0xebb355406a3126c2 + .quad 0xd26383a868c8c393 + .quad 0x6c0c6429e5b97a82 + .quad 0x5065f158c9fd2147 + .quad 0xd1997dae8e9f7374 + .quad 0xa032a2f8cfbb0816 + .quad 0xcd6cba126d445f0a + .quad 0x1ba811460accb834 + .quad 0x708169fb0c429954 + .quad 0xe14600acd76ecf67 + .quad 0x2eaab98a70e645ba + .quad 0x3981f39e58a4faf2 + + // 2^208 * 4 * G + + .quad 0x18fb8a7559230a93 + .quad 0x1d168f6960e6f45d + .quad 0x3a85a94514a93cb5 + .quad 0x38dc083705acd0fd + .quad 0xc845dfa56de66fde + .quad 0xe152a5002c40483a + .quad 0xe9d2e163c7b4f632 + .quad 0x30f4452edcbc1b65 + .quad 0x856d2782c5759740 + .quad 0xfa134569f99cbecc + .quad 0x8844fc73c0ea4e71 + .quad 0x632d9a1a593f2469 + + // 2^208 * 5 * G + + .quad 0xf6bb6b15b807cba6 + .quad 0x1823c7dfbc54f0d7 + .quad 0xbb1d97036e29670b + .quad 0x0b24f48847ed4a57 + .quad 0xbf09fd11ed0c84a7 + .quad 0x63f071810d9f693a + .quad 0x21908c2d57cf8779 + .quad 0x3a5a7df28af64ba2 + .quad 0xdcdad4be511beac7 + .quad 0xa4538075ed26ccf2 + .quad 0xe19cff9f005f9a65 + .quad 0x34fcf74475481f63 + + // 2^208 * 6 * G + + .quad 0xc197e04c789767ca + .quad 0xb8714dcb38d9467d + .quad 0x55de888283f95fa8 + .quad 0x3d3bdc164dfa63f7 + .quad 0xa5bb1dab78cfaa98 + .quad 0x5ceda267190b72f2 + .quad 0x9309c9110a92608e + .quad 0x0119a3042fb374b0 + .quad 0x67a2d89ce8c2177d + .quad 0x669da5f66895d0c1 + .quad 0xf56598e5b282a2b0 + .quad 0x56c088f1ede20a73 + + // 2^208 * 7 * G + + .quad 0x336d3d1110a86e17 + .quad 0xd7f388320b75b2fa + .quad 0xf915337625072988 + .quad 0x09674c6b99108b87 + .quad 0x581b5fac24f38f02 + .quad 0xa90be9febae30cbd + .quad 0x9a2169028acf92f0 + .quad 0x038b7ea48359038f + .quad 0x9f4ef82199316ff8 + .quad 0x2f49d282eaa78d4f + .quad 0x0971a5ab5aef3174 + .quad 0x6e5e31025969eb65 + + // 2^208 * 8 * G + + .quad 0xb16c62f587e593fb + .quad 0x4999eddeca5d3e71 + .quad 0xb491c1e014cc3e6d + .quad 0x08f5114789a8dba8 + .quad 0x3304fb0e63066222 + .quad 0xfb35068987acba3f + .quad 0xbd1924778c1061a3 + .quad 0x3058ad43d1838620 + .quad 0x323c0ffde57663d0 + .quad 0x05c3df38a22ea610 + .quad 0xbdc78abdac994f9a + .quad 0x26549fa4efe3dc99 + + // 2^212 * 1 * G + + .quad 0x738b38d787ce8f89 + .quad 0xb62658e24179a88d + .quad 0x30738c9cf151316d + .quad 0x49128c7f727275c9 + .quad 0x04dbbc17f75396b9 + .quad 0x69e6a2d7d2f86746 + .quad 0xc6409d99f53eabc6 + .quad 0x606175f6332e25d2 + .quad 0x4021370ef540e7dd + .quad 0x0910d6f5a1f1d0a5 + .quad 0x4634aacd5b06b807 + .quad 0x6a39e6356944f235 + + // 2^212 * 2 * G + + .quad 0x96cd5640df90f3e7 + .quad 0x6c3a760edbfa25ea + .quad 0x24f3ef0959e33cc4 + .quad 0x42889e7e530d2e58 + .quad 0x1da1965774049e9d + .quad 0xfbcd6ea198fe352b + .quad 0xb1cbcd50cc5236a6 + .quad 0x1f5ec83d3f9846e2 + .quad 0x8efb23c3328ccb75 + .quad 0xaf42a207dd876ee9 + .quad 0x20fbdadc5dfae796 + .quad 0x241e246b06bf9f51 + + // 2^212 * 3 * G + + .quad 0x29e68e57ad6e98f6 + .quad 0x4c9260c80b462065 + .quad 0x3f00862ea51ebb4b + .quad 0x5bc2c77fb38d9097 + .quad 0x7eaafc9a6280bbb8 + .quad 0x22a70f12f403d809 + .quad 0x31ce40bb1bfc8d20 + .quad 0x2bc65635e8bd53ee + .quad 0xe8d5dc9fa96bad93 + .quad 0xe58fb17dde1947dc + .quad 0x681532ea65185fa3 + .quad 0x1fdd6c3b034a7830 + + // 2^212 * 4 * G + + .quad 0x0a64e28c55dc18fe + .quad 0xe3df9e993399ebdd + .quad 0x79ac432370e2e652 + .quad 0x35ff7fc33ae4cc0e + .quad 0x9c13a6a52dd8f7a9 + .quad 0x2dbb1f8c3efdcabf + .quad 0x961e32405e08f7b5 + .quad 0x48c8a121bbe6c9e5 + .quad 0xfc415a7c59646445 + .quad 0xd224b2d7c128b615 + .quad 0x6035c9c905fbb912 + .quad 0x42d7a91274429fab + + // 2^212 * 5 * G + + .quad 0x4e6213e3eaf72ed3 + .quad 0x6794981a43acd4e7 + .quad 0xff547cde6eb508cb + .quad 0x6fed19dd10fcb532 + .quad 0xa9a48947933da5bc + .quad 0x4a58920ec2e979ec + .quad 0x96d8800013e5ac4c + .quad 0x453692d74b48b147 + .quad 0xdd775d99a8559c6f + .quad 0xf42a2140df003e24 + .quad 0x5223e229da928a66 + .quad 0x063f46ba6d38f22c + + // 2^212 * 6 * G + + .quad 0xd2d242895f536694 + .quad 0xca33a2c542939b2c + .quad 0x986fada6c7ddb95c + .quad 0x5a152c042f712d5d + .quad 0x39843cb737346921 + .quad 0xa747fb0738c89447 + .quad 0xcb8d8031a245307e + .quad 0x67810f8e6d82f068 + .quad 0x3eeb8fbcd2287db4 + .quad 0x72c7d3a301a03e93 + .quad 0x5473e88cbd98265a + .quad 0x7324aa515921b403 + + // 2^212 * 7 * G + + .quad 0x857942f46c3cbe8e + .quad 0xa1d364b14730c046 + .quad 0x1c8ed914d23c41bf + .quad 0x0838e161eef6d5d2 + .quad 0xad23f6dae82354cb + .quad 0x6962502ab6571a6d + .quad 0x9b651636e38e37d1 + .quad 0x5cac5005d1a3312f + .quad 0x8cc154cce9e39904 + .quad 0x5b3a040b84de6846 + .quad 0xc4d8a61cb1be5d6e + .quad 0x40fb897bd8861f02 + + // 2^212 * 8 * G + + .quad 0x84c5aa9062de37a1 + .quad 0x421da5000d1d96e1 + .quad 0x788286306a9242d9 + .quad 0x3c5e464a690d10da + .quad 0xe57ed8475ab10761 + .quad 0x71435e206fd13746 + .quad 0x342f824ecd025632 + .quad 0x4b16281ea8791e7b + .quad 0xd1c101d50b813381 + .quad 0xdee60f1176ee6828 + .quad 0x0cb68893383f6409 + .quad 0x6183c565f6ff484a + + // 2^216 * 1 * G + + .quad 0x741d5a461e6bf9d6 + .quad 0x2305b3fc7777a581 + .quad 0xd45574a26474d3d9 + .quad 0x1926e1dc6401e0ff + .quad 0xdb468549af3f666e + .quad 0xd77fcf04f14a0ea5 + .quad 0x3df23ff7a4ba0c47 + .quad 0x3a10dfe132ce3c85 + .quad 0xe07f4e8aea17cea0 + .quad 0x2fd515463a1fc1fd + .quad 0x175322fd31f2c0f1 + .quad 0x1fa1d01d861e5d15 + + // 2^216 * 2 * G + + .quad 0xcc8055947d599832 + .quad 0x1e4656da37f15520 + .quad 0x99f6f7744e059320 + .quad 0x773563bc6a75cf33 + .quad 0x38dcac00d1df94ab + .quad 0x2e712bddd1080de9 + .quad 0x7f13e93efdd5e262 + .quad 0x73fced18ee9a01e5 + .quad 0x06b1e90863139cb3 + .quad 0xa493da67c5a03ecd + .quad 0x8d77cec8ad638932 + .quad 0x1f426b701b864f44 + + // 2^216 * 3 * G + + .quad 0xefc9264c41911c01 + .quad 0xf1a3b7b817a22c25 + .quad 0x5875da6bf30f1447 + .quad 0x4e1af5271d31b090 + .quad 0xf17e35c891a12552 + .quad 0xb76b8153575e9c76 + .quad 0xfa83406f0d9b723e + .quad 0x0b76bb1b3fa7e438 + .quad 0x08b8c1f97f92939b + .quad 0xbe6771cbd444ab6e + .quad 0x22e5646399bb8017 + .quad 0x7b6dd61eb772a955 + + // 2^216 * 4 * G + + .quad 0xb7adc1e850f33d92 + .quad 0x7998fa4f608cd5cf + .quad 0xad962dbd8dfc5bdb + .quad 0x703e9bceaf1d2f4f + .quad 0x5730abf9ab01d2c7 + .quad 0x16fb76dc40143b18 + .quad 0x866cbe65a0cbb281 + .quad 0x53fa9b659bff6afe + .quad 0x6c14c8e994885455 + .quad 0x843a5d6665aed4e5 + .quad 0x181bb73ebcd65af1 + .quad 0x398d93e5c4c61f50 + + // 2^216 * 5 * G + + .quad 0x1c4bd16733e248f3 + .quad 0xbd9e128715bf0a5f + .quad 0xd43f8cf0a10b0376 + .quad 0x53b09b5ddf191b13 + .quad 0xc3877c60d2e7e3f2 + .quad 0x3b34aaa030828bb1 + .quad 0x283e26e7739ef138 + .quad 0x699c9c9002c30577 + .quad 0xf306a7235946f1cc + .quad 0x921718b5cce5d97d + .quad 0x28cdd24781b4e975 + .quad 0x51caf30c6fcdd907 + + // 2^216 * 6 * G + + .quad 0xa60ba7427674e00a + .quad 0x630e8570a17a7bf3 + .quad 0x3758563dcf3324cc + .quad 0x5504aa292383fdaa + .quad 0x737af99a18ac54c7 + .quad 0x903378dcc51cb30f + .quad 0x2b89bc334ce10cc7 + .quad 0x12ae29c189f8e99a + .quad 0xa99ec0cb1f0d01cf + .quad 0x0dd1efcc3a34f7ae + .quad 0x55ca7521d09c4e22 + .quad 0x5fd14fe958eba5ea + + // 2^216 * 7 * G + + .quad 0xb5dc2ddf2845ab2c + .quad 0x069491b10a7fe993 + .quad 0x4daaf3d64002e346 + .quad 0x093ff26e586474d1 + .quad 0x3c42fe5ebf93cb8e + .quad 0xbedfa85136d4565f + .quad 0xe0f0859e884220e8 + .quad 0x7dd73f960725d128 + .quad 0xb10d24fe68059829 + .quad 0x75730672dbaf23e5 + .quad 0x1367253ab457ac29 + .quad 0x2f59bcbc86b470a4 + + // 2^216 * 8 * G + + .quad 0x83847d429917135f + .quad 0xad1b911f567d03d7 + .quad 0x7e7748d9be77aad1 + .quad 0x5458b42e2e51af4a + .quad 0x7041d560b691c301 + .quad 0x85201b3fadd7e71e + .quad 0x16c2e16311335585 + .quad 0x2aa55e3d010828b1 + .quad 0xed5192e60c07444f + .quad 0x42c54e2d74421d10 + .quad 0x352b4c82fdb5c864 + .quad 0x13e9004a8a768664 + + // 2^220 * 1 * G + + .quad 0xcbb5b5556c032bff + .quad 0xdf7191b729297a3a + .quad 0xc1ff7326aded81bb + .quad 0x71ade8bb68be03f5 + .quad 0x1e6284c5806b467c + .quad 0xc5f6997be75d607b + .quad 0x8b67d958b378d262 + .quad 0x3d88d66a81cd8b70 + .quad 0x8b767a93204ed789 + .quad 0x762fcacb9fa0ae2a + .quad 0x771febcc6dce4887 + .quad 0x343062158ff05fb3 + + // 2^220 * 2 * G + + .quad 0xe05da1a7e1f5bf49 + .quad 0x26457d6dd4736092 + .quad 0x77dcb07773cc32f6 + .quad 0x0a5d94969cdd5fcd + .quad 0xfce219072a7b31b4 + .quad 0x4d7adc75aa578016 + .quad 0x0ec276a687479324 + .quad 0x6d6d9d5d1fda4beb + .quad 0x22b1a58ae9b08183 + .quad 0xfd95d071c15c388b + .quad 0xa9812376850a0517 + .quad 0x33384cbabb7f335e + + // 2^220 * 3 * G + + .quad 0x3c6fa2680ca2c7b5 + .quad 0x1b5082046fb64fda + .quad 0xeb53349c5431d6de + .quad 0x5278b38f6b879c89 + .quad 0x33bc627a26218b8d + .quad 0xea80b21fc7a80c61 + .quad 0x9458b12b173e9ee6 + .quad 0x076247be0e2f3059 + .quad 0x52e105f61416375a + .quad 0xec97af3685abeba4 + .quad 0x26e6b50623a67c36 + .quad 0x5cf0e856f3d4fb01 + + // 2^220 * 4 * G + + .quad 0xf6c968731ae8cab4 + .quad 0x5e20741ecb4f92c5 + .quad 0x2da53be58ccdbc3e + .quad 0x2dddfea269970df7 + .quad 0xbeaece313db342a8 + .quad 0xcba3635b842db7ee + .quad 0xe88c6620817f13ef + .quad 0x1b9438aa4e76d5c6 + .quad 0x8a50777e166f031a + .quad 0x067b39f10fb7a328 + .quad 0x1925c9a6010fbd76 + .quad 0x6df9b575cc740905 + + // 2^220 * 5 * G + + .quad 0x42c1192927f6bdcf + .quad 0x8f91917a403d61ca + .quad 0xdc1c5a668b9e1f61 + .quad 0x1596047804ec0f8d + .quad 0xecdfc35b48cade41 + .quad 0x6a88471fb2328270 + .quad 0x740a4a2440a01b6a + .quad 0x471e5796003b5f29 + .quad 0xda96bbb3aced37ac + .quad 0x7a2423b5e9208cea + .quad 0x24cc5c3038aebae2 + .quad 0x50c356afdc5dae2f + + // 2^220 * 6 * G + + .quad 0x09dcbf4341c30318 + .quad 0xeeba061183181dce + .quad 0xc179c0cedc1e29a1 + .quad 0x1dbf7b89073f35b0 + .quad 0xcfed9cdf1b31b964 + .quad 0xf486a9858ca51af3 + .quad 0x14897265ea8c1f84 + .quad 0x784a53dd932acc00 + .quad 0x2d99f9df14fc4920 + .quad 0x76ccb60cc4499fe5 + .quad 0xa4132cbbe5cf0003 + .quad 0x3f93d82354f000ea + + // 2^220 * 7 * G + + .quad 0x8183e7689e04ce85 + .quad 0x678fb71e04465341 + .quad 0xad92058f6688edac + .quad 0x5da350d3532b099a + .quad 0xeaac12d179e14978 + .quad 0xff923ff3bbebff5e + .quad 0x4af663e40663ce27 + .quad 0x0fd381a811a5f5ff + .quad 0xf256aceca436df54 + .quad 0x108b6168ae69d6e8 + .quad 0x20d986cb6b5d036c + .quad 0x655957b9fee2af50 + + // 2^220 * 8 * G + + .quad 0xaea8b07fa902030f + .quad 0xf88c766af463d143 + .quad 0x15b083663c787a60 + .quad 0x08eab1148267a4a8 + .quad 0xbdc1409bd002d0ac + .quad 0x66660245b5ccd9a6 + .quad 0x82317dc4fade85ec + .quad 0x02fe934b6ad7df0d + .quad 0xef5cf100cfb7ea74 + .quad 0x22897633a1cb42ac + .quad 0xd4ce0c54cef285e2 + .quad 0x30408c048a146a55 + + // 2^224 * 1 * G + + .quad 0x739d8845832fcedb + .quad 0xfa38d6c9ae6bf863 + .quad 0x32bc0dcab74ffef7 + .quad 0x73937e8814bce45e + .quad 0xbb2e00c9193b877f + .quad 0xece3a890e0dc506b + .quad 0xecf3b7c036de649f + .quad 0x5f46040898de9e1a + .quad 0xb9037116297bf48d + .quad 0xa9d13b22d4f06834 + .quad 0xe19715574696bdc6 + .quad 0x2cf8a4e891d5e835 + + // 2^224 * 2 * G + + .quad 0x6d93fd8707110f67 + .quad 0xdd4c09d37c38b549 + .quad 0x7cb16a4cc2736a86 + .quad 0x2049bd6e58252a09 + .quad 0x2cb5487e17d06ba2 + .quad 0x24d2381c3950196b + .quad 0xd7659c8185978a30 + .quad 0x7a6f7f2891d6a4f6 + .quad 0x7d09fd8d6a9aef49 + .quad 0xf0ee60be5b3db90b + .quad 0x4c21b52c519ebfd4 + .quad 0x6011aadfc545941d + + // 2^224 * 3 * G + + .quad 0x5f67926dcf95f83c + .quad 0x7c7e856171289071 + .quad 0xd6a1e7f3998f7a5b + .quad 0x6fc5cc1b0b62f9e0 + .quad 0x63ded0c802cbf890 + .quad 0xfbd098ca0dff6aaa + .quad 0x624d0afdb9b6ed99 + .quad 0x69ce18b779340b1e + .quad 0xd1ef5528b29879cb + .quad 0xdd1aae3cd47e9092 + .quad 0x127e0442189f2352 + .quad 0x15596b3ae57101f1 + + // 2^224 * 4 * G + + .quad 0x462739d23f9179a2 + .quad 0xff83123197d6ddcf + .quad 0x1307deb553f2148a + .quad 0x0d2237687b5f4dda + .quad 0x09ff31167e5124ca + .quad 0x0be4158bd9c745df + .quad 0x292b7d227ef556e5 + .quad 0x3aa4e241afb6d138 + .quad 0x2cc138bf2a3305f5 + .quad 0x48583f8fa2e926c3 + .quad 0x083ab1a25549d2eb + .quad 0x32fcaa6e4687a36c + + // 2^224 * 5 * G + + .quad 0x7bc56e8dc57d9af5 + .quad 0x3e0bd2ed9df0bdf2 + .quad 0xaac014de22efe4a3 + .quad 0x4627e9cefebd6a5c + .quad 0x3207a4732787ccdf + .quad 0x17e31908f213e3f8 + .quad 0xd5b2ecd7f60d964e + .quad 0x746f6336c2600be9 + .quad 0x3f4af345ab6c971c + .quad 0xe288eb729943731f + .quad 0x33596a8a0344186d + .quad 0x7b4917007ed66293 + + // 2^224 * 6 * G + + .quad 0x2d85fb5cab84b064 + .quad 0x497810d289f3bc14 + .quad 0x476adc447b15ce0c + .quad 0x122ba376f844fd7b + .quad 0x54341b28dd53a2dd + .quad 0xaa17905bdf42fc3f + .quad 0x0ff592d94dd2f8f4 + .quad 0x1d03620fe08cd37d + .quad 0xc20232cda2b4e554 + .quad 0x9ed0fd42115d187f + .quad 0x2eabb4be7dd479d9 + .quad 0x02c70bf52b68ec4c + + // 2^224 * 7 * G + + .quad 0xa287ec4b5d0b2fbb + .quad 0x415c5790074882ca + .quad 0xe044a61ec1d0815c + .quad 0x26334f0a409ef5e0 + .quad 0xace532bf458d72e1 + .quad 0x5be768e07cb73cb5 + .quad 0x56cf7d94ee8bbde7 + .quad 0x6b0697e3feb43a03 + .quad 0xb6c8f04adf62a3c0 + .quad 0x3ef000ef076da45d + .quad 0x9c9cb95849f0d2a9 + .quad 0x1cc37f43441b2fae + + // 2^224 * 8 * G + + .quad 0x508f565a5cc7324f + .quad 0xd061c4c0e506a922 + .quad 0xfb18abdb5c45ac19 + .quad 0x6c6809c10380314a + .quad 0xd76656f1c9ceaeb9 + .quad 0x1c5b15f818e5656a + .quad 0x26e72832844c2334 + .quad 0x3a346f772f196838 + .quad 0xd2d55112e2da6ac8 + .quad 0xe9bd0331b1e851ed + .quad 0x960746dd8ec67262 + .quad 0x05911b9f6ef7c5d0 + + // 2^228 * 1 * G + + .quad 0xe9dcd756b637ff2d + .quad 0xec4c348fc987f0c4 + .quad 0xced59285f3fbc7b7 + .quad 0x3305354793e1ea87 + .quad 0x01c18980c5fe9f94 + .quad 0xcd656769716fd5c8 + .quad 0x816045c3d195a086 + .quad 0x6e2b7f3266cc7982 + .quad 0xcc802468f7c3568f + .quad 0x9de9ba8219974cb3 + .quad 0xabb7229cb5b81360 + .quad 0x44e2017a6fbeba62 + + // 2^228 * 2 * G + + .quad 0xc4c2a74354dab774 + .quad 0x8e5d4c3c4eaf031a + .quad 0xb76c23d242838f17 + .quad 0x749a098f68dce4ea + .quad 0x87f82cf3b6ca6ecd + .quad 0x580f893e18f4a0c2 + .quad 0x058930072604e557 + .quad 0x6cab6ac256d19c1d + .quad 0xdcdfe0a02cc1de60 + .quad 0x032665ff51c5575b + .quad 0x2c0c32f1073abeeb + .quad 0x6a882014cd7b8606 + + // 2^228 * 3 * G + + .quad 0xa52a92fea4747fb5 + .quad 0xdc12a4491fa5ab89 + .quad 0xd82da94bb847a4ce + .quad 0x4d77edce9512cc4e + .quad 0xd111d17caf4feb6e + .quad 0x050bba42b33aa4a3 + .quad 0x17514c3ceeb46c30 + .quad 0x54bedb8b1bc27d75 + .quad 0x77c8e14577e2189c + .quad 0xa3e46f6aff99c445 + .quad 0x3144dfc86d335343 + .quad 0x3a96559e7c4216a9 + + // 2^228 * 4 * G + + .quad 0x12550d37f42ad2ee + .quad 0x8b78e00498a1fbf5 + .quad 0x5d53078233894cb2 + .quad 0x02c84e4e3e498d0c + .quad 0x4493896880baaa52 + .quad 0x4c98afc4f285940e + .quad 0xef4aa79ba45448b6 + .quad 0x5278c510a57aae7f + .quad 0xa54dd074294c0b94 + .quad 0xf55d46b8df18ffb6 + .quad 0xf06fecc58dae8366 + .quad 0x588657668190d165 + + // 2^228 * 5 * G + + .quad 0xd47712311aef7117 + .quad 0x50343101229e92c7 + .quad 0x7a95e1849d159b97 + .quad 0x2449959b8b5d29c9 + .quad 0xbf5834f03de25cc3 + .quad 0xb887c8aed6815496 + .quad 0x5105221a9481e892 + .quad 0x6760ed19f7723f93 + .quad 0x669ba3b7ac35e160 + .quad 0x2eccf73fba842056 + .quad 0x1aec1f17c0804f07 + .quad 0x0d96bc031856f4e7 + + // 2^228 * 6 * G + + .quad 0x3318be7775c52d82 + .quad 0x4cb764b554d0aab9 + .quad 0xabcf3d27cc773d91 + .quad 0x3bf4d1848123288a + .quad 0xb1d534b0cc7505e1 + .quad 0x32cd003416c35288 + .quad 0xcb36a5800762c29d + .quad 0x5bfe69b9237a0bf8 + .quad 0x183eab7e78a151ab + .quad 0xbbe990c999093763 + .quad 0xff717d6e4ac7e335 + .quad 0x4c5cddb325f39f88 + + // 2^228 * 7 * G + + .quad 0xc0f6b74d6190a6eb + .quad 0x20ea81a42db8f4e4 + .quad 0xa8bd6f7d97315760 + .quad 0x33b1d60262ac7c21 + .quad 0x57750967e7a9f902 + .quad 0x2c37fdfc4f5b467e + .quad 0xb261663a3177ba46 + .quad 0x3a375e78dc2d532b + .quad 0x8141e72f2d4dddea + .quad 0xe6eafe9862c607c8 + .quad 0x23c28458573cafd0 + .quad 0x46b9476f4ff97346 + + // 2^228 * 8 * G + + .quad 0x0c1ffea44f901e5c + .quad 0x2b0b6fb72184b782 + .quad 0xe587ff910114db88 + .quad 0x37130f364785a142 + .quad 0x1215505c0d58359f + .quad 0x2a2013c7fc28c46b + .quad 0x24a0a1af89ea664e + .quad 0x4400b638a1130e1f + .quad 0x3a01b76496ed19c3 + .quad 0x31e00ab0ed327230 + .quad 0x520a885783ca15b1 + .quad 0x06aab9875accbec7 + + // 2^232 * 1 * G + + .quad 0xc1339983f5df0ebb + .quad 0xc0f3758f512c4cac + .quad 0x2cf1130a0bb398e1 + .quad 0x6b3cecf9aa270c62 + .quad 0x5349acf3512eeaef + .quad 0x20c141d31cc1cb49 + .quad 0x24180c07a99a688d + .quad 0x555ef9d1c64b2d17 + .quad 0x36a770ba3b73bd08 + .quad 0x624aef08a3afbf0c + .quad 0x5737ff98b40946f2 + .quad 0x675f4de13381749d + + // 2^232 * 2 * G + + .quad 0x0e2c52036b1782fc + .quad 0x64816c816cad83b4 + .quad 0xd0dcbdd96964073e + .quad 0x13d99df70164c520 + .quad 0xa12ff6d93bdab31d + .quad 0x0725d80f9d652dfe + .quad 0x019c4ff39abe9487 + .quad 0x60f450b882cd3c43 + .quad 0x014b5ec321e5c0ca + .quad 0x4fcb69c9d719bfa2 + .quad 0x4e5f1c18750023a0 + .quad 0x1c06de9e55edac80 + + // 2^232 * 3 * G + + .quad 0x990f7ad6a33ec4e2 + .quad 0x6608f938be2ee08e + .quad 0x9ca143c563284515 + .quad 0x4cf38a1fec2db60d + .quad 0xffd52b40ff6d69aa + .quad 0x34530b18dc4049bb + .quad 0x5e4a5c2fa34d9897 + .quad 0x78096f8e7d32ba2d + .quad 0xa0aaaa650dfa5ce7 + .quad 0xf9c49e2a48b5478c + .quad 0x4f09cc7d7003725b + .quad 0x373cad3a26091abe + + // 2^232 * 4 * G + + .quad 0xb294634d82c9f57c + .quad 0x1fcbfde124934536 + .quad 0x9e9c4db3418cdb5a + .quad 0x0040f3d9454419fc + .quad 0xf1bea8fb89ddbbad + .quad 0x3bcb2cbc61aeaecb + .quad 0x8f58a7bb1f9b8d9d + .quad 0x21547eda5112a686 + .quad 0xdefde939fd5986d3 + .quad 0xf4272c89510a380c + .quad 0xb72ba407bb3119b9 + .quad 0x63550a334a254df4 + + // 2^232 * 5 * G + + .quad 0x6507d6edb569cf37 + .quad 0x178429b00ca52ee1 + .quad 0xea7c0090eb6bd65d + .quad 0x3eea62c7daf78f51 + .quad 0x9bba584572547b49 + .quad 0xf305c6fae2c408e0 + .quad 0x60e8fa69c734f18d + .quad 0x39a92bafaa7d767a + .quad 0x9d24c713e693274e + .quad 0x5f63857768dbd375 + .quad 0x70525560eb8ab39a + .quad 0x68436a0665c9c4cd + + // 2^232 * 6 * G + + .quad 0xbc0235e8202f3f27 + .quad 0xc75c00e264f975b0 + .quad 0x91a4e9d5a38c2416 + .quad 0x17b6e7f68ab789f9 + .quad 0x1e56d317e820107c + .quad 0xc5266844840ae965 + .quad 0xc1e0a1c6320ffc7a + .quad 0x5373669c91611472 + .quad 0x5d2814ab9a0e5257 + .quad 0x908f2084c9cab3fc + .quad 0xafcaf5885b2d1eca + .quad 0x1cb4b5a678f87d11 + + // 2^232 * 7 * G + + .quad 0xb664c06b394afc6c + .quad 0x0c88de2498da5fb1 + .quad 0x4f8d03164bcad834 + .quad 0x330bca78de7434a2 + .quad 0x6b74aa62a2a007e7 + .quad 0xf311e0b0f071c7b1 + .quad 0x5707e438000be223 + .quad 0x2dc0fd2d82ef6eac + .quad 0x982eff841119744e + .quad 0xf9695e962b074724 + .quad 0xc58ac14fbfc953fb + .quad 0x3c31be1b369f1cf5 + + // 2^232 * 8 * G + + .quad 0xb0f4864d08948aee + .quad 0x07dc19ee91ba1c6f + .quad 0x7975cdaea6aca158 + .quad 0x330b61134262d4bb + .quad 0xc168bc93f9cb4272 + .quad 0xaeb8711fc7cedb98 + .quad 0x7f0e52aa34ac8d7a + .quad 0x41cec1097e7d55bb + .quad 0xf79619d7a26d808a + .quad 0xbb1fd49e1d9e156d + .quad 0x73d7c36cdba1df27 + .quad 0x26b44cd91f28777d + + // 2^236 * 1 * G + + .quad 0x300a9035393aa6d8 + .quad 0x2b501131a12bb1cd + .quad 0x7b1ff677f093c222 + .quad 0x4309c1f8cab82bad + .quad 0xaf44842db0285f37 + .quad 0x8753189047efc8df + .quad 0x9574e091f820979a + .quad 0x0e378d6069615579 + .quad 0xd9fa917183075a55 + .quad 0x4bdb5ad26b009fdc + .quad 0x7829ad2cd63def0e + .quad 0x078fc54975fd3877 + + // 2^236 * 2 * G + + .quad 0x87dfbd1428878f2d + .quad 0x134636dd1e9421a1 + .quad 0x4f17c951257341a3 + .quad 0x5df98d4bad296cb8 + .quad 0xe2004b5bb833a98a + .quad 0x44775dec2d4c3330 + .quad 0x3aa244067eace913 + .quad 0x272630e3d58e00a9 + .quad 0xf3678fd0ecc90b54 + .quad 0xf001459b12043599 + .quad 0x26725fbc3758b89b + .quad 0x4325e4aa73a719ae + + // 2^236 * 3 * G + + .quad 0x657dc6ef433c3493 + .quad 0x65375e9f80dbf8c3 + .quad 0x47fd2d465b372dae + .quad 0x4966ab79796e7947 + .quad 0xed24629acf69f59d + .quad 0x2a4a1ccedd5abbf4 + .quad 0x3535ca1f56b2d67b + .quad 0x5d8c68d043b1b42d + .quad 0xee332d4de3b42b0a + .quad 0xd84e5a2b16a4601c + .quad 0x78243877078ba3e4 + .quad 0x77ed1eb4184ee437 + + // 2^236 * 4 * G + + .quad 0xbfd4e13f201839a0 + .quad 0xaeefffe23e3df161 + .quad 0xb65b04f06b5d1fe3 + .quad 0x52e085fb2b62fbc0 + .quad 0x185d43f89e92ed1a + .quad 0xb04a1eeafe4719c6 + .quad 0x499fbe88a6f03f4f + .quad 0x5d8b0d2f3c859bdd + .quad 0x124079eaa54cf2ba + .quad 0xd72465eb001b26e7 + .quad 0x6843bcfdc97af7fd + .quad 0x0524b42b55eacd02 + + // 2^236 * 5 * G + + .quad 0xfd0d5dbee45447b0 + .quad 0x6cec351a092005ee + .quad 0x99a47844567579cb + .quad 0x59d242a216e7fa45 + .quad 0xbc18dcad9b829eac + .quad 0x23ae7d28b5f579d0 + .quad 0xc346122a69384233 + .quad 0x1a6110b2e7d4ac89 + .quad 0x4f833f6ae66997ac + .quad 0x6849762a361839a4 + .quad 0x6985dec1970ab525 + .quad 0x53045e89dcb1f546 + + // 2^236 * 6 * G + + .quad 0xcb8bb346d75353db + .quad 0xfcfcb24bae511e22 + .quad 0xcba48d40d50ae6ef + .quad 0x26e3bae5f4f7cb5d + .quad 0x84da3cde8d45fe12 + .quad 0xbd42c218e444e2d2 + .quad 0xa85196781f7e3598 + .quad 0x7642c93f5616e2b2 + .quad 0x2323daa74595f8e4 + .quad 0xde688c8b857abeb4 + .quad 0x3fc48e961c59326e + .quad 0x0b2e73ca15c9b8ba + + // 2^236 * 7 * G + + .quad 0xd6bb4428c17f5026 + .quad 0x9eb27223fb5a9ca7 + .quad 0xe37ba5031919c644 + .quad 0x21ce380db59a6602 + .quad 0x0e3fbfaf79c03a55 + .quad 0x3077af054cbb5acf + .quad 0xd5c55245db3de39f + .quad 0x015e68c1476a4af7 + .quad 0xc1d5285220066a38 + .quad 0x95603e523570aef3 + .quad 0x832659a7226b8a4d + .quad 0x5dd689091f8eedc9 + + // 2^236 * 8 * G + + .quad 0xcbac84debfd3c856 + .quad 0x1624c348b35ff244 + .quad 0xb7f88dca5d9cad07 + .quad 0x3b0e574da2c2ebe8 + .quad 0x1d022591a5313084 + .quad 0xca2d4aaed6270872 + .quad 0x86a12b852f0bfd20 + .quad 0x56e6c439ad7da748 + .quad 0xc704ff4942bdbae6 + .quad 0x5e21ade2b2de1f79 + .quad 0xe95db3f35652fad8 + .quad 0x0822b5378f08ebc1 + + // 2^240 * 1 * G + + .quad 0x51f048478f387475 + .quad 0xb25dbcf49cbecb3c + .quad 0x9aab1244d99f2055 + .quad 0x2c709e6c1c10a5d6 + .quad 0xe1b7f29362730383 + .quad 0x4b5279ffebca8a2c + .quad 0xdafc778abfd41314 + .quad 0x7deb10149c72610f + .quad 0xcb62af6a8766ee7a + .quad 0x66cbec045553cd0e + .quad 0x588001380f0be4b5 + .quad 0x08e68e9ff62ce2ea + + // 2^240 * 2 * G + + .quad 0x34ad500a4bc130ad + .quad 0x8d38db493d0bd49c + .quad 0xa25c3d98500a89be + .quad 0x2f1f3f87eeba3b09 + .quad 0x2f2d09d50ab8f2f9 + .quad 0xacb9218dc55923df + .quad 0x4a8f342673766cb9 + .quad 0x4cb13bd738f719f5 + .quad 0xf7848c75e515b64a + .quad 0xa59501badb4a9038 + .quad 0xc20d313f3f751b50 + .quad 0x19a1e353c0ae2ee8 + + // 2^240 * 3 * G + + .quad 0x7d1c7560bafa05c3 + .quad 0xb3e1a0a0c6e55e61 + .quad 0xe3529718c0d66473 + .quad 0x41546b11c20c3486 + .quad 0xb42172cdd596bdbd + .quad 0x93e0454398eefc40 + .quad 0x9fb15347b44109b5 + .quad 0x736bd3990266ae34 + .quad 0x85532d509334b3b4 + .quad 0x46fd114b60816573 + .quad 0xcc5f5f30425c8375 + .quad 0x412295a2b87fab5c + + // 2^240 * 4 * G + + .quad 0x19c99b88f57ed6e9 + .quad 0x5393cb266df8c825 + .quad 0x5cee3213b30ad273 + .quad 0x14e153ebb52d2e34 + .quad 0x2e655261e293eac6 + .quad 0x845a92032133acdb + .quad 0x460975cb7900996b + .quad 0x0760bb8d195add80 + .quad 0x413e1a17cde6818a + .quad 0x57156da9ed69a084 + .quad 0x2cbf268f46caccb1 + .quad 0x6b34be9bc33ac5f2 + + // 2^240 * 5 * G + + .quad 0xf3df2f643a78c0b2 + .quad 0x4c3e971ef22e027c + .quad 0xec7d1c5e49c1b5a3 + .quad 0x2012c18f0922dd2d + .quad 0x11fc69656571f2d3 + .quad 0xc6c9e845530e737a + .quad 0xe33ae7a2d4fe5035 + .quad 0x01b9c7b62e6dd30b + .quad 0x880b55e55ac89d29 + .quad 0x1483241f45a0a763 + .quad 0x3d36efdfc2e76c1f + .quad 0x08af5b784e4bade8 + + // 2^240 * 6 * G + + .quad 0x283499dc881f2533 + .quad 0x9d0525da779323b6 + .quad 0x897addfb673441f4 + .quad 0x32b79d71163a168d + .quad 0xe27314d289cc2c4b + .quad 0x4be4bd11a287178d + .quad 0x18d528d6fa3364ce + .quad 0x6423c1d5afd9826e + .quad 0xcc85f8d9edfcb36a + .quad 0x22bcc28f3746e5f9 + .quad 0xe49de338f9e5d3cd + .quad 0x480a5efbc13e2dcc + + // 2^240 * 7 * G + + .quad 0x0b51e70b01622071 + .quad 0x06b505cf8b1dafc5 + .quad 0x2c6bb061ef5aabcd + .quad 0x47aa27600cb7bf31 + .quad 0xb6614ce442ce221f + .quad 0x6e199dcc4c053928 + .quad 0x663fb4a4dc1cbe03 + .quad 0x24b31d47691c8e06 + .quad 0x2a541eedc015f8c3 + .quad 0x11a4fe7e7c693f7c + .quad 0xf0af66134ea278d6 + .quad 0x545b585d14dda094 + + // 2^240 * 8 * G + + .quad 0x67bf275ea0d43a0f + .quad 0xade68e34089beebe + .quad 0x4289134cd479e72e + .quad 0x0f62f9c332ba5454 + .quad 0x6204e4d0e3b321e1 + .quad 0x3baa637a28ff1e95 + .quad 0x0b0ccffd5b99bd9e + .quad 0x4d22dc3e64c8d071 + .quad 0xfcb46589d63b5f39 + .quad 0x5cae6a3f57cbcf61 + .quad 0xfebac2d2953afa05 + .quad 0x1c0fa01a36371436 + + // 2^244 * 1 * G + + .quad 0xe7547449bc7cd692 + .quad 0x0f9abeaae6f73ddf + .quad 0x4af01ca700837e29 + .quad 0x63ab1b5d3f1bc183 + .quad 0xc11ee5e854c53fae + .quad 0x6a0b06c12b4f3ff4 + .quad 0x33540f80e0b67a72 + .quad 0x15f18fc3cd07e3ef + .quad 0x32750763b028f48c + .quad 0x06020740556a065f + .quad 0xd53bd812c3495b58 + .quad 0x08706c9b865f508d + + // 2^244 * 2 * G + + .quad 0xf37ca2ab3d343dff + .quad 0x1a8c6a2d80abc617 + .quad 0x8e49e035d4ccffca + .quad 0x48b46beebaa1d1b9 + .quad 0xcc991b4138b41246 + .quad 0x243b9c526f9ac26b + .quad 0xb9ef494db7cbabbd + .quad 0x5fba433dd082ed00 + .quad 0x9c49e355c9941ad0 + .quad 0xb9734ade74498f84 + .quad 0x41c3fed066663e5c + .quad 0x0ecfedf8e8e710b3 + + // 2^244 * 3 * G + + .quad 0x76430f9f9cd470d9 + .quad 0xb62acc9ba42f6008 + .quad 0x1898297c59adad5e + .quad 0x7789dd2db78c5080 + .quad 0x744f7463e9403762 + .quad 0xf79a8dee8dfcc9c9 + .quad 0x163a649655e4cde3 + .quad 0x3b61788db284f435 + .quad 0xb22228190d6ef6b2 + .quad 0xa94a66b246ce4bfa + .quad 0x46c1a77a4f0b6cc7 + .quad 0x4236ccffeb7338cf + + // 2^244 * 4 * G + + .quad 0x8497404d0d55e274 + .quad 0x6c6663d9c4ad2b53 + .quad 0xec2fb0d9ada95734 + .quad 0x2617e120cdb8f73c + .quad 0x3bd82dbfda777df6 + .quad 0x71b177cc0b98369e + .quad 0x1d0e8463850c3699 + .quad 0x5a71945b48e2d1f1 + .quad 0x6f203dd5405b4b42 + .quad 0x327ec60410b24509 + .quad 0x9c347230ac2a8846 + .quad 0x77de29fc11ffeb6a + + // 2^244 * 5 * G + + .quad 0xb0ac57c983b778a8 + .quad 0x53cdcca9d7fe912c + .quad 0x61c2b854ff1f59dc + .quad 0x3a1a2cf0f0de7dac + .quad 0x835e138fecced2ca + .quad 0x8c9eaf13ea963b9a + .quad 0xc95fbfc0b2160ea6 + .quad 0x575e66f3ad877892 + .quad 0x99803a27c88fcb3a + .quad 0x345a6789275ec0b0 + .quad 0x459789d0ff6c2be5 + .quad 0x62f882651e70a8b2 + + // 2^244 * 6 * G + + .quad 0x085ae2c759ff1be4 + .quad 0x149145c93b0e40b7 + .quad 0xc467e7fa7ff27379 + .quad 0x4eeecf0ad5c73a95 + .quad 0x6d822986698a19e0 + .quad 0xdc9821e174d78a71 + .quad 0x41a85f31f6cb1f47 + .quad 0x352721c2bcda9c51 + .quad 0x48329952213fc985 + .quad 0x1087cf0d368a1746 + .quad 0x8e5261b166c15aa5 + .quad 0x2d5b2d842ed24c21 + + // 2^244 * 7 * G + + .quad 0x02cfebd9ebd3ded1 + .quad 0xd45b217739021974 + .quad 0x7576f813fe30a1b7 + .quad 0x5691b6f9a34ef6c2 + .quad 0x5eb7d13d196ac533 + .quad 0x377234ecdb80be2b + .quad 0xe144cffc7cf5ae24 + .quad 0x5226bcf9c441acec + .quad 0x79ee6c7223e5b547 + .quad 0x6f5f50768330d679 + .quad 0xed73e1e96d8adce9 + .quad 0x27c3da1e1d8ccc03 + + // 2^244 * 8 * G + + .quad 0x7eb9efb23fe24c74 + .quad 0x3e50f49f1651be01 + .quad 0x3ea732dc21858dea + .quad 0x17377bd75bb810f9 + .quad 0x28302e71630ef9f6 + .quad 0xc2d4a2032b64cee0 + .quad 0x090820304b6292be + .quad 0x5fca747aa82adf18 + .quad 0x232a03c35c258ea5 + .quad 0x86f23a2c6bcb0cf1 + .quad 0x3dad8d0d2e442166 + .quad 0x04a8933cab76862b + + // 2^248 * 1 * G + + .quad 0xd2c604b622943dff + .quad 0xbc8cbece44cfb3a0 + .quad 0x5d254ff397808678 + .quad 0x0fa3614f3b1ca6bf + .quad 0x69082b0e8c936a50 + .quad 0xf9c9a035c1dac5b6 + .quad 0x6fb73e54c4dfb634 + .quad 0x4005419b1d2bc140 + .quad 0xa003febdb9be82f0 + .quad 0x2089c1af3a44ac90 + .quad 0xf8499f911954fa8e + .quad 0x1fba218aef40ab42 + + // 2^248 * 2 * G + + .quad 0xab549448fac8f53e + .quad 0x81f6e89a7ba63741 + .quad 0x74fd6c7d6c2b5e01 + .quad 0x392e3acaa8c86e42 + .quad 0x4f3e57043e7b0194 + .quad 0xa81d3eee08daaf7f + .quad 0xc839c6ab99dcdef1 + .quad 0x6c535d13ff7761d5 + .quad 0x4cbd34e93e8a35af + .quad 0x2e0781445887e816 + .quad 0x19319c76f29ab0ab + .quad 0x25e17fe4d50ac13b + + // 2^248 * 3 * G + + .quad 0x0a289bd71e04f676 + .quad 0x208e1c52d6420f95 + .quad 0x5186d8b034691fab + .quad 0x255751442a9fb351 + .quad 0x915f7ff576f121a7 + .quad 0xc34a32272fcd87e3 + .quad 0xccba2fde4d1be526 + .quad 0x6bba828f8969899b + .quad 0xe2d1bc6690fe3901 + .quad 0x4cb54a18a0997ad5 + .quad 0x971d6914af8460d4 + .quad 0x559d504f7f6b7be4 + + // 2^248 * 4 * G + + .quad 0xa7738378b3eb54d5 + .quad 0x1d69d366a5553c7c + .quad 0x0a26cf62f92800ba + .quad 0x01ab12d5807e3217 + .quad 0x9c4891e7f6d266fd + .quad 0x0744a19b0307781b + .quad 0x88388f1d6061e23b + .quad 0x123ea6a3354bd50e + .quad 0x118d189041e32d96 + .quad 0xb9ede3c2d8315848 + .quad 0x1eab4271d83245d9 + .quad 0x4a3961e2c918a154 + + // 2^248 * 5 * G + + .quad 0x71dc3be0f8e6bba0 + .quad 0xd6cef8347effe30a + .quad 0xa992425fe13a476a + .quad 0x2cd6bce3fb1db763 + .quad 0x0327d644f3233f1e + .quad 0x499a260e34fcf016 + .quad 0x83b5a716f2dab979 + .quad 0x68aceead9bd4111f + .quad 0x38b4c90ef3d7c210 + .quad 0x308e6e24b7ad040c + .quad 0x3860d9f1b7e73e23 + .quad 0x595760d5b508f597 + + // 2^248 * 6 * G + + .quad 0x6129bfe104aa6397 + .quad 0x8f960008a4a7fccb + .quad 0x3f8bc0897d909458 + .quad 0x709fa43edcb291a9 + .quad 0x882acbebfd022790 + .quad 0x89af3305c4115760 + .quad 0x65f492e37d3473f4 + .quad 0x2cb2c5df54515a2b + .quad 0xeb0a5d8c63fd2aca + .quad 0xd22bc1662e694eff + .quad 0x2723f36ef8cbb03a + .quad 0x70f029ecf0c8131f + + // 2^248 * 7 * G + + .quad 0x461307b32eed3e33 + .quad 0xae042f33a45581e7 + .quad 0xc94449d3195f0366 + .quad 0x0b7d5d8a6c314858 + .quad 0x2a6aafaa5e10b0b9 + .quad 0x78f0a370ef041aa9 + .quad 0x773efb77aa3ad61f + .quad 0x44eca5a2a74bd9e1 + .quad 0x25d448327b95d543 + .quad 0x70d38300a3340f1d + .quad 0xde1c531c60e1c52b + .quad 0x272224512c7de9e4 + + // 2^248 * 8 * G + + .quad 0x1abc92af49c5342e + .quad 0xffeed811b2e6fad0 + .quad 0xefa28c8dfcc84e29 + .quad 0x11b5df18a44cc543 + .quad 0xbf7bbb8a42a975fc + .quad 0x8c5c397796ada358 + .quad 0xe27fc76fcdedaa48 + .quad 0x19735fd7f6bc20a6 + .quad 0xe3ab90d042c84266 + .quad 0xeb848e0f7f19547e + .quad 0x2503a1d065a497b9 + .quad 0x0fef911191df895f + + // 2^252 * 1 * G + + .quad 0xb1507ca1ab1c6eb9 + .quad 0xbd448f3e16b687b3 + .quad 0x3455fb7f2c7a91ab + .quad 0x7579229e2f2adec1 + .quad 0x6ab5dcb85b1c16b7 + .quad 0x94c0fce83c7b27a5 + .quad 0xa4b11c1a735517be + .quad 0x499238d0ba0eafaa + .quad 0xecf46e527aba8b57 + .quad 0x15a08c478bd1647b + .quad 0x7af1c6a65f706fef + .quad 0x6345fa78f03a30d5 + + // 2^252 * 2 * G + + .quad 0xdf02f95f1015e7a1 + .quad 0x790ec41da9b40263 + .quad 0x4d3a0ea133ea1107 + .quad 0x54f70be7e33af8c9 + .quad 0x93d3cbe9bdd8f0a4 + .quad 0xdb152c1bfd177302 + .quad 0x7dbddc6d7f17a875 + .quad 0x3e1a71cc8f426efe + .quad 0xc83ca3e390babd62 + .quad 0x80ede3670291c833 + .quad 0xc88038ccd37900c4 + .quad 0x2c5fc0231ec31fa1 + + // 2^252 * 3 * G + + .quad 0xfeba911717038b4f + .quad 0xe5123721c9deef81 + .quad 0x1c97e4e75d0d8834 + .quad 0x68afae7a23dc3bc6 + .quad 0xc422e4d102456e65 + .quad 0x87414ac1cad47b91 + .quad 0x1592e2bba2b6ffdd + .quad 0x75d9d2bff5c2100f + .quad 0x5bd9b4763626e81c + .quad 0x89966936bca02edd + .quad 0x0a41193d61f077b3 + .quad 0x3097a24200ce5471 + + // 2^252 * 4 * G + + .quad 0x57427734c7f8b84c + .quad 0xf141a13e01b270e9 + .quad 0x02d1adfeb4e564a6 + .quad 0x4bb23d92ce83bd48 + .quad 0xa162e7246695c486 + .quad 0x131d633435a89607 + .quad 0x30521561a0d12a37 + .quad 0x56704bada6afb363 + .quad 0xaf6c4aa752f912b9 + .quad 0x5e665f6cd86770c8 + .quad 0x4c35ac83a3c8cd58 + .quad 0x2b7a29c010a58a7e + + // 2^252 * 5 * G + + .quad 0xc4007f77d0c1cec3 + .quad 0x8d1020b6bac492f8 + .quad 0x32ec29d57e69daaf + .quad 0x599408759d95fce0 + .quad 0x33810a23bf00086e + .quad 0xafce925ee736ff7c + .quad 0x3d60e670e24922d4 + .quad 0x11ce9e714f96061b + .quad 0x219ef713d815bac1 + .quad 0xf141465d485be25c + .quad 0x6d5447cc4e513c51 + .quad 0x174926be5ef44393 + + // 2^252 * 6 * G + + .quad 0xb5deb2f9fc5bd5bb + .quad 0x92daa72ae1d810e1 + .quad 0xafc4cfdcb72a1c59 + .quad 0x497d78813fc22a24 + .quad 0x3ef5d41593ea022e + .quad 0x5cbcc1a20ed0eed6 + .quad 0x8fd24ecf07382c8c + .quad 0x6fa42ead06d8e1ad + .quad 0xe276824a1f73371f + .quad 0x7f7cf01c4f5b6736 + .quad 0x7e201fe304fa46e7 + .quad 0x785a36a357808c96 + + // 2^252 * 7 * G + + .quad 0x825fbdfd63014d2b + .quad 0xc852369c6ca7578b + .quad 0x5b2fcd285c0b5df0 + .quad 0x12ab214c58048c8f + .quad 0x070442985d517bc3 + .quad 0x6acd56c7ae653678 + .quad 0x00a27983985a7763 + .quad 0x5167effae512662b + .quad 0xbd4ea9e10f53c4b6 + .quad 0x1673dc5f8ac91a14 + .quad 0xa8f81a4e2acc1aba + .quad 0x33a92a7924332a25 + + // 2^252 * 8 * G + + .quad 0x9dd1f49927996c02 + .quad 0x0cb3b058e04d1752 + .quad 0x1f7e88967fd02c3e + .quad 0x2f964268cb8b3eb1 + .quad 0x7ba95ba0218f2ada + .quad 0xcff42287330fb9ca + .quad 0xdada496d56c6d907 + .quad 0x5380c296f4beee54 + .quad 0x9d4f270466898d0a + .quad 0x3d0987990aff3f7a + .quad 0xd09ef36267daba45 + .quad 0x7761455e7b1c669c diff --git a/cbits/s2n/arm/curve25519_x25519base_alt.S b/cbits/s2n/arm/curve25519_x25519base_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/curve25519_x25519base_alt.S @@ -0,0 +1,9434 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// The x25519 function for curve25519 on base element 9 +// Input scalar[4]; output res[4] +// +// extern void curve25519_x25519base_alt +// (uint64_t res[static 4],const uint64_t scalar[static 4]); +// +// Given a scalar n, returns the X coordinate of n * G where G = (9,...) is +// the standard generator. The scalar is first slightly modified/mangled +// as specified in the relevant RFC (https://www.rfc-editor.org/rfc/rfc7748). +// +// Standard ARM ABI: X0 = res, X1 = scalar +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(curve25519_x25519base_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(curve25519_x25519base_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(curve25519_x25519base_alt) + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Stable home for the input result argument during the whole body + +#define res x23 + +// Other variables that are only needed prior to the modular inverse. + +#define tab x19 + +#define i x20 + +#define bias x21 + +#define bf x22 +#define ix x22 + +// Pointer-offset pairs for result and temporaries on stack with some aliasing. + +#define resx res, #(0*NUMSIZE) + +#define scalar sp, #(0*NUMSIZE) + +#define tabent sp, #(1*NUMSIZE) +#define ymx_2 sp, #(1*NUMSIZE) +#define xpy_2 sp, #(2*NUMSIZE) +#define kxy_2 sp, #(3*NUMSIZE) + +#define acc sp, #(4*NUMSIZE) +#define x_1 sp, #(4*NUMSIZE) +#define y_1 sp, #(5*NUMSIZE) +#define z_1 sp, #(6*NUMSIZE) +#define w_1 sp, #(7*NUMSIZE) +#define x_3 sp, #(4*NUMSIZE) +#define y_3 sp, #(5*NUMSIZE) +#define z_3 sp, #(6*NUMSIZE) +#define w_3 sp, #(7*NUMSIZE) + +#define tmpspace sp, #(8*NUMSIZE) +#define t0 sp, #(8*NUMSIZE) +#define t1 sp, #(9*NUMSIZE) +#define t2 sp, #(10*NUMSIZE) +#define t3 sp, #(11*NUMSIZE) +#define t4 sp, #(12*NUMSIZE) +#define t5 sp, #(13*NUMSIZE) + +// Total size to reserve on the stack + +#define NSPACE 14*NUMSIZE + +// Macro wrapping up the basic field operation bignum_mul_p25519_alt, only +// trivially different from a pure function call to that subroutine. + +#define mul_p25519(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x7, x8, [P2] __LF \ + mul x12, x3, x7 __LF \ + umulh x13, x3, x7 __LF \ + mul x11, x3, x8 __LF \ + umulh x14, x3, x8 __LF \ + adds x13, x13, x11 __LF \ + ldp x9, x10, [P2+16] __LF \ + mul x11, x3, x9 __LF \ + umulh x15, x3, x9 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x3, x10 __LF \ + umulh x16, x3, x10 __LF \ + adcs x15, x15, x11 __LF \ + adc x16, x16, xzr __LF \ + ldp x5, x6, [P1+16] __LF \ + mul x11, x4, x7 __LF \ + adds x13, x13, x11 __LF \ + mul x11, x4, x8 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x4, x9 __LF \ + adcs x15, x15, x11 __LF \ + mul x11, x4, x10 __LF \ + adcs x16, x16, x11 __LF \ + umulh x3, x4, x10 __LF \ + adc x3, x3, xzr __LF \ + umulh x11, x4, x7 __LF \ + adds x14, x14, x11 __LF \ + umulh x11, x4, x8 __LF \ + adcs x15, x15, x11 __LF \ + umulh x11, x4, x9 __LF \ + adcs x16, x16, x11 __LF \ + adc x3, x3, xzr __LF \ + mul x11, x5, x7 __LF \ + adds x14, x14, x11 __LF \ + mul x11, x5, x8 __LF \ + adcs x15, x15, x11 __LF \ + mul x11, x5, x9 __LF \ + adcs x16, x16, x11 __LF \ + mul x11, x5, x10 __LF \ + adcs x3, x3, x11 __LF \ + umulh x4, x5, x10 __LF \ + adc x4, x4, xzr __LF \ + umulh x11, x5, x7 __LF \ + adds x15, x15, x11 __LF \ + umulh x11, x5, x8 __LF \ + adcs x16, x16, x11 __LF \ + umulh x11, x5, x9 __LF \ + adcs x3, x3, x11 __LF \ + adc x4, x4, xzr __LF \ + mul x11, x6, x7 __LF \ + adds x15, x15, x11 __LF \ + mul x11, x6, x8 __LF \ + adcs x16, x16, x11 __LF \ + mul x11, x6, x9 __LF \ + adcs x3, x3, x11 __LF \ + mul x11, x6, x10 __LF \ + adcs x4, x4, x11 __LF \ + umulh x5, x6, x10 __LF \ + adc x5, x5, xzr __LF \ + umulh x11, x6, x7 __LF \ + adds x16, x16, x11 __LF \ + umulh x11, x6, x8 __LF \ + adcs x3, x3, x11 __LF \ + umulh x11, x6, x9 __LF \ + adcs x4, x4, x11 __LF \ + adc x5, x5, xzr __LF \ + mov x7, #0x26 __LF \ + mul x11, x7, x16 __LF \ + umulh x9, x7, x16 __LF \ + adds x12, x12, x11 __LF \ + mul x11, x7, x3 __LF \ + umulh x3, x7, x3 __LF \ + adcs x13, x13, x11 __LF \ + mul x11, x7, x4 __LF \ + umulh x4, x7, x4 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x7, x5 __LF \ + umulh x5, x7, x5 __LF \ + adcs x15, x15, x11 __LF \ + cset x16, cs __LF \ + adds x15, x15, x4 __LF \ + adc x16, x16, x5 __LF \ + cmn x15, x15 __LF \ + orr x15, x15, #0x8000000000000000 __LF \ + adc x8, x16, x16 __LF \ + mov x7, #0x13 __LF \ + madd x11, x7, x8, x7 __LF \ + adds x12, x12, x11 __LF \ + adcs x13, x13, x9 __LF \ + adcs x14, x14, x3 __LF \ + adcs x15, x15, xzr __LF \ + csel x7, x7, xzr, cc __LF \ + subs x12, x12, x7 __LF \ + sbcs x13, x13, xzr __LF \ + sbcs x14, x14, xzr __LF \ + sbc x15, x15, xzr __LF \ + and x15, x15, #0x7fffffffffffffff __LF \ + stp x12, x13, [P0] __LF \ + stp x14, x15, [P0+16] + +// A version of multiplication that only guarantees output < 2 * p_25519. +// This basically skips the +1 and final correction in quotient estimation. + +#define mul_4(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x7, x8, [P2] __LF \ + mul x12, x3, x7 __LF \ + umulh x13, x3, x7 __LF \ + mul x11, x3, x8 __LF \ + umulh x14, x3, x8 __LF \ + adds x13, x13, x11 __LF \ + ldp x9, x10, [P2+16] __LF \ + mul x11, x3, x9 __LF \ + umulh x15, x3, x9 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x3, x10 __LF \ + umulh x16, x3, x10 __LF \ + adcs x15, x15, x11 __LF \ + adc x16, x16, xzr __LF \ + ldp x5, x6, [P1+16] __LF \ + mul x11, x4, x7 __LF \ + adds x13, x13, x11 __LF \ + mul x11, x4, x8 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x4, x9 __LF \ + adcs x15, x15, x11 __LF \ + mul x11, x4, x10 __LF \ + adcs x16, x16, x11 __LF \ + umulh x3, x4, x10 __LF \ + adc x3, x3, xzr __LF \ + umulh x11, x4, x7 __LF \ + adds x14, x14, x11 __LF \ + umulh x11, x4, x8 __LF \ + adcs x15, x15, x11 __LF \ + umulh x11, x4, x9 __LF \ + adcs x16, x16, x11 __LF \ + adc x3, x3, xzr __LF \ + mul x11, x5, x7 __LF \ + adds x14, x14, x11 __LF \ + mul x11, x5, x8 __LF \ + adcs x15, x15, x11 __LF \ + mul x11, x5, x9 __LF \ + adcs x16, x16, x11 __LF \ + mul x11, x5, x10 __LF \ + adcs x3, x3, x11 __LF \ + umulh x4, x5, x10 __LF \ + adc x4, x4, xzr __LF \ + umulh x11, x5, x7 __LF \ + adds x15, x15, x11 __LF \ + umulh x11, x5, x8 __LF \ + adcs x16, x16, x11 __LF \ + umulh x11, x5, x9 __LF \ + adcs x3, x3, x11 __LF \ + adc x4, x4, xzr __LF \ + mul x11, x6, x7 __LF \ + adds x15, x15, x11 __LF \ + mul x11, x6, x8 __LF \ + adcs x16, x16, x11 __LF \ + mul x11, x6, x9 __LF \ + adcs x3, x3, x11 __LF \ + mul x11, x6, x10 __LF \ + adcs x4, x4, x11 __LF \ + umulh x5, x6, x10 __LF \ + adc x5, x5, xzr __LF \ + umulh x11, x6, x7 __LF \ + adds x16, x16, x11 __LF \ + umulh x11, x6, x8 __LF \ + adcs x3, x3, x11 __LF \ + umulh x11, x6, x9 __LF \ + adcs x4, x4, x11 __LF \ + adc x5, x5, xzr __LF \ + mov x7, #0x26 __LF \ + mul x11, x7, x16 __LF \ + umulh x9, x7, x16 __LF \ + adds x12, x12, x11 __LF \ + mul x11, x7, x3 __LF \ + umulh x3, x7, x3 __LF \ + adcs x13, x13, x11 __LF \ + mul x11, x7, x4 __LF \ + umulh x4, x7, x4 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x7, x5 __LF \ + umulh x5, x7, x5 __LF \ + adcs x15, x15, x11 __LF \ + cset x16, cs __LF \ + adds x15, x15, x4 __LF \ + adc x16, x16, x5 __LF \ + cmn x15, x15 __LF \ + bic x15, x15, #0x8000000000000000 __LF \ + adc x8, x16, x16 __LF \ + mov x7, #0x13 __LF \ + mul x11, x7, x8 __LF \ + adds x12, x12, x11 __LF \ + adcs x13, x13, x9 __LF \ + adcs x14, x14, x3 __LF \ + adc x15, x15, xzr __LF \ + stp x12, x13, [P0] __LF \ + stp x14, x15, [P0+16] + +// Modular subtraction with double modulus 2 * p_25519 = 2^256 - 38 + +#define sub_twice4(P0,P1,P2) \ + ldp x5, x6, [P1] __LF \ + ldp x4, x3, [P2] __LF \ + subs x5, x5, x4 __LF \ + sbcs x6, x6, x3 __LF \ + ldp x7, x8, [P1+16] __LF \ + ldp x4, x3, [P2+16] __LF \ + sbcs x7, x7, x4 __LF \ + sbcs x8, x8, x3 __LF \ + mov x4, #38 __LF \ + csel x3, x4, xzr, lo __LF \ + subs x5, x5, x3 __LF \ + sbcs x6, x6, xzr __LF \ + sbcs x7, x7, xzr __LF \ + sbc x8, x8, xzr __LF \ + stp x5, x6, [P0] __LF \ + stp x7, x8, [P0+16] + +// Modular addition and doubling with double modulus 2 * p_25519 = 2^256 - 38. +// This only ensures that the result fits in 4 digits, not that it is reduced +// even w.r.t. double modulus. The result is always correct modulo provided +// the sum of the inputs is < 2^256 + 2^256 - 38, so in particular provided +// at least one of them is reduced double modulo. + +#define add_twice4(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x7, x8, [P2] __LF \ + adds x3, x3, x7 __LF \ + adcs x4, x4, x8 __LF \ + ldp x5, x6, [P1+16] __LF \ + ldp x7, x8, [P2+16] __LF \ + adcs x5, x5, x7 __LF \ + adcs x6, x6, x8 __LF \ + mov x9, #38 __LF \ + csel x9, x9, xzr, cs __LF \ + adds x3, x3, x9 __LF \ + adcs x4, x4, xzr __LF \ + adcs x5, x5, xzr __LF \ + adc x6, x6, xzr __LF \ + stp x3, x4, [P0] __LF \ + stp x5, x6, [P0+16] + +#define double_twice4(P0,P1) \ + ldp x3, x4, [P1] __LF \ + adds x3, x3, x3 __LF \ + adcs x4, x4, x4 __LF \ + ldp x5, x6, [P1+16] __LF \ + adcs x5, x5, x5 __LF \ + adcs x6, x6, x6 __LF \ + mov x9, #38 __LF \ + csel x9, x9, xzr, cs __LF \ + adds x3, x3, x9 __LF \ + adcs x4, x4, xzr __LF \ + adcs x5, x5, xzr __LF \ + adc x6, x6, xzr __LF \ + stp x3, x4, [P0] __LF \ + stp x5, x6, [P0+16] + +S2N_BN_SYMBOL(curve25519_x25519base_alt): + CFI_START + +// Save regs and make room for temporaries + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_DEC_SP(NSPACE) + +// Move the output pointer to a stable place + + mov res, x0 + +// Copy the input scalar to its local variable while mangling it. +// In principle the mangling is into 01xxx...xxx000, but actually +// we only clear the top two bits so 00xxx...xxxxxx. The additional +// 2^254 * G is taken care of by the starting value for the addition +// chain below, while we never look at the three low bits at all. + + ldp x10, x11, [x1] + stp x10, x11, [scalar] + ldp x12, x13, [x1, #16] + bic x13, x13, #0xc000000000000000 + stp x12, x13, [scalar+16] + +// The main part of the computation is on the edwards25519 curve in +// extended-projective coordinates (X,Y,Z,T), representing a point +// (x,y) via x = X/Z, y = Y/Z and x * y = T/Z (so X * Y = T * Z). +// Only at the very end do we translate back to curve25519. So G +// below means the generator within edwards25519 corresponding to +// (9,...) for curve25519, via the standard isomorphism. +// +// Initialize accumulator "acc" to either (2^254 + 8) * G or just 2^254 * G +// depending on bit 3 of the scalar, the only nonzero bit of the bottom 4. +// Thus, we have effectively dealt with bits 0, 1, 2, 3, 254 and 255. + + ldr x0, [scalar] + ands xzr, x0, #8 + +#if defined(__ELF__) + adrp tab, S2N_BN_SYMBOL(curve25519_x25519base_alt_constant) + add tab, tab, :lo12:S2N_BN_SYMBOL(curve25519_x25519base_alt_constant) +#else + adrp tab, S2N_BN_SYMBOL(curve25519_x25519base_alt_constant)@PAGE + add tab, tab, S2N_BN_SYMBOL(curve25519_x25519base_alt_constant)@PAGEOFF +#endif + + ldp x0, x1, [tab] + ldp x2, x3, [tab, #96] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc] + + ldp x0, x1, [tab, #1*16] + ldp x2, x3, [tab, #96+1*16] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+1*16] + + ldp x0, x1, [tab, #2*16] + ldp x2, x3, [tab, #96+2*16] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+2*16] + + ldp x0, x1, [tab, #3*16] + ldp x2, x3, [tab, #96+3*16] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+3*16] + + mov x0, #1 + stp x0, xzr, [acc+4*16] + stp xzr, xzr, [acc+5*16] + + ldp x0, x1, [tab, #4*16] + ldp x2, x3, [tab, #96+4*16] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+6*16] + + ldp x0, x1, [tab, #5*16] + ldp x2, x3, [tab, #96+5*16] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+7*16] + +// The counter "i" tracks the bit position for which the scalar has +// already been absorbed, starting at 4 and going up in chunks of 4. +// +// The pointer "tab" points at the current block of the table for +// multiples (2^i * j) * G at the current bit position i; 1 <= j <= 8. +// +// The bias is always either 0 and 1 and needs to be added to the +// partially processed scalar implicitly. This is used to absorb 4 bits +// of scalar per iteration from 3-bit table indexing by exploiting +// negation: (16 * h + l) * G = (16 * (h + 1) - (16 - l)) * G is used +// when l >= 9. Note that we can't have any bias left over at the +// end because of the clearing of bit 255 of the scalar, meaning the +// l >= 9 case cannot arise on the last iteration. + + mov i, 4 + add tab, tab, #192 + mov bias, xzr + +// Start of the main loop, repeated 63 times for i = 4, 8, ..., 252 + +Lcurve25519_x25519base_alt_scalarloop: + +// Look at the next 4-bit field "bf", adding the previous bias as well. +// Choose the table index "ix" as bf when bf <= 8 and 16 - bf for bf >= 9, +// setting the bias to 1 for the next iteration in the latter case. + + lsr x0, i, #6 + ldr x2, [sp, x0, lsl #3] // Exploiting scalar = sp exactly + lsr x2, x2, i + and x2, x2, #15 + add bf, x2, bias + + cmp bf, 9 + cset bias, cs + + mov x0, 16 + sub x0, x0, bf + cmp bias, xzr + csel ix, x0, bf, ne + +// Perform constant-time lookup in the table to get element number "ix". +// The table entry for the affine point (x,y) is actually a triple +// (y - x,x + y,2 * d * x * y) to precompute parts of the addition. +// Note that "ix" can be 0, so we set up the appropriate identity first. + + mov x0, #1 + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, #1 + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + mov x9, xzr + mov x10, xzr + mov x11, xzr + + cmp ix, #1 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #2 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #3 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #4 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #5 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #6 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #7 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #8 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + +// We now have the triple from the table in registers as follows +// +// [x3;x2;x1;x0] = y - x +// [x7;x6;x5;x4] = x + y +// [x11;x10;x9;x8] = 2 * d * x * y +// +// In case bias = 1 we need to negate this. For Edwards curves +// -(x,y) = (-x,y), i.e. we need to negate the x coordinate. +// In this processed encoding, that amounts to swapping the +// first two fields and negating the third. +// +// The optional negation here also pretends bias = 0 whenever +// ix = 0 so that it doesn't need to handle the case of zero +// inputs, since no non-trivial table entries are zero. Note +// that in the zero case the whole negation is trivial, and +// so indeed is the swapping. + + cmp bias, #0 + + csel x12, x0, x4, eq + csel x13, x1, x5, eq + csel x14, x2, x6, eq + csel x15, x3, x7, eq + stp x12, x13, [tabent] + stp x14, x15, [tabent+16] + + csel x12, x0, x4, ne + csel x13, x1, x5, ne + csel x14, x2, x6, ne + csel x15, x3, x7, ne + stp x12, x13, [tabent+32] + stp x14, x15, [tabent+48] + + mov x0, #-19 + subs x0, x0, x8 + mov x2, #-1 + sbcs x1, x2, x9 + sbcs x2, x2, x10 + mov x3, #0x7FFFFFFFFFFFFFFF + sbc x3, x3, x11 + + cmp ix, xzr + ccmp bias, xzr, #4, ne + + csel x0, x0, x8, ne + csel x1, x1, x9, ne + stp x0, x1, [tabent+64] + csel x2, x2, x10, ne + csel x3, x3, x11, ne + stp x2, x3, [tabent+80] + +// Extended-projective and precomputed mixed addition. +// This is effectively the same as calling the standalone +// function edwards25519_pepadd(acc,acc,tabent), but we +// only retain slightly weaker normalization < 2 * p_25519 +// throughout the inner loop, so the computation is +// slightly different, and faster overall. + + double_twice4(t0,z_1) + sub_twice4(t1,y_1,x_1) + add_twice4(t2,y_1,x_1) + mul_4(t3,w_1,kxy_2) + mul_4(t1,t1,ymx_2) + mul_4(t2,t2,xpy_2) + sub_twice4(t4,t0,t3) + add_twice4(t0,t0,t3) + sub_twice4(t5,t2,t1) + add_twice4(t1,t2,t1) + mul_4(z_3,t4,t0) + mul_4(x_3,t5,t4) + mul_4(y_3,t0,t1) + mul_4(w_3,t5,t1) + +// End of the main loop; move on by 4 bits. + + add i, i, 4 + cmp i, 256 + bcc Lcurve25519_x25519base_alt_scalarloop + +// Now we need to translate from Edwards curve edwards25519 back +// to the Montgomery form curve25519. The mapping in the affine +// representations is +// +// (x,y) |-> ((1 + y) / (1 - y), c * (1 + y) / ((1 - y) * x)) +// +// For x25519, we only need the x coordinate, and we compute this as +// +// (1 + y) / (1 - y) = (x + x * y) / (x - x * y) +// = (X/Z + T/Z) / (X/Z - T/Z) +// = (X + T) / (X - T) +// = (X + T) * inverse(X - T) +// +// We could equally well use (Z + Y) / (Z - Y), but the above has the +// same cost, and it more explicitly forces zero output whenever X = 0, +// regardless of how the modular inverse behaves on zero inputs. In +// the present setting (base point 9, mangled scalar) that doesn't +// really matter anyway since X = 0 never arises, but it seems a +// little bit tidier. Note that both Edwards point (0,1) which maps to +// the Montgomery point at infinity, and Edwards (0,-1) which maps to +// Montgomery (0,0) [this is the 2-torsion point] are both by definition +// mapped to 0 by the X coordinate mapping used to define curve25519. +// +// First the addition and subtraction: + + add_twice4(t1,x_3,w_3) + sub_twice4(t2,x_3,w_3) + +// Prepare to call the modular inverse function to get t0 = 1/t2 +// Note that this works for the weakly normalized z_3 equally well. +// The non-coprime case z_3 == 0 (mod p_25519) cannot arise anyway. + + add x0, t0 + add x1, t2 + +// Inline copy of bignum_inv_p25519, identical except for stripping out +// the prologue and epilogue saving and restoring registers and making +// and reclaiming room on the stack. For more details and explanations see +// "arm/curve25519/bignum_inv_p25519.S". Note that the stack it uses for +// its own temporaries is 128 bytes, so it has no effect on variables +// that are needed in the rest of our computation here: res, t0, t1, t2. + + mov x20, x0 + mov x10, #0xffffffffffffffed + mov x11, #0xffffffffffffffff + stp x10, x11, [sp] + mov x12, #0x7fffffffffffffff + stp x11, x12, [sp, #16] + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + mov x7, #0x13 + lsr x6, x5, #63 + madd x6, x7, x6, x7 + adds x2, x2, x6 + adcs x3, x3, xzr + adcs x4, x4, xzr + orr x5, x5, #0x8000000000000000 + adcs x5, x5, xzr + csel x6, x7, xzr, cc + subs x2, x2, x6 + sbcs x3, x3, xzr + sbcs x4, x4, xzr + sbc x5, x5, xzr + and x5, x5, #0x7fffffffffffffff + stp x2, x3, [sp, #32] + stp x4, x5, [sp, #48] + stp xzr, xzr, [sp, #64] + stp xzr, xzr, [sp, #80] + mov x10, #0x2099 + movk x10, #0x7502, lsl #16 + movk x10, #0x9e23, lsl #32 + movk x10, #0xa0f9, lsl #48 + mov x11, #0x2595 + movk x11, #0x1d13, lsl #16 + movk x11, #0x8f3f, lsl #32 + movk x11, #0xa8c6, lsl #48 + mov x12, #0x5242 + movk x12, #0x5ac, lsl #16 + movk x12, #0x8938, lsl #32 + movk x12, #0x6c6c, lsl #48 + mov x13, #0x615 + movk x13, #0x4177, lsl #16 + movk x13, #0x8b2, lsl #32 + movk x13, #0x2765, lsl #48 + stp x10, x11, [sp, #96] + stp x12, x13, [sp, #112] + mov x21, #0xa + mov x22, #0x1 + b Lcurve25519_x25519base_alt_invmidloop +Lcurve25519_x25519base_alt_invloop: + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + and x0, x12, x16 + and x1, x13, x17 + add x19, x0, x1 + ldr x7, [sp] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #32] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x7, [sp, #8] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #40] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + adc x6, x6, x1 + extr x4, x2, x4, #59 + str x4, [sp] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + adc x4, x4, x1 + extr x5, x3, x5, #59 + str x5, [sp, #32] + ldr x7, [sp, #16] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #48] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + adc x5, x5, x1 + extr x2, x6, x2, #59 + str x2, [sp, #8] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + adc x2, x2, x1 + extr x3, x4, x3, #59 + str x3, [sp, #40] + ldr x7, [sp, #24] + eor x1, x7, x14 + asr x3, x1, #63 + and x3, x3, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #56] + eor x1, x8, x15 + asr x0, x1, #63 + and x0, x0, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x5, x6, #59 + str x6, [sp, #16] + extr x5, x3, x5, #59 + str x5, [sp, #24] + eor x1, x7, x16 + asr x5, x1, #63 + and x5, x5, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + asr x0, x1, #63 + and x0, x0, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x4, x2, x4, #59 + str x4, [sp, #48] + extr x2, x5, x2, #59 + str x2, [sp, #56] + ldr x7, [sp, #64] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #96] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #64] + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + str x5, [sp, #96] + adc x3, x3, x1 + ldr x7, [sp, #72] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #104] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #72] + adc x6, x6, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + str x3, [sp, #104] + adc x4, x4, x1 + ldr x7, [sp, #80] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #112] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #80] + adc x5, x5, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + str x4, [sp, #112] + adc x2, x2, x1 + ldr x7, [sp, #88] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #120] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x3, x5, #63 + ldp x0, x1, [sp, #64] + add x6, x6, x3, asr #63 + mov x3, #0x13 + mul x4, x6, x3 + add x5, x5, x6, lsl #63 + smulh x3, x6, x3 + ldr x6, [sp, #80] + adds x0, x0, x4 + adcs x1, x1, x3 + asr x3, x3, #63 + adcs x6, x6, x3 + adc x5, x5, x3 + stp x0, x1, [sp, #64] + stp x6, x5, [sp, #80] + eor x1, x7, x16 + and x5, x16, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + and x0, x17, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x6, x5, x2, #63 + ldp x0, x1, [sp, #96] + add x6, x6, x5, asr #63 + mov x5, #0x13 + mul x4, x6, x5 + add x2, x2, x6, lsl #63 + smulh x5, x6, x5 + ldr x3, [sp, #112] + adds x0, x0, x4 + adcs x1, x1, x5 + asr x5, x5, #63 + adcs x3, x3, x5 + adc x2, x2, x5 + stp x0, x1, [sp, #96] + stp x3, x2, [sp, #112] +Lcurve25519_x25519base_alt_invmidloop: + mov x1, x22 + ldr x2, [sp] + ldr x3, [sp, #32] + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x8, x4, #0x100, lsl #12 + sbfx x8, x8, #21, #21 + mov x11, #0x100000 + add x11, x11, x11, lsl #21 + add x9, x4, x11 + asr x9, x9, #42 + add x10, x5, #0x100, lsl #12 + sbfx x10, x10, #21, #21 + add x11, x5, x11 + asr x11, x11, #42 + mul x6, x8, x2 + mul x7, x9, x3 + mul x2, x10, x2 + mul x3, x11, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #21, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #42 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #21, #21 + add x15, x5, x15 + asr x15, x15, #42 + mul x6, x12, x2 + mul x7, x13, x3 + mul x2, x14, x2 + mul x3, x15, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + mul x2, x12, x8 + mul x3, x12, x9 + mul x6, x14, x8 + mul x7, x14, x9 + madd x8, x13, x10, x2 + madd x9, x13, x11, x3 + madd x16, x15, x10, x6 + madd x17, x15, x11, x7 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #22, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #43 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #22, #21 + add x15, x5, x15 + asr x15, x15, #43 + mneg x2, x12, x8 + mneg x3, x12, x9 + mneg x4, x14, x8 + mneg x5, x14, x9 + msub x10, x13, x16, x2 + msub x11, x13, x17, x3 + msub x12, x15, x16, x4 + msub x13, x15, x17, x5 + mov x22, x1 + subs x21, x21, #0x1 + b.ne Lcurve25519_x25519base_alt_invloop + ldr x0, [sp] + ldr x1, [sp, #32] + mul x0, x0, x10 + madd x1, x1, x11, x0 + asr x0, x1, #63 + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + eor x14, x14, x0 + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + eor x15, x15, x0 + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + eor x16, x16, x0 + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + eor x17, x17, x0 + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + ldr x7, [sp, #64] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #96] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #64] + adc x2, x2, x1 + ldr x7, [sp, #72] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #104] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #72] + adc x6, x6, x1 + ldr x7, [sp, #80] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #112] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #80] + adc x5, x5, x1 + ldr x7, [sp, #88] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #120] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x3, x5, #63 + ldp x0, x1, [sp, #64] + tst x3, x3 + cinc x6, x6, pl + mov x3, #0x13 + mul x4, x6, x3 + add x5, x5, x6, lsl #63 + smulh x6, x6, x3 + ldr x2, [sp, #80] + adds x0, x0, x4 + adcs x1, x1, x6 + asr x6, x6, #63 + adcs x2, x2, x6 + adcs x5, x5, x6 + csel x3, x3, xzr, mi + subs x0, x0, x3 + sbcs x1, x1, xzr + sbcs x2, x2, xzr + sbc x5, x5, xzr + and x5, x5, #0x7fffffffffffffff + mov x4, x20 + stp x0, x1, [x4] + stp x2, x5, [x4, #16] + +// The final result is (X + T) / (X - T) +// This is the only operation in the whole computation that +// fully reduces modulo p_25519 since now we want the canonical +// answer as output. + + mul_p25519(resx,t1,t0) + +// Restore stack and registers + + CFI_INC_SP(NSPACE) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(curve25519_x25519base_alt) + + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif + +// **************************************************************************** +// The precomputed data (all read-only). +// **************************************************************************** + +#if defined(__ELF__) +.section .rodata +.type S2N_BN_SYMBOL(curve25519_x25519base_alt_constant), %object +.size S2N_BN_SYMBOL(curve25519_x25519base_alt_constant), 48576 +#elif defined(__APPLE__) +.const_data +#endif + +S2N_BN_SYMBOL(curve25519_x25519base_alt_constant): + +// 2^254 * G and (2^254 + 8) * G in extended-projective coordinates +// but with Z = 1 assumed and hence left out, so they are (X,Y,T) only. + + .quad 0x251037f7cf4e861d + .quad 0x10ede0fb19fb128f + .quad 0x96c033b175f5e2c8 + .quad 0x055f070d6c15fb0d + .quad 0x7c52af2c97473e69 + .quad 0x022f82391bad8378 + .quad 0x9991e1b02adb476f + .quad 0x511144a03a99b855 + .quad 0x5fafc3b88ff2e4ae + .quad 0x855e4ff0de1230ff + .quad 0x72e302a348492870 + .quad 0x1253c19e53dbe1bc + + .quad 0x331d086e0d9abcaa + .quad 0x1e23c96d311a10c9 + .quad 0x96d0f95e58c13478 + .quad 0x2f72f7384fcfcc59 + .quad 0x39a6cd1cfd7d87c9 + .quad 0x9867a0abd8ae153a + .quad 0xa49d2a5f35986745 + .quad 0x57012940cdfe82e1 + .quad 0x5046a6532ec5544a + .quad 0x6d674004739ff6c9 + .quad 0x9bbaa44b234a70e3 + .quad 0x5e6d8901138cf386 + +// Precomputed table of multiples of generator for edwards25519 +// all in precomputed extended-projective (y-x,x+y,2*d*x*y) triples. + + // 2^4 * 1 * G + + .quad 0x7ec851ca553e2df3 + .quad 0xa71284cba64878b3 + .quad 0xe6b5e4193288d1e7 + .quad 0x4cf210ec5a9a8883 + .quad 0x322d04a52d9021f6 + .quad 0xb9c19f3375c6bf9c + .quad 0x587a3a4342d20b09 + .quad 0x143b1cf8aa64fe61 + .quad 0x9f867c7d968acaab + .quad 0x5f54258e27092729 + .quad 0xd0a7d34bea180975 + .quad 0x21b546a3374126e1 + + // 2^4 * 2 * G + + .quad 0xa94ff858a2888343 + .quad 0xce0ed4565313ed3c + .quad 0xf55c3dcfb5bf34fa + .quad 0x0a653ca5c9eab371 + .quad 0x490a7a45d185218f + .quad 0x9a15377846049335 + .quad 0x0060ea09cc31e1f6 + .quad 0x7e041577f86ee965 + .quad 0x66b2a496ce5b67f3 + .quad 0xff5492d8bd569796 + .quad 0x503cec294a592cd0 + .quad 0x566943650813acb2 + + // 2^4 * 3 * G + + .quad 0xb818db0c26620798 + .quad 0x5d5c31d9606e354a + .quad 0x0982fa4f00a8cdc7 + .quad 0x17e12bcd4653e2d4 + .quad 0x5672f9eb1dabb69d + .quad 0xba70b535afe853fc + .quad 0x47ac0f752796d66d + .quad 0x32a5351794117275 + .quad 0xd3a644a6df648437 + .quad 0x703b6559880fbfdd + .quad 0xcb852540ad3a1aa5 + .quad 0x0900b3f78e4c6468 + + // 2^4 * 4 * G + + .quad 0x0a851b9f679d651b + .quad 0xe108cb61033342f2 + .quad 0xd601f57fe88b30a3 + .quad 0x371f3acaed2dd714 + .quad 0xed280fbec816ad31 + .quad 0x52d9595bd8e6efe3 + .quad 0x0fe71772f6c623f5 + .quad 0x4314030b051e293c + .quad 0xd560005efbf0bcad + .quad 0x8eb70f2ed1870c5e + .quad 0x201f9033d084e6a0 + .quad 0x4c3a5ae1ce7b6670 + + // 2^4 * 5 * G + + .quad 0x4138a434dcb8fa95 + .quad 0x870cf67d6c96840b + .quad 0xde388574297be82c + .quad 0x7c814db27262a55a + .quad 0xbaf875e4c93da0dd + .quad 0xb93282a771b9294d + .quad 0x80d63fb7f4c6c460 + .quad 0x6de9c73dea66c181 + .quad 0x478904d5a04df8f2 + .quad 0xfafbae4ab10142d3 + .quad 0xf6c8ac63555d0998 + .quad 0x5aac4a412f90b104 + + // 2^4 * 6 * G + + .quad 0xc64f326b3ac92908 + .quad 0x5551b282e663e1e0 + .quad 0x476b35f54a1a4b83 + .quad 0x1b9da3fe189f68c2 + .quad 0x603a0d0abd7f5134 + .quad 0x8089c932e1d3ae46 + .quad 0xdf2591398798bd63 + .quad 0x1c145cd274ba0235 + .quad 0x32e8386475f3d743 + .quad 0x365b8baf6ae5d9ef + .quad 0x825238b6385b681e + .quad 0x234929c1167d65e1 + + // 2^4 * 7 * G + + .quad 0x984decaba077ade8 + .quad 0x383f77ad19eb389d + .quad 0xc7ec6b7e2954d794 + .quad 0x59c77b3aeb7c3a7a + .quad 0x48145cc21d099fcf + .quad 0x4535c192cc28d7e5 + .quad 0x80e7c1e548247e01 + .quad 0x4a5f28743b2973ee + .quad 0xd3add725225ccf62 + .quad 0x911a3381b2152c5d + .quad 0xd8b39fad5b08f87d + .quad 0x6f05606b4799fe3b + + // 2^4 * 8 * G + + .quad 0x9ffe9e92177ba962 + .quad 0x98aee71d0de5cae1 + .quad 0x3ff4ae942d831044 + .quad 0x714de12e58533ac8 + .quad 0x5b433149f91b6483 + .quad 0xadb5dc655a2cbf62 + .quad 0x87fa8412632827b3 + .quad 0x60895e91ab49f8d8 + .quad 0xe9ecf2ed0cf86c18 + .quad 0xb46d06120735dfd4 + .quad 0xbc9da09804b96be7 + .quad 0x73e2e62fd96dc26b + + // 2^8 * 1 * G + + .quad 0xed5b635449aa515e + .quad 0xa865c49f0bc6823a + .quad 0x850c1fe95b42d1c4 + .quad 0x30d76d6f03d315b9 + .quad 0x2eccdd0e632f9c1d + .quad 0x51d0b69676893115 + .quad 0x52dfb76ba8637a58 + .quad 0x6dd37d49a00eef39 + .quad 0x6c4444172106e4c7 + .quad 0xfb53d680928d7f69 + .quad 0xb4739ea4694d3f26 + .quad 0x10c697112e864bb0 + + // 2^8 * 2 * G + + .quad 0x6493c4277dbe5fde + .quad 0x265d4fad19ad7ea2 + .quad 0x0e00dfc846304590 + .quad 0x25e61cabed66fe09 + .quad 0x0ca62aa08358c805 + .quad 0x6a3d4ae37a204247 + .quad 0x7464d3a63b11eddc + .quad 0x03bf9baf550806ef + .quad 0x3f13e128cc586604 + .quad 0x6f5873ecb459747e + .quad 0xa0b63dedcc1268f5 + .quad 0x566d78634586e22c + + // 2^8 * 3 * G + + .quad 0x1637a49f9cc10834 + .quad 0xbc8e56d5a89bc451 + .quad 0x1cb5ec0f7f7fd2db + .quad 0x33975bca5ecc35d9 + .quad 0xa1054285c65a2fd0 + .quad 0x6c64112af31667c3 + .quad 0x680ae240731aee58 + .quad 0x14fba5f34793b22a + .quad 0x3cd746166985f7d4 + .quad 0x593e5e84c9c80057 + .quad 0x2fc3f2b67b61131e + .quad 0x14829cea83fc526c + + // 2^8 * 4 * G + + .quad 0xff437b8497dd95c2 + .quad 0x6c744e30aa4eb5a7 + .quad 0x9e0c5d613c85e88b + .quad 0x2fd9c71e5f758173 + .quad 0x21e70b2f4e71ecb8 + .quad 0xe656ddb940a477e3 + .quad 0xbf6556cece1d4f80 + .quad 0x05fc3bc4535d7b7e + .quad 0x24b8b3ae52afdedd + .quad 0x3495638ced3b30cf + .quad 0x33a4bc83a9be8195 + .quad 0x373767475c651f04 + + // 2^8 * 5 * G + + .quad 0x2fba99fd40d1add9 + .quad 0xb307166f96f4d027 + .quad 0x4363f05215f03bae + .quad 0x1fbea56c3b18f999 + .quad 0x634095cb14246590 + .quad 0xef12144016c15535 + .quad 0x9e38140c8910bc60 + .quad 0x6bf5905730907c8c + .quad 0x0fa778f1e1415b8a + .quad 0x06409ff7bac3a77e + .quad 0x6f52d7b89aa29a50 + .quad 0x02521cf67a635a56 + + // 2^8 * 6 * G + + .quad 0x513fee0b0a9d5294 + .quad 0x8f98e75c0fdf5a66 + .quad 0xd4618688bfe107ce + .quad 0x3fa00a7e71382ced + .quad 0xb1146720772f5ee4 + .quad 0xe8f894b196079ace + .quad 0x4af8224d00ac824a + .quad 0x001753d9f7cd6cc4 + .quad 0x3c69232d963ddb34 + .quad 0x1dde87dab4973858 + .quad 0xaad7d1f9a091f285 + .quad 0x12b5fe2fa048edb6 + + // 2^8 * 7 * G + + .quad 0x71f0fbc496fce34d + .quad 0x73b9826badf35bed + .quad 0xd2047261ff28c561 + .quad 0x749b76f96fb1206f + .quad 0xdf2b7c26ad6f1e92 + .quad 0x4b66d323504b8913 + .quad 0x8c409dc0751c8bc3 + .quad 0x6f7e93c20796c7b8 + .quad 0x1f5af604aea6ae05 + .quad 0xc12351f1bee49c99 + .quad 0x61a808b5eeff6b66 + .quad 0x0fcec10f01e02151 + + // 2^8 * 8 * G + + .quad 0x644d58a649fe1e44 + .quad 0x21fcaea231ad777e + .quad 0x02441c5a887fd0d2 + .quad 0x4901aa7183c511f3 + .quad 0x3df2d29dc4244e45 + .quad 0x2b020e7493d8de0a + .quad 0x6cc8067e820c214d + .quad 0x413779166feab90a + .quad 0x08b1b7548c1af8f0 + .quad 0xce0f7a7c246299b4 + .quad 0xf760b0f91e06d939 + .quad 0x41bb887b726d1213 + + // 2^12 * 1 * G + + .quad 0x9267806c567c49d8 + .quad 0x066d04ccca791e6a + .quad 0xa69f5645e3cc394b + .quad 0x5c95b686a0788cd2 + .quad 0x97d980e0aa39f7d2 + .quad 0x35d0384252c6b51c + .quad 0x7d43f49307cd55aa + .quad 0x56bd36cfb78ac362 + .quad 0x2ac519c10d14a954 + .quad 0xeaf474b494b5fa90 + .quad 0xe6af8382a9f87a5a + .quad 0x0dea6db1879be094 + + // 2^12 * 2 * G + + .quad 0xaa66bf547344e5ab + .quad 0xda1258888f1b4309 + .quad 0x5e87d2b3fd564b2f + .quad 0x5b2c78885483b1dd + .quad 0x15baeb74d6a8797a + .quad 0x7ef55cf1fac41732 + .quad 0x29001f5a3c8b05c5 + .quad 0x0ad7cc8752eaccfb + .quad 0x52151362793408cf + .quad 0xeb0f170319963d94 + .quad 0xa833b2fa883d9466 + .quad 0x093a7fa775003c78 + + // 2^12 * 3 * G + + .quad 0xe5107de63a16d7be + .quad 0xa377ffdc9af332cf + .quad 0x70d5bf18440b677f + .quad 0x6a252b19a4a31403 + .quad 0xb8e9604460a91286 + .quad 0x7f3fd8047778d3de + .quad 0x67d01e31bf8a5e2d + .quad 0x7b038a06c27b653e + .quad 0x9ed919d5d36990f3 + .quad 0x5213aebbdb4eb9f2 + .quad 0xc708ea054cb99135 + .quad 0x58ded57f72260e56 + + // 2^12 * 4 * G + + .quad 0x78e79dade9413d77 + .quad 0xf257f9d59729e67d + .quad 0x59db910ee37aa7e6 + .quad 0x6aa11b5bbb9e039c + .quad 0xda6d53265b0fd48b + .quad 0x8960823193bfa988 + .quad 0xd78ac93261d57e28 + .quad 0x79f2942d3a5c8143 + .quad 0x97da2f25b6c88de9 + .quad 0x251ba7eaacf20169 + .quad 0x09b44f87ef4eb4e4 + .quad 0x7d90ab1bbc6a7da5 + + // 2^12 * 5 * G + + .quad 0x9acca683a7016bfe + .quad 0x90505f4df2c50b6d + .quad 0x6b610d5fcce435aa + .quad 0x19a10d446198ff96 + .quad 0x1a07a3f496b3c397 + .quad 0x11ceaa188f4e2532 + .quad 0x7d9498d5a7751bf0 + .quad 0x19ed161f508dd8a0 + .quad 0x560a2cd687dce6ca + .quad 0x7f3568c48664cf4d + .quad 0x8741e95222803a38 + .quad 0x483bdab1595653fc + + // 2^12 * 6 * G + + .quad 0xfa780f148734fa49 + .quad 0x106f0b70360534e0 + .quad 0x2210776fe3e307bd + .quad 0x3286c109dde6a0fe + .quad 0xd6cf4d0ab4da80f6 + .quad 0x82483e45f8307fe0 + .quad 0x05005269ae6f9da4 + .quad 0x1c7052909cf7877a + .quad 0x32ee7de2874e98d4 + .quad 0x14c362e9b97e0c60 + .quad 0x5781dcde6a60a38a + .quad 0x217dd5eaaa7aa840 + + // 2^12 * 7 * G + + .quad 0x9db7c4d0248e1eb0 + .quad 0xe07697e14d74bf52 + .quad 0x1e6a9b173c562354 + .quad 0x7fa7c21f795a4965 + .quad 0x8bdf1fb9be8c0ec8 + .quad 0x00bae7f8e30a0282 + .quad 0x4963991dad6c4f6c + .quad 0x07058a6e5df6f60a + .quad 0xe9eb02c4db31f67f + .quad 0xed25fd8910bcfb2b + .quad 0x46c8131f5c5cddb4 + .quad 0x33b21c13a0cb9bce + + // 2^12 * 8 * G + + .quad 0x360692f8087d8e31 + .quad 0xf4dcc637d27163f7 + .quad 0x25a4e62065ea5963 + .quad 0x659bf72e5ac160d9 + .quad 0x9aafb9b05ee38c5b + .quad 0xbf9d2d4e071a13c7 + .quad 0x8eee6e6de933290a + .quad 0x1c3bab17ae109717 + .quad 0x1c9ab216c7cab7b0 + .quad 0x7d65d37407bbc3cc + .quad 0x52744750504a58d5 + .quad 0x09f2606b131a2990 + + // 2^16 * 1 * G + + .quad 0x40e87d44744346be + .quad 0x1d48dad415b52b25 + .quad 0x7c3a8a18a13b603e + .quad 0x4eb728c12fcdbdf7 + .quad 0x7e234c597c6691ae + .quad 0x64889d3d0a85b4c8 + .quad 0xdae2c90c354afae7 + .quad 0x0a871e070c6a9e1d + .quad 0x3301b5994bbc8989 + .quad 0x736bae3a5bdd4260 + .quad 0x0d61ade219d59e3c + .quad 0x3ee7300f2685d464 + + // 2^16 * 2 * G + + .quad 0xf5d255e49e7dd6b7 + .quad 0x8016115c610b1eac + .quad 0x3c99975d92e187ca + .quad 0x13815762979125c2 + .quad 0x43fa7947841e7518 + .quad 0xe5c6fa59639c46d7 + .quad 0xa1065e1de3052b74 + .quad 0x7d47c6a2cfb89030 + .quad 0x3fdad0148ef0d6e0 + .quad 0x9d3e749a91546f3c + .quad 0x71ec621026bb8157 + .quad 0x148cf58d34c9ec80 + + // 2^16 * 3 * G + + .quad 0x46a492f67934f027 + .quad 0x469984bef6840aa9 + .quad 0x5ca1bc2a89611854 + .quad 0x3ff2fa1ebd5dbbd4 + .quad 0xe2572f7d9ae4756d + .quad 0x56c345bb88f3487f + .quad 0x9fd10b6d6960a88d + .quad 0x278febad4eaea1b9 + .quad 0xb1aa681f8c933966 + .quad 0x8c21949c20290c98 + .quad 0x39115291219d3c52 + .quad 0x4104dd02fe9c677b + + // 2^16 * 4 * G + + .quad 0x72b2bf5e1124422a + .quad 0xa1fa0c3398a33ab5 + .quad 0x94cb6101fa52b666 + .quad 0x2c863b00afaf53d5 + .quad 0x81214e06db096ab8 + .quad 0x21a8b6c90ce44f35 + .quad 0x6524c12a409e2af5 + .quad 0x0165b5a48efca481 + .quad 0xf190a474a0846a76 + .quad 0x12eff984cd2f7cc0 + .quad 0x695e290658aa2b8f + .quad 0x591b67d9bffec8b8 + + // 2^16 * 5 * G + + .quad 0x312f0d1c80b49bfa + .quad 0x5979515eabf3ec8a + .quad 0x727033c09ef01c88 + .quad 0x3de02ec7ca8f7bcb + .quad 0x99b9b3719f18b55d + .quad 0xe465e5faa18c641e + .quad 0x61081136c29f05ed + .quad 0x489b4f867030128b + .quad 0xd232102d3aeb92ef + .quad 0xe16253b46116a861 + .quad 0x3d7eabe7190baa24 + .quad 0x49f5fbba496cbebf + + // 2^16 * 6 * G + + .quad 0x30949a108a5bcfd4 + .quad 0xdc40dd70bc6473eb + .quad 0x92c294c1307c0d1c + .quad 0x5604a86dcbfa6e74 + .quad 0x155d628c1e9c572e + .quad 0x8a4d86acc5884741 + .quad 0x91a352f6515763eb + .quad 0x06a1a6c28867515b + .quad 0x7288d1d47c1764b6 + .quad 0x72541140e0418b51 + .quad 0x9f031a6018acf6d1 + .quad 0x20989e89fe2742c6 + + // 2^16 * 7 * G + + .quad 0x499777fd3a2dcc7f + .quad 0x32857c2ca54fd892 + .quad 0xa279d864d207e3a0 + .quad 0x0403ed1d0ca67e29 + .quad 0x1674278b85eaec2e + .quad 0x5621dc077acb2bdf + .quad 0x640a4c1661cbf45a + .quad 0x730b9950f70595d3 + .quad 0xc94b2d35874ec552 + .quad 0xc5e6c8cf98246f8d + .quad 0xf7cb46fa16c035ce + .quad 0x5bd7454308303dcc + + // 2^16 * 8 * G + + .quad 0x7f9ad19528b24cc2 + .quad 0x7f6b54656335c181 + .quad 0x66b8b66e4fc07236 + .quad 0x133a78007380ad83 + .quad 0x85c4932115e7792a + .quad 0xc64c89a2bdcdddc9 + .quad 0x9d1e3da8ada3d762 + .quad 0x5bb7db123067f82c + .quad 0x0961f467c6ca62be + .quad 0x04ec21d6211952ee + .quad 0x182360779bd54770 + .quad 0x740dca6d58f0e0d2 + + // 2^20 * 1 * G + + .quad 0x50b70bf5d3f0af0b + .quad 0x4feaf48ae32e71f7 + .quad 0x60e84ed3a55bbd34 + .quad 0x00ed489b3f50d1ed + .quad 0x3906c72aed261ae5 + .quad 0x9ab68fd988e100f7 + .quad 0xf5e9059af3360197 + .quad 0x0e53dc78bf2b6d47 + .quad 0xb90829bf7971877a + .quad 0x5e4444636d17e631 + .quad 0x4d05c52e18276893 + .quad 0x27632d9a5a4a4af5 + + // 2^20 * 2 * G + + .quad 0xd11ff05154b260ce + .quad 0xd86dc38e72f95270 + .quad 0x601fcd0d267cc138 + .quad 0x2b67916429e90ccd + .quad 0xa98285d187eaffdb + .quad 0xa5b4fbbbd8d0a864 + .quad 0xb658f27f022663f7 + .quad 0x3bbc2b22d99ce282 + .quad 0xb917c952583c0a58 + .quad 0x653ff9b80fe4c6f3 + .quad 0x9b0da7d7bcdf3c0c + .quad 0x43a0eeb6ab54d60e + + // 2^20 * 3 * G + + .quad 0x396966a46d4a5487 + .quad 0xf811a18aac2bb3ba + .quad 0x66e4685b5628b26b + .quad 0x70a477029d929b92 + .quad 0x3ac6322357875fe8 + .quad 0xd9d4f4ecf5fbcb8f + .quad 0x8dee8493382bb620 + .quad 0x50c5eaa14c799fdc + .quad 0xdd0edc8bd6f2fb3c + .quad 0x54c63aa79cc7b7a0 + .quad 0xae0b032b2c8d9f1a + .quad 0x6f9ce107602967fb + + // 2^20 * 4 * G + + .quad 0xad1054b1cde1c22a + .quad 0xc4a8e90248eb32df + .quad 0x5f3e7b33accdc0ea + .quad 0x72364713fc79963e + .quad 0x139693063520e0b5 + .quad 0x437fcf7c88ea03fe + .quad 0xf7d4c40bd3c959bc + .quad 0x699154d1f893ded9 + .quad 0x315d5c75b4b27526 + .quad 0xcccb842d0236daa5 + .quad 0x22f0c8a3345fee8e + .quad 0x73975a617d39dbed + + // 2^20 * 5 * G + + .quad 0xe4024df96375da10 + .quad 0x78d3251a1830c870 + .quad 0x902b1948658cd91c + .quad 0x7e18b10b29b7438a + .quad 0x6f37f392f4433e46 + .quad 0x0e19b9a11f566b18 + .quad 0x220fb78a1fd1d662 + .quad 0x362a4258a381c94d + .quad 0x9071d9132b6beb2f + .quad 0x0f26e9ad28418247 + .quad 0xeab91ec9bdec925d + .quad 0x4be65bc8f48af2de + + // 2^20 * 6 * G + + .quad 0x78487feba36e7028 + .quad 0x5f3f13001dd8ce34 + .quad 0x934fb12d4b30c489 + .quad 0x056c244d397f0a2b + .quad 0x1d50fba257c26234 + .quad 0x7bd4823adeb0678b + .quad 0xc2b0dc6ea6538af5 + .quad 0x5665eec6351da73e + .quad 0xdb3ee00943bfb210 + .quad 0x4972018720800ac2 + .quad 0x26ab5d6173bd8667 + .quad 0x20b209c2ab204938 + + // 2^20 * 7 * G + + .quad 0x549e342ac07fb34b + .quad 0x02d8220821373d93 + .quad 0xbc262d70acd1f567 + .quad 0x7a92c9fdfbcac784 + .quad 0x1fcca94516bd3289 + .quad 0x448d65aa41420428 + .quad 0x59c3b7b216a55d62 + .quad 0x49992cc64e612cd8 + .quad 0x65bd1bea70f801de + .quad 0x1befb7c0fe49e28a + .quad 0xa86306cdb1b2ae4a + .quad 0x3b7ac0cd265c2a09 + + // 2^20 * 8 * G + + .quad 0x822bee438c01bcec + .quad 0x530cb525c0fbc73b + .quad 0x48519034c1953fe9 + .quad 0x265cc261e09a0f5b + .quad 0xf0d54e4f22ed39a7 + .quad 0xa2aae91e5608150a + .quad 0xf421b2e9eddae875 + .quad 0x31bc531d6b7de992 + .quad 0xdf3d134da980f971 + .quad 0x7a4fb8d1221a22a7 + .quad 0x3df7d42035aad6d8 + .quad 0x2a14edcc6a1a125e + + // 2^24 * 1 * G + + .quad 0xdf48ee0752cfce4e + .quad 0xc3fffaf306ec08b7 + .quad 0x05710b2ab95459c4 + .quad 0x161d25fa963ea38d + .quad 0x231a8c570478433c + .quad 0xb7b5270ec281439d + .quad 0xdbaa99eae3d9079f + .quad 0x2c03f5256c2b03d9 + .quad 0x790f18757b53a47d + .quad 0x307b0130cf0c5879 + .quad 0x31903d77257ef7f9 + .quad 0x699468bdbd96bbaf + + // 2^24 * 2 * G + + .quad 0xbd1f2f46f4dafecf + .quad 0x7cef0114a47fd6f7 + .quad 0xd31ffdda4a47b37f + .quad 0x525219a473905785 + .quad 0xd8dd3de66aa91948 + .quad 0x485064c22fc0d2cc + .quad 0x9b48246634fdea2f + .quad 0x293e1c4e6c4a2e3a + .quad 0x376e134b925112e1 + .quad 0x703778b5dca15da0 + .quad 0xb04589af461c3111 + .quad 0x5b605c447f032823 + + // 2^24 * 3 * G + + .quad 0xb965805920c47c89 + .quad 0xe7f0100c923b8fcc + .quad 0x0001256502e2ef77 + .quad 0x24a76dcea8aeb3ee + .quad 0x3be9fec6f0e7f04c + .quad 0x866a579e75e34962 + .quad 0x5542ef161e1de61a + .quad 0x2f12fef4cc5abdd5 + .quad 0x0a4522b2dfc0c740 + .quad 0x10d06e7f40c9a407 + .quad 0xc6cf144178cff668 + .quad 0x5e607b2518a43790 + + // 2^24 * 4 * G + + .quad 0x58b31d8f6cdf1818 + .quad 0x35cfa74fc36258a2 + .quad 0xe1b3ff4f66e61d6e + .quad 0x5067acab6ccdd5f7 + .quad 0xa02c431ca596cf14 + .quad 0xe3c42d40aed3e400 + .quad 0xd24526802e0f26db + .quad 0x201f33139e457068 + .quad 0xfd527f6b08039d51 + .quad 0x18b14964017c0006 + .quad 0xd5220eb02e25a4a8 + .quad 0x397cba8862460375 + + // 2^24 * 5 * G + + .quad 0x30c13093f05959b2 + .quad 0xe23aa18de9a97976 + .quad 0x222fd491721d5e26 + .quad 0x2339d320766e6c3a + .quad 0x7815c3fbc81379e7 + .quad 0xa6619420dde12af1 + .quad 0xffa9c0f885a8fdd5 + .quad 0x771b4022c1e1c252 + .quad 0xd87dd986513a2fa7 + .quad 0xf5ac9b71f9d4cf08 + .quad 0xd06bc31b1ea283b3 + .quad 0x331a189219971a76 + + // 2^24 * 6 * G + + .quad 0xf5166f45fb4f80c6 + .quad 0x9c36c7de61c775cf + .quad 0xe3d4e81b9041d91c + .quad 0x31167c6b83bdfe21 + .quad 0x26512f3a9d7572af + .quad 0x5bcbe28868074a9e + .quad 0x84edc1c11180f7c4 + .quad 0x1ac9619ff649a67b + .quad 0xf22b3842524b1068 + .quad 0x5068343bee9ce987 + .quad 0xfc9d71844a6250c8 + .quad 0x612436341f08b111 + + // 2^24 * 7 * G + + .quad 0xd99d41db874e898d + .quad 0x09fea5f16c07dc20 + .quad 0x793d2c67d00f9bbc + .quad 0x46ebe2309e5eff40 + .quad 0x8b6349e31a2d2638 + .quad 0x9ddfb7009bd3fd35 + .quad 0x7f8bf1b8a3a06ba4 + .quad 0x1522aa3178d90445 + .quad 0x2c382f5369614938 + .quad 0xdafe409ab72d6d10 + .quad 0xe8c83391b646f227 + .quad 0x45fe70f50524306c + + // 2^24 * 8 * G + + .quad 0xda4875a6960c0b8c + .quad 0x5b68d076ef0e2f20 + .quad 0x07fb51cf3d0b8fd4 + .quad 0x428d1623a0e392d4 + .quad 0x62f24920c8951491 + .quad 0x05f007c83f630ca2 + .quad 0x6fbb45d2f5c9d4b8 + .quad 0x16619f6db57a2245 + .quad 0x084f4a4401a308fd + .quad 0xa82219c376a5caac + .quad 0xdeb8de4643d1bc7d + .quad 0x1d81592d60bd38c6 + + // 2^28 * 1 * G + + .quad 0xd833d7beec2a4c38 + .quad 0x2c9162830acc20ed + .quad 0xe93a47aa92df7581 + .quad 0x702d67a3333c4a81 + .quad 0x3a4a369a2f89c8a1 + .quad 0x63137a1d7c8de80d + .quad 0xbcac008a78eda015 + .quad 0x2cb8b3a5b483b03f + .quad 0x36e417cbcb1b90a1 + .quad 0x33b3ddaa7f11794e + .quad 0x3f510808885bc607 + .quad 0x24141dc0e6a8020d + + // 2^28 * 2 * G + + .quad 0x59f73c773fefee9d + .quad 0xb3f1ef89c1cf989d + .quad 0xe35dfb42e02e545f + .quad 0x5766120b47a1b47c + .quad 0x91925dccbd83157d + .quad 0x3ca1205322cc8094 + .quad 0x28e57f183f90d6e4 + .quad 0x1a4714cede2e767b + .quad 0xdb20ba0fb8b6b7ff + .quad 0xb732c3b677511fa1 + .quad 0xa92b51c099f02d89 + .quad 0x4f3875ad489ca5f1 + + // 2^28 * 3 * G + + .quad 0xc7fc762f4932ab22 + .quad 0x7ac0edf72f4c3c1b + .quad 0x5f6b55aa9aa895e8 + .quad 0x3680274dad0a0081 + .quad 0x79ed13f6ee73eec0 + .quad 0xa5c6526d69110bb1 + .quad 0xe48928c38603860c + .quad 0x722a1446fd7059f5 + .quad 0xd0959fe9a8cf8819 + .quad 0xd0a995508475a99c + .quad 0x6eac173320b09cc5 + .quad 0x628ecf04331b1095 + + // 2^28 * 4 * G + + .quad 0x98bcb118a9d0ddbc + .quad 0xee449e3408b4802b + .quad 0x87089226b8a6b104 + .quad 0x685f349a45c7915d + .quad 0x9b41acf85c74ccf1 + .quad 0xb673318108265251 + .quad 0x99c92aed11adb147 + .quad 0x7a47d70d34ecb40f + .quad 0x60a0c4cbcc43a4f5 + .quad 0x775c66ca3677bea9 + .quad 0xa17aa1752ff8f5ed + .quad 0x11ded9020e01fdc0 + + // 2^28 * 5 * G + + .quad 0x890e7809caefe704 + .quad 0x8728296de30e8c6c + .quad 0x4c5cd2a392aeb1c9 + .quad 0x194263d15771531f + .quad 0x471f95b03bea93b7 + .quad 0x0552d7d43313abd3 + .quad 0xbd9370e2e17e3f7b + .quad 0x7b120f1db20e5bec + .quad 0x17d2fb3d86502d7a + .quad 0xb564d84450a69352 + .quad 0x7da962c8a60ed75d + .quad 0x00d0f85b318736aa + + // 2^28 * 6 * G + + .quad 0x978b142e777c84fd + .quad 0xf402644705a8c062 + .quad 0xa67ad51be7e612c7 + .quad 0x2f7b459698dd6a33 + .quad 0xa6753c1efd7621c1 + .quad 0x69c0b4a7445671f5 + .quad 0x971f527405b23c11 + .quad 0x387bc74851a8c7cd + .quad 0x81894b4d4a52a9a8 + .quad 0xadd93e12f6b8832f + .quad 0x184d8548b61bd638 + .quad 0x3f1c62dbd6c9f6cd + + // 2^28 * 7 * G + + .quad 0x2e8f1f0091910c1f + .quad 0xa4df4fe0bff2e12c + .quad 0x60c6560aee927438 + .quad 0x6338283facefc8fa + .quad 0x3fad3e40148f693d + .quad 0x052656e194eb9a72 + .quad 0x2f4dcbfd184f4e2f + .quad 0x406f8db1c482e18b + .quad 0x9e630d2c7f191ee4 + .quad 0x4fbf8301bc3ff670 + .quad 0x787d8e4e7afb73c4 + .quad 0x50d83d5be8f58fa5 + + // 2^28 * 8 * G + + .quad 0x85683916c11a1897 + .quad 0x2d69a4efe506d008 + .quad 0x39af1378f664bd01 + .quad 0x65942131361517c6 + .quad 0xc0accf90b4d3b66d + .quad 0xa7059de561732e60 + .quad 0x033d1f7870c6b0ba + .quad 0x584161cd26d946e4 + .quad 0xbbf2b1a072d27ca2 + .quad 0xbf393c59fbdec704 + .quad 0xe98dbbcee262b81e + .quad 0x02eebd0b3029b589 + + // 2^32 * 1 * G + + .quad 0x61368756a60dac5f + .quad 0x17e02f6aebabdc57 + .quad 0x7f193f2d4cce0f7d + .quad 0x20234a7789ecdcf0 + .quad 0x8765b69f7b85c5e8 + .quad 0x6ff0678bd168bab2 + .quad 0x3a70e77c1d330f9b + .quad 0x3a5f6d51b0af8e7c + .quad 0x76d20db67178b252 + .quad 0x071c34f9d51ed160 + .quad 0xf62a4a20b3e41170 + .quad 0x7cd682353cffe366 + + // 2^32 * 2 * G + + .quad 0x0be1a45bd887fab6 + .quad 0x2a846a32ba403b6e + .quad 0xd9921012e96e6000 + .quad 0x2838c8863bdc0943 + .quad 0xa665cd6068acf4f3 + .quad 0x42d92d183cd7e3d3 + .quad 0x5759389d336025d9 + .quad 0x3ef0253b2b2cd8ff + .quad 0xd16bb0cf4a465030 + .quad 0xfa496b4115c577ab + .quad 0x82cfae8af4ab419d + .quad 0x21dcb8a606a82812 + + // 2^32 * 3 * G + + .quad 0x5c6004468c9d9fc8 + .quad 0x2540096ed42aa3cb + .quad 0x125b4d4c12ee2f9c + .quad 0x0bc3d08194a31dab + .quad 0x9a8d00fabe7731ba + .quad 0x8203607e629e1889 + .quad 0xb2cc023743f3d97f + .quad 0x5d840dbf6c6f678b + .quad 0x706e380d309fe18b + .quad 0x6eb02da6b9e165c7 + .quad 0x57bbba997dae20ab + .quad 0x3a4276232ac196dd + + // 2^32 * 4 * G + + .quad 0x4b42432c8a7084fa + .quad 0x898a19e3dfb9e545 + .quad 0xbe9f00219c58e45d + .quad 0x1ff177cea16debd1 + .quad 0x3bf8c172db447ecb + .quad 0x5fcfc41fc6282dbd + .quad 0x80acffc075aa15fe + .quad 0x0770c9e824e1a9f9 + .quad 0xcf61d99a45b5b5fd + .quad 0x860984e91b3a7924 + .quad 0xe7300919303e3e89 + .quad 0x39f264fd41500b1e + + // 2^32 * 5 * G + + .quad 0xa7ad3417dbe7e29c + .quad 0xbd94376a2b9c139c + .quad 0xa0e91b8e93597ba9 + .quad 0x1712d73468889840 + .quad 0xd19b4aabfe097be1 + .quad 0xa46dfce1dfe01929 + .quad 0xc3c908942ca6f1ff + .quad 0x65c621272c35f14e + .quad 0xe72b89f8ce3193dd + .quad 0x4d103356a125c0bb + .quad 0x0419a93d2e1cfe83 + .quad 0x22f9800ab19ce272 + + // 2^32 * 6 * G + + .quad 0x605a368a3e9ef8cb + .quad 0xe3e9c022a5504715 + .quad 0x553d48b05f24248f + .quad 0x13f416cd647626e5 + .quad 0x42029fdd9a6efdac + .quad 0xb912cebe34a54941 + .quad 0x640f64b987bdf37b + .quad 0x4171a4d38598cab4 + .quad 0xfa2758aa99c94c8c + .quad 0x23006f6fb000b807 + .quad 0xfbd291ddadda5392 + .quad 0x508214fa574bd1ab + + // 2^32 * 7 * G + + .quad 0xc20269153ed6fe4b + .quad 0xa65a6739511d77c4 + .quad 0xcbde26462c14af94 + .quad 0x22f960ec6faba74b + .quad 0x461a15bb53d003d6 + .quad 0xb2102888bcf3c965 + .quad 0x27c576756c683a5a + .quad 0x3a7758a4c86cb447 + .quad 0x548111f693ae5076 + .quad 0x1dae21df1dfd54a6 + .quad 0x12248c90f3115e65 + .quad 0x5d9fd15f8de7f494 + + // 2^32 * 8 * G + + .quad 0x031408d36d63727f + .quad 0x6a379aefd7c7b533 + .quad 0xa9e18fc5ccaee24b + .quad 0x332f35914f8fbed3 + .quad 0x3f244d2aeed7521e + .quad 0x8e3a9028432e9615 + .quad 0xe164ba772e9c16d4 + .quad 0x3bc187fa47eb98d8 + .quad 0x6d470115ea86c20c + .quad 0x998ab7cb6c46d125 + .quad 0xd77832b53a660188 + .quad 0x450d81ce906fba03 + + // 2^36 * 1 * G + + .quad 0xf8ae4d2ad8453902 + .quad 0x7018058ee8db2d1d + .quad 0xaab3995fc7d2c11e + .quad 0x53b16d2324ccca79 + .quad 0x23264d66b2cae0b5 + .quad 0x7dbaed33ebca6576 + .quad 0x030ebed6f0d24ac8 + .quad 0x2a887f78f7635510 + .quad 0x2a23b9e75c012d4f + .quad 0x0c974651cae1f2ea + .quad 0x2fb63273675d70ca + .quad 0x0ba7250b864403f5 + + // 2^36 * 2 * G + + .quad 0xbb0d18fd029c6421 + .quad 0xbc2d142189298f02 + .quad 0x8347f8e68b250e96 + .quad 0x7b9f2fe8032d71c9 + .quad 0xdd63589386f86d9c + .quad 0x61699176e13a85a4 + .quad 0x2e5111954eaa7d57 + .quad 0x32c21b57fb60bdfb + .quad 0xd87823cd319e0780 + .quad 0xefc4cfc1897775c5 + .quad 0x4854fb129a0ab3f7 + .quad 0x12c49d417238c371 + + // 2^36 * 3 * G + + .quad 0x0950b533ffe83769 + .quad 0x21861c1d8e1d6bd1 + .quad 0xf022d8381302e510 + .quad 0x2509200c6391cab4 + .quad 0x09b3a01783799542 + .quad 0x626dd08faad5ee3f + .quad 0xba00bceeeb70149f + .quad 0x1421b246a0a444c9 + .quad 0x4aa43a8e8c24a7c7 + .quad 0x04c1f540d8f05ef5 + .quad 0xadba5e0c0b3eb9dc + .quad 0x2ab5504448a49ce3 + + // 2^36 * 4 * G + + .quad 0x2ed227266f0f5dec + .quad 0x9824ee415ed50824 + .quad 0x807bec7c9468d415 + .quad 0x7093bae1b521e23f + .quad 0xdc07ac631c5d3afa + .quad 0x58615171f9df8c6c + .quad 0x72a079d89d73e2b0 + .quad 0x7301f4ceb4eae15d + .quad 0x6409e759d6722c41 + .quad 0xa674e1cf72bf729b + .quad 0xbc0a24eb3c21e569 + .quad 0x390167d24ebacb23 + + // 2^36 * 5 * G + + .quad 0x27f58e3bba353f1c + .quad 0x4c47764dbf6a4361 + .quad 0xafbbc4e56e562650 + .quad 0x07db2ee6aae1a45d + .quad 0xd7bb054ba2f2120b + .quad 0xe2b9ceaeb10589b7 + .quad 0x3fe8bac8f3c0edbe + .quad 0x4cbd40767112cb69 + .quad 0x0b603cc029c58176 + .quad 0x5988e3825cb15d61 + .quad 0x2bb61413dcf0ad8d + .quad 0x7b8eec6c74183287 + + // 2^36 * 6 * G + + .quad 0xe4ca40782cd27cb0 + .quad 0xdaf9c323fbe967bd + .quad 0xb29bd34a8ad41e9e + .quad 0x72810497626ede4d + .quad 0x32fee570fc386b73 + .quad 0xda8b0141da3a8cc7 + .quad 0x975ffd0ac8968359 + .quad 0x6ee809a1b132a855 + .quad 0x9444bb31fcfd863a + .quad 0x2fe3690a3e4e48c5 + .quad 0xdc29c867d088fa25 + .quad 0x13bd1e38d173292e + + // 2^36 * 7 * G + + .quad 0xd32b4cd8696149b5 + .quad 0xe55937d781d8aab7 + .quad 0x0bcb2127ae122b94 + .quad 0x41e86fcfb14099b0 + .quad 0x223fb5cf1dfac521 + .quad 0x325c25316f554450 + .quad 0x030b98d7659177ac + .quad 0x1ed018b64f88a4bd + .quad 0x3630dfa1b802a6b0 + .quad 0x880f874742ad3bd5 + .quad 0x0af90d6ceec5a4d4 + .quad 0x746a247a37cdc5d9 + + // 2^36 * 8 * G + + .quad 0xd531b8bd2b7b9af6 + .quad 0x5005093537fc5b51 + .quad 0x232fcf25c593546d + .quad 0x20a365142bb40f49 + .quad 0x6eccd85278d941ed + .quad 0x2254ae83d22f7843 + .quad 0xc522d02e7bbfcdb7 + .quad 0x681e3351bff0e4e2 + .quad 0x8b64b59d83034f45 + .quad 0x2f8b71f21fa20efb + .quad 0x69249495ba6550e4 + .quad 0x539ef98e45d5472b + + // 2^40 * 1 * G + + .quad 0x6e7bb6a1a6205275 + .quad 0xaa4f21d7413c8e83 + .quad 0x6f56d155e88f5cb2 + .quad 0x2de25d4ba6345be1 + .quad 0xd074d8961cae743f + .quad 0xf86d18f5ee1c63ed + .quad 0x97bdc55be7f4ed29 + .quad 0x4cbad279663ab108 + .quad 0x80d19024a0d71fcd + .quad 0xc525c20afb288af8 + .quad 0xb1a3974b5f3a6419 + .quad 0x7d7fbcefe2007233 + + // 2^40 * 2 * G + + .quad 0xfaef1e6a266b2801 + .quad 0x866c68c4d5739f16 + .quad 0xf68a2fbc1b03762c + .quad 0x5975435e87b75a8d + .quad 0xcd7c5dc5f3c29094 + .quad 0xc781a29a2a9105ab + .quad 0x80c61d36421c3058 + .quad 0x4f9cd196dcd8d4d7 + .quad 0x199297d86a7b3768 + .quad 0xd0d058241ad17a63 + .quad 0xba029cad5c1c0c17 + .quad 0x7ccdd084387a0307 + + // 2^40 * 3 * G + + .quad 0xdca6422c6d260417 + .quad 0xae153d50948240bd + .quad 0xa9c0c1b4fb68c677 + .quad 0x428bd0ed61d0cf53 + .quad 0x9b0c84186760cc93 + .quad 0xcdae007a1ab32a99 + .quad 0xa88dec86620bda18 + .quad 0x3593ca848190ca44 + .quad 0x9213189a5e849aa7 + .quad 0xd4d8c33565d8facd + .quad 0x8c52545b53fdbbd1 + .quad 0x27398308da2d63e6 + + // 2^40 * 4 * G + + .quad 0x42c38d28435ed413 + .quad 0xbd50f3603278ccc9 + .quad 0xbb07ab1a79da03ef + .quad 0x269597aebe8c3355 + .quad 0xb9a10e4c0a702453 + .quad 0x0fa25866d57d1bde + .quad 0xffb9d9b5cd27daf7 + .quad 0x572c2945492c33fd + .quad 0xc77fc745d6cd30be + .quad 0xe4dfe8d3e3baaefb + .quad 0xa22c8830aa5dda0c + .quad 0x7f985498c05bca80 + + // 2^40 * 5 * G + + .quad 0x3849ce889f0be117 + .quad 0x8005ad1b7b54a288 + .quad 0x3da3c39f23fc921c + .quad 0x76c2ec470a31f304 + .quad 0xd35615520fbf6363 + .quad 0x08045a45cf4dfba6 + .quad 0xeec24fbc873fa0c2 + .quad 0x30f2653cd69b12e7 + .quad 0x8a08c938aac10c85 + .quad 0x46179b60db276bcb + .quad 0xa920c01e0e6fac70 + .quad 0x2f1273f1596473da + + // 2^40 * 6 * G + + .quad 0x4739fc7c8ae01e11 + .quad 0xfd5274904a6aab9f + .quad 0x41d98a8287728f2e + .quad 0x5d9e572ad85b69f2 + .quad 0x30488bd755a70bc0 + .quad 0x06d6b5a4f1d442e7 + .quad 0xead1a69ebc596162 + .quad 0x38ac1997edc5f784 + .quad 0x0666b517a751b13b + .quad 0x747d06867e9b858c + .quad 0xacacc011454dde49 + .quad 0x22dfcd9cbfe9e69c + + // 2^40 * 7 * G + + .quad 0x8ddbd2e0c30d0cd9 + .quad 0xad8e665facbb4333 + .quad 0x8f6b258c322a961f + .quad 0x6b2916c05448c1c7 + .quad 0x56ec59b4103be0a1 + .quad 0x2ee3baecd259f969 + .quad 0x797cb29413f5cd32 + .quad 0x0fe9877824cde472 + .quad 0x7edb34d10aba913b + .quad 0x4ea3cd822e6dac0e + .quad 0x66083dff6578f815 + .quad 0x4c303f307ff00a17 + + // 2^40 * 8 * G + + .quad 0xd30a3bd617b28c85 + .quad 0xc5d377b739773bea + .quad 0xc6c6e78c1e6a5cbf + .quad 0x0d61b8f78b2ab7c4 + .quad 0x29fc03580dd94500 + .quad 0xecd27aa46fbbec93 + .quad 0x130a155fc2e2a7f8 + .quad 0x416b151ab706a1d5 + .quad 0x56a8d7efe9c136b0 + .quad 0xbd07e5cd58e44b20 + .quad 0xafe62fda1b57e0ab + .quad 0x191a2af74277e8d2 + + // 2^44 * 1 * G + + .quad 0xd550095bab6f4985 + .quad 0x04f4cd5b4fbfaf1a + .quad 0x9d8e2ed12a0c7540 + .quad 0x2bc24e04b2212286 + .quad 0x09d4b60b2fe09a14 + .quad 0xc384f0afdbb1747e + .quad 0x58e2ea8978b5fd6e + .quad 0x519ef577b5e09b0a + .quad 0x1863d7d91124cca9 + .quad 0x7ac08145b88a708e + .quad 0x2bcd7309857031f5 + .quad 0x62337a6e8ab8fae5 + + // 2^44 * 2 * G + + .quad 0x4bcef17f06ffca16 + .quad 0xde06e1db692ae16a + .quad 0x0753702d614f42b0 + .quad 0x5f6041b45b9212d0 + .quad 0xd1ab324e1b3a1273 + .quad 0x18947cf181055340 + .quad 0x3b5d9567a98c196e + .quad 0x7fa00425802e1e68 + .quad 0x7d531574028c2705 + .quad 0x80317d69db0d75fe + .quad 0x30fface8ef8c8ddd + .quad 0x7e9de97bb6c3e998 + + // 2^44 * 3 * G + + .quad 0x1558967b9e6585a3 + .quad 0x97c99ce098e98b92 + .quad 0x10af149b6eb3adad + .quad 0x42181fe8f4d38cfa + .quad 0xf004be62a24d40dd + .quad 0xba0659910452d41f + .quad 0x81c45ee162a44234 + .quad 0x4cb829d8a22266ef + .quad 0x1dbcaa8407b86681 + .quad 0x081f001e8b26753b + .quad 0x3cd7ce6a84048e81 + .quad 0x78af11633f25f22c + + // 2^44 * 4 * G + + .quad 0x8416ebd40b50babc + .quad 0x1508722628208bee + .quad 0xa3148fafb9c1c36d + .quad 0x0d07daacd32d7d5d + .quad 0x3241c00e7d65318c + .quad 0xe6bee5dcd0e86de7 + .quad 0x118b2dc2fbc08c26 + .quad 0x680d04a7fc603dc3 + .quad 0xf9c2414a695aa3eb + .quad 0xdaa42c4c05a68f21 + .quad 0x7c6c23987f93963e + .quad 0x210e8cd30c3954e3 + + // 2^44 * 5 * G + + .quad 0xac4201f210a71c06 + .quad 0x6a65e0aef3bfb021 + .quad 0xbc42c35c393632f7 + .quad 0x56ea8db1865f0742 + .quad 0x2b50f16137fe6c26 + .quad 0xe102bcd856e404d8 + .quad 0x12b0f1414c561f6b + .quad 0x51b17bc8d028ec91 + .quad 0xfff5fb4bcf535119 + .quad 0xf4989d79df1108a0 + .quad 0xbdfcea659a3ba325 + .quad 0x18a11f1174d1a6f2 + + // 2^44 * 6 * G + + .quad 0x407375ab3f6bba29 + .quad 0x9ec3b6d8991e482e + .quad 0x99c80e82e55f92e9 + .quad 0x307c13b6fb0c0ae1 + .quad 0xfbd63cdad27a5f2c + .quad 0xf00fc4bc8aa106d7 + .quad 0x53fb5c1a8e64a430 + .quad 0x04eaabe50c1a2e85 + .quad 0x24751021cb8ab5e7 + .quad 0xfc2344495c5010eb + .quad 0x5f1e717b4e5610a1 + .quad 0x44da5f18c2710cd5 + + // 2^44 * 7 * G + + .quad 0x033cc55ff1b82eb5 + .quad 0xb15ae36d411cae52 + .quad 0xba40b6198ffbacd3 + .quad 0x768edce1532e861f + .quad 0x9156fe6b89d8eacc + .quad 0xe6b79451e23126a1 + .quad 0xbd7463d93944eb4e + .quad 0x726373f6767203ae + .quad 0xe305ca72eb7ef68a + .quad 0x662cf31f70eadb23 + .quad 0x18f026fdb4c45b68 + .quad 0x513b5384b5d2ecbd + + // 2^44 * 8 * G + + .quad 0x46d46280c729989e + .quad 0x4b93fbd05368a5dd + .quad 0x63df3f81d1765a89 + .quad 0x34cebd64b9a0a223 + .quad 0x5e2702878af34ceb + .quad 0x900b0409b946d6ae + .quad 0x6512ebf7dabd8512 + .quad 0x61d9b76988258f81 + .quad 0xa6c5a71349b7d94b + .quad 0xa3f3d15823eb9446 + .quad 0x0416fbd277484834 + .quad 0x69d45e6f2c70812f + + // 2^48 * 1 * G + + .quad 0xce16f74bc53c1431 + .quad 0x2b9725ce2072edde + .quad 0xb8b9c36fb5b23ee7 + .quad 0x7e2e0e450b5cc908 + .quad 0x9fe62b434f460efb + .quad 0xded303d4a63607d6 + .quad 0xf052210eb7a0da24 + .quad 0x237e7dbe00545b93 + .quad 0x013575ed6701b430 + .quad 0x231094e69f0bfd10 + .quad 0x75320f1583e47f22 + .quad 0x71afa699b11155e3 + + // 2^48 * 2 * G + + .quad 0x65ce6f9b3953b61d + .quad 0xc65839eaafa141e6 + .quad 0x0f435ffda9f759fe + .quad 0x021142e9c2b1c28e + .quad 0xea423c1c473b50d6 + .quad 0x51e87a1f3b38ef10 + .quad 0x9b84bf5fb2c9be95 + .quad 0x00731fbc78f89a1c + .quad 0xe430c71848f81880 + .quad 0xbf960c225ecec119 + .quad 0xb6dae0836bba15e3 + .quad 0x4c4d6f3347e15808 + + // 2^48 * 3 * G + + .quad 0x18f7eccfc17d1fc9 + .quad 0x6c75f5a651403c14 + .quad 0xdbde712bf7ee0cdf + .quad 0x193fddaaa7e47a22 + .quad 0x2f0cddfc988f1970 + .quad 0x6b916227b0b9f51b + .quad 0x6ec7b6c4779176be + .quad 0x38bf9500a88f9fa8 + .quad 0x1fd2c93c37e8876f + .quad 0xa2f61e5a18d1462c + .quad 0x5080f58239241276 + .quad 0x6a6fb99ebf0d4969 + + // 2^48 * 4 * G + + .quad 0x6a46c1bb560855eb + .quad 0x2416bb38f893f09d + .quad 0xd71d11378f71acc1 + .quad 0x75f76914a31896ea + .quad 0xeeb122b5b6e423c6 + .quad 0x939d7010f286ff8e + .quad 0x90a92a831dcf5d8c + .quad 0x136fda9f42c5eb10 + .quad 0xf94cdfb1a305bdd1 + .quad 0x0f364b9d9ff82c08 + .quad 0x2a87d8a5c3bb588a + .quad 0x022183510be8dcba + + // 2^48 * 5 * G + + .quad 0x4af766385ead2d14 + .quad 0xa08ed880ca7c5830 + .quad 0x0d13a6e610211e3d + .quad 0x6a071ce17b806c03 + .quad 0x9d5a710143307a7f + .quad 0xb063de9ec47da45f + .quad 0x22bbfe52be927ad3 + .quad 0x1387c441fd40426c + .quad 0xb5d3c3d187978af8 + .quad 0x722b5a3d7f0e4413 + .quad 0x0d7b4848bb477ca0 + .quad 0x3171b26aaf1edc92 + + // 2^48 * 6 * G + + .quad 0xa92f319097564ca8 + .quad 0xff7bb84c2275e119 + .quad 0x4f55fe37a4875150 + .quad 0x221fd4873cf0835a + .quad 0xa60db7d8b28a47d1 + .quad 0xa6bf14d61770a4f1 + .quad 0xd4a1f89353ddbd58 + .quad 0x6c514a63344243e9 + .quad 0x2322204f3a156341 + .quad 0xfb73e0e9ba0a032d + .quad 0xfce0dd4c410f030e + .quad 0x48daa596fb924aaa + + // 2^48 * 7 * G + + .quad 0x6eca8e665ca59cc7 + .quad 0xa847254b2e38aca0 + .quad 0x31afc708d21e17ce + .quad 0x676dd6fccad84af7 + .quad 0x14f61d5dc84c9793 + .quad 0x9941f9e3ef418206 + .quad 0xcdf5b88f346277ac + .quad 0x58c837fa0e8a79a9 + .quad 0x0cf9688596fc9058 + .quad 0x1ddcbbf37b56a01b + .quad 0xdcc2e77d4935d66a + .quad 0x1c4f73f2c6a57f0a + + // 2^48 * 8 * G + + .quad 0x0e7a4fbd305fa0bb + .quad 0x829d4ce054c663ad + .quad 0xf421c3832fe33848 + .quad 0x795ac80d1bf64c42 + .quad 0xb36e706efc7c3484 + .quad 0x73dfc9b4c3c1cf61 + .quad 0xeb1d79c9781cc7e5 + .quad 0x70459adb7daf675c + .quad 0x1b91db4991b42bb3 + .quad 0x572696234b02dcca + .quad 0x9fdf9ee51f8c78dc + .quad 0x5fe162848ce21fd3 + + // 2^52 * 1 * G + + .quad 0xe2790aae4d077c41 + .quad 0x8b938270db7469a3 + .quad 0x6eb632dc8abd16a2 + .quad 0x720814ecaa064b72 + .quad 0x315c29c795115389 + .quad 0xd7e0e507862f74ce + .quad 0x0c4a762185927432 + .quad 0x72de6c984a25a1e4 + .quad 0xae9ab553bf6aa310 + .quad 0x050a50a9806d6e1b + .quad 0x92bb7403adff5139 + .quad 0x0394d27645be618b + + // 2^52 * 2 * G + + .quad 0x4d572251857eedf4 + .quad 0xe3724edde19e93c5 + .quad 0x8a71420e0b797035 + .quad 0x3b3c833687abe743 + .quad 0xf5396425b23545a4 + .quad 0x15a7a27e98fbb296 + .quad 0xab6c52bc636fdd86 + .quad 0x79d995a8419334ee + .quad 0xcd8a8ea61195dd75 + .quad 0xa504d8a81dd9a82f + .quad 0x540dca81a35879b6 + .quad 0x60dd16a379c86a8a + + // 2^52 * 3 * G + + .quad 0x35a2c8487381e559 + .quad 0x596ffea6d78082cb + .quad 0xcb9771ebdba7b653 + .quad 0x5a08b5019b4da685 + .quad 0x3501d6f8153e47b8 + .quad 0xb7a9675414a2f60c + .quad 0x112ee8b6455d9523 + .quad 0x4e62a3c18112ea8a + .quad 0xc8d4ac04516ab786 + .quad 0x595af3215295b23d + .quad 0xd6edd234db0230c1 + .quad 0x0929efe8825b41cc + + // 2^52 * 4 * G + + .quad 0x5f0601d1cbd0f2d3 + .quad 0x736e412f6132bb7f + .quad 0x83604432238dde87 + .quad 0x1e3a5272f5c0753c + .quad 0x8b3172b7ad56651d + .quad 0x01581b7a3fabd717 + .quad 0x2dc94df6424df6e4 + .quad 0x30376e5d2c29284f + .quad 0xd2918da78159a59c + .quad 0x6bdc1cd93f0713f3 + .quad 0x565f7a934acd6590 + .quad 0x53daacec4cb4c128 + + // 2^52 * 5 * G + + .quad 0x4ca73bd79cc8a7d6 + .quad 0x4d4a738f47e9a9b2 + .quad 0xf4cbf12942f5fe00 + .quad 0x01a13ff9bdbf0752 + .quad 0x99852bc3852cfdb0 + .quad 0x2cc12e9559d6ed0b + .quad 0x70f9e2bf9b5ac27b + .quad 0x4f3b8c117959ae99 + .quad 0x55b6c9c82ff26412 + .quad 0x1ac4a8c91fb667a8 + .quad 0xd527bfcfeb778bf2 + .quad 0x303337da7012a3be + + // 2^52 * 6 * G + + .quad 0x955422228c1c9d7c + .quad 0x01fac1371a9b340f + .quad 0x7e8d9177925b48d7 + .quad 0x53f8ad5661b3e31b + .quad 0x976d3ccbfad2fdd1 + .quad 0xcb88839737a640a8 + .quad 0x2ff00c1d6734cb25 + .quad 0x269ff4dc789c2d2b + .quad 0x0c003fbdc08d678d + .quad 0x4d982fa37ead2b17 + .quad 0xc07e6bcdb2e582f1 + .quad 0x296c7291df412a44 + + // 2^52 * 7 * G + + .quad 0x7903de2b33daf397 + .quad 0xd0ff0619c9a624b3 + .quad 0x8a1d252b555b3e18 + .quad 0x2b6d581c52e0b7c0 + .quad 0xdfb23205dab8b59e + .quad 0x465aeaa0c8092250 + .quad 0xd133c1189a725d18 + .quad 0x2327370261f117d1 + .quad 0x3d0543d3623e7986 + .quad 0x679414c2c278a354 + .quad 0xae43f0cc726196f6 + .quad 0x7836c41f8245eaba + + // 2^52 * 8 * G + + .quad 0xe7a254db49e95a81 + .quad 0x5192d5d008b0ad73 + .quad 0x4d20e5b1d00afc07 + .quad 0x5d55f8012cf25f38 + .quad 0xca651e848011937c + .quad 0xc6b0c46e6ef41a28 + .quad 0xb7021ba75f3f8d52 + .quad 0x119dff99ead7b9fd + .quad 0x43eadfcbf4b31d4d + .quad 0xc6503f7411148892 + .quad 0xfeee68c5060d3b17 + .quad 0x329293b3dd4a0ac8 + + // 2^56 * 1 * G + + .quad 0x4e59214fe194961a + .quad 0x49be7dc70d71cd4f + .quad 0x9300cfd23b50f22d + .quad 0x4789d446fc917232 + .quad 0x2879852d5d7cb208 + .quad 0xb8dedd70687df2e7 + .quad 0xdc0bffab21687891 + .quad 0x2b44c043677daa35 + .quad 0x1a1c87ab074eb78e + .quad 0xfac6d18e99daf467 + .quad 0x3eacbbcd484f9067 + .quad 0x60c52eef2bb9a4e4 + + // 2^56 * 2 * G + + .quad 0x0b5d89bc3bfd8bf1 + .quad 0xb06b9237c9f3551a + .quad 0x0e4c16b0d53028f5 + .quad 0x10bc9c312ccfcaab + .quad 0x702bc5c27cae6d11 + .quad 0x44c7699b54a48cab + .quad 0xefbc4056ba492eb2 + .quad 0x70d77248d9b6676d + .quad 0xaa8ae84b3ec2a05b + .quad 0x98699ef4ed1781e0 + .quad 0x794513e4708e85d1 + .quad 0x63755bd3a976f413 + + // 2^56 * 3 * G + + .quad 0xb55fa03e2ad10853 + .quad 0x356f75909ee63569 + .quad 0x9ff9f1fdbe69b890 + .quad 0x0d8cc1c48bc16f84 + .quad 0x3dc7101897f1acb7 + .quad 0x5dda7d5ec165bbd8 + .quad 0x508e5b9c0fa1020f + .quad 0x2763751737c52a56 + .quad 0x029402d36eb419a9 + .quad 0xf0b44e7e77b460a5 + .quad 0xcfa86230d43c4956 + .quad 0x70c2dd8a7ad166e7 + + // 2^56 * 4 * G + + .quad 0x656194509f6fec0e + .quad 0xee2e7ea946c6518d + .quad 0x9733c1f367e09b5c + .quad 0x2e0fac6363948495 + .quad 0x91d4967db8ed7e13 + .quad 0x74252f0ad776817a + .quad 0xe40982e00d852564 + .quad 0x32b8613816a53ce5 + .quad 0x79e7f7bee448cd64 + .quad 0x6ac83a67087886d0 + .quad 0xf89fd4d9a0e4db2e + .quad 0x4179215c735a4f41 + + // 2^56 * 5 * G + + .quad 0x8c7094e7d7dced2a + .quad 0x97fb8ac347d39c70 + .quad 0xe13be033a906d902 + .quad 0x700344a30cd99d76 + .quad 0xe4ae33b9286bcd34 + .quad 0xb7ef7eb6559dd6dc + .quad 0x278b141fb3d38e1f + .quad 0x31fa85662241c286 + .quad 0xaf826c422e3622f4 + .quad 0xc12029879833502d + .quad 0x9bc1b7e12b389123 + .quad 0x24bb2312a9952489 + + // 2^56 * 6 * G + + .quad 0xb1a8ed1732de67c3 + .quad 0x3cb49418461b4948 + .quad 0x8ebd434376cfbcd2 + .quad 0x0fee3e871e188008 + .quad 0x41f80c2af5f85c6b + .quad 0x687284c304fa6794 + .quad 0x8945df99a3ba1bad + .quad 0x0d1d2af9ffeb5d16 + .quad 0xa9da8aa132621edf + .quad 0x30b822a159226579 + .quad 0x4004197ba79ac193 + .quad 0x16acd79718531d76 + + // 2^56 * 7 * G + + .quad 0x72df72af2d9b1d3d + .quad 0x63462a36a432245a + .quad 0x3ecea07916b39637 + .quad 0x123e0ef6b9302309 + .quad 0xc959c6c57887b6ad + .quad 0x94e19ead5f90feba + .quad 0x16e24e62a342f504 + .quad 0x164ed34b18161700 + .quad 0x487ed94c192fe69a + .quad 0x61ae2cea3a911513 + .quad 0x877bf6d3b9a4de27 + .quad 0x78da0fc61073f3eb + + // 2^56 * 8 * G + + .quad 0x5bf15d28e52bc66a + .quad 0x2c47e31870f01a8e + .quad 0x2419afbc06c28bdd + .quad 0x2d25deeb256b173a + .quad 0xa29f80f1680c3a94 + .quad 0x71f77e151ae9e7e6 + .quad 0x1100f15848017973 + .quad 0x054aa4b316b38ddd + .quad 0xdfc8468d19267cb8 + .quad 0x0b28789c66e54daf + .quad 0x2aeb1d2a666eec17 + .quad 0x134610a6ab7da760 + + // 2^60 * 1 * G + + .quad 0xcaf55ec27c59b23f + .quad 0x99aeed3e154d04f2 + .quad 0x68441d72e14141f4 + .quad 0x140345133932a0a2 + .quad 0xd91430e0dc028c3c + .quad 0x0eb955a85217c771 + .quad 0x4b09e1ed2c99a1fa + .quad 0x42881af2bd6a743c + .quad 0x7bfec69aab5cad3d + .quad 0xc23e8cd34cb2cfad + .quad 0x685dd14bfb37d6a2 + .quad 0x0ad6d64415677a18 + + // 2^60 * 2 * G + + .quad 0x781a439e417becb5 + .quad 0x4ac5938cd10e0266 + .quad 0x5da385110692ac24 + .quad 0x11b065a2ade31233 + .quad 0x7914892847927e9f + .quad 0x33dad6ef370aa877 + .quad 0x1f8f24fa11122703 + .quad 0x5265ac2f2adf9592 + .quad 0x405fdd309afcb346 + .quad 0xd9723d4428e63f54 + .quad 0x94c01df05f65aaae + .quad 0x43e4dc3ae14c0809 + + // 2^60 * 3 * G + + .quad 0xbc12c7f1a938a517 + .quad 0x473028ab3180b2e1 + .quad 0x3f78571efbcd254a + .quad 0x74e534426ff6f90f + .quad 0xea6f7ac3adc2c6a3 + .quad 0xd0e928f6e9717c94 + .quad 0xe2d379ead645eaf5 + .quad 0x46dd8785c51ffbbe + .quad 0x709801be375c8898 + .quad 0x4b06dab5e3fd8348 + .quad 0x75880ced27230714 + .quad 0x2b09468fdd2f4c42 + + // 2^60 * 4 * G + + .quad 0x97c749eeb701cb96 + .quad 0x83f438d4b6a369c3 + .quad 0x62962b8b9a402cd9 + .quad 0x6976c7509888df7b + .quad 0x5b97946582ffa02a + .quad 0xda096a51fea8f549 + .quad 0xa06351375f77af9b + .quad 0x1bcfde61201d1e76 + .quad 0x4a4a5490246a59a2 + .quad 0xd63ebddee87fdd90 + .quad 0xd9437c670d2371fa + .quad 0x69e87308d30f8ed6 + + // 2^60 * 5 * G + + .quad 0x435a8bb15656beb0 + .quad 0xf8fac9ba4f4d5bca + .quad 0xb9b278c41548c075 + .quad 0x3eb0ef76e892b622 + .quad 0x0f80bf028bc80303 + .quad 0x6aae16b37a18cefb + .quad 0xdd47ea47d72cd6a3 + .quad 0x61943588f4ed39aa + .quad 0xd26e5c3e91039f85 + .quad 0xc0e9e77df6f33aa9 + .quad 0xe8968c5570066a93 + .quad 0x3c34d1881faaaddd + + // 2^60 * 6 * G + + .quad 0x3f9d2b5ea09f9ec0 + .quad 0x1dab3b6fb623a890 + .quad 0xa09ba3ea72d926c4 + .quad 0x374193513fd8b36d + .quad 0xbd5b0b8f2fffe0d9 + .quad 0x6aa254103ed24fb9 + .quad 0x2ac7d7bcb26821c4 + .quad 0x605b394b60dca36a + .quad 0xb4e856e45a9d1ed2 + .quad 0xefe848766c97a9a2 + .quad 0xb104cf641e5eee7d + .quad 0x2f50b81c88a71c8f + + // 2^60 * 7 * G + + .quad 0x31723c61fc6811bb + .quad 0x9cb450486211800f + .quad 0x768933d347995753 + .quad 0x3491a53502752fcd + .quad 0x2b552ca0a7da522a + .quad 0x3230b336449b0250 + .quad 0xf2c4c5bca4b99fb9 + .quad 0x7b2c674958074a22 + .quad 0xd55165883ed28cdf + .quad 0x12d84fd2d362de39 + .quad 0x0a874ad3e3378e4f + .quad 0x000d2b1f7c763e74 + + // 2^60 * 8 * G + + .quad 0x3d420811d06d4a67 + .quad 0xbefc048590e0ffe3 + .quad 0xf870c6b7bd487bde + .quad 0x6e2a7316319afa28 + .quad 0x9624778c3e94a8ab + .quad 0x0ad6f3cee9a78bec + .quad 0x948ac7810d743c4f + .quad 0x76627935aaecfccc + .quad 0x56a8ac24d6d59a9f + .quad 0xc8db753e3096f006 + .quad 0x477f41e68f4c5299 + .quad 0x588d851cf6c86114 + + // 2^64 * 1 * G + + .quad 0x51138ec78df6b0fe + .quad 0x5397da89e575f51b + .quad 0x09207a1d717af1b9 + .quad 0x2102fdba2b20d650 + .quad 0xcd2a65e777d1f515 + .quad 0x548991878faa60f1 + .quad 0xb1b73bbcdabc06e5 + .quad 0x654878cba97cc9fb + .quad 0x969ee405055ce6a1 + .quad 0x36bca7681251ad29 + .quad 0x3a1af517aa7da415 + .quad 0x0ad725db29ecb2ba + + // 2^64 * 2 * G + + .quad 0xdc4267b1834e2457 + .quad 0xb67544b570ce1bc5 + .quad 0x1af07a0bf7d15ed7 + .quad 0x4aefcffb71a03650 + .quad 0xfec7bc0c9b056f85 + .quad 0x537d5268e7f5ffd7 + .quad 0x77afc6624312aefa + .quad 0x4f675f5302399fd9 + .quad 0xc32d36360415171e + .quad 0xcd2bef118998483b + .quad 0x870a6eadd0945110 + .quad 0x0bccbb72a2a86561 + + // 2^64 * 3 * G + + .quad 0x185e962feab1a9c8 + .quad 0x86e7e63565147dcd + .quad 0xb092e031bb5b6df2 + .quad 0x4024f0ab59d6b73e + .quad 0x186d5e4c50fe1296 + .quad 0xe0397b82fee89f7e + .quad 0x3bc7f6c5507031b0 + .quad 0x6678fd69108f37c2 + .quad 0x1586fa31636863c2 + .quad 0x07f68c48572d33f2 + .quad 0x4f73cc9f789eaefc + .quad 0x2d42e2108ead4701 + + // 2^64 * 4 * G + + .quad 0x97f5131594dfd29b + .quad 0x6155985d313f4c6a + .quad 0xeba13f0708455010 + .quad 0x676b2608b8d2d322 + .quad 0x21717b0d0f537593 + .quad 0x914e690b131e064c + .quad 0x1bb687ae752ae09f + .quad 0x420bf3a79b423c6e + .quad 0x8138ba651c5b2b47 + .quad 0x8671b6ec311b1b80 + .quad 0x7bff0cb1bc3135b0 + .quad 0x745d2ffa9c0cf1e0 + + // 2^64 * 5 * G + + .quad 0xbf525a1e2bc9c8bd + .quad 0xea5b260826479d81 + .quad 0xd511c70edf0155db + .quad 0x1ae23ceb960cf5d0 + .quad 0x6036df5721d34e6a + .quad 0xb1db8827997bb3d0 + .quad 0xd3c209c3c8756afa + .quad 0x06e15be54c1dc839 + .quad 0x5b725d871932994a + .quad 0x32351cb5ceb1dab0 + .quad 0x7dc41549dab7ca05 + .quad 0x58ded861278ec1f7 + + // 2^64 * 6 * G + + .quad 0xd8173793f266c55c + .quad 0xc8c976c5cc454e49 + .quad 0x5ce382f8bc26c3a8 + .quad 0x2ff39de85485f6f9 + .quad 0x2dfb5ba8b6c2c9a8 + .quad 0x48eeef8ef52c598c + .quad 0x33809107f12d1573 + .quad 0x08ba696b531d5bd8 + .quad 0x77ed3eeec3efc57a + .quad 0x04e05517d4ff4811 + .quad 0xea3d7a3ff1a671cb + .quad 0x120633b4947cfe54 + + // 2^64 * 7 * G + + .quad 0x0b94987891610042 + .quad 0x4ee7b13cecebfae8 + .quad 0x70be739594f0a4c0 + .quad 0x35d30a99b4d59185 + .quad 0x82bd31474912100a + .quad 0xde237b6d7e6fbe06 + .quad 0xe11e761911ea79c6 + .quad 0x07433be3cb393bde + .quad 0xff7944c05ce997f4 + .quad 0x575d3de4b05c51a3 + .quad 0x583381fd5a76847c + .quad 0x2d873ede7af6da9f + + // 2^64 * 8 * G + + .quad 0x157a316443373409 + .quad 0xfab8b7eef4aa81d9 + .quad 0xb093fee6f5a64806 + .quad 0x2e773654707fa7b6 + .quad 0xaa6202e14e5df981 + .quad 0xa20d59175015e1f5 + .quad 0x18a275d3bae21d6c + .quad 0x0543618a01600253 + .quad 0x0deabdf4974c23c1 + .quad 0xaa6f0a259dce4693 + .quad 0x04202cb8a29aba2c + .quad 0x4b1443362d07960d + + // 2^68 * 1 * G + + .quad 0x47b837f753242cec + .quad 0x256dc48cc04212f2 + .quad 0xe222fbfbe1d928c5 + .quad 0x48ea295bad8a2c07 + .quad 0x299b1c3f57c5715e + .quad 0x96cb929e6b686d90 + .quad 0x3004806447235ab3 + .quad 0x2c435c24a44d9fe1 + .quad 0x0607c97c80f8833f + .quad 0x0e851578ca25ec5b + .quad 0x54f7450b161ebb6f + .quad 0x7bcb4792a0def80e + + // 2^68 * 2 * G + + .quad 0x8487e3d02bc73659 + .quad 0x4baf8445059979df + .quad 0xd17c975adcad6fbf + .quad 0x57369f0bdefc96b6 + .quad 0x1cecd0a0045224c2 + .quad 0x757f1b1b69e53952 + .quad 0x775b7a925289f681 + .quad 0x1b6cc62016736148 + .quad 0xf1a9990175638698 + .quad 0x353dd1beeeaa60d3 + .quad 0x849471334c9ba488 + .quad 0x63fa6e6843ade311 + + // 2^68 * 3 * G + + .quad 0xd15c20536597c168 + .quad 0x9f73740098d28789 + .quad 0x18aee7f13257ba1f + .quad 0x3418bfda07346f14 + .quad 0x2195becdd24b5eb7 + .quad 0x5e41f18cc0cd44f9 + .quad 0xdf28074441ca9ede + .quad 0x07073b98f35b7d67 + .quad 0xd03c676c4ce530d4 + .quad 0x0b64c0473b5df9f4 + .quad 0x065cef8b19b3a31e + .quad 0x3084d661533102c9 + + // 2^68 * 4 * G + + .quad 0xe1f6b79ebf8469ad + .quad 0x15801004e2663135 + .quad 0x9a498330af74181b + .quad 0x3ba2504f049b673c + .quad 0x9a6ce876760321fd + .quad 0x7fe2b5109eb63ad8 + .quad 0x00e7d4ae8ac80592 + .quad 0x73d86b7abb6f723a + .quad 0x0b52b5606dba5ab6 + .quad 0xa9134f0fbbb1edab + .quad 0x30a9520d9b04a635 + .quad 0x6813b8f37973e5db + + // 2^68 * 5 * G + + .quad 0x9854b054334127c1 + .quad 0x105d047882fbff25 + .quad 0xdb49f7f944186f4f + .quad 0x1768e838bed0b900 + .quad 0xf194ca56f3157e29 + .quad 0x136d35705ef528a5 + .quad 0xdd4cef778b0599bc + .quad 0x7d5472af24f833ed + .quad 0xd0ef874daf33da47 + .quad 0x00d3be5db6e339f9 + .quad 0x3f2a8a2f9c9ceece + .quad 0x5d1aeb792352435a + + // 2^68 * 6 * G + + .quad 0xf59e6bb319cd63ca + .quad 0x670c159221d06839 + .quad 0xb06d565b2150cab6 + .quad 0x20fb199d104f12a3 + .quad 0x12c7bfaeb61ba775 + .quad 0xb84e621fe263bffd + .quad 0x0b47a5c35c840dcf + .quad 0x7e83be0bccaf8634 + .quad 0x61943dee6d99c120 + .quad 0x86101f2e460b9fe0 + .quad 0x6bb2f1518ee8598d + .quad 0x76b76289fcc475cc + + // 2^68 * 7 * G + + .quad 0x791b4cc1756286fa + .quad 0xdbced317d74a157c + .quad 0x7e732421ea72bde6 + .quad 0x01fe18491131c8e9 + .quad 0x4245f1a1522ec0b3 + .quad 0x558785b22a75656d + .quad 0x1d485a2548a1b3c0 + .quad 0x60959eccd58fe09f + .quad 0x3ebfeb7ba8ed7a09 + .quad 0x49fdc2bbe502789c + .quad 0x44ebce5d3c119428 + .quad 0x35e1eb55be947f4a + + // 2^68 * 8 * G + + .quad 0xdbdae701c5738dd3 + .quad 0xf9c6f635b26f1bee + .quad 0x61e96a8042f15ef4 + .quad 0x3aa1d11faf60a4d8 + .quad 0x14fd6dfa726ccc74 + .quad 0x3b084cfe2f53b965 + .quad 0xf33ae4f552a2c8b4 + .quad 0x59aab07a0d40166a + .quad 0x77bcec4c925eac25 + .quad 0x1848718460137738 + .quad 0x5b374337fea9f451 + .quad 0x1865e78ec8e6aa46 + + // 2^72 * 1 * G + + .quad 0xccc4b7c7b66e1f7a + .quad 0x44157e25f50c2f7e + .quad 0x3ef06dfc713eaf1c + .quad 0x582f446752da63f7 + .quad 0x967c54e91c529ccb + .quad 0x30f6269264c635fb + .quad 0x2747aff478121965 + .quad 0x17038418eaf66f5c + .quad 0xc6317bd320324ce4 + .quad 0xa81042e8a4488bc4 + .quad 0xb21ef18b4e5a1364 + .quad 0x0c2a1c4bcda28dc9 + + // 2^72 * 2 * G + + .quad 0xd24dc7d06f1f0447 + .quad 0xb2269e3edb87c059 + .quad 0xd15b0272fbb2d28f + .quad 0x7c558bd1c6f64877 + .quad 0xedc4814869bd6945 + .quad 0x0d6d907dbe1c8d22 + .quad 0xc63bd212d55cc5ab + .quad 0x5a6a9b30a314dc83 + .quad 0xd0ec1524d396463d + .quad 0x12bb628ac35a24f0 + .quad 0xa50c3a791cbc5fa4 + .quad 0x0404a5ca0afbafc3 + + // 2^72 * 3 * G + + .quad 0x8c1f40070aa743d6 + .quad 0xccbad0cb5b265ee8 + .quad 0x574b046b668fd2de + .quad 0x46395bfdcadd9633 + .quad 0x62bc9e1b2a416fd1 + .quad 0xb5c6f728e350598b + .quad 0x04343fd83d5d6967 + .quad 0x39527516e7f8ee98 + .quad 0x117fdb2d1a5d9a9c + .quad 0x9c7745bcd1005c2a + .quad 0xefd4bef154d56fea + .quad 0x76579a29e822d016 + + // 2^72 * 4 * G + + .quad 0x45b68e7e49c02a17 + .quad 0x23cd51a2bca9a37f + .quad 0x3ed65f11ec224c1b + .quad 0x43a384dc9e05bdb1 + .quad 0x333cb51352b434f2 + .quad 0xd832284993de80e1 + .quad 0xb5512887750d35ce + .quad 0x02c514bb2a2777c1 + .quad 0x684bd5da8bf1b645 + .quad 0xfb8bd37ef6b54b53 + .quad 0x313916d7a9b0d253 + .quad 0x1160920961548059 + + // 2^72 * 5 * G + + .quad 0xb44d166929dacfaa + .quad 0xda529f4c8413598f + .quad 0xe9ef63ca453d5559 + .quad 0x351e125bc5698e0b + .quad 0x7a385616369b4dcd + .quad 0x75c02ca7655c3563 + .quad 0x7dc21bf9d4f18021 + .quad 0x2f637d7491e6e042 + .quad 0xd4b49b461af67bbe + .quad 0xd603037ac8ab8961 + .quad 0x71dee19ff9a699fb + .quad 0x7f182d06e7ce2a9a + + // 2^72 * 6 * G + + .quad 0x7a7c8e64ab0168ec + .quad 0xcb5a4a5515edc543 + .quad 0x095519d347cd0eda + .quad 0x67d4ac8c343e93b0 + .quad 0x09454b728e217522 + .quad 0xaa58e8f4d484b8d8 + .quad 0xd358254d7f46903c + .quad 0x44acc043241c5217 + .quad 0x1c7d6bbb4f7a5777 + .quad 0x8b35fed4918313e1 + .quad 0x4adca1c6c96b4684 + .quad 0x556d1c8312ad71bd + + // 2^72 * 7 * G + + .quad 0x17ef40e30c8d3982 + .quad 0x31f7073e15a3fa34 + .quad 0x4f21f3cb0773646e + .quad 0x746c6c6d1d824eff + .quad 0x81f06756b11be821 + .quad 0x0faff82310a3f3dd + .quad 0xf8b2d0556a99465d + .quad 0x097abe38cc8c7f05 + .quad 0x0c49c9877ea52da4 + .quad 0x4c4369559bdc1d43 + .quad 0x022c3809f7ccebd2 + .quad 0x577e14a34bee84bd + + // 2^72 * 8 * G + + .quad 0xf0e268ac61a73b0a + .quad 0xf2fafa103791a5f5 + .quad 0xc1e13e826b6d00e9 + .quad 0x60fa7ee96fd78f42 + .quad 0x94fecebebd4dd72b + .quad 0xf46a4fda060f2211 + .quad 0x124a5977c0c8d1ff + .quad 0x705304b8fb009295 + .quad 0xb63d1d354d296ec6 + .quad 0xf3c3053e5fad31d8 + .quad 0x670b958cb4bd42ec + .quad 0x21398e0ca16353fd + + // 2^76 * 1 * G + + .quad 0x216ab2ca8da7d2ef + .quad 0x366ad9dd99f42827 + .quad 0xae64b9004fdd3c75 + .quad 0x403a395b53909e62 + .quad 0x86c5fc16861b7e9a + .quad 0xf6a330476a27c451 + .quad 0x01667267a1e93597 + .quad 0x05ffb9cd6082dfeb + .quad 0xa617fa9ff53f6139 + .quad 0x60f2b5e513e66cb6 + .quad 0xd7a8beefb3448aa4 + .quad 0x7a2932856f5ea192 + + // 2^76 * 2 * G + + .quad 0x0b39d761b02de888 + .quad 0x5f550e7ed2414e1f + .quad 0xa6bfa45822e1a940 + .quad 0x050a2f7dfd447b99 + .quad 0xb89c444879639302 + .quad 0x4ae4f19350c67f2c + .quad 0xf0b35da8c81af9c6 + .quad 0x39d0003546871017 + .quad 0x437c3b33a650db77 + .quad 0x6bafe81dbac52bb2 + .quad 0xfe99402d2db7d318 + .quad 0x2b5b7eec372ba6ce + + // 2^76 * 3 * G + + .quad 0xb3bc4bbd83f50eef + .quad 0x508f0c998c927866 + .quad 0x43e76587c8b7e66e + .quad 0x0f7655a3a47f98d9 + .quad 0xa694404d613ac8f4 + .quad 0x500c3c2bfa97e72c + .quad 0x874104d21fcec210 + .quad 0x1b205fb38604a8ee + .quad 0x55ecad37d24b133c + .quad 0x441e147d6038c90b + .quad 0x656683a1d62c6fee + .quad 0x0157d5dc87e0ecae + + // 2^76 * 4 * G + + .quad 0xf2a7af510354c13d + .quad 0xd7a0b145aa372b60 + .quad 0x2869b96a05a3d470 + .quad 0x6528e42d82460173 + .quad 0x95265514d71eb524 + .quad 0xe603d8815df14593 + .quad 0x147cdf410d4de6b7 + .quad 0x5293b1730437c850 + .quad 0x23d0e0814bccf226 + .quad 0x92c745cd8196fb93 + .quad 0x8b61796c59541e5b + .quad 0x40a44df0c021f978 + + // 2^76 * 5 * G + + .quad 0xdaa869894f20ea6a + .quad 0xea14a3d14c620618 + .quad 0x6001fccb090bf8be + .quad 0x35f4e822947e9cf0 + .quad 0x86c96e514bc5d095 + .quad 0xf20d4098fca6804a + .quad 0x27363d89c826ea5d + .quad 0x39ca36565719cacf + .quad 0x97506f2f6f87b75c + .quad 0xc624aea0034ae070 + .quad 0x1ec856e3aad34dd6 + .quad 0x055b0be0e440e58f + + // 2^76 * 6 * G + + .quad 0x6469a17d89735d12 + .quad 0xdb6f27d5e662b9f1 + .quad 0x9fcba3286a395681 + .quad 0x363b8004d269af25 + .quad 0x4d12a04b6ea33da2 + .quad 0x57cf4c15e36126dd + .quad 0x90ec9675ee44d967 + .quad 0x64ca348d2a985aac + .quad 0x99588e19e4c4912d + .quad 0xefcc3b4e1ca5ce6b + .quad 0x4522ea60fa5b98d5 + .quad 0x7064bbab1de4a819 + + // 2^76 * 7 * G + + .quad 0xb919e1515a770641 + .quad 0xa9a2e2c74e7f8039 + .quad 0x7527250b3df23109 + .quad 0x756a7330ac27b78b + .quad 0xa290c06142542129 + .quad 0xf2e2c2aebe8d5b90 + .quad 0xcf2458db76abfe1b + .quad 0x02157ade83d626bf + .quad 0x3e46972a1b9a038b + .quad 0x2e4ee66a7ee03fb4 + .quad 0x81a248776edbb4ca + .quad 0x1a944ee88ecd0563 + + // 2^76 * 8 * G + + .quad 0xd5a91d1151039372 + .quad 0x2ed377b799ca26de + .quad 0xa17202acfd366b6b + .quad 0x0730291bd6901995 + .quad 0xbb40a859182362d6 + .quad 0xb99f55778a4d1abb + .quad 0x8d18b427758559f6 + .quad 0x26c20fe74d26235a + .quad 0x648d1d9fe9cc22f5 + .quad 0x66bc561928dd577c + .quad 0x47d3ed21652439d1 + .quad 0x49d271acedaf8b49 + + // 2^80 * 1 * G + + .quad 0x89f5058a382b33f3 + .quad 0x5ae2ba0bad48c0b4 + .quad 0x8f93b503a53db36e + .quad 0x5aa3ed9d95a232e6 + .quad 0x2798aaf9b4b75601 + .quad 0x5eac72135c8dad72 + .quad 0xd2ceaa6161b7a023 + .quad 0x1bbfb284e98f7d4e + .quad 0x656777e9c7d96561 + .quad 0xcb2b125472c78036 + .quad 0x65053299d9506eee + .quad 0x4a07e14e5e8957cc + + // 2^80 * 2 * G + + .quad 0x4ee412cb980df999 + .quad 0xa315d76f3c6ec771 + .quad 0xbba5edde925c77fd + .quad 0x3f0bac391d313402 + .quad 0x240b58cdc477a49b + .quad 0xfd38dade6447f017 + .quad 0x19928d32a7c86aad + .quad 0x50af7aed84afa081 + .quad 0x6e4fde0115f65be5 + .quad 0x29982621216109b2 + .quad 0x780205810badd6d9 + .quad 0x1921a316baebd006 + + // 2^80 * 3 * G + + .quad 0x89422f7edfb870fc + .quad 0x2c296beb4f76b3bd + .quad 0x0738f1d436c24df7 + .quad 0x6458df41e273aeb0 + .quad 0xd75aad9ad9f3c18b + .quad 0x566a0eef60b1c19c + .quad 0x3e9a0bac255c0ed9 + .quad 0x7b049deca062c7f5 + .quad 0xdccbe37a35444483 + .quad 0x758879330fedbe93 + .quad 0x786004c312c5dd87 + .quad 0x6093dccbc2950e64 + + // 2^80 * 4 * G + + .quad 0x1ff39a8585e0706d + .quad 0x36d0a5d8b3e73933 + .quad 0x43b9f2e1718f453b + .quad 0x57d1ea084827a97c + .quad 0x6bdeeebe6084034b + .quad 0x3199c2b6780fb854 + .quad 0x973376abb62d0695 + .quad 0x6e3180c98b647d90 + .quad 0xee7ab6e7a128b071 + .quad 0xa4c1596d93a88baa + .quad 0xf7b4de82b2216130 + .quad 0x363e999ddd97bd18 + + // 2^80 * 5 * G + + .quad 0x96a843c135ee1fc4 + .quad 0x976eb35508e4c8cf + .quad 0xb42f6801b58cd330 + .quad 0x48ee9b78693a052b + .quad 0x2f1848dce24baec6 + .quad 0x769b7255babcaf60 + .quad 0x90cb3c6e3cefe931 + .quad 0x231f979bc6f9b355 + .quad 0x5c31de4bcc2af3c6 + .quad 0xb04bb030fe208d1f + .quad 0xb78d7009c14fb466 + .quad 0x079bfa9b08792413 + + // 2^80 * 6 * G + + .quad 0xe3903a51da300df4 + .quad 0x843964233da95ab0 + .quad 0xed3cf12d0b356480 + .quad 0x038c77f684817194 + .quad 0xf3c9ed80a2d54245 + .quad 0x0aa08b7877f63952 + .quad 0xd76dac63d1085475 + .quad 0x1ef4fb159470636b + .quad 0x854e5ee65b167bec + .quad 0x59590a4296d0cdc2 + .quad 0x72b2df3498102199 + .quad 0x575ee92a4a0bff56 + + // 2^80 * 7 * G + + .quad 0xd4c080908a182fcf + .quad 0x30e170c299489dbd + .quad 0x05babd5752f733de + .quad 0x43d4e7112cd3fd00 + .quad 0x5d46bc450aa4d801 + .quad 0xc3af1227a533b9d8 + .quad 0x389e3b262b8906c2 + .quad 0x200a1e7e382f581b + .quad 0x518db967eaf93ac5 + .quad 0x71bc989b056652c0 + .quad 0xfe2b85d9567197f5 + .quad 0x050eca52651e4e38 + + // 2^80 * 8 * G + + .quad 0xc3431ade453f0c9c + .quad 0xe9f5045eff703b9b + .quad 0xfcd97ac9ed847b3d + .quad 0x4b0ee6c21c58f4c6 + .quad 0x97ac397660e668ea + .quad 0x9b19bbfe153ab497 + .quad 0x4cb179b534eca79f + .quad 0x6151c09fa131ae57 + .quad 0x3af55c0dfdf05d96 + .quad 0xdd262ee02ab4ee7a + .quad 0x11b2bb8712171709 + .quad 0x1fef24fa800f030b + + // 2^84 * 1 * G + + .quad 0xb496123a6b6c6609 + .quad 0xa750fe8580ab5938 + .quad 0xf471bf39b7c27a5f + .quad 0x507903ce77ac193c + .quad 0xff91a66a90166220 + .quad 0xf22552ae5bf1e009 + .quad 0x7dff85d87f90df7c + .quad 0x4f620ffe0c736fb9 + .quad 0x62f90d65dfde3e34 + .quad 0xcf28c592b9fa5fad + .quad 0x99c86ef9c6164510 + .quad 0x25d448044a256c84 + + // 2^84 * 2 * G + + .quad 0xbd68230ec7e9b16f + .quad 0x0eb1b9c1c1c5795d + .quad 0x7943c8c495b6b1ff + .quad 0x2f9faf620bbacf5e + .quad 0x2c7c4415c9022b55 + .quad 0x56a0d241812eb1fe + .quad 0xf02ea1c9d7b65e0d + .quad 0x4180512fd5323b26 + .quad 0xa4ff3e698a48a5db + .quad 0xba6a3806bd95403b + .quad 0x9f7ce1af47d5b65d + .quad 0x15e087e55939d2fb + + // 2^84 * 3 * G + + .quad 0x12207543745c1496 + .quad 0xdaff3cfdda38610c + .quad 0xe4e797272c71c34f + .quad 0x39c07b1934bdede9 + .quad 0x8894186efb963f38 + .quad 0x48a00e80dc639bd5 + .quad 0xa4e8092be96c1c99 + .quad 0x5a097d54ca573661 + .quad 0x2d45892b17c9e755 + .quad 0xd033fd7289308df8 + .quad 0x6c2fe9d9525b8bd9 + .quad 0x2edbecf1c11cc079 + + // 2^84 * 4 * G + + .quad 0x1616a4e3c715a0d2 + .quad 0x53623cb0f8341d4d + .quad 0x96ef5329c7e899cb + .quad 0x3d4e8dbba668baa6 + .quad 0xee0f0fddd087a25f + .quad 0x9c7531555c3e34ee + .quad 0x660c572e8fab3ab5 + .quad 0x0854fc44544cd3b2 + .quad 0x61eba0c555edad19 + .quad 0x24b533fef0a83de6 + .quad 0x3b77042883baa5f8 + .quad 0x678f82b898a47e8d + + // 2^84 * 5 * G + + .quad 0xb1491d0bd6900c54 + .quad 0x3539722c9d132636 + .quad 0x4db928920b362bc9 + .quad 0x4d7cd1fea68b69df + .quad 0x1e09d94057775696 + .quad 0xeed1265c3cd951db + .quad 0xfa9dac2b20bce16f + .quad 0x0f7f76e0e8d089f4 + .quad 0x36d9ebc5d485b00c + .quad 0xa2596492e4adb365 + .quad 0xc1659480c2119ccd + .quad 0x45306349186e0d5f + + // 2^84 * 6 * G + + .quad 0x94ddd0c1a6cdff1d + .quad 0x55f6f115e84213ae + .quad 0x6c935f85992fcf6a + .quad 0x067ee0f54a37f16f + .quad 0x96a414ec2b072491 + .quad 0x1bb2218127a7b65b + .quad 0x6d2849596e8a4af0 + .quad 0x65f3b08ccd27765f + .quad 0xecb29fff199801f7 + .quad 0x9d361d1fa2a0f72f + .quad 0x25f11d2375fd2f49 + .quad 0x124cefe80fe10fe2 + + // 2^84 * 7 * G + + .quad 0x4c126cf9d18df255 + .quad 0xc1d471e9147a63b6 + .quad 0x2c6d3c73f3c93b5f + .quad 0x6be3a6a2e3ff86a2 + .quad 0x1518e85b31b16489 + .quad 0x8faadcb7db710bfb + .quad 0x39b0bdf4a14ae239 + .quad 0x05f4cbea503d20c1 + .quad 0xce040e9ec04145bc + .quad 0xc71ff4e208f6834c + .quad 0xbd546e8dab8847a3 + .quad 0x64666aa0a4d2aba5 + + // 2^84 * 8 * G + + .quad 0x6841435a7c06d912 + .quad 0xca123c21bb3f830b + .quad 0xd4b37b27b1cbe278 + .quad 0x1d753b84c76f5046 + .quad 0xb0c53bf73337e94c + .quad 0x7cb5697e11e14f15 + .quad 0x4b84abac1930c750 + .quad 0x28dd4abfe0640468 + .quad 0x7dc0b64c44cb9f44 + .quad 0x18a3e1ace3925dbf + .quad 0x7a3034862d0457c4 + .quad 0x4c498bf78a0c892e + + // 2^88 * 1 * G + + .quad 0x37d653fb1aa73196 + .quad 0x0f9495303fd76418 + .quad 0xad200b09fb3a17b2 + .quad 0x544d49292fc8613e + .quad 0x22d2aff530976b86 + .quad 0x8d90b806c2d24604 + .quad 0xdca1896c4de5bae5 + .quad 0x28005fe6c8340c17 + .quad 0x6aefba9f34528688 + .quad 0x5c1bff9425107da1 + .quad 0xf75bbbcd66d94b36 + .quad 0x72e472930f316dfa + + // 2^88 * 2 * G + + .quad 0x2695208c9781084f + .quad 0xb1502a0b23450ee1 + .quad 0xfd9daea603efde02 + .quad 0x5a9d2e8c2733a34c + .quad 0x07f3f635d32a7627 + .quad 0x7aaa4d865f6566f0 + .quad 0x3c85e79728d04450 + .quad 0x1fee7f000fe06438 + .quad 0x765305da03dbf7e5 + .quad 0xa4daf2491434cdbd + .quad 0x7b4ad5cdd24a88ec + .quad 0x00f94051ee040543 + + // 2^88 * 3 * G + + .quad 0x8d356b23c3d330b2 + .quad 0xf21c8b9bb0471b06 + .quad 0xb36c316c6e42b83c + .quad 0x07d79c7e8beab10d + .quad 0xd7ef93bb07af9753 + .quad 0x583ed0cf3db766a7 + .quad 0xce6998bf6e0b1ec5 + .quad 0x47b7ffd25dd40452 + .quad 0x87fbfb9cbc08dd12 + .quad 0x8a066b3ae1eec29b + .quad 0x0d57242bdb1fc1bf + .quad 0x1c3520a35ea64bb6 + + // 2^88 * 4 * G + + .quad 0x80d253a6bccba34a + .quad 0x3e61c3a13838219b + .quad 0x90c3b6019882e396 + .quad 0x1c3d05775d0ee66f + .quad 0xcda86f40216bc059 + .quad 0x1fbb231d12bcd87e + .quad 0xb4956a9e17c70990 + .quad 0x38750c3b66d12e55 + .quad 0x692ef1409422e51a + .quad 0xcbc0c73c2b5df671 + .quad 0x21014fe7744ce029 + .quad 0x0621e2c7d330487c + + // 2^88 * 5 * G + + .quad 0xaf9860cc8259838d + .quad 0x90ea48c1c69f9adc + .quad 0x6526483765581e30 + .quad 0x0007d6097bd3a5bc + .quad 0xb7ae1796b0dbf0f3 + .quad 0x54dfafb9e17ce196 + .quad 0x25923071e9aaa3b4 + .quad 0x5d8e589ca1002e9d + .quad 0xc0bf1d950842a94b + .quad 0xb2d3c363588f2e3e + .quad 0x0a961438bb51e2ef + .quad 0x1583d7783c1cbf86 + + // 2^88 * 6 * G + + .quad 0xeceea2ef5da27ae1 + .quad 0x597c3a1455670174 + .quad 0xc9a62a126609167a + .quad 0x252a5f2e81ed8f70 + .quad 0x90034704cc9d28c7 + .quad 0x1d1b679ef72cc58f + .quad 0x16e12b5fbe5b8726 + .quad 0x4958064e83c5580a + .quad 0x0d2894265066e80d + .quad 0xfcc3f785307c8c6b + .quad 0x1b53da780c1112fd + .quad 0x079c170bd843b388 + + // 2^88 * 7 * G + + .quad 0x0506ece464fa6fff + .quad 0xbee3431e6205e523 + .quad 0x3579422451b8ea42 + .quad 0x6dec05e34ac9fb00 + .quad 0xcdd6cd50c0d5d056 + .quad 0x9af7686dbb03573b + .quad 0x3ca6723ff3c3ef48 + .quad 0x6768c0d7317b8acc + .quad 0x94b625e5f155c1b3 + .quad 0x417bf3a7997b7b91 + .quad 0xc22cbddc6d6b2600 + .quad 0x51445e14ddcd52f4 + + // 2^88 * 8 * G + + .quad 0x57502b4b3b144951 + .quad 0x8e67ff6b444bbcb3 + .quad 0xb8bd6927166385db + .quad 0x13186f31e39295c8 + .quad 0x893147ab2bbea455 + .quad 0x8c53a24f92079129 + .quad 0x4b49f948be30f7a7 + .quad 0x12e990086e4fd43d + .quad 0xf10c96b37fdfbb2e + .quad 0x9f9a935e121ceaf9 + .quad 0xdf1136c43a5b983f + .quad 0x77b2e3f05d3e99af + + // 2^92 * 1 * G + + .quad 0xfd0d75879cf12657 + .quad 0xe82fef94e53a0e29 + .quad 0xcc34a7f05bbb4be7 + .quad 0x0b251172a50c38a2 + .quad 0x9532f48fcc5cd29b + .quad 0x2ba851bea3ce3671 + .quad 0x32dacaa051122941 + .quad 0x478d99d9350004f2 + .quad 0x1d5ad94890bb02c0 + .quad 0x50e208b10ec25115 + .quad 0xa26a22894ef21702 + .quad 0x4dc923343b524805 + + // 2^92 * 2 * G + + .quad 0xe3828c400f8086b6 + .quad 0x3f77e6f7979f0dc8 + .quad 0x7ef6de304df42cb4 + .quad 0x5265797cb6abd784 + .quad 0x3ad3e3ebf36c4975 + .quad 0xd75d25a537862125 + .quad 0xe873943da025a516 + .quad 0x6bbc7cb4c411c847 + .quad 0x3c6f9cd1d4a50d56 + .quad 0xb6244077c6feab7e + .quad 0x6ff9bf483580972e + .quad 0x00375883b332acfb + + // 2^92 * 3 * G + + .quad 0x0001b2cd28cb0940 + .quad 0x63fb51a06f1c24c9 + .quad 0xb5ad8691dcd5ca31 + .quad 0x67238dbd8c450660 + .quad 0xc98bec856c75c99c + .quad 0xe44184c000e33cf4 + .quad 0x0a676b9bba907634 + .quad 0x669e2cb571f379d7 + .quad 0xcb116b73a49bd308 + .quad 0x025aad6b2392729e + .quad 0xb4793efa3f55d9b1 + .quad 0x72a1056140678bb9 + + // 2^92 * 4 * G + + .quad 0xa2b6812b1cc9249d + .quad 0x62866eee21211f58 + .quad 0x2cb5c5b85df10ece + .quad 0x03a6b259e263ae00 + .quad 0x0d8d2909e2e505b6 + .quad 0x98ca78abc0291230 + .quad 0x77ef5569a9b12327 + .quad 0x7c77897b81439b47 + .quad 0xf1c1b5e2de331cb5 + .quad 0x5a9f5d8e15fca420 + .quad 0x9fa438f17bd932b1 + .quad 0x2a381bf01c6146e7 + + // 2^92 * 5 * G + + .quad 0xac9b9879cfc811c1 + .quad 0x8b7d29813756e567 + .quad 0x50da4e607c70edfc + .quad 0x5dbca62f884400b6 + .quad 0xf7c0be32b534166f + .quad 0x27e6ca6419cf70d4 + .quad 0x934df7d7a957a759 + .quad 0x5701461dabdec2aa + .quad 0x2c6747402c915c25 + .quad 0x1bdcd1a80b0d340a + .quad 0x5e5601bd07b43f5f + .quad 0x2555b4e05539a242 + + // 2^92 * 6 * G + + .quad 0x6fc09f5266ddd216 + .quad 0xdce560a7c8e37048 + .quad 0xec65939da2df62fd + .quad 0x7a869ae7e52ed192 + .quad 0x78409b1d87e463d4 + .quad 0xad4da95acdfb639d + .quad 0xec28773755259b9c + .quad 0x69c806e9c31230ab + .quad 0x7b48f57414bb3f22 + .quad 0x68c7cee4aedccc88 + .quad 0xed2f936179ed80be + .quad 0x25d70b885f77bc4b + + // 2^92 * 7 * G + + .quad 0x4151c3d9762bf4de + .quad 0x083f435f2745d82b + .quad 0x29775a2e0d23ddd5 + .quad 0x138e3a6269a5db24 + .quad 0x98459d29bb1ae4d4 + .quad 0x56b9c4c739f954ec + .quad 0x832743f6c29b4b3e + .quad 0x21ea8e2798b6878a + .quad 0x87bef4b46a5a7b9c + .quad 0xd2299d1b5fc1d062 + .quad 0x82409818dd321648 + .quad 0x5c5abeb1e5a2e03d + + // 2^92 * 8 * G + + .quad 0x14722af4b73c2ddb + .quad 0xbc470c5f5a05060d + .quad 0x00943eac2581b02e + .quad 0x0e434b3b1f499c8f + .quad 0x02cde6de1306a233 + .quad 0x7b5a52a2116f8ec7 + .quad 0xe1c681f4c1163b5b + .quad 0x241d350660d32643 + .quad 0x6be4404d0ebc52c7 + .quad 0xae46233bb1a791f5 + .quad 0x2aec170ed25db42b + .quad 0x1d8dfd966645d694 + + // 2^96 * 1 * G + + .quad 0x296fa9c59c2ec4de + .quad 0xbc8b61bf4f84f3cb + .quad 0x1c7706d917a8f908 + .quad 0x63b795fc7ad3255d + .quad 0xd598639c12ddb0a4 + .quad 0xa5d19f30c024866b + .quad 0xd17c2f0358fce460 + .quad 0x07a195152e095e8a + .quad 0xa8368f02389e5fc8 + .quad 0x90433b02cf8de43b + .quad 0xafa1fd5dc5412643 + .quad 0x3e8fe83d032f0137 + + // 2^96 * 2 * G + + .quad 0x2f8b15b90570a294 + .quad 0x94f2427067084549 + .quad 0xde1c5ae161bbfd84 + .quad 0x75ba3b797fac4007 + .quad 0x08704c8de8efd13c + .quad 0xdfc51a8e33e03731 + .quad 0xa59d5da51260cde3 + .quad 0x22d60899a6258c86 + .quad 0x6239dbc070cdd196 + .quad 0x60fe8a8b6c7d8a9a + .quad 0xb38847bceb401260 + .quad 0x0904d07b87779e5e + + // 2^96 * 3 * G + + .quad 0xb4ce1fd4ddba919c + .quad 0xcf31db3ec74c8daa + .quad 0x2c63cc63ad86cc51 + .quad 0x43e2143fbc1dde07 + .quad 0xf4322d6648f940b9 + .quad 0x06952f0cbd2d0c39 + .quad 0x167697ada081f931 + .quad 0x6240aacebaf72a6c + .quad 0xf834749c5ba295a0 + .quad 0xd6947c5bca37d25a + .quad 0x66f13ba7e7c9316a + .quad 0x56bdaf238db40cac + + // 2^96 * 4 * G + + .quad 0x362ab9e3f53533eb + .quad 0x338568d56eb93d40 + .quad 0x9e0e14521d5a5572 + .quad 0x1d24a86d83741318 + .quad 0x1310d36cc19d3bb2 + .quad 0x062a6bb7622386b9 + .quad 0x7c9b8591d7a14f5c + .quad 0x03aa31507e1e5754 + .quad 0xf4ec7648ffd4ce1f + .quad 0xe045eaf054ac8c1c + .quad 0x88d225821d09357c + .quad 0x43b261dc9aeb4859 + + // 2^96 * 5 * G + + .quad 0xe55b1e1988bb79bb + .quad 0xa09ed07dc17a359d + .quad 0xb02c2ee2603dea33 + .quad 0x326055cf5b276bc2 + .quad 0x19513d8b6c951364 + .quad 0x94fe7126000bf47b + .quad 0x028d10ddd54f9567 + .quad 0x02b4d5e242940964 + .quad 0xb4a155cb28d18df2 + .quad 0xeacc4646186ce508 + .quad 0xc49cf4936c824389 + .quad 0x27a6c809ae5d3410 + + // 2^96 * 6 * G + + .quad 0x8ba6ebcd1f0db188 + .quad 0x37d3d73a675a5be8 + .quad 0xf22edfa315f5585a + .quad 0x2cb67174ff60a17e + .quad 0xcd2c270ac43d6954 + .quad 0xdd4a3e576a66cab2 + .quad 0x79fa592469d7036c + .quad 0x221503603d8c2599 + .quad 0x59eecdf9390be1d0 + .quad 0xa9422044728ce3f1 + .quad 0x82891c667a94f0f4 + .quad 0x7b1df4b73890f436 + + // 2^96 * 7 * G + + .quad 0xe492f2e0b3b2a224 + .quad 0x7c6c9e062b551160 + .quad 0x15eb8fe20d7f7b0e + .quad 0x61fcef2658fc5992 + .quad 0x5f2e221807f8f58c + .quad 0xe3555c9fd49409d4 + .quad 0xb2aaa88d1fb6a630 + .quad 0x68698245d352e03d + .quad 0xdbb15d852a18187a + .quad 0xf3e4aad386ddacd7 + .quad 0x44bae2810ff6c482 + .quad 0x46cf4c473daf01cf + + // 2^96 * 8 * G + + .quad 0x426525ed9ec4e5f9 + .quad 0x0e5eda0116903303 + .quad 0x72b1a7f2cbe5cadc + .quad 0x29387bcd14eb5f40 + .quad 0x213c6ea7f1498140 + .quad 0x7c1e7ef8392b4854 + .quad 0x2488c38c5629ceba + .quad 0x1065aae50d8cc5bb + .quad 0x1c2c4525df200d57 + .quad 0x5c3b2dd6bfca674a + .quad 0x0a07e7b1e1834030 + .quad 0x69a198e64f1ce716 + + // 2^100 * 1 * G + + .quad 0x7afcd613efa9d697 + .quad 0x0cc45aa41c067959 + .quad 0xa56fe104c1fada96 + .quad 0x3a73b70472e40365 + .quad 0x7b26e56b9e2d4734 + .quad 0xc4c7132b81c61675 + .quad 0xef5c9525ec9cde7f + .quad 0x39c80b16e71743ad + .quad 0x0f196e0d1b826c68 + .quad 0xf71ff0e24960e3db + .quad 0x6113167023b7436c + .quad 0x0cf0ea5877da7282 + + // 2^100 * 2 * G + + .quad 0x196c80a4ddd4ccbd + .quad 0x22e6f55d95f2dd9d + .quad 0xc75e33c740d6c71b + .quad 0x7bb51279cb3c042f + .quad 0xe332ced43ba6945a + .quad 0xde0b1361e881c05d + .quad 0x1ad40f095e67ed3b + .quad 0x5da8acdab8c63d5d + .quad 0xc4b6664a3a70159f + .quad 0x76194f0f0a904e14 + .quad 0xa5614c39a4096c13 + .quad 0x6cd0ff50979feced + + // 2^100 * 3 * G + + .quad 0xc0e067e78f4428ac + .quad 0x14835ab0a61135e3 + .quad 0xf21d14f338062935 + .quad 0x6390a4c8df04849c + .quad 0x7fecfabdb04ba18e + .quad 0xd0fc7bfc3bddbcf7 + .quad 0xa41d486e057a131c + .quad 0x641a4391f2223a61 + .quad 0xc5c6b95aa606a8db + .quad 0x914b7f9eb06825f1 + .quad 0x2a731f6b44fc9eff + .quad 0x30ddf38562705cfc + + // 2^100 * 4 * G + + .quad 0x4e3dcbdad1bff7f9 + .quad 0xc9118e8220645717 + .quad 0xbacccebc0f189d56 + .quad 0x1b4822e9d4467668 + .quad 0x33bef2bd68bcd52c + .quad 0xc649dbb069482ef2 + .quad 0xb5b6ee0c41cb1aee + .quad 0x5c294d270212a7e5 + .quad 0xab360a7f25563781 + .quad 0x2512228a480f7958 + .quad 0xc75d05276114b4e3 + .quad 0x222d9625d976fe2a + + // 2^100 * 5 * G + + .quad 0x1c717f85b372ace1 + .quad 0x81930e694638bf18 + .quad 0x239cad056bc08b58 + .quad 0x0b34271c87f8fff4 + .quad 0x0f94be7e0a344f85 + .quad 0xeb2faa8c87f22c38 + .quad 0x9ce1e75e4ee16f0f + .quad 0x43e64e5418a08dea + .quad 0x8155e2521a35ce63 + .quad 0xbe100d4df912028e + .quad 0xbff80bf8a57ddcec + .quad 0x57342dc96d6bc6e4 + + // 2^100 * 6 * G + + .quad 0xefeef065c8ce5998 + .quad 0xbf029510b5cbeaa2 + .quad 0x8c64a10620b7c458 + .quad 0x35134fb231c24855 + .quad 0xf3c3bcb71e707bf6 + .quad 0x351d9b8c7291a762 + .quad 0x00502e6edad69a33 + .quad 0x522f521f1ec8807f + .quad 0x272c1f46f9a3902b + .quad 0xc91ba3b799657bcc + .quad 0xae614b304f8a1c0e + .quad 0x7afcaad70b99017b + + // 2^100 * 7 * G + + .quad 0xc25ded54a4b8be41 + .quad 0x902d13e11bb0e2dd + .quad 0x41f43233cde82ab2 + .quad 0x1085faa5c3aae7cb + .quad 0xa88141ecef842b6b + .quad 0x55e7b14797abe6c5 + .quad 0x8c748f9703784ffe + .quad 0x5b50a1f7afcd00b7 + .quad 0x9b840f66f1361315 + .quad 0x18462242701003e9 + .quad 0x65ed45fae4a25080 + .quad 0x0a2862393fda7320 + + // 2^100 * 8 * G + + .quad 0x46ab13c8347cbc9d + .quad 0x3849e8d499c12383 + .quad 0x4cea314087d64ac9 + .quad 0x1f354134b1a29ee7 + .quad 0x960e737b6ecb9d17 + .quad 0xfaf24948d67ceae1 + .quad 0x37e7a9b4d55e1b89 + .quad 0x5cb7173cb46c59eb + .quad 0x4a89e68b82b7abf0 + .quad 0xf41cd9279ba6b7b9 + .quad 0x16e6c210e18d876f + .quad 0x7cacdb0f7f1b09c6 + + // 2^104 * 1 * G + + .quad 0x9062b2e0d91a78bc + .quad 0x47c9889cc8509667 + .quad 0x9df54a66405070b8 + .quad 0x7369e6a92493a1bf + .quad 0xe1014434dcc5caed + .quad 0x47ed5d963c84fb33 + .quad 0x70019576ed86a0e7 + .quad 0x25b2697bd267f9e4 + .quad 0x9d673ffb13986864 + .quad 0x3ca5fbd9415dc7b8 + .quad 0xe04ecc3bdf273b5e + .quad 0x1420683db54e4cd2 + + // 2^104 * 2 * G + + .quad 0xb478bd1e249dd197 + .quad 0x620c35005e58c102 + .quad 0xfb02d32fccbaac5c + .quad 0x60b63bebf508a72d + .quad 0x34eebb6fc1cc5ad0 + .quad 0x6a1b0ce99646ac8b + .quad 0xd3b0da49a66bde53 + .quad 0x31e83b4161d081c1 + .quad 0x97e8c7129e062b4f + .quad 0x49e48f4f29320ad8 + .quad 0x5bece14b6f18683f + .quad 0x55cf1eb62d550317 + + // 2^104 * 3 * G + + .quad 0x5879101065c23d58 + .quad 0x8b9d086d5094819c + .quad 0xe2402fa912c55fa7 + .quad 0x669a6564570891d4 + .quad 0x3076b5e37df58c52 + .quad 0xd73ab9dde799cc36 + .quad 0xbd831ce34913ee20 + .quad 0x1a56fbaa62ba0133 + .quad 0x943e6b505c9dc9ec + .quad 0x302557bba77c371a + .quad 0x9873ae5641347651 + .quad 0x13c4836799c58a5c + + // 2^104 * 4 * G + + .quad 0x423a5d465ab3e1b9 + .quad 0xfc13c187c7f13f61 + .quad 0x19f83664ecb5b9b6 + .quad 0x66f80c93a637b607 + .quad 0xc4dcfb6a5d8bd080 + .quad 0xdeebc4ec571a4842 + .quad 0xd4b2e883b8e55365 + .quad 0x50bdc87dc8e5b827 + .quad 0x606d37836edfe111 + .quad 0x32353e15f011abd9 + .quad 0x64b03ac325b73b96 + .quad 0x1dd56444725fd5ae + + // 2^104 * 5 * G + + .quad 0x8fa47ff83362127d + .quad 0xbc9f6ac471cd7c15 + .quad 0x6e71454349220c8b + .quad 0x0e645912219f732e + .quad 0xc297e60008bac89a + .quad 0x7d4cea11eae1c3e0 + .quad 0xf3e38be19fe7977c + .quad 0x3a3a450f63a305cd + .quad 0x078f2f31d8394627 + .quad 0x389d3183de94a510 + .quad 0xd1e36c6d17996f80 + .quad 0x318c8d9393a9a87b + + // 2^104 * 6 * G + + .quad 0xf2745d032afffe19 + .quad 0x0c9f3c497f24db66 + .quad 0xbc98d3e3ba8598ef + .quad 0x224c7c679a1d5314 + .quad 0x5d669e29ab1dd398 + .quad 0xfc921658342d9e3b + .quad 0x55851dfdf35973cd + .quad 0x509a41c325950af6 + .quad 0xbdc06edca6f925e9 + .quad 0x793ef3f4641b1f33 + .quad 0x82ec12809d833e89 + .quad 0x05bff02328a11389 + + // 2^104 * 7 * G + + .quad 0x3632137023cae00b + .quad 0x544acf0ad1accf59 + .quad 0x96741049d21a1c88 + .quad 0x780b8cc3fa2a44a7 + .quad 0x6881a0dd0dc512e4 + .quad 0x4fe70dc844a5fafe + .quad 0x1f748e6b8f4a5240 + .quad 0x576277cdee01a3ea + .quad 0x1ef38abc234f305f + .quad 0x9a577fbd1405de08 + .quad 0x5e82a51434e62a0d + .quad 0x5ff418726271b7a1 + + // 2^104 * 8 * G + + .quad 0x398e080c1789db9d + .quad 0xa7602025f3e778f5 + .quad 0xfa98894c06bd035d + .quad 0x106a03dc25a966be + .quad 0xe5db47e813b69540 + .quad 0xf35d2a3b432610e1 + .quad 0xac1f26e938781276 + .quad 0x29d4db8ca0a0cb69 + .quad 0xd9ad0aaf333353d0 + .quad 0x38669da5acd309e5 + .quad 0x3c57658ac888f7f0 + .quad 0x4ab38a51052cbefa + + // 2^108 * 1 * G + + .quad 0xdfdacbee4324c0e9 + .quad 0x054442883f955bb7 + .quad 0xdef7aaa8ea31609f + .quad 0x68aee70642287cff + .quad 0xf68fe2e8809de054 + .quad 0xe3bc096a9c82bad1 + .quad 0x076353d40aadbf45 + .quad 0x7b9b1fb5dea1959e + .quad 0xf01cc8f17471cc0c + .quad 0x95242e37579082bb + .quad 0x27776093d3e46b5f + .quad 0x2d13d55a28bd85fb + + // 2^108 * 2 * G + + .quad 0xfac5d2065b35b8da + .quad 0xa8da8a9a85624bb7 + .quad 0xccd2ca913d21cd0f + .quad 0x6b8341ee8bf90d58 + .quad 0xbf019cce7aee7a52 + .quad 0xa8ded2b6e454ead3 + .quad 0x3c619f0b87a8bb19 + .quad 0x3619b5d7560916d8 + .quad 0x3579f26b0282c4b2 + .quad 0x64d592f24fafefae + .quad 0xb7cded7b28c8c7c0 + .quad 0x6a927b6b7173a8d7 + + // 2^108 * 3 * G + + .quad 0x1f6db24f986e4656 + .quad 0x1021c02ed1e9105b + .quad 0xf8ff3fff2cc0a375 + .quad 0x1d2a6bf8c6c82592 + .quad 0x8d7040863ece88eb + .quad 0xf0e307a980eec08c + .quad 0xac2250610d788fda + .quad 0x056d92a43a0d478d + .quad 0x1b05a196fc3da5a1 + .quad 0x77d7a8c243b59ed0 + .quad 0x06da3d6297d17918 + .quad 0x66fbb494f12353f7 + + // 2^108 * 4 * G + + .quad 0x751a50b9d85c0fb8 + .quad 0xd1afdc258bcf097b + .quad 0x2f16a6a38309a969 + .quad 0x14ddff9ee5b00659 + .quad 0xd6d70996f12309d6 + .quad 0xdbfb2385e9c3d539 + .quad 0x46d602b0f7552411 + .quad 0x270a0b0557843e0c + .quad 0x61ff0640a7862bcc + .quad 0x81cac09a5f11abfe + .quad 0x9047830455d12abb + .quad 0x19a4bde1945ae873 + + // 2^108 * 5 * G + + .quad 0x9b9f26f520a6200a + .quad 0x64804443cf13eaf8 + .quad 0x8a63673f8631edd3 + .quad 0x72bbbce11ed39dc1 + .quad 0x40c709dec076c49f + .quad 0x657bfaf27f3e53f6 + .quad 0x40662331eca042c4 + .quad 0x14b375487eb4df04 + .quad 0xae853c94ab66dc47 + .quad 0xeb62343edf762d6e + .quad 0xf08e0e186fb2f7d1 + .quad 0x4f0b1c02700ab37a + + // 2^108 * 6 * G + + .quad 0xe1706787d81951fa + .quad 0xa10a2c8eb290c77b + .quad 0xe7382fa03ed66773 + .quad 0x0a4d84710bcc4b54 + .quad 0x79fd21ccc1b2e23f + .quad 0x4ae7c281453df52a + .quad 0xc8172ec9d151486b + .quad 0x68abe9443e0a7534 + .quad 0xda12c6c407831dcb + .quad 0x0da230d74d5c510d + .quad 0x4ab1531e6bd404e1 + .quad 0x4106b166bcf440ef + + // 2^108 * 7 * G + + .quad 0x02e57a421cd23668 + .quad 0x4ad9fb5d0eaef6fd + .quad 0x954e6727b1244480 + .quad 0x7f792f9d2699f331 + .quad 0xa485ccd539e4ecf2 + .quad 0x5aa3f3ad0555bab5 + .quad 0x145e3439937df82d + .quad 0x1238b51e1214283f + .quad 0x0b886b925fd4d924 + .quad 0x60906f7a3626a80d + .quad 0xecd367b4b98abd12 + .quad 0x2876beb1def344cf + + // 2^108 * 8 * G + + .quad 0xdc84e93563144691 + .quad 0x632fe8a0d61f23f4 + .quad 0x4caa800612a9a8d5 + .quad 0x48f9dbfa0e9918d3 + .quad 0xd594b3333a8a85f8 + .quad 0x4ea37689e78d7d58 + .quad 0x73bf9f455e8e351f + .quad 0x5507d7d2bc41ebb4 + .quad 0x1ceb2903299572fc + .quad 0x7c8ccaa29502d0ee + .quad 0x91bfa43411cce67b + .quad 0x5784481964a831e7 + + // 2^112 * 1 * G + + .quad 0xda7c2b256768d593 + .quad 0x98c1c0574422ca13 + .quad 0xf1a80bd5ca0ace1d + .quad 0x29cdd1adc088a690 + .quad 0xd6cfd1ef5fddc09c + .quad 0xe82b3efdf7575dce + .quad 0x25d56b5d201634c2 + .quad 0x3041c6bb04ed2b9b + .quad 0x0ff2f2f9d956e148 + .quad 0xade797759f356b2e + .quad 0x1a4698bb5f6c025c + .quad 0x104bbd6814049a7b + + // 2^112 * 2 * G + + .quad 0x51f0fd3168f1ed67 + .quad 0x2c811dcdd86f3bc2 + .quad 0x44dc5c4304d2f2de + .quad 0x5be8cc57092a7149 + .quad 0xa95d9a5fd67ff163 + .quad 0xe92be69d4cc75681 + .quad 0xb7f8024cde20f257 + .quad 0x204f2a20fb072df5 + .quad 0xc8143b3d30ebb079 + .quad 0x7589155abd652e30 + .quad 0x653c3c318f6d5c31 + .quad 0x2570fb17c279161f + + // 2^112 * 3 * G + + .quad 0x3efa367f2cb61575 + .quad 0xf5f96f761cd6026c + .quad 0xe8c7142a65b52562 + .quad 0x3dcb65ea53030acd + .quad 0x192ea9550bb8245a + .quad 0xc8e6fba88f9050d1 + .quad 0x7986ea2d88a4c935 + .quad 0x241c5f91de018668 + .quad 0x28d8172940de6caa + .quad 0x8fbf2cf022d9733a + .quad 0x16d7fcdd235b01d1 + .quad 0x08420edd5fcdf0e5 + + // 2^112 * 4 * G + + .quad 0xcdff20ab8362fa4a + .quad 0x57e118d4e21a3e6e + .quad 0xe3179617fc39e62b + .quad 0x0d9a53efbc1769fd + .quad 0x0358c34e04f410ce + .quad 0xb6135b5a276e0685 + .quad 0x5d9670c7ebb91521 + .quad 0x04d654f321db889c + .quad 0x5e7dc116ddbdb5d5 + .quad 0x2954deb68da5dd2d + .quad 0x1cb608173334a292 + .quad 0x4a7a4f2618991ad7 + + // 2^112 * 5 * G + + .quad 0xf4a718025fb15f95 + .quad 0x3df65f346b5c1b8f + .quad 0xcdfcf08500e01112 + .quad 0x11b50c4cddd31848 + .quad 0x24c3b291af372a4b + .quad 0x93da8270718147f2 + .quad 0xdd84856486899ef2 + .quad 0x4a96314223e0ee33 + .quad 0xa6e8274408a4ffd6 + .quad 0x738e177e9c1576d9 + .quad 0x773348b63d02b3f2 + .quad 0x4f4bce4dce6bcc51 + + // 2^112 * 6 * G + + .quad 0xa71fce5ae2242584 + .quad 0x26ea725692f58a9e + .quad 0xd21a09d71cea3cf4 + .quad 0x73fcdd14b71c01e6 + .quad 0x30e2616ec49d0b6f + .quad 0xe456718fcaec2317 + .quad 0x48eb409bf26b4fa6 + .quad 0x3042cee561595f37 + .quad 0x427e7079449bac41 + .quad 0x855ae36dbce2310a + .quad 0x4cae76215f841a7c + .quad 0x389e740c9a9ce1d6 + + // 2^112 * 7 * G + + .quad 0x64fcb3ae34dcb9ce + .quad 0x97500323e348d0ad + .quad 0x45b3f07d62c6381b + .quad 0x61545379465a6788 + .quad 0xc9bd78f6570eac28 + .quad 0xe55b0b3227919ce1 + .quad 0x65fc3eaba19b91ed + .quad 0x25c425e5d6263690 + .quad 0x3f3e06a6f1d7de6e + .quad 0x3ef976278e062308 + .quad 0x8c14f6264e8a6c77 + .quad 0x6539a08915484759 + + // 2^112 * 8 * G + + .quad 0xe9d21f74c3d2f773 + .quad 0xc150544125c46845 + .quad 0x624e5ce8f9b99e33 + .quad 0x11c5e4aac5cd186c + .quad 0xddc4dbd414bb4a19 + .quad 0x19b2bc3c98424f8e + .quad 0x48a89fd736ca7169 + .quad 0x0f65320ef019bd90 + .quad 0xd486d1b1cafde0c6 + .quad 0x4f3fe6e3163b5181 + .quad 0x59a8af0dfaf2939a + .quad 0x4cabc7bdec33072a + + // 2^116 * 1 * G + + .quad 0x16faa8fb532f7428 + .quad 0xdbd42ea046a4e272 + .quad 0x5337653b8b9ea480 + .quad 0x4065947223973f03 + .quad 0xf7c0a19c1a54a044 + .quad 0x4a1c5e2477bd9fbb + .quad 0xa6e3ca115af22972 + .quad 0x1819bb953f2e9e0d + .quad 0x498fbb795e042e84 + .quad 0x7d0dd89a7698b714 + .quad 0x8bfb0ba427fe6295 + .quad 0x36ba82e721200524 + + // 2^116 * 2 * G + + .quad 0xd60ecbb74245ec41 + .quad 0xfd9be89e34348716 + .quad 0xc9240afee42284de + .quad 0x4472f648d0531db4 + .quad 0xc8d69d0a57274ed5 + .quad 0x45ba803260804b17 + .quad 0xdf3cda102255dfac + .quad 0x77d221232709b339 + .quad 0x498a6d7064ad94d8 + .quad 0xa5b5c8fd9af62263 + .quad 0x8ca8ed0545c141f4 + .quad 0x2c63bec3662d358c + + // 2^116 * 3 * G + + .quad 0x7fe60d8bea787955 + .quad 0xb9dc117eb5f401b7 + .quad 0x91c7c09a19355cce + .quad 0x22692ef59442bedf + .quad 0x9a518b3a8586f8bf + .quad 0x9ee71af6cbb196f0 + .quad 0xaa0625e6a2385cf2 + .quad 0x1deb2176ddd7c8d1 + .quad 0x8563d19a2066cf6c + .quad 0x401bfd8c4dcc7cd7 + .quad 0xd976a6becd0d8f62 + .quad 0x67cfd773a278b05e + + // 2^116 * 4 * G + + .quad 0x8dec31faef3ee475 + .quad 0x99dbff8a9e22fd92 + .quad 0x512d11594e26cab1 + .quad 0x0cde561eec4310b9 + .quad 0x2d5fa9855a4e586a + .quad 0x65f8f7a449beab7e + .quad 0xaa074dddf21d33d3 + .quad 0x185cba721bcb9dee + .quad 0x93869da3f4e3cb41 + .quad 0xbf0392f540f7977e + .quad 0x026204fcd0463b83 + .quad 0x3ec91a769eec6eed + + // 2^116 * 5 * G + + .quad 0x1e9df75bf78166ad + .quad 0x4dfda838eb0cd7af + .quad 0xba002ed8c1eaf988 + .quad 0x13fedb3e11f33cfc + .quad 0x0fad2fb7b0a3402f + .quad 0x46615ecbfb69f4a8 + .quad 0xf745bcc8c5f8eaa6 + .quad 0x7a5fa8794a94e896 + .quad 0x52958faa13cd67a1 + .quad 0x965ee0818bdbb517 + .quad 0x16e58daa2e8845b3 + .quad 0x357d397d5499da8f + + // 2^116 * 6 * G + + .quad 0x1ebfa05fb0bace6c + .quad 0xc934620c1caf9a1e + .quad 0xcc771cc41d82b61a + .quad 0x2d94a16aa5f74fec + .quad 0x481dacb4194bfbf8 + .quad 0x4d77e3f1bae58299 + .quad 0x1ef4612e7d1372a0 + .quad 0x3a8d867e70ff69e1 + .quad 0x6f58cd5d55aff958 + .quad 0xba3eaa5c75567721 + .quad 0x75c123999165227d + .quad 0x69be1343c2f2b35e + + // 2^116 * 7 * G + + .quad 0x0e091d5ee197c92a + .quad 0x4f51019f2945119f + .quad 0x143679b9f034e99c + .quad 0x7d88112e4d24c696 + .quad 0x82bbbdac684b8de3 + .quad 0xa2f4c7d03fca0718 + .quad 0x337f92fbe096aaa8 + .quad 0x200d4d8c63587376 + .quad 0x208aed4b4893b32b + .quad 0x3efbf23ebe59b964 + .quad 0xd762deb0dba5e507 + .quad 0x69607bd681bd9d94 + + // 2^116 * 8 * G + + .quad 0xf6be021068de1ce1 + .quad 0xe8d518e70edcbc1f + .quad 0xe3effdd01b5505a5 + .quad 0x35f63353d3ec3fd0 + .quad 0x3b7f3bd49323a902 + .quad 0x7c21b5566b2c6e53 + .quad 0xe5ba8ff53a7852a7 + .quad 0x28bc77a5838ece00 + .quad 0x63ba78a8e25d8036 + .quad 0x63651e0094333490 + .quad 0x48d82f20288ce532 + .quad 0x3a31abfa36b57524 + + // 2^120 * 1 * G + + .quad 0x239e9624089c0a2e + .quad 0xc748c4c03afe4738 + .quad 0x17dbed2a764fa12a + .quad 0x639b93f0321c8582 + .quad 0xc08f788f3f78d289 + .quad 0xfe30a72ca1404d9f + .quad 0xf2778bfccf65cc9d + .quad 0x7ee498165acb2021 + .quad 0x7bd508e39111a1c3 + .quad 0x2b2b90d480907489 + .quad 0xe7d2aec2ae72fd19 + .quad 0x0edf493c85b602a6 + + // 2^120 * 2 * G + + .quad 0xaecc8158599b5a68 + .quad 0xea574f0febade20e + .quad 0x4fe41d7422b67f07 + .quad 0x403b92e3019d4fb4 + .quad 0x6767c4d284764113 + .quad 0xa090403ff7f5f835 + .quad 0x1c8fcffacae6bede + .quad 0x04c00c54d1dfa369 + .quad 0x4dc22f818b465cf8 + .quad 0x71a0f35a1480eff8 + .quad 0xaee8bfad04c7d657 + .quad 0x355bb12ab26176f4 + + // 2^120 * 3 * G + + .quad 0xa71e64cc7493bbf4 + .quad 0xe5bd84d9eca3b0c3 + .quad 0x0a6bc50cfa05e785 + .quad 0x0f9b8132182ec312 + .quad 0xa301dac75a8c7318 + .quad 0xed90039db3ceaa11 + .quad 0x6f077cbf3bae3f2d + .quad 0x7518eaf8e052ad8e + .quad 0xa48859c41b7f6c32 + .quad 0x0f2d60bcf4383298 + .quad 0x1815a929c9b1d1d9 + .quad 0x47c3871bbb1755c4 + + // 2^120 * 4 * G + + .quad 0x5144539771ec4f48 + .quad 0xf805b17dc98c5d6e + .quad 0xf762c11a47c3c66b + .quad 0x00b89b85764699dc + .quad 0xfbe65d50c85066b0 + .quad 0x62ecc4b0b3a299b0 + .quad 0xe53754ea441ae8e0 + .quad 0x08fea02ce8d48d5f + .quad 0x824ddd7668deead0 + .quad 0xc86445204b685d23 + .quad 0xb514cfcd5d89d665 + .quad 0x473829a74f75d537 + + // 2^120 * 5 * G + + .quad 0x82d2da754679c418 + .quad 0xe63bd7d8b2618df0 + .quad 0x355eef24ac47eb0a + .quad 0x2078684c4833c6b4 + .quad 0x23d9533aad3902c9 + .quad 0x64c2ddceef03588f + .quad 0x15257390cfe12fb4 + .quad 0x6c668b4d44e4d390 + .quad 0x3b48cf217a78820c + .quad 0xf76a0ab281273e97 + .quad 0xa96c65a78c8eed7b + .quad 0x7411a6054f8a433f + + // 2^120 * 6 * G + + .quad 0x4d659d32b99dc86d + .quad 0x044cdc75603af115 + .quad 0xb34c712cdcc2e488 + .quad 0x7c136574fb8134ff + .quad 0x579ae53d18b175b4 + .quad 0x68713159f392a102 + .quad 0x8455ecba1eef35f5 + .quad 0x1ec9a872458c398f + .quad 0xb8e6a4d400a2509b + .quad 0x9b81d7020bc882b4 + .quad 0x57e7cc9bf1957561 + .quad 0x3add88a5c7cd6460 + + // 2^120 * 7 * G + + .quad 0xab895770b635dcf2 + .quad 0x02dfef6cf66c1fbc + .quad 0x85530268beb6d187 + .quad 0x249929fccc879e74 + .quad 0x85c298d459393046 + .quad 0x8f7e35985ff659ec + .quad 0x1d2ca22af2f66e3a + .quad 0x61ba1131a406a720 + .quad 0xa3d0a0f116959029 + .quad 0x023b6b6cba7ebd89 + .quad 0x7bf15a3e26783307 + .quad 0x5620310cbbd8ece7 + + // 2^120 * 8 * G + + .quad 0x528993434934d643 + .quad 0xb9dbf806a51222f5 + .quad 0x8f6d878fc3f41c22 + .quad 0x37676a2a4d9d9730 + .quad 0x6646b5f477e285d6 + .quad 0x40e8ff676c8f6193 + .quad 0xa6ec7311abb594dd + .quad 0x7ec846f3658cec4d + .quad 0x9b5e8f3f1da22ec7 + .quad 0x130f1d776c01cd13 + .quad 0x214c8fcfa2989fb8 + .quad 0x6daaf723399b9dd5 + + // 2^124 * 1 * G + + .quad 0x591e4a5610628564 + .quad 0x2a4bb87ca8b4df34 + .quad 0xde2a2572e7a38e43 + .quad 0x3cbdabd9fee5046e + .quad 0x81aebbdd2cd13070 + .quad 0x962e4325f85a0e9e + .quad 0xde9391aacadffecb + .quad 0x53177fda52c230e6 + .quad 0xa7bc970650b9de79 + .quad 0x3d12a7fbc301b59b + .quad 0x02652e68d36ae38c + .quad 0x79d739835a6199dc + + // 2^124 * 2 * G + + .quad 0xd9354df64131c1bd + .quad 0x758094a186ec5822 + .quad 0x4464ee12e459f3c2 + .quad 0x6c11fce4cb133282 + .quad 0x21c9d9920d591737 + .quad 0x9bea41d2e9b46cd6 + .quad 0xe20e84200d89bfca + .quad 0x79d99f946eae5ff8 + .quad 0xf17b483568673205 + .quad 0x387deae83caad96c + .quad 0x61b471fd56ffe386 + .quad 0x31741195b745a599 + + // 2^124 * 3 * G + + .quad 0xe8d10190b77a360b + .quad 0x99b983209995e702 + .quad 0xbd4fdff8fa0247aa + .quad 0x2772e344e0d36a87 + .quad 0x17f8ba683b02a047 + .quad 0x50212096feefb6c8 + .quad 0x70139be21556cbe2 + .quad 0x203e44a11d98915b + .quad 0xd6863eba37b9e39f + .quad 0x105bc169723b5a23 + .quad 0x104f6459a65c0762 + .quad 0x567951295b4d38d4 + + // 2^124 * 4 * G + + .quad 0x535fd60613037524 + .quad 0xe210adf6b0fbc26a + .quad 0xac8d0a9b23e990ae + .quad 0x47204d08d72fdbf9 + .quad 0x07242eb30d4b497f + .quad 0x1ef96306b9bccc87 + .quad 0x37950934d8116f45 + .quad 0x05468d6201405b04 + .quad 0x00f565a9f93267de + .quad 0xcecfd78dc0d58e8a + .quad 0xa215e2dcf318e28e + .quad 0x4599ee919b633352 + + // 2^124 * 5 * G + + .quad 0xd3c220ca70e0e76b + .quad 0xb12bea58ea9f3094 + .quad 0x294ddec8c3271282 + .quad 0x0c3539e1a1d1d028 + .quad 0xac746d6b861ae579 + .quad 0x31ab0650f6aea9dc + .quad 0x241d661140256d4c + .quad 0x2f485e853d21a5de + .quad 0x329744839c0833f3 + .quad 0x6fe6257fd2abc484 + .quad 0x5327d1814b358817 + .quad 0x65712585893fe9bc + + // 2^124 * 6 * G + + .quad 0x9c102fb732a61161 + .quad 0xe48e10dd34d520a8 + .quad 0x365c63546f9a9176 + .quad 0x32f6fe4c046f6006 + .quad 0x81c29f1bd708ee3f + .quad 0xddcb5a05ae6407d0 + .quad 0x97aec1d7d2a3eba7 + .quad 0x1590521a91d50831 + .quad 0x40a3a11ec7910acc + .quad 0x9013dff8f16d27ae + .quad 0x1a9720d8abb195d4 + .quad 0x1bb9fe452ea98463 + + // 2^124 * 7 * G + + .quad 0xe9d1d950b3d54f9e + .quad 0x2d5f9cbee00d33c1 + .quad 0x51c2c656a04fc6ac + .quad 0x65c091ee3c1cbcc9 + .quad 0xcf5e6c95cc36747c + .quad 0x294201536b0bc30d + .quad 0x453ac67cee797af0 + .quad 0x5eae6ab32a8bb3c9 + .quad 0x7083661114f118ea + .quad 0x2b37b87b94349cad + .quad 0x7273f51cb4e99f40 + .quad 0x78a2a95823d75698 + + // 2^124 * 8 * G + + .quad 0xa2b072e95c8c2ace + .quad 0x69cffc96651e9c4b + .quad 0x44328ef842e7b42b + .quad 0x5dd996c122aadeb3 + .quad 0xb4f23c425ef83207 + .quad 0xabf894d3c9a934b5 + .quad 0xd0708c1339fd87f7 + .quad 0x1876789117166130 + .quad 0x925b5ef0670c507c + .quad 0x819bc842b93c33bf + .quad 0x10792e9a70dd003f + .quad 0x59ad4b7a6e28dc74 + + // 2^128 * 1 * G + + .quad 0x5f3a7562eb3dbe47 + .quad 0xf7ea38548ebda0b8 + .quad 0x00c3e53145747299 + .quad 0x1304e9e71627d551 + .quad 0x583b04bfacad8ea2 + .quad 0x29b743e8148be884 + .quad 0x2b1e583b0810c5db + .quad 0x2b5449e58eb3bbaa + .quad 0x789814d26adc9cfe + .quad 0x3c1bab3f8b48dd0b + .quad 0xda0fe1fff979c60a + .quad 0x4468de2d7c2dd693 + + // 2^128 * 2 * G + + .quad 0x51bb355e9419469e + .quad 0x33e6dc4c23ddc754 + .quad 0x93a5b6d6447f9962 + .quad 0x6cce7c6ffb44bd63 + .quad 0x4b9ad8c6f86307ce + .quad 0x21113531435d0c28 + .quad 0xd4a866c5657a772c + .quad 0x5da6427e63247352 + .quad 0x1a94c688deac22ca + .quad 0xb9066ef7bbae1ff8 + .quad 0x88ad8c388d59580f + .quad 0x58f29abfe79f2ca8 + + // 2^128 * 3 * G + + .quad 0xe90ecfab8de73e68 + .quad 0x54036f9f377e76a5 + .quad 0xf0495b0bbe015982 + .quad 0x577629c4a7f41e36 + .quad 0x4b5a64bf710ecdf6 + .quad 0xb14ce538462c293c + .quad 0x3643d056d50b3ab9 + .quad 0x6af93724185b4870 + .quad 0x3220024509c6a888 + .quad 0xd2e036134b558973 + .quad 0x83e236233c33289f + .quad 0x701f25bb0caec18f + + // 2^128 * 4 * G + + .quad 0xc3a8b0f8e4616ced + .quad 0xf700660e9e25a87d + .quad 0x61e3061ff4bca59c + .quad 0x2e0c92bfbdc40be9 + .quad 0x9d18f6d97cbec113 + .quad 0x844a06e674bfdbe4 + .quad 0x20f5b522ac4e60d6 + .quad 0x720a5bc050955e51 + .quad 0x0c3f09439b805a35 + .quad 0xe84e8b376242abfc + .quad 0x691417f35c229346 + .quad 0x0e9b9cbb144ef0ec + + // 2^128 * 5 * G + + .quad 0xfbbad48ffb5720ad + .quad 0xee81916bdbf90d0e + .quad 0xd4813152635543bf + .quad 0x221104eb3f337bd8 + .quad 0x8dee9bd55db1beee + .quad 0xc9c3ab370a723fb9 + .quad 0x44a8f1bf1c68d791 + .quad 0x366d44191cfd3cde + .quad 0x9e3c1743f2bc8c14 + .quad 0x2eda26fcb5856c3b + .quad 0xccb82f0e68a7fb97 + .quad 0x4167a4e6bc593244 + + // 2^128 * 6 * G + + .quad 0x643b9d2876f62700 + .quad 0x5d1d9d400e7668eb + .quad 0x1b4b430321fc0684 + .quad 0x7938bb7e2255246a + .quad 0xc2be2665f8ce8fee + .quad 0xe967ff14e880d62c + .quad 0xf12e6e7e2f364eee + .quad 0x34b33370cb7ed2f6 + .quad 0xcdc591ee8681d6cc + .quad 0xce02109ced85a753 + .quad 0xed7485c158808883 + .quad 0x1176fc6e2dfe65e4 + + // 2^128 * 7 * G + + .quad 0xb4af6cd05b9c619b + .quad 0x2ddfc9f4b2a58480 + .quad 0x3d4fa502ebe94dc4 + .quad 0x08fc3a4c677d5f34 + .quad 0xdb90e28949770eb8 + .quad 0x98fbcc2aacf440a3 + .quad 0x21354ffeded7879b + .quad 0x1f6a3e54f26906b6 + .quad 0x60a4c199d30734ea + .quad 0x40c085b631165cd6 + .quad 0xe2333e23f7598295 + .quad 0x4f2fad0116b900d1 + + // 2^128 * 8 * G + + .quad 0x44beb24194ae4e54 + .quad 0x5f541c511857ef6c + .quad 0xa61e6b2d368d0498 + .quad 0x445484a4972ef7ab + .quad 0x962cd91db73bb638 + .quad 0xe60577aafc129c08 + .quad 0x6f619b39f3b61689 + .quad 0x3451995f2944ee81 + .quad 0x9152fcd09fea7d7c + .quad 0x4a816c94b0935cf6 + .quad 0x258e9aaa47285c40 + .quad 0x10b89ca6042893b7 + + // 2^132 * 1 * G + + .quad 0x9b2a426e3b646025 + .quad 0x32127190385ce4cf + .quad 0xa25cffc2dd6dea45 + .quad 0x06409010bea8de75 + .quad 0xd67cded679d34aa0 + .quad 0xcc0b9ec0cc4db39f + .quad 0xa535a456e35d190f + .quad 0x2e05d9eaf61f6fef + .quad 0xc447901ad61beb59 + .quad 0x661f19bce5dc880a + .quad 0x24685482b7ca6827 + .quad 0x293c778cefe07f26 + + // 2^132 * 2 * G + + .quad 0x86809e7007069096 + .quad 0xaad75b15e4e50189 + .quad 0x07f35715a21a0147 + .quad 0x0487f3f112815d5e + .quad 0x16c795d6a11ff200 + .quad 0xcb70d0e2b15815c9 + .quad 0x89f293209b5395b5 + .quad 0x50b8c2d031e47b4f + .quad 0x48350c08068a4962 + .quad 0x6ffdd05351092c9a + .quad 0x17af4f4aaf6fc8dd + .quad 0x4b0553b53cdba58b + + // 2^132 * 3 * G + + .quad 0x9c65fcbe1b32ff79 + .quad 0xeb75ea9f03b50f9b + .quad 0xfced2a6c6c07e606 + .quad 0x35106cd551717908 + .quad 0xbf05211b27c152d4 + .quad 0x5ec26849bd1af639 + .quad 0x5e0b2caa8e6fab98 + .quad 0x054c8bdd50bd0840 + .quad 0x38a0b12f1dcf073d + .quad 0x4b60a8a3b7f6a276 + .quad 0xfed5ac25d3404f9a + .quad 0x72e82d5e5505c229 + + // 2^132 * 4 * G + + .quad 0x6b0b697ff0d844c8 + .quad 0xbb12f85cd979cb49 + .quad 0xd2a541c6c1da0f1f + .quad 0x7b7c242958ce7211 + .quad 0x00d9cdfd69771d02 + .quad 0x410276cd6cfbf17e + .quad 0x4c45306c1cb12ec7 + .quad 0x2857bf1627500861 + .quad 0x9f21903f0101689e + .quad 0xd779dfd3bf861005 + .quad 0xa122ee5f3deb0f1b + .quad 0x510df84b485a00d4 + + // 2^132 * 5 * G + + .quad 0xa54133bb9277a1fa + .quad 0x74ec3b6263991237 + .quad 0x1a3c54dc35d2f15a + .quad 0x2d347144e482ba3a + .quad 0x24b3c887c70ac15e + .quad 0xb0f3a557fb81b732 + .quad 0x9b2cde2fe578cc1b + .quad 0x4cf7ed0703b54f8e + .quad 0x6bd47c6598fbee0f + .quad 0x9e4733e2ab55be2d + .quad 0x1093f624127610c5 + .quad 0x4e05e26ad0a1eaa4 + + // 2^132 * 6 * G + + .quad 0xda9b6b624b531f20 + .quad 0x429a760e77509abb + .quad 0xdbe9f522e823cb80 + .quad 0x618f1856880c8f82 + .quad 0x1833c773e18fe6c0 + .quad 0xe3c4711ad3c87265 + .quad 0x3bfd3c4f0116b283 + .quad 0x1955875eb4cd4db8 + .quad 0x6da6de8f0e399799 + .quad 0x7ad61aa440fda178 + .quad 0xb32cd8105e3563dd + .quad 0x15f6beae2ae340ae + + // 2^132 * 7 * G + + .quad 0x862bcb0c31ec3a62 + .quad 0x810e2b451138f3c2 + .quad 0x788ec4b839dac2a4 + .quad 0x28f76867ae2a9281 + .quad 0xba9a0f7b9245e215 + .quad 0xf368612dd98c0dbb + .quad 0x2e84e4cbf220b020 + .quad 0x6ba92fe962d90eda + .quad 0x3e4df9655884e2aa + .quad 0xbd62fbdbdbd465a5 + .quad 0xd7596caa0de9e524 + .quad 0x6e8042ccb2b1b3d7 + + // 2^132 * 8 * G + + .quad 0xf10d3c29ce28ca6e + .quad 0xbad34540fcb6093d + .quad 0xe7426ed7a2ea2d3f + .quad 0x08af9d4e4ff298b9 + .quad 0x1530653616521f7e + .quad 0x660d06b896203dba + .quad 0x2d3989bc545f0879 + .quad 0x4b5303af78ebd7b0 + .quad 0x72f8a6c3bebcbde8 + .quad 0x4f0fca4adc3a8e89 + .quad 0x6fa9d4e8c7bfdf7a + .quad 0x0dcf2d679b624eb7 + + // 2^136 * 1 * G + + .quad 0x3d5947499718289c + .quad 0x12ebf8c524533f26 + .quad 0x0262bfcb14c3ef15 + .quad 0x20b878d577b7518e + .quad 0x753941be5a45f06e + .quad 0xd07caeed6d9c5f65 + .quad 0x11776b9c72ff51b6 + .quad 0x17d2d1d9ef0d4da9 + .quad 0x27f2af18073f3e6a + .quad 0xfd3fe519d7521069 + .quad 0x22e3b72c3ca60022 + .quad 0x72214f63cc65c6a7 + + // 2^136 * 2 * G + + .quad 0xb4e37f405307a693 + .quad 0xaba714d72f336795 + .quad 0xd6fbd0a773761099 + .quad 0x5fdf48c58171cbc9 + .quad 0x1d9db7b9f43b29c9 + .quad 0xd605824a4f518f75 + .quad 0xf2c072bd312f9dc4 + .quad 0x1f24ac855a1545b0 + .quad 0x24d608328e9505aa + .quad 0x4748c1d10c1420ee + .quad 0xc7ffe45c06fb25a2 + .quad 0x00ba739e2ae395e6 + + // 2^136 * 3 * G + + .quad 0x592e98de5c8790d6 + .quad 0xe5bfb7d345c2a2df + .quad 0x115a3b60f9b49922 + .quad 0x03283a3e67ad78f3 + .quad 0xae4426f5ea88bb26 + .quad 0x360679d984973bfb + .quad 0x5c9f030c26694e50 + .quad 0x72297de7d518d226 + .quad 0x48241dc7be0cb939 + .quad 0x32f19b4d8b633080 + .quad 0xd3dfc90d02289308 + .quad 0x05e1296846271945 + + // 2^136 * 4 * G + + .quad 0xba82eeb32d9c495a + .quad 0xceefc8fcf12bb97c + .quad 0xb02dabae93b5d1e0 + .quad 0x39c00c9c13698d9b + .quad 0xadbfbbc8242c4550 + .quad 0xbcc80cecd03081d9 + .quad 0x843566a6f5c8df92 + .quad 0x78cf25d38258ce4c + .quad 0x15ae6b8e31489d68 + .quad 0xaa851cab9c2bf087 + .quad 0xc9a75a97f04efa05 + .quad 0x006b52076b3ff832 + + // 2^136 * 5 * G + + .quad 0x29e0cfe19d95781c + .quad 0xb681df18966310e2 + .quad 0x57df39d370516b39 + .quad 0x4d57e3443bc76122 + .quad 0xf5cb7e16b9ce082d + .quad 0x3407f14c417abc29 + .quad 0xd4b36bce2bf4a7ab + .quad 0x7de2e9561a9f75ce + .quad 0xde70d4f4b6a55ecb + .quad 0x4801527f5d85db99 + .quad 0xdbc9c440d3ee9a81 + .quad 0x6b2a90af1a6029ed + + // 2^136 * 6 * G + + .quad 0x6923f4fc9ae61e97 + .quad 0x5735281de03f5fd1 + .quad 0xa764ae43e6edd12d + .quad 0x5fd8f4e9d12d3e4a + .quad 0x77ebf3245bb2d80a + .quad 0xd8301b472fb9079b + .quad 0xc647e6f24cee7333 + .quad 0x465812c8276c2109 + .quad 0x4d43beb22a1062d9 + .quad 0x7065fb753831dc16 + .quad 0x180d4a7bde2968d7 + .quad 0x05b32c2b1cb16790 + + // 2^136 * 7 * G + + .quad 0xc8c05eccd24da8fd + .quad 0xa1cf1aac05dfef83 + .quad 0xdbbeeff27df9cd61 + .quad 0x3b5556a37b471e99 + .quad 0xf7fca42c7ad58195 + .quad 0x3214286e4333f3cc + .quad 0xb6c29d0d340b979d + .quad 0x31771a48567307e1 + .quad 0x32b0c524e14dd482 + .quad 0xedb351541a2ba4b6 + .quad 0xa3d16048282b5af3 + .quad 0x4fc079d27a7336eb + + // 2^136 * 8 * G + + .quad 0x51c938b089bf2f7f + .quad 0x2497bd6502dfe9a7 + .quad 0xffffc09c7880e453 + .quad 0x124567cecaf98e92 + .quad 0xdc348b440c86c50d + .quad 0x1337cbc9cc94e651 + .quad 0x6422f74d643e3cb9 + .quad 0x241170c2bae3cd08 + .quad 0x3ff9ab860ac473b4 + .quad 0xf0911dee0113e435 + .quad 0x4ae75060ebc6c4af + .quad 0x3f8612966c87000d + + // 2^140 * 1 * G + + .quad 0x0c9c5303f7957be4 + .quad 0xa3c31a20e085c145 + .quad 0xb0721d71d0850050 + .quad 0x0aba390eab0bf2da + .quad 0x529fdffe638c7bf3 + .quad 0xdf2b9e60388b4995 + .quad 0xe027b34f1bad0249 + .quad 0x7bc92fc9b9fa74ed + .quad 0x9f97ef2e801ad9f9 + .quad 0x83697d5479afda3a + .quad 0xe906b3ffbd596b50 + .quad 0x02672b37dd3fb8e0 + + // 2^140 * 2 * G + + .quad 0x48b2ca8b260885e4 + .quad 0xa4286bec82b34c1c + .quad 0x937e1a2617f58f74 + .quad 0x741d1fcbab2ca2a5 + .quad 0xee9ba729398ca7f5 + .quad 0xeb9ca6257a4849db + .quad 0x29eb29ce7ec544e1 + .quad 0x232ca21ef736e2c8 + .quad 0xbf61423d253fcb17 + .quad 0x08803ceafa39eb14 + .quad 0xf18602df9851c7af + .quad 0x0400f3a049e3414b + + // 2^140 * 3 * G + + .quad 0xabce0476ba61c55b + .quad 0x36a3d6d7c4d39716 + .quad 0x6eb259d5e8d82d09 + .quad 0x0c9176e984d756fb + .quad 0x2efba412a06e7b06 + .quad 0x146785452c8d2560 + .quad 0xdf9713ebd67a91c7 + .quad 0x32830ac7157eadf3 + .quad 0x0e782a7ab73769e8 + .quad 0x04a05d7875b18e2c + .quad 0x29525226ebcceae1 + .quad 0x0d794f8383eba820 + + // 2^140 * 4 * G + + .quad 0xff35f5cb9e1516f4 + .quad 0xee805bcf648aae45 + .quad 0xf0d73c2bb93a9ef3 + .quad 0x097b0bf22092a6c2 + .quad 0x7be44ce7a7a2e1ac + .quad 0x411fd93efad1b8b7 + .quad 0x1734a1d70d5f7c9b + .quad 0x0d6592233127db16 + .quad 0xc48bab1521a9d733 + .quad 0xa6c2eaead61abb25 + .quad 0x625c6c1cc6cb4305 + .quad 0x7fc90fea93eb3a67 + + // 2^140 * 5 * G + + .quad 0x0408f1fe1f5c5926 + .quad 0x1a8f2f5e3b258bf4 + .quad 0x40a951a2fdc71669 + .quad 0x6598ee93c98b577e + .quad 0xc527deb59c7cb23d + .quad 0x955391695328404e + .quad 0xd64392817ccf2c7a + .quad 0x6ce97dabf7d8fa11 + .quad 0x25b5a8e50ef7c48f + .quad 0xeb6034116f2ce532 + .quad 0xc5e75173e53de537 + .quad 0x73119fa08c12bb03 + + // 2^140 * 6 * G + + .quad 0xed30129453f1a4cb + .quad 0xbce621c9c8f53787 + .quad 0xfacb2b1338bee7b9 + .quad 0x3025798a9ea8428c + .quad 0x7845b94d21f4774d + .quad 0xbf62f16c7897b727 + .quad 0x671857c03c56522b + .quad 0x3cd6a85295621212 + .quad 0x3fecde923aeca999 + .quad 0xbdaa5b0062e8c12f + .quad 0x67b99dfc96988ade + .quad 0x3f52c02852661036 + + // 2^140 * 7 * G + + .quad 0xffeaa48e2a1351c6 + .quad 0x28624754fa7f53d7 + .quad 0x0b5ba9e57582ddf1 + .quad 0x60c0104ba696ac59 + .quad 0x9258bf99eec416c6 + .quad 0xac8a5017a9d2f671 + .quad 0x629549ab16dea4ab + .quad 0x05d0e85c99091569 + .quad 0x051de020de9cbe97 + .quad 0xfa07fc56b50bcf74 + .quad 0x378cec9f0f11df65 + .quad 0x36853c69ab96de4d + + // 2^140 * 8 * G + + .quad 0x36d9b8de78f39b2d + .quad 0x7f42ed71a847b9ec + .quad 0x241cd1d679bd3fde + .quad 0x6a704fec92fbce6b + .quad 0x4433c0b0fac5e7be + .quad 0x724bae854c08dcbe + .quad 0xf1f24cc446978f9b + .quad 0x4a0aff6d62825fc8 + .quad 0xe917fb9e61095301 + .quad 0xc102df9402a092f8 + .quad 0xbf09e2f5fa66190b + .quad 0x681109bee0dcfe37 + + // 2^144 * 1 * G + + .quad 0x559a0cc9782a0dde + .quad 0x551dcdb2ea718385 + .quad 0x7f62865b31ef238c + .quad 0x504aa7767973613d + .quad 0x9c18fcfa36048d13 + .quad 0x29159db373899ddd + .quad 0xdc9f350b9f92d0aa + .quad 0x26f57eee878a19d4 + .quad 0x0cab2cd55687efb1 + .quad 0x5180d162247af17b + .quad 0x85c15a344f5a2467 + .quad 0x4041943d9dba3069 + + // 2^144 * 2 * G + + .quad 0xc3c0eeba43ebcc96 + .quad 0x8d749c9c26ea9caf + .quad 0xd9fa95ee1c77ccc6 + .quad 0x1420a1d97684340f + .quad 0x4b217743a26caadd + .quad 0x47a6b424648ab7ce + .quad 0xcb1d4f7a03fbc9e3 + .quad 0x12d931429800d019 + .quad 0x00c67799d337594f + .quad 0x5e3c5140b23aa47b + .quad 0x44182854e35ff395 + .quad 0x1b4f92314359a012 + + // 2^144 * 3 * G + + .quad 0x3e5c109d89150951 + .quad 0x39cefa912de9696a + .quad 0x20eae43f975f3020 + .quad 0x239b572a7f132dae + .quad 0x33cf3030a49866b1 + .quad 0x251f73d2215f4859 + .quad 0xab82aa4051def4f6 + .quad 0x5ff191d56f9a23f6 + .quad 0x819ed433ac2d9068 + .quad 0x2883ab795fc98523 + .quad 0xef4572805593eb3d + .quad 0x020c526a758f36cb + + // 2^144 * 4 * G + + .quad 0x779834f89ed8dbbc + .quad 0xc8f2aaf9dc7ca46c + .quad 0xa9524cdca3e1b074 + .quad 0x02aacc4615313877 + .quad 0xe931ef59f042cc89 + .quad 0x2c589c9d8e124bb6 + .quad 0xadc8e18aaec75997 + .quad 0x452cfe0a5602c50c + .quad 0x86a0f7a0647877df + .quad 0xbbc464270e607c9f + .quad 0xab17ea25f1fb11c9 + .quad 0x4cfb7d7b304b877b + + // 2^144 * 5 * G + + .quad 0x72b43d6cb89b75fe + .quad 0x54c694d99c6adc80 + .quad 0xb8c3aa373ee34c9f + .quad 0x14b4622b39075364 + .quad 0xe28699c29789ef12 + .quad 0x2b6ecd71df57190d + .quad 0xc343c857ecc970d0 + .quad 0x5b1d4cbc434d3ac5 + .quad 0xb6fb2615cc0a9f26 + .quad 0x3a4f0e2bb88dcce5 + .quad 0x1301498b3369a705 + .quad 0x2f98f71258592dd1 + + // 2^144 * 6 * G + + .quad 0x0c94a74cb50f9e56 + .quad 0x5b1ff4a98e8e1320 + .quad 0x9a2acc2182300f67 + .quad 0x3a6ae249d806aaf9 + .quad 0x2e12ae444f54a701 + .quad 0xfcfe3ef0a9cbd7de + .quad 0xcebf890d75835de0 + .quad 0x1d8062e9e7614554 + .quad 0x657ada85a9907c5a + .quad 0x1a0ea8b591b90f62 + .quad 0x8d0e1dfbdf34b4e9 + .quad 0x298b8ce8aef25ff3 + + // 2^144 * 7 * G + + .quad 0x2a927953eff70cb2 + .quad 0x4b89c92a79157076 + .quad 0x9418457a30a7cf6a + .quad 0x34b8a8404d5ce485 + .quad 0x837a72ea0a2165de + .quad 0x3fab07b40bcf79f6 + .quad 0x521636c77738ae70 + .quad 0x6ba6271803a7d7dc + .quad 0xc26eecb583693335 + .quad 0xd5a813df63b5fefd + .quad 0xa293aa9aa4b22573 + .quad 0x71d62bdd465e1c6a + + // 2^144 * 8 * G + + .quad 0x6533cc28d378df80 + .quad 0xf6db43790a0fa4b4 + .quad 0xe3645ff9f701da5a + .quad 0x74d5f317f3172ba4 + .quad 0xcd2db5dab1f75ef5 + .quad 0xd77f95cf16b065f5 + .quad 0x14571fea3f49f085 + .quad 0x1c333621262b2b3d + .quad 0xa86fe55467d9ca81 + .quad 0x398b7c752b298c37 + .quad 0xda6d0892e3ac623b + .quad 0x4aebcc4547e9d98c + + // 2^148 * 1 * G + + .quad 0x53175a7205d21a77 + .quad 0xb0c04422d3b934d4 + .quad 0xadd9f24bdd5deadc + .quad 0x074f46e69f10ff8c + .quad 0x0de9b204a059a445 + .quad 0xe15cb4aa4b17ad0f + .quad 0xe1bbec521f79c557 + .quad 0x2633f1b9d071081b + .quad 0xc1fb4177018b9910 + .quad 0xa6ea20dc6c0fe140 + .quad 0xd661f3e74354c6ff + .quad 0x5ecb72e6f1a3407a + + // 2^148 * 2 * G + + .quad 0xa515a31b2259fb4e + .quad 0x0960f3972bcac52f + .quad 0xedb52fec8d3454cb + .quad 0x382e2720c476c019 + .quad 0xfeeae106e8e86997 + .quad 0x9863337f98d09383 + .quad 0x9470480eaa06ebef + .quad 0x038b6898d4c5c2d0 + .quad 0xf391c51d8ace50a6 + .quad 0x3142d0b9ae2d2948 + .quad 0xdb4d5a1a7f24ca80 + .quad 0x21aeba8b59250ea8 + + // 2^148 * 3 * G + + .quad 0x24f13b34cf405530 + .quad 0x3c44ea4a43088af7 + .quad 0x5dd5c5170006a482 + .quad 0x118eb8f8890b086d + .quad 0x53853600f0087f23 + .quad 0x4c461879da7d5784 + .quad 0x6af303deb41f6860 + .quad 0x0a3c16c5c27c18ed + .quad 0x17e49c17cc947f3d + .quad 0xccc6eda6aac1d27b + .quad 0xdf6092ceb0f08e56 + .quad 0x4909b3e22c67c36b + + // 2^148 * 4 * G + + .quad 0x9c9c85ea63fe2e89 + .quad 0xbe1baf910e9412ec + .quad 0x8f7baa8a86fbfe7b + .quad 0x0fb17f9fef968b6c + .quad 0x59a16676706ff64e + .quad 0x10b953dd0d86a53d + .quad 0x5848e1e6ce5c0b96 + .quad 0x2d8b78e712780c68 + .quad 0x79d5c62eafc3902b + .quad 0x773a215289e80728 + .quad 0xc38ae640e10120b9 + .quad 0x09ae23717b2b1a6d + + // 2^148 * 5 * G + + .quad 0xbb6a192a4e4d083c + .quad 0x34ace0630029e192 + .quad 0x98245a59aafabaeb + .quad 0x6d9c8a9ada97faac + .quad 0x10ab8fa1ad32b1d0 + .quad 0xe9aced1be2778b24 + .quad 0xa8856bc0373de90f + .quad 0x66f35ddddda53996 + .quad 0xd27d9afb24997323 + .quad 0x1bb7e07ef6f01d2e + .quad 0x2ba7472df52ecc7f + .quad 0x03019b4f646f9dc8 + + // 2^148 * 6 * G + + .quad 0x04a186b5565345cd + .quad 0xeee76610bcc4116a + .quad 0x689c73b478fb2a45 + .quad 0x387dcbff65697512 + .quad 0xaf09b214e6b3dc6b + .quad 0x3f7573b5ad7d2f65 + .quad 0xd019d988100a23b0 + .quad 0x392b63a58b5c35f7 + .quad 0x4093addc9c07c205 + .quad 0xc565be15f532c37e + .quad 0x63dbecfd1583402a + .quad 0x61722b4aef2e032e + + // 2^148 * 7 * G + + .quad 0x0012aafeecbd47af + .quad 0x55a266fb1cd46309 + .quad 0xf203eb680967c72c + .quad 0x39633944ca3c1429 + .quad 0xd6b07a5581cb0e3c + .quad 0x290ff006d9444969 + .quad 0x08680b6a16dcda1f + .quad 0x5568d2b75a06de59 + .quad 0x8d0cb88c1b37cfe1 + .quad 0x05b6a5a3053818f3 + .quad 0xf2e9bc04b787d959 + .quad 0x6beba1249add7f64 + + // 2^148 * 8 * G + + .quad 0x1d06005ca5b1b143 + .quad 0x6d4c6bb87fd1cda2 + .quad 0x6ef5967653fcffe7 + .quad 0x097c29e8c1ce1ea5 + .quad 0x5c3cecb943f5a53b + .quad 0x9cc9a61d06c08df2 + .quad 0xcfba639a85895447 + .quad 0x5a845ae80df09fd5 + .quad 0x4ce97dbe5deb94ca + .quad 0x38d0a4388c709c48 + .quad 0xc43eced4a169d097 + .quad 0x0a1249fff7e587c3 + + // 2^152 * 1 * G + + .quad 0x12f0071b276d01c9 + .quad 0xe7b8bac586c48c70 + .quad 0x5308129b71d6fba9 + .quad 0x5d88fbf95a3db792 + .quad 0x0b408d9e7354b610 + .quad 0x806b32535ba85b6e + .quad 0xdbe63a034a58a207 + .quad 0x173bd9ddc9a1df2c + .quad 0x2b500f1efe5872df + .quad 0x58d6582ed43918c1 + .quad 0xe6ed278ec9673ae0 + .quad 0x06e1cd13b19ea319 + + // 2^152 * 2 * G + + .quad 0x40d0ad516f166f23 + .quad 0x118e32931fab6abe + .quad 0x3fe35e14a04d088e + .quad 0x3080603526e16266 + .quad 0x472baf629e5b0353 + .quad 0x3baa0b90278d0447 + .quad 0x0c785f469643bf27 + .quad 0x7f3a6a1a8d837b13 + .quad 0xf7e644395d3d800b + .quad 0x95a8d555c901edf6 + .quad 0x68cd7830592c6339 + .quad 0x30d0fded2e51307e + + // 2^152 * 3 * G + + .quad 0xe0594d1af21233b3 + .quad 0x1bdbe78ef0cc4d9c + .quad 0x6965187f8f499a77 + .quad 0x0a9214202c099868 + .quad 0x9cb4971e68b84750 + .quad 0xa09572296664bbcf + .quad 0x5c8de72672fa412b + .quad 0x4615084351c589d9 + .quad 0xbc9019c0aeb9a02e + .quad 0x55c7110d16034cae + .quad 0x0e6df501659932ec + .quad 0x3bca0d2895ca5dfe + + // 2^152 * 4 * G + + .quad 0x40f031bc3c5d62a4 + .quad 0x19fc8b3ecff07a60 + .quad 0x98183da2130fb545 + .quad 0x5631deddae8f13cd + .quad 0x9c688eb69ecc01bf + .quad 0xf0bc83ada644896f + .quad 0xca2d955f5f7a9fe2 + .quad 0x4ea8b4038df28241 + .quad 0x2aed460af1cad202 + .quad 0x46305305a48cee83 + .quad 0x9121774549f11a5f + .quad 0x24ce0930542ca463 + + // 2^152 * 5 * G + + .quad 0x1fe890f5fd06c106 + .quad 0xb5c468355d8810f2 + .quad 0x827808fe6e8caf3e + .quad 0x41d4e3c28a06d74b + .quad 0x3fcfa155fdf30b85 + .quad 0xd2f7168e36372ea4 + .quad 0xb2e064de6492f844 + .quad 0x549928a7324f4280 + .quad 0xf26e32a763ee1a2e + .quad 0xae91e4b7d25ffdea + .quad 0xbc3bd33bd17f4d69 + .quad 0x491b66dec0dcff6a + + // 2^152 * 6 * G + + .quad 0x98f5b13dc7ea32a7 + .quad 0xe3d5f8cc7e16db98 + .quad 0xac0abf52cbf8d947 + .quad 0x08f338d0c85ee4ac + .quad 0x75f04a8ed0da64a1 + .quad 0xed222caf67e2284b + .quad 0x8234a3791f7b7ba4 + .quad 0x4cf6b8b0b7018b67 + .quad 0xc383a821991a73bd + .quad 0xab27bc01df320c7a + .quad 0xc13d331b84777063 + .quad 0x530d4a82eb078a99 + + // 2^152 * 7 * G + + .quad 0x004c3630e1f94825 + .quad 0x7e2d78268cab535a + .quad 0xc7482323cc84ff8b + .quad 0x65ea753f101770b9 + .quad 0x6d6973456c9abf9e + .quad 0x257fb2fc4900a880 + .quad 0x2bacf412c8cfb850 + .quad 0x0db3e7e00cbfbd5b + .quad 0x3d66fc3ee2096363 + .quad 0x81d62c7f61b5cb6b + .quad 0x0fbe044213443b1a + .quad 0x02a4ec1921e1a1db + + // 2^152 * 8 * G + + .quad 0x5ce6259a3b24b8a2 + .quad 0xb8577acc45afa0b8 + .quad 0xcccbe6e88ba07037 + .quad 0x3d143c51127809bf + .quad 0xf5c86162f1cf795f + .quad 0x118c861926ee57f2 + .quad 0x172124851c063578 + .quad 0x36d12b5dec067fcf + .quad 0x126d279179154557 + .quad 0xd5e48f5cfc783a0a + .quad 0x36bdb6e8df179bac + .quad 0x2ef517885ba82859 + + // 2^156 * 1 * G + + .quad 0x88bd438cd11e0d4a + .quad 0x30cb610d43ccf308 + .quad 0xe09a0e3791937bcc + .quad 0x4559135b25b1720c + .quad 0x1ea436837c6da1e9 + .quad 0xf9c189af1fb9bdbe + .quad 0x303001fcce5dd155 + .quad 0x28a7c99ebc57be52 + .quad 0xb8fd9399e8d19e9d + .quad 0x908191cb962423ff + .quad 0xb2b948d747c742a3 + .quad 0x37f33226d7fb44c4 + + // 2^156 * 2 * G + + .quad 0x0dae8767b55f6e08 + .quad 0x4a43b3b35b203a02 + .quad 0xe3725a6e80af8c79 + .quad 0x0f7a7fd1705fa7a3 + .quad 0x33912553c821b11d + .quad 0x66ed42c241e301df + .quad 0x066fcc11104222fd + .quad 0x307a3b41c192168f + .quad 0x8eeb5d076eb55ce0 + .quad 0x2fc536bfaa0d925a + .quad 0xbe81830fdcb6c6e8 + .quad 0x556c7045827baf52 + + // 2^156 * 3 * G + + .quad 0x8e2b517302e9d8b7 + .quad 0xe3e52269248714e8 + .quad 0xbd4fbd774ca960b5 + .quad 0x6f4b4199c5ecada9 + .quad 0xb94b90022bf44406 + .quad 0xabd4237eff90b534 + .quad 0x7600a960faf86d3a + .quad 0x2f45abdac2322ee3 + .quad 0x61af4912c8ef8a6a + .quad 0xe58fa4fe43fb6e5e + .quad 0xb5afcc5d6fd427cf + .quad 0x6a5393281e1e11eb + + // 2^156 * 4 * G + + .quad 0xf3da5139a5d1ee89 + .quad 0x8145457cff936988 + .quad 0x3f622fed00e188c4 + .quad 0x0f513815db8b5a3d + .quad 0x0fff04fe149443cf + .quad 0x53cac6d9865cddd7 + .quad 0x31385b03531ed1b7 + .quad 0x5846a27cacd1039d + .quad 0x4ff5cdac1eb08717 + .quad 0x67e8b29590f2e9bc + .quad 0x44093b5e237afa99 + .quad 0x0d414bed8708b8b2 + + // 2^156 * 5 * G + + .quad 0xcfb68265fd0e75f6 + .quad 0xe45b3e28bb90e707 + .quad 0x7242a8de9ff92c7a + .quad 0x685b3201933202dd + .quad 0x81886a92294ac9e8 + .quad 0x23162b45d55547be + .quad 0x94cfbc4403715983 + .quad 0x50eb8fdb134bc401 + .quad 0xc0b73ec6d6b330cd + .quad 0x84e44807132faff1 + .quad 0x732b7352c4a5dee1 + .quad 0x5d7c7cf1aa7cd2d2 + + // 2^156 * 6 * G + + .quad 0xaf3b46bf7a4aafa2 + .quad 0xb78705ec4d40d411 + .quad 0x114f0c6aca7c15e3 + .quad 0x3f364faaa9489d4d + .quad 0x33d1013e9b73a562 + .quad 0x925cef5748ec26e1 + .quad 0xa7fce614dd468058 + .quad 0x78b0fad41e9aa438 + .quad 0xbf56a431ed05b488 + .quad 0xa533e66c9c495c7e + .quad 0xe8652baf87f3651a + .quad 0x0241800059d66c33 + + // 2^156 * 7 * G + + .quad 0xceb077fea37a5be4 + .quad 0xdb642f02e5a5eeb7 + .quad 0xc2e6d0c5471270b8 + .quad 0x4771b65538e4529c + .quad 0x28350c7dcf38ea01 + .quad 0x7c6cdbc0b2917ab6 + .quad 0xace7cfbe857082f7 + .quad 0x4d2845aba2d9a1e0 + .quad 0xbb537fe0447070de + .quad 0xcba744436dd557df + .quad 0xd3b5a3473600dbcb + .quad 0x4aeabbe6f9ffd7f8 + + // 2^156 * 8 * G + + .quad 0x4630119e40d8f78c + .quad 0xa01a9bc53c710e11 + .quad 0x486d2b258910dd79 + .quad 0x1e6c47b3db0324e5 + .quad 0x6a2134bcc4a9c8f2 + .quad 0xfbf8fd1c8ace2e37 + .quad 0x000ae3049911a0ba + .quad 0x046e3a616bc89b9e + .quad 0x14e65442f03906be + .quad 0x4a019d54e362be2a + .quad 0x68ccdfec8dc230c7 + .quad 0x7cfb7e3faf6b861c + + // 2^160 * 1 * G + + .quad 0x4637974e8c58aedc + .quad 0xb9ef22fbabf041a4 + .quad 0xe185d956e980718a + .quad 0x2f1b78fab143a8a6 + .quad 0x96eebffb305b2f51 + .quad 0xd3f938ad889596b8 + .quad 0xf0f52dc746d5dd25 + .quad 0x57968290bb3a0095 + .quad 0xf71ab8430a20e101 + .quad 0xf393658d24f0ec47 + .quad 0xcf7509a86ee2eed1 + .quad 0x7dc43e35dc2aa3e1 + + // 2^160 * 2 * G + + .quad 0x85966665887dd9c3 + .quad 0xc90f9b314bb05355 + .quad 0xc6e08df8ef2079b1 + .quad 0x7ef72016758cc12f + .quad 0x5a782a5c273e9718 + .quad 0x3576c6995e4efd94 + .quad 0x0f2ed8051f237d3e + .quad 0x044fb81d82d50a99 + .quad 0xc1df18c5a907e3d9 + .quad 0x57b3371dce4c6359 + .quad 0xca704534b201bb49 + .quad 0x7f79823f9c30dd2e + + // 2^160 * 3 * G + + .quad 0x8334d239a3b513e8 + .quad 0xc13670d4b91fa8d8 + .quad 0x12b54136f590bd33 + .quad 0x0a4e0373d784d9b4 + .quad 0x6a9c1ff068f587ba + .quad 0x0827894e0050c8de + .quad 0x3cbf99557ded5be7 + .quad 0x64a9b0431c06d6f0 + .quad 0x2eb3d6a15b7d2919 + .quad 0xb0b4f6a0d53a8235 + .quad 0x7156ce4389a45d47 + .quad 0x071a7d0ace18346c + + // 2^160 * 4 * G + + .quad 0xd3072daac887ba0b + .quad 0x01262905bfa562ee + .quad 0xcf543002c0ef768b + .quad 0x2c3bcc7146ea7e9c + .quad 0xcc0c355220e14431 + .quad 0x0d65950709b15141 + .quad 0x9af5621b209d5f36 + .quad 0x7c69bcf7617755d3 + .quad 0x07f0d7eb04e8295f + .quad 0x10db18252f50f37d + .quad 0xe951a9a3171798d7 + .quad 0x6f5a9a7322aca51d + + // 2^160 * 5 * G + + .quad 0x8ba1000c2f41c6c5 + .quad 0xc49f79c10cfefb9b + .quad 0x4efa47703cc51c9f + .quad 0x494e21a2e147afca + .quad 0xe729d4eba3d944be + .quad 0x8d9e09408078af9e + .quad 0x4525567a47869c03 + .quad 0x02ab9680ee8d3b24 + .quad 0xefa48a85dde50d9a + .quad 0x219a224e0fb9a249 + .quad 0xfa091f1dd91ef6d9 + .quad 0x6b5d76cbea46bb34 + + // 2^160 * 6 * G + + .quad 0x8857556cec0cd994 + .quad 0x6472dc6f5cd01dba + .quad 0xaf0169148f42b477 + .quad 0x0ae333f685277354 + .quad 0xe0f941171e782522 + .quad 0xf1e6ae74036936d3 + .quad 0x408b3ea2d0fcc746 + .quad 0x16fb869c03dd313e + .quad 0x288e199733b60962 + .quad 0x24fc72b4d8abe133 + .quad 0x4811f7ed0991d03e + .quad 0x3f81e38b8f70d075 + + // 2^160 * 7 * G + + .quad 0x7f910fcc7ed9affe + .quad 0x545cb8a12465874b + .quad 0xa8397ed24b0c4704 + .quad 0x50510fc104f50993 + .quad 0x0adb7f355f17c824 + .quad 0x74b923c3d74299a4 + .quad 0xd57c3e8bcbf8eaf7 + .quad 0x0ad3e2d34cdedc3d + .quad 0x6f0c0fc5336e249d + .quad 0x745ede19c331cfd9 + .quad 0xf2d6fd0009eefe1c + .quad 0x127c158bf0fa1ebe + + // 2^160 * 8 * G + + .quad 0xf6197c422e9879a2 + .quad 0xa44addd452ca3647 + .quad 0x9b413fc14b4eaccb + .quad 0x354ef87d07ef4f68 + .quad 0xdea28fc4ae51b974 + .quad 0x1d9973d3744dfe96 + .quad 0x6240680b873848a8 + .quad 0x4ed82479d167df95 + .quad 0xfee3b52260c5d975 + .quad 0x50352efceb41b0b8 + .quad 0x8808ac30a9f6653c + .quad 0x302d92d20539236d + + // 2^164 * 1 * G + + .quad 0x4c59023fcb3efb7c + .quad 0x6c2fcb99c63c2a94 + .quad 0xba4190e2c3c7e084 + .quad 0x0e545daea51874d9 + .quad 0x957b8b8b0df53c30 + .quad 0x2a1c770a8e60f098 + .quad 0xbbc7a670345796de + .quad 0x22a48f9a90c99bc9 + .quad 0x6b7dc0dc8d3fac58 + .quad 0x5497cd6ce6e42bfd + .quad 0x542f7d1bf400d305 + .quad 0x4159f47f048d9136 + + // 2^164 * 2 * G + + .quad 0x20ad660839e31e32 + .quad 0xf81e1bd58405be50 + .quad 0xf8064056f4dabc69 + .quad 0x14d23dd4ce71b975 + .quad 0x748515a8bbd24839 + .quad 0x77128347afb02b55 + .quad 0x50ba2ac649a2a17f + .quad 0x060525513ad730f1 + .quad 0xf2398e098aa27f82 + .quad 0x6d7982bb89a1b024 + .quad 0xfa694084214dd24c + .quad 0x71ab966fa32301c3 + + // 2^164 * 3 * G + + .quad 0x2dcbd8e34ded02fc + .quad 0x1151f3ec596f22aa + .quad 0xbca255434e0328da + .quad 0x35768fbe92411b22 + .quad 0xb1088a0702809955 + .quad 0x43b273ea0b43c391 + .quad 0xca9b67aefe0686ed + .quad 0x605eecbf8335f4ed + .quad 0x83200a656c340431 + .quad 0x9fcd71678ee59c2f + .quad 0x75d4613f71300f8a + .quad 0x7a912faf60f542f9 + + // 2^164 * 4 * G + + .quad 0xb204585e5edc1a43 + .quad 0x9f0e16ee5897c73c + .quad 0x5b82c0ae4e70483c + .quad 0x624a170e2bddf9be + .quad 0x253f4f8dfa2d5597 + .quad 0x25e49c405477130c + .quad 0x00c052e5996b1102 + .quad 0x33cb966e33bb6c4a + .quad 0x597028047f116909 + .quad 0x828ac41c1e564467 + .quad 0x70417dbde6217387 + .quad 0x721627aefbac4384 + + // 2^164 * 5 * G + + .quad 0x97d03bc38736add5 + .quad 0x2f1422afc532b130 + .quad 0x3aa68a057101bbc4 + .quad 0x4c946cf7e74f9fa7 + .quad 0xfd3097bc410b2f22 + .quad 0xf1a05da7b5cfa844 + .quad 0x61289a1def57ca74 + .quad 0x245ea199bb821902 + .quad 0xaedca66978d477f8 + .quad 0x1898ba3c29117fe1 + .quad 0xcf73f983720cbd58 + .quad 0x67da12e6b8b56351 + + // 2^164 * 6 * G + + .quad 0x7067e187b4bd6e07 + .quad 0x6e8f0203c7d1fe74 + .quad 0x93c6aa2f38c85a30 + .quad 0x76297d1f3d75a78a + .quad 0x2b7ef3d38ec8308c + .quad 0x828fd7ec71eb94ab + .quad 0x807c3b36c5062abd + .quad 0x0cb64cb831a94141 + .quad 0x3030fc33534c6378 + .quad 0xb9635c5ce541e861 + .quad 0x15d9a9bed9b2c728 + .quad 0x49233ea3f3775dcb + + // 2^164 * 7 * G + + .quad 0x629398fa8dbffc3a + .quad 0xe12fe52dd54db455 + .quad 0xf3be11dfdaf25295 + .quad 0x628b140dce5e7b51 + .quad 0x7b3985fe1c9f249b + .quad 0x4fd6b2d5a1233293 + .quad 0xceb345941adf4d62 + .quad 0x6987ff6f542de50c + .quad 0x47e241428f83753c + .quad 0x6317bebc866af997 + .quad 0xdabb5b433d1a9829 + .quad 0x074d8d245287fb2d + + // 2^164 * 8 * G + + .quad 0x8337d9cd440bfc31 + .quad 0x729d2ca1af318fd7 + .quad 0xa040a4a4772c2070 + .quad 0x46002ef03a7349be + .quad 0x481875c6c0e31488 + .quad 0x219429b2e22034b4 + .quad 0x7223c98a31283b65 + .quad 0x3420d60b342277f9 + .quad 0xfaa23adeaffe65f7 + .quad 0x78261ed45be0764c + .quad 0x441c0a1e2f164403 + .quad 0x5aea8e567a87d395 + + // 2^168 * 1 * G + + .quad 0x7813c1a2bca4283d + .quad 0xed62f091a1863dd9 + .quad 0xaec7bcb8c268fa86 + .quad 0x10e5d3b76f1cae4c + .quad 0x2dbc6fb6e4e0f177 + .quad 0x04e1bf29a4bd6a93 + .quad 0x5e1966d4787af6e8 + .quad 0x0edc5f5eb426d060 + .quad 0x5453bfd653da8e67 + .quad 0xe9dc1eec24a9f641 + .quad 0xbf87263b03578a23 + .quad 0x45b46c51361cba72 + + // 2^168 * 2 * G + + .quad 0xa9402abf314f7fa1 + .quad 0xe257f1dc8e8cf450 + .quad 0x1dbbd54b23a8be84 + .quad 0x2177bfa36dcb713b + .quad 0xce9d4ddd8a7fe3e4 + .quad 0xab13645676620e30 + .quad 0x4b594f7bb30e9958 + .quad 0x5c1c0aef321229df + .quad 0x37081bbcfa79db8f + .quad 0x6048811ec25f59b3 + .quad 0x087a76659c832487 + .quad 0x4ae619387d8ab5bb + + // 2^168 * 3 * G + + .quad 0x8ddbf6aa5344a32e + .quad 0x7d88eab4b41b4078 + .quad 0x5eb0eb974a130d60 + .quad 0x1a00d91b17bf3e03 + .quad 0x61117e44985bfb83 + .quad 0xfce0462a71963136 + .quad 0x83ac3448d425904b + .quad 0x75685abe5ba43d64 + .quad 0x6e960933eb61f2b2 + .quad 0x543d0fa8c9ff4952 + .quad 0xdf7275107af66569 + .quad 0x135529b623b0e6aa + + // 2^168 * 4 * G + + .quad 0x18f0dbd7add1d518 + .quad 0x979f7888cfc11f11 + .quad 0x8732e1f07114759b + .quad 0x79b5b81a65ca3a01 + .quad 0xf5c716bce22e83fe + .quad 0xb42beb19e80985c1 + .quad 0xec9da63714254aae + .quad 0x5972ea051590a613 + .quad 0x0fd4ac20dc8f7811 + .quad 0x9a9ad294ac4d4fa8 + .quad 0xc01b2d64b3360434 + .quad 0x4f7e9c95905f3bdb + + // 2^168 * 5 * G + + .quad 0x62674bbc5781302e + .quad 0xd8520f3989addc0f + .quad 0x8c2999ae53fbd9c6 + .quad 0x31993ad92e638e4c + .quad 0x71c8443d355299fe + .quad 0x8bcd3b1cdbebead7 + .quad 0x8092499ef1a49466 + .quad 0x1942eec4a144adc8 + .quad 0x7dac5319ae234992 + .quad 0x2c1b3d910cea3e92 + .quad 0x553ce494253c1122 + .quad 0x2a0a65314ef9ca75 + + // 2^168 * 6 * G + + .quad 0x2db7937ff7f927c2 + .quad 0xdb741f0617d0a635 + .quad 0x5982f3a21155af76 + .quad 0x4cf6e218647c2ded + .quad 0xcf361acd3c1c793a + .quad 0x2f9ebcac5a35bc3b + .quad 0x60e860e9a8cda6ab + .quad 0x055dc39b6dea1a13 + .quad 0xb119227cc28d5bb6 + .quad 0x07e24ebc774dffab + .quad 0xa83c78cee4a32c89 + .quad 0x121a307710aa24b6 + + // 2^168 * 7 * G + + .quad 0xe4db5d5e9f034a97 + .quad 0xe153fc093034bc2d + .quad 0x460546919551d3b1 + .quad 0x333fc76c7a40e52d + .quad 0xd659713ec77483c9 + .quad 0x88bfe077b82b96af + .quad 0x289e28231097bcd3 + .quad 0x527bb94a6ced3a9b + .quad 0x563d992a995b482e + .quad 0x3405d07c6e383801 + .quad 0x485035de2f64d8e5 + .quad 0x6b89069b20a7a9f7 + + // 2^168 * 8 * G + + .quad 0x812aa0416270220d + .quad 0x995a89faf9245b4e + .quad 0xffadc4ce5072ef05 + .quad 0x23bc2103aa73eb73 + .quad 0x4082fa8cb5c7db77 + .quad 0x068686f8c734c155 + .quad 0x29e6c8d9f6e7a57e + .quad 0x0473d308a7639bcf + .quad 0xcaee792603589e05 + .quad 0x2b4b421246dcc492 + .quad 0x02a1ef74e601a94f + .quad 0x102f73bfde04341a + + // 2^172 * 1 * G + + .quad 0xb5a2d50c7ec20d3e + .quad 0xc64bdd6ea0c97263 + .quad 0x56e89052c1ff734d + .quad 0x4929c6f72b2ffaba + .quad 0x358ecba293a36247 + .quad 0xaf8f9862b268fd65 + .quad 0x412f7e9968a01c89 + .quad 0x5786f312cd754524 + .quad 0x337788ffca14032c + .quad 0xf3921028447f1ee3 + .quad 0x8b14071f231bccad + .quad 0x4c817b4bf2344783 + + // 2^172 * 2 * G + + .quad 0x0ff853852871b96e + .quad 0xe13e9fab60c3f1bb + .quad 0xeefd595325344402 + .quad 0x0a37c37075b7744b + .quad 0x413ba057a40b4484 + .quad 0xba4c2e1a4f5f6a43 + .quad 0x614ba0a5aee1d61c + .quad 0x78a1531a8b05dc53 + .quad 0x6cbdf1703ad0562b + .quad 0x8ecf4830c92521a3 + .quad 0xdaebd303fd8424e7 + .quad 0x72ad82a42e5ec56f + + // 2^172 * 3 * G + + .quad 0x3f9e8e35bafb65f6 + .quad 0x39d69ec8f27293a1 + .quad 0x6cb8cd958cf6a3d0 + .quad 0x1734778173adae6d + .quad 0xc368939167024bc3 + .quad 0x8e69d16d49502fda + .quad 0xfcf2ec3ce45f4b29 + .quad 0x065f669ea3b4cbc4 + .quad 0x8a00aec75532db4d + .quad 0xb869a4e443e31bb1 + .quad 0x4a0f8552d3a7f515 + .quad 0x19adeb7c303d7c08 + + // 2^172 * 4 * G + + .quad 0xc720cb6153ead9a3 + .quad 0x55b2c97f512b636e + .quad 0xb1e35b5fd40290b1 + .quad 0x2fd9ccf13b530ee2 + .quad 0x9d05ba7d43c31794 + .quad 0x2470c8ff93322526 + .quad 0x8323dec816197438 + .quad 0x2852709881569b53 + .quad 0x07bd475b47f796b8 + .quad 0xd2c7b013542c8f54 + .quad 0x2dbd23f43b24f87e + .quad 0x6551afd77b0901d6 + + // 2^172 * 5 * G + + .quad 0x4546baaf54aac27f + .quad 0xf6f66fecb2a45a28 + .quad 0x582d1b5b562bcfe8 + .quad 0x44b123f3920f785f + .quad 0x68a24ce3a1d5c9ac + .quad 0xbb77a33d10ff6461 + .quad 0x0f86ce4425d3166e + .quad 0x56507c0950b9623b + .quad 0x1206f0b7d1713e63 + .quad 0x353fe3d915bafc74 + .quad 0x194ceb970ad9d94d + .quad 0x62fadd7cf9d03ad3 + + // 2^172 * 6 * G + + .quad 0xc6b5967b5598a074 + .quad 0x5efe91ce8e493e25 + .quad 0xd4b72c4549280888 + .quad 0x20ef1149a26740c2 + .quad 0x3cd7bc61e7ce4594 + .quad 0xcd6b35a9b7dd267e + .quad 0xa080abc84366ef27 + .quad 0x6ec7c46f59c79711 + .quad 0x2f07ad636f09a8a2 + .quad 0x8697e6ce24205e7d + .quad 0xc0aefc05ee35a139 + .quad 0x15e80958b5f9d897 + + // 2^172 * 7 * G + + .quad 0x25a5ef7d0c3e235b + .quad 0x6c39c17fbe134ee7 + .quad 0xc774e1342dc5c327 + .quad 0x021354b892021f39 + .quad 0x4dd1ed355bb061c4 + .quad 0x42dc0cef941c0700 + .quad 0x61305dc1fd86340e + .quad 0x56b2cc930e55a443 + .quad 0x1df79da6a6bfc5a2 + .quad 0x02f3a2749fde4369 + .quad 0xb323d9f2cda390a7 + .quad 0x7be0847b8774d363 + + // 2^172 * 8 * G + + .quad 0x8c99cc5a8b3f55c3 + .quad 0x0611d7253fded2a0 + .quad 0xed2995ff36b70a36 + .quad 0x1f699a54d78a2619 + .quad 0x1466f5af5307fa11 + .quad 0x817fcc7ded6c0af2 + .quad 0x0a6de44ec3a4a3fb + .quad 0x74071475bc927d0b + .quad 0xe77292f373e7ea8a + .quad 0x296537d2cb045a31 + .quad 0x1bd0653ed3274fde + .quad 0x2f9a2c4476bd2966 + + // 2^176 * 1 * G + + .quad 0xeb18b9ab7f5745c6 + .quad 0x023a8aee5787c690 + .quad 0xb72712da2df7afa9 + .quad 0x36597d25ea5c013d + .quad 0xa2b4dae0b5511c9a + .quad 0x7ac860292bffff06 + .quad 0x981f375df5504234 + .quad 0x3f6bd725da4ea12d + .quad 0x734d8d7b106058ac + .quad 0xd940579e6fc6905f + .quad 0x6466f8f99202932d + .quad 0x7b7ecc19da60d6d0 + + // 2^176 * 2 * G + + .quad 0x78c2373c695c690d + .quad 0xdd252e660642906e + .quad 0x951d44444ae12bd2 + .quad 0x4235ad7601743956 + .quad 0x6dae4a51a77cfa9b + .quad 0x82263654e7a38650 + .quad 0x09bbffcd8f2d82db + .quad 0x03bedc661bf5caba + .quad 0x6258cb0d078975f5 + .quad 0x492942549189f298 + .quad 0xa0cab423e2e36ee4 + .quad 0x0e7ce2b0cdf066a1 + + // 2^176 * 3 * G + + .quad 0xc494643ac48c85a3 + .quad 0xfd361df43c6139ad + .quad 0x09db17dd3ae94d48 + .quad 0x666e0a5d8fb4674a + .quad 0xfea6fedfd94b70f9 + .quad 0xf130c051c1fcba2d + .quad 0x4882d47e7f2fab89 + .quad 0x615256138aeceeb5 + .quad 0x2abbf64e4870cb0d + .quad 0xcd65bcf0aa458b6b + .quad 0x9abe4eba75e8985d + .quad 0x7f0bc810d514dee4 + + // 2^176 * 4 * G + + .quad 0xb9006ba426f4136f + .quad 0x8d67369e57e03035 + .quad 0xcbc8dfd94f463c28 + .quad 0x0d1f8dbcf8eedbf5 + .quad 0x83ac9dad737213a0 + .quad 0x9ff6f8ba2ef72e98 + .quad 0x311e2edd43ec6957 + .quad 0x1d3a907ddec5ab75 + .quad 0xba1693313ed081dc + .quad 0x29329fad851b3480 + .quad 0x0128013c030321cb + .quad 0x00011b44a31bfde3 + + // 2^176 * 5 * G + + .quad 0x3fdfa06c3fc66c0c + .quad 0x5d40e38e4dd60dd2 + .quad 0x7ae38b38268e4d71 + .quad 0x3ac48d916e8357e1 + .quad 0x16561f696a0aa75c + .quad 0xc1bf725c5852bd6a + .quad 0x11a8dd7f9a7966ad + .quad 0x63d988a2d2851026 + .quad 0x00120753afbd232e + .quad 0xe92bceb8fdd8f683 + .quad 0xf81669b384e72b91 + .quad 0x33fad52b2368a066 + + // 2^176 * 6 * G + + .quad 0x540649c6c5e41e16 + .quad 0x0af86430333f7735 + .quad 0xb2acfcd2f305e746 + .quad 0x16c0f429a256dca7 + .quad 0x8d2cc8d0c422cfe8 + .quad 0x072b4f7b05a13acb + .quad 0xa3feb6e6ecf6a56f + .quad 0x3cc355ccb90a71e2 + .quad 0xe9b69443903e9131 + .quad 0xb8a494cb7a5637ce + .quad 0xc87cd1a4baba9244 + .quad 0x631eaf426bae7568 + + // 2^176 * 7 * G + + .quad 0xb3e90410da66fe9f + .quad 0x85dd4b526c16e5a6 + .quad 0xbc3d97611ef9bf83 + .quad 0x5599648b1ea919b5 + .quad 0x47d975b9a3700de8 + .quad 0x7280c5fbe2f80552 + .quad 0x53658f2732e45de1 + .quad 0x431f2c7f665f80b5 + .quad 0xd6026344858f7b19 + .quad 0x14ab352fa1ea514a + .quad 0x8900441a2090a9d7 + .quad 0x7b04715f91253b26 + + // 2^176 * 8 * G + + .quad 0x83edbd28acf6ae43 + .quad 0x86357c8b7d5c7ab4 + .quad 0xc0404769b7eb2c44 + .quad 0x59b37bf5c2f6583f + .quad 0xb376c280c4e6bac6 + .quad 0x970ed3dd6d1d9b0b + .quad 0xb09a9558450bf944 + .quad 0x48d0acfa57cde223 + .quad 0xb60f26e47dabe671 + .quad 0xf1d1a197622f3a37 + .quad 0x4208ce7ee9960394 + .quad 0x16234191336d3bdb + + // 2^180 * 1 * G + + .quad 0xf19aeac733a63aef + .quad 0x2c7fba5d4442454e + .quad 0x5da87aa04795e441 + .quad 0x413051e1a4e0b0f5 + .quad 0x852dd1fd3d578bbe + .quad 0x2b65ce72c3286108 + .quad 0x658c07f4eace2273 + .quad 0x0933f804ec38ab40 + .quad 0xa7ab69798d496476 + .quad 0x8121aadefcb5abc8 + .quad 0xa5dc12ef7b539472 + .quad 0x07fd47065e45351a + + // 2^180 * 2 * G + + .quad 0xc8583c3d258d2bcd + .quad 0x17029a4daf60b73f + .quad 0xfa0fc9d6416a3781 + .quad 0x1c1e5fba38b3fb23 + .quad 0x304211559ae8e7c3 + .quad 0xf281b229944882a5 + .quad 0x8a13ac2e378250e4 + .quad 0x014afa0954ba48f4 + .quad 0xcb3197001bb3666c + .quad 0x330060524bffecb9 + .quad 0x293711991a88233c + .quad 0x291884363d4ed364 + + // 2^180 * 3 * G + + .quad 0x033c6805dc4babfa + .quad 0x2c15bf5e5596ecc1 + .quad 0x1bc70624b59b1d3b + .quad 0x3ede9850a19f0ec5 + .quad 0xfb9d37c3bc1ab6eb + .quad 0x02be14534d57a240 + .quad 0xf4d73415f8a5e1f6 + .quad 0x5964f4300ccc8188 + .quad 0xe44a23152d096800 + .quad 0x5c08c55970866996 + .quad 0xdf2db60a46affb6e + .quad 0x579155c1f856fd89 + + // 2^180 * 4 * G + + .quad 0x96324edd12e0c9ef + .quad 0x468b878df2420297 + .quad 0x199a3776a4f573be + .quad 0x1e7fbcf18e91e92a + .quad 0xb5f16b630817e7a6 + .quad 0x808c69233c351026 + .quad 0x324a983b54cef201 + .quad 0x53c092084a485345 + .quad 0xd2d41481f1cbafbf + .quad 0x231d2db6716174e5 + .quad 0x0b7d7656e2a55c98 + .quad 0x3e955cd82aa495f6 + + // 2^180 * 5 * G + + .quad 0xe48f535e3ed15433 + .quad 0xd075692a0d7270a3 + .quad 0x40fbd21daade6387 + .quad 0x14264887cf4495f5 + .quad 0xab39f3ef61bb3a3f + .quad 0x8eb400652eb9193e + .quad 0xb5de6ecc38c11f74 + .quad 0x654d7e9626f3c49f + .quad 0xe564cfdd5c7d2ceb + .quad 0x82eeafded737ccb9 + .quad 0x6107db62d1f9b0ab + .quad 0x0b6baac3b4358dbb + + // 2^180 * 6 * G + + .quad 0x7ae62bcb8622fe98 + .quad 0x47762256ceb891af + .quad 0x1a5a92bcf2e406b4 + .quad 0x7d29401784e41501 + .quad 0x204abad63700a93b + .quad 0xbe0023d3da779373 + .quad 0xd85f0346633ab709 + .quad 0x00496dc490820412 + .quad 0x1c74b88dc27e6360 + .quad 0x074854268d14850c + .quad 0xa145fb7b3e0dcb30 + .quad 0x10843f1b43803b23 + + // 2^180 * 7 * G + + .quad 0xc5f90455376276dd + .quad 0xce59158dd7645cd9 + .quad 0x92f65d511d366b39 + .quad 0x11574b6e526996c4 + .quad 0xd56f672de324689b + .quad 0xd1da8aedb394a981 + .quad 0xdd7b58fe9168cfed + .quad 0x7ce246cd4d56c1e8 + .quad 0xb8f4308e7f80be53 + .quad 0x5f3cb8cb34a9d397 + .quad 0x18a961bd33cc2b2c + .quad 0x710045fb3a9af671 + + // 2^180 * 8 * G + + .quad 0x73f93d36101b95eb + .quad 0xfaef33794f6f4486 + .quad 0x5651735f8f15e562 + .quad 0x7fa3f19058b40da1 + .quad 0xa03fc862059d699e + .quad 0x2370cfa19a619e69 + .quad 0xc4fe3b122f823deb + .quad 0x1d1b056fa7f0844e + .quad 0x1bc64631e56bf61f + .quad 0xd379ab106e5382a3 + .quad 0x4d58c57e0540168d + .quad 0x566256628442d8e4 + + // 2^184 * 1 * G + + .quad 0xb9e499def6267ff6 + .quad 0x7772ca7b742c0843 + .quad 0x23a0153fe9a4f2b1 + .quad 0x2cdfdfecd5d05006 + .quad 0xdd499cd61ff38640 + .quad 0x29cd9bc3063625a0 + .quad 0x51e2d8023dd73dc3 + .quad 0x4a25707a203b9231 + .quad 0x2ab7668a53f6ed6a + .quad 0x304242581dd170a1 + .quad 0x4000144c3ae20161 + .quad 0x5721896d248e49fc + + // 2^184 * 2 * G + + .quad 0x0b6e5517fd181bae + .quad 0x9022629f2bb963b4 + .quad 0x5509bce932064625 + .quad 0x578edd74f63c13da + .quad 0x285d5091a1d0da4e + .quad 0x4baa6fa7b5fe3e08 + .quad 0x63e5177ce19393b3 + .quad 0x03c935afc4b030fd + .quad 0x997276c6492b0c3d + .quad 0x47ccc2c4dfe205fc + .quad 0xdcd29b84dd623a3c + .quad 0x3ec2ab590288c7a2 + + // 2^184 * 3 * G + + .quad 0xa1a0d27be4d87bb9 + .quad 0xa98b4deb61391aed + .quad 0x99a0ddd073cb9b83 + .quad 0x2dd5c25a200fcace + .quad 0xa7213a09ae32d1cb + .quad 0x0f2b87df40f5c2d5 + .quad 0x0baea4c6e81eab29 + .quad 0x0e1bf66c6adbac5e + .quad 0xe2abd5e9792c887e + .quad 0x1a020018cb926d5d + .quad 0xbfba69cdbaae5f1e + .quad 0x730548b35ae88f5f + + // 2^184 * 4 * G + + .quad 0xc43551a3cba8b8ee + .quad 0x65a26f1db2115f16 + .quad 0x760f4f52ab8c3850 + .quad 0x3043443b411db8ca + .quad 0x805b094ba1d6e334 + .quad 0xbf3ef17709353f19 + .quad 0x423f06cb0622702b + .quad 0x585a2277d87845dd + .quad 0xa18a5f8233d48962 + .quad 0x6698c4b5ec78257f + .quad 0xa78e6fa5373e41ff + .quad 0x7656278950ef981f + + // 2^184 * 5 * G + + .quad 0x38c3cf59d51fc8c0 + .quad 0x9bedd2fd0506b6f2 + .quad 0x26bf109fab570e8f + .quad 0x3f4160a8c1b846a6 + .quad 0xe17073a3ea86cf9d + .quad 0x3a8cfbb707155fdc + .quad 0x4853e7fc31838a8e + .quad 0x28bbf484b613f616 + .quad 0xf2612f5c6f136c7c + .quad 0xafead107f6dd11be + .quad 0x527e9ad213de6f33 + .quad 0x1e79cb358188f75d + + // 2^184 * 6 * G + + .quad 0x013436c3eef7e3f1 + .quad 0x828b6a7ffe9e10f8 + .quad 0x7ff908e5bcf9defc + .quad 0x65d7951b3a3b3831 + .quad 0x77e953d8f5e08181 + .quad 0x84a50c44299dded9 + .quad 0xdc6c2d0c864525e5 + .quad 0x478ab52d39d1f2f4 + .quad 0x66a6a4d39252d159 + .quad 0xe5dde1bc871ac807 + .quad 0xb82c6b40a6c1c96f + .quad 0x16d87a411a212214 + + // 2^184 * 7 * G + + .quad 0xb3bd7e5a42066215 + .quad 0x879be3cd0c5a24c1 + .quad 0x57c05db1d6f994b7 + .quad 0x28f87c8165f38ca6 + .quad 0xfba4d5e2d54e0583 + .quad 0xe21fafd72ebd99fa + .quad 0x497ac2736ee9778f + .quad 0x1f990b577a5a6dde + .quad 0xa3344ead1be8f7d6 + .quad 0x7d1e50ebacea798f + .quad 0x77c6569e520de052 + .quad 0x45882fe1534d6d3e + + // 2^184 * 8 * G + + .quad 0x6669345d757983d6 + .quad 0x62b6ed1117aa11a6 + .quad 0x7ddd1857985e128f + .quad 0x688fe5b8f626f6dd + .quad 0xd8ac9929943c6fe4 + .quad 0xb5f9f161a38392a2 + .quad 0x2699db13bec89af3 + .quad 0x7dcf843ce405f074 + .quad 0x6c90d6484a4732c0 + .quad 0xd52143fdca563299 + .quad 0xb3be28c3915dc6e1 + .quad 0x6739687e7327191b + + // 2^188 * 1 * G + + .quad 0x9f65c5ea200814cf + .quad 0x840536e169a31740 + .quad 0x8b0ed13925c8b4ad + .quad 0x0080dbafe936361d + .quad 0x8ce5aad0c9cb971f + .quad 0x1156aaa99fd54a29 + .quad 0x41f7247015af9b78 + .quad 0x1fe8cca8420f49aa + .quad 0x72a1848f3c0cc82a + .quad 0x38c560c2877c9e54 + .quad 0x5004e228ce554140 + .quad 0x042418a103429d71 + + // 2^188 * 2 * G + + .quad 0x899dea51abf3ff5f + .quad 0x9b93a8672fc2d8ba + .quad 0x2c38cb97be6ebd5c + .quad 0x114d578497263b5d + .quad 0x58e84c6f20816247 + .quad 0x8db2b2b6e36fd793 + .quad 0x977182561d484d85 + .quad 0x0822024f8632abd7 + .quad 0xb301bb7c6b1beca3 + .quad 0x55393f6dc6eb1375 + .quad 0x910d281097b6e4eb + .quad 0x1ad4548d9d479ea3 + + // 2^188 * 3 * G + + .quad 0xcd5a7da0389a48fd + .quad 0xb38fa4aa9a78371e + .quad 0xc6d9761b2cdb8e6c + .quad 0x35cf51dbc97e1443 + .quad 0xa06fe66d0fe9fed3 + .quad 0xa8733a401c587909 + .quad 0x30d14d800df98953 + .quad 0x41ce5876c7b30258 + .quad 0x59ac3bc5d670c022 + .quad 0xeae67c109b119406 + .quad 0x9798bdf0b3782fda + .quad 0x651e3201fd074092 + + // 2^188 * 4 * G + + .quad 0xd63d8483ef30c5cf + .quad 0x4cd4b4962361cc0c + .quad 0xee90e500a48426ac + .quad 0x0af51d7d18c14eeb + .quad 0xa57ba4a01efcae9e + .quad 0x769f4beedc308a94 + .quad 0xd1f10eeb3603cb2e + .quad 0x4099ce5e7e441278 + .quad 0x1ac98e4f8a5121e9 + .quad 0x7dae9544dbfa2fe0 + .quad 0x8320aa0dd6430df9 + .quad 0x667282652c4a2fb5 + + // 2^188 * 5 * G + + .quad 0x874621f4d86bc9ab + .quad 0xb54c7bbe56fe6fea + .quad 0x077a24257fadc22c + .quad 0x1ab53be419b90d39 + .quad 0xada8b6e02946db23 + .quad 0x1c0ce51a7b253ab7 + .quad 0x8448c85a66dd485b + .quad 0x7f1fc025d0675adf + .quad 0xd8ee1b18319ea6aa + .quad 0x004d88083a21f0da + .quad 0x3bd6aa1d883a4f4b + .quad 0x4db9a3a6dfd9fd14 + + // 2^188 * 6 * G + + .quad 0x8ce7b23bb99c0755 + .quad 0x35c5d6edc4f50f7a + .quad 0x7e1e2ed2ed9b50c3 + .quad 0x36305f16e8934da1 + .quad 0xd95b00bbcbb77c68 + .quad 0xddbc846a91f17849 + .quad 0x7cf700aebe28d9b3 + .quad 0x5ce1285c85d31f3e + .quad 0x31b6972d98b0bde8 + .quad 0x7d920706aca6de5b + .quad 0xe67310f8908a659f + .quad 0x50fac2a6efdf0235 + + // 2^188 * 7 * G + + .quad 0xf3d3a9f35b880f5a + .quad 0xedec050cdb03e7c2 + .quad 0xa896981ff9f0b1a2 + .quad 0x49a4ae2bac5e34a4 + .quad 0x295b1c86f6f449bc + .quad 0x51b2e84a1f0ab4dd + .quad 0xc001cb30aa8e551d + .quad 0x6a28d35944f43662 + .quad 0x28bb12ee04a740e0 + .quad 0x14313bbd9bce8174 + .quad 0x72f5b5e4e8c10c40 + .quad 0x7cbfb19936adcd5b + + // 2^188 * 8 * G + + .quad 0xa311ddc26b89792d + .quad 0x1b30b4c6da512664 + .quad 0x0ca77b4ccf150859 + .quad 0x1de443df1b009408 + .quad 0x8e793a7acc36e6e0 + .quad 0xf9fab7a37d586eed + .quad 0x3a4f9692bae1f4e4 + .quad 0x1c14b03eff5f447e + .quad 0x19647bd114a85291 + .quad 0x57b76cb21034d3af + .quad 0x6329db440f9d6dfa + .quad 0x5ef43e586a571493 + + // 2^192 * 1 * G + + .quad 0xef782014385675a6 + .quad 0xa2649f30aafda9e8 + .quad 0x4cd1eb505cdfa8cb + .quad 0x46115aba1d4dc0b3 + .quad 0xa66dcc9dc80c1ac0 + .quad 0x97a05cf41b38a436 + .quad 0xa7ebf3be95dbd7c6 + .quad 0x7da0b8f68d7e7dab + .quad 0xd40f1953c3b5da76 + .quad 0x1dac6f7321119e9b + .quad 0x03cc6021feb25960 + .quad 0x5a5f887e83674b4b + + // 2^192 * 2 * G + + .quad 0x8f6301cf70a13d11 + .quad 0xcfceb815350dd0c4 + .quad 0xf70297d4a4bca47e + .quad 0x3669b656e44d1434 + .quad 0x9e9628d3a0a643b9 + .quad 0xb5c3cb00e6c32064 + .quad 0x9b5302897c2dec32 + .quad 0x43e37ae2d5d1c70c + .quad 0x387e3f06eda6e133 + .quad 0x67301d5199a13ac0 + .quad 0xbd5ad8f836263811 + .quad 0x6a21e6cd4fd5e9be + + // 2^192 * 3 * G + + .quad 0xf1c6170a3046e65f + .quad 0x58712a2a00d23524 + .quad 0x69dbbd3c8c82b755 + .quad 0x586bf9f1a195ff57 + .quad 0xef4129126699b2e3 + .quad 0x71d30847708d1301 + .quad 0x325432d01182b0bd + .quad 0x45371b07001e8b36 + .quad 0xa6db088d5ef8790b + .quad 0x5278f0dc610937e5 + .quad 0xac0349d261a16eb8 + .quad 0x0eafb03790e52179 + + // 2^192 * 4 * G + + .quad 0x960555c13748042f + .quad 0x219a41e6820baa11 + .quad 0x1c81f73873486d0c + .quad 0x309acc675a02c661 + .quad 0x5140805e0f75ae1d + .quad 0xec02fbe32662cc30 + .quad 0x2cebdf1eea92396d + .quad 0x44ae3344c5435bb3 + .quad 0x9cf289b9bba543ee + .quad 0xf3760e9d5ac97142 + .quad 0x1d82e5c64f9360aa + .quad 0x62d5221b7f94678f + + // 2^192 * 5 * G + + .quad 0x524c299c18d0936d + .quad 0xc86bb56c8a0c1a0c + .quad 0xa375052edb4a8631 + .quad 0x5c0efde4bc754562 + .quad 0x7585d4263af77a3c + .quad 0xdfae7b11fee9144d + .quad 0xa506708059f7193d + .quad 0x14f29a5383922037 + .quad 0xdf717edc25b2d7f5 + .quad 0x21f970db99b53040 + .quad 0xda9234b7c3ed4c62 + .quad 0x5e72365c7bee093e + + // 2^192 * 6 * G + + .quad 0x575bfc074571217f + .quad 0x3779675d0694d95b + .quad 0x9a0a37bbf4191e33 + .quad 0x77f1104c47b4eabc + .quad 0x7d9339062f08b33e + .quad 0x5b9659e5df9f32be + .quad 0xacff3dad1f9ebdfd + .quad 0x70b20555cb7349b7 + .quad 0xbe5113c555112c4c + .quad 0x6688423a9a881fcd + .quad 0x446677855e503b47 + .quad 0x0e34398f4a06404a + + // 2^192 * 7 * G + + .quad 0xb67d22d93ecebde8 + .quad 0x09b3e84127822f07 + .quad 0x743fa61fb05b6d8d + .quad 0x5e5405368a362372 + .quad 0x18930b093e4b1928 + .quad 0x7de3e10e73f3f640 + .quad 0xf43217da73395d6f + .quad 0x6f8aded6ca379c3e + .quad 0xe340123dfdb7b29a + .quad 0x487b97e1a21ab291 + .quad 0xf9967d02fde6949e + .quad 0x780de72ec8d3de97 + + // 2^192 * 8 * G + + .quad 0x0ae28545089ae7bc + .quad 0x388ddecf1c7f4d06 + .quad 0x38ac15510a4811b8 + .quad 0x0eb28bf671928ce4 + .quad 0x671feaf300f42772 + .quad 0x8f72eb2a2a8c41aa + .quad 0x29a17fd797373292 + .quad 0x1defc6ad32b587a6 + .quad 0xaf5bbe1aef5195a7 + .quad 0x148c1277917b15ed + .quad 0x2991f7fb7ae5da2e + .quad 0x467d201bf8dd2867 + + // 2^196 * 1 * G + + .quad 0x7906ee72f7bd2e6b + .quad 0x05d270d6109abf4e + .quad 0x8d5cfe45b941a8a4 + .quad 0x44c218671c974287 + .quad 0x745f9d56296bc318 + .quad 0x993580d4d8152e65 + .quad 0xb0e5b13f5839e9ce + .quad 0x51fc2b28d43921c0 + .quad 0x1b8fd11795e2a98c + .quad 0x1c4e5ee12b6b6291 + .quad 0x5b30e7107424b572 + .quad 0x6e6b9de84c4f4ac6 + + // 2^196 * 2 * G + + .quad 0xdff25fce4b1de151 + .quad 0xd841c0c7e11c4025 + .quad 0x2554b3c854749c87 + .quad 0x2d292459908e0df9 + .quad 0x6b7c5f10f80cb088 + .quad 0x736b54dc56e42151 + .quad 0xc2b620a5c6ef99c4 + .quad 0x5f4c802cc3a06f42 + .quad 0x9b65c8f17d0752da + .quad 0x881ce338c77ee800 + .quad 0xc3b514f05b62f9e3 + .quad 0x66ed5dd5bec10d48 + + // 2^196 * 3 * G + + .quad 0x7d38a1c20bb2089d + .quad 0x808334e196ccd412 + .quad 0xc4a70b8c6c97d313 + .quad 0x2eacf8bc03007f20 + .quad 0xf0adf3c9cbca047d + .quad 0x81c3b2cbf4552f6b + .quad 0xcfda112d44735f93 + .quad 0x1f23a0c77e20048c + .quad 0xf235467be5bc1570 + .quad 0x03d2d9020dbab38c + .quad 0x27529aa2fcf9e09e + .quad 0x0840bef29d34bc50 + + // 2^196 * 4 * G + + .quad 0x796dfb35dc10b287 + .quad 0x27176bcd5c7ff29d + .quad 0x7f3d43e8c7b24905 + .quad 0x0304f5a191c54276 + .quad 0xcd54e06b7f37e4eb + .quad 0x8cc15f87f5e96cca + .quad 0xb8248bb0d3597dce + .quad 0x246affa06074400c + .quad 0x37d88e68fbe45321 + .quad 0x86097548c0d75032 + .quad 0x4e9b13ef894a0d35 + .quad 0x25a83cac5753d325 + + // 2^196 * 5 * G + + .quad 0x10222f48eed8165e + .quad 0x623fc1234b8bcf3a + .quad 0x1e145c09c221e8f0 + .quad 0x7ccfa59fca782630 + .quad 0x9f0f66293952b6e2 + .quad 0x33db5e0e0934267b + .quad 0xff45252bd609fedc + .quad 0x06be10f5c506e0c9 + .quad 0x1a9615a9b62a345f + .quad 0x22050c564a52fecc + .quad 0xa7a2788528bc0dfe + .quad 0x5e82770a1a1ee71d + + // 2^196 * 6 * G + + .quad 0x35425183ad896a5c + .quad 0xe8673afbe78d52f6 + .quad 0x2c66f25f92a35f64 + .quad 0x09d04f3b3b86b102 + .quad 0xe802e80a42339c74 + .quad 0x34175166a7fffae5 + .quad 0x34865d1f1c408cae + .quad 0x2cca982c605bc5ee + .quad 0xfd2d5d35197dbe6e + .quad 0x207c2eea8be4ffa3 + .quad 0x2613d8db325ae918 + .quad 0x7a325d1727741d3e + + // 2^196 * 7 * G + + .quad 0xd036b9bbd16dfde2 + .quad 0xa2055757c497a829 + .quad 0x8e6cc966a7f12667 + .quad 0x4d3b1a791239c180 + .quad 0xecd27d017e2a076a + .quad 0xd788689f1636495e + .quad 0x52a61af0919233e5 + .quad 0x2a479df17bb1ae64 + .quad 0x9e5eee8e33db2710 + .quad 0x189854ded6c43ca5 + .quad 0xa41c22c592718138 + .quad 0x27ad5538a43a5e9b + + // 2^196 * 8 * G + + .quad 0x2746dd4b15350d61 + .quad 0xd03fcbc8ee9521b7 + .quad 0xe86e365a138672ca + .quad 0x510e987f7e7d89e2 + .quad 0xcb5a7d638e47077c + .quad 0x8db7536120a1c059 + .quad 0x549e1e4d8bedfdcc + .quad 0x080153b7503b179d + .quad 0xdda69d930a3ed3e3 + .quad 0x3d386ef1cd60a722 + .quad 0xc817ad58bdaa4ee6 + .quad 0x23be8d554fe7372a + + // 2^200 * 1 * G + + .quad 0x95fe919a74ef4fad + .quad 0x3a827becf6a308a2 + .quad 0x964e01d309a47b01 + .quad 0x71c43c4f5ba3c797 + .quad 0xbc1ef4bd567ae7a9 + .quad 0x3f624cb2d64498bd + .quad 0xe41064d22c1f4ec8 + .quad 0x2ef9c5a5ba384001 + .quad 0xb6fd6df6fa9e74cd + .quad 0xf18278bce4af267a + .quad 0x8255b3d0f1ef990e + .quad 0x5a758ca390c5f293 + + // 2^200 * 2 * G + + .quad 0xa2b72710d9462495 + .quad 0x3aa8c6d2d57d5003 + .quad 0xe3d400bfa0b487ca + .quad 0x2dbae244b3eb72ec + .quad 0x8ce0918b1d61dc94 + .quad 0x8ded36469a813066 + .quad 0xd4e6a829afe8aad3 + .quad 0x0a738027f639d43f + .quad 0x980f4a2f57ffe1cc + .quad 0x00670d0de1839843 + .quad 0x105c3f4a49fb15fd + .quad 0x2698ca635126a69c + + // 2^200 * 3 * G + + .quad 0xe765318832b0ba78 + .quad 0x381831f7925cff8b + .quad 0x08a81b91a0291fcc + .quad 0x1fb43dcc49caeb07 + .quad 0x2e3d702f5e3dd90e + .quad 0x9e3f0918e4d25386 + .quad 0x5e773ef6024da96a + .quad 0x3c004b0c4afa3332 + .quad 0x9aa946ac06f4b82b + .quad 0x1ca284a5a806c4f3 + .quad 0x3ed3265fc6cd4787 + .quad 0x6b43fd01cd1fd217 + + // 2^200 * 4 * G + + .quad 0xc7a75d4b4697c544 + .quad 0x15fdf848df0fffbf + .quad 0x2868b9ebaa46785a + .quad 0x5a68d7105b52f714 + .quad 0xb5c742583e760ef3 + .quad 0x75dc52b9ee0ab990 + .quad 0xbf1427c2072b923f + .quad 0x73420b2d6ff0d9f0 + .quad 0xaf2cf6cb9e851e06 + .quad 0x8f593913c62238c4 + .quad 0xda8ab89699fbf373 + .quad 0x3db5632fea34bc9e + + // 2^200 * 5 * G + + .quad 0xf46eee2bf75dd9d8 + .quad 0x0d17b1f6396759a5 + .quad 0x1bf2d131499e7273 + .quad 0x04321adf49d75f13 + .quad 0x2e4990b1829825d5 + .quad 0xedeaeb873e9a8991 + .quad 0xeef03d394c704af8 + .quad 0x59197ea495df2b0e + .quad 0x04e16019e4e55aae + .quad 0xe77b437a7e2f92e9 + .quad 0xc7ce2dc16f159aa4 + .quad 0x45eafdc1f4d70cc0 + + // 2^200 * 6 * G + + .quad 0x698401858045d72b + .quad 0x4c22faa2cf2f0651 + .quad 0x941a36656b222dc6 + .quad 0x5a5eebc80362dade + .quad 0xb60e4624cfccb1ed + .quad 0x59dbc292bd5c0395 + .quad 0x31a09d1ddc0481c9 + .quad 0x3f73ceea5d56d940 + .quad 0xb7a7bfd10a4e8dc6 + .quad 0xbe57007e44c9b339 + .quad 0x60c1207f1557aefa + .quad 0x26058891266218db + + // 2^200 * 7 * G + + .quad 0x59f704a68360ff04 + .quad 0xc3d93fde7661e6f4 + .quad 0x831b2a7312873551 + .quad 0x54ad0c2e4e615d57 + .quad 0x4c818e3cc676e542 + .quad 0x5e422c9303ceccad + .quad 0xec07cccab4129f08 + .quad 0x0dedfa10b24443b8 + .quad 0xee3b67d5b82b522a + .quad 0x36f163469fa5c1eb + .quad 0xa5b4d2f26ec19fd3 + .quad 0x62ecb2baa77a9408 + + // 2^200 * 8 * G + + .quad 0xe5ed795261152b3d + .quad 0x4962357d0eddd7d1 + .quad 0x7482c8d0b96b4c71 + .quad 0x2e59f919a966d8be + .quad 0x92072836afb62874 + .quad 0x5fcd5e8579e104a5 + .quad 0x5aad01adc630a14a + .quad 0x61913d5075663f98 + .quad 0x0dc62d361a3231da + .quad 0xfa47583294200270 + .quad 0x02d801513f9594ce + .quad 0x3ddbc2a131c05d5c + + // 2^204 * 1 * G + + .quad 0x3f50a50a4ffb81ef + .quad 0xb1e035093bf420bf + .quad 0x9baa8e1cc6aa2cd0 + .quad 0x32239861fa237a40 + .quad 0xfb735ac2004a35d1 + .quad 0x31de0f433a6607c3 + .quad 0x7b8591bfc528d599 + .quad 0x55be9a25f5bb050c + .quad 0x0d005acd33db3dbf + .quad 0x0111b37c80ac35e2 + .quad 0x4892d66c6f88ebeb + .quad 0x770eadb16508fbcd + + // 2^204 * 2 * G + + .quad 0x8451f9e05e4e89dd + .quad 0xc06302ffbc793937 + .quad 0x5d22749556a6495c + .quad 0x09a6755ca05603fb + .quad 0xf1d3b681a05071b9 + .quad 0x2207659a3592ff3a + .quad 0x5f0169297881e40e + .quad 0x16bedd0e86ba374e + .quad 0x5ecccc4f2c2737b5 + .quad 0x43b79e0c2dccb703 + .quad 0x33e008bc4ec43df3 + .quad 0x06c1b840f07566c0 + + // 2^204 * 3 * G + + .quad 0x7688a5c6a388f877 + .quad 0x02a96c14deb2b6ac + .quad 0x64c9f3431b8c2af8 + .quad 0x3628435554a1eed6 + .quad 0x69ee9e7f9b02805c + .quad 0xcbff828a547d1640 + .quad 0x3d93a869b2430968 + .quad 0x46b7b8cd3fe26972 + .quad 0xe9812086fe7eebe0 + .quad 0x4cba6be72f515437 + .quad 0x1d04168b516efae9 + .quad 0x5ea1391043982cb9 + + // 2^204 * 4 * G + + .quad 0x49125c9cf4702ee1 + .quad 0x4520b71f8b25b32d + .quad 0x33193026501fef7e + .quad 0x656d8997c8d2eb2b + .quad 0x6f2b3be4d5d3b002 + .quad 0xafec33d96a09c880 + .quad 0x035f73a4a8bcc4cc + .quad 0x22c5b9284662198b + .quad 0xcb58c8fe433d8939 + .quad 0x89a0cb2e6a8d7e50 + .quad 0x79ca955309fbbe5a + .quad 0x0c626616cd7fc106 + + // 2^204 * 5 * G + + .quad 0x1ffeb80a4879b61f + .quad 0x6396726e4ada21ed + .quad 0x33c7b093368025ba + .quad 0x471aa0c6f3c31788 + .quad 0x8fdfc379fbf454b1 + .quad 0x45a5a970f1a4b771 + .quad 0xac921ef7bad35915 + .quad 0x42d088dca81c2192 + .quad 0x8fda0f37a0165199 + .quad 0x0adadb77c8a0e343 + .quad 0x20fbfdfcc875e820 + .quad 0x1cf2bea80c2206e7 + + // 2^204 * 6 * G + + .quad 0xc2ddf1deb36202ac + .quad 0x92a5fe09d2e27aa5 + .quad 0x7d1648f6fc09f1d3 + .quad 0x74c2cc0513bc4959 + .quad 0x982d6e1a02c0412f + .quad 0x90fa4c83db58e8fe + .quad 0x01c2f5bcdcb18bc0 + .quad 0x686e0c90216abc66 + .quad 0x1fadbadba54395a7 + .quad 0xb41a02a0ae0da66a + .quad 0xbf19f598bba37c07 + .quad 0x6a12b8acde48430d + + // 2^204 * 7 * G + + .quad 0xf8daea1f39d495d9 + .quad 0x592c190e525f1dfc + .quad 0xdb8cbd04c9991d1b + .quad 0x11f7fda3d88f0cb7 + .quad 0x793bdd801aaeeb5f + .quad 0x00a2a0aac1518871 + .quad 0xe8a373a31f2136b4 + .quad 0x48aab888fc91ef19 + .quad 0x041f7e925830f40e + .quad 0x002d6ca979661c06 + .quad 0x86dc9ff92b046a2e + .quad 0x760360928b0493d1 + + // 2^204 * 8 * G + + .quad 0x21bb41c6120cf9c6 + .quad 0xeab2aa12decda59b + .quad 0xc1a72d020aa48b34 + .quad 0x215d4d27e87d3b68 + .quad 0xb43108e5695a0b05 + .quad 0x6cb00ee8ad37a38b + .quad 0x5edad6eea3537381 + .quad 0x3f2602d4b6dc3224 + .quad 0xc8b247b65bcaf19c + .quad 0x49779dc3b1b2c652 + .quad 0x89a180bbd5ece2e2 + .quad 0x13f098a3cec8e039 + + // 2^208 * 1 * G + + .quad 0x9adc0ff9ce5ec54b + .quad 0x039c2a6b8c2f130d + .quad 0x028007c7f0f89515 + .quad 0x78968314ac04b36b + .quad 0xf3aa57a22796bb14 + .quad 0x883abab79b07da21 + .quad 0xe54be21831a0391c + .quad 0x5ee7fb38d83205f9 + .quad 0x538dfdcb41446a8e + .quad 0xa5acfda9434937f9 + .quad 0x46af908d263c8c78 + .quad 0x61d0633c9bca0d09 + + // 2^208 * 2 * G + + .quad 0x63744935ffdb2566 + .quad 0xc5bd6b89780b68bb + .quad 0x6f1b3280553eec03 + .quad 0x6e965fd847aed7f5 + .quad 0xada328bcf8fc73df + .quad 0xee84695da6f037fc + .quad 0x637fb4db38c2a909 + .quad 0x5b23ac2df8067bdc + .quad 0x9ad2b953ee80527b + .quad 0xe88f19aafade6d8d + .quad 0x0e711704150e82cf + .quad 0x79b9bbb9dd95dedc + + // 2^208 * 3 * G + + .quad 0xebb355406a3126c2 + .quad 0xd26383a868c8c393 + .quad 0x6c0c6429e5b97a82 + .quad 0x5065f158c9fd2147 + .quad 0xd1997dae8e9f7374 + .quad 0xa032a2f8cfbb0816 + .quad 0xcd6cba126d445f0a + .quad 0x1ba811460accb834 + .quad 0x708169fb0c429954 + .quad 0xe14600acd76ecf67 + .quad 0x2eaab98a70e645ba + .quad 0x3981f39e58a4faf2 + + // 2^208 * 4 * G + + .quad 0x18fb8a7559230a93 + .quad 0x1d168f6960e6f45d + .quad 0x3a85a94514a93cb5 + .quad 0x38dc083705acd0fd + .quad 0xc845dfa56de66fde + .quad 0xe152a5002c40483a + .quad 0xe9d2e163c7b4f632 + .quad 0x30f4452edcbc1b65 + .quad 0x856d2782c5759740 + .quad 0xfa134569f99cbecc + .quad 0x8844fc73c0ea4e71 + .quad 0x632d9a1a593f2469 + + // 2^208 * 5 * G + + .quad 0xf6bb6b15b807cba6 + .quad 0x1823c7dfbc54f0d7 + .quad 0xbb1d97036e29670b + .quad 0x0b24f48847ed4a57 + .quad 0xbf09fd11ed0c84a7 + .quad 0x63f071810d9f693a + .quad 0x21908c2d57cf8779 + .quad 0x3a5a7df28af64ba2 + .quad 0xdcdad4be511beac7 + .quad 0xa4538075ed26ccf2 + .quad 0xe19cff9f005f9a65 + .quad 0x34fcf74475481f63 + + // 2^208 * 6 * G + + .quad 0xc197e04c789767ca + .quad 0xb8714dcb38d9467d + .quad 0x55de888283f95fa8 + .quad 0x3d3bdc164dfa63f7 + .quad 0xa5bb1dab78cfaa98 + .quad 0x5ceda267190b72f2 + .quad 0x9309c9110a92608e + .quad 0x0119a3042fb374b0 + .quad 0x67a2d89ce8c2177d + .quad 0x669da5f66895d0c1 + .quad 0xf56598e5b282a2b0 + .quad 0x56c088f1ede20a73 + + // 2^208 * 7 * G + + .quad 0x336d3d1110a86e17 + .quad 0xd7f388320b75b2fa + .quad 0xf915337625072988 + .quad 0x09674c6b99108b87 + .quad 0x581b5fac24f38f02 + .quad 0xa90be9febae30cbd + .quad 0x9a2169028acf92f0 + .quad 0x038b7ea48359038f + .quad 0x9f4ef82199316ff8 + .quad 0x2f49d282eaa78d4f + .quad 0x0971a5ab5aef3174 + .quad 0x6e5e31025969eb65 + + // 2^208 * 8 * G + + .quad 0xb16c62f587e593fb + .quad 0x4999eddeca5d3e71 + .quad 0xb491c1e014cc3e6d + .quad 0x08f5114789a8dba8 + .quad 0x3304fb0e63066222 + .quad 0xfb35068987acba3f + .quad 0xbd1924778c1061a3 + .quad 0x3058ad43d1838620 + .quad 0x323c0ffde57663d0 + .quad 0x05c3df38a22ea610 + .quad 0xbdc78abdac994f9a + .quad 0x26549fa4efe3dc99 + + // 2^212 * 1 * G + + .quad 0x738b38d787ce8f89 + .quad 0xb62658e24179a88d + .quad 0x30738c9cf151316d + .quad 0x49128c7f727275c9 + .quad 0x04dbbc17f75396b9 + .quad 0x69e6a2d7d2f86746 + .quad 0xc6409d99f53eabc6 + .quad 0x606175f6332e25d2 + .quad 0x4021370ef540e7dd + .quad 0x0910d6f5a1f1d0a5 + .quad 0x4634aacd5b06b807 + .quad 0x6a39e6356944f235 + + // 2^212 * 2 * G + + .quad 0x96cd5640df90f3e7 + .quad 0x6c3a760edbfa25ea + .quad 0x24f3ef0959e33cc4 + .quad 0x42889e7e530d2e58 + .quad 0x1da1965774049e9d + .quad 0xfbcd6ea198fe352b + .quad 0xb1cbcd50cc5236a6 + .quad 0x1f5ec83d3f9846e2 + .quad 0x8efb23c3328ccb75 + .quad 0xaf42a207dd876ee9 + .quad 0x20fbdadc5dfae796 + .quad 0x241e246b06bf9f51 + + // 2^212 * 3 * G + + .quad 0x29e68e57ad6e98f6 + .quad 0x4c9260c80b462065 + .quad 0x3f00862ea51ebb4b + .quad 0x5bc2c77fb38d9097 + .quad 0x7eaafc9a6280bbb8 + .quad 0x22a70f12f403d809 + .quad 0x31ce40bb1bfc8d20 + .quad 0x2bc65635e8bd53ee + .quad 0xe8d5dc9fa96bad93 + .quad 0xe58fb17dde1947dc + .quad 0x681532ea65185fa3 + .quad 0x1fdd6c3b034a7830 + + // 2^212 * 4 * G + + .quad 0x0a64e28c55dc18fe + .quad 0xe3df9e993399ebdd + .quad 0x79ac432370e2e652 + .quad 0x35ff7fc33ae4cc0e + .quad 0x9c13a6a52dd8f7a9 + .quad 0x2dbb1f8c3efdcabf + .quad 0x961e32405e08f7b5 + .quad 0x48c8a121bbe6c9e5 + .quad 0xfc415a7c59646445 + .quad 0xd224b2d7c128b615 + .quad 0x6035c9c905fbb912 + .quad 0x42d7a91274429fab + + // 2^212 * 5 * G + + .quad 0x4e6213e3eaf72ed3 + .quad 0x6794981a43acd4e7 + .quad 0xff547cde6eb508cb + .quad 0x6fed19dd10fcb532 + .quad 0xa9a48947933da5bc + .quad 0x4a58920ec2e979ec + .quad 0x96d8800013e5ac4c + .quad 0x453692d74b48b147 + .quad 0xdd775d99a8559c6f + .quad 0xf42a2140df003e24 + .quad 0x5223e229da928a66 + .quad 0x063f46ba6d38f22c + + // 2^212 * 6 * G + + .quad 0xd2d242895f536694 + .quad 0xca33a2c542939b2c + .quad 0x986fada6c7ddb95c + .quad 0x5a152c042f712d5d + .quad 0x39843cb737346921 + .quad 0xa747fb0738c89447 + .quad 0xcb8d8031a245307e + .quad 0x67810f8e6d82f068 + .quad 0x3eeb8fbcd2287db4 + .quad 0x72c7d3a301a03e93 + .quad 0x5473e88cbd98265a + .quad 0x7324aa515921b403 + + // 2^212 * 7 * G + + .quad 0x857942f46c3cbe8e + .quad 0xa1d364b14730c046 + .quad 0x1c8ed914d23c41bf + .quad 0x0838e161eef6d5d2 + .quad 0xad23f6dae82354cb + .quad 0x6962502ab6571a6d + .quad 0x9b651636e38e37d1 + .quad 0x5cac5005d1a3312f + .quad 0x8cc154cce9e39904 + .quad 0x5b3a040b84de6846 + .quad 0xc4d8a61cb1be5d6e + .quad 0x40fb897bd8861f02 + + // 2^212 * 8 * G + + .quad 0x84c5aa9062de37a1 + .quad 0x421da5000d1d96e1 + .quad 0x788286306a9242d9 + .quad 0x3c5e464a690d10da + .quad 0xe57ed8475ab10761 + .quad 0x71435e206fd13746 + .quad 0x342f824ecd025632 + .quad 0x4b16281ea8791e7b + .quad 0xd1c101d50b813381 + .quad 0xdee60f1176ee6828 + .quad 0x0cb68893383f6409 + .quad 0x6183c565f6ff484a + + // 2^216 * 1 * G + + .quad 0x741d5a461e6bf9d6 + .quad 0x2305b3fc7777a581 + .quad 0xd45574a26474d3d9 + .quad 0x1926e1dc6401e0ff + .quad 0xdb468549af3f666e + .quad 0xd77fcf04f14a0ea5 + .quad 0x3df23ff7a4ba0c47 + .quad 0x3a10dfe132ce3c85 + .quad 0xe07f4e8aea17cea0 + .quad 0x2fd515463a1fc1fd + .quad 0x175322fd31f2c0f1 + .quad 0x1fa1d01d861e5d15 + + // 2^216 * 2 * G + + .quad 0xcc8055947d599832 + .quad 0x1e4656da37f15520 + .quad 0x99f6f7744e059320 + .quad 0x773563bc6a75cf33 + .quad 0x38dcac00d1df94ab + .quad 0x2e712bddd1080de9 + .quad 0x7f13e93efdd5e262 + .quad 0x73fced18ee9a01e5 + .quad 0x06b1e90863139cb3 + .quad 0xa493da67c5a03ecd + .quad 0x8d77cec8ad638932 + .quad 0x1f426b701b864f44 + + // 2^216 * 3 * G + + .quad 0xefc9264c41911c01 + .quad 0xf1a3b7b817a22c25 + .quad 0x5875da6bf30f1447 + .quad 0x4e1af5271d31b090 + .quad 0xf17e35c891a12552 + .quad 0xb76b8153575e9c76 + .quad 0xfa83406f0d9b723e + .quad 0x0b76bb1b3fa7e438 + .quad 0x08b8c1f97f92939b + .quad 0xbe6771cbd444ab6e + .quad 0x22e5646399bb8017 + .quad 0x7b6dd61eb772a955 + + // 2^216 * 4 * G + + .quad 0xb7adc1e850f33d92 + .quad 0x7998fa4f608cd5cf + .quad 0xad962dbd8dfc5bdb + .quad 0x703e9bceaf1d2f4f + .quad 0x5730abf9ab01d2c7 + .quad 0x16fb76dc40143b18 + .quad 0x866cbe65a0cbb281 + .quad 0x53fa9b659bff6afe + .quad 0x6c14c8e994885455 + .quad 0x843a5d6665aed4e5 + .quad 0x181bb73ebcd65af1 + .quad 0x398d93e5c4c61f50 + + // 2^216 * 5 * G + + .quad 0x1c4bd16733e248f3 + .quad 0xbd9e128715bf0a5f + .quad 0xd43f8cf0a10b0376 + .quad 0x53b09b5ddf191b13 + .quad 0xc3877c60d2e7e3f2 + .quad 0x3b34aaa030828bb1 + .quad 0x283e26e7739ef138 + .quad 0x699c9c9002c30577 + .quad 0xf306a7235946f1cc + .quad 0x921718b5cce5d97d + .quad 0x28cdd24781b4e975 + .quad 0x51caf30c6fcdd907 + + // 2^216 * 6 * G + + .quad 0xa60ba7427674e00a + .quad 0x630e8570a17a7bf3 + .quad 0x3758563dcf3324cc + .quad 0x5504aa292383fdaa + .quad 0x737af99a18ac54c7 + .quad 0x903378dcc51cb30f + .quad 0x2b89bc334ce10cc7 + .quad 0x12ae29c189f8e99a + .quad 0xa99ec0cb1f0d01cf + .quad 0x0dd1efcc3a34f7ae + .quad 0x55ca7521d09c4e22 + .quad 0x5fd14fe958eba5ea + + // 2^216 * 7 * G + + .quad 0xb5dc2ddf2845ab2c + .quad 0x069491b10a7fe993 + .quad 0x4daaf3d64002e346 + .quad 0x093ff26e586474d1 + .quad 0x3c42fe5ebf93cb8e + .quad 0xbedfa85136d4565f + .quad 0xe0f0859e884220e8 + .quad 0x7dd73f960725d128 + .quad 0xb10d24fe68059829 + .quad 0x75730672dbaf23e5 + .quad 0x1367253ab457ac29 + .quad 0x2f59bcbc86b470a4 + + // 2^216 * 8 * G + + .quad 0x83847d429917135f + .quad 0xad1b911f567d03d7 + .quad 0x7e7748d9be77aad1 + .quad 0x5458b42e2e51af4a + .quad 0x7041d560b691c301 + .quad 0x85201b3fadd7e71e + .quad 0x16c2e16311335585 + .quad 0x2aa55e3d010828b1 + .quad 0xed5192e60c07444f + .quad 0x42c54e2d74421d10 + .quad 0x352b4c82fdb5c864 + .quad 0x13e9004a8a768664 + + // 2^220 * 1 * G + + .quad 0xcbb5b5556c032bff + .quad 0xdf7191b729297a3a + .quad 0xc1ff7326aded81bb + .quad 0x71ade8bb68be03f5 + .quad 0x1e6284c5806b467c + .quad 0xc5f6997be75d607b + .quad 0x8b67d958b378d262 + .quad 0x3d88d66a81cd8b70 + .quad 0x8b767a93204ed789 + .quad 0x762fcacb9fa0ae2a + .quad 0x771febcc6dce4887 + .quad 0x343062158ff05fb3 + + // 2^220 * 2 * G + + .quad 0xe05da1a7e1f5bf49 + .quad 0x26457d6dd4736092 + .quad 0x77dcb07773cc32f6 + .quad 0x0a5d94969cdd5fcd + .quad 0xfce219072a7b31b4 + .quad 0x4d7adc75aa578016 + .quad 0x0ec276a687479324 + .quad 0x6d6d9d5d1fda4beb + .quad 0x22b1a58ae9b08183 + .quad 0xfd95d071c15c388b + .quad 0xa9812376850a0517 + .quad 0x33384cbabb7f335e + + // 2^220 * 3 * G + + .quad 0x3c6fa2680ca2c7b5 + .quad 0x1b5082046fb64fda + .quad 0xeb53349c5431d6de + .quad 0x5278b38f6b879c89 + .quad 0x33bc627a26218b8d + .quad 0xea80b21fc7a80c61 + .quad 0x9458b12b173e9ee6 + .quad 0x076247be0e2f3059 + .quad 0x52e105f61416375a + .quad 0xec97af3685abeba4 + .quad 0x26e6b50623a67c36 + .quad 0x5cf0e856f3d4fb01 + + // 2^220 * 4 * G + + .quad 0xf6c968731ae8cab4 + .quad 0x5e20741ecb4f92c5 + .quad 0x2da53be58ccdbc3e + .quad 0x2dddfea269970df7 + .quad 0xbeaece313db342a8 + .quad 0xcba3635b842db7ee + .quad 0xe88c6620817f13ef + .quad 0x1b9438aa4e76d5c6 + .quad 0x8a50777e166f031a + .quad 0x067b39f10fb7a328 + .quad 0x1925c9a6010fbd76 + .quad 0x6df9b575cc740905 + + // 2^220 * 5 * G + + .quad 0x42c1192927f6bdcf + .quad 0x8f91917a403d61ca + .quad 0xdc1c5a668b9e1f61 + .quad 0x1596047804ec0f8d + .quad 0xecdfc35b48cade41 + .quad 0x6a88471fb2328270 + .quad 0x740a4a2440a01b6a + .quad 0x471e5796003b5f29 + .quad 0xda96bbb3aced37ac + .quad 0x7a2423b5e9208cea + .quad 0x24cc5c3038aebae2 + .quad 0x50c356afdc5dae2f + + // 2^220 * 6 * G + + .quad 0x09dcbf4341c30318 + .quad 0xeeba061183181dce + .quad 0xc179c0cedc1e29a1 + .quad 0x1dbf7b89073f35b0 + .quad 0xcfed9cdf1b31b964 + .quad 0xf486a9858ca51af3 + .quad 0x14897265ea8c1f84 + .quad 0x784a53dd932acc00 + .quad 0x2d99f9df14fc4920 + .quad 0x76ccb60cc4499fe5 + .quad 0xa4132cbbe5cf0003 + .quad 0x3f93d82354f000ea + + // 2^220 * 7 * G + + .quad 0x8183e7689e04ce85 + .quad 0x678fb71e04465341 + .quad 0xad92058f6688edac + .quad 0x5da350d3532b099a + .quad 0xeaac12d179e14978 + .quad 0xff923ff3bbebff5e + .quad 0x4af663e40663ce27 + .quad 0x0fd381a811a5f5ff + .quad 0xf256aceca436df54 + .quad 0x108b6168ae69d6e8 + .quad 0x20d986cb6b5d036c + .quad 0x655957b9fee2af50 + + // 2^220 * 8 * G + + .quad 0xaea8b07fa902030f + .quad 0xf88c766af463d143 + .quad 0x15b083663c787a60 + .quad 0x08eab1148267a4a8 + .quad 0xbdc1409bd002d0ac + .quad 0x66660245b5ccd9a6 + .quad 0x82317dc4fade85ec + .quad 0x02fe934b6ad7df0d + .quad 0xef5cf100cfb7ea74 + .quad 0x22897633a1cb42ac + .quad 0xd4ce0c54cef285e2 + .quad 0x30408c048a146a55 + + // 2^224 * 1 * G + + .quad 0x739d8845832fcedb + .quad 0xfa38d6c9ae6bf863 + .quad 0x32bc0dcab74ffef7 + .quad 0x73937e8814bce45e + .quad 0xbb2e00c9193b877f + .quad 0xece3a890e0dc506b + .quad 0xecf3b7c036de649f + .quad 0x5f46040898de9e1a + .quad 0xb9037116297bf48d + .quad 0xa9d13b22d4f06834 + .quad 0xe19715574696bdc6 + .quad 0x2cf8a4e891d5e835 + + // 2^224 * 2 * G + + .quad 0x6d93fd8707110f67 + .quad 0xdd4c09d37c38b549 + .quad 0x7cb16a4cc2736a86 + .quad 0x2049bd6e58252a09 + .quad 0x2cb5487e17d06ba2 + .quad 0x24d2381c3950196b + .quad 0xd7659c8185978a30 + .quad 0x7a6f7f2891d6a4f6 + .quad 0x7d09fd8d6a9aef49 + .quad 0xf0ee60be5b3db90b + .quad 0x4c21b52c519ebfd4 + .quad 0x6011aadfc545941d + + // 2^224 * 3 * G + + .quad 0x5f67926dcf95f83c + .quad 0x7c7e856171289071 + .quad 0xd6a1e7f3998f7a5b + .quad 0x6fc5cc1b0b62f9e0 + .quad 0x63ded0c802cbf890 + .quad 0xfbd098ca0dff6aaa + .quad 0x624d0afdb9b6ed99 + .quad 0x69ce18b779340b1e + .quad 0xd1ef5528b29879cb + .quad 0xdd1aae3cd47e9092 + .quad 0x127e0442189f2352 + .quad 0x15596b3ae57101f1 + + // 2^224 * 4 * G + + .quad 0x462739d23f9179a2 + .quad 0xff83123197d6ddcf + .quad 0x1307deb553f2148a + .quad 0x0d2237687b5f4dda + .quad 0x09ff31167e5124ca + .quad 0x0be4158bd9c745df + .quad 0x292b7d227ef556e5 + .quad 0x3aa4e241afb6d138 + .quad 0x2cc138bf2a3305f5 + .quad 0x48583f8fa2e926c3 + .quad 0x083ab1a25549d2eb + .quad 0x32fcaa6e4687a36c + + // 2^224 * 5 * G + + .quad 0x7bc56e8dc57d9af5 + .quad 0x3e0bd2ed9df0bdf2 + .quad 0xaac014de22efe4a3 + .quad 0x4627e9cefebd6a5c + .quad 0x3207a4732787ccdf + .quad 0x17e31908f213e3f8 + .quad 0xd5b2ecd7f60d964e + .quad 0x746f6336c2600be9 + .quad 0x3f4af345ab6c971c + .quad 0xe288eb729943731f + .quad 0x33596a8a0344186d + .quad 0x7b4917007ed66293 + + // 2^224 * 6 * G + + .quad 0x2d85fb5cab84b064 + .quad 0x497810d289f3bc14 + .quad 0x476adc447b15ce0c + .quad 0x122ba376f844fd7b + .quad 0x54341b28dd53a2dd + .quad 0xaa17905bdf42fc3f + .quad 0x0ff592d94dd2f8f4 + .quad 0x1d03620fe08cd37d + .quad 0xc20232cda2b4e554 + .quad 0x9ed0fd42115d187f + .quad 0x2eabb4be7dd479d9 + .quad 0x02c70bf52b68ec4c + + // 2^224 * 7 * G + + .quad 0xa287ec4b5d0b2fbb + .quad 0x415c5790074882ca + .quad 0xe044a61ec1d0815c + .quad 0x26334f0a409ef5e0 + .quad 0xace532bf458d72e1 + .quad 0x5be768e07cb73cb5 + .quad 0x56cf7d94ee8bbde7 + .quad 0x6b0697e3feb43a03 + .quad 0xb6c8f04adf62a3c0 + .quad 0x3ef000ef076da45d + .quad 0x9c9cb95849f0d2a9 + .quad 0x1cc37f43441b2fae + + // 2^224 * 8 * G + + .quad 0x508f565a5cc7324f + .quad 0xd061c4c0e506a922 + .quad 0xfb18abdb5c45ac19 + .quad 0x6c6809c10380314a + .quad 0xd76656f1c9ceaeb9 + .quad 0x1c5b15f818e5656a + .quad 0x26e72832844c2334 + .quad 0x3a346f772f196838 + .quad 0xd2d55112e2da6ac8 + .quad 0xe9bd0331b1e851ed + .quad 0x960746dd8ec67262 + .quad 0x05911b9f6ef7c5d0 + + // 2^228 * 1 * G + + .quad 0xe9dcd756b637ff2d + .quad 0xec4c348fc987f0c4 + .quad 0xced59285f3fbc7b7 + .quad 0x3305354793e1ea87 + .quad 0x01c18980c5fe9f94 + .quad 0xcd656769716fd5c8 + .quad 0x816045c3d195a086 + .quad 0x6e2b7f3266cc7982 + .quad 0xcc802468f7c3568f + .quad 0x9de9ba8219974cb3 + .quad 0xabb7229cb5b81360 + .quad 0x44e2017a6fbeba62 + + // 2^228 * 2 * G + + .quad 0xc4c2a74354dab774 + .quad 0x8e5d4c3c4eaf031a + .quad 0xb76c23d242838f17 + .quad 0x749a098f68dce4ea + .quad 0x87f82cf3b6ca6ecd + .quad 0x580f893e18f4a0c2 + .quad 0x058930072604e557 + .quad 0x6cab6ac256d19c1d + .quad 0xdcdfe0a02cc1de60 + .quad 0x032665ff51c5575b + .quad 0x2c0c32f1073abeeb + .quad 0x6a882014cd7b8606 + + // 2^228 * 3 * G + + .quad 0xa52a92fea4747fb5 + .quad 0xdc12a4491fa5ab89 + .quad 0xd82da94bb847a4ce + .quad 0x4d77edce9512cc4e + .quad 0xd111d17caf4feb6e + .quad 0x050bba42b33aa4a3 + .quad 0x17514c3ceeb46c30 + .quad 0x54bedb8b1bc27d75 + .quad 0x77c8e14577e2189c + .quad 0xa3e46f6aff99c445 + .quad 0x3144dfc86d335343 + .quad 0x3a96559e7c4216a9 + + // 2^228 * 4 * G + + .quad 0x12550d37f42ad2ee + .quad 0x8b78e00498a1fbf5 + .quad 0x5d53078233894cb2 + .quad 0x02c84e4e3e498d0c + .quad 0x4493896880baaa52 + .quad 0x4c98afc4f285940e + .quad 0xef4aa79ba45448b6 + .quad 0x5278c510a57aae7f + .quad 0xa54dd074294c0b94 + .quad 0xf55d46b8df18ffb6 + .quad 0xf06fecc58dae8366 + .quad 0x588657668190d165 + + // 2^228 * 5 * G + + .quad 0xd47712311aef7117 + .quad 0x50343101229e92c7 + .quad 0x7a95e1849d159b97 + .quad 0x2449959b8b5d29c9 + .quad 0xbf5834f03de25cc3 + .quad 0xb887c8aed6815496 + .quad 0x5105221a9481e892 + .quad 0x6760ed19f7723f93 + .quad 0x669ba3b7ac35e160 + .quad 0x2eccf73fba842056 + .quad 0x1aec1f17c0804f07 + .quad 0x0d96bc031856f4e7 + + // 2^228 * 6 * G + + .quad 0x3318be7775c52d82 + .quad 0x4cb764b554d0aab9 + .quad 0xabcf3d27cc773d91 + .quad 0x3bf4d1848123288a + .quad 0xb1d534b0cc7505e1 + .quad 0x32cd003416c35288 + .quad 0xcb36a5800762c29d + .quad 0x5bfe69b9237a0bf8 + .quad 0x183eab7e78a151ab + .quad 0xbbe990c999093763 + .quad 0xff717d6e4ac7e335 + .quad 0x4c5cddb325f39f88 + + // 2^228 * 7 * G + + .quad 0xc0f6b74d6190a6eb + .quad 0x20ea81a42db8f4e4 + .quad 0xa8bd6f7d97315760 + .quad 0x33b1d60262ac7c21 + .quad 0x57750967e7a9f902 + .quad 0x2c37fdfc4f5b467e + .quad 0xb261663a3177ba46 + .quad 0x3a375e78dc2d532b + .quad 0x8141e72f2d4dddea + .quad 0xe6eafe9862c607c8 + .quad 0x23c28458573cafd0 + .quad 0x46b9476f4ff97346 + + // 2^228 * 8 * G + + .quad 0x0c1ffea44f901e5c + .quad 0x2b0b6fb72184b782 + .quad 0xe587ff910114db88 + .quad 0x37130f364785a142 + .quad 0x1215505c0d58359f + .quad 0x2a2013c7fc28c46b + .quad 0x24a0a1af89ea664e + .quad 0x4400b638a1130e1f + .quad 0x3a01b76496ed19c3 + .quad 0x31e00ab0ed327230 + .quad 0x520a885783ca15b1 + .quad 0x06aab9875accbec7 + + // 2^232 * 1 * G + + .quad 0xc1339983f5df0ebb + .quad 0xc0f3758f512c4cac + .quad 0x2cf1130a0bb398e1 + .quad 0x6b3cecf9aa270c62 + .quad 0x5349acf3512eeaef + .quad 0x20c141d31cc1cb49 + .quad 0x24180c07a99a688d + .quad 0x555ef9d1c64b2d17 + .quad 0x36a770ba3b73bd08 + .quad 0x624aef08a3afbf0c + .quad 0x5737ff98b40946f2 + .quad 0x675f4de13381749d + + // 2^232 * 2 * G + + .quad 0x0e2c52036b1782fc + .quad 0x64816c816cad83b4 + .quad 0xd0dcbdd96964073e + .quad 0x13d99df70164c520 + .quad 0xa12ff6d93bdab31d + .quad 0x0725d80f9d652dfe + .quad 0x019c4ff39abe9487 + .quad 0x60f450b882cd3c43 + .quad 0x014b5ec321e5c0ca + .quad 0x4fcb69c9d719bfa2 + .quad 0x4e5f1c18750023a0 + .quad 0x1c06de9e55edac80 + + // 2^232 * 3 * G + + .quad 0x990f7ad6a33ec4e2 + .quad 0x6608f938be2ee08e + .quad 0x9ca143c563284515 + .quad 0x4cf38a1fec2db60d + .quad 0xffd52b40ff6d69aa + .quad 0x34530b18dc4049bb + .quad 0x5e4a5c2fa34d9897 + .quad 0x78096f8e7d32ba2d + .quad 0xa0aaaa650dfa5ce7 + .quad 0xf9c49e2a48b5478c + .quad 0x4f09cc7d7003725b + .quad 0x373cad3a26091abe + + // 2^232 * 4 * G + + .quad 0xb294634d82c9f57c + .quad 0x1fcbfde124934536 + .quad 0x9e9c4db3418cdb5a + .quad 0x0040f3d9454419fc + .quad 0xf1bea8fb89ddbbad + .quad 0x3bcb2cbc61aeaecb + .quad 0x8f58a7bb1f9b8d9d + .quad 0x21547eda5112a686 + .quad 0xdefde939fd5986d3 + .quad 0xf4272c89510a380c + .quad 0xb72ba407bb3119b9 + .quad 0x63550a334a254df4 + + // 2^232 * 5 * G + + .quad 0x6507d6edb569cf37 + .quad 0x178429b00ca52ee1 + .quad 0xea7c0090eb6bd65d + .quad 0x3eea62c7daf78f51 + .quad 0x9bba584572547b49 + .quad 0xf305c6fae2c408e0 + .quad 0x60e8fa69c734f18d + .quad 0x39a92bafaa7d767a + .quad 0x9d24c713e693274e + .quad 0x5f63857768dbd375 + .quad 0x70525560eb8ab39a + .quad 0x68436a0665c9c4cd + + // 2^232 * 6 * G + + .quad 0xbc0235e8202f3f27 + .quad 0xc75c00e264f975b0 + .quad 0x91a4e9d5a38c2416 + .quad 0x17b6e7f68ab789f9 + .quad 0x1e56d317e820107c + .quad 0xc5266844840ae965 + .quad 0xc1e0a1c6320ffc7a + .quad 0x5373669c91611472 + .quad 0x5d2814ab9a0e5257 + .quad 0x908f2084c9cab3fc + .quad 0xafcaf5885b2d1eca + .quad 0x1cb4b5a678f87d11 + + // 2^232 * 7 * G + + .quad 0xb664c06b394afc6c + .quad 0x0c88de2498da5fb1 + .quad 0x4f8d03164bcad834 + .quad 0x330bca78de7434a2 + .quad 0x6b74aa62a2a007e7 + .quad 0xf311e0b0f071c7b1 + .quad 0x5707e438000be223 + .quad 0x2dc0fd2d82ef6eac + .quad 0x982eff841119744e + .quad 0xf9695e962b074724 + .quad 0xc58ac14fbfc953fb + .quad 0x3c31be1b369f1cf5 + + // 2^232 * 8 * G + + .quad 0xb0f4864d08948aee + .quad 0x07dc19ee91ba1c6f + .quad 0x7975cdaea6aca158 + .quad 0x330b61134262d4bb + .quad 0xc168bc93f9cb4272 + .quad 0xaeb8711fc7cedb98 + .quad 0x7f0e52aa34ac8d7a + .quad 0x41cec1097e7d55bb + .quad 0xf79619d7a26d808a + .quad 0xbb1fd49e1d9e156d + .quad 0x73d7c36cdba1df27 + .quad 0x26b44cd91f28777d + + // 2^236 * 1 * G + + .quad 0x300a9035393aa6d8 + .quad 0x2b501131a12bb1cd + .quad 0x7b1ff677f093c222 + .quad 0x4309c1f8cab82bad + .quad 0xaf44842db0285f37 + .quad 0x8753189047efc8df + .quad 0x9574e091f820979a + .quad 0x0e378d6069615579 + .quad 0xd9fa917183075a55 + .quad 0x4bdb5ad26b009fdc + .quad 0x7829ad2cd63def0e + .quad 0x078fc54975fd3877 + + // 2^236 * 2 * G + + .quad 0x87dfbd1428878f2d + .quad 0x134636dd1e9421a1 + .quad 0x4f17c951257341a3 + .quad 0x5df98d4bad296cb8 + .quad 0xe2004b5bb833a98a + .quad 0x44775dec2d4c3330 + .quad 0x3aa244067eace913 + .quad 0x272630e3d58e00a9 + .quad 0xf3678fd0ecc90b54 + .quad 0xf001459b12043599 + .quad 0x26725fbc3758b89b + .quad 0x4325e4aa73a719ae + + // 2^236 * 3 * G + + .quad 0x657dc6ef433c3493 + .quad 0x65375e9f80dbf8c3 + .quad 0x47fd2d465b372dae + .quad 0x4966ab79796e7947 + .quad 0xed24629acf69f59d + .quad 0x2a4a1ccedd5abbf4 + .quad 0x3535ca1f56b2d67b + .quad 0x5d8c68d043b1b42d + .quad 0xee332d4de3b42b0a + .quad 0xd84e5a2b16a4601c + .quad 0x78243877078ba3e4 + .quad 0x77ed1eb4184ee437 + + // 2^236 * 4 * G + + .quad 0xbfd4e13f201839a0 + .quad 0xaeefffe23e3df161 + .quad 0xb65b04f06b5d1fe3 + .quad 0x52e085fb2b62fbc0 + .quad 0x185d43f89e92ed1a + .quad 0xb04a1eeafe4719c6 + .quad 0x499fbe88a6f03f4f + .quad 0x5d8b0d2f3c859bdd + .quad 0x124079eaa54cf2ba + .quad 0xd72465eb001b26e7 + .quad 0x6843bcfdc97af7fd + .quad 0x0524b42b55eacd02 + + // 2^236 * 5 * G + + .quad 0xfd0d5dbee45447b0 + .quad 0x6cec351a092005ee + .quad 0x99a47844567579cb + .quad 0x59d242a216e7fa45 + .quad 0xbc18dcad9b829eac + .quad 0x23ae7d28b5f579d0 + .quad 0xc346122a69384233 + .quad 0x1a6110b2e7d4ac89 + .quad 0x4f833f6ae66997ac + .quad 0x6849762a361839a4 + .quad 0x6985dec1970ab525 + .quad 0x53045e89dcb1f546 + + // 2^236 * 6 * G + + .quad 0xcb8bb346d75353db + .quad 0xfcfcb24bae511e22 + .quad 0xcba48d40d50ae6ef + .quad 0x26e3bae5f4f7cb5d + .quad 0x84da3cde8d45fe12 + .quad 0xbd42c218e444e2d2 + .quad 0xa85196781f7e3598 + .quad 0x7642c93f5616e2b2 + .quad 0x2323daa74595f8e4 + .quad 0xde688c8b857abeb4 + .quad 0x3fc48e961c59326e + .quad 0x0b2e73ca15c9b8ba + + // 2^236 * 7 * G + + .quad 0xd6bb4428c17f5026 + .quad 0x9eb27223fb5a9ca7 + .quad 0xe37ba5031919c644 + .quad 0x21ce380db59a6602 + .quad 0x0e3fbfaf79c03a55 + .quad 0x3077af054cbb5acf + .quad 0xd5c55245db3de39f + .quad 0x015e68c1476a4af7 + .quad 0xc1d5285220066a38 + .quad 0x95603e523570aef3 + .quad 0x832659a7226b8a4d + .quad 0x5dd689091f8eedc9 + + // 2^236 * 8 * G + + .quad 0xcbac84debfd3c856 + .quad 0x1624c348b35ff244 + .quad 0xb7f88dca5d9cad07 + .quad 0x3b0e574da2c2ebe8 + .quad 0x1d022591a5313084 + .quad 0xca2d4aaed6270872 + .quad 0x86a12b852f0bfd20 + .quad 0x56e6c439ad7da748 + .quad 0xc704ff4942bdbae6 + .quad 0x5e21ade2b2de1f79 + .quad 0xe95db3f35652fad8 + .quad 0x0822b5378f08ebc1 + + // 2^240 * 1 * G + + .quad 0x51f048478f387475 + .quad 0xb25dbcf49cbecb3c + .quad 0x9aab1244d99f2055 + .quad 0x2c709e6c1c10a5d6 + .quad 0xe1b7f29362730383 + .quad 0x4b5279ffebca8a2c + .quad 0xdafc778abfd41314 + .quad 0x7deb10149c72610f + .quad 0xcb62af6a8766ee7a + .quad 0x66cbec045553cd0e + .quad 0x588001380f0be4b5 + .quad 0x08e68e9ff62ce2ea + + // 2^240 * 2 * G + + .quad 0x34ad500a4bc130ad + .quad 0x8d38db493d0bd49c + .quad 0xa25c3d98500a89be + .quad 0x2f1f3f87eeba3b09 + .quad 0x2f2d09d50ab8f2f9 + .quad 0xacb9218dc55923df + .quad 0x4a8f342673766cb9 + .quad 0x4cb13bd738f719f5 + .quad 0xf7848c75e515b64a + .quad 0xa59501badb4a9038 + .quad 0xc20d313f3f751b50 + .quad 0x19a1e353c0ae2ee8 + + // 2^240 * 3 * G + + .quad 0x7d1c7560bafa05c3 + .quad 0xb3e1a0a0c6e55e61 + .quad 0xe3529718c0d66473 + .quad 0x41546b11c20c3486 + .quad 0xb42172cdd596bdbd + .quad 0x93e0454398eefc40 + .quad 0x9fb15347b44109b5 + .quad 0x736bd3990266ae34 + .quad 0x85532d509334b3b4 + .quad 0x46fd114b60816573 + .quad 0xcc5f5f30425c8375 + .quad 0x412295a2b87fab5c + + // 2^240 * 4 * G + + .quad 0x19c99b88f57ed6e9 + .quad 0x5393cb266df8c825 + .quad 0x5cee3213b30ad273 + .quad 0x14e153ebb52d2e34 + .quad 0x2e655261e293eac6 + .quad 0x845a92032133acdb + .quad 0x460975cb7900996b + .quad 0x0760bb8d195add80 + .quad 0x413e1a17cde6818a + .quad 0x57156da9ed69a084 + .quad 0x2cbf268f46caccb1 + .quad 0x6b34be9bc33ac5f2 + + // 2^240 * 5 * G + + .quad 0xf3df2f643a78c0b2 + .quad 0x4c3e971ef22e027c + .quad 0xec7d1c5e49c1b5a3 + .quad 0x2012c18f0922dd2d + .quad 0x11fc69656571f2d3 + .quad 0xc6c9e845530e737a + .quad 0xe33ae7a2d4fe5035 + .quad 0x01b9c7b62e6dd30b + .quad 0x880b55e55ac89d29 + .quad 0x1483241f45a0a763 + .quad 0x3d36efdfc2e76c1f + .quad 0x08af5b784e4bade8 + + // 2^240 * 6 * G + + .quad 0x283499dc881f2533 + .quad 0x9d0525da779323b6 + .quad 0x897addfb673441f4 + .quad 0x32b79d71163a168d + .quad 0xe27314d289cc2c4b + .quad 0x4be4bd11a287178d + .quad 0x18d528d6fa3364ce + .quad 0x6423c1d5afd9826e + .quad 0xcc85f8d9edfcb36a + .quad 0x22bcc28f3746e5f9 + .quad 0xe49de338f9e5d3cd + .quad 0x480a5efbc13e2dcc + + // 2^240 * 7 * G + + .quad 0x0b51e70b01622071 + .quad 0x06b505cf8b1dafc5 + .quad 0x2c6bb061ef5aabcd + .quad 0x47aa27600cb7bf31 + .quad 0xb6614ce442ce221f + .quad 0x6e199dcc4c053928 + .quad 0x663fb4a4dc1cbe03 + .quad 0x24b31d47691c8e06 + .quad 0x2a541eedc015f8c3 + .quad 0x11a4fe7e7c693f7c + .quad 0xf0af66134ea278d6 + .quad 0x545b585d14dda094 + + // 2^240 * 8 * G + + .quad 0x67bf275ea0d43a0f + .quad 0xade68e34089beebe + .quad 0x4289134cd479e72e + .quad 0x0f62f9c332ba5454 + .quad 0x6204e4d0e3b321e1 + .quad 0x3baa637a28ff1e95 + .quad 0x0b0ccffd5b99bd9e + .quad 0x4d22dc3e64c8d071 + .quad 0xfcb46589d63b5f39 + .quad 0x5cae6a3f57cbcf61 + .quad 0xfebac2d2953afa05 + .quad 0x1c0fa01a36371436 + + // 2^244 * 1 * G + + .quad 0xe7547449bc7cd692 + .quad 0x0f9abeaae6f73ddf + .quad 0x4af01ca700837e29 + .quad 0x63ab1b5d3f1bc183 + .quad 0xc11ee5e854c53fae + .quad 0x6a0b06c12b4f3ff4 + .quad 0x33540f80e0b67a72 + .quad 0x15f18fc3cd07e3ef + .quad 0x32750763b028f48c + .quad 0x06020740556a065f + .quad 0xd53bd812c3495b58 + .quad 0x08706c9b865f508d + + // 2^244 * 2 * G + + .quad 0xf37ca2ab3d343dff + .quad 0x1a8c6a2d80abc617 + .quad 0x8e49e035d4ccffca + .quad 0x48b46beebaa1d1b9 + .quad 0xcc991b4138b41246 + .quad 0x243b9c526f9ac26b + .quad 0xb9ef494db7cbabbd + .quad 0x5fba433dd082ed00 + .quad 0x9c49e355c9941ad0 + .quad 0xb9734ade74498f84 + .quad 0x41c3fed066663e5c + .quad 0x0ecfedf8e8e710b3 + + // 2^244 * 3 * G + + .quad 0x76430f9f9cd470d9 + .quad 0xb62acc9ba42f6008 + .quad 0x1898297c59adad5e + .quad 0x7789dd2db78c5080 + .quad 0x744f7463e9403762 + .quad 0xf79a8dee8dfcc9c9 + .quad 0x163a649655e4cde3 + .quad 0x3b61788db284f435 + .quad 0xb22228190d6ef6b2 + .quad 0xa94a66b246ce4bfa + .quad 0x46c1a77a4f0b6cc7 + .quad 0x4236ccffeb7338cf + + // 2^244 * 4 * G + + .quad 0x8497404d0d55e274 + .quad 0x6c6663d9c4ad2b53 + .quad 0xec2fb0d9ada95734 + .quad 0x2617e120cdb8f73c + .quad 0x3bd82dbfda777df6 + .quad 0x71b177cc0b98369e + .quad 0x1d0e8463850c3699 + .quad 0x5a71945b48e2d1f1 + .quad 0x6f203dd5405b4b42 + .quad 0x327ec60410b24509 + .quad 0x9c347230ac2a8846 + .quad 0x77de29fc11ffeb6a + + // 2^244 * 5 * G + + .quad 0xb0ac57c983b778a8 + .quad 0x53cdcca9d7fe912c + .quad 0x61c2b854ff1f59dc + .quad 0x3a1a2cf0f0de7dac + .quad 0x835e138fecced2ca + .quad 0x8c9eaf13ea963b9a + .quad 0xc95fbfc0b2160ea6 + .quad 0x575e66f3ad877892 + .quad 0x99803a27c88fcb3a + .quad 0x345a6789275ec0b0 + .quad 0x459789d0ff6c2be5 + .quad 0x62f882651e70a8b2 + + // 2^244 * 6 * G + + .quad 0x085ae2c759ff1be4 + .quad 0x149145c93b0e40b7 + .quad 0xc467e7fa7ff27379 + .quad 0x4eeecf0ad5c73a95 + .quad 0x6d822986698a19e0 + .quad 0xdc9821e174d78a71 + .quad 0x41a85f31f6cb1f47 + .quad 0x352721c2bcda9c51 + .quad 0x48329952213fc985 + .quad 0x1087cf0d368a1746 + .quad 0x8e5261b166c15aa5 + .quad 0x2d5b2d842ed24c21 + + // 2^244 * 7 * G + + .quad 0x02cfebd9ebd3ded1 + .quad 0xd45b217739021974 + .quad 0x7576f813fe30a1b7 + .quad 0x5691b6f9a34ef6c2 + .quad 0x5eb7d13d196ac533 + .quad 0x377234ecdb80be2b + .quad 0xe144cffc7cf5ae24 + .quad 0x5226bcf9c441acec + .quad 0x79ee6c7223e5b547 + .quad 0x6f5f50768330d679 + .quad 0xed73e1e96d8adce9 + .quad 0x27c3da1e1d8ccc03 + + // 2^244 * 8 * G + + .quad 0x7eb9efb23fe24c74 + .quad 0x3e50f49f1651be01 + .quad 0x3ea732dc21858dea + .quad 0x17377bd75bb810f9 + .quad 0x28302e71630ef9f6 + .quad 0xc2d4a2032b64cee0 + .quad 0x090820304b6292be + .quad 0x5fca747aa82adf18 + .quad 0x232a03c35c258ea5 + .quad 0x86f23a2c6bcb0cf1 + .quad 0x3dad8d0d2e442166 + .quad 0x04a8933cab76862b + + // 2^248 * 1 * G + + .quad 0xd2c604b622943dff + .quad 0xbc8cbece44cfb3a0 + .quad 0x5d254ff397808678 + .quad 0x0fa3614f3b1ca6bf + .quad 0x69082b0e8c936a50 + .quad 0xf9c9a035c1dac5b6 + .quad 0x6fb73e54c4dfb634 + .quad 0x4005419b1d2bc140 + .quad 0xa003febdb9be82f0 + .quad 0x2089c1af3a44ac90 + .quad 0xf8499f911954fa8e + .quad 0x1fba218aef40ab42 + + // 2^248 * 2 * G + + .quad 0xab549448fac8f53e + .quad 0x81f6e89a7ba63741 + .quad 0x74fd6c7d6c2b5e01 + .quad 0x392e3acaa8c86e42 + .quad 0x4f3e57043e7b0194 + .quad 0xa81d3eee08daaf7f + .quad 0xc839c6ab99dcdef1 + .quad 0x6c535d13ff7761d5 + .quad 0x4cbd34e93e8a35af + .quad 0x2e0781445887e816 + .quad 0x19319c76f29ab0ab + .quad 0x25e17fe4d50ac13b + + // 2^248 * 3 * G + + .quad 0x0a289bd71e04f676 + .quad 0x208e1c52d6420f95 + .quad 0x5186d8b034691fab + .quad 0x255751442a9fb351 + .quad 0x915f7ff576f121a7 + .quad 0xc34a32272fcd87e3 + .quad 0xccba2fde4d1be526 + .quad 0x6bba828f8969899b + .quad 0xe2d1bc6690fe3901 + .quad 0x4cb54a18a0997ad5 + .quad 0x971d6914af8460d4 + .quad 0x559d504f7f6b7be4 + + // 2^248 * 4 * G + + .quad 0xa7738378b3eb54d5 + .quad 0x1d69d366a5553c7c + .quad 0x0a26cf62f92800ba + .quad 0x01ab12d5807e3217 + .quad 0x9c4891e7f6d266fd + .quad 0x0744a19b0307781b + .quad 0x88388f1d6061e23b + .quad 0x123ea6a3354bd50e + .quad 0x118d189041e32d96 + .quad 0xb9ede3c2d8315848 + .quad 0x1eab4271d83245d9 + .quad 0x4a3961e2c918a154 + + // 2^248 * 5 * G + + .quad 0x71dc3be0f8e6bba0 + .quad 0xd6cef8347effe30a + .quad 0xa992425fe13a476a + .quad 0x2cd6bce3fb1db763 + .quad 0x0327d644f3233f1e + .quad 0x499a260e34fcf016 + .quad 0x83b5a716f2dab979 + .quad 0x68aceead9bd4111f + .quad 0x38b4c90ef3d7c210 + .quad 0x308e6e24b7ad040c + .quad 0x3860d9f1b7e73e23 + .quad 0x595760d5b508f597 + + // 2^248 * 6 * G + + .quad 0x6129bfe104aa6397 + .quad 0x8f960008a4a7fccb + .quad 0x3f8bc0897d909458 + .quad 0x709fa43edcb291a9 + .quad 0x882acbebfd022790 + .quad 0x89af3305c4115760 + .quad 0x65f492e37d3473f4 + .quad 0x2cb2c5df54515a2b + .quad 0xeb0a5d8c63fd2aca + .quad 0xd22bc1662e694eff + .quad 0x2723f36ef8cbb03a + .quad 0x70f029ecf0c8131f + + // 2^248 * 7 * G + + .quad 0x461307b32eed3e33 + .quad 0xae042f33a45581e7 + .quad 0xc94449d3195f0366 + .quad 0x0b7d5d8a6c314858 + .quad 0x2a6aafaa5e10b0b9 + .quad 0x78f0a370ef041aa9 + .quad 0x773efb77aa3ad61f + .quad 0x44eca5a2a74bd9e1 + .quad 0x25d448327b95d543 + .quad 0x70d38300a3340f1d + .quad 0xde1c531c60e1c52b + .quad 0x272224512c7de9e4 + + // 2^248 * 8 * G + + .quad 0x1abc92af49c5342e + .quad 0xffeed811b2e6fad0 + .quad 0xefa28c8dfcc84e29 + .quad 0x11b5df18a44cc543 + .quad 0xbf7bbb8a42a975fc + .quad 0x8c5c397796ada358 + .quad 0xe27fc76fcdedaa48 + .quad 0x19735fd7f6bc20a6 + .quad 0xe3ab90d042c84266 + .quad 0xeb848e0f7f19547e + .quad 0x2503a1d065a497b9 + .quad 0x0fef911191df895f + + // 2^252 * 1 * G + + .quad 0xb1507ca1ab1c6eb9 + .quad 0xbd448f3e16b687b3 + .quad 0x3455fb7f2c7a91ab + .quad 0x7579229e2f2adec1 + .quad 0x6ab5dcb85b1c16b7 + .quad 0x94c0fce83c7b27a5 + .quad 0xa4b11c1a735517be + .quad 0x499238d0ba0eafaa + .quad 0xecf46e527aba8b57 + .quad 0x15a08c478bd1647b + .quad 0x7af1c6a65f706fef + .quad 0x6345fa78f03a30d5 + + // 2^252 * 2 * G + + .quad 0xdf02f95f1015e7a1 + .quad 0x790ec41da9b40263 + .quad 0x4d3a0ea133ea1107 + .quad 0x54f70be7e33af8c9 + .quad 0x93d3cbe9bdd8f0a4 + .quad 0xdb152c1bfd177302 + .quad 0x7dbddc6d7f17a875 + .quad 0x3e1a71cc8f426efe + .quad 0xc83ca3e390babd62 + .quad 0x80ede3670291c833 + .quad 0xc88038ccd37900c4 + .quad 0x2c5fc0231ec31fa1 + + // 2^252 * 3 * G + + .quad 0xfeba911717038b4f + .quad 0xe5123721c9deef81 + .quad 0x1c97e4e75d0d8834 + .quad 0x68afae7a23dc3bc6 + .quad 0xc422e4d102456e65 + .quad 0x87414ac1cad47b91 + .quad 0x1592e2bba2b6ffdd + .quad 0x75d9d2bff5c2100f + .quad 0x5bd9b4763626e81c + .quad 0x89966936bca02edd + .quad 0x0a41193d61f077b3 + .quad 0x3097a24200ce5471 + + // 2^252 * 4 * G + + .quad 0x57427734c7f8b84c + .quad 0xf141a13e01b270e9 + .quad 0x02d1adfeb4e564a6 + .quad 0x4bb23d92ce83bd48 + .quad 0xa162e7246695c486 + .quad 0x131d633435a89607 + .quad 0x30521561a0d12a37 + .quad 0x56704bada6afb363 + .quad 0xaf6c4aa752f912b9 + .quad 0x5e665f6cd86770c8 + .quad 0x4c35ac83a3c8cd58 + .quad 0x2b7a29c010a58a7e + + // 2^252 * 5 * G + + .quad 0xc4007f77d0c1cec3 + .quad 0x8d1020b6bac492f8 + .quad 0x32ec29d57e69daaf + .quad 0x599408759d95fce0 + .quad 0x33810a23bf00086e + .quad 0xafce925ee736ff7c + .quad 0x3d60e670e24922d4 + .quad 0x11ce9e714f96061b + .quad 0x219ef713d815bac1 + .quad 0xf141465d485be25c + .quad 0x6d5447cc4e513c51 + .quad 0x174926be5ef44393 + + // 2^252 * 6 * G + + .quad 0xb5deb2f9fc5bd5bb + .quad 0x92daa72ae1d810e1 + .quad 0xafc4cfdcb72a1c59 + .quad 0x497d78813fc22a24 + .quad 0x3ef5d41593ea022e + .quad 0x5cbcc1a20ed0eed6 + .quad 0x8fd24ecf07382c8c + .quad 0x6fa42ead06d8e1ad + .quad 0xe276824a1f73371f + .quad 0x7f7cf01c4f5b6736 + .quad 0x7e201fe304fa46e7 + .quad 0x785a36a357808c96 + + // 2^252 * 7 * G + + .quad 0x825fbdfd63014d2b + .quad 0xc852369c6ca7578b + .quad 0x5b2fcd285c0b5df0 + .quad 0x12ab214c58048c8f + .quad 0x070442985d517bc3 + .quad 0x6acd56c7ae653678 + .quad 0x00a27983985a7763 + .quad 0x5167effae512662b + .quad 0xbd4ea9e10f53c4b6 + .quad 0x1673dc5f8ac91a14 + .quad 0xa8f81a4e2acc1aba + .quad 0x33a92a7924332a25 + + // 2^252 * 8 * G + + .quad 0x9dd1f49927996c02 + .quad 0x0cb3b058e04d1752 + .quad 0x1f7e88967fd02c3e + .quad 0x2f964268cb8b3eb1 + .quad 0x7ba95ba0218f2ada + .quad 0xcff42287330fb9ca + .quad 0xdada496d56c6d907 + .quad 0x5380c296f4beee54 + .quad 0x9d4f270466898d0a + .quad 0x3d0987990aff3f7a + .quad 0xd09ef36267daba45 + .quad 0x7761455e7b1c669c diff --git a/cbits/s2n/arm/edwards25519_encode.S b/cbits/s2n/arm/edwards25519_encode.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/edwards25519_encode.S @@ -0,0 +1,136 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Encode edwards25519 point into compressed form as 256-bit number +// Input p[8]; output z[32] (bytes) +// +// extern void edwards25519_encode(uint8_t z[static 32], +// const uint64_t p[static 8]); +// +// This assumes that the input buffer p points to a pair of 256-bit +// numbers x (at p) and y (at p+4) representing a point (x,y) on the +// edwards25519 curve. It is assumed that both x and y are < p_25519 +// but there is no checking of this, nor of the fact that (x,y) is +// in fact on the curve. +// +// The output in z is a little-endian array of bytes corresponding to +// the standard compressed encoding of a point as 2^255 * x_0 + y +// where x_0 is the least significant bit of x. +// See "https://datatracker.ietf.org/doc/html/rfc8032#section-5.1.2" +// In this implementation, y is simply truncated to 255 bits, but if +// it is reduced mod p_25519 as expected this does not affect values. +// +// Standard ARM ABI: X0 = z, X1 = p +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(edwards25519_encode) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(edwards25519_encode) + S2N_BN_SYM_PRIVACY_DIRECTIVE(edwards25519_encode) + .text + .balign 4 + +#define z x0 +#define p x1 + +#define y0 x2 +#define y1 x3 +#define y2 x4 +#define y3 x5 +#define y0short w2 +#define y1short w3 +#define y2short w4 +#define y3short w5 +#define xb x6 + +S2N_BN_SYMBOL(edwards25519_encode): + CFI_START + +// Load lowest word of x coordinate in xb and full y as [y3;y2;y1;y0]. + + ldr xb, [p] + ldp y0, y1, [p, #32] + ldp y2, y3, [p, #48] + +// Compute the encoded form, making the LSB of x the MSB of the encoding + + and y3, y3, #0x7FFFFFFFFFFFFFFF + orr y3, y3, xb, lsl #63 + +// Write back in a byte-oriented fashion to be independent of endianness + + strb y0short, [z] + lsr y0, y0, #8 + strb y0short, [z, #1] + lsr y0, y0, #8 + strb y0short, [z, #2] + lsr y0, y0, #8 + strb y0short, [z, #3] + lsr y0, y0, #8 + strb y0short, [z, #4] + lsr y0, y0, #8 + strb y0short, [z, #5] + lsr y0, y0, #8 + strb y0short, [z, #6] + lsr y0, y0, #8 + strb y0short, [z, #7] + + strb y1short, [z, #8] + lsr y1, y1, #8 + strb y1short, [z, #9] + lsr y1, y1, #8 + strb y1short, [z, #10] + lsr y1, y1, #8 + strb y1short, [z, #11] + lsr y1, y1, #8 + strb y1short, [z, #12] + lsr y1, y1, #8 + strb y1short, [z, #13] + lsr y1, y1, #8 + strb y1short, [z, #14] + lsr y1, y1, #8 + strb y1short, [z, #15] + + strb y2short, [z, #16] + lsr y2, y2, #8 + strb y2short, [z, #17] + lsr y2, y2, #8 + strb y2short, [z, #18] + lsr y2, y2, #8 + strb y2short, [z, #19] + lsr y2, y2, #8 + strb y2short, [z, #20] + lsr y2, y2, #8 + strb y2short, [z, #21] + lsr y2, y2, #8 + strb y2short, [z, #22] + lsr y2, y2, #8 + strb y2short, [z, #23] + + strb y3short, [z, #24] + lsr y3, y3, #8 + strb y3short, [z, #25] + lsr y3, y3, #8 + strb y3short, [z, #26] + lsr y3, y3, #8 + strb y3short, [z, #27] + lsr y3, y3, #8 + strb y3short, [z, #28] + lsr y3, y3, #8 + strb y3short, [z, #29] + lsr y3, y3, #8 + strb y3short, [z, #30] + lsr y3, y3, #8 + strb y3short, [z, #31] + +// Return + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(edwards25519_encode) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/arm/edwards25519_scalarmulbase.S b/cbits/s2n/arm/edwards25519_scalarmulbase.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/edwards25519_scalarmulbase.S @@ -0,0 +1,9635 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Scalar multiplication for the edwards25519 standard basepoint +// Input scalar[4]; output res[8] +// +// extern void edwards25519_scalarmulbase +// (uint64_t res[static 8],const uint64_t scalar[static 4]); +// +// Given a scalar n, returns point (X,Y) = n * B where B = (...,4/5) is +// the standard basepoint for the edwards25519 (Ed25519) curve. +// +// Standard ARM ABI: X0 = res, X1 = scalar +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(edwards25519_scalarmulbase) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(edwards25519_scalarmulbase) + S2N_BN_SYM_PRIVACY_DIRECTIVE(edwards25519_scalarmulbase) + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Stable home for the input result argument during the whole body + +#define res x23 + +// Other variables that are only needed prior to the modular inverse. + +#define tab x19 + +#define i x20 + +#define bias x21 + +#define bf x22 +#define ix x22 + +// Pointer-offset pairs for result and temporaries on stack with some aliasing. + +#define resx res, #(0*NUMSIZE) +#define resy res, #(1*NUMSIZE) + +#define scalar sp, #(0*NUMSIZE) + +#define tabent sp, #(1*NUMSIZE) +#define ymx_2 sp, #(1*NUMSIZE) +#define xpy_2 sp, #(2*NUMSIZE) +#define kxy_2 sp, #(3*NUMSIZE) + +#define acc sp, #(4*NUMSIZE) +#define x_1 sp, #(4*NUMSIZE) +#define y_1 sp, #(5*NUMSIZE) +#define z_1 sp, #(6*NUMSIZE) +#define w_1 sp, #(7*NUMSIZE) +#define x_3 sp, #(4*NUMSIZE) +#define y_3 sp, #(5*NUMSIZE) +#define z_3 sp, #(6*NUMSIZE) +#define w_3 sp, #(7*NUMSIZE) + +#define tmpspace sp, #(8*NUMSIZE) +#define t0 sp, #(8*NUMSIZE) +#define t1 sp, #(9*NUMSIZE) +#define t2 sp, #(10*NUMSIZE) +#define t3 sp, #(11*NUMSIZE) +#define t4 sp, #(12*NUMSIZE) +#define t5 sp, #(13*NUMSIZE) + +// Total size to reserve on the stack + +#define NSPACE 14*NUMSIZE + +// Load 64-bit immediate into a register + +#define movbig(nn,n3,n2,n1,n0) \ + movz nn, n0 __LF \ + movk nn, n1, lsl #16 __LF \ + movk nn, n2, lsl #32 __LF \ + movk nn, n3, lsl #48 + +// Macro wrapping up the basic field operation bignum_mul_p25519, only +// trivially different from a pure function call to that subroutine. + +#define mul_p25519(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x5, x6, [P2] __LF \ + umull x7, w3, w5 __LF \ + lsr x0, x3, #32 __LF \ + umull x15, w0, w5 __LF \ + lsr x16, x5, #32 __LF \ + umull x8, w16, w0 __LF \ + umull x16, w3, w16 __LF \ + adds x7, x7, x15, lsl #32 __LF \ + lsr x15, x15, #32 __LF \ + adc x8, x8, x15 __LF \ + adds x7, x7, x16, lsl #32 __LF \ + lsr x16, x16, #32 __LF \ + adc x8, x8, x16 __LF \ + mul x9, x4, x6 __LF \ + umulh x10, x4, x6 __LF \ + subs x4, x4, x3 __LF \ + cneg x4, x4, cc __LF \ + csetm x16, cc __LF \ + adds x9, x9, x8 __LF \ + adc x10, x10, xzr __LF \ + subs x3, x5, x6 __LF \ + cneg x3, x3, cc __LF \ + cinv x16, x16, cc __LF \ + mul x15, x4, x3 __LF \ + umulh x3, x4, x3 __LF \ + adds x8, x7, x9 __LF \ + adcs x9, x9, x10 __LF \ + adc x10, x10, xzr __LF \ + cmn x16, #0x1 __LF \ + eor x15, x15, x16 __LF \ + adcs x8, x15, x8 __LF \ + eor x3, x3, x16 __LF \ + adcs x9, x3, x9 __LF \ + adc x10, x10, x16 __LF \ + ldp x3, x4, [P1+16] __LF \ + ldp x5, x6, [P2+16] __LF \ + umull x11, w3, w5 __LF \ + lsr x0, x3, #32 __LF \ + umull x15, w0, w5 __LF \ + lsr x16, x5, #32 __LF \ + umull x12, w16, w0 __LF \ + umull x16, w3, w16 __LF \ + adds x11, x11, x15, lsl #32 __LF \ + lsr x15, x15, #32 __LF \ + adc x12, x12, x15 __LF \ + adds x11, x11, x16, lsl #32 __LF \ + lsr x16, x16, #32 __LF \ + adc x12, x12, x16 __LF \ + mul x13, x4, x6 __LF \ + umulh x14, x4, x6 __LF \ + subs x4, x4, x3 __LF \ + cneg x4, x4, cc __LF \ + csetm x16, cc __LF \ + adds x13, x13, x12 __LF \ + adc x14, x14, xzr __LF \ + subs x3, x5, x6 __LF \ + cneg x3, x3, cc __LF \ + cinv x16, x16, cc __LF \ + mul x15, x4, x3 __LF \ + umulh x3, x4, x3 __LF \ + adds x12, x11, x13 __LF \ + adcs x13, x13, x14 __LF \ + adc x14, x14, xzr __LF \ + cmn x16, #0x1 __LF \ + eor x15, x15, x16 __LF \ + adcs x12, x15, x12 __LF \ + eor x3, x3, x16 __LF \ + adcs x13, x3, x13 __LF \ + adc x14, x14, x16 __LF \ + ldp x3, x4, [P1+16] __LF \ + ldp x15, x16, [P1] __LF \ + subs x3, x3, x15 __LF \ + sbcs x4, x4, x16 __LF \ + csetm x16, cc __LF \ + ldp x15, x0, [P2] __LF \ + subs x5, x15, x5 __LF \ + sbcs x6, x0, x6 __LF \ + csetm x0, cc __LF \ + eor x3, x3, x16 __LF \ + subs x3, x3, x16 __LF \ + eor x4, x4, x16 __LF \ + sbc x4, x4, x16 __LF \ + eor x5, x5, x0 __LF \ + subs x5, x5, x0 __LF \ + eor x6, x6, x0 __LF \ + sbc x6, x6, x0 __LF \ + eor x16, x0, x16 __LF \ + adds x11, x11, x9 __LF \ + adcs x12, x12, x10 __LF \ + adcs x13, x13, xzr __LF \ + adc x14, x14, xzr __LF \ + mul x2, x3, x5 __LF \ + umulh x0, x3, x5 __LF \ + mul x15, x4, x6 __LF \ + umulh x1, x4, x6 __LF \ + subs x4, x4, x3 __LF \ + cneg x4, x4, cc __LF \ + csetm x9, cc __LF \ + adds x15, x15, x0 __LF \ + adc x1, x1, xzr __LF \ + subs x6, x5, x6 __LF \ + cneg x6, x6, cc __LF \ + cinv x9, x9, cc __LF \ + mul x5, x4, x6 __LF \ + umulh x6, x4, x6 __LF \ + adds x0, x2, x15 __LF \ + adcs x15, x15, x1 __LF \ + adc x1, x1, xzr __LF \ + cmn x9, #0x1 __LF \ + eor x5, x5, x9 __LF \ + adcs x0, x5, x0 __LF \ + eor x6, x6, x9 __LF \ + adcs x15, x6, x15 __LF \ + adc x1, x1, x9 __LF \ + adds x9, x11, x7 __LF \ + adcs x10, x12, x8 __LF \ + adcs x11, x13, x11 __LF \ + adcs x12, x14, x12 __LF \ + adcs x13, x13, xzr __LF \ + adc x14, x14, xzr __LF \ + cmn x16, #0x1 __LF \ + eor x2, x2, x16 __LF \ + adcs x9, x2, x9 __LF \ + eor x0, x0, x16 __LF \ + adcs x10, x0, x10 __LF \ + eor x15, x15, x16 __LF \ + adcs x11, x15, x11 __LF \ + eor x1, x1, x16 __LF \ + adcs x12, x1, x12 __LF \ + adcs x13, x13, x16 __LF \ + adc x14, x14, x16 __LF \ + mov x3, #0x26 __LF \ + umull x4, w11, w3 __LF \ + add x4, x4, w7, uxtw __LF \ + lsr x7, x7, #32 __LF \ + lsr x11, x11, #32 __LF \ + umaddl x11, w11, w3, x7 __LF \ + mov x7, x4 __LF \ + umull x4, w12, w3 __LF \ + add x4, x4, w8, uxtw __LF \ + lsr x8, x8, #32 __LF \ + lsr x12, x12, #32 __LF \ + umaddl x12, w12, w3, x8 __LF \ + mov x8, x4 __LF \ + umull x4, w13, w3 __LF \ + add x4, x4, w9, uxtw __LF \ + lsr x9, x9, #32 __LF \ + lsr x13, x13, #32 __LF \ + umaddl x13, w13, w3, x9 __LF \ + mov x9, x4 __LF \ + umull x4, w14, w3 __LF \ + add x4, x4, w10, uxtw __LF \ + lsr x10, x10, #32 __LF \ + lsr x14, x14, #32 __LF \ + umaddl x14, w14, w3, x10 __LF \ + mov x10, x4 __LF \ + lsr x0, x14, #31 __LF \ + mov x5, #0x13 __LF \ + umaddl x5, w5, w0, x5 __LF \ + add x7, x7, x5 __LF \ + adds x7, x7, x11, lsl #32 __LF \ + extr x3, x12, x11, #32 __LF \ + adcs x8, x8, x3 __LF \ + extr x3, x13, x12, #32 __LF \ + adcs x9, x9, x3 __LF \ + extr x3, x14, x13, #32 __LF \ + lsl x5, x0, #63 __LF \ + eor x10, x10, x5 __LF \ + adc x10, x10, x3 __LF \ + mov x3, #0x13 __LF \ + tst x10, #0x8000000000000000 __LF \ + csel x3, x3, xzr, pl __LF \ + subs x7, x7, x3 __LF \ + sbcs x8, x8, xzr __LF \ + sbcs x9, x9, xzr __LF \ + sbc x10, x10, xzr __LF \ + and x10, x10, #0x7fffffffffffffff __LF \ + stp x7, x8, [P0] __LF \ + stp x9, x10, [P0+16] + +// A version of multiplication that only guarantees output < 2 * p_25519. +// This basically skips the +1 and final correction in quotient estimation. + +#define mul_4(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x5, x6, [P2] __LF \ + umull x7, w3, w5 __LF \ + lsr x0, x3, #32 __LF \ + umull x15, w0, w5 __LF \ + lsr x16, x5, #32 __LF \ + umull x8, w16, w0 __LF \ + umull x16, w3, w16 __LF \ + adds x7, x7, x15, lsl #32 __LF \ + lsr x15, x15, #32 __LF \ + adc x8, x8, x15 __LF \ + adds x7, x7, x16, lsl #32 __LF \ + lsr x16, x16, #32 __LF \ + adc x8, x8, x16 __LF \ + mul x9, x4, x6 __LF \ + umulh x10, x4, x6 __LF \ + subs x4, x4, x3 __LF \ + cneg x4, x4, cc __LF \ + csetm x16, cc __LF \ + adds x9, x9, x8 __LF \ + adc x10, x10, xzr __LF \ + subs x3, x5, x6 __LF \ + cneg x3, x3, cc __LF \ + cinv x16, x16, cc __LF \ + mul x15, x4, x3 __LF \ + umulh x3, x4, x3 __LF \ + adds x8, x7, x9 __LF \ + adcs x9, x9, x10 __LF \ + adc x10, x10, xzr __LF \ + cmn x16, #0x1 __LF \ + eor x15, x15, x16 __LF \ + adcs x8, x15, x8 __LF \ + eor x3, x3, x16 __LF \ + adcs x9, x3, x9 __LF \ + adc x10, x10, x16 __LF \ + ldp x3, x4, [P1+16] __LF \ + ldp x5, x6, [P2+16] __LF \ + umull x11, w3, w5 __LF \ + lsr x0, x3, #32 __LF \ + umull x15, w0, w5 __LF \ + lsr x16, x5, #32 __LF \ + umull x12, w16, w0 __LF \ + umull x16, w3, w16 __LF \ + adds x11, x11, x15, lsl #32 __LF \ + lsr x15, x15, #32 __LF \ + adc x12, x12, x15 __LF \ + adds x11, x11, x16, lsl #32 __LF \ + lsr x16, x16, #32 __LF \ + adc x12, x12, x16 __LF \ + mul x13, x4, x6 __LF \ + umulh x14, x4, x6 __LF \ + subs x4, x4, x3 __LF \ + cneg x4, x4, cc __LF \ + csetm x16, cc __LF \ + adds x13, x13, x12 __LF \ + adc x14, x14, xzr __LF \ + subs x3, x5, x6 __LF \ + cneg x3, x3, cc __LF \ + cinv x16, x16, cc __LF \ + mul x15, x4, x3 __LF \ + umulh x3, x4, x3 __LF \ + adds x12, x11, x13 __LF \ + adcs x13, x13, x14 __LF \ + adc x14, x14, xzr __LF \ + cmn x16, #0x1 __LF \ + eor x15, x15, x16 __LF \ + adcs x12, x15, x12 __LF \ + eor x3, x3, x16 __LF \ + adcs x13, x3, x13 __LF \ + adc x14, x14, x16 __LF \ + ldp x3, x4, [P1+16] __LF \ + ldp x15, x16, [P1] __LF \ + subs x3, x3, x15 __LF \ + sbcs x4, x4, x16 __LF \ + csetm x16, cc __LF \ + ldp x15, x0, [P2] __LF \ + subs x5, x15, x5 __LF \ + sbcs x6, x0, x6 __LF \ + csetm x0, cc __LF \ + eor x3, x3, x16 __LF \ + subs x3, x3, x16 __LF \ + eor x4, x4, x16 __LF \ + sbc x4, x4, x16 __LF \ + eor x5, x5, x0 __LF \ + subs x5, x5, x0 __LF \ + eor x6, x6, x0 __LF \ + sbc x6, x6, x0 __LF \ + eor x16, x0, x16 __LF \ + adds x11, x11, x9 __LF \ + adcs x12, x12, x10 __LF \ + adcs x13, x13, xzr __LF \ + adc x14, x14, xzr __LF \ + mul x2, x3, x5 __LF \ + umulh x0, x3, x5 __LF \ + mul x15, x4, x6 __LF \ + umulh x1, x4, x6 __LF \ + subs x4, x4, x3 __LF \ + cneg x4, x4, cc __LF \ + csetm x9, cc __LF \ + adds x15, x15, x0 __LF \ + adc x1, x1, xzr __LF \ + subs x6, x5, x6 __LF \ + cneg x6, x6, cc __LF \ + cinv x9, x9, cc __LF \ + mul x5, x4, x6 __LF \ + umulh x6, x4, x6 __LF \ + adds x0, x2, x15 __LF \ + adcs x15, x15, x1 __LF \ + adc x1, x1, xzr __LF \ + cmn x9, #0x1 __LF \ + eor x5, x5, x9 __LF \ + adcs x0, x5, x0 __LF \ + eor x6, x6, x9 __LF \ + adcs x15, x6, x15 __LF \ + adc x1, x1, x9 __LF \ + adds x9, x11, x7 __LF \ + adcs x10, x12, x8 __LF \ + adcs x11, x13, x11 __LF \ + adcs x12, x14, x12 __LF \ + adcs x13, x13, xzr __LF \ + adc x14, x14, xzr __LF \ + cmn x16, #0x1 __LF \ + eor x2, x2, x16 __LF \ + adcs x9, x2, x9 __LF \ + eor x0, x0, x16 __LF \ + adcs x10, x0, x10 __LF \ + eor x15, x15, x16 __LF \ + adcs x11, x15, x11 __LF \ + eor x1, x1, x16 __LF \ + adcs x12, x1, x12 __LF \ + adcs x13, x13, x16 __LF \ + adc x14, x14, x16 __LF \ + mov x3, #0x26 __LF \ + umull x4, w11, w3 __LF \ + add x4, x4, w7, uxtw __LF \ + lsr x7, x7, #32 __LF \ + lsr x11, x11, #32 __LF \ + umaddl x11, w11, w3, x7 __LF \ + mov x7, x4 __LF \ + umull x4, w12, w3 __LF \ + add x4, x4, w8, uxtw __LF \ + lsr x8, x8, #32 __LF \ + lsr x12, x12, #32 __LF \ + umaddl x12, w12, w3, x8 __LF \ + mov x8, x4 __LF \ + umull x4, w13, w3 __LF \ + add x4, x4, w9, uxtw __LF \ + lsr x9, x9, #32 __LF \ + lsr x13, x13, #32 __LF \ + umaddl x13, w13, w3, x9 __LF \ + mov x9, x4 __LF \ + umull x4, w14, w3 __LF \ + add x4, x4, w10, uxtw __LF \ + lsr x10, x10, #32 __LF \ + lsr x14, x14, #32 __LF \ + umaddl x14, w14, w3, x10 __LF \ + mov x10, x4 __LF \ + lsr x0, x14, #31 __LF \ + mov x5, #0x13 __LF \ + umull x5, w5, w0 __LF \ + add x7, x7, x5 __LF \ + adds x7, x7, x11, lsl #32 __LF \ + extr x3, x12, x11, #32 __LF \ + adcs x8, x8, x3 __LF \ + extr x3, x13, x12, #32 __LF \ + adcs x9, x9, x3 __LF \ + extr x3, x14, x13, #32 __LF \ + lsl x5, x0, #63 __LF \ + eor x10, x10, x5 __LF \ + adc x10, x10, x3 __LF \ + stp x7, x8, [P0] __LF \ + stp x9, x10, [P0+16] + +// Modular subtraction with double modulus 2 * p_25519 = 2^256 - 38 + +#define sub_twice4(P0,P1,P2) \ + ldp x5, x6, [P1] __LF \ + ldp x4, x3, [P2] __LF \ + subs x5, x5, x4 __LF \ + sbcs x6, x6, x3 __LF \ + ldp x7, x8, [P1+16] __LF \ + ldp x4, x3, [P2+16] __LF \ + sbcs x7, x7, x4 __LF \ + sbcs x8, x8, x3 __LF \ + mov x4, #38 __LF \ + csel x3, x4, xzr, lo __LF \ + subs x5, x5, x3 __LF \ + sbcs x6, x6, xzr __LF \ + sbcs x7, x7, xzr __LF \ + sbc x8, x8, xzr __LF \ + stp x5, x6, [P0] __LF \ + stp x7, x8, [P0+16] + +// Modular addition and doubling with double modulus 2 * p_25519 = 2^256 - 38. +// This only ensures that the result fits in 4 digits, not that it is reduced +// even w.r.t. double modulus. The result is always correct modulo provided +// the sum of the inputs is < 2^256 + 2^256 - 38, so in particular provided +// at least one of them is reduced double modulo. + +#define add_twice4(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x7, x8, [P2] __LF \ + adds x3, x3, x7 __LF \ + adcs x4, x4, x8 __LF \ + ldp x5, x6, [P1+16] __LF \ + ldp x7, x8, [P2+16] __LF \ + adcs x5, x5, x7 __LF \ + adcs x6, x6, x8 __LF \ + mov x9, #38 __LF \ + csel x9, x9, xzr, cs __LF \ + adds x3, x3, x9 __LF \ + adcs x4, x4, xzr __LF \ + adcs x5, x5, xzr __LF \ + adc x6, x6, xzr __LF \ + stp x3, x4, [P0] __LF \ + stp x5, x6, [P0+16] + +#define double_twice4(P0,P1) \ + ldp x3, x4, [P1] __LF \ + adds x3, x3, x3 __LF \ + adcs x4, x4, x4 __LF \ + ldp x5, x6, [P1+16] __LF \ + adcs x5, x5, x5 __LF \ + adcs x6, x6, x6 __LF \ + mov x9, #38 __LF \ + csel x9, x9, xzr, cs __LF \ + adds x3, x3, x9 __LF \ + adcs x4, x4, xzr __LF \ + adcs x5, x5, xzr __LF \ + adc x6, x6, xzr __LF \ + stp x3, x4, [P0] __LF \ + stp x5, x6, [P0+16] + +S2N_BN_SYMBOL(edwards25519_scalarmulbase): + CFI_START + +// Save regs and make room for temporaries + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_DEC_SP(NSPACE) + +// Move the output pointer to a stable place + + mov res, x0 + +// Copy the input scalar x to its local variable while reducing it +// modulo 2^252 + m where m = 27742317777372353535851937790883648493; +// this is the order of the basepoint so this doesn't change the result. +// First do q = floor(x/2^252) and x' = x - q * (2^252 + m), which gives +// an initial result -15 * m <= x' < 2^252 + + ldp x10, x11, [x1] + ldp x12, x13, [x1, #16] + + lsr x9, x13, #60 + + movbig(x0,#0x5812,#0x631a,#0x5cf5,#0xd3ed); + movbig(x1,#0x14de,#0xf9de,#0xa2f7,#0x9cd6); + + mul x2, x9, x0 + mul x3, x9, x1 + umulh x4, x9, x0 + umulh x5, x9, x1 + + adds x3, x3, x4 + adc x4, x5, xzr + lsl x5, x9, #60 + + subs x10, x10, x2 + sbcs x11, x11, x3 + sbcs x12, x12, x4 + sbcs x13, x13, x5 + +// If x' < 0 then just directly negate it; this makes sure the +// reduced argument is strictly 0 <= x' < 2^252, but now we need +// to record (done via bit 255 of the reduced scalar, which is +// ignored in the main loop) when we negated so we can flip +// the sign of the eventual point to compensate. + + csetm x9, cc + adds xzr, x9, x9 + eor x10, x10, x9 + adcs x10, x10, xzr + eor x11, x11, x9 + adcs x11, x11, xzr + eor x12, x12, x9 + adcs x12, x12, xzr + eor x13, x13, x9 + adc x13, x13, xzr + + and x9, x9, #0x8000000000000000 + orr x13, x13, x9 + +// And before we store the scalar, test and reset bit 251 to +// initialize the main loop just below. + + stp x10, x11, [scalar] + tst x13, #0x0800000000000000 + bic x13, x13, #0x0800000000000000 + stp x12, x13, [scalar+16] + +// The main part of the computation is in extended-projective coordinates +// (X,Y,Z,T), representing an affine point on the edwards25519 curve +// (x,y) via x = X/Z, y = Y/Z and x * y = T/Z (so X * Y = T * Z). +// In comments B means the standard basepoint (x,4/5) = +// (0x216....f25d51a,0x6666..666658). +// +// Initialize accumulator "acc" to either 0 or 2^251 * B depending on +// bit 251 of the (reduced) scalar. That leaves bits 0..250 to handle. + +#if defined(__ELF__) + adrp tab, S2N_BN_SYMBOL(edwards25519_scalarmulbase_constant) + add tab, tab, :lo12:S2N_BN_SYMBOL(edwards25519_scalarmulbase_constant) +#else + adrp tab, S2N_BN_SYMBOL(edwards25519_scalarmulbase_constant)@PAGE + add tab, tab, S2N_BN_SYMBOL(edwards25519_scalarmulbase_constant)@PAGEOFF +#endif + + ldp x0, x1, [tab] + ldp x2, x3, [tab, #96] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc] + + ldp x0, x1, [tab, #1*16] + ldp x2, x3, [tab, #(96+1*16)] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+1*16] + + ldp x0, x1, [tab, #2*16] + ldp x2, x3, [tab, #(96+2*16)] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+2*16] + + ldp x0, x1, [tab, #3*16] + ldp x2, x3, [tab, #(96+3*16)] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+3*16] + + mov x0, #1 + stp x0, xzr, [acc+4*16] + stp xzr, xzr, [acc+5*16] + + ldp x0, x1, [tab, #4*16] + ldp x2, x3, [tab, #(96+4*16)] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+6*16] + + ldp x0, x1, [tab, #5*16] + ldp x2, x3, [tab, #(96+5*16)] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+7*16] + +// The counter "i" tracks the bit position for which the scalar has +// already been absorbed, starting at 0 and going up in chunks of 4. +// +// The pointer "tab" points at the current block of the table for +// multiples (2^i * j) * B at the current bit position i; 1 <= j <= 8. +// +// The bias is always either 0 and 1 and needs to be added to the +// partially processed scalar implicitly. This is used to absorb 4 bits +// of scalar per iteration from 3-bit table indexing by exploiting +// negation: (16 * h + l) * B = (16 * (h + 1) - (16 - l)) * B is used +// when l >= 9. Note that we can't have any bias left over at the +// end because we made sure bit 251 is clear in the reduced scalar. + + mov i, 0 + add tab, tab, #192 + mov bias, xzr + +// Start of the main loop, repeated 63 times for i = 0, 4, 8, ..., 248 + +Ledwards25519_scalarmulbase_scalarloop: + +// Look at the next 4-bit field "bf", adding the previous bias as well. +// Choose the table index "ix" as bf when bf <= 8 and 16 - bf for bf >= 9, +// setting the bias to 1 for the next iteration in the latter case. + + lsr x0, i, #6 + ldr x2, [sp, x0, lsl #3] // Exploiting scalar = sp exactly + lsr x2, x2, i + and x2, x2, #15 + add bf, x2, bias + + cmp bf, 9 + cset bias, cs + + mov x0, 16 + sub x0, x0, bf + cmp bias, xzr + csel ix, x0, bf, ne + +// Perform constant-time lookup in the table to get element number "ix". +// The table entry for the affine point (x,y) is actually a triple +// (y - x,x + y,2 * d * x * y) to precompute parts of the addition. +// Note that "ix" can be 0, so we set up the appropriate identity first. + + mov x0, #1 + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, #1 + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + mov x9, xzr + mov x10, xzr + mov x11, xzr + + cmp ix, #1 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #2 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #3 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #4 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #5 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #6 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #7 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #8 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + +// We now have the triple from the table in registers as follows +// +// [x3;x2;x1;x0] = y - x +// [x7;x6;x5;x4] = x + y +// [x11;x10;x9;x8] = 2 * d * x * y +// +// In case bias = 1 we need to negate this. For Edwards curves +// -(x,y) = (-x,y), i.e. we need to negate the x coordinate. +// In this processed encoding, that amounts to swapping the +// first two fields and negating the third. +// +// The optional negation here also pretends bias = 0 whenever +// ix = 0 so that it doesn't need to handle the case of zero +// inputs, since no non-trivial table entries are zero. Note +// that in the zero case the whole negation is trivial, and +// so indeed is the swapping. + + cmp bias, #0 + + csel x12, x0, x4, eq + csel x13, x1, x5, eq + csel x14, x2, x6, eq + csel x15, x3, x7, eq + stp x12, x13, [tabent] + stp x14, x15, [tabent+16] + + csel x12, x0, x4, ne + csel x13, x1, x5, ne + csel x14, x2, x6, ne + csel x15, x3, x7, ne + stp x12, x13, [tabent+32] + stp x14, x15, [tabent+48] + + mov x0, #-19 + subs x0, x0, x8 + mov x2, #-1 + sbcs x1, x2, x9 + sbcs x2, x2, x10 + mov x3, #0x7FFFFFFFFFFFFFFF + sbc x3, x3, x11 + + cmp ix, xzr + ccmp bias, xzr, #4, ne + + csel x0, x0, x8, ne + csel x1, x1, x9, ne + stp x0, x1, [tabent+64] + csel x2, x2, x10, ne + csel x3, x3, x11, ne + stp x2, x3, [tabent+80] + +// Extended-projective and precomputed mixed addition. +// This is effectively the same as calling the standalone +// function edwards25519_pepadd_alt(acc,acc,tabent), but we +// only retain slightly weaker normalization < 2 * p_25519 +// throughout the inner loop, so the computation is +// slightly different, and faster overall. + + double_twice4(t0,z_1) + sub_twice4(t1,y_1,x_1) + add_twice4(t2,y_1,x_1) + mul_4(t3,w_1,kxy_2) + mul_4(t1,t1,ymx_2) + mul_4(t2,t2,xpy_2) + sub_twice4(t4,t0,t3) + add_twice4(t0,t0,t3) + sub_twice4(t5,t2,t1) + add_twice4(t1,t2,t1) + mul_4(z_3,t4,t0) + mul_4(x_3,t5,t4) + mul_4(y_3,t0,t1) + mul_4(w_3,t5,t1) + +// End of the main loop; move on by 4 bits. + + add i, i, 4 + cmp i, 252 + bcc Ledwards25519_scalarmulbase_scalarloop + +// Insert the optional negation of the projective X coordinate, and +// so by extension the final affine x coordinate x = X/Z and thus +// the point P = (x,y). We only know X < 2 * p_25519, so we do the +// negation as 2 * p_25519 - X to keep it nonnegative. From this +// point on we don't need any normalization of the coordinates +// except for making sure that they fit in 4 digits. + + ldp x0, x1, [x_3] + ldp x2, x3, [x_3+16] + mov x4, #0xffffffffffffffda + subs x4, x4, x0 + mov x7, #0xffffffffffffffff + sbcs x5, x7, x1 + sbcs x6, x7, x2 + sbc x7, x7, x3 + ldr x10, [scalar+24] + tst x10, #0x8000000000000000 + csel x0, x4, x0, ne + csel x1, x5, x1, ne + csel x2, x6, x2, ne + csel x3, x7, x3, ne + stp x0, x1, [x_3] + stp x2, x3, [x_3+16] + +// Now we need to map out of the extended-projective representation +// (X,Y,Z,W) back to the affine form (x,y) = (X/Z,Y/Z). This means +// first calling the modular inverse to get w_3 = 1/z_3. + + add x0, w_3 + add x1, z_3 + +// Inline copy of bignum_inv_p25519, identical except for stripping out +// the prologue and epilogue saving and restoring registers and making +// and reclaiming room on the stack. For more details and explanations see +// "arm/curve25519/bignum_inv_p25519.S". Note that the stack it uses for +// its own temporaries is 128 bytes, so it has no effect on variables +// that are needed in the rest of our computation here: res, w_3, x_3 +// and y_3. + + mov x20, x0 + mov x10, #0xffffffffffffffed + mov x11, #0xffffffffffffffff + stp x10, x11, [sp] + mov x12, #0x7fffffffffffffff + stp x11, x12, [sp, #16] + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + mov x7, #0x13 + lsr x6, x5, #63 + madd x6, x7, x6, x7 + adds x2, x2, x6 + adcs x3, x3, xzr + adcs x4, x4, xzr + orr x5, x5, #0x8000000000000000 + adcs x5, x5, xzr + csel x6, x7, xzr, cc + subs x2, x2, x6 + sbcs x3, x3, xzr + sbcs x4, x4, xzr + sbc x5, x5, xzr + and x5, x5, #0x7fffffffffffffff + stp x2, x3, [sp, #32] + stp x4, x5, [sp, #48] + stp xzr, xzr, [sp, #64] + stp xzr, xzr, [sp, #80] + mov x10, #0x2099 + movk x10, #0x7502, lsl #16 + movk x10, #0x9e23, lsl #32 + movk x10, #0xa0f9, lsl #48 + mov x11, #0x2595 + movk x11, #0x1d13, lsl #16 + movk x11, #0x8f3f, lsl #32 + movk x11, #0xa8c6, lsl #48 + mov x12, #0x5242 + movk x12, #0x5ac, lsl #16 + movk x12, #0x8938, lsl #32 + movk x12, #0x6c6c, lsl #48 + mov x13, #0x615 + movk x13, #0x4177, lsl #16 + movk x13, #0x8b2, lsl #32 + movk x13, #0x2765, lsl #48 + stp x10, x11, [sp, #96] + stp x12, x13, [sp, #112] + mov x21, #0xa + mov x22, #0x1 + b Ledwards25519_scalarmulbase_invmidloop +Ledwards25519_scalarmulbase_invloop: + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + and x0, x12, x16 + and x1, x13, x17 + add x19, x0, x1 + ldr x7, [sp] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #32] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x7, [sp, #8] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #40] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + adc x6, x6, x1 + extr x4, x2, x4, #59 + str x4, [sp] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + adc x4, x4, x1 + extr x5, x3, x5, #59 + str x5, [sp, #32] + ldr x7, [sp, #16] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #48] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + adc x5, x5, x1 + extr x2, x6, x2, #59 + str x2, [sp, #8] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + adc x2, x2, x1 + extr x3, x4, x3, #59 + str x3, [sp, #40] + ldr x7, [sp, #24] + eor x1, x7, x14 + asr x3, x1, #63 + and x3, x3, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #56] + eor x1, x8, x15 + asr x0, x1, #63 + and x0, x0, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x5, x6, #59 + str x6, [sp, #16] + extr x5, x3, x5, #59 + str x5, [sp, #24] + eor x1, x7, x16 + asr x5, x1, #63 + and x5, x5, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + asr x0, x1, #63 + and x0, x0, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x4, x2, x4, #59 + str x4, [sp, #48] + extr x2, x5, x2, #59 + str x2, [sp, #56] + ldr x7, [sp, #64] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #96] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #64] + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + str x5, [sp, #96] + adc x3, x3, x1 + ldr x7, [sp, #72] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #104] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #72] + adc x6, x6, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + str x3, [sp, #104] + adc x4, x4, x1 + ldr x7, [sp, #80] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #112] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #80] + adc x5, x5, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + str x4, [sp, #112] + adc x2, x2, x1 + ldr x7, [sp, #88] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #120] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x3, x5, #63 + ldp x0, x1, [sp, #64] + add x6, x6, x3, asr #63 + mov x3, #0x13 + mul x4, x6, x3 + add x5, x5, x6, lsl #63 + smulh x3, x6, x3 + ldr x6, [sp, #80] + adds x0, x0, x4 + adcs x1, x1, x3 + asr x3, x3, #63 + adcs x6, x6, x3 + adc x5, x5, x3 + stp x0, x1, [sp, #64] + stp x6, x5, [sp, #80] + eor x1, x7, x16 + and x5, x16, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + and x0, x17, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x6, x5, x2, #63 + ldp x0, x1, [sp, #96] + add x6, x6, x5, asr #63 + mov x5, #0x13 + mul x4, x6, x5 + add x2, x2, x6, lsl #63 + smulh x5, x6, x5 + ldr x3, [sp, #112] + adds x0, x0, x4 + adcs x1, x1, x5 + asr x5, x5, #63 + adcs x3, x3, x5 + adc x2, x2, x5 + stp x0, x1, [sp, #96] + stp x3, x2, [sp, #112] +Ledwards25519_scalarmulbase_invmidloop: + mov x1, x22 + ldr x2, [sp] + ldr x3, [sp, #32] + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x8, x4, #0x100, lsl #12 + sbfx x8, x8, #21, #21 + mov x11, #0x100000 + add x11, x11, x11, lsl #21 + add x9, x4, x11 + asr x9, x9, #42 + add x10, x5, #0x100, lsl #12 + sbfx x10, x10, #21, #21 + add x11, x5, x11 + asr x11, x11, #42 + mul x6, x8, x2 + mul x7, x9, x3 + mul x2, x10, x2 + mul x3, x11, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #21, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #42 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #21, #21 + add x15, x5, x15 + asr x15, x15, #42 + mul x6, x12, x2 + mul x7, x13, x3 + mul x2, x14, x2 + mul x3, x15, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + mul x2, x12, x8 + mul x3, x12, x9 + mul x6, x14, x8 + mul x7, x14, x9 + madd x8, x13, x10, x2 + madd x9, x13, x11, x3 + madd x16, x15, x10, x6 + madd x17, x15, x11, x7 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #22, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #43 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #22, #21 + add x15, x5, x15 + asr x15, x15, #43 + mneg x2, x12, x8 + mneg x3, x12, x9 + mneg x4, x14, x8 + mneg x5, x14, x9 + msub x10, x13, x16, x2 + msub x11, x13, x17, x3 + msub x12, x15, x16, x4 + msub x13, x15, x17, x5 + mov x22, x1 + subs x21, x21, #0x1 + b.ne Ledwards25519_scalarmulbase_invloop + ldr x0, [sp] + ldr x1, [sp, #32] + mul x0, x0, x10 + madd x1, x1, x11, x0 + asr x0, x1, #63 + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + eor x14, x14, x0 + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + eor x15, x15, x0 + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + eor x16, x16, x0 + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + eor x17, x17, x0 + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + ldr x7, [sp, #64] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #96] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #64] + adc x2, x2, x1 + ldr x7, [sp, #72] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #104] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #72] + adc x6, x6, x1 + ldr x7, [sp, #80] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #112] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #80] + adc x5, x5, x1 + ldr x7, [sp, #88] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #120] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x3, x5, #63 + ldp x0, x1, [sp, #64] + tst x3, x3 + cinc x6, x6, pl + mov x3, #0x13 + mul x4, x6, x3 + add x5, x5, x6, lsl #63 + smulh x6, x6, x3 + ldr x2, [sp, #80] + adds x0, x0, x4 + adcs x1, x1, x6 + asr x6, x6, #63 + adcs x2, x2, x6 + adcs x5, x5, x6 + csel x3, x3, xzr, mi + subs x0, x0, x3 + sbcs x1, x1, xzr + sbcs x2, x2, xzr + sbc x5, x5, xzr + and x5, x5, #0x7fffffffffffffff + mov x4, x20 + stp x0, x1, [x4] + stp x2, x5, [x4, #16] + +// The final result is x = X * inv(Z), y = Y * inv(Z). +// These are the only operations in the whole computation that +// fully reduce modulo p_25519 since now we want the canonical +// answer as output. + + mul_p25519(resx,x_3,w_3) + mul_p25519(resy,y_3,w_3) + +// Restore stack and registers + + CFI_INC_SP(NSPACE) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(edwards25519_scalarmulbase) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif + +// **************************************************************************** +// The precomputed data (all read-only). +// **************************************************************************** + +#if defined(__ELF__) +.section .rodata +.type S2N_BN_SYMBOL(edwards25519_scalarmulbase_constant), %object +.size S2N_BN_SYMBOL(edwards25519_scalarmulbase_constant), 48576 +#elif defined(__APPLE__) +.const_data +#endif + +S2N_BN_SYMBOL(edwards25519_scalarmulbase_constant): + +// 0 * B = 0 and 2^251 * B in extended-projective coordinates +// but with Z = 1 assumed and hence left out, so they are (X,Y,T) only. + + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000001 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + + .quad 0x525f946d7c7220e7 + .quad 0x4636b0b2f1e35444 + .quad 0x796e9d70e892ae0f + .quad 0x03dec05fa937adb1 + .quad 0x6d1c271cc6375515 + .quad 0x462588c4a4ca4f14 + .quad 0x691129fee55afc39 + .quad 0x15949f784d8472f5 + .quad 0xbd89e510afad0049 + .quad 0x4d1f08c073b9860e + .quad 0x07716e8b2d00af9d + .quad 0x70d685f68f859714 + +// Precomputed table of multiples of generator for edwards25519 +// all in precomputed extended-projective (y-x,x+y,2*d*x*y) triples. + + // 2^0 * 1 * G + + .quad 0x9d103905d740913e + .quad 0xfd399f05d140beb3 + .quad 0xa5c18434688f8a09 + .quad 0x44fd2f9298f81267 + .quad 0x2fbc93c6f58c3b85 + .quad 0xcf932dc6fb8c0e19 + .quad 0x270b4898643d42c2 + .quad 0x07cf9d3a33d4ba65 + .quad 0xabc91205877aaa68 + .quad 0x26d9e823ccaac49e + .quad 0x5a1b7dcbdd43598c + .quad 0x6f117b689f0c65a8 + + // 2^0 * 2 * G + + .quad 0x8a99a56042b4d5a8 + .quad 0x8f2b810c4e60acf6 + .quad 0xe09e236bb16e37aa + .quad 0x6bb595a669c92555 + .quad 0x9224e7fc933c71d7 + .quad 0x9f469d967a0ff5b5 + .quad 0x5aa69a65e1d60702 + .quad 0x590c063fa87d2e2e + .quad 0x43faa8b3a59b7a5f + .quad 0x36c16bdd5d9acf78 + .quad 0x500fa0840b3d6a31 + .quad 0x701af5b13ea50b73 + + // 2^0 * 3 * G + + .quad 0x56611fe8a4fcd265 + .quad 0x3bd353fde5c1ba7d + .quad 0x8131f31a214bd6bd + .quad 0x2ab91587555bda62 + .quad 0xaf25b0a84cee9730 + .quad 0x025a8430e8864b8a + .quad 0xc11b50029f016732 + .quad 0x7a164e1b9a80f8f4 + .quad 0x14ae933f0dd0d889 + .quad 0x589423221c35da62 + .quad 0xd170e5458cf2db4c + .quad 0x5a2826af12b9b4c6 + + // 2^0 * 4 * G + + .quad 0x95fe050a056818bf + .quad 0x327e89715660faa9 + .quad 0xc3e8e3cd06a05073 + .quad 0x27933f4c7445a49a + .quad 0x287351b98efc099f + .quad 0x6765c6f47dfd2538 + .quad 0xca348d3dfb0a9265 + .quad 0x680e910321e58727 + .quad 0x5a13fbe9c476ff09 + .quad 0x6e9e39457b5cc172 + .quad 0x5ddbdcf9102b4494 + .quad 0x7f9d0cbf63553e2b + + // 2^0 * 5 * G + + .quad 0x7f9182c3a447d6ba + .quad 0xd50014d14b2729b7 + .quad 0xe33cf11cb864a087 + .quad 0x154a7e73eb1b55f3 + .quad 0xa212bc4408a5bb33 + .quad 0x8d5048c3c75eed02 + .quad 0xdd1beb0c5abfec44 + .quad 0x2945ccf146e206eb + .quad 0xbcbbdbf1812a8285 + .quad 0x270e0807d0bdd1fc + .quad 0xb41b670b1bbda72d + .quad 0x43aabe696b3bb69a + + // 2^0 * 6 * G + + .quad 0x499806b67b7d8ca4 + .quad 0x575be28427d22739 + .quad 0xbb085ce7204553b9 + .quad 0x38b64c41ae417884 + .quad 0x3a0ceeeb77157131 + .quad 0x9b27158900c8af88 + .quad 0x8065b668da59a736 + .quad 0x51e57bb6a2cc38bd + .quad 0x85ac326702ea4b71 + .quad 0xbe70e00341a1bb01 + .quad 0x53e4a24b083bc144 + .quad 0x10b8e91a9f0d61e3 + + // 2^0 * 7 * G + + .quad 0xba6f2c9aaa3221b1 + .quad 0x6ca021533bba23a7 + .quad 0x9dea764f92192c3a + .quad 0x1d6edd5d2e5317e0 + .quad 0x6b1a5cd0944ea3bf + .quad 0x7470353ab39dc0d2 + .quad 0x71b2528228542e49 + .quad 0x461bea69283c927e + .quad 0xf1836dc801b8b3a2 + .quad 0xb3035f47053ea49a + .quad 0x529c41ba5877adf3 + .quad 0x7a9fbb1c6a0f90a7 + + // 2^0 * 8 * G + + .quad 0xe2a75dedf39234d9 + .quad 0x963d7680e1b558f9 + .quad 0x2c2741ac6e3c23fb + .quad 0x3a9024a1320e01c3 + .quad 0x59b7596604dd3e8f + .quad 0x6cb30377e288702c + .quad 0xb1339c665ed9c323 + .quad 0x0915e76061bce52f + .quad 0xe7c1f5d9c9a2911a + .quad 0xb8a371788bcca7d7 + .quad 0x636412190eb62a32 + .quad 0x26907c5c2ecc4e95 + + // 2^4 * 1 * B + + .quad 0x7ec851ca553e2df3 + .quad 0xa71284cba64878b3 + .quad 0xe6b5e4193288d1e7 + .quad 0x4cf210ec5a9a8883 + .quad 0x322d04a52d9021f6 + .quad 0xb9c19f3375c6bf9c + .quad 0x587a3a4342d20b09 + .quad 0x143b1cf8aa64fe61 + .quad 0x9f867c7d968acaab + .quad 0x5f54258e27092729 + .quad 0xd0a7d34bea180975 + .quad 0x21b546a3374126e1 + + // 2^4 * 2 * B + + .quad 0xa94ff858a2888343 + .quad 0xce0ed4565313ed3c + .quad 0xf55c3dcfb5bf34fa + .quad 0x0a653ca5c9eab371 + .quad 0x490a7a45d185218f + .quad 0x9a15377846049335 + .quad 0x0060ea09cc31e1f6 + .quad 0x7e041577f86ee965 + .quad 0x66b2a496ce5b67f3 + .quad 0xff5492d8bd569796 + .quad 0x503cec294a592cd0 + .quad 0x566943650813acb2 + + // 2^4 * 3 * B + + .quad 0xb818db0c26620798 + .quad 0x5d5c31d9606e354a + .quad 0x0982fa4f00a8cdc7 + .quad 0x17e12bcd4653e2d4 + .quad 0x5672f9eb1dabb69d + .quad 0xba70b535afe853fc + .quad 0x47ac0f752796d66d + .quad 0x32a5351794117275 + .quad 0xd3a644a6df648437 + .quad 0x703b6559880fbfdd + .quad 0xcb852540ad3a1aa5 + .quad 0x0900b3f78e4c6468 + + // 2^4 * 4 * B + + .quad 0x0a851b9f679d651b + .quad 0xe108cb61033342f2 + .quad 0xd601f57fe88b30a3 + .quad 0x371f3acaed2dd714 + .quad 0xed280fbec816ad31 + .quad 0x52d9595bd8e6efe3 + .quad 0x0fe71772f6c623f5 + .quad 0x4314030b051e293c + .quad 0xd560005efbf0bcad + .quad 0x8eb70f2ed1870c5e + .quad 0x201f9033d084e6a0 + .quad 0x4c3a5ae1ce7b6670 + + // 2^4 * 5 * B + + .quad 0x4138a434dcb8fa95 + .quad 0x870cf67d6c96840b + .quad 0xde388574297be82c + .quad 0x7c814db27262a55a + .quad 0xbaf875e4c93da0dd + .quad 0xb93282a771b9294d + .quad 0x80d63fb7f4c6c460 + .quad 0x6de9c73dea66c181 + .quad 0x478904d5a04df8f2 + .quad 0xfafbae4ab10142d3 + .quad 0xf6c8ac63555d0998 + .quad 0x5aac4a412f90b104 + + // 2^4 * 6 * B + + .quad 0xc64f326b3ac92908 + .quad 0x5551b282e663e1e0 + .quad 0x476b35f54a1a4b83 + .quad 0x1b9da3fe189f68c2 + .quad 0x603a0d0abd7f5134 + .quad 0x8089c932e1d3ae46 + .quad 0xdf2591398798bd63 + .quad 0x1c145cd274ba0235 + .quad 0x32e8386475f3d743 + .quad 0x365b8baf6ae5d9ef + .quad 0x825238b6385b681e + .quad 0x234929c1167d65e1 + + // 2^4 * 7 * B + + .quad 0x984decaba077ade8 + .quad 0x383f77ad19eb389d + .quad 0xc7ec6b7e2954d794 + .quad 0x59c77b3aeb7c3a7a + .quad 0x48145cc21d099fcf + .quad 0x4535c192cc28d7e5 + .quad 0x80e7c1e548247e01 + .quad 0x4a5f28743b2973ee + .quad 0xd3add725225ccf62 + .quad 0x911a3381b2152c5d + .quad 0xd8b39fad5b08f87d + .quad 0x6f05606b4799fe3b + + // 2^4 * 8 * B + + .quad 0x9ffe9e92177ba962 + .quad 0x98aee71d0de5cae1 + .quad 0x3ff4ae942d831044 + .quad 0x714de12e58533ac8 + .quad 0x5b433149f91b6483 + .quad 0xadb5dc655a2cbf62 + .quad 0x87fa8412632827b3 + .quad 0x60895e91ab49f8d8 + .quad 0xe9ecf2ed0cf86c18 + .quad 0xb46d06120735dfd4 + .quad 0xbc9da09804b96be7 + .quad 0x73e2e62fd96dc26b + + // 2^8 * 1 * B + + .quad 0xed5b635449aa515e + .quad 0xa865c49f0bc6823a + .quad 0x850c1fe95b42d1c4 + .quad 0x30d76d6f03d315b9 + .quad 0x2eccdd0e632f9c1d + .quad 0x51d0b69676893115 + .quad 0x52dfb76ba8637a58 + .quad 0x6dd37d49a00eef39 + .quad 0x6c4444172106e4c7 + .quad 0xfb53d680928d7f69 + .quad 0xb4739ea4694d3f26 + .quad 0x10c697112e864bb0 + + // 2^8 * 2 * B + + .quad 0x6493c4277dbe5fde + .quad 0x265d4fad19ad7ea2 + .quad 0x0e00dfc846304590 + .quad 0x25e61cabed66fe09 + .quad 0x0ca62aa08358c805 + .quad 0x6a3d4ae37a204247 + .quad 0x7464d3a63b11eddc + .quad 0x03bf9baf550806ef + .quad 0x3f13e128cc586604 + .quad 0x6f5873ecb459747e + .quad 0xa0b63dedcc1268f5 + .quad 0x566d78634586e22c + + // 2^8 * 3 * B + + .quad 0x1637a49f9cc10834 + .quad 0xbc8e56d5a89bc451 + .quad 0x1cb5ec0f7f7fd2db + .quad 0x33975bca5ecc35d9 + .quad 0xa1054285c65a2fd0 + .quad 0x6c64112af31667c3 + .quad 0x680ae240731aee58 + .quad 0x14fba5f34793b22a + .quad 0x3cd746166985f7d4 + .quad 0x593e5e84c9c80057 + .quad 0x2fc3f2b67b61131e + .quad 0x14829cea83fc526c + + // 2^8 * 4 * B + + .quad 0xff437b8497dd95c2 + .quad 0x6c744e30aa4eb5a7 + .quad 0x9e0c5d613c85e88b + .quad 0x2fd9c71e5f758173 + .quad 0x21e70b2f4e71ecb8 + .quad 0xe656ddb940a477e3 + .quad 0xbf6556cece1d4f80 + .quad 0x05fc3bc4535d7b7e + .quad 0x24b8b3ae52afdedd + .quad 0x3495638ced3b30cf + .quad 0x33a4bc83a9be8195 + .quad 0x373767475c651f04 + + // 2^8 * 5 * B + + .quad 0x2fba99fd40d1add9 + .quad 0xb307166f96f4d027 + .quad 0x4363f05215f03bae + .quad 0x1fbea56c3b18f999 + .quad 0x634095cb14246590 + .quad 0xef12144016c15535 + .quad 0x9e38140c8910bc60 + .quad 0x6bf5905730907c8c + .quad 0x0fa778f1e1415b8a + .quad 0x06409ff7bac3a77e + .quad 0x6f52d7b89aa29a50 + .quad 0x02521cf67a635a56 + + // 2^8 * 6 * B + + .quad 0x513fee0b0a9d5294 + .quad 0x8f98e75c0fdf5a66 + .quad 0xd4618688bfe107ce + .quad 0x3fa00a7e71382ced + .quad 0xb1146720772f5ee4 + .quad 0xe8f894b196079ace + .quad 0x4af8224d00ac824a + .quad 0x001753d9f7cd6cc4 + .quad 0x3c69232d963ddb34 + .quad 0x1dde87dab4973858 + .quad 0xaad7d1f9a091f285 + .quad 0x12b5fe2fa048edb6 + + // 2^8 * 7 * B + + .quad 0x71f0fbc496fce34d + .quad 0x73b9826badf35bed + .quad 0xd2047261ff28c561 + .quad 0x749b76f96fb1206f + .quad 0xdf2b7c26ad6f1e92 + .quad 0x4b66d323504b8913 + .quad 0x8c409dc0751c8bc3 + .quad 0x6f7e93c20796c7b8 + .quad 0x1f5af604aea6ae05 + .quad 0xc12351f1bee49c99 + .quad 0x61a808b5eeff6b66 + .quad 0x0fcec10f01e02151 + + // 2^8 * 8 * B + + .quad 0x644d58a649fe1e44 + .quad 0x21fcaea231ad777e + .quad 0x02441c5a887fd0d2 + .quad 0x4901aa7183c511f3 + .quad 0x3df2d29dc4244e45 + .quad 0x2b020e7493d8de0a + .quad 0x6cc8067e820c214d + .quad 0x413779166feab90a + .quad 0x08b1b7548c1af8f0 + .quad 0xce0f7a7c246299b4 + .quad 0xf760b0f91e06d939 + .quad 0x41bb887b726d1213 + + // 2^12 * 1 * B + + .quad 0x9267806c567c49d8 + .quad 0x066d04ccca791e6a + .quad 0xa69f5645e3cc394b + .quad 0x5c95b686a0788cd2 + .quad 0x97d980e0aa39f7d2 + .quad 0x35d0384252c6b51c + .quad 0x7d43f49307cd55aa + .quad 0x56bd36cfb78ac362 + .quad 0x2ac519c10d14a954 + .quad 0xeaf474b494b5fa90 + .quad 0xe6af8382a9f87a5a + .quad 0x0dea6db1879be094 + + // 2^12 * 2 * B + + .quad 0xaa66bf547344e5ab + .quad 0xda1258888f1b4309 + .quad 0x5e87d2b3fd564b2f + .quad 0x5b2c78885483b1dd + .quad 0x15baeb74d6a8797a + .quad 0x7ef55cf1fac41732 + .quad 0x29001f5a3c8b05c5 + .quad 0x0ad7cc8752eaccfb + .quad 0x52151362793408cf + .quad 0xeb0f170319963d94 + .quad 0xa833b2fa883d9466 + .quad 0x093a7fa775003c78 + + // 2^12 * 3 * B + + .quad 0xe5107de63a16d7be + .quad 0xa377ffdc9af332cf + .quad 0x70d5bf18440b677f + .quad 0x6a252b19a4a31403 + .quad 0xb8e9604460a91286 + .quad 0x7f3fd8047778d3de + .quad 0x67d01e31bf8a5e2d + .quad 0x7b038a06c27b653e + .quad 0x9ed919d5d36990f3 + .quad 0x5213aebbdb4eb9f2 + .quad 0xc708ea054cb99135 + .quad 0x58ded57f72260e56 + + // 2^12 * 4 * B + + .quad 0x78e79dade9413d77 + .quad 0xf257f9d59729e67d + .quad 0x59db910ee37aa7e6 + .quad 0x6aa11b5bbb9e039c + .quad 0xda6d53265b0fd48b + .quad 0x8960823193bfa988 + .quad 0xd78ac93261d57e28 + .quad 0x79f2942d3a5c8143 + .quad 0x97da2f25b6c88de9 + .quad 0x251ba7eaacf20169 + .quad 0x09b44f87ef4eb4e4 + .quad 0x7d90ab1bbc6a7da5 + + // 2^12 * 5 * B + + .quad 0x9acca683a7016bfe + .quad 0x90505f4df2c50b6d + .quad 0x6b610d5fcce435aa + .quad 0x19a10d446198ff96 + .quad 0x1a07a3f496b3c397 + .quad 0x11ceaa188f4e2532 + .quad 0x7d9498d5a7751bf0 + .quad 0x19ed161f508dd8a0 + .quad 0x560a2cd687dce6ca + .quad 0x7f3568c48664cf4d + .quad 0x8741e95222803a38 + .quad 0x483bdab1595653fc + + // 2^12 * 6 * B + + .quad 0xfa780f148734fa49 + .quad 0x106f0b70360534e0 + .quad 0x2210776fe3e307bd + .quad 0x3286c109dde6a0fe + .quad 0xd6cf4d0ab4da80f6 + .quad 0x82483e45f8307fe0 + .quad 0x05005269ae6f9da4 + .quad 0x1c7052909cf7877a + .quad 0x32ee7de2874e98d4 + .quad 0x14c362e9b97e0c60 + .quad 0x5781dcde6a60a38a + .quad 0x217dd5eaaa7aa840 + + // 2^12 * 7 * B + + .quad 0x9db7c4d0248e1eb0 + .quad 0xe07697e14d74bf52 + .quad 0x1e6a9b173c562354 + .quad 0x7fa7c21f795a4965 + .quad 0x8bdf1fb9be8c0ec8 + .quad 0x00bae7f8e30a0282 + .quad 0x4963991dad6c4f6c + .quad 0x07058a6e5df6f60a + .quad 0xe9eb02c4db31f67f + .quad 0xed25fd8910bcfb2b + .quad 0x46c8131f5c5cddb4 + .quad 0x33b21c13a0cb9bce + + // 2^12 * 8 * B + + .quad 0x360692f8087d8e31 + .quad 0xf4dcc637d27163f7 + .quad 0x25a4e62065ea5963 + .quad 0x659bf72e5ac160d9 + .quad 0x9aafb9b05ee38c5b + .quad 0xbf9d2d4e071a13c7 + .quad 0x8eee6e6de933290a + .quad 0x1c3bab17ae109717 + .quad 0x1c9ab216c7cab7b0 + .quad 0x7d65d37407bbc3cc + .quad 0x52744750504a58d5 + .quad 0x09f2606b131a2990 + + // 2^16 * 1 * B + + .quad 0x40e87d44744346be + .quad 0x1d48dad415b52b25 + .quad 0x7c3a8a18a13b603e + .quad 0x4eb728c12fcdbdf7 + .quad 0x7e234c597c6691ae + .quad 0x64889d3d0a85b4c8 + .quad 0xdae2c90c354afae7 + .quad 0x0a871e070c6a9e1d + .quad 0x3301b5994bbc8989 + .quad 0x736bae3a5bdd4260 + .quad 0x0d61ade219d59e3c + .quad 0x3ee7300f2685d464 + + // 2^16 * 2 * B + + .quad 0xf5d255e49e7dd6b7 + .quad 0x8016115c610b1eac + .quad 0x3c99975d92e187ca + .quad 0x13815762979125c2 + .quad 0x43fa7947841e7518 + .quad 0xe5c6fa59639c46d7 + .quad 0xa1065e1de3052b74 + .quad 0x7d47c6a2cfb89030 + .quad 0x3fdad0148ef0d6e0 + .quad 0x9d3e749a91546f3c + .quad 0x71ec621026bb8157 + .quad 0x148cf58d34c9ec80 + + // 2^16 * 3 * B + + .quad 0x46a492f67934f027 + .quad 0x469984bef6840aa9 + .quad 0x5ca1bc2a89611854 + .quad 0x3ff2fa1ebd5dbbd4 + .quad 0xe2572f7d9ae4756d + .quad 0x56c345bb88f3487f + .quad 0x9fd10b6d6960a88d + .quad 0x278febad4eaea1b9 + .quad 0xb1aa681f8c933966 + .quad 0x8c21949c20290c98 + .quad 0x39115291219d3c52 + .quad 0x4104dd02fe9c677b + + // 2^16 * 4 * B + + .quad 0x72b2bf5e1124422a + .quad 0xa1fa0c3398a33ab5 + .quad 0x94cb6101fa52b666 + .quad 0x2c863b00afaf53d5 + .quad 0x81214e06db096ab8 + .quad 0x21a8b6c90ce44f35 + .quad 0x6524c12a409e2af5 + .quad 0x0165b5a48efca481 + .quad 0xf190a474a0846a76 + .quad 0x12eff984cd2f7cc0 + .quad 0x695e290658aa2b8f + .quad 0x591b67d9bffec8b8 + + // 2^16 * 5 * B + + .quad 0x312f0d1c80b49bfa + .quad 0x5979515eabf3ec8a + .quad 0x727033c09ef01c88 + .quad 0x3de02ec7ca8f7bcb + .quad 0x99b9b3719f18b55d + .quad 0xe465e5faa18c641e + .quad 0x61081136c29f05ed + .quad 0x489b4f867030128b + .quad 0xd232102d3aeb92ef + .quad 0xe16253b46116a861 + .quad 0x3d7eabe7190baa24 + .quad 0x49f5fbba496cbebf + + // 2^16 * 6 * B + + .quad 0x30949a108a5bcfd4 + .quad 0xdc40dd70bc6473eb + .quad 0x92c294c1307c0d1c + .quad 0x5604a86dcbfa6e74 + .quad 0x155d628c1e9c572e + .quad 0x8a4d86acc5884741 + .quad 0x91a352f6515763eb + .quad 0x06a1a6c28867515b + .quad 0x7288d1d47c1764b6 + .quad 0x72541140e0418b51 + .quad 0x9f031a6018acf6d1 + .quad 0x20989e89fe2742c6 + + // 2^16 * 7 * B + + .quad 0x499777fd3a2dcc7f + .quad 0x32857c2ca54fd892 + .quad 0xa279d864d207e3a0 + .quad 0x0403ed1d0ca67e29 + .quad 0x1674278b85eaec2e + .quad 0x5621dc077acb2bdf + .quad 0x640a4c1661cbf45a + .quad 0x730b9950f70595d3 + .quad 0xc94b2d35874ec552 + .quad 0xc5e6c8cf98246f8d + .quad 0xf7cb46fa16c035ce + .quad 0x5bd7454308303dcc + + // 2^16 * 8 * B + + .quad 0x7f9ad19528b24cc2 + .quad 0x7f6b54656335c181 + .quad 0x66b8b66e4fc07236 + .quad 0x133a78007380ad83 + .quad 0x85c4932115e7792a + .quad 0xc64c89a2bdcdddc9 + .quad 0x9d1e3da8ada3d762 + .quad 0x5bb7db123067f82c + .quad 0x0961f467c6ca62be + .quad 0x04ec21d6211952ee + .quad 0x182360779bd54770 + .quad 0x740dca6d58f0e0d2 + + // 2^20 * 1 * B + + .quad 0x50b70bf5d3f0af0b + .quad 0x4feaf48ae32e71f7 + .quad 0x60e84ed3a55bbd34 + .quad 0x00ed489b3f50d1ed + .quad 0x3906c72aed261ae5 + .quad 0x9ab68fd988e100f7 + .quad 0xf5e9059af3360197 + .quad 0x0e53dc78bf2b6d47 + .quad 0xb90829bf7971877a + .quad 0x5e4444636d17e631 + .quad 0x4d05c52e18276893 + .quad 0x27632d9a5a4a4af5 + + // 2^20 * 2 * B + + .quad 0xd11ff05154b260ce + .quad 0xd86dc38e72f95270 + .quad 0x601fcd0d267cc138 + .quad 0x2b67916429e90ccd + .quad 0xa98285d187eaffdb + .quad 0xa5b4fbbbd8d0a864 + .quad 0xb658f27f022663f7 + .quad 0x3bbc2b22d99ce282 + .quad 0xb917c952583c0a58 + .quad 0x653ff9b80fe4c6f3 + .quad 0x9b0da7d7bcdf3c0c + .quad 0x43a0eeb6ab54d60e + + // 2^20 * 3 * B + + .quad 0x396966a46d4a5487 + .quad 0xf811a18aac2bb3ba + .quad 0x66e4685b5628b26b + .quad 0x70a477029d929b92 + .quad 0x3ac6322357875fe8 + .quad 0xd9d4f4ecf5fbcb8f + .quad 0x8dee8493382bb620 + .quad 0x50c5eaa14c799fdc + .quad 0xdd0edc8bd6f2fb3c + .quad 0x54c63aa79cc7b7a0 + .quad 0xae0b032b2c8d9f1a + .quad 0x6f9ce107602967fb + + // 2^20 * 4 * B + + .quad 0xad1054b1cde1c22a + .quad 0xc4a8e90248eb32df + .quad 0x5f3e7b33accdc0ea + .quad 0x72364713fc79963e + .quad 0x139693063520e0b5 + .quad 0x437fcf7c88ea03fe + .quad 0xf7d4c40bd3c959bc + .quad 0x699154d1f893ded9 + .quad 0x315d5c75b4b27526 + .quad 0xcccb842d0236daa5 + .quad 0x22f0c8a3345fee8e + .quad 0x73975a617d39dbed + + // 2^20 * 5 * B + + .quad 0xe4024df96375da10 + .quad 0x78d3251a1830c870 + .quad 0x902b1948658cd91c + .quad 0x7e18b10b29b7438a + .quad 0x6f37f392f4433e46 + .quad 0x0e19b9a11f566b18 + .quad 0x220fb78a1fd1d662 + .quad 0x362a4258a381c94d + .quad 0x9071d9132b6beb2f + .quad 0x0f26e9ad28418247 + .quad 0xeab91ec9bdec925d + .quad 0x4be65bc8f48af2de + + // 2^20 * 6 * B + + .quad 0x78487feba36e7028 + .quad 0x5f3f13001dd8ce34 + .quad 0x934fb12d4b30c489 + .quad 0x056c244d397f0a2b + .quad 0x1d50fba257c26234 + .quad 0x7bd4823adeb0678b + .quad 0xc2b0dc6ea6538af5 + .quad 0x5665eec6351da73e + .quad 0xdb3ee00943bfb210 + .quad 0x4972018720800ac2 + .quad 0x26ab5d6173bd8667 + .quad 0x20b209c2ab204938 + + // 2^20 * 7 * B + + .quad 0x549e342ac07fb34b + .quad 0x02d8220821373d93 + .quad 0xbc262d70acd1f567 + .quad 0x7a92c9fdfbcac784 + .quad 0x1fcca94516bd3289 + .quad 0x448d65aa41420428 + .quad 0x59c3b7b216a55d62 + .quad 0x49992cc64e612cd8 + .quad 0x65bd1bea70f801de + .quad 0x1befb7c0fe49e28a + .quad 0xa86306cdb1b2ae4a + .quad 0x3b7ac0cd265c2a09 + + // 2^20 * 8 * B + + .quad 0x822bee438c01bcec + .quad 0x530cb525c0fbc73b + .quad 0x48519034c1953fe9 + .quad 0x265cc261e09a0f5b + .quad 0xf0d54e4f22ed39a7 + .quad 0xa2aae91e5608150a + .quad 0xf421b2e9eddae875 + .quad 0x31bc531d6b7de992 + .quad 0xdf3d134da980f971 + .quad 0x7a4fb8d1221a22a7 + .quad 0x3df7d42035aad6d8 + .quad 0x2a14edcc6a1a125e + + // 2^24 * 1 * B + + .quad 0xdf48ee0752cfce4e + .quad 0xc3fffaf306ec08b7 + .quad 0x05710b2ab95459c4 + .quad 0x161d25fa963ea38d + .quad 0x231a8c570478433c + .quad 0xb7b5270ec281439d + .quad 0xdbaa99eae3d9079f + .quad 0x2c03f5256c2b03d9 + .quad 0x790f18757b53a47d + .quad 0x307b0130cf0c5879 + .quad 0x31903d77257ef7f9 + .quad 0x699468bdbd96bbaf + + // 2^24 * 2 * B + + .quad 0xbd1f2f46f4dafecf + .quad 0x7cef0114a47fd6f7 + .quad 0xd31ffdda4a47b37f + .quad 0x525219a473905785 + .quad 0xd8dd3de66aa91948 + .quad 0x485064c22fc0d2cc + .quad 0x9b48246634fdea2f + .quad 0x293e1c4e6c4a2e3a + .quad 0x376e134b925112e1 + .quad 0x703778b5dca15da0 + .quad 0xb04589af461c3111 + .quad 0x5b605c447f032823 + + // 2^24 * 3 * B + + .quad 0xb965805920c47c89 + .quad 0xe7f0100c923b8fcc + .quad 0x0001256502e2ef77 + .quad 0x24a76dcea8aeb3ee + .quad 0x3be9fec6f0e7f04c + .quad 0x866a579e75e34962 + .quad 0x5542ef161e1de61a + .quad 0x2f12fef4cc5abdd5 + .quad 0x0a4522b2dfc0c740 + .quad 0x10d06e7f40c9a407 + .quad 0xc6cf144178cff668 + .quad 0x5e607b2518a43790 + + // 2^24 * 4 * B + + .quad 0x58b31d8f6cdf1818 + .quad 0x35cfa74fc36258a2 + .quad 0xe1b3ff4f66e61d6e + .quad 0x5067acab6ccdd5f7 + .quad 0xa02c431ca596cf14 + .quad 0xe3c42d40aed3e400 + .quad 0xd24526802e0f26db + .quad 0x201f33139e457068 + .quad 0xfd527f6b08039d51 + .quad 0x18b14964017c0006 + .quad 0xd5220eb02e25a4a8 + .quad 0x397cba8862460375 + + // 2^24 * 5 * B + + .quad 0x30c13093f05959b2 + .quad 0xe23aa18de9a97976 + .quad 0x222fd491721d5e26 + .quad 0x2339d320766e6c3a + .quad 0x7815c3fbc81379e7 + .quad 0xa6619420dde12af1 + .quad 0xffa9c0f885a8fdd5 + .quad 0x771b4022c1e1c252 + .quad 0xd87dd986513a2fa7 + .quad 0xf5ac9b71f9d4cf08 + .quad 0xd06bc31b1ea283b3 + .quad 0x331a189219971a76 + + // 2^24 * 6 * B + + .quad 0xf5166f45fb4f80c6 + .quad 0x9c36c7de61c775cf + .quad 0xe3d4e81b9041d91c + .quad 0x31167c6b83bdfe21 + .quad 0x26512f3a9d7572af + .quad 0x5bcbe28868074a9e + .quad 0x84edc1c11180f7c4 + .quad 0x1ac9619ff649a67b + .quad 0xf22b3842524b1068 + .quad 0x5068343bee9ce987 + .quad 0xfc9d71844a6250c8 + .quad 0x612436341f08b111 + + // 2^24 * 7 * B + + .quad 0xd99d41db874e898d + .quad 0x09fea5f16c07dc20 + .quad 0x793d2c67d00f9bbc + .quad 0x46ebe2309e5eff40 + .quad 0x8b6349e31a2d2638 + .quad 0x9ddfb7009bd3fd35 + .quad 0x7f8bf1b8a3a06ba4 + .quad 0x1522aa3178d90445 + .quad 0x2c382f5369614938 + .quad 0xdafe409ab72d6d10 + .quad 0xe8c83391b646f227 + .quad 0x45fe70f50524306c + + // 2^24 * 8 * B + + .quad 0xda4875a6960c0b8c + .quad 0x5b68d076ef0e2f20 + .quad 0x07fb51cf3d0b8fd4 + .quad 0x428d1623a0e392d4 + .quad 0x62f24920c8951491 + .quad 0x05f007c83f630ca2 + .quad 0x6fbb45d2f5c9d4b8 + .quad 0x16619f6db57a2245 + .quad 0x084f4a4401a308fd + .quad 0xa82219c376a5caac + .quad 0xdeb8de4643d1bc7d + .quad 0x1d81592d60bd38c6 + + // 2^28 * 1 * B + + .quad 0xd833d7beec2a4c38 + .quad 0x2c9162830acc20ed + .quad 0xe93a47aa92df7581 + .quad 0x702d67a3333c4a81 + .quad 0x3a4a369a2f89c8a1 + .quad 0x63137a1d7c8de80d + .quad 0xbcac008a78eda015 + .quad 0x2cb8b3a5b483b03f + .quad 0x36e417cbcb1b90a1 + .quad 0x33b3ddaa7f11794e + .quad 0x3f510808885bc607 + .quad 0x24141dc0e6a8020d + + // 2^28 * 2 * B + + .quad 0x59f73c773fefee9d + .quad 0xb3f1ef89c1cf989d + .quad 0xe35dfb42e02e545f + .quad 0x5766120b47a1b47c + .quad 0x91925dccbd83157d + .quad 0x3ca1205322cc8094 + .quad 0x28e57f183f90d6e4 + .quad 0x1a4714cede2e767b + .quad 0xdb20ba0fb8b6b7ff + .quad 0xb732c3b677511fa1 + .quad 0xa92b51c099f02d89 + .quad 0x4f3875ad489ca5f1 + + // 2^28 * 3 * B + + .quad 0xc7fc762f4932ab22 + .quad 0x7ac0edf72f4c3c1b + .quad 0x5f6b55aa9aa895e8 + .quad 0x3680274dad0a0081 + .quad 0x79ed13f6ee73eec0 + .quad 0xa5c6526d69110bb1 + .quad 0xe48928c38603860c + .quad 0x722a1446fd7059f5 + .quad 0xd0959fe9a8cf8819 + .quad 0xd0a995508475a99c + .quad 0x6eac173320b09cc5 + .quad 0x628ecf04331b1095 + + // 2^28 * 4 * B + + .quad 0x98bcb118a9d0ddbc + .quad 0xee449e3408b4802b + .quad 0x87089226b8a6b104 + .quad 0x685f349a45c7915d + .quad 0x9b41acf85c74ccf1 + .quad 0xb673318108265251 + .quad 0x99c92aed11adb147 + .quad 0x7a47d70d34ecb40f + .quad 0x60a0c4cbcc43a4f5 + .quad 0x775c66ca3677bea9 + .quad 0xa17aa1752ff8f5ed + .quad 0x11ded9020e01fdc0 + + // 2^28 * 5 * B + + .quad 0x890e7809caefe704 + .quad 0x8728296de30e8c6c + .quad 0x4c5cd2a392aeb1c9 + .quad 0x194263d15771531f + .quad 0x471f95b03bea93b7 + .quad 0x0552d7d43313abd3 + .quad 0xbd9370e2e17e3f7b + .quad 0x7b120f1db20e5bec + .quad 0x17d2fb3d86502d7a + .quad 0xb564d84450a69352 + .quad 0x7da962c8a60ed75d + .quad 0x00d0f85b318736aa + + // 2^28 * 6 * B + + .quad 0x978b142e777c84fd + .quad 0xf402644705a8c062 + .quad 0xa67ad51be7e612c7 + .quad 0x2f7b459698dd6a33 + .quad 0xa6753c1efd7621c1 + .quad 0x69c0b4a7445671f5 + .quad 0x971f527405b23c11 + .quad 0x387bc74851a8c7cd + .quad 0x81894b4d4a52a9a8 + .quad 0xadd93e12f6b8832f + .quad 0x184d8548b61bd638 + .quad 0x3f1c62dbd6c9f6cd + + // 2^28 * 7 * B + + .quad 0x2e8f1f0091910c1f + .quad 0xa4df4fe0bff2e12c + .quad 0x60c6560aee927438 + .quad 0x6338283facefc8fa + .quad 0x3fad3e40148f693d + .quad 0x052656e194eb9a72 + .quad 0x2f4dcbfd184f4e2f + .quad 0x406f8db1c482e18b + .quad 0x9e630d2c7f191ee4 + .quad 0x4fbf8301bc3ff670 + .quad 0x787d8e4e7afb73c4 + .quad 0x50d83d5be8f58fa5 + + // 2^28 * 8 * B + + .quad 0x85683916c11a1897 + .quad 0x2d69a4efe506d008 + .quad 0x39af1378f664bd01 + .quad 0x65942131361517c6 + .quad 0xc0accf90b4d3b66d + .quad 0xa7059de561732e60 + .quad 0x033d1f7870c6b0ba + .quad 0x584161cd26d946e4 + .quad 0xbbf2b1a072d27ca2 + .quad 0xbf393c59fbdec704 + .quad 0xe98dbbcee262b81e + .quad 0x02eebd0b3029b589 + + // 2^32 * 1 * B + + .quad 0x61368756a60dac5f + .quad 0x17e02f6aebabdc57 + .quad 0x7f193f2d4cce0f7d + .quad 0x20234a7789ecdcf0 + .quad 0x8765b69f7b85c5e8 + .quad 0x6ff0678bd168bab2 + .quad 0x3a70e77c1d330f9b + .quad 0x3a5f6d51b0af8e7c + .quad 0x76d20db67178b252 + .quad 0x071c34f9d51ed160 + .quad 0xf62a4a20b3e41170 + .quad 0x7cd682353cffe366 + + // 2^32 * 2 * B + + .quad 0x0be1a45bd887fab6 + .quad 0x2a846a32ba403b6e + .quad 0xd9921012e96e6000 + .quad 0x2838c8863bdc0943 + .quad 0xa665cd6068acf4f3 + .quad 0x42d92d183cd7e3d3 + .quad 0x5759389d336025d9 + .quad 0x3ef0253b2b2cd8ff + .quad 0xd16bb0cf4a465030 + .quad 0xfa496b4115c577ab + .quad 0x82cfae8af4ab419d + .quad 0x21dcb8a606a82812 + + // 2^32 * 3 * B + + .quad 0x5c6004468c9d9fc8 + .quad 0x2540096ed42aa3cb + .quad 0x125b4d4c12ee2f9c + .quad 0x0bc3d08194a31dab + .quad 0x9a8d00fabe7731ba + .quad 0x8203607e629e1889 + .quad 0xb2cc023743f3d97f + .quad 0x5d840dbf6c6f678b + .quad 0x706e380d309fe18b + .quad 0x6eb02da6b9e165c7 + .quad 0x57bbba997dae20ab + .quad 0x3a4276232ac196dd + + // 2^32 * 4 * B + + .quad 0x4b42432c8a7084fa + .quad 0x898a19e3dfb9e545 + .quad 0xbe9f00219c58e45d + .quad 0x1ff177cea16debd1 + .quad 0x3bf8c172db447ecb + .quad 0x5fcfc41fc6282dbd + .quad 0x80acffc075aa15fe + .quad 0x0770c9e824e1a9f9 + .quad 0xcf61d99a45b5b5fd + .quad 0x860984e91b3a7924 + .quad 0xe7300919303e3e89 + .quad 0x39f264fd41500b1e + + // 2^32 * 5 * B + + .quad 0xa7ad3417dbe7e29c + .quad 0xbd94376a2b9c139c + .quad 0xa0e91b8e93597ba9 + .quad 0x1712d73468889840 + .quad 0xd19b4aabfe097be1 + .quad 0xa46dfce1dfe01929 + .quad 0xc3c908942ca6f1ff + .quad 0x65c621272c35f14e + .quad 0xe72b89f8ce3193dd + .quad 0x4d103356a125c0bb + .quad 0x0419a93d2e1cfe83 + .quad 0x22f9800ab19ce272 + + // 2^32 * 6 * B + + .quad 0x605a368a3e9ef8cb + .quad 0xe3e9c022a5504715 + .quad 0x553d48b05f24248f + .quad 0x13f416cd647626e5 + .quad 0x42029fdd9a6efdac + .quad 0xb912cebe34a54941 + .quad 0x640f64b987bdf37b + .quad 0x4171a4d38598cab4 + .quad 0xfa2758aa99c94c8c + .quad 0x23006f6fb000b807 + .quad 0xfbd291ddadda5392 + .quad 0x508214fa574bd1ab + + // 2^32 * 7 * B + + .quad 0xc20269153ed6fe4b + .quad 0xa65a6739511d77c4 + .quad 0xcbde26462c14af94 + .quad 0x22f960ec6faba74b + .quad 0x461a15bb53d003d6 + .quad 0xb2102888bcf3c965 + .quad 0x27c576756c683a5a + .quad 0x3a7758a4c86cb447 + .quad 0x548111f693ae5076 + .quad 0x1dae21df1dfd54a6 + .quad 0x12248c90f3115e65 + .quad 0x5d9fd15f8de7f494 + + // 2^32 * 8 * B + + .quad 0x031408d36d63727f + .quad 0x6a379aefd7c7b533 + .quad 0xa9e18fc5ccaee24b + .quad 0x332f35914f8fbed3 + .quad 0x3f244d2aeed7521e + .quad 0x8e3a9028432e9615 + .quad 0xe164ba772e9c16d4 + .quad 0x3bc187fa47eb98d8 + .quad 0x6d470115ea86c20c + .quad 0x998ab7cb6c46d125 + .quad 0xd77832b53a660188 + .quad 0x450d81ce906fba03 + + // 2^36 * 1 * B + + .quad 0xf8ae4d2ad8453902 + .quad 0x7018058ee8db2d1d + .quad 0xaab3995fc7d2c11e + .quad 0x53b16d2324ccca79 + .quad 0x23264d66b2cae0b5 + .quad 0x7dbaed33ebca6576 + .quad 0x030ebed6f0d24ac8 + .quad 0x2a887f78f7635510 + .quad 0x2a23b9e75c012d4f + .quad 0x0c974651cae1f2ea + .quad 0x2fb63273675d70ca + .quad 0x0ba7250b864403f5 + + // 2^36 * 2 * B + + .quad 0xbb0d18fd029c6421 + .quad 0xbc2d142189298f02 + .quad 0x8347f8e68b250e96 + .quad 0x7b9f2fe8032d71c9 + .quad 0xdd63589386f86d9c + .quad 0x61699176e13a85a4 + .quad 0x2e5111954eaa7d57 + .quad 0x32c21b57fb60bdfb + .quad 0xd87823cd319e0780 + .quad 0xefc4cfc1897775c5 + .quad 0x4854fb129a0ab3f7 + .quad 0x12c49d417238c371 + + // 2^36 * 3 * B + + .quad 0x0950b533ffe83769 + .quad 0x21861c1d8e1d6bd1 + .quad 0xf022d8381302e510 + .quad 0x2509200c6391cab4 + .quad 0x09b3a01783799542 + .quad 0x626dd08faad5ee3f + .quad 0xba00bceeeb70149f + .quad 0x1421b246a0a444c9 + .quad 0x4aa43a8e8c24a7c7 + .quad 0x04c1f540d8f05ef5 + .quad 0xadba5e0c0b3eb9dc + .quad 0x2ab5504448a49ce3 + + // 2^36 * 4 * B + + .quad 0x2ed227266f0f5dec + .quad 0x9824ee415ed50824 + .quad 0x807bec7c9468d415 + .quad 0x7093bae1b521e23f + .quad 0xdc07ac631c5d3afa + .quad 0x58615171f9df8c6c + .quad 0x72a079d89d73e2b0 + .quad 0x7301f4ceb4eae15d + .quad 0x6409e759d6722c41 + .quad 0xa674e1cf72bf729b + .quad 0xbc0a24eb3c21e569 + .quad 0x390167d24ebacb23 + + // 2^36 * 5 * B + + .quad 0x27f58e3bba353f1c + .quad 0x4c47764dbf6a4361 + .quad 0xafbbc4e56e562650 + .quad 0x07db2ee6aae1a45d + .quad 0xd7bb054ba2f2120b + .quad 0xe2b9ceaeb10589b7 + .quad 0x3fe8bac8f3c0edbe + .quad 0x4cbd40767112cb69 + .quad 0x0b603cc029c58176 + .quad 0x5988e3825cb15d61 + .quad 0x2bb61413dcf0ad8d + .quad 0x7b8eec6c74183287 + + // 2^36 * 6 * B + + .quad 0xe4ca40782cd27cb0 + .quad 0xdaf9c323fbe967bd + .quad 0xb29bd34a8ad41e9e + .quad 0x72810497626ede4d + .quad 0x32fee570fc386b73 + .quad 0xda8b0141da3a8cc7 + .quad 0x975ffd0ac8968359 + .quad 0x6ee809a1b132a855 + .quad 0x9444bb31fcfd863a + .quad 0x2fe3690a3e4e48c5 + .quad 0xdc29c867d088fa25 + .quad 0x13bd1e38d173292e + + // 2^36 * 7 * B + + .quad 0xd32b4cd8696149b5 + .quad 0xe55937d781d8aab7 + .quad 0x0bcb2127ae122b94 + .quad 0x41e86fcfb14099b0 + .quad 0x223fb5cf1dfac521 + .quad 0x325c25316f554450 + .quad 0x030b98d7659177ac + .quad 0x1ed018b64f88a4bd + .quad 0x3630dfa1b802a6b0 + .quad 0x880f874742ad3bd5 + .quad 0x0af90d6ceec5a4d4 + .quad 0x746a247a37cdc5d9 + + // 2^36 * 8 * B + + .quad 0xd531b8bd2b7b9af6 + .quad 0x5005093537fc5b51 + .quad 0x232fcf25c593546d + .quad 0x20a365142bb40f49 + .quad 0x6eccd85278d941ed + .quad 0x2254ae83d22f7843 + .quad 0xc522d02e7bbfcdb7 + .quad 0x681e3351bff0e4e2 + .quad 0x8b64b59d83034f45 + .quad 0x2f8b71f21fa20efb + .quad 0x69249495ba6550e4 + .quad 0x539ef98e45d5472b + + // 2^40 * 1 * B + + .quad 0x6e7bb6a1a6205275 + .quad 0xaa4f21d7413c8e83 + .quad 0x6f56d155e88f5cb2 + .quad 0x2de25d4ba6345be1 + .quad 0xd074d8961cae743f + .quad 0xf86d18f5ee1c63ed + .quad 0x97bdc55be7f4ed29 + .quad 0x4cbad279663ab108 + .quad 0x80d19024a0d71fcd + .quad 0xc525c20afb288af8 + .quad 0xb1a3974b5f3a6419 + .quad 0x7d7fbcefe2007233 + + // 2^40 * 2 * B + + .quad 0xfaef1e6a266b2801 + .quad 0x866c68c4d5739f16 + .quad 0xf68a2fbc1b03762c + .quad 0x5975435e87b75a8d + .quad 0xcd7c5dc5f3c29094 + .quad 0xc781a29a2a9105ab + .quad 0x80c61d36421c3058 + .quad 0x4f9cd196dcd8d4d7 + .quad 0x199297d86a7b3768 + .quad 0xd0d058241ad17a63 + .quad 0xba029cad5c1c0c17 + .quad 0x7ccdd084387a0307 + + // 2^40 * 3 * B + + .quad 0xdca6422c6d260417 + .quad 0xae153d50948240bd + .quad 0xa9c0c1b4fb68c677 + .quad 0x428bd0ed61d0cf53 + .quad 0x9b0c84186760cc93 + .quad 0xcdae007a1ab32a99 + .quad 0xa88dec86620bda18 + .quad 0x3593ca848190ca44 + .quad 0x9213189a5e849aa7 + .quad 0xd4d8c33565d8facd + .quad 0x8c52545b53fdbbd1 + .quad 0x27398308da2d63e6 + + // 2^40 * 4 * B + + .quad 0x42c38d28435ed413 + .quad 0xbd50f3603278ccc9 + .quad 0xbb07ab1a79da03ef + .quad 0x269597aebe8c3355 + .quad 0xb9a10e4c0a702453 + .quad 0x0fa25866d57d1bde + .quad 0xffb9d9b5cd27daf7 + .quad 0x572c2945492c33fd + .quad 0xc77fc745d6cd30be + .quad 0xe4dfe8d3e3baaefb + .quad 0xa22c8830aa5dda0c + .quad 0x7f985498c05bca80 + + // 2^40 * 5 * B + + .quad 0x3849ce889f0be117 + .quad 0x8005ad1b7b54a288 + .quad 0x3da3c39f23fc921c + .quad 0x76c2ec470a31f304 + .quad 0xd35615520fbf6363 + .quad 0x08045a45cf4dfba6 + .quad 0xeec24fbc873fa0c2 + .quad 0x30f2653cd69b12e7 + .quad 0x8a08c938aac10c85 + .quad 0x46179b60db276bcb + .quad 0xa920c01e0e6fac70 + .quad 0x2f1273f1596473da + + // 2^40 * 6 * B + + .quad 0x4739fc7c8ae01e11 + .quad 0xfd5274904a6aab9f + .quad 0x41d98a8287728f2e + .quad 0x5d9e572ad85b69f2 + .quad 0x30488bd755a70bc0 + .quad 0x06d6b5a4f1d442e7 + .quad 0xead1a69ebc596162 + .quad 0x38ac1997edc5f784 + .quad 0x0666b517a751b13b + .quad 0x747d06867e9b858c + .quad 0xacacc011454dde49 + .quad 0x22dfcd9cbfe9e69c + + // 2^40 * 7 * B + + .quad 0x8ddbd2e0c30d0cd9 + .quad 0xad8e665facbb4333 + .quad 0x8f6b258c322a961f + .quad 0x6b2916c05448c1c7 + .quad 0x56ec59b4103be0a1 + .quad 0x2ee3baecd259f969 + .quad 0x797cb29413f5cd32 + .quad 0x0fe9877824cde472 + .quad 0x7edb34d10aba913b + .quad 0x4ea3cd822e6dac0e + .quad 0x66083dff6578f815 + .quad 0x4c303f307ff00a17 + + // 2^40 * 8 * B + + .quad 0xd30a3bd617b28c85 + .quad 0xc5d377b739773bea + .quad 0xc6c6e78c1e6a5cbf + .quad 0x0d61b8f78b2ab7c4 + .quad 0x29fc03580dd94500 + .quad 0xecd27aa46fbbec93 + .quad 0x130a155fc2e2a7f8 + .quad 0x416b151ab706a1d5 + .quad 0x56a8d7efe9c136b0 + .quad 0xbd07e5cd58e44b20 + .quad 0xafe62fda1b57e0ab + .quad 0x191a2af74277e8d2 + + // 2^44 * 1 * B + + .quad 0xd550095bab6f4985 + .quad 0x04f4cd5b4fbfaf1a + .quad 0x9d8e2ed12a0c7540 + .quad 0x2bc24e04b2212286 + .quad 0x09d4b60b2fe09a14 + .quad 0xc384f0afdbb1747e + .quad 0x58e2ea8978b5fd6e + .quad 0x519ef577b5e09b0a + .quad 0x1863d7d91124cca9 + .quad 0x7ac08145b88a708e + .quad 0x2bcd7309857031f5 + .quad 0x62337a6e8ab8fae5 + + // 2^44 * 2 * B + + .quad 0x4bcef17f06ffca16 + .quad 0xde06e1db692ae16a + .quad 0x0753702d614f42b0 + .quad 0x5f6041b45b9212d0 + .quad 0xd1ab324e1b3a1273 + .quad 0x18947cf181055340 + .quad 0x3b5d9567a98c196e + .quad 0x7fa00425802e1e68 + .quad 0x7d531574028c2705 + .quad 0x80317d69db0d75fe + .quad 0x30fface8ef8c8ddd + .quad 0x7e9de97bb6c3e998 + + // 2^44 * 3 * B + + .quad 0x1558967b9e6585a3 + .quad 0x97c99ce098e98b92 + .quad 0x10af149b6eb3adad + .quad 0x42181fe8f4d38cfa + .quad 0xf004be62a24d40dd + .quad 0xba0659910452d41f + .quad 0x81c45ee162a44234 + .quad 0x4cb829d8a22266ef + .quad 0x1dbcaa8407b86681 + .quad 0x081f001e8b26753b + .quad 0x3cd7ce6a84048e81 + .quad 0x78af11633f25f22c + + // 2^44 * 4 * B + + .quad 0x8416ebd40b50babc + .quad 0x1508722628208bee + .quad 0xa3148fafb9c1c36d + .quad 0x0d07daacd32d7d5d + .quad 0x3241c00e7d65318c + .quad 0xe6bee5dcd0e86de7 + .quad 0x118b2dc2fbc08c26 + .quad 0x680d04a7fc603dc3 + .quad 0xf9c2414a695aa3eb + .quad 0xdaa42c4c05a68f21 + .quad 0x7c6c23987f93963e + .quad 0x210e8cd30c3954e3 + + // 2^44 * 5 * B + + .quad 0xac4201f210a71c06 + .quad 0x6a65e0aef3bfb021 + .quad 0xbc42c35c393632f7 + .quad 0x56ea8db1865f0742 + .quad 0x2b50f16137fe6c26 + .quad 0xe102bcd856e404d8 + .quad 0x12b0f1414c561f6b + .quad 0x51b17bc8d028ec91 + .quad 0xfff5fb4bcf535119 + .quad 0xf4989d79df1108a0 + .quad 0xbdfcea659a3ba325 + .quad 0x18a11f1174d1a6f2 + + // 2^44 * 6 * B + + .quad 0x407375ab3f6bba29 + .quad 0x9ec3b6d8991e482e + .quad 0x99c80e82e55f92e9 + .quad 0x307c13b6fb0c0ae1 + .quad 0xfbd63cdad27a5f2c + .quad 0xf00fc4bc8aa106d7 + .quad 0x53fb5c1a8e64a430 + .quad 0x04eaabe50c1a2e85 + .quad 0x24751021cb8ab5e7 + .quad 0xfc2344495c5010eb + .quad 0x5f1e717b4e5610a1 + .quad 0x44da5f18c2710cd5 + + // 2^44 * 7 * B + + .quad 0x033cc55ff1b82eb5 + .quad 0xb15ae36d411cae52 + .quad 0xba40b6198ffbacd3 + .quad 0x768edce1532e861f + .quad 0x9156fe6b89d8eacc + .quad 0xe6b79451e23126a1 + .quad 0xbd7463d93944eb4e + .quad 0x726373f6767203ae + .quad 0xe305ca72eb7ef68a + .quad 0x662cf31f70eadb23 + .quad 0x18f026fdb4c45b68 + .quad 0x513b5384b5d2ecbd + + // 2^44 * 8 * B + + .quad 0x46d46280c729989e + .quad 0x4b93fbd05368a5dd + .quad 0x63df3f81d1765a89 + .quad 0x34cebd64b9a0a223 + .quad 0x5e2702878af34ceb + .quad 0x900b0409b946d6ae + .quad 0x6512ebf7dabd8512 + .quad 0x61d9b76988258f81 + .quad 0xa6c5a71349b7d94b + .quad 0xa3f3d15823eb9446 + .quad 0x0416fbd277484834 + .quad 0x69d45e6f2c70812f + + // 2^48 * 1 * B + + .quad 0xce16f74bc53c1431 + .quad 0x2b9725ce2072edde + .quad 0xb8b9c36fb5b23ee7 + .quad 0x7e2e0e450b5cc908 + .quad 0x9fe62b434f460efb + .quad 0xded303d4a63607d6 + .quad 0xf052210eb7a0da24 + .quad 0x237e7dbe00545b93 + .quad 0x013575ed6701b430 + .quad 0x231094e69f0bfd10 + .quad 0x75320f1583e47f22 + .quad 0x71afa699b11155e3 + + // 2^48 * 2 * B + + .quad 0x65ce6f9b3953b61d + .quad 0xc65839eaafa141e6 + .quad 0x0f435ffda9f759fe + .quad 0x021142e9c2b1c28e + .quad 0xea423c1c473b50d6 + .quad 0x51e87a1f3b38ef10 + .quad 0x9b84bf5fb2c9be95 + .quad 0x00731fbc78f89a1c + .quad 0xe430c71848f81880 + .quad 0xbf960c225ecec119 + .quad 0xb6dae0836bba15e3 + .quad 0x4c4d6f3347e15808 + + // 2^48 * 3 * B + + .quad 0x18f7eccfc17d1fc9 + .quad 0x6c75f5a651403c14 + .quad 0xdbde712bf7ee0cdf + .quad 0x193fddaaa7e47a22 + .quad 0x2f0cddfc988f1970 + .quad 0x6b916227b0b9f51b + .quad 0x6ec7b6c4779176be + .quad 0x38bf9500a88f9fa8 + .quad 0x1fd2c93c37e8876f + .quad 0xa2f61e5a18d1462c + .quad 0x5080f58239241276 + .quad 0x6a6fb99ebf0d4969 + + // 2^48 * 4 * B + + .quad 0x6a46c1bb560855eb + .quad 0x2416bb38f893f09d + .quad 0xd71d11378f71acc1 + .quad 0x75f76914a31896ea + .quad 0xeeb122b5b6e423c6 + .quad 0x939d7010f286ff8e + .quad 0x90a92a831dcf5d8c + .quad 0x136fda9f42c5eb10 + .quad 0xf94cdfb1a305bdd1 + .quad 0x0f364b9d9ff82c08 + .quad 0x2a87d8a5c3bb588a + .quad 0x022183510be8dcba + + // 2^48 * 5 * B + + .quad 0x4af766385ead2d14 + .quad 0xa08ed880ca7c5830 + .quad 0x0d13a6e610211e3d + .quad 0x6a071ce17b806c03 + .quad 0x9d5a710143307a7f + .quad 0xb063de9ec47da45f + .quad 0x22bbfe52be927ad3 + .quad 0x1387c441fd40426c + .quad 0xb5d3c3d187978af8 + .quad 0x722b5a3d7f0e4413 + .quad 0x0d7b4848bb477ca0 + .quad 0x3171b26aaf1edc92 + + // 2^48 * 6 * B + + .quad 0xa92f319097564ca8 + .quad 0xff7bb84c2275e119 + .quad 0x4f55fe37a4875150 + .quad 0x221fd4873cf0835a + .quad 0xa60db7d8b28a47d1 + .quad 0xa6bf14d61770a4f1 + .quad 0xd4a1f89353ddbd58 + .quad 0x6c514a63344243e9 + .quad 0x2322204f3a156341 + .quad 0xfb73e0e9ba0a032d + .quad 0xfce0dd4c410f030e + .quad 0x48daa596fb924aaa + + // 2^48 * 7 * B + + .quad 0x6eca8e665ca59cc7 + .quad 0xa847254b2e38aca0 + .quad 0x31afc708d21e17ce + .quad 0x676dd6fccad84af7 + .quad 0x14f61d5dc84c9793 + .quad 0x9941f9e3ef418206 + .quad 0xcdf5b88f346277ac + .quad 0x58c837fa0e8a79a9 + .quad 0x0cf9688596fc9058 + .quad 0x1ddcbbf37b56a01b + .quad 0xdcc2e77d4935d66a + .quad 0x1c4f73f2c6a57f0a + + // 2^48 * 8 * B + + .quad 0x0e7a4fbd305fa0bb + .quad 0x829d4ce054c663ad + .quad 0xf421c3832fe33848 + .quad 0x795ac80d1bf64c42 + .quad 0xb36e706efc7c3484 + .quad 0x73dfc9b4c3c1cf61 + .quad 0xeb1d79c9781cc7e5 + .quad 0x70459adb7daf675c + .quad 0x1b91db4991b42bb3 + .quad 0x572696234b02dcca + .quad 0x9fdf9ee51f8c78dc + .quad 0x5fe162848ce21fd3 + + // 2^52 * 1 * B + + .quad 0xe2790aae4d077c41 + .quad 0x8b938270db7469a3 + .quad 0x6eb632dc8abd16a2 + .quad 0x720814ecaa064b72 + .quad 0x315c29c795115389 + .quad 0xd7e0e507862f74ce + .quad 0x0c4a762185927432 + .quad 0x72de6c984a25a1e4 + .quad 0xae9ab553bf6aa310 + .quad 0x050a50a9806d6e1b + .quad 0x92bb7403adff5139 + .quad 0x0394d27645be618b + + // 2^52 * 2 * B + + .quad 0x4d572251857eedf4 + .quad 0xe3724edde19e93c5 + .quad 0x8a71420e0b797035 + .quad 0x3b3c833687abe743 + .quad 0xf5396425b23545a4 + .quad 0x15a7a27e98fbb296 + .quad 0xab6c52bc636fdd86 + .quad 0x79d995a8419334ee + .quad 0xcd8a8ea61195dd75 + .quad 0xa504d8a81dd9a82f + .quad 0x540dca81a35879b6 + .quad 0x60dd16a379c86a8a + + // 2^52 * 3 * B + + .quad 0x35a2c8487381e559 + .quad 0x596ffea6d78082cb + .quad 0xcb9771ebdba7b653 + .quad 0x5a08b5019b4da685 + .quad 0x3501d6f8153e47b8 + .quad 0xb7a9675414a2f60c + .quad 0x112ee8b6455d9523 + .quad 0x4e62a3c18112ea8a + .quad 0xc8d4ac04516ab786 + .quad 0x595af3215295b23d + .quad 0xd6edd234db0230c1 + .quad 0x0929efe8825b41cc + + // 2^52 * 4 * B + + .quad 0x5f0601d1cbd0f2d3 + .quad 0x736e412f6132bb7f + .quad 0x83604432238dde87 + .quad 0x1e3a5272f5c0753c + .quad 0x8b3172b7ad56651d + .quad 0x01581b7a3fabd717 + .quad 0x2dc94df6424df6e4 + .quad 0x30376e5d2c29284f + .quad 0xd2918da78159a59c + .quad 0x6bdc1cd93f0713f3 + .quad 0x565f7a934acd6590 + .quad 0x53daacec4cb4c128 + + // 2^52 * 5 * B + + .quad 0x4ca73bd79cc8a7d6 + .quad 0x4d4a738f47e9a9b2 + .quad 0xf4cbf12942f5fe00 + .quad 0x01a13ff9bdbf0752 + .quad 0x99852bc3852cfdb0 + .quad 0x2cc12e9559d6ed0b + .quad 0x70f9e2bf9b5ac27b + .quad 0x4f3b8c117959ae99 + .quad 0x55b6c9c82ff26412 + .quad 0x1ac4a8c91fb667a8 + .quad 0xd527bfcfeb778bf2 + .quad 0x303337da7012a3be + + // 2^52 * 6 * B + + .quad 0x955422228c1c9d7c + .quad 0x01fac1371a9b340f + .quad 0x7e8d9177925b48d7 + .quad 0x53f8ad5661b3e31b + .quad 0x976d3ccbfad2fdd1 + .quad 0xcb88839737a640a8 + .quad 0x2ff00c1d6734cb25 + .quad 0x269ff4dc789c2d2b + .quad 0x0c003fbdc08d678d + .quad 0x4d982fa37ead2b17 + .quad 0xc07e6bcdb2e582f1 + .quad 0x296c7291df412a44 + + // 2^52 * 7 * B + + .quad 0x7903de2b33daf397 + .quad 0xd0ff0619c9a624b3 + .quad 0x8a1d252b555b3e18 + .quad 0x2b6d581c52e0b7c0 + .quad 0xdfb23205dab8b59e + .quad 0x465aeaa0c8092250 + .quad 0xd133c1189a725d18 + .quad 0x2327370261f117d1 + .quad 0x3d0543d3623e7986 + .quad 0x679414c2c278a354 + .quad 0xae43f0cc726196f6 + .quad 0x7836c41f8245eaba + + // 2^52 * 8 * B + + .quad 0xe7a254db49e95a81 + .quad 0x5192d5d008b0ad73 + .quad 0x4d20e5b1d00afc07 + .quad 0x5d55f8012cf25f38 + .quad 0xca651e848011937c + .quad 0xc6b0c46e6ef41a28 + .quad 0xb7021ba75f3f8d52 + .quad 0x119dff99ead7b9fd + .quad 0x43eadfcbf4b31d4d + .quad 0xc6503f7411148892 + .quad 0xfeee68c5060d3b17 + .quad 0x329293b3dd4a0ac8 + + // 2^56 * 1 * B + + .quad 0x4e59214fe194961a + .quad 0x49be7dc70d71cd4f + .quad 0x9300cfd23b50f22d + .quad 0x4789d446fc917232 + .quad 0x2879852d5d7cb208 + .quad 0xb8dedd70687df2e7 + .quad 0xdc0bffab21687891 + .quad 0x2b44c043677daa35 + .quad 0x1a1c87ab074eb78e + .quad 0xfac6d18e99daf467 + .quad 0x3eacbbcd484f9067 + .quad 0x60c52eef2bb9a4e4 + + // 2^56 * 2 * B + + .quad 0x0b5d89bc3bfd8bf1 + .quad 0xb06b9237c9f3551a + .quad 0x0e4c16b0d53028f5 + .quad 0x10bc9c312ccfcaab + .quad 0x702bc5c27cae6d11 + .quad 0x44c7699b54a48cab + .quad 0xefbc4056ba492eb2 + .quad 0x70d77248d9b6676d + .quad 0xaa8ae84b3ec2a05b + .quad 0x98699ef4ed1781e0 + .quad 0x794513e4708e85d1 + .quad 0x63755bd3a976f413 + + // 2^56 * 3 * B + + .quad 0xb55fa03e2ad10853 + .quad 0x356f75909ee63569 + .quad 0x9ff9f1fdbe69b890 + .quad 0x0d8cc1c48bc16f84 + .quad 0x3dc7101897f1acb7 + .quad 0x5dda7d5ec165bbd8 + .quad 0x508e5b9c0fa1020f + .quad 0x2763751737c52a56 + .quad 0x029402d36eb419a9 + .quad 0xf0b44e7e77b460a5 + .quad 0xcfa86230d43c4956 + .quad 0x70c2dd8a7ad166e7 + + // 2^56 * 4 * B + + .quad 0x656194509f6fec0e + .quad 0xee2e7ea946c6518d + .quad 0x9733c1f367e09b5c + .quad 0x2e0fac6363948495 + .quad 0x91d4967db8ed7e13 + .quad 0x74252f0ad776817a + .quad 0xe40982e00d852564 + .quad 0x32b8613816a53ce5 + .quad 0x79e7f7bee448cd64 + .quad 0x6ac83a67087886d0 + .quad 0xf89fd4d9a0e4db2e + .quad 0x4179215c735a4f41 + + // 2^56 * 5 * B + + .quad 0x8c7094e7d7dced2a + .quad 0x97fb8ac347d39c70 + .quad 0xe13be033a906d902 + .quad 0x700344a30cd99d76 + .quad 0xe4ae33b9286bcd34 + .quad 0xb7ef7eb6559dd6dc + .quad 0x278b141fb3d38e1f + .quad 0x31fa85662241c286 + .quad 0xaf826c422e3622f4 + .quad 0xc12029879833502d + .quad 0x9bc1b7e12b389123 + .quad 0x24bb2312a9952489 + + // 2^56 * 6 * B + + .quad 0xb1a8ed1732de67c3 + .quad 0x3cb49418461b4948 + .quad 0x8ebd434376cfbcd2 + .quad 0x0fee3e871e188008 + .quad 0x41f80c2af5f85c6b + .quad 0x687284c304fa6794 + .quad 0x8945df99a3ba1bad + .quad 0x0d1d2af9ffeb5d16 + .quad 0xa9da8aa132621edf + .quad 0x30b822a159226579 + .quad 0x4004197ba79ac193 + .quad 0x16acd79718531d76 + + // 2^56 * 7 * B + + .quad 0x72df72af2d9b1d3d + .quad 0x63462a36a432245a + .quad 0x3ecea07916b39637 + .quad 0x123e0ef6b9302309 + .quad 0xc959c6c57887b6ad + .quad 0x94e19ead5f90feba + .quad 0x16e24e62a342f504 + .quad 0x164ed34b18161700 + .quad 0x487ed94c192fe69a + .quad 0x61ae2cea3a911513 + .quad 0x877bf6d3b9a4de27 + .quad 0x78da0fc61073f3eb + + // 2^56 * 8 * B + + .quad 0x5bf15d28e52bc66a + .quad 0x2c47e31870f01a8e + .quad 0x2419afbc06c28bdd + .quad 0x2d25deeb256b173a + .quad 0xa29f80f1680c3a94 + .quad 0x71f77e151ae9e7e6 + .quad 0x1100f15848017973 + .quad 0x054aa4b316b38ddd + .quad 0xdfc8468d19267cb8 + .quad 0x0b28789c66e54daf + .quad 0x2aeb1d2a666eec17 + .quad 0x134610a6ab7da760 + + // 2^60 * 1 * B + + .quad 0xcaf55ec27c59b23f + .quad 0x99aeed3e154d04f2 + .quad 0x68441d72e14141f4 + .quad 0x140345133932a0a2 + .quad 0xd91430e0dc028c3c + .quad 0x0eb955a85217c771 + .quad 0x4b09e1ed2c99a1fa + .quad 0x42881af2bd6a743c + .quad 0x7bfec69aab5cad3d + .quad 0xc23e8cd34cb2cfad + .quad 0x685dd14bfb37d6a2 + .quad 0x0ad6d64415677a18 + + // 2^60 * 2 * B + + .quad 0x781a439e417becb5 + .quad 0x4ac5938cd10e0266 + .quad 0x5da385110692ac24 + .quad 0x11b065a2ade31233 + .quad 0x7914892847927e9f + .quad 0x33dad6ef370aa877 + .quad 0x1f8f24fa11122703 + .quad 0x5265ac2f2adf9592 + .quad 0x405fdd309afcb346 + .quad 0xd9723d4428e63f54 + .quad 0x94c01df05f65aaae + .quad 0x43e4dc3ae14c0809 + + // 2^60 * 3 * B + + .quad 0xbc12c7f1a938a517 + .quad 0x473028ab3180b2e1 + .quad 0x3f78571efbcd254a + .quad 0x74e534426ff6f90f + .quad 0xea6f7ac3adc2c6a3 + .quad 0xd0e928f6e9717c94 + .quad 0xe2d379ead645eaf5 + .quad 0x46dd8785c51ffbbe + .quad 0x709801be375c8898 + .quad 0x4b06dab5e3fd8348 + .quad 0x75880ced27230714 + .quad 0x2b09468fdd2f4c42 + + // 2^60 * 4 * B + + .quad 0x97c749eeb701cb96 + .quad 0x83f438d4b6a369c3 + .quad 0x62962b8b9a402cd9 + .quad 0x6976c7509888df7b + .quad 0x5b97946582ffa02a + .quad 0xda096a51fea8f549 + .quad 0xa06351375f77af9b + .quad 0x1bcfde61201d1e76 + .quad 0x4a4a5490246a59a2 + .quad 0xd63ebddee87fdd90 + .quad 0xd9437c670d2371fa + .quad 0x69e87308d30f8ed6 + + // 2^60 * 5 * B + + .quad 0x435a8bb15656beb0 + .quad 0xf8fac9ba4f4d5bca + .quad 0xb9b278c41548c075 + .quad 0x3eb0ef76e892b622 + .quad 0x0f80bf028bc80303 + .quad 0x6aae16b37a18cefb + .quad 0xdd47ea47d72cd6a3 + .quad 0x61943588f4ed39aa + .quad 0xd26e5c3e91039f85 + .quad 0xc0e9e77df6f33aa9 + .quad 0xe8968c5570066a93 + .quad 0x3c34d1881faaaddd + + // 2^60 * 6 * B + + .quad 0x3f9d2b5ea09f9ec0 + .quad 0x1dab3b6fb623a890 + .quad 0xa09ba3ea72d926c4 + .quad 0x374193513fd8b36d + .quad 0xbd5b0b8f2fffe0d9 + .quad 0x6aa254103ed24fb9 + .quad 0x2ac7d7bcb26821c4 + .quad 0x605b394b60dca36a + .quad 0xb4e856e45a9d1ed2 + .quad 0xefe848766c97a9a2 + .quad 0xb104cf641e5eee7d + .quad 0x2f50b81c88a71c8f + + // 2^60 * 7 * B + + .quad 0x31723c61fc6811bb + .quad 0x9cb450486211800f + .quad 0x768933d347995753 + .quad 0x3491a53502752fcd + .quad 0x2b552ca0a7da522a + .quad 0x3230b336449b0250 + .quad 0xf2c4c5bca4b99fb9 + .quad 0x7b2c674958074a22 + .quad 0xd55165883ed28cdf + .quad 0x12d84fd2d362de39 + .quad 0x0a874ad3e3378e4f + .quad 0x000d2b1f7c763e74 + + // 2^60 * 8 * B + + .quad 0x3d420811d06d4a67 + .quad 0xbefc048590e0ffe3 + .quad 0xf870c6b7bd487bde + .quad 0x6e2a7316319afa28 + .quad 0x9624778c3e94a8ab + .quad 0x0ad6f3cee9a78bec + .quad 0x948ac7810d743c4f + .quad 0x76627935aaecfccc + .quad 0x56a8ac24d6d59a9f + .quad 0xc8db753e3096f006 + .quad 0x477f41e68f4c5299 + .quad 0x588d851cf6c86114 + + // 2^64 * 1 * B + + .quad 0x51138ec78df6b0fe + .quad 0x5397da89e575f51b + .quad 0x09207a1d717af1b9 + .quad 0x2102fdba2b20d650 + .quad 0xcd2a65e777d1f515 + .quad 0x548991878faa60f1 + .quad 0xb1b73bbcdabc06e5 + .quad 0x654878cba97cc9fb + .quad 0x969ee405055ce6a1 + .quad 0x36bca7681251ad29 + .quad 0x3a1af517aa7da415 + .quad 0x0ad725db29ecb2ba + + // 2^64 * 2 * B + + .quad 0xdc4267b1834e2457 + .quad 0xb67544b570ce1bc5 + .quad 0x1af07a0bf7d15ed7 + .quad 0x4aefcffb71a03650 + .quad 0xfec7bc0c9b056f85 + .quad 0x537d5268e7f5ffd7 + .quad 0x77afc6624312aefa + .quad 0x4f675f5302399fd9 + .quad 0xc32d36360415171e + .quad 0xcd2bef118998483b + .quad 0x870a6eadd0945110 + .quad 0x0bccbb72a2a86561 + + // 2^64 * 3 * B + + .quad 0x185e962feab1a9c8 + .quad 0x86e7e63565147dcd + .quad 0xb092e031bb5b6df2 + .quad 0x4024f0ab59d6b73e + .quad 0x186d5e4c50fe1296 + .quad 0xe0397b82fee89f7e + .quad 0x3bc7f6c5507031b0 + .quad 0x6678fd69108f37c2 + .quad 0x1586fa31636863c2 + .quad 0x07f68c48572d33f2 + .quad 0x4f73cc9f789eaefc + .quad 0x2d42e2108ead4701 + + // 2^64 * 4 * B + + .quad 0x97f5131594dfd29b + .quad 0x6155985d313f4c6a + .quad 0xeba13f0708455010 + .quad 0x676b2608b8d2d322 + .quad 0x21717b0d0f537593 + .quad 0x914e690b131e064c + .quad 0x1bb687ae752ae09f + .quad 0x420bf3a79b423c6e + .quad 0x8138ba651c5b2b47 + .quad 0x8671b6ec311b1b80 + .quad 0x7bff0cb1bc3135b0 + .quad 0x745d2ffa9c0cf1e0 + + // 2^64 * 5 * B + + .quad 0xbf525a1e2bc9c8bd + .quad 0xea5b260826479d81 + .quad 0xd511c70edf0155db + .quad 0x1ae23ceb960cf5d0 + .quad 0x6036df5721d34e6a + .quad 0xb1db8827997bb3d0 + .quad 0xd3c209c3c8756afa + .quad 0x06e15be54c1dc839 + .quad 0x5b725d871932994a + .quad 0x32351cb5ceb1dab0 + .quad 0x7dc41549dab7ca05 + .quad 0x58ded861278ec1f7 + + // 2^64 * 6 * B + + .quad 0xd8173793f266c55c + .quad 0xc8c976c5cc454e49 + .quad 0x5ce382f8bc26c3a8 + .quad 0x2ff39de85485f6f9 + .quad 0x2dfb5ba8b6c2c9a8 + .quad 0x48eeef8ef52c598c + .quad 0x33809107f12d1573 + .quad 0x08ba696b531d5bd8 + .quad 0x77ed3eeec3efc57a + .quad 0x04e05517d4ff4811 + .quad 0xea3d7a3ff1a671cb + .quad 0x120633b4947cfe54 + + // 2^64 * 7 * B + + .quad 0x0b94987891610042 + .quad 0x4ee7b13cecebfae8 + .quad 0x70be739594f0a4c0 + .quad 0x35d30a99b4d59185 + .quad 0x82bd31474912100a + .quad 0xde237b6d7e6fbe06 + .quad 0xe11e761911ea79c6 + .quad 0x07433be3cb393bde + .quad 0xff7944c05ce997f4 + .quad 0x575d3de4b05c51a3 + .quad 0x583381fd5a76847c + .quad 0x2d873ede7af6da9f + + // 2^64 * 8 * B + + .quad 0x157a316443373409 + .quad 0xfab8b7eef4aa81d9 + .quad 0xb093fee6f5a64806 + .quad 0x2e773654707fa7b6 + .quad 0xaa6202e14e5df981 + .quad 0xa20d59175015e1f5 + .quad 0x18a275d3bae21d6c + .quad 0x0543618a01600253 + .quad 0x0deabdf4974c23c1 + .quad 0xaa6f0a259dce4693 + .quad 0x04202cb8a29aba2c + .quad 0x4b1443362d07960d + + // 2^68 * 1 * B + + .quad 0x47b837f753242cec + .quad 0x256dc48cc04212f2 + .quad 0xe222fbfbe1d928c5 + .quad 0x48ea295bad8a2c07 + .quad 0x299b1c3f57c5715e + .quad 0x96cb929e6b686d90 + .quad 0x3004806447235ab3 + .quad 0x2c435c24a44d9fe1 + .quad 0x0607c97c80f8833f + .quad 0x0e851578ca25ec5b + .quad 0x54f7450b161ebb6f + .quad 0x7bcb4792a0def80e + + // 2^68 * 2 * B + + .quad 0x8487e3d02bc73659 + .quad 0x4baf8445059979df + .quad 0xd17c975adcad6fbf + .quad 0x57369f0bdefc96b6 + .quad 0x1cecd0a0045224c2 + .quad 0x757f1b1b69e53952 + .quad 0x775b7a925289f681 + .quad 0x1b6cc62016736148 + .quad 0xf1a9990175638698 + .quad 0x353dd1beeeaa60d3 + .quad 0x849471334c9ba488 + .quad 0x63fa6e6843ade311 + + // 2^68 * 3 * B + + .quad 0xd15c20536597c168 + .quad 0x9f73740098d28789 + .quad 0x18aee7f13257ba1f + .quad 0x3418bfda07346f14 + .quad 0x2195becdd24b5eb7 + .quad 0x5e41f18cc0cd44f9 + .quad 0xdf28074441ca9ede + .quad 0x07073b98f35b7d67 + .quad 0xd03c676c4ce530d4 + .quad 0x0b64c0473b5df9f4 + .quad 0x065cef8b19b3a31e + .quad 0x3084d661533102c9 + + // 2^68 * 4 * B + + .quad 0xe1f6b79ebf8469ad + .quad 0x15801004e2663135 + .quad 0x9a498330af74181b + .quad 0x3ba2504f049b673c + .quad 0x9a6ce876760321fd + .quad 0x7fe2b5109eb63ad8 + .quad 0x00e7d4ae8ac80592 + .quad 0x73d86b7abb6f723a + .quad 0x0b52b5606dba5ab6 + .quad 0xa9134f0fbbb1edab + .quad 0x30a9520d9b04a635 + .quad 0x6813b8f37973e5db + + // 2^68 * 5 * B + + .quad 0x9854b054334127c1 + .quad 0x105d047882fbff25 + .quad 0xdb49f7f944186f4f + .quad 0x1768e838bed0b900 + .quad 0xf194ca56f3157e29 + .quad 0x136d35705ef528a5 + .quad 0xdd4cef778b0599bc + .quad 0x7d5472af24f833ed + .quad 0xd0ef874daf33da47 + .quad 0x00d3be5db6e339f9 + .quad 0x3f2a8a2f9c9ceece + .quad 0x5d1aeb792352435a + + // 2^68 * 6 * B + + .quad 0xf59e6bb319cd63ca + .quad 0x670c159221d06839 + .quad 0xb06d565b2150cab6 + .quad 0x20fb199d104f12a3 + .quad 0x12c7bfaeb61ba775 + .quad 0xb84e621fe263bffd + .quad 0x0b47a5c35c840dcf + .quad 0x7e83be0bccaf8634 + .quad 0x61943dee6d99c120 + .quad 0x86101f2e460b9fe0 + .quad 0x6bb2f1518ee8598d + .quad 0x76b76289fcc475cc + + // 2^68 * 7 * B + + .quad 0x791b4cc1756286fa + .quad 0xdbced317d74a157c + .quad 0x7e732421ea72bde6 + .quad 0x01fe18491131c8e9 + .quad 0x4245f1a1522ec0b3 + .quad 0x558785b22a75656d + .quad 0x1d485a2548a1b3c0 + .quad 0x60959eccd58fe09f + .quad 0x3ebfeb7ba8ed7a09 + .quad 0x49fdc2bbe502789c + .quad 0x44ebce5d3c119428 + .quad 0x35e1eb55be947f4a + + // 2^68 * 8 * B + + .quad 0xdbdae701c5738dd3 + .quad 0xf9c6f635b26f1bee + .quad 0x61e96a8042f15ef4 + .quad 0x3aa1d11faf60a4d8 + .quad 0x14fd6dfa726ccc74 + .quad 0x3b084cfe2f53b965 + .quad 0xf33ae4f552a2c8b4 + .quad 0x59aab07a0d40166a + .quad 0x77bcec4c925eac25 + .quad 0x1848718460137738 + .quad 0x5b374337fea9f451 + .quad 0x1865e78ec8e6aa46 + + // 2^72 * 1 * B + + .quad 0xccc4b7c7b66e1f7a + .quad 0x44157e25f50c2f7e + .quad 0x3ef06dfc713eaf1c + .quad 0x582f446752da63f7 + .quad 0x967c54e91c529ccb + .quad 0x30f6269264c635fb + .quad 0x2747aff478121965 + .quad 0x17038418eaf66f5c + .quad 0xc6317bd320324ce4 + .quad 0xa81042e8a4488bc4 + .quad 0xb21ef18b4e5a1364 + .quad 0x0c2a1c4bcda28dc9 + + // 2^72 * 2 * B + + .quad 0xd24dc7d06f1f0447 + .quad 0xb2269e3edb87c059 + .quad 0xd15b0272fbb2d28f + .quad 0x7c558bd1c6f64877 + .quad 0xedc4814869bd6945 + .quad 0x0d6d907dbe1c8d22 + .quad 0xc63bd212d55cc5ab + .quad 0x5a6a9b30a314dc83 + .quad 0xd0ec1524d396463d + .quad 0x12bb628ac35a24f0 + .quad 0xa50c3a791cbc5fa4 + .quad 0x0404a5ca0afbafc3 + + // 2^72 * 3 * B + + .quad 0x8c1f40070aa743d6 + .quad 0xccbad0cb5b265ee8 + .quad 0x574b046b668fd2de + .quad 0x46395bfdcadd9633 + .quad 0x62bc9e1b2a416fd1 + .quad 0xb5c6f728e350598b + .quad 0x04343fd83d5d6967 + .quad 0x39527516e7f8ee98 + .quad 0x117fdb2d1a5d9a9c + .quad 0x9c7745bcd1005c2a + .quad 0xefd4bef154d56fea + .quad 0x76579a29e822d016 + + // 2^72 * 4 * B + + .quad 0x45b68e7e49c02a17 + .quad 0x23cd51a2bca9a37f + .quad 0x3ed65f11ec224c1b + .quad 0x43a384dc9e05bdb1 + .quad 0x333cb51352b434f2 + .quad 0xd832284993de80e1 + .quad 0xb5512887750d35ce + .quad 0x02c514bb2a2777c1 + .quad 0x684bd5da8bf1b645 + .quad 0xfb8bd37ef6b54b53 + .quad 0x313916d7a9b0d253 + .quad 0x1160920961548059 + + // 2^72 * 5 * B + + .quad 0xb44d166929dacfaa + .quad 0xda529f4c8413598f + .quad 0xe9ef63ca453d5559 + .quad 0x351e125bc5698e0b + .quad 0x7a385616369b4dcd + .quad 0x75c02ca7655c3563 + .quad 0x7dc21bf9d4f18021 + .quad 0x2f637d7491e6e042 + .quad 0xd4b49b461af67bbe + .quad 0xd603037ac8ab8961 + .quad 0x71dee19ff9a699fb + .quad 0x7f182d06e7ce2a9a + + // 2^72 * 6 * B + + .quad 0x7a7c8e64ab0168ec + .quad 0xcb5a4a5515edc543 + .quad 0x095519d347cd0eda + .quad 0x67d4ac8c343e93b0 + .quad 0x09454b728e217522 + .quad 0xaa58e8f4d484b8d8 + .quad 0xd358254d7f46903c + .quad 0x44acc043241c5217 + .quad 0x1c7d6bbb4f7a5777 + .quad 0x8b35fed4918313e1 + .quad 0x4adca1c6c96b4684 + .quad 0x556d1c8312ad71bd + + // 2^72 * 7 * B + + .quad 0x17ef40e30c8d3982 + .quad 0x31f7073e15a3fa34 + .quad 0x4f21f3cb0773646e + .quad 0x746c6c6d1d824eff + .quad 0x81f06756b11be821 + .quad 0x0faff82310a3f3dd + .quad 0xf8b2d0556a99465d + .quad 0x097abe38cc8c7f05 + .quad 0x0c49c9877ea52da4 + .quad 0x4c4369559bdc1d43 + .quad 0x022c3809f7ccebd2 + .quad 0x577e14a34bee84bd + + // 2^72 * 8 * B + + .quad 0xf0e268ac61a73b0a + .quad 0xf2fafa103791a5f5 + .quad 0xc1e13e826b6d00e9 + .quad 0x60fa7ee96fd78f42 + .quad 0x94fecebebd4dd72b + .quad 0xf46a4fda060f2211 + .quad 0x124a5977c0c8d1ff + .quad 0x705304b8fb009295 + .quad 0xb63d1d354d296ec6 + .quad 0xf3c3053e5fad31d8 + .quad 0x670b958cb4bd42ec + .quad 0x21398e0ca16353fd + + // 2^76 * 1 * B + + .quad 0x216ab2ca8da7d2ef + .quad 0x366ad9dd99f42827 + .quad 0xae64b9004fdd3c75 + .quad 0x403a395b53909e62 + .quad 0x86c5fc16861b7e9a + .quad 0xf6a330476a27c451 + .quad 0x01667267a1e93597 + .quad 0x05ffb9cd6082dfeb + .quad 0xa617fa9ff53f6139 + .quad 0x60f2b5e513e66cb6 + .quad 0xd7a8beefb3448aa4 + .quad 0x7a2932856f5ea192 + + // 2^76 * 2 * B + + .quad 0x0b39d761b02de888 + .quad 0x5f550e7ed2414e1f + .quad 0xa6bfa45822e1a940 + .quad 0x050a2f7dfd447b99 + .quad 0xb89c444879639302 + .quad 0x4ae4f19350c67f2c + .quad 0xf0b35da8c81af9c6 + .quad 0x39d0003546871017 + .quad 0x437c3b33a650db77 + .quad 0x6bafe81dbac52bb2 + .quad 0xfe99402d2db7d318 + .quad 0x2b5b7eec372ba6ce + + // 2^76 * 3 * B + + .quad 0xb3bc4bbd83f50eef + .quad 0x508f0c998c927866 + .quad 0x43e76587c8b7e66e + .quad 0x0f7655a3a47f98d9 + .quad 0xa694404d613ac8f4 + .quad 0x500c3c2bfa97e72c + .quad 0x874104d21fcec210 + .quad 0x1b205fb38604a8ee + .quad 0x55ecad37d24b133c + .quad 0x441e147d6038c90b + .quad 0x656683a1d62c6fee + .quad 0x0157d5dc87e0ecae + + // 2^76 * 4 * B + + .quad 0xf2a7af510354c13d + .quad 0xd7a0b145aa372b60 + .quad 0x2869b96a05a3d470 + .quad 0x6528e42d82460173 + .quad 0x95265514d71eb524 + .quad 0xe603d8815df14593 + .quad 0x147cdf410d4de6b7 + .quad 0x5293b1730437c850 + .quad 0x23d0e0814bccf226 + .quad 0x92c745cd8196fb93 + .quad 0x8b61796c59541e5b + .quad 0x40a44df0c021f978 + + // 2^76 * 5 * B + + .quad 0xdaa869894f20ea6a + .quad 0xea14a3d14c620618 + .quad 0x6001fccb090bf8be + .quad 0x35f4e822947e9cf0 + .quad 0x86c96e514bc5d095 + .quad 0xf20d4098fca6804a + .quad 0x27363d89c826ea5d + .quad 0x39ca36565719cacf + .quad 0x97506f2f6f87b75c + .quad 0xc624aea0034ae070 + .quad 0x1ec856e3aad34dd6 + .quad 0x055b0be0e440e58f + + // 2^76 * 6 * B + + .quad 0x6469a17d89735d12 + .quad 0xdb6f27d5e662b9f1 + .quad 0x9fcba3286a395681 + .quad 0x363b8004d269af25 + .quad 0x4d12a04b6ea33da2 + .quad 0x57cf4c15e36126dd + .quad 0x90ec9675ee44d967 + .quad 0x64ca348d2a985aac + .quad 0x99588e19e4c4912d + .quad 0xefcc3b4e1ca5ce6b + .quad 0x4522ea60fa5b98d5 + .quad 0x7064bbab1de4a819 + + // 2^76 * 7 * B + + .quad 0xb919e1515a770641 + .quad 0xa9a2e2c74e7f8039 + .quad 0x7527250b3df23109 + .quad 0x756a7330ac27b78b + .quad 0xa290c06142542129 + .quad 0xf2e2c2aebe8d5b90 + .quad 0xcf2458db76abfe1b + .quad 0x02157ade83d626bf + .quad 0x3e46972a1b9a038b + .quad 0x2e4ee66a7ee03fb4 + .quad 0x81a248776edbb4ca + .quad 0x1a944ee88ecd0563 + + // 2^76 * 8 * B + + .quad 0xd5a91d1151039372 + .quad 0x2ed377b799ca26de + .quad 0xa17202acfd366b6b + .quad 0x0730291bd6901995 + .quad 0xbb40a859182362d6 + .quad 0xb99f55778a4d1abb + .quad 0x8d18b427758559f6 + .quad 0x26c20fe74d26235a + .quad 0x648d1d9fe9cc22f5 + .quad 0x66bc561928dd577c + .quad 0x47d3ed21652439d1 + .quad 0x49d271acedaf8b49 + + // 2^80 * 1 * B + + .quad 0x89f5058a382b33f3 + .quad 0x5ae2ba0bad48c0b4 + .quad 0x8f93b503a53db36e + .quad 0x5aa3ed9d95a232e6 + .quad 0x2798aaf9b4b75601 + .quad 0x5eac72135c8dad72 + .quad 0xd2ceaa6161b7a023 + .quad 0x1bbfb284e98f7d4e + .quad 0x656777e9c7d96561 + .quad 0xcb2b125472c78036 + .quad 0x65053299d9506eee + .quad 0x4a07e14e5e8957cc + + // 2^80 * 2 * B + + .quad 0x4ee412cb980df999 + .quad 0xa315d76f3c6ec771 + .quad 0xbba5edde925c77fd + .quad 0x3f0bac391d313402 + .quad 0x240b58cdc477a49b + .quad 0xfd38dade6447f017 + .quad 0x19928d32a7c86aad + .quad 0x50af7aed84afa081 + .quad 0x6e4fde0115f65be5 + .quad 0x29982621216109b2 + .quad 0x780205810badd6d9 + .quad 0x1921a316baebd006 + + // 2^80 * 3 * B + + .quad 0x89422f7edfb870fc + .quad 0x2c296beb4f76b3bd + .quad 0x0738f1d436c24df7 + .quad 0x6458df41e273aeb0 + .quad 0xd75aad9ad9f3c18b + .quad 0x566a0eef60b1c19c + .quad 0x3e9a0bac255c0ed9 + .quad 0x7b049deca062c7f5 + .quad 0xdccbe37a35444483 + .quad 0x758879330fedbe93 + .quad 0x786004c312c5dd87 + .quad 0x6093dccbc2950e64 + + // 2^80 * 4 * B + + .quad 0x1ff39a8585e0706d + .quad 0x36d0a5d8b3e73933 + .quad 0x43b9f2e1718f453b + .quad 0x57d1ea084827a97c + .quad 0x6bdeeebe6084034b + .quad 0x3199c2b6780fb854 + .quad 0x973376abb62d0695 + .quad 0x6e3180c98b647d90 + .quad 0xee7ab6e7a128b071 + .quad 0xa4c1596d93a88baa + .quad 0xf7b4de82b2216130 + .quad 0x363e999ddd97bd18 + + // 2^80 * 5 * B + + .quad 0x96a843c135ee1fc4 + .quad 0x976eb35508e4c8cf + .quad 0xb42f6801b58cd330 + .quad 0x48ee9b78693a052b + .quad 0x2f1848dce24baec6 + .quad 0x769b7255babcaf60 + .quad 0x90cb3c6e3cefe931 + .quad 0x231f979bc6f9b355 + .quad 0x5c31de4bcc2af3c6 + .quad 0xb04bb030fe208d1f + .quad 0xb78d7009c14fb466 + .quad 0x079bfa9b08792413 + + // 2^80 * 6 * B + + .quad 0xe3903a51da300df4 + .quad 0x843964233da95ab0 + .quad 0xed3cf12d0b356480 + .quad 0x038c77f684817194 + .quad 0xf3c9ed80a2d54245 + .quad 0x0aa08b7877f63952 + .quad 0xd76dac63d1085475 + .quad 0x1ef4fb159470636b + .quad 0x854e5ee65b167bec + .quad 0x59590a4296d0cdc2 + .quad 0x72b2df3498102199 + .quad 0x575ee92a4a0bff56 + + // 2^80 * 7 * B + + .quad 0xd4c080908a182fcf + .quad 0x30e170c299489dbd + .quad 0x05babd5752f733de + .quad 0x43d4e7112cd3fd00 + .quad 0x5d46bc450aa4d801 + .quad 0xc3af1227a533b9d8 + .quad 0x389e3b262b8906c2 + .quad 0x200a1e7e382f581b + .quad 0x518db967eaf93ac5 + .quad 0x71bc989b056652c0 + .quad 0xfe2b85d9567197f5 + .quad 0x050eca52651e4e38 + + // 2^80 * 8 * B + + .quad 0xc3431ade453f0c9c + .quad 0xe9f5045eff703b9b + .quad 0xfcd97ac9ed847b3d + .quad 0x4b0ee6c21c58f4c6 + .quad 0x97ac397660e668ea + .quad 0x9b19bbfe153ab497 + .quad 0x4cb179b534eca79f + .quad 0x6151c09fa131ae57 + .quad 0x3af55c0dfdf05d96 + .quad 0xdd262ee02ab4ee7a + .quad 0x11b2bb8712171709 + .quad 0x1fef24fa800f030b + + // 2^84 * 1 * B + + .quad 0xb496123a6b6c6609 + .quad 0xa750fe8580ab5938 + .quad 0xf471bf39b7c27a5f + .quad 0x507903ce77ac193c + .quad 0xff91a66a90166220 + .quad 0xf22552ae5bf1e009 + .quad 0x7dff85d87f90df7c + .quad 0x4f620ffe0c736fb9 + .quad 0x62f90d65dfde3e34 + .quad 0xcf28c592b9fa5fad + .quad 0x99c86ef9c6164510 + .quad 0x25d448044a256c84 + + // 2^84 * 2 * B + + .quad 0xbd68230ec7e9b16f + .quad 0x0eb1b9c1c1c5795d + .quad 0x7943c8c495b6b1ff + .quad 0x2f9faf620bbacf5e + .quad 0x2c7c4415c9022b55 + .quad 0x56a0d241812eb1fe + .quad 0xf02ea1c9d7b65e0d + .quad 0x4180512fd5323b26 + .quad 0xa4ff3e698a48a5db + .quad 0xba6a3806bd95403b + .quad 0x9f7ce1af47d5b65d + .quad 0x15e087e55939d2fb + + // 2^84 * 3 * B + + .quad 0x12207543745c1496 + .quad 0xdaff3cfdda38610c + .quad 0xe4e797272c71c34f + .quad 0x39c07b1934bdede9 + .quad 0x8894186efb963f38 + .quad 0x48a00e80dc639bd5 + .quad 0xa4e8092be96c1c99 + .quad 0x5a097d54ca573661 + .quad 0x2d45892b17c9e755 + .quad 0xd033fd7289308df8 + .quad 0x6c2fe9d9525b8bd9 + .quad 0x2edbecf1c11cc079 + + // 2^84 * 4 * B + + .quad 0x1616a4e3c715a0d2 + .quad 0x53623cb0f8341d4d + .quad 0x96ef5329c7e899cb + .quad 0x3d4e8dbba668baa6 + .quad 0xee0f0fddd087a25f + .quad 0x9c7531555c3e34ee + .quad 0x660c572e8fab3ab5 + .quad 0x0854fc44544cd3b2 + .quad 0x61eba0c555edad19 + .quad 0x24b533fef0a83de6 + .quad 0x3b77042883baa5f8 + .quad 0x678f82b898a47e8d + + // 2^84 * 5 * B + + .quad 0xb1491d0bd6900c54 + .quad 0x3539722c9d132636 + .quad 0x4db928920b362bc9 + .quad 0x4d7cd1fea68b69df + .quad 0x1e09d94057775696 + .quad 0xeed1265c3cd951db + .quad 0xfa9dac2b20bce16f + .quad 0x0f7f76e0e8d089f4 + .quad 0x36d9ebc5d485b00c + .quad 0xa2596492e4adb365 + .quad 0xc1659480c2119ccd + .quad 0x45306349186e0d5f + + // 2^84 * 6 * B + + .quad 0x94ddd0c1a6cdff1d + .quad 0x55f6f115e84213ae + .quad 0x6c935f85992fcf6a + .quad 0x067ee0f54a37f16f + .quad 0x96a414ec2b072491 + .quad 0x1bb2218127a7b65b + .quad 0x6d2849596e8a4af0 + .quad 0x65f3b08ccd27765f + .quad 0xecb29fff199801f7 + .quad 0x9d361d1fa2a0f72f + .quad 0x25f11d2375fd2f49 + .quad 0x124cefe80fe10fe2 + + // 2^84 * 7 * B + + .quad 0x4c126cf9d18df255 + .quad 0xc1d471e9147a63b6 + .quad 0x2c6d3c73f3c93b5f + .quad 0x6be3a6a2e3ff86a2 + .quad 0x1518e85b31b16489 + .quad 0x8faadcb7db710bfb + .quad 0x39b0bdf4a14ae239 + .quad 0x05f4cbea503d20c1 + .quad 0xce040e9ec04145bc + .quad 0xc71ff4e208f6834c + .quad 0xbd546e8dab8847a3 + .quad 0x64666aa0a4d2aba5 + + // 2^84 * 8 * B + + .quad 0x6841435a7c06d912 + .quad 0xca123c21bb3f830b + .quad 0xd4b37b27b1cbe278 + .quad 0x1d753b84c76f5046 + .quad 0xb0c53bf73337e94c + .quad 0x7cb5697e11e14f15 + .quad 0x4b84abac1930c750 + .quad 0x28dd4abfe0640468 + .quad 0x7dc0b64c44cb9f44 + .quad 0x18a3e1ace3925dbf + .quad 0x7a3034862d0457c4 + .quad 0x4c498bf78a0c892e + + // 2^88 * 1 * B + + .quad 0x37d653fb1aa73196 + .quad 0x0f9495303fd76418 + .quad 0xad200b09fb3a17b2 + .quad 0x544d49292fc8613e + .quad 0x22d2aff530976b86 + .quad 0x8d90b806c2d24604 + .quad 0xdca1896c4de5bae5 + .quad 0x28005fe6c8340c17 + .quad 0x6aefba9f34528688 + .quad 0x5c1bff9425107da1 + .quad 0xf75bbbcd66d94b36 + .quad 0x72e472930f316dfa + + // 2^88 * 2 * B + + .quad 0x2695208c9781084f + .quad 0xb1502a0b23450ee1 + .quad 0xfd9daea603efde02 + .quad 0x5a9d2e8c2733a34c + .quad 0x07f3f635d32a7627 + .quad 0x7aaa4d865f6566f0 + .quad 0x3c85e79728d04450 + .quad 0x1fee7f000fe06438 + .quad 0x765305da03dbf7e5 + .quad 0xa4daf2491434cdbd + .quad 0x7b4ad5cdd24a88ec + .quad 0x00f94051ee040543 + + // 2^88 * 3 * B + + .quad 0x8d356b23c3d330b2 + .quad 0xf21c8b9bb0471b06 + .quad 0xb36c316c6e42b83c + .quad 0x07d79c7e8beab10d + .quad 0xd7ef93bb07af9753 + .quad 0x583ed0cf3db766a7 + .quad 0xce6998bf6e0b1ec5 + .quad 0x47b7ffd25dd40452 + .quad 0x87fbfb9cbc08dd12 + .quad 0x8a066b3ae1eec29b + .quad 0x0d57242bdb1fc1bf + .quad 0x1c3520a35ea64bb6 + + // 2^88 * 4 * B + + .quad 0x80d253a6bccba34a + .quad 0x3e61c3a13838219b + .quad 0x90c3b6019882e396 + .quad 0x1c3d05775d0ee66f + .quad 0xcda86f40216bc059 + .quad 0x1fbb231d12bcd87e + .quad 0xb4956a9e17c70990 + .quad 0x38750c3b66d12e55 + .quad 0x692ef1409422e51a + .quad 0xcbc0c73c2b5df671 + .quad 0x21014fe7744ce029 + .quad 0x0621e2c7d330487c + + // 2^88 * 5 * B + + .quad 0xaf9860cc8259838d + .quad 0x90ea48c1c69f9adc + .quad 0x6526483765581e30 + .quad 0x0007d6097bd3a5bc + .quad 0xb7ae1796b0dbf0f3 + .quad 0x54dfafb9e17ce196 + .quad 0x25923071e9aaa3b4 + .quad 0x5d8e589ca1002e9d + .quad 0xc0bf1d950842a94b + .quad 0xb2d3c363588f2e3e + .quad 0x0a961438bb51e2ef + .quad 0x1583d7783c1cbf86 + + // 2^88 * 6 * B + + .quad 0xeceea2ef5da27ae1 + .quad 0x597c3a1455670174 + .quad 0xc9a62a126609167a + .quad 0x252a5f2e81ed8f70 + .quad 0x90034704cc9d28c7 + .quad 0x1d1b679ef72cc58f + .quad 0x16e12b5fbe5b8726 + .quad 0x4958064e83c5580a + .quad 0x0d2894265066e80d + .quad 0xfcc3f785307c8c6b + .quad 0x1b53da780c1112fd + .quad 0x079c170bd843b388 + + // 2^88 * 7 * B + + .quad 0x0506ece464fa6fff + .quad 0xbee3431e6205e523 + .quad 0x3579422451b8ea42 + .quad 0x6dec05e34ac9fb00 + .quad 0xcdd6cd50c0d5d056 + .quad 0x9af7686dbb03573b + .quad 0x3ca6723ff3c3ef48 + .quad 0x6768c0d7317b8acc + .quad 0x94b625e5f155c1b3 + .quad 0x417bf3a7997b7b91 + .quad 0xc22cbddc6d6b2600 + .quad 0x51445e14ddcd52f4 + + // 2^88 * 8 * B + + .quad 0x57502b4b3b144951 + .quad 0x8e67ff6b444bbcb3 + .quad 0xb8bd6927166385db + .quad 0x13186f31e39295c8 + .quad 0x893147ab2bbea455 + .quad 0x8c53a24f92079129 + .quad 0x4b49f948be30f7a7 + .quad 0x12e990086e4fd43d + .quad 0xf10c96b37fdfbb2e + .quad 0x9f9a935e121ceaf9 + .quad 0xdf1136c43a5b983f + .quad 0x77b2e3f05d3e99af + + // 2^92 * 1 * B + + .quad 0xfd0d75879cf12657 + .quad 0xe82fef94e53a0e29 + .quad 0xcc34a7f05bbb4be7 + .quad 0x0b251172a50c38a2 + .quad 0x9532f48fcc5cd29b + .quad 0x2ba851bea3ce3671 + .quad 0x32dacaa051122941 + .quad 0x478d99d9350004f2 + .quad 0x1d5ad94890bb02c0 + .quad 0x50e208b10ec25115 + .quad 0xa26a22894ef21702 + .quad 0x4dc923343b524805 + + // 2^92 * 2 * B + + .quad 0xe3828c400f8086b6 + .quad 0x3f77e6f7979f0dc8 + .quad 0x7ef6de304df42cb4 + .quad 0x5265797cb6abd784 + .quad 0x3ad3e3ebf36c4975 + .quad 0xd75d25a537862125 + .quad 0xe873943da025a516 + .quad 0x6bbc7cb4c411c847 + .quad 0x3c6f9cd1d4a50d56 + .quad 0xb6244077c6feab7e + .quad 0x6ff9bf483580972e + .quad 0x00375883b332acfb + + // 2^92 * 3 * B + + .quad 0x0001b2cd28cb0940 + .quad 0x63fb51a06f1c24c9 + .quad 0xb5ad8691dcd5ca31 + .quad 0x67238dbd8c450660 + .quad 0xc98bec856c75c99c + .quad 0xe44184c000e33cf4 + .quad 0x0a676b9bba907634 + .quad 0x669e2cb571f379d7 + .quad 0xcb116b73a49bd308 + .quad 0x025aad6b2392729e + .quad 0xb4793efa3f55d9b1 + .quad 0x72a1056140678bb9 + + // 2^92 * 4 * B + + .quad 0xa2b6812b1cc9249d + .quad 0x62866eee21211f58 + .quad 0x2cb5c5b85df10ece + .quad 0x03a6b259e263ae00 + .quad 0x0d8d2909e2e505b6 + .quad 0x98ca78abc0291230 + .quad 0x77ef5569a9b12327 + .quad 0x7c77897b81439b47 + .quad 0xf1c1b5e2de331cb5 + .quad 0x5a9f5d8e15fca420 + .quad 0x9fa438f17bd932b1 + .quad 0x2a381bf01c6146e7 + + // 2^92 * 5 * B + + .quad 0xac9b9879cfc811c1 + .quad 0x8b7d29813756e567 + .quad 0x50da4e607c70edfc + .quad 0x5dbca62f884400b6 + .quad 0xf7c0be32b534166f + .quad 0x27e6ca6419cf70d4 + .quad 0x934df7d7a957a759 + .quad 0x5701461dabdec2aa + .quad 0x2c6747402c915c25 + .quad 0x1bdcd1a80b0d340a + .quad 0x5e5601bd07b43f5f + .quad 0x2555b4e05539a242 + + // 2^92 * 6 * B + + .quad 0x6fc09f5266ddd216 + .quad 0xdce560a7c8e37048 + .quad 0xec65939da2df62fd + .quad 0x7a869ae7e52ed192 + .quad 0x78409b1d87e463d4 + .quad 0xad4da95acdfb639d + .quad 0xec28773755259b9c + .quad 0x69c806e9c31230ab + .quad 0x7b48f57414bb3f22 + .quad 0x68c7cee4aedccc88 + .quad 0xed2f936179ed80be + .quad 0x25d70b885f77bc4b + + // 2^92 * 7 * B + + .quad 0x4151c3d9762bf4de + .quad 0x083f435f2745d82b + .quad 0x29775a2e0d23ddd5 + .quad 0x138e3a6269a5db24 + .quad 0x98459d29bb1ae4d4 + .quad 0x56b9c4c739f954ec + .quad 0x832743f6c29b4b3e + .quad 0x21ea8e2798b6878a + .quad 0x87bef4b46a5a7b9c + .quad 0xd2299d1b5fc1d062 + .quad 0x82409818dd321648 + .quad 0x5c5abeb1e5a2e03d + + // 2^92 * 8 * B + + .quad 0x14722af4b73c2ddb + .quad 0xbc470c5f5a05060d + .quad 0x00943eac2581b02e + .quad 0x0e434b3b1f499c8f + .quad 0x02cde6de1306a233 + .quad 0x7b5a52a2116f8ec7 + .quad 0xe1c681f4c1163b5b + .quad 0x241d350660d32643 + .quad 0x6be4404d0ebc52c7 + .quad 0xae46233bb1a791f5 + .quad 0x2aec170ed25db42b + .quad 0x1d8dfd966645d694 + + // 2^96 * 1 * B + + .quad 0x296fa9c59c2ec4de + .quad 0xbc8b61bf4f84f3cb + .quad 0x1c7706d917a8f908 + .quad 0x63b795fc7ad3255d + .quad 0xd598639c12ddb0a4 + .quad 0xa5d19f30c024866b + .quad 0xd17c2f0358fce460 + .quad 0x07a195152e095e8a + .quad 0xa8368f02389e5fc8 + .quad 0x90433b02cf8de43b + .quad 0xafa1fd5dc5412643 + .quad 0x3e8fe83d032f0137 + + // 2^96 * 2 * B + + .quad 0x2f8b15b90570a294 + .quad 0x94f2427067084549 + .quad 0xde1c5ae161bbfd84 + .quad 0x75ba3b797fac4007 + .quad 0x08704c8de8efd13c + .quad 0xdfc51a8e33e03731 + .quad 0xa59d5da51260cde3 + .quad 0x22d60899a6258c86 + .quad 0x6239dbc070cdd196 + .quad 0x60fe8a8b6c7d8a9a + .quad 0xb38847bceb401260 + .quad 0x0904d07b87779e5e + + // 2^96 * 3 * B + + .quad 0xb4ce1fd4ddba919c + .quad 0xcf31db3ec74c8daa + .quad 0x2c63cc63ad86cc51 + .quad 0x43e2143fbc1dde07 + .quad 0xf4322d6648f940b9 + .quad 0x06952f0cbd2d0c39 + .quad 0x167697ada081f931 + .quad 0x6240aacebaf72a6c + .quad 0xf834749c5ba295a0 + .quad 0xd6947c5bca37d25a + .quad 0x66f13ba7e7c9316a + .quad 0x56bdaf238db40cac + + // 2^96 * 4 * B + + .quad 0x362ab9e3f53533eb + .quad 0x338568d56eb93d40 + .quad 0x9e0e14521d5a5572 + .quad 0x1d24a86d83741318 + .quad 0x1310d36cc19d3bb2 + .quad 0x062a6bb7622386b9 + .quad 0x7c9b8591d7a14f5c + .quad 0x03aa31507e1e5754 + .quad 0xf4ec7648ffd4ce1f + .quad 0xe045eaf054ac8c1c + .quad 0x88d225821d09357c + .quad 0x43b261dc9aeb4859 + + // 2^96 * 5 * B + + .quad 0xe55b1e1988bb79bb + .quad 0xa09ed07dc17a359d + .quad 0xb02c2ee2603dea33 + .quad 0x326055cf5b276bc2 + .quad 0x19513d8b6c951364 + .quad 0x94fe7126000bf47b + .quad 0x028d10ddd54f9567 + .quad 0x02b4d5e242940964 + .quad 0xb4a155cb28d18df2 + .quad 0xeacc4646186ce508 + .quad 0xc49cf4936c824389 + .quad 0x27a6c809ae5d3410 + + // 2^96 * 6 * B + + .quad 0x8ba6ebcd1f0db188 + .quad 0x37d3d73a675a5be8 + .quad 0xf22edfa315f5585a + .quad 0x2cb67174ff60a17e + .quad 0xcd2c270ac43d6954 + .quad 0xdd4a3e576a66cab2 + .quad 0x79fa592469d7036c + .quad 0x221503603d8c2599 + .quad 0x59eecdf9390be1d0 + .quad 0xa9422044728ce3f1 + .quad 0x82891c667a94f0f4 + .quad 0x7b1df4b73890f436 + + // 2^96 * 7 * B + + .quad 0xe492f2e0b3b2a224 + .quad 0x7c6c9e062b551160 + .quad 0x15eb8fe20d7f7b0e + .quad 0x61fcef2658fc5992 + .quad 0x5f2e221807f8f58c + .quad 0xe3555c9fd49409d4 + .quad 0xb2aaa88d1fb6a630 + .quad 0x68698245d352e03d + .quad 0xdbb15d852a18187a + .quad 0xf3e4aad386ddacd7 + .quad 0x44bae2810ff6c482 + .quad 0x46cf4c473daf01cf + + // 2^96 * 8 * B + + .quad 0x426525ed9ec4e5f9 + .quad 0x0e5eda0116903303 + .quad 0x72b1a7f2cbe5cadc + .quad 0x29387bcd14eb5f40 + .quad 0x213c6ea7f1498140 + .quad 0x7c1e7ef8392b4854 + .quad 0x2488c38c5629ceba + .quad 0x1065aae50d8cc5bb + .quad 0x1c2c4525df200d57 + .quad 0x5c3b2dd6bfca674a + .quad 0x0a07e7b1e1834030 + .quad 0x69a198e64f1ce716 + + // 2^100 * 1 * B + + .quad 0x7afcd613efa9d697 + .quad 0x0cc45aa41c067959 + .quad 0xa56fe104c1fada96 + .quad 0x3a73b70472e40365 + .quad 0x7b26e56b9e2d4734 + .quad 0xc4c7132b81c61675 + .quad 0xef5c9525ec9cde7f + .quad 0x39c80b16e71743ad + .quad 0x0f196e0d1b826c68 + .quad 0xf71ff0e24960e3db + .quad 0x6113167023b7436c + .quad 0x0cf0ea5877da7282 + + // 2^100 * 2 * B + + .quad 0x196c80a4ddd4ccbd + .quad 0x22e6f55d95f2dd9d + .quad 0xc75e33c740d6c71b + .quad 0x7bb51279cb3c042f + .quad 0xe332ced43ba6945a + .quad 0xde0b1361e881c05d + .quad 0x1ad40f095e67ed3b + .quad 0x5da8acdab8c63d5d + .quad 0xc4b6664a3a70159f + .quad 0x76194f0f0a904e14 + .quad 0xa5614c39a4096c13 + .quad 0x6cd0ff50979feced + + // 2^100 * 3 * B + + .quad 0xc0e067e78f4428ac + .quad 0x14835ab0a61135e3 + .quad 0xf21d14f338062935 + .quad 0x6390a4c8df04849c + .quad 0x7fecfabdb04ba18e + .quad 0xd0fc7bfc3bddbcf7 + .quad 0xa41d486e057a131c + .quad 0x641a4391f2223a61 + .quad 0xc5c6b95aa606a8db + .quad 0x914b7f9eb06825f1 + .quad 0x2a731f6b44fc9eff + .quad 0x30ddf38562705cfc + + // 2^100 * 4 * B + + .quad 0x4e3dcbdad1bff7f9 + .quad 0xc9118e8220645717 + .quad 0xbacccebc0f189d56 + .quad 0x1b4822e9d4467668 + .quad 0x33bef2bd68bcd52c + .quad 0xc649dbb069482ef2 + .quad 0xb5b6ee0c41cb1aee + .quad 0x5c294d270212a7e5 + .quad 0xab360a7f25563781 + .quad 0x2512228a480f7958 + .quad 0xc75d05276114b4e3 + .quad 0x222d9625d976fe2a + + // 2^100 * 5 * B + + .quad 0x1c717f85b372ace1 + .quad 0x81930e694638bf18 + .quad 0x239cad056bc08b58 + .quad 0x0b34271c87f8fff4 + .quad 0x0f94be7e0a344f85 + .quad 0xeb2faa8c87f22c38 + .quad 0x9ce1e75e4ee16f0f + .quad 0x43e64e5418a08dea + .quad 0x8155e2521a35ce63 + .quad 0xbe100d4df912028e + .quad 0xbff80bf8a57ddcec + .quad 0x57342dc96d6bc6e4 + + // 2^100 * 6 * B + + .quad 0xefeef065c8ce5998 + .quad 0xbf029510b5cbeaa2 + .quad 0x8c64a10620b7c458 + .quad 0x35134fb231c24855 + .quad 0xf3c3bcb71e707bf6 + .quad 0x351d9b8c7291a762 + .quad 0x00502e6edad69a33 + .quad 0x522f521f1ec8807f + .quad 0x272c1f46f9a3902b + .quad 0xc91ba3b799657bcc + .quad 0xae614b304f8a1c0e + .quad 0x7afcaad70b99017b + + // 2^100 * 7 * B + + .quad 0xc25ded54a4b8be41 + .quad 0x902d13e11bb0e2dd + .quad 0x41f43233cde82ab2 + .quad 0x1085faa5c3aae7cb + .quad 0xa88141ecef842b6b + .quad 0x55e7b14797abe6c5 + .quad 0x8c748f9703784ffe + .quad 0x5b50a1f7afcd00b7 + .quad 0x9b840f66f1361315 + .quad 0x18462242701003e9 + .quad 0x65ed45fae4a25080 + .quad 0x0a2862393fda7320 + + // 2^100 * 8 * B + + .quad 0x46ab13c8347cbc9d + .quad 0x3849e8d499c12383 + .quad 0x4cea314087d64ac9 + .quad 0x1f354134b1a29ee7 + .quad 0x960e737b6ecb9d17 + .quad 0xfaf24948d67ceae1 + .quad 0x37e7a9b4d55e1b89 + .quad 0x5cb7173cb46c59eb + .quad 0x4a89e68b82b7abf0 + .quad 0xf41cd9279ba6b7b9 + .quad 0x16e6c210e18d876f + .quad 0x7cacdb0f7f1b09c6 + + // 2^104 * 1 * B + + .quad 0x9062b2e0d91a78bc + .quad 0x47c9889cc8509667 + .quad 0x9df54a66405070b8 + .quad 0x7369e6a92493a1bf + .quad 0xe1014434dcc5caed + .quad 0x47ed5d963c84fb33 + .quad 0x70019576ed86a0e7 + .quad 0x25b2697bd267f9e4 + .quad 0x9d673ffb13986864 + .quad 0x3ca5fbd9415dc7b8 + .quad 0xe04ecc3bdf273b5e + .quad 0x1420683db54e4cd2 + + // 2^104 * 2 * B + + .quad 0xb478bd1e249dd197 + .quad 0x620c35005e58c102 + .quad 0xfb02d32fccbaac5c + .quad 0x60b63bebf508a72d + .quad 0x34eebb6fc1cc5ad0 + .quad 0x6a1b0ce99646ac8b + .quad 0xd3b0da49a66bde53 + .quad 0x31e83b4161d081c1 + .quad 0x97e8c7129e062b4f + .quad 0x49e48f4f29320ad8 + .quad 0x5bece14b6f18683f + .quad 0x55cf1eb62d550317 + + // 2^104 * 3 * B + + .quad 0x5879101065c23d58 + .quad 0x8b9d086d5094819c + .quad 0xe2402fa912c55fa7 + .quad 0x669a6564570891d4 + .quad 0x3076b5e37df58c52 + .quad 0xd73ab9dde799cc36 + .quad 0xbd831ce34913ee20 + .quad 0x1a56fbaa62ba0133 + .quad 0x943e6b505c9dc9ec + .quad 0x302557bba77c371a + .quad 0x9873ae5641347651 + .quad 0x13c4836799c58a5c + + // 2^104 * 4 * B + + .quad 0x423a5d465ab3e1b9 + .quad 0xfc13c187c7f13f61 + .quad 0x19f83664ecb5b9b6 + .quad 0x66f80c93a637b607 + .quad 0xc4dcfb6a5d8bd080 + .quad 0xdeebc4ec571a4842 + .quad 0xd4b2e883b8e55365 + .quad 0x50bdc87dc8e5b827 + .quad 0x606d37836edfe111 + .quad 0x32353e15f011abd9 + .quad 0x64b03ac325b73b96 + .quad 0x1dd56444725fd5ae + + // 2^104 * 5 * B + + .quad 0x8fa47ff83362127d + .quad 0xbc9f6ac471cd7c15 + .quad 0x6e71454349220c8b + .quad 0x0e645912219f732e + .quad 0xc297e60008bac89a + .quad 0x7d4cea11eae1c3e0 + .quad 0xf3e38be19fe7977c + .quad 0x3a3a450f63a305cd + .quad 0x078f2f31d8394627 + .quad 0x389d3183de94a510 + .quad 0xd1e36c6d17996f80 + .quad 0x318c8d9393a9a87b + + // 2^104 * 6 * B + + .quad 0xf2745d032afffe19 + .quad 0x0c9f3c497f24db66 + .quad 0xbc98d3e3ba8598ef + .quad 0x224c7c679a1d5314 + .quad 0x5d669e29ab1dd398 + .quad 0xfc921658342d9e3b + .quad 0x55851dfdf35973cd + .quad 0x509a41c325950af6 + .quad 0xbdc06edca6f925e9 + .quad 0x793ef3f4641b1f33 + .quad 0x82ec12809d833e89 + .quad 0x05bff02328a11389 + + // 2^104 * 7 * B + + .quad 0x3632137023cae00b + .quad 0x544acf0ad1accf59 + .quad 0x96741049d21a1c88 + .quad 0x780b8cc3fa2a44a7 + .quad 0x6881a0dd0dc512e4 + .quad 0x4fe70dc844a5fafe + .quad 0x1f748e6b8f4a5240 + .quad 0x576277cdee01a3ea + .quad 0x1ef38abc234f305f + .quad 0x9a577fbd1405de08 + .quad 0x5e82a51434e62a0d + .quad 0x5ff418726271b7a1 + + // 2^104 * 8 * B + + .quad 0x398e080c1789db9d + .quad 0xa7602025f3e778f5 + .quad 0xfa98894c06bd035d + .quad 0x106a03dc25a966be + .quad 0xe5db47e813b69540 + .quad 0xf35d2a3b432610e1 + .quad 0xac1f26e938781276 + .quad 0x29d4db8ca0a0cb69 + .quad 0xd9ad0aaf333353d0 + .quad 0x38669da5acd309e5 + .quad 0x3c57658ac888f7f0 + .quad 0x4ab38a51052cbefa + + // 2^108 * 1 * B + + .quad 0xdfdacbee4324c0e9 + .quad 0x054442883f955bb7 + .quad 0xdef7aaa8ea31609f + .quad 0x68aee70642287cff + .quad 0xf68fe2e8809de054 + .quad 0xe3bc096a9c82bad1 + .quad 0x076353d40aadbf45 + .quad 0x7b9b1fb5dea1959e + .quad 0xf01cc8f17471cc0c + .quad 0x95242e37579082bb + .quad 0x27776093d3e46b5f + .quad 0x2d13d55a28bd85fb + + // 2^108 * 2 * B + + .quad 0xfac5d2065b35b8da + .quad 0xa8da8a9a85624bb7 + .quad 0xccd2ca913d21cd0f + .quad 0x6b8341ee8bf90d58 + .quad 0xbf019cce7aee7a52 + .quad 0xa8ded2b6e454ead3 + .quad 0x3c619f0b87a8bb19 + .quad 0x3619b5d7560916d8 + .quad 0x3579f26b0282c4b2 + .quad 0x64d592f24fafefae + .quad 0xb7cded7b28c8c7c0 + .quad 0x6a927b6b7173a8d7 + + // 2^108 * 3 * B + + .quad 0x1f6db24f986e4656 + .quad 0x1021c02ed1e9105b + .quad 0xf8ff3fff2cc0a375 + .quad 0x1d2a6bf8c6c82592 + .quad 0x8d7040863ece88eb + .quad 0xf0e307a980eec08c + .quad 0xac2250610d788fda + .quad 0x056d92a43a0d478d + .quad 0x1b05a196fc3da5a1 + .quad 0x77d7a8c243b59ed0 + .quad 0x06da3d6297d17918 + .quad 0x66fbb494f12353f7 + + // 2^108 * 4 * B + + .quad 0x751a50b9d85c0fb8 + .quad 0xd1afdc258bcf097b + .quad 0x2f16a6a38309a969 + .quad 0x14ddff9ee5b00659 + .quad 0xd6d70996f12309d6 + .quad 0xdbfb2385e9c3d539 + .quad 0x46d602b0f7552411 + .quad 0x270a0b0557843e0c + .quad 0x61ff0640a7862bcc + .quad 0x81cac09a5f11abfe + .quad 0x9047830455d12abb + .quad 0x19a4bde1945ae873 + + // 2^108 * 5 * B + + .quad 0x9b9f26f520a6200a + .quad 0x64804443cf13eaf8 + .quad 0x8a63673f8631edd3 + .quad 0x72bbbce11ed39dc1 + .quad 0x40c709dec076c49f + .quad 0x657bfaf27f3e53f6 + .quad 0x40662331eca042c4 + .quad 0x14b375487eb4df04 + .quad 0xae853c94ab66dc47 + .quad 0xeb62343edf762d6e + .quad 0xf08e0e186fb2f7d1 + .quad 0x4f0b1c02700ab37a + + // 2^108 * 6 * B + + .quad 0xe1706787d81951fa + .quad 0xa10a2c8eb290c77b + .quad 0xe7382fa03ed66773 + .quad 0x0a4d84710bcc4b54 + .quad 0x79fd21ccc1b2e23f + .quad 0x4ae7c281453df52a + .quad 0xc8172ec9d151486b + .quad 0x68abe9443e0a7534 + .quad 0xda12c6c407831dcb + .quad 0x0da230d74d5c510d + .quad 0x4ab1531e6bd404e1 + .quad 0x4106b166bcf440ef + + // 2^108 * 7 * B + + .quad 0x02e57a421cd23668 + .quad 0x4ad9fb5d0eaef6fd + .quad 0x954e6727b1244480 + .quad 0x7f792f9d2699f331 + .quad 0xa485ccd539e4ecf2 + .quad 0x5aa3f3ad0555bab5 + .quad 0x145e3439937df82d + .quad 0x1238b51e1214283f + .quad 0x0b886b925fd4d924 + .quad 0x60906f7a3626a80d + .quad 0xecd367b4b98abd12 + .quad 0x2876beb1def344cf + + // 2^108 * 8 * B + + .quad 0xdc84e93563144691 + .quad 0x632fe8a0d61f23f4 + .quad 0x4caa800612a9a8d5 + .quad 0x48f9dbfa0e9918d3 + .quad 0xd594b3333a8a85f8 + .quad 0x4ea37689e78d7d58 + .quad 0x73bf9f455e8e351f + .quad 0x5507d7d2bc41ebb4 + .quad 0x1ceb2903299572fc + .quad 0x7c8ccaa29502d0ee + .quad 0x91bfa43411cce67b + .quad 0x5784481964a831e7 + + // 2^112 * 1 * B + + .quad 0xda7c2b256768d593 + .quad 0x98c1c0574422ca13 + .quad 0xf1a80bd5ca0ace1d + .quad 0x29cdd1adc088a690 + .quad 0xd6cfd1ef5fddc09c + .quad 0xe82b3efdf7575dce + .quad 0x25d56b5d201634c2 + .quad 0x3041c6bb04ed2b9b + .quad 0x0ff2f2f9d956e148 + .quad 0xade797759f356b2e + .quad 0x1a4698bb5f6c025c + .quad 0x104bbd6814049a7b + + // 2^112 * 2 * B + + .quad 0x51f0fd3168f1ed67 + .quad 0x2c811dcdd86f3bc2 + .quad 0x44dc5c4304d2f2de + .quad 0x5be8cc57092a7149 + .quad 0xa95d9a5fd67ff163 + .quad 0xe92be69d4cc75681 + .quad 0xb7f8024cde20f257 + .quad 0x204f2a20fb072df5 + .quad 0xc8143b3d30ebb079 + .quad 0x7589155abd652e30 + .quad 0x653c3c318f6d5c31 + .quad 0x2570fb17c279161f + + // 2^112 * 3 * B + + .quad 0x3efa367f2cb61575 + .quad 0xf5f96f761cd6026c + .quad 0xe8c7142a65b52562 + .quad 0x3dcb65ea53030acd + .quad 0x192ea9550bb8245a + .quad 0xc8e6fba88f9050d1 + .quad 0x7986ea2d88a4c935 + .quad 0x241c5f91de018668 + .quad 0x28d8172940de6caa + .quad 0x8fbf2cf022d9733a + .quad 0x16d7fcdd235b01d1 + .quad 0x08420edd5fcdf0e5 + + // 2^112 * 4 * B + + .quad 0xcdff20ab8362fa4a + .quad 0x57e118d4e21a3e6e + .quad 0xe3179617fc39e62b + .quad 0x0d9a53efbc1769fd + .quad 0x0358c34e04f410ce + .quad 0xb6135b5a276e0685 + .quad 0x5d9670c7ebb91521 + .quad 0x04d654f321db889c + .quad 0x5e7dc116ddbdb5d5 + .quad 0x2954deb68da5dd2d + .quad 0x1cb608173334a292 + .quad 0x4a7a4f2618991ad7 + + // 2^112 * 5 * B + + .quad 0xf4a718025fb15f95 + .quad 0x3df65f346b5c1b8f + .quad 0xcdfcf08500e01112 + .quad 0x11b50c4cddd31848 + .quad 0x24c3b291af372a4b + .quad 0x93da8270718147f2 + .quad 0xdd84856486899ef2 + .quad 0x4a96314223e0ee33 + .quad 0xa6e8274408a4ffd6 + .quad 0x738e177e9c1576d9 + .quad 0x773348b63d02b3f2 + .quad 0x4f4bce4dce6bcc51 + + // 2^112 * 6 * B + + .quad 0xa71fce5ae2242584 + .quad 0x26ea725692f58a9e + .quad 0xd21a09d71cea3cf4 + .quad 0x73fcdd14b71c01e6 + .quad 0x30e2616ec49d0b6f + .quad 0xe456718fcaec2317 + .quad 0x48eb409bf26b4fa6 + .quad 0x3042cee561595f37 + .quad 0x427e7079449bac41 + .quad 0x855ae36dbce2310a + .quad 0x4cae76215f841a7c + .quad 0x389e740c9a9ce1d6 + + // 2^112 * 7 * B + + .quad 0x64fcb3ae34dcb9ce + .quad 0x97500323e348d0ad + .quad 0x45b3f07d62c6381b + .quad 0x61545379465a6788 + .quad 0xc9bd78f6570eac28 + .quad 0xe55b0b3227919ce1 + .quad 0x65fc3eaba19b91ed + .quad 0x25c425e5d6263690 + .quad 0x3f3e06a6f1d7de6e + .quad 0x3ef976278e062308 + .quad 0x8c14f6264e8a6c77 + .quad 0x6539a08915484759 + + // 2^112 * 8 * B + + .quad 0xe9d21f74c3d2f773 + .quad 0xc150544125c46845 + .quad 0x624e5ce8f9b99e33 + .quad 0x11c5e4aac5cd186c + .quad 0xddc4dbd414bb4a19 + .quad 0x19b2bc3c98424f8e + .quad 0x48a89fd736ca7169 + .quad 0x0f65320ef019bd90 + .quad 0xd486d1b1cafde0c6 + .quad 0x4f3fe6e3163b5181 + .quad 0x59a8af0dfaf2939a + .quad 0x4cabc7bdec33072a + + // 2^116 * 1 * B + + .quad 0x16faa8fb532f7428 + .quad 0xdbd42ea046a4e272 + .quad 0x5337653b8b9ea480 + .quad 0x4065947223973f03 + .quad 0xf7c0a19c1a54a044 + .quad 0x4a1c5e2477bd9fbb + .quad 0xa6e3ca115af22972 + .quad 0x1819bb953f2e9e0d + .quad 0x498fbb795e042e84 + .quad 0x7d0dd89a7698b714 + .quad 0x8bfb0ba427fe6295 + .quad 0x36ba82e721200524 + + // 2^116 * 2 * B + + .quad 0xd60ecbb74245ec41 + .quad 0xfd9be89e34348716 + .quad 0xc9240afee42284de + .quad 0x4472f648d0531db4 + .quad 0xc8d69d0a57274ed5 + .quad 0x45ba803260804b17 + .quad 0xdf3cda102255dfac + .quad 0x77d221232709b339 + .quad 0x498a6d7064ad94d8 + .quad 0xa5b5c8fd9af62263 + .quad 0x8ca8ed0545c141f4 + .quad 0x2c63bec3662d358c + + // 2^116 * 3 * B + + .quad 0x7fe60d8bea787955 + .quad 0xb9dc117eb5f401b7 + .quad 0x91c7c09a19355cce + .quad 0x22692ef59442bedf + .quad 0x9a518b3a8586f8bf + .quad 0x9ee71af6cbb196f0 + .quad 0xaa0625e6a2385cf2 + .quad 0x1deb2176ddd7c8d1 + .quad 0x8563d19a2066cf6c + .quad 0x401bfd8c4dcc7cd7 + .quad 0xd976a6becd0d8f62 + .quad 0x67cfd773a278b05e + + // 2^116 * 4 * B + + .quad 0x8dec31faef3ee475 + .quad 0x99dbff8a9e22fd92 + .quad 0x512d11594e26cab1 + .quad 0x0cde561eec4310b9 + .quad 0x2d5fa9855a4e586a + .quad 0x65f8f7a449beab7e + .quad 0xaa074dddf21d33d3 + .quad 0x185cba721bcb9dee + .quad 0x93869da3f4e3cb41 + .quad 0xbf0392f540f7977e + .quad 0x026204fcd0463b83 + .quad 0x3ec91a769eec6eed + + // 2^116 * 5 * B + + .quad 0x1e9df75bf78166ad + .quad 0x4dfda838eb0cd7af + .quad 0xba002ed8c1eaf988 + .quad 0x13fedb3e11f33cfc + .quad 0x0fad2fb7b0a3402f + .quad 0x46615ecbfb69f4a8 + .quad 0xf745bcc8c5f8eaa6 + .quad 0x7a5fa8794a94e896 + .quad 0x52958faa13cd67a1 + .quad 0x965ee0818bdbb517 + .quad 0x16e58daa2e8845b3 + .quad 0x357d397d5499da8f + + // 2^116 * 6 * B + + .quad 0x1ebfa05fb0bace6c + .quad 0xc934620c1caf9a1e + .quad 0xcc771cc41d82b61a + .quad 0x2d94a16aa5f74fec + .quad 0x481dacb4194bfbf8 + .quad 0x4d77e3f1bae58299 + .quad 0x1ef4612e7d1372a0 + .quad 0x3a8d867e70ff69e1 + .quad 0x6f58cd5d55aff958 + .quad 0xba3eaa5c75567721 + .quad 0x75c123999165227d + .quad 0x69be1343c2f2b35e + + // 2^116 * 7 * B + + .quad 0x0e091d5ee197c92a + .quad 0x4f51019f2945119f + .quad 0x143679b9f034e99c + .quad 0x7d88112e4d24c696 + .quad 0x82bbbdac684b8de3 + .quad 0xa2f4c7d03fca0718 + .quad 0x337f92fbe096aaa8 + .quad 0x200d4d8c63587376 + .quad 0x208aed4b4893b32b + .quad 0x3efbf23ebe59b964 + .quad 0xd762deb0dba5e507 + .quad 0x69607bd681bd9d94 + + // 2^116 * 8 * B + + .quad 0xf6be021068de1ce1 + .quad 0xe8d518e70edcbc1f + .quad 0xe3effdd01b5505a5 + .quad 0x35f63353d3ec3fd0 + .quad 0x3b7f3bd49323a902 + .quad 0x7c21b5566b2c6e53 + .quad 0xe5ba8ff53a7852a7 + .quad 0x28bc77a5838ece00 + .quad 0x63ba78a8e25d8036 + .quad 0x63651e0094333490 + .quad 0x48d82f20288ce532 + .quad 0x3a31abfa36b57524 + + // 2^120 * 1 * B + + .quad 0x239e9624089c0a2e + .quad 0xc748c4c03afe4738 + .quad 0x17dbed2a764fa12a + .quad 0x639b93f0321c8582 + .quad 0xc08f788f3f78d289 + .quad 0xfe30a72ca1404d9f + .quad 0xf2778bfccf65cc9d + .quad 0x7ee498165acb2021 + .quad 0x7bd508e39111a1c3 + .quad 0x2b2b90d480907489 + .quad 0xe7d2aec2ae72fd19 + .quad 0x0edf493c85b602a6 + + // 2^120 * 2 * B + + .quad 0xaecc8158599b5a68 + .quad 0xea574f0febade20e + .quad 0x4fe41d7422b67f07 + .quad 0x403b92e3019d4fb4 + .quad 0x6767c4d284764113 + .quad 0xa090403ff7f5f835 + .quad 0x1c8fcffacae6bede + .quad 0x04c00c54d1dfa369 + .quad 0x4dc22f818b465cf8 + .quad 0x71a0f35a1480eff8 + .quad 0xaee8bfad04c7d657 + .quad 0x355bb12ab26176f4 + + // 2^120 * 3 * B + + .quad 0xa71e64cc7493bbf4 + .quad 0xe5bd84d9eca3b0c3 + .quad 0x0a6bc50cfa05e785 + .quad 0x0f9b8132182ec312 + .quad 0xa301dac75a8c7318 + .quad 0xed90039db3ceaa11 + .quad 0x6f077cbf3bae3f2d + .quad 0x7518eaf8e052ad8e + .quad 0xa48859c41b7f6c32 + .quad 0x0f2d60bcf4383298 + .quad 0x1815a929c9b1d1d9 + .quad 0x47c3871bbb1755c4 + + // 2^120 * 4 * B + + .quad 0x5144539771ec4f48 + .quad 0xf805b17dc98c5d6e + .quad 0xf762c11a47c3c66b + .quad 0x00b89b85764699dc + .quad 0xfbe65d50c85066b0 + .quad 0x62ecc4b0b3a299b0 + .quad 0xe53754ea441ae8e0 + .quad 0x08fea02ce8d48d5f + .quad 0x824ddd7668deead0 + .quad 0xc86445204b685d23 + .quad 0xb514cfcd5d89d665 + .quad 0x473829a74f75d537 + + // 2^120 * 5 * B + + .quad 0x82d2da754679c418 + .quad 0xe63bd7d8b2618df0 + .quad 0x355eef24ac47eb0a + .quad 0x2078684c4833c6b4 + .quad 0x23d9533aad3902c9 + .quad 0x64c2ddceef03588f + .quad 0x15257390cfe12fb4 + .quad 0x6c668b4d44e4d390 + .quad 0x3b48cf217a78820c + .quad 0xf76a0ab281273e97 + .quad 0xa96c65a78c8eed7b + .quad 0x7411a6054f8a433f + + // 2^120 * 6 * B + + .quad 0x4d659d32b99dc86d + .quad 0x044cdc75603af115 + .quad 0xb34c712cdcc2e488 + .quad 0x7c136574fb8134ff + .quad 0x579ae53d18b175b4 + .quad 0x68713159f392a102 + .quad 0x8455ecba1eef35f5 + .quad 0x1ec9a872458c398f + .quad 0xb8e6a4d400a2509b + .quad 0x9b81d7020bc882b4 + .quad 0x57e7cc9bf1957561 + .quad 0x3add88a5c7cd6460 + + // 2^120 * 7 * B + + .quad 0xab895770b635dcf2 + .quad 0x02dfef6cf66c1fbc + .quad 0x85530268beb6d187 + .quad 0x249929fccc879e74 + .quad 0x85c298d459393046 + .quad 0x8f7e35985ff659ec + .quad 0x1d2ca22af2f66e3a + .quad 0x61ba1131a406a720 + .quad 0xa3d0a0f116959029 + .quad 0x023b6b6cba7ebd89 + .quad 0x7bf15a3e26783307 + .quad 0x5620310cbbd8ece7 + + // 2^120 * 8 * B + + .quad 0x528993434934d643 + .quad 0xb9dbf806a51222f5 + .quad 0x8f6d878fc3f41c22 + .quad 0x37676a2a4d9d9730 + .quad 0x6646b5f477e285d6 + .quad 0x40e8ff676c8f6193 + .quad 0xa6ec7311abb594dd + .quad 0x7ec846f3658cec4d + .quad 0x9b5e8f3f1da22ec7 + .quad 0x130f1d776c01cd13 + .quad 0x214c8fcfa2989fb8 + .quad 0x6daaf723399b9dd5 + + // 2^124 * 1 * B + + .quad 0x591e4a5610628564 + .quad 0x2a4bb87ca8b4df34 + .quad 0xde2a2572e7a38e43 + .quad 0x3cbdabd9fee5046e + .quad 0x81aebbdd2cd13070 + .quad 0x962e4325f85a0e9e + .quad 0xde9391aacadffecb + .quad 0x53177fda52c230e6 + .quad 0xa7bc970650b9de79 + .quad 0x3d12a7fbc301b59b + .quad 0x02652e68d36ae38c + .quad 0x79d739835a6199dc + + // 2^124 * 2 * B + + .quad 0xd9354df64131c1bd + .quad 0x758094a186ec5822 + .quad 0x4464ee12e459f3c2 + .quad 0x6c11fce4cb133282 + .quad 0x21c9d9920d591737 + .quad 0x9bea41d2e9b46cd6 + .quad 0xe20e84200d89bfca + .quad 0x79d99f946eae5ff8 + .quad 0xf17b483568673205 + .quad 0x387deae83caad96c + .quad 0x61b471fd56ffe386 + .quad 0x31741195b745a599 + + // 2^124 * 3 * B + + .quad 0xe8d10190b77a360b + .quad 0x99b983209995e702 + .quad 0xbd4fdff8fa0247aa + .quad 0x2772e344e0d36a87 + .quad 0x17f8ba683b02a047 + .quad 0x50212096feefb6c8 + .quad 0x70139be21556cbe2 + .quad 0x203e44a11d98915b + .quad 0xd6863eba37b9e39f + .quad 0x105bc169723b5a23 + .quad 0x104f6459a65c0762 + .quad 0x567951295b4d38d4 + + // 2^124 * 4 * B + + .quad 0x535fd60613037524 + .quad 0xe210adf6b0fbc26a + .quad 0xac8d0a9b23e990ae + .quad 0x47204d08d72fdbf9 + .quad 0x07242eb30d4b497f + .quad 0x1ef96306b9bccc87 + .quad 0x37950934d8116f45 + .quad 0x05468d6201405b04 + .quad 0x00f565a9f93267de + .quad 0xcecfd78dc0d58e8a + .quad 0xa215e2dcf318e28e + .quad 0x4599ee919b633352 + + // 2^124 * 5 * B + + .quad 0xd3c220ca70e0e76b + .quad 0xb12bea58ea9f3094 + .quad 0x294ddec8c3271282 + .quad 0x0c3539e1a1d1d028 + .quad 0xac746d6b861ae579 + .quad 0x31ab0650f6aea9dc + .quad 0x241d661140256d4c + .quad 0x2f485e853d21a5de + .quad 0x329744839c0833f3 + .quad 0x6fe6257fd2abc484 + .quad 0x5327d1814b358817 + .quad 0x65712585893fe9bc + + // 2^124 * 6 * B + + .quad 0x9c102fb732a61161 + .quad 0xe48e10dd34d520a8 + .quad 0x365c63546f9a9176 + .quad 0x32f6fe4c046f6006 + .quad 0x81c29f1bd708ee3f + .quad 0xddcb5a05ae6407d0 + .quad 0x97aec1d7d2a3eba7 + .quad 0x1590521a91d50831 + .quad 0x40a3a11ec7910acc + .quad 0x9013dff8f16d27ae + .quad 0x1a9720d8abb195d4 + .quad 0x1bb9fe452ea98463 + + // 2^124 * 7 * B + + .quad 0xe9d1d950b3d54f9e + .quad 0x2d5f9cbee00d33c1 + .quad 0x51c2c656a04fc6ac + .quad 0x65c091ee3c1cbcc9 + .quad 0xcf5e6c95cc36747c + .quad 0x294201536b0bc30d + .quad 0x453ac67cee797af0 + .quad 0x5eae6ab32a8bb3c9 + .quad 0x7083661114f118ea + .quad 0x2b37b87b94349cad + .quad 0x7273f51cb4e99f40 + .quad 0x78a2a95823d75698 + + // 2^124 * 8 * B + + .quad 0xa2b072e95c8c2ace + .quad 0x69cffc96651e9c4b + .quad 0x44328ef842e7b42b + .quad 0x5dd996c122aadeb3 + .quad 0xb4f23c425ef83207 + .quad 0xabf894d3c9a934b5 + .quad 0xd0708c1339fd87f7 + .quad 0x1876789117166130 + .quad 0x925b5ef0670c507c + .quad 0x819bc842b93c33bf + .quad 0x10792e9a70dd003f + .quad 0x59ad4b7a6e28dc74 + + // 2^128 * 1 * B + + .quad 0x5f3a7562eb3dbe47 + .quad 0xf7ea38548ebda0b8 + .quad 0x00c3e53145747299 + .quad 0x1304e9e71627d551 + .quad 0x583b04bfacad8ea2 + .quad 0x29b743e8148be884 + .quad 0x2b1e583b0810c5db + .quad 0x2b5449e58eb3bbaa + .quad 0x789814d26adc9cfe + .quad 0x3c1bab3f8b48dd0b + .quad 0xda0fe1fff979c60a + .quad 0x4468de2d7c2dd693 + + // 2^128 * 2 * B + + .quad 0x51bb355e9419469e + .quad 0x33e6dc4c23ddc754 + .quad 0x93a5b6d6447f9962 + .quad 0x6cce7c6ffb44bd63 + .quad 0x4b9ad8c6f86307ce + .quad 0x21113531435d0c28 + .quad 0xd4a866c5657a772c + .quad 0x5da6427e63247352 + .quad 0x1a94c688deac22ca + .quad 0xb9066ef7bbae1ff8 + .quad 0x88ad8c388d59580f + .quad 0x58f29abfe79f2ca8 + + // 2^128 * 3 * B + + .quad 0xe90ecfab8de73e68 + .quad 0x54036f9f377e76a5 + .quad 0xf0495b0bbe015982 + .quad 0x577629c4a7f41e36 + .quad 0x4b5a64bf710ecdf6 + .quad 0xb14ce538462c293c + .quad 0x3643d056d50b3ab9 + .quad 0x6af93724185b4870 + .quad 0x3220024509c6a888 + .quad 0xd2e036134b558973 + .quad 0x83e236233c33289f + .quad 0x701f25bb0caec18f + + // 2^128 * 4 * B + + .quad 0xc3a8b0f8e4616ced + .quad 0xf700660e9e25a87d + .quad 0x61e3061ff4bca59c + .quad 0x2e0c92bfbdc40be9 + .quad 0x9d18f6d97cbec113 + .quad 0x844a06e674bfdbe4 + .quad 0x20f5b522ac4e60d6 + .quad 0x720a5bc050955e51 + .quad 0x0c3f09439b805a35 + .quad 0xe84e8b376242abfc + .quad 0x691417f35c229346 + .quad 0x0e9b9cbb144ef0ec + + // 2^128 * 5 * B + + .quad 0xfbbad48ffb5720ad + .quad 0xee81916bdbf90d0e + .quad 0xd4813152635543bf + .quad 0x221104eb3f337bd8 + .quad 0x8dee9bd55db1beee + .quad 0xc9c3ab370a723fb9 + .quad 0x44a8f1bf1c68d791 + .quad 0x366d44191cfd3cde + .quad 0x9e3c1743f2bc8c14 + .quad 0x2eda26fcb5856c3b + .quad 0xccb82f0e68a7fb97 + .quad 0x4167a4e6bc593244 + + // 2^128 * 6 * B + + .quad 0x643b9d2876f62700 + .quad 0x5d1d9d400e7668eb + .quad 0x1b4b430321fc0684 + .quad 0x7938bb7e2255246a + .quad 0xc2be2665f8ce8fee + .quad 0xe967ff14e880d62c + .quad 0xf12e6e7e2f364eee + .quad 0x34b33370cb7ed2f6 + .quad 0xcdc591ee8681d6cc + .quad 0xce02109ced85a753 + .quad 0xed7485c158808883 + .quad 0x1176fc6e2dfe65e4 + + // 2^128 * 7 * B + + .quad 0xb4af6cd05b9c619b + .quad 0x2ddfc9f4b2a58480 + .quad 0x3d4fa502ebe94dc4 + .quad 0x08fc3a4c677d5f34 + .quad 0xdb90e28949770eb8 + .quad 0x98fbcc2aacf440a3 + .quad 0x21354ffeded7879b + .quad 0x1f6a3e54f26906b6 + .quad 0x60a4c199d30734ea + .quad 0x40c085b631165cd6 + .quad 0xe2333e23f7598295 + .quad 0x4f2fad0116b900d1 + + // 2^128 * 8 * B + + .quad 0x44beb24194ae4e54 + .quad 0x5f541c511857ef6c + .quad 0xa61e6b2d368d0498 + .quad 0x445484a4972ef7ab + .quad 0x962cd91db73bb638 + .quad 0xe60577aafc129c08 + .quad 0x6f619b39f3b61689 + .quad 0x3451995f2944ee81 + .quad 0x9152fcd09fea7d7c + .quad 0x4a816c94b0935cf6 + .quad 0x258e9aaa47285c40 + .quad 0x10b89ca6042893b7 + + // 2^132 * 1 * B + + .quad 0x9b2a426e3b646025 + .quad 0x32127190385ce4cf + .quad 0xa25cffc2dd6dea45 + .quad 0x06409010bea8de75 + .quad 0xd67cded679d34aa0 + .quad 0xcc0b9ec0cc4db39f + .quad 0xa535a456e35d190f + .quad 0x2e05d9eaf61f6fef + .quad 0xc447901ad61beb59 + .quad 0x661f19bce5dc880a + .quad 0x24685482b7ca6827 + .quad 0x293c778cefe07f26 + + // 2^132 * 2 * B + + .quad 0x86809e7007069096 + .quad 0xaad75b15e4e50189 + .quad 0x07f35715a21a0147 + .quad 0x0487f3f112815d5e + .quad 0x16c795d6a11ff200 + .quad 0xcb70d0e2b15815c9 + .quad 0x89f293209b5395b5 + .quad 0x50b8c2d031e47b4f + .quad 0x48350c08068a4962 + .quad 0x6ffdd05351092c9a + .quad 0x17af4f4aaf6fc8dd + .quad 0x4b0553b53cdba58b + + // 2^132 * 3 * B + + .quad 0x9c65fcbe1b32ff79 + .quad 0xeb75ea9f03b50f9b + .quad 0xfced2a6c6c07e606 + .quad 0x35106cd551717908 + .quad 0xbf05211b27c152d4 + .quad 0x5ec26849bd1af639 + .quad 0x5e0b2caa8e6fab98 + .quad 0x054c8bdd50bd0840 + .quad 0x38a0b12f1dcf073d + .quad 0x4b60a8a3b7f6a276 + .quad 0xfed5ac25d3404f9a + .quad 0x72e82d5e5505c229 + + // 2^132 * 4 * B + + .quad 0x6b0b697ff0d844c8 + .quad 0xbb12f85cd979cb49 + .quad 0xd2a541c6c1da0f1f + .quad 0x7b7c242958ce7211 + .quad 0x00d9cdfd69771d02 + .quad 0x410276cd6cfbf17e + .quad 0x4c45306c1cb12ec7 + .quad 0x2857bf1627500861 + .quad 0x9f21903f0101689e + .quad 0xd779dfd3bf861005 + .quad 0xa122ee5f3deb0f1b + .quad 0x510df84b485a00d4 + + // 2^132 * 5 * B + + .quad 0xa54133bb9277a1fa + .quad 0x74ec3b6263991237 + .quad 0x1a3c54dc35d2f15a + .quad 0x2d347144e482ba3a + .quad 0x24b3c887c70ac15e + .quad 0xb0f3a557fb81b732 + .quad 0x9b2cde2fe578cc1b + .quad 0x4cf7ed0703b54f8e + .quad 0x6bd47c6598fbee0f + .quad 0x9e4733e2ab55be2d + .quad 0x1093f624127610c5 + .quad 0x4e05e26ad0a1eaa4 + + // 2^132 * 6 * B + + .quad 0xda9b6b624b531f20 + .quad 0x429a760e77509abb + .quad 0xdbe9f522e823cb80 + .quad 0x618f1856880c8f82 + .quad 0x1833c773e18fe6c0 + .quad 0xe3c4711ad3c87265 + .quad 0x3bfd3c4f0116b283 + .quad 0x1955875eb4cd4db8 + .quad 0x6da6de8f0e399799 + .quad 0x7ad61aa440fda178 + .quad 0xb32cd8105e3563dd + .quad 0x15f6beae2ae340ae + + // 2^132 * 7 * B + + .quad 0x862bcb0c31ec3a62 + .quad 0x810e2b451138f3c2 + .quad 0x788ec4b839dac2a4 + .quad 0x28f76867ae2a9281 + .quad 0xba9a0f7b9245e215 + .quad 0xf368612dd98c0dbb + .quad 0x2e84e4cbf220b020 + .quad 0x6ba92fe962d90eda + .quad 0x3e4df9655884e2aa + .quad 0xbd62fbdbdbd465a5 + .quad 0xd7596caa0de9e524 + .quad 0x6e8042ccb2b1b3d7 + + // 2^132 * 8 * B + + .quad 0xf10d3c29ce28ca6e + .quad 0xbad34540fcb6093d + .quad 0xe7426ed7a2ea2d3f + .quad 0x08af9d4e4ff298b9 + .quad 0x1530653616521f7e + .quad 0x660d06b896203dba + .quad 0x2d3989bc545f0879 + .quad 0x4b5303af78ebd7b0 + .quad 0x72f8a6c3bebcbde8 + .quad 0x4f0fca4adc3a8e89 + .quad 0x6fa9d4e8c7bfdf7a + .quad 0x0dcf2d679b624eb7 + + // 2^136 * 1 * B + + .quad 0x3d5947499718289c + .quad 0x12ebf8c524533f26 + .quad 0x0262bfcb14c3ef15 + .quad 0x20b878d577b7518e + .quad 0x753941be5a45f06e + .quad 0xd07caeed6d9c5f65 + .quad 0x11776b9c72ff51b6 + .quad 0x17d2d1d9ef0d4da9 + .quad 0x27f2af18073f3e6a + .quad 0xfd3fe519d7521069 + .quad 0x22e3b72c3ca60022 + .quad 0x72214f63cc65c6a7 + + // 2^136 * 2 * B + + .quad 0xb4e37f405307a693 + .quad 0xaba714d72f336795 + .quad 0xd6fbd0a773761099 + .quad 0x5fdf48c58171cbc9 + .quad 0x1d9db7b9f43b29c9 + .quad 0xd605824a4f518f75 + .quad 0xf2c072bd312f9dc4 + .quad 0x1f24ac855a1545b0 + .quad 0x24d608328e9505aa + .quad 0x4748c1d10c1420ee + .quad 0xc7ffe45c06fb25a2 + .quad 0x00ba739e2ae395e6 + + // 2^136 * 3 * B + + .quad 0x592e98de5c8790d6 + .quad 0xe5bfb7d345c2a2df + .quad 0x115a3b60f9b49922 + .quad 0x03283a3e67ad78f3 + .quad 0xae4426f5ea88bb26 + .quad 0x360679d984973bfb + .quad 0x5c9f030c26694e50 + .quad 0x72297de7d518d226 + .quad 0x48241dc7be0cb939 + .quad 0x32f19b4d8b633080 + .quad 0xd3dfc90d02289308 + .quad 0x05e1296846271945 + + // 2^136 * 4 * B + + .quad 0xba82eeb32d9c495a + .quad 0xceefc8fcf12bb97c + .quad 0xb02dabae93b5d1e0 + .quad 0x39c00c9c13698d9b + .quad 0xadbfbbc8242c4550 + .quad 0xbcc80cecd03081d9 + .quad 0x843566a6f5c8df92 + .quad 0x78cf25d38258ce4c + .quad 0x15ae6b8e31489d68 + .quad 0xaa851cab9c2bf087 + .quad 0xc9a75a97f04efa05 + .quad 0x006b52076b3ff832 + + // 2^136 * 5 * B + + .quad 0x29e0cfe19d95781c + .quad 0xb681df18966310e2 + .quad 0x57df39d370516b39 + .quad 0x4d57e3443bc76122 + .quad 0xf5cb7e16b9ce082d + .quad 0x3407f14c417abc29 + .quad 0xd4b36bce2bf4a7ab + .quad 0x7de2e9561a9f75ce + .quad 0xde70d4f4b6a55ecb + .quad 0x4801527f5d85db99 + .quad 0xdbc9c440d3ee9a81 + .quad 0x6b2a90af1a6029ed + + // 2^136 * 6 * B + + .quad 0x6923f4fc9ae61e97 + .quad 0x5735281de03f5fd1 + .quad 0xa764ae43e6edd12d + .quad 0x5fd8f4e9d12d3e4a + .quad 0x77ebf3245bb2d80a + .quad 0xd8301b472fb9079b + .quad 0xc647e6f24cee7333 + .quad 0x465812c8276c2109 + .quad 0x4d43beb22a1062d9 + .quad 0x7065fb753831dc16 + .quad 0x180d4a7bde2968d7 + .quad 0x05b32c2b1cb16790 + + // 2^136 * 7 * B + + .quad 0xc8c05eccd24da8fd + .quad 0xa1cf1aac05dfef83 + .quad 0xdbbeeff27df9cd61 + .quad 0x3b5556a37b471e99 + .quad 0xf7fca42c7ad58195 + .quad 0x3214286e4333f3cc + .quad 0xb6c29d0d340b979d + .quad 0x31771a48567307e1 + .quad 0x32b0c524e14dd482 + .quad 0xedb351541a2ba4b6 + .quad 0xa3d16048282b5af3 + .quad 0x4fc079d27a7336eb + + // 2^136 * 8 * B + + .quad 0x51c938b089bf2f7f + .quad 0x2497bd6502dfe9a7 + .quad 0xffffc09c7880e453 + .quad 0x124567cecaf98e92 + .quad 0xdc348b440c86c50d + .quad 0x1337cbc9cc94e651 + .quad 0x6422f74d643e3cb9 + .quad 0x241170c2bae3cd08 + .quad 0x3ff9ab860ac473b4 + .quad 0xf0911dee0113e435 + .quad 0x4ae75060ebc6c4af + .quad 0x3f8612966c87000d + + // 2^140 * 1 * B + + .quad 0x0c9c5303f7957be4 + .quad 0xa3c31a20e085c145 + .quad 0xb0721d71d0850050 + .quad 0x0aba390eab0bf2da + .quad 0x529fdffe638c7bf3 + .quad 0xdf2b9e60388b4995 + .quad 0xe027b34f1bad0249 + .quad 0x7bc92fc9b9fa74ed + .quad 0x9f97ef2e801ad9f9 + .quad 0x83697d5479afda3a + .quad 0xe906b3ffbd596b50 + .quad 0x02672b37dd3fb8e0 + + // 2^140 * 2 * B + + .quad 0x48b2ca8b260885e4 + .quad 0xa4286bec82b34c1c + .quad 0x937e1a2617f58f74 + .quad 0x741d1fcbab2ca2a5 + .quad 0xee9ba729398ca7f5 + .quad 0xeb9ca6257a4849db + .quad 0x29eb29ce7ec544e1 + .quad 0x232ca21ef736e2c8 + .quad 0xbf61423d253fcb17 + .quad 0x08803ceafa39eb14 + .quad 0xf18602df9851c7af + .quad 0x0400f3a049e3414b + + // 2^140 * 3 * B + + .quad 0xabce0476ba61c55b + .quad 0x36a3d6d7c4d39716 + .quad 0x6eb259d5e8d82d09 + .quad 0x0c9176e984d756fb + .quad 0x2efba412a06e7b06 + .quad 0x146785452c8d2560 + .quad 0xdf9713ebd67a91c7 + .quad 0x32830ac7157eadf3 + .quad 0x0e782a7ab73769e8 + .quad 0x04a05d7875b18e2c + .quad 0x29525226ebcceae1 + .quad 0x0d794f8383eba820 + + // 2^140 * 4 * B + + .quad 0xff35f5cb9e1516f4 + .quad 0xee805bcf648aae45 + .quad 0xf0d73c2bb93a9ef3 + .quad 0x097b0bf22092a6c2 + .quad 0x7be44ce7a7a2e1ac + .quad 0x411fd93efad1b8b7 + .quad 0x1734a1d70d5f7c9b + .quad 0x0d6592233127db16 + .quad 0xc48bab1521a9d733 + .quad 0xa6c2eaead61abb25 + .quad 0x625c6c1cc6cb4305 + .quad 0x7fc90fea93eb3a67 + + // 2^140 * 5 * B + + .quad 0x0408f1fe1f5c5926 + .quad 0x1a8f2f5e3b258bf4 + .quad 0x40a951a2fdc71669 + .quad 0x6598ee93c98b577e + .quad 0xc527deb59c7cb23d + .quad 0x955391695328404e + .quad 0xd64392817ccf2c7a + .quad 0x6ce97dabf7d8fa11 + .quad 0x25b5a8e50ef7c48f + .quad 0xeb6034116f2ce532 + .quad 0xc5e75173e53de537 + .quad 0x73119fa08c12bb03 + + // 2^140 * 6 * B + + .quad 0xed30129453f1a4cb + .quad 0xbce621c9c8f53787 + .quad 0xfacb2b1338bee7b9 + .quad 0x3025798a9ea8428c + .quad 0x7845b94d21f4774d + .quad 0xbf62f16c7897b727 + .quad 0x671857c03c56522b + .quad 0x3cd6a85295621212 + .quad 0x3fecde923aeca999 + .quad 0xbdaa5b0062e8c12f + .quad 0x67b99dfc96988ade + .quad 0x3f52c02852661036 + + // 2^140 * 7 * B + + .quad 0xffeaa48e2a1351c6 + .quad 0x28624754fa7f53d7 + .quad 0x0b5ba9e57582ddf1 + .quad 0x60c0104ba696ac59 + .quad 0x9258bf99eec416c6 + .quad 0xac8a5017a9d2f671 + .quad 0x629549ab16dea4ab + .quad 0x05d0e85c99091569 + .quad 0x051de020de9cbe97 + .quad 0xfa07fc56b50bcf74 + .quad 0x378cec9f0f11df65 + .quad 0x36853c69ab96de4d + + // 2^140 * 8 * B + + .quad 0x36d9b8de78f39b2d + .quad 0x7f42ed71a847b9ec + .quad 0x241cd1d679bd3fde + .quad 0x6a704fec92fbce6b + .quad 0x4433c0b0fac5e7be + .quad 0x724bae854c08dcbe + .quad 0xf1f24cc446978f9b + .quad 0x4a0aff6d62825fc8 + .quad 0xe917fb9e61095301 + .quad 0xc102df9402a092f8 + .quad 0xbf09e2f5fa66190b + .quad 0x681109bee0dcfe37 + + // 2^144 * 1 * B + + .quad 0x559a0cc9782a0dde + .quad 0x551dcdb2ea718385 + .quad 0x7f62865b31ef238c + .quad 0x504aa7767973613d + .quad 0x9c18fcfa36048d13 + .quad 0x29159db373899ddd + .quad 0xdc9f350b9f92d0aa + .quad 0x26f57eee878a19d4 + .quad 0x0cab2cd55687efb1 + .quad 0x5180d162247af17b + .quad 0x85c15a344f5a2467 + .quad 0x4041943d9dba3069 + + // 2^144 * 2 * B + + .quad 0xc3c0eeba43ebcc96 + .quad 0x8d749c9c26ea9caf + .quad 0xd9fa95ee1c77ccc6 + .quad 0x1420a1d97684340f + .quad 0x4b217743a26caadd + .quad 0x47a6b424648ab7ce + .quad 0xcb1d4f7a03fbc9e3 + .quad 0x12d931429800d019 + .quad 0x00c67799d337594f + .quad 0x5e3c5140b23aa47b + .quad 0x44182854e35ff395 + .quad 0x1b4f92314359a012 + + // 2^144 * 3 * B + + .quad 0x3e5c109d89150951 + .quad 0x39cefa912de9696a + .quad 0x20eae43f975f3020 + .quad 0x239b572a7f132dae + .quad 0x33cf3030a49866b1 + .quad 0x251f73d2215f4859 + .quad 0xab82aa4051def4f6 + .quad 0x5ff191d56f9a23f6 + .quad 0x819ed433ac2d9068 + .quad 0x2883ab795fc98523 + .quad 0xef4572805593eb3d + .quad 0x020c526a758f36cb + + // 2^144 * 4 * B + + .quad 0x779834f89ed8dbbc + .quad 0xc8f2aaf9dc7ca46c + .quad 0xa9524cdca3e1b074 + .quad 0x02aacc4615313877 + .quad 0xe931ef59f042cc89 + .quad 0x2c589c9d8e124bb6 + .quad 0xadc8e18aaec75997 + .quad 0x452cfe0a5602c50c + .quad 0x86a0f7a0647877df + .quad 0xbbc464270e607c9f + .quad 0xab17ea25f1fb11c9 + .quad 0x4cfb7d7b304b877b + + // 2^144 * 5 * B + + .quad 0x72b43d6cb89b75fe + .quad 0x54c694d99c6adc80 + .quad 0xb8c3aa373ee34c9f + .quad 0x14b4622b39075364 + .quad 0xe28699c29789ef12 + .quad 0x2b6ecd71df57190d + .quad 0xc343c857ecc970d0 + .quad 0x5b1d4cbc434d3ac5 + .quad 0xb6fb2615cc0a9f26 + .quad 0x3a4f0e2bb88dcce5 + .quad 0x1301498b3369a705 + .quad 0x2f98f71258592dd1 + + // 2^144 * 6 * B + + .quad 0x0c94a74cb50f9e56 + .quad 0x5b1ff4a98e8e1320 + .quad 0x9a2acc2182300f67 + .quad 0x3a6ae249d806aaf9 + .quad 0x2e12ae444f54a701 + .quad 0xfcfe3ef0a9cbd7de + .quad 0xcebf890d75835de0 + .quad 0x1d8062e9e7614554 + .quad 0x657ada85a9907c5a + .quad 0x1a0ea8b591b90f62 + .quad 0x8d0e1dfbdf34b4e9 + .quad 0x298b8ce8aef25ff3 + + // 2^144 * 7 * B + + .quad 0x2a927953eff70cb2 + .quad 0x4b89c92a79157076 + .quad 0x9418457a30a7cf6a + .quad 0x34b8a8404d5ce485 + .quad 0x837a72ea0a2165de + .quad 0x3fab07b40bcf79f6 + .quad 0x521636c77738ae70 + .quad 0x6ba6271803a7d7dc + .quad 0xc26eecb583693335 + .quad 0xd5a813df63b5fefd + .quad 0xa293aa9aa4b22573 + .quad 0x71d62bdd465e1c6a + + // 2^144 * 8 * B + + .quad 0x6533cc28d378df80 + .quad 0xf6db43790a0fa4b4 + .quad 0xe3645ff9f701da5a + .quad 0x74d5f317f3172ba4 + .quad 0xcd2db5dab1f75ef5 + .quad 0xd77f95cf16b065f5 + .quad 0x14571fea3f49f085 + .quad 0x1c333621262b2b3d + .quad 0xa86fe55467d9ca81 + .quad 0x398b7c752b298c37 + .quad 0xda6d0892e3ac623b + .quad 0x4aebcc4547e9d98c + + // 2^148 * 1 * B + + .quad 0x53175a7205d21a77 + .quad 0xb0c04422d3b934d4 + .quad 0xadd9f24bdd5deadc + .quad 0x074f46e69f10ff8c + .quad 0x0de9b204a059a445 + .quad 0xe15cb4aa4b17ad0f + .quad 0xe1bbec521f79c557 + .quad 0x2633f1b9d071081b + .quad 0xc1fb4177018b9910 + .quad 0xa6ea20dc6c0fe140 + .quad 0xd661f3e74354c6ff + .quad 0x5ecb72e6f1a3407a + + // 2^148 * 2 * B + + .quad 0xa515a31b2259fb4e + .quad 0x0960f3972bcac52f + .quad 0xedb52fec8d3454cb + .quad 0x382e2720c476c019 + .quad 0xfeeae106e8e86997 + .quad 0x9863337f98d09383 + .quad 0x9470480eaa06ebef + .quad 0x038b6898d4c5c2d0 + .quad 0xf391c51d8ace50a6 + .quad 0x3142d0b9ae2d2948 + .quad 0xdb4d5a1a7f24ca80 + .quad 0x21aeba8b59250ea8 + + // 2^148 * 3 * B + + .quad 0x24f13b34cf405530 + .quad 0x3c44ea4a43088af7 + .quad 0x5dd5c5170006a482 + .quad 0x118eb8f8890b086d + .quad 0x53853600f0087f23 + .quad 0x4c461879da7d5784 + .quad 0x6af303deb41f6860 + .quad 0x0a3c16c5c27c18ed + .quad 0x17e49c17cc947f3d + .quad 0xccc6eda6aac1d27b + .quad 0xdf6092ceb0f08e56 + .quad 0x4909b3e22c67c36b + + // 2^148 * 4 * B + + .quad 0x9c9c85ea63fe2e89 + .quad 0xbe1baf910e9412ec + .quad 0x8f7baa8a86fbfe7b + .quad 0x0fb17f9fef968b6c + .quad 0x59a16676706ff64e + .quad 0x10b953dd0d86a53d + .quad 0x5848e1e6ce5c0b96 + .quad 0x2d8b78e712780c68 + .quad 0x79d5c62eafc3902b + .quad 0x773a215289e80728 + .quad 0xc38ae640e10120b9 + .quad 0x09ae23717b2b1a6d + + // 2^148 * 5 * B + + .quad 0xbb6a192a4e4d083c + .quad 0x34ace0630029e192 + .quad 0x98245a59aafabaeb + .quad 0x6d9c8a9ada97faac + .quad 0x10ab8fa1ad32b1d0 + .quad 0xe9aced1be2778b24 + .quad 0xa8856bc0373de90f + .quad 0x66f35ddddda53996 + .quad 0xd27d9afb24997323 + .quad 0x1bb7e07ef6f01d2e + .quad 0x2ba7472df52ecc7f + .quad 0x03019b4f646f9dc8 + + // 2^148 * 6 * B + + .quad 0x04a186b5565345cd + .quad 0xeee76610bcc4116a + .quad 0x689c73b478fb2a45 + .quad 0x387dcbff65697512 + .quad 0xaf09b214e6b3dc6b + .quad 0x3f7573b5ad7d2f65 + .quad 0xd019d988100a23b0 + .quad 0x392b63a58b5c35f7 + .quad 0x4093addc9c07c205 + .quad 0xc565be15f532c37e + .quad 0x63dbecfd1583402a + .quad 0x61722b4aef2e032e + + // 2^148 * 7 * B + + .quad 0x0012aafeecbd47af + .quad 0x55a266fb1cd46309 + .quad 0xf203eb680967c72c + .quad 0x39633944ca3c1429 + .quad 0xd6b07a5581cb0e3c + .quad 0x290ff006d9444969 + .quad 0x08680b6a16dcda1f + .quad 0x5568d2b75a06de59 + .quad 0x8d0cb88c1b37cfe1 + .quad 0x05b6a5a3053818f3 + .quad 0xf2e9bc04b787d959 + .quad 0x6beba1249add7f64 + + // 2^148 * 8 * B + + .quad 0x1d06005ca5b1b143 + .quad 0x6d4c6bb87fd1cda2 + .quad 0x6ef5967653fcffe7 + .quad 0x097c29e8c1ce1ea5 + .quad 0x5c3cecb943f5a53b + .quad 0x9cc9a61d06c08df2 + .quad 0xcfba639a85895447 + .quad 0x5a845ae80df09fd5 + .quad 0x4ce97dbe5deb94ca + .quad 0x38d0a4388c709c48 + .quad 0xc43eced4a169d097 + .quad 0x0a1249fff7e587c3 + + // 2^152 * 1 * B + + .quad 0x12f0071b276d01c9 + .quad 0xe7b8bac586c48c70 + .quad 0x5308129b71d6fba9 + .quad 0x5d88fbf95a3db792 + .quad 0x0b408d9e7354b610 + .quad 0x806b32535ba85b6e + .quad 0xdbe63a034a58a207 + .quad 0x173bd9ddc9a1df2c + .quad 0x2b500f1efe5872df + .quad 0x58d6582ed43918c1 + .quad 0xe6ed278ec9673ae0 + .quad 0x06e1cd13b19ea319 + + // 2^152 * 2 * B + + .quad 0x40d0ad516f166f23 + .quad 0x118e32931fab6abe + .quad 0x3fe35e14a04d088e + .quad 0x3080603526e16266 + .quad 0x472baf629e5b0353 + .quad 0x3baa0b90278d0447 + .quad 0x0c785f469643bf27 + .quad 0x7f3a6a1a8d837b13 + .quad 0xf7e644395d3d800b + .quad 0x95a8d555c901edf6 + .quad 0x68cd7830592c6339 + .quad 0x30d0fded2e51307e + + // 2^152 * 3 * B + + .quad 0xe0594d1af21233b3 + .quad 0x1bdbe78ef0cc4d9c + .quad 0x6965187f8f499a77 + .quad 0x0a9214202c099868 + .quad 0x9cb4971e68b84750 + .quad 0xa09572296664bbcf + .quad 0x5c8de72672fa412b + .quad 0x4615084351c589d9 + .quad 0xbc9019c0aeb9a02e + .quad 0x55c7110d16034cae + .quad 0x0e6df501659932ec + .quad 0x3bca0d2895ca5dfe + + // 2^152 * 4 * B + + .quad 0x40f031bc3c5d62a4 + .quad 0x19fc8b3ecff07a60 + .quad 0x98183da2130fb545 + .quad 0x5631deddae8f13cd + .quad 0x9c688eb69ecc01bf + .quad 0xf0bc83ada644896f + .quad 0xca2d955f5f7a9fe2 + .quad 0x4ea8b4038df28241 + .quad 0x2aed460af1cad202 + .quad 0x46305305a48cee83 + .quad 0x9121774549f11a5f + .quad 0x24ce0930542ca463 + + // 2^152 * 5 * B + + .quad 0x1fe890f5fd06c106 + .quad 0xb5c468355d8810f2 + .quad 0x827808fe6e8caf3e + .quad 0x41d4e3c28a06d74b + .quad 0x3fcfa155fdf30b85 + .quad 0xd2f7168e36372ea4 + .quad 0xb2e064de6492f844 + .quad 0x549928a7324f4280 + .quad 0xf26e32a763ee1a2e + .quad 0xae91e4b7d25ffdea + .quad 0xbc3bd33bd17f4d69 + .quad 0x491b66dec0dcff6a + + // 2^152 * 6 * B + + .quad 0x98f5b13dc7ea32a7 + .quad 0xe3d5f8cc7e16db98 + .quad 0xac0abf52cbf8d947 + .quad 0x08f338d0c85ee4ac + .quad 0x75f04a8ed0da64a1 + .quad 0xed222caf67e2284b + .quad 0x8234a3791f7b7ba4 + .quad 0x4cf6b8b0b7018b67 + .quad 0xc383a821991a73bd + .quad 0xab27bc01df320c7a + .quad 0xc13d331b84777063 + .quad 0x530d4a82eb078a99 + + // 2^152 * 7 * B + + .quad 0x004c3630e1f94825 + .quad 0x7e2d78268cab535a + .quad 0xc7482323cc84ff8b + .quad 0x65ea753f101770b9 + .quad 0x6d6973456c9abf9e + .quad 0x257fb2fc4900a880 + .quad 0x2bacf412c8cfb850 + .quad 0x0db3e7e00cbfbd5b + .quad 0x3d66fc3ee2096363 + .quad 0x81d62c7f61b5cb6b + .quad 0x0fbe044213443b1a + .quad 0x02a4ec1921e1a1db + + // 2^152 * 8 * B + + .quad 0x5ce6259a3b24b8a2 + .quad 0xb8577acc45afa0b8 + .quad 0xcccbe6e88ba07037 + .quad 0x3d143c51127809bf + .quad 0xf5c86162f1cf795f + .quad 0x118c861926ee57f2 + .quad 0x172124851c063578 + .quad 0x36d12b5dec067fcf + .quad 0x126d279179154557 + .quad 0xd5e48f5cfc783a0a + .quad 0x36bdb6e8df179bac + .quad 0x2ef517885ba82859 + + // 2^156 * 1 * B + + .quad 0x88bd438cd11e0d4a + .quad 0x30cb610d43ccf308 + .quad 0xe09a0e3791937bcc + .quad 0x4559135b25b1720c + .quad 0x1ea436837c6da1e9 + .quad 0xf9c189af1fb9bdbe + .quad 0x303001fcce5dd155 + .quad 0x28a7c99ebc57be52 + .quad 0xb8fd9399e8d19e9d + .quad 0x908191cb962423ff + .quad 0xb2b948d747c742a3 + .quad 0x37f33226d7fb44c4 + + // 2^156 * 2 * B + + .quad 0x0dae8767b55f6e08 + .quad 0x4a43b3b35b203a02 + .quad 0xe3725a6e80af8c79 + .quad 0x0f7a7fd1705fa7a3 + .quad 0x33912553c821b11d + .quad 0x66ed42c241e301df + .quad 0x066fcc11104222fd + .quad 0x307a3b41c192168f + .quad 0x8eeb5d076eb55ce0 + .quad 0x2fc536bfaa0d925a + .quad 0xbe81830fdcb6c6e8 + .quad 0x556c7045827baf52 + + // 2^156 * 3 * B + + .quad 0x8e2b517302e9d8b7 + .quad 0xe3e52269248714e8 + .quad 0xbd4fbd774ca960b5 + .quad 0x6f4b4199c5ecada9 + .quad 0xb94b90022bf44406 + .quad 0xabd4237eff90b534 + .quad 0x7600a960faf86d3a + .quad 0x2f45abdac2322ee3 + .quad 0x61af4912c8ef8a6a + .quad 0xe58fa4fe43fb6e5e + .quad 0xb5afcc5d6fd427cf + .quad 0x6a5393281e1e11eb + + // 2^156 * 4 * B + + .quad 0xf3da5139a5d1ee89 + .quad 0x8145457cff936988 + .quad 0x3f622fed00e188c4 + .quad 0x0f513815db8b5a3d + .quad 0x0fff04fe149443cf + .quad 0x53cac6d9865cddd7 + .quad 0x31385b03531ed1b7 + .quad 0x5846a27cacd1039d + .quad 0x4ff5cdac1eb08717 + .quad 0x67e8b29590f2e9bc + .quad 0x44093b5e237afa99 + .quad 0x0d414bed8708b8b2 + + // 2^156 * 5 * B + + .quad 0xcfb68265fd0e75f6 + .quad 0xe45b3e28bb90e707 + .quad 0x7242a8de9ff92c7a + .quad 0x685b3201933202dd + .quad 0x81886a92294ac9e8 + .quad 0x23162b45d55547be + .quad 0x94cfbc4403715983 + .quad 0x50eb8fdb134bc401 + .quad 0xc0b73ec6d6b330cd + .quad 0x84e44807132faff1 + .quad 0x732b7352c4a5dee1 + .quad 0x5d7c7cf1aa7cd2d2 + + // 2^156 * 6 * B + + .quad 0xaf3b46bf7a4aafa2 + .quad 0xb78705ec4d40d411 + .quad 0x114f0c6aca7c15e3 + .quad 0x3f364faaa9489d4d + .quad 0x33d1013e9b73a562 + .quad 0x925cef5748ec26e1 + .quad 0xa7fce614dd468058 + .quad 0x78b0fad41e9aa438 + .quad 0xbf56a431ed05b488 + .quad 0xa533e66c9c495c7e + .quad 0xe8652baf87f3651a + .quad 0x0241800059d66c33 + + // 2^156 * 7 * B + + .quad 0xceb077fea37a5be4 + .quad 0xdb642f02e5a5eeb7 + .quad 0xc2e6d0c5471270b8 + .quad 0x4771b65538e4529c + .quad 0x28350c7dcf38ea01 + .quad 0x7c6cdbc0b2917ab6 + .quad 0xace7cfbe857082f7 + .quad 0x4d2845aba2d9a1e0 + .quad 0xbb537fe0447070de + .quad 0xcba744436dd557df + .quad 0xd3b5a3473600dbcb + .quad 0x4aeabbe6f9ffd7f8 + + // 2^156 * 8 * B + + .quad 0x4630119e40d8f78c + .quad 0xa01a9bc53c710e11 + .quad 0x486d2b258910dd79 + .quad 0x1e6c47b3db0324e5 + .quad 0x6a2134bcc4a9c8f2 + .quad 0xfbf8fd1c8ace2e37 + .quad 0x000ae3049911a0ba + .quad 0x046e3a616bc89b9e + .quad 0x14e65442f03906be + .quad 0x4a019d54e362be2a + .quad 0x68ccdfec8dc230c7 + .quad 0x7cfb7e3faf6b861c + + // 2^160 * 1 * B + + .quad 0x4637974e8c58aedc + .quad 0xb9ef22fbabf041a4 + .quad 0xe185d956e980718a + .quad 0x2f1b78fab143a8a6 + .quad 0x96eebffb305b2f51 + .quad 0xd3f938ad889596b8 + .quad 0xf0f52dc746d5dd25 + .quad 0x57968290bb3a0095 + .quad 0xf71ab8430a20e101 + .quad 0xf393658d24f0ec47 + .quad 0xcf7509a86ee2eed1 + .quad 0x7dc43e35dc2aa3e1 + + // 2^160 * 2 * B + + .quad 0x85966665887dd9c3 + .quad 0xc90f9b314bb05355 + .quad 0xc6e08df8ef2079b1 + .quad 0x7ef72016758cc12f + .quad 0x5a782a5c273e9718 + .quad 0x3576c6995e4efd94 + .quad 0x0f2ed8051f237d3e + .quad 0x044fb81d82d50a99 + .quad 0xc1df18c5a907e3d9 + .quad 0x57b3371dce4c6359 + .quad 0xca704534b201bb49 + .quad 0x7f79823f9c30dd2e + + // 2^160 * 3 * B + + .quad 0x8334d239a3b513e8 + .quad 0xc13670d4b91fa8d8 + .quad 0x12b54136f590bd33 + .quad 0x0a4e0373d784d9b4 + .quad 0x6a9c1ff068f587ba + .quad 0x0827894e0050c8de + .quad 0x3cbf99557ded5be7 + .quad 0x64a9b0431c06d6f0 + .quad 0x2eb3d6a15b7d2919 + .quad 0xb0b4f6a0d53a8235 + .quad 0x7156ce4389a45d47 + .quad 0x071a7d0ace18346c + + // 2^160 * 4 * B + + .quad 0xd3072daac887ba0b + .quad 0x01262905bfa562ee + .quad 0xcf543002c0ef768b + .quad 0x2c3bcc7146ea7e9c + .quad 0xcc0c355220e14431 + .quad 0x0d65950709b15141 + .quad 0x9af5621b209d5f36 + .quad 0x7c69bcf7617755d3 + .quad 0x07f0d7eb04e8295f + .quad 0x10db18252f50f37d + .quad 0xe951a9a3171798d7 + .quad 0x6f5a9a7322aca51d + + // 2^160 * 5 * B + + .quad 0x8ba1000c2f41c6c5 + .quad 0xc49f79c10cfefb9b + .quad 0x4efa47703cc51c9f + .quad 0x494e21a2e147afca + .quad 0xe729d4eba3d944be + .quad 0x8d9e09408078af9e + .quad 0x4525567a47869c03 + .quad 0x02ab9680ee8d3b24 + .quad 0xefa48a85dde50d9a + .quad 0x219a224e0fb9a249 + .quad 0xfa091f1dd91ef6d9 + .quad 0x6b5d76cbea46bb34 + + // 2^160 * 6 * B + + .quad 0x8857556cec0cd994 + .quad 0x6472dc6f5cd01dba + .quad 0xaf0169148f42b477 + .quad 0x0ae333f685277354 + .quad 0xe0f941171e782522 + .quad 0xf1e6ae74036936d3 + .quad 0x408b3ea2d0fcc746 + .quad 0x16fb869c03dd313e + .quad 0x288e199733b60962 + .quad 0x24fc72b4d8abe133 + .quad 0x4811f7ed0991d03e + .quad 0x3f81e38b8f70d075 + + // 2^160 * 7 * B + + .quad 0x7f910fcc7ed9affe + .quad 0x545cb8a12465874b + .quad 0xa8397ed24b0c4704 + .quad 0x50510fc104f50993 + .quad 0x0adb7f355f17c824 + .quad 0x74b923c3d74299a4 + .quad 0xd57c3e8bcbf8eaf7 + .quad 0x0ad3e2d34cdedc3d + .quad 0x6f0c0fc5336e249d + .quad 0x745ede19c331cfd9 + .quad 0xf2d6fd0009eefe1c + .quad 0x127c158bf0fa1ebe + + // 2^160 * 8 * B + + .quad 0xf6197c422e9879a2 + .quad 0xa44addd452ca3647 + .quad 0x9b413fc14b4eaccb + .quad 0x354ef87d07ef4f68 + .quad 0xdea28fc4ae51b974 + .quad 0x1d9973d3744dfe96 + .quad 0x6240680b873848a8 + .quad 0x4ed82479d167df95 + .quad 0xfee3b52260c5d975 + .quad 0x50352efceb41b0b8 + .quad 0x8808ac30a9f6653c + .quad 0x302d92d20539236d + + // 2^164 * 1 * B + + .quad 0x4c59023fcb3efb7c + .quad 0x6c2fcb99c63c2a94 + .quad 0xba4190e2c3c7e084 + .quad 0x0e545daea51874d9 + .quad 0x957b8b8b0df53c30 + .quad 0x2a1c770a8e60f098 + .quad 0xbbc7a670345796de + .quad 0x22a48f9a90c99bc9 + .quad 0x6b7dc0dc8d3fac58 + .quad 0x5497cd6ce6e42bfd + .quad 0x542f7d1bf400d305 + .quad 0x4159f47f048d9136 + + // 2^164 * 2 * B + + .quad 0x20ad660839e31e32 + .quad 0xf81e1bd58405be50 + .quad 0xf8064056f4dabc69 + .quad 0x14d23dd4ce71b975 + .quad 0x748515a8bbd24839 + .quad 0x77128347afb02b55 + .quad 0x50ba2ac649a2a17f + .quad 0x060525513ad730f1 + .quad 0xf2398e098aa27f82 + .quad 0x6d7982bb89a1b024 + .quad 0xfa694084214dd24c + .quad 0x71ab966fa32301c3 + + // 2^164 * 3 * B + + .quad 0x2dcbd8e34ded02fc + .quad 0x1151f3ec596f22aa + .quad 0xbca255434e0328da + .quad 0x35768fbe92411b22 + .quad 0xb1088a0702809955 + .quad 0x43b273ea0b43c391 + .quad 0xca9b67aefe0686ed + .quad 0x605eecbf8335f4ed + .quad 0x83200a656c340431 + .quad 0x9fcd71678ee59c2f + .quad 0x75d4613f71300f8a + .quad 0x7a912faf60f542f9 + + // 2^164 * 4 * B + + .quad 0xb204585e5edc1a43 + .quad 0x9f0e16ee5897c73c + .quad 0x5b82c0ae4e70483c + .quad 0x624a170e2bddf9be + .quad 0x253f4f8dfa2d5597 + .quad 0x25e49c405477130c + .quad 0x00c052e5996b1102 + .quad 0x33cb966e33bb6c4a + .quad 0x597028047f116909 + .quad 0x828ac41c1e564467 + .quad 0x70417dbde6217387 + .quad 0x721627aefbac4384 + + // 2^164 * 5 * B + + .quad 0x97d03bc38736add5 + .quad 0x2f1422afc532b130 + .quad 0x3aa68a057101bbc4 + .quad 0x4c946cf7e74f9fa7 + .quad 0xfd3097bc410b2f22 + .quad 0xf1a05da7b5cfa844 + .quad 0x61289a1def57ca74 + .quad 0x245ea199bb821902 + .quad 0xaedca66978d477f8 + .quad 0x1898ba3c29117fe1 + .quad 0xcf73f983720cbd58 + .quad 0x67da12e6b8b56351 + + // 2^164 * 6 * B + + .quad 0x7067e187b4bd6e07 + .quad 0x6e8f0203c7d1fe74 + .quad 0x93c6aa2f38c85a30 + .quad 0x76297d1f3d75a78a + .quad 0x2b7ef3d38ec8308c + .quad 0x828fd7ec71eb94ab + .quad 0x807c3b36c5062abd + .quad 0x0cb64cb831a94141 + .quad 0x3030fc33534c6378 + .quad 0xb9635c5ce541e861 + .quad 0x15d9a9bed9b2c728 + .quad 0x49233ea3f3775dcb + + // 2^164 * 7 * B + + .quad 0x629398fa8dbffc3a + .quad 0xe12fe52dd54db455 + .quad 0xf3be11dfdaf25295 + .quad 0x628b140dce5e7b51 + .quad 0x7b3985fe1c9f249b + .quad 0x4fd6b2d5a1233293 + .quad 0xceb345941adf4d62 + .quad 0x6987ff6f542de50c + .quad 0x47e241428f83753c + .quad 0x6317bebc866af997 + .quad 0xdabb5b433d1a9829 + .quad 0x074d8d245287fb2d + + // 2^164 * 8 * B + + .quad 0x8337d9cd440bfc31 + .quad 0x729d2ca1af318fd7 + .quad 0xa040a4a4772c2070 + .quad 0x46002ef03a7349be + .quad 0x481875c6c0e31488 + .quad 0x219429b2e22034b4 + .quad 0x7223c98a31283b65 + .quad 0x3420d60b342277f9 + .quad 0xfaa23adeaffe65f7 + .quad 0x78261ed45be0764c + .quad 0x441c0a1e2f164403 + .quad 0x5aea8e567a87d395 + + // 2^168 * 1 * B + + .quad 0x7813c1a2bca4283d + .quad 0xed62f091a1863dd9 + .quad 0xaec7bcb8c268fa86 + .quad 0x10e5d3b76f1cae4c + .quad 0x2dbc6fb6e4e0f177 + .quad 0x04e1bf29a4bd6a93 + .quad 0x5e1966d4787af6e8 + .quad 0x0edc5f5eb426d060 + .quad 0x5453bfd653da8e67 + .quad 0xe9dc1eec24a9f641 + .quad 0xbf87263b03578a23 + .quad 0x45b46c51361cba72 + + // 2^168 * 2 * B + + .quad 0xa9402abf314f7fa1 + .quad 0xe257f1dc8e8cf450 + .quad 0x1dbbd54b23a8be84 + .quad 0x2177bfa36dcb713b + .quad 0xce9d4ddd8a7fe3e4 + .quad 0xab13645676620e30 + .quad 0x4b594f7bb30e9958 + .quad 0x5c1c0aef321229df + .quad 0x37081bbcfa79db8f + .quad 0x6048811ec25f59b3 + .quad 0x087a76659c832487 + .quad 0x4ae619387d8ab5bb + + // 2^168 * 3 * B + + .quad 0x8ddbf6aa5344a32e + .quad 0x7d88eab4b41b4078 + .quad 0x5eb0eb974a130d60 + .quad 0x1a00d91b17bf3e03 + .quad 0x61117e44985bfb83 + .quad 0xfce0462a71963136 + .quad 0x83ac3448d425904b + .quad 0x75685abe5ba43d64 + .quad 0x6e960933eb61f2b2 + .quad 0x543d0fa8c9ff4952 + .quad 0xdf7275107af66569 + .quad 0x135529b623b0e6aa + + // 2^168 * 4 * B + + .quad 0x18f0dbd7add1d518 + .quad 0x979f7888cfc11f11 + .quad 0x8732e1f07114759b + .quad 0x79b5b81a65ca3a01 + .quad 0xf5c716bce22e83fe + .quad 0xb42beb19e80985c1 + .quad 0xec9da63714254aae + .quad 0x5972ea051590a613 + .quad 0x0fd4ac20dc8f7811 + .quad 0x9a9ad294ac4d4fa8 + .quad 0xc01b2d64b3360434 + .quad 0x4f7e9c95905f3bdb + + // 2^168 * 5 * B + + .quad 0x62674bbc5781302e + .quad 0xd8520f3989addc0f + .quad 0x8c2999ae53fbd9c6 + .quad 0x31993ad92e638e4c + .quad 0x71c8443d355299fe + .quad 0x8bcd3b1cdbebead7 + .quad 0x8092499ef1a49466 + .quad 0x1942eec4a144adc8 + .quad 0x7dac5319ae234992 + .quad 0x2c1b3d910cea3e92 + .quad 0x553ce494253c1122 + .quad 0x2a0a65314ef9ca75 + + // 2^168 * 6 * B + + .quad 0x2db7937ff7f927c2 + .quad 0xdb741f0617d0a635 + .quad 0x5982f3a21155af76 + .quad 0x4cf6e218647c2ded + .quad 0xcf361acd3c1c793a + .quad 0x2f9ebcac5a35bc3b + .quad 0x60e860e9a8cda6ab + .quad 0x055dc39b6dea1a13 + .quad 0xb119227cc28d5bb6 + .quad 0x07e24ebc774dffab + .quad 0xa83c78cee4a32c89 + .quad 0x121a307710aa24b6 + + // 2^168 * 7 * B + + .quad 0xe4db5d5e9f034a97 + .quad 0xe153fc093034bc2d + .quad 0x460546919551d3b1 + .quad 0x333fc76c7a40e52d + .quad 0xd659713ec77483c9 + .quad 0x88bfe077b82b96af + .quad 0x289e28231097bcd3 + .quad 0x527bb94a6ced3a9b + .quad 0x563d992a995b482e + .quad 0x3405d07c6e383801 + .quad 0x485035de2f64d8e5 + .quad 0x6b89069b20a7a9f7 + + // 2^168 * 8 * B + + .quad 0x812aa0416270220d + .quad 0x995a89faf9245b4e + .quad 0xffadc4ce5072ef05 + .quad 0x23bc2103aa73eb73 + .quad 0x4082fa8cb5c7db77 + .quad 0x068686f8c734c155 + .quad 0x29e6c8d9f6e7a57e + .quad 0x0473d308a7639bcf + .quad 0xcaee792603589e05 + .quad 0x2b4b421246dcc492 + .quad 0x02a1ef74e601a94f + .quad 0x102f73bfde04341a + + // 2^172 * 1 * B + + .quad 0xb5a2d50c7ec20d3e + .quad 0xc64bdd6ea0c97263 + .quad 0x56e89052c1ff734d + .quad 0x4929c6f72b2ffaba + .quad 0x358ecba293a36247 + .quad 0xaf8f9862b268fd65 + .quad 0x412f7e9968a01c89 + .quad 0x5786f312cd754524 + .quad 0x337788ffca14032c + .quad 0xf3921028447f1ee3 + .quad 0x8b14071f231bccad + .quad 0x4c817b4bf2344783 + + // 2^172 * 2 * B + + .quad 0x0ff853852871b96e + .quad 0xe13e9fab60c3f1bb + .quad 0xeefd595325344402 + .quad 0x0a37c37075b7744b + .quad 0x413ba057a40b4484 + .quad 0xba4c2e1a4f5f6a43 + .quad 0x614ba0a5aee1d61c + .quad 0x78a1531a8b05dc53 + .quad 0x6cbdf1703ad0562b + .quad 0x8ecf4830c92521a3 + .quad 0xdaebd303fd8424e7 + .quad 0x72ad82a42e5ec56f + + // 2^172 * 3 * B + + .quad 0x3f9e8e35bafb65f6 + .quad 0x39d69ec8f27293a1 + .quad 0x6cb8cd958cf6a3d0 + .quad 0x1734778173adae6d + .quad 0xc368939167024bc3 + .quad 0x8e69d16d49502fda + .quad 0xfcf2ec3ce45f4b29 + .quad 0x065f669ea3b4cbc4 + .quad 0x8a00aec75532db4d + .quad 0xb869a4e443e31bb1 + .quad 0x4a0f8552d3a7f515 + .quad 0x19adeb7c303d7c08 + + // 2^172 * 4 * B + + .quad 0xc720cb6153ead9a3 + .quad 0x55b2c97f512b636e + .quad 0xb1e35b5fd40290b1 + .quad 0x2fd9ccf13b530ee2 + .quad 0x9d05ba7d43c31794 + .quad 0x2470c8ff93322526 + .quad 0x8323dec816197438 + .quad 0x2852709881569b53 + .quad 0x07bd475b47f796b8 + .quad 0xd2c7b013542c8f54 + .quad 0x2dbd23f43b24f87e + .quad 0x6551afd77b0901d6 + + // 2^172 * 5 * B + + .quad 0x4546baaf54aac27f + .quad 0xf6f66fecb2a45a28 + .quad 0x582d1b5b562bcfe8 + .quad 0x44b123f3920f785f + .quad 0x68a24ce3a1d5c9ac + .quad 0xbb77a33d10ff6461 + .quad 0x0f86ce4425d3166e + .quad 0x56507c0950b9623b + .quad 0x1206f0b7d1713e63 + .quad 0x353fe3d915bafc74 + .quad 0x194ceb970ad9d94d + .quad 0x62fadd7cf9d03ad3 + + // 2^172 * 6 * B + + .quad 0xc6b5967b5598a074 + .quad 0x5efe91ce8e493e25 + .quad 0xd4b72c4549280888 + .quad 0x20ef1149a26740c2 + .quad 0x3cd7bc61e7ce4594 + .quad 0xcd6b35a9b7dd267e + .quad 0xa080abc84366ef27 + .quad 0x6ec7c46f59c79711 + .quad 0x2f07ad636f09a8a2 + .quad 0x8697e6ce24205e7d + .quad 0xc0aefc05ee35a139 + .quad 0x15e80958b5f9d897 + + // 2^172 * 7 * B + + .quad 0x25a5ef7d0c3e235b + .quad 0x6c39c17fbe134ee7 + .quad 0xc774e1342dc5c327 + .quad 0x021354b892021f39 + .quad 0x4dd1ed355bb061c4 + .quad 0x42dc0cef941c0700 + .quad 0x61305dc1fd86340e + .quad 0x56b2cc930e55a443 + .quad 0x1df79da6a6bfc5a2 + .quad 0x02f3a2749fde4369 + .quad 0xb323d9f2cda390a7 + .quad 0x7be0847b8774d363 + + // 2^172 * 8 * B + + .quad 0x8c99cc5a8b3f55c3 + .quad 0x0611d7253fded2a0 + .quad 0xed2995ff36b70a36 + .quad 0x1f699a54d78a2619 + .quad 0x1466f5af5307fa11 + .quad 0x817fcc7ded6c0af2 + .quad 0x0a6de44ec3a4a3fb + .quad 0x74071475bc927d0b + .quad 0xe77292f373e7ea8a + .quad 0x296537d2cb045a31 + .quad 0x1bd0653ed3274fde + .quad 0x2f9a2c4476bd2966 + + // 2^176 * 1 * B + + .quad 0xeb18b9ab7f5745c6 + .quad 0x023a8aee5787c690 + .quad 0xb72712da2df7afa9 + .quad 0x36597d25ea5c013d + .quad 0xa2b4dae0b5511c9a + .quad 0x7ac860292bffff06 + .quad 0x981f375df5504234 + .quad 0x3f6bd725da4ea12d + .quad 0x734d8d7b106058ac + .quad 0xd940579e6fc6905f + .quad 0x6466f8f99202932d + .quad 0x7b7ecc19da60d6d0 + + // 2^176 * 2 * B + + .quad 0x78c2373c695c690d + .quad 0xdd252e660642906e + .quad 0x951d44444ae12bd2 + .quad 0x4235ad7601743956 + .quad 0x6dae4a51a77cfa9b + .quad 0x82263654e7a38650 + .quad 0x09bbffcd8f2d82db + .quad 0x03bedc661bf5caba + .quad 0x6258cb0d078975f5 + .quad 0x492942549189f298 + .quad 0xa0cab423e2e36ee4 + .quad 0x0e7ce2b0cdf066a1 + + // 2^176 * 3 * B + + .quad 0xc494643ac48c85a3 + .quad 0xfd361df43c6139ad + .quad 0x09db17dd3ae94d48 + .quad 0x666e0a5d8fb4674a + .quad 0xfea6fedfd94b70f9 + .quad 0xf130c051c1fcba2d + .quad 0x4882d47e7f2fab89 + .quad 0x615256138aeceeb5 + .quad 0x2abbf64e4870cb0d + .quad 0xcd65bcf0aa458b6b + .quad 0x9abe4eba75e8985d + .quad 0x7f0bc810d514dee4 + + // 2^176 * 4 * B + + .quad 0xb9006ba426f4136f + .quad 0x8d67369e57e03035 + .quad 0xcbc8dfd94f463c28 + .quad 0x0d1f8dbcf8eedbf5 + .quad 0x83ac9dad737213a0 + .quad 0x9ff6f8ba2ef72e98 + .quad 0x311e2edd43ec6957 + .quad 0x1d3a907ddec5ab75 + .quad 0xba1693313ed081dc + .quad 0x29329fad851b3480 + .quad 0x0128013c030321cb + .quad 0x00011b44a31bfde3 + + // 2^176 * 5 * B + + .quad 0x3fdfa06c3fc66c0c + .quad 0x5d40e38e4dd60dd2 + .quad 0x7ae38b38268e4d71 + .quad 0x3ac48d916e8357e1 + .quad 0x16561f696a0aa75c + .quad 0xc1bf725c5852bd6a + .quad 0x11a8dd7f9a7966ad + .quad 0x63d988a2d2851026 + .quad 0x00120753afbd232e + .quad 0xe92bceb8fdd8f683 + .quad 0xf81669b384e72b91 + .quad 0x33fad52b2368a066 + + // 2^176 * 6 * B + + .quad 0x540649c6c5e41e16 + .quad 0x0af86430333f7735 + .quad 0xb2acfcd2f305e746 + .quad 0x16c0f429a256dca7 + .quad 0x8d2cc8d0c422cfe8 + .quad 0x072b4f7b05a13acb + .quad 0xa3feb6e6ecf6a56f + .quad 0x3cc355ccb90a71e2 + .quad 0xe9b69443903e9131 + .quad 0xb8a494cb7a5637ce + .quad 0xc87cd1a4baba9244 + .quad 0x631eaf426bae7568 + + // 2^176 * 7 * B + + .quad 0xb3e90410da66fe9f + .quad 0x85dd4b526c16e5a6 + .quad 0xbc3d97611ef9bf83 + .quad 0x5599648b1ea919b5 + .quad 0x47d975b9a3700de8 + .quad 0x7280c5fbe2f80552 + .quad 0x53658f2732e45de1 + .quad 0x431f2c7f665f80b5 + .quad 0xd6026344858f7b19 + .quad 0x14ab352fa1ea514a + .quad 0x8900441a2090a9d7 + .quad 0x7b04715f91253b26 + + // 2^176 * 8 * B + + .quad 0x83edbd28acf6ae43 + .quad 0x86357c8b7d5c7ab4 + .quad 0xc0404769b7eb2c44 + .quad 0x59b37bf5c2f6583f + .quad 0xb376c280c4e6bac6 + .quad 0x970ed3dd6d1d9b0b + .quad 0xb09a9558450bf944 + .quad 0x48d0acfa57cde223 + .quad 0xb60f26e47dabe671 + .quad 0xf1d1a197622f3a37 + .quad 0x4208ce7ee9960394 + .quad 0x16234191336d3bdb + + // 2^180 * 1 * B + + .quad 0xf19aeac733a63aef + .quad 0x2c7fba5d4442454e + .quad 0x5da87aa04795e441 + .quad 0x413051e1a4e0b0f5 + .quad 0x852dd1fd3d578bbe + .quad 0x2b65ce72c3286108 + .quad 0x658c07f4eace2273 + .quad 0x0933f804ec38ab40 + .quad 0xa7ab69798d496476 + .quad 0x8121aadefcb5abc8 + .quad 0xa5dc12ef7b539472 + .quad 0x07fd47065e45351a + + // 2^180 * 2 * B + + .quad 0xc8583c3d258d2bcd + .quad 0x17029a4daf60b73f + .quad 0xfa0fc9d6416a3781 + .quad 0x1c1e5fba38b3fb23 + .quad 0x304211559ae8e7c3 + .quad 0xf281b229944882a5 + .quad 0x8a13ac2e378250e4 + .quad 0x014afa0954ba48f4 + .quad 0xcb3197001bb3666c + .quad 0x330060524bffecb9 + .quad 0x293711991a88233c + .quad 0x291884363d4ed364 + + // 2^180 * 3 * B + + .quad 0x033c6805dc4babfa + .quad 0x2c15bf5e5596ecc1 + .quad 0x1bc70624b59b1d3b + .quad 0x3ede9850a19f0ec5 + .quad 0xfb9d37c3bc1ab6eb + .quad 0x02be14534d57a240 + .quad 0xf4d73415f8a5e1f6 + .quad 0x5964f4300ccc8188 + .quad 0xe44a23152d096800 + .quad 0x5c08c55970866996 + .quad 0xdf2db60a46affb6e + .quad 0x579155c1f856fd89 + + // 2^180 * 4 * B + + .quad 0x96324edd12e0c9ef + .quad 0x468b878df2420297 + .quad 0x199a3776a4f573be + .quad 0x1e7fbcf18e91e92a + .quad 0xb5f16b630817e7a6 + .quad 0x808c69233c351026 + .quad 0x324a983b54cef201 + .quad 0x53c092084a485345 + .quad 0xd2d41481f1cbafbf + .quad 0x231d2db6716174e5 + .quad 0x0b7d7656e2a55c98 + .quad 0x3e955cd82aa495f6 + + // 2^180 * 5 * B + + .quad 0xe48f535e3ed15433 + .quad 0xd075692a0d7270a3 + .quad 0x40fbd21daade6387 + .quad 0x14264887cf4495f5 + .quad 0xab39f3ef61bb3a3f + .quad 0x8eb400652eb9193e + .quad 0xb5de6ecc38c11f74 + .quad 0x654d7e9626f3c49f + .quad 0xe564cfdd5c7d2ceb + .quad 0x82eeafded737ccb9 + .quad 0x6107db62d1f9b0ab + .quad 0x0b6baac3b4358dbb + + // 2^180 * 6 * B + + .quad 0x7ae62bcb8622fe98 + .quad 0x47762256ceb891af + .quad 0x1a5a92bcf2e406b4 + .quad 0x7d29401784e41501 + .quad 0x204abad63700a93b + .quad 0xbe0023d3da779373 + .quad 0xd85f0346633ab709 + .quad 0x00496dc490820412 + .quad 0x1c74b88dc27e6360 + .quad 0x074854268d14850c + .quad 0xa145fb7b3e0dcb30 + .quad 0x10843f1b43803b23 + + // 2^180 * 7 * B + + .quad 0xc5f90455376276dd + .quad 0xce59158dd7645cd9 + .quad 0x92f65d511d366b39 + .quad 0x11574b6e526996c4 + .quad 0xd56f672de324689b + .quad 0xd1da8aedb394a981 + .quad 0xdd7b58fe9168cfed + .quad 0x7ce246cd4d56c1e8 + .quad 0xb8f4308e7f80be53 + .quad 0x5f3cb8cb34a9d397 + .quad 0x18a961bd33cc2b2c + .quad 0x710045fb3a9af671 + + // 2^180 * 8 * B + + .quad 0x73f93d36101b95eb + .quad 0xfaef33794f6f4486 + .quad 0x5651735f8f15e562 + .quad 0x7fa3f19058b40da1 + .quad 0xa03fc862059d699e + .quad 0x2370cfa19a619e69 + .quad 0xc4fe3b122f823deb + .quad 0x1d1b056fa7f0844e + .quad 0x1bc64631e56bf61f + .quad 0xd379ab106e5382a3 + .quad 0x4d58c57e0540168d + .quad 0x566256628442d8e4 + + // 2^184 * 1 * B + + .quad 0xb9e499def6267ff6 + .quad 0x7772ca7b742c0843 + .quad 0x23a0153fe9a4f2b1 + .quad 0x2cdfdfecd5d05006 + .quad 0xdd499cd61ff38640 + .quad 0x29cd9bc3063625a0 + .quad 0x51e2d8023dd73dc3 + .quad 0x4a25707a203b9231 + .quad 0x2ab7668a53f6ed6a + .quad 0x304242581dd170a1 + .quad 0x4000144c3ae20161 + .quad 0x5721896d248e49fc + + // 2^184 * 2 * B + + .quad 0x0b6e5517fd181bae + .quad 0x9022629f2bb963b4 + .quad 0x5509bce932064625 + .quad 0x578edd74f63c13da + .quad 0x285d5091a1d0da4e + .quad 0x4baa6fa7b5fe3e08 + .quad 0x63e5177ce19393b3 + .quad 0x03c935afc4b030fd + .quad 0x997276c6492b0c3d + .quad 0x47ccc2c4dfe205fc + .quad 0xdcd29b84dd623a3c + .quad 0x3ec2ab590288c7a2 + + // 2^184 * 3 * B + + .quad 0xa1a0d27be4d87bb9 + .quad 0xa98b4deb61391aed + .quad 0x99a0ddd073cb9b83 + .quad 0x2dd5c25a200fcace + .quad 0xa7213a09ae32d1cb + .quad 0x0f2b87df40f5c2d5 + .quad 0x0baea4c6e81eab29 + .quad 0x0e1bf66c6adbac5e + .quad 0xe2abd5e9792c887e + .quad 0x1a020018cb926d5d + .quad 0xbfba69cdbaae5f1e + .quad 0x730548b35ae88f5f + + // 2^184 * 4 * B + + .quad 0xc43551a3cba8b8ee + .quad 0x65a26f1db2115f16 + .quad 0x760f4f52ab8c3850 + .quad 0x3043443b411db8ca + .quad 0x805b094ba1d6e334 + .quad 0xbf3ef17709353f19 + .quad 0x423f06cb0622702b + .quad 0x585a2277d87845dd + .quad 0xa18a5f8233d48962 + .quad 0x6698c4b5ec78257f + .quad 0xa78e6fa5373e41ff + .quad 0x7656278950ef981f + + // 2^184 * 5 * B + + .quad 0x38c3cf59d51fc8c0 + .quad 0x9bedd2fd0506b6f2 + .quad 0x26bf109fab570e8f + .quad 0x3f4160a8c1b846a6 + .quad 0xe17073a3ea86cf9d + .quad 0x3a8cfbb707155fdc + .quad 0x4853e7fc31838a8e + .quad 0x28bbf484b613f616 + .quad 0xf2612f5c6f136c7c + .quad 0xafead107f6dd11be + .quad 0x527e9ad213de6f33 + .quad 0x1e79cb358188f75d + + // 2^184 * 6 * B + + .quad 0x013436c3eef7e3f1 + .quad 0x828b6a7ffe9e10f8 + .quad 0x7ff908e5bcf9defc + .quad 0x65d7951b3a3b3831 + .quad 0x77e953d8f5e08181 + .quad 0x84a50c44299dded9 + .quad 0xdc6c2d0c864525e5 + .quad 0x478ab52d39d1f2f4 + .quad 0x66a6a4d39252d159 + .quad 0xe5dde1bc871ac807 + .quad 0xb82c6b40a6c1c96f + .quad 0x16d87a411a212214 + + // 2^184 * 7 * B + + .quad 0xb3bd7e5a42066215 + .quad 0x879be3cd0c5a24c1 + .quad 0x57c05db1d6f994b7 + .quad 0x28f87c8165f38ca6 + .quad 0xfba4d5e2d54e0583 + .quad 0xe21fafd72ebd99fa + .quad 0x497ac2736ee9778f + .quad 0x1f990b577a5a6dde + .quad 0xa3344ead1be8f7d6 + .quad 0x7d1e50ebacea798f + .quad 0x77c6569e520de052 + .quad 0x45882fe1534d6d3e + + // 2^184 * 8 * B + + .quad 0x6669345d757983d6 + .quad 0x62b6ed1117aa11a6 + .quad 0x7ddd1857985e128f + .quad 0x688fe5b8f626f6dd + .quad 0xd8ac9929943c6fe4 + .quad 0xb5f9f161a38392a2 + .quad 0x2699db13bec89af3 + .quad 0x7dcf843ce405f074 + .quad 0x6c90d6484a4732c0 + .quad 0xd52143fdca563299 + .quad 0xb3be28c3915dc6e1 + .quad 0x6739687e7327191b + + // 2^188 * 1 * B + + .quad 0x9f65c5ea200814cf + .quad 0x840536e169a31740 + .quad 0x8b0ed13925c8b4ad + .quad 0x0080dbafe936361d + .quad 0x8ce5aad0c9cb971f + .quad 0x1156aaa99fd54a29 + .quad 0x41f7247015af9b78 + .quad 0x1fe8cca8420f49aa + .quad 0x72a1848f3c0cc82a + .quad 0x38c560c2877c9e54 + .quad 0x5004e228ce554140 + .quad 0x042418a103429d71 + + // 2^188 * 2 * B + + .quad 0x899dea51abf3ff5f + .quad 0x9b93a8672fc2d8ba + .quad 0x2c38cb97be6ebd5c + .quad 0x114d578497263b5d + .quad 0x58e84c6f20816247 + .quad 0x8db2b2b6e36fd793 + .quad 0x977182561d484d85 + .quad 0x0822024f8632abd7 + .quad 0xb301bb7c6b1beca3 + .quad 0x55393f6dc6eb1375 + .quad 0x910d281097b6e4eb + .quad 0x1ad4548d9d479ea3 + + // 2^188 * 3 * B + + .quad 0xcd5a7da0389a48fd + .quad 0xb38fa4aa9a78371e + .quad 0xc6d9761b2cdb8e6c + .quad 0x35cf51dbc97e1443 + .quad 0xa06fe66d0fe9fed3 + .quad 0xa8733a401c587909 + .quad 0x30d14d800df98953 + .quad 0x41ce5876c7b30258 + .quad 0x59ac3bc5d670c022 + .quad 0xeae67c109b119406 + .quad 0x9798bdf0b3782fda + .quad 0x651e3201fd074092 + + // 2^188 * 4 * B + + .quad 0xd63d8483ef30c5cf + .quad 0x4cd4b4962361cc0c + .quad 0xee90e500a48426ac + .quad 0x0af51d7d18c14eeb + .quad 0xa57ba4a01efcae9e + .quad 0x769f4beedc308a94 + .quad 0xd1f10eeb3603cb2e + .quad 0x4099ce5e7e441278 + .quad 0x1ac98e4f8a5121e9 + .quad 0x7dae9544dbfa2fe0 + .quad 0x8320aa0dd6430df9 + .quad 0x667282652c4a2fb5 + + // 2^188 * 5 * B + + .quad 0x874621f4d86bc9ab + .quad 0xb54c7bbe56fe6fea + .quad 0x077a24257fadc22c + .quad 0x1ab53be419b90d39 + .quad 0xada8b6e02946db23 + .quad 0x1c0ce51a7b253ab7 + .quad 0x8448c85a66dd485b + .quad 0x7f1fc025d0675adf + .quad 0xd8ee1b18319ea6aa + .quad 0x004d88083a21f0da + .quad 0x3bd6aa1d883a4f4b + .quad 0x4db9a3a6dfd9fd14 + + // 2^188 * 6 * B + + .quad 0x8ce7b23bb99c0755 + .quad 0x35c5d6edc4f50f7a + .quad 0x7e1e2ed2ed9b50c3 + .quad 0x36305f16e8934da1 + .quad 0xd95b00bbcbb77c68 + .quad 0xddbc846a91f17849 + .quad 0x7cf700aebe28d9b3 + .quad 0x5ce1285c85d31f3e + .quad 0x31b6972d98b0bde8 + .quad 0x7d920706aca6de5b + .quad 0xe67310f8908a659f + .quad 0x50fac2a6efdf0235 + + // 2^188 * 7 * B + + .quad 0xf3d3a9f35b880f5a + .quad 0xedec050cdb03e7c2 + .quad 0xa896981ff9f0b1a2 + .quad 0x49a4ae2bac5e34a4 + .quad 0x295b1c86f6f449bc + .quad 0x51b2e84a1f0ab4dd + .quad 0xc001cb30aa8e551d + .quad 0x6a28d35944f43662 + .quad 0x28bb12ee04a740e0 + .quad 0x14313bbd9bce8174 + .quad 0x72f5b5e4e8c10c40 + .quad 0x7cbfb19936adcd5b + + // 2^188 * 8 * B + + .quad 0xa311ddc26b89792d + .quad 0x1b30b4c6da512664 + .quad 0x0ca77b4ccf150859 + .quad 0x1de443df1b009408 + .quad 0x8e793a7acc36e6e0 + .quad 0xf9fab7a37d586eed + .quad 0x3a4f9692bae1f4e4 + .quad 0x1c14b03eff5f447e + .quad 0x19647bd114a85291 + .quad 0x57b76cb21034d3af + .quad 0x6329db440f9d6dfa + .quad 0x5ef43e586a571493 + + // 2^192 * 1 * B + + .quad 0xef782014385675a6 + .quad 0xa2649f30aafda9e8 + .quad 0x4cd1eb505cdfa8cb + .quad 0x46115aba1d4dc0b3 + .quad 0xa66dcc9dc80c1ac0 + .quad 0x97a05cf41b38a436 + .quad 0xa7ebf3be95dbd7c6 + .quad 0x7da0b8f68d7e7dab + .quad 0xd40f1953c3b5da76 + .quad 0x1dac6f7321119e9b + .quad 0x03cc6021feb25960 + .quad 0x5a5f887e83674b4b + + // 2^192 * 2 * B + + .quad 0x8f6301cf70a13d11 + .quad 0xcfceb815350dd0c4 + .quad 0xf70297d4a4bca47e + .quad 0x3669b656e44d1434 + .quad 0x9e9628d3a0a643b9 + .quad 0xb5c3cb00e6c32064 + .quad 0x9b5302897c2dec32 + .quad 0x43e37ae2d5d1c70c + .quad 0x387e3f06eda6e133 + .quad 0x67301d5199a13ac0 + .quad 0xbd5ad8f836263811 + .quad 0x6a21e6cd4fd5e9be + + // 2^192 * 3 * B + + .quad 0xf1c6170a3046e65f + .quad 0x58712a2a00d23524 + .quad 0x69dbbd3c8c82b755 + .quad 0x586bf9f1a195ff57 + .quad 0xef4129126699b2e3 + .quad 0x71d30847708d1301 + .quad 0x325432d01182b0bd + .quad 0x45371b07001e8b36 + .quad 0xa6db088d5ef8790b + .quad 0x5278f0dc610937e5 + .quad 0xac0349d261a16eb8 + .quad 0x0eafb03790e52179 + + // 2^192 * 4 * B + + .quad 0x960555c13748042f + .quad 0x219a41e6820baa11 + .quad 0x1c81f73873486d0c + .quad 0x309acc675a02c661 + .quad 0x5140805e0f75ae1d + .quad 0xec02fbe32662cc30 + .quad 0x2cebdf1eea92396d + .quad 0x44ae3344c5435bb3 + .quad 0x9cf289b9bba543ee + .quad 0xf3760e9d5ac97142 + .quad 0x1d82e5c64f9360aa + .quad 0x62d5221b7f94678f + + // 2^192 * 5 * B + + .quad 0x524c299c18d0936d + .quad 0xc86bb56c8a0c1a0c + .quad 0xa375052edb4a8631 + .quad 0x5c0efde4bc754562 + .quad 0x7585d4263af77a3c + .quad 0xdfae7b11fee9144d + .quad 0xa506708059f7193d + .quad 0x14f29a5383922037 + .quad 0xdf717edc25b2d7f5 + .quad 0x21f970db99b53040 + .quad 0xda9234b7c3ed4c62 + .quad 0x5e72365c7bee093e + + // 2^192 * 6 * B + + .quad 0x575bfc074571217f + .quad 0x3779675d0694d95b + .quad 0x9a0a37bbf4191e33 + .quad 0x77f1104c47b4eabc + .quad 0x7d9339062f08b33e + .quad 0x5b9659e5df9f32be + .quad 0xacff3dad1f9ebdfd + .quad 0x70b20555cb7349b7 + .quad 0xbe5113c555112c4c + .quad 0x6688423a9a881fcd + .quad 0x446677855e503b47 + .quad 0x0e34398f4a06404a + + // 2^192 * 7 * B + + .quad 0xb67d22d93ecebde8 + .quad 0x09b3e84127822f07 + .quad 0x743fa61fb05b6d8d + .quad 0x5e5405368a362372 + .quad 0x18930b093e4b1928 + .quad 0x7de3e10e73f3f640 + .quad 0xf43217da73395d6f + .quad 0x6f8aded6ca379c3e + .quad 0xe340123dfdb7b29a + .quad 0x487b97e1a21ab291 + .quad 0xf9967d02fde6949e + .quad 0x780de72ec8d3de97 + + // 2^192 * 8 * B + + .quad 0x0ae28545089ae7bc + .quad 0x388ddecf1c7f4d06 + .quad 0x38ac15510a4811b8 + .quad 0x0eb28bf671928ce4 + .quad 0x671feaf300f42772 + .quad 0x8f72eb2a2a8c41aa + .quad 0x29a17fd797373292 + .quad 0x1defc6ad32b587a6 + .quad 0xaf5bbe1aef5195a7 + .quad 0x148c1277917b15ed + .quad 0x2991f7fb7ae5da2e + .quad 0x467d201bf8dd2867 + + // 2^196 * 1 * B + + .quad 0x7906ee72f7bd2e6b + .quad 0x05d270d6109abf4e + .quad 0x8d5cfe45b941a8a4 + .quad 0x44c218671c974287 + .quad 0x745f9d56296bc318 + .quad 0x993580d4d8152e65 + .quad 0xb0e5b13f5839e9ce + .quad 0x51fc2b28d43921c0 + .quad 0x1b8fd11795e2a98c + .quad 0x1c4e5ee12b6b6291 + .quad 0x5b30e7107424b572 + .quad 0x6e6b9de84c4f4ac6 + + // 2^196 * 2 * B + + .quad 0xdff25fce4b1de151 + .quad 0xd841c0c7e11c4025 + .quad 0x2554b3c854749c87 + .quad 0x2d292459908e0df9 + .quad 0x6b7c5f10f80cb088 + .quad 0x736b54dc56e42151 + .quad 0xc2b620a5c6ef99c4 + .quad 0x5f4c802cc3a06f42 + .quad 0x9b65c8f17d0752da + .quad 0x881ce338c77ee800 + .quad 0xc3b514f05b62f9e3 + .quad 0x66ed5dd5bec10d48 + + // 2^196 * 3 * B + + .quad 0x7d38a1c20bb2089d + .quad 0x808334e196ccd412 + .quad 0xc4a70b8c6c97d313 + .quad 0x2eacf8bc03007f20 + .quad 0xf0adf3c9cbca047d + .quad 0x81c3b2cbf4552f6b + .quad 0xcfda112d44735f93 + .quad 0x1f23a0c77e20048c + .quad 0xf235467be5bc1570 + .quad 0x03d2d9020dbab38c + .quad 0x27529aa2fcf9e09e + .quad 0x0840bef29d34bc50 + + // 2^196 * 4 * B + + .quad 0x796dfb35dc10b287 + .quad 0x27176bcd5c7ff29d + .quad 0x7f3d43e8c7b24905 + .quad 0x0304f5a191c54276 + .quad 0xcd54e06b7f37e4eb + .quad 0x8cc15f87f5e96cca + .quad 0xb8248bb0d3597dce + .quad 0x246affa06074400c + .quad 0x37d88e68fbe45321 + .quad 0x86097548c0d75032 + .quad 0x4e9b13ef894a0d35 + .quad 0x25a83cac5753d325 + + // 2^196 * 5 * B + + .quad 0x10222f48eed8165e + .quad 0x623fc1234b8bcf3a + .quad 0x1e145c09c221e8f0 + .quad 0x7ccfa59fca782630 + .quad 0x9f0f66293952b6e2 + .quad 0x33db5e0e0934267b + .quad 0xff45252bd609fedc + .quad 0x06be10f5c506e0c9 + .quad 0x1a9615a9b62a345f + .quad 0x22050c564a52fecc + .quad 0xa7a2788528bc0dfe + .quad 0x5e82770a1a1ee71d + + // 2^196 * 6 * B + + .quad 0x35425183ad896a5c + .quad 0xe8673afbe78d52f6 + .quad 0x2c66f25f92a35f64 + .quad 0x09d04f3b3b86b102 + .quad 0xe802e80a42339c74 + .quad 0x34175166a7fffae5 + .quad 0x34865d1f1c408cae + .quad 0x2cca982c605bc5ee + .quad 0xfd2d5d35197dbe6e + .quad 0x207c2eea8be4ffa3 + .quad 0x2613d8db325ae918 + .quad 0x7a325d1727741d3e + + // 2^196 * 7 * B + + .quad 0xd036b9bbd16dfde2 + .quad 0xa2055757c497a829 + .quad 0x8e6cc966a7f12667 + .quad 0x4d3b1a791239c180 + .quad 0xecd27d017e2a076a + .quad 0xd788689f1636495e + .quad 0x52a61af0919233e5 + .quad 0x2a479df17bb1ae64 + .quad 0x9e5eee8e33db2710 + .quad 0x189854ded6c43ca5 + .quad 0xa41c22c592718138 + .quad 0x27ad5538a43a5e9b + + // 2^196 * 8 * B + + .quad 0x2746dd4b15350d61 + .quad 0xd03fcbc8ee9521b7 + .quad 0xe86e365a138672ca + .quad 0x510e987f7e7d89e2 + .quad 0xcb5a7d638e47077c + .quad 0x8db7536120a1c059 + .quad 0x549e1e4d8bedfdcc + .quad 0x080153b7503b179d + .quad 0xdda69d930a3ed3e3 + .quad 0x3d386ef1cd60a722 + .quad 0xc817ad58bdaa4ee6 + .quad 0x23be8d554fe7372a + + // 2^200 * 1 * B + + .quad 0x95fe919a74ef4fad + .quad 0x3a827becf6a308a2 + .quad 0x964e01d309a47b01 + .quad 0x71c43c4f5ba3c797 + .quad 0xbc1ef4bd567ae7a9 + .quad 0x3f624cb2d64498bd + .quad 0xe41064d22c1f4ec8 + .quad 0x2ef9c5a5ba384001 + .quad 0xb6fd6df6fa9e74cd + .quad 0xf18278bce4af267a + .quad 0x8255b3d0f1ef990e + .quad 0x5a758ca390c5f293 + + // 2^200 * 2 * B + + .quad 0xa2b72710d9462495 + .quad 0x3aa8c6d2d57d5003 + .quad 0xe3d400bfa0b487ca + .quad 0x2dbae244b3eb72ec + .quad 0x8ce0918b1d61dc94 + .quad 0x8ded36469a813066 + .quad 0xd4e6a829afe8aad3 + .quad 0x0a738027f639d43f + .quad 0x980f4a2f57ffe1cc + .quad 0x00670d0de1839843 + .quad 0x105c3f4a49fb15fd + .quad 0x2698ca635126a69c + + // 2^200 * 3 * B + + .quad 0xe765318832b0ba78 + .quad 0x381831f7925cff8b + .quad 0x08a81b91a0291fcc + .quad 0x1fb43dcc49caeb07 + .quad 0x2e3d702f5e3dd90e + .quad 0x9e3f0918e4d25386 + .quad 0x5e773ef6024da96a + .quad 0x3c004b0c4afa3332 + .quad 0x9aa946ac06f4b82b + .quad 0x1ca284a5a806c4f3 + .quad 0x3ed3265fc6cd4787 + .quad 0x6b43fd01cd1fd217 + + // 2^200 * 4 * B + + .quad 0xc7a75d4b4697c544 + .quad 0x15fdf848df0fffbf + .quad 0x2868b9ebaa46785a + .quad 0x5a68d7105b52f714 + .quad 0xb5c742583e760ef3 + .quad 0x75dc52b9ee0ab990 + .quad 0xbf1427c2072b923f + .quad 0x73420b2d6ff0d9f0 + .quad 0xaf2cf6cb9e851e06 + .quad 0x8f593913c62238c4 + .quad 0xda8ab89699fbf373 + .quad 0x3db5632fea34bc9e + + // 2^200 * 5 * B + + .quad 0xf46eee2bf75dd9d8 + .quad 0x0d17b1f6396759a5 + .quad 0x1bf2d131499e7273 + .quad 0x04321adf49d75f13 + .quad 0x2e4990b1829825d5 + .quad 0xedeaeb873e9a8991 + .quad 0xeef03d394c704af8 + .quad 0x59197ea495df2b0e + .quad 0x04e16019e4e55aae + .quad 0xe77b437a7e2f92e9 + .quad 0xc7ce2dc16f159aa4 + .quad 0x45eafdc1f4d70cc0 + + // 2^200 * 6 * B + + .quad 0x698401858045d72b + .quad 0x4c22faa2cf2f0651 + .quad 0x941a36656b222dc6 + .quad 0x5a5eebc80362dade + .quad 0xb60e4624cfccb1ed + .quad 0x59dbc292bd5c0395 + .quad 0x31a09d1ddc0481c9 + .quad 0x3f73ceea5d56d940 + .quad 0xb7a7bfd10a4e8dc6 + .quad 0xbe57007e44c9b339 + .quad 0x60c1207f1557aefa + .quad 0x26058891266218db + + // 2^200 * 7 * B + + .quad 0x59f704a68360ff04 + .quad 0xc3d93fde7661e6f4 + .quad 0x831b2a7312873551 + .quad 0x54ad0c2e4e615d57 + .quad 0x4c818e3cc676e542 + .quad 0x5e422c9303ceccad + .quad 0xec07cccab4129f08 + .quad 0x0dedfa10b24443b8 + .quad 0xee3b67d5b82b522a + .quad 0x36f163469fa5c1eb + .quad 0xa5b4d2f26ec19fd3 + .quad 0x62ecb2baa77a9408 + + // 2^200 * 8 * B + + .quad 0xe5ed795261152b3d + .quad 0x4962357d0eddd7d1 + .quad 0x7482c8d0b96b4c71 + .quad 0x2e59f919a966d8be + .quad 0x92072836afb62874 + .quad 0x5fcd5e8579e104a5 + .quad 0x5aad01adc630a14a + .quad 0x61913d5075663f98 + .quad 0x0dc62d361a3231da + .quad 0xfa47583294200270 + .quad 0x02d801513f9594ce + .quad 0x3ddbc2a131c05d5c + + // 2^204 * 1 * B + + .quad 0x3f50a50a4ffb81ef + .quad 0xb1e035093bf420bf + .quad 0x9baa8e1cc6aa2cd0 + .quad 0x32239861fa237a40 + .quad 0xfb735ac2004a35d1 + .quad 0x31de0f433a6607c3 + .quad 0x7b8591bfc528d599 + .quad 0x55be9a25f5bb050c + .quad 0x0d005acd33db3dbf + .quad 0x0111b37c80ac35e2 + .quad 0x4892d66c6f88ebeb + .quad 0x770eadb16508fbcd + + // 2^204 * 2 * B + + .quad 0x8451f9e05e4e89dd + .quad 0xc06302ffbc793937 + .quad 0x5d22749556a6495c + .quad 0x09a6755ca05603fb + .quad 0xf1d3b681a05071b9 + .quad 0x2207659a3592ff3a + .quad 0x5f0169297881e40e + .quad 0x16bedd0e86ba374e + .quad 0x5ecccc4f2c2737b5 + .quad 0x43b79e0c2dccb703 + .quad 0x33e008bc4ec43df3 + .quad 0x06c1b840f07566c0 + + // 2^204 * 3 * B + + .quad 0x7688a5c6a388f877 + .quad 0x02a96c14deb2b6ac + .quad 0x64c9f3431b8c2af8 + .quad 0x3628435554a1eed6 + .quad 0x69ee9e7f9b02805c + .quad 0xcbff828a547d1640 + .quad 0x3d93a869b2430968 + .quad 0x46b7b8cd3fe26972 + .quad 0xe9812086fe7eebe0 + .quad 0x4cba6be72f515437 + .quad 0x1d04168b516efae9 + .quad 0x5ea1391043982cb9 + + // 2^204 * 4 * B + + .quad 0x49125c9cf4702ee1 + .quad 0x4520b71f8b25b32d + .quad 0x33193026501fef7e + .quad 0x656d8997c8d2eb2b + .quad 0x6f2b3be4d5d3b002 + .quad 0xafec33d96a09c880 + .quad 0x035f73a4a8bcc4cc + .quad 0x22c5b9284662198b + .quad 0xcb58c8fe433d8939 + .quad 0x89a0cb2e6a8d7e50 + .quad 0x79ca955309fbbe5a + .quad 0x0c626616cd7fc106 + + // 2^204 * 5 * B + + .quad 0x1ffeb80a4879b61f + .quad 0x6396726e4ada21ed + .quad 0x33c7b093368025ba + .quad 0x471aa0c6f3c31788 + .quad 0x8fdfc379fbf454b1 + .quad 0x45a5a970f1a4b771 + .quad 0xac921ef7bad35915 + .quad 0x42d088dca81c2192 + .quad 0x8fda0f37a0165199 + .quad 0x0adadb77c8a0e343 + .quad 0x20fbfdfcc875e820 + .quad 0x1cf2bea80c2206e7 + + // 2^204 * 6 * B + + .quad 0xc2ddf1deb36202ac + .quad 0x92a5fe09d2e27aa5 + .quad 0x7d1648f6fc09f1d3 + .quad 0x74c2cc0513bc4959 + .quad 0x982d6e1a02c0412f + .quad 0x90fa4c83db58e8fe + .quad 0x01c2f5bcdcb18bc0 + .quad 0x686e0c90216abc66 + .quad 0x1fadbadba54395a7 + .quad 0xb41a02a0ae0da66a + .quad 0xbf19f598bba37c07 + .quad 0x6a12b8acde48430d + + // 2^204 * 7 * B + + .quad 0xf8daea1f39d495d9 + .quad 0x592c190e525f1dfc + .quad 0xdb8cbd04c9991d1b + .quad 0x11f7fda3d88f0cb7 + .quad 0x793bdd801aaeeb5f + .quad 0x00a2a0aac1518871 + .quad 0xe8a373a31f2136b4 + .quad 0x48aab888fc91ef19 + .quad 0x041f7e925830f40e + .quad 0x002d6ca979661c06 + .quad 0x86dc9ff92b046a2e + .quad 0x760360928b0493d1 + + // 2^204 * 8 * B + + .quad 0x21bb41c6120cf9c6 + .quad 0xeab2aa12decda59b + .quad 0xc1a72d020aa48b34 + .quad 0x215d4d27e87d3b68 + .quad 0xb43108e5695a0b05 + .quad 0x6cb00ee8ad37a38b + .quad 0x5edad6eea3537381 + .quad 0x3f2602d4b6dc3224 + .quad 0xc8b247b65bcaf19c + .quad 0x49779dc3b1b2c652 + .quad 0x89a180bbd5ece2e2 + .quad 0x13f098a3cec8e039 + + // 2^208 * 1 * B + + .quad 0x9adc0ff9ce5ec54b + .quad 0x039c2a6b8c2f130d + .quad 0x028007c7f0f89515 + .quad 0x78968314ac04b36b + .quad 0xf3aa57a22796bb14 + .quad 0x883abab79b07da21 + .quad 0xe54be21831a0391c + .quad 0x5ee7fb38d83205f9 + .quad 0x538dfdcb41446a8e + .quad 0xa5acfda9434937f9 + .quad 0x46af908d263c8c78 + .quad 0x61d0633c9bca0d09 + + // 2^208 * 2 * B + + .quad 0x63744935ffdb2566 + .quad 0xc5bd6b89780b68bb + .quad 0x6f1b3280553eec03 + .quad 0x6e965fd847aed7f5 + .quad 0xada328bcf8fc73df + .quad 0xee84695da6f037fc + .quad 0x637fb4db38c2a909 + .quad 0x5b23ac2df8067bdc + .quad 0x9ad2b953ee80527b + .quad 0xe88f19aafade6d8d + .quad 0x0e711704150e82cf + .quad 0x79b9bbb9dd95dedc + + // 2^208 * 3 * B + + .quad 0xebb355406a3126c2 + .quad 0xd26383a868c8c393 + .quad 0x6c0c6429e5b97a82 + .quad 0x5065f158c9fd2147 + .quad 0xd1997dae8e9f7374 + .quad 0xa032a2f8cfbb0816 + .quad 0xcd6cba126d445f0a + .quad 0x1ba811460accb834 + .quad 0x708169fb0c429954 + .quad 0xe14600acd76ecf67 + .quad 0x2eaab98a70e645ba + .quad 0x3981f39e58a4faf2 + + // 2^208 * 4 * B + + .quad 0x18fb8a7559230a93 + .quad 0x1d168f6960e6f45d + .quad 0x3a85a94514a93cb5 + .quad 0x38dc083705acd0fd + .quad 0xc845dfa56de66fde + .quad 0xe152a5002c40483a + .quad 0xe9d2e163c7b4f632 + .quad 0x30f4452edcbc1b65 + .quad 0x856d2782c5759740 + .quad 0xfa134569f99cbecc + .quad 0x8844fc73c0ea4e71 + .quad 0x632d9a1a593f2469 + + // 2^208 * 5 * B + + .quad 0xf6bb6b15b807cba6 + .quad 0x1823c7dfbc54f0d7 + .quad 0xbb1d97036e29670b + .quad 0x0b24f48847ed4a57 + .quad 0xbf09fd11ed0c84a7 + .quad 0x63f071810d9f693a + .quad 0x21908c2d57cf8779 + .quad 0x3a5a7df28af64ba2 + .quad 0xdcdad4be511beac7 + .quad 0xa4538075ed26ccf2 + .quad 0xe19cff9f005f9a65 + .quad 0x34fcf74475481f63 + + // 2^208 * 6 * B + + .quad 0xc197e04c789767ca + .quad 0xb8714dcb38d9467d + .quad 0x55de888283f95fa8 + .quad 0x3d3bdc164dfa63f7 + .quad 0xa5bb1dab78cfaa98 + .quad 0x5ceda267190b72f2 + .quad 0x9309c9110a92608e + .quad 0x0119a3042fb374b0 + .quad 0x67a2d89ce8c2177d + .quad 0x669da5f66895d0c1 + .quad 0xf56598e5b282a2b0 + .quad 0x56c088f1ede20a73 + + // 2^208 * 7 * B + + .quad 0x336d3d1110a86e17 + .quad 0xd7f388320b75b2fa + .quad 0xf915337625072988 + .quad 0x09674c6b99108b87 + .quad 0x581b5fac24f38f02 + .quad 0xa90be9febae30cbd + .quad 0x9a2169028acf92f0 + .quad 0x038b7ea48359038f + .quad 0x9f4ef82199316ff8 + .quad 0x2f49d282eaa78d4f + .quad 0x0971a5ab5aef3174 + .quad 0x6e5e31025969eb65 + + // 2^208 * 8 * B + + .quad 0xb16c62f587e593fb + .quad 0x4999eddeca5d3e71 + .quad 0xb491c1e014cc3e6d + .quad 0x08f5114789a8dba8 + .quad 0x3304fb0e63066222 + .quad 0xfb35068987acba3f + .quad 0xbd1924778c1061a3 + .quad 0x3058ad43d1838620 + .quad 0x323c0ffde57663d0 + .quad 0x05c3df38a22ea610 + .quad 0xbdc78abdac994f9a + .quad 0x26549fa4efe3dc99 + + // 2^212 * 1 * B + + .quad 0x738b38d787ce8f89 + .quad 0xb62658e24179a88d + .quad 0x30738c9cf151316d + .quad 0x49128c7f727275c9 + .quad 0x04dbbc17f75396b9 + .quad 0x69e6a2d7d2f86746 + .quad 0xc6409d99f53eabc6 + .quad 0x606175f6332e25d2 + .quad 0x4021370ef540e7dd + .quad 0x0910d6f5a1f1d0a5 + .quad 0x4634aacd5b06b807 + .quad 0x6a39e6356944f235 + + // 2^212 * 2 * B + + .quad 0x96cd5640df90f3e7 + .quad 0x6c3a760edbfa25ea + .quad 0x24f3ef0959e33cc4 + .quad 0x42889e7e530d2e58 + .quad 0x1da1965774049e9d + .quad 0xfbcd6ea198fe352b + .quad 0xb1cbcd50cc5236a6 + .quad 0x1f5ec83d3f9846e2 + .quad 0x8efb23c3328ccb75 + .quad 0xaf42a207dd876ee9 + .quad 0x20fbdadc5dfae796 + .quad 0x241e246b06bf9f51 + + // 2^212 * 3 * B + + .quad 0x29e68e57ad6e98f6 + .quad 0x4c9260c80b462065 + .quad 0x3f00862ea51ebb4b + .quad 0x5bc2c77fb38d9097 + .quad 0x7eaafc9a6280bbb8 + .quad 0x22a70f12f403d809 + .quad 0x31ce40bb1bfc8d20 + .quad 0x2bc65635e8bd53ee + .quad 0xe8d5dc9fa96bad93 + .quad 0xe58fb17dde1947dc + .quad 0x681532ea65185fa3 + .quad 0x1fdd6c3b034a7830 + + // 2^212 * 4 * B + + .quad 0x0a64e28c55dc18fe + .quad 0xe3df9e993399ebdd + .quad 0x79ac432370e2e652 + .quad 0x35ff7fc33ae4cc0e + .quad 0x9c13a6a52dd8f7a9 + .quad 0x2dbb1f8c3efdcabf + .quad 0x961e32405e08f7b5 + .quad 0x48c8a121bbe6c9e5 + .quad 0xfc415a7c59646445 + .quad 0xd224b2d7c128b615 + .quad 0x6035c9c905fbb912 + .quad 0x42d7a91274429fab + + // 2^212 * 5 * B + + .quad 0x4e6213e3eaf72ed3 + .quad 0x6794981a43acd4e7 + .quad 0xff547cde6eb508cb + .quad 0x6fed19dd10fcb532 + .quad 0xa9a48947933da5bc + .quad 0x4a58920ec2e979ec + .quad 0x96d8800013e5ac4c + .quad 0x453692d74b48b147 + .quad 0xdd775d99a8559c6f + .quad 0xf42a2140df003e24 + .quad 0x5223e229da928a66 + .quad 0x063f46ba6d38f22c + + // 2^212 * 6 * B + + .quad 0xd2d242895f536694 + .quad 0xca33a2c542939b2c + .quad 0x986fada6c7ddb95c + .quad 0x5a152c042f712d5d + .quad 0x39843cb737346921 + .quad 0xa747fb0738c89447 + .quad 0xcb8d8031a245307e + .quad 0x67810f8e6d82f068 + .quad 0x3eeb8fbcd2287db4 + .quad 0x72c7d3a301a03e93 + .quad 0x5473e88cbd98265a + .quad 0x7324aa515921b403 + + // 2^212 * 7 * B + + .quad 0x857942f46c3cbe8e + .quad 0xa1d364b14730c046 + .quad 0x1c8ed914d23c41bf + .quad 0x0838e161eef6d5d2 + .quad 0xad23f6dae82354cb + .quad 0x6962502ab6571a6d + .quad 0x9b651636e38e37d1 + .quad 0x5cac5005d1a3312f + .quad 0x8cc154cce9e39904 + .quad 0x5b3a040b84de6846 + .quad 0xc4d8a61cb1be5d6e + .quad 0x40fb897bd8861f02 + + // 2^212 * 8 * B + + .quad 0x84c5aa9062de37a1 + .quad 0x421da5000d1d96e1 + .quad 0x788286306a9242d9 + .quad 0x3c5e464a690d10da + .quad 0xe57ed8475ab10761 + .quad 0x71435e206fd13746 + .quad 0x342f824ecd025632 + .quad 0x4b16281ea8791e7b + .quad 0xd1c101d50b813381 + .quad 0xdee60f1176ee6828 + .quad 0x0cb68893383f6409 + .quad 0x6183c565f6ff484a + + // 2^216 * 1 * B + + .quad 0x741d5a461e6bf9d6 + .quad 0x2305b3fc7777a581 + .quad 0xd45574a26474d3d9 + .quad 0x1926e1dc6401e0ff + .quad 0xdb468549af3f666e + .quad 0xd77fcf04f14a0ea5 + .quad 0x3df23ff7a4ba0c47 + .quad 0x3a10dfe132ce3c85 + .quad 0xe07f4e8aea17cea0 + .quad 0x2fd515463a1fc1fd + .quad 0x175322fd31f2c0f1 + .quad 0x1fa1d01d861e5d15 + + // 2^216 * 2 * B + + .quad 0xcc8055947d599832 + .quad 0x1e4656da37f15520 + .quad 0x99f6f7744e059320 + .quad 0x773563bc6a75cf33 + .quad 0x38dcac00d1df94ab + .quad 0x2e712bddd1080de9 + .quad 0x7f13e93efdd5e262 + .quad 0x73fced18ee9a01e5 + .quad 0x06b1e90863139cb3 + .quad 0xa493da67c5a03ecd + .quad 0x8d77cec8ad638932 + .quad 0x1f426b701b864f44 + + // 2^216 * 3 * B + + .quad 0xefc9264c41911c01 + .quad 0xf1a3b7b817a22c25 + .quad 0x5875da6bf30f1447 + .quad 0x4e1af5271d31b090 + .quad 0xf17e35c891a12552 + .quad 0xb76b8153575e9c76 + .quad 0xfa83406f0d9b723e + .quad 0x0b76bb1b3fa7e438 + .quad 0x08b8c1f97f92939b + .quad 0xbe6771cbd444ab6e + .quad 0x22e5646399bb8017 + .quad 0x7b6dd61eb772a955 + + // 2^216 * 4 * B + + .quad 0xb7adc1e850f33d92 + .quad 0x7998fa4f608cd5cf + .quad 0xad962dbd8dfc5bdb + .quad 0x703e9bceaf1d2f4f + .quad 0x5730abf9ab01d2c7 + .quad 0x16fb76dc40143b18 + .quad 0x866cbe65a0cbb281 + .quad 0x53fa9b659bff6afe + .quad 0x6c14c8e994885455 + .quad 0x843a5d6665aed4e5 + .quad 0x181bb73ebcd65af1 + .quad 0x398d93e5c4c61f50 + + // 2^216 * 5 * B + + .quad 0x1c4bd16733e248f3 + .quad 0xbd9e128715bf0a5f + .quad 0xd43f8cf0a10b0376 + .quad 0x53b09b5ddf191b13 + .quad 0xc3877c60d2e7e3f2 + .quad 0x3b34aaa030828bb1 + .quad 0x283e26e7739ef138 + .quad 0x699c9c9002c30577 + .quad 0xf306a7235946f1cc + .quad 0x921718b5cce5d97d + .quad 0x28cdd24781b4e975 + .quad 0x51caf30c6fcdd907 + + // 2^216 * 6 * B + + .quad 0xa60ba7427674e00a + .quad 0x630e8570a17a7bf3 + .quad 0x3758563dcf3324cc + .quad 0x5504aa292383fdaa + .quad 0x737af99a18ac54c7 + .quad 0x903378dcc51cb30f + .quad 0x2b89bc334ce10cc7 + .quad 0x12ae29c189f8e99a + .quad 0xa99ec0cb1f0d01cf + .quad 0x0dd1efcc3a34f7ae + .quad 0x55ca7521d09c4e22 + .quad 0x5fd14fe958eba5ea + + // 2^216 * 7 * B + + .quad 0xb5dc2ddf2845ab2c + .quad 0x069491b10a7fe993 + .quad 0x4daaf3d64002e346 + .quad 0x093ff26e586474d1 + .quad 0x3c42fe5ebf93cb8e + .quad 0xbedfa85136d4565f + .quad 0xe0f0859e884220e8 + .quad 0x7dd73f960725d128 + .quad 0xb10d24fe68059829 + .quad 0x75730672dbaf23e5 + .quad 0x1367253ab457ac29 + .quad 0x2f59bcbc86b470a4 + + // 2^216 * 8 * B + + .quad 0x83847d429917135f + .quad 0xad1b911f567d03d7 + .quad 0x7e7748d9be77aad1 + .quad 0x5458b42e2e51af4a + .quad 0x7041d560b691c301 + .quad 0x85201b3fadd7e71e + .quad 0x16c2e16311335585 + .quad 0x2aa55e3d010828b1 + .quad 0xed5192e60c07444f + .quad 0x42c54e2d74421d10 + .quad 0x352b4c82fdb5c864 + .quad 0x13e9004a8a768664 + + // 2^220 * 1 * B + + .quad 0xcbb5b5556c032bff + .quad 0xdf7191b729297a3a + .quad 0xc1ff7326aded81bb + .quad 0x71ade8bb68be03f5 + .quad 0x1e6284c5806b467c + .quad 0xc5f6997be75d607b + .quad 0x8b67d958b378d262 + .quad 0x3d88d66a81cd8b70 + .quad 0x8b767a93204ed789 + .quad 0x762fcacb9fa0ae2a + .quad 0x771febcc6dce4887 + .quad 0x343062158ff05fb3 + + // 2^220 * 2 * B + + .quad 0xe05da1a7e1f5bf49 + .quad 0x26457d6dd4736092 + .quad 0x77dcb07773cc32f6 + .quad 0x0a5d94969cdd5fcd + .quad 0xfce219072a7b31b4 + .quad 0x4d7adc75aa578016 + .quad 0x0ec276a687479324 + .quad 0x6d6d9d5d1fda4beb + .quad 0x22b1a58ae9b08183 + .quad 0xfd95d071c15c388b + .quad 0xa9812376850a0517 + .quad 0x33384cbabb7f335e + + // 2^220 * 3 * B + + .quad 0x3c6fa2680ca2c7b5 + .quad 0x1b5082046fb64fda + .quad 0xeb53349c5431d6de + .quad 0x5278b38f6b879c89 + .quad 0x33bc627a26218b8d + .quad 0xea80b21fc7a80c61 + .quad 0x9458b12b173e9ee6 + .quad 0x076247be0e2f3059 + .quad 0x52e105f61416375a + .quad 0xec97af3685abeba4 + .quad 0x26e6b50623a67c36 + .quad 0x5cf0e856f3d4fb01 + + // 2^220 * 4 * B + + .quad 0xf6c968731ae8cab4 + .quad 0x5e20741ecb4f92c5 + .quad 0x2da53be58ccdbc3e + .quad 0x2dddfea269970df7 + .quad 0xbeaece313db342a8 + .quad 0xcba3635b842db7ee + .quad 0xe88c6620817f13ef + .quad 0x1b9438aa4e76d5c6 + .quad 0x8a50777e166f031a + .quad 0x067b39f10fb7a328 + .quad 0x1925c9a6010fbd76 + .quad 0x6df9b575cc740905 + + // 2^220 * 5 * B + + .quad 0x42c1192927f6bdcf + .quad 0x8f91917a403d61ca + .quad 0xdc1c5a668b9e1f61 + .quad 0x1596047804ec0f8d + .quad 0xecdfc35b48cade41 + .quad 0x6a88471fb2328270 + .quad 0x740a4a2440a01b6a + .quad 0x471e5796003b5f29 + .quad 0xda96bbb3aced37ac + .quad 0x7a2423b5e9208cea + .quad 0x24cc5c3038aebae2 + .quad 0x50c356afdc5dae2f + + // 2^220 * 6 * B + + .quad 0x09dcbf4341c30318 + .quad 0xeeba061183181dce + .quad 0xc179c0cedc1e29a1 + .quad 0x1dbf7b89073f35b0 + .quad 0xcfed9cdf1b31b964 + .quad 0xf486a9858ca51af3 + .quad 0x14897265ea8c1f84 + .quad 0x784a53dd932acc00 + .quad 0x2d99f9df14fc4920 + .quad 0x76ccb60cc4499fe5 + .quad 0xa4132cbbe5cf0003 + .quad 0x3f93d82354f000ea + + // 2^220 * 7 * B + + .quad 0x8183e7689e04ce85 + .quad 0x678fb71e04465341 + .quad 0xad92058f6688edac + .quad 0x5da350d3532b099a + .quad 0xeaac12d179e14978 + .quad 0xff923ff3bbebff5e + .quad 0x4af663e40663ce27 + .quad 0x0fd381a811a5f5ff + .quad 0xf256aceca436df54 + .quad 0x108b6168ae69d6e8 + .quad 0x20d986cb6b5d036c + .quad 0x655957b9fee2af50 + + // 2^220 * 8 * B + + .quad 0xaea8b07fa902030f + .quad 0xf88c766af463d143 + .quad 0x15b083663c787a60 + .quad 0x08eab1148267a4a8 + .quad 0xbdc1409bd002d0ac + .quad 0x66660245b5ccd9a6 + .quad 0x82317dc4fade85ec + .quad 0x02fe934b6ad7df0d + .quad 0xef5cf100cfb7ea74 + .quad 0x22897633a1cb42ac + .quad 0xd4ce0c54cef285e2 + .quad 0x30408c048a146a55 + + // 2^224 * 1 * B + + .quad 0x739d8845832fcedb + .quad 0xfa38d6c9ae6bf863 + .quad 0x32bc0dcab74ffef7 + .quad 0x73937e8814bce45e + .quad 0xbb2e00c9193b877f + .quad 0xece3a890e0dc506b + .quad 0xecf3b7c036de649f + .quad 0x5f46040898de9e1a + .quad 0xb9037116297bf48d + .quad 0xa9d13b22d4f06834 + .quad 0xe19715574696bdc6 + .quad 0x2cf8a4e891d5e835 + + // 2^224 * 2 * B + + .quad 0x6d93fd8707110f67 + .quad 0xdd4c09d37c38b549 + .quad 0x7cb16a4cc2736a86 + .quad 0x2049bd6e58252a09 + .quad 0x2cb5487e17d06ba2 + .quad 0x24d2381c3950196b + .quad 0xd7659c8185978a30 + .quad 0x7a6f7f2891d6a4f6 + .quad 0x7d09fd8d6a9aef49 + .quad 0xf0ee60be5b3db90b + .quad 0x4c21b52c519ebfd4 + .quad 0x6011aadfc545941d + + // 2^224 * 3 * B + + .quad 0x5f67926dcf95f83c + .quad 0x7c7e856171289071 + .quad 0xd6a1e7f3998f7a5b + .quad 0x6fc5cc1b0b62f9e0 + .quad 0x63ded0c802cbf890 + .quad 0xfbd098ca0dff6aaa + .quad 0x624d0afdb9b6ed99 + .quad 0x69ce18b779340b1e + .quad 0xd1ef5528b29879cb + .quad 0xdd1aae3cd47e9092 + .quad 0x127e0442189f2352 + .quad 0x15596b3ae57101f1 + + // 2^224 * 4 * B + + .quad 0x462739d23f9179a2 + .quad 0xff83123197d6ddcf + .quad 0x1307deb553f2148a + .quad 0x0d2237687b5f4dda + .quad 0x09ff31167e5124ca + .quad 0x0be4158bd9c745df + .quad 0x292b7d227ef556e5 + .quad 0x3aa4e241afb6d138 + .quad 0x2cc138bf2a3305f5 + .quad 0x48583f8fa2e926c3 + .quad 0x083ab1a25549d2eb + .quad 0x32fcaa6e4687a36c + + // 2^224 * 5 * B + + .quad 0x7bc56e8dc57d9af5 + .quad 0x3e0bd2ed9df0bdf2 + .quad 0xaac014de22efe4a3 + .quad 0x4627e9cefebd6a5c + .quad 0x3207a4732787ccdf + .quad 0x17e31908f213e3f8 + .quad 0xd5b2ecd7f60d964e + .quad 0x746f6336c2600be9 + .quad 0x3f4af345ab6c971c + .quad 0xe288eb729943731f + .quad 0x33596a8a0344186d + .quad 0x7b4917007ed66293 + + // 2^224 * 6 * B + + .quad 0x2d85fb5cab84b064 + .quad 0x497810d289f3bc14 + .quad 0x476adc447b15ce0c + .quad 0x122ba376f844fd7b + .quad 0x54341b28dd53a2dd + .quad 0xaa17905bdf42fc3f + .quad 0x0ff592d94dd2f8f4 + .quad 0x1d03620fe08cd37d + .quad 0xc20232cda2b4e554 + .quad 0x9ed0fd42115d187f + .quad 0x2eabb4be7dd479d9 + .quad 0x02c70bf52b68ec4c + + // 2^224 * 7 * B + + .quad 0xa287ec4b5d0b2fbb + .quad 0x415c5790074882ca + .quad 0xe044a61ec1d0815c + .quad 0x26334f0a409ef5e0 + .quad 0xace532bf458d72e1 + .quad 0x5be768e07cb73cb5 + .quad 0x56cf7d94ee8bbde7 + .quad 0x6b0697e3feb43a03 + .quad 0xb6c8f04adf62a3c0 + .quad 0x3ef000ef076da45d + .quad 0x9c9cb95849f0d2a9 + .quad 0x1cc37f43441b2fae + + // 2^224 * 8 * B + + .quad 0x508f565a5cc7324f + .quad 0xd061c4c0e506a922 + .quad 0xfb18abdb5c45ac19 + .quad 0x6c6809c10380314a + .quad 0xd76656f1c9ceaeb9 + .quad 0x1c5b15f818e5656a + .quad 0x26e72832844c2334 + .quad 0x3a346f772f196838 + .quad 0xd2d55112e2da6ac8 + .quad 0xe9bd0331b1e851ed + .quad 0x960746dd8ec67262 + .quad 0x05911b9f6ef7c5d0 + + // 2^228 * 1 * B + + .quad 0xe9dcd756b637ff2d + .quad 0xec4c348fc987f0c4 + .quad 0xced59285f3fbc7b7 + .quad 0x3305354793e1ea87 + .quad 0x01c18980c5fe9f94 + .quad 0xcd656769716fd5c8 + .quad 0x816045c3d195a086 + .quad 0x6e2b7f3266cc7982 + .quad 0xcc802468f7c3568f + .quad 0x9de9ba8219974cb3 + .quad 0xabb7229cb5b81360 + .quad 0x44e2017a6fbeba62 + + // 2^228 * 2 * B + + .quad 0xc4c2a74354dab774 + .quad 0x8e5d4c3c4eaf031a + .quad 0xb76c23d242838f17 + .quad 0x749a098f68dce4ea + .quad 0x87f82cf3b6ca6ecd + .quad 0x580f893e18f4a0c2 + .quad 0x058930072604e557 + .quad 0x6cab6ac256d19c1d + .quad 0xdcdfe0a02cc1de60 + .quad 0x032665ff51c5575b + .quad 0x2c0c32f1073abeeb + .quad 0x6a882014cd7b8606 + + // 2^228 * 3 * B + + .quad 0xa52a92fea4747fb5 + .quad 0xdc12a4491fa5ab89 + .quad 0xd82da94bb847a4ce + .quad 0x4d77edce9512cc4e + .quad 0xd111d17caf4feb6e + .quad 0x050bba42b33aa4a3 + .quad 0x17514c3ceeb46c30 + .quad 0x54bedb8b1bc27d75 + .quad 0x77c8e14577e2189c + .quad 0xa3e46f6aff99c445 + .quad 0x3144dfc86d335343 + .quad 0x3a96559e7c4216a9 + + // 2^228 * 4 * B + + .quad 0x12550d37f42ad2ee + .quad 0x8b78e00498a1fbf5 + .quad 0x5d53078233894cb2 + .quad 0x02c84e4e3e498d0c + .quad 0x4493896880baaa52 + .quad 0x4c98afc4f285940e + .quad 0xef4aa79ba45448b6 + .quad 0x5278c510a57aae7f + .quad 0xa54dd074294c0b94 + .quad 0xf55d46b8df18ffb6 + .quad 0xf06fecc58dae8366 + .quad 0x588657668190d165 + + // 2^228 * 5 * B + + .quad 0xd47712311aef7117 + .quad 0x50343101229e92c7 + .quad 0x7a95e1849d159b97 + .quad 0x2449959b8b5d29c9 + .quad 0xbf5834f03de25cc3 + .quad 0xb887c8aed6815496 + .quad 0x5105221a9481e892 + .quad 0x6760ed19f7723f93 + .quad 0x669ba3b7ac35e160 + .quad 0x2eccf73fba842056 + .quad 0x1aec1f17c0804f07 + .quad 0x0d96bc031856f4e7 + + // 2^228 * 6 * B + + .quad 0x3318be7775c52d82 + .quad 0x4cb764b554d0aab9 + .quad 0xabcf3d27cc773d91 + .quad 0x3bf4d1848123288a + .quad 0xb1d534b0cc7505e1 + .quad 0x32cd003416c35288 + .quad 0xcb36a5800762c29d + .quad 0x5bfe69b9237a0bf8 + .quad 0x183eab7e78a151ab + .quad 0xbbe990c999093763 + .quad 0xff717d6e4ac7e335 + .quad 0x4c5cddb325f39f88 + + // 2^228 * 7 * B + + .quad 0xc0f6b74d6190a6eb + .quad 0x20ea81a42db8f4e4 + .quad 0xa8bd6f7d97315760 + .quad 0x33b1d60262ac7c21 + .quad 0x57750967e7a9f902 + .quad 0x2c37fdfc4f5b467e + .quad 0xb261663a3177ba46 + .quad 0x3a375e78dc2d532b + .quad 0x8141e72f2d4dddea + .quad 0xe6eafe9862c607c8 + .quad 0x23c28458573cafd0 + .quad 0x46b9476f4ff97346 + + // 2^228 * 8 * B + + .quad 0x0c1ffea44f901e5c + .quad 0x2b0b6fb72184b782 + .quad 0xe587ff910114db88 + .quad 0x37130f364785a142 + .quad 0x1215505c0d58359f + .quad 0x2a2013c7fc28c46b + .quad 0x24a0a1af89ea664e + .quad 0x4400b638a1130e1f + .quad 0x3a01b76496ed19c3 + .quad 0x31e00ab0ed327230 + .quad 0x520a885783ca15b1 + .quad 0x06aab9875accbec7 + + // 2^232 * 1 * B + + .quad 0xc1339983f5df0ebb + .quad 0xc0f3758f512c4cac + .quad 0x2cf1130a0bb398e1 + .quad 0x6b3cecf9aa270c62 + .quad 0x5349acf3512eeaef + .quad 0x20c141d31cc1cb49 + .quad 0x24180c07a99a688d + .quad 0x555ef9d1c64b2d17 + .quad 0x36a770ba3b73bd08 + .quad 0x624aef08a3afbf0c + .quad 0x5737ff98b40946f2 + .quad 0x675f4de13381749d + + // 2^232 * 2 * B + + .quad 0x0e2c52036b1782fc + .quad 0x64816c816cad83b4 + .quad 0xd0dcbdd96964073e + .quad 0x13d99df70164c520 + .quad 0xa12ff6d93bdab31d + .quad 0x0725d80f9d652dfe + .quad 0x019c4ff39abe9487 + .quad 0x60f450b882cd3c43 + .quad 0x014b5ec321e5c0ca + .quad 0x4fcb69c9d719bfa2 + .quad 0x4e5f1c18750023a0 + .quad 0x1c06de9e55edac80 + + // 2^232 * 3 * B + + .quad 0x990f7ad6a33ec4e2 + .quad 0x6608f938be2ee08e + .quad 0x9ca143c563284515 + .quad 0x4cf38a1fec2db60d + .quad 0xffd52b40ff6d69aa + .quad 0x34530b18dc4049bb + .quad 0x5e4a5c2fa34d9897 + .quad 0x78096f8e7d32ba2d + .quad 0xa0aaaa650dfa5ce7 + .quad 0xf9c49e2a48b5478c + .quad 0x4f09cc7d7003725b + .quad 0x373cad3a26091abe + + // 2^232 * 4 * B + + .quad 0xb294634d82c9f57c + .quad 0x1fcbfde124934536 + .quad 0x9e9c4db3418cdb5a + .quad 0x0040f3d9454419fc + .quad 0xf1bea8fb89ddbbad + .quad 0x3bcb2cbc61aeaecb + .quad 0x8f58a7bb1f9b8d9d + .quad 0x21547eda5112a686 + .quad 0xdefde939fd5986d3 + .quad 0xf4272c89510a380c + .quad 0xb72ba407bb3119b9 + .quad 0x63550a334a254df4 + + // 2^232 * 5 * B + + .quad 0x6507d6edb569cf37 + .quad 0x178429b00ca52ee1 + .quad 0xea7c0090eb6bd65d + .quad 0x3eea62c7daf78f51 + .quad 0x9bba584572547b49 + .quad 0xf305c6fae2c408e0 + .quad 0x60e8fa69c734f18d + .quad 0x39a92bafaa7d767a + .quad 0x9d24c713e693274e + .quad 0x5f63857768dbd375 + .quad 0x70525560eb8ab39a + .quad 0x68436a0665c9c4cd + + // 2^232 * 6 * B + + .quad 0xbc0235e8202f3f27 + .quad 0xc75c00e264f975b0 + .quad 0x91a4e9d5a38c2416 + .quad 0x17b6e7f68ab789f9 + .quad 0x1e56d317e820107c + .quad 0xc5266844840ae965 + .quad 0xc1e0a1c6320ffc7a + .quad 0x5373669c91611472 + .quad 0x5d2814ab9a0e5257 + .quad 0x908f2084c9cab3fc + .quad 0xafcaf5885b2d1eca + .quad 0x1cb4b5a678f87d11 + + // 2^232 * 7 * B + + .quad 0xb664c06b394afc6c + .quad 0x0c88de2498da5fb1 + .quad 0x4f8d03164bcad834 + .quad 0x330bca78de7434a2 + .quad 0x6b74aa62a2a007e7 + .quad 0xf311e0b0f071c7b1 + .quad 0x5707e438000be223 + .quad 0x2dc0fd2d82ef6eac + .quad 0x982eff841119744e + .quad 0xf9695e962b074724 + .quad 0xc58ac14fbfc953fb + .quad 0x3c31be1b369f1cf5 + + // 2^232 * 8 * B + + .quad 0xb0f4864d08948aee + .quad 0x07dc19ee91ba1c6f + .quad 0x7975cdaea6aca158 + .quad 0x330b61134262d4bb + .quad 0xc168bc93f9cb4272 + .quad 0xaeb8711fc7cedb98 + .quad 0x7f0e52aa34ac8d7a + .quad 0x41cec1097e7d55bb + .quad 0xf79619d7a26d808a + .quad 0xbb1fd49e1d9e156d + .quad 0x73d7c36cdba1df27 + .quad 0x26b44cd91f28777d + + // 2^236 * 1 * B + + .quad 0x300a9035393aa6d8 + .quad 0x2b501131a12bb1cd + .quad 0x7b1ff677f093c222 + .quad 0x4309c1f8cab82bad + .quad 0xaf44842db0285f37 + .quad 0x8753189047efc8df + .quad 0x9574e091f820979a + .quad 0x0e378d6069615579 + .quad 0xd9fa917183075a55 + .quad 0x4bdb5ad26b009fdc + .quad 0x7829ad2cd63def0e + .quad 0x078fc54975fd3877 + + // 2^236 * 2 * B + + .quad 0x87dfbd1428878f2d + .quad 0x134636dd1e9421a1 + .quad 0x4f17c951257341a3 + .quad 0x5df98d4bad296cb8 + .quad 0xe2004b5bb833a98a + .quad 0x44775dec2d4c3330 + .quad 0x3aa244067eace913 + .quad 0x272630e3d58e00a9 + .quad 0xf3678fd0ecc90b54 + .quad 0xf001459b12043599 + .quad 0x26725fbc3758b89b + .quad 0x4325e4aa73a719ae + + // 2^236 * 3 * B + + .quad 0x657dc6ef433c3493 + .quad 0x65375e9f80dbf8c3 + .quad 0x47fd2d465b372dae + .quad 0x4966ab79796e7947 + .quad 0xed24629acf69f59d + .quad 0x2a4a1ccedd5abbf4 + .quad 0x3535ca1f56b2d67b + .quad 0x5d8c68d043b1b42d + .quad 0xee332d4de3b42b0a + .quad 0xd84e5a2b16a4601c + .quad 0x78243877078ba3e4 + .quad 0x77ed1eb4184ee437 + + // 2^236 * 4 * B + + .quad 0xbfd4e13f201839a0 + .quad 0xaeefffe23e3df161 + .quad 0xb65b04f06b5d1fe3 + .quad 0x52e085fb2b62fbc0 + .quad 0x185d43f89e92ed1a + .quad 0xb04a1eeafe4719c6 + .quad 0x499fbe88a6f03f4f + .quad 0x5d8b0d2f3c859bdd + .quad 0x124079eaa54cf2ba + .quad 0xd72465eb001b26e7 + .quad 0x6843bcfdc97af7fd + .quad 0x0524b42b55eacd02 + + // 2^236 * 5 * B + + .quad 0xfd0d5dbee45447b0 + .quad 0x6cec351a092005ee + .quad 0x99a47844567579cb + .quad 0x59d242a216e7fa45 + .quad 0xbc18dcad9b829eac + .quad 0x23ae7d28b5f579d0 + .quad 0xc346122a69384233 + .quad 0x1a6110b2e7d4ac89 + .quad 0x4f833f6ae66997ac + .quad 0x6849762a361839a4 + .quad 0x6985dec1970ab525 + .quad 0x53045e89dcb1f546 + + // 2^236 * 6 * B + + .quad 0xcb8bb346d75353db + .quad 0xfcfcb24bae511e22 + .quad 0xcba48d40d50ae6ef + .quad 0x26e3bae5f4f7cb5d + .quad 0x84da3cde8d45fe12 + .quad 0xbd42c218e444e2d2 + .quad 0xa85196781f7e3598 + .quad 0x7642c93f5616e2b2 + .quad 0x2323daa74595f8e4 + .quad 0xde688c8b857abeb4 + .quad 0x3fc48e961c59326e + .quad 0x0b2e73ca15c9b8ba + + // 2^236 * 7 * B + + .quad 0xd6bb4428c17f5026 + .quad 0x9eb27223fb5a9ca7 + .quad 0xe37ba5031919c644 + .quad 0x21ce380db59a6602 + .quad 0x0e3fbfaf79c03a55 + .quad 0x3077af054cbb5acf + .quad 0xd5c55245db3de39f + .quad 0x015e68c1476a4af7 + .quad 0xc1d5285220066a38 + .quad 0x95603e523570aef3 + .quad 0x832659a7226b8a4d + .quad 0x5dd689091f8eedc9 + + // 2^236 * 8 * B + + .quad 0xcbac84debfd3c856 + .quad 0x1624c348b35ff244 + .quad 0xb7f88dca5d9cad07 + .quad 0x3b0e574da2c2ebe8 + .quad 0x1d022591a5313084 + .quad 0xca2d4aaed6270872 + .quad 0x86a12b852f0bfd20 + .quad 0x56e6c439ad7da748 + .quad 0xc704ff4942bdbae6 + .quad 0x5e21ade2b2de1f79 + .quad 0xe95db3f35652fad8 + .quad 0x0822b5378f08ebc1 + + // 2^240 * 1 * B + + .quad 0x51f048478f387475 + .quad 0xb25dbcf49cbecb3c + .quad 0x9aab1244d99f2055 + .quad 0x2c709e6c1c10a5d6 + .quad 0xe1b7f29362730383 + .quad 0x4b5279ffebca8a2c + .quad 0xdafc778abfd41314 + .quad 0x7deb10149c72610f + .quad 0xcb62af6a8766ee7a + .quad 0x66cbec045553cd0e + .quad 0x588001380f0be4b5 + .quad 0x08e68e9ff62ce2ea + + // 2^240 * 2 * B + + .quad 0x34ad500a4bc130ad + .quad 0x8d38db493d0bd49c + .quad 0xa25c3d98500a89be + .quad 0x2f1f3f87eeba3b09 + .quad 0x2f2d09d50ab8f2f9 + .quad 0xacb9218dc55923df + .quad 0x4a8f342673766cb9 + .quad 0x4cb13bd738f719f5 + .quad 0xf7848c75e515b64a + .quad 0xa59501badb4a9038 + .quad 0xc20d313f3f751b50 + .quad 0x19a1e353c0ae2ee8 + + // 2^240 * 3 * B + + .quad 0x7d1c7560bafa05c3 + .quad 0xb3e1a0a0c6e55e61 + .quad 0xe3529718c0d66473 + .quad 0x41546b11c20c3486 + .quad 0xb42172cdd596bdbd + .quad 0x93e0454398eefc40 + .quad 0x9fb15347b44109b5 + .quad 0x736bd3990266ae34 + .quad 0x85532d509334b3b4 + .quad 0x46fd114b60816573 + .quad 0xcc5f5f30425c8375 + .quad 0x412295a2b87fab5c + + // 2^240 * 4 * B + + .quad 0x19c99b88f57ed6e9 + .quad 0x5393cb266df8c825 + .quad 0x5cee3213b30ad273 + .quad 0x14e153ebb52d2e34 + .quad 0x2e655261e293eac6 + .quad 0x845a92032133acdb + .quad 0x460975cb7900996b + .quad 0x0760bb8d195add80 + .quad 0x413e1a17cde6818a + .quad 0x57156da9ed69a084 + .quad 0x2cbf268f46caccb1 + .quad 0x6b34be9bc33ac5f2 + + // 2^240 * 5 * B + + .quad 0xf3df2f643a78c0b2 + .quad 0x4c3e971ef22e027c + .quad 0xec7d1c5e49c1b5a3 + .quad 0x2012c18f0922dd2d + .quad 0x11fc69656571f2d3 + .quad 0xc6c9e845530e737a + .quad 0xe33ae7a2d4fe5035 + .quad 0x01b9c7b62e6dd30b + .quad 0x880b55e55ac89d29 + .quad 0x1483241f45a0a763 + .quad 0x3d36efdfc2e76c1f + .quad 0x08af5b784e4bade8 + + // 2^240 * 6 * B + + .quad 0x283499dc881f2533 + .quad 0x9d0525da779323b6 + .quad 0x897addfb673441f4 + .quad 0x32b79d71163a168d + .quad 0xe27314d289cc2c4b + .quad 0x4be4bd11a287178d + .quad 0x18d528d6fa3364ce + .quad 0x6423c1d5afd9826e + .quad 0xcc85f8d9edfcb36a + .quad 0x22bcc28f3746e5f9 + .quad 0xe49de338f9e5d3cd + .quad 0x480a5efbc13e2dcc + + // 2^240 * 7 * B + + .quad 0x0b51e70b01622071 + .quad 0x06b505cf8b1dafc5 + .quad 0x2c6bb061ef5aabcd + .quad 0x47aa27600cb7bf31 + .quad 0xb6614ce442ce221f + .quad 0x6e199dcc4c053928 + .quad 0x663fb4a4dc1cbe03 + .quad 0x24b31d47691c8e06 + .quad 0x2a541eedc015f8c3 + .quad 0x11a4fe7e7c693f7c + .quad 0xf0af66134ea278d6 + .quad 0x545b585d14dda094 + + // 2^240 * 8 * B + + .quad 0x67bf275ea0d43a0f + .quad 0xade68e34089beebe + .quad 0x4289134cd479e72e + .quad 0x0f62f9c332ba5454 + .quad 0x6204e4d0e3b321e1 + .quad 0x3baa637a28ff1e95 + .quad 0x0b0ccffd5b99bd9e + .quad 0x4d22dc3e64c8d071 + .quad 0xfcb46589d63b5f39 + .quad 0x5cae6a3f57cbcf61 + .quad 0xfebac2d2953afa05 + .quad 0x1c0fa01a36371436 + + // 2^244 * 1 * B + + .quad 0xe7547449bc7cd692 + .quad 0x0f9abeaae6f73ddf + .quad 0x4af01ca700837e29 + .quad 0x63ab1b5d3f1bc183 + .quad 0xc11ee5e854c53fae + .quad 0x6a0b06c12b4f3ff4 + .quad 0x33540f80e0b67a72 + .quad 0x15f18fc3cd07e3ef + .quad 0x32750763b028f48c + .quad 0x06020740556a065f + .quad 0xd53bd812c3495b58 + .quad 0x08706c9b865f508d + + // 2^244 * 2 * B + + .quad 0xf37ca2ab3d343dff + .quad 0x1a8c6a2d80abc617 + .quad 0x8e49e035d4ccffca + .quad 0x48b46beebaa1d1b9 + .quad 0xcc991b4138b41246 + .quad 0x243b9c526f9ac26b + .quad 0xb9ef494db7cbabbd + .quad 0x5fba433dd082ed00 + .quad 0x9c49e355c9941ad0 + .quad 0xb9734ade74498f84 + .quad 0x41c3fed066663e5c + .quad 0x0ecfedf8e8e710b3 + + // 2^244 * 3 * B + + .quad 0x76430f9f9cd470d9 + .quad 0xb62acc9ba42f6008 + .quad 0x1898297c59adad5e + .quad 0x7789dd2db78c5080 + .quad 0x744f7463e9403762 + .quad 0xf79a8dee8dfcc9c9 + .quad 0x163a649655e4cde3 + .quad 0x3b61788db284f435 + .quad 0xb22228190d6ef6b2 + .quad 0xa94a66b246ce4bfa + .quad 0x46c1a77a4f0b6cc7 + .quad 0x4236ccffeb7338cf + + // 2^244 * 4 * B + + .quad 0x8497404d0d55e274 + .quad 0x6c6663d9c4ad2b53 + .quad 0xec2fb0d9ada95734 + .quad 0x2617e120cdb8f73c + .quad 0x3bd82dbfda777df6 + .quad 0x71b177cc0b98369e + .quad 0x1d0e8463850c3699 + .quad 0x5a71945b48e2d1f1 + .quad 0x6f203dd5405b4b42 + .quad 0x327ec60410b24509 + .quad 0x9c347230ac2a8846 + .quad 0x77de29fc11ffeb6a + + // 2^244 * 5 * B + + .quad 0xb0ac57c983b778a8 + .quad 0x53cdcca9d7fe912c + .quad 0x61c2b854ff1f59dc + .quad 0x3a1a2cf0f0de7dac + .quad 0x835e138fecced2ca + .quad 0x8c9eaf13ea963b9a + .quad 0xc95fbfc0b2160ea6 + .quad 0x575e66f3ad877892 + .quad 0x99803a27c88fcb3a + .quad 0x345a6789275ec0b0 + .quad 0x459789d0ff6c2be5 + .quad 0x62f882651e70a8b2 + + // 2^244 * 6 * B + + .quad 0x085ae2c759ff1be4 + .quad 0x149145c93b0e40b7 + .quad 0xc467e7fa7ff27379 + .quad 0x4eeecf0ad5c73a95 + .quad 0x6d822986698a19e0 + .quad 0xdc9821e174d78a71 + .quad 0x41a85f31f6cb1f47 + .quad 0x352721c2bcda9c51 + .quad 0x48329952213fc985 + .quad 0x1087cf0d368a1746 + .quad 0x8e5261b166c15aa5 + .quad 0x2d5b2d842ed24c21 + + // 2^244 * 7 * B + + .quad 0x02cfebd9ebd3ded1 + .quad 0xd45b217739021974 + .quad 0x7576f813fe30a1b7 + .quad 0x5691b6f9a34ef6c2 + .quad 0x5eb7d13d196ac533 + .quad 0x377234ecdb80be2b + .quad 0xe144cffc7cf5ae24 + .quad 0x5226bcf9c441acec + .quad 0x79ee6c7223e5b547 + .quad 0x6f5f50768330d679 + .quad 0xed73e1e96d8adce9 + .quad 0x27c3da1e1d8ccc03 + + // 2^244 * 8 * B + + .quad 0x7eb9efb23fe24c74 + .quad 0x3e50f49f1651be01 + .quad 0x3ea732dc21858dea + .quad 0x17377bd75bb810f9 + .quad 0x28302e71630ef9f6 + .quad 0xc2d4a2032b64cee0 + .quad 0x090820304b6292be + .quad 0x5fca747aa82adf18 + .quad 0x232a03c35c258ea5 + .quad 0x86f23a2c6bcb0cf1 + .quad 0x3dad8d0d2e442166 + .quad 0x04a8933cab76862b + + // 2^248 * 1 * B + + .quad 0xd2c604b622943dff + .quad 0xbc8cbece44cfb3a0 + .quad 0x5d254ff397808678 + .quad 0x0fa3614f3b1ca6bf + .quad 0x69082b0e8c936a50 + .quad 0xf9c9a035c1dac5b6 + .quad 0x6fb73e54c4dfb634 + .quad 0x4005419b1d2bc140 + .quad 0xa003febdb9be82f0 + .quad 0x2089c1af3a44ac90 + .quad 0xf8499f911954fa8e + .quad 0x1fba218aef40ab42 + + // 2^248 * 2 * B + + .quad 0xab549448fac8f53e + .quad 0x81f6e89a7ba63741 + .quad 0x74fd6c7d6c2b5e01 + .quad 0x392e3acaa8c86e42 + .quad 0x4f3e57043e7b0194 + .quad 0xa81d3eee08daaf7f + .quad 0xc839c6ab99dcdef1 + .quad 0x6c535d13ff7761d5 + .quad 0x4cbd34e93e8a35af + .quad 0x2e0781445887e816 + .quad 0x19319c76f29ab0ab + .quad 0x25e17fe4d50ac13b + + // 2^248 * 3 * B + + .quad 0x0a289bd71e04f676 + .quad 0x208e1c52d6420f95 + .quad 0x5186d8b034691fab + .quad 0x255751442a9fb351 + .quad 0x915f7ff576f121a7 + .quad 0xc34a32272fcd87e3 + .quad 0xccba2fde4d1be526 + .quad 0x6bba828f8969899b + .quad 0xe2d1bc6690fe3901 + .quad 0x4cb54a18a0997ad5 + .quad 0x971d6914af8460d4 + .quad 0x559d504f7f6b7be4 + + // 2^248 * 4 * B + + .quad 0xa7738378b3eb54d5 + .quad 0x1d69d366a5553c7c + .quad 0x0a26cf62f92800ba + .quad 0x01ab12d5807e3217 + .quad 0x9c4891e7f6d266fd + .quad 0x0744a19b0307781b + .quad 0x88388f1d6061e23b + .quad 0x123ea6a3354bd50e + .quad 0x118d189041e32d96 + .quad 0xb9ede3c2d8315848 + .quad 0x1eab4271d83245d9 + .quad 0x4a3961e2c918a154 + + // 2^248 * 5 * B + + .quad 0x71dc3be0f8e6bba0 + .quad 0xd6cef8347effe30a + .quad 0xa992425fe13a476a + .quad 0x2cd6bce3fb1db763 + .quad 0x0327d644f3233f1e + .quad 0x499a260e34fcf016 + .quad 0x83b5a716f2dab979 + .quad 0x68aceead9bd4111f + .quad 0x38b4c90ef3d7c210 + .quad 0x308e6e24b7ad040c + .quad 0x3860d9f1b7e73e23 + .quad 0x595760d5b508f597 + + // 2^248 * 6 * B + + .quad 0x6129bfe104aa6397 + .quad 0x8f960008a4a7fccb + .quad 0x3f8bc0897d909458 + .quad 0x709fa43edcb291a9 + .quad 0x882acbebfd022790 + .quad 0x89af3305c4115760 + .quad 0x65f492e37d3473f4 + .quad 0x2cb2c5df54515a2b + .quad 0xeb0a5d8c63fd2aca + .quad 0xd22bc1662e694eff + .quad 0x2723f36ef8cbb03a + .quad 0x70f029ecf0c8131f + + // 2^248 * 7 * B + + .quad 0x461307b32eed3e33 + .quad 0xae042f33a45581e7 + .quad 0xc94449d3195f0366 + .quad 0x0b7d5d8a6c314858 + .quad 0x2a6aafaa5e10b0b9 + .quad 0x78f0a370ef041aa9 + .quad 0x773efb77aa3ad61f + .quad 0x44eca5a2a74bd9e1 + .quad 0x25d448327b95d543 + .quad 0x70d38300a3340f1d + .quad 0xde1c531c60e1c52b + .quad 0x272224512c7de9e4 + + // 2^248 * 8 * B + + .quad 0x1abc92af49c5342e + .quad 0xffeed811b2e6fad0 + .quad 0xefa28c8dfcc84e29 + .quad 0x11b5df18a44cc543 + .quad 0xbf7bbb8a42a975fc + .quad 0x8c5c397796ada358 + .quad 0xe27fc76fcdedaa48 + .quad 0x19735fd7f6bc20a6 + .quad 0xe3ab90d042c84266 + .quad 0xeb848e0f7f19547e + .quad 0x2503a1d065a497b9 + .quad 0x0fef911191df895f diff --git a/cbits/s2n/arm/edwards25519_scalarmulbase_alt.S b/cbits/s2n/arm/edwards25519_scalarmulbase_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/edwards25519_scalarmulbase_alt.S @@ -0,0 +1,9477 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Scalar multiplication for the edwards25519 standard basepoint +// Input scalar[4]; output res[8] +// +// extern void edwards25519_scalarmulbase_alt +// (uint64_t res[static 8],const uint64_t scalar[static 4]); +// +// Given a scalar n, returns point (X,Y) = n * B where B = (...,4/5) is +// the standard basepoint for the edwards25519 (Ed25519) curve. +// +// Standard ARM ABI: X0 = res, X1 = scalar +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(edwards25519_scalarmulbase_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(edwards25519_scalarmulbase_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(edwards25519_scalarmulbase_alt) + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Stable home for the input result argument during the whole body + +#define res x23 + +// Other variables that are only needed prior to the modular inverse. + +#define tab x19 + +#define i x20 + +#define bias x21 + +#define bf x22 +#define ix x22 + +// Pointer-offset pairs for result and temporaries on stack with some aliasing. + +#define resx res, #(0*NUMSIZE) +#define resy res, #(1*NUMSIZE) + +#define scalar sp, #(0*NUMSIZE) + +#define tabent sp, #(1*NUMSIZE) +#define ymx_2 sp, #(1*NUMSIZE) +#define xpy_2 sp, #(2*NUMSIZE) +#define kxy_2 sp, #(3*NUMSIZE) + +#define acc sp, #(4*NUMSIZE) +#define x_1 sp, #(4*NUMSIZE) +#define y_1 sp, #(5*NUMSIZE) +#define z_1 sp, #(6*NUMSIZE) +#define w_1 sp, #(7*NUMSIZE) +#define x_3 sp, #(4*NUMSIZE) +#define y_3 sp, #(5*NUMSIZE) +#define z_3 sp, #(6*NUMSIZE) +#define w_3 sp, #(7*NUMSIZE) + +#define tmpspace sp, #(8*NUMSIZE) +#define t0 sp, #(8*NUMSIZE) +#define t1 sp, #(9*NUMSIZE) +#define t2 sp, #(10*NUMSIZE) +#define t3 sp, #(11*NUMSIZE) +#define t4 sp, #(12*NUMSIZE) +#define t5 sp, #(13*NUMSIZE) + +// Total size to reserve on the stack + +#define NSPACE 14*NUMSIZE + +// Load 64-bit immediate into a register + +#define movbig(nn,n3,n2,n1,n0) \ + movz nn, n0 __LF \ + movk nn, n1, lsl #16 __LF \ + movk nn, n2, lsl #32 __LF \ + movk nn, n3, lsl #48 + +// Macro wrapping up the basic field operation bignum_mul_p25519_alt, only +// trivially different from a pure function call to that subroutine. + +#define mul_p25519(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x7, x8, [P2] __LF \ + mul x12, x3, x7 __LF \ + umulh x13, x3, x7 __LF \ + mul x11, x3, x8 __LF \ + umulh x14, x3, x8 __LF \ + adds x13, x13, x11 __LF \ + ldp x9, x10, [P2+16] __LF \ + mul x11, x3, x9 __LF \ + umulh x15, x3, x9 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x3, x10 __LF \ + umulh x16, x3, x10 __LF \ + adcs x15, x15, x11 __LF \ + adc x16, x16, xzr __LF \ + ldp x5, x6, [P1+16] __LF \ + mul x11, x4, x7 __LF \ + adds x13, x13, x11 __LF \ + mul x11, x4, x8 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x4, x9 __LF \ + adcs x15, x15, x11 __LF \ + mul x11, x4, x10 __LF \ + adcs x16, x16, x11 __LF \ + umulh x3, x4, x10 __LF \ + adc x3, x3, xzr __LF \ + umulh x11, x4, x7 __LF \ + adds x14, x14, x11 __LF \ + umulh x11, x4, x8 __LF \ + adcs x15, x15, x11 __LF \ + umulh x11, x4, x9 __LF \ + adcs x16, x16, x11 __LF \ + adc x3, x3, xzr __LF \ + mul x11, x5, x7 __LF \ + adds x14, x14, x11 __LF \ + mul x11, x5, x8 __LF \ + adcs x15, x15, x11 __LF \ + mul x11, x5, x9 __LF \ + adcs x16, x16, x11 __LF \ + mul x11, x5, x10 __LF \ + adcs x3, x3, x11 __LF \ + umulh x4, x5, x10 __LF \ + adc x4, x4, xzr __LF \ + umulh x11, x5, x7 __LF \ + adds x15, x15, x11 __LF \ + umulh x11, x5, x8 __LF \ + adcs x16, x16, x11 __LF \ + umulh x11, x5, x9 __LF \ + adcs x3, x3, x11 __LF \ + adc x4, x4, xzr __LF \ + mul x11, x6, x7 __LF \ + adds x15, x15, x11 __LF \ + mul x11, x6, x8 __LF \ + adcs x16, x16, x11 __LF \ + mul x11, x6, x9 __LF \ + adcs x3, x3, x11 __LF \ + mul x11, x6, x10 __LF \ + adcs x4, x4, x11 __LF \ + umulh x5, x6, x10 __LF \ + adc x5, x5, xzr __LF \ + umulh x11, x6, x7 __LF \ + adds x16, x16, x11 __LF \ + umulh x11, x6, x8 __LF \ + adcs x3, x3, x11 __LF \ + umulh x11, x6, x9 __LF \ + adcs x4, x4, x11 __LF \ + adc x5, x5, xzr __LF \ + mov x7, #0x26 __LF \ + mul x11, x7, x16 __LF \ + umulh x9, x7, x16 __LF \ + adds x12, x12, x11 __LF \ + mul x11, x7, x3 __LF \ + umulh x3, x7, x3 __LF \ + adcs x13, x13, x11 __LF \ + mul x11, x7, x4 __LF \ + umulh x4, x7, x4 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x7, x5 __LF \ + umulh x5, x7, x5 __LF \ + adcs x15, x15, x11 __LF \ + cset x16, cs __LF \ + adds x15, x15, x4 __LF \ + adc x16, x16, x5 __LF \ + cmn x15, x15 __LF \ + orr x15, x15, #0x8000000000000000 __LF \ + adc x8, x16, x16 __LF \ + mov x7, #0x13 __LF \ + madd x11, x7, x8, x7 __LF \ + adds x12, x12, x11 __LF \ + adcs x13, x13, x9 __LF \ + adcs x14, x14, x3 __LF \ + adcs x15, x15, xzr __LF \ + csel x7, x7, xzr, cc __LF \ + subs x12, x12, x7 __LF \ + sbcs x13, x13, xzr __LF \ + sbcs x14, x14, xzr __LF \ + sbc x15, x15, xzr __LF \ + and x15, x15, #0x7fffffffffffffff __LF \ + stp x12, x13, [P0] __LF \ + stp x14, x15, [P0+16] + +// A version of multiplication that only guarantees output < 2 * p_25519. +// This basically skips the +1 and final correction in quotient estimation. + +#define mul_4(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x7, x8, [P2] __LF \ + mul x12, x3, x7 __LF \ + umulh x13, x3, x7 __LF \ + mul x11, x3, x8 __LF \ + umulh x14, x3, x8 __LF \ + adds x13, x13, x11 __LF \ + ldp x9, x10, [P2+16] __LF \ + mul x11, x3, x9 __LF \ + umulh x15, x3, x9 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x3, x10 __LF \ + umulh x16, x3, x10 __LF \ + adcs x15, x15, x11 __LF \ + adc x16, x16, xzr __LF \ + ldp x5, x6, [P1+16] __LF \ + mul x11, x4, x7 __LF \ + adds x13, x13, x11 __LF \ + mul x11, x4, x8 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x4, x9 __LF \ + adcs x15, x15, x11 __LF \ + mul x11, x4, x10 __LF \ + adcs x16, x16, x11 __LF \ + umulh x3, x4, x10 __LF \ + adc x3, x3, xzr __LF \ + umulh x11, x4, x7 __LF \ + adds x14, x14, x11 __LF \ + umulh x11, x4, x8 __LF \ + adcs x15, x15, x11 __LF \ + umulh x11, x4, x9 __LF \ + adcs x16, x16, x11 __LF \ + adc x3, x3, xzr __LF \ + mul x11, x5, x7 __LF \ + adds x14, x14, x11 __LF \ + mul x11, x5, x8 __LF \ + adcs x15, x15, x11 __LF \ + mul x11, x5, x9 __LF \ + adcs x16, x16, x11 __LF \ + mul x11, x5, x10 __LF \ + adcs x3, x3, x11 __LF \ + umulh x4, x5, x10 __LF \ + adc x4, x4, xzr __LF \ + umulh x11, x5, x7 __LF \ + adds x15, x15, x11 __LF \ + umulh x11, x5, x8 __LF \ + adcs x16, x16, x11 __LF \ + umulh x11, x5, x9 __LF \ + adcs x3, x3, x11 __LF \ + adc x4, x4, xzr __LF \ + mul x11, x6, x7 __LF \ + adds x15, x15, x11 __LF \ + mul x11, x6, x8 __LF \ + adcs x16, x16, x11 __LF \ + mul x11, x6, x9 __LF \ + adcs x3, x3, x11 __LF \ + mul x11, x6, x10 __LF \ + adcs x4, x4, x11 __LF \ + umulh x5, x6, x10 __LF \ + adc x5, x5, xzr __LF \ + umulh x11, x6, x7 __LF \ + adds x16, x16, x11 __LF \ + umulh x11, x6, x8 __LF \ + adcs x3, x3, x11 __LF \ + umulh x11, x6, x9 __LF \ + adcs x4, x4, x11 __LF \ + adc x5, x5, xzr __LF \ + mov x7, #0x26 __LF \ + mul x11, x7, x16 __LF \ + umulh x9, x7, x16 __LF \ + adds x12, x12, x11 __LF \ + mul x11, x7, x3 __LF \ + umulh x3, x7, x3 __LF \ + adcs x13, x13, x11 __LF \ + mul x11, x7, x4 __LF \ + umulh x4, x7, x4 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x7, x5 __LF \ + umulh x5, x7, x5 __LF \ + adcs x15, x15, x11 __LF \ + cset x16, cs __LF \ + adds x15, x15, x4 __LF \ + adc x16, x16, x5 __LF \ + cmn x15, x15 __LF \ + bic x15, x15, #0x8000000000000000 __LF \ + adc x8, x16, x16 __LF \ + mov x7, #0x13 __LF \ + mul x11, x7, x8 __LF \ + adds x12, x12, x11 __LF \ + adcs x13, x13, x9 __LF \ + adcs x14, x14, x3 __LF \ + adc x15, x15, xzr __LF \ + stp x12, x13, [P0] __LF \ + stp x14, x15, [P0+16] + +// Modular subtraction with double modulus 2 * p_25519 = 2^256 - 38 + +#define sub_twice4(P0,P1,P2) \ + ldp x5, x6, [P1] __LF \ + ldp x4, x3, [P2] __LF \ + subs x5, x5, x4 __LF \ + sbcs x6, x6, x3 __LF \ + ldp x7, x8, [P1+16] __LF \ + ldp x4, x3, [P2+16] __LF \ + sbcs x7, x7, x4 __LF \ + sbcs x8, x8, x3 __LF \ + mov x4, #38 __LF \ + csel x3, x4, xzr, lo __LF \ + subs x5, x5, x3 __LF \ + sbcs x6, x6, xzr __LF \ + sbcs x7, x7, xzr __LF \ + sbc x8, x8, xzr __LF \ + stp x5, x6, [P0] __LF \ + stp x7, x8, [P0+16] + +// Modular addition and doubling with double modulus 2 * p_25519 = 2^256 - 38. +// This only ensures that the result fits in 4 digits, not that it is reduced +// even w.r.t. double modulus. The result is always correct modulo provided +// the sum of the inputs is < 2^256 + 2^256 - 38, so in particular provided +// at least one of them is reduced double modulo. + +#define add_twice4(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x7, x8, [P2] __LF \ + adds x3, x3, x7 __LF \ + adcs x4, x4, x8 __LF \ + ldp x5, x6, [P1+16] __LF \ + ldp x7, x8, [P2+16] __LF \ + adcs x5, x5, x7 __LF \ + adcs x6, x6, x8 __LF \ + mov x9, #38 __LF \ + csel x9, x9, xzr, cs __LF \ + adds x3, x3, x9 __LF \ + adcs x4, x4, xzr __LF \ + adcs x5, x5, xzr __LF \ + adc x6, x6, xzr __LF \ + stp x3, x4, [P0] __LF \ + stp x5, x6, [P0+16] + +#define double_twice4(P0,P1) \ + ldp x3, x4, [P1] __LF \ + adds x3, x3, x3 __LF \ + adcs x4, x4, x4 __LF \ + ldp x5, x6, [P1+16] __LF \ + adcs x5, x5, x5 __LF \ + adcs x6, x6, x6 __LF \ + mov x9, #38 __LF \ + csel x9, x9, xzr, cs __LF \ + adds x3, x3, x9 __LF \ + adcs x4, x4, xzr __LF \ + adcs x5, x5, xzr __LF \ + adc x6, x6, xzr __LF \ + stp x3, x4, [P0] __LF \ + stp x5, x6, [P0+16] + +S2N_BN_SYMBOL(edwards25519_scalarmulbase_alt): + CFI_START + +// Save regs and make room for temporaries + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_DEC_SP(NSPACE) + +// Move the output pointer to a stable place + + mov res, x0 + +// Copy the input scalar x to its local variable while reducing it +// modulo 2^252 + m where m = 27742317777372353535851937790883648493; +// this is the order of the basepoint so this doesn't change the result. +// First do q = floor(x/2^252) and x' = x - q * (2^252 + m), which gives +// an initial result -15 * m <= x' < 2^252 + + ldp x10, x11, [x1] + ldp x12, x13, [x1, #16] + + lsr x9, x13, #60 + + movbig(x0,#0x5812,#0x631a,#0x5cf5,#0xd3ed); + movbig(x1,#0x14de,#0xf9de,#0xa2f7,#0x9cd6); + + mul x2, x9, x0 + mul x3, x9, x1 + umulh x4, x9, x0 + umulh x5, x9, x1 + + adds x3, x3, x4 + adc x4, x5, xzr + lsl x5, x9, #60 + + subs x10, x10, x2 + sbcs x11, x11, x3 + sbcs x12, x12, x4 + sbcs x13, x13, x5 + +// If x' < 0 then just directly negate it; this makes sure the +// reduced argument is strictly 0 <= x' < 2^252, but now we need +// to record (done via bit 255 of the reduced scalar, which is +// ignored in the main loop) when we negated so we can flip +// the sign of the eventual point to compensate. + + csetm x9, cc + adds xzr, x9, x9 + eor x10, x10, x9 + adcs x10, x10, xzr + eor x11, x11, x9 + adcs x11, x11, xzr + eor x12, x12, x9 + adcs x12, x12, xzr + eor x13, x13, x9 + adc x13, x13, xzr + + and x9, x9, #0x8000000000000000 + orr x13, x13, x9 + +// And before we store the scalar, test and reset bit 251 to +// initialize the main loop just below. + + stp x10, x11, [scalar] + tst x13, #0x0800000000000000 + bic x13, x13, #0x0800000000000000 + stp x12, x13, [scalar+16] + +// The main part of the computation is in extended-projective coordinates +// (X,Y,Z,T), representing an affine point on the edwards25519 curve +// (x,y) via x = X/Z, y = Y/Z and x * y = T/Z (so X * Y = T * Z). +// In comments B means the standard basepoint (x,4/5) = +// (0x216....f25d51a,0x6666..666658). +// +// Initialize accumulator "acc" to either 0 or 2^251 * B depending on +// bit 251 of the (reduced) scalar. That leaves bits 0..250 to handle. + +#if defined(__ELF__) + adrp tab, S2N_BN_SYMBOL(edwards25519_scalarmulbase_alt_constant) + add tab, tab, :lo12:S2N_BN_SYMBOL(edwards25519_scalarmulbase_alt_constant) +#else + adrp tab, S2N_BN_SYMBOL(edwards25519_scalarmulbase_alt_constant)@PAGE + add tab, tab, S2N_BN_SYMBOL(edwards25519_scalarmulbase_alt_constant)@PAGEOFF +#endif + + ldp x0, x1, [tab] + ldp x2, x3, [tab, #96] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc] + + ldp x0, x1, [tab, #1*16] + ldp x2, x3, [tab, #(96+1*16)] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+1*16] + + ldp x0, x1, [tab, #2*16] + ldp x2, x3, [tab, #(96+2*16)] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+2*16] + + ldp x0, x1, [tab, #3*16] + ldp x2, x3, [tab, #(96+3*16)] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+3*16] + + mov x0, #1 + stp x0, xzr, [acc+4*16] + stp xzr, xzr, [acc+5*16] + + ldp x0, x1, [tab, #4*16] + ldp x2, x3, [tab, #(96+4*16)] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+6*16] + + ldp x0, x1, [tab, #5*16] + ldp x2, x3, [tab, #(96+5*16)] + csel x0, x0, x2, eq + csel x1, x1, x3, eq + stp x0, x1, [acc+7*16] + +// The counter "i" tracks the bit position for which the scalar has +// already been absorbed, starting at 0 and going up in chunks of 4. +// +// The pointer "tab" points at the current block of the table for +// multiples (2^i * j) * B at the current bit position i; 1 <= j <= 8. +// +// The bias is always either 0 and 1 and needs to be added to the +// partially processed scalar implicitly. This is used to absorb 4 bits +// of scalar per iteration from 3-bit table indexing by exploiting +// negation: (16 * h + l) * B = (16 * (h + 1) - (16 - l)) * B is used +// when l >= 9. Note that we can't have any bias left over at the +// end because we made sure bit 251 is clear in the reduced scalar. + + mov i, 0 + add tab, tab, #192 + mov bias, xzr + +// Start of the main loop, repeated 63 times for i = 0, 4, 8, ..., 248 + +Ledwards25519_scalarmulbase_alt_scalarloop: + +// Look at the next 4-bit field "bf", adding the previous bias as well. +// Choose the table index "ix" as bf when bf <= 8 and 16 - bf for bf >= 9, +// setting the bias to 1 for the next iteration in the latter case. + + lsr x0, i, #6 + ldr x2, [sp, x0, lsl #3] // Exploiting scalar = sp exactly + lsr x2, x2, i + and x2, x2, #15 + add bf, x2, bias + + cmp bf, 9 + cset bias, cs + + mov x0, 16 + sub x0, x0, bf + cmp bias, xzr + csel ix, x0, bf, ne + +// Perform constant-time lookup in the table to get element number "ix". +// The table entry for the affine point (x,y) is actually a triple +// (y - x,x + y,2 * d * x * y) to precompute parts of the addition. +// Note that "ix" can be 0, so we set up the appropriate identity first. + + mov x0, #1 + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, #1 + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + mov x9, xzr + mov x10, xzr + mov x11, xzr + + cmp ix, #1 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #2 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #3 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #4 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #5 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #6 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #7 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + + cmp ix, #8 + ldp x12, x13, [tab] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x12, x13, [tab, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x12, x13, [tab, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x12, x13, [tab, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x12, x13, [tab, #64] + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x12, x13, [tab, #80] + csel x10, x10, x12, ne + csel x11, x11, x13, ne + add tab, tab, #96 + +// We now have the triple from the table in registers as follows +// +// [x3;x2;x1;x0] = y - x +// [x7;x6;x5;x4] = x + y +// [x11;x10;x9;x8] = 2 * d * x * y +// +// In case bias = 1 we need to negate this. For Edwards curves +// -(x,y) = (-x,y), i.e. we need to negate the x coordinate. +// In this processed encoding, that amounts to swapping the +// first two fields and negating the third. +// +// The optional negation here also pretends bias = 0 whenever +// ix = 0 so that it doesn't need to handle the case of zero +// inputs, since no non-trivial table entries are zero. Note +// that in the zero case the whole negation is trivial, and +// so indeed is the swapping. + + cmp bias, #0 + + csel x12, x0, x4, eq + csel x13, x1, x5, eq + csel x14, x2, x6, eq + csel x15, x3, x7, eq + stp x12, x13, [tabent] + stp x14, x15, [tabent+16] + + csel x12, x0, x4, ne + csel x13, x1, x5, ne + csel x14, x2, x6, ne + csel x15, x3, x7, ne + stp x12, x13, [tabent+32] + stp x14, x15, [tabent+48] + + mov x0, #-19 + subs x0, x0, x8 + mov x2, #-1 + sbcs x1, x2, x9 + sbcs x2, x2, x10 + mov x3, #0x7FFFFFFFFFFFFFFF + sbc x3, x3, x11 + + cmp ix, xzr + ccmp bias, xzr, #4, ne + + csel x0, x0, x8, ne + csel x1, x1, x9, ne + stp x0, x1, [tabent+64] + csel x2, x2, x10, ne + csel x3, x3, x11, ne + stp x2, x3, [tabent+80] + +// Extended-projective and precomputed mixed addition. +// This is effectively the same as calling the standalone +// function edwards25519_pepadd_alt(acc,acc,tabent), but we +// only retain slightly weaker normalization < 2 * p_25519 +// throughout the inner loop, so the computation is +// slightly different, and faster overall. + + double_twice4(t0,z_1) + sub_twice4(t1,y_1,x_1) + add_twice4(t2,y_1,x_1) + mul_4(t3,w_1,kxy_2) + mul_4(t1,t1,ymx_2) + mul_4(t2,t2,xpy_2) + sub_twice4(t4,t0,t3) + add_twice4(t0,t0,t3) + sub_twice4(t5,t2,t1) + add_twice4(t1,t2,t1) + mul_4(z_3,t4,t0) + mul_4(x_3,t5,t4) + mul_4(y_3,t0,t1) + mul_4(w_3,t5,t1) + +// End of the main loop; move on by 4 bits. + + add i, i, 4 + cmp i, 252 + bcc Ledwards25519_scalarmulbase_alt_scalarloop + +// Insert the optional negation of the projective X coordinate, and +// so by extension the final affine x coordinate x = X/Z and thus +// the point P = (x,y). We only know X < 2 * p_25519, so we do the +// negation as 2 * p_25519 - X to keep it nonnegative. From this +// point on we don't need any normalization of the coordinates +// except for making sure that they fit in 4 digits. + + ldp x0, x1, [x_3] + ldp x2, x3, [x_3+16] + mov x4, #0xffffffffffffffda + subs x4, x4, x0 + mov x7, #0xffffffffffffffff + sbcs x5, x7, x1 + sbcs x6, x7, x2 + sbc x7, x7, x3 + ldr x10, [scalar+24] + tst x10, #0x8000000000000000 + csel x0, x4, x0, ne + csel x1, x5, x1, ne + csel x2, x6, x2, ne + csel x3, x7, x3, ne + stp x0, x1, [x_3] + stp x2, x3, [x_3+16] + +// Now we need to map out of the extended-projective representation +// (X,Y,Z,W) back to the affine form (x,y) = (X/Z,Y/Z). This means +// first calling the modular inverse to get w_3 = 1/z_3. + + add x0, w_3 + add x1, z_3 + +// Inline copy of bignum_inv_p25519, identical except for stripping out +// the prologue and epilogue saving and restoring registers and making +// and reclaiming room on the stack. For more details and explanations see +// "arm/curve25519/bignum_inv_p25519.S". Note that the stack it uses for +// its own temporaries is 128 bytes, so it has no effect on variables +// that are needed in the rest of our computation here: res, w_3, x_3 +// and y_3. + + mov x20, x0 + mov x10, #0xffffffffffffffed + mov x11, #0xffffffffffffffff + stp x10, x11, [sp] + mov x12, #0x7fffffffffffffff + stp x11, x12, [sp, #16] + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + mov x7, #0x13 + lsr x6, x5, #63 + madd x6, x7, x6, x7 + adds x2, x2, x6 + adcs x3, x3, xzr + adcs x4, x4, xzr + orr x5, x5, #0x8000000000000000 + adcs x5, x5, xzr + csel x6, x7, xzr, cc + subs x2, x2, x6 + sbcs x3, x3, xzr + sbcs x4, x4, xzr + sbc x5, x5, xzr + and x5, x5, #0x7fffffffffffffff + stp x2, x3, [sp, #32] + stp x4, x5, [sp, #48] + stp xzr, xzr, [sp, #64] + stp xzr, xzr, [sp, #80] + mov x10, #0x2099 + movk x10, #0x7502, lsl #16 + movk x10, #0x9e23, lsl #32 + movk x10, #0xa0f9, lsl #48 + mov x11, #0x2595 + movk x11, #0x1d13, lsl #16 + movk x11, #0x8f3f, lsl #32 + movk x11, #0xa8c6, lsl #48 + mov x12, #0x5242 + movk x12, #0x5ac, lsl #16 + movk x12, #0x8938, lsl #32 + movk x12, #0x6c6c, lsl #48 + mov x13, #0x615 + movk x13, #0x4177, lsl #16 + movk x13, #0x8b2, lsl #32 + movk x13, #0x2765, lsl #48 + stp x10, x11, [sp, #96] + stp x12, x13, [sp, #112] + mov x21, #0xa + mov x22, #0x1 + b Ledwards25519_scalarmulbase_alt_invmidloop +Ledwards25519_scalarmulbase_alt_invloop: + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + and x0, x12, x16 + and x1, x13, x17 + add x19, x0, x1 + ldr x7, [sp] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #32] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x7, [sp, #8] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #40] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + adc x6, x6, x1 + extr x4, x2, x4, #59 + str x4, [sp] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + adc x4, x4, x1 + extr x5, x3, x5, #59 + str x5, [sp, #32] + ldr x7, [sp, #16] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #48] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + adc x5, x5, x1 + extr x2, x6, x2, #59 + str x2, [sp, #8] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + adc x2, x2, x1 + extr x3, x4, x3, #59 + str x3, [sp, #40] + ldr x7, [sp, #24] + eor x1, x7, x14 + asr x3, x1, #63 + and x3, x3, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #56] + eor x1, x8, x15 + asr x0, x1, #63 + and x0, x0, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x5, x6, #59 + str x6, [sp, #16] + extr x5, x3, x5, #59 + str x5, [sp, #24] + eor x1, x7, x16 + asr x5, x1, #63 + and x5, x5, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + asr x0, x1, #63 + and x0, x0, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x4, x2, x4, #59 + str x4, [sp, #48] + extr x2, x5, x2, #59 + str x2, [sp, #56] + ldr x7, [sp, #64] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #96] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #64] + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + str x5, [sp, #96] + adc x3, x3, x1 + ldr x7, [sp, #72] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #104] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #72] + adc x6, x6, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + str x3, [sp, #104] + adc x4, x4, x1 + ldr x7, [sp, #80] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #112] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #80] + adc x5, x5, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + str x4, [sp, #112] + adc x2, x2, x1 + ldr x7, [sp, #88] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #120] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x3, x5, #63 + ldp x0, x1, [sp, #64] + add x6, x6, x3, asr #63 + mov x3, #0x13 + mul x4, x6, x3 + add x5, x5, x6, lsl #63 + smulh x3, x6, x3 + ldr x6, [sp, #80] + adds x0, x0, x4 + adcs x1, x1, x3 + asr x3, x3, #63 + adcs x6, x6, x3 + adc x5, x5, x3 + stp x0, x1, [sp, #64] + stp x6, x5, [sp, #80] + eor x1, x7, x16 + and x5, x16, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + and x0, x17, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x6, x5, x2, #63 + ldp x0, x1, [sp, #96] + add x6, x6, x5, asr #63 + mov x5, #0x13 + mul x4, x6, x5 + add x2, x2, x6, lsl #63 + smulh x5, x6, x5 + ldr x3, [sp, #112] + adds x0, x0, x4 + adcs x1, x1, x5 + asr x5, x5, #63 + adcs x3, x3, x5 + adc x2, x2, x5 + stp x0, x1, [sp, #96] + stp x3, x2, [sp, #112] +Ledwards25519_scalarmulbase_alt_invmidloop: + mov x1, x22 + ldr x2, [sp] + ldr x3, [sp, #32] + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x8, x4, #0x100, lsl #12 + sbfx x8, x8, #21, #21 + mov x11, #0x100000 + add x11, x11, x11, lsl #21 + add x9, x4, x11 + asr x9, x9, #42 + add x10, x5, #0x100, lsl #12 + sbfx x10, x10, #21, #21 + add x11, x5, x11 + asr x11, x11, #42 + mul x6, x8, x2 + mul x7, x9, x3 + mul x2, x10, x2 + mul x3, x11, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #21, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #42 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #21, #21 + add x15, x5, x15 + asr x15, x15, #42 + mul x6, x12, x2 + mul x7, x13, x3 + mul x2, x14, x2 + mul x3, x15, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + mul x2, x12, x8 + mul x3, x12, x9 + mul x6, x14, x8 + mul x7, x14, x9 + madd x8, x13, x10, x2 + madd x9, x13, x11, x3 + madd x16, x15, x10, x6 + madd x17, x15, x11, x7 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #22, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #43 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #22, #21 + add x15, x5, x15 + asr x15, x15, #43 + mneg x2, x12, x8 + mneg x3, x12, x9 + mneg x4, x14, x8 + mneg x5, x14, x9 + msub x10, x13, x16, x2 + msub x11, x13, x17, x3 + msub x12, x15, x16, x4 + msub x13, x15, x17, x5 + mov x22, x1 + subs x21, x21, #0x1 + b.ne Ledwards25519_scalarmulbase_alt_invloop + ldr x0, [sp] + ldr x1, [sp, #32] + mul x0, x0, x10 + madd x1, x1, x11, x0 + asr x0, x1, #63 + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + eor x14, x14, x0 + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + eor x15, x15, x0 + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + eor x16, x16, x0 + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + eor x17, x17, x0 + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + ldr x7, [sp, #64] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #96] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #64] + adc x2, x2, x1 + ldr x7, [sp, #72] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #104] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #72] + adc x6, x6, x1 + ldr x7, [sp, #80] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #112] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #80] + adc x5, x5, x1 + ldr x7, [sp, #88] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #120] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x3, x5, #63 + ldp x0, x1, [sp, #64] + tst x3, x3 + cinc x6, x6, pl + mov x3, #0x13 + mul x4, x6, x3 + add x5, x5, x6, lsl #63 + smulh x6, x6, x3 + ldr x2, [sp, #80] + adds x0, x0, x4 + adcs x1, x1, x6 + asr x6, x6, #63 + adcs x2, x2, x6 + adcs x5, x5, x6 + csel x3, x3, xzr, mi + subs x0, x0, x3 + sbcs x1, x1, xzr + sbcs x2, x2, xzr + sbc x5, x5, xzr + and x5, x5, #0x7fffffffffffffff + mov x4, x20 + stp x0, x1, [x4] + stp x2, x5, [x4, #16] + +// The final result is x = X * inv(Z), y = Y * inv(Z). +// These are the only operations in the whole computation that +// fully reduce modulo p_25519 since now we want the canonical +// answer as output. + + mul_p25519(resx,x_3,w_3) + mul_p25519(resy,y_3,w_3) + +// Restore stack and registers + + CFI_INC_SP(NSPACE) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(edwards25519_scalarmulbase_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif + +// **************************************************************************** +// The precomputed data (all read-only). +// **************************************************************************** + +#if defined(__ELF__) +.section .rodata +.type S2N_BN_SYMBOL(edwards25519_scalarmulbase_alt_constant), %object +.size S2N_BN_SYMBOL(edwards25519_scalarmulbase_alt_constant), 48576 +#elif defined(__APPLE__) +.const_data +#endif + +S2N_BN_SYMBOL(edwards25519_scalarmulbase_alt_constant): + +// 0 * B = 0 and 2^251 * B in extended-projective coordinates +// but with Z = 1 assumed and hence left out, so they are (X,Y,T) only. + + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000001 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + + .quad 0x525f946d7c7220e7 + .quad 0x4636b0b2f1e35444 + .quad 0x796e9d70e892ae0f + .quad 0x03dec05fa937adb1 + .quad 0x6d1c271cc6375515 + .quad 0x462588c4a4ca4f14 + .quad 0x691129fee55afc39 + .quad 0x15949f784d8472f5 + .quad 0xbd89e510afad0049 + .quad 0x4d1f08c073b9860e + .quad 0x07716e8b2d00af9d + .quad 0x70d685f68f859714 + +// Precomputed table of multiples of generator for edwards25519 +// all in precomputed extended-projective (y-x,x+y,2*d*x*y) triples. + + // 2^0 * 1 * G + + .quad 0x9d103905d740913e + .quad 0xfd399f05d140beb3 + .quad 0xa5c18434688f8a09 + .quad 0x44fd2f9298f81267 + .quad 0x2fbc93c6f58c3b85 + .quad 0xcf932dc6fb8c0e19 + .quad 0x270b4898643d42c2 + .quad 0x07cf9d3a33d4ba65 + .quad 0xabc91205877aaa68 + .quad 0x26d9e823ccaac49e + .quad 0x5a1b7dcbdd43598c + .quad 0x6f117b689f0c65a8 + + // 2^0 * 2 * G + + .quad 0x8a99a56042b4d5a8 + .quad 0x8f2b810c4e60acf6 + .quad 0xe09e236bb16e37aa + .quad 0x6bb595a669c92555 + .quad 0x9224e7fc933c71d7 + .quad 0x9f469d967a0ff5b5 + .quad 0x5aa69a65e1d60702 + .quad 0x590c063fa87d2e2e + .quad 0x43faa8b3a59b7a5f + .quad 0x36c16bdd5d9acf78 + .quad 0x500fa0840b3d6a31 + .quad 0x701af5b13ea50b73 + + // 2^0 * 3 * G + + .quad 0x56611fe8a4fcd265 + .quad 0x3bd353fde5c1ba7d + .quad 0x8131f31a214bd6bd + .quad 0x2ab91587555bda62 + .quad 0xaf25b0a84cee9730 + .quad 0x025a8430e8864b8a + .quad 0xc11b50029f016732 + .quad 0x7a164e1b9a80f8f4 + .quad 0x14ae933f0dd0d889 + .quad 0x589423221c35da62 + .quad 0xd170e5458cf2db4c + .quad 0x5a2826af12b9b4c6 + + // 2^0 * 4 * G + + .quad 0x95fe050a056818bf + .quad 0x327e89715660faa9 + .quad 0xc3e8e3cd06a05073 + .quad 0x27933f4c7445a49a + .quad 0x287351b98efc099f + .quad 0x6765c6f47dfd2538 + .quad 0xca348d3dfb0a9265 + .quad 0x680e910321e58727 + .quad 0x5a13fbe9c476ff09 + .quad 0x6e9e39457b5cc172 + .quad 0x5ddbdcf9102b4494 + .quad 0x7f9d0cbf63553e2b + + // 2^0 * 5 * G + + .quad 0x7f9182c3a447d6ba + .quad 0xd50014d14b2729b7 + .quad 0xe33cf11cb864a087 + .quad 0x154a7e73eb1b55f3 + .quad 0xa212bc4408a5bb33 + .quad 0x8d5048c3c75eed02 + .quad 0xdd1beb0c5abfec44 + .quad 0x2945ccf146e206eb + .quad 0xbcbbdbf1812a8285 + .quad 0x270e0807d0bdd1fc + .quad 0xb41b670b1bbda72d + .quad 0x43aabe696b3bb69a + + // 2^0 * 6 * G + + .quad 0x499806b67b7d8ca4 + .quad 0x575be28427d22739 + .quad 0xbb085ce7204553b9 + .quad 0x38b64c41ae417884 + .quad 0x3a0ceeeb77157131 + .quad 0x9b27158900c8af88 + .quad 0x8065b668da59a736 + .quad 0x51e57bb6a2cc38bd + .quad 0x85ac326702ea4b71 + .quad 0xbe70e00341a1bb01 + .quad 0x53e4a24b083bc144 + .quad 0x10b8e91a9f0d61e3 + + // 2^0 * 7 * G + + .quad 0xba6f2c9aaa3221b1 + .quad 0x6ca021533bba23a7 + .quad 0x9dea764f92192c3a + .quad 0x1d6edd5d2e5317e0 + .quad 0x6b1a5cd0944ea3bf + .quad 0x7470353ab39dc0d2 + .quad 0x71b2528228542e49 + .quad 0x461bea69283c927e + .quad 0xf1836dc801b8b3a2 + .quad 0xb3035f47053ea49a + .quad 0x529c41ba5877adf3 + .quad 0x7a9fbb1c6a0f90a7 + + // 2^0 * 8 * G + + .quad 0xe2a75dedf39234d9 + .quad 0x963d7680e1b558f9 + .quad 0x2c2741ac6e3c23fb + .quad 0x3a9024a1320e01c3 + .quad 0x59b7596604dd3e8f + .quad 0x6cb30377e288702c + .quad 0xb1339c665ed9c323 + .quad 0x0915e76061bce52f + .quad 0xe7c1f5d9c9a2911a + .quad 0xb8a371788bcca7d7 + .quad 0x636412190eb62a32 + .quad 0x26907c5c2ecc4e95 + + // 2^4 * 1 * B + + .quad 0x7ec851ca553e2df3 + .quad 0xa71284cba64878b3 + .quad 0xe6b5e4193288d1e7 + .quad 0x4cf210ec5a9a8883 + .quad 0x322d04a52d9021f6 + .quad 0xb9c19f3375c6bf9c + .quad 0x587a3a4342d20b09 + .quad 0x143b1cf8aa64fe61 + .quad 0x9f867c7d968acaab + .quad 0x5f54258e27092729 + .quad 0xd0a7d34bea180975 + .quad 0x21b546a3374126e1 + + // 2^4 * 2 * B + + .quad 0xa94ff858a2888343 + .quad 0xce0ed4565313ed3c + .quad 0xf55c3dcfb5bf34fa + .quad 0x0a653ca5c9eab371 + .quad 0x490a7a45d185218f + .quad 0x9a15377846049335 + .quad 0x0060ea09cc31e1f6 + .quad 0x7e041577f86ee965 + .quad 0x66b2a496ce5b67f3 + .quad 0xff5492d8bd569796 + .quad 0x503cec294a592cd0 + .quad 0x566943650813acb2 + + // 2^4 * 3 * B + + .quad 0xb818db0c26620798 + .quad 0x5d5c31d9606e354a + .quad 0x0982fa4f00a8cdc7 + .quad 0x17e12bcd4653e2d4 + .quad 0x5672f9eb1dabb69d + .quad 0xba70b535afe853fc + .quad 0x47ac0f752796d66d + .quad 0x32a5351794117275 + .quad 0xd3a644a6df648437 + .quad 0x703b6559880fbfdd + .quad 0xcb852540ad3a1aa5 + .quad 0x0900b3f78e4c6468 + + // 2^4 * 4 * B + + .quad 0x0a851b9f679d651b + .quad 0xe108cb61033342f2 + .quad 0xd601f57fe88b30a3 + .quad 0x371f3acaed2dd714 + .quad 0xed280fbec816ad31 + .quad 0x52d9595bd8e6efe3 + .quad 0x0fe71772f6c623f5 + .quad 0x4314030b051e293c + .quad 0xd560005efbf0bcad + .quad 0x8eb70f2ed1870c5e + .quad 0x201f9033d084e6a0 + .quad 0x4c3a5ae1ce7b6670 + + // 2^4 * 5 * B + + .quad 0x4138a434dcb8fa95 + .quad 0x870cf67d6c96840b + .quad 0xde388574297be82c + .quad 0x7c814db27262a55a + .quad 0xbaf875e4c93da0dd + .quad 0xb93282a771b9294d + .quad 0x80d63fb7f4c6c460 + .quad 0x6de9c73dea66c181 + .quad 0x478904d5a04df8f2 + .quad 0xfafbae4ab10142d3 + .quad 0xf6c8ac63555d0998 + .quad 0x5aac4a412f90b104 + + // 2^4 * 6 * B + + .quad 0xc64f326b3ac92908 + .quad 0x5551b282e663e1e0 + .quad 0x476b35f54a1a4b83 + .quad 0x1b9da3fe189f68c2 + .quad 0x603a0d0abd7f5134 + .quad 0x8089c932e1d3ae46 + .quad 0xdf2591398798bd63 + .quad 0x1c145cd274ba0235 + .quad 0x32e8386475f3d743 + .quad 0x365b8baf6ae5d9ef + .quad 0x825238b6385b681e + .quad 0x234929c1167d65e1 + + // 2^4 * 7 * B + + .quad 0x984decaba077ade8 + .quad 0x383f77ad19eb389d + .quad 0xc7ec6b7e2954d794 + .quad 0x59c77b3aeb7c3a7a + .quad 0x48145cc21d099fcf + .quad 0x4535c192cc28d7e5 + .quad 0x80e7c1e548247e01 + .quad 0x4a5f28743b2973ee + .quad 0xd3add725225ccf62 + .quad 0x911a3381b2152c5d + .quad 0xd8b39fad5b08f87d + .quad 0x6f05606b4799fe3b + + // 2^4 * 8 * B + + .quad 0x9ffe9e92177ba962 + .quad 0x98aee71d0de5cae1 + .quad 0x3ff4ae942d831044 + .quad 0x714de12e58533ac8 + .quad 0x5b433149f91b6483 + .quad 0xadb5dc655a2cbf62 + .quad 0x87fa8412632827b3 + .quad 0x60895e91ab49f8d8 + .quad 0xe9ecf2ed0cf86c18 + .quad 0xb46d06120735dfd4 + .quad 0xbc9da09804b96be7 + .quad 0x73e2e62fd96dc26b + + // 2^8 * 1 * B + + .quad 0xed5b635449aa515e + .quad 0xa865c49f0bc6823a + .quad 0x850c1fe95b42d1c4 + .quad 0x30d76d6f03d315b9 + .quad 0x2eccdd0e632f9c1d + .quad 0x51d0b69676893115 + .quad 0x52dfb76ba8637a58 + .quad 0x6dd37d49a00eef39 + .quad 0x6c4444172106e4c7 + .quad 0xfb53d680928d7f69 + .quad 0xb4739ea4694d3f26 + .quad 0x10c697112e864bb0 + + // 2^8 * 2 * B + + .quad 0x6493c4277dbe5fde + .quad 0x265d4fad19ad7ea2 + .quad 0x0e00dfc846304590 + .quad 0x25e61cabed66fe09 + .quad 0x0ca62aa08358c805 + .quad 0x6a3d4ae37a204247 + .quad 0x7464d3a63b11eddc + .quad 0x03bf9baf550806ef + .quad 0x3f13e128cc586604 + .quad 0x6f5873ecb459747e + .quad 0xa0b63dedcc1268f5 + .quad 0x566d78634586e22c + + // 2^8 * 3 * B + + .quad 0x1637a49f9cc10834 + .quad 0xbc8e56d5a89bc451 + .quad 0x1cb5ec0f7f7fd2db + .quad 0x33975bca5ecc35d9 + .quad 0xa1054285c65a2fd0 + .quad 0x6c64112af31667c3 + .quad 0x680ae240731aee58 + .quad 0x14fba5f34793b22a + .quad 0x3cd746166985f7d4 + .quad 0x593e5e84c9c80057 + .quad 0x2fc3f2b67b61131e + .quad 0x14829cea83fc526c + + // 2^8 * 4 * B + + .quad 0xff437b8497dd95c2 + .quad 0x6c744e30aa4eb5a7 + .quad 0x9e0c5d613c85e88b + .quad 0x2fd9c71e5f758173 + .quad 0x21e70b2f4e71ecb8 + .quad 0xe656ddb940a477e3 + .quad 0xbf6556cece1d4f80 + .quad 0x05fc3bc4535d7b7e + .quad 0x24b8b3ae52afdedd + .quad 0x3495638ced3b30cf + .quad 0x33a4bc83a9be8195 + .quad 0x373767475c651f04 + + // 2^8 * 5 * B + + .quad 0x2fba99fd40d1add9 + .quad 0xb307166f96f4d027 + .quad 0x4363f05215f03bae + .quad 0x1fbea56c3b18f999 + .quad 0x634095cb14246590 + .quad 0xef12144016c15535 + .quad 0x9e38140c8910bc60 + .quad 0x6bf5905730907c8c + .quad 0x0fa778f1e1415b8a + .quad 0x06409ff7bac3a77e + .quad 0x6f52d7b89aa29a50 + .quad 0x02521cf67a635a56 + + // 2^8 * 6 * B + + .quad 0x513fee0b0a9d5294 + .quad 0x8f98e75c0fdf5a66 + .quad 0xd4618688bfe107ce + .quad 0x3fa00a7e71382ced + .quad 0xb1146720772f5ee4 + .quad 0xe8f894b196079ace + .quad 0x4af8224d00ac824a + .quad 0x001753d9f7cd6cc4 + .quad 0x3c69232d963ddb34 + .quad 0x1dde87dab4973858 + .quad 0xaad7d1f9a091f285 + .quad 0x12b5fe2fa048edb6 + + // 2^8 * 7 * B + + .quad 0x71f0fbc496fce34d + .quad 0x73b9826badf35bed + .quad 0xd2047261ff28c561 + .quad 0x749b76f96fb1206f + .quad 0xdf2b7c26ad6f1e92 + .quad 0x4b66d323504b8913 + .quad 0x8c409dc0751c8bc3 + .quad 0x6f7e93c20796c7b8 + .quad 0x1f5af604aea6ae05 + .quad 0xc12351f1bee49c99 + .quad 0x61a808b5eeff6b66 + .quad 0x0fcec10f01e02151 + + // 2^8 * 8 * B + + .quad 0x644d58a649fe1e44 + .quad 0x21fcaea231ad777e + .quad 0x02441c5a887fd0d2 + .quad 0x4901aa7183c511f3 + .quad 0x3df2d29dc4244e45 + .quad 0x2b020e7493d8de0a + .quad 0x6cc8067e820c214d + .quad 0x413779166feab90a + .quad 0x08b1b7548c1af8f0 + .quad 0xce0f7a7c246299b4 + .quad 0xf760b0f91e06d939 + .quad 0x41bb887b726d1213 + + // 2^12 * 1 * B + + .quad 0x9267806c567c49d8 + .quad 0x066d04ccca791e6a + .quad 0xa69f5645e3cc394b + .quad 0x5c95b686a0788cd2 + .quad 0x97d980e0aa39f7d2 + .quad 0x35d0384252c6b51c + .quad 0x7d43f49307cd55aa + .quad 0x56bd36cfb78ac362 + .quad 0x2ac519c10d14a954 + .quad 0xeaf474b494b5fa90 + .quad 0xe6af8382a9f87a5a + .quad 0x0dea6db1879be094 + + // 2^12 * 2 * B + + .quad 0xaa66bf547344e5ab + .quad 0xda1258888f1b4309 + .quad 0x5e87d2b3fd564b2f + .quad 0x5b2c78885483b1dd + .quad 0x15baeb74d6a8797a + .quad 0x7ef55cf1fac41732 + .quad 0x29001f5a3c8b05c5 + .quad 0x0ad7cc8752eaccfb + .quad 0x52151362793408cf + .quad 0xeb0f170319963d94 + .quad 0xa833b2fa883d9466 + .quad 0x093a7fa775003c78 + + // 2^12 * 3 * B + + .quad 0xe5107de63a16d7be + .quad 0xa377ffdc9af332cf + .quad 0x70d5bf18440b677f + .quad 0x6a252b19a4a31403 + .quad 0xb8e9604460a91286 + .quad 0x7f3fd8047778d3de + .quad 0x67d01e31bf8a5e2d + .quad 0x7b038a06c27b653e + .quad 0x9ed919d5d36990f3 + .quad 0x5213aebbdb4eb9f2 + .quad 0xc708ea054cb99135 + .quad 0x58ded57f72260e56 + + // 2^12 * 4 * B + + .quad 0x78e79dade9413d77 + .quad 0xf257f9d59729e67d + .quad 0x59db910ee37aa7e6 + .quad 0x6aa11b5bbb9e039c + .quad 0xda6d53265b0fd48b + .quad 0x8960823193bfa988 + .quad 0xd78ac93261d57e28 + .quad 0x79f2942d3a5c8143 + .quad 0x97da2f25b6c88de9 + .quad 0x251ba7eaacf20169 + .quad 0x09b44f87ef4eb4e4 + .quad 0x7d90ab1bbc6a7da5 + + // 2^12 * 5 * B + + .quad 0x9acca683a7016bfe + .quad 0x90505f4df2c50b6d + .quad 0x6b610d5fcce435aa + .quad 0x19a10d446198ff96 + .quad 0x1a07a3f496b3c397 + .quad 0x11ceaa188f4e2532 + .quad 0x7d9498d5a7751bf0 + .quad 0x19ed161f508dd8a0 + .quad 0x560a2cd687dce6ca + .quad 0x7f3568c48664cf4d + .quad 0x8741e95222803a38 + .quad 0x483bdab1595653fc + + // 2^12 * 6 * B + + .quad 0xfa780f148734fa49 + .quad 0x106f0b70360534e0 + .quad 0x2210776fe3e307bd + .quad 0x3286c109dde6a0fe + .quad 0xd6cf4d0ab4da80f6 + .quad 0x82483e45f8307fe0 + .quad 0x05005269ae6f9da4 + .quad 0x1c7052909cf7877a + .quad 0x32ee7de2874e98d4 + .quad 0x14c362e9b97e0c60 + .quad 0x5781dcde6a60a38a + .quad 0x217dd5eaaa7aa840 + + // 2^12 * 7 * B + + .quad 0x9db7c4d0248e1eb0 + .quad 0xe07697e14d74bf52 + .quad 0x1e6a9b173c562354 + .quad 0x7fa7c21f795a4965 + .quad 0x8bdf1fb9be8c0ec8 + .quad 0x00bae7f8e30a0282 + .quad 0x4963991dad6c4f6c + .quad 0x07058a6e5df6f60a + .quad 0xe9eb02c4db31f67f + .quad 0xed25fd8910bcfb2b + .quad 0x46c8131f5c5cddb4 + .quad 0x33b21c13a0cb9bce + + // 2^12 * 8 * B + + .quad 0x360692f8087d8e31 + .quad 0xf4dcc637d27163f7 + .quad 0x25a4e62065ea5963 + .quad 0x659bf72e5ac160d9 + .quad 0x9aafb9b05ee38c5b + .quad 0xbf9d2d4e071a13c7 + .quad 0x8eee6e6de933290a + .quad 0x1c3bab17ae109717 + .quad 0x1c9ab216c7cab7b0 + .quad 0x7d65d37407bbc3cc + .quad 0x52744750504a58d5 + .quad 0x09f2606b131a2990 + + // 2^16 * 1 * B + + .quad 0x40e87d44744346be + .quad 0x1d48dad415b52b25 + .quad 0x7c3a8a18a13b603e + .quad 0x4eb728c12fcdbdf7 + .quad 0x7e234c597c6691ae + .quad 0x64889d3d0a85b4c8 + .quad 0xdae2c90c354afae7 + .quad 0x0a871e070c6a9e1d + .quad 0x3301b5994bbc8989 + .quad 0x736bae3a5bdd4260 + .quad 0x0d61ade219d59e3c + .quad 0x3ee7300f2685d464 + + // 2^16 * 2 * B + + .quad 0xf5d255e49e7dd6b7 + .quad 0x8016115c610b1eac + .quad 0x3c99975d92e187ca + .quad 0x13815762979125c2 + .quad 0x43fa7947841e7518 + .quad 0xe5c6fa59639c46d7 + .quad 0xa1065e1de3052b74 + .quad 0x7d47c6a2cfb89030 + .quad 0x3fdad0148ef0d6e0 + .quad 0x9d3e749a91546f3c + .quad 0x71ec621026bb8157 + .quad 0x148cf58d34c9ec80 + + // 2^16 * 3 * B + + .quad 0x46a492f67934f027 + .quad 0x469984bef6840aa9 + .quad 0x5ca1bc2a89611854 + .quad 0x3ff2fa1ebd5dbbd4 + .quad 0xe2572f7d9ae4756d + .quad 0x56c345bb88f3487f + .quad 0x9fd10b6d6960a88d + .quad 0x278febad4eaea1b9 + .quad 0xb1aa681f8c933966 + .quad 0x8c21949c20290c98 + .quad 0x39115291219d3c52 + .quad 0x4104dd02fe9c677b + + // 2^16 * 4 * B + + .quad 0x72b2bf5e1124422a + .quad 0xa1fa0c3398a33ab5 + .quad 0x94cb6101fa52b666 + .quad 0x2c863b00afaf53d5 + .quad 0x81214e06db096ab8 + .quad 0x21a8b6c90ce44f35 + .quad 0x6524c12a409e2af5 + .quad 0x0165b5a48efca481 + .quad 0xf190a474a0846a76 + .quad 0x12eff984cd2f7cc0 + .quad 0x695e290658aa2b8f + .quad 0x591b67d9bffec8b8 + + // 2^16 * 5 * B + + .quad 0x312f0d1c80b49bfa + .quad 0x5979515eabf3ec8a + .quad 0x727033c09ef01c88 + .quad 0x3de02ec7ca8f7bcb + .quad 0x99b9b3719f18b55d + .quad 0xe465e5faa18c641e + .quad 0x61081136c29f05ed + .quad 0x489b4f867030128b + .quad 0xd232102d3aeb92ef + .quad 0xe16253b46116a861 + .quad 0x3d7eabe7190baa24 + .quad 0x49f5fbba496cbebf + + // 2^16 * 6 * B + + .quad 0x30949a108a5bcfd4 + .quad 0xdc40dd70bc6473eb + .quad 0x92c294c1307c0d1c + .quad 0x5604a86dcbfa6e74 + .quad 0x155d628c1e9c572e + .quad 0x8a4d86acc5884741 + .quad 0x91a352f6515763eb + .quad 0x06a1a6c28867515b + .quad 0x7288d1d47c1764b6 + .quad 0x72541140e0418b51 + .quad 0x9f031a6018acf6d1 + .quad 0x20989e89fe2742c6 + + // 2^16 * 7 * B + + .quad 0x499777fd3a2dcc7f + .quad 0x32857c2ca54fd892 + .quad 0xa279d864d207e3a0 + .quad 0x0403ed1d0ca67e29 + .quad 0x1674278b85eaec2e + .quad 0x5621dc077acb2bdf + .quad 0x640a4c1661cbf45a + .quad 0x730b9950f70595d3 + .quad 0xc94b2d35874ec552 + .quad 0xc5e6c8cf98246f8d + .quad 0xf7cb46fa16c035ce + .quad 0x5bd7454308303dcc + + // 2^16 * 8 * B + + .quad 0x7f9ad19528b24cc2 + .quad 0x7f6b54656335c181 + .quad 0x66b8b66e4fc07236 + .quad 0x133a78007380ad83 + .quad 0x85c4932115e7792a + .quad 0xc64c89a2bdcdddc9 + .quad 0x9d1e3da8ada3d762 + .quad 0x5bb7db123067f82c + .quad 0x0961f467c6ca62be + .quad 0x04ec21d6211952ee + .quad 0x182360779bd54770 + .quad 0x740dca6d58f0e0d2 + + // 2^20 * 1 * B + + .quad 0x50b70bf5d3f0af0b + .quad 0x4feaf48ae32e71f7 + .quad 0x60e84ed3a55bbd34 + .quad 0x00ed489b3f50d1ed + .quad 0x3906c72aed261ae5 + .quad 0x9ab68fd988e100f7 + .quad 0xf5e9059af3360197 + .quad 0x0e53dc78bf2b6d47 + .quad 0xb90829bf7971877a + .quad 0x5e4444636d17e631 + .quad 0x4d05c52e18276893 + .quad 0x27632d9a5a4a4af5 + + // 2^20 * 2 * B + + .quad 0xd11ff05154b260ce + .quad 0xd86dc38e72f95270 + .quad 0x601fcd0d267cc138 + .quad 0x2b67916429e90ccd + .quad 0xa98285d187eaffdb + .quad 0xa5b4fbbbd8d0a864 + .quad 0xb658f27f022663f7 + .quad 0x3bbc2b22d99ce282 + .quad 0xb917c952583c0a58 + .quad 0x653ff9b80fe4c6f3 + .quad 0x9b0da7d7bcdf3c0c + .quad 0x43a0eeb6ab54d60e + + // 2^20 * 3 * B + + .quad 0x396966a46d4a5487 + .quad 0xf811a18aac2bb3ba + .quad 0x66e4685b5628b26b + .quad 0x70a477029d929b92 + .quad 0x3ac6322357875fe8 + .quad 0xd9d4f4ecf5fbcb8f + .quad 0x8dee8493382bb620 + .quad 0x50c5eaa14c799fdc + .quad 0xdd0edc8bd6f2fb3c + .quad 0x54c63aa79cc7b7a0 + .quad 0xae0b032b2c8d9f1a + .quad 0x6f9ce107602967fb + + // 2^20 * 4 * B + + .quad 0xad1054b1cde1c22a + .quad 0xc4a8e90248eb32df + .quad 0x5f3e7b33accdc0ea + .quad 0x72364713fc79963e + .quad 0x139693063520e0b5 + .quad 0x437fcf7c88ea03fe + .quad 0xf7d4c40bd3c959bc + .quad 0x699154d1f893ded9 + .quad 0x315d5c75b4b27526 + .quad 0xcccb842d0236daa5 + .quad 0x22f0c8a3345fee8e + .quad 0x73975a617d39dbed + + // 2^20 * 5 * B + + .quad 0xe4024df96375da10 + .quad 0x78d3251a1830c870 + .quad 0x902b1948658cd91c + .quad 0x7e18b10b29b7438a + .quad 0x6f37f392f4433e46 + .quad 0x0e19b9a11f566b18 + .quad 0x220fb78a1fd1d662 + .quad 0x362a4258a381c94d + .quad 0x9071d9132b6beb2f + .quad 0x0f26e9ad28418247 + .quad 0xeab91ec9bdec925d + .quad 0x4be65bc8f48af2de + + // 2^20 * 6 * B + + .quad 0x78487feba36e7028 + .quad 0x5f3f13001dd8ce34 + .quad 0x934fb12d4b30c489 + .quad 0x056c244d397f0a2b + .quad 0x1d50fba257c26234 + .quad 0x7bd4823adeb0678b + .quad 0xc2b0dc6ea6538af5 + .quad 0x5665eec6351da73e + .quad 0xdb3ee00943bfb210 + .quad 0x4972018720800ac2 + .quad 0x26ab5d6173bd8667 + .quad 0x20b209c2ab204938 + + // 2^20 * 7 * B + + .quad 0x549e342ac07fb34b + .quad 0x02d8220821373d93 + .quad 0xbc262d70acd1f567 + .quad 0x7a92c9fdfbcac784 + .quad 0x1fcca94516bd3289 + .quad 0x448d65aa41420428 + .quad 0x59c3b7b216a55d62 + .quad 0x49992cc64e612cd8 + .quad 0x65bd1bea70f801de + .quad 0x1befb7c0fe49e28a + .quad 0xa86306cdb1b2ae4a + .quad 0x3b7ac0cd265c2a09 + + // 2^20 * 8 * B + + .quad 0x822bee438c01bcec + .quad 0x530cb525c0fbc73b + .quad 0x48519034c1953fe9 + .quad 0x265cc261e09a0f5b + .quad 0xf0d54e4f22ed39a7 + .quad 0xa2aae91e5608150a + .quad 0xf421b2e9eddae875 + .quad 0x31bc531d6b7de992 + .quad 0xdf3d134da980f971 + .quad 0x7a4fb8d1221a22a7 + .quad 0x3df7d42035aad6d8 + .quad 0x2a14edcc6a1a125e + + // 2^24 * 1 * B + + .quad 0xdf48ee0752cfce4e + .quad 0xc3fffaf306ec08b7 + .quad 0x05710b2ab95459c4 + .quad 0x161d25fa963ea38d + .quad 0x231a8c570478433c + .quad 0xb7b5270ec281439d + .quad 0xdbaa99eae3d9079f + .quad 0x2c03f5256c2b03d9 + .quad 0x790f18757b53a47d + .quad 0x307b0130cf0c5879 + .quad 0x31903d77257ef7f9 + .quad 0x699468bdbd96bbaf + + // 2^24 * 2 * B + + .quad 0xbd1f2f46f4dafecf + .quad 0x7cef0114a47fd6f7 + .quad 0xd31ffdda4a47b37f + .quad 0x525219a473905785 + .quad 0xd8dd3de66aa91948 + .quad 0x485064c22fc0d2cc + .quad 0x9b48246634fdea2f + .quad 0x293e1c4e6c4a2e3a + .quad 0x376e134b925112e1 + .quad 0x703778b5dca15da0 + .quad 0xb04589af461c3111 + .quad 0x5b605c447f032823 + + // 2^24 * 3 * B + + .quad 0xb965805920c47c89 + .quad 0xe7f0100c923b8fcc + .quad 0x0001256502e2ef77 + .quad 0x24a76dcea8aeb3ee + .quad 0x3be9fec6f0e7f04c + .quad 0x866a579e75e34962 + .quad 0x5542ef161e1de61a + .quad 0x2f12fef4cc5abdd5 + .quad 0x0a4522b2dfc0c740 + .quad 0x10d06e7f40c9a407 + .quad 0xc6cf144178cff668 + .quad 0x5e607b2518a43790 + + // 2^24 * 4 * B + + .quad 0x58b31d8f6cdf1818 + .quad 0x35cfa74fc36258a2 + .quad 0xe1b3ff4f66e61d6e + .quad 0x5067acab6ccdd5f7 + .quad 0xa02c431ca596cf14 + .quad 0xe3c42d40aed3e400 + .quad 0xd24526802e0f26db + .quad 0x201f33139e457068 + .quad 0xfd527f6b08039d51 + .quad 0x18b14964017c0006 + .quad 0xd5220eb02e25a4a8 + .quad 0x397cba8862460375 + + // 2^24 * 5 * B + + .quad 0x30c13093f05959b2 + .quad 0xe23aa18de9a97976 + .quad 0x222fd491721d5e26 + .quad 0x2339d320766e6c3a + .quad 0x7815c3fbc81379e7 + .quad 0xa6619420dde12af1 + .quad 0xffa9c0f885a8fdd5 + .quad 0x771b4022c1e1c252 + .quad 0xd87dd986513a2fa7 + .quad 0xf5ac9b71f9d4cf08 + .quad 0xd06bc31b1ea283b3 + .quad 0x331a189219971a76 + + // 2^24 * 6 * B + + .quad 0xf5166f45fb4f80c6 + .quad 0x9c36c7de61c775cf + .quad 0xe3d4e81b9041d91c + .quad 0x31167c6b83bdfe21 + .quad 0x26512f3a9d7572af + .quad 0x5bcbe28868074a9e + .quad 0x84edc1c11180f7c4 + .quad 0x1ac9619ff649a67b + .quad 0xf22b3842524b1068 + .quad 0x5068343bee9ce987 + .quad 0xfc9d71844a6250c8 + .quad 0x612436341f08b111 + + // 2^24 * 7 * B + + .quad 0xd99d41db874e898d + .quad 0x09fea5f16c07dc20 + .quad 0x793d2c67d00f9bbc + .quad 0x46ebe2309e5eff40 + .quad 0x8b6349e31a2d2638 + .quad 0x9ddfb7009bd3fd35 + .quad 0x7f8bf1b8a3a06ba4 + .quad 0x1522aa3178d90445 + .quad 0x2c382f5369614938 + .quad 0xdafe409ab72d6d10 + .quad 0xe8c83391b646f227 + .quad 0x45fe70f50524306c + + // 2^24 * 8 * B + + .quad 0xda4875a6960c0b8c + .quad 0x5b68d076ef0e2f20 + .quad 0x07fb51cf3d0b8fd4 + .quad 0x428d1623a0e392d4 + .quad 0x62f24920c8951491 + .quad 0x05f007c83f630ca2 + .quad 0x6fbb45d2f5c9d4b8 + .quad 0x16619f6db57a2245 + .quad 0x084f4a4401a308fd + .quad 0xa82219c376a5caac + .quad 0xdeb8de4643d1bc7d + .quad 0x1d81592d60bd38c6 + + // 2^28 * 1 * B + + .quad 0xd833d7beec2a4c38 + .quad 0x2c9162830acc20ed + .quad 0xe93a47aa92df7581 + .quad 0x702d67a3333c4a81 + .quad 0x3a4a369a2f89c8a1 + .quad 0x63137a1d7c8de80d + .quad 0xbcac008a78eda015 + .quad 0x2cb8b3a5b483b03f + .quad 0x36e417cbcb1b90a1 + .quad 0x33b3ddaa7f11794e + .quad 0x3f510808885bc607 + .quad 0x24141dc0e6a8020d + + // 2^28 * 2 * B + + .quad 0x59f73c773fefee9d + .quad 0xb3f1ef89c1cf989d + .quad 0xe35dfb42e02e545f + .quad 0x5766120b47a1b47c + .quad 0x91925dccbd83157d + .quad 0x3ca1205322cc8094 + .quad 0x28e57f183f90d6e4 + .quad 0x1a4714cede2e767b + .quad 0xdb20ba0fb8b6b7ff + .quad 0xb732c3b677511fa1 + .quad 0xa92b51c099f02d89 + .quad 0x4f3875ad489ca5f1 + + // 2^28 * 3 * B + + .quad 0xc7fc762f4932ab22 + .quad 0x7ac0edf72f4c3c1b + .quad 0x5f6b55aa9aa895e8 + .quad 0x3680274dad0a0081 + .quad 0x79ed13f6ee73eec0 + .quad 0xa5c6526d69110bb1 + .quad 0xe48928c38603860c + .quad 0x722a1446fd7059f5 + .quad 0xd0959fe9a8cf8819 + .quad 0xd0a995508475a99c + .quad 0x6eac173320b09cc5 + .quad 0x628ecf04331b1095 + + // 2^28 * 4 * B + + .quad 0x98bcb118a9d0ddbc + .quad 0xee449e3408b4802b + .quad 0x87089226b8a6b104 + .quad 0x685f349a45c7915d + .quad 0x9b41acf85c74ccf1 + .quad 0xb673318108265251 + .quad 0x99c92aed11adb147 + .quad 0x7a47d70d34ecb40f + .quad 0x60a0c4cbcc43a4f5 + .quad 0x775c66ca3677bea9 + .quad 0xa17aa1752ff8f5ed + .quad 0x11ded9020e01fdc0 + + // 2^28 * 5 * B + + .quad 0x890e7809caefe704 + .quad 0x8728296de30e8c6c + .quad 0x4c5cd2a392aeb1c9 + .quad 0x194263d15771531f + .quad 0x471f95b03bea93b7 + .quad 0x0552d7d43313abd3 + .quad 0xbd9370e2e17e3f7b + .quad 0x7b120f1db20e5bec + .quad 0x17d2fb3d86502d7a + .quad 0xb564d84450a69352 + .quad 0x7da962c8a60ed75d + .quad 0x00d0f85b318736aa + + // 2^28 * 6 * B + + .quad 0x978b142e777c84fd + .quad 0xf402644705a8c062 + .quad 0xa67ad51be7e612c7 + .quad 0x2f7b459698dd6a33 + .quad 0xa6753c1efd7621c1 + .quad 0x69c0b4a7445671f5 + .quad 0x971f527405b23c11 + .quad 0x387bc74851a8c7cd + .quad 0x81894b4d4a52a9a8 + .quad 0xadd93e12f6b8832f + .quad 0x184d8548b61bd638 + .quad 0x3f1c62dbd6c9f6cd + + // 2^28 * 7 * B + + .quad 0x2e8f1f0091910c1f + .quad 0xa4df4fe0bff2e12c + .quad 0x60c6560aee927438 + .quad 0x6338283facefc8fa + .quad 0x3fad3e40148f693d + .quad 0x052656e194eb9a72 + .quad 0x2f4dcbfd184f4e2f + .quad 0x406f8db1c482e18b + .quad 0x9e630d2c7f191ee4 + .quad 0x4fbf8301bc3ff670 + .quad 0x787d8e4e7afb73c4 + .quad 0x50d83d5be8f58fa5 + + // 2^28 * 8 * B + + .quad 0x85683916c11a1897 + .quad 0x2d69a4efe506d008 + .quad 0x39af1378f664bd01 + .quad 0x65942131361517c6 + .quad 0xc0accf90b4d3b66d + .quad 0xa7059de561732e60 + .quad 0x033d1f7870c6b0ba + .quad 0x584161cd26d946e4 + .quad 0xbbf2b1a072d27ca2 + .quad 0xbf393c59fbdec704 + .quad 0xe98dbbcee262b81e + .quad 0x02eebd0b3029b589 + + // 2^32 * 1 * B + + .quad 0x61368756a60dac5f + .quad 0x17e02f6aebabdc57 + .quad 0x7f193f2d4cce0f7d + .quad 0x20234a7789ecdcf0 + .quad 0x8765b69f7b85c5e8 + .quad 0x6ff0678bd168bab2 + .quad 0x3a70e77c1d330f9b + .quad 0x3a5f6d51b0af8e7c + .quad 0x76d20db67178b252 + .quad 0x071c34f9d51ed160 + .quad 0xf62a4a20b3e41170 + .quad 0x7cd682353cffe366 + + // 2^32 * 2 * B + + .quad 0x0be1a45bd887fab6 + .quad 0x2a846a32ba403b6e + .quad 0xd9921012e96e6000 + .quad 0x2838c8863bdc0943 + .quad 0xa665cd6068acf4f3 + .quad 0x42d92d183cd7e3d3 + .quad 0x5759389d336025d9 + .quad 0x3ef0253b2b2cd8ff + .quad 0xd16bb0cf4a465030 + .quad 0xfa496b4115c577ab + .quad 0x82cfae8af4ab419d + .quad 0x21dcb8a606a82812 + + // 2^32 * 3 * B + + .quad 0x5c6004468c9d9fc8 + .quad 0x2540096ed42aa3cb + .quad 0x125b4d4c12ee2f9c + .quad 0x0bc3d08194a31dab + .quad 0x9a8d00fabe7731ba + .quad 0x8203607e629e1889 + .quad 0xb2cc023743f3d97f + .quad 0x5d840dbf6c6f678b + .quad 0x706e380d309fe18b + .quad 0x6eb02da6b9e165c7 + .quad 0x57bbba997dae20ab + .quad 0x3a4276232ac196dd + + // 2^32 * 4 * B + + .quad 0x4b42432c8a7084fa + .quad 0x898a19e3dfb9e545 + .quad 0xbe9f00219c58e45d + .quad 0x1ff177cea16debd1 + .quad 0x3bf8c172db447ecb + .quad 0x5fcfc41fc6282dbd + .quad 0x80acffc075aa15fe + .quad 0x0770c9e824e1a9f9 + .quad 0xcf61d99a45b5b5fd + .quad 0x860984e91b3a7924 + .quad 0xe7300919303e3e89 + .quad 0x39f264fd41500b1e + + // 2^32 * 5 * B + + .quad 0xa7ad3417dbe7e29c + .quad 0xbd94376a2b9c139c + .quad 0xa0e91b8e93597ba9 + .quad 0x1712d73468889840 + .quad 0xd19b4aabfe097be1 + .quad 0xa46dfce1dfe01929 + .quad 0xc3c908942ca6f1ff + .quad 0x65c621272c35f14e + .quad 0xe72b89f8ce3193dd + .quad 0x4d103356a125c0bb + .quad 0x0419a93d2e1cfe83 + .quad 0x22f9800ab19ce272 + + // 2^32 * 6 * B + + .quad 0x605a368a3e9ef8cb + .quad 0xe3e9c022a5504715 + .quad 0x553d48b05f24248f + .quad 0x13f416cd647626e5 + .quad 0x42029fdd9a6efdac + .quad 0xb912cebe34a54941 + .quad 0x640f64b987bdf37b + .quad 0x4171a4d38598cab4 + .quad 0xfa2758aa99c94c8c + .quad 0x23006f6fb000b807 + .quad 0xfbd291ddadda5392 + .quad 0x508214fa574bd1ab + + // 2^32 * 7 * B + + .quad 0xc20269153ed6fe4b + .quad 0xa65a6739511d77c4 + .quad 0xcbde26462c14af94 + .quad 0x22f960ec6faba74b + .quad 0x461a15bb53d003d6 + .quad 0xb2102888bcf3c965 + .quad 0x27c576756c683a5a + .quad 0x3a7758a4c86cb447 + .quad 0x548111f693ae5076 + .quad 0x1dae21df1dfd54a6 + .quad 0x12248c90f3115e65 + .quad 0x5d9fd15f8de7f494 + + // 2^32 * 8 * B + + .quad 0x031408d36d63727f + .quad 0x6a379aefd7c7b533 + .quad 0xa9e18fc5ccaee24b + .quad 0x332f35914f8fbed3 + .quad 0x3f244d2aeed7521e + .quad 0x8e3a9028432e9615 + .quad 0xe164ba772e9c16d4 + .quad 0x3bc187fa47eb98d8 + .quad 0x6d470115ea86c20c + .quad 0x998ab7cb6c46d125 + .quad 0xd77832b53a660188 + .quad 0x450d81ce906fba03 + + // 2^36 * 1 * B + + .quad 0xf8ae4d2ad8453902 + .quad 0x7018058ee8db2d1d + .quad 0xaab3995fc7d2c11e + .quad 0x53b16d2324ccca79 + .quad 0x23264d66b2cae0b5 + .quad 0x7dbaed33ebca6576 + .quad 0x030ebed6f0d24ac8 + .quad 0x2a887f78f7635510 + .quad 0x2a23b9e75c012d4f + .quad 0x0c974651cae1f2ea + .quad 0x2fb63273675d70ca + .quad 0x0ba7250b864403f5 + + // 2^36 * 2 * B + + .quad 0xbb0d18fd029c6421 + .quad 0xbc2d142189298f02 + .quad 0x8347f8e68b250e96 + .quad 0x7b9f2fe8032d71c9 + .quad 0xdd63589386f86d9c + .quad 0x61699176e13a85a4 + .quad 0x2e5111954eaa7d57 + .quad 0x32c21b57fb60bdfb + .quad 0xd87823cd319e0780 + .quad 0xefc4cfc1897775c5 + .quad 0x4854fb129a0ab3f7 + .quad 0x12c49d417238c371 + + // 2^36 * 3 * B + + .quad 0x0950b533ffe83769 + .quad 0x21861c1d8e1d6bd1 + .quad 0xf022d8381302e510 + .quad 0x2509200c6391cab4 + .quad 0x09b3a01783799542 + .quad 0x626dd08faad5ee3f + .quad 0xba00bceeeb70149f + .quad 0x1421b246a0a444c9 + .quad 0x4aa43a8e8c24a7c7 + .quad 0x04c1f540d8f05ef5 + .quad 0xadba5e0c0b3eb9dc + .quad 0x2ab5504448a49ce3 + + // 2^36 * 4 * B + + .quad 0x2ed227266f0f5dec + .quad 0x9824ee415ed50824 + .quad 0x807bec7c9468d415 + .quad 0x7093bae1b521e23f + .quad 0xdc07ac631c5d3afa + .quad 0x58615171f9df8c6c + .quad 0x72a079d89d73e2b0 + .quad 0x7301f4ceb4eae15d + .quad 0x6409e759d6722c41 + .quad 0xa674e1cf72bf729b + .quad 0xbc0a24eb3c21e569 + .quad 0x390167d24ebacb23 + + // 2^36 * 5 * B + + .quad 0x27f58e3bba353f1c + .quad 0x4c47764dbf6a4361 + .quad 0xafbbc4e56e562650 + .quad 0x07db2ee6aae1a45d + .quad 0xd7bb054ba2f2120b + .quad 0xe2b9ceaeb10589b7 + .quad 0x3fe8bac8f3c0edbe + .quad 0x4cbd40767112cb69 + .quad 0x0b603cc029c58176 + .quad 0x5988e3825cb15d61 + .quad 0x2bb61413dcf0ad8d + .quad 0x7b8eec6c74183287 + + // 2^36 * 6 * B + + .quad 0xe4ca40782cd27cb0 + .quad 0xdaf9c323fbe967bd + .quad 0xb29bd34a8ad41e9e + .quad 0x72810497626ede4d + .quad 0x32fee570fc386b73 + .quad 0xda8b0141da3a8cc7 + .quad 0x975ffd0ac8968359 + .quad 0x6ee809a1b132a855 + .quad 0x9444bb31fcfd863a + .quad 0x2fe3690a3e4e48c5 + .quad 0xdc29c867d088fa25 + .quad 0x13bd1e38d173292e + + // 2^36 * 7 * B + + .quad 0xd32b4cd8696149b5 + .quad 0xe55937d781d8aab7 + .quad 0x0bcb2127ae122b94 + .quad 0x41e86fcfb14099b0 + .quad 0x223fb5cf1dfac521 + .quad 0x325c25316f554450 + .quad 0x030b98d7659177ac + .quad 0x1ed018b64f88a4bd + .quad 0x3630dfa1b802a6b0 + .quad 0x880f874742ad3bd5 + .quad 0x0af90d6ceec5a4d4 + .quad 0x746a247a37cdc5d9 + + // 2^36 * 8 * B + + .quad 0xd531b8bd2b7b9af6 + .quad 0x5005093537fc5b51 + .quad 0x232fcf25c593546d + .quad 0x20a365142bb40f49 + .quad 0x6eccd85278d941ed + .quad 0x2254ae83d22f7843 + .quad 0xc522d02e7bbfcdb7 + .quad 0x681e3351bff0e4e2 + .quad 0x8b64b59d83034f45 + .quad 0x2f8b71f21fa20efb + .quad 0x69249495ba6550e4 + .quad 0x539ef98e45d5472b + + // 2^40 * 1 * B + + .quad 0x6e7bb6a1a6205275 + .quad 0xaa4f21d7413c8e83 + .quad 0x6f56d155e88f5cb2 + .quad 0x2de25d4ba6345be1 + .quad 0xd074d8961cae743f + .quad 0xf86d18f5ee1c63ed + .quad 0x97bdc55be7f4ed29 + .quad 0x4cbad279663ab108 + .quad 0x80d19024a0d71fcd + .quad 0xc525c20afb288af8 + .quad 0xb1a3974b5f3a6419 + .quad 0x7d7fbcefe2007233 + + // 2^40 * 2 * B + + .quad 0xfaef1e6a266b2801 + .quad 0x866c68c4d5739f16 + .quad 0xf68a2fbc1b03762c + .quad 0x5975435e87b75a8d + .quad 0xcd7c5dc5f3c29094 + .quad 0xc781a29a2a9105ab + .quad 0x80c61d36421c3058 + .quad 0x4f9cd196dcd8d4d7 + .quad 0x199297d86a7b3768 + .quad 0xd0d058241ad17a63 + .quad 0xba029cad5c1c0c17 + .quad 0x7ccdd084387a0307 + + // 2^40 * 3 * B + + .quad 0xdca6422c6d260417 + .quad 0xae153d50948240bd + .quad 0xa9c0c1b4fb68c677 + .quad 0x428bd0ed61d0cf53 + .quad 0x9b0c84186760cc93 + .quad 0xcdae007a1ab32a99 + .quad 0xa88dec86620bda18 + .quad 0x3593ca848190ca44 + .quad 0x9213189a5e849aa7 + .quad 0xd4d8c33565d8facd + .quad 0x8c52545b53fdbbd1 + .quad 0x27398308da2d63e6 + + // 2^40 * 4 * B + + .quad 0x42c38d28435ed413 + .quad 0xbd50f3603278ccc9 + .quad 0xbb07ab1a79da03ef + .quad 0x269597aebe8c3355 + .quad 0xb9a10e4c0a702453 + .quad 0x0fa25866d57d1bde + .quad 0xffb9d9b5cd27daf7 + .quad 0x572c2945492c33fd + .quad 0xc77fc745d6cd30be + .quad 0xe4dfe8d3e3baaefb + .quad 0xa22c8830aa5dda0c + .quad 0x7f985498c05bca80 + + // 2^40 * 5 * B + + .quad 0x3849ce889f0be117 + .quad 0x8005ad1b7b54a288 + .quad 0x3da3c39f23fc921c + .quad 0x76c2ec470a31f304 + .quad 0xd35615520fbf6363 + .quad 0x08045a45cf4dfba6 + .quad 0xeec24fbc873fa0c2 + .quad 0x30f2653cd69b12e7 + .quad 0x8a08c938aac10c85 + .quad 0x46179b60db276bcb + .quad 0xa920c01e0e6fac70 + .quad 0x2f1273f1596473da + + // 2^40 * 6 * B + + .quad 0x4739fc7c8ae01e11 + .quad 0xfd5274904a6aab9f + .quad 0x41d98a8287728f2e + .quad 0x5d9e572ad85b69f2 + .quad 0x30488bd755a70bc0 + .quad 0x06d6b5a4f1d442e7 + .quad 0xead1a69ebc596162 + .quad 0x38ac1997edc5f784 + .quad 0x0666b517a751b13b + .quad 0x747d06867e9b858c + .quad 0xacacc011454dde49 + .quad 0x22dfcd9cbfe9e69c + + // 2^40 * 7 * B + + .quad 0x8ddbd2e0c30d0cd9 + .quad 0xad8e665facbb4333 + .quad 0x8f6b258c322a961f + .quad 0x6b2916c05448c1c7 + .quad 0x56ec59b4103be0a1 + .quad 0x2ee3baecd259f969 + .quad 0x797cb29413f5cd32 + .quad 0x0fe9877824cde472 + .quad 0x7edb34d10aba913b + .quad 0x4ea3cd822e6dac0e + .quad 0x66083dff6578f815 + .quad 0x4c303f307ff00a17 + + // 2^40 * 8 * B + + .quad 0xd30a3bd617b28c85 + .quad 0xc5d377b739773bea + .quad 0xc6c6e78c1e6a5cbf + .quad 0x0d61b8f78b2ab7c4 + .quad 0x29fc03580dd94500 + .quad 0xecd27aa46fbbec93 + .quad 0x130a155fc2e2a7f8 + .quad 0x416b151ab706a1d5 + .quad 0x56a8d7efe9c136b0 + .quad 0xbd07e5cd58e44b20 + .quad 0xafe62fda1b57e0ab + .quad 0x191a2af74277e8d2 + + // 2^44 * 1 * B + + .quad 0xd550095bab6f4985 + .quad 0x04f4cd5b4fbfaf1a + .quad 0x9d8e2ed12a0c7540 + .quad 0x2bc24e04b2212286 + .quad 0x09d4b60b2fe09a14 + .quad 0xc384f0afdbb1747e + .quad 0x58e2ea8978b5fd6e + .quad 0x519ef577b5e09b0a + .quad 0x1863d7d91124cca9 + .quad 0x7ac08145b88a708e + .quad 0x2bcd7309857031f5 + .quad 0x62337a6e8ab8fae5 + + // 2^44 * 2 * B + + .quad 0x4bcef17f06ffca16 + .quad 0xde06e1db692ae16a + .quad 0x0753702d614f42b0 + .quad 0x5f6041b45b9212d0 + .quad 0xd1ab324e1b3a1273 + .quad 0x18947cf181055340 + .quad 0x3b5d9567a98c196e + .quad 0x7fa00425802e1e68 + .quad 0x7d531574028c2705 + .quad 0x80317d69db0d75fe + .quad 0x30fface8ef8c8ddd + .quad 0x7e9de97bb6c3e998 + + // 2^44 * 3 * B + + .quad 0x1558967b9e6585a3 + .quad 0x97c99ce098e98b92 + .quad 0x10af149b6eb3adad + .quad 0x42181fe8f4d38cfa + .quad 0xf004be62a24d40dd + .quad 0xba0659910452d41f + .quad 0x81c45ee162a44234 + .quad 0x4cb829d8a22266ef + .quad 0x1dbcaa8407b86681 + .quad 0x081f001e8b26753b + .quad 0x3cd7ce6a84048e81 + .quad 0x78af11633f25f22c + + // 2^44 * 4 * B + + .quad 0x8416ebd40b50babc + .quad 0x1508722628208bee + .quad 0xa3148fafb9c1c36d + .quad 0x0d07daacd32d7d5d + .quad 0x3241c00e7d65318c + .quad 0xe6bee5dcd0e86de7 + .quad 0x118b2dc2fbc08c26 + .quad 0x680d04a7fc603dc3 + .quad 0xf9c2414a695aa3eb + .quad 0xdaa42c4c05a68f21 + .quad 0x7c6c23987f93963e + .quad 0x210e8cd30c3954e3 + + // 2^44 * 5 * B + + .quad 0xac4201f210a71c06 + .quad 0x6a65e0aef3bfb021 + .quad 0xbc42c35c393632f7 + .quad 0x56ea8db1865f0742 + .quad 0x2b50f16137fe6c26 + .quad 0xe102bcd856e404d8 + .quad 0x12b0f1414c561f6b + .quad 0x51b17bc8d028ec91 + .quad 0xfff5fb4bcf535119 + .quad 0xf4989d79df1108a0 + .quad 0xbdfcea659a3ba325 + .quad 0x18a11f1174d1a6f2 + + // 2^44 * 6 * B + + .quad 0x407375ab3f6bba29 + .quad 0x9ec3b6d8991e482e + .quad 0x99c80e82e55f92e9 + .quad 0x307c13b6fb0c0ae1 + .quad 0xfbd63cdad27a5f2c + .quad 0xf00fc4bc8aa106d7 + .quad 0x53fb5c1a8e64a430 + .quad 0x04eaabe50c1a2e85 + .quad 0x24751021cb8ab5e7 + .quad 0xfc2344495c5010eb + .quad 0x5f1e717b4e5610a1 + .quad 0x44da5f18c2710cd5 + + // 2^44 * 7 * B + + .quad 0x033cc55ff1b82eb5 + .quad 0xb15ae36d411cae52 + .quad 0xba40b6198ffbacd3 + .quad 0x768edce1532e861f + .quad 0x9156fe6b89d8eacc + .quad 0xe6b79451e23126a1 + .quad 0xbd7463d93944eb4e + .quad 0x726373f6767203ae + .quad 0xe305ca72eb7ef68a + .quad 0x662cf31f70eadb23 + .quad 0x18f026fdb4c45b68 + .quad 0x513b5384b5d2ecbd + + // 2^44 * 8 * B + + .quad 0x46d46280c729989e + .quad 0x4b93fbd05368a5dd + .quad 0x63df3f81d1765a89 + .quad 0x34cebd64b9a0a223 + .quad 0x5e2702878af34ceb + .quad 0x900b0409b946d6ae + .quad 0x6512ebf7dabd8512 + .quad 0x61d9b76988258f81 + .quad 0xa6c5a71349b7d94b + .quad 0xa3f3d15823eb9446 + .quad 0x0416fbd277484834 + .quad 0x69d45e6f2c70812f + + // 2^48 * 1 * B + + .quad 0xce16f74bc53c1431 + .quad 0x2b9725ce2072edde + .quad 0xb8b9c36fb5b23ee7 + .quad 0x7e2e0e450b5cc908 + .quad 0x9fe62b434f460efb + .quad 0xded303d4a63607d6 + .quad 0xf052210eb7a0da24 + .quad 0x237e7dbe00545b93 + .quad 0x013575ed6701b430 + .quad 0x231094e69f0bfd10 + .quad 0x75320f1583e47f22 + .quad 0x71afa699b11155e3 + + // 2^48 * 2 * B + + .quad 0x65ce6f9b3953b61d + .quad 0xc65839eaafa141e6 + .quad 0x0f435ffda9f759fe + .quad 0x021142e9c2b1c28e + .quad 0xea423c1c473b50d6 + .quad 0x51e87a1f3b38ef10 + .quad 0x9b84bf5fb2c9be95 + .quad 0x00731fbc78f89a1c + .quad 0xe430c71848f81880 + .quad 0xbf960c225ecec119 + .quad 0xb6dae0836bba15e3 + .quad 0x4c4d6f3347e15808 + + // 2^48 * 3 * B + + .quad 0x18f7eccfc17d1fc9 + .quad 0x6c75f5a651403c14 + .quad 0xdbde712bf7ee0cdf + .quad 0x193fddaaa7e47a22 + .quad 0x2f0cddfc988f1970 + .quad 0x6b916227b0b9f51b + .quad 0x6ec7b6c4779176be + .quad 0x38bf9500a88f9fa8 + .quad 0x1fd2c93c37e8876f + .quad 0xa2f61e5a18d1462c + .quad 0x5080f58239241276 + .quad 0x6a6fb99ebf0d4969 + + // 2^48 * 4 * B + + .quad 0x6a46c1bb560855eb + .quad 0x2416bb38f893f09d + .quad 0xd71d11378f71acc1 + .quad 0x75f76914a31896ea + .quad 0xeeb122b5b6e423c6 + .quad 0x939d7010f286ff8e + .quad 0x90a92a831dcf5d8c + .quad 0x136fda9f42c5eb10 + .quad 0xf94cdfb1a305bdd1 + .quad 0x0f364b9d9ff82c08 + .quad 0x2a87d8a5c3bb588a + .quad 0x022183510be8dcba + + // 2^48 * 5 * B + + .quad 0x4af766385ead2d14 + .quad 0xa08ed880ca7c5830 + .quad 0x0d13a6e610211e3d + .quad 0x6a071ce17b806c03 + .quad 0x9d5a710143307a7f + .quad 0xb063de9ec47da45f + .quad 0x22bbfe52be927ad3 + .quad 0x1387c441fd40426c + .quad 0xb5d3c3d187978af8 + .quad 0x722b5a3d7f0e4413 + .quad 0x0d7b4848bb477ca0 + .quad 0x3171b26aaf1edc92 + + // 2^48 * 6 * B + + .quad 0xa92f319097564ca8 + .quad 0xff7bb84c2275e119 + .quad 0x4f55fe37a4875150 + .quad 0x221fd4873cf0835a + .quad 0xa60db7d8b28a47d1 + .quad 0xa6bf14d61770a4f1 + .quad 0xd4a1f89353ddbd58 + .quad 0x6c514a63344243e9 + .quad 0x2322204f3a156341 + .quad 0xfb73e0e9ba0a032d + .quad 0xfce0dd4c410f030e + .quad 0x48daa596fb924aaa + + // 2^48 * 7 * B + + .quad 0x6eca8e665ca59cc7 + .quad 0xa847254b2e38aca0 + .quad 0x31afc708d21e17ce + .quad 0x676dd6fccad84af7 + .quad 0x14f61d5dc84c9793 + .quad 0x9941f9e3ef418206 + .quad 0xcdf5b88f346277ac + .quad 0x58c837fa0e8a79a9 + .quad 0x0cf9688596fc9058 + .quad 0x1ddcbbf37b56a01b + .quad 0xdcc2e77d4935d66a + .quad 0x1c4f73f2c6a57f0a + + // 2^48 * 8 * B + + .quad 0x0e7a4fbd305fa0bb + .quad 0x829d4ce054c663ad + .quad 0xf421c3832fe33848 + .quad 0x795ac80d1bf64c42 + .quad 0xb36e706efc7c3484 + .quad 0x73dfc9b4c3c1cf61 + .quad 0xeb1d79c9781cc7e5 + .quad 0x70459adb7daf675c + .quad 0x1b91db4991b42bb3 + .quad 0x572696234b02dcca + .quad 0x9fdf9ee51f8c78dc + .quad 0x5fe162848ce21fd3 + + // 2^52 * 1 * B + + .quad 0xe2790aae4d077c41 + .quad 0x8b938270db7469a3 + .quad 0x6eb632dc8abd16a2 + .quad 0x720814ecaa064b72 + .quad 0x315c29c795115389 + .quad 0xd7e0e507862f74ce + .quad 0x0c4a762185927432 + .quad 0x72de6c984a25a1e4 + .quad 0xae9ab553bf6aa310 + .quad 0x050a50a9806d6e1b + .quad 0x92bb7403adff5139 + .quad 0x0394d27645be618b + + // 2^52 * 2 * B + + .quad 0x4d572251857eedf4 + .quad 0xe3724edde19e93c5 + .quad 0x8a71420e0b797035 + .quad 0x3b3c833687abe743 + .quad 0xf5396425b23545a4 + .quad 0x15a7a27e98fbb296 + .quad 0xab6c52bc636fdd86 + .quad 0x79d995a8419334ee + .quad 0xcd8a8ea61195dd75 + .quad 0xa504d8a81dd9a82f + .quad 0x540dca81a35879b6 + .quad 0x60dd16a379c86a8a + + // 2^52 * 3 * B + + .quad 0x35a2c8487381e559 + .quad 0x596ffea6d78082cb + .quad 0xcb9771ebdba7b653 + .quad 0x5a08b5019b4da685 + .quad 0x3501d6f8153e47b8 + .quad 0xb7a9675414a2f60c + .quad 0x112ee8b6455d9523 + .quad 0x4e62a3c18112ea8a + .quad 0xc8d4ac04516ab786 + .quad 0x595af3215295b23d + .quad 0xd6edd234db0230c1 + .quad 0x0929efe8825b41cc + + // 2^52 * 4 * B + + .quad 0x5f0601d1cbd0f2d3 + .quad 0x736e412f6132bb7f + .quad 0x83604432238dde87 + .quad 0x1e3a5272f5c0753c + .quad 0x8b3172b7ad56651d + .quad 0x01581b7a3fabd717 + .quad 0x2dc94df6424df6e4 + .quad 0x30376e5d2c29284f + .quad 0xd2918da78159a59c + .quad 0x6bdc1cd93f0713f3 + .quad 0x565f7a934acd6590 + .quad 0x53daacec4cb4c128 + + // 2^52 * 5 * B + + .quad 0x4ca73bd79cc8a7d6 + .quad 0x4d4a738f47e9a9b2 + .quad 0xf4cbf12942f5fe00 + .quad 0x01a13ff9bdbf0752 + .quad 0x99852bc3852cfdb0 + .quad 0x2cc12e9559d6ed0b + .quad 0x70f9e2bf9b5ac27b + .quad 0x4f3b8c117959ae99 + .quad 0x55b6c9c82ff26412 + .quad 0x1ac4a8c91fb667a8 + .quad 0xd527bfcfeb778bf2 + .quad 0x303337da7012a3be + + // 2^52 * 6 * B + + .quad 0x955422228c1c9d7c + .quad 0x01fac1371a9b340f + .quad 0x7e8d9177925b48d7 + .quad 0x53f8ad5661b3e31b + .quad 0x976d3ccbfad2fdd1 + .quad 0xcb88839737a640a8 + .quad 0x2ff00c1d6734cb25 + .quad 0x269ff4dc789c2d2b + .quad 0x0c003fbdc08d678d + .quad 0x4d982fa37ead2b17 + .quad 0xc07e6bcdb2e582f1 + .quad 0x296c7291df412a44 + + // 2^52 * 7 * B + + .quad 0x7903de2b33daf397 + .quad 0xd0ff0619c9a624b3 + .quad 0x8a1d252b555b3e18 + .quad 0x2b6d581c52e0b7c0 + .quad 0xdfb23205dab8b59e + .quad 0x465aeaa0c8092250 + .quad 0xd133c1189a725d18 + .quad 0x2327370261f117d1 + .quad 0x3d0543d3623e7986 + .quad 0x679414c2c278a354 + .quad 0xae43f0cc726196f6 + .quad 0x7836c41f8245eaba + + // 2^52 * 8 * B + + .quad 0xe7a254db49e95a81 + .quad 0x5192d5d008b0ad73 + .quad 0x4d20e5b1d00afc07 + .quad 0x5d55f8012cf25f38 + .quad 0xca651e848011937c + .quad 0xc6b0c46e6ef41a28 + .quad 0xb7021ba75f3f8d52 + .quad 0x119dff99ead7b9fd + .quad 0x43eadfcbf4b31d4d + .quad 0xc6503f7411148892 + .quad 0xfeee68c5060d3b17 + .quad 0x329293b3dd4a0ac8 + + // 2^56 * 1 * B + + .quad 0x4e59214fe194961a + .quad 0x49be7dc70d71cd4f + .quad 0x9300cfd23b50f22d + .quad 0x4789d446fc917232 + .quad 0x2879852d5d7cb208 + .quad 0xb8dedd70687df2e7 + .quad 0xdc0bffab21687891 + .quad 0x2b44c043677daa35 + .quad 0x1a1c87ab074eb78e + .quad 0xfac6d18e99daf467 + .quad 0x3eacbbcd484f9067 + .quad 0x60c52eef2bb9a4e4 + + // 2^56 * 2 * B + + .quad 0x0b5d89bc3bfd8bf1 + .quad 0xb06b9237c9f3551a + .quad 0x0e4c16b0d53028f5 + .quad 0x10bc9c312ccfcaab + .quad 0x702bc5c27cae6d11 + .quad 0x44c7699b54a48cab + .quad 0xefbc4056ba492eb2 + .quad 0x70d77248d9b6676d + .quad 0xaa8ae84b3ec2a05b + .quad 0x98699ef4ed1781e0 + .quad 0x794513e4708e85d1 + .quad 0x63755bd3a976f413 + + // 2^56 * 3 * B + + .quad 0xb55fa03e2ad10853 + .quad 0x356f75909ee63569 + .quad 0x9ff9f1fdbe69b890 + .quad 0x0d8cc1c48bc16f84 + .quad 0x3dc7101897f1acb7 + .quad 0x5dda7d5ec165bbd8 + .quad 0x508e5b9c0fa1020f + .quad 0x2763751737c52a56 + .quad 0x029402d36eb419a9 + .quad 0xf0b44e7e77b460a5 + .quad 0xcfa86230d43c4956 + .quad 0x70c2dd8a7ad166e7 + + // 2^56 * 4 * B + + .quad 0x656194509f6fec0e + .quad 0xee2e7ea946c6518d + .quad 0x9733c1f367e09b5c + .quad 0x2e0fac6363948495 + .quad 0x91d4967db8ed7e13 + .quad 0x74252f0ad776817a + .quad 0xe40982e00d852564 + .quad 0x32b8613816a53ce5 + .quad 0x79e7f7bee448cd64 + .quad 0x6ac83a67087886d0 + .quad 0xf89fd4d9a0e4db2e + .quad 0x4179215c735a4f41 + + // 2^56 * 5 * B + + .quad 0x8c7094e7d7dced2a + .quad 0x97fb8ac347d39c70 + .quad 0xe13be033a906d902 + .quad 0x700344a30cd99d76 + .quad 0xe4ae33b9286bcd34 + .quad 0xb7ef7eb6559dd6dc + .quad 0x278b141fb3d38e1f + .quad 0x31fa85662241c286 + .quad 0xaf826c422e3622f4 + .quad 0xc12029879833502d + .quad 0x9bc1b7e12b389123 + .quad 0x24bb2312a9952489 + + // 2^56 * 6 * B + + .quad 0xb1a8ed1732de67c3 + .quad 0x3cb49418461b4948 + .quad 0x8ebd434376cfbcd2 + .quad 0x0fee3e871e188008 + .quad 0x41f80c2af5f85c6b + .quad 0x687284c304fa6794 + .quad 0x8945df99a3ba1bad + .quad 0x0d1d2af9ffeb5d16 + .quad 0xa9da8aa132621edf + .quad 0x30b822a159226579 + .quad 0x4004197ba79ac193 + .quad 0x16acd79718531d76 + + // 2^56 * 7 * B + + .quad 0x72df72af2d9b1d3d + .quad 0x63462a36a432245a + .quad 0x3ecea07916b39637 + .quad 0x123e0ef6b9302309 + .quad 0xc959c6c57887b6ad + .quad 0x94e19ead5f90feba + .quad 0x16e24e62a342f504 + .quad 0x164ed34b18161700 + .quad 0x487ed94c192fe69a + .quad 0x61ae2cea3a911513 + .quad 0x877bf6d3b9a4de27 + .quad 0x78da0fc61073f3eb + + // 2^56 * 8 * B + + .quad 0x5bf15d28e52bc66a + .quad 0x2c47e31870f01a8e + .quad 0x2419afbc06c28bdd + .quad 0x2d25deeb256b173a + .quad 0xa29f80f1680c3a94 + .quad 0x71f77e151ae9e7e6 + .quad 0x1100f15848017973 + .quad 0x054aa4b316b38ddd + .quad 0xdfc8468d19267cb8 + .quad 0x0b28789c66e54daf + .quad 0x2aeb1d2a666eec17 + .quad 0x134610a6ab7da760 + + // 2^60 * 1 * B + + .quad 0xcaf55ec27c59b23f + .quad 0x99aeed3e154d04f2 + .quad 0x68441d72e14141f4 + .quad 0x140345133932a0a2 + .quad 0xd91430e0dc028c3c + .quad 0x0eb955a85217c771 + .quad 0x4b09e1ed2c99a1fa + .quad 0x42881af2bd6a743c + .quad 0x7bfec69aab5cad3d + .quad 0xc23e8cd34cb2cfad + .quad 0x685dd14bfb37d6a2 + .quad 0x0ad6d64415677a18 + + // 2^60 * 2 * B + + .quad 0x781a439e417becb5 + .quad 0x4ac5938cd10e0266 + .quad 0x5da385110692ac24 + .quad 0x11b065a2ade31233 + .quad 0x7914892847927e9f + .quad 0x33dad6ef370aa877 + .quad 0x1f8f24fa11122703 + .quad 0x5265ac2f2adf9592 + .quad 0x405fdd309afcb346 + .quad 0xd9723d4428e63f54 + .quad 0x94c01df05f65aaae + .quad 0x43e4dc3ae14c0809 + + // 2^60 * 3 * B + + .quad 0xbc12c7f1a938a517 + .quad 0x473028ab3180b2e1 + .quad 0x3f78571efbcd254a + .quad 0x74e534426ff6f90f + .quad 0xea6f7ac3adc2c6a3 + .quad 0xd0e928f6e9717c94 + .quad 0xe2d379ead645eaf5 + .quad 0x46dd8785c51ffbbe + .quad 0x709801be375c8898 + .quad 0x4b06dab5e3fd8348 + .quad 0x75880ced27230714 + .quad 0x2b09468fdd2f4c42 + + // 2^60 * 4 * B + + .quad 0x97c749eeb701cb96 + .quad 0x83f438d4b6a369c3 + .quad 0x62962b8b9a402cd9 + .quad 0x6976c7509888df7b + .quad 0x5b97946582ffa02a + .quad 0xda096a51fea8f549 + .quad 0xa06351375f77af9b + .quad 0x1bcfde61201d1e76 + .quad 0x4a4a5490246a59a2 + .quad 0xd63ebddee87fdd90 + .quad 0xd9437c670d2371fa + .quad 0x69e87308d30f8ed6 + + // 2^60 * 5 * B + + .quad 0x435a8bb15656beb0 + .quad 0xf8fac9ba4f4d5bca + .quad 0xb9b278c41548c075 + .quad 0x3eb0ef76e892b622 + .quad 0x0f80bf028bc80303 + .quad 0x6aae16b37a18cefb + .quad 0xdd47ea47d72cd6a3 + .quad 0x61943588f4ed39aa + .quad 0xd26e5c3e91039f85 + .quad 0xc0e9e77df6f33aa9 + .quad 0xe8968c5570066a93 + .quad 0x3c34d1881faaaddd + + // 2^60 * 6 * B + + .quad 0x3f9d2b5ea09f9ec0 + .quad 0x1dab3b6fb623a890 + .quad 0xa09ba3ea72d926c4 + .quad 0x374193513fd8b36d + .quad 0xbd5b0b8f2fffe0d9 + .quad 0x6aa254103ed24fb9 + .quad 0x2ac7d7bcb26821c4 + .quad 0x605b394b60dca36a + .quad 0xb4e856e45a9d1ed2 + .quad 0xefe848766c97a9a2 + .quad 0xb104cf641e5eee7d + .quad 0x2f50b81c88a71c8f + + // 2^60 * 7 * B + + .quad 0x31723c61fc6811bb + .quad 0x9cb450486211800f + .quad 0x768933d347995753 + .quad 0x3491a53502752fcd + .quad 0x2b552ca0a7da522a + .quad 0x3230b336449b0250 + .quad 0xf2c4c5bca4b99fb9 + .quad 0x7b2c674958074a22 + .quad 0xd55165883ed28cdf + .quad 0x12d84fd2d362de39 + .quad 0x0a874ad3e3378e4f + .quad 0x000d2b1f7c763e74 + + // 2^60 * 8 * B + + .quad 0x3d420811d06d4a67 + .quad 0xbefc048590e0ffe3 + .quad 0xf870c6b7bd487bde + .quad 0x6e2a7316319afa28 + .quad 0x9624778c3e94a8ab + .quad 0x0ad6f3cee9a78bec + .quad 0x948ac7810d743c4f + .quad 0x76627935aaecfccc + .quad 0x56a8ac24d6d59a9f + .quad 0xc8db753e3096f006 + .quad 0x477f41e68f4c5299 + .quad 0x588d851cf6c86114 + + // 2^64 * 1 * B + + .quad 0x51138ec78df6b0fe + .quad 0x5397da89e575f51b + .quad 0x09207a1d717af1b9 + .quad 0x2102fdba2b20d650 + .quad 0xcd2a65e777d1f515 + .quad 0x548991878faa60f1 + .quad 0xb1b73bbcdabc06e5 + .quad 0x654878cba97cc9fb + .quad 0x969ee405055ce6a1 + .quad 0x36bca7681251ad29 + .quad 0x3a1af517aa7da415 + .quad 0x0ad725db29ecb2ba + + // 2^64 * 2 * B + + .quad 0xdc4267b1834e2457 + .quad 0xb67544b570ce1bc5 + .quad 0x1af07a0bf7d15ed7 + .quad 0x4aefcffb71a03650 + .quad 0xfec7bc0c9b056f85 + .quad 0x537d5268e7f5ffd7 + .quad 0x77afc6624312aefa + .quad 0x4f675f5302399fd9 + .quad 0xc32d36360415171e + .quad 0xcd2bef118998483b + .quad 0x870a6eadd0945110 + .quad 0x0bccbb72a2a86561 + + // 2^64 * 3 * B + + .quad 0x185e962feab1a9c8 + .quad 0x86e7e63565147dcd + .quad 0xb092e031bb5b6df2 + .quad 0x4024f0ab59d6b73e + .quad 0x186d5e4c50fe1296 + .quad 0xe0397b82fee89f7e + .quad 0x3bc7f6c5507031b0 + .quad 0x6678fd69108f37c2 + .quad 0x1586fa31636863c2 + .quad 0x07f68c48572d33f2 + .quad 0x4f73cc9f789eaefc + .quad 0x2d42e2108ead4701 + + // 2^64 * 4 * B + + .quad 0x97f5131594dfd29b + .quad 0x6155985d313f4c6a + .quad 0xeba13f0708455010 + .quad 0x676b2608b8d2d322 + .quad 0x21717b0d0f537593 + .quad 0x914e690b131e064c + .quad 0x1bb687ae752ae09f + .quad 0x420bf3a79b423c6e + .quad 0x8138ba651c5b2b47 + .quad 0x8671b6ec311b1b80 + .quad 0x7bff0cb1bc3135b0 + .quad 0x745d2ffa9c0cf1e0 + + // 2^64 * 5 * B + + .quad 0xbf525a1e2bc9c8bd + .quad 0xea5b260826479d81 + .quad 0xd511c70edf0155db + .quad 0x1ae23ceb960cf5d0 + .quad 0x6036df5721d34e6a + .quad 0xb1db8827997bb3d0 + .quad 0xd3c209c3c8756afa + .quad 0x06e15be54c1dc839 + .quad 0x5b725d871932994a + .quad 0x32351cb5ceb1dab0 + .quad 0x7dc41549dab7ca05 + .quad 0x58ded861278ec1f7 + + // 2^64 * 6 * B + + .quad 0xd8173793f266c55c + .quad 0xc8c976c5cc454e49 + .quad 0x5ce382f8bc26c3a8 + .quad 0x2ff39de85485f6f9 + .quad 0x2dfb5ba8b6c2c9a8 + .quad 0x48eeef8ef52c598c + .quad 0x33809107f12d1573 + .quad 0x08ba696b531d5bd8 + .quad 0x77ed3eeec3efc57a + .quad 0x04e05517d4ff4811 + .quad 0xea3d7a3ff1a671cb + .quad 0x120633b4947cfe54 + + // 2^64 * 7 * B + + .quad 0x0b94987891610042 + .quad 0x4ee7b13cecebfae8 + .quad 0x70be739594f0a4c0 + .quad 0x35d30a99b4d59185 + .quad 0x82bd31474912100a + .quad 0xde237b6d7e6fbe06 + .quad 0xe11e761911ea79c6 + .quad 0x07433be3cb393bde + .quad 0xff7944c05ce997f4 + .quad 0x575d3de4b05c51a3 + .quad 0x583381fd5a76847c + .quad 0x2d873ede7af6da9f + + // 2^64 * 8 * B + + .quad 0x157a316443373409 + .quad 0xfab8b7eef4aa81d9 + .quad 0xb093fee6f5a64806 + .quad 0x2e773654707fa7b6 + .quad 0xaa6202e14e5df981 + .quad 0xa20d59175015e1f5 + .quad 0x18a275d3bae21d6c + .quad 0x0543618a01600253 + .quad 0x0deabdf4974c23c1 + .quad 0xaa6f0a259dce4693 + .quad 0x04202cb8a29aba2c + .quad 0x4b1443362d07960d + + // 2^68 * 1 * B + + .quad 0x47b837f753242cec + .quad 0x256dc48cc04212f2 + .quad 0xe222fbfbe1d928c5 + .quad 0x48ea295bad8a2c07 + .quad 0x299b1c3f57c5715e + .quad 0x96cb929e6b686d90 + .quad 0x3004806447235ab3 + .quad 0x2c435c24a44d9fe1 + .quad 0x0607c97c80f8833f + .quad 0x0e851578ca25ec5b + .quad 0x54f7450b161ebb6f + .quad 0x7bcb4792a0def80e + + // 2^68 * 2 * B + + .quad 0x8487e3d02bc73659 + .quad 0x4baf8445059979df + .quad 0xd17c975adcad6fbf + .quad 0x57369f0bdefc96b6 + .quad 0x1cecd0a0045224c2 + .quad 0x757f1b1b69e53952 + .quad 0x775b7a925289f681 + .quad 0x1b6cc62016736148 + .quad 0xf1a9990175638698 + .quad 0x353dd1beeeaa60d3 + .quad 0x849471334c9ba488 + .quad 0x63fa6e6843ade311 + + // 2^68 * 3 * B + + .quad 0xd15c20536597c168 + .quad 0x9f73740098d28789 + .quad 0x18aee7f13257ba1f + .quad 0x3418bfda07346f14 + .quad 0x2195becdd24b5eb7 + .quad 0x5e41f18cc0cd44f9 + .quad 0xdf28074441ca9ede + .quad 0x07073b98f35b7d67 + .quad 0xd03c676c4ce530d4 + .quad 0x0b64c0473b5df9f4 + .quad 0x065cef8b19b3a31e + .quad 0x3084d661533102c9 + + // 2^68 * 4 * B + + .quad 0xe1f6b79ebf8469ad + .quad 0x15801004e2663135 + .quad 0x9a498330af74181b + .quad 0x3ba2504f049b673c + .quad 0x9a6ce876760321fd + .quad 0x7fe2b5109eb63ad8 + .quad 0x00e7d4ae8ac80592 + .quad 0x73d86b7abb6f723a + .quad 0x0b52b5606dba5ab6 + .quad 0xa9134f0fbbb1edab + .quad 0x30a9520d9b04a635 + .quad 0x6813b8f37973e5db + + // 2^68 * 5 * B + + .quad 0x9854b054334127c1 + .quad 0x105d047882fbff25 + .quad 0xdb49f7f944186f4f + .quad 0x1768e838bed0b900 + .quad 0xf194ca56f3157e29 + .quad 0x136d35705ef528a5 + .quad 0xdd4cef778b0599bc + .quad 0x7d5472af24f833ed + .quad 0xd0ef874daf33da47 + .quad 0x00d3be5db6e339f9 + .quad 0x3f2a8a2f9c9ceece + .quad 0x5d1aeb792352435a + + // 2^68 * 6 * B + + .quad 0xf59e6bb319cd63ca + .quad 0x670c159221d06839 + .quad 0xb06d565b2150cab6 + .quad 0x20fb199d104f12a3 + .quad 0x12c7bfaeb61ba775 + .quad 0xb84e621fe263bffd + .quad 0x0b47a5c35c840dcf + .quad 0x7e83be0bccaf8634 + .quad 0x61943dee6d99c120 + .quad 0x86101f2e460b9fe0 + .quad 0x6bb2f1518ee8598d + .quad 0x76b76289fcc475cc + + // 2^68 * 7 * B + + .quad 0x791b4cc1756286fa + .quad 0xdbced317d74a157c + .quad 0x7e732421ea72bde6 + .quad 0x01fe18491131c8e9 + .quad 0x4245f1a1522ec0b3 + .quad 0x558785b22a75656d + .quad 0x1d485a2548a1b3c0 + .quad 0x60959eccd58fe09f + .quad 0x3ebfeb7ba8ed7a09 + .quad 0x49fdc2bbe502789c + .quad 0x44ebce5d3c119428 + .quad 0x35e1eb55be947f4a + + // 2^68 * 8 * B + + .quad 0xdbdae701c5738dd3 + .quad 0xf9c6f635b26f1bee + .quad 0x61e96a8042f15ef4 + .quad 0x3aa1d11faf60a4d8 + .quad 0x14fd6dfa726ccc74 + .quad 0x3b084cfe2f53b965 + .quad 0xf33ae4f552a2c8b4 + .quad 0x59aab07a0d40166a + .quad 0x77bcec4c925eac25 + .quad 0x1848718460137738 + .quad 0x5b374337fea9f451 + .quad 0x1865e78ec8e6aa46 + + // 2^72 * 1 * B + + .quad 0xccc4b7c7b66e1f7a + .quad 0x44157e25f50c2f7e + .quad 0x3ef06dfc713eaf1c + .quad 0x582f446752da63f7 + .quad 0x967c54e91c529ccb + .quad 0x30f6269264c635fb + .quad 0x2747aff478121965 + .quad 0x17038418eaf66f5c + .quad 0xc6317bd320324ce4 + .quad 0xa81042e8a4488bc4 + .quad 0xb21ef18b4e5a1364 + .quad 0x0c2a1c4bcda28dc9 + + // 2^72 * 2 * B + + .quad 0xd24dc7d06f1f0447 + .quad 0xb2269e3edb87c059 + .quad 0xd15b0272fbb2d28f + .quad 0x7c558bd1c6f64877 + .quad 0xedc4814869bd6945 + .quad 0x0d6d907dbe1c8d22 + .quad 0xc63bd212d55cc5ab + .quad 0x5a6a9b30a314dc83 + .quad 0xd0ec1524d396463d + .quad 0x12bb628ac35a24f0 + .quad 0xa50c3a791cbc5fa4 + .quad 0x0404a5ca0afbafc3 + + // 2^72 * 3 * B + + .quad 0x8c1f40070aa743d6 + .quad 0xccbad0cb5b265ee8 + .quad 0x574b046b668fd2de + .quad 0x46395bfdcadd9633 + .quad 0x62bc9e1b2a416fd1 + .quad 0xb5c6f728e350598b + .quad 0x04343fd83d5d6967 + .quad 0x39527516e7f8ee98 + .quad 0x117fdb2d1a5d9a9c + .quad 0x9c7745bcd1005c2a + .quad 0xefd4bef154d56fea + .quad 0x76579a29e822d016 + + // 2^72 * 4 * B + + .quad 0x45b68e7e49c02a17 + .quad 0x23cd51a2bca9a37f + .quad 0x3ed65f11ec224c1b + .quad 0x43a384dc9e05bdb1 + .quad 0x333cb51352b434f2 + .quad 0xd832284993de80e1 + .quad 0xb5512887750d35ce + .quad 0x02c514bb2a2777c1 + .quad 0x684bd5da8bf1b645 + .quad 0xfb8bd37ef6b54b53 + .quad 0x313916d7a9b0d253 + .quad 0x1160920961548059 + + // 2^72 * 5 * B + + .quad 0xb44d166929dacfaa + .quad 0xda529f4c8413598f + .quad 0xe9ef63ca453d5559 + .quad 0x351e125bc5698e0b + .quad 0x7a385616369b4dcd + .quad 0x75c02ca7655c3563 + .quad 0x7dc21bf9d4f18021 + .quad 0x2f637d7491e6e042 + .quad 0xd4b49b461af67bbe + .quad 0xd603037ac8ab8961 + .quad 0x71dee19ff9a699fb + .quad 0x7f182d06e7ce2a9a + + // 2^72 * 6 * B + + .quad 0x7a7c8e64ab0168ec + .quad 0xcb5a4a5515edc543 + .quad 0x095519d347cd0eda + .quad 0x67d4ac8c343e93b0 + .quad 0x09454b728e217522 + .quad 0xaa58e8f4d484b8d8 + .quad 0xd358254d7f46903c + .quad 0x44acc043241c5217 + .quad 0x1c7d6bbb4f7a5777 + .quad 0x8b35fed4918313e1 + .quad 0x4adca1c6c96b4684 + .quad 0x556d1c8312ad71bd + + // 2^72 * 7 * B + + .quad 0x17ef40e30c8d3982 + .quad 0x31f7073e15a3fa34 + .quad 0x4f21f3cb0773646e + .quad 0x746c6c6d1d824eff + .quad 0x81f06756b11be821 + .quad 0x0faff82310a3f3dd + .quad 0xf8b2d0556a99465d + .quad 0x097abe38cc8c7f05 + .quad 0x0c49c9877ea52da4 + .quad 0x4c4369559bdc1d43 + .quad 0x022c3809f7ccebd2 + .quad 0x577e14a34bee84bd + + // 2^72 * 8 * B + + .quad 0xf0e268ac61a73b0a + .quad 0xf2fafa103791a5f5 + .quad 0xc1e13e826b6d00e9 + .quad 0x60fa7ee96fd78f42 + .quad 0x94fecebebd4dd72b + .quad 0xf46a4fda060f2211 + .quad 0x124a5977c0c8d1ff + .quad 0x705304b8fb009295 + .quad 0xb63d1d354d296ec6 + .quad 0xf3c3053e5fad31d8 + .quad 0x670b958cb4bd42ec + .quad 0x21398e0ca16353fd + + // 2^76 * 1 * B + + .quad 0x216ab2ca8da7d2ef + .quad 0x366ad9dd99f42827 + .quad 0xae64b9004fdd3c75 + .quad 0x403a395b53909e62 + .quad 0x86c5fc16861b7e9a + .quad 0xf6a330476a27c451 + .quad 0x01667267a1e93597 + .quad 0x05ffb9cd6082dfeb + .quad 0xa617fa9ff53f6139 + .quad 0x60f2b5e513e66cb6 + .quad 0xd7a8beefb3448aa4 + .quad 0x7a2932856f5ea192 + + // 2^76 * 2 * B + + .quad 0x0b39d761b02de888 + .quad 0x5f550e7ed2414e1f + .quad 0xa6bfa45822e1a940 + .quad 0x050a2f7dfd447b99 + .quad 0xb89c444879639302 + .quad 0x4ae4f19350c67f2c + .quad 0xf0b35da8c81af9c6 + .quad 0x39d0003546871017 + .quad 0x437c3b33a650db77 + .quad 0x6bafe81dbac52bb2 + .quad 0xfe99402d2db7d318 + .quad 0x2b5b7eec372ba6ce + + // 2^76 * 3 * B + + .quad 0xb3bc4bbd83f50eef + .quad 0x508f0c998c927866 + .quad 0x43e76587c8b7e66e + .quad 0x0f7655a3a47f98d9 + .quad 0xa694404d613ac8f4 + .quad 0x500c3c2bfa97e72c + .quad 0x874104d21fcec210 + .quad 0x1b205fb38604a8ee + .quad 0x55ecad37d24b133c + .quad 0x441e147d6038c90b + .quad 0x656683a1d62c6fee + .quad 0x0157d5dc87e0ecae + + // 2^76 * 4 * B + + .quad 0xf2a7af510354c13d + .quad 0xd7a0b145aa372b60 + .quad 0x2869b96a05a3d470 + .quad 0x6528e42d82460173 + .quad 0x95265514d71eb524 + .quad 0xe603d8815df14593 + .quad 0x147cdf410d4de6b7 + .quad 0x5293b1730437c850 + .quad 0x23d0e0814bccf226 + .quad 0x92c745cd8196fb93 + .quad 0x8b61796c59541e5b + .quad 0x40a44df0c021f978 + + // 2^76 * 5 * B + + .quad 0xdaa869894f20ea6a + .quad 0xea14a3d14c620618 + .quad 0x6001fccb090bf8be + .quad 0x35f4e822947e9cf0 + .quad 0x86c96e514bc5d095 + .quad 0xf20d4098fca6804a + .quad 0x27363d89c826ea5d + .quad 0x39ca36565719cacf + .quad 0x97506f2f6f87b75c + .quad 0xc624aea0034ae070 + .quad 0x1ec856e3aad34dd6 + .quad 0x055b0be0e440e58f + + // 2^76 * 6 * B + + .quad 0x6469a17d89735d12 + .quad 0xdb6f27d5e662b9f1 + .quad 0x9fcba3286a395681 + .quad 0x363b8004d269af25 + .quad 0x4d12a04b6ea33da2 + .quad 0x57cf4c15e36126dd + .quad 0x90ec9675ee44d967 + .quad 0x64ca348d2a985aac + .quad 0x99588e19e4c4912d + .quad 0xefcc3b4e1ca5ce6b + .quad 0x4522ea60fa5b98d5 + .quad 0x7064bbab1de4a819 + + // 2^76 * 7 * B + + .quad 0xb919e1515a770641 + .quad 0xa9a2e2c74e7f8039 + .quad 0x7527250b3df23109 + .quad 0x756a7330ac27b78b + .quad 0xa290c06142542129 + .quad 0xf2e2c2aebe8d5b90 + .quad 0xcf2458db76abfe1b + .quad 0x02157ade83d626bf + .quad 0x3e46972a1b9a038b + .quad 0x2e4ee66a7ee03fb4 + .quad 0x81a248776edbb4ca + .quad 0x1a944ee88ecd0563 + + // 2^76 * 8 * B + + .quad 0xd5a91d1151039372 + .quad 0x2ed377b799ca26de + .quad 0xa17202acfd366b6b + .quad 0x0730291bd6901995 + .quad 0xbb40a859182362d6 + .quad 0xb99f55778a4d1abb + .quad 0x8d18b427758559f6 + .quad 0x26c20fe74d26235a + .quad 0x648d1d9fe9cc22f5 + .quad 0x66bc561928dd577c + .quad 0x47d3ed21652439d1 + .quad 0x49d271acedaf8b49 + + // 2^80 * 1 * B + + .quad 0x89f5058a382b33f3 + .quad 0x5ae2ba0bad48c0b4 + .quad 0x8f93b503a53db36e + .quad 0x5aa3ed9d95a232e6 + .quad 0x2798aaf9b4b75601 + .quad 0x5eac72135c8dad72 + .quad 0xd2ceaa6161b7a023 + .quad 0x1bbfb284e98f7d4e + .quad 0x656777e9c7d96561 + .quad 0xcb2b125472c78036 + .quad 0x65053299d9506eee + .quad 0x4a07e14e5e8957cc + + // 2^80 * 2 * B + + .quad 0x4ee412cb980df999 + .quad 0xa315d76f3c6ec771 + .quad 0xbba5edde925c77fd + .quad 0x3f0bac391d313402 + .quad 0x240b58cdc477a49b + .quad 0xfd38dade6447f017 + .quad 0x19928d32a7c86aad + .quad 0x50af7aed84afa081 + .quad 0x6e4fde0115f65be5 + .quad 0x29982621216109b2 + .quad 0x780205810badd6d9 + .quad 0x1921a316baebd006 + + // 2^80 * 3 * B + + .quad 0x89422f7edfb870fc + .quad 0x2c296beb4f76b3bd + .quad 0x0738f1d436c24df7 + .quad 0x6458df41e273aeb0 + .quad 0xd75aad9ad9f3c18b + .quad 0x566a0eef60b1c19c + .quad 0x3e9a0bac255c0ed9 + .quad 0x7b049deca062c7f5 + .quad 0xdccbe37a35444483 + .quad 0x758879330fedbe93 + .quad 0x786004c312c5dd87 + .quad 0x6093dccbc2950e64 + + // 2^80 * 4 * B + + .quad 0x1ff39a8585e0706d + .quad 0x36d0a5d8b3e73933 + .quad 0x43b9f2e1718f453b + .quad 0x57d1ea084827a97c + .quad 0x6bdeeebe6084034b + .quad 0x3199c2b6780fb854 + .quad 0x973376abb62d0695 + .quad 0x6e3180c98b647d90 + .quad 0xee7ab6e7a128b071 + .quad 0xa4c1596d93a88baa + .quad 0xf7b4de82b2216130 + .quad 0x363e999ddd97bd18 + + // 2^80 * 5 * B + + .quad 0x96a843c135ee1fc4 + .quad 0x976eb35508e4c8cf + .quad 0xb42f6801b58cd330 + .quad 0x48ee9b78693a052b + .quad 0x2f1848dce24baec6 + .quad 0x769b7255babcaf60 + .quad 0x90cb3c6e3cefe931 + .quad 0x231f979bc6f9b355 + .quad 0x5c31de4bcc2af3c6 + .quad 0xb04bb030fe208d1f + .quad 0xb78d7009c14fb466 + .quad 0x079bfa9b08792413 + + // 2^80 * 6 * B + + .quad 0xe3903a51da300df4 + .quad 0x843964233da95ab0 + .quad 0xed3cf12d0b356480 + .quad 0x038c77f684817194 + .quad 0xf3c9ed80a2d54245 + .quad 0x0aa08b7877f63952 + .quad 0xd76dac63d1085475 + .quad 0x1ef4fb159470636b + .quad 0x854e5ee65b167bec + .quad 0x59590a4296d0cdc2 + .quad 0x72b2df3498102199 + .quad 0x575ee92a4a0bff56 + + // 2^80 * 7 * B + + .quad 0xd4c080908a182fcf + .quad 0x30e170c299489dbd + .quad 0x05babd5752f733de + .quad 0x43d4e7112cd3fd00 + .quad 0x5d46bc450aa4d801 + .quad 0xc3af1227a533b9d8 + .quad 0x389e3b262b8906c2 + .quad 0x200a1e7e382f581b + .quad 0x518db967eaf93ac5 + .quad 0x71bc989b056652c0 + .quad 0xfe2b85d9567197f5 + .quad 0x050eca52651e4e38 + + // 2^80 * 8 * B + + .quad 0xc3431ade453f0c9c + .quad 0xe9f5045eff703b9b + .quad 0xfcd97ac9ed847b3d + .quad 0x4b0ee6c21c58f4c6 + .quad 0x97ac397660e668ea + .quad 0x9b19bbfe153ab497 + .quad 0x4cb179b534eca79f + .quad 0x6151c09fa131ae57 + .quad 0x3af55c0dfdf05d96 + .quad 0xdd262ee02ab4ee7a + .quad 0x11b2bb8712171709 + .quad 0x1fef24fa800f030b + + // 2^84 * 1 * B + + .quad 0xb496123a6b6c6609 + .quad 0xa750fe8580ab5938 + .quad 0xf471bf39b7c27a5f + .quad 0x507903ce77ac193c + .quad 0xff91a66a90166220 + .quad 0xf22552ae5bf1e009 + .quad 0x7dff85d87f90df7c + .quad 0x4f620ffe0c736fb9 + .quad 0x62f90d65dfde3e34 + .quad 0xcf28c592b9fa5fad + .quad 0x99c86ef9c6164510 + .quad 0x25d448044a256c84 + + // 2^84 * 2 * B + + .quad 0xbd68230ec7e9b16f + .quad 0x0eb1b9c1c1c5795d + .quad 0x7943c8c495b6b1ff + .quad 0x2f9faf620bbacf5e + .quad 0x2c7c4415c9022b55 + .quad 0x56a0d241812eb1fe + .quad 0xf02ea1c9d7b65e0d + .quad 0x4180512fd5323b26 + .quad 0xa4ff3e698a48a5db + .quad 0xba6a3806bd95403b + .quad 0x9f7ce1af47d5b65d + .quad 0x15e087e55939d2fb + + // 2^84 * 3 * B + + .quad 0x12207543745c1496 + .quad 0xdaff3cfdda38610c + .quad 0xe4e797272c71c34f + .quad 0x39c07b1934bdede9 + .quad 0x8894186efb963f38 + .quad 0x48a00e80dc639bd5 + .quad 0xa4e8092be96c1c99 + .quad 0x5a097d54ca573661 + .quad 0x2d45892b17c9e755 + .quad 0xd033fd7289308df8 + .quad 0x6c2fe9d9525b8bd9 + .quad 0x2edbecf1c11cc079 + + // 2^84 * 4 * B + + .quad 0x1616a4e3c715a0d2 + .quad 0x53623cb0f8341d4d + .quad 0x96ef5329c7e899cb + .quad 0x3d4e8dbba668baa6 + .quad 0xee0f0fddd087a25f + .quad 0x9c7531555c3e34ee + .quad 0x660c572e8fab3ab5 + .quad 0x0854fc44544cd3b2 + .quad 0x61eba0c555edad19 + .quad 0x24b533fef0a83de6 + .quad 0x3b77042883baa5f8 + .quad 0x678f82b898a47e8d + + // 2^84 * 5 * B + + .quad 0xb1491d0bd6900c54 + .quad 0x3539722c9d132636 + .quad 0x4db928920b362bc9 + .quad 0x4d7cd1fea68b69df + .quad 0x1e09d94057775696 + .quad 0xeed1265c3cd951db + .quad 0xfa9dac2b20bce16f + .quad 0x0f7f76e0e8d089f4 + .quad 0x36d9ebc5d485b00c + .quad 0xa2596492e4adb365 + .quad 0xc1659480c2119ccd + .quad 0x45306349186e0d5f + + // 2^84 * 6 * B + + .quad 0x94ddd0c1a6cdff1d + .quad 0x55f6f115e84213ae + .quad 0x6c935f85992fcf6a + .quad 0x067ee0f54a37f16f + .quad 0x96a414ec2b072491 + .quad 0x1bb2218127a7b65b + .quad 0x6d2849596e8a4af0 + .quad 0x65f3b08ccd27765f + .quad 0xecb29fff199801f7 + .quad 0x9d361d1fa2a0f72f + .quad 0x25f11d2375fd2f49 + .quad 0x124cefe80fe10fe2 + + // 2^84 * 7 * B + + .quad 0x4c126cf9d18df255 + .quad 0xc1d471e9147a63b6 + .quad 0x2c6d3c73f3c93b5f + .quad 0x6be3a6a2e3ff86a2 + .quad 0x1518e85b31b16489 + .quad 0x8faadcb7db710bfb + .quad 0x39b0bdf4a14ae239 + .quad 0x05f4cbea503d20c1 + .quad 0xce040e9ec04145bc + .quad 0xc71ff4e208f6834c + .quad 0xbd546e8dab8847a3 + .quad 0x64666aa0a4d2aba5 + + // 2^84 * 8 * B + + .quad 0x6841435a7c06d912 + .quad 0xca123c21bb3f830b + .quad 0xd4b37b27b1cbe278 + .quad 0x1d753b84c76f5046 + .quad 0xb0c53bf73337e94c + .quad 0x7cb5697e11e14f15 + .quad 0x4b84abac1930c750 + .quad 0x28dd4abfe0640468 + .quad 0x7dc0b64c44cb9f44 + .quad 0x18a3e1ace3925dbf + .quad 0x7a3034862d0457c4 + .quad 0x4c498bf78a0c892e + + // 2^88 * 1 * B + + .quad 0x37d653fb1aa73196 + .quad 0x0f9495303fd76418 + .quad 0xad200b09fb3a17b2 + .quad 0x544d49292fc8613e + .quad 0x22d2aff530976b86 + .quad 0x8d90b806c2d24604 + .quad 0xdca1896c4de5bae5 + .quad 0x28005fe6c8340c17 + .quad 0x6aefba9f34528688 + .quad 0x5c1bff9425107da1 + .quad 0xf75bbbcd66d94b36 + .quad 0x72e472930f316dfa + + // 2^88 * 2 * B + + .quad 0x2695208c9781084f + .quad 0xb1502a0b23450ee1 + .quad 0xfd9daea603efde02 + .quad 0x5a9d2e8c2733a34c + .quad 0x07f3f635d32a7627 + .quad 0x7aaa4d865f6566f0 + .quad 0x3c85e79728d04450 + .quad 0x1fee7f000fe06438 + .quad 0x765305da03dbf7e5 + .quad 0xa4daf2491434cdbd + .quad 0x7b4ad5cdd24a88ec + .quad 0x00f94051ee040543 + + // 2^88 * 3 * B + + .quad 0x8d356b23c3d330b2 + .quad 0xf21c8b9bb0471b06 + .quad 0xb36c316c6e42b83c + .quad 0x07d79c7e8beab10d + .quad 0xd7ef93bb07af9753 + .quad 0x583ed0cf3db766a7 + .quad 0xce6998bf6e0b1ec5 + .quad 0x47b7ffd25dd40452 + .quad 0x87fbfb9cbc08dd12 + .quad 0x8a066b3ae1eec29b + .quad 0x0d57242bdb1fc1bf + .quad 0x1c3520a35ea64bb6 + + // 2^88 * 4 * B + + .quad 0x80d253a6bccba34a + .quad 0x3e61c3a13838219b + .quad 0x90c3b6019882e396 + .quad 0x1c3d05775d0ee66f + .quad 0xcda86f40216bc059 + .quad 0x1fbb231d12bcd87e + .quad 0xb4956a9e17c70990 + .quad 0x38750c3b66d12e55 + .quad 0x692ef1409422e51a + .quad 0xcbc0c73c2b5df671 + .quad 0x21014fe7744ce029 + .quad 0x0621e2c7d330487c + + // 2^88 * 5 * B + + .quad 0xaf9860cc8259838d + .quad 0x90ea48c1c69f9adc + .quad 0x6526483765581e30 + .quad 0x0007d6097bd3a5bc + .quad 0xb7ae1796b0dbf0f3 + .quad 0x54dfafb9e17ce196 + .quad 0x25923071e9aaa3b4 + .quad 0x5d8e589ca1002e9d + .quad 0xc0bf1d950842a94b + .quad 0xb2d3c363588f2e3e + .quad 0x0a961438bb51e2ef + .quad 0x1583d7783c1cbf86 + + // 2^88 * 6 * B + + .quad 0xeceea2ef5da27ae1 + .quad 0x597c3a1455670174 + .quad 0xc9a62a126609167a + .quad 0x252a5f2e81ed8f70 + .quad 0x90034704cc9d28c7 + .quad 0x1d1b679ef72cc58f + .quad 0x16e12b5fbe5b8726 + .quad 0x4958064e83c5580a + .quad 0x0d2894265066e80d + .quad 0xfcc3f785307c8c6b + .quad 0x1b53da780c1112fd + .quad 0x079c170bd843b388 + + // 2^88 * 7 * B + + .quad 0x0506ece464fa6fff + .quad 0xbee3431e6205e523 + .quad 0x3579422451b8ea42 + .quad 0x6dec05e34ac9fb00 + .quad 0xcdd6cd50c0d5d056 + .quad 0x9af7686dbb03573b + .quad 0x3ca6723ff3c3ef48 + .quad 0x6768c0d7317b8acc + .quad 0x94b625e5f155c1b3 + .quad 0x417bf3a7997b7b91 + .quad 0xc22cbddc6d6b2600 + .quad 0x51445e14ddcd52f4 + + // 2^88 * 8 * B + + .quad 0x57502b4b3b144951 + .quad 0x8e67ff6b444bbcb3 + .quad 0xb8bd6927166385db + .quad 0x13186f31e39295c8 + .quad 0x893147ab2bbea455 + .quad 0x8c53a24f92079129 + .quad 0x4b49f948be30f7a7 + .quad 0x12e990086e4fd43d + .quad 0xf10c96b37fdfbb2e + .quad 0x9f9a935e121ceaf9 + .quad 0xdf1136c43a5b983f + .quad 0x77b2e3f05d3e99af + + // 2^92 * 1 * B + + .quad 0xfd0d75879cf12657 + .quad 0xe82fef94e53a0e29 + .quad 0xcc34a7f05bbb4be7 + .quad 0x0b251172a50c38a2 + .quad 0x9532f48fcc5cd29b + .quad 0x2ba851bea3ce3671 + .quad 0x32dacaa051122941 + .quad 0x478d99d9350004f2 + .quad 0x1d5ad94890bb02c0 + .quad 0x50e208b10ec25115 + .quad 0xa26a22894ef21702 + .quad 0x4dc923343b524805 + + // 2^92 * 2 * B + + .quad 0xe3828c400f8086b6 + .quad 0x3f77e6f7979f0dc8 + .quad 0x7ef6de304df42cb4 + .quad 0x5265797cb6abd784 + .quad 0x3ad3e3ebf36c4975 + .quad 0xd75d25a537862125 + .quad 0xe873943da025a516 + .quad 0x6bbc7cb4c411c847 + .quad 0x3c6f9cd1d4a50d56 + .quad 0xb6244077c6feab7e + .quad 0x6ff9bf483580972e + .quad 0x00375883b332acfb + + // 2^92 * 3 * B + + .quad 0x0001b2cd28cb0940 + .quad 0x63fb51a06f1c24c9 + .quad 0xb5ad8691dcd5ca31 + .quad 0x67238dbd8c450660 + .quad 0xc98bec856c75c99c + .quad 0xe44184c000e33cf4 + .quad 0x0a676b9bba907634 + .quad 0x669e2cb571f379d7 + .quad 0xcb116b73a49bd308 + .quad 0x025aad6b2392729e + .quad 0xb4793efa3f55d9b1 + .quad 0x72a1056140678bb9 + + // 2^92 * 4 * B + + .quad 0xa2b6812b1cc9249d + .quad 0x62866eee21211f58 + .quad 0x2cb5c5b85df10ece + .quad 0x03a6b259e263ae00 + .quad 0x0d8d2909e2e505b6 + .quad 0x98ca78abc0291230 + .quad 0x77ef5569a9b12327 + .quad 0x7c77897b81439b47 + .quad 0xf1c1b5e2de331cb5 + .quad 0x5a9f5d8e15fca420 + .quad 0x9fa438f17bd932b1 + .quad 0x2a381bf01c6146e7 + + // 2^92 * 5 * B + + .quad 0xac9b9879cfc811c1 + .quad 0x8b7d29813756e567 + .quad 0x50da4e607c70edfc + .quad 0x5dbca62f884400b6 + .quad 0xf7c0be32b534166f + .quad 0x27e6ca6419cf70d4 + .quad 0x934df7d7a957a759 + .quad 0x5701461dabdec2aa + .quad 0x2c6747402c915c25 + .quad 0x1bdcd1a80b0d340a + .quad 0x5e5601bd07b43f5f + .quad 0x2555b4e05539a242 + + // 2^92 * 6 * B + + .quad 0x6fc09f5266ddd216 + .quad 0xdce560a7c8e37048 + .quad 0xec65939da2df62fd + .quad 0x7a869ae7e52ed192 + .quad 0x78409b1d87e463d4 + .quad 0xad4da95acdfb639d + .quad 0xec28773755259b9c + .quad 0x69c806e9c31230ab + .quad 0x7b48f57414bb3f22 + .quad 0x68c7cee4aedccc88 + .quad 0xed2f936179ed80be + .quad 0x25d70b885f77bc4b + + // 2^92 * 7 * B + + .quad 0x4151c3d9762bf4de + .quad 0x083f435f2745d82b + .quad 0x29775a2e0d23ddd5 + .quad 0x138e3a6269a5db24 + .quad 0x98459d29bb1ae4d4 + .quad 0x56b9c4c739f954ec + .quad 0x832743f6c29b4b3e + .quad 0x21ea8e2798b6878a + .quad 0x87bef4b46a5a7b9c + .quad 0xd2299d1b5fc1d062 + .quad 0x82409818dd321648 + .quad 0x5c5abeb1e5a2e03d + + // 2^92 * 8 * B + + .quad 0x14722af4b73c2ddb + .quad 0xbc470c5f5a05060d + .quad 0x00943eac2581b02e + .quad 0x0e434b3b1f499c8f + .quad 0x02cde6de1306a233 + .quad 0x7b5a52a2116f8ec7 + .quad 0xe1c681f4c1163b5b + .quad 0x241d350660d32643 + .quad 0x6be4404d0ebc52c7 + .quad 0xae46233bb1a791f5 + .quad 0x2aec170ed25db42b + .quad 0x1d8dfd966645d694 + + // 2^96 * 1 * B + + .quad 0x296fa9c59c2ec4de + .quad 0xbc8b61bf4f84f3cb + .quad 0x1c7706d917a8f908 + .quad 0x63b795fc7ad3255d + .quad 0xd598639c12ddb0a4 + .quad 0xa5d19f30c024866b + .quad 0xd17c2f0358fce460 + .quad 0x07a195152e095e8a + .quad 0xa8368f02389e5fc8 + .quad 0x90433b02cf8de43b + .quad 0xafa1fd5dc5412643 + .quad 0x3e8fe83d032f0137 + + // 2^96 * 2 * B + + .quad 0x2f8b15b90570a294 + .quad 0x94f2427067084549 + .quad 0xde1c5ae161bbfd84 + .quad 0x75ba3b797fac4007 + .quad 0x08704c8de8efd13c + .quad 0xdfc51a8e33e03731 + .quad 0xa59d5da51260cde3 + .quad 0x22d60899a6258c86 + .quad 0x6239dbc070cdd196 + .quad 0x60fe8a8b6c7d8a9a + .quad 0xb38847bceb401260 + .quad 0x0904d07b87779e5e + + // 2^96 * 3 * B + + .quad 0xb4ce1fd4ddba919c + .quad 0xcf31db3ec74c8daa + .quad 0x2c63cc63ad86cc51 + .quad 0x43e2143fbc1dde07 + .quad 0xf4322d6648f940b9 + .quad 0x06952f0cbd2d0c39 + .quad 0x167697ada081f931 + .quad 0x6240aacebaf72a6c + .quad 0xf834749c5ba295a0 + .quad 0xd6947c5bca37d25a + .quad 0x66f13ba7e7c9316a + .quad 0x56bdaf238db40cac + + // 2^96 * 4 * B + + .quad 0x362ab9e3f53533eb + .quad 0x338568d56eb93d40 + .quad 0x9e0e14521d5a5572 + .quad 0x1d24a86d83741318 + .quad 0x1310d36cc19d3bb2 + .quad 0x062a6bb7622386b9 + .quad 0x7c9b8591d7a14f5c + .quad 0x03aa31507e1e5754 + .quad 0xf4ec7648ffd4ce1f + .quad 0xe045eaf054ac8c1c + .quad 0x88d225821d09357c + .quad 0x43b261dc9aeb4859 + + // 2^96 * 5 * B + + .quad 0xe55b1e1988bb79bb + .quad 0xa09ed07dc17a359d + .quad 0xb02c2ee2603dea33 + .quad 0x326055cf5b276bc2 + .quad 0x19513d8b6c951364 + .quad 0x94fe7126000bf47b + .quad 0x028d10ddd54f9567 + .quad 0x02b4d5e242940964 + .quad 0xb4a155cb28d18df2 + .quad 0xeacc4646186ce508 + .quad 0xc49cf4936c824389 + .quad 0x27a6c809ae5d3410 + + // 2^96 * 6 * B + + .quad 0x8ba6ebcd1f0db188 + .quad 0x37d3d73a675a5be8 + .quad 0xf22edfa315f5585a + .quad 0x2cb67174ff60a17e + .quad 0xcd2c270ac43d6954 + .quad 0xdd4a3e576a66cab2 + .quad 0x79fa592469d7036c + .quad 0x221503603d8c2599 + .quad 0x59eecdf9390be1d0 + .quad 0xa9422044728ce3f1 + .quad 0x82891c667a94f0f4 + .quad 0x7b1df4b73890f436 + + // 2^96 * 7 * B + + .quad 0xe492f2e0b3b2a224 + .quad 0x7c6c9e062b551160 + .quad 0x15eb8fe20d7f7b0e + .quad 0x61fcef2658fc5992 + .quad 0x5f2e221807f8f58c + .quad 0xe3555c9fd49409d4 + .quad 0xb2aaa88d1fb6a630 + .quad 0x68698245d352e03d + .quad 0xdbb15d852a18187a + .quad 0xf3e4aad386ddacd7 + .quad 0x44bae2810ff6c482 + .quad 0x46cf4c473daf01cf + + // 2^96 * 8 * B + + .quad 0x426525ed9ec4e5f9 + .quad 0x0e5eda0116903303 + .quad 0x72b1a7f2cbe5cadc + .quad 0x29387bcd14eb5f40 + .quad 0x213c6ea7f1498140 + .quad 0x7c1e7ef8392b4854 + .quad 0x2488c38c5629ceba + .quad 0x1065aae50d8cc5bb + .quad 0x1c2c4525df200d57 + .quad 0x5c3b2dd6bfca674a + .quad 0x0a07e7b1e1834030 + .quad 0x69a198e64f1ce716 + + // 2^100 * 1 * B + + .quad 0x7afcd613efa9d697 + .quad 0x0cc45aa41c067959 + .quad 0xa56fe104c1fada96 + .quad 0x3a73b70472e40365 + .quad 0x7b26e56b9e2d4734 + .quad 0xc4c7132b81c61675 + .quad 0xef5c9525ec9cde7f + .quad 0x39c80b16e71743ad + .quad 0x0f196e0d1b826c68 + .quad 0xf71ff0e24960e3db + .quad 0x6113167023b7436c + .quad 0x0cf0ea5877da7282 + + // 2^100 * 2 * B + + .quad 0x196c80a4ddd4ccbd + .quad 0x22e6f55d95f2dd9d + .quad 0xc75e33c740d6c71b + .quad 0x7bb51279cb3c042f + .quad 0xe332ced43ba6945a + .quad 0xde0b1361e881c05d + .quad 0x1ad40f095e67ed3b + .quad 0x5da8acdab8c63d5d + .quad 0xc4b6664a3a70159f + .quad 0x76194f0f0a904e14 + .quad 0xa5614c39a4096c13 + .quad 0x6cd0ff50979feced + + // 2^100 * 3 * B + + .quad 0xc0e067e78f4428ac + .quad 0x14835ab0a61135e3 + .quad 0xf21d14f338062935 + .quad 0x6390a4c8df04849c + .quad 0x7fecfabdb04ba18e + .quad 0xd0fc7bfc3bddbcf7 + .quad 0xa41d486e057a131c + .quad 0x641a4391f2223a61 + .quad 0xc5c6b95aa606a8db + .quad 0x914b7f9eb06825f1 + .quad 0x2a731f6b44fc9eff + .quad 0x30ddf38562705cfc + + // 2^100 * 4 * B + + .quad 0x4e3dcbdad1bff7f9 + .quad 0xc9118e8220645717 + .quad 0xbacccebc0f189d56 + .quad 0x1b4822e9d4467668 + .quad 0x33bef2bd68bcd52c + .quad 0xc649dbb069482ef2 + .quad 0xb5b6ee0c41cb1aee + .quad 0x5c294d270212a7e5 + .quad 0xab360a7f25563781 + .quad 0x2512228a480f7958 + .quad 0xc75d05276114b4e3 + .quad 0x222d9625d976fe2a + + // 2^100 * 5 * B + + .quad 0x1c717f85b372ace1 + .quad 0x81930e694638bf18 + .quad 0x239cad056bc08b58 + .quad 0x0b34271c87f8fff4 + .quad 0x0f94be7e0a344f85 + .quad 0xeb2faa8c87f22c38 + .quad 0x9ce1e75e4ee16f0f + .quad 0x43e64e5418a08dea + .quad 0x8155e2521a35ce63 + .quad 0xbe100d4df912028e + .quad 0xbff80bf8a57ddcec + .quad 0x57342dc96d6bc6e4 + + // 2^100 * 6 * B + + .quad 0xefeef065c8ce5998 + .quad 0xbf029510b5cbeaa2 + .quad 0x8c64a10620b7c458 + .quad 0x35134fb231c24855 + .quad 0xf3c3bcb71e707bf6 + .quad 0x351d9b8c7291a762 + .quad 0x00502e6edad69a33 + .quad 0x522f521f1ec8807f + .quad 0x272c1f46f9a3902b + .quad 0xc91ba3b799657bcc + .quad 0xae614b304f8a1c0e + .quad 0x7afcaad70b99017b + + // 2^100 * 7 * B + + .quad 0xc25ded54a4b8be41 + .quad 0x902d13e11bb0e2dd + .quad 0x41f43233cde82ab2 + .quad 0x1085faa5c3aae7cb + .quad 0xa88141ecef842b6b + .quad 0x55e7b14797abe6c5 + .quad 0x8c748f9703784ffe + .quad 0x5b50a1f7afcd00b7 + .quad 0x9b840f66f1361315 + .quad 0x18462242701003e9 + .quad 0x65ed45fae4a25080 + .quad 0x0a2862393fda7320 + + // 2^100 * 8 * B + + .quad 0x46ab13c8347cbc9d + .quad 0x3849e8d499c12383 + .quad 0x4cea314087d64ac9 + .quad 0x1f354134b1a29ee7 + .quad 0x960e737b6ecb9d17 + .quad 0xfaf24948d67ceae1 + .quad 0x37e7a9b4d55e1b89 + .quad 0x5cb7173cb46c59eb + .quad 0x4a89e68b82b7abf0 + .quad 0xf41cd9279ba6b7b9 + .quad 0x16e6c210e18d876f + .quad 0x7cacdb0f7f1b09c6 + + // 2^104 * 1 * B + + .quad 0x9062b2e0d91a78bc + .quad 0x47c9889cc8509667 + .quad 0x9df54a66405070b8 + .quad 0x7369e6a92493a1bf + .quad 0xe1014434dcc5caed + .quad 0x47ed5d963c84fb33 + .quad 0x70019576ed86a0e7 + .quad 0x25b2697bd267f9e4 + .quad 0x9d673ffb13986864 + .quad 0x3ca5fbd9415dc7b8 + .quad 0xe04ecc3bdf273b5e + .quad 0x1420683db54e4cd2 + + // 2^104 * 2 * B + + .quad 0xb478bd1e249dd197 + .quad 0x620c35005e58c102 + .quad 0xfb02d32fccbaac5c + .quad 0x60b63bebf508a72d + .quad 0x34eebb6fc1cc5ad0 + .quad 0x6a1b0ce99646ac8b + .quad 0xd3b0da49a66bde53 + .quad 0x31e83b4161d081c1 + .quad 0x97e8c7129e062b4f + .quad 0x49e48f4f29320ad8 + .quad 0x5bece14b6f18683f + .quad 0x55cf1eb62d550317 + + // 2^104 * 3 * B + + .quad 0x5879101065c23d58 + .quad 0x8b9d086d5094819c + .quad 0xe2402fa912c55fa7 + .quad 0x669a6564570891d4 + .quad 0x3076b5e37df58c52 + .quad 0xd73ab9dde799cc36 + .quad 0xbd831ce34913ee20 + .quad 0x1a56fbaa62ba0133 + .quad 0x943e6b505c9dc9ec + .quad 0x302557bba77c371a + .quad 0x9873ae5641347651 + .quad 0x13c4836799c58a5c + + // 2^104 * 4 * B + + .quad 0x423a5d465ab3e1b9 + .quad 0xfc13c187c7f13f61 + .quad 0x19f83664ecb5b9b6 + .quad 0x66f80c93a637b607 + .quad 0xc4dcfb6a5d8bd080 + .quad 0xdeebc4ec571a4842 + .quad 0xd4b2e883b8e55365 + .quad 0x50bdc87dc8e5b827 + .quad 0x606d37836edfe111 + .quad 0x32353e15f011abd9 + .quad 0x64b03ac325b73b96 + .quad 0x1dd56444725fd5ae + + // 2^104 * 5 * B + + .quad 0x8fa47ff83362127d + .quad 0xbc9f6ac471cd7c15 + .quad 0x6e71454349220c8b + .quad 0x0e645912219f732e + .quad 0xc297e60008bac89a + .quad 0x7d4cea11eae1c3e0 + .quad 0xf3e38be19fe7977c + .quad 0x3a3a450f63a305cd + .quad 0x078f2f31d8394627 + .quad 0x389d3183de94a510 + .quad 0xd1e36c6d17996f80 + .quad 0x318c8d9393a9a87b + + // 2^104 * 6 * B + + .quad 0xf2745d032afffe19 + .quad 0x0c9f3c497f24db66 + .quad 0xbc98d3e3ba8598ef + .quad 0x224c7c679a1d5314 + .quad 0x5d669e29ab1dd398 + .quad 0xfc921658342d9e3b + .quad 0x55851dfdf35973cd + .quad 0x509a41c325950af6 + .quad 0xbdc06edca6f925e9 + .quad 0x793ef3f4641b1f33 + .quad 0x82ec12809d833e89 + .quad 0x05bff02328a11389 + + // 2^104 * 7 * B + + .quad 0x3632137023cae00b + .quad 0x544acf0ad1accf59 + .quad 0x96741049d21a1c88 + .quad 0x780b8cc3fa2a44a7 + .quad 0x6881a0dd0dc512e4 + .quad 0x4fe70dc844a5fafe + .quad 0x1f748e6b8f4a5240 + .quad 0x576277cdee01a3ea + .quad 0x1ef38abc234f305f + .quad 0x9a577fbd1405de08 + .quad 0x5e82a51434e62a0d + .quad 0x5ff418726271b7a1 + + // 2^104 * 8 * B + + .quad 0x398e080c1789db9d + .quad 0xa7602025f3e778f5 + .quad 0xfa98894c06bd035d + .quad 0x106a03dc25a966be + .quad 0xe5db47e813b69540 + .quad 0xf35d2a3b432610e1 + .quad 0xac1f26e938781276 + .quad 0x29d4db8ca0a0cb69 + .quad 0xd9ad0aaf333353d0 + .quad 0x38669da5acd309e5 + .quad 0x3c57658ac888f7f0 + .quad 0x4ab38a51052cbefa + + // 2^108 * 1 * B + + .quad 0xdfdacbee4324c0e9 + .quad 0x054442883f955bb7 + .quad 0xdef7aaa8ea31609f + .quad 0x68aee70642287cff + .quad 0xf68fe2e8809de054 + .quad 0xe3bc096a9c82bad1 + .quad 0x076353d40aadbf45 + .quad 0x7b9b1fb5dea1959e + .quad 0xf01cc8f17471cc0c + .quad 0x95242e37579082bb + .quad 0x27776093d3e46b5f + .quad 0x2d13d55a28bd85fb + + // 2^108 * 2 * B + + .quad 0xfac5d2065b35b8da + .quad 0xa8da8a9a85624bb7 + .quad 0xccd2ca913d21cd0f + .quad 0x6b8341ee8bf90d58 + .quad 0xbf019cce7aee7a52 + .quad 0xa8ded2b6e454ead3 + .quad 0x3c619f0b87a8bb19 + .quad 0x3619b5d7560916d8 + .quad 0x3579f26b0282c4b2 + .quad 0x64d592f24fafefae + .quad 0xb7cded7b28c8c7c0 + .quad 0x6a927b6b7173a8d7 + + // 2^108 * 3 * B + + .quad 0x1f6db24f986e4656 + .quad 0x1021c02ed1e9105b + .quad 0xf8ff3fff2cc0a375 + .quad 0x1d2a6bf8c6c82592 + .quad 0x8d7040863ece88eb + .quad 0xf0e307a980eec08c + .quad 0xac2250610d788fda + .quad 0x056d92a43a0d478d + .quad 0x1b05a196fc3da5a1 + .quad 0x77d7a8c243b59ed0 + .quad 0x06da3d6297d17918 + .quad 0x66fbb494f12353f7 + + // 2^108 * 4 * B + + .quad 0x751a50b9d85c0fb8 + .quad 0xd1afdc258bcf097b + .quad 0x2f16a6a38309a969 + .quad 0x14ddff9ee5b00659 + .quad 0xd6d70996f12309d6 + .quad 0xdbfb2385e9c3d539 + .quad 0x46d602b0f7552411 + .quad 0x270a0b0557843e0c + .quad 0x61ff0640a7862bcc + .quad 0x81cac09a5f11abfe + .quad 0x9047830455d12abb + .quad 0x19a4bde1945ae873 + + // 2^108 * 5 * B + + .quad 0x9b9f26f520a6200a + .quad 0x64804443cf13eaf8 + .quad 0x8a63673f8631edd3 + .quad 0x72bbbce11ed39dc1 + .quad 0x40c709dec076c49f + .quad 0x657bfaf27f3e53f6 + .quad 0x40662331eca042c4 + .quad 0x14b375487eb4df04 + .quad 0xae853c94ab66dc47 + .quad 0xeb62343edf762d6e + .quad 0xf08e0e186fb2f7d1 + .quad 0x4f0b1c02700ab37a + + // 2^108 * 6 * B + + .quad 0xe1706787d81951fa + .quad 0xa10a2c8eb290c77b + .quad 0xe7382fa03ed66773 + .quad 0x0a4d84710bcc4b54 + .quad 0x79fd21ccc1b2e23f + .quad 0x4ae7c281453df52a + .quad 0xc8172ec9d151486b + .quad 0x68abe9443e0a7534 + .quad 0xda12c6c407831dcb + .quad 0x0da230d74d5c510d + .quad 0x4ab1531e6bd404e1 + .quad 0x4106b166bcf440ef + + // 2^108 * 7 * B + + .quad 0x02e57a421cd23668 + .quad 0x4ad9fb5d0eaef6fd + .quad 0x954e6727b1244480 + .quad 0x7f792f9d2699f331 + .quad 0xa485ccd539e4ecf2 + .quad 0x5aa3f3ad0555bab5 + .quad 0x145e3439937df82d + .quad 0x1238b51e1214283f + .quad 0x0b886b925fd4d924 + .quad 0x60906f7a3626a80d + .quad 0xecd367b4b98abd12 + .quad 0x2876beb1def344cf + + // 2^108 * 8 * B + + .quad 0xdc84e93563144691 + .quad 0x632fe8a0d61f23f4 + .quad 0x4caa800612a9a8d5 + .quad 0x48f9dbfa0e9918d3 + .quad 0xd594b3333a8a85f8 + .quad 0x4ea37689e78d7d58 + .quad 0x73bf9f455e8e351f + .quad 0x5507d7d2bc41ebb4 + .quad 0x1ceb2903299572fc + .quad 0x7c8ccaa29502d0ee + .quad 0x91bfa43411cce67b + .quad 0x5784481964a831e7 + + // 2^112 * 1 * B + + .quad 0xda7c2b256768d593 + .quad 0x98c1c0574422ca13 + .quad 0xf1a80bd5ca0ace1d + .quad 0x29cdd1adc088a690 + .quad 0xd6cfd1ef5fddc09c + .quad 0xe82b3efdf7575dce + .quad 0x25d56b5d201634c2 + .quad 0x3041c6bb04ed2b9b + .quad 0x0ff2f2f9d956e148 + .quad 0xade797759f356b2e + .quad 0x1a4698bb5f6c025c + .quad 0x104bbd6814049a7b + + // 2^112 * 2 * B + + .quad 0x51f0fd3168f1ed67 + .quad 0x2c811dcdd86f3bc2 + .quad 0x44dc5c4304d2f2de + .quad 0x5be8cc57092a7149 + .quad 0xa95d9a5fd67ff163 + .quad 0xe92be69d4cc75681 + .quad 0xb7f8024cde20f257 + .quad 0x204f2a20fb072df5 + .quad 0xc8143b3d30ebb079 + .quad 0x7589155abd652e30 + .quad 0x653c3c318f6d5c31 + .quad 0x2570fb17c279161f + + // 2^112 * 3 * B + + .quad 0x3efa367f2cb61575 + .quad 0xf5f96f761cd6026c + .quad 0xe8c7142a65b52562 + .quad 0x3dcb65ea53030acd + .quad 0x192ea9550bb8245a + .quad 0xc8e6fba88f9050d1 + .quad 0x7986ea2d88a4c935 + .quad 0x241c5f91de018668 + .quad 0x28d8172940de6caa + .quad 0x8fbf2cf022d9733a + .quad 0x16d7fcdd235b01d1 + .quad 0x08420edd5fcdf0e5 + + // 2^112 * 4 * B + + .quad 0xcdff20ab8362fa4a + .quad 0x57e118d4e21a3e6e + .quad 0xe3179617fc39e62b + .quad 0x0d9a53efbc1769fd + .quad 0x0358c34e04f410ce + .quad 0xb6135b5a276e0685 + .quad 0x5d9670c7ebb91521 + .quad 0x04d654f321db889c + .quad 0x5e7dc116ddbdb5d5 + .quad 0x2954deb68da5dd2d + .quad 0x1cb608173334a292 + .quad 0x4a7a4f2618991ad7 + + // 2^112 * 5 * B + + .quad 0xf4a718025fb15f95 + .quad 0x3df65f346b5c1b8f + .quad 0xcdfcf08500e01112 + .quad 0x11b50c4cddd31848 + .quad 0x24c3b291af372a4b + .quad 0x93da8270718147f2 + .quad 0xdd84856486899ef2 + .quad 0x4a96314223e0ee33 + .quad 0xa6e8274408a4ffd6 + .quad 0x738e177e9c1576d9 + .quad 0x773348b63d02b3f2 + .quad 0x4f4bce4dce6bcc51 + + // 2^112 * 6 * B + + .quad 0xa71fce5ae2242584 + .quad 0x26ea725692f58a9e + .quad 0xd21a09d71cea3cf4 + .quad 0x73fcdd14b71c01e6 + .quad 0x30e2616ec49d0b6f + .quad 0xe456718fcaec2317 + .quad 0x48eb409bf26b4fa6 + .quad 0x3042cee561595f37 + .quad 0x427e7079449bac41 + .quad 0x855ae36dbce2310a + .quad 0x4cae76215f841a7c + .quad 0x389e740c9a9ce1d6 + + // 2^112 * 7 * B + + .quad 0x64fcb3ae34dcb9ce + .quad 0x97500323e348d0ad + .quad 0x45b3f07d62c6381b + .quad 0x61545379465a6788 + .quad 0xc9bd78f6570eac28 + .quad 0xe55b0b3227919ce1 + .quad 0x65fc3eaba19b91ed + .quad 0x25c425e5d6263690 + .quad 0x3f3e06a6f1d7de6e + .quad 0x3ef976278e062308 + .quad 0x8c14f6264e8a6c77 + .quad 0x6539a08915484759 + + // 2^112 * 8 * B + + .quad 0xe9d21f74c3d2f773 + .quad 0xc150544125c46845 + .quad 0x624e5ce8f9b99e33 + .quad 0x11c5e4aac5cd186c + .quad 0xddc4dbd414bb4a19 + .quad 0x19b2bc3c98424f8e + .quad 0x48a89fd736ca7169 + .quad 0x0f65320ef019bd90 + .quad 0xd486d1b1cafde0c6 + .quad 0x4f3fe6e3163b5181 + .quad 0x59a8af0dfaf2939a + .quad 0x4cabc7bdec33072a + + // 2^116 * 1 * B + + .quad 0x16faa8fb532f7428 + .quad 0xdbd42ea046a4e272 + .quad 0x5337653b8b9ea480 + .quad 0x4065947223973f03 + .quad 0xf7c0a19c1a54a044 + .quad 0x4a1c5e2477bd9fbb + .quad 0xa6e3ca115af22972 + .quad 0x1819bb953f2e9e0d + .quad 0x498fbb795e042e84 + .quad 0x7d0dd89a7698b714 + .quad 0x8bfb0ba427fe6295 + .quad 0x36ba82e721200524 + + // 2^116 * 2 * B + + .quad 0xd60ecbb74245ec41 + .quad 0xfd9be89e34348716 + .quad 0xc9240afee42284de + .quad 0x4472f648d0531db4 + .quad 0xc8d69d0a57274ed5 + .quad 0x45ba803260804b17 + .quad 0xdf3cda102255dfac + .quad 0x77d221232709b339 + .quad 0x498a6d7064ad94d8 + .quad 0xa5b5c8fd9af62263 + .quad 0x8ca8ed0545c141f4 + .quad 0x2c63bec3662d358c + + // 2^116 * 3 * B + + .quad 0x7fe60d8bea787955 + .quad 0xb9dc117eb5f401b7 + .quad 0x91c7c09a19355cce + .quad 0x22692ef59442bedf + .quad 0x9a518b3a8586f8bf + .quad 0x9ee71af6cbb196f0 + .quad 0xaa0625e6a2385cf2 + .quad 0x1deb2176ddd7c8d1 + .quad 0x8563d19a2066cf6c + .quad 0x401bfd8c4dcc7cd7 + .quad 0xd976a6becd0d8f62 + .quad 0x67cfd773a278b05e + + // 2^116 * 4 * B + + .quad 0x8dec31faef3ee475 + .quad 0x99dbff8a9e22fd92 + .quad 0x512d11594e26cab1 + .quad 0x0cde561eec4310b9 + .quad 0x2d5fa9855a4e586a + .quad 0x65f8f7a449beab7e + .quad 0xaa074dddf21d33d3 + .quad 0x185cba721bcb9dee + .quad 0x93869da3f4e3cb41 + .quad 0xbf0392f540f7977e + .quad 0x026204fcd0463b83 + .quad 0x3ec91a769eec6eed + + // 2^116 * 5 * B + + .quad 0x1e9df75bf78166ad + .quad 0x4dfda838eb0cd7af + .quad 0xba002ed8c1eaf988 + .quad 0x13fedb3e11f33cfc + .quad 0x0fad2fb7b0a3402f + .quad 0x46615ecbfb69f4a8 + .quad 0xf745bcc8c5f8eaa6 + .quad 0x7a5fa8794a94e896 + .quad 0x52958faa13cd67a1 + .quad 0x965ee0818bdbb517 + .quad 0x16e58daa2e8845b3 + .quad 0x357d397d5499da8f + + // 2^116 * 6 * B + + .quad 0x1ebfa05fb0bace6c + .quad 0xc934620c1caf9a1e + .quad 0xcc771cc41d82b61a + .quad 0x2d94a16aa5f74fec + .quad 0x481dacb4194bfbf8 + .quad 0x4d77e3f1bae58299 + .quad 0x1ef4612e7d1372a0 + .quad 0x3a8d867e70ff69e1 + .quad 0x6f58cd5d55aff958 + .quad 0xba3eaa5c75567721 + .quad 0x75c123999165227d + .quad 0x69be1343c2f2b35e + + // 2^116 * 7 * B + + .quad 0x0e091d5ee197c92a + .quad 0x4f51019f2945119f + .quad 0x143679b9f034e99c + .quad 0x7d88112e4d24c696 + .quad 0x82bbbdac684b8de3 + .quad 0xa2f4c7d03fca0718 + .quad 0x337f92fbe096aaa8 + .quad 0x200d4d8c63587376 + .quad 0x208aed4b4893b32b + .quad 0x3efbf23ebe59b964 + .quad 0xd762deb0dba5e507 + .quad 0x69607bd681bd9d94 + + // 2^116 * 8 * B + + .quad 0xf6be021068de1ce1 + .quad 0xe8d518e70edcbc1f + .quad 0xe3effdd01b5505a5 + .quad 0x35f63353d3ec3fd0 + .quad 0x3b7f3bd49323a902 + .quad 0x7c21b5566b2c6e53 + .quad 0xe5ba8ff53a7852a7 + .quad 0x28bc77a5838ece00 + .quad 0x63ba78a8e25d8036 + .quad 0x63651e0094333490 + .quad 0x48d82f20288ce532 + .quad 0x3a31abfa36b57524 + + // 2^120 * 1 * B + + .quad 0x239e9624089c0a2e + .quad 0xc748c4c03afe4738 + .quad 0x17dbed2a764fa12a + .quad 0x639b93f0321c8582 + .quad 0xc08f788f3f78d289 + .quad 0xfe30a72ca1404d9f + .quad 0xf2778bfccf65cc9d + .quad 0x7ee498165acb2021 + .quad 0x7bd508e39111a1c3 + .quad 0x2b2b90d480907489 + .quad 0xe7d2aec2ae72fd19 + .quad 0x0edf493c85b602a6 + + // 2^120 * 2 * B + + .quad 0xaecc8158599b5a68 + .quad 0xea574f0febade20e + .quad 0x4fe41d7422b67f07 + .quad 0x403b92e3019d4fb4 + .quad 0x6767c4d284764113 + .quad 0xa090403ff7f5f835 + .quad 0x1c8fcffacae6bede + .quad 0x04c00c54d1dfa369 + .quad 0x4dc22f818b465cf8 + .quad 0x71a0f35a1480eff8 + .quad 0xaee8bfad04c7d657 + .quad 0x355bb12ab26176f4 + + // 2^120 * 3 * B + + .quad 0xa71e64cc7493bbf4 + .quad 0xe5bd84d9eca3b0c3 + .quad 0x0a6bc50cfa05e785 + .quad 0x0f9b8132182ec312 + .quad 0xa301dac75a8c7318 + .quad 0xed90039db3ceaa11 + .quad 0x6f077cbf3bae3f2d + .quad 0x7518eaf8e052ad8e + .quad 0xa48859c41b7f6c32 + .quad 0x0f2d60bcf4383298 + .quad 0x1815a929c9b1d1d9 + .quad 0x47c3871bbb1755c4 + + // 2^120 * 4 * B + + .quad 0x5144539771ec4f48 + .quad 0xf805b17dc98c5d6e + .quad 0xf762c11a47c3c66b + .quad 0x00b89b85764699dc + .quad 0xfbe65d50c85066b0 + .quad 0x62ecc4b0b3a299b0 + .quad 0xe53754ea441ae8e0 + .quad 0x08fea02ce8d48d5f + .quad 0x824ddd7668deead0 + .quad 0xc86445204b685d23 + .quad 0xb514cfcd5d89d665 + .quad 0x473829a74f75d537 + + // 2^120 * 5 * B + + .quad 0x82d2da754679c418 + .quad 0xe63bd7d8b2618df0 + .quad 0x355eef24ac47eb0a + .quad 0x2078684c4833c6b4 + .quad 0x23d9533aad3902c9 + .quad 0x64c2ddceef03588f + .quad 0x15257390cfe12fb4 + .quad 0x6c668b4d44e4d390 + .quad 0x3b48cf217a78820c + .quad 0xf76a0ab281273e97 + .quad 0xa96c65a78c8eed7b + .quad 0x7411a6054f8a433f + + // 2^120 * 6 * B + + .quad 0x4d659d32b99dc86d + .quad 0x044cdc75603af115 + .quad 0xb34c712cdcc2e488 + .quad 0x7c136574fb8134ff + .quad 0x579ae53d18b175b4 + .quad 0x68713159f392a102 + .quad 0x8455ecba1eef35f5 + .quad 0x1ec9a872458c398f + .quad 0xb8e6a4d400a2509b + .quad 0x9b81d7020bc882b4 + .quad 0x57e7cc9bf1957561 + .quad 0x3add88a5c7cd6460 + + // 2^120 * 7 * B + + .quad 0xab895770b635dcf2 + .quad 0x02dfef6cf66c1fbc + .quad 0x85530268beb6d187 + .quad 0x249929fccc879e74 + .quad 0x85c298d459393046 + .quad 0x8f7e35985ff659ec + .quad 0x1d2ca22af2f66e3a + .quad 0x61ba1131a406a720 + .quad 0xa3d0a0f116959029 + .quad 0x023b6b6cba7ebd89 + .quad 0x7bf15a3e26783307 + .quad 0x5620310cbbd8ece7 + + // 2^120 * 8 * B + + .quad 0x528993434934d643 + .quad 0xb9dbf806a51222f5 + .quad 0x8f6d878fc3f41c22 + .quad 0x37676a2a4d9d9730 + .quad 0x6646b5f477e285d6 + .quad 0x40e8ff676c8f6193 + .quad 0xa6ec7311abb594dd + .quad 0x7ec846f3658cec4d + .quad 0x9b5e8f3f1da22ec7 + .quad 0x130f1d776c01cd13 + .quad 0x214c8fcfa2989fb8 + .quad 0x6daaf723399b9dd5 + + // 2^124 * 1 * B + + .quad 0x591e4a5610628564 + .quad 0x2a4bb87ca8b4df34 + .quad 0xde2a2572e7a38e43 + .quad 0x3cbdabd9fee5046e + .quad 0x81aebbdd2cd13070 + .quad 0x962e4325f85a0e9e + .quad 0xde9391aacadffecb + .quad 0x53177fda52c230e6 + .quad 0xa7bc970650b9de79 + .quad 0x3d12a7fbc301b59b + .quad 0x02652e68d36ae38c + .quad 0x79d739835a6199dc + + // 2^124 * 2 * B + + .quad 0xd9354df64131c1bd + .quad 0x758094a186ec5822 + .quad 0x4464ee12e459f3c2 + .quad 0x6c11fce4cb133282 + .quad 0x21c9d9920d591737 + .quad 0x9bea41d2e9b46cd6 + .quad 0xe20e84200d89bfca + .quad 0x79d99f946eae5ff8 + .quad 0xf17b483568673205 + .quad 0x387deae83caad96c + .quad 0x61b471fd56ffe386 + .quad 0x31741195b745a599 + + // 2^124 * 3 * B + + .quad 0xe8d10190b77a360b + .quad 0x99b983209995e702 + .quad 0xbd4fdff8fa0247aa + .quad 0x2772e344e0d36a87 + .quad 0x17f8ba683b02a047 + .quad 0x50212096feefb6c8 + .quad 0x70139be21556cbe2 + .quad 0x203e44a11d98915b + .quad 0xd6863eba37b9e39f + .quad 0x105bc169723b5a23 + .quad 0x104f6459a65c0762 + .quad 0x567951295b4d38d4 + + // 2^124 * 4 * B + + .quad 0x535fd60613037524 + .quad 0xe210adf6b0fbc26a + .quad 0xac8d0a9b23e990ae + .quad 0x47204d08d72fdbf9 + .quad 0x07242eb30d4b497f + .quad 0x1ef96306b9bccc87 + .quad 0x37950934d8116f45 + .quad 0x05468d6201405b04 + .quad 0x00f565a9f93267de + .quad 0xcecfd78dc0d58e8a + .quad 0xa215e2dcf318e28e + .quad 0x4599ee919b633352 + + // 2^124 * 5 * B + + .quad 0xd3c220ca70e0e76b + .quad 0xb12bea58ea9f3094 + .quad 0x294ddec8c3271282 + .quad 0x0c3539e1a1d1d028 + .quad 0xac746d6b861ae579 + .quad 0x31ab0650f6aea9dc + .quad 0x241d661140256d4c + .quad 0x2f485e853d21a5de + .quad 0x329744839c0833f3 + .quad 0x6fe6257fd2abc484 + .quad 0x5327d1814b358817 + .quad 0x65712585893fe9bc + + // 2^124 * 6 * B + + .quad 0x9c102fb732a61161 + .quad 0xe48e10dd34d520a8 + .quad 0x365c63546f9a9176 + .quad 0x32f6fe4c046f6006 + .quad 0x81c29f1bd708ee3f + .quad 0xddcb5a05ae6407d0 + .quad 0x97aec1d7d2a3eba7 + .quad 0x1590521a91d50831 + .quad 0x40a3a11ec7910acc + .quad 0x9013dff8f16d27ae + .quad 0x1a9720d8abb195d4 + .quad 0x1bb9fe452ea98463 + + // 2^124 * 7 * B + + .quad 0xe9d1d950b3d54f9e + .quad 0x2d5f9cbee00d33c1 + .quad 0x51c2c656a04fc6ac + .quad 0x65c091ee3c1cbcc9 + .quad 0xcf5e6c95cc36747c + .quad 0x294201536b0bc30d + .quad 0x453ac67cee797af0 + .quad 0x5eae6ab32a8bb3c9 + .quad 0x7083661114f118ea + .quad 0x2b37b87b94349cad + .quad 0x7273f51cb4e99f40 + .quad 0x78a2a95823d75698 + + // 2^124 * 8 * B + + .quad 0xa2b072e95c8c2ace + .quad 0x69cffc96651e9c4b + .quad 0x44328ef842e7b42b + .quad 0x5dd996c122aadeb3 + .quad 0xb4f23c425ef83207 + .quad 0xabf894d3c9a934b5 + .quad 0xd0708c1339fd87f7 + .quad 0x1876789117166130 + .quad 0x925b5ef0670c507c + .quad 0x819bc842b93c33bf + .quad 0x10792e9a70dd003f + .quad 0x59ad4b7a6e28dc74 + + // 2^128 * 1 * B + + .quad 0x5f3a7562eb3dbe47 + .quad 0xf7ea38548ebda0b8 + .quad 0x00c3e53145747299 + .quad 0x1304e9e71627d551 + .quad 0x583b04bfacad8ea2 + .quad 0x29b743e8148be884 + .quad 0x2b1e583b0810c5db + .quad 0x2b5449e58eb3bbaa + .quad 0x789814d26adc9cfe + .quad 0x3c1bab3f8b48dd0b + .quad 0xda0fe1fff979c60a + .quad 0x4468de2d7c2dd693 + + // 2^128 * 2 * B + + .quad 0x51bb355e9419469e + .quad 0x33e6dc4c23ddc754 + .quad 0x93a5b6d6447f9962 + .quad 0x6cce7c6ffb44bd63 + .quad 0x4b9ad8c6f86307ce + .quad 0x21113531435d0c28 + .quad 0xd4a866c5657a772c + .quad 0x5da6427e63247352 + .quad 0x1a94c688deac22ca + .quad 0xb9066ef7bbae1ff8 + .quad 0x88ad8c388d59580f + .quad 0x58f29abfe79f2ca8 + + // 2^128 * 3 * B + + .quad 0xe90ecfab8de73e68 + .quad 0x54036f9f377e76a5 + .quad 0xf0495b0bbe015982 + .quad 0x577629c4a7f41e36 + .quad 0x4b5a64bf710ecdf6 + .quad 0xb14ce538462c293c + .quad 0x3643d056d50b3ab9 + .quad 0x6af93724185b4870 + .quad 0x3220024509c6a888 + .quad 0xd2e036134b558973 + .quad 0x83e236233c33289f + .quad 0x701f25bb0caec18f + + // 2^128 * 4 * B + + .quad 0xc3a8b0f8e4616ced + .quad 0xf700660e9e25a87d + .quad 0x61e3061ff4bca59c + .quad 0x2e0c92bfbdc40be9 + .quad 0x9d18f6d97cbec113 + .quad 0x844a06e674bfdbe4 + .quad 0x20f5b522ac4e60d6 + .quad 0x720a5bc050955e51 + .quad 0x0c3f09439b805a35 + .quad 0xe84e8b376242abfc + .quad 0x691417f35c229346 + .quad 0x0e9b9cbb144ef0ec + + // 2^128 * 5 * B + + .quad 0xfbbad48ffb5720ad + .quad 0xee81916bdbf90d0e + .quad 0xd4813152635543bf + .quad 0x221104eb3f337bd8 + .quad 0x8dee9bd55db1beee + .quad 0xc9c3ab370a723fb9 + .quad 0x44a8f1bf1c68d791 + .quad 0x366d44191cfd3cde + .quad 0x9e3c1743f2bc8c14 + .quad 0x2eda26fcb5856c3b + .quad 0xccb82f0e68a7fb97 + .quad 0x4167a4e6bc593244 + + // 2^128 * 6 * B + + .quad 0x643b9d2876f62700 + .quad 0x5d1d9d400e7668eb + .quad 0x1b4b430321fc0684 + .quad 0x7938bb7e2255246a + .quad 0xc2be2665f8ce8fee + .quad 0xe967ff14e880d62c + .quad 0xf12e6e7e2f364eee + .quad 0x34b33370cb7ed2f6 + .quad 0xcdc591ee8681d6cc + .quad 0xce02109ced85a753 + .quad 0xed7485c158808883 + .quad 0x1176fc6e2dfe65e4 + + // 2^128 * 7 * B + + .quad 0xb4af6cd05b9c619b + .quad 0x2ddfc9f4b2a58480 + .quad 0x3d4fa502ebe94dc4 + .quad 0x08fc3a4c677d5f34 + .quad 0xdb90e28949770eb8 + .quad 0x98fbcc2aacf440a3 + .quad 0x21354ffeded7879b + .quad 0x1f6a3e54f26906b6 + .quad 0x60a4c199d30734ea + .quad 0x40c085b631165cd6 + .quad 0xe2333e23f7598295 + .quad 0x4f2fad0116b900d1 + + // 2^128 * 8 * B + + .quad 0x44beb24194ae4e54 + .quad 0x5f541c511857ef6c + .quad 0xa61e6b2d368d0498 + .quad 0x445484a4972ef7ab + .quad 0x962cd91db73bb638 + .quad 0xe60577aafc129c08 + .quad 0x6f619b39f3b61689 + .quad 0x3451995f2944ee81 + .quad 0x9152fcd09fea7d7c + .quad 0x4a816c94b0935cf6 + .quad 0x258e9aaa47285c40 + .quad 0x10b89ca6042893b7 + + // 2^132 * 1 * B + + .quad 0x9b2a426e3b646025 + .quad 0x32127190385ce4cf + .quad 0xa25cffc2dd6dea45 + .quad 0x06409010bea8de75 + .quad 0xd67cded679d34aa0 + .quad 0xcc0b9ec0cc4db39f + .quad 0xa535a456e35d190f + .quad 0x2e05d9eaf61f6fef + .quad 0xc447901ad61beb59 + .quad 0x661f19bce5dc880a + .quad 0x24685482b7ca6827 + .quad 0x293c778cefe07f26 + + // 2^132 * 2 * B + + .quad 0x86809e7007069096 + .quad 0xaad75b15e4e50189 + .quad 0x07f35715a21a0147 + .quad 0x0487f3f112815d5e + .quad 0x16c795d6a11ff200 + .quad 0xcb70d0e2b15815c9 + .quad 0x89f293209b5395b5 + .quad 0x50b8c2d031e47b4f + .quad 0x48350c08068a4962 + .quad 0x6ffdd05351092c9a + .quad 0x17af4f4aaf6fc8dd + .quad 0x4b0553b53cdba58b + + // 2^132 * 3 * B + + .quad 0x9c65fcbe1b32ff79 + .quad 0xeb75ea9f03b50f9b + .quad 0xfced2a6c6c07e606 + .quad 0x35106cd551717908 + .quad 0xbf05211b27c152d4 + .quad 0x5ec26849bd1af639 + .quad 0x5e0b2caa8e6fab98 + .quad 0x054c8bdd50bd0840 + .quad 0x38a0b12f1dcf073d + .quad 0x4b60a8a3b7f6a276 + .quad 0xfed5ac25d3404f9a + .quad 0x72e82d5e5505c229 + + // 2^132 * 4 * B + + .quad 0x6b0b697ff0d844c8 + .quad 0xbb12f85cd979cb49 + .quad 0xd2a541c6c1da0f1f + .quad 0x7b7c242958ce7211 + .quad 0x00d9cdfd69771d02 + .quad 0x410276cd6cfbf17e + .quad 0x4c45306c1cb12ec7 + .quad 0x2857bf1627500861 + .quad 0x9f21903f0101689e + .quad 0xd779dfd3bf861005 + .quad 0xa122ee5f3deb0f1b + .quad 0x510df84b485a00d4 + + // 2^132 * 5 * B + + .quad 0xa54133bb9277a1fa + .quad 0x74ec3b6263991237 + .quad 0x1a3c54dc35d2f15a + .quad 0x2d347144e482ba3a + .quad 0x24b3c887c70ac15e + .quad 0xb0f3a557fb81b732 + .quad 0x9b2cde2fe578cc1b + .quad 0x4cf7ed0703b54f8e + .quad 0x6bd47c6598fbee0f + .quad 0x9e4733e2ab55be2d + .quad 0x1093f624127610c5 + .quad 0x4e05e26ad0a1eaa4 + + // 2^132 * 6 * B + + .quad 0xda9b6b624b531f20 + .quad 0x429a760e77509abb + .quad 0xdbe9f522e823cb80 + .quad 0x618f1856880c8f82 + .quad 0x1833c773e18fe6c0 + .quad 0xe3c4711ad3c87265 + .quad 0x3bfd3c4f0116b283 + .quad 0x1955875eb4cd4db8 + .quad 0x6da6de8f0e399799 + .quad 0x7ad61aa440fda178 + .quad 0xb32cd8105e3563dd + .quad 0x15f6beae2ae340ae + + // 2^132 * 7 * B + + .quad 0x862bcb0c31ec3a62 + .quad 0x810e2b451138f3c2 + .quad 0x788ec4b839dac2a4 + .quad 0x28f76867ae2a9281 + .quad 0xba9a0f7b9245e215 + .quad 0xf368612dd98c0dbb + .quad 0x2e84e4cbf220b020 + .quad 0x6ba92fe962d90eda + .quad 0x3e4df9655884e2aa + .quad 0xbd62fbdbdbd465a5 + .quad 0xd7596caa0de9e524 + .quad 0x6e8042ccb2b1b3d7 + + // 2^132 * 8 * B + + .quad 0xf10d3c29ce28ca6e + .quad 0xbad34540fcb6093d + .quad 0xe7426ed7a2ea2d3f + .quad 0x08af9d4e4ff298b9 + .quad 0x1530653616521f7e + .quad 0x660d06b896203dba + .quad 0x2d3989bc545f0879 + .quad 0x4b5303af78ebd7b0 + .quad 0x72f8a6c3bebcbde8 + .quad 0x4f0fca4adc3a8e89 + .quad 0x6fa9d4e8c7bfdf7a + .quad 0x0dcf2d679b624eb7 + + // 2^136 * 1 * B + + .quad 0x3d5947499718289c + .quad 0x12ebf8c524533f26 + .quad 0x0262bfcb14c3ef15 + .quad 0x20b878d577b7518e + .quad 0x753941be5a45f06e + .quad 0xd07caeed6d9c5f65 + .quad 0x11776b9c72ff51b6 + .quad 0x17d2d1d9ef0d4da9 + .quad 0x27f2af18073f3e6a + .quad 0xfd3fe519d7521069 + .quad 0x22e3b72c3ca60022 + .quad 0x72214f63cc65c6a7 + + // 2^136 * 2 * B + + .quad 0xb4e37f405307a693 + .quad 0xaba714d72f336795 + .quad 0xd6fbd0a773761099 + .quad 0x5fdf48c58171cbc9 + .quad 0x1d9db7b9f43b29c9 + .quad 0xd605824a4f518f75 + .quad 0xf2c072bd312f9dc4 + .quad 0x1f24ac855a1545b0 + .quad 0x24d608328e9505aa + .quad 0x4748c1d10c1420ee + .quad 0xc7ffe45c06fb25a2 + .quad 0x00ba739e2ae395e6 + + // 2^136 * 3 * B + + .quad 0x592e98de5c8790d6 + .quad 0xe5bfb7d345c2a2df + .quad 0x115a3b60f9b49922 + .quad 0x03283a3e67ad78f3 + .quad 0xae4426f5ea88bb26 + .quad 0x360679d984973bfb + .quad 0x5c9f030c26694e50 + .quad 0x72297de7d518d226 + .quad 0x48241dc7be0cb939 + .quad 0x32f19b4d8b633080 + .quad 0xd3dfc90d02289308 + .quad 0x05e1296846271945 + + // 2^136 * 4 * B + + .quad 0xba82eeb32d9c495a + .quad 0xceefc8fcf12bb97c + .quad 0xb02dabae93b5d1e0 + .quad 0x39c00c9c13698d9b + .quad 0xadbfbbc8242c4550 + .quad 0xbcc80cecd03081d9 + .quad 0x843566a6f5c8df92 + .quad 0x78cf25d38258ce4c + .quad 0x15ae6b8e31489d68 + .quad 0xaa851cab9c2bf087 + .quad 0xc9a75a97f04efa05 + .quad 0x006b52076b3ff832 + + // 2^136 * 5 * B + + .quad 0x29e0cfe19d95781c + .quad 0xb681df18966310e2 + .quad 0x57df39d370516b39 + .quad 0x4d57e3443bc76122 + .quad 0xf5cb7e16b9ce082d + .quad 0x3407f14c417abc29 + .quad 0xd4b36bce2bf4a7ab + .quad 0x7de2e9561a9f75ce + .quad 0xde70d4f4b6a55ecb + .quad 0x4801527f5d85db99 + .quad 0xdbc9c440d3ee9a81 + .quad 0x6b2a90af1a6029ed + + // 2^136 * 6 * B + + .quad 0x6923f4fc9ae61e97 + .quad 0x5735281de03f5fd1 + .quad 0xa764ae43e6edd12d + .quad 0x5fd8f4e9d12d3e4a + .quad 0x77ebf3245bb2d80a + .quad 0xd8301b472fb9079b + .quad 0xc647e6f24cee7333 + .quad 0x465812c8276c2109 + .quad 0x4d43beb22a1062d9 + .quad 0x7065fb753831dc16 + .quad 0x180d4a7bde2968d7 + .quad 0x05b32c2b1cb16790 + + // 2^136 * 7 * B + + .quad 0xc8c05eccd24da8fd + .quad 0xa1cf1aac05dfef83 + .quad 0xdbbeeff27df9cd61 + .quad 0x3b5556a37b471e99 + .quad 0xf7fca42c7ad58195 + .quad 0x3214286e4333f3cc + .quad 0xb6c29d0d340b979d + .quad 0x31771a48567307e1 + .quad 0x32b0c524e14dd482 + .quad 0xedb351541a2ba4b6 + .quad 0xa3d16048282b5af3 + .quad 0x4fc079d27a7336eb + + // 2^136 * 8 * B + + .quad 0x51c938b089bf2f7f + .quad 0x2497bd6502dfe9a7 + .quad 0xffffc09c7880e453 + .quad 0x124567cecaf98e92 + .quad 0xdc348b440c86c50d + .quad 0x1337cbc9cc94e651 + .quad 0x6422f74d643e3cb9 + .quad 0x241170c2bae3cd08 + .quad 0x3ff9ab860ac473b4 + .quad 0xf0911dee0113e435 + .quad 0x4ae75060ebc6c4af + .quad 0x3f8612966c87000d + + // 2^140 * 1 * B + + .quad 0x0c9c5303f7957be4 + .quad 0xa3c31a20e085c145 + .quad 0xb0721d71d0850050 + .quad 0x0aba390eab0bf2da + .quad 0x529fdffe638c7bf3 + .quad 0xdf2b9e60388b4995 + .quad 0xe027b34f1bad0249 + .quad 0x7bc92fc9b9fa74ed + .quad 0x9f97ef2e801ad9f9 + .quad 0x83697d5479afda3a + .quad 0xe906b3ffbd596b50 + .quad 0x02672b37dd3fb8e0 + + // 2^140 * 2 * B + + .quad 0x48b2ca8b260885e4 + .quad 0xa4286bec82b34c1c + .quad 0x937e1a2617f58f74 + .quad 0x741d1fcbab2ca2a5 + .quad 0xee9ba729398ca7f5 + .quad 0xeb9ca6257a4849db + .quad 0x29eb29ce7ec544e1 + .quad 0x232ca21ef736e2c8 + .quad 0xbf61423d253fcb17 + .quad 0x08803ceafa39eb14 + .quad 0xf18602df9851c7af + .quad 0x0400f3a049e3414b + + // 2^140 * 3 * B + + .quad 0xabce0476ba61c55b + .quad 0x36a3d6d7c4d39716 + .quad 0x6eb259d5e8d82d09 + .quad 0x0c9176e984d756fb + .quad 0x2efba412a06e7b06 + .quad 0x146785452c8d2560 + .quad 0xdf9713ebd67a91c7 + .quad 0x32830ac7157eadf3 + .quad 0x0e782a7ab73769e8 + .quad 0x04a05d7875b18e2c + .quad 0x29525226ebcceae1 + .quad 0x0d794f8383eba820 + + // 2^140 * 4 * B + + .quad 0xff35f5cb9e1516f4 + .quad 0xee805bcf648aae45 + .quad 0xf0d73c2bb93a9ef3 + .quad 0x097b0bf22092a6c2 + .quad 0x7be44ce7a7a2e1ac + .quad 0x411fd93efad1b8b7 + .quad 0x1734a1d70d5f7c9b + .quad 0x0d6592233127db16 + .quad 0xc48bab1521a9d733 + .quad 0xa6c2eaead61abb25 + .quad 0x625c6c1cc6cb4305 + .quad 0x7fc90fea93eb3a67 + + // 2^140 * 5 * B + + .quad 0x0408f1fe1f5c5926 + .quad 0x1a8f2f5e3b258bf4 + .quad 0x40a951a2fdc71669 + .quad 0x6598ee93c98b577e + .quad 0xc527deb59c7cb23d + .quad 0x955391695328404e + .quad 0xd64392817ccf2c7a + .quad 0x6ce97dabf7d8fa11 + .quad 0x25b5a8e50ef7c48f + .quad 0xeb6034116f2ce532 + .quad 0xc5e75173e53de537 + .quad 0x73119fa08c12bb03 + + // 2^140 * 6 * B + + .quad 0xed30129453f1a4cb + .quad 0xbce621c9c8f53787 + .quad 0xfacb2b1338bee7b9 + .quad 0x3025798a9ea8428c + .quad 0x7845b94d21f4774d + .quad 0xbf62f16c7897b727 + .quad 0x671857c03c56522b + .quad 0x3cd6a85295621212 + .quad 0x3fecde923aeca999 + .quad 0xbdaa5b0062e8c12f + .quad 0x67b99dfc96988ade + .quad 0x3f52c02852661036 + + // 2^140 * 7 * B + + .quad 0xffeaa48e2a1351c6 + .quad 0x28624754fa7f53d7 + .quad 0x0b5ba9e57582ddf1 + .quad 0x60c0104ba696ac59 + .quad 0x9258bf99eec416c6 + .quad 0xac8a5017a9d2f671 + .quad 0x629549ab16dea4ab + .quad 0x05d0e85c99091569 + .quad 0x051de020de9cbe97 + .quad 0xfa07fc56b50bcf74 + .quad 0x378cec9f0f11df65 + .quad 0x36853c69ab96de4d + + // 2^140 * 8 * B + + .quad 0x36d9b8de78f39b2d + .quad 0x7f42ed71a847b9ec + .quad 0x241cd1d679bd3fde + .quad 0x6a704fec92fbce6b + .quad 0x4433c0b0fac5e7be + .quad 0x724bae854c08dcbe + .quad 0xf1f24cc446978f9b + .quad 0x4a0aff6d62825fc8 + .quad 0xe917fb9e61095301 + .quad 0xc102df9402a092f8 + .quad 0xbf09e2f5fa66190b + .quad 0x681109bee0dcfe37 + + // 2^144 * 1 * B + + .quad 0x559a0cc9782a0dde + .quad 0x551dcdb2ea718385 + .quad 0x7f62865b31ef238c + .quad 0x504aa7767973613d + .quad 0x9c18fcfa36048d13 + .quad 0x29159db373899ddd + .quad 0xdc9f350b9f92d0aa + .quad 0x26f57eee878a19d4 + .quad 0x0cab2cd55687efb1 + .quad 0x5180d162247af17b + .quad 0x85c15a344f5a2467 + .quad 0x4041943d9dba3069 + + // 2^144 * 2 * B + + .quad 0xc3c0eeba43ebcc96 + .quad 0x8d749c9c26ea9caf + .quad 0xd9fa95ee1c77ccc6 + .quad 0x1420a1d97684340f + .quad 0x4b217743a26caadd + .quad 0x47a6b424648ab7ce + .quad 0xcb1d4f7a03fbc9e3 + .quad 0x12d931429800d019 + .quad 0x00c67799d337594f + .quad 0x5e3c5140b23aa47b + .quad 0x44182854e35ff395 + .quad 0x1b4f92314359a012 + + // 2^144 * 3 * B + + .quad 0x3e5c109d89150951 + .quad 0x39cefa912de9696a + .quad 0x20eae43f975f3020 + .quad 0x239b572a7f132dae + .quad 0x33cf3030a49866b1 + .quad 0x251f73d2215f4859 + .quad 0xab82aa4051def4f6 + .quad 0x5ff191d56f9a23f6 + .quad 0x819ed433ac2d9068 + .quad 0x2883ab795fc98523 + .quad 0xef4572805593eb3d + .quad 0x020c526a758f36cb + + // 2^144 * 4 * B + + .quad 0x779834f89ed8dbbc + .quad 0xc8f2aaf9dc7ca46c + .quad 0xa9524cdca3e1b074 + .quad 0x02aacc4615313877 + .quad 0xe931ef59f042cc89 + .quad 0x2c589c9d8e124bb6 + .quad 0xadc8e18aaec75997 + .quad 0x452cfe0a5602c50c + .quad 0x86a0f7a0647877df + .quad 0xbbc464270e607c9f + .quad 0xab17ea25f1fb11c9 + .quad 0x4cfb7d7b304b877b + + // 2^144 * 5 * B + + .quad 0x72b43d6cb89b75fe + .quad 0x54c694d99c6adc80 + .quad 0xb8c3aa373ee34c9f + .quad 0x14b4622b39075364 + .quad 0xe28699c29789ef12 + .quad 0x2b6ecd71df57190d + .quad 0xc343c857ecc970d0 + .quad 0x5b1d4cbc434d3ac5 + .quad 0xb6fb2615cc0a9f26 + .quad 0x3a4f0e2bb88dcce5 + .quad 0x1301498b3369a705 + .quad 0x2f98f71258592dd1 + + // 2^144 * 6 * B + + .quad 0x0c94a74cb50f9e56 + .quad 0x5b1ff4a98e8e1320 + .quad 0x9a2acc2182300f67 + .quad 0x3a6ae249d806aaf9 + .quad 0x2e12ae444f54a701 + .quad 0xfcfe3ef0a9cbd7de + .quad 0xcebf890d75835de0 + .quad 0x1d8062e9e7614554 + .quad 0x657ada85a9907c5a + .quad 0x1a0ea8b591b90f62 + .quad 0x8d0e1dfbdf34b4e9 + .quad 0x298b8ce8aef25ff3 + + // 2^144 * 7 * B + + .quad 0x2a927953eff70cb2 + .quad 0x4b89c92a79157076 + .quad 0x9418457a30a7cf6a + .quad 0x34b8a8404d5ce485 + .quad 0x837a72ea0a2165de + .quad 0x3fab07b40bcf79f6 + .quad 0x521636c77738ae70 + .quad 0x6ba6271803a7d7dc + .quad 0xc26eecb583693335 + .quad 0xd5a813df63b5fefd + .quad 0xa293aa9aa4b22573 + .quad 0x71d62bdd465e1c6a + + // 2^144 * 8 * B + + .quad 0x6533cc28d378df80 + .quad 0xf6db43790a0fa4b4 + .quad 0xe3645ff9f701da5a + .quad 0x74d5f317f3172ba4 + .quad 0xcd2db5dab1f75ef5 + .quad 0xd77f95cf16b065f5 + .quad 0x14571fea3f49f085 + .quad 0x1c333621262b2b3d + .quad 0xa86fe55467d9ca81 + .quad 0x398b7c752b298c37 + .quad 0xda6d0892e3ac623b + .quad 0x4aebcc4547e9d98c + + // 2^148 * 1 * B + + .quad 0x53175a7205d21a77 + .quad 0xb0c04422d3b934d4 + .quad 0xadd9f24bdd5deadc + .quad 0x074f46e69f10ff8c + .quad 0x0de9b204a059a445 + .quad 0xe15cb4aa4b17ad0f + .quad 0xe1bbec521f79c557 + .quad 0x2633f1b9d071081b + .quad 0xc1fb4177018b9910 + .quad 0xa6ea20dc6c0fe140 + .quad 0xd661f3e74354c6ff + .quad 0x5ecb72e6f1a3407a + + // 2^148 * 2 * B + + .quad 0xa515a31b2259fb4e + .quad 0x0960f3972bcac52f + .quad 0xedb52fec8d3454cb + .quad 0x382e2720c476c019 + .quad 0xfeeae106e8e86997 + .quad 0x9863337f98d09383 + .quad 0x9470480eaa06ebef + .quad 0x038b6898d4c5c2d0 + .quad 0xf391c51d8ace50a6 + .quad 0x3142d0b9ae2d2948 + .quad 0xdb4d5a1a7f24ca80 + .quad 0x21aeba8b59250ea8 + + // 2^148 * 3 * B + + .quad 0x24f13b34cf405530 + .quad 0x3c44ea4a43088af7 + .quad 0x5dd5c5170006a482 + .quad 0x118eb8f8890b086d + .quad 0x53853600f0087f23 + .quad 0x4c461879da7d5784 + .quad 0x6af303deb41f6860 + .quad 0x0a3c16c5c27c18ed + .quad 0x17e49c17cc947f3d + .quad 0xccc6eda6aac1d27b + .quad 0xdf6092ceb0f08e56 + .quad 0x4909b3e22c67c36b + + // 2^148 * 4 * B + + .quad 0x9c9c85ea63fe2e89 + .quad 0xbe1baf910e9412ec + .quad 0x8f7baa8a86fbfe7b + .quad 0x0fb17f9fef968b6c + .quad 0x59a16676706ff64e + .quad 0x10b953dd0d86a53d + .quad 0x5848e1e6ce5c0b96 + .quad 0x2d8b78e712780c68 + .quad 0x79d5c62eafc3902b + .quad 0x773a215289e80728 + .quad 0xc38ae640e10120b9 + .quad 0x09ae23717b2b1a6d + + // 2^148 * 5 * B + + .quad 0xbb6a192a4e4d083c + .quad 0x34ace0630029e192 + .quad 0x98245a59aafabaeb + .quad 0x6d9c8a9ada97faac + .quad 0x10ab8fa1ad32b1d0 + .quad 0xe9aced1be2778b24 + .quad 0xa8856bc0373de90f + .quad 0x66f35ddddda53996 + .quad 0xd27d9afb24997323 + .quad 0x1bb7e07ef6f01d2e + .quad 0x2ba7472df52ecc7f + .quad 0x03019b4f646f9dc8 + + // 2^148 * 6 * B + + .quad 0x04a186b5565345cd + .quad 0xeee76610bcc4116a + .quad 0x689c73b478fb2a45 + .quad 0x387dcbff65697512 + .quad 0xaf09b214e6b3dc6b + .quad 0x3f7573b5ad7d2f65 + .quad 0xd019d988100a23b0 + .quad 0x392b63a58b5c35f7 + .quad 0x4093addc9c07c205 + .quad 0xc565be15f532c37e + .quad 0x63dbecfd1583402a + .quad 0x61722b4aef2e032e + + // 2^148 * 7 * B + + .quad 0x0012aafeecbd47af + .quad 0x55a266fb1cd46309 + .quad 0xf203eb680967c72c + .quad 0x39633944ca3c1429 + .quad 0xd6b07a5581cb0e3c + .quad 0x290ff006d9444969 + .quad 0x08680b6a16dcda1f + .quad 0x5568d2b75a06de59 + .quad 0x8d0cb88c1b37cfe1 + .quad 0x05b6a5a3053818f3 + .quad 0xf2e9bc04b787d959 + .quad 0x6beba1249add7f64 + + // 2^148 * 8 * B + + .quad 0x1d06005ca5b1b143 + .quad 0x6d4c6bb87fd1cda2 + .quad 0x6ef5967653fcffe7 + .quad 0x097c29e8c1ce1ea5 + .quad 0x5c3cecb943f5a53b + .quad 0x9cc9a61d06c08df2 + .quad 0xcfba639a85895447 + .quad 0x5a845ae80df09fd5 + .quad 0x4ce97dbe5deb94ca + .quad 0x38d0a4388c709c48 + .quad 0xc43eced4a169d097 + .quad 0x0a1249fff7e587c3 + + // 2^152 * 1 * B + + .quad 0x12f0071b276d01c9 + .quad 0xe7b8bac586c48c70 + .quad 0x5308129b71d6fba9 + .quad 0x5d88fbf95a3db792 + .quad 0x0b408d9e7354b610 + .quad 0x806b32535ba85b6e + .quad 0xdbe63a034a58a207 + .quad 0x173bd9ddc9a1df2c + .quad 0x2b500f1efe5872df + .quad 0x58d6582ed43918c1 + .quad 0xe6ed278ec9673ae0 + .quad 0x06e1cd13b19ea319 + + // 2^152 * 2 * B + + .quad 0x40d0ad516f166f23 + .quad 0x118e32931fab6abe + .quad 0x3fe35e14a04d088e + .quad 0x3080603526e16266 + .quad 0x472baf629e5b0353 + .quad 0x3baa0b90278d0447 + .quad 0x0c785f469643bf27 + .quad 0x7f3a6a1a8d837b13 + .quad 0xf7e644395d3d800b + .quad 0x95a8d555c901edf6 + .quad 0x68cd7830592c6339 + .quad 0x30d0fded2e51307e + + // 2^152 * 3 * B + + .quad 0xe0594d1af21233b3 + .quad 0x1bdbe78ef0cc4d9c + .quad 0x6965187f8f499a77 + .quad 0x0a9214202c099868 + .quad 0x9cb4971e68b84750 + .quad 0xa09572296664bbcf + .quad 0x5c8de72672fa412b + .quad 0x4615084351c589d9 + .quad 0xbc9019c0aeb9a02e + .quad 0x55c7110d16034cae + .quad 0x0e6df501659932ec + .quad 0x3bca0d2895ca5dfe + + // 2^152 * 4 * B + + .quad 0x40f031bc3c5d62a4 + .quad 0x19fc8b3ecff07a60 + .quad 0x98183da2130fb545 + .quad 0x5631deddae8f13cd + .quad 0x9c688eb69ecc01bf + .quad 0xf0bc83ada644896f + .quad 0xca2d955f5f7a9fe2 + .quad 0x4ea8b4038df28241 + .quad 0x2aed460af1cad202 + .quad 0x46305305a48cee83 + .quad 0x9121774549f11a5f + .quad 0x24ce0930542ca463 + + // 2^152 * 5 * B + + .quad 0x1fe890f5fd06c106 + .quad 0xb5c468355d8810f2 + .quad 0x827808fe6e8caf3e + .quad 0x41d4e3c28a06d74b + .quad 0x3fcfa155fdf30b85 + .quad 0xd2f7168e36372ea4 + .quad 0xb2e064de6492f844 + .quad 0x549928a7324f4280 + .quad 0xf26e32a763ee1a2e + .quad 0xae91e4b7d25ffdea + .quad 0xbc3bd33bd17f4d69 + .quad 0x491b66dec0dcff6a + + // 2^152 * 6 * B + + .quad 0x98f5b13dc7ea32a7 + .quad 0xe3d5f8cc7e16db98 + .quad 0xac0abf52cbf8d947 + .quad 0x08f338d0c85ee4ac + .quad 0x75f04a8ed0da64a1 + .quad 0xed222caf67e2284b + .quad 0x8234a3791f7b7ba4 + .quad 0x4cf6b8b0b7018b67 + .quad 0xc383a821991a73bd + .quad 0xab27bc01df320c7a + .quad 0xc13d331b84777063 + .quad 0x530d4a82eb078a99 + + // 2^152 * 7 * B + + .quad 0x004c3630e1f94825 + .quad 0x7e2d78268cab535a + .quad 0xc7482323cc84ff8b + .quad 0x65ea753f101770b9 + .quad 0x6d6973456c9abf9e + .quad 0x257fb2fc4900a880 + .quad 0x2bacf412c8cfb850 + .quad 0x0db3e7e00cbfbd5b + .quad 0x3d66fc3ee2096363 + .quad 0x81d62c7f61b5cb6b + .quad 0x0fbe044213443b1a + .quad 0x02a4ec1921e1a1db + + // 2^152 * 8 * B + + .quad 0x5ce6259a3b24b8a2 + .quad 0xb8577acc45afa0b8 + .quad 0xcccbe6e88ba07037 + .quad 0x3d143c51127809bf + .quad 0xf5c86162f1cf795f + .quad 0x118c861926ee57f2 + .quad 0x172124851c063578 + .quad 0x36d12b5dec067fcf + .quad 0x126d279179154557 + .quad 0xd5e48f5cfc783a0a + .quad 0x36bdb6e8df179bac + .quad 0x2ef517885ba82859 + + // 2^156 * 1 * B + + .quad 0x88bd438cd11e0d4a + .quad 0x30cb610d43ccf308 + .quad 0xe09a0e3791937bcc + .quad 0x4559135b25b1720c + .quad 0x1ea436837c6da1e9 + .quad 0xf9c189af1fb9bdbe + .quad 0x303001fcce5dd155 + .quad 0x28a7c99ebc57be52 + .quad 0xb8fd9399e8d19e9d + .quad 0x908191cb962423ff + .quad 0xb2b948d747c742a3 + .quad 0x37f33226d7fb44c4 + + // 2^156 * 2 * B + + .quad 0x0dae8767b55f6e08 + .quad 0x4a43b3b35b203a02 + .quad 0xe3725a6e80af8c79 + .quad 0x0f7a7fd1705fa7a3 + .quad 0x33912553c821b11d + .quad 0x66ed42c241e301df + .quad 0x066fcc11104222fd + .quad 0x307a3b41c192168f + .quad 0x8eeb5d076eb55ce0 + .quad 0x2fc536bfaa0d925a + .quad 0xbe81830fdcb6c6e8 + .quad 0x556c7045827baf52 + + // 2^156 * 3 * B + + .quad 0x8e2b517302e9d8b7 + .quad 0xe3e52269248714e8 + .quad 0xbd4fbd774ca960b5 + .quad 0x6f4b4199c5ecada9 + .quad 0xb94b90022bf44406 + .quad 0xabd4237eff90b534 + .quad 0x7600a960faf86d3a + .quad 0x2f45abdac2322ee3 + .quad 0x61af4912c8ef8a6a + .quad 0xe58fa4fe43fb6e5e + .quad 0xb5afcc5d6fd427cf + .quad 0x6a5393281e1e11eb + + // 2^156 * 4 * B + + .quad 0xf3da5139a5d1ee89 + .quad 0x8145457cff936988 + .quad 0x3f622fed00e188c4 + .quad 0x0f513815db8b5a3d + .quad 0x0fff04fe149443cf + .quad 0x53cac6d9865cddd7 + .quad 0x31385b03531ed1b7 + .quad 0x5846a27cacd1039d + .quad 0x4ff5cdac1eb08717 + .quad 0x67e8b29590f2e9bc + .quad 0x44093b5e237afa99 + .quad 0x0d414bed8708b8b2 + + // 2^156 * 5 * B + + .quad 0xcfb68265fd0e75f6 + .quad 0xe45b3e28bb90e707 + .quad 0x7242a8de9ff92c7a + .quad 0x685b3201933202dd + .quad 0x81886a92294ac9e8 + .quad 0x23162b45d55547be + .quad 0x94cfbc4403715983 + .quad 0x50eb8fdb134bc401 + .quad 0xc0b73ec6d6b330cd + .quad 0x84e44807132faff1 + .quad 0x732b7352c4a5dee1 + .quad 0x5d7c7cf1aa7cd2d2 + + // 2^156 * 6 * B + + .quad 0xaf3b46bf7a4aafa2 + .quad 0xb78705ec4d40d411 + .quad 0x114f0c6aca7c15e3 + .quad 0x3f364faaa9489d4d + .quad 0x33d1013e9b73a562 + .quad 0x925cef5748ec26e1 + .quad 0xa7fce614dd468058 + .quad 0x78b0fad41e9aa438 + .quad 0xbf56a431ed05b488 + .quad 0xa533e66c9c495c7e + .quad 0xe8652baf87f3651a + .quad 0x0241800059d66c33 + + // 2^156 * 7 * B + + .quad 0xceb077fea37a5be4 + .quad 0xdb642f02e5a5eeb7 + .quad 0xc2e6d0c5471270b8 + .quad 0x4771b65538e4529c + .quad 0x28350c7dcf38ea01 + .quad 0x7c6cdbc0b2917ab6 + .quad 0xace7cfbe857082f7 + .quad 0x4d2845aba2d9a1e0 + .quad 0xbb537fe0447070de + .quad 0xcba744436dd557df + .quad 0xd3b5a3473600dbcb + .quad 0x4aeabbe6f9ffd7f8 + + // 2^156 * 8 * B + + .quad 0x4630119e40d8f78c + .quad 0xa01a9bc53c710e11 + .quad 0x486d2b258910dd79 + .quad 0x1e6c47b3db0324e5 + .quad 0x6a2134bcc4a9c8f2 + .quad 0xfbf8fd1c8ace2e37 + .quad 0x000ae3049911a0ba + .quad 0x046e3a616bc89b9e + .quad 0x14e65442f03906be + .quad 0x4a019d54e362be2a + .quad 0x68ccdfec8dc230c7 + .quad 0x7cfb7e3faf6b861c + + // 2^160 * 1 * B + + .quad 0x4637974e8c58aedc + .quad 0xb9ef22fbabf041a4 + .quad 0xe185d956e980718a + .quad 0x2f1b78fab143a8a6 + .quad 0x96eebffb305b2f51 + .quad 0xd3f938ad889596b8 + .quad 0xf0f52dc746d5dd25 + .quad 0x57968290bb3a0095 + .quad 0xf71ab8430a20e101 + .quad 0xf393658d24f0ec47 + .quad 0xcf7509a86ee2eed1 + .quad 0x7dc43e35dc2aa3e1 + + // 2^160 * 2 * B + + .quad 0x85966665887dd9c3 + .quad 0xc90f9b314bb05355 + .quad 0xc6e08df8ef2079b1 + .quad 0x7ef72016758cc12f + .quad 0x5a782a5c273e9718 + .quad 0x3576c6995e4efd94 + .quad 0x0f2ed8051f237d3e + .quad 0x044fb81d82d50a99 + .quad 0xc1df18c5a907e3d9 + .quad 0x57b3371dce4c6359 + .quad 0xca704534b201bb49 + .quad 0x7f79823f9c30dd2e + + // 2^160 * 3 * B + + .quad 0x8334d239a3b513e8 + .quad 0xc13670d4b91fa8d8 + .quad 0x12b54136f590bd33 + .quad 0x0a4e0373d784d9b4 + .quad 0x6a9c1ff068f587ba + .quad 0x0827894e0050c8de + .quad 0x3cbf99557ded5be7 + .quad 0x64a9b0431c06d6f0 + .quad 0x2eb3d6a15b7d2919 + .quad 0xb0b4f6a0d53a8235 + .quad 0x7156ce4389a45d47 + .quad 0x071a7d0ace18346c + + // 2^160 * 4 * B + + .quad 0xd3072daac887ba0b + .quad 0x01262905bfa562ee + .quad 0xcf543002c0ef768b + .quad 0x2c3bcc7146ea7e9c + .quad 0xcc0c355220e14431 + .quad 0x0d65950709b15141 + .quad 0x9af5621b209d5f36 + .quad 0x7c69bcf7617755d3 + .quad 0x07f0d7eb04e8295f + .quad 0x10db18252f50f37d + .quad 0xe951a9a3171798d7 + .quad 0x6f5a9a7322aca51d + + // 2^160 * 5 * B + + .quad 0x8ba1000c2f41c6c5 + .quad 0xc49f79c10cfefb9b + .quad 0x4efa47703cc51c9f + .quad 0x494e21a2e147afca + .quad 0xe729d4eba3d944be + .quad 0x8d9e09408078af9e + .quad 0x4525567a47869c03 + .quad 0x02ab9680ee8d3b24 + .quad 0xefa48a85dde50d9a + .quad 0x219a224e0fb9a249 + .quad 0xfa091f1dd91ef6d9 + .quad 0x6b5d76cbea46bb34 + + // 2^160 * 6 * B + + .quad 0x8857556cec0cd994 + .quad 0x6472dc6f5cd01dba + .quad 0xaf0169148f42b477 + .quad 0x0ae333f685277354 + .quad 0xe0f941171e782522 + .quad 0xf1e6ae74036936d3 + .quad 0x408b3ea2d0fcc746 + .quad 0x16fb869c03dd313e + .quad 0x288e199733b60962 + .quad 0x24fc72b4d8abe133 + .quad 0x4811f7ed0991d03e + .quad 0x3f81e38b8f70d075 + + // 2^160 * 7 * B + + .quad 0x7f910fcc7ed9affe + .quad 0x545cb8a12465874b + .quad 0xa8397ed24b0c4704 + .quad 0x50510fc104f50993 + .quad 0x0adb7f355f17c824 + .quad 0x74b923c3d74299a4 + .quad 0xd57c3e8bcbf8eaf7 + .quad 0x0ad3e2d34cdedc3d + .quad 0x6f0c0fc5336e249d + .quad 0x745ede19c331cfd9 + .quad 0xf2d6fd0009eefe1c + .quad 0x127c158bf0fa1ebe + + // 2^160 * 8 * B + + .quad 0xf6197c422e9879a2 + .quad 0xa44addd452ca3647 + .quad 0x9b413fc14b4eaccb + .quad 0x354ef87d07ef4f68 + .quad 0xdea28fc4ae51b974 + .quad 0x1d9973d3744dfe96 + .quad 0x6240680b873848a8 + .quad 0x4ed82479d167df95 + .quad 0xfee3b52260c5d975 + .quad 0x50352efceb41b0b8 + .quad 0x8808ac30a9f6653c + .quad 0x302d92d20539236d + + // 2^164 * 1 * B + + .quad 0x4c59023fcb3efb7c + .quad 0x6c2fcb99c63c2a94 + .quad 0xba4190e2c3c7e084 + .quad 0x0e545daea51874d9 + .quad 0x957b8b8b0df53c30 + .quad 0x2a1c770a8e60f098 + .quad 0xbbc7a670345796de + .quad 0x22a48f9a90c99bc9 + .quad 0x6b7dc0dc8d3fac58 + .quad 0x5497cd6ce6e42bfd + .quad 0x542f7d1bf400d305 + .quad 0x4159f47f048d9136 + + // 2^164 * 2 * B + + .quad 0x20ad660839e31e32 + .quad 0xf81e1bd58405be50 + .quad 0xf8064056f4dabc69 + .quad 0x14d23dd4ce71b975 + .quad 0x748515a8bbd24839 + .quad 0x77128347afb02b55 + .quad 0x50ba2ac649a2a17f + .quad 0x060525513ad730f1 + .quad 0xf2398e098aa27f82 + .quad 0x6d7982bb89a1b024 + .quad 0xfa694084214dd24c + .quad 0x71ab966fa32301c3 + + // 2^164 * 3 * B + + .quad 0x2dcbd8e34ded02fc + .quad 0x1151f3ec596f22aa + .quad 0xbca255434e0328da + .quad 0x35768fbe92411b22 + .quad 0xb1088a0702809955 + .quad 0x43b273ea0b43c391 + .quad 0xca9b67aefe0686ed + .quad 0x605eecbf8335f4ed + .quad 0x83200a656c340431 + .quad 0x9fcd71678ee59c2f + .quad 0x75d4613f71300f8a + .quad 0x7a912faf60f542f9 + + // 2^164 * 4 * B + + .quad 0xb204585e5edc1a43 + .quad 0x9f0e16ee5897c73c + .quad 0x5b82c0ae4e70483c + .quad 0x624a170e2bddf9be + .quad 0x253f4f8dfa2d5597 + .quad 0x25e49c405477130c + .quad 0x00c052e5996b1102 + .quad 0x33cb966e33bb6c4a + .quad 0x597028047f116909 + .quad 0x828ac41c1e564467 + .quad 0x70417dbde6217387 + .quad 0x721627aefbac4384 + + // 2^164 * 5 * B + + .quad 0x97d03bc38736add5 + .quad 0x2f1422afc532b130 + .quad 0x3aa68a057101bbc4 + .quad 0x4c946cf7e74f9fa7 + .quad 0xfd3097bc410b2f22 + .quad 0xf1a05da7b5cfa844 + .quad 0x61289a1def57ca74 + .quad 0x245ea199bb821902 + .quad 0xaedca66978d477f8 + .quad 0x1898ba3c29117fe1 + .quad 0xcf73f983720cbd58 + .quad 0x67da12e6b8b56351 + + // 2^164 * 6 * B + + .quad 0x7067e187b4bd6e07 + .quad 0x6e8f0203c7d1fe74 + .quad 0x93c6aa2f38c85a30 + .quad 0x76297d1f3d75a78a + .quad 0x2b7ef3d38ec8308c + .quad 0x828fd7ec71eb94ab + .quad 0x807c3b36c5062abd + .quad 0x0cb64cb831a94141 + .quad 0x3030fc33534c6378 + .quad 0xb9635c5ce541e861 + .quad 0x15d9a9bed9b2c728 + .quad 0x49233ea3f3775dcb + + // 2^164 * 7 * B + + .quad 0x629398fa8dbffc3a + .quad 0xe12fe52dd54db455 + .quad 0xf3be11dfdaf25295 + .quad 0x628b140dce5e7b51 + .quad 0x7b3985fe1c9f249b + .quad 0x4fd6b2d5a1233293 + .quad 0xceb345941adf4d62 + .quad 0x6987ff6f542de50c + .quad 0x47e241428f83753c + .quad 0x6317bebc866af997 + .quad 0xdabb5b433d1a9829 + .quad 0x074d8d245287fb2d + + // 2^164 * 8 * B + + .quad 0x8337d9cd440bfc31 + .quad 0x729d2ca1af318fd7 + .quad 0xa040a4a4772c2070 + .quad 0x46002ef03a7349be + .quad 0x481875c6c0e31488 + .quad 0x219429b2e22034b4 + .quad 0x7223c98a31283b65 + .quad 0x3420d60b342277f9 + .quad 0xfaa23adeaffe65f7 + .quad 0x78261ed45be0764c + .quad 0x441c0a1e2f164403 + .quad 0x5aea8e567a87d395 + + // 2^168 * 1 * B + + .quad 0x7813c1a2bca4283d + .quad 0xed62f091a1863dd9 + .quad 0xaec7bcb8c268fa86 + .quad 0x10e5d3b76f1cae4c + .quad 0x2dbc6fb6e4e0f177 + .quad 0x04e1bf29a4bd6a93 + .quad 0x5e1966d4787af6e8 + .quad 0x0edc5f5eb426d060 + .quad 0x5453bfd653da8e67 + .quad 0xe9dc1eec24a9f641 + .quad 0xbf87263b03578a23 + .quad 0x45b46c51361cba72 + + // 2^168 * 2 * B + + .quad 0xa9402abf314f7fa1 + .quad 0xe257f1dc8e8cf450 + .quad 0x1dbbd54b23a8be84 + .quad 0x2177bfa36dcb713b + .quad 0xce9d4ddd8a7fe3e4 + .quad 0xab13645676620e30 + .quad 0x4b594f7bb30e9958 + .quad 0x5c1c0aef321229df + .quad 0x37081bbcfa79db8f + .quad 0x6048811ec25f59b3 + .quad 0x087a76659c832487 + .quad 0x4ae619387d8ab5bb + + // 2^168 * 3 * B + + .quad 0x8ddbf6aa5344a32e + .quad 0x7d88eab4b41b4078 + .quad 0x5eb0eb974a130d60 + .quad 0x1a00d91b17bf3e03 + .quad 0x61117e44985bfb83 + .quad 0xfce0462a71963136 + .quad 0x83ac3448d425904b + .quad 0x75685abe5ba43d64 + .quad 0x6e960933eb61f2b2 + .quad 0x543d0fa8c9ff4952 + .quad 0xdf7275107af66569 + .quad 0x135529b623b0e6aa + + // 2^168 * 4 * B + + .quad 0x18f0dbd7add1d518 + .quad 0x979f7888cfc11f11 + .quad 0x8732e1f07114759b + .quad 0x79b5b81a65ca3a01 + .quad 0xf5c716bce22e83fe + .quad 0xb42beb19e80985c1 + .quad 0xec9da63714254aae + .quad 0x5972ea051590a613 + .quad 0x0fd4ac20dc8f7811 + .quad 0x9a9ad294ac4d4fa8 + .quad 0xc01b2d64b3360434 + .quad 0x4f7e9c95905f3bdb + + // 2^168 * 5 * B + + .quad 0x62674bbc5781302e + .quad 0xd8520f3989addc0f + .quad 0x8c2999ae53fbd9c6 + .quad 0x31993ad92e638e4c + .quad 0x71c8443d355299fe + .quad 0x8bcd3b1cdbebead7 + .quad 0x8092499ef1a49466 + .quad 0x1942eec4a144adc8 + .quad 0x7dac5319ae234992 + .quad 0x2c1b3d910cea3e92 + .quad 0x553ce494253c1122 + .quad 0x2a0a65314ef9ca75 + + // 2^168 * 6 * B + + .quad 0x2db7937ff7f927c2 + .quad 0xdb741f0617d0a635 + .quad 0x5982f3a21155af76 + .quad 0x4cf6e218647c2ded + .quad 0xcf361acd3c1c793a + .quad 0x2f9ebcac5a35bc3b + .quad 0x60e860e9a8cda6ab + .quad 0x055dc39b6dea1a13 + .quad 0xb119227cc28d5bb6 + .quad 0x07e24ebc774dffab + .quad 0xa83c78cee4a32c89 + .quad 0x121a307710aa24b6 + + // 2^168 * 7 * B + + .quad 0xe4db5d5e9f034a97 + .quad 0xe153fc093034bc2d + .quad 0x460546919551d3b1 + .quad 0x333fc76c7a40e52d + .quad 0xd659713ec77483c9 + .quad 0x88bfe077b82b96af + .quad 0x289e28231097bcd3 + .quad 0x527bb94a6ced3a9b + .quad 0x563d992a995b482e + .quad 0x3405d07c6e383801 + .quad 0x485035de2f64d8e5 + .quad 0x6b89069b20a7a9f7 + + // 2^168 * 8 * B + + .quad 0x812aa0416270220d + .quad 0x995a89faf9245b4e + .quad 0xffadc4ce5072ef05 + .quad 0x23bc2103aa73eb73 + .quad 0x4082fa8cb5c7db77 + .quad 0x068686f8c734c155 + .quad 0x29e6c8d9f6e7a57e + .quad 0x0473d308a7639bcf + .quad 0xcaee792603589e05 + .quad 0x2b4b421246dcc492 + .quad 0x02a1ef74e601a94f + .quad 0x102f73bfde04341a + + // 2^172 * 1 * B + + .quad 0xb5a2d50c7ec20d3e + .quad 0xc64bdd6ea0c97263 + .quad 0x56e89052c1ff734d + .quad 0x4929c6f72b2ffaba + .quad 0x358ecba293a36247 + .quad 0xaf8f9862b268fd65 + .quad 0x412f7e9968a01c89 + .quad 0x5786f312cd754524 + .quad 0x337788ffca14032c + .quad 0xf3921028447f1ee3 + .quad 0x8b14071f231bccad + .quad 0x4c817b4bf2344783 + + // 2^172 * 2 * B + + .quad 0x0ff853852871b96e + .quad 0xe13e9fab60c3f1bb + .quad 0xeefd595325344402 + .quad 0x0a37c37075b7744b + .quad 0x413ba057a40b4484 + .quad 0xba4c2e1a4f5f6a43 + .quad 0x614ba0a5aee1d61c + .quad 0x78a1531a8b05dc53 + .quad 0x6cbdf1703ad0562b + .quad 0x8ecf4830c92521a3 + .quad 0xdaebd303fd8424e7 + .quad 0x72ad82a42e5ec56f + + // 2^172 * 3 * B + + .quad 0x3f9e8e35bafb65f6 + .quad 0x39d69ec8f27293a1 + .quad 0x6cb8cd958cf6a3d0 + .quad 0x1734778173adae6d + .quad 0xc368939167024bc3 + .quad 0x8e69d16d49502fda + .quad 0xfcf2ec3ce45f4b29 + .quad 0x065f669ea3b4cbc4 + .quad 0x8a00aec75532db4d + .quad 0xb869a4e443e31bb1 + .quad 0x4a0f8552d3a7f515 + .quad 0x19adeb7c303d7c08 + + // 2^172 * 4 * B + + .quad 0xc720cb6153ead9a3 + .quad 0x55b2c97f512b636e + .quad 0xb1e35b5fd40290b1 + .quad 0x2fd9ccf13b530ee2 + .quad 0x9d05ba7d43c31794 + .quad 0x2470c8ff93322526 + .quad 0x8323dec816197438 + .quad 0x2852709881569b53 + .quad 0x07bd475b47f796b8 + .quad 0xd2c7b013542c8f54 + .quad 0x2dbd23f43b24f87e + .quad 0x6551afd77b0901d6 + + // 2^172 * 5 * B + + .quad 0x4546baaf54aac27f + .quad 0xf6f66fecb2a45a28 + .quad 0x582d1b5b562bcfe8 + .quad 0x44b123f3920f785f + .quad 0x68a24ce3a1d5c9ac + .quad 0xbb77a33d10ff6461 + .quad 0x0f86ce4425d3166e + .quad 0x56507c0950b9623b + .quad 0x1206f0b7d1713e63 + .quad 0x353fe3d915bafc74 + .quad 0x194ceb970ad9d94d + .quad 0x62fadd7cf9d03ad3 + + // 2^172 * 6 * B + + .quad 0xc6b5967b5598a074 + .quad 0x5efe91ce8e493e25 + .quad 0xd4b72c4549280888 + .quad 0x20ef1149a26740c2 + .quad 0x3cd7bc61e7ce4594 + .quad 0xcd6b35a9b7dd267e + .quad 0xa080abc84366ef27 + .quad 0x6ec7c46f59c79711 + .quad 0x2f07ad636f09a8a2 + .quad 0x8697e6ce24205e7d + .quad 0xc0aefc05ee35a139 + .quad 0x15e80958b5f9d897 + + // 2^172 * 7 * B + + .quad 0x25a5ef7d0c3e235b + .quad 0x6c39c17fbe134ee7 + .quad 0xc774e1342dc5c327 + .quad 0x021354b892021f39 + .quad 0x4dd1ed355bb061c4 + .quad 0x42dc0cef941c0700 + .quad 0x61305dc1fd86340e + .quad 0x56b2cc930e55a443 + .quad 0x1df79da6a6bfc5a2 + .quad 0x02f3a2749fde4369 + .quad 0xb323d9f2cda390a7 + .quad 0x7be0847b8774d363 + + // 2^172 * 8 * B + + .quad 0x8c99cc5a8b3f55c3 + .quad 0x0611d7253fded2a0 + .quad 0xed2995ff36b70a36 + .quad 0x1f699a54d78a2619 + .quad 0x1466f5af5307fa11 + .quad 0x817fcc7ded6c0af2 + .quad 0x0a6de44ec3a4a3fb + .quad 0x74071475bc927d0b + .quad 0xe77292f373e7ea8a + .quad 0x296537d2cb045a31 + .quad 0x1bd0653ed3274fde + .quad 0x2f9a2c4476bd2966 + + // 2^176 * 1 * B + + .quad 0xeb18b9ab7f5745c6 + .quad 0x023a8aee5787c690 + .quad 0xb72712da2df7afa9 + .quad 0x36597d25ea5c013d + .quad 0xa2b4dae0b5511c9a + .quad 0x7ac860292bffff06 + .quad 0x981f375df5504234 + .quad 0x3f6bd725da4ea12d + .quad 0x734d8d7b106058ac + .quad 0xd940579e6fc6905f + .quad 0x6466f8f99202932d + .quad 0x7b7ecc19da60d6d0 + + // 2^176 * 2 * B + + .quad 0x78c2373c695c690d + .quad 0xdd252e660642906e + .quad 0x951d44444ae12bd2 + .quad 0x4235ad7601743956 + .quad 0x6dae4a51a77cfa9b + .quad 0x82263654e7a38650 + .quad 0x09bbffcd8f2d82db + .quad 0x03bedc661bf5caba + .quad 0x6258cb0d078975f5 + .quad 0x492942549189f298 + .quad 0xa0cab423e2e36ee4 + .quad 0x0e7ce2b0cdf066a1 + + // 2^176 * 3 * B + + .quad 0xc494643ac48c85a3 + .quad 0xfd361df43c6139ad + .quad 0x09db17dd3ae94d48 + .quad 0x666e0a5d8fb4674a + .quad 0xfea6fedfd94b70f9 + .quad 0xf130c051c1fcba2d + .quad 0x4882d47e7f2fab89 + .quad 0x615256138aeceeb5 + .quad 0x2abbf64e4870cb0d + .quad 0xcd65bcf0aa458b6b + .quad 0x9abe4eba75e8985d + .quad 0x7f0bc810d514dee4 + + // 2^176 * 4 * B + + .quad 0xb9006ba426f4136f + .quad 0x8d67369e57e03035 + .quad 0xcbc8dfd94f463c28 + .quad 0x0d1f8dbcf8eedbf5 + .quad 0x83ac9dad737213a0 + .quad 0x9ff6f8ba2ef72e98 + .quad 0x311e2edd43ec6957 + .quad 0x1d3a907ddec5ab75 + .quad 0xba1693313ed081dc + .quad 0x29329fad851b3480 + .quad 0x0128013c030321cb + .quad 0x00011b44a31bfde3 + + // 2^176 * 5 * B + + .quad 0x3fdfa06c3fc66c0c + .quad 0x5d40e38e4dd60dd2 + .quad 0x7ae38b38268e4d71 + .quad 0x3ac48d916e8357e1 + .quad 0x16561f696a0aa75c + .quad 0xc1bf725c5852bd6a + .quad 0x11a8dd7f9a7966ad + .quad 0x63d988a2d2851026 + .quad 0x00120753afbd232e + .quad 0xe92bceb8fdd8f683 + .quad 0xf81669b384e72b91 + .quad 0x33fad52b2368a066 + + // 2^176 * 6 * B + + .quad 0x540649c6c5e41e16 + .quad 0x0af86430333f7735 + .quad 0xb2acfcd2f305e746 + .quad 0x16c0f429a256dca7 + .quad 0x8d2cc8d0c422cfe8 + .quad 0x072b4f7b05a13acb + .quad 0xa3feb6e6ecf6a56f + .quad 0x3cc355ccb90a71e2 + .quad 0xe9b69443903e9131 + .quad 0xb8a494cb7a5637ce + .quad 0xc87cd1a4baba9244 + .quad 0x631eaf426bae7568 + + // 2^176 * 7 * B + + .quad 0xb3e90410da66fe9f + .quad 0x85dd4b526c16e5a6 + .quad 0xbc3d97611ef9bf83 + .quad 0x5599648b1ea919b5 + .quad 0x47d975b9a3700de8 + .quad 0x7280c5fbe2f80552 + .quad 0x53658f2732e45de1 + .quad 0x431f2c7f665f80b5 + .quad 0xd6026344858f7b19 + .quad 0x14ab352fa1ea514a + .quad 0x8900441a2090a9d7 + .quad 0x7b04715f91253b26 + + // 2^176 * 8 * B + + .quad 0x83edbd28acf6ae43 + .quad 0x86357c8b7d5c7ab4 + .quad 0xc0404769b7eb2c44 + .quad 0x59b37bf5c2f6583f + .quad 0xb376c280c4e6bac6 + .quad 0x970ed3dd6d1d9b0b + .quad 0xb09a9558450bf944 + .quad 0x48d0acfa57cde223 + .quad 0xb60f26e47dabe671 + .quad 0xf1d1a197622f3a37 + .quad 0x4208ce7ee9960394 + .quad 0x16234191336d3bdb + + // 2^180 * 1 * B + + .quad 0xf19aeac733a63aef + .quad 0x2c7fba5d4442454e + .quad 0x5da87aa04795e441 + .quad 0x413051e1a4e0b0f5 + .quad 0x852dd1fd3d578bbe + .quad 0x2b65ce72c3286108 + .quad 0x658c07f4eace2273 + .quad 0x0933f804ec38ab40 + .quad 0xa7ab69798d496476 + .quad 0x8121aadefcb5abc8 + .quad 0xa5dc12ef7b539472 + .quad 0x07fd47065e45351a + + // 2^180 * 2 * B + + .quad 0xc8583c3d258d2bcd + .quad 0x17029a4daf60b73f + .quad 0xfa0fc9d6416a3781 + .quad 0x1c1e5fba38b3fb23 + .quad 0x304211559ae8e7c3 + .quad 0xf281b229944882a5 + .quad 0x8a13ac2e378250e4 + .quad 0x014afa0954ba48f4 + .quad 0xcb3197001bb3666c + .quad 0x330060524bffecb9 + .quad 0x293711991a88233c + .quad 0x291884363d4ed364 + + // 2^180 * 3 * B + + .quad 0x033c6805dc4babfa + .quad 0x2c15bf5e5596ecc1 + .quad 0x1bc70624b59b1d3b + .quad 0x3ede9850a19f0ec5 + .quad 0xfb9d37c3bc1ab6eb + .quad 0x02be14534d57a240 + .quad 0xf4d73415f8a5e1f6 + .quad 0x5964f4300ccc8188 + .quad 0xe44a23152d096800 + .quad 0x5c08c55970866996 + .quad 0xdf2db60a46affb6e + .quad 0x579155c1f856fd89 + + // 2^180 * 4 * B + + .quad 0x96324edd12e0c9ef + .quad 0x468b878df2420297 + .quad 0x199a3776a4f573be + .quad 0x1e7fbcf18e91e92a + .quad 0xb5f16b630817e7a6 + .quad 0x808c69233c351026 + .quad 0x324a983b54cef201 + .quad 0x53c092084a485345 + .quad 0xd2d41481f1cbafbf + .quad 0x231d2db6716174e5 + .quad 0x0b7d7656e2a55c98 + .quad 0x3e955cd82aa495f6 + + // 2^180 * 5 * B + + .quad 0xe48f535e3ed15433 + .quad 0xd075692a0d7270a3 + .quad 0x40fbd21daade6387 + .quad 0x14264887cf4495f5 + .quad 0xab39f3ef61bb3a3f + .quad 0x8eb400652eb9193e + .quad 0xb5de6ecc38c11f74 + .quad 0x654d7e9626f3c49f + .quad 0xe564cfdd5c7d2ceb + .quad 0x82eeafded737ccb9 + .quad 0x6107db62d1f9b0ab + .quad 0x0b6baac3b4358dbb + + // 2^180 * 6 * B + + .quad 0x7ae62bcb8622fe98 + .quad 0x47762256ceb891af + .quad 0x1a5a92bcf2e406b4 + .quad 0x7d29401784e41501 + .quad 0x204abad63700a93b + .quad 0xbe0023d3da779373 + .quad 0xd85f0346633ab709 + .quad 0x00496dc490820412 + .quad 0x1c74b88dc27e6360 + .quad 0x074854268d14850c + .quad 0xa145fb7b3e0dcb30 + .quad 0x10843f1b43803b23 + + // 2^180 * 7 * B + + .quad 0xc5f90455376276dd + .quad 0xce59158dd7645cd9 + .quad 0x92f65d511d366b39 + .quad 0x11574b6e526996c4 + .quad 0xd56f672de324689b + .quad 0xd1da8aedb394a981 + .quad 0xdd7b58fe9168cfed + .quad 0x7ce246cd4d56c1e8 + .quad 0xb8f4308e7f80be53 + .quad 0x5f3cb8cb34a9d397 + .quad 0x18a961bd33cc2b2c + .quad 0x710045fb3a9af671 + + // 2^180 * 8 * B + + .quad 0x73f93d36101b95eb + .quad 0xfaef33794f6f4486 + .quad 0x5651735f8f15e562 + .quad 0x7fa3f19058b40da1 + .quad 0xa03fc862059d699e + .quad 0x2370cfa19a619e69 + .quad 0xc4fe3b122f823deb + .quad 0x1d1b056fa7f0844e + .quad 0x1bc64631e56bf61f + .quad 0xd379ab106e5382a3 + .quad 0x4d58c57e0540168d + .quad 0x566256628442d8e4 + + // 2^184 * 1 * B + + .quad 0xb9e499def6267ff6 + .quad 0x7772ca7b742c0843 + .quad 0x23a0153fe9a4f2b1 + .quad 0x2cdfdfecd5d05006 + .quad 0xdd499cd61ff38640 + .quad 0x29cd9bc3063625a0 + .quad 0x51e2d8023dd73dc3 + .quad 0x4a25707a203b9231 + .quad 0x2ab7668a53f6ed6a + .quad 0x304242581dd170a1 + .quad 0x4000144c3ae20161 + .quad 0x5721896d248e49fc + + // 2^184 * 2 * B + + .quad 0x0b6e5517fd181bae + .quad 0x9022629f2bb963b4 + .quad 0x5509bce932064625 + .quad 0x578edd74f63c13da + .quad 0x285d5091a1d0da4e + .quad 0x4baa6fa7b5fe3e08 + .quad 0x63e5177ce19393b3 + .quad 0x03c935afc4b030fd + .quad 0x997276c6492b0c3d + .quad 0x47ccc2c4dfe205fc + .quad 0xdcd29b84dd623a3c + .quad 0x3ec2ab590288c7a2 + + // 2^184 * 3 * B + + .quad 0xa1a0d27be4d87bb9 + .quad 0xa98b4deb61391aed + .quad 0x99a0ddd073cb9b83 + .quad 0x2dd5c25a200fcace + .quad 0xa7213a09ae32d1cb + .quad 0x0f2b87df40f5c2d5 + .quad 0x0baea4c6e81eab29 + .quad 0x0e1bf66c6adbac5e + .quad 0xe2abd5e9792c887e + .quad 0x1a020018cb926d5d + .quad 0xbfba69cdbaae5f1e + .quad 0x730548b35ae88f5f + + // 2^184 * 4 * B + + .quad 0xc43551a3cba8b8ee + .quad 0x65a26f1db2115f16 + .quad 0x760f4f52ab8c3850 + .quad 0x3043443b411db8ca + .quad 0x805b094ba1d6e334 + .quad 0xbf3ef17709353f19 + .quad 0x423f06cb0622702b + .quad 0x585a2277d87845dd + .quad 0xa18a5f8233d48962 + .quad 0x6698c4b5ec78257f + .quad 0xa78e6fa5373e41ff + .quad 0x7656278950ef981f + + // 2^184 * 5 * B + + .quad 0x38c3cf59d51fc8c0 + .quad 0x9bedd2fd0506b6f2 + .quad 0x26bf109fab570e8f + .quad 0x3f4160a8c1b846a6 + .quad 0xe17073a3ea86cf9d + .quad 0x3a8cfbb707155fdc + .quad 0x4853e7fc31838a8e + .quad 0x28bbf484b613f616 + .quad 0xf2612f5c6f136c7c + .quad 0xafead107f6dd11be + .quad 0x527e9ad213de6f33 + .quad 0x1e79cb358188f75d + + // 2^184 * 6 * B + + .quad 0x013436c3eef7e3f1 + .quad 0x828b6a7ffe9e10f8 + .quad 0x7ff908e5bcf9defc + .quad 0x65d7951b3a3b3831 + .quad 0x77e953d8f5e08181 + .quad 0x84a50c44299dded9 + .quad 0xdc6c2d0c864525e5 + .quad 0x478ab52d39d1f2f4 + .quad 0x66a6a4d39252d159 + .quad 0xe5dde1bc871ac807 + .quad 0xb82c6b40a6c1c96f + .quad 0x16d87a411a212214 + + // 2^184 * 7 * B + + .quad 0xb3bd7e5a42066215 + .quad 0x879be3cd0c5a24c1 + .quad 0x57c05db1d6f994b7 + .quad 0x28f87c8165f38ca6 + .quad 0xfba4d5e2d54e0583 + .quad 0xe21fafd72ebd99fa + .quad 0x497ac2736ee9778f + .quad 0x1f990b577a5a6dde + .quad 0xa3344ead1be8f7d6 + .quad 0x7d1e50ebacea798f + .quad 0x77c6569e520de052 + .quad 0x45882fe1534d6d3e + + // 2^184 * 8 * B + + .quad 0x6669345d757983d6 + .quad 0x62b6ed1117aa11a6 + .quad 0x7ddd1857985e128f + .quad 0x688fe5b8f626f6dd + .quad 0xd8ac9929943c6fe4 + .quad 0xb5f9f161a38392a2 + .quad 0x2699db13bec89af3 + .quad 0x7dcf843ce405f074 + .quad 0x6c90d6484a4732c0 + .quad 0xd52143fdca563299 + .quad 0xb3be28c3915dc6e1 + .quad 0x6739687e7327191b + + // 2^188 * 1 * B + + .quad 0x9f65c5ea200814cf + .quad 0x840536e169a31740 + .quad 0x8b0ed13925c8b4ad + .quad 0x0080dbafe936361d + .quad 0x8ce5aad0c9cb971f + .quad 0x1156aaa99fd54a29 + .quad 0x41f7247015af9b78 + .quad 0x1fe8cca8420f49aa + .quad 0x72a1848f3c0cc82a + .quad 0x38c560c2877c9e54 + .quad 0x5004e228ce554140 + .quad 0x042418a103429d71 + + // 2^188 * 2 * B + + .quad 0x899dea51abf3ff5f + .quad 0x9b93a8672fc2d8ba + .quad 0x2c38cb97be6ebd5c + .quad 0x114d578497263b5d + .quad 0x58e84c6f20816247 + .quad 0x8db2b2b6e36fd793 + .quad 0x977182561d484d85 + .quad 0x0822024f8632abd7 + .quad 0xb301bb7c6b1beca3 + .quad 0x55393f6dc6eb1375 + .quad 0x910d281097b6e4eb + .quad 0x1ad4548d9d479ea3 + + // 2^188 * 3 * B + + .quad 0xcd5a7da0389a48fd + .quad 0xb38fa4aa9a78371e + .quad 0xc6d9761b2cdb8e6c + .quad 0x35cf51dbc97e1443 + .quad 0xa06fe66d0fe9fed3 + .quad 0xa8733a401c587909 + .quad 0x30d14d800df98953 + .quad 0x41ce5876c7b30258 + .quad 0x59ac3bc5d670c022 + .quad 0xeae67c109b119406 + .quad 0x9798bdf0b3782fda + .quad 0x651e3201fd074092 + + // 2^188 * 4 * B + + .quad 0xd63d8483ef30c5cf + .quad 0x4cd4b4962361cc0c + .quad 0xee90e500a48426ac + .quad 0x0af51d7d18c14eeb + .quad 0xa57ba4a01efcae9e + .quad 0x769f4beedc308a94 + .quad 0xd1f10eeb3603cb2e + .quad 0x4099ce5e7e441278 + .quad 0x1ac98e4f8a5121e9 + .quad 0x7dae9544dbfa2fe0 + .quad 0x8320aa0dd6430df9 + .quad 0x667282652c4a2fb5 + + // 2^188 * 5 * B + + .quad 0x874621f4d86bc9ab + .quad 0xb54c7bbe56fe6fea + .quad 0x077a24257fadc22c + .quad 0x1ab53be419b90d39 + .quad 0xada8b6e02946db23 + .quad 0x1c0ce51a7b253ab7 + .quad 0x8448c85a66dd485b + .quad 0x7f1fc025d0675adf + .quad 0xd8ee1b18319ea6aa + .quad 0x004d88083a21f0da + .quad 0x3bd6aa1d883a4f4b + .quad 0x4db9a3a6dfd9fd14 + + // 2^188 * 6 * B + + .quad 0x8ce7b23bb99c0755 + .quad 0x35c5d6edc4f50f7a + .quad 0x7e1e2ed2ed9b50c3 + .quad 0x36305f16e8934da1 + .quad 0xd95b00bbcbb77c68 + .quad 0xddbc846a91f17849 + .quad 0x7cf700aebe28d9b3 + .quad 0x5ce1285c85d31f3e + .quad 0x31b6972d98b0bde8 + .quad 0x7d920706aca6de5b + .quad 0xe67310f8908a659f + .quad 0x50fac2a6efdf0235 + + // 2^188 * 7 * B + + .quad 0xf3d3a9f35b880f5a + .quad 0xedec050cdb03e7c2 + .quad 0xa896981ff9f0b1a2 + .quad 0x49a4ae2bac5e34a4 + .quad 0x295b1c86f6f449bc + .quad 0x51b2e84a1f0ab4dd + .quad 0xc001cb30aa8e551d + .quad 0x6a28d35944f43662 + .quad 0x28bb12ee04a740e0 + .quad 0x14313bbd9bce8174 + .quad 0x72f5b5e4e8c10c40 + .quad 0x7cbfb19936adcd5b + + // 2^188 * 8 * B + + .quad 0xa311ddc26b89792d + .quad 0x1b30b4c6da512664 + .quad 0x0ca77b4ccf150859 + .quad 0x1de443df1b009408 + .quad 0x8e793a7acc36e6e0 + .quad 0xf9fab7a37d586eed + .quad 0x3a4f9692bae1f4e4 + .quad 0x1c14b03eff5f447e + .quad 0x19647bd114a85291 + .quad 0x57b76cb21034d3af + .quad 0x6329db440f9d6dfa + .quad 0x5ef43e586a571493 + + // 2^192 * 1 * B + + .quad 0xef782014385675a6 + .quad 0xa2649f30aafda9e8 + .quad 0x4cd1eb505cdfa8cb + .quad 0x46115aba1d4dc0b3 + .quad 0xa66dcc9dc80c1ac0 + .quad 0x97a05cf41b38a436 + .quad 0xa7ebf3be95dbd7c6 + .quad 0x7da0b8f68d7e7dab + .quad 0xd40f1953c3b5da76 + .quad 0x1dac6f7321119e9b + .quad 0x03cc6021feb25960 + .quad 0x5a5f887e83674b4b + + // 2^192 * 2 * B + + .quad 0x8f6301cf70a13d11 + .quad 0xcfceb815350dd0c4 + .quad 0xf70297d4a4bca47e + .quad 0x3669b656e44d1434 + .quad 0x9e9628d3a0a643b9 + .quad 0xb5c3cb00e6c32064 + .quad 0x9b5302897c2dec32 + .quad 0x43e37ae2d5d1c70c + .quad 0x387e3f06eda6e133 + .quad 0x67301d5199a13ac0 + .quad 0xbd5ad8f836263811 + .quad 0x6a21e6cd4fd5e9be + + // 2^192 * 3 * B + + .quad 0xf1c6170a3046e65f + .quad 0x58712a2a00d23524 + .quad 0x69dbbd3c8c82b755 + .quad 0x586bf9f1a195ff57 + .quad 0xef4129126699b2e3 + .quad 0x71d30847708d1301 + .quad 0x325432d01182b0bd + .quad 0x45371b07001e8b36 + .quad 0xa6db088d5ef8790b + .quad 0x5278f0dc610937e5 + .quad 0xac0349d261a16eb8 + .quad 0x0eafb03790e52179 + + // 2^192 * 4 * B + + .quad 0x960555c13748042f + .quad 0x219a41e6820baa11 + .quad 0x1c81f73873486d0c + .quad 0x309acc675a02c661 + .quad 0x5140805e0f75ae1d + .quad 0xec02fbe32662cc30 + .quad 0x2cebdf1eea92396d + .quad 0x44ae3344c5435bb3 + .quad 0x9cf289b9bba543ee + .quad 0xf3760e9d5ac97142 + .quad 0x1d82e5c64f9360aa + .quad 0x62d5221b7f94678f + + // 2^192 * 5 * B + + .quad 0x524c299c18d0936d + .quad 0xc86bb56c8a0c1a0c + .quad 0xa375052edb4a8631 + .quad 0x5c0efde4bc754562 + .quad 0x7585d4263af77a3c + .quad 0xdfae7b11fee9144d + .quad 0xa506708059f7193d + .quad 0x14f29a5383922037 + .quad 0xdf717edc25b2d7f5 + .quad 0x21f970db99b53040 + .quad 0xda9234b7c3ed4c62 + .quad 0x5e72365c7bee093e + + // 2^192 * 6 * B + + .quad 0x575bfc074571217f + .quad 0x3779675d0694d95b + .quad 0x9a0a37bbf4191e33 + .quad 0x77f1104c47b4eabc + .quad 0x7d9339062f08b33e + .quad 0x5b9659e5df9f32be + .quad 0xacff3dad1f9ebdfd + .quad 0x70b20555cb7349b7 + .quad 0xbe5113c555112c4c + .quad 0x6688423a9a881fcd + .quad 0x446677855e503b47 + .quad 0x0e34398f4a06404a + + // 2^192 * 7 * B + + .quad 0xb67d22d93ecebde8 + .quad 0x09b3e84127822f07 + .quad 0x743fa61fb05b6d8d + .quad 0x5e5405368a362372 + .quad 0x18930b093e4b1928 + .quad 0x7de3e10e73f3f640 + .quad 0xf43217da73395d6f + .quad 0x6f8aded6ca379c3e + .quad 0xe340123dfdb7b29a + .quad 0x487b97e1a21ab291 + .quad 0xf9967d02fde6949e + .quad 0x780de72ec8d3de97 + + // 2^192 * 8 * B + + .quad 0x0ae28545089ae7bc + .quad 0x388ddecf1c7f4d06 + .quad 0x38ac15510a4811b8 + .quad 0x0eb28bf671928ce4 + .quad 0x671feaf300f42772 + .quad 0x8f72eb2a2a8c41aa + .quad 0x29a17fd797373292 + .quad 0x1defc6ad32b587a6 + .quad 0xaf5bbe1aef5195a7 + .quad 0x148c1277917b15ed + .quad 0x2991f7fb7ae5da2e + .quad 0x467d201bf8dd2867 + + // 2^196 * 1 * B + + .quad 0x7906ee72f7bd2e6b + .quad 0x05d270d6109abf4e + .quad 0x8d5cfe45b941a8a4 + .quad 0x44c218671c974287 + .quad 0x745f9d56296bc318 + .quad 0x993580d4d8152e65 + .quad 0xb0e5b13f5839e9ce + .quad 0x51fc2b28d43921c0 + .quad 0x1b8fd11795e2a98c + .quad 0x1c4e5ee12b6b6291 + .quad 0x5b30e7107424b572 + .quad 0x6e6b9de84c4f4ac6 + + // 2^196 * 2 * B + + .quad 0xdff25fce4b1de151 + .quad 0xd841c0c7e11c4025 + .quad 0x2554b3c854749c87 + .quad 0x2d292459908e0df9 + .quad 0x6b7c5f10f80cb088 + .quad 0x736b54dc56e42151 + .quad 0xc2b620a5c6ef99c4 + .quad 0x5f4c802cc3a06f42 + .quad 0x9b65c8f17d0752da + .quad 0x881ce338c77ee800 + .quad 0xc3b514f05b62f9e3 + .quad 0x66ed5dd5bec10d48 + + // 2^196 * 3 * B + + .quad 0x7d38a1c20bb2089d + .quad 0x808334e196ccd412 + .quad 0xc4a70b8c6c97d313 + .quad 0x2eacf8bc03007f20 + .quad 0xf0adf3c9cbca047d + .quad 0x81c3b2cbf4552f6b + .quad 0xcfda112d44735f93 + .quad 0x1f23a0c77e20048c + .quad 0xf235467be5bc1570 + .quad 0x03d2d9020dbab38c + .quad 0x27529aa2fcf9e09e + .quad 0x0840bef29d34bc50 + + // 2^196 * 4 * B + + .quad 0x796dfb35dc10b287 + .quad 0x27176bcd5c7ff29d + .quad 0x7f3d43e8c7b24905 + .quad 0x0304f5a191c54276 + .quad 0xcd54e06b7f37e4eb + .quad 0x8cc15f87f5e96cca + .quad 0xb8248bb0d3597dce + .quad 0x246affa06074400c + .quad 0x37d88e68fbe45321 + .quad 0x86097548c0d75032 + .quad 0x4e9b13ef894a0d35 + .quad 0x25a83cac5753d325 + + // 2^196 * 5 * B + + .quad 0x10222f48eed8165e + .quad 0x623fc1234b8bcf3a + .quad 0x1e145c09c221e8f0 + .quad 0x7ccfa59fca782630 + .quad 0x9f0f66293952b6e2 + .quad 0x33db5e0e0934267b + .quad 0xff45252bd609fedc + .quad 0x06be10f5c506e0c9 + .quad 0x1a9615a9b62a345f + .quad 0x22050c564a52fecc + .quad 0xa7a2788528bc0dfe + .quad 0x5e82770a1a1ee71d + + // 2^196 * 6 * B + + .quad 0x35425183ad896a5c + .quad 0xe8673afbe78d52f6 + .quad 0x2c66f25f92a35f64 + .quad 0x09d04f3b3b86b102 + .quad 0xe802e80a42339c74 + .quad 0x34175166a7fffae5 + .quad 0x34865d1f1c408cae + .quad 0x2cca982c605bc5ee + .quad 0xfd2d5d35197dbe6e + .quad 0x207c2eea8be4ffa3 + .quad 0x2613d8db325ae918 + .quad 0x7a325d1727741d3e + + // 2^196 * 7 * B + + .quad 0xd036b9bbd16dfde2 + .quad 0xa2055757c497a829 + .quad 0x8e6cc966a7f12667 + .quad 0x4d3b1a791239c180 + .quad 0xecd27d017e2a076a + .quad 0xd788689f1636495e + .quad 0x52a61af0919233e5 + .quad 0x2a479df17bb1ae64 + .quad 0x9e5eee8e33db2710 + .quad 0x189854ded6c43ca5 + .quad 0xa41c22c592718138 + .quad 0x27ad5538a43a5e9b + + // 2^196 * 8 * B + + .quad 0x2746dd4b15350d61 + .quad 0xd03fcbc8ee9521b7 + .quad 0xe86e365a138672ca + .quad 0x510e987f7e7d89e2 + .quad 0xcb5a7d638e47077c + .quad 0x8db7536120a1c059 + .quad 0x549e1e4d8bedfdcc + .quad 0x080153b7503b179d + .quad 0xdda69d930a3ed3e3 + .quad 0x3d386ef1cd60a722 + .quad 0xc817ad58bdaa4ee6 + .quad 0x23be8d554fe7372a + + // 2^200 * 1 * B + + .quad 0x95fe919a74ef4fad + .quad 0x3a827becf6a308a2 + .quad 0x964e01d309a47b01 + .quad 0x71c43c4f5ba3c797 + .quad 0xbc1ef4bd567ae7a9 + .quad 0x3f624cb2d64498bd + .quad 0xe41064d22c1f4ec8 + .quad 0x2ef9c5a5ba384001 + .quad 0xb6fd6df6fa9e74cd + .quad 0xf18278bce4af267a + .quad 0x8255b3d0f1ef990e + .quad 0x5a758ca390c5f293 + + // 2^200 * 2 * B + + .quad 0xa2b72710d9462495 + .quad 0x3aa8c6d2d57d5003 + .quad 0xe3d400bfa0b487ca + .quad 0x2dbae244b3eb72ec + .quad 0x8ce0918b1d61dc94 + .quad 0x8ded36469a813066 + .quad 0xd4e6a829afe8aad3 + .quad 0x0a738027f639d43f + .quad 0x980f4a2f57ffe1cc + .quad 0x00670d0de1839843 + .quad 0x105c3f4a49fb15fd + .quad 0x2698ca635126a69c + + // 2^200 * 3 * B + + .quad 0xe765318832b0ba78 + .quad 0x381831f7925cff8b + .quad 0x08a81b91a0291fcc + .quad 0x1fb43dcc49caeb07 + .quad 0x2e3d702f5e3dd90e + .quad 0x9e3f0918e4d25386 + .quad 0x5e773ef6024da96a + .quad 0x3c004b0c4afa3332 + .quad 0x9aa946ac06f4b82b + .quad 0x1ca284a5a806c4f3 + .quad 0x3ed3265fc6cd4787 + .quad 0x6b43fd01cd1fd217 + + // 2^200 * 4 * B + + .quad 0xc7a75d4b4697c544 + .quad 0x15fdf848df0fffbf + .quad 0x2868b9ebaa46785a + .quad 0x5a68d7105b52f714 + .quad 0xb5c742583e760ef3 + .quad 0x75dc52b9ee0ab990 + .quad 0xbf1427c2072b923f + .quad 0x73420b2d6ff0d9f0 + .quad 0xaf2cf6cb9e851e06 + .quad 0x8f593913c62238c4 + .quad 0xda8ab89699fbf373 + .quad 0x3db5632fea34bc9e + + // 2^200 * 5 * B + + .quad 0xf46eee2bf75dd9d8 + .quad 0x0d17b1f6396759a5 + .quad 0x1bf2d131499e7273 + .quad 0x04321adf49d75f13 + .quad 0x2e4990b1829825d5 + .quad 0xedeaeb873e9a8991 + .quad 0xeef03d394c704af8 + .quad 0x59197ea495df2b0e + .quad 0x04e16019e4e55aae + .quad 0xe77b437a7e2f92e9 + .quad 0xc7ce2dc16f159aa4 + .quad 0x45eafdc1f4d70cc0 + + // 2^200 * 6 * B + + .quad 0x698401858045d72b + .quad 0x4c22faa2cf2f0651 + .quad 0x941a36656b222dc6 + .quad 0x5a5eebc80362dade + .quad 0xb60e4624cfccb1ed + .quad 0x59dbc292bd5c0395 + .quad 0x31a09d1ddc0481c9 + .quad 0x3f73ceea5d56d940 + .quad 0xb7a7bfd10a4e8dc6 + .quad 0xbe57007e44c9b339 + .quad 0x60c1207f1557aefa + .quad 0x26058891266218db + + // 2^200 * 7 * B + + .quad 0x59f704a68360ff04 + .quad 0xc3d93fde7661e6f4 + .quad 0x831b2a7312873551 + .quad 0x54ad0c2e4e615d57 + .quad 0x4c818e3cc676e542 + .quad 0x5e422c9303ceccad + .quad 0xec07cccab4129f08 + .quad 0x0dedfa10b24443b8 + .quad 0xee3b67d5b82b522a + .quad 0x36f163469fa5c1eb + .quad 0xa5b4d2f26ec19fd3 + .quad 0x62ecb2baa77a9408 + + // 2^200 * 8 * B + + .quad 0xe5ed795261152b3d + .quad 0x4962357d0eddd7d1 + .quad 0x7482c8d0b96b4c71 + .quad 0x2e59f919a966d8be + .quad 0x92072836afb62874 + .quad 0x5fcd5e8579e104a5 + .quad 0x5aad01adc630a14a + .quad 0x61913d5075663f98 + .quad 0x0dc62d361a3231da + .quad 0xfa47583294200270 + .quad 0x02d801513f9594ce + .quad 0x3ddbc2a131c05d5c + + // 2^204 * 1 * B + + .quad 0x3f50a50a4ffb81ef + .quad 0xb1e035093bf420bf + .quad 0x9baa8e1cc6aa2cd0 + .quad 0x32239861fa237a40 + .quad 0xfb735ac2004a35d1 + .quad 0x31de0f433a6607c3 + .quad 0x7b8591bfc528d599 + .quad 0x55be9a25f5bb050c + .quad 0x0d005acd33db3dbf + .quad 0x0111b37c80ac35e2 + .quad 0x4892d66c6f88ebeb + .quad 0x770eadb16508fbcd + + // 2^204 * 2 * B + + .quad 0x8451f9e05e4e89dd + .quad 0xc06302ffbc793937 + .quad 0x5d22749556a6495c + .quad 0x09a6755ca05603fb + .quad 0xf1d3b681a05071b9 + .quad 0x2207659a3592ff3a + .quad 0x5f0169297881e40e + .quad 0x16bedd0e86ba374e + .quad 0x5ecccc4f2c2737b5 + .quad 0x43b79e0c2dccb703 + .quad 0x33e008bc4ec43df3 + .quad 0x06c1b840f07566c0 + + // 2^204 * 3 * B + + .quad 0x7688a5c6a388f877 + .quad 0x02a96c14deb2b6ac + .quad 0x64c9f3431b8c2af8 + .quad 0x3628435554a1eed6 + .quad 0x69ee9e7f9b02805c + .quad 0xcbff828a547d1640 + .quad 0x3d93a869b2430968 + .quad 0x46b7b8cd3fe26972 + .quad 0xe9812086fe7eebe0 + .quad 0x4cba6be72f515437 + .quad 0x1d04168b516efae9 + .quad 0x5ea1391043982cb9 + + // 2^204 * 4 * B + + .quad 0x49125c9cf4702ee1 + .quad 0x4520b71f8b25b32d + .quad 0x33193026501fef7e + .quad 0x656d8997c8d2eb2b + .quad 0x6f2b3be4d5d3b002 + .quad 0xafec33d96a09c880 + .quad 0x035f73a4a8bcc4cc + .quad 0x22c5b9284662198b + .quad 0xcb58c8fe433d8939 + .quad 0x89a0cb2e6a8d7e50 + .quad 0x79ca955309fbbe5a + .quad 0x0c626616cd7fc106 + + // 2^204 * 5 * B + + .quad 0x1ffeb80a4879b61f + .quad 0x6396726e4ada21ed + .quad 0x33c7b093368025ba + .quad 0x471aa0c6f3c31788 + .quad 0x8fdfc379fbf454b1 + .quad 0x45a5a970f1a4b771 + .quad 0xac921ef7bad35915 + .quad 0x42d088dca81c2192 + .quad 0x8fda0f37a0165199 + .quad 0x0adadb77c8a0e343 + .quad 0x20fbfdfcc875e820 + .quad 0x1cf2bea80c2206e7 + + // 2^204 * 6 * B + + .quad 0xc2ddf1deb36202ac + .quad 0x92a5fe09d2e27aa5 + .quad 0x7d1648f6fc09f1d3 + .quad 0x74c2cc0513bc4959 + .quad 0x982d6e1a02c0412f + .quad 0x90fa4c83db58e8fe + .quad 0x01c2f5bcdcb18bc0 + .quad 0x686e0c90216abc66 + .quad 0x1fadbadba54395a7 + .quad 0xb41a02a0ae0da66a + .quad 0xbf19f598bba37c07 + .quad 0x6a12b8acde48430d + + // 2^204 * 7 * B + + .quad 0xf8daea1f39d495d9 + .quad 0x592c190e525f1dfc + .quad 0xdb8cbd04c9991d1b + .quad 0x11f7fda3d88f0cb7 + .quad 0x793bdd801aaeeb5f + .quad 0x00a2a0aac1518871 + .quad 0xe8a373a31f2136b4 + .quad 0x48aab888fc91ef19 + .quad 0x041f7e925830f40e + .quad 0x002d6ca979661c06 + .quad 0x86dc9ff92b046a2e + .quad 0x760360928b0493d1 + + // 2^204 * 8 * B + + .quad 0x21bb41c6120cf9c6 + .quad 0xeab2aa12decda59b + .quad 0xc1a72d020aa48b34 + .quad 0x215d4d27e87d3b68 + .quad 0xb43108e5695a0b05 + .quad 0x6cb00ee8ad37a38b + .quad 0x5edad6eea3537381 + .quad 0x3f2602d4b6dc3224 + .quad 0xc8b247b65bcaf19c + .quad 0x49779dc3b1b2c652 + .quad 0x89a180bbd5ece2e2 + .quad 0x13f098a3cec8e039 + + // 2^208 * 1 * B + + .quad 0x9adc0ff9ce5ec54b + .quad 0x039c2a6b8c2f130d + .quad 0x028007c7f0f89515 + .quad 0x78968314ac04b36b + .quad 0xf3aa57a22796bb14 + .quad 0x883abab79b07da21 + .quad 0xe54be21831a0391c + .quad 0x5ee7fb38d83205f9 + .quad 0x538dfdcb41446a8e + .quad 0xa5acfda9434937f9 + .quad 0x46af908d263c8c78 + .quad 0x61d0633c9bca0d09 + + // 2^208 * 2 * B + + .quad 0x63744935ffdb2566 + .quad 0xc5bd6b89780b68bb + .quad 0x6f1b3280553eec03 + .quad 0x6e965fd847aed7f5 + .quad 0xada328bcf8fc73df + .quad 0xee84695da6f037fc + .quad 0x637fb4db38c2a909 + .quad 0x5b23ac2df8067bdc + .quad 0x9ad2b953ee80527b + .quad 0xe88f19aafade6d8d + .quad 0x0e711704150e82cf + .quad 0x79b9bbb9dd95dedc + + // 2^208 * 3 * B + + .quad 0xebb355406a3126c2 + .quad 0xd26383a868c8c393 + .quad 0x6c0c6429e5b97a82 + .quad 0x5065f158c9fd2147 + .quad 0xd1997dae8e9f7374 + .quad 0xa032a2f8cfbb0816 + .quad 0xcd6cba126d445f0a + .quad 0x1ba811460accb834 + .quad 0x708169fb0c429954 + .quad 0xe14600acd76ecf67 + .quad 0x2eaab98a70e645ba + .quad 0x3981f39e58a4faf2 + + // 2^208 * 4 * B + + .quad 0x18fb8a7559230a93 + .quad 0x1d168f6960e6f45d + .quad 0x3a85a94514a93cb5 + .quad 0x38dc083705acd0fd + .quad 0xc845dfa56de66fde + .quad 0xe152a5002c40483a + .quad 0xe9d2e163c7b4f632 + .quad 0x30f4452edcbc1b65 + .quad 0x856d2782c5759740 + .quad 0xfa134569f99cbecc + .quad 0x8844fc73c0ea4e71 + .quad 0x632d9a1a593f2469 + + // 2^208 * 5 * B + + .quad 0xf6bb6b15b807cba6 + .quad 0x1823c7dfbc54f0d7 + .quad 0xbb1d97036e29670b + .quad 0x0b24f48847ed4a57 + .quad 0xbf09fd11ed0c84a7 + .quad 0x63f071810d9f693a + .quad 0x21908c2d57cf8779 + .quad 0x3a5a7df28af64ba2 + .quad 0xdcdad4be511beac7 + .quad 0xa4538075ed26ccf2 + .quad 0xe19cff9f005f9a65 + .quad 0x34fcf74475481f63 + + // 2^208 * 6 * B + + .quad 0xc197e04c789767ca + .quad 0xb8714dcb38d9467d + .quad 0x55de888283f95fa8 + .quad 0x3d3bdc164dfa63f7 + .quad 0xa5bb1dab78cfaa98 + .quad 0x5ceda267190b72f2 + .quad 0x9309c9110a92608e + .quad 0x0119a3042fb374b0 + .quad 0x67a2d89ce8c2177d + .quad 0x669da5f66895d0c1 + .quad 0xf56598e5b282a2b0 + .quad 0x56c088f1ede20a73 + + // 2^208 * 7 * B + + .quad 0x336d3d1110a86e17 + .quad 0xd7f388320b75b2fa + .quad 0xf915337625072988 + .quad 0x09674c6b99108b87 + .quad 0x581b5fac24f38f02 + .quad 0xa90be9febae30cbd + .quad 0x9a2169028acf92f0 + .quad 0x038b7ea48359038f + .quad 0x9f4ef82199316ff8 + .quad 0x2f49d282eaa78d4f + .quad 0x0971a5ab5aef3174 + .quad 0x6e5e31025969eb65 + + // 2^208 * 8 * B + + .quad 0xb16c62f587e593fb + .quad 0x4999eddeca5d3e71 + .quad 0xb491c1e014cc3e6d + .quad 0x08f5114789a8dba8 + .quad 0x3304fb0e63066222 + .quad 0xfb35068987acba3f + .quad 0xbd1924778c1061a3 + .quad 0x3058ad43d1838620 + .quad 0x323c0ffde57663d0 + .quad 0x05c3df38a22ea610 + .quad 0xbdc78abdac994f9a + .quad 0x26549fa4efe3dc99 + + // 2^212 * 1 * B + + .quad 0x738b38d787ce8f89 + .quad 0xb62658e24179a88d + .quad 0x30738c9cf151316d + .quad 0x49128c7f727275c9 + .quad 0x04dbbc17f75396b9 + .quad 0x69e6a2d7d2f86746 + .quad 0xc6409d99f53eabc6 + .quad 0x606175f6332e25d2 + .quad 0x4021370ef540e7dd + .quad 0x0910d6f5a1f1d0a5 + .quad 0x4634aacd5b06b807 + .quad 0x6a39e6356944f235 + + // 2^212 * 2 * B + + .quad 0x96cd5640df90f3e7 + .quad 0x6c3a760edbfa25ea + .quad 0x24f3ef0959e33cc4 + .quad 0x42889e7e530d2e58 + .quad 0x1da1965774049e9d + .quad 0xfbcd6ea198fe352b + .quad 0xb1cbcd50cc5236a6 + .quad 0x1f5ec83d3f9846e2 + .quad 0x8efb23c3328ccb75 + .quad 0xaf42a207dd876ee9 + .quad 0x20fbdadc5dfae796 + .quad 0x241e246b06bf9f51 + + // 2^212 * 3 * B + + .quad 0x29e68e57ad6e98f6 + .quad 0x4c9260c80b462065 + .quad 0x3f00862ea51ebb4b + .quad 0x5bc2c77fb38d9097 + .quad 0x7eaafc9a6280bbb8 + .quad 0x22a70f12f403d809 + .quad 0x31ce40bb1bfc8d20 + .quad 0x2bc65635e8bd53ee + .quad 0xe8d5dc9fa96bad93 + .quad 0xe58fb17dde1947dc + .quad 0x681532ea65185fa3 + .quad 0x1fdd6c3b034a7830 + + // 2^212 * 4 * B + + .quad 0x0a64e28c55dc18fe + .quad 0xe3df9e993399ebdd + .quad 0x79ac432370e2e652 + .quad 0x35ff7fc33ae4cc0e + .quad 0x9c13a6a52dd8f7a9 + .quad 0x2dbb1f8c3efdcabf + .quad 0x961e32405e08f7b5 + .quad 0x48c8a121bbe6c9e5 + .quad 0xfc415a7c59646445 + .quad 0xd224b2d7c128b615 + .quad 0x6035c9c905fbb912 + .quad 0x42d7a91274429fab + + // 2^212 * 5 * B + + .quad 0x4e6213e3eaf72ed3 + .quad 0x6794981a43acd4e7 + .quad 0xff547cde6eb508cb + .quad 0x6fed19dd10fcb532 + .quad 0xa9a48947933da5bc + .quad 0x4a58920ec2e979ec + .quad 0x96d8800013e5ac4c + .quad 0x453692d74b48b147 + .quad 0xdd775d99a8559c6f + .quad 0xf42a2140df003e24 + .quad 0x5223e229da928a66 + .quad 0x063f46ba6d38f22c + + // 2^212 * 6 * B + + .quad 0xd2d242895f536694 + .quad 0xca33a2c542939b2c + .quad 0x986fada6c7ddb95c + .quad 0x5a152c042f712d5d + .quad 0x39843cb737346921 + .quad 0xa747fb0738c89447 + .quad 0xcb8d8031a245307e + .quad 0x67810f8e6d82f068 + .quad 0x3eeb8fbcd2287db4 + .quad 0x72c7d3a301a03e93 + .quad 0x5473e88cbd98265a + .quad 0x7324aa515921b403 + + // 2^212 * 7 * B + + .quad 0x857942f46c3cbe8e + .quad 0xa1d364b14730c046 + .quad 0x1c8ed914d23c41bf + .quad 0x0838e161eef6d5d2 + .quad 0xad23f6dae82354cb + .quad 0x6962502ab6571a6d + .quad 0x9b651636e38e37d1 + .quad 0x5cac5005d1a3312f + .quad 0x8cc154cce9e39904 + .quad 0x5b3a040b84de6846 + .quad 0xc4d8a61cb1be5d6e + .quad 0x40fb897bd8861f02 + + // 2^212 * 8 * B + + .quad 0x84c5aa9062de37a1 + .quad 0x421da5000d1d96e1 + .quad 0x788286306a9242d9 + .quad 0x3c5e464a690d10da + .quad 0xe57ed8475ab10761 + .quad 0x71435e206fd13746 + .quad 0x342f824ecd025632 + .quad 0x4b16281ea8791e7b + .quad 0xd1c101d50b813381 + .quad 0xdee60f1176ee6828 + .quad 0x0cb68893383f6409 + .quad 0x6183c565f6ff484a + + // 2^216 * 1 * B + + .quad 0x741d5a461e6bf9d6 + .quad 0x2305b3fc7777a581 + .quad 0xd45574a26474d3d9 + .quad 0x1926e1dc6401e0ff + .quad 0xdb468549af3f666e + .quad 0xd77fcf04f14a0ea5 + .quad 0x3df23ff7a4ba0c47 + .quad 0x3a10dfe132ce3c85 + .quad 0xe07f4e8aea17cea0 + .quad 0x2fd515463a1fc1fd + .quad 0x175322fd31f2c0f1 + .quad 0x1fa1d01d861e5d15 + + // 2^216 * 2 * B + + .quad 0xcc8055947d599832 + .quad 0x1e4656da37f15520 + .quad 0x99f6f7744e059320 + .quad 0x773563bc6a75cf33 + .quad 0x38dcac00d1df94ab + .quad 0x2e712bddd1080de9 + .quad 0x7f13e93efdd5e262 + .quad 0x73fced18ee9a01e5 + .quad 0x06b1e90863139cb3 + .quad 0xa493da67c5a03ecd + .quad 0x8d77cec8ad638932 + .quad 0x1f426b701b864f44 + + // 2^216 * 3 * B + + .quad 0xefc9264c41911c01 + .quad 0xf1a3b7b817a22c25 + .quad 0x5875da6bf30f1447 + .quad 0x4e1af5271d31b090 + .quad 0xf17e35c891a12552 + .quad 0xb76b8153575e9c76 + .quad 0xfa83406f0d9b723e + .quad 0x0b76bb1b3fa7e438 + .quad 0x08b8c1f97f92939b + .quad 0xbe6771cbd444ab6e + .quad 0x22e5646399bb8017 + .quad 0x7b6dd61eb772a955 + + // 2^216 * 4 * B + + .quad 0xb7adc1e850f33d92 + .quad 0x7998fa4f608cd5cf + .quad 0xad962dbd8dfc5bdb + .quad 0x703e9bceaf1d2f4f + .quad 0x5730abf9ab01d2c7 + .quad 0x16fb76dc40143b18 + .quad 0x866cbe65a0cbb281 + .quad 0x53fa9b659bff6afe + .quad 0x6c14c8e994885455 + .quad 0x843a5d6665aed4e5 + .quad 0x181bb73ebcd65af1 + .quad 0x398d93e5c4c61f50 + + // 2^216 * 5 * B + + .quad 0x1c4bd16733e248f3 + .quad 0xbd9e128715bf0a5f + .quad 0xd43f8cf0a10b0376 + .quad 0x53b09b5ddf191b13 + .quad 0xc3877c60d2e7e3f2 + .quad 0x3b34aaa030828bb1 + .quad 0x283e26e7739ef138 + .quad 0x699c9c9002c30577 + .quad 0xf306a7235946f1cc + .quad 0x921718b5cce5d97d + .quad 0x28cdd24781b4e975 + .quad 0x51caf30c6fcdd907 + + // 2^216 * 6 * B + + .quad 0xa60ba7427674e00a + .quad 0x630e8570a17a7bf3 + .quad 0x3758563dcf3324cc + .quad 0x5504aa292383fdaa + .quad 0x737af99a18ac54c7 + .quad 0x903378dcc51cb30f + .quad 0x2b89bc334ce10cc7 + .quad 0x12ae29c189f8e99a + .quad 0xa99ec0cb1f0d01cf + .quad 0x0dd1efcc3a34f7ae + .quad 0x55ca7521d09c4e22 + .quad 0x5fd14fe958eba5ea + + // 2^216 * 7 * B + + .quad 0xb5dc2ddf2845ab2c + .quad 0x069491b10a7fe993 + .quad 0x4daaf3d64002e346 + .quad 0x093ff26e586474d1 + .quad 0x3c42fe5ebf93cb8e + .quad 0xbedfa85136d4565f + .quad 0xe0f0859e884220e8 + .quad 0x7dd73f960725d128 + .quad 0xb10d24fe68059829 + .quad 0x75730672dbaf23e5 + .quad 0x1367253ab457ac29 + .quad 0x2f59bcbc86b470a4 + + // 2^216 * 8 * B + + .quad 0x83847d429917135f + .quad 0xad1b911f567d03d7 + .quad 0x7e7748d9be77aad1 + .quad 0x5458b42e2e51af4a + .quad 0x7041d560b691c301 + .quad 0x85201b3fadd7e71e + .quad 0x16c2e16311335585 + .quad 0x2aa55e3d010828b1 + .quad 0xed5192e60c07444f + .quad 0x42c54e2d74421d10 + .quad 0x352b4c82fdb5c864 + .quad 0x13e9004a8a768664 + + // 2^220 * 1 * B + + .quad 0xcbb5b5556c032bff + .quad 0xdf7191b729297a3a + .quad 0xc1ff7326aded81bb + .quad 0x71ade8bb68be03f5 + .quad 0x1e6284c5806b467c + .quad 0xc5f6997be75d607b + .quad 0x8b67d958b378d262 + .quad 0x3d88d66a81cd8b70 + .quad 0x8b767a93204ed789 + .quad 0x762fcacb9fa0ae2a + .quad 0x771febcc6dce4887 + .quad 0x343062158ff05fb3 + + // 2^220 * 2 * B + + .quad 0xe05da1a7e1f5bf49 + .quad 0x26457d6dd4736092 + .quad 0x77dcb07773cc32f6 + .quad 0x0a5d94969cdd5fcd + .quad 0xfce219072a7b31b4 + .quad 0x4d7adc75aa578016 + .quad 0x0ec276a687479324 + .quad 0x6d6d9d5d1fda4beb + .quad 0x22b1a58ae9b08183 + .quad 0xfd95d071c15c388b + .quad 0xa9812376850a0517 + .quad 0x33384cbabb7f335e + + // 2^220 * 3 * B + + .quad 0x3c6fa2680ca2c7b5 + .quad 0x1b5082046fb64fda + .quad 0xeb53349c5431d6de + .quad 0x5278b38f6b879c89 + .quad 0x33bc627a26218b8d + .quad 0xea80b21fc7a80c61 + .quad 0x9458b12b173e9ee6 + .quad 0x076247be0e2f3059 + .quad 0x52e105f61416375a + .quad 0xec97af3685abeba4 + .quad 0x26e6b50623a67c36 + .quad 0x5cf0e856f3d4fb01 + + // 2^220 * 4 * B + + .quad 0xf6c968731ae8cab4 + .quad 0x5e20741ecb4f92c5 + .quad 0x2da53be58ccdbc3e + .quad 0x2dddfea269970df7 + .quad 0xbeaece313db342a8 + .quad 0xcba3635b842db7ee + .quad 0xe88c6620817f13ef + .quad 0x1b9438aa4e76d5c6 + .quad 0x8a50777e166f031a + .quad 0x067b39f10fb7a328 + .quad 0x1925c9a6010fbd76 + .quad 0x6df9b575cc740905 + + // 2^220 * 5 * B + + .quad 0x42c1192927f6bdcf + .quad 0x8f91917a403d61ca + .quad 0xdc1c5a668b9e1f61 + .quad 0x1596047804ec0f8d + .quad 0xecdfc35b48cade41 + .quad 0x6a88471fb2328270 + .quad 0x740a4a2440a01b6a + .quad 0x471e5796003b5f29 + .quad 0xda96bbb3aced37ac + .quad 0x7a2423b5e9208cea + .quad 0x24cc5c3038aebae2 + .quad 0x50c356afdc5dae2f + + // 2^220 * 6 * B + + .quad 0x09dcbf4341c30318 + .quad 0xeeba061183181dce + .quad 0xc179c0cedc1e29a1 + .quad 0x1dbf7b89073f35b0 + .quad 0xcfed9cdf1b31b964 + .quad 0xf486a9858ca51af3 + .quad 0x14897265ea8c1f84 + .quad 0x784a53dd932acc00 + .quad 0x2d99f9df14fc4920 + .quad 0x76ccb60cc4499fe5 + .quad 0xa4132cbbe5cf0003 + .quad 0x3f93d82354f000ea + + // 2^220 * 7 * B + + .quad 0x8183e7689e04ce85 + .quad 0x678fb71e04465341 + .quad 0xad92058f6688edac + .quad 0x5da350d3532b099a + .quad 0xeaac12d179e14978 + .quad 0xff923ff3bbebff5e + .quad 0x4af663e40663ce27 + .quad 0x0fd381a811a5f5ff + .quad 0xf256aceca436df54 + .quad 0x108b6168ae69d6e8 + .quad 0x20d986cb6b5d036c + .quad 0x655957b9fee2af50 + + // 2^220 * 8 * B + + .quad 0xaea8b07fa902030f + .quad 0xf88c766af463d143 + .quad 0x15b083663c787a60 + .quad 0x08eab1148267a4a8 + .quad 0xbdc1409bd002d0ac + .quad 0x66660245b5ccd9a6 + .quad 0x82317dc4fade85ec + .quad 0x02fe934b6ad7df0d + .quad 0xef5cf100cfb7ea74 + .quad 0x22897633a1cb42ac + .quad 0xd4ce0c54cef285e2 + .quad 0x30408c048a146a55 + + // 2^224 * 1 * B + + .quad 0x739d8845832fcedb + .quad 0xfa38d6c9ae6bf863 + .quad 0x32bc0dcab74ffef7 + .quad 0x73937e8814bce45e + .quad 0xbb2e00c9193b877f + .quad 0xece3a890e0dc506b + .quad 0xecf3b7c036de649f + .quad 0x5f46040898de9e1a + .quad 0xb9037116297bf48d + .quad 0xa9d13b22d4f06834 + .quad 0xe19715574696bdc6 + .quad 0x2cf8a4e891d5e835 + + // 2^224 * 2 * B + + .quad 0x6d93fd8707110f67 + .quad 0xdd4c09d37c38b549 + .quad 0x7cb16a4cc2736a86 + .quad 0x2049bd6e58252a09 + .quad 0x2cb5487e17d06ba2 + .quad 0x24d2381c3950196b + .quad 0xd7659c8185978a30 + .quad 0x7a6f7f2891d6a4f6 + .quad 0x7d09fd8d6a9aef49 + .quad 0xf0ee60be5b3db90b + .quad 0x4c21b52c519ebfd4 + .quad 0x6011aadfc545941d + + // 2^224 * 3 * B + + .quad 0x5f67926dcf95f83c + .quad 0x7c7e856171289071 + .quad 0xd6a1e7f3998f7a5b + .quad 0x6fc5cc1b0b62f9e0 + .quad 0x63ded0c802cbf890 + .quad 0xfbd098ca0dff6aaa + .quad 0x624d0afdb9b6ed99 + .quad 0x69ce18b779340b1e + .quad 0xd1ef5528b29879cb + .quad 0xdd1aae3cd47e9092 + .quad 0x127e0442189f2352 + .quad 0x15596b3ae57101f1 + + // 2^224 * 4 * B + + .quad 0x462739d23f9179a2 + .quad 0xff83123197d6ddcf + .quad 0x1307deb553f2148a + .quad 0x0d2237687b5f4dda + .quad 0x09ff31167e5124ca + .quad 0x0be4158bd9c745df + .quad 0x292b7d227ef556e5 + .quad 0x3aa4e241afb6d138 + .quad 0x2cc138bf2a3305f5 + .quad 0x48583f8fa2e926c3 + .quad 0x083ab1a25549d2eb + .quad 0x32fcaa6e4687a36c + + // 2^224 * 5 * B + + .quad 0x7bc56e8dc57d9af5 + .quad 0x3e0bd2ed9df0bdf2 + .quad 0xaac014de22efe4a3 + .quad 0x4627e9cefebd6a5c + .quad 0x3207a4732787ccdf + .quad 0x17e31908f213e3f8 + .quad 0xd5b2ecd7f60d964e + .quad 0x746f6336c2600be9 + .quad 0x3f4af345ab6c971c + .quad 0xe288eb729943731f + .quad 0x33596a8a0344186d + .quad 0x7b4917007ed66293 + + // 2^224 * 6 * B + + .quad 0x2d85fb5cab84b064 + .quad 0x497810d289f3bc14 + .quad 0x476adc447b15ce0c + .quad 0x122ba376f844fd7b + .quad 0x54341b28dd53a2dd + .quad 0xaa17905bdf42fc3f + .quad 0x0ff592d94dd2f8f4 + .quad 0x1d03620fe08cd37d + .quad 0xc20232cda2b4e554 + .quad 0x9ed0fd42115d187f + .quad 0x2eabb4be7dd479d9 + .quad 0x02c70bf52b68ec4c + + // 2^224 * 7 * B + + .quad 0xa287ec4b5d0b2fbb + .quad 0x415c5790074882ca + .quad 0xe044a61ec1d0815c + .quad 0x26334f0a409ef5e0 + .quad 0xace532bf458d72e1 + .quad 0x5be768e07cb73cb5 + .quad 0x56cf7d94ee8bbde7 + .quad 0x6b0697e3feb43a03 + .quad 0xb6c8f04adf62a3c0 + .quad 0x3ef000ef076da45d + .quad 0x9c9cb95849f0d2a9 + .quad 0x1cc37f43441b2fae + + // 2^224 * 8 * B + + .quad 0x508f565a5cc7324f + .quad 0xd061c4c0e506a922 + .quad 0xfb18abdb5c45ac19 + .quad 0x6c6809c10380314a + .quad 0xd76656f1c9ceaeb9 + .quad 0x1c5b15f818e5656a + .quad 0x26e72832844c2334 + .quad 0x3a346f772f196838 + .quad 0xd2d55112e2da6ac8 + .quad 0xe9bd0331b1e851ed + .quad 0x960746dd8ec67262 + .quad 0x05911b9f6ef7c5d0 + + // 2^228 * 1 * B + + .quad 0xe9dcd756b637ff2d + .quad 0xec4c348fc987f0c4 + .quad 0xced59285f3fbc7b7 + .quad 0x3305354793e1ea87 + .quad 0x01c18980c5fe9f94 + .quad 0xcd656769716fd5c8 + .quad 0x816045c3d195a086 + .quad 0x6e2b7f3266cc7982 + .quad 0xcc802468f7c3568f + .quad 0x9de9ba8219974cb3 + .quad 0xabb7229cb5b81360 + .quad 0x44e2017a6fbeba62 + + // 2^228 * 2 * B + + .quad 0xc4c2a74354dab774 + .quad 0x8e5d4c3c4eaf031a + .quad 0xb76c23d242838f17 + .quad 0x749a098f68dce4ea + .quad 0x87f82cf3b6ca6ecd + .quad 0x580f893e18f4a0c2 + .quad 0x058930072604e557 + .quad 0x6cab6ac256d19c1d + .quad 0xdcdfe0a02cc1de60 + .quad 0x032665ff51c5575b + .quad 0x2c0c32f1073abeeb + .quad 0x6a882014cd7b8606 + + // 2^228 * 3 * B + + .quad 0xa52a92fea4747fb5 + .quad 0xdc12a4491fa5ab89 + .quad 0xd82da94bb847a4ce + .quad 0x4d77edce9512cc4e + .quad 0xd111d17caf4feb6e + .quad 0x050bba42b33aa4a3 + .quad 0x17514c3ceeb46c30 + .quad 0x54bedb8b1bc27d75 + .quad 0x77c8e14577e2189c + .quad 0xa3e46f6aff99c445 + .quad 0x3144dfc86d335343 + .quad 0x3a96559e7c4216a9 + + // 2^228 * 4 * B + + .quad 0x12550d37f42ad2ee + .quad 0x8b78e00498a1fbf5 + .quad 0x5d53078233894cb2 + .quad 0x02c84e4e3e498d0c + .quad 0x4493896880baaa52 + .quad 0x4c98afc4f285940e + .quad 0xef4aa79ba45448b6 + .quad 0x5278c510a57aae7f + .quad 0xa54dd074294c0b94 + .quad 0xf55d46b8df18ffb6 + .quad 0xf06fecc58dae8366 + .quad 0x588657668190d165 + + // 2^228 * 5 * B + + .quad 0xd47712311aef7117 + .quad 0x50343101229e92c7 + .quad 0x7a95e1849d159b97 + .quad 0x2449959b8b5d29c9 + .quad 0xbf5834f03de25cc3 + .quad 0xb887c8aed6815496 + .quad 0x5105221a9481e892 + .quad 0x6760ed19f7723f93 + .quad 0x669ba3b7ac35e160 + .quad 0x2eccf73fba842056 + .quad 0x1aec1f17c0804f07 + .quad 0x0d96bc031856f4e7 + + // 2^228 * 6 * B + + .quad 0x3318be7775c52d82 + .quad 0x4cb764b554d0aab9 + .quad 0xabcf3d27cc773d91 + .quad 0x3bf4d1848123288a + .quad 0xb1d534b0cc7505e1 + .quad 0x32cd003416c35288 + .quad 0xcb36a5800762c29d + .quad 0x5bfe69b9237a0bf8 + .quad 0x183eab7e78a151ab + .quad 0xbbe990c999093763 + .quad 0xff717d6e4ac7e335 + .quad 0x4c5cddb325f39f88 + + // 2^228 * 7 * B + + .quad 0xc0f6b74d6190a6eb + .quad 0x20ea81a42db8f4e4 + .quad 0xa8bd6f7d97315760 + .quad 0x33b1d60262ac7c21 + .quad 0x57750967e7a9f902 + .quad 0x2c37fdfc4f5b467e + .quad 0xb261663a3177ba46 + .quad 0x3a375e78dc2d532b + .quad 0x8141e72f2d4dddea + .quad 0xe6eafe9862c607c8 + .quad 0x23c28458573cafd0 + .quad 0x46b9476f4ff97346 + + // 2^228 * 8 * B + + .quad 0x0c1ffea44f901e5c + .quad 0x2b0b6fb72184b782 + .quad 0xe587ff910114db88 + .quad 0x37130f364785a142 + .quad 0x1215505c0d58359f + .quad 0x2a2013c7fc28c46b + .quad 0x24a0a1af89ea664e + .quad 0x4400b638a1130e1f + .quad 0x3a01b76496ed19c3 + .quad 0x31e00ab0ed327230 + .quad 0x520a885783ca15b1 + .quad 0x06aab9875accbec7 + + // 2^232 * 1 * B + + .quad 0xc1339983f5df0ebb + .quad 0xc0f3758f512c4cac + .quad 0x2cf1130a0bb398e1 + .quad 0x6b3cecf9aa270c62 + .quad 0x5349acf3512eeaef + .quad 0x20c141d31cc1cb49 + .quad 0x24180c07a99a688d + .quad 0x555ef9d1c64b2d17 + .quad 0x36a770ba3b73bd08 + .quad 0x624aef08a3afbf0c + .quad 0x5737ff98b40946f2 + .quad 0x675f4de13381749d + + // 2^232 * 2 * B + + .quad 0x0e2c52036b1782fc + .quad 0x64816c816cad83b4 + .quad 0xd0dcbdd96964073e + .quad 0x13d99df70164c520 + .quad 0xa12ff6d93bdab31d + .quad 0x0725d80f9d652dfe + .quad 0x019c4ff39abe9487 + .quad 0x60f450b882cd3c43 + .quad 0x014b5ec321e5c0ca + .quad 0x4fcb69c9d719bfa2 + .quad 0x4e5f1c18750023a0 + .quad 0x1c06de9e55edac80 + + // 2^232 * 3 * B + + .quad 0x990f7ad6a33ec4e2 + .quad 0x6608f938be2ee08e + .quad 0x9ca143c563284515 + .quad 0x4cf38a1fec2db60d + .quad 0xffd52b40ff6d69aa + .quad 0x34530b18dc4049bb + .quad 0x5e4a5c2fa34d9897 + .quad 0x78096f8e7d32ba2d + .quad 0xa0aaaa650dfa5ce7 + .quad 0xf9c49e2a48b5478c + .quad 0x4f09cc7d7003725b + .quad 0x373cad3a26091abe + + // 2^232 * 4 * B + + .quad 0xb294634d82c9f57c + .quad 0x1fcbfde124934536 + .quad 0x9e9c4db3418cdb5a + .quad 0x0040f3d9454419fc + .quad 0xf1bea8fb89ddbbad + .quad 0x3bcb2cbc61aeaecb + .quad 0x8f58a7bb1f9b8d9d + .quad 0x21547eda5112a686 + .quad 0xdefde939fd5986d3 + .quad 0xf4272c89510a380c + .quad 0xb72ba407bb3119b9 + .quad 0x63550a334a254df4 + + // 2^232 * 5 * B + + .quad 0x6507d6edb569cf37 + .quad 0x178429b00ca52ee1 + .quad 0xea7c0090eb6bd65d + .quad 0x3eea62c7daf78f51 + .quad 0x9bba584572547b49 + .quad 0xf305c6fae2c408e0 + .quad 0x60e8fa69c734f18d + .quad 0x39a92bafaa7d767a + .quad 0x9d24c713e693274e + .quad 0x5f63857768dbd375 + .quad 0x70525560eb8ab39a + .quad 0x68436a0665c9c4cd + + // 2^232 * 6 * B + + .quad 0xbc0235e8202f3f27 + .quad 0xc75c00e264f975b0 + .quad 0x91a4e9d5a38c2416 + .quad 0x17b6e7f68ab789f9 + .quad 0x1e56d317e820107c + .quad 0xc5266844840ae965 + .quad 0xc1e0a1c6320ffc7a + .quad 0x5373669c91611472 + .quad 0x5d2814ab9a0e5257 + .quad 0x908f2084c9cab3fc + .quad 0xafcaf5885b2d1eca + .quad 0x1cb4b5a678f87d11 + + // 2^232 * 7 * B + + .quad 0xb664c06b394afc6c + .quad 0x0c88de2498da5fb1 + .quad 0x4f8d03164bcad834 + .quad 0x330bca78de7434a2 + .quad 0x6b74aa62a2a007e7 + .quad 0xf311e0b0f071c7b1 + .quad 0x5707e438000be223 + .quad 0x2dc0fd2d82ef6eac + .quad 0x982eff841119744e + .quad 0xf9695e962b074724 + .quad 0xc58ac14fbfc953fb + .quad 0x3c31be1b369f1cf5 + + // 2^232 * 8 * B + + .quad 0xb0f4864d08948aee + .quad 0x07dc19ee91ba1c6f + .quad 0x7975cdaea6aca158 + .quad 0x330b61134262d4bb + .quad 0xc168bc93f9cb4272 + .quad 0xaeb8711fc7cedb98 + .quad 0x7f0e52aa34ac8d7a + .quad 0x41cec1097e7d55bb + .quad 0xf79619d7a26d808a + .quad 0xbb1fd49e1d9e156d + .quad 0x73d7c36cdba1df27 + .quad 0x26b44cd91f28777d + + // 2^236 * 1 * B + + .quad 0x300a9035393aa6d8 + .quad 0x2b501131a12bb1cd + .quad 0x7b1ff677f093c222 + .quad 0x4309c1f8cab82bad + .quad 0xaf44842db0285f37 + .quad 0x8753189047efc8df + .quad 0x9574e091f820979a + .quad 0x0e378d6069615579 + .quad 0xd9fa917183075a55 + .quad 0x4bdb5ad26b009fdc + .quad 0x7829ad2cd63def0e + .quad 0x078fc54975fd3877 + + // 2^236 * 2 * B + + .quad 0x87dfbd1428878f2d + .quad 0x134636dd1e9421a1 + .quad 0x4f17c951257341a3 + .quad 0x5df98d4bad296cb8 + .quad 0xe2004b5bb833a98a + .quad 0x44775dec2d4c3330 + .quad 0x3aa244067eace913 + .quad 0x272630e3d58e00a9 + .quad 0xf3678fd0ecc90b54 + .quad 0xf001459b12043599 + .quad 0x26725fbc3758b89b + .quad 0x4325e4aa73a719ae + + // 2^236 * 3 * B + + .quad 0x657dc6ef433c3493 + .quad 0x65375e9f80dbf8c3 + .quad 0x47fd2d465b372dae + .quad 0x4966ab79796e7947 + .quad 0xed24629acf69f59d + .quad 0x2a4a1ccedd5abbf4 + .quad 0x3535ca1f56b2d67b + .quad 0x5d8c68d043b1b42d + .quad 0xee332d4de3b42b0a + .quad 0xd84e5a2b16a4601c + .quad 0x78243877078ba3e4 + .quad 0x77ed1eb4184ee437 + + // 2^236 * 4 * B + + .quad 0xbfd4e13f201839a0 + .quad 0xaeefffe23e3df161 + .quad 0xb65b04f06b5d1fe3 + .quad 0x52e085fb2b62fbc0 + .quad 0x185d43f89e92ed1a + .quad 0xb04a1eeafe4719c6 + .quad 0x499fbe88a6f03f4f + .quad 0x5d8b0d2f3c859bdd + .quad 0x124079eaa54cf2ba + .quad 0xd72465eb001b26e7 + .quad 0x6843bcfdc97af7fd + .quad 0x0524b42b55eacd02 + + // 2^236 * 5 * B + + .quad 0xfd0d5dbee45447b0 + .quad 0x6cec351a092005ee + .quad 0x99a47844567579cb + .quad 0x59d242a216e7fa45 + .quad 0xbc18dcad9b829eac + .quad 0x23ae7d28b5f579d0 + .quad 0xc346122a69384233 + .quad 0x1a6110b2e7d4ac89 + .quad 0x4f833f6ae66997ac + .quad 0x6849762a361839a4 + .quad 0x6985dec1970ab525 + .quad 0x53045e89dcb1f546 + + // 2^236 * 6 * B + + .quad 0xcb8bb346d75353db + .quad 0xfcfcb24bae511e22 + .quad 0xcba48d40d50ae6ef + .quad 0x26e3bae5f4f7cb5d + .quad 0x84da3cde8d45fe12 + .quad 0xbd42c218e444e2d2 + .quad 0xa85196781f7e3598 + .quad 0x7642c93f5616e2b2 + .quad 0x2323daa74595f8e4 + .quad 0xde688c8b857abeb4 + .quad 0x3fc48e961c59326e + .quad 0x0b2e73ca15c9b8ba + + // 2^236 * 7 * B + + .quad 0xd6bb4428c17f5026 + .quad 0x9eb27223fb5a9ca7 + .quad 0xe37ba5031919c644 + .quad 0x21ce380db59a6602 + .quad 0x0e3fbfaf79c03a55 + .quad 0x3077af054cbb5acf + .quad 0xd5c55245db3de39f + .quad 0x015e68c1476a4af7 + .quad 0xc1d5285220066a38 + .quad 0x95603e523570aef3 + .quad 0x832659a7226b8a4d + .quad 0x5dd689091f8eedc9 + + // 2^236 * 8 * B + + .quad 0xcbac84debfd3c856 + .quad 0x1624c348b35ff244 + .quad 0xb7f88dca5d9cad07 + .quad 0x3b0e574da2c2ebe8 + .quad 0x1d022591a5313084 + .quad 0xca2d4aaed6270872 + .quad 0x86a12b852f0bfd20 + .quad 0x56e6c439ad7da748 + .quad 0xc704ff4942bdbae6 + .quad 0x5e21ade2b2de1f79 + .quad 0xe95db3f35652fad8 + .quad 0x0822b5378f08ebc1 + + // 2^240 * 1 * B + + .quad 0x51f048478f387475 + .quad 0xb25dbcf49cbecb3c + .quad 0x9aab1244d99f2055 + .quad 0x2c709e6c1c10a5d6 + .quad 0xe1b7f29362730383 + .quad 0x4b5279ffebca8a2c + .quad 0xdafc778abfd41314 + .quad 0x7deb10149c72610f + .quad 0xcb62af6a8766ee7a + .quad 0x66cbec045553cd0e + .quad 0x588001380f0be4b5 + .quad 0x08e68e9ff62ce2ea + + // 2^240 * 2 * B + + .quad 0x34ad500a4bc130ad + .quad 0x8d38db493d0bd49c + .quad 0xa25c3d98500a89be + .quad 0x2f1f3f87eeba3b09 + .quad 0x2f2d09d50ab8f2f9 + .quad 0xacb9218dc55923df + .quad 0x4a8f342673766cb9 + .quad 0x4cb13bd738f719f5 + .quad 0xf7848c75e515b64a + .quad 0xa59501badb4a9038 + .quad 0xc20d313f3f751b50 + .quad 0x19a1e353c0ae2ee8 + + // 2^240 * 3 * B + + .quad 0x7d1c7560bafa05c3 + .quad 0xb3e1a0a0c6e55e61 + .quad 0xe3529718c0d66473 + .quad 0x41546b11c20c3486 + .quad 0xb42172cdd596bdbd + .quad 0x93e0454398eefc40 + .quad 0x9fb15347b44109b5 + .quad 0x736bd3990266ae34 + .quad 0x85532d509334b3b4 + .quad 0x46fd114b60816573 + .quad 0xcc5f5f30425c8375 + .quad 0x412295a2b87fab5c + + // 2^240 * 4 * B + + .quad 0x19c99b88f57ed6e9 + .quad 0x5393cb266df8c825 + .quad 0x5cee3213b30ad273 + .quad 0x14e153ebb52d2e34 + .quad 0x2e655261e293eac6 + .quad 0x845a92032133acdb + .quad 0x460975cb7900996b + .quad 0x0760bb8d195add80 + .quad 0x413e1a17cde6818a + .quad 0x57156da9ed69a084 + .quad 0x2cbf268f46caccb1 + .quad 0x6b34be9bc33ac5f2 + + // 2^240 * 5 * B + + .quad 0xf3df2f643a78c0b2 + .quad 0x4c3e971ef22e027c + .quad 0xec7d1c5e49c1b5a3 + .quad 0x2012c18f0922dd2d + .quad 0x11fc69656571f2d3 + .quad 0xc6c9e845530e737a + .quad 0xe33ae7a2d4fe5035 + .quad 0x01b9c7b62e6dd30b + .quad 0x880b55e55ac89d29 + .quad 0x1483241f45a0a763 + .quad 0x3d36efdfc2e76c1f + .quad 0x08af5b784e4bade8 + + // 2^240 * 6 * B + + .quad 0x283499dc881f2533 + .quad 0x9d0525da779323b6 + .quad 0x897addfb673441f4 + .quad 0x32b79d71163a168d + .quad 0xe27314d289cc2c4b + .quad 0x4be4bd11a287178d + .quad 0x18d528d6fa3364ce + .quad 0x6423c1d5afd9826e + .quad 0xcc85f8d9edfcb36a + .quad 0x22bcc28f3746e5f9 + .quad 0xe49de338f9e5d3cd + .quad 0x480a5efbc13e2dcc + + // 2^240 * 7 * B + + .quad 0x0b51e70b01622071 + .quad 0x06b505cf8b1dafc5 + .quad 0x2c6bb061ef5aabcd + .quad 0x47aa27600cb7bf31 + .quad 0xb6614ce442ce221f + .quad 0x6e199dcc4c053928 + .quad 0x663fb4a4dc1cbe03 + .quad 0x24b31d47691c8e06 + .quad 0x2a541eedc015f8c3 + .quad 0x11a4fe7e7c693f7c + .quad 0xf0af66134ea278d6 + .quad 0x545b585d14dda094 + + // 2^240 * 8 * B + + .quad 0x67bf275ea0d43a0f + .quad 0xade68e34089beebe + .quad 0x4289134cd479e72e + .quad 0x0f62f9c332ba5454 + .quad 0x6204e4d0e3b321e1 + .quad 0x3baa637a28ff1e95 + .quad 0x0b0ccffd5b99bd9e + .quad 0x4d22dc3e64c8d071 + .quad 0xfcb46589d63b5f39 + .quad 0x5cae6a3f57cbcf61 + .quad 0xfebac2d2953afa05 + .quad 0x1c0fa01a36371436 + + // 2^244 * 1 * B + + .quad 0xe7547449bc7cd692 + .quad 0x0f9abeaae6f73ddf + .quad 0x4af01ca700837e29 + .quad 0x63ab1b5d3f1bc183 + .quad 0xc11ee5e854c53fae + .quad 0x6a0b06c12b4f3ff4 + .quad 0x33540f80e0b67a72 + .quad 0x15f18fc3cd07e3ef + .quad 0x32750763b028f48c + .quad 0x06020740556a065f + .quad 0xd53bd812c3495b58 + .quad 0x08706c9b865f508d + + // 2^244 * 2 * B + + .quad 0xf37ca2ab3d343dff + .quad 0x1a8c6a2d80abc617 + .quad 0x8e49e035d4ccffca + .quad 0x48b46beebaa1d1b9 + .quad 0xcc991b4138b41246 + .quad 0x243b9c526f9ac26b + .quad 0xb9ef494db7cbabbd + .quad 0x5fba433dd082ed00 + .quad 0x9c49e355c9941ad0 + .quad 0xb9734ade74498f84 + .quad 0x41c3fed066663e5c + .quad 0x0ecfedf8e8e710b3 + + // 2^244 * 3 * B + + .quad 0x76430f9f9cd470d9 + .quad 0xb62acc9ba42f6008 + .quad 0x1898297c59adad5e + .quad 0x7789dd2db78c5080 + .quad 0x744f7463e9403762 + .quad 0xf79a8dee8dfcc9c9 + .quad 0x163a649655e4cde3 + .quad 0x3b61788db284f435 + .quad 0xb22228190d6ef6b2 + .quad 0xa94a66b246ce4bfa + .quad 0x46c1a77a4f0b6cc7 + .quad 0x4236ccffeb7338cf + + // 2^244 * 4 * B + + .quad 0x8497404d0d55e274 + .quad 0x6c6663d9c4ad2b53 + .quad 0xec2fb0d9ada95734 + .quad 0x2617e120cdb8f73c + .quad 0x3bd82dbfda777df6 + .quad 0x71b177cc0b98369e + .quad 0x1d0e8463850c3699 + .quad 0x5a71945b48e2d1f1 + .quad 0x6f203dd5405b4b42 + .quad 0x327ec60410b24509 + .quad 0x9c347230ac2a8846 + .quad 0x77de29fc11ffeb6a + + // 2^244 * 5 * B + + .quad 0xb0ac57c983b778a8 + .quad 0x53cdcca9d7fe912c + .quad 0x61c2b854ff1f59dc + .quad 0x3a1a2cf0f0de7dac + .quad 0x835e138fecced2ca + .quad 0x8c9eaf13ea963b9a + .quad 0xc95fbfc0b2160ea6 + .quad 0x575e66f3ad877892 + .quad 0x99803a27c88fcb3a + .quad 0x345a6789275ec0b0 + .quad 0x459789d0ff6c2be5 + .quad 0x62f882651e70a8b2 + + // 2^244 * 6 * B + + .quad 0x085ae2c759ff1be4 + .quad 0x149145c93b0e40b7 + .quad 0xc467e7fa7ff27379 + .quad 0x4eeecf0ad5c73a95 + .quad 0x6d822986698a19e0 + .quad 0xdc9821e174d78a71 + .quad 0x41a85f31f6cb1f47 + .quad 0x352721c2bcda9c51 + .quad 0x48329952213fc985 + .quad 0x1087cf0d368a1746 + .quad 0x8e5261b166c15aa5 + .quad 0x2d5b2d842ed24c21 + + // 2^244 * 7 * B + + .quad 0x02cfebd9ebd3ded1 + .quad 0xd45b217739021974 + .quad 0x7576f813fe30a1b7 + .quad 0x5691b6f9a34ef6c2 + .quad 0x5eb7d13d196ac533 + .quad 0x377234ecdb80be2b + .quad 0xe144cffc7cf5ae24 + .quad 0x5226bcf9c441acec + .quad 0x79ee6c7223e5b547 + .quad 0x6f5f50768330d679 + .quad 0xed73e1e96d8adce9 + .quad 0x27c3da1e1d8ccc03 + + // 2^244 * 8 * B + + .quad 0x7eb9efb23fe24c74 + .quad 0x3e50f49f1651be01 + .quad 0x3ea732dc21858dea + .quad 0x17377bd75bb810f9 + .quad 0x28302e71630ef9f6 + .quad 0xc2d4a2032b64cee0 + .quad 0x090820304b6292be + .quad 0x5fca747aa82adf18 + .quad 0x232a03c35c258ea5 + .quad 0x86f23a2c6bcb0cf1 + .quad 0x3dad8d0d2e442166 + .quad 0x04a8933cab76862b + + // 2^248 * 1 * B + + .quad 0xd2c604b622943dff + .quad 0xbc8cbece44cfb3a0 + .quad 0x5d254ff397808678 + .quad 0x0fa3614f3b1ca6bf + .quad 0x69082b0e8c936a50 + .quad 0xf9c9a035c1dac5b6 + .quad 0x6fb73e54c4dfb634 + .quad 0x4005419b1d2bc140 + .quad 0xa003febdb9be82f0 + .quad 0x2089c1af3a44ac90 + .quad 0xf8499f911954fa8e + .quad 0x1fba218aef40ab42 + + // 2^248 * 2 * B + + .quad 0xab549448fac8f53e + .quad 0x81f6e89a7ba63741 + .quad 0x74fd6c7d6c2b5e01 + .quad 0x392e3acaa8c86e42 + .quad 0x4f3e57043e7b0194 + .quad 0xa81d3eee08daaf7f + .quad 0xc839c6ab99dcdef1 + .quad 0x6c535d13ff7761d5 + .quad 0x4cbd34e93e8a35af + .quad 0x2e0781445887e816 + .quad 0x19319c76f29ab0ab + .quad 0x25e17fe4d50ac13b + + // 2^248 * 3 * B + + .quad 0x0a289bd71e04f676 + .quad 0x208e1c52d6420f95 + .quad 0x5186d8b034691fab + .quad 0x255751442a9fb351 + .quad 0x915f7ff576f121a7 + .quad 0xc34a32272fcd87e3 + .quad 0xccba2fde4d1be526 + .quad 0x6bba828f8969899b + .quad 0xe2d1bc6690fe3901 + .quad 0x4cb54a18a0997ad5 + .quad 0x971d6914af8460d4 + .quad 0x559d504f7f6b7be4 + + // 2^248 * 4 * B + + .quad 0xa7738378b3eb54d5 + .quad 0x1d69d366a5553c7c + .quad 0x0a26cf62f92800ba + .quad 0x01ab12d5807e3217 + .quad 0x9c4891e7f6d266fd + .quad 0x0744a19b0307781b + .quad 0x88388f1d6061e23b + .quad 0x123ea6a3354bd50e + .quad 0x118d189041e32d96 + .quad 0xb9ede3c2d8315848 + .quad 0x1eab4271d83245d9 + .quad 0x4a3961e2c918a154 + + // 2^248 * 5 * B + + .quad 0x71dc3be0f8e6bba0 + .quad 0xd6cef8347effe30a + .quad 0xa992425fe13a476a + .quad 0x2cd6bce3fb1db763 + .quad 0x0327d644f3233f1e + .quad 0x499a260e34fcf016 + .quad 0x83b5a716f2dab979 + .quad 0x68aceead9bd4111f + .quad 0x38b4c90ef3d7c210 + .quad 0x308e6e24b7ad040c + .quad 0x3860d9f1b7e73e23 + .quad 0x595760d5b508f597 + + // 2^248 * 6 * B + + .quad 0x6129bfe104aa6397 + .quad 0x8f960008a4a7fccb + .quad 0x3f8bc0897d909458 + .quad 0x709fa43edcb291a9 + .quad 0x882acbebfd022790 + .quad 0x89af3305c4115760 + .quad 0x65f492e37d3473f4 + .quad 0x2cb2c5df54515a2b + .quad 0xeb0a5d8c63fd2aca + .quad 0xd22bc1662e694eff + .quad 0x2723f36ef8cbb03a + .quad 0x70f029ecf0c8131f + + // 2^248 * 7 * B + + .quad 0x461307b32eed3e33 + .quad 0xae042f33a45581e7 + .quad 0xc94449d3195f0366 + .quad 0x0b7d5d8a6c314858 + .quad 0x2a6aafaa5e10b0b9 + .quad 0x78f0a370ef041aa9 + .quad 0x773efb77aa3ad61f + .quad 0x44eca5a2a74bd9e1 + .quad 0x25d448327b95d543 + .quad 0x70d38300a3340f1d + .quad 0xde1c531c60e1c52b + .quad 0x272224512c7de9e4 + + // 2^248 * 8 * B + + .quad 0x1abc92af49c5342e + .quad 0xffeed811b2e6fad0 + .quad 0xefa28c8dfcc84e29 + .quad 0x11b5df18a44cc543 + .quad 0xbf7bbb8a42a975fc + .quad 0x8c5c397796ada358 + .quad 0xe27fc76fcdedaa48 + .quad 0x19735fd7f6bc20a6 + .quad 0xe3ab90d042c84266 + .quad 0xeb848e0f7f19547e + .quad 0x2503a1d065a497b9 + .quad 0x0fef911191df895f diff --git a/cbits/s2n/arm/p256_montjadd.S b/cbits/s2n/arm/p256_montjadd.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/p256_montjadd.S @@ -0,0 +1,3165 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Point addition on NIST curve P-256 in Montgomery-Jacobian coordinates +// +// extern void p256_montjadd(uint64_t p3[static 12], const uint64_t p1[static 12], +// const uint64_t p2[static 12]); +// +// Does p3 := p1 + p2 where all points are regarded as Jacobian triples with +// each coordinate in the Montgomery domain, i.e. x' = (2^256 * x) mod p_256. +// A Jacobian triple (x',y',z') represents affine point (x/z^2,y/z^3). +// +// Standard ARM ABI: X0 = p3, X1 = p1, X2 = p2 +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + +// This is functionally equivalent to p256_montjadd in unopt/p256_montjadd.S. +// This is the result of doing the following sequence of optimizations: +// 1. Function inlining +// 2. Eliminating redundant load/store instructions +// 3. Folding (add addr, const) + load/store +// Function inlining is done manually. The second and third optimizations are +// done by a script. + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_montjadd) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_montjadd) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_montjadd) + + .text + .balign 4 + +#define NUMSIZE 32 +#define NSPACE NUMSIZE*7 + +S2N_BN_SYMBOL(p256_montjadd): + CFI_START + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x26) + CFI_PUSH2(x27,x30) + CFI_DEC_SP(NSPACE) + + mov x21, x0 + mov x22, x1 + mov x23, x2 + mov x0, sp + ldr q19, [x22, #64] + ldp x9, x13, [x22, #64] + ldr q23, [x22, #80] + ldr q0, [x22, #64] + ldp x1, x10, [x22, #80] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x9, x13 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x9, x13 + umulh x15, x9, x1 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x9, x13 + mov x12, v1.d[0] + csetm x5, cc + cneg x6, x14, cc + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc + cinv x2, x5, cc + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x13, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x19, x3, x16, cs + csel x14, x8, x14, cs + csel x12, x11, x12, cs + csel x20, x5, x2, cs + stp x14, x12, [x0, #16] + stp x19, x20, [x0] + ldr q19, [x23, #64] + ldp x9, x13, [x23, #64] + ldr q23, [x23, #80] + ldr q0, [x23, #64] + ldp x1, x10, [x23, #80] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x9, x13 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x9, x13 + umulh x15, x9, x1 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x9, x13 + mov x12, v1.d[0] + csetm x5, cc + cneg x6, x14, cc + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc + cinv x2, x5, cc + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x13, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x16, x3, x16, cs + csel x14, x8, x14, cs + csel x12, x11, x12, cs + csel x2, x5, x2, cs + stp x14, x12, [sp, #176] + stp x16, x2, [sp, #160] + ldr q20, [x22, #32] + ldp x7, x17, [x23, #64] + ldr q0, [x23, #64] + ldp x6, x10, [x22, #32] + ldp x11, x15, [x23, #80] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc + cneg x13, x4, cc + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [x22, #48] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc + shl v17.2d, v27.2d, #32 + csetm x15, cc + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [x23, #80] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [x22, #48] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x24, x3, x13 + adcs x25, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x24 + adcs x15, x16, x25 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x11, x11, x13 + and x1, x1, x13 + adcs x4, x4, x1 + and x1, x12, x13 + stp x11, x4, [sp, #192] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #208] + ldr q20, [x23, #32] + ldp x7, x17, [x22, #64] + ldr q0, [x22, #64] + ldp x6, x10, [x23, #32] + ldp x11, x15, [x22, #80] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc + cneg x13, x4, cc + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [x23, #48] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc + shl v17.2d, v27.2d, #32 + csetm x15, cc + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [x22, #80] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [x23, #48] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x24, x3, x13 + adcs x25, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x24 + adcs x15, x16, x25 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x24, x11, x13 + and x1, x1, x13 + adcs x25, x4, x1 + and x1, x12, x13 + stp x24, x25, [sp, #32] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #48] + mov x1, sp + ldr q20, [x23, #0] + ldr q0, [x1] + ldp x6, x10, [x23, #0] + ldp x11, x15, [x1, #16] + rev64 v16.4s, v20.4s + subs x4, x19, x20 + csetm x3, cc + cneg x13, x4, cc + mul v16.4s, v16.4s, v0.4s + umulh x12, x20, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x19 + ldr q20, [x23, #16] + sbcs x5, x15, x20 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x19, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc + shl v17.2d, v27.2d, #32 + csetm x15, cc + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [x1, #16] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [x23, #16] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x19, x3, x13 + adcs x20, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x19 + adcs x15, x16, x20 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x11, x11, x13 + and x1, x1, x13 + adcs x4, x4, x1 + and x1, x12, x13 + stp x11, x4, [sp, #64] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #80] + ldr q20, [x22, #0] + ldp x7, x17, [sp, #160] + ldr q0, [sp, #160] + ldp x6, x10, [x22, #0] + ldp x11, x15, [sp, #176] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc + cneg x13, x4, cc + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [x22, #16] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc + shl v17.2d, v27.2d, #32 + csetm x15, cc + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [sp, #176] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [x22, #16] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x19, x3, x13 + adcs x20, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x19 + adcs x15, x16, x20 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x11, x11, x13 + and x1, x1, x13 + adcs x4, x4, x1 + and x1, x12, x13 + stp x11, x4, [sp, #128] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #144] + mov x1, sp + ldr q20, [sp, #32] + ldp x7, x17, [x1] + ldr q0, [x1] + ldp x11, x15, [x1, #16] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc + cneg x13, x4, cc + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x25 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [sp, #48] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x24 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc + shl v17.2d, v27.2d, #32 + csetm x15, cc + subs x9, x25, x24 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [x1, #16] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #48] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x24, x7 + sbcs x9, x25, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x19, x3, x13 + adcs x20, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x24, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x25, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x19 + adcs x15, x16, x20 + eor x5, x17, x4 + adcs x9, x1, x24 + eor x1, x10, x5 + adcs x16, x2, x25 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x19, x11, x13 + and x1, x1, x13 + adcs x20, x4, x1 + and x1, x12, x13 + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #48] + ldr q20, [sp, #192] + ldp x7, x17, [sp, #160] + ldr q0, [sp, #160] + ldp x6, x10, [sp, #192] + ldp x11, x15, [sp, #176] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc + cneg x13, x4, cc + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [sp, #208] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc + shl v17.2d, v27.2d, #32 + csetm x15, cc + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [sp, #176] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #208] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x24, x3, x13 + adcs x25, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x24 + adcs x15, x16, x25 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x9, x11, x13 + and x1, x1, x13 + adcs x10, x4, x1 + and x1, x12, x13 + stp x9, x10, [sp, #192] + adcs x11, x7, xzr + adc x12, x17, x1 + stp x11, x12, [sp, #208] + ldp x5, x6, [sp, #64] + ldp x4, x3, [sp, #128] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [sp, #144] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x13, x5, x3 + and x4, x3, #0xffffffff + adcs x24, x6, x4 + adcs x25, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x26, x8, x4 + stp x13, x24, [sp, #160] + stp x25, x26, [sp, #176] + subs x5, x19, x9 + sbcs x6, x20, x10 + ldp x7, x8, [sp, #48] + sbcs x7, x7, x11 + sbcs x8, x8, x12 + csetm x3, cc + adds x19, x5, x3 + and x4, x3, #0xffffffff + adcs x20, x6, x4 + adcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x8, x8, x4 + stp x19, x20, [sp, #32] + stp x7, x8, [sp, #48] + ldr q19, [sp, #160] + ldr q23, [sp, #176] + ldr q0, [sp, #160] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x13, x24 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x13, x24 + umulh x15, x13, x25 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x13, x24 + mov x12, v1.d[0] + csetm x5, cc + cneg x6, x14, cc + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x26, x25 + mov x9, v1.d[1] + cneg x17, x2, cc + cinv x2, x5, cc + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x24, x26 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x25, x26 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x25, x26 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x26, x26 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x26, x26 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x25, x25 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x25, x25 + adcs x2, x2, x9 + mov x6, #0xffffffff + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x24, x3, x16, cs + csel x25, x8, x14, cs + csel x26, x11, x12, cs + csel x27, x5, x2, cs + stp x25, x26, [sp, #112] + stp x24, x27, [sp, #96] + mov x0, sp + ldr q19, [sp, #32] + ldr q23, [sp, #48] + ldr q0, [sp, #32] + ldp x1, x10, [sp, #48] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x19, x20 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x19, x20 + umulh x15, x19, x1 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x19, x20 + mov x12, v1.d[0] + csetm x5, cc + cneg x6, x14, cc + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc + cinv x2, x5, cc + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x20, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x16, x3, x16, cs + csel x14, x8, x14, cs + csel x12, x11, x12, cs + csel x2, x5, x2, cs + stp x14, x12, [x0, #16] + stp x16, x2, [x0] + ldr q20, [sp, #128] + ldr q0, [sp, #96] + ldp x6, x10, [sp, #128] + rev64 v16.4s, v20.4s + subs x4, x24, x27 + csetm x3, cc + cneg x13, x4, cc + mul v16.4s, v16.4s, v0.4s + umulh x12, x27, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x25, x24 + ldr q20, [sp, #144] + sbcs x5, x26, x27 + ngc x17, xzr + subs x8, x25, x26 + uaddlp v27.2d, v16.4s + umulh x4, x24, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc + shl v17.2d, v27.2d, #32 + csetm x15, cc + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [sp, #112] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #144] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x19, x3, x13 + adcs x20, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x25, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x26, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x19 + adcs x15, x16, x20 + eor x5, x17, x4 + adcs x9, x1, x25 + eor x1, x10, x5 + adcs x16, x2, x26 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x19, x11, x13 + and x1, x1, x13 + adcs x20, x4, x1 + and x1, x12, x13 + stp x19, x20, [sp, #128] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #144] + ldr q20, [sp, #64] + ldr q0, [sp, #96] + ldp x6, x10, [sp, #64] + ldp x11, x15, [sp, #112] + rev64 v16.4s, v20.4s + subs x4, x24, x27 + csetm x3, cc + cneg x13, x4, cc + mul v16.4s, v16.4s, v0.4s + umulh x12, x27, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x24 + ldr q20, [sp, #80] + sbcs x5, x15, x27 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x24, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc + shl v17.2d, v27.2d, #32 + csetm x15, cc + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [sp, #112] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #80] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x24, x3, x13 + adcs x25, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x24 + adcs x15, x16, x25 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x9, x11, x13 + and x1, x1, x13 + adcs x10, x4, x1 + and x1, x12, x13 + stp x9, x10, [sp, #64] + adcs x11, x7, xzr + adc x12, x17, x1 + stp x11, x12, [sp, #80] + mov x0, sp + mov x1, sp + ldp x5, x6, [x1] + subs x5, x5, x19 + sbcs x6, x6, x20 + ldp x7, x8, [x1, #16] + ldp x4, x3, [sp, #144] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x24, x5, x3 + and x4, x3, #0xffffffff + adcs x25, x6, x4 + adcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x8, x8, x4 + stp x7, x8, [x0, #16] + subs x5, x9, x19 + sbcs x6, x10, x20 + ldp x4, x3, [sp, #144] + sbcs x7, x11, x4 + sbcs x8, x12, x3 + csetm x3, cc + adds x5, x5, x3 + and x4, x3, #0xffffffff + adcs x6, x6, x4 + adcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x8, x8, x4 + stp x5, x6, [sp, #96] + stp x7, x8, [sp, #112] + ldr q20, [x22, #64] + ldp x7, x17, [sp, #160] + ldr q0, [sp, #160] + ldp x6, x10, [x22, #64] + ldp x11, x15, [sp, #176] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc + cneg x13, x4, cc + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [x22, #80] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc + shl v17.2d, v27.2d, #32 + csetm x15, cc + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [sp, #176] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [x22, #80] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x19, x3, x13 + adcs x20, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x19 + adcs x15, x16, x20 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x11, x11, x13 + and x1, x1, x13 + adcs x4, x4, x1 + and x1, x12, x13 + stp x11, x4, [sp, #160] + adcs x19, x7, xzr + adc x20, x17, x1 + stp x19, x20, [sp, #176] + mov x0, sp + mov x1, sp + ldp x4, x3, [sp, #64] + subs x5, x24, x4 + sbcs x6, x25, x3 + ldp x7, x8, [x1, #16] + ldp x4, x3, [sp, #80] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x9, x5, x3 + and x4, x3, #0xffffffff + adcs x10, x6, x4 + adcs x11, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x3, x8, x4 + stp x9, x10, [x0] + stp x11, x3, [x0, #16] + ldp x5, x6, [sp, #128] + subs x5, x5, x9 + sbcs x6, x6, x10 + ldp x7, x8, [sp, #144] + sbcs x7, x7, x11 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + and x4, x3, #0xffffffff + adcs x6, x6, x4 + adcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x8, x8, x4 + stp x5, x6, [sp, #128] + stp x7, x8, [sp, #144] + ldr q20, [sp, #192] + ldp x7, x17, [sp, #96] + ldr q0, [sp, #96] + ldp x6, x10, [sp, #192] + ldp x11, x15, [sp, #112] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc + cneg x13, x4, cc + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [sp, #208] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc + shl v17.2d, v27.2d, #32 + csetm x15, cc + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [sp, #112] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #208] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x24, x3, x13 + adcs x25, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x24 + adcs x15, x16, x25 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x11, x11, x13 + and x1, x1, x13 + adcs x4, x4, x1 + and x1, x12, x13 + stp x11, x4, [sp, #96] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #112] + ldr q20, [x23, #64] + ldp x7, x17, [sp, #160] + ldr q0, [sp, #160] + ldp x6, x10, [x23, #64] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc + cneg x13, x4, cc + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x19, x7 + ldr q20, [x23, #80] + sbcs x5, x20, x17 + ngc x17, xzr + subs x8, x19, x20 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc + shl v17.2d, v27.2d, #32 + csetm x15, cc + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [sp, #176] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [x23, #80] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x19, x3, x13 + adcs x20, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x24, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x25, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x19 + adcs x15, x16, x20 + eor x5, x17, x4 + adcs x9, x1, x24 + eor x1, x10, x5 + adcs x16, x2, x25 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x19, x11, x13 + and x1, x1, x13 + adcs x20, x4, x1 + and x1, x12, x13 + stp x19, x20, [sp, #160] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #176] + ldr q20, [sp, #128] + ldp x7, x17, [sp, #32] + ldr q0, [sp, #32] + ldp x6, x10, [sp, #128] + ldp x11, x15, [sp, #48] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc + cneg x13, x4, cc + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [sp, #144] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc + shl v17.2d, v27.2d, #32 + csetm x15, cc + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [sp, #48] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #144] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x24, x3, x13 + adcs x25, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x24 + adcs x15, x16, x25 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x5, x11, x13 + and x1, x1, x13 + adcs x6, x4, x1 + and x1, x12, x13 + adcs x7, x7, xzr + adc x9, x17, x1 + ldp x4, x3, [sp, #96] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x4, x3, [sp, #112] + sbcs x7, x7, x4 + sbcs x8, x9, x3 + csetm x3, cc + adds x15, x5, x3 + and x4, x3, #0xffffffff + adcs x24, x6, x4 + adcs x25, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x26, x8, x4 + stp x15, x24, [sp, #128] + stp x25, x26, [sp, #144] + ldp x0, x1, [x22, #64] + ldp x2, x3, [x22, #80] + orr x12, x0, x1 + orr x13, x2, x3 + orr x12, x12, x13 + cmp x12, xzr + cset x12, ne + ldp x4, x5, [x23, #64] + ldp x6, x7, [x23, #80] + orr x13, x4, x5 + orr x14, x6, x7 + orr x13, x13, x14 + cmp x13, xzr + cset x13, ne + cmp x13, x12 + csel x8, x0, x19, cc + csel x9, x1, x20, cc + csel x8, x4, x8, hi + csel x9, x5, x9, hi + ldp x10, x11, [sp, #176] + csel x10, x2, x10, cc + csel x11, x3, x11, cc + csel x10, x6, x10, hi + csel x11, x7, x11, hi + ldp x12, x13, [x22] + ldp x0, x1, [sp] + csel x0, x12, x0, cc + csel x1, x13, x1, cc + ldp x12, x13, [x23] + csel x0, x12, x0, hi + csel x1, x13, x1, hi + ldp x12, x13, [x22, #16] + ldp x2, x3, [sp, #16] + csel x2, x12, x2, cc + csel x3, x13, x3, cc + ldp x12, x13, [x23, #16] + csel x2, x12, x2, hi + csel x3, x13, x3, hi + ldp x12, x13, [x22, #32] + csel x4, x12, x15, cc + csel x5, x13, x24, cc + ldp x12, x13, [x23, #32] + csel x4, x12, x4, hi + csel x5, x13, x5, hi + ldp x12, x13, [x22, #48] + csel x6, x12, x25, cc + csel x7, x13, x26, cc + ldp x12, x13, [x23, #48] + csel x6, x12, x6, hi + csel x7, x13, x7, hi + stp x0, x1, [x21] + stp x2, x3, [x21, #16] + stp x4, x5, [x21, #32] + stp x6, x7, [x21, #48] + stp x8, x9, [x21, #64] + stp x10, x11, [x21, #80] + + CFI_INC_SP(NSPACE) + CFI_POP2(x27,x30) + CFI_POP2(x25,x26) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_montjadd) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/p256_montjadd_alt.S b/cbits/s2n/arm/p256_montjadd_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/p256_montjadd_alt.S @@ -0,0 +1,555 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Point addition on NIST curve P-256 in Montgomery-Jacobian coordinates +// +// extern void p256_montjadd_alt(uint64_t p3[static 12], +// const uint64_t p1[static 12], +// const uint64_t p2[static 12]); +// +// Does p3 := p1 + p2 where all points are regarded as Jacobian triples with +// each coordinate in the Montgomery domain, i.e. x' = (2^256 * x) mod p_256. +// A Jacobian triple (x',y',z') represents affine point (x/z^2,y/z^3). +// +// Standard ARM ABI: X0 = p3, X1 = p1, X2 = p2 +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_montjadd_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_montjadd_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_montjadd_alt) + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Stable homes for input arguments during main code sequence + +#define input_z x15 +#define input_x x16 +#define input_y x17 + +// Pointer-offset pairs for inputs and outputs + +#define x_1 input_x, #0 +#define y_1 input_x, #NUMSIZE +#define z_1 input_x, #(2*NUMSIZE) + +#define x_2 input_y, #0 +#define y_2 input_y, #NUMSIZE +#define z_2 input_y, #(2*NUMSIZE) + +#define x_3 input_z, #0 +#define y_3 input_z, #NUMSIZE +#define z_3 input_z, #(2*NUMSIZE) + +// Pointer-offset pairs for temporaries, with some aliasing +// #NSPACE is the total stack needed for these temporaries + +#define z1sq sp, #(NUMSIZE*0) +#define ww sp, #(NUMSIZE*0) +#define resx sp, #(NUMSIZE*0) + +#define yd sp, #(NUMSIZE*1) +#define y2a sp, #(NUMSIZE*1) + +#define x2a sp, #(NUMSIZE*2) +#define zzx2 sp, #(NUMSIZE*2) + +#define zz sp, #(NUMSIZE*3) +#define t1 sp, #(NUMSIZE*3) + +#define t2 sp, #(NUMSIZE*4) +#define x1a sp, #(NUMSIZE*4) +#define zzx1 sp, #(NUMSIZE*4) +#define resy sp, #(NUMSIZE*4) + +#define xd sp, #(NUMSIZE*5) +#define z2sq sp, #(NUMSIZE*5) +#define resz sp, #(NUMSIZE*5) + +#define y1a sp, #(NUMSIZE*6) + +#define NSPACE NUMSIZE*7 + +// Corresponds to bignum_montmul_p256_alt except registers + +#define montmul_p256(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x7, x8, [P2] __LF \ + mul x12, x3, x7 __LF \ + umulh x13, x3, x7 __LF \ + mul x11, x3, x8 __LF \ + umulh x14, x3, x8 __LF \ + adds x13, x13, x11 __LF \ + ldp x9, x10, [P2+16] __LF \ + mul x11, x3, x9 __LF \ + umulh x0, x3, x9 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x3, x10 __LF \ + umulh x1, x3, x10 __LF \ + adcs x0, x0, x11 __LF \ + adc x1, x1, xzr __LF \ + ldp x5, x6, [P1+16] __LF \ + mul x11, x4, x7 __LF \ + adds x13, x13, x11 __LF \ + mul x11, x4, x8 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x4, x9 __LF \ + adcs x0, x0, x11 __LF \ + mul x11, x4, x10 __LF \ + adcs x1, x1, x11 __LF \ + umulh x3, x4, x10 __LF \ + adc x3, x3, xzr __LF \ + umulh x11, x4, x7 __LF \ + adds x14, x14, x11 __LF \ + umulh x11, x4, x8 __LF \ + adcs x0, x0, x11 __LF \ + umulh x11, x4, x9 __LF \ + adcs x1, x1, x11 __LF \ + adc x3, x3, xzr __LF \ + mul x11, x5, x7 __LF \ + adds x14, x14, x11 __LF \ + mul x11, x5, x8 __LF \ + adcs x0, x0, x11 __LF \ + mul x11, x5, x9 __LF \ + adcs x1, x1, x11 __LF \ + mul x11, x5, x10 __LF \ + adcs x3, x3, x11 __LF \ + umulh x4, x5, x10 __LF \ + adc x4, x4, xzr __LF \ + umulh x11, x5, x7 __LF \ + adds x0, x0, x11 __LF \ + umulh x11, x5, x8 __LF \ + adcs x1, x1, x11 __LF \ + umulh x11, x5, x9 __LF \ + adcs x3, x3, x11 __LF \ + adc x4, x4, xzr __LF \ + mul x11, x6, x7 __LF \ + adds x0, x0, x11 __LF \ + mul x11, x6, x8 __LF \ + adcs x1, x1, x11 __LF \ + mul x11, x6, x9 __LF \ + adcs x3, x3, x11 __LF \ + mul x11, x6, x10 __LF \ + adcs x4, x4, x11 __LF \ + umulh x5, x6, x10 __LF \ + adc x5, x5, xzr __LF \ + mov x10, #0xffffffff00000001 __LF \ + adds x13, x13, x12, lsl #32 __LF \ + lsr x11, x12, #32 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x12, x10 __LF \ + umulh x12, x12, x10 __LF \ + adcs x0, x0, x11 __LF \ + adc x12, x12, xzr __LF \ + umulh x11, x6, x7 __LF \ + adds x1, x1, x11 __LF \ + umulh x11, x6, x8 __LF \ + adcs x3, x3, x11 __LF \ + umulh x11, x6, x9 __LF \ + adcs x4, x4, x11 __LF \ + adc x5, x5, xzr __LF \ + adds x14, x14, x13, lsl #32 __LF \ + lsr x11, x13, #32 __LF \ + adcs x0, x0, x11 __LF \ + mul x11, x13, x10 __LF \ + umulh x13, x13, x10 __LF \ + adcs x12, x12, x11 __LF \ + adc x13, x13, xzr __LF \ + adds x0, x0, x14, lsl #32 __LF \ + lsr x11, x14, #32 __LF \ + adcs x12, x12, x11 __LF \ + mul x11, x14, x10 __LF \ + umulh x14, x14, x10 __LF \ + adcs x13, x13, x11 __LF \ + adc x14, x14, xzr __LF \ + adds x12, x12, x0, lsl #32 __LF \ + lsr x11, x0, #32 __LF \ + adcs x13, x13, x11 __LF \ + mul x11, x0, x10 __LF \ + umulh x0, x0, x10 __LF \ + adcs x14, x14, x11 __LF \ + adc x0, x0, xzr __LF \ + adds x12, x12, x1 __LF \ + adcs x13, x13, x3 __LF \ + adcs x14, x14, x4 __LF \ + adcs x0, x0, x5 __LF \ + cset x8, cs __LF \ + mov x11, #0xffffffff __LF \ + adds x1, x12, #0x1 __LF \ + sbcs x3, x13, x11 __LF \ + sbcs x4, x14, xzr __LF \ + sbcs x5, x0, x10 __LF \ + sbcs xzr, x8, xzr __LF \ + csel x12, x12, x1, cc __LF \ + csel x13, x13, x3, cc __LF \ + csel x14, x14, x4, cc __LF \ + csel x0, x0, x5, cc __LF \ + stp x12, x13, [P0] __LF \ + stp x14, x0, [P0+16] + +// Corresponds exactly to bignum_montsqr_p256_alt + +#define montsqr_p256(P0,P1) \ + ldp x2, x3, [P1] __LF \ + mul x9, x2, x3 __LF \ + umulh x10, x2, x3 __LF \ + ldp x4, x5, [P1+16] __LF \ + mul x11, x2, x5 __LF \ + umulh x12, x2, x5 __LF \ + mul x6, x2, x4 __LF \ + umulh x7, x2, x4 __LF \ + adds x10, x10, x6 __LF \ + adcs x11, x11, x7 __LF \ + mul x6, x3, x4 __LF \ + umulh x7, x3, x4 __LF \ + adc x7, x7, xzr __LF \ + adds x11, x11, x6 __LF \ + mul x13, x4, x5 __LF \ + umulh x14, x4, x5 __LF \ + adcs x12, x12, x7 __LF \ + mul x6, x3, x5 __LF \ + umulh x7, x3, x5 __LF \ + adc x7, x7, xzr __LF \ + adds x12, x12, x6 __LF \ + adcs x13, x13, x7 __LF \ + adc x14, x14, xzr __LF \ + adds x9, x9, x9 __LF \ + adcs x10, x10, x10 __LF \ + adcs x11, x11, x11 __LF \ + adcs x12, x12, x12 __LF \ + adcs x13, x13, x13 __LF \ + adcs x14, x14, x14 __LF \ + cset x7, cs __LF \ + umulh x6, x2, x2 __LF \ + mul x8, x2, x2 __LF \ + adds x9, x9, x6 __LF \ + mul x6, x3, x3 __LF \ + adcs x10, x10, x6 __LF \ + umulh x6, x3, x3 __LF \ + adcs x11, x11, x6 __LF \ + mul x6, x4, x4 __LF \ + adcs x12, x12, x6 __LF \ + umulh x6, x4, x4 __LF \ + adcs x13, x13, x6 __LF \ + mul x6, x5, x5 __LF \ + adcs x14, x14, x6 __LF \ + umulh x6, x5, x5 __LF \ + adc x7, x7, x6 __LF \ + adds x9, x9, x8, lsl #32 __LF \ + lsr x3, x8, #32 __LF \ + adcs x10, x10, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x8, x3 __LF \ + umulh x8, x8, x3 __LF \ + adcs x11, x11, x2 __LF \ + adc x8, x8, xzr __LF \ + adds x10, x10, x9, lsl #32 __LF \ + lsr x3, x9, #32 __LF \ + adcs x11, x11, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x9, x3 __LF \ + umulh x9, x9, x3 __LF \ + adcs x8, x8, x2 __LF \ + adc x9, x9, xzr __LF \ + adds x11, x11, x10, lsl #32 __LF \ + lsr x3, x10, #32 __LF \ + adcs x8, x8, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x10, x3 __LF \ + umulh x10, x10, x3 __LF \ + adcs x9, x9, x2 __LF \ + adc x10, x10, xzr __LF \ + adds x8, x8, x11, lsl #32 __LF \ + lsr x3, x11, #32 __LF \ + adcs x9, x9, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x11, x3 __LF \ + umulh x11, x11, x3 __LF \ + adcs x10, x10, x2 __LF \ + adc x11, x11, xzr __LF \ + adds x8, x8, x12 __LF \ + adcs x9, x9, x13 __LF \ + adcs x10, x10, x14 __LF \ + adcs x11, x11, x7 __LF \ + cset x2, cs __LF \ + mov x3, #0xffffffff __LF \ + mov x5, #0xffffffff00000001 __LF \ + adds x12, x8, #0x1 __LF \ + sbcs x13, x9, x3 __LF \ + sbcs x14, x10, xzr __LF \ + sbcs x7, x11, x5 __LF \ + sbcs xzr, x2, xzr __LF \ + csel x8, x8, x12, cc __LF \ + csel x9, x9, x13, cc __LF \ + csel x10, x10, x14, cc __LF \ + csel x11, x11, x7, cc __LF \ + stp x8, x9, [P0] __LF \ + stp x10, x11, [P0+16] + +// Almost-Montgomery variant which we use when an input to other muls +// with the other argument fully reduced (which is always safe). + +#define amontsqr_p256(P0,P1) \ + ldp x2, x3, [P1] __LF \ + mul x9, x2, x3 __LF \ + umulh x10, x2, x3 __LF \ + ldp x4, x5, [P1+16] __LF \ + mul x11, x2, x5 __LF \ + umulh x12, x2, x5 __LF \ + mul x6, x2, x4 __LF \ + umulh x7, x2, x4 __LF \ + adds x10, x10, x6 __LF \ + adcs x11, x11, x7 __LF \ + mul x6, x3, x4 __LF \ + umulh x7, x3, x4 __LF \ + adc x7, x7, xzr __LF \ + adds x11, x11, x6 __LF \ + mul x13, x4, x5 __LF \ + umulh x14, x4, x5 __LF \ + adcs x12, x12, x7 __LF \ + mul x6, x3, x5 __LF \ + umulh x7, x3, x5 __LF \ + adc x7, x7, xzr __LF \ + adds x12, x12, x6 __LF \ + adcs x13, x13, x7 __LF \ + adc x14, x14, xzr __LF \ + adds x9, x9, x9 __LF \ + adcs x10, x10, x10 __LF \ + adcs x11, x11, x11 __LF \ + adcs x12, x12, x12 __LF \ + adcs x13, x13, x13 __LF \ + adcs x14, x14, x14 __LF \ + cset x7, cs __LF \ + umulh x6, x2, x2 __LF \ + mul x8, x2, x2 __LF \ + adds x9, x9, x6 __LF \ + mul x6, x3, x3 __LF \ + adcs x10, x10, x6 __LF \ + umulh x6, x3, x3 __LF \ + adcs x11, x11, x6 __LF \ + mul x6, x4, x4 __LF \ + adcs x12, x12, x6 __LF \ + umulh x6, x4, x4 __LF \ + adcs x13, x13, x6 __LF \ + mul x6, x5, x5 __LF \ + adcs x14, x14, x6 __LF \ + umulh x6, x5, x5 __LF \ + adc x7, x7, x6 __LF \ + adds x9, x9, x8, lsl #32 __LF \ + lsr x3, x8, #32 __LF \ + adcs x10, x10, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x8, x3 __LF \ + umulh x8, x8, x3 __LF \ + adcs x11, x11, x2 __LF \ + adc x8, x8, xzr __LF \ + adds x10, x10, x9, lsl #32 __LF \ + lsr x3, x9, #32 __LF \ + adcs x11, x11, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x9, x3 __LF \ + umulh x9, x9, x3 __LF \ + adcs x8, x8, x2 __LF \ + adc x9, x9, xzr __LF \ + adds x11, x11, x10, lsl #32 __LF \ + lsr x3, x10, #32 __LF \ + adcs x8, x8, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x10, x3 __LF \ + umulh x10, x10, x3 __LF \ + adcs x9, x9, x2 __LF \ + adc x10, x10, xzr __LF \ + adds x8, x8, x11, lsl #32 __LF \ + lsr x3, x11, #32 __LF \ + adcs x9, x9, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x11, x3 __LF \ + umulh x11, x11, x3 __LF \ + adcs x10, x10, x2 __LF \ + adc x11, x11, xzr __LF \ + adds x8, x8, x12 __LF \ + adcs x9, x9, x13 __LF \ + adcs x10, x10, x14 __LF \ + adcs x11, x11, x7 __LF \ + mov x2, #0xffffffffffffffff __LF \ + csel x2, xzr, x2, cc __LF \ + mov x3, #0xffffffff __LF \ + csel x3, xzr, x3, cc __LF \ + mov x5, #0xffffffff00000001 __LF \ + csel x5, xzr, x5, cc __LF \ + subs x8, x8, x2 __LF \ + sbcs x9, x9, x3 __LF \ + sbcs x10, x10, xzr __LF \ + sbc x11, x11, x5 __LF \ + stp x8, x9, [P0] __LF \ + stp x10, x11, [P0+16] + +// Corresponds exactly to bignum_sub_p256 + +#define sub_p256(P0,P1,P2) \ + ldp x5, x6, [P1] __LF \ + ldp x4, x3, [P2] __LF \ + subs x5, x5, x4 __LF \ + sbcs x6, x6, x3 __LF \ + ldp x7, x8, [P1+16] __LF \ + ldp x4, x3, [P2+16] __LF \ + sbcs x7, x7, x4 __LF \ + sbcs x8, x8, x3 __LF \ + csetm x3, cc __LF \ + adds x5, x5, x3 __LF \ + mov x4, #0xffffffff __LF \ + and x4, x4, x3 __LF \ + adcs x6, x6, x4 __LF \ + adcs x7, x7, xzr __LF \ + mov x4, #0xffffffff00000001 __LF \ + and x4, x4, x3 __LF \ + adc x8, x8, x4 __LF \ + stp x5, x6, [P0] __LF \ + stp x7, x8, [P0+16] + +S2N_BN_SYMBOL(p256_montjadd_alt): + CFI_START + +// Make room on stack for temporary variables +// Move the input arguments to stable places + + CFI_DEC_SP(NSPACE) + + mov input_z, x0 + mov input_x, x1 + mov input_y, x2 + +// Main code, just a sequence of basic field operations +// 12 * multiply + 4 * square + 7 * subtract + + amontsqr_p256(z1sq,z_1) + amontsqr_p256(z2sq,z_2) + + montmul_p256(y1a,z_2,y_1) + montmul_p256(y2a,z_1,y_2) + + montmul_p256(x2a,z1sq,x_2) + montmul_p256(x1a,z2sq,x_1) + montmul_p256(y2a,z1sq,y2a) + montmul_p256(y1a,z2sq,y1a) + + sub_p256(xd,x2a,x1a) + sub_p256(yd,y2a,y1a) + + amontsqr_p256(zz,xd) + montsqr_p256(ww,yd) + + montmul_p256(zzx1,zz,x1a) + montmul_p256(zzx2,zz,x2a) + + sub_p256(resx,ww,zzx1) + sub_p256(t1,zzx2,zzx1) + + montmul_p256(xd,xd,z_1) + + sub_p256(resx,resx,zzx2) + + sub_p256(t2,zzx1,resx) + + montmul_p256(t1,t1,y1a) + montmul_p256(resz,xd,z_2) + montmul_p256(t2,yd,t2) + + sub_p256(resy,t2,t1) + +// Load in the z coordinates of the inputs to check for P1 = 0 and P2 = 0 +// The condition codes get set by a comparison (P2 != 0) - (P1 != 0) +// So "HI" <=> CF /\ ~ZF <=> P1 = 0 /\ ~(P2 = 0) +// and "LO" <=> ~CF <=> ~(P1 = 0) /\ P2 = 0 + + ldp x0, x1, [z_1] + ldp x2, x3, [z_1+16] + + orr x12, x0, x1 + orr x13, x2, x3 + orr x12, x12, x13 + cmp x12, xzr + cset x12, ne + + ldp x4, x5, [z_2] + ldp x6, x7, [z_2+16] + + orr x13, x4, x5 + orr x14, x6, x7 + orr x13, x13, x14 + cmp x13, xzr + cset x13, ne + + cmp x13, x12 + +// Multiplex the outputs accordingly, re-using the z's in registers + + ldp x8, x9, [resz] + csel x8, x0, x8, lo + csel x9, x1, x9, lo + csel x8, x4, x8, hi + csel x9, x5, x9, hi + ldp x10, x11, [resz+16] + csel x10, x2, x10, lo + csel x11, x3, x11, lo + csel x10, x6, x10, hi + csel x11, x7, x11, hi + + ldp x12, x13, [x_1] + ldp x0, x1, [resx] + csel x0, x12, x0, lo + csel x1, x13, x1, lo + ldp x12, x13, [x_2] + csel x0, x12, x0, hi + csel x1, x13, x1, hi + + ldp x12, x13, [x_1+16] + ldp x2, x3, [resx+16] + csel x2, x12, x2, lo + csel x3, x13, x3, lo + ldp x12, x13, [x_2+16] + csel x2, x12, x2, hi + csel x3, x13, x3, hi + + ldp x12, x13, [y_1] + ldp x4, x5, [resy] + csel x4, x12, x4, lo + csel x5, x13, x5, lo + ldp x12, x13, [y_2] + csel x4, x12, x4, hi + csel x5, x13, x5, hi + + ldp x12, x13, [y_1+16] + ldp x6, x7, [resy+16] + csel x6, x12, x6, lo + csel x7, x13, x7, lo + ldp x12, x13, [y_2+16] + csel x6, x12, x6, hi + csel x7, x13, x7, hi + +// Finally store back the multiplexed values + + stp x0, x1, [x_3] + stp x2, x3, [x_3+16] + stp x4, x5, [y_3] + stp x6, x7, [y_3+16] + stp x8, x9, [z_3] + stp x10, x11, [z_3+16] + +// Restore registers and return + + CFI_INC_SP(NSPACE) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_montjadd_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/p256_montjdouble.S b/cbits/s2n/arm/p256_montjdouble.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/p256_montjdouble.S @@ -0,0 +1,1555 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Point doubling on NIST curve P-256 in Montgomery-Jacobian coordinates +// +// extern void p256_montjdouble(uint64_t p3[static 12], +// const uint64_t p1[static 12]); +// +// Does p3 := 2 * p1 where all points are regarded as Jacobian triples with +// each coordinate in the Montgomery domain, i.e. x' = (2^256 * x) mod p_256. +// A Jacobian triple (x',y',z') represents affine point (x/z^2,y/z^3). +// +// Standard ARM ABI: X0 = p3, X1 = p1 +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + +// This is functionally equivalent to p256_montjdouble in unopt/p256_montjdouble.S. +// This is the result of doing the following sequence of optimizations: +// 1. Function inlining +// 2. Eliminating redundant load/store instructions +// 3. Folding (add addr, const) + load/store +// Function inlining is done manually. The second and third optimizations are +// done by a script. + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_montjdouble) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_montjdouble) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_montjdouble) + .text + .balign 4 + +#define NUMSIZE 32 +#define NSPACE NUMSIZE*6 + +S2N_BN_SYMBOL(p256_montjdouble): + CFI_START + + CFI_DEC_SP(NSPACE+80) + CFI_STACKSAVE2(x19,x20,NSPACE) + CFI_STACKSAVE2(x21,x22,NSPACE+16) + CFI_STACKSAVE2(x23,x24,NSPACE+32) + CFI_STACKSAVE2(x25,x26,NSPACE+48) + CFI_STACKSAVE1Z(x27,NSPACE+64) + + mov x19, x0 + mov x20, x1 + mov x0, sp + ldr q19, [x20, #64] + ldp x9, x13, [x20, #64] + ldr q23, [x20, #80] + ldr q0, [x20, #64] + ldp x1, x10, [x20, #80] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x9, x13 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x9, x13 + umulh x15, x9, x1 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x9, x13 + mov x12, v1.d[0] + csetm x5, cc + cneg x6, x14, cc + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc + cinv x2, x5, cc + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x13, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x21, x3, x16, cs + csel x22, x8, x14, cs + csel x23, x11, x12, cs + csel x24, x5, x2, cs + stp x22, x23, [x0, #16] + stp x21, x24, [x0] + ldr q19, [x20, #32] + ldp x9, x13, [x20, #32] + ldr q23, [x20, #48] + ldr q0, [x20, #32] + ldp x1, x10, [x20, #48] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x9, x13 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x9, x13 + umulh x15, x9, x1 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x9, x13 + mov x12, v1.d[0] + csetm x5, cc + cneg x6, x14, cc + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc + cinv x2, x5, cc + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x13, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x16, x3, x16, cs + csel x14, x8, x14, cs + csel x12, x11, x12, cs + csel x2, x5, x2, cs + stp x14, x12, [sp, #48] + stp x16, x2, [sp, #32] + ldp x5, x6, [x20, #0] + subs x5, x5, x21 + sbcs x6, x6, x24 + ldp x7, x8, [x20, #16] + sbcs x7, x7, x22 + sbcs x8, x8, x23 + csetm x3, cc + adds x10, x5, x3 + and x4, x3, #0xffffffff + adcs x25, x6, x4 + adcs x26, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x27, x8, x4 + stp x10, x25, [sp, #96] + stp x26, x27, [sp, #112] + ldp x5, x6, [x20] + adds x5, x5, x21 + adcs x6, x6, x24 + ldp x7, x8, [x20, #16] + adcs x7, x7, x22 + adcs x8, x8, x23 + csetm x3, cs + subs x9, x5, x3 + and x1, x3, #0xffffffff + sbcs x5, x6, x1 + sbcs x7, x7, xzr + and x2, x3, #0xffffffff00000001 + sbc x8, x8, x2 + stp x9, x5, [sp, #64] + stp x7, x8, [sp, #80] + ldr q20, [sp, #96] + ldr q0, [sp, #64] + rev64 v16.4s, v20.4s + subs x4, x9, x5 + csetm x3, cc + cneg x13, x4, cc + mul v16.4s, v16.4s, v0.4s + umulh x12, x5, x25 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x7, x9 + ldr q20, [sp, #112] + sbcs x5, x8, x5 + ngc x17, xzr + subs x8, x7, x8 + uaddlp v27.2d, v16.4s + umulh x4, x9, x10 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc + shl v17.2d, v27.2d, #32 + csetm x15, cc + subs x9, x25, x10 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [sp, #80] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x10, x26 + sbcs x9, x25, x27 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x27, x26 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x21, x3, x13 + adcs x22, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x23, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x24, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x21 + adcs x15, x16, x22 + eor x5, x17, x4 + adcs x9, x1, x23 + eor x1, x10, x5 + adcs x16, x2, x24 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x21, x11, x13 + and x1, x1, x13 + adcs x22, x4, x1 + and x1, x12, x13 + stp x21, x22, [sp, #96] + adcs x23, x7, xzr + adc x24, x17, x1 + stp x23, x24, [sp, #112] + ldp x4, x5, [x20, #32] + ldp x8, x9, [x20, #64] + adds x4, x4, x8 + adcs x5, x5, x9 + ldp x6, x7, [x20, #48] + ldp x10, x11, [x20, #80] + adcs x6, x6, x10 + adcs x7, x7, x11 + adc x3, xzr, xzr + adds x8, x4, #0x1 + mov x9, #0xffffffff + sbcs x9, x5, x9 + sbcs x10, x6, xzr + mov x11, #0xffffffff00000001 + sbcs x11, x7, x11 + sbcs x3, x3, xzr + csel x4, x4, x8, cc + csel x5, x5, x9, cc + csel x6, x6, x10, cc + csel x7, x7, x11, cc + stp x4, x5, [sp, #64] + stp x6, x7, [sp, #80] + ldr q20, [sp, #32] + ldp x7, x17, [x20, #0] + ldr q0, [x20, #0] + ldp x6, x10, [sp, #32] + ldp x11, x15, [x20, #16] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc + cneg x13, x4, cc + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [sp, #48] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc + shl v17.2d, v27.2d, #32 + csetm x15, cc + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [x20, #16] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #48] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x20, x3, x13 + adcs x25, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x20 + adcs x15, x16, x25 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x20, x11, x13 + and x1, x1, x13 + adcs x25, x4, x1 + and x1, x12, x13 + stp x20, x25, [sp, #128] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #144] + ldr q19, [sp, #96] + ldr q23, [sp, #112] + ldr q0, [sp, #96] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x21, x22 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x21, x22 + umulh x15, x21, x23 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x21, x22 + mov x12, v1.d[0] + csetm x5, cc + cneg x6, x14, cc + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x24, x23 + mov x9, v1.d[1] + cneg x17, x2, cc + cinv x2, x5, cc + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x22, x24 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x23, x24 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x23, x24 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x24, x24 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x24, x24 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x23, x23 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x23, x23 + adcs x2, x2, x9 + mov x6, #0xffffffff + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x21, x3, x16, cs + csel x22, x8, x14, cs + csel x23, x11, x12, cs + csel x24, x5, x2, cs + ldr q19, [sp, #64] + ldp x9, x13, [sp, #64] + ldr q23, [sp, #80] + ldr q0, [sp, #64] + ldp x1, x10, [sp, #80] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x9, x13 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x9, x13 + umulh x15, x9, x1 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x9, x13 + mov x12, v1.d[0] + csetm x5, cc + cneg x6, x14, cc + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc + cinv x2, x5, cc + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x13, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x13, x3, x16, cs + csel x14, x8, x14, cs + csel x15, x11, x12, cs + csel x26, x5, x2, cs + mov x1, #0x9 + mov x2, #0xffffffffffffffff + subs x9, x2, x21 + mov x2, #0xffffffff + sbcs x10, x2, x24 + ngcs x11, x22 + mov x2, #0xffffffff00000001 + sbc x12, x2, x23 + mul x3, x1, x9 + mul x4, x1, x10 + mul x5, x1, x11 + mul x6, x1, x12 + umulh x9, x1, x9 + umulh x10, x1, x10 + umulh x11, x1, x11 + umulh x7, x1, x12 + adds x4, x4, x9 + adcs x5, x5, x10 + adcs x6, x6, x11 + adc x7, x7, xzr + mov x1, #0xc + mul x8, x20, x1 + umulh x9, x20, x1 + adds x3, x3, x8 + mul x8, x25, x1 + umulh x10, x25, x1 + adcs x4, x4, x8 + ldp x11, x12, [sp, #144] + mul x8, x11, x1 + umulh x11, x11, x1 + adcs x5, x5, x8 + mul x8, x12, x1 + umulh x12, x12, x1 + adcs x6, x6, x8 + adc x7, x7, xzr + adds x4, x4, x9 + adcs x5, x5, x10 + adcs x6, x6, x11 + adc x7, x7, x12 + add x8, x7, #0x1 + lsl x10, x8, #32 + adds x6, x6, x10 + adc x7, x7, xzr + neg x9, x8 + sub x10, x10, #0x1 + subs x3, x3, x9 + sbcs x4, x4, x10 + sbcs x5, x5, xzr + sbcs x6, x6, x8 + sbc x8, x7, x8 + adds x20, x3, x8 + and x9, x8, #0xffffffff + adcs x21, x4, x9 + adcs x22, x5, xzr + neg x10, x9 + adc x23, x6, x10 + stp x20, x21, [sp, #160] + stp x22, x23, [sp, #176] + mov x2, sp + ldp x4, x3, [x2] + subs x5, x13, x4 + sbcs x6, x26, x3 + ldp x4, x3, [x2, #16] + sbcs x7, x14, x4 + sbcs x8, x15, x3 + csetm x3, cc + adds x5, x5, x3 + and x4, x3, #0xffffffff + adcs x6, x6, x4 + adcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x8, x8, x4 + stp x5, x6, [sp, #64] + stp x7, x8, [sp, #80] + mov x0, sp + ldr q19, [sp, #32] + ldp x9, x13, [sp, #32] + ldr q23, [sp, #48] + ldr q0, [sp, #32] + ldp x1, x10, [sp, #48] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x9, x13 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x9, x13 + umulh x15, x9, x1 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x9, x13 + mov x12, v1.d[0] + csetm x5, cc + cneg x6, x14, cc + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc + cinv x2, x5, cc + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x13, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x24, x3, x16, cs + csel x25, x8, x14, cs + csel x26, x11, x12, cs + csel x27, x5, x2, cs + stp x25, x26, [x0, #16] + stp x24, x27, [x0] + ldr q20, [sp, #96] + ldr q0, [sp, #160] + ldp x6, x10, [sp, #96] + rev64 v16.4s, v20.4s + subs x4, x20, x21 + csetm x3, cc + cneg x13, x4, cc + mul v16.4s, v16.4s, v0.4s + umulh x12, x21, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x22, x20 + ldr q20, [sp, #112] + sbcs x5, x23, x21 + ngc x17, xzr + subs x8, x22, x23 + uaddlp v27.2d, v16.4s + umulh x4, x20, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc + shl v17.2d, v27.2d, #32 + csetm x15, cc + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [sp, #176] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #112] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x20, x3, x13 + adcs x21, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x22, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x23, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x20 + adcs x15, x16, x21 + eor x5, x17, x4 + adcs x9, x1, x22 + eor x1, x10, x5 + adcs x16, x2, x23 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x14, x11, x13 + and x1, x1, x13 + adcs x15, x4, x1 + and x1, x12, x13 + stp x14, x15, [sp, #96] + adcs x13, x7, xzr + adc x20, x17, x1 + stp x13, x20, [sp, #112] + ldp x5, x6, [sp, #64] + ldp x4, x3, [sp, #32] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [sp, #48] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + and x4, x3, #0xffffffff + adcs x6, x6, x4 + adcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x8, x8, x4 + stp x5, x6, [x19, #64] + stp x7, x8, [x19, #80] + ldp x1, x2, [sp, #128] + lsl x0, x1, #2 + ldp x6, x7, [sp, #160] + subs x0, x0, x6 + extr x1, x2, x1, #62 + sbcs x1, x1, x7 + ldp x3, x4, [sp, #144] + extr x2, x3, x2, #62 + ldp x6, x7, [sp, #176] + sbcs x2, x2, x6 + extr x3, x4, x3, #62 + sbcs x3, x3, x7 + lsr x4, x4, #62 + sbc x4, x4, xzr + add x5, x4, #0x1 + lsl x8, x5, #32 + negs x6, x8 + ngcs x7, xzr + sbc x8, x8, x5 + adds x0, x0, x5 + adcs x1, x1, x6 + adcs x2, x2, x7 + adcs x3, x3, x8 + csetm x5, cc + adds x0, x0, x5 + and x6, x5, #0xffffffff + adcs x1, x1, x6 + adcs x2, x2, xzr + neg x7, x6 + adc x3, x3, x7 + stp x0, x1, [x19] + stp x2, x3, [x19, #16] + mov x2, #0xffffffffffffffff + subs x9, x2, x24 + mov x2, #0xffffffff + sbcs x10, x2, x27 + ngcs x11, x25 + mov x2, #0xffffffff00000001 + sbc x12, x2, x26 + lsl x3, x9, #3 + extr x4, x10, x9, #61 + extr x5, x11, x10, #61 + extr x6, x12, x11, #61 + lsr x7, x12, #61 + mov x1, #0x3 + mul x8, x14, x1 + umulh x9, x14, x1 + adds x3, x3, x8 + mul x8, x15, x1 + umulh x10, x15, x1 + adcs x4, x4, x8 + mul x8, x13, x1 + umulh x11, x13, x1 + adcs x5, x5, x8 + mul x8, x20, x1 + umulh x12, x20, x1 + adcs x6, x6, x8 + adc x7, x7, xzr + adds x4, x4, x9 + adcs x5, x5, x10 + adcs x6, x6, x11 + adc x7, x7, x12 + add x8, x7, #0x1 + lsl x10, x8, #32 + adds x6, x6, x10 + adc x7, x7, xzr + neg x9, x8 + sub x10, x10, #0x1 + subs x3, x3, x9 + sbcs x4, x4, x10 + sbcs x5, x5, xzr + sbcs x6, x6, x8 + sbc x8, x7, x8 + adds x3, x3, x8 + and x9, x8, #0xffffffff + adcs x4, x4, x9 + adcs x5, x5, xzr + neg x10, x9 + adc x6, x6, x10 + stp x3, x4, [x19, #32] + stp x5, x6, [x19, #48] + + CFI_STACKLOAD1Z(x27,NSPACE+64) + CFI_STACKLOAD2(x25,x26,NSPACE+48) + CFI_STACKLOAD2(x23,x24,NSPACE+32) + CFI_STACKLOAD2(x21,x22,NSPACE+16) + CFI_STACKLOAD2(x19,x20,NSPACE) + CFI_INC_SP((NSPACE+80)) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_montjdouble) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/p256_montjdouble_alt.S b/cbits/s2n/arm/p256_montjdouble_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/p256_montjdouble_alt.S @@ -0,0 +1,587 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Point doubling on NIST curve P-256 in Montgomery-Jacobian coordinates +// +// extern void p256_montjdouble_alt(uint64_t p3[static 12], +// const uint64_t p1[static 12]); +// +// Does p3 := 2 * p1 where all points are regarded as Jacobian triples with +// each coordinate in the Montgomery domain, i.e. x' = (2^256 * x) mod p_256. +// A Jacobian triple (x',y',z') represents affine point (x/z^2,y/z^3). +// +// Standard ARM ABI: X0 = p3, X1 = p1 +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_montjdouble_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_montjdouble_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_montjdouble_alt) + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Stable homes for input arguments during main code sequence + +#define input_z x15 +#define input_x x16 + +// Pointer-offset pairs for inputs and outputs + +#define x_1 input_x, #0 +#define y_1 input_x, #NUMSIZE +#define z_1 input_x, #(2*NUMSIZE) + +#define x_3 input_z, #0 +#define y_3 input_z, #NUMSIZE +#define z_3 input_z, #(2*NUMSIZE) + +// Pointer-offset pairs for temporaries, with some aliasing +// #NSPACE is the total stack needed for these temporaries + +#define z2 sp, #(NUMSIZE*0) +#define y4 sp, #(NUMSIZE*0) + +#define y2 sp, #(NUMSIZE*1) + +#define t1 sp, #(NUMSIZE*2) + +#define t2 sp, #(NUMSIZE*3) +#define x2p sp, #(NUMSIZE*3) +#define dx2 sp, #(NUMSIZE*3) + +#define xy2 sp, #(NUMSIZE*4) + +#define x4p sp, #(NUMSIZE*5) +#define d sp, #(NUMSIZE*5) + +#define NSPACE NUMSIZE*6 + +// Corresponds exactly to bignum_montmul_p256_alt except registers + +#define montmul_p256(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x7, x8, [P2] __LF \ + mul x12, x3, x7 __LF \ + umulh x13, x3, x7 __LF \ + mul x11, x3, x8 __LF \ + umulh x14, x3, x8 __LF \ + adds x13, x13, x11 __LF \ + ldp x9, x10, [P2+16] __LF \ + mul x11, x3, x9 __LF \ + umulh x0, x3, x9 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x3, x10 __LF \ + umulh x1, x3, x10 __LF \ + adcs x0, x0, x11 __LF \ + adc x1, x1, xzr __LF \ + ldp x5, x6, [P1+16] __LF \ + mul x11, x4, x7 __LF \ + adds x13, x13, x11 __LF \ + mul x11, x4, x8 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x4, x9 __LF \ + adcs x0, x0, x11 __LF \ + mul x11, x4, x10 __LF \ + adcs x1, x1, x11 __LF \ + umulh x3, x4, x10 __LF \ + adc x3, x3, xzr __LF \ + umulh x11, x4, x7 __LF \ + adds x14, x14, x11 __LF \ + umulh x11, x4, x8 __LF \ + adcs x0, x0, x11 __LF \ + umulh x11, x4, x9 __LF \ + adcs x1, x1, x11 __LF \ + adc x3, x3, xzr __LF \ + mul x11, x5, x7 __LF \ + adds x14, x14, x11 __LF \ + mul x11, x5, x8 __LF \ + adcs x0, x0, x11 __LF \ + mul x11, x5, x9 __LF \ + adcs x1, x1, x11 __LF \ + mul x11, x5, x10 __LF \ + adcs x3, x3, x11 __LF \ + umulh x4, x5, x10 __LF \ + adc x4, x4, xzr __LF \ + umulh x11, x5, x7 __LF \ + adds x0, x0, x11 __LF \ + umulh x11, x5, x8 __LF \ + adcs x1, x1, x11 __LF \ + umulh x11, x5, x9 __LF \ + adcs x3, x3, x11 __LF \ + adc x4, x4, xzr __LF \ + mul x11, x6, x7 __LF \ + adds x0, x0, x11 __LF \ + mul x11, x6, x8 __LF \ + adcs x1, x1, x11 __LF \ + mul x11, x6, x9 __LF \ + adcs x3, x3, x11 __LF \ + mul x11, x6, x10 __LF \ + adcs x4, x4, x11 __LF \ + umulh x5, x6, x10 __LF \ + adc x5, x5, xzr __LF \ + mov x10, #0xffffffff00000001 __LF \ + adds x13, x13, x12, lsl #32 __LF \ + lsr x11, x12, #32 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x12, x10 __LF \ + umulh x12, x12, x10 __LF \ + adcs x0, x0, x11 __LF \ + adc x12, x12, xzr __LF \ + umulh x11, x6, x7 __LF \ + adds x1, x1, x11 __LF \ + umulh x11, x6, x8 __LF \ + adcs x3, x3, x11 __LF \ + umulh x11, x6, x9 __LF \ + adcs x4, x4, x11 __LF \ + adc x5, x5, xzr __LF \ + adds x14, x14, x13, lsl #32 __LF \ + lsr x11, x13, #32 __LF \ + adcs x0, x0, x11 __LF \ + mul x11, x13, x10 __LF \ + umulh x13, x13, x10 __LF \ + adcs x12, x12, x11 __LF \ + adc x13, x13, xzr __LF \ + adds x0, x0, x14, lsl #32 __LF \ + lsr x11, x14, #32 __LF \ + adcs x12, x12, x11 __LF \ + mul x11, x14, x10 __LF \ + umulh x14, x14, x10 __LF \ + adcs x13, x13, x11 __LF \ + adc x14, x14, xzr __LF \ + adds x12, x12, x0, lsl #32 __LF \ + lsr x11, x0, #32 __LF \ + adcs x13, x13, x11 __LF \ + mul x11, x0, x10 __LF \ + umulh x0, x0, x10 __LF \ + adcs x14, x14, x11 __LF \ + adc x0, x0, xzr __LF \ + adds x12, x12, x1 __LF \ + adcs x13, x13, x3 __LF \ + adcs x14, x14, x4 __LF \ + adcs x0, x0, x5 __LF \ + cset x8, cs __LF \ + mov x11, #0xffffffff __LF \ + adds x1, x12, #0x1 __LF \ + sbcs x3, x13, x11 __LF \ + sbcs x4, x14, xzr __LF \ + sbcs x5, x0, x10 __LF \ + sbcs xzr, x8, xzr __LF \ + csel x12, x12, x1, cc __LF \ + csel x13, x13, x3, cc __LF \ + csel x14, x14, x4, cc __LF \ + csel x0, x0, x5, cc __LF \ + stp x12, x13, [P0] __LF \ + stp x14, x0, [P0+16] + +// Corresponds exactly to bignum_montsqr_p256_alt + +#define montsqr_p256(P0,P1) \ + ldp x2, x3, [P1] __LF \ + mul x9, x2, x3 __LF \ + umulh x10, x2, x3 __LF \ + ldp x4, x5, [P1+16] __LF \ + mul x11, x2, x5 __LF \ + umulh x12, x2, x5 __LF \ + mul x6, x2, x4 __LF \ + umulh x7, x2, x4 __LF \ + adds x10, x10, x6 __LF \ + adcs x11, x11, x7 __LF \ + mul x6, x3, x4 __LF \ + umulh x7, x3, x4 __LF \ + adc x7, x7, xzr __LF \ + adds x11, x11, x6 __LF \ + mul x13, x4, x5 __LF \ + umulh x14, x4, x5 __LF \ + adcs x12, x12, x7 __LF \ + mul x6, x3, x5 __LF \ + umulh x7, x3, x5 __LF \ + adc x7, x7, xzr __LF \ + adds x12, x12, x6 __LF \ + adcs x13, x13, x7 __LF \ + adc x14, x14, xzr __LF \ + adds x9, x9, x9 __LF \ + adcs x10, x10, x10 __LF \ + adcs x11, x11, x11 __LF \ + adcs x12, x12, x12 __LF \ + adcs x13, x13, x13 __LF \ + adcs x14, x14, x14 __LF \ + cset x7, hs __LF \ + umulh x6, x2, x2 __LF \ + mul x8, x2, x2 __LF \ + adds x9, x9, x6 __LF \ + mul x6, x3, x3 __LF \ + adcs x10, x10, x6 __LF \ + umulh x6, x3, x3 __LF \ + adcs x11, x11, x6 __LF \ + mul x6, x4, x4 __LF \ + adcs x12, x12, x6 __LF \ + umulh x6, x4, x4 __LF \ + adcs x13, x13, x6 __LF \ + mul x6, x5, x5 __LF \ + adcs x14, x14, x6 __LF \ + umulh x6, x5, x5 __LF \ + adc x7, x7, x6 __LF \ + mov x5, #-4294967295 __LF \ + adds x9, x9, x8, lsl #32 __LF \ + lsr x3, x8, #32 __LF \ + adcs x10, x10, x3 __LF \ + mul x2, x8, x5 __LF \ + umulh x8, x8, x5 __LF \ + adcs x11, x11, x2 __LF \ + adc x8, x8, xzr __LF \ + adds x10, x10, x9, lsl #32 __LF \ + lsr x3, x9, #32 __LF \ + adcs x11, x11, x3 __LF \ + mul x2, x9, x5 __LF \ + umulh x9, x9, x5 __LF \ + adcs x8, x8, x2 __LF \ + adc x9, x9, xzr __LF \ + adds x11, x11, x10, lsl #32 __LF \ + lsr x3, x10, #32 __LF \ + adcs x8, x8, x3 __LF \ + mul x2, x10, x5 __LF \ + umulh x10, x10, x5 __LF \ + adcs x9, x9, x2 __LF \ + adc x10, x10, xzr __LF \ + adds x8, x8, x11, lsl #32 __LF \ + lsr x3, x11, #32 __LF \ + adcs x9, x9, x3 __LF \ + mul x2, x11, x5 __LF \ + umulh x11, x11, x5 __LF \ + adcs x10, x10, x2 __LF \ + adc x11, x11, xzr __LF \ + adds x8, x8, x12 __LF \ + adcs x9, x9, x13 __LF \ + adcs x10, x10, x14 __LF \ + adcs x11, x11, x7 __LF \ + cset x2, hs __LF \ + mov x3, #4294967295 __LF \ + adds x12, x8, #1 __LF \ + sbcs x13, x9, x3 __LF \ + sbcs x14, x10, xzr __LF \ + sbcs x7, x11, x5 __LF \ + sbcs xzr, x2, xzr __LF \ + csel x8, x8, x12, lo __LF \ + csel x9, x9, x13, lo __LF \ + csel x10, x10, x14, lo __LF \ + csel x11, x11, x7, lo __LF \ + stp x8, x9, [P0] __LF \ + stp x10, x11, [P0+16] + +// Corresponds exactly to bignum_sub_p256 + +#define sub_p256(P0,P1,P2) \ + ldp x5, x6, [P1] __LF \ + ldp x4, x3, [P2] __LF \ + subs x5, x5, x4 __LF \ + sbcs x6, x6, x3 __LF \ + ldp x7, x8, [P1+16] __LF \ + ldp x4, x3, [P2+16] __LF \ + sbcs x7, x7, x4 __LF \ + sbcs x8, x8, x3 __LF \ + csetm x3, lo __LF \ + adds x5, x5, x3 __LF \ + and x4, x3, #0xffffffff __LF \ + adcs x6, x6, x4 __LF \ + adcs x7, x7, xzr __LF \ + and x4, x3, #0xffffffff00000001 __LF \ + adc x8, x8, x4 __LF \ + stp x5, x6, [P0] __LF \ + stp x7, x8, [P0+16] + +// Corresponds exactly to bignum_add_p256 + +#define add_p256(P0,P1,P2) \ + ldp x5, x6, [P1] __LF \ + ldp x4, x3, [P2] __LF \ + adds x5, x5, x4 __LF \ + adcs x6, x6, x3 __LF \ + ldp x7, x8, [P1+16] __LF \ + ldp x4, x3, [P2+16] __LF \ + adcs x7, x7, x4 __LF \ + adcs x8, x8, x3 __LF \ + adc x3, xzr, xzr __LF \ + cmn x5, #1 __LF \ + mov x4, #4294967295 __LF \ + sbcs xzr, x6, x4 __LF \ + sbcs xzr, x7, xzr __LF \ + mov x4, #-4294967295 __LF \ + sbcs xzr, x8, x4 __LF \ + adcs x3, x3, xzr __LF \ + csetm x3, ne __LF \ + subs x5, x5, x3 __LF \ + and x4, x3, #0xffffffff __LF \ + sbcs x6, x6, x4 __LF \ + sbcs x7, x7, xzr __LF \ + and x4, x3, #0xffffffff00000001 __LF \ + sbc x8, x8, x4 __LF \ + stp x5, x6, [P0] __LF \ + stp x7, x8, [P0+16] + +// A weak version of add that only guarantees sum in 4 digits + +#define weakadd_p256(P0,P1,P2) \ + ldp x5, x6, [P1] __LF \ + ldp x4, x3, [P2] __LF \ + adds x5, x5, x4 __LF \ + adcs x6, x6, x3 __LF \ + ldp x7, x8, [P1+16] __LF \ + ldp x4, x3, [P2+16] __LF \ + adcs x7, x7, x4 __LF \ + adcs x8, x8, x3 __LF \ + csetm x3, cs __LF \ + subs x5, x5, x3 __LF \ + and x1, x3, #4294967295 __LF \ + sbcs x6, x6, x1 __LF \ + sbcs x7, x7, xzr __LF \ + and x2, x3, #-4294967295 __LF \ + sbc x8, x8, x2 __LF \ + stp x5, x6, [P0] __LF \ + stp x7, x8, [P0+16] + +// P0 = C * P1 - D * P2 computed as D * (p_256 - P2) + C * P1 +// Quotient estimation is done just as q = h + 1 as in bignum_triple_p256 +// This also applies to the other functions following. + +#define cmsub_p256(P0,C,P1,D,P2) \ + mov x1, D __LF \ + mov x2, #-1 __LF \ + ldp x9, x10, [P2] __LF \ + subs x9, x2, x9 __LF \ + mov x2, #4294967295 __LF \ + sbcs x10, x2, x10 __LF \ + ldp x11, x12, [P2+16] __LF \ + sbcs x11, xzr, x11 __LF \ + mov x2, #-4294967295 __LF \ + sbc x12, x2, x12 __LF \ + mul x3, x1, x9 __LF \ + mul x4, x1, x10 __LF \ + mul x5, x1, x11 __LF \ + mul x6, x1, x12 __LF \ + umulh x9, x1, x9 __LF \ + umulh x10, x1, x10 __LF \ + umulh x11, x1, x11 __LF \ + umulh x7, x1, x12 __LF \ + adds x4, x4, x9 __LF \ + adcs x5, x5, x10 __LF \ + adcs x6, x6, x11 __LF \ + adc x7, x7, xzr __LF \ + mov x1, C __LF \ + ldp x9, x10, [P1] __LF \ + mul x8, x9, x1 __LF \ + umulh x9, x9, x1 __LF \ + adds x3, x3, x8 __LF \ + mul x8, x10, x1 __LF \ + umulh x10, x10, x1 __LF \ + adcs x4, x4, x8 __LF \ + ldp x11, x12, [P1+16] __LF \ + mul x8, x11, x1 __LF \ + umulh x11, x11, x1 __LF \ + adcs x5, x5, x8 __LF \ + mul x8, x12, x1 __LF \ + umulh x12, x12, x1 __LF \ + adcs x6, x6, x8 __LF \ + adc x7, x7, xzr __LF \ + adds x4, x4, x9 __LF \ + adcs x5, x5, x10 __LF \ + adcs x6, x6, x11 __LF \ + adc x7, x7, x12 __LF \ + add x8, x7, #1 __LF \ + lsl x10, x8, #32 __LF \ + adds x6, x6, x10 __LF \ + adc x7, x7, xzr __LF \ + neg x9, x8 __LF \ + sub x10, x10, #1 __LF \ + subs x3, x3, x9 __LF \ + sbcs x4, x4, x10 __LF \ + sbcs x5, x5, xzr __LF \ + sbcs x6, x6, x8 __LF \ + sbc x8, x7, x8 __LF \ + adds x3, x3, x8 __LF \ + and x9, x8, #4294967295 __LF \ + adcs x4, x4, x9 __LF \ + adcs x5, x5, xzr __LF \ + neg x10, x9 __LF \ + adc x6, x6, x10 __LF \ + stp x3, x4, [P0] __LF \ + stp x5, x6, [P0+16] + +// P0 = 4 * P1 - P2, by direct subtraction of P2; the method +// in bignum_cmul_p256 etc. for quotient estimation still +// works when the value to be reduced is negative, as +// long as it is > -p_256, which is the case here. The +// actual accumulation of q * p_256 is done a bit differently +// so it works for the q = 0 case. + +#define cmsub41_p256(P0,P1,P2) \ + ldp x1, x2, [P1] __LF \ + lsl x0, x1, #2 __LF \ + ldp x6, x7, [P2] __LF \ + subs x0, x0, x6 __LF \ + extr x1, x2, x1, #62 __LF \ + sbcs x1, x1, x7 __LF \ + ldp x3, x4, [P1+16] __LF \ + extr x2, x3, x2, #62 __LF \ + ldp x6, x7, [P2+16] __LF \ + sbcs x2, x2, x6 __LF \ + extr x3, x4, x3, #62 __LF \ + sbcs x3, x3, x7 __LF \ + lsr x4, x4, #62 __LF \ + sbc x4, x4, xzr __LF \ + add x5, x4, #1 __LF \ + lsl x8, x5, #32 __LF \ + subs x6, xzr, x8 __LF \ + sbcs x7, xzr, xzr __LF \ + sbc x8, x8, x5 __LF \ + adds x0, x0, x5 __LF \ + adcs x1, x1, x6 __LF \ + adcs x2, x2, x7 __LF \ + adcs x3, x3, x8 __LF \ + csetm x5, cc __LF \ + adds x0, x0, x5 __LF \ + and x6, x5, #4294967295 __LF \ + adcs x1, x1, x6 __LF \ + adcs x2, x2, xzr __LF \ + neg x7, x6 __LF \ + adc x3, x3, x7 __LF \ + stp x0, x1, [P0] __LF \ + stp x2, x3, [P0+16] + +// P0 = 3 * P1 - 8 * P2, computed as (p_256 - P2) << 3 + 3 * P1 + +#define cmsub38_p256(P0,P1,P2) \ + mov x1, 8 __LF \ + mov x2, #-1 __LF \ + ldp x9, x10, [P2] __LF \ + subs x9, x2, x9 __LF \ + mov x2, #4294967295 __LF \ + sbcs x10, x2, x10 __LF \ + ldp x11, x12, [P2+16] __LF \ + sbcs x11, xzr, x11 __LF \ + mov x2, #-4294967295 __LF \ + sbc x12, x2, x12 __LF \ + lsl x3, x9, #3 __LF \ + extr x4, x10, x9, #61 __LF \ + extr x5, x11, x10, #61 __LF \ + extr x6, x12, x11, #61 __LF \ + lsr x7, x12, #61 __LF \ + mov x1, 3 __LF \ + ldp x9, x10, [P1] __LF \ + mul x8, x9, x1 __LF \ + umulh x9, x9, x1 __LF \ + adds x3, x3, x8 __LF \ + mul x8, x10, x1 __LF \ + umulh x10, x10, x1 __LF \ + adcs x4, x4, x8 __LF \ + ldp x11, x12, [P1+16] __LF \ + mul x8, x11, x1 __LF \ + umulh x11, x11, x1 __LF \ + adcs x5, x5, x8 __LF \ + mul x8, x12, x1 __LF \ + umulh x12, x12, x1 __LF \ + adcs x6, x6, x8 __LF \ + adc x7, x7, xzr __LF \ + adds x4, x4, x9 __LF \ + adcs x5, x5, x10 __LF \ + adcs x6, x6, x11 __LF \ + adc x7, x7, x12 __LF \ + add x8, x7, #1 __LF \ + lsl x10, x8, #32 __LF \ + adds x6, x6, x10 __LF \ + adc x7, x7, xzr __LF \ + neg x9, x8 __LF \ + sub x10, x10, #1 __LF \ + subs x3, x3, x9 __LF \ + sbcs x4, x4, x10 __LF \ + sbcs x5, x5, xzr __LF \ + sbcs x6, x6, x8 __LF \ + sbc x8, x7, x8 __LF \ + adds x3, x3, x8 __LF \ + and x9, x8, #4294967295 __LF \ + adcs x4, x4, x9 __LF \ + adcs x5, x5, xzr __LF \ + neg x10, x9 __LF \ + adc x6, x6, x10 __LF \ + stp x3, x4, [P0] __LF \ + stp x5, x6, [P0+16] + +S2N_BN_SYMBOL(p256_montjdouble_alt): + CFI_START + +// Make room on stack for temporary variables + + CFI_DEC_SP(NSPACE) + +// Move the input arguments to stable places + + mov input_z, x0 + mov input_x, x1 + +// Main code, just a sequence of basic field operations + +// z2 = z^2 +// y2 = y^2 + + montsqr_p256(z2,z_1) + montsqr_p256(y2,y_1) + +// x2p = x^2 - z^4 = (x + z^2) * (x - z^2) + + sub_p256(t2,x_1,z2) + weakadd_p256(t1,x_1,z2) + montmul_p256(x2p,t1,t2) + +// t1 = y + z +// xy2 = x * y^2 +// x4p = x2p^2 + + add_p256(t1,y_1,z_1) + montmul_p256(xy2,x_1,y2) + montsqr_p256(x4p,x2p) + +// t1 = (y + z)^2 + + montsqr_p256(t1,t1) + +// d = 12 * xy2 - 9 * x4p +// t1 = y^2 + 2 * y * z + + cmsub_p256(d,12,xy2,9,x4p) + sub_p256(t1,t1,z2) + +// y4 = y^4 + + montsqr_p256(y4,y2) + +// dx2 = d * x2p + + montmul_p256(dx2,d,x2p) + +// z_3' = 2 * y * z + + sub_p256(z_3,t1,y2) + +// x' = 4 * xy2 - d + + cmsub41_p256(x_3,xy2,d) + +// y' = 3 * dx2 - 8 * y4 + + cmsub38_p256(y_3,dx2,y4) + +// Restore stack and return + + CFI_INC_SP(NSPACE) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_montjdouble_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/p256_montjmixadd.S b/cbits/s2n/arm/p256_montjmixadd.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/p256_montjmixadd.S @@ -0,0 +1,513 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Point mixed addition on NIST curve P-256 in Montgomery-Jacobian coordinates +// +// extern void p256_montjmixadd(uint64_t p3[static 12], +// const uint64_t p1[static 12], +// const uint64_t p2[static 8]); +// +// Does p3 := p1 + p2 where all points are regarded as Jacobian triples with +// each coordinate in the Montgomery domain, i.e. x' = (2^256 * x) mod p_256. +// A Jacobian triple (x',y',z') represents affine point (x/z^2,y/z^3). +// The "mixed" part means that p2 only has x and y coordinates, with the +// implicit z coordinate assumed to be the identity. +// +// Standard ARM ABI: X0 = p3, X1 = p1, X2 = p2 +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_montjmixadd) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_montjmixadd) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_montjmixadd) + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Stable homes for input arguments during main code sequence + +#define input_z x17 +#define input_x x19 +#define input_y x20 + +// Pointer-offset pairs for inputs and outputs + +#define x_1 input_x, #0 +#define y_1 input_x, #NUMSIZE +#define z_1 input_x, #(2*NUMSIZE) + +#define x_2 input_y, #0 +#define y_2 input_y, #NUMSIZE + +#define x_3 input_z, #0 +#define y_3 input_z, #NUMSIZE +#define z_3 input_z, #(2*NUMSIZE) + +// Pointer-offset pairs for temporaries, with some aliasing +// #NSPACE is the total stack needed for these temporaries + +#define zp2 sp, #(NUMSIZE*0) +#define ww sp, #(NUMSIZE*0) +#define resx sp, #(NUMSIZE*0) + +#define yd sp, #(NUMSIZE*1) +#define y2a sp, #(NUMSIZE*1) + +#define x2a sp, #(NUMSIZE*2) +#define zzx2 sp, #(NUMSIZE*2) + +#define zz sp, #(NUMSIZE*3) +#define t1 sp, #(NUMSIZE*3) + +#define t2 sp, #(NUMSIZE*4) +#define zzx1 sp, #(NUMSIZE*4) +#define resy sp, #(NUMSIZE*4) + +#define xd sp, #(NUMSIZE*5) +#define resz sp, #(NUMSIZE*5) + +#define NSPACE NUMSIZE*6 + +// Corresponds to bignum_montmul_p256 but uses x0 in place of x17 + +#define montmul_p256(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x5, x6, [P1+16] __LF \ + ldp x7, x8, [P2] __LF \ + ldp x9, x10, [P2+16] __LF \ + mul x11, x3, x7 __LF \ + mul x13, x4, x8 __LF \ + umulh x12, x3, x7 __LF \ + adds x16, x11, x13 __LF \ + umulh x14, x4, x8 __LF \ + adcs x0, x12, x14 __LF \ + adcs x14, x14, xzr __LF \ + adds x12, x12, x16 __LF \ + adcs x13, x13, x0 __LF \ + adcs x14, x14, xzr __LF \ + subs x15, x3, x4 __LF \ + cneg x15, x15, lo __LF \ + csetm x1, lo __LF \ + subs x0, x8, x7 __LF \ + cneg x0, x0, lo __LF \ + mul x16, x15, x0 __LF \ + umulh x0, x15, x0 __LF \ + cinv x1, x1, lo __LF \ + eor x16, x16, x1 __LF \ + eor x0, x0, x1 __LF \ + cmn x1, #1 __LF \ + adcs x12, x12, x16 __LF \ + adcs x13, x13, x0 __LF \ + adc x14, x14, x1 __LF \ + lsl x0, x11, #32 __LF \ + subs x1, x11, x0 __LF \ + lsr x16, x11, #32 __LF \ + sbc x11, x11, x16 __LF \ + adds x12, x12, x0 __LF \ + adcs x13, x13, x16 __LF \ + adcs x14, x14, x1 __LF \ + adc x11, x11, xzr __LF \ + lsl x0, x12, #32 __LF \ + subs x1, x12, x0 __LF \ + lsr x16, x12, #32 __LF \ + sbc x12, x12, x16 __LF \ + adds x13, x13, x0 __LF \ + adcs x14, x14, x16 __LF \ + adcs x11, x11, x1 __LF \ + adc x12, x12, xzr __LF \ + stp x13, x14, [P0] __LF \ + stp x11, x12, [P0+16] __LF \ + mul x11, x5, x9 __LF \ + mul x13, x6, x10 __LF \ + umulh x12, x5, x9 __LF \ + adds x16, x11, x13 __LF \ + umulh x14, x6, x10 __LF \ + adcs x0, x12, x14 __LF \ + adcs x14, x14, xzr __LF \ + adds x12, x12, x16 __LF \ + adcs x13, x13, x0 __LF \ + adcs x14, x14, xzr __LF \ + subs x15, x5, x6 __LF \ + cneg x15, x15, lo __LF \ + csetm x1, lo __LF \ + subs x0, x10, x9 __LF \ + cneg x0, x0, lo __LF \ + mul x16, x15, x0 __LF \ + umulh x0, x15, x0 __LF \ + cinv x1, x1, lo __LF \ + eor x16, x16, x1 __LF \ + eor x0, x0, x1 __LF \ + cmn x1, #1 __LF \ + adcs x12, x12, x16 __LF \ + adcs x13, x13, x0 __LF \ + adc x14, x14, x1 __LF \ + subs x3, x5, x3 __LF \ + sbcs x4, x6, x4 __LF \ + ngc x5, xzr __LF \ + cmn x5, #1 __LF \ + eor x3, x3, x5 __LF \ + adcs x3, x3, xzr __LF \ + eor x4, x4, x5 __LF \ + adcs x4, x4, xzr __LF \ + subs x7, x7, x9 __LF \ + sbcs x8, x8, x10 __LF \ + ngc x9, xzr __LF \ + cmn x9, #1 __LF \ + eor x7, x7, x9 __LF \ + adcs x7, x7, xzr __LF \ + eor x8, x8, x9 __LF \ + adcs x8, x8, xzr __LF \ + eor x10, x5, x9 __LF \ + ldp x15, x1, [P0] __LF \ + adds x15, x11, x15 __LF \ + adcs x1, x12, x1 __LF \ + ldp x5, x9, [P0+16] __LF \ + adcs x5, x13, x5 __LF \ + adcs x9, x14, x9 __LF \ + adc x2, xzr, xzr __LF \ + mul x11, x3, x7 __LF \ + mul x13, x4, x8 __LF \ + umulh x12, x3, x7 __LF \ + adds x16, x11, x13 __LF \ + umulh x14, x4, x8 __LF \ + adcs x0, x12, x14 __LF \ + adcs x14, x14, xzr __LF \ + adds x12, x12, x16 __LF \ + adcs x13, x13, x0 __LF \ + adcs x14, x14, xzr __LF \ + subs x3, x3, x4 __LF \ + cneg x3, x3, lo __LF \ + csetm x4, lo __LF \ + subs x0, x8, x7 __LF \ + cneg x0, x0, lo __LF \ + mul x16, x3, x0 __LF \ + umulh x0, x3, x0 __LF \ + cinv x4, x4, lo __LF \ + eor x16, x16, x4 __LF \ + eor x0, x0, x4 __LF \ + cmn x4, #1 __LF \ + adcs x12, x12, x16 __LF \ + adcs x13, x13, x0 __LF \ + adc x14, x14, x4 __LF \ + cmn x10, #1 __LF \ + eor x11, x11, x10 __LF \ + adcs x11, x11, x15 __LF \ + eor x12, x12, x10 __LF \ + adcs x12, x12, x1 __LF \ + eor x13, x13, x10 __LF \ + adcs x13, x13, x5 __LF \ + eor x14, x14, x10 __LF \ + adcs x14, x14, x9 __LF \ + adcs x3, x2, x10 __LF \ + adcs x4, x10, xzr __LF \ + adc x10, x10, xzr __LF \ + adds x13, x13, x15 __LF \ + adcs x14, x14, x1 __LF \ + adcs x3, x3, x5 __LF \ + adcs x4, x4, x9 __LF \ + adc x10, x10, x2 __LF \ + lsl x0, x11, #32 __LF \ + subs x1, x11, x0 __LF \ + lsr x16, x11, #32 __LF \ + sbc x11, x11, x16 __LF \ + adds x12, x12, x0 __LF \ + adcs x13, x13, x16 __LF \ + adcs x14, x14, x1 __LF \ + adc x11, x11, xzr __LF \ + lsl x0, x12, #32 __LF \ + subs x1, x12, x0 __LF \ + lsr x16, x12, #32 __LF \ + sbc x12, x12, x16 __LF \ + adds x13, x13, x0 __LF \ + adcs x14, x14, x16 __LF \ + adcs x11, x11, x1 __LF \ + adc x12, x12, xzr __LF \ + adds x3, x3, x11 __LF \ + adcs x4, x4, x12 __LF \ + adc x10, x10, xzr __LF \ + add x2, x10, #1 __LF \ + lsl x16, x2, #32 __LF \ + adds x4, x4, x16 __LF \ + adc x10, x10, xzr __LF \ + neg x15, x2 __LF \ + sub x16, x16, #1 __LF \ + subs x13, x13, x15 __LF \ + sbcs x14, x14, x16 __LF \ + sbcs x3, x3, xzr __LF \ + sbcs x4, x4, x2 __LF \ + sbcs x7, x10, x2 __LF \ + adds x13, x13, x7 __LF \ + mov x10, #4294967295 __LF \ + and x10, x10, x7 __LF \ + adcs x14, x14, x10 __LF \ + adcs x3, x3, xzr __LF \ + mov x10, #-4294967295 __LF \ + and x10, x10, x7 __LF \ + adc x4, x4, x10 __LF \ + stp x13, x14, [P0] __LF \ + stp x3, x4, [P0+16] + +// Corresponds to bignum_montsqr_p256 but uses x0 in place of x17 + +#define montsqr_p256(P0,P1) \ + ldp x2, x3, [P1] __LF \ + ldp x4, x5, [P1+16] __LF \ + umull x15, w2, w2 __LF \ + lsr x11, x2, #32 __LF \ + umull x16, w11, w11 __LF \ + umull x11, w2, w11 __LF \ + adds x15, x15, x11, lsl #33 __LF \ + lsr x11, x11, #31 __LF \ + adc x16, x16, x11 __LF \ + umull x0, w3, w3 __LF \ + lsr x11, x3, #32 __LF \ + umull x1, w11, w11 __LF \ + umull x11, w3, w11 __LF \ + mul x12, x2, x3 __LF \ + umulh x13, x2, x3 __LF \ + adds x0, x0, x11, lsl #33 __LF \ + lsr x11, x11, #31 __LF \ + adc x1, x1, x11 __LF \ + adds x12, x12, x12 __LF \ + adcs x13, x13, x13 __LF \ + adc x1, x1, xzr __LF \ + adds x16, x16, x12 __LF \ + adcs x0, x0, x13 __LF \ + adc x1, x1, xzr __LF \ + lsl x12, x15, #32 __LF \ + subs x13, x15, x12 __LF \ + lsr x11, x15, #32 __LF \ + sbc x15, x15, x11 __LF \ + adds x16, x16, x12 __LF \ + adcs x0, x0, x11 __LF \ + adcs x1, x1, x13 __LF \ + adc x15, x15, xzr __LF \ + lsl x12, x16, #32 __LF \ + subs x13, x16, x12 __LF \ + lsr x11, x16, #32 __LF \ + sbc x16, x16, x11 __LF \ + adds x0, x0, x12 __LF \ + adcs x1, x1, x11 __LF \ + adcs x15, x15, x13 __LF \ + adc x16, x16, xzr __LF \ + mul x6, x2, x4 __LF \ + mul x14, x3, x5 __LF \ + umulh x8, x2, x4 __LF \ + subs x10, x2, x3 __LF \ + cneg x10, x10, lo __LF \ + csetm x13, lo __LF \ + subs x12, x5, x4 __LF \ + cneg x12, x12, lo __LF \ + mul x11, x10, x12 __LF \ + umulh x12, x10, x12 __LF \ + cinv x13, x13, lo __LF \ + eor x11, x11, x13 __LF \ + eor x12, x12, x13 __LF \ + adds x7, x6, x8 __LF \ + adc x8, x8, xzr __LF \ + umulh x9, x3, x5 __LF \ + adds x7, x7, x14 __LF \ + adcs x8, x8, x9 __LF \ + adc x9, x9, xzr __LF \ + adds x8, x8, x14 __LF \ + adc x9, x9, xzr __LF \ + cmn x13, #1 __LF \ + adcs x7, x7, x11 __LF \ + adcs x8, x8, x12 __LF \ + adc x9, x9, x13 __LF \ + adds x6, x6, x6 __LF \ + adcs x7, x7, x7 __LF \ + adcs x8, x8, x8 __LF \ + adcs x9, x9, x9 __LF \ + adc x10, xzr, xzr __LF \ + adds x6, x6, x0 __LF \ + adcs x7, x7, x1 __LF \ + adcs x8, x8, x15 __LF \ + adcs x9, x9, x16 __LF \ + adc x10, x10, xzr __LF \ + lsl x12, x6, #32 __LF \ + subs x13, x6, x12 __LF \ + lsr x11, x6, #32 __LF \ + sbc x6, x6, x11 __LF \ + adds x7, x7, x12 __LF \ + adcs x8, x8, x11 __LF \ + adcs x9, x9, x13 __LF \ + adcs x10, x10, x6 __LF \ + adc x6, xzr, xzr __LF \ + lsl x12, x7, #32 __LF \ + subs x13, x7, x12 __LF \ + lsr x11, x7, #32 __LF \ + sbc x7, x7, x11 __LF \ + adds x8, x8, x12 __LF \ + adcs x9, x9, x11 __LF \ + adcs x10, x10, x13 __LF \ + adcs x6, x6, x7 __LF \ + adc x7, xzr, xzr __LF \ + mul x11, x4, x4 __LF \ + adds x8, x8, x11 __LF \ + mul x12, x5, x5 __LF \ + umulh x11, x4, x4 __LF \ + adcs x9, x9, x11 __LF \ + adcs x10, x10, x12 __LF \ + umulh x12, x5, x5 __LF \ + adcs x6, x6, x12 __LF \ + adc x7, x7, xzr __LF \ + mul x11, x4, x5 __LF \ + umulh x12, x4, x5 __LF \ + adds x11, x11, x11 __LF \ + adcs x12, x12, x12 __LF \ + adc x13, xzr, xzr __LF \ + adds x9, x9, x11 __LF \ + adcs x10, x10, x12 __LF \ + adcs x6, x6, x13 __LF \ + adcs x7, x7, xzr __LF \ + mov x11, #4294967295 __LF \ + adds x5, x8, #1 __LF \ + sbcs x11, x9, x11 __LF \ + mov x13, #-4294967295 __LF \ + sbcs x12, x10, xzr __LF \ + sbcs x13, x6, x13 __LF \ + sbcs xzr, x7, xzr __LF \ + csel x8, x5, x8, hs __LF \ + csel x9, x11, x9, hs __LF \ + csel x10, x12, x10, hs __LF \ + csel x6, x13, x6, hs __LF \ + stp x8, x9, [P0] __LF \ + stp x10, x6, [P0+16] + +// Corresponds exactly to bignum_sub_p256 + +#define sub_p256(P0,P1,P2) \ + ldp x5, x6, [P1] __LF \ + ldp x4, x3, [P2] __LF \ + subs x5, x5, x4 __LF \ + sbcs x6, x6, x3 __LF \ + ldp x7, x8, [P1+16] __LF \ + ldp x4, x3, [P2+16] __LF \ + sbcs x7, x7, x4 __LF \ + sbcs x8, x8, x3 __LF \ + csetm x3, cc __LF \ + adds x5, x5, x3 __LF \ + mov x4, #0xffffffff __LF \ + and x4, x4, x3 __LF \ + adcs x6, x6, x4 __LF \ + adcs x7, x7, xzr __LF \ + mov x4, #0xffffffff00000001 __LF \ + and x4, x4, x3 __LF \ + adc x8, x8, x4 __LF \ + stp x5, x6, [P0] __LF \ + stp x7, x8, [P0+16] + +S2N_BN_SYMBOL(p256_montjmixadd): + CFI_START + +// Save regs and make room on stack for temporary variables + + CFI_PUSH2(x19,x20) + CFI_DEC_SP(NSPACE) + +// Move the input arguments to stable places + + mov input_z, x0 + mov input_x, x1 + mov input_y, x2 + +// Main code, just a sequence of basic field operations +// 8 * multiply + 3 * square + 7 * subtract + + montsqr_p256(zp2,z_1) + montmul_p256(y2a,z_1,y_2) + + montmul_p256(x2a,zp2,x_2) + montmul_p256(y2a,zp2,y2a) + + sub_p256(xd,x2a,x_1) + sub_p256(yd,y2a,y_1) + + montsqr_p256(zz,xd) + montsqr_p256(ww,yd) + + montmul_p256(zzx1,zz,x_1) + montmul_p256(zzx2,zz,x2a) + + sub_p256(resx,ww,zzx1) + sub_p256(t1,zzx2,zzx1) + + montmul_p256(resz,xd,z_1) + + sub_p256(resx,resx,zzx2) + + sub_p256(t2,zzx1,resx) + + montmul_p256(t1,t1,y_1) + montmul_p256(t2,yd,t2) + + sub_p256(resy,t2,t1) + +// Test if z_1 = 0 to decide if p1 = 0 (up to projective equivalence) + + ldp x0, x1, [z_1] + ldp x2, x3, [z_1+16] + orr x4, x0, x1 + orr x5, x2, x3 + orr x4, x4, x5 + cmp x4, xzr + +// Multiplex: if p1 <> 0 just copy the computed result from the staging area. +// If p1 = 0 then return the point p2 augmented with a z = 1 coordinate (in +// Montgomery form so not the simple constant 1 but rather 2^256 - p_256), +// hence giving 0 + p2 = p2 for the final result. + + ldp x0, x1, [resx] + ldp x12, x13, [x_2] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x2, x3, [resx+16] + ldp x12, x13, [x_2+16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + + ldp x4, x5, [resy] + ldp x12, x13, [y_2] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x6, x7, [resy+16] + ldp x12, x13, [y_2+16] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + + ldp x8, x9, [resz] + mov x12, #0x0000000000000001 + mov x13, #0xffffffff00000000 + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x10, x11, [resz+16] + mov x12, #0xffffffffffffffff + mov x13, #0x00000000fffffffe + csel x10, x10, x12, ne + csel x11, x11, x13, ne + + stp x0, x1, [x_3] + stp x2, x3, [x_3+16] + stp x4, x5, [y_3] + stp x6, x7, [y_3+16] + stp x8, x9, [z_3] + stp x10, x11, [z_3+16] + +// Restore registers and return + + CFI_INC_SP(NSPACE) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_montjmixadd) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/p256_montjmixadd_alt.S b/cbits/s2n/arm/p256_montjmixadd_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/p256_montjmixadd_alt.S @@ -0,0 +1,517 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Point mixed addition on NIST curve P-256 in Montgomery-Jacobian coordinates +// +// extern void p256_montjmixadd_alt(uint64_t p3[static 12], +// const uint64_t p1[static 12], +// const uint64_t p2[static 8]); +// +// Does p3 := p1 + p2 where all points are regarded as Jacobian triples with +// each coordinate in the Montgomery domain, i.e. x' = (2^256 * x) mod p_256. +// A Jacobian triple (x',y',z') represents affine point (x/z^2,y/z^3). +// The "mixed" part means that p2 only has x and y coordinates, with the +// implicit z coordinate assumed to be the identity. +// +// Standard ARM ABI: X0 = p3, X1 = p1, X2 = p2 +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_montjmixadd_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_montjmixadd_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_montjmixadd_alt) + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Stable homes for input arguments during main code sequence + +#define input_z x15 +#define input_x x16 +#define input_y x17 + +// Pointer-offset pairs for inputs and outputs + +#define x_1 input_x, #0 +#define y_1 input_x, #NUMSIZE +#define z_1 input_x, #(2*NUMSIZE) + +#define x_2 input_y, #0 +#define y_2 input_y, #NUMSIZE + +#define x_3 input_z, #0 +#define y_3 input_z, #NUMSIZE +#define z_3 input_z, #(2*NUMSIZE) + +// Pointer-offset pairs for temporaries, with some aliasing +// #NSPACE is the total stack needed for these temporaries + +#define zp2 sp, #(NUMSIZE*0) +#define ww sp, #(NUMSIZE*0) +#define resx sp, #(NUMSIZE*0) + +#define yd sp, #(NUMSIZE*1) +#define y2a sp, #(NUMSIZE*1) + +#define x2a sp, #(NUMSIZE*2) +#define zzx2 sp, #(NUMSIZE*2) + +#define zz sp, #(NUMSIZE*3) +#define t1 sp, #(NUMSIZE*3) + +#define t2 sp, #(NUMSIZE*4) +#define zzx1 sp, #(NUMSIZE*4) +#define resy sp, #(NUMSIZE*4) + +#define xd sp, #(NUMSIZE*5) +#define resz sp, #(NUMSIZE*5) + +#define NSPACE NUMSIZE*6 + +// Corresponds to bignum_montmul_p256_alt except registers + +#define montmul_p256(P0,P1,P2) \ + ldp x3, x4, [P1] __LF \ + ldp x7, x8, [P2] __LF \ + mul x12, x3, x7 __LF \ + umulh x13, x3, x7 __LF \ + mul x11, x3, x8 __LF \ + umulh x14, x3, x8 __LF \ + adds x13, x13, x11 __LF \ + ldp x9, x10, [P2+16] __LF \ + mul x11, x3, x9 __LF \ + umulh x0, x3, x9 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x3, x10 __LF \ + umulh x1, x3, x10 __LF \ + adcs x0, x0, x11 __LF \ + adc x1, x1, xzr __LF \ + ldp x5, x6, [P1+16] __LF \ + mul x11, x4, x7 __LF \ + adds x13, x13, x11 __LF \ + mul x11, x4, x8 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x4, x9 __LF \ + adcs x0, x0, x11 __LF \ + mul x11, x4, x10 __LF \ + adcs x1, x1, x11 __LF \ + umulh x3, x4, x10 __LF \ + adc x3, x3, xzr __LF \ + umulh x11, x4, x7 __LF \ + adds x14, x14, x11 __LF \ + umulh x11, x4, x8 __LF \ + adcs x0, x0, x11 __LF \ + umulh x11, x4, x9 __LF \ + adcs x1, x1, x11 __LF \ + adc x3, x3, xzr __LF \ + mul x11, x5, x7 __LF \ + adds x14, x14, x11 __LF \ + mul x11, x5, x8 __LF \ + adcs x0, x0, x11 __LF \ + mul x11, x5, x9 __LF \ + adcs x1, x1, x11 __LF \ + mul x11, x5, x10 __LF \ + adcs x3, x3, x11 __LF \ + umulh x4, x5, x10 __LF \ + adc x4, x4, xzr __LF \ + umulh x11, x5, x7 __LF \ + adds x0, x0, x11 __LF \ + umulh x11, x5, x8 __LF \ + adcs x1, x1, x11 __LF \ + umulh x11, x5, x9 __LF \ + adcs x3, x3, x11 __LF \ + adc x4, x4, xzr __LF \ + mul x11, x6, x7 __LF \ + adds x0, x0, x11 __LF \ + mul x11, x6, x8 __LF \ + adcs x1, x1, x11 __LF \ + mul x11, x6, x9 __LF \ + adcs x3, x3, x11 __LF \ + mul x11, x6, x10 __LF \ + adcs x4, x4, x11 __LF \ + umulh x5, x6, x10 __LF \ + adc x5, x5, xzr __LF \ + mov x10, #0xffffffff00000001 __LF \ + adds x13, x13, x12, lsl #32 __LF \ + lsr x11, x12, #32 __LF \ + adcs x14, x14, x11 __LF \ + mul x11, x12, x10 __LF \ + umulh x12, x12, x10 __LF \ + adcs x0, x0, x11 __LF \ + adc x12, x12, xzr __LF \ + umulh x11, x6, x7 __LF \ + adds x1, x1, x11 __LF \ + umulh x11, x6, x8 __LF \ + adcs x3, x3, x11 __LF \ + umulh x11, x6, x9 __LF \ + adcs x4, x4, x11 __LF \ + adc x5, x5, xzr __LF \ + adds x14, x14, x13, lsl #32 __LF \ + lsr x11, x13, #32 __LF \ + adcs x0, x0, x11 __LF \ + mul x11, x13, x10 __LF \ + umulh x13, x13, x10 __LF \ + adcs x12, x12, x11 __LF \ + adc x13, x13, xzr __LF \ + adds x0, x0, x14, lsl #32 __LF \ + lsr x11, x14, #32 __LF \ + adcs x12, x12, x11 __LF \ + mul x11, x14, x10 __LF \ + umulh x14, x14, x10 __LF \ + adcs x13, x13, x11 __LF \ + adc x14, x14, xzr __LF \ + adds x12, x12, x0, lsl #32 __LF \ + lsr x11, x0, #32 __LF \ + adcs x13, x13, x11 __LF \ + mul x11, x0, x10 __LF \ + umulh x0, x0, x10 __LF \ + adcs x14, x14, x11 __LF \ + adc x0, x0, xzr __LF \ + adds x12, x12, x1 __LF \ + adcs x13, x13, x3 __LF \ + adcs x14, x14, x4 __LF \ + adcs x0, x0, x5 __LF \ + cset x8, cs __LF \ + mov x11, #0xffffffff __LF \ + adds x1, x12, #0x1 __LF \ + sbcs x3, x13, x11 __LF \ + sbcs x4, x14, xzr __LF \ + sbcs x5, x0, x10 __LF \ + sbcs xzr, x8, xzr __LF \ + csel x12, x12, x1, cc __LF \ + csel x13, x13, x3, cc __LF \ + csel x14, x14, x4, cc __LF \ + csel x0, x0, x5, cc __LF \ + stp x12, x13, [P0] __LF \ + stp x14, x0, [P0+16] + +// Corresponds exactly to bignum_montsqr_p256_alt + +#define montsqr_p256(P0,P1) \ + ldp x2, x3, [P1] __LF \ + mul x9, x2, x3 __LF \ + umulh x10, x2, x3 __LF \ + ldp x4, x5, [P1+16] __LF \ + mul x11, x2, x5 __LF \ + umulh x12, x2, x5 __LF \ + mul x6, x2, x4 __LF \ + umulh x7, x2, x4 __LF \ + adds x10, x10, x6 __LF \ + adcs x11, x11, x7 __LF \ + mul x6, x3, x4 __LF \ + umulh x7, x3, x4 __LF \ + adc x7, x7, xzr __LF \ + adds x11, x11, x6 __LF \ + mul x13, x4, x5 __LF \ + umulh x14, x4, x5 __LF \ + adcs x12, x12, x7 __LF \ + mul x6, x3, x5 __LF \ + umulh x7, x3, x5 __LF \ + adc x7, x7, xzr __LF \ + adds x12, x12, x6 __LF \ + adcs x13, x13, x7 __LF \ + adc x14, x14, xzr __LF \ + adds x9, x9, x9 __LF \ + adcs x10, x10, x10 __LF \ + adcs x11, x11, x11 __LF \ + adcs x12, x12, x12 __LF \ + adcs x13, x13, x13 __LF \ + adcs x14, x14, x14 __LF \ + cset x7, cs __LF \ + umulh x6, x2, x2 __LF \ + mul x8, x2, x2 __LF \ + adds x9, x9, x6 __LF \ + mul x6, x3, x3 __LF \ + adcs x10, x10, x6 __LF \ + umulh x6, x3, x3 __LF \ + adcs x11, x11, x6 __LF \ + mul x6, x4, x4 __LF \ + adcs x12, x12, x6 __LF \ + umulh x6, x4, x4 __LF \ + adcs x13, x13, x6 __LF \ + mul x6, x5, x5 __LF \ + adcs x14, x14, x6 __LF \ + umulh x6, x5, x5 __LF \ + adc x7, x7, x6 __LF \ + adds x9, x9, x8, lsl #32 __LF \ + lsr x3, x8, #32 __LF \ + adcs x10, x10, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x8, x3 __LF \ + umulh x8, x8, x3 __LF \ + adcs x11, x11, x2 __LF \ + adc x8, x8, xzr __LF \ + adds x10, x10, x9, lsl #32 __LF \ + lsr x3, x9, #32 __LF \ + adcs x11, x11, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x9, x3 __LF \ + umulh x9, x9, x3 __LF \ + adcs x8, x8, x2 __LF \ + adc x9, x9, xzr __LF \ + adds x11, x11, x10, lsl #32 __LF \ + lsr x3, x10, #32 __LF \ + adcs x8, x8, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x10, x3 __LF \ + umulh x10, x10, x3 __LF \ + adcs x9, x9, x2 __LF \ + adc x10, x10, xzr __LF \ + adds x8, x8, x11, lsl #32 __LF \ + lsr x3, x11, #32 __LF \ + adcs x9, x9, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x11, x3 __LF \ + umulh x11, x11, x3 __LF \ + adcs x10, x10, x2 __LF \ + adc x11, x11, xzr __LF \ + adds x8, x8, x12 __LF \ + adcs x9, x9, x13 __LF \ + adcs x10, x10, x14 __LF \ + adcs x11, x11, x7 __LF \ + cset x2, cs __LF \ + mov x3, #0xffffffff __LF \ + mov x5, #0xffffffff00000001 __LF \ + adds x12, x8, #0x1 __LF \ + sbcs x13, x9, x3 __LF \ + sbcs x14, x10, xzr __LF \ + sbcs x7, x11, x5 __LF \ + sbcs xzr, x2, xzr __LF \ + csel x8, x8, x12, cc __LF \ + csel x9, x9, x13, cc __LF \ + csel x10, x10, x14, cc __LF \ + csel x11, x11, x7, cc __LF \ + stp x8, x9, [P0] __LF \ + stp x10, x11, [P0+16] + +// Almost-Montgomery variant which we use when an input to other muls +// with the other argument fully reduced (which is always safe). + +#define amontsqr_p256(P0,P1) \ + ldp x2, x3, [P1] __LF \ + mul x9, x2, x3 __LF \ + umulh x10, x2, x3 __LF \ + ldp x4, x5, [P1+16] __LF \ + mul x11, x2, x5 __LF \ + umulh x12, x2, x5 __LF \ + mul x6, x2, x4 __LF \ + umulh x7, x2, x4 __LF \ + adds x10, x10, x6 __LF \ + adcs x11, x11, x7 __LF \ + mul x6, x3, x4 __LF \ + umulh x7, x3, x4 __LF \ + adc x7, x7, xzr __LF \ + adds x11, x11, x6 __LF \ + mul x13, x4, x5 __LF \ + umulh x14, x4, x5 __LF \ + adcs x12, x12, x7 __LF \ + mul x6, x3, x5 __LF \ + umulh x7, x3, x5 __LF \ + adc x7, x7, xzr __LF \ + adds x12, x12, x6 __LF \ + adcs x13, x13, x7 __LF \ + adc x14, x14, xzr __LF \ + adds x9, x9, x9 __LF \ + adcs x10, x10, x10 __LF \ + adcs x11, x11, x11 __LF \ + adcs x12, x12, x12 __LF \ + adcs x13, x13, x13 __LF \ + adcs x14, x14, x14 __LF \ + cset x7, cs __LF \ + umulh x6, x2, x2 __LF \ + mul x8, x2, x2 __LF \ + adds x9, x9, x6 __LF \ + mul x6, x3, x3 __LF \ + adcs x10, x10, x6 __LF \ + umulh x6, x3, x3 __LF \ + adcs x11, x11, x6 __LF \ + mul x6, x4, x4 __LF \ + adcs x12, x12, x6 __LF \ + umulh x6, x4, x4 __LF \ + adcs x13, x13, x6 __LF \ + mul x6, x5, x5 __LF \ + adcs x14, x14, x6 __LF \ + umulh x6, x5, x5 __LF \ + adc x7, x7, x6 __LF \ + adds x9, x9, x8, lsl #32 __LF \ + lsr x3, x8, #32 __LF \ + adcs x10, x10, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x8, x3 __LF \ + umulh x8, x8, x3 __LF \ + adcs x11, x11, x2 __LF \ + adc x8, x8, xzr __LF \ + adds x10, x10, x9, lsl #32 __LF \ + lsr x3, x9, #32 __LF \ + adcs x11, x11, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x9, x3 __LF \ + umulh x9, x9, x3 __LF \ + adcs x8, x8, x2 __LF \ + adc x9, x9, xzr __LF \ + adds x11, x11, x10, lsl #32 __LF \ + lsr x3, x10, #32 __LF \ + adcs x8, x8, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x10, x3 __LF \ + umulh x10, x10, x3 __LF \ + adcs x9, x9, x2 __LF \ + adc x10, x10, xzr __LF \ + adds x8, x8, x11, lsl #32 __LF \ + lsr x3, x11, #32 __LF \ + adcs x9, x9, x3 __LF \ + mov x3, #0xffffffff00000001 __LF \ + mul x2, x11, x3 __LF \ + umulh x11, x11, x3 __LF \ + adcs x10, x10, x2 __LF \ + adc x11, x11, xzr __LF \ + adds x8, x8, x12 __LF \ + adcs x9, x9, x13 __LF \ + adcs x10, x10, x14 __LF \ + adcs x11, x11, x7 __LF \ + mov x2, #0xffffffffffffffff __LF \ + csel x2, xzr, x2, cc __LF \ + mov x3, #0xffffffff __LF \ + csel x3, xzr, x3, cc __LF \ + mov x5, #0xffffffff00000001 __LF \ + csel x5, xzr, x5, cc __LF \ + subs x8, x8, x2 __LF \ + sbcs x9, x9, x3 __LF \ + sbcs x10, x10, xzr __LF \ + sbc x11, x11, x5 __LF \ + stp x8, x9, [P0] __LF \ + stp x10, x11, [P0+16] + +// Corresponds exactly to bignum_sub_p256 + +#define sub_p256(P0,P1,P2) \ + ldp x5, x6, [P1] __LF \ + ldp x4, x3, [P2] __LF \ + subs x5, x5, x4 __LF \ + sbcs x6, x6, x3 __LF \ + ldp x7, x8, [P1+16] __LF \ + ldp x4, x3, [P2+16] __LF \ + sbcs x7, x7, x4 __LF \ + sbcs x8, x8, x3 __LF \ + csetm x3, cc __LF \ + adds x5, x5, x3 __LF \ + mov x4, #0xffffffff __LF \ + and x4, x4, x3 __LF \ + adcs x6, x6, x4 __LF \ + adcs x7, x7, xzr __LF \ + mov x4, #0xffffffff00000001 __LF \ + and x4, x4, x3 __LF \ + adc x8, x8, x4 __LF \ + stp x5, x6, [P0] __LF \ + stp x7, x8, [P0+16] + +S2N_BN_SYMBOL(p256_montjmixadd_alt): + CFI_START + +// Make room on stack for temporary variables +// Move the input arguments to stable places + + CFI_DEC_SP(NSPACE) + + mov input_z, x0 + mov input_x, x1 + mov input_y, x2 + +// Main code, just a sequence of basic field operations +// 8 * multiply + 3 * square + 7 * subtract + + amontsqr_p256(zp2,z_1) + montmul_p256(y2a,z_1,y_2) + + montmul_p256(x2a,zp2,x_2) + montmul_p256(y2a,zp2,y2a) + + sub_p256(xd,x2a,x_1) + sub_p256(yd,y2a,y_1) + + amontsqr_p256(zz,xd) + montsqr_p256(ww,yd) + + montmul_p256(zzx1,zz,x_1) + montmul_p256(zzx2,zz,x2a) + + sub_p256(resx,ww,zzx1) + sub_p256(t1,zzx2,zzx1) + + montmul_p256(resz,xd,z_1) + + sub_p256(resx,resx,zzx2) + + sub_p256(t2,zzx1,resx) + + montmul_p256(t1,t1,y_1) + montmul_p256(t2,yd,t2) + + sub_p256(resy,t2,t1) + +// Test if z_1 = 0 to decide if p1 = 0 (up to projective equivalence) + + ldp x0, x1, [z_1] + ldp x2, x3, [z_1+16] + orr x4, x0, x1 + orr x5, x2, x3 + orr x4, x4, x5 + cmp x4, xzr + +// Multiplex: if p1 <> 0 just copy the computed result from the staging area. +// If p1 = 0 then return the point p2 augmented with a z = 1 coordinate (in +// Montgomery form so not the simple constant 1 but rather 2^256 - p_256), +// hence giving 0 + p2 = p2 for the final result. + + ldp x0, x1, [resx] + ldp x12, x13, [x_2] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x2, x3, [resx+16] + ldp x12, x13, [x_2+16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + + ldp x4, x5, [resy] + ldp x12, x13, [y_2] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x6, x7, [resy+16] + ldp x12, x13, [y_2+16] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + + ldp x8, x9, [resz] + mov x12, #0x0000000000000001 + mov x13, #0xffffffff00000000 + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x10, x11, [resz+16] + mov x12, #0xffffffffffffffff + mov x13, #0x00000000fffffffe + csel x10, x10, x12, ne + csel x11, x11, x13, ne + + stp x0, x1, [x_3] + stp x2, x3, [x_3+16] + stp x4, x5, [y_3] + stp x6, x7, [y_3+16] + stp x8, x9, [z_3] + stp x10, x11, [z_3+16] + +// Restore stack and return + + CFI_INC_SP(NSPACE) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_montjmixadd_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/p256_scalarmul.S b/cbits/s2n/arm/p256_scalarmul.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/p256_scalarmul.S @@ -0,0 +1,8620 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Scalar multiplication for P-256 +// Input scalar[4], point[8]; output res[8] +// +// extern void p256_scalarmul +// (uint64_t res[static 8], +// const uint64_t scalar[static 4], +// const uint64_t point[static 8]); +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-256, returns the point (X,Y) = n * P. The input and output +// are affine points, and in the case of the point at infinity as +// the result, (0,0) is returned. +// +// Standard ARM ABI: X0 = res, X1 = scalar, X2 = point +// ---------------------------------------------------------------------------- + + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_scalarmul) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_scalarmul) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_scalarmul) + + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Safe copies of inputs (res lasts the whole code, point not so long) +// and additional values in variables, with some aliasing + +#define res x19 +#define sgn x20 +#define j x20 +#define point x21 + +// Intermediate variables on the stack. The last z2, z3 values can +// safely be overlaid on the table, which is no longer needed at the end. + +#define scalarb sp, #(0*NUMSIZE) +#define acc sp, #(1*NUMSIZE) +#define tabent sp, #(4*NUMSIZE) + +#define tab sp, #(7*NUMSIZE) + +#define z2 sp, #(7*NUMSIZE) +#define z3 sp, #(8*NUMSIZE) + +#define NSPACE 31*NUMSIZE + +// Loading large constants + +#define movbig(nn,n3,n2,n1,n0) \ + movz nn, n0 __LF \ + movk nn, n1, lsl #16 __LF \ + movk nn, n2, lsl #32 __LF \ + movk nn, n3, lsl #48 + +S2N_BN_SYMBOL(p256_scalarmul): + CFI_START + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x30) + CFI_DEC_SP(NSPACE) + +// Preserve the "res" and "point" input arguments. We load and process the +// scalar immediately so we don't bother preserving that input argument. +// Also, "point" is only needed early on and so its register gets re-used. + + mov res, x0 + mov point, x2 + +// Load the digits of group order n_256 = [x12;x13;x14;x15] + + movbig(x12, #0xf3b9, #0xcac2, #0xfc63, #0x2551) + movbig(x13, #0xbce6, #0xfaad, #0xa717, #0x9e84) + mov x14, #0xffffffffffffffff + mov x15, #0xffffffff00000000 + +// First, reduce the input scalar mod n_256, i.e. conditionally subtract n_256 + + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + + subs x6, x2, x12 + sbcs x7, x3, x13 + sbcs x8, x4, x14 + sbcs x9, x5, x15 + + csel x2, x2, x6, cc + csel x3, x3, x7, cc + csel x4, x4, x8, cc + csel x5, x5, x9, cc + +// Now if the top bit of the reduced scalar is set, negate it mod n_256, +// i.e. do n |-> n_256 - n. Remember the sign as "sgn" so we can +// correspondingly negate the point below. + + subs x6, x12, x2 + sbcs x7, x13, x3 + sbcs x8, x14, x4 + sbc x9, x15, x5 + + tst x5, #0x8000000000000000 + csel x2, x2, x6, eq + csel x3, x3, x7, eq + csel x4, x4, x8, eq + csel x5, x5, x9, eq + cset sgn, ne + +// In either case then add the recoding constant 0x08888...888 to allow +// signed digits. + + mov x6, 0x8888888888888888 + adds x2, x2, x6 + adcs x3, x3, x6 + bic x7, x6, #0xF000000000000000 + adcs x4, x4, x6 + adc x5, x5, x7 + + stp x2, x3, [scalarb] + stp x4, x5, [scalarb+16] + +// Set the tab[0] table entry to Montgomery-Jacobian point = 1 * P +// The z coordinate is just the Montgomery form of the constant 1. + + add x0, tab + mov x1, point + CFI_BL(Lp256_scalarmul_local_tomont_p256) + + add x1, point, #32 + add x0, tab+32 + CFI_BL(Lp256_scalarmul_local_tomont_p256) + + mov x0, #0x0000000000000001 + mov x1, #0xffffffff00000000 + stp x0, x1, [tab+64] + mov x2, #0xffffffffffffffff + mov x3, #0x00000000fffffffe + stp x2, x3, [tab+80] + +// If the top bit of the scalar was set, negate (y coordinate of) the point + + ldp x4, x5, [tab+32] + ldp x6, x7, [tab+48] + + mov x0, 0xffffffffffffffff + subs x0, x0, x4 + mov x1, 0x00000000ffffffff + sbcs x1, x1, x5 + mov x3, 0xffffffff00000001 + sbcs x2, xzr, x6 + sbc x3, x3, x7 + + cmp sgn, xzr + csel x4, x0, x4, ne + csel x5, x1, x5, ne + csel x6, x2, x6, ne + csel x7, x3, x7, ne + + stp x4, x5, [tab+32] + stp x6, x7, [tab+48] + +// Compute and record tab[1] = 2 * p, ..., tab[7] = 8 * P + + add x0, tab+96*1 + add x1, tab + CFI_BL(Lp256_scalarmul_local_p256_montjdouble) + + add x0, tab+96*2 + add x1, tab+96*1 + add x2, tab + CFI_BL(Lp256_scalarmul_local_p256_montjmixadd) + + add x0, tab+96*3 + add x1, tab+96*1 + CFI_BL(Lp256_scalarmul_local_p256_montjdouble) + + add x0, tab+96*4 + add x1, tab+96*3 + add x2, tab + CFI_BL(Lp256_scalarmul_local_p256_montjmixadd) + + add x0, tab+96*5 + add x1, tab+96*2 + CFI_BL(Lp256_scalarmul_local_p256_montjdouble) + + add x0, tab+96*6 + add x1, tab+96*5 + add x2, tab + CFI_BL(Lp256_scalarmul_local_p256_montjmixadd) + + add x0, tab+96*7 + add x1, tab+96*3 + CFI_BL(Lp256_scalarmul_local_p256_montjdouble) + +// Initialize the accumulator as a table entry for top 4 bits (unrecoded) + + ldr x14, [scalarb+24] + lsr x14, x14, #60 + + mov x0, xzr + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, xzr + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + mov x9, xzr + mov x10, xzr + mov x11, xzr + add x15, tab + + .set i, 1 +.rep 8 + cmp x14, #i + ldp x12, x13, [x15] + csel x0, x12, x0, eq + csel x1, x13, x1, eq + ldp x12, x13, [x15, #16] + csel x2, x12, x2, eq + csel x3, x13, x3, eq + ldp x12, x13, [x15, #32] + csel x4, x12, x4, eq + csel x5, x13, x5, eq + ldp x12, x13, [x15, #48] + csel x6, x12, x6, eq + csel x7, x13, x7, eq + ldp x12, x13, [x15, #64] + csel x8, x12, x8, eq + csel x9, x13, x9, eq + ldp x12, x13, [x15, #80] + csel x10, x12, x10, eq + csel x11, x13, x11, eq + add x15, x15, #96 + .set i, (i+1) +.endr + stp x0, x1, [acc] + stp x2, x3, [acc+16] + stp x4, x5, [acc+32] + stp x6, x7, [acc+48] + stp x8, x9, [acc+64] + stp x10, x11, [acc+80] + + mov j, #252 + +// Main loop over size-4 bitfields: double 4 times then add signed digit + +Lp256_scalarmul_loop: + sub j, j, #4 + + add x0, acc + add x1, acc + CFI_BL(Lp256_scalarmul_local_p256_montjdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp256_scalarmul_local_p256_montjdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp256_scalarmul_local_p256_montjdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp256_scalarmul_local_p256_montjdouble) + + lsr x2, j, #6 + ldr x14, [sp, x2, lsl #3] // Exploits scalarb = sp exactly + lsr x14, x14, j + and x14, x14, #15 + + subs x14, x14, #8 + cset x16, lo // x16 = sign of digit (1 = negative) + cneg x14, x14, lo // x14 = absolute value of digit + +// Conditionally select the table entry tab[i-1] = i * P in constant time + + mov x0, xzr + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, xzr + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + mov x9, xzr + mov x10, xzr + mov x11, xzr + add x15, tab + .set i, 1 +.rep 8 + cmp x14, #i + ldp x12, x13, [x15] + csel x0, x12, x0, eq + csel x1, x13, x1, eq + ldp x12, x13, [x15, #16] + csel x2, x12, x2, eq + csel x3, x13, x3, eq + ldp x12, x13, [x15, #32] + csel x4, x12, x4, eq + csel x5, x13, x5, eq + ldp x12, x13, [x15, #48] + csel x6, x12, x6, eq + csel x7, x13, x7, eq + ldp x12, x13, [x15, #64] + csel x8, x12, x8, eq + csel x9, x13, x9, eq + ldp x12, x13, [x15, #80] + csel x10, x12, x10, eq + csel x11, x13, x11, eq + add x15, x15, #96 + .set i, (i+1) +.endr + +// Store it to "tabent" with the y coordinate optionally negated + + stp x0, x1, [tabent] + stp x2, x3, [tabent+16] + + mov x0, 0xffffffffffffffff + subs x0, x0, x4 + mov x1, 0x00000000ffffffff + sbcs x1, x1, x5 + mov x3, 0xffffffff00000001 + sbcs x2, xzr, x6 + sbc x3, x3, x7 + + cmp x16, xzr + csel x4, x0, x4, ne + csel x5, x1, x5, ne + csel x6, x2, x6, ne + csel x7, x3, x7, ne + + stp x4, x5, [tabent+32] + stp x6, x7, [tabent+48] + stp x8, x9, [tabent+64] + stp x10, x11, [tabent+80] + + add x0, acc + add x1, acc + add x2, tabent + CFI_BL(Lp256_scalarmul_local_p256_montjadd) + + cbnz j, Lp256_scalarmul_loop + +// That's the end of the main loop, and we just need to translate +// back from the Jacobian representation to affine. First of all, +// let z2 = 1/z^2 and z3 = 1/z^3, both without Montgomery form + + add x0, z2 + add x1, acc+64 + CFI_BL(Lp256_scalarmul_local_montsqr_p256) + + add x0, z3 + add x2, z2 + add x1, acc+64 + CFI_BL(Lp256_scalarmul_local_montmul_p256) + + add x0, z2 + add x1, z3 + CFI_BL(Lp256_scalarmul_local_demont_p256) + + add x0, z3 + add x1, z2 + CFI_BL(Lp256_scalarmul_local_inv_p256) + + add x0, z2 + add x2, z3 + add x1, acc+64 + CFI_BL(Lp256_scalarmul_local_montmul_p256) + +// Convert back from Jacobian (X,Y,Z) |-> (X/Z^2, Y/Z^3) + + add x1, acc + add x2, z2 + mov x0, res + CFI_BL(Lp256_scalarmul_local_montmul_p256) + + add x0, res, #32 + add x1, acc+32 + add x2, z3 + CFI_BL(Lp256_scalarmul_local_montmul_p256) + +// Restore stack and registers and return + + CFI_INC_SP(NSPACE) + CFI_POP2(x21,x30) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_scalarmul) + +// Local copies of subroutines, complete clones at the moment + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_local_demont_p256) + +Lp256_scalarmul_local_demont_p256: + CFI_START + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + lsl x7, x2, #32 + subs x8, x2, x7 + lsr x6, x2, #32 + sbc x2, x2, x6 + adds x3, x3, x7 + adcs x4, x4, x6 + adcs x5, x5, x8 + adc x2, x2, xzr + lsl x7, x3, #32 + subs x8, x3, x7 + lsr x6, x3, #32 + sbc x3, x3, x6 + adds x4, x4, x7 + adcs x5, x5, x6 + adcs x2, x2, x8 + adc x3, x3, xzr + lsl x7, x4, #32 + subs x8, x4, x7 + lsr x6, x4, #32 + sbc x4, x4, x6 + adds x5, x5, x7 + adcs x2, x2, x6 + adcs x3, x3, x8 + adc x4, x4, xzr + lsl x7, x5, #32 + subs x8, x5, x7 + lsr x6, x5, #32 + sbc x5, x5, x6 + adds x2, x2, x7 + adcs x3, x3, x6 + adcs x4, x4, x8 + adc x5, x5, xzr + stp x2, x3, [x0] + stp x4, x5, [x0, #16] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_local_demont_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_local_inv_p256) + +Lp256_scalarmul_local_inv_p256: + CFI_START + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_DEC_SP(160) + mov x20, x0 + mov x10, #0xffffffffffffffff + mov x11, #0xffffffff + mov x13, #0xffffffff00000001 + stp x10, x11, [sp] + stp xzr, x13, [sp, #16] + str xzr, [sp, #32] + ldp x2, x3, [x1] + subs x10, x2, x10 + sbcs x11, x3, x11 + ldp x4, x5, [x1, #16] + sbcs x12, x4, xzr + sbcs x13, x5, x13 + csel x2, x2, x10, cc + csel x3, x3, x11, cc + csel x4, x4, x12, cc + csel x5, x5, x13, cc + stp x2, x3, [sp, #48] + stp x4, x5, [sp, #64] + str xzr, [sp, #80] + stp xzr, xzr, [sp, #96] + stp xzr, xzr, [sp, #112] + mov x10, #0x4000000000000 + stp x10, xzr, [sp, #128] + stp xzr, xzr, [sp, #144] + mov x21, #0xa + mov x22, #0x1 + b Lp256_scalarmul_inv_midloop +Lp256_scalarmul_inv_loop: + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + and x0, x12, x16 + and x1, x13, x17 + add x19, x0, x1 + ldr x7, [sp] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #48] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x7, [sp, #8] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #56] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + adc x6, x6, x1 + extr x4, x2, x4, #59 + str x4, [sp] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + adc x4, x4, x1 + extr x5, x3, x5, #59 + str x5, [sp, #48] + ldr x7, [sp, #16] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #64] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + adc x5, x5, x1 + extr x2, x6, x2, #59 + str x2, [sp, #8] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + adc x2, x2, x1 + extr x3, x4, x3, #59 + str x3, [sp, #56] + ldr x7, [sp, #24] + eor x1, x7, x14 + ldr x23, [sp, #32] + eor x3, x23, x14 + and x3, x3, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #72] + eor x1, x8, x15 + ldr x24, [sp, #80] + eor x0, x24, x15 + and x0, x0, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x5, x6, #59 + str x6, [sp, #16] + extr x5, x3, x5, #59 + str x5, [sp, #24] + asr x3, x3, #59 + str x3, [sp, #32] + eor x1, x7, x16 + eor x5, x23, x16 + and x5, x5, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + eor x0, x24, x17 + and x0, x0, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x4, x2, x4, #59 + str x4, [sp, #64] + extr x2, x5, x2, #59 + str x2, [sp, #72] + asr x5, x5, #59 + str x5, [sp, #80] + ldr x7, [sp, #96] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #128] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #96] + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + str x5, [sp, #128] + adc x3, x3, x1 + ldr x7, [sp, #104] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #136] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #104] + adc x6, x6, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + str x3, [sp, #136] + adc x4, x4, x1 + ldr x7, [sp, #112] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #144] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #112] + adc x5, x5, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + str x4, [sp, #144] + adc x2, x2, x1 + ldr x7, [sp, #120] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #152] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + ldp x0, x1, [sp, #96] + ldr x6, [sp, #112] + mov x14, #0xe000000000000000 + adds x0, x0, x14 + sbcs x1, x1, xzr + mov x11, #0x1fffffff + adcs x6, x6, x11 + mov x10, #0x2000000000000000 + adcs x5, x5, x10 + mov x14, #0x1fffffffe0000000 + adc x3, x3, x14 + lsl x11, x0, #32 + subs x14, x0, x11 + lsr x10, x0, #32 + sbc x0, x0, x10 + adds x1, x1, x11 + adcs x6, x6, x10 + adcs x5, x5, x14 + adcs x3, x3, x0 + mov x14, #0xffffffffffffffff + mov x11, #0xffffffff + mov x10, #0xffffffff00000001 + csel x14, x14, xzr, cs + csel x11, x11, xzr, cs + csel x10, x10, xzr, cs + subs x1, x1, x14 + sbcs x6, x6, x11 + sbcs x5, x5, xzr + sbc x3, x3, x10 + stp x1, x6, [sp, #96] + stp x5, x3, [sp, #112] + eor x1, x7, x16 + and x5, x16, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + and x0, x17, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + ldp x0, x1, [sp, #128] + ldr x3, [sp, #144] + mov x14, #0xe000000000000000 + adds x0, x0, x14 + sbcs x1, x1, xzr + mov x11, #0x1fffffff + adcs x3, x3, x11 + mov x10, #0x2000000000000000 + adcs x2, x2, x10 + mov x14, #0x1fffffffe0000000 + adc x5, x5, x14 + lsl x11, x0, #32 + subs x14, x0, x11 + lsr x10, x0, #32 + sbc x0, x0, x10 + adds x1, x1, x11 + adcs x3, x3, x10 + adcs x2, x2, x14 + adcs x5, x5, x0 + mov x14, #0xffffffffffffffff + mov x11, #0xffffffff + mov x10, #0xffffffff00000001 + csel x14, x14, xzr, cs + csel x11, x11, xzr, cs + csel x10, x10, xzr, cs + subs x1, x1, x14 + sbcs x3, x3, x11 + sbcs x2, x2, xzr + sbc x5, x5, x10 + stp x1, x3, [sp, #128] + stp x2, x5, [sp, #144] +Lp256_scalarmul_inv_midloop: + mov x1, x22 + ldr x2, [sp] + ldr x3, [sp, #48] + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x8, x4, #0x100, lsl #12 + sbfx x8, x8, #21, #21 + mov x11, #0x100000 + add x11, x11, x11, lsl #21 + add x9, x4, x11 + asr x9, x9, #42 + add x10, x5, #0x100, lsl #12 + sbfx x10, x10, #21, #21 + add x11, x5, x11 + asr x11, x11, #42 + mul x6, x8, x2 + mul x7, x9, x3 + mul x2, x10, x2 + mul x3, x11, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #21, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #42 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #21, #21 + add x15, x5, x15 + asr x15, x15, #42 + mul x6, x12, x2 + mul x7, x13, x3 + mul x2, x14, x2 + mul x3, x15, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + mul x2, x12, x8 + mul x3, x12, x9 + mul x6, x14, x8 + mul x7, x14, x9 + madd x8, x13, x10, x2 + madd x9, x13, x11, x3 + madd x16, x15, x10, x6 + madd x17, x15, x11, x7 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #22, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #43 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #22, #21 + add x15, x5, x15 + asr x15, x15, #43 + mneg x2, x12, x8 + mneg x3, x12, x9 + mneg x4, x14, x8 + mneg x5, x14, x9 + msub x10, x13, x16, x2 + msub x11, x13, x17, x3 + msub x12, x15, x16, x4 + msub x13, x15, x17, x5 + mov x22, x1 + subs x21, x21, #0x1 + b.ne Lp256_scalarmul_inv_loop + ldr x0, [sp] + ldr x1, [sp, #48] + mul x0, x0, x10 + madd x1, x1, x11, x0 + asr x0, x1, #63 + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + eor x14, x14, x0 + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + eor x15, x15, x0 + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + eor x16, x16, x0 + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + eor x17, x17, x0 + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + ldr x7, [sp, #96] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #128] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #96] + adc x2, x2, x1 + ldr x7, [sp, #104] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #136] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #104] + adc x6, x6, x1 + ldr x7, [sp, #112] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #144] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #112] + adc x5, x5, x1 + ldr x7, [sp, #120] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #152] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + ldp x0, x1, [sp, #96] + ldr x2, [sp, #112] + mov x14, #0xe000000000000000 + adds x0, x0, x14 + sbcs x1, x1, xzr + mov x11, #0x1fffffff + adcs x2, x2, x11 + mov x10, #0x2000000000000000 + adcs x5, x5, x10 + mov x14, #0x1fffffffe0000000 + adc x3, x3, x14 + lsl x11, x0, #32 + subs x14, x0, x11 + lsr x10, x0, #32 + sbc x0, x0, x10 + adds x1, x1, x11 + adcs x2, x2, x10 + adcs x5, x5, x14 + adcs x3, x3, x0 + mov x14, #0xffffffffffffffff + mov x11, #0xffffffff + mov x10, #0xffffffff00000001 + csel x14, x14, xzr, cs + csel x11, x11, xzr, cs + csel x10, x10, xzr, cs + subs x1, x1, x14 + sbcs x2, x2, x11 + sbcs x5, x5, xzr + sbc x3, x3, x10 + mov x10, #0xffffffffffffffff + subs x10, x1, x10 + mov x11, #0xffffffff + sbcs x11, x2, x11 + mov x13, #0xffffffff00000001 + sbcs x12, x5, xzr + sbcs x13, x3, x13 + csel x10, x1, x10, cc + csel x11, x2, x11, cc + csel x12, x5, x12, cc + csel x13, x3, x13, cc + stp x10, x11, [x20] + stp x12, x13, [x20, #16] + CFI_INC_SP(160) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_local_inv_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_local_montmul_p256) + +Lp256_scalarmul_local_montmul_p256: + CFI_START + ldr q20, [x2] + ldp x7, x17, [x1] + ldr q0, [x1] + ldp x6, x10, [x2] + ldp x11, x15, [x1, #16] + rev64 v16.4S, v20.4S + subs x4, x7, x17 + csetm x3, cc + cneg x13, x4, cc + mul v16.4S, v16.4S, v0.4S + umulh x12, x17, x10 + uzp1 v28.4S, v20.4S, v0.4S + subs x14, x11, x7 + ldr q20, [x2, #16] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2D, v16.4S + umulh x4, x7, x6 + uzp1 v21.4S, v0.4S, v0.4S + cneg x11, x8, cc + shl v17.2D, v27.2D, #32 + csetm x15, cc + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2D, v21.2S, v28.2S + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [x1, #16] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2S, v20.2D + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4S, v20.4S, v20.4S + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2S, v28.2D + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [x2, #16] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x9, x3, x13 + adcs x3, x8, x7 + umulh x8, x14, x11 + umull v21.2D, v0.2S, v1.2S + adcs x12, x10, x12 + umull v3.2D, v0.2S, v16.2S + adc x15, x15, xzr + rev64 v24.4S, v20.4S + stp x12, x15, [x0, #16] + movi v2.2D, #0x00000000ffffffff + mul x10, x14, x11 + mul v4.4S, v24.4S, v28.4S + subs x13, x14, x5 + uzp2 v19.4S, v28.4S, v28.4S + csetm x15, cc + usra v3.2D, v21.2D, #32 + mul x7, x5, x1 + umull v21.2D, v19.2S, v16.2S + cneg x13, x13, cc + uaddlp v5.2D, v4.4S + subs x11, x1, x11 + and v16.16B, v3.16B, v2.16B + umulh x5, x5, x1 + shl v24.2D, v5.2D, #32 + cneg x11, x11, cc + umlal v16.2D, v19.2S, v1.2S + cinv x12, x15, cc + umlal v24.2D, v0.2S, v1.2S + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + stp x9, x3, [x0] + usra v21.2D, v3.2D, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2D, v16.2D, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + ldp x15, x8, [x0] + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + ldp x9, x13, [x0, #16] + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x15 + adcs x15, x16, x8 + eor x5, x17, x4 + adcs x9, x1, x9 + eor x1, x10, x5 + adcs x16, x2, x13 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x11, x11, x13 + and x1, x1, x13 + adcs x4, x4, x1 + and x1, x12, x13 + stp x11, x4, [x0] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [x0, #16] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_local_montmul_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_local_montsqr_p256) + +Lp256_scalarmul_local_montsqr_p256: + CFI_START + ldr q19, [x1] + ldp x9, x13, [x1] + ldr q23, [x1, #16] + ldr q0, [x1] + ldp x1, x10, [x1, #16] + uzp2 v29.4S, v19.4S, v19.4S + xtn v4.2S, v19.2D + umulh x8, x9, x13 + rev64 v20.4S, v23.4S + umull v16.2D, v19.2S, v19.2S + umull v1.2D, v29.2S, v4.2S + mul v20.4S, v20.4S, v0.4S + subs x14, x9, x13 + umulh x15, x9, x1 + mov x16, v16.d[1] + umull2 v4.2D, v19.4S, v19.4S + mov x4, v16.d[0] + uzp1 v17.4S, v23.4S, v0.4S + uaddlp v19.2D, v20.4S + lsr x7, x8, #63 + mul x11, x9, x13 + mov x12, v1.d[0] + csetm x5, cc + cneg x6, x14, cc + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc + cinv x2, x5, cc + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4S, v0.4S, v0.4S + shl v19.2D, v19.2D, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2D, v20.2S, v17.2S + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x13, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x16, x3, x16, cs + csel x14, x8, x14, cs + csel x12, x11, x12, cs + csel x2, x5, x2, cs + stp x14, x12, [x0, #16] + stp x16, x2, [x0] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_local_montsqr_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_local_tomont_p256) + +Lp256_scalarmul_local_tomont_p256: + CFI_START + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + mov x1, #0xffffffffffffffff + mov x7, #0xffffffff + mov x9, #0xffffffff00000001 + subs x1, x2, x1 + sbcs x7, x3, x7 + sbcs x8, x4, xzr + sbcs x9, x5, x9 + csel x2, x2, x1, cc + csel x3, x3, x7, cc + csel x4, x4, x8, cc + csel x5, x5, x9, cc + cmp xzr, xzr + extr x9, x5, x4, #32 + adcs xzr, x4, x9 + lsr x9, x5, #32 + adcs x9, x5, x9 + csetm x6, cs + orr x9, x9, x6 + lsl x7, x9, #32 + lsr x8, x9, #32 + adds x4, x4, x7 + adc x5, x5, x8 + negs x6, x9 + sbcs x7, x7, xzr + sbc x8, x8, xzr + negs x6, x6 + sbcs x2, x2, x7 + sbcs x3, x3, x8 + sbcs x4, x4, x9 + sbcs x5, x5, x9 + adds x6, x6, x5 + mov x7, #0xffffffff + and x7, x7, x5 + adcs x2, x2, x7 + adcs x3, x3, xzr + mov x7, #0xffffffff00000001 + and x7, x7, x5 + adc x4, x4, x7 + cmp xzr, xzr + extr x9, x4, x3, #32 + adcs xzr, x3, x9 + lsr x9, x4, #32 + adcs x9, x4, x9 + csetm x5, cs + orr x9, x9, x5 + lsl x7, x9, #32 + lsr x8, x9, #32 + adds x3, x3, x7 + adc x4, x4, x8 + negs x5, x9 + sbcs x7, x7, xzr + sbc x8, x8, xzr + negs x5, x5 + sbcs x6, x6, x7 + sbcs x2, x2, x8 + sbcs x3, x3, x9 + sbcs x4, x4, x9 + adds x5, x5, x4 + mov x7, #0xffffffff + and x7, x7, x4 + adcs x6, x6, x7 + adcs x2, x2, xzr + mov x7, #0xffffffff00000001 + and x7, x7, x4 + adc x3, x3, x7 + cmp xzr, xzr + extr x9, x3, x2, #32 + adcs xzr, x2, x9 + lsr x9, x3, #32 + adcs x9, x3, x9 + csetm x4, cs + orr x9, x9, x4 + lsl x7, x9, #32 + lsr x8, x9, #32 + adds x2, x2, x7 + adc x3, x3, x8 + negs x4, x9 + sbcs x7, x7, xzr + sbc x8, x8, xzr + negs x4, x4 + sbcs x5, x5, x7 + sbcs x6, x6, x8 + sbcs x2, x2, x9 + sbcs x3, x3, x9 + adds x4, x4, x3 + mov x7, #0xffffffff + and x7, x7, x3 + adcs x5, x5, x7 + adcs x6, x6, xzr + mov x7, #0xffffffff00000001 + and x7, x7, x3 + adc x2, x2, x7 + cmp xzr, xzr + extr x9, x2, x6, #32 + adcs xzr, x6, x9 + lsr x9, x2, #32 + adcs x9, x2, x9 + csetm x3, cs + orr x9, x9, x3 + lsl x7, x9, #32 + lsr x8, x9, #32 + adds x6, x6, x7 + adc x2, x2, x8 + negs x3, x9 + sbcs x7, x7, xzr + sbc x8, x8, xzr + negs x3, x3 + sbcs x4, x4, x7 + sbcs x5, x5, x8 + sbcs x6, x6, x9 + sbcs x2, x2, x9 + adds x3, x3, x2 + mov x7, #0xffffffff + and x7, x7, x2 + adcs x4, x4, x7 + adcs x5, x5, xzr + mov x7, #0xffffffff00000001 + and x7, x7, x2 + adc x6, x6, x7 + stp x3, x4, [x0] + stp x5, x6, [x0, #16] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_local_tomont_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_local_p256_montjadd) + +Lp256_scalarmul_local_p256_montjadd: + CFI_START + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x26) + CFI_PUSH2(x27,x30) + CFI_DEC_SP(224) + mov x21, x0 + mov x22, x1 + mov x23, x2 + mov x0, sp + ldr q19, [x22, #64] + ldp x9, x13, [x22, #64] + ldr q23, [x22, #80] + ldr q0, [x22, #64] + ldp x1, x10, [x22, #80] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x9, x13 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x9, x13 + umulh x15, x9, x1 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x9, x13 + mov x12, v1.d[0] + csetm x5, cc // cc = lo, ul, last + cneg x6, x14, cc // cc = lo, ul, last + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc // cc = lo, ul, last + cinv x2, x5, cc // cc = lo, ul, last + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x13, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff // #4294967295 + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 // #-4294967295 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x19, x3, x16, cs // cs = hs, nlast + csel x14, x8, x14, cs // cs = hs, nlast + csel x12, x11, x12, cs // cs = hs, nlast + csel x20, x5, x2, cs // cs = hs, nlast + stp x14, x12, [x0, #16] + stp x19, x20, [x0] + ldr q19, [x23, #64] + ldp x9, x13, [x23, #64] + ldr q23, [x23, #80] + ldr q0, [x23, #64] + ldp x1, x10, [x23, #80] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x9, x13 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x9, x13 + umulh x15, x9, x1 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x9, x13 + mov x12, v1.d[0] + csetm x5, cc // cc = lo, ul, last + cneg x6, x14, cc // cc = lo, ul, last + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc // cc = lo, ul, last + cinv x2, x5, cc // cc = lo, ul, last + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x13, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff // #4294967295 + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 // #-4294967295 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x16, x3, x16, cs // cs = hs, nlast + csel x14, x8, x14, cs // cs = hs, nlast + csel x12, x11, x12, cs // cs = hs, nlast + csel x2, x5, x2, cs // cs = hs, nlast + stp x14, x12, [sp, #176] + stp x16, x2, [sp, #160] + ldr q20, [x22, #32] + ldp x7, x17, [x23, #64] + ldr q0, [x23, #64] + ldp x6, x10, [x22, #32] + ldp x11, x15, [x23, #80] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc // cc = lo, ul, last + cneg x13, x4, cc // cc = lo, ul, last + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [x22, #48] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc // cc = lo, ul, last + shl v17.2d, v27.2d, #32 + csetm x15, cc // cc = lo, ul, last + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc // cc = lo, ul, last + cinv x9, x3, cc // cc = lo, ul, last + cmn x17, #0x1 + ldr q28, [x23, #80] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [x22, #48] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc // cc = lo, ul, last + cneg x6, x6, cc // cc = lo, ul, last + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x24, x3, x13 + adcs x25, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc // cc = lo, ul, last + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc // cc = lo, ul, last + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc // cc = lo, ul, last + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc // cc = lo, ul, last + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x24 + adcs x15, x16, x25 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff // #4294967295 + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 // #-4294967295 + adds x11, x11, x13 + and x1, x1, x13 + adcs x4, x4, x1 + and x1, x12, x13 + stp x11, x4, [sp, #192] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #208] + ldr q20, [x23, #32] + ldp x7, x17, [x22, #64] + ldr q0, [x22, #64] + ldp x6, x10, [x23, #32] + ldp x11, x15, [x22, #80] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc // cc = lo, ul, last + cneg x13, x4, cc // cc = lo, ul, last + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [x23, #48] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc // cc = lo, ul, last + shl v17.2d, v27.2d, #32 + csetm x15, cc // cc = lo, ul, last + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc // cc = lo, ul, last + cinv x9, x3, cc // cc = lo, ul, last + cmn x17, #0x1 + ldr q28, [x22, #80] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [x23, #48] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc // cc = lo, ul, last + cneg x6, x6, cc // cc = lo, ul, last + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x24, x3, x13 + adcs x25, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc // cc = lo, ul, last + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc // cc = lo, ul, last + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc // cc = lo, ul, last + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc // cc = lo, ul, last + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x24 + adcs x15, x16, x25 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff // #4294967295 + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 // #-4294967295 + adds x24, x11, x13 + and x1, x1, x13 + adcs x25, x4, x1 + and x1, x12, x13 + stp x24, x25, [sp, #32] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #48] + mov x1, sp + ldr q20, [x23] + ldr q0, [x1] + ldp x6, x10, [x23] + ldp x11, x15, [x1, #16] + rev64 v16.4s, v20.4s + subs x4, x19, x20 + csetm x3, cc // cc = lo, ul, last + cneg x13, x4, cc // cc = lo, ul, last + mul v16.4s, v16.4s, v0.4s + umulh x12, x20, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x19 + ldr q20, [x23, #16] + sbcs x5, x15, x20 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x19, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc // cc = lo, ul, last + shl v17.2d, v27.2d, #32 + csetm x15, cc // cc = lo, ul, last + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc // cc = lo, ul, last + cinv x9, x3, cc // cc = lo, ul, last + cmn x17, #0x1 + ldr q28, [x1, #16] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [x23, #16] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc // cc = lo, ul, last + cneg x6, x6, cc // cc = lo, ul, last + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x19, x3, x13 + adcs x20, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc // cc = lo, ul, last + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc // cc = lo, ul, last + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc // cc = lo, ul, last + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc // cc = lo, ul, last + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x19 + adcs x15, x16, x20 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff // #4294967295 + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 // #-4294967295 + adds x11, x11, x13 + and x1, x1, x13 + adcs x4, x4, x1 + and x1, x12, x13 + stp x11, x4, [sp, #64] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #80] + ldr q20, [x22] + ldp x7, x17, [sp, #160] + ldr q0, [sp, #160] + ldp x6, x10, [x22] + ldp x11, x15, [sp, #176] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc // cc = lo, ul, last + cneg x13, x4, cc // cc = lo, ul, last + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [x22, #16] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc // cc = lo, ul, last + shl v17.2d, v27.2d, #32 + csetm x15, cc // cc = lo, ul, last + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc // cc = lo, ul, last + cinv x9, x3, cc // cc = lo, ul, last + cmn x17, #0x1 + ldr q28, [sp, #176] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [x22, #16] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc // cc = lo, ul, last + cneg x6, x6, cc // cc = lo, ul, last + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x19, x3, x13 + adcs x20, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc // cc = lo, ul, last + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc // cc = lo, ul, last + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc // cc = lo, ul, last + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc // cc = lo, ul, last + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x19 + adcs x15, x16, x20 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff // #4294967295 + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 // #-4294967295 + adds x11, x11, x13 + and x1, x1, x13 + adcs x4, x4, x1 + and x1, x12, x13 + stp x11, x4, [sp, #128] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #144] + mov x1, sp + ldr q20, [sp, #32] + ldp x7, x17, [x1] + ldr q0, [x1] + ldp x11, x15, [x1, #16] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc // cc = lo, ul, last + cneg x13, x4, cc // cc = lo, ul, last + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x25 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [sp, #48] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x24 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc // cc = lo, ul, last + shl v17.2d, v27.2d, #32 + csetm x15, cc // cc = lo, ul, last + subs x9, x25, x24 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc // cc = lo, ul, last + cinv x9, x3, cc // cc = lo, ul, last + cmn x17, #0x1 + ldr q28, [x1, #16] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #48] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x24, x7 + sbcs x9, x25, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc // cc = lo, ul, last + cneg x6, x6, cc // cc = lo, ul, last + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x19, x3, x13 + adcs x20, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x24, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x25, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc // cc = lo, ul, last + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc // cc = lo, ul, last + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc // cc = lo, ul, last + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc // cc = lo, ul, last + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x19 + adcs x15, x16, x20 + eor x5, x17, x4 + adcs x9, x1, x24 + eor x1, x10, x5 + adcs x16, x2, x25 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff // #4294967295 + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 // #-4294967295 + adds x19, x11, x13 + and x1, x1, x13 + adcs x20, x4, x1 + and x1, x12, x13 + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #48] + ldr q20, [sp, #192] + ldp x7, x17, [sp, #160] + ldr q0, [sp, #160] + ldp x6, x10, [sp, #192] + ldp x11, x15, [sp, #176] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc // cc = lo, ul, last + cneg x13, x4, cc // cc = lo, ul, last + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [sp, #208] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc // cc = lo, ul, last + shl v17.2d, v27.2d, #32 + csetm x15, cc // cc = lo, ul, last + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc // cc = lo, ul, last + cinv x9, x3, cc // cc = lo, ul, last + cmn x17, #0x1 + ldr q28, [sp, #176] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #208] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc // cc = lo, ul, last + cneg x6, x6, cc // cc = lo, ul, last + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x24, x3, x13 + adcs x25, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc // cc = lo, ul, last + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc // cc = lo, ul, last + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc // cc = lo, ul, last + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc // cc = lo, ul, last + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x24 + adcs x15, x16, x25 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff // #4294967295 + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 // #-4294967295 + adds x9, x11, x13 + and x1, x1, x13 + adcs x10, x4, x1 + and x1, x12, x13 + stp x9, x10, [sp, #192] + adcs x11, x7, xzr + adc x12, x17, x1 + stp x11, x12, [sp, #208] + ldp x5, x6, [sp, #64] + ldp x4, x3, [sp, #128] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [sp, #144] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc // cc = lo, ul, last + adds x13, x5, x3 + and x4, x3, #0xffffffff + adcs x24, x6, x4 + adcs x25, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x26, x8, x4 + stp x13, x24, [sp, #160] + stp x25, x26, [sp, #176] + subs x5, x19, x9 + sbcs x6, x20, x10 + ldp x7, x8, [sp, #48] + sbcs x7, x7, x11 + sbcs x8, x8, x12 + csetm x3, cc // cc = lo, ul, last + adds x19, x5, x3 + and x4, x3, #0xffffffff + adcs x20, x6, x4 + adcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x8, x8, x4 + stp x19, x20, [sp, #32] + stp x7, x8, [sp, #48] + ldr q19, [sp, #160] + ldr q23, [sp, #176] + ldr q0, [sp, #160] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x13, x24 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x13, x24 + umulh x15, x13, x25 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x13, x24 + mov x12, v1.d[0] + csetm x5, cc // cc = lo, ul, last + cneg x6, x14, cc // cc = lo, ul, last + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x26, x25 + mov x9, v1.d[1] + cneg x17, x2, cc // cc = lo, ul, last + cinv x2, x5, cc // cc = lo, ul, last + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x24, x26 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x25, x26 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x25, x26 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x26, x26 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x26, x26 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x25, x25 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x25, x25 + adcs x2, x2, x9 + mov x6, #0xffffffff // #4294967295 + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 // #-4294967295 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x24, x3, x16, cs // cs = hs, nlast + csel x25, x8, x14, cs // cs = hs, nlast + csel x26, x11, x12, cs // cs = hs, nlast + csel x27, x5, x2, cs // cs = hs, nlast + stp x25, x26, [sp, #112] + stp x24, x27, [sp, #96] + mov x0, sp + ldr q19, [sp, #32] + ldr q23, [sp, #48] + ldr q0, [sp, #32] + ldp x1, x10, [sp, #48] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x19, x20 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x19, x20 + umulh x15, x19, x1 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x19, x20 + mov x12, v1.d[0] + csetm x5, cc // cc = lo, ul, last + cneg x6, x14, cc // cc = lo, ul, last + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc // cc = lo, ul, last + cinv x2, x5, cc // cc = lo, ul, last + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x20, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff // #4294967295 + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 // #-4294967295 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x16, x3, x16, cs // cs = hs, nlast + csel x14, x8, x14, cs // cs = hs, nlast + csel x12, x11, x12, cs // cs = hs, nlast + csel x2, x5, x2, cs // cs = hs, nlast + stp x14, x12, [x0, #16] + stp x16, x2, [x0] + ldr q20, [sp, #128] + ldr q0, [sp, #96] + ldp x6, x10, [sp, #128] + rev64 v16.4s, v20.4s + subs x4, x24, x27 + csetm x3, cc // cc = lo, ul, last + cneg x13, x4, cc // cc = lo, ul, last + mul v16.4s, v16.4s, v0.4s + umulh x12, x27, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x25, x24 + ldr q20, [sp, #144] + sbcs x5, x26, x27 + ngc x17, xzr + subs x8, x25, x26 + uaddlp v27.2d, v16.4s + umulh x4, x24, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc // cc = lo, ul, last + shl v17.2d, v27.2d, #32 + csetm x15, cc // cc = lo, ul, last + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc // cc = lo, ul, last + cinv x9, x3, cc // cc = lo, ul, last + cmn x17, #0x1 + ldr q28, [sp, #112] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #144] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc // cc = lo, ul, last + cneg x6, x6, cc // cc = lo, ul, last + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x19, x3, x13 + adcs x20, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x25, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x26, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc // cc = lo, ul, last + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc // cc = lo, ul, last + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc // cc = lo, ul, last + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc // cc = lo, ul, last + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x19 + adcs x15, x16, x20 + eor x5, x17, x4 + adcs x9, x1, x25 + eor x1, x10, x5 + adcs x16, x2, x26 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff // #4294967295 + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 // #-4294967295 + adds x19, x11, x13 + and x1, x1, x13 + adcs x20, x4, x1 + and x1, x12, x13 + stp x19, x20, [sp, #128] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #144] + ldr q20, [sp, #64] + ldr q0, [sp, #96] + ldp x6, x10, [sp, #64] + ldp x11, x15, [sp, #112] + rev64 v16.4s, v20.4s + subs x4, x24, x27 + csetm x3, cc // cc = lo, ul, last + cneg x13, x4, cc // cc = lo, ul, last + mul v16.4s, v16.4s, v0.4s + umulh x12, x27, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x24 + ldr q20, [sp, #80] + sbcs x5, x15, x27 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x24, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc // cc = lo, ul, last + shl v17.2d, v27.2d, #32 + csetm x15, cc // cc = lo, ul, last + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc // cc = lo, ul, last + cinv x9, x3, cc // cc = lo, ul, last + cmn x17, #0x1 + ldr q28, [sp, #112] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #80] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc // cc = lo, ul, last + cneg x6, x6, cc // cc = lo, ul, last + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x24, x3, x13 + adcs x25, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc // cc = lo, ul, last + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc // cc = lo, ul, last + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc // cc = lo, ul, last + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc // cc = lo, ul, last + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x24 + adcs x15, x16, x25 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff // #4294967295 + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 // #-4294967295 + adds x9, x11, x13 + and x1, x1, x13 + adcs x10, x4, x1 + and x1, x12, x13 + stp x9, x10, [sp, #64] + adcs x11, x7, xzr + adc x12, x17, x1 + stp x11, x12, [sp, #80] + mov x0, sp + mov x1, sp + ldp x5, x6, [x1] + subs x5, x5, x19 + sbcs x6, x6, x20 + ldp x7, x8, [x1, #16] + ldp x4, x3, [sp, #144] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc // cc = lo, ul, last + adds x24, x5, x3 + and x4, x3, #0xffffffff + adcs x25, x6, x4 + adcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x8, x8, x4 + stp x7, x8, [x0, #16] + subs x5, x9, x19 + sbcs x6, x10, x20 + ldp x4, x3, [sp, #144] + sbcs x7, x11, x4 + sbcs x8, x12, x3 + csetm x3, cc // cc = lo, ul, last + adds x5, x5, x3 + and x4, x3, #0xffffffff + adcs x6, x6, x4 + adcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x8, x8, x4 + stp x5, x6, [sp, #96] + stp x7, x8, [sp, #112] + ldr q20, [x22, #64] + ldp x7, x17, [sp, #160] + ldr q0, [sp, #160] + ldp x6, x10, [x22, #64] + ldp x11, x15, [sp, #176] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc // cc = lo, ul, last + cneg x13, x4, cc // cc = lo, ul, last + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [x22, #80] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc // cc = lo, ul, last + shl v17.2d, v27.2d, #32 + csetm x15, cc // cc = lo, ul, last + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc // cc = lo, ul, last + cinv x9, x3, cc // cc = lo, ul, last + cmn x17, #0x1 + ldr q28, [sp, #176] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [x22, #80] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc // cc = lo, ul, last + cneg x6, x6, cc // cc = lo, ul, last + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x19, x3, x13 + adcs x20, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc // cc = lo, ul, last + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc // cc = lo, ul, last + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc // cc = lo, ul, last + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc // cc = lo, ul, last + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x19 + adcs x15, x16, x20 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff // #4294967295 + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 // #-4294967295 + adds x11, x11, x13 + and x1, x1, x13 + adcs x4, x4, x1 + and x1, x12, x13 + stp x11, x4, [sp, #160] + adcs x19, x7, xzr + adc x20, x17, x1 + stp x19, x20, [sp, #176] + mov x0, sp + mov x1, sp + ldp x4, x3, [sp, #64] + subs x5, x24, x4 + sbcs x6, x25, x3 + ldp x7, x8, [x1, #16] + ldp x4, x3, [sp, #80] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc // cc = lo, ul, last + adds x9, x5, x3 + and x4, x3, #0xffffffff + adcs x10, x6, x4 + adcs x11, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x3, x8, x4 + stp x9, x10, [x0] + stp x11, x3, [x0, #16] + ldp x5, x6, [sp, #128] + subs x5, x5, x9 + sbcs x6, x6, x10 + ldp x7, x8, [sp, #144] + sbcs x7, x7, x11 + sbcs x8, x8, x3 + csetm x3, cc // cc = lo, ul, last + adds x5, x5, x3 + and x4, x3, #0xffffffff + adcs x6, x6, x4 + adcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x8, x8, x4 + stp x5, x6, [sp, #128] + stp x7, x8, [sp, #144] + ldr q20, [sp, #192] + ldp x7, x17, [sp, #96] + ldr q0, [sp, #96] + ldp x6, x10, [sp, #192] + ldp x11, x15, [sp, #112] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc // cc = lo, ul, last + cneg x13, x4, cc // cc = lo, ul, last + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [sp, #208] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc // cc = lo, ul, last + shl v17.2d, v27.2d, #32 + csetm x15, cc // cc = lo, ul, last + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc // cc = lo, ul, last + cinv x9, x3, cc // cc = lo, ul, last + cmn x17, #0x1 + ldr q28, [sp, #112] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #208] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc // cc = lo, ul, last + cneg x6, x6, cc // cc = lo, ul, last + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x24, x3, x13 + adcs x25, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc // cc = lo, ul, last + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc // cc = lo, ul, last + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc // cc = lo, ul, last + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc // cc = lo, ul, last + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x24 + adcs x15, x16, x25 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff // #4294967295 + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 // #-4294967295 + adds x11, x11, x13 + and x1, x1, x13 + adcs x4, x4, x1 + and x1, x12, x13 + stp x11, x4, [sp, #96] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #112] + ldr q20, [x23, #64] + ldp x7, x17, [sp, #160] + ldr q0, [sp, #160] + ldp x6, x10, [x23, #64] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc // cc = lo, ul, last + cneg x13, x4, cc // cc = lo, ul, last + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x19, x7 + ldr q20, [x23, #80] + sbcs x5, x20, x17 + ngc x17, xzr + subs x8, x19, x20 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc // cc = lo, ul, last + shl v17.2d, v27.2d, #32 + csetm x15, cc // cc = lo, ul, last + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc // cc = lo, ul, last + cinv x9, x3, cc // cc = lo, ul, last + cmn x17, #0x1 + ldr q28, [sp, #176] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [x23, #80] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc // cc = lo, ul, last + cneg x6, x6, cc // cc = lo, ul, last + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x19, x3, x13 + adcs x20, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x24, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x25, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc // cc = lo, ul, last + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc // cc = lo, ul, last + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc // cc = lo, ul, last + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc // cc = lo, ul, last + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x19 + adcs x15, x16, x20 + eor x5, x17, x4 + adcs x9, x1, x24 + eor x1, x10, x5 + adcs x16, x2, x25 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff // #4294967295 + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 // #-4294967295 + adds x19, x11, x13 + and x1, x1, x13 + adcs x20, x4, x1 + and x1, x12, x13 + stp x19, x20, [sp, #160] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #176] + ldr q20, [sp, #128] + ldp x7, x17, [sp, #32] + ldr q0, [sp, #32] + ldp x6, x10, [sp, #128] + ldp x11, x15, [sp, #48] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc // cc = lo, ul, last + cneg x13, x4, cc // cc = lo, ul, last + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [sp, #144] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc // cc = lo, ul, last + shl v17.2d, v27.2d, #32 + csetm x15, cc // cc = lo, ul, last + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc // cc = lo, ul, last + cinv x9, x3, cc // cc = lo, ul, last + cmn x17, #0x1 + ldr q28, [sp, #48] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #144] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc // cc = lo, ul, last + cneg x6, x6, cc // cc = lo, ul, last + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x24, x3, x13 + adcs x25, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc // cc = lo, ul, last + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc // cc = lo, ul, last + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc // cc = lo, ul, last + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc // cc = lo, ul, last + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x24 + adcs x15, x16, x25 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff // #4294967295 + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 // #-4294967295 + adds x5, x11, x13 + and x1, x1, x13 + adcs x6, x4, x1 + and x1, x12, x13 + adcs x7, x7, xzr + adc x9, x17, x1 + ldp x4, x3, [sp, #96] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x4, x3, [sp, #112] + sbcs x7, x7, x4 + sbcs x8, x9, x3 + csetm x3, cc // cc = lo, ul, last + adds x15, x5, x3 + and x4, x3, #0xffffffff + adcs x24, x6, x4 + adcs x25, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x26, x8, x4 + stp x15, x24, [sp, #128] + stp x25, x26, [sp, #144] + ldp x0, x1, [x22, #64] + ldp x2, x3, [x22, #80] + orr x12, x0, x1 + orr x13, x2, x3 + orr x12, x12, x13 + cmp x12, xzr + cset x12, ne // ne = any + ldp x4, x5, [x23, #64] + ldp x6, x7, [x23, #80] + orr x13, x4, x5 + orr x14, x6, x7 + orr x13, x13, x14 + cmp x13, xzr + cset x13, ne // ne = any + cmp x13, x12 + csel x8, x0, x19, cc // cc = lo, ul, last + csel x9, x1, x20, cc // cc = lo, ul, last + csel x8, x4, x8, hi // hi = pmore + csel x9, x5, x9, hi // hi = pmore + ldp x10, x11, [sp, #176] + csel x10, x2, x10, cc // cc = lo, ul, last + csel x11, x3, x11, cc // cc = lo, ul, last + csel x10, x6, x10, hi // hi = pmore + csel x11, x7, x11, hi // hi = pmore + ldp x12, x13, [x22] + ldp x0, x1, [sp] + csel x0, x12, x0, cc // cc = lo, ul, last + csel x1, x13, x1, cc // cc = lo, ul, last + ldp x12, x13, [x23] + csel x0, x12, x0, hi // hi = pmore + csel x1, x13, x1, hi // hi = pmore + ldp x12, x13, [x22, #16] + ldp x2, x3, [sp, #16] + csel x2, x12, x2, cc // cc = lo, ul, last + csel x3, x13, x3, cc // cc = lo, ul, last + ldp x12, x13, [x23, #16] + csel x2, x12, x2, hi // hi = pmore + csel x3, x13, x3, hi // hi = pmore + ldp x12, x13, [x22, #32] + csel x4, x12, x15, cc // cc = lo, ul, last + csel x5, x13, x24, cc // cc = lo, ul, last + ldp x12, x13, [x23, #32] + csel x4, x12, x4, hi // hi = pmore + csel x5, x13, x5, hi // hi = pmore + ldp x12, x13, [x22, #48] + csel x6, x12, x25, cc // cc = lo, ul, last + csel x7, x13, x26, cc // cc = lo, ul, last + ldp x12, x13, [x23, #48] + csel x6, x12, x6, hi // hi = pmore + csel x7, x13, x7, hi // hi = pmore + stp x0, x1, [x21] + stp x2, x3, [x21, #16] + stp x4, x5, [x21, #32] + stp x6, x7, [x21, #48] + stp x8, x9, [x21, #64] + stp x10, x11, [x21, #80] + CFI_INC_SP(224) + CFI_POP2(x27,x30) + CFI_POP2(x25,x26) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_local_p256_montjadd) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_local_p256_montjdouble) + +Lp256_scalarmul_local_p256_montjdouble: + CFI_START + CFI_DEC_SP(272) + stp x19, x20, [sp, #192] + stp x21, x22, [sp, #208] + stp x23, x24, [sp, #224] + stp x25, x26, [sp, #240] + stp x27, xzr, [sp, #256] + mov x19, x0 + mov x20, x1 + mov x0, sp + ldr q19, [x20, #64] + ldp x9, x13, [x20, #64] + ldr q23, [x20, #80] + ldr q0, [x20, #64] + ldp x1, x10, [x20, #80] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x9, x13 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x9, x13 + umulh x15, x9, x1 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x9, x13 + mov x12, v1.d[0] + csetm x5, cc // cc = lo, ul, last + cneg x6, x14, cc // cc = lo, ul, last + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc // cc = lo, ul, last + cinv x2, x5, cc // cc = lo, ul, last + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x13, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff // #4294967295 + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 // #-4294967295 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x21, x3, x16, cs // cs = hs, nlast + csel x22, x8, x14, cs // cs = hs, nlast + csel x23, x11, x12, cs // cs = hs, nlast + csel x24, x5, x2, cs // cs = hs, nlast + stp x22, x23, [x0, #16] + stp x21, x24, [x0] + ldr q19, [x20, #32] + ldp x9, x13, [x20, #32] + ldr q23, [x20, #48] + ldr q0, [x20, #32] + ldp x1, x10, [x20, #48] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x9, x13 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x9, x13 + umulh x15, x9, x1 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x9, x13 + mov x12, v1.d[0] + csetm x5, cc // cc = lo, ul, last + cneg x6, x14, cc // cc = lo, ul, last + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc // cc = lo, ul, last + cinv x2, x5, cc // cc = lo, ul, last + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x13, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff // #4294967295 + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 // #-4294967295 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x16, x3, x16, cs // cs = hs, nlast + csel x14, x8, x14, cs // cs = hs, nlast + csel x12, x11, x12, cs // cs = hs, nlast + csel x2, x5, x2, cs // cs = hs, nlast + stp x14, x12, [sp, #48] + stp x16, x2, [sp, #32] + ldp x5, x6, [x20] + subs x5, x5, x21 + sbcs x6, x6, x24 + ldp x7, x8, [x20, #16] + sbcs x7, x7, x22 + sbcs x8, x8, x23 + csetm x3, cc // cc = lo, ul, last + adds x10, x5, x3 + and x4, x3, #0xffffffff + adcs x25, x6, x4 + adcs x26, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x27, x8, x4 + stp x10, x25, [sp, #96] + stp x26, x27, [sp, #112] + ldp x5, x6, [x20] + adds x5, x5, x21 + adcs x6, x6, x24 + ldp x7, x8, [x20, #16] + adcs x7, x7, x22 + adcs x8, x8, x23 + csetm x3, cs // cs = hs, nlast + subs x9, x5, x3 + and x1, x3, #0xffffffff + sbcs x5, x6, x1 + sbcs x7, x7, xzr + and x2, x3, #0xffffffff00000001 + sbc x8, x8, x2 + stp x9, x5, [sp, #64] + stp x7, x8, [sp, #80] + ldr q20, [sp, #96] + ldr q0, [sp, #64] + rev64 v16.4s, v20.4s + subs x4, x9, x5 + csetm x3, cc // cc = lo, ul, last + cneg x13, x4, cc // cc = lo, ul, last + mul v16.4s, v16.4s, v0.4s + umulh x12, x5, x25 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x7, x9 + ldr q20, [sp, #112] + sbcs x5, x8, x5 + ngc x17, xzr + subs x8, x7, x8 + uaddlp v27.2d, v16.4s + umulh x4, x9, x10 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc // cc = lo, ul, last + shl v17.2d, v27.2d, #32 + csetm x15, cc // cc = lo, ul, last + subs x9, x25, x10 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc // cc = lo, ul, last + cinv x9, x3, cc // cc = lo, ul, last + cmn x17, #0x1 + ldr q28, [sp, #80] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x10, x26 + sbcs x9, x25, x27 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x27, x26 + cinv x2, x15, cc // cc = lo, ul, last + cneg x6, x6, cc // cc = lo, ul, last + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x21, x3, x13 + adcs x22, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x23, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x24, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc // cc = lo, ul, last + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc // cc = lo, ul, last + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc // cc = lo, ul, last + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc // cc = lo, ul, last + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x21 + adcs x15, x16, x22 + eor x5, x17, x4 + adcs x9, x1, x23 + eor x1, x10, x5 + adcs x16, x2, x24 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff // #4294967295 + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 // #-4294967295 + adds x21, x11, x13 + and x1, x1, x13 + adcs x22, x4, x1 + and x1, x12, x13 + stp x21, x22, [sp, #96] + adcs x23, x7, xzr + adc x24, x17, x1 + stp x23, x24, [sp, #112] + ldp x4, x5, [x20, #32] + ldp x8, x9, [x20, #64] + adds x4, x4, x8 + adcs x5, x5, x9 + ldp x6, x7, [x20, #48] + ldp x10, x11, [x20, #80] + adcs x6, x6, x10 + adcs x7, x7, x11 + adc x3, xzr, xzr + adds x8, x4, #0x1 + mov x9, #0xffffffff // #4294967295 + sbcs x9, x5, x9 + sbcs x10, x6, xzr + mov x11, #0xffffffff00000001 // #-4294967295 + sbcs x11, x7, x11 + sbcs x3, x3, xzr + csel x4, x4, x8, cc // cc = lo, ul, last + csel x5, x5, x9, cc // cc = lo, ul, last + csel x6, x6, x10, cc // cc = lo, ul, last + csel x7, x7, x11, cc // cc = lo, ul, last + stp x4, x5, [sp, #64] + stp x6, x7, [sp, #80] + ldr q20, [sp, #32] + ldp x7, x17, [x20] + ldr q0, [x20] + ldp x6, x10, [sp, #32] + ldp x11, x15, [x20, #16] + rev64 v16.4s, v20.4s + subs x4, x7, x17 + csetm x3, cc // cc = lo, ul, last + cneg x13, x4, cc // cc = lo, ul, last + mul v16.4s, v16.4s, v0.4s + umulh x12, x17, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x11, x7 + ldr q20, [sp, #48] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2d, v16.4s + umulh x4, x7, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc // cc = lo, ul, last + shl v17.2d, v27.2d, #32 + csetm x15, cc // cc = lo, ul, last + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc // cc = lo, ul, last + cinv x9, x3, cc // cc = lo, ul, last + cmn x17, #0x1 + ldr q28, [x20, #16] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #48] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc // cc = lo, ul, last + cneg x6, x6, cc // cc = lo, ul, last + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x20, x3, x13 + adcs x25, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x26, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x27, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc // cc = lo, ul, last + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc // cc = lo, ul, last + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc // cc = lo, ul, last + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc // cc = lo, ul, last + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x20 + adcs x15, x16, x25 + eor x5, x17, x4 + adcs x9, x1, x26 + eor x1, x10, x5 + adcs x16, x2, x27 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff // #4294967295 + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 // #-4294967295 + adds x20, x11, x13 + and x1, x1, x13 + adcs x25, x4, x1 + and x1, x12, x13 + stp x20, x25, [sp, #128] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [sp, #144] + ldr q19, [sp, #96] + ldr q23, [sp, #112] + ldr q0, [sp, #96] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x21, x22 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x21, x22 + umulh x15, x21, x23 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x21, x22 + mov x12, v1.d[0] + csetm x5, cc // cc = lo, ul, last + cneg x6, x14, cc // cc = lo, ul, last + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x24, x23 + mov x9, v1.d[1] + cneg x17, x2, cc // cc = lo, ul, last + cinv x2, x5, cc // cc = lo, ul, last + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x22, x24 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x23, x24 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x23, x24 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x24, x24 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x24, x24 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x23, x23 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x23, x23 + adcs x2, x2, x9 + mov x6, #0xffffffff // #4294967295 + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 // #-4294967295 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x21, x3, x16, cs // cs = hs, nlast + csel x22, x8, x14, cs // cs = hs, nlast + csel x23, x11, x12, cs // cs = hs, nlast + csel x24, x5, x2, cs // cs = hs, nlast + ldr q19, [sp, #64] + ldp x9, x13, [sp, #64] + ldr q23, [sp, #80] + ldr q0, [sp, #64] + ldp x1, x10, [sp, #80] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x9, x13 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x9, x13 + umulh x15, x9, x1 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x9, x13 + mov x12, v1.d[0] + csetm x5, cc // cc = lo, ul, last + cneg x6, x14, cc // cc = lo, ul, last + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc // cc = lo, ul, last + cinv x2, x5, cc // cc = lo, ul, last + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x13, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff // #4294967295 + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 // #-4294967295 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x13, x3, x16, cs // cs = hs, nlast + csel x14, x8, x14, cs // cs = hs, nlast + csel x15, x11, x12, cs // cs = hs, nlast + csel x26, x5, x2, cs // cs = hs, nlast + mov x1, #0x9 // #9 + mov x2, #0xffffffffffffffff // #-1 + subs x9, x2, x21 + mov x2, #0xffffffff // #4294967295 + sbcs x10, x2, x24 + ngcs x11, x22 + mov x2, #0xffffffff00000001 // #-4294967295 + sbc x12, x2, x23 + mul x3, x1, x9 + mul x4, x1, x10 + mul x5, x1, x11 + mul x6, x1, x12 + umulh x9, x1, x9 + umulh x10, x1, x10 + umulh x11, x1, x11 + umulh x7, x1, x12 + adds x4, x4, x9 + adcs x5, x5, x10 + adcs x6, x6, x11 + adc x7, x7, xzr + mov x1, #0xc // #12 + mul x8, x20, x1 + umulh x9, x20, x1 + adds x3, x3, x8 + mul x8, x25, x1 + umulh x10, x25, x1 + adcs x4, x4, x8 + ldp x11, x12, [sp, #144] + mul x8, x11, x1 + umulh x11, x11, x1 + adcs x5, x5, x8 + mul x8, x12, x1 + umulh x12, x12, x1 + adcs x6, x6, x8 + adc x7, x7, xzr + adds x4, x4, x9 + adcs x5, x5, x10 + adcs x6, x6, x11 + adc x7, x7, x12 + add x8, x7, #0x1 + lsl x10, x8, #32 + adds x6, x6, x10 + adc x7, x7, xzr + neg x9, x8 + sub x10, x10, #0x1 + subs x3, x3, x9 + sbcs x4, x4, x10 + sbcs x5, x5, xzr + sbcs x6, x6, x8 + sbc x8, x7, x8 + adds x20, x3, x8 + and x9, x8, #0xffffffff + adcs x21, x4, x9 + adcs x22, x5, xzr + neg x10, x9 + adc x23, x6, x10 + stp x20, x21, [sp, #160] + stp x22, x23, [sp, #176] + mov x2, sp + ldp x4, x3, [x2] + subs x5, x13, x4 + sbcs x6, x26, x3 + ldp x4, x3, [x2, #16] + sbcs x7, x14, x4 + sbcs x8, x15, x3 + csetm x3, cc // cc = lo, ul, last + adds x5, x5, x3 + and x4, x3, #0xffffffff + adcs x6, x6, x4 + adcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x8, x8, x4 + stp x5, x6, [sp, #64] + stp x7, x8, [sp, #80] + mov x0, sp + ldr q19, [sp, #32] + ldp x9, x13, [sp, #32] + ldr q23, [sp, #48] + ldr q0, [sp, #32] + ldp x1, x10, [sp, #48] + uzp2 v29.4s, v19.4s, v19.4s + xtn v4.2s, v19.2d + umulh x8, x9, x13 + rev64 v20.4s, v23.4s + umull v16.2d, v19.2s, v19.2s + umull v1.2d, v29.2s, v4.2s + mul v20.4s, v20.4s, v0.4s + subs x14, x9, x13 + umulh x15, x9, x1 + mov x16, v16.d[1] + umull2 v4.2d, v19.4s, v19.4s + mov x4, v16.d[0] + uzp1 v17.4s, v23.4s, v0.4s + uaddlp v19.2d, v20.4s + lsr x7, x8, #63 + mul x11, x9, x13 + mov x12, v1.d[0] + csetm x5, cc // cc = lo, ul, last + cneg x6, x14, cc // cc = lo, ul, last + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc // cc = lo, ul, last + cinv x2, x5, cc // cc = lo, ul, last + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4s, v0.4s, v0.4s + shl v19.2d, v19.2d, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2d, v20.2s, v17.2s + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x13, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff // #4294967295 + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 // #-4294967295 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x24, x3, x16, cs // cs = hs, nlast + csel x25, x8, x14, cs // cs = hs, nlast + csel x26, x11, x12, cs // cs = hs, nlast + csel x27, x5, x2, cs // cs = hs, nlast + stp x25, x26, [x0, #16] + stp x24, x27, [x0] + ldr q20, [sp, #96] + ldr q0, [sp, #160] + ldp x6, x10, [sp, #96] + rev64 v16.4s, v20.4s + subs x4, x20, x21 + csetm x3, cc // cc = lo, ul, last + cneg x13, x4, cc // cc = lo, ul, last + mul v16.4s, v16.4s, v0.4s + umulh x12, x21, x10 + uzp1 v28.4s, v20.4s, v0.4s + subs x14, x22, x20 + ldr q20, [sp, #112] + sbcs x5, x23, x21 + ngc x17, xzr + subs x8, x22, x23 + uaddlp v27.2d, v16.4s + umulh x4, x20, x6 + uzp1 v21.4s, v0.4s, v0.4s + cneg x11, x8, cc // cc = lo, ul, last + shl v17.2d, v27.2d, #32 + csetm x15, cc // cc = lo, ul, last + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2d, v21.2s, v28.2s + cneg x8, x9, cc // cc = lo, ul, last + cinv x9, x3, cc // cc = lo, ul, last + cmn x17, #0x1 + ldr q28, [sp, #176] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2s, v20.2d + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4s, v20.4s, v20.4s + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2s, v28.2d + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [sp, #112] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc // cc = lo, ul, last + cneg x6, x6, cc // cc = lo, ul, last + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x20, x3, x13 + adcs x21, x8, x7 + umulh x8, x14, x11 + umull v21.2d, v0.2s, v1.2s + adcs x22, x10, x12 + umull v3.2d, v0.2s, v16.2s + adc x23, x15, xzr + rev64 v24.4s, v20.4s + movi v2.2d, #0xffffffff + mul x10, x14, x11 + mul v4.4s, v24.4s, v28.4s + subs x13, x14, x5 + uzp2 v19.4s, v28.4s, v28.4s + csetm x15, cc // cc = lo, ul, last + usra v3.2d, v21.2d, #32 + mul x7, x5, x1 + umull v21.2d, v19.2s, v16.2s + cneg x13, x13, cc // cc = lo, ul, last + uaddlp v5.2d, v4.4s + subs x11, x1, x11 + and v16.16b, v3.16b, v2.16b + umulh x5, x5, x1 + shl v24.2d, v5.2d, #32 + cneg x11, x11, cc // cc = lo, ul, last + umlal v16.2d, v19.2s, v1.2s + cinv x12, x15, cc // cc = lo, ul, last + umlal v24.2d, v0.2s, v1.2s + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + usra v21.2d, v3.2d, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2d, v16.2d, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x20 + adcs x15, x16, x21 + eor x5, x17, x4 + adcs x9, x1, x22 + eor x1, x10, x5 + adcs x16, x2, x23 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff // #4294967295 + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 // #-4294967295 + adds x14, x11, x13 + and x1, x1, x13 + adcs x15, x4, x1 + and x1, x12, x13 + stp x14, x15, [sp, #96] + adcs x13, x7, xzr + adc x20, x17, x1 + stp x13, x20, [sp, #112] + ldp x5, x6, [sp, #64] + ldp x4, x3, [sp, #32] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [sp, #48] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc // cc = lo, ul, last + adds x5, x5, x3 + and x4, x3, #0xffffffff + adcs x6, x6, x4 + adcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x8, x8, x4 + stp x5, x6, [x19, #64] + stp x7, x8, [x19, #80] + ldp x1, x2, [sp, #128] + lsl x0, x1, #2 + ldp x6, x7, [sp, #160] + subs x0, x0, x6 + extr x1, x2, x1, #62 + sbcs x1, x1, x7 + ldp x3, x4, [sp, #144] + extr x2, x3, x2, #62 + ldp x6, x7, [sp, #176] + sbcs x2, x2, x6 + extr x3, x4, x3, #62 + sbcs x3, x3, x7 + lsr x4, x4, #62 + sbc x4, x4, xzr + add x5, x4, #0x1 + lsl x8, x5, #32 + negs x6, x8 + ngcs x7, xzr + sbc x8, x8, x5 + adds x0, x0, x5 + adcs x1, x1, x6 + adcs x2, x2, x7 + adcs x3, x3, x8 + csetm x5, cc // cc = lo, ul, last + adds x0, x0, x5 + and x6, x5, #0xffffffff + adcs x1, x1, x6 + adcs x2, x2, xzr + neg x7, x6 + adc x3, x3, x7 + stp x0, x1, [x19] + stp x2, x3, [x19, #16] + mov x2, #0xffffffffffffffff // #-1 + subs x9, x2, x24 + mov x2, #0xffffffff // #4294967295 + sbcs x10, x2, x27 + ngcs x11, x25 + mov x2, #0xffffffff00000001 // #-4294967295 + sbc x12, x2, x26 + lsl x3, x9, #3 + extr x4, x10, x9, #61 + extr x5, x11, x10, #61 + extr x6, x12, x11, #61 + lsr x7, x12, #61 + mov x1, #0x3 // #3 + mul x8, x14, x1 + umulh x9, x14, x1 + adds x3, x3, x8 + mul x8, x15, x1 + umulh x10, x15, x1 + adcs x4, x4, x8 + mul x8, x13, x1 + umulh x11, x13, x1 + adcs x5, x5, x8 + mul x8, x20, x1 + umulh x12, x20, x1 + adcs x6, x6, x8 + adc x7, x7, xzr + adds x4, x4, x9 + adcs x5, x5, x10 + adcs x6, x6, x11 + adc x7, x7, x12 + add x8, x7, #0x1 + lsl x10, x8, #32 + adds x6, x6, x10 + adc x7, x7, xzr + neg x9, x8 + sub x10, x10, #0x1 + subs x3, x3, x9 + sbcs x4, x4, x10 + sbcs x5, x5, xzr + sbcs x6, x6, x8 + sbc x8, x7, x8 + adds x3, x3, x8 + and x9, x8, #0xffffffff + adcs x4, x4, x9 + adcs x5, x5, xzr + neg x10, x9 + adc x6, x6, x10 + stp x3, x4, [x19, #32] + stp x5, x6, [x19, #48] + ldp x27, xzr, [sp, #256] + ldp x25, x26, [sp, #240] + ldp x23, x24, [sp, #224] + ldp x21, x22, [sp, #208] + ldp x19, x20, [sp, #192] + CFI_INC_SP(272) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_local_p256_montjdouble) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_local_p256_montjmixadd) + +Lp256_scalarmul_local_p256_montjmixadd: + CFI_START + CFI_PUSH2(x19,x20) + CFI_DEC_SP(192) + mov x17, x0 + mov x19, x1 + mov x20, x2 + ldp x2, x3, [x19, #64] + ldp x4, x5, [x19, #80] + umull x15, w2, w2 + lsr x11, x2, #32 + umull x16, w11, w11 + umull x11, w2, w11 + adds x15, x15, x11, lsl #33 + lsr x11, x11, #31 + adc x16, x16, x11 + umull x0, w3, w3 + lsr x11, x3, #32 + umull x1, w11, w11 + umull x11, w3, w11 + mul x12, x2, x3 + umulh x13, x2, x3 + adds x0, x0, x11, lsl #33 + lsr x11, x11, #31 + adc x1, x1, x11 + adds x12, x12, x12 + adcs x13, x13, x13 + adc x1, x1, xzr + adds x16, x16, x12 + adcs x0, x0, x13 + adc x1, x1, xzr + lsl x12, x15, #32 + subs x13, x15, x12 + lsr x11, x15, #32 + sbc x15, x15, x11 + adds x16, x16, x12 + adcs x0, x0, x11 + adcs x1, x1, x13 + adc x15, x15, xzr + lsl x12, x16, #32 + subs x13, x16, x12 + lsr x11, x16, #32 + sbc x16, x16, x11 + adds x0, x0, x12 + adcs x1, x1, x11 + adcs x15, x15, x13 + adc x16, x16, xzr + mul x6, x2, x4 + mul x14, x3, x5 + umulh x8, x2, x4 + subs x10, x2, x3 + cneg x10, x10, cc + csetm x13, cc + subs x12, x5, x4 + cneg x12, x12, cc + mul x11, x10, x12 + umulh x12, x10, x12 + cinv x13, x13, cc + eor x11, x11, x13 + eor x12, x12, x13 + adds x7, x6, x8 + adc x8, x8, xzr + umulh x9, x3, x5 + adds x7, x7, x14 + adcs x8, x8, x9 + adc x9, x9, xzr + adds x8, x8, x14 + adc x9, x9, xzr + cmn x13, #0x1 + adcs x7, x7, x11 + adcs x8, x8, x12 + adc x9, x9, x13 + adds x6, x6, x6 + adcs x7, x7, x7 + adcs x8, x8, x8 + adcs x9, x9, x9 + adc x10, xzr, xzr + adds x6, x6, x0 + adcs x7, x7, x1 + adcs x8, x8, x15 + adcs x9, x9, x16 + adc x10, x10, xzr + lsl x12, x6, #32 + subs x13, x6, x12 + lsr x11, x6, #32 + sbc x6, x6, x11 + adds x7, x7, x12 + adcs x8, x8, x11 + adcs x9, x9, x13 + adcs x10, x10, x6 + adc x6, xzr, xzr + lsl x12, x7, #32 + subs x13, x7, x12 + lsr x11, x7, #32 + sbc x7, x7, x11 + adds x8, x8, x12 + adcs x9, x9, x11 + adcs x10, x10, x13 + adcs x6, x6, x7 + adc x7, xzr, xzr + mul x11, x4, x4 + adds x8, x8, x11 + mul x12, x5, x5 + umulh x11, x4, x4 + adcs x9, x9, x11 + adcs x10, x10, x12 + umulh x12, x5, x5 + adcs x6, x6, x12 + adc x7, x7, xzr + mul x11, x4, x5 + umulh x12, x4, x5 + adds x11, x11, x11 + adcs x12, x12, x12 + adc x13, xzr, xzr + adds x9, x9, x11 + adcs x10, x10, x12 + adcs x6, x6, x13 + adcs x7, x7, xzr + mov x11, #0xffffffff + adds x5, x8, #0x1 + sbcs x11, x9, x11 + mov x13, #0xffffffff00000001 + sbcs x12, x10, xzr + sbcs x13, x6, x13 + sbcs xzr, x7, xzr + csel x8, x5, x8, cs + csel x9, x11, x9, cs + csel x10, x12, x10, cs + csel x6, x13, x6, cs + stp x8, x9, [sp] + stp x10, x6, [sp, #16] + ldp x3, x4, [x19, #64] + ldp x5, x6, [x19, #80] + ldp x7, x8, [x20, #32] + ldp x9, x10, [x20, #48] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #32] + stp x11, x12, [sp, #48] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #32] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #48] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #32] + stp x3, x4, [sp, #48] + ldp x3, x4, [sp] + ldp x5, x6, [sp, #16] + ldp x7, x8, [x20] + ldp x9, x10, [x20, #16] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #64] + stp x11, x12, [sp, #80] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #64] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #80] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #64] + stp x3, x4, [sp, #80] + ldp x3, x4, [sp] + ldp x5, x6, [sp, #16] + ldp x7, x8, [sp, #32] + ldp x9, x10, [sp, #48] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #32] + stp x11, x12, [sp, #48] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #32] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #48] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #32] + stp x3, x4, [sp, #48] + ldp x5, x6, [sp, #64] + ldp x4, x3, [x19] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [x19, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #160] + stp x7, x8, [sp, #176] + ldp x5, x6, [sp, #32] + ldp x4, x3, [x19, #32] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #48] + ldp x4, x3, [x19, #48] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #32] + stp x7, x8, [sp, #48] + ldp x2, x3, [sp, #160] + ldp x4, x5, [sp, #176] + umull x15, w2, w2 + lsr x11, x2, #32 + umull x16, w11, w11 + umull x11, w2, w11 + adds x15, x15, x11, lsl #33 + lsr x11, x11, #31 + adc x16, x16, x11 + umull x0, w3, w3 + lsr x11, x3, #32 + umull x1, w11, w11 + umull x11, w3, w11 + mul x12, x2, x3 + umulh x13, x2, x3 + adds x0, x0, x11, lsl #33 + lsr x11, x11, #31 + adc x1, x1, x11 + adds x12, x12, x12 + adcs x13, x13, x13 + adc x1, x1, xzr + adds x16, x16, x12 + adcs x0, x0, x13 + adc x1, x1, xzr + lsl x12, x15, #32 + subs x13, x15, x12 + lsr x11, x15, #32 + sbc x15, x15, x11 + adds x16, x16, x12 + adcs x0, x0, x11 + adcs x1, x1, x13 + adc x15, x15, xzr + lsl x12, x16, #32 + subs x13, x16, x12 + lsr x11, x16, #32 + sbc x16, x16, x11 + adds x0, x0, x12 + adcs x1, x1, x11 + adcs x15, x15, x13 + adc x16, x16, xzr + mul x6, x2, x4 + mul x14, x3, x5 + umulh x8, x2, x4 + subs x10, x2, x3 + cneg x10, x10, cc + csetm x13, cc + subs x12, x5, x4 + cneg x12, x12, cc + mul x11, x10, x12 + umulh x12, x10, x12 + cinv x13, x13, cc + eor x11, x11, x13 + eor x12, x12, x13 + adds x7, x6, x8 + adc x8, x8, xzr + umulh x9, x3, x5 + adds x7, x7, x14 + adcs x8, x8, x9 + adc x9, x9, xzr + adds x8, x8, x14 + adc x9, x9, xzr + cmn x13, #0x1 + adcs x7, x7, x11 + adcs x8, x8, x12 + adc x9, x9, x13 + adds x6, x6, x6 + adcs x7, x7, x7 + adcs x8, x8, x8 + adcs x9, x9, x9 + adc x10, xzr, xzr + adds x6, x6, x0 + adcs x7, x7, x1 + adcs x8, x8, x15 + adcs x9, x9, x16 + adc x10, x10, xzr + lsl x12, x6, #32 + subs x13, x6, x12 + lsr x11, x6, #32 + sbc x6, x6, x11 + adds x7, x7, x12 + adcs x8, x8, x11 + adcs x9, x9, x13 + adcs x10, x10, x6 + adc x6, xzr, xzr + lsl x12, x7, #32 + subs x13, x7, x12 + lsr x11, x7, #32 + sbc x7, x7, x11 + adds x8, x8, x12 + adcs x9, x9, x11 + adcs x10, x10, x13 + adcs x6, x6, x7 + adc x7, xzr, xzr + mul x11, x4, x4 + adds x8, x8, x11 + mul x12, x5, x5 + umulh x11, x4, x4 + adcs x9, x9, x11 + adcs x10, x10, x12 + umulh x12, x5, x5 + adcs x6, x6, x12 + adc x7, x7, xzr + mul x11, x4, x5 + umulh x12, x4, x5 + adds x11, x11, x11 + adcs x12, x12, x12 + adc x13, xzr, xzr + adds x9, x9, x11 + adcs x10, x10, x12 + adcs x6, x6, x13 + adcs x7, x7, xzr + mov x11, #0xffffffff + adds x5, x8, #0x1 + sbcs x11, x9, x11 + mov x13, #0xffffffff00000001 + sbcs x12, x10, xzr + sbcs x13, x6, x13 + sbcs xzr, x7, xzr + csel x8, x5, x8, cs + csel x9, x11, x9, cs + csel x10, x12, x10, cs + csel x6, x13, x6, cs + stp x8, x9, [sp, #96] + stp x10, x6, [sp, #112] + ldp x2, x3, [sp, #32] + ldp x4, x5, [sp, #48] + umull x15, w2, w2 + lsr x11, x2, #32 + umull x16, w11, w11 + umull x11, w2, w11 + adds x15, x15, x11, lsl #33 + lsr x11, x11, #31 + adc x16, x16, x11 + umull x0, w3, w3 + lsr x11, x3, #32 + umull x1, w11, w11 + umull x11, w3, w11 + mul x12, x2, x3 + umulh x13, x2, x3 + adds x0, x0, x11, lsl #33 + lsr x11, x11, #31 + adc x1, x1, x11 + adds x12, x12, x12 + adcs x13, x13, x13 + adc x1, x1, xzr + adds x16, x16, x12 + adcs x0, x0, x13 + adc x1, x1, xzr + lsl x12, x15, #32 + subs x13, x15, x12 + lsr x11, x15, #32 + sbc x15, x15, x11 + adds x16, x16, x12 + adcs x0, x0, x11 + adcs x1, x1, x13 + adc x15, x15, xzr + lsl x12, x16, #32 + subs x13, x16, x12 + lsr x11, x16, #32 + sbc x16, x16, x11 + adds x0, x0, x12 + adcs x1, x1, x11 + adcs x15, x15, x13 + adc x16, x16, xzr + mul x6, x2, x4 + mul x14, x3, x5 + umulh x8, x2, x4 + subs x10, x2, x3 + cneg x10, x10, cc + csetm x13, cc + subs x12, x5, x4 + cneg x12, x12, cc + mul x11, x10, x12 + umulh x12, x10, x12 + cinv x13, x13, cc + eor x11, x11, x13 + eor x12, x12, x13 + adds x7, x6, x8 + adc x8, x8, xzr + umulh x9, x3, x5 + adds x7, x7, x14 + adcs x8, x8, x9 + adc x9, x9, xzr + adds x8, x8, x14 + adc x9, x9, xzr + cmn x13, #0x1 + adcs x7, x7, x11 + adcs x8, x8, x12 + adc x9, x9, x13 + adds x6, x6, x6 + adcs x7, x7, x7 + adcs x8, x8, x8 + adcs x9, x9, x9 + adc x10, xzr, xzr + adds x6, x6, x0 + adcs x7, x7, x1 + adcs x8, x8, x15 + adcs x9, x9, x16 + adc x10, x10, xzr + lsl x12, x6, #32 + subs x13, x6, x12 + lsr x11, x6, #32 + sbc x6, x6, x11 + adds x7, x7, x12 + adcs x8, x8, x11 + adcs x9, x9, x13 + adcs x10, x10, x6 + adc x6, xzr, xzr + lsl x12, x7, #32 + subs x13, x7, x12 + lsr x11, x7, #32 + sbc x7, x7, x11 + adds x8, x8, x12 + adcs x9, x9, x11 + adcs x10, x10, x13 + adcs x6, x6, x7 + adc x7, xzr, xzr + mul x11, x4, x4 + adds x8, x8, x11 + mul x12, x5, x5 + umulh x11, x4, x4 + adcs x9, x9, x11 + adcs x10, x10, x12 + umulh x12, x5, x5 + adcs x6, x6, x12 + adc x7, x7, xzr + mul x11, x4, x5 + umulh x12, x4, x5 + adds x11, x11, x11 + adcs x12, x12, x12 + adc x13, xzr, xzr + adds x9, x9, x11 + adcs x10, x10, x12 + adcs x6, x6, x13 + adcs x7, x7, xzr + mov x11, #0xffffffff + adds x5, x8, #0x1 + sbcs x11, x9, x11 + mov x13, #0xffffffff00000001 + sbcs x12, x10, xzr + sbcs x13, x6, x13 + sbcs xzr, x7, xzr + csel x8, x5, x8, cs + csel x9, x11, x9, cs + csel x10, x12, x10, cs + csel x6, x13, x6, cs + stp x8, x9, [sp] + stp x10, x6, [sp, #16] + ldp x3, x4, [sp, #96] + ldp x5, x6, [sp, #112] + ldp x7, x8, [x19] + ldp x9, x10, [x19, #16] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #128] + stp x11, x12, [sp, #144] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #128] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #144] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #128] + stp x3, x4, [sp, #144] + ldp x3, x4, [sp, #96] + ldp x5, x6, [sp, #112] + ldp x7, x8, [sp, #64] + ldp x9, x10, [sp, #80] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #64] + stp x11, x12, [sp, #80] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #64] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #80] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #64] + stp x3, x4, [sp, #80] + ldp x5, x6, [sp] + ldp x4, x3, [sp, #128] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #16] + ldp x4, x3, [sp, #144] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp] + stp x7, x8, [sp, #16] + ldp x5, x6, [sp, #64] + ldp x4, x3, [sp, #128] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [sp, #144] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #96] + stp x7, x8, [sp, #112] + ldp x3, x4, [sp, #160] + ldp x5, x6, [sp, #176] + ldp x7, x8, [x19, #64] + ldp x9, x10, [x19, #80] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #160] + stp x11, x12, [sp, #176] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #160] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #176] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #160] + stp x3, x4, [sp, #176] + ldp x5, x6, [sp] + ldp x4, x3, [sp, #64] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #16] + ldp x4, x3, [sp, #80] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp] + stp x7, x8, [sp, #16] + ldp x5, x6, [sp, #128] + ldp x4, x3, [sp] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #144] + ldp x4, x3, [sp, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #128] + stp x7, x8, [sp, #144] + ldp x3, x4, [sp, #96] + ldp x5, x6, [sp, #112] + ldp x7, x8, [x19, #32] + ldp x9, x10, [x19, #48] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #96] + stp x11, x12, [sp, #112] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #96] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #112] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #96] + stp x3, x4, [sp, #112] + ldp x3, x4, [sp, #32] + ldp x5, x6, [sp, #48] + ldp x7, x8, [sp, #128] + ldp x9, x10, [sp, #144] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #128] + stp x11, x12, [sp, #144] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #128] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #144] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #128] + stp x3, x4, [sp, #144] + ldp x5, x6, [sp, #128] + ldp x4, x3, [sp, #96] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #144] + ldp x4, x3, [sp, #112] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #128] + stp x7, x8, [sp, #144] + ldp x0, x1, [x19, #64] + ldp x2, x3, [x19, #80] + orr x4, x0, x1 + orr x5, x2, x3 + orr x4, x4, x5 + cmp x4, xzr + ldp x0, x1, [sp] + ldp x12, x13, [x20] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x2, x3, [sp, #16] + ldp x12, x13, [x20, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x4, x5, [sp, #128] + ldp x12, x13, [x20, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x6, x7, [sp, #144] + ldp x12, x13, [x20, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x8, x9, [sp, #160] + mov x12, #0x1 + mov x13, #0xffffffff00000000 + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x10, x11, [sp, #176] + mov x12, #0xffffffffffffffff + mov x13, #0xfffffffe + csel x10, x10, x12, ne + csel x11, x11, x13, ne + stp x0, x1, [x17] + stp x2, x3, [x17, #16] + stp x4, x5, [x17, #32] + stp x6, x7, [x17, #48] + stp x8, x9, [x17, #64] + stp x10, x11, [x17, #80] + CFI_INC_SP(192) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_local_p256_montjmixadd) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/p256_scalarmul_alt.S b/cbits/s2n/arm/p256_scalarmul_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/p256_scalarmul_alt.S @@ -0,0 +1,6235 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Scalar multiplication for P-256 +// Input scalar[4], point[8]; output res[8] +// +// extern void p256_scalarmul_alt +// (uint64_t res[static 8], +// const uint64_t scalar[static 4], +// const uint64_t point[static 8]); +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-256, returns the point (X,Y) = n * P. The input and output +// are affine points, and in the case of the point at infinity as +// the result, (0,0) is returned. +// +// Standard ARM ABI: X0 = res, X1 = scalar, X2 = point +// ---------------------------------------------------------------------------- + + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_scalarmul_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_scalarmul_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_scalarmul_alt) + + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Safe copies of inputs (res lasts the whole code, point not so long) +// and additional values in variables, with some aliasing + +#define res x19 +#define sgn x20 +#define j x20 +#define point x21 + +// Intermediate variables on the stack. The last z2, z3 values can +// safely be overlaid on the table, which is no longer needed at the end. + +#define scalarb sp, #(0*NUMSIZE) +#define acc sp, #(1*NUMSIZE) +#define tabent sp, #(4*NUMSIZE) + +#define tab sp, #(7*NUMSIZE) + +#define z2 sp, #(7*NUMSIZE) +#define z3 sp, #(8*NUMSIZE) + +#define NSPACE 31*NUMSIZE + +// Loading large constants + +#define movbig(nn,n3,n2,n1,n0) \ + movz nn, n0 __LF \ + movk nn, n1, lsl #16 __LF \ + movk nn, n2, lsl #32 __LF \ + movk nn, n3, lsl #48 + +S2N_BN_SYMBOL(p256_scalarmul_alt): + CFI_START + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x30) + CFI_DEC_SP(NSPACE) + +// Preserve the "res" and "point" input arguments. We load and process the +// scalar immediately so we don't bother preserving that input argument. +// Also, "point" is only needed early on and so its register gets re-used. + + mov res, x0 + mov point, x2 + +// Load the digits of group order n_256 = [x12;x13;x14;x15] + + movbig(x12, #0xf3b9, #0xcac2, #0xfc63, #0x2551) + movbig(x13, #0xbce6, #0xfaad, #0xa717, #0x9e84) + mov x14, #0xffffffffffffffff + mov x15, #0xffffffff00000000 + +// First, reduce the input scalar mod n_256, i.e. conditionally subtract n_256 + + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + + subs x6, x2, x12 + sbcs x7, x3, x13 + sbcs x8, x4, x14 + sbcs x9, x5, x15 + + csel x2, x2, x6, cc + csel x3, x3, x7, cc + csel x4, x4, x8, cc + csel x5, x5, x9, cc + +// Now if the top bit of the reduced scalar is set, negate it mod n_256, +// i.e. do n |-> n_256 - n. Remember the sign as "sgn" so we can +// correspondingly negate the point below. + + subs x6, x12, x2 + sbcs x7, x13, x3 + sbcs x8, x14, x4 + sbc x9, x15, x5 + + tst x5, #0x8000000000000000 + csel x2, x2, x6, eq + csel x3, x3, x7, eq + csel x4, x4, x8, eq + csel x5, x5, x9, eq + cset sgn, ne + +// In either case then add the recoding constant 0x08888...888 to allow +// signed digits. + + mov x6, 0x8888888888888888 + adds x2, x2, x6 + adcs x3, x3, x6 + bic x7, x6, #0xF000000000000000 + adcs x4, x4, x6 + adc x5, x5, x7 + + stp x2, x3, [scalarb] + stp x4, x5, [scalarb+16] + +// Set the tab[0] table entry to Montgomery-Jacobian point = 1 * P +// The z coordinate is just the Montgomery form of the constant 1. + + add x0, tab + mov x1, point + CFI_BL(Lp256_scalarmul_alt_local_tomont_p256) + + add x1, point, #32 + add x0, tab+32 + CFI_BL(Lp256_scalarmul_alt_local_tomont_p256) + + mov x0, #0x0000000000000001 + mov x1, #0xffffffff00000000 + stp x0, x1, [tab+64] + mov x2, #0xffffffffffffffff + mov x3, #0x00000000fffffffe + stp x2, x3, [tab+80] + +// If the top bit of the scalar was set, negate (y coordinate of) the point + + ldp x4, x5, [tab+32] + ldp x6, x7, [tab+48] + + mov x0, 0xffffffffffffffff + subs x0, x0, x4 + mov x1, 0x00000000ffffffff + sbcs x1, x1, x5 + mov x3, 0xffffffff00000001 + sbcs x2, xzr, x6 + sbc x3, x3, x7 + + cmp sgn, xzr + csel x4, x0, x4, ne + csel x5, x1, x5, ne + csel x6, x2, x6, ne + csel x7, x3, x7, ne + + stp x4, x5, [tab+32] + stp x6, x7, [tab+48] + +// Compute and record tab[1] = 2 * p, ..., tab[7] = 8 * P + + add x0, tab+96*1 + add x1, tab + CFI_BL(Lp256_scalarmul_alt_local_p256_montjdouble) + + add x0, tab+96*2 + add x1, tab+96*1 + add x2, tab + CFI_BL(Lp256_scalarmul_alt_local_p256_montjmixadd) + + add x0, tab+96*3 + add x1, tab+96*1 + CFI_BL(Lp256_scalarmul_alt_local_p256_montjdouble) + + add x0, tab+96*4 + add x1, tab+96*3 + add x2, tab + CFI_BL(Lp256_scalarmul_alt_local_p256_montjmixadd) + + add x0, tab+96*5 + add x1, tab+96*2 + CFI_BL(Lp256_scalarmul_alt_local_p256_montjdouble) + + add x0, tab+96*6 + add x1, tab+96*5 + add x2, tab + CFI_BL(Lp256_scalarmul_alt_local_p256_montjmixadd) + + add x0, tab+96*7 + add x1, tab+96*3 + CFI_BL(Lp256_scalarmul_alt_local_p256_montjdouble) + +// Initialize the accumulator as a table entry for top 4 bits (unrecoded) + + ldr x14, [scalarb+24] + lsr x14, x14, #60 + + mov x0, xzr + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, xzr + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + mov x9, xzr + mov x10, xzr + mov x11, xzr + add x15, tab + + .set i, 1 +.rep 8 + cmp x14, #i + ldp x12, x13, [x15] + csel x0, x12, x0, eq + csel x1, x13, x1, eq + ldp x12, x13, [x15, #16] + csel x2, x12, x2, eq + csel x3, x13, x3, eq + ldp x12, x13, [x15, #32] + csel x4, x12, x4, eq + csel x5, x13, x5, eq + ldp x12, x13, [x15, #48] + csel x6, x12, x6, eq + csel x7, x13, x7, eq + ldp x12, x13, [x15, #64] + csel x8, x12, x8, eq + csel x9, x13, x9, eq + ldp x12, x13, [x15, #80] + csel x10, x12, x10, eq + csel x11, x13, x11, eq + add x15, x15, #96 + .set i, (i+1) +.endr + stp x0, x1, [acc] + stp x2, x3, [acc+16] + stp x4, x5, [acc+32] + stp x6, x7, [acc+48] + stp x8, x9, [acc+64] + stp x10, x11, [acc+80] + + mov j, #252 + +// Main loop over size-4 bitfields: double 4 times then add signed digit + +Lp256_scalarmul_alt_loop: + sub j, j, #4 + + add x0, acc + add x1, acc + CFI_BL(Lp256_scalarmul_alt_local_p256_montjdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp256_scalarmul_alt_local_p256_montjdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp256_scalarmul_alt_local_p256_montjdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp256_scalarmul_alt_local_p256_montjdouble) + + lsr x2, j, #6 + ldr x14, [sp, x2, lsl #3] // Exploits scalarb = sp exactly + lsr x14, x14, j + and x14, x14, #15 + + subs x14, x14, #8 + cset x16, lo // x16 = sign of digit (1 = negative) + cneg x14, x14, lo // x14 = absolute value of digit + +// Conditionally select the table entry tab[i-1] = i * P in constant time + + mov x0, xzr + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, xzr + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + mov x9, xzr + mov x10, xzr + mov x11, xzr + add x15, tab + .set i, 1 +.rep 8 + cmp x14, #i + ldp x12, x13, [x15] + csel x0, x12, x0, eq + csel x1, x13, x1, eq + ldp x12, x13, [x15, #16] + csel x2, x12, x2, eq + csel x3, x13, x3, eq + ldp x12, x13, [x15, #32] + csel x4, x12, x4, eq + csel x5, x13, x5, eq + ldp x12, x13, [x15, #48] + csel x6, x12, x6, eq + csel x7, x13, x7, eq + ldp x12, x13, [x15, #64] + csel x8, x12, x8, eq + csel x9, x13, x9, eq + ldp x12, x13, [x15, #80] + csel x10, x12, x10, eq + csel x11, x13, x11, eq + add x15, x15, #96 + .set i, (i+1) +.endr + +// Store it to "tabent" with the y coordinate optionally negated + + stp x0, x1, [tabent] + stp x2, x3, [tabent+16] + + mov x0, 0xffffffffffffffff + subs x0, x0, x4 + mov x1, 0x00000000ffffffff + sbcs x1, x1, x5 + mov x3, 0xffffffff00000001 + sbcs x2, xzr, x6 + sbc x3, x3, x7 + + cmp x16, xzr + csel x4, x0, x4, ne + csel x5, x1, x5, ne + csel x6, x2, x6, ne + csel x7, x3, x7, ne + + stp x4, x5, [tabent+32] + stp x6, x7, [tabent+48] + stp x8, x9, [tabent+64] + stp x10, x11, [tabent+80] + + add x0, acc + add x1, acc + add x2, tabent + CFI_BL(Lp256_scalarmul_alt_local_p256_montjadd) + + cbnz j, Lp256_scalarmul_alt_loop + +// That's the end of the main loop, and we just need to translate +// back from the Jacobian representation to affine. First of all, +// let z2 = 1/z^2 and z3 = 1/z^3, both without Montgomery form + + add x0, z2 + add x1, acc+64 + CFI_BL(Lp256_scalarmul_alt_local_montsqr_p256) + + add x0, z3 + add x2, z2 + add x1, acc+64 + CFI_BL(Lp256_scalarmul_alt_local_montmul_p256) + + add x0, z2 + add x1, z3 + CFI_BL(Lp256_scalarmul_alt_local_demont_p256) + + add x0, z3 + add x1, z2 + CFI_BL(Lp256_scalarmul_alt_local_inv_p256) + + add x0, z2 + add x2, z3 + add x1, acc+64 + CFI_BL(Lp256_scalarmul_alt_local_montmul_p256) + +// Convert back from Jacobian (X,Y,Z) |-> (X/Z^2, Y/Z^3) + + add x1, acc + add x2, z2 + mov x0, res + CFI_BL(Lp256_scalarmul_alt_local_montmul_p256) + + add x0, res, #32 + add x1, acc+32 + add x2, z3 + CFI_BL(Lp256_scalarmul_alt_local_montmul_p256) + +// Restore stack and registers and return + + CFI_INC_SP(NSPACE) + CFI_POP2(x21,x30) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_scalarmul_alt) + +// Local copies of subroutines, complete clones at the moment + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_alt_local_demont_p256) + +Lp256_scalarmul_alt_local_demont_p256: + CFI_START + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + lsl x7, x2, #32 + subs x8, x2, x7 + lsr x6, x2, #32 + sbc x2, x2, x6 + adds x3, x3, x7 + adcs x4, x4, x6 + adcs x5, x5, x8 + adc x2, x2, xzr + lsl x7, x3, #32 + subs x8, x3, x7 + lsr x6, x3, #32 + sbc x3, x3, x6 + adds x4, x4, x7 + adcs x5, x5, x6 + adcs x2, x2, x8 + adc x3, x3, xzr + lsl x7, x4, #32 + subs x8, x4, x7 + lsr x6, x4, #32 + sbc x4, x4, x6 + adds x5, x5, x7 + adcs x2, x2, x6 + adcs x3, x3, x8 + adc x4, x4, xzr + lsl x7, x5, #32 + subs x8, x5, x7 + lsr x6, x5, #32 + sbc x5, x5, x6 + adds x2, x2, x7 + adcs x3, x3, x6 + adcs x4, x4, x8 + adc x5, x5, xzr + stp x2, x3, [x0] + stp x4, x5, [x0, #16] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_alt_local_demont_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_alt_local_inv_p256) + +Lp256_scalarmul_alt_local_inv_p256: + CFI_START + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_DEC_SP(160) + mov x20, x0 + mov x10, #0xffffffffffffffff + mov x11, #0xffffffff + mov x13, #0xffffffff00000001 + stp x10, x11, [sp] + stp xzr, x13, [sp, #16] + str xzr, [sp, #32] + ldp x2, x3, [x1] + subs x10, x2, x10 + sbcs x11, x3, x11 + ldp x4, x5, [x1, #16] + sbcs x12, x4, xzr + sbcs x13, x5, x13 + csel x2, x2, x10, cc + csel x3, x3, x11, cc + csel x4, x4, x12, cc + csel x5, x5, x13, cc + stp x2, x3, [sp, #48] + stp x4, x5, [sp, #64] + str xzr, [sp, #80] + stp xzr, xzr, [sp, #96] + stp xzr, xzr, [sp, #112] + mov x10, #0x4000000000000 + stp x10, xzr, [sp, #128] + stp xzr, xzr, [sp, #144] + mov x21, #0xa + mov x22, #0x1 + b Lp256_scalarmul_alt_inv_midloop +Lp256_scalarmul_alt_inv_loop: + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + and x0, x12, x16 + and x1, x13, x17 + add x19, x0, x1 + ldr x7, [sp] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #48] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x7, [sp, #8] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #56] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + adc x6, x6, x1 + extr x4, x2, x4, #59 + str x4, [sp] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + adc x4, x4, x1 + extr x5, x3, x5, #59 + str x5, [sp, #48] + ldr x7, [sp, #16] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #64] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + adc x5, x5, x1 + extr x2, x6, x2, #59 + str x2, [sp, #8] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + adc x2, x2, x1 + extr x3, x4, x3, #59 + str x3, [sp, #56] + ldr x7, [sp, #24] + eor x1, x7, x14 + ldr x23, [sp, #32] + eor x3, x23, x14 + and x3, x3, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #72] + eor x1, x8, x15 + ldr x24, [sp, #80] + eor x0, x24, x15 + and x0, x0, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x5, x6, #59 + str x6, [sp, #16] + extr x5, x3, x5, #59 + str x5, [sp, #24] + asr x3, x3, #59 + str x3, [sp, #32] + eor x1, x7, x16 + eor x5, x23, x16 + and x5, x5, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + eor x0, x24, x17 + and x0, x0, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x4, x2, x4, #59 + str x4, [sp, #64] + extr x2, x5, x2, #59 + str x2, [sp, #72] + asr x5, x5, #59 + str x5, [sp, #80] + ldr x7, [sp, #96] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #128] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #96] + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + str x5, [sp, #128] + adc x3, x3, x1 + ldr x7, [sp, #104] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #136] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #104] + adc x6, x6, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + str x3, [sp, #136] + adc x4, x4, x1 + ldr x7, [sp, #112] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #144] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #112] + adc x5, x5, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + str x4, [sp, #144] + adc x2, x2, x1 + ldr x7, [sp, #120] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #152] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + ldp x0, x1, [sp, #96] + ldr x6, [sp, #112] + mov x14, #0xe000000000000000 + adds x0, x0, x14 + sbcs x1, x1, xzr + mov x11, #0x1fffffff + adcs x6, x6, x11 + mov x10, #0x2000000000000000 + adcs x5, x5, x10 + mov x14, #0x1fffffffe0000000 + adc x3, x3, x14 + lsl x11, x0, #32 + subs x14, x0, x11 + lsr x10, x0, #32 + sbc x0, x0, x10 + adds x1, x1, x11 + adcs x6, x6, x10 + adcs x5, x5, x14 + adcs x3, x3, x0 + mov x14, #0xffffffffffffffff + mov x11, #0xffffffff + mov x10, #0xffffffff00000001 + csel x14, x14, xzr, cs + csel x11, x11, xzr, cs + csel x10, x10, xzr, cs + subs x1, x1, x14 + sbcs x6, x6, x11 + sbcs x5, x5, xzr + sbc x3, x3, x10 + stp x1, x6, [sp, #96] + stp x5, x3, [sp, #112] + eor x1, x7, x16 + and x5, x16, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + and x0, x17, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + ldp x0, x1, [sp, #128] + ldr x3, [sp, #144] + mov x14, #0xe000000000000000 + adds x0, x0, x14 + sbcs x1, x1, xzr + mov x11, #0x1fffffff + adcs x3, x3, x11 + mov x10, #0x2000000000000000 + adcs x2, x2, x10 + mov x14, #0x1fffffffe0000000 + adc x5, x5, x14 + lsl x11, x0, #32 + subs x14, x0, x11 + lsr x10, x0, #32 + sbc x0, x0, x10 + adds x1, x1, x11 + adcs x3, x3, x10 + adcs x2, x2, x14 + adcs x5, x5, x0 + mov x14, #0xffffffffffffffff + mov x11, #0xffffffff + mov x10, #0xffffffff00000001 + csel x14, x14, xzr, cs + csel x11, x11, xzr, cs + csel x10, x10, xzr, cs + subs x1, x1, x14 + sbcs x3, x3, x11 + sbcs x2, x2, xzr + sbc x5, x5, x10 + stp x1, x3, [sp, #128] + stp x2, x5, [sp, #144] +Lp256_scalarmul_alt_inv_midloop: + mov x1, x22 + ldr x2, [sp] + ldr x3, [sp, #48] + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x8, x4, #0x100, lsl #12 + sbfx x8, x8, #21, #21 + mov x11, #0x100000 + add x11, x11, x11, lsl #21 + add x9, x4, x11 + asr x9, x9, #42 + add x10, x5, #0x100, lsl #12 + sbfx x10, x10, #21, #21 + add x11, x5, x11 + asr x11, x11, #42 + mul x6, x8, x2 + mul x7, x9, x3 + mul x2, x10, x2 + mul x3, x11, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #21, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #42 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #21, #21 + add x15, x5, x15 + asr x15, x15, #42 + mul x6, x12, x2 + mul x7, x13, x3 + mul x2, x14, x2 + mul x3, x15, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + mul x2, x12, x8 + mul x3, x12, x9 + mul x6, x14, x8 + mul x7, x14, x9 + madd x8, x13, x10, x2 + madd x9, x13, x11, x3 + madd x16, x15, x10, x6 + madd x17, x15, x11, x7 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #22, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #43 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #22, #21 + add x15, x5, x15 + asr x15, x15, #43 + mneg x2, x12, x8 + mneg x3, x12, x9 + mneg x4, x14, x8 + mneg x5, x14, x9 + msub x10, x13, x16, x2 + msub x11, x13, x17, x3 + msub x12, x15, x16, x4 + msub x13, x15, x17, x5 + mov x22, x1 + subs x21, x21, #0x1 + b.ne Lp256_scalarmul_alt_inv_loop + ldr x0, [sp] + ldr x1, [sp, #48] + mul x0, x0, x10 + madd x1, x1, x11, x0 + asr x0, x1, #63 + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + eor x14, x14, x0 + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + eor x15, x15, x0 + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + eor x16, x16, x0 + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + eor x17, x17, x0 + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + ldr x7, [sp, #96] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #128] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #96] + adc x2, x2, x1 + ldr x7, [sp, #104] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #136] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #104] + adc x6, x6, x1 + ldr x7, [sp, #112] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #144] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #112] + adc x5, x5, x1 + ldr x7, [sp, #120] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #152] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + ldp x0, x1, [sp, #96] + ldr x2, [sp, #112] + mov x14, #0xe000000000000000 + adds x0, x0, x14 + sbcs x1, x1, xzr + mov x11, #0x1fffffff + adcs x2, x2, x11 + mov x10, #0x2000000000000000 + adcs x5, x5, x10 + mov x14, #0x1fffffffe0000000 + adc x3, x3, x14 + lsl x11, x0, #32 + subs x14, x0, x11 + lsr x10, x0, #32 + sbc x0, x0, x10 + adds x1, x1, x11 + adcs x2, x2, x10 + adcs x5, x5, x14 + adcs x3, x3, x0 + mov x14, #0xffffffffffffffff + mov x11, #0xffffffff + mov x10, #0xffffffff00000001 + csel x14, x14, xzr, cs + csel x11, x11, xzr, cs + csel x10, x10, xzr, cs + subs x1, x1, x14 + sbcs x2, x2, x11 + sbcs x5, x5, xzr + sbc x3, x3, x10 + mov x10, #0xffffffffffffffff + subs x10, x1, x10 + mov x11, #0xffffffff + sbcs x11, x2, x11 + mov x13, #0xffffffff00000001 + sbcs x12, x5, xzr + sbcs x13, x3, x13 + csel x10, x1, x10, cc + csel x11, x2, x11, cc + csel x12, x5, x12, cc + csel x13, x3, x13, cc + stp x10, x11, [x20] + stp x12, x13, [x20, #16] + CFI_INC_SP(160) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_alt_local_inv_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_alt_local_montmul_p256) + +Lp256_scalarmul_alt_local_montmul_p256: + CFI_START + ldp x3, x4, [x1] + ldp x7, x8, [x2] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x2, #16] + mul x11, x3, x9 + umulh x15, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x16, x3, x10 + adcs x15, x15, x11 + adc x16, x16, xzr + ldp x5, x6, [x1, #16] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x15, x15, x11 + mul x11, x4, x10 + adcs x16, x16, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x15, x15, x11 + umulh x11, x4, x9 + adcs x16, x16, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x15, x15, x11 + mul x11, x5, x9 + adcs x16, x16, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x15, x15, x11 + umulh x11, x5, x8 + adcs x16, x16, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x15, x15, x11 + mul x11, x6, x8 + adcs x16, x16, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x15, x15, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x16, x16, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x15, x15, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x15, x15, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x15, lsl #32 + lsr x11, x15, #32 + adcs x13, x13, x11 + mul x11, x15, x10 + umulh x15, x15, x10 + adcs x14, x14, x11 + adc x15, x15, xzr + adds x12, x12, x16 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x15, x15, x5 + cset x8, cs + mov x11, #0xffffffff + adds x16, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x15, x10 + sbcs xzr, x8, xzr + csel x12, x12, x16, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x15, x15, x5, cc + stp x12, x13, [x0] + stp x14, x15, [x0, #16] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_alt_local_montmul_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_alt_local_montsqr_p256) + +Lp256_scalarmul_alt_local_montsqr_p256: + CFI_START + ldp x2, x3, [x1] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [x1, #16] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + mov x5, #0xffffffff00000001 + adds x9, x9, x8, lsl #32 + lsr x2, x8, #32 + adcs x10, x10, x2 + mul x2, x8, x5 + umulh x8, x8, x5 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x2, x9, #32 + adcs x11, x11, x2 + mul x2, x9, x5 + umulh x9, x9, x5 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x2, x10, #32 + adcs x8, x8, x2 + mul x2, x10, x5 + umulh x10, x10, x5 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x2, x11, #32 + adcs x9, x9, x2 + mul x2, x11, x5 + umulh x11, x11, x5 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + cset x2, cs + mov x3, #0xffffffff + adds x12, x8, #0x1 + sbcs x13, x9, x3 + sbcs x14, x10, xzr + sbcs x7, x11, x5 + sbcs xzr, x2, xzr + csel x8, x8, x12, cc + csel x9, x9, x13, cc + csel x10, x10, x14, cc + csel x11, x11, x7, cc + stp x8, x9, [x0] + stp x10, x11, [x0, #16] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_alt_local_montsqr_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_alt_local_tomont_p256) + +Lp256_scalarmul_alt_local_tomont_p256: + CFI_START + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + mov x1, #0xffffffffffffffff + mov x7, #0xffffffff + mov x9, #0xffffffff00000001 + subs x1, x2, x1 + sbcs x7, x3, x7 + sbcs x8, x4, xzr + sbcs x9, x5, x9 + csel x2, x2, x1, cc + csel x3, x3, x7, cc + csel x4, x4, x8, cc + csel x5, x5, x9, cc + cmp xzr, xzr + extr x9, x5, x4, #32 + adcs xzr, x4, x9 + lsr x9, x5, #32 + adcs x9, x5, x9 + csetm x6, cs + orr x9, x9, x6 + lsl x7, x9, #32 + lsr x8, x9, #32 + adds x4, x4, x7 + adc x5, x5, x8 + negs x6, x9 + sbcs x7, x7, xzr + sbc x8, x8, xzr + negs x6, x6 + sbcs x2, x2, x7 + sbcs x3, x3, x8 + sbcs x4, x4, x9 + sbcs x5, x5, x9 + adds x6, x6, x5 + mov x7, #0xffffffff + and x7, x7, x5 + adcs x2, x2, x7 + adcs x3, x3, xzr + mov x7, #0xffffffff00000001 + and x7, x7, x5 + adc x4, x4, x7 + cmp xzr, xzr + extr x9, x4, x3, #32 + adcs xzr, x3, x9 + lsr x9, x4, #32 + adcs x9, x4, x9 + csetm x5, cs + orr x9, x9, x5 + lsl x7, x9, #32 + lsr x8, x9, #32 + adds x3, x3, x7 + adc x4, x4, x8 + negs x5, x9 + sbcs x7, x7, xzr + sbc x8, x8, xzr + negs x5, x5 + sbcs x6, x6, x7 + sbcs x2, x2, x8 + sbcs x3, x3, x9 + sbcs x4, x4, x9 + adds x5, x5, x4 + mov x7, #0xffffffff + and x7, x7, x4 + adcs x6, x6, x7 + adcs x2, x2, xzr + mov x7, #0xffffffff00000001 + and x7, x7, x4 + adc x3, x3, x7 + cmp xzr, xzr + extr x9, x3, x2, #32 + adcs xzr, x2, x9 + lsr x9, x3, #32 + adcs x9, x3, x9 + csetm x4, cs + orr x9, x9, x4 + lsl x7, x9, #32 + lsr x8, x9, #32 + adds x2, x2, x7 + adc x3, x3, x8 + negs x4, x9 + sbcs x7, x7, xzr + sbc x8, x8, xzr + negs x4, x4 + sbcs x5, x5, x7 + sbcs x6, x6, x8 + sbcs x2, x2, x9 + sbcs x3, x3, x9 + adds x4, x4, x3 + mov x7, #0xffffffff + and x7, x7, x3 + adcs x5, x5, x7 + adcs x6, x6, xzr + mov x7, #0xffffffff00000001 + and x7, x7, x3 + adc x2, x2, x7 + cmp xzr, xzr + extr x9, x2, x6, #32 + adcs xzr, x6, x9 + lsr x9, x2, #32 + adcs x9, x2, x9 + csetm x3, cs + orr x9, x9, x3 + lsl x7, x9, #32 + lsr x8, x9, #32 + adds x6, x6, x7 + adc x2, x2, x8 + negs x3, x9 + sbcs x7, x7, xzr + sbc x8, x8, xzr + negs x3, x3 + sbcs x4, x4, x7 + sbcs x5, x5, x8 + sbcs x6, x6, x9 + sbcs x2, x2, x9 + adds x3, x3, x2 + mov x7, #0xffffffff + and x7, x7, x2 + adcs x4, x4, x7 + adcs x5, x5, xzr + mov x7, #0xffffffff00000001 + and x7, x7, x2 + adc x6, x6, x7 + stp x3, x4, [x0] + stp x5, x6, [x0, #16] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_alt_local_tomont_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_alt_local_p256_montjadd) + +Lp256_scalarmul_alt_local_p256_montjadd: + CFI_START + CFI_DEC_SP(224) + mov x15, x0 + mov x16, x1 + mov x17, x2 + ldp x2, x3, [x16, #64] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [x16, #80] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + adds x9, x9, x8, lsl #32 + lsr x3, x8, #32 + adcs x10, x10, x3 + mov x3, #0xffffffff00000001 + mul x2, x8, x3 + umulh x8, x8, x3 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x3, x9, #32 + adcs x11, x11, x3 + mov x3, #0xffffffff00000001 + mul x2, x9, x3 + umulh x9, x9, x3 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x3, x10, #32 + adcs x8, x8, x3 + mov x3, #0xffffffff00000001 + mul x2, x10, x3 + umulh x10, x10, x3 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x3, x11, #32 + adcs x9, x9, x3 + mov x3, #0xffffffff00000001 + mul x2, x11, x3 + umulh x11, x11, x3 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + mov x2, #0xffffffffffffffff + csel x2, xzr, x2, cc + mov x3, #0xffffffff + csel x3, xzr, x3, cc + mov x5, #0xffffffff00000001 + csel x5, xzr, x5, cc + subs x8, x8, x2 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbc x11, x11, x5 + stp x8, x9, [sp] + stp x10, x11, [sp, #16] + ldp x2, x3, [x17, #64] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [x17, #80] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + adds x9, x9, x8, lsl #32 + lsr x3, x8, #32 + adcs x10, x10, x3 + mov x3, #0xffffffff00000001 + mul x2, x8, x3 + umulh x8, x8, x3 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x3, x9, #32 + adcs x11, x11, x3 + mov x3, #0xffffffff00000001 + mul x2, x9, x3 + umulh x9, x9, x3 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x3, x10, #32 + adcs x8, x8, x3 + mov x3, #0xffffffff00000001 + mul x2, x10, x3 + umulh x10, x10, x3 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x3, x11, #32 + adcs x9, x9, x3 + mov x3, #0xffffffff00000001 + mul x2, x11, x3 + umulh x11, x11, x3 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + mov x2, #0xffffffffffffffff + csel x2, xzr, x2, cc + mov x3, #0xffffffff + csel x3, xzr, x3, cc + mov x5, #0xffffffff00000001 + csel x5, xzr, x5, cc + subs x8, x8, x2 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbc x11, x11, x5 + stp x8, x9, [sp, #160] + stp x10, x11, [sp, #176] + ldp x3, x4, [x17, #64] + ldp x7, x8, [x16, #32] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x16, #48] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [x17, #80] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #192] + stp x14, x0, [sp, #208] + ldp x3, x4, [x16, #64] + ldp x7, x8, [x17, #32] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x17, #48] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [x16, #80] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #32] + stp x14, x0, [sp, #48] + ldp x3, x4, [sp] + ldp x7, x8, [x17] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x17, #16] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #16] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #64] + stp x14, x0, [sp, #80] + ldp x3, x4, [sp, #160] + ldp x7, x8, [x16] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x16, #16] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #176] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #128] + stp x14, x0, [sp, #144] + ldp x3, x4, [sp] + ldp x7, x8, [sp, #32] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [sp, #48] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #16] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #32] + stp x14, x0, [sp, #48] + ldp x3, x4, [sp, #160] + ldp x7, x8, [sp, #192] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [sp, #208] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #176] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #192] + stp x14, x0, [sp, #208] + ldp x5, x6, [sp, #64] + ldp x4, x3, [sp, #128] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [sp, #144] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #160] + stp x7, x8, [sp, #176] + ldp x5, x6, [sp, #32] + ldp x4, x3, [sp, #192] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #48] + ldp x4, x3, [sp, #208] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #32] + stp x7, x8, [sp, #48] + ldp x2, x3, [sp, #160] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [sp, #176] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + adds x9, x9, x8, lsl #32 + lsr x3, x8, #32 + adcs x10, x10, x3 + mov x3, #0xffffffff00000001 + mul x2, x8, x3 + umulh x8, x8, x3 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x3, x9, #32 + adcs x11, x11, x3 + mov x3, #0xffffffff00000001 + mul x2, x9, x3 + umulh x9, x9, x3 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x3, x10, #32 + adcs x8, x8, x3 + mov x3, #0xffffffff00000001 + mul x2, x10, x3 + umulh x10, x10, x3 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x3, x11, #32 + adcs x9, x9, x3 + mov x3, #0xffffffff00000001 + mul x2, x11, x3 + umulh x11, x11, x3 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + mov x2, #0xffffffffffffffff + csel x2, xzr, x2, cc + mov x3, #0xffffffff + csel x3, xzr, x3, cc + mov x5, #0xffffffff00000001 + csel x5, xzr, x5, cc + subs x8, x8, x2 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbc x11, x11, x5 + stp x8, x9, [sp, #96] + stp x10, x11, [sp, #112] + ldp x2, x3, [sp, #32] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [sp, #48] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + adds x9, x9, x8, lsl #32 + lsr x3, x8, #32 + adcs x10, x10, x3 + mov x3, #0xffffffff00000001 + mul x2, x8, x3 + umulh x8, x8, x3 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x3, x9, #32 + adcs x11, x11, x3 + mov x3, #0xffffffff00000001 + mul x2, x9, x3 + umulh x9, x9, x3 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x3, x10, #32 + adcs x8, x8, x3 + mov x3, #0xffffffff00000001 + mul x2, x10, x3 + umulh x10, x10, x3 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x3, x11, #32 + adcs x9, x9, x3 + mov x3, #0xffffffff00000001 + mul x2, x11, x3 + umulh x11, x11, x3 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + cset x2, cs + mov x3, #0xffffffff + mov x5, #0xffffffff00000001 + adds x12, x8, #0x1 + sbcs x13, x9, x3 + sbcs x14, x10, xzr + sbcs x7, x11, x5 + sbcs xzr, x2, xzr + csel x8, x8, x12, cc + csel x9, x9, x13, cc + csel x10, x10, x14, cc + csel x11, x11, x7, cc + stp x8, x9, [sp] + stp x10, x11, [sp, #16] + ldp x3, x4, [sp, #96] + ldp x7, x8, [sp, #128] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [sp, #144] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #112] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #128] + stp x14, x0, [sp, #144] + ldp x3, x4, [sp, #96] + ldp x7, x8, [sp, #64] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [sp, #80] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #112] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #64] + stp x14, x0, [sp, #80] + ldp x5, x6, [sp] + ldp x4, x3, [sp, #128] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #16] + ldp x4, x3, [sp, #144] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp] + stp x7, x8, [sp, #16] + ldp x5, x6, [sp, #64] + ldp x4, x3, [sp, #128] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [sp, #144] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #96] + stp x7, x8, [sp, #112] + ldp x3, x4, [sp, #160] + ldp x7, x8, [x16, #64] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x16, #80] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #176] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #160] + stp x14, x0, [sp, #176] + ldp x5, x6, [sp] + ldp x4, x3, [sp, #64] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #16] + ldp x4, x3, [sp, #80] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp] + stp x7, x8, [sp, #16] + ldp x5, x6, [sp, #128] + ldp x4, x3, [sp] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #144] + ldp x4, x3, [sp, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #128] + stp x7, x8, [sp, #144] + ldp x3, x4, [sp, #96] + ldp x7, x8, [sp, #192] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [sp, #208] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #112] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #96] + stp x14, x0, [sp, #112] + ldp x3, x4, [sp, #160] + ldp x7, x8, [x17, #64] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x17, #80] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #176] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #160] + stp x14, x0, [sp, #176] + ldp x3, x4, [sp, #32] + ldp x7, x8, [sp, #128] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [sp, #144] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #48] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #128] + stp x14, x0, [sp, #144] + ldp x5, x6, [sp, #128] + ldp x4, x3, [sp, #96] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #144] + ldp x4, x3, [sp, #112] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #128] + stp x7, x8, [sp, #144] + ldp x0, x1, [x16, #64] + ldp x2, x3, [x16, #80] + orr x12, x0, x1 + orr x13, x2, x3 + orr x12, x12, x13 + cmp x12, xzr + cset x12, ne + ldp x4, x5, [x17, #64] + ldp x6, x7, [x17, #80] + orr x13, x4, x5 + orr x14, x6, x7 + orr x13, x13, x14 + cmp x13, xzr + cset x13, ne + cmp x13, x12 + ldp x8, x9, [sp, #160] + csel x8, x0, x8, cc + csel x9, x1, x9, cc + csel x8, x4, x8, hi + csel x9, x5, x9, hi + ldp x10, x11, [sp, #176] + csel x10, x2, x10, cc + csel x11, x3, x11, cc + csel x10, x6, x10, hi + csel x11, x7, x11, hi + ldp x12, x13, [x16] + ldp x0, x1, [sp] + csel x0, x12, x0, cc + csel x1, x13, x1, cc + ldp x12, x13, [x17] + csel x0, x12, x0, hi + csel x1, x13, x1, hi + ldp x12, x13, [x16, #16] + ldp x2, x3, [sp, #16] + csel x2, x12, x2, cc + csel x3, x13, x3, cc + ldp x12, x13, [x17, #16] + csel x2, x12, x2, hi + csel x3, x13, x3, hi + ldp x12, x13, [x16, #32] + ldp x4, x5, [sp, #128] + csel x4, x12, x4, cc + csel x5, x13, x5, cc + ldp x12, x13, [x17, #32] + csel x4, x12, x4, hi + csel x5, x13, x5, hi + ldp x12, x13, [x16, #48] + ldp x6, x7, [sp, #144] + csel x6, x12, x6, cc + csel x7, x13, x7, cc + ldp x12, x13, [x17, #48] + csel x6, x12, x6, hi + csel x7, x13, x7, hi + stp x0, x1, [x15] + stp x2, x3, [x15, #16] + stp x4, x5, [x15, #32] + stp x6, x7, [x15, #48] + stp x8, x9, [x15, #64] + stp x10, x11, [x15, #80] + CFI_INC_SP(224) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_alt_local_p256_montjadd) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_alt_local_p256_montjdouble) + +Lp256_scalarmul_alt_local_p256_montjdouble: + CFI_START + CFI_DEC_SP(192) + mov x15, x0 + mov x16, x1 + ldp x2, x3, [x16, #64] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [x16, #80] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + mov x5, #0xffffffff00000001 + adds x9, x9, x8, lsl #32 + lsr x3, x8, #32 + adcs x10, x10, x3 + mul x2, x8, x5 + umulh x8, x8, x5 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x3, x9, #32 + adcs x11, x11, x3 + mul x2, x9, x5 + umulh x9, x9, x5 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x3, x10, #32 + adcs x8, x8, x3 + mul x2, x10, x5 + umulh x10, x10, x5 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x3, x11, #32 + adcs x9, x9, x3 + mul x2, x11, x5 + umulh x11, x11, x5 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + cset x2, cs + mov x3, #0xffffffff + adds x12, x8, #0x1 + sbcs x13, x9, x3 + sbcs x14, x10, xzr + sbcs x7, x11, x5 + sbcs xzr, x2, xzr + csel x8, x8, x12, cc + csel x9, x9, x13, cc + csel x10, x10, x14, cc + csel x11, x11, x7, cc + stp x8, x9, [sp] + stp x10, x11, [sp, #16] + ldp x2, x3, [x16, #32] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [x16, #48] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + mov x5, #0xffffffff00000001 + adds x9, x9, x8, lsl #32 + lsr x3, x8, #32 + adcs x10, x10, x3 + mul x2, x8, x5 + umulh x8, x8, x5 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x3, x9, #32 + adcs x11, x11, x3 + mul x2, x9, x5 + umulh x9, x9, x5 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x3, x10, #32 + adcs x8, x8, x3 + mul x2, x10, x5 + umulh x10, x10, x5 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x3, x11, #32 + adcs x9, x9, x3 + mul x2, x11, x5 + umulh x11, x11, x5 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + cset x2, cs + mov x3, #0xffffffff + adds x12, x8, #0x1 + sbcs x13, x9, x3 + sbcs x14, x10, xzr + sbcs x7, x11, x5 + sbcs xzr, x2, xzr + csel x8, x8, x12, cc + csel x9, x9, x13, cc + csel x10, x10, x14, cc + csel x11, x11, x7, cc + stp x8, x9, [sp, #32] + stp x10, x11, [sp, #48] + ldp x5, x6, [x16] + ldp x4, x3, [sp] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [x16, #16] + ldp x4, x3, [sp, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + and x4, x3, #0xffffffff + adcs x6, x6, x4 + adcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x8, x8, x4 + stp x5, x6, [sp, #96] + stp x7, x8, [sp, #112] + ldp x5, x6, [x16] + ldp x4, x3, [sp] + adds x5, x5, x4 + adcs x6, x6, x3 + ldp x7, x8, [x16, #16] + ldp x4, x3, [sp, #16] + adcs x7, x7, x4 + adcs x8, x8, x3 + csetm x3, cs + subs x5, x5, x3 + and x1, x3, #0xffffffff + sbcs x6, x6, x1 + sbcs x7, x7, xzr + and x2, x3, #0xffffffff00000001 + sbc x8, x8, x2 + stp x5, x6, [sp, #64] + stp x7, x8, [sp, #80] + ldp x3, x4, [sp, #64] + ldp x7, x8, [sp, #96] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [sp, #112] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #80] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #96] + stp x14, x0, [sp, #112] + ldp x5, x6, [x16, #32] + ldp x4, x3, [x16, #64] + adds x5, x5, x4 + adcs x6, x6, x3 + ldp x7, x8, [x16, #48] + ldp x4, x3, [x16, #80] + adcs x7, x7, x4 + adcs x8, x8, x3 + adc x3, xzr, xzr + cmn x5, #0x1 + mov x4, #0xffffffff + sbcs xzr, x6, x4 + sbcs xzr, x7, xzr + mov x4, #0xffffffff00000001 + sbcs xzr, x8, x4 + adcs x3, x3, xzr + csetm x3, ne + subs x5, x5, x3 + and x4, x3, #0xffffffff + sbcs x6, x6, x4 + sbcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + sbc x8, x8, x4 + stp x5, x6, [sp, #64] + stp x7, x8, [sp, #80] + ldp x3, x4, [x16] + ldp x7, x8, [sp, #32] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [sp, #48] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [x16, #16] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #128] + stp x14, x0, [sp, #144] + ldp x2, x3, [sp, #96] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [sp, #112] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + mov x5, #0xffffffff00000001 + adds x9, x9, x8, lsl #32 + lsr x3, x8, #32 + adcs x10, x10, x3 + mul x2, x8, x5 + umulh x8, x8, x5 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x3, x9, #32 + adcs x11, x11, x3 + mul x2, x9, x5 + umulh x9, x9, x5 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x3, x10, #32 + adcs x8, x8, x3 + mul x2, x10, x5 + umulh x10, x10, x5 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x3, x11, #32 + adcs x9, x9, x3 + mul x2, x11, x5 + umulh x11, x11, x5 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + cset x2, cs + mov x3, #0xffffffff + adds x12, x8, #0x1 + sbcs x13, x9, x3 + sbcs x14, x10, xzr + sbcs x7, x11, x5 + sbcs xzr, x2, xzr + csel x8, x8, x12, cc + csel x9, x9, x13, cc + csel x10, x10, x14, cc + csel x11, x11, x7, cc + stp x8, x9, [sp, #160] + stp x10, x11, [sp, #176] + ldp x2, x3, [sp, #64] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [sp, #80] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + mov x5, #0xffffffff00000001 + adds x9, x9, x8, lsl #32 + lsr x3, x8, #32 + adcs x10, x10, x3 + mul x2, x8, x5 + umulh x8, x8, x5 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x3, x9, #32 + adcs x11, x11, x3 + mul x2, x9, x5 + umulh x9, x9, x5 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x3, x10, #32 + adcs x8, x8, x3 + mul x2, x10, x5 + umulh x10, x10, x5 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x3, x11, #32 + adcs x9, x9, x3 + mul x2, x11, x5 + umulh x11, x11, x5 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + cset x2, cs + mov x3, #0xffffffff + adds x12, x8, #0x1 + sbcs x13, x9, x3 + sbcs x14, x10, xzr + sbcs x7, x11, x5 + sbcs xzr, x2, xzr + csel x8, x8, x12, cc + csel x9, x9, x13, cc + csel x10, x10, x14, cc + csel x11, x11, x7, cc + stp x8, x9, [sp, #64] + stp x10, x11, [sp, #80] + mov x1, #0x9 + mov x2, #0xffffffffffffffff + ldp x9, x10, [sp, #160] + subs x9, x2, x9 + mov x2, #0xffffffff + sbcs x10, x2, x10 + ldp x11, x12, [sp, #176] + ngcs x11, x11 + mov x2, #0xffffffff00000001 + sbc x12, x2, x12 + mul x3, x1, x9 + mul x4, x1, x10 + mul x5, x1, x11 + mul x6, x1, x12 + umulh x9, x1, x9 + umulh x10, x1, x10 + umulh x11, x1, x11 + umulh x7, x1, x12 + adds x4, x4, x9 + adcs x5, x5, x10 + adcs x6, x6, x11 + adc x7, x7, xzr + mov x1, #0xc + ldp x9, x10, [sp, #128] + mul x8, x9, x1 + umulh x9, x9, x1 + adds x3, x3, x8 + mul x8, x10, x1 + umulh x10, x10, x1 + adcs x4, x4, x8 + ldp x11, x12, [sp, #144] + mul x8, x11, x1 + umulh x11, x11, x1 + adcs x5, x5, x8 + mul x8, x12, x1 + umulh x12, x12, x1 + adcs x6, x6, x8 + adc x7, x7, xzr + adds x4, x4, x9 + adcs x5, x5, x10 + adcs x6, x6, x11 + adc x7, x7, x12 + add x8, x7, #0x1 + lsl x10, x8, #32 + adds x6, x6, x10 + adc x7, x7, xzr + neg x9, x8 + sub x10, x10, #0x1 + subs x3, x3, x9 + sbcs x4, x4, x10 + sbcs x5, x5, xzr + sbcs x6, x6, x8 + sbc x8, x7, x8 + adds x3, x3, x8 + and x9, x8, #0xffffffff + adcs x4, x4, x9 + adcs x5, x5, xzr + neg x10, x9 + adc x6, x6, x10 + stp x3, x4, [sp, #160] + stp x5, x6, [sp, #176] + ldp x5, x6, [sp, #64] + ldp x4, x3, [sp] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [sp, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + and x4, x3, #0xffffffff + adcs x6, x6, x4 + adcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x8, x8, x4 + stp x5, x6, [sp, #64] + stp x7, x8, [sp, #80] + ldp x2, x3, [sp, #32] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [sp, #48] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + mov x5, #0xffffffff00000001 + adds x9, x9, x8, lsl #32 + lsr x3, x8, #32 + adcs x10, x10, x3 + mul x2, x8, x5 + umulh x8, x8, x5 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x3, x9, #32 + adcs x11, x11, x3 + mul x2, x9, x5 + umulh x9, x9, x5 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x3, x10, #32 + adcs x8, x8, x3 + mul x2, x10, x5 + umulh x10, x10, x5 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x3, x11, #32 + adcs x9, x9, x3 + mul x2, x11, x5 + umulh x11, x11, x5 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + cset x2, cs + mov x3, #0xffffffff + adds x12, x8, #0x1 + sbcs x13, x9, x3 + sbcs x14, x10, xzr + sbcs x7, x11, x5 + sbcs xzr, x2, xzr + csel x8, x8, x12, cc + csel x9, x9, x13, cc + csel x10, x10, x14, cc + csel x11, x11, x7, cc + stp x8, x9, [sp] + stp x10, x11, [sp, #16] + ldp x3, x4, [sp, #160] + ldp x7, x8, [sp, #96] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [sp, #112] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #176] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #96] + stp x14, x0, [sp, #112] + ldp x5, x6, [sp, #64] + ldp x4, x3, [sp, #32] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [sp, #48] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + and x4, x3, #0xffffffff + adcs x6, x6, x4 + adcs x7, x7, xzr + and x4, x3, #0xffffffff00000001 + adc x8, x8, x4 + stp x5, x6, [x15, #64] + stp x7, x8, [x15, #80] + ldp x1, x2, [sp, #128] + lsl x0, x1, #2 + ldp x6, x7, [sp, #160] + subs x0, x0, x6 + extr x1, x2, x1, #62 + sbcs x1, x1, x7 + ldp x3, x4, [sp, #144] + extr x2, x3, x2, #62 + ldp x6, x7, [sp, #176] + sbcs x2, x2, x6 + extr x3, x4, x3, #62 + sbcs x3, x3, x7 + lsr x4, x4, #62 + sbc x4, x4, xzr + add x5, x4, #0x1 + lsl x8, x5, #32 + negs x6, x8 + ngcs x7, xzr + sbc x8, x8, x5 + adds x0, x0, x5 + adcs x1, x1, x6 + adcs x2, x2, x7 + adcs x3, x3, x8 + csetm x5, cc + adds x0, x0, x5 + and x6, x5, #0xffffffff + adcs x1, x1, x6 + adcs x2, x2, xzr + neg x7, x6 + adc x3, x3, x7 + stp x0, x1, [x15] + stp x2, x3, [x15, #16] + mov x1, #0x8 + mov x2, #0xffffffffffffffff + ldp x9, x10, [sp] + subs x9, x2, x9 + mov x2, #0xffffffff + sbcs x10, x2, x10 + ldp x11, x12, [sp, #16] + ngcs x11, x11 + mov x2, #0xffffffff00000001 + sbc x12, x2, x12 + lsl x3, x9, #3 + extr x4, x10, x9, #61 + extr x5, x11, x10, #61 + extr x6, x12, x11, #61 + lsr x7, x12, #61 + mov x1, #0x3 + ldp x9, x10, [sp, #96] + mul x8, x9, x1 + umulh x9, x9, x1 + adds x3, x3, x8 + mul x8, x10, x1 + umulh x10, x10, x1 + adcs x4, x4, x8 + ldp x11, x12, [sp, #112] + mul x8, x11, x1 + umulh x11, x11, x1 + adcs x5, x5, x8 + mul x8, x12, x1 + umulh x12, x12, x1 + adcs x6, x6, x8 + adc x7, x7, xzr + adds x4, x4, x9 + adcs x5, x5, x10 + adcs x6, x6, x11 + adc x7, x7, x12 + add x8, x7, #0x1 + lsl x10, x8, #32 + adds x6, x6, x10 + adc x7, x7, xzr + neg x9, x8 + sub x10, x10, #0x1 + subs x3, x3, x9 + sbcs x4, x4, x10 + sbcs x5, x5, xzr + sbcs x6, x6, x8 + sbc x8, x7, x8 + adds x3, x3, x8 + and x9, x8, #0xffffffff + adcs x4, x4, x9 + adcs x5, x5, xzr + neg x10, x9 + adc x6, x6, x10 + stp x3, x4, [x15, #32] + stp x5, x6, [x15, #48] + CFI_INC_SP(192) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_alt_local_p256_montjdouble) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_alt_local_p256_montjmixadd) + +Lp256_scalarmul_alt_local_p256_montjmixadd: + CFI_START + CFI_DEC_SP(192) + mov x15, x0 + mov x16, x1 + mov x17, x2 + ldp x2, x3, [x16, #64] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [x16, #80] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + adds x9, x9, x8, lsl #32 + lsr x3, x8, #32 + adcs x10, x10, x3 + mov x3, #0xffffffff00000001 + mul x2, x8, x3 + umulh x8, x8, x3 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x3, x9, #32 + adcs x11, x11, x3 + mov x3, #0xffffffff00000001 + mul x2, x9, x3 + umulh x9, x9, x3 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x3, x10, #32 + adcs x8, x8, x3 + mov x3, #0xffffffff00000001 + mul x2, x10, x3 + umulh x10, x10, x3 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x3, x11, #32 + adcs x9, x9, x3 + mov x3, #0xffffffff00000001 + mul x2, x11, x3 + umulh x11, x11, x3 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + mov x2, #0xffffffffffffffff + csel x2, xzr, x2, cc + mov x3, #0xffffffff + csel x3, xzr, x3, cc + mov x5, #0xffffffff00000001 + csel x5, xzr, x5, cc + subs x8, x8, x2 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbc x11, x11, x5 + stp x8, x9, [sp] + stp x10, x11, [sp, #16] + ldp x3, x4, [x16, #64] + ldp x7, x8, [x17, #32] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x17, #48] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [x16, #80] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #32] + stp x14, x0, [sp, #48] + ldp x3, x4, [sp] + ldp x7, x8, [x17] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x17, #16] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #16] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #64] + stp x14, x0, [sp, #80] + ldp x3, x4, [sp] + ldp x7, x8, [sp, #32] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [sp, #48] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #16] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #32] + stp x14, x0, [sp, #48] + ldp x5, x6, [sp, #64] + ldp x4, x3, [x16] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [x16, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #160] + stp x7, x8, [sp, #176] + ldp x5, x6, [sp, #32] + ldp x4, x3, [x16, #32] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #48] + ldp x4, x3, [x16, #48] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #32] + stp x7, x8, [sp, #48] + ldp x2, x3, [sp, #160] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [sp, #176] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + adds x9, x9, x8, lsl #32 + lsr x3, x8, #32 + adcs x10, x10, x3 + mov x3, #0xffffffff00000001 + mul x2, x8, x3 + umulh x8, x8, x3 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x3, x9, #32 + adcs x11, x11, x3 + mov x3, #0xffffffff00000001 + mul x2, x9, x3 + umulh x9, x9, x3 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x3, x10, #32 + adcs x8, x8, x3 + mov x3, #0xffffffff00000001 + mul x2, x10, x3 + umulh x10, x10, x3 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x3, x11, #32 + adcs x9, x9, x3 + mov x3, #0xffffffff00000001 + mul x2, x11, x3 + umulh x11, x11, x3 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + mov x2, #0xffffffffffffffff + csel x2, xzr, x2, cc + mov x3, #0xffffffff + csel x3, xzr, x3, cc + mov x5, #0xffffffff00000001 + csel x5, xzr, x5, cc + subs x8, x8, x2 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbc x11, x11, x5 + stp x8, x9, [sp, #96] + stp x10, x11, [sp, #112] + ldp x2, x3, [sp, #32] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [sp, #48] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + adds x9, x9, x8, lsl #32 + lsr x3, x8, #32 + adcs x10, x10, x3 + mov x3, #0xffffffff00000001 + mul x2, x8, x3 + umulh x8, x8, x3 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x3, x9, #32 + adcs x11, x11, x3 + mov x3, #0xffffffff00000001 + mul x2, x9, x3 + umulh x9, x9, x3 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x3, x10, #32 + adcs x8, x8, x3 + mov x3, #0xffffffff00000001 + mul x2, x10, x3 + umulh x10, x10, x3 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x3, x11, #32 + adcs x9, x9, x3 + mov x3, #0xffffffff00000001 + mul x2, x11, x3 + umulh x11, x11, x3 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + cset x2, cs + mov x3, #0xffffffff + mov x5, #0xffffffff00000001 + adds x12, x8, #0x1 + sbcs x13, x9, x3 + sbcs x14, x10, xzr + sbcs x7, x11, x5 + sbcs xzr, x2, xzr + csel x8, x8, x12, cc + csel x9, x9, x13, cc + csel x10, x10, x14, cc + csel x11, x11, x7, cc + stp x8, x9, [sp] + stp x10, x11, [sp, #16] + ldp x3, x4, [sp, #96] + ldp x7, x8, [x16] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x16, #16] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #112] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #128] + stp x14, x0, [sp, #144] + ldp x3, x4, [sp, #96] + ldp x7, x8, [sp, #64] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [sp, #80] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #112] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #64] + stp x14, x0, [sp, #80] + ldp x5, x6, [sp] + ldp x4, x3, [sp, #128] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #16] + ldp x4, x3, [sp, #144] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp] + stp x7, x8, [sp, #16] + ldp x5, x6, [sp, #64] + ldp x4, x3, [sp, #128] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [sp, #144] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #96] + stp x7, x8, [sp, #112] + ldp x3, x4, [sp, #160] + ldp x7, x8, [x16, #64] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x16, #80] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #176] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #160] + stp x14, x0, [sp, #176] + ldp x5, x6, [sp] + ldp x4, x3, [sp, #64] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #16] + ldp x4, x3, [sp, #80] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp] + stp x7, x8, [sp, #16] + ldp x5, x6, [sp, #128] + ldp x4, x3, [sp] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #144] + ldp x4, x3, [sp, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #128] + stp x7, x8, [sp, #144] + ldp x3, x4, [sp, #96] + ldp x7, x8, [x16, #32] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x16, #48] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #112] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #96] + stp x14, x0, [sp, #112] + ldp x3, x4, [sp, #32] + ldp x7, x8, [sp, #128] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [sp, #144] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #48] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #128] + stp x14, x0, [sp, #144] + ldp x5, x6, [sp, #128] + ldp x4, x3, [sp, #96] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #144] + ldp x4, x3, [sp, #112] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #128] + stp x7, x8, [sp, #144] + ldp x0, x1, [x16, #64] + ldp x2, x3, [x16, #80] + orr x4, x0, x1 + orr x5, x2, x3 + orr x4, x4, x5 + cmp x4, xzr + ldp x0, x1, [sp] + ldp x12, x13, [x17] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x2, x3, [sp, #16] + ldp x12, x13, [x17, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x4, x5, [sp, #128] + ldp x12, x13, [x17, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x6, x7, [sp, #144] + ldp x12, x13, [x17, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x8, x9, [sp, #160] + mov x12, #0x1 + mov x13, #0xffffffff00000000 + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x10, x11, [sp, #176] + mov x12, #0xffffffffffffffff + mov x13, #0xfffffffe + csel x10, x10, x12, ne + csel x11, x11, x13, ne + stp x0, x1, [x15] + stp x2, x3, [x15, #16] + stp x4, x5, [x15, #32] + stp x6, x7, [x15, #48] + stp x8, x9, [x15, #64] + stp x10, x11, [x15, #80] + CFI_INC_SP(192) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_alt_local_p256_montjmixadd) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/p256_scalarmulbase.S b/cbits/s2n/arm/p256_scalarmulbase.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/p256_scalarmulbase.S @@ -0,0 +1,3782 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Scalar multiplication for precomputed point on NIST curve P-256 +// Input scalar[4], blocksize, table[]; output res[8] +// +// extern void p256_scalarmulbase +// (uint64_t res[static 8], +// const uint64_t scalar[static 4], +// uint64_t blocksize, +// const uint64_t *table); +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-256, the input argument "table" is expected to be a table of +// multiples of the point P in Montgomery-affine form, with each block +// corresponding to "blocksize" bits of the scalar as follows, where +// B = 2^{blocksize-1} (e.g. B = 8 for blocksize = 4): +// +// For each i,j with blocksize * i <= 256 and 1 <= j <= B +// the multiple 2^{blocksize * i} * j * P is stored at +// tab[8 * (B * i + (j - 1))], considered as uint64_t pointers +// or tab + 64 * (B * i + (j - 1)) as byte pointers. +// +// Standard ARM ABI: X0 = res, X1 = scalar, X2 = blocksize, X3 = table +// ---------------------------------------------------------------------------- + + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_scalarmulbase) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_scalarmulbase) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_scalarmulbase) + + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Safe copies of inputs and additional variables, with some aliasing + +#define res x19 +#define blocksize x20 +#define table x21 +#define i x22 +#define bf x23 +#define cf x24 +#define j x25 + +// Intermediate variables on the stack. The last z2, z3 values can +// safely be overlaid on "nacc", which is no longer needed at the end. +// Uppercase syntactic variants make x86_att version simpler to generate + +#define rscalar sp, #(0*NUMSIZE) +#define acc sp, #(1*NUMSIZE) +#define nacc sp, #(4*NUMSIZE) +#define tabent sp, #(7*NUMSIZE) + +#define z2 sp, #(4*NUMSIZE) +#define z3 sp, #(5*NUMSIZE) + +#define NSPACE 9*NUMSIZE + +// Loading large constants + +#define movbig(nn,n3,n2,n1,n0) \ + movz nn, n0 __LF \ + movk nn, n1, lsl #16 __LF \ + movk nn, n2, lsl #32 __LF \ + movk nn, n3, lsl #48 + +S2N_BN_SYMBOL(p256_scalarmulbase): + CFI_START + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x30) + CFI_DEC_SP(NSPACE) + +// Preserve the input arguments except the scalar, since that gets absorbed +// immediately. The "table" value subsequently gets shifted up each iteration +// of the loop, while "res" and "blocksize" are static throughout. + + mov res, x0 + mov blocksize, x2 + mov table, x3 + +// Load the digits of group order n_256 = [x15;x14;x13;x12] + + movbig(x12, #0xf3b9, #0xcac2, #0xfc63, #0x2551) + movbig(x13, #0xbce6, #0xfaad, #0xa717, #0x9e84) + mov x14, #0xffffffffffffffff + mov x15, #0xffffffff00000000 + +// First, reduce the input scalar mod n_256, i.e. conditionally subtract n_256 +// Store it to "rscalar" (reduced scalar) + + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + + subs x6, x2, x12 + sbcs x7, x3, x13 + sbcs x8, x4, x14 + sbcs x9, x5, x15 + + csel x2, x2, x6, cc + csel x3, x3, x7, cc + csel x4, x4, x8, cc + csel x5, x5, x9, cc + + stp x2, x3, [rscalar] + stp x4, x5, [rscalar+16] + +// Initialize the accumulator to all zeros and the "carry flag" cf to 0 + + stp xzr, xzr, [acc] + stp xzr, xzr, [acc+16] + stp xzr, xzr, [acc+32] + stp xzr, xzr, [acc+48] + stp xzr, xzr, [acc+64] + stp xzr, xzr, [acc+80] + mov cf, xzr + +// Main loop over {i >= 0 | blocksize * i <= 256}. Note the non-strict +// inequality, to allow top carry for any choices of blocksize. + + mov i, xzr + +Lp256_scalarmulbase_loop: + +// The next raw bitfield is bf = bitfield(blocksize * i,blocksize) + cf, +// adding in the deferred carry cf. We then shift the whole scalar right +// by blocksize so we can keep picking bitfield(0,blocksize). + + ldp x0, x1, [rscalar] + ldp x2, x3, [rscalar+16] + + mov x4, #1 + lsl x4, x4, blocksize + sub x4, x4, #1 + and x4, x4, x0 + add bf, x4, cf + + neg x8, blocksize + + lsl x5, x1, x8 + + lsr x0, x0, blocksize + orr x0, x0, x5 + + lsl x6, x2, x8 + lsr x1, x1, blocksize + orr x1, x1, x6 + + lsl x7, x3, x8 + lsr x2, x2, blocksize + orr x2, x2, x7 + + lsr x3, x3, blocksize + + stp x0, x1, [rscalar] + stp x2, x3, [rscalar+16] + +// Now if bf <= B we just select entry j, unnegated and set cf = 0. +// If bf > B we set j = 2 * B - bf and negate the j'th entry, setting cf = 1. +// In either case we ultimately add bf, in the latter case with deferred +// carry as 2 * B - (2 * B - bf) = bf. + + mov x0, #1 + lsl x1, x0, blocksize + lsr x0, x1, #1 + + sub x2, x1, bf + + cmp x0, bf + cset cf, cc + csel j, x2, bf, cc + +// Load table entry j - 1 for nonzero j in constant-time style. + + mov x16, #1 + lsl x16, x16, blocksize + lsr x16, x16, #1 + mov x17, j + +Lp256_scalarmulbase_tabloop: + ldp x8, x9, [table] + ldp x10, x11, [table, #16] + ldp x12, x13, [table, #32] + ldp x14, x15, [table, #48] + + subs x17, x17, #1 + csel x0, x8, x0, eq + csel x1, x9, x1, eq + csel x2, x10, x2, eq + csel x3, x11, x3, eq + csel x4, x12, x4, eq + csel x5, x13, x5, eq + csel x6, x14, x6, eq + csel x7, x15, x7, eq + + add table, table, #64 + + sub x16, x16, #1 + cbnz x16, Lp256_scalarmulbase_tabloop + +// Before storing back, optionally negate the y coordinate of the table entry + + stp x0, x1, [tabent] + stp x2, x3, [tabent+16] + + mov x0, 0xffffffffffffffff + subs x0, x0, x4 + mov x1, 0x00000000ffffffff + sbcs x1, x1, x5 + mov x3, 0xffffffff00000001 + sbcs x2, xzr, x6 + sbc x3, x3, x7 + + cmp cf, xzr + csel x4, x0, x4, ne + csel x5, x1, x5, ne + csel x6, x2, x6, ne + csel x7, x3, x7, ne + + stp x4, x5, [tabent+32] + stp x6, x7, [tabent+48] + +// Add the adjusted table point to the accumulator + + add x0, nacc + add x1, acc + add x2, tabent + CFI_BL(Lp256_scalarmulbase_local_p256_montjmixadd) + +// However, only commit that update to the accumulator if j is nonzero, +// because the mixed addition function does not handle this case directly, +// and in any case we didn't choose the table entry appropriately. + + cmp j, xzr + ldp x0, x1, [acc] + ldp x12, x13, [nacc] + csel x0, x12, x0, ne + csel x1, x13, x1, ne + + ldp x2, x3, [acc+16] + ldp x12, x13, [nacc+16] + csel x2, x12, x2, ne + csel x3, x13, x3, ne + + ldp x4, x5, [acc+32] + ldp x12, x13, [nacc+32] + csel x4, x12, x4, ne + csel x5, x13, x5, ne + + ldp x6, x7, [acc+48] + ldp x12, x13, [nacc+48] + csel x6, x12, x6, ne + csel x7, x13, x7, ne + + ldp x8, x9, [acc+64] + ldp x12, x13, [nacc+64] + csel x8, x12, x8, ne + csel x9, x13, x9, ne + + ldp x10, x11, [acc+80] + ldp x12, x13, [nacc+80] + csel x10, x12, x10, ne + csel x11, x13, x11, ne + + stp x0, x1, [acc] + stp x2, x3, [acc+16] + stp x4, x5, [acc+32] + stp x6, x7, [acc+48] + stp x8, x9, [acc+64] + stp x10, x11, [acc+80] + +// Loop while blocksize * i <= 256 + + add i, i, #1 + mul x0, blocksize, i + cmp x0, #257 + bcc Lp256_scalarmulbase_loop + +// That's the end of the main loop, and we just need to translate +// back from the Jacobian representation to affine. First of all, +// let z2 = 1/z^2 and z3 = 1/z^3, both without Montgomery form + + add x0, z2 + add x1, acc+64 + CFI_BL(Lp256_scalarmulbase_local_montsqr_p256) + + add x0, z3 + add x1, acc+64 + add x2, z2 + CFI_BL(Lp256_scalarmulbase_local_montmul_p256) + + add x0, z2 + add x1, z3 + CFI_BL(Lp256_scalarmulbase_local_demont_p256) + + add x0, z3 + add x1, z2 + CFI_BL(Lp256_scalarmulbase_local_inv_p256) + + add x0, z2 + add x1, acc+64 + add x2, z3 + CFI_BL(Lp256_scalarmulbase_local_montmul_p256) + +// Convert back from Jacobian (X,Y,Z) |-> (X/Z^2, Y/Z^3) + + mov x0, res + add x1, acc + add x2, z2 + CFI_BL(Lp256_scalarmulbase_local_montmul_p256) + + add x0, res, #32 + add x1, acc+32 + add x2, z3 + CFI_BL(Lp256_scalarmulbase_local_montmul_p256) + +// Restore stack and registers and return + + CFI_INC_SP(NSPACE) + CFI_POP2(x25,x30) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_scalarmulbase) + +// Local copies of subroutines, complete clones at the moment + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_local_demont_p256) + +Lp256_scalarmulbase_local_demont_p256: + CFI_START + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + lsl x7, x2, #32 + subs x8, x2, x7 + lsr x6, x2, #32 + sbc x2, x2, x6 + adds x3, x3, x7 + adcs x4, x4, x6 + adcs x5, x5, x8 + adc x2, x2, xzr + lsl x7, x3, #32 + subs x8, x3, x7 + lsr x6, x3, #32 + sbc x3, x3, x6 + adds x4, x4, x7 + adcs x5, x5, x6 + adcs x2, x2, x8 + adc x3, x3, xzr + lsl x7, x4, #32 + subs x8, x4, x7 + lsr x6, x4, #32 + sbc x4, x4, x6 + adds x5, x5, x7 + adcs x2, x2, x6 + adcs x3, x3, x8 + adc x4, x4, xzr + lsl x7, x5, #32 + subs x8, x5, x7 + lsr x6, x5, #32 + sbc x5, x5, x6 + adds x2, x2, x7 + adcs x3, x3, x6 + adcs x4, x4, x8 + adc x5, x5, xzr + stp x2, x3, [x0] + stp x4, x5, [x0, #16] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_local_demont_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_local_inv_p256) + +Lp256_scalarmulbase_local_inv_p256: + CFI_START + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_DEC_SP(160) + mov x20, x0 + mov x10, #0xffffffffffffffff + mov x11, #0xffffffff + mov x13, #0xffffffff00000001 + stp x10, x11, [sp] + stp xzr, x13, [sp, #16] + str xzr, [sp, #32] + ldp x2, x3, [x1] + subs x10, x2, x10 + sbcs x11, x3, x11 + ldp x4, x5, [x1, #16] + sbcs x12, x4, xzr + sbcs x13, x5, x13 + csel x2, x2, x10, cc + csel x3, x3, x11, cc + csel x4, x4, x12, cc + csel x5, x5, x13, cc + stp x2, x3, [sp, #48] + stp x4, x5, [sp, #64] + str xzr, [sp, #80] + stp xzr, xzr, [sp, #96] + stp xzr, xzr, [sp, #112] + mov x10, #0x4000000000000 + stp x10, xzr, [sp, #128] + stp xzr, xzr, [sp, #144] + mov x21, #0xa + mov x22, #0x1 + b Lp256_scalarmulbase_inv_midloop +Lp256_scalarmulbase_inv_loop: + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + and x0, x12, x16 + and x1, x13, x17 + add x19, x0, x1 + ldr x7, [sp] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #48] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x7, [sp, #8] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #56] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + adc x6, x6, x1 + extr x4, x2, x4, #59 + str x4, [sp] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + adc x4, x4, x1 + extr x5, x3, x5, #59 + str x5, [sp, #48] + ldr x7, [sp, #16] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #64] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + adc x5, x5, x1 + extr x2, x6, x2, #59 + str x2, [sp, #8] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + adc x2, x2, x1 + extr x3, x4, x3, #59 + str x3, [sp, #56] + ldr x7, [sp, #24] + eor x1, x7, x14 + ldr x23, [sp, #32] + eor x3, x23, x14 + and x3, x3, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #72] + eor x1, x8, x15 + ldr x24, [sp, #80] + eor x0, x24, x15 + and x0, x0, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x5, x6, #59 + str x6, [sp, #16] + extr x5, x3, x5, #59 + str x5, [sp, #24] + asr x3, x3, #59 + str x3, [sp, #32] + eor x1, x7, x16 + eor x5, x23, x16 + and x5, x5, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + eor x0, x24, x17 + and x0, x0, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x4, x2, x4, #59 + str x4, [sp, #64] + extr x2, x5, x2, #59 + str x2, [sp, #72] + asr x5, x5, #59 + str x5, [sp, #80] + ldr x7, [sp, #96] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #128] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #96] + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + str x5, [sp, #128] + adc x3, x3, x1 + ldr x7, [sp, #104] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #136] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #104] + adc x6, x6, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + str x3, [sp, #136] + adc x4, x4, x1 + ldr x7, [sp, #112] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #144] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #112] + adc x5, x5, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + str x4, [sp, #144] + adc x2, x2, x1 + ldr x7, [sp, #120] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #152] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + ldp x0, x1, [sp, #96] + ldr x6, [sp, #112] + mov x14, #0xe000000000000000 + adds x0, x0, x14 + sbcs x1, x1, xzr + mov x11, #0x1fffffff + adcs x6, x6, x11 + mov x10, #0x2000000000000000 + adcs x5, x5, x10 + mov x14, #0x1fffffffe0000000 + adc x3, x3, x14 + lsl x11, x0, #32 + subs x14, x0, x11 + lsr x10, x0, #32 + sbc x0, x0, x10 + adds x1, x1, x11 + adcs x6, x6, x10 + adcs x5, x5, x14 + adcs x3, x3, x0 + mov x14, #0xffffffffffffffff + mov x11, #0xffffffff + mov x10, #0xffffffff00000001 + csel x14, x14, xzr, cs + csel x11, x11, xzr, cs + csel x10, x10, xzr, cs + subs x1, x1, x14 + sbcs x6, x6, x11 + sbcs x5, x5, xzr + sbc x3, x3, x10 + stp x1, x6, [sp, #96] + stp x5, x3, [sp, #112] + eor x1, x7, x16 + and x5, x16, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + and x0, x17, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + ldp x0, x1, [sp, #128] + ldr x3, [sp, #144] + mov x14, #0xe000000000000000 + adds x0, x0, x14 + sbcs x1, x1, xzr + mov x11, #0x1fffffff + adcs x3, x3, x11 + mov x10, #0x2000000000000000 + adcs x2, x2, x10 + mov x14, #0x1fffffffe0000000 + adc x5, x5, x14 + lsl x11, x0, #32 + subs x14, x0, x11 + lsr x10, x0, #32 + sbc x0, x0, x10 + adds x1, x1, x11 + adcs x3, x3, x10 + adcs x2, x2, x14 + adcs x5, x5, x0 + mov x14, #0xffffffffffffffff + mov x11, #0xffffffff + mov x10, #0xffffffff00000001 + csel x14, x14, xzr, cs + csel x11, x11, xzr, cs + csel x10, x10, xzr, cs + subs x1, x1, x14 + sbcs x3, x3, x11 + sbcs x2, x2, xzr + sbc x5, x5, x10 + stp x1, x3, [sp, #128] + stp x2, x5, [sp, #144] +Lp256_scalarmulbase_inv_midloop: + mov x1, x22 + ldr x2, [sp] + ldr x3, [sp, #48] + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x8, x4, #0x100, lsl #12 + sbfx x8, x8, #21, #21 + mov x11, #0x100000 + add x11, x11, x11, lsl #21 + add x9, x4, x11 + asr x9, x9, #42 + add x10, x5, #0x100, lsl #12 + sbfx x10, x10, #21, #21 + add x11, x5, x11 + asr x11, x11, #42 + mul x6, x8, x2 + mul x7, x9, x3 + mul x2, x10, x2 + mul x3, x11, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #21, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #42 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #21, #21 + add x15, x5, x15 + asr x15, x15, #42 + mul x6, x12, x2 + mul x7, x13, x3 + mul x2, x14, x2 + mul x3, x15, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + mul x2, x12, x8 + mul x3, x12, x9 + mul x6, x14, x8 + mul x7, x14, x9 + madd x8, x13, x10, x2 + madd x9, x13, x11, x3 + madd x16, x15, x10, x6 + madd x17, x15, x11, x7 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #22, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #43 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #22, #21 + add x15, x5, x15 + asr x15, x15, #43 + mneg x2, x12, x8 + mneg x3, x12, x9 + mneg x4, x14, x8 + mneg x5, x14, x9 + msub x10, x13, x16, x2 + msub x11, x13, x17, x3 + msub x12, x15, x16, x4 + msub x13, x15, x17, x5 + mov x22, x1 + subs x21, x21, #0x1 + bne Lp256_scalarmulbase_inv_loop + ldr x0, [sp] + ldr x1, [sp, #48] + mul x0, x0, x10 + madd x1, x1, x11, x0 + asr x0, x1, #63 + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + eor x14, x14, x0 + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + eor x15, x15, x0 + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + eor x16, x16, x0 + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + eor x17, x17, x0 + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + ldr x7, [sp, #96] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #128] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #96] + adc x2, x2, x1 + ldr x7, [sp, #104] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #136] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #104] + adc x6, x6, x1 + ldr x7, [sp, #112] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #144] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #112] + adc x5, x5, x1 + ldr x7, [sp, #120] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #152] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + ldp x0, x1, [sp, #96] + ldr x2, [sp, #112] + mov x14, #0xe000000000000000 + adds x0, x0, x14 + sbcs x1, x1, xzr + mov x11, #0x1fffffff + adcs x2, x2, x11 + mov x10, #0x2000000000000000 + adcs x5, x5, x10 + mov x14, #0x1fffffffe0000000 + adc x3, x3, x14 + lsl x11, x0, #32 + subs x14, x0, x11 + lsr x10, x0, #32 + sbc x0, x0, x10 + adds x1, x1, x11 + adcs x2, x2, x10 + adcs x5, x5, x14 + adcs x3, x3, x0 + mov x14, #0xffffffffffffffff + mov x11, #0xffffffff + mov x10, #0xffffffff00000001 + csel x14, x14, xzr, cs + csel x11, x11, xzr, cs + csel x10, x10, xzr, cs + subs x1, x1, x14 + sbcs x2, x2, x11 + sbcs x5, x5, xzr + sbc x3, x3, x10 + mov x10, #0xffffffffffffffff + subs x10, x1, x10 + mov x11, #0xffffffff + sbcs x11, x2, x11 + mov x13, #0xffffffff00000001 + sbcs x12, x5, xzr + sbcs x13, x3, x13 + csel x10, x1, x10, cc + csel x11, x2, x11, cc + csel x12, x5, x12, cc + csel x13, x3, x13, cc + stp x10, x11, [x20] + stp x12, x13, [x20, #16] + CFI_INC_SP(160) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_local_inv_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_local_montmul_p256) + +Lp256_scalarmulbase_local_montmul_p256: + CFI_START + ldr q20, [x2] + ldp x7, x17, [x1] + ldr q0, [x1] + ldp x6, x10, [x2] + ldp x11, x15, [x1, #16] + rev64 v16.4S, v20.4S + subs x4, x7, x17 + csetm x3, cc + cneg x13, x4, cc + mul v16.4S, v16.4S, v0.4S + umulh x12, x17, x10 + uzp1 v28.4S, v20.4S, v0.4S + subs x14, x11, x7 + ldr q20, [x2, #16] + sbcs x5, x15, x17 + ngc x17, xzr + subs x8, x11, x15 + uaddlp v27.2D, v16.4S + umulh x4, x7, x6 + uzp1 v21.4S, v0.4S, v0.4S + cneg x11, x8, cc + shl v17.2D, v27.2D, #32 + csetm x15, cc + subs x9, x10, x6 + eor x7, x14, x17 + umlal v17.2D, v21.2S, v28.2S + cneg x8, x9, cc + cinv x9, x3, cc + cmn x17, #0x1 + ldr q28, [x1, #16] + adcs x14, x7, xzr + mul x7, x13, x8 + eor x1, x5, x17 + adcs x5, x1, xzr + xtn v1.2S, v20.2D + mov x1, v17.d[0] + mov x3, v17.d[1] + uzp2 v16.4S, v20.4S, v20.4S + umulh x16, x13, x8 + eor x13, x7, x9 + adds x8, x1, x3 + adcs x7, x4, x12 + xtn v0.2S, v28.2D + adcs x12, x12, xzr + adds x8, x4, x8 + adcs x3, x3, x7 + ldp x7, x2, [x2, #16] + adcs x12, x12, xzr + cmn x9, #0x1 + adcs x8, x8, x13 + eor x13, x16, x9 + adcs x16, x3, x13 + lsl x3, x1, #32 + adc x13, x12, x9 + subs x12, x6, x7 + sbcs x9, x10, x2 + lsr x10, x1, #32 + ngc x4, xzr + subs x6, x2, x7 + cinv x2, x15, cc + cneg x6, x6, cc + subs x7, x1, x3 + eor x9, x9, x4 + sbc x1, x1, x10 + adds x15, x8, x3 + adcs x3, x16, x10 + mul x16, x11, x6 + adcs x8, x13, x7 + eor x13, x12, x4 + adc x10, x1, xzr + cmn x4, #0x1 + umulh x6, x11, x6 + adcs x11, x13, xzr + adcs x1, x9, xzr + lsl x13, x15, #32 + subs x12, x15, x13 + lsr x7, x15, #32 + sbc x15, x15, x7 + adds x9, x3, x13 + adcs x3, x8, x7 + umulh x8, x14, x11 + umull v21.2D, v0.2S, v1.2S + adcs x12, x10, x12 + umull v3.2D, v0.2S, v16.2S + adc x15, x15, xzr + rev64 v24.4S, v20.4S + stp x12, x15, [x0, #16] + movi v2.2D, #0x00000000ffffffff + mul x10, x14, x11 + mul v4.4S, v24.4S, v28.4S + subs x13, x14, x5 + uzp2 v19.4S, v28.4S, v28.4S + csetm x15, cc + usra v3.2D, v21.2D, #32 + mul x7, x5, x1 + umull v21.2D, v19.2S, v16.2S + cneg x13, x13, cc + uaddlp v5.2D, v4.4S + subs x11, x1, x11 + and v16.16B, v3.16B, v2.16B + umulh x5, x5, x1 + shl v24.2D, v5.2D, #32 + cneg x11, x11, cc + umlal v16.2D, v19.2S, v1.2S + cinv x12, x15, cc + umlal v24.2D, v0.2S, v1.2S + adds x15, x10, x7 + mul x14, x13, x11 + eor x1, x6, x2 + adcs x6, x8, x5 + stp x9, x3, [x0] + usra v21.2D, v3.2D, #32 + adcs x9, x5, xzr + umulh x11, x13, x11 + adds x15, x8, x15 + adcs x7, x7, x6 + eor x8, x14, x12 + usra v21.2D, v16.2D, #32 + adcs x13, x9, xzr + cmn x12, #0x1 + mov x9, v24.d[1] + adcs x14, x15, x8 + eor x6, x11, x12 + adcs x6, x7, x6 + mov x5, v24.d[0] + mov x11, v21.d[1] + mov x7, v21.d[0] + adc x3, x13, x12 + adds x12, x5, x9 + adcs x13, x7, x11 + ldp x15, x8, [x0] + adcs x11, x11, xzr + adds x12, x7, x12 + eor x16, x16, x2 + adcs x7, x9, x13 + adcs x11, x11, xzr + cmn x2, #0x1 + ldp x9, x13, [x0, #16] + adcs x16, x12, x16 + adcs x1, x7, x1 + adc x2, x11, x2 + adds x7, x5, x15 + adcs x15, x16, x8 + eor x5, x17, x4 + adcs x9, x1, x9 + eor x1, x10, x5 + adcs x16, x2, x13 + adc x2, xzr, xzr + cmn x5, #0x1 + eor x13, x14, x5 + adcs x14, x1, x7 + eor x1, x6, x5 + adcs x6, x13, x15 + adcs x10, x1, x9 + eor x4, x3, x5 + mov x1, #0xffffffff + adcs x8, x4, x16 + lsr x13, x14, #32 + adcs x17, x2, x5 + adcs x11, x5, xzr + adc x4, x5, xzr + adds x12, x10, x7 + adcs x7, x8, x15 + adcs x5, x17, x9 + adcs x9, x11, x16 + lsl x11, x14, #32 + adc x10, x4, x2 + subs x17, x14, x11 + sbc x4, x14, x13 + adds x11, x6, x11 + adcs x12, x12, x13 + lsl x15, x11, #32 + adcs x17, x7, x17 + lsr x7, x11, #32 + adc x13, x4, xzr + subs x4, x11, x15 + sbc x11, x11, x7 + adds x8, x12, x15 + adcs x15, x17, x7 + adcs x4, x13, x4 + adc x11, x11, xzr + adds x7, x5, x4 + adcs x17, x9, x11 + adc x13, x10, xzr + add x12, x13, #0x1 + neg x11, x12 + lsl x4, x12, #32 + adds x17, x17, x4 + sub x4, x4, #0x1 + adc x13, x13, xzr + subs x11, x8, x11 + sbcs x4, x15, x4 + sbcs x7, x7, xzr + sbcs x17, x17, x12 + sbcs x13, x13, x12 + mov x12, #0xffffffff00000001 + adds x11, x11, x13 + and x1, x1, x13 + adcs x4, x4, x1 + and x1, x12, x13 + stp x11, x4, [x0] + adcs x4, x7, xzr + adc x1, x17, x1 + stp x4, x1, [x0, #16] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_local_montmul_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_local_montsqr_p256) + +Lp256_scalarmulbase_local_montsqr_p256: + CFI_START + ldr q19, [x1] + ldp x9, x13, [x1] + ldr q23, [x1, #16] + ldr q0, [x1] + ldp x1, x10, [x1, #16] + uzp2 v29.4S, v19.4S, v19.4S + xtn v4.2S, v19.2D + umulh x8, x9, x13 + rev64 v20.4S, v23.4S + umull v16.2D, v19.2S, v19.2S + umull v1.2D, v29.2S, v4.2S + mul v20.4S, v20.4S, v0.4S + subs x14, x9, x13 + umulh x15, x9, x1 + mov x16, v16.d[1] + umull2 v4.2D, v19.4S, v19.4S + mov x4, v16.d[0] + uzp1 v17.4S, v23.4S, v0.4S + uaddlp v19.2D, v20.4S + lsr x7, x8, #63 + mul x11, x9, x13 + mov x12, v1.d[0] + csetm x5, cc + cneg x6, x14, cc + mov x3, v4.d[1] + mov x14, v4.d[0] + subs x2, x10, x1 + mov x9, v1.d[1] + cneg x17, x2, cc + cinv x2, x5, cc + adds x5, x4, x12, lsl #33 + extr x4, x8, x11, #63 + lsr x8, x12, #31 + uzp1 v20.4S, v0.4S, v0.4S + shl v19.2D, v19.2D, #32 + adc x16, x16, x8 + adds x8, x14, x9, lsl #33 + lsr x14, x9, #31 + lsl x9, x5, #32 + umlal v19.2D, v20.2S, v17.2S + adc x14, x3, x14 + adds x16, x16, x11, lsl #1 + lsr x3, x5, #32 + umulh x12, x6, x17 + adcs x4, x8, x4 + adc x11, x14, x7 + subs x8, x5, x9 + sbc x5, x5, x3 + adds x16, x16, x9 + mov x14, v19.d[0] + mul x17, x6, x17 + adcs x3, x4, x3 + lsl x7, x16, #32 + umulh x13, x13, x10 + adcs x11, x11, x8 + lsr x8, x16, #32 + adc x5, x5, xzr + subs x9, x16, x7 + sbc x16, x16, x8 + adds x7, x3, x7 + mov x3, v19.d[1] + adcs x6, x11, x8 + umulh x11, x1, x10 + adcs x5, x5, x9 + eor x8, x12, x2 + adc x9, x16, xzr + adds x16, x14, x15 + adc x15, x15, xzr + adds x12, x16, x3 + eor x16, x17, x2 + mul x4, x1, x10 + adcs x15, x15, x13 + adc x17, x13, xzr + adds x15, x15, x3 + adc x3, x17, xzr + cmn x2, #0x1 + mul x17, x10, x10 + adcs x12, x12, x16 + adcs x16, x15, x8 + umulh x10, x10, x10 + adc x2, x3, x2 + adds x14, x14, x14 + adcs x12, x12, x12 + adcs x16, x16, x16 + adcs x2, x2, x2 + adc x15, xzr, xzr + adds x14, x14, x7 + mul x3, x1, x1 + adcs x12, x12, x6 + lsr x7, x14, #32 + adcs x16, x16, x5 + lsl x5, x14, #32 + umulh x13, x1, x1 + adcs x2, x2, x9 + mov x6, #0xffffffff + adc x15, x15, xzr + adds x8, x4, x4 + adcs x1, x11, x11 + mov x11, #0xffffffff00000001 + adc x4, xzr, xzr + subs x9, x14, x5 + sbc x14, x14, x7 + adds x12, x12, x5 + adcs x16, x16, x7 + lsl x5, x12, #32 + lsr x7, x12, #32 + adcs x2, x2, x9 + adcs x14, x15, x14 + adc x15, xzr, xzr + subs x9, x12, x5 + sbc x12, x12, x7 + adds x16, x16, x5 + adcs x2, x2, x7 + adcs x14, x14, x9 + adcs x12, x15, x12 + adc x15, xzr, xzr + adds x16, x16, x3 + adcs x2, x2, x13 + adcs x14, x14, x17 + adcs x12, x12, x10 + adc x15, x15, xzr + adds x2, x2, x8 + adcs x14, x14, x1 + adcs x12, x12, x4 + adcs x15, x15, xzr + adds x3, x16, #0x1 + sbcs x5, x2, x6 + sbcs x8, x14, xzr + sbcs x11, x12, x11 + sbcs xzr, x15, xzr + csel x16, x3, x16, cs + csel x14, x8, x14, cs + csel x12, x11, x12, cs + csel x2, x5, x2, cs + stp x14, x12, [x0, #16] + stp x16, x2, [x0] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_local_montsqr_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_local_p256_montjmixadd) + +Lp256_scalarmulbase_local_p256_montjmixadd: + CFI_START + CFI_PUSH2(x19,x20) + CFI_DEC_SP(192) + mov x17, x0 + mov x19, x1 + mov x20, x2 + ldp x2, x3, [x19, #64] + ldp x4, x5, [x19, #80] + umull x15, w2, w2 + lsr x11, x2, #32 + umull x16, w11, w11 + umull x11, w2, w11 + adds x15, x15, x11, lsl #33 + lsr x11, x11, #31 + adc x16, x16, x11 + umull x0, w3, w3 + lsr x11, x3, #32 + umull x1, w11, w11 + umull x11, w3, w11 + mul x12, x2, x3 + umulh x13, x2, x3 + adds x0, x0, x11, lsl #33 + lsr x11, x11, #31 + adc x1, x1, x11 + adds x12, x12, x12 + adcs x13, x13, x13 + adc x1, x1, xzr + adds x16, x16, x12 + adcs x0, x0, x13 + adc x1, x1, xzr + lsl x12, x15, #32 + subs x13, x15, x12 + lsr x11, x15, #32 + sbc x15, x15, x11 + adds x16, x16, x12 + adcs x0, x0, x11 + adcs x1, x1, x13 + adc x15, x15, xzr + lsl x12, x16, #32 + subs x13, x16, x12 + lsr x11, x16, #32 + sbc x16, x16, x11 + adds x0, x0, x12 + adcs x1, x1, x11 + adcs x15, x15, x13 + adc x16, x16, xzr + mul x6, x2, x4 + mul x14, x3, x5 + umulh x8, x2, x4 + subs x10, x2, x3 + cneg x10, x10, cc + csetm x13, cc + subs x12, x5, x4 + cneg x12, x12, cc + mul x11, x10, x12 + umulh x12, x10, x12 + cinv x13, x13, cc + eor x11, x11, x13 + eor x12, x12, x13 + adds x7, x6, x8 + adc x8, x8, xzr + umulh x9, x3, x5 + adds x7, x7, x14 + adcs x8, x8, x9 + adc x9, x9, xzr + adds x8, x8, x14 + adc x9, x9, xzr + cmn x13, #0x1 + adcs x7, x7, x11 + adcs x8, x8, x12 + adc x9, x9, x13 + adds x6, x6, x6 + adcs x7, x7, x7 + adcs x8, x8, x8 + adcs x9, x9, x9 + adc x10, xzr, xzr + adds x6, x6, x0 + adcs x7, x7, x1 + adcs x8, x8, x15 + adcs x9, x9, x16 + adc x10, x10, xzr + lsl x12, x6, #32 + subs x13, x6, x12 + lsr x11, x6, #32 + sbc x6, x6, x11 + adds x7, x7, x12 + adcs x8, x8, x11 + adcs x9, x9, x13 + adcs x10, x10, x6 + adc x6, xzr, xzr + lsl x12, x7, #32 + subs x13, x7, x12 + lsr x11, x7, #32 + sbc x7, x7, x11 + adds x8, x8, x12 + adcs x9, x9, x11 + adcs x10, x10, x13 + adcs x6, x6, x7 + adc x7, xzr, xzr + mul x11, x4, x4 + adds x8, x8, x11 + mul x12, x5, x5 + umulh x11, x4, x4 + adcs x9, x9, x11 + adcs x10, x10, x12 + umulh x12, x5, x5 + adcs x6, x6, x12 + adc x7, x7, xzr + mul x11, x4, x5 + umulh x12, x4, x5 + adds x11, x11, x11 + adcs x12, x12, x12 + adc x13, xzr, xzr + adds x9, x9, x11 + adcs x10, x10, x12 + adcs x6, x6, x13 + adcs x7, x7, xzr + mov x11, #0xffffffff + adds x5, x8, #0x1 + sbcs x11, x9, x11 + mov x13, #0xffffffff00000001 + sbcs x12, x10, xzr + sbcs x13, x6, x13 + sbcs xzr, x7, xzr + csel x8, x5, x8, cs + csel x9, x11, x9, cs + csel x10, x12, x10, cs + csel x6, x13, x6, cs + stp x8, x9, [sp] + stp x10, x6, [sp, #16] + ldp x3, x4, [x19, #64] + ldp x5, x6, [x19, #80] + ldp x7, x8, [x20, #32] + ldp x9, x10, [x20, #48] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #32] + stp x11, x12, [sp, #48] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #32] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #48] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #32] + stp x3, x4, [sp, #48] + ldp x3, x4, [sp] + ldp x5, x6, [sp, #16] + ldp x7, x8, [x20] + ldp x9, x10, [x20, #16] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #64] + stp x11, x12, [sp, #80] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #64] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #80] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #64] + stp x3, x4, [sp, #80] + ldp x3, x4, [sp] + ldp x5, x6, [sp, #16] + ldp x7, x8, [sp, #32] + ldp x9, x10, [sp, #48] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #32] + stp x11, x12, [sp, #48] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #32] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #48] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #32] + stp x3, x4, [sp, #48] + ldp x5, x6, [sp, #64] + ldp x4, x3, [x19] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [x19, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #160] + stp x7, x8, [sp, #176] + ldp x5, x6, [sp, #32] + ldp x4, x3, [x19, #32] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #48] + ldp x4, x3, [x19, #48] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #32] + stp x7, x8, [sp, #48] + ldp x2, x3, [sp, #160] + ldp x4, x5, [sp, #176] + umull x15, w2, w2 + lsr x11, x2, #32 + umull x16, w11, w11 + umull x11, w2, w11 + adds x15, x15, x11, lsl #33 + lsr x11, x11, #31 + adc x16, x16, x11 + umull x0, w3, w3 + lsr x11, x3, #32 + umull x1, w11, w11 + umull x11, w3, w11 + mul x12, x2, x3 + umulh x13, x2, x3 + adds x0, x0, x11, lsl #33 + lsr x11, x11, #31 + adc x1, x1, x11 + adds x12, x12, x12 + adcs x13, x13, x13 + adc x1, x1, xzr + adds x16, x16, x12 + adcs x0, x0, x13 + adc x1, x1, xzr + lsl x12, x15, #32 + subs x13, x15, x12 + lsr x11, x15, #32 + sbc x15, x15, x11 + adds x16, x16, x12 + adcs x0, x0, x11 + adcs x1, x1, x13 + adc x15, x15, xzr + lsl x12, x16, #32 + subs x13, x16, x12 + lsr x11, x16, #32 + sbc x16, x16, x11 + adds x0, x0, x12 + adcs x1, x1, x11 + adcs x15, x15, x13 + adc x16, x16, xzr + mul x6, x2, x4 + mul x14, x3, x5 + umulh x8, x2, x4 + subs x10, x2, x3 + cneg x10, x10, cc + csetm x13, cc + subs x12, x5, x4 + cneg x12, x12, cc + mul x11, x10, x12 + umulh x12, x10, x12 + cinv x13, x13, cc + eor x11, x11, x13 + eor x12, x12, x13 + adds x7, x6, x8 + adc x8, x8, xzr + umulh x9, x3, x5 + adds x7, x7, x14 + adcs x8, x8, x9 + adc x9, x9, xzr + adds x8, x8, x14 + adc x9, x9, xzr + cmn x13, #0x1 + adcs x7, x7, x11 + adcs x8, x8, x12 + adc x9, x9, x13 + adds x6, x6, x6 + adcs x7, x7, x7 + adcs x8, x8, x8 + adcs x9, x9, x9 + adc x10, xzr, xzr + adds x6, x6, x0 + adcs x7, x7, x1 + adcs x8, x8, x15 + adcs x9, x9, x16 + adc x10, x10, xzr + lsl x12, x6, #32 + subs x13, x6, x12 + lsr x11, x6, #32 + sbc x6, x6, x11 + adds x7, x7, x12 + adcs x8, x8, x11 + adcs x9, x9, x13 + adcs x10, x10, x6 + adc x6, xzr, xzr + lsl x12, x7, #32 + subs x13, x7, x12 + lsr x11, x7, #32 + sbc x7, x7, x11 + adds x8, x8, x12 + adcs x9, x9, x11 + adcs x10, x10, x13 + adcs x6, x6, x7 + adc x7, xzr, xzr + mul x11, x4, x4 + adds x8, x8, x11 + mul x12, x5, x5 + umulh x11, x4, x4 + adcs x9, x9, x11 + adcs x10, x10, x12 + umulh x12, x5, x5 + adcs x6, x6, x12 + adc x7, x7, xzr + mul x11, x4, x5 + umulh x12, x4, x5 + adds x11, x11, x11 + adcs x12, x12, x12 + adc x13, xzr, xzr + adds x9, x9, x11 + adcs x10, x10, x12 + adcs x6, x6, x13 + adcs x7, x7, xzr + mov x11, #0xffffffff + adds x5, x8, #0x1 + sbcs x11, x9, x11 + mov x13, #0xffffffff00000001 + sbcs x12, x10, xzr + sbcs x13, x6, x13 + sbcs xzr, x7, xzr + csel x8, x5, x8, cs + csel x9, x11, x9, cs + csel x10, x12, x10, cs + csel x6, x13, x6, cs + stp x8, x9, [sp, #96] + stp x10, x6, [sp, #112] + ldp x2, x3, [sp, #32] + ldp x4, x5, [sp, #48] + umull x15, w2, w2 + lsr x11, x2, #32 + umull x16, w11, w11 + umull x11, w2, w11 + adds x15, x15, x11, lsl #33 + lsr x11, x11, #31 + adc x16, x16, x11 + umull x0, w3, w3 + lsr x11, x3, #32 + umull x1, w11, w11 + umull x11, w3, w11 + mul x12, x2, x3 + umulh x13, x2, x3 + adds x0, x0, x11, lsl #33 + lsr x11, x11, #31 + adc x1, x1, x11 + adds x12, x12, x12 + adcs x13, x13, x13 + adc x1, x1, xzr + adds x16, x16, x12 + adcs x0, x0, x13 + adc x1, x1, xzr + lsl x12, x15, #32 + subs x13, x15, x12 + lsr x11, x15, #32 + sbc x15, x15, x11 + adds x16, x16, x12 + adcs x0, x0, x11 + adcs x1, x1, x13 + adc x15, x15, xzr + lsl x12, x16, #32 + subs x13, x16, x12 + lsr x11, x16, #32 + sbc x16, x16, x11 + adds x0, x0, x12 + adcs x1, x1, x11 + adcs x15, x15, x13 + adc x16, x16, xzr + mul x6, x2, x4 + mul x14, x3, x5 + umulh x8, x2, x4 + subs x10, x2, x3 + cneg x10, x10, cc + csetm x13, cc + subs x12, x5, x4 + cneg x12, x12, cc + mul x11, x10, x12 + umulh x12, x10, x12 + cinv x13, x13, cc + eor x11, x11, x13 + eor x12, x12, x13 + adds x7, x6, x8 + adc x8, x8, xzr + umulh x9, x3, x5 + adds x7, x7, x14 + adcs x8, x8, x9 + adc x9, x9, xzr + adds x8, x8, x14 + adc x9, x9, xzr + cmn x13, #0x1 + adcs x7, x7, x11 + adcs x8, x8, x12 + adc x9, x9, x13 + adds x6, x6, x6 + adcs x7, x7, x7 + adcs x8, x8, x8 + adcs x9, x9, x9 + adc x10, xzr, xzr + adds x6, x6, x0 + adcs x7, x7, x1 + adcs x8, x8, x15 + adcs x9, x9, x16 + adc x10, x10, xzr + lsl x12, x6, #32 + subs x13, x6, x12 + lsr x11, x6, #32 + sbc x6, x6, x11 + adds x7, x7, x12 + adcs x8, x8, x11 + adcs x9, x9, x13 + adcs x10, x10, x6 + adc x6, xzr, xzr + lsl x12, x7, #32 + subs x13, x7, x12 + lsr x11, x7, #32 + sbc x7, x7, x11 + adds x8, x8, x12 + adcs x9, x9, x11 + adcs x10, x10, x13 + adcs x6, x6, x7 + adc x7, xzr, xzr + mul x11, x4, x4 + adds x8, x8, x11 + mul x12, x5, x5 + umulh x11, x4, x4 + adcs x9, x9, x11 + adcs x10, x10, x12 + umulh x12, x5, x5 + adcs x6, x6, x12 + adc x7, x7, xzr + mul x11, x4, x5 + umulh x12, x4, x5 + adds x11, x11, x11 + adcs x12, x12, x12 + adc x13, xzr, xzr + adds x9, x9, x11 + adcs x10, x10, x12 + adcs x6, x6, x13 + adcs x7, x7, xzr + mov x11, #0xffffffff + adds x5, x8, #0x1 + sbcs x11, x9, x11 + mov x13, #0xffffffff00000001 + sbcs x12, x10, xzr + sbcs x13, x6, x13 + sbcs xzr, x7, xzr + csel x8, x5, x8, cs + csel x9, x11, x9, cs + csel x10, x12, x10, cs + csel x6, x13, x6, cs + stp x8, x9, [sp] + stp x10, x6, [sp, #16] + ldp x3, x4, [sp, #96] + ldp x5, x6, [sp, #112] + ldp x7, x8, [x19] + ldp x9, x10, [x19, #16] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #128] + stp x11, x12, [sp, #144] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #128] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #144] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #128] + stp x3, x4, [sp, #144] + ldp x3, x4, [sp, #96] + ldp x5, x6, [sp, #112] + ldp x7, x8, [sp, #64] + ldp x9, x10, [sp, #80] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #64] + stp x11, x12, [sp, #80] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #64] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #80] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #64] + stp x3, x4, [sp, #80] + ldp x5, x6, [sp] + ldp x4, x3, [sp, #128] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #16] + ldp x4, x3, [sp, #144] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp] + stp x7, x8, [sp, #16] + ldp x5, x6, [sp, #64] + ldp x4, x3, [sp, #128] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [sp, #144] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #96] + stp x7, x8, [sp, #112] + ldp x3, x4, [sp, #160] + ldp x5, x6, [sp, #176] + ldp x7, x8, [x19, #64] + ldp x9, x10, [x19, #80] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #160] + stp x11, x12, [sp, #176] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #160] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #176] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #160] + stp x3, x4, [sp, #176] + ldp x5, x6, [sp] + ldp x4, x3, [sp, #64] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #16] + ldp x4, x3, [sp, #80] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp] + stp x7, x8, [sp, #16] + ldp x5, x6, [sp, #128] + ldp x4, x3, [sp] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #144] + ldp x4, x3, [sp, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #128] + stp x7, x8, [sp, #144] + ldp x3, x4, [sp, #96] + ldp x5, x6, [sp, #112] + ldp x7, x8, [x19, #32] + ldp x9, x10, [x19, #48] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #96] + stp x11, x12, [sp, #112] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #96] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #112] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #96] + stp x3, x4, [sp, #112] + ldp x3, x4, [sp, #32] + ldp x5, x6, [sp, #48] + ldp x7, x8, [sp, #128] + ldp x9, x10, [sp, #144] + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x3, x4 + cneg x15, x15, cc + csetm x1, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + stp x13, x14, [sp, #128] + stp x11, x12, [sp, #144] + mul x11, x5, x9 + mul x13, x6, x10 + umulh x12, x5, x9 + adds x16, x11, x13 + umulh x14, x6, x10 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x15, x5, x6 + cneg x15, x15, cc + csetm x1, cc + subs x0, x10, x9 + cneg x0, x0, cc + mul x16, x15, x0 + umulh x0, x15, x0 + cinv x1, x1, cc + eor x16, x16, x1 + eor x0, x0, x1 + cmn x1, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x1 + subs x3, x5, x3 + sbcs x4, x6, x4 + ngc x5, xzr + cmn x5, #0x1 + eor x3, x3, x5 + adcs x3, x3, xzr + eor x4, x4, x5 + adcs x4, x4, xzr + subs x7, x7, x9 + sbcs x8, x8, x10 + ngc x9, xzr + cmn x9, #0x1 + eor x7, x7, x9 + adcs x7, x7, xzr + eor x8, x8, x9 + adcs x8, x8, xzr + eor x10, x5, x9 + ldp x15, x1, [sp, #128] + adds x15, x11, x15 + adcs x1, x12, x1 + ldp x5, x9, [sp, #144] + adcs x5, x13, x5 + adcs x9, x14, x9 + adc x2, xzr, xzr + mul x11, x3, x7 + mul x13, x4, x8 + umulh x12, x3, x7 + adds x16, x11, x13 + umulh x14, x4, x8 + adcs x0, x12, x14 + adcs x14, x14, xzr + adds x12, x12, x16 + adcs x13, x13, x0 + adcs x14, x14, xzr + subs x3, x3, x4 + cneg x3, x3, cc + csetm x4, cc + subs x0, x8, x7 + cneg x0, x0, cc + mul x16, x3, x0 + umulh x0, x3, x0 + cinv x4, x4, cc + eor x16, x16, x4 + eor x0, x0, x4 + cmn x4, #0x1 + adcs x12, x12, x16 + adcs x13, x13, x0 + adc x14, x14, x4 + cmn x10, #0x1 + eor x11, x11, x10 + adcs x11, x11, x15 + eor x12, x12, x10 + adcs x12, x12, x1 + eor x13, x13, x10 + adcs x13, x13, x5 + eor x14, x14, x10 + adcs x14, x14, x9 + adcs x3, x2, x10 + adcs x4, x10, xzr + adc x10, x10, xzr + adds x13, x13, x15 + adcs x14, x14, x1 + adcs x3, x3, x5 + adcs x4, x4, x9 + adc x10, x10, x2 + lsl x0, x11, #32 + subs x1, x11, x0 + lsr x16, x11, #32 + sbc x11, x11, x16 + adds x12, x12, x0 + adcs x13, x13, x16 + adcs x14, x14, x1 + adc x11, x11, xzr + lsl x0, x12, #32 + subs x1, x12, x0 + lsr x16, x12, #32 + sbc x12, x12, x16 + adds x13, x13, x0 + adcs x14, x14, x16 + adcs x11, x11, x1 + adc x12, x12, xzr + adds x3, x3, x11 + adcs x4, x4, x12 + adc x10, x10, xzr + add x2, x10, #0x1 + lsl x16, x2, #32 + adds x4, x4, x16 + adc x10, x10, xzr + neg x15, x2 + sub x16, x16, #0x1 + subs x13, x13, x15 + sbcs x14, x14, x16 + sbcs x3, x3, xzr + sbcs x4, x4, x2 + sbcs x7, x10, x2 + adds x13, x13, x7 + mov x10, #0xffffffff + and x10, x10, x7 + adcs x14, x14, x10 + adcs x3, x3, xzr + mov x10, #0xffffffff00000001 + and x10, x10, x7 + adc x4, x4, x10 + stp x13, x14, [sp, #128] + stp x3, x4, [sp, #144] + ldp x5, x6, [sp, #128] + ldp x4, x3, [sp, #96] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #144] + ldp x4, x3, [sp, #112] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #128] + stp x7, x8, [sp, #144] + ldp x0, x1, [x19, #64] + ldp x2, x3, [x19, #80] + orr x4, x0, x1 + orr x5, x2, x3 + orr x4, x4, x5 + cmp x4, xzr + ldp x0, x1, [sp] + ldp x12, x13, [x20] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x2, x3, [sp, #16] + ldp x12, x13, [x20, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x4, x5, [sp, #128] + ldp x12, x13, [x20, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x6, x7, [sp, #144] + ldp x12, x13, [x20, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x8, x9, [sp, #160] + mov x12, #0x1 + mov x13, #0xffffffff00000000 + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x10, x11, [sp, #176] + mov x12, #0xffffffffffffffff + mov x13, #0xfffffffe + csel x10, x10, x12, ne + csel x11, x11, x13, ne + stp x0, x1, [x17] + stp x2, x3, [x17, #16] + stp x4, x5, [x17, #32] + stp x6, x7, [x17, #48] + stp x8, x9, [x17, #64] + stp x10, x11, [x17, #80] + CFI_INC_SP(192) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_local_p256_montjmixadd) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/p256_scalarmulbase_alt.S b/cbits/s2n/arm/p256_scalarmulbase_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/p256_scalarmulbase_alt.S @@ -0,0 +1,3057 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Scalar multiplication for precomputed point on NIST curve P-256 +// Input scalar[4], blocksize, table[]; output res[8] +// +// extern void p256_scalarmulbase_alt +// (uint64_t res[static 8], +// const uint64_t scalar[static 4], +// uint64_t blocksize, +// const uint64_t *table); +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-256, the input argument "table" is expected to be a table of +// multiples of the point P in Montgomery-affine form, with each block +// corresponding to "blocksize" bits of the scalar as follows, where +// B = 2^{blocksize-1} (e.g. B = 8 for blocksize = 4): +// +// For each i,j with blocksize * i <= 256 and 1 <= j <= B +// the multiple 2^{blocksize * i} * j * P is stored at +// tab[8 * (B * i + (j - 1))], considered as uint64_t pointers +// or tab + 64 * (B * i + (j - 1)) as byte pointers. +// +// Standard ARM ABI: X0 = res, X1 = scalar, X2 = blocksize, X3 = table +// ---------------------------------------------------------------------------- + + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_scalarmulbase_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_scalarmulbase_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_scalarmulbase_alt) + + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Safe copies of inputs and additional variables, with some aliasing + +#define res x19 +#define blocksize x20 +#define table x21 +#define i x22 +#define bf x23 +#define cf x24 +#define j x25 + +// Intermediate variables on the stack. The last z2, z3 values can +// safely be overlaid on "nacc", which is no longer needed at the end. +// Uppercase syntactic variants make x86_att version simpler to generate + +#define rscalar sp, #(0*NUMSIZE) +#define acc sp, #(1*NUMSIZE) +#define nacc sp, #(4*NUMSIZE) +#define tabent sp, #(7*NUMSIZE) + +#define z2 sp, #(4*NUMSIZE) +#define z3 sp, #(5*NUMSIZE) + +#define NSPACE 9*NUMSIZE + +// Loading large constants + +#define movbig(nn,n3,n2,n1,n0) \ + movz nn, n0 __LF \ + movk nn, n1, lsl #16 __LF \ + movk nn, n2, lsl #32 __LF \ + movk nn, n3, lsl #48 + +S2N_BN_SYMBOL(p256_scalarmulbase_alt): + CFI_START + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x30) + CFI_DEC_SP(NSPACE) + +// Preserve the input arguments except the scalar, since that gets absorbed +// immediately. The "table" value subsequently gets shifted up each iteration +// of the loop, while "res" and "blocksize" are static throughout. + + mov res, x0 + mov blocksize, x2 + mov table, x3 + +// Load the digits of group order n_256 = [x15;x14;x13;x12] + + movbig(x12, #0xf3b9, #0xcac2, #0xfc63, #0x2551) + movbig(x13, #0xbce6, #0xfaad, #0xa717, #0x9e84) + mov x14, #0xffffffffffffffff + mov x15, #0xffffffff00000000 + +// First, reduce the input scalar mod n_256, i.e. conditionally subtract n_256 +// Store it to "rscalar" (reduced scalar) + + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + + subs x6, x2, x12 + sbcs x7, x3, x13 + sbcs x8, x4, x14 + sbcs x9, x5, x15 + + csel x2, x2, x6, cc + csel x3, x3, x7, cc + csel x4, x4, x8, cc + csel x5, x5, x9, cc + + stp x2, x3, [rscalar] + stp x4, x5, [rscalar+16] + +// Initialize the accumulator to all zeros and the "carry flag" cf to 0 + + stp xzr, xzr, [acc] + stp xzr, xzr, [acc+16] + stp xzr, xzr, [acc+32] + stp xzr, xzr, [acc+48] + stp xzr, xzr, [acc+64] + stp xzr, xzr, [acc+80] + mov cf, xzr + +// Main loop over {i >= 0 | blocksize * i <= 256}. Note the non-strict +// inequality, to allow top carry for any choices of blocksize. + + mov i, xzr + +Lp256_scalarmulbase_alt_loop: + +// The next raw bitfield is bf = bitfield(blocksize * i,blocksize) + cf, +// adding in the deferred carry cf. We then shift the whole scalar right +// by blocksize so we can keep picking bitfield(0,blocksize). + + ldp x0, x1, [rscalar] + ldp x2, x3, [rscalar+16] + + mov x4, #1 + lsl x4, x4, blocksize + sub x4, x4, #1 + and x4, x4, x0 + add bf, x4, cf + + neg x8, blocksize + + lsl x5, x1, x8 + + lsr x0, x0, blocksize + orr x0, x0, x5 + + lsl x6, x2, x8 + lsr x1, x1, blocksize + orr x1, x1, x6 + + lsl x7, x3, x8 + lsr x2, x2, blocksize + orr x2, x2, x7 + + lsr x3, x3, blocksize + + stp x0, x1, [rscalar] + stp x2, x3, [rscalar+16] + +// Now if bf <= B we just select entry j, unnegated and set cf = 0. +// If bf > B we set j = 2 * B - bf and negate the j'th entry, setting cf = 1. +// In either case we ultimately add bf, in the latter case with deferred +// carry as 2 * B - (2 * B - bf) = bf. + + mov x0, #1 + lsl x1, x0, blocksize + lsr x0, x1, #1 + + sub x2, x1, bf + + cmp x0, bf + cset cf, cc + csel j, x2, bf, cc + +// Load table entry j - 1 for nonzero j in constant-time style. + + mov x16, #1 + lsl x16, x16, blocksize + lsr x16, x16, #1 + mov x17, j + +Lp256_scalarmulbase_alt_tabloop: + ldp x8, x9, [table] + ldp x10, x11, [table, #16] + ldp x12, x13, [table, #32] + ldp x14, x15, [table, #48] + + subs x17, x17, #1 + csel x0, x8, x0, eq + csel x1, x9, x1, eq + csel x2, x10, x2, eq + csel x3, x11, x3, eq + csel x4, x12, x4, eq + csel x5, x13, x5, eq + csel x6, x14, x6, eq + csel x7, x15, x7, eq + + add table, table, #64 + + sub x16, x16, #1 + cbnz x16, Lp256_scalarmulbase_alt_tabloop + +// Before storing back, optionally negate the y coordinate of the table entry + + stp x0, x1, [tabent] + stp x2, x3, [tabent+16] + + mov x0, 0xffffffffffffffff + subs x0, x0, x4 + mov x1, 0x00000000ffffffff + sbcs x1, x1, x5 + mov x3, 0xffffffff00000001 + sbcs x2, xzr, x6 + sbc x3, x3, x7 + + cmp cf, xzr + csel x4, x0, x4, ne + csel x5, x1, x5, ne + csel x6, x2, x6, ne + csel x7, x3, x7, ne + + stp x4, x5, [tabent+32] + stp x6, x7, [tabent+48] + +// Add the adjusted table point to the accumulator + + add x0, nacc + add x1, acc + add x2, tabent + CFI_BL(Lp256_scalarmulbase_alt_local_p256_montjmixadd) + +// However, only commit that update to the accumulator if j is nonzero, +// because the mixed addition function does not handle this case directly, +// and in any case we didn't choose the table entry appropriately. + + cmp j, xzr + ldp x0, x1, [acc] + ldp x12, x13, [nacc] + csel x0, x12, x0, ne + csel x1, x13, x1, ne + + ldp x2, x3, [acc+16] + ldp x12, x13, [nacc+16] + csel x2, x12, x2, ne + csel x3, x13, x3, ne + + ldp x4, x5, [acc+32] + ldp x12, x13, [nacc+32] + csel x4, x12, x4, ne + csel x5, x13, x5, ne + + ldp x6, x7, [acc+48] + ldp x12, x13, [nacc+48] + csel x6, x12, x6, ne + csel x7, x13, x7, ne + + ldp x8, x9, [acc+64] + ldp x12, x13, [nacc+64] + csel x8, x12, x8, ne + csel x9, x13, x9, ne + + ldp x10, x11, [acc+80] + ldp x12, x13, [nacc+80] + csel x10, x12, x10, ne + csel x11, x13, x11, ne + + stp x0, x1, [acc] + stp x2, x3, [acc+16] + stp x4, x5, [acc+32] + stp x6, x7, [acc+48] + stp x8, x9, [acc+64] + stp x10, x11, [acc+80] + +// Loop while blocksize * i <= 256 + + add i, i, #1 + mul x0, blocksize, i + cmp x0, #257 + bcc Lp256_scalarmulbase_alt_loop + +// That's the end of the main loop, and we just need to translate +// back from the Jacobian representation to affine. First of all, +// let z2 = 1/z^2 and z3 = 1/z^3, both without Montgomery form + + add x0, z2 + add x1, acc+64 + CFI_BL(Lp256_scalarmulbase_alt_local_montsqr_p256) + + add x0, z3 + add x1, acc+64 + add x2, z2 + CFI_BL(Lp256_scalarmulbase_alt_local_montmul_p256) + + add x0, z2 + add x1, z3 + CFI_BL(Lp256_scalarmulbase_alt_local_demont_p256) + + add x0, z3 + add x1, z2 + CFI_BL(Lp256_scalarmulbase_alt_local_inv_p256) + + add x0, z2 + add x1, acc+64 + add x2, z3 + CFI_BL(Lp256_scalarmulbase_alt_local_montmul_p256) + +// Convert back from Jacobian (X,Y,Z) |-> (X/Z^2, Y/Z^3) + + mov x0, res + add x1, acc + add x2, z2 + CFI_BL(Lp256_scalarmulbase_alt_local_montmul_p256) + + add x0, res, #32 + add x1, acc+32 + add x2, z3 + CFI_BL(Lp256_scalarmulbase_alt_local_montmul_p256) + +// Restore stack and registers and return + + CFI_INC_SP(NSPACE) + CFI_POP2(x25,x30) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_scalarmulbase_alt) + +// Local copies of subroutines, complete clones at the moment + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_alt_local_demont_p256) + +Lp256_scalarmulbase_alt_local_demont_p256: + CFI_START + ldp x2, x3, [x1] + ldp x4, x5, [x1, #16] + lsl x7, x2, #32 + subs x8, x2, x7 + lsr x6, x2, #32 + sbc x2, x2, x6 + adds x3, x3, x7 + adcs x4, x4, x6 + adcs x5, x5, x8 + adc x2, x2, xzr + lsl x7, x3, #32 + subs x8, x3, x7 + lsr x6, x3, #32 + sbc x3, x3, x6 + adds x4, x4, x7 + adcs x5, x5, x6 + adcs x2, x2, x8 + adc x3, x3, xzr + lsl x7, x4, #32 + subs x8, x4, x7 + lsr x6, x4, #32 + sbc x4, x4, x6 + adds x5, x5, x7 + adcs x2, x2, x6 + adcs x3, x3, x8 + adc x4, x4, xzr + lsl x7, x5, #32 + subs x8, x5, x7 + lsr x6, x5, #32 + sbc x5, x5, x6 + adds x2, x2, x7 + adcs x3, x3, x6 + adcs x4, x4, x8 + adc x5, x5, xzr + stp x2, x3, [x0] + stp x4, x5, [x0, #16] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_alt_local_demont_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_alt_local_inv_p256) + +Lp256_scalarmulbase_alt_local_inv_p256: + CFI_START + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_DEC_SP(160) + mov x20, x0 + mov x10, #0xffffffffffffffff + mov x11, #0xffffffff + mov x13, #0xffffffff00000001 + stp x10, x11, [sp] + stp xzr, x13, [sp, #16] + str xzr, [sp, #32] + ldp x2, x3, [x1] + subs x10, x2, x10 + sbcs x11, x3, x11 + ldp x4, x5, [x1, #16] + sbcs x12, x4, xzr + sbcs x13, x5, x13 + csel x2, x2, x10, cc + csel x3, x3, x11, cc + csel x4, x4, x12, cc + csel x5, x5, x13, cc + stp x2, x3, [sp, #48] + stp x4, x5, [sp, #64] + str xzr, [sp, #80] + stp xzr, xzr, [sp, #96] + stp xzr, xzr, [sp, #112] + mov x10, #0x4000000000000 + stp x10, xzr, [sp, #128] + stp xzr, xzr, [sp, #144] + mov x21, #0xa + mov x22, #0x1 + b Lp256_scalarmulbase_alt_inv_midloop +Lp256_scalarmulbase_alt_inv_loop: + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + and x0, x12, x16 + and x1, x13, x17 + add x19, x0, x1 + ldr x7, [sp] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #48] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x7, [sp, #8] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #56] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + adc x6, x6, x1 + extr x4, x2, x4, #59 + str x4, [sp] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + adc x4, x4, x1 + extr x5, x3, x5, #59 + str x5, [sp, #48] + ldr x7, [sp, #16] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #64] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + adc x5, x5, x1 + extr x2, x6, x2, #59 + str x2, [sp, #8] + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + adc x2, x2, x1 + extr x3, x4, x3, #59 + str x3, [sp, #56] + ldr x7, [sp, #24] + eor x1, x7, x14 + ldr x23, [sp, #32] + eor x3, x23, x14 + and x3, x3, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #72] + eor x1, x8, x15 + ldr x24, [sp, #80] + eor x0, x24, x15 + and x0, x0, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + extr x6, x5, x6, #59 + str x6, [sp, #16] + extr x5, x3, x5, #59 + str x5, [sp, #24] + asr x3, x3, #59 + str x3, [sp, #32] + eor x1, x7, x16 + eor x5, x23, x16 + and x5, x5, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + eor x0, x24, x17 + and x0, x0, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + extr x4, x2, x4, #59 + str x4, [sp, #64] + extr x2, x5, x2, #59 + str x2, [sp, #72] + asr x5, x5, #59 + str x5, [sp, #80] + ldr x7, [sp, #96] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #128] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #96] + adc x2, x2, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x5, x19, x0 + adc x3, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x5, x5, x0 + str x5, [sp, #128] + adc x3, x3, x1 + ldr x7, [sp, #104] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #136] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #104] + adc x6, x6, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x3, x3, x0 + adc x4, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x3, x3, x0 + str x3, [sp, #136] + adc x4, x4, x1 + ldr x7, [sp, #112] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #144] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #112] + adc x5, x5, x1 + eor x1, x7, x16 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x4, x4, x0 + adc x2, xzr, x1 + eor x1, x8, x17 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x4, x4, x0 + str x4, [sp, #144] + adc x2, x2, x1 + ldr x7, [sp, #120] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #152] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + ldp x0, x1, [sp, #96] + ldr x6, [sp, #112] + mov x14, #0xe000000000000000 + adds x0, x0, x14 + sbcs x1, x1, xzr + mov x11, #0x1fffffff + adcs x6, x6, x11 + mov x10, #0x2000000000000000 + adcs x5, x5, x10 + mov x14, #0x1fffffffe0000000 + adc x3, x3, x14 + lsl x11, x0, #32 + subs x14, x0, x11 + lsr x10, x0, #32 + sbc x0, x0, x10 + adds x1, x1, x11 + adcs x6, x6, x10 + adcs x5, x5, x14 + adcs x3, x3, x0 + mov x14, #0xffffffffffffffff + mov x11, #0xffffffff + mov x10, #0xffffffff00000001 + csel x14, x14, xzr, cs + csel x11, x11, xzr, cs + csel x10, x10, xzr, cs + subs x1, x1, x14 + sbcs x6, x6, x11 + sbcs x5, x5, xzr + sbc x3, x3, x10 + stp x1, x6, [sp, #96] + stp x5, x3, [sp, #112] + eor x1, x7, x16 + and x5, x16, x12 + neg x5, x5 + mul x0, x1, x12 + umulh x1, x1, x12 + adds x2, x2, x0 + adc x5, x5, x1 + eor x1, x8, x17 + and x0, x17, x13 + sub x5, x5, x0 + mul x0, x1, x13 + umulh x1, x1, x13 + adds x2, x2, x0 + adc x5, x5, x1 + ldp x0, x1, [sp, #128] + ldr x3, [sp, #144] + mov x14, #0xe000000000000000 + adds x0, x0, x14 + sbcs x1, x1, xzr + mov x11, #0x1fffffff + adcs x3, x3, x11 + mov x10, #0x2000000000000000 + adcs x2, x2, x10 + mov x14, #0x1fffffffe0000000 + adc x5, x5, x14 + lsl x11, x0, #32 + subs x14, x0, x11 + lsr x10, x0, #32 + sbc x0, x0, x10 + adds x1, x1, x11 + adcs x3, x3, x10 + adcs x2, x2, x14 + adcs x5, x5, x0 + mov x14, #0xffffffffffffffff + mov x11, #0xffffffff + mov x10, #0xffffffff00000001 + csel x14, x14, xzr, cs + csel x11, x11, xzr, cs + csel x10, x10, xzr, cs + subs x1, x1, x14 + sbcs x3, x3, x11 + sbcs x2, x2, xzr + sbc x5, x5, x10 + stp x1, x3, [sp, #128] + stp x2, x5, [sp, #144] +Lp256_scalarmulbase_alt_inv_midloop: + mov x1, x22 + ldr x2, [sp] + ldr x3, [sp, #48] + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x8, x4, #0x100, lsl #12 + sbfx x8, x8, #21, #21 + mov x11, #0x100000 + add x11, x11, x11, lsl #21 + add x9, x4, x11 + asr x9, x9, #42 + add x10, x5, #0x100, lsl #12 + sbfx x10, x10, #21, #21 + add x11, x5, x11 + asr x11, x11, #42 + mul x6, x8, x2 + mul x7, x9, x3 + mul x2, x10, x2 + mul x3, x11, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #21, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #42 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #21, #21 + add x15, x5, x15 + asr x15, x15, #42 + mul x6, x12, x2 + mul x7, x13, x3 + mul x2, x14, x2 + mul x3, x15, x3 + add x4, x6, x7 + add x5, x2, x3 + asr x2, x4, #20 + asr x3, x5, #20 + and x4, x2, #0xfffff + orr x4, x4, #0xfffffe0000000000 + and x5, x3, #0xfffff + orr x5, x5, #0xc000000000000000 + tst x5, #0x1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + mul x2, x12, x8 + mul x3, x12, x9 + mul x6, x14, x8 + mul x7, x14, x9 + madd x8, x13, x10, x2 + madd x9, x13, x11, x3 + madd x16, x15, x10, x6 + madd x17, x15, x11, x7 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + tst x5, #0x2 + asr x5, x5, #1 + csel x6, x4, xzr, ne + ccmp x1, xzr, #0x8, ne + cneg x1, x1, ge + cneg x6, x6, ge + csel x4, x5, x4, ge + add x5, x5, x6 + add x1, x1, #0x2 + asr x5, x5, #1 + add x12, x4, #0x100, lsl #12 + sbfx x12, x12, #22, #21 + mov x15, #0x100000 + add x15, x15, x15, lsl #21 + add x13, x4, x15 + asr x13, x13, #43 + add x14, x5, #0x100, lsl #12 + sbfx x14, x14, #22, #21 + add x15, x5, x15 + asr x15, x15, #43 + mneg x2, x12, x8 + mneg x3, x12, x9 + mneg x4, x14, x8 + mneg x5, x14, x9 + msub x10, x13, x16, x2 + msub x11, x13, x17, x3 + msub x12, x15, x16, x4 + msub x13, x15, x17, x5 + mov x22, x1 + subs x21, x21, #0x1 + bne Lp256_scalarmulbase_alt_inv_loop + ldr x0, [sp] + ldr x1, [sp, #48] + mul x0, x0, x10 + madd x1, x1, x11, x0 + asr x0, x1, #63 + cmp x10, xzr + csetm x14, mi + cneg x10, x10, mi + eor x14, x14, x0 + cmp x11, xzr + csetm x15, mi + cneg x11, x11, mi + eor x15, x15, x0 + cmp x12, xzr + csetm x16, mi + cneg x12, x12, mi + eor x16, x16, x0 + cmp x13, xzr + csetm x17, mi + cneg x13, x13, mi + eor x17, x17, x0 + and x0, x10, x14 + and x1, x11, x15 + add x9, x0, x1 + ldr x7, [sp, #96] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x4, x9, x0 + adc x2, xzr, x1 + ldr x8, [sp, #128] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x4, x4, x0 + str x4, [sp, #96] + adc x2, x2, x1 + ldr x7, [sp, #104] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x2, x2, x0 + adc x6, xzr, x1 + ldr x8, [sp, #136] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x2, x2, x0 + str x2, [sp, #104] + adc x6, x6, x1 + ldr x7, [sp, #112] + eor x1, x7, x14 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x6, x6, x0 + adc x5, xzr, x1 + ldr x8, [sp, #144] + eor x1, x8, x15 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x6, x6, x0 + str x6, [sp, #112] + adc x5, x5, x1 + ldr x7, [sp, #120] + eor x1, x7, x14 + and x3, x14, x10 + neg x3, x3 + mul x0, x1, x10 + umulh x1, x1, x10 + adds x5, x5, x0 + adc x3, x3, x1 + ldr x8, [sp, #152] + eor x1, x8, x15 + and x0, x15, x11 + sub x3, x3, x0 + mul x0, x1, x11 + umulh x1, x1, x11 + adds x5, x5, x0 + adc x3, x3, x1 + ldp x0, x1, [sp, #96] + ldr x2, [sp, #112] + mov x14, #0xe000000000000000 + adds x0, x0, x14 + sbcs x1, x1, xzr + mov x11, #0x1fffffff + adcs x2, x2, x11 + mov x10, #0x2000000000000000 + adcs x5, x5, x10 + mov x14, #0x1fffffffe0000000 + adc x3, x3, x14 + lsl x11, x0, #32 + subs x14, x0, x11 + lsr x10, x0, #32 + sbc x0, x0, x10 + adds x1, x1, x11 + adcs x2, x2, x10 + adcs x5, x5, x14 + adcs x3, x3, x0 + mov x14, #0xffffffffffffffff + mov x11, #0xffffffff + mov x10, #0xffffffff00000001 + csel x14, x14, xzr, cs + csel x11, x11, xzr, cs + csel x10, x10, xzr, cs + subs x1, x1, x14 + sbcs x2, x2, x11 + sbcs x5, x5, xzr + sbc x3, x3, x10 + mov x10, #0xffffffffffffffff + subs x10, x1, x10 + mov x11, #0xffffffff + sbcs x11, x2, x11 + mov x13, #0xffffffff00000001 + sbcs x12, x5, xzr + sbcs x13, x3, x13 + csel x10, x1, x10, cc + csel x11, x2, x11, cc + csel x12, x5, x12, cc + csel x13, x3, x13, cc + stp x10, x11, [x20] + stp x12, x13, [x20, #16] + CFI_INC_SP(160) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_alt_local_inv_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_alt_local_montmul_p256) + +Lp256_scalarmulbase_alt_local_montmul_p256: + CFI_START + ldp x3, x4, [x1] + ldp x7, x8, [x2] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x2, #16] + mul x11, x3, x9 + umulh x15, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x16, x3, x10 + adcs x15, x15, x11 + adc x16, x16, xzr + ldp x5, x6, [x1, #16] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x15, x15, x11 + mul x11, x4, x10 + adcs x16, x16, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x15, x15, x11 + umulh x11, x4, x9 + adcs x16, x16, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x15, x15, x11 + mul x11, x5, x9 + adcs x16, x16, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x15, x15, x11 + umulh x11, x5, x8 + adcs x16, x16, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x15, x15, x11 + mul x11, x6, x8 + adcs x16, x16, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x15, x15, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x16, x16, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x15, x15, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x15, x15, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x15, lsl #32 + lsr x11, x15, #32 + adcs x13, x13, x11 + mul x11, x15, x10 + umulh x15, x15, x10 + adcs x14, x14, x11 + adc x15, x15, xzr + adds x12, x12, x16 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x15, x15, x5 + cset x8, cs + mov x11, #0xffffffff + adds x16, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x15, x10 + sbcs xzr, x8, xzr + csel x12, x12, x16, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x15, x15, x5, cc + stp x12, x13, [x0] + stp x14, x15, [x0, #16] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_alt_local_montmul_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_alt_local_montsqr_p256) + +Lp256_scalarmulbase_alt_local_montsqr_p256: + CFI_START + ldp x2, x3, [x1] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [x1, #16] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + mov x5, #0xffffffff00000001 + adds x9, x9, x8, lsl #32 + lsr x2, x8, #32 + adcs x10, x10, x2 + mul x2, x8, x5 + umulh x8, x8, x5 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x2, x9, #32 + adcs x11, x11, x2 + mul x2, x9, x5 + umulh x9, x9, x5 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x2, x10, #32 + adcs x8, x8, x2 + mul x2, x10, x5 + umulh x10, x10, x5 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x2, x11, #32 + adcs x9, x9, x2 + mul x2, x11, x5 + umulh x11, x11, x5 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + cset x2, cs + mov x3, #0xffffffff + adds x12, x8, #0x1 + sbcs x13, x9, x3 + sbcs x14, x10, xzr + sbcs x7, x11, x5 + sbcs xzr, x2, xzr + csel x8, x8, x12, cc + csel x9, x9, x13, cc + csel x10, x10, x14, cc + csel x11, x11, x7, cc + stp x8, x9, [x0] + stp x10, x11, [x0, #16] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_alt_local_montsqr_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_alt_local_p256_montjmixadd) + +Lp256_scalarmulbase_alt_local_p256_montjmixadd: + CFI_START + CFI_DEC_SP(192) + mov x15, x0 + mov x16, x1 + mov x17, x2 + ldp x2, x3, [x16, #64] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [x16, #80] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + adds x9, x9, x8, lsl #32 + lsr x3, x8, #32 + adcs x10, x10, x3 + mov x3, #0xffffffff00000001 + mul x2, x8, x3 + umulh x8, x8, x3 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x3, x9, #32 + adcs x11, x11, x3 + mov x3, #0xffffffff00000001 + mul x2, x9, x3 + umulh x9, x9, x3 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x3, x10, #32 + adcs x8, x8, x3 + mov x3, #0xffffffff00000001 + mul x2, x10, x3 + umulh x10, x10, x3 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x3, x11, #32 + adcs x9, x9, x3 + mov x3, #0xffffffff00000001 + mul x2, x11, x3 + umulh x11, x11, x3 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + mov x2, #0xffffffffffffffff + csel x2, xzr, x2, cc + mov x3, #0xffffffff + csel x3, xzr, x3, cc + mov x5, #0xffffffff00000001 + csel x5, xzr, x5, cc + subs x8, x8, x2 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbc x11, x11, x5 + stp x8, x9, [sp] + stp x10, x11, [sp, #16] + ldp x3, x4, [x16, #64] + ldp x7, x8, [x17, #32] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x17, #48] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [x16, #80] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #32] + stp x14, x0, [sp, #48] + ldp x3, x4, [sp] + ldp x7, x8, [x17] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x17, #16] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #16] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #64] + stp x14, x0, [sp, #80] + ldp x3, x4, [sp] + ldp x7, x8, [sp, #32] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [sp, #48] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #16] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #32] + stp x14, x0, [sp, #48] + ldp x5, x6, [sp, #64] + ldp x4, x3, [x16] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [x16, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #160] + stp x7, x8, [sp, #176] + ldp x5, x6, [sp, #32] + ldp x4, x3, [x16, #32] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #48] + ldp x4, x3, [x16, #48] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #32] + stp x7, x8, [sp, #48] + ldp x2, x3, [sp, #160] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [sp, #176] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + adds x9, x9, x8, lsl #32 + lsr x3, x8, #32 + adcs x10, x10, x3 + mov x3, #0xffffffff00000001 + mul x2, x8, x3 + umulh x8, x8, x3 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x3, x9, #32 + adcs x11, x11, x3 + mov x3, #0xffffffff00000001 + mul x2, x9, x3 + umulh x9, x9, x3 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x3, x10, #32 + adcs x8, x8, x3 + mov x3, #0xffffffff00000001 + mul x2, x10, x3 + umulh x10, x10, x3 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x3, x11, #32 + adcs x9, x9, x3 + mov x3, #0xffffffff00000001 + mul x2, x11, x3 + umulh x11, x11, x3 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + mov x2, #0xffffffffffffffff + csel x2, xzr, x2, cc + mov x3, #0xffffffff + csel x3, xzr, x3, cc + mov x5, #0xffffffff00000001 + csel x5, xzr, x5, cc + subs x8, x8, x2 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbc x11, x11, x5 + stp x8, x9, [sp, #96] + stp x10, x11, [sp, #112] + ldp x2, x3, [sp, #32] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [sp, #48] + mul x11, x2, x5 + umulh x12, x2, x5 + mul x6, x2, x4 + umulh x7, x2, x4 + adds x10, x10, x6 + adcs x11, x11, x7 + mul x6, x3, x4 + umulh x7, x3, x4 + adc x7, x7, xzr + adds x11, x11, x6 + mul x13, x4, x5 + umulh x14, x4, x5 + adcs x12, x12, x7 + mul x6, x3, x5 + umulh x7, x3, x5 + adc x7, x7, xzr + adds x12, x12, x6 + adcs x13, x13, x7 + adc x14, x14, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + cset x7, cs + umulh x6, x2, x2 + mul x8, x2, x2 + adds x9, x9, x6 + mul x6, x3, x3 + adcs x10, x10, x6 + umulh x6, x3, x3 + adcs x11, x11, x6 + mul x6, x4, x4 + adcs x12, x12, x6 + umulh x6, x4, x4 + adcs x13, x13, x6 + mul x6, x5, x5 + adcs x14, x14, x6 + umulh x6, x5, x5 + adc x7, x7, x6 + adds x9, x9, x8, lsl #32 + lsr x3, x8, #32 + adcs x10, x10, x3 + mov x3, #0xffffffff00000001 + mul x2, x8, x3 + umulh x8, x8, x3 + adcs x11, x11, x2 + adc x8, x8, xzr + adds x10, x10, x9, lsl #32 + lsr x3, x9, #32 + adcs x11, x11, x3 + mov x3, #0xffffffff00000001 + mul x2, x9, x3 + umulh x9, x9, x3 + adcs x8, x8, x2 + adc x9, x9, xzr + adds x11, x11, x10, lsl #32 + lsr x3, x10, #32 + adcs x8, x8, x3 + mov x3, #0xffffffff00000001 + mul x2, x10, x3 + umulh x10, x10, x3 + adcs x9, x9, x2 + adc x10, x10, xzr + adds x8, x8, x11, lsl #32 + lsr x3, x11, #32 + adcs x9, x9, x3 + mov x3, #0xffffffff00000001 + mul x2, x11, x3 + umulh x11, x11, x3 + adcs x10, x10, x2 + adc x11, x11, xzr + adds x8, x8, x12 + adcs x9, x9, x13 + adcs x10, x10, x14 + adcs x11, x11, x7 + cset x2, cs + mov x3, #0xffffffff + mov x5, #0xffffffff00000001 + adds x12, x8, #0x1 + sbcs x13, x9, x3 + sbcs x14, x10, xzr + sbcs x7, x11, x5 + sbcs xzr, x2, xzr + csel x8, x8, x12, cc + csel x9, x9, x13, cc + csel x10, x10, x14, cc + csel x11, x11, x7, cc + stp x8, x9, [sp] + stp x10, x11, [sp, #16] + ldp x3, x4, [sp, #96] + ldp x7, x8, [x16] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x16, #16] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #112] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #128] + stp x14, x0, [sp, #144] + ldp x3, x4, [sp, #96] + ldp x7, x8, [sp, #64] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [sp, #80] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #112] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #64] + stp x14, x0, [sp, #80] + ldp x5, x6, [sp] + ldp x4, x3, [sp, #128] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #16] + ldp x4, x3, [sp, #144] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp] + stp x7, x8, [sp, #16] + ldp x5, x6, [sp, #64] + ldp x4, x3, [sp, #128] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #80] + ldp x4, x3, [sp, #144] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #96] + stp x7, x8, [sp, #112] + ldp x3, x4, [sp, #160] + ldp x7, x8, [x16, #64] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x16, #80] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #176] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #160] + stp x14, x0, [sp, #176] + ldp x5, x6, [sp] + ldp x4, x3, [sp, #64] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #16] + ldp x4, x3, [sp, #80] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp] + stp x7, x8, [sp, #16] + ldp x5, x6, [sp, #128] + ldp x4, x3, [sp] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #144] + ldp x4, x3, [sp, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #128] + stp x7, x8, [sp, #144] + ldp x3, x4, [sp, #96] + ldp x7, x8, [x16, #32] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [x16, #48] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #112] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #96] + stp x14, x0, [sp, #112] + ldp x3, x4, [sp, #32] + ldp x7, x8, [sp, #128] + mul x12, x3, x7 + umulh x13, x3, x7 + mul x11, x3, x8 + umulh x14, x3, x8 + adds x13, x13, x11 + ldp x9, x10, [sp, #144] + mul x11, x3, x9 + umulh x0, x3, x9 + adcs x14, x14, x11 + mul x11, x3, x10 + umulh x1, x3, x10 + adcs x0, x0, x11 + adc x1, x1, xzr + ldp x5, x6, [sp, #48] + mul x11, x4, x7 + adds x13, x13, x11 + mul x11, x4, x8 + adcs x14, x14, x11 + mul x11, x4, x9 + adcs x0, x0, x11 + mul x11, x4, x10 + adcs x1, x1, x11 + umulh x3, x4, x10 + adc x3, x3, xzr + umulh x11, x4, x7 + adds x14, x14, x11 + umulh x11, x4, x8 + adcs x0, x0, x11 + umulh x11, x4, x9 + adcs x1, x1, x11 + adc x3, x3, xzr + mul x11, x5, x7 + adds x14, x14, x11 + mul x11, x5, x8 + adcs x0, x0, x11 + mul x11, x5, x9 + adcs x1, x1, x11 + mul x11, x5, x10 + adcs x3, x3, x11 + umulh x4, x5, x10 + adc x4, x4, xzr + umulh x11, x5, x7 + adds x0, x0, x11 + umulh x11, x5, x8 + adcs x1, x1, x11 + umulh x11, x5, x9 + adcs x3, x3, x11 + adc x4, x4, xzr + mul x11, x6, x7 + adds x0, x0, x11 + mul x11, x6, x8 + adcs x1, x1, x11 + mul x11, x6, x9 + adcs x3, x3, x11 + mul x11, x6, x10 + adcs x4, x4, x11 + umulh x5, x6, x10 + adc x5, x5, xzr + mov x10, #0xffffffff00000001 + adds x13, x13, x12, lsl #32 + lsr x11, x12, #32 + adcs x14, x14, x11 + mul x11, x12, x10 + umulh x12, x12, x10 + adcs x0, x0, x11 + adc x12, x12, xzr + umulh x11, x6, x7 + adds x1, x1, x11 + umulh x11, x6, x8 + adcs x3, x3, x11 + umulh x11, x6, x9 + adcs x4, x4, x11 + adc x5, x5, xzr + adds x14, x14, x13, lsl #32 + lsr x11, x13, #32 + adcs x0, x0, x11 + mul x11, x13, x10 + umulh x13, x13, x10 + adcs x12, x12, x11 + adc x13, x13, xzr + adds x0, x0, x14, lsl #32 + lsr x11, x14, #32 + adcs x12, x12, x11 + mul x11, x14, x10 + umulh x14, x14, x10 + adcs x13, x13, x11 + adc x14, x14, xzr + adds x12, x12, x0, lsl #32 + lsr x11, x0, #32 + adcs x13, x13, x11 + mul x11, x0, x10 + umulh x0, x0, x10 + adcs x14, x14, x11 + adc x0, x0, xzr + adds x12, x12, x1 + adcs x13, x13, x3 + adcs x14, x14, x4 + adcs x0, x0, x5 + cset x8, cs + mov x11, #0xffffffff + adds x1, x12, #0x1 + sbcs x3, x13, x11 + sbcs x4, x14, xzr + sbcs x5, x0, x10 + sbcs xzr, x8, xzr + csel x12, x12, x1, cc + csel x13, x13, x3, cc + csel x14, x14, x4, cc + csel x0, x0, x5, cc + stp x12, x13, [sp, #128] + stp x14, x0, [sp, #144] + ldp x5, x6, [sp, #128] + ldp x4, x3, [sp, #96] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #144] + ldp x4, x3, [sp, #112] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + csetm x3, cc + adds x5, x5, x3 + mov x4, #0xffffffff + and x4, x4, x3 + adcs x6, x6, x4 + adcs x7, x7, xzr + mov x4, #0xffffffff00000001 + and x4, x4, x3 + adc x8, x8, x4 + stp x5, x6, [sp, #128] + stp x7, x8, [sp, #144] + ldp x0, x1, [x16, #64] + ldp x2, x3, [x16, #80] + orr x4, x0, x1 + orr x5, x2, x3 + orr x4, x4, x5 + cmp x4, xzr + ldp x0, x1, [sp] + ldp x12, x13, [x17] + csel x0, x0, x12, ne + csel x1, x1, x13, ne + ldp x2, x3, [sp, #16] + ldp x12, x13, [x17, #16] + csel x2, x2, x12, ne + csel x3, x3, x13, ne + ldp x4, x5, [sp, #128] + ldp x12, x13, [x17, #32] + csel x4, x4, x12, ne + csel x5, x5, x13, ne + ldp x6, x7, [sp, #144] + ldp x12, x13, [x17, #48] + csel x6, x6, x12, ne + csel x7, x7, x13, ne + ldp x8, x9, [sp, #160] + mov x12, #0x1 + mov x13, #0xffffffff00000000 + csel x8, x8, x12, ne + csel x9, x9, x13, ne + ldp x10, x11, [sp, #176] + mov x12, #0xffffffffffffffff + mov x13, #0xfffffffe + csel x10, x10, x12, ne + csel x11, x11, x13, ne + stp x0, x1, [x15] + stp x2, x3, [x15, #16] + stp x4, x5, [x15, #32] + stp x6, x7, [x15, #48] + stp x8, x9, [x15, #64] + stp x10, x11, [x15, #80] + CFI_INC_SP(192) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_alt_local_p256_montjmixadd) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/p384_montjscalarmul.S b/cbits/s2n/arm/p384_montjscalarmul.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/p384_montjscalarmul.S @@ -0,0 +1,10004 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Montgomery-Jacobian form scalar multiplication for P-384 +// Input scalar[6], point[18]; output res[18] +// +// extern void p384_montjscalarmul +// (uint64_t res[static 18], +// const uint64_t scalar[static 6], +// const uint64_t point[static 18]); +// +// This function is a variant of its affine point version p384_scalarmul. +// Here, input and output points are assumed to be in Jacobian form with +// their coordinates in the Montgomery domain. Thus, if priming indicates +// Montgomery form, x' = (2^384 * x) mod p_384 etc., each point argument +// is a triple (x',y',z') representing the affine point (x/z^2,y/z^3) when +// z' is nonzero or the point at infinity (group identity) if z' = 0. +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-384, returns a representation of n * P. If the result is the +// point at infinity (either because the input point was or because the +// scalar was a multiple of p_384) then the output is guaranteed to +// represent the point at infinity, i.e. to have its z coordinate zero. +// +// Standard ARM ABI: X0 = res, X1 = scalar, X2 = point +// ---------------------------------------------------------------------------- + + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p384_montjscalarmul) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p384_montjscalarmul) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p384_montjscalarmul) + + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 48 +#define JACSIZE (3*NUMSIZE) + +// Safe copies of input res and additional values in variables. + +#define bf x22 +#define sgn x23 +#define j x24 +#define res x25 + +// Intermediate variables on the stack. +// The table is 16 entries, each of size JACSIZE = 3 * NUMSIZE + +#define scalarb sp, #(0*NUMSIZE) +#define acc sp, #(1*NUMSIZE) +#define tabent sp, #(4*NUMSIZE) + +#define tab sp, #(7*NUMSIZE) + +#define NSPACE 55*NUMSIZE + +// Avoid using .rep for the sake of the BoringSSL/AWS-LC delocator, +// which doesn't accept repetitions, assembler macros etc. + +#define selectblock(I) \ + cmp bf, #(1*I) __LF \ + ldp x20, x21, [x19] __LF \ + csel x0, x20, x0, eq __LF \ + csel x1, x21, x1, eq __LF \ + ldp x20, x21, [x19, #16] __LF \ + csel x2, x20, x2, eq __LF \ + csel x3, x21, x3, eq __LF \ + ldp x20, x21, [x19, #32] __LF \ + csel x4, x20, x4, eq __LF \ + csel x5, x21, x5, eq __LF \ + ldp x20, x21, [x19, #48] __LF \ + csel x6, x20, x6, eq __LF \ + csel x7, x21, x7, eq __LF \ + ldp x20, x21, [x19, #64] __LF \ + csel x8, x20, x8, eq __LF \ + csel x9, x21, x9, eq __LF \ + ldp x20, x21, [x19, #80] __LF \ + csel x10, x20, x10, eq __LF \ + csel x11, x21, x11, eq __LF \ + ldp x20, x21, [x19, #96] __LF \ + csel x12, x20, x12, eq __LF \ + csel x13, x21, x13, eq __LF \ + ldp x20, x21, [x19, #112] __LF \ + csel x14, x20, x14, eq __LF \ + csel x15, x21, x15, eq __LF \ + ldp x20, x21, [x19, #128] __LF \ + csel x16, x20, x16, eq __LF \ + csel x17, x21, x17, eq __LF \ + add x19, x19, #JACSIZE + +// Loading large constants + +#define movbig(nn,n3,n2,n1,n0) \ + movz nn, n0 __LF \ + movk nn, n1, lsl #16 __LF \ + movk nn, n2, lsl #32 __LF \ + movk nn, n3, lsl #48 + +S2N_BN_SYMBOL(p384_montjscalarmul): + CFI_START + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x30) + CFI_DEC_SP(NSPACE) + +// Preserve the "res" input argument; others get processed early. + + mov res, x0 + +// Reduce the input scalar mod n_384, i.e. conditionally subtract n_384. +// Store it to "scalarb". + + ldp x3, x4, [x1] + movbig(x15, #0xecec, #0x196a, #0xccc5, #0x2973) + ldp x5, x6, [x1, #16] + movbig(x16, #0x581a, #0x0db2, #0x48b0, #0xa77a) + ldp x7, x8, [x1, #32] + movbig(x17, #0xc763, #0x4d81, #0xf437, #0x2ddf) + + subs x9, x3, x15 + sbcs x10, x4, x16 + sbcs x11, x5, x17 + adcs x12, x6, xzr + adcs x13, x7, xzr + adcs x14, x8, xzr + + csel x3, x3, x9, cc + csel x4, x4, x10, cc + csel x5, x5, x11, cc + csel x6, x6, x12, cc + csel x7, x7, x13, cc + csel x8, x8, x14, cc + + stp x3, x4, [scalarb] + stp x5, x6, [scalarb+16] + stp x7, x8, [scalarb+32] + +// Set the tab[0] table entry to the input point = 1 * P + + ldp x10, x11, [x2] + stp x10, x11, [tab] + ldp x12, x13, [x2, #16] + stp x12, x13, [tab+16] + ldp x14, x15, [x2, #32] + stp x14, x15, [tab+32] + + ldp x10, x11, [x2, #48] + stp x10, x11, [tab+48] + ldp x12, x13, [x2, #64] + stp x12, x13, [tab+64] + ldp x14, x15, [x2, #80] + stp x14, x15, [tab+80] + + ldp x10, x11, [x2, #96] + stp x10, x11, [tab+96] + ldp x12, x13, [x2, #112] + stp x12, x13, [tab+112] + ldp x14, x15, [x2, #128] + stp x14, x15, [tab+128] + +// Compute and record tab[1] = 2 * p, ..., tab[15] = 16 * P + + add x0, tab+JACSIZE*1 + add x1, tab + CFI_BL(Lp384_montjscalarmul_p384_montjdouble) + + add x0, tab+JACSIZE*2 + add x1, tab+JACSIZE*1 + add x2, tab + CFI_BL(Lp384_montjscalarmul_p384_montjadd) + + add x0, tab+JACSIZE*3 + add x1, tab+JACSIZE*1 + CFI_BL(Lp384_montjscalarmul_p384_montjdouble) + + add x0, tab+JACSIZE*4 + add x1, tab+JACSIZE*3 + add x2, tab + CFI_BL(Lp384_montjscalarmul_p384_montjadd) + + add x0, tab+JACSIZE*5 + add x1, tab+JACSIZE*2 + CFI_BL(Lp384_montjscalarmul_p384_montjdouble) + + add x0, tab+JACSIZE*6 + add x1, tab+JACSIZE*5 + add x2, tab + CFI_BL(Lp384_montjscalarmul_p384_montjadd) + + add x0, tab+JACSIZE*7 + add x1, tab+JACSIZE*3 + CFI_BL(Lp384_montjscalarmul_p384_montjdouble) + + add x0, tab+JACSIZE*8 + add x1, tab+JACSIZE*7 + add x2, tab + CFI_BL(Lp384_montjscalarmul_p384_montjadd) + + add x0, tab+JACSIZE*9 + add x1, tab+JACSIZE*4 + CFI_BL(Lp384_montjscalarmul_p384_montjdouble) + + add x0, tab+JACSIZE*10 + add x1, tab+JACSIZE*9 + add x2, tab + CFI_BL(Lp384_montjscalarmul_p384_montjadd) + + add x0, tab+JACSIZE*11 + add x1, tab+JACSIZE*5 + CFI_BL(Lp384_montjscalarmul_p384_montjdouble) + + add x0, tab+JACSIZE*12 + add x1, tab+JACSIZE*11 + add x2, tab + CFI_BL(Lp384_montjscalarmul_p384_montjadd) + + add x0, tab+JACSIZE*13 + add x1, tab+JACSIZE*6 + CFI_BL(Lp384_montjscalarmul_p384_montjdouble) + + add x0, tab+JACSIZE*14 + add x1, tab+JACSIZE*13 + add x2, tab + CFI_BL(Lp384_montjscalarmul_p384_montjadd) + + add x0, tab+JACSIZE*15 + add x1, tab+JACSIZE*7 + CFI_BL(Lp384_montjscalarmul_p384_montjdouble) + +// Add the recoding constant sum_i(16 * 32^i) to the scalar to allow signed +// digits. The digits of the constant, in lowest-to-highest order, are as +// follows; they are generated dynamically since none is a simple ARM load. +// +// 0x0842108421084210 +// 0x1084210842108421 +// 0x2108421084210842 +// 0x4210842108421084 +// 0x8421084210842108 +// 0x0842108421084210 + + ldp x0, x1, [scalarb] + ldp x2, x3, [scalarb+16] + ldp x4, x5, [scalarb+32] + movbig(x8, #0x1084, #0x2108, #0x4210, #0x8421) + adds x0, x0, x8, lsr #1 + adcs x1, x1, x8 + lsl x8, x8, #1 + adcs x2, x2, x8 + lsl x8, x8, #1 + adcs x3, x3, x8 + lsl x8, x8, #1 + adcs x4, x4, x8 + lsr x8, x8, #4 + adcs x5, x5, x8 + cset x6, cs + +// Record the top bitfield then shift the whole scalar left 4 bits +// to align the top of the next bitfield with the MSB (bits 379..383). + + extr bf, x6, x5, #60 + extr x5, x5, x4, #60 + extr x4, x4, x3, #60 + extr x3, x3, x2, #60 + extr x2, x2, x1, #60 + extr x1, x1, x0, #60 + lsl x0, x0, #4 + stp x0, x1, [scalarb] + stp x2, x3, [scalarb+16] + stp x4, x5, [scalarb+32] + +// Initialize the accumulator to the corresponding entry using constant-time +// lookup in the table. This top digit, uniquely, is not recoded so there is +// no sign adjustment to make. + + mov x0, xzr + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, xzr + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + mov x9, xzr + mov x10, xzr + mov x11, xzr + mov x12, xzr + mov x13, xzr + mov x14, xzr + mov x15, xzr + mov x16, xzr + mov x17, xzr + + add x19, tab + + selectblock(1) + selectblock(2) + selectblock(3) + selectblock(4) + selectblock(5) + selectblock(6) + selectblock(7) + selectblock(8) + selectblock(9) + selectblock(10) + selectblock(11) + selectblock(12) + selectblock(13) + selectblock(14) + selectblock(15) + selectblock(16) + + stp x0, x1, [acc] + stp x2, x3, [acc+16] + stp x4, x5, [acc+32] + stp x6, x7, [acc+48] + stp x8, x9, [acc+64] + stp x10, x11, [acc+80] + stp x12, x13, [acc+96] + stp x14, x15, [acc+112] + stp x16, x17, [acc+128] + + mov j, #380 + +// Main loop over size-5 bitfields: double 5 times then add signed digit +// At each stage we shift the scalar left by 5 bits so we can simply pick +// the top 5 bits as the bitfield, saving some fiddle over indexing. + +Lp384_montjscalarmul_mainloop: + sub j, j, #5 + + add x0, acc + add x1, acc + CFI_BL(Lp384_montjscalarmul_p384_montjdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp384_montjscalarmul_p384_montjdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp384_montjscalarmul_p384_montjdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp384_montjscalarmul_p384_montjdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp384_montjscalarmul_p384_montjdouble) + +// Choose the bitfield and adjust it to sign and magnitude + + ldp x0, x1, [scalarb] + ldp x2, x3, [scalarb+16] + ldp x4, x5, [scalarb+32] + lsr bf, x5, #59 + extr x5, x5, x4, #59 + extr x4, x4, x3, #59 + extr x3, x3, x2, #59 + extr x2, x2, x1, #59 + extr x1, x1, x0, #59 + lsl x0, x0, #5 + stp x0, x1, [scalarb] + stp x2, x3, [scalarb+16] + stp x4, x5, [scalarb+32] + + subs bf, bf, #16 + cset sgn, lo // sgn = sign of digit (1 = negative) + cneg bf, bf, lo // bf = absolute value of digit + +// Conditionally select the table entry tab[i-1] = i * P in constant time + + mov x0, xzr + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, xzr + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + mov x9, xzr + mov x10, xzr + mov x11, xzr + mov x12, xzr + mov x13, xzr + mov x14, xzr + mov x15, xzr + mov x16, xzr + mov x17, xzr + + add x19, tab + + selectblock(1) + selectblock(2) + selectblock(3) + selectblock(4) + selectblock(5) + selectblock(6) + selectblock(7) + selectblock(8) + selectblock(9) + selectblock(10) + selectblock(11) + selectblock(12) + selectblock(13) + selectblock(14) + selectblock(15) + selectblock(16) + +// Store it to "tabent" with the y coordinate optionally negated. +// This is done carefully to give coordinates < p_384 even in +// the degenerate case y = 0 (when z = 0 for points on the curve). + + stp x0, x1, [tabent] + stp x2, x3, [tabent+16] + stp x4, x5, [tabent+32] + + stp x12, x13, [tabent+96] + stp x14, x15, [tabent+112] + stp x16, x17, [tabent+128] + + mov x0, #0x00000000ffffffff + subs x0, x0, x6 + orr x12, x6, x7 + mov x1, #0xffffffff00000000 + sbcs x1, x1, x7 + orr x13, x8, x9 + mov x2, #0xfffffffffffffffe + sbcs x2, x2, x8 + orr x14, x10, x11 + mov x5, #0xffffffffffffffff + sbcs x3, x5, x9 + orr x12, x12, x13 + sbcs x4, x5, x10 + orr x12, x12, x14 + sbcs x5, x5, x11 + + cmp sgn, xzr + ccmp x12, xzr, #4, ne + + csel x6, x0, x6, ne + csel x7, x1, x7, ne + csel x8, x2, x8, ne + csel x9, x3, x9, ne + csel x10, x4, x10, ne + csel x11, x5, x11, ne + + stp x6, x7, [tabent+48] + stp x8, x9, [tabent+64] + stp x10, x11, [tabent+80] + +// Add to the accumulator + + add x0, acc + add x1, acc + add x2, tabent + CFI_BL(Lp384_montjscalarmul_p384_montjadd) + + cbnz j, Lp384_montjscalarmul_mainloop + +// That's the end of the main loop, and we just need to copy the +// result in "acc" to the output. + + ldp x0, x1, [acc] + stp x0, x1, [res] + ldp x0, x1, [acc+16] + stp x0, x1, [res, #16] + ldp x0, x1, [acc+32] + stp x0, x1, [res, #32] + ldp x0, x1, [acc+48] + stp x0, x1, [res, #48] + ldp x0, x1, [acc+64] + stp x0, x1, [res, #64] + ldp x0, x1, [acc+80] + stp x0, x1, [res, #80] + ldp x0, x1, [acc+96] + stp x0, x1, [res, #96] + ldp x0, x1, [acc+112] + stp x0, x1, [res, #112] + ldp x0, x1, [acc+128] + stp x0, x1, [res, #128] + +// Restore stack and registers and return + + CFI_INC_SP(NSPACE) + CFI_POP2(x25,x30) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p384_montjscalarmul) + +// Local copies of subroutines, complete clones at the moment + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp384_montjscalarmul_p384_montjadd) + +Lp384_montjscalarmul_p384_montjadd: + CFI_START + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x26) + CFI_PUSH1Z(x27) + CFI_DEC_SP(384) + mov x24, x0 + mov x25, x1 + mov x26, x2 + mov x0, sp + ldr q1, [x25, #96] + ldp x9, x2, [x25, #96] + ldr q0, [x25, #96] + ldp x4, x6, [x25, #112] + rev64 v21.4s, v1.4s + uzp2 v28.4s, v1.4s, v1.4s + umulh x7, x9, x2 + xtn v17.2s, v1.2d + mul v27.4s, v21.4s, v0.4s + ldr q20, [x25, #128] + xtn v30.2s, v0.2d + ldr q1, [x25, #128] + uzp2 v31.4s, v0.4s, v0.4s + ldp x5, x10, [x25, #128] + umulh x8, x9, x4 + uaddlp v3.2d, v27.4s + umull v16.2d, v30.2s, v17.2s + mul x16, x9, x4 + umull v27.2d, v30.2s, v28.2s + shrn v0.2s, v20.2d, #32 + xtn v7.2s, v20.2d + shl v20.2d, v3.2d, #32 + umull v3.2d, v31.2s, v28.2s + mul x3, x2, x4 + umlal v20.2d, v30.2s, v17.2s + umull v22.2d, v7.2s, v0.2s + usra v27.2d, v16.2d, #32 + umulh x11, x2, x4 + movi v21.2d, #0xffffffff + uzp2 v28.4s, v1.4s, v1.4s + adds x15, x16, x7 + and v5.16b, v27.16b, v21.16b + adcs x3, x3, x8 + usra v3.2d, v27.2d, #32 + dup v29.2d, x6 + adcs x16, x11, xzr + mov x14, v20.d[0] + umlal v5.2d, v31.2s, v17.2s + mul x8, x9, x2 + mov x7, v20.d[1] + shl v19.2d, v22.2d, #33 + xtn v25.2s, v29.2d + rev64 v31.4s, v1.4s + lsl x13, x14, #32 + uzp2 v6.4s, v29.4s, v29.4s + umlal v19.2d, v7.2s, v7.2s + usra v3.2d, v5.2d, #32 + adds x1, x8, x8 + umulh x8, x4, x4 + add x12, x13, x14 + mul v17.4s, v31.4s, v29.4s + xtn v4.2s, v1.2d + adcs x14, x15, x15 + lsr x13, x12, #32 + adcs x15, x3, x3 + umull v31.2d, v25.2s, v28.2s + adcs x11, x16, x16 + umull v21.2d, v25.2s, v4.2s + mov x17, v3.d[0] + umull v18.2d, v6.2s, v28.2s + adc x16, x8, xzr + uaddlp v16.2d, v17.4s + movi v1.2d, #0xffffffff + subs x13, x13, x12 + usra v31.2d, v21.2d, #32 + sbc x8, x12, xzr + adds x17, x17, x1 + mul x1, x4, x4 + shl v28.2d, v16.2d, #32 + mov x3, v3.d[1] + adcs x14, x7, x14 + extr x7, x8, x13, #32 + adcs x13, x3, x15 + and v3.16b, v31.16b, v1.16b + adcs x11, x1, x11 + lsr x1, x8, #32 + umlal v3.2d, v6.2s, v4.2s + usra v18.2d, v31.2d, #32 + adc x3, x16, xzr + adds x1, x1, x12 + umlal v28.2d, v25.2s, v4.2s + adc x16, xzr, xzr + subs x15, x17, x7 + sbcs x7, x14, x1 + lsl x1, x15, #32 + sbcs x16, x13, x16 + add x8, x1, x15 + usra v18.2d, v3.2d, #32 + sbcs x14, x11, xzr + lsr x1, x8, #32 + sbcs x17, x3, xzr + sbc x11, x12, xzr + subs x13, x1, x8 + umulh x12, x4, x10 + sbc x1, x8, xzr + extr x13, x1, x13, #32 + lsr x1, x1, #32 + adds x15, x1, x8 + adc x1, xzr, xzr + subs x7, x7, x13 + sbcs x13, x16, x15 + lsl x3, x7, #32 + umulh x16, x2, x5 + sbcs x15, x14, x1 + add x7, x3, x7 + sbcs x3, x17, xzr + lsr x1, x7, #32 + sbcs x14, x11, xzr + sbc x11, x8, xzr + subs x8, x1, x7 + sbc x1, x7, xzr + extr x8, x1, x8, #32 + lsr x1, x1, #32 + adds x1, x1, x7 + adc x17, xzr, xzr + subs x13, x13, x8 + umulh x8, x9, x6 + sbcs x1, x15, x1 + sbcs x15, x3, x17 + sbcs x3, x14, xzr + mul x17, x2, x5 + sbcs x11, x11, xzr + stp x13, x1, [x0] + sbc x14, x7, xzr + mul x7, x4, x10 + subs x1, x9, x2 + stp x15, x3, [x0, #16] + csetm x15, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + stp x11, x14, [x0, #32] + mul x14, x9, x6 + adds x17, x8, x17 + adcs x7, x16, x7 + adc x13, x12, xzr + subs x12, x5, x6 + cneg x3, x12, cc // cc = lo, ul, last + cinv x16, x15, cc // cc = lo, ul, last + mul x8, x1, x3 + umulh x1, x1, x3 + eor x12, x8, x16 + adds x11, x17, x14 + adcs x3, x7, x17 + adcs x15, x13, x7 + adc x8, x13, xzr + adds x3, x3, x14 + adcs x15, x15, x17 + adcs x17, x8, x7 + eor x1, x1, x16 + adc x13, x13, xzr + subs x9, x9, x4 + csetm x8, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x4, x2, x4 + cneg x4, x4, cc // cc = lo, ul, last + csetm x7, cc // cc = lo, ul, last + subs x2, x10, x6 + cinv x8, x8, cc // cc = lo, ul, last + cneg x2, x2, cc // cc = lo, ul, last + cmn x16, #0x1 + adcs x11, x11, x12 + mul x12, x9, x2 + adcs x3, x3, x1 + adcs x15, x15, x16 + umulh x9, x9, x2 + adcs x17, x17, x16 + adc x13, x13, x16 + subs x1, x10, x5 + cinv x2, x7, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + eor x9, x9, x8 + cmn x8, #0x1 + eor x7, x12, x8 + mul x12, x4, x1 + adcs x3, x3, x7 + adcs x7, x15, x9 + adcs x15, x17, x8 + ldp x9, x17, [x0, #16] + umulh x4, x4, x1 + adc x8, x13, x8 + cmn x2, #0x1 + eor x1, x12, x2 + adcs x1, x7, x1 + ldp x7, x16, [x0] + eor x12, x4, x2 + adcs x4, x15, x12 + ldp x15, x12, [x0, #32] + adc x8, x8, x2 + adds x13, x14, x14 + umulh x14, x5, x10 + adcs x2, x11, x11 + adcs x3, x3, x3 + adcs x1, x1, x1 + adcs x4, x4, x4 + adcs x11, x8, x8 + adc x8, xzr, xzr + adds x13, x13, x7 + adcs x2, x2, x16 + mul x16, x5, x10 + adcs x3, x3, x9 + adcs x1, x1, x17 + umulh x5, x5, x5 + lsl x9, x13, #32 + add x9, x9, x13 + adcs x4, x4, x15 + mov x13, v28.d[1] + adcs x15, x11, x12 + lsr x7, x9, #32 + adc x11, x8, xzr + subs x7, x7, x9 + umulh x10, x10, x10 + sbc x17, x9, xzr + extr x7, x17, x7, #32 + lsr x17, x17, #32 + adds x17, x17, x9 + adc x12, xzr, xzr + subs x8, x2, x7 + sbcs x17, x3, x17 + lsl x7, x8, #32 + sbcs x2, x1, x12 + add x3, x7, x8 + sbcs x12, x4, xzr + lsr x1, x3, #32 + sbcs x7, x15, xzr + sbc x15, x9, xzr + subs x1, x1, x3 + sbc x4, x3, xzr + lsr x9, x4, #32 + extr x8, x4, x1, #32 + adds x9, x9, x3 + adc x4, xzr, xzr + subs x1, x17, x8 + lsl x17, x1, #32 + sbcs x8, x2, x9 + sbcs x9, x12, x4 + add x17, x17, x1 + mov x1, v18.d[1] + lsr x2, x17, #32 + sbcs x7, x7, xzr + mov x12, v18.d[0] + sbcs x15, x15, xzr + sbc x3, x3, xzr + subs x4, x2, x17 + sbc x2, x17, xzr + adds x12, x13, x12 + adcs x16, x16, x1 + lsr x13, x2, #32 + extr x1, x2, x4, #32 + adc x2, x14, xzr + adds x4, x13, x17 + mul x13, x6, x6 + adc x14, xzr, xzr + subs x1, x8, x1 + sbcs x4, x9, x4 + mov x9, v28.d[0] + sbcs x7, x7, x14 + sbcs x8, x15, xzr + sbcs x3, x3, xzr + sbc x14, x17, xzr + adds x17, x9, x9 + adcs x12, x12, x12 + mov x15, v19.d[0] + adcs x9, x16, x16 + umulh x6, x6, x6 + adcs x16, x2, x2 + adc x2, xzr, xzr + adds x11, x11, x8 + adcs x3, x3, xzr + adcs x14, x14, xzr + adcs x8, xzr, xzr + adds x13, x1, x13 + mov x1, v19.d[1] + adcs x6, x4, x6 + mov x4, #0xffffffff // #4294967295 + adcs x15, x7, x15 + adcs x7, x11, x5 + adcs x1, x3, x1 + adcs x14, x14, x10 + adc x11, x8, xzr + adds x6, x6, x17 + adcs x8, x15, x12 + adcs x3, x7, x9 + adcs x15, x1, x16 + mov x16, #0xffffffff00000001 // #-4294967295 + adcs x14, x14, x2 + mov x2, #0x1 // #1 + adc x17, x11, xzr + cmn x13, x16 + adcs xzr, x6, x4 + adcs xzr, x8, x2 + adcs xzr, x3, xzr + adcs xzr, x15, xzr + adcs xzr, x14, xzr + adc x1, x17, xzr + neg x9, x1 + and x1, x16, x9 + adds x11, x13, x1 + and x13, x4, x9 + adcs x5, x6, x13 + and x1, x2, x9 + adcs x7, x8, x1 + stp x11, x5, [x0] + adcs x11, x3, xzr + adcs x2, x15, xzr + stp x7, x11, [x0, #16] + adc x17, x14, xzr + stp x2, x17, [x0, #32] + ldr q1, [x26, #96] + ldp x9, x2, [x26, #96] + ldr q0, [x26, #96] + ldp x4, x6, [x26, #112] + rev64 v21.4s, v1.4s + uzp2 v28.4s, v1.4s, v1.4s + umulh x7, x9, x2 + xtn v17.2s, v1.2d + mul v27.4s, v21.4s, v0.4s + ldr q20, [x26, #128] + xtn v30.2s, v0.2d + ldr q1, [x26, #128] + uzp2 v31.4s, v0.4s, v0.4s + ldp x5, x10, [x26, #128] + umulh x8, x9, x4 + uaddlp v3.2d, v27.4s + umull v16.2d, v30.2s, v17.2s + mul x16, x9, x4 + umull v27.2d, v30.2s, v28.2s + shrn v0.2s, v20.2d, #32 + xtn v7.2s, v20.2d + shl v20.2d, v3.2d, #32 + umull v3.2d, v31.2s, v28.2s + mul x3, x2, x4 + umlal v20.2d, v30.2s, v17.2s + umull v22.2d, v7.2s, v0.2s + usra v27.2d, v16.2d, #32 + umulh x11, x2, x4 + movi v21.2d, #0xffffffff + uzp2 v28.4s, v1.4s, v1.4s + adds x15, x16, x7 + and v5.16b, v27.16b, v21.16b + adcs x3, x3, x8 + usra v3.2d, v27.2d, #32 + dup v29.2d, x6 + adcs x16, x11, xzr + mov x14, v20.d[0] + umlal v5.2d, v31.2s, v17.2s + mul x8, x9, x2 + mov x7, v20.d[1] + shl v19.2d, v22.2d, #33 + xtn v25.2s, v29.2d + rev64 v31.4s, v1.4s + lsl x13, x14, #32 + uzp2 v6.4s, v29.4s, v29.4s + umlal v19.2d, v7.2s, v7.2s + usra v3.2d, v5.2d, #32 + adds x1, x8, x8 + umulh x8, x4, x4 + add x12, x13, x14 + mul v17.4s, v31.4s, v29.4s + xtn v4.2s, v1.2d + adcs x14, x15, x15 + lsr x13, x12, #32 + adcs x15, x3, x3 + umull v31.2d, v25.2s, v28.2s + adcs x11, x16, x16 + umull v21.2d, v25.2s, v4.2s + mov x17, v3.d[0] + umull v18.2d, v6.2s, v28.2s + adc x16, x8, xzr + uaddlp v16.2d, v17.4s + movi v1.2d, #0xffffffff + subs x13, x13, x12 + usra v31.2d, v21.2d, #32 + sbc x8, x12, xzr + adds x17, x17, x1 + mul x1, x4, x4 + shl v28.2d, v16.2d, #32 + mov x3, v3.d[1] + adcs x14, x7, x14 + extr x7, x8, x13, #32 + adcs x13, x3, x15 + and v3.16b, v31.16b, v1.16b + adcs x11, x1, x11 + lsr x1, x8, #32 + umlal v3.2d, v6.2s, v4.2s + usra v18.2d, v31.2d, #32 + adc x3, x16, xzr + adds x1, x1, x12 + umlal v28.2d, v25.2s, v4.2s + adc x16, xzr, xzr + subs x15, x17, x7 + sbcs x7, x14, x1 + lsl x1, x15, #32 + sbcs x16, x13, x16 + add x8, x1, x15 + usra v18.2d, v3.2d, #32 + sbcs x14, x11, xzr + lsr x1, x8, #32 + sbcs x17, x3, xzr + sbc x11, x12, xzr + subs x13, x1, x8 + umulh x12, x4, x10 + sbc x1, x8, xzr + extr x13, x1, x13, #32 + lsr x1, x1, #32 + adds x15, x1, x8 + adc x1, xzr, xzr + subs x7, x7, x13 + sbcs x13, x16, x15 + lsl x3, x7, #32 + umulh x16, x2, x5 + sbcs x15, x14, x1 + add x7, x3, x7 + sbcs x3, x17, xzr + lsr x1, x7, #32 + sbcs x14, x11, xzr + sbc x11, x8, xzr + subs x8, x1, x7 + sbc x1, x7, xzr + extr x8, x1, x8, #32 + lsr x1, x1, #32 + adds x1, x1, x7 + adc x17, xzr, xzr + subs x13, x13, x8 + umulh x8, x9, x6 + sbcs x1, x15, x1 + sbcs x15, x3, x17 + sbcs x3, x14, xzr + mul x17, x2, x5 + sbcs x11, x11, xzr + stp x13, x1, [sp, #240] + sbc x14, x7, xzr + mul x7, x4, x10 + subs x1, x9, x2 + stp x15, x3, [sp, #256] + csetm x15, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + stp x11, x14, [sp, #272] + mul x14, x9, x6 + adds x17, x8, x17 + adcs x7, x16, x7 + adc x13, x12, xzr + subs x12, x5, x6 + cneg x3, x12, cc // cc = lo, ul, last + cinv x16, x15, cc // cc = lo, ul, last + mul x8, x1, x3 + umulh x1, x1, x3 + eor x12, x8, x16 + adds x11, x17, x14 + adcs x3, x7, x17 + adcs x15, x13, x7 + adc x8, x13, xzr + adds x3, x3, x14 + adcs x15, x15, x17 + adcs x17, x8, x7 + eor x1, x1, x16 + adc x13, x13, xzr + subs x9, x9, x4 + csetm x8, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x4, x2, x4 + cneg x4, x4, cc // cc = lo, ul, last + csetm x7, cc // cc = lo, ul, last + subs x2, x10, x6 + cinv x8, x8, cc // cc = lo, ul, last + cneg x2, x2, cc // cc = lo, ul, last + cmn x16, #0x1 + adcs x11, x11, x12 + mul x12, x9, x2 + adcs x3, x3, x1 + adcs x15, x15, x16 + umulh x9, x9, x2 + adcs x17, x17, x16 + adc x13, x13, x16 + subs x1, x10, x5 + cinv x2, x7, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + eor x9, x9, x8 + cmn x8, #0x1 + eor x7, x12, x8 + mul x12, x4, x1 + adcs x3, x3, x7 + adcs x7, x15, x9 + adcs x15, x17, x8 + ldp x9, x17, [sp, #256] + umulh x4, x4, x1 + adc x8, x13, x8 + cmn x2, #0x1 + eor x1, x12, x2 + adcs x1, x7, x1 + ldp x7, x16, [sp, #240] + eor x12, x4, x2 + adcs x4, x15, x12 + ldp x15, x12, [sp, #272] + adc x8, x8, x2 + adds x13, x14, x14 + umulh x14, x5, x10 + adcs x2, x11, x11 + adcs x3, x3, x3 + adcs x1, x1, x1 + adcs x4, x4, x4 + adcs x11, x8, x8 + adc x8, xzr, xzr + adds x13, x13, x7 + adcs x2, x2, x16 + mul x16, x5, x10 + adcs x3, x3, x9 + adcs x1, x1, x17 + umulh x5, x5, x5 + lsl x9, x13, #32 + add x9, x9, x13 + adcs x4, x4, x15 + mov x13, v28.d[1] + adcs x15, x11, x12 + lsr x7, x9, #32 + adc x11, x8, xzr + subs x7, x7, x9 + umulh x10, x10, x10 + sbc x17, x9, xzr + extr x7, x17, x7, #32 + lsr x17, x17, #32 + adds x17, x17, x9 + adc x12, xzr, xzr + subs x8, x2, x7 + sbcs x17, x3, x17 + lsl x7, x8, #32 + sbcs x2, x1, x12 + add x3, x7, x8 + sbcs x12, x4, xzr + lsr x1, x3, #32 + sbcs x7, x15, xzr + sbc x15, x9, xzr + subs x1, x1, x3 + sbc x4, x3, xzr + lsr x9, x4, #32 + extr x8, x4, x1, #32 + adds x9, x9, x3 + adc x4, xzr, xzr + subs x1, x17, x8 + lsl x17, x1, #32 + sbcs x8, x2, x9 + sbcs x9, x12, x4 + add x17, x17, x1 + mov x1, v18.d[1] + lsr x2, x17, #32 + sbcs x7, x7, xzr + mov x12, v18.d[0] + sbcs x15, x15, xzr + sbc x3, x3, xzr + subs x4, x2, x17 + sbc x2, x17, xzr + adds x12, x13, x12 + adcs x16, x16, x1 + lsr x13, x2, #32 + extr x1, x2, x4, #32 + adc x2, x14, xzr + adds x4, x13, x17 + mul x13, x6, x6 + adc x14, xzr, xzr + subs x1, x8, x1 + sbcs x4, x9, x4 + mov x9, v28.d[0] + sbcs x7, x7, x14 + sbcs x8, x15, xzr + sbcs x3, x3, xzr + sbc x14, x17, xzr + adds x17, x9, x9 + adcs x12, x12, x12 + mov x15, v19.d[0] + adcs x9, x16, x16 + umulh x6, x6, x6 + adcs x16, x2, x2 + adc x2, xzr, xzr + adds x11, x11, x8 + adcs x3, x3, xzr + adcs x14, x14, xzr + adcs x8, xzr, xzr + adds x13, x1, x13 + mov x1, v19.d[1] + adcs x6, x4, x6 + mov x4, #0xffffffff // #4294967295 + adcs x15, x7, x15 + adcs x7, x11, x5 + adcs x1, x3, x1 + adcs x14, x14, x10 + adc x11, x8, xzr + adds x6, x6, x17 + adcs x8, x15, x12 + adcs x3, x7, x9 + adcs x15, x1, x16 + mov x16, #0xffffffff00000001 // #-4294967295 + adcs x14, x14, x2 + mov x2, #0x1 // #1 + adc x17, x11, xzr + cmn x13, x16 + adcs xzr, x6, x4 + adcs xzr, x8, x2 + adcs xzr, x3, xzr + adcs xzr, x15, xzr + adcs xzr, x14, xzr + adc x1, x17, xzr + neg x9, x1 + and x1, x16, x9 + adds x11, x13, x1 + and x13, x4, x9 + adcs x5, x6, x13 + and x1, x2, x9 + adcs x7, x8, x1 + stp x11, x5, [sp, #240] + adcs x11, x3, xzr + adcs x2, x15, xzr + stp x7, x11, [sp, #256] + adc x17, x14, xzr + stp x2, x17, [sp, #272] + stp x23, x24, [sp, #0x150] + ldr q3, [x26, #96] + ldr q25, [x25, #48] + ldp x13, x23, [x25, #48] + ldp x3, x21, [x26, #96] + rev64 v23.4s, v25.4s + uzp1 v17.4s, v25.4s, v3.4s + umulh x15, x3, x13 + mul v6.4s, v23.4s, v3.4s + uzp1 v3.4s, v3.4s, v3.4s + ldr q27, [x25, #80] + ldp x8, x24, [x26, #112] + subs x6, x3, x21 + ldr q0, [x26, #128] + movi v23.2d, #0xffffffff + csetm x10, cc // cc = lo, ul, last + umulh x19, x21, x23 + rev64 v4.4s, v27.4s + uzp2 v25.4s, v27.4s, v27.4s + cneg x4, x6, cc // cc = lo, ul, last + subs x7, x23, x13 + xtn v22.2s, v0.2d + xtn v24.2s, v27.2d + cneg x20, x7, cc // cc = lo, ul, last + ldp x6, x14, [x25, #64] + mul v27.4s, v4.4s, v0.4s + uaddlp v20.2d, v6.4s + cinv x5, x10, cc // cc = lo, ul, last + mul x16, x4, x20 + uzp2 v6.4s, v0.4s, v0.4s + umull v21.2d, v22.2s, v25.2s + shl v0.2d, v20.2d, #32 + umlal v0.2d, v3.2s, v17.2s + mul x22, x8, x6 + umull v1.2d, v6.2s, v25.2s + subs x12, x3, x8 + umull v20.2d, v22.2s, v24.2s + cneg x17, x12, cc // cc = lo, ul, last + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc // cc = lo, ul, last + usra v21.2d, v20.2d, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2d, v21.2d, #32 + adds x22, x15, x7 + and v26.16b, v21.16b, v23.16b + adcs x16, x12, x15 + uaddlp v25.2d, v27.4s + adcs x9, x19, x12 + umlal v26.2d, v6.2s, v24.2s + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2d, v25.2d, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc // cc = lo, ul, last + cinv x10, x10, cc // cc = lo, ul, last + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc // cc = lo, ul, last + eor x19, x19, x10 + csetm x4, cc // cc = lo, ul, last + subs x16, x6, x23 + cneg x16, x16, cc // cc = lo, ul, last + umlal v27.2d, v22.2s, v24.2s + mul x15, x20, x16 + cinv x4, x4, cc // cc = lo, ul, last + cmn x10, #0x1 + usra v1.2d, v26.2d, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [x25, #80] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [x26, #128] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x16, x4, x16 + mov x4, v27.d[0] + sbcs x11, x20, x11 + sbcs x20, x9, x12 + stp x16, x11, [sp, #288] + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [sp, #304] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc // cc = lo, ul, last + csetm x2, cc // cc = lo, ul, last + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc // cc = lo, ul, last + cneg x19, x19, cc // cc = lo, ul, last + stp x9, x20, [sp, #320] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc // cc = lo, ul, last + csetm x12, cc // cc = lo, ul, last + subs x9, x17, x14 + cinv x12, x12, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc // cc = lo, ul, last + cneg x24, x10, cc // cc = lo, ul, last + subs x10, x17, x15 + cinv x7, x7, cc // cc = lo, ul, last + cneg x10, x10, cc // cc = lo, ul, last + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + ldp x20, x9, [sp, #288] + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [sp, #304] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [sp, #320] + adds x20, x22, x20 + mul x10, x13, x14 + adcs x11, x11, x9 + eor x9, x8, x21 + adcs x21, x19, x17 + stp x20, x11, [sp, #288] + adcs x12, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + stp x21, x12, [sp, #304] + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [sp, #320] + cneg x3, x21, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc // cc = lo, ul, last + csetm x16, cc // cc = lo, ul, last + subs x21, x6, x15 + cneg x22, x21, cc // cc = lo, ul, last + cinv x21, x24, cc // cc = lo, ul, last + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + subs x20, x14, x15 + cinv x24, x24, cc // cc = lo, ul, last + mul x22, x3, x22 + cneg x3, x20, cc // cc = lo, ul, last + subs x13, x6, x14 + cneg x20, x13, cc // cc = lo, ul, last + cinv x15, x16, cc // cc = lo, ul, last + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe // #-2 + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [sp, #288] + ldp x21, x12, [sp, #304] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [sp, #320] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x21 + eor x1, x22, x9 + adcs x24, x23, x12 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x21 + adcs x15, x17, x12 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff // #4294967295 + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc // cc = lo, ul, last + and x11, x16, x23 + and x20, x14, x23 + adds x22, x6, x20 + eor x3, x20, x23 + adcs x5, x7, x3 + adcs x14, x24, x11 + stp x22, x5, [sp, #288] + adcs x5, x13, x23 + adcs x21, x8, x23 + stp x14, x5, [sp, #304] + adc x12, x15, x23 + stp x21, x12, [sp, #320] + ldr q3, [x25, #96] + ldr q25, [x26, #48] + ldp x13, x23, [x26, #48] + ldp x3, x21, [x25, #96] + rev64 v23.4s, v25.4s + uzp1 v17.4s, v25.4s, v3.4s + umulh x15, x3, x13 + mul v6.4s, v23.4s, v3.4s + uzp1 v3.4s, v3.4s, v3.4s + ldr q27, [x26, #80] + ldp x8, x24, [x25, #112] + subs x6, x3, x21 + ldr q0, [x25, #128] + movi v23.2d, #0xffffffff + csetm x10, cc // cc = lo, ul, last + umulh x19, x21, x23 + rev64 v4.4s, v27.4s + uzp2 v25.4s, v27.4s, v27.4s + cneg x4, x6, cc // cc = lo, ul, last + subs x7, x23, x13 + xtn v22.2s, v0.2d + xtn v24.2s, v27.2d + cneg x20, x7, cc // cc = lo, ul, last + ldp x6, x14, [x26, #64] + mul v27.4s, v4.4s, v0.4s + uaddlp v20.2d, v6.4s + cinv x5, x10, cc // cc = lo, ul, last + mul x16, x4, x20 + uzp2 v6.4s, v0.4s, v0.4s + umull v21.2d, v22.2s, v25.2s + shl v0.2d, v20.2d, #32 + umlal v0.2d, v3.2s, v17.2s + mul x22, x8, x6 + umull v1.2d, v6.2s, v25.2s + subs x12, x3, x8 + umull v20.2d, v22.2s, v24.2s + cneg x17, x12, cc // cc = lo, ul, last + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc // cc = lo, ul, last + usra v21.2d, v20.2d, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2d, v21.2d, #32 + adds x22, x15, x7 + and v26.16b, v21.16b, v23.16b + adcs x16, x12, x15 + uaddlp v25.2d, v27.4s + adcs x9, x19, x12 + umlal v26.2d, v6.2s, v24.2s + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2d, v25.2d, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc // cc = lo, ul, last + cinv x10, x10, cc // cc = lo, ul, last + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc // cc = lo, ul, last + eor x19, x19, x10 + csetm x4, cc // cc = lo, ul, last + subs x16, x6, x23 + cneg x16, x16, cc // cc = lo, ul, last + umlal v27.2d, v22.2s, v24.2s + mul x15, x20, x16 + cinv x4, x4, cc // cc = lo, ul, last + cmn x10, #0x1 + usra v1.2d, v26.2d, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [x26, #80] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [x25, #128] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x16, x4, x16 + mov x4, v27.d[0] + sbcs x11, x20, x11 + sbcs x20, x9, x12 + stp x16, x11, [sp, #48] + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [sp, #64] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc // cc = lo, ul, last + csetm x2, cc // cc = lo, ul, last + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc // cc = lo, ul, last + cneg x19, x19, cc // cc = lo, ul, last + stp x9, x20, [sp, #80] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc // cc = lo, ul, last + csetm x12, cc // cc = lo, ul, last + subs x9, x17, x14 + cinv x12, x12, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc // cc = lo, ul, last + cneg x24, x10, cc // cc = lo, ul, last + subs x10, x17, x15 + cinv x7, x7, cc // cc = lo, ul, last + cneg x10, x10, cc // cc = lo, ul, last + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + ldp x20, x9, [sp, #48] + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [sp, #64] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [sp, #80] + adds x20, x22, x20 + mul x10, x13, x14 + adcs x11, x11, x9 + eor x9, x8, x21 + adcs x21, x19, x17 + stp x20, x11, [sp, #48] + adcs x12, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + stp x21, x12, [sp, #64] + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [sp, #80] + cneg x3, x21, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc // cc = lo, ul, last + csetm x16, cc // cc = lo, ul, last + subs x21, x6, x15 + cneg x22, x21, cc // cc = lo, ul, last + cinv x21, x24, cc // cc = lo, ul, last + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + subs x20, x14, x15 + cinv x24, x24, cc // cc = lo, ul, last + mul x22, x3, x22 + cneg x3, x20, cc // cc = lo, ul, last + subs x13, x6, x14 + cneg x20, x13, cc // cc = lo, ul, last + cinv x15, x16, cc // cc = lo, ul, last + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe // #-2 + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [sp, #48] + ldp x21, x12, [sp, #64] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [sp, #80] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x21 + eor x1, x22, x9 + adcs x24, x23, x12 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x21 + adcs x15, x17, x12 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff // #4294967295 + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc // cc = lo, ul, last + and x11, x16, x23 + and x20, x14, x23 + adds x22, x6, x20 + eor x3, x20, x23 + adcs x5, x7, x3 + adcs x14, x24, x11 + stp x22, x5, [sp, #48] + adcs x5, x13, x23 + adcs x21, x8, x23 + stp x14, x5, [sp, #64] + adc x12, x15, x23 + stp x21, x12, [sp, #80] + mov x1, sp + ldr q3, [x1] + ldr q25, [x26] + ldp x13, x23, [x26] + ldp x3, x21, [x1] + rev64 v23.4s, v25.4s + uzp1 v17.4s, v25.4s, v3.4s + umulh x15, x3, x13 + mul v6.4s, v23.4s, v3.4s + uzp1 v3.4s, v3.4s, v3.4s + ldr q27, [x26, #32] + ldp x8, x24, [x1, #16] + subs x6, x3, x21 + ldr q0, [x1, #32] + movi v23.2d, #0xffffffff + csetm x10, cc // cc = lo, ul, last + umulh x19, x21, x23 + rev64 v4.4s, v27.4s + uzp2 v25.4s, v27.4s, v27.4s + cneg x4, x6, cc // cc = lo, ul, last + subs x7, x23, x13 + xtn v22.2s, v0.2d + xtn v24.2s, v27.2d + cneg x20, x7, cc // cc = lo, ul, last + ldp x6, x14, [x26, #16] + mul v27.4s, v4.4s, v0.4s + uaddlp v20.2d, v6.4s + cinv x5, x10, cc // cc = lo, ul, last + mul x16, x4, x20 + uzp2 v6.4s, v0.4s, v0.4s + umull v21.2d, v22.2s, v25.2s + shl v0.2d, v20.2d, #32 + umlal v0.2d, v3.2s, v17.2s + mul x22, x8, x6 + umull v1.2d, v6.2s, v25.2s + subs x12, x3, x8 + umull v20.2d, v22.2s, v24.2s + cneg x17, x12, cc // cc = lo, ul, last + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc // cc = lo, ul, last + usra v21.2d, v20.2d, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2d, v21.2d, #32 + adds x22, x15, x7 + and v26.16b, v21.16b, v23.16b + adcs x16, x12, x15 + uaddlp v25.2d, v27.4s + adcs x9, x19, x12 + umlal v26.2d, v6.2s, v24.2s + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2d, v25.2d, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc // cc = lo, ul, last + cinv x10, x10, cc // cc = lo, ul, last + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc // cc = lo, ul, last + eor x19, x19, x10 + csetm x4, cc // cc = lo, ul, last + subs x16, x6, x23 + cneg x16, x16, cc // cc = lo, ul, last + umlal v27.2d, v22.2s, v24.2s + mul x15, x20, x16 + cinv x4, x4, cc // cc = lo, ul, last + cmn x10, #0x1 + usra v1.2d, v26.2d, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [x26, #32] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [x1, #32] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x16, x4, x16 + mov x4, v27.d[0] + sbcs x11, x20, x11 + sbcs x20, x9, x12 + stp x16, x11, [sp, #96] + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [sp, #112] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc // cc = lo, ul, last + csetm x2, cc // cc = lo, ul, last + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc // cc = lo, ul, last + cneg x19, x19, cc // cc = lo, ul, last + stp x9, x20, [sp, #128] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc // cc = lo, ul, last + csetm x12, cc // cc = lo, ul, last + subs x9, x17, x14 + cinv x12, x12, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc // cc = lo, ul, last + cneg x24, x10, cc // cc = lo, ul, last + subs x10, x17, x15 + cinv x7, x7, cc // cc = lo, ul, last + cneg x10, x10, cc // cc = lo, ul, last + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + ldp x20, x9, [sp, #96] + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [sp, #112] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [sp, #128] + adds x20, x22, x20 + mul x10, x13, x14 + adcs x11, x11, x9 + eor x9, x8, x21 + adcs x21, x19, x17 + stp x20, x11, [sp, #96] + adcs x12, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + stp x21, x12, [sp, #112] + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [sp, #128] + cneg x3, x21, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc // cc = lo, ul, last + csetm x16, cc // cc = lo, ul, last + subs x21, x6, x15 + cneg x22, x21, cc // cc = lo, ul, last + cinv x21, x24, cc // cc = lo, ul, last + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + subs x20, x14, x15 + cinv x24, x24, cc // cc = lo, ul, last + mul x22, x3, x22 + cneg x3, x20, cc // cc = lo, ul, last + subs x13, x6, x14 + cneg x20, x13, cc // cc = lo, ul, last + cinv x15, x16, cc // cc = lo, ul, last + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe // #-2 + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [sp, #96] + ldp x21, x12, [sp, #112] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [sp, #128] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x21 + eor x1, x22, x9 + adcs x24, x23, x12 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x21 + adcs x15, x17, x12 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff // #4294967295 + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc // cc = lo, ul, last + and x11, x16, x23 + and x20, x14, x23 + adds x22, x6, x20 + eor x3, x20, x23 + adcs x5, x7, x3 + adcs x14, x24, x11 + stp x22, x5, [sp, #96] + adcs x5, x13, x23 + adcs x21, x8, x23 + stp x14, x5, [sp, #112] + adc x12, x15, x23 + stp x21, x12, [sp, #128] + ldr q3, [sp, #240] + ldr q25, [x25] + ldp x13, x23, [x25] + ldp x3, x21, [sp, #240] + rev64 v23.4s, v25.4s + uzp1 v17.4s, v25.4s, v3.4s + umulh x15, x3, x13 + mul v6.4s, v23.4s, v3.4s + uzp1 v3.4s, v3.4s, v3.4s + ldr q27, [x25, #32] + ldp x8, x24, [sp, #256] + subs x6, x3, x21 + ldr q0, [sp, #272] + movi v23.2d, #0xffffffff + csetm x10, cc // cc = lo, ul, last + umulh x19, x21, x23 + rev64 v4.4s, v27.4s + uzp2 v25.4s, v27.4s, v27.4s + cneg x4, x6, cc // cc = lo, ul, last + subs x7, x23, x13 + xtn v22.2s, v0.2d + xtn v24.2s, v27.2d + cneg x20, x7, cc // cc = lo, ul, last + ldp x6, x14, [x25, #16] + mul v27.4s, v4.4s, v0.4s + uaddlp v20.2d, v6.4s + cinv x5, x10, cc // cc = lo, ul, last + mul x16, x4, x20 + uzp2 v6.4s, v0.4s, v0.4s + umull v21.2d, v22.2s, v25.2s + shl v0.2d, v20.2d, #32 + umlal v0.2d, v3.2s, v17.2s + mul x22, x8, x6 + umull v1.2d, v6.2s, v25.2s + subs x12, x3, x8 + umull v20.2d, v22.2s, v24.2s + cneg x17, x12, cc // cc = lo, ul, last + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc // cc = lo, ul, last + usra v21.2d, v20.2d, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2d, v21.2d, #32 + adds x22, x15, x7 + and v26.16b, v21.16b, v23.16b + adcs x16, x12, x15 + uaddlp v25.2d, v27.4s + adcs x9, x19, x12 + umlal v26.2d, v6.2s, v24.2s + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2d, v25.2d, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc // cc = lo, ul, last + cinv x10, x10, cc // cc = lo, ul, last + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc // cc = lo, ul, last + eor x19, x19, x10 + csetm x4, cc // cc = lo, ul, last + subs x16, x6, x23 + cneg x16, x16, cc // cc = lo, ul, last + umlal v27.2d, v22.2s, v24.2s + mul x15, x20, x16 + cinv x4, x4, cc // cc = lo, ul, last + cmn x10, #0x1 + usra v1.2d, v26.2d, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [x25, #32] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [sp, #272] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x16, x4, x16 + mov x4, v27.d[0] + sbcs x11, x20, x11 + sbcs x20, x9, x12 + stp x16, x11, [sp, #192] + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [sp, #208] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc // cc = lo, ul, last + csetm x2, cc // cc = lo, ul, last + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc // cc = lo, ul, last + cneg x19, x19, cc // cc = lo, ul, last + stp x9, x20, [sp, #224] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc // cc = lo, ul, last + csetm x12, cc // cc = lo, ul, last + subs x9, x17, x14 + cinv x12, x12, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc // cc = lo, ul, last + cneg x24, x10, cc // cc = lo, ul, last + subs x10, x17, x15 + cinv x7, x7, cc // cc = lo, ul, last + cneg x10, x10, cc // cc = lo, ul, last + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + ldp x20, x9, [sp, #192] + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [sp, #208] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [sp, #224] + adds x20, x22, x20 + mul x10, x13, x14 + adcs x11, x11, x9 + eor x9, x8, x21 + adcs x21, x19, x17 + stp x20, x11, [sp, #192] + adcs x12, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + stp x21, x12, [sp, #208] + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [sp, #224] + cneg x3, x21, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc // cc = lo, ul, last + csetm x16, cc // cc = lo, ul, last + subs x21, x6, x15 + cneg x22, x21, cc // cc = lo, ul, last + cinv x21, x24, cc // cc = lo, ul, last + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + subs x20, x14, x15 + cinv x24, x24, cc // cc = lo, ul, last + mul x22, x3, x22 + cneg x3, x20, cc // cc = lo, ul, last + subs x13, x6, x14 + cneg x20, x13, cc // cc = lo, ul, last + cinv x15, x16, cc // cc = lo, ul, last + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe // #-2 + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [sp, #192] + ldp x21, x12, [sp, #208] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [sp, #224] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x21 + eor x1, x22, x9 + adcs x24, x23, x12 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x21 + adcs x15, x17, x12 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff // #4294967295 + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc // cc = lo, ul, last + and x11, x16, x23 + and x20, x14, x23 + adds x22, x6, x20 + eor x3, x20, x23 + adcs x5, x7, x3 + adcs x14, x24, x11 + stp x22, x5, [sp, #192] + adcs x5, x13, x23 + adcs x21, x8, x23 + stp x14, x5, [sp, #208] + adc x12, x15, x23 + stp x21, x12, [sp, #224] + mov x1, sp + ldr q3, [x1] + ldr q25, [sp, #48] + ldp x13, x23, [sp, #48] + ldp x3, x21, [x1] + rev64 v23.4s, v25.4s + uzp1 v17.4s, v25.4s, v3.4s + umulh x15, x3, x13 + mul v6.4s, v23.4s, v3.4s + uzp1 v3.4s, v3.4s, v3.4s + ldr q27, [sp, #80] + ldp x8, x24, [x1, #16] + subs x6, x3, x21 + ldr q0, [x1, #32] + movi v23.2d, #0xffffffff + csetm x10, cc // cc = lo, ul, last + umulh x19, x21, x23 + rev64 v4.4s, v27.4s + uzp2 v25.4s, v27.4s, v27.4s + cneg x4, x6, cc // cc = lo, ul, last + subs x7, x23, x13 + xtn v22.2s, v0.2d + xtn v24.2s, v27.2d + cneg x20, x7, cc // cc = lo, ul, last + ldp x6, x14, [sp, #64] + mul v27.4s, v4.4s, v0.4s + uaddlp v20.2d, v6.4s + cinv x5, x10, cc // cc = lo, ul, last + mul x16, x4, x20 + uzp2 v6.4s, v0.4s, v0.4s + umull v21.2d, v22.2s, v25.2s + shl v0.2d, v20.2d, #32 + umlal v0.2d, v3.2s, v17.2s + mul x22, x8, x6 + umull v1.2d, v6.2s, v25.2s + subs x12, x3, x8 + umull v20.2d, v22.2s, v24.2s + cneg x17, x12, cc // cc = lo, ul, last + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc // cc = lo, ul, last + usra v21.2d, v20.2d, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2d, v21.2d, #32 + adds x22, x15, x7 + and v26.16b, v21.16b, v23.16b + adcs x16, x12, x15 + uaddlp v25.2d, v27.4s + adcs x9, x19, x12 + umlal v26.2d, v6.2s, v24.2s + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2d, v25.2d, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc // cc = lo, ul, last + cinv x10, x10, cc // cc = lo, ul, last + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc // cc = lo, ul, last + eor x19, x19, x10 + csetm x4, cc // cc = lo, ul, last + subs x16, x6, x23 + cneg x16, x16, cc // cc = lo, ul, last + umlal v27.2d, v22.2s, v24.2s + mul x15, x20, x16 + cinv x4, x4, cc // cc = lo, ul, last + cmn x10, #0x1 + usra v1.2d, v26.2d, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [sp, #80] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [x1, #32] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x16, x4, x16 + mov x4, v27.d[0] + sbcs x11, x20, x11 + sbcs x20, x9, x12 + stp x16, x11, [sp, #48] + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [sp, #64] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc // cc = lo, ul, last + csetm x2, cc // cc = lo, ul, last + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc // cc = lo, ul, last + cneg x19, x19, cc // cc = lo, ul, last + stp x9, x20, [sp, #80] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc // cc = lo, ul, last + csetm x12, cc // cc = lo, ul, last + subs x9, x17, x14 + cinv x12, x12, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc // cc = lo, ul, last + cneg x24, x10, cc // cc = lo, ul, last + subs x10, x17, x15 + cinv x7, x7, cc // cc = lo, ul, last + cneg x10, x10, cc // cc = lo, ul, last + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + ldp x20, x9, [sp, #48] + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [sp, #64] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [sp, #80] + adds x20, x22, x20 + mul x10, x13, x14 + adcs x11, x11, x9 + eor x9, x8, x21 + adcs x21, x19, x17 + stp x20, x11, [sp, #48] + adcs x12, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + stp x21, x12, [sp, #64] + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [sp, #80] + cneg x3, x21, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc // cc = lo, ul, last + csetm x16, cc // cc = lo, ul, last + subs x21, x6, x15 + cneg x22, x21, cc // cc = lo, ul, last + cinv x21, x24, cc // cc = lo, ul, last + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + subs x20, x14, x15 + cinv x24, x24, cc // cc = lo, ul, last + mul x22, x3, x22 + cneg x3, x20, cc // cc = lo, ul, last + subs x13, x6, x14 + cneg x20, x13, cc // cc = lo, ul, last + cinv x15, x16, cc // cc = lo, ul, last + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe // #-2 + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [sp, #48] + ldp x21, x12, [sp, #64] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [sp, #80] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x21 + eor x1, x22, x9 + adcs x24, x23, x12 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x21 + adcs x15, x17, x12 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff // #4294967295 + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc // cc = lo, ul, last + and x11, x16, x23 + and x20, x14, x23 + adds x22, x6, x20 + eor x3, x20, x23 + adcs x5, x7, x3 + adcs x14, x24, x11 + stp x22, x5, [sp, #48] + adcs x5, x13, x23 + adcs x21, x8, x23 + stp x14, x5, [sp, #64] + adc x12, x15, x23 + stp x21, x12, [sp, #80] + ldr q3, [sp, #240] + ldr q25, [sp, #288] + ldp x13, x23, [sp, #288] + ldp x3, x21, [sp, #240] + rev64 v23.4s, v25.4s + uzp1 v17.4s, v25.4s, v3.4s + umulh x15, x3, x13 + mul v6.4s, v23.4s, v3.4s + uzp1 v3.4s, v3.4s, v3.4s + ldr q27, [sp, #320] + ldp x8, x24, [sp, #256] + subs x6, x3, x21 + ldr q0, [sp, #272] + movi v23.2d, #0xffffffff + csetm x10, cc // cc = lo, ul, last + umulh x19, x21, x23 + rev64 v4.4s, v27.4s + uzp2 v25.4s, v27.4s, v27.4s + cneg x4, x6, cc // cc = lo, ul, last + subs x7, x23, x13 + xtn v22.2s, v0.2d + xtn v24.2s, v27.2d + cneg x20, x7, cc // cc = lo, ul, last + ldp x6, x14, [sp, #304] + mul v27.4s, v4.4s, v0.4s + uaddlp v20.2d, v6.4s + cinv x5, x10, cc // cc = lo, ul, last + mul x16, x4, x20 + uzp2 v6.4s, v0.4s, v0.4s + umull v21.2d, v22.2s, v25.2s + shl v0.2d, v20.2d, #32 + umlal v0.2d, v3.2s, v17.2s + mul x22, x8, x6 + umull v1.2d, v6.2s, v25.2s + subs x12, x3, x8 + umull v20.2d, v22.2s, v24.2s + cneg x17, x12, cc // cc = lo, ul, last + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc // cc = lo, ul, last + usra v21.2d, v20.2d, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2d, v21.2d, #32 + adds x22, x15, x7 + and v26.16b, v21.16b, v23.16b + adcs x16, x12, x15 + uaddlp v25.2d, v27.4s + adcs x9, x19, x12 + umlal v26.2d, v6.2s, v24.2s + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2d, v25.2d, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc // cc = lo, ul, last + cinv x10, x10, cc // cc = lo, ul, last + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc // cc = lo, ul, last + eor x19, x19, x10 + csetm x4, cc // cc = lo, ul, last + subs x16, x6, x23 + cneg x16, x16, cc // cc = lo, ul, last + umlal v27.2d, v22.2s, v24.2s + mul x15, x20, x16 + cinv x4, x4, cc // cc = lo, ul, last + cmn x10, #0x1 + usra v1.2d, v26.2d, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [sp, #320] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [sp, #272] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x16, x4, x16 + mov x4, v27.d[0] + sbcs x11, x20, x11 + sbcs x20, x9, x12 + stp x16, x11, [sp, #288] + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [sp, #304] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc // cc = lo, ul, last + csetm x2, cc // cc = lo, ul, last + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc // cc = lo, ul, last + cneg x19, x19, cc // cc = lo, ul, last + stp x9, x20, [sp, #320] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc // cc = lo, ul, last + csetm x12, cc // cc = lo, ul, last + subs x9, x17, x14 + cinv x12, x12, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc // cc = lo, ul, last + cneg x24, x10, cc // cc = lo, ul, last + subs x10, x17, x15 + cinv x7, x7, cc // cc = lo, ul, last + cneg x10, x10, cc // cc = lo, ul, last + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + ldp x20, x9, [sp, #288] + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [sp, #304] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [sp, #320] + adds x20, x22, x20 + mul x10, x13, x14 + adcs x11, x11, x9 + eor x9, x8, x21 + adcs x21, x19, x17 + stp x20, x11, [sp, #288] + adcs x12, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + stp x21, x12, [sp, #304] + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [sp, #320] + cneg x3, x21, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc // cc = lo, ul, last + csetm x16, cc // cc = lo, ul, last + subs x21, x6, x15 + cneg x22, x21, cc // cc = lo, ul, last + cinv x21, x24, cc // cc = lo, ul, last + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + subs x20, x14, x15 + cinv x24, x24, cc // cc = lo, ul, last + mul x22, x3, x22 + cneg x3, x20, cc // cc = lo, ul, last + subs x13, x6, x14 + cneg x20, x13, cc // cc = lo, ul, last + cinv x15, x16, cc // cc = lo, ul, last + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe // #-2 + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [sp, #288] + ldp x21, x12, [sp, #304] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [sp, #320] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x21 + eor x1, x22, x9 + adcs x24, x23, x12 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x21 + adcs x15, x17, x12 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff // #4294967295 + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc // cc = lo, ul, last + and x11, x16, x23 + and x20, x14, x23 + adds x22, x6, x20 + eor x3, x20, x23 + adcs x5, x7, x3 + adcs x2, x24, x11 + stp x22, x5, [sp, #288] + adcs x11, x13, x23 + adcs x12, x8, x23 + stp x2, x11, [sp, #304] + adc x13, x15, x23 + stp x12, x13, [sp, #320] + ldp x5, x6, [sp, #96] + ldp x4, x3, [sp, #192] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #112] + ldp x4, x3, [sp, #208] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #128] + ldp x4, x3, [sp, #224] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, cc // cc = lo, ul, last + mov x4, #0xffffffff // #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #0xfffffffffffffffe // #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [sp, #240] + stp x7, x8, [sp, #256] + stp x9, x10, [sp, #272] + ldp x5, x6, [sp, #48] + ldp x4, x3, [sp, #288] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #64] + sbcs x7, x7, x2 + sbcs x8, x8, x11 + ldp x9, x10, [sp, #80] + sbcs x9, x9, x12 + sbcs x10, x10, x13 + csetm x3, cc // cc = lo, ul, last + mov x4, #0xffffffff // #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #0xfffffffffffffffe // #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [sp, #48] + stp x7, x8, [sp, #64] + stp x9, x10, [sp, #80] + ldr q1, [sp, #240] + ldp x9, x2, [sp, #240] + ldr q0, [sp, #240] + ldp x4, x6, [sp, #256] + rev64 v21.4s, v1.4s + uzp2 v28.4s, v1.4s, v1.4s + umulh x7, x9, x2 + xtn v17.2s, v1.2d + mul v27.4s, v21.4s, v0.4s + ldr q20, [sp, #272] + xtn v30.2s, v0.2d + ldr q1, [sp, #272] + uzp2 v31.4s, v0.4s, v0.4s + ldp x5, x10, [sp, #272] + umulh x8, x9, x4 + uaddlp v3.2d, v27.4s + umull v16.2d, v30.2s, v17.2s + mul x16, x9, x4 + umull v27.2d, v30.2s, v28.2s + shrn v0.2s, v20.2d, #32 + xtn v7.2s, v20.2d + shl v20.2d, v3.2d, #32 + umull v3.2d, v31.2s, v28.2s + mul x3, x2, x4 + umlal v20.2d, v30.2s, v17.2s + umull v22.2d, v7.2s, v0.2s + usra v27.2d, v16.2d, #32 + umulh x11, x2, x4 + movi v21.2d, #0xffffffff + uzp2 v28.4s, v1.4s, v1.4s + adds x15, x16, x7 + and v5.16b, v27.16b, v21.16b + adcs x3, x3, x8 + usra v3.2d, v27.2d, #32 + dup v29.2d, x6 + adcs x16, x11, xzr + mov x14, v20.d[0] + umlal v5.2d, v31.2s, v17.2s + mul x8, x9, x2 + mov x7, v20.d[1] + shl v19.2d, v22.2d, #33 + xtn v25.2s, v29.2d + rev64 v31.4s, v1.4s + lsl x13, x14, #32 + uzp2 v6.4s, v29.4s, v29.4s + umlal v19.2d, v7.2s, v7.2s + usra v3.2d, v5.2d, #32 + adds x1, x8, x8 + umulh x8, x4, x4 + add x12, x13, x14 + mul v17.4s, v31.4s, v29.4s + xtn v4.2s, v1.2d + adcs x14, x15, x15 + lsr x13, x12, #32 + adcs x15, x3, x3 + umull v31.2d, v25.2s, v28.2s + adcs x11, x16, x16 + umull v21.2d, v25.2s, v4.2s + mov x17, v3.d[0] + umull v18.2d, v6.2s, v28.2s + adc x16, x8, xzr + uaddlp v16.2d, v17.4s + movi v1.2d, #0xffffffff + subs x13, x13, x12 + usra v31.2d, v21.2d, #32 + sbc x8, x12, xzr + adds x17, x17, x1 + mul x1, x4, x4 + shl v28.2d, v16.2d, #32 + mov x3, v3.d[1] + adcs x14, x7, x14 + extr x7, x8, x13, #32 + adcs x13, x3, x15 + and v3.16b, v31.16b, v1.16b + adcs x11, x1, x11 + lsr x1, x8, #32 + umlal v3.2d, v6.2s, v4.2s + usra v18.2d, v31.2d, #32 + adc x3, x16, xzr + adds x1, x1, x12 + umlal v28.2d, v25.2s, v4.2s + adc x16, xzr, xzr + subs x15, x17, x7 + sbcs x7, x14, x1 + lsl x1, x15, #32 + sbcs x16, x13, x16 + add x8, x1, x15 + usra v18.2d, v3.2d, #32 + sbcs x14, x11, xzr + lsr x1, x8, #32 + sbcs x17, x3, xzr + sbc x11, x12, xzr + subs x13, x1, x8 + umulh x12, x4, x10 + sbc x1, x8, xzr + extr x13, x1, x13, #32 + lsr x1, x1, #32 + adds x15, x1, x8 + adc x1, xzr, xzr + subs x7, x7, x13 + sbcs x13, x16, x15 + lsl x3, x7, #32 + umulh x16, x2, x5 + sbcs x15, x14, x1 + add x7, x3, x7 + sbcs x3, x17, xzr + lsr x1, x7, #32 + sbcs x14, x11, xzr + sbc x11, x8, xzr + subs x8, x1, x7 + sbc x1, x7, xzr + extr x8, x1, x8, #32 + lsr x1, x1, #32 + adds x1, x1, x7 + adc x17, xzr, xzr + subs x13, x13, x8 + umulh x8, x9, x6 + sbcs x1, x15, x1 + sbcs x15, x3, x17 + sbcs x3, x14, xzr + mul x17, x2, x5 + sbcs x11, x11, xzr + stp x13, x1, [sp, #144] + sbc x14, x7, xzr + mul x7, x4, x10 + subs x1, x9, x2 + stp x15, x3, [sp, #160] + csetm x15, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + stp x11, x14, [sp, #176] + mul x14, x9, x6 + adds x17, x8, x17 + adcs x7, x16, x7 + adc x13, x12, xzr + subs x12, x5, x6 + cneg x3, x12, cc // cc = lo, ul, last + cinv x16, x15, cc // cc = lo, ul, last + mul x8, x1, x3 + umulh x1, x1, x3 + eor x12, x8, x16 + adds x11, x17, x14 + adcs x3, x7, x17 + adcs x15, x13, x7 + adc x8, x13, xzr + adds x3, x3, x14 + adcs x15, x15, x17 + adcs x17, x8, x7 + eor x1, x1, x16 + adc x13, x13, xzr + subs x9, x9, x4 + csetm x8, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x4, x2, x4 + cneg x4, x4, cc // cc = lo, ul, last + csetm x7, cc // cc = lo, ul, last + subs x2, x10, x6 + cinv x8, x8, cc // cc = lo, ul, last + cneg x2, x2, cc // cc = lo, ul, last + cmn x16, #0x1 + adcs x11, x11, x12 + mul x12, x9, x2 + adcs x3, x3, x1 + adcs x15, x15, x16 + umulh x9, x9, x2 + adcs x17, x17, x16 + adc x13, x13, x16 + subs x1, x10, x5 + cinv x2, x7, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + eor x9, x9, x8 + cmn x8, #0x1 + eor x7, x12, x8 + mul x12, x4, x1 + adcs x3, x3, x7 + adcs x7, x15, x9 + adcs x15, x17, x8 + ldp x9, x17, [sp, #160] + umulh x4, x4, x1 + adc x8, x13, x8 + cmn x2, #0x1 + eor x1, x12, x2 + adcs x1, x7, x1 + ldp x7, x16, [sp, #144] + eor x12, x4, x2 + adcs x4, x15, x12 + ldp x15, x12, [sp, #176] + adc x8, x8, x2 + adds x13, x14, x14 + umulh x14, x5, x10 + adcs x2, x11, x11 + adcs x3, x3, x3 + adcs x1, x1, x1 + adcs x4, x4, x4 + adcs x11, x8, x8 + adc x8, xzr, xzr + adds x13, x13, x7 + adcs x2, x2, x16 + mul x16, x5, x10 + adcs x3, x3, x9 + adcs x1, x1, x17 + umulh x5, x5, x5 + lsl x9, x13, #32 + add x9, x9, x13 + adcs x4, x4, x15 + mov x13, v28.d[1] + adcs x15, x11, x12 + lsr x7, x9, #32 + adc x11, x8, xzr + subs x7, x7, x9 + umulh x10, x10, x10 + sbc x17, x9, xzr + extr x7, x17, x7, #32 + lsr x17, x17, #32 + adds x17, x17, x9 + adc x12, xzr, xzr + subs x8, x2, x7 + sbcs x17, x3, x17 + lsl x7, x8, #32 + sbcs x2, x1, x12 + add x3, x7, x8 + sbcs x12, x4, xzr + lsr x1, x3, #32 + sbcs x7, x15, xzr + sbc x15, x9, xzr + subs x1, x1, x3 + sbc x4, x3, xzr + lsr x9, x4, #32 + extr x8, x4, x1, #32 + adds x9, x9, x3 + adc x4, xzr, xzr + subs x1, x17, x8 + lsl x17, x1, #32 + sbcs x8, x2, x9 + sbcs x9, x12, x4 + add x17, x17, x1 + mov x1, v18.d[1] + lsr x2, x17, #32 + sbcs x7, x7, xzr + mov x12, v18.d[0] + sbcs x15, x15, xzr + sbc x3, x3, xzr + subs x4, x2, x17 + sbc x2, x17, xzr + adds x12, x13, x12 + adcs x16, x16, x1 + lsr x13, x2, #32 + extr x1, x2, x4, #32 + adc x2, x14, xzr + adds x4, x13, x17 + mul x13, x6, x6 + adc x14, xzr, xzr + subs x1, x8, x1 + sbcs x4, x9, x4 + mov x9, v28.d[0] + sbcs x7, x7, x14 + sbcs x8, x15, xzr + sbcs x3, x3, xzr + sbc x14, x17, xzr + adds x17, x9, x9 + adcs x12, x12, x12 + mov x15, v19.d[0] + adcs x9, x16, x16 + umulh x6, x6, x6 + adcs x16, x2, x2 + adc x2, xzr, xzr + adds x11, x11, x8 + adcs x3, x3, xzr + adcs x14, x14, xzr + adcs x8, xzr, xzr + adds x13, x1, x13 + mov x1, v19.d[1] + adcs x6, x4, x6 + mov x4, #0xffffffff // #4294967295 + adcs x15, x7, x15 + adcs x7, x11, x5 + adcs x1, x3, x1 + adcs x14, x14, x10 + adc x11, x8, xzr + adds x6, x6, x17 + adcs x8, x15, x12 + adcs x3, x7, x9 + adcs x15, x1, x16 + mov x16, #0xffffffff00000001 // #-4294967295 + adcs x14, x14, x2 + mov x2, #0x1 // #1 + adc x17, x11, xzr + cmn x13, x16 + adcs xzr, x6, x4 + adcs xzr, x8, x2 + adcs xzr, x3, xzr + adcs xzr, x15, xzr + adcs xzr, x14, xzr + adc x1, x17, xzr + neg x9, x1 + and x1, x16, x9 + adds x11, x13, x1 + and x13, x4, x9 + adcs x5, x6, x13 + and x1, x2, x9 + adcs x7, x8, x1 + stp x11, x5, [sp, #144] + adcs x11, x3, xzr + adcs x2, x15, xzr + stp x7, x11, [sp, #160] + adc x17, x14, xzr + stp x2, x17, [sp, #176] + mov x0, sp + ldr q1, [sp, #48] + ldp x9, x2, [sp, #48] + ldr q0, [sp, #48] + ldp x4, x6, [sp, #64] + rev64 v21.4s, v1.4s + uzp2 v28.4s, v1.4s, v1.4s + umulh x7, x9, x2 + xtn v17.2s, v1.2d + mul v27.4s, v21.4s, v0.4s + ldr q20, [sp, #80] + xtn v30.2s, v0.2d + ldr q1, [sp, #80] + uzp2 v31.4s, v0.4s, v0.4s + ldp x5, x10, [sp, #80] + umulh x8, x9, x4 + uaddlp v3.2d, v27.4s + umull v16.2d, v30.2s, v17.2s + mul x16, x9, x4 + umull v27.2d, v30.2s, v28.2s + shrn v0.2s, v20.2d, #32 + xtn v7.2s, v20.2d + shl v20.2d, v3.2d, #32 + umull v3.2d, v31.2s, v28.2s + mul x3, x2, x4 + umlal v20.2d, v30.2s, v17.2s + umull v22.2d, v7.2s, v0.2s + usra v27.2d, v16.2d, #32 + umulh x11, x2, x4 + movi v21.2d, #0xffffffff + uzp2 v28.4s, v1.4s, v1.4s + adds x15, x16, x7 + and v5.16b, v27.16b, v21.16b + adcs x3, x3, x8 + usra v3.2d, v27.2d, #32 + dup v29.2d, x6 + adcs x16, x11, xzr + mov x14, v20.d[0] + umlal v5.2d, v31.2s, v17.2s + mul x8, x9, x2 + mov x7, v20.d[1] + shl v19.2d, v22.2d, #33 + xtn v25.2s, v29.2d + rev64 v31.4s, v1.4s + lsl x13, x14, #32 + uzp2 v6.4s, v29.4s, v29.4s + umlal v19.2d, v7.2s, v7.2s + usra v3.2d, v5.2d, #32 + adds x1, x8, x8 + umulh x8, x4, x4 + add x12, x13, x14 + mul v17.4s, v31.4s, v29.4s + xtn v4.2s, v1.2d + adcs x14, x15, x15 + lsr x13, x12, #32 + adcs x15, x3, x3 + umull v31.2d, v25.2s, v28.2s + adcs x11, x16, x16 + umull v21.2d, v25.2s, v4.2s + mov x17, v3.d[0] + umull v18.2d, v6.2s, v28.2s + adc x16, x8, xzr + uaddlp v16.2d, v17.4s + movi v1.2d, #0xffffffff + subs x13, x13, x12 + usra v31.2d, v21.2d, #32 + sbc x8, x12, xzr + adds x17, x17, x1 + mul x1, x4, x4 + shl v28.2d, v16.2d, #32 + mov x3, v3.d[1] + adcs x14, x7, x14 + extr x7, x8, x13, #32 + adcs x13, x3, x15 + and v3.16b, v31.16b, v1.16b + adcs x11, x1, x11 + lsr x1, x8, #32 + umlal v3.2d, v6.2s, v4.2s + usra v18.2d, v31.2d, #32 + adc x3, x16, xzr + adds x1, x1, x12 + umlal v28.2d, v25.2s, v4.2s + adc x16, xzr, xzr + subs x15, x17, x7 + sbcs x7, x14, x1 + lsl x1, x15, #32 + sbcs x16, x13, x16 + add x8, x1, x15 + usra v18.2d, v3.2d, #32 + sbcs x14, x11, xzr + lsr x1, x8, #32 + sbcs x17, x3, xzr + sbc x11, x12, xzr + subs x13, x1, x8 + umulh x12, x4, x10 + sbc x1, x8, xzr + extr x13, x1, x13, #32 + lsr x1, x1, #32 + adds x15, x1, x8 + adc x1, xzr, xzr + subs x7, x7, x13 + sbcs x13, x16, x15 + lsl x3, x7, #32 + umulh x16, x2, x5 + sbcs x15, x14, x1 + add x7, x3, x7 + sbcs x3, x17, xzr + lsr x1, x7, #32 + sbcs x14, x11, xzr + sbc x11, x8, xzr + subs x8, x1, x7 + sbc x1, x7, xzr + extr x8, x1, x8, #32 + lsr x1, x1, #32 + adds x1, x1, x7 + adc x17, xzr, xzr + subs x13, x13, x8 + umulh x8, x9, x6 + sbcs x1, x15, x1 + sbcs x15, x3, x17 + sbcs x3, x14, xzr + mul x17, x2, x5 + sbcs x11, x11, xzr + stp x13, x1, [x0] + sbc x14, x7, xzr + mul x7, x4, x10 + subs x1, x9, x2 + stp x15, x3, [x0, #16] + csetm x15, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + stp x11, x14, [x0, #32] + mul x14, x9, x6 + adds x17, x8, x17 + adcs x7, x16, x7 + adc x13, x12, xzr + subs x12, x5, x6 + cneg x3, x12, cc // cc = lo, ul, last + cinv x16, x15, cc // cc = lo, ul, last + mul x8, x1, x3 + umulh x1, x1, x3 + eor x12, x8, x16 + adds x11, x17, x14 + adcs x3, x7, x17 + adcs x15, x13, x7 + adc x8, x13, xzr + adds x3, x3, x14 + adcs x15, x15, x17 + adcs x17, x8, x7 + eor x1, x1, x16 + adc x13, x13, xzr + subs x9, x9, x4 + csetm x8, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x4, x2, x4 + cneg x4, x4, cc // cc = lo, ul, last + csetm x7, cc // cc = lo, ul, last + subs x2, x10, x6 + cinv x8, x8, cc // cc = lo, ul, last + cneg x2, x2, cc // cc = lo, ul, last + cmn x16, #0x1 + adcs x11, x11, x12 + mul x12, x9, x2 + adcs x3, x3, x1 + adcs x15, x15, x16 + umulh x9, x9, x2 + adcs x17, x17, x16 + adc x13, x13, x16 + subs x1, x10, x5 + cinv x2, x7, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + eor x9, x9, x8 + cmn x8, #0x1 + eor x7, x12, x8 + mul x12, x4, x1 + adcs x3, x3, x7 + adcs x7, x15, x9 + adcs x15, x17, x8 + ldp x9, x17, [x0, #16] + umulh x4, x4, x1 + adc x8, x13, x8 + cmn x2, #0x1 + eor x1, x12, x2 + adcs x1, x7, x1 + ldp x7, x16, [x0] + eor x12, x4, x2 + adcs x4, x15, x12 + ldp x15, x12, [x0, #32] + adc x8, x8, x2 + adds x13, x14, x14 + umulh x14, x5, x10 + adcs x2, x11, x11 + adcs x3, x3, x3 + adcs x1, x1, x1 + adcs x4, x4, x4 + adcs x11, x8, x8 + adc x8, xzr, xzr + adds x13, x13, x7 + adcs x2, x2, x16 + mul x16, x5, x10 + adcs x3, x3, x9 + adcs x1, x1, x17 + umulh x5, x5, x5 + lsl x9, x13, #32 + add x9, x9, x13 + adcs x4, x4, x15 + mov x13, v28.d[1] + adcs x15, x11, x12 + lsr x7, x9, #32 + adc x11, x8, xzr + subs x7, x7, x9 + umulh x10, x10, x10 + sbc x17, x9, xzr + extr x7, x17, x7, #32 + lsr x17, x17, #32 + adds x17, x17, x9 + adc x12, xzr, xzr + subs x8, x2, x7 + sbcs x17, x3, x17 + lsl x7, x8, #32 + sbcs x2, x1, x12 + add x3, x7, x8 + sbcs x12, x4, xzr + lsr x1, x3, #32 + sbcs x7, x15, xzr + sbc x15, x9, xzr + subs x1, x1, x3 + sbc x4, x3, xzr + lsr x9, x4, #32 + extr x8, x4, x1, #32 + adds x9, x9, x3 + adc x4, xzr, xzr + subs x1, x17, x8 + lsl x17, x1, #32 + sbcs x8, x2, x9 + sbcs x9, x12, x4 + add x17, x17, x1 + mov x1, v18.d[1] + lsr x2, x17, #32 + sbcs x7, x7, xzr + mov x12, v18.d[0] + sbcs x15, x15, xzr + sbc x3, x3, xzr + subs x4, x2, x17 + sbc x2, x17, xzr + adds x12, x13, x12 + adcs x16, x16, x1 + lsr x13, x2, #32 + extr x1, x2, x4, #32 + adc x2, x14, xzr + adds x4, x13, x17 + mul x13, x6, x6 + adc x14, xzr, xzr + subs x1, x8, x1 + sbcs x4, x9, x4 + mov x9, v28.d[0] + sbcs x7, x7, x14 + sbcs x8, x15, xzr + sbcs x3, x3, xzr + sbc x14, x17, xzr + adds x17, x9, x9 + adcs x12, x12, x12 + mov x15, v19.d[0] + adcs x9, x16, x16 + umulh x6, x6, x6 + adcs x16, x2, x2 + adc x2, xzr, xzr + adds x11, x11, x8 + adcs x3, x3, xzr + adcs x14, x14, xzr + adcs x8, xzr, xzr + adds x13, x1, x13 + mov x1, v19.d[1] + adcs x6, x4, x6 + mov x4, #0xffffffff // #4294967295 + adcs x15, x7, x15 + adcs x7, x11, x5 + adcs x1, x3, x1 + adcs x14, x14, x10 + adc x11, x8, xzr + adds x6, x6, x17 + adcs x8, x15, x12 + adcs x3, x7, x9 + adcs x15, x1, x16 + mov x16, #0xffffffff00000001 // #-4294967295 + adcs x14, x14, x2 + mov x2, #0x1 // #1 + adc x17, x11, xzr + cmn x13, x16 + adcs xzr, x6, x4 + adcs xzr, x8, x2 + adcs xzr, x3, xzr + adcs xzr, x15, xzr + adcs xzr, x14, xzr + adc x1, x17, xzr + neg x9, x1 + and x1, x16, x9 + adds x11, x13, x1 + and x13, x4, x9 + adcs x5, x6, x13 + and x1, x2, x9 + adcs x7, x8, x1 + stp x11, x5, [x0] + adcs x11, x3, xzr + adcs x2, x15, xzr + stp x7, x11, [x0, #16] + adc x17, x14, xzr + stp x2, x17, [x0, #32] + ldr q3, [sp, #144] + ldr q25, [sp, #192] + ldp x13, x23, [sp, #192] + ldp x3, x21, [sp, #144] + rev64 v23.4s, v25.4s + uzp1 v17.4s, v25.4s, v3.4s + umulh x15, x3, x13 + mul v6.4s, v23.4s, v3.4s + uzp1 v3.4s, v3.4s, v3.4s + ldr q27, [sp, #224] + ldp x8, x24, [sp, #160] + subs x6, x3, x21 + ldr q0, [sp, #176] + movi v23.2d, #0xffffffff + csetm x10, cc // cc = lo, ul, last + umulh x19, x21, x23 + rev64 v4.4s, v27.4s + uzp2 v25.4s, v27.4s, v27.4s + cneg x4, x6, cc // cc = lo, ul, last + subs x7, x23, x13 + xtn v22.2s, v0.2d + xtn v24.2s, v27.2d + cneg x20, x7, cc // cc = lo, ul, last + ldp x6, x14, [sp, #208] + mul v27.4s, v4.4s, v0.4s + uaddlp v20.2d, v6.4s + cinv x5, x10, cc // cc = lo, ul, last + mul x16, x4, x20 + uzp2 v6.4s, v0.4s, v0.4s + umull v21.2d, v22.2s, v25.2s + shl v0.2d, v20.2d, #32 + umlal v0.2d, v3.2s, v17.2s + mul x22, x8, x6 + umull v1.2d, v6.2s, v25.2s + subs x12, x3, x8 + umull v20.2d, v22.2s, v24.2s + cneg x17, x12, cc // cc = lo, ul, last + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc // cc = lo, ul, last + usra v21.2d, v20.2d, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2d, v21.2d, #32 + adds x22, x15, x7 + and v26.16b, v21.16b, v23.16b + adcs x16, x12, x15 + uaddlp v25.2d, v27.4s + adcs x9, x19, x12 + umlal v26.2d, v6.2s, v24.2s + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2d, v25.2d, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc // cc = lo, ul, last + cinv x10, x10, cc // cc = lo, ul, last + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc // cc = lo, ul, last + eor x19, x19, x10 + csetm x4, cc // cc = lo, ul, last + subs x16, x6, x23 + cneg x16, x16, cc // cc = lo, ul, last + umlal v27.2d, v22.2s, v24.2s + mul x15, x20, x16 + cinv x4, x4, cc // cc = lo, ul, last + cmn x10, #0x1 + usra v1.2d, v26.2d, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [sp, #224] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [sp, #176] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x16, x4, x16 + mov x4, v27.d[0] + sbcs x11, x20, x11 + sbcs x20, x9, x12 + stp x16, x11, [sp, #192] + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [sp, #208] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc // cc = lo, ul, last + csetm x2, cc // cc = lo, ul, last + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc // cc = lo, ul, last + cneg x19, x19, cc // cc = lo, ul, last + stp x9, x20, [sp, #224] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc // cc = lo, ul, last + csetm x12, cc // cc = lo, ul, last + subs x9, x17, x14 + cinv x12, x12, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc // cc = lo, ul, last + cneg x24, x10, cc // cc = lo, ul, last + subs x10, x17, x15 + cinv x7, x7, cc // cc = lo, ul, last + cneg x10, x10, cc // cc = lo, ul, last + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + ldp x20, x9, [sp, #192] + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [sp, #208] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [sp, #224] + adds x20, x22, x20 + mul x10, x13, x14 + adcs x11, x11, x9 + eor x9, x8, x21 + adcs x21, x19, x17 + stp x20, x11, [sp, #192] + adcs x12, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + stp x21, x12, [sp, #208] + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [sp, #224] + cneg x3, x21, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc // cc = lo, ul, last + csetm x16, cc // cc = lo, ul, last + subs x21, x6, x15 + cneg x22, x21, cc // cc = lo, ul, last + cinv x21, x24, cc // cc = lo, ul, last + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + subs x20, x14, x15 + cinv x24, x24, cc // cc = lo, ul, last + mul x22, x3, x22 + cneg x3, x20, cc // cc = lo, ul, last + subs x13, x6, x14 + cneg x20, x13, cc // cc = lo, ul, last + cinv x15, x16, cc // cc = lo, ul, last + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe // #-2 + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [sp, #192] + ldp x21, x12, [sp, #208] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [sp, #224] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x21 + eor x1, x22, x9 + adcs x24, x23, x12 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x21 + adcs x15, x17, x12 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff // #4294967295 + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc // cc = lo, ul, last + and x11, x16, x23 + and x20, x14, x23 + adds x22, x6, x20 + eor x3, x20, x23 + adcs x5, x7, x3 + adcs x14, x24, x11 + stp x22, x5, [sp, #192] + adcs x5, x13, x23 + adcs x21, x8, x23 + stp x14, x5, [sp, #208] + adc x12, x15, x23 + stp x21, x12, [sp, #224] + ldr q3, [sp, #144] + ldr q25, [sp, #96] + ldp x13, x23, [sp, #96] + ldp x3, x21, [sp, #144] + rev64 v23.4s, v25.4s + uzp1 v17.4s, v25.4s, v3.4s + umulh x15, x3, x13 + mul v6.4s, v23.4s, v3.4s + uzp1 v3.4s, v3.4s, v3.4s + ldr q27, [sp, #128] + ldp x8, x24, [sp, #160] + subs x6, x3, x21 + ldr q0, [sp, #176] + movi v23.2d, #0xffffffff + csetm x10, cc // cc = lo, ul, last + umulh x19, x21, x23 + rev64 v4.4s, v27.4s + uzp2 v25.4s, v27.4s, v27.4s + cneg x4, x6, cc // cc = lo, ul, last + subs x7, x23, x13 + xtn v22.2s, v0.2d + xtn v24.2s, v27.2d + cneg x20, x7, cc // cc = lo, ul, last + ldp x6, x14, [sp, #112] + mul v27.4s, v4.4s, v0.4s + uaddlp v20.2d, v6.4s + cinv x5, x10, cc // cc = lo, ul, last + mul x16, x4, x20 + uzp2 v6.4s, v0.4s, v0.4s + umull v21.2d, v22.2s, v25.2s + shl v0.2d, v20.2d, #32 + umlal v0.2d, v3.2s, v17.2s + mul x22, x8, x6 + umull v1.2d, v6.2s, v25.2s + subs x12, x3, x8 + umull v20.2d, v22.2s, v24.2s + cneg x17, x12, cc // cc = lo, ul, last + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc // cc = lo, ul, last + usra v21.2d, v20.2d, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2d, v21.2d, #32 + adds x22, x15, x7 + and v26.16b, v21.16b, v23.16b + adcs x16, x12, x15 + uaddlp v25.2d, v27.4s + adcs x9, x19, x12 + umlal v26.2d, v6.2s, v24.2s + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2d, v25.2d, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc // cc = lo, ul, last + cinv x10, x10, cc // cc = lo, ul, last + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc // cc = lo, ul, last + eor x19, x19, x10 + csetm x4, cc // cc = lo, ul, last + subs x16, x6, x23 + cneg x16, x16, cc // cc = lo, ul, last + umlal v27.2d, v22.2s, v24.2s + mul x15, x20, x16 + cinv x4, x4, cc // cc = lo, ul, last + cmn x10, #0x1 + usra v1.2d, v26.2d, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [sp, #128] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [sp, #176] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x16, x4, x16 + mov x4, v27.d[0] + sbcs x11, x20, x11 + sbcs x20, x9, x12 + stp x16, x11, [sp, #96] + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [sp, #112] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc // cc = lo, ul, last + csetm x2, cc // cc = lo, ul, last + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc // cc = lo, ul, last + cneg x19, x19, cc // cc = lo, ul, last + stp x9, x20, [sp, #128] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc // cc = lo, ul, last + csetm x12, cc // cc = lo, ul, last + subs x9, x17, x14 + cinv x12, x12, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc // cc = lo, ul, last + cneg x24, x10, cc // cc = lo, ul, last + subs x10, x17, x15 + cinv x7, x7, cc // cc = lo, ul, last + cneg x10, x10, cc // cc = lo, ul, last + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + ldp x20, x9, [sp, #96] + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [sp, #112] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [sp, #128] + adds x20, x22, x20 + mul x10, x13, x14 + adcs x11, x11, x9 + eor x9, x8, x21 + adcs x21, x19, x17 + stp x20, x11, [sp, #96] + adcs x12, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + stp x21, x12, [sp, #112] + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [sp, #128] + cneg x3, x21, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc // cc = lo, ul, last + csetm x16, cc // cc = lo, ul, last + subs x21, x6, x15 + cneg x22, x21, cc // cc = lo, ul, last + cinv x21, x24, cc // cc = lo, ul, last + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + subs x20, x14, x15 + cinv x24, x24, cc // cc = lo, ul, last + mul x22, x3, x22 + cneg x3, x20, cc // cc = lo, ul, last + subs x13, x6, x14 + cneg x20, x13, cc // cc = lo, ul, last + cinv x15, x16, cc // cc = lo, ul, last + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe // #-2 + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [sp, #96] + ldp x21, x12, [sp, #112] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [sp, #128] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x21 + eor x1, x22, x9 + adcs x24, x23, x12 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x21 + adcs x15, x17, x12 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff // #4294967295 + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc // cc = lo, ul, last + and x11, x16, x23 + and x20, x14, x23 + adds x22, x6, x20 + eor x3, x20, x23 + adcs x5, x7, x3 + adcs x2, x24, x11 + stp x22, x5, [sp, #96] + adcs x11, x13, x23 + adcs x12, x8, x23 + stp x2, x11, [sp, #112] + adc x13, x15, x23 + stp x12, x13, [sp, #128] + mov x0, sp + mov x1, sp + ldp x5, x6, [x1] + ldp x4, x3, [sp, #192] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [x1, #16] + ldp x4, x3, [sp, #208] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [x1, #32] + ldp x4, x3, [sp, #224] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, cc // cc = lo, ul, last + mov x4, #0xffffffff // #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #0xfffffffffffffffe // #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [x0] + stp x7, x8, [x0, #16] + stp x9, x10, [x0, #32] + ldp x5, x6, [sp, #96] + ldp x4, x3, [sp, #192] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x4, x3, [sp, #208] + sbcs x7, x2, x4 + sbcs x8, x11, x3 + ldp x4, x3, [sp, #224] + sbcs x9, x12, x4 + sbcs x10, x13, x3 + csetm x3, cc // cc = lo, ul, last + mov x4, #0xffffffff // #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #0xfffffffffffffffe // #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [sp, #144] + stp x7, x8, [sp, #160] + stp x9, x10, [sp, #176] + ldr q3, [sp, #240] + ldr q25, [x25, #96] + ldp x13, x23, [x25, #96] + ldp x3, x21, [sp, #240] + rev64 v23.4s, v25.4s + uzp1 v17.4s, v25.4s, v3.4s + umulh x15, x3, x13 + mul v6.4s, v23.4s, v3.4s + uzp1 v3.4s, v3.4s, v3.4s + ldr q27, [x25, #128] + ldp x8, x24, [sp, #256] + subs x6, x3, x21 + ldr q0, [sp, #272] + movi v23.2d, #0xffffffff + csetm x10, cc // cc = lo, ul, last + umulh x19, x21, x23 + rev64 v4.4s, v27.4s + uzp2 v25.4s, v27.4s, v27.4s + cneg x4, x6, cc // cc = lo, ul, last + subs x7, x23, x13 + xtn v22.2s, v0.2d + xtn v24.2s, v27.2d + cneg x20, x7, cc // cc = lo, ul, last + ldp x6, x14, [x25, #112] + mul v27.4s, v4.4s, v0.4s + uaddlp v20.2d, v6.4s + cinv x5, x10, cc // cc = lo, ul, last + mul x16, x4, x20 + uzp2 v6.4s, v0.4s, v0.4s + umull v21.2d, v22.2s, v25.2s + shl v0.2d, v20.2d, #32 + umlal v0.2d, v3.2s, v17.2s + mul x22, x8, x6 + umull v1.2d, v6.2s, v25.2s + subs x12, x3, x8 + umull v20.2d, v22.2s, v24.2s + cneg x17, x12, cc // cc = lo, ul, last + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc // cc = lo, ul, last + usra v21.2d, v20.2d, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2d, v21.2d, #32 + adds x22, x15, x7 + and v26.16b, v21.16b, v23.16b + adcs x16, x12, x15 + uaddlp v25.2d, v27.4s + adcs x9, x19, x12 + umlal v26.2d, v6.2s, v24.2s + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2d, v25.2d, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc // cc = lo, ul, last + cinv x10, x10, cc // cc = lo, ul, last + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc // cc = lo, ul, last + eor x19, x19, x10 + csetm x4, cc // cc = lo, ul, last + subs x16, x6, x23 + cneg x16, x16, cc // cc = lo, ul, last + umlal v27.2d, v22.2s, v24.2s + mul x15, x20, x16 + cinv x4, x4, cc // cc = lo, ul, last + cmn x10, #0x1 + usra v1.2d, v26.2d, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [x25, #128] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [sp, #272] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x16, x4, x16 + mov x4, v27.d[0] + sbcs x11, x20, x11 + sbcs x20, x9, x12 + stp x16, x11, [sp, #240] + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [sp, #256] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc // cc = lo, ul, last + csetm x2, cc // cc = lo, ul, last + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc // cc = lo, ul, last + cneg x19, x19, cc // cc = lo, ul, last + stp x9, x20, [sp, #272] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc // cc = lo, ul, last + csetm x12, cc // cc = lo, ul, last + subs x9, x17, x14 + cinv x12, x12, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc // cc = lo, ul, last + cneg x24, x10, cc // cc = lo, ul, last + subs x10, x17, x15 + cinv x7, x7, cc // cc = lo, ul, last + cneg x10, x10, cc // cc = lo, ul, last + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + ldp x20, x9, [sp, #240] + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [sp, #256] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [sp, #272] + adds x20, x22, x20 + mul x10, x13, x14 + adcs x11, x11, x9 + eor x9, x8, x21 + adcs x21, x19, x17 + stp x20, x11, [sp, #240] + adcs x12, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + stp x21, x12, [sp, #256] + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [sp, #272] + cneg x3, x21, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc // cc = lo, ul, last + csetm x16, cc // cc = lo, ul, last + subs x21, x6, x15 + cneg x22, x21, cc // cc = lo, ul, last + cinv x21, x24, cc // cc = lo, ul, last + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + subs x20, x14, x15 + cinv x24, x24, cc // cc = lo, ul, last + mul x22, x3, x22 + cneg x3, x20, cc // cc = lo, ul, last + subs x13, x6, x14 + cneg x20, x13, cc // cc = lo, ul, last + cinv x15, x16, cc // cc = lo, ul, last + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe // #-2 + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [sp, #240] + ldp x21, x12, [sp, #256] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [sp, #272] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x21 + eor x1, x22, x9 + adcs x24, x23, x12 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x21 + adcs x15, x17, x12 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff // #4294967295 + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc // cc = lo, ul, last + and x11, x16, x23 + and x20, x14, x23 + adds x22, x6, x20 + eor x3, x20, x23 + adcs x5, x7, x3 + adcs x14, x24, x11 + stp x22, x5, [sp, #240] + adcs x5, x13, x23 + adcs x21, x8, x23 + stp x14, x5, [sp, #256] + adc x12, x15, x23 + stp x21, x12, [sp, #272] + mov x0, sp + mov x1, sp + ldp x5, x6, [x1] + ldp x4, x3, [sp, #96] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [x1, #16] + ldp x4, x3, [sp, #112] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [x1, #32] + ldp x4, x3, [sp, #128] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, cc // cc = lo, ul, last + mov x4, #0xffffffff // #4294967295 + and x4, x4, x3 + adds x2, x5, x4 + eor x4, x4, x3 + adcs x11, x6, x4 + mov x4, #0xfffffffffffffffe // #-2 + and x4, x4, x3 + adcs x4, x7, x4 + adcs x12, x8, x3 + adcs x13, x9, x3 + adc x3, x10, x3 + stp x2, x11, [x0] + stp x4, x12, [x0, #16] + stp x13, x3, [x0, #32] + ldp x5, x6, [sp, #192] + subs x5, x5, x2 + sbcs x6, x6, x11 + ldp x7, x8, [sp, #208] + sbcs x7, x7, x4 + sbcs x8, x8, x12 + ldp x9, x10, [sp, #224] + sbcs x9, x9, x13 + sbcs x10, x10, x3 + csetm x3, cc // cc = lo, ul, last + mov x4, #0xffffffff // #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #0xfffffffffffffffe // #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [sp, #192] + stp x7, x8, [sp, #208] + stp x9, x10, [sp, #224] + ldr q3, [sp, #144] + ldr q25, [sp, #288] + ldp x13, x23, [sp, #288] + ldp x3, x21, [sp, #144] + rev64 v23.4s, v25.4s + uzp1 v17.4s, v25.4s, v3.4s + umulh x15, x3, x13 + mul v6.4s, v23.4s, v3.4s + uzp1 v3.4s, v3.4s, v3.4s + ldr q27, [sp, #320] + ldp x8, x24, [sp, #160] + subs x6, x3, x21 + ldr q0, [sp, #176] + movi v23.2d, #0xffffffff + csetm x10, cc // cc = lo, ul, last + umulh x19, x21, x23 + rev64 v4.4s, v27.4s + uzp2 v25.4s, v27.4s, v27.4s + cneg x4, x6, cc // cc = lo, ul, last + subs x7, x23, x13 + xtn v22.2s, v0.2d + xtn v24.2s, v27.2d + cneg x20, x7, cc // cc = lo, ul, last + ldp x6, x14, [sp, #304] + mul v27.4s, v4.4s, v0.4s + uaddlp v20.2d, v6.4s + cinv x5, x10, cc // cc = lo, ul, last + mul x16, x4, x20 + uzp2 v6.4s, v0.4s, v0.4s + umull v21.2d, v22.2s, v25.2s + shl v0.2d, v20.2d, #32 + umlal v0.2d, v3.2s, v17.2s + mul x22, x8, x6 + umull v1.2d, v6.2s, v25.2s + subs x12, x3, x8 + umull v20.2d, v22.2s, v24.2s + cneg x17, x12, cc // cc = lo, ul, last + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc // cc = lo, ul, last + usra v21.2d, v20.2d, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2d, v21.2d, #32 + adds x22, x15, x7 + and v26.16b, v21.16b, v23.16b + adcs x16, x12, x15 + uaddlp v25.2d, v27.4s + adcs x9, x19, x12 + umlal v26.2d, v6.2s, v24.2s + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2d, v25.2d, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc // cc = lo, ul, last + cinv x10, x10, cc // cc = lo, ul, last + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc // cc = lo, ul, last + eor x19, x19, x10 + csetm x4, cc // cc = lo, ul, last + subs x16, x6, x23 + cneg x16, x16, cc // cc = lo, ul, last + umlal v27.2d, v22.2s, v24.2s + mul x15, x20, x16 + cinv x4, x4, cc // cc = lo, ul, last + cmn x10, #0x1 + usra v1.2d, v26.2d, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [sp, #320] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [sp, #176] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x16, x4, x16 + mov x4, v27.d[0] + sbcs x11, x20, x11 + sbcs x20, x9, x12 + stp x16, x11, [sp, #144] + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [sp, #160] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc // cc = lo, ul, last + csetm x2, cc // cc = lo, ul, last + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc // cc = lo, ul, last + cneg x19, x19, cc // cc = lo, ul, last + stp x9, x20, [sp, #176] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc // cc = lo, ul, last + csetm x12, cc // cc = lo, ul, last + subs x9, x17, x14 + cinv x12, x12, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc // cc = lo, ul, last + cneg x24, x10, cc // cc = lo, ul, last + subs x10, x17, x15 + cinv x7, x7, cc // cc = lo, ul, last + cneg x10, x10, cc // cc = lo, ul, last + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + ldp x20, x9, [sp, #144] + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [sp, #160] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [sp, #176] + adds x20, x22, x20 + mul x10, x13, x14 + adcs x11, x11, x9 + eor x9, x8, x21 + adcs x21, x19, x17 + stp x20, x11, [sp, #144] + adcs x12, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + stp x21, x12, [sp, #160] + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [sp, #176] + cneg x3, x21, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc // cc = lo, ul, last + csetm x16, cc // cc = lo, ul, last + subs x21, x6, x15 + cneg x22, x21, cc // cc = lo, ul, last + cinv x21, x24, cc // cc = lo, ul, last + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + subs x20, x14, x15 + cinv x24, x24, cc // cc = lo, ul, last + mul x22, x3, x22 + cneg x3, x20, cc // cc = lo, ul, last + subs x13, x6, x14 + cneg x20, x13, cc // cc = lo, ul, last + cinv x15, x16, cc // cc = lo, ul, last + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe // #-2 + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [sp, #144] + ldp x21, x12, [sp, #160] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [sp, #176] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x21 + eor x1, x22, x9 + adcs x24, x23, x12 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x21 + adcs x15, x17, x12 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff // #4294967295 + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc // cc = lo, ul, last + and x11, x16, x23 + and x20, x14, x23 + adds x22, x6, x20 + eor x3, x20, x23 + adcs x5, x7, x3 + adcs x14, x24, x11 + stp x22, x5, [sp, #144] + adcs x5, x13, x23 + adcs x21, x8, x23 + stp x14, x5, [sp, #160] + adc x12, x15, x23 + stp x21, x12, [sp, #176] + ldr q3, [sp, #240] + ldr q25, [x26, #96] + ldp x13, x23, [x26, #96] + ldp x3, x21, [sp, #240] + rev64 v23.4s, v25.4s + uzp1 v17.4s, v25.4s, v3.4s + umulh x15, x3, x13 + mul v6.4s, v23.4s, v3.4s + uzp1 v3.4s, v3.4s, v3.4s + ldr q27, [x26, #128] + ldp x8, x24, [sp, #256] + subs x6, x3, x21 + ldr q0, [sp, #272] + movi v23.2d, #0xffffffff + csetm x10, cc // cc = lo, ul, last + umulh x19, x21, x23 + rev64 v4.4s, v27.4s + uzp2 v25.4s, v27.4s, v27.4s + cneg x4, x6, cc // cc = lo, ul, last + subs x7, x23, x13 + xtn v22.2s, v0.2d + xtn v24.2s, v27.2d + cneg x20, x7, cc // cc = lo, ul, last + ldp x6, x14, [x26, #112] + mul v27.4s, v4.4s, v0.4s + uaddlp v20.2d, v6.4s + cinv x5, x10, cc // cc = lo, ul, last + mul x16, x4, x20 + uzp2 v6.4s, v0.4s, v0.4s + umull v21.2d, v22.2s, v25.2s + shl v0.2d, v20.2d, #32 + umlal v0.2d, v3.2s, v17.2s + mul x22, x8, x6 + umull v1.2d, v6.2s, v25.2s + subs x12, x3, x8 + umull v20.2d, v22.2s, v24.2s + cneg x17, x12, cc // cc = lo, ul, last + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc // cc = lo, ul, last + usra v21.2d, v20.2d, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2d, v21.2d, #32 + adds x22, x15, x7 + and v26.16b, v21.16b, v23.16b + adcs x16, x12, x15 + uaddlp v25.2d, v27.4s + adcs x9, x19, x12 + umlal v26.2d, v6.2s, v24.2s + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2d, v25.2d, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc // cc = lo, ul, last + cinv x10, x10, cc // cc = lo, ul, last + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc // cc = lo, ul, last + eor x19, x19, x10 + csetm x4, cc // cc = lo, ul, last + subs x16, x6, x23 + cneg x16, x16, cc // cc = lo, ul, last + umlal v27.2d, v22.2s, v24.2s + mul x15, x20, x16 + cinv x4, x4, cc // cc = lo, ul, last + cmn x10, #0x1 + usra v1.2d, v26.2d, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [x26, #128] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [sp, #272] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x16, x4, x16 + mov x4, v27.d[0] + sbcs x11, x20, x11 + sbcs x20, x9, x12 + stp x16, x11, [sp, #240] + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [sp, #256] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc // cc = lo, ul, last + csetm x2, cc // cc = lo, ul, last + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc // cc = lo, ul, last + cneg x19, x19, cc // cc = lo, ul, last + stp x9, x20, [sp, #272] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc // cc = lo, ul, last + csetm x12, cc // cc = lo, ul, last + subs x9, x17, x14 + cinv x12, x12, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc // cc = lo, ul, last + cneg x24, x10, cc // cc = lo, ul, last + subs x10, x17, x15 + cinv x7, x7, cc // cc = lo, ul, last + cneg x10, x10, cc // cc = lo, ul, last + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + ldp x20, x9, [sp, #240] + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [sp, #256] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [sp, #272] + adds x20, x22, x20 + mul x10, x13, x14 + adcs x11, x11, x9 + eor x9, x8, x21 + adcs x21, x19, x17 + stp x20, x11, [sp, #240] + adcs x12, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + stp x21, x12, [sp, #256] + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [sp, #272] + cneg x3, x21, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc // cc = lo, ul, last + csetm x16, cc // cc = lo, ul, last + subs x21, x6, x15 + cneg x22, x21, cc // cc = lo, ul, last + cinv x21, x24, cc // cc = lo, ul, last + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + subs x20, x14, x15 + cinv x24, x24, cc // cc = lo, ul, last + mul x22, x3, x22 + cneg x3, x20, cc // cc = lo, ul, last + subs x13, x6, x14 + cneg x20, x13, cc // cc = lo, ul, last + cinv x15, x16, cc // cc = lo, ul, last + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe // #-2 + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [sp, #240] + ldp x21, x12, [sp, #256] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [sp, #272] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x21 + eor x1, x22, x9 + adcs x24, x23, x12 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x21 + adcs x15, x17, x12 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff // #4294967295 + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc // cc = lo, ul, last + and x11, x16, x23 + and x20, x14, x23 + adds x22, x6, x20 + eor x3, x20, x23 + adcs x5, x7, x3 + adcs x14, x24, x11 + stp x22, x5, [sp, #240] + adcs x5, x13, x23 + adcs x21, x8, x23 + stp x14, x5, [sp, #256] + adc x12, x15, x23 + stp x21, x12, [sp, #272] + ldp x2, x27, [sp, #0x150] + ldr q3, [sp, #48] + ldr q25, [sp, #192] + ldp x13, x23, [sp, #192] + ldp x3, x21, [sp, #48] + rev64 v23.4s, v25.4s + uzp1 v17.4s, v25.4s, v3.4s + umulh x15, x3, x13 + mul v6.4s, v23.4s, v3.4s + uzp1 v3.4s, v3.4s, v3.4s + ldr q27, [sp, #224] + ldp x8, x24, [sp, #64] + subs x6, x3, x21 + ldr q0, [sp, #80] + movi v23.2d, #0xffffffff + csetm x10, cc // cc = lo, ul, last + umulh x19, x21, x23 + rev64 v4.4s, v27.4s + uzp2 v25.4s, v27.4s, v27.4s + cneg x4, x6, cc // cc = lo, ul, last + subs x7, x23, x13 + xtn v22.2s, v0.2d + xtn v24.2s, v27.2d + cneg x20, x7, cc // cc = lo, ul, last + ldp x6, x14, [sp, #208] + mul v27.4s, v4.4s, v0.4s + uaddlp v20.2d, v6.4s + cinv x5, x10, cc // cc = lo, ul, last + mul x16, x4, x20 + uzp2 v6.4s, v0.4s, v0.4s + umull v21.2d, v22.2s, v25.2s + shl v0.2d, v20.2d, #32 + umlal v0.2d, v3.2s, v17.2s + mul x22, x8, x6 + umull v1.2d, v6.2s, v25.2s + subs x12, x3, x8 + umull v20.2d, v22.2s, v24.2s + cneg x17, x12, cc // cc = lo, ul, last + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc // cc = lo, ul, last + usra v21.2d, v20.2d, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2d, v21.2d, #32 + adds x22, x15, x7 + and v26.16b, v21.16b, v23.16b + adcs x16, x12, x15 + uaddlp v25.2d, v27.4s + adcs x9, x19, x12 + umlal v26.2d, v6.2s, v24.2s + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2d, v25.2d, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc // cc = lo, ul, last + cinv x10, x10, cc // cc = lo, ul, last + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc // cc = lo, ul, last + eor x19, x19, x10 + csetm x4, cc // cc = lo, ul, last + subs x16, x6, x23 + cneg x16, x16, cc // cc = lo, ul, last + umlal v27.2d, v22.2s, v24.2s + mul x15, x20, x16 + cinv x4, x4, cc // cc = lo, ul, last + cmn x10, #0x1 + usra v1.2d, v26.2d, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [sp, #224] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [sp, #80] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x16, x4, x16 + mov x4, v27.d[0] + sbcs x11, x20, x11 + sbcs x20, x9, x12 + stp x16, x11, [sp, #192] + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [sp, #208] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc // cc = lo, ul, last + csetm x2, cc // cc = lo, ul, last + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc // cc = lo, ul, last + cneg x19, x19, cc // cc = lo, ul, last + stp x9, x20, [sp, #224] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc // cc = lo, ul, last + csetm x12, cc // cc = lo, ul, last + subs x9, x17, x14 + cinv x12, x12, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc // cc = lo, ul, last + cneg x24, x10, cc // cc = lo, ul, last + subs x10, x17, x15 + cinv x7, x7, cc // cc = lo, ul, last + cneg x10, x10, cc // cc = lo, ul, last + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + ldp x20, x9, [sp, #192] + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [sp, #208] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [sp, #224] + adds x20, x22, x20 + mul x10, x13, x14 + adcs x11, x11, x9 + eor x9, x8, x21 + adcs x21, x19, x17 + stp x20, x11, [sp, #192] + adcs x12, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + stp x21, x12, [sp, #208] + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [sp, #224] + cneg x3, x21, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc // cc = lo, ul, last + csetm x16, cc // cc = lo, ul, last + subs x21, x6, x15 + cneg x22, x21, cc // cc = lo, ul, last + cinv x21, x24, cc // cc = lo, ul, last + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + subs x20, x14, x15 + cinv x24, x24, cc // cc = lo, ul, last + mul x22, x3, x22 + cneg x3, x20, cc // cc = lo, ul, last + subs x13, x6, x14 + cneg x20, x13, cc // cc = lo, ul, last + cinv x15, x16, cc // cc = lo, ul, last + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe // #-2 + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [sp, #192] + ldp x21, x12, [sp, #208] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [sp, #224] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x21 + eor x1, x22, x9 + adcs x24, x23, x12 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x21 + adcs x15, x17, x12 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff // #4294967295 + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc // cc = lo, ul, last + and x11, x16, x23 + and x20, x14, x23 + adds x2, x6, x20 + eor x3, x20, x23 + adcs x6, x7, x3 + adcs x7, x24, x11 + adcs x9, x13, x23 + adcs x10, x8, x23 + adc x11, x15, x23 + ldp x4, x3, [sp, #144] + subs x5, x2, x4 + sbcs x6, x6, x3 + ldp x4, x3, [sp, #160] + sbcs x7, x7, x4 + sbcs x8, x9, x3 + ldp x4, x3, [sp, #176] + sbcs x9, x10, x4 + sbcs x10, x11, x3 + csetm x3, cc // cc = lo, ul, last + mov x4, #0xffffffff // #4294967295 + and x4, x4, x3 + adds x19, x5, x4 + eor x4, x4, x3 + adcs x24, x6, x4 + mov x4, #0xfffffffffffffffe // #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x7, x8, [sp, #208] + stp x9, x10, [sp, #224] + ldp x0, x1, [x25, #96] + ldp x2, x3, [x25, #112] + ldp x4, x5, [x25, #128] + orr x20, x0, x1 + orr x21, x2, x3 + orr x22, x4, x5 + orr x20, x20, x21 + orr x20, x20, x22 + cmp x20, xzr + cset x20, ne // ne = any + ldp x6, x7, [x26, #96] + ldp x8, x9, [x26, #112] + ldp x10, x11, [x26, #128] + orr x21, x6, x7 + orr x22, x8, x9 + orr x23, x10, x11 + orr x21, x21, x22 + orr x21, x21, x23 + cmp x21, xzr + cset x21, ne // ne = any + cmp x21, x20 + ldp x12, x13, [sp, #240] + csel x12, x0, x12, cc // cc = lo, ul, last + csel x13, x1, x13, cc // cc = lo, ul, last + csel x12, x6, x12, hi // hi = pmore + csel x13, x7, x13, hi // hi = pmore + ldp x14, x15, [sp, #256] + csel x14, x2, x14, cc // cc = lo, ul, last + csel x15, x3, x15, cc // cc = lo, ul, last + csel x14, x8, x14, hi // hi = pmore + csel x15, x9, x15, hi // hi = pmore + ldp x16, x17, [sp, #272] + csel x16, x4, x16, cc // cc = lo, ul, last + csel x17, x5, x17, cc // cc = lo, ul, last + csel x16, x10, x16, hi // hi = pmore + csel x17, x11, x17, hi // hi = pmore + ldp x20, x21, [x25] + ldp x0, x1, [sp] + csel x0, x20, x0, cc // cc = lo, ul, last + csel x1, x21, x1, cc // cc = lo, ul, last + ldp x20, x21, [x26] + csel x0, x20, x0, hi // hi = pmore + csel x1, x21, x1, hi // hi = pmore + ldp x20, x21, [x25, #16] + ldp x2, x3, [sp, #16] + csel x2, x20, x2, cc // cc = lo, ul, last + csel x3, x21, x3, cc // cc = lo, ul, last + ldp x20, x21, [x26, #16] + csel x2, x20, x2, hi // hi = pmore + csel x3, x21, x3, hi // hi = pmore + ldp x20, x21, [x25, #32] + ldp x4, x5, [sp, #32] + csel x4, x20, x4, cc // cc = lo, ul, last + csel x5, x21, x5, cc // cc = lo, ul, last + ldp x20, x21, [x26, #32] + csel x4, x20, x4, hi // hi = pmore + csel x5, x21, x5, hi // hi = pmore + ldp x20, x21, [x25, #48] + csel x6, x20, x19, cc // cc = lo, ul, last + csel x7, x21, x24, cc // cc = lo, ul, last + ldp x20, x21, [x26, #48] + csel x6, x20, x6, hi // hi = pmore + csel x7, x21, x7, hi // hi = pmore + ldp x20, x21, [x25, #64] + ldp x8, x9, [sp, #208] + csel x8, x20, x8, cc // cc = lo, ul, last + csel x9, x21, x9, cc // cc = lo, ul, last + ldp x20, x21, [x26, #64] + csel x8, x20, x8, hi // hi = pmore + csel x9, x21, x9, hi // hi = pmore + ldp x20, x21, [x25, #80] + ldp x10, x11, [sp, #224] + csel x10, x20, x10, cc // cc = lo, ul, last + csel x11, x21, x11, cc // cc = lo, ul, last + ldp x20, x21, [x26, #80] + csel x10, x20, x10, hi // hi = pmore + csel x11, x21, x11, hi // hi = pmore + stp x0, x1, [x27] + stp x2, x3, [x27, #16] + stp x4, x5, [x27, #32] + stp x6, x7, [x27, #48] + stp x8, x9, [x27, #64] + stp x10, x11, [x27, #80] + stp x12, x13, [x27, #96] + stp x14, x15, [x27, #112] + stp x16, x17, [x27, #128] + CFI_INC_SP(384) + CFI_POP1Z(x27) + CFI_POP2(x25,x26) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp384_montjscalarmul_p384_montjadd) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp384_montjscalarmul_p384_montjdouble) + +Lp384_montjscalarmul_p384_montjdouble: + CFI_START + CFI_DEC_SP(416) + stp x19, x20, [sp, #336] + stp x21, x22, [sp, #352] + stp x23, x24, [sp, #368] + stp x25, x26, [sp, #384] + stp x27, xzr, [sp, #400] + mov x25, x0 + mov x26, x1 + mov x0, sp + ldr q1, [x26, #96] + ldp x9, x2, [x26, #96] + ldr q0, [x26, #96] + ldp x4, x6, [x26, #112] + rev64 v21.4s, v1.4s + uzp2 v28.4s, v1.4s, v1.4s + umulh x7, x9, x2 + xtn v17.2s, v1.2d + mul v27.4s, v21.4s, v0.4s + ldr q20, [x26, #128] + xtn v30.2s, v0.2d + ldr q1, [x26, #128] + uzp2 v31.4s, v0.4s, v0.4s + ldp x5, x10, [x26, #128] + umulh x8, x9, x4 + uaddlp v3.2d, v27.4s + umull v16.2d, v30.2s, v17.2s + mul x16, x9, x4 + umull v27.2d, v30.2s, v28.2s + shrn v0.2s, v20.2d, #32 + xtn v7.2s, v20.2d + shl v20.2d, v3.2d, #32 + umull v3.2d, v31.2s, v28.2s + mul x3, x2, x4 + umlal v20.2d, v30.2s, v17.2s + umull v22.2d, v7.2s, v0.2s + usra v27.2d, v16.2d, #32 + umulh x11, x2, x4 + movi v21.2d, #0xffffffff + uzp2 v28.4s, v1.4s, v1.4s + adds x15, x16, x7 + and v5.16b, v27.16b, v21.16b + adcs x3, x3, x8 + usra v3.2d, v27.2d, #32 + dup v29.2d, x6 + adcs x16, x11, xzr + mov x14, v20.d[0] + umlal v5.2d, v31.2s, v17.2s + mul x8, x9, x2 + mov x7, v20.d[1] + shl v19.2d, v22.2d, #33 + xtn v25.2s, v29.2d + rev64 v31.4s, v1.4s + lsl x13, x14, #32 + uzp2 v6.4s, v29.4s, v29.4s + umlal v19.2d, v7.2s, v7.2s + usra v3.2d, v5.2d, #32 + adds x1, x8, x8 + umulh x8, x4, x4 + add x12, x13, x14 + mul v17.4s, v31.4s, v29.4s + xtn v4.2s, v1.2d + adcs x14, x15, x15 + lsr x13, x12, #32 + adcs x15, x3, x3 + umull v31.2d, v25.2s, v28.2s + adcs x11, x16, x16 + umull v21.2d, v25.2s, v4.2s + mov x17, v3.d[0] + umull v18.2d, v6.2s, v28.2s + adc x16, x8, xzr + uaddlp v16.2d, v17.4s + movi v1.2d, #0xffffffff + subs x13, x13, x12 + usra v31.2d, v21.2d, #32 + sbc x8, x12, xzr + adds x17, x17, x1 + mul x1, x4, x4 + shl v28.2d, v16.2d, #32 + mov x3, v3.d[1] + adcs x14, x7, x14 + extr x7, x8, x13, #32 + adcs x13, x3, x15 + and v3.16b, v31.16b, v1.16b + adcs x11, x1, x11 + lsr x1, x8, #32 + umlal v3.2d, v6.2s, v4.2s + usra v18.2d, v31.2d, #32 + adc x3, x16, xzr + adds x1, x1, x12 + umlal v28.2d, v25.2s, v4.2s + adc x16, xzr, xzr + subs x15, x17, x7 + sbcs x7, x14, x1 + lsl x1, x15, #32 + sbcs x16, x13, x16 + add x8, x1, x15 + usra v18.2d, v3.2d, #32 + sbcs x14, x11, xzr + lsr x1, x8, #32 + sbcs x17, x3, xzr + sbc x11, x12, xzr + subs x13, x1, x8 + umulh x12, x4, x10 + sbc x1, x8, xzr + extr x13, x1, x13, #32 + lsr x1, x1, #32 + adds x15, x1, x8 + adc x1, xzr, xzr + subs x7, x7, x13 + sbcs x13, x16, x15 + lsl x3, x7, #32 + umulh x16, x2, x5 + sbcs x15, x14, x1 + add x7, x3, x7 + sbcs x3, x17, xzr + lsr x1, x7, #32 + sbcs x14, x11, xzr + sbc x11, x8, xzr + subs x8, x1, x7 + sbc x1, x7, xzr + extr x8, x1, x8, #32 + lsr x1, x1, #32 + adds x1, x1, x7 + adc x17, xzr, xzr + subs x13, x13, x8 + umulh x8, x9, x6 + sbcs x1, x15, x1 + sbcs x15, x3, x17 + sbcs x3, x14, xzr + mul x17, x2, x5 + sbcs x11, x11, xzr + stp x13, x1, [x0] + sbc x14, x7, xzr + mul x7, x4, x10 + subs x1, x9, x2 + stp x15, x3, [x0, #16] + csetm x15, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + stp x11, x14, [x0, #32] + mul x14, x9, x6 + adds x17, x8, x17 + adcs x7, x16, x7 + adc x13, x12, xzr + subs x12, x5, x6 + cneg x3, x12, cc // cc = lo, ul, last + cinv x16, x15, cc // cc = lo, ul, last + mul x8, x1, x3 + umulh x1, x1, x3 + eor x12, x8, x16 + adds x11, x17, x14 + adcs x3, x7, x17 + adcs x15, x13, x7 + adc x8, x13, xzr + adds x3, x3, x14 + adcs x15, x15, x17 + adcs x17, x8, x7 + eor x1, x1, x16 + adc x13, x13, xzr + subs x9, x9, x4 + csetm x8, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x4, x2, x4 + cneg x4, x4, cc // cc = lo, ul, last + csetm x7, cc // cc = lo, ul, last + subs x2, x10, x6 + cinv x8, x8, cc // cc = lo, ul, last + cneg x2, x2, cc // cc = lo, ul, last + cmn x16, #0x1 + adcs x11, x11, x12 + mul x12, x9, x2 + adcs x3, x3, x1 + adcs x15, x15, x16 + umulh x9, x9, x2 + adcs x17, x17, x16 + adc x13, x13, x16 + subs x1, x10, x5 + cinv x2, x7, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + eor x9, x9, x8 + cmn x8, #0x1 + eor x7, x12, x8 + mul x12, x4, x1 + adcs x3, x3, x7 + adcs x7, x15, x9 + adcs x15, x17, x8 + ldp x9, x17, [x0, #16] + umulh x4, x4, x1 + adc x8, x13, x8 + cmn x2, #0x1 + eor x1, x12, x2 + adcs x1, x7, x1 + ldp x7, x16, [x0] + eor x12, x4, x2 + adcs x4, x15, x12 + ldp x15, x12, [x0, #32] + adc x8, x8, x2 + adds x13, x14, x14 + umulh x14, x5, x10 + adcs x2, x11, x11 + adcs x3, x3, x3 + adcs x1, x1, x1 + adcs x4, x4, x4 + adcs x11, x8, x8 + adc x8, xzr, xzr + adds x13, x13, x7 + adcs x2, x2, x16 + mul x16, x5, x10 + adcs x3, x3, x9 + adcs x1, x1, x17 + umulh x5, x5, x5 + lsl x9, x13, #32 + add x9, x9, x13 + adcs x4, x4, x15 + mov x13, v28.d[1] + adcs x15, x11, x12 + lsr x7, x9, #32 + adc x11, x8, xzr + subs x7, x7, x9 + umulh x10, x10, x10 + sbc x17, x9, xzr + extr x7, x17, x7, #32 + lsr x17, x17, #32 + adds x17, x17, x9 + adc x12, xzr, xzr + subs x8, x2, x7 + sbcs x17, x3, x17 + lsl x7, x8, #32 + sbcs x2, x1, x12 + add x3, x7, x8 + sbcs x12, x4, xzr + lsr x1, x3, #32 + sbcs x7, x15, xzr + sbc x15, x9, xzr + subs x1, x1, x3 + sbc x4, x3, xzr + lsr x9, x4, #32 + extr x8, x4, x1, #32 + adds x9, x9, x3 + adc x4, xzr, xzr + subs x1, x17, x8 + lsl x17, x1, #32 + sbcs x8, x2, x9 + sbcs x9, x12, x4 + add x17, x17, x1 + mov x1, v18.d[1] + lsr x2, x17, #32 + sbcs x7, x7, xzr + mov x12, v18.d[0] + sbcs x15, x15, xzr + sbc x3, x3, xzr + subs x4, x2, x17 + sbc x2, x17, xzr + adds x12, x13, x12 + adcs x16, x16, x1 + lsr x13, x2, #32 + extr x1, x2, x4, #32 + adc x2, x14, xzr + adds x4, x13, x17 + mul x13, x6, x6 + adc x14, xzr, xzr + subs x1, x8, x1 + sbcs x4, x9, x4 + mov x9, v28.d[0] + sbcs x7, x7, x14 + sbcs x8, x15, xzr + sbcs x3, x3, xzr + sbc x14, x17, xzr + adds x17, x9, x9 + adcs x12, x12, x12 + mov x15, v19.d[0] + adcs x9, x16, x16 + umulh x6, x6, x6 + adcs x16, x2, x2 + adc x2, xzr, xzr + adds x11, x11, x8 + adcs x3, x3, xzr + adcs x14, x14, xzr + adcs x8, xzr, xzr + adds x13, x1, x13 + mov x1, v19.d[1] + adcs x6, x4, x6 + mov x4, #0xffffffff // #4294967295 + adcs x15, x7, x15 + adcs x7, x11, x5 + adcs x1, x3, x1 + adcs x14, x14, x10 + adc x11, x8, xzr + adds x6, x6, x17 + adcs x8, x15, x12 + adcs x3, x7, x9 + adcs x15, x1, x16 + mov x16, #0xffffffff00000001 // #-4294967295 + adcs x14, x14, x2 + mov x2, #0x1 // #1 + adc x17, x11, xzr + cmn x13, x16 + adcs xzr, x6, x4 + adcs xzr, x8, x2 + adcs xzr, x3, xzr + adcs xzr, x15, xzr + adcs xzr, x14, xzr + adc x1, x17, xzr + neg x9, x1 + and x1, x16, x9 + adds x11, x13, x1 + and x13, x4, x9 + adcs x5, x6, x13 + and x1, x2, x9 + adcs x7, x8, x1 + stp x11, x5, [x0] + adcs x11, x3, xzr + adcs x2, x15, xzr + stp x7, x11, [x0, #16] + adc x17, x14, xzr + stp x2, x17, [x0, #32] + ldr q1, [x26, #48] + ldp x9, x2, [x26, #48] + ldr q0, [x26, #48] + ldp x4, x6, [x26, #64] + rev64 v21.4s, v1.4s + uzp2 v28.4s, v1.4s, v1.4s + umulh x7, x9, x2 + xtn v17.2s, v1.2d + mul v27.4s, v21.4s, v0.4s + ldr q20, [x26, #80] + xtn v30.2s, v0.2d + ldr q1, [x26, #80] + uzp2 v31.4s, v0.4s, v0.4s + ldp x5, x10, [x26, #80] + umulh x8, x9, x4 + uaddlp v3.2d, v27.4s + umull v16.2d, v30.2s, v17.2s + mul x16, x9, x4 + umull v27.2d, v30.2s, v28.2s + shrn v0.2s, v20.2d, #32 + xtn v7.2s, v20.2d + shl v20.2d, v3.2d, #32 + umull v3.2d, v31.2s, v28.2s + mul x3, x2, x4 + umlal v20.2d, v30.2s, v17.2s + umull v22.2d, v7.2s, v0.2s + usra v27.2d, v16.2d, #32 + umulh x11, x2, x4 + movi v21.2d, #0xffffffff + uzp2 v28.4s, v1.4s, v1.4s + adds x15, x16, x7 + and v5.16b, v27.16b, v21.16b + adcs x3, x3, x8 + usra v3.2d, v27.2d, #32 + dup v29.2d, x6 + adcs x16, x11, xzr + mov x14, v20.d[0] + umlal v5.2d, v31.2s, v17.2s + mul x8, x9, x2 + mov x7, v20.d[1] + shl v19.2d, v22.2d, #33 + xtn v25.2s, v29.2d + rev64 v31.4s, v1.4s + lsl x13, x14, #32 + uzp2 v6.4s, v29.4s, v29.4s + umlal v19.2d, v7.2s, v7.2s + usra v3.2d, v5.2d, #32 + adds x1, x8, x8 + umulh x8, x4, x4 + add x12, x13, x14 + mul v17.4s, v31.4s, v29.4s + xtn v4.2s, v1.2d + adcs x14, x15, x15 + lsr x13, x12, #32 + adcs x15, x3, x3 + umull v31.2d, v25.2s, v28.2s + adcs x11, x16, x16 + umull v21.2d, v25.2s, v4.2s + mov x17, v3.d[0] + umull v18.2d, v6.2s, v28.2s + adc x16, x8, xzr + uaddlp v16.2d, v17.4s + movi v1.2d, #0xffffffff + subs x13, x13, x12 + usra v31.2d, v21.2d, #32 + sbc x8, x12, xzr + adds x17, x17, x1 + mul x1, x4, x4 + shl v28.2d, v16.2d, #32 + mov x3, v3.d[1] + adcs x14, x7, x14 + extr x7, x8, x13, #32 + adcs x13, x3, x15 + and v3.16b, v31.16b, v1.16b + adcs x11, x1, x11 + lsr x1, x8, #32 + umlal v3.2d, v6.2s, v4.2s + usra v18.2d, v31.2d, #32 + adc x3, x16, xzr + adds x1, x1, x12 + umlal v28.2d, v25.2s, v4.2s + adc x16, xzr, xzr + subs x15, x17, x7 + sbcs x7, x14, x1 + lsl x1, x15, #32 + sbcs x16, x13, x16 + add x8, x1, x15 + usra v18.2d, v3.2d, #32 + sbcs x14, x11, xzr + lsr x1, x8, #32 + sbcs x17, x3, xzr + sbc x11, x12, xzr + subs x13, x1, x8 + umulh x12, x4, x10 + sbc x1, x8, xzr + extr x13, x1, x13, #32 + lsr x1, x1, #32 + adds x15, x1, x8 + adc x1, xzr, xzr + subs x7, x7, x13 + sbcs x13, x16, x15 + lsl x3, x7, #32 + umulh x16, x2, x5 + sbcs x15, x14, x1 + add x7, x3, x7 + sbcs x3, x17, xzr + lsr x1, x7, #32 + sbcs x14, x11, xzr + sbc x11, x8, xzr + subs x8, x1, x7 + sbc x1, x7, xzr + extr x8, x1, x8, #32 + lsr x1, x1, #32 + adds x1, x1, x7 + adc x17, xzr, xzr + subs x13, x13, x8 + umulh x8, x9, x6 + sbcs x1, x15, x1 + sbcs x15, x3, x17 + sbcs x3, x14, xzr + mul x17, x2, x5 + sbcs x11, x11, xzr + stp x13, x1, [sp, #48] + sbc x14, x7, xzr + mul x7, x4, x10 + subs x1, x9, x2 + stp x15, x3, [sp, #64] + csetm x15, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + stp x11, x14, [sp, #80] + mul x14, x9, x6 + adds x17, x8, x17 + adcs x7, x16, x7 + adc x13, x12, xzr + subs x12, x5, x6 + cneg x3, x12, cc // cc = lo, ul, last + cinv x16, x15, cc // cc = lo, ul, last + mul x8, x1, x3 + umulh x1, x1, x3 + eor x12, x8, x16 + adds x11, x17, x14 + adcs x3, x7, x17 + adcs x15, x13, x7 + adc x8, x13, xzr + adds x3, x3, x14 + adcs x15, x15, x17 + adcs x17, x8, x7 + eor x1, x1, x16 + adc x13, x13, xzr + subs x9, x9, x4 + csetm x8, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x4, x2, x4 + cneg x4, x4, cc // cc = lo, ul, last + csetm x7, cc // cc = lo, ul, last + subs x2, x10, x6 + cinv x8, x8, cc // cc = lo, ul, last + cneg x2, x2, cc // cc = lo, ul, last + cmn x16, #0x1 + adcs x11, x11, x12 + mul x12, x9, x2 + adcs x3, x3, x1 + adcs x15, x15, x16 + umulh x9, x9, x2 + adcs x17, x17, x16 + adc x13, x13, x16 + subs x1, x10, x5 + cinv x2, x7, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + eor x9, x9, x8 + cmn x8, #0x1 + eor x7, x12, x8 + mul x12, x4, x1 + adcs x3, x3, x7 + adcs x7, x15, x9 + adcs x15, x17, x8 + ldp x9, x17, [sp, #64] + umulh x4, x4, x1 + adc x8, x13, x8 + cmn x2, #0x1 + eor x1, x12, x2 + adcs x1, x7, x1 + ldp x7, x16, [sp, #48] + eor x12, x4, x2 + adcs x4, x15, x12 + ldp x15, x12, [sp, #80] + adc x8, x8, x2 + adds x13, x14, x14 + umulh x14, x5, x10 + adcs x2, x11, x11 + adcs x3, x3, x3 + adcs x1, x1, x1 + adcs x4, x4, x4 + adcs x11, x8, x8 + adc x8, xzr, xzr + adds x13, x13, x7 + adcs x2, x2, x16 + mul x16, x5, x10 + adcs x3, x3, x9 + adcs x1, x1, x17 + umulh x5, x5, x5 + lsl x9, x13, #32 + add x9, x9, x13 + adcs x4, x4, x15 + mov x13, v28.d[1] + adcs x15, x11, x12 + lsr x7, x9, #32 + adc x11, x8, xzr + subs x7, x7, x9 + umulh x10, x10, x10 + sbc x17, x9, xzr + extr x7, x17, x7, #32 + lsr x17, x17, #32 + adds x17, x17, x9 + adc x12, xzr, xzr + subs x8, x2, x7 + sbcs x17, x3, x17 + lsl x7, x8, #32 + sbcs x2, x1, x12 + add x3, x7, x8 + sbcs x12, x4, xzr + lsr x1, x3, #32 + sbcs x7, x15, xzr + sbc x15, x9, xzr + subs x1, x1, x3 + sbc x4, x3, xzr + lsr x9, x4, #32 + extr x8, x4, x1, #32 + adds x9, x9, x3 + adc x4, xzr, xzr + subs x1, x17, x8 + lsl x17, x1, #32 + sbcs x8, x2, x9 + sbcs x9, x12, x4 + add x17, x17, x1 + mov x1, v18.d[1] + lsr x2, x17, #32 + sbcs x7, x7, xzr + mov x12, v18.d[0] + sbcs x15, x15, xzr + sbc x3, x3, xzr + subs x4, x2, x17 + sbc x2, x17, xzr + adds x12, x13, x12 + adcs x16, x16, x1 + lsr x13, x2, #32 + extr x1, x2, x4, #32 + adc x2, x14, xzr + adds x4, x13, x17 + mul x13, x6, x6 + adc x14, xzr, xzr + subs x1, x8, x1 + sbcs x4, x9, x4 + mov x9, v28.d[0] + sbcs x7, x7, x14 + sbcs x8, x15, xzr + sbcs x3, x3, xzr + sbc x14, x17, xzr + adds x17, x9, x9 + adcs x12, x12, x12 + mov x15, v19.d[0] + adcs x9, x16, x16 + umulh x6, x6, x6 + adcs x16, x2, x2 + adc x2, xzr, xzr + adds x11, x11, x8 + adcs x3, x3, xzr + adcs x14, x14, xzr + adcs x8, xzr, xzr + adds x13, x1, x13 + mov x1, v19.d[1] + adcs x6, x4, x6 + mov x4, #0xffffffff // #4294967295 + adcs x15, x7, x15 + adcs x7, x11, x5 + adcs x1, x3, x1 + adcs x14, x14, x10 + adc x11, x8, xzr + adds x6, x6, x17 + adcs x8, x15, x12 + adcs x3, x7, x9 + adcs x15, x1, x16 + mov x16, #0xffffffff00000001 // #-4294967295 + adcs x14, x14, x2 + mov x2, #0x1 // #1 + adc x17, x11, xzr + cmn x13, x16 + adcs xzr, x6, x4 + adcs xzr, x8, x2 + adcs xzr, x3, xzr + adcs xzr, x15, xzr + adcs xzr, x14, xzr + adc x1, x17, xzr + neg x9, x1 + and x1, x16, x9 + adds x11, x13, x1 + and x13, x4, x9 + adcs x5, x6, x13 + and x1, x2, x9 + adcs x7, x8, x1 + stp x11, x5, [sp, #48] + adcs x11, x3, xzr + adcs x2, x15, xzr + stp x7, x11, [sp, #64] + adc x17, x14, xzr + stp x2, x17, [sp, #80] + ldp x5, x6, [x26] + ldp x4, x3, [sp] + adds x5, x5, x4 + adcs x6, x6, x3 + ldp x7, x8, [x26, #16] + ldp x4, x3, [sp, #16] + adcs x7, x7, x4 + adcs x8, x8, x3 + ldp x9, x10, [x26, #32] + ldp x4, x3, [sp, #32] + adcs x9, x9, x4 + adcs x10, x10, x3 + csetm x3, cs // cs = hs, nlast + mov x4, #0xffffffff // #4294967295 + and x4, x4, x3 + subs x5, x5, x4 + eor x4, x4, x3 + sbcs x6, x6, x4 + mov x4, #0xfffffffffffffffe // #-2 + and x4, x4, x3 + sbcs x7, x7, x4 + sbcs x8, x8, x3 + sbcs x9, x9, x3 + sbc x10, x10, x3 + stp x5, x6, [sp, #240] + stp x7, x8, [sp, #256] + stp x9, x10, [sp, #272] + mov x2, sp + ldp x5, x6, [x26] + ldp x4, x3, [x2] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [x26, #16] + ldp x4, x3, [x2, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [x26, #32] + ldp x4, x3, [x2, #32] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, cc // cc = lo, ul, last + mov x4, #0xffffffff // #4294967295 + and x4, x4, x3 + adds x13, x5, x4 + eor x4, x4, x3 + adcs x23, x6, x4 + mov x4, #0xfffffffffffffffe // #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x13, x23, [sp, #192] + stp x7, x8, [sp, #208] + stp x9, x10, [sp, #224] + ldr q3, [sp, #240] + ldr q25, [sp, #192] + ldp x3, x21, [sp, #240] + rev64 v23.4s, v25.4s + uzp1 v17.4s, v25.4s, v3.4s + umulh x15, x3, x13 + mul v6.4s, v23.4s, v3.4s + uzp1 v3.4s, v3.4s, v3.4s + ldr q27, [sp, #224] + ldp x8, x24, [sp, #256] + subs x6, x3, x21 + ldr q0, [sp, #272] + movi v23.2d, #0xffffffff + csetm x10, cc // cc = lo, ul, last + umulh x19, x21, x23 + rev64 v4.4s, v27.4s + uzp2 v25.4s, v27.4s, v27.4s + cneg x4, x6, cc // cc = lo, ul, last + subs x7, x23, x13 + xtn v22.2s, v0.2d + xtn v24.2s, v27.2d + cneg x20, x7, cc // cc = lo, ul, last + ldp x6, x14, [sp, #208] + mul v27.4s, v4.4s, v0.4s + uaddlp v20.2d, v6.4s + cinv x5, x10, cc // cc = lo, ul, last + mul x16, x4, x20 + uzp2 v6.4s, v0.4s, v0.4s + umull v21.2d, v22.2s, v25.2s + shl v0.2d, v20.2d, #32 + umlal v0.2d, v3.2s, v17.2s + mul x22, x8, x6 + umull v1.2d, v6.2s, v25.2s + subs x12, x3, x8 + umull v20.2d, v22.2s, v24.2s + cneg x17, x12, cc // cc = lo, ul, last + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc // cc = lo, ul, last + usra v21.2d, v20.2d, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2d, v21.2d, #32 + adds x22, x15, x7 + and v26.16b, v21.16b, v23.16b + adcs x16, x12, x15 + uaddlp v25.2d, v27.4s + adcs x9, x19, x12 + umlal v26.2d, v6.2s, v24.2s + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2d, v25.2d, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc // cc = lo, ul, last + cinv x10, x10, cc // cc = lo, ul, last + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc // cc = lo, ul, last + eor x19, x19, x10 + csetm x4, cc // cc = lo, ul, last + subs x16, x6, x23 + cneg x16, x16, cc // cc = lo, ul, last + umlal v27.2d, v22.2s, v24.2s + mul x15, x20, x16 + cinv x4, x4, cc // cc = lo, ul, last + cmn x10, #0x1 + usra v1.2d, v26.2d, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [sp, #224] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [sp, #272] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x16, x4, x16 + mov x4, v27.d[0] + sbcs x11, x20, x11 + sbcs x20, x9, x12 + stp x16, x11, [sp, #96] + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [sp, #112] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc // cc = lo, ul, last + csetm x2, cc // cc = lo, ul, last + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc // cc = lo, ul, last + cneg x19, x19, cc // cc = lo, ul, last + stp x9, x20, [sp, #128] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc // cc = lo, ul, last + csetm x12, cc // cc = lo, ul, last + subs x9, x17, x14 + cinv x12, x12, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc // cc = lo, ul, last + cneg x24, x10, cc // cc = lo, ul, last + subs x10, x17, x15 + cinv x7, x7, cc // cc = lo, ul, last + cneg x10, x10, cc // cc = lo, ul, last + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + ldp x20, x9, [sp, #96] + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [sp, #112] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [sp, #128] + adds x20, x22, x20 + mul x10, x13, x14 + adcs x11, x11, x9 + eor x9, x8, x21 + adcs x21, x19, x17 + stp x20, x11, [sp, #96] + adcs x12, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + stp x21, x12, [sp, #112] + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [sp, #128] + cneg x3, x21, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc // cc = lo, ul, last + csetm x16, cc // cc = lo, ul, last + subs x21, x6, x15 + cneg x22, x21, cc // cc = lo, ul, last + cinv x21, x24, cc // cc = lo, ul, last + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + subs x20, x14, x15 + cinv x24, x24, cc // cc = lo, ul, last + mul x22, x3, x22 + cneg x3, x20, cc // cc = lo, ul, last + subs x13, x6, x14 + cneg x20, x13, cc // cc = lo, ul, last + cinv x15, x16, cc // cc = lo, ul, last + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe // #-2 + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [sp, #96] + ldp x21, x12, [sp, #112] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [sp, #128] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x21 + eor x1, x22, x9 + adcs x24, x23, x12 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x21 + adcs x15, x17, x12 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff // #4294967295 + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc // cc = lo, ul, last + and x11, x16, x23 + and x20, x14, x23 + adds x22, x6, x20 + eor x3, x20, x23 + adcs x5, x7, x3 + adcs x14, x24, x11 + stp x22, x5, [sp, #96] + adcs x5, x13, x23 + adcs x21, x8, x23 + stp x14, x5, [sp, #112] + adc x12, x15, x23 + stp x21, x12, [sp, #128] + ldp x5, x6, [x26, #48] + ldp x4, x3, [x26, #96] + adds x5, x5, x4 + adcs x6, x6, x3 + ldp x7, x8, [x26, #64] + ldp x4, x3, [x26, #112] + adcs x7, x7, x4 + adcs x8, x8, x3 + ldp x9, x10, [x26, #80] + ldp x4, x3, [x26, #128] + adcs x9, x9, x4 + adcs x10, x10, x3 + adc x3, xzr, xzr + mov x4, #0xffffffff // #4294967295 + cmp x5, x4 + mov x4, #0xffffffff00000000 // #-4294967296 + sbcs xzr, x6, x4 + mov x4, #0xfffffffffffffffe // #-2 + sbcs xzr, x7, x4 + adcs xzr, x8, xzr + adcs xzr, x9, xzr + adcs xzr, x10, xzr + adcs x3, x3, xzr + csetm x3, ne // ne = any + mov x4, #0xffffffff // #4294967295 + and x4, x4, x3 + subs x5, x5, x4 + eor x4, x4, x3 + sbcs x6, x6, x4 + mov x4, #0xfffffffffffffffe // #-2 + and x4, x4, x3 + sbcs x7, x7, x4 + sbcs x8, x8, x3 + sbcs x9, x9, x3 + sbc x10, x10, x3 + stp x5, x6, [sp, #240] + stp x7, x8, [sp, #256] + stp x9, x10, [sp, #272] + ldr q1, [sp, #96] + ldp x9, x2, [sp, #96] + ldr q0, [sp, #96] + ldp x4, x6, [sp, #112] + rev64 v21.4s, v1.4s + uzp2 v28.4s, v1.4s, v1.4s + umulh x7, x9, x2 + xtn v17.2s, v1.2d + mul v27.4s, v21.4s, v0.4s + ldr q20, [sp, #128] + xtn v30.2s, v0.2d + ldr q1, [sp, #128] + uzp2 v31.4s, v0.4s, v0.4s + ldp x5, x10, [sp, #128] + umulh x8, x9, x4 + uaddlp v3.2d, v27.4s + umull v16.2d, v30.2s, v17.2s + mul x16, x9, x4 + umull v27.2d, v30.2s, v28.2s + shrn v0.2s, v20.2d, #32 + xtn v7.2s, v20.2d + shl v20.2d, v3.2d, #32 + umull v3.2d, v31.2s, v28.2s + mul x3, x2, x4 + umlal v20.2d, v30.2s, v17.2s + umull v22.2d, v7.2s, v0.2s + usra v27.2d, v16.2d, #32 + umulh x11, x2, x4 + movi v21.2d, #0xffffffff + uzp2 v28.4s, v1.4s, v1.4s + adds x15, x16, x7 + and v5.16b, v27.16b, v21.16b + adcs x3, x3, x8 + usra v3.2d, v27.2d, #32 + dup v29.2d, x6 + adcs x16, x11, xzr + mov x14, v20.d[0] + umlal v5.2d, v31.2s, v17.2s + mul x8, x9, x2 + mov x7, v20.d[1] + shl v19.2d, v22.2d, #33 + xtn v25.2s, v29.2d + rev64 v31.4s, v1.4s + lsl x13, x14, #32 + uzp2 v6.4s, v29.4s, v29.4s + umlal v19.2d, v7.2s, v7.2s + usra v3.2d, v5.2d, #32 + adds x1, x8, x8 + umulh x8, x4, x4 + add x12, x13, x14 + mul v17.4s, v31.4s, v29.4s + xtn v4.2s, v1.2d + adcs x14, x15, x15 + lsr x13, x12, #32 + adcs x15, x3, x3 + umull v31.2d, v25.2s, v28.2s + adcs x11, x16, x16 + umull v21.2d, v25.2s, v4.2s + mov x17, v3.d[0] + umull v18.2d, v6.2s, v28.2s + adc x16, x8, xzr + uaddlp v16.2d, v17.4s + movi v1.2d, #0xffffffff + subs x13, x13, x12 + usra v31.2d, v21.2d, #32 + sbc x8, x12, xzr + adds x17, x17, x1 + mul x1, x4, x4 + shl v28.2d, v16.2d, #32 + mov x3, v3.d[1] + adcs x14, x7, x14 + extr x7, x8, x13, #32 + adcs x13, x3, x15 + and v3.16b, v31.16b, v1.16b + adcs x11, x1, x11 + lsr x1, x8, #32 + umlal v3.2d, v6.2s, v4.2s + usra v18.2d, v31.2d, #32 + adc x3, x16, xzr + adds x1, x1, x12 + umlal v28.2d, v25.2s, v4.2s + adc x16, xzr, xzr + subs x15, x17, x7 + sbcs x7, x14, x1 + lsl x1, x15, #32 + sbcs x16, x13, x16 + add x8, x1, x15 + usra v18.2d, v3.2d, #32 + sbcs x14, x11, xzr + lsr x1, x8, #32 + sbcs x17, x3, xzr + sbc x11, x12, xzr + subs x13, x1, x8 + umulh x12, x4, x10 + sbc x1, x8, xzr + extr x13, x1, x13, #32 + lsr x1, x1, #32 + adds x15, x1, x8 + adc x1, xzr, xzr + subs x7, x7, x13 + sbcs x13, x16, x15 + lsl x3, x7, #32 + umulh x16, x2, x5 + sbcs x15, x14, x1 + add x7, x3, x7 + sbcs x3, x17, xzr + lsr x1, x7, #32 + sbcs x14, x11, xzr + sbc x11, x8, xzr + subs x8, x1, x7 + sbc x1, x7, xzr + extr x8, x1, x8, #32 + lsr x1, x1, #32 + adds x1, x1, x7 + adc x17, xzr, xzr + subs x13, x13, x8 + umulh x8, x9, x6 + sbcs x1, x15, x1 + sbcs x15, x3, x17 + sbcs x3, x14, xzr + mul x17, x2, x5 + sbcs x11, x11, xzr + stp x13, x1, [sp, #288] + sbc x14, x7, xzr + mul x7, x4, x10 + subs x1, x9, x2 + stp x15, x3, [sp, #304] + csetm x15, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + stp x11, x14, [sp, #320] + mul x14, x9, x6 + adds x17, x8, x17 + adcs x7, x16, x7 + adc x13, x12, xzr + subs x12, x5, x6 + cneg x3, x12, cc // cc = lo, ul, last + cinv x16, x15, cc // cc = lo, ul, last + mul x8, x1, x3 + umulh x1, x1, x3 + eor x12, x8, x16 + adds x11, x17, x14 + adcs x3, x7, x17 + adcs x15, x13, x7 + adc x8, x13, xzr + adds x3, x3, x14 + adcs x15, x15, x17 + adcs x17, x8, x7 + eor x1, x1, x16 + adc x13, x13, xzr + subs x9, x9, x4 + csetm x8, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x4, x2, x4 + cneg x4, x4, cc // cc = lo, ul, last + csetm x7, cc // cc = lo, ul, last + subs x2, x10, x6 + cinv x8, x8, cc // cc = lo, ul, last + cneg x2, x2, cc // cc = lo, ul, last + cmn x16, #0x1 + adcs x11, x11, x12 + mul x12, x9, x2 + adcs x3, x3, x1 + adcs x15, x15, x16 + umulh x9, x9, x2 + adcs x17, x17, x16 + adc x13, x13, x16 + subs x1, x10, x5 + cinv x2, x7, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + eor x9, x9, x8 + cmn x8, #0x1 + eor x7, x12, x8 + mul x12, x4, x1 + adcs x3, x3, x7 + adcs x7, x15, x9 + adcs x15, x17, x8 + ldp x9, x17, [sp, #304] + umulh x4, x4, x1 + adc x8, x13, x8 + cmn x2, #0x1 + eor x1, x12, x2 + adcs x1, x7, x1 + ldp x7, x16, [sp, #288] + eor x12, x4, x2 + adcs x4, x15, x12 + ldp x15, x12, [sp, #320] + adc x8, x8, x2 + adds x13, x14, x14 + umulh x14, x5, x10 + adcs x2, x11, x11 + adcs x3, x3, x3 + adcs x1, x1, x1 + adcs x4, x4, x4 + adcs x11, x8, x8 + adc x8, xzr, xzr + adds x13, x13, x7 + adcs x2, x2, x16 + mul x16, x5, x10 + adcs x3, x3, x9 + adcs x1, x1, x17 + umulh x5, x5, x5 + lsl x9, x13, #32 + add x9, x9, x13 + adcs x4, x4, x15 + mov x13, v28.d[1] + adcs x15, x11, x12 + lsr x7, x9, #32 + adc x11, x8, xzr + subs x7, x7, x9 + umulh x10, x10, x10 + sbc x17, x9, xzr + extr x7, x17, x7, #32 + lsr x17, x17, #32 + adds x17, x17, x9 + adc x12, xzr, xzr + subs x8, x2, x7 + sbcs x17, x3, x17 + lsl x7, x8, #32 + sbcs x2, x1, x12 + add x3, x7, x8 + sbcs x12, x4, xzr + lsr x1, x3, #32 + sbcs x7, x15, xzr + sbc x15, x9, xzr + subs x1, x1, x3 + sbc x4, x3, xzr + lsr x9, x4, #32 + extr x8, x4, x1, #32 + adds x9, x9, x3 + adc x4, xzr, xzr + subs x1, x17, x8 + lsl x17, x1, #32 + sbcs x8, x2, x9 + sbcs x9, x12, x4 + add x17, x17, x1 + mov x1, v18.d[1] + lsr x2, x17, #32 + sbcs x7, x7, xzr + mov x12, v18.d[0] + sbcs x15, x15, xzr + sbc x3, x3, xzr + subs x4, x2, x17 + sbc x2, x17, xzr + adds x12, x13, x12 + adcs x16, x16, x1 + lsr x13, x2, #32 + extr x1, x2, x4, #32 + adc x2, x14, xzr + adds x4, x13, x17 + mul x13, x6, x6 + adc x14, xzr, xzr + subs x1, x8, x1 + sbcs x4, x9, x4 + mov x9, v28.d[0] + sbcs x7, x7, x14 + sbcs x8, x15, xzr + sbcs x3, x3, xzr + sbc x14, x17, xzr + adds x17, x9, x9 + adcs x12, x12, x12 + mov x15, v19.d[0] + adcs x9, x16, x16 + umulh x6, x6, x6 + adcs x16, x2, x2 + adc x2, xzr, xzr + adds x11, x11, x8 + adcs x3, x3, xzr + adcs x14, x14, xzr + adcs x8, xzr, xzr + adds x13, x1, x13 + mov x1, v19.d[1] + adcs x6, x4, x6 + mov x4, #0xffffffff // #4294967295 + adcs x15, x7, x15 + adcs x7, x11, x5 + adcs x1, x3, x1 + adcs x14, x14, x10 + adc x11, x8, xzr + adds x6, x6, x17 + adcs x8, x15, x12 + adcs x3, x7, x9 + adcs x15, x1, x16 + mov x16, #0xffffffff00000001 // #-4294967295 + adcs x14, x14, x2 + mov x2, #0x1 // #1 + adc x17, x11, xzr + cmn x13, x16 + adcs xzr, x6, x4 + adcs xzr, x8, x2 + adcs xzr, x3, xzr + adcs xzr, x15, xzr + adcs xzr, x14, xzr + adc x1, x17, xzr + neg x9, x1 + and x1, x16, x9 + adds x11, x13, x1 + and x13, x4, x9 + adcs x5, x6, x13 + and x1, x2, x9 + adcs x7, x8, x1 + stp x11, x5, [sp, #288] + adcs x11, x3, xzr + adcs x2, x15, xzr + stp x7, x11, [sp, #304] + adc x17, x14, xzr + stp x2, x17, [sp, #320] + ldr q3, [x26] + ldr q25, [sp, #48] + ldp x13, x23, [sp, #48] + ldp x3, x21, [x26] + rev64 v23.4s, v25.4s + uzp1 v17.4s, v25.4s, v3.4s + umulh x15, x3, x13 + mul v6.4s, v23.4s, v3.4s + uzp1 v3.4s, v3.4s, v3.4s + ldr q27, [sp, #80] + ldp x8, x24, [x26, #16] + subs x6, x3, x21 + ldr q0, [x26, #32] + movi v23.2d, #0xffffffff + csetm x10, cc // cc = lo, ul, last + umulh x19, x21, x23 + rev64 v4.4s, v27.4s + uzp2 v25.4s, v27.4s, v27.4s + cneg x4, x6, cc // cc = lo, ul, last + subs x7, x23, x13 + xtn v22.2s, v0.2d + xtn v24.2s, v27.2d + cneg x20, x7, cc // cc = lo, ul, last + ldp x6, x14, [sp, #64] + mul v27.4s, v4.4s, v0.4s + uaddlp v20.2d, v6.4s + cinv x5, x10, cc // cc = lo, ul, last + mul x16, x4, x20 + uzp2 v6.4s, v0.4s, v0.4s + umull v21.2d, v22.2s, v25.2s + shl v0.2d, v20.2d, #32 + umlal v0.2d, v3.2s, v17.2s + mul x22, x8, x6 + umull v1.2d, v6.2s, v25.2s + subs x12, x3, x8 + umull v20.2d, v22.2s, v24.2s + cneg x17, x12, cc // cc = lo, ul, last + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc // cc = lo, ul, last + usra v21.2d, v20.2d, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2d, v21.2d, #32 + adds x22, x15, x7 + and v26.16b, v21.16b, v23.16b + adcs x16, x12, x15 + uaddlp v25.2d, v27.4s + adcs x9, x19, x12 + umlal v26.2d, v6.2s, v24.2s + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2d, v25.2d, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc // cc = lo, ul, last + cinv x10, x10, cc // cc = lo, ul, last + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc // cc = lo, ul, last + eor x19, x19, x10 + csetm x4, cc // cc = lo, ul, last + subs x16, x6, x23 + cneg x16, x16, cc // cc = lo, ul, last + umlal v27.2d, v22.2s, v24.2s + mul x15, x20, x16 + cinv x4, x4, cc // cc = lo, ul, last + cmn x10, #0x1 + usra v1.2d, v26.2d, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [sp, #80] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [x26, #32] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x26, x4, x16 + mov x4, v27.d[0] + sbcs x27, x20, x11 + sbcs x20, x9, x12 + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [sp, #160] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc // cc = lo, ul, last + csetm x2, cc // cc = lo, ul, last + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc // cc = lo, ul, last + cneg x19, x19, cc // cc = lo, ul, last + stp x9, x20, [sp, #176] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc // cc = lo, ul, last + csetm x12, cc // cc = lo, ul, last + subs x9, x17, x14 + cinv x12, x12, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc // cc = lo, ul, last + cneg x24, x10, cc // cc = lo, ul, last + subs x10, x17, x15 + cinv x7, x7, cc // cc = lo, ul, last + cneg x10, x10, cc // cc = lo, ul, last + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [sp, #160] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [sp, #176] + adds x20, x22, x26 + mul x10, x13, x14 + adcs x11, x11, x27 + eor x9, x8, x21 + adcs x26, x19, x17 + stp x20, x11, [sp, #144] + adcs x27, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [sp, #176] + cneg x3, x21, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc // cc = lo, ul, last + csetm x16, cc // cc = lo, ul, last + subs x21, x6, x15 + cneg x22, x21, cc // cc = lo, ul, last + cinv x21, x24, cc // cc = lo, ul, last + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + subs x20, x14, x15 + cinv x24, x24, cc // cc = lo, ul, last + mul x22, x3, x22 + cneg x3, x20, cc // cc = lo, ul, last + subs x13, x6, x14 + cneg x20, x13, cc // cc = lo, ul, last + cinv x15, x16, cc // cc = lo, ul, last + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe // #-2 + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [sp, #144] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [sp, #176] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x26 + eor x1, x22, x9 + adcs x24, x23, x27 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x26 + adcs x15, x17, x27 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff // #4294967295 + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc // cc = lo, ul, last + and x11, x16, x23 + and x20, x14, x23 + adds x22, x6, x20 + eor x3, x20, x23 + adcs x5, x7, x3 + adcs x14, x24, x11 + stp x22, x5, [sp, #144] + adcs x5, x13, x23 + adcs x21, x8, x23 + stp x14, x5, [sp, #160] + adc x12, x15, x23 + stp x21, x12, [sp, #176] + ldr q1, [sp, #240] + ldp x9, x2, [sp, #240] + ldr q0, [sp, #240] + ldp x4, x6, [sp, #256] + rev64 v21.4s, v1.4s + uzp2 v28.4s, v1.4s, v1.4s + umulh x7, x9, x2 + xtn v17.2s, v1.2d + mul v27.4s, v21.4s, v0.4s + ldr q20, [sp, #272] + xtn v30.2s, v0.2d + ldr q1, [sp, #272] + uzp2 v31.4s, v0.4s, v0.4s + ldp x5, x10, [sp, #272] + umulh x8, x9, x4 + uaddlp v3.2d, v27.4s + umull v16.2d, v30.2s, v17.2s + mul x16, x9, x4 + umull v27.2d, v30.2s, v28.2s + shrn v0.2s, v20.2d, #32 + xtn v7.2s, v20.2d + shl v20.2d, v3.2d, #32 + umull v3.2d, v31.2s, v28.2s + mul x3, x2, x4 + umlal v20.2d, v30.2s, v17.2s + umull v22.2d, v7.2s, v0.2s + usra v27.2d, v16.2d, #32 + umulh x11, x2, x4 + movi v21.2d, #0xffffffff + uzp2 v28.4s, v1.4s, v1.4s + adds x15, x16, x7 + and v5.16b, v27.16b, v21.16b + adcs x3, x3, x8 + usra v3.2d, v27.2d, #32 + dup v29.2d, x6 + adcs x16, x11, xzr + mov x14, v20.d[0] + umlal v5.2d, v31.2s, v17.2s + mul x8, x9, x2 + mov x7, v20.d[1] + shl v19.2d, v22.2d, #33 + xtn v25.2s, v29.2d + rev64 v31.4s, v1.4s + lsl x13, x14, #32 + uzp2 v6.4s, v29.4s, v29.4s + umlal v19.2d, v7.2s, v7.2s + usra v3.2d, v5.2d, #32 + adds x1, x8, x8 + umulh x8, x4, x4 + add x12, x13, x14 + mul v17.4s, v31.4s, v29.4s + xtn v4.2s, v1.2d + adcs x14, x15, x15 + lsr x13, x12, #32 + adcs x15, x3, x3 + umull v31.2d, v25.2s, v28.2s + adcs x11, x16, x16 + umull v21.2d, v25.2s, v4.2s + mov x17, v3.d[0] + umull v18.2d, v6.2s, v28.2s + adc x16, x8, xzr + uaddlp v16.2d, v17.4s + movi v1.2d, #0xffffffff + subs x13, x13, x12 + usra v31.2d, v21.2d, #32 + sbc x8, x12, xzr + adds x17, x17, x1 + mul x1, x4, x4 + shl v28.2d, v16.2d, #32 + mov x3, v3.d[1] + adcs x14, x7, x14 + extr x7, x8, x13, #32 + adcs x13, x3, x15 + and v3.16b, v31.16b, v1.16b + adcs x11, x1, x11 + lsr x1, x8, #32 + umlal v3.2d, v6.2s, v4.2s + usra v18.2d, v31.2d, #32 + adc x3, x16, xzr + adds x1, x1, x12 + umlal v28.2d, v25.2s, v4.2s + adc x16, xzr, xzr + subs x15, x17, x7 + sbcs x7, x14, x1 + lsl x1, x15, #32 + sbcs x16, x13, x16 + add x8, x1, x15 + usra v18.2d, v3.2d, #32 + sbcs x14, x11, xzr + lsr x1, x8, #32 + sbcs x17, x3, xzr + sbc x11, x12, xzr + subs x13, x1, x8 + umulh x12, x4, x10 + sbc x1, x8, xzr + extr x13, x1, x13, #32 + lsr x1, x1, #32 + adds x15, x1, x8 + adc x1, xzr, xzr + subs x7, x7, x13 + sbcs x13, x16, x15 + lsl x3, x7, #32 + umulh x16, x2, x5 + sbcs x15, x14, x1 + add x7, x3, x7 + sbcs x3, x17, xzr + lsr x1, x7, #32 + sbcs x14, x11, xzr + sbc x11, x8, xzr + subs x8, x1, x7 + sbc x1, x7, xzr + extr x8, x1, x8, #32 + lsr x1, x1, #32 + adds x1, x1, x7 + adc x17, xzr, xzr + subs x13, x13, x8 + umulh x8, x9, x6 + sbcs x1, x15, x1 + sbcs x19, x3, x17 + sbcs x20, x14, xzr + mul x17, x2, x5 + sbcs x11, x11, xzr + stp x13, x1, [sp, #192] + sbc x14, x7, xzr + mul x7, x4, x10 + subs x1, x9, x2 + csetm x15, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + stp x11, x14, [sp, #224] + mul x14, x9, x6 + adds x17, x8, x17 + adcs x7, x16, x7 + adc x13, x12, xzr + subs x12, x5, x6 + cneg x3, x12, cc // cc = lo, ul, last + cinv x16, x15, cc // cc = lo, ul, last + mul x8, x1, x3 + umulh x1, x1, x3 + eor x12, x8, x16 + adds x11, x17, x14 + adcs x3, x7, x17 + adcs x15, x13, x7 + adc x8, x13, xzr + adds x3, x3, x14 + adcs x15, x15, x17 + adcs x17, x8, x7 + eor x1, x1, x16 + adc x13, x13, xzr + subs x9, x9, x4 + csetm x8, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x4, x2, x4 + cneg x4, x4, cc // cc = lo, ul, last + csetm x7, cc // cc = lo, ul, last + subs x2, x10, x6 + cinv x8, x8, cc // cc = lo, ul, last + cneg x2, x2, cc // cc = lo, ul, last + cmn x16, #0x1 + adcs x11, x11, x12 + mul x12, x9, x2 + adcs x3, x3, x1 + adcs x15, x15, x16 + umulh x9, x9, x2 + adcs x17, x17, x16 + adc x13, x13, x16 + subs x1, x10, x5 + cinv x2, x7, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + eor x9, x9, x8 + cmn x8, #0x1 + eor x7, x12, x8 + mul x12, x4, x1 + adcs x3, x3, x7 + adcs x7, x15, x9 + adcs x15, x17, x8 + umulh x4, x4, x1 + adc x8, x13, x8 + cmn x2, #0x1 + eor x1, x12, x2 + adcs x1, x7, x1 + ldp x7, x16, [sp, #192] + eor x12, x4, x2 + adcs x4, x15, x12 + ldp x15, x12, [sp, #224] + adc x8, x8, x2 + adds x13, x14, x14 + umulh x14, x5, x10 + adcs x2, x11, x11 + adcs x3, x3, x3 + adcs x1, x1, x1 + adcs x4, x4, x4 + adcs x11, x8, x8 + adc x8, xzr, xzr + adds x13, x13, x7 + adcs x2, x2, x16 + mul x16, x5, x10 + adcs x3, x3, x19 + adcs x1, x1, x20 + umulh x5, x5, x5 + lsl x9, x13, #32 + add x9, x9, x13 + adcs x4, x4, x15 + mov x13, v28.d[1] + adcs x15, x11, x12 + lsr x7, x9, #32 + adc x11, x8, xzr + subs x7, x7, x9 + umulh x10, x10, x10 + sbc x17, x9, xzr + extr x7, x17, x7, #32 + lsr x17, x17, #32 + adds x17, x17, x9 + adc x12, xzr, xzr + subs x8, x2, x7 + sbcs x17, x3, x17 + lsl x7, x8, #32 + sbcs x2, x1, x12 + add x3, x7, x8 + sbcs x12, x4, xzr + lsr x1, x3, #32 + sbcs x7, x15, xzr + sbc x15, x9, xzr + subs x1, x1, x3 + sbc x4, x3, xzr + lsr x9, x4, #32 + extr x8, x4, x1, #32 + adds x9, x9, x3 + adc x4, xzr, xzr + subs x1, x17, x8 + lsl x17, x1, #32 + sbcs x8, x2, x9 + sbcs x9, x12, x4 + add x17, x17, x1 + mov x1, v18.d[1] + lsr x2, x17, #32 + sbcs x7, x7, xzr + mov x12, v18.d[0] + sbcs x15, x15, xzr + sbc x3, x3, xzr + subs x4, x2, x17 + sbc x2, x17, xzr + adds x12, x13, x12 + adcs x16, x16, x1 + lsr x13, x2, #32 + extr x1, x2, x4, #32 + adc x2, x14, xzr + adds x4, x13, x17 + mul x13, x6, x6 + adc x14, xzr, xzr + subs x1, x8, x1 + sbcs x4, x9, x4 + mov x9, v28.d[0] + sbcs x7, x7, x14 + sbcs x8, x15, xzr + sbcs x3, x3, xzr + sbc x14, x17, xzr + adds x17, x9, x9 + adcs x12, x12, x12 + mov x15, v19.d[0] + adcs x9, x16, x16 + umulh x6, x6, x6 + adcs x16, x2, x2 + adc x2, xzr, xzr + adds x11, x11, x8 + adcs x3, x3, xzr + adcs x14, x14, xzr + adcs x8, xzr, xzr + adds x13, x1, x13 + mov x1, v19.d[1] + adcs x6, x4, x6 + mov x4, #0xffffffff // #4294967295 + adcs x15, x7, x15 + adcs x7, x11, x5 + adcs x1, x3, x1 + adcs x14, x14, x10 + adc x11, x8, xzr + adds x6, x6, x17 + adcs x8, x15, x12 + adcs x3, x7, x9 + adcs x15, x1, x16 + mov x16, #0xffffffff00000001 // #-4294967295 + adcs x14, x14, x2 + mov x2, #0x1 // #1 + adc x17, x11, xzr + cmn x13, x16 + adcs xzr, x6, x4 + adcs xzr, x8, x2 + adcs xzr, x3, xzr + adcs xzr, x15, xzr + adcs xzr, x14, xzr + adc x1, x17, xzr + neg x9, x1 + and x1, x16, x9 + adds x19, x13, x1 + and x13, x4, x9 + adcs x20, x6, x13 + and x1, x2, x9 + adcs x7, x8, x1 + adcs x11, x3, xzr + adcs x2, x15, xzr + stp x7, x11, [sp, #208] + adc x17, x14, xzr + stp x2, x17, [sp, #224] + ldp x0, x1, [sp, #288] + mov x6, #0xffffffff // #4294967295 + subs x6, x6, x0 + mov x7, #0xffffffff00000000 // #-4294967296 + sbcs x7, x7, x1 + ldp x0, x1, [sp, #304] + mov x8, #0xfffffffffffffffe // #-2 + sbcs x8, x8, x0 + mov x13, #0xffffffffffffffff // #-1 + sbcs x9, x13, x1 + ldp x0, x1, [sp, #320] + sbcs x10, x13, x0 + sbc x11, x13, x1 + mov x12, #0x9 // #9 + mul x0, x12, x6 + mul x1, x12, x7 + mul x2, x12, x8 + mul x3, x12, x9 + mul x4, x12, x10 + mul x5, x12, x11 + umulh x6, x12, x6 + umulh x7, x12, x7 + umulh x8, x12, x8 + umulh x9, x12, x9 + umulh x10, x12, x10 + umulh x12, x12, x11 + adds x1, x1, x6 + adcs x2, x2, x7 + adcs x3, x3, x8 + adcs x4, x4, x9 + adcs x5, x5, x10 + mov x6, #0x1 // #1 + adc x6, x12, x6 + ldp x8, x9, [sp, #144] + ldp x10, x11, [sp, #160] + ldp x12, x13, [sp, #176] + mov x14, #0xc // #12 + mul x15, x14, x8 + umulh x8, x14, x8 + adds x0, x0, x15 + mul x15, x14, x9 + umulh x9, x14, x9 + adcs x1, x1, x15 + mul x15, x14, x10 + umulh x10, x14, x10 + adcs x2, x2, x15 + mul x15, x14, x11 + umulh x11, x14, x11 + adcs x3, x3, x15 + mul x15, x14, x12 + umulh x12, x14, x12 + adcs x4, x4, x15 + mul x15, x14, x13 + umulh x13, x14, x13 + adcs x5, x5, x15 + adc x6, x6, xzr + adds x1, x1, x8 + adcs x2, x2, x9 + adcs x3, x3, x10 + adcs x4, x4, x11 + adcs x5, x5, x12 + adcs x6, x6, x13 + lsl x7, x6, #32 + subs x8, x6, x7 + sbc x7, x7, xzr + adds x0, x0, x8 + adcs x1, x1, x7 + adcs x2, x2, x6 + adcs x3, x3, xzr + adcs x4, x4, xzr + adcs x5, x5, xzr + csetm x6, cc // cc = lo, ul, last + mov x7, #0xffffffff // #4294967295 + and x7, x7, x6 + adds x0, x0, x7 + eor x7, x7, x6 + adcs x1, x1, x7 + mov x7, #0xfffffffffffffffe // #-2 + and x7, x7, x6 + adcs x2, x2, x7 + adcs x3, x3, x6 + adcs x4, x4, x6 + adc x5, x5, x6 + stp x0, x1, [sp, #288] + stp x2, x3, [sp, #304] + stp x4, x5, [sp, #320] + mov x2, sp + ldp x4, x3, [x2] + subs x5, x19, x4 + sbcs x6, x20, x3 + ldp x7, x8, [sp, #208] + ldp x4, x3, [x2, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #224] + ldp x4, x3, [x2, #32] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, cc // cc = lo, ul, last + mov x4, #0xffffffff // #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #0xfffffffffffffffe // #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [sp, #240] + stp x7, x8, [sp, #256] + stp x9, x10, [sp, #272] + ldr q1, [sp, #48] + ldp x9, x2, [sp, #48] + ldr q0, [sp, #48] + ldp x4, x6, [sp, #64] + rev64 v21.4s, v1.4s + uzp2 v28.4s, v1.4s, v1.4s + umulh x7, x9, x2 + xtn v17.2s, v1.2d + mul v27.4s, v21.4s, v0.4s + ldr q20, [sp, #80] + xtn v30.2s, v0.2d + ldr q1, [sp, #80] + uzp2 v31.4s, v0.4s, v0.4s + ldp x5, x10, [sp, #80] + umulh x8, x9, x4 + uaddlp v3.2d, v27.4s + umull v16.2d, v30.2s, v17.2s + mul x16, x9, x4 + umull v27.2d, v30.2s, v28.2s + shrn v0.2s, v20.2d, #32 + xtn v7.2s, v20.2d + shl v20.2d, v3.2d, #32 + umull v3.2d, v31.2s, v28.2s + mul x3, x2, x4 + umlal v20.2d, v30.2s, v17.2s + umull v22.2d, v7.2s, v0.2s + usra v27.2d, v16.2d, #32 + umulh x11, x2, x4 + movi v21.2d, #0xffffffff + uzp2 v28.4s, v1.4s, v1.4s + adds x15, x16, x7 + and v5.16b, v27.16b, v21.16b + adcs x3, x3, x8 + usra v3.2d, v27.2d, #32 + dup v29.2d, x6 + adcs x16, x11, xzr + mov x14, v20.d[0] + umlal v5.2d, v31.2s, v17.2s + mul x8, x9, x2 + mov x7, v20.d[1] + shl v19.2d, v22.2d, #33 + xtn v25.2s, v29.2d + rev64 v31.4s, v1.4s + lsl x13, x14, #32 + uzp2 v6.4s, v29.4s, v29.4s + umlal v19.2d, v7.2s, v7.2s + usra v3.2d, v5.2d, #32 + adds x1, x8, x8 + umulh x8, x4, x4 + add x12, x13, x14 + mul v17.4s, v31.4s, v29.4s + xtn v4.2s, v1.2d + adcs x14, x15, x15 + lsr x13, x12, #32 + adcs x15, x3, x3 + umull v31.2d, v25.2s, v28.2s + adcs x11, x16, x16 + umull v21.2d, v25.2s, v4.2s + mov x17, v3.d[0] + umull v18.2d, v6.2s, v28.2s + adc x16, x8, xzr + uaddlp v16.2d, v17.4s + movi v1.2d, #0xffffffff + subs x13, x13, x12 + usra v31.2d, v21.2d, #32 + sbc x8, x12, xzr + adds x17, x17, x1 + mul x1, x4, x4 + shl v28.2d, v16.2d, #32 + mov x3, v3.d[1] + adcs x14, x7, x14 + extr x7, x8, x13, #32 + adcs x13, x3, x15 + and v3.16b, v31.16b, v1.16b + adcs x11, x1, x11 + lsr x1, x8, #32 + umlal v3.2d, v6.2s, v4.2s + usra v18.2d, v31.2d, #32 + adc x3, x16, xzr + adds x1, x1, x12 + umlal v28.2d, v25.2s, v4.2s + adc x16, xzr, xzr + subs x15, x17, x7 + sbcs x7, x14, x1 + lsl x1, x15, #32 + sbcs x16, x13, x16 + add x8, x1, x15 + usra v18.2d, v3.2d, #32 + sbcs x14, x11, xzr + lsr x1, x8, #32 + sbcs x17, x3, xzr + sbc x11, x12, xzr + subs x13, x1, x8 + umulh x12, x4, x10 + sbc x1, x8, xzr + extr x13, x1, x13, #32 + lsr x1, x1, #32 + adds x15, x1, x8 + adc x1, xzr, xzr + subs x7, x7, x13 + sbcs x13, x16, x15 + lsl x3, x7, #32 + umulh x16, x2, x5 + sbcs x15, x14, x1 + add x7, x3, x7 + sbcs x3, x17, xzr + lsr x1, x7, #32 + sbcs x14, x11, xzr + sbc x11, x8, xzr + subs x8, x1, x7 + sbc x1, x7, xzr + extr x8, x1, x8, #32 + lsr x1, x1, #32 + adds x1, x1, x7 + adc x17, xzr, xzr + subs x13, x13, x8 + umulh x8, x9, x6 + sbcs x1, x15, x1 + sbcs x19, x3, x17 + sbcs x20, x14, xzr + mul x17, x2, x5 + sbcs x11, x11, xzr + stp x13, x1, [sp, #192] + sbc x14, x7, xzr + mul x7, x4, x10 + subs x1, x9, x2 + csetm x15, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + stp x11, x14, [sp, #224] + mul x14, x9, x6 + adds x17, x8, x17 + adcs x7, x16, x7 + adc x13, x12, xzr + subs x12, x5, x6 + cneg x3, x12, cc // cc = lo, ul, last + cinv x16, x15, cc // cc = lo, ul, last + mul x8, x1, x3 + umulh x1, x1, x3 + eor x12, x8, x16 + adds x11, x17, x14 + adcs x3, x7, x17 + adcs x15, x13, x7 + adc x8, x13, xzr + adds x3, x3, x14 + adcs x15, x15, x17 + adcs x17, x8, x7 + eor x1, x1, x16 + adc x13, x13, xzr + subs x9, x9, x4 + csetm x8, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x4, x2, x4 + cneg x4, x4, cc // cc = lo, ul, last + csetm x7, cc // cc = lo, ul, last + subs x2, x10, x6 + cinv x8, x8, cc // cc = lo, ul, last + cneg x2, x2, cc // cc = lo, ul, last + cmn x16, #0x1 + adcs x11, x11, x12 + mul x12, x9, x2 + adcs x3, x3, x1 + adcs x15, x15, x16 + umulh x9, x9, x2 + adcs x17, x17, x16 + adc x13, x13, x16 + subs x1, x10, x5 + cinv x2, x7, cc // cc = lo, ul, last + cneg x1, x1, cc // cc = lo, ul, last + eor x9, x9, x8 + cmn x8, #0x1 + eor x7, x12, x8 + mul x12, x4, x1 + adcs x3, x3, x7 + adcs x7, x15, x9 + adcs x15, x17, x8 + umulh x4, x4, x1 + adc x8, x13, x8 + cmn x2, #0x1 + eor x1, x12, x2 + adcs x1, x7, x1 + ldp x7, x16, [sp, #192] + eor x12, x4, x2 + adcs x4, x15, x12 + ldp x15, x12, [sp, #224] + adc x8, x8, x2 + adds x13, x14, x14 + umulh x14, x5, x10 + adcs x2, x11, x11 + adcs x3, x3, x3 + adcs x1, x1, x1 + adcs x4, x4, x4 + adcs x11, x8, x8 + adc x8, xzr, xzr + adds x13, x13, x7 + adcs x2, x2, x16 + mul x16, x5, x10 + adcs x3, x3, x19 + adcs x1, x1, x20 + umulh x5, x5, x5 + lsl x9, x13, #32 + add x9, x9, x13 + adcs x4, x4, x15 + mov x13, v28.d[1] + adcs x15, x11, x12 + lsr x7, x9, #32 + adc x11, x8, xzr + subs x7, x7, x9 + umulh x10, x10, x10 + sbc x17, x9, xzr + extr x7, x17, x7, #32 + lsr x17, x17, #32 + adds x17, x17, x9 + adc x12, xzr, xzr + subs x8, x2, x7 + sbcs x17, x3, x17 + lsl x7, x8, #32 + sbcs x2, x1, x12 + add x3, x7, x8 + sbcs x12, x4, xzr + lsr x1, x3, #32 + sbcs x7, x15, xzr + sbc x15, x9, xzr + subs x1, x1, x3 + sbc x4, x3, xzr + lsr x9, x4, #32 + extr x8, x4, x1, #32 + adds x9, x9, x3 + adc x4, xzr, xzr + subs x1, x17, x8 + lsl x17, x1, #32 + sbcs x8, x2, x9 + sbcs x9, x12, x4 + add x17, x17, x1 + mov x1, v18.d[1] + lsr x2, x17, #32 + sbcs x7, x7, xzr + mov x12, v18.d[0] + sbcs x15, x15, xzr + sbc x3, x3, xzr + subs x4, x2, x17 + sbc x2, x17, xzr + adds x12, x13, x12 + adcs x16, x16, x1 + lsr x13, x2, #32 + extr x1, x2, x4, #32 + adc x2, x14, xzr + adds x4, x13, x17 + mul x13, x6, x6 + adc x14, xzr, xzr + subs x1, x8, x1 + sbcs x4, x9, x4 + mov x9, v28.d[0] + sbcs x7, x7, x14 + sbcs x8, x15, xzr + sbcs x3, x3, xzr + sbc x14, x17, xzr + adds x17, x9, x9 + adcs x12, x12, x12 + mov x15, v19.d[0] + adcs x9, x16, x16 + umulh x6, x6, x6 + adcs x16, x2, x2 + adc x2, xzr, xzr + adds x11, x11, x8 + adcs x3, x3, xzr + adcs x14, x14, xzr + adcs x8, xzr, xzr + adds x13, x1, x13 + mov x1, v19.d[1] + adcs x6, x4, x6 + mov x4, #0xffffffff // #4294967295 + adcs x15, x7, x15 + adcs x7, x11, x5 + adcs x1, x3, x1 + adcs x14, x14, x10 + adc x11, x8, xzr + adds x6, x6, x17 + adcs x8, x15, x12 + adcs x3, x7, x9 + adcs x15, x1, x16 + mov x16, #0xffffffff00000001 // #-4294967295 + adcs x14, x14, x2 + mov x2, #0x1 // #1 + adc x17, x11, xzr + cmn x13, x16 + adcs xzr, x6, x4 + adcs xzr, x8, x2 + adcs xzr, x3, xzr + adcs xzr, x15, xzr + adcs xzr, x14, xzr + adc x1, x17, xzr + neg x9, x1 + and x1, x16, x9 + adds x11, x13, x1 + and x13, x4, x9 + adcs x5, x6, x13 + and x1, x2, x9 + adcs x7, x8, x1 + stp x11, x5, [sp, #192] + adcs x11, x3, xzr + adcs x2, x15, xzr + stp x7, x11, [sp, #208] + adc x17, x14, xzr + stp x2, x17, [sp, #224] + ldp x5, x6, [sp, #240] + ldp x4, x3, [sp, #48] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #256] + ldp x4, x3, [sp, #64] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #272] + ldp x4, x3, [sp, #80] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, cc // cc = lo, ul, last + mov x4, #0xffffffff // #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #0xfffffffffffffffe // #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [x25, #96] + stp x7, x8, [x25, #112] + stp x9, x10, [x25, #128] + ldr q3, [sp, #288] + ldr q25, [sp, #96] + ldp x13, x23, [sp, #96] + ldp x3, x21, [sp, #288] + rev64 v23.4s, v25.4s + uzp1 v17.4s, v25.4s, v3.4s + umulh x15, x3, x13 + mul v6.4s, v23.4s, v3.4s + uzp1 v3.4s, v3.4s, v3.4s + ldr q27, [sp, #128] + ldp x8, x24, [sp, #304] + subs x6, x3, x21 + ldr q0, [sp, #320] + movi v23.2d, #0xffffffff + csetm x10, cc // cc = lo, ul, last + umulh x19, x21, x23 + rev64 v4.4s, v27.4s + uzp2 v25.4s, v27.4s, v27.4s + cneg x4, x6, cc // cc = lo, ul, last + subs x7, x23, x13 + xtn v22.2s, v0.2d + xtn v24.2s, v27.2d + cneg x20, x7, cc // cc = lo, ul, last + ldp x6, x14, [sp, #112] + mul v27.4s, v4.4s, v0.4s + uaddlp v20.2d, v6.4s + cinv x5, x10, cc // cc = lo, ul, last + mul x16, x4, x20 + uzp2 v6.4s, v0.4s, v0.4s + umull v21.2d, v22.2s, v25.2s + shl v0.2d, v20.2d, #32 + umlal v0.2d, v3.2s, v17.2s + mul x22, x8, x6 + umull v1.2d, v6.2s, v25.2s + subs x12, x3, x8 + umull v20.2d, v22.2s, v24.2s + cneg x17, x12, cc // cc = lo, ul, last + umulh x9, x8, x6 + mov x12, v0.d[1] + eor x11, x16, x5 + mov x7, v0.d[0] + csetm x10, cc // cc = lo, ul, last + usra v21.2d, v20.2d, #32 + adds x15, x15, x12 + adcs x12, x19, x22 + umulh x20, x4, x20 + adc x19, x9, xzr + usra v1.2d, v21.2d, #32 + adds x22, x15, x7 + and v26.16b, v21.16b, v23.16b + adcs x16, x12, x15 + uaddlp v25.2d, v27.4s + adcs x9, x19, x12 + umlal v26.2d, v6.2s, v24.2s + adc x4, x19, xzr + adds x16, x16, x7 + shl v27.2d, v25.2d, #32 + adcs x9, x9, x15 + adcs x4, x4, x12 + eor x12, x20, x5 + adc x15, x19, xzr + subs x20, x6, x13 + cneg x20, x20, cc // cc = lo, ul, last + cinv x10, x10, cc // cc = lo, ul, last + cmn x5, #0x1 + mul x19, x17, x20 + adcs x11, x22, x11 + adcs x12, x16, x12 + adcs x9, x9, x5 + umulh x17, x17, x20 + adcs x22, x4, x5 + adc x5, x15, x5 + subs x16, x21, x8 + cneg x20, x16, cc // cc = lo, ul, last + eor x19, x19, x10 + csetm x4, cc // cc = lo, ul, last + subs x16, x6, x23 + cneg x16, x16, cc // cc = lo, ul, last + umlal v27.2d, v22.2s, v24.2s + mul x15, x20, x16 + cinv x4, x4, cc // cc = lo, ul, last + cmn x10, #0x1 + usra v1.2d, v26.2d, #32 + adcs x19, x12, x19 + eor x17, x17, x10 + adcs x9, x9, x17 + adcs x22, x22, x10 + lsl x12, x7, #32 + umulh x20, x20, x16 + eor x16, x15, x4 + ldp x15, x17, [sp, #128] + add x2, x12, x7 + adc x7, x5, x10 + ldp x5, x10, [sp, #320] + lsr x1, x2, #32 + eor x12, x20, x4 + subs x1, x1, x2 + sbc x20, x2, xzr + cmn x4, #0x1 + adcs x9, x9, x16 + extr x1, x20, x1, #32 + lsr x20, x20, #32 + adcs x22, x22, x12 + adc x16, x7, x4 + adds x12, x20, x2 + umulh x7, x24, x14 + adc x4, xzr, xzr + subs x1, x11, x1 + sbcs x20, x19, x12 + sbcs x12, x9, x4 + lsl x9, x1, #32 + add x1, x9, x1 + sbcs x9, x22, xzr + mul x22, x24, x14 + sbcs x16, x16, xzr + lsr x4, x1, #32 + sbc x19, x2, xzr + subs x4, x4, x1 + sbc x11, x1, xzr + extr x2, x11, x4, #32 + lsr x4, x11, #32 + adds x4, x4, x1 + adc x11, xzr, xzr + subs x2, x20, x2 + sbcs x4, x12, x4 + sbcs x20, x9, x11 + lsl x12, x2, #32 + add x2, x12, x2 + sbcs x9, x16, xzr + lsr x11, x2, #32 + sbcs x19, x19, xzr + sbc x1, x1, xzr + subs x16, x11, x2 + sbc x12, x2, xzr + extr x16, x12, x16, #32 + lsr x12, x12, #32 + adds x11, x12, x2 + adc x12, xzr, xzr + subs x26, x4, x16 + mov x4, v27.d[0] + sbcs x27, x20, x11 + sbcs x20, x9, x12 + sbcs x11, x19, xzr + sbcs x9, x1, xzr + stp x20, x11, [sp, #256] + mov x1, v1.d[0] + sbc x20, x2, xzr + subs x12, x24, x5 + mov x11, v27.d[1] + cneg x16, x12, cc // cc = lo, ul, last + csetm x2, cc // cc = lo, ul, last + subs x19, x15, x14 + mov x12, v1.d[1] + cinv x2, x2, cc // cc = lo, ul, last + cneg x19, x19, cc // cc = lo, ul, last + stp x9, x20, [sp, #272] + mul x9, x16, x19 + adds x4, x7, x4 + adcs x11, x1, x11 + adc x1, x12, xzr + adds x20, x4, x22 + umulh x19, x16, x19 + adcs x7, x11, x4 + eor x16, x9, x2 + adcs x9, x1, x11 + adc x12, x1, xzr + adds x7, x7, x22 + adcs x4, x9, x4 + adcs x9, x12, x11 + adc x12, x1, xzr + cmn x2, #0x1 + eor x1, x19, x2 + adcs x11, x20, x16 + adcs x19, x7, x1 + adcs x1, x4, x2 + adcs x20, x9, x2 + adc x2, x12, x2 + subs x12, x24, x10 + cneg x16, x12, cc // cc = lo, ul, last + csetm x12, cc // cc = lo, ul, last + subs x9, x17, x14 + cinv x12, x12, cc // cc = lo, ul, last + cneg x9, x9, cc // cc = lo, ul, last + subs x3, x24, x3 + sbcs x21, x5, x21 + mul x24, x16, x9 + sbcs x4, x10, x8 + ngc x8, xzr + subs x10, x5, x10 + eor x5, x24, x12 + csetm x7, cc // cc = lo, ul, last + cneg x24, x10, cc // cc = lo, ul, last + subs x10, x17, x15 + cinv x7, x7, cc // cc = lo, ul, last + cneg x10, x10, cc // cc = lo, ul, last + subs x14, x13, x14 + sbcs x15, x23, x15 + eor x13, x21, x8 + mul x23, x24, x10 + sbcs x17, x6, x17 + eor x6, x3, x8 + ngc x21, xzr + umulh x9, x16, x9 + cmn x8, #0x1 + eor x3, x23, x7 + adcs x23, x6, xzr + adcs x13, x13, xzr + eor x16, x4, x8 + adc x16, x16, xzr + eor x4, x17, x21 + umulh x17, x24, x10 + cmn x21, #0x1 + eor x24, x14, x21 + eor x6, x15, x21 + adcs x15, x24, xzr + adcs x14, x6, xzr + adc x6, x4, xzr + cmn x12, #0x1 + eor x4, x9, x12 + adcs x19, x19, x5 + umulh x5, x23, x15 + adcs x1, x1, x4 + adcs x10, x20, x12 + eor x4, x17, x7 + adc x2, x2, x12 + cmn x7, #0x1 + adcs x12, x1, x3 + ldp x17, x24, [sp, #256] + mul x1, x16, x6 + adcs x3, x10, x4 + adc x2, x2, x7 + ldp x7, x4, [sp, #272] + adds x20, x22, x26 + mul x10, x13, x14 + adcs x11, x11, x27 + eor x9, x8, x21 + adcs x26, x19, x17 + stp x20, x11, [sp, #240] + adcs x27, x12, x24 + mul x8, x23, x15 + adcs x3, x3, x7 + adcs x12, x2, x4 + adc x19, xzr, xzr + subs x21, x23, x16 + umulh x2, x16, x6 + stp x3, x12, [sp, #272] + cneg x3, x21, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + umulh x11, x13, x14 + subs x21, x13, x16 + eor x7, x8, x9 + cneg x17, x21, cc // cc = lo, ul, last + csetm x16, cc // cc = lo, ul, last + subs x21, x6, x15 + cneg x22, x21, cc // cc = lo, ul, last + cinv x21, x24, cc // cc = lo, ul, last + subs x20, x23, x13 + umulh x12, x3, x22 + cneg x23, x20, cc // cc = lo, ul, last + csetm x24, cc // cc = lo, ul, last + subs x20, x14, x15 + cinv x24, x24, cc // cc = lo, ul, last + mul x22, x3, x22 + cneg x3, x20, cc // cc = lo, ul, last + subs x13, x6, x14 + cneg x20, x13, cc // cc = lo, ul, last + cinv x15, x16, cc // cc = lo, ul, last + adds x13, x5, x10 + mul x4, x23, x3 + adcs x11, x11, x1 + adc x14, x2, xzr + adds x5, x13, x8 + adcs x16, x11, x13 + umulh x23, x23, x3 + adcs x3, x14, x11 + adc x1, x14, xzr + adds x10, x16, x8 + adcs x6, x3, x13 + adcs x8, x1, x11 + umulh x13, x17, x20 + eor x1, x4, x24 + adc x4, x14, xzr + cmn x24, #0x1 + adcs x1, x5, x1 + eor x16, x23, x24 + eor x11, x1, x9 + adcs x23, x10, x16 + eor x2, x22, x21 + adcs x3, x6, x24 + mul x14, x17, x20 + eor x17, x13, x15 + adcs x13, x8, x24 + adc x8, x4, x24 + cmn x21, #0x1 + adcs x6, x23, x2 + mov x16, #0xfffffffffffffffe // #-2 + eor x20, x12, x21 + adcs x20, x3, x20 + eor x23, x14, x15 + adcs x2, x13, x21 + adc x8, x8, x21 + cmn x15, #0x1 + ldp x5, x4, [sp, #240] + adcs x22, x20, x23 + eor x23, x22, x9 + adcs x17, x2, x17 + adc x22, x8, x15 + cmn x9, #0x1 + adcs x15, x7, x5 + ldp x10, x14, [sp, #272] + eor x1, x6, x9 + lsl x2, x15, #32 + adcs x8, x11, x4 + adcs x13, x1, x26 + eor x1, x22, x9 + adcs x24, x23, x27 + eor x11, x17, x9 + adcs x23, x11, x10 + adcs x7, x1, x14 + adcs x17, x9, x19 + adcs x20, x9, xzr + add x1, x2, x15 + lsr x3, x1, #32 + adcs x11, x9, xzr + adc x9, x9, xzr + subs x3, x3, x1 + sbc x6, x1, xzr + adds x24, x24, x5 + adcs x4, x23, x4 + extr x3, x6, x3, #32 + lsr x6, x6, #32 + adcs x21, x7, x26 + adcs x15, x17, x27 + adcs x7, x20, x10 + adcs x20, x11, x14 + mov x14, #0xffffffff // #4294967295 + adc x22, x9, x19 + adds x12, x6, x1 + adc x10, xzr, xzr + subs x3, x8, x3 + sbcs x12, x13, x12 + lsl x9, x3, #32 + add x3, x9, x3 + sbcs x10, x24, x10 + sbcs x24, x4, xzr + lsr x9, x3, #32 + sbcs x21, x21, xzr + sbc x1, x1, xzr + subs x9, x9, x3 + sbc x13, x3, xzr + extr x9, x13, x9, #32 + lsr x13, x13, #32 + adds x13, x13, x3 + adc x6, xzr, xzr + subs x12, x12, x9 + sbcs x17, x10, x13 + lsl x2, x12, #32 + sbcs x10, x24, x6 + add x9, x2, x12 + sbcs x6, x21, xzr + lsr x5, x9, #32 + sbcs x21, x1, xzr + sbc x13, x3, xzr + subs x8, x5, x9 + sbc x19, x9, xzr + lsr x12, x19, #32 + extr x3, x19, x8, #32 + adds x8, x12, x9 + adc x1, xzr, xzr + subs x2, x17, x3 + sbcs x12, x10, x8 + sbcs x5, x6, x1 + sbcs x3, x21, xzr + sbcs x19, x13, xzr + sbc x24, x9, xzr + adds x23, x15, x3 + adcs x8, x7, x19 + adcs x11, x20, x24 + adc x9, x22, xzr + add x24, x9, #0x1 + lsl x7, x24, #32 + subs x21, x24, x7 + sbc x10, x7, xzr + adds x6, x2, x21 + adcs x7, x12, x10 + adcs x24, x5, x24 + adcs x13, x23, xzr + adcs x8, x8, xzr + adcs x15, x11, xzr + csetm x23, cc // cc = lo, ul, last + and x11, x16, x23 + and x20, x14, x23 + adds x22, x6, x20 + eor x3, x20, x23 + adcs x5, x7, x3 + adcs x14, x24, x11 + stp x22, x5, [sp, #240] + adcs x5, x13, x23 + adcs x12, x8, x23 + stp x14, x5, [sp, #256] + adc x19, x15, x23 + ldp x1, x2, [sp, #144] + ldp x3, x4, [sp, #160] + ldp x5, x6, [sp, #176] + lsl x0, x1, #2 + ldp x7, x8, [sp, #288] + subs x0, x0, x7 + extr x1, x2, x1, #62 + sbcs x1, x1, x8 + ldp x7, x8, [sp, #304] + extr x2, x3, x2, #62 + sbcs x2, x2, x7 + extr x3, x4, x3, #62 + sbcs x3, x3, x8 + extr x4, x5, x4, #62 + ldp x7, x8, [sp, #320] + sbcs x4, x4, x7 + extr x5, x6, x5, #62 + sbcs x5, x5, x8 + lsr x6, x6, #62 + adc x6, x6, xzr + lsl x7, x6, #32 + subs x8, x6, x7 + sbc x7, x7, xzr + adds x0, x0, x8 + adcs x1, x1, x7 + adcs x2, x2, x6 + adcs x3, x3, xzr + adcs x4, x4, xzr + adcs x5, x5, xzr + csetm x8, cc // cc = lo, ul, last + mov x9, #0xffffffff // #4294967295 + and x9, x9, x8 + adds x0, x0, x9 + eor x9, x9, x8 + adcs x1, x1, x9 + mov x9, #0xfffffffffffffffe // #-2 + and x9, x9, x8 + adcs x2, x2, x9 + adcs x3, x3, x8 + adcs x4, x4, x8 + adc x5, x5, x8 + stp x0, x1, [x25] + stp x2, x3, [x25, #16] + stp x4, x5, [x25, #32] + ldp x0, x1, [sp, #192] + mov x6, #0xffffffff // #4294967295 + subs x6, x6, x0 + mov x7, #0xffffffff00000000 // #-4294967296 + sbcs x7, x7, x1 + ldp x0, x1, [sp, #208] + mov x8, #0xfffffffffffffffe // #-2 + sbcs x8, x8, x0 + mov x13, #0xffffffffffffffff // #-1 + sbcs x9, x13, x1 + ldp x0, x1, [sp, #224] + sbcs x10, x13, x0 + sbc x11, x13, x1 + lsl x0, x6, #3 + extr x1, x7, x6, #61 + extr x2, x8, x7, #61 + extr x3, x9, x8, #61 + extr x4, x10, x9, #61 + extr x5, x11, x10, #61 + lsr x6, x11, #61 + add x6, x6, #0x1 + ldp x8, x9, [sp, #240] + ldp x10, x11, [sp, #256] + mov x14, #0x3 // #3 + mul x15, x14, x8 + umulh x8, x14, x8 + adds x0, x0, x15 + mul x15, x14, x9 + umulh x9, x14, x9 + adcs x1, x1, x15 + mul x15, x14, x10 + umulh x10, x14, x10 + adcs x2, x2, x15 + mul x15, x14, x11 + umulh x11, x14, x11 + adcs x3, x3, x15 + mul x15, x14, x12 + umulh x12, x14, x12 + adcs x4, x4, x15 + mul x15, x14, x19 + umulh x13, x14, x19 + adcs x5, x5, x15 + adc x6, x6, xzr + adds x1, x1, x8 + adcs x2, x2, x9 + adcs x3, x3, x10 + adcs x4, x4, x11 + adcs x5, x5, x12 + adcs x6, x6, x13 + lsl x7, x6, #32 + subs x8, x6, x7 + sbc x7, x7, xzr + adds x0, x0, x8 + adcs x1, x1, x7 + adcs x2, x2, x6 + adcs x3, x3, xzr + adcs x4, x4, xzr + adcs x5, x5, xzr + csetm x6, cc // cc = lo, ul, last + mov x7, #0xffffffff // #4294967295 + and x7, x7, x6 + adds x0, x0, x7 + eor x7, x7, x6 + adcs x1, x1, x7 + mov x7, #0xfffffffffffffffe // #-2 + and x7, x7, x6 + adcs x2, x2, x7 + adcs x3, x3, x6 + adcs x4, x4, x6 + adc x5, x5, x6 + stp x0, x1, [x25, #48] + stp x2, x3, [x25, #64] + stp x4, x5, [x25, #80] + ldp x19, x20, [sp, #336] + ldp x21, x22, [sp, #352] + ldp x23, x24, [sp, #368] + ldp x25, x26, [sp, #384] + ldp x27, xzr, [sp, #400] + CFI_INC_SP(416) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp384_montjscalarmul_p384_montjdouble) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/p384_montjscalarmul_alt.S b/cbits/s2n/arm/p384_montjscalarmul_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/p384_montjscalarmul_alt.S @@ -0,0 +1,7155 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Montgomery-Jacobian form scalar multiplication for P-384 +// Input scalar[6], point[18]; output res[18] +// +// extern void p384_montjscalarmul_alt +// (uint64_t res[static 18], +// const uint64_t scalar[static 6], +// const uint64_t point[static 18]); +// +// This function is a variant of its affine point version p384_scalarmul_alt. +// Here, input and output points are assumed to be in Jacobian form with +// their coordinates in the Montgomery domain. Thus, if priming indicates +// Montgomery form, x' = (2^384 * x) mod p_384 etc., each point argument +// is a triple (x',y',z') representing the affine point (x/z^2,y/z^3) when +// z' is nonzero or the point at infinity (group identity) if z' = 0. +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-384, returns a representation of n * P. If the result is the +// point at infinity (either because the input point was or because the +// scalar was a multiple of p_384) then the output is guaranteed to +// represent the point at infinity, i.e. to have its z coordinate zero. +// +// Standard ARM ABI: X0 = res, X1 = scalar, X2 = point +// ---------------------------------------------------------------------------- + + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p384_montjscalarmul_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p384_montjscalarmul_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p384_montjscalarmul_alt) + + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 48 +#define JACSIZE (3*NUMSIZE) + +// Safe copies of input res and additional values in variables. + +#define bf x22 +#define sgn x23 +#define j x24 +#define res x25 + +// Intermediate variables on the stack. +// The table is 16 entries, each of size JACSIZE = 3 * NUMSIZE + +#define scalarb sp, #(0*NUMSIZE) +#define acc sp, #(1*NUMSIZE) +#define tabent sp, #(4*NUMSIZE) + +#define tab sp, #(7*NUMSIZE) + +#define NSPACE 55*NUMSIZE + +// Avoid using .rep for the sake of the BoringSSL/AWS-LC delocator, +// which doesn't accept repetitions, assembler macros etc. + +#define selectblock(I) \ + cmp bf, #(1*I) __LF \ + ldp x20, x21, [x19] __LF \ + csel x0, x20, x0, eq __LF \ + csel x1, x21, x1, eq __LF \ + ldp x20, x21, [x19, #16] __LF \ + csel x2, x20, x2, eq __LF \ + csel x3, x21, x3, eq __LF \ + ldp x20, x21, [x19, #32] __LF \ + csel x4, x20, x4, eq __LF \ + csel x5, x21, x5, eq __LF \ + ldp x20, x21, [x19, #48] __LF \ + csel x6, x20, x6, eq __LF \ + csel x7, x21, x7, eq __LF \ + ldp x20, x21, [x19, #64] __LF \ + csel x8, x20, x8, eq __LF \ + csel x9, x21, x9, eq __LF \ + ldp x20, x21, [x19, #80] __LF \ + csel x10, x20, x10, eq __LF \ + csel x11, x21, x11, eq __LF \ + ldp x20, x21, [x19, #96] __LF \ + csel x12, x20, x12, eq __LF \ + csel x13, x21, x13, eq __LF \ + ldp x20, x21, [x19, #112] __LF \ + csel x14, x20, x14, eq __LF \ + csel x15, x21, x15, eq __LF \ + ldp x20, x21, [x19, #128] __LF \ + csel x16, x20, x16, eq __LF \ + csel x17, x21, x17, eq __LF \ + add x19, x19, #JACSIZE + +// Loading large constants + +#define movbig(nn,n3,n2,n1,n0) \ + movz nn, n0 __LF \ + movk nn, n1, lsl #16 __LF \ + movk nn, n2, lsl #32 __LF \ + movk nn, n3, lsl #48 + +S2N_BN_SYMBOL(p384_montjscalarmul_alt): + CFI_START + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x30) + CFI_DEC_SP(NSPACE) + +// Preserve the "res" input argument; others get processed early. + + mov res, x0 + +// Reduce the input scalar mod n_384, i.e. conditionally subtract n_384. +// Store it to "scalarb". + + ldp x3, x4, [x1] + movbig(x15, #0xecec, #0x196a, #0xccc5, #0x2973) + ldp x5, x6, [x1, #16] + movbig(x16, #0x581a, #0x0db2, #0x48b0, #0xa77a) + ldp x7, x8, [x1, #32] + movbig(x17, #0xc763, #0x4d81, #0xf437, #0x2ddf) + + subs x9, x3, x15 + sbcs x10, x4, x16 + sbcs x11, x5, x17 + adcs x12, x6, xzr + adcs x13, x7, xzr + adcs x14, x8, xzr + + csel x3, x3, x9, cc + csel x4, x4, x10, cc + csel x5, x5, x11, cc + csel x6, x6, x12, cc + csel x7, x7, x13, cc + csel x8, x8, x14, cc + + stp x3, x4, [scalarb] + stp x5, x6, [scalarb+16] + stp x7, x8, [scalarb+32] + +// Set the tab[0] table entry to the input point = 1 * P + + ldp x10, x11, [x2] + stp x10, x11, [tab] + ldp x12, x13, [x2, #16] + stp x12, x13, [tab+16] + ldp x14, x15, [x2, #32] + stp x14, x15, [tab+32] + + ldp x10, x11, [x2, #48] + stp x10, x11, [tab+48] + ldp x12, x13, [x2, #64] + stp x12, x13, [tab+64] + ldp x14, x15, [x2, #80] + stp x14, x15, [tab+80] + + ldp x10, x11, [x2, #96] + stp x10, x11, [tab+96] + ldp x12, x13, [x2, #112] + stp x12, x13, [tab+112] + ldp x14, x15, [x2, #128] + stp x14, x15, [tab+128] + +// Compute and record tab[1] = 2 * p, ..., tab[15] = 16 * P + + add x0, tab+JACSIZE*1 + add x1, tab + CFI_BL(Lp384_montjscalarmul_alt_p384_montjdouble) + + add x0, tab+JACSIZE*2 + add x1, tab+JACSIZE*1 + add x2, tab + CFI_BL(Lp384_montjscalarmul_alt_p384_montjadd) + + add x0, tab+JACSIZE*3 + add x1, tab+JACSIZE*1 + CFI_BL(Lp384_montjscalarmul_alt_p384_montjdouble) + + add x0, tab+JACSIZE*4 + add x1, tab+JACSIZE*3 + add x2, tab + CFI_BL(Lp384_montjscalarmul_alt_p384_montjadd) + + add x0, tab+JACSIZE*5 + add x1, tab+JACSIZE*2 + CFI_BL(Lp384_montjscalarmul_alt_p384_montjdouble) + + add x0, tab+JACSIZE*6 + add x1, tab+JACSIZE*5 + add x2, tab + CFI_BL(Lp384_montjscalarmul_alt_p384_montjadd) + + add x0, tab+JACSIZE*7 + add x1, tab+JACSIZE*3 + CFI_BL(Lp384_montjscalarmul_alt_p384_montjdouble) + + add x0, tab+JACSIZE*8 + add x1, tab+JACSIZE*7 + add x2, tab + CFI_BL(Lp384_montjscalarmul_alt_p384_montjadd) + + add x0, tab+JACSIZE*9 + add x1, tab+JACSIZE*4 + CFI_BL(Lp384_montjscalarmul_alt_p384_montjdouble) + + add x0, tab+JACSIZE*10 + add x1, tab+JACSIZE*9 + add x2, tab + CFI_BL(Lp384_montjscalarmul_alt_p384_montjadd) + + add x0, tab+JACSIZE*11 + add x1, tab+JACSIZE*5 + CFI_BL(Lp384_montjscalarmul_alt_p384_montjdouble) + + add x0, tab+JACSIZE*12 + add x1, tab+JACSIZE*11 + add x2, tab + CFI_BL(Lp384_montjscalarmul_alt_p384_montjadd) + + add x0, tab+JACSIZE*13 + add x1, tab+JACSIZE*6 + CFI_BL(Lp384_montjscalarmul_alt_p384_montjdouble) + + add x0, tab+JACSIZE*14 + add x1, tab+JACSIZE*13 + add x2, tab + CFI_BL(Lp384_montjscalarmul_alt_p384_montjadd) + + add x0, tab+JACSIZE*15 + add x1, tab+JACSIZE*7 + CFI_BL(Lp384_montjscalarmul_alt_p384_montjdouble) + +// Add the recoding constant sum_i(16 * 32^i) to the scalar to allow signed +// digits. The digits of the constant, in lowest-to-highest order, are as +// follows; they are generated dynamically since none is a simple ARM load. +// +// 0x0842108421084210 +// 0x1084210842108421 +// 0x2108421084210842 +// 0x4210842108421084 +// 0x8421084210842108 +// 0x0842108421084210 + + ldp x0, x1, [scalarb] + ldp x2, x3, [scalarb+16] + ldp x4, x5, [scalarb+32] + movbig(x8, #0x1084, #0x2108, #0x4210, #0x8421) + adds x0, x0, x8, lsr #1 + adcs x1, x1, x8 + lsl x8, x8, #1 + adcs x2, x2, x8 + lsl x8, x8, #1 + adcs x3, x3, x8 + lsl x8, x8, #1 + adcs x4, x4, x8 + lsr x8, x8, #4 + adcs x5, x5, x8 + cset x6, cs + +// Record the top bitfield then shift the whole scalar left 4 bits +// to align the top of the next bitfield with the MSB (bits 379..383). + + extr bf, x6, x5, #60 + extr x5, x5, x4, #60 + extr x4, x4, x3, #60 + extr x3, x3, x2, #60 + extr x2, x2, x1, #60 + extr x1, x1, x0, #60 + lsl x0, x0, #4 + stp x0, x1, [scalarb] + stp x2, x3, [scalarb+16] + stp x4, x5, [scalarb+32] + +// Initialize the accumulator to the corresponding entry using constant-time +// lookup in the table. This top digit, uniquely, is not recoded so there is +// no sign adjustment to make. + + mov x0, xzr + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, xzr + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + mov x9, xzr + mov x10, xzr + mov x11, xzr + mov x12, xzr + mov x13, xzr + mov x14, xzr + mov x15, xzr + mov x16, xzr + mov x17, xzr + + add x19, tab + + selectblock(1) + selectblock(2) + selectblock(3) + selectblock(4) + selectblock(5) + selectblock(6) + selectblock(7) + selectblock(8) + selectblock(9) + selectblock(10) + selectblock(11) + selectblock(12) + selectblock(13) + selectblock(14) + selectblock(15) + selectblock(16) + + stp x0, x1, [acc] + stp x2, x3, [acc+16] + stp x4, x5, [acc+32] + stp x6, x7, [acc+48] + stp x8, x9, [acc+64] + stp x10, x11, [acc+80] + stp x12, x13, [acc+96] + stp x14, x15, [acc+112] + stp x16, x17, [acc+128] + + mov j, #380 + +// Main loop over size-5 bitfields: double 5 times then add signed digit +// At each stage we shift the scalar left by 5 bits so we can simply pick +// the top 5 bits as the bitfield, saving some fiddle over indexing. + +Lp384_montjscalarmul_alt_mainloop: + sub j, j, #5 + + add x0, acc + add x1, acc + CFI_BL(Lp384_montjscalarmul_alt_p384_montjdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp384_montjscalarmul_alt_p384_montjdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp384_montjscalarmul_alt_p384_montjdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp384_montjscalarmul_alt_p384_montjdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp384_montjscalarmul_alt_p384_montjdouble) + +// Choose the bitfield and adjust it to sign and magnitude + + ldp x0, x1, [scalarb] + ldp x2, x3, [scalarb+16] + ldp x4, x5, [scalarb+32] + lsr bf, x5, #59 + extr x5, x5, x4, #59 + extr x4, x4, x3, #59 + extr x3, x3, x2, #59 + extr x2, x2, x1, #59 + extr x1, x1, x0, #59 + lsl x0, x0, #5 + stp x0, x1, [scalarb] + stp x2, x3, [scalarb+16] + stp x4, x5, [scalarb+32] + + subs bf, bf, #16 + cset sgn, lo // sgn = sign of digit (1 = negative) + cneg bf, bf, lo // bf = absolute value of digit + +// Conditionally select the table entry tab[i-1] = i * P in constant time + + mov x0, xzr + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, xzr + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + mov x9, xzr + mov x10, xzr + mov x11, xzr + mov x12, xzr + mov x13, xzr + mov x14, xzr + mov x15, xzr + mov x16, xzr + mov x17, xzr + + add x19, tab + + selectblock(1) + selectblock(2) + selectblock(3) + selectblock(4) + selectblock(5) + selectblock(6) + selectblock(7) + selectblock(8) + selectblock(9) + selectblock(10) + selectblock(11) + selectblock(12) + selectblock(13) + selectblock(14) + selectblock(15) + selectblock(16) + +// Store it to "tabent" with the y coordinate optionally negated. +// This is done carefully to give coordinates < p_384 even in +// the degenerate case y = 0 (when z = 0 for points on the curve). + + stp x0, x1, [tabent] + stp x2, x3, [tabent+16] + stp x4, x5, [tabent+32] + + stp x12, x13, [tabent+96] + stp x14, x15, [tabent+112] + stp x16, x17, [tabent+128] + + mov x0, #0x00000000ffffffff + subs x0, x0, x6 + orr x12, x6, x7 + mov x1, #0xffffffff00000000 + sbcs x1, x1, x7 + orr x13, x8, x9 + mov x2, #0xfffffffffffffffe + sbcs x2, x2, x8 + orr x14, x10, x11 + mov x5, #0xffffffffffffffff + sbcs x3, x5, x9 + orr x12, x12, x13 + sbcs x4, x5, x10 + orr x12, x12, x14 + sbcs x5, x5, x11 + + cmp sgn, xzr + ccmp x12, xzr, #4, ne + + csel x6, x0, x6, ne + csel x7, x1, x7, ne + csel x8, x2, x8, ne + csel x9, x3, x9, ne + csel x10, x4, x10, ne + csel x11, x5, x11, ne + + stp x6, x7, [tabent+48] + stp x8, x9, [tabent+64] + stp x10, x11, [tabent+80] + +// Add to the accumulator + + add x0, acc + add x1, acc + add x2, tabent + CFI_BL(Lp384_montjscalarmul_alt_p384_montjadd) + + cbnz j, Lp384_montjscalarmul_alt_mainloop + +// That's the end of the main loop, and we just need to copy the +// result in "acc" to the output. + + ldp x0, x1, [acc] + stp x0, x1, [res] + ldp x0, x1, [acc+16] + stp x0, x1, [res, #16] + ldp x0, x1, [acc+32] + stp x0, x1, [res, #32] + ldp x0, x1, [acc+48] + stp x0, x1, [res, #48] + ldp x0, x1, [acc+64] + stp x0, x1, [res, #64] + ldp x0, x1, [acc+80] + stp x0, x1, [res, #80] + ldp x0, x1, [acc+96] + stp x0, x1, [res, #96] + ldp x0, x1, [acc+112] + stp x0, x1, [res, #112] + ldp x0, x1, [acc+128] + stp x0, x1, [res, #128] + +// Restore stack and registers and return + + CFI_INC_SP(NSPACE) + CFI_POP2(x25,x30) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p384_montjscalarmul_alt) + +// Local copies of subroutines, complete clones at the moment + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp384_montjscalarmul_alt_p384_montjadd) + +Lp384_montjscalarmul_alt_p384_montjadd: + CFI_START + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x26) + CFI_DEC_SP(336) + mov x24, x0 + mov x25, x1 + mov x26, x2 + ldp x2, x3, [x25, #96] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [x25, #112] + mul x8, x2, x4 + adds x10, x10, x8 + mul x11, x2, x5 + mul x8, x3, x4 + adcs x11, x11, x8 + umulh x12, x2, x5 + mul x8, x3, x5 + adcs x12, x12, x8 + ldp x6, x7, [x25, #128] + mul x13, x2, x7 + mul x8, x3, x6 + adcs x13, x13, x8 + umulh x14, x2, x7 + mul x8, x3, x7 + adcs x14, x14, x8 + mul x15, x5, x6 + adcs x15, x15, xzr + umulh x16, x5, x6 + adc x16, x16, xzr + umulh x8, x2, x4 + adds x11, x11, x8 + umulh x8, x3, x4 + adcs x12, x12, x8 + umulh x8, x3, x5 + adcs x13, x13, x8 + umulh x8, x3, x6 + adcs x14, x14, x8 + umulh x8, x3, x7 + adcs x15, x15, x8 + adc x16, x16, xzr + mul x8, x2, x6 + adds x12, x12, x8 + mul x8, x4, x5 + adcs x13, x13, x8 + mul x8, x4, x6 + adcs x14, x14, x8 + mul x8, x4, x7 + adcs x15, x15, x8 + mul x8, x5, x7 + adcs x16, x16, x8 + mul x17, x6, x7 + adcs x17, x17, xzr + umulh x19, x6, x7 + adc x19, x19, xzr + umulh x8, x2, x6 + adds x13, x13, x8 + umulh x8, x4, x5 + adcs x14, x14, x8 + umulh x8, x4, x6 + adcs x15, x15, x8 + umulh x8, x4, x7 + adcs x16, x16, x8 + umulh x8, x5, x7 + adcs x17, x17, x8 + adc x19, x19, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + adcs x15, x15, x15 + adcs x16, x16, x16 + adcs x17, x17, x17 + adcs x19, x19, x19 + cset x20, hs + umulh x8, x2, x2 + mul x2, x2, x2 + adds x9, x9, x8 + mul x8, x3, x3 + adcs x10, x10, x8 + umulh x8, x3, x3 + adcs x11, x11, x8 + mul x8, x4, x4 + adcs x12, x12, x8 + umulh x8, x4, x4 + adcs x13, x13, x8 + mul x8, x5, x5 + adcs x14, x14, x8 + umulh x8, x5, x5 + adcs x15, x15, x8 + mul x8, x6, x6 + adcs x16, x16, x8 + umulh x8, x6, x6 + adcs x17, x17, x8 + mul x8, x7, x7 + adcs x19, x19, x8 + umulh x8, x7, x7 + adc x20, x20, x8 + lsl x5, x2, #32 + add x2, x5, x2 + mov x5, #-4294967295 + umulh x5, x5, x2 + mov x4, #4294967295 + mul x3, x4, x2 + umulh x4, x4, x2 + adds x5, x5, x3 + adcs x4, x4, x2 + adc x3, xzr, xzr + subs x9, x9, x5 + sbcs x10, x10, x4 + sbcs x11, x11, x3 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x2, x2, xzr + lsl x5, x9, #32 + add x9, x5, x9 + mov x5, #-4294967295 + umulh x5, x5, x9 + mov x4, #4294967295 + mul x3, x4, x9 + umulh x4, x4, x9 + adds x5, x5, x3 + adcs x4, x4, x9 + adc x3, xzr, xzr + subs x10, x10, x5 + sbcs x11, x11, x4 + sbcs x12, x12, x3 + sbcs x13, x13, xzr + sbcs x2, x2, xzr + sbc x9, x9, xzr + lsl x5, x10, #32 + add x10, x5, x10 + mov x5, #-4294967295 + umulh x5, x5, x10 + mov x4, #4294967295 + mul x3, x4, x10 + umulh x4, x4, x10 + adds x5, x5, x3 + adcs x4, x4, x10 + adc x3, xzr, xzr + subs x11, x11, x5 + sbcs x12, x12, x4 + sbcs x13, x13, x3 + sbcs x2, x2, xzr + sbcs x9, x9, xzr + sbc x10, x10, xzr + lsl x5, x11, #32 + add x11, x5, x11 + mov x5, #-4294967295 + umulh x5, x5, x11 + mov x4, #4294967295 + mul x3, x4, x11 + umulh x4, x4, x11 + adds x5, x5, x3 + adcs x4, x4, x11 + adc x3, xzr, xzr + subs x12, x12, x5 + sbcs x13, x13, x4 + sbcs x2, x2, x3 + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbc x11, x11, xzr + lsl x5, x12, #32 + add x12, x5, x12 + mov x5, #-4294967295 + umulh x5, x5, x12 + mov x4, #4294967295 + mul x3, x4, x12 + umulh x4, x4, x12 + adds x5, x5, x3 + adcs x4, x4, x12 + adc x3, xzr, xzr + subs x13, x13, x5 + sbcs x2, x2, x4 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbc x12, x12, xzr + lsl x5, x13, #32 + add x13, x5, x13 + mov x5, #-4294967295 + umulh x5, x5, x13 + mov x4, #4294967295 + mul x3, x4, x13 + umulh x4, x4, x13 + adds x5, x5, x3 + adcs x4, x4, x13 + adc x3, xzr, xzr + subs x2, x2, x5 + sbcs x9, x9, x4 + sbcs x10, x10, x3 + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + adds x2, x2, x14 + adcs x9, x9, x15 + adcs x10, x10, x16 + adcs x11, x11, x17 + adcs x12, x12, x19 + adcs x13, x13, x20 + mov x14, #-4294967295 + mov x15, #4294967295 + csel x14, x14, xzr, hs + csel x15, x15, xzr, hs + cset x16, hs + adds x2, x2, x14 + adcs x9, x9, x15 + adcs x10, x10, x16 + adcs x11, x11, xzr + adcs x12, x12, xzr + adc x13, x13, xzr + stp x2, x9, [sp] + stp x10, x11, [sp, #16] + stp x12, x13, [sp, #32] + ldp x2, x3, [x26, #96] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [x26, #112] + mul x8, x2, x4 + adds x10, x10, x8 + mul x11, x2, x5 + mul x8, x3, x4 + adcs x11, x11, x8 + umulh x12, x2, x5 + mul x8, x3, x5 + adcs x12, x12, x8 + ldp x6, x7, [x26, #128] + mul x13, x2, x7 + mul x8, x3, x6 + adcs x13, x13, x8 + umulh x14, x2, x7 + mul x8, x3, x7 + adcs x14, x14, x8 + mul x15, x5, x6 + adcs x15, x15, xzr + umulh x16, x5, x6 + adc x16, x16, xzr + umulh x8, x2, x4 + adds x11, x11, x8 + umulh x8, x3, x4 + adcs x12, x12, x8 + umulh x8, x3, x5 + adcs x13, x13, x8 + umulh x8, x3, x6 + adcs x14, x14, x8 + umulh x8, x3, x7 + adcs x15, x15, x8 + adc x16, x16, xzr + mul x8, x2, x6 + adds x12, x12, x8 + mul x8, x4, x5 + adcs x13, x13, x8 + mul x8, x4, x6 + adcs x14, x14, x8 + mul x8, x4, x7 + adcs x15, x15, x8 + mul x8, x5, x7 + adcs x16, x16, x8 + mul x17, x6, x7 + adcs x17, x17, xzr + umulh x19, x6, x7 + adc x19, x19, xzr + umulh x8, x2, x6 + adds x13, x13, x8 + umulh x8, x4, x5 + adcs x14, x14, x8 + umulh x8, x4, x6 + adcs x15, x15, x8 + umulh x8, x4, x7 + adcs x16, x16, x8 + umulh x8, x5, x7 + adcs x17, x17, x8 + adc x19, x19, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + adcs x15, x15, x15 + adcs x16, x16, x16 + adcs x17, x17, x17 + adcs x19, x19, x19 + cset x20, hs + umulh x8, x2, x2 + mul x2, x2, x2 + adds x9, x9, x8 + mul x8, x3, x3 + adcs x10, x10, x8 + umulh x8, x3, x3 + adcs x11, x11, x8 + mul x8, x4, x4 + adcs x12, x12, x8 + umulh x8, x4, x4 + adcs x13, x13, x8 + mul x8, x5, x5 + adcs x14, x14, x8 + umulh x8, x5, x5 + adcs x15, x15, x8 + mul x8, x6, x6 + adcs x16, x16, x8 + umulh x8, x6, x6 + adcs x17, x17, x8 + mul x8, x7, x7 + adcs x19, x19, x8 + umulh x8, x7, x7 + adc x20, x20, x8 + lsl x5, x2, #32 + add x2, x5, x2 + mov x5, #-4294967295 + umulh x5, x5, x2 + mov x4, #4294967295 + mul x3, x4, x2 + umulh x4, x4, x2 + adds x5, x5, x3 + adcs x4, x4, x2 + adc x3, xzr, xzr + subs x9, x9, x5 + sbcs x10, x10, x4 + sbcs x11, x11, x3 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x2, x2, xzr + lsl x5, x9, #32 + add x9, x5, x9 + mov x5, #-4294967295 + umulh x5, x5, x9 + mov x4, #4294967295 + mul x3, x4, x9 + umulh x4, x4, x9 + adds x5, x5, x3 + adcs x4, x4, x9 + adc x3, xzr, xzr + subs x10, x10, x5 + sbcs x11, x11, x4 + sbcs x12, x12, x3 + sbcs x13, x13, xzr + sbcs x2, x2, xzr + sbc x9, x9, xzr + lsl x5, x10, #32 + add x10, x5, x10 + mov x5, #-4294967295 + umulh x5, x5, x10 + mov x4, #4294967295 + mul x3, x4, x10 + umulh x4, x4, x10 + adds x5, x5, x3 + adcs x4, x4, x10 + adc x3, xzr, xzr + subs x11, x11, x5 + sbcs x12, x12, x4 + sbcs x13, x13, x3 + sbcs x2, x2, xzr + sbcs x9, x9, xzr + sbc x10, x10, xzr + lsl x5, x11, #32 + add x11, x5, x11 + mov x5, #-4294967295 + umulh x5, x5, x11 + mov x4, #4294967295 + mul x3, x4, x11 + umulh x4, x4, x11 + adds x5, x5, x3 + adcs x4, x4, x11 + adc x3, xzr, xzr + subs x12, x12, x5 + sbcs x13, x13, x4 + sbcs x2, x2, x3 + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbc x11, x11, xzr + lsl x5, x12, #32 + add x12, x5, x12 + mov x5, #-4294967295 + umulh x5, x5, x12 + mov x4, #4294967295 + mul x3, x4, x12 + umulh x4, x4, x12 + adds x5, x5, x3 + adcs x4, x4, x12 + adc x3, xzr, xzr + subs x13, x13, x5 + sbcs x2, x2, x4 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbc x12, x12, xzr + lsl x5, x13, #32 + add x13, x5, x13 + mov x5, #-4294967295 + umulh x5, x5, x13 + mov x4, #4294967295 + mul x3, x4, x13 + umulh x4, x4, x13 + adds x5, x5, x3 + adcs x4, x4, x13 + adc x3, xzr, xzr + subs x2, x2, x5 + sbcs x9, x9, x4 + sbcs x10, x10, x3 + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + adds x2, x2, x14 + adcs x9, x9, x15 + adcs x10, x10, x16 + adcs x11, x11, x17 + adcs x12, x12, x19 + adcs x13, x13, x20 + mov x14, #-4294967295 + mov x15, #4294967295 + csel x14, x14, xzr, hs + csel x15, x15, xzr, hs + cset x16, hs + adds x2, x2, x14 + adcs x9, x9, x15 + adcs x10, x10, x16 + adcs x11, x11, xzr + adcs x12, x12, xzr + adc x13, x13, xzr + stp x2, x9, [sp, #240] + stp x10, x11, [sp, #256] + stp x12, x13, [sp, #272] + ldp x3, x4, [x26, #96] + ldp x5, x6, [x25, #48] + mul x12, x3, x5 + umulh x13, x3, x5 + mul x11, x3, x6 + umulh x14, x3, x6 + adds x13, x13, x11 + ldp x7, x8, [x25, #64] + mul x11, x3, x7 + umulh x15, x3, x7 + adcs x14, x14, x11 + mul x11, x3, x8 + umulh x16, x3, x8 + adcs x15, x15, x11 + ldp x9, x10, [x25, #80] + mul x11, x3, x9 + umulh x17, x3, x9 + adcs x16, x16, x11 + mul x11, x3, x10 + umulh x19, x3, x10 + adcs x17, x17, x11 + adc x19, x19, xzr + mul x11, x4, x5 + adds x13, x13, x11 + mul x11, x4, x6 + adcs x14, x14, x11 + mul x11, x4, x7 + adcs x15, x15, x11 + mul x11, x4, x8 + adcs x16, x16, x11 + mul x11, x4, x9 + adcs x17, x17, x11 + mul x11, x4, x10 + adcs x19, x19, x11 + cset x20, hs + umulh x11, x4, x5 + adds x14, x14, x11 + umulh x11, x4, x6 + adcs x15, x15, x11 + umulh x11, x4, x7 + adcs x16, x16, x11 + umulh x11, x4, x8 + adcs x17, x17, x11 + umulh x11, x4, x9 + adcs x19, x19, x11 + umulh x11, x4, x10 + adc x20, x20, x11 + ldp x3, x4, [x26, #112] + mul x11, x3, x5 + adds x14, x14, x11 + mul x11, x3, x6 + adcs x15, x15, x11 + mul x11, x3, x7 + adcs x16, x16, x11 + mul x11, x3, x8 + adcs x17, x17, x11 + mul x11, x3, x9 + adcs x19, x19, x11 + mul x11, x3, x10 + adcs x20, x20, x11 + cset x21, hs + umulh x11, x3, x5 + adds x15, x15, x11 + umulh x11, x3, x6 + adcs x16, x16, x11 + umulh x11, x3, x7 + adcs x17, x17, x11 + umulh x11, x3, x8 + adcs x19, x19, x11 + umulh x11, x3, x9 + adcs x20, x20, x11 + umulh x11, x3, x10 + adc x21, x21, x11 + mul x11, x4, x5 + adds x15, x15, x11 + mul x11, x4, x6 + adcs x16, x16, x11 + mul x11, x4, x7 + adcs x17, x17, x11 + mul x11, x4, x8 + adcs x19, x19, x11 + mul x11, x4, x9 + adcs x20, x20, x11 + mul x11, x4, x10 + adcs x21, x21, x11 + cset x22, hs + umulh x11, x4, x5 + adds x16, x16, x11 + umulh x11, x4, x6 + adcs x17, x17, x11 + umulh x11, x4, x7 + adcs x19, x19, x11 + umulh x11, x4, x8 + adcs x20, x20, x11 + umulh x11, x4, x9 + adcs x21, x21, x11 + umulh x11, x4, x10 + adc x22, x22, x11 + ldp x3, x4, [x26, #128] + mul x11, x3, x5 + adds x16, x16, x11 + mul x11, x3, x6 + adcs x17, x17, x11 + mul x11, x3, x7 + adcs x19, x19, x11 + mul x11, x3, x8 + adcs x20, x20, x11 + mul x11, x3, x9 + adcs x21, x21, x11 + mul x11, x3, x10 + adcs x22, x22, x11 + cset x2, hs + umulh x11, x3, x5 + adds x17, x17, x11 + umulh x11, x3, x6 + adcs x19, x19, x11 + umulh x11, x3, x7 + adcs x20, x20, x11 + umulh x11, x3, x8 + adcs x21, x21, x11 + umulh x11, x3, x9 + adcs x22, x22, x11 + umulh x11, x3, x10 + adc x2, x2, x11 + mul x11, x4, x5 + adds x17, x17, x11 + mul x11, x4, x6 + adcs x19, x19, x11 + mul x11, x4, x7 + adcs x20, x20, x11 + mul x11, x4, x8 + adcs x21, x21, x11 + mul x11, x4, x9 + adcs x22, x22, x11 + mul x11, x4, x10 + adcs x2, x2, x11 + cset x1, hs + umulh x11, x4, x5 + adds x19, x19, x11 + umulh x11, x4, x6 + adcs x20, x20, x11 + umulh x11, x4, x7 + adcs x21, x21, x11 + umulh x11, x4, x8 + adcs x22, x22, x11 + umulh x11, x4, x9 + adcs x2, x2, x11 + umulh x11, x4, x10 + adc x1, x1, x11 + lsl x7, x12, #32 + add x12, x7, x12 + mov x7, #-4294967295 + umulh x7, x7, x12 + mov x6, #4294967295 + mul x5, x6, x12 + umulh x6, x6, x12 + adds x7, x7, x5 + adcs x6, x6, x12 + adc x5, xzr, xzr + subs x13, x13, x7 + sbcs x14, x14, x6 + sbcs x15, x15, x5 + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x12, x12, xzr + lsl x7, x13, #32 + add x13, x7, x13 + mov x7, #-4294967295 + umulh x7, x7, x13 + mov x6, #4294967295 + mul x5, x6, x13 + umulh x6, x6, x13 + adds x7, x7, x5 + adcs x6, x6, x13 + adc x5, xzr, xzr + subs x14, x14, x7 + sbcs x15, x15, x6 + sbcs x16, x16, x5 + sbcs x17, x17, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + lsl x7, x14, #32 + add x14, x7, x14 + mov x7, #-4294967295 + umulh x7, x7, x14 + mov x6, #4294967295 + mul x5, x6, x14 + umulh x6, x6, x14 + adds x7, x7, x5 + adcs x6, x6, x14 + adc x5, xzr, xzr + subs x15, x15, x7 + sbcs x16, x16, x6 + sbcs x17, x17, x5 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + lsl x7, x15, #32 + add x15, x7, x15 + mov x7, #-4294967295 + umulh x7, x7, x15 + mov x6, #4294967295 + mul x5, x6, x15 + umulh x6, x6, x15 + adds x7, x7, x5 + adcs x6, x6, x15 + adc x5, xzr, xzr + subs x16, x16, x7 + sbcs x17, x17, x6 + sbcs x12, x12, x5 + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbc x15, x15, xzr + lsl x7, x16, #32 + add x16, x7, x16 + mov x7, #-4294967295 + umulh x7, x7, x16 + mov x6, #4294967295 + mul x5, x6, x16 + umulh x6, x6, x16 + adds x7, x7, x5 + adcs x6, x6, x16 + adc x5, xzr, xzr + subs x17, x17, x7 + sbcs x12, x12, x6 + sbcs x13, x13, x5 + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbc x16, x16, xzr + lsl x7, x17, #32 + add x17, x7, x17 + mov x7, #-4294967295 + umulh x7, x7, x17 + mov x6, #4294967295 + mul x5, x6, x17 + umulh x6, x6, x17 + adds x7, x7, x5 + adcs x6, x6, x17 + adc x5, xzr, xzr + subs x12, x12, x7 + sbcs x13, x13, x6 + sbcs x14, x14, x5 + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbc x17, x17, xzr + adds x12, x12, x19 + adcs x13, x13, x20 + adcs x14, x14, x21 + adcs x15, x15, x22 + adcs x16, x16, x2 + adcs x17, x17, x1 + adc x10, xzr, xzr + mov x11, #-4294967295 + adds x19, x12, x11 + mov x11, #4294967295 + adcs x20, x13, x11 + mov x11, #1 + adcs x21, x14, x11 + adcs x22, x15, xzr + adcs x2, x16, xzr + adcs x1, x17, xzr + adcs x10, x10, xzr + csel x12, x12, x19, eq + csel x13, x13, x20, eq + csel x14, x14, x21, eq + csel x15, x15, x22, eq + csel x16, x16, x2, eq + csel x17, x17, x1, eq + stp x12, x13, [sp, #288] + stp x14, x15, [sp, #304] + stp x16, x17, [sp, #320] + ldp x3, x4, [x25, #96] + ldp x5, x6, [x26, #48] + mul x12, x3, x5 + umulh x13, x3, x5 + mul x11, x3, x6 + umulh x14, x3, x6 + adds x13, x13, x11 + ldp x7, x8, [x26, #64] + mul x11, x3, x7 + umulh x15, x3, x7 + adcs x14, x14, x11 + mul x11, x3, x8 + umulh x16, x3, x8 + adcs x15, x15, x11 + ldp x9, x10, [x26, #80] + mul x11, x3, x9 + umulh x17, x3, x9 + adcs x16, x16, x11 + mul x11, x3, x10 + umulh x19, x3, x10 + adcs x17, x17, x11 + adc x19, x19, xzr + mul x11, x4, x5 + adds x13, x13, x11 + mul x11, x4, x6 + adcs x14, x14, x11 + mul x11, x4, x7 + adcs x15, x15, x11 + mul x11, x4, x8 + adcs x16, x16, x11 + mul x11, x4, x9 + adcs x17, x17, x11 + mul x11, x4, x10 + adcs x19, x19, x11 + cset x20, hs + umulh x11, x4, x5 + adds x14, x14, x11 + umulh x11, x4, x6 + adcs x15, x15, x11 + umulh x11, x4, x7 + adcs x16, x16, x11 + umulh x11, x4, x8 + adcs x17, x17, x11 + umulh x11, x4, x9 + adcs x19, x19, x11 + umulh x11, x4, x10 + adc x20, x20, x11 + ldp x3, x4, [x25, #112] + mul x11, x3, x5 + adds x14, x14, x11 + mul x11, x3, x6 + adcs x15, x15, x11 + mul x11, x3, x7 + adcs x16, x16, x11 + mul x11, x3, x8 + adcs x17, x17, x11 + mul x11, x3, x9 + adcs x19, x19, x11 + mul x11, x3, x10 + adcs x20, x20, x11 + cset x21, hs + umulh x11, x3, x5 + adds x15, x15, x11 + umulh x11, x3, x6 + adcs x16, x16, x11 + umulh x11, x3, x7 + adcs x17, x17, x11 + umulh x11, x3, x8 + adcs x19, x19, x11 + umulh x11, x3, x9 + adcs x20, x20, x11 + umulh x11, x3, x10 + adc x21, x21, x11 + mul x11, x4, x5 + adds x15, x15, x11 + mul x11, x4, x6 + adcs x16, x16, x11 + mul x11, x4, x7 + adcs x17, x17, x11 + mul x11, x4, x8 + adcs x19, x19, x11 + mul x11, x4, x9 + adcs x20, x20, x11 + mul x11, x4, x10 + adcs x21, x21, x11 + cset x22, hs + umulh x11, x4, x5 + adds x16, x16, x11 + umulh x11, x4, x6 + adcs x17, x17, x11 + umulh x11, x4, x7 + adcs x19, x19, x11 + umulh x11, x4, x8 + adcs x20, x20, x11 + umulh x11, x4, x9 + adcs x21, x21, x11 + umulh x11, x4, x10 + adc x22, x22, x11 + ldp x3, x4, [x25, #128] + mul x11, x3, x5 + adds x16, x16, x11 + mul x11, x3, x6 + adcs x17, x17, x11 + mul x11, x3, x7 + adcs x19, x19, x11 + mul x11, x3, x8 + adcs x20, x20, x11 + mul x11, x3, x9 + adcs x21, x21, x11 + mul x11, x3, x10 + adcs x22, x22, x11 + cset x2, hs + umulh x11, x3, x5 + adds x17, x17, x11 + umulh x11, x3, x6 + adcs x19, x19, x11 + umulh x11, x3, x7 + adcs x20, x20, x11 + umulh x11, x3, x8 + adcs x21, x21, x11 + umulh x11, x3, x9 + adcs x22, x22, x11 + umulh x11, x3, x10 + adc x2, x2, x11 + mul x11, x4, x5 + adds x17, x17, x11 + mul x11, x4, x6 + adcs x19, x19, x11 + mul x11, x4, x7 + adcs x20, x20, x11 + mul x11, x4, x8 + adcs x21, x21, x11 + mul x11, x4, x9 + adcs x22, x22, x11 + mul x11, x4, x10 + adcs x2, x2, x11 + cset x1, hs + umulh x11, x4, x5 + adds x19, x19, x11 + umulh x11, x4, x6 + adcs x20, x20, x11 + umulh x11, x4, x7 + adcs x21, x21, x11 + umulh x11, x4, x8 + adcs x22, x22, x11 + umulh x11, x4, x9 + adcs x2, x2, x11 + umulh x11, x4, x10 + adc x1, x1, x11 + lsl x7, x12, #32 + add x12, x7, x12 + mov x7, #-4294967295 + umulh x7, x7, x12 + mov x6, #4294967295 + mul x5, x6, x12 + umulh x6, x6, x12 + adds x7, x7, x5 + adcs x6, x6, x12 + adc x5, xzr, xzr + subs x13, x13, x7 + sbcs x14, x14, x6 + sbcs x15, x15, x5 + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x12, x12, xzr + lsl x7, x13, #32 + add x13, x7, x13 + mov x7, #-4294967295 + umulh x7, x7, x13 + mov x6, #4294967295 + mul x5, x6, x13 + umulh x6, x6, x13 + adds x7, x7, x5 + adcs x6, x6, x13 + adc x5, xzr, xzr + subs x14, x14, x7 + sbcs x15, x15, x6 + sbcs x16, x16, x5 + sbcs x17, x17, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + lsl x7, x14, #32 + add x14, x7, x14 + mov x7, #-4294967295 + umulh x7, x7, x14 + mov x6, #4294967295 + mul x5, x6, x14 + umulh x6, x6, x14 + adds x7, x7, x5 + adcs x6, x6, x14 + adc x5, xzr, xzr + subs x15, x15, x7 + sbcs x16, x16, x6 + sbcs x17, x17, x5 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + lsl x7, x15, #32 + add x15, x7, x15 + mov x7, #-4294967295 + umulh x7, x7, x15 + mov x6, #4294967295 + mul x5, x6, x15 + umulh x6, x6, x15 + adds x7, x7, x5 + adcs x6, x6, x15 + adc x5, xzr, xzr + subs x16, x16, x7 + sbcs x17, x17, x6 + sbcs x12, x12, x5 + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbc x15, x15, xzr + lsl x7, x16, #32 + add x16, x7, x16 + mov x7, #-4294967295 + umulh x7, x7, x16 + mov x6, #4294967295 + mul x5, x6, x16 + umulh x6, x6, x16 + adds x7, x7, x5 + adcs x6, x6, x16 + adc x5, xzr, xzr + subs x17, x17, x7 + sbcs x12, x12, x6 + sbcs x13, x13, x5 + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbc x16, x16, xzr + lsl x7, x17, #32 + add x17, x7, x17 + mov x7, #-4294967295 + umulh x7, x7, x17 + mov x6, #4294967295 + mul x5, x6, x17 + umulh x6, x6, x17 + adds x7, x7, x5 + adcs x6, x6, x17 + adc x5, xzr, xzr + subs x12, x12, x7 + sbcs x13, x13, x6 + sbcs x14, x14, x5 + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbc x17, x17, xzr + adds x12, x12, x19 + adcs x13, x13, x20 + adcs x14, x14, x21 + adcs x15, x15, x22 + adcs x16, x16, x2 + adcs x17, x17, x1 + adc x10, xzr, xzr + mov x11, #-4294967295 + adds x19, x12, x11 + mov x11, #4294967295 + adcs x20, x13, x11 + mov x11, #1 + adcs x21, x14, x11 + adcs x22, x15, xzr + adcs x2, x16, xzr + adcs x1, x17, xzr + adcs x10, x10, xzr + csel x12, x12, x19, eq + csel x13, x13, x20, eq + csel x14, x14, x21, eq + csel x15, x15, x22, eq + csel x16, x16, x2, eq + csel x17, x17, x1, eq + stp x12, x13, [sp, #48] + stp x14, x15, [sp, #64] + stp x16, x17, [sp, #80] + ldp x3, x4, [sp] + ldp x5, x6, [x26] + mul x12, x3, x5 + umulh x13, x3, x5 + mul x11, x3, x6 + umulh x14, x3, x6 + adds x13, x13, x11 + ldp x7, x8, [x26, #16] + mul x11, x3, x7 + umulh x15, x3, x7 + adcs x14, x14, x11 + mul x11, x3, x8 + umulh x16, x3, x8 + adcs x15, x15, x11 + ldp x9, x10, [x26, #32] + mul x11, x3, x9 + umulh x17, x3, x9 + adcs x16, x16, x11 + mul x11, x3, x10 + umulh x19, x3, x10 + adcs x17, x17, x11 + adc x19, x19, xzr + mul x11, x4, x5 + adds x13, x13, x11 + mul x11, x4, x6 + adcs x14, x14, x11 + mul x11, x4, x7 + adcs x15, x15, x11 + mul x11, x4, x8 + adcs x16, x16, x11 + mul x11, x4, x9 + adcs x17, x17, x11 + mul x11, x4, x10 + adcs x19, x19, x11 + cset x20, hs + umulh x11, x4, x5 + adds x14, x14, x11 + umulh x11, x4, x6 + adcs x15, x15, x11 + umulh x11, x4, x7 + adcs x16, x16, x11 + umulh x11, x4, x8 + adcs x17, x17, x11 + umulh x11, x4, x9 + adcs x19, x19, x11 + umulh x11, x4, x10 + adc x20, x20, x11 + ldp x3, x4, [sp, #16] + mul x11, x3, x5 + adds x14, x14, x11 + mul x11, x3, x6 + adcs x15, x15, x11 + mul x11, x3, x7 + adcs x16, x16, x11 + mul x11, x3, x8 + adcs x17, x17, x11 + mul x11, x3, x9 + adcs x19, x19, x11 + mul x11, x3, x10 + adcs x20, x20, x11 + cset x21, hs + umulh x11, x3, x5 + adds x15, x15, x11 + umulh x11, x3, x6 + adcs x16, x16, x11 + umulh x11, x3, x7 + adcs x17, x17, x11 + umulh x11, x3, x8 + adcs x19, x19, x11 + umulh x11, x3, x9 + adcs x20, x20, x11 + umulh x11, x3, x10 + adc x21, x21, x11 + mul x11, x4, x5 + adds x15, x15, x11 + mul x11, x4, x6 + adcs x16, x16, x11 + mul x11, x4, x7 + adcs x17, x17, x11 + mul x11, x4, x8 + adcs x19, x19, x11 + mul x11, x4, x9 + adcs x20, x20, x11 + mul x11, x4, x10 + adcs x21, x21, x11 + cset x22, hs + umulh x11, x4, x5 + adds x16, x16, x11 + umulh x11, x4, x6 + adcs x17, x17, x11 + umulh x11, x4, x7 + adcs x19, x19, x11 + umulh x11, x4, x8 + adcs x20, x20, x11 + umulh x11, x4, x9 + adcs x21, x21, x11 + umulh x11, x4, x10 + adc x22, x22, x11 + ldp x3, x4, [sp, #32] + mul x11, x3, x5 + adds x16, x16, x11 + mul x11, x3, x6 + adcs x17, x17, x11 + mul x11, x3, x7 + adcs x19, x19, x11 + mul x11, x3, x8 + adcs x20, x20, x11 + mul x11, x3, x9 + adcs x21, x21, x11 + mul x11, x3, x10 + adcs x22, x22, x11 + cset x2, hs + umulh x11, x3, x5 + adds x17, x17, x11 + umulh x11, x3, x6 + adcs x19, x19, x11 + umulh x11, x3, x7 + adcs x20, x20, x11 + umulh x11, x3, x8 + adcs x21, x21, x11 + umulh x11, x3, x9 + adcs x22, x22, x11 + umulh x11, x3, x10 + adc x2, x2, x11 + mul x11, x4, x5 + adds x17, x17, x11 + mul x11, x4, x6 + adcs x19, x19, x11 + mul x11, x4, x7 + adcs x20, x20, x11 + mul x11, x4, x8 + adcs x21, x21, x11 + mul x11, x4, x9 + adcs x22, x22, x11 + mul x11, x4, x10 + adcs x2, x2, x11 + cset x1, hs + umulh x11, x4, x5 + adds x19, x19, x11 + umulh x11, x4, x6 + adcs x20, x20, x11 + umulh x11, x4, x7 + adcs x21, x21, x11 + umulh x11, x4, x8 + adcs x22, x22, x11 + umulh x11, x4, x9 + adcs x2, x2, x11 + umulh x11, x4, x10 + adc x1, x1, x11 + lsl x7, x12, #32 + add x12, x7, x12 + mov x7, #-4294967295 + umulh x7, x7, x12 + mov x6, #4294967295 + mul x5, x6, x12 + umulh x6, x6, x12 + adds x7, x7, x5 + adcs x6, x6, x12 + adc x5, xzr, xzr + subs x13, x13, x7 + sbcs x14, x14, x6 + sbcs x15, x15, x5 + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x12, x12, xzr + lsl x7, x13, #32 + add x13, x7, x13 + mov x7, #-4294967295 + umulh x7, x7, x13 + mov x6, #4294967295 + mul x5, x6, x13 + umulh x6, x6, x13 + adds x7, x7, x5 + adcs x6, x6, x13 + adc x5, xzr, xzr + subs x14, x14, x7 + sbcs x15, x15, x6 + sbcs x16, x16, x5 + sbcs x17, x17, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + lsl x7, x14, #32 + add x14, x7, x14 + mov x7, #-4294967295 + umulh x7, x7, x14 + mov x6, #4294967295 + mul x5, x6, x14 + umulh x6, x6, x14 + adds x7, x7, x5 + adcs x6, x6, x14 + adc x5, xzr, xzr + subs x15, x15, x7 + sbcs x16, x16, x6 + sbcs x17, x17, x5 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + lsl x7, x15, #32 + add x15, x7, x15 + mov x7, #-4294967295 + umulh x7, x7, x15 + mov x6, #4294967295 + mul x5, x6, x15 + umulh x6, x6, x15 + adds x7, x7, x5 + adcs x6, x6, x15 + adc x5, xzr, xzr + subs x16, x16, x7 + sbcs x17, x17, x6 + sbcs x12, x12, x5 + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbc x15, x15, xzr + lsl x7, x16, #32 + add x16, x7, x16 + mov x7, #-4294967295 + umulh x7, x7, x16 + mov x6, #4294967295 + mul x5, x6, x16 + umulh x6, x6, x16 + adds x7, x7, x5 + adcs x6, x6, x16 + adc x5, xzr, xzr + subs x17, x17, x7 + sbcs x12, x12, x6 + sbcs x13, x13, x5 + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbc x16, x16, xzr + lsl x7, x17, #32 + add x17, x7, x17 + mov x7, #-4294967295 + umulh x7, x7, x17 + mov x6, #4294967295 + mul x5, x6, x17 + umulh x6, x6, x17 + adds x7, x7, x5 + adcs x6, x6, x17 + adc x5, xzr, xzr + subs x12, x12, x7 + sbcs x13, x13, x6 + sbcs x14, x14, x5 + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbc x17, x17, xzr + adds x12, x12, x19 + adcs x13, x13, x20 + adcs x14, x14, x21 + adcs x15, x15, x22 + adcs x16, x16, x2 + adcs x17, x17, x1 + adc x10, xzr, xzr + mov x11, #-4294967295 + adds x19, x12, x11 + mov x11, #4294967295 + adcs x20, x13, x11 + mov x11, #1 + adcs x21, x14, x11 + adcs x22, x15, xzr + adcs x2, x16, xzr + adcs x1, x17, xzr + adcs x10, x10, xzr + csel x12, x12, x19, eq + csel x13, x13, x20, eq + csel x14, x14, x21, eq + csel x15, x15, x22, eq + csel x16, x16, x2, eq + csel x17, x17, x1, eq + stp x12, x13, [sp, #96] + stp x14, x15, [sp, #112] + stp x16, x17, [sp, #128] + ldp x3, x4, [sp, #240] + ldp x5, x6, [x25] + mul x12, x3, x5 + umulh x13, x3, x5 + mul x11, x3, x6 + umulh x14, x3, x6 + adds x13, x13, x11 + ldp x7, x8, [x25, #16] + mul x11, x3, x7 + umulh x15, x3, x7 + adcs x14, x14, x11 + mul x11, x3, x8 + umulh x16, x3, x8 + adcs x15, x15, x11 + ldp x9, x10, [x25, #32] + mul x11, x3, x9 + umulh x17, x3, x9 + adcs x16, x16, x11 + mul x11, x3, x10 + umulh x19, x3, x10 + adcs x17, x17, x11 + adc x19, x19, xzr + mul x11, x4, x5 + adds x13, x13, x11 + mul x11, x4, x6 + adcs x14, x14, x11 + mul x11, x4, x7 + adcs x15, x15, x11 + mul x11, x4, x8 + adcs x16, x16, x11 + mul x11, x4, x9 + adcs x17, x17, x11 + mul x11, x4, x10 + adcs x19, x19, x11 + cset x20, hs + umulh x11, x4, x5 + adds x14, x14, x11 + umulh x11, x4, x6 + adcs x15, x15, x11 + umulh x11, x4, x7 + adcs x16, x16, x11 + umulh x11, x4, x8 + adcs x17, x17, x11 + umulh x11, x4, x9 + adcs x19, x19, x11 + umulh x11, x4, x10 + adc x20, x20, x11 + ldp x3, x4, [sp, #256] + mul x11, x3, x5 + adds x14, x14, x11 + mul x11, x3, x6 + adcs x15, x15, x11 + mul x11, x3, x7 + adcs x16, x16, x11 + mul x11, x3, x8 + adcs x17, x17, x11 + mul x11, x3, x9 + adcs x19, x19, x11 + mul x11, x3, x10 + adcs x20, x20, x11 + cset x21, hs + umulh x11, x3, x5 + adds x15, x15, x11 + umulh x11, x3, x6 + adcs x16, x16, x11 + umulh x11, x3, x7 + adcs x17, x17, x11 + umulh x11, x3, x8 + adcs x19, x19, x11 + umulh x11, x3, x9 + adcs x20, x20, x11 + umulh x11, x3, x10 + adc x21, x21, x11 + mul x11, x4, x5 + adds x15, x15, x11 + mul x11, x4, x6 + adcs x16, x16, x11 + mul x11, x4, x7 + adcs x17, x17, x11 + mul x11, x4, x8 + adcs x19, x19, x11 + mul x11, x4, x9 + adcs x20, x20, x11 + mul x11, x4, x10 + adcs x21, x21, x11 + cset x22, hs + umulh x11, x4, x5 + adds x16, x16, x11 + umulh x11, x4, x6 + adcs x17, x17, x11 + umulh x11, x4, x7 + adcs x19, x19, x11 + umulh x11, x4, x8 + adcs x20, x20, x11 + umulh x11, x4, x9 + adcs x21, x21, x11 + umulh x11, x4, x10 + adc x22, x22, x11 + ldp x3, x4, [sp, #272] + mul x11, x3, x5 + adds x16, x16, x11 + mul x11, x3, x6 + adcs x17, x17, x11 + mul x11, x3, x7 + adcs x19, x19, x11 + mul x11, x3, x8 + adcs x20, x20, x11 + mul x11, x3, x9 + adcs x21, x21, x11 + mul x11, x3, x10 + adcs x22, x22, x11 + cset x2, hs + umulh x11, x3, x5 + adds x17, x17, x11 + umulh x11, x3, x6 + adcs x19, x19, x11 + umulh x11, x3, x7 + adcs x20, x20, x11 + umulh x11, x3, x8 + adcs x21, x21, x11 + umulh x11, x3, x9 + adcs x22, x22, x11 + umulh x11, x3, x10 + adc x2, x2, x11 + mul x11, x4, x5 + adds x17, x17, x11 + mul x11, x4, x6 + adcs x19, x19, x11 + mul x11, x4, x7 + adcs x20, x20, x11 + mul x11, x4, x8 + adcs x21, x21, x11 + mul x11, x4, x9 + adcs x22, x22, x11 + mul x11, x4, x10 + adcs x2, x2, x11 + cset x1, hs + umulh x11, x4, x5 + adds x19, x19, x11 + umulh x11, x4, x6 + adcs x20, x20, x11 + umulh x11, x4, x7 + adcs x21, x21, x11 + umulh x11, x4, x8 + adcs x22, x22, x11 + umulh x11, x4, x9 + adcs x2, x2, x11 + umulh x11, x4, x10 + adc x1, x1, x11 + lsl x7, x12, #32 + add x12, x7, x12 + mov x7, #-4294967295 + umulh x7, x7, x12 + mov x6, #4294967295 + mul x5, x6, x12 + umulh x6, x6, x12 + adds x7, x7, x5 + adcs x6, x6, x12 + adc x5, xzr, xzr + subs x13, x13, x7 + sbcs x14, x14, x6 + sbcs x15, x15, x5 + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x12, x12, xzr + lsl x7, x13, #32 + add x13, x7, x13 + mov x7, #-4294967295 + umulh x7, x7, x13 + mov x6, #4294967295 + mul x5, x6, x13 + umulh x6, x6, x13 + adds x7, x7, x5 + adcs x6, x6, x13 + adc x5, xzr, xzr + subs x14, x14, x7 + sbcs x15, x15, x6 + sbcs x16, x16, x5 + sbcs x17, x17, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + lsl x7, x14, #32 + add x14, x7, x14 + mov x7, #-4294967295 + umulh x7, x7, x14 + mov x6, #4294967295 + mul x5, x6, x14 + umulh x6, x6, x14 + adds x7, x7, x5 + adcs x6, x6, x14 + adc x5, xzr, xzr + subs x15, x15, x7 + sbcs x16, x16, x6 + sbcs x17, x17, x5 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + lsl x7, x15, #32 + add x15, x7, x15 + mov x7, #-4294967295 + umulh x7, x7, x15 + mov x6, #4294967295 + mul x5, x6, x15 + umulh x6, x6, x15 + adds x7, x7, x5 + adcs x6, x6, x15 + adc x5, xzr, xzr + subs x16, x16, x7 + sbcs x17, x17, x6 + sbcs x12, x12, x5 + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbc x15, x15, xzr + lsl x7, x16, #32 + add x16, x7, x16 + mov x7, #-4294967295 + umulh x7, x7, x16 + mov x6, #4294967295 + mul x5, x6, x16 + umulh x6, x6, x16 + adds x7, x7, x5 + adcs x6, x6, x16 + adc x5, xzr, xzr + subs x17, x17, x7 + sbcs x12, x12, x6 + sbcs x13, x13, x5 + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbc x16, x16, xzr + lsl x7, x17, #32 + add x17, x7, x17 + mov x7, #-4294967295 + umulh x7, x7, x17 + mov x6, #4294967295 + mul x5, x6, x17 + umulh x6, x6, x17 + adds x7, x7, x5 + adcs x6, x6, x17 + adc x5, xzr, xzr + subs x12, x12, x7 + sbcs x13, x13, x6 + sbcs x14, x14, x5 + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbc x17, x17, xzr + adds x12, x12, x19 + adcs x13, x13, x20 + adcs x14, x14, x21 + adcs x15, x15, x22 + adcs x16, x16, x2 + adcs x17, x17, x1 + adc x10, xzr, xzr + mov x11, #-4294967295 + adds x19, x12, x11 + mov x11, #4294967295 + adcs x20, x13, x11 + mov x11, #1 + adcs x21, x14, x11 + adcs x22, x15, xzr + adcs x2, x16, xzr + adcs x1, x17, xzr + adcs x10, x10, xzr + csel x12, x12, x19, eq + csel x13, x13, x20, eq + csel x14, x14, x21, eq + csel x15, x15, x22, eq + csel x16, x16, x2, eq + csel x17, x17, x1, eq + stp x12, x13, [sp, #192] + stp x14, x15, [sp, #208] + stp x16, x17, [sp, #224] + ldp x3, x4, [sp] + ldp x5, x6, [sp, #48] + mul x12, x3, x5 + umulh x13, x3, x5 + mul x11, x3, x6 + umulh x14, x3, x6 + adds x13, x13, x11 + ldp x7, x8, [sp, #64] + mul x11, x3, x7 + umulh x15, x3, x7 + adcs x14, x14, x11 + mul x11, x3, x8 + umulh x16, x3, x8 + adcs x15, x15, x11 + ldp x9, x10, [sp, #80] + mul x11, x3, x9 + umulh x17, x3, x9 + adcs x16, x16, x11 + mul x11, x3, x10 + umulh x19, x3, x10 + adcs x17, x17, x11 + adc x19, x19, xzr + mul x11, x4, x5 + adds x13, x13, x11 + mul x11, x4, x6 + adcs x14, x14, x11 + mul x11, x4, x7 + adcs x15, x15, x11 + mul x11, x4, x8 + adcs x16, x16, x11 + mul x11, x4, x9 + adcs x17, x17, x11 + mul x11, x4, x10 + adcs x19, x19, x11 + cset x20, hs + umulh x11, x4, x5 + adds x14, x14, x11 + umulh x11, x4, x6 + adcs x15, x15, x11 + umulh x11, x4, x7 + adcs x16, x16, x11 + umulh x11, x4, x8 + adcs x17, x17, x11 + umulh x11, x4, x9 + adcs x19, x19, x11 + umulh x11, x4, x10 + adc x20, x20, x11 + ldp x3, x4, [sp, #16] + mul x11, x3, x5 + adds x14, x14, x11 + mul x11, x3, x6 + adcs x15, x15, x11 + mul x11, x3, x7 + adcs x16, x16, x11 + mul x11, x3, x8 + adcs x17, x17, x11 + mul x11, x3, x9 + adcs x19, x19, x11 + mul x11, x3, x10 + adcs x20, x20, x11 + cset x21, hs + umulh x11, x3, x5 + adds x15, x15, x11 + umulh x11, x3, x6 + adcs x16, x16, x11 + umulh x11, x3, x7 + adcs x17, x17, x11 + umulh x11, x3, x8 + adcs x19, x19, x11 + umulh x11, x3, x9 + adcs x20, x20, x11 + umulh x11, x3, x10 + adc x21, x21, x11 + mul x11, x4, x5 + adds x15, x15, x11 + mul x11, x4, x6 + adcs x16, x16, x11 + mul x11, x4, x7 + adcs x17, x17, x11 + mul x11, x4, x8 + adcs x19, x19, x11 + mul x11, x4, x9 + adcs x20, x20, x11 + mul x11, x4, x10 + adcs x21, x21, x11 + cset x22, hs + umulh x11, x4, x5 + adds x16, x16, x11 + umulh x11, x4, x6 + adcs x17, x17, x11 + umulh x11, x4, x7 + adcs x19, x19, x11 + umulh x11, x4, x8 + adcs x20, x20, x11 + umulh x11, x4, x9 + adcs x21, x21, x11 + umulh x11, x4, x10 + adc x22, x22, x11 + ldp x3, x4, [sp, #32] + mul x11, x3, x5 + adds x16, x16, x11 + mul x11, x3, x6 + adcs x17, x17, x11 + mul x11, x3, x7 + adcs x19, x19, x11 + mul x11, x3, x8 + adcs x20, x20, x11 + mul x11, x3, x9 + adcs x21, x21, x11 + mul x11, x3, x10 + adcs x22, x22, x11 + cset x2, hs + umulh x11, x3, x5 + adds x17, x17, x11 + umulh x11, x3, x6 + adcs x19, x19, x11 + umulh x11, x3, x7 + adcs x20, x20, x11 + umulh x11, x3, x8 + adcs x21, x21, x11 + umulh x11, x3, x9 + adcs x22, x22, x11 + umulh x11, x3, x10 + adc x2, x2, x11 + mul x11, x4, x5 + adds x17, x17, x11 + mul x11, x4, x6 + adcs x19, x19, x11 + mul x11, x4, x7 + adcs x20, x20, x11 + mul x11, x4, x8 + adcs x21, x21, x11 + mul x11, x4, x9 + adcs x22, x22, x11 + mul x11, x4, x10 + adcs x2, x2, x11 + cset x1, hs + umulh x11, x4, x5 + adds x19, x19, x11 + umulh x11, x4, x6 + adcs x20, x20, x11 + umulh x11, x4, x7 + adcs x21, x21, x11 + umulh x11, x4, x8 + adcs x22, x22, x11 + umulh x11, x4, x9 + adcs x2, x2, x11 + umulh x11, x4, x10 + adc x1, x1, x11 + lsl x7, x12, #32 + add x12, x7, x12 + mov x7, #-4294967295 + umulh x7, x7, x12 + mov x6, #4294967295 + mul x5, x6, x12 + umulh x6, x6, x12 + adds x7, x7, x5 + adcs x6, x6, x12 + adc x5, xzr, xzr + subs x13, x13, x7 + sbcs x14, x14, x6 + sbcs x15, x15, x5 + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x12, x12, xzr + lsl x7, x13, #32 + add x13, x7, x13 + mov x7, #-4294967295 + umulh x7, x7, x13 + mov x6, #4294967295 + mul x5, x6, x13 + umulh x6, x6, x13 + adds x7, x7, x5 + adcs x6, x6, x13 + adc x5, xzr, xzr + subs x14, x14, x7 + sbcs x15, x15, x6 + sbcs x16, x16, x5 + sbcs x17, x17, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + lsl x7, x14, #32 + add x14, x7, x14 + mov x7, #-4294967295 + umulh x7, x7, x14 + mov x6, #4294967295 + mul x5, x6, x14 + umulh x6, x6, x14 + adds x7, x7, x5 + adcs x6, x6, x14 + adc x5, xzr, xzr + subs x15, x15, x7 + sbcs x16, x16, x6 + sbcs x17, x17, x5 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + lsl x7, x15, #32 + add x15, x7, x15 + mov x7, #-4294967295 + umulh x7, x7, x15 + mov x6, #4294967295 + mul x5, x6, x15 + umulh x6, x6, x15 + adds x7, x7, x5 + adcs x6, x6, x15 + adc x5, xzr, xzr + subs x16, x16, x7 + sbcs x17, x17, x6 + sbcs x12, x12, x5 + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbc x15, x15, xzr + lsl x7, x16, #32 + add x16, x7, x16 + mov x7, #-4294967295 + umulh x7, x7, x16 + mov x6, #4294967295 + mul x5, x6, x16 + umulh x6, x6, x16 + adds x7, x7, x5 + adcs x6, x6, x16 + adc x5, xzr, xzr + subs x17, x17, x7 + sbcs x12, x12, x6 + sbcs x13, x13, x5 + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbc x16, x16, xzr + lsl x7, x17, #32 + add x17, x7, x17 + mov x7, #-4294967295 + umulh x7, x7, x17 + mov x6, #4294967295 + mul x5, x6, x17 + umulh x6, x6, x17 + adds x7, x7, x5 + adcs x6, x6, x17 + adc x5, xzr, xzr + subs x12, x12, x7 + sbcs x13, x13, x6 + sbcs x14, x14, x5 + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbc x17, x17, xzr + adds x12, x12, x19 + adcs x13, x13, x20 + adcs x14, x14, x21 + adcs x15, x15, x22 + adcs x16, x16, x2 + adcs x17, x17, x1 + adc x10, xzr, xzr + mov x11, #-4294967295 + adds x19, x12, x11 + mov x11, #4294967295 + adcs x20, x13, x11 + mov x11, #1 + adcs x21, x14, x11 + adcs x22, x15, xzr + adcs x2, x16, xzr + adcs x1, x17, xzr + adcs x10, x10, xzr + csel x12, x12, x19, eq + csel x13, x13, x20, eq + csel x14, x14, x21, eq + csel x15, x15, x22, eq + csel x16, x16, x2, eq + csel x17, x17, x1, eq + stp x12, x13, [sp, #48] + stp x14, x15, [sp, #64] + stp x16, x17, [sp, #80] + ldp x3, x4, [sp, #240] + ldp x5, x6, [sp, #288] + mul x12, x3, x5 + umulh x13, x3, x5 + mul x11, x3, x6 + umulh x14, x3, x6 + adds x13, x13, x11 + ldp x7, x8, [sp, #304] + mul x11, x3, x7 + umulh x15, x3, x7 + adcs x14, x14, x11 + mul x11, x3, x8 + umulh x16, x3, x8 + adcs x15, x15, x11 + ldp x9, x10, [sp, #320] + mul x11, x3, x9 + umulh x17, x3, x9 + adcs x16, x16, x11 + mul x11, x3, x10 + umulh x19, x3, x10 + adcs x17, x17, x11 + adc x19, x19, xzr + mul x11, x4, x5 + adds x13, x13, x11 + mul x11, x4, x6 + adcs x14, x14, x11 + mul x11, x4, x7 + adcs x15, x15, x11 + mul x11, x4, x8 + adcs x16, x16, x11 + mul x11, x4, x9 + adcs x17, x17, x11 + mul x11, x4, x10 + adcs x19, x19, x11 + cset x20, hs + umulh x11, x4, x5 + adds x14, x14, x11 + umulh x11, x4, x6 + adcs x15, x15, x11 + umulh x11, x4, x7 + adcs x16, x16, x11 + umulh x11, x4, x8 + adcs x17, x17, x11 + umulh x11, x4, x9 + adcs x19, x19, x11 + umulh x11, x4, x10 + adc x20, x20, x11 + ldp x3, x4, [sp, #256] + mul x11, x3, x5 + adds x14, x14, x11 + mul x11, x3, x6 + adcs x15, x15, x11 + mul x11, x3, x7 + adcs x16, x16, x11 + mul x11, x3, x8 + adcs x17, x17, x11 + mul x11, x3, x9 + adcs x19, x19, x11 + mul x11, x3, x10 + adcs x20, x20, x11 + cset x21, hs + umulh x11, x3, x5 + adds x15, x15, x11 + umulh x11, x3, x6 + adcs x16, x16, x11 + umulh x11, x3, x7 + adcs x17, x17, x11 + umulh x11, x3, x8 + adcs x19, x19, x11 + umulh x11, x3, x9 + adcs x20, x20, x11 + umulh x11, x3, x10 + adc x21, x21, x11 + mul x11, x4, x5 + adds x15, x15, x11 + mul x11, x4, x6 + adcs x16, x16, x11 + mul x11, x4, x7 + adcs x17, x17, x11 + mul x11, x4, x8 + adcs x19, x19, x11 + mul x11, x4, x9 + adcs x20, x20, x11 + mul x11, x4, x10 + adcs x21, x21, x11 + cset x22, hs + umulh x11, x4, x5 + adds x16, x16, x11 + umulh x11, x4, x6 + adcs x17, x17, x11 + umulh x11, x4, x7 + adcs x19, x19, x11 + umulh x11, x4, x8 + adcs x20, x20, x11 + umulh x11, x4, x9 + adcs x21, x21, x11 + umulh x11, x4, x10 + adc x22, x22, x11 + ldp x3, x4, [sp, #272] + mul x11, x3, x5 + adds x16, x16, x11 + mul x11, x3, x6 + adcs x17, x17, x11 + mul x11, x3, x7 + adcs x19, x19, x11 + mul x11, x3, x8 + adcs x20, x20, x11 + mul x11, x3, x9 + adcs x21, x21, x11 + mul x11, x3, x10 + adcs x22, x22, x11 + cset x2, hs + umulh x11, x3, x5 + adds x17, x17, x11 + umulh x11, x3, x6 + adcs x19, x19, x11 + umulh x11, x3, x7 + adcs x20, x20, x11 + umulh x11, x3, x8 + adcs x21, x21, x11 + umulh x11, x3, x9 + adcs x22, x22, x11 + umulh x11, x3, x10 + adc x2, x2, x11 + mul x11, x4, x5 + adds x17, x17, x11 + mul x11, x4, x6 + adcs x19, x19, x11 + mul x11, x4, x7 + adcs x20, x20, x11 + mul x11, x4, x8 + adcs x21, x21, x11 + mul x11, x4, x9 + adcs x22, x22, x11 + mul x11, x4, x10 + adcs x2, x2, x11 + cset x1, hs + umulh x11, x4, x5 + adds x19, x19, x11 + umulh x11, x4, x6 + adcs x20, x20, x11 + umulh x11, x4, x7 + adcs x21, x21, x11 + umulh x11, x4, x8 + adcs x22, x22, x11 + umulh x11, x4, x9 + adcs x2, x2, x11 + umulh x11, x4, x10 + adc x1, x1, x11 + lsl x7, x12, #32 + add x12, x7, x12 + mov x7, #-4294967295 + umulh x7, x7, x12 + mov x6, #4294967295 + mul x5, x6, x12 + umulh x6, x6, x12 + adds x7, x7, x5 + adcs x6, x6, x12 + adc x5, xzr, xzr + subs x13, x13, x7 + sbcs x14, x14, x6 + sbcs x15, x15, x5 + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x12, x12, xzr + lsl x7, x13, #32 + add x13, x7, x13 + mov x7, #-4294967295 + umulh x7, x7, x13 + mov x6, #4294967295 + mul x5, x6, x13 + umulh x6, x6, x13 + adds x7, x7, x5 + adcs x6, x6, x13 + adc x5, xzr, xzr + subs x14, x14, x7 + sbcs x15, x15, x6 + sbcs x16, x16, x5 + sbcs x17, x17, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + lsl x7, x14, #32 + add x14, x7, x14 + mov x7, #-4294967295 + umulh x7, x7, x14 + mov x6, #4294967295 + mul x5, x6, x14 + umulh x6, x6, x14 + adds x7, x7, x5 + adcs x6, x6, x14 + adc x5, xzr, xzr + subs x15, x15, x7 + sbcs x16, x16, x6 + sbcs x17, x17, x5 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + lsl x7, x15, #32 + add x15, x7, x15 + mov x7, #-4294967295 + umulh x7, x7, x15 + mov x6, #4294967295 + mul x5, x6, x15 + umulh x6, x6, x15 + adds x7, x7, x5 + adcs x6, x6, x15 + adc x5, xzr, xzr + subs x16, x16, x7 + sbcs x17, x17, x6 + sbcs x12, x12, x5 + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbc x15, x15, xzr + lsl x7, x16, #32 + add x16, x7, x16 + mov x7, #-4294967295 + umulh x7, x7, x16 + mov x6, #4294967295 + mul x5, x6, x16 + umulh x6, x6, x16 + adds x7, x7, x5 + adcs x6, x6, x16 + adc x5, xzr, xzr + subs x17, x17, x7 + sbcs x12, x12, x6 + sbcs x13, x13, x5 + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbc x16, x16, xzr + lsl x7, x17, #32 + add x17, x7, x17 + mov x7, #-4294967295 + umulh x7, x7, x17 + mov x6, #4294967295 + mul x5, x6, x17 + umulh x6, x6, x17 + adds x7, x7, x5 + adcs x6, x6, x17 + adc x5, xzr, xzr + subs x12, x12, x7 + sbcs x13, x13, x6 + sbcs x14, x14, x5 + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbc x17, x17, xzr + adds x12, x12, x19 + adcs x13, x13, x20 + adcs x14, x14, x21 + adcs x15, x15, x22 + adcs x16, x16, x2 + adcs x17, x17, x1 + adc x10, xzr, xzr + mov x11, #-4294967295 + adds x19, x12, x11 + mov x11, #4294967295 + adcs x20, x13, x11 + mov x11, #1 + adcs x21, x14, x11 + adcs x22, x15, xzr + adcs x2, x16, xzr + adcs x1, x17, xzr + adcs x10, x10, xzr + csel x12, x12, x19, eq + csel x13, x13, x20, eq + csel x14, x14, x21, eq + csel x15, x15, x22, eq + csel x16, x16, x2, eq + csel x17, x17, x1, eq + stp x12, x13, [sp, #288] + stp x14, x15, [sp, #304] + stp x16, x17, [sp, #320] + ldp x5, x6, [sp, #96] + ldp x4, x3, [sp, #192] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #112] + ldp x4, x3, [sp, #208] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #128] + ldp x4, x3, [sp, #224] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, lo + mov x4, #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [sp, #240] + stp x7, x8, [sp, #256] + stp x9, x10, [sp, #272] + ldp x5, x6, [sp, #48] + ldp x4, x3, [sp, #288] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #64] + ldp x4, x3, [sp, #304] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #80] + ldp x4, x3, [sp, #320] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, lo + mov x4, #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [sp, #48] + stp x7, x8, [sp, #64] + stp x9, x10, [sp, #80] + ldp x2, x3, [sp, #240] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [sp, #256] + mul x8, x2, x4 + adds x10, x10, x8 + mul x11, x2, x5 + mul x8, x3, x4 + adcs x11, x11, x8 + umulh x12, x2, x5 + mul x8, x3, x5 + adcs x12, x12, x8 + ldp x6, x7, [sp, #272] + mul x13, x2, x7 + mul x8, x3, x6 + adcs x13, x13, x8 + umulh x14, x2, x7 + mul x8, x3, x7 + adcs x14, x14, x8 + mul x15, x5, x6 + adcs x15, x15, xzr + umulh x16, x5, x6 + adc x16, x16, xzr + umulh x8, x2, x4 + adds x11, x11, x8 + umulh x8, x3, x4 + adcs x12, x12, x8 + umulh x8, x3, x5 + adcs x13, x13, x8 + umulh x8, x3, x6 + adcs x14, x14, x8 + umulh x8, x3, x7 + adcs x15, x15, x8 + adc x16, x16, xzr + mul x8, x2, x6 + adds x12, x12, x8 + mul x8, x4, x5 + adcs x13, x13, x8 + mul x8, x4, x6 + adcs x14, x14, x8 + mul x8, x4, x7 + adcs x15, x15, x8 + mul x8, x5, x7 + adcs x16, x16, x8 + mul x17, x6, x7 + adcs x17, x17, xzr + umulh x19, x6, x7 + adc x19, x19, xzr + umulh x8, x2, x6 + adds x13, x13, x8 + umulh x8, x4, x5 + adcs x14, x14, x8 + umulh x8, x4, x6 + adcs x15, x15, x8 + umulh x8, x4, x7 + adcs x16, x16, x8 + umulh x8, x5, x7 + adcs x17, x17, x8 + adc x19, x19, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + adcs x15, x15, x15 + adcs x16, x16, x16 + adcs x17, x17, x17 + adcs x19, x19, x19 + cset x20, hs + umulh x8, x2, x2 + mul x2, x2, x2 + adds x9, x9, x8 + mul x8, x3, x3 + adcs x10, x10, x8 + umulh x8, x3, x3 + adcs x11, x11, x8 + mul x8, x4, x4 + adcs x12, x12, x8 + umulh x8, x4, x4 + adcs x13, x13, x8 + mul x8, x5, x5 + adcs x14, x14, x8 + umulh x8, x5, x5 + adcs x15, x15, x8 + mul x8, x6, x6 + adcs x16, x16, x8 + umulh x8, x6, x6 + adcs x17, x17, x8 + mul x8, x7, x7 + adcs x19, x19, x8 + umulh x8, x7, x7 + adc x20, x20, x8 + lsl x5, x2, #32 + add x2, x5, x2 + mov x5, #-4294967295 + umulh x5, x5, x2 + mov x4, #4294967295 + mul x3, x4, x2 + umulh x4, x4, x2 + adds x5, x5, x3 + adcs x4, x4, x2 + adc x3, xzr, xzr + subs x9, x9, x5 + sbcs x10, x10, x4 + sbcs x11, x11, x3 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x2, x2, xzr + lsl x5, x9, #32 + add x9, x5, x9 + mov x5, #-4294967295 + umulh x5, x5, x9 + mov x4, #4294967295 + mul x3, x4, x9 + umulh x4, x4, x9 + adds x5, x5, x3 + adcs x4, x4, x9 + adc x3, xzr, xzr + subs x10, x10, x5 + sbcs x11, x11, x4 + sbcs x12, x12, x3 + sbcs x13, x13, xzr + sbcs x2, x2, xzr + sbc x9, x9, xzr + lsl x5, x10, #32 + add x10, x5, x10 + mov x5, #-4294967295 + umulh x5, x5, x10 + mov x4, #4294967295 + mul x3, x4, x10 + umulh x4, x4, x10 + adds x5, x5, x3 + adcs x4, x4, x10 + adc x3, xzr, xzr + subs x11, x11, x5 + sbcs x12, x12, x4 + sbcs x13, x13, x3 + sbcs x2, x2, xzr + sbcs x9, x9, xzr + sbc x10, x10, xzr + lsl x5, x11, #32 + add x11, x5, x11 + mov x5, #-4294967295 + umulh x5, x5, x11 + mov x4, #4294967295 + mul x3, x4, x11 + umulh x4, x4, x11 + adds x5, x5, x3 + adcs x4, x4, x11 + adc x3, xzr, xzr + subs x12, x12, x5 + sbcs x13, x13, x4 + sbcs x2, x2, x3 + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbc x11, x11, xzr + lsl x5, x12, #32 + add x12, x5, x12 + mov x5, #-4294967295 + umulh x5, x5, x12 + mov x4, #4294967295 + mul x3, x4, x12 + umulh x4, x4, x12 + adds x5, x5, x3 + adcs x4, x4, x12 + adc x3, xzr, xzr + subs x13, x13, x5 + sbcs x2, x2, x4 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbc x12, x12, xzr + lsl x5, x13, #32 + add x13, x5, x13 + mov x5, #-4294967295 + umulh x5, x5, x13 + mov x4, #4294967295 + mul x3, x4, x13 + umulh x4, x4, x13 + adds x5, x5, x3 + adcs x4, x4, x13 + adc x3, xzr, xzr + subs x2, x2, x5 + sbcs x9, x9, x4 + sbcs x10, x10, x3 + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + adds x2, x2, x14 + adcs x9, x9, x15 + adcs x10, x10, x16 + adcs x11, x11, x17 + adcs x12, x12, x19 + adcs x13, x13, x20 + mov x14, #-4294967295 + mov x15, #4294967295 + csel x14, x14, xzr, hs + csel x15, x15, xzr, hs + cset x16, hs + adds x2, x2, x14 + adcs x9, x9, x15 + adcs x10, x10, x16 + adcs x11, x11, xzr + adcs x12, x12, xzr + adc x13, x13, xzr + stp x2, x9, [sp, #144] + stp x10, x11, [sp, #160] + stp x12, x13, [sp, #176] + ldp x2, x3, [sp, #48] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [sp, #64] + mul x8, x2, x4 + adds x10, x10, x8 + mul x11, x2, x5 + mul x8, x3, x4 + adcs x11, x11, x8 + umulh x12, x2, x5 + mul x8, x3, x5 + adcs x12, x12, x8 + ldp x6, x7, [sp, #80] + mul x13, x2, x7 + mul x8, x3, x6 + adcs x13, x13, x8 + umulh x14, x2, x7 + mul x8, x3, x7 + adcs x14, x14, x8 + mul x15, x5, x6 + adcs x15, x15, xzr + umulh x16, x5, x6 + adc x16, x16, xzr + umulh x8, x2, x4 + adds x11, x11, x8 + umulh x8, x3, x4 + adcs x12, x12, x8 + umulh x8, x3, x5 + adcs x13, x13, x8 + umulh x8, x3, x6 + adcs x14, x14, x8 + umulh x8, x3, x7 + adcs x15, x15, x8 + adc x16, x16, xzr + mul x8, x2, x6 + adds x12, x12, x8 + mul x8, x4, x5 + adcs x13, x13, x8 + mul x8, x4, x6 + adcs x14, x14, x8 + mul x8, x4, x7 + adcs x15, x15, x8 + mul x8, x5, x7 + adcs x16, x16, x8 + mul x17, x6, x7 + adcs x17, x17, xzr + umulh x19, x6, x7 + adc x19, x19, xzr + umulh x8, x2, x6 + adds x13, x13, x8 + umulh x8, x4, x5 + adcs x14, x14, x8 + umulh x8, x4, x6 + adcs x15, x15, x8 + umulh x8, x4, x7 + adcs x16, x16, x8 + umulh x8, x5, x7 + adcs x17, x17, x8 + adc x19, x19, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + adcs x15, x15, x15 + adcs x16, x16, x16 + adcs x17, x17, x17 + adcs x19, x19, x19 + cset x20, hs + umulh x8, x2, x2 + mul x2, x2, x2 + adds x9, x9, x8 + mul x8, x3, x3 + adcs x10, x10, x8 + umulh x8, x3, x3 + adcs x11, x11, x8 + mul x8, x4, x4 + adcs x12, x12, x8 + umulh x8, x4, x4 + adcs x13, x13, x8 + mul x8, x5, x5 + adcs x14, x14, x8 + umulh x8, x5, x5 + adcs x15, x15, x8 + mul x8, x6, x6 + adcs x16, x16, x8 + umulh x8, x6, x6 + adcs x17, x17, x8 + mul x8, x7, x7 + adcs x19, x19, x8 + umulh x8, x7, x7 + adc x20, x20, x8 + lsl x5, x2, #32 + add x2, x5, x2 + mov x5, #-4294967295 + umulh x5, x5, x2 + mov x4, #4294967295 + mul x3, x4, x2 + umulh x4, x4, x2 + adds x5, x5, x3 + adcs x4, x4, x2 + adc x3, xzr, xzr + subs x9, x9, x5 + sbcs x10, x10, x4 + sbcs x11, x11, x3 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x2, x2, xzr + lsl x5, x9, #32 + add x9, x5, x9 + mov x5, #-4294967295 + umulh x5, x5, x9 + mov x4, #4294967295 + mul x3, x4, x9 + umulh x4, x4, x9 + adds x5, x5, x3 + adcs x4, x4, x9 + adc x3, xzr, xzr + subs x10, x10, x5 + sbcs x11, x11, x4 + sbcs x12, x12, x3 + sbcs x13, x13, xzr + sbcs x2, x2, xzr + sbc x9, x9, xzr + lsl x5, x10, #32 + add x10, x5, x10 + mov x5, #-4294967295 + umulh x5, x5, x10 + mov x4, #4294967295 + mul x3, x4, x10 + umulh x4, x4, x10 + adds x5, x5, x3 + adcs x4, x4, x10 + adc x3, xzr, xzr + subs x11, x11, x5 + sbcs x12, x12, x4 + sbcs x13, x13, x3 + sbcs x2, x2, xzr + sbcs x9, x9, xzr + sbc x10, x10, xzr + lsl x5, x11, #32 + add x11, x5, x11 + mov x5, #-4294967295 + umulh x5, x5, x11 + mov x4, #4294967295 + mul x3, x4, x11 + umulh x4, x4, x11 + adds x5, x5, x3 + adcs x4, x4, x11 + adc x3, xzr, xzr + subs x12, x12, x5 + sbcs x13, x13, x4 + sbcs x2, x2, x3 + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbc x11, x11, xzr + lsl x5, x12, #32 + add x12, x5, x12 + mov x5, #-4294967295 + umulh x5, x5, x12 + mov x4, #4294967295 + mul x3, x4, x12 + umulh x4, x4, x12 + adds x5, x5, x3 + adcs x4, x4, x12 + adc x3, xzr, xzr + subs x13, x13, x5 + sbcs x2, x2, x4 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbc x12, x12, xzr + lsl x5, x13, #32 + add x13, x5, x13 + mov x5, #-4294967295 + umulh x5, x5, x13 + mov x4, #4294967295 + mul x3, x4, x13 + umulh x4, x4, x13 + adds x5, x5, x3 + adcs x4, x4, x13 + adc x3, xzr, xzr + subs x2, x2, x5 + sbcs x9, x9, x4 + sbcs x10, x10, x3 + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + adds x2, x2, x14 + adcs x9, x9, x15 + adcs x10, x10, x16 + adcs x11, x11, x17 + adcs x12, x12, x19 + adcs x13, x13, x20 + adc x6, xzr, xzr + mov x8, #-4294967295 + adds x14, x2, x8 + mov x8, #4294967295 + adcs x15, x9, x8 + mov x8, #1 + adcs x16, x10, x8 + adcs x17, x11, xzr + adcs x19, x12, xzr + adcs x20, x13, xzr + adcs x6, x6, xzr + csel x2, x2, x14, eq + csel x9, x9, x15, eq + csel x10, x10, x16, eq + csel x11, x11, x17, eq + csel x12, x12, x19, eq + csel x13, x13, x20, eq + stp x2, x9, [sp] + stp x10, x11, [sp, #16] + stp x12, x13, [sp, #32] + ldp x3, x4, [sp, #144] + ldp x5, x6, [sp, #192] + mul x12, x3, x5 + umulh x13, x3, x5 + mul x11, x3, x6 + umulh x14, x3, x6 + adds x13, x13, x11 + ldp x7, x8, [sp, #208] + mul x11, x3, x7 + umulh x15, x3, x7 + adcs x14, x14, x11 + mul x11, x3, x8 + umulh x16, x3, x8 + adcs x15, x15, x11 + ldp x9, x10, [sp, #224] + mul x11, x3, x9 + umulh x17, x3, x9 + adcs x16, x16, x11 + mul x11, x3, x10 + umulh x19, x3, x10 + adcs x17, x17, x11 + adc x19, x19, xzr + mul x11, x4, x5 + adds x13, x13, x11 + mul x11, x4, x6 + adcs x14, x14, x11 + mul x11, x4, x7 + adcs x15, x15, x11 + mul x11, x4, x8 + adcs x16, x16, x11 + mul x11, x4, x9 + adcs x17, x17, x11 + mul x11, x4, x10 + adcs x19, x19, x11 + cset x20, hs + umulh x11, x4, x5 + adds x14, x14, x11 + umulh x11, x4, x6 + adcs x15, x15, x11 + umulh x11, x4, x7 + adcs x16, x16, x11 + umulh x11, x4, x8 + adcs x17, x17, x11 + umulh x11, x4, x9 + adcs x19, x19, x11 + umulh x11, x4, x10 + adc x20, x20, x11 + ldp x3, x4, [sp, #160] + mul x11, x3, x5 + adds x14, x14, x11 + mul x11, x3, x6 + adcs x15, x15, x11 + mul x11, x3, x7 + adcs x16, x16, x11 + mul x11, x3, x8 + adcs x17, x17, x11 + mul x11, x3, x9 + adcs x19, x19, x11 + mul x11, x3, x10 + adcs x20, x20, x11 + cset x21, hs + umulh x11, x3, x5 + adds x15, x15, x11 + umulh x11, x3, x6 + adcs x16, x16, x11 + umulh x11, x3, x7 + adcs x17, x17, x11 + umulh x11, x3, x8 + adcs x19, x19, x11 + umulh x11, x3, x9 + adcs x20, x20, x11 + umulh x11, x3, x10 + adc x21, x21, x11 + mul x11, x4, x5 + adds x15, x15, x11 + mul x11, x4, x6 + adcs x16, x16, x11 + mul x11, x4, x7 + adcs x17, x17, x11 + mul x11, x4, x8 + adcs x19, x19, x11 + mul x11, x4, x9 + adcs x20, x20, x11 + mul x11, x4, x10 + adcs x21, x21, x11 + cset x22, hs + umulh x11, x4, x5 + adds x16, x16, x11 + umulh x11, x4, x6 + adcs x17, x17, x11 + umulh x11, x4, x7 + adcs x19, x19, x11 + umulh x11, x4, x8 + adcs x20, x20, x11 + umulh x11, x4, x9 + adcs x21, x21, x11 + umulh x11, x4, x10 + adc x22, x22, x11 + ldp x3, x4, [sp, #176] + mul x11, x3, x5 + adds x16, x16, x11 + mul x11, x3, x6 + adcs x17, x17, x11 + mul x11, x3, x7 + adcs x19, x19, x11 + mul x11, x3, x8 + adcs x20, x20, x11 + mul x11, x3, x9 + adcs x21, x21, x11 + mul x11, x3, x10 + adcs x22, x22, x11 + cset x2, hs + umulh x11, x3, x5 + adds x17, x17, x11 + umulh x11, x3, x6 + adcs x19, x19, x11 + umulh x11, x3, x7 + adcs x20, x20, x11 + umulh x11, x3, x8 + adcs x21, x21, x11 + umulh x11, x3, x9 + adcs x22, x22, x11 + umulh x11, x3, x10 + adc x2, x2, x11 + mul x11, x4, x5 + adds x17, x17, x11 + mul x11, x4, x6 + adcs x19, x19, x11 + mul x11, x4, x7 + adcs x20, x20, x11 + mul x11, x4, x8 + adcs x21, x21, x11 + mul x11, x4, x9 + adcs x22, x22, x11 + mul x11, x4, x10 + adcs x2, x2, x11 + cset x1, hs + umulh x11, x4, x5 + adds x19, x19, x11 + umulh x11, x4, x6 + adcs x20, x20, x11 + umulh x11, x4, x7 + adcs x21, x21, x11 + umulh x11, x4, x8 + adcs x22, x22, x11 + umulh x11, x4, x9 + adcs x2, x2, x11 + umulh x11, x4, x10 + adc x1, x1, x11 + lsl x7, x12, #32 + add x12, x7, x12 + mov x7, #-4294967295 + umulh x7, x7, x12 + mov x6, #4294967295 + mul x5, x6, x12 + umulh x6, x6, x12 + adds x7, x7, x5 + adcs x6, x6, x12 + adc x5, xzr, xzr + subs x13, x13, x7 + sbcs x14, x14, x6 + sbcs x15, x15, x5 + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x12, x12, xzr + lsl x7, x13, #32 + add x13, x7, x13 + mov x7, #-4294967295 + umulh x7, x7, x13 + mov x6, #4294967295 + mul x5, x6, x13 + umulh x6, x6, x13 + adds x7, x7, x5 + adcs x6, x6, x13 + adc x5, xzr, xzr + subs x14, x14, x7 + sbcs x15, x15, x6 + sbcs x16, x16, x5 + sbcs x17, x17, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + lsl x7, x14, #32 + add x14, x7, x14 + mov x7, #-4294967295 + umulh x7, x7, x14 + mov x6, #4294967295 + mul x5, x6, x14 + umulh x6, x6, x14 + adds x7, x7, x5 + adcs x6, x6, x14 + adc x5, xzr, xzr + subs x15, x15, x7 + sbcs x16, x16, x6 + sbcs x17, x17, x5 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + lsl x7, x15, #32 + add x15, x7, x15 + mov x7, #-4294967295 + umulh x7, x7, x15 + mov x6, #4294967295 + mul x5, x6, x15 + umulh x6, x6, x15 + adds x7, x7, x5 + adcs x6, x6, x15 + adc x5, xzr, xzr + subs x16, x16, x7 + sbcs x17, x17, x6 + sbcs x12, x12, x5 + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbc x15, x15, xzr + lsl x7, x16, #32 + add x16, x7, x16 + mov x7, #-4294967295 + umulh x7, x7, x16 + mov x6, #4294967295 + mul x5, x6, x16 + umulh x6, x6, x16 + adds x7, x7, x5 + adcs x6, x6, x16 + adc x5, xzr, xzr + subs x17, x17, x7 + sbcs x12, x12, x6 + sbcs x13, x13, x5 + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbc x16, x16, xzr + lsl x7, x17, #32 + add x17, x7, x17 + mov x7, #-4294967295 + umulh x7, x7, x17 + mov x6, #4294967295 + mul x5, x6, x17 + umulh x6, x6, x17 + adds x7, x7, x5 + adcs x6, x6, x17 + adc x5, xzr, xzr + subs x12, x12, x7 + sbcs x13, x13, x6 + sbcs x14, x14, x5 + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbc x17, x17, xzr + adds x12, x12, x19 + adcs x13, x13, x20 + adcs x14, x14, x21 + adcs x15, x15, x22 + adcs x16, x16, x2 + adcs x17, x17, x1 + adc x10, xzr, xzr + mov x11, #-4294967295 + adds x19, x12, x11 + mov x11, #4294967295 + adcs x20, x13, x11 + mov x11, #1 + adcs x21, x14, x11 + adcs x22, x15, xzr + adcs x2, x16, xzr + adcs x1, x17, xzr + adcs x10, x10, xzr + csel x12, x12, x19, eq + csel x13, x13, x20, eq + csel x14, x14, x21, eq + csel x15, x15, x22, eq + csel x16, x16, x2, eq + csel x17, x17, x1, eq + stp x12, x13, [sp, #192] + stp x14, x15, [sp, #208] + stp x16, x17, [sp, #224] + ldp x3, x4, [sp, #144] + ldp x5, x6, [sp, #96] + mul x12, x3, x5 + umulh x13, x3, x5 + mul x11, x3, x6 + umulh x14, x3, x6 + adds x13, x13, x11 + ldp x7, x8, [sp, #112] + mul x11, x3, x7 + umulh x15, x3, x7 + adcs x14, x14, x11 + mul x11, x3, x8 + umulh x16, x3, x8 + adcs x15, x15, x11 + ldp x9, x10, [sp, #128] + mul x11, x3, x9 + umulh x17, x3, x9 + adcs x16, x16, x11 + mul x11, x3, x10 + umulh x19, x3, x10 + adcs x17, x17, x11 + adc x19, x19, xzr + mul x11, x4, x5 + adds x13, x13, x11 + mul x11, x4, x6 + adcs x14, x14, x11 + mul x11, x4, x7 + adcs x15, x15, x11 + mul x11, x4, x8 + adcs x16, x16, x11 + mul x11, x4, x9 + adcs x17, x17, x11 + mul x11, x4, x10 + adcs x19, x19, x11 + cset x20, hs + umulh x11, x4, x5 + adds x14, x14, x11 + umulh x11, x4, x6 + adcs x15, x15, x11 + umulh x11, x4, x7 + adcs x16, x16, x11 + umulh x11, x4, x8 + adcs x17, x17, x11 + umulh x11, x4, x9 + adcs x19, x19, x11 + umulh x11, x4, x10 + adc x20, x20, x11 + ldp x3, x4, [sp, #160] + mul x11, x3, x5 + adds x14, x14, x11 + mul x11, x3, x6 + adcs x15, x15, x11 + mul x11, x3, x7 + adcs x16, x16, x11 + mul x11, x3, x8 + adcs x17, x17, x11 + mul x11, x3, x9 + adcs x19, x19, x11 + mul x11, x3, x10 + adcs x20, x20, x11 + cset x21, hs + umulh x11, x3, x5 + adds x15, x15, x11 + umulh x11, x3, x6 + adcs x16, x16, x11 + umulh x11, x3, x7 + adcs x17, x17, x11 + umulh x11, x3, x8 + adcs x19, x19, x11 + umulh x11, x3, x9 + adcs x20, x20, x11 + umulh x11, x3, x10 + adc x21, x21, x11 + mul x11, x4, x5 + adds x15, x15, x11 + mul x11, x4, x6 + adcs x16, x16, x11 + mul x11, x4, x7 + adcs x17, x17, x11 + mul x11, x4, x8 + adcs x19, x19, x11 + mul x11, x4, x9 + adcs x20, x20, x11 + mul x11, x4, x10 + adcs x21, x21, x11 + cset x22, hs + umulh x11, x4, x5 + adds x16, x16, x11 + umulh x11, x4, x6 + adcs x17, x17, x11 + umulh x11, x4, x7 + adcs x19, x19, x11 + umulh x11, x4, x8 + adcs x20, x20, x11 + umulh x11, x4, x9 + adcs x21, x21, x11 + umulh x11, x4, x10 + adc x22, x22, x11 + ldp x3, x4, [sp, #176] + mul x11, x3, x5 + adds x16, x16, x11 + mul x11, x3, x6 + adcs x17, x17, x11 + mul x11, x3, x7 + adcs x19, x19, x11 + mul x11, x3, x8 + adcs x20, x20, x11 + mul x11, x3, x9 + adcs x21, x21, x11 + mul x11, x3, x10 + adcs x22, x22, x11 + cset x2, hs + umulh x11, x3, x5 + adds x17, x17, x11 + umulh x11, x3, x6 + adcs x19, x19, x11 + umulh x11, x3, x7 + adcs x20, x20, x11 + umulh x11, x3, x8 + adcs x21, x21, x11 + umulh x11, x3, x9 + adcs x22, x22, x11 + umulh x11, x3, x10 + adc x2, x2, x11 + mul x11, x4, x5 + adds x17, x17, x11 + mul x11, x4, x6 + adcs x19, x19, x11 + mul x11, x4, x7 + adcs x20, x20, x11 + mul x11, x4, x8 + adcs x21, x21, x11 + mul x11, x4, x9 + adcs x22, x22, x11 + mul x11, x4, x10 + adcs x2, x2, x11 + cset x1, hs + umulh x11, x4, x5 + adds x19, x19, x11 + umulh x11, x4, x6 + adcs x20, x20, x11 + umulh x11, x4, x7 + adcs x21, x21, x11 + umulh x11, x4, x8 + adcs x22, x22, x11 + umulh x11, x4, x9 + adcs x2, x2, x11 + umulh x11, x4, x10 + adc x1, x1, x11 + lsl x7, x12, #32 + add x12, x7, x12 + mov x7, #-4294967295 + umulh x7, x7, x12 + mov x6, #4294967295 + mul x5, x6, x12 + umulh x6, x6, x12 + adds x7, x7, x5 + adcs x6, x6, x12 + adc x5, xzr, xzr + subs x13, x13, x7 + sbcs x14, x14, x6 + sbcs x15, x15, x5 + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x12, x12, xzr + lsl x7, x13, #32 + add x13, x7, x13 + mov x7, #-4294967295 + umulh x7, x7, x13 + mov x6, #4294967295 + mul x5, x6, x13 + umulh x6, x6, x13 + adds x7, x7, x5 + adcs x6, x6, x13 + adc x5, xzr, xzr + subs x14, x14, x7 + sbcs x15, x15, x6 + sbcs x16, x16, x5 + sbcs x17, x17, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + lsl x7, x14, #32 + add x14, x7, x14 + mov x7, #-4294967295 + umulh x7, x7, x14 + mov x6, #4294967295 + mul x5, x6, x14 + umulh x6, x6, x14 + adds x7, x7, x5 + adcs x6, x6, x14 + adc x5, xzr, xzr + subs x15, x15, x7 + sbcs x16, x16, x6 + sbcs x17, x17, x5 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + lsl x7, x15, #32 + add x15, x7, x15 + mov x7, #-4294967295 + umulh x7, x7, x15 + mov x6, #4294967295 + mul x5, x6, x15 + umulh x6, x6, x15 + adds x7, x7, x5 + adcs x6, x6, x15 + adc x5, xzr, xzr + subs x16, x16, x7 + sbcs x17, x17, x6 + sbcs x12, x12, x5 + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbc x15, x15, xzr + lsl x7, x16, #32 + add x16, x7, x16 + mov x7, #-4294967295 + umulh x7, x7, x16 + mov x6, #4294967295 + mul x5, x6, x16 + umulh x6, x6, x16 + adds x7, x7, x5 + adcs x6, x6, x16 + adc x5, xzr, xzr + subs x17, x17, x7 + sbcs x12, x12, x6 + sbcs x13, x13, x5 + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbc x16, x16, xzr + lsl x7, x17, #32 + add x17, x7, x17 + mov x7, #-4294967295 + umulh x7, x7, x17 + mov x6, #4294967295 + mul x5, x6, x17 + umulh x6, x6, x17 + adds x7, x7, x5 + adcs x6, x6, x17 + adc x5, xzr, xzr + subs x12, x12, x7 + sbcs x13, x13, x6 + sbcs x14, x14, x5 + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbc x17, x17, xzr + adds x12, x12, x19 + adcs x13, x13, x20 + adcs x14, x14, x21 + adcs x15, x15, x22 + adcs x16, x16, x2 + adcs x17, x17, x1 + adc x10, xzr, xzr + mov x11, #-4294967295 + adds x19, x12, x11 + mov x11, #4294967295 + adcs x20, x13, x11 + mov x11, #1 + adcs x21, x14, x11 + adcs x22, x15, xzr + adcs x2, x16, xzr + adcs x1, x17, xzr + adcs x10, x10, xzr + csel x12, x12, x19, eq + csel x13, x13, x20, eq + csel x14, x14, x21, eq + csel x15, x15, x22, eq + csel x16, x16, x2, eq + csel x17, x17, x1, eq + stp x12, x13, [sp, #96] + stp x14, x15, [sp, #112] + stp x16, x17, [sp, #128] + ldp x5, x6, [sp] + ldp x4, x3, [sp, #192] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #16] + ldp x4, x3, [sp, #208] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #32] + ldp x4, x3, [sp, #224] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, lo + mov x4, #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [sp] + stp x7, x8, [sp, #16] + stp x9, x10, [sp, #32] + ldp x5, x6, [sp, #96] + ldp x4, x3, [sp, #192] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #112] + ldp x4, x3, [sp, #208] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #128] + ldp x4, x3, [sp, #224] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, lo + mov x4, #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [sp, #144] + stp x7, x8, [sp, #160] + stp x9, x10, [sp, #176] + ldp x3, x4, [sp, #240] + ldp x5, x6, [x25, #96] + mul x12, x3, x5 + umulh x13, x3, x5 + mul x11, x3, x6 + umulh x14, x3, x6 + adds x13, x13, x11 + ldp x7, x8, [x25, #112] + mul x11, x3, x7 + umulh x15, x3, x7 + adcs x14, x14, x11 + mul x11, x3, x8 + umulh x16, x3, x8 + adcs x15, x15, x11 + ldp x9, x10, [x25, #128] + mul x11, x3, x9 + umulh x17, x3, x9 + adcs x16, x16, x11 + mul x11, x3, x10 + umulh x19, x3, x10 + adcs x17, x17, x11 + adc x19, x19, xzr + mul x11, x4, x5 + adds x13, x13, x11 + mul x11, x4, x6 + adcs x14, x14, x11 + mul x11, x4, x7 + adcs x15, x15, x11 + mul x11, x4, x8 + adcs x16, x16, x11 + mul x11, x4, x9 + adcs x17, x17, x11 + mul x11, x4, x10 + adcs x19, x19, x11 + cset x20, hs + umulh x11, x4, x5 + adds x14, x14, x11 + umulh x11, x4, x6 + adcs x15, x15, x11 + umulh x11, x4, x7 + adcs x16, x16, x11 + umulh x11, x4, x8 + adcs x17, x17, x11 + umulh x11, x4, x9 + adcs x19, x19, x11 + umulh x11, x4, x10 + adc x20, x20, x11 + ldp x3, x4, [sp, #256] + mul x11, x3, x5 + adds x14, x14, x11 + mul x11, x3, x6 + adcs x15, x15, x11 + mul x11, x3, x7 + adcs x16, x16, x11 + mul x11, x3, x8 + adcs x17, x17, x11 + mul x11, x3, x9 + adcs x19, x19, x11 + mul x11, x3, x10 + adcs x20, x20, x11 + cset x21, hs + umulh x11, x3, x5 + adds x15, x15, x11 + umulh x11, x3, x6 + adcs x16, x16, x11 + umulh x11, x3, x7 + adcs x17, x17, x11 + umulh x11, x3, x8 + adcs x19, x19, x11 + umulh x11, x3, x9 + adcs x20, x20, x11 + umulh x11, x3, x10 + adc x21, x21, x11 + mul x11, x4, x5 + adds x15, x15, x11 + mul x11, x4, x6 + adcs x16, x16, x11 + mul x11, x4, x7 + adcs x17, x17, x11 + mul x11, x4, x8 + adcs x19, x19, x11 + mul x11, x4, x9 + adcs x20, x20, x11 + mul x11, x4, x10 + adcs x21, x21, x11 + cset x22, hs + umulh x11, x4, x5 + adds x16, x16, x11 + umulh x11, x4, x6 + adcs x17, x17, x11 + umulh x11, x4, x7 + adcs x19, x19, x11 + umulh x11, x4, x8 + adcs x20, x20, x11 + umulh x11, x4, x9 + adcs x21, x21, x11 + umulh x11, x4, x10 + adc x22, x22, x11 + ldp x3, x4, [sp, #272] + mul x11, x3, x5 + adds x16, x16, x11 + mul x11, x3, x6 + adcs x17, x17, x11 + mul x11, x3, x7 + adcs x19, x19, x11 + mul x11, x3, x8 + adcs x20, x20, x11 + mul x11, x3, x9 + adcs x21, x21, x11 + mul x11, x3, x10 + adcs x22, x22, x11 + cset x2, hs + umulh x11, x3, x5 + adds x17, x17, x11 + umulh x11, x3, x6 + adcs x19, x19, x11 + umulh x11, x3, x7 + adcs x20, x20, x11 + umulh x11, x3, x8 + adcs x21, x21, x11 + umulh x11, x3, x9 + adcs x22, x22, x11 + umulh x11, x3, x10 + adc x2, x2, x11 + mul x11, x4, x5 + adds x17, x17, x11 + mul x11, x4, x6 + adcs x19, x19, x11 + mul x11, x4, x7 + adcs x20, x20, x11 + mul x11, x4, x8 + adcs x21, x21, x11 + mul x11, x4, x9 + adcs x22, x22, x11 + mul x11, x4, x10 + adcs x2, x2, x11 + cset x1, hs + umulh x11, x4, x5 + adds x19, x19, x11 + umulh x11, x4, x6 + adcs x20, x20, x11 + umulh x11, x4, x7 + adcs x21, x21, x11 + umulh x11, x4, x8 + adcs x22, x22, x11 + umulh x11, x4, x9 + adcs x2, x2, x11 + umulh x11, x4, x10 + adc x1, x1, x11 + lsl x7, x12, #32 + add x12, x7, x12 + mov x7, #-4294967295 + umulh x7, x7, x12 + mov x6, #4294967295 + mul x5, x6, x12 + umulh x6, x6, x12 + adds x7, x7, x5 + adcs x6, x6, x12 + adc x5, xzr, xzr + subs x13, x13, x7 + sbcs x14, x14, x6 + sbcs x15, x15, x5 + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x12, x12, xzr + lsl x7, x13, #32 + add x13, x7, x13 + mov x7, #-4294967295 + umulh x7, x7, x13 + mov x6, #4294967295 + mul x5, x6, x13 + umulh x6, x6, x13 + adds x7, x7, x5 + adcs x6, x6, x13 + adc x5, xzr, xzr + subs x14, x14, x7 + sbcs x15, x15, x6 + sbcs x16, x16, x5 + sbcs x17, x17, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + lsl x7, x14, #32 + add x14, x7, x14 + mov x7, #-4294967295 + umulh x7, x7, x14 + mov x6, #4294967295 + mul x5, x6, x14 + umulh x6, x6, x14 + adds x7, x7, x5 + adcs x6, x6, x14 + adc x5, xzr, xzr + subs x15, x15, x7 + sbcs x16, x16, x6 + sbcs x17, x17, x5 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + lsl x7, x15, #32 + add x15, x7, x15 + mov x7, #-4294967295 + umulh x7, x7, x15 + mov x6, #4294967295 + mul x5, x6, x15 + umulh x6, x6, x15 + adds x7, x7, x5 + adcs x6, x6, x15 + adc x5, xzr, xzr + subs x16, x16, x7 + sbcs x17, x17, x6 + sbcs x12, x12, x5 + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbc x15, x15, xzr + lsl x7, x16, #32 + add x16, x7, x16 + mov x7, #-4294967295 + umulh x7, x7, x16 + mov x6, #4294967295 + mul x5, x6, x16 + umulh x6, x6, x16 + adds x7, x7, x5 + adcs x6, x6, x16 + adc x5, xzr, xzr + subs x17, x17, x7 + sbcs x12, x12, x6 + sbcs x13, x13, x5 + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbc x16, x16, xzr + lsl x7, x17, #32 + add x17, x7, x17 + mov x7, #-4294967295 + umulh x7, x7, x17 + mov x6, #4294967295 + mul x5, x6, x17 + umulh x6, x6, x17 + adds x7, x7, x5 + adcs x6, x6, x17 + adc x5, xzr, xzr + subs x12, x12, x7 + sbcs x13, x13, x6 + sbcs x14, x14, x5 + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbc x17, x17, xzr + adds x12, x12, x19 + adcs x13, x13, x20 + adcs x14, x14, x21 + adcs x15, x15, x22 + adcs x16, x16, x2 + adcs x17, x17, x1 + adc x10, xzr, xzr + mov x11, #-4294967295 + adds x19, x12, x11 + mov x11, #4294967295 + adcs x20, x13, x11 + mov x11, #1 + adcs x21, x14, x11 + adcs x22, x15, xzr + adcs x2, x16, xzr + adcs x1, x17, xzr + adcs x10, x10, xzr + csel x12, x12, x19, eq + csel x13, x13, x20, eq + csel x14, x14, x21, eq + csel x15, x15, x22, eq + csel x16, x16, x2, eq + csel x17, x17, x1, eq + stp x12, x13, [sp, #240] + stp x14, x15, [sp, #256] + stp x16, x17, [sp, #272] + ldp x5, x6, [sp] + ldp x4, x3, [sp, #96] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #16] + ldp x4, x3, [sp, #112] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #32] + ldp x4, x3, [sp, #128] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, lo + mov x4, #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [sp] + stp x7, x8, [sp, #16] + stp x9, x10, [sp, #32] + ldp x5, x6, [sp, #192] + ldp x4, x3, [sp] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #208] + ldp x4, x3, [sp, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #224] + ldp x4, x3, [sp, #32] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, lo + mov x4, #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [sp, #192] + stp x7, x8, [sp, #208] + stp x9, x10, [sp, #224] + ldp x3, x4, [sp, #144] + ldp x5, x6, [sp, #288] + mul x12, x3, x5 + umulh x13, x3, x5 + mul x11, x3, x6 + umulh x14, x3, x6 + adds x13, x13, x11 + ldp x7, x8, [sp, #304] + mul x11, x3, x7 + umulh x15, x3, x7 + adcs x14, x14, x11 + mul x11, x3, x8 + umulh x16, x3, x8 + adcs x15, x15, x11 + ldp x9, x10, [sp, #320] + mul x11, x3, x9 + umulh x17, x3, x9 + adcs x16, x16, x11 + mul x11, x3, x10 + umulh x19, x3, x10 + adcs x17, x17, x11 + adc x19, x19, xzr + mul x11, x4, x5 + adds x13, x13, x11 + mul x11, x4, x6 + adcs x14, x14, x11 + mul x11, x4, x7 + adcs x15, x15, x11 + mul x11, x4, x8 + adcs x16, x16, x11 + mul x11, x4, x9 + adcs x17, x17, x11 + mul x11, x4, x10 + adcs x19, x19, x11 + cset x20, hs + umulh x11, x4, x5 + adds x14, x14, x11 + umulh x11, x4, x6 + adcs x15, x15, x11 + umulh x11, x4, x7 + adcs x16, x16, x11 + umulh x11, x4, x8 + adcs x17, x17, x11 + umulh x11, x4, x9 + adcs x19, x19, x11 + umulh x11, x4, x10 + adc x20, x20, x11 + ldp x3, x4, [sp, #160] + mul x11, x3, x5 + adds x14, x14, x11 + mul x11, x3, x6 + adcs x15, x15, x11 + mul x11, x3, x7 + adcs x16, x16, x11 + mul x11, x3, x8 + adcs x17, x17, x11 + mul x11, x3, x9 + adcs x19, x19, x11 + mul x11, x3, x10 + adcs x20, x20, x11 + cset x21, hs + umulh x11, x3, x5 + adds x15, x15, x11 + umulh x11, x3, x6 + adcs x16, x16, x11 + umulh x11, x3, x7 + adcs x17, x17, x11 + umulh x11, x3, x8 + adcs x19, x19, x11 + umulh x11, x3, x9 + adcs x20, x20, x11 + umulh x11, x3, x10 + adc x21, x21, x11 + mul x11, x4, x5 + adds x15, x15, x11 + mul x11, x4, x6 + adcs x16, x16, x11 + mul x11, x4, x7 + adcs x17, x17, x11 + mul x11, x4, x8 + adcs x19, x19, x11 + mul x11, x4, x9 + adcs x20, x20, x11 + mul x11, x4, x10 + adcs x21, x21, x11 + cset x22, hs + umulh x11, x4, x5 + adds x16, x16, x11 + umulh x11, x4, x6 + adcs x17, x17, x11 + umulh x11, x4, x7 + adcs x19, x19, x11 + umulh x11, x4, x8 + adcs x20, x20, x11 + umulh x11, x4, x9 + adcs x21, x21, x11 + umulh x11, x4, x10 + adc x22, x22, x11 + ldp x3, x4, [sp, #176] + mul x11, x3, x5 + adds x16, x16, x11 + mul x11, x3, x6 + adcs x17, x17, x11 + mul x11, x3, x7 + adcs x19, x19, x11 + mul x11, x3, x8 + adcs x20, x20, x11 + mul x11, x3, x9 + adcs x21, x21, x11 + mul x11, x3, x10 + adcs x22, x22, x11 + cset x2, hs + umulh x11, x3, x5 + adds x17, x17, x11 + umulh x11, x3, x6 + adcs x19, x19, x11 + umulh x11, x3, x7 + adcs x20, x20, x11 + umulh x11, x3, x8 + adcs x21, x21, x11 + umulh x11, x3, x9 + adcs x22, x22, x11 + umulh x11, x3, x10 + adc x2, x2, x11 + mul x11, x4, x5 + adds x17, x17, x11 + mul x11, x4, x6 + adcs x19, x19, x11 + mul x11, x4, x7 + adcs x20, x20, x11 + mul x11, x4, x8 + adcs x21, x21, x11 + mul x11, x4, x9 + adcs x22, x22, x11 + mul x11, x4, x10 + adcs x2, x2, x11 + cset x1, hs + umulh x11, x4, x5 + adds x19, x19, x11 + umulh x11, x4, x6 + adcs x20, x20, x11 + umulh x11, x4, x7 + adcs x21, x21, x11 + umulh x11, x4, x8 + adcs x22, x22, x11 + umulh x11, x4, x9 + adcs x2, x2, x11 + umulh x11, x4, x10 + adc x1, x1, x11 + lsl x7, x12, #32 + add x12, x7, x12 + mov x7, #-4294967295 + umulh x7, x7, x12 + mov x6, #4294967295 + mul x5, x6, x12 + umulh x6, x6, x12 + adds x7, x7, x5 + adcs x6, x6, x12 + adc x5, xzr, xzr + subs x13, x13, x7 + sbcs x14, x14, x6 + sbcs x15, x15, x5 + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x12, x12, xzr + lsl x7, x13, #32 + add x13, x7, x13 + mov x7, #-4294967295 + umulh x7, x7, x13 + mov x6, #4294967295 + mul x5, x6, x13 + umulh x6, x6, x13 + adds x7, x7, x5 + adcs x6, x6, x13 + adc x5, xzr, xzr + subs x14, x14, x7 + sbcs x15, x15, x6 + sbcs x16, x16, x5 + sbcs x17, x17, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + lsl x7, x14, #32 + add x14, x7, x14 + mov x7, #-4294967295 + umulh x7, x7, x14 + mov x6, #4294967295 + mul x5, x6, x14 + umulh x6, x6, x14 + adds x7, x7, x5 + adcs x6, x6, x14 + adc x5, xzr, xzr + subs x15, x15, x7 + sbcs x16, x16, x6 + sbcs x17, x17, x5 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + lsl x7, x15, #32 + add x15, x7, x15 + mov x7, #-4294967295 + umulh x7, x7, x15 + mov x6, #4294967295 + mul x5, x6, x15 + umulh x6, x6, x15 + adds x7, x7, x5 + adcs x6, x6, x15 + adc x5, xzr, xzr + subs x16, x16, x7 + sbcs x17, x17, x6 + sbcs x12, x12, x5 + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbc x15, x15, xzr + lsl x7, x16, #32 + add x16, x7, x16 + mov x7, #-4294967295 + umulh x7, x7, x16 + mov x6, #4294967295 + mul x5, x6, x16 + umulh x6, x6, x16 + adds x7, x7, x5 + adcs x6, x6, x16 + adc x5, xzr, xzr + subs x17, x17, x7 + sbcs x12, x12, x6 + sbcs x13, x13, x5 + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbc x16, x16, xzr + lsl x7, x17, #32 + add x17, x7, x17 + mov x7, #-4294967295 + umulh x7, x7, x17 + mov x6, #4294967295 + mul x5, x6, x17 + umulh x6, x6, x17 + adds x7, x7, x5 + adcs x6, x6, x17 + adc x5, xzr, xzr + subs x12, x12, x7 + sbcs x13, x13, x6 + sbcs x14, x14, x5 + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbc x17, x17, xzr + adds x12, x12, x19 + adcs x13, x13, x20 + adcs x14, x14, x21 + adcs x15, x15, x22 + adcs x16, x16, x2 + adcs x17, x17, x1 + adc x10, xzr, xzr + mov x11, #-4294967295 + adds x19, x12, x11 + mov x11, #4294967295 + adcs x20, x13, x11 + mov x11, #1 + adcs x21, x14, x11 + adcs x22, x15, xzr + adcs x2, x16, xzr + adcs x1, x17, xzr + adcs x10, x10, xzr + csel x12, x12, x19, eq + csel x13, x13, x20, eq + csel x14, x14, x21, eq + csel x15, x15, x22, eq + csel x16, x16, x2, eq + csel x17, x17, x1, eq + stp x12, x13, [sp, #144] + stp x14, x15, [sp, #160] + stp x16, x17, [sp, #176] + ldp x3, x4, [sp, #240] + ldp x5, x6, [x26, #96] + mul x12, x3, x5 + umulh x13, x3, x5 + mul x11, x3, x6 + umulh x14, x3, x6 + adds x13, x13, x11 + ldp x7, x8, [x26, #112] + mul x11, x3, x7 + umulh x15, x3, x7 + adcs x14, x14, x11 + mul x11, x3, x8 + umulh x16, x3, x8 + adcs x15, x15, x11 + ldp x9, x10, [x26, #128] + mul x11, x3, x9 + umulh x17, x3, x9 + adcs x16, x16, x11 + mul x11, x3, x10 + umulh x19, x3, x10 + adcs x17, x17, x11 + adc x19, x19, xzr + mul x11, x4, x5 + adds x13, x13, x11 + mul x11, x4, x6 + adcs x14, x14, x11 + mul x11, x4, x7 + adcs x15, x15, x11 + mul x11, x4, x8 + adcs x16, x16, x11 + mul x11, x4, x9 + adcs x17, x17, x11 + mul x11, x4, x10 + adcs x19, x19, x11 + cset x20, hs + umulh x11, x4, x5 + adds x14, x14, x11 + umulh x11, x4, x6 + adcs x15, x15, x11 + umulh x11, x4, x7 + adcs x16, x16, x11 + umulh x11, x4, x8 + adcs x17, x17, x11 + umulh x11, x4, x9 + adcs x19, x19, x11 + umulh x11, x4, x10 + adc x20, x20, x11 + ldp x3, x4, [sp, #256] + mul x11, x3, x5 + adds x14, x14, x11 + mul x11, x3, x6 + adcs x15, x15, x11 + mul x11, x3, x7 + adcs x16, x16, x11 + mul x11, x3, x8 + adcs x17, x17, x11 + mul x11, x3, x9 + adcs x19, x19, x11 + mul x11, x3, x10 + adcs x20, x20, x11 + cset x21, hs + umulh x11, x3, x5 + adds x15, x15, x11 + umulh x11, x3, x6 + adcs x16, x16, x11 + umulh x11, x3, x7 + adcs x17, x17, x11 + umulh x11, x3, x8 + adcs x19, x19, x11 + umulh x11, x3, x9 + adcs x20, x20, x11 + umulh x11, x3, x10 + adc x21, x21, x11 + mul x11, x4, x5 + adds x15, x15, x11 + mul x11, x4, x6 + adcs x16, x16, x11 + mul x11, x4, x7 + adcs x17, x17, x11 + mul x11, x4, x8 + adcs x19, x19, x11 + mul x11, x4, x9 + adcs x20, x20, x11 + mul x11, x4, x10 + adcs x21, x21, x11 + cset x22, hs + umulh x11, x4, x5 + adds x16, x16, x11 + umulh x11, x4, x6 + adcs x17, x17, x11 + umulh x11, x4, x7 + adcs x19, x19, x11 + umulh x11, x4, x8 + adcs x20, x20, x11 + umulh x11, x4, x9 + adcs x21, x21, x11 + umulh x11, x4, x10 + adc x22, x22, x11 + ldp x3, x4, [sp, #272] + mul x11, x3, x5 + adds x16, x16, x11 + mul x11, x3, x6 + adcs x17, x17, x11 + mul x11, x3, x7 + adcs x19, x19, x11 + mul x11, x3, x8 + adcs x20, x20, x11 + mul x11, x3, x9 + adcs x21, x21, x11 + mul x11, x3, x10 + adcs x22, x22, x11 + cset x2, hs + umulh x11, x3, x5 + adds x17, x17, x11 + umulh x11, x3, x6 + adcs x19, x19, x11 + umulh x11, x3, x7 + adcs x20, x20, x11 + umulh x11, x3, x8 + adcs x21, x21, x11 + umulh x11, x3, x9 + adcs x22, x22, x11 + umulh x11, x3, x10 + adc x2, x2, x11 + mul x11, x4, x5 + adds x17, x17, x11 + mul x11, x4, x6 + adcs x19, x19, x11 + mul x11, x4, x7 + adcs x20, x20, x11 + mul x11, x4, x8 + adcs x21, x21, x11 + mul x11, x4, x9 + adcs x22, x22, x11 + mul x11, x4, x10 + adcs x2, x2, x11 + cset x1, hs + umulh x11, x4, x5 + adds x19, x19, x11 + umulh x11, x4, x6 + adcs x20, x20, x11 + umulh x11, x4, x7 + adcs x21, x21, x11 + umulh x11, x4, x8 + adcs x22, x22, x11 + umulh x11, x4, x9 + adcs x2, x2, x11 + umulh x11, x4, x10 + adc x1, x1, x11 + lsl x7, x12, #32 + add x12, x7, x12 + mov x7, #-4294967295 + umulh x7, x7, x12 + mov x6, #4294967295 + mul x5, x6, x12 + umulh x6, x6, x12 + adds x7, x7, x5 + adcs x6, x6, x12 + adc x5, xzr, xzr + subs x13, x13, x7 + sbcs x14, x14, x6 + sbcs x15, x15, x5 + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x12, x12, xzr + lsl x7, x13, #32 + add x13, x7, x13 + mov x7, #-4294967295 + umulh x7, x7, x13 + mov x6, #4294967295 + mul x5, x6, x13 + umulh x6, x6, x13 + adds x7, x7, x5 + adcs x6, x6, x13 + adc x5, xzr, xzr + subs x14, x14, x7 + sbcs x15, x15, x6 + sbcs x16, x16, x5 + sbcs x17, x17, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + lsl x7, x14, #32 + add x14, x7, x14 + mov x7, #-4294967295 + umulh x7, x7, x14 + mov x6, #4294967295 + mul x5, x6, x14 + umulh x6, x6, x14 + adds x7, x7, x5 + adcs x6, x6, x14 + adc x5, xzr, xzr + subs x15, x15, x7 + sbcs x16, x16, x6 + sbcs x17, x17, x5 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + lsl x7, x15, #32 + add x15, x7, x15 + mov x7, #-4294967295 + umulh x7, x7, x15 + mov x6, #4294967295 + mul x5, x6, x15 + umulh x6, x6, x15 + adds x7, x7, x5 + adcs x6, x6, x15 + adc x5, xzr, xzr + subs x16, x16, x7 + sbcs x17, x17, x6 + sbcs x12, x12, x5 + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbc x15, x15, xzr + lsl x7, x16, #32 + add x16, x7, x16 + mov x7, #-4294967295 + umulh x7, x7, x16 + mov x6, #4294967295 + mul x5, x6, x16 + umulh x6, x6, x16 + adds x7, x7, x5 + adcs x6, x6, x16 + adc x5, xzr, xzr + subs x17, x17, x7 + sbcs x12, x12, x6 + sbcs x13, x13, x5 + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbc x16, x16, xzr + lsl x7, x17, #32 + add x17, x7, x17 + mov x7, #-4294967295 + umulh x7, x7, x17 + mov x6, #4294967295 + mul x5, x6, x17 + umulh x6, x6, x17 + adds x7, x7, x5 + adcs x6, x6, x17 + adc x5, xzr, xzr + subs x12, x12, x7 + sbcs x13, x13, x6 + sbcs x14, x14, x5 + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbc x17, x17, xzr + adds x12, x12, x19 + adcs x13, x13, x20 + adcs x14, x14, x21 + adcs x15, x15, x22 + adcs x16, x16, x2 + adcs x17, x17, x1 + adc x10, xzr, xzr + mov x11, #-4294967295 + adds x19, x12, x11 + mov x11, #4294967295 + adcs x20, x13, x11 + mov x11, #1 + adcs x21, x14, x11 + adcs x22, x15, xzr + adcs x2, x16, xzr + adcs x1, x17, xzr + adcs x10, x10, xzr + csel x12, x12, x19, eq + csel x13, x13, x20, eq + csel x14, x14, x21, eq + csel x15, x15, x22, eq + csel x16, x16, x2, eq + csel x17, x17, x1, eq + stp x12, x13, [sp, #240] + stp x14, x15, [sp, #256] + stp x16, x17, [sp, #272] + ldp x3, x4, [sp, #48] + ldp x5, x6, [sp, #192] + mul x12, x3, x5 + umulh x13, x3, x5 + mul x11, x3, x6 + umulh x14, x3, x6 + adds x13, x13, x11 + ldp x7, x8, [sp, #208] + mul x11, x3, x7 + umulh x15, x3, x7 + adcs x14, x14, x11 + mul x11, x3, x8 + umulh x16, x3, x8 + adcs x15, x15, x11 + ldp x9, x10, [sp, #224] + mul x11, x3, x9 + umulh x17, x3, x9 + adcs x16, x16, x11 + mul x11, x3, x10 + umulh x19, x3, x10 + adcs x17, x17, x11 + adc x19, x19, xzr + mul x11, x4, x5 + adds x13, x13, x11 + mul x11, x4, x6 + adcs x14, x14, x11 + mul x11, x4, x7 + adcs x15, x15, x11 + mul x11, x4, x8 + adcs x16, x16, x11 + mul x11, x4, x9 + adcs x17, x17, x11 + mul x11, x4, x10 + adcs x19, x19, x11 + cset x20, hs + umulh x11, x4, x5 + adds x14, x14, x11 + umulh x11, x4, x6 + adcs x15, x15, x11 + umulh x11, x4, x7 + adcs x16, x16, x11 + umulh x11, x4, x8 + adcs x17, x17, x11 + umulh x11, x4, x9 + adcs x19, x19, x11 + umulh x11, x4, x10 + adc x20, x20, x11 + ldp x3, x4, [sp, #64] + mul x11, x3, x5 + adds x14, x14, x11 + mul x11, x3, x6 + adcs x15, x15, x11 + mul x11, x3, x7 + adcs x16, x16, x11 + mul x11, x3, x8 + adcs x17, x17, x11 + mul x11, x3, x9 + adcs x19, x19, x11 + mul x11, x3, x10 + adcs x20, x20, x11 + cset x21, hs + umulh x11, x3, x5 + adds x15, x15, x11 + umulh x11, x3, x6 + adcs x16, x16, x11 + umulh x11, x3, x7 + adcs x17, x17, x11 + umulh x11, x3, x8 + adcs x19, x19, x11 + umulh x11, x3, x9 + adcs x20, x20, x11 + umulh x11, x3, x10 + adc x21, x21, x11 + mul x11, x4, x5 + adds x15, x15, x11 + mul x11, x4, x6 + adcs x16, x16, x11 + mul x11, x4, x7 + adcs x17, x17, x11 + mul x11, x4, x8 + adcs x19, x19, x11 + mul x11, x4, x9 + adcs x20, x20, x11 + mul x11, x4, x10 + adcs x21, x21, x11 + cset x22, hs + umulh x11, x4, x5 + adds x16, x16, x11 + umulh x11, x4, x6 + adcs x17, x17, x11 + umulh x11, x4, x7 + adcs x19, x19, x11 + umulh x11, x4, x8 + adcs x20, x20, x11 + umulh x11, x4, x9 + adcs x21, x21, x11 + umulh x11, x4, x10 + adc x22, x22, x11 + ldp x3, x4, [sp, #80] + mul x11, x3, x5 + adds x16, x16, x11 + mul x11, x3, x6 + adcs x17, x17, x11 + mul x11, x3, x7 + adcs x19, x19, x11 + mul x11, x3, x8 + adcs x20, x20, x11 + mul x11, x3, x9 + adcs x21, x21, x11 + mul x11, x3, x10 + adcs x22, x22, x11 + cset x2, hs + umulh x11, x3, x5 + adds x17, x17, x11 + umulh x11, x3, x6 + adcs x19, x19, x11 + umulh x11, x3, x7 + adcs x20, x20, x11 + umulh x11, x3, x8 + adcs x21, x21, x11 + umulh x11, x3, x9 + adcs x22, x22, x11 + umulh x11, x3, x10 + adc x2, x2, x11 + mul x11, x4, x5 + adds x17, x17, x11 + mul x11, x4, x6 + adcs x19, x19, x11 + mul x11, x4, x7 + adcs x20, x20, x11 + mul x11, x4, x8 + adcs x21, x21, x11 + mul x11, x4, x9 + adcs x22, x22, x11 + mul x11, x4, x10 + adcs x2, x2, x11 + cset x1, hs + umulh x11, x4, x5 + adds x19, x19, x11 + umulh x11, x4, x6 + adcs x20, x20, x11 + umulh x11, x4, x7 + adcs x21, x21, x11 + umulh x11, x4, x8 + adcs x22, x22, x11 + umulh x11, x4, x9 + adcs x2, x2, x11 + umulh x11, x4, x10 + adc x1, x1, x11 + lsl x7, x12, #32 + add x12, x7, x12 + mov x7, #-4294967295 + umulh x7, x7, x12 + mov x6, #4294967295 + mul x5, x6, x12 + umulh x6, x6, x12 + adds x7, x7, x5 + adcs x6, x6, x12 + adc x5, xzr, xzr + subs x13, x13, x7 + sbcs x14, x14, x6 + sbcs x15, x15, x5 + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x12, x12, xzr + lsl x7, x13, #32 + add x13, x7, x13 + mov x7, #-4294967295 + umulh x7, x7, x13 + mov x6, #4294967295 + mul x5, x6, x13 + umulh x6, x6, x13 + adds x7, x7, x5 + adcs x6, x6, x13 + adc x5, xzr, xzr + subs x14, x14, x7 + sbcs x15, x15, x6 + sbcs x16, x16, x5 + sbcs x17, x17, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + lsl x7, x14, #32 + add x14, x7, x14 + mov x7, #-4294967295 + umulh x7, x7, x14 + mov x6, #4294967295 + mul x5, x6, x14 + umulh x6, x6, x14 + adds x7, x7, x5 + adcs x6, x6, x14 + adc x5, xzr, xzr + subs x15, x15, x7 + sbcs x16, x16, x6 + sbcs x17, x17, x5 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + lsl x7, x15, #32 + add x15, x7, x15 + mov x7, #-4294967295 + umulh x7, x7, x15 + mov x6, #4294967295 + mul x5, x6, x15 + umulh x6, x6, x15 + adds x7, x7, x5 + adcs x6, x6, x15 + adc x5, xzr, xzr + subs x16, x16, x7 + sbcs x17, x17, x6 + sbcs x12, x12, x5 + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbc x15, x15, xzr + lsl x7, x16, #32 + add x16, x7, x16 + mov x7, #-4294967295 + umulh x7, x7, x16 + mov x6, #4294967295 + mul x5, x6, x16 + umulh x6, x6, x16 + adds x7, x7, x5 + adcs x6, x6, x16 + adc x5, xzr, xzr + subs x17, x17, x7 + sbcs x12, x12, x6 + sbcs x13, x13, x5 + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbc x16, x16, xzr + lsl x7, x17, #32 + add x17, x7, x17 + mov x7, #-4294967295 + umulh x7, x7, x17 + mov x6, #4294967295 + mul x5, x6, x17 + umulh x6, x6, x17 + adds x7, x7, x5 + adcs x6, x6, x17 + adc x5, xzr, xzr + subs x12, x12, x7 + sbcs x13, x13, x6 + sbcs x14, x14, x5 + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbc x17, x17, xzr + adds x12, x12, x19 + adcs x13, x13, x20 + adcs x14, x14, x21 + adcs x15, x15, x22 + adcs x16, x16, x2 + adcs x17, x17, x1 + adc x10, xzr, xzr + mov x11, #-4294967295 + adds x19, x12, x11 + mov x11, #4294967295 + adcs x20, x13, x11 + mov x11, #1 + adcs x21, x14, x11 + adcs x22, x15, xzr + adcs x2, x16, xzr + adcs x1, x17, xzr + adcs x10, x10, xzr + csel x12, x12, x19, eq + csel x13, x13, x20, eq + csel x14, x14, x21, eq + csel x15, x15, x22, eq + csel x16, x16, x2, eq + csel x17, x17, x1, eq + stp x12, x13, [sp, #192] + stp x14, x15, [sp, #208] + stp x16, x17, [sp, #224] + ldp x5, x6, [sp, #192] + ldp x4, x3, [sp, #144] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #208] + ldp x4, x3, [sp, #160] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #224] + ldp x4, x3, [sp, #176] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, lo + mov x4, #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [sp, #192] + stp x7, x8, [sp, #208] + stp x9, x10, [sp, #224] + ldp x0, x1, [x25, #96] + ldp x2, x3, [x25, #112] + ldp x4, x5, [x25, #128] + orr x20, x0, x1 + orr x21, x2, x3 + orr x22, x4, x5 + orr x20, x20, x21 + orr x20, x20, x22 + cmp x20, xzr + cset x20, ne + ldp x6, x7, [x26, #96] + ldp x8, x9, [x26, #112] + ldp x10, x11, [x26, #128] + orr x21, x6, x7 + orr x22, x8, x9 + orr x23, x10, x11 + orr x21, x21, x22 + orr x21, x21, x23 + cmp x21, xzr + cset x21, ne + cmp x21, x20 + ldp x12, x13, [sp, #240] + csel x12, x0, x12, lo + csel x13, x1, x13, lo + csel x12, x6, x12, hi + csel x13, x7, x13, hi + ldp x14, x15, [sp, #256] + csel x14, x2, x14, lo + csel x15, x3, x15, lo + csel x14, x8, x14, hi + csel x15, x9, x15, hi + ldp x16, x17, [sp, #272] + csel x16, x4, x16, lo + csel x17, x5, x17, lo + csel x16, x10, x16, hi + csel x17, x11, x17, hi + ldp x20, x21, [x25] + ldp x0, x1, [sp] + csel x0, x20, x0, lo + csel x1, x21, x1, lo + ldp x20, x21, [x26] + csel x0, x20, x0, hi + csel x1, x21, x1, hi + ldp x20, x21, [x25, #16] + ldp x2, x3, [sp, #16] + csel x2, x20, x2, lo + csel x3, x21, x3, lo + ldp x20, x21, [x26, #16] + csel x2, x20, x2, hi + csel x3, x21, x3, hi + ldp x20, x21, [x25, #32] + ldp x4, x5, [sp, #32] + csel x4, x20, x4, lo + csel x5, x21, x5, lo + ldp x20, x21, [x26, #32] + csel x4, x20, x4, hi + csel x5, x21, x5, hi + ldp x20, x21, [x25, #48] + ldp x6, x7, [sp, #192] + csel x6, x20, x6, lo + csel x7, x21, x7, lo + ldp x20, x21, [x26, #48] + csel x6, x20, x6, hi + csel x7, x21, x7, hi + ldp x20, x21, [x25, #64] + ldp x8, x9, [sp, #208] + csel x8, x20, x8, lo + csel x9, x21, x9, lo + ldp x20, x21, [x26, #64] + csel x8, x20, x8, hi + csel x9, x21, x9, hi + ldp x20, x21, [x25, #80] + ldp x10, x11, [sp, #224] + csel x10, x20, x10, lo + csel x11, x21, x11, lo + ldp x20, x21, [x26, #80] + csel x10, x20, x10, hi + csel x11, x21, x11, hi + stp x0, x1, [x24] + stp x2, x3, [x24, #16] + stp x4, x5, [x24, #32] + stp x6, x7, [x24, #48] + stp x8, x9, [x24, #64] + stp x10, x11, [x24, #80] + stp x12, x13, [x24, #96] + stp x14, x15, [x24, #112] + stp x16, x17, [x24, #128] + CFI_INC_SP(336) + CFI_POP2(x25,x26) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp384_montjscalarmul_alt_p384_montjadd) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp384_montjscalarmul_alt_p384_montjdouble) + +Lp384_montjscalarmul_alt_p384_montjdouble: + CFI_START + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_DEC_SP(336) + mov x23, x0 + mov x24, x1 + ldp x2, x3, [x24, #96] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [x24, #112] + mul x8, x2, x4 + adds x10, x10, x8 + mul x11, x2, x5 + mul x8, x3, x4 + adcs x11, x11, x8 + umulh x12, x2, x5 + mul x8, x3, x5 + adcs x12, x12, x8 + ldp x6, x7, [x24, #128] + mul x13, x2, x7 + mul x8, x3, x6 + adcs x13, x13, x8 + umulh x14, x2, x7 + mul x8, x3, x7 + adcs x14, x14, x8 + mul x15, x5, x6 + adcs x15, x15, xzr + umulh x16, x5, x6 + adc x16, x16, xzr + umulh x8, x2, x4 + adds x11, x11, x8 + umulh x8, x3, x4 + adcs x12, x12, x8 + umulh x8, x3, x5 + adcs x13, x13, x8 + umulh x8, x3, x6 + adcs x14, x14, x8 + umulh x8, x3, x7 + adcs x15, x15, x8 + adc x16, x16, xzr + mul x8, x2, x6 + adds x12, x12, x8 + mul x8, x4, x5 + adcs x13, x13, x8 + mul x8, x4, x6 + adcs x14, x14, x8 + mul x8, x4, x7 + adcs x15, x15, x8 + mul x8, x5, x7 + adcs x16, x16, x8 + mul x17, x6, x7 + adcs x17, x17, xzr + umulh x19, x6, x7 + adc x19, x19, xzr + umulh x8, x2, x6 + adds x13, x13, x8 + umulh x8, x4, x5 + adcs x14, x14, x8 + umulh x8, x4, x6 + adcs x15, x15, x8 + umulh x8, x4, x7 + adcs x16, x16, x8 + umulh x8, x5, x7 + adcs x17, x17, x8 + adc x19, x19, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + adcs x15, x15, x15 + adcs x16, x16, x16 + adcs x17, x17, x17 + adcs x19, x19, x19 + cset x20, hs + umulh x8, x2, x2 + mul x2, x2, x2 + adds x9, x9, x8 + mul x8, x3, x3 + adcs x10, x10, x8 + umulh x8, x3, x3 + adcs x11, x11, x8 + mul x8, x4, x4 + adcs x12, x12, x8 + umulh x8, x4, x4 + adcs x13, x13, x8 + mul x8, x5, x5 + adcs x14, x14, x8 + umulh x8, x5, x5 + adcs x15, x15, x8 + mul x8, x6, x6 + adcs x16, x16, x8 + umulh x8, x6, x6 + adcs x17, x17, x8 + mul x8, x7, x7 + adcs x19, x19, x8 + umulh x8, x7, x7 + adc x20, x20, x8 + lsl x5, x2, #32 + add x2, x5, x2 + mov x5, #-4294967295 + umulh x5, x5, x2 + mov x4, #4294967295 + mul x3, x4, x2 + umulh x4, x4, x2 + adds x5, x5, x3 + adcs x4, x4, x2 + adc x3, xzr, xzr + subs x9, x9, x5 + sbcs x10, x10, x4 + sbcs x11, x11, x3 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x2, x2, xzr + lsl x5, x9, #32 + add x9, x5, x9 + mov x5, #-4294967295 + umulh x5, x5, x9 + mov x4, #4294967295 + mul x3, x4, x9 + umulh x4, x4, x9 + adds x5, x5, x3 + adcs x4, x4, x9 + adc x3, xzr, xzr + subs x10, x10, x5 + sbcs x11, x11, x4 + sbcs x12, x12, x3 + sbcs x13, x13, xzr + sbcs x2, x2, xzr + sbc x9, x9, xzr + lsl x5, x10, #32 + add x10, x5, x10 + mov x5, #-4294967295 + umulh x5, x5, x10 + mov x4, #4294967295 + mul x3, x4, x10 + umulh x4, x4, x10 + adds x5, x5, x3 + adcs x4, x4, x10 + adc x3, xzr, xzr + subs x11, x11, x5 + sbcs x12, x12, x4 + sbcs x13, x13, x3 + sbcs x2, x2, xzr + sbcs x9, x9, xzr + sbc x10, x10, xzr + lsl x5, x11, #32 + add x11, x5, x11 + mov x5, #-4294967295 + umulh x5, x5, x11 + mov x4, #4294967295 + mul x3, x4, x11 + umulh x4, x4, x11 + adds x5, x5, x3 + adcs x4, x4, x11 + adc x3, xzr, xzr + subs x12, x12, x5 + sbcs x13, x13, x4 + sbcs x2, x2, x3 + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbc x11, x11, xzr + lsl x5, x12, #32 + add x12, x5, x12 + mov x5, #-4294967295 + umulh x5, x5, x12 + mov x4, #4294967295 + mul x3, x4, x12 + umulh x4, x4, x12 + adds x5, x5, x3 + adcs x4, x4, x12 + adc x3, xzr, xzr + subs x13, x13, x5 + sbcs x2, x2, x4 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbc x12, x12, xzr + lsl x5, x13, #32 + add x13, x5, x13 + mov x5, #-4294967295 + umulh x5, x5, x13 + mov x4, #4294967295 + mul x3, x4, x13 + umulh x4, x4, x13 + adds x5, x5, x3 + adcs x4, x4, x13 + adc x3, xzr, xzr + subs x2, x2, x5 + sbcs x9, x9, x4 + sbcs x10, x10, x3 + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + adds x2, x2, x14 + adcs x9, x9, x15 + adcs x10, x10, x16 + adcs x11, x11, x17 + adcs x12, x12, x19 + adcs x13, x13, x20 + adc x6, xzr, xzr + mov x8, #-4294967295 + adds x14, x2, x8 + mov x8, #4294967295 + adcs x15, x9, x8 + mov x8, #1 + adcs x16, x10, x8 + adcs x17, x11, xzr + adcs x19, x12, xzr + adcs x20, x13, xzr + adcs x6, x6, xzr + csel x2, x2, x14, eq + csel x9, x9, x15, eq + csel x10, x10, x16, eq + csel x11, x11, x17, eq + csel x12, x12, x19, eq + csel x13, x13, x20, eq + stp x2, x9, [sp] + stp x10, x11, [sp, #16] + stp x12, x13, [sp, #32] + ldp x2, x3, [x24, #48] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [x24, #64] + mul x8, x2, x4 + adds x10, x10, x8 + mul x11, x2, x5 + mul x8, x3, x4 + adcs x11, x11, x8 + umulh x12, x2, x5 + mul x8, x3, x5 + adcs x12, x12, x8 + ldp x6, x7, [x24, #80] + mul x13, x2, x7 + mul x8, x3, x6 + adcs x13, x13, x8 + umulh x14, x2, x7 + mul x8, x3, x7 + adcs x14, x14, x8 + mul x15, x5, x6 + adcs x15, x15, xzr + umulh x16, x5, x6 + adc x16, x16, xzr + umulh x8, x2, x4 + adds x11, x11, x8 + umulh x8, x3, x4 + adcs x12, x12, x8 + umulh x8, x3, x5 + adcs x13, x13, x8 + umulh x8, x3, x6 + adcs x14, x14, x8 + umulh x8, x3, x7 + adcs x15, x15, x8 + adc x16, x16, xzr + mul x8, x2, x6 + adds x12, x12, x8 + mul x8, x4, x5 + adcs x13, x13, x8 + mul x8, x4, x6 + adcs x14, x14, x8 + mul x8, x4, x7 + adcs x15, x15, x8 + mul x8, x5, x7 + adcs x16, x16, x8 + mul x17, x6, x7 + adcs x17, x17, xzr + umulh x19, x6, x7 + adc x19, x19, xzr + umulh x8, x2, x6 + adds x13, x13, x8 + umulh x8, x4, x5 + adcs x14, x14, x8 + umulh x8, x4, x6 + adcs x15, x15, x8 + umulh x8, x4, x7 + adcs x16, x16, x8 + umulh x8, x5, x7 + adcs x17, x17, x8 + adc x19, x19, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + adcs x15, x15, x15 + adcs x16, x16, x16 + adcs x17, x17, x17 + adcs x19, x19, x19 + cset x20, hs + umulh x8, x2, x2 + mul x2, x2, x2 + adds x9, x9, x8 + mul x8, x3, x3 + adcs x10, x10, x8 + umulh x8, x3, x3 + adcs x11, x11, x8 + mul x8, x4, x4 + adcs x12, x12, x8 + umulh x8, x4, x4 + adcs x13, x13, x8 + mul x8, x5, x5 + adcs x14, x14, x8 + umulh x8, x5, x5 + adcs x15, x15, x8 + mul x8, x6, x6 + adcs x16, x16, x8 + umulh x8, x6, x6 + adcs x17, x17, x8 + mul x8, x7, x7 + adcs x19, x19, x8 + umulh x8, x7, x7 + adc x20, x20, x8 + lsl x5, x2, #32 + add x2, x5, x2 + mov x5, #-4294967295 + umulh x5, x5, x2 + mov x4, #4294967295 + mul x3, x4, x2 + umulh x4, x4, x2 + adds x5, x5, x3 + adcs x4, x4, x2 + adc x3, xzr, xzr + subs x9, x9, x5 + sbcs x10, x10, x4 + sbcs x11, x11, x3 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x2, x2, xzr + lsl x5, x9, #32 + add x9, x5, x9 + mov x5, #-4294967295 + umulh x5, x5, x9 + mov x4, #4294967295 + mul x3, x4, x9 + umulh x4, x4, x9 + adds x5, x5, x3 + adcs x4, x4, x9 + adc x3, xzr, xzr + subs x10, x10, x5 + sbcs x11, x11, x4 + sbcs x12, x12, x3 + sbcs x13, x13, xzr + sbcs x2, x2, xzr + sbc x9, x9, xzr + lsl x5, x10, #32 + add x10, x5, x10 + mov x5, #-4294967295 + umulh x5, x5, x10 + mov x4, #4294967295 + mul x3, x4, x10 + umulh x4, x4, x10 + adds x5, x5, x3 + adcs x4, x4, x10 + adc x3, xzr, xzr + subs x11, x11, x5 + sbcs x12, x12, x4 + sbcs x13, x13, x3 + sbcs x2, x2, xzr + sbcs x9, x9, xzr + sbc x10, x10, xzr + lsl x5, x11, #32 + add x11, x5, x11 + mov x5, #-4294967295 + umulh x5, x5, x11 + mov x4, #4294967295 + mul x3, x4, x11 + umulh x4, x4, x11 + adds x5, x5, x3 + adcs x4, x4, x11 + adc x3, xzr, xzr + subs x12, x12, x5 + sbcs x13, x13, x4 + sbcs x2, x2, x3 + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbc x11, x11, xzr + lsl x5, x12, #32 + add x12, x5, x12 + mov x5, #-4294967295 + umulh x5, x5, x12 + mov x4, #4294967295 + mul x3, x4, x12 + umulh x4, x4, x12 + adds x5, x5, x3 + adcs x4, x4, x12 + adc x3, xzr, xzr + subs x13, x13, x5 + sbcs x2, x2, x4 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbc x12, x12, xzr + lsl x5, x13, #32 + add x13, x5, x13 + mov x5, #-4294967295 + umulh x5, x5, x13 + mov x4, #4294967295 + mul x3, x4, x13 + umulh x4, x4, x13 + adds x5, x5, x3 + adcs x4, x4, x13 + adc x3, xzr, xzr + subs x2, x2, x5 + sbcs x9, x9, x4 + sbcs x10, x10, x3 + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + adds x2, x2, x14 + adcs x9, x9, x15 + adcs x10, x10, x16 + adcs x11, x11, x17 + adcs x12, x12, x19 + adcs x13, x13, x20 + adc x6, xzr, xzr + mov x8, #-4294967295 + adds x14, x2, x8 + mov x8, #4294967295 + adcs x15, x9, x8 + mov x8, #1 + adcs x16, x10, x8 + adcs x17, x11, xzr + adcs x19, x12, xzr + adcs x20, x13, xzr + adcs x6, x6, xzr + csel x2, x2, x14, eq + csel x9, x9, x15, eq + csel x10, x10, x16, eq + csel x11, x11, x17, eq + csel x12, x12, x19, eq + csel x13, x13, x20, eq + stp x2, x9, [sp, #48] + stp x10, x11, [sp, #64] + stp x12, x13, [sp, #80] + ldp x5, x6, [x24] + ldp x4, x3, [sp] + adds x5, x5, x4 + adcs x6, x6, x3 + ldp x7, x8, [x24, #16] + ldp x4, x3, [sp, #16] + adcs x7, x7, x4 + adcs x8, x8, x3 + ldp x9, x10, [x24, #32] + ldp x4, x3, [sp, #32] + adcs x9, x9, x4 + adcs x10, x10, x3 + csetm x3, hs + mov x4, #4294967295 + and x4, x4, x3 + subs x5, x5, x4 + eor x4, x4, x3 + sbcs x6, x6, x4 + mov x4, #-2 + and x4, x4, x3 + sbcs x7, x7, x4 + sbcs x8, x8, x3 + sbcs x9, x9, x3 + sbc x10, x10, x3 + stp x5, x6, [sp, #240] + stp x7, x8, [sp, #256] + stp x9, x10, [sp, #272] + ldp x5, x6, [x24] + ldp x4, x3, [sp] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [x24, #16] + ldp x4, x3, [sp, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [x24, #32] + ldp x4, x3, [sp, #32] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, lo + mov x4, #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [sp, #192] + stp x7, x8, [sp, #208] + stp x9, x10, [sp, #224] + ldp x3, x4, [sp, #240] + ldp x5, x6, [sp, #192] + mul x12, x3, x5 + umulh x13, x3, x5 + mul x11, x3, x6 + umulh x14, x3, x6 + adds x13, x13, x11 + ldp x7, x8, [sp, #208] + mul x11, x3, x7 + umulh x15, x3, x7 + adcs x14, x14, x11 + mul x11, x3, x8 + umulh x16, x3, x8 + adcs x15, x15, x11 + ldp x9, x10, [sp, #224] + mul x11, x3, x9 + umulh x17, x3, x9 + adcs x16, x16, x11 + mul x11, x3, x10 + umulh x19, x3, x10 + adcs x17, x17, x11 + adc x19, x19, xzr + mul x11, x4, x5 + adds x13, x13, x11 + mul x11, x4, x6 + adcs x14, x14, x11 + mul x11, x4, x7 + adcs x15, x15, x11 + mul x11, x4, x8 + adcs x16, x16, x11 + mul x11, x4, x9 + adcs x17, x17, x11 + mul x11, x4, x10 + adcs x19, x19, x11 + cset x20, hs + umulh x11, x4, x5 + adds x14, x14, x11 + umulh x11, x4, x6 + adcs x15, x15, x11 + umulh x11, x4, x7 + adcs x16, x16, x11 + umulh x11, x4, x8 + adcs x17, x17, x11 + umulh x11, x4, x9 + adcs x19, x19, x11 + umulh x11, x4, x10 + adc x20, x20, x11 + ldp x3, x4, [sp, #256] + mul x11, x3, x5 + adds x14, x14, x11 + mul x11, x3, x6 + adcs x15, x15, x11 + mul x11, x3, x7 + adcs x16, x16, x11 + mul x11, x3, x8 + adcs x17, x17, x11 + mul x11, x3, x9 + adcs x19, x19, x11 + mul x11, x3, x10 + adcs x20, x20, x11 + cset x21, hs + umulh x11, x3, x5 + adds x15, x15, x11 + umulh x11, x3, x6 + adcs x16, x16, x11 + umulh x11, x3, x7 + adcs x17, x17, x11 + umulh x11, x3, x8 + adcs x19, x19, x11 + umulh x11, x3, x9 + adcs x20, x20, x11 + umulh x11, x3, x10 + adc x21, x21, x11 + mul x11, x4, x5 + adds x15, x15, x11 + mul x11, x4, x6 + adcs x16, x16, x11 + mul x11, x4, x7 + adcs x17, x17, x11 + mul x11, x4, x8 + adcs x19, x19, x11 + mul x11, x4, x9 + adcs x20, x20, x11 + mul x11, x4, x10 + adcs x21, x21, x11 + cset x22, hs + umulh x11, x4, x5 + adds x16, x16, x11 + umulh x11, x4, x6 + adcs x17, x17, x11 + umulh x11, x4, x7 + adcs x19, x19, x11 + umulh x11, x4, x8 + adcs x20, x20, x11 + umulh x11, x4, x9 + adcs x21, x21, x11 + umulh x11, x4, x10 + adc x22, x22, x11 + ldp x3, x4, [sp, #272] + mul x11, x3, x5 + adds x16, x16, x11 + mul x11, x3, x6 + adcs x17, x17, x11 + mul x11, x3, x7 + adcs x19, x19, x11 + mul x11, x3, x8 + adcs x20, x20, x11 + mul x11, x3, x9 + adcs x21, x21, x11 + mul x11, x3, x10 + adcs x22, x22, x11 + cset x2, hs + umulh x11, x3, x5 + adds x17, x17, x11 + umulh x11, x3, x6 + adcs x19, x19, x11 + umulh x11, x3, x7 + adcs x20, x20, x11 + umulh x11, x3, x8 + adcs x21, x21, x11 + umulh x11, x3, x9 + adcs x22, x22, x11 + umulh x11, x3, x10 + adc x2, x2, x11 + mul x11, x4, x5 + adds x17, x17, x11 + mul x11, x4, x6 + adcs x19, x19, x11 + mul x11, x4, x7 + adcs x20, x20, x11 + mul x11, x4, x8 + adcs x21, x21, x11 + mul x11, x4, x9 + adcs x22, x22, x11 + mul x11, x4, x10 + adcs x2, x2, x11 + cset x1, hs + umulh x11, x4, x5 + adds x19, x19, x11 + umulh x11, x4, x6 + adcs x20, x20, x11 + umulh x11, x4, x7 + adcs x21, x21, x11 + umulh x11, x4, x8 + adcs x22, x22, x11 + umulh x11, x4, x9 + adcs x2, x2, x11 + umulh x11, x4, x10 + adc x1, x1, x11 + lsl x7, x12, #32 + add x12, x7, x12 + mov x7, #-4294967295 + umulh x7, x7, x12 + mov x6, #4294967295 + mul x5, x6, x12 + umulh x6, x6, x12 + adds x7, x7, x5 + adcs x6, x6, x12 + adc x5, xzr, xzr + subs x13, x13, x7 + sbcs x14, x14, x6 + sbcs x15, x15, x5 + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x12, x12, xzr + lsl x7, x13, #32 + add x13, x7, x13 + mov x7, #-4294967295 + umulh x7, x7, x13 + mov x6, #4294967295 + mul x5, x6, x13 + umulh x6, x6, x13 + adds x7, x7, x5 + adcs x6, x6, x13 + adc x5, xzr, xzr + subs x14, x14, x7 + sbcs x15, x15, x6 + sbcs x16, x16, x5 + sbcs x17, x17, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + lsl x7, x14, #32 + add x14, x7, x14 + mov x7, #-4294967295 + umulh x7, x7, x14 + mov x6, #4294967295 + mul x5, x6, x14 + umulh x6, x6, x14 + adds x7, x7, x5 + adcs x6, x6, x14 + adc x5, xzr, xzr + subs x15, x15, x7 + sbcs x16, x16, x6 + sbcs x17, x17, x5 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + lsl x7, x15, #32 + add x15, x7, x15 + mov x7, #-4294967295 + umulh x7, x7, x15 + mov x6, #4294967295 + mul x5, x6, x15 + umulh x6, x6, x15 + adds x7, x7, x5 + adcs x6, x6, x15 + adc x5, xzr, xzr + subs x16, x16, x7 + sbcs x17, x17, x6 + sbcs x12, x12, x5 + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbc x15, x15, xzr + lsl x7, x16, #32 + add x16, x7, x16 + mov x7, #-4294967295 + umulh x7, x7, x16 + mov x6, #4294967295 + mul x5, x6, x16 + umulh x6, x6, x16 + adds x7, x7, x5 + adcs x6, x6, x16 + adc x5, xzr, xzr + subs x17, x17, x7 + sbcs x12, x12, x6 + sbcs x13, x13, x5 + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbc x16, x16, xzr + lsl x7, x17, #32 + add x17, x7, x17 + mov x7, #-4294967295 + umulh x7, x7, x17 + mov x6, #4294967295 + mul x5, x6, x17 + umulh x6, x6, x17 + adds x7, x7, x5 + adcs x6, x6, x17 + adc x5, xzr, xzr + subs x12, x12, x7 + sbcs x13, x13, x6 + sbcs x14, x14, x5 + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbc x17, x17, xzr + adds x12, x12, x19 + adcs x13, x13, x20 + adcs x14, x14, x21 + adcs x15, x15, x22 + adcs x16, x16, x2 + adcs x17, x17, x1 + adc x10, xzr, xzr + mov x11, #-4294967295 + adds x19, x12, x11 + mov x11, #4294967295 + adcs x20, x13, x11 + mov x11, #1 + adcs x21, x14, x11 + adcs x22, x15, xzr + adcs x2, x16, xzr + adcs x1, x17, xzr + adcs x10, x10, xzr + csel x12, x12, x19, eq + csel x13, x13, x20, eq + csel x14, x14, x21, eq + csel x15, x15, x22, eq + csel x16, x16, x2, eq + csel x17, x17, x1, eq + stp x12, x13, [sp, #96] + stp x14, x15, [sp, #112] + stp x16, x17, [sp, #128] + ldp x5, x6, [x24, #48] + ldp x4, x3, [x24, #96] + adds x5, x5, x4 + adcs x6, x6, x3 + ldp x7, x8, [x24, #64] + ldp x4, x3, [x24, #112] + adcs x7, x7, x4 + adcs x8, x8, x3 + ldp x9, x10, [x24, #80] + ldp x4, x3, [x24, #128] + adcs x9, x9, x4 + adcs x10, x10, x3 + adc x3, xzr, xzr + mov x4, #4294967295 + cmp x5, x4 + mov x4, #-4294967296 + sbcs xzr, x6, x4 + mov x4, #-2 + sbcs xzr, x7, x4 + adcs xzr, x8, xzr + adcs xzr, x9, xzr + adcs xzr, x10, xzr + adcs x3, x3, xzr + csetm x3, ne + mov x4, #4294967295 + and x4, x4, x3 + subs x5, x5, x4 + eor x4, x4, x3 + sbcs x6, x6, x4 + mov x4, #-2 + and x4, x4, x3 + sbcs x7, x7, x4 + sbcs x8, x8, x3 + sbcs x9, x9, x3 + sbc x10, x10, x3 + stp x5, x6, [sp, #240] + stp x7, x8, [sp, #256] + stp x9, x10, [sp, #272] + ldp x2, x3, [sp, #96] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [sp, #112] + mul x8, x2, x4 + adds x10, x10, x8 + mul x11, x2, x5 + mul x8, x3, x4 + adcs x11, x11, x8 + umulh x12, x2, x5 + mul x8, x3, x5 + adcs x12, x12, x8 + ldp x6, x7, [sp, #128] + mul x13, x2, x7 + mul x8, x3, x6 + adcs x13, x13, x8 + umulh x14, x2, x7 + mul x8, x3, x7 + adcs x14, x14, x8 + mul x15, x5, x6 + adcs x15, x15, xzr + umulh x16, x5, x6 + adc x16, x16, xzr + umulh x8, x2, x4 + adds x11, x11, x8 + umulh x8, x3, x4 + adcs x12, x12, x8 + umulh x8, x3, x5 + adcs x13, x13, x8 + umulh x8, x3, x6 + adcs x14, x14, x8 + umulh x8, x3, x7 + adcs x15, x15, x8 + adc x16, x16, xzr + mul x8, x2, x6 + adds x12, x12, x8 + mul x8, x4, x5 + adcs x13, x13, x8 + mul x8, x4, x6 + adcs x14, x14, x8 + mul x8, x4, x7 + adcs x15, x15, x8 + mul x8, x5, x7 + adcs x16, x16, x8 + mul x17, x6, x7 + adcs x17, x17, xzr + umulh x19, x6, x7 + adc x19, x19, xzr + umulh x8, x2, x6 + adds x13, x13, x8 + umulh x8, x4, x5 + adcs x14, x14, x8 + umulh x8, x4, x6 + adcs x15, x15, x8 + umulh x8, x4, x7 + adcs x16, x16, x8 + umulh x8, x5, x7 + adcs x17, x17, x8 + adc x19, x19, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + adcs x15, x15, x15 + adcs x16, x16, x16 + adcs x17, x17, x17 + adcs x19, x19, x19 + cset x20, hs + umulh x8, x2, x2 + mul x2, x2, x2 + adds x9, x9, x8 + mul x8, x3, x3 + adcs x10, x10, x8 + umulh x8, x3, x3 + adcs x11, x11, x8 + mul x8, x4, x4 + adcs x12, x12, x8 + umulh x8, x4, x4 + adcs x13, x13, x8 + mul x8, x5, x5 + adcs x14, x14, x8 + umulh x8, x5, x5 + adcs x15, x15, x8 + mul x8, x6, x6 + adcs x16, x16, x8 + umulh x8, x6, x6 + adcs x17, x17, x8 + mul x8, x7, x7 + adcs x19, x19, x8 + umulh x8, x7, x7 + adc x20, x20, x8 + lsl x5, x2, #32 + add x2, x5, x2 + mov x5, #-4294967295 + umulh x5, x5, x2 + mov x4, #4294967295 + mul x3, x4, x2 + umulh x4, x4, x2 + adds x5, x5, x3 + adcs x4, x4, x2 + adc x3, xzr, xzr + subs x9, x9, x5 + sbcs x10, x10, x4 + sbcs x11, x11, x3 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x2, x2, xzr + lsl x5, x9, #32 + add x9, x5, x9 + mov x5, #-4294967295 + umulh x5, x5, x9 + mov x4, #4294967295 + mul x3, x4, x9 + umulh x4, x4, x9 + adds x5, x5, x3 + adcs x4, x4, x9 + adc x3, xzr, xzr + subs x10, x10, x5 + sbcs x11, x11, x4 + sbcs x12, x12, x3 + sbcs x13, x13, xzr + sbcs x2, x2, xzr + sbc x9, x9, xzr + lsl x5, x10, #32 + add x10, x5, x10 + mov x5, #-4294967295 + umulh x5, x5, x10 + mov x4, #4294967295 + mul x3, x4, x10 + umulh x4, x4, x10 + adds x5, x5, x3 + adcs x4, x4, x10 + adc x3, xzr, xzr + subs x11, x11, x5 + sbcs x12, x12, x4 + sbcs x13, x13, x3 + sbcs x2, x2, xzr + sbcs x9, x9, xzr + sbc x10, x10, xzr + lsl x5, x11, #32 + add x11, x5, x11 + mov x5, #-4294967295 + umulh x5, x5, x11 + mov x4, #4294967295 + mul x3, x4, x11 + umulh x4, x4, x11 + adds x5, x5, x3 + adcs x4, x4, x11 + adc x3, xzr, xzr + subs x12, x12, x5 + sbcs x13, x13, x4 + sbcs x2, x2, x3 + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbc x11, x11, xzr + lsl x5, x12, #32 + add x12, x5, x12 + mov x5, #-4294967295 + umulh x5, x5, x12 + mov x4, #4294967295 + mul x3, x4, x12 + umulh x4, x4, x12 + adds x5, x5, x3 + adcs x4, x4, x12 + adc x3, xzr, xzr + subs x13, x13, x5 + sbcs x2, x2, x4 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbc x12, x12, xzr + lsl x5, x13, #32 + add x13, x5, x13 + mov x5, #-4294967295 + umulh x5, x5, x13 + mov x4, #4294967295 + mul x3, x4, x13 + umulh x4, x4, x13 + adds x5, x5, x3 + adcs x4, x4, x13 + adc x3, xzr, xzr + subs x2, x2, x5 + sbcs x9, x9, x4 + sbcs x10, x10, x3 + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + adds x2, x2, x14 + adcs x9, x9, x15 + adcs x10, x10, x16 + adcs x11, x11, x17 + adcs x12, x12, x19 + adcs x13, x13, x20 + adc x6, xzr, xzr + mov x8, #-4294967295 + adds x14, x2, x8 + mov x8, #4294967295 + adcs x15, x9, x8 + mov x8, #1 + adcs x16, x10, x8 + adcs x17, x11, xzr + adcs x19, x12, xzr + adcs x20, x13, xzr + adcs x6, x6, xzr + csel x2, x2, x14, eq + csel x9, x9, x15, eq + csel x10, x10, x16, eq + csel x11, x11, x17, eq + csel x12, x12, x19, eq + csel x13, x13, x20, eq + stp x2, x9, [sp, #288] + stp x10, x11, [sp, #304] + stp x12, x13, [sp, #320] + ldp x3, x4, [x24] + ldp x5, x6, [sp, #48] + mul x12, x3, x5 + umulh x13, x3, x5 + mul x11, x3, x6 + umulh x14, x3, x6 + adds x13, x13, x11 + ldp x7, x8, [sp, #64] + mul x11, x3, x7 + umulh x15, x3, x7 + adcs x14, x14, x11 + mul x11, x3, x8 + umulh x16, x3, x8 + adcs x15, x15, x11 + ldp x9, x10, [sp, #80] + mul x11, x3, x9 + umulh x17, x3, x9 + adcs x16, x16, x11 + mul x11, x3, x10 + umulh x19, x3, x10 + adcs x17, x17, x11 + adc x19, x19, xzr + mul x11, x4, x5 + adds x13, x13, x11 + mul x11, x4, x6 + adcs x14, x14, x11 + mul x11, x4, x7 + adcs x15, x15, x11 + mul x11, x4, x8 + adcs x16, x16, x11 + mul x11, x4, x9 + adcs x17, x17, x11 + mul x11, x4, x10 + adcs x19, x19, x11 + cset x20, hs + umulh x11, x4, x5 + adds x14, x14, x11 + umulh x11, x4, x6 + adcs x15, x15, x11 + umulh x11, x4, x7 + adcs x16, x16, x11 + umulh x11, x4, x8 + adcs x17, x17, x11 + umulh x11, x4, x9 + adcs x19, x19, x11 + umulh x11, x4, x10 + adc x20, x20, x11 + ldp x3, x4, [x24, #16] + mul x11, x3, x5 + adds x14, x14, x11 + mul x11, x3, x6 + adcs x15, x15, x11 + mul x11, x3, x7 + adcs x16, x16, x11 + mul x11, x3, x8 + adcs x17, x17, x11 + mul x11, x3, x9 + adcs x19, x19, x11 + mul x11, x3, x10 + adcs x20, x20, x11 + cset x21, hs + umulh x11, x3, x5 + adds x15, x15, x11 + umulh x11, x3, x6 + adcs x16, x16, x11 + umulh x11, x3, x7 + adcs x17, x17, x11 + umulh x11, x3, x8 + adcs x19, x19, x11 + umulh x11, x3, x9 + adcs x20, x20, x11 + umulh x11, x3, x10 + adc x21, x21, x11 + mul x11, x4, x5 + adds x15, x15, x11 + mul x11, x4, x6 + adcs x16, x16, x11 + mul x11, x4, x7 + adcs x17, x17, x11 + mul x11, x4, x8 + adcs x19, x19, x11 + mul x11, x4, x9 + adcs x20, x20, x11 + mul x11, x4, x10 + adcs x21, x21, x11 + cset x22, hs + umulh x11, x4, x5 + adds x16, x16, x11 + umulh x11, x4, x6 + adcs x17, x17, x11 + umulh x11, x4, x7 + adcs x19, x19, x11 + umulh x11, x4, x8 + adcs x20, x20, x11 + umulh x11, x4, x9 + adcs x21, x21, x11 + umulh x11, x4, x10 + adc x22, x22, x11 + ldp x3, x4, [x24, #32] + mul x11, x3, x5 + adds x16, x16, x11 + mul x11, x3, x6 + adcs x17, x17, x11 + mul x11, x3, x7 + adcs x19, x19, x11 + mul x11, x3, x8 + adcs x20, x20, x11 + mul x11, x3, x9 + adcs x21, x21, x11 + mul x11, x3, x10 + adcs x22, x22, x11 + cset x2, hs + umulh x11, x3, x5 + adds x17, x17, x11 + umulh x11, x3, x6 + adcs x19, x19, x11 + umulh x11, x3, x7 + adcs x20, x20, x11 + umulh x11, x3, x8 + adcs x21, x21, x11 + umulh x11, x3, x9 + adcs x22, x22, x11 + umulh x11, x3, x10 + adc x2, x2, x11 + mul x11, x4, x5 + adds x17, x17, x11 + mul x11, x4, x6 + adcs x19, x19, x11 + mul x11, x4, x7 + adcs x20, x20, x11 + mul x11, x4, x8 + adcs x21, x21, x11 + mul x11, x4, x9 + adcs x22, x22, x11 + mul x11, x4, x10 + adcs x2, x2, x11 + cset x1, hs + umulh x11, x4, x5 + adds x19, x19, x11 + umulh x11, x4, x6 + adcs x20, x20, x11 + umulh x11, x4, x7 + adcs x21, x21, x11 + umulh x11, x4, x8 + adcs x22, x22, x11 + umulh x11, x4, x9 + adcs x2, x2, x11 + umulh x11, x4, x10 + adc x1, x1, x11 + lsl x7, x12, #32 + add x12, x7, x12 + mov x7, #-4294967295 + umulh x7, x7, x12 + mov x6, #4294967295 + mul x5, x6, x12 + umulh x6, x6, x12 + adds x7, x7, x5 + adcs x6, x6, x12 + adc x5, xzr, xzr + subs x13, x13, x7 + sbcs x14, x14, x6 + sbcs x15, x15, x5 + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x12, x12, xzr + lsl x7, x13, #32 + add x13, x7, x13 + mov x7, #-4294967295 + umulh x7, x7, x13 + mov x6, #4294967295 + mul x5, x6, x13 + umulh x6, x6, x13 + adds x7, x7, x5 + adcs x6, x6, x13 + adc x5, xzr, xzr + subs x14, x14, x7 + sbcs x15, x15, x6 + sbcs x16, x16, x5 + sbcs x17, x17, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + lsl x7, x14, #32 + add x14, x7, x14 + mov x7, #-4294967295 + umulh x7, x7, x14 + mov x6, #4294967295 + mul x5, x6, x14 + umulh x6, x6, x14 + adds x7, x7, x5 + adcs x6, x6, x14 + adc x5, xzr, xzr + subs x15, x15, x7 + sbcs x16, x16, x6 + sbcs x17, x17, x5 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + lsl x7, x15, #32 + add x15, x7, x15 + mov x7, #-4294967295 + umulh x7, x7, x15 + mov x6, #4294967295 + mul x5, x6, x15 + umulh x6, x6, x15 + adds x7, x7, x5 + adcs x6, x6, x15 + adc x5, xzr, xzr + subs x16, x16, x7 + sbcs x17, x17, x6 + sbcs x12, x12, x5 + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbc x15, x15, xzr + lsl x7, x16, #32 + add x16, x7, x16 + mov x7, #-4294967295 + umulh x7, x7, x16 + mov x6, #4294967295 + mul x5, x6, x16 + umulh x6, x6, x16 + adds x7, x7, x5 + adcs x6, x6, x16 + adc x5, xzr, xzr + subs x17, x17, x7 + sbcs x12, x12, x6 + sbcs x13, x13, x5 + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbc x16, x16, xzr + lsl x7, x17, #32 + add x17, x7, x17 + mov x7, #-4294967295 + umulh x7, x7, x17 + mov x6, #4294967295 + mul x5, x6, x17 + umulh x6, x6, x17 + adds x7, x7, x5 + adcs x6, x6, x17 + adc x5, xzr, xzr + subs x12, x12, x7 + sbcs x13, x13, x6 + sbcs x14, x14, x5 + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbc x17, x17, xzr + adds x12, x12, x19 + adcs x13, x13, x20 + adcs x14, x14, x21 + adcs x15, x15, x22 + adcs x16, x16, x2 + adcs x17, x17, x1 + adc x10, xzr, xzr + mov x11, #-4294967295 + adds x19, x12, x11 + mov x11, #4294967295 + adcs x20, x13, x11 + mov x11, #1 + adcs x21, x14, x11 + adcs x22, x15, xzr + adcs x2, x16, xzr + adcs x1, x17, xzr + adcs x10, x10, xzr + csel x12, x12, x19, eq + csel x13, x13, x20, eq + csel x14, x14, x21, eq + csel x15, x15, x22, eq + csel x16, x16, x2, eq + csel x17, x17, x1, eq + stp x12, x13, [sp, #144] + stp x14, x15, [sp, #160] + stp x16, x17, [sp, #176] + ldp x2, x3, [sp, #240] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [sp, #256] + mul x8, x2, x4 + adds x10, x10, x8 + mul x11, x2, x5 + mul x8, x3, x4 + adcs x11, x11, x8 + umulh x12, x2, x5 + mul x8, x3, x5 + adcs x12, x12, x8 + ldp x6, x7, [sp, #272] + mul x13, x2, x7 + mul x8, x3, x6 + adcs x13, x13, x8 + umulh x14, x2, x7 + mul x8, x3, x7 + adcs x14, x14, x8 + mul x15, x5, x6 + adcs x15, x15, xzr + umulh x16, x5, x6 + adc x16, x16, xzr + umulh x8, x2, x4 + adds x11, x11, x8 + umulh x8, x3, x4 + adcs x12, x12, x8 + umulh x8, x3, x5 + adcs x13, x13, x8 + umulh x8, x3, x6 + adcs x14, x14, x8 + umulh x8, x3, x7 + adcs x15, x15, x8 + adc x16, x16, xzr + mul x8, x2, x6 + adds x12, x12, x8 + mul x8, x4, x5 + adcs x13, x13, x8 + mul x8, x4, x6 + adcs x14, x14, x8 + mul x8, x4, x7 + adcs x15, x15, x8 + mul x8, x5, x7 + adcs x16, x16, x8 + mul x17, x6, x7 + adcs x17, x17, xzr + umulh x19, x6, x7 + adc x19, x19, xzr + umulh x8, x2, x6 + adds x13, x13, x8 + umulh x8, x4, x5 + adcs x14, x14, x8 + umulh x8, x4, x6 + adcs x15, x15, x8 + umulh x8, x4, x7 + adcs x16, x16, x8 + umulh x8, x5, x7 + adcs x17, x17, x8 + adc x19, x19, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + adcs x15, x15, x15 + adcs x16, x16, x16 + adcs x17, x17, x17 + adcs x19, x19, x19 + cset x20, hs + umulh x8, x2, x2 + mul x2, x2, x2 + adds x9, x9, x8 + mul x8, x3, x3 + adcs x10, x10, x8 + umulh x8, x3, x3 + adcs x11, x11, x8 + mul x8, x4, x4 + adcs x12, x12, x8 + umulh x8, x4, x4 + adcs x13, x13, x8 + mul x8, x5, x5 + adcs x14, x14, x8 + umulh x8, x5, x5 + adcs x15, x15, x8 + mul x8, x6, x6 + adcs x16, x16, x8 + umulh x8, x6, x6 + adcs x17, x17, x8 + mul x8, x7, x7 + adcs x19, x19, x8 + umulh x8, x7, x7 + adc x20, x20, x8 + lsl x5, x2, #32 + add x2, x5, x2 + mov x5, #-4294967295 + umulh x5, x5, x2 + mov x4, #4294967295 + mul x3, x4, x2 + umulh x4, x4, x2 + adds x5, x5, x3 + adcs x4, x4, x2 + adc x3, xzr, xzr + subs x9, x9, x5 + sbcs x10, x10, x4 + sbcs x11, x11, x3 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x2, x2, xzr + lsl x5, x9, #32 + add x9, x5, x9 + mov x5, #-4294967295 + umulh x5, x5, x9 + mov x4, #4294967295 + mul x3, x4, x9 + umulh x4, x4, x9 + adds x5, x5, x3 + adcs x4, x4, x9 + adc x3, xzr, xzr + subs x10, x10, x5 + sbcs x11, x11, x4 + sbcs x12, x12, x3 + sbcs x13, x13, xzr + sbcs x2, x2, xzr + sbc x9, x9, xzr + lsl x5, x10, #32 + add x10, x5, x10 + mov x5, #-4294967295 + umulh x5, x5, x10 + mov x4, #4294967295 + mul x3, x4, x10 + umulh x4, x4, x10 + adds x5, x5, x3 + adcs x4, x4, x10 + adc x3, xzr, xzr + subs x11, x11, x5 + sbcs x12, x12, x4 + sbcs x13, x13, x3 + sbcs x2, x2, xzr + sbcs x9, x9, xzr + sbc x10, x10, xzr + lsl x5, x11, #32 + add x11, x5, x11 + mov x5, #-4294967295 + umulh x5, x5, x11 + mov x4, #4294967295 + mul x3, x4, x11 + umulh x4, x4, x11 + adds x5, x5, x3 + adcs x4, x4, x11 + adc x3, xzr, xzr + subs x12, x12, x5 + sbcs x13, x13, x4 + sbcs x2, x2, x3 + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbc x11, x11, xzr + lsl x5, x12, #32 + add x12, x5, x12 + mov x5, #-4294967295 + umulh x5, x5, x12 + mov x4, #4294967295 + mul x3, x4, x12 + umulh x4, x4, x12 + adds x5, x5, x3 + adcs x4, x4, x12 + adc x3, xzr, xzr + subs x13, x13, x5 + sbcs x2, x2, x4 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbc x12, x12, xzr + lsl x5, x13, #32 + add x13, x5, x13 + mov x5, #-4294967295 + umulh x5, x5, x13 + mov x4, #4294967295 + mul x3, x4, x13 + umulh x4, x4, x13 + adds x5, x5, x3 + adcs x4, x4, x13 + adc x3, xzr, xzr + subs x2, x2, x5 + sbcs x9, x9, x4 + sbcs x10, x10, x3 + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + adds x2, x2, x14 + adcs x9, x9, x15 + adcs x10, x10, x16 + adcs x11, x11, x17 + adcs x12, x12, x19 + adcs x13, x13, x20 + adc x6, xzr, xzr + mov x8, #-4294967295 + adds x14, x2, x8 + mov x8, #4294967295 + adcs x15, x9, x8 + mov x8, #1 + adcs x16, x10, x8 + adcs x17, x11, xzr + adcs x19, x12, xzr + adcs x20, x13, xzr + adcs x6, x6, xzr + csel x2, x2, x14, eq + csel x9, x9, x15, eq + csel x10, x10, x16, eq + csel x11, x11, x17, eq + csel x12, x12, x19, eq + csel x13, x13, x20, eq + stp x2, x9, [sp, #192] + stp x10, x11, [sp, #208] + stp x12, x13, [sp, #224] + ldp x0, x1, [sp, #288] + mov x6, #4294967295 + subs x6, x6, x0 + mov x7, #-4294967296 + sbcs x7, x7, x1 + ldp x0, x1, [sp, #304] + mov x8, #-2 + sbcs x8, x8, x0 + mov x13, #-1 + sbcs x9, x13, x1 + ldp x0, x1, [sp, #320] + sbcs x10, x13, x0 + sbc x11, x13, x1 + mov x12, #9 + mul x0, x12, x6 + mul x1, x12, x7 + mul x2, x12, x8 + mul x3, x12, x9 + mul x4, x12, x10 + mul x5, x12, x11 + umulh x6, x12, x6 + umulh x7, x12, x7 + umulh x8, x12, x8 + umulh x9, x12, x9 + umulh x10, x12, x10 + umulh x12, x12, x11 + adds x1, x1, x6 + adcs x2, x2, x7 + adcs x3, x3, x8 + adcs x4, x4, x9 + adcs x5, x5, x10 + mov x6, #1 + adc x6, x12, x6 + ldp x8, x9, [sp, #144] + ldp x10, x11, [sp, #160] + ldp x12, x13, [sp, #176] + mov x14, #12 + mul x15, x14, x8 + umulh x8, x14, x8 + adds x0, x0, x15 + mul x15, x14, x9 + umulh x9, x14, x9 + adcs x1, x1, x15 + mul x15, x14, x10 + umulh x10, x14, x10 + adcs x2, x2, x15 + mul x15, x14, x11 + umulh x11, x14, x11 + adcs x3, x3, x15 + mul x15, x14, x12 + umulh x12, x14, x12 + adcs x4, x4, x15 + mul x15, x14, x13 + umulh x13, x14, x13 + adcs x5, x5, x15 + adc x6, x6, xzr + adds x1, x1, x8 + adcs x2, x2, x9 + adcs x3, x3, x10 + adcs x4, x4, x11 + adcs x5, x5, x12 + adcs x6, x6, x13 + lsl x7, x6, #32 + subs x8, x6, x7 + sbc x7, x7, xzr + adds x0, x0, x8 + adcs x1, x1, x7 + adcs x2, x2, x6 + adcs x3, x3, xzr + adcs x4, x4, xzr + adcs x5, x5, xzr + csetm x6, lo + mov x7, #4294967295 + and x7, x7, x6 + adds x0, x0, x7 + eor x7, x7, x6 + adcs x1, x1, x7 + mov x7, #-2 + and x7, x7, x6 + adcs x2, x2, x7 + adcs x3, x3, x6 + adcs x4, x4, x6 + adc x5, x5, x6 + stp x0, x1, [sp, #288] + stp x2, x3, [sp, #304] + stp x4, x5, [sp, #320] + ldp x5, x6, [sp, #192] + ldp x4, x3, [sp] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #208] + ldp x4, x3, [sp, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #224] + ldp x4, x3, [sp, #32] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, lo + mov x4, #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [sp, #240] + stp x7, x8, [sp, #256] + stp x9, x10, [sp, #272] + ldp x2, x3, [sp, #48] + mul x9, x2, x3 + umulh x10, x2, x3 + ldp x4, x5, [sp, #64] + mul x8, x2, x4 + adds x10, x10, x8 + mul x11, x2, x5 + mul x8, x3, x4 + adcs x11, x11, x8 + umulh x12, x2, x5 + mul x8, x3, x5 + adcs x12, x12, x8 + ldp x6, x7, [sp, #80] + mul x13, x2, x7 + mul x8, x3, x6 + adcs x13, x13, x8 + umulh x14, x2, x7 + mul x8, x3, x7 + adcs x14, x14, x8 + mul x15, x5, x6 + adcs x15, x15, xzr + umulh x16, x5, x6 + adc x16, x16, xzr + umulh x8, x2, x4 + adds x11, x11, x8 + umulh x8, x3, x4 + adcs x12, x12, x8 + umulh x8, x3, x5 + adcs x13, x13, x8 + umulh x8, x3, x6 + adcs x14, x14, x8 + umulh x8, x3, x7 + adcs x15, x15, x8 + adc x16, x16, xzr + mul x8, x2, x6 + adds x12, x12, x8 + mul x8, x4, x5 + adcs x13, x13, x8 + mul x8, x4, x6 + adcs x14, x14, x8 + mul x8, x4, x7 + adcs x15, x15, x8 + mul x8, x5, x7 + adcs x16, x16, x8 + mul x17, x6, x7 + adcs x17, x17, xzr + umulh x19, x6, x7 + adc x19, x19, xzr + umulh x8, x2, x6 + adds x13, x13, x8 + umulh x8, x4, x5 + adcs x14, x14, x8 + umulh x8, x4, x6 + adcs x15, x15, x8 + umulh x8, x4, x7 + adcs x16, x16, x8 + umulh x8, x5, x7 + adcs x17, x17, x8 + adc x19, x19, xzr + adds x9, x9, x9 + adcs x10, x10, x10 + adcs x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + adcs x15, x15, x15 + adcs x16, x16, x16 + adcs x17, x17, x17 + adcs x19, x19, x19 + cset x20, hs + umulh x8, x2, x2 + mul x2, x2, x2 + adds x9, x9, x8 + mul x8, x3, x3 + adcs x10, x10, x8 + umulh x8, x3, x3 + adcs x11, x11, x8 + mul x8, x4, x4 + adcs x12, x12, x8 + umulh x8, x4, x4 + adcs x13, x13, x8 + mul x8, x5, x5 + adcs x14, x14, x8 + umulh x8, x5, x5 + adcs x15, x15, x8 + mul x8, x6, x6 + adcs x16, x16, x8 + umulh x8, x6, x6 + adcs x17, x17, x8 + mul x8, x7, x7 + adcs x19, x19, x8 + umulh x8, x7, x7 + adc x20, x20, x8 + lsl x5, x2, #32 + add x2, x5, x2 + mov x5, #-4294967295 + umulh x5, x5, x2 + mov x4, #4294967295 + mul x3, x4, x2 + umulh x4, x4, x2 + adds x5, x5, x3 + adcs x4, x4, x2 + adc x3, xzr, xzr + subs x9, x9, x5 + sbcs x10, x10, x4 + sbcs x11, x11, x3 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x2, x2, xzr + lsl x5, x9, #32 + add x9, x5, x9 + mov x5, #-4294967295 + umulh x5, x5, x9 + mov x4, #4294967295 + mul x3, x4, x9 + umulh x4, x4, x9 + adds x5, x5, x3 + adcs x4, x4, x9 + adc x3, xzr, xzr + subs x10, x10, x5 + sbcs x11, x11, x4 + sbcs x12, x12, x3 + sbcs x13, x13, xzr + sbcs x2, x2, xzr + sbc x9, x9, xzr + lsl x5, x10, #32 + add x10, x5, x10 + mov x5, #-4294967295 + umulh x5, x5, x10 + mov x4, #4294967295 + mul x3, x4, x10 + umulh x4, x4, x10 + adds x5, x5, x3 + adcs x4, x4, x10 + adc x3, xzr, xzr + subs x11, x11, x5 + sbcs x12, x12, x4 + sbcs x13, x13, x3 + sbcs x2, x2, xzr + sbcs x9, x9, xzr + sbc x10, x10, xzr + lsl x5, x11, #32 + add x11, x5, x11 + mov x5, #-4294967295 + umulh x5, x5, x11 + mov x4, #4294967295 + mul x3, x4, x11 + umulh x4, x4, x11 + adds x5, x5, x3 + adcs x4, x4, x11 + adc x3, xzr, xzr + subs x12, x12, x5 + sbcs x13, x13, x4 + sbcs x2, x2, x3 + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbc x11, x11, xzr + lsl x5, x12, #32 + add x12, x5, x12 + mov x5, #-4294967295 + umulh x5, x5, x12 + mov x4, #4294967295 + mul x3, x4, x12 + umulh x4, x4, x12 + adds x5, x5, x3 + adcs x4, x4, x12 + adc x3, xzr, xzr + subs x13, x13, x5 + sbcs x2, x2, x4 + sbcs x9, x9, x3 + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbc x12, x12, xzr + lsl x5, x13, #32 + add x13, x5, x13 + mov x5, #-4294967295 + umulh x5, x5, x13 + mov x4, #4294967295 + mul x3, x4, x13 + umulh x4, x4, x13 + adds x5, x5, x3 + adcs x4, x4, x13 + adc x3, xzr, xzr + subs x2, x2, x5 + sbcs x9, x9, x4 + sbcs x10, x10, x3 + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + adds x2, x2, x14 + adcs x9, x9, x15 + adcs x10, x10, x16 + adcs x11, x11, x17 + adcs x12, x12, x19 + adcs x13, x13, x20 + adc x6, xzr, xzr + mov x8, #-4294967295 + adds x14, x2, x8 + mov x8, #4294967295 + adcs x15, x9, x8 + mov x8, #1 + adcs x16, x10, x8 + adcs x17, x11, xzr + adcs x19, x12, xzr + adcs x20, x13, xzr + adcs x6, x6, xzr + csel x2, x2, x14, eq + csel x9, x9, x15, eq + csel x10, x10, x16, eq + csel x11, x11, x17, eq + csel x12, x12, x19, eq + csel x13, x13, x20, eq + stp x2, x9, [sp, #192] + stp x10, x11, [sp, #208] + stp x12, x13, [sp, #224] + ldp x5, x6, [sp, #240] + ldp x4, x3, [sp, #48] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #256] + ldp x4, x3, [sp, #64] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #272] + ldp x4, x3, [sp, #80] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + csetm x3, lo + mov x4, #4294967295 + and x4, x4, x3 + adds x5, x5, x4 + eor x4, x4, x3 + adcs x6, x6, x4 + mov x4, #-2 + and x4, x4, x3 + adcs x7, x7, x4 + adcs x8, x8, x3 + adcs x9, x9, x3 + adc x10, x10, x3 + stp x5, x6, [x23, #96] + stp x7, x8, [x23, #112] + stp x9, x10, [x23, #128] + ldp x3, x4, [sp, #288] + ldp x5, x6, [sp, #96] + mul x12, x3, x5 + umulh x13, x3, x5 + mul x11, x3, x6 + umulh x14, x3, x6 + adds x13, x13, x11 + ldp x7, x8, [sp, #112] + mul x11, x3, x7 + umulh x15, x3, x7 + adcs x14, x14, x11 + mul x11, x3, x8 + umulh x16, x3, x8 + adcs x15, x15, x11 + ldp x9, x10, [sp, #128] + mul x11, x3, x9 + umulh x17, x3, x9 + adcs x16, x16, x11 + mul x11, x3, x10 + umulh x19, x3, x10 + adcs x17, x17, x11 + adc x19, x19, xzr + mul x11, x4, x5 + adds x13, x13, x11 + mul x11, x4, x6 + adcs x14, x14, x11 + mul x11, x4, x7 + adcs x15, x15, x11 + mul x11, x4, x8 + adcs x16, x16, x11 + mul x11, x4, x9 + adcs x17, x17, x11 + mul x11, x4, x10 + adcs x19, x19, x11 + cset x20, hs + umulh x11, x4, x5 + adds x14, x14, x11 + umulh x11, x4, x6 + adcs x15, x15, x11 + umulh x11, x4, x7 + adcs x16, x16, x11 + umulh x11, x4, x8 + adcs x17, x17, x11 + umulh x11, x4, x9 + adcs x19, x19, x11 + umulh x11, x4, x10 + adc x20, x20, x11 + ldp x3, x4, [sp, #304] + mul x11, x3, x5 + adds x14, x14, x11 + mul x11, x3, x6 + adcs x15, x15, x11 + mul x11, x3, x7 + adcs x16, x16, x11 + mul x11, x3, x8 + adcs x17, x17, x11 + mul x11, x3, x9 + adcs x19, x19, x11 + mul x11, x3, x10 + adcs x20, x20, x11 + cset x21, hs + umulh x11, x3, x5 + adds x15, x15, x11 + umulh x11, x3, x6 + adcs x16, x16, x11 + umulh x11, x3, x7 + adcs x17, x17, x11 + umulh x11, x3, x8 + adcs x19, x19, x11 + umulh x11, x3, x9 + adcs x20, x20, x11 + umulh x11, x3, x10 + adc x21, x21, x11 + mul x11, x4, x5 + adds x15, x15, x11 + mul x11, x4, x6 + adcs x16, x16, x11 + mul x11, x4, x7 + adcs x17, x17, x11 + mul x11, x4, x8 + adcs x19, x19, x11 + mul x11, x4, x9 + adcs x20, x20, x11 + mul x11, x4, x10 + adcs x21, x21, x11 + cset x22, hs + umulh x11, x4, x5 + adds x16, x16, x11 + umulh x11, x4, x6 + adcs x17, x17, x11 + umulh x11, x4, x7 + adcs x19, x19, x11 + umulh x11, x4, x8 + adcs x20, x20, x11 + umulh x11, x4, x9 + adcs x21, x21, x11 + umulh x11, x4, x10 + adc x22, x22, x11 + ldp x3, x4, [sp, #320] + mul x11, x3, x5 + adds x16, x16, x11 + mul x11, x3, x6 + adcs x17, x17, x11 + mul x11, x3, x7 + adcs x19, x19, x11 + mul x11, x3, x8 + adcs x20, x20, x11 + mul x11, x3, x9 + adcs x21, x21, x11 + mul x11, x3, x10 + adcs x22, x22, x11 + cset x2, hs + umulh x11, x3, x5 + adds x17, x17, x11 + umulh x11, x3, x6 + adcs x19, x19, x11 + umulh x11, x3, x7 + adcs x20, x20, x11 + umulh x11, x3, x8 + adcs x21, x21, x11 + umulh x11, x3, x9 + adcs x22, x22, x11 + umulh x11, x3, x10 + adc x2, x2, x11 + mul x11, x4, x5 + adds x17, x17, x11 + mul x11, x4, x6 + adcs x19, x19, x11 + mul x11, x4, x7 + adcs x20, x20, x11 + mul x11, x4, x8 + adcs x21, x21, x11 + mul x11, x4, x9 + adcs x22, x22, x11 + mul x11, x4, x10 + adcs x2, x2, x11 + cset x1, hs + umulh x11, x4, x5 + adds x19, x19, x11 + umulh x11, x4, x6 + adcs x20, x20, x11 + umulh x11, x4, x7 + adcs x21, x21, x11 + umulh x11, x4, x8 + adcs x22, x22, x11 + umulh x11, x4, x9 + adcs x2, x2, x11 + umulh x11, x4, x10 + adc x1, x1, x11 + lsl x7, x12, #32 + add x12, x7, x12 + mov x7, #-4294967295 + umulh x7, x7, x12 + mov x6, #4294967295 + mul x5, x6, x12 + umulh x6, x6, x12 + adds x7, x7, x5 + adcs x6, x6, x12 + adc x5, xzr, xzr + subs x13, x13, x7 + sbcs x14, x14, x6 + sbcs x15, x15, x5 + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x12, x12, xzr + lsl x7, x13, #32 + add x13, x7, x13 + mov x7, #-4294967295 + umulh x7, x7, x13 + mov x6, #4294967295 + mul x5, x6, x13 + umulh x6, x6, x13 + adds x7, x7, x5 + adcs x6, x6, x13 + adc x5, xzr, xzr + subs x14, x14, x7 + sbcs x15, x15, x6 + sbcs x16, x16, x5 + sbcs x17, x17, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + lsl x7, x14, #32 + add x14, x7, x14 + mov x7, #-4294967295 + umulh x7, x7, x14 + mov x6, #4294967295 + mul x5, x6, x14 + umulh x6, x6, x14 + adds x7, x7, x5 + adcs x6, x6, x14 + adc x5, xzr, xzr + subs x15, x15, x7 + sbcs x16, x16, x6 + sbcs x17, x17, x5 + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + lsl x7, x15, #32 + add x15, x7, x15 + mov x7, #-4294967295 + umulh x7, x7, x15 + mov x6, #4294967295 + mul x5, x6, x15 + umulh x6, x6, x15 + adds x7, x7, x5 + adcs x6, x6, x15 + adc x5, xzr, xzr + subs x16, x16, x7 + sbcs x17, x17, x6 + sbcs x12, x12, x5 + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbc x15, x15, xzr + lsl x7, x16, #32 + add x16, x7, x16 + mov x7, #-4294967295 + umulh x7, x7, x16 + mov x6, #4294967295 + mul x5, x6, x16 + umulh x6, x6, x16 + adds x7, x7, x5 + adcs x6, x6, x16 + adc x5, xzr, xzr + subs x17, x17, x7 + sbcs x12, x12, x6 + sbcs x13, x13, x5 + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbc x16, x16, xzr + lsl x7, x17, #32 + add x17, x7, x17 + mov x7, #-4294967295 + umulh x7, x7, x17 + mov x6, #4294967295 + mul x5, x6, x17 + umulh x6, x6, x17 + adds x7, x7, x5 + adcs x6, x6, x17 + adc x5, xzr, xzr + subs x12, x12, x7 + sbcs x13, x13, x6 + sbcs x14, x14, x5 + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbc x17, x17, xzr + adds x12, x12, x19 + adcs x13, x13, x20 + adcs x14, x14, x21 + adcs x15, x15, x22 + adcs x16, x16, x2 + adcs x17, x17, x1 + adc x10, xzr, xzr + mov x11, #-4294967295 + adds x19, x12, x11 + mov x11, #4294967295 + adcs x20, x13, x11 + mov x11, #1 + adcs x21, x14, x11 + adcs x22, x15, xzr + adcs x2, x16, xzr + adcs x1, x17, xzr + adcs x10, x10, xzr + csel x12, x12, x19, eq + csel x13, x13, x20, eq + csel x14, x14, x21, eq + csel x15, x15, x22, eq + csel x16, x16, x2, eq + csel x17, x17, x1, eq + stp x12, x13, [sp, #240] + stp x14, x15, [sp, #256] + stp x16, x17, [sp, #272] + ldp x1, x2, [sp, #144] + ldp x3, x4, [sp, #160] + ldp x5, x6, [sp, #176] + lsl x0, x1, #2 + ldp x7, x8, [sp, #288] + subs x0, x0, x7 + extr x1, x2, x1, #62 + sbcs x1, x1, x8 + ldp x7, x8, [sp, #304] + extr x2, x3, x2, #62 + sbcs x2, x2, x7 + extr x3, x4, x3, #62 + sbcs x3, x3, x8 + extr x4, x5, x4, #62 + ldp x7, x8, [sp, #320] + sbcs x4, x4, x7 + extr x5, x6, x5, #62 + sbcs x5, x5, x8 + lsr x6, x6, #62 + adc x6, x6, xzr + lsl x7, x6, #32 + subs x8, x6, x7 + sbc x7, x7, xzr + adds x0, x0, x8 + adcs x1, x1, x7 + adcs x2, x2, x6 + adcs x3, x3, xzr + adcs x4, x4, xzr + adcs x5, x5, xzr + csetm x8, lo + mov x9, #4294967295 + and x9, x9, x8 + adds x0, x0, x9 + eor x9, x9, x8 + adcs x1, x1, x9 + mov x9, #-2 + and x9, x9, x8 + adcs x2, x2, x9 + adcs x3, x3, x8 + adcs x4, x4, x8 + adc x5, x5, x8 + stp x0, x1, [x23] + stp x2, x3, [x23, #16] + stp x4, x5, [x23, #32] + ldp x0, x1, [sp, #192] + mov x6, #4294967295 + subs x6, x6, x0 + mov x7, #-4294967296 + sbcs x7, x7, x1 + ldp x0, x1, [sp, #208] + mov x8, #-2 + sbcs x8, x8, x0 + mov x13, #-1 + sbcs x9, x13, x1 + ldp x0, x1, [sp, #224] + sbcs x10, x13, x0 + sbc x11, x13, x1 + lsl x0, x6, #3 + extr x1, x7, x6, #61 + extr x2, x8, x7, #61 + extr x3, x9, x8, #61 + extr x4, x10, x9, #61 + extr x5, x11, x10, #61 + lsr x6, x11, #61 + add x6, x6, #1 + ldp x8, x9, [sp, #240] + ldp x10, x11, [sp, #256] + ldp x12, x13, [sp, #272] + mov x14, #3 + mul x15, x14, x8 + umulh x8, x14, x8 + adds x0, x0, x15 + mul x15, x14, x9 + umulh x9, x14, x9 + adcs x1, x1, x15 + mul x15, x14, x10 + umulh x10, x14, x10 + adcs x2, x2, x15 + mul x15, x14, x11 + umulh x11, x14, x11 + adcs x3, x3, x15 + mul x15, x14, x12 + umulh x12, x14, x12 + adcs x4, x4, x15 + mul x15, x14, x13 + umulh x13, x14, x13 + adcs x5, x5, x15 + adc x6, x6, xzr + adds x1, x1, x8 + adcs x2, x2, x9 + adcs x3, x3, x10 + adcs x4, x4, x11 + adcs x5, x5, x12 + adcs x6, x6, x13 + lsl x7, x6, #32 + subs x8, x6, x7 + sbc x7, x7, xzr + adds x0, x0, x8 + adcs x1, x1, x7 + adcs x2, x2, x6 + adcs x3, x3, xzr + adcs x4, x4, xzr + adcs x5, x5, xzr + csetm x6, lo + mov x7, #4294967295 + and x7, x7, x6 + adds x0, x0, x7 + eor x7, x7, x6 + adcs x1, x1, x7 + mov x7, #-2 + and x7, x7, x6 + adcs x2, x2, x7 + adcs x3, x3, x6 + adcs x4, x4, x6 + adc x5, x5, x6 + stp x0, x1, [x23, #48] + stp x2, x3, [x23, #64] + stp x4, x5, [x23, #80] + CFI_INC_SP(336) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp384_montjscalarmul_alt_p384_montjdouble) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/p521_jscalarmul.S b/cbits/s2n/arm/p521_jscalarmul.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/p521_jscalarmul.S @@ -0,0 +1,2747 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Jacobian form scalar multiplication for P-521 +// Input scalar[9], point[27]; output res[27] +// +// extern void p521_jscalarmul +// (uint64_t res[static 27], +// const uint64_t scalar[static 9], +// const uint64_t point[static 27]); +// +// This function is a variant of its affine point version p521_scalarmul. +// Here, input and output points are assumed to be in Jacobian form with +// a triple (x,y,z) representing the affine point (x/z^2,y/z^3) when +// z is nonzero or the point at infinity (group identity) if z = 0. +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-521, returns a representation of n * P. If the result is the +// point at infinity (either because the input point was or because the +// scalar was a multiple of p_521) then the output is guaranteed to +// represent the point at infinity, i.e. to have its z coordinate zero. +// +// Standard ARM ABI: X0 = res, X1 = scalar, X2 = point +// ---------------------------------------------------------------------------- + + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p521_jscalarmul) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p521_jscalarmul) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p521_jscalarmul) + + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 72 +#define JACSIZE (3*NUMSIZE) + +// Safe copies of input res and additional values in variables. + +#define tabup x15 +#define bf x16 +#define sgn x17 +#define j x19 +#define res x20 + +// Intermediate variables on the stack. +// The table is 16 entries, each of size JACSIZE = 3 * NUMSIZE + +#define scalarb sp, #(0*NUMSIZE) +#define acc sp, #(1*NUMSIZE) +#define tabent sp, #(4*NUMSIZE) + +#define tab sp, #(7*NUMSIZE) + +// Round up to maintain stack alignment + +#define NSPACE 3968 + +#define selectblock(I) \ + cmp bf, #(1*I) __LF \ + ldp x10, x11, [tabup] __LF \ + csel x0, x10, x0, eq __LF \ + csel x1, x11, x1, eq __LF \ + ldp x10, x11, [tabup, #16] __LF \ + csel x2, x10, x2, eq __LF \ + csel x3, x11, x3, eq __LF \ + ldp x10, x11, [tabup, #32] __LF \ + csel x4, x10, x4, eq __LF \ + csel x5, x11, x5, eq __LF \ + ldp x10, x11, [tabup, #48] __LF \ + csel x6, x10, x6, eq __LF \ + csel x7, x11, x7, eq __LF \ + ldr x10, [tabup, #64] __LF \ + csel x8, x10, x8, eq __LF \ + add tabup, tabup, #JACSIZE + +// Loading large constants + +#define movbig(nn,n3,n2,n1,n0) \ + movz nn, n0 __LF \ + movk nn, n1, lsl #16 __LF \ + movk nn, n2, lsl #32 __LF \ + movk nn, n3, lsl #48 + +S2N_BN_SYMBOL(p521_jscalarmul): + CFI_START + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x30) + CFI_DEC_SP(NSPACE) + +// Preserve the "res" input argument; others get processed early. + + mov res, x0 + +// Reduce the input scalar mod n_521 and store it to "scalarb". + + mov x19, x2 + add x0, scalarb + CFI_BL(Lp521_jscalarmul_bignum_mod_n521_9) + mov x2, x19 + +// Set the tab[0] table entry to the input point = 1 * P, but also +// reduce all coordinates modulo p. In principle we assume reduction +// as a precondition, but this reduces the scope for surprise, e.g. +// making sure that any input with z = 0 is treated as zero, even +// if the other coordinates are not in fact reduced. + + add x0, tab + mov x1, x19 + CFI_BL(Lp521_jscalarmul_bignum_mod_p521_9) + + add x0, tab+NUMSIZE + add x1, x19, #NUMSIZE + CFI_BL(Lp521_jscalarmul_bignum_mod_p521_9) + + add x0, tab+2*NUMSIZE + add x1, x19, #(2*NUMSIZE) + CFI_BL(Lp521_jscalarmul_bignum_mod_p521_9) + +// If bit 520 of the scalar is set, then negate the scalar mod n_521, +// i.e. do scalar |-> n_521 - scalar, and also the point to compensate +// by negating its y coordinate. This further step is not needed by +// the indexing scheme (the top window is only a couple of bits either +// way), but is convenient to exclude a problem with the specific value +// scalar = n_521 - 18, where the last Jacobian addition is of the form +// (n_521 - 9) * P + -(9 * P) and hence is a degenerate doubling case. + + ldp x0, x1, [scalarb] + movbig(x10, #0xbb6f, #0xb71e, #0x9138, #0x6409) + subs x10, x10, x0 + movbig(x11, #0x3bb5, #0xc9b8, #0x899c, #0x47ae) + sbcs x11, x11, x1 + ldp x2, x3, [scalarb+16] + movbig(x12, #0x7fcc, #0x0148, #0xf709, #0xa5d0) + sbcs x12, x12, x2 + movbig(x13, #0x5186, #0x8783, #0xbf2f, #0x966b) + sbcs x13, x13, x3 + ldp x4, x5, [scalarb+32] + mov x14, 0xfffffffffffffffa + sbcs x14, x14, x4 + mov x15, 0xffffffffffffffff + sbcs x15, x15, x5 + ldp x6, x7, [scalarb+48] + mov x16, 0xffffffffffffffff + sbcs x16, x16, x6 + mov x17, 0xffffffffffffffff + sbcs x17, x17, x7 + ldr x8, [scalarb+64] + mov x19, 0x00000000000001ff + sbc x19, x19, x8 + tst x8, 0x100 + csetm x9, ne + csel x0, x10, x0, ne + csel x1, x11, x1, ne + csel x2, x12, x2, ne + csel x3, x13, x3, ne + csel x4, x14, x4, ne + csel x5, x15, x5, ne + csel x6, x16, x6, ne + csel x7, x17, x7, ne + csel x8, x19, x8, ne + stp x0, x1, [scalarb] + stp x2, x3, [scalarb+16] + stp x4, x5, [scalarb+32] + stp x6, x7, [scalarb+48] + str x8, [scalarb+64] + + add tabup, tab + ldp x0, x1, [tabup, #NUMSIZE] + ldp x2, x3, [tabup, #NUMSIZE+16] + ldp x4, x5, [tabup, #NUMSIZE+32] + ldp x6, x7, [tabup, #NUMSIZE+48] + ldr x8, [tabup, #NUMSIZE+64] + orr x10, x0, x1 + orr x11, x2, x3 + orr x12, x4, x5 + orr x13, x6, x7 + orr x10, x10, x11 + orr x12, x12, x13 + orr x12, x12, x8 + orr x10, x10, x12 + cmp x10, xzr + csel x9, x9, xzr, ne + eor x0, x0, x9 + eor x1, x1, x9 + eor x2, x2, x9 + eor x3, x3, x9 + eor x4, x4, x9 + eor x5, x5, x9 + eor x6, x6, x9 + eor x7, x7, x9 + and x9, x9, #0x1FF + eor x8, x8, x9 + stp x0, x1, [tabup, #NUMSIZE] + stp x2, x3, [tabup, #NUMSIZE+16] + stp x4, x5, [tabup, #NUMSIZE+32] + stp x6, x7, [tabup, #NUMSIZE+48] + str x8, [tabup, #NUMSIZE+64] + +// Compute and record tab[1] = 2 * p, ..., tab[15] = 16 * P + + add x0, tab+JACSIZE*1 + add x1, tab + CFI_BL(Lp521_jscalarmul_jdouble) + + add x0, tab+JACSIZE*2 + add x1, tab+JACSIZE*1 + add x2, tab + CFI_BL(Lp521_jscalarmul_jadd) + + add x0, tab+JACSIZE*3 + add x1, tab+JACSIZE*1 + CFI_BL(Lp521_jscalarmul_jdouble) + + add x0, tab+JACSIZE*4 + add x1, tab+JACSIZE*3 + add x2, tab + CFI_BL(Lp521_jscalarmul_jadd) + + add x0, tab+JACSIZE*5 + add x1, tab+JACSIZE*2 + CFI_BL(Lp521_jscalarmul_jdouble) + + add x0, tab+JACSIZE*6 + add x1, tab+JACSIZE*5 + add x2, tab + CFI_BL(Lp521_jscalarmul_jadd) + + add x0, tab+JACSIZE*7 + add x1, tab+JACSIZE*3 + CFI_BL(Lp521_jscalarmul_jdouble) + + add x0, tab+JACSIZE*8 + add x1, tab+JACSIZE*7 + add x2, tab + CFI_BL(Lp521_jscalarmul_jadd) + + add x0, tab+JACSIZE*9 + add x1, tab+JACSIZE*4 + CFI_BL(Lp521_jscalarmul_jdouble) + + add x0, tab+JACSIZE*10 + add x1, tab+JACSIZE*9 + add x2, tab + CFI_BL(Lp521_jscalarmul_jadd) + + add x0, tab+JACSIZE*11 + add x1, tab+JACSIZE*5 + CFI_BL(Lp521_jscalarmul_jdouble) + + add x0, tab+JACSIZE*12 + add x1, tab+JACSIZE*11 + add x2, tab + CFI_BL(Lp521_jscalarmul_jadd) + + add x0, tab+JACSIZE*13 + add x1, tab+JACSIZE*6 + CFI_BL(Lp521_jscalarmul_jdouble) + + add x0, tab+JACSIZE*14 + add x1, tab+JACSIZE*13 + add x2, tab + CFI_BL(Lp521_jscalarmul_jadd) + + add x0, tab+JACSIZE*15 + add x1, tab+JACSIZE*7 + CFI_BL(Lp521_jscalarmul_jdouble) + +// Add the recoding constant sum_i(16 * 32^i) to the scalar to allow signed +// digits. The digits of the constant, in lowest-to-highest order, are as +// follows; they are generated dynamically since none is a simple ARM load. +// +// 0x0842108421084210 +// 0x1084210842108421 +// 0x2108421084210842 +// 0x4210842108421084 +// 0x8421084210842108 +// 0x0842108421084210 +// 0x1084210842108421 +// 0x2108421084210842 +// 0x0000000000000084 + + ldp x0, x1, [scalarb] + ldp x2, x3, [scalarb+16] + ldp x4, x5, [scalarb+32] + ldp x6, x7, [scalarb+48] + ldr x8, [scalarb+64] + + movbig(x10, #0x1084, #0x2108, #0x4210, #0x8421) + adds x0, x0, x10, lsr #1 + adcs x1, x1, x10 + lsl x10, x10, #1 + adcs x2, x2, x10 + lsl x10, x10, #1 + adcs x3, x3, x10 + lsl x10, x10, #1 + adcs x4, x4, x10 + lsr x11, x10, #4 + adcs x5, x5, x11 + lsr x10, x10, #3 + adcs x6, x6, x10 + lsl x10, x10, #1 + adcs x7, x7, x10 + lsl x10, x10, #1 + and x10, x10, #0xFF + adc x8, x8, x10 + +// Because of the initial reduction the top bitfield (>= bits 520) is <= 1, +// i.e. just a single bit. Record that in "bf", then shift the whole +// scalar left 56 bits to align the top of the next bitfield with the MSB +// (bits 571..575). + + lsr bf, x8, #8 + extr x8, x8, x7, #8 + extr x7, x7, x6, #8 + extr x6, x6, x5, #8 + extr x5, x5, x4, #8 + extr x4, x4, x3, #8 + extr x3, x3, x2, #8 + extr x2, x2, x1, #8 + extr x1, x1, x0, #8 + lsl x0, x0, #56 + stp x0, x1, [scalarb] + stp x2, x3, [scalarb+16] + stp x4, x5, [scalarb+32] + stp x6, x7, [scalarb+48] + str x8, [scalarb+64] + +// According to the top bit, initialize the accumulator to P or 0. This top +// digit, uniquely, is not recoded so there is no sign adjustment to make. +// We only really need to adjust the z coordinate to zero, but do all three. + + add tabup, tab + cmp bf, xzr + + ldp x0, x1, [tabup] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc] + ldp x0, x1, [tabup, #16] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+16] + ldp x0, x1, [tabup, #32] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+32] + ldp x0, x1, [tabup, #48] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+48] + ldp x0, x1, [tabup, #64] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+64] + ldp x0, x1, [tabup, #80] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+80] + ldp x0, x1, [tabup, #96] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+96] + ldp x0, x1, [tabup, #112] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+112] + ldp x0, x1, [tabup, #128] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+128] + ldp x0, x1, [tabup, #144] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+144] + ldp x0, x1, [tabup, #160] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+160] + ldp x0, x1, [tabup, #176] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+176] + ldp x0, x1, [tabup, #192] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+192] + ldr x0, [tabup, #208] + csel x0, x0, xzr, ne + str x0, [acc+208] + +// Main loop over size-5 bitfields: double 5 times then add signed digit +// At each stage we shift the scalar left by 5 bits so we can simply pick +// the top 5 bits as the bitfield, saving some fiddle over indexing. + + mov j, #520 + +Lp521_jscalarmul_mainloop: + sub j, j, #5 + + add x0, acc + add x1, acc + CFI_BL(Lp521_jscalarmul_jdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp521_jscalarmul_jdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp521_jscalarmul_jdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp521_jscalarmul_jdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp521_jscalarmul_jdouble) + +// Choose the bitfield and adjust it to sign and magnitude + + ldp x0, x1, [scalarb] + ldp x2, x3, [scalarb+16] + ldp x4, x5, [scalarb+32] + ldp x6, x7, [scalarb+48] + ldr x8, [scalarb+64] + lsr bf, x8, #59 + extr x8, x8, x7, #59 + extr x7, x7, x6, #59 + extr x6, x6, x5, #59 + extr x5, x5, x4, #59 + extr x4, x4, x3, #59 + extr x3, x3, x2, #59 + extr x2, x2, x1, #59 + extr x1, x1, x0, #59 + lsl x0, x0, #5 + stp x0, x1, [scalarb] + stp x2, x3, [scalarb+16] + stp x4, x5, [scalarb+32] + stp x6, x7, [scalarb+48] + str x8, [scalarb+64] + + subs bf, bf, #16 + csetm sgn, lo // sgn = sign of digit (1 = negative) + cneg bf, bf, lo // bf = absolute value of digit + +// Conditionally select the table entry tab[i-1] = i * P in constant time + + mov x0, xzr + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, xzr + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + add tabup, tab + selectblock(1) + selectblock(2) + selectblock(3) + selectblock(4) + selectblock(5) + selectblock(6) + selectblock(7) + selectblock(8) + selectblock(9) + selectblock(10) + selectblock(11) + selectblock(12) + selectblock(13) + selectblock(14) + selectblock(15) + selectblock(16) + stp x0, x1, [tabent] + stp x2, x3, [tabent+16] + stp x4, x5, [tabent+32] + stp x6, x7, [tabent+48] + str x8, [tabent+64] + + mov x0, xzr + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, xzr + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + add tabup, tab+2*NUMSIZE + selectblock(1) + selectblock(2) + selectblock(3) + selectblock(4) + selectblock(5) + selectblock(6) + selectblock(7) + selectblock(8) + selectblock(9) + selectblock(10) + selectblock(11) + selectblock(12) + selectblock(13) + selectblock(14) + selectblock(15) + selectblock(16) + stp x0, x1, [tabent+2*NUMSIZE] + stp x2, x3, [tabent+2*NUMSIZE+16] + stp x4, x5, [tabent+2*NUMSIZE+32] + stp x6, x7, [tabent+2*NUMSIZE+48] + str x8, [tabent+2*NUMSIZE+64] + + mov x0, xzr + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, xzr + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + add tabup, tab+NUMSIZE + selectblock(1) + selectblock(2) + selectblock(3) + selectblock(4) + selectblock(5) + selectblock(6) + selectblock(7) + selectblock(8) + selectblock(9) + selectblock(10) + selectblock(11) + selectblock(12) + selectblock(13) + selectblock(14) + selectblock(15) + selectblock(16) + +// Store it to "tabent" with the y coordinate optionally negated. +// This is done carefully to give coordinates < p_521 even in +// the degenerate case y = 0 (when z = 0 for points on the curve). + + orr x10, x0, x1 + orr x11, x2, x3 + orr x12, x4, x5 + orr x13, x6, x7 + orr x10, x10, x11 + orr x12, x12, x13 + orr x12, x12, x8 + orr x10, x10, x12 + cmp x10, xzr + csel sgn, sgn, xzr, ne + + eor x0, x0, sgn + eor x1, x1, sgn + eor x2, x2, sgn + eor x3, x3, sgn + eor x4, x4, sgn + eor x5, x5, sgn + eor x6, x6, sgn + eor x7, x7, sgn + and sgn, sgn, #0x1FF + eor x8, x8, sgn + + stp x0, x1, [tabent+NUMSIZE] + stp x2, x3, [tabent+NUMSIZE+16] + stp x4, x5, [tabent+NUMSIZE+32] + stp x6, x7, [tabent+NUMSIZE+48] + str x8, [tabent+NUMSIZE+64] + +// Add to the accumulator + + add x0, acc + add x1, acc + add x2, tabent + CFI_BL(Lp521_jscalarmul_jadd) + + cbnz j, Lp521_jscalarmul_mainloop + +// That's the end of the main loop, and we just need to copy the +// result in "acc" to the output. + + ldp x0, x1, [acc] + stp x0, x1, [res] + ldp x0, x1, [acc+16] + stp x0, x1, [res, #16] + ldp x0, x1, [acc+32] + stp x0, x1, [res, #32] + ldp x0, x1, [acc+48] + stp x0, x1, [res, #48] + ldp x0, x1, [acc+64] + stp x0, x1, [res, #64] + ldp x0, x1, [acc+80] + stp x0, x1, [res, #80] + ldp x0, x1, [acc+96] + stp x0, x1, [res, #96] + ldp x0, x1, [acc+112] + stp x0, x1, [res, #112] + ldp x0, x1, [acc+128] + stp x0, x1, [res, #128] + ldp x0, x1, [acc+144] + stp x0, x1, [res, #144] + ldp x0, x1, [acc+160] + stp x0, x1, [res, #160] + ldp x0, x1, [acc+176] + stp x0, x1, [res, #176] + ldp x0, x1, [acc+192] + stp x0, x1, [res, #192] + ldr x0, [acc+208] + str x0, [res, #208] + +// Restore stack and registers and return + + CFI_INC_SP(NSPACE) + CFI_POP2(x21,x30) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p521_jscalarmul) + +// Local copies of subroutines, complete clones at the moment except +// that we share multiplication and squaring between the point operations. + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_bignum_mod_p521_9) + +Lp521_jscalarmul_bignum_mod_p521_9: + CFI_START + ldr x12, [x1, #64] + lsr x2, x12, #9 + cmp xzr, xzr + ldp x4, x5, [x1] + adcs xzr, x4, x2 + adcs xzr, x5, xzr + ldp x6, x7, [x1, #16] + and x3, x6, x7 + adcs xzr, x3, xzr + ldp x8, x9, [x1, #32] + and x3, x8, x9 + adcs xzr, x3, xzr + ldp x10, x11, [x1, #48] + and x3, x10, x11 + adcs xzr, x3, xzr + orr x3, x12, #0xfffffffffffffe00 + adcs x3, x3, xzr + adcs x4, x4, x2 + adcs x5, x5, xzr + adcs x6, x6, xzr + adcs x7, x7, xzr + adcs x8, x8, xzr + adcs x9, x9, xzr + adcs x10, x10, xzr + adcs x11, x11, xzr + adc x12, x12, xzr + and x12, x12, #0x1ff + stp x4, x5, [x0] + stp x6, x7, [x0, #16] + stp x8, x9, [x0, #32] + stp x10, x11, [x0, #48] + str x12, [x0, #64] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_bignum_mod_p521_9) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_bignum_mod_n521_9) + +Lp521_jscalarmul_bignum_mod_n521_9: + CFI_START + ldr x14, [x1, #64] + lsr x15, x14, #9 + add x15, x15, #1 + mov x2, #39927 + movk x2, #28359, lsl #16 + movk x2, #18657, lsl #32 + movk x2, #17552, lsl #48 + mul x6, x2, x15 + mov x3, #47185 + movk x3, #30307, lsl #16 + movk x3, #13895, lsl #32 + movk x3, #50250, lsl #48 + mul x7, x3, x15 + mov x4, #23087 + movk x4, #2294, lsl #16 + movk x4, #65207, lsl #32 + movk x4, #32819, lsl #48 + mul x8, x4, x15 + mov x5, #27028 + movk x5, #16592, lsl #16 + movk x5, #30844, lsl #32 + movk x5, #44665, lsl #48 + mul x9, x5, x15 + lsl x10, x15, #2 + add x10, x10, x15 + umulh x13, x2, x15 + adds x7, x7, x13 + umulh x13, x3, x15 + adcs x8, x8, x13 + umulh x13, x4, x15 + adcs x9, x9, x13 + umulh x13, x5, x15 + adc x10, x10, x13 + ldp x12, x13, [x1] + adds x6, x6, x12 + adcs x7, x7, x13 + ldp x12, x13, [x1, #16] + adcs x8, x8, x12 + adcs x9, x9, x13 + ldp x13, x11, [x1, #32] + adcs x10, x10, x13 + adcs x11, x11, xzr + ldp x12, x13, [x1, #48] + adcs x12, x12, xzr + adcs x13, x13, xzr + orr x14, x14, #0xfffffffffffffe00 + adcs x14, x14, xzr + csetm x15, lo + and x2, x2, x15 + subs x6, x6, x2 + and x3, x3, x15 + sbcs x7, x7, x3 + and x4, x4, x15 + sbcs x8, x8, x4 + and x5, x5, x15 + sbcs x9, x9, x5 + mov x2, #5 + and x2, x2, x15 + sbcs x10, x10, x2 + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + and x14, x14, #0x1ff + stp x6, x7, [x0] + stp x8, x9, [x0, #16] + stp x10, x11, [x0, #32] + stp x12, x13, [x0, #48] + str x14, [x0, #64] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_bignum_mod_n521_9) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_jadd) + +Lp521_jscalarmul_jadd: + CFI_START + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x26) + CFI_PUSH2(x27,x28) + CFI_PUSH2(x29,x30) + CFI_DEC_SP(576) + mov x26, x0 + mov x27, x1 + mov x28, x2 + mov x0, sp + add x1, x27, #0x90 + CFI_BL(Lp521_jscalarmul_sqr_p521) + add x0, sp, #0x168 + add x1, x28, #0x90 + CFI_BL(Lp521_jscalarmul_sqr_p521) + add x0, sp, #0x1f8 + add x1, x28, #0x90 + add x2, x27, #0x48 + CFI_BL(Lp521_jscalarmul_mul_p521) + add x0, sp, #0x48 + add x1, x27, #0x90 + add x2, x28, #0x48 + CFI_BL(Lp521_jscalarmul_mul_p521) + add x0, sp, #0x90 + mov x1, sp + add x2, x28, #0x0 + CFI_BL(Lp521_jscalarmul_mul_p521) + add x0, sp, #0x120 + add x1, sp, #0x168 + add x2, x27, #0x0 + CFI_BL(Lp521_jscalarmul_mul_p521) + add x0, sp, #0x48 + mov x1, sp + add x2, sp, #0x48 + CFI_BL(Lp521_jscalarmul_mul_p521) + add x0, sp, #0x1f8 + add x1, sp, #0x168 + add x2, sp, #0x1f8 + CFI_BL(Lp521_jscalarmul_mul_p521) + add x0, sp, #0x168 + add x1, sp, #0x90 + add x2, sp, #0x120 + CFI_BL(Lp521_jscalarmul_sub_p521) + add x0, sp, #0x48 + add x1, sp, #0x48 + add x2, sp, #0x1f8 + CFI_BL(Lp521_jscalarmul_sub_p521) + add x0, sp, #0xd8 + add x1, sp, #0x168 + CFI_BL(Lp521_jscalarmul_sqr_p521) + mov x0, sp + add x1, sp, #0x48 + CFI_BL(Lp521_jscalarmul_sqr_p521) + add x0, sp, #0x120 + add x1, sp, #0xd8 + add x2, sp, #0x120 + CFI_BL(Lp521_jscalarmul_mul_p521) + add x0, sp, #0x90 + add x1, sp, #0xd8 + add x2, sp, #0x90 + CFI_BL(Lp521_jscalarmul_mul_p521) + mov x0, sp + mov x1, sp + add x2, sp, #0x120 + CFI_BL(Lp521_jscalarmul_sub_p521) + add x0, sp, #0xd8 + add x1, sp, #0x90 + add x2, sp, #0x120 + CFI_BL(Lp521_jscalarmul_sub_p521) + add x0, sp, #0x168 + add x1, sp, #0x168 + add x2, x27, #0x90 + CFI_BL(Lp521_jscalarmul_mul_p521) + mov x0, sp + mov x1, sp + add x2, sp, #0x90 + CFI_BL(Lp521_jscalarmul_sub_p521) + add x0, sp, #0x120 + add x1, sp, #0x120 + mov x2, sp + CFI_BL(Lp521_jscalarmul_sub_p521) + add x0, sp, #0xd8 + add x1, sp, #0xd8 + add x2, sp, #0x1f8 + CFI_BL(Lp521_jscalarmul_mul_p521) + add x0, sp, #0x168 + add x1, sp, #0x168 + add x2, x28, #0x90 + CFI_BL(Lp521_jscalarmul_mul_p521) + add x0, sp, #0x120 + add x1, sp, #0x48 + add x2, sp, #0x120 + CFI_BL(Lp521_jscalarmul_mul_p521) + add x0, sp, #0x120 + add x1, sp, #0x120 + add x2, sp, #0xd8 + CFI_BL(Lp521_jscalarmul_sub_p521) + ldp x0, x1, [x27, #144] + ldp x2, x3, [x27, #160] + ldp x4, x5, [x27, #176] + ldp x6, x7, [x27, #192] + ldr x8, [x27, #208] + orr x20, x0, x1 + orr x21, x2, x3 + orr x22, x4, x5 + orr x23, x6, x7 + orr x20, x20, x21 + orr x22, x22, x23 + orr x20, x20, x8 + orr x20, x20, x22 + cmp x20, xzr + cset x20, ne + ldp x10, x11, [x28, #144] + ldp x12, x13, [x28, #160] + ldp x14, x15, [x28, #176] + ldp x16, x17, [x28, #192] + ldr x19, [x28, #208] + orr x21, x10, x11 + orr x22, x12, x13 + orr x23, x14, x15 + orr x24, x16, x17 + orr x21, x21, x22 + orr x23, x23, x24 + orr x21, x21, x19 + orr x21, x21, x23 + csel x0, x0, x10, ne + csel x1, x1, x11, ne + csel x2, x2, x12, ne + csel x3, x3, x13, ne + csel x4, x4, x14, ne + csel x5, x5, x15, ne + csel x6, x6, x16, ne + csel x7, x7, x17, ne + csel x8, x8, x19, ne + cmp x21, xzr + cset x21, ne + cmp x21, x20 + ldp x10, x11, [sp, #360] + ldp x12, x13, [sp, #376] + ldp x14, x15, [sp, #392] + ldp x16, x17, [sp, #408] + ldr x19, [sp, #424] + csel x0, x0, x10, ne + csel x1, x1, x11, ne + csel x2, x2, x12, ne + csel x3, x3, x13, ne + csel x4, x4, x14, ne + csel x5, x5, x15, ne + csel x6, x6, x16, ne + csel x7, x7, x17, ne + csel x8, x8, x19, ne + stp x0, x1, [sp, #360] + stp x2, x3, [sp, #376] + stp x4, x5, [sp, #392] + stp x6, x7, [sp, #408] + str x8, [sp, #424] + ldp x20, x21, [x27] + ldp x0, x1, [sp] + csel x0, x20, x0, cc + csel x1, x21, x1, cc + ldp x20, x21, [x28] + csel x0, x20, x0, hi + csel x1, x21, x1, hi + ldp x20, x21, [x27, #16] + ldp x2, x3, [sp, #16] + csel x2, x20, x2, cc + csel x3, x21, x3, cc + ldp x20, x21, [x28, #16] + csel x2, x20, x2, hi + csel x3, x21, x3, hi + ldp x20, x21, [x27, #32] + ldp x4, x5, [sp, #32] + csel x4, x20, x4, cc + csel x5, x21, x5, cc + ldp x20, x21, [x28, #32] + csel x4, x20, x4, hi + csel x5, x21, x5, hi + ldp x20, x21, [x27, #48] + ldp x6, x7, [sp, #48] + csel x6, x20, x6, cc + csel x7, x21, x7, cc + ldp x20, x21, [x28, #48] + csel x6, x20, x6, hi + csel x7, x21, x7, hi + ldr x20, [x27, #64] + ldr x8, [sp, #64] + csel x8, x20, x8, cc + ldr x21, [x28, #64] + csel x8, x21, x8, hi + ldp x20, x21, [x27, #72] + ldp x10, x11, [sp, #288] + csel x10, x20, x10, cc + csel x11, x21, x11, cc + ldp x20, x21, [x28, #72] + csel x10, x20, x10, hi + csel x11, x21, x11, hi + ldp x20, x21, [x27, #88] + ldp x12, x13, [sp, #304] + csel x12, x20, x12, cc + csel x13, x21, x13, cc + ldp x20, x21, [x28, #88] + csel x12, x20, x12, hi + csel x13, x21, x13, hi + ldp x20, x21, [x27, #104] + ldp x14, x15, [sp, #320] + csel x14, x20, x14, cc + csel x15, x21, x15, cc + ldp x20, x21, [x28, #104] + csel x14, x20, x14, hi + csel x15, x21, x15, hi + ldp x20, x21, [x27, #120] + ldp x16, x17, [sp, #336] + csel x16, x20, x16, cc + csel x17, x21, x17, cc + ldp x20, x21, [x28, #120] + csel x16, x20, x16, hi + csel x17, x21, x17, hi + ldr x20, [x27, #136] + ldr x19, [sp, #352] + csel x19, x20, x19, cc + ldr x21, [x28, #136] + csel x19, x21, x19, hi + stp x0, x1, [x26] + stp x2, x3, [x26, #16] + stp x4, x5, [x26, #32] + stp x6, x7, [x26, #48] + str x8, [x26, #64] + ldp x0, x1, [sp, #360] + ldp x2, x3, [sp, #376] + ldp x4, x5, [sp, #392] + ldp x6, x7, [sp, #408] + ldr x8, [sp, #424] + stp x10, x11, [x26, #72] + stp x12, x13, [x26, #88] + stp x14, x15, [x26, #104] + stp x16, x17, [x26, #120] + str x19, [x26, #136] + stp x0, x1, [x26, #144] + stp x2, x3, [x26, #160] + stp x4, x5, [x26, #176] + stp x6, x7, [x26, #192] + str x8, [x26, #208] + CFI_INC_SP(576) + CFI_POP2(x29,x30) + CFI_POP2(x27,x28) + CFI_POP2(x25,x26) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_jadd) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_jdouble) + +Lp521_jscalarmul_jdouble: + CFI_START + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x26) + CFI_PUSH2(x27,x28) + CFI_PUSH2(x29,x30) + CFI_DEC_SP(512) + mov x26, x0 + mov x27, x1 + mov x0, sp + add x1, x27, #0x90 + CFI_BL(Lp521_jscalarmul_sqr_p521) + add x0, sp, #0x48 + add x1, x27, #0x48 + CFI_BL(Lp521_jscalarmul_sqr_p521) + ldp x5, x6, [x27] + ldp x4, x3, [sp] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [x27, #16] + ldp x4, x3, [sp, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [x27, #32] + ldp x4, x3, [sp, #32] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + ldp x11, x12, [x27, #48] + ldp x4, x3, [sp, #48] + sbcs x11, x11, x4 + sbcs x12, x12, x3 + ldr x13, [x27, #64] + ldr x4, [sp, #64] + sbcs x13, x13, x4 + sbcs x5, x5, xzr + sbcs x6, x6, xzr + sbcs x7, x7, xzr + sbcs x8, x8, xzr + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbcs x13, x13, xzr + and x13, x13, #0x1ff + stp x5, x6, [sp, #216] + stp x7, x8, [sp, #232] + stp x9, x10, [sp, #248] + stp x11, x12, [sp, #264] + str x13, [sp, #280] + cmp xzr, xzr + ldp x5, x6, [x27] + ldp x4, x3, [sp] + adcs x5, x5, x4 + adcs x6, x6, x3 + ldp x7, x8, [x27, #16] + ldp x4, x3, [sp, #16] + adcs x7, x7, x4 + adcs x8, x8, x3 + ldp x9, x10, [x27, #32] + ldp x4, x3, [sp, #32] + adcs x9, x9, x4 + adcs x10, x10, x3 + ldp x11, x12, [x27, #48] + ldp x4, x3, [sp, #48] + adcs x11, x11, x4 + adcs x12, x12, x3 + ldr x13, [x27, #64] + ldr x4, [sp, #64] + adc x13, x13, x4 + subs x4, x13, #0x200 + csetm x4, cs + sbcs x5, x5, xzr + and x4, x4, #0x200 + sbcs x6, x6, xzr + sbcs x7, x7, xzr + sbcs x8, x8, xzr + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbc x13, x13, x4 + stp x5, x6, [sp, #144] + stp x7, x8, [sp, #160] + stp x9, x10, [sp, #176] + stp x11, x12, [sp, #192] + str x13, [sp, #208] + add x0, sp, #0xd8 + add x1, sp, #0x90 + add x2, sp, #0xd8 + CFI_BL(Lp521_jscalarmul_mul_p521) + cmp xzr, xzr + ldp x5, x6, [x27, #72] + ldp x4, x3, [x27, #144] + adcs x5, x5, x4 + adcs x6, x6, x3 + ldp x7, x8, [x27, #88] + ldp x4, x3, [x27, #160] + adcs x7, x7, x4 + adcs x8, x8, x3 + ldp x9, x10, [x27, #104] + ldp x4, x3, [x27, #176] + adcs x9, x9, x4 + adcs x10, x10, x3 + ldp x11, x12, [x27, #120] + ldp x4, x3, [x27, #192] + adcs x11, x11, x4 + adcs x12, x12, x3 + ldr x13, [x27, #136] + ldr x4, [x27, #208] + adc x13, x13, x4 + subs x4, x13, #0x200 + csetm x4, cs + sbcs x5, x5, xzr + and x4, x4, #0x200 + sbcs x6, x6, xzr + sbcs x7, x7, xzr + sbcs x8, x8, xzr + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbc x13, x13, x4 + stp x5, x6, [sp, #144] + stp x7, x8, [sp, #160] + stp x9, x10, [sp, #176] + stp x11, x12, [sp, #192] + str x13, [sp, #208] + add x0, sp, #0x120 + add x1, x27, #0x0 + add x2, sp, #0x48 + CFI_BL(Lp521_jscalarmul_mul_p521) + add x0, sp, #0x168 + add x1, sp, #0xd8 + CFI_BL(Lp521_jscalarmul_sqr_p521) + add x0, sp, #0x90 + add x1, sp, #0x90 + CFI_BL(Lp521_jscalarmul_sqr_p521) + ldp x6, x7, [sp, #288] + mov x1, #0xc + mul x3, x1, x6 + mul x4, x1, x7 + umulh x6, x1, x6 + adds x4, x4, x6 + umulh x7, x1, x7 + ldp x8, x9, [sp, #304] + mul x5, x1, x8 + mul x6, x1, x9 + umulh x8, x1, x8 + adcs x5, x5, x7 + umulh x9, x1, x9 + adcs x6, x6, x8 + ldp x10, x11, [sp, #320] + mul x7, x1, x10 + mul x8, x1, x11 + umulh x10, x1, x10 + adcs x7, x7, x9 + umulh x11, x1, x11 + adcs x8, x8, x10 + ldp x12, x13, [sp, #336] + mul x9, x1, x12 + mul x10, x1, x13 + umulh x12, x1, x12 + adcs x9, x9, x11 + umulh x13, x1, x13 + adcs x10, x10, x12 + ldr x14, [sp, #352] + mul x11, x1, x14 + adc x11, x11, x13 + mov x1, #0x9 + ldp x20, x21, [sp, #360] + mvn x20, x20 + mul x0, x1, x20 + umulh x20, x1, x20 + adds x3, x3, x0 + mvn x21, x21 + mul x0, x1, x21 + umulh x21, x1, x21 + adcs x4, x4, x0 + ldp x22, x23, [sp, #376] + mvn x22, x22 + mul x0, x1, x22 + umulh x22, x1, x22 + adcs x5, x5, x0 + mvn x23, x23 + mul x0, x1, x23 + umulh x23, x1, x23 + adcs x6, x6, x0 + ldp x17, x19, [sp, #392] + mvn x17, x17 + mul x0, x1, x17 + umulh x17, x1, x17 + adcs x7, x7, x0 + mvn x19, x19 + mul x0, x1, x19 + umulh x19, x1, x19 + adcs x8, x8, x0 + ldp x2, x16, [sp, #408] + mvn x2, x2 + mul x0, x1, x2 + umulh x2, x1, x2 + adcs x9, x9, x0 + mvn x16, x16 + mul x0, x1, x16 + umulh x16, x1, x16 + adcs x10, x10, x0 + ldr x0, [sp, #424] + eor x0, x0, #0x1ff + mul x0, x1, x0 + adc x11, x11, x0 + adds x4, x4, x20 + adcs x5, x5, x21 + and x15, x4, x5 + adcs x6, x6, x22 + and x15, x15, x6 + adcs x7, x7, x23 + and x15, x15, x7 + adcs x8, x8, x17 + and x15, x15, x8 + adcs x9, x9, x19 + and x15, x15, x9 + adcs x10, x10, x2 + and x15, x15, x10 + adc x11, x11, x16 + lsr x12, x11, #9 + orr x11, x11, #0xfffffffffffffe00 + cmp xzr, xzr + adcs xzr, x3, x12 + adcs xzr, x15, xzr + adcs xzr, x11, xzr + adcs x3, x3, x12 + adcs x4, x4, xzr + adcs x5, x5, xzr + adcs x6, x6, xzr + adcs x7, x7, xzr + adcs x8, x8, xzr + adcs x9, x9, xzr + adcs x10, x10, xzr + adc x11, x11, xzr + and x11, x11, #0x1ff + stp x3, x4, [sp, #360] + stp x5, x6, [sp, #376] + stp x7, x8, [sp, #392] + stp x9, x10, [sp, #408] + str x11, [sp, #424] + ldp x5, x6, [sp, #144] + ldp x4, x3, [sp] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #160] + ldp x4, x3, [sp, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #176] + ldp x4, x3, [sp, #32] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + ldp x11, x12, [sp, #192] + ldp x4, x3, [sp, #48] + sbcs x11, x11, x4 + sbcs x12, x12, x3 + ldr x13, [sp, #208] + ldr x4, [sp, #64] + sbcs x13, x13, x4 + sbcs x5, x5, xzr + sbcs x6, x6, xzr + sbcs x7, x7, xzr + sbcs x8, x8, xzr + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbcs x13, x13, xzr + and x13, x13, #0x1ff + stp x5, x6, [sp, #144] + stp x7, x8, [sp, #160] + stp x9, x10, [sp, #176] + stp x11, x12, [sp, #192] + str x13, [sp, #208] + mov x0, sp + add x1, sp, #0x48 + CFI_BL(Lp521_jscalarmul_sqr_p521) + add x0, sp, #0xd8 + add x1, sp, #0x168 + add x2, sp, #0xd8 + CFI_BL(Lp521_jscalarmul_mul_p521) + ldp x5, x6, [sp, #144] + ldp x4, x3, [sp, #72] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #160] + ldp x4, x3, [sp, #88] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #176] + ldp x4, x3, [sp, #104] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + ldp x11, x12, [sp, #192] + ldp x4, x3, [sp, #120] + sbcs x11, x11, x4 + sbcs x12, x12, x3 + ldr x13, [sp, #208] + ldr x4, [sp, #136] + sbcs x13, x13, x4 + sbcs x5, x5, xzr + sbcs x6, x6, xzr + sbcs x7, x7, xzr + sbcs x8, x8, xzr + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbcs x13, x13, xzr + and x13, x13, #0x1ff + stp x5, x6, [x26, #144] + stp x7, x8, [x26, #160] + stp x9, x10, [x26, #176] + stp x11, x12, [x26, #192] + str x13, [x26, #208] + ldp x6, x7, [sp, #288] + lsl x3, x6, #2 + extr x4, x7, x6, #62 + ldp x8, x9, [sp, #304] + extr x5, x8, x7, #62 + extr x6, x9, x8, #62 + ldp x10, x11, [sp, #320] + extr x7, x10, x9, #62 + extr x8, x11, x10, #62 + ldp x12, x13, [sp, #336] + extr x9, x12, x11, #62 + extr x10, x13, x12, #62 + ldr x14, [sp, #352] + extr x11, x14, x13, #62 + ldp x0, x1, [sp, #360] + mvn x0, x0 + adds x3, x3, x0 + sbcs x4, x4, x1 + ldp x0, x1, [sp, #376] + sbcs x5, x5, x0 + and x15, x4, x5 + sbcs x6, x6, x1 + and x15, x15, x6 + ldp x0, x1, [sp, #392] + sbcs x7, x7, x0 + and x15, x15, x7 + sbcs x8, x8, x1 + and x15, x15, x8 + ldp x0, x1, [sp, #408] + sbcs x9, x9, x0 + and x15, x15, x9 + sbcs x10, x10, x1 + and x15, x15, x10 + ldr x0, [sp, #424] + eor x0, x0, #0x1ff + adc x11, x11, x0 + lsr x12, x11, #9 + orr x11, x11, #0xfffffffffffffe00 + cmp xzr, xzr + adcs xzr, x3, x12 + adcs xzr, x15, xzr + adcs xzr, x11, xzr + adcs x3, x3, x12 + adcs x4, x4, xzr + adcs x5, x5, xzr + adcs x6, x6, xzr + adcs x7, x7, xzr + adcs x8, x8, xzr + adcs x9, x9, xzr + adcs x10, x10, xzr + adc x11, x11, xzr + and x11, x11, #0x1ff + stp x3, x4, [x26] + stp x5, x6, [x26, #16] + stp x7, x8, [x26, #32] + stp x9, x10, [x26, #48] + str x11, [x26, #64] + ldp x6, x7, [sp, #216] + lsl x3, x6, #1 + adds x3, x3, x6 + extr x4, x7, x6, #63 + adcs x4, x4, x7 + ldp x8, x9, [sp, #232] + extr x5, x8, x7, #63 + adcs x5, x5, x8 + extr x6, x9, x8, #63 + adcs x6, x6, x9 + ldp x10, x11, [sp, #248] + extr x7, x10, x9, #63 + adcs x7, x7, x10 + extr x8, x11, x10, #63 + adcs x8, x8, x11 + ldp x12, x13, [sp, #264] + extr x9, x12, x11, #63 + adcs x9, x9, x12 + extr x10, x13, x12, #63 + adcs x10, x10, x13 + ldr x14, [sp, #280] + extr x11, x14, x13, #63 + adc x11, x11, x14 + ldp x20, x21, [sp] + mvn x20, x20 + lsl x0, x20, #3 + adds x3, x3, x0 + mvn x21, x21 + extr x0, x21, x20, #61 + adcs x4, x4, x0 + ldp x22, x23, [sp, #16] + mvn x22, x22 + extr x0, x22, x21, #61 + adcs x5, x5, x0 + and x15, x4, x5 + mvn x23, x23 + extr x0, x23, x22, #61 + adcs x6, x6, x0 + and x15, x15, x6 + ldp x20, x21, [sp, #32] + mvn x20, x20 + extr x0, x20, x23, #61 + adcs x7, x7, x0 + and x15, x15, x7 + mvn x21, x21 + extr x0, x21, x20, #61 + adcs x8, x8, x0 + and x15, x15, x8 + ldp x22, x23, [sp, #48] + mvn x22, x22 + extr x0, x22, x21, #61 + adcs x9, x9, x0 + and x15, x15, x9 + mvn x23, x23 + extr x0, x23, x22, #61 + adcs x10, x10, x0 + and x15, x15, x10 + ldr x0, [sp, #64] + eor x0, x0, #0x1ff + extr x0, x0, x23, #61 + adc x11, x11, x0 + lsr x12, x11, #9 + orr x11, x11, #0xfffffffffffffe00 + cmp xzr, xzr + adcs xzr, x3, x12 + adcs xzr, x15, xzr + adcs xzr, x11, xzr + adcs x3, x3, x12 + adcs x4, x4, xzr + adcs x5, x5, xzr + adcs x6, x6, xzr + adcs x7, x7, xzr + adcs x8, x8, xzr + adcs x9, x9, xzr + adcs x10, x10, xzr + adc x11, x11, xzr + and x11, x11, #0x1ff + stp x3, x4, [x26, #72] + stp x5, x6, [x26, #88] + stp x7, x8, [x26, #104] + stp x9, x10, [x26, #120] + str x11, [x26, #136] + CFI_INC_SP(512) + CFI_POP2(x29,x30) + CFI_POP2(x27,x28) + CFI_POP2(x25,x26) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_jdouble) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_mul_p521) + +Lp521_jscalarmul_mul_p521: + CFI_START + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x26) + CFI_DEC_SP(80) + ldr q6, [x2] + ldp x10, x17, [x1, #16] + ldr q4, [x1] + ldr q16, [x2, #32] + ldp x5, x20, [x2, #16] + ldr q2, [x1, #32] + movi v31.2D, #0x00000000ffffffff + uzp2 v17.4S, v6.4S, v6.4S + rev64 v7.4S, v6.4S + ldp x15, x21, [x1] + xtn v25.2S, v6.2D + xtn v22.2S, v4.2D + subs x14, x10, x17 + mul v7.4S, v7.4S, v4.4S + csetm x8, cc + rev64 v3.4S, v16.4S + xtn v1.2S, v16.2D + ldp x13, x16, [x2] + mul x26, x10, x5 + uzp2 v16.4S, v16.4S, v16.4S + uaddlp v26.2D, v7.4S + cneg x4, x14, cc + subs x24, x15, x21 + xtn v5.2S, v2.2D + mul v28.4S, v3.4S, v2.4S + shl v26.2D, v26.2D, #32 + mul x22, x17, x20 + umull v20.2D, v22.2S, v25.2S + uzp2 v6.4S, v4.4S, v4.4S + umull v18.2D, v22.2S, v17.2S + uzp2 v4.4S, v2.4S, v2.4S + cneg x14, x24, cc + csetm x7, cc + umulh x11, x17, x20 + usra v18.2D, v20.2D, #32 + uaddlp v7.2D, v28.4S + subs x19, x16, x13 + umlal v26.2D, v22.2S, v25.2S + cneg x19, x19, cc + shl v28.2D, v7.2D, #32 + umull v7.2D, v5.2S, v1.2S + umull v30.2D, v5.2S, v16.2S + cinv x6, x7, cc + mul x25, x14, x19 + umlal v28.2D, v5.2S, v1.2S + umull v21.2D, v6.2S, v17.2S + umulh x14, x14, x19 + usra v30.2D, v7.2D, #32 + subs x9, x20, x5 + and v29.16B, v18.16B, v31.16B + cinv x23, x8, cc + mov x8, v26.d[1] + cneg x12, x9, cc + usra v21.2D, v18.2D, #32 + umlal v29.2D, v6.2S, v25.2S + mul x24, x4, x12 + umull v18.2D, v4.2S, v16.2S + movi v25.2D, #0x00000000ffffffff + eor x9, x14, x6 + and v7.16B, v30.16B, v25.16B + usra v21.2D, v29.2D, #32 + umulh x7, x10, x5 + usra v18.2D, v30.2D, #32 + umlal v7.2D, v4.2S, v1.2S + mov x19, v21.d[0] + umulh x3, x4, x12 + mov x14, v21.d[1] + usra v18.2D, v7.2D, #32 + adds x4, x8, x19 + mov x8, v26.d[0] + adcs x19, x26, x14 + adcs x14, x22, x7 + adc x12, x11, xzr + adds x11, x4, x8 + adcs x26, x19, x4 + adcs x22, x14, x19 + eor x4, x24, x23 + adcs x14, x12, x14 + eor x7, x25, x6 + adc x25, xzr, x12 + eor x19, x3, x23 + adds x3, x26, x8 + adcs x24, x22, x11 + adcs x12, x14, x26 + adcs x22, x25, x22 + adcs x26, xzr, x14 + adc x14, xzr, x25 + cmn x23, #0x1 + adcs x22, x22, x4 + adcs x19, x26, x19 + adc x25, x14, x23 + subs x14, x21, x17 + cneg x23, x14, cc + csetm x26, cc + subs x4, x20, x16 + cneg x14, x4, cc + cinv x4, x26, cc + cmn x6, #0x1 + adcs x11, x11, x7 + mul x7, x23, x14 + adcs x9, x3, x9 + adcs x26, x24, x6 + umulh x3, x23, x14 + adcs x14, x12, x6 + adcs x22, x22, x6 + adcs x12, x19, x6 + extr x24, x11, x8, #55 + adc x6, x25, x6 + subs x19, x15, x17 + csetm x17, cc + cneg x23, x19, cc + subs x19, x20, x13 + lsl x25, x8, #9 + eor x8, x7, x4 + cneg x20, x19, cc + umulh x7, x23, x20 + cinv x19, x17, cc + subs x17, x15, x10 + csetm x15, cc + stp x25, x24, [sp, #32] + cneg x24, x17, cc + mul x20, x23, x20 + subs x25, x5, x13 + cneg x13, x25, cc + cinv x15, x15, cc + mul x25, x24, x13 + subs x21, x21, x10 + csetm x23, cc + cneg x17, x21, cc + subs x21, x5, x16 + umulh x13, x24, x13 + cinv x10, x23, cc + cneg x23, x21, cc + cmn x4, #0x1 + adcs x14, x14, x8 + eor x21, x3, x4 + adcs x21, x22, x21 + eor x5, x20, x19 + adcs x24, x12, x4 + mul x12, x17, x23 + eor x8, x25, x15 + adc x25, x6, x4 + cmn x15, #0x1 + adcs x6, x9, x8 + ldp x20, x8, [x2, #48] + eor x9, x13, x15 + adcs x4, x26, x9 + umulh x26, x17, x23 + ldp x17, x13, [x1, #48] + adcs x9, x14, x15 + adcs x16, x21, x15 + adcs x14, x24, x15 + eor x21, x7, x19 + mul x23, x17, x20 + adc x24, x25, x15 + cmn x19, #0x1 + adcs x7, x4, x5 + adcs x9, x9, x21 + umulh x3, x13, x8 + adcs x16, x16, x19 + adcs x22, x14, x19 + eor x5, x12, x10 + adc x12, x24, x19 + cmn x10, #0x1 + adcs x19, x7, x5 + eor x14, x26, x10 + mov x7, v28.d[1] + adcs x24, x9, x14 + extr x4, x19, x6, #55 + umulh x15, x17, x20 + mov x14, v18.d[1] + lsr x9, x19, #55 + adcs x5, x16, x10 + mov x16, v18.d[0] + adcs x19, x22, x10 + str x9, [sp, #64] + extr x25, x6, x11, #55 + adc x21, x12, x10 + subs x26, x17, x13 + stp x25, x4, [sp, #48] + stp x19, x21, [sp, #16] + csetm x6, cc + cneg x4, x26, cc + mul x19, x13, x8 + subs x11, x8, x20 + stp x24, x5, [sp] + ldp x21, x10, [x1, #32] + cinv x12, x6, cc + cneg x6, x11, cc + mov x9, v28.d[0] + umulh x25, x4, x6 + adds x22, x7, x16 + ldp x16, x5, [x2, #32] + adcs x14, x23, x14 + adcs x11, x19, x15 + adc x24, x3, xzr + adds x3, x22, x9 + adcs x15, x14, x22 + mul x22, x4, x6 + adcs x6, x11, x14 + adcs x4, x24, x11 + eor x14, x25, x12 + adc x26, xzr, x24 + subs x7, x21, x10 + csetm x23, cc + cneg x19, x7, cc + subs x24, x5, x16 + cneg x11, x24, cc + cinv x7, x23, cc + adds x25, x15, x9 + eor x23, x22, x12 + adcs x22, x6, x3 + mul x24, x19, x11 + adcs x15, x4, x15 + adcs x6, x26, x6 + umulh x19, x19, x11 + adcs x11, xzr, x4 + adc x26, xzr, x26 + cmn x12, #0x1 + adcs x4, x6, x23 + eor x6, x24, x7 + adcs x14, x11, x14 + adc x26, x26, x12 + subs x11, x10, x13 + cneg x12, x11, cc + csetm x11, cc + eor x19, x19, x7 + subs x24, x8, x5 + cinv x11, x11, cc + cneg x24, x24, cc + cmn x7, #0x1 + adcs x3, x3, x6 + mul x23, x12, x24 + adcs x25, x25, x19 + adcs x6, x22, x7 + umulh x19, x12, x24 + adcs x22, x15, x7 + adcs x12, x4, x7 + eor x24, x23, x11 + adcs x4, x14, x7 + adc x26, x26, x7 + eor x19, x19, x11 + subs x14, x21, x17 + cneg x7, x14, cc + csetm x14, cc + subs x23, x20, x16 + cinv x14, x14, cc + cneg x23, x23, cc + cmn x11, #0x1 + adcs x22, x22, x24 + mul x24, x7, x23 + adcs x15, x12, x19 + adcs x4, x4, x11 + adc x19, x26, x11 + umulh x26, x7, x23 + subs x7, x21, x13 + eor x11, x24, x14 + cneg x23, x7, cc + csetm x12, cc + subs x7, x8, x16 + cneg x7, x7, cc + cinv x12, x12, cc + cmn x14, #0x1 + eor x26, x26, x14 + adcs x11, x25, x11 + mul x25, x23, x7 + adcs x26, x6, x26 + adcs x6, x22, x14 + adcs x24, x15, x14 + umulh x23, x23, x7 + adcs x4, x4, x14 + adc x22, x19, x14 + eor x14, x25, x12 + eor x7, x23, x12 + cmn x12, #0x1 + adcs x14, x26, x14 + ldp x19, x25, [x2] + ldp x15, x23, [x2, #16] + adcs x26, x6, x7 + adcs x24, x24, x12 + adcs x7, x4, x12 + adc x4, x22, x12 + subs x19, x19, x16 + ldp x16, x22, [x1] + sbcs x6, x25, x5 + ldp x12, x25, [x1, #16] + sbcs x15, x15, x20 + sbcs x8, x23, x8 + csetm x23, cc + subs x21, x21, x16 + eor x16, x19, x23 + sbcs x19, x10, x22 + eor x22, x6, x23 + eor x8, x8, x23 + sbcs x6, x17, x12 + sbcs x13, x13, x25 + csetm x12, cc + subs x10, x10, x17 + cneg x17, x10, cc + csetm x25, cc + subs x5, x20, x5 + eor x10, x19, x12 + cneg x19, x5, cc + eor x20, x15, x23 + eor x21, x21, x12 + cinv x15, x25, cc + mul x25, x17, x19 + subs x16, x16, x23 + sbcs x5, x22, x23 + eor x6, x6, x12 + sbcs x20, x20, x23 + eor x22, x13, x12 + sbc x8, x8, x23 + subs x21, x21, x12 + umulh x19, x17, x19 + sbcs x10, x10, x12 + sbcs x17, x6, x12 + eor x6, x19, x15 + eor x19, x25, x15 + umulh x25, x17, x20 + sbc x13, x22, x12 + cmn x15, #0x1 + adcs x22, x14, x19 + adcs x19, x26, x6 + ldp x6, x26, [sp] + adcs x14, x24, x15 + umulh x24, x21, x16 + adcs x7, x7, x15 + adc x15, x4, x15 + adds x4, x9, x6 + eor x9, x23, x12 + adcs x12, x3, x26 + stp x4, x12, [sp] + ldp x4, x26, [sp, #16] + umulh x12, x10, x5 + ldp x6, x23, [sp, #32] + adcs x3, x11, x4 + mul x4, x13, x8 + adcs x26, x22, x26 + ldp x22, x11, [sp, #48] + adcs x6, x19, x6 + stp x3, x26, [sp, #16] + mul x26, x10, x5 + adcs x14, x14, x23 + stp x6, x14, [sp, #32] + ldr x6, [sp, #64] + adcs x22, x7, x22 + adcs x14, x15, x11 + mul x11, x17, x20 + adc x19, x6, xzr + stp x22, x14, [sp, #48] + adds x14, x26, x24 + str x19, [sp, #64] + umulh x19, x13, x8 + adcs x7, x11, x12 + adcs x22, x4, x25 + mul x6, x21, x16 + adc x19, x19, xzr + subs x11, x17, x13 + cneg x12, x11, cc + csetm x11, cc + subs x24, x8, x20 + cinv x11, x11, cc + cneg x24, x24, cc + adds x4, x14, x6 + adcs x14, x7, x14 + mul x3, x12, x24 + adcs x7, x22, x7 + adcs x22, x19, x22 + umulh x12, x12, x24 + adc x24, xzr, x19 + adds x19, x14, x6 + eor x3, x3, x11 + adcs x26, x7, x4 + adcs x14, x22, x14 + adcs x25, x24, x7 + adcs x23, xzr, x22 + eor x7, x12, x11 + adc x12, xzr, x24 + subs x22, x21, x10 + cneg x24, x22, cc + csetm x22, cc + subs x15, x5, x16 + cinv x22, x22, cc + cneg x15, x15, cc + cmn x11, #0x1 + adcs x3, x25, x3 + mul x25, x24, x15 + adcs x23, x23, x7 + adc x11, x12, x11 + subs x7, x10, x13 + umulh x15, x24, x15 + cneg x12, x7, cc + csetm x7, cc + eor x24, x25, x22 + eor x25, x15, x22 + cmn x22, #0x1 + adcs x24, x4, x24 + adcs x19, x19, x25 + adcs x15, x26, x22 + adcs x4, x14, x22 + adcs x26, x3, x22 + adcs x25, x23, x22 + adc x23, x11, x22 + subs x14, x21, x17 + cneg x3, x14, cc + csetm x11, cc + subs x14, x8, x5 + cneg x14, x14, cc + cinv x7, x7, cc + subs x13, x21, x13 + cneg x21, x13, cc + csetm x13, cc + mul x22, x12, x14 + subs x8, x8, x16 + cinv x13, x13, cc + umulh x14, x12, x14 + cneg x12, x8, cc + subs x8, x20, x16 + cneg x8, x8, cc + cinv x16, x11, cc + eor x22, x22, x7 + cmn x7, #0x1 + eor x14, x14, x7 + adcs x4, x4, x22 + mul x11, x3, x8 + adcs x22, x26, x14 + adcs x14, x25, x7 + eor x25, x24, x9 + adc x26, x23, x7 + umulh x7, x3, x8 + subs x17, x10, x17 + cneg x24, x17, cc + eor x3, x11, x16 + csetm x11, cc + subs x20, x20, x5 + cneg x5, x20, cc + cinv x11, x11, cc + cmn x16, #0x1 + mul x17, x21, x12 + eor x8, x7, x16 + adcs x10, x19, x3 + and x19, x9, #0x1ff + adcs x20, x15, x8 + umulh x15, x21, x12 + eor x12, x10, x9 + eor x8, x6, x9 + adcs x6, x4, x16 + adcs x4, x22, x16 + adcs x21, x14, x16 + adc x7, x26, x16 + mul x10, x24, x5 + cmn x13, #0x1 + ldp x3, x14, [x1] + eor x17, x17, x13 + umulh x5, x24, x5 + adcs x20, x20, x17 + eor x17, x15, x13 + adcs x16, x6, x17 + eor x22, x10, x11 + adcs x23, x4, x13 + extr x10, x14, x3, #52 + and x26, x3, #0xfffffffffffff + adcs x24, x21, x13 + and x15, x10, #0xfffffffffffff + adc x6, x7, x13 + cmn x11, #0x1 + adcs x17, x20, x22 + eor x4, x5, x11 + ldp x21, x10, [sp] + adcs x7, x16, x4 + eor x16, x17, x9 + eor x13, x7, x9 + ldp x3, x17, [sp, #16] + adcs x7, x23, x11 + eor x23, x7, x9 + ldp x5, x22, [sp, #32] + adcs x7, x24, x11 + adc x24, x6, x11 + ldr x6, [x2, #64] + adds x20, x8, x21 + lsl x11, x20, #9 + eor x4, x7, x9 + orr x7, x11, x19 + eor x8, x24, x9 + adcs x11, x25, x10 + mul x26, x6, x26 + ldp x19, x24, [sp, #48] + adcs x12, x12, x3 + adcs x16, x16, x17 + adcs x9, x13, x5 + ldr x25, [sp, #64] + extr x20, x11, x20, #55 + adcs x13, x23, x22 + adcs x4, x4, x19 + extr x23, x12, x11, #55 + adcs x8, x8, x24 + adc x11, x25, xzr + adds x21, x9, x21 + extr x9, x16, x12, #55 + lsr x12, x16, #55 + adcs x10, x13, x10 + mul x15, x6, x15 + adcs x13, x4, x3 + ldp x16, x4, [x2] + ldr x3, [x1, #64] + adcs x17, x8, x17 + adcs x5, x5, x7 + adcs x20, x22, x20 + adcs x8, x19, x23 + and x22, x16, #0xfffffffffffff + ldp x19, x7, [x1, #16] + adcs x9, x24, x9 + extr x24, x4, x16, #52 + adc x16, x12, x25 + mul x22, x3, x22 + and x25, x24, #0xfffffffffffff + extr x14, x19, x14, #40 + and x12, x14, #0xfffffffffffff + extr x23, x7, x19, #28 + ldp x19, x24, [x2, #16] + mul x14, x3, x25 + and x23, x23, #0xfffffffffffff + add x22, x26, x22 + lsl x11, x11, #48 + lsr x26, x22, #52 + lsl x25, x22, #12 + mul x22, x6, x12 + extr x12, x19, x4, #40 + add x4, x15, x14 + mul x15, x6, x23 + add x4, x4, x26 + extr x23, x24, x19, #28 + ldp x14, x19, [x1, #32] + and x26, x12, #0xfffffffffffff + extr x12, x4, x25, #12 + and x25, x23, #0xfffffffffffff + adds x21, x21, x12 + mul x12, x3, x26 + extr x23, x14, x7, #16 + and x23, x23, #0xfffffffffffff + mul x7, x3, x25 + ldp x25, x26, [x2, #32] + add x12, x22, x12 + extr x22, x19, x14, #56 + mul x23, x6, x23 + lsr x14, x14, #4 + extr x24, x25, x24, #16 + add x7, x15, x7 + and x15, x24, #0xfffffffffffff + and x22, x22, #0xfffffffffffff + lsr x24, x4, #52 + mul x15, x3, x15 + and x14, x14, #0xfffffffffffff + add x12, x12, x24 + lsl x24, x4, #12 + lsr x4, x12, #52 + extr x24, x12, x24, #24 + adcs x10, x10, x24 + lsl x24, x12, #12 + add x12, x7, x4 + mul x22, x6, x22 + add x4, x23, x15 + extr x7, x12, x24, #36 + adcs x13, x13, x7 + lsl x15, x12, #12 + add x7, x4, x11 + lsr x24, x12, #52 + ldp x23, x11, [x2, #48] + add x4, x7, x24 + mul x12, x6, x14 + extr x7, x26, x25, #56 + extr x14, x4, x15, #48 + and x2, x7, #0xfffffffffffff + extr x24, x11, x23, #32 + ldp x15, x7, [x1, #48] + and x1, x24, #0xfffffffffffff + lsr x24, x4, #52 + mul x2, x3, x2 + extr x26, x23, x26, #44 + lsr x23, x25, #4 + and x23, x23, #0xfffffffffffff + and x25, x26, #0xfffffffffffff + extr x26, x7, x15, #32 + extr x19, x15, x19, #44 + mul x23, x3, x23 + and x15, x26, #0xfffffffffffff + lsl x26, x4, #12 + and x4, x19, #0xfffffffffffff + lsr x11, x11, #20 + mul x19, x6, x4 + adcs x17, x17, x14 + add x14, x22, x2 + add x22, x12, x23 + lsr x7, x7, #20 + add x22, x22, x24 + extr x2, x22, x26, #60 + mul x24, x3, x25 + lsr x22, x22, #52 + add x14, x14, x22 + lsl x22, x2, #8 + extr x22, x14, x22, #8 + lsl x2, x14, #12 + mul x1, x3, x1 + adcs x12, x5, x22 + mul x5, x6, x15 + and x26, x10, x13 + and x4, x26, x17 + add x23, x19, x24 + lsr x14, x14, #52 + mul x22, x3, x11 + add x11, x23, x14 + extr x25, x11, x2, #20 + lsl x19, x11, #12 + adcs x25, x20, x25 + and x14, x4, x12 + add x1, x5, x1 + and x14, x14, x25 + mul x15, x6, x7 + add x26, x15, x22 + mul x6, x6, x3 + lsr x22, x11, #52 + add x4, x1, x22 + lsr x1, x4, #52 + extr x3, x4, x19, #32 + lsl x15, x4, #12 + add x7, x26, x1 + adcs x23, x8, x3 + extr x20, x7, x15, #44 + and x3, x14, x23 + lsr x19, x7, #44 + adcs x7, x9, x20 + add x11, x6, x19 + adc x4, x16, x11 + lsr x14, x4, #9 + cmp xzr, xzr + and x15, x3, x7 + orr x3, x4, #0xfffffffffffffe00 + adcs xzr, x21, x14 + adcs xzr, x15, xzr + adcs xzr, x3, xzr + adcs x11, x21, x14 + and x14, x11, #0x1ff + adcs x1, x10, xzr + extr x10, x1, x11, #9 + str x14, [x0, #64] + adcs x14, x13, xzr + extr x11, x14, x1, #9 + adcs x1, x17, xzr + extr x4, x1, x14, #9 + stp x10, x11, [x0] + adcs x11, x12, xzr + extr x14, x11, x1, #9 + adcs x10, x25, xzr + extr x11, x10, x11, #9 + stp x4, x14, [x0, #16] + adcs x14, x23, xzr + extr x10, x14, x10, #9 + adcs x1, x7, xzr + stp x11, x10, [x0, #32] + extr x14, x1, x14, #9 + adc x10, x3, xzr + extr x26, x10, x1, #9 + stp x14, x26, [x0, #48] + CFI_INC_SP(80) + CFI_POP2(x25,x26) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_mul_p521) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_sqr_p521) + +Lp521_jscalarmul_sqr_p521: + CFI_START + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + ldr q23, [x1, #32] + ldp x9, x2, [x1, #32] + ldr q16, [x1, #32] + ldr q20, [x1, #48] + ldp x6, x13, [x1, #48] + rev64 v2.4S, v23.4S + mul x14, x9, x2 + ldr q31, [x1, #48] + subs x22, x9, x2 + uzp2 v26.4S, v23.4S, v23.4S + mul v30.4S, v2.4S, v16.4S + xtn v0.2S, v20.2D + csetm x12, cc + xtn v21.2S, v16.2D + xtn v23.2S, v23.2D + umulh x10, x9, x6 + rev64 v27.4S, v31.4S + umull v2.2D, v21.2S, v26.2S + cneg x23, x22, cc + uaddlp v25.2D, v30.4S + umull v18.2D, v21.2S, v23.2S + mul x22, x9, x6 + mul v6.4S, v27.4S, v20.4S + uzp2 v17.4S, v20.4S, v20.4S + shl v20.2D, v25.2D, #32 + uzp2 v27.4S, v31.4S, v31.4S + mul x16, x2, x13 + umlal v20.2D, v21.2S, v23.2S + usra v2.2D, v18.2D, #32 + adds x8, x22, x10 + umull v25.2D, v17.2S, v27.2S + xtn v31.2S, v31.2D + movi v1.2D, #0xffffffff + adc x3, x10, xzr + umulh x21, x2, x13 + uzp2 v21.4S, v16.4S, v16.4S + umull v18.2D, v0.2S, v27.2S + subs x19, x13, x6 + and v7.16B, v2.16B, v1.16B + umull v27.2D, v0.2S, v31.2S + cneg x20, x19, cc + movi v30.2D, #0xffffffff + umull v16.2D, v21.2S, v26.2S + umlal v7.2D, v21.2S, v23.2S + mul x19, x23, x20 + cinv x7, x12, cc + uaddlp v6.2D, v6.4S + eor x12, x19, x7 + adds x11, x8, x16 + umulh x10, x23, x20 + ldr q1, [x1] + usra v16.2D, v2.2D, #32 + adcs x19, x3, x21 + shl v2.2D, v6.2D, #32 + adc x20, x21, xzr + adds x17, x19, x16 + usra v18.2D, v27.2D, #32 + adc x19, x20, xzr + cmn x7, #0x1 + umlal v2.2D, v0.2S, v31.2S + umulh x16, x9, x2 + adcs x8, x11, x12 + usra v16.2D, v7.2D, #32 + ldr x12, [x1, #64] + eor x20, x10, x7 + umulh x10, x6, x13 + mov x23, v2.d[0] + mov x3, v2.d[1] + adcs x21, x17, x20 + usra v25.2D, v18.2D, #32 + and v23.16B, v18.16B, v30.16B + adc x7, x19, x7 + adds x22, x22, x22 + ldr q7, [x1, #16] + adcs x17, x8, x8 + umlal v23.2D, v17.2S, v31.2S + mov x19, v16.d[0] + mul x11, x12, x12 + ldr q4, [x1] + usra v25.2D, v23.2D, #32 + add x5, x12, x12 + adcs x15, x21, x21 + ldr q28, [x1] + mov x12, v20.d[1] + adcs x24, x7, x7 + mov x21, v16.d[1] + adc x4, xzr, xzr + adds x19, x19, x14 + ldr q18, [x1, #16] + xtn v26.2S, v1.2D + adcs x8, x12, x16 + adc x21, x21, xzr + adds x7, x19, x14 + xtn v23.2S, v7.2D + rev64 v21.4S, v28.4S + adcs x12, x8, x16 + ldp x20, x19, [x1] + mov x16, v25.d[1] + xtn v22.2S, v28.2D + adc x14, x21, xzr + adds x8, x22, x12 + uzp2 v24.4S, v28.4S, v28.4S + rev64 v28.4S, v18.4S + mul x12, x6, x13 + mul v16.4S, v21.4S, v1.4S + shrn v31.2S, v7.2D, #32 + adcs x22, x17, x14 + mov x14, v25.d[0] + and x21, x20, #0xfffffffffffff + umull v17.2D, v26.2S, v24.2S + ldr q2, [x1, #32] + adcs x17, x15, xzr + ldr q30, [x1, #48] + umull v7.2D, v26.2S, v22.2S + adcs x15, x24, xzr + ldr q0, [x1, #16] + movi v6.2D, #0xffffffff + adc x4, x4, xzr + adds x14, x14, x12 + uzp1 v27.4S, v18.4S, v4.4S + uzp2 v19.4S, v1.4S, v1.4S + adcs x24, x3, x10 + mul x3, x5, x21 + umull v29.2D, v23.2S, v31.2S + ldr q5, [x1] + adc x21, x16, xzr + adds x16, x14, x12 + extr x12, x19, x20, #52 + umull v18.2D, v19.2S, v24.2S + adcs x24, x24, x10 + and x10, x12, #0xfffffffffffff + ldp x14, x12, [x1, #16] + usra v17.2D, v7.2D, #32 + adc x21, x21, xzr + adds x23, x23, x17 + mul x17, x5, x10 + shl v21.2D, v29.2D, #33 + lsl x10, x3, #12 + lsr x1, x3, #52 + rev64 v29.4S, v2.4S + uaddlp v25.2D, v16.4S + add x17, x17, x1 + adcs x16, x16, x15 + extr x3, x14, x19, #40 + mov x15, v20.d[0] + extr x10, x17, x10, #12 + and x3, x3, #0xfffffffffffff + shl v3.2D, v25.2D, #32 + and v6.16B, v17.16B, v6.16B + mul x1, x5, x3 + usra v18.2D, v17.2D, #32 + adcs x3, x24, x4 + extr x4, x12, x14, #28 + umlal v6.2D, v19.2S, v22.2S + xtn v20.2S, v2.2D + umlal v3.2D, v26.2S, v22.2S + movi v26.2D, #0xffffffff + lsr x24, x17, #52 + and x4, x4, #0xfffffffffffff + uzp2 v19.4S, v2.4S, v2.4S + add x1, x1, x24 + mul x24, x5, x4 + lsl x4, x17, #12 + xtn v24.2S, v5.2D + extr x17, x1, x4, #24 + adc x21, x21, xzr + umlal v21.2D, v23.2S, v23.2S + adds x4, x15, x10 + lsl x10, x1, #12 + adcs x15, x7, x17 + mul v23.4S, v28.4S, v4.4S + and x7, x4, #0x1ff + lsr x17, x1, #52 + umulh x1, x19, x12 + uzp2 v17.4S, v5.4S, v5.4S + extr x4, x15, x4, #9 + add x24, x24, x17 + mul v29.4S, v29.4S, v5.4S + extr x17, x24, x10, #36 + extr x10, x9, x12, #16 + uzp1 v28.4S, v4.4S, v4.4S + adcs x17, x8, x17 + and x8, x10, #0xfffffffffffff + umull v16.2D, v24.2S, v20.2S + extr x10, x17, x15, #9 + mul x15, x5, x8 + stp x4, x10, [x0] + lsl x4, x24, #12 + lsr x8, x9, #4 + uaddlp v4.2D, v23.4S + and x8, x8, #0xfffffffffffff + umull v23.2D, v24.2S, v19.2S + mul x8, x5, x8 + extr x10, x2, x9, #56 + lsr x24, x24, #52 + and x10, x10, #0xfffffffffffff + add x15, x15, x24 + extr x4, x15, x4, #48 + mul x24, x5, x10 + lsr x10, x15, #52 + usra v23.2D, v16.2D, #32 + add x10, x8, x10 + shl v4.2D, v4.2D, #32 + adcs x22, x22, x4 + extr x4, x6, x2, #44 + lsl x15, x15, #12 + lsr x8, x10, #52 + extr x15, x10, x15, #60 + and x10, x4, #0xfffffffffffff + umlal v4.2D, v28.2S, v27.2S + add x8, x24, x8 + extr x4, x13, x6, #32 + mul x24, x5, x10 + uzp2 v16.4S, v30.4S, v30.4S + lsl x10, x15, #8 + rev64 v28.4S, v30.4S + and x15, x4, #0xfffffffffffff + extr x4, x8, x10, #8 + mul x10, x5, x15 + lsl x15, x8, #12 + adcs x23, x23, x4 + lsr x4, x8, #52 + lsr x8, x13, #20 + add x4, x24, x4 + mul x8, x5, x8 + lsr x24, x4, #52 + extr x15, x4, x15, #20 + lsl x4, x4, #12 + add x10, x10, x24 + adcs x15, x16, x15 + extr x4, x10, x4, #32 + umulh x5, x20, x14 + adcs x3, x3, x4 + usra v18.2D, v6.2D, #32 + lsl x16, x10, #12 + extr x24, x15, x23, #9 + lsr x10, x10, #52 + uzp2 v27.4S, v0.4S, v0.4S + add x8, x8, x10 + extr x10, x3, x15, #9 + extr x4, x22, x17, #9 + and v25.16B, v23.16B, v26.16B + lsr x17, x8, #44 + extr x15, x8, x16, #44 + extr x16, x23, x22, #9 + xtn v7.2S, v30.2D + mov x8, v4.d[0] + stp x24, x10, [x0, #32] + uaddlp v30.2D, v29.4S + stp x4, x16, [x0, #16] + umulh x24, x20, x19 + adcs x15, x21, x15 + adc x16, x11, x17 + subs x11, x20, x19 + xtn v5.2S, v0.2D + csetm x17, cc + extr x3, x15, x3, #9 + mov x22, v4.d[1] + cneg x21, x11, cc + subs x10, x12, x14 + mul v31.4S, v28.4S, v0.4S + cneg x10, x10, cc + cinv x11, x17, cc + shl v4.2D, v30.2D, #32 + umull v28.2D, v5.2S, v16.2S + extr x23, x16, x15, #9 + adds x4, x8, x5 + mul x17, x21, x10 + umull v22.2D, v5.2S, v7.2S + adc x15, x5, xzr + adds x4, x4, x22 + uaddlp v2.2D, v31.4S + lsr x5, x16, #9 + adcs x16, x15, x1 + mov x15, v18.d[0] + adc x1, x1, xzr + umulh x10, x21, x10 + adds x22, x16, x22 + umlal v4.2D, v24.2S, v20.2S + umull v30.2D, v27.2S, v16.2S + stp x3, x23, [x0, #48] + add x3, x7, x5 + adc x16, x1, xzr + usra v28.2D, v22.2D, #32 + mul x23, x20, x19 + eor x1, x17, x11 + cmn x11, #0x1 + mov x17, v18.d[1] + umull v18.2D, v17.2S, v19.2S + adcs x7, x4, x1 + eor x1, x10, x11 + umlal v25.2D, v17.2S, v20.2S + movi v16.2D, #0xffffffff + adcs x22, x22, x1 + usra v18.2D, v23.2D, #32 + umulh x4, x14, x14 + adc x1, x16, x11 + adds x10, x8, x8 + shl v23.2D, v2.2D, #32 + str x3, [x0, #64] + adcs x5, x7, x7 + and v16.16B, v28.16B, v16.16B + usra v30.2D, v28.2D, #32 + adcs x7, x22, x22 + mov x21, v3.d[1] + adcs x11, x1, x1 + umlal v16.2D, v27.2S, v7.2S + adc x22, xzr, xzr + adds x16, x15, x23 + mul x8, x14, x12 + umlal v23.2D, v5.2S, v7.2S + usra v18.2D, v25.2D, #32 + umulh x15, x14, x12 + adcs x21, x21, x24 + usra v30.2D, v16.2D, #32 + adc x1, x17, xzr + adds x3, x16, x23 + adcs x21, x21, x24 + adc x1, x1, xzr + adds x24, x10, x21 + umulh x21, x12, x12 + adcs x16, x5, x1 + adcs x10, x7, xzr + mov x17, v21.d[1] + adcs x23, x11, xzr + adc x5, x22, xzr + adds x1, x4, x8 + adcs x22, x17, x15 + ldp x17, x4, [x0] + mov x11, v21.d[0] + adc x21, x21, xzr + adds x1, x1, x8 + adcs x15, x22, x15 + adc x8, x21, xzr + adds x22, x11, x10 + mov x21, v3.d[0] + adcs x11, x1, x23 + ldp x1, x10, [x0, #16] + adcs x15, x15, x5 + adc x7, x8, xzr + adds x8, x17, x21 + mov x23, v4.d[1] + ldp x5, x21, [x0, #32] + adcs x17, x4, x3 + ldr x4, [x0, #64] + mov x3, v18.d[0] + adcs x24, x1, x24 + stp x8, x17, [x0] + adcs x17, x10, x16 + ldp x1, x16, [x0, #48] + adcs x5, x5, x22 + adcs x8, x21, x11 + stp x5, x8, [x0, #32] + adcs x1, x1, x15 + mov x15, v23.d[1] + adcs x21, x16, x7 + stp x1, x21, [x0, #48] + adc x10, x4, xzr + subs x7, x14, x12 + mov x16, v18.d[1] + cneg x5, x7, cc + csetm x4, cc + subs x11, x13, x6 + mov x8, v23.d[0] + cneg x7, x11, cc + cinv x21, x4, cc + mov x11, v30.d[0] + adds x4, x23, x3 + mul x22, x5, x7 + mov x23, v30.d[1] + adcs x8, x8, x16 + adcs x16, x15, x11 + adc x11, x23, xzr + umulh x3, x5, x7 + stp x24, x17, [x0, #16] + mov x5, v4.d[0] + subs x15, x20, x19 + cneg x7, x15, cc + str x10, [x0, #64] + csetm x1, cc + subs x24, x2, x9 + cneg x17, x24, cc + cinv x15, x1, cc + adds x23, x4, x5 + umulh x1, x7, x17 + adcs x24, x8, x4 + adcs x10, x16, x8 + eor x8, x22, x21 + adcs x16, x11, x16 + mul x22, x7, x17 + eor x17, x1, x15 + adc x1, xzr, x11 + adds x11, x24, x5 + eor x7, x3, x21 + adcs x3, x10, x23 + adcs x24, x16, x24 + adcs x4, x1, x10 + eor x10, x22, x15 + adcs x16, xzr, x16 + adc x1, xzr, x1 + cmn x21, #0x1 + adcs x8, x4, x8 + adcs x22, x16, x7 + adc x7, x1, x21 + subs x21, x19, x12 + csetm x4, cc + cneg x1, x21, cc + subs x21, x13, x2 + cinv x16, x4, cc + cneg x4, x21, cc + cmn x15, #0x1 + adcs x21, x23, x10 + mul x23, x1, x4 + adcs x11, x11, x17 + adcs x3, x3, x15 + umulh x1, x1, x4 + adcs x24, x24, x15 + adcs x8, x8, x15 + adcs x22, x22, x15 + eor x17, x23, x16 + adc x15, x7, x15 + subs x7, x20, x14 + cneg x7, x7, cc + csetm x4, cc + subs x10, x20, x12 + cneg x23, x10, cc + csetm x10, cc + subs x12, x6, x9 + cinv x20, x4, cc + cneg x12, x12, cc + cmn x16, #0x1 + eor x1, x1, x16 + adcs x17, x24, x17 + mul x4, x7, x12 + adcs x8, x8, x1 + umulh x1, x7, x12 + adcs x24, x22, x16 + adc x7, x15, x16 + subs x12, x13, x9 + cneg x12, x12, cc + cinv x13, x10, cc + subs x19, x19, x14 + mul x9, x23, x12 + cneg x19, x19, cc + csetm x10, cc + eor x16, x1, x20 + subs x22, x6, x2 + umulh x12, x23, x12 + eor x1, x4, x20 + cinv x4, x10, cc + cneg x22, x22, cc + cmn x20, #0x1 + adcs x15, x11, x1 + eor x6, x12, x13 + adcs x10, x3, x16 + adcs x17, x17, x20 + eor x23, x9, x13 + adcs x2, x8, x20 + mul x11, x19, x22 + adcs x24, x24, x20 + adc x7, x7, x20 + cmn x13, #0x1 + adcs x3, x10, x23 + umulh x22, x19, x22 + adcs x17, x17, x6 + eor x12, x22, x4 + extr x22, x15, x21, #63 + adcs x8, x2, x13 + extr x21, x21, x5, #63 + ldp x16, x23, [x0] + adcs x20, x24, x13 + eor x1, x11, x4 + adc x6, x7, x13 + cmn x4, #0x1 + ldp x2, x7, [x0, #16] + adcs x1, x3, x1 + extr x19, x1, x15, #63 + adcs x14, x17, x12 + extr x1, x14, x1, #63 + lsl x17, x5, #1 + adcs x8, x8, x4 + extr x12, x8, x14, #8 + ldp x15, x11, [x0, #32] + adcs x9, x20, x4 + adc x3, x6, x4 + adds x16, x12, x16 + extr x6, x9, x8, #8 + ldp x14, x12, [x0, #48] + extr x8, x3, x9, #8 + adcs x20, x6, x23 + ldr x24, [x0, #64] + lsr x6, x3, #8 + adcs x8, x8, x2 + and x2, x1, #0x1ff + and x1, x20, x8 + adcs x4, x6, x7 + adcs x3, x17, x15 + and x1, x1, x4 + adcs x9, x21, x11 + and x1, x1, x3 + adcs x6, x22, x14 + and x1, x1, x9 + and x21, x1, x6 + adcs x14, x19, x12 + adc x1, x24, x2 + cmp xzr, xzr + orr x12, x1, #0xfffffffffffffe00 + lsr x1, x1, #9 + adcs xzr, x16, x1 + and x21, x21, x14 + adcs xzr, x21, xzr + adcs xzr, x12, xzr + adcs x21, x16, x1 + adcs x1, x20, xzr + adcs x19, x8, xzr + stp x21, x1, [x0] + adcs x1, x4, xzr + adcs x21, x3, xzr + stp x19, x1, [x0, #16] + adcs x1, x9, xzr + stp x21, x1, [x0, #32] + adcs x21, x6, xzr + adcs x1, x14, xzr + stp x21, x1, [x0, #48] + adc x1, x12, xzr + and x1, x1, #0x1ff + str x1, [x0, #64] + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_sqr_p521) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_sub_p521) + +Lp521_jscalarmul_sub_p521: + CFI_START + ldp x5, x6, [x1] + ldp x4, x3, [x2] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [x1, #16] + ldp x4, x3, [x2, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [x1, #32] + ldp x4, x3, [x2, #32] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + ldp x11, x12, [x1, #48] + ldp x4, x3, [x2, #48] + sbcs x11, x11, x4 + sbcs x12, x12, x3 + ldr x13, [x1, #64] + ldr x4, [x2, #64] + sbcs x13, x13, x4 + sbcs x5, x5, xzr + sbcs x6, x6, xzr + sbcs x7, x7, xzr + sbcs x8, x8, xzr + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbcs x13, x13, xzr + and x13, x13, #0x1ff + stp x5, x6, [x0] + stp x7, x8, [x0, #16] + stp x9, x10, [x0, #32] + stp x11, x12, [x0, #48] + str x13, [x0, #64] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_sub_p521) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/arm/p521_jscalarmul_alt.S b/cbits/s2n/arm/p521_jscalarmul_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/arm/p521_jscalarmul_alt.S @@ -0,0 +1,2143 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Jacobian form scalar multiplication for P-521 +// Input scalar[9], point[27]; output res[27] +// +// extern void p521_jscalarmul_alt +// (uint64_t res[static 27], +// const uint64_t scalar[static 9], +// const uint64_t point[static 27]); +// +// This function is a variant of its affine point version p521_scalarmul. +// Here, input and output points are assumed to be in Jacobian form with +// a triple (x,y,z) representing the affine point (x/z^2,y/z^3) when +// z is nonzero or the point at infinity (group identity) if z = 0. +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-521, returns a representation of n * P. If the result is the +// point at infinity (either because the input point was or because the +// scalar was a multiple of p_521) then the output is guaranteed to +// represent the point at infinity, i.e. to have its z coordinate zero. +// +// Standard ARM ABI: X0 = res, X1 = scalar, X2 = point +// ---------------------------------------------------------------------------- + + +#include "_internal_s2n_bignum_arm.h" + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p521_jscalarmul_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p521_jscalarmul_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p521_jscalarmul_alt) + + + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 72 +#define JACSIZE (3*NUMSIZE) + +// Safe copies of input res and additional values in variables. + +#define tabup x15 +#define bf x16 +#define sgn x17 +#define j x19 +#define res x20 + +// Intermediate variables on the stack. +// The table is 16 entries, each of size JACSIZE = 3 * NUMSIZE + +#define scalarb sp, #(0*NUMSIZE) +#define acc sp, #(1*NUMSIZE) +#define tabent sp, #(4*NUMSIZE) + +#define tab sp, #(7*NUMSIZE) + +// Round up to maintain stack alignment + +#define NSPACE 3968 + +#define selectblock(I) \ + cmp bf, #(1*I) __LF \ + ldp x10, x11, [tabup] __LF \ + csel x0, x10, x0, eq __LF \ + csel x1, x11, x1, eq __LF \ + ldp x10, x11, [tabup, #16] __LF \ + csel x2, x10, x2, eq __LF \ + csel x3, x11, x3, eq __LF \ + ldp x10, x11, [tabup, #32] __LF \ + csel x4, x10, x4, eq __LF \ + csel x5, x11, x5, eq __LF \ + ldp x10, x11, [tabup, #48] __LF \ + csel x6, x10, x6, eq __LF \ + csel x7, x11, x7, eq __LF \ + ldr x10, [tabup, #64] __LF \ + csel x8, x10, x8, eq __LF \ + add tabup, tabup, #JACSIZE + +// Loading large constants + +#define movbig(nn,n3,n2,n1,n0) \ + movz nn, n0 __LF \ + movk nn, n1, lsl #16 __LF \ + movk nn, n2, lsl #32 __LF \ + movk nn, n3, lsl #48 + +S2N_BN_SYMBOL(p521_jscalarmul_alt): + CFI_START + + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x30) + CFI_DEC_SP(NSPACE) + +// Preserve the "res" input argument; others get processed early. + + mov res, x0 + +// Reduce the input scalar mod n_521 and store it to "scalarb". + + mov x19, x2 + add x0, scalarb + CFI_BL(Lp521_jscalarmul_alt_bignum_mod_n521_9) + mov x2, x19 + +// Set the tab[0] table entry to the input point = 1 * P, but also +// reduce all coordinates modulo p. In principle we assume reduction +// as a precondition, but this reduces the scope for surprise, e.g. +// making sure that any input with z = 0 is treated as zero, even +// if the other coordinates are not in fact reduced. + + add x0, tab + mov x1, x19 + CFI_BL(Lp521_jscalarmul_alt_bignum_mod_p521_9) + + add x0, tab+NUMSIZE + add x1, x19, #NUMSIZE + CFI_BL(Lp521_jscalarmul_alt_bignum_mod_p521_9) + + add x0, tab+2*NUMSIZE + add x1, x19, #(2*NUMSIZE) + CFI_BL(Lp521_jscalarmul_alt_bignum_mod_p521_9) + +// If bit 520 of the scalar is set, then negate the scalar mod n_521, +// i.e. do scalar |-> n_521 - scalar, and also the point to compensate +// by negating its y coordinate. This further step is not needed by +// the indexing scheme (the top window is only a couple of bits either +// way), but is convenient to exclude a problem with the specific value +// scalar = n_521 - 18, where the last Jacobian addition is of the form +// (n_521 - 9) * P + -(9 * P) and hence is a degenerate doubling case. + + ldp x0, x1, [scalarb] + movbig(x10, #0xbb6f, #0xb71e, #0x9138, #0x6409) + subs x10, x10, x0 + movbig(x11, #0x3bb5, #0xc9b8, #0x899c, #0x47ae) + sbcs x11, x11, x1 + ldp x2, x3, [scalarb+16] + movbig(x12, #0x7fcc, #0x0148, #0xf709, #0xa5d0) + sbcs x12, x12, x2 + movbig(x13, #0x5186, #0x8783, #0xbf2f, #0x966b) + sbcs x13, x13, x3 + ldp x4, x5, [scalarb+32] + mov x14, 0xfffffffffffffffa + sbcs x14, x14, x4 + mov x15, 0xffffffffffffffff + sbcs x15, x15, x5 + ldp x6, x7, [scalarb+48] + mov x16, 0xffffffffffffffff + sbcs x16, x16, x6 + mov x17, 0xffffffffffffffff + sbcs x17, x17, x7 + ldr x8, [scalarb+64] + mov x19, 0x00000000000001ff + sbc x19, x19, x8 + tst x8, 0x100 + csetm x9, ne + csel x0, x10, x0, ne + csel x1, x11, x1, ne + csel x2, x12, x2, ne + csel x3, x13, x3, ne + csel x4, x14, x4, ne + csel x5, x15, x5, ne + csel x6, x16, x6, ne + csel x7, x17, x7, ne + csel x8, x19, x8, ne + stp x0, x1, [scalarb] + stp x2, x3, [scalarb+16] + stp x4, x5, [scalarb+32] + stp x6, x7, [scalarb+48] + str x8, [scalarb+64] + + add tabup, tab + ldp x0, x1, [tabup, #NUMSIZE] + ldp x2, x3, [tabup, #NUMSIZE+16] + ldp x4, x5, [tabup, #NUMSIZE+32] + ldp x6, x7, [tabup, #NUMSIZE+48] + ldr x8, [tabup, #NUMSIZE+64] + orr x10, x0, x1 + orr x11, x2, x3 + orr x12, x4, x5 + orr x13, x6, x7 + orr x10, x10, x11 + orr x12, x12, x13 + orr x12, x12, x8 + orr x10, x10, x12 + cmp x10, xzr + csel x9, x9, xzr, ne + eor x0, x0, x9 + eor x1, x1, x9 + eor x2, x2, x9 + eor x3, x3, x9 + eor x4, x4, x9 + eor x5, x5, x9 + eor x6, x6, x9 + eor x7, x7, x9 + and x9, x9, #0x1FF + eor x8, x8, x9 + stp x0, x1, [tabup, #NUMSIZE] + stp x2, x3, [tabup, #NUMSIZE+16] + stp x4, x5, [tabup, #NUMSIZE+32] + stp x6, x7, [tabup, #NUMSIZE+48] + str x8, [tabup, #NUMSIZE+64] + +// Compute and record tab[1] = 2 * p, ..., tab[15] = 16 * P + + add x0, tab+JACSIZE*1 + add x1, tab + CFI_BL(Lp521_jscalarmul_alt_jdouble) + + add x0, tab+JACSIZE*2 + add x1, tab+JACSIZE*1 + add x2, tab + CFI_BL(Lp521_jscalarmul_alt_jadd) + + add x0, tab+JACSIZE*3 + add x1, tab+JACSIZE*1 + CFI_BL(Lp521_jscalarmul_alt_jdouble) + + add x0, tab+JACSIZE*4 + add x1, tab+JACSIZE*3 + add x2, tab + CFI_BL(Lp521_jscalarmul_alt_jadd) + + add x0, tab+JACSIZE*5 + add x1, tab+JACSIZE*2 + CFI_BL(Lp521_jscalarmul_alt_jdouble) + + add x0, tab+JACSIZE*6 + add x1, tab+JACSIZE*5 + add x2, tab + CFI_BL(Lp521_jscalarmul_alt_jadd) + + add x0, tab+JACSIZE*7 + add x1, tab+JACSIZE*3 + CFI_BL(Lp521_jscalarmul_alt_jdouble) + + add x0, tab+JACSIZE*8 + add x1, tab+JACSIZE*7 + add x2, tab + CFI_BL(Lp521_jscalarmul_alt_jadd) + + add x0, tab+JACSIZE*9 + add x1, tab+JACSIZE*4 + CFI_BL(Lp521_jscalarmul_alt_jdouble) + + add x0, tab+JACSIZE*10 + add x1, tab+JACSIZE*9 + add x2, tab + CFI_BL(Lp521_jscalarmul_alt_jadd) + + add x0, tab+JACSIZE*11 + add x1, tab+JACSIZE*5 + CFI_BL(Lp521_jscalarmul_alt_jdouble) + + add x0, tab+JACSIZE*12 + add x1, tab+JACSIZE*11 + add x2, tab + CFI_BL(Lp521_jscalarmul_alt_jadd) + + add x0, tab+JACSIZE*13 + add x1, tab+JACSIZE*6 + CFI_BL(Lp521_jscalarmul_alt_jdouble) + + add x0, tab+JACSIZE*14 + add x1, tab+JACSIZE*13 + add x2, tab + CFI_BL(Lp521_jscalarmul_alt_jadd) + + add x0, tab+JACSIZE*15 + add x1, tab+JACSIZE*7 + CFI_BL(Lp521_jscalarmul_alt_jdouble) + +// Add the recoding constant sum_i(16 * 32^i) to the scalar to allow signed +// digits. The digits of the constant, in lowest-to-highest order, are as +// follows; they are generated dynamically since none is a simple ARM load. +// +// 0x0842108421084210 +// 0x1084210842108421 +// 0x2108421084210842 +// 0x4210842108421084 +// 0x8421084210842108 +// 0x0842108421084210 +// 0x1084210842108421 +// 0x2108421084210842 +// 0x0000000000000084 + + ldp x0, x1, [scalarb] + ldp x2, x3, [scalarb+16] + ldp x4, x5, [scalarb+32] + ldp x6, x7, [scalarb+48] + ldr x8, [scalarb+64] + + movbig(x10, #0x1084, #0x2108, #0x4210, #0x8421) + adds x0, x0, x10, lsr #1 + adcs x1, x1, x10 + lsl x10, x10, #1 + adcs x2, x2, x10 + lsl x10, x10, #1 + adcs x3, x3, x10 + lsl x10, x10, #1 + adcs x4, x4, x10 + lsr x11, x10, #4 + adcs x5, x5, x11 + lsr x10, x10, #3 + adcs x6, x6, x10 + lsl x10, x10, #1 + adcs x7, x7, x10 + lsl x10, x10, #1 + and x10, x10, #0xFF + adc x8, x8, x10 + +// Because of the initial reduction the top bitfield (>= bits 520) is <= 1, +// i.e. just a single bit. Record that in "bf", then shift the whole +// scalar left 56 bits to align the top of the next bitfield with the MSB +// (bits 571..575). + + lsr bf, x8, #8 + extr x8, x8, x7, #8 + extr x7, x7, x6, #8 + extr x6, x6, x5, #8 + extr x5, x5, x4, #8 + extr x4, x4, x3, #8 + extr x3, x3, x2, #8 + extr x2, x2, x1, #8 + extr x1, x1, x0, #8 + lsl x0, x0, #56 + stp x0, x1, [scalarb] + stp x2, x3, [scalarb+16] + stp x4, x5, [scalarb+32] + stp x6, x7, [scalarb+48] + str x8, [scalarb+64] + +// According to the top bit, initialize the accumulator to P or 0. This top +// digit, uniquely, is not recoded so there is no sign adjustment to make. +// We only really need to adjust the z coordinate to zero, but do all three. + + add tabup, tab + cmp bf, xzr + + ldp x0, x1, [tabup] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc] + ldp x0, x1, [tabup, #16] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+16] + ldp x0, x1, [tabup, #32] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+32] + ldp x0, x1, [tabup, #48] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+48] + ldp x0, x1, [tabup, #64] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+64] + ldp x0, x1, [tabup, #80] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+80] + ldp x0, x1, [tabup, #96] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+96] + ldp x0, x1, [tabup, #112] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+112] + ldp x0, x1, [tabup, #128] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+128] + ldp x0, x1, [tabup, #144] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+144] + ldp x0, x1, [tabup, #160] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+160] + ldp x0, x1, [tabup, #176] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+176] + ldp x0, x1, [tabup, #192] + csel x0, x0, xzr, ne + csel x1, x1, xzr, ne + stp x0, x1, [acc+192] + ldr x0, [tabup, #208] + csel x0, x0, xzr, ne + str x0, [acc+208] + +// Main loop over size-5 bitfields: double 5 times then add signed digit +// At each stage we shift the scalar left by 5 bits so we can simply pick +// the top 5 bits as the bitfield, saving some fiddle over indexing. + + mov j, #520 + +Lp521_jscalarmul_alt_mainloop: + sub j, j, #5 + + add x0, acc + add x1, acc + CFI_BL(Lp521_jscalarmul_alt_jdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp521_jscalarmul_alt_jdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp521_jscalarmul_alt_jdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp521_jscalarmul_alt_jdouble) + + add x0, acc + add x1, acc + CFI_BL(Lp521_jscalarmul_alt_jdouble) + +// Choose the bitfield and adjust it to sign and magnitude + + ldp x0, x1, [scalarb] + ldp x2, x3, [scalarb+16] + ldp x4, x5, [scalarb+32] + ldp x6, x7, [scalarb+48] + ldr x8, [scalarb+64] + lsr bf, x8, #59 + extr x8, x8, x7, #59 + extr x7, x7, x6, #59 + extr x6, x6, x5, #59 + extr x5, x5, x4, #59 + extr x4, x4, x3, #59 + extr x3, x3, x2, #59 + extr x2, x2, x1, #59 + extr x1, x1, x0, #59 + lsl x0, x0, #5 + stp x0, x1, [scalarb] + stp x2, x3, [scalarb+16] + stp x4, x5, [scalarb+32] + stp x6, x7, [scalarb+48] + str x8, [scalarb+64] + + subs bf, bf, #16 + csetm sgn, lo // sgn = sign of digit (1 = negative) + cneg bf, bf, lo // bf = absolute value of digit + +// Conditionally select the table entry tab[i-1] = i * P in constant time + + mov x0, xzr + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, xzr + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + add tabup, tab + selectblock(1) + selectblock(2) + selectblock(3) + selectblock(4) + selectblock(5) + selectblock(6) + selectblock(7) + selectblock(8) + selectblock(9) + selectblock(10) + selectblock(11) + selectblock(12) + selectblock(13) + selectblock(14) + selectblock(15) + selectblock(16) + stp x0, x1, [tabent] + stp x2, x3, [tabent+16] + stp x4, x5, [tabent+32] + stp x6, x7, [tabent+48] + str x8, [tabent+64] + + mov x0, xzr + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, xzr + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + add tabup, tab+2*NUMSIZE + selectblock(1) + selectblock(2) + selectblock(3) + selectblock(4) + selectblock(5) + selectblock(6) + selectblock(7) + selectblock(8) + selectblock(9) + selectblock(10) + selectblock(11) + selectblock(12) + selectblock(13) + selectblock(14) + selectblock(15) + selectblock(16) + stp x0, x1, [tabent+2*NUMSIZE] + stp x2, x3, [tabent+2*NUMSIZE+16] + stp x4, x5, [tabent+2*NUMSIZE+32] + stp x6, x7, [tabent+2*NUMSIZE+48] + str x8, [tabent+2*NUMSIZE+64] + + mov x0, xzr + mov x1, xzr + mov x2, xzr + mov x3, xzr + mov x4, xzr + mov x5, xzr + mov x6, xzr + mov x7, xzr + mov x8, xzr + add tabup, tab+NUMSIZE + selectblock(1) + selectblock(2) + selectblock(3) + selectblock(4) + selectblock(5) + selectblock(6) + selectblock(7) + selectblock(8) + selectblock(9) + selectblock(10) + selectblock(11) + selectblock(12) + selectblock(13) + selectblock(14) + selectblock(15) + selectblock(16) + +// Store it to "tabent" with the y coordinate optionally negated. +// This is done carefully to give coordinates < p_521 even in +// the degenerate case y = 0 (when z = 0 for points on the curve). + + orr x10, x0, x1 + orr x11, x2, x3 + orr x12, x4, x5 + orr x13, x6, x7 + orr x10, x10, x11 + orr x12, x12, x13 + orr x12, x12, x8 + orr x10, x10, x12 + cmp x10, xzr + csel sgn, sgn, xzr, ne + + eor x0, x0, sgn + eor x1, x1, sgn + eor x2, x2, sgn + eor x3, x3, sgn + eor x4, x4, sgn + eor x5, x5, sgn + eor x6, x6, sgn + eor x7, x7, sgn + and sgn, sgn, #0x1FF + eor x8, x8, sgn + + stp x0, x1, [tabent+NUMSIZE] + stp x2, x3, [tabent+NUMSIZE+16] + stp x4, x5, [tabent+NUMSIZE+32] + stp x6, x7, [tabent+NUMSIZE+48] + str x8, [tabent+NUMSIZE+64] + +// Add to the accumulator + + add x0, acc + add x1, acc + add x2, tabent + CFI_BL(Lp521_jscalarmul_alt_jadd) + + cbnz j, Lp521_jscalarmul_alt_mainloop + +// That's the end of the main loop, and we just need to copy the +// result in "acc" to the output. + + ldp x0, x1, [acc] + stp x0, x1, [res] + ldp x0, x1, [acc+16] + stp x0, x1, [res, #16] + ldp x0, x1, [acc+32] + stp x0, x1, [res, #32] + ldp x0, x1, [acc+48] + stp x0, x1, [res, #48] + ldp x0, x1, [acc+64] + stp x0, x1, [res, #64] + ldp x0, x1, [acc+80] + stp x0, x1, [res, #80] + ldp x0, x1, [acc+96] + stp x0, x1, [res, #96] + ldp x0, x1, [acc+112] + stp x0, x1, [res, #112] + ldp x0, x1, [acc+128] + stp x0, x1, [res, #128] + ldp x0, x1, [acc+144] + stp x0, x1, [res, #144] + ldp x0, x1, [acc+160] + stp x0, x1, [res, #160] + ldp x0, x1, [acc+176] + stp x0, x1, [res, #176] + ldp x0, x1, [acc+192] + stp x0, x1, [res, #192] + ldr x0, [acc+208] + str x0, [res, #208] + +// Restore stack and registers and return + + CFI_INC_SP(NSPACE) + CFI_POP2(x21,x30) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p521_jscalarmul_alt) + +// Local copies of subroutines, complete clones at the moment except +// that we share multiplication and squaring between the point operations. + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_alt_bignum_mod_p521_9) + +Lp521_jscalarmul_alt_bignum_mod_p521_9: + CFI_START + ldr x12, [x1, #64] + lsr x2, x12, #9 + cmp xzr, xzr + ldp x4, x5, [x1] + adcs xzr, x4, x2 + adcs xzr, x5, xzr + ldp x6, x7, [x1, #16] + and x3, x6, x7 + adcs xzr, x3, xzr + ldp x8, x9, [x1, #32] + and x3, x8, x9 + adcs xzr, x3, xzr + ldp x10, x11, [x1, #48] + and x3, x10, x11 + adcs xzr, x3, xzr + orr x3, x12, #0xfffffffffffffe00 + adcs x3, x3, xzr + adcs x4, x4, x2 + adcs x5, x5, xzr + adcs x6, x6, xzr + adcs x7, x7, xzr + adcs x8, x8, xzr + adcs x9, x9, xzr + adcs x10, x10, xzr + adcs x11, x11, xzr + adc x12, x12, xzr + and x12, x12, #0x1ff + stp x4, x5, [x0] + stp x6, x7, [x0, #16] + stp x8, x9, [x0, #32] + stp x10, x11, [x0, #48] + str x12, [x0, #64] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_alt_bignum_mod_p521_9) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_alt_bignum_mod_n521_9) + +Lp521_jscalarmul_alt_bignum_mod_n521_9: + CFI_START + ldr x14, [x1, #64] + lsr x15, x14, #9 + add x15, x15, #1 + mov x2, #39927 + movk x2, #28359, lsl #16 + movk x2, #18657, lsl #32 + movk x2, #17552, lsl #48 + mul x6, x2, x15 + mov x3, #47185 + movk x3, #30307, lsl #16 + movk x3, #13895, lsl #32 + movk x3, #50250, lsl #48 + mul x7, x3, x15 + mov x4, #23087 + movk x4, #2294, lsl #16 + movk x4, #65207, lsl #32 + movk x4, #32819, lsl #48 + mul x8, x4, x15 + mov x5, #27028 + movk x5, #16592, lsl #16 + movk x5, #30844, lsl #32 + movk x5, #44665, lsl #48 + mul x9, x5, x15 + lsl x10, x15, #2 + add x10, x10, x15 + umulh x13, x2, x15 + adds x7, x7, x13 + umulh x13, x3, x15 + adcs x8, x8, x13 + umulh x13, x4, x15 + adcs x9, x9, x13 + umulh x13, x5, x15 + adc x10, x10, x13 + ldp x12, x13, [x1] + adds x6, x6, x12 + adcs x7, x7, x13 + ldp x12, x13, [x1, #16] + adcs x8, x8, x12 + adcs x9, x9, x13 + ldp x13, x11, [x1, #32] + adcs x10, x10, x13 + adcs x11, x11, xzr + ldp x12, x13, [x1, #48] + adcs x12, x12, xzr + adcs x13, x13, xzr + orr x14, x14, #0xfffffffffffffe00 + adcs x14, x14, xzr + csetm x15, lo + and x2, x2, x15 + subs x6, x6, x2 + and x3, x3, x15 + sbcs x7, x7, x3 + and x4, x4, x15 + sbcs x8, x8, x4 + and x5, x5, x15 + sbcs x9, x9, x5 + mov x2, #5 + and x2, x2, x15 + sbcs x10, x10, x2 + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbc x14, x14, xzr + and x14, x14, #0x1ff + stp x6, x7, [x0] + stp x8, x9, [x0, #16] + stp x10, x11, [x0, #32] + stp x12, x13, [x0, #48] + str x14, [x0, #64] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_alt_bignum_mod_n521_9) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_alt_jadd) + +Lp521_jscalarmul_alt_jadd: + CFI_START + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x26) + CFI_PUSH2(x27,x28) + CFI_PUSH2(x29,x30) + CFI_DEC_SP(576) + mov x27, x0 + mov x28, x1 + mov x29, x2 + mov x0, sp + add x1, x28, #0x90 + CFI_BL(Lp521_jscalarmul_alt_sqr_p521) + add x0, sp, #0x168 + add x1, x29, #0x90 + CFI_BL(Lp521_jscalarmul_alt_sqr_p521) + add x0, sp, #0x1f8 + add x1, x29, #0x90 + add x2, x28, #0x48 + CFI_BL(Lp521_jscalarmul_alt_mul_p521) + add x0, sp, #0x48 + add x1, x28, #0x90 + add x2, x29, #0x48 + CFI_BL(Lp521_jscalarmul_alt_mul_p521) + add x0, sp, #0x90 + mov x1, sp + add x2, x29, #0x0 + CFI_BL(Lp521_jscalarmul_alt_mul_p521) + add x0, sp, #0x120 + add x1, sp, #0x168 + add x2, x28, #0x0 + CFI_BL(Lp521_jscalarmul_alt_mul_p521) + add x0, sp, #0x48 + mov x1, sp + add x2, sp, #0x48 + CFI_BL(Lp521_jscalarmul_alt_mul_p521) + add x0, sp, #0x1f8 + add x1, sp, #0x168 + add x2, sp, #0x1f8 + CFI_BL(Lp521_jscalarmul_alt_mul_p521) + add x0, sp, #0x168 + add x1, sp, #0x90 + add x2, sp, #0x120 + CFI_BL(Lp521_jscalarmul_alt_sub_p521) + add x0, sp, #0x48 + add x1, sp, #0x48 + add x2, sp, #0x1f8 + CFI_BL(Lp521_jscalarmul_alt_sub_p521) + add x0, sp, #0xd8 + add x1, sp, #0x168 + CFI_BL(Lp521_jscalarmul_alt_sqr_p521) + mov x0, sp + add x1, sp, #0x48 + CFI_BL(Lp521_jscalarmul_alt_sqr_p521) + add x0, sp, #0x120 + add x1, sp, #0xd8 + add x2, sp, #0x120 + CFI_BL(Lp521_jscalarmul_alt_mul_p521) + add x0, sp, #0x90 + add x1, sp, #0xd8 + add x2, sp, #0x90 + CFI_BL(Lp521_jscalarmul_alt_mul_p521) + mov x0, sp + mov x1, sp + add x2, sp, #0x120 + CFI_BL(Lp521_jscalarmul_alt_sub_p521) + add x0, sp, #0xd8 + add x1, sp, #0x90 + add x2, sp, #0x120 + CFI_BL(Lp521_jscalarmul_alt_sub_p521) + add x0, sp, #0x168 + add x1, sp, #0x168 + add x2, x28, #0x90 + CFI_BL(Lp521_jscalarmul_alt_mul_p521) + mov x0, sp + mov x1, sp + add x2, sp, #0x90 + CFI_BL(Lp521_jscalarmul_alt_sub_p521) + add x0, sp, #0x120 + add x1, sp, #0x120 + mov x2, sp + CFI_BL(Lp521_jscalarmul_alt_sub_p521) + add x0, sp, #0xd8 + add x1, sp, #0xd8 + add x2, sp, #0x1f8 + CFI_BL(Lp521_jscalarmul_alt_mul_p521) + add x0, sp, #0x168 + add x1, sp, #0x168 + add x2, x29, #0x90 + CFI_BL(Lp521_jscalarmul_alt_mul_p521) + add x0, sp, #0x120 + add x1, sp, #0x48 + add x2, sp, #0x120 + CFI_BL(Lp521_jscalarmul_alt_mul_p521) + add x0, sp, #0x120 + add x1, sp, #0x120 + add x2, sp, #0xd8 + CFI_BL(Lp521_jscalarmul_alt_sub_p521) + ldp x0, x1, [x28, #144] + ldp x2, x3, [x28, #160] + ldp x4, x5, [x28, #176] + ldp x6, x7, [x28, #192] + ldr x8, [x28, #208] + orr x20, x0, x1 + orr x21, x2, x3 + orr x22, x4, x5 + orr x23, x6, x7 + orr x20, x20, x21 + orr x22, x22, x23 + orr x20, x20, x8 + orr x20, x20, x22 + cmp x20, xzr + cset x20, ne + ldp x10, x11, [x29, #144] + ldp x12, x13, [x29, #160] + ldp x14, x15, [x29, #176] + ldp x16, x17, [x29, #192] + ldr x19, [x29, #208] + orr x21, x10, x11 + orr x22, x12, x13 + orr x23, x14, x15 + orr x24, x16, x17 + orr x21, x21, x22 + orr x23, x23, x24 + orr x21, x21, x19 + orr x21, x21, x23 + csel x0, x0, x10, ne + csel x1, x1, x11, ne + csel x2, x2, x12, ne + csel x3, x3, x13, ne + csel x4, x4, x14, ne + csel x5, x5, x15, ne + csel x6, x6, x16, ne + csel x7, x7, x17, ne + csel x8, x8, x19, ne + cmp x21, xzr + cset x21, ne + cmp x21, x20 + ldp x10, x11, [sp, #360] + ldp x12, x13, [sp, #376] + ldp x14, x15, [sp, #392] + ldp x16, x17, [sp, #408] + ldr x19, [sp, #424] + csel x0, x0, x10, ne + csel x1, x1, x11, ne + csel x2, x2, x12, ne + csel x3, x3, x13, ne + csel x4, x4, x14, ne + csel x5, x5, x15, ne + csel x6, x6, x16, ne + csel x7, x7, x17, ne + csel x8, x8, x19, ne + stp x0, x1, [sp, #360] + stp x2, x3, [sp, #376] + stp x4, x5, [sp, #392] + stp x6, x7, [sp, #408] + str x8, [sp, #424] + ldp x20, x21, [x28] + ldp x0, x1, [sp] + csel x0, x20, x0, cc + csel x1, x21, x1, cc + ldp x20, x21, [x29] + csel x0, x20, x0, hi + csel x1, x21, x1, hi + ldp x20, x21, [x28, #16] + ldp x2, x3, [sp, #16] + csel x2, x20, x2, cc + csel x3, x21, x3, cc + ldp x20, x21, [x29, #16] + csel x2, x20, x2, hi + csel x3, x21, x3, hi + ldp x20, x21, [x28, #32] + ldp x4, x5, [sp, #32] + csel x4, x20, x4, cc + csel x5, x21, x5, cc + ldp x20, x21, [x29, #32] + csel x4, x20, x4, hi + csel x5, x21, x5, hi + ldp x20, x21, [x28, #48] + ldp x6, x7, [sp, #48] + csel x6, x20, x6, cc + csel x7, x21, x7, cc + ldp x20, x21, [x29, #48] + csel x6, x20, x6, hi + csel x7, x21, x7, hi + ldr x20, [x28, #64] + ldr x8, [sp, #64] + csel x8, x20, x8, cc + ldr x21, [x29, #64] + csel x8, x21, x8, hi + ldp x20, x21, [x28, #72] + ldp x10, x11, [sp, #288] + csel x10, x20, x10, cc + csel x11, x21, x11, cc + ldp x20, x21, [x29, #72] + csel x10, x20, x10, hi + csel x11, x21, x11, hi + ldp x20, x21, [x28, #88] + ldp x12, x13, [sp, #304] + csel x12, x20, x12, cc + csel x13, x21, x13, cc + ldp x20, x21, [x29, #88] + csel x12, x20, x12, hi + csel x13, x21, x13, hi + ldp x20, x21, [x28, #104] + ldp x14, x15, [sp, #320] + csel x14, x20, x14, cc + csel x15, x21, x15, cc + ldp x20, x21, [x29, #104] + csel x14, x20, x14, hi + csel x15, x21, x15, hi + ldp x20, x21, [x28, #120] + ldp x16, x17, [sp, #336] + csel x16, x20, x16, cc + csel x17, x21, x17, cc + ldp x20, x21, [x29, #120] + csel x16, x20, x16, hi + csel x17, x21, x17, hi + ldr x20, [x28, #136] + ldr x19, [sp, #352] + csel x19, x20, x19, cc + ldr x21, [x29, #136] + csel x19, x21, x19, hi + stp x0, x1, [x27] + stp x2, x3, [x27, #16] + stp x4, x5, [x27, #32] + stp x6, x7, [x27, #48] + str x8, [x27, #64] + ldp x0, x1, [sp, #360] + ldp x2, x3, [sp, #376] + ldp x4, x5, [sp, #392] + ldp x6, x7, [sp, #408] + ldr x8, [sp, #424] + stp x10, x11, [x27, #72] + stp x12, x13, [x27, #88] + stp x14, x15, [x27, #104] + stp x16, x17, [x27, #120] + str x19, [x27, #136] + stp x0, x1, [x27, #144] + stp x2, x3, [x27, #160] + stp x4, x5, [x27, #176] + stp x6, x7, [x27, #192] + str x8, [x27, #208] + CFI_INC_SP(576) + CFI_POP2(x29,x30) + CFI_POP2(x27,x28) + CFI_POP2(x25,x26) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_alt_jadd) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_alt_jdouble) + +Lp521_jscalarmul_alt_jdouble: + CFI_START + CFI_PUSH2(x19,x20) + CFI_PUSH2(x21,x22) + CFI_PUSH2(x23,x24) + CFI_PUSH2(x25,x26) + CFI_PUSH2(x27,x28) + CFI_PUSH2(x29,x30) + CFI_DEC_SP(512) + mov x27, x0 + mov x28, x1 + mov x0, sp + add x1, x28, #0x90 + CFI_BL(Lp521_jscalarmul_alt_sqr_p521) + add x0, sp, #0x48 + add x1, x28, #0x48 + CFI_BL(Lp521_jscalarmul_alt_sqr_p521) + ldp x5, x6, [x28] + ldp x4, x3, [sp] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [x28, #16] + ldp x4, x3, [sp, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [x28, #32] + ldp x4, x3, [sp, #32] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + ldp x11, x12, [x28, #48] + ldp x4, x3, [sp, #48] + sbcs x11, x11, x4 + sbcs x12, x12, x3 + ldr x13, [x28, #64] + ldr x4, [sp, #64] + sbcs x13, x13, x4 + sbcs x5, x5, xzr + sbcs x6, x6, xzr + sbcs x7, x7, xzr + sbcs x8, x8, xzr + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbcs x13, x13, xzr + and x13, x13, #0x1ff + stp x5, x6, [sp, #216] + stp x7, x8, [sp, #232] + stp x9, x10, [sp, #248] + stp x11, x12, [sp, #264] + str x13, [sp, #280] + cmp xzr, xzr + ldp x5, x6, [x28] + ldp x4, x3, [sp] + adcs x5, x5, x4 + adcs x6, x6, x3 + ldp x7, x8, [x28, #16] + ldp x4, x3, [sp, #16] + adcs x7, x7, x4 + adcs x8, x8, x3 + ldp x9, x10, [x28, #32] + ldp x4, x3, [sp, #32] + adcs x9, x9, x4 + adcs x10, x10, x3 + ldp x11, x12, [x28, #48] + ldp x4, x3, [sp, #48] + adcs x11, x11, x4 + adcs x12, x12, x3 + ldr x13, [x28, #64] + ldr x4, [sp, #64] + adc x13, x13, x4 + subs x4, x13, #0x200 + csetm x4, cs + sbcs x5, x5, xzr + and x4, x4, #0x200 + sbcs x6, x6, xzr + sbcs x7, x7, xzr + sbcs x8, x8, xzr + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbc x13, x13, x4 + stp x5, x6, [sp, #144] + stp x7, x8, [sp, #160] + stp x9, x10, [sp, #176] + stp x11, x12, [sp, #192] + str x13, [sp, #208] + add x0, sp, #0xd8 + add x1, sp, #0x90 + add x2, sp, #0xd8 + CFI_BL(Lp521_jscalarmul_alt_mul_p521) + cmp xzr, xzr + ldp x5, x6, [x28, #72] + ldp x4, x3, [x28, #144] + adcs x5, x5, x4 + adcs x6, x6, x3 + ldp x7, x8, [x28, #88] + ldp x4, x3, [x28, #160] + adcs x7, x7, x4 + adcs x8, x8, x3 + ldp x9, x10, [x28, #104] + ldp x4, x3, [x28, #176] + adcs x9, x9, x4 + adcs x10, x10, x3 + ldp x11, x12, [x28, #120] + ldp x4, x3, [x28, #192] + adcs x11, x11, x4 + adcs x12, x12, x3 + ldr x13, [x28, #136] + ldr x4, [x28, #208] + adc x13, x13, x4 + subs x4, x13, #0x200 + csetm x4, cs + sbcs x5, x5, xzr + and x4, x4, #0x200 + sbcs x6, x6, xzr + sbcs x7, x7, xzr + sbcs x8, x8, xzr + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbc x13, x13, x4 + stp x5, x6, [sp, #144] + stp x7, x8, [sp, #160] + stp x9, x10, [sp, #176] + stp x11, x12, [sp, #192] + str x13, [sp, #208] + add x0, sp, #0x120 + add x1, x28, #0x0 + add x2, sp, #0x48 + CFI_BL(Lp521_jscalarmul_alt_mul_p521) + add x0, sp, #0x168 + add x1, sp, #0xd8 + CFI_BL(Lp521_jscalarmul_alt_sqr_p521) + add x0, sp, #0x90 + add x1, sp, #0x90 + CFI_BL(Lp521_jscalarmul_alt_sqr_p521) + ldp x6, x7, [sp, #288] + mov x1, #0xc + mul x3, x1, x6 + mul x4, x1, x7 + umulh x6, x1, x6 + adds x4, x4, x6 + umulh x7, x1, x7 + ldp x8, x9, [sp, #304] + mul x5, x1, x8 + mul x6, x1, x9 + umulh x8, x1, x8 + adcs x5, x5, x7 + umulh x9, x1, x9 + adcs x6, x6, x8 + ldp x10, x11, [sp, #320] + mul x7, x1, x10 + mul x8, x1, x11 + umulh x10, x1, x10 + adcs x7, x7, x9 + umulh x11, x1, x11 + adcs x8, x8, x10 + ldp x12, x13, [sp, #336] + mul x9, x1, x12 + mul x10, x1, x13 + umulh x12, x1, x12 + adcs x9, x9, x11 + umulh x13, x1, x13 + adcs x10, x10, x12 + ldr x14, [sp, #352] + mul x11, x1, x14 + adc x11, x11, x13 + mov x1, #0x9 + ldp x20, x21, [sp, #360] + mvn x20, x20 + mul x0, x1, x20 + umulh x20, x1, x20 + adds x3, x3, x0 + mvn x21, x21 + mul x0, x1, x21 + umulh x21, x1, x21 + adcs x4, x4, x0 + ldp x22, x23, [sp, #376] + mvn x22, x22 + mul x0, x1, x22 + umulh x22, x1, x22 + adcs x5, x5, x0 + mvn x23, x23 + mul x0, x1, x23 + umulh x23, x1, x23 + adcs x6, x6, x0 + ldp x17, x19, [sp, #392] + mvn x17, x17 + mul x0, x1, x17 + umulh x17, x1, x17 + adcs x7, x7, x0 + mvn x19, x19 + mul x0, x1, x19 + umulh x19, x1, x19 + adcs x8, x8, x0 + ldp x2, x16, [sp, #408] + mvn x2, x2 + mul x0, x1, x2 + umulh x2, x1, x2 + adcs x9, x9, x0 + mvn x16, x16 + mul x0, x1, x16 + umulh x16, x1, x16 + adcs x10, x10, x0 + ldr x0, [sp, #424] + eor x0, x0, #0x1ff + mul x0, x1, x0 + adc x11, x11, x0 + adds x4, x4, x20 + adcs x5, x5, x21 + and x15, x4, x5 + adcs x6, x6, x22 + and x15, x15, x6 + adcs x7, x7, x23 + and x15, x15, x7 + adcs x8, x8, x17 + and x15, x15, x8 + adcs x9, x9, x19 + and x15, x15, x9 + adcs x10, x10, x2 + and x15, x15, x10 + adc x11, x11, x16 + lsr x12, x11, #9 + orr x11, x11, #0xfffffffffffffe00 + cmp xzr, xzr + adcs xzr, x3, x12 + adcs xzr, x15, xzr + adcs xzr, x11, xzr + adcs x3, x3, x12 + adcs x4, x4, xzr + adcs x5, x5, xzr + adcs x6, x6, xzr + adcs x7, x7, xzr + adcs x8, x8, xzr + adcs x9, x9, xzr + adcs x10, x10, xzr + adc x11, x11, xzr + and x11, x11, #0x1ff + stp x3, x4, [sp, #360] + stp x5, x6, [sp, #376] + stp x7, x8, [sp, #392] + stp x9, x10, [sp, #408] + str x11, [sp, #424] + ldp x5, x6, [sp, #144] + ldp x4, x3, [sp] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #160] + ldp x4, x3, [sp, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #176] + ldp x4, x3, [sp, #32] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + ldp x11, x12, [sp, #192] + ldp x4, x3, [sp, #48] + sbcs x11, x11, x4 + sbcs x12, x12, x3 + ldr x13, [sp, #208] + ldr x4, [sp, #64] + sbcs x13, x13, x4 + sbcs x5, x5, xzr + sbcs x6, x6, xzr + sbcs x7, x7, xzr + sbcs x8, x8, xzr + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbcs x13, x13, xzr + and x13, x13, #0x1ff + stp x5, x6, [sp, #144] + stp x7, x8, [sp, #160] + stp x9, x10, [sp, #176] + stp x11, x12, [sp, #192] + str x13, [sp, #208] + mov x0, sp + add x1, sp, #0x48 + CFI_BL(Lp521_jscalarmul_alt_sqr_p521) + add x0, sp, #0xd8 + add x1, sp, #0x168 + add x2, sp, #0xd8 + CFI_BL(Lp521_jscalarmul_alt_mul_p521) + ldp x5, x6, [sp, #144] + ldp x4, x3, [sp, #72] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [sp, #160] + ldp x4, x3, [sp, #88] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [sp, #176] + ldp x4, x3, [sp, #104] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + ldp x11, x12, [sp, #192] + ldp x4, x3, [sp, #120] + sbcs x11, x11, x4 + sbcs x12, x12, x3 + ldr x13, [sp, #208] + ldr x4, [sp, #136] + sbcs x13, x13, x4 + sbcs x5, x5, xzr + sbcs x6, x6, xzr + sbcs x7, x7, xzr + sbcs x8, x8, xzr + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbcs x13, x13, xzr + and x13, x13, #0x1ff + stp x5, x6, [x27, #144] + stp x7, x8, [x27, #160] + stp x9, x10, [x27, #176] + stp x11, x12, [x27, #192] + str x13, [x27, #208] + ldp x6, x7, [sp, #288] + lsl x3, x6, #2 + extr x4, x7, x6, #62 + ldp x8, x9, [sp, #304] + extr x5, x8, x7, #62 + extr x6, x9, x8, #62 + ldp x10, x11, [sp, #320] + extr x7, x10, x9, #62 + extr x8, x11, x10, #62 + ldp x12, x13, [sp, #336] + extr x9, x12, x11, #62 + extr x10, x13, x12, #62 + ldr x14, [sp, #352] + extr x11, x14, x13, #62 + ldp x0, x1, [sp, #360] + mvn x0, x0 + adds x3, x3, x0 + sbcs x4, x4, x1 + ldp x0, x1, [sp, #376] + sbcs x5, x5, x0 + and x15, x4, x5 + sbcs x6, x6, x1 + and x15, x15, x6 + ldp x0, x1, [sp, #392] + sbcs x7, x7, x0 + and x15, x15, x7 + sbcs x8, x8, x1 + and x15, x15, x8 + ldp x0, x1, [sp, #408] + sbcs x9, x9, x0 + and x15, x15, x9 + sbcs x10, x10, x1 + and x15, x15, x10 + ldr x0, [sp, #424] + eor x0, x0, #0x1ff + adc x11, x11, x0 + lsr x12, x11, #9 + orr x11, x11, #0xfffffffffffffe00 + cmp xzr, xzr + adcs xzr, x3, x12 + adcs xzr, x15, xzr + adcs xzr, x11, xzr + adcs x3, x3, x12 + adcs x4, x4, xzr + adcs x5, x5, xzr + adcs x6, x6, xzr + adcs x7, x7, xzr + adcs x8, x8, xzr + adcs x9, x9, xzr + adcs x10, x10, xzr + adc x11, x11, xzr + and x11, x11, #0x1ff + stp x3, x4, [x27] + stp x5, x6, [x27, #16] + stp x7, x8, [x27, #32] + stp x9, x10, [x27, #48] + str x11, [x27, #64] + ldp x6, x7, [sp, #216] + lsl x3, x6, #1 + adds x3, x3, x6 + extr x4, x7, x6, #63 + adcs x4, x4, x7 + ldp x8, x9, [sp, #232] + extr x5, x8, x7, #63 + adcs x5, x5, x8 + extr x6, x9, x8, #63 + adcs x6, x6, x9 + ldp x10, x11, [sp, #248] + extr x7, x10, x9, #63 + adcs x7, x7, x10 + extr x8, x11, x10, #63 + adcs x8, x8, x11 + ldp x12, x13, [sp, #264] + extr x9, x12, x11, #63 + adcs x9, x9, x12 + extr x10, x13, x12, #63 + adcs x10, x10, x13 + ldr x14, [sp, #280] + extr x11, x14, x13, #63 + adc x11, x11, x14 + ldp x20, x21, [sp] + mvn x20, x20 + lsl x0, x20, #3 + adds x3, x3, x0 + mvn x21, x21 + extr x0, x21, x20, #61 + adcs x4, x4, x0 + ldp x22, x23, [sp, #16] + mvn x22, x22 + extr x0, x22, x21, #61 + adcs x5, x5, x0 + and x15, x4, x5 + mvn x23, x23 + extr x0, x23, x22, #61 + adcs x6, x6, x0 + and x15, x15, x6 + ldp x20, x21, [sp, #32] + mvn x20, x20 + extr x0, x20, x23, #61 + adcs x7, x7, x0 + and x15, x15, x7 + mvn x21, x21 + extr x0, x21, x20, #61 + adcs x8, x8, x0 + and x15, x15, x8 + ldp x22, x23, [sp, #48] + mvn x22, x22 + extr x0, x22, x21, #61 + adcs x9, x9, x0 + and x15, x15, x9 + mvn x23, x23 + extr x0, x23, x22, #61 + adcs x10, x10, x0 + and x15, x15, x10 + ldr x0, [sp, #64] + eor x0, x0, #0x1ff + extr x0, x0, x23, #61 + adc x11, x11, x0 + lsr x12, x11, #9 + orr x11, x11, #0xfffffffffffffe00 + cmp xzr, xzr + adcs xzr, x3, x12 + adcs xzr, x15, xzr + adcs xzr, x11, xzr + adcs x3, x3, x12 + adcs x4, x4, xzr + adcs x5, x5, xzr + adcs x6, x6, xzr + adcs x7, x7, xzr + adcs x8, x8, xzr + adcs x9, x9, xzr + adcs x10, x10, xzr + adc x11, x11, xzr + and x11, x11, #0x1ff + stp x3, x4, [x27, #72] + stp x5, x6, [x27, #88] + stp x7, x8, [x27, #104] + stp x9, x10, [x27, #120] + str x11, [x27, #136] + CFI_INC_SP(512) + CFI_POP2(x29,x30) + CFI_POP2(x27,x28) + CFI_POP2(x25,x26) + CFI_POP2(x23,x24) + CFI_POP2(x21,x22) + CFI_POP2(x19,x20) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_alt_jdouble) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_alt_mul_p521) + +Lp521_jscalarmul_alt_mul_p521: + CFI_START + ldp x3, x4, [x1] + ldp x5, x6, [x2] + mul x15, x3, x5 + umulh x16, x3, x5 + mul x14, x3, x6 + umulh x17, x3, x6 + adds x16, x16, x14 + ldp x7, x8, [x2, #16] + mul x14, x3, x7 + umulh x19, x3, x7 + adcs x17, x17, x14 + mul x14, x3, x8 + umulh x20, x3, x8 + adcs x19, x19, x14 + ldp x9, x10, [x2, #32] + mul x14, x3, x9 + umulh x21, x3, x9 + adcs x20, x20, x14 + mul x14, x3, x10 + umulh x22, x3, x10 + adcs x21, x21, x14 + ldp x11, x12, [x2, #48] + mul x14, x3, x11 + umulh x23, x3, x11 + adcs x22, x22, x14 + ldr x13, [x2, #64] + mul x14, x3, x12 + umulh x24, x3, x12 + adcs x23, x23, x14 + mul x14, x3, x13 + umulh x25, x3, x13 + adcs x24, x24, x14 + adc x25, x25, xzr + mul x14, x4, x5 + adds x16, x16, x14 + mul x14, x4, x6 + adcs x17, x17, x14 + mul x14, x4, x7 + adcs x19, x19, x14 + mul x14, x4, x8 + adcs x20, x20, x14 + mul x14, x4, x9 + adcs x21, x21, x14 + mul x14, x4, x10 + adcs x22, x22, x14 + mul x14, x4, x11 + adcs x23, x23, x14 + mul x14, x4, x12 + adcs x24, x24, x14 + mul x14, x4, x13 + adcs x25, x25, x14 + cset x26, cs + umulh x14, x4, x5 + adds x17, x17, x14 + umulh x14, x4, x6 + adcs x19, x19, x14 + umulh x14, x4, x7 + adcs x20, x20, x14 + umulh x14, x4, x8 + adcs x21, x21, x14 + umulh x14, x4, x9 + adcs x22, x22, x14 + umulh x14, x4, x10 + adcs x23, x23, x14 + umulh x14, x4, x11 + adcs x24, x24, x14 + umulh x14, x4, x12 + adcs x25, x25, x14 + umulh x14, x4, x13 + adc x26, x26, x14 + stp x15, x16, [sp, #432] + ldp x3, x4, [x1, #16] + mul x14, x3, x5 + adds x17, x17, x14 + mul x14, x3, x6 + adcs x19, x19, x14 + mul x14, x3, x7 + adcs x20, x20, x14 + mul x14, x3, x8 + adcs x21, x21, x14 + mul x14, x3, x9 + adcs x22, x22, x14 + mul x14, x3, x10 + adcs x23, x23, x14 + mul x14, x3, x11 + adcs x24, x24, x14 + mul x14, x3, x12 + adcs x25, x25, x14 + mul x14, x3, x13 + adcs x26, x26, x14 + cset x15, cs + umulh x14, x3, x5 + adds x19, x19, x14 + umulh x14, x3, x6 + adcs x20, x20, x14 + umulh x14, x3, x7 + adcs x21, x21, x14 + umulh x14, x3, x8 + adcs x22, x22, x14 + umulh x14, x3, x9 + adcs x23, x23, x14 + umulh x14, x3, x10 + adcs x24, x24, x14 + umulh x14, x3, x11 + adcs x25, x25, x14 + umulh x14, x3, x12 + adcs x26, x26, x14 + umulh x14, x3, x13 + adc x15, x15, x14 + mul x14, x4, x5 + adds x19, x19, x14 + mul x14, x4, x6 + adcs x20, x20, x14 + mul x14, x4, x7 + adcs x21, x21, x14 + mul x14, x4, x8 + adcs x22, x22, x14 + mul x14, x4, x9 + adcs x23, x23, x14 + mul x14, x4, x10 + adcs x24, x24, x14 + mul x14, x4, x11 + adcs x25, x25, x14 + mul x14, x4, x12 + adcs x26, x26, x14 + mul x14, x4, x13 + adcs x15, x15, x14 + cset x16, cs + umulh x14, x4, x5 + adds x20, x20, x14 + umulh x14, x4, x6 + adcs x21, x21, x14 + umulh x14, x4, x7 + adcs x22, x22, x14 + umulh x14, x4, x8 + adcs x23, x23, x14 + umulh x14, x4, x9 + adcs x24, x24, x14 + umulh x14, x4, x10 + adcs x25, x25, x14 + umulh x14, x4, x11 + adcs x26, x26, x14 + umulh x14, x4, x12 + adcs x15, x15, x14 + umulh x14, x4, x13 + adc x16, x16, x14 + stp x17, x19, [sp, #448] + ldp x3, x4, [x1, #32] + mul x14, x3, x5 + adds x20, x20, x14 + mul x14, x3, x6 + adcs x21, x21, x14 + mul x14, x3, x7 + adcs x22, x22, x14 + mul x14, x3, x8 + adcs x23, x23, x14 + mul x14, x3, x9 + adcs x24, x24, x14 + mul x14, x3, x10 + adcs x25, x25, x14 + mul x14, x3, x11 + adcs x26, x26, x14 + mul x14, x3, x12 + adcs x15, x15, x14 + mul x14, x3, x13 + adcs x16, x16, x14 + cset x17, cs + umulh x14, x3, x5 + adds x21, x21, x14 + umulh x14, x3, x6 + adcs x22, x22, x14 + umulh x14, x3, x7 + adcs x23, x23, x14 + umulh x14, x3, x8 + adcs x24, x24, x14 + umulh x14, x3, x9 + adcs x25, x25, x14 + umulh x14, x3, x10 + adcs x26, x26, x14 + umulh x14, x3, x11 + adcs x15, x15, x14 + umulh x14, x3, x12 + adcs x16, x16, x14 + umulh x14, x3, x13 + adc x17, x17, x14 + mul x14, x4, x5 + adds x21, x21, x14 + mul x14, x4, x6 + adcs x22, x22, x14 + mul x14, x4, x7 + adcs x23, x23, x14 + mul x14, x4, x8 + adcs x24, x24, x14 + mul x14, x4, x9 + adcs x25, x25, x14 + mul x14, x4, x10 + adcs x26, x26, x14 + mul x14, x4, x11 + adcs x15, x15, x14 + mul x14, x4, x12 + adcs x16, x16, x14 + mul x14, x4, x13 + adcs x17, x17, x14 + cset x19, cs + umulh x14, x4, x5 + adds x22, x22, x14 + umulh x14, x4, x6 + adcs x23, x23, x14 + umulh x14, x4, x7 + adcs x24, x24, x14 + umulh x14, x4, x8 + adcs x25, x25, x14 + umulh x14, x4, x9 + adcs x26, x26, x14 + umulh x14, x4, x10 + adcs x15, x15, x14 + umulh x14, x4, x11 + adcs x16, x16, x14 + umulh x14, x4, x12 + adcs x17, x17, x14 + umulh x14, x4, x13 + adc x19, x19, x14 + stp x20, x21, [sp, #464] + ldp x3, x4, [x1, #48] + mul x14, x3, x5 + adds x22, x22, x14 + mul x14, x3, x6 + adcs x23, x23, x14 + mul x14, x3, x7 + adcs x24, x24, x14 + mul x14, x3, x8 + adcs x25, x25, x14 + mul x14, x3, x9 + adcs x26, x26, x14 + mul x14, x3, x10 + adcs x15, x15, x14 + mul x14, x3, x11 + adcs x16, x16, x14 + mul x14, x3, x12 + adcs x17, x17, x14 + mul x14, x3, x13 + adcs x19, x19, x14 + cset x20, cs + umulh x14, x3, x5 + adds x23, x23, x14 + umulh x14, x3, x6 + adcs x24, x24, x14 + umulh x14, x3, x7 + adcs x25, x25, x14 + umulh x14, x3, x8 + adcs x26, x26, x14 + umulh x14, x3, x9 + adcs x15, x15, x14 + umulh x14, x3, x10 + adcs x16, x16, x14 + umulh x14, x3, x11 + adcs x17, x17, x14 + umulh x14, x3, x12 + adcs x19, x19, x14 + umulh x14, x3, x13 + adc x20, x20, x14 + mul x14, x4, x5 + adds x23, x23, x14 + mul x14, x4, x6 + adcs x24, x24, x14 + mul x14, x4, x7 + adcs x25, x25, x14 + mul x14, x4, x8 + adcs x26, x26, x14 + mul x14, x4, x9 + adcs x15, x15, x14 + mul x14, x4, x10 + adcs x16, x16, x14 + mul x14, x4, x11 + adcs x17, x17, x14 + mul x14, x4, x12 + adcs x19, x19, x14 + mul x14, x4, x13 + adcs x20, x20, x14 + cset x21, cs + umulh x14, x4, x5 + adds x24, x24, x14 + umulh x14, x4, x6 + adcs x25, x25, x14 + umulh x14, x4, x7 + adcs x26, x26, x14 + umulh x14, x4, x8 + adcs x15, x15, x14 + umulh x14, x4, x9 + adcs x16, x16, x14 + umulh x14, x4, x10 + adcs x17, x17, x14 + umulh x14, x4, x11 + adcs x19, x19, x14 + umulh x14, x4, x12 + adcs x20, x20, x14 + umulh x14, x4, x13 + adc x21, x21, x14 + stp x22, x23, [sp, #480] + ldr x3, [x1, #64] + mul x14, x3, x5 + adds x24, x24, x14 + mul x14, x3, x6 + adcs x25, x25, x14 + mul x14, x3, x7 + adcs x26, x26, x14 + mul x14, x3, x8 + adcs x15, x15, x14 + mul x14, x3, x9 + adcs x16, x16, x14 + mul x14, x3, x10 + adcs x17, x17, x14 + mul x14, x3, x11 + adcs x19, x19, x14 + mul x14, x3, x12 + adcs x20, x20, x14 + mul x14, x3, x13 + adc x21, x21, x14 + umulh x14, x3, x5 + adds x25, x25, x14 + umulh x14, x3, x6 + adcs x26, x26, x14 + umulh x14, x3, x7 + adcs x15, x15, x14 + umulh x14, x3, x8 + adcs x16, x16, x14 + umulh x14, x3, x9 + adcs x17, x17, x14 + umulh x14, x3, x10 + adcs x19, x19, x14 + umulh x14, x3, x11 + adcs x20, x20, x14 + umulh x14, x3, x12 + adc x21, x21, x14 + cmp xzr, xzr + ldp x5, x6, [sp, #432] + extr x14, x25, x24, #9 + adcs x5, x5, x14 + extr x14, x26, x25, #9 + adcs x6, x6, x14 + ldp x7, x8, [sp, #448] + extr x14, x15, x26, #9 + adcs x7, x7, x14 + extr x14, x16, x15, #9 + adcs x8, x8, x14 + ldp x9, x10, [sp, #464] + extr x14, x17, x16, #9 + adcs x9, x9, x14 + extr x14, x19, x17, #9 + adcs x10, x10, x14 + ldp x11, x12, [sp, #480] + extr x14, x20, x19, #9 + adcs x11, x11, x14 + extr x14, x21, x20, #9 + adcs x12, x12, x14 + orr x13, x24, #0xfffffffffffffe00 + lsr x14, x21, #9 + adcs x13, x13, x14 + sbcs x5, x5, xzr + sbcs x6, x6, xzr + sbcs x7, x7, xzr + sbcs x8, x8, xzr + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbc x13, x13, xzr + and x13, x13, #0x1ff + stp x5, x6, [x0] + stp x7, x8, [x0, #16] + stp x9, x10, [x0, #32] + stp x11, x12, [x0, #48] + str x13, [x0, #64] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_alt_mul_p521) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_alt_sqr_p521) + +Lp521_jscalarmul_alt_sqr_p521: + CFI_START + ldp x2, x3, [x1] + mul x11, x2, x3 + umulh x12, x2, x3 + ldp x4, x5, [x1, #16] + mul x10, x2, x4 + umulh x13, x2, x4 + adds x12, x12, x10 + ldp x6, x7, [x1, #32] + mul x10, x2, x5 + umulh x14, x2, x5 + adcs x13, x13, x10 + ldp x8, x9, [x1, #48] + mul x10, x2, x6 + umulh x15, x2, x6 + adcs x14, x14, x10 + mul x10, x2, x7 + umulh x16, x2, x7 + adcs x15, x15, x10 + mul x10, x2, x8 + umulh x17, x2, x8 + adcs x16, x16, x10 + mul x10, x2, x9 + umulh x19, x2, x9 + adcs x17, x17, x10 + adc x19, x19, xzr + mul x10, x3, x4 + adds x13, x13, x10 + mul x10, x3, x5 + adcs x14, x14, x10 + mul x10, x3, x6 + adcs x15, x15, x10 + mul x10, x3, x7 + adcs x16, x16, x10 + mul x10, x3, x8 + adcs x17, x17, x10 + mul x10, x3, x9 + adcs x19, x19, x10 + cset x20, cs + umulh x10, x3, x4 + adds x14, x14, x10 + umulh x10, x3, x5 + adcs x15, x15, x10 + umulh x10, x3, x6 + adcs x16, x16, x10 + umulh x10, x3, x7 + adcs x17, x17, x10 + umulh x10, x3, x8 + adcs x19, x19, x10 + umulh x10, x3, x9 + adc x20, x20, x10 + mul x10, x6, x7 + umulh x21, x6, x7 + adds x20, x20, x10 + adc x21, x21, xzr + mul x10, x4, x5 + adds x15, x15, x10 + mul x10, x4, x6 + adcs x16, x16, x10 + mul x10, x4, x7 + adcs x17, x17, x10 + mul x10, x4, x8 + adcs x19, x19, x10 + mul x10, x4, x9 + adcs x20, x20, x10 + mul x10, x6, x8 + adcs x21, x21, x10 + cset x22, cs + umulh x10, x4, x5 + adds x16, x16, x10 + umulh x10, x4, x6 + adcs x17, x17, x10 + umulh x10, x4, x7 + adcs x19, x19, x10 + umulh x10, x4, x8 + adcs x20, x20, x10 + umulh x10, x4, x9 + adcs x21, x21, x10 + umulh x10, x6, x8 + adc x22, x22, x10 + mul x10, x7, x8 + umulh x23, x7, x8 + adds x22, x22, x10 + adc x23, x23, xzr + mul x10, x5, x6 + adds x17, x17, x10 + mul x10, x5, x7 + adcs x19, x19, x10 + mul x10, x5, x8 + adcs x20, x20, x10 + mul x10, x5, x9 + adcs x21, x21, x10 + mul x10, x6, x9 + adcs x22, x22, x10 + mul x10, x7, x9 + adcs x23, x23, x10 + cset x24, cs + umulh x10, x5, x6 + adds x19, x19, x10 + umulh x10, x5, x7 + adcs x20, x20, x10 + umulh x10, x5, x8 + adcs x21, x21, x10 + umulh x10, x5, x9 + adcs x22, x22, x10 + umulh x10, x6, x9 + adcs x23, x23, x10 + umulh x10, x7, x9 + adc x24, x24, x10 + mul x10, x8, x9 + umulh x25, x8, x9 + adds x24, x24, x10 + adc x25, x25, xzr + adds x11, x11, x11 + adcs x12, x12, x12 + adcs x13, x13, x13 + adcs x14, x14, x14 + adcs x15, x15, x15 + adcs x16, x16, x16 + adcs x17, x17, x17 + adcs x19, x19, x19 + adcs x20, x20, x20 + adcs x21, x21, x21 + adcs x22, x22, x22 + adcs x23, x23, x23 + adcs x24, x24, x24 + adcs x25, x25, x25 + cset x26, cs + umulh x10, x2, x2 + adds x11, x11, x10 + mul x10, x3, x3 + adcs x12, x12, x10 + umulh x10, x3, x3 + adcs x13, x13, x10 + mul x10, x4, x4 + adcs x14, x14, x10 + umulh x10, x4, x4 + adcs x15, x15, x10 + mul x10, x5, x5 + adcs x16, x16, x10 + umulh x10, x5, x5 + adcs x17, x17, x10 + mul x10, x6, x6 + adcs x19, x19, x10 + umulh x10, x6, x6 + adcs x20, x20, x10 + mul x10, x7, x7 + adcs x21, x21, x10 + umulh x10, x7, x7 + adcs x22, x22, x10 + mul x10, x8, x8 + adcs x23, x23, x10 + umulh x10, x8, x8 + adcs x24, x24, x10 + mul x10, x9, x9 + adcs x25, x25, x10 + umulh x10, x9, x9 + adc x26, x26, x10 + ldr x1, [x1, #64] + add x1, x1, x1 + mul x10, x1, x2 + adds x19, x19, x10 + umulh x10, x1, x2 + adcs x20, x20, x10 + mul x10, x1, x4 + adcs x21, x21, x10 + umulh x10, x1, x4 + adcs x22, x22, x10 + mul x10, x1, x6 + adcs x23, x23, x10 + umulh x10, x1, x6 + adcs x24, x24, x10 + mul x10, x1, x8 + adcs x25, x25, x10 + umulh x10, x1, x8 + adcs x26, x26, x10 + lsr x4, x1, #1 + mul x4, x4, x4 + adc x4, x4, xzr + mul x10, x1, x3 + adds x20, x20, x10 + umulh x10, x1, x3 + adcs x21, x21, x10 + mul x10, x1, x5 + adcs x22, x22, x10 + umulh x10, x1, x5 + adcs x23, x23, x10 + mul x10, x1, x7 + adcs x24, x24, x10 + umulh x10, x1, x7 + adcs x25, x25, x10 + mul x10, x1, x9 + adcs x26, x26, x10 + umulh x10, x1, x9 + adc x4, x4, x10 + mul x2, x2, x2 + cmp xzr, xzr + extr x10, x20, x19, #9 + adcs x2, x2, x10 + extr x10, x21, x20, #9 + adcs x11, x11, x10 + extr x10, x22, x21, #9 + adcs x12, x12, x10 + extr x10, x23, x22, #9 + adcs x13, x13, x10 + extr x10, x24, x23, #9 + adcs x14, x14, x10 + extr x10, x25, x24, #9 + adcs x15, x15, x10 + extr x10, x26, x25, #9 + adcs x16, x16, x10 + extr x10, x4, x26, #9 + adcs x17, x17, x10 + orr x19, x19, #0xfffffffffffffe00 + lsr x10, x4, #9 + adcs x19, x19, x10 + sbcs x2, x2, xzr + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbcs x13, x13, xzr + sbcs x14, x14, xzr + sbcs x15, x15, xzr + sbcs x16, x16, xzr + sbcs x17, x17, xzr + sbc x19, x19, xzr + and x19, x19, #0x1ff + stp x2, x11, [x0] + stp x12, x13, [x0, #16] + stp x14, x15, [x0, #32] + stp x16, x17, [x0, #48] + str x19, [x0, #64] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_alt_sqr_p521) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_alt_sub_p521) + +Lp521_jscalarmul_alt_sub_p521: + CFI_START + ldp x5, x6, [x1] + ldp x4, x3, [x2] + subs x5, x5, x4 + sbcs x6, x6, x3 + ldp x7, x8, [x1, #16] + ldp x4, x3, [x2, #16] + sbcs x7, x7, x4 + sbcs x8, x8, x3 + ldp x9, x10, [x1, #32] + ldp x4, x3, [x2, #32] + sbcs x9, x9, x4 + sbcs x10, x10, x3 + ldp x11, x12, [x1, #48] + ldp x4, x3, [x2, #48] + sbcs x11, x11, x4 + sbcs x12, x12, x3 + ldr x13, [x1, #64] + ldr x4, [x2, #64] + sbcs x13, x13, x4 + sbcs x5, x5, xzr + sbcs x6, x6, xzr + sbcs x7, x7, xzr + sbcs x8, x8, xzr + sbcs x9, x9, xzr + sbcs x10, x10, xzr + sbcs x11, x11, xzr + sbcs x12, x12, xzr + sbcs x13, x13, xzr + and x13, x13, #0x1ff + stp x5, x6, [x0] + stp x7, x8, [x0, #16] + stp x9, x10, [x0, #32] + stp x11, x12, [x0, #48] + str x13, [x0, #64] + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_alt_sub_p521) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/import.sh b/cbits/s2n/import.sh new file mode 100644 --- /dev/null +++ b/cbits/s2n/import.sh @@ -0,0 +1,131 @@ +#!/bin/sh +# Re-import the vendored parts of AWS's s2n-bignum. +# +# Only the files crypton calls are kept, and they are kept unmodified -- the +# dispatch that chooses between the two variants of each is in +# cbits/p256/p256_s2n.c, not in here. Run this from cbits/s2n: +# +# ./import.sh [commit] +# +# and commit the result together with the COMMIT line it writes, so that the +# tree always says which upstream revision it holds. +set -eu + +REPO=https://github.com/awslabs/s2n-bignum +REV=${1:-main} +HERE=$(cd "$(dirname "$0")" && pwd) +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT + +git clone -q "$REPO" "$TMP/s2n" +git -C "$TMP/s2n" checkout -q "$REV" + +# The headers every vendored file includes. Not x86_att/'s own header of +# the same name, which is crypton's -- it makes the x86-64 files assemble on +# Windows and hands on to this one; see README.md. Nothing here copies into +# that name, and nothing should. +for h in _internal_s2n_bignum_arm.h _internal_s2n_bignum_x86_att.h; do + cp "$TMP/s2n/include/$h" "$HERE/include/$h" +done +cp "$TMP/s2n/LICENSE" "$HERE/LICENSE" + +# Both variants of each routine are taken, since which one is faster is not +# the same question on the two architectures -- see README.md. Where the +# upstream tree has no separate _alt file the plain one defines both symbols, +# so "copy it if it is there" gets the right set either way. +# x86-64 only, for the things AArch64 does not want -- see README.md. +take_x86() { + dir=$1 + name=$2 + cp "$TMP/s2n/x86_att/$dir/$name.S" "$HERE/x86_att/$name.S" +} + +take() { + curve=$1 + name=$2 + for arch in arm x86_att; do + for v in "" _alt; do + src="$TMP/s2n/$arch/$curve/$name$v.S" + if [ -f "$src" ]; then + cp "$src" "$HERE/$arch/$name$v.S" + fi + done + done +} + +# P-256: variable-point scalar multiplication, affine in and out, and the +# fixed-base one, which reads a table of its own that +# cbits/p256/gen_base_table.py builds. +take p256 p256_scalarmul +take p256 p256_scalarmulbase + +# The Jacobian point operations, which ECDSA verification walks itself: it +# multiplies two scalars at once, in variable time, which is allowed there +# because everything it touches is public. See cbits/p256/p256_verify.c. +take p256 p256_montjadd +take p256 p256_montjdouble +take p256 p256_montjmixadd +take p256 bignum_tomont_p256 +take p256 bignum_demont_p256 +take p256 bignum_neg_p256 + +# P-384 and P-521 have no affine wrapper upstream, so the Montgomery and +# Jacobian conversions are built here out of these; the glue is in +# cbits/crypton_ecc_s2n.c. +take p384 p384_montjscalarmul +take p384 bignum_tomont_p384 +take p384 bignum_deamont_p384 +take p384 bignum_montmul_p384 +take p384 bignum_montsqr_p384 +take p384 bignum_montinv_p384 + +take p521 p521_jscalarmul +take p521 bignum_mul_p521 +take p521 bignum_sqr_p521 +take p521 bignum_inv_p521 + +# X25519, both the general one and the fixed-base one that a key is +# generated with. The word form, which both architectures have, rather than +# the byte form that only AArch64 has: they measure the same and this way +# there is one code path. +take curve25519 curve25519_x25519 +take curve25519 curve25519_x25519base + +# Ed25519's base point multiplication, which signing does twice -- once for +# the nonce's point and once for the public key it derives from the secret +# key every time -- and the encoding of the result, which has one form. +take curve25519 edwards25519_scalarmulbase +take curve25519 edwards25519_encode + +# Inversion modulo an odd number of any size, which is what ECDSA does once +# per signature and once per verification. It uses no instruction beyond +# the base architecture, so there is one of it and no run-time question. +take generic bignum_modinv + +# Modular exponentiation at RSA sizes. Only x86-64: on AArch64 crypton's C +# is the faster of the two, measured, so nothing is taken for it. The +# Karatsuba multiplications and the reduction all want ADX. +take_x86 fastmul bignum_kmul_16_32 +take_x86 fastmul bignum_ksqr_16_32 +take_x86 fastmul bignum_kmul_32_64 +take_x86 fastmul bignum_ksqr_32_64 +take_x86 fastmul bignum_emontredc_8n +# RSA-3072's 24-limb square, built from these where the 32-limb Karatsuba +# would take its operands padded +take_x86 fastmul bignum_sqr_8_16 +take_x86 fastmul bignum_mul_8_16 +# and, for the sizes there is no Karatsuba routine for, the schoolbook +# multiplication and squaring of any size, which bignum_emontredc_8n then +# reduces as it reduces the others +take_x86 generic bignum_mul +take_x86 generic bignum_sqr + +# R^2 mod m, which every exponentiation and every setup in crypton_ecc.c +# begins with. Both architectures: the C climbs to it by a ladder of +# Montgomery squarings and this is faster on both. Taken from upstream's +# main when it went in rather than by this script, which is why it was +# missing here; it is the same file at the COMMIT above. +take generic bignum_montifier + +git -C "$TMP/s2n" rev-parse HEAD > "$HERE/COMMIT" +echo "imported $(cat "$HERE/COMMIT")" diff --git a/cbits/s2n/include/_internal_s2n_bignum_arm.h b/cbits/s2n/include/_internal_s2n_bignum_arm.h new file mode 100644 --- /dev/null +++ b/cbits/s2n/include/_internal_s2n_bignum_arm.h @@ -0,0 +1,103 @@ +#ifdef __APPLE__ +# define S2N_BN_SYMBOL(NAME) _##NAME +# if defined(__AARCH64EL__) || defined(__ARMEL__) +# define __LF %% +# else +# define __LF ; +# endif +#else +# define S2N_BN_SYMBOL(name) name +# define __LF ; +#endif + +#define S2N_BN_SYM_VISIBILITY_DIRECTIVE(name) .globl S2N_BN_SYMBOL(name) + +#ifdef S2N_BN_HIDE_SYMBOLS +# ifdef __APPLE__ +# define S2N_BN_SYM_PRIVACY_DIRECTIVE(name) .private_extern S2N_BN_SYMBOL(name) +# else +# define S2N_BN_SYM_PRIVACY_DIRECTIVE(name) .hidden S2N_BN_SYMBOL(name) +# endif +#else +# define S2N_BN_SYM_PRIVACY_DIRECTIVE(name) /* NO-OP: S2N_BN_SYM_PRIVACY_DIRECTIVE */ +#endif + +#ifdef __APPLE__ +# define S2N_BN_FUNCTION_TYPE_DIRECTIVE(name) /* Not used in Mach-O */ +#else +# define S2N_BN_FUNCTION_TYPE_DIRECTIVE(name) .type name, %function +#endif + +#ifdef __APPLE__ +# define S2N_BN_SIZE_DIRECTIVE(name) /* Not used in Mach-O */ +#else +# define S2N_BN_SIZE_DIRECTIVE(name) .size S2N_BN_SYMBOL(name), .-S2N_BN_SYMBOL(name) +#endif + +// Enable branch target identification (BTI) support unless explicitly disabled +// with -DNO_IBT, mirroring the x86 _CET_ENDBR machinery. AARCH64_VALID_CALL_TARGET +// is emitted at each entry point unconditionally by default, since BTI 'c' is in +// the hint space and so behaves as a NOP on all pre-Armv8.5-A processors. The name +// matches AWS-LC's macro, whose definition we defer to if already present, just as +// the x86 side defers to . Unlike CET, BTI also needs a .note.gnu.property +// section: it has GNU_PROPERTY_AARCH64_FEATURE_1_AND semantics, so one object +// without the note silently disables BTI program-wide, hence emitting it here. + +#if NO_IBT +# if defined(AARCH64_VALID_CALL_TARGET) +# error "The s2n-bignum build option NO_IBT was configured, but AARCH64_VALID_CALL_TARGET is defined in this compilation unit. That is weird, so failing the build." +# endif +# define AARCH64_VALID_CALL_TARGET +#elif !defined(AARCH64_VALID_CALL_TARGET) +# define AARCH64_VALID_CALL_TARGET hint #34 /* BTI c */ +# ifndef __APPLE__ + .pushsection .note.gnu.property, "a" + .balign 8 + .long 4 /* n_namesz: sizeof "GNU\0" */ + .long 0x10 /* n_descsz: 16 bytes of property data */ + .long 0x5 /* n_type: NT_GNU_PROPERTY_TYPE_0 */ + .asciz "GNU" + .long 0xc0000000 /* pr_type: GNU_PROPERTY_AARCH64_FEATURE_1_AND */ + .long 4 /* pr_datasz: 4 bytes */ + .long 1 /* pr_data: GNU_PROPERTY_AARCH64_FEATURE_1_BTI */ + .long 0 /* pad to 8-byte alignment */ + .popsection +# endif +#endif + +// Variants of instructions including CFI (call frame information) annotations + +#define CFI_START .cfi_startproc +#define CFI_RET ret __LF .cfi_endproc + +#define CFI_BL(target) bl target + +#define CFI_PUSH2(lo,hi) stp lo, hi, [sp, #-16]! __LF .cfi_adjust_cfa_offset 16 __LF .cfi_rel_offset lo, 0 __LF .cfi_rel_offset hi, 8 +#define CFI_PUSH1Z(reg) stp reg, xzr, [sp, #-16]! __LF .cfi_adjust_cfa_offset 16 __LF .cfi_rel_offset reg, 0 + +#define CFI_POP2(lo,hi) ldp lo, hi, [sp], #16 __LF .cfi_adjust_cfa_offset -16 __LF .cfi_restore lo __LF .cfi_restore hi +#define CFI_POP1Z(reg) ldp reg, xzr, [sp], #16 __LF .cfi_adjust_cfa_offset -16 __LF .cfi_restore reg + +#define CFI_STACKSAVE2(lo,hi,offset) stp lo, hi, [sp, #(offset)] __LF .cfi_rel_offset lo, offset __LF .cfi_rel_offset hi, offset+8 + +// This is an alternative to CFI_STACKSAVE2 to work around delocator problems +// in the AWS-LC FIPS build, avoiding certain composite expressions. It is +// expected that offset8 = offset+8 as in an invocation of CFI_STACKSAVE2. +// Likewise the (offset+0) oddities in the following macros are driven by +// delocator problems. + +#define CFI_STACKSAVE2X(lo,hi,offset,offset8) stp lo, hi, [sp, #(offset+0)] __LF .cfi_rel_offset lo, offset __LF .cfi_rel_offset hi, offset8 + +#define CFI_STACKSAVE1Z(reg,offset) stp reg, xzr, [sp, #(offset+0)] __LF .cfi_rel_offset reg, offset + +#define CFI_STACKLOAD2(lo,hi,offset) ldp lo, hi, [sp, #(offset+0)] __LF .cfi_restore lo __LF .cfi_restore hi +#define CFI_STACKLOAD1Z(reg,offset) ldp reg, xzr, [sp, #(offset+0)] __LF .cfi_restore reg + +// It would be better to use -(offset) not -offset, but again there seem +// to be delocator issues. We adopt a discipline of not using dangerous +// composite expressions in this macro, e.g. parenthesizing the argument +// or just using numeric constants or products where the association is +// not a problem. + +#define CFI_INC_SP(offset) add sp, sp, #(offset+0) __LF .cfi_adjust_cfa_offset -offset +#define CFI_DEC_SP(offset) sub sp, sp, #(offset+0) __LF .cfi_adjust_cfa_offset offset diff --git a/cbits/s2n/include/_internal_s2n_bignum_x86_att.h b/cbits/s2n/include/_internal_s2n_bignum_x86_att.h new file mode 100644 --- /dev/null +++ b/cbits/s2n/include/_internal_s2n_bignum_x86_att.h @@ -0,0 +1,68 @@ +#ifdef __APPLE__ +# define S2N_BN_SYMBOL(NAME) _##NAME +#else +# define S2N_BN_SYMBOL(name) name +#endif + +#define S2N_BN_SYM_VISIBILITY_DIRECTIVE(name) .globl S2N_BN_SYMBOL(name) + +#ifdef S2N_BN_HIDE_SYMBOLS +# ifdef __APPLE__ +# define S2N_BN_SYM_PRIVACY_DIRECTIVE(name) .private_extern S2N_BN_SYMBOL(name) +# else +# define S2N_BN_SYM_PRIVACY_DIRECTIVE(name) .hidden S2N_BN_SYMBOL(name) +# endif +#else +# define S2N_BN_SYM_PRIVACY_DIRECTIVE(name) /* NO-OP: S2N_BN_SYM_PRIVACY_DIRECTIVE */ +#endif + +#ifdef __APPLE__ +# define S2N_BN_FUNCTION_TYPE_DIRECTIVE(name) /* Not used in Mach-O */ +#else +# define S2N_BN_FUNCTION_TYPE_DIRECTIVE(name) .type name, %function +#endif + +#ifdef __APPLE__ +# define S2N_BN_SIZE_DIRECTIVE(name) /* Not used in Mach-O */ +#else +# define S2N_BN_SIZE_DIRECTIVE(name) .size S2N_BN_SYMBOL(name), .-S2N_BN_SYMBOL(name) +#endif + +// Enable indirect branch tracking support unless explicitly disabled +// with -DNO_IBT. If the platform supports CET, simply inherit this from +// the usual header. Otherwise manually define _CET_ENDBR, used at each +// x86 entry point, to be the ENDBR64 instruction, with an explicit byte +// sequence for compilers/assemblers that don't know about it. Note that +// it is safe to use ENDBR64 on all platforms, since the encoding is by +// design interpreted as a NOP on all pre-CET x86_64 processors. The only +// downside is a small increase in code size and potentially a modest +// slowdown from executing one more instruction. + +#if NO_IBT +# if defined(_CET_ENDBR) +# error "The s2n-bignum build option NO_IBT was configured, but _CET_ENDBR is defined in this compilation unit. That is weird, so failing the build." +# endif +# define _CET_ENDBR +#elif defined(__CET__) +# include +#elif !defined(_CET_ENDBR) +# define _CET_ENDBR .byte 0xf3,0x0f,0x1e,0xfa +#endif + +// Variants of instructions including CFI (call frame information) annotations + +#define CFI_START .cfi_startproc +#define CFI_RET retq ; .cfi_endproc + +#define CFI_CALL(target) callq target + +#define CFI_PUSH(reg) pushq reg ; .cfi_adjust_cfa_offset 8 ; .cfi_rel_offset reg, 0 +#define CFI_POP(reg) popq reg ; .cfi_adjust_cfa_offset -8 ; .cfi_restore reg + +#define CFI_INC_RSP(offset) addq $offset, %rsp ; .cfi_adjust_cfa_offset -offset +#define CFI_DEC_RSP(offset) subq $offset, %rsp ; .cfi_adjust_cfa_offset offset + +#define CFI_STACKSAVE(reg,offset) mov reg, offset(%rsp) ; .cfi_rel_offset reg, offset +#define CFI_STACKLOAD(reg,offset) mov offset(%rsp), reg ; .cfi_restore reg +#define CFI_STACKSAVEU(reg,offset) movups reg, offset(%rsp) ; .cfi_rel_offset reg, offset +#define CFI_STACKLOADU(reg,offset) movups offset(%rsp), reg ; .cfi_restore reg diff --git a/cbits/s2n/x86_att/_internal_s2n_bignum_x86_att.h b/cbits/s2n/x86_att/_internal_s2n_bignum_x86_att.h new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/_internal_s2n_bignum_x86_att.h @@ -0,0 +1,33 @@ +/* + * crypton's, not s2n-bignum's: what the vendored assembly needs on Windows, + * kept out of the vendored files so that import.sh can replace them as it + * always has. + * + * Every file in this directory includes "_internal_s2n_bignum_x86_att.h", + * and a quoted include looks first in the directory of the file that asks + * for it -- so this is the one each of them gets, and it hands on to + * upstream's own in cbits/s2n/include. On Linux and macOS it changes + * nothing. + * + * On Windows, two things. The ELF directives .type and .size are refused by + * the COFF assemblers -- gcc's says "junk at end of line" and clang's + * "expected absolute expression" -- and there is nothing in a PE object for + * them to say, so they are emptied, as upstream does for Mach-O. And the + * calling convention is Microsoft's, not System V's: each file carries a + * prologue that moves its arguments into the registers it was written + * for, under WINDOWS_ABI, which upstream leaves to the build to define. + */ +/* By a path from this file rather than by #include_next: clang, finding + * this file next to the one that asked for it, starts #include_next from + * the top of the search and finds this file again, without end. */ +#include "../include/_internal_s2n_bignum_x86_att.h" + +#if defined(_WIN32) +# undef S2N_BN_FUNCTION_TYPE_DIRECTIVE +# define S2N_BN_FUNCTION_TYPE_DIRECTIVE(name) /* not used in COFF */ +# undef S2N_BN_SIZE_DIRECTIVE +# define S2N_BN_SIZE_DIRECTIVE(name) /* not used in COFF */ +# ifndef WINDOWS_ABI +# define WINDOWS_ABI 1 +# endif +#endif diff --git a/cbits/s2n/x86_att/bignum_deamont_p384.S b/cbits/s2n/x86_att/bignum_deamont_p384.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_deamont_p384.S @@ -0,0 +1,184 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Convert from almost-Montgomery form, z := (x / 2^384) mod p_384 +// Input x[6]; output z[6] +// +// extern void bignum_deamont_p384(uint64_t z[static 6], +// const uint64_t x[static 6]); +// +// Convert a 6-digit bignum x out of its (optionally almost) Montgomery form, +// "almost" meaning any 6-digit input will work, with no range restriction. +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_deamont_p384) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_deamont_p384) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_deamont_p384) + .text + +#define z %rdi +#define x %rsi + +// Additional temps in the correction phase + +#define u %rax +#define v %rcx +#define w %rdx + +#define vshort %ecx + +// Core one-step "short" Montgomery reduction macro. Takes input in +// [d5;d4;d3;d2;d1;d0] and returns result in [d6;d5;d4;d3;d2;d1], +// adding to the existing contents of [d5;d4;d3;d2;d1;d0]. This +// is intended only for 6-word inputs as in mapping out of Montgomery, +// not for the general case of Montgomery multiplication. It is fine +// for d6 to be the same register as d0. +// +// Parms: montreds(d6,d5,d4,d3,d2,d1,d0) +// +// We want to add (2^384 - 2^128 - 2^96 + 2^32 - 1) * w +// where w = [d0 + (d0<<32)] mod 2^64 + +#define montreds(d6,d5,d4,d3,d2,d1,d0) \ +/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64 */ \ + movq d0, %rdx ; \ + shlq $32, %rdx ; \ + addq d0, %rdx ; \ +/* Construct [%rsi;%rcx;%rax;-] = (2^384 - p_384) * w */ \ +/* We know the lowest word will cancel so we can re-use d0 */ \ +/* as a temp. */ \ + xorq %rsi, %rsi ; \ + movq $0xffffffff00000001, %rax ; \ + mulxq %rax, %rcx, %rax ; \ + movl $0x00000000ffffffff, %ecx ; \ + mulxq %rcx, d0, %rcx ; \ + adcq d0, %rax ; \ + adcq %rdx, %rcx ; \ + adcq $0, %rsi ; \ +/* Now subtract that and add 2^384 * w */ \ + subq %rax, d1 ; \ + sbbq %rcx, d2 ; \ + sbbq %rsi, d3 ; \ + sbbq $0, d4 ; \ + sbbq $0, d5 ; \ + movq %rdx, d6 ; \ + sbbq $0, d6 + +S2N_BN_SYMBOL(bignum_deamont_p384): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Save more registers to play with + + CFI_PUSH(%r12) + CFI_PUSH(%r13) + +// Set up an initial window [%r13,%r12,%r11,%r10,%r9,%r8] = x + + movq (x), %r8 + movq 8(x), %r9 + movq 16(x), %r10 + movq 24(x), %r11 + movq 32(x), %r12 + movq 40(x), %r13 + +// Montgomery reduce window 0 + + montreds(%r8,%r13,%r12,%r11,%r10,%r9,%r8) + +// Montgomery reduce window 1 + + montreds(%r9,%r8,%r13,%r12,%r11,%r10,%r9) + +// Montgomery reduce window 2 + + montreds(%r10,%r9,%r8,%r13,%r12,%r11,%r10) + +// Montgomery reduce window 3 + + montreds(%r11,%r10,%r9,%r8,%r13,%r12,%r11) + +// Montgomery reduce window 4 + + montreds(%r12,%r11,%r10,%r9,%r8,%r13,%r12) + +// Montgomery reduce window 5 + + montreds(%r13,%r12,%r11,%r10,%r9,%r8,%r13) + +// Do a test addition of dd = [%r13;%r12;%r11;%r10;%r9;%r8] and +// 2^384 - p_384 = [0;0;0;1;v;u], hence setting CF iff +// dd + (2^384 - p_384) >= 2^384, hence iff dd >= p_384. + + movq $0xffffffff00000001, u + movl $0x00000000ffffffff, vshort + + movq %r8, w + addq u, w + movq %r9, w + adcq v, w + movq %r10, w + adcq $1, w + movq %r11, w + adcq $0, w + movq %r12, w + adcq $0, w + movq %r13, w + adcq $0, w + +// Convert CF to a bitmask in w + + sbbq w, w + +// Masked addition of 2^384 - p_384, hence subtraction of p_384 + + andq w, u + andq w, v + andq $1, w + + addq u, %r8 + adcq v, %r9 + adcq w, %r10 + adcq $0, %r11 + adcq $0, %r12 + adcq $0, %r13 + +// Write back the result + + movq %r8, (z) + movq %r9, 8(z) + movq %r10, 16(z) + movq %r11, 24(z) + movq %r12, 32(z) + movq %r13, 40(z) + +// Restore registers and return + + CFI_POP(%r13) + CFI_POP(%r12) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_deamont_p384) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_deamont_p384_alt.S b/cbits/s2n/x86_att/bignum_deamont_p384_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_deamont_p384_alt.S @@ -0,0 +1,184 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Convert from almost-Montgomery form, z := (x / 2^384) mod p_384 +// Input x[6]; output z[6] +// +// extern void bignum_deamont_p384_alt(uint64_t z[static 6], +// const uint64_t x[static 6]); +// +// Convert a 6-digit bignum x out of its (optionally almost) Montgomery form, +// "almost" meaning any 6-digit input will work, with no range restriction. +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_deamont_p384_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_deamont_p384_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_deamont_p384_alt) + .text + +#define z %rdi +#define x %rsi + +// Additional temps in the correction phase + +#define u %rax +#define v %rcx +#define w %rdx + +#define vshort %ecx + +// Core one-step "short" Montgomery reduction macro. Takes input in +// [d5;d4;d3;d2;d1;d0] and returns result in [d6;d5;d4;d3;d2;d1], +// adding to the existing [d5;d4;d3;d2;d1] and re-using d0 as a +// temporary internally, as well as %rax, %rcx and %rdx. +// It is OK for d6 and d0 to be the same register (they often are) +// +// We want to add (2^384 - 2^128 - 2^96 + 2^32 - 1) * w +// where w = [d0 + (d0<<32)] mod 2^64 +// +// montreds(d6,d5,d4,d3,d2,d1,d0) + +#define montreds(d6,d5,d4,d3,d2,d1,d0) \ +/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64 */ \ + movq d0, %rcx ; \ + shlq $32, %rcx ; \ + addq d0, %rcx ; \ +/* Construct [%rax;%rdx;d0;-] = (2^384 - p_384) * w */ \ +/* We know the lowest word will cancel so we can re-use d0 */ \ +/* and %rcx as temps. */ \ + movq $0xffffffff00000001, %rax ; \ + mulq %rcx; \ + movq %rdx, d0 ; \ + movq $0x00000000ffffffff, %rax ; \ + mulq %rcx; \ + addq %rax, d0 ; \ + movl $0, %eax ; \ + adcq %rcx, %rdx ; \ + adcl %eax, %eax ; \ +/* Now subtract that and add 2^384 * w */ \ + subq d0, d1 ; \ + sbbq %rdx, d2 ; \ + sbbq %rax, d3 ; \ + sbbq $0, d4 ; \ + sbbq $0, d5 ; \ + movq %rcx, d6 ; \ + sbbq $0, d6 + +S2N_BN_SYMBOL(bignum_deamont_p384_alt): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Save more registers to play with + + CFI_PUSH(%r12) + CFI_PUSH(%r13) + +// Set up an initial window [%r13,%r12,%r11,%r10,%r9,%r8] = x + + movq (x), %r8 + movq 8(x), %r9 + movq 16(x), %r10 + movq 24(x), %r11 + movq 32(x), %r12 + movq 40(x), %r13 + +// Montgomery reduce window 0 + + montreds(%r8,%r13,%r12,%r11,%r10,%r9,%r8) + +// Montgomery reduce window 1 + + montreds(%r9,%r8,%r13,%r12,%r11,%r10,%r9) + +// Montgomery reduce window 2 + + montreds(%r10,%r9,%r8,%r13,%r12,%r11,%r10) + +// Montgomery reduce window 3 + + montreds(%r11,%r10,%r9,%r8,%r13,%r12,%r11) + +// Montgomery reduce window 4 + + montreds(%r12,%r11,%r10,%r9,%r8,%r13,%r12) + +// Montgomery reduce window 5 + + montreds(%r13,%r12,%r11,%r10,%r9,%r8,%r13) + +// Do a test addition of dd = [%r13;%r12;%r11;%r10;%r9;%r8] and +// 2^384 - p_384 = [0;0;0;1;v;u], hence setting CF iff +// dd + (2^384 - p_384) >= 2^384, hence iff dd >= p_384. + + movq $0xffffffff00000001, u + movl $0x00000000ffffffff, vshort + + movq %r8, w + addq u, w + movq %r9, w + adcq v, w + movq %r10, w + adcq $1, w + movq %r11, w + adcq $0, w + movq %r12, w + adcq $0, w + movq %r13, w + adcq $0, w + +// Convert CF to a bitmask in w + + sbbq w, w + +// Masked addition of 2^384 - p_384, hence subtraction of p_384 + + andq w, u + andq w, v + andq $1, w + + addq u, %r8 + adcq v, %r9 + adcq w, %r10 + adcq $0, %r11 + adcq $0, %r12 + adcq $0, %r13 + +// Write back the result + + movq %r8, (z) + movq %r9, 8(z) + movq %r10, 16(z) + movq %r11, 24(z) + movq %r12, 32(z) + movq %r13, 40(z) + +// Restore registers and return + + CFI_POP(%r13) + CFI_POP(%r12) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_deamont_p384_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_demont_p256.S b/cbits/s2n/x86_att/bignum_demont_p256.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_demont_p256.S @@ -0,0 +1,116 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Convert from Montgomery form z := (x / 2^256) mod p_256, assuming x reduced +// Input x[4]; output z[4] +// +// extern void bignum_demont_p256(uint64_t z[static 4], +// const uint64_t x[static 4]); +// +// This assumes the input is < p_256 for correctness. If this is not the case, +// use the variant "bignum_deamont_p256" instead. +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_demont_p256) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_demont_p256) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_demont_p256) + .text + +#define z %rdi +#define x %rsi + +// Add %rdx * m into a register-pair (high,low) +// maintaining consistent double-carrying with adcx and adox, +// using %rax and %rcx as temporaries + +#define mulpadd(high,low,m) \ + mulxq m, %rax, %rcx ; \ + adcxq %rax, low ; \ + adoxq %rcx, high + +S2N_BN_SYMBOL(bignum_demont_p256): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Save one more register to play with + + CFI_PUSH(%rbx) + +// Set up an initial 4-word window [%r11,%r10,%r9,%r8] = x + + movq (x), %r8 + movq 8(x), %r9 + movq 16(x), %r10 + movq 24(x), %r11 + +// Fill in two zeros to the left + + xorq %rbx, %rbx + xorq %rsi, %rsi + +// Montgomery reduce windows 0 and 1 together + + movq $0x0000000100000000, %rdx + mulpadd(%r10,%r9,%r8) + mulpadd(%r11,%r10,%r9) + movq $0xffffffff00000001, %rdx + mulpadd(%rbx,%r11,%r8) + mulpadd(%rsi,%rbx,%r9) + movl $0, %r8d + adcxq %r8, %rsi + +// Append just one more leading zero (by the above %r8 = 0 already). + + xorq %r9, %r9 + +// Montgomery reduce windows 2 and 3 together + + movq $0x0000000100000000, %rdx + mulpadd(%rbx,%r11,%r10) + mulpadd(%rsi,%rbx,%r11) + movq $0xffffffff00000001, %rdx + mulpadd(%r8,%rsi,%r10) + mulpadd(%r9,%r8,%r11) + movl $0, %r10d + adcxq %r10, %r9 + +// Since the input was assumed reduced modulo, i.e. < p, we actually know that +// 2^256 * [carries; %r9;%r8;%rsi;%rbx] is <= (p - 1) + (2^256 - 1) p +// and hence [carries; %r9;%r8;%rsi;%rbx] < p. This means in fact carries = 0 +// and [%r9;%r8;%rsi;%rbx] is already our answer, without further correction. +// Write that back. + + movq %rbx, (z) + movq %rsi, 8(z) + movq %r8, 16(z) + movq %r9, 24(z) + +// Restore saved register and return + + CFI_POP(%rbx) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_demont_p256) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_demont_p256_alt.S b/cbits/s2n/x86_att/bignum_demont_p256_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_demont_p256_alt.S @@ -0,0 +1,134 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Convert from Montgomery form z := (x / 2^256) mod p_256, assuming x reduced +// Input x[4]; output z[4] +// +// extern void bignum_demont_p256_alt(uint64_t z[static 4], +// const uint64_t x[static 4]); +// +// This assumes the input is < p_256 for correctness. If this is not the case, +// use the variant "bignum_deamont_p256" instead. +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_demont_p256_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_demont_p256_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_demont_p256_alt) + .text + +#define z %rdi +#define x %rsi + +// Add %rdx * m into a register-pair (high,low) +// maintaining consistent double-carrying with adcx and adox, +// using %rax and %rcx as temporaries + +#define mulpado(high,low,m) \ + mulxq m, %rax, %rcx ; \ + adcxq %rax, low ; \ + adoxq %rcx, high + +// Add %rcx * m into a register-pair (high,low) maintaining consistent +// carry-catching with carry (negated, as bitmask) and using %rax and %rdx +// as temporaries + +#define mulpadd(carry,high,low,m) \ + movq m, %rax ; \ + mulq %rcx; \ + subq carry, %rdx ; \ + addq %rax, low ; \ + adcq %rdx, high ; \ + sbbq carry, carry + +// Initial version assuming no carry-in + +#define mulpadi(carry,high,low,m) \ + movq m, %rax ; \ + mulq %rcx; \ + addq %rax, low ; \ + adcq %rdx, high ; \ + sbbq carry, carry + +// Version with no carry in or out + +#define mulpadn(high,low,m) \ + movq m, %rax ; \ + mulq %rcx; \ + addq %rax, low ; \ + adcq %rdx, high + +S2N_BN_SYMBOL(bignum_demont_p256_alt): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Set up an initial 4-word window [%r11,%r10,%r9,%r8] = x + + movq (x), %r8 + movq 8(x), %r9 + movq 16(x), %r10 + movq 24(x), %r11 + +// Load constant 2^32; %rcx toggles between this and (1 - %rcx) below + + movq $0x0000000100000000, %rcx + +// Montgomery reduce windows 0 and 1 together as [%r8;%rsi;%r11;%r10] + + mulpadi(%rsi,%r10,%r9,%r8) + mulpadd(%rsi,%r11,%r10,%r9) + negq %rcx + negq %rsi + incq %rcx + mulpadi(%r8,%rsi,%r11,%r8) + negq %r8 + mulpadn(%r8,%rsi,%r9) + +// Montgomery reduce windows 2 and 3 together as [%r10;%r9;%r8;%rsi] + + negq %rcx + incq %rcx + mulpadi(%r9,%rsi,%r11,%r10) + mulpadd(%r9,%r8,%rsi,%r11) + negq %rcx + negq %r9 + incq %rcx + mulpadi(%r10,%r9,%r8,%r10) + negq %r10 + mulpadn(%r10,%r9,%r11) + +// Since the input was assumed reduced modulo, i.e. < p, we actually know that +// 2^256 * [carries; %r10;%r9;%r8;%rsi] is <= (p - 1) + (2^256 - 1) p +// and hence [carries; %r10;%r9;%r8;%rsi] < p. This means in fact carries = 0 +// and [%r10;%r9;%r8;%rsi] is already our answer, without further correction. +// Write that back. + + movq %rsi, (z) + movq %r8, 8(z) + movq %r9, 16(z) + movq %r10, 24(z) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_demont_p256_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_emontredc_8n.S b/cbits/s2n/x86_att/bignum_emontredc_8n.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_emontredc_8n.S @@ -0,0 +1,427 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Extended Montgomery reduce in 8-digit blocks, results in input-output buffer +// Inputs z[2*k], m[k], w; outputs function return (extra result bit) and z[2*k] +// +// extern uint64_t bignum_emontredc_8n(uint64_t k, uint64_t *z, const uint64_t *m, +// uint64_t w); +// +// Functionally equivalent to bignum_emontredc (see that file for more detail). +// But in general assumes that the input k is a multiple of 8. +// +// Standard x86-64 ABI: RDI = k, RSI = z, RDX = m, RCX = w, returns RAX +// Microsoft x64 ABI: RCX = k, RDX = z, R8 = m, R9 = w, returns RAX +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_emontredc_8n) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_emontredc_8n) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_emontredc_8n) + .text + .balign 32 + +// Original input parameters are here + +#define z %rsi +#define w %rcx + +// This is copied in early once we stash away k + +#define m %rdi + +// A variable z pointer + +#define zz %rbp + +// Stack-based variables + +#define carry (%rsp) +#define innercount 8(%rsp) +#define outercount 16(%rsp) +#define k8m1 24(%rsp) + +// ----------------------------------------------------------------------------- +// Standard macros as used in pure multiplier arrays +// ----------------------------------------------------------------------------- + +// mulpadd i, j adds z[i] * rdx (now assumed = m[j]) into the window at i+j + +.macro mulpadd arg1,arg2 + mulxq 8*\arg1(z), %rax, %rbx +.if ((\arg1 + \arg2) % 8 == 0) + adcxq %rax, %r8 + adoxq %rbx, %r9 +.elseif ((\arg1 + \arg2) % 8 == 1) + adcxq %rax, %r9 + adoxq %rbx, %r10 +.elseif ((\arg1 + \arg2) % 8 == 2) + adcxq %rax, %r10 + adoxq %rbx, %r11 +.elseif ((\arg1 + \arg2) % 8 == 3) + adcxq %rax, %r11 + adoxq %rbx, %r12 +.elseif ((\arg1 + \arg2) % 8 == 4) + adcxq %rax, %r12 + adoxq %rbx, %r13 +.elseif ((\arg1 + \arg2) % 8 == 5) + adcxq %rax, %r13 + adoxq %rbx, %r14 +.elseif ((\arg1 + \arg2) % 8 == 6) + adcxq %rax, %r14 + adoxq %rbx, %r15 +.elseif ((\arg1 + \arg2) % 8 == 7) + adcxq %rax, %r15 + adoxq %rbx, %r8 +.endif + +.endm + +// addrow i adds z[i] + zz[0..7] * m[j] into the window + +.macro addrow arg1 + movq 8*\arg1(m), %rdx + xorl %eax, %eax // Get a known flag state + +.if (\arg1 % 8 == 0) + adoxq 8*\arg1(zz), %r8 +.elseif (\arg1 % 8 == 1) + adoxq 8*\arg1(zz), %r9 +.elseif (\arg1 % 8 == 2) + adoxq 8*\arg1(zz), %r10 +.elseif (\arg1 % 8 == 3) + adoxq 8*\arg1(zz), %r11 +.elseif (\arg1 % 8 == 4) + adoxq 8*\arg1(zz), %r12 +.elseif (\arg1 % 8 == 5) + adoxq 8*\arg1(zz), %r13 +.elseif (\arg1 % 8 == 6) + adoxq 8*\arg1(zz), %r14 +.elseif (\arg1 % 8 == 7) + adoxq 8*\arg1(zz), %r15 +.endif + + mulpadd 0, \arg1 + +.if (\arg1 % 8 == 0) + movq %r8, 8*\arg1(zz) + movl $0, %r8d +.elseif (\arg1 % 8 == 1) + movq %r9, 8*\arg1(zz) + movl $0, %r9d +.elseif (\arg1 % 8 == 2) + movq %r10, 8*\arg1(zz) + movl $0, %r10d +.elseif (\arg1 % 8 == 3) + movq %r11, 8*\arg1(zz) + movl $0, %r11d +.elseif (\arg1 % 8 == 4) + movq %r12, 8*\arg1(zz) + movl $0, %r12d +.elseif (\arg1 % 8 == 5) + movq %r13, 8*\arg1(zz) + movl $0, %r13d +.elseif (\arg1 % 8 == 6) + movq %r14, 8*\arg1(zz) + movl $0, %r14d +.elseif (\arg1 % 8 == 7) + movq %r15, 8*\arg1(zz) + movl $0, %r15d +.endif + + mulpadd 1, \arg1 + mulpadd 2, \arg1 + mulpadd 3, \arg1 + mulpadd 4, \arg1 + mulpadd 5, \arg1 + mulpadd 6, \arg1 + mulpadd 7, \arg1 + +.if (\arg1 % 8 == 0) + adcq $0, %r8 +.elseif (\arg1 % 8 == 1) + adcq $0, %r9 +.elseif (\arg1 % 8 == 2) + adcq $0, %r10 +.elseif (\arg1 % 8 == 3) + adcq $0, %r11 +.elseif (\arg1 % 8 == 4) + adcq $0, %r12 +.elseif (\arg1 % 8 == 5) + adcq $0, %r13 +.elseif (\arg1 % 8 == 6) + adcq $0, %r14 +.elseif (\arg1 % 8 == 7) + adcq $0, %r15 +.endif + + +.endm + +// ----------------------------------------------------------------------------- +// Anti-matter versions with z and m switched, and also not writing back the z +// words, but the inverses instead, *and* also adding in the z[0..7] at the +// beginning. The aim is to use this in Montgomery where we discover z[j] +// entries as we go along. +// ----------------------------------------------------------------------------- + +.macro mulpadda arg1,arg2 + mulxq 8*\arg1(m), %rax, %rbx +.if ((\arg1 + \arg2) % 8 == 0) + adcxq %rax, %r8 + adoxq %rbx, %r9 +.elseif ((\arg1 + \arg2) % 8 == 1) + adcxq %rax, %r9 + adoxq %rbx, %r10 +.elseif ((\arg1 + \arg2) % 8 == 2) + adcxq %rax, %r10 + adoxq %rbx, %r11 +.elseif ((\arg1 + \arg2) % 8 == 3) + adcxq %rax, %r11 + adoxq %rbx, %r12 +.elseif ((\arg1 + \arg2) % 8 == 4) + adcxq %rax, %r12 + adoxq %rbx, %r13 +.elseif ((\arg1 + \arg2) % 8 == 5) + adcxq %rax, %r13 + adoxq %rbx, %r14 +.elseif ((\arg1 + \arg2) % 8 == 6) + adcxq %rax, %r14 + adoxq %rbx, %r15 +.elseif ((\arg1 + \arg2) % 8 == 7) + adcxq %rax, %r15 + adoxq %rbx, %r8 +.endif + +.endm + +.macro adurowa arg1 + movq w, %rdx // Get the word-level modular inverse + xorl %eax, %eax // Get a known flag state +.if (\arg1 % 8 == 0) + mulxq %r8, %rdx, %rax +.elseif (\arg1 % 8 == 1) + mulxq %r9, %rdx, %rax +.elseif (\arg1 % 8 == 2) + mulxq %r10, %rdx, %rax +.elseif (\arg1 % 8 == 3) + mulxq %r11, %rdx, %rax +.elseif (\arg1 % 8 == 4) + mulxq %r12, %rdx, %rax +.elseif (\arg1 % 8 == 5) + mulxq %r13, %rdx, %rax +.elseif (\arg1 % 8 == 6) + mulxq %r14, %rdx, %rax +.elseif (\arg1 % 8 == 7) + mulxq %r15, %rdx, %rax +.endif + + movq %rdx, 8*\arg1(z) // Store multiplier word + + mulpadda 0, \arg1 + + // Note that the bottom reg of the window is zero by construction + // So it's safe just to use "mulpadda 7" here + + mulpadda 1, \arg1 + mulpadda 2, \arg1 + mulpadda 3, \arg1 + mulpadda 4, \arg1 + mulpadda 5, \arg1 + mulpadda 6, \arg1 + mulpadda 7, \arg1 // window lowest = 0 beforehand by construction + +.if (\arg1 % 8 == 0) + adcq $0, %r8 +.elseif (\arg1 % 8 == 1) + adcq $0, %r9 +.elseif (\arg1 % 8 == 2) + adcq $0, %r10 +.elseif (\arg1 % 8 == 3) + adcq $0, %r11 +.elseif (\arg1 % 8 == 4) + adcq $0, %r12 +.elseif (\arg1 % 8 == 5) + adcq $0, %r13 +.elseif (\arg1 % 8 == 6) + adcq $0, %r14 +.elseif (\arg1 % 8 == 7) + adcq $0, %r15 +.endif + +.endm + +.macro adurowza + movq w, %rdx // Get the word-level modular inverse + xorl %eax, %eax // Get a known flag state + + movq (z), %r8 // %r8 = zeroth word + mulxq %r8, %rdx, %rax // Compute multiplier word + movq %rdx, (z) // Store multiplier word + movq 8(z), %r9 + + mulpadda 0, 0 + movq 16(z), %r10 + mulpadda 1, 0 + movq 24(z), %r11 + mulpadda 2, 0 + movq 32(z), %r12 + mulpadda 3, 0 + movq 40(z), %r13 + mulpadda 4, 0 + movq 48(z), %r14 + mulpadda 5, 0 + movq 56(z), %r15 + mulpadda 6, 0 + mulpadda 7, 0 // r8 = 0 beforehand by construction + adcq $0, %r8 +.endm + +// ----------------------------------------------------------------------------- +// Hybrid top, doing an 8 block specially then multiple additional 8 blocks +// ----------------------------------------------------------------------------- + +// Multiply-add: z := z + x[i...i+7] * m + +.macro addrows + + adurowza + adurowa 1 + adurowa 2 + adurowa 3 + adurowa 4 + adurowa 5 + adurowa 6 + adurowa 7 + + movq z, zz + + movq k8m1, %rax + testq %rax, %rax + jz Lbignum_emontredc_8n_innerend + movq %rax, innercount +Lbignum_emontredc_8n_innerloop: + addq $64, zz + addq $64, m + addrow 0 + addrow 1 + addrow 2 + addrow 3 + addrow 4 + addrow 5 + addrow 6 + addrow 7 + subq $64, innercount + jnz Lbignum_emontredc_8n_innerloop + + movq k8m1, %rax +Lbignum_emontredc_8n_innerend: + subq %rax, m + + movq carry, %rbx + negq %rbx + adcq %r8, 64(z,%rax,1) + adcq %r9, 72(z,%rax,1) + adcq %r10, 80(z,%rax,1) + adcq %r11, 88(z,%rax,1) + adcq %r12, 96(z,%rax,1) + adcq %r13, 104(z,%rax,1) + adcq %r14, 112(z,%rax,1) + adcq %r15, 120(z,%rax,1) + movl $0, %eax + adcq $0, %rax + movq %rax, carry +.endm + +// ----------------------------------------------------------------------------- +// Main code. +// ----------------------------------------------------------------------------- + +S2N_BN_SYMBOL(bignum_emontredc_8n): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + movq %r9, %rcx +#endif + +// Save more registers to play with + + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Pre-initialize the return value to 0 just in case of early exit below + + xorl %eax, %eax + +// Divide the input k by 8, and push k8m1 = (k/8 - 1)<<6 which is used as +// the scaled inner loop counter / pointer adjustment repeatedly. Also push +// k/8 itself which is here initializing the outer loop count. + + shrq $3, %rdi + jz Lbignum_emontredc_8n_end + + leaq -1(%rdi), %rbx + shlq $6, %rbx + CFI_PUSH(%rbx) + CFI_PUSH(%rdi) + +// Make space for two more variables, and set between-stages carry to 0 + + CFI_DEC_RSP(16) + movq $0, carry + +// Copy m into its main home + + movq %rdx, m + +// Now just systematically add in the rows + +Lbignum_emontredc_8n_outerloop: + addrows + addq $64, z + subq $1, outercount + jnz Lbignum_emontredc_8n_outerloop + +// Pop the carry-out "p", which was stored at [%rsp], put in %rax for return + + CFI_POP(%rax) + +// Adjust the stack + + CFI_INC_RSP(24) + +// Reset of epilog + +Lbignum_emontredc_8n_end: + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + CFI_POP(%rbp) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_emontredc_8n) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_inv_p521.S b/cbits/s2n/x86_att/bignum_inv_p521.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_inv_p521.S @@ -0,0 +1,2093 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Modular inverse modulo p_521 = 2^521 - 1 +// Input x[9]; output z[9] +// +// extern void bignum_inv_p521(uint64_t z[static 9],const uint64_t x[static 9]); +// +// Assuming the 9-digit input x is coprime to p_521, i.e. is not divisible +// by it, returns z < p_521 such that x * z == 1 (mod p_521). Note that +// x does not need to be reduced modulo p_521, but the output always is. +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_inv_p521) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_inv_p521) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_inv_p521) + .text + .balign 32 + +// Size in bytes of a 64-bit word + +#define N 8 + +// Pointer-offset pairs for temporaries on stack + +#define f 0(%rsp) +#define g (9*N)(%rsp) +#define u (18*N)(%rsp) +#define v (27*N)(%rsp) +#define tmp (36*N)(%rsp) +#define tmp2 (37*N)(%rsp) +#define i (38*N)(%rsp) +#define d (39*N)(%rsp) + +#define mat (40*N)(%rsp) + +// Backup for the input pointer + +#define res (44*N)(%rsp) + +// Total size to reserve on the stack + +#define NSPACE 45*N + +// Syntactic variants to make x86_att version simpler to generate + +#define F 0 +#define G (9*N) +#define U (18*N) +#define V (27*N) +#define MAT (40*N) + +#define ff (%rsp) +#define gg (9*N)(%rsp) + +// Very similar to a subroutine call to the s2n-bignum word_divstep59. +// But different in register usage and returning the final matrix as +// +// [ %r8 %r10] +// [ %r12 %r14] +// +// and also returning the matrix still negated (which doesn't matter) + +#define divstep59(din,fin,gin) \ + movq din, %rsi ; \ + movq fin, %rdx ; \ + movq gin, %rcx ; \ + movq %rdx, %rbx ; \ + andq $0xfffff, %rbx ; \ + movabsq $0xfffffe0000000000, %rax ; \ + orq %rax, %rbx ; \ + andq $0xfffff, %rcx ; \ + movabsq $0xc000000000000000, %rax ; \ + orq %rax, %rcx ; \ + movq $0xfffffffffffffffe, %rax ; \ + xorl %ebp, %ebp ; \ + movl $0x2, %edx ; \ + movq %rbx, %rdi ; \ + movq %rax, %r8 ; \ + testq %rsi, %rsi ; \ + cmovs %rbp, %r8 ; \ + testq $0x1, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + sarq $1, %rcx ; \ + movl $0x100000, %eax ; \ + leaq (%rbx,%rax), %rdx ; \ + leaq (%rcx,%rax), %rdi ; \ + shlq $0x16, %rdx ; \ + shlq $0x16, %rdi ; \ + sarq $0x2b, %rdx ; \ + sarq $0x2b, %rdi ; \ + movabsq $0x20000100000, %rax ; \ + leaq (%rbx,%rax), %rbx ; \ + leaq (%rcx,%rax), %rcx ; \ + sarq $0x2a, %rbx ; \ + sarq $0x2a, %rcx ; \ + movq %rdx, MAT(%rsp) ; \ + movq %rbx, MAT+0x8(%rsp) ; \ + movq %rdi, MAT+0x10(%rsp) ; \ + movq %rcx, MAT+0x18(%rsp) ; \ + movq fin, %r12 ; \ + imulq %r12, %rdi ; \ + imulq %rdx, %r12 ; \ + movq gin, %r13 ; \ + imulq %r13, %rbx ; \ + imulq %rcx, %r13 ; \ + addq %rbx, %r12 ; \ + addq %rdi, %r13 ; \ + sarq $0x14, %r12 ; \ + sarq $0x14, %r13 ; \ + movq %r12, %rbx ; \ + andq $0xfffff, %rbx ; \ + movabsq $0xfffffe0000000000, %rax ; \ + orq %rax, %rbx ; \ + movq %r13, %rcx ; \ + andq $0xfffff, %rcx ; \ + movabsq $0xc000000000000000, %rax ; \ + orq %rax, %rcx ; \ + movq $0xfffffffffffffffe, %rax ; \ + movl $0x2, %edx ; \ + movq %rbx, %rdi ; \ + movq %rax, %r8 ; \ + testq %rsi, %rsi ; \ + cmovs %rbp, %r8 ; \ + testq $0x1, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + sarq $1, %rcx ; \ + movl $0x100000, %eax ; \ + leaq (%rbx,%rax), %r8 ; \ + leaq (%rcx,%rax), %r10 ; \ + shlq $0x16, %r8 ; \ + shlq $0x16, %r10 ; \ + sarq $0x2b, %r8 ; \ + sarq $0x2b, %r10 ; \ + movabsq $0x20000100000, %rax ; \ + leaq (%rbx,%rax), %r15 ; \ + leaq (%rcx,%rax), %r11 ; \ + sarq $0x2a, %r15 ; \ + sarq $0x2a, %r11 ; \ + movq %r13, %rbx ; \ + movq %r12, %rcx ; \ + imulq %r8, %r12 ; \ + imulq %r15, %rbx ; \ + addq %rbx, %r12 ; \ + imulq %r11, %r13 ; \ + imulq %r10, %rcx ; \ + addq %rcx, %r13 ; \ + sarq $0x14, %r12 ; \ + sarq $0x14, %r13 ; \ + movq %r12, %rbx ; \ + andq $0xfffff, %rbx ; \ + movabsq $0xfffffe0000000000, %rax ; \ + orq %rax, %rbx ; \ + movq %r13, %rcx ; \ + andq $0xfffff, %rcx ; \ + movabsq $0xc000000000000000, %rax ; \ + orq %rax, %rcx ; \ + movq MAT(%rsp), %rax ; \ + imulq %r8, %rax ; \ + movq MAT+0x10(%rsp), %rdx ; \ + imulq %r15, %rdx ; \ + imulq MAT+0x8(%rsp), %r8 ; \ + imulq MAT+0x18(%rsp), %r15 ; \ + addq %r8, %r15 ; \ + leaq (%rax,%rdx), %r9 ; \ + movq MAT(%rsp), %rax ; \ + imulq %r10, %rax ; \ + movq MAT+0x10(%rsp), %rdx ; \ + imulq %r11, %rdx ; \ + imulq MAT+0x8(%rsp), %r10 ; \ + imulq MAT+0x18(%rsp), %r11 ; \ + addq %r10, %r11 ; \ + leaq (%rax,%rdx), %r13 ; \ + movq $0xfffffffffffffffe, %rax ; \ + movl $0x2, %edx ; \ + movq %rbx, %rdi ; \ + movq %rax, %r8 ; \ + testq %rsi, %rsi ; \ + cmovs %rbp, %r8 ; \ + testq $0x1, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + sarq $1, %rcx ; \ + movl $0x100000, %eax ; \ + leaq (%rbx,%rax), %r8 ; \ + leaq (%rcx,%rax), %r12 ; \ + shlq $0x15, %r8 ; \ + shlq $0x15, %r12 ; \ + sarq $0x2b, %r8 ; \ + sarq $0x2b, %r12 ; \ + movabsq $0x20000100000, %rax ; \ + leaq (%rbx,%rax), %r10 ; \ + leaq (%rcx,%rax), %r14 ; \ + sarq $0x2b, %r10 ; \ + sarq $0x2b, %r14 ; \ + movq %r9, %rax ; \ + imulq %r8, %rax ; \ + movq %r13, %rdx ; \ + imulq %r10, %rdx ; \ + imulq %r15, %r8 ; \ + imulq %r11, %r10 ; \ + addq %r8, %r10 ; \ + leaq (%rax,%rdx), %r8 ; \ + movq %r9, %rax ; \ + imulq %r12, %rax ; \ + movq %r13, %rdx ; \ + imulq %r14, %rdx ; \ + imulq %r15, %r12 ; \ + imulq %r11, %r14 ; \ + addq %r12, %r14 ; \ + leaq (%rax,%rdx), %r12 + +S2N_BN_SYMBOL(bignum_inv_p521): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Save registers and make room for temporaries + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + + CFI_DEC_RSP(NSPACE) + +// Save the return pointer for the end so we can overwrite %rdi later + + movq %rdi, res + +// Copy the prime p_521 = 2^521 - 1 into the f variable + + xorl %eax, %eax + notq %rax + movq %rax, F(%rsp) + movq %rax, F+8(%rsp) + movq %rax, F+16(%rsp) + movq %rax, F+24(%rsp) + movq %rax, F+32(%rsp) + movq %rax, F+40(%rsp) + movq %rax, F+48(%rsp) + movq %rax, F+56(%rsp) + movl $0x1FF, %eax + movq %rax, F+64(%rsp) + +// Copy the input into the g variable, but reduce it strictly mod p_521 +// so that g <= f as assumed in the bound proof. This code fragment is +// very similar to bignum_mod_p521_9. + + movq 64(%rsi), %r8 + movl $0x1FF, %ebx + andq %r8, %rbx + shrq $9, %r8 + + stc + adcq (%rsi), %r8 + movq 8(%rsi), %r9 + adcq $0, %r9 + movq 16(%rsi), %r10 + adcq $0, %r10 + movq 24(%rsi), %r11 + adcq $0, %r11 + movq 32(%rsi), %r12 + adcq $0, %r12 + movq 40(%rsi), %r13 + adcq $0, %r13 + movq 48(%rsi), %r14 + adcq $0, %r14 + movq 56(%rsi), %r15 + adcq $0, %r15 + adcq $0, %rbx + + cmpq $512, %rbx + + sbbq $0, %r8 + movq %r8, G(%rsp) + sbbq $0, %r9 + movq %r9, G+8(%rsp) + sbbq $0, %r10 + movq %r10, G+16(%rsp) + sbbq $0, %r11 + movq %r11, G+24(%rsp) + sbbq $0, %r12 + movq %r12, G+32(%rsp) + sbbq $0, %r13 + movq %r13, G+40(%rsp) + sbbq $0, %r14 + movq %r14, G+48(%rsp) + sbbq $0, %r15 + movq %r15, G+56(%rsp) + sbbq $0, %rbx + andq $0x1FF, %rbx + movq %rbx, G+64(%rsp) + +// Also maintain weakly reduced < 2*p_521 vector [u,v] such that +// [f,g] == x * 2^{1239-59*i} * [u,v] (mod p_521) +// starting with [p_521,x] == x * 2^{1239-59*0} * [0,2^-1239] (mod p_521) +// Note that because (2^{a+521} == 2^a) (mod p_521) we simply have +// (2^-1239 == 2^324) (mod p_521) so the constant initializer is simple. +// +// Based on the standard divstep bound, for inputs <= 2^b we need at least +// n >= (9437 * b + 1) / 4096. Since b is 521, that means 1201 iterations. +// Since we package divstep in multiples of 59 bits, we do 21 blocks of 59 +// making *1239* total. (With a bit more effort we could avoid the full 59 +// divsteps and use a shorter tail computation, but we keep it simple.) +// Hence, after the 21st iteration we have [f,g] == x * [u,v] and since +// |f| = 1 we get the modular inverse from u by flipping its sign with f. + + xorl %eax, %eax + movq %rax, U(%rsp) + movq %rax, U+8(%rsp) + movq %rax, U+16(%rsp) + movq %rax, U+24(%rsp) + movq %rax, U+32(%rsp) + movq %rax, U+40(%rsp) + movq %rax, U+48(%rsp) + movq %rax, U+56(%rsp) + movq %rax, U+64(%rsp) + + movl $16, %ebx + movq %rax, V(%rsp) + movq %rax, V+8(%rsp) + movq %rax, V+16(%rsp) + movq %rax, V+24(%rsp) + movq %rax, V+32(%rsp) + movq %rbx, V+40(%rsp) + movq %rax, V+48(%rsp) + movq %rax, V+56(%rsp) + movq %rax, V+64(%rsp) + +// Start of main loop. We jump into the middle so that the divstep +// portion is common to the special 21st iteration after a uniform +// first 20. + + movq $21, i + movq $1, d + jmp Lbignum_inv_p521_midloop + +Lbignum_inv_p521_loop: + +// Separate out the matrix into sign-magnitude pairs + + movq %r8, %r9 + sarq $63, %r9 + xorq %r9, %r8 + subq %r9, %r8 + + movq %r10, %r11 + sarq $63, %r11 + xorq %r11, %r10 + subq %r11, %r10 + + movq %r12, %r13 + sarq $63, %r13 + xorq %r13, %r12 + subq %r13, %r12 + + movq %r14, %r15 + sarq $63, %r15 + xorq %r15, %r14 + subq %r15, %r14 + +// Adjust the initial values to allow for complement instead of negation +// This initial offset is the same for [f,g] and [u,v] compositions. +// Save it in temporary storage for the [u,v] part and do [f,g] first. + + movq %r8, %rax + andq %r9, %rax + movq %r10, %rdi + andq %r11, %rdi + addq %rax, %rdi + movq %rdi, tmp + + movq %r12, %rax + andq %r13, %rax + movq %r14, %rsi + andq %r15, %rsi + addq %rax, %rsi + movq %rsi, tmp2 + +// Now the computation of the updated f and g values. This maintains a +// 2-word carry between stages so we can conveniently insert the shift +// right by 59 before storing back, and not overwrite digits we need +// again of the old f and g values. +// +// Digit 0 of [f,g] + + xorl %ebx, %ebx + movq F(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rdi + adcq %rdx, %rbx + movq G(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rdi + adcq %rdx, %rbx + + xorl %ebp, %ebp + movq F(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rbp + movq G(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + +// Digit 1 of [f,g] + + xorl %ecx, %ecx + movq F+N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq G+N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + shrdq $59, %rbx, %rdi + movq %rdi, F(%rsp) + + xorl %edi, %edi + movq F+N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rbp + adcq %rdx, %rdi + movq G+N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rdi + shrdq $59, %rbp, %rsi + movq %rsi, G(%rsp) + +// Digit 2 of [f,g] + + xorl %esi, %esi + movq F+2*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rsi + movq G+2*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rsi + shrdq $59, %rcx, %rbx + movq %rbx, F+N(%rsp) + + xorl %ebx, %ebx + movq F+2*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rdi + adcq %rdx, %rbx + movq G+2*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rdi + adcq %rdx, %rbx + shrdq $59, %rdi, %rbp + movq %rbp, G+N(%rsp) + +// Digit 3 of [f,g] + + xorl %ebp, %ebp + movq F+3*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rsi + adcq %rdx, %rbp + movq G+3*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rsi + adcq %rdx, %rbp + shrdq $59, %rsi, %rcx + movq %rcx, F+2*N(%rsp) + + xorl %ecx, %ecx + movq F+3*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rbx + adcq %rdx, %rcx + movq G+3*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbx + adcq %rdx, %rcx + shrdq $59, %rbx, %rdi + movq %rdi, G+2*N(%rsp) + +// Digit 4 of [f,g] + + xorl %edi, %edi + movq F+4*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbp + adcq %rdx, %rdi + movq G+4*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbp + adcq %rdx, %rdi + shrdq $59, %rbp, %rsi + movq %rsi, F+3*N(%rsp) + + xorl %esi, %esi + movq F+4*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rcx + adcq %rdx, %rsi + movq G+4*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rcx + adcq %rdx, %rsi + shrdq $59, %rcx, %rbx + movq %rbx, G+3*N(%rsp) + +// Digit 5 of [f,g] + + xorl %ebx, %ebx + movq F+5*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rdi + adcq %rdx, %rbx + movq G+5*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rdi + adcq %rdx, %rbx + shrdq $59, %rdi, %rbp + movq %rbp, F+4*N(%rsp) + + xorl %ebp, %ebp + movq F+5*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rbp + movq G+5*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + shrdq $59, %rsi, %rcx + movq %rcx, G+4*N(%rsp) + +// Digit 6 of [f,g] + + xorl %ecx, %ecx + movq F+6*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq G+6*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + shrdq $59, %rbx, %rdi + movq %rdi, F+5*N(%rsp) + + xorl %edi, %edi + movq F+6*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rbp + adcq %rdx, %rdi + movq G+6*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rdi + shrdq $59, %rbp, %rsi + movq %rsi, G+5*N(%rsp) + +// Digit 7 of [f,g] + + xorl %esi, %esi + movq F+7*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rsi + movq G+7*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rsi + shrdq $59, %rcx, %rbx + movq %rbx, F+6*N(%rsp) + + xorl %ebx, %ebx + movq F+7*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rdi + adcq %rdx, %rbx + movq G+7*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rdi + adcq %rdx, %rbx + shrdq $59, %rdi, %rbp + movq %rbp, G+6*N(%rsp) + +// Digits 8 and 9 of [f,g] + + movq F+8*N(%rsp), %rax + xorq %r9, %rax + movq %rax, %rbp + sarq $63, %rbp + andq %r8, %rbp + negq %rbp + mulq %r8 + addq %rax, %rsi + adcq %rdx, %rbp + movq G+8*N(%rsp), %rax + xorq %r11, %rax + movq %rax, %rdx + sarq $63, %rdx + andq %r10, %rdx + subq %rdx, %rbp + mulq %r10 + addq %rax, %rsi + adcq %rdx, %rbp + shrdq $59, %rsi, %rcx + movq %rcx, F+7*N(%rsp) + shrdq $59, %rbp, %rsi + + movq F+8*N(%rsp), %rax + movq %rsi, F+8*N(%rsp) + + xorq %r13, %rax + movq %rax, %rsi + sarq $63, %rsi + andq %r12, %rsi + negq %rsi + mulq %r12 + addq %rax, %rbx + adcq %rdx, %rsi + movq G+8*N(%rsp), %rax + xorq %r15, %rax + movq %rax, %rdx + sarq $63, %rdx + andq %r14, %rdx + subq %rdx, %rsi + mulq %r14 + addq %rax, %rbx + adcq %rdx, %rsi + shrdq $59, %rbx, %rdi + movq %rdi, G+7*N(%rsp) + shrdq $59, %rsi, %rbx + movq %rbx, G+8*N(%rsp) + +// Get the initial carries back from storage and do the [u,v] accumulation + + movq tmp, %rbx + movq tmp2, %rbp + +// Digit 0 of [u,v] + + xorl %ecx, %ecx + movq U(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq V(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + + xorl %esi, %esi + movq U(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, U(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq V(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, V(%rsp) + +// Digit 1 of [u,v] + + xorl %ebx, %ebx + movq U+N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq V+N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + + xorl %ebp, %ebp + movq U+N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, U+N(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq V+N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, V+N(%rsp) + +// Digit 2 of [u,v] + + xorl %ecx, %ecx + movq U+2*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq V+2*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + + xorl %esi, %esi + movq U+2*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, U+2*N(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq V+2*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, V+2*N(%rsp) + +// Digit 3 of [u,v] + + xorl %ebx, %ebx + movq U+3*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq V+3*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + + xorl %ebp, %ebp + movq U+3*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, U+3*N(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq V+3*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, V+3*N(%rsp) + +// Digit 4 of [u,v] + + xorl %ecx, %ecx + movq U+4*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq V+4*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + + xorl %esi, %esi + movq U+4*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, U+4*N(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq V+4*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, V+4*N(%rsp) + +// Digit 5 of [u,v] + + xorl %ebx, %ebx + movq U+5*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq V+5*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + + xorl %ebp, %ebp + movq U+5*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, U+5*N(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq V+5*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, V+5*N(%rsp) + +// Digit 6 of [u,v] + + xorl %ecx, %ecx + movq U+6*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq V+6*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + + xorl %esi, %esi + movq U+6*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, U+6*N(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq V+6*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, V+6*N(%rsp) + +// Digit 7 of [u,v] + + xorl %ebx, %ebx + movq U+7*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq V+7*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + + xorl %ebp, %ebp + movq U+7*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, U+7*N(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq V+7*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, V+7*N(%rsp) + +// Digits 8 and 9 of u (top is unsigned) + + movq U+8*N(%rsp), %rax + xorq %r9, %rax + movq %r9, %rcx + andq %r8, %rcx + negq %rcx + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq V+8*N(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rcx + mulq %r10 + addq %rax, %rbx + adcq %rcx, %rdx + +// Modular reduction of u + + movq %rdx, %rax + shldq $55, %rbx, %rdx + sarq $63, %rax + addq %rax, %rdx + movq %rdx, %rax + shlq $9, %rdx + subq %rdx, %rbx + movq %rax, %rdx + sarq $63, %rax + movq U(%rsp), %rcx + addq %rdx, %rcx + movq %rcx, U(%rsp) + movq U+N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, U+N(%rsp) + movq U+2*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, U+2*N(%rsp) + movq U+3*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, U+3*N(%rsp) + movq U+4*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, U+4*N(%rsp) + movq U+5*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, U+5*N(%rsp) + movq U+6*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, U+6*N(%rsp) + movq U+7*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, U+7*N(%rsp) + adcq %rax, %rbx + +// Preload for last use of old u digit 8 + + movq U+8*N(%rsp), %rax + movq %rbx, U+8*N(%rsp) + +// Digits 8 and 9 of v (top is unsigned) + + xorq %r13, %rax + movq %r13, %rbx + andq %r12, %rbx + negq %rbx + mulq %r12 + addq %rax, %rbp + adcq %rdx, %rbx + movq V+8*N(%rsp), %rax + xorq %r15, %rax + movq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rbx + mulq %r14 + addq %rax, %rbp + adcq %rbx, %rdx + +// Modular reduction of v + + movq %rdx, %rax + shldq $55, %rbp, %rdx + sarq $63, %rax + addq %rax, %rdx + movq %rdx, %rax + shlq $9, %rdx + subq %rdx, %rbp + movq %rax, %rdx + sarq $63, %rax + movq V(%rsp), %rcx + addq %rdx, %rcx + movq %rcx, V(%rsp) + movq V+N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, V+N(%rsp) + movq V+2*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, V+2*N(%rsp) + movq V+3*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, V+3*N(%rsp) + movq V+4*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, V+4*N(%rsp) + movq V+5*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, V+5*N(%rsp) + movq V+6*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, V+6*N(%rsp) + movq V+7*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, V+7*N(%rsp) + adcq %rax, %rbp + movq %rbp, V+8*N(%rsp) + +Lbignum_inv_p521_midloop: + + divstep59(d,ff,gg) + movq %rsi, d + +// Next iteration + + decq i + jnz Lbignum_inv_p521_loop + +// The 21st and last iteration does not need anything except the +// u value and the sign of f; the latter can be obtained from the +// lowest word of f. So it's done differently from the main loop. +// Find the sign of the new f. For this we just need one digit +// since we know (for in-scope cases) that f is either +1 or -1. +// We don't explicitly shift right by 59 either, but looking at +// bit 63 (or any bit >= 60) of the unshifted result is enough +// to distinguish -1 from +1; this is then made into a mask. + + movq F(%rsp), %rax + movq G(%rsp), %rcx + imulq %r8, %rax + imulq %r10, %rcx + addq %rcx, %rax + sarq $63, %rax + +// Now separate out the matrix into sign-magnitude pairs +// and adjust each one based on the sign of f. +// +// Note that at this point we expect |f|=1 and we got its +// sign above, so then since [f,0] == x * [u,v] (mod p_521) +// we want to flip the sign of u according to that of f. + + movq %r8, %r9 + sarq $63, %r9 + xorq %r9, %r8 + subq %r9, %r8 + xorq %rax, %r9 + + movq %r10, %r11 + sarq $63, %r11 + xorq %r11, %r10 + subq %r11, %r10 + xorq %rax, %r11 + + movq %r12, %r13 + sarq $63, %r13 + xorq %r13, %r12 + subq %r13, %r12 + xorq %rax, %r13 + + movq %r14, %r15 + sarq $63, %r15 + xorq %r15, %r14 + subq %r15, %r14 + xorq %rax, %r15 + +// Adjust the initial value to allow for complement instead of negation + + movq %r8, %rax + andq %r9, %rax + movq %r10, %rbx + andq %r11, %rbx + addq %rax, %rbx + +// Digit 0 of u + + xorl %ecx, %ecx + movq U(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq V(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + movq %rbx, U(%rsp) + adcq %rdx, %rcx + +// Digit 1 of u + + xorl %ebx, %ebx + movq U+N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq V+N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + movq %rcx, U+N(%rsp) + adcq %rdx, %rbx + +// Digit 2 of u + + xorl %ecx, %ecx + movq U+2*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq V+2*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + movq %rbx, U+2*N(%rsp) + adcq %rdx, %rcx + +// Digit 3 of u + + xorl %ebx, %ebx + movq U+3*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq V+3*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + movq %rcx, U+3*N(%rsp) + adcq %rdx, %rbx + +// Digit 4 of u + + xorl %ecx, %ecx + movq U+4*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq V+4*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + movq %rbx, U+4*N(%rsp) + adcq %rdx, %rcx + +// Digit 5 of u + + xorl %ebx, %ebx + movq U+5*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq V+5*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + movq %rcx, U+5*N(%rsp) + adcq %rdx, %rbx + +// Digit 6 of u + + xorl %ecx, %ecx + movq U+6*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq V+6*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + movq %rbx, U+6*N(%rsp) + adcq %rdx, %rcx + +// Digit 7 of u + + xorl %ebx, %ebx + movq U+7*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq V+7*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + movq %rcx, U+7*N(%rsp) + adcq %rdx, %rbx + +// Digits 8 and 9 of u (top is unsigned) + + movq U+8*N(%rsp), %rax + xorq %r9, %rax + movq %r9, %rcx + andq %r8, %rcx + negq %rcx + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq V+8*N(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rcx + mulq %r10 + addq %rax, %rbx + adcq %rcx, %rdx + +// Modular reduction of u + + movq %rdx, %rax + shldq $55, %rbx, %rdx + sarq $63, %rax + addq %rax, %rdx + movq %rdx, %rax + shlq $9, %rdx + subq %rdx, %rbx + movq %rax, %rdx + sarq $63, %rax + movq U(%rsp), %rcx + addq %rdx, %rcx + movq %rcx, U(%rsp) + movq U+N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, U+N(%rsp) + movq U+2*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, U+2*N(%rsp) + movq U+3*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, U+3*N(%rsp) + movq U+4*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, U+4*N(%rsp) + movq U+5*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, U+5*N(%rsp) + movq U+6*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, U+6*N(%rsp) + movq U+7*N(%rsp), %rcx + adcq %rax, %rcx + movq %rcx, U+7*N(%rsp) + adcq %rax, %rbx + movq %rbx, U+8*N(%rsp) + +// Further strict reduction ready for the output, which just means +// a conditional subtraction of p_521 + + xorl %eax, %eax + notq %rax + movq U(%rsp), %r8 + subq %rax, %r8 + movq U+N(%rsp), %r9 + sbbq %rax, %r9 + movq U+2*N(%rsp), %r10 + sbbq %rax, %r10 + movq U+3*N(%rsp), %r11 + sbbq %rax, %r11 + movq U+4*N(%rsp), %r12 + sbbq %rax, %r12 + movq U+5*N(%rsp), %r13 + sbbq %rax, %r13 + movq U+6*N(%rsp), %r14 + sbbq %rax, %r14 + movq U+7*N(%rsp), %r15 + sbbq %rax, %r15 + movl $0x1FF, %eax + movq U+8*N(%rsp), %rbp + sbbq %rax, %rbp + + cmovcq U(%rsp), %r8 + cmovcq U+N(%rsp), %r9 + cmovcq U+2*N(%rsp), %r10 + cmovcq U+3*N(%rsp), %r11 + cmovcq U+4*N(%rsp), %r12 + cmovcq U+5*N(%rsp), %r13 + cmovcq U+6*N(%rsp), %r14 + cmovcq U+7*N(%rsp), %r15 + cmovcq U+8*N(%rsp), %rbp + +// Store it back to the final output + + movq res, %rdi + movq %r8, (%rdi) + movq %r9, N(%rdi) + movq %r10, 2*N(%rdi) + movq %r11, 3*N(%rdi) + movq %r12, 4*N(%rdi) + movq %r13, 5*N(%rdi) + movq %r14, 6*N(%rdi) + movq %r15, 7*N(%rdi) + movq %rbp, 8*N(%rdi) + +// Restore stack and registers + + CFI_INC_RSP(NSPACE) + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_inv_p521) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_kmul_16_32.S b/cbits/s2n/x86_att/bignum_kmul_16_32.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_kmul_16_32.S @@ -0,0 +1,513 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Multiply z := x * y +// Inputs x[16], y[16]; output z[32]; temporary buffer t[>=32] +// +// extern void bignum_kmul_16_32(uint64_t z[static 32], +// const uint64_t x[static 16], +// const uint64_t y[static 16], +// uint64_t t[static 32]); +// +// In this x86 code the final temporary space argument t is unused, but +// it is retained in the prototype above for API consistency with ARM. +// +// Standard x86-64 ABI: RDI = z, RSI = x, RDX = y, RCX = t +// Microsoft x64 ABI: RCX = z, RDX = x, R8 = y, R9 = t +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_kmul_16_32) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_kmul_16_32) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_kmul_16_32) + .text + +// These parameters are kept where they come in + +#define z %rdi +#define x %rsi + +// This one gets moved to free up %rdx for muls + +#define y %rcx + +// Often used for zero + +#define zero %rbp +#define zeroe %ebp + +// mulpadd i, j adds x[i] * rdx (now assumed = y[j]) into the window at i+j + +.macro mulpadd arg1,arg2 + mulxq 8*\arg1(x), %rax, %rbx +.if ((\arg1 + \arg2) % 8 == 0) + adcxq %rax, %r8 + adoxq %rbx, %r9 +.elseif ((\arg1 + \arg2) % 8 == 1) + adcxq %rax, %r9 + adoxq %rbx, %r10 +.elseif ((\arg1 + \arg2) % 8 == 2) + adcxq %rax, %r10 + adoxq %rbx, %r11 +.elseif ((\arg1 + \arg2) % 8 == 3) + adcxq %rax, %r11 + adoxq %rbx, %r12 +.elseif ((\arg1 + \arg2) % 8 == 4) + adcxq %rax, %r12 + adoxq %rbx, %r13 +.elseif ((\arg1 + \arg2) % 8 == 5) + adcxq %rax, %r13 + adoxq %rbx, %r14 +.elseif ((\arg1 + \arg2) % 8 == 6) + adcxq %rax, %r14 + adoxq %rbx, %r15 +.elseif ((\arg1 + \arg2) % 8 == 7) + adcxq %rax, %r15 + adoxq %rbx, %r8 +.endif + +.endm + +// mulpade i, j adds x[i] * rdx (now assumed = y[j]) into the window at i+j +// but re-creates the top word assuming nothing to add there + +.macro mulpade arg1,arg2 +.if ((\arg1 + \arg2) % 8 == 0) + mulxq 8*\arg1(x), %rax, %r9 + adcxq %rax, %r8 + adoxq zero, %r9 +.elseif ((\arg1 + \arg2) % 8 == 1) + mulxq 8*\arg1(x), %rax, %r10 + adcxq %rax, %r9 + adoxq zero, %r10 +.elseif ((\arg1 + \arg2) % 8 == 2) + mulxq 8*\arg1(x), %rax, %r11 + adcxq %rax, %r10 + adoxq zero, %r11 +.elseif ((\arg1 + \arg2) % 8 == 3) + mulxq 8*\arg1(x), %rax, %r12 + adcxq %rax, %r11 + adoxq zero, %r12 +.elseif ((\arg1 + \arg2) % 8 == 4) + mulxq 8*\arg1(x), %rax, %r13 + adcxq %rax, %r12 + adoxq zero, %r13 +.elseif ((\arg1 + \arg2) % 8 == 5) + mulxq 8*\arg1(x), %rax, %r14 + adcxq %rax, %r13 + adoxq zero, %r14 +.elseif ((\arg1 + \arg2) % 8 == 6) + mulxq 8*\arg1(x), %rax, %r15 + adcxq %rax, %r14 + adoxq zero, %r15 +.elseif ((\arg1 + \arg2) % 8 == 7) + mulxq 8*\arg1(x), %rax, %r8 + adcxq %rax, %r15 + adoxq zero, %r8 +.endif + +.endm + +// addrow i adds z[i] + x[0..7] * y[i] into the window + +.macro addrow arg1 + movq 8*\arg1(y), %rdx + xorl zeroe, zeroe + +.if (\arg1 % 8 == 0) + adoxq 8*\arg1(z), %r8 +.elseif (\arg1 % 8 == 1) + adoxq 8*\arg1(z), %r9 +.elseif (\arg1 % 8 == 2) + adoxq 8*\arg1(z), %r10 +.elseif (\arg1 % 8 == 3) + adoxq 8*\arg1(z), %r11 +.elseif (\arg1 % 8 == 4) + adoxq 8*\arg1(z), %r12 +.elseif (\arg1 % 8 == 5) + adoxq 8*\arg1(z), %r13 +.elseif (\arg1 % 8 == 6) + adoxq 8*\arg1(z), %r14 +.elseif (\arg1 % 8 == 7) + adoxq 8*\arg1(z), %r15 +.endif + + mulpadd 0, \arg1 + +.if (\arg1 % 8 == 0) + movq %r8, 8*\arg1(z) +.elseif (\arg1 % 8 == 1) + movq %r9, 8*\arg1(z) +.elseif (\arg1 % 8 == 2) + movq %r10, 8*\arg1(z) +.elseif (\arg1 % 8 == 3) + movq %r11, 8*\arg1(z) +.elseif (\arg1 % 8 == 4) + movq %r12, 8*\arg1(z) +.elseif (\arg1 % 8 == 5) + movq %r13, 8*\arg1(z) +.elseif (\arg1 % 8 == 6) + movq %r14, 8*\arg1(z) +.elseif (\arg1 % 8 == 7) + movq %r15, 8*\arg1(z) +.endif + + mulpadd 1, \arg1 + mulpadd 2, \arg1 + mulpadd 3, \arg1 + mulpadd 4, \arg1 + mulpadd 5, \arg1 + mulpadd 6, \arg1 + mulpade 7, \arg1 + +.if (\arg1 % 8 == 0) + adcq zero, %r8 +.elseif (\arg1 % 8 == 1) + adcq zero, %r9 +.elseif (\arg1 % 8 == 2) + adcq zero, %r10 +.elseif (\arg1 % 8 == 3) + adcq zero, %r11 +.elseif (\arg1 % 8 == 4) + adcq zero, %r12 +.elseif (\arg1 % 8 == 5) + adcq zero, %r13 +.elseif (\arg1 % 8 == 6) + adcq zero, %r14 +.elseif (\arg1 % 8 == 7) + adcq zero, %r15 +.endif + +.endm + +// Special zero version of addrow, setting up the window from scratch + +.macro addrowz + movq (y), %rdx + xorl zeroe, zeroe + + mulxq (x), %rax, %r9 + adcq %rax, (z) + + mulxq 8(x), %rax, %r10 + adcq %rax, %r9 + + mulxq 16(x), %rax, %r11 + adcq %rax, %r10 + + mulxq 24(x), %rax, %r12 + adcq %rax, %r11 + + mulxq 32(x), %rax, %r13 + adcq %rax, %r12 + + mulxq 40(x), %rax, %r14 + adcq %rax, %r13 + + mulxq 48(x), %rax, %r15 + adcq %rax, %r14 + + mulxq 56(x), %rax, %r8 + adcq %rax, %r15 + + adcq zero, %r8 +.endm + +// This is a variant where we add the initial z[0..7] at the outset. +// This makes the initialization process a bit less wasteful. By doing +// a block of 8 we get the same effect except that we add z[0..7] +// +// adurow i adds 2^{7*64} * z[i+7] + x[0..7] * y[i] into the window + +.macro adurow arg1 + movq 8*\arg1(y), %rdx + xorl zeroe, zeroe + + mulpadd 0, \arg1 + +.if (\arg1 % 8 == 0) + movq %r8, 8*\arg1(z) +.elseif (\arg1 % 8 == 1) + movq %r9, 8*\arg1(z) +.elseif (\arg1 % 8 == 2) + movq %r10, 8*\arg1(z) +.elseif (\arg1 % 8 == 3) + movq %r11, 8*\arg1(z) +.elseif (\arg1 % 8 == 4) + movq %r12, 8*\arg1(z) +.elseif (\arg1 % 8 == 5) + movq %r13, 8*\arg1(z) +.elseif (\arg1 % 8 == 6) + movq %r14, 8*\arg1(z) +.elseif (\arg1 % 8 == 7) + movq %r15, 8*\arg1(z) +.endif + + mulpadd 1, \arg1 + mulpadd 2, \arg1 + mulpadd 3, \arg1 + mulpadd 4, \arg1 + mulpadd 5, \arg1 + mulpadd 6, \arg1 + mulpade 7, \arg1 + +.if (\arg1 % 8 == 0) + adcq zero, %r8 +.elseif (\arg1 % 8 == 1) + adcq zero, %r9 +.elseif (\arg1 % 8 == 2) + adcq zero, %r10 +.elseif (\arg1 % 8 == 3) + adcq zero, %r11 +.elseif (\arg1 % 8 == 4) + adcq zero, %r12 +.elseif (\arg1 % 8 == 5) + adcq zero, %r13 +.elseif (\arg1 % 8 == 6) + adcq zero, %r14 +.elseif (\arg1 % 8 == 7) + adcq zero, %r15 +.endif + +.endm + +// Special "adurow 0" case to do first stage + +.macro adurowz + movq (y), %rdx + xorl zeroe, zeroe + + movq (z), %r8 + movq 8(z), %r9 + + mulpadd 0, 0 + movq %r8, (z) + + movq 16(z), %r10 + mulpadd 1, 0 + movq 24(z), %r11 + mulpadd 2, 0 + movq 32(z), %r12 + mulpadd 3, 0 + movq 40(z), %r13 + mulpadd 4, 0 + movq 48(z), %r14 + mulpadd 5, 0 + movq 56(z), %r15 + mulpadd 6, 0 + + mulxq 56(x), %rax, %r8 + adcxq %rax, %r15 + adoxq zero, %r8 + adcxq zero, %r8 +.endm + +// Multiply-add: z := z + x[0..7] * y + +.macro addrows + adurowz + adurow 1 + adurow 2 + adurow 3 + adurow 4 + adurow 5 + adurow 6 + adurow 7 + addrow 8 + addrow 9 + addrow 10 + addrow 11 + addrow 12 + addrow 13 + addrow 14 + addrow 15 + + movq %r8, 128(z) + movq %r9, 136(z) + movq %r10, 144(z) + movq %r11, 152(z) + movq %r12, 160(z) + movq %r13, 168(z) + movq %r14, 176(z) + movq %r15, 184(z) + +.endm + +// mulrow i adds x[0..7] * y[i] into the window +// just like addrow but no addition of z[i] + +.macro mulrow arg1 + movq 8*\arg1(y), %rdx + xorl zeroe, zeroe + + mulpadd 0, \arg1 + +.if (\arg1 % 8 == 0) + movq %r8, 8*\arg1(z) +.elseif (\arg1 % 8 == 1) + movq %r9, 8*\arg1(z) +.elseif (\arg1 % 8 == 2) + movq %r10, 8*\arg1(z) +.elseif (\arg1 % 8 == 3) + movq %r11, 8*\arg1(z) +.elseif (\arg1 % 8 == 4) + movq %r12, 8*\arg1(z) +.elseif (\arg1 % 8 == 5) + movq %r13, 8*\arg1(z) +.elseif (\arg1 % 8 == 6) + movq %r14, 8*\arg1(z) +.elseif (\arg1 % 8 == 7) + movq %r15, 8*\arg1(z) +.endif + + mulpadd 1, \arg1 + mulpadd 2, \arg1 + mulpadd 3, \arg1 + mulpadd 4, \arg1 + mulpadd 5, \arg1 + mulpadd 6, \arg1 + mulpade 7, \arg1 + +.if (\arg1 % 8 == 0) + adcq zero, %r8 +.elseif (\arg1 % 8 == 1) + adcq zero, %r9 +.elseif (\arg1 % 8 == 2) + adcq zero, %r10 +.elseif (\arg1 % 8 == 3) + adcq zero, %r11 +.elseif (\arg1 % 8 == 4) + adcq zero, %r12 +.elseif (\arg1 % 8 == 5) + adcq zero, %r13 +.elseif (\arg1 % 8 == 6) + adcq zero, %r14 +.elseif (\arg1 % 8 == 7) + adcq zero, %r15 +.endif + + +.endm + +// Special zero version of mulrow, setting up the window from scratch + +.macro mulrowz + movq (y), %rdx + xorl zeroe, zeroe + + mulxq (x), %rax, %r9 + movq %rax, (z) + + mulxq 8(x), %rax, %r10 + adcxq %rax, %r9 + + mulxq 16(x), %rax, %r11 + adcxq %rax, %r10 + + mulxq 24(x), %rax, %r12 + adcxq %rax, %r11 + + mulxq 32(x), %rax, %r13 + adcxq %rax, %r12 + + mulxq 40(x), %rax, %r14 + adcxq %rax, %r13 + + mulxq 48(x), %rax, %r15 + adcxq %rax, %r14 + + mulxq 56(x), %rax, %r8 + adcxq %rax, %r15 + + adcq zero, %r8 +.endm + +// Multiply-add: z := x[0..7] * y plus window + +.macro mulrows + mulrowz + mulrow 1 + mulrow 2 + mulrow 3 + mulrow 4 + mulrow 5 + mulrow 6 + mulrow 7 + + mulrow 8 + mulrow 9 + mulrow 10 + mulrow 11 + mulrow 12 + mulrow 13 + mulrow 14 + mulrow 15 + + movq %r8, 128(z) + movq %r9, 136(z) + movq %r10, 144(z) + movq %r11, 152(z) + movq %r12, 160(z) + movq %r13, 168(z) + movq %r14, 176(z) + movq %r15, 184(z) + +.endm + + +S2N_BN_SYMBOL(bignum_kmul_16_32): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx +#endif + +// Save more registers to play with + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Move y into its permanent home, freeing up %rdx for its special role in muls + + movq %rdx, y + +// Do the zeroth row as a pure product then the next as multiply-add + + mulrows + + addq $64, z + addq $64, x + addrows + +// Restore registers and return + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_kmul_16_32) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_kmul_32_64.S b/cbits/s2n/x86_att/bignum_kmul_32_64.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_kmul_32_64.S @@ -0,0 +1,1161 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Multiply z := x * y +// Inputs x[32], y[32]; output z[64]; temporary buffer t[>=96] +// +// extern void bignum_kmul_32_64(uint64_t z[static 64], +// const uint64_t x[static 32], +// const uint64_t y[static 32], +// uint64_t t[static 96]); +// +// This is a Karatsuba-style function multiplying half-sized results +// internally and using temporary buffer t for intermediate results. The size +// of 96 is an overstatement for compatibility with the ARM version; it +// actually only uses 65 elements of t (64 + 1 for a stashed sign). +// +// Standard x86-64 ABI: RDI = z, RSI = x, RDX = y, RCX = t +// Microsoft x64 ABI: RCX = z, RDX = x, R8 = y, R9 = t +// ----------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_kmul_32_64) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_kmul_32_64) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_kmul_32_64) + + + .text + +#define K 16 + +#define z %rdi +#define x %rsi +#define y %rcx + +#define s %r9 + +// We re-use the y variable to point at t later on, when this seems clearer + +#define t %rcx + +S2N_BN_SYMBOL(bignum_kmul_32_64): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + movq %r9, %rcx +#endif + +// Save callee-saved registers and also push t onto the stack; we'll +// use this space to back up both t and later z. Then move the y variable +// into its longer-term home for the first few stages. + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + + CFI_PUSH(%rcx) + movq %rdx, y + +// Multiply the low halves + + CFI_CALL(Lbignum_kmul_32_64_local_bignum_kmul_16_32) + +// Multiply the high halves + + leaq 16*K-0x40(%rdi), %rdi + leaq 8*K-0x40(%rsi), %rsi + leaq 8*K(%rcx), %rcx + CFI_CALL(Lbignum_kmul_32_64_local_bignum_kmul_16_32) + +// Establish %r8 as the t pointer and use the cell to back up z now + + movq (%rsp), %r8 + subq $16*K+0x40, %rdi + movq %rdi, (%rsp) + +// Form |x_lo - x_hi| starting at t + + movq -8*K-0x40(%rsi), %rax + subq -8*K-0x40+8*K(%rsi), %rax + movq %rax, (%r8) + .set I, 1 + .rep K-1 + movq -8*K-0x40+8*I(%rsi), %rax + sbbq -8*K-0x40+8*K+8*I(%rsi), %rax + movq %rax, 8*I(%r8) + .set I, (I+1) + .endr + + movl $0, %ebx + sbbq s, s // Maintain CF, set ZF for cmovs, record sign + + .set I, 0 + .rep K + movq 8*I(%r8), %rdx + movq %rdx, %rax + notq %rdx + cmovzq %rax, %rdx + adcxq %rbx, %rdx + movq %rdx, 8*I(%r8) + .set I, (I+1) + .endr + +// Form |y_hi - y_lo| (note opposite order) starting at t[K] + + movq -8*K+8*K(%rcx), %rax + subq -8*K(%rcx), %rax + movq %rax, 8*K(%r8) + .set I, 1 + .rep K-1 + movq -8*K+8*K+8*I(%rcx), %rax + sbbq -8*K+8*I(%rcx), %rax + movq %rax, 8*K+8*I(%r8) + .set I, (I+1) + .endr + + movl $0, %ebx + sbbq %rbp, %rbp // Maintain CF, set ZF for cmovs + + .set I, 0 + .rep K + movq 8*K+8*I(%r8), %rdx + movq %rdx, %rax + notq %rdx + cmovzq %rax, %rdx + adcxq %rbx, %rdx + movq %rdx, 8*K+8*I(%r8) + .set I, (I+1) + .endr + +// Stash the final sign with which to add things at t[4*K] + + xorq %rbp, s + movq s, 32*K(%r8) + +// Multiply the absolute differences, putting the result at t[2*K] +// This has the side-effect of putting t in the "right" register %rcx +// so after the load of z, we have both z and t pointers straight. + + movq %r8, %rcx + leaq 8*K(%r8), %rsi + leaq 16*K(%r8), %rdi + CFI_CALL(Lbignum_kmul_32_64_local_bignum_kmul_16_32) + movq (%rsp), z + +// Compose the middle parts [2,1] + [1,0] + [3,2], saving carry in %rbx. +// Put the sum at t, overwriting the absolute differences we no longer need. + + xorl %ebx, %ebx + .set I, 0 + .rep 2*K + movq 8*K+8*I(z), %rax + adcxq 8*I(z), %rax + adoxq 16*K+8*I(z), %rax + movq %rax, 8*I(t) + .set I, (I+1) + .endr + adoxq %rbx, %rbx + adcq $0, %rbx + +// Sign-aware addition or subtraction of the complicated term. +// We double-negate it to set CF/ZF while not spoiling its +// actual form: note that we eventually adcx to it below. + + movq 32*K(t), s + negq s + negq s + + .set I, 0 + .rep 2*K + movq 16*K+8*I(t), %rdx + movq %rdx, %rax + notq %rdx + cmovzq %rax, %rdx + adcxq 8*I(t), %rdx + movq %rdx, 8*K+8*I(z) + .set I, (I+1) + .endr + +// Bump the accumulated carry. This must end up >= 0 because it's the top +// word of a value of the form ... + h * h' + l * l' - (h - l) * (h' - l') >= 0 + + adcxq s, %rbx + +// Finally propagate the carry to the top part + + xorl %eax, %eax + addq %rbx, 24*K(z) + .set I, 1 + .rep K-1 + adcq %rax, 24*K+8*I(z) + .set I, (I+1) + .endr + +// Restore and return. The first pop is not needed for the ABI but +// we need to adjust the stack anyway so it seems reasonable. + + CFI_POP(%rcx) + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +// Local copy of half-length subroutine. This has a slightly different +// interface, expecting y argument in %rcx directly, and not doing any +// save-restore of the other registers. It naturally moves z and x on by +// 0x40, which we compensate for when it is called by adjusting offsets. + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lbignum_kmul_32_64_local_bignum_kmul_16_32) + +Lbignum_kmul_32_64_local_bignum_kmul_16_32: + CFI_START + movq (%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %r9 + movq %rax, (%rdi) + mulxq 0x8(%rsi), %rax, %r10 + adcxq %rax, %r9 + mulxq 0x10(%rsi), %rax, %r11 + adcxq %rax, %r10 + mulxq 0x18(%rsi), %rax, %r12 + adcxq %rax, %r11 + mulxq 0x20(%rsi), %rax, %r13 + adcxq %rax, %r12 + mulxq 0x28(%rsi), %rax, %r14 + adcxq %rax, %r13 + mulxq 0x30(%rsi), %rax, %r15 + adcxq %rax, %r14 + mulxq 0x38(%rsi), %rax, %r8 + adcxq %rax, %r15 + adcq %rbp, %r8 + movq 0x8(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + movq %r9, 0x8(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x38(%rsi), %rax, %r9 + adcxq %rax, %r8 + adoxq %rbp, %r9 + adcq %rbp, %r9 + movq 0x10(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq %r10, 0x10(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x38(%rsi), %rax, %r10 + adcxq %rax, %r9 + adoxq %rbp, %r10 + adcq %rbp, %r10 + movq 0x18(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq %r11, 0x18(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x38(%rsi), %rax, %r11 + adcxq %rax, %r10 + adoxq %rbp, %r11 + adcq %rbp, %r11 + movq 0x20(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq %r12, 0x20(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x38(%rsi), %rax, %r12 + adcxq %rax, %r11 + adoxq %rbp, %r12 + adcq %rbp, %r12 + movq 0x28(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + movq %r13, 0x28(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x38(%rsi), %rax, %r13 + adcxq %rax, %r12 + adoxq %rbp, %r13 + adcq %rbp, %r13 + movq 0x30(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + movq %r14, 0x30(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x38(%rsi), %rax, %r14 + adcxq %rax, %r13 + adoxq %rbp, %r14 + adcq %rbp, %r14 + movq 0x38(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + movq %r15, 0x38(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x38(%rsi), %rax, %r15 + adcxq %rax, %r14 + adoxq %rbp, %r15 + adcq %rbp, %r15 + movq 0x40(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + movq %r8, 0x40(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x38(%rsi), %rax, %r8 + adcxq %rax, %r15 + adoxq %rbp, %r8 + adcq %rbp, %r8 + movq 0x48(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + movq %r9, 0x48(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x38(%rsi), %rax, %r9 + adcxq %rax, %r8 + adoxq %rbp, %r9 + adcq %rbp, %r9 + movq 0x50(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq %r10, 0x50(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x38(%rsi), %rax, %r10 + adcxq %rax, %r9 + adoxq %rbp, %r10 + adcq %rbp, %r10 + movq 0x58(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq %r11, 0x58(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x38(%rsi), %rax, %r11 + adcxq %rax, %r10 + adoxq %rbp, %r11 + adcq %rbp, %r11 + movq 0x60(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq %r12, 0x60(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x38(%rsi), %rax, %r12 + adcxq %rax, %r11 + adoxq %rbp, %r12 + adcq %rbp, %r12 + movq 0x68(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + movq %r13, 0x68(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x38(%rsi), %rax, %r13 + adcxq %rax, %r12 + adoxq %rbp, %r13 + adcq %rbp, %r13 + movq 0x70(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + movq %r14, 0x70(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x38(%rsi), %rax, %r14 + adcxq %rax, %r13 + adoxq %rbp, %r14 + adcq %rbp, %r14 + movq 0x78(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + movq %r15, 0x78(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x38(%rsi), %rax, %r15 + adcxq %rax, %r14 + adoxq %rbp, %r15 + adcq %rbp, %r15 + movq %r8, 0x80(%rdi) + movq %r9, 0x88(%rdi) + movq %r10, 0x90(%rdi) + movq %r11, 0x98(%rdi) + movq %r12, 0xa0(%rdi) + movq %r13, 0xa8(%rdi) + movq %r14, 0xb0(%rdi) + movq %r15, 0xb8(%rdi) + addq $0x40, %rdi + addq $0x40, %rsi + movq (%rcx), %rdx + xorl %ebp, %ebp + movq (%rdi), %r8 + movq 0x8(%rdi), %r9 + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + movq %r8, (%rdi) + movq 0x10(%rdi), %r10 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + movq 0x18(%rdi), %r11 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq 0x20(%rdi), %r12 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x28(%rdi), %r13 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq 0x30(%rdi), %r14 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + movq 0x38(%rdi), %r15 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x38(%rsi), %rax, %r8 + adcxq %rax, %r15 + adoxq %rbp, %r8 + adcxq %rbp, %r8 + movq 0x8(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + movq %r9, 0x8(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x38(%rsi), %rax, %r9 + adcxq %rax, %r8 + adoxq %rbp, %r9 + adcq %rbp, %r9 + movq 0x10(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq %r10, 0x10(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x38(%rsi), %rax, %r10 + adcxq %rax, %r9 + adoxq %rbp, %r10 + adcq %rbp, %r10 + movq 0x18(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq %r11, 0x18(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x38(%rsi), %rax, %r11 + adcxq %rax, %r10 + adoxq %rbp, %r11 + adcq %rbp, %r11 + movq 0x20(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq %r12, 0x20(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x38(%rsi), %rax, %r12 + adcxq %rax, %r11 + adoxq %rbp, %r12 + adcq %rbp, %r12 + movq 0x28(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + movq %r13, 0x28(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x38(%rsi), %rax, %r13 + adcxq %rax, %r12 + adoxq %rbp, %r13 + adcq %rbp, %r13 + movq 0x30(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + movq %r14, 0x30(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x38(%rsi), %rax, %r14 + adcxq %rax, %r13 + adoxq %rbp, %r14 + adcq %rbp, %r14 + movq 0x38(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + movq %r15, 0x38(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x38(%rsi), %rax, %r15 + adcxq %rax, %r14 + adoxq %rbp, %r15 + adcq %rbp, %r15 + movq 0x40(%rcx), %rdx + xorl %ebp, %ebp + adoxq 0x40(%rdi), %r8 + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + movq %r8, 0x40(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x38(%rsi), %rax, %r8 + adcxq %rax, %r15 + adoxq %rbp, %r8 + adcq %rbp, %r8 + movq 0x48(%rcx), %rdx + xorl %ebp, %ebp + adoxq 0x48(%rdi), %r9 + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + movq %r9, 0x48(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x38(%rsi), %rax, %r9 + adcxq %rax, %r8 + adoxq %rbp, %r9 + adcq %rbp, %r9 + movq 0x50(%rcx), %rdx + xorl %ebp, %ebp + adoxq 0x50(%rdi), %r10 + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq %r10, 0x50(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x38(%rsi), %rax, %r10 + adcxq %rax, %r9 + adoxq %rbp, %r10 + adcq %rbp, %r10 + movq 0x58(%rcx), %rdx + xorl %ebp, %ebp + adoxq 0x58(%rdi), %r11 + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq %r11, 0x58(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x38(%rsi), %rax, %r11 + adcxq %rax, %r10 + adoxq %rbp, %r11 + adcq %rbp, %r11 + movq 0x60(%rcx), %rdx + xorl %ebp, %ebp + adoxq 0x60(%rdi), %r12 + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq %r12, 0x60(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x38(%rsi), %rax, %r12 + adcxq %rax, %r11 + adoxq %rbp, %r12 + adcq %rbp, %r12 + movq 0x68(%rcx), %rdx + xorl %ebp, %ebp + adoxq 0x68(%rdi), %r13 + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + movq %r13, 0x68(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x38(%rsi), %rax, %r13 + adcxq %rax, %r12 + adoxq %rbp, %r13 + adcq %rbp, %r13 + movq 0x70(%rcx), %rdx + xorl %ebp, %ebp + adoxq 0x70(%rdi), %r14 + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + movq %r14, 0x70(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x38(%rsi), %rax, %r14 + adcxq %rax, %r13 + adoxq %rbp, %r14 + adcq %rbp, %r14 + movq 0x78(%rcx), %rdx + xorl %ebp, %ebp + adoxq 0x78(%rdi), %r15 + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + movq %r15, 0x78(%rdi) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x38(%rsi), %rax, %r15 + adcxq %rax, %r14 + adoxq %rbp, %r15 + adcq %rbp, %r15 + movq %r8, 0x80(%rdi) + movq %r9, 0x88(%rdi) + movq %r10, 0x90(%rdi) + movq %r11, 0x98(%rdi) + movq %r12, 0xa0(%rdi) + movq %r13, 0xa8(%rdi) + movq %r14, 0xb0(%rdi) + movq %r15, 0xb8(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lbignum_kmul_32_64_local_bignum_kmul_16_32) + +S2N_BN_SIZE_DIRECTIVE(bignum_kmul_32_64) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_ksqr_16_32.S b/cbits/s2n/x86_att/bignum_ksqr_16_32.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_ksqr_16_32.S @@ -0,0 +1,545 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Square, z := x^2 +// Input x[16]; output z[32]; temporary buffer t[>=24] +// +// extern void bignum_ksqr_16_32(uint64_t z[static 32], +// const uint64_t x[static 16], +// uint64_t t[static 24]); +// +// In this x86 code the final temporary space argument t is unused, but +// it is retained in the prototype above for API consistency with ARM. +// +// Standard x86-64 ABI: RDI = z, RSI = x, RDX = t +// Microsoft x64 ABI: RCX = z, RDX = x, R8 = t +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_ksqr_16_32) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_ksqr_16_32) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_ksqr_16_32) + .text + +#define z %rdi +#define x %rsi + +// A zero register + +#define zero %rbp +#define zeroe %ebp + +// ------------------------------------------------------------------------ +// mulpadd i, j adds rdx * x[i] into the window at the i+j point +// ------------------------------------------------------------------------ + +.macro mulpadd arg1,arg2 + mulxq 8*\arg1(x), %rax, %rcx +.if ((\arg1 + \arg2) % 8 == 0) + adcxq %rax, %r8 + adoxq %rcx, %r9 +.elseif ((\arg1 + \arg2) % 8 == 1) + adcxq %rax, %r9 + adoxq %rcx, %r10 +.elseif ((\arg1 + \arg2) % 8 == 2) + adcxq %rax, %r10 + adoxq %rcx, %r11 +.elseif ((\arg1 + \arg2) % 8 == 3) + adcxq %rax, %r11 + adoxq %rcx, %r12 +.elseif ((\arg1 + \arg2) % 8 == 4) + adcxq %rax, %r12 + adoxq %rcx, %r13 +.elseif ((\arg1 + \arg2) % 8 == 5) + adcxq %rax, %r13 + adoxq %rcx, %r14 +.elseif ((\arg1 + \arg2) % 8 == 6) + adcxq %rax, %r14 + adoxq %rcx, %r15 +.elseif ((\arg1 + \arg2) % 8 == 7) + adcxq %rax, %r15 + adoxq %rcx, %r8 +.endif + +.endm + +// ------------------------------------------------------------------------ +// mulpade i, j adds rdx * x[i] into the window at i+j +// but re-creates the top word assuming nothing to add there +// ------------------------------------------------------------------------ + +.macro mulpade arg1,arg2 +.if ((\arg1 + \arg2) % 8 == 0) + mulxq 8*\arg1(x), %rax, %r9 + adcxq %rax, %r8 + adoxq zero, %r9 +.elseif ((\arg1 + \arg2) % 8 == 1) + mulxq 8*\arg1(x), %rax, %r10 + adcxq %rax, %r9 + adoxq zero, %r10 +.elseif ((\arg1 + \arg2) % 8 == 2) + mulxq 8*\arg1(x), %rax, %r11 + adcxq %rax, %r10 + adoxq zero, %r11 +.elseif ((\arg1 + \arg2) % 8 == 3) + mulxq 8*\arg1(x), %rax, %r12 + adcxq %rax, %r11 + adoxq zero, %r12 +.elseif ((\arg1 + \arg2) % 8 == 4) + mulxq 8*\arg1(x), %rax, %r13 + adcxq %rax, %r12 + adoxq zero, %r13 +.elseif ((\arg1 + \arg2) % 8 == 5) + mulxq 8*\arg1(x), %rax, %r14 + adcxq %rax, %r13 + adoxq zero, %r14 +.elseif ((\arg1 + \arg2) % 8 == 6) + mulxq 8*\arg1(x), %rax, %r15 + adcxq %rax, %r14 + adoxq zero, %r15 +.elseif ((\arg1 + \arg2) % 8 == 7) + mulxq 8*\arg1(x), %rax, %r8 + adcxq %rax, %r15 + adoxq zero, %r8 +.endif + +.endm + +// ------------------------------------------------------------------------ +// addrow i,j adds z[i+j] + x[i..i+7] * x[j] into the window +// ------------------------------------------------------------------------ + +.macro addrow arg1,arg2 + movq 8*\arg2(x), %rdx + xorl zeroe, zeroe // Get a known flag state and give a zero reg + +.if ((\arg1 + \arg2) % 8 == 0) + adoxq 8*(\arg1+\arg2)(z), %r8 +.elseif ((\arg1 + \arg2) % 8 == 1) + adoxq 8*(\arg1+\arg2)(z), %r9 +.elseif ((\arg1 + \arg2) % 8 == 2) + adoxq 8*(\arg1+\arg2)(z), %r10 +.elseif ((\arg1 + \arg2) % 8 == 3) + adoxq 8*(\arg1+\arg2)(z), %r11 +.elseif ((\arg1 + \arg2) % 8 == 4) + adoxq 8*(\arg1+\arg2)(z), %r12 +.elseif ((\arg1 + \arg2) % 8 == 5) + adoxq 8*(\arg1+\arg2)(z), %r13 +.elseif ((\arg1 + \arg2) % 8 == 6) + adoxq 8*(\arg1+\arg2)(z), %r14 +.elseif ((\arg1 + \arg2) % 8 == 7) + adoxq 8*(\arg1+\arg2)(z), %r15 +.endif + + mulpadd \arg1, \arg2 + +.if ((\arg1 + \arg2) % 8 == 0) + movq %r8, 8*(\arg1+\arg2)(z) +.elseif ((\arg1 + \arg2) % 8 == 1) + movq %r9, 8*(\arg1+\arg2)(z) +.elseif ((\arg1 + \arg2) % 8 == 2) + movq %r10, 8*(\arg1+\arg2)(z) +.elseif ((\arg1 + \arg2) % 8 == 3) + movq %r11, 8*(\arg1+\arg2)(z) +.elseif ((\arg1 + \arg2) % 8 == 4) + movq %r12, 8*(\arg1+\arg2)(z) +.elseif ((\arg1 + \arg2) % 8 == 5) + movq %r13, 8*(\arg1+\arg2)(z) +.elseif ((\arg1 + \arg2) % 8 == 6) + movq %r14, 8*(\arg1+\arg2)(z) +.elseif ((\arg1 + \arg2) % 8 == 7) + movq %r15, 8*(\arg1+\arg2)(z) +.endif + + mulpadd (\arg1+1), \arg2 + mulpadd (\arg1+2), \arg2 + mulpadd (\arg1+3), \arg2 + mulpadd (\arg1+4), \arg2 + mulpadd (\arg1+5), \arg2 + mulpade (\arg1+6), \arg2 + mulpade (\arg1+7), \arg2 + +.if ((\arg1 + \arg2) % 8 == 0) + adcxq zero, %r8 +.elseif ((\arg1 + \arg2) % 8 == 1) + adcxq zero, %r9 +.elseif ((\arg1 + \arg2) % 8 == 2) + adcxq zero, %r10 +.elseif ((\arg1 + \arg2) % 8 == 3) + adcxq zero, %r11 +.elseif ((\arg1 + \arg2) % 8 == 4) + adcxq zero, %r12 +.elseif ((\arg1 + \arg2) % 8 == 5) + adcxq zero, %r13 +.elseif ((\arg1 + \arg2) % 8 == 6) + adcxq zero, %r14 +.elseif ((\arg1 + \arg2) % 8 == 7) + adcxq zero, %r15 +.endif + + +.endm + + +// ------------------------------------------------------------------------ +// Adds off-diagonal part of x[i..i+7]^2 into the window, writes 0..7 back +// ------------------------------------------------------------------------ + +.macro sqr arg1 + + xorl zeroe, zeroe + +// Set up the initial window + + movq 16*\arg1+8(z), %r9 + movq 16*\arg1+16(z), %r10 + movq 16*\arg1+24(z), %r11 + movq 16*\arg1+32(z), %r12 + movq 16*\arg1+40(z), %r13 + movq 16*\arg1+48(z), %r14 + movq 16*\arg1+56(z), %r15 + +// Add in the first diagonal [%r8..%r10] + 2 wb = 10 + 20 + 30 + 40 + 50 + 60 + 70 + + movq 8*\arg1(x), %rdx + mulpadd (\arg1+1), (\arg1+0) + movq %r9, 16*\arg1+8(z) + mulpadd (\arg1+2), (\arg1+0) + movq %r10, 16*\arg1+16(z) + mulpadd (\arg1+3), (\arg1+0) + mulpadd (\arg1+4), (\arg1+0) + mulpadd (\arg1+5), (\arg1+0) + mulpadd (\arg1+6), (\arg1+0) + mulpade (\arg1+7), (\arg1+0) + adcxq zero, %r8 + +// Add in the next diagonal = 21 + 31 + 41 + 51 + 61 + 71 + 54 + + xorl zeroe, zeroe + movq 8*\arg1+8(x), %rdx + mulpadd (\arg1+2), (\arg1+1) + movq %r11, 16*\arg1+24(z) + mulpadd (\arg1+3), (\arg1+1) + movq %r12, 16*\arg1+32(z) + mulpadd (\arg1+4), (\arg1+1) + mulpadd (\arg1+5), (\arg1+1) + mulpadd (\arg1+6), (\arg1+1) + mulpade (\arg1+7), (\arg1+1) + movq 8*\arg1+32(x), %rdx + mulpade (\arg1+5), (\arg1+4) + adcxq zero, %r10 + +// And the next one = 32 + 42 + 52 + 62 + 72 + 64 + 65 + + xorl zeroe, zeroe + movq 8*\arg1+16(x), %rdx + mulpadd (\arg1+3), (\arg1+2) + movq %r13, 16*\arg1+40(z) + mulpadd (\arg1+4), (\arg1+2) + movq %r14, 16*\arg1+48(z) + mulpadd (\arg1+5), (\arg1+2) + mulpadd (\arg1+6), (\arg1+2) + mulpadd (\arg1+7), (\arg1+2) + movq 8*\arg1+48(x), %rdx + mulpade (\arg1+4), (\arg1+6) + mulpade (\arg1+5), (\arg1+6) + adcxq zero, %r12 + +// And the final one = 43 + 53 + 63 + 73 + 74 + 75 + 76 + + xorl zeroe, zeroe + movq 8*\arg1+24(x), %rdx + mulpadd (\arg1+4), (\arg1+3) + movq %r15, 16*\arg1+56(z) + mulpadd (\arg1+5), (\arg1+3) + mulpadd (\arg1+6), (\arg1+3) + mulpadd (\arg1+7), (\arg1+3) + movq 8*\arg1+56(x), %rdx + mulpadd (\arg1+4), (\arg1+7) + mulpade (\arg1+5), (\arg1+7) + mulpade (\arg1+6), (\arg1+7) + adcxq zero, %r14 +.endm + +// ------------------------------------------------------------------------ +// Multiply-add: z := z + x[i...i+7] * x +// ------------------------------------------------------------------------ + +.macro addrows arg1 + + sqr \arg1 + + .set I, (\arg1+8) +.rep (8-\arg1) + addrow \arg1, I + .set I, (I+1) +.endr + + movq %r8, 8*(16+\arg1)(z) + movq %r9, 8*(17+\arg1)(z) + movq %r10, 8*(18+\arg1)(z) + movq %r11, 8*(19+\arg1)(z) + movq %r12, 8*(20+\arg1)(z) + movq %r13, 8*(21+\arg1)(z) + movq %r14, 8*(22+\arg1)(z) +.endm + + +// ------------------------------------------------------------------------ +// mulrow i,j adds x[i..i+7] * x[j] into the window +// just like addrow but no addition of z[i+j] +// ------------------------------------------------------------------------ + +.macro mulrow arg1,arg2 + movq 8*\arg2(x), %rdx + xorl zeroe, zeroe // Get a known flag state and give a zero reg + + mulpadd \arg1, \arg2 + +.if ((\arg1 + \arg2) % 8 == 0) + movq %r8, 8*(\arg1+\arg2)(z) +.elseif ((\arg1 + \arg2) % 8 == 1) + movq %r9, 8*(\arg1+\arg2)(z) +.elseif ((\arg1 + \arg2) % 8 == 2) + movq %r10, 8*(\arg1+\arg2)(z) +.elseif ((\arg1 + \arg2) % 8 == 3) + movq %r11, 8*(\arg1+\arg2)(z) +.elseif ((\arg1 + \arg2) % 8 == 4) + movq %r12, 8*(\arg1+\arg2)(z) +.elseif ((\arg1 + \arg2) % 8 == 5) + movq %r13, 8*(\arg1+\arg2)(z) +.elseif ((\arg1 + \arg2) % 8 == 6) + movq %r14, 8*(\arg1+\arg2)(z) +.elseif ((\arg1 + \arg2) % 8 == 7) + movq %r15, 8*(\arg1+\arg2)(z) +.endif + + mulpadd (\arg1+1), \arg2 + mulpadd (\arg1+2), \arg2 + mulpadd (\arg1+3), \arg2 + mulpadd (\arg1+4), \arg2 + mulpadd (\arg1+5), \arg2 +.if ((\arg1 + \arg2) % 8 == 0) + mulpade (\arg1+6), \arg2 +.else + mulpadd (\arg1+6), \arg2 +.endif + + mulpade (\arg1+7), \arg2 + +.if ((\arg1 + \arg2) % 8 == 0) + adcxq zero, %r8 +.elseif ((\arg1 + \arg2) % 8 == 1) + adcxq zero, %r9 +.elseif ((\arg1 + \arg2) % 8 == 2) + adcxq zero, %r10 +.elseif ((\arg1 + \arg2) % 8 == 3) + adcxq zero, %r11 +.elseif ((\arg1 + \arg2) % 8 == 4) + adcxq zero, %r12 +.elseif ((\arg1 + \arg2) % 8 == 5) + adcxq zero, %r13 +.elseif ((\arg1 + \arg2) % 8 == 6) + adcxq zero, %r14 +.elseif ((\arg1 + \arg2) % 8 == 7) + adcxq zero, %r15 +.endif + + +.endm + +// ------------------------------------------------------------------------ +// Compute off-diagonal part of x[0..7]^2, write back 1..7 elements and +// set up the high part in the standard register window. DOES NOT WRITE z[0]! +// ------------------------------------------------------------------------ + +.macro sqrz + + xorl zeroe, zeroe + +// Set initial window [%r8..%r10] + 2 wb = 10 + 20 + 30 + 40 + 50 + 60 + 70 + + movq (x), %rdx + mulxq 8(x), %r9, %rax + movq %r9, 8(z) + mulxq 16(x), %r10, %rcx + adcxq %rax, %r10 + movq %r10, 16(z) + mulxq 24(x), %r11, %rax + adcxq %rcx, %r11 + mulxq 32(x), %r12, %rcx + adcxq %rax, %r12 + mulxq 40(x), %r13, %rax + adcxq %rcx, %r13 + mulxq 48(x), %r14, %rcx + adcxq %rax, %r14 + mulxq 56(x), %r15, %r8 + adcxq %rcx, %r15 + adcxq zero, %r8 + +// Add in the next diagonal = 21 + 31 + 41 + 51 + 61 + 71 + 54 + + xorl zeroe, zeroe + movq 8(x), %rdx + mulpadd 2, 1 + movq %r11, 24(z) + mulpadd 3, 1 + movq %r12, 32(z) + mulpadd 4, 1 + mulpadd 5, 1 + mulpadd 6, 1 + mulpade 7, 1 + movq 32(x), %rdx + mulpade 5, 4 + adcxq zero, %r10 + +// And the next one = 32 + 42 + 52 + 62 + 72 + 64 + 65 + + xorl zeroe, zeroe + movq 16(x), %rdx + mulpadd 3, 2 + movq %r13, 40(z) + mulpadd 4, 2 + movq %r14, 48(z) + mulpadd 5, 2 + mulpadd 6, 2 + mulpadd 7, 2 + movq 48(x), %rdx + mulpade 4, 6 + mulpade 5, 6 + adcxq zero, %r12 + +// And the final one = 43 + 53 + 63 + 73 + 74 + 75 + 76 + + xorl zeroe, zeroe + movq 24(x), %rdx + mulpadd 4, 3 + movq %r15, 56(z) + mulpadd 5, 3 + mulpadd 6, 3 + mulpadd 7, 3 + movq 56(x), %rdx + mulpadd 4, 7 + mulpade 5, 7 + mulpade 6, 7 + adcxq zero, %r14 +.endm + +// ------------------------------------------------------------------------ +// Multiply-add: z := x[0...7] * x off-diagonal elements +// ------------------------------------------------------------------------ + +.macro mulrows + sqrz + + .set I, 8 +.rep 8 + mulrow 0, I + .set I, (I+1) +.endr + + movq %r8, 128(z) + movq %r9, 136(z) + movq %r10, 144(z) + movq %r11, 152(z) + movq %r12, 160(z) + movq %r13, 168(z) + movq %r14, 176(z) + movq %r15, 184(z) +.endm + +// ------------------------------------------------------------------------ +// The actual code +// ------------------------------------------------------------------------ + + + +S2N_BN_SYMBOL(bignum_ksqr_16_32): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Save more registers to play with + + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Now just systematically add in the rows to get all off-diagonal elements + + mulrows + addrows 8 + +// Double and add the diagonal elements. Note that z[0] was never written above + + xorl zeroe, zeroe + movq (x), %rdx + mulxq %rdx, %rax, %rcx + movq %rax, (z) + + movq 8(z), %rdx + adcxq %rdx, %rdx + adoxq %rcx, %rdx + movq %rdx, 8(z) + + .set I, 1 +.rep 14 + movq 8*I(x), %rdx + mulxq %rdx, %rax, %rcx + + movq 8*(2*I)(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 8*(2*I)(z) + + movq 8*(2*I+1)(z), %rdx + adcxq %rdx, %rdx + adoxq %rcx, %rdx + movq %rdx, 8*(2*I+1)(z) + .set I, (I+1) +.endr + + movq 8*I(x), %rdx + mulxq %rdx, %rax, %rcx + + movq 8*(2*I)(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 8*(2*I)(z) + + adcxq zero, %rcx + adoxq zero, %rcx + movq %rcx, 8*(2*I+1)(z) + .set I, (I+1) + + +// Restore registers and return + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_ksqr_16_32) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_ksqr_32_64.S b/cbits/s2n/x86_att/bignum_ksqr_32_64.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_ksqr_32_64.S @@ -0,0 +1,809 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Square, z := x^2 +// Input x[32]; output z[64]; temporary buffer t[>=72] +// +// extern void bignum_ksqr_32_64(uint64_t z[static 64], +// const uint64_t x[static 32], +// uint64_t t[static 72]); +// +// This is a Karatsuba-style function squaring half-sized results +// and using temporary buffer t for intermediate results. The size of 72 +// is an overstatement for compatibility with the ARM version; it actually +// only uses 65 elements of t (64 + 1 for a suspended carry). +// +// Standard x86-64 ABI: RDI = z, RSI = x, RDX = t +// Microsoft x64 ABI: RCX = z, RDX = x, R8 = t +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_ksqr_32_64) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_ksqr_32_64) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_ksqr_32_64) + + .text + +#define K 16 + +#define z %rdi +#define x %rsi +#define t %rcx + +S2N_BN_SYMBOL(bignum_ksqr_32_64): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx +#endif + +// Save callee-preserved registers once and for all at the outset +// Later we further reshuffle the input arguments to avoid extra saves + + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Move the temp space pointer since we need %rdx for multiplications + + movq %rdx, t + +// Square the low half + + CFI_CALL(Lbignum_ksqr_32_64_local_bignum_sqr_16_32) + +// Square the high half; from here on x and z are modified + + leaq 8*K(x), x // input at x+8*K + leaq 16*K(z), z // result at z+16*K + CFI_CALL(Lbignum_ksqr_32_64_local_bignum_sqr_16_32) + +// Form |x_lo - x_hi|, stored at t + + movq -8*K(x), %rax + subq (x), %rax + movq %rax, (t) + .set I, 1 + .rep K-1 + movq -8*K+8*I(x), %rax + sbbq 8*I(x), %rax + movq %rax, 8*I(t) + .set I, (I+1) + .endr + + movl $0, %ebx + sbbq %rax, %rax // Maintain CF, set ZF for cmovs + + .set I, 0 + .rep K + movq 8*I(t), %rdx + movq %rdx, %rax + notq %rdx + adcxq %rbx, %rdx + cmovzq %rax, %rdx + movq %rdx, 8*I(t) + .set I, (I+1) + .endr + +// Compose the middle parts [2,1] + [1,0] + [3,2] +// Put the low half of this at t[K] and the top half in place at z[2*K]; a +// fully in-place version is awkward with the otherwise beneficial double +// carry chain. Stash the carry suspended from the 3k position at the end of +// the temp buffer t[4*K]. + + xorl %edx, %edx + .set I, 0 + .rep K + movq -16*K+8*K+8*I(z), %rax + adcxq -16*K+8*I(z), %rax + adoxq -16*K+16*K+8*I(z), %rax + movq %rax, 8*K+8*I(t) + .set I, (I+1) + .endr + + .rep K + movq -16*K+8*K+8*I(z), %rax + adcxq -16*K+8*I(z), %rax + adoxq -16*K+16*K+8*I(z), %rax + movq %rax, -16*K+8*K+8*I(z) + .set I, (I+1) + .endr + + adoxq %rdx, %rdx + adcq $0, %rdx + movq %rdx, 32*K(t) + +// Square the absolute difference, putting the result M at t[2*K]. +// This involves another shuffle so now t' = z_orig and x' = t_orig +// while z' points within the temp buffer to the product M itself + + movq t, x + leaq -16*K(z), t + leaq 16*K(x), z + CFI_CALL(Lbignum_ksqr_32_64_local_bignum_sqr_16_32) + +// Subtract M, pausing at the 3k position to bump down accumulated carry. +// The carry cannot go negative since it's the top word of a value +// of the form ... + h^2 + l^2 - (h - l)^2 >= 0 + + movq 8*K(x), %rax + subq (z), %rax + movq %rax, 8*K(t) + + .set I, 1 + + .rep (K-1) + movq 8*K+8*I(x), %rax + sbbq 8*I(z), %rax + movq %rax, 8*K+8*I(t) + .set I, (I+1) + .endr + + .rep K + movq 8*K+8*I(t), %rax + sbbq 8*I(z), %rax + movq %rax, 8*K+8*I(t) + .set I, (I+1) + .endr + + movq 32*K(x), %rdx + sbbq $0, %rdx + +// Finally propagate the carry to the top quarter + + xorl %eax, %eax + addq %rdx, 24*K(t) + .set I, 1 + .rep K-1 + adcq %rax, 24*K+8*I(t) + .set I, (I+1) + .endr + +// Restore registers and return + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + CFI_POP(%rbp) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +// Local copy of the half-length subroutine + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lbignum_ksqr_32_64_local_bignum_sqr_16_32) + +Lbignum_ksqr_32_64_local_bignum_sqr_16_32: + CFI_START + xorl %ebp, %ebp + movq (x), %rdx + mulxq 0x8(x), %r9, %rax + movq %r9, 0x8(z) + mulxq 0x10(x), %r10, %rbx + adcxq %rax, %r10 + movq %r10, 0x10(z) + mulxq 0x18(x), %r11, %rax + adcxq %rbx, %r11 + mulxq 0x20(x), %r12, %rbx + adcxq %rax, %r12 + mulxq 0x28(x), %r13, %rax + adcxq %rbx, %r13 + mulxq 0x30(x), %r14, %rbx + adcxq %rax, %r14 + mulxq 0x38(x), %r15, %r8 + adcxq %rbx, %r15 + adcxq %rbp, %r8 + xorl %ebp, %ebp + movq 0x8(x), %rdx + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq %r11, 0x18(z) + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq %r12, 0x20(z) + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x38(x), %rax, %r9 + adcxq %rax, %r8 + adoxq %rbp, %r9 + movq 0x20(x), %rdx + mulxq 0x28(x), %rax, %r10 + adcxq %rax, %r9 + adoxq %rbp, %r10 + adcxq %rbp, %r10 + xorl %ebp, %ebp + movq 0x10(x), %rdx + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + movq %r13, 0x28(z) + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + movq %r14, 0x30(z) + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x38(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + movq 0x30(x), %rdx + mulxq 0x20(x), %rax, %r11 + adcxq %rax, %r10 + adoxq %rbp, %r11 + mulxq 0x28(x), %rax, %r12 + adcxq %rax, %r11 + adoxq %rbp, %r12 + adcxq %rbp, %r12 + xorl %ebp, %ebp + movq 0x18(x), %rdx + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + movq %r15, 0x38(z) + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x38(x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq 0x38(x), %rdx + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x28(x), %rax, %r13 + adcxq %rax, %r12 + adoxq %rbp, %r13 + mulxq 0x30(x), %rax, %r14 + adcxq %rax, %r13 + adoxq %rbp, %r14 + adcxq %rbp, %r14 + movq 0x40(x), %rdx + xorl %ebp, %ebp + mulxq (x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + movq %r8, 0x40(z) + mulxq 0x8(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x30(x), %rax, %r15 + adcxq %rax, %r14 + adoxq %rbp, %r15 + mulxq 0x38(x), %rax, %r8 + adcxq %rax, %r15 + adoxq %rbp, %r8 + adcxq %rbp, %r8 + movq 0x48(x), %rdx + xorl %ebp, %ebp + mulxq (x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + movq %r9, 0x48(z) + mulxq 0x8(x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x38(x), %rax, %r9 + adcxq %rax, %r8 + adoxq %rbp, %r9 + adcxq %rbp, %r9 + movq 0x50(x), %rdx + xorl %ebp, %ebp + mulxq (x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq %r10, 0x50(z) + mulxq 0x8(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x38(x), %rax, %r10 + adcxq %rax, %r9 + adoxq %rbp, %r10 + adcxq %rbp, %r10 + movq 0x58(x), %rdx + xorl %ebp, %ebp + mulxq (x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq %r11, 0x58(z) + mulxq 0x8(x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x38(x), %rax, %r11 + adcxq %rax, %r10 + adoxq %rbp, %r11 + adcxq %rbp, %r11 + movq 0x60(x), %rdx + xorl %ebp, %ebp + mulxq (x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq %r12, 0x60(z) + mulxq 0x8(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x38(x), %rax, %r12 + adcxq %rax, %r11 + adoxq %rbp, %r12 + adcxq %rbp, %r12 + movq 0x68(x), %rdx + xorl %ebp, %ebp + mulxq (x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + movq %r13, 0x68(z) + mulxq 0x8(x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x38(x), %rax, %r13 + adcxq %rax, %r12 + adoxq %rbp, %r13 + adcxq %rbp, %r13 + movq 0x70(x), %rdx + xorl %ebp, %ebp + mulxq (x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + movq %r14, 0x70(z) + mulxq 0x8(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x38(x), %rax, %r14 + adcxq %rax, %r13 + adoxq %rbp, %r14 + adcxq %rbp, %r14 + movq 0x78(x), %rdx + xorl %ebp, %ebp + mulxq (x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + movq %r15, 0x78(z) + mulxq 0x8(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x38(x), %rax, %r15 + adcxq %rax, %r14 + adoxq %rbp, %r15 + adcxq %rbp, %r15 + movq %r8, 0x80(z) + movq %r9, 0x88(z) + movq %r10, 0x90(z) + movq %r11, 0x98(z) + movq %r12, 0xa0(z) + movq %r13, 0xa8(z) + movq %r14, 0xb0(z) + movq %r15, 0xb8(z) + xorl %ebp, %ebp + movq 0x88(z), %r9 + movq 0x90(z), %r10 + movq 0x98(z), %r11 + movq 0xa0(z), %r12 + movq 0xa8(z), %r13 + movq 0xb0(z), %r14 + movq 0xb8(z), %r15 + movq 0x40(x), %rdx + mulxq 0x48(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + movq %r9, 0x88(z) + mulxq 0x50(x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq %r10, 0x90(z) + mulxq 0x58(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x60(x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x68(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x70(x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x78(x), %rax, %r8 + adcxq %rax, %r15 + adoxq %rbp, %r8 + adcxq %rbp, %r8 + xorl %ebp, %ebp + movq 0x48(x), %rdx + mulxq 0x50(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq %r11, 0x98(z) + mulxq 0x58(x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq %r12, 0xa0(z) + mulxq 0x60(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x68(x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x70(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x78(x), %rax, %r9 + adcxq %rax, %r8 + adoxq %rbp, %r9 + movq 0x60(x), %rdx + mulxq 0x68(x), %rax, %r10 + adcxq %rax, %r9 + adoxq %rbp, %r10 + adcxq %rbp, %r10 + xorl %ebp, %ebp + movq 0x50(x), %rdx + mulxq 0x58(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + movq %r13, 0xa8(z) + mulxq 0x60(x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + movq %r14, 0xb0(z) + mulxq 0x68(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x70(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x78(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + movq 0x70(x), %rdx + mulxq 0x60(x), %rax, %r11 + adcxq %rax, %r10 + adoxq %rbp, %r11 + mulxq 0x68(x), %rax, %r12 + adcxq %rax, %r11 + adoxq %rbp, %r12 + adcxq %rbp, %r12 + xorl %ebp, %ebp + movq 0x58(x), %rdx + mulxq 0x60(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + movq %r15, 0xb8(z) + mulxq 0x68(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x70(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x78(x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq 0x78(x), %rdx + mulxq 0x60(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x68(x), %rax, %r13 + adcxq %rax, %r12 + adoxq %rbp, %r13 + mulxq 0x70(x), %rax, %r14 + adcxq %rax, %r13 + adoxq %rbp, %r14 + adcxq %rbp, %r14 + movq %r8, 0xc0(z) + movq %r9, 0xc8(z) + movq %r10, 0xd0(z) + movq %r11, 0xd8(z) + movq %r12, 0xe0(z) + movq %r13, 0xe8(z) + movq %r14, 0xf0(z) + xorl %ebp, %ebp + movq (x), %rdx + mulxq %rdx, %rax, %rbx + movq %rax, (z) + movq 0x8(z), %rdx + adcxq %rdx, %rdx + adoxq %rbx, %rdx + movq %rdx, 0x8(z) + movq 0x8(x), %rdx + mulxq %rdx, %rax, %rbx + movq 0x10(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 0x10(z) + movq 0x18(z), %rdx + adcxq %rdx, %rdx + adoxq %rbx, %rdx + movq %rdx, 0x18(z) + movq 0x10(x), %rdx + mulxq %rdx, %rax, %rbx + movq 0x20(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 0x20(z) + movq 0x28(z), %rdx + adcxq %rdx, %rdx + adoxq %rbx, %rdx + movq %rdx, 0x28(z) + movq 0x18(x), %rdx + mulxq %rdx, %rax, %rbx + movq 0x30(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 0x30(z) + movq 0x38(z), %rdx + adcxq %rdx, %rdx + adoxq %rbx, %rdx + movq %rdx, 0x38(z) + movq 0x20(x), %rdx + mulxq %rdx, %rax, %rbx + movq 0x40(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 0x40(z) + movq 0x48(z), %rdx + adcxq %rdx, %rdx + adoxq %rbx, %rdx + movq %rdx, 0x48(z) + movq 0x28(x), %rdx + mulxq %rdx, %rax, %rbx + movq 0x50(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 0x50(z) + movq 0x58(z), %rdx + adcxq %rdx, %rdx + adoxq %rbx, %rdx + movq %rdx, 0x58(z) + movq 0x30(x), %rdx + mulxq %rdx, %rax, %rbx + movq 0x60(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 0x60(z) + movq 0x68(z), %rdx + adcxq %rdx, %rdx + adoxq %rbx, %rdx + movq %rdx, 0x68(z) + movq 0x38(x), %rdx + mulxq %rdx, %rax, %rbx + movq 0x70(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 0x70(z) + movq 0x78(z), %rdx + adcxq %rdx, %rdx + adoxq %rbx, %rdx + movq %rdx, 0x78(z) + movq 0x40(x), %rdx + mulxq %rdx, %rax, %rbx + movq 0x80(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 0x80(z) + movq 0x88(z), %rdx + adcxq %rdx, %rdx + adoxq %rbx, %rdx + movq %rdx, 0x88(z) + movq 0x48(x), %rdx + mulxq %rdx, %rax, %rbx + movq 0x90(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 0x90(z) + movq 0x98(z), %rdx + adcxq %rdx, %rdx + adoxq %rbx, %rdx + movq %rdx, 0x98(z) + movq 0x50(x), %rdx + mulxq %rdx, %rax, %rbx + movq 0xa0(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 0xa0(z) + movq 0xa8(z), %rdx + adcxq %rdx, %rdx + adoxq %rbx, %rdx + movq %rdx, 0xa8(z) + movq 0x58(x), %rdx + mulxq %rdx, %rax, %rbx + movq 0xb0(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 0xb0(z) + movq 0xb8(z), %rdx + adcxq %rdx, %rdx + adoxq %rbx, %rdx + movq %rdx, 0xb8(z) + movq 0x60(x), %rdx + mulxq %rdx, %rax, %rbx + movq 0xc0(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 0xc0(z) + movq 0xc8(z), %rdx + adcxq %rdx, %rdx + adoxq %rbx, %rdx + movq %rdx, 0xc8(z) + movq 0x68(x), %rdx + mulxq %rdx, %rax, %rbx + movq 0xd0(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 0xd0(z) + movq 0xd8(z), %rdx + adcxq %rdx, %rdx + adoxq %rbx, %rdx + movq %rdx, 0xd8(z) + movq 0x70(x), %rdx + mulxq %rdx, %rax, %rbx + movq 0xe0(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 0xe0(z) + movq 0xe8(z), %rdx + adcxq %rdx, %rdx + adoxq %rbx, %rdx + movq %rdx, 0xe8(z) + movq 0x78(x), %rdx + mulxq %rdx, %rax, %rbx + movq 0xf0(z), %rdx + adcxq %rdx, %rdx + adoxq %rax, %rdx + movq %rdx, 0xf0(z) + adcxq %rbp, %rbx + adoxq %rbp, %rbx + movq %rbx, 0xf8(z) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lbignum_ksqr_32_64_local_bignum_sqr_16_32) + +S2N_BN_SIZE_DIRECTIVE(bignum_ksqr_32_64) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_modinv.S b/cbits/s2n/x86_att/bignum_modinv.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_modinv.S @@ -0,0 +1,714 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Invert modulo m, z = (1/a) mod b, assuming b is an odd number > 1, coprime a +// Inputs a[k], b[k]; output z[k]; temporary buffer t[>=3*k] +// +// extern void bignum_modinv(uint64_t k, uint64_t *z, const uint64_t *a, +// const uint64_t *b, uint64_t *t); +// +// k-digit (digit=64 bits) "z := a^-1 mod b" (modular inverse of a modulo b) +// using t as a temporary buffer (t at least 3*k words = 24*k bytes), and +// assuming that a and b are coprime *and* that b is an odd number > 1. +// +// Standard x86-64 ABI: RDI = k, RSI = z, RDX = a, RCX = b, R8 = t +// Microsoft x64 ABI: RCX = k, RDX = z, R8 = a, R9 = b, [RSP+40] = t +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_modinv) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_modinv) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_modinv) + .text + .balign 32 + +// We get CHUNKSIZE bits per outer iteration, 64 minus a few for proxy errors + +#define CHUNKSIZE 58 + +// These variables are so fundamental we keep them consistently in registers. +// k actually stays where it was at the beginning, while l gets set up later + +#define k %rdi +#define l %r13 + +// These are kept on the stack since there aren't enough registers + +#define mat_mm (%rsp) +#define mat_mn 8(%rsp) +#define mat_nm 16(%rsp) +#define mat_nn 24(%rsp) +#define t 32(%rsp) +// Modular inverse +#define v 40(%rsp) +// We reconstruct n as m + 8*k as needed +#define m 48(%rsp) +#define w 56(%rsp) +#define z 64(%rsp) +// Original b pointer, not b the temp +#define bm 72(%rsp) + +#define STACKVARSIZE 80 + +// These get set to m/n or w/z during the cross-multiplications etc. +// Otherwise they can be used as additional temporaries + +#define p1 %r8 +#define p2 %r15 + +// These are shorthands for common temporary registers + +#define a %rax +#define b %rbx +#define c %rcx +#define d %rdx +#define i %r9 + +// Temporaries for the top proxy selection part + +#define c1 %r10 +#define c2 %r11 +#define h1 %r12 +#define h2 %rbp +#define l1 %r14 +#define l2 %rsi + +// Re-use for the actual proxies; m_hi = h1 and n_hi = h2 are assumed + +#define m_hi %r12 +#define n_hi %rbp +#define m_lo %r14 +#define n_lo %rsi + +// Re-use for the matrix entries in the inner loop, though they +// get spilled to the corresponding memory locations mat_... + +#define m_m %r10 +#define m_n %r11 +#define n_m %rcx +#define n_n %rdx + +#define ashort %eax +#define ishort %r9d +#define m_mshort %r10d +#define m_nshort %r11d +#define n_mshort %ecx +#define n_nshort %edx + +S2N_BN_SYMBOL(bignum_modinv): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + movq %r9, %rcx + movq 56(%rsp), %r8 +#endif + +// Save all required registers and make room on stack for all the above vars + + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(STACKVARSIZE) + +// If k = 0 then do nothing (this is out of scope anyway) + + testq k, k + jz Lbignum_modinv_end + +// Set up the additional two buffers m and n beyond w in temp space +// and record all pointers m, n, w and z in stack-based variables + + movq %rsi, z + movq %r8, w + movq %rcx, bm + leaq (%r8,k,8), %r10 + movq %r10, m + leaq (%r10,k,8), p2 + +// Initialize the main buffers with their starting values: +// m = a, n = b, w = b (to be tweaked to b - 1) and z = 0 + + xorq %r11, %r11 + xorq i, i +Lbignum_modinv_copyloop: + movq (%rdx,i,8), a + movq (%rcx,i,8), b + movq a, (%r10,i,8) + movq b, (p2,i,8) + movq b, (%r8,i,8) + movq %r11, (%rsi,i,8) + incq i + cmpq k, i + jc Lbignum_modinv_copyloop + +// Tweak down w to b - 1 (this crude approach is safe as b needs to be odd +// for it to be in scope). We have then established the congruence invariant: +// +// a * w == -m (mod b) +// a * z == n (mod b) +// +// This, with the bounds w <= b and z <= b, is maintained round the outer loop + + movq (%r8), a + movq a, b + decq b + movq b, (%r8) + +// Compute v = negated modular inverse of b mod 2^64, reusing a from above +// This is used for Montgomery reduction operations each time round the loop + + movq a, h2 + movq a, h1 + shlq $2, h2 + subq h2, h1 + xorq $2, h1 + + movq h1, h2 + imulq a, h2 + movl $2, ashort + addq h2, a + addq $1, h2 + + imulq a, h1 + + imulq h2, h2 + movl $1, ashort + addq h2, a + imulq a, h1 + + imulq h2, h2 + movl $1, ashort + addq h2, a + imulq a, h1 + + imulq h2, h2 + movl $1, ashort + addq h2, a + imulq a, h1 + + movq h1, v + +// Set up the outer loop count of 128 * k +// The invariant is that m * n < 2^t at all times. + + movq k, a + shlq $7, a + movq a, t + +// Start of the main outer loop iterated t / CHUNKSIZE times + +Lbignum_modinv_outerloop: + +// We need only bother with sharper l = min k (ceil(t/64)) digits +// for the computations on m and n (but we still need k for w and z). +// Either both m and n fit in l digits, or m has become zero and so +// nothing happens in the loop anyway and this makes no difference. + + movq t, l + addq $63, l + shrq $6, l + cmpq k, l + cmovncq k, l + +// Select upper and lower proxies for both m and n to drive the inner +// loop. The lower proxies are simply the lowest digits themselves, +// m_lo = m[0] and n_lo = n[0], while the upper proxies are bitfields +// of the two inputs selected so their top bit (63) aligns with the +// most significant bit of *either* of the two inputs. + + xorq h1, h1 // Previous high and low for m + xorq l1, l1 + xorq h2, h2 // Previous high and low for n + xorq l2, l2 + xorq c2, c2 // Mask flag: previous word of one was nonzero + // and in this case h1 and h2 are those words + + movq m, p1 + leaq (p1,k,8), p2 + xorq i, i +Lbignum_modinv_toploop: + movq (p1,i,8), b + movq (p2,i,8), c + movq c2, c1 + andq h1, c1 + andq h2, c2 + movq b, a + orq c, a + negq a + cmovcq c1, l1 + cmovcq c2, l2 + cmovcq b, h1 + cmovcq c, h2 + sbbq c2, c2 + incq i + cmpq l, i + jc Lbignum_modinv_toploop + + movq h1, a + orq h2, a + bsrq a, c + xorq $63, c + shldq %cl, l1, h1 + shldq %cl, l2, h2 + +// m_lo = m[0], n_lo = n[0]; + + movq (p1), %rax + movq %rax, m_lo + + movq (p2), %rax + movq %rax, n_lo + +// Now the inner loop, with i as loop counter from CHUNKSIZE down. +// This records a matrix of updates to apply to the initial +// values of m and n with, at stage j: +// +// sgn * m' = (m_m * m - m_n * n) / 2^j +// -sgn * n' = (n_m * m - n_n * n) / 2^j +// +// where "sgn" is either +1 or -1, and we lose track of which except +// that both instance above are the same. This throwing away the sign +// costs nothing (since we have to correct in general anyway because +// of the proxied comparison) and makes things a bit simpler. But it +// is simply the parity of the number of times the first condition, +// used as the swapping criterion, fires in this loop. + + movl $1, m_mshort + movl $0, m_nshort + movl $0, n_mshort + movl $1, n_nshort + movl $CHUNKSIZE, ishort + +// Stash more variables over the inner loop to free up regs + + movq k, mat_mn + movq l, mat_nm + movq p1, mat_mm + movq p2, mat_nn + +// Conceptually in the inner loop we follow these steps: +// +// * If m_lo is odd and m_hi < n_hi, then swap the four pairs +// (m_hi,n_hi); (m_lo,n_lo); (m_m,n_m); (m_n,n_n) +// +// * Now, if m_lo is odd (old or new, doesn't matter as initial n_lo is odd) +// m_hi := m_hi - n_hi, m_lo := m_lo - n_lo +// m_m := m_m + n_m, m_n := m_n + n_n +// +// * Halve and double them +// m_hi := m_hi / 2, m_lo := m_lo / 2 +// n_m := n_m * 2, n_n := n_n * 2 +// +// The actual computation computes updates before actually swapping and +// then corrects as needed. + +Lbignum_modinv_innerloop: + + xorl %eax, %eax + xorl %ebx, %ebx + xorq p1, p1 + xorq p2, p2 + btq $0, m_lo + + cmovcq n_hi, %rax + cmovcq n_lo, %rbx + cmovcq n_m, p1 + cmovcq n_n, p2 + + movq m_lo, l + subq %rbx, m_lo + subq l, %rbx + movq m_hi, k + subq %rax, k + cmovcq m_hi, n_hi + leaq -1(k), m_hi + cmovcq %rbx, m_lo + cmovcq l, n_lo + notq m_hi + cmovcq m_m, n_m + cmovcq m_n, n_n + cmovncq k, m_hi + + shrq $1, m_lo + addq p1, m_m + addq p2, m_n + shrq $1, m_hi + addq n_m, n_m + addq n_n, n_n + +// End of the inner for-loop + + decq i + jnz Lbignum_modinv_innerloop + +// Unstash the temporary variables + + movq mat_mn, k + movq mat_nm, l + movq mat_mm, p1 + movq mat_nn, p2 + +// Put the matrix entries in memory since we're out of registers +// We pull them out repeatedly in the next loop + + movq m_m, mat_mm + movq m_n, mat_mn + movq n_m, mat_nm + movq n_n, mat_nn + +// Apply the update to w and z, using addition in this case, and also take +// the chance to shift an additional 6 = 64-CHUNKSIZE bits to be ready for a +// Montgomery multiplication. Because we know that m_m + m_n <= 2^58 and +// w, z <= b < 2^{64k}, we know that both of these fit in k+1 words. +// We do this before the m-n update to allow us to play with c1 and c2 here. +// +// l1::w = 2^6 * (m_m * w + m_n * z) +// l2::z = 2^6 * (n_m * w + n_n * z) +// +// with c1 and c2 recording previous words for the shifting part + + movq w, p1 + movq z, p2 + xorq l1, l1 + xorq l2, l2 + xorq c1, c1 + xorq c2, c2 + xorq i, i +Lbignum_modinv_congloop: + + movq (p1,i,8), c + movq mat_mm, a + mulq c + addq a, l1 + adcq $0, d + movq d, h1 // Now h1::l1 := m_m * w + l1_in + + movq mat_nm, a + mulq c + addq a, l2 + adcq $0, d + movq d, h2 // Now h2::l2 := n_m * w + l2_in + + movq (p2,i,8), c + movq mat_mn, a + mulq c + addq a, l1 + adcq d, h1 // h1::l1 := m_m * w + m_n * z + l1_in + shrdq $CHUNKSIZE, l1, c1 + movq c1, (p1,i,8) + movq l1, c1 + movq h1, l1 + + movq mat_nn, a + mulq c + addq a, l2 + adcq d, h2 // h2::l2 := n_m * w + n_n * z + l2_in + shrdq $CHUNKSIZE, l2, c2 + movq c2, (p2,i,8) + movq l2, c2 + movq h2, l2 + + incq i + cmpq k, i + jc Lbignum_modinv_congloop + + shldq $64-CHUNKSIZE, c1, l1 + shldq $64-CHUNKSIZE, c2, l2 + +// Do a Montgomery reduction of l1::w + + movq bm, p2 + + movq (p1), b + movq v, h1 + imulq b, h1 + movq (p2), a + mulq h1 + addq b, a // Will be zero but want the carry + movq %rdx, c1 + movl $1, ishort + movq k, c + decq c + jz Lbignum_modinv_wmontend + +Lbignum_modinv_wmontloop: + adcq (p1,i,8), c1 + sbbq b, b + movq (p2,i,8), a + mulq h1 + subq b, %rdx + addq c1, a + movq a, -8(p1,i,8) + movq %rdx, c1 + incq i + decq c + jnz Lbignum_modinv_wmontloop + +Lbignum_modinv_wmontend: + adcq l1, c1 + movq c1, -8(p1,k,8) + sbbq c1, c1 + negq c1 + + movq k, c + xorq i, i +Lbignum_modinv_wcmploop: + movq (p1,i,8), a + sbbq (p2,i,8), a + incq i + decq c + jnz Lbignum_modinv_wcmploop + sbbq $0, c1 + sbbq c1, c1 + notq c1 + + xorq c, c + xorq i, i +Lbignum_modinv_wcorrloop: + movq (p1,i,8), a + movq (p2,i,8), b + andq c1, b + negq c + sbbq b, a + sbbq c, c + movq a, (p1,i,8) + incq i + cmpq k, i + jc Lbignum_modinv_wcorrloop + +// Do a Montgomery reduction of l2::z + + movq z, p1 + + movq (p1), b + movq v, h2 + imulq b, h2 + movq (p2), a + mulq h2 + addq b, a // Will be zero but want the carry + movq %rdx, c2 + movl $1, ishort + movq k, c + decq c + jz Lbignum_modinv_zmontend + +Lbignum_modinv_zmontloop: + adcq (p1,i,8), c2 + sbbq b, b + movq (p2,i,8), a + mulq h2 + subq b, %rdx + addq c2, a + movq a, -8(p1,i,8) + movq %rdx, c2 + incq i + decq c + jnz Lbignum_modinv_zmontloop + +Lbignum_modinv_zmontend: + adcq l2, c2 + movq c2, -8(p1,k,8) + sbbq c2, c2 + negq c2 + + movq k, c + xorq i, i +Lbignum_modinv_zcmploop: + movq (p1,i,8), a + sbbq (p2,i,8), a + incq i + decq c + jnz Lbignum_modinv_zcmploop + sbbq $0, c2 + sbbq c2, c2 + notq c2 + + xorq c, c + xorq i, i +Lbignum_modinv_zcorrloop: + movq (p1,i,8), a + movq (p2,i,8), b + andq c2, b + negq c + sbbq b, a + sbbq c, c + movq a, (p1,i,8) + incq i + cmpq k, i + jc Lbignum_modinv_zcorrloop + +// Now actually compute the updates to m and n corresponding to the matrix, +// and correct the signs if they have gone negative. First we compute the +// (k+1)-sized updates with the following invariant (here h1 and h2 are in +// fact carry bitmasks, either 0 or -1): +// +// h1::l1::m = m_m * m - m_n * n +// h2::l2::n = n_m * m - n_n * n + + movq m, p1 + leaq (p1,k,8), p2 + xorq i, i + xorq h1, h1 + xorq l1, l1 + xorq h2, h2 + xorq l2, l2 +Lbignum_modinv_crossloop: + + movq (p1,i,8), c + movq mat_mm, a + mulq c + addq a, l1 + adcq $0, d + movq d, c1 // Now c1::l1 is +ve part 1 + + movq mat_nm, a + mulq c + addq a, l2 + adcq $0, d + movq d, c2 // Now c2::l2 is +ve part 2 + + movq (p2,i,8), c + movq mat_mn, a + mulq c + subq h1, d // Now d::a is -ve part 1 + + subq a, l1 + sbbq d, c1 + sbbq h1, h1 + movq l1, (p1,i,8) + movq c1, l1 + + movq mat_nn, a + mulq c + subq h2, d // Now d::a is -ve part 2 + + subq a, l2 + sbbq d, c2 + sbbq h2, h2 + movq l2, (p2,i,8) + movq c2, l2 + + incq i + cmpq l, i + jc Lbignum_modinv_crossloop + +// Now fix the signs of m and n if they have gone negative + + xorq i, i + movq h1, c1 // carry-in coded up as well + movq h2, c2 // carry-in coded up as well + xorq h1, l1 // for the Lbignum_modinv_end digit + xorq h2, l2 // for the Lbignum_modinv_end digit +Lbignum_modinv_optnegloop: + movq (p1,i,8), a + xorq h1, a + negq c1 + adcq $0, a + sbbq c1, c1 + movq a, (p1,i,8) + movq (p2,i,8), a + xorq h2, a + negq c2 + adcq $0, a + sbbq c2, c2 + movq a, (p2,i,8) + incq i + cmpq l, i + jc Lbignum_modinv_optnegloop + subq c1, l1 + subq c2, l2 + +// Now shift them right CHUNKSIZE bits + + movq l, i +Lbignum_modinv_shiftloop: + movq -8(p1,i,8), a + movq a, c1 + shrdq $CHUNKSIZE, l1, a + movq a, -8(p1,i,8) + movq c1, l1 + movq -8(p2,i,8), a + movq a, c2 + shrdq $CHUNKSIZE, l2, a + movq a, -8(p2,i,8) + movq c2, l2 + decq i + jnz Lbignum_modinv_shiftloop + +// Finally, use the signs h1 and h2 to do optional modular negations of +// w and z respectively, flipping h2 to make signs work. We don't make +// any checks for zero values, but we certainly retain w <= b and z <= b. +// This is enough for the Montgomery step in the next iteration to give +// strict reduction w < b amd z < b, and anyway when we terminate we +// could not have z = b since it violates the coprimality assumption for +// in-scope cases. + + notq h2 + movq bm, c + movq w, p1 + movq z, p2 + movq h1, c1 + movq h2, c2 + xorq i, i +Lbignum_modinv_fliploop: + movq h2, d + movq (c,i,8), a + andq a, d + andq h1, a + movq (p1,i,8), b + xorq h1, b + negq c1 + adcq b, a + sbbq c1, c1 + movq a, (p1,i,8) + movq (p2,i,8), b + xorq h2, b + negq c2 + adcq b, d + sbbq c2, c2 + movq d, (p2,i,8) + incq i + cmpq k, i + jc Lbignum_modinv_fliploop + +// End of main loop. We can stop if t' <= 0 since then m * n < 2^0, which +// since n is odd and m and n are coprime (in the in-scope cases) means +// m = 0, n = 1 and hence from the congruence invariant a * z == 1 (mod b). +// Moreover we do in fact need to maintain strictly t > 0 in the main loop, +// or the computation of the optimized digit bound l could collapse to 0. + + subq $CHUNKSIZE, t + jnbe Lbignum_modinv_outerloop + +Lbignum_modinv_end: + CFI_INC_RSP(STACKVARSIZE) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + CFI_POP(%rbp) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_modinv) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_montifier.S b/cbits/s2n/x86_att/bignum_montifier.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_montifier.S @@ -0,0 +1,545 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Compute "montification" constant z := 2^{128k} mod m +// Input m[k]; output z[k]; temporary buffer t[>=k] +// +// extern void bignum_montifier(uint64_t k, uint64_t *z, const uint64_t *m, +// uint64_t *t); +// +// The last argument points to a temporary buffer t that should have size >= k. +// This is called "montifier" because given any other k-digit number x, +// whether or not it's reduced modulo m, it can be mapped to its Montgomery +// representation (2^{64k} * x) mod m just by Montgomery multiplication by z. +// +// Standard x86-64 ABI: RDI = k, RSI = z, RDX = m, RCX = t +// Microsoft x64 ABI: RCX = k, RDX = z, R8 = m, R9 = t +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montifier) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montifier) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montifier) + .text + .balign 32 + +#define k %rdi +#define z %rsi + +// These two inputs get moved to different places since RCX and RDX are special + +#define m %r12 +#define t %r13 + +// Other variables + +#define i %rbx +// Modular inverse; aliased to i, but we never use them together +#define w %rbx +#define j %rbp +// Matters that this is RAX for special use in multiplies +#define a %rax +// Matters that this is RDX for special use in multiplies +#define d %rdx +// Matters that this is RCX as CL=lo(c) is assumed in shifts +#define c %rcx +#define h %r11 +#define l %r10 +#define b %r9 +#define n %r8 + +// Some aliases for the values b and n + +#define q %r8 +#define r %r9 + +#define ashort %eax +#define ishort %ebx +#define jshort %ebp +#define qshort %r8d + + +S2N_BN_SYMBOL(bignum_montifier): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + movq %r9, %rcx +#endif + +// Save some additional registers for use, copy args out of RCX and RDX + + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + + movq %rdx, m + movq %rcx, t + +// If k = 0 the whole operation is trivial + + testq k, k + jz Lbignum_montifier_end + +// Copy the input m into the temporary buffer t. The temporary register +// c matters since we want it to hold the highest digit, ready for the +// normalization phase. + + xorq i, i +Lbignum_montifier_copyinloop: + movq (m,i,8), c + movq c, (t,i,8) + incq i + cmpq k, i + jc Lbignum_montifier_copyinloop + +// Do a rather stupid but constant-time digit normalization, conditionally +// shifting left (k-1) times based on whether the top word is zero. +// With careful binary striding this could be O(k*log(k)) instead of O(k^2) +// while still retaining the constant-time style. +// The "neg c" sets the zeroness predicate (~CF) for the entire inner loop + + movq k, i + decq i + jz Lbignum_montifier_normalized +Lbignum_montifier_normloop: + xorq j, j + movq k, h + negq c + movl $0, ashort +Lbignum_montifier_shufloop: + movq a, c + movq (t,j,8), a + cmovcq a, c + movq c, (t,j,8) + incq j + decq h + jnz Lbignum_montifier_shufloop + decq i + jnz Lbignum_montifier_normloop + +// We now have the top digit nonzero, assuming the input was nonzero, +// and as per the invariant of the loop above, c holds that digit. So +// now just count c's leading zeros and shift t bitwise that many bits. +// Note that we don't care about the result of bsr for zero inputs so +// the simple xor-ing with 63 is safe. + +Lbignum_montifier_normalized: + + bsrq c, c + xorq $63, c + + xorq b, b + xorq i, i +Lbignum_montifier_bitloop: + movq (t,i,8), a + movq a, j + shldq %cl, b, a + movq a, (t,i,8) + movq j, b + incq i + cmpq k, i + jc Lbignum_montifier_bitloop + +// Let h be the high word of n, which in all the in-scope cases is >= 2^63. +// Now successively form q = 2^i div h and r = 2^i mod h as i goes from +// 64 to 126. We avoid just using division out of constant-time concerns +// (at the least we would need to fix up h = 0 for out-of-scope inputs) and +// don't bother with Newton-Raphson, since this stupid simple loop doesn't +// contribute much of the overall runtime at typical sizes. + + movq -8(t,k,8), h + movl $1, qshort + movq h, r + negq r + movl $62, ishort +Lbignum_montifier_estloop: + + addq q, q + movq h, a + subq r, a + cmpq a, r // CF <=> r < h - r <=> 2 * r < h + sbbq a, a + notq a // a = bitmask(2 * r >= h) + subq a, q + addq r, r + andq h, a + subq a, r + decq i + jnz Lbignum_montifier_estloop + +// Strictly speaking the above loop doesn't quite give the true remainder +// and quotient in the special case r = h = 2^63, so fix it up. We get +// q = 2^63 - 1 and r = 2^63 and really want q = 2^63 and r = 0. This is +// supererogatory, because the main property of q used below still holds +// in this case unless the initial m = 1, and then anyway the overall +// specification (congruence modulo m) holds degenerately. But it seems +// nicer to get a "true" quotient and remainder. + + incq r + cmpq r, h + adcq $0, q + +// So now we have q and r with 2^126 = q * h + r (imagining r = 0 in the +// fixed-up case above: note that we never actually use the computed +// value of r below and so didn't adjust it). And we can assume the ranges +// q <= 2^63 and r < h < 2^64. +// +// The idea is to use q as a first quotient estimate for a remainder +// of 2^{p+62} mod n, where p = 64 * k. We have, splitting n into the +// high and low parts h and l: +// +// 2^{p+62} - q * n = 2^{p+62} - q * (2^{p-64} * h + l) +// = 2^{p+62} - (2^{p-64} * (q * h) + q * l) +// = 2^{p+62} - 2^{p-64} * (2^126 - r) - q * l +// = 2^{p-64} * r - q * l +// +// Note that 2^{p-64} * r < 2^{p-64} * h <= n +// and also q * l < 2^63 * 2^{p-64} = 2^{p-1} <= n +// so |diff| = |2^{p-64} * r - q * l| < n. +// +// If in fact diff >= 0 then it is already 2^{p+62} mod n. +// otherwise diff + n is the right answer. +// +// To (maybe?) make the computation slightly easier we actually flip +// the sign and compute d = q * n - 2^{p+62}. Then the answer is either +// -d (when negative) or n - d; in either case we effectively negate d. +// This negating tweak in fact spoils the result for cases where +// 2^{p+62} mod n = 0, when we get n instead. However the only case +// where this can happen is m = 1, when the whole spec holds trivially, +// and actually the remainder of the logic below works anyway since +// the latter part of the code only needs a congruence for the k-digit +// result, not strict modular reduction (the doublings will maintain +// the non-strict inequality). + + xorq c, c + xorq i, i +Lbignum_montifier_mulloop: + movq (t,i,8), a + mulq q + addq c, a + adcq $0, d + movq a, (z,i,8) + movq d, c + incq i + cmpq k, i + jc Lbignum_montifier_mulloop + +// Now c is the high word of the product, so subtract 2^62 +// and then turn it into a bitmask in q = h + + movq $0x4000000000000000, a + subq a, c + sbbq q, q + notq q + +// Now do [c] * n - d for our final answer + + xorq c, c + xorq i, i +Lbignum_montifier_remloop: + movq (t,i,8), a + andq q, a + negq c + sbbq (z,i,8), a + sbbq c, c + movq a, (z,i,8) + incq i + cmpq k, i + jc Lbignum_montifier_remloop + +// Now still need to do a couple of modular doublings to get us all the +// way up to 2^{p+64} == r from initial 2^{p+62} == r (mod n). + + xorq c, c + xorq j, j + xorq b, b +Lbignum_montifier_dubloop1: + movq (z,j,8), a + shrdq $63, a, c + negq b + sbbq (t,j,8), c + sbbq b, b + movq c, (z,j,8) + movq a, c + incq j + cmpq k, j + jc Lbignum_montifier_dubloop1 + shrq $63, c + addq b, c + xorq j, j + xorq b, b +Lbignum_montifier_corrloop1: + movq (t,j,8), a + andq c, a + negq b + adcq (z,j,8), a + sbbq b, b + movq a, (z,j,8) + incq j + cmpq k, j + jc Lbignum_montifier_corrloop1 + +// This is not exactly the same: we also copy output to t giving the +// initialization t_1 = r == 2^{p+64} mod n for the main loop next. + + xorq c, c + xorq j, j + xorq b, b +Lbignum_montifier_dubloop2: + movq (z,j,8), a + shrdq $63, a, c + negq b + sbbq (t,j,8), c + sbbq b, b + movq c, (z,j,8) + movq a, c + incq j + cmpq k, j + jc Lbignum_montifier_dubloop2 + shrq $63, c + addq b, c + xorq j, j + xorq b, b +Lbignum_montifier_corrloop2: + movq (t,j,8), a + andq c, a + negq b + adcq (z,j,8), a + sbbq b, b + movq a, (z,j,8) + movq a, (t,j,8) + incq j + cmpq k, j + jc Lbignum_montifier_corrloop2 + +// We then successively generate (k+1)-digit values satisfying +// t_i == 2^{p+64*i} mod n, each of which is stored in h::t. Finish +// initialization by zeroing h initially + + xorq h, h + +// Then if t_i = 2^{p} * h + l +// we have t_{i+1} == 2^64 * t_i +// = (2^{p+64} * h) + (2^64 * l) +// == r * h + l<<64 +// Do this 2*k more times so we end up == 2^{192*k+64}, one more than we want +// +// Writing B = 2^{64k}, the possible correction of adding r, which for +// a (k+1)-digit result is equivalent to subtracting q = 2^{64*(k+1)} - r +// would give the overall worst-case value minus q of +// [ B * (B^k - 1) + (B - 1) * r ] - [B^{k+1} - r] +// = B * (r - 1) < B^{k+1} so we keep inside k+1 digits as required. +// +// This implementation makes the shift implicit by starting b with the +// "previous" digit (initially 0) to offset things by 1. + + leaq (k,k), i +Lbignum_montifier_modloop: + xorq b, b + movq k, n + xorq j, j + xorq c, c +Lbignum_montifier_cmaloop: + adcq b, c + sbbq l, l + movq (z,j,8), a + mulq h + subq l, d + addq c, a + movq (t,j,8), b + movq a, (t,j,8) + movq d, c + incq j + decq n + jnz Lbignum_montifier_cmaloop + adcq c, b + movq b, h + + sbbq l, l + + xorq j, j + xorq c, c +Lbignum_montifier_oaloop: + movq (t,j,8), a + movq (z,j,8), b + andq l, b + negq c + adcq b, a + sbbq c, c + movq a, (t,j,8) + incq j + cmpq k, j + jc Lbignum_montifier_oaloop + subq c, h + + decq i + jnz Lbignum_montifier_modloop + +// Compute the negated modular inverse w (same register as i, not used again). + + movq (m), a + movq a, c + movq a, w + shlq $2, c + subq c, w + xorq $2, w + movq w, c + imulq a, c + movl $2, ashort + addq c, a + addq $1, c + imulq a, w + imulq c, c + movl $1, ashort + addq c, a + imulq a, w + imulq c, c + movl $1, ashort + addq c, a + imulq a, w + imulq c, c + movl $1, ashort + addq c, a + imulq a, w + +// Now do one almost-Montgomery reduction w.r.t. the original m +// which lops off one 2^64 from the congruence and, with the usual +// almost-Montgomery correction, gets us back inside k digits + + movq (t), c + movq w, b + imulq c, b + + movq (m), a + mulq b + addq c, a + movq d, c + movl $1, jshort + movq k, n + decq n + jz Lbignum_montifier_amontend +Lbignum_montifier_amontloop: + adcq (t,j,8), c + sbbq l, l + movq (m,j,8), a + mulq b + subq l, d + addq c, a + movq a, -8(t,j,8) + movq d, c + incq j + decq n + jnz Lbignum_montifier_amontloop +Lbignum_montifier_amontend: + adcq c, h + sbbq l, l + movq h, -8(t,k,8) + + xorq j, j + xorq c, c +Lbignum_montifier_aosloop: + movq (t,j,8), a + movq (m,j,8), b + andq l, b + negq c + sbbq b, a + sbbq c, c + movq a, (z,j,8) + incq j + cmpq k, j + jc Lbignum_montifier_aosloop + +// So far, the code (basically a variant of bignum_amontifier) has produced +// a k-digit value z == 2^{192k} (mod m), not necessarily fully reduced mod m. +// We now do a short Montgomery reduction (similar to bignum_demont) so that +// we achieve full reduction mod m while lopping 2^{64k} off the congruence. +// We recycle h as the somewhat strangely-named outer loop counter. + + movq k, h + +Lbignum_montifier_montouterloop: + movq (z), c + movq w, b + imulq c, b + movq (m), a + mulq b + addq c, a + movq d, c + movl $1, jshort + movq k, n + decq n + jz Lbignum_montifier_montend +Lbignum_montifier_montloop: + adcq (z,j,8), c + sbbq l, l + movq (m,j,8), a + mulq b + subq l, d + addq c, a + movq a, -8(z,j,8) + movq d, c + incq j + decq n + jnz Lbignum_montifier_montloop +Lbignum_montifier_montend: + adcq $0, c + movq c, -8(z,k,8) + + decq h + jnz Lbignum_montifier_montouterloop + +// Now do a comparison of z with m to set a final correction mask +// indicating that z >= m and so we need to subtract m. + + xorq j, j + movq k, n +Lbignum_montifier_cmploop: + movq (z,j,8), a + sbbq (m,j,8), a + incq j + decq n + jnz Lbignum_montifier_cmploop + sbbq d, d + notq d + +// Now do a masked subtraction of m for the final reduced result. + + xorq l, l + xorq j, j +Lbignum_montifier_corrloop: + movq (m,j,8), a + andq d, a + negq l + sbbq a, (z,j,8) + sbbq l, l + incq j + cmpq k, j + jc Lbignum_montifier_corrloop + +Lbignum_montifier_end: + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + CFI_POP(%rbp) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_montifier) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_montinv_p384.S b/cbits/s2n/x86_att/bignum_montinv_p384.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_montinv_p384.S @@ -0,0 +1,1834 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Montgomery inverse modulo p_384 = 2^384 - 2^128 - 2^96 + 2^32 - 1 +// Input x[6]; output z[6] +// +// extern void bignum_montinv_p384(uint64_t z[static 6], +// const uint64_t x[static 6]); +// +// If the 6-digit input x is coprime to p_384, i.e. is not divisible +// by it, returns z < p_384 such that x * z == 2^768 (mod p_384). This +// is effectively "Montgomery inverse" because if we consider x and z as +// Montgomery forms of X and Z, i.e. x == 2^384 * X and z == 2^384 * Z +// (both mod p_384) then X * Z == 1 (mod p_384). That is, this function +// gives the analog of the modular inverse bignum_inv_p384 but with both +// input and output in the Montgomery domain. Note that x does not need +// to be reduced modulo p_384, but the output always is. If the input +// is divisible (i.e. is 0 or p_384), then there can be no solution to +// the congruence x * z == 2^768 (mod p_384), and z = 0 is returned. +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montinv_p384) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montinv_p384) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montinv_p384) + .text + .balign 32 + +// Size in bytes of a 64-bit word + +#define N 8 + +// Pointer-offset pairs for temporaries on stack +// The u and v variables are 6 words each as expected, but the f and g +// variables are 8 words each -- they need to have at least one extra +// word for a sign word, and to preserve alignment we "round up" to 8. +// In fact, we currently keep an extra word in u and v as well. + +#define f 0(%rsp) +#define g (8*N)(%rsp) +#define u (16*N)(%rsp) +#define v (24*N)(%rsp) +#define tmp (32*N)(%rsp) +#define tmp2 (33*N)(%rsp) +#define i (34*N)(%rsp) +#define d (35*N)(%rsp) + +#define mat (36*N)(%rsp) + +// Backup for the input pointer + +#define res (40*N)(%rsp) + +// Total size to reserve on the stack + +#define NSPACE 42*N + +// Syntactic variants to make x86_att version simpler to generate + +#define F 0 +#define G (8*N) +#define U (16*N) +#define V (24*N) +#define MAT (36*N) + +#define ff (%rsp) +#define gg (8*N)(%rsp) + +// --------------------------------------------------------------------------- +// Core signed almost-Montgomery reduction macro from P[6..0] to P[5..0]. +// --------------------------------------------------------------------------- + +#define amontred(P) \ +/* We only know the input is -2^444 < x < 2^444. To do traditional */ \ +/* unsigned Montgomery reduction, start by adding 2^61 * p_384. */ \ + movq $0xe000000000000000, %r8 ; \ + xorl %eax, %eax ; \ + addq P, %r8 ; \ + movq $0x000000001fffffff, %r9 ; \ + leaq -1(%rax), %rax ; \ + adcq N+P, %r9 ; \ + movq $0xdfffffffe0000000, %r10 ; \ + adcq 2*N+P, %r10 ; \ + movq 3*N+P, %r11 ; \ + adcq %rax, %r11 ; \ + movq 4*N+P, %r12 ; \ + adcq %rax, %r12 ; \ + movq 5*N+P, %r13 ; \ + adcq %rax, %r13 ; \ + movq $0x1fffffffffffffff, %r14 ; \ + adcq 6*N+P, %r14 ; \ +/* Correction multiplier is %rbx = w = [d0 + (d0<<32)] mod 2^64 */ \ + movq %r8, %rbx ; \ + shlq $32, %rbx ; \ + addq %r8, %rbx ; \ +/* Construct [%rbp;%rdx;%rax;-] = (2^384 - p_384) * w */ \ +/* We know lowest word will cancel so can re-use %r8 as a temp */ \ + xorl %ebp, %ebp ; \ + movq $0xffffffff00000001, %rax ; \ + mulq %rbx; \ + movq %rdx, %r8 ; \ + movq $0x00000000ffffffff, %rax ; \ + mulq %rbx; \ + addq %r8, %rax ; \ + adcq %rbx, %rdx ; \ + adcl %ebp, %ebp ; \ +/* Now subtract that and add 2^384 * w, catching carry in %rax */ \ + subq %rax, %r9 ; \ + sbbq %rdx, %r10 ; \ + sbbq %rbp, %r11 ; \ + sbbq $0, %r12 ; \ + sbbq $0, %r13 ; \ + sbbq $0, %r14 ; \ + sbbq %rax, %rax ; \ + addq %rbx, %r14 ; \ + adcq $0, %rax ; \ +/* Now if top is nonzero we subtract p_384 (almost-Montgomery) */ \ + negq %rax; \ + movq $0x00000000ffffffff, %rbx ; \ + andq %rax, %rbx ; \ + movq $0xffffffff00000000, %rcx ; \ + andq %rax, %rcx ; \ + movq $0xfffffffffffffffe, %rdx ; \ + andq %rax, %rdx ; \ + subq %rbx, %r9 ; \ + movq %r9, P ; \ + sbbq %rcx, %r10 ; \ + movq %r10, N+P ; \ + sbbq %rdx, %r11 ; \ + movq %r11, 2*N+P ; \ + sbbq %rax, %r12 ; \ + movq %r12, 3*N+P ; \ + sbbq %rax, %r13 ; \ + movq %r13, 4*N+P ; \ + sbbq %rax, %r14 ; \ + movq %r14, 5*N+P + +// Very similar to a subroutine call to the s2n-bignum word_divstep59. +// But different in register usage and returning the final matrix as +// +// [ %r8 %r10] +// [ %r12 %r14] +// +// and also returning the matrix still negated (which doesn't matter) + +#define divstep59(din,fin,gin) \ + movq din, %rsi ; \ + movq fin, %rdx ; \ + movq gin, %rcx ; \ + movq %rdx, %rbx ; \ + andq $0xfffff, %rbx ; \ + movabsq $0xfffffe0000000000, %rax ; \ + orq %rax, %rbx ; \ + andq $0xfffff, %rcx ; \ + movabsq $0xc000000000000000, %rax ; \ + orq %rax, %rcx ; \ + movq $0xfffffffffffffffe, %rax ; \ + xorl %ebp, %ebp ; \ + movl $0x2, %edx ; \ + movq %rbx, %rdi ; \ + movq %rax, %r8 ; \ + testq %rsi, %rsi ; \ + cmovs %rbp, %r8 ; \ + testq $0x1, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + sarq $1, %rcx ; \ + movl $0x100000, %eax ; \ + leaq (%rbx,%rax), %rdx ; \ + leaq (%rcx,%rax), %rdi ; \ + shlq $0x16, %rdx ; \ + shlq $0x16, %rdi ; \ + sarq $0x2b, %rdx ; \ + sarq $0x2b, %rdi ; \ + movabsq $0x20000100000, %rax ; \ + leaq (%rbx,%rax), %rbx ; \ + leaq (%rcx,%rax), %rcx ; \ + sarq $0x2a, %rbx ; \ + sarq $0x2a, %rcx ; \ + movq %rdx, MAT(%rsp) ; \ + movq %rbx, MAT+0x8(%rsp) ; \ + movq %rdi, MAT+0x10(%rsp) ; \ + movq %rcx, MAT+0x18(%rsp) ; \ + movq fin, %r12 ; \ + imulq %r12, %rdi ; \ + imulq %rdx, %r12 ; \ + movq gin, %r13 ; \ + imulq %r13, %rbx ; \ + imulq %rcx, %r13 ; \ + addq %rbx, %r12 ; \ + addq %rdi, %r13 ; \ + sarq $0x14, %r12 ; \ + sarq $0x14, %r13 ; \ + movq %r12, %rbx ; \ + andq $0xfffff, %rbx ; \ + movabsq $0xfffffe0000000000, %rax ; \ + orq %rax, %rbx ; \ + movq %r13, %rcx ; \ + andq $0xfffff, %rcx ; \ + movabsq $0xc000000000000000, %rax ; \ + orq %rax, %rcx ; \ + movq $0xfffffffffffffffe, %rax ; \ + movl $0x2, %edx ; \ + movq %rbx, %rdi ; \ + movq %rax, %r8 ; \ + testq %rsi, %rsi ; \ + cmovs %rbp, %r8 ; \ + testq $0x1, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + sarq $1, %rcx ; \ + movl $0x100000, %eax ; \ + leaq (%rbx,%rax), %r8 ; \ + leaq (%rcx,%rax), %r10 ; \ + shlq $0x16, %r8 ; \ + shlq $0x16, %r10 ; \ + sarq $0x2b, %r8 ; \ + sarq $0x2b, %r10 ; \ + movabsq $0x20000100000, %rax ; \ + leaq (%rbx,%rax), %r15 ; \ + leaq (%rcx,%rax), %r11 ; \ + sarq $0x2a, %r15 ; \ + sarq $0x2a, %r11 ; \ + movq %r13, %rbx ; \ + movq %r12, %rcx ; \ + imulq %r8, %r12 ; \ + imulq %r15, %rbx ; \ + addq %rbx, %r12 ; \ + imulq %r11, %r13 ; \ + imulq %r10, %rcx ; \ + addq %rcx, %r13 ; \ + sarq $0x14, %r12 ; \ + sarq $0x14, %r13 ; \ + movq %r12, %rbx ; \ + andq $0xfffff, %rbx ; \ + movabsq $0xfffffe0000000000, %rax ; \ + orq %rax, %rbx ; \ + movq %r13, %rcx ; \ + andq $0xfffff, %rcx ; \ + movabsq $0xc000000000000000, %rax ; \ + orq %rax, %rcx ; \ + movq MAT(%rsp), %rax ; \ + imulq %r8, %rax ; \ + movq MAT+0x10(%rsp), %rdx ; \ + imulq %r15, %rdx ; \ + imulq MAT+0x8(%rsp), %r8 ; \ + imulq MAT+0x18(%rsp), %r15 ; \ + addq %r8, %r15 ; \ + leaq (%rax,%rdx), %r9 ; \ + movq MAT(%rsp), %rax ; \ + imulq %r10, %rax ; \ + movq MAT+0x10(%rsp), %rdx ; \ + imulq %r11, %rdx ; \ + imulq MAT+0x8(%rsp), %r10 ; \ + imulq MAT+0x18(%rsp), %r11 ; \ + addq %r10, %r11 ; \ + leaq (%rax,%rdx), %r13 ; \ + movq $0xfffffffffffffffe, %rax ; \ + movl $0x2, %edx ; \ + movq %rbx, %rdi ; \ + movq %rax, %r8 ; \ + testq %rsi, %rsi ; \ + cmovs %rbp, %r8 ; \ + testq $0x1, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + cmovs %rbp, %r8 ; \ + movq %rbx, %rdi ; \ + testq %rdx, %rcx ; \ + cmoveq %rbp, %r8 ; \ + cmoveq %rbp, %rdi ; \ + sarq $1, %rcx ; \ + xorq %r8, %rdi ; \ + xorq %r8, %rsi ; \ + btq $0x3f, %r8 ; \ + cmovbq %rcx, %rbx ; \ + movq %rax, %r8 ; \ + subq %rax, %rsi ; \ + leaq (%rcx,%rdi), %rcx ; \ + sarq $1, %rcx ; \ + movl $0x100000, %eax ; \ + leaq (%rbx,%rax), %r8 ; \ + leaq (%rcx,%rax), %r12 ; \ + shlq $0x15, %r8 ; \ + shlq $0x15, %r12 ; \ + sarq $0x2b, %r8 ; \ + sarq $0x2b, %r12 ; \ + movabsq $0x20000100000, %rax ; \ + leaq (%rbx,%rax), %r10 ; \ + leaq (%rcx,%rax), %r14 ; \ + sarq $0x2b, %r10 ; \ + sarq $0x2b, %r14 ; \ + movq %r9, %rax ; \ + imulq %r8, %rax ; \ + movq %r13, %rdx ; \ + imulq %r10, %rdx ; \ + imulq %r15, %r8 ; \ + imulq %r11, %r10 ; \ + addq %r8, %r10 ; \ + leaq (%rax,%rdx), %r8 ; \ + movq %r9, %rax ; \ + imulq %r12, %rax ; \ + movq %r13, %rdx ; \ + imulq %r14, %rdx ; \ + imulq %r15, %r12 ; \ + imulq %r11, %r14 ; \ + addq %r12, %r14 ; \ + leaq (%rax,%rdx), %r12 + +S2N_BN_SYMBOL(bignum_montinv_p384): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Save registers and make room for temporaries + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + + CFI_DEC_RSP(NSPACE) + +// Save the return pointer for the end so we can overwrite %rdi later + + movq %rdi, res + +// Copy the constant p_384 into f including the 7th zero digit + + movl $0xffffffff, %eax + movq %rax, F(%rsp) + movq %rax, %rbx + notq %rbx + movq %rbx, F+N(%rsp) + xorl %ebp, %ebp + leaq -2(%rbp), %rcx + movq %rcx, F+2*N(%rsp) + leaq -1(%rbp), %rdx + movq %rdx, F+3*N(%rsp) + movq %rdx, F+4*N(%rsp) + movq %rdx, F+5*N(%rsp) + movq %rbp, F+6*N(%rsp) + +// Copy input but to g, reduced mod p_384 so that g <= f as assumed +// in the divstep bound proof. + + movq (%rsi), %r8 + subq %rax, %r8 + movq N(%rsi), %r9 + sbbq %rbx, %r9 + movq 2*N(%rsi), %r10 + sbbq %rcx, %r10 + movq 3*N(%rsi), %r11 + sbbq %rdx, %r11 + movq 4*N(%rsi), %r12 + sbbq %rdx, %r12 + movq 5*N(%rsi), %r13 + sbbq %rdx, %r13 + + cmovcq (%rsi), %r8 + cmovcq N(%rsi), %r9 + cmovcq 2*N(%rsi), %r10 + cmovcq 3*N(%rsi), %r11 + cmovcq 4*N(%rsi), %r12 + cmovcq 5*N(%rsi), %r13 + + movq %r8, G(%rsp) + movq %r9, G+N(%rsp) + movq %r10, G+2*N(%rsp) + movq %r11, G+3*N(%rsp) + movq %r12, G+4*N(%rsp) + movq %r13, G+5*N(%rsp) + movq %rbp, G+6*N(%rsp) + +// Also maintain reduced < 2^384 vector [u,v] such that +// [f,g] == x * 2^{5*i-843} * [u,v] (mod p_384) +// starting with [p_384,x] == x * 2^{5*0-843} * [0,2^843] (mod p_384) +// The weird-looking 5*i modifications come in because we are doing +// 64-bit word-sized Montgomery reductions at each stage, which is +// 5 bits more than the 59-bit requirement to keep things stable. +// After the 15th and last iteration and sign adjustment, when +// f == 1 for in-scope cases, we have x * 2^{75-843} * u == 1, i.e. +// x * u == 2^768 as required. + + xorl %eax, %eax + movq %rax, U(%rsp) + movq %rax, U+N(%rsp) + movq %rax, U+2*N(%rsp) + movq %rax, U+3*N(%rsp) + movq %rax, U+4*N(%rsp) + movq %rax, U+5*N(%rsp) + +// The starting constant 2^843 mod p_384 is +// 0x0000000000000800:00001000000007ff:fffff00000000000 +// :00001000000007ff:fffff00000000800:0000000000000000 +// where colons separate 64-bit subwords, least significant at the right. +// These are constructed dynamically to reduce large constant loads. + + movq %rax, V(%rsp) + movq $0xfffff00000000800, %rcx + movq %rcx, V+N(%rsp) + movq $0x00001000000007ff, %rdx + movq %rdx, V+2*N(%rsp) + btr $11, %rcx + movq %rcx, V+3*N(%rsp) + movq %rdx, V+4*N(%rsp) + bts $11, %rax + movq %rax, V+5*N(%rsp) + +// Start of main loop. We jump into the middle so that the divstep +// portion is common to the special fifteenth iteration after a uniform +// first 14. + + movq $15, i + movq $1, d + jmp Lbignum_montinv_p384_midloop + +Lbignum_montinv_p384_loop: + +// Separate out the matrix into sign-magnitude pairs + + movq %r8, %r9 + sarq $63, %r9 + xorq %r9, %r8 + subq %r9, %r8 + + movq %r10, %r11 + sarq $63, %r11 + xorq %r11, %r10 + subq %r11, %r10 + + movq %r12, %r13 + sarq $63, %r13 + xorq %r13, %r12 + subq %r13, %r12 + + movq %r14, %r15 + sarq $63, %r15 + xorq %r15, %r14 + subq %r15, %r14 + +// Adjust the initial values to allow for complement instead of negation +// This initial offset is the same for [f,g] and [u,v] compositions. +// Save it in temporary storage for the [u,v] part and do [f,g] first. + + movq %r8, %rax + andq %r9, %rax + movq %r10, %rdi + andq %r11, %rdi + addq %rax, %rdi + movq %rdi, tmp + + movq %r12, %rax + andq %r13, %rax + movq %r14, %rsi + andq %r15, %rsi + addq %rax, %rsi + movq %rsi, tmp2 + +// Now the computation of the updated f and g values. This maintains a +// 2-word carry between stages so we can conveniently insert the shift +// right by 59 before storing back, and not overwrite digits we need +// again of the old f and g values. +// +// Digit 0 of [f,g] + + xorl %ebx, %ebx + movq F(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rdi + adcq %rdx, %rbx + movq G(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rdi + adcq %rdx, %rbx + + xorl %ebp, %ebp + movq F(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rbp + movq G(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + +// Digit 1 of [f,g] + + xorl %ecx, %ecx + movq F+N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq G+N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + shrdq $59, %rbx, %rdi + movq %rdi, F(%rsp) + + xorl %edi, %edi + movq F+N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rbp + adcq %rdx, %rdi + movq G+N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rdi + shrdq $59, %rbp, %rsi + movq %rsi, G(%rsp) + +// Digit 2 of [f,g] + + xorl %esi, %esi + movq F+2*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rsi + movq G+2*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rsi + shrdq $59, %rcx, %rbx + movq %rbx, F+N(%rsp) + + xorl %ebx, %ebx + movq F+2*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rdi + adcq %rdx, %rbx + movq G+2*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rdi + adcq %rdx, %rbx + shrdq $59, %rdi, %rbp + movq %rbp, G+N(%rsp) + +// Digit 3 of [f,g] + + xorl %ebp, %ebp + movq F+3*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rsi + adcq %rdx, %rbp + movq G+3*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rsi + adcq %rdx, %rbp + shrdq $59, %rsi, %rcx + movq %rcx, F+2*N(%rsp) + + xorl %ecx, %ecx + movq F+3*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rbx + adcq %rdx, %rcx + movq G+3*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbx + adcq %rdx, %rcx + shrdq $59, %rbx, %rdi + movq %rdi, G+2*N(%rsp) + +// Digit 4 of [f,g] + + xorl %edi, %edi + movq F+4*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbp + adcq %rdx, %rdi + movq G+4*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbp + adcq %rdx, %rdi + shrdq $59, %rbp, %rsi + movq %rsi, F+3*N(%rsp) + + xorl %esi, %esi + movq F+4*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rcx + adcq %rdx, %rsi + movq G+4*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rcx + adcq %rdx, %rsi + shrdq $59, %rcx, %rbx + movq %rbx, G+3*N(%rsp) + +// Digits 5 and 6 of [f,g] + + movq F+5*N(%rsp), %rax + xorq %r9, %rax + movq F+6*N(%rsp), %rbx + xorq %r9, %rbx + andq %r8, %rbx + negq %rbx + mulq %r8 + addq %rax, %rdi + adcq %rdx, %rbx + movq G+5*N(%rsp), %rax + xorq %r11, %rax + movq G+6*N(%rsp), %rdx + xorq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbx + mulq %r10 + addq %rax, %rdi + adcq %rdx, %rbx + shrdq $59, %rdi, %rbp + movq %rbp, F+4*N(%rsp) + shrdq $59, %rbx, %rdi + sarq $59, %rbx + + movq F+5*N(%rsp), %rax + movq %rdi, F+5*N(%rsp) + + movq F+6*N(%rsp), %rdi + movq %rbx, F+6*N(%rsp) + + xorq %r13, %rax + xorq %r13, %rdi + andq %r12, %rdi + negq %rdi + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rdi + movq G+5*N(%rsp), %rax + xorq %r15, %rax + movq G+6*N(%rsp), %rdx + xorq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rdi + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rdi + shrdq $59, %rsi, %rcx + movq %rcx, G+4*N(%rsp) + shrdq $59, %rdi, %rsi + movq %rsi, G+5*N(%rsp) + sarq $59, %rdi + movq %rdi, G+6*N(%rsp) + +// Get the initial carries back from storage and do the [u,v] accumulation + + movq tmp, %rbx + movq tmp2, %rbp + +// Digit 0 of [u,v] + + xorl %ecx, %ecx + movq U(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq V(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + + xorl %esi, %esi + movq U(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, U(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq V(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, V(%rsp) + +// Digit 1 of [u,v] + + xorl %ebx, %ebx + movq U+N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq V+N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + + xorl %ebp, %ebp + movq U+N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, U+N(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq V+N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, V+N(%rsp) + +// Digit 2 of [u,v] + + xorl %ecx, %ecx + movq U+2*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq V+2*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + + xorl %esi, %esi + movq U+2*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, U+2*N(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq V+2*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, V+2*N(%rsp) + +// Digit 3 of [u,v] + + xorl %ebx, %ebx + movq U+3*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq V+3*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + + xorl %ebp, %ebp + movq U+3*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, U+3*N(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq V+3*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, V+3*N(%rsp) + +// Digit 4 of [u,v] + + xorl %ecx, %ecx + movq U+4*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq V+4*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + + xorl %esi, %esi + movq U+4*N(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, U+4*N(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq V+4*N(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, V+4*N(%rsp) + +// Digits 5 and 6 of u (top is unsigned) + + movq U+5*N(%rsp), %rax + xorq %r9, %rax + movq %r9, %rbx + andq %r8, %rbx + negq %rbx + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq V+5*N(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbx + mulq %r10 + addq %rax, %rcx + adcq %rbx, %rdx + +// Preload for last use of old u digit 3 + + movq U+5*N(%rsp), %rax + movq %rcx, U+5*N(%rsp) + movq %rdx, U+6*N(%rsp) + +// Digits 5 and 6 of v (top is unsigned) + + xorq %r13, %rax + movq %r13, %rcx + andq %r12, %rcx + negq %rcx + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rcx + movq V+5*N(%rsp), %rax + xorq %r15, %rax + movq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rcx + mulq %r14 + addq %rax, %rsi + adcq %rcx, %rdx + movq %rsi, V+5*N(%rsp) + movq %rdx, V+6*N(%rsp) + +// Montgomery reduction of u + + amontred(u) + +// Montgomery reduction of v + + amontred(v) + +Lbignum_montinv_p384_midloop: + + divstep59(d,ff,gg) + movq %rsi, d + +// Next iteration + + decq i + jnz Lbignum_montinv_p384_loop + +// The 15th and last iteration does not need anything except the +// u value and the sign of f; the latter can be obtained from the +// lowest word of f. So it's done differently from the main loop. +// Find the sign of the new f. For this we just need one digit +// since we know (for in-scope cases) that f is either +1 or -1. +// We don't explicitly shift right by 59 either, but looking at +// bit 63 (or any bit >= 60) of the unshifted result is enough +// to distinguish -1 from +1; this is then made into a mask. + + movq F(%rsp), %rax + movq G(%rsp), %rcx + imulq %r8, %rax + imulq %r10, %rcx + addq %rcx, %rax + sarq $63, %rax + +// Now separate out the matrix into sign-magnitude pairs +// and adjust each one based on the sign of f. +// +// Note that at this point we expect |f|=1 and we got its +// sign above, so then since [f,0] == x * 2^{-768} [u,v] (mod p_384) +// we want to flip the sign of u according to that of f. + + movq %r8, %r9 + sarq $63, %r9 + xorq %r9, %r8 + subq %r9, %r8 + xorq %rax, %r9 + + movq %r10, %r11 + sarq $63, %r11 + xorq %r11, %r10 + subq %r11, %r10 + xorq %rax, %r11 + + movq %r12, %r13 + sarq $63, %r13 + xorq %r13, %r12 + subq %r13, %r12 + xorq %rax, %r13 + + movq %r14, %r15 + sarq $63, %r15 + xorq %r15, %r14 + subq %r15, %r14 + xorq %rax, %r15 + +// Adjust the initial value to allow for complement instead of negation + + movq %r8, %rax + andq %r9, %rax + movq %r10, %r12 + andq %r11, %r12 + addq %rax, %r12 + +// Digit 0 of [u] + + xorl %r13d, %r13d + movq U(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r12 + adcq %rdx, %r13 + movq V(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r12 + movq %r12, U(%rsp) + adcq %rdx, %r13 + +// Digit 1 of [u] + + xorl %r14d, %r14d + movq U+N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r13 + adcq %rdx, %r14 + movq V+N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r13 + movq %r13, U+N(%rsp) + adcq %rdx, %r14 + +// Digit 2 of [u] + + xorl %r15d, %r15d + movq U+2*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r14 + adcq %rdx, %r15 + movq V+2*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r14 + movq %r14, U+2*N(%rsp) + adcq %rdx, %r15 + +// Digit 3 of [u] + + xorl %r14d, %r14d + movq U+3*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r15 + adcq %rdx, %r14 + movq V+3*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r15 + movq %r15, U+3*N(%rsp) + adcq %rdx, %r14 + +// Digit 4 of [u] + + xorl %r15d, %r15d + movq U+4*N(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r14 + adcq %rdx, %r15 + movq V+4*N(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r14 + movq %r14, U+4*N(%rsp) + adcq %rdx, %r15 + +// Digits 5 and 6 of u (top is unsigned) + + movq U+5*N(%rsp), %rax + xorq %r9, %rax + andq %r8, %r9 + negq %r9 + mulq %r8 + addq %rax, %r15 + adcq %rdx, %r9 + movq V+5*N(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %r9 + mulq %r10 + addq %rax, %r15 + movq %r15, U+5*N(%rsp) + adcq %rdx, %r9 + movq %r9, U+6*N(%rsp) + +// Montgomery reduce u + + amontred(u) + +// Perform final strict reduction mod p_384 and copy to output + + movl $0xffffffff, %eax + movq %rax, %rbx + notq %rbx + xorl %ebp, %ebp + leaq -2(%rbp), %rcx + leaq -1(%rbp), %rdx + + movq U(%rsp), %r8 + subq %rax, %r8 + movq U+N(%rsp), %r9 + sbbq %rbx, %r9 + movq U+2*N(%rsp), %r10 + sbbq %rcx, %r10 + movq U+3*N(%rsp), %r11 + sbbq %rdx, %r11 + movq U+4*N(%rsp), %r12 + sbbq %rdx, %r12 + movq U+5*N(%rsp), %r13 + sbbq %rdx, %r13 + + cmovcq U(%rsp), %r8 + cmovcq U+N(%rsp), %r9 + cmovcq U+2*N(%rsp), %r10 + cmovcq U+3*N(%rsp), %r11 + cmovcq U+4*N(%rsp), %r12 + cmovcq U+5*N(%rsp), %r13 + + movq res, %rdi + movq %r8, (%rdi) + movq %r9, N(%rdi) + movq %r10, 2*N(%rdi) + movq %r11, 3*N(%rdi) + movq %r12, 4*N(%rdi) + movq %r13, 5*N(%rdi) + +// Restore stack and registers + + CFI_INC_RSP(NSPACE) + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_montinv_p384) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_montmul_p384.S b/cbits/s2n/x86_att/bignum_montmul_p384.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_montmul_p384.S @@ -0,0 +1,291 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Montgomery multiply, z := (x * y / 2^384) mod p_384 +// Inputs x[6], y[6]; output z[6] +// +// extern void bignum_montmul_p384(uint64_t z[static 6], +// const uint64_t x[static 6], +// const uint64_t y[static 6]); +// +// Does z := (2^{-384} * x * y) mod p_384, assuming that the inputs x and y +// satisfy x * y <= 2^384 * p_384 (in particular this is true if we are in +// the "usual" case x < p_384 and y < p_384). +// +// Standard x86-64 ABI: RDI = z, RSI = x, RDX = y +// Microsoft x64 ABI: RCX = z, RDX = x, R8 = y +// ----------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montmul_p384) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montmul_p384) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montmul_p384) + .text + +#define z %rdi +#define x %rsi + +// We move the y argument here so we can use %rdx for multipliers + +#define y %rcx + +// Some temp registers for the last correction stage + +#define d %rax +#define u %rdx +#define v %rcx +#define w %rbx + +// Add %rdx * m into a register-pair (high,low) +// maintaining consistent double-carrying with adcx and adox, +// using %rax and %rbx as temporaries + +#define mulpadd(high,low,m) \ + mulxq m, %rax, %rbx ; \ + adcxq %rax, low ; \ + adoxq %rbx, high + +// Core one-step Montgomery reduction macro. Takes input in +// [d7;d6;d5;d4;d3;d2;d1;d0] and returns result in [d7;d6;d5;d4;d3;d2;d1], +// adding to the existing contents, re-using d0 as a temporary internally +// +// We want to add (2^384 - 2^128 - 2^96 + 2^32 - 1) * w +// where w = [d0 + (d0<<32)] mod 2^64 +// +// montredc(d7,d6,d5,d4,d3,d2,d1,d0) +// +// This particular variant, with its mix of addition and subtraction +// at the top, is not intended to maintain a coherent carry or borrow out. +// It is assumed the final result would fit in [d7;d6;d5;d4;d3;d2;d1]. +// which is always the case here as the top word is even always in {0,1} + +#define montredc(d7,d6,d5,d4,d3,d2,d1,d0) \ +/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64 */ \ + movq d0, %rdx ; \ + shlq $32, %rdx ; \ + addq d0, %rdx ; \ +/* Construct [%rbp;%rbx;%rax;-] = (2^384 - p_384) * w */ \ +/* We know the lowest word will cancel so we can re-use d0 as a temp */ \ + xorl %ebp, %ebp ; \ + movq $0xffffffff00000001, %rax ; \ + mulxq %rax, %rbx, %rax ; \ + movl $0x00000000ffffffff, %ebx ; \ + mulxq %rbx, d0, %rbx ; \ + adcq d0, %rax ; \ + adcq %rdx, %rbx ; \ + adcl %ebp, %ebp ; \ +/* Now subtract that and add 2^384 * w */ \ + subq %rax, d1 ; \ + sbbq %rbx, d2 ; \ + sbbq %rbp, d3 ; \ + sbbq $0, d4 ; \ + sbbq $0, d5 ; \ + sbbq $0, %rdx ; \ + addq %rdx, d6 ; \ + adcq $0, d7 + +S2N_BN_SYMBOL(bignum_montmul_p384): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx +#endif + +// Save more registers to play with + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Copy y into a safe register to start with + + movq %rdx, y + +// Do row 0 computation, which is a bit different: +// set up initial window [%r14,%r13,%r12,%r11,%r10,%r9,%r8] = y[0] * x +// Unlike later, we only need a single carry chain + + movq (y), %rdx + xorl %r15d, %r15d + mulxq (x), %r8, %r9 + mulxq 8(x), %rbx, %r10 + addq %rbx, %r9 + mulxq 16(x), %rbx, %r11 + adcq %rbx, %r10 + mulxq 24(x), %rbx, %r12 + adcq %rbx, %r11 + mulxq 32(x), %rbx, %r13 + adcq %rbx, %r12 + mulxq 40(x), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + +// Montgomery reduce the zeroth window + + montredc(%r15, %r14,%r13,%r12,%r11,%r10,%r9,%r8) + +// Add row 1 + + movq 8(y), %rdx + xorl %r8d, %r8d + mulpadd(%r10,%r9,(x)) + mulpadd(%r11,%r10, 8(x)) + mulpadd(%r12,%r11,16(x)) + mulpadd(%r13,%r12,24(x)) + mulpadd(%r14,%r13,32(x)) + adoxq %r8, %r15 + mulxq 40(x), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + +// Montgomery reduce window 1 + + montredc(%r8, %r15,%r14,%r13,%r12,%r11,%r10,%r9) + +// Add row 2 + + movq 16(y), %rdx + xorl %r9d, %r9d + mulpadd(%r11,%r10,(x)) + mulpadd(%r12,%r11,8(x)) + mulpadd(%r13,%r12,16(x)) + mulpadd(%r14,%r13,24(x)) + mulpadd(%r15,%r14,32(x)) + adoxq %r9, %r8 + mulxq 40(x), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + +// Montgomery reduce window 2 + + montredc(%r9, %r8,%r15,%r14,%r13,%r12,%r11,%r10) + +// Add row 3 + + movq 24(y), %rdx + xorl %r10d, %r10d + mulpadd(%r12,%r11,(x)) + mulpadd(%r13,%r12,8(x)) + mulpadd(%r14,%r13,16(x)) + mulpadd(%r15,%r14,24(x)) + mulpadd(%r8,%r15,32(x)) + adoxq %r10, %r9 + mulxq 40(x), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + +// Montgomery reduce window 3 + + montredc(%r10, %r9,%r8,%r15,%r14,%r13,%r12,%r11) + +// Add row 4 + + movq 32(y), %rdx + xorl %r11d, %r11d + mulpadd(%r13,%r12,(x)) + mulpadd(%r14,%r13,8(x)) + mulpadd(%r15,%r14,16(x)) + mulpadd(%r8,%r15,24(x)) + mulpadd(%r9,%r8,32(x)) + adoxq %r11, %r10 + mulxq 40(x), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + +// Montgomery reduce window 4 + + montredc(%r11, %r10,%r9,%r8,%r15,%r14,%r13,%r12) + +// Add row 5 + + movq 40(y), %rdx + xorl %r12d, %r12d + mulpadd(%r14,%r13,(x)) + mulpadd(%r15,%r14,8(x)) + mulpadd(%r8,%r15,16(x)) + mulpadd(%r9,%r8,24(x)) + mulpadd(%r10,%r9,32(x)) + adoxq %r12, %r11 + mulxq 40(x), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + +// Montgomery reduce window 5 + + montredc(%r12, %r11,%r10,%r9,%r8,%r15,%r14,%r13) + +// We now have a pre-reduced 7-word form z = [%r12; %r11;%r10;%r9;%r8;%r15;%r14] +// Next, accumulate in different registers z - p_384, or more precisely +// +// [%r12; %r13;%rbp;%rdx;%rcx;%rbx;%rax] = z + (2^384 - p_384) + + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + + movq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0x00000000ffffffff, %ebx + adcq %r15, %rbx + movl $0x0000000000000001, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0, %r12 + +// ~ZF <=> %r12 >= 1 <=> z + (2^384 - p_384) >= 2^384 <=> z >= p_384, which +// determines whether to use the further reduced argument or the original z. + + cmovnzq %rax, %r14 + cmovnzq %rbx, %r15 + cmovnzq %rcx, %r8 + cmovnzq %rdx, %r9 + cmovnzq %rbp, %r10 + cmovnzq %r13, %r11 + +// Write back the result + + movq %r14, (z) + movq %r15, 8(z) + movq %r8, 16(z) + movq %r9, 24(z) + movq %r10, 32(z) + movq %r11, 40(z) + +// Restore registers and return + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_montmul_p384) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_montmul_p384_alt.S b/cbits/s2n/x86_att/bignum_montmul_p384_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_montmul_p384_alt.S @@ -0,0 +1,316 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Montgomery multiply, z := (x * y / 2^384) mod p_384 +// Inputs x[6], y[6]; output z[6] +// +// extern void bignum_montmul_p384_alt(uint64_t z[static 6], +// const uint64_t x[static 6], +// const uint64_t y[static 6]); +// +// Does z := (2^{-384} * x * y) mod p_384, assuming that the inputs x and y +// satisfy x * y <= 2^384 * p_384 (in particular this is true if we are in +// the "usual" case x < p_384 and y < p_384). +// +// Standard x86-64 ABI: RDI = z, RSI = x, RDX = y +// Microsoft x64 ABI: RCX = z, RDX = x, R8 = y +// ----------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montmul_p384_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montmul_p384_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montmul_p384_alt) + .text + +#define z %rdi +#define x %rsi + +// We move the y argument here so we can use %rdx for multipliers + +#define y %rcx + +// Some temp registers for the last correction stage + +#define d %rax +#define u %rdx +#define v %rcx +#define w %rbx + +// Add %rbx * m into a register-pair (high,low) maintaining consistent +// carry-catching with carry (negated, as bitmask) and using %rax and %rdx +// as temporaries + +#define mulpadd(carry,high,low,m) \ + movq m, %rax ; \ + mulq %rbx; \ + subq carry, %rdx ; \ + addq %rax, low ; \ + adcq %rdx, high ; \ + sbbq carry, carry + +// Initial version assuming no carry-in + +#define mulpadi(carry,high,low,m) \ + movq m, %rax ; \ + mulq %rbx; \ + addq %rax, low ; \ + adcq %rdx, high ; \ + sbbq carry, carry + +// End version not catching the top carry-out + +#define mulpade(carry,high,low,m) \ + movq m, %rax ; \ + mulq %rbx; \ + subq carry, %rdx ; \ + addq %rax, low ; \ + adcq %rdx, high + +// Core one-step Montgomery reduction macro. Takes input in +// [d7;d6;d5;d4;d3;d2;d1;d0] and returns result in [d7;d6;d5;d4;d3;d2;d1], +// adding to the existing contents, re-using d0 as a temporary internally +// +// We want to add (2^384 - 2^128 - 2^96 + 2^32 - 1) * w +// where w = [d0 + (d0<<32)] mod 2^64 +// +// montredc(d7,d6,d5,d4,d3,d2,d1,d0) +// +// This particular variant, with its mix of addition and subtraction +// at the top, is not intended to maintain a coherent carry or borrow out. +// It is assumed the final result would fit in [d7;d6;d5;d4;d3;d2;d1]. +// which is always the case here as the top word is even always in {0,1} + +#define montredc(d7,d6,d5,d4,d3,d2,d1,d0) \ +/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64 */ \ + movq d0, %rbx ; \ + shlq $32, %rbx ; \ + addq d0, %rbx ; \ +/* Construct [%rbp;%rdx;%rax;-] = (2^384 - p_384) * w */ \ +/* We know the lowest word will cancel so we can re-use d0 as a temp */ \ + xorl %ebp, %ebp ; \ + movq $0xffffffff00000001, %rax ; \ + mulq %rbx; \ + movq %rdx, d0 ; \ + movq $0x00000000ffffffff, %rax ; \ + mulq %rbx; \ + addq d0, %rax ; \ + adcq %rbx, %rdx ; \ + adcl %ebp, %ebp ; \ +/* Now subtract that and add 2^384 * w */ \ + subq %rax, d1 ; \ + sbbq %rdx, d2 ; \ + sbbq %rbp, d3 ; \ + sbbq $0, d4 ; \ + sbbq $0, d5 ; \ + sbbq $0, %rbx ; \ + addq %rbx, d6 ; \ + adcq $0, d7 + +S2N_BN_SYMBOL(bignum_montmul_p384_alt): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx +#endif + +// Save more registers to play with + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Copy y into a safe register to start with + + movq %rdx, y + +// Do row 0 computation, which is a bit different: +// set up initial window [%r14,%r13,%r12,%r11,%r10,%r9,%r8] = y[0] * x +// Unlike later, we only need a single carry chain + + movq (y), %rbx + movq (x), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + + movq 8(x), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + + movq 16(x), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + + movq 24(x), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + + movq 32(x), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + + movq 40(x), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + + xorl %r15d, %r15d + +// Montgomery reduce the zeroth window + + montredc(%r15, %r14,%r13,%r12,%r11,%r10,%r9,%r8) + +// Add row 1 + + movq 8(y), %rbx + mulpadi(%r8,%r10,%r9,(x)) + mulpadd(%r8,%r11,%r10,8(x)) + mulpadd(%r8,%r12,%r11,16(x)) + mulpadd(%r8,%r13,%r12,24(x)) + mulpadd(%r8,%r14,%r13,32(x)) + mulpadd(%r8,%r15,%r14,40(x)) + negq %r8 + +// Montgomery reduce window 1 + + montredc(%r8, %r15,%r14,%r13,%r12,%r11,%r10,%r9) + +// Add row 2 + + movq 16(y), %rbx + mulpadi(%r9,%r11,%r10,(x)) + mulpadd(%r9,%r12,%r11,8(x)) + mulpadd(%r9,%r13,%r12,16(x)) + mulpadd(%r9,%r14,%r13,24(x)) + mulpadd(%r9,%r15,%r14,32(x)) + mulpadd(%r9,%r8,%r15,40(x)) + negq %r9 + +// Montgomery reduce window 2 + + montredc(%r9, %r8,%r15,%r14,%r13,%r12,%r11,%r10) + +// Add row 3 + + movq 24(y), %rbx + mulpadi(%r10,%r12,%r11,(x)) + mulpadd(%r10,%r13,%r12,8(x)) + mulpadd(%r10,%r14,%r13,16(x)) + mulpadd(%r10,%r15,%r14,24(x)) + mulpadd(%r10,%r8,%r15,32(x)) + mulpadd(%r10,%r9,%r8,40(x)) + negq %r10 + +// Montgomery reduce window 3 + + montredc(%r10, %r9,%r8,%r15,%r14,%r13,%r12,%r11) + +// Add row 4 + + movq 32(y), %rbx + mulpadi(%r11,%r13,%r12,(x)) + mulpadd(%r11,%r14,%r13,8(x)) + mulpadd(%r11,%r15,%r14,16(x)) + mulpadd(%r11,%r8,%r15,24(x)) + mulpadd(%r11,%r9,%r8,32(x)) + mulpadd(%r11,%r10,%r9,40(x)) + negq %r11 + +// Montgomery reduce window 4 + + montredc(%r11, %r10,%r9,%r8,%r15,%r14,%r13,%r12) + +// Add row 5 + + movq 40(y), %rbx + mulpadi(%r12,%r14,%r13,(x)) + mulpadd(%r12,%r15,%r14,8(x)) + mulpadd(%r12,%r8,%r15,16(x)) + mulpadd(%r12,%r9,%r8,24(x)) + mulpadd(%r12,%r10,%r9,32(x)) + mulpadd(%r12,%r11,%r10,40(x)) + negq %r12 + +// Montgomery reduce window 5 + + montredc(%r12, %r11,%r10,%r9,%r8,%r15,%r14,%r13) + +// We now have a pre-reduced 7-word form z = [%r12; %r11;%r10;%r9;%r8;%r15;%r14] +// Next, accumulate in different registers z - p_384, or more precisely +// +// [%r12; %r13;%rbp;%rdx;%rcx;%rbx;%rax] = z + (2^384 - p_384) + + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + + movq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0x00000000ffffffff, %ebx + adcq %r15, %rbx + movl $0x0000000000000001, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0, %r12 + +// ~ZF <=> %r12 >= 1 <=> z + (2^384 - p_384) >= 2^384 <=> z >= p_384, which +// determines whether to use the further reduced argument or the original z. + + cmovnzq %rax, %r14 + cmovnzq %rbx, %r15 + cmovnzq %rcx, %r8 + cmovnzq %rdx, %r9 + cmovnzq %rbp, %r10 + cmovnzq %r13, %r11 + +// Write back the result + + movq %r14, (z) + movq %r15, 8(z) + movq %r8, 16(z) + movq %r9, 24(z) + movq %r10, 32(z) + movq %r11, 40(z) + +// Restore registers and return + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_montmul_p384_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_montsqr_p384.S b/cbits/s2n/x86_att/bignum_montsqr_p384.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_montsqr_p384.S @@ -0,0 +1,294 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Montgomery square, z := (x^2 / 2^384) mod p_384 +// Input x[6]; output z[6] +// +// extern void bignum_montsqr_p384(uint64_t z[static 6], +// const uint64_t x[static 6]); +// +// Does z := (x^2 / 2^384) mod p_384, assuming x^2 <= 2^384 * p_384, which is +// guaranteed in particular if x < p_384 initially (the "intended" case). +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montsqr_p384) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montsqr_p384) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montsqr_p384) + .text + +#define z %rdi +#define x %rsi + +// Some temp registers for the last correction stage + +#define d %rax +#define u %rdx +#define v %r10 +#define w %r11 + +// A zero register, very often + +#define zero %rbp +#define zeroe %ebp + +// Add %rdx * m into a register-pair (high,low) +// maintaining consistent double-carrying with adcx and adox, +// using %rax and %rbx as temporaries + +#define mulpadd(high,low,m) \ + mulxq m, %rax, %rbx ; \ + adcxq %rax, low ; \ + adoxq %rbx, high + +// Core one-step "short" Montgomery reduction macro. Takes input in +// [d5;d4;d3;d2;d1;d0] and returns result in [d6;d5;d4;d3;d2;d1], +// adding to the existing [d5;d4;d3;d2;d1] and re-using d0 as a +// temporary internally, as well as %rax, %rbx and %rdx. +// It is OK for d6 and d0 to be the same register (they often are) +// +// We want to add (2^384 - 2^128 - 2^96 + 2^32 - 1) * w +// where w = [d0 + (d0<<32)] mod 2^64 +// +// montreds(d6,d5,d4,d3,d2,d1,d0) + +#define montreds(d6,d5,d4,d3,d2,d1,d0) \ +/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64 */ \ + movq d0, %rdx ; \ + shlq $32, %rdx ; \ + addq d0, %rdx ; \ +/* Construct [%rbx;d0;%rax;-] = (2^384 - p_384) * w */ \ +/* We know the lowest word will cancel so we can re-use d0 */ \ +/* and %rbx as temps. */ \ + movq $0xffffffff00000001, %rax ; \ + mulxq %rax, d0, %rax ; \ + movl $0x00000000ffffffff, %ebx ; \ + mulxq %rbx, %rbx, d0 ; \ + addq %rbx, %rax ; \ + adcq %rdx, d0 ; \ + movl $0, %ebx ; \ + adcq %rbx, %rbx ; \ +/* Now subtract that and add 2^384 * w */ \ + subq %rax, d1 ; \ + sbbq d0, d2 ; \ + sbbq %rbx, d3 ; \ + sbbq $0, d4 ; \ + sbbq $0, d5 ; \ + movq %rdx, d6 ; \ + sbbq $0, d6 + +S2N_BN_SYMBOL(bignum_montsqr_p384): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Save more registers to play with + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Set up an initial window [%rcx;%r15;...%r9] = [34;05;03;01] +// Note that we are using %rcx as the first step past the rotating window + + movq (x), %rdx + mulxq 8(x), %r9, %r10 + mulxq 24(x), %r11, %r12 + mulxq 40(x), %r13, %r14 + movq 24(x), %rdx + mulxq 32(x), %r15, %rcx + +// Clear our zero register, and also initialize the flags for the carry chain + + xorl zeroe, zeroe + +// Chain in the addition of 02 + 12 + 13 + 14 + 15 to that window +// (no carry-out possible) + + movq 16(x), %rdx + mulpadd(%r11,%r10,(x)) + mulpadd(%r12,%r11,8(x)) + movq 8(x), %rdx + mulpadd(%r13,%r12,24(x)) + mulpadd(%r14,%r13,32(x)) + mulpadd(%r15,%r14,40(x)) + adcxq zero, %r15 + adoxq zero, %rcx + adcq zero, %rcx + +// Again zero out the flags. Actually they are already cleared but it may +// help decouple these in the OOO engine not to wait for the chain above + + xorl zeroe, zeroe + +// Now chain in the 04 + 23 + 24 + 25 + 35 + 45 terms +// We are running out of registers in our rotating window, so we start +// using %rbx (and hence need care with using mulpadd after this). Thus +// our result so far is in [%rbp;%rbx;%rcx;%r15;...%r9] + + movq 32(x), %rdx + mulpadd(%r13,%r12,(x)) + movq 16(x), %rdx + mulpadd(%r14,%r13,24(x)) + mulpadd(%r15,%r14,32(x)) + mulxq 40(x), %rax, %rdx + adcxq %rax, %r15 + adoxq %rdx, %rcx + +// First set up the last couple of spots in our window, [%rbp;%rbx] = 45 +// then add the last other term 35 + + movq 40(x), %rdx + mulxq 32(x), %rbx, %rbp + mulxq 24(x), %rax, %rdx + adcxq %rax, %rcx + adoxq %rdx, %rbx + movl $0, %eax + adcxq %rax, %rbx + adoxq %rax, %rbp + adcq %rax, %rbp + +// Just for a clear fresh start for the flags; we don't use the zero + + xorq %rax, %rax + +// Double and add to the 00 + 11 + 22 + 33 + 44 + 55 terms +// For one glorious moment the entire squaring result is all in the +// register file as [%rsi;%rbp;%rbx;%rcx;%r15;...;%r8] +// (since we've now finished with x we can re-use %rsi) + + movq (x), %rdx + mulxq (x), %r8, %rax + adcxq %r9, %r9 + adoxq %rax, %r9 + movq 8(x), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 16(x), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 24(x), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %r15, %r15 + adoxq %rdx, %r15 + movq 32(x), %rdx + mulxq %rdx, %rax, %rdx + adcxq %rcx, %rcx + adoxq %rax, %rcx + adcxq %rbx, %rbx + adoxq %rdx, %rbx + movq 40(x), %rdx + mulxq %rdx, %rax, %rsi + adcxq %rbp, %rbp + adoxq %rax, %rbp + movl $0, %eax + adcxq %rax, %rsi + adoxq %rax, %rsi + +// We need just *one* more register as a temp for the Montgomery steps. +// Since we are writing to the z buffer anyway, make use of that to stash %rbx. + + movq %rbx, (z) + +// Montgomery reduce the %r13,...,%r8 window 6 times + + montreds(%r8,%r13,%r12,%r11,%r10,%r9,%r8) + montreds(%r9,%r8,%r13,%r12,%r11,%r10,%r9) + montreds(%r10,%r9,%r8,%r13,%r12,%r11,%r10) + montreds(%r11,%r10,%r9,%r8,%r13,%r12,%r11) + montreds(%r12,%r11,%r10,%r9,%r8,%r13,%r12) + montreds(%r13,%r12,%r11,%r10,%r9,%r8,%r13) + +// Now we can safely restore %rbx before accumulating + + movq (z), %rbx + + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rsi + movl $0, %r8d + adcq %r8, %r8 + +// We now have a pre-reduced 7-word form z = [%r8; %rsi;%rbp;%rbx;%rcx;%r15;%r14] +// Next, accumulate in different registers z - p_384, or more precisely +// +// [%r8; %r13;%r12;%r11;%r10;%r9;%rax] = z + (2^384 - p_384) + + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0x00000000ffffffff, %r9d + adcq %r15, %r9 + movl $0x0000000000000001, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rsi, %r13 + adcq $0, %r8 + +// ~ZF <=> %r12 >= 1 <=> z + (2^384 - p_384) >= 2^384 <=> z >= p_384, which +// determines whether to use the further reduced argument or the original z. + + cmovnzq %rax, %r14 + cmovnzq %r9, %r15 + cmovnzq %r10, %rcx + cmovnzq %r11, %rbx + cmovnzq %r12, %rbp + cmovnzq %r13, %rsi + +// Write back the result + + movq %r14, (z) + movq %r15, 8(z) + movq %rcx, 16(z) + movq %rbx, 24(z) + movq %rbp, 32(z) + movq %rsi, 40(z) + +// Restore registers and return + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_montsqr_p384) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_montsqr_p384_alt.S b/cbits/s2n/x86_att/bignum_montsqr_p384_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_montsqr_p384_alt.S @@ -0,0 +1,339 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Montgomery square, z := (x^2 / 2^384) mod p_384 +// Input x[6]; output z[6] +// +// extern void bignum_montsqr_p384_alt(uint64_t z[static 6], +// const uint64_t x[static 6]); +// +// Does z := (x^2 / 2^384) mod p_384, assuming x^2 <= 2^384 * p_384, which is +// guaranteed in particular if x < p_384 initially (the "intended" case). +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_montsqr_p384_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_montsqr_p384_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_montsqr_p384_alt) + .text + +#define z %rdi +#define x %rsi + +// Some temp registers for the last correction stage + +#define d %rax +#define u %rdx +#define v %r10 +#define w %r11 + +// A zero register, very often + +#define zero %rbp +#define zeroe %ebp + +// Add %rbx * m into a register-pair (high,low) maintaining consistent +// carry-catching with carry (negated, as bitmask) and using %rax and %rdx +// as temporaries + +#define mulpadd(carry,high,low,m) \ + movq m, %rax ; \ + mulq %rbx; \ + subq carry, %rdx ; \ + addq %rax, low ; \ + adcq %rdx, high ; \ + sbbq carry, carry + +// Initial version assuming no carry-in + +#define mulpadi(carry,high,low,m) \ + movq m, %rax ; \ + mulq %rbx; \ + addq %rax, low ; \ + adcq %rdx, high ; \ + sbbq carry, carry + +// End version not catching the top carry-out + +#define mulpade(carry,high,low,m) \ + movq m, %rax ; \ + mulq %rbx; \ + subq carry, %rdx ; \ + addq %rax, low ; \ + adcq %rdx, high + +// Core one-step "short" Montgomery reduction macro. Takes input in +// [d5;d4;d3;d2;d1;d0] and returns result in [d6;d5;d4;d3;d2;d1], +// adding to the existing [d5;d4;d3;d2;d1] and re-using d0 as a +// temporary internally, as well as %rax, %rbx and %rdx. +// It is OK for d6 and d0 to be the same register (they often are) +// +// We want to add (2^384 - 2^128 - 2^96 + 2^32 - 1) * w +// where w = [d0 + (d0<<32)] mod 2^64 +// +// montreds(d6,d5,d4,d3,d2,d1,d0) + +#define montreds(d6,d5,d4,d3,d2,d1,d0) \ +/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64 */ \ + movq d0, %rbx ; \ + shlq $32, %rbx ; \ + addq d0, %rbx ; \ +/* Construct [%rax;%rdx;d0;-] = (2^384 - p_384) * w */ \ +/* We know the lowest word will cancel so we can re-use d0 */ \ +/* and %rbx as temps. */ \ + movq $0xffffffff00000001, %rax ; \ + mulq %rbx; \ + movq %rdx, d0 ; \ + movq $0x00000000ffffffff, %rax ; \ + mulq %rbx; \ + addq %rax, d0 ; \ + movl $0, %eax ; \ + adcq %rbx, %rdx ; \ + adcl %eax, %eax ; \ +/* Now subtract that and add 2^384 * w */ \ + subq d0, d1 ; \ + sbbq %rdx, d2 ; \ + sbbq %rax, d3 ; \ + sbbq $0, d4 ; \ + sbbq $0, d5 ; \ + movq %rbx, d6 ; \ + sbbq $0, d6 + +S2N_BN_SYMBOL(bignum_montsqr_p384_alt): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Save more registers to play with + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Set up an initial window [%rcx;%r15;...%r9] = [34;05;03;01] +// Note that we are using %rcx as the first step past the rotating window + + movq (x), %rbx + movq 8(x), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + + movq 24(x), %rax + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + + movq 40(x), %rax + mulq %rbx + movq %rax, %r13 + movq %rdx, %r14 + + movq 24(x), %rax + mulq 32(x) + movq %rax, %r15 + movq %rdx, %rcx + +// Chain in the addition of 02 + 12 + 13 + 14 + 15 to that window +// (no carry-out possible) + + movq 16(x), %rbx + mulpadi(%rbp,%r11,%r10,(x)) + mulpadd(%rbp,%r12,%r11,8(x)) + movq 8(x), %rbx + mulpadd(%rbp,%r13,%r12,24(x)) + mulpadd(%rbp,%r14,%r13,32(x)) + mulpade(%rbp,%r15,%r14,40(x)) + adcq $0, %rcx + +// Now chain in the 04 + 23 + 24 + 25 + 35 + 45 terms +// We are running out of registers in our rotating window, so we start +// using %rbx (and hence need care with using mulpadd after this). Thus +// our result so far is in [%rbp;%rbx;%rcx;%r15;...%r9] + + movq 32(x), %rbx + mulpadi(%rbp,%r13,%r12,(x)) + movq 16(x), %rbx + mulpadd(%rbp,%r14,%r13,24(x)) + mulpadd(%rbp,%r15,%r14,32(x)) + mulpadd(%rbp,%rcx,%r15,40(x)) + + xorl %ebx, %ebx + movq 24(x), %rax + mulq 40(x) + subq %rbp, %rdx + xorl %ebp, %ebp + addq %rax, %rcx + adcq %rdx, %rbx + adcl %ebp, %ebp + movq 32(x), %rax + mulq 40(x) + addq %rax, %rbx + adcq %rdx, %rbp + +// Double the window as [%r8;%rbp;%rbx;%rcx;%r15;...%r9] + + xorl %r8d, %r8d + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %r15, %r15 + adcq %rcx, %rcx + adcq %rbx, %rbx + adcq %rbp, %rbp + adcl %r8d, %r8d + +// Add the doubled window to the 00 + 11 + 22 + 33 + 44 + 55 terms +// For one glorious moment the entire squaring result is all in the +// register file as [%rsi;%rbp;%rbx;%rcx;%r15;...;%r8] +// (since we've now finished with x we can re-use %rsi). But since +// we are so close to running out of registers, we do a bit of +// reshuffling and temporary storage in the output buffer. + + movq (x), %rax + mulq %rax + movq %r8, (z) + movq %rax, %r8 + movq 8(x), %rax + movq %rbp, 8(z) + addq %rdx, %r9 + sbbq %rbp, %rbp + + mulq %rax + negq %rbp + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + + movq 16(x), %rax + mulq %rax + negq %rbp + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + + movq 24(x), %rax + mulq %rax + negq %rbp + adcq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + + movq 32(x), %rax + mulq %rax + negq %rbp + adcq %rax, %rcx + adcq %rdx, %rbx + sbbq %rbp, %rbp + + movq 40(x), %rax + mulq %rax + negq %rbp + adcq 8(z), %rax + adcq (z), %rdx + movq %rax, %rbp + movq %rdx, %rsi + +// We need just *one* more register as a temp for the Montgomery steps. +// Since we are writing to the z buffer anyway, make use of that again +// to stash %rbx. + + movq %rbx, (z) + +// Montgomery reduce the %r13,...,%r8 window 6 times + + montreds(%r8,%r13,%r12,%r11,%r10,%r9,%r8) + montreds(%r9,%r8,%r13,%r12,%r11,%r10,%r9) + montreds(%r10,%r9,%r8,%r13,%r12,%r11,%r10) + montreds(%r11,%r10,%r9,%r8,%r13,%r12,%r11) + montreds(%r12,%r11,%r10,%r9,%r8,%r13,%r12) + montreds(%r13,%r12,%r11,%r10,%r9,%r8,%r13) + +// Now we can safely restore %rbx before accumulating + + movq (z), %rbx + + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rsi + movl $0, %r8d + adcq %r8, %r8 + +// We now have a pre-reduced 7-word form z = [%r8; %rsi;%rbp;%rbx;%rcx;%r15;%r14] +// Next, accumulate in different registers z - p_384, or more precisely +// +// [%r8; %r13;%r12;%r11;%r10;%r9;%rax] = z + (2^384 - p_384) + + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0x00000000ffffffff, %r9d + adcq %r15, %r9 + movl $0x0000000000000001, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rsi, %r13 + adcq $0, %r8 + +// ~ZF <=> %r12 >= 1 <=> z + (2^384 - p_384) >= 2^384 <=> z >= p_384, which +// determines whether to use the further reduced argument or the original z. + + cmovnzq %rax, %r14 + cmovnzq %r9, %r15 + cmovnzq %r10, %rcx + cmovnzq %r11, %rbx + cmovnzq %r12, %rbp + cmovnzq %r13, %rsi + +// Write back the result + + movq %r14, (z) + movq %r15, 8(z) + movq %rcx, 16(z) + movq %rbx, 24(z) + movq %rbp, 32(z) + movq %rsi, 40(z) + +// Restore registers and return + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_montsqr_p384_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_mul.S b/cbits/s2n/x86_att/bignum_mul.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_mul.S @@ -0,0 +1,160 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Multiply z := x * y +// Inputs x[m], y[n]; output z[k] +// +// extern void bignum_mul(uint64_t k, uint64_t *z, uint64_t m, const uint64_t *x, +// uint64_t n, const uint64_t *y); +// +// Does the "z := x * y" operation where x is m digits, y is n, result z is k. +// Truncates the result in general unless k >= m + n +// +// Standard x86-64 ABI: RDI = k, RSI = z, RDX = m, RCX = x, R8 = n, R9 = y +// Microsoft x64 ABI: RCX = k, RDX = z, R8 = m, R9 = x, [RSP+40] = n, [RSP+48] = y +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_mul) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_mul) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_mul) + .text + .balign 32 + +// These are actually right + +#define p %rdi +#define z %rsi +#define n %r8 + +// These are not + +#define c %r15 +#define h %r14 +#define l %r13 +#define x %r12 +#define y %r11 +#define i %rbx +#define k %r10 +#define m %rbp + +// These are always local scratch since multiplier result is in these + +#define a %rax +#define d %rdx + + + +S2N_BN_SYMBOL(bignum_mul): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + movq %r9, %rcx + movq 56(%rsp), %r8 + movq 64(%rsp), %r9 +#endif + +// We use too many registers, and also we need %rax:%rdx for multiplications + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + movq %rdx, m + +// If the result size is zero, do nothing +// Note that even if either or both inputs has size zero, we can't +// just give up because we at least need to zero the output array +// If we did a multiply-add variant, however, then we could + + testq p, p + jz Lbignum_mul_end + +// Set initial 2-part sum to zero (we zero c inside the body) + + xorq h, h + xorq l, l + +// Otherwise do outer loop k = 0 ... k = p - 1 + + xorq k, k + +Lbignum_mul_outerloop: + +// Zero our carry term first; we eventually want it and a zero is useful now +// Set a = max 0 (k + 1 - n), i = min (k + 1) m +// This defines the range a <= j < i for the inner summation +// Note that since k < p < 2^64 we can assume k + 1 doesn't overflow +// And since we want to increment it anyway, we might as well do it now + + xorq c, c // c = 0 + incq k // k = k + 1 + + movq k, a // a = k + 1 + subq n, a // a = k + 1 - n + cmovcq c, a // a = max 0 (k + 1 - n) + + movq m, i // i = m + cmpq m, k // CF <=> k + 1 < m + cmovcq k, i // i = min (k + 1) m + +// Turn i into a loop count, and skip things if it's <= 0 +// Otherwise set up initial pointers x -> x0[a] and y -> y0[k - a] +// and then launch into the main inner loop, postdecrementing i + + movq k, d + subq i, d + subq a, i + jbe Lbignum_mul_innerend + leaq (%rcx,a,8), x + leaq -8(%r9,d,8), y + +Lbignum_mul_innerloop: + movq (y,i,8), %rax + mulq (x) + addq $8, x + addq %rax, l + adcq %rdx, h + adcq $0, c + decq i + jnz Lbignum_mul_innerloop + +Lbignum_mul_innerend: + + movq l, (z) + movq h, l + movq c, h + addq $8, z + + cmpq p, k + jc Lbignum_mul_outerloop + +Lbignum_mul_end: + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_mul) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_mul_8_16.S b/cbits/s2n/x86_att/bignum_mul_8_16.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_mul_8_16.S @@ -0,0 +1,264 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Multiply z := x * y +// Inputs x[8], y[8]; output z[16] +// +// extern void bignum_mul_8_16(uint64_t z[static 16], const uint64_t x[static 8], +// const uint64_t y[static 8]); +// +// Standard x86-64 ABI: RDI = z, RSI = x, RDX = y +// Microsoft x64 ABI: RCX = z, RDX = x, R8 = y +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_mul_8_16) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_mul_8_16) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_mul_8_16) + .text + +// These are actually right + +#define z %rdi +#define x %rsi + +// Copied in or set up + +#define y %rcx + +// A zero register + +#define zero %rbp +#define zeroe %ebp + +// mulpadd i, j adds x[i] * rdx (now assumed = y[j]) into the window at i+j + +.macro mulpadd arg1,arg2 + mulxq 8*\arg1(x), %rax, %rbx +.if ((\arg1 + \arg2) % 8 == 0) + adcxq %rax, %r8 + adoxq %rbx, %r9 +.elseif ((\arg1 + \arg2) % 8 == 1) + adcxq %rax, %r9 + adoxq %rbx, %r10 +.elseif ((\arg1 + \arg2) % 8 == 2) + adcxq %rax, %r10 + adoxq %rbx, %r11 +.elseif ((\arg1 + \arg2) % 8 == 3) + adcxq %rax, %r11 + adoxq %rbx, %r12 +.elseif ((\arg1 + \arg2) % 8 == 4) + adcxq %rax, %r12 + adoxq %rbx, %r13 +.elseif ((\arg1 + \arg2) % 8 == 5) + adcxq %rax, %r13 + adoxq %rbx, %r14 +.elseif ((\arg1 + \arg2) % 8 == 6) + adcxq %rax, %r14 + adoxq %rbx, %r15 +.elseif ((\arg1 + \arg2) % 8 == 7) + adcxq %rax, %r15 + adoxq %rbx, %r8 +.endif + +.endm + +// mulpade i, j adds x[i] * rdx (now assumed = y[j]) into the window at i+j +// but re-creates the top word assuming nothing to add there + +.macro mulpade arg1,arg2 +.if ((\arg1 + \arg2) % 8 == 0) + mulxq 8*\arg1(x), %rax, %r9 + adcxq %rax, %r8 + adoxq zero, %r9 +.elseif ((\arg1 + \arg2) % 8 == 1) + mulxq 8*\arg1(x), %rax, %r10 + adcxq %rax, %r9 + adoxq zero, %r10 +.elseif ((\arg1 + \arg2) % 8 == 2) + mulxq 8*\arg1(x), %rax, %r11 + adcxq %rax, %r10 + adoxq zero, %r11 +.elseif ((\arg1 + \arg2) % 8 == 3) + mulxq 8*\arg1(x), %rax, %r12 + adcxq %rax, %r11 + adoxq zero, %r12 +.elseif ((\arg1 + \arg2) % 8 == 4) + mulxq 8*\arg1(x), %rax, %r13 + adcxq %rax, %r12 + adoxq zero, %r13 +.elseif ((\arg1 + \arg2) % 8 == 5) + mulxq 8*\arg1(x), %rax, %r14 + adcxq %rax, %r13 + adoxq zero, %r14 +.elseif ((\arg1 + \arg2) % 8 == 6) + mulxq 8*\arg1(x), %rax, %r15 + adcxq %rax, %r14 + adoxq zero, %r15 +.elseif ((\arg1 + \arg2) % 8 == 7) + mulxq 8*\arg1(x), %rax, %r8 + adcxq %rax, %r15 + adoxq zero, %r8 +.endif + +.endm + +// Add in the whole j'th row + +.macro addrow arg1 + movq 8*\arg1(y), %rdx + xorl zeroe, zeroe + + mulpadd 0, \arg1 + +.if (\arg1 % 8 == 0) + movq %r8, 8*\arg1(z) +.elseif (\arg1 % 8 == 1) + movq %r9, 8*\arg1(z) +.elseif (\arg1 % 8 == 2) + movq %r10, 8*\arg1(z) +.elseif (\arg1 % 8 == 3) + movq %r11, 8*\arg1(z) +.elseif (\arg1 % 8 == 4) + movq %r12, 8*\arg1(z) +.elseif (\arg1 % 8 == 5) + movq %r13, 8*\arg1(z) +.elseif (\arg1 % 8 == 6) + movq %r14, 8*\arg1(z) +.elseif (\arg1 % 8 == 7) + movq %r15, 8*\arg1(z) +.endif + + mulpadd 1, \arg1 + mulpadd 2, \arg1 + mulpadd 3, \arg1 + mulpadd 4, \arg1 + mulpadd 5, \arg1 + mulpadd 6, \arg1 + mulpade 7, \arg1 + +.if (\arg1 % 8 == 0) + adcq zero, %r8 +.elseif (\arg1 % 8 == 1) + adcq zero, %r9 +.elseif (\arg1 % 8 == 2) + adcq zero, %r10 +.elseif (\arg1 % 8 == 3) + adcq zero, %r11 +.elseif (\arg1 % 8 == 4) + adcq zero, %r12 +.elseif (\arg1 % 8 == 5) + adcq zero, %r13 +.elseif (\arg1 % 8 == 6) + adcq zero, %r14 +.elseif (\arg1 % 8 == 7) + adcq zero, %r15 +.endif + +.endm + + +S2N_BN_SYMBOL(bignum_mul_8_16): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx +#endif + +// Save more registers to play with + + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Copy y into a safe register to start with + + movq %rdx, y + +// Zero a register, which also makes sure we don't get a fake carry-in + + xorl zeroe, zeroe + +// Do the zeroth row, which is a bit different +// Write back the zero-zero product and then accumulate +// %r8,%r15,%r14,%r13,%r12,%r11,%r10,%r9 as y[0] * x from 1..8 + + movq (y), %rdx + + mulxq (x), %r8, %r9 + movq %r8, (z) + + mulxq 8(x), %rbx, %r10 + adcq %rbx, %r9 + + mulxq 16(x), %rbx, %r11 + adcq %rbx, %r10 + + mulxq 24(x), %rbx, %r12 + adcq %rbx, %r11 + + mulxq 32(x), %rbx, %r13 + adcq %rbx, %r12 + + mulxq 40(x), %rbx, %r14 + adcq %rbx, %r13 + + mulxq 48(x), %rbx, %r15 + adcq %rbx, %r14 + + mulxq 56(x), %rbx, %r8 + adcq %rbx, %r15 + adcq zero, %r8 + +// Now all the other rows in a uniform pattern + + addrow 1 + addrow 2 + addrow 3 + addrow 4 + addrow 5 + addrow 6 + addrow 7 + +// Now write back the additional columns + + movq %r8, 64(z) + movq %r9, 72(z) + movq %r10, 80(z) + movq %r11, 88(z) + movq %r12, 96(z) + movq %r13, 104(z) + movq %r14, 112(z) + movq %r15, 120(z) + +// Real epilog + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + CFI_POP(%rbp) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_mul_8_16) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_mul_p521.S b/cbits/s2n/x86_att/bignum_mul_p521.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_mul_p521.S @@ -0,0 +1,394 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Multiply modulo p_521, z := (x * y) mod p_521, assuming x and y reduced +// Inputs x[9], y[9]; output z[9] +// +// extern void bignum_mul_p521(uint64_t z[static 9], const uint64_t x[static 9], +// const uint64_t y[static 9]); +// +// Standard x86-64 ABI: RDI = z, RSI = x, RDX = y +// Microsoft x64 ABI: RCX = z, RDX = x, R8 = y +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_mul_p521) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_mul_p521) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_mul_p521) + .text + +#define z %rdi +#define x %rsi + +// Copied in + +#define y %rcx + +// mulpadd (high,low,x) adds rdx * x to a register-pair (high,low) +// maintaining consistent double-carrying with adcx and adox, +// using %rax and %rbx as temporaries. + +#define mulpadd(high,low,x) \ + mulxq x, %rax, %rbx ; \ + adcxq %rax, low ; \ + adoxq %rbx, high + +S2N_BN_SYMBOL(bignum_mul_p521): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx +#endif + +// Save more registers to play with and make temporary space on stack + + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(64) + +// Copy y into a safe register to start with + + movq %rdx, y + +// Clone of the main body of bignum_8_16, writing back the low 8 words to +// the temporary buffer on the stack and keeping the top half in %r15,...,%r8 + + xorl %ebp, %ebp + movq (y), %rdx + mulxq (x), %r8, %r9 + movq %r8, (%rsp) + mulxq 0x8(x), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x10(x), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x18(x), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0x20(x), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0x28(x), %rbx, %r14 + adcq %rbx, %r13 + mulxq 0x30(x), %rbx, %r15 + adcq %rbx, %r14 + mulxq 0x38(x), %rbx, %r8 + adcq %rbx, %r15 + adcq %rbp, %r8 + movq 0x8(y), %rdx + xorl %ebp, %ebp + mulxq (x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + movq %r9, 0x8(%rsp) + mulxq 0x8(x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x38(x), %rax, %r9 + adcxq %rax, %r8 + adoxq %rbp, %r9 + adcq %rbp, %r9 + movq 0x10(y), %rdx + xorl %ebp, %ebp + mulxq (x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq %r10, 0x10(%rsp) + mulxq 0x8(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x38(x), %rax, %r10 + adcxq %rax, %r9 + adoxq %rbp, %r10 + adcq %rbp, %r10 + movq 0x18(y), %rdx + xorl %ebp, %ebp + mulxq (x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq %r11, 0x18(%rsp) + mulxq 0x8(x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x38(x), %rax, %r11 + adcxq %rax, %r10 + adoxq %rbp, %r11 + adcq %rbp, %r11 + movq 0x20(y), %rdx + xorl %ebp, %ebp + mulxq (x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq %r12, 0x20(%rsp) + mulxq 0x8(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x38(x), %rax, %r12 + adcxq %rax, %r11 + adoxq %rbp, %r12 + adcq %rbp, %r12 + movq 0x28(y), %rdx + xorl %ebp, %ebp + mulxq (x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + movq %r13, 0x28(%rsp) + mulxq 0x8(x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x38(x), %rax, %r13 + adcxq %rax, %r12 + adoxq %rbp, %r13 + adcq %rbp, %r13 + movq 0x30(y), %rdx + xorl %ebp, %ebp + mulxq (x), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + movq %r14, 0x30(%rsp) + mulxq 0x8(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x38(x), %rax, %r14 + adcxq %rax, %r13 + adoxq %rbp, %r14 + adcq %rbp, %r14 + movq 0x38(y), %rdx + xorl %ebp, %ebp + mulxq (x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + movq %r15, 0x38(%rsp) + mulxq 0x8(x), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x10(x), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x18(x), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x20(x), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x28(x), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x30(x), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x38(x), %rax, %r15 + adcxq %rax, %r14 + adoxq %rbp, %r15 + adcq %rbp, %r15 + +// Accumulate x[8] * y[0..7], extending the window to %rbp,%r15,...,%r8 + + movq 64(x), %rdx + xorl %ebp, %ebp + mulpadd(%r9,%r8,(y)) + mulpadd(%r10,%r9,8(y)) + mulpadd(%r11,%r10,16(y)) + mulpadd(%r12,%r11,24(y)) + mulpadd(%r13,%r12,32(y)) + mulpadd(%r14,%r13,40(y)) + mulpadd(%r15,%r14,48(y)) + mulxq 56(y), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbp, %rbx + adcq %rbx, %rbp + +// Accumulate y[8] * x[0..8] within this extended window %rbp,%r15,...,%r8 + + movq 64(y), %rdx + xorl %eax, %eax + mulpadd(%r9,%r8,(x)) + mulpadd(%r10,%r9,8(x)) + mulpadd(%r11,%r10,16(x)) + mulpadd(%r12,%r11,24(x)) + mulpadd(%r13,%r12,32(x)) + mulpadd(%r14,%r13,40(x)) + mulpadd(%r15,%r14,48(x)) + mulxq 56(x), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %rbp + mulxq 64(x), %rax, %rbx + adcq %rax, %rbp + +// Rotate the upper portion right 9 bits since 2^512 == 2^-9 (mod p_521) +// Let rotated result %rbp,%r15,%r14,...,%r8 be h (high) and %rsp[0..7] be l (low) + + movq %r8, %rax + andq $0x1FF, %rax + shrdq $9, %r9, %r8 + shrdq $9, %r10, %r9 + shrdq $9, %r11, %r10 + shrdq $9, %r12, %r11 + shrdq $9, %r13, %r12 + shrdq $9, %r14, %r13 + shrdq $9, %r15, %r14 + shrdq $9, %rbp, %r15 + shrq $9, %rbp + addq %rax, %rbp + +// Force carry-in then add to get s = h + l + 1 +// but actually add all 1s in the top 53 bits to get simple carry out + + stc + adcq (%rsp), %r8 + adcq 8(%rsp), %r9 + adcq 16(%rsp), %r10 + adcq 24(%rsp), %r11 + adcq 32(%rsp), %r12 + adcq 40(%rsp), %r13 + adcq 48(%rsp), %r14 + adcq 56(%rsp), %r15 + adcq $~0x1FF, %rbp + +// Now CF is set <=> h + l + 1 >= 2^521 <=> h + l >= p_521, +// in which case the lower 521 bits are already right. Otherwise if +// CF is clear, we want to subtract 1. Hence subtract the complement +// of the carry flag then mask the top word, which scrubs the +// padding in either case. Write digits back as they are created. + + cmc + sbbq $0, %r8 + movq %r8, (z) + sbbq $0, %r9 + movq %r9, 8(z) + sbbq $0, %r10 + movq %r10, 16(z) + sbbq $0, %r11 + movq %r11, 24(z) + sbbq $0, %r12 + movq %r12, 32(z) + sbbq $0, %r13 + movq %r13, 40(z) + sbbq $0, %r14 + movq %r14, 48(z) + sbbq $0, %r15 + movq %r15, 56(z) + sbbq $0, %rbp + andq $0x1FF, %rbp + movq %rbp, 64(z) + +// Restore registers and return + + CFI_INC_RSP(64) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + CFI_POP(%rbp) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_mul_p521) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_mul_p521_alt.S b/cbits/s2n/x86_att/bignum_mul_p521_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_mul_p521_alt.S @@ -0,0 +1,321 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Multiply modulo p_521, z := (x * y) mod p_521, assuming x and y reduced +// Inputs x[9], y[9]; output z[9] +// +// extern void bignum_mul_p521_alt(uint64_t z[static 9], +// const uint64_t x[static 9], +// const uint64_t y[static 9]); +// +// Standard x86-64 ABI: RDI = z, RSI = x, RDX = y +// Microsoft x64 ABI: RCX = z, RDX = x, R8 = y +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_mul_p521_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_mul_p521_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_mul_p521_alt) + .text + +#define z %rdi +#define x %rsi + +// This is moved from %rdx to free it for muls + +#define y %rcx + +// Macro for the key "multiply and add to (c,h,l)" step + +#define combadd(c,h,l,numa,numb) \ + movq numa, %rax ; \ + mulq numb; \ + addq %rax, l ; \ + adcq %rdx, h ; \ + adcq $0, c + +// A minutely shorter form for when c = 0 initially + +#define combadz(c,h,l,numa,numb) \ + movq numa, %rax ; \ + mulq numb; \ + addq %rax, l ; \ + adcq %rdx, h ; \ + adcq c, c + +// A short form where we don't expect a top carry + +#define combads(h,l,numa,numb) \ + movq numa, %rax ; \ + mulq numb; \ + addq %rax, l ; \ + adcq %rdx, h + +S2N_BN_SYMBOL(bignum_mul_p521_alt): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx +#endif + +// Make more registers available and make temporary space on stack + + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(72) + +// Copy y into a safe register to start with + + movq %rdx, y + +// Start doing a conventional columnwise multiplication, +// temporarily storing the lower 9 digits to the stack. +// Start with result term 0 + + movq (x), %rax + mulq (y) + + movq %rax, (%rsp) + movq %rdx, %r9 + xorq %r10, %r10 + +// Result term 1 + + xorq %r11, %r11 + combads(%r10,%r9,(x),8(y)) + combadz(%r11,%r10,%r9,8(x),(y)) + movq %r9, 8(%rsp) + +// Result term 2 + + xorq %r12, %r12 + combadz(%r12,%r11,%r10,(x),16(y)) + combadd(%r12,%r11,%r10,8(x),8(y)) + combadd(%r12,%r11,%r10,16(x),(y)) + movq %r10, 16(%rsp) + +// Result term 3 + + xorq %r13, %r13 + combadz(%r13,%r12,%r11,(x),24(y)) + combadd(%r13,%r12,%r11,8(x),16(y)) + combadd(%r13,%r12,%r11,16(x),8(y)) + combadd(%r13,%r12,%r11,24(x),(y)) + movq %r11, 24(%rsp) + +// Result term 4 + + xorq %r14, %r14 + combadz(%r14,%r13,%r12,(x),32(y)) + combadd(%r14,%r13,%r12,8(x),24(y)) + combadd(%r14,%r13,%r12,16(x),16(y)) + combadd(%r14,%r13,%r12,24(x),8(y)) + combadd(%r14,%r13,%r12,32(x),(y)) + movq %r12, 32(%rsp) + +// Result term 5 + + xorq %r15, %r15 + combadz(%r15,%r14,%r13,(x),40(y)) + combadd(%r15,%r14,%r13,8(x),32(y)) + combadd(%r15,%r14,%r13,16(x),24(y)) + combadd(%r15,%r14,%r13,24(x),16(y)) + combadd(%r15,%r14,%r13,32(x),8(y)) + combadd(%r15,%r14,%r13,40(x),(y)) + movq %r13, 40(%rsp) + +// Result term 6 + + xorq %r8, %r8 + combadz(%r8,%r15,%r14,(x),48(y)) + combadd(%r8,%r15,%r14,8(x),40(y)) + combadd(%r8,%r15,%r14,16(x),32(y)) + combadd(%r8,%r15,%r14,24(x),24(y)) + combadd(%r8,%r15,%r14,32(x),16(y)) + combadd(%r8,%r15,%r14,40(x),8(y)) + combadd(%r8,%r15,%r14,48(x),(y)) + movq %r14, 48(%rsp) + +// Result term 7 + + xorq %r9, %r9 + combadz(%r9,%r8,%r15,(x),56(y)) + combadd(%r9,%r8,%r15,8(x),48(y)) + combadd(%r9,%r8,%r15,16(x),40(y)) + combadd(%r9,%r8,%r15,24(x),32(y)) + combadd(%r9,%r8,%r15,32(x),24(y)) + combadd(%r9,%r8,%r15,40(x),16(y)) + combadd(%r9,%r8,%r15,48(x),8(y)) + combadd(%r9,%r8,%r15,56(x),(y)) + movq %r15, 56(%rsp) + +// Result term 8 + + xorq %r10, %r10 + combadz(%r10,%r9,%r8,(x),64(y)) + combadd(%r10,%r9,%r8,8(x),56(y)) + combadd(%r10,%r9,%r8,16(x),48(y)) + combadd(%r10,%r9,%r8,24(x),40(y)) + combadd(%r10,%r9,%r8,32(x),32(y)) + combadd(%r10,%r9,%r8,40(x),24(y)) + combadd(%r10,%r9,%r8,48(x),16(y)) + combadd(%r10,%r9,%r8,56(x),8(y)) + combadd(%r10,%r9,%r8,64(x),(y)) + movq %r8, 64(%rsp) + +// At this point we suspend writing back results and collect them +// in a register window. Next is result term 9 + + xorq %r11, %r11 + combadz(%r11,%r10,%r9,8(x),64(y)) + combadd(%r11,%r10,%r9,16(x),56(y)) + combadd(%r11,%r10,%r9,24(x),48(y)) + combadd(%r11,%r10,%r9,32(x),40(y)) + combadd(%r11,%r10,%r9,40(x),32(y)) + combadd(%r11,%r10,%r9,48(x),24(y)) + combadd(%r11,%r10,%r9,56(x),16(y)) + combadd(%r11,%r10,%r9,64(x),8(y)) + +// Result term 10 + + xorq %r12, %r12 + combadz(%r12,%r11,%r10,16(x),64(y)) + combadd(%r12,%r11,%r10,24(x),56(y)) + combadd(%r12,%r11,%r10,32(x),48(y)) + combadd(%r12,%r11,%r10,40(x),40(y)) + combadd(%r12,%r11,%r10,48(x),32(y)) + combadd(%r12,%r11,%r10,56(x),24(y)) + combadd(%r12,%r11,%r10,64(x),16(y)) + +// Result term 11 + + xorq %r13, %r13 + combadz(%r13,%r12,%r11,24(x),64(y)) + combadd(%r13,%r12,%r11,32(x),56(y)) + combadd(%r13,%r12,%r11,40(x),48(y)) + combadd(%r13,%r12,%r11,48(x),40(y)) + combadd(%r13,%r12,%r11,56(x),32(y)) + combadd(%r13,%r12,%r11,64(x),24(y)) + +// Result term 12 + + xorq %r14, %r14 + combadz(%r14,%r13,%r12,32(x),64(y)) + combadd(%r14,%r13,%r12,40(x),56(y)) + combadd(%r14,%r13,%r12,48(x),48(y)) + combadd(%r14,%r13,%r12,56(x),40(y)) + combadd(%r14,%r13,%r12,64(x),32(y)) + +// Result term 13 + + xorq %r15, %r15 + combadz(%r15,%r14,%r13,40(x),64(y)) + combadd(%r15,%r14,%r13,48(x),56(y)) + combadd(%r15,%r14,%r13,56(x),48(y)) + combadd(%r15,%r14,%r13,64(x),40(y)) + +// Result term 14 + + xorq %r8, %r8 + combadz(%r8,%r15,%r14,48(x),64(y)) + combadd(%r8,%r15,%r14,56(x),56(y)) + combadd(%r8,%r15,%r14,64(x),48(y)) + +// Result term 15 + + combads(%r8,%r15,56(x),64(y)) + combads(%r8,%r15,64(x),56(y)) + +// Result term 16 + + movq 64(x), %rax + imulq 64(y), %rax + addq %r8, %rax + +// Now the upper portion is [%rax;%r15;%r14;%r13;%r12;%r11;%r10;%r9;[%rsp+64]]. +// Rotate the upper portion right 9 bits since 2^512 == 2^-9 (mod p_521) +// Let rotated result %rdx,%r15,%r14,...,%r8 be h (high) and %rsp[0..7] be l (low) + + movq 64(%rsp), %r8 + movq %r8, %rdx + andq $0x1FF, %rdx + shrdq $9, %r9, %r8 + shrdq $9, %r10, %r9 + shrdq $9, %r11, %r10 + shrdq $9, %r12, %r11 + shrdq $9, %r13, %r12 + shrdq $9, %r14, %r13 + shrdq $9, %r15, %r14 + shrdq $9, %rax, %r15 + shrq $9, %rax + addq %rax, %rdx + +// Force carry-in then add to get s = h + l + 1 +// but actually add all 1s in the top 53 bits to get simple carry out + + stc + adcq (%rsp), %r8 + adcq 8(%rsp), %r9 + adcq 16(%rsp), %r10 + adcq 24(%rsp), %r11 + adcq 32(%rsp), %r12 + adcq 40(%rsp), %r13 + adcq 48(%rsp), %r14 + adcq 56(%rsp), %r15 + adcq $~0x1FF, %rdx + +// Now CF is set <=> h + l + 1 >= 2^521 <=> h + l >= p_521, +// in which case the lower 521 bits are already right. Otherwise if +// CF is clear, we want to subtract 1. Hence subtract the complement +// of the carry flag then mask the top word, which scrubs the +// padding in either case. Write digits back as they are created. + + cmc + sbbq $0, %r8 + movq %r8, (z) + sbbq $0, %r9 + movq %r9, 8(z) + sbbq $0, %r10 + movq %r10, 16(z) + sbbq $0, %r11 + movq %r11, 24(z) + sbbq $0, %r12 + movq %r12, 32(z) + sbbq $0, %r13 + movq %r13, 40(z) + sbbq $0, %r14 + movq %r14, 48(z) + sbbq $0, %r15 + movq %r15, 56(z) + sbbq $0, %rdx + andq $0x1FF, %rdx + movq %rdx, 64(z) + +// Restore registers and return + + CFI_INC_RSP(72) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_mul_p521_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_neg_p256.S b/cbits/s2n/x86_att/bignum_neg_p256.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_neg_p256.S @@ -0,0 +1,97 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Negate modulo p_256, z := (-x) mod p_256, assuming x reduced +// Input x[4]; output z[4] +// +// extern void bignum_neg_p256(uint64_t z[static 4], const uint64_t x[static 4]); +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_neg_p256) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_neg_p256) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_neg_p256) + .text + +#define z %rdi +#define x %rsi + +#define q %rdx + +#define d0 %rax +#define d1 %rcx +#define d2 %r8 +#define d3 %r9 + +#define n1 %r10 +#define n3 %r11 + +#define d0short %eax +#define n1short %r10d + +S2N_BN_SYMBOL(bignum_neg_p256): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Load the input digits as [d3;d2;d1;d0] and also set a bitmask q +// for the input being nonzero, so that we avoid doing -0 = p_256 +// and hence maintain strict modular reduction + + movq (x), d0 + movq 8(x), d1 + movq d0, n1 + orq d1, n1 + movq 16(x), d2 + movq 24(x), d3 + movq d2, n3 + orq d3, n3 + orq n1, n3 + negq n3 + sbbq q, q + +// Load the non-trivial words of p_256 = [n3;0;n1;-1] and mask them with q + + movl $0x00000000ffffffff, n1short + movq $0xffffffff00000001, n3 + andq q, n1 + andq q, n3 + +// Do the subtraction, getting it as [n3;d0;n1;q] to avoid moves + + subq d0, q + movl $0, d0short + sbbq d1, n1 + sbbq d2, d0 + sbbq d3, n3 + +// Write back + + movq q, (z) + movq n1, 8(z) + movq d0, 16(z) + movq n3, 24(z) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_neg_p256) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_sqr.S b/cbits/s2n/x86_att/bignum_sqr.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_sqr.S @@ -0,0 +1,190 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Square z := x^2 +// Input x[n]; output z[k] +// +// extern void bignum_sqr(uint64_t k, uint64_t *z, uint64_t n, const uint64_t *x); +// +// Does the "z := x^2" operation where x is n digits and result z is k. +// Truncates the result in general unless k >= 2 * n +// +// Standard x86-64 ABI: RDI = k, RSI = z, RDX = n, RCX = x +// Microsoft x64 ABI: RCX = k, RDX = z, R8 = n, R9 = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_sqr) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_sqr) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_sqr) + .text + .balign 32 + +// First three are where arguments come in, but n is moved. + +#define p %rdi +#define z %rsi +#define x %rcx +#define n %r8 + +// These are always local scratch since multiplier result is in these + +#define a %rax +#define d %rdx + +// Other variables + +#define i %rbx +#define ll %rbp +#define hh %r9 +#define k %r10 +#define y %r11 +#define htop %r12 +#define l %r13 +#define h %r14 +#define c %r15 + +// Short versions + +#define llshort %ebp + +S2N_BN_SYMBOL(bignum_sqr): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + movq %r9, %rcx +#endif + +// We use too many registers, and also we need %rax:%rdx for multiplications + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + movq %rdx, n + +// If p = 0 the result is trivial and nothing needs doing + + testq p, p + jz Lbignum_sqr_end + +// initialize (hh,ll) = 0 + + xorl llshort, llshort + xorq hh, hh + +// Iterate outer loop from k = 0 ... k = p - 1 producing result digits + + xorq k, k + +Lbignum_sqr_outerloop: + +// First let bot = MAX 0 (k + 1 - n) and top = MIN (k + 1) n +// We want to accumulate all x[i] * x[k - i] for bot <= i < top +// For the optimization of squaring we avoid duplication and do +// 2 * x[i] * x[k - i] for i < htop, where htop = MIN ((k+1)/2) n +// Initialize i = bot; in fact just compute bot as i directly. + + xorq c, c + leaq 1(k), i + movq i, htop + shrq $1, htop + subq n, i + cmovcq c, i + cmpq n, htop + cmovncq n, htop + +// Initialize the three-part local sum (c,h,l); c was already done above + + xorq l, l + xorq h, h + +// If htop <= bot then main doubled part of the sum is empty + + cmpq htop, i + jnc Lbignum_sqr_nosumming + +// Use a moving pointer for [y] = x[k-i] for the cofactor + + movq k, a + subq i, a + leaq (x,a,8), y + +// Do the main part of the sum x[i] * x[k - i] for 2 * i < k + +Lbignum_sqr_innerloop: + movq (x,i,8), a + mulq (y) + addq a, l + adcq d, h + adcq $0, c + subq $8, y + incq i + cmpq htop, i + jc Lbignum_sqr_innerloop + +// Now double it + + addq l, l + adcq h, h + adcq c, c + +// If k is even (which means 2 * i = k) and i < n add the extra x[i]^2 term + +Lbignum_sqr_nosumming: + testq $1, k + jnz Lbignum_sqr_innerend + cmpq n, i + jnc Lbignum_sqr_innerend + + movq (x,i,8), a + mulq a + addq a, l + adcq d, h + adcq $0, c + +// Now add the local sum into the global sum, store and shift + +Lbignum_sqr_innerend: + addq ll, l + movq l, (z,k,8) + adcq hh, h + movq h, ll + adcq $0, c + movq c, hh + + incq k + cmpq p, k + jc Lbignum_sqr_outerloop + +// Restore registers and return + +Lbignum_sqr_end: + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_sqr) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_sqr_8_16.S b/cbits/s2n/x86_att/bignum_sqr_8_16.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_sqr_8_16.S @@ -0,0 +1,302 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Square, z := x^2 +// Input x[8]; output z[16] +// +// extern void bignum_sqr_8_16(uint64_t z[static 16], const uint64_t x[static 8]); +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_sqr_8_16) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_sqr_8_16) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_sqr_8_16) + .text + +// These are actually right + +#define z %rdi +#define x %rsi + +// A zero register + +#define zero %rbp +#define zeroe %ebp + +// mulpadd i, j adds rdx * x[i] into the window at the i+j point + +.macro mulpadd arg1,arg2 + mulxq 8*\arg1(x), %rax, %rcx +.if ((\arg1 + \arg2) % 8 == 0) + adcxq %rax, %r8 + adoxq %rcx, %r9 +.elseif ((\arg1 + \arg2) % 8 == 1) + adcxq %rax, %r9 + adoxq %rcx, %r10 +.elseif ((\arg1 + \arg2) % 8 == 2) + adcxq %rax, %r10 + adoxq %rcx, %r11 +.elseif ((\arg1 + \arg2) % 8 == 3) + adcxq %rax, %r11 + adoxq %rcx, %r12 +.elseif ((\arg1 + \arg2) % 8 == 4) + adcxq %rax, %r12 + adoxq %rcx, %r13 +.elseif ((\arg1 + \arg2) % 8 == 5) + adcxq %rax, %r13 + adoxq %rcx, %r14 +.elseif ((\arg1 + \arg2) % 8 == 6) + adcxq %rax, %r14 + adoxq %rcx, %r15 +.elseif ((\arg1 + \arg2) % 8 == 7) + adcxq %rax, %r15 + adoxq %rcx, %r8 +.endif + +.endm + +// mulpade i, j adds rdx * x[i] into the window at i+j +// but re-creates the top word assuming nothing to add there + +.macro mulpade arg1,arg2 +.if ((\arg1 + \arg2) % 8 == 0) + mulxq 8*\arg1(x), %rax, %r9 + adcxq %rax, %r8 + adoxq zero, %r9 +.elseif ((\arg1 + \arg2) % 8 == 1) + mulxq 8*\arg1(x), %rax, %r10 + adcxq %rax, %r9 + adoxq zero, %r10 +.elseif ((\arg1 + \arg2) % 8 == 2) + mulxq 8*\arg1(x), %rax, %r11 + adcxq %rax, %r10 + adoxq zero, %r11 +.elseif ((\arg1 + \arg2) % 8 == 3) + mulxq 8*\arg1(x), %rax, %r12 + adcxq %rax, %r11 + adoxq zero, %r12 +.elseif ((\arg1 + \arg2) % 8 == 4) + mulxq 8*\arg1(x), %rax, %r13 + adcxq %rax, %r12 + adoxq zero, %r13 +.elseif ((\arg1 + \arg2) % 8 == 5) + mulxq 8*\arg1(x), %rax, %r14 + adcxq %rax, %r13 + adoxq zero, %r14 +.elseif ((\arg1 + \arg2) % 8 == 6) + mulxq 8*\arg1(x), %rax, %r15 + adcxq %rax, %r14 + adoxq zero, %r15 +.elseif ((\arg1 + \arg2) % 8 == 7) + mulxq 8*\arg1(x), %rax, %r8 + adcxq %rax, %r15 + adoxq zero, %r8 +.endif + +.endm + +.macro diagonals + + xorl zeroe, zeroe + +// Set initial window [%r8..%r10] + 2 wb = 10 + 20 + 30 + 40 + 50 + 60 + 70 + + movq (x), %rdx + mulxq 8(x), %r9, %rax + movq %r9, 8(z) + mulxq 16(x), %r10, %rcx + adcxq %rax, %r10 + movq %r10, 16(z) + mulxq 24(x), %r11, %rax + adcxq %rcx, %r11 + mulxq 32(x), %r12, %rcx + adcxq %rax, %r12 + mulxq 40(x), %r13, %rax + adcxq %rcx, %r13 + mulxq 48(x), %r14, %rcx + adcxq %rax, %r14 + mulxq 56(x), %r15, %r8 + adcxq %rcx, %r15 + adcxq zero, %r8 + +// Add in the next diagonal = 21 + 31 + 41 + 51 + 61 + 71 + 54 + + xorl zeroe, zeroe + movq 8(x), %rdx + mulpadd 2, 1 + movq %r11, 24(z) + mulpadd 3, 1 + movq %r12, 32(z) + mulpadd 4, 1 + mulpadd 5, 1 + mulpadd 6, 1 + mulpade 7, 1 + movq 32(x), %rdx + mulpade 5, 4 + adcxq zero, %r10 + +// And the next one = 32 + 42 + 52 + 62 + 72 + 64 + 65 + + xorl zeroe, zeroe + movq 16(x), %rdx + mulpadd 3, 2 + movq %r13, 40(z) + mulpadd 4, 2 + movq %r14, 48(z) + mulpadd 5, 2 + mulpadd 6, 2 + mulpadd 7, 2 + movq 48(x), %rdx + mulpade 4, 6 + mulpade 5, 6 + adcxq zero, %r12 + +// And the final one = 43 + 53 + 63 + 73 + 74 + 75 + 76 + + xorl zeroe, zeroe + movq 24(x), %rdx + mulpadd 4, 3 + movq %r15, 56(z) + mulpadd 5, 3 + movq %r8, 64(z) + mulpadd 6, 3 + mulpadd 7, 3 + movq 56(x), %rdx + mulpadd 4, 7 + mulpade 5, 7 + mulpade 6, 7 + adcxq zero, %r14 + +// Double and add things; use z[1]..z[8] and thereafter the registers +// %r9..%r15 which haven't been written back yet + + xorl zeroe, zeroe + movq (x), %rdx + mulxq %rdx, %rax, %rcx + movq %rax, (z) + movq 8(z), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 8(z) + + movq 16(z), %rax + movq 8(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %rax, %rax + adoxq %rdx, %rax + movq %rax, 16(z) + movq 24(z), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 24(z) + + movq 32(z), %rax + movq 16(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %rax, %rax + adoxq %rdx, %rax + movq %rax, 32(z) + movq 40(z), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 40(z) + + movq 48(z), %rax + movq 24(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %rax, %rax + adoxq %rdx, %rax + movq %rax, 48(z) + movq 56(z), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 56(z) + + movq 64(z), %rax + movq 32(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %rax, %rax + adoxq %rdx, %rax + movq %rax, 64(z) + adcxq %r9, %r9 + adoxq %rcx, %r9 + movq %r9, 72(z) + + movq 40(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %r10, %r10 + adoxq %rdx, %r10 + movq %r10, 80(z) + adcxq %r11, %r11 + adoxq %rcx, %r11 + movq %r11, 88(z) + + movq 48(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %r12, %r12 + adoxq %rdx, %r12 + movq %r12, 96(z) + adcxq %r13, %r13 + adoxq %rcx, %r13 + movq %r13, 104(z) + + movq 56(x), %rdx + mulxq %rdx, %rdx, %r15 + adcxq %r14, %r14 + adoxq %rdx, %r14 + movq %r14, 112(z) + adcxq zero, %r15 + adoxq zero, %r15 + movq %r15, 120(z) + +.endm + + +S2N_BN_SYMBOL(bignum_sqr_8_16): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Save more registers to play with + + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Do the multiplication + + diagonals + +// Real epilog + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_sqr_8_16) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_sqr_p521.S b/cbits/s2n/x86_att/bignum_sqr_p521.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_sqr_p521.S @@ -0,0 +1,302 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Square modulo p_521, z := (x^2) mod p_521, assuming x reduced +// Input x[9]; output z[9] +// +// extern void bignum_sqr_p521(uint64_t z[static 9], const uint64_t x[static 9]); +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_sqr_p521) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_sqr_p521) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_sqr_p521) + .text + +#define z %rdi +#define x %rsi + +// A zero register + +#define zero %rbp +#define zeroe %ebp + +// mulpadd(high,low,i) adds %rdx * x[i] to a register-pair (high,low) +// maintaining consistent double-carrying with adcx and adox, +// using %rax and %rcx as temporaries. + +#define mulpadd(high,low,I) \ + mulxq I(x), %rax, %rcx ; \ + adcxq %rax, low ; \ + adoxq %rcx, high + +// mulpade(high,low,i) adds %rdx * x[i] to a register-pair (high,low) +// maintaining consistent double-carrying with adcx and adox, +// using %rax as a temporary, assuming high created from scratch +// and that zero has value zero. + +#define mulpade(high,low,I) \ + mulxq I(x), %rax, high ; \ + adcxq %rax, low ; \ + adoxq zero, high + +S2N_BN_SYMBOL(bignum_sqr_p521): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Save more registers to play with and make temporary space on stack + + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(64) + +// Do a basic 8x8 squaring stashing in %rsp[0..7] but keeping the +// top half in the usual rotating register window %r15,...,%r8. Except +// for the lack of full writeback this is the same as bignum_sqr_8_16. + + xorl zeroe, zeroe + + movq (x), %rdx + mulxq 8(x), %r9, %rax + movq %r9, 8(%rsp) + mulxq 16(x), %r10, %rcx + adcxq %rax, %r10 + movq %r10, 16(%rsp) + mulxq 24(x), %r11, %rax + adcxq %rcx, %r11 + mulxq 32(x), %r12, %rcx + adcxq %rax, %r12 + mulxq 40(x), %r13, %rax + adcxq %rcx, %r13 + mulxq 48(x), %r14, %rcx + adcxq %rax, %r14 + mulxq 56(x), %r15, %r8 + adcxq %rcx, %r15 + adcxq zero, %r8 + + xorl zeroe, zeroe + movq 8(x), %rdx + mulpadd(%r12,%r11,16) + movq %r11, 24(%rsp) + mulpadd(%r13,%r12,24) + movq %r12, 32(%rsp) + mulpadd(%r14,%r13,32) + mulpadd(%r15,%r14,40) + mulpadd(%r8,%r15,48) + mulpade(%r9,%r8,56) + movq 32(x), %rdx + mulpade(%r10,%r9,40) + adcxq zero, %r10 + + xorl zeroe, zeroe + movq 16(x), %rdx + mulpadd(%r14,%r13,24) + movq %r13, 40(%rsp) + mulpadd(%r15,%r14,32) + movq %r14, 48(%rsp) + mulpadd(%r8,%r15,40) + mulpadd(%r9,%r8,48) + mulpadd(%r10,%r9,56) + movq 48(x), %rdx + mulpade(%r11,%r10,32) + mulpade(%r12,%r11,40) + adcxq zero, %r12 + + xorl zeroe, zeroe + movq 24(x), %rdx + mulpadd(%r8,%r15,32) + movq %r15, 56(%rsp) + mulpadd(%r9,%r8,40) + mulpadd(%r10,%r9,48) + mulpadd(%r11,%r10,56) + movq 56(x), %rdx + mulpadd(%r12,%r11,32) + mulpade(%r13,%r12,40) + mulpade(%r14,%r13,48) + adcxq zero, %r14 + + xorl zeroe, zeroe + movq (x), %rdx + mulxq %rdx, %rax, %rcx + movq %rax, (%rsp) + movq 8(%rsp), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 8(%rsp) + + movq 16(%rsp), %rax + movq 8(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %rax, %rax + adoxq %rdx, %rax + movq %rax, 16(%rsp) + movq 24(%rsp), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 24(%rsp) + + movq 32(%rsp), %rax + movq 16(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %rax, %rax + adoxq %rdx, %rax + movq %rax, 32(%rsp) + movq 40(%rsp), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 40(%rsp) + + movq 48(%rsp), %rax + movq 24(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %rax, %rax + adoxq %rdx, %rax + movq %rax, 48(%rsp) + movq 56(%rsp), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 56(%rsp) + + movq 32(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %r8, %r8 + adoxq %rdx, %r8 + adcxq %r9, %r9 + adoxq %rcx, %r9 + + movq 40(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %r10, %r10 + adoxq %rdx, %r10 + adcxq %r11, %r11 + adoxq %rcx, %r11 + + movq 48(x), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %r12, %r12 + adoxq %rdx, %r12 + adcxq %r13, %r13 + adoxq %rcx, %r13 + + movq 56(x), %rdx + mulxq %rdx, %rdx, %r15 + adcxq %r14, %r14 + adoxq %rdx, %r14 + adcxq zero, %r15 + adoxq zero, %r15 + +// Augment the high part with the contribution from the top little word C. +// If we write the input as 2^512 * C + x then we are otherwise just doing +// x^2, so we need to add to the high part 2^512 * C^2 + (2 * C) * x. +// The initial doubling add of C also clears the CF and OF flags as desired. +// We extend the window now to the 9-element %rbp,%r15,%r14,...,%r8. + + movq 64(x), %rdx + movq %rdx, %rbp + imulq %rbp, %rbp + addq %rdx, %rdx + mulpadd(%r9,%r8,0) + mulpadd(%r10,%r9,8) + mulpadd(%r11,%r10,16) + mulpadd(%r12,%r11,24) + mulpadd(%r13,%r12,32) + mulpadd(%r14,%r13,40) + mulpadd(%r15,%r14,48) + mulxq 56(x), %rax, %rcx + adcxq %rax, %r15 + adoxq %rcx, %rbp + adcq $0, %rbp + +// Rotate the upper portion right 9 bits since 2^512 == 2^-9 (mod p_521) +// Let rotated result %rbp,%r15,%r14,...,%r8 be h (high) and %rsp[0..7] be l (low) + + movq %r8, %rax + andq $0x1FF, %rax + shrdq $9, %r9, %r8 + shrdq $9, %r10, %r9 + shrdq $9, %r11, %r10 + shrdq $9, %r12, %r11 + shrdq $9, %r13, %r12 + shrdq $9, %r14, %r13 + shrdq $9, %r15, %r14 + shrdq $9, %rbp, %r15 + shrq $9, %rbp + addq %rax, %rbp + +// Force carry-in then add to get s = h + l + 1 +// but actually add all 1s in the top 53 bits to get simple carry out + + stc + adcq (%rsp), %r8 + adcq 8(%rsp), %r9 + adcq 16(%rsp), %r10 + adcq 24(%rsp), %r11 + adcq 32(%rsp), %r12 + adcq 40(%rsp), %r13 + adcq 48(%rsp), %r14 + adcq 56(%rsp), %r15 + adcq $~0x1FF, %rbp + +// Now CF is set <=> h + l + 1 >= 2^521 <=> h + l >= p_521, +// in which case the lower 521 bits are already right. Otherwise if +// CF is clear, we want to subtract 1. Hence subtract the complement +// of the carry flag then mask the top word, which scrubs the +// padding in either case. Write digits back as they are created. + + cmc + sbbq $0, %r8 + movq %r8, (z) + sbbq $0, %r9 + movq %r9, 8(z) + sbbq $0, %r10 + movq %r10, 16(z) + sbbq $0, %r11 + movq %r11, 24(z) + sbbq $0, %r12 + movq %r12, 32(z) + sbbq $0, %r13 + movq %r13, 40(z) + sbbq $0, %r14 + movq %r14, 48(z) + sbbq $0, %r15 + movq %r15, 56(z) + sbbq $0, %rbp + andq $0x1FF, %rbp + movq %rbp, 64(z) + +// Restore registers and return + + CFI_INC_RSP(64) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_sqr_p521) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_sqr_p521_alt.S b/cbits/s2n/x86_att/bignum_sqr_p521_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_sqr_p521_alt.S @@ -0,0 +1,315 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Square modulo p_521, z := (x^2) mod p_521, assuming x reduced +// Input x[9]; output z[9] +// +// extern void bignum_sqr_p521_alt(uint64_t z[static 9], +// const uint64_t x[static 9]); +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_sqr_p521_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_sqr_p521_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_sqr_p521_alt) + .text + +// Input arguments + +#define z %rdi +#define x %rsi + +// Macro for the key "multiply and add to (c,h,l)" step + +#define combadd(c,h,l,numa,numb) \ + movq numa, %rax ; \ + mulq numb; \ + addq %rax, l ; \ + adcq %rdx, h ; \ + adcq $0, c + +// Set up initial window (c,h,l) = numa * numb + +#define combaddz(c,h,l,numa,numb) \ + movq numa, %rax ; \ + mulq numb; \ + xorq c, c ; \ + movq %rax, l ; \ + movq %rdx, h + +// Doubling step (c,h,l) = 2 * (c,hh,ll) + (0,h,l) + +#define doubladd(c,h,l,hh,ll) \ + addq ll, ll ; \ + adcq hh, hh ; \ + adcq c, c ; \ + addq ll, l ; \ + adcq hh, h ; \ + adcq $0, c + +// Square term incorporation (c,h,l) += numba^2 + +#define combadd1(c,h,l,numa) \ + movq numa, %rax ; \ + mulq %rax; \ + addq %rax, l ; \ + adcq %rdx, h ; \ + adcq $0, c + +// A short form where we don't expect a top carry + +#define combads(h,l,numa) \ + movq numa, %rax ; \ + mulq %rax; \ + addq %rax, l ; \ + adcq %rdx, h + +// A version doubling directly before adding, for single non-square terms + +#define combadd2(c,h,l,numa,numb) \ + movq numa, %rax ; \ + mulq numb; \ + addq %rax, %rax ; \ + adcq %rdx, %rdx ; \ + adcq $0, c ; \ + addq %rax, l ; \ + adcq %rdx, h ; \ + adcq $0, c + +S2N_BN_SYMBOL(bignum_sqr_p521_alt): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Make more registers available and make temporary space on stack + + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(72) + +// Start doing a conventional columnwise squaring, +// temporarily storing the lower 9 digits on the stack. +// Start with result term 0 + + movq (x), %rax + mulq %rax + + movq %rax, (%rsp) + movq %rdx, %r9 + xorq %r10, %r10 + +// Result term 1 + + xorq %r11, %r11 + combadd2(%r11,%r10,%r9,(x),8(x)) + movq %r9, 8(%rsp) + +// Result term 2 + + xorq %r12, %r12 + combadd1(%r12,%r11,%r10,8(x)) + combadd2(%r12,%r11,%r10,(x),16(x)) + movq %r10, 16(%rsp) + +// Result term 3 + + combaddz(%r13,%rcx,%rbx,(x),24(x)) + combadd(%r13,%rcx,%rbx,8(x),16(x)) + doubladd(%r13,%r12,%r11,%rcx,%rbx) + movq %r11, 24(%rsp) + +// Result term 4 + + combaddz(%r14,%rcx,%rbx,(x),32(x)) + combadd(%r14,%rcx,%rbx,8(x),24(x)) + doubladd(%r14,%r13,%r12,%rcx,%rbx) + combadd1(%r14,%r13,%r12,16(x)) + movq %r12, 32(%rsp) + +// Result term 5 + + combaddz(%r15,%rcx,%rbx,(x),40(x)) + combadd(%r15,%rcx,%rbx,8(x),32(x)) + combadd(%r15,%rcx,%rbx,16(x),24(x)) + doubladd(%r15,%r14,%r13,%rcx,%rbx) + movq %r13, 40(%rsp) + +// Result term 6 + + combaddz(%r8,%rcx,%rbx,(x),48(x)) + combadd(%r8,%rcx,%rbx,8(x),40(x)) + combadd(%r8,%rcx,%rbx,16(x),32(x)) + doubladd(%r8,%r15,%r14,%rcx,%rbx) + combadd1(%r8,%r15,%r14,24(x)) + movq %r14, 48(%rsp) + +// Result term 7 + + combaddz(%r9,%rcx,%rbx,(x),56(x)) + combadd(%r9,%rcx,%rbx,8(x),48(x)) + combadd(%r9,%rcx,%rbx,16(x),40(x)) + combadd(%r9,%rcx,%rbx,24(x),32(x)) + doubladd(%r9,%r8,%r15,%rcx,%rbx) + movq %r15, 56(%rsp) + +// Result term 8 + + combaddz(%r10,%rcx,%rbx,(x),64(x)) + combadd(%r10,%rcx,%rbx,8(x),56(x)) + combadd(%r10,%rcx,%rbx,16(x),48(x)) + combadd(%r10,%rcx,%rbx,24(x),40(x)) + doubladd(%r10,%r9,%r8,%rcx,%rbx) + combadd1(%r10,%r9,%r8,32(x)) + movq %r8, 64(%rsp) + +// We now stop writing back and keep remaining results in a register window. +// Continue with result term 9 + + combaddz(%r11,%rcx,%rbx,8(x),64(x)) + combadd(%r11,%rcx,%rbx,16(x),56(x)) + combadd(%r11,%rcx,%rbx,24(x),48(x)) + combadd(%r11,%rcx,%rbx,32(x),40(x)) + doubladd(%r11,%r10,%r9,%rcx,%rbx) + +// Result term 10 + + combaddz(%r12,%rcx,%rbx,16(x),64(x)) + combadd(%r12,%rcx,%rbx,24(x),56(x)) + combadd(%r12,%rcx,%rbx,32(x),48(x)) + doubladd(%r12,%r11,%r10,%rcx,%rbx) + combadd1(%r12,%r11,%r10,40(x)) + +// Result term 11 + + combaddz(%r13,%rcx,%rbx,24(x),64(x)) + combadd(%r13,%rcx,%rbx,32(x),56(x)) + combadd(%r13,%rcx,%rbx,40(x),48(x)) + doubladd(%r13,%r12,%r11,%rcx,%rbx) + +// Result term 12 + + combaddz(%r14,%rcx,%rbx,32(x),64(x)) + combadd(%r14,%rcx,%rbx,40(x),56(x)) + doubladd(%r14,%r13,%r12,%rcx,%rbx) + combadd1(%r14,%r13,%r12,48(x)) + +// Result term 13 + + combaddz(%r15,%rcx,%rbx,40(x),64(x)) + combadd(%r15,%rcx,%rbx,48(x),56(x)) + doubladd(%r15,%r14,%r13,%rcx,%rbx); + +// Result term 14 + + xorq %r8, %r8 + combadd1(%r8,%r15,%r14,56(x)) + combadd2(%r8,%r15,%r14,48(x),64(x)) + +// Result term 15 + + movq 56(x), %rax + mulq 64(x) + addq %rax, %rax + adcq %rdx, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + +// Result term 16 + + movq 64(x), %rax + imulq %rax, %rax + addq %r8, %rax + +// Now the upper portion is [%rax;%r15;%r14;%r13;%r12;%r11;%r10;%r9;[%rsp+64]]. +// Rotate the upper portion right 9 bits since 2^512 == 2^-9 (mod p_521) +// Let rotated result %rdx,%r15,%r14,...,%r8 be h (high) and %rsp[0..7] be l (low) + + movq 64(%rsp), %r8 + movq %r8, %rdx + andq $0x1FF, %rdx + shrdq $9, %r9, %r8 + shrdq $9, %r10, %r9 + shrdq $9, %r11, %r10 + shrdq $9, %r12, %r11 + shrdq $9, %r13, %r12 + shrdq $9, %r14, %r13 + shrdq $9, %r15, %r14 + shrdq $9, %rax, %r15 + shrq $9, %rax + addq %rax, %rdx + +// Force carry-in then add to get s = h + l + 1 +// but actually add all 1s in the top 53 bits to get simple carry out + + stc + adcq (%rsp), %r8 + adcq 8(%rsp), %r9 + adcq 16(%rsp), %r10 + adcq 24(%rsp), %r11 + adcq 32(%rsp), %r12 + adcq 40(%rsp), %r13 + adcq 48(%rsp), %r14 + adcq 56(%rsp), %r15 + adcq $~0x1FF, %rdx + +// Now CF is set <=> h + l + 1 >= 2^521 <=> h + l >= p_521, +// in which case the lower 521 bits are already right. Otherwise if +// CF is clear, we want to subtract 1. Hence subtract the complement +// of the carry flag then mask the top word, which scrubs the +// padding in either case. Write digits back as they are created. + + cmc + sbbq $0, %r8 + movq %r8, (z) + sbbq $0, %r9 + movq %r9, 8(z) + sbbq $0, %r10 + movq %r10, 16(z) + sbbq $0, %r11 + movq %r11, 24(z) + sbbq $0, %r12 + movq %r12, 32(z) + sbbq $0, %r13 + movq %r13, 40(z) + sbbq $0, %r14 + movq %r14, 48(z) + sbbq $0, %r15 + movq %r15, 56(z) + sbbq $0, %rdx + andq $0x1FF, %rdx + movq %rdx, 64(z) + +// Restore registers and return + + CFI_INC_RSP(72) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_sqr_p521_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_tomont_p256.S b/cbits/s2n/x86_att/bignum_tomont_p256.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_tomont_p256.S @@ -0,0 +1,195 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Convert to Montgomery form z := (2^256 * x) mod p_256 +// Input x[4]; output z[4] +// +// extern void bignum_tomont_p256(uint64_t z[static 4], +// const uint64_t x[static 4]); +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_tomont_p256) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_tomont_p256) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_tomont_p256) + .text + +#define z %rdi +#define x %rsi + +// Some temp registers for the last correction stage + +#define d %rax +#define u %rdx +#define v %rcx + +#define dshort %eax +#define ushort %edx + +// Add %rdx * m into a register-pair (high,low) +// maintaining consistent double-carrying with adcx and adox, +// using %rax and %rbx as temporaries + +#define mulpadd(high,low,m) \ + mulxq m, %rax, %rcx ; \ + adcxq %rax, low ; \ + adoxq %rcx, high + +S2N_BN_SYMBOL(bignum_tomont_p256): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// We are essentially just doing a Montgomery multiplication of x and the +// precomputed constant y = 2^512 mod p, so the code is almost the same +// modulo a few registers and the change from loading y[i] to using constants. +// Because there is no y pointer to keep, we use one register less. + + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Do row 0 computation, which is a bit different: +// set up initial window [%r12,%r11,%r10,%r9,%r8] = y[0] * x +// Unlike later, we only need a single carry chain + + xorq %r13, %r13 + movl $0x0000000000000003, %edx + mulxq (x), %r8, %r9 + mulxq 8(x), %rcx, %r10 + adcxq %rcx, %r9 + mulxq 16(x), %rcx, %r11 + adcxq %rcx, %r10 + mulxq 24(x), %rcx, %r12 + adcxq %rcx, %r11 + adcxq %r13, %r12 + +// Add row 1 + + movq $0xfffffffbffffffff, %rdx + xorq %r14, %r14 + mulpadd(%r10,%r9,(x)) + mulpadd(%r11,%r10,8(x)) + mulpadd(%r12,%r11,16(x)) + mulpadd(%r13,%r12,24(x)) + adcq %r14, %r13 + +// Montgomery reduce windows 0 and 1 together + + xorq %r15, %r15 + movq $0x0000000100000000, %rdx + mulpadd(%r10,%r9,%r8) + mulpadd(%r11,%r10,%r9) + movq $0xffffffff00000001, %rdx + mulpadd(%r12,%r11,%r8) + mulpadd(%r13,%r12,%r9) + adcxq %r15, %r13 + adoxq %r15, %r14 + adcxq %r15, %r14 + +// Add row 2 + + movq $0xfffffffffffffffe, %rdx + xorq %r8, %r8 + mulpadd(%r11,%r10,(x)) + mulpadd(%r12,%r11,8(x)) + mulpadd(%r13,%r12,16(x)) + mulpadd(%r14,%r13,24(x)) + adcxq %r8, %r14 + adoxq %r8, %r15 + adcxq %r8, %r15 + +// Add row 3 + + movq $0x00000004fffffffd, %rdx + xorq %r9, %r9 + mulpadd(%r12,%r11,(x)) + mulpadd(%r13,%r12,8(x)) + mulpadd(%r14,%r13,16(x)) + mulpadd(%r15,%r14,24(x)) + adcxq %r9, %r15 + adoxq %r9, %r8 + adcxq %r9, %r8 + +// Montgomery reduce windows 2 and 3 together + + xorq %r9, %r9 + movq $0x0000000100000000, %rdx + mulpadd(%r12,%r11,%r10) + mulpadd(%r13,%r12,%r11) + movq $0xffffffff00000001, %rdx + mulpadd(%r14,%r13,%r10) + mulpadd(%r15,%r14,%r11) + adcxq %r9, %r15 + adoxq %r9, %r8 + adcxq %r9, %r8 + +// We now have a pre-reduced 5-word form [%r8; %r15;%r14;%r13;%r12] +// Load non-trivial digits of p_256 = [v; 0; u; -1] + + movl $0x00000000ffffffff, ushort + movq $0xffffffff00000001, v + +// Now do the subtraction (0,p_256-1) - (%r8,%r15,%r14,%r13,%r12) to get the carry + + movq $-2, d + subq %r12, d + movq u, d + sbbq %r13, d + movl $0, dshort + sbbq %r14, d + movq v, d + sbbq %r15, d + +// This last comparison in the chain will actually even set the mask +// for us, so we don't need to separately create it from the carry. +// This means p_256 - 1 < (c,d1,d0,d5,d4), i.e. we are so far >= p_256 + + movl $0, dshort + sbbq %r8, d + andq d, u + andq d, v + +// Do a masked subtraction of p_256 and write back + + subq d, %r12 + sbbq u, %r13 + sbbq $0, %r14 + sbbq v, %r15 + + movq %r12, (z) + movq %r13, 8(z) + movq %r14, 16(z) + movq %r15, 24(z) + +// Restore registers and return + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_tomont_p256) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_tomont_p256_alt.S b/cbits/s2n/x86_att/bignum_tomont_p256_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_tomont_p256_alt.S @@ -0,0 +1,203 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Convert to Montgomery form z := (2^256 * x) mod p_256 +// Input x[4]; output z[4] +// +// extern void bignum_tomont_p256_alt(uint64_t z[static 4], +// const uint64_t x[static 4]); +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_tomont_p256_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_tomont_p256_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_tomont_p256_alt) + .text + +#define z %rdi +#define x %rsi + +// Add %rcx * m into a register-pair (high,low) maintaining consistent +// carry-catching with carry (negated, as bitmask) and using %rax and %rdx +// as temporaries + +#define mulpadd(carry,high,low,m) \ + movq m, %rax ; \ + mulq %rcx; \ + subq carry, %rdx ; \ + addq %rax, low ; \ + adcq %rdx, high ; \ + sbbq carry, carry + +// Initial version assuming no carry-in + +#define mulpadi(carry,high,low,m) \ + movq m, %rax ; \ + mulq %rcx; \ + addq %rax, low ; \ + adcq %rdx, high ; \ + sbbq carry, carry + +// End version not catching the top carry-out + +#define mulpade(carry,high,low,m) \ + movq m, %rax ; \ + mulq %rcx; \ + subq carry, %rdx ; \ + addq %rax, low ; \ + adcq %rdx, high + +S2N_BN_SYMBOL(bignum_tomont_p256_alt): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Save more registers to play with + + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Do row 0 computation, which is a bit different: +// set up initial window [%r12,%r11,%r10,%r9,%r8] = y[0] * x +// Unlike later, we only need a single carry chain + + movl $0x0000000000000003, %ecx + movq (x), %rax + mulq %rcx + movq %rax, %r8 + movq %rdx, %r9 + + movq 8(x), %rax + mulq %rcx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + + movq 16(x), %rax + mulq %rcx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + + movq 24(x), %rax + mulq %rcx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + +// Add row 1 + + movq $0xfffffffbffffffff, %rcx + xorl %r13d, %r13d + mulpadi(%r14,%r10,%r9,(x)) + mulpadd(%r14,%r11,%r10,8(x)) + mulpadd(%r14,%r12,%r11,16(x)) + mulpade(%r14,%r13,%r12,24(x)) + +// Montgomery reduce windows 0 and 1 together + + xorl %r14d, %r14d + movq $0x0000000100000000, %rcx + mulpadi(%r15,%r10,%r9,%r8) + mulpadd(%r15,%r11,%r10,%r9) + notq %rcx + leaq 2(%rcx), %rcx + mulpadd(%r15,%r12,%r11,%r8) + mulpade(%r15,%r13,%r12,%r9) + adcq %r14, %r14 + +// Add row 2 + + movq $0xfffffffffffffffe, %rcx + xorl %r15d, %r15d + mulpadi(%r8,%r11,%r10,(x)) + mulpadd(%r8,%r12,%r11,8(x)) + mulpadd(%r8,%r13,%r12,16(x)) + mulpade(%r8,%r14,%r13,24(x)) + adcq %r15, %r15 + +// Add row 3 + + movq $0x00000004fffffffd, %rcx + xorl %r8d, %r8d + mulpadi(%r9,%r12,%r11,(x)) + mulpadd(%r9,%r13,%r12,8(x)) + mulpadd(%r9,%r14,%r13,16(x)) + mulpade(%r9,%r15,%r14,24(x)) + adcq %r8, %r8 + +// Montgomery reduce windows 2 and 3 together + + movq $0x0000000100000000, %rcx + mulpadi(%r9,%r12,%r11,%r10) + mulpadd(%r9,%r13,%r12,%r11) + notq %rcx + leaq 2(%rcx), %rcx + mulpadd(%r9,%r14,%r13,%r10) + mulpadd(%r9,%r15,%r14,%r11) + subq %r9, %r8 + +// We now have a pre-reduced 5-word form [%r8; %r15;%r14;%r13;%r12] +// Load [%rax;%r11;%r9;%rcx;%rdx] = 2^320 - p_256, re-using earlier numbers a bit +// Do [%rax;%r11;%r9;%rcx;%rdx] = [%r8;%r15;%r14;%r13;%r12] + (2^320 - p_256) + + xorl %edx, %edx + leaq -1(%rdx), %r9 + incq %rdx + addq %r12, %rdx + decq %rcx + adcq %r13, %rcx + movq %r9, %rax + adcq %r14, %r9 + movl $0x00000000fffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + +// Now carry is set if r + (2^320 - p_256) >= 2^320, i.e. r >= p_256 +// where r is the pre-reduced form. So conditionally select the +// output accordingly. + + cmovcq %rdx, %r12 + cmovcq %rcx, %r13 + cmovcq %r9, %r14 + cmovcq %r11, %r15 + +// Write back reduced value + + movq %r12, (z) + movq %r13, 8(z) + movq %r14, 16(z) + movq %r15, 24(z) + +// Restore registers and return + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_tomont_p256_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_tomont_p384.S b/cbits/s2n/x86_att/bignum_tomont_p384.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_tomont_p384.S @@ -0,0 +1,295 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Convert to Montgomery form z := (2^384 * x) mod p_384 +// Input x[6]; output z[6] +// +// extern void bignum_tomont_p384(uint64_t z[static 6], +// const uint64_t x[static 6]); +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_tomont_p384) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_tomont_p384) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_tomont_p384) + .text + +#define z %rdi +#define x %rsi + +// Fairly consistently used as a zero register + +#define zero %rbp + +// Some temp registers for the last correction stage + +#define d %rax +#define u %rdx +#define v %rcx +#define w %rsi + +#define vshort %ecx +#define wshort %esi + +// Add %rdx * m into a register-pair (high,low) +// maintaining consistent double-carrying with adcx and adox, +// using %rax and %rcx as temporaries + +#define mulpadd(high,low,m) \ + mulxq m, %rax, %rcx ; \ + adcxq %rax, low ; \ + adoxq %rcx, high + +// Core one-step Montgomery reduction macro. Takes input in +// [d7;d6;d5;d4;d3;d2;d1;d0] and returns result in [d7;d6;d5;d4;d3;d2;d1], +// adding to the existing contents, re-using d0 as a temporary internally +// +// We want to add (2^384 - 2^128 - 2^96 + 2^32 - 1) * w +// where w = [d0 + (d0<<32)] mod 2^64 +// +// montredc(d7,d6,d5,d4,d3,d2,d1,d0) +// +// This particular variant, with its mix of addition and subtraction +// at the top, is not intended to maintain a coherent carry or borrow out. +// It is assumed the final result would fit in [d7;d6;d5;d4;d3;d2;d1]. +// which is always the case here as the top word is even always in {0,1} + +#define montredc(d7,d6,d5,d4,d3,d2,d1,d0) \ +/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64 */ \ + movq d0, %rdx ; \ + shlq $32, %rdx ; \ + addq d0, %rdx ; \ +/* Construct [%rbp;%rcx;%rax;-] = (2^384 - p_384) * w */ \ +/* We know the lowest word will cancel so we can re-use d0 as a temp */ \ + xorl %ebp, %ebp ; \ + movq $0xffffffff00000001, %rax ; \ + mulxq %rax, %rcx, %rax ; \ + movl $0x00000000ffffffff, %ecx ; \ + mulxq %rcx, d0, %rcx ; \ + adcq d0, %rax ; \ + adcq %rdx, %rcx ; \ + adcl %ebp, %ebp ; \ +/* Now subtract that and add 2^384 * w */ \ + subq %rax, d1 ; \ + sbbq %rcx, d2 ; \ + sbbq %rbp, d3 ; \ + sbbq $0, d4 ; \ + sbbq $0, d5 ; \ + sbbq $0, %rdx ; \ + addq %rdx, d6 ; \ + adcq $0, d7 + +S2N_BN_SYMBOL(bignum_tomont_p384): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// We are essentially just doing a Montgomery multiplication of x and the +// precomputed constant y = 2^768 mod p, so the code is almost the same +// modulo a few registers and the change from loading y[i] to using constants, +// plus the easy digits y[4] = 1 and y[5] = 0 being treated specially. +// Because there is no y pointer to keep, we use one register less. + + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Do row 0 computation, which is a bit different: +// set up initial window [%r14,%r13,%r12,%r11,%r10,%r9,%r8] = y[0] * x +// Unlike later, we only need a single carry chain + + movq $0xfffffffe00000001, %rdx + mulxq (x), %r8, %r9 + mulxq 8(x), %rcx, %r10 + addq %rcx, %r9 + mulxq 16(x), %rcx, %r11 + adcq %rcx, %r10 + mulxq 24(x), %rcx, %r12 + adcq %rcx, %r11 + mulxq 32(x), %rcx, %r13 + adcq %rcx, %r12 + mulxq 40(x), %rcx, %r14 + adcq %rcx, %r13 + adcq $0, %r14 + +// Montgomery reduce the zeroth window + + xorq %r15, %r15 + montredc(%r15, %r14,%r13,%r12,%r11,%r10,%r9,%r8) + +// Add row 1 + + xorq zero, zero + movq $0x0000000200000000, %rdx + xorq %r8, %r8 + mulpadd(%r10,%r9,(x)) + mulpadd(%r11,%r10,8(x)) + mulpadd(%r12,%r11,16(x)) + mulpadd(%r13,%r12,24(x)) + mulpadd(%r14,%r13,32(x)) + mulpadd(%r15,%r14,40(x)) + adcxq zero, %r15 + adoxq zero, %r8 + adcxq zero, %r8 + +// Montgomery reduce window 1 + + montredc(%r8, %r15,%r14,%r13,%r12,%r11,%r10,%r9) + +// Add row 2 + + xorq zero, zero + movq $0xfffffffe00000000, %rdx + xorq %r9, %r9 + mulpadd(%r11,%r10,(x)) + mulpadd(%r12,%r11,8(x)) + mulpadd(%r13,%r12,16(x)) + mulpadd(%r14,%r13,24(x)) + mulpadd(%r15,%r14,32(x)) + mulpadd(%r8,%r15,40(x)) + adcxq zero, %r8 + adoxq zero, %r9 + adcxq zero, %r9 + +// Montgomery reduce window 2 + + montredc(%r9, %r8,%r15,%r14,%r13,%r12,%r11,%r10) + +// Add row 3 + + xorq zero, zero + movq $0x0000000200000000, %rdx + xorq %r10, %r10 + mulpadd(%r12,%r11,(x)) + mulpadd(%r13,%r12,8(x)) + mulpadd(%r14,%r13,16(x)) + mulpadd(%r15,%r14,24(x)) + mulpadd(%r8,%r15,32(x)) + mulpadd(%r9,%r8,40(x)) + adcxq zero, %r9 + adoxq zero, %r10 + adcxq zero, %r10 + +// Montgomery reduce window 3 + + montredc(%r10, %r9,%r8,%r15,%r14,%r13,%r12,%r11) + +// Add row 4. The multiplier y[4] = 1, so we just add x to the window +// while extending it with one more digit, initially this carry + + xorq %r11, %r11 + addq (x), %r12 + adcq 8(x), %r13 + adcq 16(x), %r14 + adcq 24(x), %r15 + adcq 32(x), %r8 + adcq 40(x), %r9 + adcq $0, %r10 + adcq $0, %r11 + +// Montgomery reduce window 4 + + montredc(%r11, %r10,%r9,%r8,%r15,%r14,%r13,%r12) + +// Add row 5, The multiplier y[5] = 0, so this is trivial: all we do is +// bring down another zero digit into the window. + + xorq %r12, %r12 + +// Montgomery reduce window 5 + + montredc(%r12, %r11,%r10,%r9,%r8,%r15,%r14,%r13) + +// We now have a pre-reduced 7-word form [%r12;%r11;%r10;%r9;%r8;%r15;%r14] + +// We know, writing B = 2^{6*64} that the full implicit result is +// B^2 c <= z + (B - 1) * p < B * p + (B - 1) * p < 2 * B * p, +// so the top half is certainly < 2 * p. If c = 1 already, we know +// subtracting p will give the reduced modulus. But now we do a +// comparison to catch cases where the residue is >= p. +// First set [0;0;0;w;v;u] = 2^384 - p_384 + + movq $0xffffffff00000001, u + movl $0x00000000ffffffff, vshort + movl $0x0000000000000001, wshort + +// Let dd = [%r11;%r10;%r9;%r8;%r15;%r14] be the topless 6-word intermediate result. +// Set CF if the addition dd + (2^384 - p_384) >= 2^384, hence iff dd >= p_384. + + movq %r14, d + addq u, d + movq %r15, d + adcq v, d + movq %r8, d + adcq w, d + movq %r9, d + adcq $0, d + movq %r10, d + adcq $0, d + movq %r11, d + adcq $0, d + +// Now just add this new carry into the existing %r12. It's easy to see they +// can't both be 1 by our range assumptions, so this gives us a {0,1} flag + + adcq $0, %r12 + +// Now convert it into a bitmask + + negq %r12 + +// Masked addition of 2^384 - p_384, hence subtraction of p_384 + + andq %r12, u + andq %r12, v + andq %r12, w + + addq u, %r14 + adcq v, %r15 + adcq w, %r8 + adcq $0, %r9 + adcq $0, %r10 + adcq $0, %r11 + +// Write back the result + + movq %r14, (z) + movq %r15, 8(z) + movq %r8, 16(z) + movq %r9, 24(z) + movq %r10, 32(z) + movq %r11, 40(z) + +// Restore registers and return + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_tomont_p384) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/bignum_tomont_p384_alt.S b/cbits/s2n/x86_att/bignum_tomont_p384_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/bignum_tomont_p384_alt.S @@ -0,0 +1,324 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Convert to Montgomery form z := (2^384 * x) mod p_384 +// Input x[6]; output z[6] +// +// extern void bignum_tomont_p384_alt(uint64_t z[static 6], +// const uint64_t x[static 6]); +// +// Standard x86-64 ABI: RDI = z, RSI = x +// Microsoft x64 ABI: RCX = z, RDX = x +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(bignum_tomont_p384_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(bignum_tomont_p384_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(bignum_tomont_p384_alt) + .text + +#define z %rdi +#define x %rsi + +// Some temp registers for the last correction stage + +#define d %rax +#define u %rdx +#define v %rcx +#define w %rsi + +#define vshort %ecx +#define wshort %esi + +// Add %rbx * m into a register-pair (high,low) maintaining consistent +// carry-catching with carry (negated, as bitmask) and using %rax and %rdx +// as temporaries + +#define mulpadd(carry,high,low,m) \ + movq m, %rax ; \ + mulq %rbx; \ + subq carry, %rdx ; \ + addq %rax, low ; \ + adcq %rdx, high ; \ + sbbq carry, carry + +// Initial version assuming no carry-in + +#define mulpadi(carry,high,low,m) \ + movq m, %rax ; \ + mulq %rbx; \ + addq %rax, low ; \ + adcq %rdx, high ; \ + sbbq carry, carry + +// End version not catching the top carry-out + +#define mulpade(carry,high,low,m) \ + movq m, %rax ; \ + mulq %rbx; \ + subq carry, %rdx ; \ + addq %rax, low ; \ + adcq %rdx, high + +// Core one-step Montgomery reduction macro. Takes input in +// [d7;d6;d5;d4;d3;d2;d1;d0] and returns result in [d7;d6;d5;d4;d3;d2;d1], +// adding to the existing contents, re-using d0 as a temporary internally +// +// We want to add (2^384 - 2^128 - 2^96 + 2^32 - 1) * w +// where w = [d0 + (d0<<32)] mod 2^64 +// +// montredc(d7,d6,d5,d4,d3,d2,d1,d0) +// +// This particular variant, with its mix of addition and subtraction +// at the top, is not intended to maintain a coherent carry or borrow out. +// It is assumed the final result would fit in [d7;d6;d5;d4;d3;d2;d1]. +// which is always the case here as the top word is even always in {0,1} + +#define montredc(d7,d6,d5,d4,d3,d2,d1,d0) \ +/* Our correction multiplier is w = [d0 + (d0<<32)] mod 2^64 */ \ + movq d0, %rbx ; \ + shlq $32, %rbx ; \ + addq d0, %rbx ; \ +/* Construct [%rcx;%rdx;%rax;-] = (2^384 - p_384) * w */ \ +/* We know the lowest word will cancel so we can re-use d0 as a temp */ \ + xorl %ecx, %ecx ; \ + movq $0xffffffff00000001, %rax ; \ + mulq %rbx; \ + movq %rdx, d0 ; \ + movq $0x00000000ffffffff, %rax ; \ + mulq %rbx; \ + addq d0, %rax ; \ + adcq %rbx, %rdx ; \ + adcl %ecx, %ecx ; \ +/* Now subtract that and add 2^384 * w */ \ + subq %rax, d1 ; \ + sbbq %rdx, d2 ; \ + sbbq %rcx, d3 ; \ + sbbq $0, d4 ; \ + sbbq $0, d5 ; \ + sbbq $0, %rbx ; \ + addq %rbx, d6 ; \ + adcq $0, d7 + +S2N_BN_SYMBOL(bignum_tomont_p384_alt): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// We are essentially just doing a Montgomery multiplication of x and the +// precomputed constant y = 2^768 mod p, so the code is almost the same +// modulo a few registers and the change from loading y[i] to using constants, +// plus the easy digits y[4] = 1 and y[5] = 0 being treated specially. +// Because there is no y pointer to keep, we use one register less. + + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + +// Do row 0 computation, which is a bit different: +// set up initial window [%r14,%r13,%r12,%r11,%r10,%r9,%r8] = y[0] * x +// Unlike later, we only need a single carry chain + + movq $0xfffffffe00000001, %rbx + movq (x), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + + movq 8(x), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + + movq 16(x), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + + movq 24(x), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + + movq 32(x), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + + movq 40(x), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + + xorl %r15d, %r15d + +// Montgomery reduce the zeroth window + + montredc(%r15, %r14,%r13,%r12,%r11,%r10,%r9,%r8) + +// Add row 1 + + movq $0x0000000200000000, %rbx + mulpadi(%r8,%r10,%r9,(x)) + mulpadd(%r8,%r11,%r10,8(x)) + mulpadd(%r8,%r12,%r11,16(x)) + mulpadd(%r8,%r13,%r12,24(x)) + mulpadd(%r8,%r14,%r13,32(x)) + mulpadd(%r8,%r15,%r14,40(x)) + negq %r8 + +// Montgomery reduce window 1 + + montredc(%r8, %r15,%r14,%r13,%r12,%r11,%r10,%r9) + +// Add row 2 + + movq $0xfffffffe00000000, %rbx + mulpadi(%r9,%r11,%r10,(x)) + mulpadd(%r9,%r12,%r11,8(x)) + mulpadd(%r9,%r13,%r12,16(x)) + mulpadd(%r9,%r14,%r13,24(x)) + mulpadd(%r9,%r15,%r14,32(x)) + mulpadd(%r9,%r8,%r15,40(x)) + negq %r9 + +// Montgomery reduce window 2 + + montredc(%r9, %r8,%r15,%r14,%r13,%r12,%r11,%r10) + +// Add row 3 + + movq $0x0000000200000000, %rbx + mulpadi(%r10,%r12,%r11,(x)) + mulpadd(%r10,%r13,%r12,8(x)) + mulpadd(%r10,%r14,%r13,16(x)) + mulpadd(%r10,%r15,%r14,24(x)) + mulpadd(%r10,%r8,%r15,32(x)) + mulpadd(%r10,%r9,%r8,40(x)) + negq %r10 + +// Montgomery reduce window 3 + + montredc(%r10, %r9,%r8,%r15,%r14,%r13,%r12,%r11) + +// Add row 4. The multiplier y[4] = 1, so we just add x to the window +// while extending it with one more digit, initially this carry + + xorq %r11, %r11 + addq (x), %r12 + adcq 8(x), %r13 + adcq 16(x), %r14 + adcq 24(x), %r15 + adcq 32(x), %r8 + adcq 40(x), %r9 + adcq %r11, %r10 + adcq %r11, %r11 + +// Montgomery reduce window 4 + + montredc(%r11, %r10,%r9,%r8,%r15,%r14,%r13,%r12) + +// Add row 5, The multiplier y[5] = 0, so this is trivial: all we do is +// bring down another zero digit into the window. + + xorq %r12, %r12 + +// Montgomery reduce window 5 + + montredc(%r12, %r11,%r10,%r9,%r8,%r15,%r14,%r13) + +// We now have a pre-reduced 7-word form [%r12;%r11;%r10;%r9;%r8;%r15;%r14] + +// We know, writing B = 2^{6*64} that the full implicit result is +// B^2 c <= z + (B - 1) * p < B * p + (B - 1) * p < 2 * B * p, +// so the top half is certainly < 2 * p. If c = 1 already, we know +// subtracting p will give the reduced modulus. But now we do a +// comparison to catch cases where the residue is >= p. +// First set [0;0;0;w;v;u] = 2^384 - p_384 + + movq $0xffffffff00000001, u + movl $0x00000000ffffffff, vshort + movl $0x0000000000000001, wshort + +// Let dd = [%r11;%r10;%r9;%r8;%r15;%r14] be the topless 6-word intermediate result. +// Set CF if the addition dd + (2^384 - p_384) >= 2^384, hence iff dd >= p_384. + + movq %r14, d + addq u, d + movq %r15, d + adcq v, d + movq %r8, d + adcq w, d + movq %r9, d + adcq $0, d + movq %r10, d + adcq $0, d + movq %r11, d + adcq $0, d + +// Now just add this new carry into the existing %r12. It's easy to see they +// can't both be 1 by our range assumptions, so this gives us a {0,1} flag + + adcq $0, %r12 + +// Now convert it into a bitmask + + negq %r12 + +// Masked addition of 2^384 - p_384, hence subtraction of p_384 + + andq %r12, u + andq %r12, v + andq %r12, w + + addq u, %r14 + adcq v, %r15 + adcq w, %r8 + adcq $0, %r9 + adcq $0, %r10 + adcq $0, %r11 + +// Write back the result + + movq %r14, (z) + movq %r15, 8(z) + movq %r8, 16(z) + movq %r9, 24(z) + movq %r10, 32(z) + movq %r11, 40(z) + +// Restore registers and return + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(bignum_tomont_p384_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/curve25519_x25519.S b/cbits/s2n/x86_att/curve25519_x25519.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/curve25519_x25519.S @@ -0,0 +1,2189 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// The x25519 function for curve25519 +// Inputs scalar[4], point[4]; output res[4] +// +// extern void curve25519_x25519 +// (uint64_t res[static 4],const uint64_t scalar[static 4], +// const uint64_t point[static 4]); +// +// The function has a second prototype considering the arguments as arrays +// of bytes rather than 64-bit words. The underlying code is the same, since +// the x86 platform is little-endian. +// +// extern void curve25519_x25519_byte +// (uint8_t res[static 32],const uint8_t scalar[static 32], +// const uint8_t point[static 32]); +// +// Given a scalar n and the X coordinate of an input point P = (X,Y) on +// curve25519 (Y can live in any extension field of characteristic 2^255-19), +// this returns the X coordinate of n * P = (X, Y), or 0 when n * P is the +// point at infinity. Both n and X inputs are first slightly modified/mangled +// as specified in the relevant RFC (https://www.rfc-editor.org/rfc/rfc7748); +// in particular the lower three bits of n are set to zero. Does not implement +// the zero-check specified in Section 6.1. +// +// Standard x86-64 ABI: RDI = res, RSI = scalar, RDX = point +// Microsoft x64 ABI: RCX = res, RDX = scalar, R8 = point +// ---------------------------------------------------------------------------- +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(curve25519_x25519) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(curve25519_x25519) + S2N_BN_SYM_PRIVACY_DIRECTIVE(curve25519_x25519) + S2N_BN_SYM_VISIBILITY_DIRECTIVE(curve25519_x25519_byte) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(curve25519_x25519_byte) + S2N_BN_SYM_PRIVACY_DIRECTIVE(curve25519_x25519_byte) + .text + .balign 32 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Stable homes for the input result argument during the whole body +// and other variables that are only needed prior to the modular inverse. + +#define res 12*NUMSIZE(%rsp) +#define i 12*NUMSIZE+8(%rsp) +#define swap 12*NUMSIZE+16(%rsp) + +// Pointers to result x coord to be written, assuming the base "res" +// has been loaded into %rbp + +#define resx 0(%rbp) + +// Pointer-offset pairs for temporaries on stack with some aliasing. +// Both dmsn and dnsm need space for >= 5 digits, and we allocate 8 + +#define scalar (0*NUMSIZE)(%rsp) + +#define pointx (1*NUMSIZE)(%rsp) + +#define dm (2*NUMSIZE)(%rsp) + +#define zm (3*NUMSIZE)(%rsp) +#define sm (3*NUMSIZE)(%rsp) +#define dpro (3*NUMSIZE)(%rsp) + +#define sn (4*NUMSIZE)(%rsp) + +#define dn (5*NUMSIZE)(%rsp) +#define e (5*NUMSIZE)(%rsp) + +#define dmsn (6*NUMSIZE)(%rsp) +#define p (6*NUMSIZE)(%rsp) +#define zn (7*NUMSIZE)(%rsp) + +#define xm (8*NUMSIZE)(%rsp) +#define dnsm (8*NUMSIZE)(%rsp) +#define spro (8*NUMSIZE)(%rsp) + +#define xn (10*NUMSIZE)(%rsp) +#define s (10*NUMSIZE)(%rsp) + +#define d (11*NUMSIZE)(%rsp) + +// Total size to reserve on the stack +// This includes space for the 3 other variables above +// and rounds up to a multiple of 32 + +#define NSPACE 13*NUMSIZE + +// Macro wrapping up the basic field operation bignum_mul_p25519, only +// trivially different from a pure function call to that subroutine. + +#define mul_p25519(P0,P1,P2) \ + xorl %edi, %edi ; \ + movq P2, %rdx ; \ + mulxq P1, %r8, %r9 ; \ + mulxq 0x8+P1, %rax, %r10 ; \ + addq %rax, %r9 ; \ + mulxq 0x10+P1, %rax, %r11 ; \ + adcq %rax, %r10 ; \ + mulxq 0x18+P1, %rax, %r12 ; \ + adcq %rax, %r11 ; \ + adcq %rdi, %r12 ; \ + xorl %edi, %edi ; \ + movq 0x8+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x18+P1, %rax, %r13 ; \ + adcxq %rax, %r12 ; \ + adoxq %rdi, %r13 ; \ + adcxq %rdi, %r13 ; \ + xorl %edi, %edi ; \ + movq 0x10+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x18+P1, %rax, %r14 ; \ + adcxq %rax, %r13 ; \ + adoxq %rdi, %r14 ; \ + adcxq %rdi, %r14 ; \ + xorl %edi, %edi ; \ + movq 0x18+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + mulxq 0x18+P1, %rax, %r15 ; \ + adcxq %rax, %r14 ; \ + adoxq %rdi, %r15 ; \ + adcxq %rdi, %r15 ; \ + movl $0x26, %edx ; \ + xorl %edi, %edi ; \ + mulxq %r12, %rax, %rbx ; \ + adcxq %rax, %r8 ; \ + adoxq %rbx, %r9 ; \ + mulxq %r13, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq %r14, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq %r15, %rax, %r12 ; \ + adcxq %rax, %r11 ; \ + adoxq %rdi, %r12 ; \ + adcxq %rdi, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + movl $0x13, %edx ; \ + incq %r12; \ + bts $63, %r11 ; \ + mulxq %r12, %rax, %rbx ; \ + addq %rax, %r8 ; \ + adcq %rbx, %r9 ; \ + adcq %rdi, %r10 ; \ + adcq %rdi, %r11 ; \ + sbbq %rax, %rax ; \ + notq %rax; \ + andq %rdx, %rax ; \ + subq %rax, %r8 ; \ + sbbq %rdi, %r9 ; \ + sbbq %rdi, %r10 ; \ + sbbq %rdi, %r11 ; \ + btr $63, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// A version of multiplication that only guarantees output < 2 * p_25519. +// This basically skips the +1 and final correction in quotient estimation. + +#define mul_4(P0,P1,P2) \ + xorl %ecx, %ecx ; \ + movq P2, %rdx ; \ + mulxq P1, %r8, %r9 ; \ + mulxq 0x8+P1, %rax, %r10 ; \ + addq %rax, %r9 ; \ + mulxq 0x10+P1, %rax, %r11 ; \ + adcq %rax, %r10 ; \ + mulxq 0x18+P1, %rax, %r12 ; \ + adcq %rax, %r11 ; \ + adcq %rcx, %r12 ; \ + xorl %ecx, %ecx ; \ + movq 0x8+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x18+P1, %rax, %r13 ; \ + adcxq %rax, %r12 ; \ + adoxq %rcx, %r13 ; \ + adcxq %rcx, %r13 ; \ + xorl %ecx, %ecx ; \ + movq 0x10+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x18+P1, %rax, %r14 ; \ + adcxq %rax, %r13 ; \ + adoxq %rcx, %r14 ; \ + adcxq %rcx, %r14 ; \ + xorl %ecx, %ecx ; \ + movq 0x18+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + mulxq 0x18+P1, %rax, %r15 ; \ + adcxq %rax, %r14 ; \ + adoxq %rcx, %r15 ; \ + adcxq %rcx, %r15 ; \ + movl $0x26, %edx ; \ + xorl %ecx, %ecx ; \ + mulxq %r12, %rax, %rbx ; \ + adcxq %rax, %r8 ; \ + adoxq %rbx, %r9 ; \ + mulxq %r13, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq %r14, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq %r15, %rax, %r12 ; \ + adcxq %rax, %r11 ; \ + adoxq %rcx, %r12 ; \ + adcxq %rcx, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + btr $0x3f, %r11 ; \ + movl $0x13, %edx ; \ + imulq %r12, %rdx ; \ + addq %rdx, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Multiplication just giving a 5-digit result (actually < 39 * p_25519) +// by not doing anything beyond the first stage of reduction + +#define mul_5(P0,P1,P2) \ + xorl %edi, %edi ; \ + movq P2, %rdx ; \ + mulxq P1, %r8, %r9 ; \ + mulxq 0x8+P1, %rax, %r10 ; \ + addq %rax, %r9 ; \ + mulxq 0x10+P1, %rax, %r11 ; \ + adcq %rax, %r10 ; \ + mulxq 0x18+P1, %rax, %r12 ; \ + adcq %rax, %r11 ; \ + adcq %rdi, %r12 ; \ + xorl %edi, %edi ; \ + movq 0x8+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x18+P1, %rax, %r13 ; \ + adcxq %rax, %r12 ; \ + adoxq %rdi, %r13 ; \ + adcxq %rdi, %r13 ; \ + xorl %edi, %edi ; \ + movq 0x10+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x18+P1, %rax, %r14 ; \ + adcxq %rax, %r13 ; \ + adoxq %rdi, %r14 ; \ + adcxq %rdi, %r14 ; \ + xorl %edi, %edi ; \ + movq 0x18+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + mulxq 0x18+P1, %rax, %r15 ; \ + adcxq %rax, %r14 ; \ + adoxq %rdi, %r15 ; \ + adcxq %rdi, %r15 ; \ + movl $0x26, %edx ; \ + xorl %edi, %edi ; \ + mulxq %r12, %rax, %rbx ; \ + adcxq %rax, %r8 ; \ + adoxq %rbx, %r9 ; \ + mulxq %r13, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq %r14, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq %r15, %rax, %r12 ; \ + adcxq %rax, %r11 ; \ + adoxq %rdi, %r12 ; \ + adcxq %rdi, %r12 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 ; \ + movq %r12, 0x20+P0 + +// Squaring just giving a result < 2 * p_25519, which is done by +// basically skipping the +1 in the quotient estimate and the final +// optional correction. + +#define sqr_4(P0,P1) \ + movq P1, %rdx ; \ + mulxq %rdx, %r8, %r15 ; \ + mulxq 0x8+P1, %r9, %r10 ; \ + mulxq 0x18+P1, %r11, %r12 ; \ + movq 0x10+P1, %rdx ; \ + mulxq 0x18+P1, %r13, %r14 ; \ + xorl %ebx, %ebx ; \ + mulxq P1, %rax, %rcx ; \ + adcxq %rax, %r10 ; \ + adoxq %rcx, %r11 ; \ + mulxq 0x8+P1, %rax, %rcx ; \ + adcxq %rax, %r11 ; \ + adoxq %rcx, %r12 ; \ + movq 0x18+P1, %rdx ; \ + mulxq 0x8+P1, %rax, %rcx ; \ + adcxq %rax, %r12 ; \ + adoxq %rcx, %r13 ; \ + adcxq %rbx, %r13 ; \ + adoxq %rbx, %r14 ; \ + adcq %rbx, %r14 ; \ + xorl %ebx, %ebx ; \ + adcxq %r9, %r9 ; \ + adoxq %r15, %r9 ; \ + movq 0x8+P1, %rdx ; \ + mulxq %rdx, %rax, %rdx ; \ + adcxq %r10, %r10 ; \ + adoxq %rax, %r10 ; \ + adcxq %r11, %r11 ; \ + adoxq %rdx, %r11 ; \ + movq 0x10+P1, %rdx ; \ + mulxq %rdx, %rax, %rdx ; \ + adcxq %r12, %r12 ; \ + adoxq %rax, %r12 ; \ + adcxq %r13, %r13 ; \ + adoxq %rdx, %r13 ; \ + movq 0x18+P1, %rdx ; \ + mulxq %rdx, %rax, %r15 ; \ + adcxq %r14, %r14 ; \ + adoxq %rax, %r14 ; \ + adcxq %rbx, %r15 ; \ + adoxq %rbx, %r15 ; \ + movl $0x26, %edx ; \ + xorl %ebx, %ebx ; \ + mulxq %r12, %rax, %rcx ; \ + adcxq %rax, %r8 ; \ + adoxq %rcx, %r9 ; \ + mulxq %r13, %rax, %rcx ; \ + adcxq %rax, %r9 ; \ + adoxq %rcx, %r10 ; \ + mulxq %r14, %rax, %rcx ; \ + adcxq %rax, %r10 ; \ + adoxq %rcx, %r11 ; \ + mulxq %r15, %rax, %r12 ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + adcxq %rbx, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + btr $0x3f, %r11 ; \ + movl $0x13, %edx ; \ + imulq %r12, %rdx ; \ + addq %rdx, %r8 ; \ + adcq %rbx, %r9 ; \ + adcq %rbx, %r10 ; \ + adcq %rbx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Add 5-digit inputs and normalize to 4 digits + +#define add5_4(P0,P1,P2) \ + movq P1, %r8 ; \ + addq P2, %r8 ; \ + movq 8+P1, %r9 ; \ + adcq 8+P2, %r9 ; \ + movq 16+P1, %r10 ; \ + adcq 16+P2, %r10 ; \ + movq 24+P1, %r11 ; \ + adcq 24+P2, %r11 ; \ + movq 32+P1, %r12 ; \ + adcq 32+P2, %r12 ; \ + xorl %ebx, %ebx ; \ + shldq $0x1, %r11, %r12 ; \ + btr $0x3f, %r11 ; \ + movl $0x13, %edx ; \ + imulq %r12, %rdx ; \ + addq %rdx, %r8 ; \ + adcq %rbx, %r9 ; \ + adcq %rbx, %r10 ; \ + adcq %rbx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Modular addition with double modulus 2 * p_25519 = 2^256 - 38. +// This only ensures that the result fits in 4 digits, not that it is reduced +// even w.r.t. double modulus. The result is always correct modulo provided +// the sum of the inputs is < 2^256 + 2^256 - 38, so in particular provided +// at least one of them is reduced double modulo. + +#define add_twice4(P0,P1,P2) \ + movq P1, %r8 ; \ + xorl %ecx, %ecx ; \ + addq P2, %r8 ; \ + movq 0x8+P1, %r9 ; \ + adcq 0x8+P2, %r9 ; \ + movq 0x10+P1, %r10 ; \ + adcq 0x10+P2, %r10 ; \ + movq 0x18+P1, %r11 ; \ + adcq 0x18+P2, %r11 ; \ + movl $38, %eax ; \ + cmovncq %rcx, %rax ; \ + addq %rax, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Modular subtraction with double modulus 2 * p_25519 = 2^256 - 38 + +#define sub_twice4(P0,P1,P2) \ + movq P1, %r8 ; \ + xorl %ebx, %ebx ; \ + subq P2, %r8 ; \ + movq 8+P1, %r9 ; \ + sbbq 8+P2, %r9 ; \ + movl $38, %ecx ; \ + movq 16+P1, %r10 ; \ + sbbq 16+P2, %r10 ; \ + movq 24+P1, %rax ; \ + sbbq 24+P2, %rax ; \ + cmovncq %rbx, %rcx ; \ + subq %rcx, %r8 ; \ + sbbq %rbx, %r9 ; \ + sbbq %rbx, %r10 ; \ + sbbq %rbx, %rax ; \ + movq %r8, P0 ; \ + movq %r9, 8+P0 ; \ + movq %r10, 16+P0 ; \ + movq %rax, 24+P0 + +// 5-digit subtraction with upward bias to make it positive, adding +// 1000 * (2^255 - 19) = 2^256 * 500 - 19000, then normalizing to 4 digits + +#define sub5_4(P0,P1,P2) \ + movq P1, %r8 ; \ + subq P2, %r8 ; \ + movq 8+P1, %r9 ; \ + sbbq 8+P2, %r9 ; \ + movq 16+P1, %r10 ; \ + sbbq 16+P2, %r10 ; \ + movq 24+P1, %r11 ; \ + sbbq 24+P2, %r11 ; \ + movq 32+P1, %r12 ; \ + sbbq 32+P2, %r12 ; \ + xorl %ebx, %ebx ; \ + subq $19000, %r8 ; \ + sbbq %rbx, %r9 ; \ + sbbq %rbx, %r10 ; \ + sbbq %rbx, %r11 ; \ + sbbq %rbx, %r12 ; \ + addq $500, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + btr $0x3f, %r11 ; \ + movl $0x13, %edx ; \ + imulq %r12, %rdx ; \ + addq %rdx, %r8 ; \ + adcq %rbx, %r9 ; \ + adcq %rbx, %r10 ; \ + adcq %rbx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Combined z = c * x + y with reduction only < 2 * p_25519 +// It is assumed that 19 * (c * x + y) < 2^60 * 2^256 so we +// don't need a high mul in the final part. + +#define cmadd_4(P0,C1,P2,P3) \ + movq P3, %r8 ; \ + movq 8+P3, %r9 ; \ + movq 16+P3, %r10 ; \ + movq 24+P3, %r11 ; \ + xorl %edi, %edi ; \ + movq $C1, %rdx ; \ + mulxq P2, %rax, %rbx ; \ + adcxq %rax, %r8 ; \ + adoxq %rbx, %r9 ; \ + mulxq 8+P2, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq 16+P2, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 24+P2, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rdi, %rbx ; \ + adcxq %rdi, %rbx ; \ + shldq $0x1, %r11, %rbx ; \ + btr $63, %r11 ; \ + movl $0x13, %edx ; \ + imulq %rdx, %rbx ; \ + addq %rbx, %r8 ; \ + adcq %rdi, %r9 ; \ + adcq %rdi, %r10 ; \ + adcq %rdi, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Multiplex: z := if NZ then x else y + +#define mux_4(P0,P1,P2) \ + movq P1, %rax ; \ + movq P2, %rcx ; \ + cmovzq %rcx, %rax ; \ + movq %rax, P0 ; \ + movq 8+P1, %rax ; \ + movq 8+P2, %rcx ; \ + cmovzq %rcx, %rax ; \ + movq %rax, 8+P0 ; \ + movq 16+P1, %rax ; \ + movq 16+P2, %rcx ; \ + cmovzq %rcx, %rax ; \ + movq %rax, 16+P0 ; \ + movq 24+P1, %rax ; \ + movq 24+P2, %rcx ; \ + cmovzq %rcx, %rax ; \ + movq %rax, 24+P0 + +S2N_BN_SYMBOL(curve25519_x25519): +S2N_BN_SYMBOL(curve25519_x25519_byte): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx +#endif + +// Save registers, make room for temps, preserve input arguments. + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(NSPACE) + +// Move the output pointer to a stable place + + movq %rdi, res + +// Copy the inputs to the local variables with minimal mangling: +// +// - The scalar is in principle turned into 01xxx...xxx000 but +// in the structure below the special handling of these bits is +// explicit in the main computation; the scalar is just copied. +// +// - The point x coord is reduced mod 2^255 by masking off the +// top bit. In the main loop we only need reduction < 2 * p_25519. + + movq (%rsi), %rax + movq %rax, (%rsp) + movq 8(%rsi), %rax + movq %rax, 8(%rsp) + movq 16(%rsi), %rax + movq %rax, 16(%rsp) + movq 24(%rsi), %rax + movq %rax, 24(%rsp) + + movq (%rdx), %r8 + movq 8(%rdx), %r9 + movq 16(%rdx), %r10 + movq 24(%rdx), %r11 + btr $63, %r11 + movq %r8, 32(%rsp) + movq %r9, 40(%rsp) + movq %r10, 48(%rsp) + movq %r11, 56(%rsp) + +// Initialize with explicit doubling in order to handle set bit 254. +// Set swap = 1 and (xm,zm) = (x,1) then double as (xn,zn) = 2 * (x,1). +// We use the fact that the point x coordinate is still in registers. +// Since zm = 1 we could do the doubling with an operation count of +// 2 * S + M instead of 2 * S + 2 * M, but it doesn't seem worth +// the slight complication arising from a different linear combination. + + movl $1, %eax + movq %rax, swap + movq %r8, 256(%rsp) + movq %rax, 96(%rsp) + xorl %eax, %eax + movq %r9, 264(%rsp) + movq %rax, 104(%rsp) + movq %r10, 272(%rsp) + movq %rax, 112(%rsp) + movq %r11, 280(%rsp) + movq %rax, 120(%rsp) + + sub_twice4(d,xm,zm) + add_twice4(s,xm,zm) + sqr_4(d,d) + sqr_4(s,s) + sub_twice4(p,s,d) + cmadd_4(e,0x1db42,p,d) + mul_4(xn,s,d) + mul_4(zn,p,e) + +// The main loop over unmodified bits from i = 253, ..., i = 3 (inclusive). +// This is a classic Montgomery ladder, with the main coordinates only +// reduced mod 2 * p_25519, some intermediate results even more loosely. + + movl $253, %eax + movq %rax, i + +Lcurve25519_x25519_scalarloop: + +// sm = xm + zm; sn = xn + zn; dm = xm - zm; dn = xn - zn + + sub_twice4(dm,xm,zm) + add_twice4(sn,xn,zn) + sub_twice4(dn,xn,zn) + add_twice4(sm,xm,zm) + +// DOUBLING: mux d = xt - zt and s = xt + zt for appropriate choice of (xt,zt) + + movq i, %rdx + movq %rdx, %rcx + shrq $6, %rdx + movq (%rsp,%rdx,8), %rdx + shrq %cl, %rdx + andq $1, %rdx + cmpq swap, %rdx + movq %rdx, swap + mux_4(d,dm,dn) + mux_4(s,sm,sn) + +// ADDING: dmsn = dm * sn; dnsm = sm * dn + + mul_5(dnsm,sm,dn) + mul_5(dmsn,sn,dm) + +// DOUBLING: d = (xt - zt)^2 + + sqr_4(d,d) + +// ADDING: dpro = (dmsn - dnsm)^2, spro = (dmsn + dnsm)^2 +// DOUBLING: s = (xt + zt)^2 + + sub5_4(dpro,dmsn,dnsm) + add5_4(spro,dmsn,dnsm) + sqr_4(s,s) + sqr_4(dpro,dpro) + +// DOUBLING: p = 4 * xt * zt = s - d + + sub_twice4(p,s,d) + +// ADDING: xm' = (dmsn + dnsm)^2 + + sqr_4(xm,spro) + +// DOUBLING: e = 121666 * p + d + + cmadd_4(e,0x1db42,p,d) + +// DOUBLING: xn' = (xt + zt)^2 * (xt - zt)^2 = s * d + + mul_4(xn,s,d) + +// DOUBLING: zn' = (4 * xt * zt) * ((xt - zt)^2 + 121666 * (4 * xt * zt)) +// = p * (d + 121666 * p) + + mul_4(zn,p,e) + +// ADDING: zm' = x * (dmsn - dnsm)^2 + + mul_4(zm,dpro,pointx) + +// Loop down as far as 3 (inclusive) + + movq i, %rax + subq $1, %rax + movq %rax, i + cmpq $3, %rax + jnc Lcurve25519_x25519_scalarloop + +// Multiplex directly into (xn,zn) then do three pure doubling steps; +// this accounts for the implicit zeroing of the three lowest bits +// of the scalar. + + movq swap, %rdx + testq %rdx, %rdx + mux_4(xn,xm,xn) + mux_4(zn,zm,zn) + + sub_twice4(d,xn,zn) + add_twice4(s,xn,zn) + sqr_4(d,d) + sqr_4(s,s) + sub_twice4(p,s,d) + cmadd_4(e,0x1db42,p,d) + mul_4(xn,s,d) + mul_4(zn,p,e) + + sub_twice4(d,xn,zn) + add_twice4(s,xn,zn) + sqr_4(d,d) + sqr_4(s,s) + sub_twice4(p,s,d) + cmadd_4(e,0x1db42,p,d) + mul_4(xn,s,d) + mul_4(zn,p,e) + + sub_twice4(d,xn,zn) + add_twice4(s,xn,zn) + sqr_4(d,d) + sqr_4(s,s) + sub_twice4(p,s,d) + cmadd_4(e,0x1db42,p,d) + mul_4(xn,s,d) + mul_4(zn,p,e) + +// The projective result of the scalar multiplication is now (xn,zn). +// Prepare to call the modular inverse function to get zn' = 1/zn + + leaq 224(%rsp), %rdi + leaq 224(%rsp), %rsi + +// Inline copy of bignum_inv_p25519, identical except for stripping out +// the prologue and epilogue saving and restoring registers and making +// and reclaiming room on the stack. For more details and explanations see +// "x86/curve25519/bignum_inv_p25519.S". Note that the stack it uses for +// its own temporaries is 208 bytes, so it has no effect on variables +// that are needed in the rest of our computation here: res, xn and zn. + + movq %rdi, 0xc0(%rsp) + xorl %eax, %eax + leaq -0x13(%rax), %rcx + notq %rax + movq %rcx, (%rsp) + movq %rax, 0x8(%rsp) + movq %rax, 0x10(%rsp) + btr $0x3f, %rax + movq %rax, 0x18(%rsp) + movq (%rsi), %rdx + movq 0x8(%rsi), %rcx + movq 0x10(%rsi), %r8 + movq 0x18(%rsi), %r9 + movl $0x1, %eax + xorl %r10d, %r10d + bts $0x3f, %r9 + adcq %r10, %rax + imulq $0x13, %rax, %rax + addq %rax, %rdx + adcq %r10, %rcx + adcq %r10, %r8 + adcq %r10, %r9 + movl $0x13, %eax + cmovbq %r10, %rax + subq %rax, %rdx + sbbq %r10, %rcx + sbbq %r10, %r8 + sbbq %r10, %r9 + btr $0x3f, %r9 + movq %rdx, 0x20(%rsp) + movq %rcx, 0x28(%rsp) + movq %r8, 0x30(%rsp) + movq %r9, 0x38(%rsp) + xorl %eax, %eax + movq %rax, 0x40(%rsp) + movq %rax, 0x48(%rsp) + movq %rax, 0x50(%rsp) + movq %rax, 0x58(%rsp) + movabsq $0xa0f99e2375022099, %rax + movq %rax, 0x60(%rsp) + movabsq $0xa8c68f3f1d132595, %rax + movq %rax, 0x68(%rsp) + movabsq $0x6c6c893805ac5242, %rax + movq %rax, 0x70(%rsp) + movabsq $0x276508b241770615, %rax + movq %rax, 0x78(%rsp) + movq $0xa, 0x90(%rsp) + movq $0x1, 0x98(%rsp) + jmp Lcurve25519_x25519_midloop +Lcurve25519_x25519_inverseloop: + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + movq %r8, %rax + andq %r9, %rax + movq %r10, %rdi + andq %r11, %rdi + addq %rax, %rdi + movq %rdi, 0x80(%rsp) + movq %r12, %rax + andq %r13, %rax + movq %r14, %rsi + andq %r15, %rsi + addq %rax, %rsi + movq %rsi, 0x88(%rsp) + xorl %ebx, %ebx + movq (%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x20(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rdi + adcq %rdx, %rbx + xorl %ebp, %ebp + movq (%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x20(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + xorl %ecx, %ecx + movq 0x8(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x28(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + shrdq $0x3b, %rbx, %rdi + movq %rdi, (%rsp) + xorl %edi, %edi + movq 0x8(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rbp + adcq %rdx, %rdi + movq 0x28(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rdi + shrdq $0x3b, %rbp, %rsi + movq %rsi, 0x20(%rsp) + xorl %esi, %esi + movq 0x10(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rsi + movq 0x30(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rsi + shrdq $0x3b, %rcx, %rbx + movq %rbx, 0x8(%rsp) + xorl %ebx, %ebx + movq 0x10(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x30(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rdi + adcq %rdx, %rbx + shrdq $0x3b, %rdi, %rbp + movq %rbp, 0x28(%rsp) + movq 0x18(%rsp), %rax + xorq %r9, %rax + movq %rax, %rbp + sarq $0x3f, %rbp + andq %r8, %rbp + negq %rbp + mulq %r8 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x38(%rsp), %rax + xorq %r11, %rax + movq %rax, %rdx + sarq $0x3f, %rdx + andq %r10, %rdx + subq %rdx, %rbp + mulq %r10 + addq %rax, %rsi + adcq %rdx, %rbp + shrdq $0x3b, %rsi, %rcx + movq %rcx, 0x10(%rsp) + shrdq $0x3b, %rbp, %rsi + movq 0x18(%rsp), %rax + movq %rsi, 0x18(%rsp) + xorq %r13, %rax + movq %rax, %rsi + sarq $0x3f, %rsi + andq %r12, %rsi + negq %rsi + mulq %r12 + addq %rax, %rbx + adcq %rdx, %rsi + movq 0x38(%rsp), %rax + xorq %r15, %rax + movq %rax, %rdx + sarq $0x3f, %rdx + andq %r14, %rdx + subq %rdx, %rsi + mulq %r14 + addq %rax, %rbx + adcq %rdx, %rsi + shrdq $0x3b, %rbx, %rdi + movq %rdi, 0x30(%rsp) + shrdq $0x3b, %rsi, %rbx + movq %rbx, 0x38(%rsp) + movq 0x80(%rsp), %rbx + movq 0x88(%rsp), %rbp + xorl %ecx, %ecx + movq 0x40(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x60(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x40(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x40(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x60(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x60(%rsp) + xorl %ebx, %ebx + movq 0x48(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x68(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + xorl %ebp, %ebp + movq 0x48(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, 0x48(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x68(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, 0x68(%rsp) + xorl %ecx, %ecx + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x70(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x50(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x50(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x70(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x70(%rsp) + movq 0x58(%rsp), %rax + xorq %r9, %rax + movq %r9, %rbx + andq %r8, %rbx + negq %rbx + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x78(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbx + mulq %r10 + addq %rax, %rcx + adcq %rbx, %rdx + movq %rdx, %rbx + shldq $0x1, %rcx, %rdx + sarq $0x3f, %rbx + addq %rbx, %rdx + movl $0x13, %eax + imulq %rdx + movq 0x40(%rsp), %r8 + addq %rax, %r8 + movq %r8, 0x40(%rsp) + movq 0x48(%rsp), %r8 + adcq %rdx, %r8 + movq %r8, 0x48(%rsp) + movq 0x50(%rsp), %r8 + adcq %rbx, %r8 + movq %r8, 0x50(%rsp) + adcq %rbx, %rcx + shlq $0x3f, %rax + addq %rax, %rcx + movq 0x58(%rsp), %rax + movq %rcx, 0x58(%rsp) + xorq %r13, %rax + movq %r13, %rcx + andq %r12, %rcx + negq %rcx + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rcx + movq 0x78(%rsp), %rax + xorq %r15, %rax + movq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rcx + mulq %r14 + addq %rax, %rsi + adcq %rcx, %rdx + movq %rdx, %rcx + shldq $0x1, %rsi, %rdx + sarq $0x3f, %rcx + movl $0x13, %eax + addq %rcx, %rdx + imulq %rdx + movq 0x60(%rsp), %r8 + addq %rax, %r8 + movq %r8, 0x60(%rsp) + movq 0x68(%rsp), %r8 + adcq %rdx, %r8 + movq %r8, 0x68(%rsp) + movq 0x70(%rsp), %r8 + adcq %rcx, %r8 + movq %r8, 0x70(%rsp) + adcq %rcx, %rsi + shlq $0x3f, %rax + addq %rax, %rsi + movq %rsi, 0x78(%rsp) +Lcurve25519_x25519_midloop: + movq 0x98(%rsp), %rsi + movq (%rsp), %rdx + movq 0x20(%rsp), %rcx + movq %rdx, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + xorl %ebp, %ebp + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %rdx + leaq (%rcx,%rax), %rdi + shlq $0x16, %rdx + shlq $0x16, %rdi + sarq $0x2b, %rdx + sarq $0x2b, %rdi + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %rbx + leaq (%rcx,%rax), %rcx + sarq $0x2a, %rbx + sarq $0x2a, %rcx + movq %rdx, 0xa0(%rsp) + movq %rbx, 0xa8(%rsp) + movq %rdi, 0xb0(%rsp) + movq %rcx, 0xb8(%rsp) + movq (%rsp), %r12 + imulq %r12, %rdi + imulq %rdx, %r12 + movq 0x20(%rsp), %r13 + imulq %r13, %rbx + imulq %rcx, %r13 + addq %rbx, %r12 + addq %rdi, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r10 + shlq $0x16, %r8 + shlq $0x16, %r10 + sarq $0x2b, %r8 + sarq $0x2b, %r10 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r15 + leaq (%rcx,%rax), %r11 + sarq $0x2a, %r15 + sarq $0x2a, %r11 + movq %r13, %rbx + movq %r12, %rcx + imulq %r8, %r12 + imulq %r15, %rbx + addq %rbx, %r12 + imulq %r11, %r13 + imulq %r10, %rcx + addq %rcx, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq 0xa0(%rsp), %rax + imulq %r8, %rax + movq 0xb0(%rsp), %rdx + imulq %r15, %rdx + imulq 0xa8(%rsp), %r8 + imulq 0xb8(%rsp), %r15 + addq %r8, %r15 + leaq (%rax,%rdx), %r9 + movq 0xa0(%rsp), %rax + imulq %r10, %rax + movq 0xb0(%rsp), %rdx + imulq %r11, %rdx + imulq 0xa8(%rsp), %r10 + imulq 0xb8(%rsp), %r11 + addq %r10, %r11 + leaq (%rax,%rdx), %r13 + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r12 + shlq $0x15, %r8 + shlq $0x15, %r12 + sarq $0x2b, %r8 + sarq $0x2b, %r12 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r10 + leaq (%rcx,%rax), %r14 + sarq $0x2b, %r10 + sarq $0x2b, %r14 + movq %r9, %rax + imulq %r8, %rax + movq %r13, %rdx + imulq %r10, %rdx + imulq %r15, %r8 + imulq %r11, %r10 + addq %r8, %r10 + leaq (%rax,%rdx), %r8 + movq %r9, %rax + imulq %r12, %rax + movq %r13, %rdx + imulq %r14, %rdx + imulq %r15, %r12 + imulq %r11, %r14 + addq %r12, %r14 + leaq (%rax,%rdx), %r12 + movq %rsi, 0x98(%rsp) + decq 0x90(%rsp) + jne Lcurve25519_x25519_inverseloop + movq (%rsp), %rax + movq 0x20(%rsp), %rcx + imulq %r8, %rax + imulq %r10, %rcx + addq %rcx, %rax + sarq $0x3f, %rax + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + xorq %rax, %r9 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + xorq %rax, %r11 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + xorq %rax, %r13 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + xorq %rax, %r15 + movq %r8, %rax + andq %r9, %rax + movq %r10, %r12 + andq %r11, %r12 + addq %rax, %r12 + xorl %r13d, %r13d + movq 0x40(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x60(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movq 0x48(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r13 + adcq %rdx, %r14 + movq 0x68(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x70(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x58(%rsp), %rax + xorq %r9, %rax + andq %r8, %r9 + negq %r9 + mulq %r8 + addq %rax, %r15 + adcq %rdx, %r9 + movq 0x78(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %r9 + mulq %r10 + addq %rax, %r15 + adcq %rdx, %r9 + movq %r9, %rax + shldq $0x1, %r15, %rax + sarq $0x3f, %r9 + movl $0x13, %ebx + leaq 0x1(%rax,%r9,1), %rax + imulq %rbx + xorl %ebp, %ebp + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r9, %r14 + adcq %r9, %r15 + shlq $0x3f, %rax + addq %rax, %r15 + cmovns %rbp, %rbx + subq %rbx, %r12 + sbbq %rbp, %r13 + sbbq %rbp, %r14 + sbbq %rbp, %r15 + btr $0x3f, %r15 + movq 0xc0(%rsp), %rdi + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + +// Now the result is xn * (1/zn), fully reduced modulo p. +// Note that in the degenerate case zn = 0 (mod p_25519), the +// modular inverse code above will produce 1/zn = 0, giving +// the correct overall X25519 result of zero for the point at +// infinity. + + movq res, %rbp + mul_p25519(resx,xn,zn) + +// Restore stack and registers + + CFI_INC_RSP(NSPACE) + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(curve25519_x25519) +S2N_BN_SIZE_DIRECTIVE(curve25519_x25519_byte) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/curve25519_x25519_alt.S b/cbits/s2n/x86_att/curve25519_x25519_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/curve25519_x25519_alt.S @@ -0,0 +1,2350 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// The x25519 function for curve25519 +// Inputs scalar[4], point[4]; output res[4] +// +// extern void curve25519_x25519_alt +// (uint64_t res[static 4],const uint64_t scalar[static 4], +// const uint64_t point[static 4]); +// +// The function has a second prototype considering the arguments as arrays +// of bytes rather than 64-bit words. The underlying code is the same, since +// the x86 platform is little-endian. +// +// extern void curve25519_x25519_byte_alt +// (uint8_t res[static 32],const uint8_t scalar[static 32], +// const uint8_t point[static 32]); +// +// Given a scalar n and the X coordinate of an input point P = (X,Y) on +// curve25519 (Y can live in any extension field of characteristic 2^255-19), +// this returns the X coordinate of n * P = (X, Y), or 0 when n * P is the +// point at infinity. Both n and X inputs are first slightly modified/mangled +// as specified in the relevant RFC (https://www.rfc-editor.org/rfc/rfc7748); +// in particular the lower three bits of n are set to zero. Does not implement +// the zero-check specified in Section 6.1. +// +// Standard x86-64 ABI: RDI = res, RSI = scalar, RDX = point +// Microsoft x64 ABI: RCX = res, RDX = scalar, R8 = point +// ---------------------------------------------------------------------------- +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(curve25519_x25519_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(curve25519_x25519_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(curve25519_x25519_alt) + S2N_BN_SYM_VISIBILITY_DIRECTIVE(curve25519_x25519_byte_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(curve25519_x25519_byte_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(curve25519_x25519_byte_alt) + .text + .balign 32 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Stable homes for the input result argument during the whole body +// and other variables that are only needed prior to the modular inverse. + +#define res 12*NUMSIZE(%rsp) +#define i 12*NUMSIZE+8(%rsp) +#define swap 12*NUMSIZE+16(%rsp) + +// Pointers to result x coord to be written, assuming the base "res" +// has been loaded into %rbp + +#define resx 0(%rbp) + +// Pointer-offset pairs for temporaries on stack with some aliasing. +// Both dmsn and dnsm need space for >= 5 digits, and we allocate 8 + +#define scalar (0*NUMSIZE)(%rsp) + +#define pointx (1*NUMSIZE)(%rsp) + +#define dm (2*NUMSIZE)(%rsp) + +#define zm (3*NUMSIZE)(%rsp) +#define sm (3*NUMSIZE)(%rsp) +#define dpro (3*NUMSIZE)(%rsp) + +#define sn (4*NUMSIZE)(%rsp) + +#define dn (5*NUMSIZE)(%rsp) +#define e (5*NUMSIZE)(%rsp) + +#define dmsn (6*NUMSIZE)(%rsp) +#define p (6*NUMSIZE)(%rsp) +#define zn (7*NUMSIZE)(%rsp) + +#define xm (8*NUMSIZE)(%rsp) +#define dnsm (8*NUMSIZE)(%rsp) +#define spro (8*NUMSIZE)(%rsp) + +#define xn (10*NUMSIZE)(%rsp) +#define s (10*NUMSIZE)(%rsp) + +#define d (11*NUMSIZE)(%rsp) + +// Total size to reserve on the stack +// This includes space for the 3 other variables above +// and rounds up to a multiple of 32 + +#define NSPACE 13*NUMSIZE + +// Macro wrapping up the basic field operation bignum_mul_p25519_alt, only +// trivially different from a pure function call to that subroutine. + +#define mul_p25519(P0,P1,P2) \ + movq P1, %rax ; \ + mulq P2; \ + movq %rax, %r8 ; \ + movq %rdx, %r9 ; \ + xorq %r10, %r10 ; \ + xorq %r11, %r11 ; \ + movq P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + movq 0x8+P1, %rax ; \ + mulq P2; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + adcq $0x0, %r11 ; \ + xorq %r12, %r12 ; \ + movq P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq %r12, %r12 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + movq 0x10+P1, %rax ; \ + mulq P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + xorq %r13, %r13 ; \ + movq P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq %r13, %r13 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + movq 0x18+P1, %rax ; \ + mulq P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + xorq %r14, %r14 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq %r14, %r14 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + xorq %r15, %r15 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq %r15, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq $0x0, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + movl $0x26, %esi ; \ + movq %r12, %rax ; \ + mulq %rsi; \ + addq %rax, %r8 ; \ + adcq %rdx, %r9 ; \ + sbbq %rcx, %rcx ; \ + movq %r13, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %rcx, %rcx ; \ + movq %r14, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rcx, %rcx ; \ + movq %r15, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + xorq %rcx, %rcx ; \ + addq %rax, %r11 ; \ + movq %rdx, %r12 ; \ + adcq %rcx, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + leaq 0x1(%r12), %rax ; \ + movl $0x13, %esi ; \ + bts $63, %r11 ; \ + imulq %rsi, %rax ; \ + addq %rax, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + sbbq %rax, %rax ; \ + notq %rax; \ + andq %rsi, %rax ; \ + subq %rax, %r8 ; \ + sbbq %rcx, %r9 ; \ + sbbq %rcx, %r10 ; \ + sbbq %rcx, %r11 ; \ + btr $63, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// A version of multiplication that only guarantees output < 2 * p_25519. +// This basically skips the +1 and final correction in quotient estimation. + +#define mul_4(P0,P1,P2) \ + movq P1, %rax ; \ + mulq P2; \ + movq %rax, %r8 ; \ + movq %rdx, %r9 ; \ + xorq %r10, %r10 ; \ + xorq %r11, %r11 ; \ + movq P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + movq 0x8+P1, %rax ; \ + mulq P2; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + adcq $0x0, %r11 ; \ + xorq %r12, %r12 ; \ + movq P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq %r12, %r12 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + movq 0x10+P1, %rax ; \ + mulq P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + xorq %r13, %r13 ; \ + movq P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq %r13, %r13 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + movq 0x18+P1, %rax ; \ + mulq P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + xorq %r14, %r14 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq %r14, %r14 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + xorq %r15, %r15 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq %r15, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq $0x0, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + movl $0x26, %esi ; \ + movq %r12, %rax ; \ + mulq %rsi; \ + addq %rax, %r8 ; \ + adcq %rdx, %r9 ; \ + sbbq %rcx, %rcx ; \ + movq %r13, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %rcx, %rcx ; \ + movq %r14, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rcx, %rcx ; \ + movq %r15, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + xorq %rcx, %rcx ; \ + addq %rax, %r11 ; \ + movq %rdx, %r12 ; \ + adcq %rcx, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + btr $0x3f, %r11 ; \ + movl $0x13, %edx ; \ + imulq %r12, %rdx ; \ + addq %rdx, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Multiplication just giving a 5-digit result (actually < 39 * p_25519) +// by not doing anything beyond the first stage of reduction + +#define mul_5(P0,P1,P2) \ + movq P1, %rax ; \ + mulq P2; \ + movq %rax, %r8 ; \ + movq %rdx, %r9 ; \ + xorq %r10, %r10 ; \ + xorq %r11, %r11 ; \ + movq P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + movq 0x8+P1, %rax ; \ + mulq P2; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + adcq $0x0, %r11 ; \ + xorq %r12, %r12 ; \ + movq P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq %r12, %r12 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + movq 0x10+P1, %rax ; \ + mulq P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + xorq %r13, %r13 ; \ + movq P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq %r13, %r13 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + movq 0x18+P1, %rax ; \ + mulq P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + xorq %r14, %r14 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq %r14, %r14 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + xorq %r15, %r15 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq %r15, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq $0x0, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + movl $0x26, %esi ; \ + movq %r12, %rax ; \ + mulq %rsi; \ + addq %rax, %r8 ; \ + adcq %rdx, %r9 ; \ + sbbq %rcx, %rcx ; \ + movq %r13, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %rcx, %rcx ; \ + movq %r14, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rcx, %rcx ; \ + movq %r15, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + xorq %rcx, %rcx ; \ + addq %rax, %r11 ; \ + movq %rdx, %r12 ; \ + adcq %rcx, %r12 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 ; \ + movq %r12, 0x20+P0 + +// Squaring just giving a result < 2 * p_25519, which is done by +// basically skipping the +1 in the quotient estimate and the final +// optional correction. + +#define sqr_4(P0,P1) \ + movq P1, %rax ; \ + mulq %rax; \ + movq %rax, %r8 ; \ + movq %rdx, %r9 ; \ + xorq %r10, %r10 ; \ + xorq %r11, %r11 ; \ + movq P1, %rax ; \ + mulq 0x8+P1; \ + addq %rax, %rax ; \ + adcq %rdx, %rdx ; \ + adcq $0x0, %r11 ; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + adcq $0x0, %r11 ; \ + xorq %r12, %r12 ; \ + movq 0x8+P1, %rax ; \ + mulq %rax; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + movq P1, %rax ; \ + mulq 0x10+P1; \ + addq %rax, %rax ; \ + adcq %rdx, %rdx ; \ + adcq $0x0, %r12 ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + xorq %r13, %r13 ; \ + movq P1, %rax ; \ + mulq 0x18+P1; \ + addq %rax, %rax ; \ + adcq %rdx, %rdx ; \ + adcq $0x0, %r13 ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x10+P1; \ + addq %rax, %rax ; \ + adcq %rdx, %rdx ; \ + adcq $0x0, %r13 ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + xorq %r14, %r14 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x18+P1; \ + addq %rax, %rax ; \ + adcq %rdx, %rdx ; \ + adcq $0x0, %r14 ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + movq 0x10+P1, %rax ; \ + mulq %rax; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + xorq %r15, %r15 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x18+P1; \ + addq %rax, %rax ; \ + adcq %rdx, %rdx ; \ + adcq $0x0, %r15 ; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq $0x0, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq %rax; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + movl $0x26, %esi ; \ + movq %r12, %rax ; \ + mulq %rsi; \ + addq %rax, %r8 ; \ + adcq %rdx, %r9 ; \ + sbbq %rcx, %rcx ; \ + movq %r13, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %rcx, %rcx ; \ + movq %r14, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rcx, %rcx ; \ + movq %r15, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + xorq %rcx, %rcx ; \ + addq %rax, %r11 ; \ + movq %rdx, %r12 ; \ + adcq %rcx, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + btr $0x3f, %r11 ; \ + movl $0x13, %edx ; \ + imulq %r12, %rdx ; \ + addq %rdx, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Add 5-digit inputs and normalize to 4 digits + +#define add5_4(P0,P1,P2) \ + movq P1, %r8 ; \ + addq P2, %r8 ; \ + movq 8+P1, %r9 ; \ + adcq 8+P2, %r9 ; \ + movq 16+P1, %r10 ; \ + adcq 16+P2, %r10 ; \ + movq 24+P1, %r11 ; \ + adcq 24+P2, %r11 ; \ + movq 32+P1, %r12 ; \ + adcq 32+P2, %r12 ; \ + xorl %ebx, %ebx ; \ + shldq $0x1, %r11, %r12 ; \ + btr $0x3f, %r11 ; \ + movl $0x13, %edx ; \ + imulq %r12, %rdx ; \ + addq %rdx, %r8 ; \ + adcq %rbx, %r9 ; \ + adcq %rbx, %r10 ; \ + adcq %rbx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Modular addition with double modulus 2 * p_25519 = 2^256 - 38. +// This only ensures that the result fits in 4 digits, not that it is reduced +// even w.r.t. double modulus. The result is always correct modulo provided +// the sum of the inputs is < 2^256 + 2^256 - 38, so in particular provided +// at least one of them is reduced double modulo. + +#define add_twice4(P0,P1,P2) \ + movq P1, %r8 ; \ + xorl %ecx, %ecx ; \ + addq P2, %r8 ; \ + movq 0x8+P1, %r9 ; \ + adcq 0x8+P2, %r9 ; \ + movq 0x10+P1, %r10 ; \ + adcq 0x10+P2, %r10 ; \ + movq 0x18+P1, %r11 ; \ + adcq 0x18+P2, %r11 ; \ + movl $38, %eax ; \ + cmovncq %rcx, %rax ; \ + addq %rax, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Modular subtraction with double modulus 2 * p_25519 = 2^256 - 38 + +#define sub_twice4(P0,P1,P2) \ + movq P1, %r8 ; \ + xorl %ebx, %ebx ; \ + subq P2, %r8 ; \ + movq 8+P1, %r9 ; \ + sbbq 8+P2, %r9 ; \ + movl $38, %ecx ; \ + movq 16+P1, %r10 ; \ + sbbq 16+P2, %r10 ; \ + movq 24+P1, %rax ; \ + sbbq 24+P2, %rax ; \ + cmovncq %rbx, %rcx ; \ + subq %rcx, %r8 ; \ + sbbq %rbx, %r9 ; \ + sbbq %rbx, %r10 ; \ + sbbq %rbx, %rax ; \ + movq %r8, P0 ; \ + movq %r9, 8+P0 ; \ + movq %r10, 16+P0 ; \ + movq %rax, 24+P0 + +// 5-digit subtraction with upward bias to make it positive, adding +// 1000 * (2^255 - 19) = 2^256 * 500 - 19000, then normalizing to 4 digits + +#define sub5_4(P0,P1,P2) \ + movq P1, %r8 ; \ + subq P2, %r8 ; \ + movq 8+P1, %r9 ; \ + sbbq 8+P2, %r9 ; \ + movq 16+P1, %r10 ; \ + sbbq 16+P2, %r10 ; \ + movq 24+P1, %r11 ; \ + sbbq 24+P2, %r11 ; \ + movq 32+P1, %r12 ; \ + sbbq 32+P2, %r12 ; \ + xorl %ebx, %ebx ; \ + subq $19000, %r8 ; \ + sbbq %rbx, %r9 ; \ + sbbq %rbx, %r10 ; \ + sbbq %rbx, %r11 ; \ + sbbq %rbx, %r12 ; \ + addq $500, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + btr $0x3f, %r11 ; \ + movl $0x13, %edx ; \ + imulq %r12, %rdx ; \ + addq %rdx, %r8 ; \ + adcq %rbx, %r9 ; \ + adcq %rbx, %r10 ; \ + adcq %rbx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Combined z = c * x + y with reduction only < 2 * p_25519 +// It is assumed that 19 * (c * x + y) < 2^60 * 2^256 so we +// don't need a high mul in the final part. + +#define cmadd_4(P0,C1,P2,P3) \ + movq $C1, %rsi ; \ + movq P2, %rax ; \ + mulq %rsi; \ + movq %rax, %r8 ; \ + movq %rdx, %r9 ; \ + movq 0x8+P2, %rax ; \ + xorq %r10, %r10 ; \ + mulq %rsi; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + movq 0x10+P2, %rax ; \ + mulq %rsi; \ + addq %rax, %r10 ; \ + adcq $0x0, %rdx ; \ + movq 0x18+P2, %rax ; \ + movq %rdx, %r11 ; \ + mulq %rsi; \ + xorl %esi, %esi ; \ + addq %rax, %r11 ; \ + adcq %rsi, %rdx ; \ + addq P3, %r8 ; \ + adcq 0x8+P3, %r9 ; \ + adcq 0x10+P3, %r10 ; \ + adcq 0x18+P3, %r11 ; \ + adcq %rsi, %rdx ; \ + shldq $0x1, %r11, %rdx ; \ + btr $63, %r11 ; \ + movl $0x13, %ebx ; \ + imulq %rbx, %rdx ; \ + addq %rdx, %r8 ; \ + adcq %rsi, %r9 ; \ + adcq %rsi, %r10 ; \ + adcq %rsi, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Multiplex: z := if NZ then x else y + +#define mux_4(P0,P1,P2) \ + movq P1, %rax ; \ + movq P2, %rcx ; \ + cmovzq %rcx, %rax ; \ + movq %rax, P0 ; \ + movq 8+P1, %rax ; \ + movq 8+P2, %rcx ; \ + cmovzq %rcx, %rax ; \ + movq %rax, 8+P0 ; \ + movq 16+P1, %rax ; \ + movq 16+P2, %rcx ; \ + cmovzq %rcx, %rax ; \ + movq %rax, 16+P0 ; \ + movq 24+P1, %rax ; \ + movq 24+P2, %rcx ; \ + cmovzq %rcx, %rax ; \ + movq %rax, 24+P0 + +S2N_BN_SYMBOL(curve25519_x25519_alt): +S2N_BN_SYMBOL(curve25519_x25519_byte_alt): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx +#endif + +// Save registers, make room for temps, preserve input arguments. + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(NSPACE) + +// Move the output pointer to a stable place + + movq %rdi, res + +// Copy the inputs to the local variables with minimal mangling: +// +// - The scalar is in principle turned into 01xxx...xxx000 but +// in the structure below the special handling of these bits is +// explicit in the main computation; the scalar is just copied. +// +// - The point x coord is reduced mod 2^255 by masking off the +// top bit. In the main loop we only need reduction < 2 * p_25519. + + movq (%rsi), %rax + movq %rax, (%rsp) + movq 8(%rsi), %rax + movq %rax, 8(%rsp) + movq 16(%rsi), %rax + movq %rax, 16(%rsp) + movq 24(%rsi), %rax + movq %rax, 24(%rsp) + + movq (%rdx), %r8 + movq 8(%rdx), %r9 + movq 16(%rdx), %r10 + movq 24(%rdx), %r11 + btr $63, %r11 + movq %r8, 32(%rsp) + movq %r9, 40(%rsp) + movq %r10, 48(%rsp) + movq %r11, 56(%rsp) + +// Initialize with explicit doubling in order to handle set bit 254. +// Set swap = 1 and (xm,zm) = (x,1) then double as (xn,zn) = 2 * (x,1). +// We use the fact that the point x coordinate is still in registers. +// Since zm = 1 we could do the doubling with an operation count of +// 2 * S + M instead of 2 * S + 2 * M, but it doesn't seem worth +// the slight complication arising from a different linear combination. + + movl $1, %eax + movq %rax, swap + movq %r8, 256(%rsp) + movq %rax, 96(%rsp) + xorl %eax, %eax + movq %r9, 264(%rsp) + movq %rax, 104(%rsp) + movq %r10, 272(%rsp) + movq %rax, 112(%rsp) + movq %r11, 280(%rsp) + movq %rax, 120(%rsp) + + sub_twice4(d,xm,zm) + add_twice4(s,xm,zm) + sqr_4(d,d) + sqr_4(s,s) + sub_twice4(p,s,d) + cmadd_4(e,0x1db42,p,d) + mul_4(xn,s,d) + mul_4(zn,p,e) + +// The main loop over unmodified bits from i = 253, ..., i = 3 (inclusive). +// This is a classic Montgomery ladder, with the main coordinates only +// reduced mod 2 * p_25519, some intermediate results even more loosely. + + movl $253, %eax + movq %rax, i + +Lcurve25519_x25519_alt_scalarloop: + +// sm = xm + zm; sn = xn + zn; dm = xm - zm; dn = xn - zn + + sub_twice4(dm,xm,zm) + add_twice4(sn,xn,zn) + sub_twice4(dn,xn,zn) + add_twice4(sm,xm,zm) + +// DOUBLING: mux d = xt - zt and s = xt + zt for appropriate choice of (xt,zt) + + movq i, %rdx + movq %rdx, %rcx + shrq $6, %rdx + movq (%rsp,%rdx,8), %rdx + shrq %cl, %rdx + andq $1, %rdx + cmpq swap, %rdx + movq %rdx, swap + mux_4(d,dm,dn) + mux_4(s,sm,sn) + +// ADDING: dmsn = dm * sn; dnsm = sm * dn + + mul_5(dnsm,sm,dn) + mul_5(dmsn,sn,dm) + +// DOUBLING: d = (xt - zt)^2 + + sqr_4(d,d) + +// ADDING: dpro = (dmsn - dnsm)^2, spro = (dmsn + dnsm)^2 +// DOUBLING: s = (xt + zt)^2 + + sub5_4(dpro,dmsn,dnsm) + add5_4(spro,dmsn,dnsm) + sqr_4(s,s) + sqr_4(dpro,dpro) + +// DOUBLING: p = 4 * xt * zt = s - d + + sub_twice4(p,s,d) + +// ADDING: xm' = (dmsn + dnsm)^2 + + sqr_4(xm,spro) + +// DOUBLING: e = 121666 * p + d + + cmadd_4(e,0x1db42,p,d) + +// DOUBLING: xn' = (xt + zt)^2 * (xt - zt)^2 = s * d + + mul_4(xn,s,d) + +// DOUBLING: zn' = (4 * xt * zt) * ((xt - zt)^2 + 121666 * (4 * xt * zt)) +// = p * (d + 121666 * p) + + mul_4(zn,p,e) + +// ADDING: zm' = x * (dmsn - dnsm)^2 + + mul_4(zm,dpro,pointx) + +// Loop down as far as 3 (inclusive) + + movq i, %rax + subq $1, %rax + movq %rax, i + cmpq $3, %rax + jnc Lcurve25519_x25519_alt_scalarloop + +// Multiplex directly into (xn,zn) then do three pure doubling steps; +// this accounts for the implicit zeroing of the three lowest bits +// of the scalar. + + movq swap, %rdx + testq %rdx, %rdx + mux_4(xn,xm,xn) + mux_4(zn,zm,zn) + + sub_twice4(d,xn,zn) + add_twice4(s,xn,zn) + sqr_4(d,d) + sqr_4(s,s) + sub_twice4(p,s,d) + cmadd_4(e,0x1db42,p,d) + mul_4(xn,s,d) + mul_4(zn,p,e) + + sub_twice4(d,xn,zn) + add_twice4(s,xn,zn) + sqr_4(d,d) + sqr_4(s,s) + sub_twice4(p,s,d) + cmadd_4(e,0x1db42,p,d) + mul_4(xn,s,d) + mul_4(zn,p,e) + + sub_twice4(d,xn,zn) + add_twice4(s,xn,zn) + sqr_4(d,d) + sqr_4(s,s) + sub_twice4(p,s,d) + cmadd_4(e,0x1db42,p,d) + mul_4(xn,s,d) + mul_4(zn,p,e) + +// The projective result of the scalar multiplication is now (xn,zn). +// Prepare to call the modular inverse function to get zn' = 1/zn + + leaq 224(%rsp), %rdi + leaq 224(%rsp), %rsi + +// Inline copy of bignum_inv_p25519, identical except for stripping out +// the prologue and epilogue saving and restoring registers and making +// and reclaiming room on the stack. For more details and explanations see +// "x86/curve25519/bignum_inv_p25519.S". Note that the stack it uses for +// its own temporaries is 208 bytes, so it has no effect on variables +// that are needed in the rest of our computation here: res, xn and zn. + + movq %rdi, 0xc0(%rsp) + xorl %eax, %eax + leaq -0x13(%rax), %rcx + notq %rax + movq %rcx, (%rsp) + movq %rax, 0x8(%rsp) + movq %rax, 0x10(%rsp) + btr $0x3f, %rax + movq %rax, 0x18(%rsp) + movq (%rsi), %rdx + movq 0x8(%rsi), %rcx + movq 0x10(%rsi), %r8 + movq 0x18(%rsi), %r9 + movl $0x1, %eax + xorl %r10d, %r10d + bts $0x3f, %r9 + adcq %r10, %rax + imulq $0x13, %rax, %rax + addq %rax, %rdx + adcq %r10, %rcx + adcq %r10, %r8 + adcq %r10, %r9 + movl $0x13, %eax + cmovbq %r10, %rax + subq %rax, %rdx + sbbq %r10, %rcx + sbbq %r10, %r8 + sbbq %r10, %r9 + btr $0x3f, %r9 + movq %rdx, 0x20(%rsp) + movq %rcx, 0x28(%rsp) + movq %r8, 0x30(%rsp) + movq %r9, 0x38(%rsp) + xorl %eax, %eax + movq %rax, 0x40(%rsp) + movq %rax, 0x48(%rsp) + movq %rax, 0x50(%rsp) + movq %rax, 0x58(%rsp) + movabsq $0xa0f99e2375022099, %rax + movq %rax, 0x60(%rsp) + movabsq $0xa8c68f3f1d132595, %rax + movq %rax, 0x68(%rsp) + movabsq $0x6c6c893805ac5242, %rax + movq %rax, 0x70(%rsp) + movabsq $0x276508b241770615, %rax + movq %rax, 0x78(%rsp) + movq $0xa, 0x90(%rsp) + movq $0x1, 0x98(%rsp) + jmp Lcurve25519_x25519_alt_midloop +Lcurve25519_x25519_alt_inverseloop: + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + movq %r8, %rax + andq %r9, %rax + movq %r10, %rdi + andq %r11, %rdi + addq %rax, %rdi + movq %rdi, 0x80(%rsp) + movq %r12, %rax + andq %r13, %rax + movq %r14, %rsi + andq %r15, %rsi + addq %rax, %rsi + movq %rsi, 0x88(%rsp) + xorl %ebx, %ebx + movq (%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x20(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rdi + adcq %rdx, %rbx + xorl %ebp, %ebp + movq (%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x20(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + xorl %ecx, %ecx + movq 0x8(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x28(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + shrdq $0x3b, %rbx, %rdi + movq %rdi, (%rsp) + xorl %edi, %edi + movq 0x8(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rbp + adcq %rdx, %rdi + movq 0x28(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rdi + shrdq $0x3b, %rbp, %rsi + movq %rsi, 0x20(%rsp) + xorl %esi, %esi + movq 0x10(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rsi + movq 0x30(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rsi + shrdq $0x3b, %rcx, %rbx + movq %rbx, 0x8(%rsp) + xorl %ebx, %ebx + movq 0x10(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x30(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rdi + adcq %rdx, %rbx + shrdq $0x3b, %rdi, %rbp + movq %rbp, 0x28(%rsp) + movq 0x18(%rsp), %rax + xorq %r9, %rax + movq %rax, %rbp + sarq $0x3f, %rbp + andq %r8, %rbp + negq %rbp + mulq %r8 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x38(%rsp), %rax + xorq %r11, %rax + movq %rax, %rdx + sarq $0x3f, %rdx + andq %r10, %rdx + subq %rdx, %rbp + mulq %r10 + addq %rax, %rsi + adcq %rdx, %rbp + shrdq $0x3b, %rsi, %rcx + movq %rcx, 0x10(%rsp) + shrdq $0x3b, %rbp, %rsi + movq 0x18(%rsp), %rax + movq %rsi, 0x18(%rsp) + xorq %r13, %rax + movq %rax, %rsi + sarq $0x3f, %rsi + andq %r12, %rsi + negq %rsi + mulq %r12 + addq %rax, %rbx + adcq %rdx, %rsi + movq 0x38(%rsp), %rax + xorq %r15, %rax + movq %rax, %rdx + sarq $0x3f, %rdx + andq %r14, %rdx + subq %rdx, %rsi + mulq %r14 + addq %rax, %rbx + adcq %rdx, %rsi + shrdq $0x3b, %rbx, %rdi + movq %rdi, 0x30(%rsp) + shrdq $0x3b, %rsi, %rbx + movq %rbx, 0x38(%rsp) + movq 0x80(%rsp), %rbx + movq 0x88(%rsp), %rbp + xorl %ecx, %ecx + movq 0x40(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x60(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x40(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x40(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x60(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x60(%rsp) + xorl %ebx, %ebx + movq 0x48(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x68(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + xorl %ebp, %ebp + movq 0x48(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, 0x48(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x68(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, 0x68(%rsp) + xorl %ecx, %ecx + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x70(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x50(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x50(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x70(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x70(%rsp) + movq 0x58(%rsp), %rax + xorq %r9, %rax + movq %r9, %rbx + andq %r8, %rbx + negq %rbx + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x78(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbx + mulq %r10 + addq %rax, %rcx + adcq %rbx, %rdx + movq %rdx, %rbx + shldq $0x1, %rcx, %rdx + sarq $0x3f, %rbx + addq %rbx, %rdx + movl $0x13, %eax + imulq %rdx + movq 0x40(%rsp), %r8 + addq %rax, %r8 + movq %r8, 0x40(%rsp) + movq 0x48(%rsp), %r8 + adcq %rdx, %r8 + movq %r8, 0x48(%rsp) + movq 0x50(%rsp), %r8 + adcq %rbx, %r8 + movq %r8, 0x50(%rsp) + adcq %rbx, %rcx + shlq $0x3f, %rax + addq %rax, %rcx + movq 0x58(%rsp), %rax + movq %rcx, 0x58(%rsp) + xorq %r13, %rax + movq %r13, %rcx + andq %r12, %rcx + negq %rcx + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rcx + movq 0x78(%rsp), %rax + xorq %r15, %rax + movq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rcx + mulq %r14 + addq %rax, %rsi + adcq %rcx, %rdx + movq %rdx, %rcx + shldq $0x1, %rsi, %rdx + sarq $0x3f, %rcx + movl $0x13, %eax + addq %rcx, %rdx + imulq %rdx + movq 0x60(%rsp), %r8 + addq %rax, %r8 + movq %r8, 0x60(%rsp) + movq 0x68(%rsp), %r8 + adcq %rdx, %r8 + movq %r8, 0x68(%rsp) + movq 0x70(%rsp), %r8 + adcq %rcx, %r8 + movq %r8, 0x70(%rsp) + adcq %rcx, %rsi + shlq $0x3f, %rax + addq %rax, %rsi + movq %rsi, 0x78(%rsp) +Lcurve25519_x25519_alt_midloop: + movq 0x98(%rsp), %rsi + movq (%rsp), %rdx + movq 0x20(%rsp), %rcx + movq %rdx, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + xorl %ebp, %ebp + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %rdx + leaq (%rcx,%rax), %rdi + shlq $0x16, %rdx + shlq $0x16, %rdi + sarq $0x2b, %rdx + sarq $0x2b, %rdi + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %rbx + leaq (%rcx,%rax), %rcx + sarq $0x2a, %rbx + sarq $0x2a, %rcx + movq %rdx, 0xa0(%rsp) + movq %rbx, 0xa8(%rsp) + movq %rdi, 0xb0(%rsp) + movq %rcx, 0xb8(%rsp) + movq (%rsp), %r12 + imulq %r12, %rdi + imulq %rdx, %r12 + movq 0x20(%rsp), %r13 + imulq %r13, %rbx + imulq %rcx, %r13 + addq %rbx, %r12 + addq %rdi, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r10 + shlq $0x16, %r8 + shlq $0x16, %r10 + sarq $0x2b, %r8 + sarq $0x2b, %r10 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r15 + leaq (%rcx,%rax), %r11 + sarq $0x2a, %r15 + sarq $0x2a, %r11 + movq %r13, %rbx + movq %r12, %rcx + imulq %r8, %r12 + imulq %r15, %rbx + addq %rbx, %r12 + imulq %r11, %r13 + imulq %r10, %rcx + addq %rcx, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq 0xa0(%rsp), %rax + imulq %r8, %rax + movq 0xb0(%rsp), %rdx + imulq %r15, %rdx + imulq 0xa8(%rsp), %r8 + imulq 0xb8(%rsp), %r15 + addq %r8, %r15 + leaq (%rax,%rdx), %r9 + movq 0xa0(%rsp), %rax + imulq %r10, %rax + movq 0xb0(%rsp), %rdx + imulq %r11, %rdx + imulq 0xa8(%rsp), %r10 + imulq 0xb8(%rsp), %r11 + addq %r10, %r11 + leaq (%rax,%rdx), %r13 + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r12 + shlq $0x15, %r8 + shlq $0x15, %r12 + sarq $0x2b, %r8 + sarq $0x2b, %r12 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r10 + leaq (%rcx,%rax), %r14 + sarq $0x2b, %r10 + sarq $0x2b, %r14 + movq %r9, %rax + imulq %r8, %rax + movq %r13, %rdx + imulq %r10, %rdx + imulq %r15, %r8 + imulq %r11, %r10 + addq %r8, %r10 + leaq (%rax,%rdx), %r8 + movq %r9, %rax + imulq %r12, %rax + movq %r13, %rdx + imulq %r14, %rdx + imulq %r15, %r12 + imulq %r11, %r14 + addq %r12, %r14 + leaq (%rax,%rdx), %r12 + movq %rsi, 0x98(%rsp) + decq 0x90(%rsp) + jne Lcurve25519_x25519_alt_inverseloop + movq (%rsp), %rax + movq 0x20(%rsp), %rcx + imulq %r8, %rax + imulq %r10, %rcx + addq %rcx, %rax + sarq $0x3f, %rax + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + xorq %rax, %r9 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + xorq %rax, %r11 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + xorq %rax, %r13 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + xorq %rax, %r15 + movq %r8, %rax + andq %r9, %rax + movq %r10, %r12 + andq %r11, %r12 + addq %rax, %r12 + xorl %r13d, %r13d + movq 0x40(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x60(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movq 0x48(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r13 + adcq %rdx, %r14 + movq 0x68(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x70(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x58(%rsp), %rax + xorq %r9, %rax + andq %r8, %r9 + negq %r9 + mulq %r8 + addq %rax, %r15 + adcq %rdx, %r9 + movq 0x78(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %r9 + mulq %r10 + addq %rax, %r15 + adcq %rdx, %r9 + movq %r9, %rax + shldq $0x1, %r15, %rax + sarq $0x3f, %r9 + movl $0x13, %ebx + leaq 0x1(%rax,%r9,1), %rax + imulq %rbx + xorl %ebp, %ebp + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r9, %r14 + adcq %r9, %r15 + shlq $0x3f, %rax + addq %rax, %r15 + cmovns %rbp, %rbx + subq %rbx, %r12 + sbbq %rbp, %r13 + sbbq %rbp, %r14 + sbbq %rbp, %r15 + btr $0x3f, %r15 + movq 0xc0(%rsp), %rdi + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + +// Now the result is xn * (1/zn), fully reduced modulo p. +// Note that in the degenerate case zn = 0 (mod p_25519), the +// modular inverse code above will produce 1/zn = 0, giving +// the correct overall X25519 result of zero for the point at +// infinity. + + movq res, %rbp + mul_p25519(resx,xn,zn) + +// Restore stack and registers + + CFI_INC_RSP(NSPACE) + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(curve25519_x25519_alt) +S2N_BN_SIZE_DIRECTIVE(curve25519_x25519_byte_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/curve25519_x25519base.S b/cbits/s2n/x86_att/curve25519_x25519base.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/curve25519_x25519base.S @@ -0,0 +1,9890 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// The x25519 function for curve25519 on base element 9 +// Input scalar[4]; output res[4] +// +// extern void curve25519_x25519base +// (uint64_t res[static 4], const uint64_t scalar[static 4]); +// +// The function has a second prototype considering the arguments as arrays +// of bytes rather than 64-bit words. The underlying code is the same, since +// the x86 platform is little-endian. +// +// extern void curve25519_x25519base_byte +// (uint8_t res[static 32],const uint8_t scalar[static 32]); +// +// Given a scalar n, returns the X coordinate of n * G where G = (9,...) is +// the standard generator. The scalar is first slightly modified/mangled +// as specified in the relevant RFC (https://www.rfc-editor.org/rfc/rfc7748). +// +// Standard x86-64 ABI: RDI = res, RSI = scalar +// Microsoft x64 ABI: RCX = res, RDX = scalar +// ---------------------------------------------------------------------------- +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(curve25519_x25519base) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(curve25519_x25519base) + S2N_BN_SYM_PRIVACY_DIRECTIVE(curve25519_x25519base) + S2N_BN_SYM_VISIBILITY_DIRECTIVE(curve25519_x25519base_byte) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(curve25519_x25519base_byte) + S2N_BN_SYM_PRIVACY_DIRECTIVE(curve25519_x25519base_byte) + + .text + .balign 32 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Pointer-offset pairs for result and temporaries on stack with some aliasing. +// The result "resx" assumes the "res" pointer has been preloaded into %rbp. + +#define resx (0*NUMSIZE)(%rbp) + +#define scalar (0*NUMSIZE)(%rsp) + +#define tabent (1*NUMSIZE)(%rsp) +#define ymx_2 (1*NUMSIZE)(%rsp) +#define xpy_2 (2*NUMSIZE)(%rsp) +#define kxy_2 (3*NUMSIZE)(%rsp) + +#define acc (4*NUMSIZE)(%rsp) +#define x_1 (4*NUMSIZE)(%rsp) +#define y_1 (5*NUMSIZE)(%rsp) +#define z_1 (6*NUMSIZE)(%rsp) +#define w_1 (7*NUMSIZE)(%rsp) +#define x_3 (4*NUMSIZE)(%rsp) +#define y_3 (5*NUMSIZE)(%rsp) +#define z_3 (6*NUMSIZE)(%rsp) +#define w_3 (7*NUMSIZE)(%rsp) + +#define tmpspace (8*NUMSIZE)(%rsp) +#define t0 (8*NUMSIZE)(%rsp) +#define t1 (9*NUMSIZE)(%rsp) +#define t2 (10*NUMSIZE)(%rsp) +#define t3 (11*NUMSIZE)(%rsp) +#define t4 (12*NUMSIZE)(%rsp) +#define t5 (13*NUMSIZE)(%rsp) + +// Stable homes for the input result pointer, and other variables + +#define res 14*NUMSIZE(%rsp) + +#define i 14*NUMSIZE+8(%rsp) + +#define bias 14*NUMSIZE+16(%rsp) + +#define bf 14*NUMSIZE+24(%rsp) +#define ix 14*NUMSIZE+24(%rsp) + +#define tab 15*NUMSIZE(%rsp) + +// Total size to reserve on the stack + +#define NSPACE 488 + +// Macro wrapping up the basic field multiplication, only trivially +// different from a pure function call to bignum_mul_p25519. + +#define mul_p25519(P0,P1,P2) \ + xorl %esi, %esi ; \ + movq P2, %rdx ; \ + mulxq P1, %r8, %r9 ; \ + mulxq 0x8+P1, %rax, %r10 ; \ + addq %rax, %r9 ; \ + mulxq 0x10+P1, %rax, %r11 ; \ + adcq %rax, %r10 ; \ + mulxq 0x18+P1, %rax, %r12 ; \ + adcq %rax, %r11 ; \ + adcq %rsi, %r12 ; \ + xorl %esi, %esi ; \ + movq 0x8+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x18+P1, %rax, %r13 ; \ + adcxq %rax, %r12 ; \ + adoxq %rsi, %r13 ; \ + adcxq %rsi, %r13 ; \ + xorl %esi, %esi ; \ + movq 0x10+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x18+P1, %rax, %r14 ; \ + adcxq %rax, %r13 ; \ + adoxq %rsi, %r14 ; \ + adcxq %rsi, %r14 ; \ + xorl %esi, %esi ; \ + movq 0x18+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + mulxq 0x18+P1, %rax, %r15 ; \ + adcxq %rax, %r14 ; \ + adoxq %rsi, %r15 ; \ + adcxq %rsi, %r15 ; \ + movl $0x26, %edx ; \ + xorl %esi, %esi ; \ + mulxq %r12, %rax, %rbx ; \ + adcxq %rax, %r8 ; \ + adoxq %rbx, %r9 ; \ + mulxq %r13, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq %r14, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq %r15, %rax, %r12 ; \ + adcxq %rax, %r11 ; \ + adoxq %rsi, %r12 ; \ + adcxq %rsi, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + movl $0x13, %edx ; \ + incq %r12; \ + bts $63, %r11 ; \ + mulxq %r12, %rax, %rbx ; \ + addq %rax, %r8 ; \ + adcq %rbx, %r9 ; \ + adcq %rsi, %r10 ; \ + adcq %rsi, %r11 ; \ + sbbq %rax, %rax ; \ + notq %rax; \ + andq %rdx, %rax ; \ + subq %rax, %r8 ; \ + sbbq %rsi, %r9 ; \ + sbbq %rsi, %r10 ; \ + sbbq %rsi, %r11 ; \ + btr $63, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// A version of multiplication that only guarantees output < 2 * p_25519. +// This basically skips the +1 and final correction in quotient estimation. + +#define mul_4(P0,P1,P2) \ + xorl %ecx, %ecx ; \ + movq P2, %rdx ; \ + mulxq P1, %r8, %r9 ; \ + mulxq 0x8+P1, %rax, %r10 ; \ + addq %rax, %r9 ; \ + mulxq 0x10+P1, %rax, %r11 ; \ + adcq %rax, %r10 ; \ + mulxq 0x18+P1, %rax, %r12 ; \ + adcq %rax, %r11 ; \ + adcq %rcx, %r12 ; \ + xorl %ecx, %ecx ; \ + movq 0x8+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x18+P1, %rax, %r13 ; \ + adcxq %rax, %r12 ; \ + adoxq %rcx, %r13 ; \ + adcxq %rcx, %r13 ; \ + xorl %ecx, %ecx ; \ + movq 0x10+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x18+P1, %rax, %r14 ; \ + adcxq %rax, %r13 ; \ + adoxq %rcx, %r14 ; \ + adcxq %rcx, %r14 ; \ + xorl %ecx, %ecx ; \ + movq 0x18+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + mulxq 0x18+P1, %rax, %r15 ; \ + adcxq %rax, %r14 ; \ + adoxq %rcx, %r15 ; \ + adcxq %rcx, %r15 ; \ + movl $0x26, %edx ; \ + xorl %ecx, %ecx ; \ + mulxq %r12, %rax, %rbx ; \ + adcxq %rax, %r8 ; \ + adoxq %rbx, %r9 ; \ + mulxq %r13, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq %r14, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq %r15, %rax, %r12 ; \ + adcxq %rax, %r11 ; \ + adoxq %rcx, %r12 ; \ + adcxq %rcx, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + btr $0x3f, %r11 ; \ + movl $0x13, %edx ; \ + imulq %r12, %rdx ; \ + addq %rdx, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Modular subtraction with double modulus 2 * p_25519 = 2^256 - 38 + +#define sub_twice4(P0,P1,P2) \ + movq P1, %r8 ; \ + xorl %ebx, %ebx ; \ + subq P2, %r8 ; \ + movq 8+P1, %r9 ; \ + sbbq 8+P2, %r9 ; \ + movl $38, %ecx ; \ + movq 16+P1, %r10 ; \ + sbbq 16+P2, %r10 ; \ + movq 24+P1, %rax ; \ + sbbq 24+P2, %rax ; \ + cmovncq %rbx, %rcx ; \ + subq %rcx, %r8 ; \ + sbbq %rbx, %r9 ; \ + sbbq %rbx, %r10 ; \ + sbbq %rbx, %rax ; \ + movq %r8, P0 ; \ + movq %r9, 8+P0 ; \ + movq %r10, 16+P0 ; \ + movq %rax, 24+P0 + +// Modular addition and doubling with double modulus 2 * p_25519 = 2^256 - 38. +// This only ensures that the result fits in 4 digits, not that it is reduced +// even w.r.t. double modulus. The result is always correct modulo provided +// the sum of the inputs is < 2^256 + 2^256 - 38, so in particular provided +// at least one of them is reduced double modulo. + +#define add_twice4(P0,P1,P2) \ + movq P1, %r8 ; \ + xorl %ecx, %ecx ; \ + addq P2, %r8 ; \ + movq 0x8+P1, %r9 ; \ + adcq 0x8+P2, %r9 ; \ + movq 0x10+P1, %r10 ; \ + adcq 0x10+P2, %r10 ; \ + movq 0x18+P1, %r11 ; \ + adcq 0x18+P2, %r11 ; \ + movl $38, %eax ; \ + cmovncq %rcx, %rax ; \ + addq %rax, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +#define double_twice4(P0,P1) \ + movq P1, %r8 ; \ + xorl %ecx, %ecx ; \ + addq %r8, %r8 ; \ + movq 0x8+P1, %r9 ; \ + adcq %r9, %r9 ; \ + movq 0x10+P1, %r10 ; \ + adcq %r10, %r10 ; \ + movq 0x18+P1, %r11 ; \ + adcq %r11, %r11 ; \ + movl $38, %eax ; \ + cmovncq %rcx, %rax ; \ + addq %rax, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +S2N_BN_SYMBOL(curve25519_x25519base): +S2N_BN_SYMBOL(curve25519_x25519base_byte): + CFI_START + _CET_ENDBR + +// In this case the Windows form literally makes a subroutine call. +// This avoids hassle arising from keeping code and data together. + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + CFI_CALL(Lcurve25519_x25519base_standard) + CFI_POP(%rsi) + CFI_POP(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(curve25519_x25519base) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lcurve25519_x25519base_standard) + +Lcurve25519_x25519base_standard: + CFI_START +#endif + +// Save registers, make room for temps, preserve input arguments. + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(NSPACE) + +// Move the output pointer to a stable place + + movq %rdi, res + +// Copy the input scalar to its local variable while mangling it. +// In principle the mangling is into 01xxx...xxx000, but actually +// we only clear the top two bits so 00xxx...xxxxxx. The additional +// 2^254 * G is taken care of by the starting value for the addition +// chain below, while we never look at the three low bits at all. + + movq (%rsi), %rax + movq %rax, (%rsp) + movq 8(%rsi), %rax + movq %rax, 8(%rsp) + movq 16(%rsi), %rax + movq %rax, 16(%rsp) + movq $0x3fffffffffffffff, %rax + andq 24(%rsi), %rax + movq %rax, 24(%rsp) + +// The main part of the computation is on the edwards25519 curve in +// extended-projective coordinates (X,Y,Z,T), representing a point +// (x,y) via x = X/Z, y = Y/Z and x * y = T/Z (so X * Y = T * Z). +// Only at the very end do we translate back to curve25519. So G +// below means the generator within edwards25519 corresponding to +// (9,...) for curve25519, via the standard isomorphism. +// +// Initialize accumulator "acc" to either (2^254 + 8) * G or just 2^254 * G +// depending on bit 3 of the scalar, the only nonzero bit of the bottom 4. +// Thus, we have effectively dealt with bits 0, 1, 2, 3, 254 and 255. + + movq (%rsp), %rax + andq $8, %rax + + leaq S2N_BN_SYMBOL(curve25519_x25519base_constant)(%rip), %r10 + leaq 8*12(%r10), %r11 + + movq (%r10), %rax + movq (%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*16(%rsp) + + movq 8*1(%r10), %rax + movq 8*1(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*17(%rsp) + + movq 8*2(%r10), %rax + movq 8*2(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*18(%rsp) + + movq 8*3(%r10), %rax + movq 8*3(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*19(%rsp) + + movq 8*4(%r10), %rax + movq 8*4(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*20(%rsp) + + movq 8*5(%r10), %rax + movq 8*5(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*21(%rsp) + + movq 8*6(%r10), %rax + movq 8*6(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*22(%rsp) + + movq 8*7(%r10), %rax + movq 8*7(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*23(%rsp) + + movl $1, %eax + movq %rax, 8*24(%rsp) + movl $0, %eax + movq %rax, 8*25(%rsp) + movq %rax, 8*26(%rsp) + movq %rax, 8*27(%rsp) + + movq 8*8(%r10), %rax + movq 8*8(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*28(%rsp) + + movq 8*9(%r10), %rax + movq 8*9(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*29(%rsp) + + movq 8*10(%r10), %rax + movq 8*10(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*30(%rsp) + + movq 8*11(%r10), %rax + movq 8*11(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*31(%rsp) + +// The counter "i" tracks the bit position for which the scalar has +// already been absorbed, starting at 4 and going up in chunks of 4. +// +// The pointer "tab" points at the current block of the table for +// multiples (2^i * j) * G at the current bit position i; 1 <= j <= 8. +// +// The bias is always either 0 and 1 and needs to be added to the +// partially processed scalar implicitly. This is used to absorb 4 bits +// of scalar per iteration from 3-bit table indexing by exploiting +// negation: (16 * h + l) * G = (16 * (h + 1) - (16 - l)) * G is used +// when l >= 9. Note that we can't have any bias left over at the +// end because of the clearing of bit 255 of the scalar, meaning the +// l >= 9 case cannot arise on the last iteration. + + movq $4, i + leaq 8*24(%r10), %rax + movq %rax, tab + movq $0, bias + +// Start of the main loop, repeated 63 times for i = 4, 8, ..., 252 + +Lcurve25519_x25519base_scalarloop: + +// Look at the next 4-bit field "bf", adding the previous bias as well. +// Choose the table index "ix" as bf when bf <= 8 and 16 - bf for bf >= 9, +// setting the bias to 1 for the next iteration in the latter case. + + movq i, %rax + movq %rax, %rcx + shrq $6, %rax + movq (%rsp,%rax,8), %rax // Exploiting scalar = sp exactly + shrq %cl, %rax + andq $15, %rax + addq bias, %rax + movq %rax, bf + + cmpq $9, bf + sbbq %rax, %rax + incq %rax + movq %rax, bias + + movq $16, %rdi + subq bf, %rdi + cmpq $0, bias + cmovzq bf, %rdi + movq %rdi, ix + +// Perform constant-time lookup in the table to get element number "ix". +// The table entry for the affine point (x,y) is actually a triple +// (y - x,x + y,2 * d * x * y) to precompute parts of the addition. +// Note that "ix" can be 0, so we set up the appropriate identity first. + + movl $1, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + movl $1, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + xorl %r13d, %r13d + xorl %r14d, %r14d + xorl %r15d, %r15d + + movq tab, %rbp + + cmpq $1, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $2, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $3, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $4, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $5, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $6, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $7, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $8, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + + addq $96, %rbp + movq %rbp, tab + +// We now have the triple from the table in registers as follows +// +// [%rdx;%rcx;%rbx;%rax] = y - x +// [%r11;%r10;%r9;%r8] = x + y +// [%r15;%r14;%r13;%r12] = 2 * d * x * y +// +// In case bias = 1 we need to negate this. For Edwards curves +// -(x,y) = (-x,y), i.e. we need to negate the x coordinate. +// In this processed encoding, that amounts to swapping the +// first two fields and negating the third. +// +// The optional negation here also pretends bias = 0 whenever +// ix = 0 so that it doesn't need to handle the case of zero +// inputs, since no non-trivial table entries are zero. Note +// that in the zero case the whole negation is trivial, and +// so indeed is the swapping. + + cmpq $0, bias + + movq %rax, %rsi + cmovnzq %r8, %rsi + cmovnzq %rax, %r8 + movq %rsi, 32(%rsp) + movq %r8, 64(%rsp) + + movq %rbx, %rsi + cmovnzq %r9, %rsi + cmovnzq %rbx, %r9 + movq %rsi, 40(%rsp) + movq %r9, 72(%rsp) + + movq %rcx, %rsi + cmovnzq %r10, %rsi + cmovnzq %rcx, %r10 + movq %rsi, 48(%rsp) + movq %r10, 80(%rsp) + + movq %rdx, %rsi + cmovnzq %r11, %rsi + cmovnzq %rdx, %r11 + movq %rsi, 56(%rsp) + movq %r11, 88(%rsp) + + movq $-19, %rax + movq $-1, %rbx + movq $-1, %rcx + movq $0x7fffffffffffffff, %rdx + subq %r12, %rax + sbbq %r13, %rbx + sbbq %r14, %rcx + sbbq %r15, %rdx + + movq ix, %r8 + movq bias, %r9 + testq %r8, %r8 + cmovzq %r8, %r9 + testq %r9, %r9 + + cmovzq %r12, %rax + cmovzq %r13, %rbx + cmovzq %r14, %rcx + cmovzq %r15, %rdx + movq %rax, 96(%rsp) + movq %rbx, 104(%rsp) + movq %rcx, 112(%rsp) + movq %rdx, 120(%rsp) + +// Extended-projective and precomputed mixed addition. +// This is effectively the same as calling the standalone +// function edwards25519_pepadd(acc,acc,tabent), but we +// only retain slightly weaker normalization < 2 * p_25519 +// throughout the inner loop, so the computation is +// slightly different, and faster overall. + + double_twice4(t0,z_1) + sub_twice4(t1,y_1,x_1) + add_twice4(t2,y_1,x_1) + mul_4(t3,w_1,kxy_2) + mul_4(t1,t1,ymx_2) + mul_4(t2,t2,xpy_2) + sub_twice4(t4,t0,t3) + add_twice4(t0,t0,t3) + sub_twice4(t5,t2,t1) + add_twice4(t1,t2,t1) + mul_4(z_3,t4,t0) + mul_4(x_3,t5,t4) + mul_4(y_3,t0,t1) + mul_4(w_3,t5,t1) + +// End of the main loop; move on by 4 bits. + + addq $4, i + cmpq $256, i + jc Lcurve25519_x25519base_scalarloop + +// Now we need to translate from Edwards curve edwards25519 back +// to the Montgomery form curve25519. The mapping in the affine +// representations is +// +// (x,y) |-> ((1 + y) / (1 - y), c * (1 + y) / ((1 - y) * x)) +// +// For x25519, we only need the x coordinate, and we compute this as +// +// (1 + y) / (1 - y) = (x + x * y) / (x - x * y) +// = (X/Z + T/Z) / (X/Z - T/Z) +// = (X + T) / (X - T) +// = (X + T) * inverse(X - T) +// +// We could equally well use (Z + Y) / (Z - Y), but the above has the +// same cost, and it more explicitly forces zero output whenever X = 0, +// regardless of how the modular inverse behaves on zero inputs. In +// the present setting (base point 9, mangled scalar) that doesn't +// really matter anyway since X = 0 never arises, but it seems a +// little bit tidier. Note that both Edwards point (0,1) which maps to +// the Montgomery point at infinity, and Edwards (0,-1) which maps to +// Montgomery (0,0) [this is the 2-torsion point] are both by definition +// mapped to 0 by the X coordinate mapping used to define curve25519. +// +// First the addition and subtraction: + + add_twice4(t1,x_3,w_3) + sub_twice4(t2,x_3,w_3) + +// Prepare to call the modular inverse function to get t0 = 1/t2 +// Note that this works for the weakly normalized z_3 equally well. +// The non-coprime case z_3 == 0 (mod p_25519) cannot arise anyway. + + leaq 256(%rsp), %rdi + leaq 320(%rsp), %rsi + +// Inline copy of bignum_inv_p25519, identical except for stripping out +// the prologue and epilogue saving and restoring registers and making +// and reclaiming room on the stack. For more details and explanations see +// "x86/curve25519/bignum_inv_p25519.S". Note that the stack it uses for +// its own temporaries is 208 bytes, so it has no effect on variables +// that are needed in the rest of our computation here: res, t0, t1, t2. + + movq %rdi, 0xc0(%rsp) + xorl %eax, %eax + leaq -0x13(%rax), %rcx + notq %rax + movq %rcx, (%rsp) + movq %rax, 0x8(%rsp) + movq %rax, 0x10(%rsp) + btr $0x3f, %rax + movq %rax, 0x18(%rsp) + movq (%rsi), %rdx + movq 0x8(%rsi), %rcx + movq 0x10(%rsi), %r8 + movq 0x18(%rsi), %r9 + movl $0x1, %eax + xorl %r10d, %r10d + bts $0x3f, %r9 + adcq %r10, %rax + imulq $0x13, %rax, %rax + addq %rax, %rdx + adcq %r10, %rcx + adcq %r10, %r8 + adcq %r10, %r9 + movl $0x13, %eax + cmovbq %r10, %rax + subq %rax, %rdx + sbbq %r10, %rcx + sbbq %r10, %r8 + sbbq %r10, %r9 + btr $0x3f, %r9 + movq %rdx, 0x20(%rsp) + movq %rcx, 0x28(%rsp) + movq %r8, 0x30(%rsp) + movq %r9, 0x38(%rsp) + xorl %eax, %eax + movq %rax, 0x40(%rsp) + movq %rax, 0x48(%rsp) + movq %rax, 0x50(%rsp) + movq %rax, 0x58(%rsp) + movabsq $0xa0f99e2375022099, %rax + movq %rax, 0x60(%rsp) + movabsq $0xa8c68f3f1d132595, %rax + movq %rax, 0x68(%rsp) + movabsq $0x6c6c893805ac5242, %rax + movq %rax, 0x70(%rsp) + movabsq $0x276508b241770615, %rax + movq %rax, 0x78(%rsp) + movq $0xa, 0x90(%rsp) + movq $0x1, 0x98(%rsp) + jmp Lcurve25519_x25519base_midloop +Lcurve25519_x25519base_inverseloop: + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + movq %r8, %rax + andq %r9, %rax + movq %r10, %rdi + andq %r11, %rdi + addq %rax, %rdi + movq %rdi, 0x80(%rsp) + movq %r12, %rax + andq %r13, %rax + movq %r14, %rsi + andq %r15, %rsi + addq %rax, %rsi + movq %rsi, 0x88(%rsp) + xorl %ebx, %ebx + movq (%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x20(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rdi + adcq %rdx, %rbx + xorl %ebp, %ebp + movq (%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x20(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + xorl %ecx, %ecx + movq 0x8(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x28(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + shrdq $0x3b, %rbx, %rdi + movq %rdi, (%rsp) + xorl %edi, %edi + movq 0x8(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rbp + adcq %rdx, %rdi + movq 0x28(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rdi + shrdq $0x3b, %rbp, %rsi + movq %rsi, 0x20(%rsp) + xorl %esi, %esi + movq 0x10(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rsi + movq 0x30(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rsi + shrdq $0x3b, %rcx, %rbx + movq %rbx, 0x8(%rsp) + xorl %ebx, %ebx + movq 0x10(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x30(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rdi + adcq %rdx, %rbx + shrdq $0x3b, %rdi, %rbp + movq %rbp, 0x28(%rsp) + movq 0x18(%rsp), %rax + xorq %r9, %rax + movq %rax, %rbp + sarq $0x3f, %rbp + andq %r8, %rbp + negq %rbp + mulq %r8 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x38(%rsp), %rax + xorq %r11, %rax + movq %rax, %rdx + sarq $0x3f, %rdx + andq %r10, %rdx + subq %rdx, %rbp + mulq %r10 + addq %rax, %rsi + adcq %rdx, %rbp + shrdq $0x3b, %rsi, %rcx + movq %rcx, 0x10(%rsp) + shrdq $0x3b, %rbp, %rsi + movq 0x18(%rsp), %rax + movq %rsi, 0x18(%rsp) + xorq %r13, %rax + movq %rax, %rsi + sarq $0x3f, %rsi + andq %r12, %rsi + negq %rsi + mulq %r12 + addq %rax, %rbx + adcq %rdx, %rsi + movq 0x38(%rsp), %rax + xorq %r15, %rax + movq %rax, %rdx + sarq $0x3f, %rdx + andq %r14, %rdx + subq %rdx, %rsi + mulq %r14 + addq %rax, %rbx + adcq %rdx, %rsi + shrdq $0x3b, %rbx, %rdi + movq %rdi, 0x30(%rsp) + shrdq $0x3b, %rsi, %rbx + movq %rbx, 0x38(%rsp) + movq 0x80(%rsp), %rbx + movq 0x88(%rsp), %rbp + xorl %ecx, %ecx + movq 0x40(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x60(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x40(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x40(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x60(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x60(%rsp) + xorl %ebx, %ebx + movq 0x48(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x68(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + xorl %ebp, %ebp + movq 0x48(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, 0x48(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x68(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, 0x68(%rsp) + xorl %ecx, %ecx + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x70(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x50(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x50(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x70(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x70(%rsp) + movq 0x58(%rsp), %rax + xorq %r9, %rax + movq %r9, %rbx + andq %r8, %rbx + negq %rbx + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x78(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbx + mulq %r10 + addq %rax, %rcx + adcq %rbx, %rdx + movq %rdx, %rbx + shldq $0x1, %rcx, %rdx + sarq $0x3f, %rbx + addq %rbx, %rdx + movl $0x13, %eax + imulq %rdx + movq 0x40(%rsp), %r8 + addq %rax, %r8 + movq %r8, 0x40(%rsp) + movq 0x48(%rsp), %r8 + adcq %rdx, %r8 + movq %r8, 0x48(%rsp) + movq 0x50(%rsp), %r8 + adcq %rbx, %r8 + movq %r8, 0x50(%rsp) + adcq %rbx, %rcx + shlq $0x3f, %rax + addq %rax, %rcx + movq 0x58(%rsp), %rax + movq %rcx, 0x58(%rsp) + xorq %r13, %rax + movq %r13, %rcx + andq %r12, %rcx + negq %rcx + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rcx + movq 0x78(%rsp), %rax + xorq %r15, %rax + movq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rcx + mulq %r14 + addq %rax, %rsi + adcq %rcx, %rdx + movq %rdx, %rcx + shldq $0x1, %rsi, %rdx + sarq $0x3f, %rcx + movl $0x13, %eax + addq %rcx, %rdx + imulq %rdx + movq 0x60(%rsp), %r8 + addq %rax, %r8 + movq %r8, 0x60(%rsp) + movq 0x68(%rsp), %r8 + adcq %rdx, %r8 + movq %r8, 0x68(%rsp) + movq 0x70(%rsp), %r8 + adcq %rcx, %r8 + movq %r8, 0x70(%rsp) + adcq %rcx, %rsi + shlq $0x3f, %rax + addq %rax, %rsi + movq %rsi, 0x78(%rsp) +Lcurve25519_x25519base_midloop: + movq 0x98(%rsp), %rsi + movq (%rsp), %rdx + movq 0x20(%rsp), %rcx + movq %rdx, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + xorl %ebp, %ebp + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %rdx + leaq (%rcx,%rax), %rdi + shlq $0x16, %rdx + shlq $0x16, %rdi + sarq $0x2b, %rdx + sarq $0x2b, %rdi + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %rbx + leaq (%rcx,%rax), %rcx + sarq $0x2a, %rbx + sarq $0x2a, %rcx + movq %rdx, 0xa0(%rsp) + movq %rbx, 0xa8(%rsp) + movq %rdi, 0xb0(%rsp) + movq %rcx, 0xb8(%rsp) + movq (%rsp), %r12 + imulq %r12, %rdi + imulq %rdx, %r12 + movq 0x20(%rsp), %r13 + imulq %r13, %rbx + imulq %rcx, %r13 + addq %rbx, %r12 + addq %rdi, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r10 + shlq $0x16, %r8 + shlq $0x16, %r10 + sarq $0x2b, %r8 + sarq $0x2b, %r10 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r15 + leaq (%rcx,%rax), %r11 + sarq $0x2a, %r15 + sarq $0x2a, %r11 + movq %r13, %rbx + movq %r12, %rcx + imulq %r8, %r12 + imulq %r15, %rbx + addq %rbx, %r12 + imulq %r11, %r13 + imulq %r10, %rcx + addq %rcx, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq 0xa0(%rsp), %rax + imulq %r8, %rax + movq 0xb0(%rsp), %rdx + imulq %r15, %rdx + imulq 0xa8(%rsp), %r8 + imulq 0xb8(%rsp), %r15 + addq %r8, %r15 + leaq (%rax,%rdx), %r9 + movq 0xa0(%rsp), %rax + imulq %r10, %rax + movq 0xb0(%rsp), %rdx + imulq %r11, %rdx + imulq 0xa8(%rsp), %r10 + imulq 0xb8(%rsp), %r11 + addq %r10, %r11 + leaq (%rax,%rdx), %r13 + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r12 + shlq $0x15, %r8 + shlq $0x15, %r12 + sarq $0x2b, %r8 + sarq $0x2b, %r12 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r10 + leaq (%rcx,%rax), %r14 + sarq $0x2b, %r10 + sarq $0x2b, %r14 + movq %r9, %rax + imulq %r8, %rax + movq %r13, %rdx + imulq %r10, %rdx + imulq %r15, %r8 + imulq %r11, %r10 + addq %r8, %r10 + leaq (%rax,%rdx), %r8 + movq %r9, %rax + imulq %r12, %rax + movq %r13, %rdx + imulq %r14, %rdx + imulq %r15, %r12 + imulq %r11, %r14 + addq %r12, %r14 + leaq (%rax,%rdx), %r12 + movq %rsi, 0x98(%rsp) + decq 0x90(%rsp) + jne Lcurve25519_x25519base_inverseloop + movq (%rsp), %rax + movq 0x20(%rsp), %rcx + imulq %r8, %rax + imulq %r10, %rcx + addq %rcx, %rax + sarq $0x3f, %rax + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + xorq %rax, %r9 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + xorq %rax, %r11 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + xorq %rax, %r13 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + xorq %rax, %r15 + movq %r8, %rax + andq %r9, %rax + movq %r10, %r12 + andq %r11, %r12 + addq %rax, %r12 + xorl %r13d, %r13d + movq 0x40(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x60(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movq 0x48(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r13 + adcq %rdx, %r14 + movq 0x68(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x70(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x58(%rsp), %rax + xorq %r9, %rax + andq %r8, %r9 + negq %r9 + mulq %r8 + addq %rax, %r15 + adcq %rdx, %r9 + movq 0x78(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %r9 + mulq %r10 + addq %rax, %r15 + adcq %rdx, %r9 + movq %r9, %rax + shldq $0x1, %r15, %rax + sarq $0x3f, %r9 + movl $0x13, %ebx + leaq 0x1(%rax,%r9,1), %rax + imulq %rbx + xorl %ebp, %ebp + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r9, %r14 + adcq %r9, %r15 + shlq $0x3f, %rax + addq %rax, %r15 + cmovns %rbp, %rbx + subq %rbx, %r12 + sbbq %rbp, %r13 + sbbq %rbp, %r14 + sbbq %rbp, %r15 + btr $0x3f, %r15 + movq 0xc0(%rsp), %rdi + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + +// The final result is (X + T) / (X - T) +// This is the only operation in the whole computation that +// fully reduces modulo p_25519 since now we want the canonical +// answer as output. + + movq res, %rbp + mul_p25519(resx,t1,t0) + +// Restore stack and registers + + CFI_INC_RSP(NSPACE) + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +#if WINDOWS_ABI +S2N_BN_SIZE_DIRECTIVE(Lcurve25519_x25519base_standard) +#else +S2N_BN_SIZE_DIRECTIVE(curve25519_x25519base) +#endif + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif + +// **************************************************************************** +// The precomputed data (all read-only). +// **************************************************************************** + +#if defined(__ELF__) +.section .rodata +.type S2N_BN_SYMBOL(curve25519_x25519base_constant), %object +.size S2N_BN_SYMBOL(curve25519_x25519base_constant), 48576 +#elif defined(__APPLE__) +.const_data +#endif + +S2N_BN_SYMBOL(curve25519_x25519base_constant): + +// 2^254 * G and (2^254 + 8) * G in extended-projective coordinates +// but with z = 1 assumed and hence left out, so they are (X,Y,T) only. + + .quad 0x251037f7cf4e861d + .quad 0x10ede0fb19fb128f + .quad 0x96c033b175f5e2c8 + .quad 0x055f070d6c15fb0d + + .quad 0x7c52af2c97473e69 + .quad 0x022f82391bad8378 + .quad 0x9991e1b02adb476f + .quad 0x511144a03a99b855 + + .quad 0x5fafc3b88ff2e4ae + .quad 0x855e4ff0de1230ff + .quad 0x72e302a348492870 + .quad 0x1253c19e53dbe1bc + + + .quad 0x331d086e0d9abcaa + .quad 0x1e23c96d311a10c9 + .quad 0x96d0f95e58c13478 + .quad 0x2f72f7384fcfcc59 + + .quad 0x39a6cd1cfd7d87c9 + .quad 0x9867a0abd8ae153a + .quad 0xa49d2a5f35986745 + .quad 0x57012940cdfe82e1 + + .quad 0x5046a6532ec5544a + .quad 0x6d674004739ff6c9 + .quad 0x9bbaa44b234a70e3 + .quad 0x5e6d8901138cf386 + +// Precomputed table of multiples of generator for edwards25519 +// all in precomputed extended-projective (y-x,x+y,2*d*x*y) triples. + + // 2^4 * 1 * G + + .quad 0x7ec851ca553e2df3 + .quad 0xa71284cba64878b3 + .quad 0xe6b5e4193288d1e7 + .quad 0x4cf210ec5a9a8883 + .quad 0x322d04a52d9021f6 + .quad 0xb9c19f3375c6bf9c + .quad 0x587a3a4342d20b09 + .quad 0x143b1cf8aa64fe61 + .quad 0x9f867c7d968acaab + .quad 0x5f54258e27092729 + .quad 0xd0a7d34bea180975 + .quad 0x21b546a3374126e1 + + // 2^4 * 2 * G + + .quad 0xa94ff858a2888343 + .quad 0xce0ed4565313ed3c + .quad 0xf55c3dcfb5bf34fa + .quad 0x0a653ca5c9eab371 + .quad 0x490a7a45d185218f + .quad 0x9a15377846049335 + .quad 0x0060ea09cc31e1f6 + .quad 0x7e041577f86ee965 + .quad 0x66b2a496ce5b67f3 + .quad 0xff5492d8bd569796 + .quad 0x503cec294a592cd0 + .quad 0x566943650813acb2 + + // 2^4 * 3 * G + + .quad 0xb818db0c26620798 + .quad 0x5d5c31d9606e354a + .quad 0x0982fa4f00a8cdc7 + .quad 0x17e12bcd4653e2d4 + .quad 0x5672f9eb1dabb69d + .quad 0xba70b535afe853fc + .quad 0x47ac0f752796d66d + .quad 0x32a5351794117275 + .quad 0xd3a644a6df648437 + .quad 0x703b6559880fbfdd + .quad 0xcb852540ad3a1aa5 + .quad 0x0900b3f78e4c6468 + + // 2^4 * 4 * G + + .quad 0x0a851b9f679d651b + .quad 0xe108cb61033342f2 + .quad 0xd601f57fe88b30a3 + .quad 0x371f3acaed2dd714 + .quad 0xed280fbec816ad31 + .quad 0x52d9595bd8e6efe3 + .quad 0x0fe71772f6c623f5 + .quad 0x4314030b051e293c + .quad 0xd560005efbf0bcad + .quad 0x8eb70f2ed1870c5e + .quad 0x201f9033d084e6a0 + .quad 0x4c3a5ae1ce7b6670 + + // 2^4 * 5 * G + + .quad 0x4138a434dcb8fa95 + .quad 0x870cf67d6c96840b + .quad 0xde388574297be82c + .quad 0x7c814db27262a55a + .quad 0xbaf875e4c93da0dd + .quad 0xb93282a771b9294d + .quad 0x80d63fb7f4c6c460 + .quad 0x6de9c73dea66c181 + .quad 0x478904d5a04df8f2 + .quad 0xfafbae4ab10142d3 + .quad 0xf6c8ac63555d0998 + .quad 0x5aac4a412f90b104 + + // 2^4 * 6 * G + + .quad 0xc64f326b3ac92908 + .quad 0x5551b282e663e1e0 + .quad 0x476b35f54a1a4b83 + .quad 0x1b9da3fe189f68c2 + .quad 0x603a0d0abd7f5134 + .quad 0x8089c932e1d3ae46 + .quad 0xdf2591398798bd63 + .quad 0x1c145cd274ba0235 + .quad 0x32e8386475f3d743 + .quad 0x365b8baf6ae5d9ef + .quad 0x825238b6385b681e + .quad 0x234929c1167d65e1 + + // 2^4 * 7 * G + + .quad 0x984decaba077ade8 + .quad 0x383f77ad19eb389d + .quad 0xc7ec6b7e2954d794 + .quad 0x59c77b3aeb7c3a7a + .quad 0x48145cc21d099fcf + .quad 0x4535c192cc28d7e5 + .quad 0x80e7c1e548247e01 + .quad 0x4a5f28743b2973ee + .quad 0xd3add725225ccf62 + .quad 0x911a3381b2152c5d + .quad 0xd8b39fad5b08f87d + .quad 0x6f05606b4799fe3b + + // 2^4 * 8 * G + + .quad 0x9ffe9e92177ba962 + .quad 0x98aee71d0de5cae1 + .quad 0x3ff4ae942d831044 + .quad 0x714de12e58533ac8 + .quad 0x5b433149f91b6483 + .quad 0xadb5dc655a2cbf62 + .quad 0x87fa8412632827b3 + .quad 0x60895e91ab49f8d8 + .quad 0xe9ecf2ed0cf86c18 + .quad 0xb46d06120735dfd4 + .quad 0xbc9da09804b96be7 + .quad 0x73e2e62fd96dc26b + + // 2^8 * 1 * G + + .quad 0xed5b635449aa515e + .quad 0xa865c49f0bc6823a + .quad 0x850c1fe95b42d1c4 + .quad 0x30d76d6f03d315b9 + .quad 0x2eccdd0e632f9c1d + .quad 0x51d0b69676893115 + .quad 0x52dfb76ba8637a58 + .quad 0x6dd37d49a00eef39 + .quad 0x6c4444172106e4c7 + .quad 0xfb53d680928d7f69 + .quad 0xb4739ea4694d3f26 + .quad 0x10c697112e864bb0 + + // 2^8 * 2 * G + + .quad 0x6493c4277dbe5fde + .quad 0x265d4fad19ad7ea2 + .quad 0x0e00dfc846304590 + .quad 0x25e61cabed66fe09 + .quad 0x0ca62aa08358c805 + .quad 0x6a3d4ae37a204247 + .quad 0x7464d3a63b11eddc + .quad 0x03bf9baf550806ef + .quad 0x3f13e128cc586604 + .quad 0x6f5873ecb459747e + .quad 0xa0b63dedcc1268f5 + .quad 0x566d78634586e22c + + // 2^8 * 3 * G + + .quad 0x1637a49f9cc10834 + .quad 0xbc8e56d5a89bc451 + .quad 0x1cb5ec0f7f7fd2db + .quad 0x33975bca5ecc35d9 + .quad 0xa1054285c65a2fd0 + .quad 0x6c64112af31667c3 + .quad 0x680ae240731aee58 + .quad 0x14fba5f34793b22a + .quad 0x3cd746166985f7d4 + .quad 0x593e5e84c9c80057 + .quad 0x2fc3f2b67b61131e + .quad 0x14829cea83fc526c + + // 2^8 * 4 * G + + .quad 0xff437b8497dd95c2 + .quad 0x6c744e30aa4eb5a7 + .quad 0x9e0c5d613c85e88b + .quad 0x2fd9c71e5f758173 + .quad 0x21e70b2f4e71ecb8 + .quad 0xe656ddb940a477e3 + .quad 0xbf6556cece1d4f80 + .quad 0x05fc3bc4535d7b7e + .quad 0x24b8b3ae52afdedd + .quad 0x3495638ced3b30cf + .quad 0x33a4bc83a9be8195 + .quad 0x373767475c651f04 + + // 2^8 * 5 * G + + .quad 0x2fba99fd40d1add9 + .quad 0xb307166f96f4d027 + .quad 0x4363f05215f03bae + .quad 0x1fbea56c3b18f999 + .quad 0x634095cb14246590 + .quad 0xef12144016c15535 + .quad 0x9e38140c8910bc60 + .quad 0x6bf5905730907c8c + .quad 0x0fa778f1e1415b8a + .quad 0x06409ff7bac3a77e + .quad 0x6f52d7b89aa29a50 + .quad 0x02521cf67a635a56 + + // 2^8 * 6 * G + + .quad 0x513fee0b0a9d5294 + .quad 0x8f98e75c0fdf5a66 + .quad 0xd4618688bfe107ce + .quad 0x3fa00a7e71382ced + .quad 0xb1146720772f5ee4 + .quad 0xe8f894b196079ace + .quad 0x4af8224d00ac824a + .quad 0x001753d9f7cd6cc4 + .quad 0x3c69232d963ddb34 + .quad 0x1dde87dab4973858 + .quad 0xaad7d1f9a091f285 + .quad 0x12b5fe2fa048edb6 + + // 2^8 * 7 * G + + .quad 0x71f0fbc496fce34d + .quad 0x73b9826badf35bed + .quad 0xd2047261ff28c561 + .quad 0x749b76f96fb1206f + .quad 0xdf2b7c26ad6f1e92 + .quad 0x4b66d323504b8913 + .quad 0x8c409dc0751c8bc3 + .quad 0x6f7e93c20796c7b8 + .quad 0x1f5af604aea6ae05 + .quad 0xc12351f1bee49c99 + .quad 0x61a808b5eeff6b66 + .quad 0x0fcec10f01e02151 + + // 2^8 * 8 * G + + .quad 0x644d58a649fe1e44 + .quad 0x21fcaea231ad777e + .quad 0x02441c5a887fd0d2 + .quad 0x4901aa7183c511f3 + .quad 0x3df2d29dc4244e45 + .quad 0x2b020e7493d8de0a + .quad 0x6cc8067e820c214d + .quad 0x413779166feab90a + .quad 0x08b1b7548c1af8f0 + .quad 0xce0f7a7c246299b4 + .quad 0xf760b0f91e06d939 + .quad 0x41bb887b726d1213 + + // 2^12 * 1 * G + + .quad 0x9267806c567c49d8 + .quad 0x066d04ccca791e6a + .quad 0xa69f5645e3cc394b + .quad 0x5c95b686a0788cd2 + .quad 0x97d980e0aa39f7d2 + .quad 0x35d0384252c6b51c + .quad 0x7d43f49307cd55aa + .quad 0x56bd36cfb78ac362 + .quad 0x2ac519c10d14a954 + .quad 0xeaf474b494b5fa90 + .quad 0xe6af8382a9f87a5a + .quad 0x0dea6db1879be094 + + // 2^12 * 2 * G + + .quad 0xaa66bf547344e5ab + .quad 0xda1258888f1b4309 + .quad 0x5e87d2b3fd564b2f + .quad 0x5b2c78885483b1dd + .quad 0x15baeb74d6a8797a + .quad 0x7ef55cf1fac41732 + .quad 0x29001f5a3c8b05c5 + .quad 0x0ad7cc8752eaccfb + .quad 0x52151362793408cf + .quad 0xeb0f170319963d94 + .quad 0xa833b2fa883d9466 + .quad 0x093a7fa775003c78 + + // 2^12 * 3 * G + + .quad 0xe5107de63a16d7be + .quad 0xa377ffdc9af332cf + .quad 0x70d5bf18440b677f + .quad 0x6a252b19a4a31403 + .quad 0xb8e9604460a91286 + .quad 0x7f3fd8047778d3de + .quad 0x67d01e31bf8a5e2d + .quad 0x7b038a06c27b653e + .quad 0x9ed919d5d36990f3 + .quad 0x5213aebbdb4eb9f2 + .quad 0xc708ea054cb99135 + .quad 0x58ded57f72260e56 + + // 2^12 * 4 * G + + .quad 0x78e79dade9413d77 + .quad 0xf257f9d59729e67d + .quad 0x59db910ee37aa7e6 + .quad 0x6aa11b5bbb9e039c + .quad 0xda6d53265b0fd48b + .quad 0x8960823193bfa988 + .quad 0xd78ac93261d57e28 + .quad 0x79f2942d3a5c8143 + .quad 0x97da2f25b6c88de9 + .quad 0x251ba7eaacf20169 + .quad 0x09b44f87ef4eb4e4 + .quad 0x7d90ab1bbc6a7da5 + + // 2^12 * 5 * G + + .quad 0x9acca683a7016bfe + .quad 0x90505f4df2c50b6d + .quad 0x6b610d5fcce435aa + .quad 0x19a10d446198ff96 + .quad 0x1a07a3f496b3c397 + .quad 0x11ceaa188f4e2532 + .quad 0x7d9498d5a7751bf0 + .quad 0x19ed161f508dd8a0 + .quad 0x560a2cd687dce6ca + .quad 0x7f3568c48664cf4d + .quad 0x8741e95222803a38 + .quad 0x483bdab1595653fc + + // 2^12 * 6 * G + + .quad 0xfa780f148734fa49 + .quad 0x106f0b70360534e0 + .quad 0x2210776fe3e307bd + .quad 0x3286c109dde6a0fe + .quad 0xd6cf4d0ab4da80f6 + .quad 0x82483e45f8307fe0 + .quad 0x05005269ae6f9da4 + .quad 0x1c7052909cf7877a + .quad 0x32ee7de2874e98d4 + .quad 0x14c362e9b97e0c60 + .quad 0x5781dcde6a60a38a + .quad 0x217dd5eaaa7aa840 + + // 2^12 * 7 * G + + .quad 0x9db7c4d0248e1eb0 + .quad 0xe07697e14d74bf52 + .quad 0x1e6a9b173c562354 + .quad 0x7fa7c21f795a4965 + .quad 0x8bdf1fb9be8c0ec8 + .quad 0x00bae7f8e30a0282 + .quad 0x4963991dad6c4f6c + .quad 0x07058a6e5df6f60a + .quad 0xe9eb02c4db31f67f + .quad 0xed25fd8910bcfb2b + .quad 0x46c8131f5c5cddb4 + .quad 0x33b21c13a0cb9bce + + // 2^12 * 8 * G + + .quad 0x360692f8087d8e31 + .quad 0xf4dcc637d27163f7 + .quad 0x25a4e62065ea5963 + .quad 0x659bf72e5ac160d9 + .quad 0x9aafb9b05ee38c5b + .quad 0xbf9d2d4e071a13c7 + .quad 0x8eee6e6de933290a + .quad 0x1c3bab17ae109717 + .quad 0x1c9ab216c7cab7b0 + .quad 0x7d65d37407bbc3cc + .quad 0x52744750504a58d5 + .quad 0x09f2606b131a2990 + + // 2^16 * 1 * G + + .quad 0x40e87d44744346be + .quad 0x1d48dad415b52b25 + .quad 0x7c3a8a18a13b603e + .quad 0x4eb728c12fcdbdf7 + .quad 0x7e234c597c6691ae + .quad 0x64889d3d0a85b4c8 + .quad 0xdae2c90c354afae7 + .quad 0x0a871e070c6a9e1d + .quad 0x3301b5994bbc8989 + .quad 0x736bae3a5bdd4260 + .quad 0x0d61ade219d59e3c + .quad 0x3ee7300f2685d464 + + // 2^16 * 2 * G + + .quad 0xf5d255e49e7dd6b7 + .quad 0x8016115c610b1eac + .quad 0x3c99975d92e187ca + .quad 0x13815762979125c2 + .quad 0x43fa7947841e7518 + .quad 0xe5c6fa59639c46d7 + .quad 0xa1065e1de3052b74 + .quad 0x7d47c6a2cfb89030 + .quad 0x3fdad0148ef0d6e0 + .quad 0x9d3e749a91546f3c + .quad 0x71ec621026bb8157 + .quad 0x148cf58d34c9ec80 + + // 2^16 * 3 * G + + .quad 0x46a492f67934f027 + .quad 0x469984bef6840aa9 + .quad 0x5ca1bc2a89611854 + .quad 0x3ff2fa1ebd5dbbd4 + .quad 0xe2572f7d9ae4756d + .quad 0x56c345bb88f3487f + .quad 0x9fd10b6d6960a88d + .quad 0x278febad4eaea1b9 + .quad 0xb1aa681f8c933966 + .quad 0x8c21949c20290c98 + .quad 0x39115291219d3c52 + .quad 0x4104dd02fe9c677b + + // 2^16 * 4 * G + + .quad 0x72b2bf5e1124422a + .quad 0xa1fa0c3398a33ab5 + .quad 0x94cb6101fa52b666 + .quad 0x2c863b00afaf53d5 + .quad 0x81214e06db096ab8 + .quad 0x21a8b6c90ce44f35 + .quad 0x6524c12a409e2af5 + .quad 0x0165b5a48efca481 + .quad 0xf190a474a0846a76 + .quad 0x12eff984cd2f7cc0 + .quad 0x695e290658aa2b8f + .quad 0x591b67d9bffec8b8 + + // 2^16 * 5 * G + + .quad 0x312f0d1c80b49bfa + .quad 0x5979515eabf3ec8a + .quad 0x727033c09ef01c88 + .quad 0x3de02ec7ca8f7bcb + .quad 0x99b9b3719f18b55d + .quad 0xe465e5faa18c641e + .quad 0x61081136c29f05ed + .quad 0x489b4f867030128b + .quad 0xd232102d3aeb92ef + .quad 0xe16253b46116a861 + .quad 0x3d7eabe7190baa24 + .quad 0x49f5fbba496cbebf + + // 2^16 * 6 * G + + .quad 0x30949a108a5bcfd4 + .quad 0xdc40dd70bc6473eb + .quad 0x92c294c1307c0d1c + .quad 0x5604a86dcbfa6e74 + .quad 0x155d628c1e9c572e + .quad 0x8a4d86acc5884741 + .quad 0x91a352f6515763eb + .quad 0x06a1a6c28867515b + .quad 0x7288d1d47c1764b6 + .quad 0x72541140e0418b51 + .quad 0x9f031a6018acf6d1 + .quad 0x20989e89fe2742c6 + + // 2^16 * 7 * G + + .quad 0x499777fd3a2dcc7f + .quad 0x32857c2ca54fd892 + .quad 0xa279d864d207e3a0 + .quad 0x0403ed1d0ca67e29 + .quad 0x1674278b85eaec2e + .quad 0x5621dc077acb2bdf + .quad 0x640a4c1661cbf45a + .quad 0x730b9950f70595d3 + .quad 0xc94b2d35874ec552 + .quad 0xc5e6c8cf98246f8d + .quad 0xf7cb46fa16c035ce + .quad 0x5bd7454308303dcc + + // 2^16 * 8 * G + + .quad 0x7f9ad19528b24cc2 + .quad 0x7f6b54656335c181 + .quad 0x66b8b66e4fc07236 + .quad 0x133a78007380ad83 + .quad 0x85c4932115e7792a + .quad 0xc64c89a2bdcdddc9 + .quad 0x9d1e3da8ada3d762 + .quad 0x5bb7db123067f82c + .quad 0x0961f467c6ca62be + .quad 0x04ec21d6211952ee + .quad 0x182360779bd54770 + .quad 0x740dca6d58f0e0d2 + + // 2^20 * 1 * G + + .quad 0x50b70bf5d3f0af0b + .quad 0x4feaf48ae32e71f7 + .quad 0x60e84ed3a55bbd34 + .quad 0x00ed489b3f50d1ed + .quad 0x3906c72aed261ae5 + .quad 0x9ab68fd988e100f7 + .quad 0xf5e9059af3360197 + .quad 0x0e53dc78bf2b6d47 + .quad 0xb90829bf7971877a + .quad 0x5e4444636d17e631 + .quad 0x4d05c52e18276893 + .quad 0x27632d9a5a4a4af5 + + // 2^20 * 2 * G + + .quad 0xd11ff05154b260ce + .quad 0xd86dc38e72f95270 + .quad 0x601fcd0d267cc138 + .quad 0x2b67916429e90ccd + .quad 0xa98285d187eaffdb + .quad 0xa5b4fbbbd8d0a864 + .quad 0xb658f27f022663f7 + .quad 0x3bbc2b22d99ce282 + .quad 0xb917c952583c0a58 + .quad 0x653ff9b80fe4c6f3 + .quad 0x9b0da7d7bcdf3c0c + .quad 0x43a0eeb6ab54d60e + + // 2^20 * 3 * G + + .quad 0x396966a46d4a5487 + .quad 0xf811a18aac2bb3ba + .quad 0x66e4685b5628b26b + .quad 0x70a477029d929b92 + .quad 0x3ac6322357875fe8 + .quad 0xd9d4f4ecf5fbcb8f + .quad 0x8dee8493382bb620 + .quad 0x50c5eaa14c799fdc + .quad 0xdd0edc8bd6f2fb3c + .quad 0x54c63aa79cc7b7a0 + .quad 0xae0b032b2c8d9f1a + .quad 0x6f9ce107602967fb + + // 2^20 * 4 * G + + .quad 0xad1054b1cde1c22a + .quad 0xc4a8e90248eb32df + .quad 0x5f3e7b33accdc0ea + .quad 0x72364713fc79963e + .quad 0x139693063520e0b5 + .quad 0x437fcf7c88ea03fe + .quad 0xf7d4c40bd3c959bc + .quad 0x699154d1f893ded9 + .quad 0x315d5c75b4b27526 + .quad 0xcccb842d0236daa5 + .quad 0x22f0c8a3345fee8e + .quad 0x73975a617d39dbed + + // 2^20 * 5 * G + + .quad 0xe4024df96375da10 + .quad 0x78d3251a1830c870 + .quad 0x902b1948658cd91c + .quad 0x7e18b10b29b7438a + .quad 0x6f37f392f4433e46 + .quad 0x0e19b9a11f566b18 + .quad 0x220fb78a1fd1d662 + .quad 0x362a4258a381c94d + .quad 0x9071d9132b6beb2f + .quad 0x0f26e9ad28418247 + .quad 0xeab91ec9bdec925d + .quad 0x4be65bc8f48af2de + + // 2^20 * 6 * G + + .quad 0x78487feba36e7028 + .quad 0x5f3f13001dd8ce34 + .quad 0x934fb12d4b30c489 + .quad 0x056c244d397f0a2b + .quad 0x1d50fba257c26234 + .quad 0x7bd4823adeb0678b + .quad 0xc2b0dc6ea6538af5 + .quad 0x5665eec6351da73e + .quad 0xdb3ee00943bfb210 + .quad 0x4972018720800ac2 + .quad 0x26ab5d6173bd8667 + .quad 0x20b209c2ab204938 + + // 2^20 * 7 * G + + .quad 0x549e342ac07fb34b + .quad 0x02d8220821373d93 + .quad 0xbc262d70acd1f567 + .quad 0x7a92c9fdfbcac784 + .quad 0x1fcca94516bd3289 + .quad 0x448d65aa41420428 + .quad 0x59c3b7b216a55d62 + .quad 0x49992cc64e612cd8 + .quad 0x65bd1bea70f801de + .quad 0x1befb7c0fe49e28a + .quad 0xa86306cdb1b2ae4a + .quad 0x3b7ac0cd265c2a09 + + // 2^20 * 8 * G + + .quad 0x822bee438c01bcec + .quad 0x530cb525c0fbc73b + .quad 0x48519034c1953fe9 + .quad 0x265cc261e09a0f5b + .quad 0xf0d54e4f22ed39a7 + .quad 0xa2aae91e5608150a + .quad 0xf421b2e9eddae875 + .quad 0x31bc531d6b7de992 + .quad 0xdf3d134da980f971 + .quad 0x7a4fb8d1221a22a7 + .quad 0x3df7d42035aad6d8 + .quad 0x2a14edcc6a1a125e + + // 2^24 * 1 * G + + .quad 0xdf48ee0752cfce4e + .quad 0xc3fffaf306ec08b7 + .quad 0x05710b2ab95459c4 + .quad 0x161d25fa963ea38d + .quad 0x231a8c570478433c + .quad 0xb7b5270ec281439d + .quad 0xdbaa99eae3d9079f + .quad 0x2c03f5256c2b03d9 + .quad 0x790f18757b53a47d + .quad 0x307b0130cf0c5879 + .quad 0x31903d77257ef7f9 + .quad 0x699468bdbd96bbaf + + // 2^24 * 2 * G + + .quad 0xbd1f2f46f4dafecf + .quad 0x7cef0114a47fd6f7 + .quad 0xd31ffdda4a47b37f + .quad 0x525219a473905785 + .quad 0xd8dd3de66aa91948 + .quad 0x485064c22fc0d2cc + .quad 0x9b48246634fdea2f + .quad 0x293e1c4e6c4a2e3a + .quad 0x376e134b925112e1 + .quad 0x703778b5dca15da0 + .quad 0xb04589af461c3111 + .quad 0x5b605c447f032823 + + // 2^24 * 3 * G + + .quad 0xb965805920c47c89 + .quad 0xe7f0100c923b8fcc + .quad 0x0001256502e2ef77 + .quad 0x24a76dcea8aeb3ee + .quad 0x3be9fec6f0e7f04c + .quad 0x866a579e75e34962 + .quad 0x5542ef161e1de61a + .quad 0x2f12fef4cc5abdd5 + .quad 0x0a4522b2dfc0c740 + .quad 0x10d06e7f40c9a407 + .quad 0xc6cf144178cff668 + .quad 0x5e607b2518a43790 + + // 2^24 * 4 * G + + .quad 0x58b31d8f6cdf1818 + .quad 0x35cfa74fc36258a2 + .quad 0xe1b3ff4f66e61d6e + .quad 0x5067acab6ccdd5f7 + .quad 0xa02c431ca596cf14 + .quad 0xe3c42d40aed3e400 + .quad 0xd24526802e0f26db + .quad 0x201f33139e457068 + .quad 0xfd527f6b08039d51 + .quad 0x18b14964017c0006 + .quad 0xd5220eb02e25a4a8 + .quad 0x397cba8862460375 + + // 2^24 * 5 * G + + .quad 0x30c13093f05959b2 + .quad 0xe23aa18de9a97976 + .quad 0x222fd491721d5e26 + .quad 0x2339d320766e6c3a + .quad 0x7815c3fbc81379e7 + .quad 0xa6619420dde12af1 + .quad 0xffa9c0f885a8fdd5 + .quad 0x771b4022c1e1c252 + .quad 0xd87dd986513a2fa7 + .quad 0xf5ac9b71f9d4cf08 + .quad 0xd06bc31b1ea283b3 + .quad 0x331a189219971a76 + + // 2^24 * 6 * G + + .quad 0xf5166f45fb4f80c6 + .quad 0x9c36c7de61c775cf + .quad 0xe3d4e81b9041d91c + .quad 0x31167c6b83bdfe21 + .quad 0x26512f3a9d7572af + .quad 0x5bcbe28868074a9e + .quad 0x84edc1c11180f7c4 + .quad 0x1ac9619ff649a67b + .quad 0xf22b3842524b1068 + .quad 0x5068343bee9ce987 + .quad 0xfc9d71844a6250c8 + .quad 0x612436341f08b111 + + // 2^24 * 7 * G + + .quad 0xd99d41db874e898d + .quad 0x09fea5f16c07dc20 + .quad 0x793d2c67d00f9bbc + .quad 0x46ebe2309e5eff40 + .quad 0x8b6349e31a2d2638 + .quad 0x9ddfb7009bd3fd35 + .quad 0x7f8bf1b8a3a06ba4 + .quad 0x1522aa3178d90445 + .quad 0x2c382f5369614938 + .quad 0xdafe409ab72d6d10 + .quad 0xe8c83391b646f227 + .quad 0x45fe70f50524306c + + // 2^24 * 8 * G + + .quad 0xda4875a6960c0b8c + .quad 0x5b68d076ef0e2f20 + .quad 0x07fb51cf3d0b8fd4 + .quad 0x428d1623a0e392d4 + .quad 0x62f24920c8951491 + .quad 0x05f007c83f630ca2 + .quad 0x6fbb45d2f5c9d4b8 + .quad 0x16619f6db57a2245 + .quad 0x084f4a4401a308fd + .quad 0xa82219c376a5caac + .quad 0xdeb8de4643d1bc7d + .quad 0x1d81592d60bd38c6 + + // 2^28 * 1 * G + + .quad 0xd833d7beec2a4c38 + .quad 0x2c9162830acc20ed + .quad 0xe93a47aa92df7581 + .quad 0x702d67a3333c4a81 + .quad 0x3a4a369a2f89c8a1 + .quad 0x63137a1d7c8de80d + .quad 0xbcac008a78eda015 + .quad 0x2cb8b3a5b483b03f + .quad 0x36e417cbcb1b90a1 + .quad 0x33b3ddaa7f11794e + .quad 0x3f510808885bc607 + .quad 0x24141dc0e6a8020d + + // 2^28 * 2 * G + + .quad 0x59f73c773fefee9d + .quad 0xb3f1ef89c1cf989d + .quad 0xe35dfb42e02e545f + .quad 0x5766120b47a1b47c + .quad 0x91925dccbd83157d + .quad 0x3ca1205322cc8094 + .quad 0x28e57f183f90d6e4 + .quad 0x1a4714cede2e767b + .quad 0xdb20ba0fb8b6b7ff + .quad 0xb732c3b677511fa1 + .quad 0xa92b51c099f02d89 + .quad 0x4f3875ad489ca5f1 + + // 2^28 * 3 * G + + .quad 0xc7fc762f4932ab22 + .quad 0x7ac0edf72f4c3c1b + .quad 0x5f6b55aa9aa895e8 + .quad 0x3680274dad0a0081 + .quad 0x79ed13f6ee73eec0 + .quad 0xa5c6526d69110bb1 + .quad 0xe48928c38603860c + .quad 0x722a1446fd7059f5 + .quad 0xd0959fe9a8cf8819 + .quad 0xd0a995508475a99c + .quad 0x6eac173320b09cc5 + .quad 0x628ecf04331b1095 + + // 2^28 * 4 * G + + .quad 0x98bcb118a9d0ddbc + .quad 0xee449e3408b4802b + .quad 0x87089226b8a6b104 + .quad 0x685f349a45c7915d + .quad 0x9b41acf85c74ccf1 + .quad 0xb673318108265251 + .quad 0x99c92aed11adb147 + .quad 0x7a47d70d34ecb40f + .quad 0x60a0c4cbcc43a4f5 + .quad 0x775c66ca3677bea9 + .quad 0xa17aa1752ff8f5ed + .quad 0x11ded9020e01fdc0 + + // 2^28 * 5 * G + + .quad 0x890e7809caefe704 + .quad 0x8728296de30e8c6c + .quad 0x4c5cd2a392aeb1c9 + .quad 0x194263d15771531f + .quad 0x471f95b03bea93b7 + .quad 0x0552d7d43313abd3 + .quad 0xbd9370e2e17e3f7b + .quad 0x7b120f1db20e5bec + .quad 0x17d2fb3d86502d7a + .quad 0xb564d84450a69352 + .quad 0x7da962c8a60ed75d + .quad 0x00d0f85b318736aa + + // 2^28 * 6 * G + + .quad 0x978b142e777c84fd + .quad 0xf402644705a8c062 + .quad 0xa67ad51be7e612c7 + .quad 0x2f7b459698dd6a33 + .quad 0xa6753c1efd7621c1 + .quad 0x69c0b4a7445671f5 + .quad 0x971f527405b23c11 + .quad 0x387bc74851a8c7cd + .quad 0x81894b4d4a52a9a8 + .quad 0xadd93e12f6b8832f + .quad 0x184d8548b61bd638 + .quad 0x3f1c62dbd6c9f6cd + + // 2^28 * 7 * G + + .quad 0x2e8f1f0091910c1f + .quad 0xa4df4fe0bff2e12c + .quad 0x60c6560aee927438 + .quad 0x6338283facefc8fa + .quad 0x3fad3e40148f693d + .quad 0x052656e194eb9a72 + .quad 0x2f4dcbfd184f4e2f + .quad 0x406f8db1c482e18b + .quad 0x9e630d2c7f191ee4 + .quad 0x4fbf8301bc3ff670 + .quad 0x787d8e4e7afb73c4 + .quad 0x50d83d5be8f58fa5 + + // 2^28 * 8 * G + + .quad 0x85683916c11a1897 + .quad 0x2d69a4efe506d008 + .quad 0x39af1378f664bd01 + .quad 0x65942131361517c6 + .quad 0xc0accf90b4d3b66d + .quad 0xa7059de561732e60 + .quad 0x033d1f7870c6b0ba + .quad 0x584161cd26d946e4 + .quad 0xbbf2b1a072d27ca2 + .quad 0xbf393c59fbdec704 + .quad 0xe98dbbcee262b81e + .quad 0x02eebd0b3029b589 + + // 2^32 * 1 * G + + .quad 0x61368756a60dac5f + .quad 0x17e02f6aebabdc57 + .quad 0x7f193f2d4cce0f7d + .quad 0x20234a7789ecdcf0 + .quad 0x8765b69f7b85c5e8 + .quad 0x6ff0678bd168bab2 + .quad 0x3a70e77c1d330f9b + .quad 0x3a5f6d51b0af8e7c + .quad 0x76d20db67178b252 + .quad 0x071c34f9d51ed160 + .quad 0xf62a4a20b3e41170 + .quad 0x7cd682353cffe366 + + // 2^32 * 2 * G + + .quad 0x0be1a45bd887fab6 + .quad 0x2a846a32ba403b6e + .quad 0xd9921012e96e6000 + .quad 0x2838c8863bdc0943 + .quad 0xa665cd6068acf4f3 + .quad 0x42d92d183cd7e3d3 + .quad 0x5759389d336025d9 + .quad 0x3ef0253b2b2cd8ff + .quad 0xd16bb0cf4a465030 + .quad 0xfa496b4115c577ab + .quad 0x82cfae8af4ab419d + .quad 0x21dcb8a606a82812 + + // 2^32 * 3 * G + + .quad 0x5c6004468c9d9fc8 + .quad 0x2540096ed42aa3cb + .quad 0x125b4d4c12ee2f9c + .quad 0x0bc3d08194a31dab + .quad 0x9a8d00fabe7731ba + .quad 0x8203607e629e1889 + .quad 0xb2cc023743f3d97f + .quad 0x5d840dbf6c6f678b + .quad 0x706e380d309fe18b + .quad 0x6eb02da6b9e165c7 + .quad 0x57bbba997dae20ab + .quad 0x3a4276232ac196dd + + // 2^32 * 4 * G + + .quad 0x4b42432c8a7084fa + .quad 0x898a19e3dfb9e545 + .quad 0xbe9f00219c58e45d + .quad 0x1ff177cea16debd1 + .quad 0x3bf8c172db447ecb + .quad 0x5fcfc41fc6282dbd + .quad 0x80acffc075aa15fe + .quad 0x0770c9e824e1a9f9 + .quad 0xcf61d99a45b5b5fd + .quad 0x860984e91b3a7924 + .quad 0xe7300919303e3e89 + .quad 0x39f264fd41500b1e + + // 2^32 * 5 * G + + .quad 0xa7ad3417dbe7e29c + .quad 0xbd94376a2b9c139c + .quad 0xa0e91b8e93597ba9 + .quad 0x1712d73468889840 + .quad 0xd19b4aabfe097be1 + .quad 0xa46dfce1dfe01929 + .quad 0xc3c908942ca6f1ff + .quad 0x65c621272c35f14e + .quad 0xe72b89f8ce3193dd + .quad 0x4d103356a125c0bb + .quad 0x0419a93d2e1cfe83 + .quad 0x22f9800ab19ce272 + + // 2^32 * 6 * G + + .quad 0x605a368a3e9ef8cb + .quad 0xe3e9c022a5504715 + .quad 0x553d48b05f24248f + .quad 0x13f416cd647626e5 + .quad 0x42029fdd9a6efdac + .quad 0xb912cebe34a54941 + .quad 0x640f64b987bdf37b + .quad 0x4171a4d38598cab4 + .quad 0xfa2758aa99c94c8c + .quad 0x23006f6fb000b807 + .quad 0xfbd291ddadda5392 + .quad 0x508214fa574bd1ab + + // 2^32 * 7 * G + + .quad 0xc20269153ed6fe4b + .quad 0xa65a6739511d77c4 + .quad 0xcbde26462c14af94 + .quad 0x22f960ec6faba74b + .quad 0x461a15bb53d003d6 + .quad 0xb2102888bcf3c965 + .quad 0x27c576756c683a5a + .quad 0x3a7758a4c86cb447 + .quad 0x548111f693ae5076 + .quad 0x1dae21df1dfd54a6 + .quad 0x12248c90f3115e65 + .quad 0x5d9fd15f8de7f494 + + // 2^32 * 8 * G + + .quad 0x031408d36d63727f + .quad 0x6a379aefd7c7b533 + .quad 0xa9e18fc5ccaee24b + .quad 0x332f35914f8fbed3 + .quad 0x3f244d2aeed7521e + .quad 0x8e3a9028432e9615 + .quad 0xe164ba772e9c16d4 + .quad 0x3bc187fa47eb98d8 + .quad 0x6d470115ea86c20c + .quad 0x998ab7cb6c46d125 + .quad 0xd77832b53a660188 + .quad 0x450d81ce906fba03 + + // 2^36 * 1 * G + + .quad 0xf8ae4d2ad8453902 + .quad 0x7018058ee8db2d1d + .quad 0xaab3995fc7d2c11e + .quad 0x53b16d2324ccca79 + .quad 0x23264d66b2cae0b5 + .quad 0x7dbaed33ebca6576 + .quad 0x030ebed6f0d24ac8 + .quad 0x2a887f78f7635510 + .quad 0x2a23b9e75c012d4f + .quad 0x0c974651cae1f2ea + .quad 0x2fb63273675d70ca + .quad 0x0ba7250b864403f5 + + // 2^36 * 2 * G + + .quad 0xbb0d18fd029c6421 + .quad 0xbc2d142189298f02 + .quad 0x8347f8e68b250e96 + .quad 0x7b9f2fe8032d71c9 + .quad 0xdd63589386f86d9c + .quad 0x61699176e13a85a4 + .quad 0x2e5111954eaa7d57 + .quad 0x32c21b57fb60bdfb + .quad 0xd87823cd319e0780 + .quad 0xefc4cfc1897775c5 + .quad 0x4854fb129a0ab3f7 + .quad 0x12c49d417238c371 + + // 2^36 * 3 * G + + .quad 0x0950b533ffe83769 + .quad 0x21861c1d8e1d6bd1 + .quad 0xf022d8381302e510 + .quad 0x2509200c6391cab4 + .quad 0x09b3a01783799542 + .quad 0x626dd08faad5ee3f + .quad 0xba00bceeeb70149f + .quad 0x1421b246a0a444c9 + .quad 0x4aa43a8e8c24a7c7 + .quad 0x04c1f540d8f05ef5 + .quad 0xadba5e0c0b3eb9dc + .quad 0x2ab5504448a49ce3 + + // 2^36 * 4 * G + + .quad 0x2ed227266f0f5dec + .quad 0x9824ee415ed50824 + .quad 0x807bec7c9468d415 + .quad 0x7093bae1b521e23f + .quad 0xdc07ac631c5d3afa + .quad 0x58615171f9df8c6c + .quad 0x72a079d89d73e2b0 + .quad 0x7301f4ceb4eae15d + .quad 0x6409e759d6722c41 + .quad 0xa674e1cf72bf729b + .quad 0xbc0a24eb3c21e569 + .quad 0x390167d24ebacb23 + + // 2^36 * 5 * G + + .quad 0x27f58e3bba353f1c + .quad 0x4c47764dbf6a4361 + .quad 0xafbbc4e56e562650 + .quad 0x07db2ee6aae1a45d + .quad 0xd7bb054ba2f2120b + .quad 0xe2b9ceaeb10589b7 + .quad 0x3fe8bac8f3c0edbe + .quad 0x4cbd40767112cb69 + .quad 0x0b603cc029c58176 + .quad 0x5988e3825cb15d61 + .quad 0x2bb61413dcf0ad8d + .quad 0x7b8eec6c74183287 + + // 2^36 * 6 * G + + .quad 0xe4ca40782cd27cb0 + .quad 0xdaf9c323fbe967bd + .quad 0xb29bd34a8ad41e9e + .quad 0x72810497626ede4d + .quad 0x32fee570fc386b73 + .quad 0xda8b0141da3a8cc7 + .quad 0x975ffd0ac8968359 + .quad 0x6ee809a1b132a855 + .quad 0x9444bb31fcfd863a + .quad 0x2fe3690a3e4e48c5 + .quad 0xdc29c867d088fa25 + .quad 0x13bd1e38d173292e + + // 2^36 * 7 * G + + .quad 0xd32b4cd8696149b5 + .quad 0xe55937d781d8aab7 + .quad 0x0bcb2127ae122b94 + .quad 0x41e86fcfb14099b0 + .quad 0x223fb5cf1dfac521 + .quad 0x325c25316f554450 + .quad 0x030b98d7659177ac + .quad 0x1ed018b64f88a4bd + .quad 0x3630dfa1b802a6b0 + .quad 0x880f874742ad3bd5 + .quad 0x0af90d6ceec5a4d4 + .quad 0x746a247a37cdc5d9 + + // 2^36 * 8 * G + + .quad 0xd531b8bd2b7b9af6 + .quad 0x5005093537fc5b51 + .quad 0x232fcf25c593546d + .quad 0x20a365142bb40f49 + .quad 0x6eccd85278d941ed + .quad 0x2254ae83d22f7843 + .quad 0xc522d02e7bbfcdb7 + .quad 0x681e3351bff0e4e2 + .quad 0x8b64b59d83034f45 + .quad 0x2f8b71f21fa20efb + .quad 0x69249495ba6550e4 + .quad 0x539ef98e45d5472b + + // 2^40 * 1 * G + + .quad 0x6e7bb6a1a6205275 + .quad 0xaa4f21d7413c8e83 + .quad 0x6f56d155e88f5cb2 + .quad 0x2de25d4ba6345be1 + .quad 0xd074d8961cae743f + .quad 0xf86d18f5ee1c63ed + .quad 0x97bdc55be7f4ed29 + .quad 0x4cbad279663ab108 + .quad 0x80d19024a0d71fcd + .quad 0xc525c20afb288af8 + .quad 0xb1a3974b5f3a6419 + .quad 0x7d7fbcefe2007233 + + // 2^40 * 2 * G + + .quad 0xfaef1e6a266b2801 + .quad 0x866c68c4d5739f16 + .quad 0xf68a2fbc1b03762c + .quad 0x5975435e87b75a8d + .quad 0xcd7c5dc5f3c29094 + .quad 0xc781a29a2a9105ab + .quad 0x80c61d36421c3058 + .quad 0x4f9cd196dcd8d4d7 + .quad 0x199297d86a7b3768 + .quad 0xd0d058241ad17a63 + .quad 0xba029cad5c1c0c17 + .quad 0x7ccdd084387a0307 + + // 2^40 * 3 * G + + .quad 0xdca6422c6d260417 + .quad 0xae153d50948240bd + .quad 0xa9c0c1b4fb68c677 + .quad 0x428bd0ed61d0cf53 + .quad 0x9b0c84186760cc93 + .quad 0xcdae007a1ab32a99 + .quad 0xa88dec86620bda18 + .quad 0x3593ca848190ca44 + .quad 0x9213189a5e849aa7 + .quad 0xd4d8c33565d8facd + .quad 0x8c52545b53fdbbd1 + .quad 0x27398308da2d63e6 + + // 2^40 * 4 * G + + .quad 0x42c38d28435ed413 + .quad 0xbd50f3603278ccc9 + .quad 0xbb07ab1a79da03ef + .quad 0x269597aebe8c3355 + .quad 0xb9a10e4c0a702453 + .quad 0x0fa25866d57d1bde + .quad 0xffb9d9b5cd27daf7 + .quad 0x572c2945492c33fd + .quad 0xc77fc745d6cd30be + .quad 0xe4dfe8d3e3baaefb + .quad 0xa22c8830aa5dda0c + .quad 0x7f985498c05bca80 + + // 2^40 * 5 * G + + .quad 0x3849ce889f0be117 + .quad 0x8005ad1b7b54a288 + .quad 0x3da3c39f23fc921c + .quad 0x76c2ec470a31f304 + .quad 0xd35615520fbf6363 + .quad 0x08045a45cf4dfba6 + .quad 0xeec24fbc873fa0c2 + .quad 0x30f2653cd69b12e7 + .quad 0x8a08c938aac10c85 + .quad 0x46179b60db276bcb + .quad 0xa920c01e0e6fac70 + .quad 0x2f1273f1596473da + + // 2^40 * 6 * G + + .quad 0x4739fc7c8ae01e11 + .quad 0xfd5274904a6aab9f + .quad 0x41d98a8287728f2e + .quad 0x5d9e572ad85b69f2 + .quad 0x30488bd755a70bc0 + .quad 0x06d6b5a4f1d442e7 + .quad 0xead1a69ebc596162 + .quad 0x38ac1997edc5f784 + .quad 0x0666b517a751b13b + .quad 0x747d06867e9b858c + .quad 0xacacc011454dde49 + .quad 0x22dfcd9cbfe9e69c + + // 2^40 * 7 * G + + .quad 0x8ddbd2e0c30d0cd9 + .quad 0xad8e665facbb4333 + .quad 0x8f6b258c322a961f + .quad 0x6b2916c05448c1c7 + .quad 0x56ec59b4103be0a1 + .quad 0x2ee3baecd259f969 + .quad 0x797cb29413f5cd32 + .quad 0x0fe9877824cde472 + .quad 0x7edb34d10aba913b + .quad 0x4ea3cd822e6dac0e + .quad 0x66083dff6578f815 + .quad 0x4c303f307ff00a17 + + // 2^40 * 8 * G + + .quad 0xd30a3bd617b28c85 + .quad 0xc5d377b739773bea + .quad 0xc6c6e78c1e6a5cbf + .quad 0x0d61b8f78b2ab7c4 + .quad 0x29fc03580dd94500 + .quad 0xecd27aa46fbbec93 + .quad 0x130a155fc2e2a7f8 + .quad 0x416b151ab706a1d5 + .quad 0x56a8d7efe9c136b0 + .quad 0xbd07e5cd58e44b20 + .quad 0xafe62fda1b57e0ab + .quad 0x191a2af74277e8d2 + + // 2^44 * 1 * G + + .quad 0xd550095bab6f4985 + .quad 0x04f4cd5b4fbfaf1a + .quad 0x9d8e2ed12a0c7540 + .quad 0x2bc24e04b2212286 + .quad 0x09d4b60b2fe09a14 + .quad 0xc384f0afdbb1747e + .quad 0x58e2ea8978b5fd6e + .quad 0x519ef577b5e09b0a + .quad 0x1863d7d91124cca9 + .quad 0x7ac08145b88a708e + .quad 0x2bcd7309857031f5 + .quad 0x62337a6e8ab8fae5 + + // 2^44 * 2 * G + + .quad 0x4bcef17f06ffca16 + .quad 0xde06e1db692ae16a + .quad 0x0753702d614f42b0 + .quad 0x5f6041b45b9212d0 + .quad 0xd1ab324e1b3a1273 + .quad 0x18947cf181055340 + .quad 0x3b5d9567a98c196e + .quad 0x7fa00425802e1e68 + .quad 0x7d531574028c2705 + .quad 0x80317d69db0d75fe + .quad 0x30fface8ef8c8ddd + .quad 0x7e9de97bb6c3e998 + + // 2^44 * 3 * G + + .quad 0x1558967b9e6585a3 + .quad 0x97c99ce098e98b92 + .quad 0x10af149b6eb3adad + .quad 0x42181fe8f4d38cfa + .quad 0xf004be62a24d40dd + .quad 0xba0659910452d41f + .quad 0x81c45ee162a44234 + .quad 0x4cb829d8a22266ef + .quad 0x1dbcaa8407b86681 + .quad 0x081f001e8b26753b + .quad 0x3cd7ce6a84048e81 + .quad 0x78af11633f25f22c + + // 2^44 * 4 * G + + .quad 0x8416ebd40b50babc + .quad 0x1508722628208bee + .quad 0xa3148fafb9c1c36d + .quad 0x0d07daacd32d7d5d + .quad 0x3241c00e7d65318c + .quad 0xe6bee5dcd0e86de7 + .quad 0x118b2dc2fbc08c26 + .quad 0x680d04a7fc603dc3 + .quad 0xf9c2414a695aa3eb + .quad 0xdaa42c4c05a68f21 + .quad 0x7c6c23987f93963e + .quad 0x210e8cd30c3954e3 + + // 2^44 * 5 * G + + .quad 0xac4201f210a71c06 + .quad 0x6a65e0aef3bfb021 + .quad 0xbc42c35c393632f7 + .quad 0x56ea8db1865f0742 + .quad 0x2b50f16137fe6c26 + .quad 0xe102bcd856e404d8 + .quad 0x12b0f1414c561f6b + .quad 0x51b17bc8d028ec91 + .quad 0xfff5fb4bcf535119 + .quad 0xf4989d79df1108a0 + .quad 0xbdfcea659a3ba325 + .quad 0x18a11f1174d1a6f2 + + // 2^44 * 6 * G + + .quad 0x407375ab3f6bba29 + .quad 0x9ec3b6d8991e482e + .quad 0x99c80e82e55f92e9 + .quad 0x307c13b6fb0c0ae1 + .quad 0xfbd63cdad27a5f2c + .quad 0xf00fc4bc8aa106d7 + .quad 0x53fb5c1a8e64a430 + .quad 0x04eaabe50c1a2e85 + .quad 0x24751021cb8ab5e7 + .quad 0xfc2344495c5010eb + .quad 0x5f1e717b4e5610a1 + .quad 0x44da5f18c2710cd5 + + // 2^44 * 7 * G + + .quad 0x033cc55ff1b82eb5 + .quad 0xb15ae36d411cae52 + .quad 0xba40b6198ffbacd3 + .quad 0x768edce1532e861f + .quad 0x9156fe6b89d8eacc + .quad 0xe6b79451e23126a1 + .quad 0xbd7463d93944eb4e + .quad 0x726373f6767203ae + .quad 0xe305ca72eb7ef68a + .quad 0x662cf31f70eadb23 + .quad 0x18f026fdb4c45b68 + .quad 0x513b5384b5d2ecbd + + // 2^44 * 8 * G + + .quad 0x46d46280c729989e + .quad 0x4b93fbd05368a5dd + .quad 0x63df3f81d1765a89 + .quad 0x34cebd64b9a0a223 + .quad 0x5e2702878af34ceb + .quad 0x900b0409b946d6ae + .quad 0x6512ebf7dabd8512 + .quad 0x61d9b76988258f81 + .quad 0xa6c5a71349b7d94b + .quad 0xa3f3d15823eb9446 + .quad 0x0416fbd277484834 + .quad 0x69d45e6f2c70812f + + // 2^48 * 1 * G + + .quad 0xce16f74bc53c1431 + .quad 0x2b9725ce2072edde + .quad 0xb8b9c36fb5b23ee7 + .quad 0x7e2e0e450b5cc908 + .quad 0x9fe62b434f460efb + .quad 0xded303d4a63607d6 + .quad 0xf052210eb7a0da24 + .quad 0x237e7dbe00545b93 + .quad 0x013575ed6701b430 + .quad 0x231094e69f0bfd10 + .quad 0x75320f1583e47f22 + .quad 0x71afa699b11155e3 + + // 2^48 * 2 * G + + .quad 0x65ce6f9b3953b61d + .quad 0xc65839eaafa141e6 + .quad 0x0f435ffda9f759fe + .quad 0x021142e9c2b1c28e + .quad 0xea423c1c473b50d6 + .quad 0x51e87a1f3b38ef10 + .quad 0x9b84bf5fb2c9be95 + .quad 0x00731fbc78f89a1c + .quad 0xe430c71848f81880 + .quad 0xbf960c225ecec119 + .quad 0xb6dae0836bba15e3 + .quad 0x4c4d6f3347e15808 + + // 2^48 * 3 * G + + .quad 0x18f7eccfc17d1fc9 + .quad 0x6c75f5a651403c14 + .quad 0xdbde712bf7ee0cdf + .quad 0x193fddaaa7e47a22 + .quad 0x2f0cddfc988f1970 + .quad 0x6b916227b0b9f51b + .quad 0x6ec7b6c4779176be + .quad 0x38bf9500a88f9fa8 + .quad 0x1fd2c93c37e8876f + .quad 0xa2f61e5a18d1462c + .quad 0x5080f58239241276 + .quad 0x6a6fb99ebf0d4969 + + // 2^48 * 4 * G + + .quad 0x6a46c1bb560855eb + .quad 0x2416bb38f893f09d + .quad 0xd71d11378f71acc1 + .quad 0x75f76914a31896ea + .quad 0xeeb122b5b6e423c6 + .quad 0x939d7010f286ff8e + .quad 0x90a92a831dcf5d8c + .quad 0x136fda9f42c5eb10 + .quad 0xf94cdfb1a305bdd1 + .quad 0x0f364b9d9ff82c08 + .quad 0x2a87d8a5c3bb588a + .quad 0x022183510be8dcba + + // 2^48 * 5 * G + + .quad 0x4af766385ead2d14 + .quad 0xa08ed880ca7c5830 + .quad 0x0d13a6e610211e3d + .quad 0x6a071ce17b806c03 + .quad 0x9d5a710143307a7f + .quad 0xb063de9ec47da45f + .quad 0x22bbfe52be927ad3 + .quad 0x1387c441fd40426c + .quad 0xb5d3c3d187978af8 + .quad 0x722b5a3d7f0e4413 + .quad 0x0d7b4848bb477ca0 + .quad 0x3171b26aaf1edc92 + + // 2^48 * 6 * G + + .quad 0xa92f319097564ca8 + .quad 0xff7bb84c2275e119 + .quad 0x4f55fe37a4875150 + .quad 0x221fd4873cf0835a + .quad 0xa60db7d8b28a47d1 + .quad 0xa6bf14d61770a4f1 + .quad 0xd4a1f89353ddbd58 + .quad 0x6c514a63344243e9 + .quad 0x2322204f3a156341 + .quad 0xfb73e0e9ba0a032d + .quad 0xfce0dd4c410f030e + .quad 0x48daa596fb924aaa + + // 2^48 * 7 * G + + .quad 0x6eca8e665ca59cc7 + .quad 0xa847254b2e38aca0 + .quad 0x31afc708d21e17ce + .quad 0x676dd6fccad84af7 + .quad 0x14f61d5dc84c9793 + .quad 0x9941f9e3ef418206 + .quad 0xcdf5b88f346277ac + .quad 0x58c837fa0e8a79a9 + .quad 0x0cf9688596fc9058 + .quad 0x1ddcbbf37b56a01b + .quad 0xdcc2e77d4935d66a + .quad 0x1c4f73f2c6a57f0a + + // 2^48 * 8 * G + + .quad 0x0e7a4fbd305fa0bb + .quad 0x829d4ce054c663ad + .quad 0xf421c3832fe33848 + .quad 0x795ac80d1bf64c42 + .quad 0xb36e706efc7c3484 + .quad 0x73dfc9b4c3c1cf61 + .quad 0xeb1d79c9781cc7e5 + .quad 0x70459adb7daf675c + .quad 0x1b91db4991b42bb3 + .quad 0x572696234b02dcca + .quad 0x9fdf9ee51f8c78dc + .quad 0x5fe162848ce21fd3 + + // 2^52 * 1 * G + + .quad 0xe2790aae4d077c41 + .quad 0x8b938270db7469a3 + .quad 0x6eb632dc8abd16a2 + .quad 0x720814ecaa064b72 + .quad 0x315c29c795115389 + .quad 0xd7e0e507862f74ce + .quad 0x0c4a762185927432 + .quad 0x72de6c984a25a1e4 + .quad 0xae9ab553bf6aa310 + .quad 0x050a50a9806d6e1b + .quad 0x92bb7403adff5139 + .quad 0x0394d27645be618b + + // 2^52 * 2 * G + + .quad 0x4d572251857eedf4 + .quad 0xe3724edde19e93c5 + .quad 0x8a71420e0b797035 + .quad 0x3b3c833687abe743 + .quad 0xf5396425b23545a4 + .quad 0x15a7a27e98fbb296 + .quad 0xab6c52bc636fdd86 + .quad 0x79d995a8419334ee + .quad 0xcd8a8ea61195dd75 + .quad 0xa504d8a81dd9a82f + .quad 0x540dca81a35879b6 + .quad 0x60dd16a379c86a8a + + // 2^52 * 3 * G + + .quad 0x35a2c8487381e559 + .quad 0x596ffea6d78082cb + .quad 0xcb9771ebdba7b653 + .quad 0x5a08b5019b4da685 + .quad 0x3501d6f8153e47b8 + .quad 0xb7a9675414a2f60c + .quad 0x112ee8b6455d9523 + .quad 0x4e62a3c18112ea8a + .quad 0xc8d4ac04516ab786 + .quad 0x595af3215295b23d + .quad 0xd6edd234db0230c1 + .quad 0x0929efe8825b41cc + + // 2^52 * 4 * G + + .quad 0x5f0601d1cbd0f2d3 + .quad 0x736e412f6132bb7f + .quad 0x83604432238dde87 + .quad 0x1e3a5272f5c0753c + .quad 0x8b3172b7ad56651d + .quad 0x01581b7a3fabd717 + .quad 0x2dc94df6424df6e4 + .quad 0x30376e5d2c29284f + .quad 0xd2918da78159a59c + .quad 0x6bdc1cd93f0713f3 + .quad 0x565f7a934acd6590 + .quad 0x53daacec4cb4c128 + + // 2^52 * 5 * G + + .quad 0x4ca73bd79cc8a7d6 + .quad 0x4d4a738f47e9a9b2 + .quad 0xf4cbf12942f5fe00 + .quad 0x01a13ff9bdbf0752 + .quad 0x99852bc3852cfdb0 + .quad 0x2cc12e9559d6ed0b + .quad 0x70f9e2bf9b5ac27b + .quad 0x4f3b8c117959ae99 + .quad 0x55b6c9c82ff26412 + .quad 0x1ac4a8c91fb667a8 + .quad 0xd527bfcfeb778bf2 + .quad 0x303337da7012a3be + + // 2^52 * 6 * G + + .quad 0x955422228c1c9d7c + .quad 0x01fac1371a9b340f + .quad 0x7e8d9177925b48d7 + .quad 0x53f8ad5661b3e31b + .quad 0x976d3ccbfad2fdd1 + .quad 0xcb88839737a640a8 + .quad 0x2ff00c1d6734cb25 + .quad 0x269ff4dc789c2d2b + .quad 0x0c003fbdc08d678d + .quad 0x4d982fa37ead2b17 + .quad 0xc07e6bcdb2e582f1 + .quad 0x296c7291df412a44 + + // 2^52 * 7 * G + + .quad 0x7903de2b33daf397 + .quad 0xd0ff0619c9a624b3 + .quad 0x8a1d252b555b3e18 + .quad 0x2b6d581c52e0b7c0 + .quad 0xdfb23205dab8b59e + .quad 0x465aeaa0c8092250 + .quad 0xd133c1189a725d18 + .quad 0x2327370261f117d1 + .quad 0x3d0543d3623e7986 + .quad 0x679414c2c278a354 + .quad 0xae43f0cc726196f6 + .quad 0x7836c41f8245eaba + + // 2^52 * 8 * G + + .quad 0xe7a254db49e95a81 + .quad 0x5192d5d008b0ad73 + .quad 0x4d20e5b1d00afc07 + .quad 0x5d55f8012cf25f38 + .quad 0xca651e848011937c + .quad 0xc6b0c46e6ef41a28 + .quad 0xb7021ba75f3f8d52 + .quad 0x119dff99ead7b9fd + .quad 0x43eadfcbf4b31d4d + .quad 0xc6503f7411148892 + .quad 0xfeee68c5060d3b17 + .quad 0x329293b3dd4a0ac8 + + // 2^56 * 1 * G + + .quad 0x4e59214fe194961a + .quad 0x49be7dc70d71cd4f + .quad 0x9300cfd23b50f22d + .quad 0x4789d446fc917232 + .quad 0x2879852d5d7cb208 + .quad 0xb8dedd70687df2e7 + .quad 0xdc0bffab21687891 + .quad 0x2b44c043677daa35 + .quad 0x1a1c87ab074eb78e + .quad 0xfac6d18e99daf467 + .quad 0x3eacbbcd484f9067 + .quad 0x60c52eef2bb9a4e4 + + // 2^56 * 2 * G + + .quad 0x0b5d89bc3bfd8bf1 + .quad 0xb06b9237c9f3551a + .quad 0x0e4c16b0d53028f5 + .quad 0x10bc9c312ccfcaab + .quad 0x702bc5c27cae6d11 + .quad 0x44c7699b54a48cab + .quad 0xefbc4056ba492eb2 + .quad 0x70d77248d9b6676d + .quad 0xaa8ae84b3ec2a05b + .quad 0x98699ef4ed1781e0 + .quad 0x794513e4708e85d1 + .quad 0x63755bd3a976f413 + + // 2^56 * 3 * G + + .quad 0xb55fa03e2ad10853 + .quad 0x356f75909ee63569 + .quad 0x9ff9f1fdbe69b890 + .quad 0x0d8cc1c48bc16f84 + .quad 0x3dc7101897f1acb7 + .quad 0x5dda7d5ec165bbd8 + .quad 0x508e5b9c0fa1020f + .quad 0x2763751737c52a56 + .quad 0x029402d36eb419a9 + .quad 0xf0b44e7e77b460a5 + .quad 0xcfa86230d43c4956 + .quad 0x70c2dd8a7ad166e7 + + // 2^56 * 4 * G + + .quad 0x656194509f6fec0e + .quad 0xee2e7ea946c6518d + .quad 0x9733c1f367e09b5c + .quad 0x2e0fac6363948495 + .quad 0x91d4967db8ed7e13 + .quad 0x74252f0ad776817a + .quad 0xe40982e00d852564 + .quad 0x32b8613816a53ce5 + .quad 0x79e7f7bee448cd64 + .quad 0x6ac83a67087886d0 + .quad 0xf89fd4d9a0e4db2e + .quad 0x4179215c735a4f41 + + // 2^56 * 5 * G + + .quad 0x8c7094e7d7dced2a + .quad 0x97fb8ac347d39c70 + .quad 0xe13be033a906d902 + .quad 0x700344a30cd99d76 + .quad 0xe4ae33b9286bcd34 + .quad 0xb7ef7eb6559dd6dc + .quad 0x278b141fb3d38e1f + .quad 0x31fa85662241c286 + .quad 0xaf826c422e3622f4 + .quad 0xc12029879833502d + .quad 0x9bc1b7e12b389123 + .quad 0x24bb2312a9952489 + + // 2^56 * 6 * G + + .quad 0xb1a8ed1732de67c3 + .quad 0x3cb49418461b4948 + .quad 0x8ebd434376cfbcd2 + .quad 0x0fee3e871e188008 + .quad 0x41f80c2af5f85c6b + .quad 0x687284c304fa6794 + .quad 0x8945df99a3ba1bad + .quad 0x0d1d2af9ffeb5d16 + .quad 0xa9da8aa132621edf + .quad 0x30b822a159226579 + .quad 0x4004197ba79ac193 + .quad 0x16acd79718531d76 + + // 2^56 * 7 * G + + .quad 0x72df72af2d9b1d3d + .quad 0x63462a36a432245a + .quad 0x3ecea07916b39637 + .quad 0x123e0ef6b9302309 + .quad 0xc959c6c57887b6ad + .quad 0x94e19ead5f90feba + .quad 0x16e24e62a342f504 + .quad 0x164ed34b18161700 + .quad 0x487ed94c192fe69a + .quad 0x61ae2cea3a911513 + .quad 0x877bf6d3b9a4de27 + .quad 0x78da0fc61073f3eb + + // 2^56 * 8 * G + + .quad 0x5bf15d28e52bc66a + .quad 0x2c47e31870f01a8e + .quad 0x2419afbc06c28bdd + .quad 0x2d25deeb256b173a + .quad 0xa29f80f1680c3a94 + .quad 0x71f77e151ae9e7e6 + .quad 0x1100f15848017973 + .quad 0x054aa4b316b38ddd + .quad 0xdfc8468d19267cb8 + .quad 0x0b28789c66e54daf + .quad 0x2aeb1d2a666eec17 + .quad 0x134610a6ab7da760 + + // 2^60 * 1 * G + + .quad 0xcaf55ec27c59b23f + .quad 0x99aeed3e154d04f2 + .quad 0x68441d72e14141f4 + .quad 0x140345133932a0a2 + .quad 0xd91430e0dc028c3c + .quad 0x0eb955a85217c771 + .quad 0x4b09e1ed2c99a1fa + .quad 0x42881af2bd6a743c + .quad 0x7bfec69aab5cad3d + .quad 0xc23e8cd34cb2cfad + .quad 0x685dd14bfb37d6a2 + .quad 0x0ad6d64415677a18 + + // 2^60 * 2 * G + + .quad 0x781a439e417becb5 + .quad 0x4ac5938cd10e0266 + .quad 0x5da385110692ac24 + .quad 0x11b065a2ade31233 + .quad 0x7914892847927e9f + .quad 0x33dad6ef370aa877 + .quad 0x1f8f24fa11122703 + .quad 0x5265ac2f2adf9592 + .quad 0x405fdd309afcb346 + .quad 0xd9723d4428e63f54 + .quad 0x94c01df05f65aaae + .quad 0x43e4dc3ae14c0809 + + // 2^60 * 3 * G + + .quad 0xbc12c7f1a938a517 + .quad 0x473028ab3180b2e1 + .quad 0x3f78571efbcd254a + .quad 0x74e534426ff6f90f + .quad 0xea6f7ac3adc2c6a3 + .quad 0xd0e928f6e9717c94 + .quad 0xe2d379ead645eaf5 + .quad 0x46dd8785c51ffbbe + .quad 0x709801be375c8898 + .quad 0x4b06dab5e3fd8348 + .quad 0x75880ced27230714 + .quad 0x2b09468fdd2f4c42 + + // 2^60 * 4 * G + + .quad 0x97c749eeb701cb96 + .quad 0x83f438d4b6a369c3 + .quad 0x62962b8b9a402cd9 + .quad 0x6976c7509888df7b + .quad 0x5b97946582ffa02a + .quad 0xda096a51fea8f549 + .quad 0xa06351375f77af9b + .quad 0x1bcfde61201d1e76 + .quad 0x4a4a5490246a59a2 + .quad 0xd63ebddee87fdd90 + .quad 0xd9437c670d2371fa + .quad 0x69e87308d30f8ed6 + + // 2^60 * 5 * G + + .quad 0x435a8bb15656beb0 + .quad 0xf8fac9ba4f4d5bca + .quad 0xb9b278c41548c075 + .quad 0x3eb0ef76e892b622 + .quad 0x0f80bf028bc80303 + .quad 0x6aae16b37a18cefb + .quad 0xdd47ea47d72cd6a3 + .quad 0x61943588f4ed39aa + .quad 0xd26e5c3e91039f85 + .quad 0xc0e9e77df6f33aa9 + .quad 0xe8968c5570066a93 + .quad 0x3c34d1881faaaddd + + // 2^60 * 6 * G + + .quad 0x3f9d2b5ea09f9ec0 + .quad 0x1dab3b6fb623a890 + .quad 0xa09ba3ea72d926c4 + .quad 0x374193513fd8b36d + .quad 0xbd5b0b8f2fffe0d9 + .quad 0x6aa254103ed24fb9 + .quad 0x2ac7d7bcb26821c4 + .quad 0x605b394b60dca36a + .quad 0xb4e856e45a9d1ed2 + .quad 0xefe848766c97a9a2 + .quad 0xb104cf641e5eee7d + .quad 0x2f50b81c88a71c8f + + // 2^60 * 7 * G + + .quad 0x31723c61fc6811bb + .quad 0x9cb450486211800f + .quad 0x768933d347995753 + .quad 0x3491a53502752fcd + .quad 0x2b552ca0a7da522a + .quad 0x3230b336449b0250 + .quad 0xf2c4c5bca4b99fb9 + .quad 0x7b2c674958074a22 + .quad 0xd55165883ed28cdf + .quad 0x12d84fd2d362de39 + .quad 0x0a874ad3e3378e4f + .quad 0x000d2b1f7c763e74 + + // 2^60 * 8 * G + + .quad 0x3d420811d06d4a67 + .quad 0xbefc048590e0ffe3 + .quad 0xf870c6b7bd487bde + .quad 0x6e2a7316319afa28 + .quad 0x9624778c3e94a8ab + .quad 0x0ad6f3cee9a78bec + .quad 0x948ac7810d743c4f + .quad 0x76627935aaecfccc + .quad 0x56a8ac24d6d59a9f + .quad 0xc8db753e3096f006 + .quad 0x477f41e68f4c5299 + .quad 0x588d851cf6c86114 + + // 2^64 * 1 * G + + .quad 0x51138ec78df6b0fe + .quad 0x5397da89e575f51b + .quad 0x09207a1d717af1b9 + .quad 0x2102fdba2b20d650 + .quad 0xcd2a65e777d1f515 + .quad 0x548991878faa60f1 + .quad 0xb1b73bbcdabc06e5 + .quad 0x654878cba97cc9fb + .quad 0x969ee405055ce6a1 + .quad 0x36bca7681251ad29 + .quad 0x3a1af517aa7da415 + .quad 0x0ad725db29ecb2ba + + // 2^64 * 2 * G + + .quad 0xdc4267b1834e2457 + .quad 0xb67544b570ce1bc5 + .quad 0x1af07a0bf7d15ed7 + .quad 0x4aefcffb71a03650 + .quad 0xfec7bc0c9b056f85 + .quad 0x537d5268e7f5ffd7 + .quad 0x77afc6624312aefa + .quad 0x4f675f5302399fd9 + .quad 0xc32d36360415171e + .quad 0xcd2bef118998483b + .quad 0x870a6eadd0945110 + .quad 0x0bccbb72a2a86561 + + // 2^64 * 3 * G + + .quad 0x185e962feab1a9c8 + .quad 0x86e7e63565147dcd + .quad 0xb092e031bb5b6df2 + .quad 0x4024f0ab59d6b73e + .quad 0x186d5e4c50fe1296 + .quad 0xe0397b82fee89f7e + .quad 0x3bc7f6c5507031b0 + .quad 0x6678fd69108f37c2 + .quad 0x1586fa31636863c2 + .quad 0x07f68c48572d33f2 + .quad 0x4f73cc9f789eaefc + .quad 0x2d42e2108ead4701 + + // 2^64 * 4 * G + + .quad 0x97f5131594dfd29b + .quad 0x6155985d313f4c6a + .quad 0xeba13f0708455010 + .quad 0x676b2608b8d2d322 + .quad 0x21717b0d0f537593 + .quad 0x914e690b131e064c + .quad 0x1bb687ae752ae09f + .quad 0x420bf3a79b423c6e + .quad 0x8138ba651c5b2b47 + .quad 0x8671b6ec311b1b80 + .quad 0x7bff0cb1bc3135b0 + .quad 0x745d2ffa9c0cf1e0 + + // 2^64 * 5 * G + + .quad 0xbf525a1e2bc9c8bd + .quad 0xea5b260826479d81 + .quad 0xd511c70edf0155db + .quad 0x1ae23ceb960cf5d0 + .quad 0x6036df5721d34e6a + .quad 0xb1db8827997bb3d0 + .quad 0xd3c209c3c8756afa + .quad 0x06e15be54c1dc839 + .quad 0x5b725d871932994a + .quad 0x32351cb5ceb1dab0 + .quad 0x7dc41549dab7ca05 + .quad 0x58ded861278ec1f7 + + // 2^64 * 6 * G + + .quad 0xd8173793f266c55c + .quad 0xc8c976c5cc454e49 + .quad 0x5ce382f8bc26c3a8 + .quad 0x2ff39de85485f6f9 + .quad 0x2dfb5ba8b6c2c9a8 + .quad 0x48eeef8ef52c598c + .quad 0x33809107f12d1573 + .quad 0x08ba696b531d5bd8 + .quad 0x77ed3eeec3efc57a + .quad 0x04e05517d4ff4811 + .quad 0xea3d7a3ff1a671cb + .quad 0x120633b4947cfe54 + + // 2^64 * 7 * G + + .quad 0x0b94987891610042 + .quad 0x4ee7b13cecebfae8 + .quad 0x70be739594f0a4c0 + .quad 0x35d30a99b4d59185 + .quad 0x82bd31474912100a + .quad 0xde237b6d7e6fbe06 + .quad 0xe11e761911ea79c6 + .quad 0x07433be3cb393bde + .quad 0xff7944c05ce997f4 + .quad 0x575d3de4b05c51a3 + .quad 0x583381fd5a76847c + .quad 0x2d873ede7af6da9f + + // 2^64 * 8 * G + + .quad 0x157a316443373409 + .quad 0xfab8b7eef4aa81d9 + .quad 0xb093fee6f5a64806 + .quad 0x2e773654707fa7b6 + .quad 0xaa6202e14e5df981 + .quad 0xa20d59175015e1f5 + .quad 0x18a275d3bae21d6c + .quad 0x0543618a01600253 + .quad 0x0deabdf4974c23c1 + .quad 0xaa6f0a259dce4693 + .quad 0x04202cb8a29aba2c + .quad 0x4b1443362d07960d + + // 2^68 * 1 * G + + .quad 0x47b837f753242cec + .quad 0x256dc48cc04212f2 + .quad 0xe222fbfbe1d928c5 + .quad 0x48ea295bad8a2c07 + .quad 0x299b1c3f57c5715e + .quad 0x96cb929e6b686d90 + .quad 0x3004806447235ab3 + .quad 0x2c435c24a44d9fe1 + .quad 0x0607c97c80f8833f + .quad 0x0e851578ca25ec5b + .quad 0x54f7450b161ebb6f + .quad 0x7bcb4792a0def80e + + // 2^68 * 2 * G + + .quad 0x8487e3d02bc73659 + .quad 0x4baf8445059979df + .quad 0xd17c975adcad6fbf + .quad 0x57369f0bdefc96b6 + .quad 0x1cecd0a0045224c2 + .quad 0x757f1b1b69e53952 + .quad 0x775b7a925289f681 + .quad 0x1b6cc62016736148 + .quad 0xf1a9990175638698 + .quad 0x353dd1beeeaa60d3 + .quad 0x849471334c9ba488 + .quad 0x63fa6e6843ade311 + + // 2^68 * 3 * G + + .quad 0xd15c20536597c168 + .quad 0x9f73740098d28789 + .quad 0x18aee7f13257ba1f + .quad 0x3418bfda07346f14 + .quad 0x2195becdd24b5eb7 + .quad 0x5e41f18cc0cd44f9 + .quad 0xdf28074441ca9ede + .quad 0x07073b98f35b7d67 + .quad 0xd03c676c4ce530d4 + .quad 0x0b64c0473b5df9f4 + .quad 0x065cef8b19b3a31e + .quad 0x3084d661533102c9 + + // 2^68 * 4 * G + + .quad 0xe1f6b79ebf8469ad + .quad 0x15801004e2663135 + .quad 0x9a498330af74181b + .quad 0x3ba2504f049b673c + .quad 0x9a6ce876760321fd + .quad 0x7fe2b5109eb63ad8 + .quad 0x00e7d4ae8ac80592 + .quad 0x73d86b7abb6f723a + .quad 0x0b52b5606dba5ab6 + .quad 0xa9134f0fbbb1edab + .quad 0x30a9520d9b04a635 + .quad 0x6813b8f37973e5db + + // 2^68 * 5 * G + + .quad 0x9854b054334127c1 + .quad 0x105d047882fbff25 + .quad 0xdb49f7f944186f4f + .quad 0x1768e838bed0b900 + .quad 0xf194ca56f3157e29 + .quad 0x136d35705ef528a5 + .quad 0xdd4cef778b0599bc + .quad 0x7d5472af24f833ed + .quad 0xd0ef874daf33da47 + .quad 0x00d3be5db6e339f9 + .quad 0x3f2a8a2f9c9ceece + .quad 0x5d1aeb792352435a + + // 2^68 * 6 * G + + .quad 0xf59e6bb319cd63ca + .quad 0x670c159221d06839 + .quad 0xb06d565b2150cab6 + .quad 0x20fb199d104f12a3 + .quad 0x12c7bfaeb61ba775 + .quad 0xb84e621fe263bffd + .quad 0x0b47a5c35c840dcf + .quad 0x7e83be0bccaf8634 + .quad 0x61943dee6d99c120 + .quad 0x86101f2e460b9fe0 + .quad 0x6bb2f1518ee8598d + .quad 0x76b76289fcc475cc + + // 2^68 * 7 * G + + .quad 0x791b4cc1756286fa + .quad 0xdbced317d74a157c + .quad 0x7e732421ea72bde6 + .quad 0x01fe18491131c8e9 + .quad 0x4245f1a1522ec0b3 + .quad 0x558785b22a75656d + .quad 0x1d485a2548a1b3c0 + .quad 0x60959eccd58fe09f + .quad 0x3ebfeb7ba8ed7a09 + .quad 0x49fdc2bbe502789c + .quad 0x44ebce5d3c119428 + .quad 0x35e1eb55be947f4a + + // 2^68 * 8 * G + + .quad 0xdbdae701c5738dd3 + .quad 0xf9c6f635b26f1bee + .quad 0x61e96a8042f15ef4 + .quad 0x3aa1d11faf60a4d8 + .quad 0x14fd6dfa726ccc74 + .quad 0x3b084cfe2f53b965 + .quad 0xf33ae4f552a2c8b4 + .quad 0x59aab07a0d40166a + .quad 0x77bcec4c925eac25 + .quad 0x1848718460137738 + .quad 0x5b374337fea9f451 + .quad 0x1865e78ec8e6aa46 + + // 2^72 * 1 * G + + .quad 0xccc4b7c7b66e1f7a + .quad 0x44157e25f50c2f7e + .quad 0x3ef06dfc713eaf1c + .quad 0x582f446752da63f7 + .quad 0x967c54e91c529ccb + .quad 0x30f6269264c635fb + .quad 0x2747aff478121965 + .quad 0x17038418eaf66f5c + .quad 0xc6317bd320324ce4 + .quad 0xa81042e8a4488bc4 + .quad 0xb21ef18b4e5a1364 + .quad 0x0c2a1c4bcda28dc9 + + // 2^72 * 2 * G + + .quad 0xd24dc7d06f1f0447 + .quad 0xb2269e3edb87c059 + .quad 0xd15b0272fbb2d28f + .quad 0x7c558bd1c6f64877 + .quad 0xedc4814869bd6945 + .quad 0x0d6d907dbe1c8d22 + .quad 0xc63bd212d55cc5ab + .quad 0x5a6a9b30a314dc83 + .quad 0xd0ec1524d396463d + .quad 0x12bb628ac35a24f0 + .quad 0xa50c3a791cbc5fa4 + .quad 0x0404a5ca0afbafc3 + + // 2^72 * 3 * G + + .quad 0x8c1f40070aa743d6 + .quad 0xccbad0cb5b265ee8 + .quad 0x574b046b668fd2de + .quad 0x46395bfdcadd9633 + .quad 0x62bc9e1b2a416fd1 + .quad 0xb5c6f728e350598b + .quad 0x04343fd83d5d6967 + .quad 0x39527516e7f8ee98 + .quad 0x117fdb2d1a5d9a9c + .quad 0x9c7745bcd1005c2a + .quad 0xefd4bef154d56fea + .quad 0x76579a29e822d016 + + // 2^72 * 4 * G + + .quad 0x45b68e7e49c02a17 + .quad 0x23cd51a2bca9a37f + .quad 0x3ed65f11ec224c1b + .quad 0x43a384dc9e05bdb1 + .quad 0x333cb51352b434f2 + .quad 0xd832284993de80e1 + .quad 0xb5512887750d35ce + .quad 0x02c514bb2a2777c1 + .quad 0x684bd5da8bf1b645 + .quad 0xfb8bd37ef6b54b53 + .quad 0x313916d7a9b0d253 + .quad 0x1160920961548059 + + // 2^72 * 5 * G + + .quad 0xb44d166929dacfaa + .quad 0xda529f4c8413598f + .quad 0xe9ef63ca453d5559 + .quad 0x351e125bc5698e0b + .quad 0x7a385616369b4dcd + .quad 0x75c02ca7655c3563 + .quad 0x7dc21bf9d4f18021 + .quad 0x2f637d7491e6e042 + .quad 0xd4b49b461af67bbe + .quad 0xd603037ac8ab8961 + .quad 0x71dee19ff9a699fb + .quad 0x7f182d06e7ce2a9a + + // 2^72 * 6 * G + + .quad 0x7a7c8e64ab0168ec + .quad 0xcb5a4a5515edc543 + .quad 0x095519d347cd0eda + .quad 0x67d4ac8c343e93b0 + .quad 0x09454b728e217522 + .quad 0xaa58e8f4d484b8d8 + .quad 0xd358254d7f46903c + .quad 0x44acc043241c5217 + .quad 0x1c7d6bbb4f7a5777 + .quad 0x8b35fed4918313e1 + .quad 0x4adca1c6c96b4684 + .quad 0x556d1c8312ad71bd + + // 2^72 * 7 * G + + .quad 0x17ef40e30c8d3982 + .quad 0x31f7073e15a3fa34 + .quad 0x4f21f3cb0773646e + .quad 0x746c6c6d1d824eff + .quad 0x81f06756b11be821 + .quad 0x0faff82310a3f3dd + .quad 0xf8b2d0556a99465d + .quad 0x097abe38cc8c7f05 + .quad 0x0c49c9877ea52da4 + .quad 0x4c4369559bdc1d43 + .quad 0x022c3809f7ccebd2 + .quad 0x577e14a34bee84bd + + // 2^72 * 8 * G + + .quad 0xf0e268ac61a73b0a + .quad 0xf2fafa103791a5f5 + .quad 0xc1e13e826b6d00e9 + .quad 0x60fa7ee96fd78f42 + .quad 0x94fecebebd4dd72b + .quad 0xf46a4fda060f2211 + .quad 0x124a5977c0c8d1ff + .quad 0x705304b8fb009295 + .quad 0xb63d1d354d296ec6 + .quad 0xf3c3053e5fad31d8 + .quad 0x670b958cb4bd42ec + .quad 0x21398e0ca16353fd + + // 2^76 * 1 * G + + .quad 0x216ab2ca8da7d2ef + .quad 0x366ad9dd99f42827 + .quad 0xae64b9004fdd3c75 + .quad 0x403a395b53909e62 + .quad 0x86c5fc16861b7e9a + .quad 0xf6a330476a27c451 + .quad 0x01667267a1e93597 + .quad 0x05ffb9cd6082dfeb + .quad 0xa617fa9ff53f6139 + .quad 0x60f2b5e513e66cb6 + .quad 0xd7a8beefb3448aa4 + .quad 0x7a2932856f5ea192 + + // 2^76 * 2 * G + + .quad 0x0b39d761b02de888 + .quad 0x5f550e7ed2414e1f + .quad 0xa6bfa45822e1a940 + .quad 0x050a2f7dfd447b99 + .quad 0xb89c444879639302 + .quad 0x4ae4f19350c67f2c + .quad 0xf0b35da8c81af9c6 + .quad 0x39d0003546871017 + .quad 0x437c3b33a650db77 + .quad 0x6bafe81dbac52bb2 + .quad 0xfe99402d2db7d318 + .quad 0x2b5b7eec372ba6ce + + // 2^76 * 3 * G + + .quad 0xb3bc4bbd83f50eef + .quad 0x508f0c998c927866 + .quad 0x43e76587c8b7e66e + .quad 0x0f7655a3a47f98d9 + .quad 0xa694404d613ac8f4 + .quad 0x500c3c2bfa97e72c + .quad 0x874104d21fcec210 + .quad 0x1b205fb38604a8ee + .quad 0x55ecad37d24b133c + .quad 0x441e147d6038c90b + .quad 0x656683a1d62c6fee + .quad 0x0157d5dc87e0ecae + + // 2^76 * 4 * G + + .quad 0xf2a7af510354c13d + .quad 0xd7a0b145aa372b60 + .quad 0x2869b96a05a3d470 + .quad 0x6528e42d82460173 + .quad 0x95265514d71eb524 + .quad 0xe603d8815df14593 + .quad 0x147cdf410d4de6b7 + .quad 0x5293b1730437c850 + .quad 0x23d0e0814bccf226 + .quad 0x92c745cd8196fb93 + .quad 0x8b61796c59541e5b + .quad 0x40a44df0c021f978 + + // 2^76 * 5 * G + + .quad 0xdaa869894f20ea6a + .quad 0xea14a3d14c620618 + .quad 0x6001fccb090bf8be + .quad 0x35f4e822947e9cf0 + .quad 0x86c96e514bc5d095 + .quad 0xf20d4098fca6804a + .quad 0x27363d89c826ea5d + .quad 0x39ca36565719cacf + .quad 0x97506f2f6f87b75c + .quad 0xc624aea0034ae070 + .quad 0x1ec856e3aad34dd6 + .quad 0x055b0be0e440e58f + + // 2^76 * 6 * G + + .quad 0x6469a17d89735d12 + .quad 0xdb6f27d5e662b9f1 + .quad 0x9fcba3286a395681 + .quad 0x363b8004d269af25 + .quad 0x4d12a04b6ea33da2 + .quad 0x57cf4c15e36126dd + .quad 0x90ec9675ee44d967 + .quad 0x64ca348d2a985aac + .quad 0x99588e19e4c4912d + .quad 0xefcc3b4e1ca5ce6b + .quad 0x4522ea60fa5b98d5 + .quad 0x7064bbab1de4a819 + + // 2^76 * 7 * G + + .quad 0xb919e1515a770641 + .quad 0xa9a2e2c74e7f8039 + .quad 0x7527250b3df23109 + .quad 0x756a7330ac27b78b + .quad 0xa290c06142542129 + .quad 0xf2e2c2aebe8d5b90 + .quad 0xcf2458db76abfe1b + .quad 0x02157ade83d626bf + .quad 0x3e46972a1b9a038b + .quad 0x2e4ee66a7ee03fb4 + .quad 0x81a248776edbb4ca + .quad 0x1a944ee88ecd0563 + + // 2^76 * 8 * G + + .quad 0xd5a91d1151039372 + .quad 0x2ed377b799ca26de + .quad 0xa17202acfd366b6b + .quad 0x0730291bd6901995 + .quad 0xbb40a859182362d6 + .quad 0xb99f55778a4d1abb + .quad 0x8d18b427758559f6 + .quad 0x26c20fe74d26235a + .quad 0x648d1d9fe9cc22f5 + .quad 0x66bc561928dd577c + .quad 0x47d3ed21652439d1 + .quad 0x49d271acedaf8b49 + + // 2^80 * 1 * G + + .quad 0x89f5058a382b33f3 + .quad 0x5ae2ba0bad48c0b4 + .quad 0x8f93b503a53db36e + .quad 0x5aa3ed9d95a232e6 + .quad 0x2798aaf9b4b75601 + .quad 0x5eac72135c8dad72 + .quad 0xd2ceaa6161b7a023 + .quad 0x1bbfb284e98f7d4e + .quad 0x656777e9c7d96561 + .quad 0xcb2b125472c78036 + .quad 0x65053299d9506eee + .quad 0x4a07e14e5e8957cc + + // 2^80 * 2 * G + + .quad 0x4ee412cb980df999 + .quad 0xa315d76f3c6ec771 + .quad 0xbba5edde925c77fd + .quad 0x3f0bac391d313402 + .quad 0x240b58cdc477a49b + .quad 0xfd38dade6447f017 + .quad 0x19928d32a7c86aad + .quad 0x50af7aed84afa081 + .quad 0x6e4fde0115f65be5 + .quad 0x29982621216109b2 + .quad 0x780205810badd6d9 + .quad 0x1921a316baebd006 + + // 2^80 * 3 * G + + .quad 0x89422f7edfb870fc + .quad 0x2c296beb4f76b3bd + .quad 0x0738f1d436c24df7 + .quad 0x6458df41e273aeb0 + .quad 0xd75aad9ad9f3c18b + .quad 0x566a0eef60b1c19c + .quad 0x3e9a0bac255c0ed9 + .quad 0x7b049deca062c7f5 + .quad 0xdccbe37a35444483 + .quad 0x758879330fedbe93 + .quad 0x786004c312c5dd87 + .quad 0x6093dccbc2950e64 + + // 2^80 * 4 * G + + .quad 0x1ff39a8585e0706d + .quad 0x36d0a5d8b3e73933 + .quad 0x43b9f2e1718f453b + .quad 0x57d1ea084827a97c + .quad 0x6bdeeebe6084034b + .quad 0x3199c2b6780fb854 + .quad 0x973376abb62d0695 + .quad 0x6e3180c98b647d90 + .quad 0xee7ab6e7a128b071 + .quad 0xa4c1596d93a88baa + .quad 0xf7b4de82b2216130 + .quad 0x363e999ddd97bd18 + + // 2^80 * 5 * G + + .quad 0x96a843c135ee1fc4 + .quad 0x976eb35508e4c8cf + .quad 0xb42f6801b58cd330 + .quad 0x48ee9b78693a052b + .quad 0x2f1848dce24baec6 + .quad 0x769b7255babcaf60 + .quad 0x90cb3c6e3cefe931 + .quad 0x231f979bc6f9b355 + .quad 0x5c31de4bcc2af3c6 + .quad 0xb04bb030fe208d1f + .quad 0xb78d7009c14fb466 + .quad 0x079bfa9b08792413 + + // 2^80 * 6 * G + + .quad 0xe3903a51da300df4 + .quad 0x843964233da95ab0 + .quad 0xed3cf12d0b356480 + .quad 0x038c77f684817194 + .quad 0xf3c9ed80a2d54245 + .quad 0x0aa08b7877f63952 + .quad 0xd76dac63d1085475 + .quad 0x1ef4fb159470636b + .quad 0x854e5ee65b167bec + .quad 0x59590a4296d0cdc2 + .quad 0x72b2df3498102199 + .quad 0x575ee92a4a0bff56 + + // 2^80 * 7 * G + + .quad 0xd4c080908a182fcf + .quad 0x30e170c299489dbd + .quad 0x05babd5752f733de + .quad 0x43d4e7112cd3fd00 + .quad 0x5d46bc450aa4d801 + .quad 0xc3af1227a533b9d8 + .quad 0x389e3b262b8906c2 + .quad 0x200a1e7e382f581b + .quad 0x518db967eaf93ac5 + .quad 0x71bc989b056652c0 + .quad 0xfe2b85d9567197f5 + .quad 0x050eca52651e4e38 + + // 2^80 * 8 * G + + .quad 0xc3431ade453f0c9c + .quad 0xe9f5045eff703b9b + .quad 0xfcd97ac9ed847b3d + .quad 0x4b0ee6c21c58f4c6 + .quad 0x97ac397660e668ea + .quad 0x9b19bbfe153ab497 + .quad 0x4cb179b534eca79f + .quad 0x6151c09fa131ae57 + .quad 0x3af55c0dfdf05d96 + .quad 0xdd262ee02ab4ee7a + .quad 0x11b2bb8712171709 + .quad 0x1fef24fa800f030b + + // 2^84 * 1 * G + + .quad 0xb496123a6b6c6609 + .quad 0xa750fe8580ab5938 + .quad 0xf471bf39b7c27a5f + .quad 0x507903ce77ac193c + .quad 0xff91a66a90166220 + .quad 0xf22552ae5bf1e009 + .quad 0x7dff85d87f90df7c + .quad 0x4f620ffe0c736fb9 + .quad 0x62f90d65dfde3e34 + .quad 0xcf28c592b9fa5fad + .quad 0x99c86ef9c6164510 + .quad 0x25d448044a256c84 + + // 2^84 * 2 * G + + .quad 0xbd68230ec7e9b16f + .quad 0x0eb1b9c1c1c5795d + .quad 0x7943c8c495b6b1ff + .quad 0x2f9faf620bbacf5e + .quad 0x2c7c4415c9022b55 + .quad 0x56a0d241812eb1fe + .quad 0xf02ea1c9d7b65e0d + .quad 0x4180512fd5323b26 + .quad 0xa4ff3e698a48a5db + .quad 0xba6a3806bd95403b + .quad 0x9f7ce1af47d5b65d + .quad 0x15e087e55939d2fb + + // 2^84 * 3 * G + + .quad 0x12207543745c1496 + .quad 0xdaff3cfdda38610c + .quad 0xe4e797272c71c34f + .quad 0x39c07b1934bdede9 + .quad 0x8894186efb963f38 + .quad 0x48a00e80dc639bd5 + .quad 0xa4e8092be96c1c99 + .quad 0x5a097d54ca573661 + .quad 0x2d45892b17c9e755 + .quad 0xd033fd7289308df8 + .quad 0x6c2fe9d9525b8bd9 + .quad 0x2edbecf1c11cc079 + + // 2^84 * 4 * G + + .quad 0x1616a4e3c715a0d2 + .quad 0x53623cb0f8341d4d + .quad 0x96ef5329c7e899cb + .quad 0x3d4e8dbba668baa6 + .quad 0xee0f0fddd087a25f + .quad 0x9c7531555c3e34ee + .quad 0x660c572e8fab3ab5 + .quad 0x0854fc44544cd3b2 + .quad 0x61eba0c555edad19 + .quad 0x24b533fef0a83de6 + .quad 0x3b77042883baa5f8 + .quad 0x678f82b898a47e8d + + // 2^84 * 5 * G + + .quad 0xb1491d0bd6900c54 + .quad 0x3539722c9d132636 + .quad 0x4db928920b362bc9 + .quad 0x4d7cd1fea68b69df + .quad 0x1e09d94057775696 + .quad 0xeed1265c3cd951db + .quad 0xfa9dac2b20bce16f + .quad 0x0f7f76e0e8d089f4 + .quad 0x36d9ebc5d485b00c + .quad 0xa2596492e4adb365 + .quad 0xc1659480c2119ccd + .quad 0x45306349186e0d5f + + // 2^84 * 6 * G + + .quad 0x94ddd0c1a6cdff1d + .quad 0x55f6f115e84213ae + .quad 0x6c935f85992fcf6a + .quad 0x067ee0f54a37f16f + .quad 0x96a414ec2b072491 + .quad 0x1bb2218127a7b65b + .quad 0x6d2849596e8a4af0 + .quad 0x65f3b08ccd27765f + .quad 0xecb29fff199801f7 + .quad 0x9d361d1fa2a0f72f + .quad 0x25f11d2375fd2f49 + .quad 0x124cefe80fe10fe2 + + // 2^84 * 7 * G + + .quad 0x4c126cf9d18df255 + .quad 0xc1d471e9147a63b6 + .quad 0x2c6d3c73f3c93b5f + .quad 0x6be3a6a2e3ff86a2 + .quad 0x1518e85b31b16489 + .quad 0x8faadcb7db710bfb + .quad 0x39b0bdf4a14ae239 + .quad 0x05f4cbea503d20c1 + .quad 0xce040e9ec04145bc + .quad 0xc71ff4e208f6834c + .quad 0xbd546e8dab8847a3 + .quad 0x64666aa0a4d2aba5 + + // 2^84 * 8 * G + + .quad 0x6841435a7c06d912 + .quad 0xca123c21bb3f830b + .quad 0xd4b37b27b1cbe278 + .quad 0x1d753b84c76f5046 + .quad 0xb0c53bf73337e94c + .quad 0x7cb5697e11e14f15 + .quad 0x4b84abac1930c750 + .quad 0x28dd4abfe0640468 + .quad 0x7dc0b64c44cb9f44 + .quad 0x18a3e1ace3925dbf + .quad 0x7a3034862d0457c4 + .quad 0x4c498bf78a0c892e + + // 2^88 * 1 * G + + .quad 0x37d653fb1aa73196 + .quad 0x0f9495303fd76418 + .quad 0xad200b09fb3a17b2 + .quad 0x544d49292fc8613e + .quad 0x22d2aff530976b86 + .quad 0x8d90b806c2d24604 + .quad 0xdca1896c4de5bae5 + .quad 0x28005fe6c8340c17 + .quad 0x6aefba9f34528688 + .quad 0x5c1bff9425107da1 + .quad 0xf75bbbcd66d94b36 + .quad 0x72e472930f316dfa + + // 2^88 * 2 * G + + .quad 0x2695208c9781084f + .quad 0xb1502a0b23450ee1 + .quad 0xfd9daea603efde02 + .quad 0x5a9d2e8c2733a34c + .quad 0x07f3f635d32a7627 + .quad 0x7aaa4d865f6566f0 + .quad 0x3c85e79728d04450 + .quad 0x1fee7f000fe06438 + .quad 0x765305da03dbf7e5 + .quad 0xa4daf2491434cdbd + .quad 0x7b4ad5cdd24a88ec + .quad 0x00f94051ee040543 + + // 2^88 * 3 * G + + .quad 0x8d356b23c3d330b2 + .quad 0xf21c8b9bb0471b06 + .quad 0xb36c316c6e42b83c + .quad 0x07d79c7e8beab10d + .quad 0xd7ef93bb07af9753 + .quad 0x583ed0cf3db766a7 + .quad 0xce6998bf6e0b1ec5 + .quad 0x47b7ffd25dd40452 + .quad 0x87fbfb9cbc08dd12 + .quad 0x8a066b3ae1eec29b + .quad 0x0d57242bdb1fc1bf + .quad 0x1c3520a35ea64bb6 + + // 2^88 * 4 * G + + .quad 0x80d253a6bccba34a + .quad 0x3e61c3a13838219b + .quad 0x90c3b6019882e396 + .quad 0x1c3d05775d0ee66f + .quad 0xcda86f40216bc059 + .quad 0x1fbb231d12bcd87e + .quad 0xb4956a9e17c70990 + .quad 0x38750c3b66d12e55 + .quad 0x692ef1409422e51a + .quad 0xcbc0c73c2b5df671 + .quad 0x21014fe7744ce029 + .quad 0x0621e2c7d330487c + + // 2^88 * 5 * G + + .quad 0xaf9860cc8259838d + .quad 0x90ea48c1c69f9adc + .quad 0x6526483765581e30 + .quad 0x0007d6097bd3a5bc + .quad 0xb7ae1796b0dbf0f3 + .quad 0x54dfafb9e17ce196 + .quad 0x25923071e9aaa3b4 + .quad 0x5d8e589ca1002e9d + .quad 0xc0bf1d950842a94b + .quad 0xb2d3c363588f2e3e + .quad 0x0a961438bb51e2ef + .quad 0x1583d7783c1cbf86 + + // 2^88 * 6 * G + + .quad 0xeceea2ef5da27ae1 + .quad 0x597c3a1455670174 + .quad 0xc9a62a126609167a + .quad 0x252a5f2e81ed8f70 + .quad 0x90034704cc9d28c7 + .quad 0x1d1b679ef72cc58f + .quad 0x16e12b5fbe5b8726 + .quad 0x4958064e83c5580a + .quad 0x0d2894265066e80d + .quad 0xfcc3f785307c8c6b + .quad 0x1b53da780c1112fd + .quad 0x079c170bd843b388 + + // 2^88 * 7 * G + + .quad 0x0506ece464fa6fff + .quad 0xbee3431e6205e523 + .quad 0x3579422451b8ea42 + .quad 0x6dec05e34ac9fb00 + .quad 0xcdd6cd50c0d5d056 + .quad 0x9af7686dbb03573b + .quad 0x3ca6723ff3c3ef48 + .quad 0x6768c0d7317b8acc + .quad 0x94b625e5f155c1b3 + .quad 0x417bf3a7997b7b91 + .quad 0xc22cbddc6d6b2600 + .quad 0x51445e14ddcd52f4 + + // 2^88 * 8 * G + + .quad 0x57502b4b3b144951 + .quad 0x8e67ff6b444bbcb3 + .quad 0xb8bd6927166385db + .quad 0x13186f31e39295c8 + .quad 0x893147ab2bbea455 + .quad 0x8c53a24f92079129 + .quad 0x4b49f948be30f7a7 + .quad 0x12e990086e4fd43d + .quad 0xf10c96b37fdfbb2e + .quad 0x9f9a935e121ceaf9 + .quad 0xdf1136c43a5b983f + .quad 0x77b2e3f05d3e99af + + // 2^92 * 1 * G + + .quad 0xfd0d75879cf12657 + .quad 0xe82fef94e53a0e29 + .quad 0xcc34a7f05bbb4be7 + .quad 0x0b251172a50c38a2 + .quad 0x9532f48fcc5cd29b + .quad 0x2ba851bea3ce3671 + .quad 0x32dacaa051122941 + .quad 0x478d99d9350004f2 + .quad 0x1d5ad94890bb02c0 + .quad 0x50e208b10ec25115 + .quad 0xa26a22894ef21702 + .quad 0x4dc923343b524805 + + // 2^92 * 2 * G + + .quad 0xe3828c400f8086b6 + .quad 0x3f77e6f7979f0dc8 + .quad 0x7ef6de304df42cb4 + .quad 0x5265797cb6abd784 + .quad 0x3ad3e3ebf36c4975 + .quad 0xd75d25a537862125 + .quad 0xe873943da025a516 + .quad 0x6bbc7cb4c411c847 + .quad 0x3c6f9cd1d4a50d56 + .quad 0xb6244077c6feab7e + .quad 0x6ff9bf483580972e + .quad 0x00375883b332acfb + + // 2^92 * 3 * G + + .quad 0x0001b2cd28cb0940 + .quad 0x63fb51a06f1c24c9 + .quad 0xb5ad8691dcd5ca31 + .quad 0x67238dbd8c450660 + .quad 0xc98bec856c75c99c + .quad 0xe44184c000e33cf4 + .quad 0x0a676b9bba907634 + .quad 0x669e2cb571f379d7 + .quad 0xcb116b73a49bd308 + .quad 0x025aad6b2392729e + .quad 0xb4793efa3f55d9b1 + .quad 0x72a1056140678bb9 + + // 2^92 * 4 * G + + .quad 0xa2b6812b1cc9249d + .quad 0x62866eee21211f58 + .quad 0x2cb5c5b85df10ece + .quad 0x03a6b259e263ae00 + .quad 0x0d8d2909e2e505b6 + .quad 0x98ca78abc0291230 + .quad 0x77ef5569a9b12327 + .quad 0x7c77897b81439b47 + .quad 0xf1c1b5e2de331cb5 + .quad 0x5a9f5d8e15fca420 + .quad 0x9fa438f17bd932b1 + .quad 0x2a381bf01c6146e7 + + // 2^92 * 5 * G + + .quad 0xac9b9879cfc811c1 + .quad 0x8b7d29813756e567 + .quad 0x50da4e607c70edfc + .quad 0x5dbca62f884400b6 + .quad 0xf7c0be32b534166f + .quad 0x27e6ca6419cf70d4 + .quad 0x934df7d7a957a759 + .quad 0x5701461dabdec2aa + .quad 0x2c6747402c915c25 + .quad 0x1bdcd1a80b0d340a + .quad 0x5e5601bd07b43f5f + .quad 0x2555b4e05539a242 + + // 2^92 * 6 * G + + .quad 0x6fc09f5266ddd216 + .quad 0xdce560a7c8e37048 + .quad 0xec65939da2df62fd + .quad 0x7a869ae7e52ed192 + .quad 0x78409b1d87e463d4 + .quad 0xad4da95acdfb639d + .quad 0xec28773755259b9c + .quad 0x69c806e9c31230ab + .quad 0x7b48f57414bb3f22 + .quad 0x68c7cee4aedccc88 + .quad 0xed2f936179ed80be + .quad 0x25d70b885f77bc4b + + // 2^92 * 7 * G + + .quad 0x4151c3d9762bf4de + .quad 0x083f435f2745d82b + .quad 0x29775a2e0d23ddd5 + .quad 0x138e3a6269a5db24 + .quad 0x98459d29bb1ae4d4 + .quad 0x56b9c4c739f954ec + .quad 0x832743f6c29b4b3e + .quad 0x21ea8e2798b6878a + .quad 0x87bef4b46a5a7b9c + .quad 0xd2299d1b5fc1d062 + .quad 0x82409818dd321648 + .quad 0x5c5abeb1e5a2e03d + + // 2^92 * 8 * G + + .quad 0x14722af4b73c2ddb + .quad 0xbc470c5f5a05060d + .quad 0x00943eac2581b02e + .quad 0x0e434b3b1f499c8f + .quad 0x02cde6de1306a233 + .quad 0x7b5a52a2116f8ec7 + .quad 0xe1c681f4c1163b5b + .quad 0x241d350660d32643 + .quad 0x6be4404d0ebc52c7 + .quad 0xae46233bb1a791f5 + .quad 0x2aec170ed25db42b + .quad 0x1d8dfd966645d694 + + // 2^96 * 1 * G + + .quad 0x296fa9c59c2ec4de + .quad 0xbc8b61bf4f84f3cb + .quad 0x1c7706d917a8f908 + .quad 0x63b795fc7ad3255d + .quad 0xd598639c12ddb0a4 + .quad 0xa5d19f30c024866b + .quad 0xd17c2f0358fce460 + .quad 0x07a195152e095e8a + .quad 0xa8368f02389e5fc8 + .quad 0x90433b02cf8de43b + .quad 0xafa1fd5dc5412643 + .quad 0x3e8fe83d032f0137 + + // 2^96 * 2 * G + + .quad 0x2f8b15b90570a294 + .quad 0x94f2427067084549 + .quad 0xde1c5ae161bbfd84 + .quad 0x75ba3b797fac4007 + .quad 0x08704c8de8efd13c + .quad 0xdfc51a8e33e03731 + .quad 0xa59d5da51260cde3 + .quad 0x22d60899a6258c86 + .quad 0x6239dbc070cdd196 + .quad 0x60fe8a8b6c7d8a9a + .quad 0xb38847bceb401260 + .quad 0x0904d07b87779e5e + + // 2^96 * 3 * G + + .quad 0xb4ce1fd4ddba919c + .quad 0xcf31db3ec74c8daa + .quad 0x2c63cc63ad86cc51 + .quad 0x43e2143fbc1dde07 + .quad 0xf4322d6648f940b9 + .quad 0x06952f0cbd2d0c39 + .quad 0x167697ada081f931 + .quad 0x6240aacebaf72a6c + .quad 0xf834749c5ba295a0 + .quad 0xd6947c5bca37d25a + .quad 0x66f13ba7e7c9316a + .quad 0x56bdaf238db40cac + + // 2^96 * 4 * G + + .quad 0x362ab9e3f53533eb + .quad 0x338568d56eb93d40 + .quad 0x9e0e14521d5a5572 + .quad 0x1d24a86d83741318 + .quad 0x1310d36cc19d3bb2 + .quad 0x062a6bb7622386b9 + .quad 0x7c9b8591d7a14f5c + .quad 0x03aa31507e1e5754 + .quad 0xf4ec7648ffd4ce1f + .quad 0xe045eaf054ac8c1c + .quad 0x88d225821d09357c + .quad 0x43b261dc9aeb4859 + + // 2^96 * 5 * G + + .quad 0xe55b1e1988bb79bb + .quad 0xa09ed07dc17a359d + .quad 0xb02c2ee2603dea33 + .quad 0x326055cf5b276bc2 + .quad 0x19513d8b6c951364 + .quad 0x94fe7126000bf47b + .quad 0x028d10ddd54f9567 + .quad 0x02b4d5e242940964 + .quad 0xb4a155cb28d18df2 + .quad 0xeacc4646186ce508 + .quad 0xc49cf4936c824389 + .quad 0x27a6c809ae5d3410 + + // 2^96 * 6 * G + + .quad 0x8ba6ebcd1f0db188 + .quad 0x37d3d73a675a5be8 + .quad 0xf22edfa315f5585a + .quad 0x2cb67174ff60a17e + .quad 0xcd2c270ac43d6954 + .quad 0xdd4a3e576a66cab2 + .quad 0x79fa592469d7036c + .quad 0x221503603d8c2599 + .quad 0x59eecdf9390be1d0 + .quad 0xa9422044728ce3f1 + .quad 0x82891c667a94f0f4 + .quad 0x7b1df4b73890f436 + + // 2^96 * 7 * G + + .quad 0xe492f2e0b3b2a224 + .quad 0x7c6c9e062b551160 + .quad 0x15eb8fe20d7f7b0e + .quad 0x61fcef2658fc5992 + .quad 0x5f2e221807f8f58c + .quad 0xe3555c9fd49409d4 + .quad 0xb2aaa88d1fb6a630 + .quad 0x68698245d352e03d + .quad 0xdbb15d852a18187a + .quad 0xf3e4aad386ddacd7 + .quad 0x44bae2810ff6c482 + .quad 0x46cf4c473daf01cf + + // 2^96 * 8 * G + + .quad 0x426525ed9ec4e5f9 + .quad 0x0e5eda0116903303 + .quad 0x72b1a7f2cbe5cadc + .quad 0x29387bcd14eb5f40 + .quad 0x213c6ea7f1498140 + .quad 0x7c1e7ef8392b4854 + .quad 0x2488c38c5629ceba + .quad 0x1065aae50d8cc5bb + .quad 0x1c2c4525df200d57 + .quad 0x5c3b2dd6bfca674a + .quad 0x0a07e7b1e1834030 + .quad 0x69a198e64f1ce716 + + // 2^100 * 1 * G + + .quad 0x7afcd613efa9d697 + .quad 0x0cc45aa41c067959 + .quad 0xa56fe104c1fada96 + .quad 0x3a73b70472e40365 + .quad 0x7b26e56b9e2d4734 + .quad 0xc4c7132b81c61675 + .quad 0xef5c9525ec9cde7f + .quad 0x39c80b16e71743ad + .quad 0x0f196e0d1b826c68 + .quad 0xf71ff0e24960e3db + .quad 0x6113167023b7436c + .quad 0x0cf0ea5877da7282 + + // 2^100 * 2 * G + + .quad 0x196c80a4ddd4ccbd + .quad 0x22e6f55d95f2dd9d + .quad 0xc75e33c740d6c71b + .quad 0x7bb51279cb3c042f + .quad 0xe332ced43ba6945a + .quad 0xde0b1361e881c05d + .quad 0x1ad40f095e67ed3b + .quad 0x5da8acdab8c63d5d + .quad 0xc4b6664a3a70159f + .quad 0x76194f0f0a904e14 + .quad 0xa5614c39a4096c13 + .quad 0x6cd0ff50979feced + + // 2^100 * 3 * G + + .quad 0xc0e067e78f4428ac + .quad 0x14835ab0a61135e3 + .quad 0xf21d14f338062935 + .quad 0x6390a4c8df04849c + .quad 0x7fecfabdb04ba18e + .quad 0xd0fc7bfc3bddbcf7 + .quad 0xa41d486e057a131c + .quad 0x641a4391f2223a61 + .quad 0xc5c6b95aa606a8db + .quad 0x914b7f9eb06825f1 + .quad 0x2a731f6b44fc9eff + .quad 0x30ddf38562705cfc + + // 2^100 * 4 * G + + .quad 0x4e3dcbdad1bff7f9 + .quad 0xc9118e8220645717 + .quad 0xbacccebc0f189d56 + .quad 0x1b4822e9d4467668 + .quad 0x33bef2bd68bcd52c + .quad 0xc649dbb069482ef2 + .quad 0xb5b6ee0c41cb1aee + .quad 0x5c294d270212a7e5 + .quad 0xab360a7f25563781 + .quad 0x2512228a480f7958 + .quad 0xc75d05276114b4e3 + .quad 0x222d9625d976fe2a + + // 2^100 * 5 * G + + .quad 0x1c717f85b372ace1 + .quad 0x81930e694638bf18 + .quad 0x239cad056bc08b58 + .quad 0x0b34271c87f8fff4 + .quad 0x0f94be7e0a344f85 + .quad 0xeb2faa8c87f22c38 + .quad 0x9ce1e75e4ee16f0f + .quad 0x43e64e5418a08dea + .quad 0x8155e2521a35ce63 + .quad 0xbe100d4df912028e + .quad 0xbff80bf8a57ddcec + .quad 0x57342dc96d6bc6e4 + + // 2^100 * 6 * G + + .quad 0xefeef065c8ce5998 + .quad 0xbf029510b5cbeaa2 + .quad 0x8c64a10620b7c458 + .quad 0x35134fb231c24855 + .quad 0xf3c3bcb71e707bf6 + .quad 0x351d9b8c7291a762 + .quad 0x00502e6edad69a33 + .quad 0x522f521f1ec8807f + .quad 0x272c1f46f9a3902b + .quad 0xc91ba3b799657bcc + .quad 0xae614b304f8a1c0e + .quad 0x7afcaad70b99017b + + // 2^100 * 7 * G + + .quad 0xc25ded54a4b8be41 + .quad 0x902d13e11bb0e2dd + .quad 0x41f43233cde82ab2 + .quad 0x1085faa5c3aae7cb + .quad 0xa88141ecef842b6b + .quad 0x55e7b14797abe6c5 + .quad 0x8c748f9703784ffe + .quad 0x5b50a1f7afcd00b7 + .quad 0x9b840f66f1361315 + .quad 0x18462242701003e9 + .quad 0x65ed45fae4a25080 + .quad 0x0a2862393fda7320 + + // 2^100 * 8 * G + + .quad 0x46ab13c8347cbc9d + .quad 0x3849e8d499c12383 + .quad 0x4cea314087d64ac9 + .quad 0x1f354134b1a29ee7 + .quad 0x960e737b6ecb9d17 + .quad 0xfaf24948d67ceae1 + .quad 0x37e7a9b4d55e1b89 + .quad 0x5cb7173cb46c59eb + .quad 0x4a89e68b82b7abf0 + .quad 0xf41cd9279ba6b7b9 + .quad 0x16e6c210e18d876f + .quad 0x7cacdb0f7f1b09c6 + + // 2^104 * 1 * G + + .quad 0x9062b2e0d91a78bc + .quad 0x47c9889cc8509667 + .quad 0x9df54a66405070b8 + .quad 0x7369e6a92493a1bf + .quad 0xe1014434dcc5caed + .quad 0x47ed5d963c84fb33 + .quad 0x70019576ed86a0e7 + .quad 0x25b2697bd267f9e4 + .quad 0x9d673ffb13986864 + .quad 0x3ca5fbd9415dc7b8 + .quad 0xe04ecc3bdf273b5e + .quad 0x1420683db54e4cd2 + + // 2^104 * 2 * G + + .quad 0xb478bd1e249dd197 + .quad 0x620c35005e58c102 + .quad 0xfb02d32fccbaac5c + .quad 0x60b63bebf508a72d + .quad 0x34eebb6fc1cc5ad0 + .quad 0x6a1b0ce99646ac8b + .quad 0xd3b0da49a66bde53 + .quad 0x31e83b4161d081c1 + .quad 0x97e8c7129e062b4f + .quad 0x49e48f4f29320ad8 + .quad 0x5bece14b6f18683f + .quad 0x55cf1eb62d550317 + + // 2^104 * 3 * G + + .quad 0x5879101065c23d58 + .quad 0x8b9d086d5094819c + .quad 0xe2402fa912c55fa7 + .quad 0x669a6564570891d4 + .quad 0x3076b5e37df58c52 + .quad 0xd73ab9dde799cc36 + .quad 0xbd831ce34913ee20 + .quad 0x1a56fbaa62ba0133 + .quad 0x943e6b505c9dc9ec + .quad 0x302557bba77c371a + .quad 0x9873ae5641347651 + .quad 0x13c4836799c58a5c + + // 2^104 * 4 * G + + .quad 0x423a5d465ab3e1b9 + .quad 0xfc13c187c7f13f61 + .quad 0x19f83664ecb5b9b6 + .quad 0x66f80c93a637b607 + .quad 0xc4dcfb6a5d8bd080 + .quad 0xdeebc4ec571a4842 + .quad 0xd4b2e883b8e55365 + .quad 0x50bdc87dc8e5b827 + .quad 0x606d37836edfe111 + .quad 0x32353e15f011abd9 + .quad 0x64b03ac325b73b96 + .quad 0x1dd56444725fd5ae + + // 2^104 * 5 * G + + .quad 0x8fa47ff83362127d + .quad 0xbc9f6ac471cd7c15 + .quad 0x6e71454349220c8b + .quad 0x0e645912219f732e + .quad 0xc297e60008bac89a + .quad 0x7d4cea11eae1c3e0 + .quad 0xf3e38be19fe7977c + .quad 0x3a3a450f63a305cd + .quad 0x078f2f31d8394627 + .quad 0x389d3183de94a510 + .quad 0xd1e36c6d17996f80 + .quad 0x318c8d9393a9a87b + + // 2^104 * 6 * G + + .quad 0xf2745d032afffe19 + .quad 0x0c9f3c497f24db66 + .quad 0xbc98d3e3ba8598ef + .quad 0x224c7c679a1d5314 + .quad 0x5d669e29ab1dd398 + .quad 0xfc921658342d9e3b + .quad 0x55851dfdf35973cd + .quad 0x509a41c325950af6 + .quad 0xbdc06edca6f925e9 + .quad 0x793ef3f4641b1f33 + .quad 0x82ec12809d833e89 + .quad 0x05bff02328a11389 + + // 2^104 * 7 * G + + .quad 0x3632137023cae00b + .quad 0x544acf0ad1accf59 + .quad 0x96741049d21a1c88 + .quad 0x780b8cc3fa2a44a7 + .quad 0x6881a0dd0dc512e4 + .quad 0x4fe70dc844a5fafe + .quad 0x1f748e6b8f4a5240 + .quad 0x576277cdee01a3ea + .quad 0x1ef38abc234f305f + .quad 0x9a577fbd1405de08 + .quad 0x5e82a51434e62a0d + .quad 0x5ff418726271b7a1 + + // 2^104 * 8 * G + + .quad 0x398e080c1789db9d + .quad 0xa7602025f3e778f5 + .quad 0xfa98894c06bd035d + .quad 0x106a03dc25a966be + .quad 0xe5db47e813b69540 + .quad 0xf35d2a3b432610e1 + .quad 0xac1f26e938781276 + .quad 0x29d4db8ca0a0cb69 + .quad 0xd9ad0aaf333353d0 + .quad 0x38669da5acd309e5 + .quad 0x3c57658ac888f7f0 + .quad 0x4ab38a51052cbefa + + // 2^108 * 1 * G + + .quad 0xdfdacbee4324c0e9 + .quad 0x054442883f955bb7 + .quad 0xdef7aaa8ea31609f + .quad 0x68aee70642287cff + .quad 0xf68fe2e8809de054 + .quad 0xe3bc096a9c82bad1 + .quad 0x076353d40aadbf45 + .quad 0x7b9b1fb5dea1959e + .quad 0xf01cc8f17471cc0c + .quad 0x95242e37579082bb + .quad 0x27776093d3e46b5f + .quad 0x2d13d55a28bd85fb + + // 2^108 * 2 * G + + .quad 0xfac5d2065b35b8da + .quad 0xa8da8a9a85624bb7 + .quad 0xccd2ca913d21cd0f + .quad 0x6b8341ee8bf90d58 + .quad 0xbf019cce7aee7a52 + .quad 0xa8ded2b6e454ead3 + .quad 0x3c619f0b87a8bb19 + .quad 0x3619b5d7560916d8 + .quad 0x3579f26b0282c4b2 + .quad 0x64d592f24fafefae + .quad 0xb7cded7b28c8c7c0 + .quad 0x6a927b6b7173a8d7 + + // 2^108 * 3 * G + + .quad 0x1f6db24f986e4656 + .quad 0x1021c02ed1e9105b + .quad 0xf8ff3fff2cc0a375 + .quad 0x1d2a6bf8c6c82592 + .quad 0x8d7040863ece88eb + .quad 0xf0e307a980eec08c + .quad 0xac2250610d788fda + .quad 0x056d92a43a0d478d + .quad 0x1b05a196fc3da5a1 + .quad 0x77d7a8c243b59ed0 + .quad 0x06da3d6297d17918 + .quad 0x66fbb494f12353f7 + + // 2^108 * 4 * G + + .quad 0x751a50b9d85c0fb8 + .quad 0xd1afdc258bcf097b + .quad 0x2f16a6a38309a969 + .quad 0x14ddff9ee5b00659 + .quad 0xd6d70996f12309d6 + .quad 0xdbfb2385e9c3d539 + .quad 0x46d602b0f7552411 + .quad 0x270a0b0557843e0c + .quad 0x61ff0640a7862bcc + .quad 0x81cac09a5f11abfe + .quad 0x9047830455d12abb + .quad 0x19a4bde1945ae873 + + // 2^108 * 5 * G + + .quad 0x9b9f26f520a6200a + .quad 0x64804443cf13eaf8 + .quad 0x8a63673f8631edd3 + .quad 0x72bbbce11ed39dc1 + .quad 0x40c709dec076c49f + .quad 0x657bfaf27f3e53f6 + .quad 0x40662331eca042c4 + .quad 0x14b375487eb4df04 + .quad 0xae853c94ab66dc47 + .quad 0xeb62343edf762d6e + .quad 0xf08e0e186fb2f7d1 + .quad 0x4f0b1c02700ab37a + + // 2^108 * 6 * G + + .quad 0xe1706787d81951fa + .quad 0xa10a2c8eb290c77b + .quad 0xe7382fa03ed66773 + .quad 0x0a4d84710bcc4b54 + .quad 0x79fd21ccc1b2e23f + .quad 0x4ae7c281453df52a + .quad 0xc8172ec9d151486b + .quad 0x68abe9443e0a7534 + .quad 0xda12c6c407831dcb + .quad 0x0da230d74d5c510d + .quad 0x4ab1531e6bd404e1 + .quad 0x4106b166bcf440ef + + // 2^108 * 7 * G + + .quad 0x02e57a421cd23668 + .quad 0x4ad9fb5d0eaef6fd + .quad 0x954e6727b1244480 + .quad 0x7f792f9d2699f331 + .quad 0xa485ccd539e4ecf2 + .quad 0x5aa3f3ad0555bab5 + .quad 0x145e3439937df82d + .quad 0x1238b51e1214283f + .quad 0x0b886b925fd4d924 + .quad 0x60906f7a3626a80d + .quad 0xecd367b4b98abd12 + .quad 0x2876beb1def344cf + + // 2^108 * 8 * G + + .quad 0xdc84e93563144691 + .quad 0x632fe8a0d61f23f4 + .quad 0x4caa800612a9a8d5 + .quad 0x48f9dbfa0e9918d3 + .quad 0xd594b3333a8a85f8 + .quad 0x4ea37689e78d7d58 + .quad 0x73bf9f455e8e351f + .quad 0x5507d7d2bc41ebb4 + .quad 0x1ceb2903299572fc + .quad 0x7c8ccaa29502d0ee + .quad 0x91bfa43411cce67b + .quad 0x5784481964a831e7 + + // 2^112 * 1 * G + + .quad 0xda7c2b256768d593 + .quad 0x98c1c0574422ca13 + .quad 0xf1a80bd5ca0ace1d + .quad 0x29cdd1adc088a690 + .quad 0xd6cfd1ef5fddc09c + .quad 0xe82b3efdf7575dce + .quad 0x25d56b5d201634c2 + .quad 0x3041c6bb04ed2b9b + .quad 0x0ff2f2f9d956e148 + .quad 0xade797759f356b2e + .quad 0x1a4698bb5f6c025c + .quad 0x104bbd6814049a7b + + // 2^112 * 2 * G + + .quad 0x51f0fd3168f1ed67 + .quad 0x2c811dcdd86f3bc2 + .quad 0x44dc5c4304d2f2de + .quad 0x5be8cc57092a7149 + .quad 0xa95d9a5fd67ff163 + .quad 0xe92be69d4cc75681 + .quad 0xb7f8024cde20f257 + .quad 0x204f2a20fb072df5 + .quad 0xc8143b3d30ebb079 + .quad 0x7589155abd652e30 + .quad 0x653c3c318f6d5c31 + .quad 0x2570fb17c279161f + + // 2^112 * 3 * G + + .quad 0x3efa367f2cb61575 + .quad 0xf5f96f761cd6026c + .quad 0xe8c7142a65b52562 + .quad 0x3dcb65ea53030acd + .quad 0x192ea9550bb8245a + .quad 0xc8e6fba88f9050d1 + .quad 0x7986ea2d88a4c935 + .quad 0x241c5f91de018668 + .quad 0x28d8172940de6caa + .quad 0x8fbf2cf022d9733a + .quad 0x16d7fcdd235b01d1 + .quad 0x08420edd5fcdf0e5 + + // 2^112 * 4 * G + + .quad 0xcdff20ab8362fa4a + .quad 0x57e118d4e21a3e6e + .quad 0xe3179617fc39e62b + .quad 0x0d9a53efbc1769fd + .quad 0x0358c34e04f410ce + .quad 0xb6135b5a276e0685 + .quad 0x5d9670c7ebb91521 + .quad 0x04d654f321db889c + .quad 0x5e7dc116ddbdb5d5 + .quad 0x2954deb68da5dd2d + .quad 0x1cb608173334a292 + .quad 0x4a7a4f2618991ad7 + + // 2^112 * 5 * G + + .quad 0xf4a718025fb15f95 + .quad 0x3df65f346b5c1b8f + .quad 0xcdfcf08500e01112 + .quad 0x11b50c4cddd31848 + .quad 0x24c3b291af372a4b + .quad 0x93da8270718147f2 + .quad 0xdd84856486899ef2 + .quad 0x4a96314223e0ee33 + .quad 0xa6e8274408a4ffd6 + .quad 0x738e177e9c1576d9 + .quad 0x773348b63d02b3f2 + .quad 0x4f4bce4dce6bcc51 + + // 2^112 * 6 * G + + .quad 0xa71fce5ae2242584 + .quad 0x26ea725692f58a9e + .quad 0xd21a09d71cea3cf4 + .quad 0x73fcdd14b71c01e6 + .quad 0x30e2616ec49d0b6f + .quad 0xe456718fcaec2317 + .quad 0x48eb409bf26b4fa6 + .quad 0x3042cee561595f37 + .quad 0x427e7079449bac41 + .quad 0x855ae36dbce2310a + .quad 0x4cae76215f841a7c + .quad 0x389e740c9a9ce1d6 + + // 2^112 * 7 * G + + .quad 0x64fcb3ae34dcb9ce + .quad 0x97500323e348d0ad + .quad 0x45b3f07d62c6381b + .quad 0x61545379465a6788 + .quad 0xc9bd78f6570eac28 + .quad 0xe55b0b3227919ce1 + .quad 0x65fc3eaba19b91ed + .quad 0x25c425e5d6263690 + .quad 0x3f3e06a6f1d7de6e + .quad 0x3ef976278e062308 + .quad 0x8c14f6264e8a6c77 + .quad 0x6539a08915484759 + + // 2^112 * 8 * G + + .quad 0xe9d21f74c3d2f773 + .quad 0xc150544125c46845 + .quad 0x624e5ce8f9b99e33 + .quad 0x11c5e4aac5cd186c + .quad 0xddc4dbd414bb4a19 + .quad 0x19b2bc3c98424f8e + .quad 0x48a89fd736ca7169 + .quad 0x0f65320ef019bd90 + .quad 0xd486d1b1cafde0c6 + .quad 0x4f3fe6e3163b5181 + .quad 0x59a8af0dfaf2939a + .quad 0x4cabc7bdec33072a + + // 2^116 * 1 * G + + .quad 0x16faa8fb532f7428 + .quad 0xdbd42ea046a4e272 + .quad 0x5337653b8b9ea480 + .quad 0x4065947223973f03 + .quad 0xf7c0a19c1a54a044 + .quad 0x4a1c5e2477bd9fbb + .quad 0xa6e3ca115af22972 + .quad 0x1819bb953f2e9e0d + .quad 0x498fbb795e042e84 + .quad 0x7d0dd89a7698b714 + .quad 0x8bfb0ba427fe6295 + .quad 0x36ba82e721200524 + + // 2^116 * 2 * G + + .quad 0xd60ecbb74245ec41 + .quad 0xfd9be89e34348716 + .quad 0xc9240afee42284de + .quad 0x4472f648d0531db4 + .quad 0xc8d69d0a57274ed5 + .quad 0x45ba803260804b17 + .quad 0xdf3cda102255dfac + .quad 0x77d221232709b339 + .quad 0x498a6d7064ad94d8 + .quad 0xa5b5c8fd9af62263 + .quad 0x8ca8ed0545c141f4 + .quad 0x2c63bec3662d358c + + // 2^116 * 3 * G + + .quad 0x7fe60d8bea787955 + .quad 0xb9dc117eb5f401b7 + .quad 0x91c7c09a19355cce + .quad 0x22692ef59442bedf + .quad 0x9a518b3a8586f8bf + .quad 0x9ee71af6cbb196f0 + .quad 0xaa0625e6a2385cf2 + .quad 0x1deb2176ddd7c8d1 + .quad 0x8563d19a2066cf6c + .quad 0x401bfd8c4dcc7cd7 + .quad 0xd976a6becd0d8f62 + .quad 0x67cfd773a278b05e + + // 2^116 * 4 * G + + .quad 0x8dec31faef3ee475 + .quad 0x99dbff8a9e22fd92 + .quad 0x512d11594e26cab1 + .quad 0x0cde561eec4310b9 + .quad 0x2d5fa9855a4e586a + .quad 0x65f8f7a449beab7e + .quad 0xaa074dddf21d33d3 + .quad 0x185cba721bcb9dee + .quad 0x93869da3f4e3cb41 + .quad 0xbf0392f540f7977e + .quad 0x026204fcd0463b83 + .quad 0x3ec91a769eec6eed + + // 2^116 * 5 * G + + .quad 0x1e9df75bf78166ad + .quad 0x4dfda838eb0cd7af + .quad 0xba002ed8c1eaf988 + .quad 0x13fedb3e11f33cfc + .quad 0x0fad2fb7b0a3402f + .quad 0x46615ecbfb69f4a8 + .quad 0xf745bcc8c5f8eaa6 + .quad 0x7a5fa8794a94e896 + .quad 0x52958faa13cd67a1 + .quad 0x965ee0818bdbb517 + .quad 0x16e58daa2e8845b3 + .quad 0x357d397d5499da8f + + // 2^116 * 6 * G + + .quad 0x1ebfa05fb0bace6c + .quad 0xc934620c1caf9a1e + .quad 0xcc771cc41d82b61a + .quad 0x2d94a16aa5f74fec + .quad 0x481dacb4194bfbf8 + .quad 0x4d77e3f1bae58299 + .quad 0x1ef4612e7d1372a0 + .quad 0x3a8d867e70ff69e1 + .quad 0x6f58cd5d55aff958 + .quad 0xba3eaa5c75567721 + .quad 0x75c123999165227d + .quad 0x69be1343c2f2b35e + + // 2^116 * 7 * G + + .quad 0x0e091d5ee197c92a + .quad 0x4f51019f2945119f + .quad 0x143679b9f034e99c + .quad 0x7d88112e4d24c696 + .quad 0x82bbbdac684b8de3 + .quad 0xa2f4c7d03fca0718 + .quad 0x337f92fbe096aaa8 + .quad 0x200d4d8c63587376 + .quad 0x208aed4b4893b32b + .quad 0x3efbf23ebe59b964 + .quad 0xd762deb0dba5e507 + .quad 0x69607bd681bd9d94 + + // 2^116 * 8 * G + + .quad 0xf6be021068de1ce1 + .quad 0xe8d518e70edcbc1f + .quad 0xe3effdd01b5505a5 + .quad 0x35f63353d3ec3fd0 + .quad 0x3b7f3bd49323a902 + .quad 0x7c21b5566b2c6e53 + .quad 0xe5ba8ff53a7852a7 + .quad 0x28bc77a5838ece00 + .quad 0x63ba78a8e25d8036 + .quad 0x63651e0094333490 + .quad 0x48d82f20288ce532 + .quad 0x3a31abfa36b57524 + + // 2^120 * 1 * G + + .quad 0x239e9624089c0a2e + .quad 0xc748c4c03afe4738 + .quad 0x17dbed2a764fa12a + .quad 0x639b93f0321c8582 + .quad 0xc08f788f3f78d289 + .quad 0xfe30a72ca1404d9f + .quad 0xf2778bfccf65cc9d + .quad 0x7ee498165acb2021 + .quad 0x7bd508e39111a1c3 + .quad 0x2b2b90d480907489 + .quad 0xe7d2aec2ae72fd19 + .quad 0x0edf493c85b602a6 + + // 2^120 * 2 * G + + .quad 0xaecc8158599b5a68 + .quad 0xea574f0febade20e + .quad 0x4fe41d7422b67f07 + .quad 0x403b92e3019d4fb4 + .quad 0x6767c4d284764113 + .quad 0xa090403ff7f5f835 + .quad 0x1c8fcffacae6bede + .quad 0x04c00c54d1dfa369 + .quad 0x4dc22f818b465cf8 + .quad 0x71a0f35a1480eff8 + .quad 0xaee8bfad04c7d657 + .quad 0x355bb12ab26176f4 + + // 2^120 * 3 * G + + .quad 0xa71e64cc7493bbf4 + .quad 0xe5bd84d9eca3b0c3 + .quad 0x0a6bc50cfa05e785 + .quad 0x0f9b8132182ec312 + .quad 0xa301dac75a8c7318 + .quad 0xed90039db3ceaa11 + .quad 0x6f077cbf3bae3f2d + .quad 0x7518eaf8e052ad8e + .quad 0xa48859c41b7f6c32 + .quad 0x0f2d60bcf4383298 + .quad 0x1815a929c9b1d1d9 + .quad 0x47c3871bbb1755c4 + + // 2^120 * 4 * G + + .quad 0x5144539771ec4f48 + .quad 0xf805b17dc98c5d6e + .quad 0xf762c11a47c3c66b + .quad 0x00b89b85764699dc + .quad 0xfbe65d50c85066b0 + .quad 0x62ecc4b0b3a299b0 + .quad 0xe53754ea441ae8e0 + .quad 0x08fea02ce8d48d5f + .quad 0x824ddd7668deead0 + .quad 0xc86445204b685d23 + .quad 0xb514cfcd5d89d665 + .quad 0x473829a74f75d537 + + // 2^120 * 5 * G + + .quad 0x82d2da754679c418 + .quad 0xe63bd7d8b2618df0 + .quad 0x355eef24ac47eb0a + .quad 0x2078684c4833c6b4 + .quad 0x23d9533aad3902c9 + .quad 0x64c2ddceef03588f + .quad 0x15257390cfe12fb4 + .quad 0x6c668b4d44e4d390 + .quad 0x3b48cf217a78820c + .quad 0xf76a0ab281273e97 + .quad 0xa96c65a78c8eed7b + .quad 0x7411a6054f8a433f + + // 2^120 * 6 * G + + .quad 0x4d659d32b99dc86d + .quad 0x044cdc75603af115 + .quad 0xb34c712cdcc2e488 + .quad 0x7c136574fb8134ff + .quad 0x579ae53d18b175b4 + .quad 0x68713159f392a102 + .quad 0x8455ecba1eef35f5 + .quad 0x1ec9a872458c398f + .quad 0xb8e6a4d400a2509b + .quad 0x9b81d7020bc882b4 + .quad 0x57e7cc9bf1957561 + .quad 0x3add88a5c7cd6460 + + // 2^120 * 7 * G + + .quad 0xab895770b635dcf2 + .quad 0x02dfef6cf66c1fbc + .quad 0x85530268beb6d187 + .quad 0x249929fccc879e74 + .quad 0x85c298d459393046 + .quad 0x8f7e35985ff659ec + .quad 0x1d2ca22af2f66e3a + .quad 0x61ba1131a406a720 + .quad 0xa3d0a0f116959029 + .quad 0x023b6b6cba7ebd89 + .quad 0x7bf15a3e26783307 + .quad 0x5620310cbbd8ece7 + + // 2^120 * 8 * G + + .quad 0x528993434934d643 + .quad 0xb9dbf806a51222f5 + .quad 0x8f6d878fc3f41c22 + .quad 0x37676a2a4d9d9730 + .quad 0x6646b5f477e285d6 + .quad 0x40e8ff676c8f6193 + .quad 0xa6ec7311abb594dd + .quad 0x7ec846f3658cec4d + .quad 0x9b5e8f3f1da22ec7 + .quad 0x130f1d776c01cd13 + .quad 0x214c8fcfa2989fb8 + .quad 0x6daaf723399b9dd5 + + // 2^124 * 1 * G + + .quad 0x591e4a5610628564 + .quad 0x2a4bb87ca8b4df34 + .quad 0xde2a2572e7a38e43 + .quad 0x3cbdabd9fee5046e + .quad 0x81aebbdd2cd13070 + .quad 0x962e4325f85a0e9e + .quad 0xde9391aacadffecb + .quad 0x53177fda52c230e6 + .quad 0xa7bc970650b9de79 + .quad 0x3d12a7fbc301b59b + .quad 0x02652e68d36ae38c + .quad 0x79d739835a6199dc + + // 2^124 * 2 * G + + .quad 0xd9354df64131c1bd + .quad 0x758094a186ec5822 + .quad 0x4464ee12e459f3c2 + .quad 0x6c11fce4cb133282 + .quad 0x21c9d9920d591737 + .quad 0x9bea41d2e9b46cd6 + .quad 0xe20e84200d89bfca + .quad 0x79d99f946eae5ff8 + .quad 0xf17b483568673205 + .quad 0x387deae83caad96c + .quad 0x61b471fd56ffe386 + .quad 0x31741195b745a599 + + // 2^124 * 3 * G + + .quad 0xe8d10190b77a360b + .quad 0x99b983209995e702 + .quad 0xbd4fdff8fa0247aa + .quad 0x2772e344e0d36a87 + .quad 0x17f8ba683b02a047 + .quad 0x50212096feefb6c8 + .quad 0x70139be21556cbe2 + .quad 0x203e44a11d98915b + .quad 0xd6863eba37b9e39f + .quad 0x105bc169723b5a23 + .quad 0x104f6459a65c0762 + .quad 0x567951295b4d38d4 + + // 2^124 * 4 * G + + .quad 0x535fd60613037524 + .quad 0xe210adf6b0fbc26a + .quad 0xac8d0a9b23e990ae + .quad 0x47204d08d72fdbf9 + .quad 0x07242eb30d4b497f + .quad 0x1ef96306b9bccc87 + .quad 0x37950934d8116f45 + .quad 0x05468d6201405b04 + .quad 0x00f565a9f93267de + .quad 0xcecfd78dc0d58e8a + .quad 0xa215e2dcf318e28e + .quad 0x4599ee919b633352 + + // 2^124 * 5 * G + + .quad 0xd3c220ca70e0e76b + .quad 0xb12bea58ea9f3094 + .quad 0x294ddec8c3271282 + .quad 0x0c3539e1a1d1d028 + .quad 0xac746d6b861ae579 + .quad 0x31ab0650f6aea9dc + .quad 0x241d661140256d4c + .quad 0x2f485e853d21a5de + .quad 0x329744839c0833f3 + .quad 0x6fe6257fd2abc484 + .quad 0x5327d1814b358817 + .quad 0x65712585893fe9bc + + // 2^124 * 6 * G + + .quad 0x9c102fb732a61161 + .quad 0xe48e10dd34d520a8 + .quad 0x365c63546f9a9176 + .quad 0x32f6fe4c046f6006 + .quad 0x81c29f1bd708ee3f + .quad 0xddcb5a05ae6407d0 + .quad 0x97aec1d7d2a3eba7 + .quad 0x1590521a91d50831 + .quad 0x40a3a11ec7910acc + .quad 0x9013dff8f16d27ae + .quad 0x1a9720d8abb195d4 + .quad 0x1bb9fe452ea98463 + + // 2^124 * 7 * G + + .quad 0xe9d1d950b3d54f9e + .quad 0x2d5f9cbee00d33c1 + .quad 0x51c2c656a04fc6ac + .quad 0x65c091ee3c1cbcc9 + .quad 0xcf5e6c95cc36747c + .quad 0x294201536b0bc30d + .quad 0x453ac67cee797af0 + .quad 0x5eae6ab32a8bb3c9 + .quad 0x7083661114f118ea + .quad 0x2b37b87b94349cad + .quad 0x7273f51cb4e99f40 + .quad 0x78a2a95823d75698 + + // 2^124 * 8 * G + + .quad 0xa2b072e95c8c2ace + .quad 0x69cffc96651e9c4b + .quad 0x44328ef842e7b42b + .quad 0x5dd996c122aadeb3 + .quad 0xb4f23c425ef83207 + .quad 0xabf894d3c9a934b5 + .quad 0xd0708c1339fd87f7 + .quad 0x1876789117166130 + .quad 0x925b5ef0670c507c + .quad 0x819bc842b93c33bf + .quad 0x10792e9a70dd003f + .quad 0x59ad4b7a6e28dc74 + + // 2^128 * 1 * G + + .quad 0x5f3a7562eb3dbe47 + .quad 0xf7ea38548ebda0b8 + .quad 0x00c3e53145747299 + .quad 0x1304e9e71627d551 + .quad 0x583b04bfacad8ea2 + .quad 0x29b743e8148be884 + .quad 0x2b1e583b0810c5db + .quad 0x2b5449e58eb3bbaa + .quad 0x789814d26adc9cfe + .quad 0x3c1bab3f8b48dd0b + .quad 0xda0fe1fff979c60a + .quad 0x4468de2d7c2dd693 + + // 2^128 * 2 * G + + .quad 0x51bb355e9419469e + .quad 0x33e6dc4c23ddc754 + .quad 0x93a5b6d6447f9962 + .quad 0x6cce7c6ffb44bd63 + .quad 0x4b9ad8c6f86307ce + .quad 0x21113531435d0c28 + .quad 0xd4a866c5657a772c + .quad 0x5da6427e63247352 + .quad 0x1a94c688deac22ca + .quad 0xb9066ef7bbae1ff8 + .quad 0x88ad8c388d59580f + .quad 0x58f29abfe79f2ca8 + + // 2^128 * 3 * G + + .quad 0xe90ecfab8de73e68 + .quad 0x54036f9f377e76a5 + .quad 0xf0495b0bbe015982 + .quad 0x577629c4a7f41e36 + .quad 0x4b5a64bf710ecdf6 + .quad 0xb14ce538462c293c + .quad 0x3643d056d50b3ab9 + .quad 0x6af93724185b4870 + .quad 0x3220024509c6a888 + .quad 0xd2e036134b558973 + .quad 0x83e236233c33289f + .quad 0x701f25bb0caec18f + + // 2^128 * 4 * G + + .quad 0xc3a8b0f8e4616ced + .quad 0xf700660e9e25a87d + .quad 0x61e3061ff4bca59c + .quad 0x2e0c92bfbdc40be9 + .quad 0x9d18f6d97cbec113 + .quad 0x844a06e674bfdbe4 + .quad 0x20f5b522ac4e60d6 + .quad 0x720a5bc050955e51 + .quad 0x0c3f09439b805a35 + .quad 0xe84e8b376242abfc + .quad 0x691417f35c229346 + .quad 0x0e9b9cbb144ef0ec + + // 2^128 * 5 * G + + .quad 0xfbbad48ffb5720ad + .quad 0xee81916bdbf90d0e + .quad 0xd4813152635543bf + .quad 0x221104eb3f337bd8 + .quad 0x8dee9bd55db1beee + .quad 0xc9c3ab370a723fb9 + .quad 0x44a8f1bf1c68d791 + .quad 0x366d44191cfd3cde + .quad 0x9e3c1743f2bc8c14 + .quad 0x2eda26fcb5856c3b + .quad 0xccb82f0e68a7fb97 + .quad 0x4167a4e6bc593244 + + // 2^128 * 6 * G + + .quad 0x643b9d2876f62700 + .quad 0x5d1d9d400e7668eb + .quad 0x1b4b430321fc0684 + .quad 0x7938bb7e2255246a + .quad 0xc2be2665f8ce8fee + .quad 0xe967ff14e880d62c + .quad 0xf12e6e7e2f364eee + .quad 0x34b33370cb7ed2f6 + .quad 0xcdc591ee8681d6cc + .quad 0xce02109ced85a753 + .quad 0xed7485c158808883 + .quad 0x1176fc6e2dfe65e4 + + // 2^128 * 7 * G + + .quad 0xb4af6cd05b9c619b + .quad 0x2ddfc9f4b2a58480 + .quad 0x3d4fa502ebe94dc4 + .quad 0x08fc3a4c677d5f34 + .quad 0xdb90e28949770eb8 + .quad 0x98fbcc2aacf440a3 + .quad 0x21354ffeded7879b + .quad 0x1f6a3e54f26906b6 + .quad 0x60a4c199d30734ea + .quad 0x40c085b631165cd6 + .quad 0xe2333e23f7598295 + .quad 0x4f2fad0116b900d1 + + // 2^128 * 8 * G + + .quad 0x44beb24194ae4e54 + .quad 0x5f541c511857ef6c + .quad 0xa61e6b2d368d0498 + .quad 0x445484a4972ef7ab + .quad 0x962cd91db73bb638 + .quad 0xe60577aafc129c08 + .quad 0x6f619b39f3b61689 + .quad 0x3451995f2944ee81 + .quad 0x9152fcd09fea7d7c + .quad 0x4a816c94b0935cf6 + .quad 0x258e9aaa47285c40 + .quad 0x10b89ca6042893b7 + + // 2^132 * 1 * G + + .quad 0x9b2a426e3b646025 + .quad 0x32127190385ce4cf + .quad 0xa25cffc2dd6dea45 + .quad 0x06409010bea8de75 + .quad 0xd67cded679d34aa0 + .quad 0xcc0b9ec0cc4db39f + .quad 0xa535a456e35d190f + .quad 0x2e05d9eaf61f6fef + .quad 0xc447901ad61beb59 + .quad 0x661f19bce5dc880a + .quad 0x24685482b7ca6827 + .quad 0x293c778cefe07f26 + + // 2^132 * 2 * G + + .quad 0x86809e7007069096 + .quad 0xaad75b15e4e50189 + .quad 0x07f35715a21a0147 + .quad 0x0487f3f112815d5e + .quad 0x16c795d6a11ff200 + .quad 0xcb70d0e2b15815c9 + .quad 0x89f293209b5395b5 + .quad 0x50b8c2d031e47b4f + .quad 0x48350c08068a4962 + .quad 0x6ffdd05351092c9a + .quad 0x17af4f4aaf6fc8dd + .quad 0x4b0553b53cdba58b + + // 2^132 * 3 * G + + .quad 0x9c65fcbe1b32ff79 + .quad 0xeb75ea9f03b50f9b + .quad 0xfced2a6c6c07e606 + .quad 0x35106cd551717908 + .quad 0xbf05211b27c152d4 + .quad 0x5ec26849bd1af639 + .quad 0x5e0b2caa8e6fab98 + .quad 0x054c8bdd50bd0840 + .quad 0x38a0b12f1dcf073d + .quad 0x4b60a8a3b7f6a276 + .quad 0xfed5ac25d3404f9a + .quad 0x72e82d5e5505c229 + + // 2^132 * 4 * G + + .quad 0x6b0b697ff0d844c8 + .quad 0xbb12f85cd979cb49 + .quad 0xd2a541c6c1da0f1f + .quad 0x7b7c242958ce7211 + .quad 0x00d9cdfd69771d02 + .quad 0x410276cd6cfbf17e + .quad 0x4c45306c1cb12ec7 + .quad 0x2857bf1627500861 + .quad 0x9f21903f0101689e + .quad 0xd779dfd3bf861005 + .quad 0xa122ee5f3deb0f1b + .quad 0x510df84b485a00d4 + + // 2^132 * 5 * G + + .quad 0xa54133bb9277a1fa + .quad 0x74ec3b6263991237 + .quad 0x1a3c54dc35d2f15a + .quad 0x2d347144e482ba3a + .quad 0x24b3c887c70ac15e + .quad 0xb0f3a557fb81b732 + .quad 0x9b2cde2fe578cc1b + .quad 0x4cf7ed0703b54f8e + .quad 0x6bd47c6598fbee0f + .quad 0x9e4733e2ab55be2d + .quad 0x1093f624127610c5 + .quad 0x4e05e26ad0a1eaa4 + + // 2^132 * 6 * G + + .quad 0xda9b6b624b531f20 + .quad 0x429a760e77509abb + .quad 0xdbe9f522e823cb80 + .quad 0x618f1856880c8f82 + .quad 0x1833c773e18fe6c0 + .quad 0xe3c4711ad3c87265 + .quad 0x3bfd3c4f0116b283 + .quad 0x1955875eb4cd4db8 + .quad 0x6da6de8f0e399799 + .quad 0x7ad61aa440fda178 + .quad 0xb32cd8105e3563dd + .quad 0x15f6beae2ae340ae + + // 2^132 * 7 * G + + .quad 0x862bcb0c31ec3a62 + .quad 0x810e2b451138f3c2 + .quad 0x788ec4b839dac2a4 + .quad 0x28f76867ae2a9281 + .quad 0xba9a0f7b9245e215 + .quad 0xf368612dd98c0dbb + .quad 0x2e84e4cbf220b020 + .quad 0x6ba92fe962d90eda + .quad 0x3e4df9655884e2aa + .quad 0xbd62fbdbdbd465a5 + .quad 0xd7596caa0de9e524 + .quad 0x6e8042ccb2b1b3d7 + + // 2^132 * 8 * G + + .quad 0xf10d3c29ce28ca6e + .quad 0xbad34540fcb6093d + .quad 0xe7426ed7a2ea2d3f + .quad 0x08af9d4e4ff298b9 + .quad 0x1530653616521f7e + .quad 0x660d06b896203dba + .quad 0x2d3989bc545f0879 + .quad 0x4b5303af78ebd7b0 + .quad 0x72f8a6c3bebcbde8 + .quad 0x4f0fca4adc3a8e89 + .quad 0x6fa9d4e8c7bfdf7a + .quad 0x0dcf2d679b624eb7 + + // 2^136 * 1 * G + + .quad 0x3d5947499718289c + .quad 0x12ebf8c524533f26 + .quad 0x0262bfcb14c3ef15 + .quad 0x20b878d577b7518e + .quad 0x753941be5a45f06e + .quad 0xd07caeed6d9c5f65 + .quad 0x11776b9c72ff51b6 + .quad 0x17d2d1d9ef0d4da9 + .quad 0x27f2af18073f3e6a + .quad 0xfd3fe519d7521069 + .quad 0x22e3b72c3ca60022 + .quad 0x72214f63cc65c6a7 + + // 2^136 * 2 * G + + .quad 0xb4e37f405307a693 + .quad 0xaba714d72f336795 + .quad 0xd6fbd0a773761099 + .quad 0x5fdf48c58171cbc9 + .quad 0x1d9db7b9f43b29c9 + .quad 0xd605824a4f518f75 + .quad 0xf2c072bd312f9dc4 + .quad 0x1f24ac855a1545b0 + .quad 0x24d608328e9505aa + .quad 0x4748c1d10c1420ee + .quad 0xc7ffe45c06fb25a2 + .quad 0x00ba739e2ae395e6 + + // 2^136 * 3 * G + + .quad 0x592e98de5c8790d6 + .quad 0xe5bfb7d345c2a2df + .quad 0x115a3b60f9b49922 + .quad 0x03283a3e67ad78f3 + .quad 0xae4426f5ea88bb26 + .quad 0x360679d984973bfb + .quad 0x5c9f030c26694e50 + .quad 0x72297de7d518d226 + .quad 0x48241dc7be0cb939 + .quad 0x32f19b4d8b633080 + .quad 0xd3dfc90d02289308 + .quad 0x05e1296846271945 + + // 2^136 * 4 * G + + .quad 0xba82eeb32d9c495a + .quad 0xceefc8fcf12bb97c + .quad 0xb02dabae93b5d1e0 + .quad 0x39c00c9c13698d9b + .quad 0xadbfbbc8242c4550 + .quad 0xbcc80cecd03081d9 + .quad 0x843566a6f5c8df92 + .quad 0x78cf25d38258ce4c + .quad 0x15ae6b8e31489d68 + .quad 0xaa851cab9c2bf087 + .quad 0xc9a75a97f04efa05 + .quad 0x006b52076b3ff832 + + // 2^136 * 5 * G + + .quad 0x29e0cfe19d95781c + .quad 0xb681df18966310e2 + .quad 0x57df39d370516b39 + .quad 0x4d57e3443bc76122 + .quad 0xf5cb7e16b9ce082d + .quad 0x3407f14c417abc29 + .quad 0xd4b36bce2bf4a7ab + .quad 0x7de2e9561a9f75ce + .quad 0xde70d4f4b6a55ecb + .quad 0x4801527f5d85db99 + .quad 0xdbc9c440d3ee9a81 + .quad 0x6b2a90af1a6029ed + + // 2^136 * 6 * G + + .quad 0x6923f4fc9ae61e97 + .quad 0x5735281de03f5fd1 + .quad 0xa764ae43e6edd12d + .quad 0x5fd8f4e9d12d3e4a + .quad 0x77ebf3245bb2d80a + .quad 0xd8301b472fb9079b + .quad 0xc647e6f24cee7333 + .quad 0x465812c8276c2109 + .quad 0x4d43beb22a1062d9 + .quad 0x7065fb753831dc16 + .quad 0x180d4a7bde2968d7 + .quad 0x05b32c2b1cb16790 + + // 2^136 * 7 * G + + .quad 0xc8c05eccd24da8fd + .quad 0xa1cf1aac05dfef83 + .quad 0xdbbeeff27df9cd61 + .quad 0x3b5556a37b471e99 + .quad 0xf7fca42c7ad58195 + .quad 0x3214286e4333f3cc + .quad 0xb6c29d0d340b979d + .quad 0x31771a48567307e1 + .quad 0x32b0c524e14dd482 + .quad 0xedb351541a2ba4b6 + .quad 0xa3d16048282b5af3 + .quad 0x4fc079d27a7336eb + + // 2^136 * 8 * G + + .quad 0x51c938b089bf2f7f + .quad 0x2497bd6502dfe9a7 + .quad 0xffffc09c7880e453 + .quad 0x124567cecaf98e92 + .quad 0xdc348b440c86c50d + .quad 0x1337cbc9cc94e651 + .quad 0x6422f74d643e3cb9 + .quad 0x241170c2bae3cd08 + .quad 0x3ff9ab860ac473b4 + .quad 0xf0911dee0113e435 + .quad 0x4ae75060ebc6c4af + .quad 0x3f8612966c87000d + + // 2^140 * 1 * G + + .quad 0x0c9c5303f7957be4 + .quad 0xa3c31a20e085c145 + .quad 0xb0721d71d0850050 + .quad 0x0aba390eab0bf2da + .quad 0x529fdffe638c7bf3 + .quad 0xdf2b9e60388b4995 + .quad 0xe027b34f1bad0249 + .quad 0x7bc92fc9b9fa74ed + .quad 0x9f97ef2e801ad9f9 + .quad 0x83697d5479afda3a + .quad 0xe906b3ffbd596b50 + .quad 0x02672b37dd3fb8e0 + + // 2^140 * 2 * G + + .quad 0x48b2ca8b260885e4 + .quad 0xa4286bec82b34c1c + .quad 0x937e1a2617f58f74 + .quad 0x741d1fcbab2ca2a5 + .quad 0xee9ba729398ca7f5 + .quad 0xeb9ca6257a4849db + .quad 0x29eb29ce7ec544e1 + .quad 0x232ca21ef736e2c8 + .quad 0xbf61423d253fcb17 + .quad 0x08803ceafa39eb14 + .quad 0xf18602df9851c7af + .quad 0x0400f3a049e3414b + + // 2^140 * 3 * G + + .quad 0xabce0476ba61c55b + .quad 0x36a3d6d7c4d39716 + .quad 0x6eb259d5e8d82d09 + .quad 0x0c9176e984d756fb + .quad 0x2efba412a06e7b06 + .quad 0x146785452c8d2560 + .quad 0xdf9713ebd67a91c7 + .quad 0x32830ac7157eadf3 + .quad 0x0e782a7ab73769e8 + .quad 0x04a05d7875b18e2c + .quad 0x29525226ebcceae1 + .quad 0x0d794f8383eba820 + + // 2^140 * 4 * G + + .quad 0xff35f5cb9e1516f4 + .quad 0xee805bcf648aae45 + .quad 0xf0d73c2bb93a9ef3 + .quad 0x097b0bf22092a6c2 + .quad 0x7be44ce7a7a2e1ac + .quad 0x411fd93efad1b8b7 + .quad 0x1734a1d70d5f7c9b + .quad 0x0d6592233127db16 + .quad 0xc48bab1521a9d733 + .quad 0xa6c2eaead61abb25 + .quad 0x625c6c1cc6cb4305 + .quad 0x7fc90fea93eb3a67 + + // 2^140 * 5 * G + + .quad 0x0408f1fe1f5c5926 + .quad 0x1a8f2f5e3b258bf4 + .quad 0x40a951a2fdc71669 + .quad 0x6598ee93c98b577e + .quad 0xc527deb59c7cb23d + .quad 0x955391695328404e + .quad 0xd64392817ccf2c7a + .quad 0x6ce97dabf7d8fa11 + .quad 0x25b5a8e50ef7c48f + .quad 0xeb6034116f2ce532 + .quad 0xc5e75173e53de537 + .quad 0x73119fa08c12bb03 + + // 2^140 * 6 * G + + .quad 0xed30129453f1a4cb + .quad 0xbce621c9c8f53787 + .quad 0xfacb2b1338bee7b9 + .quad 0x3025798a9ea8428c + .quad 0x7845b94d21f4774d + .quad 0xbf62f16c7897b727 + .quad 0x671857c03c56522b + .quad 0x3cd6a85295621212 + .quad 0x3fecde923aeca999 + .quad 0xbdaa5b0062e8c12f + .quad 0x67b99dfc96988ade + .quad 0x3f52c02852661036 + + // 2^140 * 7 * G + + .quad 0xffeaa48e2a1351c6 + .quad 0x28624754fa7f53d7 + .quad 0x0b5ba9e57582ddf1 + .quad 0x60c0104ba696ac59 + .quad 0x9258bf99eec416c6 + .quad 0xac8a5017a9d2f671 + .quad 0x629549ab16dea4ab + .quad 0x05d0e85c99091569 + .quad 0x051de020de9cbe97 + .quad 0xfa07fc56b50bcf74 + .quad 0x378cec9f0f11df65 + .quad 0x36853c69ab96de4d + + // 2^140 * 8 * G + + .quad 0x36d9b8de78f39b2d + .quad 0x7f42ed71a847b9ec + .quad 0x241cd1d679bd3fde + .quad 0x6a704fec92fbce6b + .quad 0x4433c0b0fac5e7be + .quad 0x724bae854c08dcbe + .quad 0xf1f24cc446978f9b + .quad 0x4a0aff6d62825fc8 + .quad 0xe917fb9e61095301 + .quad 0xc102df9402a092f8 + .quad 0xbf09e2f5fa66190b + .quad 0x681109bee0dcfe37 + + // 2^144 * 1 * G + + .quad 0x559a0cc9782a0dde + .quad 0x551dcdb2ea718385 + .quad 0x7f62865b31ef238c + .quad 0x504aa7767973613d + .quad 0x9c18fcfa36048d13 + .quad 0x29159db373899ddd + .quad 0xdc9f350b9f92d0aa + .quad 0x26f57eee878a19d4 + .quad 0x0cab2cd55687efb1 + .quad 0x5180d162247af17b + .quad 0x85c15a344f5a2467 + .quad 0x4041943d9dba3069 + + // 2^144 * 2 * G + + .quad 0xc3c0eeba43ebcc96 + .quad 0x8d749c9c26ea9caf + .quad 0xd9fa95ee1c77ccc6 + .quad 0x1420a1d97684340f + .quad 0x4b217743a26caadd + .quad 0x47a6b424648ab7ce + .quad 0xcb1d4f7a03fbc9e3 + .quad 0x12d931429800d019 + .quad 0x00c67799d337594f + .quad 0x5e3c5140b23aa47b + .quad 0x44182854e35ff395 + .quad 0x1b4f92314359a012 + + // 2^144 * 3 * G + + .quad 0x3e5c109d89150951 + .quad 0x39cefa912de9696a + .quad 0x20eae43f975f3020 + .quad 0x239b572a7f132dae + .quad 0x33cf3030a49866b1 + .quad 0x251f73d2215f4859 + .quad 0xab82aa4051def4f6 + .quad 0x5ff191d56f9a23f6 + .quad 0x819ed433ac2d9068 + .quad 0x2883ab795fc98523 + .quad 0xef4572805593eb3d + .quad 0x020c526a758f36cb + + // 2^144 * 4 * G + + .quad 0x779834f89ed8dbbc + .quad 0xc8f2aaf9dc7ca46c + .quad 0xa9524cdca3e1b074 + .quad 0x02aacc4615313877 + .quad 0xe931ef59f042cc89 + .quad 0x2c589c9d8e124bb6 + .quad 0xadc8e18aaec75997 + .quad 0x452cfe0a5602c50c + .quad 0x86a0f7a0647877df + .quad 0xbbc464270e607c9f + .quad 0xab17ea25f1fb11c9 + .quad 0x4cfb7d7b304b877b + + // 2^144 * 5 * G + + .quad 0x72b43d6cb89b75fe + .quad 0x54c694d99c6adc80 + .quad 0xb8c3aa373ee34c9f + .quad 0x14b4622b39075364 + .quad 0xe28699c29789ef12 + .quad 0x2b6ecd71df57190d + .quad 0xc343c857ecc970d0 + .quad 0x5b1d4cbc434d3ac5 + .quad 0xb6fb2615cc0a9f26 + .quad 0x3a4f0e2bb88dcce5 + .quad 0x1301498b3369a705 + .quad 0x2f98f71258592dd1 + + // 2^144 * 6 * G + + .quad 0x0c94a74cb50f9e56 + .quad 0x5b1ff4a98e8e1320 + .quad 0x9a2acc2182300f67 + .quad 0x3a6ae249d806aaf9 + .quad 0x2e12ae444f54a701 + .quad 0xfcfe3ef0a9cbd7de + .quad 0xcebf890d75835de0 + .quad 0x1d8062e9e7614554 + .quad 0x657ada85a9907c5a + .quad 0x1a0ea8b591b90f62 + .quad 0x8d0e1dfbdf34b4e9 + .quad 0x298b8ce8aef25ff3 + + // 2^144 * 7 * G + + .quad 0x2a927953eff70cb2 + .quad 0x4b89c92a79157076 + .quad 0x9418457a30a7cf6a + .quad 0x34b8a8404d5ce485 + .quad 0x837a72ea0a2165de + .quad 0x3fab07b40bcf79f6 + .quad 0x521636c77738ae70 + .quad 0x6ba6271803a7d7dc + .quad 0xc26eecb583693335 + .quad 0xd5a813df63b5fefd + .quad 0xa293aa9aa4b22573 + .quad 0x71d62bdd465e1c6a + + // 2^144 * 8 * G + + .quad 0x6533cc28d378df80 + .quad 0xf6db43790a0fa4b4 + .quad 0xe3645ff9f701da5a + .quad 0x74d5f317f3172ba4 + .quad 0xcd2db5dab1f75ef5 + .quad 0xd77f95cf16b065f5 + .quad 0x14571fea3f49f085 + .quad 0x1c333621262b2b3d + .quad 0xa86fe55467d9ca81 + .quad 0x398b7c752b298c37 + .quad 0xda6d0892e3ac623b + .quad 0x4aebcc4547e9d98c + + // 2^148 * 1 * G + + .quad 0x53175a7205d21a77 + .quad 0xb0c04422d3b934d4 + .quad 0xadd9f24bdd5deadc + .quad 0x074f46e69f10ff8c + .quad 0x0de9b204a059a445 + .quad 0xe15cb4aa4b17ad0f + .quad 0xe1bbec521f79c557 + .quad 0x2633f1b9d071081b + .quad 0xc1fb4177018b9910 + .quad 0xa6ea20dc6c0fe140 + .quad 0xd661f3e74354c6ff + .quad 0x5ecb72e6f1a3407a + + // 2^148 * 2 * G + + .quad 0xa515a31b2259fb4e + .quad 0x0960f3972bcac52f + .quad 0xedb52fec8d3454cb + .quad 0x382e2720c476c019 + .quad 0xfeeae106e8e86997 + .quad 0x9863337f98d09383 + .quad 0x9470480eaa06ebef + .quad 0x038b6898d4c5c2d0 + .quad 0xf391c51d8ace50a6 + .quad 0x3142d0b9ae2d2948 + .quad 0xdb4d5a1a7f24ca80 + .quad 0x21aeba8b59250ea8 + + // 2^148 * 3 * G + + .quad 0x24f13b34cf405530 + .quad 0x3c44ea4a43088af7 + .quad 0x5dd5c5170006a482 + .quad 0x118eb8f8890b086d + .quad 0x53853600f0087f23 + .quad 0x4c461879da7d5784 + .quad 0x6af303deb41f6860 + .quad 0x0a3c16c5c27c18ed + .quad 0x17e49c17cc947f3d + .quad 0xccc6eda6aac1d27b + .quad 0xdf6092ceb0f08e56 + .quad 0x4909b3e22c67c36b + + // 2^148 * 4 * G + + .quad 0x9c9c85ea63fe2e89 + .quad 0xbe1baf910e9412ec + .quad 0x8f7baa8a86fbfe7b + .quad 0x0fb17f9fef968b6c + .quad 0x59a16676706ff64e + .quad 0x10b953dd0d86a53d + .quad 0x5848e1e6ce5c0b96 + .quad 0x2d8b78e712780c68 + .quad 0x79d5c62eafc3902b + .quad 0x773a215289e80728 + .quad 0xc38ae640e10120b9 + .quad 0x09ae23717b2b1a6d + + // 2^148 * 5 * G + + .quad 0xbb6a192a4e4d083c + .quad 0x34ace0630029e192 + .quad 0x98245a59aafabaeb + .quad 0x6d9c8a9ada97faac + .quad 0x10ab8fa1ad32b1d0 + .quad 0xe9aced1be2778b24 + .quad 0xa8856bc0373de90f + .quad 0x66f35ddddda53996 + .quad 0xd27d9afb24997323 + .quad 0x1bb7e07ef6f01d2e + .quad 0x2ba7472df52ecc7f + .quad 0x03019b4f646f9dc8 + + // 2^148 * 6 * G + + .quad 0x04a186b5565345cd + .quad 0xeee76610bcc4116a + .quad 0x689c73b478fb2a45 + .quad 0x387dcbff65697512 + .quad 0xaf09b214e6b3dc6b + .quad 0x3f7573b5ad7d2f65 + .quad 0xd019d988100a23b0 + .quad 0x392b63a58b5c35f7 + .quad 0x4093addc9c07c205 + .quad 0xc565be15f532c37e + .quad 0x63dbecfd1583402a + .quad 0x61722b4aef2e032e + + // 2^148 * 7 * G + + .quad 0x0012aafeecbd47af + .quad 0x55a266fb1cd46309 + .quad 0xf203eb680967c72c + .quad 0x39633944ca3c1429 + .quad 0xd6b07a5581cb0e3c + .quad 0x290ff006d9444969 + .quad 0x08680b6a16dcda1f + .quad 0x5568d2b75a06de59 + .quad 0x8d0cb88c1b37cfe1 + .quad 0x05b6a5a3053818f3 + .quad 0xf2e9bc04b787d959 + .quad 0x6beba1249add7f64 + + // 2^148 * 8 * G + + .quad 0x1d06005ca5b1b143 + .quad 0x6d4c6bb87fd1cda2 + .quad 0x6ef5967653fcffe7 + .quad 0x097c29e8c1ce1ea5 + .quad 0x5c3cecb943f5a53b + .quad 0x9cc9a61d06c08df2 + .quad 0xcfba639a85895447 + .quad 0x5a845ae80df09fd5 + .quad 0x4ce97dbe5deb94ca + .quad 0x38d0a4388c709c48 + .quad 0xc43eced4a169d097 + .quad 0x0a1249fff7e587c3 + + // 2^152 * 1 * G + + .quad 0x12f0071b276d01c9 + .quad 0xe7b8bac586c48c70 + .quad 0x5308129b71d6fba9 + .quad 0x5d88fbf95a3db792 + .quad 0x0b408d9e7354b610 + .quad 0x806b32535ba85b6e + .quad 0xdbe63a034a58a207 + .quad 0x173bd9ddc9a1df2c + .quad 0x2b500f1efe5872df + .quad 0x58d6582ed43918c1 + .quad 0xe6ed278ec9673ae0 + .quad 0x06e1cd13b19ea319 + + // 2^152 * 2 * G + + .quad 0x40d0ad516f166f23 + .quad 0x118e32931fab6abe + .quad 0x3fe35e14a04d088e + .quad 0x3080603526e16266 + .quad 0x472baf629e5b0353 + .quad 0x3baa0b90278d0447 + .quad 0x0c785f469643bf27 + .quad 0x7f3a6a1a8d837b13 + .quad 0xf7e644395d3d800b + .quad 0x95a8d555c901edf6 + .quad 0x68cd7830592c6339 + .quad 0x30d0fded2e51307e + + // 2^152 * 3 * G + + .quad 0xe0594d1af21233b3 + .quad 0x1bdbe78ef0cc4d9c + .quad 0x6965187f8f499a77 + .quad 0x0a9214202c099868 + .quad 0x9cb4971e68b84750 + .quad 0xa09572296664bbcf + .quad 0x5c8de72672fa412b + .quad 0x4615084351c589d9 + .quad 0xbc9019c0aeb9a02e + .quad 0x55c7110d16034cae + .quad 0x0e6df501659932ec + .quad 0x3bca0d2895ca5dfe + + // 2^152 * 4 * G + + .quad 0x40f031bc3c5d62a4 + .quad 0x19fc8b3ecff07a60 + .quad 0x98183da2130fb545 + .quad 0x5631deddae8f13cd + .quad 0x9c688eb69ecc01bf + .quad 0xf0bc83ada644896f + .quad 0xca2d955f5f7a9fe2 + .quad 0x4ea8b4038df28241 + .quad 0x2aed460af1cad202 + .quad 0x46305305a48cee83 + .quad 0x9121774549f11a5f + .quad 0x24ce0930542ca463 + + // 2^152 * 5 * G + + .quad 0x1fe890f5fd06c106 + .quad 0xb5c468355d8810f2 + .quad 0x827808fe6e8caf3e + .quad 0x41d4e3c28a06d74b + .quad 0x3fcfa155fdf30b85 + .quad 0xd2f7168e36372ea4 + .quad 0xb2e064de6492f844 + .quad 0x549928a7324f4280 + .quad 0xf26e32a763ee1a2e + .quad 0xae91e4b7d25ffdea + .quad 0xbc3bd33bd17f4d69 + .quad 0x491b66dec0dcff6a + + // 2^152 * 6 * G + + .quad 0x98f5b13dc7ea32a7 + .quad 0xe3d5f8cc7e16db98 + .quad 0xac0abf52cbf8d947 + .quad 0x08f338d0c85ee4ac + .quad 0x75f04a8ed0da64a1 + .quad 0xed222caf67e2284b + .quad 0x8234a3791f7b7ba4 + .quad 0x4cf6b8b0b7018b67 + .quad 0xc383a821991a73bd + .quad 0xab27bc01df320c7a + .quad 0xc13d331b84777063 + .quad 0x530d4a82eb078a99 + + // 2^152 * 7 * G + + .quad 0x004c3630e1f94825 + .quad 0x7e2d78268cab535a + .quad 0xc7482323cc84ff8b + .quad 0x65ea753f101770b9 + .quad 0x6d6973456c9abf9e + .quad 0x257fb2fc4900a880 + .quad 0x2bacf412c8cfb850 + .quad 0x0db3e7e00cbfbd5b + .quad 0x3d66fc3ee2096363 + .quad 0x81d62c7f61b5cb6b + .quad 0x0fbe044213443b1a + .quad 0x02a4ec1921e1a1db + + // 2^152 * 8 * G + + .quad 0x5ce6259a3b24b8a2 + .quad 0xb8577acc45afa0b8 + .quad 0xcccbe6e88ba07037 + .quad 0x3d143c51127809bf + .quad 0xf5c86162f1cf795f + .quad 0x118c861926ee57f2 + .quad 0x172124851c063578 + .quad 0x36d12b5dec067fcf + .quad 0x126d279179154557 + .quad 0xd5e48f5cfc783a0a + .quad 0x36bdb6e8df179bac + .quad 0x2ef517885ba82859 + + // 2^156 * 1 * G + + .quad 0x88bd438cd11e0d4a + .quad 0x30cb610d43ccf308 + .quad 0xe09a0e3791937bcc + .quad 0x4559135b25b1720c + .quad 0x1ea436837c6da1e9 + .quad 0xf9c189af1fb9bdbe + .quad 0x303001fcce5dd155 + .quad 0x28a7c99ebc57be52 + .quad 0xb8fd9399e8d19e9d + .quad 0x908191cb962423ff + .quad 0xb2b948d747c742a3 + .quad 0x37f33226d7fb44c4 + + // 2^156 * 2 * G + + .quad 0x0dae8767b55f6e08 + .quad 0x4a43b3b35b203a02 + .quad 0xe3725a6e80af8c79 + .quad 0x0f7a7fd1705fa7a3 + .quad 0x33912553c821b11d + .quad 0x66ed42c241e301df + .quad 0x066fcc11104222fd + .quad 0x307a3b41c192168f + .quad 0x8eeb5d076eb55ce0 + .quad 0x2fc536bfaa0d925a + .quad 0xbe81830fdcb6c6e8 + .quad 0x556c7045827baf52 + + // 2^156 * 3 * G + + .quad 0x8e2b517302e9d8b7 + .quad 0xe3e52269248714e8 + .quad 0xbd4fbd774ca960b5 + .quad 0x6f4b4199c5ecada9 + .quad 0xb94b90022bf44406 + .quad 0xabd4237eff90b534 + .quad 0x7600a960faf86d3a + .quad 0x2f45abdac2322ee3 + .quad 0x61af4912c8ef8a6a + .quad 0xe58fa4fe43fb6e5e + .quad 0xb5afcc5d6fd427cf + .quad 0x6a5393281e1e11eb + + // 2^156 * 4 * G + + .quad 0xf3da5139a5d1ee89 + .quad 0x8145457cff936988 + .quad 0x3f622fed00e188c4 + .quad 0x0f513815db8b5a3d + .quad 0x0fff04fe149443cf + .quad 0x53cac6d9865cddd7 + .quad 0x31385b03531ed1b7 + .quad 0x5846a27cacd1039d + .quad 0x4ff5cdac1eb08717 + .quad 0x67e8b29590f2e9bc + .quad 0x44093b5e237afa99 + .quad 0x0d414bed8708b8b2 + + // 2^156 * 5 * G + + .quad 0xcfb68265fd0e75f6 + .quad 0xe45b3e28bb90e707 + .quad 0x7242a8de9ff92c7a + .quad 0x685b3201933202dd + .quad 0x81886a92294ac9e8 + .quad 0x23162b45d55547be + .quad 0x94cfbc4403715983 + .quad 0x50eb8fdb134bc401 + .quad 0xc0b73ec6d6b330cd + .quad 0x84e44807132faff1 + .quad 0x732b7352c4a5dee1 + .quad 0x5d7c7cf1aa7cd2d2 + + // 2^156 * 6 * G + + .quad 0xaf3b46bf7a4aafa2 + .quad 0xb78705ec4d40d411 + .quad 0x114f0c6aca7c15e3 + .quad 0x3f364faaa9489d4d + .quad 0x33d1013e9b73a562 + .quad 0x925cef5748ec26e1 + .quad 0xa7fce614dd468058 + .quad 0x78b0fad41e9aa438 + .quad 0xbf56a431ed05b488 + .quad 0xa533e66c9c495c7e + .quad 0xe8652baf87f3651a + .quad 0x0241800059d66c33 + + // 2^156 * 7 * G + + .quad 0xceb077fea37a5be4 + .quad 0xdb642f02e5a5eeb7 + .quad 0xc2e6d0c5471270b8 + .quad 0x4771b65538e4529c + .quad 0x28350c7dcf38ea01 + .quad 0x7c6cdbc0b2917ab6 + .quad 0xace7cfbe857082f7 + .quad 0x4d2845aba2d9a1e0 + .quad 0xbb537fe0447070de + .quad 0xcba744436dd557df + .quad 0xd3b5a3473600dbcb + .quad 0x4aeabbe6f9ffd7f8 + + // 2^156 * 8 * G + + .quad 0x4630119e40d8f78c + .quad 0xa01a9bc53c710e11 + .quad 0x486d2b258910dd79 + .quad 0x1e6c47b3db0324e5 + .quad 0x6a2134bcc4a9c8f2 + .quad 0xfbf8fd1c8ace2e37 + .quad 0x000ae3049911a0ba + .quad 0x046e3a616bc89b9e + .quad 0x14e65442f03906be + .quad 0x4a019d54e362be2a + .quad 0x68ccdfec8dc230c7 + .quad 0x7cfb7e3faf6b861c + + // 2^160 * 1 * G + + .quad 0x4637974e8c58aedc + .quad 0xb9ef22fbabf041a4 + .quad 0xe185d956e980718a + .quad 0x2f1b78fab143a8a6 + .quad 0x96eebffb305b2f51 + .quad 0xd3f938ad889596b8 + .quad 0xf0f52dc746d5dd25 + .quad 0x57968290bb3a0095 + .quad 0xf71ab8430a20e101 + .quad 0xf393658d24f0ec47 + .quad 0xcf7509a86ee2eed1 + .quad 0x7dc43e35dc2aa3e1 + + // 2^160 * 2 * G + + .quad 0x85966665887dd9c3 + .quad 0xc90f9b314bb05355 + .quad 0xc6e08df8ef2079b1 + .quad 0x7ef72016758cc12f + .quad 0x5a782a5c273e9718 + .quad 0x3576c6995e4efd94 + .quad 0x0f2ed8051f237d3e + .quad 0x044fb81d82d50a99 + .quad 0xc1df18c5a907e3d9 + .quad 0x57b3371dce4c6359 + .quad 0xca704534b201bb49 + .quad 0x7f79823f9c30dd2e + + // 2^160 * 3 * G + + .quad 0x8334d239a3b513e8 + .quad 0xc13670d4b91fa8d8 + .quad 0x12b54136f590bd33 + .quad 0x0a4e0373d784d9b4 + .quad 0x6a9c1ff068f587ba + .quad 0x0827894e0050c8de + .quad 0x3cbf99557ded5be7 + .quad 0x64a9b0431c06d6f0 + .quad 0x2eb3d6a15b7d2919 + .quad 0xb0b4f6a0d53a8235 + .quad 0x7156ce4389a45d47 + .quad 0x071a7d0ace18346c + + // 2^160 * 4 * G + + .quad 0xd3072daac887ba0b + .quad 0x01262905bfa562ee + .quad 0xcf543002c0ef768b + .quad 0x2c3bcc7146ea7e9c + .quad 0xcc0c355220e14431 + .quad 0x0d65950709b15141 + .quad 0x9af5621b209d5f36 + .quad 0x7c69bcf7617755d3 + .quad 0x07f0d7eb04e8295f + .quad 0x10db18252f50f37d + .quad 0xe951a9a3171798d7 + .quad 0x6f5a9a7322aca51d + + // 2^160 * 5 * G + + .quad 0x8ba1000c2f41c6c5 + .quad 0xc49f79c10cfefb9b + .quad 0x4efa47703cc51c9f + .quad 0x494e21a2e147afca + .quad 0xe729d4eba3d944be + .quad 0x8d9e09408078af9e + .quad 0x4525567a47869c03 + .quad 0x02ab9680ee8d3b24 + .quad 0xefa48a85dde50d9a + .quad 0x219a224e0fb9a249 + .quad 0xfa091f1dd91ef6d9 + .quad 0x6b5d76cbea46bb34 + + // 2^160 * 6 * G + + .quad 0x8857556cec0cd994 + .quad 0x6472dc6f5cd01dba + .quad 0xaf0169148f42b477 + .quad 0x0ae333f685277354 + .quad 0xe0f941171e782522 + .quad 0xf1e6ae74036936d3 + .quad 0x408b3ea2d0fcc746 + .quad 0x16fb869c03dd313e + .quad 0x288e199733b60962 + .quad 0x24fc72b4d8abe133 + .quad 0x4811f7ed0991d03e + .quad 0x3f81e38b8f70d075 + + // 2^160 * 7 * G + + .quad 0x7f910fcc7ed9affe + .quad 0x545cb8a12465874b + .quad 0xa8397ed24b0c4704 + .quad 0x50510fc104f50993 + .quad 0x0adb7f355f17c824 + .quad 0x74b923c3d74299a4 + .quad 0xd57c3e8bcbf8eaf7 + .quad 0x0ad3e2d34cdedc3d + .quad 0x6f0c0fc5336e249d + .quad 0x745ede19c331cfd9 + .quad 0xf2d6fd0009eefe1c + .quad 0x127c158bf0fa1ebe + + // 2^160 * 8 * G + + .quad 0xf6197c422e9879a2 + .quad 0xa44addd452ca3647 + .quad 0x9b413fc14b4eaccb + .quad 0x354ef87d07ef4f68 + .quad 0xdea28fc4ae51b974 + .quad 0x1d9973d3744dfe96 + .quad 0x6240680b873848a8 + .quad 0x4ed82479d167df95 + .quad 0xfee3b52260c5d975 + .quad 0x50352efceb41b0b8 + .quad 0x8808ac30a9f6653c + .quad 0x302d92d20539236d + + // 2^164 * 1 * G + + .quad 0x4c59023fcb3efb7c + .quad 0x6c2fcb99c63c2a94 + .quad 0xba4190e2c3c7e084 + .quad 0x0e545daea51874d9 + .quad 0x957b8b8b0df53c30 + .quad 0x2a1c770a8e60f098 + .quad 0xbbc7a670345796de + .quad 0x22a48f9a90c99bc9 + .quad 0x6b7dc0dc8d3fac58 + .quad 0x5497cd6ce6e42bfd + .quad 0x542f7d1bf400d305 + .quad 0x4159f47f048d9136 + + // 2^164 * 2 * G + + .quad 0x20ad660839e31e32 + .quad 0xf81e1bd58405be50 + .quad 0xf8064056f4dabc69 + .quad 0x14d23dd4ce71b975 + .quad 0x748515a8bbd24839 + .quad 0x77128347afb02b55 + .quad 0x50ba2ac649a2a17f + .quad 0x060525513ad730f1 + .quad 0xf2398e098aa27f82 + .quad 0x6d7982bb89a1b024 + .quad 0xfa694084214dd24c + .quad 0x71ab966fa32301c3 + + // 2^164 * 3 * G + + .quad 0x2dcbd8e34ded02fc + .quad 0x1151f3ec596f22aa + .quad 0xbca255434e0328da + .quad 0x35768fbe92411b22 + .quad 0xb1088a0702809955 + .quad 0x43b273ea0b43c391 + .quad 0xca9b67aefe0686ed + .quad 0x605eecbf8335f4ed + .quad 0x83200a656c340431 + .quad 0x9fcd71678ee59c2f + .quad 0x75d4613f71300f8a + .quad 0x7a912faf60f542f9 + + // 2^164 * 4 * G + + .quad 0xb204585e5edc1a43 + .quad 0x9f0e16ee5897c73c + .quad 0x5b82c0ae4e70483c + .quad 0x624a170e2bddf9be + .quad 0x253f4f8dfa2d5597 + .quad 0x25e49c405477130c + .quad 0x00c052e5996b1102 + .quad 0x33cb966e33bb6c4a + .quad 0x597028047f116909 + .quad 0x828ac41c1e564467 + .quad 0x70417dbde6217387 + .quad 0x721627aefbac4384 + + // 2^164 * 5 * G + + .quad 0x97d03bc38736add5 + .quad 0x2f1422afc532b130 + .quad 0x3aa68a057101bbc4 + .quad 0x4c946cf7e74f9fa7 + .quad 0xfd3097bc410b2f22 + .quad 0xf1a05da7b5cfa844 + .quad 0x61289a1def57ca74 + .quad 0x245ea199bb821902 + .quad 0xaedca66978d477f8 + .quad 0x1898ba3c29117fe1 + .quad 0xcf73f983720cbd58 + .quad 0x67da12e6b8b56351 + + // 2^164 * 6 * G + + .quad 0x7067e187b4bd6e07 + .quad 0x6e8f0203c7d1fe74 + .quad 0x93c6aa2f38c85a30 + .quad 0x76297d1f3d75a78a + .quad 0x2b7ef3d38ec8308c + .quad 0x828fd7ec71eb94ab + .quad 0x807c3b36c5062abd + .quad 0x0cb64cb831a94141 + .quad 0x3030fc33534c6378 + .quad 0xb9635c5ce541e861 + .quad 0x15d9a9bed9b2c728 + .quad 0x49233ea3f3775dcb + + // 2^164 * 7 * G + + .quad 0x629398fa8dbffc3a + .quad 0xe12fe52dd54db455 + .quad 0xf3be11dfdaf25295 + .quad 0x628b140dce5e7b51 + .quad 0x7b3985fe1c9f249b + .quad 0x4fd6b2d5a1233293 + .quad 0xceb345941adf4d62 + .quad 0x6987ff6f542de50c + .quad 0x47e241428f83753c + .quad 0x6317bebc866af997 + .quad 0xdabb5b433d1a9829 + .quad 0x074d8d245287fb2d + + // 2^164 * 8 * G + + .quad 0x8337d9cd440bfc31 + .quad 0x729d2ca1af318fd7 + .quad 0xa040a4a4772c2070 + .quad 0x46002ef03a7349be + .quad 0x481875c6c0e31488 + .quad 0x219429b2e22034b4 + .quad 0x7223c98a31283b65 + .quad 0x3420d60b342277f9 + .quad 0xfaa23adeaffe65f7 + .quad 0x78261ed45be0764c + .quad 0x441c0a1e2f164403 + .quad 0x5aea8e567a87d395 + + // 2^168 * 1 * G + + .quad 0x7813c1a2bca4283d + .quad 0xed62f091a1863dd9 + .quad 0xaec7bcb8c268fa86 + .quad 0x10e5d3b76f1cae4c + .quad 0x2dbc6fb6e4e0f177 + .quad 0x04e1bf29a4bd6a93 + .quad 0x5e1966d4787af6e8 + .quad 0x0edc5f5eb426d060 + .quad 0x5453bfd653da8e67 + .quad 0xe9dc1eec24a9f641 + .quad 0xbf87263b03578a23 + .quad 0x45b46c51361cba72 + + // 2^168 * 2 * G + + .quad 0xa9402abf314f7fa1 + .quad 0xe257f1dc8e8cf450 + .quad 0x1dbbd54b23a8be84 + .quad 0x2177bfa36dcb713b + .quad 0xce9d4ddd8a7fe3e4 + .quad 0xab13645676620e30 + .quad 0x4b594f7bb30e9958 + .quad 0x5c1c0aef321229df + .quad 0x37081bbcfa79db8f + .quad 0x6048811ec25f59b3 + .quad 0x087a76659c832487 + .quad 0x4ae619387d8ab5bb + + // 2^168 * 3 * G + + .quad 0x8ddbf6aa5344a32e + .quad 0x7d88eab4b41b4078 + .quad 0x5eb0eb974a130d60 + .quad 0x1a00d91b17bf3e03 + .quad 0x61117e44985bfb83 + .quad 0xfce0462a71963136 + .quad 0x83ac3448d425904b + .quad 0x75685abe5ba43d64 + .quad 0x6e960933eb61f2b2 + .quad 0x543d0fa8c9ff4952 + .quad 0xdf7275107af66569 + .quad 0x135529b623b0e6aa + + // 2^168 * 4 * G + + .quad 0x18f0dbd7add1d518 + .quad 0x979f7888cfc11f11 + .quad 0x8732e1f07114759b + .quad 0x79b5b81a65ca3a01 + .quad 0xf5c716bce22e83fe + .quad 0xb42beb19e80985c1 + .quad 0xec9da63714254aae + .quad 0x5972ea051590a613 + .quad 0x0fd4ac20dc8f7811 + .quad 0x9a9ad294ac4d4fa8 + .quad 0xc01b2d64b3360434 + .quad 0x4f7e9c95905f3bdb + + // 2^168 * 5 * G + + .quad 0x62674bbc5781302e + .quad 0xd8520f3989addc0f + .quad 0x8c2999ae53fbd9c6 + .quad 0x31993ad92e638e4c + .quad 0x71c8443d355299fe + .quad 0x8bcd3b1cdbebead7 + .quad 0x8092499ef1a49466 + .quad 0x1942eec4a144adc8 + .quad 0x7dac5319ae234992 + .quad 0x2c1b3d910cea3e92 + .quad 0x553ce494253c1122 + .quad 0x2a0a65314ef9ca75 + + // 2^168 * 6 * G + + .quad 0x2db7937ff7f927c2 + .quad 0xdb741f0617d0a635 + .quad 0x5982f3a21155af76 + .quad 0x4cf6e218647c2ded + .quad 0xcf361acd3c1c793a + .quad 0x2f9ebcac5a35bc3b + .quad 0x60e860e9a8cda6ab + .quad 0x055dc39b6dea1a13 + .quad 0xb119227cc28d5bb6 + .quad 0x07e24ebc774dffab + .quad 0xa83c78cee4a32c89 + .quad 0x121a307710aa24b6 + + // 2^168 * 7 * G + + .quad 0xe4db5d5e9f034a97 + .quad 0xe153fc093034bc2d + .quad 0x460546919551d3b1 + .quad 0x333fc76c7a40e52d + .quad 0xd659713ec77483c9 + .quad 0x88bfe077b82b96af + .quad 0x289e28231097bcd3 + .quad 0x527bb94a6ced3a9b + .quad 0x563d992a995b482e + .quad 0x3405d07c6e383801 + .quad 0x485035de2f64d8e5 + .quad 0x6b89069b20a7a9f7 + + // 2^168 * 8 * G + + .quad 0x812aa0416270220d + .quad 0x995a89faf9245b4e + .quad 0xffadc4ce5072ef05 + .quad 0x23bc2103aa73eb73 + .quad 0x4082fa8cb5c7db77 + .quad 0x068686f8c734c155 + .quad 0x29e6c8d9f6e7a57e + .quad 0x0473d308a7639bcf + .quad 0xcaee792603589e05 + .quad 0x2b4b421246dcc492 + .quad 0x02a1ef74e601a94f + .quad 0x102f73bfde04341a + + // 2^172 * 1 * G + + .quad 0xb5a2d50c7ec20d3e + .quad 0xc64bdd6ea0c97263 + .quad 0x56e89052c1ff734d + .quad 0x4929c6f72b2ffaba + .quad 0x358ecba293a36247 + .quad 0xaf8f9862b268fd65 + .quad 0x412f7e9968a01c89 + .quad 0x5786f312cd754524 + .quad 0x337788ffca14032c + .quad 0xf3921028447f1ee3 + .quad 0x8b14071f231bccad + .quad 0x4c817b4bf2344783 + + // 2^172 * 2 * G + + .quad 0x0ff853852871b96e + .quad 0xe13e9fab60c3f1bb + .quad 0xeefd595325344402 + .quad 0x0a37c37075b7744b + .quad 0x413ba057a40b4484 + .quad 0xba4c2e1a4f5f6a43 + .quad 0x614ba0a5aee1d61c + .quad 0x78a1531a8b05dc53 + .quad 0x6cbdf1703ad0562b + .quad 0x8ecf4830c92521a3 + .quad 0xdaebd303fd8424e7 + .quad 0x72ad82a42e5ec56f + + // 2^172 * 3 * G + + .quad 0x3f9e8e35bafb65f6 + .quad 0x39d69ec8f27293a1 + .quad 0x6cb8cd958cf6a3d0 + .quad 0x1734778173adae6d + .quad 0xc368939167024bc3 + .quad 0x8e69d16d49502fda + .quad 0xfcf2ec3ce45f4b29 + .quad 0x065f669ea3b4cbc4 + .quad 0x8a00aec75532db4d + .quad 0xb869a4e443e31bb1 + .quad 0x4a0f8552d3a7f515 + .quad 0x19adeb7c303d7c08 + + // 2^172 * 4 * G + + .quad 0xc720cb6153ead9a3 + .quad 0x55b2c97f512b636e + .quad 0xb1e35b5fd40290b1 + .quad 0x2fd9ccf13b530ee2 + .quad 0x9d05ba7d43c31794 + .quad 0x2470c8ff93322526 + .quad 0x8323dec816197438 + .quad 0x2852709881569b53 + .quad 0x07bd475b47f796b8 + .quad 0xd2c7b013542c8f54 + .quad 0x2dbd23f43b24f87e + .quad 0x6551afd77b0901d6 + + // 2^172 * 5 * G + + .quad 0x4546baaf54aac27f + .quad 0xf6f66fecb2a45a28 + .quad 0x582d1b5b562bcfe8 + .quad 0x44b123f3920f785f + .quad 0x68a24ce3a1d5c9ac + .quad 0xbb77a33d10ff6461 + .quad 0x0f86ce4425d3166e + .quad 0x56507c0950b9623b + .quad 0x1206f0b7d1713e63 + .quad 0x353fe3d915bafc74 + .quad 0x194ceb970ad9d94d + .quad 0x62fadd7cf9d03ad3 + + // 2^172 * 6 * G + + .quad 0xc6b5967b5598a074 + .quad 0x5efe91ce8e493e25 + .quad 0xd4b72c4549280888 + .quad 0x20ef1149a26740c2 + .quad 0x3cd7bc61e7ce4594 + .quad 0xcd6b35a9b7dd267e + .quad 0xa080abc84366ef27 + .quad 0x6ec7c46f59c79711 + .quad 0x2f07ad636f09a8a2 + .quad 0x8697e6ce24205e7d + .quad 0xc0aefc05ee35a139 + .quad 0x15e80958b5f9d897 + + // 2^172 * 7 * G + + .quad 0x25a5ef7d0c3e235b + .quad 0x6c39c17fbe134ee7 + .quad 0xc774e1342dc5c327 + .quad 0x021354b892021f39 + .quad 0x4dd1ed355bb061c4 + .quad 0x42dc0cef941c0700 + .quad 0x61305dc1fd86340e + .quad 0x56b2cc930e55a443 + .quad 0x1df79da6a6bfc5a2 + .quad 0x02f3a2749fde4369 + .quad 0xb323d9f2cda390a7 + .quad 0x7be0847b8774d363 + + // 2^172 * 8 * G + + .quad 0x8c99cc5a8b3f55c3 + .quad 0x0611d7253fded2a0 + .quad 0xed2995ff36b70a36 + .quad 0x1f699a54d78a2619 + .quad 0x1466f5af5307fa11 + .quad 0x817fcc7ded6c0af2 + .quad 0x0a6de44ec3a4a3fb + .quad 0x74071475bc927d0b + .quad 0xe77292f373e7ea8a + .quad 0x296537d2cb045a31 + .quad 0x1bd0653ed3274fde + .quad 0x2f9a2c4476bd2966 + + // 2^176 * 1 * G + + .quad 0xeb18b9ab7f5745c6 + .quad 0x023a8aee5787c690 + .quad 0xb72712da2df7afa9 + .quad 0x36597d25ea5c013d + .quad 0xa2b4dae0b5511c9a + .quad 0x7ac860292bffff06 + .quad 0x981f375df5504234 + .quad 0x3f6bd725da4ea12d + .quad 0x734d8d7b106058ac + .quad 0xd940579e6fc6905f + .quad 0x6466f8f99202932d + .quad 0x7b7ecc19da60d6d0 + + // 2^176 * 2 * G + + .quad 0x78c2373c695c690d + .quad 0xdd252e660642906e + .quad 0x951d44444ae12bd2 + .quad 0x4235ad7601743956 + .quad 0x6dae4a51a77cfa9b + .quad 0x82263654e7a38650 + .quad 0x09bbffcd8f2d82db + .quad 0x03bedc661bf5caba + .quad 0x6258cb0d078975f5 + .quad 0x492942549189f298 + .quad 0xa0cab423e2e36ee4 + .quad 0x0e7ce2b0cdf066a1 + + // 2^176 * 3 * G + + .quad 0xc494643ac48c85a3 + .quad 0xfd361df43c6139ad + .quad 0x09db17dd3ae94d48 + .quad 0x666e0a5d8fb4674a + .quad 0xfea6fedfd94b70f9 + .quad 0xf130c051c1fcba2d + .quad 0x4882d47e7f2fab89 + .quad 0x615256138aeceeb5 + .quad 0x2abbf64e4870cb0d + .quad 0xcd65bcf0aa458b6b + .quad 0x9abe4eba75e8985d + .quad 0x7f0bc810d514dee4 + + // 2^176 * 4 * G + + .quad 0xb9006ba426f4136f + .quad 0x8d67369e57e03035 + .quad 0xcbc8dfd94f463c28 + .quad 0x0d1f8dbcf8eedbf5 + .quad 0x83ac9dad737213a0 + .quad 0x9ff6f8ba2ef72e98 + .quad 0x311e2edd43ec6957 + .quad 0x1d3a907ddec5ab75 + .quad 0xba1693313ed081dc + .quad 0x29329fad851b3480 + .quad 0x0128013c030321cb + .quad 0x00011b44a31bfde3 + + // 2^176 * 5 * G + + .quad 0x3fdfa06c3fc66c0c + .quad 0x5d40e38e4dd60dd2 + .quad 0x7ae38b38268e4d71 + .quad 0x3ac48d916e8357e1 + .quad 0x16561f696a0aa75c + .quad 0xc1bf725c5852bd6a + .quad 0x11a8dd7f9a7966ad + .quad 0x63d988a2d2851026 + .quad 0x00120753afbd232e + .quad 0xe92bceb8fdd8f683 + .quad 0xf81669b384e72b91 + .quad 0x33fad52b2368a066 + + // 2^176 * 6 * G + + .quad 0x540649c6c5e41e16 + .quad 0x0af86430333f7735 + .quad 0xb2acfcd2f305e746 + .quad 0x16c0f429a256dca7 + .quad 0x8d2cc8d0c422cfe8 + .quad 0x072b4f7b05a13acb + .quad 0xa3feb6e6ecf6a56f + .quad 0x3cc355ccb90a71e2 + .quad 0xe9b69443903e9131 + .quad 0xb8a494cb7a5637ce + .quad 0xc87cd1a4baba9244 + .quad 0x631eaf426bae7568 + + // 2^176 * 7 * G + + .quad 0xb3e90410da66fe9f + .quad 0x85dd4b526c16e5a6 + .quad 0xbc3d97611ef9bf83 + .quad 0x5599648b1ea919b5 + .quad 0x47d975b9a3700de8 + .quad 0x7280c5fbe2f80552 + .quad 0x53658f2732e45de1 + .quad 0x431f2c7f665f80b5 + .quad 0xd6026344858f7b19 + .quad 0x14ab352fa1ea514a + .quad 0x8900441a2090a9d7 + .quad 0x7b04715f91253b26 + + // 2^176 * 8 * G + + .quad 0x83edbd28acf6ae43 + .quad 0x86357c8b7d5c7ab4 + .quad 0xc0404769b7eb2c44 + .quad 0x59b37bf5c2f6583f + .quad 0xb376c280c4e6bac6 + .quad 0x970ed3dd6d1d9b0b + .quad 0xb09a9558450bf944 + .quad 0x48d0acfa57cde223 + .quad 0xb60f26e47dabe671 + .quad 0xf1d1a197622f3a37 + .quad 0x4208ce7ee9960394 + .quad 0x16234191336d3bdb + + // 2^180 * 1 * G + + .quad 0xf19aeac733a63aef + .quad 0x2c7fba5d4442454e + .quad 0x5da87aa04795e441 + .quad 0x413051e1a4e0b0f5 + .quad 0x852dd1fd3d578bbe + .quad 0x2b65ce72c3286108 + .quad 0x658c07f4eace2273 + .quad 0x0933f804ec38ab40 + .quad 0xa7ab69798d496476 + .quad 0x8121aadefcb5abc8 + .quad 0xa5dc12ef7b539472 + .quad 0x07fd47065e45351a + + // 2^180 * 2 * G + + .quad 0xc8583c3d258d2bcd + .quad 0x17029a4daf60b73f + .quad 0xfa0fc9d6416a3781 + .quad 0x1c1e5fba38b3fb23 + .quad 0x304211559ae8e7c3 + .quad 0xf281b229944882a5 + .quad 0x8a13ac2e378250e4 + .quad 0x014afa0954ba48f4 + .quad 0xcb3197001bb3666c + .quad 0x330060524bffecb9 + .quad 0x293711991a88233c + .quad 0x291884363d4ed364 + + // 2^180 * 3 * G + + .quad 0x033c6805dc4babfa + .quad 0x2c15bf5e5596ecc1 + .quad 0x1bc70624b59b1d3b + .quad 0x3ede9850a19f0ec5 + .quad 0xfb9d37c3bc1ab6eb + .quad 0x02be14534d57a240 + .quad 0xf4d73415f8a5e1f6 + .quad 0x5964f4300ccc8188 + .quad 0xe44a23152d096800 + .quad 0x5c08c55970866996 + .quad 0xdf2db60a46affb6e + .quad 0x579155c1f856fd89 + + // 2^180 * 4 * G + + .quad 0x96324edd12e0c9ef + .quad 0x468b878df2420297 + .quad 0x199a3776a4f573be + .quad 0x1e7fbcf18e91e92a + .quad 0xb5f16b630817e7a6 + .quad 0x808c69233c351026 + .quad 0x324a983b54cef201 + .quad 0x53c092084a485345 + .quad 0xd2d41481f1cbafbf + .quad 0x231d2db6716174e5 + .quad 0x0b7d7656e2a55c98 + .quad 0x3e955cd82aa495f6 + + // 2^180 * 5 * G + + .quad 0xe48f535e3ed15433 + .quad 0xd075692a0d7270a3 + .quad 0x40fbd21daade6387 + .quad 0x14264887cf4495f5 + .quad 0xab39f3ef61bb3a3f + .quad 0x8eb400652eb9193e + .quad 0xb5de6ecc38c11f74 + .quad 0x654d7e9626f3c49f + .quad 0xe564cfdd5c7d2ceb + .quad 0x82eeafded737ccb9 + .quad 0x6107db62d1f9b0ab + .quad 0x0b6baac3b4358dbb + + // 2^180 * 6 * G + + .quad 0x7ae62bcb8622fe98 + .quad 0x47762256ceb891af + .quad 0x1a5a92bcf2e406b4 + .quad 0x7d29401784e41501 + .quad 0x204abad63700a93b + .quad 0xbe0023d3da779373 + .quad 0xd85f0346633ab709 + .quad 0x00496dc490820412 + .quad 0x1c74b88dc27e6360 + .quad 0x074854268d14850c + .quad 0xa145fb7b3e0dcb30 + .quad 0x10843f1b43803b23 + + // 2^180 * 7 * G + + .quad 0xc5f90455376276dd + .quad 0xce59158dd7645cd9 + .quad 0x92f65d511d366b39 + .quad 0x11574b6e526996c4 + .quad 0xd56f672de324689b + .quad 0xd1da8aedb394a981 + .quad 0xdd7b58fe9168cfed + .quad 0x7ce246cd4d56c1e8 + .quad 0xb8f4308e7f80be53 + .quad 0x5f3cb8cb34a9d397 + .quad 0x18a961bd33cc2b2c + .quad 0x710045fb3a9af671 + + // 2^180 * 8 * G + + .quad 0x73f93d36101b95eb + .quad 0xfaef33794f6f4486 + .quad 0x5651735f8f15e562 + .quad 0x7fa3f19058b40da1 + .quad 0xa03fc862059d699e + .quad 0x2370cfa19a619e69 + .quad 0xc4fe3b122f823deb + .quad 0x1d1b056fa7f0844e + .quad 0x1bc64631e56bf61f + .quad 0xd379ab106e5382a3 + .quad 0x4d58c57e0540168d + .quad 0x566256628442d8e4 + + // 2^184 * 1 * G + + .quad 0xb9e499def6267ff6 + .quad 0x7772ca7b742c0843 + .quad 0x23a0153fe9a4f2b1 + .quad 0x2cdfdfecd5d05006 + .quad 0xdd499cd61ff38640 + .quad 0x29cd9bc3063625a0 + .quad 0x51e2d8023dd73dc3 + .quad 0x4a25707a203b9231 + .quad 0x2ab7668a53f6ed6a + .quad 0x304242581dd170a1 + .quad 0x4000144c3ae20161 + .quad 0x5721896d248e49fc + + // 2^184 * 2 * G + + .quad 0x0b6e5517fd181bae + .quad 0x9022629f2bb963b4 + .quad 0x5509bce932064625 + .quad 0x578edd74f63c13da + .quad 0x285d5091a1d0da4e + .quad 0x4baa6fa7b5fe3e08 + .quad 0x63e5177ce19393b3 + .quad 0x03c935afc4b030fd + .quad 0x997276c6492b0c3d + .quad 0x47ccc2c4dfe205fc + .quad 0xdcd29b84dd623a3c + .quad 0x3ec2ab590288c7a2 + + // 2^184 * 3 * G + + .quad 0xa1a0d27be4d87bb9 + .quad 0xa98b4deb61391aed + .quad 0x99a0ddd073cb9b83 + .quad 0x2dd5c25a200fcace + .quad 0xa7213a09ae32d1cb + .quad 0x0f2b87df40f5c2d5 + .quad 0x0baea4c6e81eab29 + .quad 0x0e1bf66c6adbac5e + .quad 0xe2abd5e9792c887e + .quad 0x1a020018cb926d5d + .quad 0xbfba69cdbaae5f1e + .quad 0x730548b35ae88f5f + + // 2^184 * 4 * G + + .quad 0xc43551a3cba8b8ee + .quad 0x65a26f1db2115f16 + .quad 0x760f4f52ab8c3850 + .quad 0x3043443b411db8ca + .quad 0x805b094ba1d6e334 + .quad 0xbf3ef17709353f19 + .quad 0x423f06cb0622702b + .quad 0x585a2277d87845dd + .quad 0xa18a5f8233d48962 + .quad 0x6698c4b5ec78257f + .quad 0xa78e6fa5373e41ff + .quad 0x7656278950ef981f + + // 2^184 * 5 * G + + .quad 0x38c3cf59d51fc8c0 + .quad 0x9bedd2fd0506b6f2 + .quad 0x26bf109fab570e8f + .quad 0x3f4160a8c1b846a6 + .quad 0xe17073a3ea86cf9d + .quad 0x3a8cfbb707155fdc + .quad 0x4853e7fc31838a8e + .quad 0x28bbf484b613f616 + .quad 0xf2612f5c6f136c7c + .quad 0xafead107f6dd11be + .quad 0x527e9ad213de6f33 + .quad 0x1e79cb358188f75d + + // 2^184 * 6 * G + + .quad 0x013436c3eef7e3f1 + .quad 0x828b6a7ffe9e10f8 + .quad 0x7ff908e5bcf9defc + .quad 0x65d7951b3a3b3831 + .quad 0x77e953d8f5e08181 + .quad 0x84a50c44299dded9 + .quad 0xdc6c2d0c864525e5 + .quad 0x478ab52d39d1f2f4 + .quad 0x66a6a4d39252d159 + .quad 0xe5dde1bc871ac807 + .quad 0xb82c6b40a6c1c96f + .quad 0x16d87a411a212214 + + // 2^184 * 7 * G + + .quad 0xb3bd7e5a42066215 + .quad 0x879be3cd0c5a24c1 + .quad 0x57c05db1d6f994b7 + .quad 0x28f87c8165f38ca6 + .quad 0xfba4d5e2d54e0583 + .quad 0xe21fafd72ebd99fa + .quad 0x497ac2736ee9778f + .quad 0x1f990b577a5a6dde + .quad 0xa3344ead1be8f7d6 + .quad 0x7d1e50ebacea798f + .quad 0x77c6569e520de052 + .quad 0x45882fe1534d6d3e + + // 2^184 * 8 * G + + .quad 0x6669345d757983d6 + .quad 0x62b6ed1117aa11a6 + .quad 0x7ddd1857985e128f + .quad 0x688fe5b8f626f6dd + .quad 0xd8ac9929943c6fe4 + .quad 0xb5f9f161a38392a2 + .quad 0x2699db13bec89af3 + .quad 0x7dcf843ce405f074 + .quad 0x6c90d6484a4732c0 + .quad 0xd52143fdca563299 + .quad 0xb3be28c3915dc6e1 + .quad 0x6739687e7327191b + + // 2^188 * 1 * G + + .quad 0x9f65c5ea200814cf + .quad 0x840536e169a31740 + .quad 0x8b0ed13925c8b4ad + .quad 0x0080dbafe936361d + .quad 0x8ce5aad0c9cb971f + .quad 0x1156aaa99fd54a29 + .quad 0x41f7247015af9b78 + .quad 0x1fe8cca8420f49aa + .quad 0x72a1848f3c0cc82a + .quad 0x38c560c2877c9e54 + .quad 0x5004e228ce554140 + .quad 0x042418a103429d71 + + // 2^188 * 2 * G + + .quad 0x899dea51abf3ff5f + .quad 0x9b93a8672fc2d8ba + .quad 0x2c38cb97be6ebd5c + .quad 0x114d578497263b5d + .quad 0x58e84c6f20816247 + .quad 0x8db2b2b6e36fd793 + .quad 0x977182561d484d85 + .quad 0x0822024f8632abd7 + .quad 0xb301bb7c6b1beca3 + .quad 0x55393f6dc6eb1375 + .quad 0x910d281097b6e4eb + .quad 0x1ad4548d9d479ea3 + + // 2^188 * 3 * G + + .quad 0xcd5a7da0389a48fd + .quad 0xb38fa4aa9a78371e + .quad 0xc6d9761b2cdb8e6c + .quad 0x35cf51dbc97e1443 + .quad 0xa06fe66d0fe9fed3 + .quad 0xa8733a401c587909 + .quad 0x30d14d800df98953 + .quad 0x41ce5876c7b30258 + .quad 0x59ac3bc5d670c022 + .quad 0xeae67c109b119406 + .quad 0x9798bdf0b3782fda + .quad 0x651e3201fd074092 + + // 2^188 * 4 * G + + .quad 0xd63d8483ef30c5cf + .quad 0x4cd4b4962361cc0c + .quad 0xee90e500a48426ac + .quad 0x0af51d7d18c14eeb + .quad 0xa57ba4a01efcae9e + .quad 0x769f4beedc308a94 + .quad 0xd1f10eeb3603cb2e + .quad 0x4099ce5e7e441278 + .quad 0x1ac98e4f8a5121e9 + .quad 0x7dae9544dbfa2fe0 + .quad 0x8320aa0dd6430df9 + .quad 0x667282652c4a2fb5 + + // 2^188 * 5 * G + + .quad 0x874621f4d86bc9ab + .quad 0xb54c7bbe56fe6fea + .quad 0x077a24257fadc22c + .quad 0x1ab53be419b90d39 + .quad 0xada8b6e02946db23 + .quad 0x1c0ce51a7b253ab7 + .quad 0x8448c85a66dd485b + .quad 0x7f1fc025d0675adf + .quad 0xd8ee1b18319ea6aa + .quad 0x004d88083a21f0da + .quad 0x3bd6aa1d883a4f4b + .quad 0x4db9a3a6dfd9fd14 + + // 2^188 * 6 * G + + .quad 0x8ce7b23bb99c0755 + .quad 0x35c5d6edc4f50f7a + .quad 0x7e1e2ed2ed9b50c3 + .quad 0x36305f16e8934da1 + .quad 0xd95b00bbcbb77c68 + .quad 0xddbc846a91f17849 + .quad 0x7cf700aebe28d9b3 + .quad 0x5ce1285c85d31f3e + .quad 0x31b6972d98b0bde8 + .quad 0x7d920706aca6de5b + .quad 0xe67310f8908a659f + .quad 0x50fac2a6efdf0235 + + // 2^188 * 7 * G + + .quad 0xf3d3a9f35b880f5a + .quad 0xedec050cdb03e7c2 + .quad 0xa896981ff9f0b1a2 + .quad 0x49a4ae2bac5e34a4 + .quad 0x295b1c86f6f449bc + .quad 0x51b2e84a1f0ab4dd + .quad 0xc001cb30aa8e551d + .quad 0x6a28d35944f43662 + .quad 0x28bb12ee04a740e0 + .quad 0x14313bbd9bce8174 + .quad 0x72f5b5e4e8c10c40 + .quad 0x7cbfb19936adcd5b + + // 2^188 * 8 * G + + .quad 0xa311ddc26b89792d + .quad 0x1b30b4c6da512664 + .quad 0x0ca77b4ccf150859 + .quad 0x1de443df1b009408 + .quad 0x8e793a7acc36e6e0 + .quad 0xf9fab7a37d586eed + .quad 0x3a4f9692bae1f4e4 + .quad 0x1c14b03eff5f447e + .quad 0x19647bd114a85291 + .quad 0x57b76cb21034d3af + .quad 0x6329db440f9d6dfa + .quad 0x5ef43e586a571493 + + // 2^192 * 1 * G + + .quad 0xef782014385675a6 + .quad 0xa2649f30aafda9e8 + .quad 0x4cd1eb505cdfa8cb + .quad 0x46115aba1d4dc0b3 + .quad 0xa66dcc9dc80c1ac0 + .quad 0x97a05cf41b38a436 + .quad 0xa7ebf3be95dbd7c6 + .quad 0x7da0b8f68d7e7dab + .quad 0xd40f1953c3b5da76 + .quad 0x1dac6f7321119e9b + .quad 0x03cc6021feb25960 + .quad 0x5a5f887e83674b4b + + // 2^192 * 2 * G + + .quad 0x8f6301cf70a13d11 + .quad 0xcfceb815350dd0c4 + .quad 0xf70297d4a4bca47e + .quad 0x3669b656e44d1434 + .quad 0x9e9628d3a0a643b9 + .quad 0xb5c3cb00e6c32064 + .quad 0x9b5302897c2dec32 + .quad 0x43e37ae2d5d1c70c + .quad 0x387e3f06eda6e133 + .quad 0x67301d5199a13ac0 + .quad 0xbd5ad8f836263811 + .quad 0x6a21e6cd4fd5e9be + + // 2^192 * 3 * G + + .quad 0xf1c6170a3046e65f + .quad 0x58712a2a00d23524 + .quad 0x69dbbd3c8c82b755 + .quad 0x586bf9f1a195ff57 + .quad 0xef4129126699b2e3 + .quad 0x71d30847708d1301 + .quad 0x325432d01182b0bd + .quad 0x45371b07001e8b36 + .quad 0xa6db088d5ef8790b + .quad 0x5278f0dc610937e5 + .quad 0xac0349d261a16eb8 + .quad 0x0eafb03790e52179 + + // 2^192 * 4 * G + + .quad 0x960555c13748042f + .quad 0x219a41e6820baa11 + .quad 0x1c81f73873486d0c + .quad 0x309acc675a02c661 + .quad 0x5140805e0f75ae1d + .quad 0xec02fbe32662cc30 + .quad 0x2cebdf1eea92396d + .quad 0x44ae3344c5435bb3 + .quad 0x9cf289b9bba543ee + .quad 0xf3760e9d5ac97142 + .quad 0x1d82e5c64f9360aa + .quad 0x62d5221b7f94678f + + // 2^192 * 5 * G + + .quad 0x524c299c18d0936d + .quad 0xc86bb56c8a0c1a0c + .quad 0xa375052edb4a8631 + .quad 0x5c0efde4bc754562 + .quad 0x7585d4263af77a3c + .quad 0xdfae7b11fee9144d + .quad 0xa506708059f7193d + .quad 0x14f29a5383922037 + .quad 0xdf717edc25b2d7f5 + .quad 0x21f970db99b53040 + .quad 0xda9234b7c3ed4c62 + .quad 0x5e72365c7bee093e + + // 2^192 * 6 * G + + .quad 0x575bfc074571217f + .quad 0x3779675d0694d95b + .quad 0x9a0a37bbf4191e33 + .quad 0x77f1104c47b4eabc + .quad 0x7d9339062f08b33e + .quad 0x5b9659e5df9f32be + .quad 0xacff3dad1f9ebdfd + .quad 0x70b20555cb7349b7 + .quad 0xbe5113c555112c4c + .quad 0x6688423a9a881fcd + .quad 0x446677855e503b47 + .quad 0x0e34398f4a06404a + + // 2^192 * 7 * G + + .quad 0xb67d22d93ecebde8 + .quad 0x09b3e84127822f07 + .quad 0x743fa61fb05b6d8d + .quad 0x5e5405368a362372 + .quad 0x18930b093e4b1928 + .quad 0x7de3e10e73f3f640 + .quad 0xf43217da73395d6f + .quad 0x6f8aded6ca379c3e + .quad 0xe340123dfdb7b29a + .quad 0x487b97e1a21ab291 + .quad 0xf9967d02fde6949e + .quad 0x780de72ec8d3de97 + + // 2^192 * 8 * G + + .quad 0x0ae28545089ae7bc + .quad 0x388ddecf1c7f4d06 + .quad 0x38ac15510a4811b8 + .quad 0x0eb28bf671928ce4 + .quad 0x671feaf300f42772 + .quad 0x8f72eb2a2a8c41aa + .quad 0x29a17fd797373292 + .quad 0x1defc6ad32b587a6 + .quad 0xaf5bbe1aef5195a7 + .quad 0x148c1277917b15ed + .quad 0x2991f7fb7ae5da2e + .quad 0x467d201bf8dd2867 + + // 2^196 * 1 * G + + .quad 0x7906ee72f7bd2e6b + .quad 0x05d270d6109abf4e + .quad 0x8d5cfe45b941a8a4 + .quad 0x44c218671c974287 + .quad 0x745f9d56296bc318 + .quad 0x993580d4d8152e65 + .quad 0xb0e5b13f5839e9ce + .quad 0x51fc2b28d43921c0 + .quad 0x1b8fd11795e2a98c + .quad 0x1c4e5ee12b6b6291 + .quad 0x5b30e7107424b572 + .quad 0x6e6b9de84c4f4ac6 + + // 2^196 * 2 * G + + .quad 0xdff25fce4b1de151 + .quad 0xd841c0c7e11c4025 + .quad 0x2554b3c854749c87 + .quad 0x2d292459908e0df9 + .quad 0x6b7c5f10f80cb088 + .quad 0x736b54dc56e42151 + .quad 0xc2b620a5c6ef99c4 + .quad 0x5f4c802cc3a06f42 + .quad 0x9b65c8f17d0752da + .quad 0x881ce338c77ee800 + .quad 0xc3b514f05b62f9e3 + .quad 0x66ed5dd5bec10d48 + + // 2^196 * 3 * G + + .quad 0x7d38a1c20bb2089d + .quad 0x808334e196ccd412 + .quad 0xc4a70b8c6c97d313 + .quad 0x2eacf8bc03007f20 + .quad 0xf0adf3c9cbca047d + .quad 0x81c3b2cbf4552f6b + .quad 0xcfda112d44735f93 + .quad 0x1f23a0c77e20048c + .quad 0xf235467be5bc1570 + .quad 0x03d2d9020dbab38c + .quad 0x27529aa2fcf9e09e + .quad 0x0840bef29d34bc50 + + // 2^196 * 4 * G + + .quad 0x796dfb35dc10b287 + .quad 0x27176bcd5c7ff29d + .quad 0x7f3d43e8c7b24905 + .quad 0x0304f5a191c54276 + .quad 0xcd54e06b7f37e4eb + .quad 0x8cc15f87f5e96cca + .quad 0xb8248bb0d3597dce + .quad 0x246affa06074400c + .quad 0x37d88e68fbe45321 + .quad 0x86097548c0d75032 + .quad 0x4e9b13ef894a0d35 + .quad 0x25a83cac5753d325 + + // 2^196 * 5 * G + + .quad 0x10222f48eed8165e + .quad 0x623fc1234b8bcf3a + .quad 0x1e145c09c221e8f0 + .quad 0x7ccfa59fca782630 + .quad 0x9f0f66293952b6e2 + .quad 0x33db5e0e0934267b + .quad 0xff45252bd609fedc + .quad 0x06be10f5c506e0c9 + .quad 0x1a9615a9b62a345f + .quad 0x22050c564a52fecc + .quad 0xa7a2788528bc0dfe + .quad 0x5e82770a1a1ee71d + + // 2^196 * 6 * G + + .quad 0x35425183ad896a5c + .quad 0xe8673afbe78d52f6 + .quad 0x2c66f25f92a35f64 + .quad 0x09d04f3b3b86b102 + .quad 0xe802e80a42339c74 + .quad 0x34175166a7fffae5 + .quad 0x34865d1f1c408cae + .quad 0x2cca982c605bc5ee + .quad 0xfd2d5d35197dbe6e + .quad 0x207c2eea8be4ffa3 + .quad 0x2613d8db325ae918 + .quad 0x7a325d1727741d3e + + // 2^196 * 7 * G + + .quad 0xd036b9bbd16dfde2 + .quad 0xa2055757c497a829 + .quad 0x8e6cc966a7f12667 + .quad 0x4d3b1a791239c180 + .quad 0xecd27d017e2a076a + .quad 0xd788689f1636495e + .quad 0x52a61af0919233e5 + .quad 0x2a479df17bb1ae64 + .quad 0x9e5eee8e33db2710 + .quad 0x189854ded6c43ca5 + .quad 0xa41c22c592718138 + .quad 0x27ad5538a43a5e9b + + // 2^196 * 8 * G + + .quad 0x2746dd4b15350d61 + .quad 0xd03fcbc8ee9521b7 + .quad 0xe86e365a138672ca + .quad 0x510e987f7e7d89e2 + .quad 0xcb5a7d638e47077c + .quad 0x8db7536120a1c059 + .quad 0x549e1e4d8bedfdcc + .quad 0x080153b7503b179d + .quad 0xdda69d930a3ed3e3 + .quad 0x3d386ef1cd60a722 + .quad 0xc817ad58bdaa4ee6 + .quad 0x23be8d554fe7372a + + // 2^200 * 1 * G + + .quad 0x95fe919a74ef4fad + .quad 0x3a827becf6a308a2 + .quad 0x964e01d309a47b01 + .quad 0x71c43c4f5ba3c797 + .quad 0xbc1ef4bd567ae7a9 + .quad 0x3f624cb2d64498bd + .quad 0xe41064d22c1f4ec8 + .quad 0x2ef9c5a5ba384001 + .quad 0xb6fd6df6fa9e74cd + .quad 0xf18278bce4af267a + .quad 0x8255b3d0f1ef990e + .quad 0x5a758ca390c5f293 + + // 2^200 * 2 * G + + .quad 0xa2b72710d9462495 + .quad 0x3aa8c6d2d57d5003 + .quad 0xe3d400bfa0b487ca + .quad 0x2dbae244b3eb72ec + .quad 0x8ce0918b1d61dc94 + .quad 0x8ded36469a813066 + .quad 0xd4e6a829afe8aad3 + .quad 0x0a738027f639d43f + .quad 0x980f4a2f57ffe1cc + .quad 0x00670d0de1839843 + .quad 0x105c3f4a49fb15fd + .quad 0x2698ca635126a69c + + // 2^200 * 3 * G + + .quad 0xe765318832b0ba78 + .quad 0x381831f7925cff8b + .quad 0x08a81b91a0291fcc + .quad 0x1fb43dcc49caeb07 + .quad 0x2e3d702f5e3dd90e + .quad 0x9e3f0918e4d25386 + .quad 0x5e773ef6024da96a + .quad 0x3c004b0c4afa3332 + .quad 0x9aa946ac06f4b82b + .quad 0x1ca284a5a806c4f3 + .quad 0x3ed3265fc6cd4787 + .quad 0x6b43fd01cd1fd217 + + // 2^200 * 4 * G + + .quad 0xc7a75d4b4697c544 + .quad 0x15fdf848df0fffbf + .quad 0x2868b9ebaa46785a + .quad 0x5a68d7105b52f714 + .quad 0xb5c742583e760ef3 + .quad 0x75dc52b9ee0ab990 + .quad 0xbf1427c2072b923f + .quad 0x73420b2d6ff0d9f0 + .quad 0xaf2cf6cb9e851e06 + .quad 0x8f593913c62238c4 + .quad 0xda8ab89699fbf373 + .quad 0x3db5632fea34bc9e + + // 2^200 * 5 * G + + .quad 0xf46eee2bf75dd9d8 + .quad 0x0d17b1f6396759a5 + .quad 0x1bf2d131499e7273 + .quad 0x04321adf49d75f13 + .quad 0x2e4990b1829825d5 + .quad 0xedeaeb873e9a8991 + .quad 0xeef03d394c704af8 + .quad 0x59197ea495df2b0e + .quad 0x04e16019e4e55aae + .quad 0xe77b437a7e2f92e9 + .quad 0xc7ce2dc16f159aa4 + .quad 0x45eafdc1f4d70cc0 + + // 2^200 * 6 * G + + .quad 0x698401858045d72b + .quad 0x4c22faa2cf2f0651 + .quad 0x941a36656b222dc6 + .quad 0x5a5eebc80362dade + .quad 0xb60e4624cfccb1ed + .quad 0x59dbc292bd5c0395 + .quad 0x31a09d1ddc0481c9 + .quad 0x3f73ceea5d56d940 + .quad 0xb7a7bfd10a4e8dc6 + .quad 0xbe57007e44c9b339 + .quad 0x60c1207f1557aefa + .quad 0x26058891266218db + + // 2^200 * 7 * G + + .quad 0x59f704a68360ff04 + .quad 0xc3d93fde7661e6f4 + .quad 0x831b2a7312873551 + .quad 0x54ad0c2e4e615d57 + .quad 0x4c818e3cc676e542 + .quad 0x5e422c9303ceccad + .quad 0xec07cccab4129f08 + .quad 0x0dedfa10b24443b8 + .quad 0xee3b67d5b82b522a + .quad 0x36f163469fa5c1eb + .quad 0xa5b4d2f26ec19fd3 + .quad 0x62ecb2baa77a9408 + + // 2^200 * 8 * G + + .quad 0xe5ed795261152b3d + .quad 0x4962357d0eddd7d1 + .quad 0x7482c8d0b96b4c71 + .quad 0x2e59f919a966d8be + .quad 0x92072836afb62874 + .quad 0x5fcd5e8579e104a5 + .quad 0x5aad01adc630a14a + .quad 0x61913d5075663f98 + .quad 0x0dc62d361a3231da + .quad 0xfa47583294200270 + .quad 0x02d801513f9594ce + .quad 0x3ddbc2a131c05d5c + + // 2^204 * 1 * G + + .quad 0x3f50a50a4ffb81ef + .quad 0xb1e035093bf420bf + .quad 0x9baa8e1cc6aa2cd0 + .quad 0x32239861fa237a40 + .quad 0xfb735ac2004a35d1 + .quad 0x31de0f433a6607c3 + .quad 0x7b8591bfc528d599 + .quad 0x55be9a25f5bb050c + .quad 0x0d005acd33db3dbf + .quad 0x0111b37c80ac35e2 + .quad 0x4892d66c6f88ebeb + .quad 0x770eadb16508fbcd + + // 2^204 * 2 * G + + .quad 0x8451f9e05e4e89dd + .quad 0xc06302ffbc793937 + .quad 0x5d22749556a6495c + .quad 0x09a6755ca05603fb + .quad 0xf1d3b681a05071b9 + .quad 0x2207659a3592ff3a + .quad 0x5f0169297881e40e + .quad 0x16bedd0e86ba374e + .quad 0x5ecccc4f2c2737b5 + .quad 0x43b79e0c2dccb703 + .quad 0x33e008bc4ec43df3 + .quad 0x06c1b840f07566c0 + + // 2^204 * 3 * G + + .quad 0x7688a5c6a388f877 + .quad 0x02a96c14deb2b6ac + .quad 0x64c9f3431b8c2af8 + .quad 0x3628435554a1eed6 + .quad 0x69ee9e7f9b02805c + .quad 0xcbff828a547d1640 + .quad 0x3d93a869b2430968 + .quad 0x46b7b8cd3fe26972 + .quad 0xe9812086fe7eebe0 + .quad 0x4cba6be72f515437 + .quad 0x1d04168b516efae9 + .quad 0x5ea1391043982cb9 + + // 2^204 * 4 * G + + .quad 0x49125c9cf4702ee1 + .quad 0x4520b71f8b25b32d + .quad 0x33193026501fef7e + .quad 0x656d8997c8d2eb2b + .quad 0x6f2b3be4d5d3b002 + .quad 0xafec33d96a09c880 + .quad 0x035f73a4a8bcc4cc + .quad 0x22c5b9284662198b + .quad 0xcb58c8fe433d8939 + .quad 0x89a0cb2e6a8d7e50 + .quad 0x79ca955309fbbe5a + .quad 0x0c626616cd7fc106 + + // 2^204 * 5 * G + + .quad 0x1ffeb80a4879b61f + .quad 0x6396726e4ada21ed + .quad 0x33c7b093368025ba + .quad 0x471aa0c6f3c31788 + .quad 0x8fdfc379fbf454b1 + .quad 0x45a5a970f1a4b771 + .quad 0xac921ef7bad35915 + .quad 0x42d088dca81c2192 + .quad 0x8fda0f37a0165199 + .quad 0x0adadb77c8a0e343 + .quad 0x20fbfdfcc875e820 + .quad 0x1cf2bea80c2206e7 + + // 2^204 * 6 * G + + .quad 0xc2ddf1deb36202ac + .quad 0x92a5fe09d2e27aa5 + .quad 0x7d1648f6fc09f1d3 + .quad 0x74c2cc0513bc4959 + .quad 0x982d6e1a02c0412f + .quad 0x90fa4c83db58e8fe + .quad 0x01c2f5bcdcb18bc0 + .quad 0x686e0c90216abc66 + .quad 0x1fadbadba54395a7 + .quad 0xb41a02a0ae0da66a + .quad 0xbf19f598bba37c07 + .quad 0x6a12b8acde48430d + + // 2^204 * 7 * G + + .quad 0xf8daea1f39d495d9 + .quad 0x592c190e525f1dfc + .quad 0xdb8cbd04c9991d1b + .quad 0x11f7fda3d88f0cb7 + .quad 0x793bdd801aaeeb5f + .quad 0x00a2a0aac1518871 + .quad 0xe8a373a31f2136b4 + .quad 0x48aab888fc91ef19 + .quad 0x041f7e925830f40e + .quad 0x002d6ca979661c06 + .quad 0x86dc9ff92b046a2e + .quad 0x760360928b0493d1 + + // 2^204 * 8 * G + + .quad 0x21bb41c6120cf9c6 + .quad 0xeab2aa12decda59b + .quad 0xc1a72d020aa48b34 + .quad 0x215d4d27e87d3b68 + .quad 0xb43108e5695a0b05 + .quad 0x6cb00ee8ad37a38b + .quad 0x5edad6eea3537381 + .quad 0x3f2602d4b6dc3224 + .quad 0xc8b247b65bcaf19c + .quad 0x49779dc3b1b2c652 + .quad 0x89a180bbd5ece2e2 + .quad 0x13f098a3cec8e039 + + // 2^208 * 1 * G + + .quad 0x9adc0ff9ce5ec54b + .quad 0x039c2a6b8c2f130d + .quad 0x028007c7f0f89515 + .quad 0x78968314ac04b36b + .quad 0xf3aa57a22796bb14 + .quad 0x883abab79b07da21 + .quad 0xe54be21831a0391c + .quad 0x5ee7fb38d83205f9 + .quad 0x538dfdcb41446a8e + .quad 0xa5acfda9434937f9 + .quad 0x46af908d263c8c78 + .quad 0x61d0633c9bca0d09 + + // 2^208 * 2 * G + + .quad 0x63744935ffdb2566 + .quad 0xc5bd6b89780b68bb + .quad 0x6f1b3280553eec03 + .quad 0x6e965fd847aed7f5 + .quad 0xada328bcf8fc73df + .quad 0xee84695da6f037fc + .quad 0x637fb4db38c2a909 + .quad 0x5b23ac2df8067bdc + .quad 0x9ad2b953ee80527b + .quad 0xe88f19aafade6d8d + .quad 0x0e711704150e82cf + .quad 0x79b9bbb9dd95dedc + + // 2^208 * 3 * G + + .quad 0xebb355406a3126c2 + .quad 0xd26383a868c8c393 + .quad 0x6c0c6429e5b97a82 + .quad 0x5065f158c9fd2147 + .quad 0xd1997dae8e9f7374 + .quad 0xa032a2f8cfbb0816 + .quad 0xcd6cba126d445f0a + .quad 0x1ba811460accb834 + .quad 0x708169fb0c429954 + .quad 0xe14600acd76ecf67 + .quad 0x2eaab98a70e645ba + .quad 0x3981f39e58a4faf2 + + // 2^208 * 4 * G + + .quad 0x18fb8a7559230a93 + .quad 0x1d168f6960e6f45d + .quad 0x3a85a94514a93cb5 + .quad 0x38dc083705acd0fd + .quad 0xc845dfa56de66fde + .quad 0xe152a5002c40483a + .quad 0xe9d2e163c7b4f632 + .quad 0x30f4452edcbc1b65 + .quad 0x856d2782c5759740 + .quad 0xfa134569f99cbecc + .quad 0x8844fc73c0ea4e71 + .quad 0x632d9a1a593f2469 + + // 2^208 * 5 * G + + .quad 0xf6bb6b15b807cba6 + .quad 0x1823c7dfbc54f0d7 + .quad 0xbb1d97036e29670b + .quad 0x0b24f48847ed4a57 + .quad 0xbf09fd11ed0c84a7 + .quad 0x63f071810d9f693a + .quad 0x21908c2d57cf8779 + .quad 0x3a5a7df28af64ba2 + .quad 0xdcdad4be511beac7 + .quad 0xa4538075ed26ccf2 + .quad 0xe19cff9f005f9a65 + .quad 0x34fcf74475481f63 + + // 2^208 * 6 * G + + .quad 0xc197e04c789767ca + .quad 0xb8714dcb38d9467d + .quad 0x55de888283f95fa8 + .quad 0x3d3bdc164dfa63f7 + .quad 0xa5bb1dab78cfaa98 + .quad 0x5ceda267190b72f2 + .quad 0x9309c9110a92608e + .quad 0x0119a3042fb374b0 + .quad 0x67a2d89ce8c2177d + .quad 0x669da5f66895d0c1 + .quad 0xf56598e5b282a2b0 + .quad 0x56c088f1ede20a73 + + // 2^208 * 7 * G + + .quad 0x336d3d1110a86e17 + .quad 0xd7f388320b75b2fa + .quad 0xf915337625072988 + .quad 0x09674c6b99108b87 + .quad 0x581b5fac24f38f02 + .quad 0xa90be9febae30cbd + .quad 0x9a2169028acf92f0 + .quad 0x038b7ea48359038f + .quad 0x9f4ef82199316ff8 + .quad 0x2f49d282eaa78d4f + .quad 0x0971a5ab5aef3174 + .quad 0x6e5e31025969eb65 + + // 2^208 * 8 * G + + .quad 0xb16c62f587e593fb + .quad 0x4999eddeca5d3e71 + .quad 0xb491c1e014cc3e6d + .quad 0x08f5114789a8dba8 + .quad 0x3304fb0e63066222 + .quad 0xfb35068987acba3f + .quad 0xbd1924778c1061a3 + .quad 0x3058ad43d1838620 + .quad 0x323c0ffde57663d0 + .quad 0x05c3df38a22ea610 + .quad 0xbdc78abdac994f9a + .quad 0x26549fa4efe3dc99 + + // 2^212 * 1 * G + + .quad 0x738b38d787ce8f89 + .quad 0xb62658e24179a88d + .quad 0x30738c9cf151316d + .quad 0x49128c7f727275c9 + .quad 0x04dbbc17f75396b9 + .quad 0x69e6a2d7d2f86746 + .quad 0xc6409d99f53eabc6 + .quad 0x606175f6332e25d2 + .quad 0x4021370ef540e7dd + .quad 0x0910d6f5a1f1d0a5 + .quad 0x4634aacd5b06b807 + .quad 0x6a39e6356944f235 + + // 2^212 * 2 * G + + .quad 0x96cd5640df90f3e7 + .quad 0x6c3a760edbfa25ea + .quad 0x24f3ef0959e33cc4 + .quad 0x42889e7e530d2e58 + .quad 0x1da1965774049e9d + .quad 0xfbcd6ea198fe352b + .quad 0xb1cbcd50cc5236a6 + .quad 0x1f5ec83d3f9846e2 + .quad 0x8efb23c3328ccb75 + .quad 0xaf42a207dd876ee9 + .quad 0x20fbdadc5dfae796 + .quad 0x241e246b06bf9f51 + + // 2^212 * 3 * G + + .quad 0x29e68e57ad6e98f6 + .quad 0x4c9260c80b462065 + .quad 0x3f00862ea51ebb4b + .quad 0x5bc2c77fb38d9097 + .quad 0x7eaafc9a6280bbb8 + .quad 0x22a70f12f403d809 + .quad 0x31ce40bb1bfc8d20 + .quad 0x2bc65635e8bd53ee + .quad 0xe8d5dc9fa96bad93 + .quad 0xe58fb17dde1947dc + .quad 0x681532ea65185fa3 + .quad 0x1fdd6c3b034a7830 + + // 2^212 * 4 * G + + .quad 0x0a64e28c55dc18fe + .quad 0xe3df9e993399ebdd + .quad 0x79ac432370e2e652 + .quad 0x35ff7fc33ae4cc0e + .quad 0x9c13a6a52dd8f7a9 + .quad 0x2dbb1f8c3efdcabf + .quad 0x961e32405e08f7b5 + .quad 0x48c8a121bbe6c9e5 + .quad 0xfc415a7c59646445 + .quad 0xd224b2d7c128b615 + .quad 0x6035c9c905fbb912 + .quad 0x42d7a91274429fab + + // 2^212 * 5 * G + + .quad 0x4e6213e3eaf72ed3 + .quad 0x6794981a43acd4e7 + .quad 0xff547cde6eb508cb + .quad 0x6fed19dd10fcb532 + .quad 0xa9a48947933da5bc + .quad 0x4a58920ec2e979ec + .quad 0x96d8800013e5ac4c + .quad 0x453692d74b48b147 + .quad 0xdd775d99a8559c6f + .quad 0xf42a2140df003e24 + .quad 0x5223e229da928a66 + .quad 0x063f46ba6d38f22c + + // 2^212 * 6 * G + + .quad 0xd2d242895f536694 + .quad 0xca33a2c542939b2c + .quad 0x986fada6c7ddb95c + .quad 0x5a152c042f712d5d + .quad 0x39843cb737346921 + .quad 0xa747fb0738c89447 + .quad 0xcb8d8031a245307e + .quad 0x67810f8e6d82f068 + .quad 0x3eeb8fbcd2287db4 + .quad 0x72c7d3a301a03e93 + .quad 0x5473e88cbd98265a + .quad 0x7324aa515921b403 + + // 2^212 * 7 * G + + .quad 0x857942f46c3cbe8e + .quad 0xa1d364b14730c046 + .quad 0x1c8ed914d23c41bf + .quad 0x0838e161eef6d5d2 + .quad 0xad23f6dae82354cb + .quad 0x6962502ab6571a6d + .quad 0x9b651636e38e37d1 + .quad 0x5cac5005d1a3312f + .quad 0x8cc154cce9e39904 + .quad 0x5b3a040b84de6846 + .quad 0xc4d8a61cb1be5d6e + .quad 0x40fb897bd8861f02 + + // 2^212 * 8 * G + + .quad 0x84c5aa9062de37a1 + .quad 0x421da5000d1d96e1 + .quad 0x788286306a9242d9 + .quad 0x3c5e464a690d10da + .quad 0xe57ed8475ab10761 + .quad 0x71435e206fd13746 + .quad 0x342f824ecd025632 + .quad 0x4b16281ea8791e7b + .quad 0xd1c101d50b813381 + .quad 0xdee60f1176ee6828 + .quad 0x0cb68893383f6409 + .quad 0x6183c565f6ff484a + + // 2^216 * 1 * G + + .quad 0x741d5a461e6bf9d6 + .quad 0x2305b3fc7777a581 + .quad 0xd45574a26474d3d9 + .quad 0x1926e1dc6401e0ff + .quad 0xdb468549af3f666e + .quad 0xd77fcf04f14a0ea5 + .quad 0x3df23ff7a4ba0c47 + .quad 0x3a10dfe132ce3c85 + .quad 0xe07f4e8aea17cea0 + .quad 0x2fd515463a1fc1fd + .quad 0x175322fd31f2c0f1 + .quad 0x1fa1d01d861e5d15 + + // 2^216 * 2 * G + + .quad 0xcc8055947d599832 + .quad 0x1e4656da37f15520 + .quad 0x99f6f7744e059320 + .quad 0x773563bc6a75cf33 + .quad 0x38dcac00d1df94ab + .quad 0x2e712bddd1080de9 + .quad 0x7f13e93efdd5e262 + .quad 0x73fced18ee9a01e5 + .quad 0x06b1e90863139cb3 + .quad 0xa493da67c5a03ecd + .quad 0x8d77cec8ad638932 + .quad 0x1f426b701b864f44 + + // 2^216 * 3 * G + + .quad 0xefc9264c41911c01 + .quad 0xf1a3b7b817a22c25 + .quad 0x5875da6bf30f1447 + .quad 0x4e1af5271d31b090 + .quad 0xf17e35c891a12552 + .quad 0xb76b8153575e9c76 + .quad 0xfa83406f0d9b723e + .quad 0x0b76bb1b3fa7e438 + .quad 0x08b8c1f97f92939b + .quad 0xbe6771cbd444ab6e + .quad 0x22e5646399bb8017 + .quad 0x7b6dd61eb772a955 + + // 2^216 * 4 * G + + .quad 0xb7adc1e850f33d92 + .quad 0x7998fa4f608cd5cf + .quad 0xad962dbd8dfc5bdb + .quad 0x703e9bceaf1d2f4f + .quad 0x5730abf9ab01d2c7 + .quad 0x16fb76dc40143b18 + .quad 0x866cbe65a0cbb281 + .quad 0x53fa9b659bff6afe + .quad 0x6c14c8e994885455 + .quad 0x843a5d6665aed4e5 + .quad 0x181bb73ebcd65af1 + .quad 0x398d93e5c4c61f50 + + // 2^216 * 5 * G + + .quad 0x1c4bd16733e248f3 + .quad 0xbd9e128715bf0a5f + .quad 0xd43f8cf0a10b0376 + .quad 0x53b09b5ddf191b13 + .quad 0xc3877c60d2e7e3f2 + .quad 0x3b34aaa030828bb1 + .quad 0x283e26e7739ef138 + .quad 0x699c9c9002c30577 + .quad 0xf306a7235946f1cc + .quad 0x921718b5cce5d97d + .quad 0x28cdd24781b4e975 + .quad 0x51caf30c6fcdd907 + + // 2^216 * 6 * G + + .quad 0xa60ba7427674e00a + .quad 0x630e8570a17a7bf3 + .quad 0x3758563dcf3324cc + .quad 0x5504aa292383fdaa + .quad 0x737af99a18ac54c7 + .quad 0x903378dcc51cb30f + .quad 0x2b89bc334ce10cc7 + .quad 0x12ae29c189f8e99a + .quad 0xa99ec0cb1f0d01cf + .quad 0x0dd1efcc3a34f7ae + .quad 0x55ca7521d09c4e22 + .quad 0x5fd14fe958eba5ea + + // 2^216 * 7 * G + + .quad 0xb5dc2ddf2845ab2c + .quad 0x069491b10a7fe993 + .quad 0x4daaf3d64002e346 + .quad 0x093ff26e586474d1 + .quad 0x3c42fe5ebf93cb8e + .quad 0xbedfa85136d4565f + .quad 0xe0f0859e884220e8 + .quad 0x7dd73f960725d128 + .quad 0xb10d24fe68059829 + .quad 0x75730672dbaf23e5 + .quad 0x1367253ab457ac29 + .quad 0x2f59bcbc86b470a4 + + // 2^216 * 8 * G + + .quad 0x83847d429917135f + .quad 0xad1b911f567d03d7 + .quad 0x7e7748d9be77aad1 + .quad 0x5458b42e2e51af4a + .quad 0x7041d560b691c301 + .quad 0x85201b3fadd7e71e + .quad 0x16c2e16311335585 + .quad 0x2aa55e3d010828b1 + .quad 0xed5192e60c07444f + .quad 0x42c54e2d74421d10 + .quad 0x352b4c82fdb5c864 + .quad 0x13e9004a8a768664 + + // 2^220 * 1 * G + + .quad 0xcbb5b5556c032bff + .quad 0xdf7191b729297a3a + .quad 0xc1ff7326aded81bb + .quad 0x71ade8bb68be03f5 + .quad 0x1e6284c5806b467c + .quad 0xc5f6997be75d607b + .quad 0x8b67d958b378d262 + .quad 0x3d88d66a81cd8b70 + .quad 0x8b767a93204ed789 + .quad 0x762fcacb9fa0ae2a + .quad 0x771febcc6dce4887 + .quad 0x343062158ff05fb3 + + // 2^220 * 2 * G + + .quad 0xe05da1a7e1f5bf49 + .quad 0x26457d6dd4736092 + .quad 0x77dcb07773cc32f6 + .quad 0x0a5d94969cdd5fcd + .quad 0xfce219072a7b31b4 + .quad 0x4d7adc75aa578016 + .quad 0x0ec276a687479324 + .quad 0x6d6d9d5d1fda4beb + .quad 0x22b1a58ae9b08183 + .quad 0xfd95d071c15c388b + .quad 0xa9812376850a0517 + .quad 0x33384cbabb7f335e + + // 2^220 * 3 * G + + .quad 0x3c6fa2680ca2c7b5 + .quad 0x1b5082046fb64fda + .quad 0xeb53349c5431d6de + .quad 0x5278b38f6b879c89 + .quad 0x33bc627a26218b8d + .quad 0xea80b21fc7a80c61 + .quad 0x9458b12b173e9ee6 + .quad 0x076247be0e2f3059 + .quad 0x52e105f61416375a + .quad 0xec97af3685abeba4 + .quad 0x26e6b50623a67c36 + .quad 0x5cf0e856f3d4fb01 + + // 2^220 * 4 * G + + .quad 0xf6c968731ae8cab4 + .quad 0x5e20741ecb4f92c5 + .quad 0x2da53be58ccdbc3e + .quad 0x2dddfea269970df7 + .quad 0xbeaece313db342a8 + .quad 0xcba3635b842db7ee + .quad 0xe88c6620817f13ef + .quad 0x1b9438aa4e76d5c6 + .quad 0x8a50777e166f031a + .quad 0x067b39f10fb7a328 + .quad 0x1925c9a6010fbd76 + .quad 0x6df9b575cc740905 + + // 2^220 * 5 * G + + .quad 0x42c1192927f6bdcf + .quad 0x8f91917a403d61ca + .quad 0xdc1c5a668b9e1f61 + .quad 0x1596047804ec0f8d + .quad 0xecdfc35b48cade41 + .quad 0x6a88471fb2328270 + .quad 0x740a4a2440a01b6a + .quad 0x471e5796003b5f29 + .quad 0xda96bbb3aced37ac + .quad 0x7a2423b5e9208cea + .quad 0x24cc5c3038aebae2 + .quad 0x50c356afdc5dae2f + + // 2^220 * 6 * G + + .quad 0x09dcbf4341c30318 + .quad 0xeeba061183181dce + .quad 0xc179c0cedc1e29a1 + .quad 0x1dbf7b89073f35b0 + .quad 0xcfed9cdf1b31b964 + .quad 0xf486a9858ca51af3 + .quad 0x14897265ea8c1f84 + .quad 0x784a53dd932acc00 + .quad 0x2d99f9df14fc4920 + .quad 0x76ccb60cc4499fe5 + .quad 0xa4132cbbe5cf0003 + .quad 0x3f93d82354f000ea + + // 2^220 * 7 * G + + .quad 0x8183e7689e04ce85 + .quad 0x678fb71e04465341 + .quad 0xad92058f6688edac + .quad 0x5da350d3532b099a + .quad 0xeaac12d179e14978 + .quad 0xff923ff3bbebff5e + .quad 0x4af663e40663ce27 + .quad 0x0fd381a811a5f5ff + .quad 0xf256aceca436df54 + .quad 0x108b6168ae69d6e8 + .quad 0x20d986cb6b5d036c + .quad 0x655957b9fee2af50 + + // 2^220 * 8 * G + + .quad 0xaea8b07fa902030f + .quad 0xf88c766af463d143 + .quad 0x15b083663c787a60 + .quad 0x08eab1148267a4a8 + .quad 0xbdc1409bd002d0ac + .quad 0x66660245b5ccd9a6 + .quad 0x82317dc4fade85ec + .quad 0x02fe934b6ad7df0d + .quad 0xef5cf100cfb7ea74 + .quad 0x22897633a1cb42ac + .quad 0xd4ce0c54cef285e2 + .quad 0x30408c048a146a55 + + // 2^224 * 1 * G + + .quad 0x739d8845832fcedb + .quad 0xfa38d6c9ae6bf863 + .quad 0x32bc0dcab74ffef7 + .quad 0x73937e8814bce45e + .quad 0xbb2e00c9193b877f + .quad 0xece3a890e0dc506b + .quad 0xecf3b7c036de649f + .quad 0x5f46040898de9e1a + .quad 0xb9037116297bf48d + .quad 0xa9d13b22d4f06834 + .quad 0xe19715574696bdc6 + .quad 0x2cf8a4e891d5e835 + + // 2^224 * 2 * G + + .quad 0x6d93fd8707110f67 + .quad 0xdd4c09d37c38b549 + .quad 0x7cb16a4cc2736a86 + .quad 0x2049bd6e58252a09 + .quad 0x2cb5487e17d06ba2 + .quad 0x24d2381c3950196b + .quad 0xd7659c8185978a30 + .quad 0x7a6f7f2891d6a4f6 + .quad 0x7d09fd8d6a9aef49 + .quad 0xf0ee60be5b3db90b + .quad 0x4c21b52c519ebfd4 + .quad 0x6011aadfc545941d + + // 2^224 * 3 * G + + .quad 0x5f67926dcf95f83c + .quad 0x7c7e856171289071 + .quad 0xd6a1e7f3998f7a5b + .quad 0x6fc5cc1b0b62f9e0 + .quad 0x63ded0c802cbf890 + .quad 0xfbd098ca0dff6aaa + .quad 0x624d0afdb9b6ed99 + .quad 0x69ce18b779340b1e + .quad 0xd1ef5528b29879cb + .quad 0xdd1aae3cd47e9092 + .quad 0x127e0442189f2352 + .quad 0x15596b3ae57101f1 + + // 2^224 * 4 * G + + .quad 0x462739d23f9179a2 + .quad 0xff83123197d6ddcf + .quad 0x1307deb553f2148a + .quad 0x0d2237687b5f4dda + .quad 0x09ff31167e5124ca + .quad 0x0be4158bd9c745df + .quad 0x292b7d227ef556e5 + .quad 0x3aa4e241afb6d138 + .quad 0x2cc138bf2a3305f5 + .quad 0x48583f8fa2e926c3 + .quad 0x083ab1a25549d2eb + .quad 0x32fcaa6e4687a36c + + // 2^224 * 5 * G + + .quad 0x7bc56e8dc57d9af5 + .quad 0x3e0bd2ed9df0bdf2 + .quad 0xaac014de22efe4a3 + .quad 0x4627e9cefebd6a5c + .quad 0x3207a4732787ccdf + .quad 0x17e31908f213e3f8 + .quad 0xd5b2ecd7f60d964e + .quad 0x746f6336c2600be9 + .quad 0x3f4af345ab6c971c + .quad 0xe288eb729943731f + .quad 0x33596a8a0344186d + .quad 0x7b4917007ed66293 + + // 2^224 * 6 * G + + .quad 0x2d85fb5cab84b064 + .quad 0x497810d289f3bc14 + .quad 0x476adc447b15ce0c + .quad 0x122ba376f844fd7b + .quad 0x54341b28dd53a2dd + .quad 0xaa17905bdf42fc3f + .quad 0x0ff592d94dd2f8f4 + .quad 0x1d03620fe08cd37d + .quad 0xc20232cda2b4e554 + .quad 0x9ed0fd42115d187f + .quad 0x2eabb4be7dd479d9 + .quad 0x02c70bf52b68ec4c + + // 2^224 * 7 * G + + .quad 0xa287ec4b5d0b2fbb + .quad 0x415c5790074882ca + .quad 0xe044a61ec1d0815c + .quad 0x26334f0a409ef5e0 + .quad 0xace532bf458d72e1 + .quad 0x5be768e07cb73cb5 + .quad 0x56cf7d94ee8bbde7 + .quad 0x6b0697e3feb43a03 + .quad 0xb6c8f04adf62a3c0 + .quad 0x3ef000ef076da45d + .quad 0x9c9cb95849f0d2a9 + .quad 0x1cc37f43441b2fae + + // 2^224 * 8 * G + + .quad 0x508f565a5cc7324f + .quad 0xd061c4c0e506a922 + .quad 0xfb18abdb5c45ac19 + .quad 0x6c6809c10380314a + .quad 0xd76656f1c9ceaeb9 + .quad 0x1c5b15f818e5656a + .quad 0x26e72832844c2334 + .quad 0x3a346f772f196838 + .quad 0xd2d55112e2da6ac8 + .quad 0xe9bd0331b1e851ed + .quad 0x960746dd8ec67262 + .quad 0x05911b9f6ef7c5d0 + + // 2^228 * 1 * G + + .quad 0xe9dcd756b637ff2d + .quad 0xec4c348fc987f0c4 + .quad 0xced59285f3fbc7b7 + .quad 0x3305354793e1ea87 + .quad 0x01c18980c5fe9f94 + .quad 0xcd656769716fd5c8 + .quad 0x816045c3d195a086 + .quad 0x6e2b7f3266cc7982 + .quad 0xcc802468f7c3568f + .quad 0x9de9ba8219974cb3 + .quad 0xabb7229cb5b81360 + .quad 0x44e2017a6fbeba62 + + // 2^228 * 2 * G + + .quad 0xc4c2a74354dab774 + .quad 0x8e5d4c3c4eaf031a + .quad 0xb76c23d242838f17 + .quad 0x749a098f68dce4ea + .quad 0x87f82cf3b6ca6ecd + .quad 0x580f893e18f4a0c2 + .quad 0x058930072604e557 + .quad 0x6cab6ac256d19c1d + .quad 0xdcdfe0a02cc1de60 + .quad 0x032665ff51c5575b + .quad 0x2c0c32f1073abeeb + .quad 0x6a882014cd7b8606 + + // 2^228 * 3 * G + + .quad 0xa52a92fea4747fb5 + .quad 0xdc12a4491fa5ab89 + .quad 0xd82da94bb847a4ce + .quad 0x4d77edce9512cc4e + .quad 0xd111d17caf4feb6e + .quad 0x050bba42b33aa4a3 + .quad 0x17514c3ceeb46c30 + .quad 0x54bedb8b1bc27d75 + .quad 0x77c8e14577e2189c + .quad 0xa3e46f6aff99c445 + .quad 0x3144dfc86d335343 + .quad 0x3a96559e7c4216a9 + + // 2^228 * 4 * G + + .quad 0x12550d37f42ad2ee + .quad 0x8b78e00498a1fbf5 + .quad 0x5d53078233894cb2 + .quad 0x02c84e4e3e498d0c + .quad 0x4493896880baaa52 + .quad 0x4c98afc4f285940e + .quad 0xef4aa79ba45448b6 + .quad 0x5278c510a57aae7f + .quad 0xa54dd074294c0b94 + .quad 0xf55d46b8df18ffb6 + .quad 0xf06fecc58dae8366 + .quad 0x588657668190d165 + + // 2^228 * 5 * G + + .quad 0xd47712311aef7117 + .quad 0x50343101229e92c7 + .quad 0x7a95e1849d159b97 + .quad 0x2449959b8b5d29c9 + .quad 0xbf5834f03de25cc3 + .quad 0xb887c8aed6815496 + .quad 0x5105221a9481e892 + .quad 0x6760ed19f7723f93 + .quad 0x669ba3b7ac35e160 + .quad 0x2eccf73fba842056 + .quad 0x1aec1f17c0804f07 + .quad 0x0d96bc031856f4e7 + + // 2^228 * 6 * G + + .quad 0x3318be7775c52d82 + .quad 0x4cb764b554d0aab9 + .quad 0xabcf3d27cc773d91 + .quad 0x3bf4d1848123288a + .quad 0xb1d534b0cc7505e1 + .quad 0x32cd003416c35288 + .quad 0xcb36a5800762c29d + .quad 0x5bfe69b9237a0bf8 + .quad 0x183eab7e78a151ab + .quad 0xbbe990c999093763 + .quad 0xff717d6e4ac7e335 + .quad 0x4c5cddb325f39f88 + + // 2^228 * 7 * G + + .quad 0xc0f6b74d6190a6eb + .quad 0x20ea81a42db8f4e4 + .quad 0xa8bd6f7d97315760 + .quad 0x33b1d60262ac7c21 + .quad 0x57750967e7a9f902 + .quad 0x2c37fdfc4f5b467e + .quad 0xb261663a3177ba46 + .quad 0x3a375e78dc2d532b + .quad 0x8141e72f2d4dddea + .quad 0xe6eafe9862c607c8 + .quad 0x23c28458573cafd0 + .quad 0x46b9476f4ff97346 + + // 2^228 * 8 * G + + .quad 0x0c1ffea44f901e5c + .quad 0x2b0b6fb72184b782 + .quad 0xe587ff910114db88 + .quad 0x37130f364785a142 + .quad 0x1215505c0d58359f + .quad 0x2a2013c7fc28c46b + .quad 0x24a0a1af89ea664e + .quad 0x4400b638a1130e1f + .quad 0x3a01b76496ed19c3 + .quad 0x31e00ab0ed327230 + .quad 0x520a885783ca15b1 + .quad 0x06aab9875accbec7 + + // 2^232 * 1 * G + + .quad 0xc1339983f5df0ebb + .quad 0xc0f3758f512c4cac + .quad 0x2cf1130a0bb398e1 + .quad 0x6b3cecf9aa270c62 + .quad 0x5349acf3512eeaef + .quad 0x20c141d31cc1cb49 + .quad 0x24180c07a99a688d + .quad 0x555ef9d1c64b2d17 + .quad 0x36a770ba3b73bd08 + .quad 0x624aef08a3afbf0c + .quad 0x5737ff98b40946f2 + .quad 0x675f4de13381749d + + // 2^232 * 2 * G + + .quad 0x0e2c52036b1782fc + .quad 0x64816c816cad83b4 + .quad 0xd0dcbdd96964073e + .quad 0x13d99df70164c520 + .quad 0xa12ff6d93bdab31d + .quad 0x0725d80f9d652dfe + .quad 0x019c4ff39abe9487 + .quad 0x60f450b882cd3c43 + .quad 0x014b5ec321e5c0ca + .quad 0x4fcb69c9d719bfa2 + .quad 0x4e5f1c18750023a0 + .quad 0x1c06de9e55edac80 + + // 2^232 * 3 * G + + .quad 0x990f7ad6a33ec4e2 + .quad 0x6608f938be2ee08e + .quad 0x9ca143c563284515 + .quad 0x4cf38a1fec2db60d + .quad 0xffd52b40ff6d69aa + .quad 0x34530b18dc4049bb + .quad 0x5e4a5c2fa34d9897 + .quad 0x78096f8e7d32ba2d + .quad 0xa0aaaa650dfa5ce7 + .quad 0xf9c49e2a48b5478c + .quad 0x4f09cc7d7003725b + .quad 0x373cad3a26091abe + + // 2^232 * 4 * G + + .quad 0xb294634d82c9f57c + .quad 0x1fcbfde124934536 + .quad 0x9e9c4db3418cdb5a + .quad 0x0040f3d9454419fc + .quad 0xf1bea8fb89ddbbad + .quad 0x3bcb2cbc61aeaecb + .quad 0x8f58a7bb1f9b8d9d + .quad 0x21547eda5112a686 + .quad 0xdefde939fd5986d3 + .quad 0xf4272c89510a380c + .quad 0xb72ba407bb3119b9 + .quad 0x63550a334a254df4 + + // 2^232 * 5 * G + + .quad 0x6507d6edb569cf37 + .quad 0x178429b00ca52ee1 + .quad 0xea7c0090eb6bd65d + .quad 0x3eea62c7daf78f51 + .quad 0x9bba584572547b49 + .quad 0xf305c6fae2c408e0 + .quad 0x60e8fa69c734f18d + .quad 0x39a92bafaa7d767a + .quad 0x9d24c713e693274e + .quad 0x5f63857768dbd375 + .quad 0x70525560eb8ab39a + .quad 0x68436a0665c9c4cd + + // 2^232 * 6 * G + + .quad 0xbc0235e8202f3f27 + .quad 0xc75c00e264f975b0 + .quad 0x91a4e9d5a38c2416 + .quad 0x17b6e7f68ab789f9 + .quad 0x1e56d317e820107c + .quad 0xc5266844840ae965 + .quad 0xc1e0a1c6320ffc7a + .quad 0x5373669c91611472 + .quad 0x5d2814ab9a0e5257 + .quad 0x908f2084c9cab3fc + .quad 0xafcaf5885b2d1eca + .quad 0x1cb4b5a678f87d11 + + // 2^232 * 7 * G + + .quad 0xb664c06b394afc6c + .quad 0x0c88de2498da5fb1 + .quad 0x4f8d03164bcad834 + .quad 0x330bca78de7434a2 + .quad 0x6b74aa62a2a007e7 + .quad 0xf311e0b0f071c7b1 + .quad 0x5707e438000be223 + .quad 0x2dc0fd2d82ef6eac + .quad 0x982eff841119744e + .quad 0xf9695e962b074724 + .quad 0xc58ac14fbfc953fb + .quad 0x3c31be1b369f1cf5 + + // 2^232 * 8 * G + + .quad 0xb0f4864d08948aee + .quad 0x07dc19ee91ba1c6f + .quad 0x7975cdaea6aca158 + .quad 0x330b61134262d4bb + .quad 0xc168bc93f9cb4272 + .quad 0xaeb8711fc7cedb98 + .quad 0x7f0e52aa34ac8d7a + .quad 0x41cec1097e7d55bb + .quad 0xf79619d7a26d808a + .quad 0xbb1fd49e1d9e156d + .quad 0x73d7c36cdba1df27 + .quad 0x26b44cd91f28777d + + // 2^236 * 1 * G + + .quad 0x300a9035393aa6d8 + .quad 0x2b501131a12bb1cd + .quad 0x7b1ff677f093c222 + .quad 0x4309c1f8cab82bad + .quad 0xaf44842db0285f37 + .quad 0x8753189047efc8df + .quad 0x9574e091f820979a + .quad 0x0e378d6069615579 + .quad 0xd9fa917183075a55 + .quad 0x4bdb5ad26b009fdc + .quad 0x7829ad2cd63def0e + .quad 0x078fc54975fd3877 + + // 2^236 * 2 * G + + .quad 0x87dfbd1428878f2d + .quad 0x134636dd1e9421a1 + .quad 0x4f17c951257341a3 + .quad 0x5df98d4bad296cb8 + .quad 0xe2004b5bb833a98a + .quad 0x44775dec2d4c3330 + .quad 0x3aa244067eace913 + .quad 0x272630e3d58e00a9 + .quad 0xf3678fd0ecc90b54 + .quad 0xf001459b12043599 + .quad 0x26725fbc3758b89b + .quad 0x4325e4aa73a719ae + + // 2^236 * 3 * G + + .quad 0x657dc6ef433c3493 + .quad 0x65375e9f80dbf8c3 + .quad 0x47fd2d465b372dae + .quad 0x4966ab79796e7947 + .quad 0xed24629acf69f59d + .quad 0x2a4a1ccedd5abbf4 + .quad 0x3535ca1f56b2d67b + .quad 0x5d8c68d043b1b42d + .quad 0xee332d4de3b42b0a + .quad 0xd84e5a2b16a4601c + .quad 0x78243877078ba3e4 + .quad 0x77ed1eb4184ee437 + + // 2^236 * 4 * G + + .quad 0xbfd4e13f201839a0 + .quad 0xaeefffe23e3df161 + .quad 0xb65b04f06b5d1fe3 + .quad 0x52e085fb2b62fbc0 + .quad 0x185d43f89e92ed1a + .quad 0xb04a1eeafe4719c6 + .quad 0x499fbe88a6f03f4f + .quad 0x5d8b0d2f3c859bdd + .quad 0x124079eaa54cf2ba + .quad 0xd72465eb001b26e7 + .quad 0x6843bcfdc97af7fd + .quad 0x0524b42b55eacd02 + + // 2^236 * 5 * G + + .quad 0xfd0d5dbee45447b0 + .quad 0x6cec351a092005ee + .quad 0x99a47844567579cb + .quad 0x59d242a216e7fa45 + .quad 0xbc18dcad9b829eac + .quad 0x23ae7d28b5f579d0 + .quad 0xc346122a69384233 + .quad 0x1a6110b2e7d4ac89 + .quad 0x4f833f6ae66997ac + .quad 0x6849762a361839a4 + .quad 0x6985dec1970ab525 + .quad 0x53045e89dcb1f546 + + // 2^236 * 6 * G + + .quad 0xcb8bb346d75353db + .quad 0xfcfcb24bae511e22 + .quad 0xcba48d40d50ae6ef + .quad 0x26e3bae5f4f7cb5d + .quad 0x84da3cde8d45fe12 + .quad 0xbd42c218e444e2d2 + .quad 0xa85196781f7e3598 + .quad 0x7642c93f5616e2b2 + .quad 0x2323daa74595f8e4 + .quad 0xde688c8b857abeb4 + .quad 0x3fc48e961c59326e + .quad 0x0b2e73ca15c9b8ba + + // 2^236 * 7 * G + + .quad 0xd6bb4428c17f5026 + .quad 0x9eb27223fb5a9ca7 + .quad 0xe37ba5031919c644 + .quad 0x21ce380db59a6602 + .quad 0x0e3fbfaf79c03a55 + .quad 0x3077af054cbb5acf + .quad 0xd5c55245db3de39f + .quad 0x015e68c1476a4af7 + .quad 0xc1d5285220066a38 + .quad 0x95603e523570aef3 + .quad 0x832659a7226b8a4d + .quad 0x5dd689091f8eedc9 + + // 2^236 * 8 * G + + .quad 0xcbac84debfd3c856 + .quad 0x1624c348b35ff244 + .quad 0xb7f88dca5d9cad07 + .quad 0x3b0e574da2c2ebe8 + .quad 0x1d022591a5313084 + .quad 0xca2d4aaed6270872 + .quad 0x86a12b852f0bfd20 + .quad 0x56e6c439ad7da748 + .quad 0xc704ff4942bdbae6 + .quad 0x5e21ade2b2de1f79 + .quad 0xe95db3f35652fad8 + .quad 0x0822b5378f08ebc1 + + // 2^240 * 1 * G + + .quad 0x51f048478f387475 + .quad 0xb25dbcf49cbecb3c + .quad 0x9aab1244d99f2055 + .quad 0x2c709e6c1c10a5d6 + .quad 0xe1b7f29362730383 + .quad 0x4b5279ffebca8a2c + .quad 0xdafc778abfd41314 + .quad 0x7deb10149c72610f + .quad 0xcb62af6a8766ee7a + .quad 0x66cbec045553cd0e + .quad 0x588001380f0be4b5 + .quad 0x08e68e9ff62ce2ea + + // 2^240 * 2 * G + + .quad 0x34ad500a4bc130ad + .quad 0x8d38db493d0bd49c + .quad 0xa25c3d98500a89be + .quad 0x2f1f3f87eeba3b09 + .quad 0x2f2d09d50ab8f2f9 + .quad 0xacb9218dc55923df + .quad 0x4a8f342673766cb9 + .quad 0x4cb13bd738f719f5 + .quad 0xf7848c75e515b64a + .quad 0xa59501badb4a9038 + .quad 0xc20d313f3f751b50 + .quad 0x19a1e353c0ae2ee8 + + // 2^240 * 3 * G + + .quad 0x7d1c7560bafa05c3 + .quad 0xb3e1a0a0c6e55e61 + .quad 0xe3529718c0d66473 + .quad 0x41546b11c20c3486 + .quad 0xb42172cdd596bdbd + .quad 0x93e0454398eefc40 + .quad 0x9fb15347b44109b5 + .quad 0x736bd3990266ae34 + .quad 0x85532d509334b3b4 + .quad 0x46fd114b60816573 + .quad 0xcc5f5f30425c8375 + .quad 0x412295a2b87fab5c + + // 2^240 * 4 * G + + .quad 0x19c99b88f57ed6e9 + .quad 0x5393cb266df8c825 + .quad 0x5cee3213b30ad273 + .quad 0x14e153ebb52d2e34 + .quad 0x2e655261e293eac6 + .quad 0x845a92032133acdb + .quad 0x460975cb7900996b + .quad 0x0760bb8d195add80 + .quad 0x413e1a17cde6818a + .quad 0x57156da9ed69a084 + .quad 0x2cbf268f46caccb1 + .quad 0x6b34be9bc33ac5f2 + + // 2^240 * 5 * G + + .quad 0xf3df2f643a78c0b2 + .quad 0x4c3e971ef22e027c + .quad 0xec7d1c5e49c1b5a3 + .quad 0x2012c18f0922dd2d + .quad 0x11fc69656571f2d3 + .quad 0xc6c9e845530e737a + .quad 0xe33ae7a2d4fe5035 + .quad 0x01b9c7b62e6dd30b + .quad 0x880b55e55ac89d29 + .quad 0x1483241f45a0a763 + .quad 0x3d36efdfc2e76c1f + .quad 0x08af5b784e4bade8 + + // 2^240 * 6 * G + + .quad 0x283499dc881f2533 + .quad 0x9d0525da779323b6 + .quad 0x897addfb673441f4 + .quad 0x32b79d71163a168d + .quad 0xe27314d289cc2c4b + .quad 0x4be4bd11a287178d + .quad 0x18d528d6fa3364ce + .quad 0x6423c1d5afd9826e + .quad 0xcc85f8d9edfcb36a + .quad 0x22bcc28f3746e5f9 + .quad 0xe49de338f9e5d3cd + .quad 0x480a5efbc13e2dcc + + // 2^240 * 7 * G + + .quad 0x0b51e70b01622071 + .quad 0x06b505cf8b1dafc5 + .quad 0x2c6bb061ef5aabcd + .quad 0x47aa27600cb7bf31 + .quad 0xb6614ce442ce221f + .quad 0x6e199dcc4c053928 + .quad 0x663fb4a4dc1cbe03 + .quad 0x24b31d47691c8e06 + .quad 0x2a541eedc015f8c3 + .quad 0x11a4fe7e7c693f7c + .quad 0xf0af66134ea278d6 + .quad 0x545b585d14dda094 + + // 2^240 * 8 * G + + .quad 0x67bf275ea0d43a0f + .quad 0xade68e34089beebe + .quad 0x4289134cd479e72e + .quad 0x0f62f9c332ba5454 + .quad 0x6204e4d0e3b321e1 + .quad 0x3baa637a28ff1e95 + .quad 0x0b0ccffd5b99bd9e + .quad 0x4d22dc3e64c8d071 + .quad 0xfcb46589d63b5f39 + .quad 0x5cae6a3f57cbcf61 + .quad 0xfebac2d2953afa05 + .quad 0x1c0fa01a36371436 + + // 2^244 * 1 * G + + .quad 0xe7547449bc7cd692 + .quad 0x0f9abeaae6f73ddf + .quad 0x4af01ca700837e29 + .quad 0x63ab1b5d3f1bc183 + .quad 0xc11ee5e854c53fae + .quad 0x6a0b06c12b4f3ff4 + .quad 0x33540f80e0b67a72 + .quad 0x15f18fc3cd07e3ef + .quad 0x32750763b028f48c + .quad 0x06020740556a065f + .quad 0xd53bd812c3495b58 + .quad 0x08706c9b865f508d + + // 2^244 * 2 * G + + .quad 0xf37ca2ab3d343dff + .quad 0x1a8c6a2d80abc617 + .quad 0x8e49e035d4ccffca + .quad 0x48b46beebaa1d1b9 + .quad 0xcc991b4138b41246 + .quad 0x243b9c526f9ac26b + .quad 0xb9ef494db7cbabbd + .quad 0x5fba433dd082ed00 + .quad 0x9c49e355c9941ad0 + .quad 0xb9734ade74498f84 + .quad 0x41c3fed066663e5c + .quad 0x0ecfedf8e8e710b3 + + // 2^244 * 3 * G + + .quad 0x76430f9f9cd470d9 + .quad 0xb62acc9ba42f6008 + .quad 0x1898297c59adad5e + .quad 0x7789dd2db78c5080 + .quad 0x744f7463e9403762 + .quad 0xf79a8dee8dfcc9c9 + .quad 0x163a649655e4cde3 + .quad 0x3b61788db284f435 + .quad 0xb22228190d6ef6b2 + .quad 0xa94a66b246ce4bfa + .quad 0x46c1a77a4f0b6cc7 + .quad 0x4236ccffeb7338cf + + // 2^244 * 4 * G + + .quad 0x8497404d0d55e274 + .quad 0x6c6663d9c4ad2b53 + .quad 0xec2fb0d9ada95734 + .quad 0x2617e120cdb8f73c + .quad 0x3bd82dbfda777df6 + .quad 0x71b177cc0b98369e + .quad 0x1d0e8463850c3699 + .quad 0x5a71945b48e2d1f1 + .quad 0x6f203dd5405b4b42 + .quad 0x327ec60410b24509 + .quad 0x9c347230ac2a8846 + .quad 0x77de29fc11ffeb6a + + // 2^244 * 5 * G + + .quad 0xb0ac57c983b778a8 + .quad 0x53cdcca9d7fe912c + .quad 0x61c2b854ff1f59dc + .quad 0x3a1a2cf0f0de7dac + .quad 0x835e138fecced2ca + .quad 0x8c9eaf13ea963b9a + .quad 0xc95fbfc0b2160ea6 + .quad 0x575e66f3ad877892 + .quad 0x99803a27c88fcb3a + .quad 0x345a6789275ec0b0 + .quad 0x459789d0ff6c2be5 + .quad 0x62f882651e70a8b2 + + // 2^244 * 6 * G + + .quad 0x085ae2c759ff1be4 + .quad 0x149145c93b0e40b7 + .quad 0xc467e7fa7ff27379 + .quad 0x4eeecf0ad5c73a95 + .quad 0x6d822986698a19e0 + .quad 0xdc9821e174d78a71 + .quad 0x41a85f31f6cb1f47 + .quad 0x352721c2bcda9c51 + .quad 0x48329952213fc985 + .quad 0x1087cf0d368a1746 + .quad 0x8e5261b166c15aa5 + .quad 0x2d5b2d842ed24c21 + + // 2^244 * 7 * G + + .quad 0x02cfebd9ebd3ded1 + .quad 0xd45b217739021974 + .quad 0x7576f813fe30a1b7 + .quad 0x5691b6f9a34ef6c2 + .quad 0x5eb7d13d196ac533 + .quad 0x377234ecdb80be2b + .quad 0xe144cffc7cf5ae24 + .quad 0x5226bcf9c441acec + .quad 0x79ee6c7223e5b547 + .quad 0x6f5f50768330d679 + .quad 0xed73e1e96d8adce9 + .quad 0x27c3da1e1d8ccc03 + + // 2^244 * 8 * G + + .quad 0x7eb9efb23fe24c74 + .quad 0x3e50f49f1651be01 + .quad 0x3ea732dc21858dea + .quad 0x17377bd75bb810f9 + .quad 0x28302e71630ef9f6 + .quad 0xc2d4a2032b64cee0 + .quad 0x090820304b6292be + .quad 0x5fca747aa82adf18 + .quad 0x232a03c35c258ea5 + .quad 0x86f23a2c6bcb0cf1 + .quad 0x3dad8d0d2e442166 + .quad 0x04a8933cab76862b + + // 2^248 * 1 * G + + .quad 0xd2c604b622943dff + .quad 0xbc8cbece44cfb3a0 + .quad 0x5d254ff397808678 + .quad 0x0fa3614f3b1ca6bf + .quad 0x69082b0e8c936a50 + .quad 0xf9c9a035c1dac5b6 + .quad 0x6fb73e54c4dfb634 + .quad 0x4005419b1d2bc140 + .quad 0xa003febdb9be82f0 + .quad 0x2089c1af3a44ac90 + .quad 0xf8499f911954fa8e + .quad 0x1fba218aef40ab42 + + // 2^248 * 2 * G + + .quad 0xab549448fac8f53e + .quad 0x81f6e89a7ba63741 + .quad 0x74fd6c7d6c2b5e01 + .quad 0x392e3acaa8c86e42 + .quad 0x4f3e57043e7b0194 + .quad 0xa81d3eee08daaf7f + .quad 0xc839c6ab99dcdef1 + .quad 0x6c535d13ff7761d5 + .quad 0x4cbd34e93e8a35af + .quad 0x2e0781445887e816 + .quad 0x19319c76f29ab0ab + .quad 0x25e17fe4d50ac13b + + // 2^248 * 3 * G + + .quad 0x0a289bd71e04f676 + .quad 0x208e1c52d6420f95 + .quad 0x5186d8b034691fab + .quad 0x255751442a9fb351 + .quad 0x915f7ff576f121a7 + .quad 0xc34a32272fcd87e3 + .quad 0xccba2fde4d1be526 + .quad 0x6bba828f8969899b + .quad 0xe2d1bc6690fe3901 + .quad 0x4cb54a18a0997ad5 + .quad 0x971d6914af8460d4 + .quad 0x559d504f7f6b7be4 + + // 2^248 * 4 * G + + .quad 0xa7738378b3eb54d5 + .quad 0x1d69d366a5553c7c + .quad 0x0a26cf62f92800ba + .quad 0x01ab12d5807e3217 + .quad 0x9c4891e7f6d266fd + .quad 0x0744a19b0307781b + .quad 0x88388f1d6061e23b + .quad 0x123ea6a3354bd50e + .quad 0x118d189041e32d96 + .quad 0xb9ede3c2d8315848 + .quad 0x1eab4271d83245d9 + .quad 0x4a3961e2c918a154 + + // 2^248 * 5 * G + + .quad 0x71dc3be0f8e6bba0 + .quad 0xd6cef8347effe30a + .quad 0xa992425fe13a476a + .quad 0x2cd6bce3fb1db763 + .quad 0x0327d644f3233f1e + .quad 0x499a260e34fcf016 + .quad 0x83b5a716f2dab979 + .quad 0x68aceead9bd4111f + .quad 0x38b4c90ef3d7c210 + .quad 0x308e6e24b7ad040c + .quad 0x3860d9f1b7e73e23 + .quad 0x595760d5b508f597 + + // 2^248 * 6 * G + + .quad 0x6129bfe104aa6397 + .quad 0x8f960008a4a7fccb + .quad 0x3f8bc0897d909458 + .quad 0x709fa43edcb291a9 + .quad 0x882acbebfd022790 + .quad 0x89af3305c4115760 + .quad 0x65f492e37d3473f4 + .quad 0x2cb2c5df54515a2b + .quad 0xeb0a5d8c63fd2aca + .quad 0xd22bc1662e694eff + .quad 0x2723f36ef8cbb03a + .quad 0x70f029ecf0c8131f + + // 2^248 * 7 * G + + .quad 0x461307b32eed3e33 + .quad 0xae042f33a45581e7 + .quad 0xc94449d3195f0366 + .quad 0x0b7d5d8a6c314858 + .quad 0x2a6aafaa5e10b0b9 + .quad 0x78f0a370ef041aa9 + .quad 0x773efb77aa3ad61f + .quad 0x44eca5a2a74bd9e1 + .quad 0x25d448327b95d543 + .quad 0x70d38300a3340f1d + .quad 0xde1c531c60e1c52b + .quad 0x272224512c7de9e4 + + // 2^248 * 8 * G + + .quad 0x1abc92af49c5342e + .quad 0xffeed811b2e6fad0 + .quad 0xefa28c8dfcc84e29 + .quad 0x11b5df18a44cc543 + .quad 0xbf7bbb8a42a975fc + .quad 0x8c5c397796ada358 + .quad 0xe27fc76fcdedaa48 + .quad 0x19735fd7f6bc20a6 + .quad 0xe3ab90d042c84266 + .quad 0xeb848e0f7f19547e + .quad 0x2503a1d065a497b9 + .quad 0x0fef911191df895f + + // 2^252 * 1 * G + + .quad 0xb1507ca1ab1c6eb9 + .quad 0xbd448f3e16b687b3 + .quad 0x3455fb7f2c7a91ab + .quad 0x7579229e2f2adec1 + .quad 0x6ab5dcb85b1c16b7 + .quad 0x94c0fce83c7b27a5 + .quad 0xa4b11c1a735517be + .quad 0x499238d0ba0eafaa + .quad 0xecf46e527aba8b57 + .quad 0x15a08c478bd1647b + .quad 0x7af1c6a65f706fef + .quad 0x6345fa78f03a30d5 + + // 2^252 * 2 * G + + .quad 0xdf02f95f1015e7a1 + .quad 0x790ec41da9b40263 + .quad 0x4d3a0ea133ea1107 + .quad 0x54f70be7e33af8c9 + .quad 0x93d3cbe9bdd8f0a4 + .quad 0xdb152c1bfd177302 + .quad 0x7dbddc6d7f17a875 + .quad 0x3e1a71cc8f426efe + .quad 0xc83ca3e390babd62 + .quad 0x80ede3670291c833 + .quad 0xc88038ccd37900c4 + .quad 0x2c5fc0231ec31fa1 + + // 2^252 * 3 * G + + .quad 0xfeba911717038b4f + .quad 0xe5123721c9deef81 + .quad 0x1c97e4e75d0d8834 + .quad 0x68afae7a23dc3bc6 + .quad 0xc422e4d102456e65 + .quad 0x87414ac1cad47b91 + .quad 0x1592e2bba2b6ffdd + .quad 0x75d9d2bff5c2100f + .quad 0x5bd9b4763626e81c + .quad 0x89966936bca02edd + .quad 0x0a41193d61f077b3 + .quad 0x3097a24200ce5471 + + // 2^252 * 4 * G + + .quad 0x57427734c7f8b84c + .quad 0xf141a13e01b270e9 + .quad 0x02d1adfeb4e564a6 + .quad 0x4bb23d92ce83bd48 + .quad 0xa162e7246695c486 + .quad 0x131d633435a89607 + .quad 0x30521561a0d12a37 + .quad 0x56704bada6afb363 + .quad 0xaf6c4aa752f912b9 + .quad 0x5e665f6cd86770c8 + .quad 0x4c35ac83a3c8cd58 + .quad 0x2b7a29c010a58a7e + + // 2^252 * 5 * G + + .quad 0xc4007f77d0c1cec3 + .quad 0x8d1020b6bac492f8 + .quad 0x32ec29d57e69daaf + .quad 0x599408759d95fce0 + .quad 0x33810a23bf00086e + .quad 0xafce925ee736ff7c + .quad 0x3d60e670e24922d4 + .quad 0x11ce9e714f96061b + .quad 0x219ef713d815bac1 + .quad 0xf141465d485be25c + .quad 0x6d5447cc4e513c51 + .quad 0x174926be5ef44393 + + // 2^252 * 6 * G + + .quad 0xb5deb2f9fc5bd5bb + .quad 0x92daa72ae1d810e1 + .quad 0xafc4cfdcb72a1c59 + .quad 0x497d78813fc22a24 + .quad 0x3ef5d41593ea022e + .quad 0x5cbcc1a20ed0eed6 + .quad 0x8fd24ecf07382c8c + .quad 0x6fa42ead06d8e1ad + .quad 0xe276824a1f73371f + .quad 0x7f7cf01c4f5b6736 + .quad 0x7e201fe304fa46e7 + .quad 0x785a36a357808c96 + + // 2^252 * 7 * G + + .quad 0x825fbdfd63014d2b + .quad 0xc852369c6ca7578b + .quad 0x5b2fcd285c0b5df0 + .quad 0x12ab214c58048c8f + .quad 0x070442985d517bc3 + .quad 0x6acd56c7ae653678 + .quad 0x00a27983985a7763 + .quad 0x5167effae512662b + .quad 0xbd4ea9e10f53c4b6 + .quad 0x1673dc5f8ac91a14 + .quad 0xa8f81a4e2acc1aba + .quad 0x33a92a7924332a25 + + // 2^252 * 8 * G + + .quad 0x9dd1f49927996c02 + .quad 0x0cb3b058e04d1752 + .quad 0x1f7e88967fd02c3e + .quad 0x2f964268cb8b3eb1 + .quad 0x7ba95ba0218f2ada + .quad 0xcff42287330fb9ca + .quad 0xdada496d56c6d907 + .quad 0x5380c296f4beee54 + .quad 0x9d4f270466898d0a + .quad 0x3d0987990aff3f7a + .quad 0xd09ef36267daba45 + .quad 0x7761455e7b1c669c diff --git a/cbits/s2n/x86_att/curve25519_x25519base_alt.S b/cbits/s2n/x86_att/curve25519_x25519base_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/curve25519_x25519base_alt.S @@ -0,0 +1,9965 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// The x25519 function for curve25519 on base element 9 +// Input scalar[4]; output res[4] +// +// extern void curve25519_x25519base_alt +// (uint64_t res[static 4],const uint64_t scalar[static 4]); +// +// The function has a second prototype considering the arguments as arrays +// of bytes rather than 64-bit words. The underlying code is the same, since +// the x86 platform is little-endian. +// +// extern void curve25519_x25519base_byte_alt +// (uint8_t res[static 32],const uint8_t scalar[static 32]); +// +// Given a scalar n, returns the X coordinate of n * G where G = (9,...) is +// the standard generator. The scalar is first slightly modified/mangled +// as specified in the relevant RFC (https://www.rfc-editor.org/rfc/rfc7748). +// +// Standard x86-64 ABI: RDI = res, RSI = scalar +// Microsoft x64 ABI: RCX = res, RDX = scalar +// ---------------------------------------------------------------------------- +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(curve25519_x25519base_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(curve25519_x25519base_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(curve25519_x25519base_alt) + S2N_BN_SYM_VISIBILITY_DIRECTIVE(curve25519_x25519base_byte_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(curve25519_x25519base_byte_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(curve25519_x25519base_byte_alt) + + .text + .balign 32 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Pointer-offset pairs for result and temporaries on stack with some aliasing. +// The result "resx" assumes the "res" pointer has been preloaded into %rbp. + +#define resx (0*NUMSIZE)(%rbp) + +#define scalar (0*NUMSIZE)(%rsp) + +#define tabent (1*NUMSIZE)(%rsp) +#define ymx_2 (1*NUMSIZE)(%rsp) +#define xpy_2 (2*NUMSIZE)(%rsp) +#define kxy_2 (3*NUMSIZE)(%rsp) + +#define acc (4*NUMSIZE)(%rsp) +#define x_1 (4*NUMSIZE)(%rsp) +#define y_1 (5*NUMSIZE)(%rsp) +#define z_1 (6*NUMSIZE)(%rsp) +#define w_1 (7*NUMSIZE)(%rsp) +#define x_3 (4*NUMSIZE)(%rsp) +#define y_3 (5*NUMSIZE)(%rsp) +#define z_3 (6*NUMSIZE)(%rsp) +#define w_3 (7*NUMSIZE)(%rsp) + +#define tmpspace (8*NUMSIZE)(%rsp) +#define t0 (8*NUMSIZE)(%rsp) +#define t1 (9*NUMSIZE)(%rsp) +#define t2 (10*NUMSIZE)(%rsp) +#define t3 (11*NUMSIZE)(%rsp) +#define t4 (12*NUMSIZE)(%rsp) +#define t5 (13*NUMSIZE)(%rsp) + +// Stable homes for the input result pointer, and other variables + +#define res 14*NUMSIZE(%rsp) + +#define i 14*NUMSIZE+8(%rsp) + +#define bias 14*NUMSIZE+16(%rsp) + +#define bf 14*NUMSIZE+24(%rsp) +#define ix 14*NUMSIZE+24(%rsp) + +#define tab 15*NUMSIZE(%rsp) + +// Total size to reserve on the stack + +#define NSPACE 488 + +// Macro wrapping up the basic field multiplication, only trivially +// different from a pure function call to bignum_mul_p25519_alt. + +#define mul_p25519(P0,P1,P2) \ + movq P1, %rax ; \ + mulq P2; \ + movq %rax, %r8 ; \ + movq %rdx, %r9 ; \ + xorq %r10, %r10 ; \ + xorq %r11, %r11 ; \ + movq P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + movq 0x8+P1, %rax ; \ + mulq P2; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + adcq $0x0, %r11 ; \ + xorq %r12, %r12 ; \ + movq P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq %r12, %r12 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + movq 0x10+P1, %rax ; \ + mulq P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + xorq %r13, %r13 ; \ + movq P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq %r13, %r13 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + movq 0x18+P1, %rax ; \ + mulq P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + xorq %r14, %r14 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq %r14, %r14 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + xorq %r15, %r15 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq %r15, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq $0x0, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + movl $0x26, %esi ; \ + movq %r12, %rax ; \ + mulq %rsi; \ + addq %rax, %r8 ; \ + adcq %rdx, %r9 ; \ + sbbq %rcx, %rcx ; \ + movq %r13, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %rcx, %rcx ; \ + movq %r14, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rcx, %rcx ; \ + movq %r15, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + xorq %rcx, %rcx ; \ + addq %rax, %r11 ; \ + movq %rdx, %r12 ; \ + adcq %rcx, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + leaq 0x1(%r12), %rax ; \ + movl $0x13, %esi ; \ + bts $63, %r11 ; \ + imulq %rsi, %rax ; \ + addq %rax, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + sbbq %rax, %rax ; \ + notq %rax; \ + andq %rsi, %rax ; \ + subq %rax, %r8 ; \ + sbbq %rcx, %r9 ; \ + sbbq %rcx, %r10 ; \ + sbbq %rcx, %r11 ; \ + btr $63, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// A version of multiplication that only guarantees output < 2 * p_25519. +// This basically skips the +1 and final correction in quotient estimation. + +#define mul_4(P0,P1,P2) \ + movq P1, %rax ; \ + mulq P2; \ + movq %rax, %r8 ; \ + movq %rdx, %r9 ; \ + xorq %r10, %r10 ; \ + xorq %r11, %r11 ; \ + movq P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + movq 0x8+P1, %rax ; \ + mulq P2; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + adcq $0x0, %r11 ; \ + xorq %r12, %r12 ; \ + movq P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq %r12, %r12 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + movq 0x10+P1, %rax ; \ + mulq P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + xorq %r13, %r13 ; \ + movq P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq %r13, %r13 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + movq 0x18+P1, %rax ; \ + mulq P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + xorq %r14, %r14 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq %r14, %r14 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + xorq %r15, %r15 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq %r15, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq $0x0, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + movl $0x26, %ebx ; \ + movq %r12, %rax ; \ + mulq %rbx; \ + addq %rax, %r8 ; \ + adcq %rdx, %r9 ; \ + sbbq %rcx, %rcx ; \ + movq %r13, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %rcx, %rcx ; \ + movq %r14, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rcx, %rcx ; \ + movq %r15, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + xorq %rcx, %rcx ; \ + addq %rax, %r11 ; \ + movq %rdx, %r12 ; \ + adcq %rcx, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + btr $0x3f, %r11 ; \ + movl $0x13, %edx ; \ + imulq %r12, %rdx ; \ + addq %rdx, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Modular subtraction with double modulus 2 * p_25519 = 2^256 - 38 + +#define sub_twice4(P0,P1,P2) \ + movq P1, %r8 ; \ + xorl %ebx, %ebx ; \ + subq P2, %r8 ; \ + movq 8+P1, %r9 ; \ + sbbq 8+P2, %r9 ; \ + movl $38, %ecx ; \ + movq 16+P1, %r10 ; \ + sbbq 16+P2, %r10 ; \ + movq 24+P1, %rax ; \ + sbbq 24+P2, %rax ; \ + cmovncq %rbx, %rcx ; \ + subq %rcx, %r8 ; \ + sbbq %rbx, %r9 ; \ + sbbq %rbx, %r10 ; \ + sbbq %rbx, %rax ; \ + movq %r8, P0 ; \ + movq %r9, 8+P0 ; \ + movq %r10, 16+P0 ; \ + movq %rax, 24+P0 + +// Modular addition and doubling with double modulus 2 * p_25519 = 2^256 - 38. +// This only ensures that the result fits in 4 digits, not that it is reduced +// even w.r.t. double modulus. The result is always correct modulo provided +// the sum of the inputs is < 2^256 + 2^256 - 38, so in particular provided +// at least one of them is reduced double modulo. + +#define add_twice4(P0,P1,P2) \ + movq P1, %r8 ; \ + xorl %ecx, %ecx ; \ + addq P2, %r8 ; \ + movq 0x8+P1, %r9 ; \ + adcq 0x8+P2, %r9 ; \ + movq 0x10+P1, %r10 ; \ + adcq 0x10+P2, %r10 ; \ + movq 0x18+P1, %r11 ; \ + adcq 0x18+P2, %r11 ; \ + movl $38, %eax ; \ + cmovncq %rcx, %rax ; \ + addq %rax, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +#define double_twice4(P0,P1) \ + movq P1, %r8 ; \ + xorl %ecx, %ecx ; \ + addq %r8, %r8 ; \ + movq 0x8+P1, %r9 ; \ + adcq %r9, %r9 ; \ + movq 0x10+P1, %r10 ; \ + adcq %r10, %r10 ; \ + movq 0x18+P1, %r11 ; \ + adcq %r11, %r11 ; \ + movl $38, %eax ; \ + cmovncq %rcx, %rax ; \ + addq %rax, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +S2N_BN_SYMBOL(curve25519_x25519base_alt): +S2N_BN_SYMBOL(curve25519_x25519base_byte_alt): + CFI_START + _CET_ENDBR + +// In this case the Windows form literally makes a subroutine call. +// This avoids hassle arising from keeping code and data together. + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + CFI_CALL(Lcurve25519_x25519base_alt_standard) + CFI_POP(%rsi) + CFI_POP(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(curve25519_x25519base_alt) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lcurve25519_x25519base_alt_standard) + +Lcurve25519_x25519base_alt_standard: + CFI_START +#endif + +// Save registers, make room for temps, preserve input arguments. + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(NSPACE) + +// Move the output pointer to a stable place + + movq %rdi, res + +// Copy the input scalar to its local variable while mangling it. +// In principle the mangling is into 01xxx...xxx000, but actually +// we only clear the top two bits so 00xxx...xxxxxx. The additional +// 2^254 * G is taken care of by the starting value for the addition +// chain below, while we never look at the three low bits at all. + + movq (%rsi), %rax + movq %rax, (%rsp) + movq 8(%rsi), %rax + movq %rax, 8(%rsp) + movq 16(%rsi), %rax + movq %rax, 16(%rsp) + movq $0x3fffffffffffffff, %rax + andq 24(%rsi), %rax + movq %rax, 24(%rsp) + +// The main part of the computation is on the edwards25519 curve in +// extended-projective coordinates (X,Y,Z,T), representing a point +// (x,y) via x = X/Z, y = Y/Z and x * y = T/Z (so X * Y = T * Z). +// Only at the very end do we translate back to curve25519. So G +// below means the generator within edwards25519 corresponding to +// (9,...) for curve25519, via the standard isomorphism. +// +// Initialize accumulator "acc" to either (2^254 + 8) * G or just 2^254 * G +// depending on bit 3 of the scalar, the only nonzero bit of the bottom 4. +// Thus, we have effectively dealt with bits 0, 1, 2, 3, 254 and 255. + + movq (%rsp), %rax + andq $8, %rax + + leaq S2N_BN_SYMBOL(curve25519_x25519base_alt_constant)(%rip), %r10 + leaq 8*12(%r10), %r11 + + movq (%r10), %rax + movq (%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*16(%rsp) + + movq 8*1(%r10), %rax + movq 8*1(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*17(%rsp) + + movq 8*2(%r10), %rax + movq 8*2(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*18(%rsp) + + movq 8*3(%r10), %rax + movq 8*3(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*19(%rsp) + + movq 8*4(%r10), %rax + movq 8*4(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*20(%rsp) + + movq 8*5(%r10), %rax + movq 8*5(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*21(%rsp) + + movq 8*6(%r10), %rax + movq 8*6(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*22(%rsp) + + movq 8*7(%r10), %rax + movq 8*7(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*23(%rsp) + + movl $1, %eax + movq %rax, 8*24(%rsp) + movl $0, %eax + movq %rax, 8*25(%rsp) + movq %rax, 8*26(%rsp) + movq %rax, 8*27(%rsp) + + movq 8*8(%r10), %rax + movq 8*8(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*28(%rsp) + + movq 8*9(%r10), %rax + movq 8*9(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*29(%rsp) + + movq 8*10(%r10), %rax + movq 8*10(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*30(%rsp) + + movq 8*11(%r10), %rax + movq 8*11(%r11), %rcx + cmovnzq %rcx, %rax + movq %rax, 8*31(%rsp) + +// The counter "i" tracks the bit position for which the scalar has +// already been absorbed, starting at 4 and going up in chunks of 4. +// +// The pointer "tab" points at the current block of the table for +// multiples (2^i * j) * G at the current bit position i; 1 <= j <= 8. +// +// The bias is always either 0 and 1 and needs to be added to the +// partially processed scalar implicitly. This is used to absorb 4 bits +// of scalar per iteration from 3-bit table indexing by exploiting +// negation: (16 * h + l) * G = (16 * (h + 1) - (16 - l)) * G is used +// when l >= 9. Note that we can't have any bias left over at the +// end because of the clearing of bit 255 of the scalar, meaning the +// l >= 9 case cannot arise on the last iteration. + + movq $4, i + leaq 8*24(%r10), %rax + movq %rax, tab + movq $0, bias + +// Start of the main loop, repeated 63 times for i = 4, 8, ..., 252 + +Lcurve25519_x25519base_alt_scalarloop: + +// Look at the next 4-bit field "bf", adding the previous bias as well. +// Choose the table index "ix" as bf when bf <= 8 and 16 - bf for bf >= 9, +// setting the bias to 1 for the next iteration in the latter case. + + movq i, %rax + movq %rax, %rcx + shrq $6, %rax + movq (%rsp,%rax,8), %rax // Exploiting scalar = sp exactly + shrq %cl, %rax + andq $15, %rax + addq bias, %rax + movq %rax, bf + + cmpq $9, bf + sbbq %rax, %rax + incq %rax + movq %rax, bias + + movq $16, %rdi + subq bf, %rdi + cmpq $0, bias + cmovzq bf, %rdi + movq %rdi, ix + +// Perform constant-time lookup in the table to get element number "ix". +// The table entry for the affine point (x,y) is actually a triple +// (y - x,x + y,2 * d * x * y) to precompute parts of the addition. +// Note that "ix" can be 0, so we set up the appropriate identity first. + + movl $1, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + movl $1, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + xorl %r13d, %r13d + xorl %r14d, %r14d + xorl %r15d, %r15d + + movq tab, %rbp + + cmpq $1, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $2, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $3, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $4, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $5, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $6, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $7, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $8, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + + addq $96, %rbp + movq %rbp, tab + +// We now have the triple from the table in registers as follows +// +// [%rdx;%rcx;%rbx;%rax] = y - x +// [%r11;%r10;%r9;%r8] = x + y +// [%r15;%r14;%r13;%r12] = 2 * d * x * y +// +// In case bias = 1 we need to negate this. For Edwards curves +// -(x,y) = (-x,y), i.e. we need to negate the x coordinate. +// In this processed encoding, that amounts to swapping the +// first two fields and negating the third. +// +// The optional negation here also pretends bias = 0 whenever +// ix = 0 so that it doesn't need to handle the case of zero +// inputs, since no non-trivial table entries are zero. Note +// that in the zero case the whole negation is trivial, and +// so indeed is the swapping. + + cmpq $0, bias + + movq %rax, %rsi + cmovnzq %r8, %rsi + cmovnzq %rax, %r8 + movq %rsi, 32(%rsp) + movq %r8, 64(%rsp) + + movq %rbx, %rsi + cmovnzq %r9, %rsi + cmovnzq %rbx, %r9 + movq %rsi, 40(%rsp) + movq %r9, 72(%rsp) + + movq %rcx, %rsi + cmovnzq %r10, %rsi + cmovnzq %rcx, %r10 + movq %rsi, 48(%rsp) + movq %r10, 80(%rsp) + + movq %rdx, %rsi + cmovnzq %r11, %rsi + cmovnzq %rdx, %r11 + movq %rsi, 56(%rsp) + movq %r11, 88(%rsp) + + movq $-19, %rax + movq $-1, %rbx + movq $-1, %rcx + movq $0x7fffffffffffffff, %rdx + subq %r12, %rax + sbbq %r13, %rbx + sbbq %r14, %rcx + sbbq %r15, %rdx + + movq ix, %r8 + movq bias, %r9 + testq %r8, %r8 + cmovzq %r8, %r9 + testq %r9, %r9 + + cmovzq %r12, %rax + cmovzq %r13, %rbx + cmovzq %r14, %rcx + cmovzq %r15, %rdx + movq %rax, 96(%rsp) + movq %rbx, 104(%rsp) + movq %rcx, 112(%rsp) + movq %rdx, 120(%rsp) + +// Extended-projective and precomputed mixed addition. +// This is effectively the same as calling the standalone +// function edwards25519_pepadd_alt(acc,acc,tabent), but we +// only retain slightly weaker normalization < 2 * p_25519 +// throughout the inner loop, so the computation is +// slightly different, and faster overall. + + double_twice4(t0,z_1) + sub_twice4(t1,y_1,x_1) + add_twice4(t2,y_1,x_1) + mul_4(t3,w_1,kxy_2) + mul_4(t1,t1,ymx_2) + mul_4(t2,t2,xpy_2) + sub_twice4(t4,t0,t3) + add_twice4(t0,t0,t3) + sub_twice4(t5,t2,t1) + add_twice4(t1,t2,t1) + mul_4(z_3,t4,t0) + mul_4(x_3,t5,t4) + mul_4(y_3,t0,t1) + mul_4(w_3,t5,t1) + +// End of the main loop; move on by 4 bits. + + addq $4, i + cmpq $256, i + jc Lcurve25519_x25519base_alt_scalarloop + +// Now we need to translate from Edwards curve edwards25519 back +// to the Montgomery form curve25519. The mapping in the affine +// representations is +// +// (x,y) |-> ((1 + y) / (1 - y), c * (1 + y) / ((1 - y) * x)) +// +// For x25519, we only need the x coordinate, and we compute this as +// +// (1 + y) / (1 - y) = (x + x * y) / (x - x * y) +// = (X/Z + T/Z) / (X/Z - T/Z) +// = (X + T) / (X - T) +// = (X + T) * inverse(X - T) +// +// We could equally well use (Z + Y) / (Z - Y), but the above has the +// same cost, and it more explicitly forces zero output whenever X = 0, +// regardless of how the modular inverse behaves on zero inputs. In +// the present setting (base point 9, mangled scalar) that doesn't +// really matter anyway since X = 0 never arises, but it seems a +// little bit tidier. Note that both Edwards point (0,1) which maps to +// the Montgomery point at infinity, and Edwards (0,-1) which maps to +// Montgomery (0,0) [this is the 2-torsion point] are both by definition +// mapped to 0 by the X coordinate mapping used to define curve25519. +// +// First the addition and subtraction: + + add_twice4(t1,x_3,w_3) + sub_twice4(t2,x_3,w_3) + +// Prepare to call the modular inverse function to get t0 = 1/t2 +// Note that this works for the weakly normalized z_3 equally well. +// The non-coprime case z_3 == 0 (mod p_25519) cannot arise anyway. + + leaq 256(%rsp), %rdi + leaq 320(%rsp), %rsi + +// Inline copy of bignum_inv_p25519, identical except for stripping out +// the prologue and epilogue saving and restoring registers and making +// and reclaiming room on the stack. For more details and explanations see +// "x86/curve25519/bignum_inv_p25519.S". Note that the stack it uses for +// its own temporaries is 208 bytes, so it has no effect on variables +// that are needed in the rest of our computation here: res, t0, t1, t2. + + movq %rdi, 0xc0(%rsp) + xorl %eax, %eax + leaq -0x13(%rax), %rcx + notq %rax + movq %rcx, (%rsp) + movq %rax, 0x8(%rsp) + movq %rax, 0x10(%rsp) + btr $0x3f, %rax + movq %rax, 0x18(%rsp) + movq (%rsi), %rdx + movq 0x8(%rsi), %rcx + movq 0x10(%rsi), %r8 + movq 0x18(%rsi), %r9 + movl $0x1, %eax + xorl %r10d, %r10d + bts $0x3f, %r9 + adcq %r10, %rax + imulq $0x13, %rax, %rax + addq %rax, %rdx + adcq %r10, %rcx + adcq %r10, %r8 + adcq %r10, %r9 + movl $0x13, %eax + cmovbq %r10, %rax + subq %rax, %rdx + sbbq %r10, %rcx + sbbq %r10, %r8 + sbbq %r10, %r9 + btr $0x3f, %r9 + movq %rdx, 0x20(%rsp) + movq %rcx, 0x28(%rsp) + movq %r8, 0x30(%rsp) + movq %r9, 0x38(%rsp) + xorl %eax, %eax + movq %rax, 0x40(%rsp) + movq %rax, 0x48(%rsp) + movq %rax, 0x50(%rsp) + movq %rax, 0x58(%rsp) + movabsq $0xa0f99e2375022099, %rax + movq %rax, 0x60(%rsp) + movabsq $0xa8c68f3f1d132595, %rax + movq %rax, 0x68(%rsp) + movabsq $0x6c6c893805ac5242, %rax + movq %rax, 0x70(%rsp) + movabsq $0x276508b241770615, %rax + movq %rax, 0x78(%rsp) + movq $0xa, 0x90(%rsp) + movq $0x1, 0x98(%rsp) + jmp Lcurve25519_x25519base_alt_midloop +Lcurve25519_x25519base_alt_inverseloop: + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + movq %r8, %rax + andq %r9, %rax + movq %r10, %rdi + andq %r11, %rdi + addq %rax, %rdi + movq %rdi, 0x80(%rsp) + movq %r12, %rax + andq %r13, %rax + movq %r14, %rsi + andq %r15, %rsi + addq %rax, %rsi + movq %rsi, 0x88(%rsp) + xorl %ebx, %ebx + movq (%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x20(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rdi + adcq %rdx, %rbx + xorl %ebp, %ebp + movq (%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x20(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + xorl %ecx, %ecx + movq 0x8(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x28(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + shrdq $0x3b, %rbx, %rdi + movq %rdi, (%rsp) + xorl %edi, %edi + movq 0x8(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rbp + adcq %rdx, %rdi + movq 0x28(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rdi + shrdq $0x3b, %rbp, %rsi + movq %rsi, 0x20(%rsp) + xorl %esi, %esi + movq 0x10(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rsi + movq 0x30(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rsi + shrdq $0x3b, %rcx, %rbx + movq %rbx, 0x8(%rsp) + xorl %ebx, %ebx + movq 0x10(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x30(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rdi + adcq %rdx, %rbx + shrdq $0x3b, %rdi, %rbp + movq %rbp, 0x28(%rsp) + movq 0x18(%rsp), %rax + xorq %r9, %rax + movq %rax, %rbp + sarq $0x3f, %rbp + andq %r8, %rbp + negq %rbp + mulq %r8 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x38(%rsp), %rax + xorq %r11, %rax + movq %rax, %rdx + sarq $0x3f, %rdx + andq %r10, %rdx + subq %rdx, %rbp + mulq %r10 + addq %rax, %rsi + adcq %rdx, %rbp + shrdq $0x3b, %rsi, %rcx + movq %rcx, 0x10(%rsp) + shrdq $0x3b, %rbp, %rsi + movq 0x18(%rsp), %rax + movq %rsi, 0x18(%rsp) + xorq %r13, %rax + movq %rax, %rsi + sarq $0x3f, %rsi + andq %r12, %rsi + negq %rsi + mulq %r12 + addq %rax, %rbx + adcq %rdx, %rsi + movq 0x38(%rsp), %rax + xorq %r15, %rax + movq %rax, %rdx + sarq $0x3f, %rdx + andq %r14, %rdx + subq %rdx, %rsi + mulq %r14 + addq %rax, %rbx + adcq %rdx, %rsi + shrdq $0x3b, %rbx, %rdi + movq %rdi, 0x30(%rsp) + shrdq $0x3b, %rsi, %rbx + movq %rbx, 0x38(%rsp) + movq 0x80(%rsp), %rbx + movq 0x88(%rsp), %rbp + xorl %ecx, %ecx + movq 0x40(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x60(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x40(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x40(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x60(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x60(%rsp) + xorl %ebx, %ebx + movq 0x48(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x68(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + xorl %ebp, %ebp + movq 0x48(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, 0x48(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x68(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, 0x68(%rsp) + xorl %ecx, %ecx + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x70(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x50(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x50(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x70(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x70(%rsp) + movq 0x58(%rsp), %rax + xorq %r9, %rax + movq %r9, %rbx + andq %r8, %rbx + negq %rbx + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x78(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbx + mulq %r10 + addq %rax, %rcx + adcq %rbx, %rdx + movq %rdx, %rbx + shldq $0x1, %rcx, %rdx + sarq $0x3f, %rbx + addq %rbx, %rdx + movl $0x13, %eax + imulq %rdx + movq 0x40(%rsp), %r8 + addq %rax, %r8 + movq %r8, 0x40(%rsp) + movq 0x48(%rsp), %r8 + adcq %rdx, %r8 + movq %r8, 0x48(%rsp) + movq 0x50(%rsp), %r8 + adcq %rbx, %r8 + movq %r8, 0x50(%rsp) + adcq %rbx, %rcx + shlq $0x3f, %rax + addq %rax, %rcx + movq 0x58(%rsp), %rax + movq %rcx, 0x58(%rsp) + xorq %r13, %rax + movq %r13, %rcx + andq %r12, %rcx + negq %rcx + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rcx + movq 0x78(%rsp), %rax + xorq %r15, %rax + movq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rcx + mulq %r14 + addq %rax, %rsi + adcq %rcx, %rdx + movq %rdx, %rcx + shldq $0x1, %rsi, %rdx + sarq $0x3f, %rcx + movl $0x13, %eax + addq %rcx, %rdx + imulq %rdx + movq 0x60(%rsp), %r8 + addq %rax, %r8 + movq %r8, 0x60(%rsp) + movq 0x68(%rsp), %r8 + adcq %rdx, %r8 + movq %r8, 0x68(%rsp) + movq 0x70(%rsp), %r8 + adcq %rcx, %r8 + movq %r8, 0x70(%rsp) + adcq %rcx, %rsi + shlq $0x3f, %rax + addq %rax, %rsi + movq %rsi, 0x78(%rsp) +Lcurve25519_x25519base_alt_midloop: + movq 0x98(%rsp), %rsi + movq (%rsp), %rdx + movq 0x20(%rsp), %rcx + movq %rdx, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + xorl %ebp, %ebp + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %rdx + leaq (%rcx,%rax), %rdi + shlq $0x16, %rdx + shlq $0x16, %rdi + sarq $0x2b, %rdx + sarq $0x2b, %rdi + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %rbx + leaq (%rcx,%rax), %rcx + sarq $0x2a, %rbx + sarq $0x2a, %rcx + movq %rdx, 0xa0(%rsp) + movq %rbx, 0xa8(%rsp) + movq %rdi, 0xb0(%rsp) + movq %rcx, 0xb8(%rsp) + movq (%rsp), %r12 + imulq %r12, %rdi + imulq %rdx, %r12 + movq 0x20(%rsp), %r13 + imulq %r13, %rbx + imulq %rcx, %r13 + addq %rbx, %r12 + addq %rdi, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r10 + shlq $0x16, %r8 + shlq $0x16, %r10 + sarq $0x2b, %r8 + sarq $0x2b, %r10 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r15 + leaq (%rcx,%rax), %r11 + sarq $0x2a, %r15 + sarq $0x2a, %r11 + movq %r13, %rbx + movq %r12, %rcx + imulq %r8, %r12 + imulq %r15, %rbx + addq %rbx, %r12 + imulq %r11, %r13 + imulq %r10, %rcx + addq %rcx, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq 0xa0(%rsp), %rax + imulq %r8, %rax + movq 0xb0(%rsp), %rdx + imulq %r15, %rdx + imulq 0xa8(%rsp), %r8 + imulq 0xb8(%rsp), %r15 + addq %r8, %r15 + leaq (%rax,%rdx), %r9 + movq 0xa0(%rsp), %rax + imulq %r10, %rax + movq 0xb0(%rsp), %rdx + imulq %r11, %rdx + imulq 0xa8(%rsp), %r10 + imulq 0xb8(%rsp), %r11 + addq %r10, %r11 + leaq (%rax,%rdx), %r13 + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r12 + shlq $0x15, %r8 + shlq $0x15, %r12 + sarq $0x2b, %r8 + sarq $0x2b, %r12 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r10 + leaq (%rcx,%rax), %r14 + sarq $0x2b, %r10 + sarq $0x2b, %r14 + movq %r9, %rax + imulq %r8, %rax + movq %r13, %rdx + imulq %r10, %rdx + imulq %r15, %r8 + imulq %r11, %r10 + addq %r8, %r10 + leaq (%rax,%rdx), %r8 + movq %r9, %rax + imulq %r12, %rax + movq %r13, %rdx + imulq %r14, %rdx + imulq %r15, %r12 + imulq %r11, %r14 + addq %r12, %r14 + leaq (%rax,%rdx), %r12 + movq %rsi, 0x98(%rsp) + decq 0x90(%rsp) + jne Lcurve25519_x25519base_alt_inverseloop + movq (%rsp), %rax + movq 0x20(%rsp), %rcx + imulq %r8, %rax + imulq %r10, %rcx + addq %rcx, %rax + sarq $0x3f, %rax + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + xorq %rax, %r9 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + xorq %rax, %r11 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + xorq %rax, %r13 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + xorq %rax, %r15 + movq %r8, %rax + andq %r9, %rax + movq %r10, %r12 + andq %r11, %r12 + addq %rax, %r12 + xorl %r13d, %r13d + movq 0x40(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x60(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movq 0x48(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r13 + adcq %rdx, %r14 + movq 0x68(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x70(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x58(%rsp), %rax + xorq %r9, %rax + andq %r8, %r9 + negq %r9 + mulq %r8 + addq %rax, %r15 + adcq %rdx, %r9 + movq 0x78(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %r9 + mulq %r10 + addq %rax, %r15 + adcq %rdx, %r9 + movq %r9, %rax + shldq $0x1, %r15, %rax + sarq $0x3f, %r9 + movl $0x13, %ebx + leaq 0x1(%rax,%r9,1), %rax + imulq %rbx + xorl %ebp, %ebp + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r9, %r14 + adcq %r9, %r15 + shlq $0x3f, %rax + addq %rax, %r15 + cmovns %rbp, %rbx + subq %rbx, %r12 + sbbq %rbp, %r13 + sbbq %rbp, %r14 + sbbq %rbp, %r15 + btr $0x3f, %r15 + movq 0xc0(%rsp), %rdi + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + +// The final result is (X + T) / (X - T) +// This is the only operation in the whole computation that +// fully reduces modulo p_25519 since now we want the canonical +// answer as output. + + movq res, %rbp + mul_p25519(resx,t1,t0) + +// Restore stack and registers + + CFI_INC_RSP(NSPACE) + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +#if WINDOWS_ABI +S2N_BN_SIZE_DIRECTIVE(Lcurve25519_x25519base_alt_standard) +#else +S2N_BN_SIZE_DIRECTIVE(curve25519_x25519base_alt) +#endif + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif + +// **************************************************************************** +// The precomputed data (all read-only). +// **************************************************************************** + +#if defined(__ELF__) +.section .rodata +.type S2N_BN_SYMBOL(curve25519_x25519base_alt_constant), %object +.size S2N_BN_SYMBOL(curve25519_x25519base_alt_constant), 48576 +#elif defined(__APPLE__) +.const_data +#endif + +S2N_BN_SYMBOL(curve25519_x25519base_alt_constant): + +// 2^254 * G and (2^254 + 8) * G in extended-projective coordinates +// but with z = 1 assumed and hence left out, so they are (X,Y,T) only. + + .quad 0x251037f7cf4e861d + .quad 0x10ede0fb19fb128f + .quad 0x96c033b175f5e2c8 + .quad 0x055f070d6c15fb0d + + .quad 0x7c52af2c97473e69 + .quad 0x022f82391bad8378 + .quad 0x9991e1b02adb476f + .quad 0x511144a03a99b855 + + .quad 0x5fafc3b88ff2e4ae + .quad 0x855e4ff0de1230ff + .quad 0x72e302a348492870 + .quad 0x1253c19e53dbe1bc + + .quad 0x331d086e0d9abcaa + .quad 0x1e23c96d311a10c9 + .quad 0x96d0f95e58c13478 + .quad 0x2f72f7384fcfcc59 + + .quad 0x39a6cd1cfd7d87c9 + .quad 0x9867a0abd8ae153a + .quad 0xa49d2a5f35986745 + .quad 0x57012940cdfe82e1 + + .quad 0x5046a6532ec5544a + .quad 0x6d674004739ff6c9 + .quad 0x9bbaa44b234a70e3 + .quad 0x5e6d8901138cf386 + +// Precomputed table of multiples of generator for edwards25519 +// all in precomputed extended-projective (y-x,x+y,2*d*x*y) triples. + + // 2^4 * 1 * G + + .quad 0x7ec851ca553e2df3 + .quad 0xa71284cba64878b3 + .quad 0xe6b5e4193288d1e7 + .quad 0x4cf210ec5a9a8883 + .quad 0x322d04a52d9021f6 + .quad 0xb9c19f3375c6bf9c + .quad 0x587a3a4342d20b09 + .quad 0x143b1cf8aa64fe61 + .quad 0x9f867c7d968acaab + .quad 0x5f54258e27092729 + .quad 0xd0a7d34bea180975 + .quad 0x21b546a3374126e1 + + // 2^4 * 2 * G + + .quad 0xa94ff858a2888343 + .quad 0xce0ed4565313ed3c + .quad 0xf55c3dcfb5bf34fa + .quad 0x0a653ca5c9eab371 + .quad 0x490a7a45d185218f + .quad 0x9a15377846049335 + .quad 0x0060ea09cc31e1f6 + .quad 0x7e041577f86ee965 + .quad 0x66b2a496ce5b67f3 + .quad 0xff5492d8bd569796 + .quad 0x503cec294a592cd0 + .quad 0x566943650813acb2 + + // 2^4 * 3 * G + + .quad 0xb818db0c26620798 + .quad 0x5d5c31d9606e354a + .quad 0x0982fa4f00a8cdc7 + .quad 0x17e12bcd4653e2d4 + .quad 0x5672f9eb1dabb69d + .quad 0xba70b535afe853fc + .quad 0x47ac0f752796d66d + .quad 0x32a5351794117275 + .quad 0xd3a644a6df648437 + .quad 0x703b6559880fbfdd + .quad 0xcb852540ad3a1aa5 + .quad 0x0900b3f78e4c6468 + + // 2^4 * 4 * G + + .quad 0x0a851b9f679d651b + .quad 0xe108cb61033342f2 + .quad 0xd601f57fe88b30a3 + .quad 0x371f3acaed2dd714 + .quad 0xed280fbec816ad31 + .quad 0x52d9595bd8e6efe3 + .quad 0x0fe71772f6c623f5 + .quad 0x4314030b051e293c + .quad 0xd560005efbf0bcad + .quad 0x8eb70f2ed1870c5e + .quad 0x201f9033d084e6a0 + .quad 0x4c3a5ae1ce7b6670 + + // 2^4 * 5 * G + + .quad 0x4138a434dcb8fa95 + .quad 0x870cf67d6c96840b + .quad 0xde388574297be82c + .quad 0x7c814db27262a55a + .quad 0xbaf875e4c93da0dd + .quad 0xb93282a771b9294d + .quad 0x80d63fb7f4c6c460 + .quad 0x6de9c73dea66c181 + .quad 0x478904d5a04df8f2 + .quad 0xfafbae4ab10142d3 + .quad 0xf6c8ac63555d0998 + .quad 0x5aac4a412f90b104 + + // 2^4 * 6 * G + + .quad 0xc64f326b3ac92908 + .quad 0x5551b282e663e1e0 + .quad 0x476b35f54a1a4b83 + .quad 0x1b9da3fe189f68c2 + .quad 0x603a0d0abd7f5134 + .quad 0x8089c932e1d3ae46 + .quad 0xdf2591398798bd63 + .quad 0x1c145cd274ba0235 + .quad 0x32e8386475f3d743 + .quad 0x365b8baf6ae5d9ef + .quad 0x825238b6385b681e + .quad 0x234929c1167d65e1 + + // 2^4 * 7 * G + + .quad 0x984decaba077ade8 + .quad 0x383f77ad19eb389d + .quad 0xc7ec6b7e2954d794 + .quad 0x59c77b3aeb7c3a7a + .quad 0x48145cc21d099fcf + .quad 0x4535c192cc28d7e5 + .quad 0x80e7c1e548247e01 + .quad 0x4a5f28743b2973ee + .quad 0xd3add725225ccf62 + .quad 0x911a3381b2152c5d + .quad 0xd8b39fad5b08f87d + .quad 0x6f05606b4799fe3b + + // 2^4 * 8 * G + + .quad 0x9ffe9e92177ba962 + .quad 0x98aee71d0de5cae1 + .quad 0x3ff4ae942d831044 + .quad 0x714de12e58533ac8 + .quad 0x5b433149f91b6483 + .quad 0xadb5dc655a2cbf62 + .quad 0x87fa8412632827b3 + .quad 0x60895e91ab49f8d8 + .quad 0xe9ecf2ed0cf86c18 + .quad 0xb46d06120735dfd4 + .quad 0xbc9da09804b96be7 + .quad 0x73e2e62fd96dc26b + + // 2^8 * 1 * G + + .quad 0xed5b635449aa515e + .quad 0xa865c49f0bc6823a + .quad 0x850c1fe95b42d1c4 + .quad 0x30d76d6f03d315b9 + .quad 0x2eccdd0e632f9c1d + .quad 0x51d0b69676893115 + .quad 0x52dfb76ba8637a58 + .quad 0x6dd37d49a00eef39 + .quad 0x6c4444172106e4c7 + .quad 0xfb53d680928d7f69 + .quad 0xb4739ea4694d3f26 + .quad 0x10c697112e864bb0 + + // 2^8 * 2 * G + + .quad 0x6493c4277dbe5fde + .quad 0x265d4fad19ad7ea2 + .quad 0x0e00dfc846304590 + .quad 0x25e61cabed66fe09 + .quad 0x0ca62aa08358c805 + .quad 0x6a3d4ae37a204247 + .quad 0x7464d3a63b11eddc + .quad 0x03bf9baf550806ef + .quad 0x3f13e128cc586604 + .quad 0x6f5873ecb459747e + .quad 0xa0b63dedcc1268f5 + .quad 0x566d78634586e22c + + // 2^8 * 3 * G + + .quad 0x1637a49f9cc10834 + .quad 0xbc8e56d5a89bc451 + .quad 0x1cb5ec0f7f7fd2db + .quad 0x33975bca5ecc35d9 + .quad 0xa1054285c65a2fd0 + .quad 0x6c64112af31667c3 + .quad 0x680ae240731aee58 + .quad 0x14fba5f34793b22a + .quad 0x3cd746166985f7d4 + .quad 0x593e5e84c9c80057 + .quad 0x2fc3f2b67b61131e + .quad 0x14829cea83fc526c + + // 2^8 * 4 * G + + .quad 0xff437b8497dd95c2 + .quad 0x6c744e30aa4eb5a7 + .quad 0x9e0c5d613c85e88b + .quad 0x2fd9c71e5f758173 + .quad 0x21e70b2f4e71ecb8 + .quad 0xe656ddb940a477e3 + .quad 0xbf6556cece1d4f80 + .quad 0x05fc3bc4535d7b7e + .quad 0x24b8b3ae52afdedd + .quad 0x3495638ced3b30cf + .quad 0x33a4bc83a9be8195 + .quad 0x373767475c651f04 + + // 2^8 * 5 * G + + .quad 0x2fba99fd40d1add9 + .quad 0xb307166f96f4d027 + .quad 0x4363f05215f03bae + .quad 0x1fbea56c3b18f999 + .quad 0x634095cb14246590 + .quad 0xef12144016c15535 + .quad 0x9e38140c8910bc60 + .quad 0x6bf5905730907c8c + .quad 0x0fa778f1e1415b8a + .quad 0x06409ff7bac3a77e + .quad 0x6f52d7b89aa29a50 + .quad 0x02521cf67a635a56 + + // 2^8 * 6 * G + + .quad 0x513fee0b0a9d5294 + .quad 0x8f98e75c0fdf5a66 + .quad 0xd4618688bfe107ce + .quad 0x3fa00a7e71382ced + .quad 0xb1146720772f5ee4 + .quad 0xe8f894b196079ace + .quad 0x4af8224d00ac824a + .quad 0x001753d9f7cd6cc4 + .quad 0x3c69232d963ddb34 + .quad 0x1dde87dab4973858 + .quad 0xaad7d1f9a091f285 + .quad 0x12b5fe2fa048edb6 + + // 2^8 * 7 * G + + .quad 0x71f0fbc496fce34d + .quad 0x73b9826badf35bed + .quad 0xd2047261ff28c561 + .quad 0x749b76f96fb1206f + .quad 0xdf2b7c26ad6f1e92 + .quad 0x4b66d323504b8913 + .quad 0x8c409dc0751c8bc3 + .quad 0x6f7e93c20796c7b8 + .quad 0x1f5af604aea6ae05 + .quad 0xc12351f1bee49c99 + .quad 0x61a808b5eeff6b66 + .quad 0x0fcec10f01e02151 + + // 2^8 * 8 * G + + .quad 0x644d58a649fe1e44 + .quad 0x21fcaea231ad777e + .quad 0x02441c5a887fd0d2 + .quad 0x4901aa7183c511f3 + .quad 0x3df2d29dc4244e45 + .quad 0x2b020e7493d8de0a + .quad 0x6cc8067e820c214d + .quad 0x413779166feab90a + .quad 0x08b1b7548c1af8f0 + .quad 0xce0f7a7c246299b4 + .quad 0xf760b0f91e06d939 + .quad 0x41bb887b726d1213 + + // 2^12 * 1 * G + + .quad 0x9267806c567c49d8 + .quad 0x066d04ccca791e6a + .quad 0xa69f5645e3cc394b + .quad 0x5c95b686a0788cd2 + .quad 0x97d980e0aa39f7d2 + .quad 0x35d0384252c6b51c + .quad 0x7d43f49307cd55aa + .quad 0x56bd36cfb78ac362 + .quad 0x2ac519c10d14a954 + .quad 0xeaf474b494b5fa90 + .quad 0xe6af8382a9f87a5a + .quad 0x0dea6db1879be094 + + // 2^12 * 2 * G + + .quad 0xaa66bf547344e5ab + .quad 0xda1258888f1b4309 + .quad 0x5e87d2b3fd564b2f + .quad 0x5b2c78885483b1dd + .quad 0x15baeb74d6a8797a + .quad 0x7ef55cf1fac41732 + .quad 0x29001f5a3c8b05c5 + .quad 0x0ad7cc8752eaccfb + .quad 0x52151362793408cf + .quad 0xeb0f170319963d94 + .quad 0xa833b2fa883d9466 + .quad 0x093a7fa775003c78 + + // 2^12 * 3 * G + + .quad 0xe5107de63a16d7be + .quad 0xa377ffdc9af332cf + .quad 0x70d5bf18440b677f + .quad 0x6a252b19a4a31403 + .quad 0xb8e9604460a91286 + .quad 0x7f3fd8047778d3de + .quad 0x67d01e31bf8a5e2d + .quad 0x7b038a06c27b653e + .quad 0x9ed919d5d36990f3 + .quad 0x5213aebbdb4eb9f2 + .quad 0xc708ea054cb99135 + .quad 0x58ded57f72260e56 + + // 2^12 * 4 * G + + .quad 0x78e79dade9413d77 + .quad 0xf257f9d59729e67d + .quad 0x59db910ee37aa7e6 + .quad 0x6aa11b5bbb9e039c + .quad 0xda6d53265b0fd48b + .quad 0x8960823193bfa988 + .quad 0xd78ac93261d57e28 + .quad 0x79f2942d3a5c8143 + .quad 0x97da2f25b6c88de9 + .quad 0x251ba7eaacf20169 + .quad 0x09b44f87ef4eb4e4 + .quad 0x7d90ab1bbc6a7da5 + + // 2^12 * 5 * G + + .quad 0x9acca683a7016bfe + .quad 0x90505f4df2c50b6d + .quad 0x6b610d5fcce435aa + .quad 0x19a10d446198ff96 + .quad 0x1a07a3f496b3c397 + .quad 0x11ceaa188f4e2532 + .quad 0x7d9498d5a7751bf0 + .quad 0x19ed161f508dd8a0 + .quad 0x560a2cd687dce6ca + .quad 0x7f3568c48664cf4d + .quad 0x8741e95222803a38 + .quad 0x483bdab1595653fc + + // 2^12 * 6 * G + + .quad 0xfa780f148734fa49 + .quad 0x106f0b70360534e0 + .quad 0x2210776fe3e307bd + .quad 0x3286c109dde6a0fe + .quad 0xd6cf4d0ab4da80f6 + .quad 0x82483e45f8307fe0 + .quad 0x05005269ae6f9da4 + .quad 0x1c7052909cf7877a + .quad 0x32ee7de2874e98d4 + .quad 0x14c362e9b97e0c60 + .quad 0x5781dcde6a60a38a + .quad 0x217dd5eaaa7aa840 + + // 2^12 * 7 * G + + .quad 0x9db7c4d0248e1eb0 + .quad 0xe07697e14d74bf52 + .quad 0x1e6a9b173c562354 + .quad 0x7fa7c21f795a4965 + .quad 0x8bdf1fb9be8c0ec8 + .quad 0x00bae7f8e30a0282 + .quad 0x4963991dad6c4f6c + .quad 0x07058a6e5df6f60a + .quad 0xe9eb02c4db31f67f + .quad 0xed25fd8910bcfb2b + .quad 0x46c8131f5c5cddb4 + .quad 0x33b21c13a0cb9bce + + // 2^12 * 8 * G + + .quad 0x360692f8087d8e31 + .quad 0xf4dcc637d27163f7 + .quad 0x25a4e62065ea5963 + .quad 0x659bf72e5ac160d9 + .quad 0x9aafb9b05ee38c5b + .quad 0xbf9d2d4e071a13c7 + .quad 0x8eee6e6de933290a + .quad 0x1c3bab17ae109717 + .quad 0x1c9ab216c7cab7b0 + .quad 0x7d65d37407bbc3cc + .quad 0x52744750504a58d5 + .quad 0x09f2606b131a2990 + + // 2^16 * 1 * G + + .quad 0x40e87d44744346be + .quad 0x1d48dad415b52b25 + .quad 0x7c3a8a18a13b603e + .quad 0x4eb728c12fcdbdf7 + .quad 0x7e234c597c6691ae + .quad 0x64889d3d0a85b4c8 + .quad 0xdae2c90c354afae7 + .quad 0x0a871e070c6a9e1d + .quad 0x3301b5994bbc8989 + .quad 0x736bae3a5bdd4260 + .quad 0x0d61ade219d59e3c + .quad 0x3ee7300f2685d464 + + // 2^16 * 2 * G + + .quad 0xf5d255e49e7dd6b7 + .quad 0x8016115c610b1eac + .quad 0x3c99975d92e187ca + .quad 0x13815762979125c2 + .quad 0x43fa7947841e7518 + .quad 0xe5c6fa59639c46d7 + .quad 0xa1065e1de3052b74 + .quad 0x7d47c6a2cfb89030 + .quad 0x3fdad0148ef0d6e0 + .quad 0x9d3e749a91546f3c + .quad 0x71ec621026bb8157 + .quad 0x148cf58d34c9ec80 + + // 2^16 * 3 * G + + .quad 0x46a492f67934f027 + .quad 0x469984bef6840aa9 + .quad 0x5ca1bc2a89611854 + .quad 0x3ff2fa1ebd5dbbd4 + .quad 0xe2572f7d9ae4756d + .quad 0x56c345bb88f3487f + .quad 0x9fd10b6d6960a88d + .quad 0x278febad4eaea1b9 + .quad 0xb1aa681f8c933966 + .quad 0x8c21949c20290c98 + .quad 0x39115291219d3c52 + .quad 0x4104dd02fe9c677b + + // 2^16 * 4 * G + + .quad 0x72b2bf5e1124422a + .quad 0xa1fa0c3398a33ab5 + .quad 0x94cb6101fa52b666 + .quad 0x2c863b00afaf53d5 + .quad 0x81214e06db096ab8 + .quad 0x21a8b6c90ce44f35 + .quad 0x6524c12a409e2af5 + .quad 0x0165b5a48efca481 + .quad 0xf190a474a0846a76 + .quad 0x12eff984cd2f7cc0 + .quad 0x695e290658aa2b8f + .quad 0x591b67d9bffec8b8 + + // 2^16 * 5 * G + + .quad 0x312f0d1c80b49bfa + .quad 0x5979515eabf3ec8a + .quad 0x727033c09ef01c88 + .quad 0x3de02ec7ca8f7bcb + .quad 0x99b9b3719f18b55d + .quad 0xe465e5faa18c641e + .quad 0x61081136c29f05ed + .quad 0x489b4f867030128b + .quad 0xd232102d3aeb92ef + .quad 0xe16253b46116a861 + .quad 0x3d7eabe7190baa24 + .quad 0x49f5fbba496cbebf + + // 2^16 * 6 * G + + .quad 0x30949a108a5bcfd4 + .quad 0xdc40dd70bc6473eb + .quad 0x92c294c1307c0d1c + .quad 0x5604a86dcbfa6e74 + .quad 0x155d628c1e9c572e + .quad 0x8a4d86acc5884741 + .quad 0x91a352f6515763eb + .quad 0x06a1a6c28867515b + .quad 0x7288d1d47c1764b6 + .quad 0x72541140e0418b51 + .quad 0x9f031a6018acf6d1 + .quad 0x20989e89fe2742c6 + + // 2^16 * 7 * G + + .quad 0x499777fd3a2dcc7f + .quad 0x32857c2ca54fd892 + .quad 0xa279d864d207e3a0 + .quad 0x0403ed1d0ca67e29 + .quad 0x1674278b85eaec2e + .quad 0x5621dc077acb2bdf + .quad 0x640a4c1661cbf45a + .quad 0x730b9950f70595d3 + .quad 0xc94b2d35874ec552 + .quad 0xc5e6c8cf98246f8d + .quad 0xf7cb46fa16c035ce + .quad 0x5bd7454308303dcc + + // 2^16 * 8 * G + + .quad 0x7f9ad19528b24cc2 + .quad 0x7f6b54656335c181 + .quad 0x66b8b66e4fc07236 + .quad 0x133a78007380ad83 + .quad 0x85c4932115e7792a + .quad 0xc64c89a2bdcdddc9 + .quad 0x9d1e3da8ada3d762 + .quad 0x5bb7db123067f82c + .quad 0x0961f467c6ca62be + .quad 0x04ec21d6211952ee + .quad 0x182360779bd54770 + .quad 0x740dca6d58f0e0d2 + + // 2^20 * 1 * G + + .quad 0x50b70bf5d3f0af0b + .quad 0x4feaf48ae32e71f7 + .quad 0x60e84ed3a55bbd34 + .quad 0x00ed489b3f50d1ed + .quad 0x3906c72aed261ae5 + .quad 0x9ab68fd988e100f7 + .quad 0xf5e9059af3360197 + .quad 0x0e53dc78bf2b6d47 + .quad 0xb90829bf7971877a + .quad 0x5e4444636d17e631 + .quad 0x4d05c52e18276893 + .quad 0x27632d9a5a4a4af5 + + // 2^20 * 2 * G + + .quad 0xd11ff05154b260ce + .quad 0xd86dc38e72f95270 + .quad 0x601fcd0d267cc138 + .quad 0x2b67916429e90ccd + .quad 0xa98285d187eaffdb + .quad 0xa5b4fbbbd8d0a864 + .quad 0xb658f27f022663f7 + .quad 0x3bbc2b22d99ce282 + .quad 0xb917c952583c0a58 + .quad 0x653ff9b80fe4c6f3 + .quad 0x9b0da7d7bcdf3c0c + .quad 0x43a0eeb6ab54d60e + + // 2^20 * 3 * G + + .quad 0x396966a46d4a5487 + .quad 0xf811a18aac2bb3ba + .quad 0x66e4685b5628b26b + .quad 0x70a477029d929b92 + .quad 0x3ac6322357875fe8 + .quad 0xd9d4f4ecf5fbcb8f + .quad 0x8dee8493382bb620 + .quad 0x50c5eaa14c799fdc + .quad 0xdd0edc8bd6f2fb3c + .quad 0x54c63aa79cc7b7a0 + .quad 0xae0b032b2c8d9f1a + .quad 0x6f9ce107602967fb + + // 2^20 * 4 * G + + .quad 0xad1054b1cde1c22a + .quad 0xc4a8e90248eb32df + .quad 0x5f3e7b33accdc0ea + .quad 0x72364713fc79963e + .quad 0x139693063520e0b5 + .quad 0x437fcf7c88ea03fe + .quad 0xf7d4c40bd3c959bc + .quad 0x699154d1f893ded9 + .quad 0x315d5c75b4b27526 + .quad 0xcccb842d0236daa5 + .quad 0x22f0c8a3345fee8e + .quad 0x73975a617d39dbed + + // 2^20 * 5 * G + + .quad 0xe4024df96375da10 + .quad 0x78d3251a1830c870 + .quad 0x902b1948658cd91c + .quad 0x7e18b10b29b7438a + .quad 0x6f37f392f4433e46 + .quad 0x0e19b9a11f566b18 + .quad 0x220fb78a1fd1d662 + .quad 0x362a4258a381c94d + .quad 0x9071d9132b6beb2f + .quad 0x0f26e9ad28418247 + .quad 0xeab91ec9bdec925d + .quad 0x4be65bc8f48af2de + + // 2^20 * 6 * G + + .quad 0x78487feba36e7028 + .quad 0x5f3f13001dd8ce34 + .quad 0x934fb12d4b30c489 + .quad 0x056c244d397f0a2b + .quad 0x1d50fba257c26234 + .quad 0x7bd4823adeb0678b + .quad 0xc2b0dc6ea6538af5 + .quad 0x5665eec6351da73e + .quad 0xdb3ee00943bfb210 + .quad 0x4972018720800ac2 + .quad 0x26ab5d6173bd8667 + .quad 0x20b209c2ab204938 + + // 2^20 * 7 * G + + .quad 0x549e342ac07fb34b + .quad 0x02d8220821373d93 + .quad 0xbc262d70acd1f567 + .quad 0x7a92c9fdfbcac784 + .quad 0x1fcca94516bd3289 + .quad 0x448d65aa41420428 + .quad 0x59c3b7b216a55d62 + .quad 0x49992cc64e612cd8 + .quad 0x65bd1bea70f801de + .quad 0x1befb7c0fe49e28a + .quad 0xa86306cdb1b2ae4a + .quad 0x3b7ac0cd265c2a09 + + // 2^20 * 8 * G + + .quad 0x822bee438c01bcec + .quad 0x530cb525c0fbc73b + .quad 0x48519034c1953fe9 + .quad 0x265cc261e09a0f5b + .quad 0xf0d54e4f22ed39a7 + .quad 0xa2aae91e5608150a + .quad 0xf421b2e9eddae875 + .quad 0x31bc531d6b7de992 + .quad 0xdf3d134da980f971 + .quad 0x7a4fb8d1221a22a7 + .quad 0x3df7d42035aad6d8 + .quad 0x2a14edcc6a1a125e + + // 2^24 * 1 * G + + .quad 0xdf48ee0752cfce4e + .quad 0xc3fffaf306ec08b7 + .quad 0x05710b2ab95459c4 + .quad 0x161d25fa963ea38d + .quad 0x231a8c570478433c + .quad 0xb7b5270ec281439d + .quad 0xdbaa99eae3d9079f + .quad 0x2c03f5256c2b03d9 + .quad 0x790f18757b53a47d + .quad 0x307b0130cf0c5879 + .quad 0x31903d77257ef7f9 + .quad 0x699468bdbd96bbaf + + // 2^24 * 2 * G + + .quad 0xbd1f2f46f4dafecf + .quad 0x7cef0114a47fd6f7 + .quad 0xd31ffdda4a47b37f + .quad 0x525219a473905785 + .quad 0xd8dd3de66aa91948 + .quad 0x485064c22fc0d2cc + .quad 0x9b48246634fdea2f + .quad 0x293e1c4e6c4a2e3a + .quad 0x376e134b925112e1 + .quad 0x703778b5dca15da0 + .quad 0xb04589af461c3111 + .quad 0x5b605c447f032823 + + // 2^24 * 3 * G + + .quad 0xb965805920c47c89 + .quad 0xe7f0100c923b8fcc + .quad 0x0001256502e2ef77 + .quad 0x24a76dcea8aeb3ee + .quad 0x3be9fec6f0e7f04c + .quad 0x866a579e75e34962 + .quad 0x5542ef161e1de61a + .quad 0x2f12fef4cc5abdd5 + .quad 0x0a4522b2dfc0c740 + .quad 0x10d06e7f40c9a407 + .quad 0xc6cf144178cff668 + .quad 0x5e607b2518a43790 + + // 2^24 * 4 * G + + .quad 0x58b31d8f6cdf1818 + .quad 0x35cfa74fc36258a2 + .quad 0xe1b3ff4f66e61d6e + .quad 0x5067acab6ccdd5f7 + .quad 0xa02c431ca596cf14 + .quad 0xe3c42d40aed3e400 + .quad 0xd24526802e0f26db + .quad 0x201f33139e457068 + .quad 0xfd527f6b08039d51 + .quad 0x18b14964017c0006 + .quad 0xd5220eb02e25a4a8 + .quad 0x397cba8862460375 + + // 2^24 * 5 * G + + .quad 0x30c13093f05959b2 + .quad 0xe23aa18de9a97976 + .quad 0x222fd491721d5e26 + .quad 0x2339d320766e6c3a + .quad 0x7815c3fbc81379e7 + .quad 0xa6619420dde12af1 + .quad 0xffa9c0f885a8fdd5 + .quad 0x771b4022c1e1c252 + .quad 0xd87dd986513a2fa7 + .quad 0xf5ac9b71f9d4cf08 + .quad 0xd06bc31b1ea283b3 + .quad 0x331a189219971a76 + + // 2^24 * 6 * G + + .quad 0xf5166f45fb4f80c6 + .quad 0x9c36c7de61c775cf + .quad 0xe3d4e81b9041d91c + .quad 0x31167c6b83bdfe21 + .quad 0x26512f3a9d7572af + .quad 0x5bcbe28868074a9e + .quad 0x84edc1c11180f7c4 + .quad 0x1ac9619ff649a67b + .quad 0xf22b3842524b1068 + .quad 0x5068343bee9ce987 + .quad 0xfc9d71844a6250c8 + .quad 0x612436341f08b111 + + // 2^24 * 7 * G + + .quad 0xd99d41db874e898d + .quad 0x09fea5f16c07dc20 + .quad 0x793d2c67d00f9bbc + .quad 0x46ebe2309e5eff40 + .quad 0x8b6349e31a2d2638 + .quad 0x9ddfb7009bd3fd35 + .quad 0x7f8bf1b8a3a06ba4 + .quad 0x1522aa3178d90445 + .quad 0x2c382f5369614938 + .quad 0xdafe409ab72d6d10 + .quad 0xe8c83391b646f227 + .quad 0x45fe70f50524306c + + // 2^24 * 8 * G + + .quad 0xda4875a6960c0b8c + .quad 0x5b68d076ef0e2f20 + .quad 0x07fb51cf3d0b8fd4 + .quad 0x428d1623a0e392d4 + .quad 0x62f24920c8951491 + .quad 0x05f007c83f630ca2 + .quad 0x6fbb45d2f5c9d4b8 + .quad 0x16619f6db57a2245 + .quad 0x084f4a4401a308fd + .quad 0xa82219c376a5caac + .quad 0xdeb8de4643d1bc7d + .quad 0x1d81592d60bd38c6 + + // 2^28 * 1 * G + + .quad 0xd833d7beec2a4c38 + .quad 0x2c9162830acc20ed + .quad 0xe93a47aa92df7581 + .quad 0x702d67a3333c4a81 + .quad 0x3a4a369a2f89c8a1 + .quad 0x63137a1d7c8de80d + .quad 0xbcac008a78eda015 + .quad 0x2cb8b3a5b483b03f + .quad 0x36e417cbcb1b90a1 + .quad 0x33b3ddaa7f11794e + .quad 0x3f510808885bc607 + .quad 0x24141dc0e6a8020d + + // 2^28 * 2 * G + + .quad 0x59f73c773fefee9d + .quad 0xb3f1ef89c1cf989d + .quad 0xe35dfb42e02e545f + .quad 0x5766120b47a1b47c + .quad 0x91925dccbd83157d + .quad 0x3ca1205322cc8094 + .quad 0x28e57f183f90d6e4 + .quad 0x1a4714cede2e767b + .quad 0xdb20ba0fb8b6b7ff + .quad 0xb732c3b677511fa1 + .quad 0xa92b51c099f02d89 + .quad 0x4f3875ad489ca5f1 + + // 2^28 * 3 * G + + .quad 0xc7fc762f4932ab22 + .quad 0x7ac0edf72f4c3c1b + .quad 0x5f6b55aa9aa895e8 + .quad 0x3680274dad0a0081 + .quad 0x79ed13f6ee73eec0 + .quad 0xa5c6526d69110bb1 + .quad 0xe48928c38603860c + .quad 0x722a1446fd7059f5 + .quad 0xd0959fe9a8cf8819 + .quad 0xd0a995508475a99c + .quad 0x6eac173320b09cc5 + .quad 0x628ecf04331b1095 + + // 2^28 * 4 * G + + .quad 0x98bcb118a9d0ddbc + .quad 0xee449e3408b4802b + .quad 0x87089226b8a6b104 + .quad 0x685f349a45c7915d + .quad 0x9b41acf85c74ccf1 + .quad 0xb673318108265251 + .quad 0x99c92aed11adb147 + .quad 0x7a47d70d34ecb40f + .quad 0x60a0c4cbcc43a4f5 + .quad 0x775c66ca3677bea9 + .quad 0xa17aa1752ff8f5ed + .quad 0x11ded9020e01fdc0 + + // 2^28 * 5 * G + + .quad 0x890e7809caefe704 + .quad 0x8728296de30e8c6c + .quad 0x4c5cd2a392aeb1c9 + .quad 0x194263d15771531f + .quad 0x471f95b03bea93b7 + .quad 0x0552d7d43313abd3 + .quad 0xbd9370e2e17e3f7b + .quad 0x7b120f1db20e5bec + .quad 0x17d2fb3d86502d7a + .quad 0xb564d84450a69352 + .quad 0x7da962c8a60ed75d + .quad 0x00d0f85b318736aa + + // 2^28 * 6 * G + + .quad 0x978b142e777c84fd + .quad 0xf402644705a8c062 + .quad 0xa67ad51be7e612c7 + .quad 0x2f7b459698dd6a33 + .quad 0xa6753c1efd7621c1 + .quad 0x69c0b4a7445671f5 + .quad 0x971f527405b23c11 + .quad 0x387bc74851a8c7cd + .quad 0x81894b4d4a52a9a8 + .quad 0xadd93e12f6b8832f + .quad 0x184d8548b61bd638 + .quad 0x3f1c62dbd6c9f6cd + + // 2^28 * 7 * G + + .quad 0x2e8f1f0091910c1f + .quad 0xa4df4fe0bff2e12c + .quad 0x60c6560aee927438 + .quad 0x6338283facefc8fa + .quad 0x3fad3e40148f693d + .quad 0x052656e194eb9a72 + .quad 0x2f4dcbfd184f4e2f + .quad 0x406f8db1c482e18b + .quad 0x9e630d2c7f191ee4 + .quad 0x4fbf8301bc3ff670 + .quad 0x787d8e4e7afb73c4 + .quad 0x50d83d5be8f58fa5 + + // 2^28 * 8 * G + + .quad 0x85683916c11a1897 + .quad 0x2d69a4efe506d008 + .quad 0x39af1378f664bd01 + .quad 0x65942131361517c6 + .quad 0xc0accf90b4d3b66d + .quad 0xa7059de561732e60 + .quad 0x033d1f7870c6b0ba + .quad 0x584161cd26d946e4 + .quad 0xbbf2b1a072d27ca2 + .quad 0xbf393c59fbdec704 + .quad 0xe98dbbcee262b81e + .quad 0x02eebd0b3029b589 + + // 2^32 * 1 * G + + .quad 0x61368756a60dac5f + .quad 0x17e02f6aebabdc57 + .quad 0x7f193f2d4cce0f7d + .quad 0x20234a7789ecdcf0 + .quad 0x8765b69f7b85c5e8 + .quad 0x6ff0678bd168bab2 + .quad 0x3a70e77c1d330f9b + .quad 0x3a5f6d51b0af8e7c + .quad 0x76d20db67178b252 + .quad 0x071c34f9d51ed160 + .quad 0xf62a4a20b3e41170 + .quad 0x7cd682353cffe366 + + // 2^32 * 2 * G + + .quad 0x0be1a45bd887fab6 + .quad 0x2a846a32ba403b6e + .quad 0xd9921012e96e6000 + .quad 0x2838c8863bdc0943 + .quad 0xa665cd6068acf4f3 + .quad 0x42d92d183cd7e3d3 + .quad 0x5759389d336025d9 + .quad 0x3ef0253b2b2cd8ff + .quad 0xd16bb0cf4a465030 + .quad 0xfa496b4115c577ab + .quad 0x82cfae8af4ab419d + .quad 0x21dcb8a606a82812 + + // 2^32 * 3 * G + + .quad 0x5c6004468c9d9fc8 + .quad 0x2540096ed42aa3cb + .quad 0x125b4d4c12ee2f9c + .quad 0x0bc3d08194a31dab + .quad 0x9a8d00fabe7731ba + .quad 0x8203607e629e1889 + .quad 0xb2cc023743f3d97f + .quad 0x5d840dbf6c6f678b + .quad 0x706e380d309fe18b + .quad 0x6eb02da6b9e165c7 + .quad 0x57bbba997dae20ab + .quad 0x3a4276232ac196dd + + // 2^32 * 4 * G + + .quad 0x4b42432c8a7084fa + .quad 0x898a19e3dfb9e545 + .quad 0xbe9f00219c58e45d + .quad 0x1ff177cea16debd1 + .quad 0x3bf8c172db447ecb + .quad 0x5fcfc41fc6282dbd + .quad 0x80acffc075aa15fe + .quad 0x0770c9e824e1a9f9 + .quad 0xcf61d99a45b5b5fd + .quad 0x860984e91b3a7924 + .quad 0xe7300919303e3e89 + .quad 0x39f264fd41500b1e + + // 2^32 * 5 * G + + .quad 0xa7ad3417dbe7e29c + .quad 0xbd94376a2b9c139c + .quad 0xa0e91b8e93597ba9 + .quad 0x1712d73468889840 + .quad 0xd19b4aabfe097be1 + .quad 0xa46dfce1dfe01929 + .quad 0xc3c908942ca6f1ff + .quad 0x65c621272c35f14e + .quad 0xe72b89f8ce3193dd + .quad 0x4d103356a125c0bb + .quad 0x0419a93d2e1cfe83 + .quad 0x22f9800ab19ce272 + + // 2^32 * 6 * G + + .quad 0x605a368a3e9ef8cb + .quad 0xe3e9c022a5504715 + .quad 0x553d48b05f24248f + .quad 0x13f416cd647626e5 + .quad 0x42029fdd9a6efdac + .quad 0xb912cebe34a54941 + .quad 0x640f64b987bdf37b + .quad 0x4171a4d38598cab4 + .quad 0xfa2758aa99c94c8c + .quad 0x23006f6fb000b807 + .quad 0xfbd291ddadda5392 + .quad 0x508214fa574bd1ab + + // 2^32 * 7 * G + + .quad 0xc20269153ed6fe4b + .quad 0xa65a6739511d77c4 + .quad 0xcbde26462c14af94 + .quad 0x22f960ec6faba74b + .quad 0x461a15bb53d003d6 + .quad 0xb2102888bcf3c965 + .quad 0x27c576756c683a5a + .quad 0x3a7758a4c86cb447 + .quad 0x548111f693ae5076 + .quad 0x1dae21df1dfd54a6 + .quad 0x12248c90f3115e65 + .quad 0x5d9fd15f8de7f494 + + // 2^32 * 8 * G + + .quad 0x031408d36d63727f + .quad 0x6a379aefd7c7b533 + .quad 0xa9e18fc5ccaee24b + .quad 0x332f35914f8fbed3 + .quad 0x3f244d2aeed7521e + .quad 0x8e3a9028432e9615 + .quad 0xe164ba772e9c16d4 + .quad 0x3bc187fa47eb98d8 + .quad 0x6d470115ea86c20c + .quad 0x998ab7cb6c46d125 + .quad 0xd77832b53a660188 + .quad 0x450d81ce906fba03 + + // 2^36 * 1 * G + + .quad 0xf8ae4d2ad8453902 + .quad 0x7018058ee8db2d1d + .quad 0xaab3995fc7d2c11e + .quad 0x53b16d2324ccca79 + .quad 0x23264d66b2cae0b5 + .quad 0x7dbaed33ebca6576 + .quad 0x030ebed6f0d24ac8 + .quad 0x2a887f78f7635510 + .quad 0x2a23b9e75c012d4f + .quad 0x0c974651cae1f2ea + .quad 0x2fb63273675d70ca + .quad 0x0ba7250b864403f5 + + // 2^36 * 2 * G + + .quad 0xbb0d18fd029c6421 + .quad 0xbc2d142189298f02 + .quad 0x8347f8e68b250e96 + .quad 0x7b9f2fe8032d71c9 + .quad 0xdd63589386f86d9c + .quad 0x61699176e13a85a4 + .quad 0x2e5111954eaa7d57 + .quad 0x32c21b57fb60bdfb + .quad 0xd87823cd319e0780 + .quad 0xefc4cfc1897775c5 + .quad 0x4854fb129a0ab3f7 + .quad 0x12c49d417238c371 + + // 2^36 * 3 * G + + .quad 0x0950b533ffe83769 + .quad 0x21861c1d8e1d6bd1 + .quad 0xf022d8381302e510 + .quad 0x2509200c6391cab4 + .quad 0x09b3a01783799542 + .quad 0x626dd08faad5ee3f + .quad 0xba00bceeeb70149f + .quad 0x1421b246a0a444c9 + .quad 0x4aa43a8e8c24a7c7 + .quad 0x04c1f540d8f05ef5 + .quad 0xadba5e0c0b3eb9dc + .quad 0x2ab5504448a49ce3 + + // 2^36 * 4 * G + + .quad 0x2ed227266f0f5dec + .quad 0x9824ee415ed50824 + .quad 0x807bec7c9468d415 + .quad 0x7093bae1b521e23f + .quad 0xdc07ac631c5d3afa + .quad 0x58615171f9df8c6c + .quad 0x72a079d89d73e2b0 + .quad 0x7301f4ceb4eae15d + .quad 0x6409e759d6722c41 + .quad 0xa674e1cf72bf729b + .quad 0xbc0a24eb3c21e569 + .quad 0x390167d24ebacb23 + + // 2^36 * 5 * G + + .quad 0x27f58e3bba353f1c + .quad 0x4c47764dbf6a4361 + .quad 0xafbbc4e56e562650 + .quad 0x07db2ee6aae1a45d + .quad 0xd7bb054ba2f2120b + .quad 0xe2b9ceaeb10589b7 + .quad 0x3fe8bac8f3c0edbe + .quad 0x4cbd40767112cb69 + .quad 0x0b603cc029c58176 + .quad 0x5988e3825cb15d61 + .quad 0x2bb61413dcf0ad8d + .quad 0x7b8eec6c74183287 + + // 2^36 * 6 * G + + .quad 0xe4ca40782cd27cb0 + .quad 0xdaf9c323fbe967bd + .quad 0xb29bd34a8ad41e9e + .quad 0x72810497626ede4d + .quad 0x32fee570fc386b73 + .quad 0xda8b0141da3a8cc7 + .quad 0x975ffd0ac8968359 + .quad 0x6ee809a1b132a855 + .quad 0x9444bb31fcfd863a + .quad 0x2fe3690a3e4e48c5 + .quad 0xdc29c867d088fa25 + .quad 0x13bd1e38d173292e + + // 2^36 * 7 * G + + .quad 0xd32b4cd8696149b5 + .quad 0xe55937d781d8aab7 + .quad 0x0bcb2127ae122b94 + .quad 0x41e86fcfb14099b0 + .quad 0x223fb5cf1dfac521 + .quad 0x325c25316f554450 + .quad 0x030b98d7659177ac + .quad 0x1ed018b64f88a4bd + .quad 0x3630dfa1b802a6b0 + .quad 0x880f874742ad3bd5 + .quad 0x0af90d6ceec5a4d4 + .quad 0x746a247a37cdc5d9 + + // 2^36 * 8 * G + + .quad 0xd531b8bd2b7b9af6 + .quad 0x5005093537fc5b51 + .quad 0x232fcf25c593546d + .quad 0x20a365142bb40f49 + .quad 0x6eccd85278d941ed + .quad 0x2254ae83d22f7843 + .quad 0xc522d02e7bbfcdb7 + .quad 0x681e3351bff0e4e2 + .quad 0x8b64b59d83034f45 + .quad 0x2f8b71f21fa20efb + .quad 0x69249495ba6550e4 + .quad 0x539ef98e45d5472b + + // 2^40 * 1 * G + + .quad 0x6e7bb6a1a6205275 + .quad 0xaa4f21d7413c8e83 + .quad 0x6f56d155e88f5cb2 + .quad 0x2de25d4ba6345be1 + .quad 0xd074d8961cae743f + .quad 0xf86d18f5ee1c63ed + .quad 0x97bdc55be7f4ed29 + .quad 0x4cbad279663ab108 + .quad 0x80d19024a0d71fcd + .quad 0xc525c20afb288af8 + .quad 0xb1a3974b5f3a6419 + .quad 0x7d7fbcefe2007233 + + // 2^40 * 2 * G + + .quad 0xfaef1e6a266b2801 + .quad 0x866c68c4d5739f16 + .quad 0xf68a2fbc1b03762c + .quad 0x5975435e87b75a8d + .quad 0xcd7c5dc5f3c29094 + .quad 0xc781a29a2a9105ab + .quad 0x80c61d36421c3058 + .quad 0x4f9cd196dcd8d4d7 + .quad 0x199297d86a7b3768 + .quad 0xd0d058241ad17a63 + .quad 0xba029cad5c1c0c17 + .quad 0x7ccdd084387a0307 + + // 2^40 * 3 * G + + .quad 0xdca6422c6d260417 + .quad 0xae153d50948240bd + .quad 0xa9c0c1b4fb68c677 + .quad 0x428bd0ed61d0cf53 + .quad 0x9b0c84186760cc93 + .quad 0xcdae007a1ab32a99 + .quad 0xa88dec86620bda18 + .quad 0x3593ca848190ca44 + .quad 0x9213189a5e849aa7 + .quad 0xd4d8c33565d8facd + .quad 0x8c52545b53fdbbd1 + .quad 0x27398308da2d63e6 + + // 2^40 * 4 * G + + .quad 0x42c38d28435ed413 + .quad 0xbd50f3603278ccc9 + .quad 0xbb07ab1a79da03ef + .quad 0x269597aebe8c3355 + .quad 0xb9a10e4c0a702453 + .quad 0x0fa25866d57d1bde + .quad 0xffb9d9b5cd27daf7 + .quad 0x572c2945492c33fd + .quad 0xc77fc745d6cd30be + .quad 0xe4dfe8d3e3baaefb + .quad 0xa22c8830aa5dda0c + .quad 0x7f985498c05bca80 + + // 2^40 * 5 * G + + .quad 0x3849ce889f0be117 + .quad 0x8005ad1b7b54a288 + .quad 0x3da3c39f23fc921c + .quad 0x76c2ec470a31f304 + .quad 0xd35615520fbf6363 + .quad 0x08045a45cf4dfba6 + .quad 0xeec24fbc873fa0c2 + .quad 0x30f2653cd69b12e7 + .quad 0x8a08c938aac10c85 + .quad 0x46179b60db276bcb + .quad 0xa920c01e0e6fac70 + .quad 0x2f1273f1596473da + + // 2^40 * 6 * G + + .quad 0x4739fc7c8ae01e11 + .quad 0xfd5274904a6aab9f + .quad 0x41d98a8287728f2e + .quad 0x5d9e572ad85b69f2 + .quad 0x30488bd755a70bc0 + .quad 0x06d6b5a4f1d442e7 + .quad 0xead1a69ebc596162 + .quad 0x38ac1997edc5f784 + .quad 0x0666b517a751b13b + .quad 0x747d06867e9b858c + .quad 0xacacc011454dde49 + .quad 0x22dfcd9cbfe9e69c + + // 2^40 * 7 * G + + .quad 0x8ddbd2e0c30d0cd9 + .quad 0xad8e665facbb4333 + .quad 0x8f6b258c322a961f + .quad 0x6b2916c05448c1c7 + .quad 0x56ec59b4103be0a1 + .quad 0x2ee3baecd259f969 + .quad 0x797cb29413f5cd32 + .quad 0x0fe9877824cde472 + .quad 0x7edb34d10aba913b + .quad 0x4ea3cd822e6dac0e + .quad 0x66083dff6578f815 + .quad 0x4c303f307ff00a17 + + // 2^40 * 8 * G + + .quad 0xd30a3bd617b28c85 + .quad 0xc5d377b739773bea + .quad 0xc6c6e78c1e6a5cbf + .quad 0x0d61b8f78b2ab7c4 + .quad 0x29fc03580dd94500 + .quad 0xecd27aa46fbbec93 + .quad 0x130a155fc2e2a7f8 + .quad 0x416b151ab706a1d5 + .quad 0x56a8d7efe9c136b0 + .quad 0xbd07e5cd58e44b20 + .quad 0xafe62fda1b57e0ab + .quad 0x191a2af74277e8d2 + + // 2^44 * 1 * G + + .quad 0xd550095bab6f4985 + .quad 0x04f4cd5b4fbfaf1a + .quad 0x9d8e2ed12a0c7540 + .quad 0x2bc24e04b2212286 + .quad 0x09d4b60b2fe09a14 + .quad 0xc384f0afdbb1747e + .quad 0x58e2ea8978b5fd6e + .quad 0x519ef577b5e09b0a + .quad 0x1863d7d91124cca9 + .quad 0x7ac08145b88a708e + .quad 0x2bcd7309857031f5 + .quad 0x62337a6e8ab8fae5 + + // 2^44 * 2 * G + + .quad 0x4bcef17f06ffca16 + .quad 0xde06e1db692ae16a + .quad 0x0753702d614f42b0 + .quad 0x5f6041b45b9212d0 + .quad 0xd1ab324e1b3a1273 + .quad 0x18947cf181055340 + .quad 0x3b5d9567a98c196e + .quad 0x7fa00425802e1e68 + .quad 0x7d531574028c2705 + .quad 0x80317d69db0d75fe + .quad 0x30fface8ef8c8ddd + .quad 0x7e9de97bb6c3e998 + + // 2^44 * 3 * G + + .quad 0x1558967b9e6585a3 + .quad 0x97c99ce098e98b92 + .quad 0x10af149b6eb3adad + .quad 0x42181fe8f4d38cfa + .quad 0xf004be62a24d40dd + .quad 0xba0659910452d41f + .quad 0x81c45ee162a44234 + .quad 0x4cb829d8a22266ef + .quad 0x1dbcaa8407b86681 + .quad 0x081f001e8b26753b + .quad 0x3cd7ce6a84048e81 + .quad 0x78af11633f25f22c + + // 2^44 * 4 * G + + .quad 0x8416ebd40b50babc + .quad 0x1508722628208bee + .quad 0xa3148fafb9c1c36d + .quad 0x0d07daacd32d7d5d + .quad 0x3241c00e7d65318c + .quad 0xe6bee5dcd0e86de7 + .quad 0x118b2dc2fbc08c26 + .quad 0x680d04a7fc603dc3 + .quad 0xf9c2414a695aa3eb + .quad 0xdaa42c4c05a68f21 + .quad 0x7c6c23987f93963e + .quad 0x210e8cd30c3954e3 + + // 2^44 * 5 * G + + .quad 0xac4201f210a71c06 + .quad 0x6a65e0aef3bfb021 + .quad 0xbc42c35c393632f7 + .quad 0x56ea8db1865f0742 + .quad 0x2b50f16137fe6c26 + .quad 0xe102bcd856e404d8 + .quad 0x12b0f1414c561f6b + .quad 0x51b17bc8d028ec91 + .quad 0xfff5fb4bcf535119 + .quad 0xf4989d79df1108a0 + .quad 0xbdfcea659a3ba325 + .quad 0x18a11f1174d1a6f2 + + // 2^44 * 6 * G + + .quad 0x407375ab3f6bba29 + .quad 0x9ec3b6d8991e482e + .quad 0x99c80e82e55f92e9 + .quad 0x307c13b6fb0c0ae1 + .quad 0xfbd63cdad27a5f2c + .quad 0xf00fc4bc8aa106d7 + .quad 0x53fb5c1a8e64a430 + .quad 0x04eaabe50c1a2e85 + .quad 0x24751021cb8ab5e7 + .quad 0xfc2344495c5010eb + .quad 0x5f1e717b4e5610a1 + .quad 0x44da5f18c2710cd5 + + // 2^44 * 7 * G + + .quad 0x033cc55ff1b82eb5 + .quad 0xb15ae36d411cae52 + .quad 0xba40b6198ffbacd3 + .quad 0x768edce1532e861f + .quad 0x9156fe6b89d8eacc + .quad 0xe6b79451e23126a1 + .quad 0xbd7463d93944eb4e + .quad 0x726373f6767203ae + .quad 0xe305ca72eb7ef68a + .quad 0x662cf31f70eadb23 + .quad 0x18f026fdb4c45b68 + .quad 0x513b5384b5d2ecbd + + // 2^44 * 8 * G + + .quad 0x46d46280c729989e + .quad 0x4b93fbd05368a5dd + .quad 0x63df3f81d1765a89 + .quad 0x34cebd64b9a0a223 + .quad 0x5e2702878af34ceb + .quad 0x900b0409b946d6ae + .quad 0x6512ebf7dabd8512 + .quad 0x61d9b76988258f81 + .quad 0xa6c5a71349b7d94b + .quad 0xa3f3d15823eb9446 + .quad 0x0416fbd277484834 + .quad 0x69d45e6f2c70812f + + // 2^48 * 1 * G + + .quad 0xce16f74bc53c1431 + .quad 0x2b9725ce2072edde + .quad 0xb8b9c36fb5b23ee7 + .quad 0x7e2e0e450b5cc908 + .quad 0x9fe62b434f460efb + .quad 0xded303d4a63607d6 + .quad 0xf052210eb7a0da24 + .quad 0x237e7dbe00545b93 + .quad 0x013575ed6701b430 + .quad 0x231094e69f0bfd10 + .quad 0x75320f1583e47f22 + .quad 0x71afa699b11155e3 + + // 2^48 * 2 * G + + .quad 0x65ce6f9b3953b61d + .quad 0xc65839eaafa141e6 + .quad 0x0f435ffda9f759fe + .quad 0x021142e9c2b1c28e + .quad 0xea423c1c473b50d6 + .quad 0x51e87a1f3b38ef10 + .quad 0x9b84bf5fb2c9be95 + .quad 0x00731fbc78f89a1c + .quad 0xe430c71848f81880 + .quad 0xbf960c225ecec119 + .quad 0xb6dae0836bba15e3 + .quad 0x4c4d6f3347e15808 + + // 2^48 * 3 * G + + .quad 0x18f7eccfc17d1fc9 + .quad 0x6c75f5a651403c14 + .quad 0xdbde712bf7ee0cdf + .quad 0x193fddaaa7e47a22 + .quad 0x2f0cddfc988f1970 + .quad 0x6b916227b0b9f51b + .quad 0x6ec7b6c4779176be + .quad 0x38bf9500a88f9fa8 + .quad 0x1fd2c93c37e8876f + .quad 0xa2f61e5a18d1462c + .quad 0x5080f58239241276 + .quad 0x6a6fb99ebf0d4969 + + // 2^48 * 4 * G + + .quad 0x6a46c1bb560855eb + .quad 0x2416bb38f893f09d + .quad 0xd71d11378f71acc1 + .quad 0x75f76914a31896ea + .quad 0xeeb122b5b6e423c6 + .quad 0x939d7010f286ff8e + .quad 0x90a92a831dcf5d8c + .quad 0x136fda9f42c5eb10 + .quad 0xf94cdfb1a305bdd1 + .quad 0x0f364b9d9ff82c08 + .quad 0x2a87d8a5c3bb588a + .quad 0x022183510be8dcba + + // 2^48 * 5 * G + + .quad 0x4af766385ead2d14 + .quad 0xa08ed880ca7c5830 + .quad 0x0d13a6e610211e3d + .quad 0x6a071ce17b806c03 + .quad 0x9d5a710143307a7f + .quad 0xb063de9ec47da45f + .quad 0x22bbfe52be927ad3 + .quad 0x1387c441fd40426c + .quad 0xb5d3c3d187978af8 + .quad 0x722b5a3d7f0e4413 + .quad 0x0d7b4848bb477ca0 + .quad 0x3171b26aaf1edc92 + + // 2^48 * 6 * G + + .quad 0xa92f319097564ca8 + .quad 0xff7bb84c2275e119 + .quad 0x4f55fe37a4875150 + .quad 0x221fd4873cf0835a + .quad 0xa60db7d8b28a47d1 + .quad 0xa6bf14d61770a4f1 + .quad 0xd4a1f89353ddbd58 + .quad 0x6c514a63344243e9 + .quad 0x2322204f3a156341 + .quad 0xfb73e0e9ba0a032d + .quad 0xfce0dd4c410f030e + .quad 0x48daa596fb924aaa + + // 2^48 * 7 * G + + .quad 0x6eca8e665ca59cc7 + .quad 0xa847254b2e38aca0 + .quad 0x31afc708d21e17ce + .quad 0x676dd6fccad84af7 + .quad 0x14f61d5dc84c9793 + .quad 0x9941f9e3ef418206 + .quad 0xcdf5b88f346277ac + .quad 0x58c837fa0e8a79a9 + .quad 0x0cf9688596fc9058 + .quad 0x1ddcbbf37b56a01b + .quad 0xdcc2e77d4935d66a + .quad 0x1c4f73f2c6a57f0a + + // 2^48 * 8 * G + + .quad 0x0e7a4fbd305fa0bb + .quad 0x829d4ce054c663ad + .quad 0xf421c3832fe33848 + .quad 0x795ac80d1bf64c42 + .quad 0xb36e706efc7c3484 + .quad 0x73dfc9b4c3c1cf61 + .quad 0xeb1d79c9781cc7e5 + .quad 0x70459adb7daf675c + .quad 0x1b91db4991b42bb3 + .quad 0x572696234b02dcca + .quad 0x9fdf9ee51f8c78dc + .quad 0x5fe162848ce21fd3 + + // 2^52 * 1 * G + + .quad 0xe2790aae4d077c41 + .quad 0x8b938270db7469a3 + .quad 0x6eb632dc8abd16a2 + .quad 0x720814ecaa064b72 + .quad 0x315c29c795115389 + .quad 0xd7e0e507862f74ce + .quad 0x0c4a762185927432 + .quad 0x72de6c984a25a1e4 + .quad 0xae9ab553bf6aa310 + .quad 0x050a50a9806d6e1b + .quad 0x92bb7403adff5139 + .quad 0x0394d27645be618b + + // 2^52 * 2 * G + + .quad 0x4d572251857eedf4 + .quad 0xe3724edde19e93c5 + .quad 0x8a71420e0b797035 + .quad 0x3b3c833687abe743 + .quad 0xf5396425b23545a4 + .quad 0x15a7a27e98fbb296 + .quad 0xab6c52bc636fdd86 + .quad 0x79d995a8419334ee + .quad 0xcd8a8ea61195dd75 + .quad 0xa504d8a81dd9a82f + .quad 0x540dca81a35879b6 + .quad 0x60dd16a379c86a8a + + // 2^52 * 3 * G + + .quad 0x35a2c8487381e559 + .quad 0x596ffea6d78082cb + .quad 0xcb9771ebdba7b653 + .quad 0x5a08b5019b4da685 + .quad 0x3501d6f8153e47b8 + .quad 0xb7a9675414a2f60c + .quad 0x112ee8b6455d9523 + .quad 0x4e62a3c18112ea8a + .quad 0xc8d4ac04516ab786 + .quad 0x595af3215295b23d + .quad 0xd6edd234db0230c1 + .quad 0x0929efe8825b41cc + + // 2^52 * 4 * G + + .quad 0x5f0601d1cbd0f2d3 + .quad 0x736e412f6132bb7f + .quad 0x83604432238dde87 + .quad 0x1e3a5272f5c0753c + .quad 0x8b3172b7ad56651d + .quad 0x01581b7a3fabd717 + .quad 0x2dc94df6424df6e4 + .quad 0x30376e5d2c29284f + .quad 0xd2918da78159a59c + .quad 0x6bdc1cd93f0713f3 + .quad 0x565f7a934acd6590 + .quad 0x53daacec4cb4c128 + + // 2^52 * 5 * G + + .quad 0x4ca73bd79cc8a7d6 + .quad 0x4d4a738f47e9a9b2 + .quad 0xf4cbf12942f5fe00 + .quad 0x01a13ff9bdbf0752 + .quad 0x99852bc3852cfdb0 + .quad 0x2cc12e9559d6ed0b + .quad 0x70f9e2bf9b5ac27b + .quad 0x4f3b8c117959ae99 + .quad 0x55b6c9c82ff26412 + .quad 0x1ac4a8c91fb667a8 + .quad 0xd527bfcfeb778bf2 + .quad 0x303337da7012a3be + + // 2^52 * 6 * G + + .quad 0x955422228c1c9d7c + .quad 0x01fac1371a9b340f + .quad 0x7e8d9177925b48d7 + .quad 0x53f8ad5661b3e31b + .quad 0x976d3ccbfad2fdd1 + .quad 0xcb88839737a640a8 + .quad 0x2ff00c1d6734cb25 + .quad 0x269ff4dc789c2d2b + .quad 0x0c003fbdc08d678d + .quad 0x4d982fa37ead2b17 + .quad 0xc07e6bcdb2e582f1 + .quad 0x296c7291df412a44 + + // 2^52 * 7 * G + + .quad 0x7903de2b33daf397 + .quad 0xd0ff0619c9a624b3 + .quad 0x8a1d252b555b3e18 + .quad 0x2b6d581c52e0b7c0 + .quad 0xdfb23205dab8b59e + .quad 0x465aeaa0c8092250 + .quad 0xd133c1189a725d18 + .quad 0x2327370261f117d1 + .quad 0x3d0543d3623e7986 + .quad 0x679414c2c278a354 + .quad 0xae43f0cc726196f6 + .quad 0x7836c41f8245eaba + + // 2^52 * 8 * G + + .quad 0xe7a254db49e95a81 + .quad 0x5192d5d008b0ad73 + .quad 0x4d20e5b1d00afc07 + .quad 0x5d55f8012cf25f38 + .quad 0xca651e848011937c + .quad 0xc6b0c46e6ef41a28 + .quad 0xb7021ba75f3f8d52 + .quad 0x119dff99ead7b9fd + .quad 0x43eadfcbf4b31d4d + .quad 0xc6503f7411148892 + .quad 0xfeee68c5060d3b17 + .quad 0x329293b3dd4a0ac8 + + // 2^56 * 1 * G + + .quad 0x4e59214fe194961a + .quad 0x49be7dc70d71cd4f + .quad 0x9300cfd23b50f22d + .quad 0x4789d446fc917232 + .quad 0x2879852d5d7cb208 + .quad 0xb8dedd70687df2e7 + .quad 0xdc0bffab21687891 + .quad 0x2b44c043677daa35 + .quad 0x1a1c87ab074eb78e + .quad 0xfac6d18e99daf467 + .quad 0x3eacbbcd484f9067 + .quad 0x60c52eef2bb9a4e4 + + // 2^56 * 2 * G + + .quad 0x0b5d89bc3bfd8bf1 + .quad 0xb06b9237c9f3551a + .quad 0x0e4c16b0d53028f5 + .quad 0x10bc9c312ccfcaab + .quad 0x702bc5c27cae6d11 + .quad 0x44c7699b54a48cab + .quad 0xefbc4056ba492eb2 + .quad 0x70d77248d9b6676d + .quad 0xaa8ae84b3ec2a05b + .quad 0x98699ef4ed1781e0 + .quad 0x794513e4708e85d1 + .quad 0x63755bd3a976f413 + + // 2^56 * 3 * G + + .quad 0xb55fa03e2ad10853 + .quad 0x356f75909ee63569 + .quad 0x9ff9f1fdbe69b890 + .quad 0x0d8cc1c48bc16f84 + .quad 0x3dc7101897f1acb7 + .quad 0x5dda7d5ec165bbd8 + .quad 0x508e5b9c0fa1020f + .quad 0x2763751737c52a56 + .quad 0x029402d36eb419a9 + .quad 0xf0b44e7e77b460a5 + .quad 0xcfa86230d43c4956 + .quad 0x70c2dd8a7ad166e7 + + // 2^56 * 4 * G + + .quad 0x656194509f6fec0e + .quad 0xee2e7ea946c6518d + .quad 0x9733c1f367e09b5c + .quad 0x2e0fac6363948495 + .quad 0x91d4967db8ed7e13 + .quad 0x74252f0ad776817a + .quad 0xe40982e00d852564 + .quad 0x32b8613816a53ce5 + .quad 0x79e7f7bee448cd64 + .quad 0x6ac83a67087886d0 + .quad 0xf89fd4d9a0e4db2e + .quad 0x4179215c735a4f41 + + // 2^56 * 5 * G + + .quad 0x8c7094e7d7dced2a + .quad 0x97fb8ac347d39c70 + .quad 0xe13be033a906d902 + .quad 0x700344a30cd99d76 + .quad 0xe4ae33b9286bcd34 + .quad 0xb7ef7eb6559dd6dc + .quad 0x278b141fb3d38e1f + .quad 0x31fa85662241c286 + .quad 0xaf826c422e3622f4 + .quad 0xc12029879833502d + .quad 0x9bc1b7e12b389123 + .quad 0x24bb2312a9952489 + + // 2^56 * 6 * G + + .quad 0xb1a8ed1732de67c3 + .quad 0x3cb49418461b4948 + .quad 0x8ebd434376cfbcd2 + .quad 0x0fee3e871e188008 + .quad 0x41f80c2af5f85c6b + .quad 0x687284c304fa6794 + .quad 0x8945df99a3ba1bad + .quad 0x0d1d2af9ffeb5d16 + .quad 0xa9da8aa132621edf + .quad 0x30b822a159226579 + .quad 0x4004197ba79ac193 + .quad 0x16acd79718531d76 + + // 2^56 * 7 * G + + .quad 0x72df72af2d9b1d3d + .quad 0x63462a36a432245a + .quad 0x3ecea07916b39637 + .quad 0x123e0ef6b9302309 + .quad 0xc959c6c57887b6ad + .quad 0x94e19ead5f90feba + .quad 0x16e24e62a342f504 + .quad 0x164ed34b18161700 + .quad 0x487ed94c192fe69a + .quad 0x61ae2cea3a911513 + .quad 0x877bf6d3b9a4de27 + .quad 0x78da0fc61073f3eb + + // 2^56 * 8 * G + + .quad 0x5bf15d28e52bc66a + .quad 0x2c47e31870f01a8e + .quad 0x2419afbc06c28bdd + .quad 0x2d25deeb256b173a + .quad 0xa29f80f1680c3a94 + .quad 0x71f77e151ae9e7e6 + .quad 0x1100f15848017973 + .quad 0x054aa4b316b38ddd + .quad 0xdfc8468d19267cb8 + .quad 0x0b28789c66e54daf + .quad 0x2aeb1d2a666eec17 + .quad 0x134610a6ab7da760 + + // 2^60 * 1 * G + + .quad 0xcaf55ec27c59b23f + .quad 0x99aeed3e154d04f2 + .quad 0x68441d72e14141f4 + .quad 0x140345133932a0a2 + .quad 0xd91430e0dc028c3c + .quad 0x0eb955a85217c771 + .quad 0x4b09e1ed2c99a1fa + .quad 0x42881af2bd6a743c + .quad 0x7bfec69aab5cad3d + .quad 0xc23e8cd34cb2cfad + .quad 0x685dd14bfb37d6a2 + .quad 0x0ad6d64415677a18 + + // 2^60 * 2 * G + + .quad 0x781a439e417becb5 + .quad 0x4ac5938cd10e0266 + .quad 0x5da385110692ac24 + .quad 0x11b065a2ade31233 + .quad 0x7914892847927e9f + .quad 0x33dad6ef370aa877 + .quad 0x1f8f24fa11122703 + .quad 0x5265ac2f2adf9592 + .quad 0x405fdd309afcb346 + .quad 0xd9723d4428e63f54 + .quad 0x94c01df05f65aaae + .quad 0x43e4dc3ae14c0809 + + // 2^60 * 3 * G + + .quad 0xbc12c7f1a938a517 + .quad 0x473028ab3180b2e1 + .quad 0x3f78571efbcd254a + .quad 0x74e534426ff6f90f + .quad 0xea6f7ac3adc2c6a3 + .quad 0xd0e928f6e9717c94 + .quad 0xe2d379ead645eaf5 + .quad 0x46dd8785c51ffbbe + .quad 0x709801be375c8898 + .quad 0x4b06dab5e3fd8348 + .quad 0x75880ced27230714 + .quad 0x2b09468fdd2f4c42 + + // 2^60 * 4 * G + + .quad 0x97c749eeb701cb96 + .quad 0x83f438d4b6a369c3 + .quad 0x62962b8b9a402cd9 + .quad 0x6976c7509888df7b + .quad 0x5b97946582ffa02a + .quad 0xda096a51fea8f549 + .quad 0xa06351375f77af9b + .quad 0x1bcfde61201d1e76 + .quad 0x4a4a5490246a59a2 + .quad 0xd63ebddee87fdd90 + .quad 0xd9437c670d2371fa + .quad 0x69e87308d30f8ed6 + + // 2^60 * 5 * G + + .quad 0x435a8bb15656beb0 + .quad 0xf8fac9ba4f4d5bca + .quad 0xb9b278c41548c075 + .quad 0x3eb0ef76e892b622 + .quad 0x0f80bf028bc80303 + .quad 0x6aae16b37a18cefb + .quad 0xdd47ea47d72cd6a3 + .quad 0x61943588f4ed39aa + .quad 0xd26e5c3e91039f85 + .quad 0xc0e9e77df6f33aa9 + .quad 0xe8968c5570066a93 + .quad 0x3c34d1881faaaddd + + // 2^60 * 6 * G + + .quad 0x3f9d2b5ea09f9ec0 + .quad 0x1dab3b6fb623a890 + .quad 0xa09ba3ea72d926c4 + .quad 0x374193513fd8b36d + .quad 0xbd5b0b8f2fffe0d9 + .quad 0x6aa254103ed24fb9 + .quad 0x2ac7d7bcb26821c4 + .quad 0x605b394b60dca36a + .quad 0xb4e856e45a9d1ed2 + .quad 0xefe848766c97a9a2 + .quad 0xb104cf641e5eee7d + .quad 0x2f50b81c88a71c8f + + // 2^60 * 7 * G + + .quad 0x31723c61fc6811bb + .quad 0x9cb450486211800f + .quad 0x768933d347995753 + .quad 0x3491a53502752fcd + .quad 0x2b552ca0a7da522a + .quad 0x3230b336449b0250 + .quad 0xf2c4c5bca4b99fb9 + .quad 0x7b2c674958074a22 + .quad 0xd55165883ed28cdf + .quad 0x12d84fd2d362de39 + .quad 0x0a874ad3e3378e4f + .quad 0x000d2b1f7c763e74 + + // 2^60 * 8 * G + + .quad 0x3d420811d06d4a67 + .quad 0xbefc048590e0ffe3 + .quad 0xf870c6b7bd487bde + .quad 0x6e2a7316319afa28 + .quad 0x9624778c3e94a8ab + .quad 0x0ad6f3cee9a78bec + .quad 0x948ac7810d743c4f + .quad 0x76627935aaecfccc + .quad 0x56a8ac24d6d59a9f + .quad 0xc8db753e3096f006 + .quad 0x477f41e68f4c5299 + .quad 0x588d851cf6c86114 + + // 2^64 * 1 * G + + .quad 0x51138ec78df6b0fe + .quad 0x5397da89e575f51b + .quad 0x09207a1d717af1b9 + .quad 0x2102fdba2b20d650 + .quad 0xcd2a65e777d1f515 + .quad 0x548991878faa60f1 + .quad 0xb1b73bbcdabc06e5 + .quad 0x654878cba97cc9fb + .quad 0x969ee405055ce6a1 + .quad 0x36bca7681251ad29 + .quad 0x3a1af517aa7da415 + .quad 0x0ad725db29ecb2ba + + // 2^64 * 2 * G + + .quad 0xdc4267b1834e2457 + .quad 0xb67544b570ce1bc5 + .quad 0x1af07a0bf7d15ed7 + .quad 0x4aefcffb71a03650 + .quad 0xfec7bc0c9b056f85 + .quad 0x537d5268e7f5ffd7 + .quad 0x77afc6624312aefa + .quad 0x4f675f5302399fd9 + .quad 0xc32d36360415171e + .quad 0xcd2bef118998483b + .quad 0x870a6eadd0945110 + .quad 0x0bccbb72a2a86561 + + // 2^64 * 3 * G + + .quad 0x185e962feab1a9c8 + .quad 0x86e7e63565147dcd + .quad 0xb092e031bb5b6df2 + .quad 0x4024f0ab59d6b73e + .quad 0x186d5e4c50fe1296 + .quad 0xe0397b82fee89f7e + .quad 0x3bc7f6c5507031b0 + .quad 0x6678fd69108f37c2 + .quad 0x1586fa31636863c2 + .quad 0x07f68c48572d33f2 + .quad 0x4f73cc9f789eaefc + .quad 0x2d42e2108ead4701 + + // 2^64 * 4 * G + + .quad 0x97f5131594dfd29b + .quad 0x6155985d313f4c6a + .quad 0xeba13f0708455010 + .quad 0x676b2608b8d2d322 + .quad 0x21717b0d0f537593 + .quad 0x914e690b131e064c + .quad 0x1bb687ae752ae09f + .quad 0x420bf3a79b423c6e + .quad 0x8138ba651c5b2b47 + .quad 0x8671b6ec311b1b80 + .quad 0x7bff0cb1bc3135b0 + .quad 0x745d2ffa9c0cf1e0 + + // 2^64 * 5 * G + + .quad 0xbf525a1e2bc9c8bd + .quad 0xea5b260826479d81 + .quad 0xd511c70edf0155db + .quad 0x1ae23ceb960cf5d0 + .quad 0x6036df5721d34e6a + .quad 0xb1db8827997bb3d0 + .quad 0xd3c209c3c8756afa + .quad 0x06e15be54c1dc839 + .quad 0x5b725d871932994a + .quad 0x32351cb5ceb1dab0 + .quad 0x7dc41549dab7ca05 + .quad 0x58ded861278ec1f7 + + // 2^64 * 6 * G + + .quad 0xd8173793f266c55c + .quad 0xc8c976c5cc454e49 + .quad 0x5ce382f8bc26c3a8 + .quad 0x2ff39de85485f6f9 + .quad 0x2dfb5ba8b6c2c9a8 + .quad 0x48eeef8ef52c598c + .quad 0x33809107f12d1573 + .quad 0x08ba696b531d5bd8 + .quad 0x77ed3eeec3efc57a + .quad 0x04e05517d4ff4811 + .quad 0xea3d7a3ff1a671cb + .quad 0x120633b4947cfe54 + + // 2^64 * 7 * G + + .quad 0x0b94987891610042 + .quad 0x4ee7b13cecebfae8 + .quad 0x70be739594f0a4c0 + .quad 0x35d30a99b4d59185 + .quad 0x82bd31474912100a + .quad 0xde237b6d7e6fbe06 + .quad 0xe11e761911ea79c6 + .quad 0x07433be3cb393bde + .quad 0xff7944c05ce997f4 + .quad 0x575d3de4b05c51a3 + .quad 0x583381fd5a76847c + .quad 0x2d873ede7af6da9f + + // 2^64 * 8 * G + + .quad 0x157a316443373409 + .quad 0xfab8b7eef4aa81d9 + .quad 0xb093fee6f5a64806 + .quad 0x2e773654707fa7b6 + .quad 0xaa6202e14e5df981 + .quad 0xa20d59175015e1f5 + .quad 0x18a275d3bae21d6c + .quad 0x0543618a01600253 + .quad 0x0deabdf4974c23c1 + .quad 0xaa6f0a259dce4693 + .quad 0x04202cb8a29aba2c + .quad 0x4b1443362d07960d + + // 2^68 * 1 * G + + .quad 0x47b837f753242cec + .quad 0x256dc48cc04212f2 + .quad 0xe222fbfbe1d928c5 + .quad 0x48ea295bad8a2c07 + .quad 0x299b1c3f57c5715e + .quad 0x96cb929e6b686d90 + .quad 0x3004806447235ab3 + .quad 0x2c435c24a44d9fe1 + .quad 0x0607c97c80f8833f + .quad 0x0e851578ca25ec5b + .quad 0x54f7450b161ebb6f + .quad 0x7bcb4792a0def80e + + // 2^68 * 2 * G + + .quad 0x8487e3d02bc73659 + .quad 0x4baf8445059979df + .quad 0xd17c975adcad6fbf + .quad 0x57369f0bdefc96b6 + .quad 0x1cecd0a0045224c2 + .quad 0x757f1b1b69e53952 + .quad 0x775b7a925289f681 + .quad 0x1b6cc62016736148 + .quad 0xf1a9990175638698 + .quad 0x353dd1beeeaa60d3 + .quad 0x849471334c9ba488 + .quad 0x63fa6e6843ade311 + + // 2^68 * 3 * G + + .quad 0xd15c20536597c168 + .quad 0x9f73740098d28789 + .quad 0x18aee7f13257ba1f + .quad 0x3418bfda07346f14 + .quad 0x2195becdd24b5eb7 + .quad 0x5e41f18cc0cd44f9 + .quad 0xdf28074441ca9ede + .quad 0x07073b98f35b7d67 + .quad 0xd03c676c4ce530d4 + .quad 0x0b64c0473b5df9f4 + .quad 0x065cef8b19b3a31e + .quad 0x3084d661533102c9 + + // 2^68 * 4 * G + + .quad 0xe1f6b79ebf8469ad + .quad 0x15801004e2663135 + .quad 0x9a498330af74181b + .quad 0x3ba2504f049b673c + .quad 0x9a6ce876760321fd + .quad 0x7fe2b5109eb63ad8 + .quad 0x00e7d4ae8ac80592 + .quad 0x73d86b7abb6f723a + .quad 0x0b52b5606dba5ab6 + .quad 0xa9134f0fbbb1edab + .quad 0x30a9520d9b04a635 + .quad 0x6813b8f37973e5db + + // 2^68 * 5 * G + + .quad 0x9854b054334127c1 + .quad 0x105d047882fbff25 + .quad 0xdb49f7f944186f4f + .quad 0x1768e838bed0b900 + .quad 0xf194ca56f3157e29 + .quad 0x136d35705ef528a5 + .quad 0xdd4cef778b0599bc + .quad 0x7d5472af24f833ed + .quad 0xd0ef874daf33da47 + .quad 0x00d3be5db6e339f9 + .quad 0x3f2a8a2f9c9ceece + .quad 0x5d1aeb792352435a + + // 2^68 * 6 * G + + .quad 0xf59e6bb319cd63ca + .quad 0x670c159221d06839 + .quad 0xb06d565b2150cab6 + .quad 0x20fb199d104f12a3 + .quad 0x12c7bfaeb61ba775 + .quad 0xb84e621fe263bffd + .quad 0x0b47a5c35c840dcf + .quad 0x7e83be0bccaf8634 + .quad 0x61943dee6d99c120 + .quad 0x86101f2e460b9fe0 + .quad 0x6bb2f1518ee8598d + .quad 0x76b76289fcc475cc + + // 2^68 * 7 * G + + .quad 0x791b4cc1756286fa + .quad 0xdbced317d74a157c + .quad 0x7e732421ea72bde6 + .quad 0x01fe18491131c8e9 + .quad 0x4245f1a1522ec0b3 + .quad 0x558785b22a75656d + .quad 0x1d485a2548a1b3c0 + .quad 0x60959eccd58fe09f + .quad 0x3ebfeb7ba8ed7a09 + .quad 0x49fdc2bbe502789c + .quad 0x44ebce5d3c119428 + .quad 0x35e1eb55be947f4a + + // 2^68 * 8 * G + + .quad 0xdbdae701c5738dd3 + .quad 0xf9c6f635b26f1bee + .quad 0x61e96a8042f15ef4 + .quad 0x3aa1d11faf60a4d8 + .quad 0x14fd6dfa726ccc74 + .quad 0x3b084cfe2f53b965 + .quad 0xf33ae4f552a2c8b4 + .quad 0x59aab07a0d40166a + .quad 0x77bcec4c925eac25 + .quad 0x1848718460137738 + .quad 0x5b374337fea9f451 + .quad 0x1865e78ec8e6aa46 + + // 2^72 * 1 * G + + .quad 0xccc4b7c7b66e1f7a + .quad 0x44157e25f50c2f7e + .quad 0x3ef06dfc713eaf1c + .quad 0x582f446752da63f7 + .quad 0x967c54e91c529ccb + .quad 0x30f6269264c635fb + .quad 0x2747aff478121965 + .quad 0x17038418eaf66f5c + .quad 0xc6317bd320324ce4 + .quad 0xa81042e8a4488bc4 + .quad 0xb21ef18b4e5a1364 + .quad 0x0c2a1c4bcda28dc9 + + // 2^72 * 2 * G + + .quad 0xd24dc7d06f1f0447 + .quad 0xb2269e3edb87c059 + .quad 0xd15b0272fbb2d28f + .quad 0x7c558bd1c6f64877 + .quad 0xedc4814869bd6945 + .quad 0x0d6d907dbe1c8d22 + .quad 0xc63bd212d55cc5ab + .quad 0x5a6a9b30a314dc83 + .quad 0xd0ec1524d396463d + .quad 0x12bb628ac35a24f0 + .quad 0xa50c3a791cbc5fa4 + .quad 0x0404a5ca0afbafc3 + + // 2^72 * 3 * G + + .quad 0x8c1f40070aa743d6 + .quad 0xccbad0cb5b265ee8 + .quad 0x574b046b668fd2de + .quad 0x46395bfdcadd9633 + .quad 0x62bc9e1b2a416fd1 + .quad 0xb5c6f728e350598b + .quad 0x04343fd83d5d6967 + .quad 0x39527516e7f8ee98 + .quad 0x117fdb2d1a5d9a9c + .quad 0x9c7745bcd1005c2a + .quad 0xefd4bef154d56fea + .quad 0x76579a29e822d016 + + // 2^72 * 4 * G + + .quad 0x45b68e7e49c02a17 + .quad 0x23cd51a2bca9a37f + .quad 0x3ed65f11ec224c1b + .quad 0x43a384dc9e05bdb1 + .quad 0x333cb51352b434f2 + .quad 0xd832284993de80e1 + .quad 0xb5512887750d35ce + .quad 0x02c514bb2a2777c1 + .quad 0x684bd5da8bf1b645 + .quad 0xfb8bd37ef6b54b53 + .quad 0x313916d7a9b0d253 + .quad 0x1160920961548059 + + // 2^72 * 5 * G + + .quad 0xb44d166929dacfaa + .quad 0xda529f4c8413598f + .quad 0xe9ef63ca453d5559 + .quad 0x351e125bc5698e0b + .quad 0x7a385616369b4dcd + .quad 0x75c02ca7655c3563 + .quad 0x7dc21bf9d4f18021 + .quad 0x2f637d7491e6e042 + .quad 0xd4b49b461af67bbe + .quad 0xd603037ac8ab8961 + .quad 0x71dee19ff9a699fb + .quad 0x7f182d06e7ce2a9a + + // 2^72 * 6 * G + + .quad 0x7a7c8e64ab0168ec + .quad 0xcb5a4a5515edc543 + .quad 0x095519d347cd0eda + .quad 0x67d4ac8c343e93b0 + .quad 0x09454b728e217522 + .quad 0xaa58e8f4d484b8d8 + .quad 0xd358254d7f46903c + .quad 0x44acc043241c5217 + .quad 0x1c7d6bbb4f7a5777 + .quad 0x8b35fed4918313e1 + .quad 0x4adca1c6c96b4684 + .quad 0x556d1c8312ad71bd + + // 2^72 * 7 * G + + .quad 0x17ef40e30c8d3982 + .quad 0x31f7073e15a3fa34 + .quad 0x4f21f3cb0773646e + .quad 0x746c6c6d1d824eff + .quad 0x81f06756b11be821 + .quad 0x0faff82310a3f3dd + .quad 0xf8b2d0556a99465d + .quad 0x097abe38cc8c7f05 + .quad 0x0c49c9877ea52da4 + .quad 0x4c4369559bdc1d43 + .quad 0x022c3809f7ccebd2 + .quad 0x577e14a34bee84bd + + // 2^72 * 8 * G + + .quad 0xf0e268ac61a73b0a + .quad 0xf2fafa103791a5f5 + .quad 0xc1e13e826b6d00e9 + .quad 0x60fa7ee96fd78f42 + .quad 0x94fecebebd4dd72b + .quad 0xf46a4fda060f2211 + .quad 0x124a5977c0c8d1ff + .quad 0x705304b8fb009295 + .quad 0xb63d1d354d296ec6 + .quad 0xf3c3053e5fad31d8 + .quad 0x670b958cb4bd42ec + .quad 0x21398e0ca16353fd + + // 2^76 * 1 * G + + .quad 0x216ab2ca8da7d2ef + .quad 0x366ad9dd99f42827 + .quad 0xae64b9004fdd3c75 + .quad 0x403a395b53909e62 + .quad 0x86c5fc16861b7e9a + .quad 0xf6a330476a27c451 + .quad 0x01667267a1e93597 + .quad 0x05ffb9cd6082dfeb + .quad 0xa617fa9ff53f6139 + .quad 0x60f2b5e513e66cb6 + .quad 0xd7a8beefb3448aa4 + .quad 0x7a2932856f5ea192 + + // 2^76 * 2 * G + + .quad 0x0b39d761b02de888 + .quad 0x5f550e7ed2414e1f + .quad 0xa6bfa45822e1a940 + .quad 0x050a2f7dfd447b99 + .quad 0xb89c444879639302 + .quad 0x4ae4f19350c67f2c + .quad 0xf0b35da8c81af9c6 + .quad 0x39d0003546871017 + .quad 0x437c3b33a650db77 + .quad 0x6bafe81dbac52bb2 + .quad 0xfe99402d2db7d318 + .quad 0x2b5b7eec372ba6ce + + // 2^76 * 3 * G + + .quad 0xb3bc4bbd83f50eef + .quad 0x508f0c998c927866 + .quad 0x43e76587c8b7e66e + .quad 0x0f7655a3a47f98d9 + .quad 0xa694404d613ac8f4 + .quad 0x500c3c2bfa97e72c + .quad 0x874104d21fcec210 + .quad 0x1b205fb38604a8ee + .quad 0x55ecad37d24b133c + .quad 0x441e147d6038c90b + .quad 0x656683a1d62c6fee + .quad 0x0157d5dc87e0ecae + + // 2^76 * 4 * G + + .quad 0xf2a7af510354c13d + .quad 0xd7a0b145aa372b60 + .quad 0x2869b96a05a3d470 + .quad 0x6528e42d82460173 + .quad 0x95265514d71eb524 + .quad 0xe603d8815df14593 + .quad 0x147cdf410d4de6b7 + .quad 0x5293b1730437c850 + .quad 0x23d0e0814bccf226 + .quad 0x92c745cd8196fb93 + .quad 0x8b61796c59541e5b + .quad 0x40a44df0c021f978 + + // 2^76 * 5 * G + + .quad 0xdaa869894f20ea6a + .quad 0xea14a3d14c620618 + .quad 0x6001fccb090bf8be + .quad 0x35f4e822947e9cf0 + .quad 0x86c96e514bc5d095 + .quad 0xf20d4098fca6804a + .quad 0x27363d89c826ea5d + .quad 0x39ca36565719cacf + .quad 0x97506f2f6f87b75c + .quad 0xc624aea0034ae070 + .quad 0x1ec856e3aad34dd6 + .quad 0x055b0be0e440e58f + + // 2^76 * 6 * G + + .quad 0x6469a17d89735d12 + .quad 0xdb6f27d5e662b9f1 + .quad 0x9fcba3286a395681 + .quad 0x363b8004d269af25 + .quad 0x4d12a04b6ea33da2 + .quad 0x57cf4c15e36126dd + .quad 0x90ec9675ee44d967 + .quad 0x64ca348d2a985aac + .quad 0x99588e19e4c4912d + .quad 0xefcc3b4e1ca5ce6b + .quad 0x4522ea60fa5b98d5 + .quad 0x7064bbab1de4a819 + + // 2^76 * 7 * G + + .quad 0xb919e1515a770641 + .quad 0xa9a2e2c74e7f8039 + .quad 0x7527250b3df23109 + .quad 0x756a7330ac27b78b + .quad 0xa290c06142542129 + .quad 0xf2e2c2aebe8d5b90 + .quad 0xcf2458db76abfe1b + .quad 0x02157ade83d626bf + .quad 0x3e46972a1b9a038b + .quad 0x2e4ee66a7ee03fb4 + .quad 0x81a248776edbb4ca + .quad 0x1a944ee88ecd0563 + + // 2^76 * 8 * G + + .quad 0xd5a91d1151039372 + .quad 0x2ed377b799ca26de + .quad 0xa17202acfd366b6b + .quad 0x0730291bd6901995 + .quad 0xbb40a859182362d6 + .quad 0xb99f55778a4d1abb + .quad 0x8d18b427758559f6 + .quad 0x26c20fe74d26235a + .quad 0x648d1d9fe9cc22f5 + .quad 0x66bc561928dd577c + .quad 0x47d3ed21652439d1 + .quad 0x49d271acedaf8b49 + + // 2^80 * 1 * G + + .quad 0x89f5058a382b33f3 + .quad 0x5ae2ba0bad48c0b4 + .quad 0x8f93b503a53db36e + .quad 0x5aa3ed9d95a232e6 + .quad 0x2798aaf9b4b75601 + .quad 0x5eac72135c8dad72 + .quad 0xd2ceaa6161b7a023 + .quad 0x1bbfb284e98f7d4e + .quad 0x656777e9c7d96561 + .quad 0xcb2b125472c78036 + .quad 0x65053299d9506eee + .quad 0x4a07e14e5e8957cc + + // 2^80 * 2 * G + + .quad 0x4ee412cb980df999 + .quad 0xa315d76f3c6ec771 + .quad 0xbba5edde925c77fd + .quad 0x3f0bac391d313402 + .quad 0x240b58cdc477a49b + .quad 0xfd38dade6447f017 + .quad 0x19928d32a7c86aad + .quad 0x50af7aed84afa081 + .quad 0x6e4fde0115f65be5 + .quad 0x29982621216109b2 + .quad 0x780205810badd6d9 + .quad 0x1921a316baebd006 + + // 2^80 * 3 * G + + .quad 0x89422f7edfb870fc + .quad 0x2c296beb4f76b3bd + .quad 0x0738f1d436c24df7 + .quad 0x6458df41e273aeb0 + .quad 0xd75aad9ad9f3c18b + .quad 0x566a0eef60b1c19c + .quad 0x3e9a0bac255c0ed9 + .quad 0x7b049deca062c7f5 + .quad 0xdccbe37a35444483 + .quad 0x758879330fedbe93 + .quad 0x786004c312c5dd87 + .quad 0x6093dccbc2950e64 + + // 2^80 * 4 * G + + .quad 0x1ff39a8585e0706d + .quad 0x36d0a5d8b3e73933 + .quad 0x43b9f2e1718f453b + .quad 0x57d1ea084827a97c + .quad 0x6bdeeebe6084034b + .quad 0x3199c2b6780fb854 + .quad 0x973376abb62d0695 + .quad 0x6e3180c98b647d90 + .quad 0xee7ab6e7a128b071 + .quad 0xa4c1596d93a88baa + .quad 0xf7b4de82b2216130 + .quad 0x363e999ddd97bd18 + + // 2^80 * 5 * G + + .quad 0x96a843c135ee1fc4 + .quad 0x976eb35508e4c8cf + .quad 0xb42f6801b58cd330 + .quad 0x48ee9b78693a052b + .quad 0x2f1848dce24baec6 + .quad 0x769b7255babcaf60 + .quad 0x90cb3c6e3cefe931 + .quad 0x231f979bc6f9b355 + .quad 0x5c31de4bcc2af3c6 + .quad 0xb04bb030fe208d1f + .quad 0xb78d7009c14fb466 + .quad 0x079bfa9b08792413 + + // 2^80 * 6 * G + + .quad 0xe3903a51da300df4 + .quad 0x843964233da95ab0 + .quad 0xed3cf12d0b356480 + .quad 0x038c77f684817194 + .quad 0xf3c9ed80a2d54245 + .quad 0x0aa08b7877f63952 + .quad 0xd76dac63d1085475 + .quad 0x1ef4fb159470636b + .quad 0x854e5ee65b167bec + .quad 0x59590a4296d0cdc2 + .quad 0x72b2df3498102199 + .quad 0x575ee92a4a0bff56 + + // 2^80 * 7 * G + + .quad 0xd4c080908a182fcf + .quad 0x30e170c299489dbd + .quad 0x05babd5752f733de + .quad 0x43d4e7112cd3fd00 + .quad 0x5d46bc450aa4d801 + .quad 0xc3af1227a533b9d8 + .quad 0x389e3b262b8906c2 + .quad 0x200a1e7e382f581b + .quad 0x518db967eaf93ac5 + .quad 0x71bc989b056652c0 + .quad 0xfe2b85d9567197f5 + .quad 0x050eca52651e4e38 + + // 2^80 * 8 * G + + .quad 0xc3431ade453f0c9c + .quad 0xe9f5045eff703b9b + .quad 0xfcd97ac9ed847b3d + .quad 0x4b0ee6c21c58f4c6 + .quad 0x97ac397660e668ea + .quad 0x9b19bbfe153ab497 + .quad 0x4cb179b534eca79f + .quad 0x6151c09fa131ae57 + .quad 0x3af55c0dfdf05d96 + .quad 0xdd262ee02ab4ee7a + .quad 0x11b2bb8712171709 + .quad 0x1fef24fa800f030b + + // 2^84 * 1 * G + + .quad 0xb496123a6b6c6609 + .quad 0xa750fe8580ab5938 + .quad 0xf471bf39b7c27a5f + .quad 0x507903ce77ac193c + .quad 0xff91a66a90166220 + .quad 0xf22552ae5bf1e009 + .quad 0x7dff85d87f90df7c + .quad 0x4f620ffe0c736fb9 + .quad 0x62f90d65dfde3e34 + .quad 0xcf28c592b9fa5fad + .quad 0x99c86ef9c6164510 + .quad 0x25d448044a256c84 + + // 2^84 * 2 * G + + .quad 0xbd68230ec7e9b16f + .quad 0x0eb1b9c1c1c5795d + .quad 0x7943c8c495b6b1ff + .quad 0x2f9faf620bbacf5e + .quad 0x2c7c4415c9022b55 + .quad 0x56a0d241812eb1fe + .quad 0xf02ea1c9d7b65e0d + .quad 0x4180512fd5323b26 + .quad 0xa4ff3e698a48a5db + .quad 0xba6a3806bd95403b + .quad 0x9f7ce1af47d5b65d + .quad 0x15e087e55939d2fb + + // 2^84 * 3 * G + + .quad 0x12207543745c1496 + .quad 0xdaff3cfdda38610c + .quad 0xe4e797272c71c34f + .quad 0x39c07b1934bdede9 + .quad 0x8894186efb963f38 + .quad 0x48a00e80dc639bd5 + .quad 0xa4e8092be96c1c99 + .quad 0x5a097d54ca573661 + .quad 0x2d45892b17c9e755 + .quad 0xd033fd7289308df8 + .quad 0x6c2fe9d9525b8bd9 + .quad 0x2edbecf1c11cc079 + + // 2^84 * 4 * G + + .quad 0x1616a4e3c715a0d2 + .quad 0x53623cb0f8341d4d + .quad 0x96ef5329c7e899cb + .quad 0x3d4e8dbba668baa6 + .quad 0xee0f0fddd087a25f + .quad 0x9c7531555c3e34ee + .quad 0x660c572e8fab3ab5 + .quad 0x0854fc44544cd3b2 + .quad 0x61eba0c555edad19 + .quad 0x24b533fef0a83de6 + .quad 0x3b77042883baa5f8 + .quad 0x678f82b898a47e8d + + // 2^84 * 5 * G + + .quad 0xb1491d0bd6900c54 + .quad 0x3539722c9d132636 + .quad 0x4db928920b362bc9 + .quad 0x4d7cd1fea68b69df + .quad 0x1e09d94057775696 + .quad 0xeed1265c3cd951db + .quad 0xfa9dac2b20bce16f + .quad 0x0f7f76e0e8d089f4 + .quad 0x36d9ebc5d485b00c + .quad 0xa2596492e4adb365 + .quad 0xc1659480c2119ccd + .quad 0x45306349186e0d5f + + // 2^84 * 6 * G + + .quad 0x94ddd0c1a6cdff1d + .quad 0x55f6f115e84213ae + .quad 0x6c935f85992fcf6a + .quad 0x067ee0f54a37f16f + .quad 0x96a414ec2b072491 + .quad 0x1bb2218127a7b65b + .quad 0x6d2849596e8a4af0 + .quad 0x65f3b08ccd27765f + .quad 0xecb29fff199801f7 + .quad 0x9d361d1fa2a0f72f + .quad 0x25f11d2375fd2f49 + .quad 0x124cefe80fe10fe2 + + // 2^84 * 7 * G + + .quad 0x4c126cf9d18df255 + .quad 0xc1d471e9147a63b6 + .quad 0x2c6d3c73f3c93b5f + .quad 0x6be3a6a2e3ff86a2 + .quad 0x1518e85b31b16489 + .quad 0x8faadcb7db710bfb + .quad 0x39b0bdf4a14ae239 + .quad 0x05f4cbea503d20c1 + .quad 0xce040e9ec04145bc + .quad 0xc71ff4e208f6834c + .quad 0xbd546e8dab8847a3 + .quad 0x64666aa0a4d2aba5 + + // 2^84 * 8 * G + + .quad 0x6841435a7c06d912 + .quad 0xca123c21bb3f830b + .quad 0xd4b37b27b1cbe278 + .quad 0x1d753b84c76f5046 + .quad 0xb0c53bf73337e94c + .quad 0x7cb5697e11e14f15 + .quad 0x4b84abac1930c750 + .quad 0x28dd4abfe0640468 + .quad 0x7dc0b64c44cb9f44 + .quad 0x18a3e1ace3925dbf + .quad 0x7a3034862d0457c4 + .quad 0x4c498bf78a0c892e + + // 2^88 * 1 * G + + .quad 0x37d653fb1aa73196 + .quad 0x0f9495303fd76418 + .quad 0xad200b09fb3a17b2 + .quad 0x544d49292fc8613e + .quad 0x22d2aff530976b86 + .quad 0x8d90b806c2d24604 + .quad 0xdca1896c4de5bae5 + .quad 0x28005fe6c8340c17 + .quad 0x6aefba9f34528688 + .quad 0x5c1bff9425107da1 + .quad 0xf75bbbcd66d94b36 + .quad 0x72e472930f316dfa + + // 2^88 * 2 * G + + .quad 0x2695208c9781084f + .quad 0xb1502a0b23450ee1 + .quad 0xfd9daea603efde02 + .quad 0x5a9d2e8c2733a34c + .quad 0x07f3f635d32a7627 + .quad 0x7aaa4d865f6566f0 + .quad 0x3c85e79728d04450 + .quad 0x1fee7f000fe06438 + .quad 0x765305da03dbf7e5 + .quad 0xa4daf2491434cdbd + .quad 0x7b4ad5cdd24a88ec + .quad 0x00f94051ee040543 + + // 2^88 * 3 * G + + .quad 0x8d356b23c3d330b2 + .quad 0xf21c8b9bb0471b06 + .quad 0xb36c316c6e42b83c + .quad 0x07d79c7e8beab10d + .quad 0xd7ef93bb07af9753 + .quad 0x583ed0cf3db766a7 + .quad 0xce6998bf6e0b1ec5 + .quad 0x47b7ffd25dd40452 + .quad 0x87fbfb9cbc08dd12 + .quad 0x8a066b3ae1eec29b + .quad 0x0d57242bdb1fc1bf + .quad 0x1c3520a35ea64bb6 + + // 2^88 * 4 * G + + .quad 0x80d253a6bccba34a + .quad 0x3e61c3a13838219b + .quad 0x90c3b6019882e396 + .quad 0x1c3d05775d0ee66f + .quad 0xcda86f40216bc059 + .quad 0x1fbb231d12bcd87e + .quad 0xb4956a9e17c70990 + .quad 0x38750c3b66d12e55 + .quad 0x692ef1409422e51a + .quad 0xcbc0c73c2b5df671 + .quad 0x21014fe7744ce029 + .quad 0x0621e2c7d330487c + + // 2^88 * 5 * G + + .quad 0xaf9860cc8259838d + .quad 0x90ea48c1c69f9adc + .quad 0x6526483765581e30 + .quad 0x0007d6097bd3a5bc + .quad 0xb7ae1796b0dbf0f3 + .quad 0x54dfafb9e17ce196 + .quad 0x25923071e9aaa3b4 + .quad 0x5d8e589ca1002e9d + .quad 0xc0bf1d950842a94b + .quad 0xb2d3c363588f2e3e + .quad 0x0a961438bb51e2ef + .quad 0x1583d7783c1cbf86 + + // 2^88 * 6 * G + + .quad 0xeceea2ef5da27ae1 + .quad 0x597c3a1455670174 + .quad 0xc9a62a126609167a + .quad 0x252a5f2e81ed8f70 + .quad 0x90034704cc9d28c7 + .quad 0x1d1b679ef72cc58f + .quad 0x16e12b5fbe5b8726 + .quad 0x4958064e83c5580a + .quad 0x0d2894265066e80d + .quad 0xfcc3f785307c8c6b + .quad 0x1b53da780c1112fd + .quad 0x079c170bd843b388 + + // 2^88 * 7 * G + + .quad 0x0506ece464fa6fff + .quad 0xbee3431e6205e523 + .quad 0x3579422451b8ea42 + .quad 0x6dec05e34ac9fb00 + .quad 0xcdd6cd50c0d5d056 + .quad 0x9af7686dbb03573b + .quad 0x3ca6723ff3c3ef48 + .quad 0x6768c0d7317b8acc + .quad 0x94b625e5f155c1b3 + .quad 0x417bf3a7997b7b91 + .quad 0xc22cbddc6d6b2600 + .quad 0x51445e14ddcd52f4 + + // 2^88 * 8 * G + + .quad 0x57502b4b3b144951 + .quad 0x8e67ff6b444bbcb3 + .quad 0xb8bd6927166385db + .quad 0x13186f31e39295c8 + .quad 0x893147ab2bbea455 + .quad 0x8c53a24f92079129 + .quad 0x4b49f948be30f7a7 + .quad 0x12e990086e4fd43d + .quad 0xf10c96b37fdfbb2e + .quad 0x9f9a935e121ceaf9 + .quad 0xdf1136c43a5b983f + .quad 0x77b2e3f05d3e99af + + // 2^92 * 1 * G + + .quad 0xfd0d75879cf12657 + .quad 0xe82fef94e53a0e29 + .quad 0xcc34a7f05bbb4be7 + .quad 0x0b251172a50c38a2 + .quad 0x9532f48fcc5cd29b + .quad 0x2ba851bea3ce3671 + .quad 0x32dacaa051122941 + .quad 0x478d99d9350004f2 + .quad 0x1d5ad94890bb02c0 + .quad 0x50e208b10ec25115 + .quad 0xa26a22894ef21702 + .quad 0x4dc923343b524805 + + // 2^92 * 2 * G + + .quad 0xe3828c400f8086b6 + .quad 0x3f77e6f7979f0dc8 + .quad 0x7ef6de304df42cb4 + .quad 0x5265797cb6abd784 + .quad 0x3ad3e3ebf36c4975 + .quad 0xd75d25a537862125 + .quad 0xe873943da025a516 + .quad 0x6bbc7cb4c411c847 + .quad 0x3c6f9cd1d4a50d56 + .quad 0xb6244077c6feab7e + .quad 0x6ff9bf483580972e + .quad 0x00375883b332acfb + + // 2^92 * 3 * G + + .quad 0x0001b2cd28cb0940 + .quad 0x63fb51a06f1c24c9 + .quad 0xb5ad8691dcd5ca31 + .quad 0x67238dbd8c450660 + .quad 0xc98bec856c75c99c + .quad 0xe44184c000e33cf4 + .quad 0x0a676b9bba907634 + .quad 0x669e2cb571f379d7 + .quad 0xcb116b73a49bd308 + .quad 0x025aad6b2392729e + .quad 0xb4793efa3f55d9b1 + .quad 0x72a1056140678bb9 + + // 2^92 * 4 * G + + .quad 0xa2b6812b1cc9249d + .quad 0x62866eee21211f58 + .quad 0x2cb5c5b85df10ece + .quad 0x03a6b259e263ae00 + .quad 0x0d8d2909e2e505b6 + .quad 0x98ca78abc0291230 + .quad 0x77ef5569a9b12327 + .quad 0x7c77897b81439b47 + .quad 0xf1c1b5e2de331cb5 + .quad 0x5a9f5d8e15fca420 + .quad 0x9fa438f17bd932b1 + .quad 0x2a381bf01c6146e7 + + // 2^92 * 5 * G + + .quad 0xac9b9879cfc811c1 + .quad 0x8b7d29813756e567 + .quad 0x50da4e607c70edfc + .quad 0x5dbca62f884400b6 + .quad 0xf7c0be32b534166f + .quad 0x27e6ca6419cf70d4 + .quad 0x934df7d7a957a759 + .quad 0x5701461dabdec2aa + .quad 0x2c6747402c915c25 + .quad 0x1bdcd1a80b0d340a + .quad 0x5e5601bd07b43f5f + .quad 0x2555b4e05539a242 + + // 2^92 * 6 * G + + .quad 0x6fc09f5266ddd216 + .quad 0xdce560a7c8e37048 + .quad 0xec65939da2df62fd + .quad 0x7a869ae7e52ed192 + .quad 0x78409b1d87e463d4 + .quad 0xad4da95acdfb639d + .quad 0xec28773755259b9c + .quad 0x69c806e9c31230ab + .quad 0x7b48f57414bb3f22 + .quad 0x68c7cee4aedccc88 + .quad 0xed2f936179ed80be + .quad 0x25d70b885f77bc4b + + // 2^92 * 7 * G + + .quad 0x4151c3d9762bf4de + .quad 0x083f435f2745d82b + .quad 0x29775a2e0d23ddd5 + .quad 0x138e3a6269a5db24 + .quad 0x98459d29bb1ae4d4 + .quad 0x56b9c4c739f954ec + .quad 0x832743f6c29b4b3e + .quad 0x21ea8e2798b6878a + .quad 0x87bef4b46a5a7b9c + .quad 0xd2299d1b5fc1d062 + .quad 0x82409818dd321648 + .quad 0x5c5abeb1e5a2e03d + + // 2^92 * 8 * G + + .quad 0x14722af4b73c2ddb + .quad 0xbc470c5f5a05060d + .quad 0x00943eac2581b02e + .quad 0x0e434b3b1f499c8f + .quad 0x02cde6de1306a233 + .quad 0x7b5a52a2116f8ec7 + .quad 0xe1c681f4c1163b5b + .quad 0x241d350660d32643 + .quad 0x6be4404d0ebc52c7 + .quad 0xae46233bb1a791f5 + .quad 0x2aec170ed25db42b + .quad 0x1d8dfd966645d694 + + // 2^96 * 1 * G + + .quad 0x296fa9c59c2ec4de + .quad 0xbc8b61bf4f84f3cb + .quad 0x1c7706d917a8f908 + .quad 0x63b795fc7ad3255d + .quad 0xd598639c12ddb0a4 + .quad 0xa5d19f30c024866b + .quad 0xd17c2f0358fce460 + .quad 0x07a195152e095e8a + .quad 0xa8368f02389e5fc8 + .quad 0x90433b02cf8de43b + .quad 0xafa1fd5dc5412643 + .quad 0x3e8fe83d032f0137 + + // 2^96 * 2 * G + + .quad 0x2f8b15b90570a294 + .quad 0x94f2427067084549 + .quad 0xde1c5ae161bbfd84 + .quad 0x75ba3b797fac4007 + .quad 0x08704c8de8efd13c + .quad 0xdfc51a8e33e03731 + .quad 0xa59d5da51260cde3 + .quad 0x22d60899a6258c86 + .quad 0x6239dbc070cdd196 + .quad 0x60fe8a8b6c7d8a9a + .quad 0xb38847bceb401260 + .quad 0x0904d07b87779e5e + + // 2^96 * 3 * G + + .quad 0xb4ce1fd4ddba919c + .quad 0xcf31db3ec74c8daa + .quad 0x2c63cc63ad86cc51 + .quad 0x43e2143fbc1dde07 + .quad 0xf4322d6648f940b9 + .quad 0x06952f0cbd2d0c39 + .quad 0x167697ada081f931 + .quad 0x6240aacebaf72a6c + .quad 0xf834749c5ba295a0 + .quad 0xd6947c5bca37d25a + .quad 0x66f13ba7e7c9316a + .quad 0x56bdaf238db40cac + + // 2^96 * 4 * G + + .quad 0x362ab9e3f53533eb + .quad 0x338568d56eb93d40 + .quad 0x9e0e14521d5a5572 + .quad 0x1d24a86d83741318 + .quad 0x1310d36cc19d3bb2 + .quad 0x062a6bb7622386b9 + .quad 0x7c9b8591d7a14f5c + .quad 0x03aa31507e1e5754 + .quad 0xf4ec7648ffd4ce1f + .quad 0xe045eaf054ac8c1c + .quad 0x88d225821d09357c + .quad 0x43b261dc9aeb4859 + + // 2^96 * 5 * G + + .quad 0xe55b1e1988bb79bb + .quad 0xa09ed07dc17a359d + .quad 0xb02c2ee2603dea33 + .quad 0x326055cf5b276bc2 + .quad 0x19513d8b6c951364 + .quad 0x94fe7126000bf47b + .quad 0x028d10ddd54f9567 + .quad 0x02b4d5e242940964 + .quad 0xb4a155cb28d18df2 + .quad 0xeacc4646186ce508 + .quad 0xc49cf4936c824389 + .quad 0x27a6c809ae5d3410 + + // 2^96 * 6 * G + + .quad 0x8ba6ebcd1f0db188 + .quad 0x37d3d73a675a5be8 + .quad 0xf22edfa315f5585a + .quad 0x2cb67174ff60a17e + .quad 0xcd2c270ac43d6954 + .quad 0xdd4a3e576a66cab2 + .quad 0x79fa592469d7036c + .quad 0x221503603d8c2599 + .quad 0x59eecdf9390be1d0 + .quad 0xa9422044728ce3f1 + .quad 0x82891c667a94f0f4 + .quad 0x7b1df4b73890f436 + + // 2^96 * 7 * G + + .quad 0xe492f2e0b3b2a224 + .quad 0x7c6c9e062b551160 + .quad 0x15eb8fe20d7f7b0e + .quad 0x61fcef2658fc5992 + .quad 0x5f2e221807f8f58c + .quad 0xe3555c9fd49409d4 + .quad 0xb2aaa88d1fb6a630 + .quad 0x68698245d352e03d + .quad 0xdbb15d852a18187a + .quad 0xf3e4aad386ddacd7 + .quad 0x44bae2810ff6c482 + .quad 0x46cf4c473daf01cf + + // 2^96 * 8 * G + + .quad 0x426525ed9ec4e5f9 + .quad 0x0e5eda0116903303 + .quad 0x72b1a7f2cbe5cadc + .quad 0x29387bcd14eb5f40 + .quad 0x213c6ea7f1498140 + .quad 0x7c1e7ef8392b4854 + .quad 0x2488c38c5629ceba + .quad 0x1065aae50d8cc5bb + .quad 0x1c2c4525df200d57 + .quad 0x5c3b2dd6bfca674a + .quad 0x0a07e7b1e1834030 + .quad 0x69a198e64f1ce716 + + // 2^100 * 1 * G + + .quad 0x7afcd613efa9d697 + .quad 0x0cc45aa41c067959 + .quad 0xa56fe104c1fada96 + .quad 0x3a73b70472e40365 + .quad 0x7b26e56b9e2d4734 + .quad 0xc4c7132b81c61675 + .quad 0xef5c9525ec9cde7f + .quad 0x39c80b16e71743ad + .quad 0x0f196e0d1b826c68 + .quad 0xf71ff0e24960e3db + .quad 0x6113167023b7436c + .quad 0x0cf0ea5877da7282 + + // 2^100 * 2 * G + + .quad 0x196c80a4ddd4ccbd + .quad 0x22e6f55d95f2dd9d + .quad 0xc75e33c740d6c71b + .quad 0x7bb51279cb3c042f + .quad 0xe332ced43ba6945a + .quad 0xde0b1361e881c05d + .quad 0x1ad40f095e67ed3b + .quad 0x5da8acdab8c63d5d + .quad 0xc4b6664a3a70159f + .quad 0x76194f0f0a904e14 + .quad 0xa5614c39a4096c13 + .quad 0x6cd0ff50979feced + + // 2^100 * 3 * G + + .quad 0xc0e067e78f4428ac + .quad 0x14835ab0a61135e3 + .quad 0xf21d14f338062935 + .quad 0x6390a4c8df04849c + .quad 0x7fecfabdb04ba18e + .quad 0xd0fc7bfc3bddbcf7 + .quad 0xa41d486e057a131c + .quad 0x641a4391f2223a61 + .quad 0xc5c6b95aa606a8db + .quad 0x914b7f9eb06825f1 + .quad 0x2a731f6b44fc9eff + .quad 0x30ddf38562705cfc + + // 2^100 * 4 * G + + .quad 0x4e3dcbdad1bff7f9 + .quad 0xc9118e8220645717 + .quad 0xbacccebc0f189d56 + .quad 0x1b4822e9d4467668 + .quad 0x33bef2bd68bcd52c + .quad 0xc649dbb069482ef2 + .quad 0xb5b6ee0c41cb1aee + .quad 0x5c294d270212a7e5 + .quad 0xab360a7f25563781 + .quad 0x2512228a480f7958 + .quad 0xc75d05276114b4e3 + .quad 0x222d9625d976fe2a + + // 2^100 * 5 * G + + .quad 0x1c717f85b372ace1 + .quad 0x81930e694638bf18 + .quad 0x239cad056bc08b58 + .quad 0x0b34271c87f8fff4 + .quad 0x0f94be7e0a344f85 + .quad 0xeb2faa8c87f22c38 + .quad 0x9ce1e75e4ee16f0f + .quad 0x43e64e5418a08dea + .quad 0x8155e2521a35ce63 + .quad 0xbe100d4df912028e + .quad 0xbff80bf8a57ddcec + .quad 0x57342dc96d6bc6e4 + + // 2^100 * 6 * G + + .quad 0xefeef065c8ce5998 + .quad 0xbf029510b5cbeaa2 + .quad 0x8c64a10620b7c458 + .quad 0x35134fb231c24855 + .quad 0xf3c3bcb71e707bf6 + .quad 0x351d9b8c7291a762 + .quad 0x00502e6edad69a33 + .quad 0x522f521f1ec8807f + .quad 0x272c1f46f9a3902b + .quad 0xc91ba3b799657bcc + .quad 0xae614b304f8a1c0e + .quad 0x7afcaad70b99017b + + // 2^100 * 7 * G + + .quad 0xc25ded54a4b8be41 + .quad 0x902d13e11bb0e2dd + .quad 0x41f43233cde82ab2 + .quad 0x1085faa5c3aae7cb + .quad 0xa88141ecef842b6b + .quad 0x55e7b14797abe6c5 + .quad 0x8c748f9703784ffe + .quad 0x5b50a1f7afcd00b7 + .quad 0x9b840f66f1361315 + .quad 0x18462242701003e9 + .quad 0x65ed45fae4a25080 + .quad 0x0a2862393fda7320 + + // 2^100 * 8 * G + + .quad 0x46ab13c8347cbc9d + .quad 0x3849e8d499c12383 + .quad 0x4cea314087d64ac9 + .quad 0x1f354134b1a29ee7 + .quad 0x960e737b6ecb9d17 + .quad 0xfaf24948d67ceae1 + .quad 0x37e7a9b4d55e1b89 + .quad 0x5cb7173cb46c59eb + .quad 0x4a89e68b82b7abf0 + .quad 0xf41cd9279ba6b7b9 + .quad 0x16e6c210e18d876f + .quad 0x7cacdb0f7f1b09c6 + + // 2^104 * 1 * G + + .quad 0x9062b2e0d91a78bc + .quad 0x47c9889cc8509667 + .quad 0x9df54a66405070b8 + .quad 0x7369e6a92493a1bf + .quad 0xe1014434dcc5caed + .quad 0x47ed5d963c84fb33 + .quad 0x70019576ed86a0e7 + .quad 0x25b2697bd267f9e4 + .quad 0x9d673ffb13986864 + .quad 0x3ca5fbd9415dc7b8 + .quad 0xe04ecc3bdf273b5e + .quad 0x1420683db54e4cd2 + + // 2^104 * 2 * G + + .quad 0xb478bd1e249dd197 + .quad 0x620c35005e58c102 + .quad 0xfb02d32fccbaac5c + .quad 0x60b63bebf508a72d + .quad 0x34eebb6fc1cc5ad0 + .quad 0x6a1b0ce99646ac8b + .quad 0xd3b0da49a66bde53 + .quad 0x31e83b4161d081c1 + .quad 0x97e8c7129e062b4f + .quad 0x49e48f4f29320ad8 + .quad 0x5bece14b6f18683f + .quad 0x55cf1eb62d550317 + + // 2^104 * 3 * G + + .quad 0x5879101065c23d58 + .quad 0x8b9d086d5094819c + .quad 0xe2402fa912c55fa7 + .quad 0x669a6564570891d4 + .quad 0x3076b5e37df58c52 + .quad 0xd73ab9dde799cc36 + .quad 0xbd831ce34913ee20 + .quad 0x1a56fbaa62ba0133 + .quad 0x943e6b505c9dc9ec + .quad 0x302557bba77c371a + .quad 0x9873ae5641347651 + .quad 0x13c4836799c58a5c + + // 2^104 * 4 * G + + .quad 0x423a5d465ab3e1b9 + .quad 0xfc13c187c7f13f61 + .quad 0x19f83664ecb5b9b6 + .quad 0x66f80c93a637b607 + .quad 0xc4dcfb6a5d8bd080 + .quad 0xdeebc4ec571a4842 + .quad 0xd4b2e883b8e55365 + .quad 0x50bdc87dc8e5b827 + .quad 0x606d37836edfe111 + .quad 0x32353e15f011abd9 + .quad 0x64b03ac325b73b96 + .quad 0x1dd56444725fd5ae + + // 2^104 * 5 * G + + .quad 0x8fa47ff83362127d + .quad 0xbc9f6ac471cd7c15 + .quad 0x6e71454349220c8b + .quad 0x0e645912219f732e + .quad 0xc297e60008bac89a + .quad 0x7d4cea11eae1c3e0 + .quad 0xf3e38be19fe7977c + .quad 0x3a3a450f63a305cd + .quad 0x078f2f31d8394627 + .quad 0x389d3183de94a510 + .quad 0xd1e36c6d17996f80 + .quad 0x318c8d9393a9a87b + + // 2^104 * 6 * G + + .quad 0xf2745d032afffe19 + .quad 0x0c9f3c497f24db66 + .quad 0xbc98d3e3ba8598ef + .quad 0x224c7c679a1d5314 + .quad 0x5d669e29ab1dd398 + .quad 0xfc921658342d9e3b + .quad 0x55851dfdf35973cd + .quad 0x509a41c325950af6 + .quad 0xbdc06edca6f925e9 + .quad 0x793ef3f4641b1f33 + .quad 0x82ec12809d833e89 + .quad 0x05bff02328a11389 + + // 2^104 * 7 * G + + .quad 0x3632137023cae00b + .quad 0x544acf0ad1accf59 + .quad 0x96741049d21a1c88 + .quad 0x780b8cc3fa2a44a7 + .quad 0x6881a0dd0dc512e4 + .quad 0x4fe70dc844a5fafe + .quad 0x1f748e6b8f4a5240 + .quad 0x576277cdee01a3ea + .quad 0x1ef38abc234f305f + .quad 0x9a577fbd1405de08 + .quad 0x5e82a51434e62a0d + .quad 0x5ff418726271b7a1 + + // 2^104 * 8 * G + + .quad 0x398e080c1789db9d + .quad 0xa7602025f3e778f5 + .quad 0xfa98894c06bd035d + .quad 0x106a03dc25a966be + .quad 0xe5db47e813b69540 + .quad 0xf35d2a3b432610e1 + .quad 0xac1f26e938781276 + .quad 0x29d4db8ca0a0cb69 + .quad 0xd9ad0aaf333353d0 + .quad 0x38669da5acd309e5 + .quad 0x3c57658ac888f7f0 + .quad 0x4ab38a51052cbefa + + // 2^108 * 1 * G + + .quad 0xdfdacbee4324c0e9 + .quad 0x054442883f955bb7 + .quad 0xdef7aaa8ea31609f + .quad 0x68aee70642287cff + .quad 0xf68fe2e8809de054 + .quad 0xe3bc096a9c82bad1 + .quad 0x076353d40aadbf45 + .quad 0x7b9b1fb5dea1959e + .quad 0xf01cc8f17471cc0c + .quad 0x95242e37579082bb + .quad 0x27776093d3e46b5f + .quad 0x2d13d55a28bd85fb + + // 2^108 * 2 * G + + .quad 0xfac5d2065b35b8da + .quad 0xa8da8a9a85624bb7 + .quad 0xccd2ca913d21cd0f + .quad 0x6b8341ee8bf90d58 + .quad 0xbf019cce7aee7a52 + .quad 0xa8ded2b6e454ead3 + .quad 0x3c619f0b87a8bb19 + .quad 0x3619b5d7560916d8 + .quad 0x3579f26b0282c4b2 + .quad 0x64d592f24fafefae + .quad 0xb7cded7b28c8c7c0 + .quad 0x6a927b6b7173a8d7 + + // 2^108 * 3 * G + + .quad 0x1f6db24f986e4656 + .quad 0x1021c02ed1e9105b + .quad 0xf8ff3fff2cc0a375 + .quad 0x1d2a6bf8c6c82592 + .quad 0x8d7040863ece88eb + .quad 0xf0e307a980eec08c + .quad 0xac2250610d788fda + .quad 0x056d92a43a0d478d + .quad 0x1b05a196fc3da5a1 + .quad 0x77d7a8c243b59ed0 + .quad 0x06da3d6297d17918 + .quad 0x66fbb494f12353f7 + + // 2^108 * 4 * G + + .quad 0x751a50b9d85c0fb8 + .quad 0xd1afdc258bcf097b + .quad 0x2f16a6a38309a969 + .quad 0x14ddff9ee5b00659 + .quad 0xd6d70996f12309d6 + .quad 0xdbfb2385e9c3d539 + .quad 0x46d602b0f7552411 + .quad 0x270a0b0557843e0c + .quad 0x61ff0640a7862bcc + .quad 0x81cac09a5f11abfe + .quad 0x9047830455d12abb + .quad 0x19a4bde1945ae873 + + // 2^108 * 5 * G + + .quad 0x9b9f26f520a6200a + .quad 0x64804443cf13eaf8 + .quad 0x8a63673f8631edd3 + .quad 0x72bbbce11ed39dc1 + .quad 0x40c709dec076c49f + .quad 0x657bfaf27f3e53f6 + .quad 0x40662331eca042c4 + .quad 0x14b375487eb4df04 + .quad 0xae853c94ab66dc47 + .quad 0xeb62343edf762d6e + .quad 0xf08e0e186fb2f7d1 + .quad 0x4f0b1c02700ab37a + + // 2^108 * 6 * G + + .quad 0xe1706787d81951fa + .quad 0xa10a2c8eb290c77b + .quad 0xe7382fa03ed66773 + .quad 0x0a4d84710bcc4b54 + .quad 0x79fd21ccc1b2e23f + .quad 0x4ae7c281453df52a + .quad 0xc8172ec9d151486b + .quad 0x68abe9443e0a7534 + .quad 0xda12c6c407831dcb + .quad 0x0da230d74d5c510d + .quad 0x4ab1531e6bd404e1 + .quad 0x4106b166bcf440ef + + // 2^108 * 7 * G + + .quad 0x02e57a421cd23668 + .quad 0x4ad9fb5d0eaef6fd + .quad 0x954e6727b1244480 + .quad 0x7f792f9d2699f331 + .quad 0xa485ccd539e4ecf2 + .quad 0x5aa3f3ad0555bab5 + .quad 0x145e3439937df82d + .quad 0x1238b51e1214283f + .quad 0x0b886b925fd4d924 + .quad 0x60906f7a3626a80d + .quad 0xecd367b4b98abd12 + .quad 0x2876beb1def344cf + + // 2^108 * 8 * G + + .quad 0xdc84e93563144691 + .quad 0x632fe8a0d61f23f4 + .quad 0x4caa800612a9a8d5 + .quad 0x48f9dbfa0e9918d3 + .quad 0xd594b3333a8a85f8 + .quad 0x4ea37689e78d7d58 + .quad 0x73bf9f455e8e351f + .quad 0x5507d7d2bc41ebb4 + .quad 0x1ceb2903299572fc + .quad 0x7c8ccaa29502d0ee + .quad 0x91bfa43411cce67b + .quad 0x5784481964a831e7 + + // 2^112 * 1 * G + + .quad 0xda7c2b256768d593 + .quad 0x98c1c0574422ca13 + .quad 0xf1a80bd5ca0ace1d + .quad 0x29cdd1adc088a690 + .quad 0xd6cfd1ef5fddc09c + .quad 0xe82b3efdf7575dce + .quad 0x25d56b5d201634c2 + .quad 0x3041c6bb04ed2b9b + .quad 0x0ff2f2f9d956e148 + .quad 0xade797759f356b2e + .quad 0x1a4698bb5f6c025c + .quad 0x104bbd6814049a7b + + // 2^112 * 2 * G + + .quad 0x51f0fd3168f1ed67 + .quad 0x2c811dcdd86f3bc2 + .quad 0x44dc5c4304d2f2de + .quad 0x5be8cc57092a7149 + .quad 0xa95d9a5fd67ff163 + .quad 0xe92be69d4cc75681 + .quad 0xb7f8024cde20f257 + .quad 0x204f2a20fb072df5 + .quad 0xc8143b3d30ebb079 + .quad 0x7589155abd652e30 + .quad 0x653c3c318f6d5c31 + .quad 0x2570fb17c279161f + + // 2^112 * 3 * G + + .quad 0x3efa367f2cb61575 + .quad 0xf5f96f761cd6026c + .quad 0xe8c7142a65b52562 + .quad 0x3dcb65ea53030acd + .quad 0x192ea9550bb8245a + .quad 0xc8e6fba88f9050d1 + .quad 0x7986ea2d88a4c935 + .quad 0x241c5f91de018668 + .quad 0x28d8172940de6caa + .quad 0x8fbf2cf022d9733a + .quad 0x16d7fcdd235b01d1 + .quad 0x08420edd5fcdf0e5 + + // 2^112 * 4 * G + + .quad 0xcdff20ab8362fa4a + .quad 0x57e118d4e21a3e6e + .quad 0xe3179617fc39e62b + .quad 0x0d9a53efbc1769fd + .quad 0x0358c34e04f410ce + .quad 0xb6135b5a276e0685 + .quad 0x5d9670c7ebb91521 + .quad 0x04d654f321db889c + .quad 0x5e7dc116ddbdb5d5 + .quad 0x2954deb68da5dd2d + .quad 0x1cb608173334a292 + .quad 0x4a7a4f2618991ad7 + + // 2^112 * 5 * G + + .quad 0xf4a718025fb15f95 + .quad 0x3df65f346b5c1b8f + .quad 0xcdfcf08500e01112 + .quad 0x11b50c4cddd31848 + .quad 0x24c3b291af372a4b + .quad 0x93da8270718147f2 + .quad 0xdd84856486899ef2 + .quad 0x4a96314223e0ee33 + .quad 0xa6e8274408a4ffd6 + .quad 0x738e177e9c1576d9 + .quad 0x773348b63d02b3f2 + .quad 0x4f4bce4dce6bcc51 + + // 2^112 * 6 * G + + .quad 0xa71fce5ae2242584 + .quad 0x26ea725692f58a9e + .quad 0xd21a09d71cea3cf4 + .quad 0x73fcdd14b71c01e6 + .quad 0x30e2616ec49d0b6f + .quad 0xe456718fcaec2317 + .quad 0x48eb409bf26b4fa6 + .quad 0x3042cee561595f37 + .quad 0x427e7079449bac41 + .quad 0x855ae36dbce2310a + .quad 0x4cae76215f841a7c + .quad 0x389e740c9a9ce1d6 + + // 2^112 * 7 * G + + .quad 0x64fcb3ae34dcb9ce + .quad 0x97500323e348d0ad + .quad 0x45b3f07d62c6381b + .quad 0x61545379465a6788 + .quad 0xc9bd78f6570eac28 + .quad 0xe55b0b3227919ce1 + .quad 0x65fc3eaba19b91ed + .quad 0x25c425e5d6263690 + .quad 0x3f3e06a6f1d7de6e + .quad 0x3ef976278e062308 + .quad 0x8c14f6264e8a6c77 + .quad 0x6539a08915484759 + + // 2^112 * 8 * G + + .quad 0xe9d21f74c3d2f773 + .quad 0xc150544125c46845 + .quad 0x624e5ce8f9b99e33 + .quad 0x11c5e4aac5cd186c + .quad 0xddc4dbd414bb4a19 + .quad 0x19b2bc3c98424f8e + .quad 0x48a89fd736ca7169 + .quad 0x0f65320ef019bd90 + .quad 0xd486d1b1cafde0c6 + .quad 0x4f3fe6e3163b5181 + .quad 0x59a8af0dfaf2939a + .quad 0x4cabc7bdec33072a + + // 2^116 * 1 * G + + .quad 0x16faa8fb532f7428 + .quad 0xdbd42ea046a4e272 + .quad 0x5337653b8b9ea480 + .quad 0x4065947223973f03 + .quad 0xf7c0a19c1a54a044 + .quad 0x4a1c5e2477bd9fbb + .quad 0xa6e3ca115af22972 + .quad 0x1819bb953f2e9e0d + .quad 0x498fbb795e042e84 + .quad 0x7d0dd89a7698b714 + .quad 0x8bfb0ba427fe6295 + .quad 0x36ba82e721200524 + + // 2^116 * 2 * G + + .quad 0xd60ecbb74245ec41 + .quad 0xfd9be89e34348716 + .quad 0xc9240afee42284de + .quad 0x4472f648d0531db4 + .quad 0xc8d69d0a57274ed5 + .quad 0x45ba803260804b17 + .quad 0xdf3cda102255dfac + .quad 0x77d221232709b339 + .quad 0x498a6d7064ad94d8 + .quad 0xa5b5c8fd9af62263 + .quad 0x8ca8ed0545c141f4 + .quad 0x2c63bec3662d358c + + // 2^116 * 3 * G + + .quad 0x7fe60d8bea787955 + .quad 0xb9dc117eb5f401b7 + .quad 0x91c7c09a19355cce + .quad 0x22692ef59442bedf + .quad 0x9a518b3a8586f8bf + .quad 0x9ee71af6cbb196f0 + .quad 0xaa0625e6a2385cf2 + .quad 0x1deb2176ddd7c8d1 + .quad 0x8563d19a2066cf6c + .quad 0x401bfd8c4dcc7cd7 + .quad 0xd976a6becd0d8f62 + .quad 0x67cfd773a278b05e + + // 2^116 * 4 * G + + .quad 0x8dec31faef3ee475 + .quad 0x99dbff8a9e22fd92 + .quad 0x512d11594e26cab1 + .quad 0x0cde561eec4310b9 + .quad 0x2d5fa9855a4e586a + .quad 0x65f8f7a449beab7e + .quad 0xaa074dddf21d33d3 + .quad 0x185cba721bcb9dee + .quad 0x93869da3f4e3cb41 + .quad 0xbf0392f540f7977e + .quad 0x026204fcd0463b83 + .quad 0x3ec91a769eec6eed + + // 2^116 * 5 * G + + .quad 0x1e9df75bf78166ad + .quad 0x4dfda838eb0cd7af + .quad 0xba002ed8c1eaf988 + .quad 0x13fedb3e11f33cfc + .quad 0x0fad2fb7b0a3402f + .quad 0x46615ecbfb69f4a8 + .quad 0xf745bcc8c5f8eaa6 + .quad 0x7a5fa8794a94e896 + .quad 0x52958faa13cd67a1 + .quad 0x965ee0818bdbb517 + .quad 0x16e58daa2e8845b3 + .quad 0x357d397d5499da8f + + // 2^116 * 6 * G + + .quad 0x1ebfa05fb0bace6c + .quad 0xc934620c1caf9a1e + .quad 0xcc771cc41d82b61a + .quad 0x2d94a16aa5f74fec + .quad 0x481dacb4194bfbf8 + .quad 0x4d77e3f1bae58299 + .quad 0x1ef4612e7d1372a0 + .quad 0x3a8d867e70ff69e1 + .quad 0x6f58cd5d55aff958 + .quad 0xba3eaa5c75567721 + .quad 0x75c123999165227d + .quad 0x69be1343c2f2b35e + + // 2^116 * 7 * G + + .quad 0x0e091d5ee197c92a + .quad 0x4f51019f2945119f + .quad 0x143679b9f034e99c + .quad 0x7d88112e4d24c696 + .quad 0x82bbbdac684b8de3 + .quad 0xa2f4c7d03fca0718 + .quad 0x337f92fbe096aaa8 + .quad 0x200d4d8c63587376 + .quad 0x208aed4b4893b32b + .quad 0x3efbf23ebe59b964 + .quad 0xd762deb0dba5e507 + .quad 0x69607bd681bd9d94 + + // 2^116 * 8 * G + + .quad 0xf6be021068de1ce1 + .quad 0xe8d518e70edcbc1f + .quad 0xe3effdd01b5505a5 + .quad 0x35f63353d3ec3fd0 + .quad 0x3b7f3bd49323a902 + .quad 0x7c21b5566b2c6e53 + .quad 0xe5ba8ff53a7852a7 + .quad 0x28bc77a5838ece00 + .quad 0x63ba78a8e25d8036 + .quad 0x63651e0094333490 + .quad 0x48d82f20288ce532 + .quad 0x3a31abfa36b57524 + + // 2^120 * 1 * G + + .quad 0x239e9624089c0a2e + .quad 0xc748c4c03afe4738 + .quad 0x17dbed2a764fa12a + .quad 0x639b93f0321c8582 + .quad 0xc08f788f3f78d289 + .quad 0xfe30a72ca1404d9f + .quad 0xf2778bfccf65cc9d + .quad 0x7ee498165acb2021 + .quad 0x7bd508e39111a1c3 + .quad 0x2b2b90d480907489 + .quad 0xe7d2aec2ae72fd19 + .quad 0x0edf493c85b602a6 + + // 2^120 * 2 * G + + .quad 0xaecc8158599b5a68 + .quad 0xea574f0febade20e + .quad 0x4fe41d7422b67f07 + .quad 0x403b92e3019d4fb4 + .quad 0x6767c4d284764113 + .quad 0xa090403ff7f5f835 + .quad 0x1c8fcffacae6bede + .quad 0x04c00c54d1dfa369 + .quad 0x4dc22f818b465cf8 + .quad 0x71a0f35a1480eff8 + .quad 0xaee8bfad04c7d657 + .quad 0x355bb12ab26176f4 + + // 2^120 * 3 * G + + .quad 0xa71e64cc7493bbf4 + .quad 0xe5bd84d9eca3b0c3 + .quad 0x0a6bc50cfa05e785 + .quad 0x0f9b8132182ec312 + .quad 0xa301dac75a8c7318 + .quad 0xed90039db3ceaa11 + .quad 0x6f077cbf3bae3f2d + .quad 0x7518eaf8e052ad8e + .quad 0xa48859c41b7f6c32 + .quad 0x0f2d60bcf4383298 + .quad 0x1815a929c9b1d1d9 + .quad 0x47c3871bbb1755c4 + + // 2^120 * 4 * G + + .quad 0x5144539771ec4f48 + .quad 0xf805b17dc98c5d6e + .quad 0xf762c11a47c3c66b + .quad 0x00b89b85764699dc + .quad 0xfbe65d50c85066b0 + .quad 0x62ecc4b0b3a299b0 + .quad 0xe53754ea441ae8e0 + .quad 0x08fea02ce8d48d5f + .quad 0x824ddd7668deead0 + .quad 0xc86445204b685d23 + .quad 0xb514cfcd5d89d665 + .quad 0x473829a74f75d537 + + // 2^120 * 5 * G + + .quad 0x82d2da754679c418 + .quad 0xe63bd7d8b2618df0 + .quad 0x355eef24ac47eb0a + .quad 0x2078684c4833c6b4 + .quad 0x23d9533aad3902c9 + .quad 0x64c2ddceef03588f + .quad 0x15257390cfe12fb4 + .quad 0x6c668b4d44e4d390 + .quad 0x3b48cf217a78820c + .quad 0xf76a0ab281273e97 + .quad 0xa96c65a78c8eed7b + .quad 0x7411a6054f8a433f + + // 2^120 * 6 * G + + .quad 0x4d659d32b99dc86d + .quad 0x044cdc75603af115 + .quad 0xb34c712cdcc2e488 + .quad 0x7c136574fb8134ff + .quad 0x579ae53d18b175b4 + .quad 0x68713159f392a102 + .quad 0x8455ecba1eef35f5 + .quad 0x1ec9a872458c398f + .quad 0xb8e6a4d400a2509b + .quad 0x9b81d7020bc882b4 + .quad 0x57e7cc9bf1957561 + .quad 0x3add88a5c7cd6460 + + // 2^120 * 7 * G + + .quad 0xab895770b635dcf2 + .quad 0x02dfef6cf66c1fbc + .quad 0x85530268beb6d187 + .quad 0x249929fccc879e74 + .quad 0x85c298d459393046 + .quad 0x8f7e35985ff659ec + .quad 0x1d2ca22af2f66e3a + .quad 0x61ba1131a406a720 + .quad 0xa3d0a0f116959029 + .quad 0x023b6b6cba7ebd89 + .quad 0x7bf15a3e26783307 + .quad 0x5620310cbbd8ece7 + + // 2^120 * 8 * G + + .quad 0x528993434934d643 + .quad 0xb9dbf806a51222f5 + .quad 0x8f6d878fc3f41c22 + .quad 0x37676a2a4d9d9730 + .quad 0x6646b5f477e285d6 + .quad 0x40e8ff676c8f6193 + .quad 0xa6ec7311abb594dd + .quad 0x7ec846f3658cec4d + .quad 0x9b5e8f3f1da22ec7 + .quad 0x130f1d776c01cd13 + .quad 0x214c8fcfa2989fb8 + .quad 0x6daaf723399b9dd5 + + // 2^124 * 1 * G + + .quad 0x591e4a5610628564 + .quad 0x2a4bb87ca8b4df34 + .quad 0xde2a2572e7a38e43 + .quad 0x3cbdabd9fee5046e + .quad 0x81aebbdd2cd13070 + .quad 0x962e4325f85a0e9e + .quad 0xde9391aacadffecb + .quad 0x53177fda52c230e6 + .quad 0xa7bc970650b9de79 + .quad 0x3d12a7fbc301b59b + .quad 0x02652e68d36ae38c + .quad 0x79d739835a6199dc + + // 2^124 * 2 * G + + .quad 0xd9354df64131c1bd + .quad 0x758094a186ec5822 + .quad 0x4464ee12e459f3c2 + .quad 0x6c11fce4cb133282 + .quad 0x21c9d9920d591737 + .quad 0x9bea41d2e9b46cd6 + .quad 0xe20e84200d89bfca + .quad 0x79d99f946eae5ff8 + .quad 0xf17b483568673205 + .quad 0x387deae83caad96c + .quad 0x61b471fd56ffe386 + .quad 0x31741195b745a599 + + // 2^124 * 3 * G + + .quad 0xe8d10190b77a360b + .quad 0x99b983209995e702 + .quad 0xbd4fdff8fa0247aa + .quad 0x2772e344e0d36a87 + .quad 0x17f8ba683b02a047 + .quad 0x50212096feefb6c8 + .quad 0x70139be21556cbe2 + .quad 0x203e44a11d98915b + .quad 0xd6863eba37b9e39f + .quad 0x105bc169723b5a23 + .quad 0x104f6459a65c0762 + .quad 0x567951295b4d38d4 + + // 2^124 * 4 * G + + .quad 0x535fd60613037524 + .quad 0xe210adf6b0fbc26a + .quad 0xac8d0a9b23e990ae + .quad 0x47204d08d72fdbf9 + .quad 0x07242eb30d4b497f + .quad 0x1ef96306b9bccc87 + .quad 0x37950934d8116f45 + .quad 0x05468d6201405b04 + .quad 0x00f565a9f93267de + .quad 0xcecfd78dc0d58e8a + .quad 0xa215e2dcf318e28e + .quad 0x4599ee919b633352 + + // 2^124 * 5 * G + + .quad 0xd3c220ca70e0e76b + .quad 0xb12bea58ea9f3094 + .quad 0x294ddec8c3271282 + .quad 0x0c3539e1a1d1d028 + .quad 0xac746d6b861ae579 + .quad 0x31ab0650f6aea9dc + .quad 0x241d661140256d4c + .quad 0x2f485e853d21a5de + .quad 0x329744839c0833f3 + .quad 0x6fe6257fd2abc484 + .quad 0x5327d1814b358817 + .quad 0x65712585893fe9bc + + // 2^124 * 6 * G + + .quad 0x9c102fb732a61161 + .quad 0xe48e10dd34d520a8 + .quad 0x365c63546f9a9176 + .quad 0x32f6fe4c046f6006 + .quad 0x81c29f1bd708ee3f + .quad 0xddcb5a05ae6407d0 + .quad 0x97aec1d7d2a3eba7 + .quad 0x1590521a91d50831 + .quad 0x40a3a11ec7910acc + .quad 0x9013dff8f16d27ae + .quad 0x1a9720d8abb195d4 + .quad 0x1bb9fe452ea98463 + + // 2^124 * 7 * G + + .quad 0xe9d1d950b3d54f9e + .quad 0x2d5f9cbee00d33c1 + .quad 0x51c2c656a04fc6ac + .quad 0x65c091ee3c1cbcc9 + .quad 0xcf5e6c95cc36747c + .quad 0x294201536b0bc30d + .quad 0x453ac67cee797af0 + .quad 0x5eae6ab32a8bb3c9 + .quad 0x7083661114f118ea + .quad 0x2b37b87b94349cad + .quad 0x7273f51cb4e99f40 + .quad 0x78a2a95823d75698 + + // 2^124 * 8 * G + + .quad 0xa2b072e95c8c2ace + .quad 0x69cffc96651e9c4b + .quad 0x44328ef842e7b42b + .quad 0x5dd996c122aadeb3 + .quad 0xb4f23c425ef83207 + .quad 0xabf894d3c9a934b5 + .quad 0xd0708c1339fd87f7 + .quad 0x1876789117166130 + .quad 0x925b5ef0670c507c + .quad 0x819bc842b93c33bf + .quad 0x10792e9a70dd003f + .quad 0x59ad4b7a6e28dc74 + + // 2^128 * 1 * G + + .quad 0x5f3a7562eb3dbe47 + .quad 0xf7ea38548ebda0b8 + .quad 0x00c3e53145747299 + .quad 0x1304e9e71627d551 + .quad 0x583b04bfacad8ea2 + .quad 0x29b743e8148be884 + .quad 0x2b1e583b0810c5db + .quad 0x2b5449e58eb3bbaa + .quad 0x789814d26adc9cfe + .quad 0x3c1bab3f8b48dd0b + .quad 0xda0fe1fff979c60a + .quad 0x4468de2d7c2dd693 + + // 2^128 * 2 * G + + .quad 0x51bb355e9419469e + .quad 0x33e6dc4c23ddc754 + .quad 0x93a5b6d6447f9962 + .quad 0x6cce7c6ffb44bd63 + .quad 0x4b9ad8c6f86307ce + .quad 0x21113531435d0c28 + .quad 0xd4a866c5657a772c + .quad 0x5da6427e63247352 + .quad 0x1a94c688deac22ca + .quad 0xb9066ef7bbae1ff8 + .quad 0x88ad8c388d59580f + .quad 0x58f29abfe79f2ca8 + + // 2^128 * 3 * G + + .quad 0xe90ecfab8de73e68 + .quad 0x54036f9f377e76a5 + .quad 0xf0495b0bbe015982 + .quad 0x577629c4a7f41e36 + .quad 0x4b5a64bf710ecdf6 + .quad 0xb14ce538462c293c + .quad 0x3643d056d50b3ab9 + .quad 0x6af93724185b4870 + .quad 0x3220024509c6a888 + .quad 0xd2e036134b558973 + .quad 0x83e236233c33289f + .quad 0x701f25bb0caec18f + + // 2^128 * 4 * G + + .quad 0xc3a8b0f8e4616ced + .quad 0xf700660e9e25a87d + .quad 0x61e3061ff4bca59c + .quad 0x2e0c92bfbdc40be9 + .quad 0x9d18f6d97cbec113 + .quad 0x844a06e674bfdbe4 + .quad 0x20f5b522ac4e60d6 + .quad 0x720a5bc050955e51 + .quad 0x0c3f09439b805a35 + .quad 0xe84e8b376242abfc + .quad 0x691417f35c229346 + .quad 0x0e9b9cbb144ef0ec + + // 2^128 * 5 * G + + .quad 0xfbbad48ffb5720ad + .quad 0xee81916bdbf90d0e + .quad 0xd4813152635543bf + .quad 0x221104eb3f337bd8 + .quad 0x8dee9bd55db1beee + .quad 0xc9c3ab370a723fb9 + .quad 0x44a8f1bf1c68d791 + .quad 0x366d44191cfd3cde + .quad 0x9e3c1743f2bc8c14 + .quad 0x2eda26fcb5856c3b + .quad 0xccb82f0e68a7fb97 + .quad 0x4167a4e6bc593244 + + // 2^128 * 6 * G + + .quad 0x643b9d2876f62700 + .quad 0x5d1d9d400e7668eb + .quad 0x1b4b430321fc0684 + .quad 0x7938bb7e2255246a + .quad 0xc2be2665f8ce8fee + .quad 0xe967ff14e880d62c + .quad 0xf12e6e7e2f364eee + .quad 0x34b33370cb7ed2f6 + .quad 0xcdc591ee8681d6cc + .quad 0xce02109ced85a753 + .quad 0xed7485c158808883 + .quad 0x1176fc6e2dfe65e4 + + // 2^128 * 7 * G + + .quad 0xb4af6cd05b9c619b + .quad 0x2ddfc9f4b2a58480 + .quad 0x3d4fa502ebe94dc4 + .quad 0x08fc3a4c677d5f34 + .quad 0xdb90e28949770eb8 + .quad 0x98fbcc2aacf440a3 + .quad 0x21354ffeded7879b + .quad 0x1f6a3e54f26906b6 + .quad 0x60a4c199d30734ea + .quad 0x40c085b631165cd6 + .quad 0xe2333e23f7598295 + .quad 0x4f2fad0116b900d1 + + // 2^128 * 8 * G + + .quad 0x44beb24194ae4e54 + .quad 0x5f541c511857ef6c + .quad 0xa61e6b2d368d0498 + .quad 0x445484a4972ef7ab + .quad 0x962cd91db73bb638 + .quad 0xe60577aafc129c08 + .quad 0x6f619b39f3b61689 + .quad 0x3451995f2944ee81 + .quad 0x9152fcd09fea7d7c + .quad 0x4a816c94b0935cf6 + .quad 0x258e9aaa47285c40 + .quad 0x10b89ca6042893b7 + + // 2^132 * 1 * G + + .quad 0x9b2a426e3b646025 + .quad 0x32127190385ce4cf + .quad 0xa25cffc2dd6dea45 + .quad 0x06409010bea8de75 + .quad 0xd67cded679d34aa0 + .quad 0xcc0b9ec0cc4db39f + .quad 0xa535a456e35d190f + .quad 0x2e05d9eaf61f6fef + .quad 0xc447901ad61beb59 + .quad 0x661f19bce5dc880a + .quad 0x24685482b7ca6827 + .quad 0x293c778cefe07f26 + + // 2^132 * 2 * G + + .quad 0x86809e7007069096 + .quad 0xaad75b15e4e50189 + .quad 0x07f35715a21a0147 + .quad 0x0487f3f112815d5e + .quad 0x16c795d6a11ff200 + .quad 0xcb70d0e2b15815c9 + .quad 0x89f293209b5395b5 + .quad 0x50b8c2d031e47b4f + .quad 0x48350c08068a4962 + .quad 0x6ffdd05351092c9a + .quad 0x17af4f4aaf6fc8dd + .quad 0x4b0553b53cdba58b + + // 2^132 * 3 * G + + .quad 0x9c65fcbe1b32ff79 + .quad 0xeb75ea9f03b50f9b + .quad 0xfced2a6c6c07e606 + .quad 0x35106cd551717908 + .quad 0xbf05211b27c152d4 + .quad 0x5ec26849bd1af639 + .quad 0x5e0b2caa8e6fab98 + .quad 0x054c8bdd50bd0840 + .quad 0x38a0b12f1dcf073d + .quad 0x4b60a8a3b7f6a276 + .quad 0xfed5ac25d3404f9a + .quad 0x72e82d5e5505c229 + + // 2^132 * 4 * G + + .quad 0x6b0b697ff0d844c8 + .quad 0xbb12f85cd979cb49 + .quad 0xd2a541c6c1da0f1f + .quad 0x7b7c242958ce7211 + .quad 0x00d9cdfd69771d02 + .quad 0x410276cd6cfbf17e + .quad 0x4c45306c1cb12ec7 + .quad 0x2857bf1627500861 + .quad 0x9f21903f0101689e + .quad 0xd779dfd3bf861005 + .quad 0xa122ee5f3deb0f1b + .quad 0x510df84b485a00d4 + + // 2^132 * 5 * G + + .quad 0xa54133bb9277a1fa + .quad 0x74ec3b6263991237 + .quad 0x1a3c54dc35d2f15a + .quad 0x2d347144e482ba3a + .quad 0x24b3c887c70ac15e + .quad 0xb0f3a557fb81b732 + .quad 0x9b2cde2fe578cc1b + .quad 0x4cf7ed0703b54f8e + .quad 0x6bd47c6598fbee0f + .quad 0x9e4733e2ab55be2d + .quad 0x1093f624127610c5 + .quad 0x4e05e26ad0a1eaa4 + + // 2^132 * 6 * G + + .quad 0xda9b6b624b531f20 + .quad 0x429a760e77509abb + .quad 0xdbe9f522e823cb80 + .quad 0x618f1856880c8f82 + .quad 0x1833c773e18fe6c0 + .quad 0xe3c4711ad3c87265 + .quad 0x3bfd3c4f0116b283 + .quad 0x1955875eb4cd4db8 + .quad 0x6da6de8f0e399799 + .quad 0x7ad61aa440fda178 + .quad 0xb32cd8105e3563dd + .quad 0x15f6beae2ae340ae + + // 2^132 * 7 * G + + .quad 0x862bcb0c31ec3a62 + .quad 0x810e2b451138f3c2 + .quad 0x788ec4b839dac2a4 + .quad 0x28f76867ae2a9281 + .quad 0xba9a0f7b9245e215 + .quad 0xf368612dd98c0dbb + .quad 0x2e84e4cbf220b020 + .quad 0x6ba92fe962d90eda + .quad 0x3e4df9655884e2aa + .quad 0xbd62fbdbdbd465a5 + .quad 0xd7596caa0de9e524 + .quad 0x6e8042ccb2b1b3d7 + + // 2^132 * 8 * G + + .quad 0xf10d3c29ce28ca6e + .quad 0xbad34540fcb6093d + .quad 0xe7426ed7a2ea2d3f + .quad 0x08af9d4e4ff298b9 + .quad 0x1530653616521f7e + .quad 0x660d06b896203dba + .quad 0x2d3989bc545f0879 + .quad 0x4b5303af78ebd7b0 + .quad 0x72f8a6c3bebcbde8 + .quad 0x4f0fca4adc3a8e89 + .quad 0x6fa9d4e8c7bfdf7a + .quad 0x0dcf2d679b624eb7 + + // 2^136 * 1 * G + + .quad 0x3d5947499718289c + .quad 0x12ebf8c524533f26 + .quad 0x0262bfcb14c3ef15 + .quad 0x20b878d577b7518e + .quad 0x753941be5a45f06e + .quad 0xd07caeed6d9c5f65 + .quad 0x11776b9c72ff51b6 + .quad 0x17d2d1d9ef0d4da9 + .quad 0x27f2af18073f3e6a + .quad 0xfd3fe519d7521069 + .quad 0x22e3b72c3ca60022 + .quad 0x72214f63cc65c6a7 + + // 2^136 * 2 * G + + .quad 0xb4e37f405307a693 + .quad 0xaba714d72f336795 + .quad 0xd6fbd0a773761099 + .quad 0x5fdf48c58171cbc9 + .quad 0x1d9db7b9f43b29c9 + .quad 0xd605824a4f518f75 + .quad 0xf2c072bd312f9dc4 + .quad 0x1f24ac855a1545b0 + .quad 0x24d608328e9505aa + .quad 0x4748c1d10c1420ee + .quad 0xc7ffe45c06fb25a2 + .quad 0x00ba739e2ae395e6 + + // 2^136 * 3 * G + + .quad 0x592e98de5c8790d6 + .quad 0xe5bfb7d345c2a2df + .quad 0x115a3b60f9b49922 + .quad 0x03283a3e67ad78f3 + .quad 0xae4426f5ea88bb26 + .quad 0x360679d984973bfb + .quad 0x5c9f030c26694e50 + .quad 0x72297de7d518d226 + .quad 0x48241dc7be0cb939 + .quad 0x32f19b4d8b633080 + .quad 0xd3dfc90d02289308 + .quad 0x05e1296846271945 + + // 2^136 * 4 * G + + .quad 0xba82eeb32d9c495a + .quad 0xceefc8fcf12bb97c + .quad 0xb02dabae93b5d1e0 + .quad 0x39c00c9c13698d9b + .quad 0xadbfbbc8242c4550 + .quad 0xbcc80cecd03081d9 + .quad 0x843566a6f5c8df92 + .quad 0x78cf25d38258ce4c + .quad 0x15ae6b8e31489d68 + .quad 0xaa851cab9c2bf087 + .quad 0xc9a75a97f04efa05 + .quad 0x006b52076b3ff832 + + // 2^136 * 5 * G + + .quad 0x29e0cfe19d95781c + .quad 0xb681df18966310e2 + .quad 0x57df39d370516b39 + .quad 0x4d57e3443bc76122 + .quad 0xf5cb7e16b9ce082d + .quad 0x3407f14c417abc29 + .quad 0xd4b36bce2bf4a7ab + .quad 0x7de2e9561a9f75ce + .quad 0xde70d4f4b6a55ecb + .quad 0x4801527f5d85db99 + .quad 0xdbc9c440d3ee9a81 + .quad 0x6b2a90af1a6029ed + + // 2^136 * 6 * G + + .quad 0x6923f4fc9ae61e97 + .quad 0x5735281de03f5fd1 + .quad 0xa764ae43e6edd12d + .quad 0x5fd8f4e9d12d3e4a + .quad 0x77ebf3245bb2d80a + .quad 0xd8301b472fb9079b + .quad 0xc647e6f24cee7333 + .quad 0x465812c8276c2109 + .quad 0x4d43beb22a1062d9 + .quad 0x7065fb753831dc16 + .quad 0x180d4a7bde2968d7 + .quad 0x05b32c2b1cb16790 + + // 2^136 * 7 * G + + .quad 0xc8c05eccd24da8fd + .quad 0xa1cf1aac05dfef83 + .quad 0xdbbeeff27df9cd61 + .quad 0x3b5556a37b471e99 + .quad 0xf7fca42c7ad58195 + .quad 0x3214286e4333f3cc + .quad 0xb6c29d0d340b979d + .quad 0x31771a48567307e1 + .quad 0x32b0c524e14dd482 + .quad 0xedb351541a2ba4b6 + .quad 0xa3d16048282b5af3 + .quad 0x4fc079d27a7336eb + + // 2^136 * 8 * G + + .quad 0x51c938b089bf2f7f + .quad 0x2497bd6502dfe9a7 + .quad 0xffffc09c7880e453 + .quad 0x124567cecaf98e92 + .quad 0xdc348b440c86c50d + .quad 0x1337cbc9cc94e651 + .quad 0x6422f74d643e3cb9 + .quad 0x241170c2bae3cd08 + .quad 0x3ff9ab860ac473b4 + .quad 0xf0911dee0113e435 + .quad 0x4ae75060ebc6c4af + .quad 0x3f8612966c87000d + + // 2^140 * 1 * G + + .quad 0x0c9c5303f7957be4 + .quad 0xa3c31a20e085c145 + .quad 0xb0721d71d0850050 + .quad 0x0aba390eab0bf2da + .quad 0x529fdffe638c7bf3 + .quad 0xdf2b9e60388b4995 + .quad 0xe027b34f1bad0249 + .quad 0x7bc92fc9b9fa74ed + .quad 0x9f97ef2e801ad9f9 + .quad 0x83697d5479afda3a + .quad 0xe906b3ffbd596b50 + .quad 0x02672b37dd3fb8e0 + + // 2^140 * 2 * G + + .quad 0x48b2ca8b260885e4 + .quad 0xa4286bec82b34c1c + .quad 0x937e1a2617f58f74 + .quad 0x741d1fcbab2ca2a5 + .quad 0xee9ba729398ca7f5 + .quad 0xeb9ca6257a4849db + .quad 0x29eb29ce7ec544e1 + .quad 0x232ca21ef736e2c8 + .quad 0xbf61423d253fcb17 + .quad 0x08803ceafa39eb14 + .quad 0xf18602df9851c7af + .quad 0x0400f3a049e3414b + + // 2^140 * 3 * G + + .quad 0xabce0476ba61c55b + .quad 0x36a3d6d7c4d39716 + .quad 0x6eb259d5e8d82d09 + .quad 0x0c9176e984d756fb + .quad 0x2efba412a06e7b06 + .quad 0x146785452c8d2560 + .quad 0xdf9713ebd67a91c7 + .quad 0x32830ac7157eadf3 + .quad 0x0e782a7ab73769e8 + .quad 0x04a05d7875b18e2c + .quad 0x29525226ebcceae1 + .quad 0x0d794f8383eba820 + + // 2^140 * 4 * G + + .quad 0xff35f5cb9e1516f4 + .quad 0xee805bcf648aae45 + .quad 0xf0d73c2bb93a9ef3 + .quad 0x097b0bf22092a6c2 + .quad 0x7be44ce7a7a2e1ac + .quad 0x411fd93efad1b8b7 + .quad 0x1734a1d70d5f7c9b + .quad 0x0d6592233127db16 + .quad 0xc48bab1521a9d733 + .quad 0xa6c2eaead61abb25 + .quad 0x625c6c1cc6cb4305 + .quad 0x7fc90fea93eb3a67 + + // 2^140 * 5 * G + + .quad 0x0408f1fe1f5c5926 + .quad 0x1a8f2f5e3b258bf4 + .quad 0x40a951a2fdc71669 + .quad 0x6598ee93c98b577e + .quad 0xc527deb59c7cb23d + .quad 0x955391695328404e + .quad 0xd64392817ccf2c7a + .quad 0x6ce97dabf7d8fa11 + .quad 0x25b5a8e50ef7c48f + .quad 0xeb6034116f2ce532 + .quad 0xc5e75173e53de537 + .quad 0x73119fa08c12bb03 + + // 2^140 * 6 * G + + .quad 0xed30129453f1a4cb + .quad 0xbce621c9c8f53787 + .quad 0xfacb2b1338bee7b9 + .quad 0x3025798a9ea8428c + .quad 0x7845b94d21f4774d + .quad 0xbf62f16c7897b727 + .quad 0x671857c03c56522b + .quad 0x3cd6a85295621212 + .quad 0x3fecde923aeca999 + .quad 0xbdaa5b0062e8c12f + .quad 0x67b99dfc96988ade + .quad 0x3f52c02852661036 + + // 2^140 * 7 * G + + .quad 0xffeaa48e2a1351c6 + .quad 0x28624754fa7f53d7 + .quad 0x0b5ba9e57582ddf1 + .quad 0x60c0104ba696ac59 + .quad 0x9258bf99eec416c6 + .quad 0xac8a5017a9d2f671 + .quad 0x629549ab16dea4ab + .quad 0x05d0e85c99091569 + .quad 0x051de020de9cbe97 + .quad 0xfa07fc56b50bcf74 + .quad 0x378cec9f0f11df65 + .quad 0x36853c69ab96de4d + + // 2^140 * 8 * G + + .quad 0x36d9b8de78f39b2d + .quad 0x7f42ed71a847b9ec + .quad 0x241cd1d679bd3fde + .quad 0x6a704fec92fbce6b + .quad 0x4433c0b0fac5e7be + .quad 0x724bae854c08dcbe + .quad 0xf1f24cc446978f9b + .quad 0x4a0aff6d62825fc8 + .quad 0xe917fb9e61095301 + .quad 0xc102df9402a092f8 + .quad 0xbf09e2f5fa66190b + .quad 0x681109bee0dcfe37 + + // 2^144 * 1 * G + + .quad 0x559a0cc9782a0dde + .quad 0x551dcdb2ea718385 + .quad 0x7f62865b31ef238c + .quad 0x504aa7767973613d + .quad 0x9c18fcfa36048d13 + .quad 0x29159db373899ddd + .quad 0xdc9f350b9f92d0aa + .quad 0x26f57eee878a19d4 + .quad 0x0cab2cd55687efb1 + .quad 0x5180d162247af17b + .quad 0x85c15a344f5a2467 + .quad 0x4041943d9dba3069 + + // 2^144 * 2 * G + + .quad 0xc3c0eeba43ebcc96 + .quad 0x8d749c9c26ea9caf + .quad 0xd9fa95ee1c77ccc6 + .quad 0x1420a1d97684340f + .quad 0x4b217743a26caadd + .quad 0x47a6b424648ab7ce + .quad 0xcb1d4f7a03fbc9e3 + .quad 0x12d931429800d019 + .quad 0x00c67799d337594f + .quad 0x5e3c5140b23aa47b + .quad 0x44182854e35ff395 + .quad 0x1b4f92314359a012 + + // 2^144 * 3 * G + + .quad 0x3e5c109d89150951 + .quad 0x39cefa912de9696a + .quad 0x20eae43f975f3020 + .quad 0x239b572a7f132dae + .quad 0x33cf3030a49866b1 + .quad 0x251f73d2215f4859 + .quad 0xab82aa4051def4f6 + .quad 0x5ff191d56f9a23f6 + .quad 0x819ed433ac2d9068 + .quad 0x2883ab795fc98523 + .quad 0xef4572805593eb3d + .quad 0x020c526a758f36cb + + // 2^144 * 4 * G + + .quad 0x779834f89ed8dbbc + .quad 0xc8f2aaf9dc7ca46c + .quad 0xa9524cdca3e1b074 + .quad 0x02aacc4615313877 + .quad 0xe931ef59f042cc89 + .quad 0x2c589c9d8e124bb6 + .quad 0xadc8e18aaec75997 + .quad 0x452cfe0a5602c50c + .quad 0x86a0f7a0647877df + .quad 0xbbc464270e607c9f + .quad 0xab17ea25f1fb11c9 + .quad 0x4cfb7d7b304b877b + + // 2^144 * 5 * G + + .quad 0x72b43d6cb89b75fe + .quad 0x54c694d99c6adc80 + .quad 0xb8c3aa373ee34c9f + .quad 0x14b4622b39075364 + .quad 0xe28699c29789ef12 + .quad 0x2b6ecd71df57190d + .quad 0xc343c857ecc970d0 + .quad 0x5b1d4cbc434d3ac5 + .quad 0xb6fb2615cc0a9f26 + .quad 0x3a4f0e2bb88dcce5 + .quad 0x1301498b3369a705 + .quad 0x2f98f71258592dd1 + + // 2^144 * 6 * G + + .quad 0x0c94a74cb50f9e56 + .quad 0x5b1ff4a98e8e1320 + .quad 0x9a2acc2182300f67 + .quad 0x3a6ae249d806aaf9 + .quad 0x2e12ae444f54a701 + .quad 0xfcfe3ef0a9cbd7de + .quad 0xcebf890d75835de0 + .quad 0x1d8062e9e7614554 + .quad 0x657ada85a9907c5a + .quad 0x1a0ea8b591b90f62 + .quad 0x8d0e1dfbdf34b4e9 + .quad 0x298b8ce8aef25ff3 + + // 2^144 * 7 * G + + .quad 0x2a927953eff70cb2 + .quad 0x4b89c92a79157076 + .quad 0x9418457a30a7cf6a + .quad 0x34b8a8404d5ce485 + .quad 0x837a72ea0a2165de + .quad 0x3fab07b40bcf79f6 + .quad 0x521636c77738ae70 + .quad 0x6ba6271803a7d7dc + .quad 0xc26eecb583693335 + .quad 0xd5a813df63b5fefd + .quad 0xa293aa9aa4b22573 + .quad 0x71d62bdd465e1c6a + + // 2^144 * 8 * G + + .quad 0x6533cc28d378df80 + .quad 0xf6db43790a0fa4b4 + .quad 0xe3645ff9f701da5a + .quad 0x74d5f317f3172ba4 + .quad 0xcd2db5dab1f75ef5 + .quad 0xd77f95cf16b065f5 + .quad 0x14571fea3f49f085 + .quad 0x1c333621262b2b3d + .quad 0xa86fe55467d9ca81 + .quad 0x398b7c752b298c37 + .quad 0xda6d0892e3ac623b + .quad 0x4aebcc4547e9d98c + + // 2^148 * 1 * G + + .quad 0x53175a7205d21a77 + .quad 0xb0c04422d3b934d4 + .quad 0xadd9f24bdd5deadc + .quad 0x074f46e69f10ff8c + .quad 0x0de9b204a059a445 + .quad 0xe15cb4aa4b17ad0f + .quad 0xe1bbec521f79c557 + .quad 0x2633f1b9d071081b + .quad 0xc1fb4177018b9910 + .quad 0xa6ea20dc6c0fe140 + .quad 0xd661f3e74354c6ff + .quad 0x5ecb72e6f1a3407a + + // 2^148 * 2 * G + + .quad 0xa515a31b2259fb4e + .quad 0x0960f3972bcac52f + .quad 0xedb52fec8d3454cb + .quad 0x382e2720c476c019 + .quad 0xfeeae106e8e86997 + .quad 0x9863337f98d09383 + .quad 0x9470480eaa06ebef + .quad 0x038b6898d4c5c2d0 + .quad 0xf391c51d8ace50a6 + .quad 0x3142d0b9ae2d2948 + .quad 0xdb4d5a1a7f24ca80 + .quad 0x21aeba8b59250ea8 + + // 2^148 * 3 * G + + .quad 0x24f13b34cf405530 + .quad 0x3c44ea4a43088af7 + .quad 0x5dd5c5170006a482 + .quad 0x118eb8f8890b086d + .quad 0x53853600f0087f23 + .quad 0x4c461879da7d5784 + .quad 0x6af303deb41f6860 + .quad 0x0a3c16c5c27c18ed + .quad 0x17e49c17cc947f3d + .quad 0xccc6eda6aac1d27b + .quad 0xdf6092ceb0f08e56 + .quad 0x4909b3e22c67c36b + + // 2^148 * 4 * G + + .quad 0x9c9c85ea63fe2e89 + .quad 0xbe1baf910e9412ec + .quad 0x8f7baa8a86fbfe7b + .quad 0x0fb17f9fef968b6c + .quad 0x59a16676706ff64e + .quad 0x10b953dd0d86a53d + .quad 0x5848e1e6ce5c0b96 + .quad 0x2d8b78e712780c68 + .quad 0x79d5c62eafc3902b + .quad 0x773a215289e80728 + .quad 0xc38ae640e10120b9 + .quad 0x09ae23717b2b1a6d + + // 2^148 * 5 * G + + .quad 0xbb6a192a4e4d083c + .quad 0x34ace0630029e192 + .quad 0x98245a59aafabaeb + .quad 0x6d9c8a9ada97faac + .quad 0x10ab8fa1ad32b1d0 + .quad 0xe9aced1be2778b24 + .quad 0xa8856bc0373de90f + .quad 0x66f35ddddda53996 + .quad 0xd27d9afb24997323 + .quad 0x1bb7e07ef6f01d2e + .quad 0x2ba7472df52ecc7f + .quad 0x03019b4f646f9dc8 + + // 2^148 * 6 * G + + .quad 0x04a186b5565345cd + .quad 0xeee76610bcc4116a + .quad 0x689c73b478fb2a45 + .quad 0x387dcbff65697512 + .quad 0xaf09b214e6b3dc6b + .quad 0x3f7573b5ad7d2f65 + .quad 0xd019d988100a23b0 + .quad 0x392b63a58b5c35f7 + .quad 0x4093addc9c07c205 + .quad 0xc565be15f532c37e + .quad 0x63dbecfd1583402a + .quad 0x61722b4aef2e032e + + // 2^148 * 7 * G + + .quad 0x0012aafeecbd47af + .quad 0x55a266fb1cd46309 + .quad 0xf203eb680967c72c + .quad 0x39633944ca3c1429 + .quad 0xd6b07a5581cb0e3c + .quad 0x290ff006d9444969 + .quad 0x08680b6a16dcda1f + .quad 0x5568d2b75a06de59 + .quad 0x8d0cb88c1b37cfe1 + .quad 0x05b6a5a3053818f3 + .quad 0xf2e9bc04b787d959 + .quad 0x6beba1249add7f64 + + // 2^148 * 8 * G + + .quad 0x1d06005ca5b1b143 + .quad 0x6d4c6bb87fd1cda2 + .quad 0x6ef5967653fcffe7 + .quad 0x097c29e8c1ce1ea5 + .quad 0x5c3cecb943f5a53b + .quad 0x9cc9a61d06c08df2 + .quad 0xcfba639a85895447 + .quad 0x5a845ae80df09fd5 + .quad 0x4ce97dbe5deb94ca + .quad 0x38d0a4388c709c48 + .quad 0xc43eced4a169d097 + .quad 0x0a1249fff7e587c3 + + // 2^152 * 1 * G + + .quad 0x12f0071b276d01c9 + .quad 0xe7b8bac586c48c70 + .quad 0x5308129b71d6fba9 + .quad 0x5d88fbf95a3db792 + .quad 0x0b408d9e7354b610 + .quad 0x806b32535ba85b6e + .quad 0xdbe63a034a58a207 + .quad 0x173bd9ddc9a1df2c + .quad 0x2b500f1efe5872df + .quad 0x58d6582ed43918c1 + .quad 0xe6ed278ec9673ae0 + .quad 0x06e1cd13b19ea319 + + // 2^152 * 2 * G + + .quad 0x40d0ad516f166f23 + .quad 0x118e32931fab6abe + .quad 0x3fe35e14a04d088e + .quad 0x3080603526e16266 + .quad 0x472baf629e5b0353 + .quad 0x3baa0b90278d0447 + .quad 0x0c785f469643bf27 + .quad 0x7f3a6a1a8d837b13 + .quad 0xf7e644395d3d800b + .quad 0x95a8d555c901edf6 + .quad 0x68cd7830592c6339 + .quad 0x30d0fded2e51307e + + // 2^152 * 3 * G + + .quad 0xe0594d1af21233b3 + .quad 0x1bdbe78ef0cc4d9c + .quad 0x6965187f8f499a77 + .quad 0x0a9214202c099868 + .quad 0x9cb4971e68b84750 + .quad 0xa09572296664bbcf + .quad 0x5c8de72672fa412b + .quad 0x4615084351c589d9 + .quad 0xbc9019c0aeb9a02e + .quad 0x55c7110d16034cae + .quad 0x0e6df501659932ec + .quad 0x3bca0d2895ca5dfe + + // 2^152 * 4 * G + + .quad 0x40f031bc3c5d62a4 + .quad 0x19fc8b3ecff07a60 + .quad 0x98183da2130fb545 + .quad 0x5631deddae8f13cd + .quad 0x9c688eb69ecc01bf + .quad 0xf0bc83ada644896f + .quad 0xca2d955f5f7a9fe2 + .quad 0x4ea8b4038df28241 + .quad 0x2aed460af1cad202 + .quad 0x46305305a48cee83 + .quad 0x9121774549f11a5f + .quad 0x24ce0930542ca463 + + // 2^152 * 5 * G + + .quad 0x1fe890f5fd06c106 + .quad 0xb5c468355d8810f2 + .quad 0x827808fe6e8caf3e + .quad 0x41d4e3c28a06d74b + .quad 0x3fcfa155fdf30b85 + .quad 0xd2f7168e36372ea4 + .quad 0xb2e064de6492f844 + .quad 0x549928a7324f4280 + .quad 0xf26e32a763ee1a2e + .quad 0xae91e4b7d25ffdea + .quad 0xbc3bd33bd17f4d69 + .quad 0x491b66dec0dcff6a + + // 2^152 * 6 * G + + .quad 0x98f5b13dc7ea32a7 + .quad 0xe3d5f8cc7e16db98 + .quad 0xac0abf52cbf8d947 + .quad 0x08f338d0c85ee4ac + .quad 0x75f04a8ed0da64a1 + .quad 0xed222caf67e2284b + .quad 0x8234a3791f7b7ba4 + .quad 0x4cf6b8b0b7018b67 + .quad 0xc383a821991a73bd + .quad 0xab27bc01df320c7a + .quad 0xc13d331b84777063 + .quad 0x530d4a82eb078a99 + + // 2^152 * 7 * G + + .quad 0x004c3630e1f94825 + .quad 0x7e2d78268cab535a + .quad 0xc7482323cc84ff8b + .quad 0x65ea753f101770b9 + .quad 0x6d6973456c9abf9e + .quad 0x257fb2fc4900a880 + .quad 0x2bacf412c8cfb850 + .quad 0x0db3e7e00cbfbd5b + .quad 0x3d66fc3ee2096363 + .quad 0x81d62c7f61b5cb6b + .quad 0x0fbe044213443b1a + .quad 0x02a4ec1921e1a1db + + // 2^152 * 8 * G + + .quad 0x5ce6259a3b24b8a2 + .quad 0xb8577acc45afa0b8 + .quad 0xcccbe6e88ba07037 + .quad 0x3d143c51127809bf + .quad 0xf5c86162f1cf795f + .quad 0x118c861926ee57f2 + .quad 0x172124851c063578 + .quad 0x36d12b5dec067fcf + .quad 0x126d279179154557 + .quad 0xd5e48f5cfc783a0a + .quad 0x36bdb6e8df179bac + .quad 0x2ef517885ba82859 + + // 2^156 * 1 * G + + .quad 0x88bd438cd11e0d4a + .quad 0x30cb610d43ccf308 + .quad 0xe09a0e3791937bcc + .quad 0x4559135b25b1720c + .quad 0x1ea436837c6da1e9 + .quad 0xf9c189af1fb9bdbe + .quad 0x303001fcce5dd155 + .quad 0x28a7c99ebc57be52 + .quad 0xb8fd9399e8d19e9d + .quad 0x908191cb962423ff + .quad 0xb2b948d747c742a3 + .quad 0x37f33226d7fb44c4 + + // 2^156 * 2 * G + + .quad 0x0dae8767b55f6e08 + .quad 0x4a43b3b35b203a02 + .quad 0xe3725a6e80af8c79 + .quad 0x0f7a7fd1705fa7a3 + .quad 0x33912553c821b11d + .quad 0x66ed42c241e301df + .quad 0x066fcc11104222fd + .quad 0x307a3b41c192168f + .quad 0x8eeb5d076eb55ce0 + .quad 0x2fc536bfaa0d925a + .quad 0xbe81830fdcb6c6e8 + .quad 0x556c7045827baf52 + + // 2^156 * 3 * G + + .quad 0x8e2b517302e9d8b7 + .quad 0xe3e52269248714e8 + .quad 0xbd4fbd774ca960b5 + .quad 0x6f4b4199c5ecada9 + .quad 0xb94b90022bf44406 + .quad 0xabd4237eff90b534 + .quad 0x7600a960faf86d3a + .quad 0x2f45abdac2322ee3 + .quad 0x61af4912c8ef8a6a + .quad 0xe58fa4fe43fb6e5e + .quad 0xb5afcc5d6fd427cf + .quad 0x6a5393281e1e11eb + + // 2^156 * 4 * G + + .quad 0xf3da5139a5d1ee89 + .quad 0x8145457cff936988 + .quad 0x3f622fed00e188c4 + .quad 0x0f513815db8b5a3d + .quad 0x0fff04fe149443cf + .quad 0x53cac6d9865cddd7 + .quad 0x31385b03531ed1b7 + .quad 0x5846a27cacd1039d + .quad 0x4ff5cdac1eb08717 + .quad 0x67e8b29590f2e9bc + .quad 0x44093b5e237afa99 + .quad 0x0d414bed8708b8b2 + + // 2^156 * 5 * G + + .quad 0xcfb68265fd0e75f6 + .quad 0xe45b3e28bb90e707 + .quad 0x7242a8de9ff92c7a + .quad 0x685b3201933202dd + .quad 0x81886a92294ac9e8 + .quad 0x23162b45d55547be + .quad 0x94cfbc4403715983 + .quad 0x50eb8fdb134bc401 + .quad 0xc0b73ec6d6b330cd + .quad 0x84e44807132faff1 + .quad 0x732b7352c4a5dee1 + .quad 0x5d7c7cf1aa7cd2d2 + + // 2^156 * 6 * G + + .quad 0xaf3b46bf7a4aafa2 + .quad 0xb78705ec4d40d411 + .quad 0x114f0c6aca7c15e3 + .quad 0x3f364faaa9489d4d + .quad 0x33d1013e9b73a562 + .quad 0x925cef5748ec26e1 + .quad 0xa7fce614dd468058 + .quad 0x78b0fad41e9aa438 + .quad 0xbf56a431ed05b488 + .quad 0xa533e66c9c495c7e + .quad 0xe8652baf87f3651a + .quad 0x0241800059d66c33 + + // 2^156 * 7 * G + + .quad 0xceb077fea37a5be4 + .quad 0xdb642f02e5a5eeb7 + .quad 0xc2e6d0c5471270b8 + .quad 0x4771b65538e4529c + .quad 0x28350c7dcf38ea01 + .quad 0x7c6cdbc0b2917ab6 + .quad 0xace7cfbe857082f7 + .quad 0x4d2845aba2d9a1e0 + .quad 0xbb537fe0447070de + .quad 0xcba744436dd557df + .quad 0xd3b5a3473600dbcb + .quad 0x4aeabbe6f9ffd7f8 + + // 2^156 * 8 * G + + .quad 0x4630119e40d8f78c + .quad 0xa01a9bc53c710e11 + .quad 0x486d2b258910dd79 + .quad 0x1e6c47b3db0324e5 + .quad 0x6a2134bcc4a9c8f2 + .quad 0xfbf8fd1c8ace2e37 + .quad 0x000ae3049911a0ba + .quad 0x046e3a616bc89b9e + .quad 0x14e65442f03906be + .quad 0x4a019d54e362be2a + .quad 0x68ccdfec8dc230c7 + .quad 0x7cfb7e3faf6b861c + + // 2^160 * 1 * G + + .quad 0x4637974e8c58aedc + .quad 0xb9ef22fbabf041a4 + .quad 0xe185d956e980718a + .quad 0x2f1b78fab143a8a6 + .quad 0x96eebffb305b2f51 + .quad 0xd3f938ad889596b8 + .quad 0xf0f52dc746d5dd25 + .quad 0x57968290bb3a0095 + .quad 0xf71ab8430a20e101 + .quad 0xf393658d24f0ec47 + .quad 0xcf7509a86ee2eed1 + .quad 0x7dc43e35dc2aa3e1 + + // 2^160 * 2 * G + + .quad 0x85966665887dd9c3 + .quad 0xc90f9b314bb05355 + .quad 0xc6e08df8ef2079b1 + .quad 0x7ef72016758cc12f + .quad 0x5a782a5c273e9718 + .quad 0x3576c6995e4efd94 + .quad 0x0f2ed8051f237d3e + .quad 0x044fb81d82d50a99 + .quad 0xc1df18c5a907e3d9 + .quad 0x57b3371dce4c6359 + .quad 0xca704534b201bb49 + .quad 0x7f79823f9c30dd2e + + // 2^160 * 3 * G + + .quad 0x8334d239a3b513e8 + .quad 0xc13670d4b91fa8d8 + .quad 0x12b54136f590bd33 + .quad 0x0a4e0373d784d9b4 + .quad 0x6a9c1ff068f587ba + .quad 0x0827894e0050c8de + .quad 0x3cbf99557ded5be7 + .quad 0x64a9b0431c06d6f0 + .quad 0x2eb3d6a15b7d2919 + .quad 0xb0b4f6a0d53a8235 + .quad 0x7156ce4389a45d47 + .quad 0x071a7d0ace18346c + + // 2^160 * 4 * G + + .quad 0xd3072daac887ba0b + .quad 0x01262905bfa562ee + .quad 0xcf543002c0ef768b + .quad 0x2c3bcc7146ea7e9c + .quad 0xcc0c355220e14431 + .quad 0x0d65950709b15141 + .quad 0x9af5621b209d5f36 + .quad 0x7c69bcf7617755d3 + .quad 0x07f0d7eb04e8295f + .quad 0x10db18252f50f37d + .quad 0xe951a9a3171798d7 + .quad 0x6f5a9a7322aca51d + + // 2^160 * 5 * G + + .quad 0x8ba1000c2f41c6c5 + .quad 0xc49f79c10cfefb9b + .quad 0x4efa47703cc51c9f + .quad 0x494e21a2e147afca + .quad 0xe729d4eba3d944be + .quad 0x8d9e09408078af9e + .quad 0x4525567a47869c03 + .quad 0x02ab9680ee8d3b24 + .quad 0xefa48a85dde50d9a + .quad 0x219a224e0fb9a249 + .quad 0xfa091f1dd91ef6d9 + .quad 0x6b5d76cbea46bb34 + + // 2^160 * 6 * G + + .quad 0x8857556cec0cd994 + .quad 0x6472dc6f5cd01dba + .quad 0xaf0169148f42b477 + .quad 0x0ae333f685277354 + .quad 0xe0f941171e782522 + .quad 0xf1e6ae74036936d3 + .quad 0x408b3ea2d0fcc746 + .quad 0x16fb869c03dd313e + .quad 0x288e199733b60962 + .quad 0x24fc72b4d8abe133 + .quad 0x4811f7ed0991d03e + .quad 0x3f81e38b8f70d075 + + // 2^160 * 7 * G + + .quad 0x7f910fcc7ed9affe + .quad 0x545cb8a12465874b + .quad 0xa8397ed24b0c4704 + .quad 0x50510fc104f50993 + .quad 0x0adb7f355f17c824 + .quad 0x74b923c3d74299a4 + .quad 0xd57c3e8bcbf8eaf7 + .quad 0x0ad3e2d34cdedc3d + .quad 0x6f0c0fc5336e249d + .quad 0x745ede19c331cfd9 + .quad 0xf2d6fd0009eefe1c + .quad 0x127c158bf0fa1ebe + + // 2^160 * 8 * G + + .quad 0xf6197c422e9879a2 + .quad 0xa44addd452ca3647 + .quad 0x9b413fc14b4eaccb + .quad 0x354ef87d07ef4f68 + .quad 0xdea28fc4ae51b974 + .quad 0x1d9973d3744dfe96 + .quad 0x6240680b873848a8 + .quad 0x4ed82479d167df95 + .quad 0xfee3b52260c5d975 + .quad 0x50352efceb41b0b8 + .quad 0x8808ac30a9f6653c + .quad 0x302d92d20539236d + + // 2^164 * 1 * G + + .quad 0x4c59023fcb3efb7c + .quad 0x6c2fcb99c63c2a94 + .quad 0xba4190e2c3c7e084 + .quad 0x0e545daea51874d9 + .quad 0x957b8b8b0df53c30 + .quad 0x2a1c770a8e60f098 + .quad 0xbbc7a670345796de + .quad 0x22a48f9a90c99bc9 + .quad 0x6b7dc0dc8d3fac58 + .quad 0x5497cd6ce6e42bfd + .quad 0x542f7d1bf400d305 + .quad 0x4159f47f048d9136 + + // 2^164 * 2 * G + + .quad 0x20ad660839e31e32 + .quad 0xf81e1bd58405be50 + .quad 0xf8064056f4dabc69 + .quad 0x14d23dd4ce71b975 + .quad 0x748515a8bbd24839 + .quad 0x77128347afb02b55 + .quad 0x50ba2ac649a2a17f + .quad 0x060525513ad730f1 + .quad 0xf2398e098aa27f82 + .quad 0x6d7982bb89a1b024 + .quad 0xfa694084214dd24c + .quad 0x71ab966fa32301c3 + + // 2^164 * 3 * G + + .quad 0x2dcbd8e34ded02fc + .quad 0x1151f3ec596f22aa + .quad 0xbca255434e0328da + .quad 0x35768fbe92411b22 + .quad 0xb1088a0702809955 + .quad 0x43b273ea0b43c391 + .quad 0xca9b67aefe0686ed + .quad 0x605eecbf8335f4ed + .quad 0x83200a656c340431 + .quad 0x9fcd71678ee59c2f + .quad 0x75d4613f71300f8a + .quad 0x7a912faf60f542f9 + + // 2^164 * 4 * G + + .quad 0xb204585e5edc1a43 + .quad 0x9f0e16ee5897c73c + .quad 0x5b82c0ae4e70483c + .quad 0x624a170e2bddf9be + .quad 0x253f4f8dfa2d5597 + .quad 0x25e49c405477130c + .quad 0x00c052e5996b1102 + .quad 0x33cb966e33bb6c4a + .quad 0x597028047f116909 + .quad 0x828ac41c1e564467 + .quad 0x70417dbde6217387 + .quad 0x721627aefbac4384 + + // 2^164 * 5 * G + + .quad 0x97d03bc38736add5 + .quad 0x2f1422afc532b130 + .quad 0x3aa68a057101bbc4 + .quad 0x4c946cf7e74f9fa7 + .quad 0xfd3097bc410b2f22 + .quad 0xf1a05da7b5cfa844 + .quad 0x61289a1def57ca74 + .quad 0x245ea199bb821902 + .quad 0xaedca66978d477f8 + .quad 0x1898ba3c29117fe1 + .quad 0xcf73f983720cbd58 + .quad 0x67da12e6b8b56351 + + // 2^164 * 6 * G + + .quad 0x7067e187b4bd6e07 + .quad 0x6e8f0203c7d1fe74 + .quad 0x93c6aa2f38c85a30 + .quad 0x76297d1f3d75a78a + .quad 0x2b7ef3d38ec8308c + .quad 0x828fd7ec71eb94ab + .quad 0x807c3b36c5062abd + .quad 0x0cb64cb831a94141 + .quad 0x3030fc33534c6378 + .quad 0xb9635c5ce541e861 + .quad 0x15d9a9bed9b2c728 + .quad 0x49233ea3f3775dcb + + // 2^164 * 7 * G + + .quad 0x629398fa8dbffc3a + .quad 0xe12fe52dd54db455 + .quad 0xf3be11dfdaf25295 + .quad 0x628b140dce5e7b51 + .quad 0x7b3985fe1c9f249b + .quad 0x4fd6b2d5a1233293 + .quad 0xceb345941adf4d62 + .quad 0x6987ff6f542de50c + .quad 0x47e241428f83753c + .quad 0x6317bebc866af997 + .quad 0xdabb5b433d1a9829 + .quad 0x074d8d245287fb2d + + // 2^164 * 8 * G + + .quad 0x8337d9cd440bfc31 + .quad 0x729d2ca1af318fd7 + .quad 0xa040a4a4772c2070 + .quad 0x46002ef03a7349be + .quad 0x481875c6c0e31488 + .quad 0x219429b2e22034b4 + .quad 0x7223c98a31283b65 + .quad 0x3420d60b342277f9 + .quad 0xfaa23adeaffe65f7 + .quad 0x78261ed45be0764c + .quad 0x441c0a1e2f164403 + .quad 0x5aea8e567a87d395 + + // 2^168 * 1 * G + + .quad 0x7813c1a2bca4283d + .quad 0xed62f091a1863dd9 + .quad 0xaec7bcb8c268fa86 + .quad 0x10e5d3b76f1cae4c + .quad 0x2dbc6fb6e4e0f177 + .quad 0x04e1bf29a4bd6a93 + .quad 0x5e1966d4787af6e8 + .quad 0x0edc5f5eb426d060 + .quad 0x5453bfd653da8e67 + .quad 0xe9dc1eec24a9f641 + .quad 0xbf87263b03578a23 + .quad 0x45b46c51361cba72 + + // 2^168 * 2 * G + + .quad 0xa9402abf314f7fa1 + .quad 0xe257f1dc8e8cf450 + .quad 0x1dbbd54b23a8be84 + .quad 0x2177bfa36dcb713b + .quad 0xce9d4ddd8a7fe3e4 + .quad 0xab13645676620e30 + .quad 0x4b594f7bb30e9958 + .quad 0x5c1c0aef321229df + .quad 0x37081bbcfa79db8f + .quad 0x6048811ec25f59b3 + .quad 0x087a76659c832487 + .quad 0x4ae619387d8ab5bb + + // 2^168 * 3 * G + + .quad 0x8ddbf6aa5344a32e + .quad 0x7d88eab4b41b4078 + .quad 0x5eb0eb974a130d60 + .quad 0x1a00d91b17bf3e03 + .quad 0x61117e44985bfb83 + .quad 0xfce0462a71963136 + .quad 0x83ac3448d425904b + .quad 0x75685abe5ba43d64 + .quad 0x6e960933eb61f2b2 + .quad 0x543d0fa8c9ff4952 + .quad 0xdf7275107af66569 + .quad 0x135529b623b0e6aa + + // 2^168 * 4 * G + + .quad 0x18f0dbd7add1d518 + .quad 0x979f7888cfc11f11 + .quad 0x8732e1f07114759b + .quad 0x79b5b81a65ca3a01 + .quad 0xf5c716bce22e83fe + .quad 0xb42beb19e80985c1 + .quad 0xec9da63714254aae + .quad 0x5972ea051590a613 + .quad 0x0fd4ac20dc8f7811 + .quad 0x9a9ad294ac4d4fa8 + .quad 0xc01b2d64b3360434 + .quad 0x4f7e9c95905f3bdb + + // 2^168 * 5 * G + + .quad 0x62674bbc5781302e + .quad 0xd8520f3989addc0f + .quad 0x8c2999ae53fbd9c6 + .quad 0x31993ad92e638e4c + .quad 0x71c8443d355299fe + .quad 0x8bcd3b1cdbebead7 + .quad 0x8092499ef1a49466 + .quad 0x1942eec4a144adc8 + .quad 0x7dac5319ae234992 + .quad 0x2c1b3d910cea3e92 + .quad 0x553ce494253c1122 + .quad 0x2a0a65314ef9ca75 + + // 2^168 * 6 * G + + .quad 0x2db7937ff7f927c2 + .quad 0xdb741f0617d0a635 + .quad 0x5982f3a21155af76 + .quad 0x4cf6e218647c2ded + .quad 0xcf361acd3c1c793a + .quad 0x2f9ebcac5a35bc3b + .quad 0x60e860e9a8cda6ab + .quad 0x055dc39b6dea1a13 + .quad 0xb119227cc28d5bb6 + .quad 0x07e24ebc774dffab + .quad 0xa83c78cee4a32c89 + .quad 0x121a307710aa24b6 + + // 2^168 * 7 * G + + .quad 0xe4db5d5e9f034a97 + .quad 0xe153fc093034bc2d + .quad 0x460546919551d3b1 + .quad 0x333fc76c7a40e52d + .quad 0xd659713ec77483c9 + .quad 0x88bfe077b82b96af + .quad 0x289e28231097bcd3 + .quad 0x527bb94a6ced3a9b + .quad 0x563d992a995b482e + .quad 0x3405d07c6e383801 + .quad 0x485035de2f64d8e5 + .quad 0x6b89069b20a7a9f7 + + // 2^168 * 8 * G + + .quad 0x812aa0416270220d + .quad 0x995a89faf9245b4e + .quad 0xffadc4ce5072ef05 + .quad 0x23bc2103aa73eb73 + .quad 0x4082fa8cb5c7db77 + .quad 0x068686f8c734c155 + .quad 0x29e6c8d9f6e7a57e + .quad 0x0473d308a7639bcf + .quad 0xcaee792603589e05 + .quad 0x2b4b421246dcc492 + .quad 0x02a1ef74e601a94f + .quad 0x102f73bfde04341a + + // 2^172 * 1 * G + + .quad 0xb5a2d50c7ec20d3e + .quad 0xc64bdd6ea0c97263 + .quad 0x56e89052c1ff734d + .quad 0x4929c6f72b2ffaba + .quad 0x358ecba293a36247 + .quad 0xaf8f9862b268fd65 + .quad 0x412f7e9968a01c89 + .quad 0x5786f312cd754524 + .quad 0x337788ffca14032c + .quad 0xf3921028447f1ee3 + .quad 0x8b14071f231bccad + .quad 0x4c817b4bf2344783 + + // 2^172 * 2 * G + + .quad 0x0ff853852871b96e + .quad 0xe13e9fab60c3f1bb + .quad 0xeefd595325344402 + .quad 0x0a37c37075b7744b + .quad 0x413ba057a40b4484 + .quad 0xba4c2e1a4f5f6a43 + .quad 0x614ba0a5aee1d61c + .quad 0x78a1531a8b05dc53 + .quad 0x6cbdf1703ad0562b + .quad 0x8ecf4830c92521a3 + .quad 0xdaebd303fd8424e7 + .quad 0x72ad82a42e5ec56f + + // 2^172 * 3 * G + + .quad 0x3f9e8e35bafb65f6 + .quad 0x39d69ec8f27293a1 + .quad 0x6cb8cd958cf6a3d0 + .quad 0x1734778173adae6d + .quad 0xc368939167024bc3 + .quad 0x8e69d16d49502fda + .quad 0xfcf2ec3ce45f4b29 + .quad 0x065f669ea3b4cbc4 + .quad 0x8a00aec75532db4d + .quad 0xb869a4e443e31bb1 + .quad 0x4a0f8552d3a7f515 + .quad 0x19adeb7c303d7c08 + + // 2^172 * 4 * G + + .quad 0xc720cb6153ead9a3 + .quad 0x55b2c97f512b636e + .quad 0xb1e35b5fd40290b1 + .quad 0x2fd9ccf13b530ee2 + .quad 0x9d05ba7d43c31794 + .quad 0x2470c8ff93322526 + .quad 0x8323dec816197438 + .quad 0x2852709881569b53 + .quad 0x07bd475b47f796b8 + .quad 0xd2c7b013542c8f54 + .quad 0x2dbd23f43b24f87e + .quad 0x6551afd77b0901d6 + + // 2^172 * 5 * G + + .quad 0x4546baaf54aac27f + .quad 0xf6f66fecb2a45a28 + .quad 0x582d1b5b562bcfe8 + .quad 0x44b123f3920f785f + .quad 0x68a24ce3a1d5c9ac + .quad 0xbb77a33d10ff6461 + .quad 0x0f86ce4425d3166e + .quad 0x56507c0950b9623b + .quad 0x1206f0b7d1713e63 + .quad 0x353fe3d915bafc74 + .quad 0x194ceb970ad9d94d + .quad 0x62fadd7cf9d03ad3 + + // 2^172 * 6 * G + + .quad 0xc6b5967b5598a074 + .quad 0x5efe91ce8e493e25 + .quad 0xd4b72c4549280888 + .quad 0x20ef1149a26740c2 + .quad 0x3cd7bc61e7ce4594 + .quad 0xcd6b35a9b7dd267e + .quad 0xa080abc84366ef27 + .quad 0x6ec7c46f59c79711 + .quad 0x2f07ad636f09a8a2 + .quad 0x8697e6ce24205e7d + .quad 0xc0aefc05ee35a139 + .quad 0x15e80958b5f9d897 + + // 2^172 * 7 * G + + .quad 0x25a5ef7d0c3e235b + .quad 0x6c39c17fbe134ee7 + .quad 0xc774e1342dc5c327 + .quad 0x021354b892021f39 + .quad 0x4dd1ed355bb061c4 + .quad 0x42dc0cef941c0700 + .quad 0x61305dc1fd86340e + .quad 0x56b2cc930e55a443 + .quad 0x1df79da6a6bfc5a2 + .quad 0x02f3a2749fde4369 + .quad 0xb323d9f2cda390a7 + .quad 0x7be0847b8774d363 + + // 2^172 * 8 * G + + .quad 0x8c99cc5a8b3f55c3 + .quad 0x0611d7253fded2a0 + .quad 0xed2995ff36b70a36 + .quad 0x1f699a54d78a2619 + .quad 0x1466f5af5307fa11 + .quad 0x817fcc7ded6c0af2 + .quad 0x0a6de44ec3a4a3fb + .quad 0x74071475bc927d0b + .quad 0xe77292f373e7ea8a + .quad 0x296537d2cb045a31 + .quad 0x1bd0653ed3274fde + .quad 0x2f9a2c4476bd2966 + + // 2^176 * 1 * G + + .quad 0xeb18b9ab7f5745c6 + .quad 0x023a8aee5787c690 + .quad 0xb72712da2df7afa9 + .quad 0x36597d25ea5c013d + .quad 0xa2b4dae0b5511c9a + .quad 0x7ac860292bffff06 + .quad 0x981f375df5504234 + .quad 0x3f6bd725da4ea12d + .quad 0x734d8d7b106058ac + .quad 0xd940579e6fc6905f + .quad 0x6466f8f99202932d + .quad 0x7b7ecc19da60d6d0 + + // 2^176 * 2 * G + + .quad 0x78c2373c695c690d + .quad 0xdd252e660642906e + .quad 0x951d44444ae12bd2 + .quad 0x4235ad7601743956 + .quad 0x6dae4a51a77cfa9b + .quad 0x82263654e7a38650 + .quad 0x09bbffcd8f2d82db + .quad 0x03bedc661bf5caba + .quad 0x6258cb0d078975f5 + .quad 0x492942549189f298 + .quad 0xa0cab423e2e36ee4 + .quad 0x0e7ce2b0cdf066a1 + + // 2^176 * 3 * G + + .quad 0xc494643ac48c85a3 + .quad 0xfd361df43c6139ad + .quad 0x09db17dd3ae94d48 + .quad 0x666e0a5d8fb4674a + .quad 0xfea6fedfd94b70f9 + .quad 0xf130c051c1fcba2d + .quad 0x4882d47e7f2fab89 + .quad 0x615256138aeceeb5 + .quad 0x2abbf64e4870cb0d + .quad 0xcd65bcf0aa458b6b + .quad 0x9abe4eba75e8985d + .quad 0x7f0bc810d514dee4 + + // 2^176 * 4 * G + + .quad 0xb9006ba426f4136f + .quad 0x8d67369e57e03035 + .quad 0xcbc8dfd94f463c28 + .quad 0x0d1f8dbcf8eedbf5 + .quad 0x83ac9dad737213a0 + .quad 0x9ff6f8ba2ef72e98 + .quad 0x311e2edd43ec6957 + .quad 0x1d3a907ddec5ab75 + .quad 0xba1693313ed081dc + .quad 0x29329fad851b3480 + .quad 0x0128013c030321cb + .quad 0x00011b44a31bfde3 + + // 2^176 * 5 * G + + .quad 0x3fdfa06c3fc66c0c + .quad 0x5d40e38e4dd60dd2 + .quad 0x7ae38b38268e4d71 + .quad 0x3ac48d916e8357e1 + .quad 0x16561f696a0aa75c + .quad 0xc1bf725c5852bd6a + .quad 0x11a8dd7f9a7966ad + .quad 0x63d988a2d2851026 + .quad 0x00120753afbd232e + .quad 0xe92bceb8fdd8f683 + .quad 0xf81669b384e72b91 + .quad 0x33fad52b2368a066 + + // 2^176 * 6 * G + + .quad 0x540649c6c5e41e16 + .quad 0x0af86430333f7735 + .quad 0xb2acfcd2f305e746 + .quad 0x16c0f429a256dca7 + .quad 0x8d2cc8d0c422cfe8 + .quad 0x072b4f7b05a13acb + .quad 0xa3feb6e6ecf6a56f + .quad 0x3cc355ccb90a71e2 + .quad 0xe9b69443903e9131 + .quad 0xb8a494cb7a5637ce + .quad 0xc87cd1a4baba9244 + .quad 0x631eaf426bae7568 + + // 2^176 * 7 * G + + .quad 0xb3e90410da66fe9f + .quad 0x85dd4b526c16e5a6 + .quad 0xbc3d97611ef9bf83 + .quad 0x5599648b1ea919b5 + .quad 0x47d975b9a3700de8 + .quad 0x7280c5fbe2f80552 + .quad 0x53658f2732e45de1 + .quad 0x431f2c7f665f80b5 + .quad 0xd6026344858f7b19 + .quad 0x14ab352fa1ea514a + .quad 0x8900441a2090a9d7 + .quad 0x7b04715f91253b26 + + // 2^176 * 8 * G + + .quad 0x83edbd28acf6ae43 + .quad 0x86357c8b7d5c7ab4 + .quad 0xc0404769b7eb2c44 + .quad 0x59b37bf5c2f6583f + .quad 0xb376c280c4e6bac6 + .quad 0x970ed3dd6d1d9b0b + .quad 0xb09a9558450bf944 + .quad 0x48d0acfa57cde223 + .quad 0xb60f26e47dabe671 + .quad 0xf1d1a197622f3a37 + .quad 0x4208ce7ee9960394 + .quad 0x16234191336d3bdb + + // 2^180 * 1 * G + + .quad 0xf19aeac733a63aef + .quad 0x2c7fba5d4442454e + .quad 0x5da87aa04795e441 + .quad 0x413051e1a4e0b0f5 + .quad 0x852dd1fd3d578bbe + .quad 0x2b65ce72c3286108 + .quad 0x658c07f4eace2273 + .quad 0x0933f804ec38ab40 + .quad 0xa7ab69798d496476 + .quad 0x8121aadefcb5abc8 + .quad 0xa5dc12ef7b539472 + .quad 0x07fd47065e45351a + + // 2^180 * 2 * G + + .quad 0xc8583c3d258d2bcd + .quad 0x17029a4daf60b73f + .quad 0xfa0fc9d6416a3781 + .quad 0x1c1e5fba38b3fb23 + .quad 0x304211559ae8e7c3 + .quad 0xf281b229944882a5 + .quad 0x8a13ac2e378250e4 + .quad 0x014afa0954ba48f4 + .quad 0xcb3197001bb3666c + .quad 0x330060524bffecb9 + .quad 0x293711991a88233c + .quad 0x291884363d4ed364 + + // 2^180 * 3 * G + + .quad 0x033c6805dc4babfa + .quad 0x2c15bf5e5596ecc1 + .quad 0x1bc70624b59b1d3b + .quad 0x3ede9850a19f0ec5 + .quad 0xfb9d37c3bc1ab6eb + .quad 0x02be14534d57a240 + .quad 0xf4d73415f8a5e1f6 + .quad 0x5964f4300ccc8188 + .quad 0xe44a23152d096800 + .quad 0x5c08c55970866996 + .quad 0xdf2db60a46affb6e + .quad 0x579155c1f856fd89 + + // 2^180 * 4 * G + + .quad 0x96324edd12e0c9ef + .quad 0x468b878df2420297 + .quad 0x199a3776a4f573be + .quad 0x1e7fbcf18e91e92a + .quad 0xb5f16b630817e7a6 + .quad 0x808c69233c351026 + .quad 0x324a983b54cef201 + .quad 0x53c092084a485345 + .quad 0xd2d41481f1cbafbf + .quad 0x231d2db6716174e5 + .quad 0x0b7d7656e2a55c98 + .quad 0x3e955cd82aa495f6 + + // 2^180 * 5 * G + + .quad 0xe48f535e3ed15433 + .quad 0xd075692a0d7270a3 + .quad 0x40fbd21daade6387 + .quad 0x14264887cf4495f5 + .quad 0xab39f3ef61bb3a3f + .quad 0x8eb400652eb9193e + .quad 0xb5de6ecc38c11f74 + .quad 0x654d7e9626f3c49f + .quad 0xe564cfdd5c7d2ceb + .quad 0x82eeafded737ccb9 + .quad 0x6107db62d1f9b0ab + .quad 0x0b6baac3b4358dbb + + // 2^180 * 6 * G + + .quad 0x7ae62bcb8622fe98 + .quad 0x47762256ceb891af + .quad 0x1a5a92bcf2e406b4 + .quad 0x7d29401784e41501 + .quad 0x204abad63700a93b + .quad 0xbe0023d3da779373 + .quad 0xd85f0346633ab709 + .quad 0x00496dc490820412 + .quad 0x1c74b88dc27e6360 + .quad 0x074854268d14850c + .quad 0xa145fb7b3e0dcb30 + .quad 0x10843f1b43803b23 + + // 2^180 * 7 * G + + .quad 0xc5f90455376276dd + .quad 0xce59158dd7645cd9 + .quad 0x92f65d511d366b39 + .quad 0x11574b6e526996c4 + .quad 0xd56f672de324689b + .quad 0xd1da8aedb394a981 + .quad 0xdd7b58fe9168cfed + .quad 0x7ce246cd4d56c1e8 + .quad 0xb8f4308e7f80be53 + .quad 0x5f3cb8cb34a9d397 + .quad 0x18a961bd33cc2b2c + .quad 0x710045fb3a9af671 + + // 2^180 * 8 * G + + .quad 0x73f93d36101b95eb + .quad 0xfaef33794f6f4486 + .quad 0x5651735f8f15e562 + .quad 0x7fa3f19058b40da1 + .quad 0xa03fc862059d699e + .quad 0x2370cfa19a619e69 + .quad 0xc4fe3b122f823deb + .quad 0x1d1b056fa7f0844e + .quad 0x1bc64631e56bf61f + .quad 0xd379ab106e5382a3 + .quad 0x4d58c57e0540168d + .quad 0x566256628442d8e4 + + // 2^184 * 1 * G + + .quad 0xb9e499def6267ff6 + .quad 0x7772ca7b742c0843 + .quad 0x23a0153fe9a4f2b1 + .quad 0x2cdfdfecd5d05006 + .quad 0xdd499cd61ff38640 + .quad 0x29cd9bc3063625a0 + .quad 0x51e2d8023dd73dc3 + .quad 0x4a25707a203b9231 + .quad 0x2ab7668a53f6ed6a + .quad 0x304242581dd170a1 + .quad 0x4000144c3ae20161 + .quad 0x5721896d248e49fc + + // 2^184 * 2 * G + + .quad 0x0b6e5517fd181bae + .quad 0x9022629f2bb963b4 + .quad 0x5509bce932064625 + .quad 0x578edd74f63c13da + .quad 0x285d5091a1d0da4e + .quad 0x4baa6fa7b5fe3e08 + .quad 0x63e5177ce19393b3 + .quad 0x03c935afc4b030fd + .quad 0x997276c6492b0c3d + .quad 0x47ccc2c4dfe205fc + .quad 0xdcd29b84dd623a3c + .quad 0x3ec2ab590288c7a2 + + // 2^184 * 3 * G + + .quad 0xa1a0d27be4d87bb9 + .quad 0xa98b4deb61391aed + .quad 0x99a0ddd073cb9b83 + .quad 0x2dd5c25a200fcace + .quad 0xa7213a09ae32d1cb + .quad 0x0f2b87df40f5c2d5 + .quad 0x0baea4c6e81eab29 + .quad 0x0e1bf66c6adbac5e + .quad 0xe2abd5e9792c887e + .quad 0x1a020018cb926d5d + .quad 0xbfba69cdbaae5f1e + .quad 0x730548b35ae88f5f + + // 2^184 * 4 * G + + .quad 0xc43551a3cba8b8ee + .quad 0x65a26f1db2115f16 + .quad 0x760f4f52ab8c3850 + .quad 0x3043443b411db8ca + .quad 0x805b094ba1d6e334 + .quad 0xbf3ef17709353f19 + .quad 0x423f06cb0622702b + .quad 0x585a2277d87845dd + .quad 0xa18a5f8233d48962 + .quad 0x6698c4b5ec78257f + .quad 0xa78e6fa5373e41ff + .quad 0x7656278950ef981f + + // 2^184 * 5 * G + + .quad 0x38c3cf59d51fc8c0 + .quad 0x9bedd2fd0506b6f2 + .quad 0x26bf109fab570e8f + .quad 0x3f4160a8c1b846a6 + .quad 0xe17073a3ea86cf9d + .quad 0x3a8cfbb707155fdc + .quad 0x4853e7fc31838a8e + .quad 0x28bbf484b613f616 + .quad 0xf2612f5c6f136c7c + .quad 0xafead107f6dd11be + .quad 0x527e9ad213de6f33 + .quad 0x1e79cb358188f75d + + // 2^184 * 6 * G + + .quad 0x013436c3eef7e3f1 + .quad 0x828b6a7ffe9e10f8 + .quad 0x7ff908e5bcf9defc + .quad 0x65d7951b3a3b3831 + .quad 0x77e953d8f5e08181 + .quad 0x84a50c44299dded9 + .quad 0xdc6c2d0c864525e5 + .quad 0x478ab52d39d1f2f4 + .quad 0x66a6a4d39252d159 + .quad 0xe5dde1bc871ac807 + .quad 0xb82c6b40a6c1c96f + .quad 0x16d87a411a212214 + + // 2^184 * 7 * G + + .quad 0xb3bd7e5a42066215 + .quad 0x879be3cd0c5a24c1 + .quad 0x57c05db1d6f994b7 + .quad 0x28f87c8165f38ca6 + .quad 0xfba4d5e2d54e0583 + .quad 0xe21fafd72ebd99fa + .quad 0x497ac2736ee9778f + .quad 0x1f990b577a5a6dde + .quad 0xa3344ead1be8f7d6 + .quad 0x7d1e50ebacea798f + .quad 0x77c6569e520de052 + .quad 0x45882fe1534d6d3e + + // 2^184 * 8 * G + + .quad 0x6669345d757983d6 + .quad 0x62b6ed1117aa11a6 + .quad 0x7ddd1857985e128f + .quad 0x688fe5b8f626f6dd + .quad 0xd8ac9929943c6fe4 + .quad 0xb5f9f161a38392a2 + .quad 0x2699db13bec89af3 + .quad 0x7dcf843ce405f074 + .quad 0x6c90d6484a4732c0 + .quad 0xd52143fdca563299 + .quad 0xb3be28c3915dc6e1 + .quad 0x6739687e7327191b + + // 2^188 * 1 * G + + .quad 0x9f65c5ea200814cf + .quad 0x840536e169a31740 + .quad 0x8b0ed13925c8b4ad + .quad 0x0080dbafe936361d + .quad 0x8ce5aad0c9cb971f + .quad 0x1156aaa99fd54a29 + .quad 0x41f7247015af9b78 + .quad 0x1fe8cca8420f49aa + .quad 0x72a1848f3c0cc82a + .quad 0x38c560c2877c9e54 + .quad 0x5004e228ce554140 + .quad 0x042418a103429d71 + + // 2^188 * 2 * G + + .quad 0x899dea51abf3ff5f + .quad 0x9b93a8672fc2d8ba + .quad 0x2c38cb97be6ebd5c + .quad 0x114d578497263b5d + .quad 0x58e84c6f20816247 + .quad 0x8db2b2b6e36fd793 + .quad 0x977182561d484d85 + .quad 0x0822024f8632abd7 + .quad 0xb301bb7c6b1beca3 + .quad 0x55393f6dc6eb1375 + .quad 0x910d281097b6e4eb + .quad 0x1ad4548d9d479ea3 + + // 2^188 * 3 * G + + .quad 0xcd5a7da0389a48fd + .quad 0xb38fa4aa9a78371e + .quad 0xc6d9761b2cdb8e6c + .quad 0x35cf51dbc97e1443 + .quad 0xa06fe66d0fe9fed3 + .quad 0xa8733a401c587909 + .quad 0x30d14d800df98953 + .quad 0x41ce5876c7b30258 + .quad 0x59ac3bc5d670c022 + .quad 0xeae67c109b119406 + .quad 0x9798bdf0b3782fda + .quad 0x651e3201fd074092 + + // 2^188 * 4 * G + + .quad 0xd63d8483ef30c5cf + .quad 0x4cd4b4962361cc0c + .quad 0xee90e500a48426ac + .quad 0x0af51d7d18c14eeb + .quad 0xa57ba4a01efcae9e + .quad 0x769f4beedc308a94 + .quad 0xd1f10eeb3603cb2e + .quad 0x4099ce5e7e441278 + .quad 0x1ac98e4f8a5121e9 + .quad 0x7dae9544dbfa2fe0 + .quad 0x8320aa0dd6430df9 + .quad 0x667282652c4a2fb5 + + // 2^188 * 5 * G + + .quad 0x874621f4d86bc9ab + .quad 0xb54c7bbe56fe6fea + .quad 0x077a24257fadc22c + .quad 0x1ab53be419b90d39 + .quad 0xada8b6e02946db23 + .quad 0x1c0ce51a7b253ab7 + .quad 0x8448c85a66dd485b + .quad 0x7f1fc025d0675adf + .quad 0xd8ee1b18319ea6aa + .quad 0x004d88083a21f0da + .quad 0x3bd6aa1d883a4f4b + .quad 0x4db9a3a6dfd9fd14 + + // 2^188 * 6 * G + + .quad 0x8ce7b23bb99c0755 + .quad 0x35c5d6edc4f50f7a + .quad 0x7e1e2ed2ed9b50c3 + .quad 0x36305f16e8934da1 + .quad 0xd95b00bbcbb77c68 + .quad 0xddbc846a91f17849 + .quad 0x7cf700aebe28d9b3 + .quad 0x5ce1285c85d31f3e + .quad 0x31b6972d98b0bde8 + .quad 0x7d920706aca6de5b + .quad 0xe67310f8908a659f + .quad 0x50fac2a6efdf0235 + + // 2^188 * 7 * G + + .quad 0xf3d3a9f35b880f5a + .quad 0xedec050cdb03e7c2 + .quad 0xa896981ff9f0b1a2 + .quad 0x49a4ae2bac5e34a4 + .quad 0x295b1c86f6f449bc + .quad 0x51b2e84a1f0ab4dd + .quad 0xc001cb30aa8e551d + .quad 0x6a28d35944f43662 + .quad 0x28bb12ee04a740e0 + .quad 0x14313bbd9bce8174 + .quad 0x72f5b5e4e8c10c40 + .quad 0x7cbfb19936adcd5b + + // 2^188 * 8 * G + + .quad 0xa311ddc26b89792d + .quad 0x1b30b4c6da512664 + .quad 0x0ca77b4ccf150859 + .quad 0x1de443df1b009408 + .quad 0x8e793a7acc36e6e0 + .quad 0xf9fab7a37d586eed + .quad 0x3a4f9692bae1f4e4 + .quad 0x1c14b03eff5f447e + .quad 0x19647bd114a85291 + .quad 0x57b76cb21034d3af + .quad 0x6329db440f9d6dfa + .quad 0x5ef43e586a571493 + + // 2^192 * 1 * G + + .quad 0xef782014385675a6 + .quad 0xa2649f30aafda9e8 + .quad 0x4cd1eb505cdfa8cb + .quad 0x46115aba1d4dc0b3 + .quad 0xa66dcc9dc80c1ac0 + .quad 0x97a05cf41b38a436 + .quad 0xa7ebf3be95dbd7c6 + .quad 0x7da0b8f68d7e7dab + .quad 0xd40f1953c3b5da76 + .quad 0x1dac6f7321119e9b + .quad 0x03cc6021feb25960 + .quad 0x5a5f887e83674b4b + + // 2^192 * 2 * G + + .quad 0x8f6301cf70a13d11 + .quad 0xcfceb815350dd0c4 + .quad 0xf70297d4a4bca47e + .quad 0x3669b656e44d1434 + .quad 0x9e9628d3a0a643b9 + .quad 0xb5c3cb00e6c32064 + .quad 0x9b5302897c2dec32 + .quad 0x43e37ae2d5d1c70c + .quad 0x387e3f06eda6e133 + .quad 0x67301d5199a13ac0 + .quad 0xbd5ad8f836263811 + .quad 0x6a21e6cd4fd5e9be + + // 2^192 * 3 * G + + .quad 0xf1c6170a3046e65f + .quad 0x58712a2a00d23524 + .quad 0x69dbbd3c8c82b755 + .quad 0x586bf9f1a195ff57 + .quad 0xef4129126699b2e3 + .quad 0x71d30847708d1301 + .quad 0x325432d01182b0bd + .quad 0x45371b07001e8b36 + .quad 0xa6db088d5ef8790b + .quad 0x5278f0dc610937e5 + .quad 0xac0349d261a16eb8 + .quad 0x0eafb03790e52179 + + // 2^192 * 4 * G + + .quad 0x960555c13748042f + .quad 0x219a41e6820baa11 + .quad 0x1c81f73873486d0c + .quad 0x309acc675a02c661 + .quad 0x5140805e0f75ae1d + .quad 0xec02fbe32662cc30 + .quad 0x2cebdf1eea92396d + .quad 0x44ae3344c5435bb3 + .quad 0x9cf289b9bba543ee + .quad 0xf3760e9d5ac97142 + .quad 0x1d82e5c64f9360aa + .quad 0x62d5221b7f94678f + + // 2^192 * 5 * G + + .quad 0x524c299c18d0936d + .quad 0xc86bb56c8a0c1a0c + .quad 0xa375052edb4a8631 + .quad 0x5c0efde4bc754562 + .quad 0x7585d4263af77a3c + .quad 0xdfae7b11fee9144d + .quad 0xa506708059f7193d + .quad 0x14f29a5383922037 + .quad 0xdf717edc25b2d7f5 + .quad 0x21f970db99b53040 + .quad 0xda9234b7c3ed4c62 + .quad 0x5e72365c7bee093e + + // 2^192 * 6 * G + + .quad 0x575bfc074571217f + .quad 0x3779675d0694d95b + .quad 0x9a0a37bbf4191e33 + .quad 0x77f1104c47b4eabc + .quad 0x7d9339062f08b33e + .quad 0x5b9659e5df9f32be + .quad 0xacff3dad1f9ebdfd + .quad 0x70b20555cb7349b7 + .quad 0xbe5113c555112c4c + .quad 0x6688423a9a881fcd + .quad 0x446677855e503b47 + .quad 0x0e34398f4a06404a + + // 2^192 * 7 * G + + .quad 0xb67d22d93ecebde8 + .quad 0x09b3e84127822f07 + .quad 0x743fa61fb05b6d8d + .quad 0x5e5405368a362372 + .quad 0x18930b093e4b1928 + .quad 0x7de3e10e73f3f640 + .quad 0xf43217da73395d6f + .quad 0x6f8aded6ca379c3e + .quad 0xe340123dfdb7b29a + .quad 0x487b97e1a21ab291 + .quad 0xf9967d02fde6949e + .quad 0x780de72ec8d3de97 + + // 2^192 * 8 * G + + .quad 0x0ae28545089ae7bc + .quad 0x388ddecf1c7f4d06 + .quad 0x38ac15510a4811b8 + .quad 0x0eb28bf671928ce4 + .quad 0x671feaf300f42772 + .quad 0x8f72eb2a2a8c41aa + .quad 0x29a17fd797373292 + .quad 0x1defc6ad32b587a6 + .quad 0xaf5bbe1aef5195a7 + .quad 0x148c1277917b15ed + .quad 0x2991f7fb7ae5da2e + .quad 0x467d201bf8dd2867 + + // 2^196 * 1 * G + + .quad 0x7906ee72f7bd2e6b + .quad 0x05d270d6109abf4e + .quad 0x8d5cfe45b941a8a4 + .quad 0x44c218671c974287 + .quad 0x745f9d56296bc318 + .quad 0x993580d4d8152e65 + .quad 0xb0e5b13f5839e9ce + .quad 0x51fc2b28d43921c0 + .quad 0x1b8fd11795e2a98c + .quad 0x1c4e5ee12b6b6291 + .quad 0x5b30e7107424b572 + .quad 0x6e6b9de84c4f4ac6 + + // 2^196 * 2 * G + + .quad 0xdff25fce4b1de151 + .quad 0xd841c0c7e11c4025 + .quad 0x2554b3c854749c87 + .quad 0x2d292459908e0df9 + .quad 0x6b7c5f10f80cb088 + .quad 0x736b54dc56e42151 + .quad 0xc2b620a5c6ef99c4 + .quad 0x5f4c802cc3a06f42 + .quad 0x9b65c8f17d0752da + .quad 0x881ce338c77ee800 + .quad 0xc3b514f05b62f9e3 + .quad 0x66ed5dd5bec10d48 + + // 2^196 * 3 * G + + .quad 0x7d38a1c20bb2089d + .quad 0x808334e196ccd412 + .quad 0xc4a70b8c6c97d313 + .quad 0x2eacf8bc03007f20 + .quad 0xf0adf3c9cbca047d + .quad 0x81c3b2cbf4552f6b + .quad 0xcfda112d44735f93 + .quad 0x1f23a0c77e20048c + .quad 0xf235467be5bc1570 + .quad 0x03d2d9020dbab38c + .quad 0x27529aa2fcf9e09e + .quad 0x0840bef29d34bc50 + + // 2^196 * 4 * G + + .quad 0x796dfb35dc10b287 + .quad 0x27176bcd5c7ff29d + .quad 0x7f3d43e8c7b24905 + .quad 0x0304f5a191c54276 + .quad 0xcd54e06b7f37e4eb + .quad 0x8cc15f87f5e96cca + .quad 0xb8248bb0d3597dce + .quad 0x246affa06074400c + .quad 0x37d88e68fbe45321 + .quad 0x86097548c0d75032 + .quad 0x4e9b13ef894a0d35 + .quad 0x25a83cac5753d325 + + // 2^196 * 5 * G + + .quad 0x10222f48eed8165e + .quad 0x623fc1234b8bcf3a + .quad 0x1e145c09c221e8f0 + .quad 0x7ccfa59fca782630 + .quad 0x9f0f66293952b6e2 + .quad 0x33db5e0e0934267b + .quad 0xff45252bd609fedc + .quad 0x06be10f5c506e0c9 + .quad 0x1a9615a9b62a345f + .quad 0x22050c564a52fecc + .quad 0xa7a2788528bc0dfe + .quad 0x5e82770a1a1ee71d + + // 2^196 * 6 * G + + .quad 0x35425183ad896a5c + .quad 0xe8673afbe78d52f6 + .quad 0x2c66f25f92a35f64 + .quad 0x09d04f3b3b86b102 + .quad 0xe802e80a42339c74 + .quad 0x34175166a7fffae5 + .quad 0x34865d1f1c408cae + .quad 0x2cca982c605bc5ee + .quad 0xfd2d5d35197dbe6e + .quad 0x207c2eea8be4ffa3 + .quad 0x2613d8db325ae918 + .quad 0x7a325d1727741d3e + + // 2^196 * 7 * G + + .quad 0xd036b9bbd16dfde2 + .quad 0xa2055757c497a829 + .quad 0x8e6cc966a7f12667 + .quad 0x4d3b1a791239c180 + .quad 0xecd27d017e2a076a + .quad 0xd788689f1636495e + .quad 0x52a61af0919233e5 + .quad 0x2a479df17bb1ae64 + .quad 0x9e5eee8e33db2710 + .quad 0x189854ded6c43ca5 + .quad 0xa41c22c592718138 + .quad 0x27ad5538a43a5e9b + + // 2^196 * 8 * G + + .quad 0x2746dd4b15350d61 + .quad 0xd03fcbc8ee9521b7 + .quad 0xe86e365a138672ca + .quad 0x510e987f7e7d89e2 + .quad 0xcb5a7d638e47077c + .quad 0x8db7536120a1c059 + .quad 0x549e1e4d8bedfdcc + .quad 0x080153b7503b179d + .quad 0xdda69d930a3ed3e3 + .quad 0x3d386ef1cd60a722 + .quad 0xc817ad58bdaa4ee6 + .quad 0x23be8d554fe7372a + + // 2^200 * 1 * G + + .quad 0x95fe919a74ef4fad + .quad 0x3a827becf6a308a2 + .quad 0x964e01d309a47b01 + .quad 0x71c43c4f5ba3c797 + .quad 0xbc1ef4bd567ae7a9 + .quad 0x3f624cb2d64498bd + .quad 0xe41064d22c1f4ec8 + .quad 0x2ef9c5a5ba384001 + .quad 0xb6fd6df6fa9e74cd + .quad 0xf18278bce4af267a + .quad 0x8255b3d0f1ef990e + .quad 0x5a758ca390c5f293 + + // 2^200 * 2 * G + + .quad 0xa2b72710d9462495 + .quad 0x3aa8c6d2d57d5003 + .quad 0xe3d400bfa0b487ca + .quad 0x2dbae244b3eb72ec + .quad 0x8ce0918b1d61dc94 + .quad 0x8ded36469a813066 + .quad 0xd4e6a829afe8aad3 + .quad 0x0a738027f639d43f + .quad 0x980f4a2f57ffe1cc + .quad 0x00670d0de1839843 + .quad 0x105c3f4a49fb15fd + .quad 0x2698ca635126a69c + + // 2^200 * 3 * G + + .quad 0xe765318832b0ba78 + .quad 0x381831f7925cff8b + .quad 0x08a81b91a0291fcc + .quad 0x1fb43dcc49caeb07 + .quad 0x2e3d702f5e3dd90e + .quad 0x9e3f0918e4d25386 + .quad 0x5e773ef6024da96a + .quad 0x3c004b0c4afa3332 + .quad 0x9aa946ac06f4b82b + .quad 0x1ca284a5a806c4f3 + .quad 0x3ed3265fc6cd4787 + .quad 0x6b43fd01cd1fd217 + + // 2^200 * 4 * G + + .quad 0xc7a75d4b4697c544 + .quad 0x15fdf848df0fffbf + .quad 0x2868b9ebaa46785a + .quad 0x5a68d7105b52f714 + .quad 0xb5c742583e760ef3 + .quad 0x75dc52b9ee0ab990 + .quad 0xbf1427c2072b923f + .quad 0x73420b2d6ff0d9f0 + .quad 0xaf2cf6cb9e851e06 + .quad 0x8f593913c62238c4 + .quad 0xda8ab89699fbf373 + .quad 0x3db5632fea34bc9e + + // 2^200 * 5 * G + + .quad 0xf46eee2bf75dd9d8 + .quad 0x0d17b1f6396759a5 + .quad 0x1bf2d131499e7273 + .quad 0x04321adf49d75f13 + .quad 0x2e4990b1829825d5 + .quad 0xedeaeb873e9a8991 + .quad 0xeef03d394c704af8 + .quad 0x59197ea495df2b0e + .quad 0x04e16019e4e55aae + .quad 0xe77b437a7e2f92e9 + .quad 0xc7ce2dc16f159aa4 + .quad 0x45eafdc1f4d70cc0 + + // 2^200 * 6 * G + + .quad 0x698401858045d72b + .quad 0x4c22faa2cf2f0651 + .quad 0x941a36656b222dc6 + .quad 0x5a5eebc80362dade + .quad 0xb60e4624cfccb1ed + .quad 0x59dbc292bd5c0395 + .quad 0x31a09d1ddc0481c9 + .quad 0x3f73ceea5d56d940 + .quad 0xb7a7bfd10a4e8dc6 + .quad 0xbe57007e44c9b339 + .quad 0x60c1207f1557aefa + .quad 0x26058891266218db + + // 2^200 * 7 * G + + .quad 0x59f704a68360ff04 + .quad 0xc3d93fde7661e6f4 + .quad 0x831b2a7312873551 + .quad 0x54ad0c2e4e615d57 + .quad 0x4c818e3cc676e542 + .quad 0x5e422c9303ceccad + .quad 0xec07cccab4129f08 + .quad 0x0dedfa10b24443b8 + .quad 0xee3b67d5b82b522a + .quad 0x36f163469fa5c1eb + .quad 0xa5b4d2f26ec19fd3 + .quad 0x62ecb2baa77a9408 + + // 2^200 * 8 * G + + .quad 0xe5ed795261152b3d + .quad 0x4962357d0eddd7d1 + .quad 0x7482c8d0b96b4c71 + .quad 0x2e59f919a966d8be + .quad 0x92072836afb62874 + .quad 0x5fcd5e8579e104a5 + .quad 0x5aad01adc630a14a + .quad 0x61913d5075663f98 + .quad 0x0dc62d361a3231da + .quad 0xfa47583294200270 + .quad 0x02d801513f9594ce + .quad 0x3ddbc2a131c05d5c + + // 2^204 * 1 * G + + .quad 0x3f50a50a4ffb81ef + .quad 0xb1e035093bf420bf + .quad 0x9baa8e1cc6aa2cd0 + .quad 0x32239861fa237a40 + .quad 0xfb735ac2004a35d1 + .quad 0x31de0f433a6607c3 + .quad 0x7b8591bfc528d599 + .quad 0x55be9a25f5bb050c + .quad 0x0d005acd33db3dbf + .quad 0x0111b37c80ac35e2 + .quad 0x4892d66c6f88ebeb + .quad 0x770eadb16508fbcd + + // 2^204 * 2 * G + + .quad 0x8451f9e05e4e89dd + .quad 0xc06302ffbc793937 + .quad 0x5d22749556a6495c + .quad 0x09a6755ca05603fb + .quad 0xf1d3b681a05071b9 + .quad 0x2207659a3592ff3a + .quad 0x5f0169297881e40e + .quad 0x16bedd0e86ba374e + .quad 0x5ecccc4f2c2737b5 + .quad 0x43b79e0c2dccb703 + .quad 0x33e008bc4ec43df3 + .quad 0x06c1b840f07566c0 + + // 2^204 * 3 * G + + .quad 0x7688a5c6a388f877 + .quad 0x02a96c14deb2b6ac + .quad 0x64c9f3431b8c2af8 + .quad 0x3628435554a1eed6 + .quad 0x69ee9e7f9b02805c + .quad 0xcbff828a547d1640 + .quad 0x3d93a869b2430968 + .quad 0x46b7b8cd3fe26972 + .quad 0xe9812086fe7eebe0 + .quad 0x4cba6be72f515437 + .quad 0x1d04168b516efae9 + .quad 0x5ea1391043982cb9 + + // 2^204 * 4 * G + + .quad 0x49125c9cf4702ee1 + .quad 0x4520b71f8b25b32d + .quad 0x33193026501fef7e + .quad 0x656d8997c8d2eb2b + .quad 0x6f2b3be4d5d3b002 + .quad 0xafec33d96a09c880 + .quad 0x035f73a4a8bcc4cc + .quad 0x22c5b9284662198b + .quad 0xcb58c8fe433d8939 + .quad 0x89a0cb2e6a8d7e50 + .quad 0x79ca955309fbbe5a + .quad 0x0c626616cd7fc106 + + // 2^204 * 5 * G + + .quad 0x1ffeb80a4879b61f + .quad 0x6396726e4ada21ed + .quad 0x33c7b093368025ba + .quad 0x471aa0c6f3c31788 + .quad 0x8fdfc379fbf454b1 + .quad 0x45a5a970f1a4b771 + .quad 0xac921ef7bad35915 + .quad 0x42d088dca81c2192 + .quad 0x8fda0f37a0165199 + .quad 0x0adadb77c8a0e343 + .quad 0x20fbfdfcc875e820 + .quad 0x1cf2bea80c2206e7 + + // 2^204 * 6 * G + + .quad 0xc2ddf1deb36202ac + .quad 0x92a5fe09d2e27aa5 + .quad 0x7d1648f6fc09f1d3 + .quad 0x74c2cc0513bc4959 + .quad 0x982d6e1a02c0412f + .quad 0x90fa4c83db58e8fe + .quad 0x01c2f5bcdcb18bc0 + .quad 0x686e0c90216abc66 + .quad 0x1fadbadba54395a7 + .quad 0xb41a02a0ae0da66a + .quad 0xbf19f598bba37c07 + .quad 0x6a12b8acde48430d + + // 2^204 * 7 * G + + .quad 0xf8daea1f39d495d9 + .quad 0x592c190e525f1dfc + .quad 0xdb8cbd04c9991d1b + .quad 0x11f7fda3d88f0cb7 + .quad 0x793bdd801aaeeb5f + .quad 0x00a2a0aac1518871 + .quad 0xe8a373a31f2136b4 + .quad 0x48aab888fc91ef19 + .quad 0x041f7e925830f40e + .quad 0x002d6ca979661c06 + .quad 0x86dc9ff92b046a2e + .quad 0x760360928b0493d1 + + // 2^204 * 8 * G + + .quad 0x21bb41c6120cf9c6 + .quad 0xeab2aa12decda59b + .quad 0xc1a72d020aa48b34 + .quad 0x215d4d27e87d3b68 + .quad 0xb43108e5695a0b05 + .quad 0x6cb00ee8ad37a38b + .quad 0x5edad6eea3537381 + .quad 0x3f2602d4b6dc3224 + .quad 0xc8b247b65bcaf19c + .quad 0x49779dc3b1b2c652 + .quad 0x89a180bbd5ece2e2 + .quad 0x13f098a3cec8e039 + + // 2^208 * 1 * G + + .quad 0x9adc0ff9ce5ec54b + .quad 0x039c2a6b8c2f130d + .quad 0x028007c7f0f89515 + .quad 0x78968314ac04b36b + .quad 0xf3aa57a22796bb14 + .quad 0x883abab79b07da21 + .quad 0xe54be21831a0391c + .quad 0x5ee7fb38d83205f9 + .quad 0x538dfdcb41446a8e + .quad 0xa5acfda9434937f9 + .quad 0x46af908d263c8c78 + .quad 0x61d0633c9bca0d09 + + // 2^208 * 2 * G + + .quad 0x63744935ffdb2566 + .quad 0xc5bd6b89780b68bb + .quad 0x6f1b3280553eec03 + .quad 0x6e965fd847aed7f5 + .quad 0xada328bcf8fc73df + .quad 0xee84695da6f037fc + .quad 0x637fb4db38c2a909 + .quad 0x5b23ac2df8067bdc + .quad 0x9ad2b953ee80527b + .quad 0xe88f19aafade6d8d + .quad 0x0e711704150e82cf + .quad 0x79b9bbb9dd95dedc + + // 2^208 * 3 * G + + .quad 0xebb355406a3126c2 + .quad 0xd26383a868c8c393 + .quad 0x6c0c6429e5b97a82 + .quad 0x5065f158c9fd2147 + .quad 0xd1997dae8e9f7374 + .quad 0xa032a2f8cfbb0816 + .quad 0xcd6cba126d445f0a + .quad 0x1ba811460accb834 + .quad 0x708169fb0c429954 + .quad 0xe14600acd76ecf67 + .quad 0x2eaab98a70e645ba + .quad 0x3981f39e58a4faf2 + + // 2^208 * 4 * G + + .quad 0x18fb8a7559230a93 + .quad 0x1d168f6960e6f45d + .quad 0x3a85a94514a93cb5 + .quad 0x38dc083705acd0fd + .quad 0xc845dfa56de66fde + .quad 0xe152a5002c40483a + .quad 0xe9d2e163c7b4f632 + .quad 0x30f4452edcbc1b65 + .quad 0x856d2782c5759740 + .quad 0xfa134569f99cbecc + .quad 0x8844fc73c0ea4e71 + .quad 0x632d9a1a593f2469 + + // 2^208 * 5 * G + + .quad 0xf6bb6b15b807cba6 + .quad 0x1823c7dfbc54f0d7 + .quad 0xbb1d97036e29670b + .quad 0x0b24f48847ed4a57 + .quad 0xbf09fd11ed0c84a7 + .quad 0x63f071810d9f693a + .quad 0x21908c2d57cf8779 + .quad 0x3a5a7df28af64ba2 + .quad 0xdcdad4be511beac7 + .quad 0xa4538075ed26ccf2 + .quad 0xe19cff9f005f9a65 + .quad 0x34fcf74475481f63 + + // 2^208 * 6 * G + + .quad 0xc197e04c789767ca + .quad 0xb8714dcb38d9467d + .quad 0x55de888283f95fa8 + .quad 0x3d3bdc164dfa63f7 + .quad 0xa5bb1dab78cfaa98 + .quad 0x5ceda267190b72f2 + .quad 0x9309c9110a92608e + .quad 0x0119a3042fb374b0 + .quad 0x67a2d89ce8c2177d + .quad 0x669da5f66895d0c1 + .quad 0xf56598e5b282a2b0 + .quad 0x56c088f1ede20a73 + + // 2^208 * 7 * G + + .quad 0x336d3d1110a86e17 + .quad 0xd7f388320b75b2fa + .quad 0xf915337625072988 + .quad 0x09674c6b99108b87 + .quad 0x581b5fac24f38f02 + .quad 0xa90be9febae30cbd + .quad 0x9a2169028acf92f0 + .quad 0x038b7ea48359038f + .quad 0x9f4ef82199316ff8 + .quad 0x2f49d282eaa78d4f + .quad 0x0971a5ab5aef3174 + .quad 0x6e5e31025969eb65 + + // 2^208 * 8 * G + + .quad 0xb16c62f587e593fb + .quad 0x4999eddeca5d3e71 + .quad 0xb491c1e014cc3e6d + .quad 0x08f5114789a8dba8 + .quad 0x3304fb0e63066222 + .quad 0xfb35068987acba3f + .quad 0xbd1924778c1061a3 + .quad 0x3058ad43d1838620 + .quad 0x323c0ffde57663d0 + .quad 0x05c3df38a22ea610 + .quad 0xbdc78abdac994f9a + .quad 0x26549fa4efe3dc99 + + // 2^212 * 1 * G + + .quad 0x738b38d787ce8f89 + .quad 0xb62658e24179a88d + .quad 0x30738c9cf151316d + .quad 0x49128c7f727275c9 + .quad 0x04dbbc17f75396b9 + .quad 0x69e6a2d7d2f86746 + .quad 0xc6409d99f53eabc6 + .quad 0x606175f6332e25d2 + .quad 0x4021370ef540e7dd + .quad 0x0910d6f5a1f1d0a5 + .quad 0x4634aacd5b06b807 + .quad 0x6a39e6356944f235 + + // 2^212 * 2 * G + + .quad 0x96cd5640df90f3e7 + .quad 0x6c3a760edbfa25ea + .quad 0x24f3ef0959e33cc4 + .quad 0x42889e7e530d2e58 + .quad 0x1da1965774049e9d + .quad 0xfbcd6ea198fe352b + .quad 0xb1cbcd50cc5236a6 + .quad 0x1f5ec83d3f9846e2 + .quad 0x8efb23c3328ccb75 + .quad 0xaf42a207dd876ee9 + .quad 0x20fbdadc5dfae796 + .quad 0x241e246b06bf9f51 + + // 2^212 * 3 * G + + .quad 0x29e68e57ad6e98f6 + .quad 0x4c9260c80b462065 + .quad 0x3f00862ea51ebb4b + .quad 0x5bc2c77fb38d9097 + .quad 0x7eaafc9a6280bbb8 + .quad 0x22a70f12f403d809 + .quad 0x31ce40bb1bfc8d20 + .quad 0x2bc65635e8bd53ee + .quad 0xe8d5dc9fa96bad93 + .quad 0xe58fb17dde1947dc + .quad 0x681532ea65185fa3 + .quad 0x1fdd6c3b034a7830 + + // 2^212 * 4 * G + + .quad 0x0a64e28c55dc18fe + .quad 0xe3df9e993399ebdd + .quad 0x79ac432370e2e652 + .quad 0x35ff7fc33ae4cc0e + .quad 0x9c13a6a52dd8f7a9 + .quad 0x2dbb1f8c3efdcabf + .quad 0x961e32405e08f7b5 + .quad 0x48c8a121bbe6c9e5 + .quad 0xfc415a7c59646445 + .quad 0xd224b2d7c128b615 + .quad 0x6035c9c905fbb912 + .quad 0x42d7a91274429fab + + // 2^212 * 5 * G + + .quad 0x4e6213e3eaf72ed3 + .quad 0x6794981a43acd4e7 + .quad 0xff547cde6eb508cb + .quad 0x6fed19dd10fcb532 + .quad 0xa9a48947933da5bc + .quad 0x4a58920ec2e979ec + .quad 0x96d8800013e5ac4c + .quad 0x453692d74b48b147 + .quad 0xdd775d99a8559c6f + .quad 0xf42a2140df003e24 + .quad 0x5223e229da928a66 + .quad 0x063f46ba6d38f22c + + // 2^212 * 6 * G + + .quad 0xd2d242895f536694 + .quad 0xca33a2c542939b2c + .quad 0x986fada6c7ddb95c + .quad 0x5a152c042f712d5d + .quad 0x39843cb737346921 + .quad 0xa747fb0738c89447 + .quad 0xcb8d8031a245307e + .quad 0x67810f8e6d82f068 + .quad 0x3eeb8fbcd2287db4 + .quad 0x72c7d3a301a03e93 + .quad 0x5473e88cbd98265a + .quad 0x7324aa515921b403 + + // 2^212 * 7 * G + + .quad 0x857942f46c3cbe8e + .quad 0xa1d364b14730c046 + .quad 0x1c8ed914d23c41bf + .quad 0x0838e161eef6d5d2 + .quad 0xad23f6dae82354cb + .quad 0x6962502ab6571a6d + .quad 0x9b651636e38e37d1 + .quad 0x5cac5005d1a3312f + .quad 0x8cc154cce9e39904 + .quad 0x5b3a040b84de6846 + .quad 0xc4d8a61cb1be5d6e + .quad 0x40fb897bd8861f02 + + // 2^212 * 8 * G + + .quad 0x84c5aa9062de37a1 + .quad 0x421da5000d1d96e1 + .quad 0x788286306a9242d9 + .quad 0x3c5e464a690d10da + .quad 0xe57ed8475ab10761 + .quad 0x71435e206fd13746 + .quad 0x342f824ecd025632 + .quad 0x4b16281ea8791e7b + .quad 0xd1c101d50b813381 + .quad 0xdee60f1176ee6828 + .quad 0x0cb68893383f6409 + .quad 0x6183c565f6ff484a + + // 2^216 * 1 * G + + .quad 0x741d5a461e6bf9d6 + .quad 0x2305b3fc7777a581 + .quad 0xd45574a26474d3d9 + .quad 0x1926e1dc6401e0ff + .quad 0xdb468549af3f666e + .quad 0xd77fcf04f14a0ea5 + .quad 0x3df23ff7a4ba0c47 + .quad 0x3a10dfe132ce3c85 + .quad 0xe07f4e8aea17cea0 + .quad 0x2fd515463a1fc1fd + .quad 0x175322fd31f2c0f1 + .quad 0x1fa1d01d861e5d15 + + // 2^216 * 2 * G + + .quad 0xcc8055947d599832 + .quad 0x1e4656da37f15520 + .quad 0x99f6f7744e059320 + .quad 0x773563bc6a75cf33 + .quad 0x38dcac00d1df94ab + .quad 0x2e712bddd1080de9 + .quad 0x7f13e93efdd5e262 + .quad 0x73fced18ee9a01e5 + .quad 0x06b1e90863139cb3 + .quad 0xa493da67c5a03ecd + .quad 0x8d77cec8ad638932 + .quad 0x1f426b701b864f44 + + // 2^216 * 3 * G + + .quad 0xefc9264c41911c01 + .quad 0xf1a3b7b817a22c25 + .quad 0x5875da6bf30f1447 + .quad 0x4e1af5271d31b090 + .quad 0xf17e35c891a12552 + .quad 0xb76b8153575e9c76 + .quad 0xfa83406f0d9b723e + .quad 0x0b76bb1b3fa7e438 + .quad 0x08b8c1f97f92939b + .quad 0xbe6771cbd444ab6e + .quad 0x22e5646399bb8017 + .quad 0x7b6dd61eb772a955 + + // 2^216 * 4 * G + + .quad 0xb7adc1e850f33d92 + .quad 0x7998fa4f608cd5cf + .quad 0xad962dbd8dfc5bdb + .quad 0x703e9bceaf1d2f4f + .quad 0x5730abf9ab01d2c7 + .quad 0x16fb76dc40143b18 + .quad 0x866cbe65a0cbb281 + .quad 0x53fa9b659bff6afe + .quad 0x6c14c8e994885455 + .quad 0x843a5d6665aed4e5 + .quad 0x181bb73ebcd65af1 + .quad 0x398d93e5c4c61f50 + + // 2^216 * 5 * G + + .quad 0x1c4bd16733e248f3 + .quad 0xbd9e128715bf0a5f + .quad 0xd43f8cf0a10b0376 + .quad 0x53b09b5ddf191b13 + .quad 0xc3877c60d2e7e3f2 + .quad 0x3b34aaa030828bb1 + .quad 0x283e26e7739ef138 + .quad 0x699c9c9002c30577 + .quad 0xf306a7235946f1cc + .quad 0x921718b5cce5d97d + .quad 0x28cdd24781b4e975 + .quad 0x51caf30c6fcdd907 + + // 2^216 * 6 * G + + .quad 0xa60ba7427674e00a + .quad 0x630e8570a17a7bf3 + .quad 0x3758563dcf3324cc + .quad 0x5504aa292383fdaa + .quad 0x737af99a18ac54c7 + .quad 0x903378dcc51cb30f + .quad 0x2b89bc334ce10cc7 + .quad 0x12ae29c189f8e99a + .quad 0xa99ec0cb1f0d01cf + .quad 0x0dd1efcc3a34f7ae + .quad 0x55ca7521d09c4e22 + .quad 0x5fd14fe958eba5ea + + // 2^216 * 7 * G + + .quad 0xb5dc2ddf2845ab2c + .quad 0x069491b10a7fe993 + .quad 0x4daaf3d64002e346 + .quad 0x093ff26e586474d1 + .quad 0x3c42fe5ebf93cb8e + .quad 0xbedfa85136d4565f + .quad 0xe0f0859e884220e8 + .quad 0x7dd73f960725d128 + .quad 0xb10d24fe68059829 + .quad 0x75730672dbaf23e5 + .quad 0x1367253ab457ac29 + .quad 0x2f59bcbc86b470a4 + + // 2^216 * 8 * G + + .quad 0x83847d429917135f + .quad 0xad1b911f567d03d7 + .quad 0x7e7748d9be77aad1 + .quad 0x5458b42e2e51af4a + .quad 0x7041d560b691c301 + .quad 0x85201b3fadd7e71e + .quad 0x16c2e16311335585 + .quad 0x2aa55e3d010828b1 + .quad 0xed5192e60c07444f + .quad 0x42c54e2d74421d10 + .quad 0x352b4c82fdb5c864 + .quad 0x13e9004a8a768664 + + // 2^220 * 1 * G + + .quad 0xcbb5b5556c032bff + .quad 0xdf7191b729297a3a + .quad 0xc1ff7326aded81bb + .quad 0x71ade8bb68be03f5 + .quad 0x1e6284c5806b467c + .quad 0xc5f6997be75d607b + .quad 0x8b67d958b378d262 + .quad 0x3d88d66a81cd8b70 + .quad 0x8b767a93204ed789 + .quad 0x762fcacb9fa0ae2a + .quad 0x771febcc6dce4887 + .quad 0x343062158ff05fb3 + + // 2^220 * 2 * G + + .quad 0xe05da1a7e1f5bf49 + .quad 0x26457d6dd4736092 + .quad 0x77dcb07773cc32f6 + .quad 0x0a5d94969cdd5fcd + .quad 0xfce219072a7b31b4 + .quad 0x4d7adc75aa578016 + .quad 0x0ec276a687479324 + .quad 0x6d6d9d5d1fda4beb + .quad 0x22b1a58ae9b08183 + .quad 0xfd95d071c15c388b + .quad 0xa9812376850a0517 + .quad 0x33384cbabb7f335e + + // 2^220 * 3 * G + + .quad 0x3c6fa2680ca2c7b5 + .quad 0x1b5082046fb64fda + .quad 0xeb53349c5431d6de + .quad 0x5278b38f6b879c89 + .quad 0x33bc627a26218b8d + .quad 0xea80b21fc7a80c61 + .quad 0x9458b12b173e9ee6 + .quad 0x076247be0e2f3059 + .quad 0x52e105f61416375a + .quad 0xec97af3685abeba4 + .quad 0x26e6b50623a67c36 + .quad 0x5cf0e856f3d4fb01 + + // 2^220 * 4 * G + + .quad 0xf6c968731ae8cab4 + .quad 0x5e20741ecb4f92c5 + .quad 0x2da53be58ccdbc3e + .quad 0x2dddfea269970df7 + .quad 0xbeaece313db342a8 + .quad 0xcba3635b842db7ee + .quad 0xe88c6620817f13ef + .quad 0x1b9438aa4e76d5c6 + .quad 0x8a50777e166f031a + .quad 0x067b39f10fb7a328 + .quad 0x1925c9a6010fbd76 + .quad 0x6df9b575cc740905 + + // 2^220 * 5 * G + + .quad 0x42c1192927f6bdcf + .quad 0x8f91917a403d61ca + .quad 0xdc1c5a668b9e1f61 + .quad 0x1596047804ec0f8d + .quad 0xecdfc35b48cade41 + .quad 0x6a88471fb2328270 + .quad 0x740a4a2440a01b6a + .quad 0x471e5796003b5f29 + .quad 0xda96bbb3aced37ac + .quad 0x7a2423b5e9208cea + .quad 0x24cc5c3038aebae2 + .quad 0x50c356afdc5dae2f + + // 2^220 * 6 * G + + .quad 0x09dcbf4341c30318 + .quad 0xeeba061183181dce + .quad 0xc179c0cedc1e29a1 + .quad 0x1dbf7b89073f35b0 + .quad 0xcfed9cdf1b31b964 + .quad 0xf486a9858ca51af3 + .quad 0x14897265ea8c1f84 + .quad 0x784a53dd932acc00 + .quad 0x2d99f9df14fc4920 + .quad 0x76ccb60cc4499fe5 + .quad 0xa4132cbbe5cf0003 + .quad 0x3f93d82354f000ea + + // 2^220 * 7 * G + + .quad 0x8183e7689e04ce85 + .quad 0x678fb71e04465341 + .quad 0xad92058f6688edac + .quad 0x5da350d3532b099a + .quad 0xeaac12d179e14978 + .quad 0xff923ff3bbebff5e + .quad 0x4af663e40663ce27 + .quad 0x0fd381a811a5f5ff + .quad 0xf256aceca436df54 + .quad 0x108b6168ae69d6e8 + .quad 0x20d986cb6b5d036c + .quad 0x655957b9fee2af50 + + // 2^220 * 8 * G + + .quad 0xaea8b07fa902030f + .quad 0xf88c766af463d143 + .quad 0x15b083663c787a60 + .quad 0x08eab1148267a4a8 + .quad 0xbdc1409bd002d0ac + .quad 0x66660245b5ccd9a6 + .quad 0x82317dc4fade85ec + .quad 0x02fe934b6ad7df0d + .quad 0xef5cf100cfb7ea74 + .quad 0x22897633a1cb42ac + .quad 0xd4ce0c54cef285e2 + .quad 0x30408c048a146a55 + + // 2^224 * 1 * G + + .quad 0x739d8845832fcedb + .quad 0xfa38d6c9ae6bf863 + .quad 0x32bc0dcab74ffef7 + .quad 0x73937e8814bce45e + .quad 0xbb2e00c9193b877f + .quad 0xece3a890e0dc506b + .quad 0xecf3b7c036de649f + .quad 0x5f46040898de9e1a + .quad 0xb9037116297bf48d + .quad 0xa9d13b22d4f06834 + .quad 0xe19715574696bdc6 + .quad 0x2cf8a4e891d5e835 + + // 2^224 * 2 * G + + .quad 0x6d93fd8707110f67 + .quad 0xdd4c09d37c38b549 + .quad 0x7cb16a4cc2736a86 + .quad 0x2049bd6e58252a09 + .quad 0x2cb5487e17d06ba2 + .quad 0x24d2381c3950196b + .quad 0xd7659c8185978a30 + .quad 0x7a6f7f2891d6a4f6 + .quad 0x7d09fd8d6a9aef49 + .quad 0xf0ee60be5b3db90b + .quad 0x4c21b52c519ebfd4 + .quad 0x6011aadfc545941d + + // 2^224 * 3 * G + + .quad 0x5f67926dcf95f83c + .quad 0x7c7e856171289071 + .quad 0xd6a1e7f3998f7a5b + .quad 0x6fc5cc1b0b62f9e0 + .quad 0x63ded0c802cbf890 + .quad 0xfbd098ca0dff6aaa + .quad 0x624d0afdb9b6ed99 + .quad 0x69ce18b779340b1e + .quad 0xd1ef5528b29879cb + .quad 0xdd1aae3cd47e9092 + .quad 0x127e0442189f2352 + .quad 0x15596b3ae57101f1 + + // 2^224 * 4 * G + + .quad 0x462739d23f9179a2 + .quad 0xff83123197d6ddcf + .quad 0x1307deb553f2148a + .quad 0x0d2237687b5f4dda + .quad 0x09ff31167e5124ca + .quad 0x0be4158bd9c745df + .quad 0x292b7d227ef556e5 + .quad 0x3aa4e241afb6d138 + .quad 0x2cc138bf2a3305f5 + .quad 0x48583f8fa2e926c3 + .quad 0x083ab1a25549d2eb + .quad 0x32fcaa6e4687a36c + + // 2^224 * 5 * G + + .quad 0x7bc56e8dc57d9af5 + .quad 0x3e0bd2ed9df0bdf2 + .quad 0xaac014de22efe4a3 + .quad 0x4627e9cefebd6a5c + .quad 0x3207a4732787ccdf + .quad 0x17e31908f213e3f8 + .quad 0xd5b2ecd7f60d964e + .quad 0x746f6336c2600be9 + .quad 0x3f4af345ab6c971c + .quad 0xe288eb729943731f + .quad 0x33596a8a0344186d + .quad 0x7b4917007ed66293 + + // 2^224 * 6 * G + + .quad 0x2d85fb5cab84b064 + .quad 0x497810d289f3bc14 + .quad 0x476adc447b15ce0c + .quad 0x122ba376f844fd7b + .quad 0x54341b28dd53a2dd + .quad 0xaa17905bdf42fc3f + .quad 0x0ff592d94dd2f8f4 + .quad 0x1d03620fe08cd37d + .quad 0xc20232cda2b4e554 + .quad 0x9ed0fd42115d187f + .quad 0x2eabb4be7dd479d9 + .quad 0x02c70bf52b68ec4c + + // 2^224 * 7 * G + + .quad 0xa287ec4b5d0b2fbb + .quad 0x415c5790074882ca + .quad 0xe044a61ec1d0815c + .quad 0x26334f0a409ef5e0 + .quad 0xace532bf458d72e1 + .quad 0x5be768e07cb73cb5 + .quad 0x56cf7d94ee8bbde7 + .quad 0x6b0697e3feb43a03 + .quad 0xb6c8f04adf62a3c0 + .quad 0x3ef000ef076da45d + .quad 0x9c9cb95849f0d2a9 + .quad 0x1cc37f43441b2fae + + // 2^224 * 8 * G + + .quad 0x508f565a5cc7324f + .quad 0xd061c4c0e506a922 + .quad 0xfb18abdb5c45ac19 + .quad 0x6c6809c10380314a + .quad 0xd76656f1c9ceaeb9 + .quad 0x1c5b15f818e5656a + .quad 0x26e72832844c2334 + .quad 0x3a346f772f196838 + .quad 0xd2d55112e2da6ac8 + .quad 0xe9bd0331b1e851ed + .quad 0x960746dd8ec67262 + .quad 0x05911b9f6ef7c5d0 + + // 2^228 * 1 * G + + .quad 0xe9dcd756b637ff2d + .quad 0xec4c348fc987f0c4 + .quad 0xced59285f3fbc7b7 + .quad 0x3305354793e1ea87 + .quad 0x01c18980c5fe9f94 + .quad 0xcd656769716fd5c8 + .quad 0x816045c3d195a086 + .quad 0x6e2b7f3266cc7982 + .quad 0xcc802468f7c3568f + .quad 0x9de9ba8219974cb3 + .quad 0xabb7229cb5b81360 + .quad 0x44e2017a6fbeba62 + + // 2^228 * 2 * G + + .quad 0xc4c2a74354dab774 + .quad 0x8e5d4c3c4eaf031a + .quad 0xb76c23d242838f17 + .quad 0x749a098f68dce4ea + .quad 0x87f82cf3b6ca6ecd + .quad 0x580f893e18f4a0c2 + .quad 0x058930072604e557 + .quad 0x6cab6ac256d19c1d + .quad 0xdcdfe0a02cc1de60 + .quad 0x032665ff51c5575b + .quad 0x2c0c32f1073abeeb + .quad 0x6a882014cd7b8606 + + // 2^228 * 3 * G + + .quad 0xa52a92fea4747fb5 + .quad 0xdc12a4491fa5ab89 + .quad 0xd82da94bb847a4ce + .quad 0x4d77edce9512cc4e + .quad 0xd111d17caf4feb6e + .quad 0x050bba42b33aa4a3 + .quad 0x17514c3ceeb46c30 + .quad 0x54bedb8b1bc27d75 + .quad 0x77c8e14577e2189c + .quad 0xa3e46f6aff99c445 + .quad 0x3144dfc86d335343 + .quad 0x3a96559e7c4216a9 + + // 2^228 * 4 * G + + .quad 0x12550d37f42ad2ee + .quad 0x8b78e00498a1fbf5 + .quad 0x5d53078233894cb2 + .quad 0x02c84e4e3e498d0c + .quad 0x4493896880baaa52 + .quad 0x4c98afc4f285940e + .quad 0xef4aa79ba45448b6 + .quad 0x5278c510a57aae7f + .quad 0xa54dd074294c0b94 + .quad 0xf55d46b8df18ffb6 + .quad 0xf06fecc58dae8366 + .quad 0x588657668190d165 + + // 2^228 * 5 * G + + .quad 0xd47712311aef7117 + .quad 0x50343101229e92c7 + .quad 0x7a95e1849d159b97 + .quad 0x2449959b8b5d29c9 + .quad 0xbf5834f03de25cc3 + .quad 0xb887c8aed6815496 + .quad 0x5105221a9481e892 + .quad 0x6760ed19f7723f93 + .quad 0x669ba3b7ac35e160 + .quad 0x2eccf73fba842056 + .quad 0x1aec1f17c0804f07 + .quad 0x0d96bc031856f4e7 + + // 2^228 * 6 * G + + .quad 0x3318be7775c52d82 + .quad 0x4cb764b554d0aab9 + .quad 0xabcf3d27cc773d91 + .quad 0x3bf4d1848123288a + .quad 0xb1d534b0cc7505e1 + .quad 0x32cd003416c35288 + .quad 0xcb36a5800762c29d + .quad 0x5bfe69b9237a0bf8 + .quad 0x183eab7e78a151ab + .quad 0xbbe990c999093763 + .quad 0xff717d6e4ac7e335 + .quad 0x4c5cddb325f39f88 + + // 2^228 * 7 * G + + .quad 0xc0f6b74d6190a6eb + .quad 0x20ea81a42db8f4e4 + .quad 0xa8bd6f7d97315760 + .quad 0x33b1d60262ac7c21 + .quad 0x57750967e7a9f902 + .quad 0x2c37fdfc4f5b467e + .quad 0xb261663a3177ba46 + .quad 0x3a375e78dc2d532b + .quad 0x8141e72f2d4dddea + .quad 0xe6eafe9862c607c8 + .quad 0x23c28458573cafd0 + .quad 0x46b9476f4ff97346 + + // 2^228 * 8 * G + + .quad 0x0c1ffea44f901e5c + .quad 0x2b0b6fb72184b782 + .quad 0xe587ff910114db88 + .quad 0x37130f364785a142 + .quad 0x1215505c0d58359f + .quad 0x2a2013c7fc28c46b + .quad 0x24a0a1af89ea664e + .quad 0x4400b638a1130e1f + .quad 0x3a01b76496ed19c3 + .quad 0x31e00ab0ed327230 + .quad 0x520a885783ca15b1 + .quad 0x06aab9875accbec7 + + // 2^232 * 1 * G + + .quad 0xc1339983f5df0ebb + .quad 0xc0f3758f512c4cac + .quad 0x2cf1130a0bb398e1 + .quad 0x6b3cecf9aa270c62 + .quad 0x5349acf3512eeaef + .quad 0x20c141d31cc1cb49 + .quad 0x24180c07a99a688d + .quad 0x555ef9d1c64b2d17 + .quad 0x36a770ba3b73bd08 + .quad 0x624aef08a3afbf0c + .quad 0x5737ff98b40946f2 + .quad 0x675f4de13381749d + + // 2^232 * 2 * G + + .quad 0x0e2c52036b1782fc + .quad 0x64816c816cad83b4 + .quad 0xd0dcbdd96964073e + .quad 0x13d99df70164c520 + .quad 0xa12ff6d93bdab31d + .quad 0x0725d80f9d652dfe + .quad 0x019c4ff39abe9487 + .quad 0x60f450b882cd3c43 + .quad 0x014b5ec321e5c0ca + .quad 0x4fcb69c9d719bfa2 + .quad 0x4e5f1c18750023a0 + .quad 0x1c06de9e55edac80 + + // 2^232 * 3 * G + + .quad 0x990f7ad6a33ec4e2 + .quad 0x6608f938be2ee08e + .quad 0x9ca143c563284515 + .quad 0x4cf38a1fec2db60d + .quad 0xffd52b40ff6d69aa + .quad 0x34530b18dc4049bb + .quad 0x5e4a5c2fa34d9897 + .quad 0x78096f8e7d32ba2d + .quad 0xa0aaaa650dfa5ce7 + .quad 0xf9c49e2a48b5478c + .quad 0x4f09cc7d7003725b + .quad 0x373cad3a26091abe + + // 2^232 * 4 * G + + .quad 0xb294634d82c9f57c + .quad 0x1fcbfde124934536 + .quad 0x9e9c4db3418cdb5a + .quad 0x0040f3d9454419fc + .quad 0xf1bea8fb89ddbbad + .quad 0x3bcb2cbc61aeaecb + .quad 0x8f58a7bb1f9b8d9d + .quad 0x21547eda5112a686 + .quad 0xdefde939fd5986d3 + .quad 0xf4272c89510a380c + .quad 0xb72ba407bb3119b9 + .quad 0x63550a334a254df4 + + // 2^232 * 5 * G + + .quad 0x6507d6edb569cf37 + .quad 0x178429b00ca52ee1 + .quad 0xea7c0090eb6bd65d + .quad 0x3eea62c7daf78f51 + .quad 0x9bba584572547b49 + .quad 0xf305c6fae2c408e0 + .quad 0x60e8fa69c734f18d + .quad 0x39a92bafaa7d767a + .quad 0x9d24c713e693274e + .quad 0x5f63857768dbd375 + .quad 0x70525560eb8ab39a + .quad 0x68436a0665c9c4cd + + // 2^232 * 6 * G + + .quad 0xbc0235e8202f3f27 + .quad 0xc75c00e264f975b0 + .quad 0x91a4e9d5a38c2416 + .quad 0x17b6e7f68ab789f9 + .quad 0x1e56d317e820107c + .quad 0xc5266844840ae965 + .quad 0xc1e0a1c6320ffc7a + .quad 0x5373669c91611472 + .quad 0x5d2814ab9a0e5257 + .quad 0x908f2084c9cab3fc + .quad 0xafcaf5885b2d1eca + .quad 0x1cb4b5a678f87d11 + + // 2^232 * 7 * G + + .quad 0xb664c06b394afc6c + .quad 0x0c88de2498da5fb1 + .quad 0x4f8d03164bcad834 + .quad 0x330bca78de7434a2 + .quad 0x6b74aa62a2a007e7 + .quad 0xf311e0b0f071c7b1 + .quad 0x5707e438000be223 + .quad 0x2dc0fd2d82ef6eac + .quad 0x982eff841119744e + .quad 0xf9695e962b074724 + .quad 0xc58ac14fbfc953fb + .quad 0x3c31be1b369f1cf5 + + // 2^232 * 8 * G + + .quad 0xb0f4864d08948aee + .quad 0x07dc19ee91ba1c6f + .quad 0x7975cdaea6aca158 + .quad 0x330b61134262d4bb + .quad 0xc168bc93f9cb4272 + .quad 0xaeb8711fc7cedb98 + .quad 0x7f0e52aa34ac8d7a + .quad 0x41cec1097e7d55bb + .quad 0xf79619d7a26d808a + .quad 0xbb1fd49e1d9e156d + .quad 0x73d7c36cdba1df27 + .quad 0x26b44cd91f28777d + + // 2^236 * 1 * G + + .quad 0x300a9035393aa6d8 + .quad 0x2b501131a12bb1cd + .quad 0x7b1ff677f093c222 + .quad 0x4309c1f8cab82bad + .quad 0xaf44842db0285f37 + .quad 0x8753189047efc8df + .quad 0x9574e091f820979a + .quad 0x0e378d6069615579 + .quad 0xd9fa917183075a55 + .quad 0x4bdb5ad26b009fdc + .quad 0x7829ad2cd63def0e + .quad 0x078fc54975fd3877 + + // 2^236 * 2 * G + + .quad 0x87dfbd1428878f2d + .quad 0x134636dd1e9421a1 + .quad 0x4f17c951257341a3 + .quad 0x5df98d4bad296cb8 + .quad 0xe2004b5bb833a98a + .quad 0x44775dec2d4c3330 + .quad 0x3aa244067eace913 + .quad 0x272630e3d58e00a9 + .quad 0xf3678fd0ecc90b54 + .quad 0xf001459b12043599 + .quad 0x26725fbc3758b89b + .quad 0x4325e4aa73a719ae + + // 2^236 * 3 * G + + .quad 0x657dc6ef433c3493 + .quad 0x65375e9f80dbf8c3 + .quad 0x47fd2d465b372dae + .quad 0x4966ab79796e7947 + .quad 0xed24629acf69f59d + .quad 0x2a4a1ccedd5abbf4 + .quad 0x3535ca1f56b2d67b + .quad 0x5d8c68d043b1b42d + .quad 0xee332d4de3b42b0a + .quad 0xd84e5a2b16a4601c + .quad 0x78243877078ba3e4 + .quad 0x77ed1eb4184ee437 + + // 2^236 * 4 * G + + .quad 0xbfd4e13f201839a0 + .quad 0xaeefffe23e3df161 + .quad 0xb65b04f06b5d1fe3 + .quad 0x52e085fb2b62fbc0 + .quad 0x185d43f89e92ed1a + .quad 0xb04a1eeafe4719c6 + .quad 0x499fbe88a6f03f4f + .quad 0x5d8b0d2f3c859bdd + .quad 0x124079eaa54cf2ba + .quad 0xd72465eb001b26e7 + .quad 0x6843bcfdc97af7fd + .quad 0x0524b42b55eacd02 + + // 2^236 * 5 * G + + .quad 0xfd0d5dbee45447b0 + .quad 0x6cec351a092005ee + .quad 0x99a47844567579cb + .quad 0x59d242a216e7fa45 + .quad 0xbc18dcad9b829eac + .quad 0x23ae7d28b5f579d0 + .quad 0xc346122a69384233 + .quad 0x1a6110b2e7d4ac89 + .quad 0x4f833f6ae66997ac + .quad 0x6849762a361839a4 + .quad 0x6985dec1970ab525 + .quad 0x53045e89dcb1f546 + + // 2^236 * 6 * G + + .quad 0xcb8bb346d75353db + .quad 0xfcfcb24bae511e22 + .quad 0xcba48d40d50ae6ef + .quad 0x26e3bae5f4f7cb5d + .quad 0x84da3cde8d45fe12 + .quad 0xbd42c218e444e2d2 + .quad 0xa85196781f7e3598 + .quad 0x7642c93f5616e2b2 + .quad 0x2323daa74595f8e4 + .quad 0xde688c8b857abeb4 + .quad 0x3fc48e961c59326e + .quad 0x0b2e73ca15c9b8ba + + // 2^236 * 7 * G + + .quad 0xd6bb4428c17f5026 + .quad 0x9eb27223fb5a9ca7 + .quad 0xe37ba5031919c644 + .quad 0x21ce380db59a6602 + .quad 0x0e3fbfaf79c03a55 + .quad 0x3077af054cbb5acf + .quad 0xd5c55245db3de39f + .quad 0x015e68c1476a4af7 + .quad 0xc1d5285220066a38 + .quad 0x95603e523570aef3 + .quad 0x832659a7226b8a4d + .quad 0x5dd689091f8eedc9 + + // 2^236 * 8 * G + + .quad 0xcbac84debfd3c856 + .quad 0x1624c348b35ff244 + .quad 0xb7f88dca5d9cad07 + .quad 0x3b0e574da2c2ebe8 + .quad 0x1d022591a5313084 + .quad 0xca2d4aaed6270872 + .quad 0x86a12b852f0bfd20 + .quad 0x56e6c439ad7da748 + .quad 0xc704ff4942bdbae6 + .quad 0x5e21ade2b2de1f79 + .quad 0xe95db3f35652fad8 + .quad 0x0822b5378f08ebc1 + + // 2^240 * 1 * G + + .quad 0x51f048478f387475 + .quad 0xb25dbcf49cbecb3c + .quad 0x9aab1244d99f2055 + .quad 0x2c709e6c1c10a5d6 + .quad 0xe1b7f29362730383 + .quad 0x4b5279ffebca8a2c + .quad 0xdafc778abfd41314 + .quad 0x7deb10149c72610f + .quad 0xcb62af6a8766ee7a + .quad 0x66cbec045553cd0e + .quad 0x588001380f0be4b5 + .quad 0x08e68e9ff62ce2ea + + // 2^240 * 2 * G + + .quad 0x34ad500a4bc130ad + .quad 0x8d38db493d0bd49c + .quad 0xa25c3d98500a89be + .quad 0x2f1f3f87eeba3b09 + .quad 0x2f2d09d50ab8f2f9 + .quad 0xacb9218dc55923df + .quad 0x4a8f342673766cb9 + .quad 0x4cb13bd738f719f5 + .quad 0xf7848c75e515b64a + .quad 0xa59501badb4a9038 + .quad 0xc20d313f3f751b50 + .quad 0x19a1e353c0ae2ee8 + + // 2^240 * 3 * G + + .quad 0x7d1c7560bafa05c3 + .quad 0xb3e1a0a0c6e55e61 + .quad 0xe3529718c0d66473 + .quad 0x41546b11c20c3486 + .quad 0xb42172cdd596bdbd + .quad 0x93e0454398eefc40 + .quad 0x9fb15347b44109b5 + .quad 0x736bd3990266ae34 + .quad 0x85532d509334b3b4 + .quad 0x46fd114b60816573 + .quad 0xcc5f5f30425c8375 + .quad 0x412295a2b87fab5c + + // 2^240 * 4 * G + + .quad 0x19c99b88f57ed6e9 + .quad 0x5393cb266df8c825 + .quad 0x5cee3213b30ad273 + .quad 0x14e153ebb52d2e34 + .quad 0x2e655261e293eac6 + .quad 0x845a92032133acdb + .quad 0x460975cb7900996b + .quad 0x0760bb8d195add80 + .quad 0x413e1a17cde6818a + .quad 0x57156da9ed69a084 + .quad 0x2cbf268f46caccb1 + .quad 0x6b34be9bc33ac5f2 + + // 2^240 * 5 * G + + .quad 0xf3df2f643a78c0b2 + .quad 0x4c3e971ef22e027c + .quad 0xec7d1c5e49c1b5a3 + .quad 0x2012c18f0922dd2d + .quad 0x11fc69656571f2d3 + .quad 0xc6c9e845530e737a + .quad 0xe33ae7a2d4fe5035 + .quad 0x01b9c7b62e6dd30b + .quad 0x880b55e55ac89d29 + .quad 0x1483241f45a0a763 + .quad 0x3d36efdfc2e76c1f + .quad 0x08af5b784e4bade8 + + // 2^240 * 6 * G + + .quad 0x283499dc881f2533 + .quad 0x9d0525da779323b6 + .quad 0x897addfb673441f4 + .quad 0x32b79d71163a168d + .quad 0xe27314d289cc2c4b + .quad 0x4be4bd11a287178d + .quad 0x18d528d6fa3364ce + .quad 0x6423c1d5afd9826e + .quad 0xcc85f8d9edfcb36a + .quad 0x22bcc28f3746e5f9 + .quad 0xe49de338f9e5d3cd + .quad 0x480a5efbc13e2dcc + + // 2^240 * 7 * G + + .quad 0x0b51e70b01622071 + .quad 0x06b505cf8b1dafc5 + .quad 0x2c6bb061ef5aabcd + .quad 0x47aa27600cb7bf31 + .quad 0xb6614ce442ce221f + .quad 0x6e199dcc4c053928 + .quad 0x663fb4a4dc1cbe03 + .quad 0x24b31d47691c8e06 + .quad 0x2a541eedc015f8c3 + .quad 0x11a4fe7e7c693f7c + .quad 0xf0af66134ea278d6 + .quad 0x545b585d14dda094 + + // 2^240 * 8 * G + + .quad 0x67bf275ea0d43a0f + .quad 0xade68e34089beebe + .quad 0x4289134cd479e72e + .quad 0x0f62f9c332ba5454 + .quad 0x6204e4d0e3b321e1 + .quad 0x3baa637a28ff1e95 + .quad 0x0b0ccffd5b99bd9e + .quad 0x4d22dc3e64c8d071 + .quad 0xfcb46589d63b5f39 + .quad 0x5cae6a3f57cbcf61 + .quad 0xfebac2d2953afa05 + .quad 0x1c0fa01a36371436 + + // 2^244 * 1 * G + + .quad 0xe7547449bc7cd692 + .quad 0x0f9abeaae6f73ddf + .quad 0x4af01ca700837e29 + .quad 0x63ab1b5d3f1bc183 + .quad 0xc11ee5e854c53fae + .quad 0x6a0b06c12b4f3ff4 + .quad 0x33540f80e0b67a72 + .quad 0x15f18fc3cd07e3ef + .quad 0x32750763b028f48c + .quad 0x06020740556a065f + .quad 0xd53bd812c3495b58 + .quad 0x08706c9b865f508d + + // 2^244 * 2 * G + + .quad 0xf37ca2ab3d343dff + .quad 0x1a8c6a2d80abc617 + .quad 0x8e49e035d4ccffca + .quad 0x48b46beebaa1d1b9 + .quad 0xcc991b4138b41246 + .quad 0x243b9c526f9ac26b + .quad 0xb9ef494db7cbabbd + .quad 0x5fba433dd082ed00 + .quad 0x9c49e355c9941ad0 + .quad 0xb9734ade74498f84 + .quad 0x41c3fed066663e5c + .quad 0x0ecfedf8e8e710b3 + + // 2^244 * 3 * G + + .quad 0x76430f9f9cd470d9 + .quad 0xb62acc9ba42f6008 + .quad 0x1898297c59adad5e + .quad 0x7789dd2db78c5080 + .quad 0x744f7463e9403762 + .quad 0xf79a8dee8dfcc9c9 + .quad 0x163a649655e4cde3 + .quad 0x3b61788db284f435 + .quad 0xb22228190d6ef6b2 + .quad 0xa94a66b246ce4bfa + .quad 0x46c1a77a4f0b6cc7 + .quad 0x4236ccffeb7338cf + + // 2^244 * 4 * G + + .quad 0x8497404d0d55e274 + .quad 0x6c6663d9c4ad2b53 + .quad 0xec2fb0d9ada95734 + .quad 0x2617e120cdb8f73c + .quad 0x3bd82dbfda777df6 + .quad 0x71b177cc0b98369e + .quad 0x1d0e8463850c3699 + .quad 0x5a71945b48e2d1f1 + .quad 0x6f203dd5405b4b42 + .quad 0x327ec60410b24509 + .quad 0x9c347230ac2a8846 + .quad 0x77de29fc11ffeb6a + + // 2^244 * 5 * G + + .quad 0xb0ac57c983b778a8 + .quad 0x53cdcca9d7fe912c + .quad 0x61c2b854ff1f59dc + .quad 0x3a1a2cf0f0de7dac + .quad 0x835e138fecced2ca + .quad 0x8c9eaf13ea963b9a + .quad 0xc95fbfc0b2160ea6 + .quad 0x575e66f3ad877892 + .quad 0x99803a27c88fcb3a + .quad 0x345a6789275ec0b0 + .quad 0x459789d0ff6c2be5 + .quad 0x62f882651e70a8b2 + + // 2^244 * 6 * G + + .quad 0x085ae2c759ff1be4 + .quad 0x149145c93b0e40b7 + .quad 0xc467e7fa7ff27379 + .quad 0x4eeecf0ad5c73a95 + .quad 0x6d822986698a19e0 + .quad 0xdc9821e174d78a71 + .quad 0x41a85f31f6cb1f47 + .quad 0x352721c2bcda9c51 + .quad 0x48329952213fc985 + .quad 0x1087cf0d368a1746 + .quad 0x8e5261b166c15aa5 + .quad 0x2d5b2d842ed24c21 + + // 2^244 * 7 * G + + .quad 0x02cfebd9ebd3ded1 + .quad 0xd45b217739021974 + .quad 0x7576f813fe30a1b7 + .quad 0x5691b6f9a34ef6c2 + .quad 0x5eb7d13d196ac533 + .quad 0x377234ecdb80be2b + .quad 0xe144cffc7cf5ae24 + .quad 0x5226bcf9c441acec + .quad 0x79ee6c7223e5b547 + .quad 0x6f5f50768330d679 + .quad 0xed73e1e96d8adce9 + .quad 0x27c3da1e1d8ccc03 + + // 2^244 * 8 * G + + .quad 0x7eb9efb23fe24c74 + .quad 0x3e50f49f1651be01 + .quad 0x3ea732dc21858dea + .quad 0x17377bd75bb810f9 + .quad 0x28302e71630ef9f6 + .quad 0xc2d4a2032b64cee0 + .quad 0x090820304b6292be + .quad 0x5fca747aa82adf18 + .quad 0x232a03c35c258ea5 + .quad 0x86f23a2c6bcb0cf1 + .quad 0x3dad8d0d2e442166 + .quad 0x04a8933cab76862b + + // 2^248 * 1 * G + + .quad 0xd2c604b622943dff + .quad 0xbc8cbece44cfb3a0 + .quad 0x5d254ff397808678 + .quad 0x0fa3614f3b1ca6bf + .quad 0x69082b0e8c936a50 + .quad 0xf9c9a035c1dac5b6 + .quad 0x6fb73e54c4dfb634 + .quad 0x4005419b1d2bc140 + .quad 0xa003febdb9be82f0 + .quad 0x2089c1af3a44ac90 + .quad 0xf8499f911954fa8e + .quad 0x1fba218aef40ab42 + + // 2^248 * 2 * G + + .quad 0xab549448fac8f53e + .quad 0x81f6e89a7ba63741 + .quad 0x74fd6c7d6c2b5e01 + .quad 0x392e3acaa8c86e42 + .quad 0x4f3e57043e7b0194 + .quad 0xa81d3eee08daaf7f + .quad 0xc839c6ab99dcdef1 + .quad 0x6c535d13ff7761d5 + .quad 0x4cbd34e93e8a35af + .quad 0x2e0781445887e816 + .quad 0x19319c76f29ab0ab + .quad 0x25e17fe4d50ac13b + + // 2^248 * 3 * G + + .quad 0x0a289bd71e04f676 + .quad 0x208e1c52d6420f95 + .quad 0x5186d8b034691fab + .quad 0x255751442a9fb351 + .quad 0x915f7ff576f121a7 + .quad 0xc34a32272fcd87e3 + .quad 0xccba2fde4d1be526 + .quad 0x6bba828f8969899b + .quad 0xe2d1bc6690fe3901 + .quad 0x4cb54a18a0997ad5 + .quad 0x971d6914af8460d4 + .quad 0x559d504f7f6b7be4 + + // 2^248 * 4 * G + + .quad 0xa7738378b3eb54d5 + .quad 0x1d69d366a5553c7c + .quad 0x0a26cf62f92800ba + .quad 0x01ab12d5807e3217 + .quad 0x9c4891e7f6d266fd + .quad 0x0744a19b0307781b + .quad 0x88388f1d6061e23b + .quad 0x123ea6a3354bd50e + .quad 0x118d189041e32d96 + .quad 0xb9ede3c2d8315848 + .quad 0x1eab4271d83245d9 + .quad 0x4a3961e2c918a154 + + // 2^248 * 5 * G + + .quad 0x71dc3be0f8e6bba0 + .quad 0xd6cef8347effe30a + .quad 0xa992425fe13a476a + .quad 0x2cd6bce3fb1db763 + .quad 0x0327d644f3233f1e + .quad 0x499a260e34fcf016 + .quad 0x83b5a716f2dab979 + .quad 0x68aceead9bd4111f + .quad 0x38b4c90ef3d7c210 + .quad 0x308e6e24b7ad040c + .quad 0x3860d9f1b7e73e23 + .quad 0x595760d5b508f597 + + // 2^248 * 6 * G + + .quad 0x6129bfe104aa6397 + .quad 0x8f960008a4a7fccb + .quad 0x3f8bc0897d909458 + .quad 0x709fa43edcb291a9 + .quad 0x882acbebfd022790 + .quad 0x89af3305c4115760 + .quad 0x65f492e37d3473f4 + .quad 0x2cb2c5df54515a2b + .quad 0xeb0a5d8c63fd2aca + .quad 0xd22bc1662e694eff + .quad 0x2723f36ef8cbb03a + .quad 0x70f029ecf0c8131f + + // 2^248 * 7 * G + + .quad 0x461307b32eed3e33 + .quad 0xae042f33a45581e7 + .quad 0xc94449d3195f0366 + .quad 0x0b7d5d8a6c314858 + .quad 0x2a6aafaa5e10b0b9 + .quad 0x78f0a370ef041aa9 + .quad 0x773efb77aa3ad61f + .quad 0x44eca5a2a74bd9e1 + .quad 0x25d448327b95d543 + .quad 0x70d38300a3340f1d + .quad 0xde1c531c60e1c52b + .quad 0x272224512c7de9e4 + + // 2^248 * 8 * G + + .quad 0x1abc92af49c5342e + .quad 0xffeed811b2e6fad0 + .quad 0xefa28c8dfcc84e29 + .quad 0x11b5df18a44cc543 + .quad 0xbf7bbb8a42a975fc + .quad 0x8c5c397796ada358 + .quad 0xe27fc76fcdedaa48 + .quad 0x19735fd7f6bc20a6 + .quad 0xe3ab90d042c84266 + .quad 0xeb848e0f7f19547e + .quad 0x2503a1d065a497b9 + .quad 0x0fef911191df895f + + // 2^252 * 1 * G + + .quad 0xb1507ca1ab1c6eb9 + .quad 0xbd448f3e16b687b3 + .quad 0x3455fb7f2c7a91ab + .quad 0x7579229e2f2adec1 + .quad 0x6ab5dcb85b1c16b7 + .quad 0x94c0fce83c7b27a5 + .quad 0xa4b11c1a735517be + .quad 0x499238d0ba0eafaa + .quad 0xecf46e527aba8b57 + .quad 0x15a08c478bd1647b + .quad 0x7af1c6a65f706fef + .quad 0x6345fa78f03a30d5 + + // 2^252 * 2 * G + + .quad 0xdf02f95f1015e7a1 + .quad 0x790ec41da9b40263 + .quad 0x4d3a0ea133ea1107 + .quad 0x54f70be7e33af8c9 + .quad 0x93d3cbe9bdd8f0a4 + .quad 0xdb152c1bfd177302 + .quad 0x7dbddc6d7f17a875 + .quad 0x3e1a71cc8f426efe + .quad 0xc83ca3e390babd62 + .quad 0x80ede3670291c833 + .quad 0xc88038ccd37900c4 + .quad 0x2c5fc0231ec31fa1 + + // 2^252 * 3 * G + + .quad 0xfeba911717038b4f + .quad 0xe5123721c9deef81 + .quad 0x1c97e4e75d0d8834 + .quad 0x68afae7a23dc3bc6 + .quad 0xc422e4d102456e65 + .quad 0x87414ac1cad47b91 + .quad 0x1592e2bba2b6ffdd + .quad 0x75d9d2bff5c2100f + .quad 0x5bd9b4763626e81c + .quad 0x89966936bca02edd + .quad 0x0a41193d61f077b3 + .quad 0x3097a24200ce5471 + + // 2^252 * 4 * G + + .quad 0x57427734c7f8b84c + .quad 0xf141a13e01b270e9 + .quad 0x02d1adfeb4e564a6 + .quad 0x4bb23d92ce83bd48 + .quad 0xa162e7246695c486 + .quad 0x131d633435a89607 + .quad 0x30521561a0d12a37 + .quad 0x56704bada6afb363 + .quad 0xaf6c4aa752f912b9 + .quad 0x5e665f6cd86770c8 + .quad 0x4c35ac83a3c8cd58 + .quad 0x2b7a29c010a58a7e + + // 2^252 * 5 * G + + .quad 0xc4007f77d0c1cec3 + .quad 0x8d1020b6bac492f8 + .quad 0x32ec29d57e69daaf + .quad 0x599408759d95fce0 + .quad 0x33810a23bf00086e + .quad 0xafce925ee736ff7c + .quad 0x3d60e670e24922d4 + .quad 0x11ce9e714f96061b + .quad 0x219ef713d815bac1 + .quad 0xf141465d485be25c + .quad 0x6d5447cc4e513c51 + .quad 0x174926be5ef44393 + + // 2^252 * 6 * G + + .quad 0xb5deb2f9fc5bd5bb + .quad 0x92daa72ae1d810e1 + .quad 0xafc4cfdcb72a1c59 + .quad 0x497d78813fc22a24 + .quad 0x3ef5d41593ea022e + .quad 0x5cbcc1a20ed0eed6 + .quad 0x8fd24ecf07382c8c + .quad 0x6fa42ead06d8e1ad + .quad 0xe276824a1f73371f + .quad 0x7f7cf01c4f5b6736 + .quad 0x7e201fe304fa46e7 + .quad 0x785a36a357808c96 + + // 2^252 * 7 * G + + .quad 0x825fbdfd63014d2b + .quad 0xc852369c6ca7578b + .quad 0x5b2fcd285c0b5df0 + .quad 0x12ab214c58048c8f + .quad 0x070442985d517bc3 + .quad 0x6acd56c7ae653678 + .quad 0x00a27983985a7763 + .quad 0x5167effae512662b + .quad 0xbd4ea9e10f53c4b6 + .quad 0x1673dc5f8ac91a14 + .quad 0xa8f81a4e2acc1aba + .quad 0x33a92a7924332a25 + + // 2^252 * 8 * G + + .quad 0x9dd1f49927996c02 + .quad 0x0cb3b058e04d1752 + .quad 0x1f7e88967fd02c3e + .quad 0x2f964268cb8b3eb1 + .quad 0x7ba95ba0218f2ada + .quad 0xcff42287330fb9ca + .quad 0xdada496d56c6d907 + .quad 0x5380c296f4beee54 + .quad 0x9d4f270466898d0a + .quad 0x3d0987990aff3f7a + .quad 0xd09ef36267daba45 + .quad 0x7761455e7b1c669c diff --git a/cbits/s2n/x86_att/edwards25519_encode.S b/cbits/s2n/x86_att/edwards25519_encode.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/edwards25519_encode.S @@ -0,0 +1,86 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Encode edwards25519 point into compressed form as 256-bit number +// Input p[8]; output z[32] (bytes) +// +// extern void edwards25519_encode(uint8_t z[static 32], +// const uint64_t p[static 8]); +// +// This assumes that the input buffer p points to a pair of 256-bit +// numbers x (at p) and y (at p+4) representing a point (x,y) on the +// edwards25519 curve. It is assumed that both x and y are < p_25519 +// but there is no checking of this, nor of the fact that (x,y) is +// in fact on the curve. +// +// The output in z is a little-endian array of bytes corresponding to +// the standard compressed encoding of a point as 2^255 * x_0 + y +// where x_0 is the least significant bit of x. +// See "https://datatracker.ietf.org/doc/html/rfc8032#section-5.1.2" +// In this implementation, y is simply truncated to 255 bits, but if +// it is reduced mod p_25519 as expected this does not affect values. +// +// Standard x86-64 ABI: RDI = z, RSI = p +// Microsoft x64 ABI: RCX = z, RDX = p +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(edwards25519_encode) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(edwards25519_encode) + S2N_BN_SYM_PRIVACY_DIRECTIVE(edwards25519_encode) + .text + +#define z %rdi +#define p %rsi +#define y0 %rax +#define y1 %rcx +#define y2 %rdx +#define y3 %r8 +#define xb %r9 + +S2N_BN_SYMBOL(edwards25519_encode): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Load lowest word of x coordinate in xb and full y as [y3;y2;y1;y0]. + + movq (p), xb + movq 32(p), y0 + movq 40(p), y1 + movq 48(p), y2 + movq 56(p), y3 + +// Compute the encoded form, making the LSB of x the MSB of the encoding + + btr $63, y3 + shlq $63, xb + orq xb, y3 + +// Store back (by the word, since x86 is little-endian anyway) + + movq y0, (z) + movq y1, 8(z) + movq y2, 16(z) + movq y3, 24(z) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(edwards25519_encode) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack,"",%progbits +#endif diff --git a/cbits/s2n/x86_att/edwards25519_scalarmulbase.S b/cbits/s2n/x86_att/edwards25519_scalarmulbase.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/edwards25519_scalarmulbase.S @@ -0,0 +1,9926 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Scalar multiplication for the edwards25519 standard basepoint +// Input scalar[4]; output res[8] +// +// extern void edwards25519_scalarmulbase +// (uint64_t res[static 8],const uint64_t scalar[static 4]); +// +// Given a scalar n, returns point (X,Y) = n * B where B = (...,4/5) is +// the standard basepoint for the edwards25519 (Ed25519) curve. +// +// Standard x86-64 ABI: RDI = res, RSI = scalar +// Microsoft x64 ABI: RCX = res, RDX = scalar +// ---------------------------------------------------------------------------- +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(edwards25519_scalarmulbase) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(edwards25519_scalarmulbase) + S2N_BN_SYM_PRIVACY_DIRECTIVE(edwards25519_scalarmulbase) + + .text + .balign 32 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Pointer-offset pairs for result and temporaries on stack with some aliasing. +// The result "resx" assumes the "res" pointer has been preloaded into %rbp. + +#define resx (0*NUMSIZE)(%rbp) +#define resy (1*NUMSIZE)(%rbp) + +#define scalar (0*NUMSIZE)(%rsp) + +#define tabent (1*NUMSIZE)(%rsp) +#define ymx_2 (1*NUMSIZE)(%rsp) +#define xpy_2 (2*NUMSIZE)(%rsp) +#define kxy_2 (3*NUMSIZE)(%rsp) + +#define t0 (4*NUMSIZE)(%rsp) +#define t1 (5*NUMSIZE)(%rsp) +#define t2 (6*NUMSIZE)(%rsp) +#define t3 (7*NUMSIZE)(%rsp) +#define t4 (8*NUMSIZE)(%rsp) +#define t5 (9*NUMSIZE)(%rsp) + +#define acc (10*NUMSIZE)(%rsp) +#define x_1 (10*NUMSIZE)(%rsp) +#define y_1 (11*NUMSIZE)(%rsp) +#define z_1 (12*NUMSIZE)(%rsp) +#define w_1 (13*NUMSIZE)(%rsp) +#define x_3 (10*NUMSIZE)(%rsp) +#define y_3 (11*NUMSIZE)(%rsp) +#define z_3 (12*NUMSIZE)(%rsp) +#define w_3 (13*NUMSIZE)(%rsp) + +// Stable homes for the input result pointer, and other variables + +#define res 14*NUMSIZE(%rsp) + +#define i 14*NUMSIZE+8(%rsp) + +#define bias 14*NUMSIZE+16(%rsp) + +#define bf 14*NUMSIZE+24(%rsp) +#define ix 14*NUMSIZE+24(%rsp) + +#define tab 15*NUMSIZE(%rsp) + +// Total size to reserve on the stack + +#define NSPACE 488 + +// Syntactic variants to make x86_att version simpler to generate + +#define SCALAR 0 +#define TABENT (1*NUMSIZE) +#define ACC (10*NUMSIZE) +#define X3 (10*NUMSIZE) +#define Z3 (12*NUMSIZE) +#define W3 (13*NUMSIZE) + +// Macro wrapping up the basic field multiplication, only trivially +// different from a pure function call to bignum_mul_p25519. + +#define mul_p25519(P0,P1,P2) \ + xorl %esi, %esi ; \ + movq P2, %rdx ; \ + mulxq P1, %r8, %r9 ; \ + mulxq 0x8+P1, %rax, %r10 ; \ + addq %rax, %r9 ; \ + mulxq 0x10+P1, %rax, %r11 ; \ + adcq %rax, %r10 ; \ + mulxq 0x18+P1, %rax, %r12 ; \ + adcq %rax, %r11 ; \ + adcq %rsi, %r12 ; \ + xorl %esi, %esi ; \ + movq 0x8+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x18+P1, %rax, %r13 ; \ + adcxq %rax, %r12 ; \ + adoxq %rsi, %r13 ; \ + adcxq %rsi, %r13 ; \ + xorl %esi, %esi ; \ + movq 0x10+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x18+P1, %rax, %r14 ; \ + adcxq %rax, %r13 ; \ + adoxq %rsi, %r14 ; \ + adcxq %rsi, %r14 ; \ + xorl %esi, %esi ; \ + movq 0x18+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + mulxq 0x18+P1, %rax, %r15 ; \ + adcxq %rax, %r14 ; \ + adoxq %rsi, %r15 ; \ + adcxq %rsi, %r15 ; \ + movl $0x26, %edx ; \ + xorl %esi, %esi ; \ + mulxq %r12, %rax, %rbx ; \ + adcxq %rax, %r8 ; \ + adoxq %rbx, %r9 ; \ + mulxq %r13, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq %r14, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq %r15, %rax, %r12 ; \ + adcxq %rax, %r11 ; \ + adoxq %rsi, %r12 ; \ + adcxq %rsi, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + movl $0x13, %edx ; \ + incq %r12; \ + bts $63, %r11 ; \ + mulxq %r12, %rax, %rbx ; \ + addq %rax, %r8 ; \ + adcq %rbx, %r9 ; \ + adcq %rsi, %r10 ; \ + adcq %rsi, %r11 ; \ + sbbq %rax, %rax ; \ + notq %rax; \ + andq %rdx, %rax ; \ + subq %rax, %r8 ; \ + sbbq %rsi, %r9 ; \ + sbbq %rsi, %r10 ; \ + sbbq %rsi, %r11 ; \ + btr $63, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// A version of multiplication that only guarantees output < 2 * p_25519. +// This basically skips the +1 and final correction in quotient estimation. + +#define mul_4(P0,P1,P2) \ + xorl %ecx, %ecx ; \ + movq P2, %rdx ; \ + mulxq P1, %r8, %r9 ; \ + mulxq 0x8+P1, %rax, %r10 ; \ + addq %rax, %r9 ; \ + mulxq 0x10+P1, %rax, %r11 ; \ + adcq %rax, %r10 ; \ + mulxq 0x18+P1, %rax, %r12 ; \ + adcq %rax, %r11 ; \ + adcq %rcx, %r12 ; \ + xorl %ecx, %ecx ; \ + movq 0x8+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x18+P1, %rax, %r13 ; \ + adcxq %rax, %r12 ; \ + adoxq %rcx, %r13 ; \ + adcxq %rcx, %r13 ; \ + xorl %ecx, %ecx ; \ + movq 0x10+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x18+P1, %rax, %r14 ; \ + adcxq %rax, %r13 ; \ + adoxq %rcx, %r14 ; \ + adcxq %rcx, %r14 ; \ + xorl %ecx, %ecx ; \ + movq 0x18+P2, %rdx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + mulxq 0x18+P1, %rax, %r15 ; \ + adcxq %rax, %r14 ; \ + adoxq %rcx, %r15 ; \ + adcxq %rcx, %r15 ; \ + movl $0x26, %edx ; \ + xorl %ecx, %ecx ; \ + mulxq %r12, %rax, %rbx ; \ + adcxq %rax, %r8 ; \ + adoxq %rbx, %r9 ; \ + mulxq %r13, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq %r14, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq %r15, %rax, %r12 ; \ + adcxq %rax, %r11 ; \ + adoxq %rcx, %r12 ; \ + adcxq %rcx, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + btr $0x3f, %r11 ; \ + movl $0x13, %edx ; \ + imulq %r12, %rdx ; \ + addq %rdx, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Modular subtraction with double modulus 2 * p_25519 = 2^256 - 38 + +#define sub_twice4(P0,P1,P2) \ + movq P1, %r8 ; \ + xorl %ebx, %ebx ; \ + subq P2, %r8 ; \ + movq 8+P1, %r9 ; \ + sbbq 8+P2, %r9 ; \ + movl $38, %ecx ; \ + movq 16+P1, %r10 ; \ + sbbq 16+P2, %r10 ; \ + movq 24+P1, %rax ; \ + sbbq 24+P2, %rax ; \ + cmovncq %rbx, %rcx ; \ + subq %rcx, %r8 ; \ + sbbq %rbx, %r9 ; \ + sbbq %rbx, %r10 ; \ + sbbq %rbx, %rax ; \ + movq %r8, P0 ; \ + movq %r9, 8+P0 ; \ + movq %r10, 16+P0 ; \ + movq %rax, 24+P0 + +// Modular addition and doubling with double modulus 2 * p_25519 = 2^256 - 38. +// This only ensures that the result fits in 4 digits, not that it is reduced +// even w.r.t. double modulus. The result is always correct modulo provided +// the sum of the inputs is < 2^256 + 2^256 - 38, so in particular provided +// at least one of them is reduced double modulo. + +#define add_twice4(P0,P1,P2) \ + movq P1, %r8 ; \ + xorl %ecx, %ecx ; \ + addq P2, %r8 ; \ + movq 0x8+P1, %r9 ; \ + adcq 0x8+P2, %r9 ; \ + movq 0x10+P1, %r10 ; \ + adcq 0x10+P2, %r10 ; \ + movq 0x18+P1, %r11 ; \ + adcq 0x18+P2, %r11 ; \ + movl $38, %eax ; \ + cmovncq %rcx, %rax ; \ + addq %rax, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +#define double_twice4(P0,P1) \ + movq P1, %r8 ; \ + xorl %ecx, %ecx ; \ + addq %r8, %r8 ; \ + movq 0x8+P1, %r9 ; \ + adcq %r9, %r9 ; \ + movq 0x10+P1, %r10 ; \ + adcq %r10, %r10 ; \ + movq 0x18+P1, %r11 ; \ + adcq %r11, %r11 ; \ + movl $38, %eax ; \ + cmovncq %rcx, %rax ; \ + addq %rax, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +S2N_BN_SYMBOL(edwards25519_scalarmulbase): + CFI_START + _CET_ENDBR + +// In this case the Windows form literally makes a subroutine call. +// This avoids hassle arising from keeping code and data together. + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + CFI_CALL(Ledwards25519_scalarmulbase_standard) + CFI_POP(%rsi) + CFI_POP(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(edwards25519_scalarmulbase) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Ledwards25519_scalarmulbase_standard) + +Ledwards25519_scalarmulbase_standard: + CFI_START +#endif + +// Save registers, make room for temps, preserve input arguments. + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(NSPACE) + +// Move the output pointer to a stable place + + movq %rdi, res + +// Copy the input scalar x to its local variable while reducing it +// modulo 2^252 + m where m = 27742317777372353535851937790883648493; +// this is the order of the basepoint so this doesn't change the result. +// First do q = floor(x/2^252) and x' = x - q * (2^252 + m), which gives +// an initial result -15 * m <= x' < 2^252 + + movq (%rsi), %r8 + movq 8(%rsi), %r9 + movq 16(%rsi), %r10 + movq 24(%rsi), %r11 + + movq %r11, %rcx + shrq $60, %rcx + + movq $0x5812631a5cf5d3ed, %rax + mulq %rcx + movq %rax, %r12 + movq %rdx, %r13 + movq $0x14def9dea2f79cd6, %rax + mulq %rcx + addq %rax, %r13 + adcq $0, %rdx + shlq $60, %rcx + + subq %r12, %r8 + sbbq %r13, %r9 + sbbq %rdx, %r10 + sbbq %rcx, %r11 + +// If x' < 0 then just directly negate it; this makes sure the +// reduced argument is strictly 0 <= x' < 2^252, but now we need +// to record (done via bit 255 of the reduced scalar, which is +// ignored in the main loop) when we negated so we can flip +// the end result to compensate. + + sbbq %rax, %rax + + xorq %rax, %r8 + xorq %rax, %r9 + xorq %rax, %r10 + xorq %rax, %r11 + + negq %rax + adcq $0, %r8 + adcq $0, %r9 + adcq $0, %r10 + adcq $0, %r11 + + shlq $63, %rax + orq %rax, %r11 + +// And before we store the scalar, test and reset bit 251 to +// initialize the main loop just below. + + movq %r8, SCALAR(%rsp) + movq %r9, SCALAR+8(%rsp) + movq %r10, SCALAR+16(%rsp) + btr $59, %r11 + movq %r11, SCALAR+24(%rsp) + +// The main part of the computation is in extended-projective coordinates +// (X,Y,Z,T), representing an affine point on the edwards25519 curve +// (x,y) via x = X/Z, y = Y/Z and x * y = T/Z (so X * Y = T * Z). +// In comments B means the standard basepoint (x,4/5) = +// (0x216....f25d51a,0x6666..666658). +// +// Initialize accumulator "acc" to either 0 or 2^251 * B depending on +// bit 251 of the (reduced) scalar. That leaves bits 0..250 to handle. + + leaq S2N_BN_SYMBOL(edwards25519_scalarmulbase_constant)(%rip), %r10 + leaq 8*12(%r10), %r11 + + movq (%r10), %rax + movq (%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC(%rsp) + + movq 8*1(%r10), %rax + movq 8*1(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+8(%rsp) + + movq 8*2(%r10), %rax + movq 8*2(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+16(%rsp) + + movq 8*3(%r10), %rax + movq 8*3(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+24(%rsp) + + movq 8*4(%r10), %rax + movq 8*4(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+32(%rsp) + + movq 8*5(%r10), %rax + movq 8*5(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+40(%rsp) + + movq 8*6(%r10), %rax + movq 8*6(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+48(%rsp) + + movq 8*7(%r10), %rax + movq 8*7(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+56(%rsp) + + movl $1, %eax + movq %rax, ACC+64(%rsp) + movl $0, %eax + movq %rax, ACC+72(%rsp) + movq %rax, ACC+80(%rsp) + movq %rax, ACC+88(%rsp) + + movq 8*8(%r10), %rax + movq 8*8(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+96(%rsp) + + movq 8*9(%r10), %rax + movq 8*9(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+104(%rsp) + + movq 8*10(%r10), %rax + movq 8*10(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+112(%rsp) + + movq 8*11(%r10), %rax + movq 8*11(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+120(%rsp) + +// The counter "i" tracks the bit position for which the scalar has +// already been absorbed, starting at 0 and going up in chunks of 4. +// +// The pointer "tab" points at the current block of the table for +// multiples (2^i * j) * B at the current bit position i; 1 <= j <= 8. +// +// The bias is always either 0 and 1 and needs to be added to the +// partially processed scalar implicitly. This is used to absorb 4 bits +// of scalar per iteration from 3-bit table indexing by exploiting +// negation: (16 * h + l) * B = (16 * (h + 1) - (16 - l)) * B is used +// when l >= 9. Note that we can't have any bias left over at the +// end because we made sure bit 251 is clear in the reduced scalar. + + movq $0, i + leaq 8*24(%r10), %rax + movq %rax, tab + movq $0, bias + +// Start of the main loop, repeated 63 times for i = 4, 8, ..., 252 + +Ledwards25519_scalarmulbase_scalarloop: + +// Look at the next 4-bit field "bf", adding the previous bias as well. +// Choose the table index "ix" as bf when bf <= 8 and 16 - bf for bf >= 9, +// setting the bias to 1 for the next iteration in the latter case. + + movq i, %rax + movq %rax, %rcx + shrq $6, %rax + movq (%rsp,%rax,8), %rax // Exploiting scalar = sp exactly + shrq %cl, %rax + andq $15, %rax + addq bias, %rax + movq %rax, bf + + cmpq $9, bf + sbbq %rax, %rax + incq %rax + movq %rax, bias + + movq $16, %rdi + subq bf, %rdi + cmpq $0, bias + cmovzq bf, %rdi + movq %rdi, ix + +// Perform constant-time lookup in the table to get element number "ix". +// The table entry for the affine point (x,y) is actually a triple +// (y - x,x + y,2 * d * x * y) to precompute parts of the addition. +// Note that "ix" can be 0, so we set up the appropriate identity first. + + movl $1, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + movl $1, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + xorl %r13d, %r13d + xorl %r14d, %r14d + xorl %r15d, %r15d + + movq tab, %rbp + + cmpq $1, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $2, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $3, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $4, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $5, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $6, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $7, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $8, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + + addq $96, %rbp + movq %rbp, tab + +// We now have the triple from the table in registers as follows +// +// [%rdx;%rcx;%rbx;%rax] = y - x +// [%r11;%r10;%r9;%r8] = x + y +// [%r15;%r14;%r13;%r12] = 2 * d * x * y +// +// In case bias = 1 we need to negate this. For Edwards curves +// -(x,y) = (-x,y), i.e. we need to negate the x coordinate. +// In this processed encoding, that amounts to swapping the +// first two fields and negating the third. +// +// The optional negation here also pretends bias = 0 whenever +// ix = 0 so that it doesn't need to handle the case of zero +// inputs, since no non-trivial table entries are zero. Note +// that in the zero case the whole negation is trivial, and +// so indeed is the swapping. + + cmpq $0, bias + + movq %rax, %rsi + cmovnzq %r8, %rsi + cmovnzq %rax, %r8 + movq %rsi, TABENT(%rsp) + movq %r8, TABENT+32(%rsp) + + movq %rbx, %rsi + cmovnzq %r9, %rsi + cmovnzq %rbx, %r9 + movq %rsi, TABENT+8(%rsp) + movq %r9, TABENT+40(%rsp) + + movq %rcx, %rsi + cmovnzq %r10, %rsi + cmovnzq %rcx, %r10 + movq %rsi, TABENT+16(%rsp) + movq %r10, TABENT+48(%rsp) + + movq %rdx, %rsi + cmovnzq %r11, %rsi + cmovnzq %rdx, %r11 + movq %rsi, TABENT+24(%rsp) + movq %r11, TABENT+56(%rsp) + + movq $-19, %rax + movq $-1, %rbx + movq $-1, %rcx + movq $0x7fffffffffffffff, %rdx + subq %r12, %rax + sbbq %r13, %rbx + sbbq %r14, %rcx + sbbq %r15, %rdx + + movq ix, %r8 + movq bias, %r9 + testq %r8, %r8 + cmovzq %r8, %r9 + testq %r9, %r9 + + cmovzq %r12, %rax + cmovzq %r13, %rbx + cmovzq %r14, %rcx + cmovzq %r15, %rdx + movq %rax, TABENT+64(%rsp) + movq %rbx, TABENT+72(%rsp) + movq %rcx, TABENT+80(%rsp) + movq %rdx, TABENT+88(%rsp) + +// Extended-projective and precomputed mixed addition. +// This is effectively the same as calling the standalone +// function edwards25519_pepadd(acc,acc,tabent), but we +// only retain slightly weaker normalization < 2 * p_25519 +// throughout the inner loop, so the computation is +// slightly different, and faster overall. + + double_twice4(t0,z_1) + sub_twice4(t1,y_1,x_1) + add_twice4(t2,y_1,x_1) + mul_4(t3,w_1,kxy_2) + mul_4(t1,t1,ymx_2) + mul_4(t2,t2,xpy_2) + sub_twice4(t4,t0,t3) + add_twice4(t0,t0,t3) + sub_twice4(t5,t2,t1) + add_twice4(t1,t2,t1) + mul_4(z_3,t4,t0) + mul_4(x_3,t5,t4) + mul_4(y_3,t0,t1) + mul_4(w_3,t5,t1) + +// End of the main loop; move on by 4 bits. + + addq $4, i + cmpq $252, i + jc Ledwards25519_scalarmulbase_scalarloop + +// Insert the optional negation of the projective X coordinate, and +// so by extension the final affine x coordinate x = X/Z and thus +// the point P = (x,y). We only know X < 2 * p_25519, so we do the +// negation as 2 * p_25519 - X to keep it nonnegative. From this +// point on we don't need any normalization of the coordinates +// except for making sure that they fit in 4 digits. + + movq X3(%rsp), %r8 + movq X3+8(%rsp), %r9 + movq X3+16(%rsp), %r10 + movq X3+24(%rsp), %r11 + movq $0xffffffffffffffda, %r12 + subq %r8, %r12 + movq $0xffffffffffffffff, %r13 + sbbq %r9, %r13 + movq $0xffffffffffffffff, %r14 + sbbq %r10, %r14 + movq $0xffffffffffffffff, %r15 + sbbq %r11, %r15 + movq SCALAR+24(%rsp), %rax + btq $63, %rax + cmovcq %r12, %r8 + cmovcq %r13, %r9 + cmovcq %r14, %r10 + cmovcq %r15, %r11 + movq %r8, X3(%rsp) + movq %r9, X3+8(%rsp) + movq %r10, X3+16(%rsp) + movq %r11, X3+24(%rsp) + +// Now we need to map out of the extended-projective representation +// (X,Y,Z,W) back to the affine form (x,y) = (X/Z,Y/Z). This means +// first calling the modular inverse to get w_3 = 1/z_3. + + leaq W3(%rsp), %rdi + leaq Z3(%rsp), %rsi + +// Inline copy of bignum_inv_p25519, identical except for stripping out +// the prologue and epilogue saving and restoring registers and making +// and reclaiming room on the stack. For more details and explanations see +// "x86/curve25519/bignum_inv_p25519.S". Note that the stack it uses for +// its own temporaries is 208 bytes, so it has no effect on variables +// that are needed in the rest of our computation here: res, x_3, y_3, +// z_3 and w_3. + + movq %rdi, 0xc0(%rsp) + xorl %eax, %eax + leaq -0x13(%rax), %rcx + notq %rax + movq %rcx, (%rsp) + movq %rax, 0x8(%rsp) + movq %rax, 0x10(%rsp) + btr $0x3f, %rax + movq %rax, 0x18(%rsp) + movq (%rsi), %rdx + movq 0x8(%rsi), %rcx + movq 0x10(%rsi), %r8 + movq 0x18(%rsi), %r9 + movl $0x1, %eax + xorl %r10d, %r10d + bts $0x3f, %r9 + adcq %r10, %rax + imulq $0x13, %rax, %rax + addq %rax, %rdx + adcq %r10, %rcx + adcq %r10, %r8 + adcq %r10, %r9 + movl $0x13, %eax + cmovbq %r10, %rax + subq %rax, %rdx + sbbq %r10, %rcx + sbbq %r10, %r8 + sbbq %r10, %r9 + btr $0x3f, %r9 + movq %rdx, 0x20(%rsp) + movq %rcx, 0x28(%rsp) + movq %r8, 0x30(%rsp) + movq %r9, 0x38(%rsp) + xorl %eax, %eax + movq %rax, 0x40(%rsp) + movq %rax, 0x48(%rsp) + movq %rax, 0x50(%rsp) + movq %rax, 0x58(%rsp) + movabsq $0xa0f99e2375022099, %rax + movq %rax, 0x60(%rsp) + movabsq $0xa8c68f3f1d132595, %rax + movq %rax, 0x68(%rsp) + movabsq $0x6c6c893805ac5242, %rax + movq %rax, 0x70(%rsp) + movabsq $0x276508b241770615, %rax + movq %rax, 0x78(%rsp) + movq $0xa, 0x90(%rsp) + movq $0x1, 0x98(%rsp) + jmp Ledwards25519_scalarmulbase_midloop +Ledwards25519_scalarmulbase_inverseloop: + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + movq %r8, %rax + andq %r9, %rax + movq %r10, %rdi + andq %r11, %rdi + addq %rax, %rdi + movq %rdi, 0x80(%rsp) + movq %r12, %rax + andq %r13, %rax + movq %r14, %rsi + andq %r15, %rsi + addq %rax, %rsi + movq %rsi, 0x88(%rsp) + xorl %ebx, %ebx + movq (%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x20(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rdi + adcq %rdx, %rbx + xorl %ebp, %ebp + movq (%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x20(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + xorl %ecx, %ecx + movq 0x8(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x28(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + shrdq $0x3b, %rbx, %rdi + movq %rdi, (%rsp) + xorl %edi, %edi + movq 0x8(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rbp + adcq %rdx, %rdi + movq 0x28(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rdi + shrdq $0x3b, %rbp, %rsi + movq %rsi, 0x20(%rsp) + xorl %esi, %esi + movq 0x10(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rsi + movq 0x30(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rsi + shrdq $0x3b, %rcx, %rbx + movq %rbx, 0x8(%rsp) + xorl %ebx, %ebx + movq 0x10(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x30(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rdi + adcq %rdx, %rbx + shrdq $0x3b, %rdi, %rbp + movq %rbp, 0x28(%rsp) + movq 0x18(%rsp), %rax + xorq %r9, %rax + movq %rax, %rbp + sarq $0x3f, %rbp + andq %r8, %rbp + negq %rbp + mulq %r8 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x38(%rsp), %rax + xorq %r11, %rax + movq %rax, %rdx + sarq $0x3f, %rdx + andq %r10, %rdx + subq %rdx, %rbp + mulq %r10 + addq %rax, %rsi + adcq %rdx, %rbp + shrdq $0x3b, %rsi, %rcx + movq %rcx, 0x10(%rsp) + shrdq $0x3b, %rbp, %rsi + movq 0x18(%rsp), %rax + movq %rsi, 0x18(%rsp) + xorq %r13, %rax + movq %rax, %rsi + sarq $0x3f, %rsi + andq %r12, %rsi + negq %rsi + mulq %r12 + addq %rax, %rbx + adcq %rdx, %rsi + movq 0x38(%rsp), %rax + xorq %r15, %rax + movq %rax, %rdx + sarq $0x3f, %rdx + andq %r14, %rdx + subq %rdx, %rsi + mulq %r14 + addq %rax, %rbx + adcq %rdx, %rsi + shrdq $0x3b, %rbx, %rdi + movq %rdi, 0x30(%rsp) + shrdq $0x3b, %rsi, %rbx + movq %rbx, 0x38(%rsp) + movq 0x80(%rsp), %rbx + movq 0x88(%rsp), %rbp + xorl %ecx, %ecx + movq 0x40(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x60(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x40(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x40(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x60(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x60(%rsp) + xorl %ebx, %ebx + movq 0x48(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x68(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + xorl %ebp, %ebp + movq 0x48(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, 0x48(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x68(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, 0x68(%rsp) + xorl %ecx, %ecx + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x70(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x50(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x50(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x70(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x70(%rsp) + movq 0x58(%rsp), %rax + xorq %r9, %rax + movq %r9, %rbx + andq %r8, %rbx + negq %rbx + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x78(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbx + mulq %r10 + addq %rax, %rcx + adcq %rbx, %rdx + movq %rdx, %rbx + shldq $0x1, %rcx, %rdx + sarq $0x3f, %rbx + addq %rbx, %rdx + movl $0x13, %eax + imulq %rdx + movq 0x40(%rsp), %r8 + addq %rax, %r8 + movq %r8, 0x40(%rsp) + movq 0x48(%rsp), %r8 + adcq %rdx, %r8 + movq %r8, 0x48(%rsp) + movq 0x50(%rsp), %r8 + adcq %rbx, %r8 + movq %r8, 0x50(%rsp) + adcq %rbx, %rcx + shlq $0x3f, %rax + addq %rax, %rcx + movq 0x58(%rsp), %rax + movq %rcx, 0x58(%rsp) + xorq %r13, %rax + movq %r13, %rcx + andq %r12, %rcx + negq %rcx + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rcx + movq 0x78(%rsp), %rax + xorq %r15, %rax + movq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rcx + mulq %r14 + addq %rax, %rsi + adcq %rcx, %rdx + movq %rdx, %rcx + shldq $0x1, %rsi, %rdx + sarq $0x3f, %rcx + movl $0x13, %eax + addq %rcx, %rdx + imulq %rdx + movq 0x60(%rsp), %r8 + addq %rax, %r8 + movq %r8, 0x60(%rsp) + movq 0x68(%rsp), %r8 + adcq %rdx, %r8 + movq %r8, 0x68(%rsp) + movq 0x70(%rsp), %r8 + adcq %rcx, %r8 + movq %r8, 0x70(%rsp) + adcq %rcx, %rsi + shlq $0x3f, %rax + addq %rax, %rsi + movq %rsi, 0x78(%rsp) +Ledwards25519_scalarmulbase_midloop: + movq 0x98(%rsp), %rsi + movq (%rsp), %rdx + movq 0x20(%rsp), %rcx + movq %rdx, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + xorl %ebp, %ebp + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %rdx + leaq (%rcx,%rax), %rdi + shlq $0x16, %rdx + shlq $0x16, %rdi + sarq $0x2b, %rdx + sarq $0x2b, %rdi + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %rbx + leaq (%rcx,%rax), %rcx + sarq $0x2a, %rbx + sarq $0x2a, %rcx + movq %rdx, 0xa0(%rsp) + movq %rbx, 0xa8(%rsp) + movq %rdi, 0xb0(%rsp) + movq %rcx, 0xb8(%rsp) + movq (%rsp), %r12 + imulq %r12, %rdi + imulq %rdx, %r12 + movq 0x20(%rsp), %r13 + imulq %r13, %rbx + imulq %rcx, %r13 + addq %rbx, %r12 + addq %rdi, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r10 + shlq $0x16, %r8 + shlq $0x16, %r10 + sarq $0x2b, %r8 + sarq $0x2b, %r10 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r15 + leaq (%rcx,%rax), %r11 + sarq $0x2a, %r15 + sarq $0x2a, %r11 + movq %r13, %rbx + movq %r12, %rcx + imulq %r8, %r12 + imulq %r15, %rbx + addq %rbx, %r12 + imulq %r11, %r13 + imulq %r10, %rcx + addq %rcx, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq 0xa0(%rsp), %rax + imulq %r8, %rax + movq 0xb0(%rsp), %rdx + imulq %r15, %rdx + imulq 0xa8(%rsp), %r8 + imulq 0xb8(%rsp), %r15 + addq %r8, %r15 + leaq (%rax,%rdx), %r9 + movq 0xa0(%rsp), %rax + imulq %r10, %rax + movq 0xb0(%rsp), %rdx + imulq %r11, %rdx + imulq 0xa8(%rsp), %r10 + imulq 0xb8(%rsp), %r11 + addq %r10, %r11 + leaq (%rax,%rdx), %r13 + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r12 + shlq $0x15, %r8 + shlq $0x15, %r12 + sarq $0x2b, %r8 + sarq $0x2b, %r12 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r10 + leaq (%rcx,%rax), %r14 + sarq $0x2b, %r10 + sarq $0x2b, %r14 + movq %r9, %rax + imulq %r8, %rax + movq %r13, %rdx + imulq %r10, %rdx + imulq %r15, %r8 + imulq %r11, %r10 + addq %r8, %r10 + leaq (%rax,%rdx), %r8 + movq %r9, %rax + imulq %r12, %rax + movq %r13, %rdx + imulq %r14, %rdx + imulq %r15, %r12 + imulq %r11, %r14 + addq %r12, %r14 + leaq (%rax,%rdx), %r12 + movq %rsi, 0x98(%rsp) + decq 0x90(%rsp) + jne Ledwards25519_scalarmulbase_inverseloop + movq (%rsp), %rax + movq 0x20(%rsp), %rcx + imulq %r8, %rax + imulq %r10, %rcx + addq %rcx, %rax + sarq $0x3f, %rax + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + xorq %rax, %r9 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + xorq %rax, %r11 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + xorq %rax, %r13 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + xorq %rax, %r15 + movq %r8, %rax + andq %r9, %rax + movq %r10, %r12 + andq %r11, %r12 + addq %rax, %r12 + xorl %r13d, %r13d + movq 0x40(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x60(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movq 0x48(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r13 + adcq %rdx, %r14 + movq 0x68(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x70(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x58(%rsp), %rax + xorq %r9, %rax + andq %r8, %r9 + negq %r9 + mulq %r8 + addq %rax, %r15 + adcq %rdx, %r9 + movq 0x78(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %r9 + mulq %r10 + addq %rax, %r15 + adcq %rdx, %r9 + movq %r9, %rax + shldq $0x1, %r15, %rax + sarq $0x3f, %r9 + movl $0x13, %ebx + leaq 0x1(%rax,%r9,1), %rax + imulq %rbx + xorl %ebp, %ebp + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r9, %r14 + adcq %r9, %r15 + shlq $0x3f, %rax + addq %rax, %r15 + cmovns %rbp, %rbx + subq %rbx, %r12 + sbbq %rbp, %r13 + sbbq %rbp, %r14 + sbbq %rbp, %r15 + btr $0x3f, %r15 + movq 0xc0(%rsp), %rdi + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + +// The final result is x = X * inv(Z), y = Y * inv(Z). +// These are the only operations in the whole computation that +// fully reduce modulo p_25519 since now we want the canonical +// answer as output. + + movq res, %rbp + mul_p25519(resx,x_3,w_3) + mul_p25519(resy,y_3,w_3) + +// Restore stack and registers + + CFI_INC_RSP(NSPACE) + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +#if WINDOWS_ABI +S2N_BN_SIZE_DIRECTIVE(Ledwards25519_scalarmulbase_standard) +#else +S2N_BN_SIZE_DIRECTIVE(edwards25519_scalarmulbase) +#endif + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif + +// **************************************************************************** +// The precomputed data (all read-only). +// **************************************************************************** + +#if defined(__ELF__) +.section .rodata +.type S2N_BN_SYMBOL(edwards25519_scalarmulbase_constant), %object +.size S2N_BN_SYMBOL(edwards25519_scalarmulbase_constant), 48576 +#elif defined(__APPLE__) +.const_data +#endif + +S2N_BN_SYMBOL(edwards25519_scalarmulbase_constant): + +// 0 * B = 0 and 2^251 * B in extended-projective coordinates +// but with Z = 1 assumed and hence left out, so they are (X,Y,T) only. + + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + + .quad 0x0000000000000001 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + + .quad 0x525f946d7c7220e7 + .quad 0x4636b0b2f1e35444 + .quad 0x796e9d70e892ae0f + .quad 0x03dec05fa937adb1 + .quad 0x6d1c271cc6375515 + .quad 0x462588c4a4ca4f14 + .quad 0x691129fee55afc39 + .quad 0x15949f784d8472f5 + .quad 0xbd89e510afad0049 + .quad 0x4d1f08c073b9860e + .quad 0x07716e8b2d00af9d + .quad 0x70d685f68f859714 + +// Precomputed table of multiples of generator for edwards25519 +// all in precomputed extended-projective (y-x,x+y,2*d*x*y) triples. + + // 2^0 * 1 * G + + .quad 0x9d103905d740913e + .quad 0xfd399f05d140beb3 + .quad 0xa5c18434688f8a09 + .quad 0x44fd2f9298f81267 + .quad 0x2fbc93c6f58c3b85 + .quad 0xcf932dc6fb8c0e19 + .quad 0x270b4898643d42c2 + .quad 0x07cf9d3a33d4ba65 + .quad 0xabc91205877aaa68 + .quad 0x26d9e823ccaac49e + .quad 0x5a1b7dcbdd43598c + .quad 0x6f117b689f0c65a8 + + // 2^0 * 2 * G + + .quad 0x8a99a56042b4d5a8 + .quad 0x8f2b810c4e60acf6 + .quad 0xe09e236bb16e37aa + .quad 0x6bb595a669c92555 + .quad 0x9224e7fc933c71d7 + .quad 0x9f469d967a0ff5b5 + .quad 0x5aa69a65e1d60702 + .quad 0x590c063fa87d2e2e + .quad 0x43faa8b3a59b7a5f + .quad 0x36c16bdd5d9acf78 + .quad 0x500fa0840b3d6a31 + .quad 0x701af5b13ea50b73 + + // 2^0 * 3 * G + + .quad 0x56611fe8a4fcd265 + .quad 0x3bd353fde5c1ba7d + .quad 0x8131f31a214bd6bd + .quad 0x2ab91587555bda62 + .quad 0xaf25b0a84cee9730 + .quad 0x025a8430e8864b8a + .quad 0xc11b50029f016732 + .quad 0x7a164e1b9a80f8f4 + .quad 0x14ae933f0dd0d889 + .quad 0x589423221c35da62 + .quad 0xd170e5458cf2db4c + .quad 0x5a2826af12b9b4c6 + + // 2^0 * 4 * G + + .quad 0x95fe050a056818bf + .quad 0x327e89715660faa9 + .quad 0xc3e8e3cd06a05073 + .quad 0x27933f4c7445a49a + .quad 0x287351b98efc099f + .quad 0x6765c6f47dfd2538 + .quad 0xca348d3dfb0a9265 + .quad 0x680e910321e58727 + .quad 0x5a13fbe9c476ff09 + .quad 0x6e9e39457b5cc172 + .quad 0x5ddbdcf9102b4494 + .quad 0x7f9d0cbf63553e2b + + // 2^0 * 5 * G + + .quad 0x7f9182c3a447d6ba + .quad 0xd50014d14b2729b7 + .quad 0xe33cf11cb864a087 + .quad 0x154a7e73eb1b55f3 + .quad 0xa212bc4408a5bb33 + .quad 0x8d5048c3c75eed02 + .quad 0xdd1beb0c5abfec44 + .quad 0x2945ccf146e206eb + .quad 0xbcbbdbf1812a8285 + .quad 0x270e0807d0bdd1fc + .quad 0xb41b670b1bbda72d + .quad 0x43aabe696b3bb69a + + // 2^0 * 6 * G + + .quad 0x499806b67b7d8ca4 + .quad 0x575be28427d22739 + .quad 0xbb085ce7204553b9 + .quad 0x38b64c41ae417884 + .quad 0x3a0ceeeb77157131 + .quad 0x9b27158900c8af88 + .quad 0x8065b668da59a736 + .quad 0x51e57bb6a2cc38bd + .quad 0x85ac326702ea4b71 + .quad 0xbe70e00341a1bb01 + .quad 0x53e4a24b083bc144 + .quad 0x10b8e91a9f0d61e3 + + // 2^0 * 7 * G + + .quad 0xba6f2c9aaa3221b1 + .quad 0x6ca021533bba23a7 + .quad 0x9dea764f92192c3a + .quad 0x1d6edd5d2e5317e0 + .quad 0x6b1a5cd0944ea3bf + .quad 0x7470353ab39dc0d2 + .quad 0x71b2528228542e49 + .quad 0x461bea69283c927e + .quad 0xf1836dc801b8b3a2 + .quad 0xb3035f47053ea49a + .quad 0x529c41ba5877adf3 + .quad 0x7a9fbb1c6a0f90a7 + + // 2^0 * 8 * G + + .quad 0xe2a75dedf39234d9 + .quad 0x963d7680e1b558f9 + .quad 0x2c2741ac6e3c23fb + .quad 0x3a9024a1320e01c3 + .quad 0x59b7596604dd3e8f + .quad 0x6cb30377e288702c + .quad 0xb1339c665ed9c323 + .quad 0x0915e76061bce52f + .quad 0xe7c1f5d9c9a2911a + .quad 0xb8a371788bcca7d7 + .quad 0x636412190eb62a32 + .quad 0x26907c5c2ecc4e95 + + // 2^4 * 1 * G + + .quad 0x7ec851ca553e2df3 + .quad 0xa71284cba64878b3 + .quad 0xe6b5e4193288d1e7 + .quad 0x4cf210ec5a9a8883 + .quad 0x322d04a52d9021f6 + .quad 0xb9c19f3375c6bf9c + .quad 0x587a3a4342d20b09 + .quad 0x143b1cf8aa64fe61 + .quad 0x9f867c7d968acaab + .quad 0x5f54258e27092729 + .quad 0xd0a7d34bea180975 + .quad 0x21b546a3374126e1 + + // 2^4 * 2 * G + + .quad 0xa94ff858a2888343 + .quad 0xce0ed4565313ed3c + .quad 0xf55c3dcfb5bf34fa + .quad 0x0a653ca5c9eab371 + .quad 0x490a7a45d185218f + .quad 0x9a15377846049335 + .quad 0x0060ea09cc31e1f6 + .quad 0x7e041577f86ee965 + .quad 0x66b2a496ce5b67f3 + .quad 0xff5492d8bd569796 + .quad 0x503cec294a592cd0 + .quad 0x566943650813acb2 + + // 2^4 * 3 * G + + .quad 0xb818db0c26620798 + .quad 0x5d5c31d9606e354a + .quad 0x0982fa4f00a8cdc7 + .quad 0x17e12bcd4653e2d4 + .quad 0x5672f9eb1dabb69d + .quad 0xba70b535afe853fc + .quad 0x47ac0f752796d66d + .quad 0x32a5351794117275 + .quad 0xd3a644a6df648437 + .quad 0x703b6559880fbfdd + .quad 0xcb852540ad3a1aa5 + .quad 0x0900b3f78e4c6468 + + // 2^4 * 4 * G + + .quad 0x0a851b9f679d651b + .quad 0xe108cb61033342f2 + .quad 0xd601f57fe88b30a3 + .quad 0x371f3acaed2dd714 + .quad 0xed280fbec816ad31 + .quad 0x52d9595bd8e6efe3 + .quad 0x0fe71772f6c623f5 + .quad 0x4314030b051e293c + .quad 0xd560005efbf0bcad + .quad 0x8eb70f2ed1870c5e + .quad 0x201f9033d084e6a0 + .quad 0x4c3a5ae1ce7b6670 + + // 2^4 * 5 * G + + .quad 0x4138a434dcb8fa95 + .quad 0x870cf67d6c96840b + .quad 0xde388574297be82c + .quad 0x7c814db27262a55a + .quad 0xbaf875e4c93da0dd + .quad 0xb93282a771b9294d + .quad 0x80d63fb7f4c6c460 + .quad 0x6de9c73dea66c181 + .quad 0x478904d5a04df8f2 + .quad 0xfafbae4ab10142d3 + .quad 0xf6c8ac63555d0998 + .quad 0x5aac4a412f90b104 + + // 2^4 * 6 * G + + .quad 0xc64f326b3ac92908 + .quad 0x5551b282e663e1e0 + .quad 0x476b35f54a1a4b83 + .quad 0x1b9da3fe189f68c2 + .quad 0x603a0d0abd7f5134 + .quad 0x8089c932e1d3ae46 + .quad 0xdf2591398798bd63 + .quad 0x1c145cd274ba0235 + .quad 0x32e8386475f3d743 + .quad 0x365b8baf6ae5d9ef + .quad 0x825238b6385b681e + .quad 0x234929c1167d65e1 + + // 2^4 * 7 * G + + .quad 0x984decaba077ade8 + .quad 0x383f77ad19eb389d + .quad 0xc7ec6b7e2954d794 + .quad 0x59c77b3aeb7c3a7a + .quad 0x48145cc21d099fcf + .quad 0x4535c192cc28d7e5 + .quad 0x80e7c1e548247e01 + .quad 0x4a5f28743b2973ee + .quad 0xd3add725225ccf62 + .quad 0x911a3381b2152c5d + .quad 0xd8b39fad5b08f87d + .quad 0x6f05606b4799fe3b + + // 2^4 * 8 * G + + .quad 0x9ffe9e92177ba962 + .quad 0x98aee71d0de5cae1 + .quad 0x3ff4ae942d831044 + .quad 0x714de12e58533ac8 + .quad 0x5b433149f91b6483 + .quad 0xadb5dc655a2cbf62 + .quad 0x87fa8412632827b3 + .quad 0x60895e91ab49f8d8 + .quad 0xe9ecf2ed0cf86c18 + .quad 0xb46d06120735dfd4 + .quad 0xbc9da09804b96be7 + .quad 0x73e2e62fd96dc26b + + // 2^8 * 1 * G + + .quad 0xed5b635449aa515e + .quad 0xa865c49f0bc6823a + .quad 0x850c1fe95b42d1c4 + .quad 0x30d76d6f03d315b9 + .quad 0x2eccdd0e632f9c1d + .quad 0x51d0b69676893115 + .quad 0x52dfb76ba8637a58 + .quad 0x6dd37d49a00eef39 + .quad 0x6c4444172106e4c7 + .quad 0xfb53d680928d7f69 + .quad 0xb4739ea4694d3f26 + .quad 0x10c697112e864bb0 + + // 2^8 * 2 * G + + .quad 0x6493c4277dbe5fde + .quad 0x265d4fad19ad7ea2 + .quad 0x0e00dfc846304590 + .quad 0x25e61cabed66fe09 + .quad 0x0ca62aa08358c805 + .quad 0x6a3d4ae37a204247 + .quad 0x7464d3a63b11eddc + .quad 0x03bf9baf550806ef + .quad 0x3f13e128cc586604 + .quad 0x6f5873ecb459747e + .quad 0xa0b63dedcc1268f5 + .quad 0x566d78634586e22c + + // 2^8 * 3 * G + + .quad 0x1637a49f9cc10834 + .quad 0xbc8e56d5a89bc451 + .quad 0x1cb5ec0f7f7fd2db + .quad 0x33975bca5ecc35d9 + .quad 0xa1054285c65a2fd0 + .quad 0x6c64112af31667c3 + .quad 0x680ae240731aee58 + .quad 0x14fba5f34793b22a + .quad 0x3cd746166985f7d4 + .quad 0x593e5e84c9c80057 + .quad 0x2fc3f2b67b61131e + .quad 0x14829cea83fc526c + + // 2^8 * 4 * G + + .quad 0xff437b8497dd95c2 + .quad 0x6c744e30aa4eb5a7 + .quad 0x9e0c5d613c85e88b + .quad 0x2fd9c71e5f758173 + .quad 0x21e70b2f4e71ecb8 + .quad 0xe656ddb940a477e3 + .quad 0xbf6556cece1d4f80 + .quad 0x05fc3bc4535d7b7e + .quad 0x24b8b3ae52afdedd + .quad 0x3495638ced3b30cf + .quad 0x33a4bc83a9be8195 + .quad 0x373767475c651f04 + + // 2^8 * 5 * G + + .quad 0x2fba99fd40d1add9 + .quad 0xb307166f96f4d027 + .quad 0x4363f05215f03bae + .quad 0x1fbea56c3b18f999 + .quad 0x634095cb14246590 + .quad 0xef12144016c15535 + .quad 0x9e38140c8910bc60 + .quad 0x6bf5905730907c8c + .quad 0x0fa778f1e1415b8a + .quad 0x06409ff7bac3a77e + .quad 0x6f52d7b89aa29a50 + .quad 0x02521cf67a635a56 + + // 2^8 * 6 * G + + .quad 0x513fee0b0a9d5294 + .quad 0x8f98e75c0fdf5a66 + .quad 0xd4618688bfe107ce + .quad 0x3fa00a7e71382ced + .quad 0xb1146720772f5ee4 + .quad 0xe8f894b196079ace + .quad 0x4af8224d00ac824a + .quad 0x001753d9f7cd6cc4 + .quad 0x3c69232d963ddb34 + .quad 0x1dde87dab4973858 + .quad 0xaad7d1f9a091f285 + .quad 0x12b5fe2fa048edb6 + + // 2^8 * 7 * G + + .quad 0x71f0fbc496fce34d + .quad 0x73b9826badf35bed + .quad 0xd2047261ff28c561 + .quad 0x749b76f96fb1206f + .quad 0xdf2b7c26ad6f1e92 + .quad 0x4b66d323504b8913 + .quad 0x8c409dc0751c8bc3 + .quad 0x6f7e93c20796c7b8 + .quad 0x1f5af604aea6ae05 + .quad 0xc12351f1bee49c99 + .quad 0x61a808b5eeff6b66 + .quad 0x0fcec10f01e02151 + + // 2^8 * 8 * G + + .quad 0x644d58a649fe1e44 + .quad 0x21fcaea231ad777e + .quad 0x02441c5a887fd0d2 + .quad 0x4901aa7183c511f3 + .quad 0x3df2d29dc4244e45 + .quad 0x2b020e7493d8de0a + .quad 0x6cc8067e820c214d + .quad 0x413779166feab90a + .quad 0x08b1b7548c1af8f0 + .quad 0xce0f7a7c246299b4 + .quad 0xf760b0f91e06d939 + .quad 0x41bb887b726d1213 + + // 2^12 * 1 * G + + .quad 0x9267806c567c49d8 + .quad 0x066d04ccca791e6a + .quad 0xa69f5645e3cc394b + .quad 0x5c95b686a0788cd2 + .quad 0x97d980e0aa39f7d2 + .quad 0x35d0384252c6b51c + .quad 0x7d43f49307cd55aa + .quad 0x56bd36cfb78ac362 + .quad 0x2ac519c10d14a954 + .quad 0xeaf474b494b5fa90 + .quad 0xe6af8382a9f87a5a + .quad 0x0dea6db1879be094 + + // 2^12 * 2 * G + + .quad 0xaa66bf547344e5ab + .quad 0xda1258888f1b4309 + .quad 0x5e87d2b3fd564b2f + .quad 0x5b2c78885483b1dd + .quad 0x15baeb74d6a8797a + .quad 0x7ef55cf1fac41732 + .quad 0x29001f5a3c8b05c5 + .quad 0x0ad7cc8752eaccfb + .quad 0x52151362793408cf + .quad 0xeb0f170319963d94 + .quad 0xa833b2fa883d9466 + .quad 0x093a7fa775003c78 + + // 2^12 * 3 * G + + .quad 0xe5107de63a16d7be + .quad 0xa377ffdc9af332cf + .quad 0x70d5bf18440b677f + .quad 0x6a252b19a4a31403 + .quad 0xb8e9604460a91286 + .quad 0x7f3fd8047778d3de + .quad 0x67d01e31bf8a5e2d + .quad 0x7b038a06c27b653e + .quad 0x9ed919d5d36990f3 + .quad 0x5213aebbdb4eb9f2 + .quad 0xc708ea054cb99135 + .quad 0x58ded57f72260e56 + + // 2^12 * 4 * G + + .quad 0x78e79dade9413d77 + .quad 0xf257f9d59729e67d + .quad 0x59db910ee37aa7e6 + .quad 0x6aa11b5bbb9e039c + .quad 0xda6d53265b0fd48b + .quad 0x8960823193bfa988 + .quad 0xd78ac93261d57e28 + .quad 0x79f2942d3a5c8143 + .quad 0x97da2f25b6c88de9 + .quad 0x251ba7eaacf20169 + .quad 0x09b44f87ef4eb4e4 + .quad 0x7d90ab1bbc6a7da5 + + // 2^12 * 5 * G + + .quad 0x9acca683a7016bfe + .quad 0x90505f4df2c50b6d + .quad 0x6b610d5fcce435aa + .quad 0x19a10d446198ff96 + .quad 0x1a07a3f496b3c397 + .quad 0x11ceaa188f4e2532 + .quad 0x7d9498d5a7751bf0 + .quad 0x19ed161f508dd8a0 + .quad 0x560a2cd687dce6ca + .quad 0x7f3568c48664cf4d + .quad 0x8741e95222803a38 + .quad 0x483bdab1595653fc + + // 2^12 * 6 * G + + .quad 0xfa780f148734fa49 + .quad 0x106f0b70360534e0 + .quad 0x2210776fe3e307bd + .quad 0x3286c109dde6a0fe + .quad 0xd6cf4d0ab4da80f6 + .quad 0x82483e45f8307fe0 + .quad 0x05005269ae6f9da4 + .quad 0x1c7052909cf7877a + .quad 0x32ee7de2874e98d4 + .quad 0x14c362e9b97e0c60 + .quad 0x5781dcde6a60a38a + .quad 0x217dd5eaaa7aa840 + + // 2^12 * 7 * G + + .quad 0x9db7c4d0248e1eb0 + .quad 0xe07697e14d74bf52 + .quad 0x1e6a9b173c562354 + .quad 0x7fa7c21f795a4965 + .quad 0x8bdf1fb9be8c0ec8 + .quad 0x00bae7f8e30a0282 + .quad 0x4963991dad6c4f6c + .quad 0x07058a6e5df6f60a + .quad 0xe9eb02c4db31f67f + .quad 0xed25fd8910bcfb2b + .quad 0x46c8131f5c5cddb4 + .quad 0x33b21c13a0cb9bce + + // 2^12 * 8 * G + + .quad 0x360692f8087d8e31 + .quad 0xf4dcc637d27163f7 + .quad 0x25a4e62065ea5963 + .quad 0x659bf72e5ac160d9 + .quad 0x9aafb9b05ee38c5b + .quad 0xbf9d2d4e071a13c7 + .quad 0x8eee6e6de933290a + .quad 0x1c3bab17ae109717 + .quad 0x1c9ab216c7cab7b0 + .quad 0x7d65d37407bbc3cc + .quad 0x52744750504a58d5 + .quad 0x09f2606b131a2990 + + // 2^16 * 1 * G + + .quad 0x40e87d44744346be + .quad 0x1d48dad415b52b25 + .quad 0x7c3a8a18a13b603e + .quad 0x4eb728c12fcdbdf7 + .quad 0x7e234c597c6691ae + .quad 0x64889d3d0a85b4c8 + .quad 0xdae2c90c354afae7 + .quad 0x0a871e070c6a9e1d + .quad 0x3301b5994bbc8989 + .quad 0x736bae3a5bdd4260 + .quad 0x0d61ade219d59e3c + .quad 0x3ee7300f2685d464 + + // 2^16 * 2 * G + + .quad 0xf5d255e49e7dd6b7 + .quad 0x8016115c610b1eac + .quad 0x3c99975d92e187ca + .quad 0x13815762979125c2 + .quad 0x43fa7947841e7518 + .quad 0xe5c6fa59639c46d7 + .quad 0xa1065e1de3052b74 + .quad 0x7d47c6a2cfb89030 + .quad 0x3fdad0148ef0d6e0 + .quad 0x9d3e749a91546f3c + .quad 0x71ec621026bb8157 + .quad 0x148cf58d34c9ec80 + + // 2^16 * 3 * G + + .quad 0x46a492f67934f027 + .quad 0x469984bef6840aa9 + .quad 0x5ca1bc2a89611854 + .quad 0x3ff2fa1ebd5dbbd4 + .quad 0xe2572f7d9ae4756d + .quad 0x56c345bb88f3487f + .quad 0x9fd10b6d6960a88d + .quad 0x278febad4eaea1b9 + .quad 0xb1aa681f8c933966 + .quad 0x8c21949c20290c98 + .quad 0x39115291219d3c52 + .quad 0x4104dd02fe9c677b + + // 2^16 * 4 * G + + .quad 0x72b2bf5e1124422a + .quad 0xa1fa0c3398a33ab5 + .quad 0x94cb6101fa52b666 + .quad 0x2c863b00afaf53d5 + .quad 0x81214e06db096ab8 + .quad 0x21a8b6c90ce44f35 + .quad 0x6524c12a409e2af5 + .quad 0x0165b5a48efca481 + .quad 0xf190a474a0846a76 + .quad 0x12eff984cd2f7cc0 + .quad 0x695e290658aa2b8f + .quad 0x591b67d9bffec8b8 + + // 2^16 * 5 * G + + .quad 0x312f0d1c80b49bfa + .quad 0x5979515eabf3ec8a + .quad 0x727033c09ef01c88 + .quad 0x3de02ec7ca8f7bcb + .quad 0x99b9b3719f18b55d + .quad 0xe465e5faa18c641e + .quad 0x61081136c29f05ed + .quad 0x489b4f867030128b + .quad 0xd232102d3aeb92ef + .quad 0xe16253b46116a861 + .quad 0x3d7eabe7190baa24 + .quad 0x49f5fbba496cbebf + + // 2^16 * 6 * G + + .quad 0x30949a108a5bcfd4 + .quad 0xdc40dd70bc6473eb + .quad 0x92c294c1307c0d1c + .quad 0x5604a86dcbfa6e74 + .quad 0x155d628c1e9c572e + .quad 0x8a4d86acc5884741 + .quad 0x91a352f6515763eb + .quad 0x06a1a6c28867515b + .quad 0x7288d1d47c1764b6 + .quad 0x72541140e0418b51 + .quad 0x9f031a6018acf6d1 + .quad 0x20989e89fe2742c6 + + // 2^16 * 7 * G + + .quad 0x499777fd3a2dcc7f + .quad 0x32857c2ca54fd892 + .quad 0xa279d864d207e3a0 + .quad 0x0403ed1d0ca67e29 + .quad 0x1674278b85eaec2e + .quad 0x5621dc077acb2bdf + .quad 0x640a4c1661cbf45a + .quad 0x730b9950f70595d3 + .quad 0xc94b2d35874ec552 + .quad 0xc5e6c8cf98246f8d + .quad 0xf7cb46fa16c035ce + .quad 0x5bd7454308303dcc + + // 2^16 * 8 * G + + .quad 0x7f9ad19528b24cc2 + .quad 0x7f6b54656335c181 + .quad 0x66b8b66e4fc07236 + .quad 0x133a78007380ad83 + .quad 0x85c4932115e7792a + .quad 0xc64c89a2bdcdddc9 + .quad 0x9d1e3da8ada3d762 + .quad 0x5bb7db123067f82c + .quad 0x0961f467c6ca62be + .quad 0x04ec21d6211952ee + .quad 0x182360779bd54770 + .quad 0x740dca6d58f0e0d2 + + // 2^20 * 1 * G + + .quad 0x50b70bf5d3f0af0b + .quad 0x4feaf48ae32e71f7 + .quad 0x60e84ed3a55bbd34 + .quad 0x00ed489b3f50d1ed + .quad 0x3906c72aed261ae5 + .quad 0x9ab68fd988e100f7 + .quad 0xf5e9059af3360197 + .quad 0x0e53dc78bf2b6d47 + .quad 0xb90829bf7971877a + .quad 0x5e4444636d17e631 + .quad 0x4d05c52e18276893 + .quad 0x27632d9a5a4a4af5 + + // 2^20 * 2 * G + + .quad 0xd11ff05154b260ce + .quad 0xd86dc38e72f95270 + .quad 0x601fcd0d267cc138 + .quad 0x2b67916429e90ccd + .quad 0xa98285d187eaffdb + .quad 0xa5b4fbbbd8d0a864 + .quad 0xb658f27f022663f7 + .quad 0x3bbc2b22d99ce282 + .quad 0xb917c952583c0a58 + .quad 0x653ff9b80fe4c6f3 + .quad 0x9b0da7d7bcdf3c0c + .quad 0x43a0eeb6ab54d60e + + // 2^20 * 3 * G + + .quad 0x396966a46d4a5487 + .quad 0xf811a18aac2bb3ba + .quad 0x66e4685b5628b26b + .quad 0x70a477029d929b92 + .quad 0x3ac6322357875fe8 + .quad 0xd9d4f4ecf5fbcb8f + .quad 0x8dee8493382bb620 + .quad 0x50c5eaa14c799fdc + .quad 0xdd0edc8bd6f2fb3c + .quad 0x54c63aa79cc7b7a0 + .quad 0xae0b032b2c8d9f1a + .quad 0x6f9ce107602967fb + + // 2^20 * 4 * G + + .quad 0xad1054b1cde1c22a + .quad 0xc4a8e90248eb32df + .quad 0x5f3e7b33accdc0ea + .quad 0x72364713fc79963e + .quad 0x139693063520e0b5 + .quad 0x437fcf7c88ea03fe + .quad 0xf7d4c40bd3c959bc + .quad 0x699154d1f893ded9 + .quad 0x315d5c75b4b27526 + .quad 0xcccb842d0236daa5 + .quad 0x22f0c8a3345fee8e + .quad 0x73975a617d39dbed + + // 2^20 * 5 * G + + .quad 0xe4024df96375da10 + .quad 0x78d3251a1830c870 + .quad 0x902b1948658cd91c + .quad 0x7e18b10b29b7438a + .quad 0x6f37f392f4433e46 + .quad 0x0e19b9a11f566b18 + .quad 0x220fb78a1fd1d662 + .quad 0x362a4258a381c94d + .quad 0x9071d9132b6beb2f + .quad 0x0f26e9ad28418247 + .quad 0xeab91ec9bdec925d + .quad 0x4be65bc8f48af2de + + // 2^20 * 6 * G + + .quad 0x78487feba36e7028 + .quad 0x5f3f13001dd8ce34 + .quad 0x934fb12d4b30c489 + .quad 0x056c244d397f0a2b + .quad 0x1d50fba257c26234 + .quad 0x7bd4823adeb0678b + .quad 0xc2b0dc6ea6538af5 + .quad 0x5665eec6351da73e + .quad 0xdb3ee00943bfb210 + .quad 0x4972018720800ac2 + .quad 0x26ab5d6173bd8667 + .quad 0x20b209c2ab204938 + + // 2^20 * 7 * G + + .quad 0x549e342ac07fb34b + .quad 0x02d8220821373d93 + .quad 0xbc262d70acd1f567 + .quad 0x7a92c9fdfbcac784 + .quad 0x1fcca94516bd3289 + .quad 0x448d65aa41420428 + .quad 0x59c3b7b216a55d62 + .quad 0x49992cc64e612cd8 + .quad 0x65bd1bea70f801de + .quad 0x1befb7c0fe49e28a + .quad 0xa86306cdb1b2ae4a + .quad 0x3b7ac0cd265c2a09 + + // 2^20 * 8 * G + + .quad 0x822bee438c01bcec + .quad 0x530cb525c0fbc73b + .quad 0x48519034c1953fe9 + .quad 0x265cc261e09a0f5b + .quad 0xf0d54e4f22ed39a7 + .quad 0xa2aae91e5608150a + .quad 0xf421b2e9eddae875 + .quad 0x31bc531d6b7de992 + .quad 0xdf3d134da980f971 + .quad 0x7a4fb8d1221a22a7 + .quad 0x3df7d42035aad6d8 + .quad 0x2a14edcc6a1a125e + + // 2^24 * 1 * G + + .quad 0xdf48ee0752cfce4e + .quad 0xc3fffaf306ec08b7 + .quad 0x05710b2ab95459c4 + .quad 0x161d25fa963ea38d + .quad 0x231a8c570478433c + .quad 0xb7b5270ec281439d + .quad 0xdbaa99eae3d9079f + .quad 0x2c03f5256c2b03d9 + .quad 0x790f18757b53a47d + .quad 0x307b0130cf0c5879 + .quad 0x31903d77257ef7f9 + .quad 0x699468bdbd96bbaf + + // 2^24 * 2 * G + + .quad 0xbd1f2f46f4dafecf + .quad 0x7cef0114a47fd6f7 + .quad 0xd31ffdda4a47b37f + .quad 0x525219a473905785 + .quad 0xd8dd3de66aa91948 + .quad 0x485064c22fc0d2cc + .quad 0x9b48246634fdea2f + .quad 0x293e1c4e6c4a2e3a + .quad 0x376e134b925112e1 + .quad 0x703778b5dca15da0 + .quad 0xb04589af461c3111 + .quad 0x5b605c447f032823 + + // 2^24 * 3 * G + + .quad 0xb965805920c47c89 + .quad 0xe7f0100c923b8fcc + .quad 0x0001256502e2ef77 + .quad 0x24a76dcea8aeb3ee + .quad 0x3be9fec6f0e7f04c + .quad 0x866a579e75e34962 + .quad 0x5542ef161e1de61a + .quad 0x2f12fef4cc5abdd5 + .quad 0x0a4522b2dfc0c740 + .quad 0x10d06e7f40c9a407 + .quad 0xc6cf144178cff668 + .quad 0x5e607b2518a43790 + + // 2^24 * 4 * G + + .quad 0x58b31d8f6cdf1818 + .quad 0x35cfa74fc36258a2 + .quad 0xe1b3ff4f66e61d6e + .quad 0x5067acab6ccdd5f7 + .quad 0xa02c431ca596cf14 + .quad 0xe3c42d40aed3e400 + .quad 0xd24526802e0f26db + .quad 0x201f33139e457068 + .quad 0xfd527f6b08039d51 + .quad 0x18b14964017c0006 + .quad 0xd5220eb02e25a4a8 + .quad 0x397cba8862460375 + + // 2^24 * 5 * G + + .quad 0x30c13093f05959b2 + .quad 0xe23aa18de9a97976 + .quad 0x222fd491721d5e26 + .quad 0x2339d320766e6c3a + .quad 0x7815c3fbc81379e7 + .quad 0xa6619420dde12af1 + .quad 0xffa9c0f885a8fdd5 + .quad 0x771b4022c1e1c252 + .quad 0xd87dd986513a2fa7 + .quad 0xf5ac9b71f9d4cf08 + .quad 0xd06bc31b1ea283b3 + .quad 0x331a189219971a76 + + // 2^24 * 6 * G + + .quad 0xf5166f45fb4f80c6 + .quad 0x9c36c7de61c775cf + .quad 0xe3d4e81b9041d91c + .quad 0x31167c6b83bdfe21 + .quad 0x26512f3a9d7572af + .quad 0x5bcbe28868074a9e + .quad 0x84edc1c11180f7c4 + .quad 0x1ac9619ff649a67b + .quad 0xf22b3842524b1068 + .quad 0x5068343bee9ce987 + .quad 0xfc9d71844a6250c8 + .quad 0x612436341f08b111 + + // 2^24 * 7 * G + + .quad 0xd99d41db874e898d + .quad 0x09fea5f16c07dc20 + .quad 0x793d2c67d00f9bbc + .quad 0x46ebe2309e5eff40 + .quad 0x8b6349e31a2d2638 + .quad 0x9ddfb7009bd3fd35 + .quad 0x7f8bf1b8a3a06ba4 + .quad 0x1522aa3178d90445 + .quad 0x2c382f5369614938 + .quad 0xdafe409ab72d6d10 + .quad 0xe8c83391b646f227 + .quad 0x45fe70f50524306c + + // 2^24 * 8 * G + + .quad 0xda4875a6960c0b8c + .quad 0x5b68d076ef0e2f20 + .quad 0x07fb51cf3d0b8fd4 + .quad 0x428d1623a0e392d4 + .quad 0x62f24920c8951491 + .quad 0x05f007c83f630ca2 + .quad 0x6fbb45d2f5c9d4b8 + .quad 0x16619f6db57a2245 + .quad 0x084f4a4401a308fd + .quad 0xa82219c376a5caac + .quad 0xdeb8de4643d1bc7d + .quad 0x1d81592d60bd38c6 + + // 2^28 * 1 * G + + .quad 0xd833d7beec2a4c38 + .quad 0x2c9162830acc20ed + .quad 0xe93a47aa92df7581 + .quad 0x702d67a3333c4a81 + .quad 0x3a4a369a2f89c8a1 + .quad 0x63137a1d7c8de80d + .quad 0xbcac008a78eda015 + .quad 0x2cb8b3a5b483b03f + .quad 0x36e417cbcb1b90a1 + .quad 0x33b3ddaa7f11794e + .quad 0x3f510808885bc607 + .quad 0x24141dc0e6a8020d + + // 2^28 * 2 * G + + .quad 0x59f73c773fefee9d + .quad 0xb3f1ef89c1cf989d + .quad 0xe35dfb42e02e545f + .quad 0x5766120b47a1b47c + .quad 0x91925dccbd83157d + .quad 0x3ca1205322cc8094 + .quad 0x28e57f183f90d6e4 + .quad 0x1a4714cede2e767b + .quad 0xdb20ba0fb8b6b7ff + .quad 0xb732c3b677511fa1 + .quad 0xa92b51c099f02d89 + .quad 0x4f3875ad489ca5f1 + + // 2^28 * 3 * G + + .quad 0xc7fc762f4932ab22 + .quad 0x7ac0edf72f4c3c1b + .quad 0x5f6b55aa9aa895e8 + .quad 0x3680274dad0a0081 + .quad 0x79ed13f6ee73eec0 + .quad 0xa5c6526d69110bb1 + .quad 0xe48928c38603860c + .quad 0x722a1446fd7059f5 + .quad 0xd0959fe9a8cf8819 + .quad 0xd0a995508475a99c + .quad 0x6eac173320b09cc5 + .quad 0x628ecf04331b1095 + + // 2^28 * 4 * G + + .quad 0x98bcb118a9d0ddbc + .quad 0xee449e3408b4802b + .quad 0x87089226b8a6b104 + .quad 0x685f349a45c7915d + .quad 0x9b41acf85c74ccf1 + .quad 0xb673318108265251 + .quad 0x99c92aed11adb147 + .quad 0x7a47d70d34ecb40f + .quad 0x60a0c4cbcc43a4f5 + .quad 0x775c66ca3677bea9 + .quad 0xa17aa1752ff8f5ed + .quad 0x11ded9020e01fdc0 + + // 2^28 * 5 * G + + .quad 0x890e7809caefe704 + .quad 0x8728296de30e8c6c + .quad 0x4c5cd2a392aeb1c9 + .quad 0x194263d15771531f + .quad 0x471f95b03bea93b7 + .quad 0x0552d7d43313abd3 + .quad 0xbd9370e2e17e3f7b + .quad 0x7b120f1db20e5bec + .quad 0x17d2fb3d86502d7a + .quad 0xb564d84450a69352 + .quad 0x7da962c8a60ed75d + .quad 0x00d0f85b318736aa + + // 2^28 * 6 * G + + .quad 0x978b142e777c84fd + .quad 0xf402644705a8c062 + .quad 0xa67ad51be7e612c7 + .quad 0x2f7b459698dd6a33 + .quad 0xa6753c1efd7621c1 + .quad 0x69c0b4a7445671f5 + .quad 0x971f527405b23c11 + .quad 0x387bc74851a8c7cd + .quad 0x81894b4d4a52a9a8 + .quad 0xadd93e12f6b8832f + .quad 0x184d8548b61bd638 + .quad 0x3f1c62dbd6c9f6cd + + // 2^28 * 7 * G + + .quad 0x2e8f1f0091910c1f + .quad 0xa4df4fe0bff2e12c + .quad 0x60c6560aee927438 + .quad 0x6338283facefc8fa + .quad 0x3fad3e40148f693d + .quad 0x052656e194eb9a72 + .quad 0x2f4dcbfd184f4e2f + .quad 0x406f8db1c482e18b + .quad 0x9e630d2c7f191ee4 + .quad 0x4fbf8301bc3ff670 + .quad 0x787d8e4e7afb73c4 + .quad 0x50d83d5be8f58fa5 + + // 2^28 * 8 * G + + .quad 0x85683916c11a1897 + .quad 0x2d69a4efe506d008 + .quad 0x39af1378f664bd01 + .quad 0x65942131361517c6 + .quad 0xc0accf90b4d3b66d + .quad 0xa7059de561732e60 + .quad 0x033d1f7870c6b0ba + .quad 0x584161cd26d946e4 + .quad 0xbbf2b1a072d27ca2 + .quad 0xbf393c59fbdec704 + .quad 0xe98dbbcee262b81e + .quad 0x02eebd0b3029b589 + + // 2^32 * 1 * G + + .quad 0x61368756a60dac5f + .quad 0x17e02f6aebabdc57 + .quad 0x7f193f2d4cce0f7d + .quad 0x20234a7789ecdcf0 + .quad 0x8765b69f7b85c5e8 + .quad 0x6ff0678bd168bab2 + .quad 0x3a70e77c1d330f9b + .quad 0x3a5f6d51b0af8e7c + .quad 0x76d20db67178b252 + .quad 0x071c34f9d51ed160 + .quad 0xf62a4a20b3e41170 + .quad 0x7cd682353cffe366 + + // 2^32 * 2 * G + + .quad 0x0be1a45bd887fab6 + .quad 0x2a846a32ba403b6e + .quad 0xd9921012e96e6000 + .quad 0x2838c8863bdc0943 + .quad 0xa665cd6068acf4f3 + .quad 0x42d92d183cd7e3d3 + .quad 0x5759389d336025d9 + .quad 0x3ef0253b2b2cd8ff + .quad 0xd16bb0cf4a465030 + .quad 0xfa496b4115c577ab + .quad 0x82cfae8af4ab419d + .quad 0x21dcb8a606a82812 + + // 2^32 * 3 * G + + .quad 0x5c6004468c9d9fc8 + .quad 0x2540096ed42aa3cb + .quad 0x125b4d4c12ee2f9c + .quad 0x0bc3d08194a31dab + .quad 0x9a8d00fabe7731ba + .quad 0x8203607e629e1889 + .quad 0xb2cc023743f3d97f + .quad 0x5d840dbf6c6f678b + .quad 0x706e380d309fe18b + .quad 0x6eb02da6b9e165c7 + .quad 0x57bbba997dae20ab + .quad 0x3a4276232ac196dd + + // 2^32 * 4 * G + + .quad 0x4b42432c8a7084fa + .quad 0x898a19e3dfb9e545 + .quad 0xbe9f00219c58e45d + .quad 0x1ff177cea16debd1 + .quad 0x3bf8c172db447ecb + .quad 0x5fcfc41fc6282dbd + .quad 0x80acffc075aa15fe + .quad 0x0770c9e824e1a9f9 + .quad 0xcf61d99a45b5b5fd + .quad 0x860984e91b3a7924 + .quad 0xe7300919303e3e89 + .quad 0x39f264fd41500b1e + + // 2^32 * 5 * G + + .quad 0xa7ad3417dbe7e29c + .quad 0xbd94376a2b9c139c + .quad 0xa0e91b8e93597ba9 + .quad 0x1712d73468889840 + .quad 0xd19b4aabfe097be1 + .quad 0xa46dfce1dfe01929 + .quad 0xc3c908942ca6f1ff + .quad 0x65c621272c35f14e + .quad 0xe72b89f8ce3193dd + .quad 0x4d103356a125c0bb + .quad 0x0419a93d2e1cfe83 + .quad 0x22f9800ab19ce272 + + // 2^32 * 6 * G + + .quad 0x605a368a3e9ef8cb + .quad 0xe3e9c022a5504715 + .quad 0x553d48b05f24248f + .quad 0x13f416cd647626e5 + .quad 0x42029fdd9a6efdac + .quad 0xb912cebe34a54941 + .quad 0x640f64b987bdf37b + .quad 0x4171a4d38598cab4 + .quad 0xfa2758aa99c94c8c + .quad 0x23006f6fb000b807 + .quad 0xfbd291ddadda5392 + .quad 0x508214fa574bd1ab + + // 2^32 * 7 * G + + .quad 0xc20269153ed6fe4b + .quad 0xa65a6739511d77c4 + .quad 0xcbde26462c14af94 + .quad 0x22f960ec6faba74b + .quad 0x461a15bb53d003d6 + .quad 0xb2102888bcf3c965 + .quad 0x27c576756c683a5a + .quad 0x3a7758a4c86cb447 + .quad 0x548111f693ae5076 + .quad 0x1dae21df1dfd54a6 + .quad 0x12248c90f3115e65 + .quad 0x5d9fd15f8de7f494 + + // 2^32 * 8 * G + + .quad 0x031408d36d63727f + .quad 0x6a379aefd7c7b533 + .quad 0xa9e18fc5ccaee24b + .quad 0x332f35914f8fbed3 + .quad 0x3f244d2aeed7521e + .quad 0x8e3a9028432e9615 + .quad 0xe164ba772e9c16d4 + .quad 0x3bc187fa47eb98d8 + .quad 0x6d470115ea86c20c + .quad 0x998ab7cb6c46d125 + .quad 0xd77832b53a660188 + .quad 0x450d81ce906fba03 + + // 2^36 * 1 * G + + .quad 0xf8ae4d2ad8453902 + .quad 0x7018058ee8db2d1d + .quad 0xaab3995fc7d2c11e + .quad 0x53b16d2324ccca79 + .quad 0x23264d66b2cae0b5 + .quad 0x7dbaed33ebca6576 + .quad 0x030ebed6f0d24ac8 + .quad 0x2a887f78f7635510 + .quad 0x2a23b9e75c012d4f + .quad 0x0c974651cae1f2ea + .quad 0x2fb63273675d70ca + .quad 0x0ba7250b864403f5 + + // 2^36 * 2 * G + + .quad 0xbb0d18fd029c6421 + .quad 0xbc2d142189298f02 + .quad 0x8347f8e68b250e96 + .quad 0x7b9f2fe8032d71c9 + .quad 0xdd63589386f86d9c + .quad 0x61699176e13a85a4 + .quad 0x2e5111954eaa7d57 + .quad 0x32c21b57fb60bdfb + .quad 0xd87823cd319e0780 + .quad 0xefc4cfc1897775c5 + .quad 0x4854fb129a0ab3f7 + .quad 0x12c49d417238c371 + + // 2^36 * 3 * G + + .quad 0x0950b533ffe83769 + .quad 0x21861c1d8e1d6bd1 + .quad 0xf022d8381302e510 + .quad 0x2509200c6391cab4 + .quad 0x09b3a01783799542 + .quad 0x626dd08faad5ee3f + .quad 0xba00bceeeb70149f + .quad 0x1421b246a0a444c9 + .quad 0x4aa43a8e8c24a7c7 + .quad 0x04c1f540d8f05ef5 + .quad 0xadba5e0c0b3eb9dc + .quad 0x2ab5504448a49ce3 + + // 2^36 * 4 * G + + .quad 0x2ed227266f0f5dec + .quad 0x9824ee415ed50824 + .quad 0x807bec7c9468d415 + .quad 0x7093bae1b521e23f + .quad 0xdc07ac631c5d3afa + .quad 0x58615171f9df8c6c + .quad 0x72a079d89d73e2b0 + .quad 0x7301f4ceb4eae15d + .quad 0x6409e759d6722c41 + .quad 0xa674e1cf72bf729b + .quad 0xbc0a24eb3c21e569 + .quad 0x390167d24ebacb23 + + // 2^36 * 5 * G + + .quad 0x27f58e3bba353f1c + .quad 0x4c47764dbf6a4361 + .quad 0xafbbc4e56e562650 + .quad 0x07db2ee6aae1a45d + .quad 0xd7bb054ba2f2120b + .quad 0xe2b9ceaeb10589b7 + .quad 0x3fe8bac8f3c0edbe + .quad 0x4cbd40767112cb69 + .quad 0x0b603cc029c58176 + .quad 0x5988e3825cb15d61 + .quad 0x2bb61413dcf0ad8d + .quad 0x7b8eec6c74183287 + + // 2^36 * 6 * G + + .quad 0xe4ca40782cd27cb0 + .quad 0xdaf9c323fbe967bd + .quad 0xb29bd34a8ad41e9e + .quad 0x72810497626ede4d + .quad 0x32fee570fc386b73 + .quad 0xda8b0141da3a8cc7 + .quad 0x975ffd0ac8968359 + .quad 0x6ee809a1b132a855 + .quad 0x9444bb31fcfd863a + .quad 0x2fe3690a3e4e48c5 + .quad 0xdc29c867d088fa25 + .quad 0x13bd1e38d173292e + + // 2^36 * 7 * G + + .quad 0xd32b4cd8696149b5 + .quad 0xe55937d781d8aab7 + .quad 0x0bcb2127ae122b94 + .quad 0x41e86fcfb14099b0 + .quad 0x223fb5cf1dfac521 + .quad 0x325c25316f554450 + .quad 0x030b98d7659177ac + .quad 0x1ed018b64f88a4bd + .quad 0x3630dfa1b802a6b0 + .quad 0x880f874742ad3bd5 + .quad 0x0af90d6ceec5a4d4 + .quad 0x746a247a37cdc5d9 + + // 2^36 * 8 * G + + .quad 0xd531b8bd2b7b9af6 + .quad 0x5005093537fc5b51 + .quad 0x232fcf25c593546d + .quad 0x20a365142bb40f49 + .quad 0x6eccd85278d941ed + .quad 0x2254ae83d22f7843 + .quad 0xc522d02e7bbfcdb7 + .quad 0x681e3351bff0e4e2 + .quad 0x8b64b59d83034f45 + .quad 0x2f8b71f21fa20efb + .quad 0x69249495ba6550e4 + .quad 0x539ef98e45d5472b + + // 2^40 * 1 * G + + .quad 0x6e7bb6a1a6205275 + .quad 0xaa4f21d7413c8e83 + .quad 0x6f56d155e88f5cb2 + .quad 0x2de25d4ba6345be1 + .quad 0xd074d8961cae743f + .quad 0xf86d18f5ee1c63ed + .quad 0x97bdc55be7f4ed29 + .quad 0x4cbad279663ab108 + .quad 0x80d19024a0d71fcd + .quad 0xc525c20afb288af8 + .quad 0xb1a3974b5f3a6419 + .quad 0x7d7fbcefe2007233 + + // 2^40 * 2 * G + + .quad 0xfaef1e6a266b2801 + .quad 0x866c68c4d5739f16 + .quad 0xf68a2fbc1b03762c + .quad 0x5975435e87b75a8d + .quad 0xcd7c5dc5f3c29094 + .quad 0xc781a29a2a9105ab + .quad 0x80c61d36421c3058 + .quad 0x4f9cd196dcd8d4d7 + .quad 0x199297d86a7b3768 + .quad 0xd0d058241ad17a63 + .quad 0xba029cad5c1c0c17 + .quad 0x7ccdd084387a0307 + + // 2^40 * 3 * G + + .quad 0xdca6422c6d260417 + .quad 0xae153d50948240bd + .quad 0xa9c0c1b4fb68c677 + .quad 0x428bd0ed61d0cf53 + .quad 0x9b0c84186760cc93 + .quad 0xcdae007a1ab32a99 + .quad 0xa88dec86620bda18 + .quad 0x3593ca848190ca44 + .quad 0x9213189a5e849aa7 + .quad 0xd4d8c33565d8facd + .quad 0x8c52545b53fdbbd1 + .quad 0x27398308da2d63e6 + + // 2^40 * 4 * G + + .quad 0x42c38d28435ed413 + .quad 0xbd50f3603278ccc9 + .quad 0xbb07ab1a79da03ef + .quad 0x269597aebe8c3355 + .quad 0xb9a10e4c0a702453 + .quad 0x0fa25866d57d1bde + .quad 0xffb9d9b5cd27daf7 + .quad 0x572c2945492c33fd + .quad 0xc77fc745d6cd30be + .quad 0xe4dfe8d3e3baaefb + .quad 0xa22c8830aa5dda0c + .quad 0x7f985498c05bca80 + + // 2^40 * 5 * G + + .quad 0x3849ce889f0be117 + .quad 0x8005ad1b7b54a288 + .quad 0x3da3c39f23fc921c + .quad 0x76c2ec470a31f304 + .quad 0xd35615520fbf6363 + .quad 0x08045a45cf4dfba6 + .quad 0xeec24fbc873fa0c2 + .quad 0x30f2653cd69b12e7 + .quad 0x8a08c938aac10c85 + .quad 0x46179b60db276bcb + .quad 0xa920c01e0e6fac70 + .quad 0x2f1273f1596473da + + // 2^40 * 6 * G + + .quad 0x4739fc7c8ae01e11 + .quad 0xfd5274904a6aab9f + .quad 0x41d98a8287728f2e + .quad 0x5d9e572ad85b69f2 + .quad 0x30488bd755a70bc0 + .quad 0x06d6b5a4f1d442e7 + .quad 0xead1a69ebc596162 + .quad 0x38ac1997edc5f784 + .quad 0x0666b517a751b13b + .quad 0x747d06867e9b858c + .quad 0xacacc011454dde49 + .quad 0x22dfcd9cbfe9e69c + + // 2^40 * 7 * G + + .quad 0x8ddbd2e0c30d0cd9 + .quad 0xad8e665facbb4333 + .quad 0x8f6b258c322a961f + .quad 0x6b2916c05448c1c7 + .quad 0x56ec59b4103be0a1 + .quad 0x2ee3baecd259f969 + .quad 0x797cb29413f5cd32 + .quad 0x0fe9877824cde472 + .quad 0x7edb34d10aba913b + .quad 0x4ea3cd822e6dac0e + .quad 0x66083dff6578f815 + .quad 0x4c303f307ff00a17 + + // 2^40 * 8 * G + + .quad 0xd30a3bd617b28c85 + .quad 0xc5d377b739773bea + .quad 0xc6c6e78c1e6a5cbf + .quad 0x0d61b8f78b2ab7c4 + .quad 0x29fc03580dd94500 + .quad 0xecd27aa46fbbec93 + .quad 0x130a155fc2e2a7f8 + .quad 0x416b151ab706a1d5 + .quad 0x56a8d7efe9c136b0 + .quad 0xbd07e5cd58e44b20 + .quad 0xafe62fda1b57e0ab + .quad 0x191a2af74277e8d2 + + // 2^44 * 1 * G + + .quad 0xd550095bab6f4985 + .quad 0x04f4cd5b4fbfaf1a + .quad 0x9d8e2ed12a0c7540 + .quad 0x2bc24e04b2212286 + .quad 0x09d4b60b2fe09a14 + .quad 0xc384f0afdbb1747e + .quad 0x58e2ea8978b5fd6e + .quad 0x519ef577b5e09b0a + .quad 0x1863d7d91124cca9 + .quad 0x7ac08145b88a708e + .quad 0x2bcd7309857031f5 + .quad 0x62337a6e8ab8fae5 + + // 2^44 * 2 * G + + .quad 0x4bcef17f06ffca16 + .quad 0xde06e1db692ae16a + .quad 0x0753702d614f42b0 + .quad 0x5f6041b45b9212d0 + .quad 0xd1ab324e1b3a1273 + .quad 0x18947cf181055340 + .quad 0x3b5d9567a98c196e + .quad 0x7fa00425802e1e68 + .quad 0x7d531574028c2705 + .quad 0x80317d69db0d75fe + .quad 0x30fface8ef8c8ddd + .quad 0x7e9de97bb6c3e998 + + // 2^44 * 3 * G + + .quad 0x1558967b9e6585a3 + .quad 0x97c99ce098e98b92 + .quad 0x10af149b6eb3adad + .quad 0x42181fe8f4d38cfa + .quad 0xf004be62a24d40dd + .quad 0xba0659910452d41f + .quad 0x81c45ee162a44234 + .quad 0x4cb829d8a22266ef + .quad 0x1dbcaa8407b86681 + .quad 0x081f001e8b26753b + .quad 0x3cd7ce6a84048e81 + .quad 0x78af11633f25f22c + + // 2^44 * 4 * G + + .quad 0x8416ebd40b50babc + .quad 0x1508722628208bee + .quad 0xa3148fafb9c1c36d + .quad 0x0d07daacd32d7d5d + .quad 0x3241c00e7d65318c + .quad 0xe6bee5dcd0e86de7 + .quad 0x118b2dc2fbc08c26 + .quad 0x680d04a7fc603dc3 + .quad 0xf9c2414a695aa3eb + .quad 0xdaa42c4c05a68f21 + .quad 0x7c6c23987f93963e + .quad 0x210e8cd30c3954e3 + + // 2^44 * 5 * G + + .quad 0xac4201f210a71c06 + .quad 0x6a65e0aef3bfb021 + .quad 0xbc42c35c393632f7 + .quad 0x56ea8db1865f0742 + .quad 0x2b50f16137fe6c26 + .quad 0xe102bcd856e404d8 + .quad 0x12b0f1414c561f6b + .quad 0x51b17bc8d028ec91 + .quad 0xfff5fb4bcf535119 + .quad 0xf4989d79df1108a0 + .quad 0xbdfcea659a3ba325 + .quad 0x18a11f1174d1a6f2 + + // 2^44 * 6 * G + + .quad 0x407375ab3f6bba29 + .quad 0x9ec3b6d8991e482e + .quad 0x99c80e82e55f92e9 + .quad 0x307c13b6fb0c0ae1 + .quad 0xfbd63cdad27a5f2c + .quad 0xf00fc4bc8aa106d7 + .quad 0x53fb5c1a8e64a430 + .quad 0x04eaabe50c1a2e85 + .quad 0x24751021cb8ab5e7 + .quad 0xfc2344495c5010eb + .quad 0x5f1e717b4e5610a1 + .quad 0x44da5f18c2710cd5 + + // 2^44 * 7 * G + + .quad 0x033cc55ff1b82eb5 + .quad 0xb15ae36d411cae52 + .quad 0xba40b6198ffbacd3 + .quad 0x768edce1532e861f + .quad 0x9156fe6b89d8eacc + .quad 0xe6b79451e23126a1 + .quad 0xbd7463d93944eb4e + .quad 0x726373f6767203ae + .quad 0xe305ca72eb7ef68a + .quad 0x662cf31f70eadb23 + .quad 0x18f026fdb4c45b68 + .quad 0x513b5384b5d2ecbd + + // 2^44 * 8 * G + + .quad 0x46d46280c729989e + .quad 0x4b93fbd05368a5dd + .quad 0x63df3f81d1765a89 + .quad 0x34cebd64b9a0a223 + .quad 0x5e2702878af34ceb + .quad 0x900b0409b946d6ae + .quad 0x6512ebf7dabd8512 + .quad 0x61d9b76988258f81 + .quad 0xa6c5a71349b7d94b + .quad 0xa3f3d15823eb9446 + .quad 0x0416fbd277484834 + .quad 0x69d45e6f2c70812f + + // 2^48 * 1 * G + + .quad 0xce16f74bc53c1431 + .quad 0x2b9725ce2072edde + .quad 0xb8b9c36fb5b23ee7 + .quad 0x7e2e0e450b5cc908 + .quad 0x9fe62b434f460efb + .quad 0xded303d4a63607d6 + .quad 0xf052210eb7a0da24 + .quad 0x237e7dbe00545b93 + .quad 0x013575ed6701b430 + .quad 0x231094e69f0bfd10 + .quad 0x75320f1583e47f22 + .quad 0x71afa699b11155e3 + + // 2^48 * 2 * G + + .quad 0x65ce6f9b3953b61d + .quad 0xc65839eaafa141e6 + .quad 0x0f435ffda9f759fe + .quad 0x021142e9c2b1c28e + .quad 0xea423c1c473b50d6 + .quad 0x51e87a1f3b38ef10 + .quad 0x9b84bf5fb2c9be95 + .quad 0x00731fbc78f89a1c + .quad 0xe430c71848f81880 + .quad 0xbf960c225ecec119 + .quad 0xb6dae0836bba15e3 + .quad 0x4c4d6f3347e15808 + + // 2^48 * 3 * G + + .quad 0x18f7eccfc17d1fc9 + .quad 0x6c75f5a651403c14 + .quad 0xdbde712bf7ee0cdf + .quad 0x193fddaaa7e47a22 + .quad 0x2f0cddfc988f1970 + .quad 0x6b916227b0b9f51b + .quad 0x6ec7b6c4779176be + .quad 0x38bf9500a88f9fa8 + .quad 0x1fd2c93c37e8876f + .quad 0xa2f61e5a18d1462c + .quad 0x5080f58239241276 + .quad 0x6a6fb99ebf0d4969 + + // 2^48 * 4 * G + + .quad 0x6a46c1bb560855eb + .quad 0x2416bb38f893f09d + .quad 0xd71d11378f71acc1 + .quad 0x75f76914a31896ea + .quad 0xeeb122b5b6e423c6 + .quad 0x939d7010f286ff8e + .quad 0x90a92a831dcf5d8c + .quad 0x136fda9f42c5eb10 + .quad 0xf94cdfb1a305bdd1 + .quad 0x0f364b9d9ff82c08 + .quad 0x2a87d8a5c3bb588a + .quad 0x022183510be8dcba + + // 2^48 * 5 * G + + .quad 0x4af766385ead2d14 + .quad 0xa08ed880ca7c5830 + .quad 0x0d13a6e610211e3d + .quad 0x6a071ce17b806c03 + .quad 0x9d5a710143307a7f + .quad 0xb063de9ec47da45f + .quad 0x22bbfe52be927ad3 + .quad 0x1387c441fd40426c + .quad 0xb5d3c3d187978af8 + .quad 0x722b5a3d7f0e4413 + .quad 0x0d7b4848bb477ca0 + .quad 0x3171b26aaf1edc92 + + // 2^48 * 6 * G + + .quad 0xa92f319097564ca8 + .quad 0xff7bb84c2275e119 + .quad 0x4f55fe37a4875150 + .quad 0x221fd4873cf0835a + .quad 0xa60db7d8b28a47d1 + .quad 0xa6bf14d61770a4f1 + .quad 0xd4a1f89353ddbd58 + .quad 0x6c514a63344243e9 + .quad 0x2322204f3a156341 + .quad 0xfb73e0e9ba0a032d + .quad 0xfce0dd4c410f030e + .quad 0x48daa596fb924aaa + + // 2^48 * 7 * G + + .quad 0x6eca8e665ca59cc7 + .quad 0xa847254b2e38aca0 + .quad 0x31afc708d21e17ce + .quad 0x676dd6fccad84af7 + .quad 0x14f61d5dc84c9793 + .quad 0x9941f9e3ef418206 + .quad 0xcdf5b88f346277ac + .quad 0x58c837fa0e8a79a9 + .quad 0x0cf9688596fc9058 + .quad 0x1ddcbbf37b56a01b + .quad 0xdcc2e77d4935d66a + .quad 0x1c4f73f2c6a57f0a + + // 2^48 * 8 * G + + .quad 0x0e7a4fbd305fa0bb + .quad 0x829d4ce054c663ad + .quad 0xf421c3832fe33848 + .quad 0x795ac80d1bf64c42 + .quad 0xb36e706efc7c3484 + .quad 0x73dfc9b4c3c1cf61 + .quad 0xeb1d79c9781cc7e5 + .quad 0x70459adb7daf675c + .quad 0x1b91db4991b42bb3 + .quad 0x572696234b02dcca + .quad 0x9fdf9ee51f8c78dc + .quad 0x5fe162848ce21fd3 + + // 2^52 * 1 * G + + .quad 0xe2790aae4d077c41 + .quad 0x8b938270db7469a3 + .quad 0x6eb632dc8abd16a2 + .quad 0x720814ecaa064b72 + .quad 0x315c29c795115389 + .quad 0xd7e0e507862f74ce + .quad 0x0c4a762185927432 + .quad 0x72de6c984a25a1e4 + .quad 0xae9ab553bf6aa310 + .quad 0x050a50a9806d6e1b + .quad 0x92bb7403adff5139 + .quad 0x0394d27645be618b + + // 2^52 * 2 * G + + .quad 0x4d572251857eedf4 + .quad 0xe3724edde19e93c5 + .quad 0x8a71420e0b797035 + .quad 0x3b3c833687abe743 + .quad 0xf5396425b23545a4 + .quad 0x15a7a27e98fbb296 + .quad 0xab6c52bc636fdd86 + .quad 0x79d995a8419334ee + .quad 0xcd8a8ea61195dd75 + .quad 0xa504d8a81dd9a82f + .quad 0x540dca81a35879b6 + .quad 0x60dd16a379c86a8a + + // 2^52 * 3 * G + + .quad 0x35a2c8487381e559 + .quad 0x596ffea6d78082cb + .quad 0xcb9771ebdba7b653 + .quad 0x5a08b5019b4da685 + .quad 0x3501d6f8153e47b8 + .quad 0xb7a9675414a2f60c + .quad 0x112ee8b6455d9523 + .quad 0x4e62a3c18112ea8a + .quad 0xc8d4ac04516ab786 + .quad 0x595af3215295b23d + .quad 0xd6edd234db0230c1 + .quad 0x0929efe8825b41cc + + // 2^52 * 4 * G + + .quad 0x5f0601d1cbd0f2d3 + .quad 0x736e412f6132bb7f + .quad 0x83604432238dde87 + .quad 0x1e3a5272f5c0753c + .quad 0x8b3172b7ad56651d + .quad 0x01581b7a3fabd717 + .quad 0x2dc94df6424df6e4 + .quad 0x30376e5d2c29284f + .quad 0xd2918da78159a59c + .quad 0x6bdc1cd93f0713f3 + .quad 0x565f7a934acd6590 + .quad 0x53daacec4cb4c128 + + // 2^52 * 5 * G + + .quad 0x4ca73bd79cc8a7d6 + .quad 0x4d4a738f47e9a9b2 + .quad 0xf4cbf12942f5fe00 + .quad 0x01a13ff9bdbf0752 + .quad 0x99852bc3852cfdb0 + .quad 0x2cc12e9559d6ed0b + .quad 0x70f9e2bf9b5ac27b + .quad 0x4f3b8c117959ae99 + .quad 0x55b6c9c82ff26412 + .quad 0x1ac4a8c91fb667a8 + .quad 0xd527bfcfeb778bf2 + .quad 0x303337da7012a3be + + // 2^52 * 6 * G + + .quad 0x955422228c1c9d7c + .quad 0x01fac1371a9b340f + .quad 0x7e8d9177925b48d7 + .quad 0x53f8ad5661b3e31b + .quad 0x976d3ccbfad2fdd1 + .quad 0xcb88839737a640a8 + .quad 0x2ff00c1d6734cb25 + .quad 0x269ff4dc789c2d2b + .quad 0x0c003fbdc08d678d + .quad 0x4d982fa37ead2b17 + .quad 0xc07e6bcdb2e582f1 + .quad 0x296c7291df412a44 + + // 2^52 * 7 * G + + .quad 0x7903de2b33daf397 + .quad 0xd0ff0619c9a624b3 + .quad 0x8a1d252b555b3e18 + .quad 0x2b6d581c52e0b7c0 + .quad 0xdfb23205dab8b59e + .quad 0x465aeaa0c8092250 + .quad 0xd133c1189a725d18 + .quad 0x2327370261f117d1 + .quad 0x3d0543d3623e7986 + .quad 0x679414c2c278a354 + .quad 0xae43f0cc726196f6 + .quad 0x7836c41f8245eaba + + // 2^52 * 8 * G + + .quad 0xe7a254db49e95a81 + .quad 0x5192d5d008b0ad73 + .quad 0x4d20e5b1d00afc07 + .quad 0x5d55f8012cf25f38 + .quad 0xca651e848011937c + .quad 0xc6b0c46e6ef41a28 + .quad 0xb7021ba75f3f8d52 + .quad 0x119dff99ead7b9fd + .quad 0x43eadfcbf4b31d4d + .quad 0xc6503f7411148892 + .quad 0xfeee68c5060d3b17 + .quad 0x329293b3dd4a0ac8 + + // 2^56 * 1 * G + + .quad 0x4e59214fe194961a + .quad 0x49be7dc70d71cd4f + .quad 0x9300cfd23b50f22d + .quad 0x4789d446fc917232 + .quad 0x2879852d5d7cb208 + .quad 0xb8dedd70687df2e7 + .quad 0xdc0bffab21687891 + .quad 0x2b44c043677daa35 + .quad 0x1a1c87ab074eb78e + .quad 0xfac6d18e99daf467 + .quad 0x3eacbbcd484f9067 + .quad 0x60c52eef2bb9a4e4 + + // 2^56 * 2 * G + + .quad 0x0b5d89bc3bfd8bf1 + .quad 0xb06b9237c9f3551a + .quad 0x0e4c16b0d53028f5 + .quad 0x10bc9c312ccfcaab + .quad 0x702bc5c27cae6d11 + .quad 0x44c7699b54a48cab + .quad 0xefbc4056ba492eb2 + .quad 0x70d77248d9b6676d + .quad 0xaa8ae84b3ec2a05b + .quad 0x98699ef4ed1781e0 + .quad 0x794513e4708e85d1 + .quad 0x63755bd3a976f413 + + // 2^56 * 3 * G + + .quad 0xb55fa03e2ad10853 + .quad 0x356f75909ee63569 + .quad 0x9ff9f1fdbe69b890 + .quad 0x0d8cc1c48bc16f84 + .quad 0x3dc7101897f1acb7 + .quad 0x5dda7d5ec165bbd8 + .quad 0x508e5b9c0fa1020f + .quad 0x2763751737c52a56 + .quad 0x029402d36eb419a9 + .quad 0xf0b44e7e77b460a5 + .quad 0xcfa86230d43c4956 + .quad 0x70c2dd8a7ad166e7 + + // 2^56 * 4 * G + + .quad 0x656194509f6fec0e + .quad 0xee2e7ea946c6518d + .quad 0x9733c1f367e09b5c + .quad 0x2e0fac6363948495 + .quad 0x91d4967db8ed7e13 + .quad 0x74252f0ad776817a + .quad 0xe40982e00d852564 + .quad 0x32b8613816a53ce5 + .quad 0x79e7f7bee448cd64 + .quad 0x6ac83a67087886d0 + .quad 0xf89fd4d9a0e4db2e + .quad 0x4179215c735a4f41 + + // 2^56 * 5 * G + + .quad 0x8c7094e7d7dced2a + .quad 0x97fb8ac347d39c70 + .quad 0xe13be033a906d902 + .quad 0x700344a30cd99d76 + .quad 0xe4ae33b9286bcd34 + .quad 0xb7ef7eb6559dd6dc + .quad 0x278b141fb3d38e1f + .quad 0x31fa85662241c286 + .quad 0xaf826c422e3622f4 + .quad 0xc12029879833502d + .quad 0x9bc1b7e12b389123 + .quad 0x24bb2312a9952489 + + // 2^56 * 6 * G + + .quad 0xb1a8ed1732de67c3 + .quad 0x3cb49418461b4948 + .quad 0x8ebd434376cfbcd2 + .quad 0x0fee3e871e188008 + .quad 0x41f80c2af5f85c6b + .quad 0x687284c304fa6794 + .quad 0x8945df99a3ba1bad + .quad 0x0d1d2af9ffeb5d16 + .quad 0xa9da8aa132621edf + .quad 0x30b822a159226579 + .quad 0x4004197ba79ac193 + .quad 0x16acd79718531d76 + + // 2^56 * 7 * G + + .quad 0x72df72af2d9b1d3d + .quad 0x63462a36a432245a + .quad 0x3ecea07916b39637 + .quad 0x123e0ef6b9302309 + .quad 0xc959c6c57887b6ad + .quad 0x94e19ead5f90feba + .quad 0x16e24e62a342f504 + .quad 0x164ed34b18161700 + .quad 0x487ed94c192fe69a + .quad 0x61ae2cea3a911513 + .quad 0x877bf6d3b9a4de27 + .quad 0x78da0fc61073f3eb + + // 2^56 * 8 * G + + .quad 0x5bf15d28e52bc66a + .quad 0x2c47e31870f01a8e + .quad 0x2419afbc06c28bdd + .quad 0x2d25deeb256b173a + .quad 0xa29f80f1680c3a94 + .quad 0x71f77e151ae9e7e6 + .quad 0x1100f15848017973 + .quad 0x054aa4b316b38ddd + .quad 0xdfc8468d19267cb8 + .quad 0x0b28789c66e54daf + .quad 0x2aeb1d2a666eec17 + .quad 0x134610a6ab7da760 + + // 2^60 * 1 * G + + .quad 0xcaf55ec27c59b23f + .quad 0x99aeed3e154d04f2 + .quad 0x68441d72e14141f4 + .quad 0x140345133932a0a2 + .quad 0xd91430e0dc028c3c + .quad 0x0eb955a85217c771 + .quad 0x4b09e1ed2c99a1fa + .quad 0x42881af2bd6a743c + .quad 0x7bfec69aab5cad3d + .quad 0xc23e8cd34cb2cfad + .quad 0x685dd14bfb37d6a2 + .quad 0x0ad6d64415677a18 + + // 2^60 * 2 * G + + .quad 0x781a439e417becb5 + .quad 0x4ac5938cd10e0266 + .quad 0x5da385110692ac24 + .quad 0x11b065a2ade31233 + .quad 0x7914892847927e9f + .quad 0x33dad6ef370aa877 + .quad 0x1f8f24fa11122703 + .quad 0x5265ac2f2adf9592 + .quad 0x405fdd309afcb346 + .quad 0xd9723d4428e63f54 + .quad 0x94c01df05f65aaae + .quad 0x43e4dc3ae14c0809 + + // 2^60 * 3 * G + + .quad 0xbc12c7f1a938a517 + .quad 0x473028ab3180b2e1 + .quad 0x3f78571efbcd254a + .quad 0x74e534426ff6f90f + .quad 0xea6f7ac3adc2c6a3 + .quad 0xd0e928f6e9717c94 + .quad 0xe2d379ead645eaf5 + .quad 0x46dd8785c51ffbbe + .quad 0x709801be375c8898 + .quad 0x4b06dab5e3fd8348 + .quad 0x75880ced27230714 + .quad 0x2b09468fdd2f4c42 + + // 2^60 * 4 * G + + .quad 0x97c749eeb701cb96 + .quad 0x83f438d4b6a369c3 + .quad 0x62962b8b9a402cd9 + .quad 0x6976c7509888df7b + .quad 0x5b97946582ffa02a + .quad 0xda096a51fea8f549 + .quad 0xa06351375f77af9b + .quad 0x1bcfde61201d1e76 + .quad 0x4a4a5490246a59a2 + .quad 0xd63ebddee87fdd90 + .quad 0xd9437c670d2371fa + .quad 0x69e87308d30f8ed6 + + // 2^60 * 5 * G + + .quad 0x435a8bb15656beb0 + .quad 0xf8fac9ba4f4d5bca + .quad 0xb9b278c41548c075 + .quad 0x3eb0ef76e892b622 + .quad 0x0f80bf028bc80303 + .quad 0x6aae16b37a18cefb + .quad 0xdd47ea47d72cd6a3 + .quad 0x61943588f4ed39aa + .quad 0xd26e5c3e91039f85 + .quad 0xc0e9e77df6f33aa9 + .quad 0xe8968c5570066a93 + .quad 0x3c34d1881faaaddd + + // 2^60 * 6 * G + + .quad 0x3f9d2b5ea09f9ec0 + .quad 0x1dab3b6fb623a890 + .quad 0xa09ba3ea72d926c4 + .quad 0x374193513fd8b36d + .quad 0xbd5b0b8f2fffe0d9 + .quad 0x6aa254103ed24fb9 + .quad 0x2ac7d7bcb26821c4 + .quad 0x605b394b60dca36a + .quad 0xb4e856e45a9d1ed2 + .quad 0xefe848766c97a9a2 + .quad 0xb104cf641e5eee7d + .quad 0x2f50b81c88a71c8f + + // 2^60 * 7 * G + + .quad 0x31723c61fc6811bb + .quad 0x9cb450486211800f + .quad 0x768933d347995753 + .quad 0x3491a53502752fcd + .quad 0x2b552ca0a7da522a + .quad 0x3230b336449b0250 + .quad 0xf2c4c5bca4b99fb9 + .quad 0x7b2c674958074a22 + .quad 0xd55165883ed28cdf + .quad 0x12d84fd2d362de39 + .quad 0x0a874ad3e3378e4f + .quad 0x000d2b1f7c763e74 + + // 2^60 * 8 * G + + .quad 0x3d420811d06d4a67 + .quad 0xbefc048590e0ffe3 + .quad 0xf870c6b7bd487bde + .quad 0x6e2a7316319afa28 + .quad 0x9624778c3e94a8ab + .quad 0x0ad6f3cee9a78bec + .quad 0x948ac7810d743c4f + .quad 0x76627935aaecfccc + .quad 0x56a8ac24d6d59a9f + .quad 0xc8db753e3096f006 + .quad 0x477f41e68f4c5299 + .quad 0x588d851cf6c86114 + + // 2^64 * 1 * G + + .quad 0x51138ec78df6b0fe + .quad 0x5397da89e575f51b + .quad 0x09207a1d717af1b9 + .quad 0x2102fdba2b20d650 + .quad 0xcd2a65e777d1f515 + .quad 0x548991878faa60f1 + .quad 0xb1b73bbcdabc06e5 + .quad 0x654878cba97cc9fb + .quad 0x969ee405055ce6a1 + .quad 0x36bca7681251ad29 + .quad 0x3a1af517aa7da415 + .quad 0x0ad725db29ecb2ba + + // 2^64 * 2 * G + + .quad 0xdc4267b1834e2457 + .quad 0xb67544b570ce1bc5 + .quad 0x1af07a0bf7d15ed7 + .quad 0x4aefcffb71a03650 + .quad 0xfec7bc0c9b056f85 + .quad 0x537d5268e7f5ffd7 + .quad 0x77afc6624312aefa + .quad 0x4f675f5302399fd9 + .quad 0xc32d36360415171e + .quad 0xcd2bef118998483b + .quad 0x870a6eadd0945110 + .quad 0x0bccbb72a2a86561 + + // 2^64 * 3 * G + + .quad 0x185e962feab1a9c8 + .quad 0x86e7e63565147dcd + .quad 0xb092e031bb5b6df2 + .quad 0x4024f0ab59d6b73e + .quad 0x186d5e4c50fe1296 + .quad 0xe0397b82fee89f7e + .quad 0x3bc7f6c5507031b0 + .quad 0x6678fd69108f37c2 + .quad 0x1586fa31636863c2 + .quad 0x07f68c48572d33f2 + .quad 0x4f73cc9f789eaefc + .quad 0x2d42e2108ead4701 + + // 2^64 * 4 * G + + .quad 0x97f5131594dfd29b + .quad 0x6155985d313f4c6a + .quad 0xeba13f0708455010 + .quad 0x676b2608b8d2d322 + .quad 0x21717b0d0f537593 + .quad 0x914e690b131e064c + .quad 0x1bb687ae752ae09f + .quad 0x420bf3a79b423c6e + .quad 0x8138ba651c5b2b47 + .quad 0x8671b6ec311b1b80 + .quad 0x7bff0cb1bc3135b0 + .quad 0x745d2ffa9c0cf1e0 + + // 2^64 * 5 * G + + .quad 0xbf525a1e2bc9c8bd + .quad 0xea5b260826479d81 + .quad 0xd511c70edf0155db + .quad 0x1ae23ceb960cf5d0 + .quad 0x6036df5721d34e6a + .quad 0xb1db8827997bb3d0 + .quad 0xd3c209c3c8756afa + .quad 0x06e15be54c1dc839 + .quad 0x5b725d871932994a + .quad 0x32351cb5ceb1dab0 + .quad 0x7dc41549dab7ca05 + .quad 0x58ded861278ec1f7 + + // 2^64 * 6 * G + + .quad 0xd8173793f266c55c + .quad 0xc8c976c5cc454e49 + .quad 0x5ce382f8bc26c3a8 + .quad 0x2ff39de85485f6f9 + .quad 0x2dfb5ba8b6c2c9a8 + .quad 0x48eeef8ef52c598c + .quad 0x33809107f12d1573 + .quad 0x08ba696b531d5bd8 + .quad 0x77ed3eeec3efc57a + .quad 0x04e05517d4ff4811 + .quad 0xea3d7a3ff1a671cb + .quad 0x120633b4947cfe54 + + // 2^64 * 7 * G + + .quad 0x0b94987891610042 + .quad 0x4ee7b13cecebfae8 + .quad 0x70be739594f0a4c0 + .quad 0x35d30a99b4d59185 + .quad 0x82bd31474912100a + .quad 0xde237b6d7e6fbe06 + .quad 0xe11e761911ea79c6 + .quad 0x07433be3cb393bde + .quad 0xff7944c05ce997f4 + .quad 0x575d3de4b05c51a3 + .quad 0x583381fd5a76847c + .quad 0x2d873ede7af6da9f + + // 2^64 * 8 * G + + .quad 0x157a316443373409 + .quad 0xfab8b7eef4aa81d9 + .quad 0xb093fee6f5a64806 + .quad 0x2e773654707fa7b6 + .quad 0xaa6202e14e5df981 + .quad 0xa20d59175015e1f5 + .quad 0x18a275d3bae21d6c + .quad 0x0543618a01600253 + .quad 0x0deabdf4974c23c1 + .quad 0xaa6f0a259dce4693 + .quad 0x04202cb8a29aba2c + .quad 0x4b1443362d07960d + + // 2^68 * 1 * G + + .quad 0x47b837f753242cec + .quad 0x256dc48cc04212f2 + .quad 0xe222fbfbe1d928c5 + .quad 0x48ea295bad8a2c07 + .quad 0x299b1c3f57c5715e + .quad 0x96cb929e6b686d90 + .quad 0x3004806447235ab3 + .quad 0x2c435c24a44d9fe1 + .quad 0x0607c97c80f8833f + .quad 0x0e851578ca25ec5b + .quad 0x54f7450b161ebb6f + .quad 0x7bcb4792a0def80e + + // 2^68 * 2 * G + + .quad 0x8487e3d02bc73659 + .quad 0x4baf8445059979df + .quad 0xd17c975adcad6fbf + .quad 0x57369f0bdefc96b6 + .quad 0x1cecd0a0045224c2 + .quad 0x757f1b1b69e53952 + .quad 0x775b7a925289f681 + .quad 0x1b6cc62016736148 + .quad 0xf1a9990175638698 + .quad 0x353dd1beeeaa60d3 + .quad 0x849471334c9ba488 + .quad 0x63fa6e6843ade311 + + // 2^68 * 3 * G + + .quad 0xd15c20536597c168 + .quad 0x9f73740098d28789 + .quad 0x18aee7f13257ba1f + .quad 0x3418bfda07346f14 + .quad 0x2195becdd24b5eb7 + .quad 0x5e41f18cc0cd44f9 + .quad 0xdf28074441ca9ede + .quad 0x07073b98f35b7d67 + .quad 0xd03c676c4ce530d4 + .quad 0x0b64c0473b5df9f4 + .quad 0x065cef8b19b3a31e + .quad 0x3084d661533102c9 + + // 2^68 * 4 * G + + .quad 0xe1f6b79ebf8469ad + .quad 0x15801004e2663135 + .quad 0x9a498330af74181b + .quad 0x3ba2504f049b673c + .quad 0x9a6ce876760321fd + .quad 0x7fe2b5109eb63ad8 + .quad 0x00e7d4ae8ac80592 + .quad 0x73d86b7abb6f723a + .quad 0x0b52b5606dba5ab6 + .quad 0xa9134f0fbbb1edab + .quad 0x30a9520d9b04a635 + .quad 0x6813b8f37973e5db + + // 2^68 * 5 * G + + .quad 0x9854b054334127c1 + .quad 0x105d047882fbff25 + .quad 0xdb49f7f944186f4f + .quad 0x1768e838bed0b900 + .quad 0xf194ca56f3157e29 + .quad 0x136d35705ef528a5 + .quad 0xdd4cef778b0599bc + .quad 0x7d5472af24f833ed + .quad 0xd0ef874daf33da47 + .quad 0x00d3be5db6e339f9 + .quad 0x3f2a8a2f9c9ceece + .quad 0x5d1aeb792352435a + + // 2^68 * 6 * G + + .quad 0xf59e6bb319cd63ca + .quad 0x670c159221d06839 + .quad 0xb06d565b2150cab6 + .quad 0x20fb199d104f12a3 + .quad 0x12c7bfaeb61ba775 + .quad 0xb84e621fe263bffd + .quad 0x0b47a5c35c840dcf + .quad 0x7e83be0bccaf8634 + .quad 0x61943dee6d99c120 + .quad 0x86101f2e460b9fe0 + .quad 0x6bb2f1518ee8598d + .quad 0x76b76289fcc475cc + + // 2^68 * 7 * G + + .quad 0x791b4cc1756286fa + .quad 0xdbced317d74a157c + .quad 0x7e732421ea72bde6 + .quad 0x01fe18491131c8e9 + .quad 0x4245f1a1522ec0b3 + .quad 0x558785b22a75656d + .quad 0x1d485a2548a1b3c0 + .quad 0x60959eccd58fe09f + .quad 0x3ebfeb7ba8ed7a09 + .quad 0x49fdc2bbe502789c + .quad 0x44ebce5d3c119428 + .quad 0x35e1eb55be947f4a + + // 2^68 * 8 * G + + .quad 0xdbdae701c5738dd3 + .quad 0xf9c6f635b26f1bee + .quad 0x61e96a8042f15ef4 + .quad 0x3aa1d11faf60a4d8 + .quad 0x14fd6dfa726ccc74 + .quad 0x3b084cfe2f53b965 + .quad 0xf33ae4f552a2c8b4 + .quad 0x59aab07a0d40166a + .quad 0x77bcec4c925eac25 + .quad 0x1848718460137738 + .quad 0x5b374337fea9f451 + .quad 0x1865e78ec8e6aa46 + + // 2^72 * 1 * G + + .quad 0xccc4b7c7b66e1f7a + .quad 0x44157e25f50c2f7e + .quad 0x3ef06dfc713eaf1c + .quad 0x582f446752da63f7 + .quad 0x967c54e91c529ccb + .quad 0x30f6269264c635fb + .quad 0x2747aff478121965 + .quad 0x17038418eaf66f5c + .quad 0xc6317bd320324ce4 + .quad 0xa81042e8a4488bc4 + .quad 0xb21ef18b4e5a1364 + .quad 0x0c2a1c4bcda28dc9 + + // 2^72 * 2 * G + + .quad 0xd24dc7d06f1f0447 + .quad 0xb2269e3edb87c059 + .quad 0xd15b0272fbb2d28f + .quad 0x7c558bd1c6f64877 + .quad 0xedc4814869bd6945 + .quad 0x0d6d907dbe1c8d22 + .quad 0xc63bd212d55cc5ab + .quad 0x5a6a9b30a314dc83 + .quad 0xd0ec1524d396463d + .quad 0x12bb628ac35a24f0 + .quad 0xa50c3a791cbc5fa4 + .quad 0x0404a5ca0afbafc3 + + // 2^72 * 3 * G + + .quad 0x8c1f40070aa743d6 + .quad 0xccbad0cb5b265ee8 + .quad 0x574b046b668fd2de + .quad 0x46395bfdcadd9633 + .quad 0x62bc9e1b2a416fd1 + .quad 0xb5c6f728e350598b + .quad 0x04343fd83d5d6967 + .quad 0x39527516e7f8ee98 + .quad 0x117fdb2d1a5d9a9c + .quad 0x9c7745bcd1005c2a + .quad 0xefd4bef154d56fea + .quad 0x76579a29e822d016 + + // 2^72 * 4 * G + + .quad 0x45b68e7e49c02a17 + .quad 0x23cd51a2bca9a37f + .quad 0x3ed65f11ec224c1b + .quad 0x43a384dc9e05bdb1 + .quad 0x333cb51352b434f2 + .quad 0xd832284993de80e1 + .quad 0xb5512887750d35ce + .quad 0x02c514bb2a2777c1 + .quad 0x684bd5da8bf1b645 + .quad 0xfb8bd37ef6b54b53 + .quad 0x313916d7a9b0d253 + .quad 0x1160920961548059 + + // 2^72 * 5 * G + + .quad 0xb44d166929dacfaa + .quad 0xda529f4c8413598f + .quad 0xe9ef63ca453d5559 + .quad 0x351e125bc5698e0b + .quad 0x7a385616369b4dcd + .quad 0x75c02ca7655c3563 + .quad 0x7dc21bf9d4f18021 + .quad 0x2f637d7491e6e042 + .quad 0xd4b49b461af67bbe + .quad 0xd603037ac8ab8961 + .quad 0x71dee19ff9a699fb + .quad 0x7f182d06e7ce2a9a + + // 2^72 * 6 * G + + .quad 0x7a7c8e64ab0168ec + .quad 0xcb5a4a5515edc543 + .quad 0x095519d347cd0eda + .quad 0x67d4ac8c343e93b0 + .quad 0x09454b728e217522 + .quad 0xaa58e8f4d484b8d8 + .quad 0xd358254d7f46903c + .quad 0x44acc043241c5217 + .quad 0x1c7d6bbb4f7a5777 + .quad 0x8b35fed4918313e1 + .quad 0x4adca1c6c96b4684 + .quad 0x556d1c8312ad71bd + + // 2^72 * 7 * G + + .quad 0x17ef40e30c8d3982 + .quad 0x31f7073e15a3fa34 + .quad 0x4f21f3cb0773646e + .quad 0x746c6c6d1d824eff + .quad 0x81f06756b11be821 + .quad 0x0faff82310a3f3dd + .quad 0xf8b2d0556a99465d + .quad 0x097abe38cc8c7f05 + .quad 0x0c49c9877ea52da4 + .quad 0x4c4369559bdc1d43 + .quad 0x022c3809f7ccebd2 + .quad 0x577e14a34bee84bd + + // 2^72 * 8 * G + + .quad 0xf0e268ac61a73b0a + .quad 0xf2fafa103791a5f5 + .quad 0xc1e13e826b6d00e9 + .quad 0x60fa7ee96fd78f42 + .quad 0x94fecebebd4dd72b + .quad 0xf46a4fda060f2211 + .quad 0x124a5977c0c8d1ff + .quad 0x705304b8fb009295 + .quad 0xb63d1d354d296ec6 + .quad 0xf3c3053e5fad31d8 + .quad 0x670b958cb4bd42ec + .quad 0x21398e0ca16353fd + + // 2^76 * 1 * G + + .quad 0x216ab2ca8da7d2ef + .quad 0x366ad9dd99f42827 + .quad 0xae64b9004fdd3c75 + .quad 0x403a395b53909e62 + .quad 0x86c5fc16861b7e9a + .quad 0xf6a330476a27c451 + .quad 0x01667267a1e93597 + .quad 0x05ffb9cd6082dfeb + .quad 0xa617fa9ff53f6139 + .quad 0x60f2b5e513e66cb6 + .quad 0xd7a8beefb3448aa4 + .quad 0x7a2932856f5ea192 + + // 2^76 * 2 * G + + .quad 0x0b39d761b02de888 + .quad 0x5f550e7ed2414e1f + .quad 0xa6bfa45822e1a940 + .quad 0x050a2f7dfd447b99 + .quad 0xb89c444879639302 + .quad 0x4ae4f19350c67f2c + .quad 0xf0b35da8c81af9c6 + .quad 0x39d0003546871017 + .quad 0x437c3b33a650db77 + .quad 0x6bafe81dbac52bb2 + .quad 0xfe99402d2db7d318 + .quad 0x2b5b7eec372ba6ce + + // 2^76 * 3 * G + + .quad 0xb3bc4bbd83f50eef + .quad 0x508f0c998c927866 + .quad 0x43e76587c8b7e66e + .quad 0x0f7655a3a47f98d9 + .quad 0xa694404d613ac8f4 + .quad 0x500c3c2bfa97e72c + .quad 0x874104d21fcec210 + .quad 0x1b205fb38604a8ee + .quad 0x55ecad37d24b133c + .quad 0x441e147d6038c90b + .quad 0x656683a1d62c6fee + .quad 0x0157d5dc87e0ecae + + // 2^76 * 4 * G + + .quad 0xf2a7af510354c13d + .quad 0xd7a0b145aa372b60 + .quad 0x2869b96a05a3d470 + .quad 0x6528e42d82460173 + .quad 0x95265514d71eb524 + .quad 0xe603d8815df14593 + .quad 0x147cdf410d4de6b7 + .quad 0x5293b1730437c850 + .quad 0x23d0e0814bccf226 + .quad 0x92c745cd8196fb93 + .quad 0x8b61796c59541e5b + .quad 0x40a44df0c021f978 + + // 2^76 * 5 * G + + .quad 0xdaa869894f20ea6a + .quad 0xea14a3d14c620618 + .quad 0x6001fccb090bf8be + .quad 0x35f4e822947e9cf0 + .quad 0x86c96e514bc5d095 + .quad 0xf20d4098fca6804a + .quad 0x27363d89c826ea5d + .quad 0x39ca36565719cacf + .quad 0x97506f2f6f87b75c + .quad 0xc624aea0034ae070 + .quad 0x1ec856e3aad34dd6 + .quad 0x055b0be0e440e58f + + // 2^76 * 6 * G + + .quad 0x6469a17d89735d12 + .quad 0xdb6f27d5e662b9f1 + .quad 0x9fcba3286a395681 + .quad 0x363b8004d269af25 + .quad 0x4d12a04b6ea33da2 + .quad 0x57cf4c15e36126dd + .quad 0x90ec9675ee44d967 + .quad 0x64ca348d2a985aac + .quad 0x99588e19e4c4912d + .quad 0xefcc3b4e1ca5ce6b + .quad 0x4522ea60fa5b98d5 + .quad 0x7064bbab1de4a819 + + // 2^76 * 7 * G + + .quad 0xb919e1515a770641 + .quad 0xa9a2e2c74e7f8039 + .quad 0x7527250b3df23109 + .quad 0x756a7330ac27b78b + .quad 0xa290c06142542129 + .quad 0xf2e2c2aebe8d5b90 + .quad 0xcf2458db76abfe1b + .quad 0x02157ade83d626bf + .quad 0x3e46972a1b9a038b + .quad 0x2e4ee66a7ee03fb4 + .quad 0x81a248776edbb4ca + .quad 0x1a944ee88ecd0563 + + // 2^76 * 8 * G + + .quad 0xd5a91d1151039372 + .quad 0x2ed377b799ca26de + .quad 0xa17202acfd366b6b + .quad 0x0730291bd6901995 + .quad 0xbb40a859182362d6 + .quad 0xb99f55778a4d1abb + .quad 0x8d18b427758559f6 + .quad 0x26c20fe74d26235a + .quad 0x648d1d9fe9cc22f5 + .quad 0x66bc561928dd577c + .quad 0x47d3ed21652439d1 + .quad 0x49d271acedaf8b49 + + // 2^80 * 1 * G + + .quad 0x89f5058a382b33f3 + .quad 0x5ae2ba0bad48c0b4 + .quad 0x8f93b503a53db36e + .quad 0x5aa3ed9d95a232e6 + .quad 0x2798aaf9b4b75601 + .quad 0x5eac72135c8dad72 + .quad 0xd2ceaa6161b7a023 + .quad 0x1bbfb284e98f7d4e + .quad 0x656777e9c7d96561 + .quad 0xcb2b125472c78036 + .quad 0x65053299d9506eee + .quad 0x4a07e14e5e8957cc + + // 2^80 * 2 * G + + .quad 0x4ee412cb980df999 + .quad 0xa315d76f3c6ec771 + .quad 0xbba5edde925c77fd + .quad 0x3f0bac391d313402 + .quad 0x240b58cdc477a49b + .quad 0xfd38dade6447f017 + .quad 0x19928d32a7c86aad + .quad 0x50af7aed84afa081 + .quad 0x6e4fde0115f65be5 + .quad 0x29982621216109b2 + .quad 0x780205810badd6d9 + .quad 0x1921a316baebd006 + + // 2^80 * 3 * G + + .quad 0x89422f7edfb870fc + .quad 0x2c296beb4f76b3bd + .quad 0x0738f1d436c24df7 + .quad 0x6458df41e273aeb0 + .quad 0xd75aad9ad9f3c18b + .quad 0x566a0eef60b1c19c + .quad 0x3e9a0bac255c0ed9 + .quad 0x7b049deca062c7f5 + .quad 0xdccbe37a35444483 + .quad 0x758879330fedbe93 + .quad 0x786004c312c5dd87 + .quad 0x6093dccbc2950e64 + + // 2^80 * 4 * G + + .quad 0x1ff39a8585e0706d + .quad 0x36d0a5d8b3e73933 + .quad 0x43b9f2e1718f453b + .quad 0x57d1ea084827a97c + .quad 0x6bdeeebe6084034b + .quad 0x3199c2b6780fb854 + .quad 0x973376abb62d0695 + .quad 0x6e3180c98b647d90 + .quad 0xee7ab6e7a128b071 + .quad 0xa4c1596d93a88baa + .quad 0xf7b4de82b2216130 + .quad 0x363e999ddd97bd18 + + // 2^80 * 5 * G + + .quad 0x96a843c135ee1fc4 + .quad 0x976eb35508e4c8cf + .quad 0xb42f6801b58cd330 + .quad 0x48ee9b78693a052b + .quad 0x2f1848dce24baec6 + .quad 0x769b7255babcaf60 + .quad 0x90cb3c6e3cefe931 + .quad 0x231f979bc6f9b355 + .quad 0x5c31de4bcc2af3c6 + .quad 0xb04bb030fe208d1f + .quad 0xb78d7009c14fb466 + .quad 0x079bfa9b08792413 + + // 2^80 * 6 * G + + .quad 0xe3903a51da300df4 + .quad 0x843964233da95ab0 + .quad 0xed3cf12d0b356480 + .quad 0x038c77f684817194 + .quad 0xf3c9ed80a2d54245 + .quad 0x0aa08b7877f63952 + .quad 0xd76dac63d1085475 + .quad 0x1ef4fb159470636b + .quad 0x854e5ee65b167bec + .quad 0x59590a4296d0cdc2 + .quad 0x72b2df3498102199 + .quad 0x575ee92a4a0bff56 + + // 2^80 * 7 * G + + .quad 0xd4c080908a182fcf + .quad 0x30e170c299489dbd + .quad 0x05babd5752f733de + .quad 0x43d4e7112cd3fd00 + .quad 0x5d46bc450aa4d801 + .quad 0xc3af1227a533b9d8 + .quad 0x389e3b262b8906c2 + .quad 0x200a1e7e382f581b + .quad 0x518db967eaf93ac5 + .quad 0x71bc989b056652c0 + .quad 0xfe2b85d9567197f5 + .quad 0x050eca52651e4e38 + + // 2^80 * 8 * G + + .quad 0xc3431ade453f0c9c + .quad 0xe9f5045eff703b9b + .quad 0xfcd97ac9ed847b3d + .quad 0x4b0ee6c21c58f4c6 + .quad 0x97ac397660e668ea + .quad 0x9b19bbfe153ab497 + .quad 0x4cb179b534eca79f + .quad 0x6151c09fa131ae57 + .quad 0x3af55c0dfdf05d96 + .quad 0xdd262ee02ab4ee7a + .quad 0x11b2bb8712171709 + .quad 0x1fef24fa800f030b + + // 2^84 * 1 * G + + .quad 0xb496123a6b6c6609 + .quad 0xa750fe8580ab5938 + .quad 0xf471bf39b7c27a5f + .quad 0x507903ce77ac193c + .quad 0xff91a66a90166220 + .quad 0xf22552ae5bf1e009 + .quad 0x7dff85d87f90df7c + .quad 0x4f620ffe0c736fb9 + .quad 0x62f90d65dfde3e34 + .quad 0xcf28c592b9fa5fad + .quad 0x99c86ef9c6164510 + .quad 0x25d448044a256c84 + + // 2^84 * 2 * G + + .quad 0xbd68230ec7e9b16f + .quad 0x0eb1b9c1c1c5795d + .quad 0x7943c8c495b6b1ff + .quad 0x2f9faf620bbacf5e + .quad 0x2c7c4415c9022b55 + .quad 0x56a0d241812eb1fe + .quad 0xf02ea1c9d7b65e0d + .quad 0x4180512fd5323b26 + .quad 0xa4ff3e698a48a5db + .quad 0xba6a3806bd95403b + .quad 0x9f7ce1af47d5b65d + .quad 0x15e087e55939d2fb + + // 2^84 * 3 * G + + .quad 0x12207543745c1496 + .quad 0xdaff3cfdda38610c + .quad 0xe4e797272c71c34f + .quad 0x39c07b1934bdede9 + .quad 0x8894186efb963f38 + .quad 0x48a00e80dc639bd5 + .quad 0xa4e8092be96c1c99 + .quad 0x5a097d54ca573661 + .quad 0x2d45892b17c9e755 + .quad 0xd033fd7289308df8 + .quad 0x6c2fe9d9525b8bd9 + .quad 0x2edbecf1c11cc079 + + // 2^84 * 4 * G + + .quad 0x1616a4e3c715a0d2 + .quad 0x53623cb0f8341d4d + .quad 0x96ef5329c7e899cb + .quad 0x3d4e8dbba668baa6 + .quad 0xee0f0fddd087a25f + .quad 0x9c7531555c3e34ee + .quad 0x660c572e8fab3ab5 + .quad 0x0854fc44544cd3b2 + .quad 0x61eba0c555edad19 + .quad 0x24b533fef0a83de6 + .quad 0x3b77042883baa5f8 + .quad 0x678f82b898a47e8d + + // 2^84 * 5 * G + + .quad 0xb1491d0bd6900c54 + .quad 0x3539722c9d132636 + .quad 0x4db928920b362bc9 + .quad 0x4d7cd1fea68b69df + .quad 0x1e09d94057775696 + .quad 0xeed1265c3cd951db + .quad 0xfa9dac2b20bce16f + .quad 0x0f7f76e0e8d089f4 + .quad 0x36d9ebc5d485b00c + .quad 0xa2596492e4adb365 + .quad 0xc1659480c2119ccd + .quad 0x45306349186e0d5f + + // 2^84 * 6 * G + + .quad 0x94ddd0c1a6cdff1d + .quad 0x55f6f115e84213ae + .quad 0x6c935f85992fcf6a + .quad 0x067ee0f54a37f16f + .quad 0x96a414ec2b072491 + .quad 0x1bb2218127a7b65b + .quad 0x6d2849596e8a4af0 + .quad 0x65f3b08ccd27765f + .quad 0xecb29fff199801f7 + .quad 0x9d361d1fa2a0f72f + .quad 0x25f11d2375fd2f49 + .quad 0x124cefe80fe10fe2 + + // 2^84 * 7 * G + + .quad 0x4c126cf9d18df255 + .quad 0xc1d471e9147a63b6 + .quad 0x2c6d3c73f3c93b5f + .quad 0x6be3a6a2e3ff86a2 + .quad 0x1518e85b31b16489 + .quad 0x8faadcb7db710bfb + .quad 0x39b0bdf4a14ae239 + .quad 0x05f4cbea503d20c1 + .quad 0xce040e9ec04145bc + .quad 0xc71ff4e208f6834c + .quad 0xbd546e8dab8847a3 + .quad 0x64666aa0a4d2aba5 + + // 2^84 * 8 * G + + .quad 0x6841435a7c06d912 + .quad 0xca123c21bb3f830b + .quad 0xd4b37b27b1cbe278 + .quad 0x1d753b84c76f5046 + .quad 0xb0c53bf73337e94c + .quad 0x7cb5697e11e14f15 + .quad 0x4b84abac1930c750 + .quad 0x28dd4abfe0640468 + .quad 0x7dc0b64c44cb9f44 + .quad 0x18a3e1ace3925dbf + .quad 0x7a3034862d0457c4 + .quad 0x4c498bf78a0c892e + + // 2^88 * 1 * G + + .quad 0x37d653fb1aa73196 + .quad 0x0f9495303fd76418 + .quad 0xad200b09fb3a17b2 + .quad 0x544d49292fc8613e + .quad 0x22d2aff530976b86 + .quad 0x8d90b806c2d24604 + .quad 0xdca1896c4de5bae5 + .quad 0x28005fe6c8340c17 + .quad 0x6aefba9f34528688 + .quad 0x5c1bff9425107da1 + .quad 0xf75bbbcd66d94b36 + .quad 0x72e472930f316dfa + + // 2^88 * 2 * G + + .quad 0x2695208c9781084f + .quad 0xb1502a0b23450ee1 + .quad 0xfd9daea603efde02 + .quad 0x5a9d2e8c2733a34c + .quad 0x07f3f635d32a7627 + .quad 0x7aaa4d865f6566f0 + .quad 0x3c85e79728d04450 + .quad 0x1fee7f000fe06438 + .quad 0x765305da03dbf7e5 + .quad 0xa4daf2491434cdbd + .quad 0x7b4ad5cdd24a88ec + .quad 0x00f94051ee040543 + + // 2^88 * 3 * G + + .quad 0x8d356b23c3d330b2 + .quad 0xf21c8b9bb0471b06 + .quad 0xb36c316c6e42b83c + .quad 0x07d79c7e8beab10d + .quad 0xd7ef93bb07af9753 + .quad 0x583ed0cf3db766a7 + .quad 0xce6998bf6e0b1ec5 + .quad 0x47b7ffd25dd40452 + .quad 0x87fbfb9cbc08dd12 + .quad 0x8a066b3ae1eec29b + .quad 0x0d57242bdb1fc1bf + .quad 0x1c3520a35ea64bb6 + + // 2^88 * 4 * G + + .quad 0x80d253a6bccba34a + .quad 0x3e61c3a13838219b + .quad 0x90c3b6019882e396 + .quad 0x1c3d05775d0ee66f + .quad 0xcda86f40216bc059 + .quad 0x1fbb231d12bcd87e + .quad 0xb4956a9e17c70990 + .quad 0x38750c3b66d12e55 + .quad 0x692ef1409422e51a + .quad 0xcbc0c73c2b5df671 + .quad 0x21014fe7744ce029 + .quad 0x0621e2c7d330487c + + // 2^88 * 5 * G + + .quad 0xaf9860cc8259838d + .quad 0x90ea48c1c69f9adc + .quad 0x6526483765581e30 + .quad 0x0007d6097bd3a5bc + .quad 0xb7ae1796b0dbf0f3 + .quad 0x54dfafb9e17ce196 + .quad 0x25923071e9aaa3b4 + .quad 0x5d8e589ca1002e9d + .quad 0xc0bf1d950842a94b + .quad 0xb2d3c363588f2e3e + .quad 0x0a961438bb51e2ef + .quad 0x1583d7783c1cbf86 + + // 2^88 * 6 * G + + .quad 0xeceea2ef5da27ae1 + .quad 0x597c3a1455670174 + .quad 0xc9a62a126609167a + .quad 0x252a5f2e81ed8f70 + .quad 0x90034704cc9d28c7 + .quad 0x1d1b679ef72cc58f + .quad 0x16e12b5fbe5b8726 + .quad 0x4958064e83c5580a + .quad 0x0d2894265066e80d + .quad 0xfcc3f785307c8c6b + .quad 0x1b53da780c1112fd + .quad 0x079c170bd843b388 + + // 2^88 * 7 * G + + .quad 0x0506ece464fa6fff + .quad 0xbee3431e6205e523 + .quad 0x3579422451b8ea42 + .quad 0x6dec05e34ac9fb00 + .quad 0xcdd6cd50c0d5d056 + .quad 0x9af7686dbb03573b + .quad 0x3ca6723ff3c3ef48 + .quad 0x6768c0d7317b8acc + .quad 0x94b625e5f155c1b3 + .quad 0x417bf3a7997b7b91 + .quad 0xc22cbddc6d6b2600 + .quad 0x51445e14ddcd52f4 + + // 2^88 * 8 * G + + .quad 0x57502b4b3b144951 + .quad 0x8e67ff6b444bbcb3 + .quad 0xb8bd6927166385db + .quad 0x13186f31e39295c8 + .quad 0x893147ab2bbea455 + .quad 0x8c53a24f92079129 + .quad 0x4b49f948be30f7a7 + .quad 0x12e990086e4fd43d + .quad 0xf10c96b37fdfbb2e + .quad 0x9f9a935e121ceaf9 + .quad 0xdf1136c43a5b983f + .quad 0x77b2e3f05d3e99af + + // 2^92 * 1 * G + + .quad 0xfd0d75879cf12657 + .quad 0xe82fef94e53a0e29 + .quad 0xcc34a7f05bbb4be7 + .quad 0x0b251172a50c38a2 + .quad 0x9532f48fcc5cd29b + .quad 0x2ba851bea3ce3671 + .quad 0x32dacaa051122941 + .quad 0x478d99d9350004f2 + .quad 0x1d5ad94890bb02c0 + .quad 0x50e208b10ec25115 + .quad 0xa26a22894ef21702 + .quad 0x4dc923343b524805 + + // 2^92 * 2 * G + + .quad 0xe3828c400f8086b6 + .quad 0x3f77e6f7979f0dc8 + .quad 0x7ef6de304df42cb4 + .quad 0x5265797cb6abd784 + .quad 0x3ad3e3ebf36c4975 + .quad 0xd75d25a537862125 + .quad 0xe873943da025a516 + .quad 0x6bbc7cb4c411c847 + .quad 0x3c6f9cd1d4a50d56 + .quad 0xb6244077c6feab7e + .quad 0x6ff9bf483580972e + .quad 0x00375883b332acfb + + // 2^92 * 3 * G + + .quad 0x0001b2cd28cb0940 + .quad 0x63fb51a06f1c24c9 + .quad 0xb5ad8691dcd5ca31 + .quad 0x67238dbd8c450660 + .quad 0xc98bec856c75c99c + .quad 0xe44184c000e33cf4 + .quad 0x0a676b9bba907634 + .quad 0x669e2cb571f379d7 + .quad 0xcb116b73a49bd308 + .quad 0x025aad6b2392729e + .quad 0xb4793efa3f55d9b1 + .quad 0x72a1056140678bb9 + + // 2^92 * 4 * G + + .quad 0xa2b6812b1cc9249d + .quad 0x62866eee21211f58 + .quad 0x2cb5c5b85df10ece + .quad 0x03a6b259e263ae00 + .quad 0x0d8d2909e2e505b6 + .quad 0x98ca78abc0291230 + .quad 0x77ef5569a9b12327 + .quad 0x7c77897b81439b47 + .quad 0xf1c1b5e2de331cb5 + .quad 0x5a9f5d8e15fca420 + .quad 0x9fa438f17bd932b1 + .quad 0x2a381bf01c6146e7 + + // 2^92 * 5 * G + + .quad 0xac9b9879cfc811c1 + .quad 0x8b7d29813756e567 + .quad 0x50da4e607c70edfc + .quad 0x5dbca62f884400b6 + .quad 0xf7c0be32b534166f + .quad 0x27e6ca6419cf70d4 + .quad 0x934df7d7a957a759 + .quad 0x5701461dabdec2aa + .quad 0x2c6747402c915c25 + .quad 0x1bdcd1a80b0d340a + .quad 0x5e5601bd07b43f5f + .quad 0x2555b4e05539a242 + + // 2^92 * 6 * G + + .quad 0x6fc09f5266ddd216 + .quad 0xdce560a7c8e37048 + .quad 0xec65939da2df62fd + .quad 0x7a869ae7e52ed192 + .quad 0x78409b1d87e463d4 + .quad 0xad4da95acdfb639d + .quad 0xec28773755259b9c + .quad 0x69c806e9c31230ab + .quad 0x7b48f57414bb3f22 + .quad 0x68c7cee4aedccc88 + .quad 0xed2f936179ed80be + .quad 0x25d70b885f77bc4b + + // 2^92 * 7 * G + + .quad 0x4151c3d9762bf4de + .quad 0x083f435f2745d82b + .quad 0x29775a2e0d23ddd5 + .quad 0x138e3a6269a5db24 + .quad 0x98459d29bb1ae4d4 + .quad 0x56b9c4c739f954ec + .quad 0x832743f6c29b4b3e + .quad 0x21ea8e2798b6878a + .quad 0x87bef4b46a5a7b9c + .quad 0xd2299d1b5fc1d062 + .quad 0x82409818dd321648 + .quad 0x5c5abeb1e5a2e03d + + // 2^92 * 8 * G + + .quad 0x14722af4b73c2ddb + .quad 0xbc470c5f5a05060d + .quad 0x00943eac2581b02e + .quad 0x0e434b3b1f499c8f + .quad 0x02cde6de1306a233 + .quad 0x7b5a52a2116f8ec7 + .quad 0xe1c681f4c1163b5b + .quad 0x241d350660d32643 + .quad 0x6be4404d0ebc52c7 + .quad 0xae46233bb1a791f5 + .quad 0x2aec170ed25db42b + .quad 0x1d8dfd966645d694 + + // 2^96 * 1 * G + + .quad 0x296fa9c59c2ec4de + .quad 0xbc8b61bf4f84f3cb + .quad 0x1c7706d917a8f908 + .quad 0x63b795fc7ad3255d + .quad 0xd598639c12ddb0a4 + .quad 0xa5d19f30c024866b + .quad 0xd17c2f0358fce460 + .quad 0x07a195152e095e8a + .quad 0xa8368f02389e5fc8 + .quad 0x90433b02cf8de43b + .quad 0xafa1fd5dc5412643 + .quad 0x3e8fe83d032f0137 + + // 2^96 * 2 * G + + .quad 0x2f8b15b90570a294 + .quad 0x94f2427067084549 + .quad 0xde1c5ae161bbfd84 + .quad 0x75ba3b797fac4007 + .quad 0x08704c8de8efd13c + .quad 0xdfc51a8e33e03731 + .quad 0xa59d5da51260cde3 + .quad 0x22d60899a6258c86 + .quad 0x6239dbc070cdd196 + .quad 0x60fe8a8b6c7d8a9a + .quad 0xb38847bceb401260 + .quad 0x0904d07b87779e5e + + // 2^96 * 3 * G + + .quad 0xb4ce1fd4ddba919c + .quad 0xcf31db3ec74c8daa + .quad 0x2c63cc63ad86cc51 + .quad 0x43e2143fbc1dde07 + .quad 0xf4322d6648f940b9 + .quad 0x06952f0cbd2d0c39 + .quad 0x167697ada081f931 + .quad 0x6240aacebaf72a6c + .quad 0xf834749c5ba295a0 + .quad 0xd6947c5bca37d25a + .quad 0x66f13ba7e7c9316a + .quad 0x56bdaf238db40cac + + // 2^96 * 4 * G + + .quad 0x362ab9e3f53533eb + .quad 0x338568d56eb93d40 + .quad 0x9e0e14521d5a5572 + .quad 0x1d24a86d83741318 + .quad 0x1310d36cc19d3bb2 + .quad 0x062a6bb7622386b9 + .quad 0x7c9b8591d7a14f5c + .quad 0x03aa31507e1e5754 + .quad 0xf4ec7648ffd4ce1f + .quad 0xe045eaf054ac8c1c + .quad 0x88d225821d09357c + .quad 0x43b261dc9aeb4859 + + // 2^96 * 5 * G + + .quad 0xe55b1e1988bb79bb + .quad 0xa09ed07dc17a359d + .quad 0xb02c2ee2603dea33 + .quad 0x326055cf5b276bc2 + .quad 0x19513d8b6c951364 + .quad 0x94fe7126000bf47b + .quad 0x028d10ddd54f9567 + .quad 0x02b4d5e242940964 + .quad 0xb4a155cb28d18df2 + .quad 0xeacc4646186ce508 + .quad 0xc49cf4936c824389 + .quad 0x27a6c809ae5d3410 + + // 2^96 * 6 * G + + .quad 0x8ba6ebcd1f0db188 + .quad 0x37d3d73a675a5be8 + .quad 0xf22edfa315f5585a + .quad 0x2cb67174ff60a17e + .quad 0xcd2c270ac43d6954 + .quad 0xdd4a3e576a66cab2 + .quad 0x79fa592469d7036c + .quad 0x221503603d8c2599 + .quad 0x59eecdf9390be1d0 + .quad 0xa9422044728ce3f1 + .quad 0x82891c667a94f0f4 + .quad 0x7b1df4b73890f436 + + // 2^96 * 7 * G + + .quad 0xe492f2e0b3b2a224 + .quad 0x7c6c9e062b551160 + .quad 0x15eb8fe20d7f7b0e + .quad 0x61fcef2658fc5992 + .quad 0x5f2e221807f8f58c + .quad 0xe3555c9fd49409d4 + .quad 0xb2aaa88d1fb6a630 + .quad 0x68698245d352e03d + .quad 0xdbb15d852a18187a + .quad 0xf3e4aad386ddacd7 + .quad 0x44bae2810ff6c482 + .quad 0x46cf4c473daf01cf + + // 2^96 * 8 * G + + .quad 0x426525ed9ec4e5f9 + .quad 0x0e5eda0116903303 + .quad 0x72b1a7f2cbe5cadc + .quad 0x29387bcd14eb5f40 + .quad 0x213c6ea7f1498140 + .quad 0x7c1e7ef8392b4854 + .quad 0x2488c38c5629ceba + .quad 0x1065aae50d8cc5bb + .quad 0x1c2c4525df200d57 + .quad 0x5c3b2dd6bfca674a + .quad 0x0a07e7b1e1834030 + .quad 0x69a198e64f1ce716 + + // 2^100 * 1 * G + + .quad 0x7afcd613efa9d697 + .quad 0x0cc45aa41c067959 + .quad 0xa56fe104c1fada96 + .quad 0x3a73b70472e40365 + .quad 0x7b26e56b9e2d4734 + .quad 0xc4c7132b81c61675 + .quad 0xef5c9525ec9cde7f + .quad 0x39c80b16e71743ad + .quad 0x0f196e0d1b826c68 + .quad 0xf71ff0e24960e3db + .quad 0x6113167023b7436c + .quad 0x0cf0ea5877da7282 + + // 2^100 * 2 * G + + .quad 0x196c80a4ddd4ccbd + .quad 0x22e6f55d95f2dd9d + .quad 0xc75e33c740d6c71b + .quad 0x7bb51279cb3c042f + .quad 0xe332ced43ba6945a + .quad 0xde0b1361e881c05d + .quad 0x1ad40f095e67ed3b + .quad 0x5da8acdab8c63d5d + .quad 0xc4b6664a3a70159f + .quad 0x76194f0f0a904e14 + .quad 0xa5614c39a4096c13 + .quad 0x6cd0ff50979feced + + // 2^100 * 3 * G + + .quad 0xc0e067e78f4428ac + .quad 0x14835ab0a61135e3 + .quad 0xf21d14f338062935 + .quad 0x6390a4c8df04849c + .quad 0x7fecfabdb04ba18e + .quad 0xd0fc7bfc3bddbcf7 + .quad 0xa41d486e057a131c + .quad 0x641a4391f2223a61 + .quad 0xc5c6b95aa606a8db + .quad 0x914b7f9eb06825f1 + .quad 0x2a731f6b44fc9eff + .quad 0x30ddf38562705cfc + + // 2^100 * 4 * G + + .quad 0x4e3dcbdad1bff7f9 + .quad 0xc9118e8220645717 + .quad 0xbacccebc0f189d56 + .quad 0x1b4822e9d4467668 + .quad 0x33bef2bd68bcd52c + .quad 0xc649dbb069482ef2 + .quad 0xb5b6ee0c41cb1aee + .quad 0x5c294d270212a7e5 + .quad 0xab360a7f25563781 + .quad 0x2512228a480f7958 + .quad 0xc75d05276114b4e3 + .quad 0x222d9625d976fe2a + + // 2^100 * 5 * G + + .quad 0x1c717f85b372ace1 + .quad 0x81930e694638bf18 + .quad 0x239cad056bc08b58 + .quad 0x0b34271c87f8fff4 + .quad 0x0f94be7e0a344f85 + .quad 0xeb2faa8c87f22c38 + .quad 0x9ce1e75e4ee16f0f + .quad 0x43e64e5418a08dea + .quad 0x8155e2521a35ce63 + .quad 0xbe100d4df912028e + .quad 0xbff80bf8a57ddcec + .quad 0x57342dc96d6bc6e4 + + // 2^100 * 6 * G + + .quad 0xefeef065c8ce5998 + .quad 0xbf029510b5cbeaa2 + .quad 0x8c64a10620b7c458 + .quad 0x35134fb231c24855 + .quad 0xf3c3bcb71e707bf6 + .quad 0x351d9b8c7291a762 + .quad 0x00502e6edad69a33 + .quad 0x522f521f1ec8807f + .quad 0x272c1f46f9a3902b + .quad 0xc91ba3b799657bcc + .quad 0xae614b304f8a1c0e + .quad 0x7afcaad70b99017b + + // 2^100 * 7 * G + + .quad 0xc25ded54a4b8be41 + .quad 0x902d13e11bb0e2dd + .quad 0x41f43233cde82ab2 + .quad 0x1085faa5c3aae7cb + .quad 0xa88141ecef842b6b + .quad 0x55e7b14797abe6c5 + .quad 0x8c748f9703784ffe + .quad 0x5b50a1f7afcd00b7 + .quad 0x9b840f66f1361315 + .quad 0x18462242701003e9 + .quad 0x65ed45fae4a25080 + .quad 0x0a2862393fda7320 + + // 2^100 * 8 * G + + .quad 0x46ab13c8347cbc9d + .quad 0x3849e8d499c12383 + .quad 0x4cea314087d64ac9 + .quad 0x1f354134b1a29ee7 + .quad 0x960e737b6ecb9d17 + .quad 0xfaf24948d67ceae1 + .quad 0x37e7a9b4d55e1b89 + .quad 0x5cb7173cb46c59eb + .quad 0x4a89e68b82b7abf0 + .quad 0xf41cd9279ba6b7b9 + .quad 0x16e6c210e18d876f + .quad 0x7cacdb0f7f1b09c6 + + // 2^104 * 1 * G + + .quad 0x9062b2e0d91a78bc + .quad 0x47c9889cc8509667 + .quad 0x9df54a66405070b8 + .quad 0x7369e6a92493a1bf + .quad 0xe1014434dcc5caed + .quad 0x47ed5d963c84fb33 + .quad 0x70019576ed86a0e7 + .quad 0x25b2697bd267f9e4 + .quad 0x9d673ffb13986864 + .quad 0x3ca5fbd9415dc7b8 + .quad 0xe04ecc3bdf273b5e + .quad 0x1420683db54e4cd2 + + // 2^104 * 2 * G + + .quad 0xb478bd1e249dd197 + .quad 0x620c35005e58c102 + .quad 0xfb02d32fccbaac5c + .quad 0x60b63bebf508a72d + .quad 0x34eebb6fc1cc5ad0 + .quad 0x6a1b0ce99646ac8b + .quad 0xd3b0da49a66bde53 + .quad 0x31e83b4161d081c1 + .quad 0x97e8c7129e062b4f + .quad 0x49e48f4f29320ad8 + .quad 0x5bece14b6f18683f + .quad 0x55cf1eb62d550317 + + // 2^104 * 3 * G + + .quad 0x5879101065c23d58 + .quad 0x8b9d086d5094819c + .quad 0xe2402fa912c55fa7 + .quad 0x669a6564570891d4 + .quad 0x3076b5e37df58c52 + .quad 0xd73ab9dde799cc36 + .quad 0xbd831ce34913ee20 + .quad 0x1a56fbaa62ba0133 + .quad 0x943e6b505c9dc9ec + .quad 0x302557bba77c371a + .quad 0x9873ae5641347651 + .quad 0x13c4836799c58a5c + + // 2^104 * 4 * G + + .quad 0x423a5d465ab3e1b9 + .quad 0xfc13c187c7f13f61 + .quad 0x19f83664ecb5b9b6 + .quad 0x66f80c93a637b607 + .quad 0xc4dcfb6a5d8bd080 + .quad 0xdeebc4ec571a4842 + .quad 0xd4b2e883b8e55365 + .quad 0x50bdc87dc8e5b827 + .quad 0x606d37836edfe111 + .quad 0x32353e15f011abd9 + .quad 0x64b03ac325b73b96 + .quad 0x1dd56444725fd5ae + + // 2^104 * 5 * G + + .quad 0x8fa47ff83362127d + .quad 0xbc9f6ac471cd7c15 + .quad 0x6e71454349220c8b + .quad 0x0e645912219f732e + .quad 0xc297e60008bac89a + .quad 0x7d4cea11eae1c3e0 + .quad 0xf3e38be19fe7977c + .quad 0x3a3a450f63a305cd + .quad 0x078f2f31d8394627 + .quad 0x389d3183de94a510 + .quad 0xd1e36c6d17996f80 + .quad 0x318c8d9393a9a87b + + // 2^104 * 6 * G + + .quad 0xf2745d032afffe19 + .quad 0x0c9f3c497f24db66 + .quad 0xbc98d3e3ba8598ef + .quad 0x224c7c679a1d5314 + .quad 0x5d669e29ab1dd398 + .quad 0xfc921658342d9e3b + .quad 0x55851dfdf35973cd + .quad 0x509a41c325950af6 + .quad 0xbdc06edca6f925e9 + .quad 0x793ef3f4641b1f33 + .quad 0x82ec12809d833e89 + .quad 0x05bff02328a11389 + + // 2^104 * 7 * G + + .quad 0x3632137023cae00b + .quad 0x544acf0ad1accf59 + .quad 0x96741049d21a1c88 + .quad 0x780b8cc3fa2a44a7 + .quad 0x6881a0dd0dc512e4 + .quad 0x4fe70dc844a5fafe + .quad 0x1f748e6b8f4a5240 + .quad 0x576277cdee01a3ea + .quad 0x1ef38abc234f305f + .quad 0x9a577fbd1405de08 + .quad 0x5e82a51434e62a0d + .quad 0x5ff418726271b7a1 + + // 2^104 * 8 * G + + .quad 0x398e080c1789db9d + .quad 0xa7602025f3e778f5 + .quad 0xfa98894c06bd035d + .quad 0x106a03dc25a966be + .quad 0xe5db47e813b69540 + .quad 0xf35d2a3b432610e1 + .quad 0xac1f26e938781276 + .quad 0x29d4db8ca0a0cb69 + .quad 0xd9ad0aaf333353d0 + .quad 0x38669da5acd309e5 + .quad 0x3c57658ac888f7f0 + .quad 0x4ab38a51052cbefa + + // 2^108 * 1 * G + + .quad 0xdfdacbee4324c0e9 + .quad 0x054442883f955bb7 + .quad 0xdef7aaa8ea31609f + .quad 0x68aee70642287cff + .quad 0xf68fe2e8809de054 + .quad 0xe3bc096a9c82bad1 + .quad 0x076353d40aadbf45 + .quad 0x7b9b1fb5dea1959e + .quad 0xf01cc8f17471cc0c + .quad 0x95242e37579082bb + .quad 0x27776093d3e46b5f + .quad 0x2d13d55a28bd85fb + + // 2^108 * 2 * G + + .quad 0xfac5d2065b35b8da + .quad 0xa8da8a9a85624bb7 + .quad 0xccd2ca913d21cd0f + .quad 0x6b8341ee8bf90d58 + .quad 0xbf019cce7aee7a52 + .quad 0xa8ded2b6e454ead3 + .quad 0x3c619f0b87a8bb19 + .quad 0x3619b5d7560916d8 + .quad 0x3579f26b0282c4b2 + .quad 0x64d592f24fafefae + .quad 0xb7cded7b28c8c7c0 + .quad 0x6a927b6b7173a8d7 + + // 2^108 * 3 * G + + .quad 0x1f6db24f986e4656 + .quad 0x1021c02ed1e9105b + .quad 0xf8ff3fff2cc0a375 + .quad 0x1d2a6bf8c6c82592 + .quad 0x8d7040863ece88eb + .quad 0xf0e307a980eec08c + .quad 0xac2250610d788fda + .quad 0x056d92a43a0d478d + .quad 0x1b05a196fc3da5a1 + .quad 0x77d7a8c243b59ed0 + .quad 0x06da3d6297d17918 + .quad 0x66fbb494f12353f7 + + // 2^108 * 4 * G + + .quad 0x751a50b9d85c0fb8 + .quad 0xd1afdc258bcf097b + .quad 0x2f16a6a38309a969 + .quad 0x14ddff9ee5b00659 + .quad 0xd6d70996f12309d6 + .quad 0xdbfb2385e9c3d539 + .quad 0x46d602b0f7552411 + .quad 0x270a0b0557843e0c + .quad 0x61ff0640a7862bcc + .quad 0x81cac09a5f11abfe + .quad 0x9047830455d12abb + .quad 0x19a4bde1945ae873 + + // 2^108 * 5 * G + + .quad 0x9b9f26f520a6200a + .quad 0x64804443cf13eaf8 + .quad 0x8a63673f8631edd3 + .quad 0x72bbbce11ed39dc1 + .quad 0x40c709dec076c49f + .quad 0x657bfaf27f3e53f6 + .quad 0x40662331eca042c4 + .quad 0x14b375487eb4df04 + .quad 0xae853c94ab66dc47 + .quad 0xeb62343edf762d6e + .quad 0xf08e0e186fb2f7d1 + .quad 0x4f0b1c02700ab37a + + // 2^108 * 6 * G + + .quad 0xe1706787d81951fa + .quad 0xa10a2c8eb290c77b + .quad 0xe7382fa03ed66773 + .quad 0x0a4d84710bcc4b54 + .quad 0x79fd21ccc1b2e23f + .quad 0x4ae7c281453df52a + .quad 0xc8172ec9d151486b + .quad 0x68abe9443e0a7534 + .quad 0xda12c6c407831dcb + .quad 0x0da230d74d5c510d + .quad 0x4ab1531e6bd404e1 + .quad 0x4106b166bcf440ef + + // 2^108 * 7 * G + + .quad 0x02e57a421cd23668 + .quad 0x4ad9fb5d0eaef6fd + .quad 0x954e6727b1244480 + .quad 0x7f792f9d2699f331 + .quad 0xa485ccd539e4ecf2 + .quad 0x5aa3f3ad0555bab5 + .quad 0x145e3439937df82d + .quad 0x1238b51e1214283f + .quad 0x0b886b925fd4d924 + .quad 0x60906f7a3626a80d + .quad 0xecd367b4b98abd12 + .quad 0x2876beb1def344cf + + // 2^108 * 8 * G + + .quad 0xdc84e93563144691 + .quad 0x632fe8a0d61f23f4 + .quad 0x4caa800612a9a8d5 + .quad 0x48f9dbfa0e9918d3 + .quad 0xd594b3333a8a85f8 + .quad 0x4ea37689e78d7d58 + .quad 0x73bf9f455e8e351f + .quad 0x5507d7d2bc41ebb4 + .quad 0x1ceb2903299572fc + .quad 0x7c8ccaa29502d0ee + .quad 0x91bfa43411cce67b + .quad 0x5784481964a831e7 + + // 2^112 * 1 * G + + .quad 0xda7c2b256768d593 + .quad 0x98c1c0574422ca13 + .quad 0xf1a80bd5ca0ace1d + .quad 0x29cdd1adc088a690 + .quad 0xd6cfd1ef5fddc09c + .quad 0xe82b3efdf7575dce + .quad 0x25d56b5d201634c2 + .quad 0x3041c6bb04ed2b9b + .quad 0x0ff2f2f9d956e148 + .quad 0xade797759f356b2e + .quad 0x1a4698bb5f6c025c + .quad 0x104bbd6814049a7b + + // 2^112 * 2 * G + + .quad 0x51f0fd3168f1ed67 + .quad 0x2c811dcdd86f3bc2 + .quad 0x44dc5c4304d2f2de + .quad 0x5be8cc57092a7149 + .quad 0xa95d9a5fd67ff163 + .quad 0xe92be69d4cc75681 + .quad 0xb7f8024cde20f257 + .quad 0x204f2a20fb072df5 + .quad 0xc8143b3d30ebb079 + .quad 0x7589155abd652e30 + .quad 0x653c3c318f6d5c31 + .quad 0x2570fb17c279161f + + // 2^112 * 3 * G + + .quad 0x3efa367f2cb61575 + .quad 0xf5f96f761cd6026c + .quad 0xe8c7142a65b52562 + .quad 0x3dcb65ea53030acd + .quad 0x192ea9550bb8245a + .quad 0xc8e6fba88f9050d1 + .quad 0x7986ea2d88a4c935 + .quad 0x241c5f91de018668 + .quad 0x28d8172940de6caa + .quad 0x8fbf2cf022d9733a + .quad 0x16d7fcdd235b01d1 + .quad 0x08420edd5fcdf0e5 + + // 2^112 * 4 * G + + .quad 0xcdff20ab8362fa4a + .quad 0x57e118d4e21a3e6e + .quad 0xe3179617fc39e62b + .quad 0x0d9a53efbc1769fd + .quad 0x0358c34e04f410ce + .quad 0xb6135b5a276e0685 + .quad 0x5d9670c7ebb91521 + .quad 0x04d654f321db889c + .quad 0x5e7dc116ddbdb5d5 + .quad 0x2954deb68da5dd2d + .quad 0x1cb608173334a292 + .quad 0x4a7a4f2618991ad7 + + // 2^112 * 5 * G + + .quad 0xf4a718025fb15f95 + .quad 0x3df65f346b5c1b8f + .quad 0xcdfcf08500e01112 + .quad 0x11b50c4cddd31848 + .quad 0x24c3b291af372a4b + .quad 0x93da8270718147f2 + .quad 0xdd84856486899ef2 + .quad 0x4a96314223e0ee33 + .quad 0xa6e8274408a4ffd6 + .quad 0x738e177e9c1576d9 + .quad 0x773348b63d02b3f2 + .quad 0x4f4bce4dce6bcc51 + + // 2^112 * 6 * G + + .quad 0xa71fce5ae2242584 + .quad 0x26ea725692f58a9e + .quad 0xd21a09d71cea3cf4 + .quad 0x73fcdd14b71c01e6 + .quad 0x30e2616ec49d0b6f + .quad 0xe456718fcaec2317 + .quad 0x48eb409bf26b4fa6 + .quad 0x3042cee561595f37 + .quad 0x427e7079449bac41 + .quad 0x855ae36dbce2310a + .quad 0x4cae76215f841a7c + .quad 0x389e740c9a9ce1d6 + + // 2^112 * 7 * G + + .quad 0x64fcb3ae34dcb9ce + .quad 0x97500323e348d0ad + .quad 0x45b3f07d62c6381b + .quad 0x61545379465a6788 + .quad 0xc9bd78f6570eac28 + .quad 0xe55b0b3227919ce1 + .quad 0x65fc3eaba19b91ed + .quad 0x25c425e5d6263690 + .quad 0x3f3e06a6f1d7de6e + .quad 0x3ef976278e062308 + .quad 0x8c14f6264e8a6c77 + .quad 0x6539a08915484759 + + // 2^112 * 8 * G + + .quad 0xe9d21f74c3d2f773 + .quad 0xc150544125c46845 + .quad 0x624e5ce8f9b99e33 + .quad 0x11c5e4aac5cd186c + .quad 0xddc4dbd414bb4a19 + .quad 0x19b2bc3c98424f8e + .quad 0x48a89fd736ca7169 + .quad 0x0f65320ef019bd90 + .quad 0xd486d1b1cafde0c6 + .quad 0x4f3fe6e3163b5181 + .quad 0x59a8af0dfaf2939a + .quad 0x4cabc7bdec33072a + + // 2^116 * 1 * G + + .quad 0x16faa8fb532f7428 + .quad 0xdbd42ea046a4e272 + .quad 0x5337653b8b9ea480 + .quad 0x4065947223973f03 + .quad 0xf7c0a19c1a54a044 + .quad 0x4a1c5e2477bd9fbb + .quad 0xa6e3ca115af22972 + .quad 0x1819bb953f2e9e0d + .quad 0x498fbb795e042e84 + .quad 0x7d0dd89a7698b714 + .quad 0x8bfb0ba427fe6295 + .quad 0x36ba82e721200524 + + // 2^116 * 2 * G + + .quad 0xd60ecbb74245ec41 + .quad 0xfd9be89e34348716 + .quad 0xc9240afee42284de + .quad 0x4472f648d0531db4 + .quad 0xc8d69d0a57274ed5 + .quad 0x45ba803260804b17 + .quad 0xdf3cda102255dfac + .quad 0x77d221232709b339 + .quad 0x498a6d7064ad94d8 + .quad 0xa5b5c8fd9af62263 + .quad 0x8ca8ed0545c141f4 + .quad 0x2c63bec3662d358c + + // 2^116 * 3 * G + + .quad 0x7fe60d8bea787955 + .quad 0xb9dc117eb5f401b7 + .quad 0x91c7c09a19355cce + .quad 0x22692ef59442bedf + .quad 0x9a518b3a8586f8bf + .quad 0x9ee71af6cbb196f0 + .quad 0xaa0625e6a2385cf2 + .quad 0x1deb2176ddd7c8d1 + .quad 0x8563d19a2066cf6c + .quad 0x401bfd8c4dcc7cd7 + .quad 0xd976a6becd0d8f62 + .quad 0x67cfd773a278b05e + + // 2^116 * 4 * G + + .quad 0x8dec31faef3ee475 + .quad 0x99dbff8a9e22fd92 + .quad 0x512d11594e26cab1 + .quad 0x0cde561eec4310b9 + .quad 0x2d5fa9855a4e586a + .quad 0x65f8f7a449beab7e + .quad 0xaa074dddf21d33d3 + .quad 0x185cba721bcb9dee + .quad 0x93869da3f4e3cb41 + .quad 0xbf0392f540f7977e + .quad 0x026204fcd0463b83 + .quad 0x3ec91a769eec6eed + + // 2^116 * 5 * G + + .quad 0x1e9df75bf78166ad + .quad 0x4dfda838eb0cd7af + .quad 0xba002ed8c1eaf988 + .quad 0x13fedb3e11f33cfc + .quad 0x0fad2fb7b0a3402f + .quad 0x46615ecbfb69f4a8 + .quad 0xf745bcc8c5f8eaa6 + .quad 0x7a5fa8794a94e896 + .quad 0x52958faa13cd67a1 + .quad 0x965ee0818bdbb517 + .quad 0x16e58daa2e8845b3 + .quad 0x357d397d5499da8f + + // 2^116 * 6 * G + + .quad 0x1ebfa05fb0bace6c + .quad 0xc934620c1caf9a1e + .quad 0xcc771cc41d82b61a + .quad 0x2d94a16aa5f74fec + .quad 0x481dacb4194bfbf8 + .quad 0x4d77e3f1bae58299 + .quad 0x1ef4612e7d1372a0 + .quad 0x3a8d867e70ff69e1 + .quad 0x6f58cd5d55aff958 + .quad 0xba3eaa5c75567721 + .quad 0x75c123999165227d + .quad 0x69be1343c2f2b35e + + // 2^116 * 7 * G + + .quad 0x0e091d5ee197c92a + .quad 0x4f51019f2945119f + .quad 0x143679b9f034e99c + .quad 0x7d88112e4d24c696 + .quad 0x82bbbdac684b8de3 + .quad 0xa2f4c7d03fca0718 + .quad 0x337f92fbe096aaa8 + .quad 0x200d4d8c63587376 + .quad 0x208aed4b4893b32b + .quad 0x3efbf23ebe59b964 + .quad 0xd762deb0dba5e507 + .quad 0x69607bd681bd9d94 + + // 2^116 * 8 * G + + .quad 0xf6be021068de1ce1 + .quad 0xe8d518e70edcbc1f + .quad 0xe3effdd01b5505a5 + .quad 0x35f63353d3ec3fd0 + .quad 0x3b7f3bd49323a902 + .quad 0x7c21b5566b2c6e53 + .quad 0xe5ba8ff53a7852a7 + .quad 0x28bc77a5838ece00 + .quad 0x63ba78a8e25d8036 + .quad 0x63651e0094333490 + .quad 0x48d82f20288ce532 + .quad 0x3a31abfa36b57524 + + // 2^120 * 1 * G + + .quad 0x239e9624089c0a2e + .quad 0xc748c4c03afe4738 + .quad 0x17dbed2a764fa12a + .quad 0x639b93f0321c8582 + .quad 0xc08f788f3f78d289 + .quad 0xfe30a72ca1404d9f + .quad 0xf2778bfccf65cc9d + .quad 0x7ee498165acb2021 + .quad 0x7bd508e39111a1c3 + .quad 0x2b2b90d480907489 + .quad 0xe7d2aec2ae72fd19 + .quad 0x0edf493c85b602a6 + + // 2^120 * 2 * G + + .quad 0xaecc8158599b5a68 + .quad 0xea574f0febade20e + .quad 0x4fe41d7422b67f07 + .quad 0x403b92e3019d4fb4 + .quad 0x6767c4d284764113 + .quad 0xa090403ff7f5f835 + .quad 0x1c8fcffacae6bede + .quad 0x04c00c54d1dfa369 + .quad 0x4dc22f818b465cf8 + .quad 0x71a0f35a1480eff8 + .quad 0xaee8bfad04c7d657 + .quad 0x355bb12ab26176f4 + + // 2^120 * 3 * G + + .quad 0xa71e64cc7493bbf4 + .quad 0xe5bd84d9eca3b0c3 + .quad 0x0a6bc50cfa05e785 + .quad 0x0f9b8132182ec312 + .quad 0xa301dac75a8c7318 + .quad 0xed90039db3ceaa11 + .quad 0x6f077cbf3bae3f2d + .quad 0x7518eaf8e052ad8e + .quad 0xa48859c41b7f6c32 + .quad 0x0f2d60bcf4383298 + .quad 0x1815a929c9b1d1d9 + .quad 0x47c3871bbb1755c4 + + // 2^120 * 4 * G + + .quad 0x5144539771ec4f48 + .quad 0xf805b17dc98c5d6e + .quad 0xf762c11a47c3c66b + .quad 0x00b89b85764699dc + .quad 0xfbe65d50c85066b0 + .quad 0x62ecc4b0b3a299b0 + .quad 0xe53754ea441ae8e0 + .quad 0x08fea02ce8d48d5f + .quad 0x824ddd7668deead0 + .quad 0xc86445204b685d23 + .quad 0xb514cfcd5d89d665 + .quad 0x473829a74f75d537 + + // 2^120 * 5 * G + + .quad 0x82d2da754679c418 + .quad 0xe63bd7d8b2618df0 + .quad 0x355eef24ac47eb0a + .quad 0x2078684c4833c6b4 + .quad 0x23d9533aad3902c9 + .quad 0x64c2ddceef03588f + .quad 0x15257390cfe12fb4 + .quad 0x6c668b4d44e4d390 + .quad 0x3b48cf217a78820c + .quad 0xf76a0ab281273e97 + .quad 0xa96c65a78c8eed7b + .quad 0x7411a6054f8a433f + + // 2^120 * 6 * G + + .quad 0x4d659d32b99dc86d + .quad 0x044cdc75603af115 + .quad 0xb34c712cdcc2e488 + .quad 0x7c136574fb8134ff + .quad 0x579ae53d18b175b4 + .quad 0x68713159f392a102 + .quad 0x8455ecba1eef35f5 + .quad 0x1ec9a872458c398f + .quad 0xb8e6a4d400a2509b + .quad 0x9b81d7020bc882b4 + .quad 0x57e7cc9bf1957561 + .quad 0x3add88a5c7cd6460 + + // 2^120 * 7 * G + + .quad 0xab895770b635dcf2 + .quad 0x02dfef6cf66c1fbc + .quad 0x85530268beb6d187 + .quad 0x249929fccc879e74 + .quad 0x85c298d459393046 + .quad 0x8f7e35985ff659ec + .quad 0x1d2ca22af2f66e3a + .quad 0x61ba1131a406a720 + .quad 0xa3d0a0f116959029 + .quad 0x023b6b6cba7ebd89 + .quad 0x7bf15a3e26783307 + .quad 0x5620310cbbd8ece7 + + // 2^120 * 8 * G + + .quad 0x528993434934d643 + .quad 0xb9dbf806a51222f5 + .quad 0x8f6d878fc3f41c22 + .quad 0x37676a2a4d9d9730 + .quad 0x6646b5f477e285d6 + .quad 0x40e8ff676c8f6193 + .quad 0xa6ec7311abb594dd + .quad 0x7ec846f3658cec4d + .quad 0x9b5e8f3f1da22ec7 + .quad 0x130f1d776c01cd13 + .quad 0x214c8fcfa2989fb8 + .quad 0x6daaf723399b9dd5 + + // 2^124 * 1 * G + + .quad 0x591e4a5610628564 + .quad 0x2a4bb87ca8b4df34 + .quad 0xde2a2572e7a38e43 + .quad 0x3cbdabd9fee5046e + .quad 0x81aebbdd2cd13070 + .quad 0x962e4325f85a0e9e + .quad 0xde9391aacadffecb + .quad 0x53177fda52c230e6 + .quad 0xa7bc970650b9de79 + .quad 0x3d12a7fbc301b59b + .quad 0x02652e68d36ae38c + .quad 0x79d739835a6199dc + + // 2^124 * 2 * G + + .quad 0xd9354df64131c1bd + .quad 0x758094a186ec5822 + .quad 0x4464ee12e459f3c2 + .quad 0x6c11fce4cb133282 + .quad 0x21c9d9920d591737 + .quad 0x9bea41d2e9b46cd6 + .quad 0xe20e84200d89bfca + .quad 0x79d99f946eae5ff8 + .quad 0xf17b483568673205 + .quad 0x387deae83caad96c + .quad 0x61b471fd56ffe386 + .quad 0x31741195b745a599 + + // 2^124 * 3 * G + + .quad 0xe8d10190b77a360b + .quad 0x99b983209995e702 + .quad 0xbd4fdff8fa0247aa + .quad 0x2772e344e0d36a87 + .quad 0x17f8ba683b02a047 + .quad 0x50212096feefb6c8 + .quad 0x70139be21556cbe2 + .quad 0x203e44a11d98915b + .quad 0xd6863eba37b9e39f + .quad 0x105bc169723b5a23 + .quad 0x104f6459a65c0762 + .quad 0x567951295b4d38d4 + + // 2^124 * 4 * G + + .quad 0x535fd60613037524 + .quad 0xe210adf6b0fbc26a + .quad 0xac8d0a9b23e990ae + .quad 0x47204d08d72fdbf9 + .quad 0x07242eb30d4b497f + .quad 0x1ef96306b9bccc87 + .quad 0x37950934d8116f45 + .quad 0x05468d6201405b04 + .quad 0x00f565a9f93267de + .quad 0xcecfd78dc0d58e8a + .quad 0xa215e2dcf318e28e + .quad 0x4599ee919b633352 + + // 2^124 * 5 * G + + .quad 0xd3c220ca70e0e76b + .quad 0xb12bea58ea9f3094 + .quad 0x294ddec8c3271282 + .quad 0x0c3539e1a1d1d028 + .quad 0xac746d6b861ae579 + .quad 0x31ab0650f6aea9dc + .quad 0x241d661140256d4c + .quad 0x2f485e853d21a5de + .quad 0x329744839c0833f3 + .quad 0x6fe6257fd2abc484 + .quad 0x5327d1814b358817 + .quad 0x65712585893fe9bc + + // 2^124 * 6 * G + + .quad 0x9c102fb732a61161 + .quad 0xe48e10dd34d520a8 + .quad 0x365c63546f9a9176 + .quad 0x32f6fe4c046f6006 + .quad 0x81c29f1bd708ee3f + .quad 0xddcb5a05ae6407d0 + .quad 0x97aec1d7d2a3eba7 + .quad 0x1590521a91d50831 + .quad 0x40a3a11ec7910acc + .quad 0x9013dff8f16d27ae + .quad 0x1a9720d8abb195d4 + .quad 0x1bb9fe452ea98463 + + // 2^124 * 7 * G + + .quad 0xe9d1d950b3d54f9e + .quad 0x2d5f9cbee00d33c1 + .quad 0x51c2c656a04fc6ac + .quad 0x65c091ee3c1cbcc9 + .quad 0xcf5e6c95cc36747c + .quad 0x294201536b0bc30d + .quad 0x453ac67cee797af0 + .quad 0x5eae6ab32a8bb3c9 + .quad 0x7083661114f118ea + .quad 0x2b37b87b94349cad + .quad 0x7273f51cb4e99f40 + .quad 0x78a2a95823d75698 + + // 2^124 * 8 * G + + .quad 0xa2b072e95c8c2ace + .quad 0x69cffc96651e9c4b + .quad 0x44328ef842e7b42b + .quad 0x5dd996c122aadeb3 + .quad 0xb4f23c425ef83207 + .quad 0xabf894d3c9a934b5 + .quad 0xd0708c1339fd87f7 + .quad 0x1876789117166130 + .quad 0x925b5ef0670c507c + .quad 0x819bc842b93c33bf + .quad 0x10792e9a70dd003f + .quad 0x59ad4b7a6e28dc74 + + // 2^128 * 1 * G + + .quad 0x5f3a7562eb3dbe47 + .quad 0xf7ea38548ebda0b8 + .quad 0x00c3e53145747299 + .quad 0x1304e9e71627d551 + .quad 0x583b04bfacad8ea2 + .quad 0x29b743e8148be884 + .quad 0x2b1e583b0810c5db + .quad 0x2b5449e58eb3bbaa + .quad 0x789814d26adc9cfe + .quad 0x3c1bab3f8b48dd0b + .quad 0xda0fe1fff979c60a + .quad 0x4468de2d7c2dd693 + + // 2^128 * 2 * G + + .quad 0x51bb355e9419469e + .quad 0x33e6dc4c23ddc754 + .quad 0x93a5b6d6447f9962 + .quad 0x6cce7c6ffb44bd63 + .quad 0x4b9ad8c6f86307ce + .quad 0x21113531435d0c28 + .quad 0xd4a866c5657a772c + .quad 0x5da6427e63247352 + .quad 0x1a94c688deac22ca + .quad 0xb9066ef7bbae1ff8 + .quad 0x88ad8c388d59580f + .quad 0x58f29abfe79f2ca8 + + // 2^128 * 3 * G + + .quad 0xe90ecfab8de73e68 + .quad 0x54036f9f377e76a5 + .quad 0xf0495b0bbe015982 + .quad 0x577629c4a7f41e36 + .quad 0x4b5a64bf710ecdf6 + .quad 0xb14ce538462c293c + .quad 0x3643d056d50b3ab9 + .quad 0x6af93724185b4870 + .quad 0x3220024509c6a888 + .quad 0xd2e036134b558973 + .quad 0x83e236233c33289f + .quad 0x701f25bb0caec18f + + // 2^128 * 4 * G + + .quad 0xc3a8b0f8e4616ced + .quad 0xf700660e9e25a87d + .quad 0x61e3061ff4bca59c + .quad 0x2e0c92bfbdc40be9 + .quad 0x9d18f6d97cbec113 + .quad 0x844a06e674bfdbe4 + .quad 0x20f5b522ac4e60d6 + .quad 0x720a5bc050955e51 + .quad 0x0c3f09439b805a35 + .quad 0xe84e8b376242abfc + .quad 0x691417f35c229346 + .quad 0x0e9b9cbb144ef0ec + + // 2^128 * 5 * G + + .quad 0xfbbad48ffb5720ad + .quad 0xee81916bdbf90d0e + .quad 0xd4813152635543bf + .quad 0x221104eb3f337bd8 + .quad 0x8dee9bd55db1beee + .quad 0xc9c3ab370a723fb9 + .quad 0x44a8f1bf1c68d791 + .quad 0x366d44191cfd3cde + .quad 0x9e3c1743f2bc8c14 + .quad 0x2eda26fcb5856c3b + .quad 0xccb82f0e68a7fb97 + .quad 0x4167a4e6bc593244 + + // 2^128 * 6 * G + + .quad 0x643b9d2876f62700 + .quad 0x5d1d9d400e7668eb + .quad 0x1b4b430321fc0684 + .quad 0x7938bb7e2255246a + .quad 0xc2be2665f8ce8fee + .quad 0xe967ff14e880d62c + .quad 0xf12e6e7e2f364eee + .quad 0x34b33370cb7ed2f6 + .quad 0xcdc591ee8681d6cc + .quad 0xce02109ced85a753 + .quad 0xed7485c158808883 + .quad 0x1176fc6e2dfe65e4 + + // 2^128 * 7 * G + + .quad 0xb4af6cd05b9c619b + .quad 0x2ddfc9f4b2a58480 + .quad 0x3d4fa502ebe94dc4 + .quad 0x08fc3a4c677d5f34 + .quad 0xdb90e28949770eb8 + .quad 0x98fbcc2aacf440a3 + .quad 0x21354ffeded7879b + .quad 0x1f6a3e54f26906b6 + .quad 0x60a4c199d30734ea + .quad 0x40c085b631165cd6 + .quad 0xe2333e23f7598295 + .quad 0x4f2fad0116b900d1 + + // 2^128 * 8 * G + + .quad 0x44beb24194ae4e54 + .quad 0x5f541c511857ef6c + .quad 0xa61e6b2d368d0498 + .quad 0x445484a4972ef7ab + .quad 0x962cd91db73bb638 + .quad 0xe60577aafc129c08 + .quad 0x6f619b39f3b61689 + .quad 0x3451995f2944ee81 + .quad 0x9152fcd09fea7d7c + .quad 0x4a816c94b0935cf6 + .quad 0x258e9aaa47285c40 + .quad 0x10b89ca6042893b7 + + // 2^132 * 1 * G + + .quad 0x9b2a426e3b646025 + .quad 0x32127190385ce4cf + .quad 0xa25cffc2dd6dea45 + .quad 0x06409010bea8de75 + .quad 0xd67cded679d34aa0 + .quad 0xcc0b9ec0cc4db39f + .quad 0xa535a456e35d190f + .quad 0x2e05d9eaf61f6fef + .quad 0xc447901ad61beb59 + .quad 0x661f19bce5dc880a + .quad 0x24685482b7ca6827 + .quad 0x293c778cefe07f26 + + // 2^132 * 2 * G + + .quad 0x86809e7007069096 + .quad 0xaad75b15e4e50189 + .quad 0x07f35715a21a0147 + .quad 0x0487f3f112815d5e + .quad 0x16c795d6a11ff200 + .quad 0xcb70d0e2b15815c9 + .quad 0x89f293209b5395b5 + .quad 0x50b8c2d031e47b4f + .quad 0x48350c08068a4962 + .quad 0x6ffdd05351092c9a + .quad 0x17af4f4aaf6fc8dd + .quad 0x4b0553b53cdba58b + + // 2^132 * 3 * G + + .quad 0x9c65fcbe1b32ff79 + .quad 0xeb75ea9f03b50f9b + .quad 0xfced2a6c6c07e606 + .quad 0x35106cd551717908 + .quad 0xbf05211b27c152d4 + .quad 0x5ec26849bd1af639 + .quad 0x5e0b2caa8e6fab98 + .quad 0x054c8bdd50bd0840 + .quad 0x38a0b12f1dcf073d + .quad 0x4b60a8a3b7f6a276 + .quad 0xfed5ac25d3404f9a + .quad 0x72e82d5e5505c229 + + // 2^132 * 4 * G + + .quad 0x6b0b697ff0d844c8 + .quad 0xbb12f85cd979cb49 + .quad 0xd2a541c6c1da0f1f + .quad 0x7b7c242958ce7211 + .quad 0x00d9cdfd69771d02 + .quad 0x410276cd6cfbf17e + .quad 0x4c45306c1cb12ec7 + .quad 0x2857bf1627500861 + .quad 0x9f21903f0101689e + .quad 0xd779dfd3bf861005 + .quad 0xa122ee5f3deb0f1b + .quad 0x510df84b485a00d4 + + // 2^132 * 5 * G + + .quad 0xa54133bb9277a1fa + .quad 0x74ec3b6263991237 + .quad 0x1a3c54dc35d2f15a + .quad 0x2d347144e482ba3a + .quad 0x24b3c887c70ac15e + .quad 0xb0f3a557fb81b732 + .quad 0x9b2cde2fe578cc1b + .quad 0x4cf7ed0703b54f8e + .quad 0x6bd47c6598fbee0f + .quad 0x9e4733e2ab55be2d + .quad 0x1093f624127610c5 + .quad 0x4e05e26ad0a1eaa4 + + // 2^132 * 6 * G + + .quad 0xda9b6b624b531f20 + .quad 0x429a760e77509abb + .quad 0xdbe9f522e823cb80 + .quad 0x618f1856880c8f82 + .quad 0x1833c773e18fe6c0 + .quad 0xe3c4711ad3c87265 + .quad 0x3bfd3c4f0116b283 + .quad 0x1955875eb4cd4db8 + .quad 0x6da6de8f0e399799 + .quad 0x7ad61aa440fda178 + .quad 0xb32cd8105e3563dd + .quad 0x15f6beae2ae340ae + + // 2^132 * 7 * G + + .quad 0x862bcb0c31ec3a62 + .quad 0x810e2b451138f3c2 + .quad 0x788ec4b839dac2a4 + .quad 0x28f76867ae2a9281 + .quad 0xba9a0f7b9245e215 + .quad 0xf368612dd98c0dbb + .quad 0x2e84e4cbf220b020 + .quad 0x6ba92fe962d90eda + .quad 0x3e4df9655884e2aa + .quad 0xbd62fbdbdbd465a5 + .quad 0xd7596caa0de9e524 + .quad 0x6e8042ccb2b1b3d7 + + // 2^132 * 8 * G + + .quad 0xf10d3c29ce28ca6e + .quad 0xbad34540fcb6093d + .quad 0xe7426ed7a2ea2d3f + .quad 0x08af9d4e4ff298b9 + .quad 0x1530653616521f7e + .quad 0x660d06b896203dba + .quad 0x2d3989bc545f0879 + .quad 0x4b5303af78ebd7b0 + .quad 0x72f8a6c3bebcbde8 + .quad 0x4f0fca4adc3a8e89 + .quad 0x6fa9d4e8c7bfdf7a + .quad 0x0dcf2d679b624eb7 + + // 2^136 * 1 * G + + .quad 0x3d5947499718289c + .quad 0x12ebf8c524533f26 + .quad 0x0262bfcb14c3ef15 + .quad 0x20b878d577b7518e + .quad 0x753941be5a45f06e + .quad 0xd07caeed6d9c5f65 + .quad 0x11776b9c72ff51b6 + .quad 0x17d2d1d9ef0d4da9 + .quad 0x27f2af18073f3e6a + .quad 0xfd3fe519d7521069 + .quad 0x22e3b72c3ca60022 + .quad 0x72214f63cc65c6a7 + + // 2^136 * 2 * G + + .quad 0xb4e37f405307a693 + .quad 0xaba714d72f336795 + .quad 0xd6fbd0a773761099 + .quad 0x5fdf48c58171cbc9 + .quad 0x1d9db7b9f43b29c9 + .quad 0xd605824a4f518f75 + .quad 0xf2c072bd312f9dc4 + .quad 0x1f24ac855a1545b0 + .quad 0x24d608328e9505aa + .quad 0x4748c1d10c1420ee + .quad 0xc7ffe45c06fb25a2 + .quad 0x00ba739e2ae395e6 + + // 2^136 * 3 * G + + .quad 0x592e98de5c8790d6 + .quad 0xe5bfb7d345c2a2df + .quad 0x115a3b60f9b49922 + .quad 0x03283a3e67ad78f3 + .quad 0xae4426f5ea88bb26 + .quad 0x360679d984973bfb + .quad 0x5c9f030c26694e50 + .quad 0x72297de7d518d226 + .quad 0x48241dc7be0cb939 + .quad 0x32f19b4d8b633080 + .quad 0xd3dfc90d02289308 + .quad 0x05e1296846271945 + + // 2^136 * 4 * G + + .quad 0xba82eeb32d9c495a + .quad 0xceefc8fcf12bb97c + .quad 0xb02dabae93b5d1e0 + .quad 0x39c00c9c13698d9b + .quad 0xadbfbbc8242c4550 + .quad 0xbcc80cecd03081d9 + .quad 0x843566a6f5c8df92 + .quad 0x78cf25d38258ce4c + .quad 0x15ae6b8e31489d68 + .quad 0xaa851cab9c2bf087 + .quad 0xc9a75a97f04efa05 + .quad 0x006b52076b3ff832 + + // 2^136 * 5 * G + + .quad 0x29e0cfe19d95781c + .quad 0xb681df18966310e2 + .quad 0x57df39d370516b39 + .quad 0x4d57e3443bc76122 + .quad 0xf5cb7e16b9ce082d + .quad 0x3407f14c417abc29 + .quad 0xd4b36bce2bf4a7ab + .quad 0x7de2e9561a9f75ce + .quad 0xde70d4f4b6a55ecb + .quad 0x4801527f5d85db99 + .quad 0xdbc9c440d3ee9a81 + .quad 0x6b2a90af1a6029ed + + // 2^136 * 6 * G + + .quad 0x6923f4fc9ae61e97 + .quad 0x5735281de03f5fd1 + .quad 0xa764ae43e6edd12d + .quad 0x5fd8f4e9d12d3e4a + .quad 0x77ebf3245bb2d80a + .quad 0xd8301b472fb9079b + .quad 0xc647e6f24cee7333 + .quad 0x465812c8276c2109 + .quad 0x4d43beb22a1062d9 + .quad 0x7065fb753831dc16 + .quad 0x180d4a7bde2968d7 + .quad 0x05b32c2b1cb16790 + + // 2^136 * 7 * G + + .quad 0xc8c05eccd24da8fd + .quad 0xa1cf1aac05dfef83 + .quad 0xdbbeeff27df9cd61 + .quad 0x3b5556a37b471e99 + .quad 0xf7fca42c7ad58195 + .quad 0x3214286e4333f3cc + .quad 0xb6c29d0d340b979d + .quad 0x31771a48567307e1 + .quad 0x32b0c524e14dd482 + .quad 0xedb351541a2ba4b6 + .quad 0xa3d16048282b5af3 + .quad 0x4fc079d27a7336eb + + // 2^136 * 8 * G + + .quad 0x51c938b089bf2f7f + .quad 0x2497bd6502dfe9a7 + .quad 0xffffc09c7880e453 + .quad 0x124567cecaf98e92 + .quad 0xdc348b440c86c50d + .quad 0x1337cbc9cc94e651 + .quad 0x6422f74d643e3cb9 + .quad 0x241170c2bae3cd08 + .quad 0x3ff9ab860ac473b4 + .quad 0xf0911dee0113e435 + .quad 0x4ae75060ebc6c4af + .quad 0x3f8612966c87000d + + // 2^140 * 1 * G + + .quad 0x0c9c5303f7957be4 + .quad 0xa3c31a20e085c145 + .quad 0xb0721d71d0850050 + .quad 0x0aba390eab0bf2da + .quad 0x529fdffe638c7bf3 + .quad 0xdf2b9e60388b4995 + .quad 0xe027b34f1bad0249 + .quad 0x7bc92fc9b9fa74ed + .quad 0x9f97ef2e801ad9f9 + .quad 0x83697d5479afda3a + .quad 0xe906b3ffbd596b50 + .quad 0x02672b37dd3fb8e0 + + // 2^140 * 2 * G + + .quad 0x48b2ca8b260885e4 + .quad 0xa4286bec82b34c1c + .quad 0x937e1a2617f58f74 + .quad 0x741d1fcbab2ca2a5 + .quad 0xee9ba729398ca7f5 + .quad 0xeb9ca6257a4849db + .quad 0x29eb29ce7ec544e1 + .quad 0x232ca21ef736e2c8 + .quad 0xbf61423d253fcb17 + .quad 0x08803ceafa39eb14 + .quad 0xf18602df9851c7af + .quad 0x0400f3a049e3414b + + // 2^140 * 3 * G + + .quad 0xabce0476ba61c55b + .quad 0x36a3d6d7c4d39716 + .quad 0x6eb259d5e8d82d09 + .quad 0x0c9176e984d756fb + .quad 0x2efba412a06e7b06 + .quad 0x146785452c8d2560 + .quad 0xdf9713ebd67a91c7 + .quad 0x32830ac7157eadf3 + .quad 0x0e782a7ab73769e8 + .quad 0x04a05d7875b18e2c + .quad 0x29525226ebcceae1 + .quad 0x0d794f8383eba820 + + // 2^140 * 4 * G + + .quad 0xff35f5cb9e1516f4 + .quad 0xee805bcf648aae45 + .quad 0xf0d73c2bb93a9ef3 + .quad 0x097b0bf22092a6c2 + .quad 0x7be44ce7a7a2e1ac + .quad 0x411fd93efad1b8b7 + .quad 0x1734a1d70d5f7c9b + .quad 0x0d6592233127db16 + .quad 0xc48bab1521a9d733 + .quad 0xa6c2eaead61abb25 + .quad 0x625c6c1cc6cb4305 + .quad 0x7fc90fea93eb3a67 + + // 2^140 * 5 * G + + .quad 0x0408f1fe1f5c5926 + .quad 0x1a8f2f5e3b258bf4 + .quad 0x40a951a2fdc71669 + .quad 0x6598ee93c98b577e + .quad 0xc527deb59c7cb23d + .quad 0x955391695328404e + .quad 0xd64392817ccf2c7a + .quad 0x6ce97dabf7d8fa11 + .quad 0x25b5a8e50ef7c48f + .quad 0xeb6034116f2ce532 + .quad 0xc5e75173e53de537 + .quad 0x73119fa08c12bb03 + + // 2^140 * 6 * G + + .quad 0xed30129453f1a4cb + .quad 0xbce621c9c8f53787 + .quad 0xfacb2b1338bee7b9 + .quad 0x3025798a9ea8428c + .quad 0x7845b94d21f4774d + .quad 0xbf62f16c7897b727 + .quad 0x671857c03c56522b + .quad 0x3cd6a85295621212 + .quad 0x3fecde923aeca999 + .quad 0xbdaa5b0062e8c12f + .quad 0x67b99dfc96988ade + .quad 0x3f52c02852661036 + + // 2^140 * 7 * G + + .quad 0xffeaa48e2a1351c6 + .quad 0x28624754fa7f53d7 + .quad 0x0b5ba9e57582ddf1 + .quad 0x60c0104ba696ac59 + .quad 0x9258bf99eec416c6 + .quad 0xac8a5017a9d2f671 + .quad 0x629549ab16dea4ab + .quad 0x05d0e85c99091569 + .quad 0x051de020de9cbe97 + .quad 0xfa07fc56b50bcf74 + .quad 0x378cec9f0f11df65 + .quad 0x36853c69ab96de4d + + // 2^140 * 8 * G + + .quad 0x36d9b8de78f39b2d + .quad 0x7f42ed71a847b9ec + .quad 0x241cd1d679bd3fde + .quad 0x6a704fec92fbce6b + .quad 0x4433c0b0fac5e7be + .quad 0x724bae854c08dcbe + .quad 0xf1f24cc446978f9b + .quad 0x4a0aff6d62825fc8 + .quad 0xe917fb9e61095301 + .quad 0xc102df9402a092f8 + .quad 0xbf09e2f5fa66190b + .quad 0x681109bee0dcfe37 + + // 2^144 * 1 * G + + .quad 0x559a0cc9782a0dde + .quad 0x551dcdb2ea718385 + .quad 0x7f62865b31ef238c + .quad 0x504aa7767973613d + .quad 0x9c18fcfa36048d13 + .quad 0x29159db373899ddd + .quad 0xdc9f350b9f92d0aa + .quad 0x26f57eee878a19d4 + .quad 0x0cab2cd55687efb1 + .quad 0x5180d162247af17b + .quad 0x85c15a344f5a2467 + .quad 0x4041943d9dba3069 + + // 2^144 * 2 * G + + .quad 0xc3c0eeba43ebcc96 + .quad 0x8d749c9c26ea9caf + .quad 0xd9fa95ee1c77ccc6 + .quad 0x1420a1d97684340f + .quad 0x4b217743a26caadd + .quad 0x47a6b424648ab7ce + .quad 0xcb1d4f7a03fbc9e3 + .quad 0x12d931429800d019 + .quad 0x00c67799d337594f + .quad 0x5e3c5140b23aa47b + .quad 0x44182854e35ff395 + .quad 0x1b4f92314359a012 + + // 2^144 * 3 * G + + .quad 0x3e5c109d89150951 + .quad 0x39cefa912de9696a + .quad 0x20eae43f975f3020 + .quad 0x239b572a7f132dae + .quad 0x33cf3030a49866b1 + .quad 0x251f73d2215f4859 + .quad 0xab82aa4051def4f6 + .quad 0x5ff191d56f9a23f6 + .quad 0x819ed433ac2d9068 + .quad 0x2883ab795fc98523 + .quad 0xef4572805593eb3d + .quad 0x020c526a758f36cb + + // 2^144 * 4 * G + + .quad 0x779834f89ed8dbbc + .quad 0xc8f2aaf9dc7ca46c + .quad 0xa9524cdca3e1b074 + .quad 0x02aacc4615313877 + .quad 0xe931ef59f042cc89 + .quad 0x2c589c9d8e124bb6 + .quad 0xadc8e18aaec75997 + .quad 0x452cfe0a5602c50c + .quad 0x86a0f7a0647877df + .quad 0xbbc464270e607c9f + .quad 0xab17ea25f1fb11c9 + .quad 0x4cfb7d7b304b877b + + // 2^144 * 5 * G + + .quad 0x72b43d6cb89b75fe + .quad 0x54c694d99c6adc80 + .quad 0xb8c3aa373ee34c9f + .quad 0x14b4622b39075364 + .quad 0xe28699c29789ef12 + .quad 0x2b6ecd71df57190d + .quad 0xc343c857ecc970d0 + .quad 0x5b1d4cbc434d3ac5 + .quad 0xb6fb2615cc0a9f26 + .quad 0x3a4f0e2bb88dcce5 + .quad 0x1301498b3369a705 + .quad 0x2f98f71258592dd1 + + // 2^144 * 6 * G + + .quad 0x0c94a74cb50f9e56 + .quad 0x5b1ff4a98e8e1320 + .quad 0x9a2acc2182300f67 + .quad 0x3a6ae249d806aaf9 + .quad 0x2e12ae444f54a701 + .quad 0xfcfe3ef0a9cbd7de + .quad 0xcebf890d75835de0 + .quad 0x1d8062e9e7614554 + .quad 0x657ada85a9907c5a + .quad 0x1a0ea8b591b90f62 + .quad 0x8d0e1dfbdf34b4e9 + .quad 0x298b8ce8aef25ff3 + + // 2^144 * 7 * G + + .quad 0x2a927953eff70cb2 + .quad 0x4b89c92a79157076 + .quad 0x9418457a30a7cf6a + .quad 0x34b8a8404d5ce485 + .quad 0x837a72ea0a2165de + .quad 0x3fab07b40bcf79f6 + .quad 0x521636c77738ae70 + .quad 0x6ba6271803a7d7dc + .quad 0xc26eecb583693335 + .quad 0xd5a813df63b5fefd + .quad 0xa293aa9aa4b22573 + .quad 0x71d62bdd465e1c6a + + // 2^144 * 8 * G + + .quad 0x6533cc28d378df80 + .quad 0xf6db43790a0fa4b4 + .quad 0xe3645ff9f701da5a + .quad 0x74d5f317f3172ba4 + .quad 0xcd2db5dab1f75ef5 + .quad 0xd77f95cf16b065f5 + .quad 0x14571fea3f49f085 + .quad 0x1c333621262b2b3d + .quad 0xa86fe55467d9ca81 + .quad 0x398b7c752b298c37 + .quad 0xda6d0892e3ac623b + .quad 0x4aebcc4547e9d98c + + // 2^148 * 1 * G + + .quad 0x53175a7205d21a77 + .quad 0xb0c04422d3b934d4 + .quad 0xadd9f24bdd5deadc + .quad 0x074f46e69f10ff8c + .quad 0x0de9b204a059a445 + .quad 0xe15cb4aa4b17ad0f + .quad 0xe1bbec521f79c557 + .quad 0x2633f1b9d071081b + .quad 0xc1fb4177018b9910 + .quad 0xa6ea20dc6c0fe140 + .quad 0xd661f3e74354c6ff + .quad 0x5ecb72e6f1a3407a + + // 2^148 * 2 * G + + .quad 0xa515a31b2259fb4e + .quad 0x0960f3972bcac52f + .quad 0xedb52fec8d3454cb + .quad 0x382e2720c476c019 + .quad 0xfeeae106e8e86997 + .quad 0x9863337f98d09383 + .quad 0x9470480eaa06ebef + .quad 0x038b6898d4c5c2d0 + .quad 0xf391c51d8ace50a6 + .quad 0x3142d0b9ae2d2948 + .quad 0xdb4d5a1a7f24ca80 + .quad 0x21aeba8b59250ea8 + + // 2^148 * 3 * G + + .quad 0x24f13b34cf405530 + .quad 0x3c44ea4a43088af7 + .quad 0x5dd5c5170006a482 + .quad 0x118eb8f8890b086d + .quad 0x53853600f0087f23 + .quad 0x4c461879da7d5784 + .quad 0x6af303deb41f6860 + .quad 0x0a3c16c5c27c18ed + .quad 0x17e49c17cc947f3d + .quad 0xccc6eda6aac1d27b + .quad 0xdf6092ceb0f08e56 + .quad 0x4909b3e22c67c36b + + // 2^148 * 4 * G + + .quad 0x9c9c85ea63fe2e89 + .quad 0xbe1baf910e9412ec + .quad 0x8f7baa8a86fbfe7b + .quad 0x0fb17f9fef968b6c + .quad 0x59a16676706ff64e + .quad 0x10b953dd0d86a53d + .quad 0x5848e1e6ce5c0b96 + .quad 0x2d8b78e712780c68 + .quad 0x79d5c62eafc3902b + .quad 0x773a215289e80728 + .quad 0xc38ae640e10120b9 + .quad 0x09ae23717b2b1a6d + + // 2^148 * 5 * G + + .quad 0xbb6a192a4e4d083c + .quad 0x34ace0630029e192 + .quad 0x98245a59aafabaeb + .quad 0x6d9c8a9ada97faac + .quad 0x10ab8fa1ad32b1d0 + .quad 0xe9aced1be2778b24 + .quad 0xa8856bc0373de90f + .quad 0x66f35ddddda53996 + .quad 0xd27d9afb24997323 + .quad 0x1bb7e07ef6f01d2e + .quad 0x2ba7472df52ecc7f + .quad 0x03019b4f646f9dc8 + + // 2^148 * 6 * G + + .quad 0x04a186b5565345cd + .quad 0xeee76610bcc4116a + .quad 0x689c73b478fb2a45 + .quad 0x387dcbff65697512 + .quad 0xaf09b214e6b3dc6b + .quad 0x3f7573b5ad7d2f65 + .quad 0xd019d988100a23b0 + .quad 0x392b63a58b5c35f7 + .quad 0x4093addc9c07c205 + .quad 0xc565be15f532c37e + .quad 0x63dbecfd1583402a + .quad 0x61722b4aef2e032e + + // 2^148 * 7 * G + + .quad 0x0012aafeecbd47af + .quad 0x55a266fb1cd46309 + .quad 0xf203eb680967c72c + .quad 0x39633944ca3c1429 + .quad 0xd6b07a5581cb0e3c + .quad 0x290ff006d9444969 + .quad 0x08680b6a16dcda1f + .quad 0x5568d2b75a06de59 + .quad 0x8d0cb88c1b37cfe1 + .quad 0x05b6a5a3053818f3 + .quad 0xf2e9bc04b787d959 + .quad 0x6beba1249add7f64 + + // 2^148 * 8 * G + + .quad 0x1d06005ca5b1b143 + .quad 0x6d4c6bb87fd1cda2 + .quad 0x6ef5967653fcffe7 + .quad 0x097c29e8c1ce1ea5 + .quad 0x5c3cecb943f5a53b + .quad 0x9cc9a61d06c08df2 + .quad 0xcfba639a85895447 + .quad 0x5a845ae80df09fd5 + .quad 0x4ce97dbe5deb94ca + .quad 0x38d0a4388c709c48 + .quad 0xc43eced4a169d097 + .quad 0x0a1249fff7e587c3 + + // 2^152 * 1 * G + + .quad 0x12f0071b276d01c9 + .quad 0xe7b8bac586c48c70 + .quad 0x5308129b71d6fba9 + .quad 0x5d88fbf95a3db792 + .quad 0x0b408d9e7354b610 + .quad 0x806b32535ba85b6e + .quad 0xdbe63a034a58a207 + .quad 0x173bd9ddc9a1df2c + .quad 0x2b500f1efe5872df + .quad 0x58d6582ed43918c1 + .quad 0xe6ed278ec9673ae0 + .quad 0x06e1cd13b19ea319 + + // 2^152 * 2 * G + + .quad 0x40d0ad516f166f23 + .quad 0x118e32931fab6abe + .quad 0x3fe35e14a04d088e + .quad 0x3080603526e16266 + .quad 0x472baf629e5b0353 + .quad 0x3baa0b90278d0447 + .quad 0x0c785f469643bf27 + .quad 0x7f3a6a1a8d837b13 + .quad 0xf7e644395d3d800b + .quad 0x95a8d555c901edf6 + .quad 0x68cd7830592c6339 + .quad 0x30d0fded2e51307e + + // 2^152 * 3 * G + + .quad 0xe0594d1af21233b3 + .quad 0x1bdbe78ef0cc4d9c + .quad 0x6965187f8f499a77 + .quad 0x0a9214202c099868 + .quad 0x9cb4971e68b84750 + .quad 0xa09572296664bbcf + .quad 0x5c8de72672fa412b + .quad 0x4615084351c589d9 + .quad 0xbc9019c0aeb9a02e + .quad 0x55c7110d16034cae + .quad 0x0e6df501659932ec + .quad 0x3bca0d2895ca5dfe + + // 2^152 * 4 * G + + .quad 0x40f031bc3c5d62a4 + .quad 0x19fc8b3ecff07a60 + .quad 0x98183da2130fb545 + .quad 0x5631deddae8f13cd + .quad 0x9c688eb69ecc01bf + .quad 0xf0bc83ada644896f + .quad 0xca2d955f5f7a9fe2 + .quad 0x4ea8b4038df28241 + .quad 0x2aed460af1cad202 + .quad 0x46305305a48cee83 + .quad 0x9121774549f11a5f + .quad 0x24ce0930542ca463 + + // 2^152 * 5 * G + + .quad 0x1fe890f5fd06c106 + .quad 0xb5c468355d8810f2 + .quad 0x827808fe6e8caf3e + .quad 0x41d4e3c28a06d74b + .quad 0x3fcfa155fdf30b85 + .quad 0xd2f7168e36372ea4 + .quad 0xb2e064de6492f844 + .quad 0x549928a7324f4280 + .quad 0xf26e32a763ee1a2e + .quad 0xae91e4b7d25ffdea + .quad 0xbc3bd33bd17f4d69 + .quad 0x491b66dec0dcff6a + + // 2^152 * 6 * G + + .quad 0x98f5b13dc7ea32a7 + .quad 0xe3d5f8cc7e16db98 + .quad 0xac0abf52cbf8d947 + .quad 0x08f338d0c85ee4ac + .quad 0x75f04a8ed0da64a1 + .quad 0xed222caf67e2284b + .quad 0x8234a3791f7b7ba4 + .quad 0x4cf6b8b0b7018b67 + .quad 0xc383a821991a73bd + .quad 0xab27bc01df320c7a + .quad 0xc13d331b84777063 + .quad 0x530d4a82eb078a99 + + // 2^152 * 7 * G + + .quad 0x004c3630e1f94825 + .quad 0x7e2d78268cab535a + .quad 0xc7482323cc84ff8b + .quad 0x65ea753f101770b9 + .quad 0x6d6973456c9abf9e + .quad 0x257fb2fc4900a880 + .quad 0x2bacf412c8cfb850 + .quad 0x0db3e7e00cbfbd5b + .quad 0x3d66fc3ee2096363 + .quad 0x81d62c7f61b5cb6b + .quad 0x0fbe044213443b1a + .quad 0x02a4ec1921e1a1db + + // 2^152 * 8 * G + + .quad 0x5ce6259a3b24b8a2 + .quad 0xb8577acc45afa0b8 + .quad 0xcccbe6e88ba07037 + .quad 0x3d143c51127809bf + .quad 0xf5c86162f1cf795f + .quad 0x118c861926ee57f2 + .quad 0x172124851c063578 + .quad 0x36d12b5dec067fcf + .quad 0x126d279179154557 + .quad 0xd5e48f5cfc783a0a + .quad 0x36bdb6e8df179bac + .quad 0x2ef517885ba82859 + + // 2^156 * 1 * G + + .quad 0x88bd438cd11e0d4a + .quad 0x30cb610d43ccf308 + .quad 0xe09a0e3791937bcc + .quad 0x4559135b25b1720c + .quad 0x1ea436837c6da1e9 + .quad 0xf9c189af1fb9bdbe + .quad 0x303001fcce5dd155 + .quad 0x28a7c99ebc57be52 + .quad 0xb8fd9399e8d19e9d + .quad 0x908191cb962423ff + .quad 0xb2b948d747c742a3 + .quad 0x37f33226d7fb44c4 + + // 2^156 * 2 * G + + .quad 0x0dae8767b55f6e08 + .quad 0x4a43b3b35b203a02 + .quad 0xe3725a6e80af8c79 + .quad 0x0f7a7fd1705fa7a3 + .quad 0x33912553c821b11d + .quad 0x66ed42c241e301df + .quad 0x066fcc11104222fd + .quad 0x307a3b41c192168f + .quad 0x8eeb5d076eb55ce0 + .quad 0x2fc536bfaa0d925a + .quad 0xbe81830fdcb6c6e8 + .quad 0x556c7045827baf52 + + // 2^156 * 3 * G + + .quad 0x8e2b517302e9d8b7 + .quad 0xe3e52269248714e8 + .quad 0xbd4fbd774ca960b5 + .quad 0x6f4b4199c5ecada9 + .quad 0xb94b90022bf44406 + .quad 0xabd4237eff90b534 + .quad 0x7600a960faf86d3a + .quad 0x2f45abdac2322ee3 + .quad 0x61af4912c8ef8a6a + .quad 0xe58fa4fe43fb6e5e + .quad 0xb5afcc5d6fd427cf + .quad 0x6a5393281e1e11eb + + // 2^156 * 4 * G + + .quad 0xf3da5139a5d1ee89 + .quad 0x8145457cff936988 + .quad 0x3f622fed00e188c4 + .quad 0x0f513815db8b5a3d + .quad 0x0fff04fe149443cf + .quad 0x53cac6d9865cddd7 + .quad 0x31385b03531ed1b7 + .quad 0x5846a27cacd1039d + .quad 0x4ff5cdac1eb08717 + .quad 0x67e8b29590f2e9bc + .quad 0x44093b5e237afa99 + .quad 0x0d414bed8708b8b2 + + // 2^156 * 5 * G + + .quad 0xcfb68265fd0e75f6 + .quad 0xe45b3e28bb90e707 + .quad 0x7242a8de9ff92c7a + .quad 0x685b3201933202dd + .quad 0x81886a92294ac9e8 + .quad 0x23162b45d55547be + .quad 0x94cfbc4403715983 + .quad 0x50eb8fdb134bc401 + .quad 0xc0b73ec6d6b330cd + .quad 0x84e44807132faff1 + .quad 0x732b7352c4a5dee1 + .quad 0x5d7c7cf1aa7cd2d2 + + // 2^156 * 6 * G + + .quad 0xaf3b46bf7a4aafa2 + .quad 0xb78705ec4d40d411 + .quad 0x114f0c6aca7c15e3 + .quad 0x3f364faaa9489d4d + .quad 0x33d1013e9b73a562 + .quad 0x925cef5748ec26e1 + .quad 0xa7fce614dd468058 + .quad 0x78b0fad41e9aa438 + .quad 0xbf56a431ed05b488 + .quad 0xa533e66c9c495c7e + .quad 0xe8652baf87f3651a + .quad 0x0241800059d66c33 + + // 2^156 * 7 * G + + .quad 0xceb077fea37a5be4 + .quad 0xdb642f02e5a5eeb7 + .quad 0xc2e6d0c5471270b8 + .quad 0x4771b65538e4529c + .quad 0x28350c7dcf38ea01 + .quad 0x7c6cdbc0b2917ab6 + .quad 0xace7cfbe857082f7 + .quad 0x4d2845aba2d9a1e0 + .quad 0xbb537fe0447070de + .quad 0xcba744436dd557df + .quad 0xd3b5a3473600dbcb + .quad 0x4aeabbe6f9ffd7f8 + + // 2^156 * 8 * G + + .quad 0x4630119e40d8f78c + .quad 0xa01a9bc53c710e11 + .quad 0x486d2b258910dd79 + .quad 0x1e6c47b3db0324e5 + .quad 0x6a2134bcc4a9c8f2 + .quad 0xfbf8fd1c8ace2e37 + .quad 0x000ae3049911a0ba + .quad 0x046e3a616bc89b9e + .quad 0x14e65442f03906be + .quad 0x4a019d54e362be2a + .quad 0x68ccdfec8dc230c7 + .quad 0x7cfb7e3faf6b861c + + // 2^160 * 1 * G + + .quad 0x4637974e8c58aedc + .quad 0xb9ef22fbabf041a4 + .quad 0xe185d956e980718a + .quad 0x2f1b78fab143a8a6 + .quad 0x96eebffb305b2f51 + .quad 0xd3f938ad889596b8 + .quad 0xf0f52dc746d5dd25 + .quad 0x57968290bb3a0095 + .quad 0xf71ab8430a20e101 + .quad 0xf393658d24f0ec47 + .quad 0xcf7509a86ee2eed1 + .quad 0x7dc43e35dc2aa3e1 + + // 2^160 * 2 * G + + .quad 0x85966665887dd9c3 + .quad 0xc90f9b314bb05355 + .quad 0xc6e08df8ef2079b1 + .quad 0x7ef72016758cc12f + .quad 0x5a782a5c273e9718 + .quad 0x3576c6995e4efd94 + .quad 0x0f2ed8051f237d3e + .quad 0x044fb81d82d50a99 + .quad 0xc1df18c5a907e3d9 + .quad 0x57b3371dce4c6359 + .quad 0xca704534b201bb49 + .quad 0x7f79823f9c30dd2e + + // 2^160 * 3 * G + + .quad 0x8334d239a3b513e8 + .quad 0xc13670d4b91fa8d8 + .quad 0x12b54136f590bd33 + .quad 0x0a4e0373d784d9b4 + .quad 0x6a9c1ff068f587ba + .quad 0x0827894e0050c8de + .quad 0x3cbf99557ded5be7 + .quad 0x64a9b0431c06d6f0 + .quad 0x2eb3d6a15b7d2919 + .quad 0xb0b4f6a0d53a8235 + .quad 0x7156ce4389a45d47 + .quad 0x071a7d0ace18346c + + // 2^160 * 4 * G + + .quad 0xd3072daac887ba0b + .quad 0x01262905bfa562ee + .quad 0xcf543002c0ef768b + .quad 0x2c3bcc7146ea7e9c + .quad 0xcc0c355220e14431 + .quad 0x0d65950709b15141 + .quad 0x9af5621b209d5f36 + .quad 0x7c69bcf7617755d3 + .quad 0x07f0d7eb04e8295f + .quad 0x10db18252f50f37d + .quad 0xe951a9a3171798d7 + .quad 0x6f5a9a7322aca51d + + // 2^160 * 5 * G + + .quad 0x8ba1000c2f41c6c5 + .quad 0xc49f79c10cfefb9b + .quad 0x4efa47703cc51c9f + .quad 0x494e21a2e147afca + .quad 0xe729d4eba3d944be + .quad 0x8d9e09408078af9e + .quad 0x4525567a47869c03 + .quad 0x02ab9680ee8d3b24 + .quad 0xefa48a85dde50d9a + .quad 0x219a224e0fb9a249 + .quad 0xfa091f1dd91ef6d9 + .quad 0x6b5d76cbea46bb34 + + // 2^160 * 6 * G + + .quad 0x8857556cec0cd994 + .quad 0x6472dc6f5cd01dba + .quad 0xaf0169148f42b477 + .quad 0x0ae333f685277354 + .quad 0xe0f941171e782522 + .quad 0xf1e6ae74036936d3 + .quad 0x408b3ea2d0fcc746 + .quad 0x16fb869c03dd313e + .quad 0x288e199733b60962 + .quad 0x24fc72b4d8abe133 + .quad 0x4811f7ed0991d03e + .quad 0x3f81e38b8f70d075 + + // 2^160 * 7 * G + + .quad 0x7f910fcc7ed9affe + .quad 0x545cb8a12465874b + .quad 0xa8397ed24b0c4704 + .quad 0x50510fc104f50993 + .quad 0x0adb7f355f17c824 + .quad 0x74b923c3d74299a4 + .quad 0xd57c3e8bcbf8eaf7 + .quad 0x0ad3e2d34cdedc3d + .quad 0x6f0c0fc5336e249d + .quad 0x745ede19c331cfd9 + .quad 0xf2d6fd0009eefe1c + .quad 0x127c158bf0fa1ebe + + // 2^160 * 8 * G + + .quad 0xf6197c422e9879a2 + .quad 0xa44addd452ca3647 + .quad 0x9b413fc14b4eaccb + .quad 0x354ef87d07ef4f68 + .quad 0xdea28fc4ae51b974 + .quad 0x1d9973d3744dfe96 + .quad 0x6240680b873848a8 + .quad 0x4ed82479d167df95 + .quad 0xfee3b52260c5d975 + .quad 0x50352efceb41b0b8 + .quad 0x8808ac30a9f6653c + .quad 0x302d92d20539236d + + // 2^164 * 1 * G + + .quad 0x4c59023fcb3efb7c + .quad 0x6c2fcb99c63c2a94 + .quad 0xba4190e2c3c7e084 + .quad 0x0e545daea51874d9 + .quad 0x957b8b8b0df53c30 + .quad 0x2a1c770a8e60f098 + .quad 0xbbc7a670345796de + .quad 0x22a48f9a90c99bc9 + .quad 0x6b7dc0dc8d3fac58 + .quad 0x5497cd6ce6e42bfd + .quad 0x542f7d1bf400d305 + .quad 0x4159f47f048d9136 + + // 2^164 * 2 * G + + .quad 0x20ad660839e31e32 + .quad 0xf81e1bd58405be50 + .quad 0xf8064056f4dabc69 + .quad 0x14d23dd4ce71b975 + .quad 0x748515a8bbd24839 + .quad 0x77128347afb02b55 + .quad 0x50ba2ac649a2a17f + .quad 0x060525513ad730f1 + .quad 0xf2398e098aa27f82 + .quad 0x6d7982bb89a1b024 + .quad 0xfa694084214dd24c + .quad 0x71ab966fa32301c3 + + // 2^164 * 3 * G + + .quad 0x2dcbd8e34ded02fc + .quad 0x1151f3ec596f22aa + .quad 0xbca255434e0328da + .quad 0x35768fbe92411b22 + .quad 0xb1088a0702809955 + .quad 0x43b273ea0b43c391 + .quad 0xca9b67aefe0686ed + .quad 0x605eecbf8335f4ed + .quad 0x83200a656c340431 + .quad 0x9fcd71678ee59c2f + .quad 0x75d4613f71300f8a + .quad 0x7a912faf60f542f9 + + // 2^164 * 4 * G + + .quad 0xb204585e5edc1a43 + .quad 0x9f0e16ee5897c73c + .quad 0x5b82c0ae4e70483c + .quad 0x624a170e2bddf9be + .quad 0x253f4f8dfa2d5597 + .quad 0x25e49c405477130c + .quad 0x00c052e5996b1102 + .quad 0x33cb966e33bb6c4a + .quad 0x597028047f116909 + .quad 0x828ac41c1e564467 + .quad 0x70417dbde6217387 + .quad 0x721627aefbac4384 + + // 2^164 * 5 * G + + .quad 0x97d03bc38736add5 + .quad 0x2f1422afc532b130 + .quad 0x3aa68a057101bbc4 + .quad 0x4c946cf7e74f9fa7 + .quad 0xfd3097bc410b2f22 + .quad 0xf1a05da7b5cfa844 + .quad 0x61289a1def57ca74 + .quad 0x245ea199bb821902 + .quad 0xaedca66978d477f8 + .quad 0x1898ba3c29117fe1 + .quad 0xcf73f983720cbd58 + .quad 0x67da12e6b8b56351 + + // 2^164 * 6 * G + + .quad 0x7067e187b4bd6e07 + .quad 0x6e8f0203c7d1fe74 + .quad 0x93c6aa2f38c85a30 + .quad 0x76297d1f3d75a78a + .quad 0x2b7ef3d38ec8308c + .quad 0x828fd7ec71eb94ab + .quad 0x807c3b36c5062abd + .quad 0x0cb64cb831a94141 + .quad 0x3030fc33534c6378 + .quad 0xb9635c5ce541e861 + .quad 0x15d9a9bed9b2c728 + .quad 0x49233ea3f3775dcb + + // 2^164 * 7 * G + + .quad 0x629398fa8dbffc3a + .quad 0xe12fe52dd54db455 + .quad 0xf3be11dfdaf25295 + .quad 0x628b140dce5e7b51 + .quad 0x7b3985fe1c9f249b + .quad 0x4fd6b2d5a1233293 + .quad 0xceb345941adf4d62 + .quad 0x6987ff6f542de50c + .quad 0x47e241428f83753c + .quad 0x6317bebc866af997 + .quad 0xdabb5b433d1a9829 + .quad 0x074d8d245287fb2d + + // 2^164 * 8 * G + + .quad 0x8337d9cd440bfc31 + .quad 0x729d2ca1af318fd7 + .quad 0xa040a4a4772c2070 + .quad 0x46002ef03a7349be + .quad 0x481875c6c0e31488 + .quad 0x219429b2e22034b4 + .quad 0x7223c98a31283b65 + .quad 0x3420d60b342277f9 + .quad 0xfaa23adeaffe65f7 + .quad 0x78261ed45be0764c + .quad 0x441c0a1e2f164403 + .quad 0x5aea8e567a87d395 + + // 2^168 * 1 * G + + .quad 0x7813c1a2bca4283d + .quad 0xed62f091a1863dd9 + .quad 0xaec7bcb8c268fa86 + .quad 0x10e5d3b76f1cae4c + .quad 0x2dbc6fb6e4e0f177 + .quad 0x04e1bf29a4bd6a93 + .quad 0x5e1966d4787af6e8 + .quad 0x0edc5f5eb426d060 + .quad 0x5453bfd653da8e67 + .quad 0xe9dc1eec24a9f641 + .quad 0xbf87263b03578a23 + .quad 0x45b46c51361cba72 + + // 2^168 * 2 * G + + .quad 0xa9402abf314f7fa1 + .quad 0xe257f1dc8e8cf450 + .quad 0x1dbbd54b23a8be84 + .quad 0x2177bfa36dcb713b + .quad 0xce9d4ddd8a7fe3e4 + .quad 0xab13645676620e30 + .quad 0x4b594f7bb30e9958 + .quad 0x5c1c0aef321229df + .quad 0x37081bbcfa79db8f + .quad 0x6048811ec25f59b3 + .quad 0x087a76659c832487 + .quad 0x4ae619387d8ab5bb + + // 2^168 * 3 * G + + .quad 0x8ddbf6aa5344a32e + .quad 0x7d88eab4b41b4078 + .quad 0x5eb0eb974a130d60 + .quad 0x1a00d91b17bf3e03 + .quad 0x61117e44985bfb83 + .quad 0xfce0462a71963136 + .quad 0x83ac3448d425904b + .quad 0x75685abe5ba43d64 + .quad 0x6e960933eb61f2b2 + .quad 0x543d0fa8c9ff4952 + .quad 0xdf7275107af66569 + .quad 0x135529b623b0e6aa + + // 2^168 * 4 * G + + .quad 0x18f0dbd7add1d518 + .quad 0x979f7888cfc11f11 + .quad 0x8732e1f07114759b + .quad 0x79b5b81a65ca3a01 + .quad 0xf5c716bce22e83fe + .quad 0xb42beb19e80985c1 + .quad 0xec9da63714254aae + .quad 0x5972ea051590a613 + .quad 0x0fd4ac20dc8f7811 + .quad 0x9a9ad294ac4d4fa8 + .quad 0xc01b2d64b3360434 + .quad 0x4f7e9c95905f3bdb + + // 2^168 * 5 * G + + .quad 0x62674bbc5781302e + .quad 0xd8520f3989addc0f + .quad 0x8c2999ae53fbd9c6 + .quad 0x31993ad92e638e4c + .quad 0x71c8443d355299fe + .quad 0x8bcd3b1cdbebead7 + .quad 0x8092499ef1a49466 + .quad 0x1942eec4a144adc8 + .quad 0x7dac5319ae234992 + .quad 0x2c1b3d910cea3e92 + .quad 0x553ce494253c1122 + .quad 0x2a0a65314ef9ca75 + + // 2^168 * 6 * G + + .quad 0x2db7937ff7f927c2 + .quad 0xdb741f0617d0a635 + .quad 0x5982f3a21155af76 + .quad 0x4cf6e218647c2ded + .quad 0xcf361acd3c1c793a + .quad 0x2f9ebcac5a35bc3b + .quad 0x60e860e9a8cda6ab + .quad 0x055dc39b6dea1a13 + .quad 0xb119227cc28d5bb6 + .quad 0x07e24ebc774dffab + .quad 0xa83c78cee4a32c89 + .quad 0x121a307710aa24b6 + + // 2^168 * 7 * G + + .quad 0xe4db5d5e9f034a97 + .quad 0xe153fc093034bc2d + .quad 0x460546919551d3b1 + .quad 0x333fc76c7a40e52d + .quad 0xd659713ec77483c9 + .quad 0x88bfe077b82b96af + .quad 0x289e28231097bcd3 + .quad 0x527bb94a6ced3a9b + .quad 0x563d992a995b482e + .quad 0x3405d07c6e383801 + .quad 0x485035de2f64d8e5 + .quad 0x6b89069b20a7a9f7 + + // 2^168 * 8 * G + + .quad 0x812aa0416270220d + .quad 0x995a89faf9245b4e + .quad 0xffadc4ce5072ef05 + .quad 0x23bc2103aa73eb73 + .quad 0x4082fa8cb5c7db77 + .quad 0x068686f8c734c155 + .quad 0x29e6c8d9f6e7a57e + .quad 0x0473d308a7639bcf + .quad 0xcaee792603589e05 + .quad 0x2b4b421246dcc492 + .quad 0x02a1ef74e601a94f + .quad 0x102f73bfde04341a + + // 2^172 * 1 * G + + .quad 0xb5a2d50c7ec20d3e + .quad 0xc64bdd6ea0c97263 + .quad 0x56e89052c1ff734d + .quad 0x4929c6f72b2ffaba + .quad 0x358ecba293a36247 + .quad 0xaf8f9862b268fd65 + .quad 0x412f7e9968a01c89 + .quad 0x5786f312cd754524 + .quad 0x337788ffca14032c + .quad 0xf3921028447f1ee3 + .quad 0x8b14071f231bccad + .quad 0x4c817b4bf2344783 + + // 2^172 * 2 * G + + .quad 0x0ff853852871b96e + .quad 0xe13e9fab60c3f1bb + .quad 0xeefd595325344402 + .quad 0x0a37c37075b7744b + .quad 0x413ba057a40b4484 + .quad 0xba4c2e1a4f5f6a43 + .quad 0x614ba0a5aee1d61c + .quad 0x78a1531a8b05dc53 + .quad 0x6cbdf1703ad0562b + .quad 0x8ecf4830c92521a3 + .quad 0xdaebd303fd8424e7 + .quad 0x72ad82a42e5ec56f + + // 2^172 * 3 * G + + .quad 0x3f9e8e35bafb65f6 + .quad 0x39d69ec8f27293a1 + .quad 0x6cb8cd958cf6a3d0 + .quad 0x1734778173adae6d + .quad 0xc368939167024bc3 + .quad 0x8e69d16d49502fda + .quad 0xfcf2ec3ce45f4b29 + .quad 0x065f669ea3b4cbc4 + .quad 0x8a00aec75532db4d + .quad 0xb869a4e443e31bb1 + .quad 0x4a0f8552d3a7f515 + .quad 0x19adeb7c303d7c08 + + // 2^172 * 4 * G + + .quad 0xc720cb6153ead9a3 + .quad 0x55b2c97f512b636e + .quad 0xb1e35b5fd40290b1 + .quad 0x2fd9ccf13b530ee2 + .quad 0x9d05ba7d43c31794 + .quad 0x2470c8ff93322526 + .quad 0x8323dec816197438 + .quad 0x2852709881569b53 + .quad 0x07bd475b47f796b8 + .quad 0xd2c7b013542c8f54 + .quad 0x2dbd23f43b24f87e + .quad 0x6551afd77b0901d6 + + // 2^172 * 5 * G + + .quad 0x4546baaf54aac27f + .quad 0xf6f66fecb2a45a28 + .quad 0x582d1b5b562bcfe8 + .quad 0x44b123f3920f785f + .quad 0x68a24ce3a1d5c9ac + .quad 0xbb77a33d10ff6461 + .quad 0x0f86ce4425d3166e + .quad 0x56507c0950b9623b + .quad 0x1206f0b7d1713e63 + .quad 0x353fe3d915bafc74 + .quad 0x194ceb970ad9d94d + .quad 0x62fadd7cf9d03ad3 + + // 2^172 * 6 * G + + .quad 0xc6b5967b5598a074 + .quad 0x5efe91ce8e493e25 + .quad 0xd4b72c4549280888 + .quad 0x20ef1149a26740c2 + .quad 0x3cd7bc61e7ce4594 + .quad 0xcd6b35a9b7dd267e + .quad 0xa080abc84366ef27 + .quad 0x6ec7c46f59c79711 + .quad 0x2f07ad636f09a8a2 + .quad 0x8697e6ce24205e7d + .quad 0xc0aefc05ee35a139 + .quad 0x15e80958b5f9d897 + + // 2^172 * 7 * G + + .quad 0x25a5ef7d0c3e235b + .quad 0x6c39c17fbe134ee7 + .quad 0xc774e1342dc5c327 + .quad 0x021354b892021f39 + .quad 0x4dd1ed355bb061c4 + .quad 0x42dc0cef941c0700 + .quad 0x61305dc1fd86340e + .quad 0x56b2cc930e55a443 + .quad 0x1df79da6a6bfc5a2 + .quad 0x02f3a2749fde4369 + .quad 0xb323d9f2cda390a7 + .quad 0x7be0847b8774d363 + + // 2^172 * 8 * G + + .quad 0x8c99cc5a8b3f55c3 + .quad 0x0611d7253fded2a0 + .quad 0xed2995ff36b70a36 + .quad 0x1f699a54d78a2619 + .quad 0x1466f5af5307fa11 + .quad 0x817fcc7ded6c0af2 + .quad 0x0a6de44ec3a4a3fb + .quad 0x74071475bc927d0b + .quad 0xe77292f373e7ea8a + .quad 0x296537d2cb045a31 + .quad 0x1bd0653ed3274fde + .quad 0x2f9a2c4476bd2966 + + // 2^176 * 1 * G + + .quad 0xeb18b9ab7f5745c6 + .quad 0x023a8aee5787c690 + .quad 0xb72712da2df7afa9 + .quad 0x36597d25ea5c013d + .quad 0xa2b4dae0b5511c9a + .quad 0x7ac860292bffff06 + .quad 0x981f375df5504234 + .quad 0x3f6bd725da4ea12d + .quad 0x734d8d7b106058ac + .quad 0xd940579e6fc6905f + .quad 0x6466f8f99202932d + .quad 0x7b7ecc19da60d6d0 + + // 2^176 * 2 * G + + .quad 0x78c2373c695c690d + .quad 0xdd252e660642906e + .quad 0x951d44444ae12bd2 + .quad 0x4235ad7601743956 + .quad 0x6dae4a51a77cfa9b + .quad 0x82263654e7a38650 + .quad 0x09bbffcd8f2d82db + .quad 0x03bedc661bf5caba + .quad 0x6258cb0d078975f5 + .quad 0x492942549189f298 + .quad 0xa0cab423e2e36ee4 + .quad 0x0e7ce2b0cdf066a1 + + // 2^176 * 3 * G + + .quad 0xc494643ac48c85a3 + .quad 0xfd361df43c6139ad + .quad 0x09db17dd3ae94d48 + .quad 0x666e0a5d8fb4674a + .quad 0xfea6fedfd94b70f9 + .quad 0xf130c051c1fcba2d + .quad 0x4882d47e7f2fab89 + .quad 0x615256138aeceeb5 + .quad 0x2abbf64e4870cb0d + .quad 0xcd65bcf0aa458b6b + .quad 0x9abe4eba75e8985d + .quad 0x7f0bc810d514dee4 + + // 2^176 * 4 * G + + .quad 0xb9006ba426f4136f + .quad 0x8d67369e57e03035 + .quad 0xcbc8dfd94f463c28 + .quad 0x0d1f8dbcf8eedbf5 + .quad 0x83ac9dad737213a0 + .quad 0x9ff6f8ba2ef72e98 + .quad 0x311e2edd43ec6957 + .quad 0x1d3a907ddec5ab75 + .quad 0xba1693313ed081dc + .quad 0x29329fad851b3480 + .quad 0x0128013c030321cb + .quad 0x00011b44a31bfde3 + + // 2^176 * 5 * G + + .quad 0x3fdfa06c3fc66c0c + .quad 0x5d40e38e4dd60dd2 + .quad 0x7ae38b38268e4d71 + .quad 0x3ac48d916e8357e1 + .quad 0x16561f696a0aa75c + .quad 0xc1bf725c5852bd6a + .quad 0x11a8dd7f9a7966ad + .quad 0x63d988a2d2851026 + .quad 0x00120753afbd232e + .quad 0xe92bceb8fdd8f683 + .quad 0xf81669b384e72b91 + .quad 0x33fad52b2368a066 + + // 2^176 * 6 * G + + .quad 0x540649c6c5e41e16 + .quad 0x0af86430333f7735 + .quad 0xb2acfcd2f305e746 + .quad 0x16c0f429a256dca7 + .quad 0x8d2cc8d0c422cfe8 + .quad 0x072b4f7b05a13acb + .quad 0xa3feb6e6ecf6a56f + .quad 0x3cc355ccb90a71e2 + .quad 0xe9b69443903e9131 + .quad 0xb8a494cb7a5637ce + .quad 0xc87cd1a4baba9244 + .quad 0x631eaf426bae7568 + + // 2^176 * 7 * G + + .quad 0xb3e90410da66fe9f + .quad 0x85dd4b526c16e5a6 + .quad 0xbc3d97611ef9bf83 + .quad 0x5599648b1ea919b5 + .quad 0x47d975b9a3700de8 + .quad 0x7280c5fbe2f80552 + .quad 0x53658f2732e45de1 + .quad 0x431f2c7f665f80b5 + .quad 0xd6026344858f7b19 + .quad 0x14ab352fa1ea514a + .quad 0x8900441a2090a9d7 + .quad 0x7b04715f91253b26 + + // 2^176 * 8 * G + + .quad 0x83edbd28acf6ae43 + .quad 0x86357c8b7d5c7ab4 + .quad 0xc0404769b7eb2c44 + .quad 0x59b37bf5c2f6583f + .quad 0xb376c280c4e6bac6 + .quad 0x970ed3dd6d1d9b0b + .quad 0xb09a9558450bf944 + .quad 0x48d0acfa57cde223 + .quad 0xb60f26e47dabe671 + .quad 0xf1d1a197622f3a37 + .quad 0x4208ce7ee9960394 + .quad 0x16234191336d3bdb + + // 2^180 * 1 * G + + .quad 0xf19aeac733a63aef + .quad 0x2c7fba5d4442454e + .quad 0x5da87aa04795e441 + .quad 0x413051e1a4e0b0f5 + .quad 0x852dd1fd3d578bbe + .quad 0x2b65ce72c3286108 + .quad 0x658c07f4eace2273 + .quad 0x0933f804ec38ab40 + .quad 0xa7ab69798d496476 + .quad 0x8121aadefcb5abc8 + .quad 0xa5dc12ef7b539472 + .quad 0x07fd47065e45351a + + // 2^180 * 2 * G + + .quad 0xc8583c3d258d2bcd + .quad 0x17029a4daf60b73f + .quad 0xfa0fc9d6416a3781 + .quad 0x1c1e5fba38b3fb23 + .quad 0x304211559ae8e7c3 + .quad 0xf281b229944882a5 + .quad 0x8a13ac2e378250e4 + .quad 0x014afa0954ba48f4 + .quad 0xcb3197001bb3666c + .quad 0x330060524bffecb9 + .quad 0x293711991a88233c + .quad 0x291884363d4ed364 + + // 2^180 * 3 * G + + .quad 0x033c6805dc4babfa + .quad 0x2c15bf5e5596ecc1 + .quad 0x1bc70624b59b1d3b + .quad 0x3ede9850a19f0ec5 + .quad 0xfb9d37c3bc1ab6eb + .quad 0x02be14534d57a240 + .quad 0xf4d73415f8a5e1f6 + .quad 0x5964f4300ccc8188 + .quad 0xe44a23152d096800 + .quad 0x5c08c55970866996 + .quad 0xdf2db60a46affb6e + .quad 0x579155c1f856fd89 + + // 2^180 * 4 * G + + .quad 0x96324edd12e0c9ef + .quad 0x468b878df2420297 + .quad 0x199a3776a4f573be + .quad 0x1e7fbcf18e91e92a + .quad 0xb5f16b630817e7a6 + .quad 0x808c69233c351026 + .quad 0x324a983b54cef201 + .quad 0x53c092084a485345 + .quad 0xd2d41481f1cbafbf + .quad 0x231d2db6716174e5 + .quad 0x0b7d7656e2a55c98 + .quad 0x3e955cd82aa495f6 + + // 2^180 * 5 * G + + .quad 0xe48f535e3ed15433 + .quad 0xd075692a0d7270a3 + .quad 0x40fbd21daade6387 + .quad 0x14264887cf4495f5 + .quad 0xab39f3ef61bb3a3f + .quad 0x8eb400652eb9193e + .quad 0xb5de6ecc38c11f74 + .quad 0x654d7e9626f3c49f + .quad 0xe564cfdd5c7d2ceb + .quad 0x82eeafded737ccb9 + .quad 0x6107db62d1f9b0ab + .quad 0x0b6baac3b4358dbb + + // 2^180 * 6 * G + + .quad 0x7ae62bcb8622fe98 + .quad 0x47762256ceb891af + .quad 0x1a5a92bcf2e406b4 + .quad 0x7d29401784e41501 + .quad 0x204abad63700a93b + .quad 0xbe0023d3da779373 + .quad 0xd85f0346633ab709 + .quad 0x00496dc490820412 + .quad 0x1c74b88dc27e6360 + .quad 0x074854268d14850c + .quad 0xa145fb7b3e0dcb30 + .quad 0x10843f1b43803b23 + + // 2^180 * 7 * G + + .quad 0xc5f90455376276dd + .quad 0xce59158dd7645cd9 + .quad 0x92f65d511d366b39 + .quad 0x11574b6e526996c4 + .quad 0xd56f672de324689b + .quad 0xd1da8aedb394a981 + .quad 0xdd7b58fe9168cfed + .quad 0x7ce246cd4d56c1e8 + .quad 0xb8f4308e7f80be53 + .quad 0x5f3cb8cb34a9d397 + .quad 0x18a961bd33cc2b2c + .quad 0x710045fb3a9af671 + + // 2^180 * 8 * G + + .quad 0x73f93d36101b95eb + .quad 0xfaef33794f6f4486 + .quad 0x5651735f8f15e562 + .quad 0x7fa3f19058b40da1 + .quad 0xa03fc862059d699e + .quad 0x2370cfa19a619e69 + .quad 0xc4fe3b122f823deb + .quad 0x1d1b056fa7f0844e + .quad 0x1bc64631e56bf61f + .quad 0xd379ab106e5382a3 + .quad 0x4d58c57e0540168d + .quad 0x566256628442d8e4 + + // 2^184 * 1 * G + + .quad 0xb9e499def6267ff6 + .quad 0x7772ca7b742c0843 + .quad 0x23a0153fe9a4f2b1 + .quad 0x2cdfdfecd5d05006 + .quad 0xdd499cd61ff38640 + .quad 0x29cd9bc3063625a0 + .quad 0x51e2d8023dd73dc3 + .quad 0x4a25707a203b9231 + .quad 0x2ab7668a53f6ed6a + .quad 0x304242581dd170a1 + .quad 0x4000144c3ae20161 + .quad 0x5721896d248e49fc + + // 2^184 * 2 * G + + .quad 0x0b6e5517fd181bae + .quad 0x9022629f2bb963b4 + .quad 0x5509bce932064625 + .quad 0x578edd74f63c13da + .quad 0x285d5091a1d0da4e + .quad 0x4baa6fa7b5fe3e08 + .quad 0x63e5177ce19393b3 + .quad 0x03c935afc4b030fd + .quad 0x997276c6492b0c3d + .quad 0x47ccc2c4dfe205fc + .quad 0xdcd29b84dd623a3c + .quad 0x3ec2ab590288c7a2 + + // 2^184 * 3 * G + + .quad 0xa1a0d27be4d87bb9 + .quad 0xa98b4deb61391aed + .quad 0x99a0ddd073cb9b83 + .quad 0x2dd5c25a200fcace + .quad 0xa7213a09ae32d1cb + .quad 0x0f2b87df40f5c2d5 + .quad 0x0baea4c6e81eab29 + .quad 0x0e1bf66c6adbac5e + .quad 0xe2abd5e9792c887e + .quad 0x1a020018cb926d5d + .quad 0xbfba69cdbaae5f1e + .quad 0x730548b35ae88f5f + + // 2^184 * 4 * G + + .quad 0xc43551a3cba8b8ee + .quad 0x65a26f1db2115f16 + .quad 0x760f4f52ab8c3850 + .quad 0x3043443b411db8ca + .quad 0x805b094ba1d6e334 + .quad 0xbf3ef17709353f19 + .quad 0x423f06cb0622702b + .quad 0x585a2277d87845dd + .quad 0xa18a5f8233d48962 + .quad 0x6698c4b5ec78257f + .quad 0xa78e6fa5373e41ff + .quad 0x7656278950ef981f + + // 2^184 * 5 * G + + .quad 0x38c3cf59d51fc8c0 + .quad 0x9bedd2fd0506b6f2 + .quad 0x26bf109fab570e8f + .quad 0x3f4160a8c1b846a6 + .quad 0xe17073a3ea86cf9d + .quad 0x3a8cfbb707155fdc + .quad 0x4853e7fc31838a8e + .quad 0x28bbf484b613f616 + .quad 0xf2612f5c6f136c7c + .quad 0xafead107f6dd11be + .quad 0x527e9ad213de6f33 + .quad 0x1e79cb358188f75d + + // 2^184 * 6 * G + + .quad 0x013436c3eef7e3f1 + .quad 0x828b6a7ffe9e10f8 + .quad 0x7ff908e5bcf9defc + .quad 0x65d7951b3a3b3831 + .quad 0x77e953d8f5e08181 + .quad 0x84a50c44299dded9 + .quad 0xdc6c2d0c864525e5 + .quad 0x478ab52d39d1f2f4 + .quad 0x66a6a4d39252d159 + .quad 0xe5dde1bc871ac807 + .quad 0xb82c6b40a6c1c96f + .quad 0x16d87a411a212214 + + // 2^184 * 7 * G + + .quad 0xb3bd7e5a42066215 + .quad 0x879be3cd0c5a24c1 + .quad 0x57c05db1d6f994b7 + .quad 0x28f87c8165f38ca6 + .quad 0xfba4d5e2d54e0583 + .quad 0xe21fafd72ebd99fa + .quad 0x497ac2736ee9778f + .quad 0x1f990b577a5a6dde + .quad 0xa3344ead1be8f7d6 + .quad 0x7d1e50ebacea798f + .quad 0x77c6569e520de052 + .quad 0x45882fe1534d6d3e + + // 2^184 * 8 * G + + .quad 0x6669345d757983d6 + .quad 0x62b6ed1117aa11a6 + .quad 0x7ddd1857985e128f + .quad 0x688fe5b8f626f6dd + .quad 0xd8ac9929943c6fe4 + .quad 0xb5f9f161a38392a2 + .quad 0x2699db13bec89af3 + .quad 0x7dcf843ce405f074 + .quad 0x6c90d6484a4732c0 + .quad 0xd52143fdca563299 + .quad 0xb3be28c3915dc6e1 + .quad 0x6739687e7327191b + + // 2^188 * 1 * G + + .quad 0x9f65c5ea200814cf + .quad 0x840536e169a31740 + .quad 0x8b0ed13925c8b4ad + .quad 0x0080dbafe936361d + .quad 0x8ce5aad0c9cb971f + .quad 0x1156aaa99fd54a29 + .quad 0x41f7247015af9b78 + .quad 0x1fe8cca8420f49aa + .quad 0x72a1848f3c0cc82a + .quad 0x38c560c2877c9e54 + .quad 0x5004e228ce554140 + .quad 0x042418a103429d71 + + // 2^188 * 2 * G + + .quad 0x899dea51abf3ff5f + .quad 0x9b93a8672fc2d8ba + .quad 0x2c38cb97be6ebd5c + .quad 0x114d578497263b5d + .quad 0x58e84c6f20816247 + .quad 0x8db2b2b6e36fd793 + .quad 0x977182561d484d85 + .quad 0x0822024f8632abd7 + .quad 0xb301bb7c6b1beca3 + .quad 0x55393f6dc6eb1375 + .quad 0x910d281097b6e4eb + .quad 0x1ad4548d9d479ea3 + + // 2^188 * 3 * G + + .quad 0xcd5a7da0389a48fd + .quad 0xb38fa4aa9a78371e + .quad 0xc6d9761b2cdb8e6c + .quad 0x35cf51dbc97e1443 + .quad 0xa06fe66d0fe9fed3 + .quad 0xa8733a401c587909 + .quad 0x30d14d800df98953 + .quad 0x41ce5876c7b30258 + .quad 0x59ac3bc5d670c022 + .quad 0xeae67c109b119406 + .quad 0x9798bdf0b3782fda + .quad 0x651e3201fd074092 + + // 2^188 * 4 * G + + .quad 0xd63d8483ef30c5cf + .quad 0x4cd4b4962361cc0c + .quad 0xee90e500a48426ac + .quad 0x0af51d7d18c14eeb + .quad 0xa57ba4a01efcae9e + .quad 0x769f4beedc308a94 + .quad 0xd1f10eeb3603cb2e + .quad 0x4099ce5e7e441278 + .quad 0x1ac98e4f8a5121e9 + .quad 0x7dae9544dbfa2fe0 + .quad 0x8320aa0dd6430df9 + .quad 0x667282652c4a2fb5 + + // 2^188 * 5 * G + + .quad 0x874621f4d86bc9ab + .quad 0xb54c7bbe56fe6fea + .quad 0x077a24257fadc22c + .quad 0x1ab53be419b90d39 + .quad 0xada8b6e02946db23 + .quad 0x1c0ce51a7b253ab7 + .quad 0x8448c85a66dd485b + .quad 0x7f1fc025d0675adf + .quad 0xd8ee1b18319ea6aa + .quad 0x004d88083a21f0da + .quad 0x3bd6aa1d883a4f4b + .quad 0x4db9a3a6dfd9fd14 + + // 2^188 * 6 * G + + .quad 0x8ce7b23bb99c0755 + .quad 0x35c5d6edc4f50f7a + .quad 0x7e1e2ed2ed9b50c3 + .quad 0x36305f16e8934da1 + .quad 0xd95b00bbcbb77c68 + .quad 0xddbc846a91f17849 + .quad 0x7cf700aebe28d9b3 + .quad 0x5ce1285c85d31f3e + .quad 0x31b6972d98b0bde8 + .quad 0x7d920706aca6de5b + .quad 0xe67310f8908a659f + .quad 0x50fac2a6efdf0235 + + // 2^188 * 7 * G + + .quad 0xf3d3a9f35b880f5a + .quad 0xedec050cdb03e7c2 + .quad 0xa896981ff9f0b1a2 + .quad 0x49a4ae2bac5e34a4 + .quad 0x295b1c86f6f449bc + .quad 0x51b2e84a1f0ab4dd + .quad 0xc001cb30aa8e551d + .quad 0x6a28d35944f43662 + .quad 0x28bb12ee04a740e0 + .quad 0x14313bbd9bce8174 + .quad 0x72f5b5e4e8c10c40 + .quad 0x7cbfb19936adcd5b + + // 2^188 * 8 * G + + .quad 0xa311ddc26b89792d + .quad 0x1b30b4c6da512664 + .quad 0x0ca77b4ccf150859 + .quad 0x1de443df1b009408 + .quad 0x8e793a7acc36e6e0 + .quad 0xf9fab7a37d586eed + .quad 0x3a4f9692bae1f4e4 + .quad 0x1c14b03eff5f447e + .quad 0x19647bd114a85291 + .quad 0x57b76cb21034d3af + .quad 0x6329db440f9d6dfa + .quad 0x5ef43e586a571493 + + // 2^192 * 1 * G + + .quad 0xef782014385675a6 + .quad 0xa2649f30aafda9e8 + .quad 0x4cd1eb505cdfa8cb + .quad 0x46115aba1d4dc0b3 + .quad 0xa66dcc9dc80c1ac0 + .quad 0x97a05cf41b38a436 + .quad 0xa7ebf3be95dbd7c6 + .quad 0x7da0b8f68d7e7dab + .quad 0xd40f1953c3b5da76 + .quad 0x1dac6f7321119e9b + .quad 0x03cc6021feb25960 + .quad 0x5a5f887e83674b4b + + // 2^192 * 2 * G + + .quad 0x8f6301cf70a13d11 + .quad 0xcfceb815350dd0c4 + .quad 0xf70297d4a4bca47e + .quad 0x3669b656e44d1434 + .quad 0x9e9628d3a0a643b9 + .quad 0xb5c3cb00e6c32064 + .quad 0x9b5302897c2dec32 + .quad 0x43e37ae2d5d1c70c + .quad 0x387e3f06eda6e133 + .quad 0x67301d5199a13ac0 + .quad 0xbd5ad8f836263811 + .quad 0x6a21e6cd4fd5e9be + + // 2^192 * 3 * G + + .quad 0xf1c6170a3046e65f + .quad 0x58712a2a00d23524 + .quad 0x69dbbd3c8c82b755 + .quad 0x586bf9f1a195ff57 + .quad 0xef4129126699b2e3 + .quad 0x71d30847708d1301 + .quad 0x325432d01182b0bd + .quad 0x45371b07001e8b36 + .quad 0xa6db088d5ef8790b + .quad 0x5278f0dc610937e5 + .quad 0xac0349d261a16eb8 + .quad 0x0eafb03790e52179 + + // 2^192 * 4 * G + + .quad 0x960555c13748042f + .quad 0x219a41e6820baa11 + .quad 0x1c81f73873486d0c + .quad 0x309acc675a02c661 + .quad 0x5140805e0f75ae1d + .quad 0xec02fbe32662cc30 + .quad 0x2cebdf1eea92396d + .quad 0x44ae3344c5435bb3 + .quad 0x9cf289b9bba543ee + .quad 0xf3760e9d5ac97142 + .quad 0x1d82e5c64f9360aa + .quad 0x62d5221b7f94678f + + // 2^192 * 5 * G + + .quad 0x524c299c18d0936d + .quad 0xc86bb56c8a0c1a0c + .quad 0xa375052edb4a8631 + .quad 0x5c0efde4bc754562 + .quad 0x7585d4263af77a3c + .quad 0xdfae7b11fee9144d + .quad 0xa506708059f7193d + .quad 0x14f29a5383922037 + .quad 0xdf717edc25b2d7f5 + .quad 0x21f970db99b53040 + .quad 0xda9234b7c3ed4c62 + .quad 0x5e72365c7bee093e + + // 2^192 * 6 * G + + .quad 0x575bfc074571217f + .quad 0x3779675d0694d95b + .quad 0x9a0a37bbf4191e33 + .quad 0x77f1104c47b4eabc + .quad 0x7d9339062f08b33e + .quad 0x5b9659e5df9f32be + .quad 0xacff3dad1f9ebdfd + .quad 0x70b20555cb7349b7 + .quad 0xbe5113c555112c4c + .quad 0x6688423a9a881fcd + .quad 0x446677855e503b47 + .quad 0x0e34398f4a06404a + + // 2^192 * 7 * G + + .quad 0xb67d22d93ecebde8 + .quad 0x09b3e84127822f07 + .quad 0x743fa61fb05b6d8d + .quad 0x5e5405368a362372 + .quad 0x18930b093e4b1928 + .quad 0x7de3e10e73f3f640 + .quad 0xf43217da73395d6f + .quad 0x6f8aded6ca379c3e + .quad 0xe340123dfdb7b29a + .quad 0x487b97e1a21ab291 + .quad 0xf9967d02fde6949e + .quad 0x780de72ec8d3de97 + + // 2^192 * 8 * G + + .quad 0x0ae28545089ae7bc + .quad 0x388ddecf1c7f4d06 + .quad 0x38ac15510a4811b8 + .quad 0x0eb28bf671928ce4 + .quad 0x671feaf300f42772 + .quad 0x8f72eb2a2a8c41aa + .quad 0x29a17fd797373292 + .quad 0x1defc6ad32b587a6 + .quad 0xaf5bbe1aef5195a7 + .quad 0x148c1277917b15ed + .quad 0x2991f7fb7ae5da2e + .quad 0x467d201bf8dd2867 + + // 2^196 * 1 * G + + .quad 0x7906ee72f7bd2e6b + .quad 0x05d270d6109abf4e + .quad 0x8d5cfe45b941a8a4 + .quad 0x44c218671c974287 + .quad 0x745f9d56296bc318 + .quad 0x993580d4d8152e65 + .quad 0xb0e5b13f5839e9ce + .quad 0x51fc2b28d43921c0 + .quad 0x1b8fd11795e2a98c + .quad 0x1c4e5ee12b6b6291 + .quad 0x5b30e7107424b572 + .quad 0x6e6b9de84c4f4ac6 + + // 2^196 * 2 * G + + .quad 0xdff25fce4b1de151 + .quad 0xd841c0c7e11c4025 + .quad 0x2554b3c854749c87 + .quad 0x2d292459908e0df9 + .quad 0x6b7c5f10f80cb088 + .quad 0x736b54dc56e42151 + .quad 0xc2b620a5c6ef99c4 + .quad 0x5f4c802cc3a06f42 + .quad 0x9b65c8f17d0752da + .quad 0x881ce338c77ee800 + .quad 0xc3b514f05b62f9e3 + .quad 0x66ed5dd5bec10d48 + + // 2^196 * 3 * G + + .quad 0x7d38a1c20bb2089d + .quad 0x808334e196ccd412 + .quad 0xc4a70b8c6c97d313 + .quad 0x2eacf8bc03007f20 + .quad 0xf0adf3c9cbca047d + .quad 0x81c3b2cbf4552f6b + .quad 0xcfda112d44735f93 + .quad 0x1f23a0c77e20048c + .quad 0xf235467be5bc1570 + .quad 0x03d2d9020dbab38c + .quad 0x27529aa2fcf9e09e + .quad 0x0840bef29d34bc50 + + // 2^196 * 4 * G + + .quad 0x796dfb35dc10b287 + .quad 0x27176bcd5c7ff29d + .quad 0x7f3d43e8c7b24905 + .quad 0x0304f5a191c54276 + .quad 0xcd54e06b7f37e4eb + .quad 0x8cc15f87f5e96cca + .quad 0xb8248bb0d3597dce + .quad 0x246affa06074400c + .quad 0x37d88e68fbe45321 + .quad 0x86097548c0d75032 + .quad 0x4e9b13ef894a0d35 + .quad 0x25a83cac5753d325 + + // 2^196 * 5 * G + + .quad 0x10222f48eed8165e + .quad 0x623fc1234b8bcf3a + .quad 0x1e145c09c221e8f0 + .quad 0x7ccfa59fca782630 + .quad 0x9f0f66293952b6e2 + .quad 0x33db5e0e0934267b + .quad 0xff45252bd609fedc + .quad 0x06be10f5c506e0c9 + .quad 0x1a9615a9b62a345f + .quad 0x22050c564a52fecc + .quad 0xa7a2788528bc0dfe + .quad 0x5e82770a1a1ee71d + + // 2^196 * 6 * G + + .quad 0x35425183ad896a5c + .quad 0xe8673afbe78d52f6 + .quad 0x2c66f25f92a35f64 + .quad 0x09d04f3b3b86b102 + .quad 0xe802e80a42339c74 + .quad 0x34175166a7fffae5 + .quad 0x34865d1f1c408cae + .quad 0x2cca982c605bc5ee + .quad 0xfd2d5d35197dbe6e + .quad 0x207c2eea8be4ffa3 + .quad 0x2613d8db325ae918 + .quad 0x7a325d1727741d3e + + // 2^196 * 7 * G + + .quad 0xd036b9bbd16dfde2 + .quad 0xa2055757c497a829 + .quad 0x8e6cc966a7f12667 + .quad 0x4d3b1a791239c180 + .quad 0xecd27d017e2a076a + .quad 0xd788689f1636495e + .quad 0x52a61af0919233e5 + .quad 0x2a479df17bb1ae64 + .quad 0x9e5eee8e33db2710 + .quad 0x189854ded6c43ca5 + .quad 0xa41c22c592718138 + .quad 0x27ad5538a43a5e9b + + // 2^196 * 8 * G + + .quad 0x2746dd4b15350d61 + .quad 0xd03fcbc8ee9521b7 + .quad 0xe86e365a138672ca + .quad 0x510e987f7e7d89e2 + .quad 0xcb5a7d638e47077c + .quad 0x8db7536120a1c059 + .quad 0x549e1e4d8bedfdcc + .quad 0x080153b7503b179d + .quad 0xdda69d930a3ed3e3 + .quad 0x3d386ef1cd60a722 + .quad 0xc817ad58bdaa4ee6 + .quad 0x23be8d554fe7372a + + // 2^200 * 1 * G + + .quad 0x95fe919a74ef4fad + .quad 0x3a827becf6a308a2 + .quad 0x964e01d309a47b01 + .quad 0x71c43c4f5ba3c797 + .quad 0xbc1ef4bd567ae7a9 + .quad 0x3f624cb2d64498bd + .quad 0xe41064d22c1f4ec8 + .quad 0x2ef9c5a5ba384001 + .quad 0xb6fd6df6fa9e74cd + .quad 0xf18278bce4af267a + .quad 0x8255b3d0f1ef990e + .quad 0x5a758ca390c5f293 + + // 2^200 * 2 * G + + .quad 0xa2b72710d9462495 + .quad 0x3aa8c6d2d57d5003 + .quad 0xe3d400bfa0b487ca + .quad 0x2dbae244b3eb72ec + .quad 0x8ce0918b1d61dc94 + .quad 0x8ded36469a813066 + .quad 0xd4e6a829afe8aad3 + .quad 0x0a738027f639d43f + .quad 0x980f4a2f57ffe1cc + .quad 0x00670d0de1839843 + .quad 0x105c3f4a49fb15fd + .quad 0x2698ca635126a69c + + // 2^200 * 3 * G + + .quad 0xe765318832b0ba78 + .quad 0x381831f7925cff8b + .quad 0x08a81b91a0291fcc + .quad 0x1fb43dcc49caeb07 + .quad 0x2e3d702f5e3dd90e + .quad 0x9e3f0918e4d25386 + .quad 0x5e773ef6024da96a + .quad 0x3c004b0c4afa3332 + .quad 0x9aa946ac06f4b82b + .quad 0x1ca284a5a806c4f3 + .quad 0x3ed3265fc6cd4787 + .quad 0x6b43fd01cd1fd217 + + // 2^200 * 4 * G + + .quad 0xc7a75d4b4697c544 + .quad 0x15fdf848df0fffbf + .quad 0x2868b9ebaa46785a + .quad 0x5a68d7105b52f714 + .quad 0xb5c742583e760ef3 + .quad 0x75dc52b9ee0ab990 + .quad 0xbf1427c2072b923f + .quad 0x73420b2d6ff0d9f0 + .quad 0xaf2cf6cb9e851e06 + .quad 0x8f593913c62238c4 + .quad 0xda8ab89699fbf373 + .quad 0x3db5632fea34bc9e + + // 2^200 * 5 * G + + .quad 0xf46eee2bf75dd9d8 + .quad 0x0d17b1f6396759a5 + .quad 0x1bf2d131499e7273 + .quad 0x04321adf49d75f13 + .quad 0x2e4990b1829825d5 + .quad 0xedeaeb873e9a8991 + .quad 0xeef03d394c704af8 + .quad 0x59197ea495df2b0e + .quad 0x04e16019e4e55aae + .quad 0xe77b437a7e2f92e9 + .quad 0xc7ce2dc16f159aa4 + .quad 0x45eafdc1f4d70cc0 + + // 2^200 * 6 * G + + .quad 0x698401858045d72b + .quad 0x4c22faa2cf2f0651 + .quad 0x941a36656b222dc6 + .quad 0x5a5eebc80362dade + .quad 0xb60e4624cfccb1ed + .quad 0x59dbc292bd5c0395 + .quad 0x31a09d1ddc0481c9 + .quad 0x3f73ceea5d56d940 + .quad 0xb7a7bfd10a4e8dc6 + .quad 0xbe57007e44c9b339 + .quad 0x60c1207f1557aefa + .quad 0x26058891266218db + + // 2^200 * 7 * G + + .quad 0x59f704a68360ff04 + .quad 0xc3d93fde7661e6f4 + .quad 0x831b2a7312873551 + .quad 0x54ad0c2e4e615d57 + .quad 0x4c818e3cc676e542 + .quad 0x5e422c9303ceccad + .quad 0xec07cccab4129f08 + .quad 0x0dedfa10b24443b8 + .quad 0xee3b67d5b82b522a + .quad 0x36f163469fa5c1eb + .quad 0xa5b4d2f26ec19fd3 + .quad 0x62ecb2baa77a9408 + + // 2^200 * 8 * G + + .quad 0xe5ed795261152b3d + .quad 0x4962357d0eddd7d1 + .quad 0x7482c8d0b96b4c71 + .quad 0x2e59f919a966d8be + .quad 0x92072836afb62874 + .quad 0x5fcd5e8579e104a5 + .quad 0x5aad01adc630a14a + .quad 0x61913d5075663f98 + .quad 0x0dc62d361a3231da + .quad 0xfa47583294200270 + .quad 0x02d801513f9594ce + .quad 0x3ddbc2a131c05d5c + + // 2^204 * 1 * G + + .quad 0x3f50a50a4ffb81ef + .quad 0xb1e035093bf420bf + .quad 0x9baa8e1cc6aa2cd0 + .quad 0x32239861fa237a40 + .quad 0xfb735ac2004a35d1 + .quad 0x31de0f433a6607c3 + .quad 0x7b8591bfc528d599 + .quad 0x55be9a25f5bb050c + .quad 0x0d005acd33db3dbf + .quad 0x0111b37c80ac35e2 + .quad 0x4892d66c6f88ebeb + .quad 0x770eadb16508fbcd + + // 2^204 * 2 * G + + .quad 0x8451f9e05e4e89dd + .quad 0xc06302ffbc793937 + .quad 0x5d22749556a6495c + .quad 0x09a6755ca05603fb + .quad 0xf1d3b681a05071b9 + .quad 0x2207659a3592ff3a + .quad 0x5f0169297881e40e + .quad 0x16bedd0e86ba374e + .quad 0x5ecccc4f2c2737b5 + .quad 0x43b79e0c2dccb703 + .quad 0x33e008bc4ec43df3 + .quad 0x06c1b840f07566c0 + + // 2^204 * 3 * G + + .quad 0x7688a5c6a388f877 + .quad 0x02a96c14deb2b6ac + .quad 0x64c9f3431b8c2af8 + .quad 0x3628435554a1eed6 + .quad 0x69ee9e7f9b02805c + .quad 0xcbff828a547d1640 + .quad 0x3d93a869b2430968 + .quad 0x46b7b8cd3fe26972 + .quad 0xe9812086fe7eebe0 + .quad 0x4cba6be72f515437 + .quad 0x1d04168b516efae9 + .quad 0x5ea1391043982cb9 + + // 2^204 * 4 * G + + .quad 0x49125c9cf4702ee1 + .quad 0x4520b71f8b25b32d + .quad 0x33193026501fef7e + .quad 0x656d8997c8d2eb2b + .quad 0x6f2b3be4d5d3b002 + .quad 0xafec33d96a09c880 + .quad 0x035f73a4a8bcc4cc + .quad 0x22c5b9284662198b + .quad 0xcb58c8fe433d8939 + .quad 0x89a0cb2e6a8d7e50 + .quad 0x79ca955309fbbe5a + .quad 0x0c626616cd7fc106 + + // 2^204 * 5 * G + + .quad 0x1ffeb80a4879b61f + .quad 0x6396726e4ada21ed + .quad 0x33c7b093368025ba + .quad 0x471aa0c6f3c31788 + .quad 0x8fdfc379fbf454b1 + .quad 0x45a5a970f1a4b771 + .quad 0xac921ef7bad35915 + .quad 0x42d088dca81c2192 + .quad 0x8fda0f37a0165199 + .quad 0x0adadb77c8a0e343 + .quad 0x20fbfdfcc875e820 + .quad 0x1cf2bea80c2206e7 + + // 2^204 * 6 * G + + .quad 0xc2ddf1deb36202ac + .quad 0x92a5fe09d2e27aa5 + .quad 0x7d1648f6fc09f1d3 + .quad 0x74c2cc0513bc4959 + .quad 0x982d6e1a02c0412f + .quad 0x90fa4c83db58e8fe + .quad 0x01c2f5bcdcb18bc0 + .quad 0x686e0c90216abc66 + .quad 0x1fadbadba54395a7 + .quad 0xb41a02a0ae0da66a + .quad 0xbf19f598bba37c07 + .quad 0x6a12b8acde48430d + + // 2^204 * 7 * G + + .quad 0xf8daea1f39d495d9 + .quad 0x592c190e525f1dfc + .quad 0xdb8cbd04c9991d1b + .quad 0x11f7fda3d88f0cb7 + .quad 0x793bdd801aaeeb5f + .quad 0x00a2a0aac1518871 + .quad 0xe8a373a31f2136b4 + .quad 0x48aab888fc91ef19 + .quad 0x041f7e925830f40e + .quad 0x002d6ca979661c06 + .quad 0x86dc9ff92b046a2e + .quad 0x760360928b0493d1 + + // 2^204 * 8 * G + + .quad 0x21bb41c6120cf9c6 + .quad 0xeab2aa12decda59b + .quad 0xc1a72d020aa48b34 + .quad 0x215d4d27e87d3b68 + .quad 0xb43108e5695a0b05 + .quad 0x6cb00ee8ad37a38b + .quad 0x5edad6eea3537381 + .quad 0x3f2602d4b6dc3224 + .quad 0xc8b247b65bcaf19c + .quad 0x49779dc3b1b2c652 + .quad 0x89a180bbd5ece2e2 + .quad 0x13f098a3cec8e039 + + // 2^208 * 1 * G + + .quad 0x9adc0ff9ce5ec54b + .quad 0x039c2a6b8c2f130d + .quad 0x028007c7f0f89515 + .quad 0x78968314ac04b36b + .quad 0xf3aa57a22796bb14 + .quad 0x883abab79b07da21 + .quad 0xe54be21831a0391c + .quad 0x5ee7fb38d83205f9 + .quad 0x538dfdcb41446a8e + .quad 0xa5acfda9434937f9 + .quad 0x46af908d263c8c78 + .quad 0x61d0633c9bca0d09 + + // 2^208 * 2 * G + + .quad 0x63744935ffdb2566 + .quad 0xc5bd6b89780b68bb + .quad 0x6f1b3280553eec03 + .quad 0x6e965fd847aed7f5 + .quad 0xada328bcf8fc73df + .quad 0xee84695da6f037fc + .quad 0x637fb4db38c2a909 + .quad 0x5b23ac2df8067bdc + .quad 0x9ad2b953ee80527b + .quad 0xe88f19aafade6d8d + .quad 0x0e711704150e82cf + .quad 0x79b9bbb9dd95dedc + + // 2^208 * 3 * G + + .quad 0xebb355406a3126c2 + .quad 0xd26383a868c8c393 + .quad 0x6c0c6429e5b97a82 + .quad 0x5065f158c9fd2147 + .quad 0xd1997dae8e9f7374 + .quad 0xa032a2f8cfbb0816 + .quad 0xcd6cba126d445f0a + .quad 0x1ba811460accb834 + .quad 0x708169fb0c429954 + .quad 0xe14600acd76ecf67 + .quad 0x2eaab98a70e645ba + .quad 0x3981f39e58a4faf2 + + // 2^208 * 4 * G + + .quad 0x18fb8a7559230a93 + .quad 0x1d168f6960e6f45d + .quad 0x3a85a94514a93cb5 + .quad 0x38dc083705acd0fd + .quad 0xc845dfa56de66fde + .quad 0xe152a5002c40483a + .quad 0xe9d2e163c7b4f632 + .quad 0x30f4452edcbc1b65 + .quad 0x856d2782c5759740 + .quad 0xfa134569f99cbecc + .quad 0x8844fc73c0ea4e71 + .quad 0x632d9a1a593f2469 + + // 2^208 * 5 * G + + .quad 0xf6bb6b15b807cba6 + .quad 0x1823c7dfbc54f0d7 + .quad 0xbb1d97036e29670b + .quad 0x0b24f48847ed4a57 + .quad 0xbf09fd11ed0c84a7 + .quad 0x63f071810d9f693a + .quad 0x21908c2d57cf8779 + .quad 0x3a5a7df28af64ba2 + .quad 0xdcdad4be511beac7 + .quad 0xa4538075ed26ccf2 + .quad 0xe19cff9f005f9a65 + .quad 0x34fcf74475481f63 + + // 2^208 * 6 * G + + .quad 0xc197e04c789767ca + .quad 0xb8714dcb38d9467d + .quad 0x55de888283f95fa8 + .quad 0x3d3bdc164dfa63f7 + .quad 0xa5bb1dab78cfaa98 + .quad 0x5ceda267190b72f2 + .quad 0x9309c9110a92608e + .quad 0x0119a3042fb374b0 + .quad 0x67a2d89ce8c2177d + .quad 0x669da5f66895d0c1 + .quad 0xf56598e5b282a2b0 + .quad 0x56c088f1ede20a73 + + // 2^208 * 7 * G + + .quad 0x336d3d1110a86e17 + .quad 0xd7f388320b75b2fa + .quad 0xf915337625072988 + .quad 0x09674c6b99108b87 + .quad 0x581b5fac24f38f02 + .quad 0xa90be9febae30cbd + .quad 0x9a2169028acf92f0 + .quad 0x038b7ea48359038f + .quad 0x9f4ef82199316ff8 + .quad 0x2f49d282eaa78d4f + .quad 0x0971a5ab5aef3174 + .quad 0x6e5e31025969eb65 + + // 2^208 * 8 * G + + .quad 0xb16c62f587e593fb + .quad 0x4999eddeca5d3e71 + .quad 0xb491c1e014cc3e6d + .quad 0x08f5114789a8dba8 + .quad 0x3304fb0e63066222 + .quad 0xfb35068987acba3f + .quad 0xbd1924778c1061a3 + .quad 0x3058ad43d1838620 + .quad 0x323c0ffde57663d0 + .quad 0x05c3df38a22ea610 + .quad 0xbdc78abdac994f9a + .quad 0x26549fa4efe3dc99 + + // 2^212 * 1 * G + + .quad 0x738b38d787ce8f89 + .quad 0xb62658e24179a88d + .quad 0x30738c9cf151316d + .quad 0x49128c7f727275c9 + .quad 0x04dbbc17f75396b9 + .quad 0x69e6a2d7d2f86746 + .quad 0xc6409d99f53eabc6 + .quad 0x606175f6332e25d2 + .quad 0x4021370ef540e7dd + .quad 0x0910d6f5a1f1d0a5 + .quad 0x4634aacd5b06b807 + .quad 0x6a39e6356944f235 + + // 2^212 * 2 * G + + .quad 0x96cd5640df90f3e7 + .quad 0x6c3a760edbfa25ea + .quad 0x24f3ef0959e33cc4 + .quad 0x42889e7e530d2e58 + .quad 0x1da1965774049e9d + .quad 0xfbcd6ea198fe352b + .quad 0xb1cbcd50cc5236a6 + .quad 0x1f5ec83d3f9846e2 + .quad 0x8efb23c3328ccb75 + .quad 0xaf42a207dd876ee9 + .quad 0x20fbdadc5dfae796 + .quad 0x241e246b06bf9f51 + + // 2^212 * 3 * G + + .quad 0x29e68e57ad6e98f6 + .quad 0x4c9260c80b462065 + .quad 0x3f00862ea51ebb4b + .quad 0x5bc2c77fb38d9097 + .quad 0x7eaafc9a6280bbb8 + .quad 0x22a70f12f403d809 + .quad 0x31ce40bb1bfc8d20 + .quad 0x2bc65635e8bd53ee + .quad 0xe8d5dc9fa96bad93 + .quad 0xe58fb17dde1947dc + .quad 0x681532ea65185fa3 + .quad 0x1fdd6c3b034a7830 + + // 2^212 * 4 * G + + .quad 0x0a64e28c55dc18fe + .quad 0xe3df9e993399ebdd + .quad 0x79ac432370e2e652 + .quad 0x35ff7fc33ae4cc0e + .quad 0x9c13a6a52dd8f7a9 + .quad 0x2dbb1f8c3efdcabf + .quad 0x961e32405e08f7b5 + .quad 0x48c8a121bbe6c9e5 + .quad 0xfc415a7c59646445 + .quad 0xd224b2d7c128b615 + .quad 0x6035c9c905fbb912 + .quad 0x42d7a91274429fab + + // 2^212 * 5 * G + + .quad 0x4e6213e3eaf72ed3 + .quad 0x6794981a43acd4e7 + .quad 0xff547cde6eb508cb + .quad 0x6fed19dd10fcb532 + .quad 0xa9a48947933da5bc + .quad 0x4a58920ec2e979ec + .quad 0x96d8800013e5ac4c + .quad 0x453692d74b48b147 + .quad 0xdd775d99a8559c6f + .quad 0xf42a2140df003e24 + .quad 0x5223e229da928a66 + .quad 0x063f46ba6d38f22c + + // 2^212 * 6 * G + + .quad 0xd2d242895f536694 + .quad 0xca33a2c542939b2c + .quad 0x986fada6c7ddb95c + .quad 0x5a152c042f712d5d + .quad 0x39843cb737346921 + .quad 0xa747fb0738c89447 + .quad 0xcb8d8031a245307e + .quad 0x67810f8e6d82f068 + .quad 0x3eeb8fbcd2287db4 + .quad 0x72c7d3a301a03e93 + .quad 0x5473e88cbd98265a + .quad 0x7324aa515921b403 + + // 2^212 * 7 * G + + .quad 0x857942f46c3cbe8e + .quad 0xa1d364b14730c046 + .quad 0x1c8ed914d23c41bf + .quad 0x0838e161eef6d5d2 + .quad 0xad23f6dae82354cb + .quad 0x6962502ab6571a6d + .quad 0x9b651636e38e37d1 + .quad 0x5cac5005d1a3312f + .quad 0x8cc154cce9e39904 + .quad 0x5b3a040b84de6846 + .quad 0xc4d8a61cb1be5d6e + .quad 0x40fb897bd8861f02 + + // 2^212 * 8 * G + + .quad 0x84c5aa9062de37a1 + .quad 0x421da5000d1d96e1 + .quad 0x788286306a9242d9 + .quad 0x3c5e464a690d10da + .quad 0xe57ed8475ab10761 + .quad 0x71435e206fd13746 + .quad 0x342f824ecd025632 + .quad 0x4b16281ea8791e7b + .quad 0xd1c101d50b813381 + .quad 0xdee60f1176ee6828 + .quad 0x0cb68893383f6409 + .quad 0x6183c565f6ff484a + + // 2^216 * 1 * G + + .quad 0x741d5a461e6bf9d6 + .quad 0x2305b3fc7777a581 + .quad 0xd45574a26474d3d9 + .quad 0x1926e1dc6401e0ff + .quad 0xdb468549af3f666e + .quad 0xd77fcf04f14a0ea5 + .quad 0x3df23ff7a4ba0c47 + .quad 0x3a10dfe132ce3c85 + .quad 0xe07f4e8aea17cea0 + .quad 0x2fd515463a1fc1fd + .quad 0x175322fd31f2c0f1 + .quad 0x1fa1d01d861e5d15 + + // 2^216 * 2 * G + + .quad 0xcc8055947d599832 + .quad 0x1e4656da37f15520 + .quad 0x99f6f7744e059320 + .quad 0x773563bc6a75cf33 + .quad 0x38dcac00d1df94ab + .quad 0x2e712bddd1080de9 + .quad 0x7f13e93efdd5e262 + .quad 0x73fced18ee9a01e5 + .quad 0x06b1e90863139cb3 + .quad 0xa493da67c5a03ecd + .quad 0x8d77cec8ad638932 + .quad 0x1f426b701b864f44 + + // 2^216 * 3 * G + + .quad 0xefc9264c41911c01 + .quad 0xf1a3b7b817a22c25 + .quad 0x5875da6bf30f1447 + .quad 0x4e1af5271d31b090 + .quad 0xf17e35c891a12552 + .quad 0xb76b8153575e9c76 + .quad 0xfa83406f0d9b723e + .quad 0x0b76bb1b3fa7e438 + .quad 0x08b8c1f97f92939b + .quad 0xbe6771cbd444ab6e + .quad 0x22e5646399bb8017 + .quad 0x7b6dd61eb772a955 + + // 2^216 * 4 * G + + .quad 0xb7adc1e850f33d92 + .quad 0x7998fa4f608cd5cf + .quad 0xad962dbd8dfc5bdb + .quad 0x703e9bceaf1d2f4f + .quad 0x5730abf9ab01d2c7 + .quad 0x16fb76dc40143b18 + .quad 0x866cbe65a0cbb281 + .quad 0x53fa9b659bff6afe + .quad 0x6c14c8e994885455 + .quad 0x843a5d6665aed4e5 + .quad 0x181bb73ebcd65af1 + .quad 0x398d93e5c4c61f50 + + // 2^216 * 5 * G + + .quad 0x1c4bd16733e248f3 + .quad 0xbd9e128715bf0a5f + .quad 0xd43f8cf0a10b0376 + .quad 0x53b09b5ddf191b13 + .quad 0xc3877c60d2e7e3f2 + .quad 0x3b34aaa030828bb1 + .quad 0x283e26e7739ef138 + .quad 0x699c9c9002c30577 + .quad 0xf306a7235946f1cc + .quad 0x921718b5cce5d97d + .quad 0x28cdd24781b4e975 + .quad 0x51caf30c6fcdd907 + + // 2^216 * 6 * G + + .quad 0xa60ba7427674e00a + .quad 0x630e8570a17a7bf3 + .quad 0x3758563dcf3324cc + .quad 0x5504aa292383fdaa + .quad 0x737af99a18ac54c7 + .quad 0x903378dcc51cb30f + .quad 0x2b89bc334ce10cc7 + .quad 0x12ae29c189f8e99a + .quad 0xa99ec0cb1f0d01cf + .quad 0x0dd1efcc3a34f7ae + .quad 0x55ca7521d09c4e22 + .quad 0x5fd14fe958eba5ea + + // 2^216 * 7 * G + + .quad 0xb5dc2ddf2845ab2c + .quad 0x069491b10a7fe993 + .quad 0x4daaf3d64002e346 + .quad 0x093ff26e586474d1 + .quad 0x3c42fe5ebf93cb8e + .quad 0xbedfa85136d4565f + .quad 0xe0f0859e884220e8 + .quad 0x7dd73f960725d128 + .quad 0xb10d24fe68059829 + .quad 0x75730672dbaf23e5 + .quad 0x1367253ab457ac29 + .quad 0x2f59bcbc86b470a4 + + // 2^216 * 8 * G + + .quad 0x83847d429917135f + .quad 0xad1b911f567d03d7 + .quad 0x7e7748d9be77aad1 + .quad 0x5458b42e2e51af4a + .quad 0x7041d560b691c301 + .quad 0x85201b3fadd7e71e + .quad 0x16c2e16311335585 + .quad 0x2aa55e3d010828b1 + .quad 0xed5192e60c07444f + .quad 0x42c54e2d74421d10 + .quad 0x352b4c82fdb5c864 + .quad 0x13e9004a8a768664 + + // 2^220 * 1 * G + + .quad 0xcbb5b5556c032bff + .quad 0xdf7191b729297a3a + .quad 0xc1ff7326aded81bb + .quad 0x71ade8bb68be03f5 + .quad 0x1e6284c5806b467c + .quad 0xc5f6997be75d607b + .quad 0x8b67d958b378d262 + .quad 0x3d88d66a81cd8b70 + .quad 0x8b767a93204ed789 + .quad 0x762fcacb9fa0ae2a + .quad 0x771febcc6dce4887 + .quad 0x343062158ff05fb3 + + // 2^220 * 2 * G + + .quad 0xe05da1a7e1f5bf49 + .quad 0x26457d6dd4736092 + .quad 0x77dcb07773cc32f6 + .quad 0x0a5d94969cdd5fcd + .quad 0xfce219072a7b31b4 + .quad 0x4d7adc75aa578016 + .quad 0x0ec276a687479324 + .quad 0x6d6d9d5d1fda4beb + .quad 0x22b1a58ae9b08183 + .quad 0xfd95d071c15c388b + .quad 0xa9812376850a0517 + .quad 0x33384cbabb7f335e + + // 2^220 * 3 * G + + .quad 0x3c6fa2680ca2c7b5 + .quad 0x1b5082046fb64fda + .quad 0xeb53349c5431d6de + .quad 0x5278b38f6b879c89 + .quad 0x33bc627a26218b8d + .quad 0xea80b21fc7a80c61 + .quad 0x9458b12b173e9ee6 + .quad 0x076247be0e2f3059 + .quad 0x52e105f61416375a + .quad 0xec97af3685abeba4 + .quad 0x26e6b50623a67c36 + .quad 0x5cf0e856f3d4fb01 + + // 2^220 * 4 * G + + .quad 0xf6c968731ae8cab4 + .quad 0x5e20741ecb4f92c5 + .quad 0x2da53be58ccdbc3e + .quad 0x2dddfea269970df7 + .quad 0xbeaece313db342a8 + .quad 0xcba3635b842db7ee + .quad 0xe88c6620817f13ef + .quad 0x1b9438aa4e76d5c6 + .quad 0x8a50777e166f031a + .quad 0x067b39f10fb7a328 + .quad 0x1925c9a6010fbd76 + .quad 0x6df9b575cc740905 + + // 2^220 * 5 * G + + .quad 0x42c1192927f6bdcf + .quad 0x8f91917a403d61ca + .quad 0xdc1c5a668b9e1f61 + .quad 0x1596047804ec0f8d + .quad 0xecdfc35b48cade41 + .quad 0x6a88471fb2328270 + .quad 0x740a4a2440a01b6a + .quad 0x471e5796003b5f29 + .quad 0xda96bbb3aced37ac + .quad 0x7a2423b5e9208cea + .quad 0x24cc5c3038aebae2 + .quad 0x50c356afdc5dae2f + + // 2^220 * 6 * G + + .quad 0x09dcbf4341c30318 + .quad 0xeeba061183181dce + .quad 0xc179c0cedc1e29a1 + .quad 0x1dbf7b89073f35b0 + .quad 0xcfed9cdf1b31b964 + .quad 0xf486a9858ca51af3 + .quad 0x14897265ea8c1f84 + .quad 0x784a53dd932acc00 + .quad 0x2d99f9df14fc4920 + .quad 0x76ccb60cc4499fe5 + .quad 0xa4132cbbe5cf0003 + .quad 0x3f93d82354f000ea + + // 2^220 * 7 * G + + .quad 0x8183e7689e04ce85 + .quad 0x678fb71e04465341 + .quad 0xad92058f6688edac + .quad 0x5da350d3532b099a + .quad 0xeaac12d179e14978 + .quad 0xff923ff3bbebff5e + .quad 0x4af663e40663ce27 + .quad 0x0fd381a811a5f5ff + .quad 0xf256aceca436df54 + .quad 0x108b6168ae69d6e8 + .quad 0x20d986cb6b5d036c + .quad 0x655957b9fee2af50 + + // 2^220 * 8 * G + + .quad 0xaea8b07fa902030f + .quad 0xf88c766af463d143 + .quad 0x15b083663c787a60 + .quad 0x08eab1148267a4a8 + .quad 0xbdc1409bd002d0ac + .quad 0x66660245b5ccd9a6 + .quad 0x82317dc4fade85ec + .quad 0x02fe934b6ad7df0d + .quad 0xef5cf100cfb7ea74 + .quad 0x22897633a1cb42ac + .quad 0xd4ce0c54cef285e2 + .quad 0x30408c048a146a55 + + // 2^224 * 1 * G + + .quad 0x739d8845832fcedb + .quad 0xfa38d6c9ae6bf863 + .quad 0x32bc0dcab74ffef7 + .quad 0x73937e8814bce45e + .quad 0xbb2e00c9193b877f + .quad 0xece3a890e0dc506b + .quad 0xecf3b7c036de649f + .quad 0x5f46040898de9e1a + .quad 0xb9037116297bf48d + .quad 0xa9d13b22d4f06834 + .quad 0xe19715574696bdc6 + .quad 0x2cf8a4e891d5e835 + + // 2^224 * 2 * G + + .quad 0x6d93fd8707110f67 + .quad 0xdd4c09d37c38b549 + .quad 0x7cb16a4cc2736a86 + .quad 0x2049bd6e58252a09 + .quad 0x2cb5487e17d06ba2 + .quad 0x24d2381c3950196b + .quad 0xd7659c8185978a30 + .quad 0x7a6f7f2891d6a4f6 + .quad 0x7d09fd8d6a9aef49 + .quad 0xf0ee60be5b3db90b + .quad 0x4c21b52c519ebfd4 + .quad 0x6011aadfc545941d + + // 2^224 * 3 * G + + .quad 0x5f67926dcf95f83c + .quad 0x7c7e856171289071 + .quad 0xd6a1e7f3998f7a5b + .quad 0x6fc5cc1b0b62f9e0 + .quad 0x63ded0c802cbf890 + .quad 0xfbd098ca0dff6aaa + .quad 0x624d0afdb9b6ed99 + .quad 0x69ce18b779340b1e + .quad 0xd1ef5528b29879cb + .quad 0xdd1aae3cd47e9092 + .quad 0x127e0442189f2352 + .quad 0x15596b3ae57101f1 + + // 2^224 * 4 * G + + .quad 0x462739d23f9179a2 + .quad 0xff83123197d6ddcf + .quad 0x1307deb553f2148a + .quad 0x0d2237687b5f4dda + .quad 0x09ff31167e5124ca + .quad 0x0be4158bd9c745df + .quad 0x292b7d227ef556e5 + .quad 0x3aa4e241afb6d138 + .quad 0x2cc138bf2a3305f5 + .quad 0x48583f8fa2e926c3 + .quad 0x083ab1a25549d2eb + .quad 0x32fcaa6e4687a36c + + // 2^224 * 5 * G + + .quad 0x7bc56e8dc57d9af5 + .quad 0x3e0bd2ed9df0bdf2 + .quad 0xaac014de22efe4a3 + .quad 0x4627e9cefebd6a5c + .quad 0x3207a4732787ccdf + .quad 0x17e31908f213e3f8 + .quad 0xd5b2ecd7f60d964e + .quad 0x746f6336c2600be9 + .quad 0x3f4af345ab6c971c + .quad 0xe288eb729943731f + .quad 0x33596a8a0344186d + .quad 0x7b4917007ed66293 + + // 2^224 * 6 * G + + .quad 0x2d85fb5cab84b064 + .quad 0x497810d289f3bc14 + .quad 0x476adc447b15ce0c + .quad 0x122ba376f844fd7b + .quad 0x54341b28dd53a2dd + .quad 0xaa17905bdf42fc3f + .quad 0x0ff592d94dd2f8f4 + .quad 0x1d03620fe08cd37d + .quad 0xc20232cda2b4e554 + .quad 0x9ed0fd42115d187f + .quad 0x2eabb4be7dd479d9 + .quad 0x02c70bf52b68ec4c + + // 2^224 * 7 * G + + .quad 0xa287ec4b5d0b2fbb + .quad 0x415c5790074882ca + .quad 0xe044a61ec1d0815c + .quad 0x26334f0a409ef5e0 + .quad 0xace532bf458d72e1 + .quad 0x5be768e07cb73cb5 + .quad 0x56cf7d94ee8bbde7 + .quad 0x6b0697e3feb43a03 + .quad 0xb6c8f04adf62a3c0 + .quad 0x3ef000ef076da45d + .quad 0x9c9cb95849f0d2a9 + .quad 0x1cc37f43441b2fae + + // 2^224 * 8 * G + + .quad 0x508f565a5cc7324f + .quad 0xd061c4c0e506a922 + .quad 0xfb18abdb5c45ac19 + .quad 0x6c6809c10380314a + .quad 0xd76656f1c9ceaeb9 + .quad 0x1c5b15f818e5656a + .quad 0x26e72832844c2334 + .quad 0x3a346f772f196838 + .quad 0xd2d55112e2da6ac8 + .quad 0xe9bd0331b1e851ed + .quad 0x960746dd8ec67262 + .quad 0x05911b9f6ef7c5d0 + + // 2^228 * 1 * G + + .quad 0xe9dcd756b637ff2d + .quad 0xec4c348fc987f0c4 + .quad 0xced59285f3fbc7b7 + .quad 0x3305354793e1ea87 + .quad 0x01c18980c5fe9f94 + .quad 0xcd656769716fd5c8 + .quad 0x816045c3d195a086 + .quad 0x6e2b7f3266cc7982 + .quad 0xcc802468f7c3568f + .quad 0x9de9ba8219974cb3 + .quad 0xabb7229cb5b81360 + .quad 0x44e2017a6fbeba62 + + // 2^228 * 2 * G + + .quad 0xc4c2a74354dab774 + .quad 0x8e5d4c3c4eaf031a + .quad 0xb76c23d242838f17 + .quad 0x749a098f68dce4ea + .quad 0x87f82cf3b6ca6ecd + .quad 0x580f893e18f4a0c2 + .quad 0x058930072604e557 + .quad 0x6cab6ac256d19c1d + .quad 0xdcdfe0a02cc1de60 + .quad 0x032665ff51c5575b + .quad 0x2c0c32f1073abeeb + .quad 0x6a882014cd7b8606 + + // 2^228 * 3 * G + + .quad 0xa52a92fea4747fb5 + .quad 0xdc12a4491fa5ab89 + .quad 0xd82da94bb847a4ce + .quad 0x4d77edce9512cc4e + .quad 0xd111d17caf4feb6e + .quad 0x050bba42b33aa4a3 + .quad 0x17514c3ceeb46c30 + .quad 0x54bedb8b1bc27d75 + .quad 0x77c8e14577e2189c + .quad 0xa3e46f6aff99c445 + .quad 0x3144dfc86d335343 + .quad 0x3a96559e7c4216a9 + + // 2^228 * 4 * G + + .quad 0x12550d37f42ad2ee + .quad 0x8b78e00498a1fbf5 + .quad 0x5d53078233894cb2 + .quad 0x02c84e4e3e498d0c + .quad 0x4493896880baaa52 + .quad 0x4c98afc4f285940e + .quad 0xef4aa79ba45448b6 + .quad 0x5278c510a57aae7f + .quad 0xa54dd074294c0b94 + .quad 0xf55d46b8df18ffb6 + .quad 0xf06fecc58dae8366 + .quad 0x588657668190d165 + + // 2^228 * 5 * G + + .quad 0xd47712311aef7117 + .quad 0x50343101229e92c7 + .quad 0x7a95e1849d159b97 + .quad 0x2449959b8b5d29c9 + .quad 0xbf5834f03de25cc3 + .quad 0xb887c8aed6815496 + .quad 0x5105221a9481e892 + .quad 0x6760ed19f7723f93 + .quad 0x669ba3b7ac35e160 + .quad 0x2eccf73fba842056 + .quad 0x1aec1f17c0804f07 + .quad 0x0d96bc031856f4e7 + + // 2^228 * 6 * G + + .quad 0x3318be7775c52d82 + .quad 0x4cb764b554d0aab9 + .quad 0xabcf3d27cc773d91 + .quad 0x3bf4d1848123288a + .quad 0xb1d534b0cc7505e1 + .quad 0x32cd003416c35288 + .quad 0xcb36a5800762c29d + .quad 0x5bfe69b9237a0bf8 + .quad 0x183eab7e78a151ab + .quad 0xbbe990c999093763 + .quad 0xff717d6e4ac7e335 + .quad 0x4c5cddb325f39f88 + + // 2^228 * 7 * G + + .quad 0xc0f6b74d6190a6eb + .quad 0x20ea81a42db8f4e4 + .quad 0xa8bd6f7d97315760 + .quad 0x33b1d60262ac7c21 + .quad 0x57750967e7a9f902 + .quad 0x2c37fdfc4f5b467e + .quad 0xb261663a3177ba46 + .quad 0x3a375e78dc2d532b + .quad 0x8141e72f2d4dddea + .quad 0xe6eafe9862c607c8 + .quad 0x23c28458573cafd0 + .quad 0x46b9476f4ff97346 + + // 2^228 * 8 * G + + .quad 0x0c1ffea44f901e5c + .quad 0x2b0b6fb72184b782 + .quad 0xe587ff910114db88 + .quad 0x37130f364785a142 + .quad 0x1215505c0d58359f + .quad 0x2a2013c7fc28c46b + .quad 0x24a0a1af89ea664e + .quad 0x4400b638a1130e1f + .quad 0x3a01b76496ed19c3 + .quad 0x31e00ab0ed327230 + .quad 0x520a885783ca15b1 + .quad 0x06aab9875accbec7 + + // 2^232 * 1 * G + + .quad 0xc1339983f5df0ebb + .quad 0xc0f3758f512c4cac + .quad 0x2cf1130a0bb398e1 + .quad 0x6b3cecf9aa270c62 + .quad 0x5349acf3512eeaef + .quad 0x20c141d31cc1cb49 + .quad 0x24180c07a99a688d + .quad 0x555ef9d1c64b2d17 + .quad 0x36a770ba3b73bd08 + .quad 0x624aef08a3afbf0c + .quad 0x5737ff98b40946f2 + .quad 0x675f4de13381749d + + // 2^232 * 2 * G + + .quad 0x0e2c52036b1782fc + .quad 0x64816c816cad83b4 + .quad 0xd0dcbdd96964073e + .quad 0x13d99df70164c520 + .quad 0xa12ff6d93bdab31d + .quad 0x0725d80f9d652dfe + .quad 0x019c4ff39abe9487 + .quad 0x60f450b882cd3c43 + .quad 0x014b5ec321e5c0ca + .quad 0x4fcb69c9d719bfa2 + .quad 0x4e5f1c18750023a0 + .quad 0x1c06de9e55edac80 + + // 2^232 * 3 * G + + .quad 0x990f7ad6a33ec4e2 + .quad 0x6608f938be2ee08e + .quad 0x9ca143c563284515 + .quad 0x4cf38a1fec2db60d + .quad 0xffd52b40ff6d69aa + .quad 0x34530b18dc4049bb + .quad 0x5e4a5c2fa34d9897 + .quad 0x78096f8e7d32ba2d + .quad 0xa0aaaa650dfa5ce7 + .quad 0xf9c49e2a48b5478c + .quad 0x4f09cc7d7003725b + .quad 0x373cad3a26091abe + + // 2^232 * 4 * G + + .quad 0xb294634d82c9f57c + .quad 0x1fcbfde124934536 + .quad 0x9e9c4db3418cdb5a + .quad 0x0040f3d9454419fc + .quad 0xf1bea8fb89ddbbad + .quad 0x3bcb2cbc61aeaecb + .quad 0x8f58a7bb1f9b8d9d + .quad 0x21547eda5112a686 + .quad 0xdefde939fd5986d3 + .quad 0xf4272c89510a380c + .quad 0xb72ba407bb3119b9 + .quad 0x63550a334a254df4 + + // 2^232 * 5 * G + + .quad 0x6507d6edb569cf37 + .quad 0x178429b00ca52ee1 + .quad 0xea7c0090eb6bd65d + .quad 0x3eea62c7daf78f51 + .quad 0x9bba584572547b49 + .quad 0xf305c6fae2c408e0 + .quad 0x60e8fa69c734f18d + .quad 0x39a92bafaa7d767a + .quad 0x9d24c713e693274e + .quad 0x5f63857768dbd375 + .quad 0x70525560eb8ab39a + .quad 0x68436a0665c9c4cd + + // 2^232 * 6 * G + + .quad 0xbc0235e8202f3f27 + .quad 0xc75c00e264f975b0 + .quad 0x91a4e9d5a38c2416 + .quad 0x17b6e7f68ab789f9 + .quad 0x1e56d317e820107c + .quad 0xc5266844840ae965 + .quad 0xc1e0a1c6320ffc7a + .quad 0x5373669c91611472 + .quad 0x5d2814ab9a0e5257 + .quad 0x908f2084c9cab3fc + .quad 0xafcaf5885b2d1eca + .quad 0x1cb4b5a678f87d11 + + // 2^232 * 7 * G + + .quad 0xb664c06b394afc6c + .quad 0x0c88de2498da5fb1 + .quad 0x4f8d03164bcad834 + .quad 0x330bca78de7434a2 + .quad 0x6b74aa62a2a007e7 + .quad 0xf311e0b0f071c7b1 + .quad 0x5707e438000be223 + .quad 0x2dc0fd2d82ef6eac + .quad 0x982eff841119744e + .quad 0xf9695e962b074724 + .quad 0xc58ac14fbfc953fb + .quad 0x3c31be1b369f1cf5 + + // 2^232 * 8 * G + + .quad 0xb0f4864d08948aee + .quad 0x07dc19ee91ba1c6f + .quad 0x7975cdaea6aca158 + .quad 0x330b61134262d4bb + .quad 0xc168bc93f9cb4272 + .quad 0xaeb8711fc7cedb98 + .quad 0x7f0e52aa34ac8d7a + .quad 0x41cec1097e7d55bb + .quad 0xf79619d7a26d808a + .quad 0xbb1fd49e1d9e156d + .quad 0x73d7c36cdba1df27 + .quad 0x26b44cd91f28777d + + // 2^236 * 1 * G + + .quad 0x300a9035393aa6d8 + .quad 0x2b501131a12bb1cd + .quad 0x7b1ff677f093c222 + .quad 0x4309c1f8cab82bad + .quad 0xaf44842db0285f37 + .quad 0x8753189047efc8df + .quad 0x9574e091f820979a + .quad 0x0e378d6069615579 + .quad 0xd9fa917183075a55 + .quad 0x4bdb5ad26b009fdc + .quad 0x7829ad2cd63def0e + .quad 0x078fc54975fd3877 + + // 2^236 * 2 * G + + .quad 0x87dfbd1428878f2d + .quad 0x134636dd1e9421a1 + .quad 0x4f17c951257341a3 + .quad 0x5df98d4bad296cb8 + .quad 0xe2004b5bb833a98a + .quad 0x44775dec2d4c3330 + .quad 0x3aa244067eace913 + .quad 0x272630e3d58e00a9 + .quad 0xf3678fd0ecc90b54 + .quad 0xf001459b12043599 + .quad 0x26725fbc3758b89b + .quad 0x4325e4aa73a719ae + + // 2^236 * 3 * G + + .quad 0x657dc6ef433c3493 + .quad 0x65375e9f80dbf8c3 + .quad 0x47fd2d465b372dae + .quad 0x4966ab79796e7947 + .quad 0xed24629acf69f59d + .quad 0x2a4a1ccedd5abbf4 + .quad 0x3535ca1f56b2d67b + .quad 0x5d8c68d043b1b42d + .quad 0xee332d4de3b42b0a + .quad 0xd84e5a2b16a4601c + .quad 0x78243877078ba3e4 + .quad 0x77ed1eb4184ee437 + + // 2^236 * 4 * G + + .quad 0xbfd4e13f201839a0 + .quad 0xaeefffe23e3df161 + .quad 0xb65b04f06b5d1fe3 + .quad 0x52e085fb2b62fbc0 + .quad 0x185d43f89e92ed1a + .quad 0xb04a1eeafe4719c6 + .quad 0x499fbe88a6f03f4f + .quad 0x5d8b0d2f3c859bdd + .quad 0x124079eaa54cf2ba + .quad 0xd72465eb001b26e7 + .quad 0x6843bcfdc97af7fd + .quad 0x0524b42b55eacd02 + + // 2^236 * 5 * G + + .quad 0xfd0d5dbee45447b0 + .quad 0x6cec351a092005ee + .quad 0x99a47844567579cb + .quad 0x59d242a216e7fa45 + .quad 0xbc18dcad9b829eac + .quad 0x23ae7d28b5f579d0 + .quad 0xc346122a69384233 + .quad 0x1a6110b2e7d4ac89 + .quad 0x4f833f6ae66997ac + .quad 0x6849762a361839a4 + .quad 0x6985dec1970ab525 + .quad 0x53045e89dcb1f546 + + // 2^236 * 6 * G + + .quad 0xcb8bb346d75353db + .quad 0xfcfcb24bae511e22 + .quad 0xcba48d40d50ae6ef + .quad 0x26e3bae5f4f7cb5d + .quad 0x84da3cde8d45fe12 + .quad 0xbd42c218e444e2d2 + .quad 0xa85196781f7e3598 + .quad 0x7642c93f5616e2b2 + .quad 0x2323daa74595f8e4 + .quad 0xde688c8b857abeb4 + .quad 0x3fc48e961c59326e + .quad 0x0b2e73ca15c9b8ba + + // 2^236 * 7 * G + + .quad 0xd6bb4428c17f5026 + .quad 0x9eb27223fb5a9ca7 + .quad 0xe37ba5031919c644 + .quad 0x21ce380db59a6602 + .quad 0x0e3fbfaf79c03a55 + .quad 0x3077af054cbb5acf + .quad 0xd5c55245db3de39f + .quad 0x015e68c1476a4af7 + .quad 0xc1d5285220066a38 + .quad 0x95603e523570aef3 + .quad 0x832659a7226b8a4d + .quad 0x5dd689091f8eedc9 + + // 2^236 * 8 * G + + .quad 0xcbac84debfd3c856 + .quad 0x1624c348b35ff244 + .quad 0xb7f88dca5d9cad07 + .quad 0x3b0e574da2c2ebe8 + .quad 0x1d022591a5313084 + .quad 0xca2d4aaed6270872 + .quad 0x86a12b852f0bfd20 + .quad 0x56e6c439ad7da748 + .quad 0xc704ff4942bdbae6 + .quad 0x5e21ade2b2de1f79 + .quad 0xe95db3f35652fad8 + .quad 0x0822b5378f08ebc1 + + // 2^240 * 1 * G + + .quad 0x51f048478f387475 + .quad 0xb25dbcf49cbecb3c + .quad 0x9aab1244d99f2055 + .quad 0x2c709e6c1c10a5d6 + .quad 0xe1b7f29362730383 + .quad 0x4b5279ffebca8a2c + .quad 0xdafc778abfd41314 + .quad 0x7deb10149c72610f + .quad 0xcb62af6a8766ee7a + .quad 0x66cbec045553cd0e + .quad 0x588001380f0be4b5 + .quad 0x08e68e9ff62ce2ea + + // 2^240 * 2 * G + + .quad 0x34ad500a4bc130ad + .quad 0x8d38db493d0bd49c + .quad 0xa25c3d98500a89be + .quad 0x2f1f3f87eeba3b09 + .quad 0x2f2d09d50ab8f2f9 + .quad 0xacb9218dc55923df + .quad 0x4a8f342673766cb9 + .quad 0x4cb13bd738f719f5 + .quad 0xf7848c75e515b64a + .quad 0xa59501badb4a9038 + .quad 0xc20d313f3f751b50 + .quad 0x19a1e353c0ae2ee8 + + // 2^240 * 3 * G + + .quad 0x7d1c7560bafa05c3 + .quad 0xb3e1a0a0c6e55e61 + .quad 0xe3529718c0d66473 + .quad 0x41546b11c20c3486 + .quad 0xb42172cdd596bdbd + .quad 0x93e0454398eefc40 + .quad 0x9fb15347b44109b5 + .quad 0x736bd3990266ae34 + .quad 0x85532d509334b3b4 + .quad 0x46fd114b60816573 + .quad 0xcc5f5f30425c8375 + .quad 0x412295a2b87fab5c + + // 2^240 * 4 * G + + .quad 0x19c99b88f57ed6e9 + .quad 0x5393cb266df8c825 + .quad 0x5cee3213b30ad273 + .quad 0x14e153ebb52d2e34 + .quad 0x2e655261e293eac6 + .quad 0x845a92032133acdb + .quad 0x460975cb7900996b + .quad 0x0760bb8d195add80 + .quad 0x413e1a17cde6818a + .quad 0x57156da9ed69a084 + .quad 0x2cbf268f46caccb1 + .quad 0x6b34be9bc33ac5f2 + + // 2^240 * 5 * G + + .quad 0xf3df2f643a78c0b2 + .quad 0x4c3e971ef22e027c + .quad 0xec7d1c5e49c1b5a3 + .quad 0x2012c18f0922dd2d + .quad 0x11fc69656571f2d3 + .quad 0xc6c9e845530e737a + .quad 0xe33ae7a2d4fe5035 + .quad 0x01b9c7b62e6dd30b + .quad 0x880b55e55ac89d29 + .quad 0x1483241f45a0a763 + .quad 0x3d36efdfc2e76c1f + .quad 0x08af5b784e4bade8 + + // 2^240 * 6 * G + + .quad 0x283499dc881f2533 + .quad 0x9d0525da779323b6 + .quad 0x897addfb673441f4 + .quad 0x32b79d71163a168d + .quad 0xe27314d289cc2c4b + .quad 0x4be4bd11a287178d + .quad 0x18d528d6fa3364ce + .quad 0x6423c1d5afd9826e + .quad 0xcc85f8d9edfcb36a + .quad 0x22bcc28f3746e5f9 + .quad 0xe49de338f9e5d3cd + .quad 0x480a5efbc13e2dcc + + // 2^240 * 7 * G + + .quad 0x0b51e70b01622071 + .quad 0x06b505cf8b1dafc5 + .quad 0x2c6bb061ef5aabcd + .quad 0x47aa27600cb7bf31 + .quad 0xb6614ce442ce221f + .quad 0x6e199dcc4c053928 + .quad 0x663fb4a4dc1cbe03 + .quad 0x24b31d47691c8e06 + .quad 0x2a541eedc015f8c3 + .quad 0x11a4fe7e7c693f7c + .quad 0xf0af66134ea278d6 + .quad 0x545b585d14dda094 + + // 2^240 * 8 * G + + .quad 0x67bf275ea0d43a0f + .quad 0xade68e34089beebe + .quad 0x4289134cd479e72e + .quad 0x0f62f9c332ba5454 + .quad 0x6204e4d0e3b321e1 + .quad 0x3baa637a28ff1e95 + .quad 0x0b0ccffd5b99bd9e + .quad 0x4d22dc3e64c8d071 + .quad 0xfcb46589d63b5f39 + .quad 0x5cae6a3f57cbcf61 + .quad 0xfebac2d2953afa05 + .quad 0x1c0fa01a36371436 + + // 2^244 * 1 * G + + .quad 0xe7547449bc7cd692 + .quad 0x0f9abeaae6f73ddf + .quad 0x4af01ca700837e29 + .quad 0x63ab1b5d3f1bc183 + .quad 0xc11ee5e854c53fae + .quad 0x6a0b06c12b4f3ff4 + .quad 0x33540f80e0b67a72 + .quad 0x15f18fc3cd07e3ef + .quad 0x32750763b028f48c + .quad 0x06020740556a065f + .quad 0xd53bd812c3495b58 + .quad 0x08706c9b865f508d + + // 2^244 * 2 * G + + .quad 0xf37ca2ab3d343dff + .quad 0x1a8c6a2d80abc617 + .quad 0x8e49e035d4ccffca + .quad 0x48b46beebaa1d1b9 + .quad 0xcc991b4138b41246 + .quad 0x243b9c526f9ac26b + .quad 0xb9ef494db7cbabbd + .quad 0x5fba433dd082ed00 + .quad 0x9c49e355c9941ad0 + .quad 0xb9734ade74498f84 + .quad 0x41c3fed066663e5c + .quad 0x0ecfedf8e8e710b3 + + // 2^244 * 3 * G + + .quad 0x76430f9f9cd470d9 + .quad 0xb62acc9ba42f6008 + .quad 0x1898297c59adad5e + .quad 0x7789dd2db78c5080 + .quad 0x744f7463e9403762 + .quad 0xf79a8dee8dfcc9c9 + .quad 0x163a649655e4cde3 + .quad 0x3b61788db284f435 + .quad 0xb22228190d6ef6b2 + .quad 0xa94a66b246ce4bfa + .quad 0x46c1a77a4f0b6cc7 + .quad 0x4236ccffeb7338cf + + // 2^244 * 4 * G + + .quad 0x8497404d0d55e274 + .quad 0x6c6663d9c4ad2b53 + .quad 0xec2fb0d9ada95734 + .quad 0x2617e120cdb8f73c + .quad 0x3bd82dbfda777df6 + .quad 0x71b177cc0b98369e + .quad 0x1d0e8463850c3699 + .quad 0x5a71945b48e2d1f1 + .quad 0x6f203dd5405b4b42 + .quad 0x327ec60410b24509 + .quad 0x9c347230ac2a8846 + .quad 0x77de29fc11ffeb6a + + // 2^244 * 5 * G + + .quad 0xb0ac57c983b778a8 + .quad 0x53cdcca9d7fe912c + .quad 0x61c2b854ff1f59dc + .quad 0x3a1a2cf0f0de7dac + .quad 0x835e138fecced2ca + .quad 0x8c9eaf13ea963b9a + .quad 0xc95fbfc0b2160ea6 + .quad 0x575e66f3ad877892 + .quad 0x99803a27c88fcb3a + .quad 0x345a6789275ec0b0 + .quad 0x459789d0ff6c2be5 + .quad 0x62f882651e70a8b2 + + // 2^244 * 6 * G + + .quad 0x085ae2c759ff1be4 + .quad 0x149145c93b0e40b7 + .quad 0xc467e7fa7ff27379 + .quad 0x4eeecf0ad5c73a95 + .quad 0x6d822986698a19e0 + .quad 0xdc9821e174d78a71 + .quad 0x41a85f31f6cb1f47 + .quad 0x352721c2bcda9c51 + .quad 0x48329952213fc985 + .quad 0x1087cf0d368a1746 + .quad 0x8e5261b166c15aa5 + .quad 0x2d5b2d842ed24c21 + + // 2^244 * 7 * G + + .quad 0x02cfebd9ebd3ded1 + .quad 0xd45b217739021974 + .quad 0x7576f813fe30a1b7 + .quad 0x5691b6f9a34ef6c2 + .quad 0x5eb7d13d196ac533 + .quad 0x377234ecdb80be2b + .quad 0xe144cffc7cf5ae24 + .quad 0x5226bcf9c441acec + .quad 0x79ee6c7223e5b547 + .quad 0x6f5f50768330d679 + .quad 0xed73e1e96d8adce9 + .quad 0x27c3da1e1d8ccc03 + + // 2^244 * 8 * G + + .quad 0x7eb9efb23fe24c74 + .quad 0x3e50f49f1651be01 + .quad 0x3ea732dc21858dea + .quad 0x17377bd75bb810f9 + .quad 0x28302e71630ef9f6 + .quad 0xc2d4a2032b64cee0 + .quad 0x090820304b6292be + .quad 0x5fca747aa82adf18 + .quad 0x232a03c35c258ea5 + .quad 0x86f23a2c6bcb0cf1 + .quad 0x3dad8d0d2e442166 + .quad 0x04a8933cab76862b + + // 2^248 * 1 * G + + .quad 0xd2c604b622943dff + .quad 0xbc8cbece44cfb3a0 + .quad 0x5d254ff397808678 + .quad 0x0fa3614f3b1ca6bf + .quad 0x69082b0e8c936a50 + .quad 0xf9c9a035c1dac5b6 + .quad 0x6fb73e54c4dfb634 + .quad 0x4005419b1d2bc140 + .quad 0xa003febdb9be82f0 + .quad 0x2089c1af3a44ac90 + .quad 0xf8499f911954fa8e + .quad 0x1fba218aef40ab42 + + // 2^248 * 2 * G + + .quad 0xab549448fac8f53e + .quad 0x81f6e89a7ba63741 + .quad 0x74fd6c7d6c2b5e01 + .quad 0x392e3acaa8c86e42 + .quad 0x4f3e57043e7b0194 + .quad 0xa81d3eee08daaf7f + .quad 0xc839c6ab99dcdef1 + .quad 0x6c535d13ff7761d5 + .quad 0x4cbd34e93e8a35af + .quad 0x2e0781445887e816 + .quad 0x19319c76f29ab0ab + .quad 0x25e17fe4d50ac13b + + // 2^248 * 3 * G + + .quad 0x0a289bd71e04f676 + .quad 0x208e1c52d6420f95 + .quad 0x5186d8b034691fab + .quad 0x255751442a9fb351 + .quad 0x915f7ff576f121a7 + .quad 0xc34a32272fcd87e3 + .quad 0xccba2fde4d1be526 + .quad 0x6bba828f8969899b + .quad 0xe2d1bc6690fe3901 + .quad 0x4cb54a18a0997ad5 + .quad 0x971d6914af8460d4 + .quad 0x559d504f7f6b7be4 + + // 2^248 * 4 * G + + .quad 0xa7738378b3eb54d5 + .quad 0x1d69d366a5553c7c + .quad 0x0a26cf62f92800ba + .quad 0x01ab12d5807e3217 + .quad 0x9c4891e7f6d266fd + .quad 0x0744a19b0307781b + .quad 0x88388f1d6061e23b + .quad 0x123ea6a3354bd50e + .quad 0x118d189041e32d96 + .quad 0xb9ede3c2d8315848 + .quad 0x1eab4271d83245d9 + .quad 0x4a3961e2c918a154 + + // 2^248 * 5 * G + + .quad 0x71dc3be0f8e6bba0 + .quad 0xd6cef8347effe30a + .quad 0xa992425fe13a476a + .quad 0x2cd6bce3fb1db763 + .quad 0x0327d644f3233f1e + .quad 0x499a260e34fcf016 + .quad 0x83b5a716f2dab979 + .quad 0x68aceead9bd4111f + .quad 0x38b4c90ef3d7c210 + .quad 0x308e6e24b7ad040c + .quad 0x3860d9f1b7e73e23 + .quad 0x595760d5b508f597 + + // 2^248 * 6 * G + + .quad 0x6129bfe104aa6397 + .quad 0x8f960008a4a7fccb + .quad 0x3f8bc0897d909458 + .quad 0x709fa43edcb291a9 + .quad 0x882acbebfd022790 + .quad 0x89af3305c4115760 + .quad 0x65f492e37d3473f4 + .quad 0x2cb2c5df54515a2b + .quad 0xeb0a5d8c63fd2aca + .quad 0xd22bc1662e694eff + .quad 0x2723f36ef8cbb03a + .quad 0x70f029ecf0c8131f + + // 2^248 * 7 * G + + .quad 0x461307b32eed3e33 + .quad 0xae042f33a45581e7 + .quad 0xc94449d3195f0366 + .quad 0x0b7d5d8a6c314858 + .quad 0x2a6aafaa5e10b0b9 + .quad 0x78f0a370ef041aa9 + .quad 0x773efb77aa3ad61f + .quad 0x44eca5a2a74bd9e1 + .quad 0x25d448327b95d543 + .quad 0x70d38300a3340f1d + .quad 0xde1c531c60e1c52b + .quad 0x272224512c7de9e4 + + // 2^248 * 8 * G + + .quad 0x1abc92af49c5342e + .quad 0xffeed811b2e6fad0 + .quad 0xefa28c8dfcc84e29 + .quad 0x11b5df18a44cc543 + .quad 0xbf7bbb8a42a975fc + .quad 0x8c5c397796ada358 + .quad 0xe27fc76fcdedaa48 + .quad 0x19735fd7f6bc20a6 + .quad 0xe3ab90d042c84266 + .quad 0xeb848e0f7f19547e + .quad 0x2503a1d065a497b9 + .quad 0x0fef911191df895f diff --git a/cbits/s2n/x86_att/edwards25519_scalarmulbase_alt.S b/cbits/s2n/x86_att/edwards25519_scalarmulbase_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/edwards25519_scalarmulbase_alt.S @@ -0,0 +1,10002 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Scalar multiplication for the edwards25519 standard basepoint +// Input scalar[4]; output res[8] +// +// extern void edwards25519_scalarmulbase_alt +// (uint64_t res[static 8],const uint64_t scalar[static 4]); +// +// Given a scalar n, returns point (X,Y) = n * B where B = (...,4/5) is +// the standard basepoint for the edwards25519 (Ed25519) curve. +// +// Standard x86-64 ABI: RDI = res, RSI = scalar +// Microsoft x64 ABI: RCX = res, RDX = scalar +// ---------------------------------------------------------------------------- +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(edwards25519_scalarmulbase_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(edwards25519_scalarmulbase_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(edwards25519_scalarmulbase_alt) + + .text + .balign 32 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Pointer-offset pairs for result and temporaries on stack with some aliasing. +// The result "resx" assumes the "res" pointer has been preloaded into %rbp. + +#define resx (0*NUMSIZE)(%rbp) +#define resy (1*NUMSIZE)(%rbp) + +#define scalar (0*NUMSIZE)(%rsp) + +#define tabent (1*NUMSIZE)(%rsp) +#define ymx_2 (1*NUMSIZE)(%rsp) +#define xpy_2 (2*NUMSIZE)(%rsp) +#define kxy_2 (3*NUMSIZE)(%rsp) + +#define t0 (4*NUMSIZE)(%rsp) +#define t1 (5*NUMSIZE)(%rsp) +#define t2 (6*NUMSIZE)(%rsp) +#define t3 (7*NUMSIZE)(%rsp) +#define t4 (8*NUMSIZE)(%rsp) +#define t5 (9*NUMSIZE)(%rsp) + +#define acc (10*NUMSIZE)(%rsp) +#define x_1 (10*NUMSIZE)(%rsp) +#define y_1 (11*NUMSIZE)(%rsp) +#define z_1 (12*NUMSIZE)(%rsp) +#define w_1 (13*NUMSIZE)(%rsp) +#define x_3 (10*NUMSIZE)(%rsp) +#define y_3 (11*NUMSIZE)(%rsp) +#define z_3 (12*NUMSIZE)(%rsp) +#define w_3 (13*NUMSIZE)(%rsp) + +// Stable homes for the input result pointer, and other variables + +#define res 14*NUMSIZE(%rsp) + +#define i 14*NUMSIZE+8(%rsp) + +#define bias 14*NUMSIZE+16(%rsp) + +#define bf 14*NUMSIZE+24(%rsp) +#define ix 14*NUMSIZE+24(%rsp) + +#define tab 15*NUMSIZE(%rsp) + +// Total size to reserve on the stack + +#define NSPACE 488 + +// Syntactic variants to make x86_att version simpler to generate + +#define SCALAR 0 +#define TABENT (1*NUMSIZE) +#define ACC (10*NUMSIZE) +#define X3 (10*NUMSIZE) +#define Z3 (12*NUMSIZE) +#define W3 (13*NUMSIZE) + +// Macro wrapping up the basic field multiplication, only trivially +// different from a pure function call to bignum_mul_p25519_alt. + +#define mul_p25519(P0,P1,P2) \ + movq P1, %rax ; \ + mulq P2; \ + movq %rax, %r8 ; \ + movq %rdx, %r9 ; \ + xorq %r10, %r10 ; \ + xorq %r11, %r11 ; \ + movq P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + movq 0x8+P1, %rax ; \ + mulq P2; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + adcq $0x0, %r11 ; \ + xorq %r12, %r12 ; \ + movq P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq %r12, %r12 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + movq 0x10+P1, %rax ; \ + mulq P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + xorq %r13, %r13 ; \ + movq P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq %r13, %r13 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + movq 0x18+P1, %rax ; \ + mulq P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + xorq %r14, %r14 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq %r14, %r14 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + xorq %r15, %r15 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq %r15, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq $0x0, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + movl $0x26, %esi ; \ + movq %r12, %rax ; \ + mulq %rsi; \ + addq %rax, %r8 ; \ + adcq %rdx, %r9 ; \ + sbbq %rcx, %rcx ; \ + movq %r13, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %rcx, %rcx ; \ + movq %r14, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rcx, %rcx ; \ + movq %r15, %rax ; \ + mulq %rsi; \ + subq %rcx, %rdx ; \ + xorq %rcx, %rcx ; \ + addq %rax, %r11 ; \ + movq %rdx, %r12 ; \ + adcq %rcx, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + leaq 0x1(%r12), %rax ; \ + movl $0x13, %esi ; \ + bts $63, %r11 ; \ + imulq %rsi, %rax ; \ + addq %rax, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + sbbq %rax, %rax ; \ + notq %rax; \ + andq %rsi, %rax ; \ + subq %rax, %r8 ; \ + sbbq %rcx, %r9 ; \ + sbbq %rcx, %r10 ; \ + sbbq %rcx, %r11 ; \ + btr $63, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// A version of multiplication that only guarantees output < 2 * p_25519. +// This basically skips the +1 and final correction in quotient estimation. + +#define mul_4(P0,P1,P2) \ + movq P1, %rax ; \ + mulq P2; \ + movq %rax, %r8 ; \ + movq %rdx, %r9 ; \ + xorq %r10, %r10 ; \ + xorq %r11, %r11 ; \ + movq P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + movq 0x8+P1, %rax ; \ + mulq P2; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + adcq $0x0, %r11 ; \ + xorq %r12, %r12 ; \ + movq P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq %r12, %r12 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + movq 0x10+P1, %rax ; \ + mulq P2; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + adcq $0x0, %r12 ; \ + xorq %r13, %r13 ; \ + movq P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq %r13, %r13 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + movq 0x18+P1, %rax ; \ + mulq P2; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + adcq $0x0, %r13 ; \ + xorq %r14, %r14 ; \ + movq 0x8+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq %r14, %r14 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x8+P2; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + xorq %r15, %r15 ; \ + movq 0x10+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq %r15, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x10+P2; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq $0x0, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq 0x18+P2; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + movl $0x26, %ebx ; \ + movq %r12, %rax ; \ + mulq %rbx; \ + addq %rax, %r8 ; \ + adcq %rdx, %r9 ; \ + sbbq %rcx, %rcx ; \ + movq %r13, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %rcx, %rcx ; \ + movq %r14, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rcx, %rcx ; \ + movq %r15, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + xorq %rcx, %rcx ; \ + addq %rax, %r11 ; \ + movq %rdx, %r12 ; \ + adcq %rcx, %r12 ; \ + shldq $0x1, %r11, %r12 ; \ + btr $0x3f, %r11 ; \ + movl $0x13, %edx ; \ + imulq %r12, %rdx ; \ + addq %rdx, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +// Modular subtraction with double modulus 2 * p_25519 = 2^256 - 38 + +#define sub_twice4(P0,P1,P2) \ + movq P1, %r8 ; \ + xorl %ebx, %ebx ; \ + subq P2, %r8 ; \ + movq 8+P1, %r9 ; \ + sbbq 8+P2, %r9 ; \ + movl $38, %ecx ; \ + movq 16+P1, %r10 ; \ + sbbq 16+P2, %r10 ; \ + movq 24+P1, %rax ; \ + sbbq 24+P2, %rax ; \ + cmovncq %rbx, %rcx ; \ + subq %rcx, %r8 ; \ + sbbq %rbx, %r9 ; \ + sbbq %rbx, %r10 ; \ + sbbq %rbx, %rax ; \ + movq %r8, P0 ; \ + movq %r9, 8+P0 ; \ + movq %r10, 16+P0 ; \ + movq %rax, 24+P0 + +// Modular addition and doubling with double modulus 2 * p_25519 = 2^256 - 38. +// This only ensures that the result fits in 4 digits, not that it is reduced +// even w.r.t. double modulus. The result is always correct modulo provided +// the sum of the inputs is < 2^256 + 2^256 - 38, so in particular provided +// at least one of them is reduced double modulo. + +#define add_twice4(P0,P1,P2) \ + movq P1, %r8 ; \ + xorl %ecx, %ecx ; \ + addq P2, %r8 ; \ + movq 0x8+P1, %r9 ; \ + adcq 0x8+P2, %r9 ; \ + movq 0x10+P1, %r10 ; \ + adcq 0x10+P2, %r10 ; \ + movq 0x18+P1, %r11 ; \ + adcq 0x18+P2, %r11 ; \ + movl $38, %eax ; \ + cmovncq %rcx, %rax ; \ + addq %rax, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +#define double_twice4(P0,P1) \ + movq P1, %r8 ; \ + xorl %ecx, %ecx ; \ + addq %r8, %r8 ; \ + movq 0x8+P1, %r9 ; \ + adcq %r9, %r9 ; \ + movq 0x10+P1, %r10 ; \ + adcq %r10, %r10 ; \ + movq 0x18+P1, %r11 ; \ + adcq %r11, %r11 ; \ + movl $38, %eax ; \ + cmovncq %rcx, %rax ; \ + addq %rax, %r8 ; \ + adcq %rcx, %r9 ; \ + adcq %rcx, %r10 ; \ + adcq %rcx, %r11 ; \ + movq %r8, P0 ; \ + movq %r9, 0x8+P0 ; \ + movq %r10, 0x10+P0 ; \ + movq %r11, 0x18+P0 + +S2N_BN_SYMBOL(edwards25519_scalarmulbase_alt): + CFI_START + _CET_ENDBR + +// In this case the Windows form literally makes a subroutine call. +// This avoids hassle arising from keeping code and data together. + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + CFI_CALL(Ledwards25519_scalarmulbase_alt_standard) + CFI_POP(%rsi) + CFI_POP(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(edwards25519_scalarmulbase_alt) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Ledwards25519_scalarmulbase_alt_standard) + +Ledwards25519_scalarmulbase_alt_standard: + CFI_START +#endif + +// Save registers, make room for temps, preserve input arguments. + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(NSPACE) + +// Move the output pointer to a stable place + + movq %rdi, res + +// Copy the input scalar x to its local variable while reducing it +// modulo 2^252 + m where m = 27742317777372353535851937790883648493; +// this is the order of the basepoint so this doesn't change the result. +// First do q = floor(x/2^252) and x' = x - q * (2^252 + m), which gives +// an initial result -15 * m <= x' < 2^252 + + movq (%rsi), %r8 + movq 8(%rsi), %r9 + movq 16(%rsi), %r10 + movq 24(%rsi), %r11 + + movq %r11, %rcx + shrq $60, %rcx + + movq $0x5812631a5cf5d3ed, %rax + mulq %rcx + movq %rax, %r12 + movq %rdx, %r13 + movq $0x14def9dea2f79cd6, %rax + mulq %rcx + addq %rax, %r13 + adcq $0, %rdx + shlq $60, %rcx + + subq %r12, %r8 + sbbq %r13, %r9 + sbbq %rdx, %r10 + sbbq %rcx, %r11 + +// If x' < 0 then just directly negate it; this makes sure the +// reduced argument is strictly 0 <= x' < 2^252, but now we need +// to record (done via bit 255 of the reduced scalar, which is +// ignored in the main loop) when we negated so we can flip +// the end result to compensate. + + sbbq %rax, %rax + + xorq %rax, %r8 + xorq %rax, %r9 + xorq %rax, %r10 + xorq %rax, %r11 + + negq %rax + adcq $0, %r8 + adcq $0, %r9 + adcq $0, %r10 + adcq $0, %r11 + + shlq $63, %rax + orq %rax, %r11 + +// And before we store the scalar, test and reset bit 251 to +// initialize the main loop just below. + + movq %r8, SCALAR(%rsp) + movq %r9, SCALAR+8(%rsp) + movq %r10, SCALAR+16(%rsp) + btr $59, %r11 + movq %r11, SCALAR+24(%rsp) + +// The main part of the computation is in extended-projective coordinates +// (X,Y,Z,T), representing an affine point on the edwards25519 curve +// (x,y) via x = X/Z, y = Y/Z and x * y = T/Z (so X * Y = T * Z). +// In comments B means the standard basepoint (x,4/5) = +// (0x216....f25d51a,0x6666..666658). +// +// Initialize accumulator "acc" to either 0 or 2^251 * B depending on +// bit 251 of the (reduced) scalar. That leaves bits 0..250 to handle. + + leaq S2N_BN_SYMBOL(edwards25519_scalarmulbase_alt_constant)(%rip), %r10 + leaq 8*12(%r10), %r11 + + movq (%r10), %rax + movq (%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC(%rsp) + + movq 8*1(%r10), %rax + movq 8*1(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+8(%rsp) + + movq 8*2(%r10), %rax + movq 8*2(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+16(%rsp) + + movq 8*3(%r10), %rax + movq 8*3(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+24(%rsp) + + movq 8*4(%r10), %rax + movq 8*4(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+32(%rsp) + + movq 8*5(%r10), %rax + movq 8*5(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+40(%rsp) + + movq 8*6(%r10), %rax + movq 8*6(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+48(%rsp) + + movq 8*7(%r10), %rax + movq 8*7(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+56(%rsp) + + movl $1, %eax + movq %rax, ACC+64(%rsp) + movl $0, %eax + movq %rax, ACC+72(%rsp) + movq %rax, ACC+80(%rsp) + movq %rax, ACC+88(%rsp) + + movq 8*8(%r10), %rax + movq 8*8(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+96(%rsp) + + movq 8*9(%r10), %rax + movq 8*9(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+104(%rsp) + + movq 8*10(%r10), %rax + movq 8*10(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+112(%rsp) + + movq 8*11(%r10), %rax + movq 8*11(%r11), %rcx + cmovcq %rcx, %rax + movq %rax, ACC+120(%rsp) + +// The counter "i" tracks the bit position for which the scalar has +// already been absorbed, starting at 0 and going up in chunks of 4. +// +// The pointer "tab" points at the current block of the table for +// multiples (2^i * j) * B at the current bit position i; 1 <= j <= 8. +// +// The bias is always either 0 and 1 and needs to be added to the +// partially processed scalar implicitly. This is used to absorb 4 bits +// of scalar per iteration from 3-bit table indexing by exploiting +// negation: (16 * h + l) * B = (16 * (h + 1) - (16 - l)) * B is used +// when l >= 9. Note that we can't have any bias left over at the +// end because we made sure bit 251 is clear in the reduced scalar. + + movq $0, i + leaq 8*24(%r10), %rax + movq %rax, tab + movq $0, bias + +// Start of the main loop, repeated 63 times for i = 4, 8, ..., 252 + +Ledwards25519_scalarmulbase_alt_scalarloop: + +// Look at the next 4-bit field "bf", adding the previous bias as well. +// Choose the table index "ix" as bf when bf <= 8 and 16 - bf for bf >= 9, +// setting the bias to 1 for the next iteration in the latter case. + + movq i, %rax + movq %rax, %rcx + shrq $6, %rax + movq (%rsp,%rax,8), %rax // Exploiting scalar = sp exactly + shrq %cl, %rax + andq $15, %rax + addq bias, %rax + movq %rax, bf + + cmpq $9, bf + sbbq %rax, %rax + incq %rax + movq %rax, bias + + movq $16, %rdi + subq bf, %rdi + cmpq $0, bias + cmovzq bf, %rdi + movq %rdi, ix + +// Perform constant-time lookup in the table to get element number "ix". +// The table entry for the affine point (x,y) is actually a triple +// (y - x,x + y,2 * d * x * y) to precompute parts of the addition. +// Note that "ix" can be 0, so we set up the appropriate identity first. + + movl $1, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + movl $1, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + xorl %r13d, %r13d + xorl %r14d, %r14d + xorl %r15d, %r15d + + movq tab, %rbp + + cmpq $1, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $2, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $3, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $4, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $5, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $6, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $7, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + addq $96, %rbp + + cmpq $8, ix + movq (%rbp), %rsi + cmovzq %rsi, %rax + movq 8(%rbp), %rsi + cmovzq %rsi, %rbx + movq 16(%rbp), %rsi + cmovzq %rsi, %rcx + movq 24(%rbp), %rsi + cmovzq %rsi, %rdx + movq 32(%rbp), %rsi + cmovzq %rsi, %r8 + movq 40(%rbp), %rsi + cmovzq %rsi, %r9 + movq 48(%rbp), %rsi + cmovzq %rsi, %r10 + movq 56(%rbp), %rsi + cmovzq %rsi, %r11 + movq 64(%rbp), %rsi + cmovzq %rsi, %r12 + movq 72(%rbp), %rsi + cmovzq %rsi, %r13 + movq 80(%rbp), %rsi + cmovzq %rsi, %r14 + movq 88(%rbp), %rsi + cmovzq %rsi, %r15 + + addq $96, %rbp + movq %rbp, tab + +// We now have the triple from the table in registers as follows +// +// [%rdx;%rcx;%rbx;%rax] = y - x +// [%r11;%r10;%r9;%r8] = x + y +// [%r15;%r14;%r13;%r12] = 2 * d * x * y +// +// In case bias = 1 we need to negate this. For Edwards curves +// -(x,y) = (-x,y), i.e. we need to negate the x coordinate. +// In this processed encoding, that amounts to swapping the +// first two fields and negating the third. +// +// The optional negation here also pretends bias = 0 whenever +// ix = 0 so that it doesn't need to handle the case of zero +// inputs, since no non-trivial table entries are zero. Note +// that in the zero case the whole negation is trivial, and +// so indeed is the swapping. + + cmpq $0, bias + + movq %rax, %rsi + cmovnzq %r8, %rsi + cmovnzq %rax, %r8 + movq %rsi, TABENT(%rsp) + movq %r8, TABENT+32(%rsp) + + movq %rbx, %rsi + cmovnzq %r9, %rsi + cmovnzq %rbx, %r9 + movq %rsi, TABENT+8(%rsp) + movq %r9, TABENT+40(%rsp) + + movq %rcx, %rsi + cmovnzq %r10, %rsi + cmovnzq %rcx, %r10 + movq %rsi, TABENT+16(%rsp) + movq %r10, TABENT+48(%rsp) + + movq %rdx, %rsi + cmovnzq %r11, %rsi + cmovnzq %rdx, %r11 + movq %rsi, TABENT+24(%rsp) + movq %r11, TABENT+56(%rsp) + + movq $-19, %rax + movq $-1, %rbx + movq $-1, %rcx + movq $0x7fffffffffffffff, %rdx + subq %r12, %rax + sbbq %r13, %rbx + sbbq %r14, %rcx + sbbq %r15, %rdx + + movq ix, %r8 + movq bias, %r9 + testq %r8, %r8 + cmovzq %r8, %r9 + testq %r9, %r9 + + cmovzq %r12, %rax + cmovzq %r13, %rbx + cmovzq %r14, %rcx + cmovzq %r15, %rdx + movq %rax, TABENT+64(%rsp) + movq %rbx, TABENT+72(%rsp) + movq %rcx, TABENT+80(%rsp) + movq %rdx, TABENT+88(%rsp) + +// Extended-projective and precomputed mixed addition. +// This is effectively the same as calling the standalone +// function edwards25519_pepadd(acc,acc,tabent), but we +// only retain slightly weaker normalization < 2 * p_25519 +// throughout the inner loop, so the computation is +// slightly different, and faster overall. + + double_twice4(t0,z_1) + sub_twice4(t1,y_1,x_1) + add_twice4(t2,y_1,x_1) + mul_4(t3,w_1,kxy_2) + mul_4(t1,t1,ymx_2) + mul_4(t2,t2,xpy_2) + sub_twice4(t4,t0,t3) + add_twice4(t0,t0,t3) + sub_twice4(t5,t2,t1) + add_twice4(t1,t2,t1) + mul_4(z_3,t4,t0) + mul_4(x_3,t5,t4) + mul_4(y_3,t0,t1) + mul_4(w_3,t5,t1) + +// End of the main loop; move on by 4 bits. + + addq $4, i + cmpq $252, i + jc Ledwards25519_scalarmulbase_alt_scalarloop + +// Insert the optional negation of the projective X coordinate, and +// so by extension the final affine x coordinate x = X/Z and thus +// the point P = (x,y). We only know X < 2 * p_25519, so we do the +// negation as 2 * p_25519 - X to keep it nonnegative. From this +// point on we don't need any normalization of the coordinates +// except for making sure that they fit in 4 digits. + + movq X3(%rsp), %r8 + movq X3+8(%rsp), %r9 + movq X3+16(%rsp), %r10 + movq X3+24(%rsp), %r11 + movq $0xffffffffffffffda, %r12 + subq %r8, %r12 + movq $0xffffffffffffffff, %r13 + sbbq %r9, %r13 + movq $0xffffffffffffffff, %r14 + sbbq %r10, %r14 + movq $0xffffffffffffffff, %r15 + sbbq %r11, %r15 + movq SCALAR+24(%rsp), %rax + btq $63, %rax + cmovcq %r12, %r8 + cmovcq %r13, %r9 + cmovcq %r14, %r10 + cmovcq %r15, %r11 + movq %r8, X3(%rsp) + movq %r9, X3+8(%rsp) + movq %r10, X3+16(%rsp) + movq %r11, X3+24(%rsp) + +// Now we need to map out of the extended-projective representation +// (X,Y,Z,W) back to the affine form (x,y) = (X/Z,Y/Z). This means +// first calling the modular inverse to get w_3 = 1/z_3. + + leaq W3(%rsp), %rdi + leaq Z3(%rsp), %rsi + +// Inline copy of bignum_inv_p25519, identical except for stripping out +// the prologue and epilogue saving and restoring registers and making +// and reclaiming room on the stack. For more details and explanations see +// "x86/curve25519/bignum_inv_p25519.S". Note that the stack it uses for +// its own temporaries is 208 bytes, so it has no effect on variables +// that are needed in the rest of our computation here: res, x_3, y_3, +// z_3 and w_3. + + movq %rdi, 0xc0(%rsp) + xorl %eax, %eax + leaq -0x13(%rax), %rcx + notq %rax + movq %rcx, (%rsp) + movq %rax, 0x8(%rsp) + movq %rax, 0x10(%rsp) + btr $0x3f, %rax + movq %rax, 0x18(%rsp) + movq (%rsi), %rdx + movq 0x8(%rsi), %rcx + movq 0x10(%rsi), %r8 + movq 0x18(%rsi), %r9 + movl $0x1, %eax + xorl %r10d, %r10d + bts $0x3f, %r9 + adcq %r10, %rax + imulq $0x13, %rax, %rax + addq %rax, %rdx + adcq %r10, %rcx + adcq %r10, %r8 + adcq %r10, %r9 + movl $0x13, %eax + cmovbq %r10, %rax + subq %rax, %rdx + sbbq %r10, %rcx + sbbq %r10, %r8 + sbbq %r10, %r9 + btr $0x3f, %r9 + movq %rdx, 0x20(%rsp) + movq %rcx, 0x28(%rsp) + movq %r8, 0x30(%rsp) + movq %r9, 0x38(%rsp) + xorl %eax, %eax + movq %rax, 0x40(%rsp) + movq %rax, 0x48(%rsp) + movq %rax, 0x50(%rsp) + movq %rax, 0x58(%rsp) + movabsq $0xa0f99e2375022099, %rax + movq %rax, 0x60(%rsp) + movabsq $0xa8c68f3f1d132595, %rax + movq %rax, 0x68(%rsp) + movabsq $0x6c6c893805ac5242, %rax + movq %rax, 0x70(%rsp) + movabsq $0x276508b241770615, %rax + movq %rax, 0x78(%rsp) + movq $0xa, 0x90(%rsp) + movq $0x1, 0x98(%rsp) + jmp Ledwards25519_scalarmulbase_alt_midloop +Ledwards25519_scalarmulbase_alt_inverseloop: + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + movq %r8, %rax + andq %r9, %rax + movq %r10, %rdi + andq %r11, %rdi + addq %rax, %rdi + movq %rdi, 0x80(%rsp) + movq %r12, %rax + andq %r13, %rax + movq %r14, %rsi + andq %r15, %rsi + addq %rax, %rsi + movq %rsi, 0x88(%rsp) + xorl %ebx, %ebx + movq (%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x20(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rdi + adcq %rdx, %rbx + xorl %ebp, %ebp + movq (%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x20(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + xorl %ecx, %ecx + movq 0x8(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x28(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + shrdq $0x3b, %rbx, %rdi + movq %rdi, (%rsp) + xorl %edi, %edi + movq 0x8(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rbp + adcq %rdx, %rdi + movq 0x28(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rdi + shrdq $0x3b, %rbp, %rsi + movq %rsi, 0x20(%rsp) + xorl %esi, %esi + movq 0x10(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rsi + movq 0x30(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rsi + shrdq $0x3b, %rcx, %rbx + movq %rbx, 0x8(%rsp) + xorl %ebx, %ebx + movq 0x10(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x30(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rdi + adcq %rdx, %rbx + shrdq $0x3b, %rdi, %rbp + movq %rbp, 0x28(%rsp) + movq 0x18(%rsp), %rax + xorq %r9, %rax + movq %rax, %rbp + sarq $0x3f, %rbp + andq %r8, %rbp + negq %rbp + mulq %r8 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x38(%rsp), %rax + xorq %r11, %rax + movq %rax, %rdx + sarq $0x3f, %rdx + andq %r10, %rdx + subq %rdx, %rbp + mulq %r10 + addq %rax, %rsi + adcq %rdx, %rbp + shrdq $0x3b, %rsi, %rcx + movq %rcx, 0x10(%rsp) + shrdq $0x3b, %rbp, %rsi + movq 0x18(%rsp), %rax + movq %rsi, 0x18(%rsp) + xorq %r13, %rax + movq %rax, %rsi + sarq $0x3f, %rsi + andq %r12, %rsi + negq %rsi + mulq %r12 + addq %rax, %rbx + adcq %rdx, %rsi + movq 0x38(%rsp), %rax + xorq %r15, %rax + movq %rax, %rdx + sarq $0x3f, %rdx + andq %r14, %rdx + subq %rdx, %rsi + mulq %r14 + addq %rax, %rbx + adcq %rdx, %rsi + shrdq $0x3b, %rbx, %rdi + movq %rdi, 0x30(%rsp) + shrdq $0x3b, %rsi, %rbx + movq %rbx, 0x38(%rsp) + movq 0x80(%rsp), %rbx + movq 0x88(%rsp), %rbp + xorl %ecx, %ecx + movq 0x40(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x60(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x40(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x40(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x60(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x60(%rsp) + xorl %ebx, %ebx + movq 0x48(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x68(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + xorl %ebp, %ebp + movq 0x48(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, 0x48(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x68(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, 0x68(%rsp) + xorl %ecx, %ecx + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x70(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x50(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x50(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x70(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x70(%rsp) + movq 0x58(%rsp), %rax + xorq %r9, %rax + movq %r9, %rbx + andq %r8, %rbx + negq %rbx + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x78(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbx + mulq %r10 + addq %rax, %rcx + adcq %rbx, %rdx + movq %rdx, %rbx + shldq $0x1, %rcx, %rdx + sarq $0x3f, %rbx + addq %rbx, %rdx + movl $0x13, %eax + imulq %rdx + movq 0x40(%rsp), %r8 + addq %rax, %r8 + movq %r8, 0x40(%rsp) + movq 0x48(%rsp), %r8 + adcq %rdx, %r8 + movq %r8, 0x48(%rsp) + movq 0x50(%rsp), %r8 + adcq %rbx, %r8 + movq %r8, 0x50(%rsp) + adcq %rbx, %rcx + shlq $0x3f, %rax + addq %rax, %rcx + movq 0x58(%rsp), %rax + movq %rcx, 0x58(%rsp) + xorq %r13, %rax + movq %r13, %rcx + andq %r12, %rcx + negq %rcx + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rcx + movq 0x78(%rsp), %rax + xorq %r15, %rax + movq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rcx + mulq %r14 + addq %rax, %rsi + adcq %rcx, %rdx + movq %rdx, %rcx + shldq $0x1, %rsi, %rdx + sarq $0x3f, %rcx + movl $0x13, %eax + addq %rcx, %rdx + imulq %rdx + movq 0x60(%rsp), %r8 + addq %rax, %r8 + movq %r8, 0x60(%rsp) + movq 0x68(%rsp), %r8 + adcq %rdx, %r8 + movq %r8, 0x68(%rsp) + movq 0x70(%rsp), %r8 + adcq %rcx, %r8 + movq %r8, 0x70(%rsp) + adcq %rcx, %rsi + shlq $0x3f, %rax + addq %rax, %rsi + movq %rsi, 0x78(%rsp) +Ledwards25519_scalarmulbase_alt_midloop: + movq 0x98(%rsp), %rsi + movq (%rsp), %rdx + movq 0x20(%rsp), %rcx + movq %rdx, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + xorl %ebp, %ebp + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %rdx + leaq (%rcx,%rax), %rdi + shlq $0x16, %rdx + shlq $0x16, %rdi + sarq $0x2b, %rdx + sarq $0x2b, %rdi + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %rbx + leaq (%rcx,%rax), %rcx + sarq $0x2a, %rbx + sarq $0x2a, %rcx + movq %rdx, 0xa0(%rsp) + movq %rbx, 0xa8(%rsp) + movq %rdi, 0xb0(%rsp) + movq %rcx, 0xb8(%rsp) + movq (%rsp), %r12 + imulq %r12, %rdi + imulq %rdx, %r12 + movq 0x20(%rsp), %r13 + imulq %r13, %rbx + imulq %rcx, %r13 + addq %rbx, %r12 + addq %rdi, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r10 + shlq $0x16, %r8 + shlq $0x16, %r10 + sarq $0x2b, %r8 + sarq $0x2b, %r10 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r15 + leaq (%rcx,%rax), %r11 + sarq $0x2a, %r15 + sarq $0x2a, %r11 + movq %r13, %rbx + movq %r12, %rcx + imulq %r8, %r12 + imulq %r15, %rbx + addq %rbx, %r12 + imulq %r11, %r13 + imulq %r10, %rcx + addq %rcx, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq 0xa0(%rsp), %rax + imulq %r8, %rax + movq 0xb0(%rsp), %rdx + imulq %r15, %rdx + imulq 0xa8(%rsp), %r8 + imulq 0xb8(%rsp), %r15 + addq %r8, %r15 + leaq (%rax,%rdx), %r9 + movq 0xa0(%rsp), %rax + imulq %r10, %rax + movq 0xb0(%rsp), %rdx + imulq %r11, %rdx + imulq 0xa8(%rsp), %r10 + imulq 0xb8(%rsp), %r11 + addq %r10, %r11 + leaq (%rax,%rdx), %r13 + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r12 + shlq $0x15, %r8 + shlq $0x15, %r12 + sarq $0x2b, %r8 + sarq $0x2b, %r12 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r10 + leaq (%rcx,%rax), %r14 + sarq $0x2b, %r10 + sarq $0x2b, %r14 + movq %r9, %rax + imulq %r8, %rax + movq %r13, %rdx + imulq %r10, %rdx + imulq %r15, %r8 + imulq %r11, %r10 + addq %r8, %r10 + leaq (%rax,%rdx), %r8 + movq %r9, %rax + imulq %r12, %rax + movq %r13, %rdx + imulq %r14, %rdx + imulq %r15, %r12 + imulq %r11, %r14 + addq %r12, %r14 + leaq (%rax,%rdx), %r12 + movq %rsi, 0x98(%rsp) + decq 0x90(%rsp) + jne Ledwards25519_scalarmulbase_alt_inverseloop + movq (%rsp), %rax + movq 0x20(%rsp), %rcx + imulq %r8, %rax + imulq %r10, %rcx + addq %rcx, %rax + sarq $0x3f, %rax + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + xorq %rax, %r9 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + xorq %rax, %r11 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + xorq %rax, %r13 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + xorq %rax, %r15 + movq %r8, %rax + andq %r9, %rax + movq %r10, %r12 + andq %r11, %r12 + addq %rax, %r12 + xorl %r13d, %r13d + movq 0x40(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x60(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movq 0x48(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r13 + adcq %rdx, %r14 + movq 0x68(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x70(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x58(%rsp), %rax + xorq %r9, %rax + andq %r8, %r9 + negq %r9 + mulq %r8 + addq %rax, %r15 + adcq %rdx, %r9 + movq 0x78(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %r9 + mulq %r10 + addq %rax, %r15 + adcq %rdx, %r9 + movq %r9, %rax + shldq $0x1, %r15, %rax + sarq $0x3f, %r9 + movl $0x13, %ebx + leaq 0x1(%rax,%r9,1), %rax + imulq %rbx + xorl %ebp, %ebp + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r9, %r14 + adcq %r9, %r15 + shlq $0x3f, %rax + addq %rax, %r15 + cmovns %rbp, %rbx + subq %rbx, %r12 + sbbq %rbp, %r13 + sbbq %rbp, %r14 + sbbq %rbp, %r15 + btr $0x3f, %r15 + movq 0xc0(%rsp), %rdi + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + +// The final result is x = X * inv(Z), y = Y * inv(Z). +// These are the only operations in the whole computation that +// fully reduce modulo p_25519 since now we want the canonical +// answer as output. + + movq res, %rbp + mul_p25519(resx,x_3,w_3) + mul_p25519(resy,y_3,w_3) + +// Restore stack and registers + + CFI_INC_RSP(NSPACE) + + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +#if WINDOWS_ABI +S2N_BN_SIZE_DIRECTIVE(Ledwards25519_scalarmulbase_alt_standard) +#else +S2N_BN_SIZE_DIRECTIVE(edwards25519_scalarmulbase_alt) +#endif + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif + +// **************************************************************************** +// The precomputed data (all read-only). +// **************************************************************************** + +#if defined(__ELF__) +.section .rodata +.type S2N_BN_SYMBOL(edwards25519_scalarmulbase_alt_constant), %object +.size S2N_BN_SYMBOL(edwards25519_scalarmulbase_alt_constant), 48576 +#elif defined(__APPLE__) +.const_data +#endif + +S2N_BN_SYMBOL(edwards25519_scalarmulbase_alt_constant): + +// 0 * B = 0 and 2^251 * B in extended-projective coordinates +// but with Z = 1 assumed and hence left out, so they are (X,Y,T) only. + + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + + .quad 0x0000000000000001 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + .quad 0x0000000000000000 + + .quad 0x525f946d7c7220e7 + .quad 0x4636b0b2f1e35444 + .quad 0x796e9d70e892ae0f + .quad 0x03dec05fa937adb1 + .quad 0x6d1c271cc6375515 + .quad 0x462588c4a4ca4f14 + .quad 0x691129fee55afc39 + .quad 0x15949f784d8472f5 + .quad 0xbd89e510afad0049 + .quad 0x4d1f08c073b9860e + .quad 0x07716e8b2d00af9d + .quad 0x70d685f68f859714 + +// Precomputed table of multiples of generator for edwards25519 +// all in precomputed extended-projective (y-x,x+y,2*d*x*y) triples. + + // 2^0 * 1 * G + + .quad 0x9d103905d740913e + .quad 0xfd399f05d140beb3 + .quad 0xa5c18434688f8a09 + .quad 0x44fd2f9298f81267 + .quad 0x2fbc93c6f58c3b85 + .quad 0xcf932dc6fb8c0e19 + .quad 0x270b4898643d42c2 + .quad 0x07cf9d3a33d4ba65 + .quad 0xabc91205877aaa68 + .quad 0x26d9e823ccaac49e + .quad 0x5a1b7dcbdd43598c + .quad 0x6f117b689f0c65a8 + + // 2^0 * 2 * G + + .quad 0x8a99a56042b4d5a8 + .quad 0x8f2b810c4e60acf6 + .quad 0xe09e236bb16e37aa + .quad 0x6bb595a669c92555 + .quad 0x9224e7fc933c71d7 + .quad 0x9f469d967a0ff5b5 + .quad 0x5aa69a65e1d60702 + .quad 0x590c063fa87d2e2e + .quad 0x43faa8b3a59b7a5f + .quad 0x36c16bdd5d9acf78 + .quad 0x500fa0840b3d6a31 + .quad 0x701af5b13ea50b73 + + // 2^0 * 3 * G + + .quad 0x56611fe8a4fcd265 + .quad 0x3bd353fde5c1ba7d + .quad 0x8131f31a214bd6bd + .quad 0x2ab91587555bda62 + .quad 0xaf25b0a84cee9730 + .quad 0x025a8430e8864b8a + .quad 0xc11b50029f016732 + .quad 0x7a164e1b9a80f8f4 + .quad 0x14ae933f0dd0d889 + .quad 0x589423221c35da62 + .quad 0xd170e5458cf2db4c + .quad 0x5a2826af12b9b4c6 + + // 2^0 * 4 * G + + .quad 0x95fe050a056818bf + .quad 0x327e89715660faa9 + .quad 0xc3e8e3cd06a05073 + .quad 0x27933f4c7445a49a + .quad 0x287351b98efc099f + .quad 0x6765c6f47dfd2538 + .quad 0xca348d3dfb0a9265 + .quad 0x680e910321e58727 + .quad 0x5a13fbe9c476ff09 + .quad 0x6e9e39457b5cc172 + .quad 0x5ddbdcf9102b4494 + .quad 0x7f9d0cbf63553e2b + + // 2^0 * 5 * G + + .quad 0x7f9182c3a447d6ba + .quad 0xd50014d14b2729b7 + .quad 0xe33cf11cb864a087 + .quad 0x154a7e73eb1b55f3 + .quad 0xa212bc4408a5bb33 + .quad 0x8d5048c3c75eed02 + .quad 0xdd1beb0c5abfec44 + .quad 0x2945ccf146e206eb + .quad 0xbcbbdbf1812a8285 + .quad 0x270e0807d0bdd1fc + .quad 0xb41b670b1bbda72d + .quad 0x43aabe696b3bb69a + + // 2^0 * 6 * G + + .quad 0x499806b67b7d8ca4 + .quad 0x575be28427d22739 + .quad 0xbb085ce7204553b9 + .quad 0x38b64c41ae417884 + .quad 0x3a0ceeeb77157131 + .quad 0x9b27158900c8af88 + .quad 0x8065b668da59a736 + .quad 0x51e57bb6a2cc38bd + .quad 0x85ac326702ea4b71 + .quad 0xbe70e00341a1bb01 + .quad 0x53e4a24b083bc144 + .quad 0x10b8e91a9f0d61e3 + + // 2^0 * 7 * G + + .quad 0xba6f2c9aaa3221b1 + .quad 0x6ca021533bba23a7 + .quad 0x9dea764f92192c3a + .quad 0x1d6edd5d2e5317e0 + .quad 0x6b1a5cd0944ea3bf + .quad 0x7470353ab39dc0d2 + .quad 0x71b2528228542e49 + .quad 0x461bea69283c927e + .quad 0xf1836dc801b8b3a2 + .quad 0xb3035f47053ea49a + .quad 0x529c41ba5877adf3 + .quad 0x7a9fbb1c6a0f90a7 + + // 2^0 * 8 * G + + .quad 0xe2a75dedf39234d9 + .quad 0x963d7680e1b558f9 + .quad 0x2c2741ac6e3c23fb + .quad 0x3a9024a1320e01c3 + .quad 0x59b7596604dd3e8f + .quad 0x6cb30377e288702c + .quad 0xb1339c665ed9c323 + .quad 0x0915e76061bce52f + .quad 0xe7c1f5d9c9a2911a + .quad 0xb8a371788bcca7d7 + .quad 0x636412190eb62a32 + .quad 0x26907c5c2ecc4e95 + + // 2^4 * 1 * G + + .quad 0x7ec851ca553e2df3 + .quad 0xa71284cba64878b3 + .quad 0xe6b5e4193288d1e7 + .quad 0x4cf210ec5a9a8883 + .quad 0x322d04a52d9021f6 + .quad 0xb9c19f3375c6bf9c + .quad 0x587a3a4342d20b09 + .quad 0x143b1cf8aa64fe61 + .quad 0x9f867c7d968acaab + .quad 0x5f54258e27092729 + .quad 0xd0a7d34bea180975 + .quad 0x21b546a3374126e1 + + // 2^4 * 2 * G + + .quad 0xa94ff858a2888343 + .quad 0xce0ed4565313ed3c + .quad 0xf55c3dcfb5bf34fa + .quad 0x0a653ca5c9eab371 + .quad 0x490a7a45d185218f + .quad 0x9a15377846049335 + .quad 0x0060ea09cc31e1f6 + .quad 0x7e041577f86ee965 + .quad 0x66b2a496ce5b67f3 + .quad 0xff5492d8bd569796 + .quad 0x503cec294a592cd0 + .quad 0x566943650813acb2 + + // 2^4 * 3 * G + + .quad 0xb818db0c26620798 + .quad 0x5d5c31d9606e354a + .quad 0x0982fa4f00a8cdc7 + .quad 0x17e12bcd4653e2d4 + .quad 0x5672f9eb1dabb69d + .quad 0xba70b535afe853fc + .quad 0x47ac0f752796d66d + .quad 0x32a5351794117275 + .quad 0xd3a644a6df648437 + .quad 0x703b6559880fbfdd + .quad 0xcb852540ad3a1aa5 + .quad 0x0900b3f78e4c6468 + + // 2^4 * 4 * G + + .quad 0x0a851b9f679d651b + .quad 0xe108cb61033342f2 + .quad 0xd601f57fe88b30a3 + .quad 0x371f3acaed2dd714 + .quad 0xed280fbec816ad31 + .quad 0x52d9595bd8e6efe3 + .quad 0x0fe71772f6c623f5 + .quad 0x4314030b051e293c + .quad 0xd560005efbf0bcad + .quad 0x8eb70f2ed1870c5e + .quad 0x201f9033d084e6a0 + .quad 0x4c3a5ae1ce7b6670 + + // 2^4 * 5 * G + + .quad 0x4138a434dcb8fa95 + .quad 0x870cf67d6c96840b + .quad 0xde388574297be82c + .quad 0x7c814db27262a55a + .quad 0xbaf875e4c93da0dd + .quad 0xb93282a771b9294d + .quad 0x80d63fb7f4c6c460 + .quad 0x6de9c73dea66c181 + .quad 0x478904d5a04df8f2 + .quad 0xfafbae4ab10142d3 + .quad 0xf6c8ac63555d0998 + .quad 0x5aac4a412f90b104 + + // 2^4 * 6 * G + + .quad 0xc64f326b3ac92908 + .quad 0x5551b282e663e1e0 + .quad 0x476b35f54a1a4b83 + .quad 0x1b9da3fe189f68c2 + .quad 0x603a0d0abd7f5134 + .quad 0x8089c932e1d3ae46 + .quad 0xdf2591398798bd63 + .quad 0x1c145cd274ba0235 + .quad 0x32e8386475f3d743 + .quad 0x365b8baf6ae5d9ef + .quad 0x825238b6385b681e + .quad 0x234929c1167d65e1 + + // 2^4 * 7 * G + + .quad 0x984decaba077ade8 + .quad 0x383f77ad19eb389d + .quad 0xc7ec6b7e2954d794 + .quad 0x59c77b3aeb7c3a7a + .quad 0x48145cc21d099fcf + .quad 0x4535c192cc28d7e5 + .quad 0x80e7c1e548247e01 + .quad 0x4a5f28743b2973ee + .quad 0xd3add725225ccf62 + .quad 0x911a3381b2152c5d + .quad 0xd8b39fad5b08f87d + .quad 0x6f05606b4799fe3b + + // 2^4 * 8 * G + + .quad 0x9ffe9e92177ba962 + .quad 0x98aee71d0de5cae1 + .quad 0x3ff4ae942d831044 + .quad 0x714de12e58533ac8 + .quad 0x5b433149f91b6483 + .quad 0xadb5dc655a2cbf62 + .quad 0x87fa8412632827b3 + .quad 0x60895e91ab49f8d8 + .quad 0xe9ecf2ed0cf86c18 + .quad 0xb46d06120735dfd4 + .quad 0xbc9da09804b96be7 + .quad 0x73e2e62fd96dc26b + + // 2^8 * 1 * G + + .quad 0xed5b635449aa515e + .quad 0xa865c49f0bc6823a + .quad 0x850c1fe95b42d1c4 + .quad 0x30d76d6f03d315b9 + .quad 0x2eccdd0e632f9c1d + .quad 0x51d0b69676893115 + .quad 0x52dfb76ba8637a58 + .quad 0x6dd37d49a00eef39 + .quad 0x6c4444172106e4c7 + .quad 0xfb53d680928d7f69 + .quad 0xb4739ea4694d3f26 + .quad 0x10c697112e864bb0 + + // 2^8 * 2 * G + + .quad 0x6493c4277dbe5fde + .quad 0x265d4fad19ad7ea2 + .quad 0x0e00dfc846304590 + .quad 0x25e61cabed66fe09 + .quad 0x0ca62aa08358c805 + .quad 0x6a3d4ae37a204247 + .quad 0x7464d3a63b11eddc + .quad 0x03bf9baf550806ef + .quad 0x3f13e128cc586604 + .quad 0x6f5873ecb459747e + .quad 0xa0b63dedcc1268f5 + .quad 0x566d78634586e22c + + // 2^8 * 3 * G + + .quad 0x1637a49f9cc10834 + .quad 0xbc8e56d5a89bc451 + .quad 0x1cb5ec0f7f7fd2db + .quad 0x33975bca5ecc35d9 + .quad 0xa1054285c65a2fd0 + .quad 0x6c64112af31667c3 + .quad 0x680ae240731aee58 + .quad 0x14fba5f34793b22a + .quad 0x3cd746166985f7d4 + .quad 0x593e5e84c9c80057 + .quad 0x2fc3f2b67b61131e + .quad 0x14829cea83fc526c + + // 2^8 * 4 * G + + .quad 0xff437b8497dd95c2 + .quad 0x6c744e30aa4eb5a7 + .quad 0x9e0c5d613c85e88b + .quad 0x2fd9c71e5f758173 + .quad 0x21e70b2f4e71ecb8 + .quad 0xe656ddb940a477e3 + .quad 0xbf6556cece1d4f80 + .quad 0x05fc3bc4535d7b7e + .quad 0x24b8b3ae52afdedd + .quad 0x3495638ced3b30cf + .quad 0x33a4bc83a9be8195 + .quad 0x373767475c651f04 + + // 2^8 * 5 * G + + .quad 0x2fba99fd40d1add9 + .quad 0xb307166f96f4d027 + .quad 0x4363f05215f03bae + .quad 0x1fbea56c3b18f999 + .quad 0x634095cb14246590 + .quad 0xef12144016c15535 + .quad 0x9e38140c8910bc60 + .quad 0x6bf5905730907c8c + .quad 0x0fa778f1e1415b8a + .quad 0x06409ff7bac3a77e + .quad 0x6f52d7b89aa29a50 + .quad 0x02521cf67a635a56 + + // 2^8 * 6 * G + + .quad 0x513fee0b0a9d5294 + .quad 0x8f98e75c0fdf5a66 + .quad 0xd4618688bfe107ce + .quad 0x3fa00a7e71382ced + .quad 0xb1146720772f5ee4 + .quad 0xe8f894b196079ace + .quad 0x4af8224d00ac824a + .quad 0x001753d9f7cd6cc4 + .quad 0x3c69232d963ddb34 + .quad 0x1dde87dab4973858 + .quad 0xaad7d1f9a091f285 + .quad 0x12b5fe2fa048edb6 + + // 2^8 * 7 * G + + .quad 0x71f0fbc496fce34d + .quad 0x73b9826badf35bed + .quad 0xd2047261ff28c561 + .quad 0x749b76f96fb1206f + .quad 0xdf2b7c26ad6f1e92 + .quad 0x4b66d323504b8913 + .quad 0x8c409dc0751c8bc3 + .quad 0x6f7e93c20796c7b8 + .quad 0x1f5af604aea6ae05 + .quad 0xc12351f1bee49c99 + .quad 0x61a808b5eeff6b66 + .quad 0x0fcec10f01e02151 + + // 2^8 * 8 * G + + .quad 0x644d58a649fe1e44 + .quad 0x21fcaea231ad777e + .quad 0x02441c5a887fd0d2 + .quad 0x4901aa7183c511f3 + .quad 0x3df2d29dc4244e45 + .quad 0x2b020e7493d8de0a + .quad 0x6cc8067e820c214d + .quad 0x413779166feab90a + .quad 0x08b1b7548c1af8f0 + .quad 0xce0f7a7c246299b4 + .quad 0xf760b0f91e06d939 + .quad 0x41bb887b726d1213 + + // 2^12 * 1 * G + + .quad 0x9267806c567c49d8 + .quad 0x066d04ccca791e6a + .quad 0xa69f5645e3cc394b + .quad 0x5c95b686a0788cd2 + .quad 0x97d980e0aa39f7d2 + .quad 0x35d0384252c6b51c + .quad 0x7d43f49307cd55aa + .quad 0x56bd36cfb78ac362 + .quad 0x2ac519c10d14a954 + .quad 0xeaf474b494b5fa90 + .quad 0xe6af8382a9f87a5a + .quad 0x0dea6db1879be094 + + // 2^12 * 2 * G + + .quad 0xaa66bf547344e5ab + .quad 0xda1258888f1b4309 + .quad 0x5e87d2b3fd564b2f + .quad 0x5b2c78885483b1dd + .quad 0x15baeb74d6a8797a + .quad 0x7ef55cf1fac41732 + .quad 0x29001f5a3c8b05c5 + .quad 0x0ad7cc8752eaccfb + .quad 0x52151362793408cf + .quad 0xeb0f170319963d94 + .quad 0xa833b2fa883d9466 + .quad 0x093a7fa775003c78 + + // 2^12 * 3 * G + + .quad 0xe5107de63a16d7be + .quad 0xa377ffdc9af332cf + .quad 0x70d5bf18440b677f + .quad 0x6a252b19a4a31403 + .quad 0xb8e9604460a91286 + .quad 0x7f3fd8047778d3de + .quad 0x67d01e31bf8a5e2d + .quad 0x7b038a06c27b653e + .quad 0x9ed919d5d36990f3 + .quad 0x5213aebbdb4eb9f2 + .quad 0xc708ea054cb99135 + .quad 0x58ded57f72260e56 + + // 2^12 * 4 * G + + .quad 0x78e79dade9413d77 + .quad 0xf257f9d59729e67d + .quad 0x59db910ee37aa7e6 + .quad 0x6aa11b5bbb9e039c + .quad 0xda6d53265b0fd48b + .quad 0x8960823193bfa988 + .quad 0xd78ac93261d57e28 + .quad 0x79f2942d3a5c8143 + .quad 0x97da2f25b6c88de9 + .quad 0x251ba7eaacf20169 + .quad 0x09b44f87ef4eb4e4 + .quad 0x7d90ab1bbc6a7da5 + + // 2^12 * 5 * G + + .quad 0x9acca683a7016bfe + .quad 0x90505f4df2c50b6d + .quad 0x6b610d5fcce435aa + .quad 0x19a10d446198ff96 + .quad 0x1a07a3f496b3c397 + .quad 0x11ceaa188f4e2532 + .quad 0x7d9498d5a7751bf0 + .quad 0x19ed161f508dd8a0 + .quad 0x560a2cd687dce6ca + .quad 0x7f3568c48664cf4d + .quad 0x8741e95222803a38 + .quad 0x483bdab1595653fc + + // 2^12 * 6 * G + + .quad 0xfa780f148734fa49 + .quad 0x106f0b70360534e0 + .quad 0x2210776fe3e307bd + .quad 0x3286c109dde6a0fe + .quad 0xd6cf4d0ab4da80f6 + .quad 0x82483e45f8307fe0 + .quad 0x05005269ae6f9da4 + .quad 0x1c7052909cf7877a + .quad 0x32ee7de2874e98d4 + .quad 0x14c362e9b97e0c60 + .quad 0x5781dcde6a60a38a + .quad 0x217dd5eaaa7aa840 + + // 2^12 * 7 * G + + .quad 0x9db7c4d0248e1eb0 + .quad 0xe07697e14d74bf52 + .quad 0x1e6a9b173c562354 + .quad 0x7fa7c21f795a4965 + .quad 0x8bdf1fb9be8c0ec8 + .quad 0x00bae7f8e30a0282 + .quad 0x4963991dad6c4f6c + .quad 0x07058a6e5df6f60a + .quad 0xe9eb02c4db31f67f + .quad 0xed25fd8910bcfb2b + .quad 0x46c8131f5c5cddb4 + .quad 0x33b21c13a0cb9bce + + // 2^12 * 8 * G + + .quad 0x360692f8087d8e31 + .quad 0xf4dcc637d27163f7 + .quad 0x25a4e62065ea5963 + .quad 0x659bf72e5ac160d9 + .quad 0x9aafb9b05ee38c5b + .quad 0xbf9d2d4e071a13c7 + .quad 0x8eee6e6de933290a + .quad 0x1c3bab17ae109717 + .quad 0x1c9ab216c7cab7b0 + .quad 0x7d65d37407bbc3cc + .quad 0x52744750504a58d5 + .quad 0x09f2606b131a2990 + + // 2^16 * 1 * G + + .quad 0x40e87d44744346be + .quad 0x1d48dad415b52b25 + .quad 0x7c3a8a18a13b603e + .quad 0x4eb728c12fcdbdf7 + .quad 0x7e234c597c6691ae + .quad 0x64889d3d0a85b4c8 + .quad 0xdae2c90c354afae7 + .quad 0x0a871e070c6a9e1d + .quad 0x3301b5994bbc8989 + .quad 0x736bae3a5bdd4260 + .quad 0x0d61ade219d59e3c + .quad 0x3ee7300f2685d464 + + // 2^16 * 2 * G + + .quad 0xf5d255e49e7dd6b7 + .quad 0x8016115c610b1eac + .quad 0x3c99975d92e187ca + .quad 0x13815762979125c2 + .quad 0x43fa7947841e7518 + .quad 0xe5c6fa59639c46d7 + .quad 0xa1065e1de3052b74 + .quad 0x7d47c6a2cfb89030 + .quad 0x3fdad0148ef0d6e0 + .quad 0x9d3e749a91546f3c + .quad 0x71ec621026bb8157 + .quad 0x148cf58d34c9ec80 + + // 2^16 * 3 * G + + .quad 0x46a492f67934f027 + .quad 0x469984bef6840aa9 + .quad 0x5ca1bc2a89611854 + .quad 0x3ff2fa1ebd5dbbd4 + .quad 0xe2572f7d9ae4756d + .quad 0x56c345bb88f3487f + .quad 0x9fd10b6d6960a88d + .quad 0x278febad4eaea1b9 + .quad 0xb1aa681f8c933966 + .quad 0x8c21949c20290c98 + .quad 0x39115291219d3c52 + .quad 0x4104dd02fe9c677b + + // 2^16 * 4 * G + + .quad 0x72b2bf5e1124422a + .quad 0xa1fa0c3398a33ab5 + .quad 0x94cb6101fa52b666 + .quad 0x2c863b00afaf53d5 + .quad 0x81214e06db096ab8 + .quad 0x21a8b6c90ce44f35 + .quad 0x6524c12a409e2af5 + .quad 0x0165b5a48efca481 + .quad 0xf190a474a0846a76 + .quad 0x12eff984cd2f7cc0 + .quad 0x695e290658aa2b8f + .quad 0x591b67d9bffec8b8 + + // 2^16 * 5 * G + + .quad 0x312f0d1c80b49bfa + .quad 0x5979515eabf3ec8a + .quad 0x727033c09ef01c88 + .quad 0x3de02ec7ca8f7bcb + .quad 0x99b9b3719f18b55d + .quad 0xe465e5faa18c641e + .quad 0x61081136c29f05ed + .quad 0x489b4f867030128b + .quad 0xd232102d3aeb92ef + .quad 0xe16253b46116a861 + .quad 0x3d7eabe7190baa24 + .quad 0x49f5fbba496cbebf + + // 2^16 * 6 * G + + .quad 0x30949a108a5bcfd4 + .quad 0xdc40dd70bc6473eb + .quad 0x92c294c1307c0d1c + .quad 0x5604a86dcbfa6e74 + .quad 0x155d628c1e9c572e + .quad 0x8a4d86acc5884741 + .quad 0x91a352f6515763eb + .quad 0x06a1a6c28867515b + .quad 0x7288d1d47c1764b6 + .quad 0x72541140e0418b51 + .quad 0x9f031a6018acf6d1 + .quad 0x20989e89fe2742c6 + + // 2^16 * 7 * G + + .quad 0x499777fd3a2dcc7f + .quad 0x32857c2ca54fd892 + .quad 0xa279d864d207e3a0 + .quad 0x0403ed1d0ca67e29 + .quad 0x1674278b85eaec2e + .quad 0x5621dc077acb2bdf + .quad 0x640a4c1661cbf45a + .quad 0x730b9950f70595d3 + .quad 0xc94b2d35874ec552 + .quad 0xc5e6c8cf98246f8d + .quad 0xf7cb46fa16c035ce + .quad 0x5bd7454308303dcc + + // 2^16 * 8 * G + + .quad 0x7f9ad19528b24cc2 + .quad 0x7f6b54656335c181 + .quad 0x66b8b66e4fc07236 + .quad 0x133a78007380ad83 + .quad 0x85c4932115e7792a + .quad 0xc64c89a2bdcdddc9 + .quad 0x9d1e3da8ada3d762 + .quad 0x5bb7db123067f82c + .quad 0x0961f467c6ca62be + .quad 0x04ec21d6211952ee + .quad 0x182360779bd54770 + .quad 0x740dca6d58f0e0d2 + + // 2^20 * 1 * G + + .quad 0x50b70bf5d3f0af0b + .quad 0x4feaf48ae32e71f7 + .quad 0x60e84ed3a55bbd34 + .quad 0x00ed489b3f50d1ed + .quad 0x3906c72aed261ae5 + .quad 0x9ab68fd988e100f7 + .quad 0xf5e9059af3360197 + .quad 0x0e53dc78bf2b6d47 + .quad 0xb90829bf7971877a + .quad 0x5e4444636d17e631 + .quad 0x4d05c52e18276893 + .quad 0x27632d9a5a4a4af5 + + // 2^20 * 2 * G + + .quad 0xd11ff05154b260ce + .quad 0xd86dc38e72f95270 + .quad 0x601fcd0d267cc138 + .quad 0x2b67916429e90ccd + .quad 0xa98285d187eaffdb + .quad 0xa5b4fbbbd8d0a864 + .quad 0xb658f27f022663f7 + .quad 0x3bbc2b22d99ce282 + .quad 0xb917c952583c0a58 + .quad 0x653ff9b80fe4c6f3 + .quad 0x9b0da7d7bcdf3c0c + .quad 0x43a0eeb6ab54d60e + + // 2^20 * 3 * G + + .quad 0x396966a46d4a5487 + .quad 0xf811a18aac2bb3ba + .quad 0x66e4685b5628b26b + .quad 0x70a477029d929b92 + .quad 0x3ac6322357875fe8 + .quad 0xd9d4f4ecf5fbcb8f + .quad 0x8dee8493382bb620 + .quad 0x50c5eaa14c799fdc + .quad 0xdd0edc8bd6f2fb3c + .quad 0x54c63aa79cc7b7a0 + .quad 0xae0b032b2c8d9f1a + .quad 0x6f9ce107602967fb + + // 2^20 * 4 * G + + .quad 0xad1054b1cde1c22a + .quad 0xc4a8e90248eb32df + .quad 0x5f3e7b33accdc0ea + .quad 0x72364713fc79963e + .quad 0x139693063520e0b5 + .quad 0x437fcf7c88ea03fe + .quad 0xf7d4c40bd3c959bc + .quad 0x699154d1f893ded9 + .quad 0x315d5c75b4b27526 + .quad 0xcccb842d0236daa5 + .quad 0x22f0c8a3345fee8e + .quad 0x73975a617d39dbed + + // 2^20 * 5 * G + + .quad 0xe4024df96375da10 + .quad 0x78d3251a1830c870 + .quad 0x902b1948658cd91c + .quad 0x7e18b10b29b7438a + .quad 0x6f37f392f4433e46 + .quad 0x0e19b9a11f566b18 + .quad 0x220fb78a1fd1d662 + .quad 0x362a4258a381c94d + .quad 0x9071d9132b6beb2f + .quad 0x0f26e9ad28418247 + .quad 0xeab91ec9bdec925d + .quad 0x4be65bc8f48af2de + + // 2^20 * 6 * G + + .quad 0x78487feba36e7028 + .quad 0x5f3f13001dd8ce34 + .quad 0x934fb12d4b30c489 + .quad 0x056c244d397f0a2b + .quad 0x1d50fba257c26234 + .quad 0x7bd4823adeb0678b + .quad 0xc2b0dc6ea6538af5 + .quad 0x5665eec6351da73e + .quad 0xdb3ee00943bfb210 + .quad 0x4972018720800ac2 + .quad 0x26ab5d6173bd8667 + .quad 0x20b209c2ab204938 + + // 2^20 * 7 * G + + .quad 0x549e342ac07fb34b + .quad 0x02d8220821373d93 + .quad 0xbc262d70acd1f567 + .quad 0x7a92c9fdfbcac784 + .quad 0x1fcca94516bd3289 + .quad 0x448d65aa41420428 + .quad 0x59c3b7b216a55d62 + .quad 0x49992cc64e612cd8 + .quad 0x65bd1bea70f801de + .quad 0x1befb7c0fe49e28a + .quad 0xa86306cdb1b2ae4a + .quad 0x3b7ac0cd265c2a09 + + // 2^20 * 8 * G + + .quad 0x822bee438c01bcec + .quad 0x530cb525c0fbc73b + .quad 0x48519034c1953fe9 + .quad 0x265cc261e09a0f5b + .quad 0xf0d54e4f22ed39a7 + .quad 0xa2aae91e5608150a + .quad 0xf421b2e9eddae875 + .quad 0x31bc531d6b7de992 + .quad 0xdf3d134da980f971 + .quad 0x7a4fb8d1221a22a7 + .quad 0x3df7d42035aad6d8 + .quad 0x2a14edcc6a1a125e + + // 2^24 * 1 * G + + .quad 0xdf48ee0752cfce4e + .quad 0xc3fffaf306ec08b7 + .quad 0x05710b2ab95459c4 + .quad 0x161d25fa963ea38d + .quad 0x231a8c570478433c + .quad 0xb7b5270ec281439d + .quad 0xdbaa99eae3d9079f + .quad 0x2c03f5256c2b03d9 + .quad 0x790f18757b53a47d + .quad 0x307b0130cf0c5879 + .quad 0x31903d77257ef7f9 + .quad 0x699468bdbd96bbaf + + // 2^24 * 2 * G + + .quad 0xbd1f2f46f4dafecf + .quad 0x7cef0114a47fd6f7 + .quad 0xd31ffdda4a47b37f + .quad 0x525219a473905785 + .quad 0xd8dd3de66aa91948 + .quad 0x485064c22fc0d2cc + .quad 0x9b48246634fdea2f + .quad 0x293e1c4e6c4a2e3a + .quad 0x376e134b925112e1 + .quad 0x703778b5dca15da0 + .quad 0xb04589af461c3111 + .quad 0x5b605c447f032823 + + // 2^24 * 3 * G + + .quad 0xb965805920c47c89 + .quad 0xe7f0100c923b8fcc + .quad 0x0001256502e2ef77 + .quad 0x24a76dcea8aeb3ee + .quad 0x3be9fec6f0e7f04c + .quad 0x866a579e75e34962 + .quad 0x5542ef161e1de61a + .quad 0x2f12fef4cc5abdd5 + .quad 0x0a4522b2dfc0c740 + .quad 0x10d06e7f40c9a407 + .quad 0xc6cf144178cff668 + .quad 0x5e607b2518a43790 + + // 2^24 * 4 * G + + .quad 0x58b31d8f6cdf1818 + .quad 0x35cfa74fc36258a2 + .quad 0xe1b3ff4f66e61d6e + .quad 0x5067acab6ccdd5f7 + .quad 0xa02c431ca596cf14 + .quad 0xe3c42d40aed3e400 + .quad 0xd24526802e0f26db + .quad 0x201f33139e457068 + .quad 0xfd527f6b08039d51 + .quad 0x18b14964017c0006 + .quad 0xd5220eb02e25a4a8 + .quad 0x397cba8862460375 + + // 2^24 * 5 * G + + .quad 0x30c13093f05959b2 + .quad 0xe23aa18de9a97976 + .quad 0x222fd491721d5e26 + .quad 0x2339d320766e6c3a + .quad 0x7815c3fbc81379e7 + .quad 0xa6619420dde12af1 + .quad 0xffa9c0f885a8fdd5 + .quad 0x771b4022c1e1c252 + .quad 0xd87dd986513a2fa7 + .quad 0xf5ac9b71f9d4cf08 + .quad 0xd06bc31b1ea283b3 + .quad 0x331a189219971a76 + + // 2^24 * 6 * G + + .quad 0xf5166f45fb4f80c6 + .quad 0x9c36c7de61c775cf + .quad 0xe3d4e81b9041d91c + .quad 0x31167c6b83bdfe21 + .quad 0x26512f3a9d7572af + .quad 0x5bcbe28868074a9e + .quad 0x84edc1c11180f7c4 + .quad 0x1ac9619ff649a67b + .quad 0xf22b3842524b1068 + .quad 0x5068343bee9ce987 + .quad 0xfc9d71844a6250c8 + .quad 0x612436341f08b111 + + // 2^24 * 7 * G + + .quad 0xd99d41db874e898d + .quad 0x09fea5f16c07dc20 + .quad 0x793d2c67d00f9bbc + .quad 0x46ebe2309e5eff40 + .quad 0x8b6349e31a2d2638 + .quad 0x9ddfb7009bd3fd35 + .quad 0x7f8bf1b8a3a06ba4 + .quad 0x1522aa3178d90445 + .quad 0x2c382f5369614938 + .quad 0xdafe409ab72d6d10 + .quad 0xe8c83391b646f227 + .quad 0x45fe70f50524306c + + // 2^24 * 8 * G + + .quad 0xda4875a6960c0b8c + .quad 0x5b68d076ef0e2f20 + .quad 0x07fb51cf3d0b8fd4 + .quad 0x428d1623a0e392d4 + .quad 0x62f24920c8951491 + .quad 0x05f007c83f630ca2 + .quad 0x6fbb45d2f5c9d4b8 + .quad 0x16619f6db57a2245 + .quad 0x084f4a4401a308fd + .quad 0xa82219c376a5caac + .quad 0xdeb8de4643d1bc7d + .quad 0x1d81592d60bd38c6 + + // 2^28 * 1 * G + + .quad 0xd833d7beec2a4c38 + .quad 0x2c9162830acc20ed + .quad 0xe93a47aa92df7581 + .quad 0x702d67a3333c4a81 + .quad 0x3a4a369a2f89c8a1 + .quad 0x63137a1d7c8de80d + .quad 0xbcac008a78eda015 + .quad 0x2cb8b3a5b483b03f + .quad 0x36e417cbcb1b90a1 + .quad 0x33b3ddaa7f11794e + .quad 0x3f510808885bc607 + .quad 0x24141dc0e6a8020d + + // 2^28 * 2 * G + + .quad 0x59f73c773fefee9d + .quad 0xb3f1ef89c1cf989d + .quad 0xe35dfb42e02e545f + .quad 0x5766120b47a1b47c + .quad 0x91925dccbd83157d + .quad 0x3ca1205322cc8094 + .quad 0x28e57f183f90d6e4 + .quad 0x1a4714cede2e767b + .quad 0xdb20ba0fb8b6b7ff + .quad 0xb732c3b677511fa1 + .quad 0xa92b51c099f02d89 + .quad 0x4f3875ad489ca5f1 + + // 2^28 * 3 * G + + .quad 0xc7fc762f4932ab22 + .quad 0x7ac0edf72f4c3c1b + .quad 0x5f6b55aa9aa895e8 + .quad 0x3680274dad0a0081 + .quad 0x79ed13f6ee73eec0 + .quad 0xa5c6526d69110bb1 + .quad 0xe48928c38603860c + .quad 0x722a1446fd7059f5 + .quad 0xd0959fe9a8cf8819 + .quad 0xd0a995508475a99c + .quad 0x6eac173320b09cc5 + .quad 0x628ecf04331b1095 + + // 2^28 * 4 * G + + .quad 0x98bcb118a9d0ddbc + .quad 0xee449e3408b4802b + .quad 0x87089226b8a6b104 + .quad 0x685f349a45c7915d + .quad 0x9b41acf85c74ccf1 + .quad 0xb673318108265251 + .quad 0x99c92aed11adb147 + .quad 0x7a47d70d34ecb40f + .quad 0x60a0c4cbcc43a4f5 + .quad 0x775c66ca3677bea9 + .quad 0xa17aa1752ff8f5ed + .quad 0x11ded9020e01fdc0 + + // 2^28 * 5 * G + + .quad 0x890e7809caefe704 + .quad 0x8728296de30e8c6c + .quad 0x4c5cd2a392aeb1c9 + .quad 0x194263d15771531f + .quad 0x471f95b03bea93b7 + .quad 0x0552d7d43313abd3 + .quad 0xbd9370e2e17e3f7b + .quad 0x7b120f1db20e5bec + .quad 0x17d2fb3d86502d7a + .quad 0xb564d84450a69352 + .quad 0x7da962c8a60ed75d + .quad 0x00d0f85b318736aa + + // 2^28 * 6 * G + + .quad 0x978b142e777c84fd + .quad 0xf402644705a8c062 + .quad 0xa67ad51be7e612c7 + .quad 0x2f7b459698dd6a33 + .quad 0xa6753c1efd7621c1 + .quad 0x69c0b4a7445671f5 + .quad 0x971f527405b23c11 + .quad 0x387bc74851a8c7cd + .quad 0x81894b4d4a52a9a8 + .quad 0xadd93e12f6b8832f + .quad 0x184d8548b61bd638 + .quad 0x3f1c62dbd6c9f6cd + + // 2^28 * 7 * G + + .quad 0x2e8f1f0091910c1f + .quad 0xa4df4fe0bff2e12c + .quad 0x60c6560aee927438 + .quad 0x6338283facefc8fa + .quad 0x3fad3e40148f693d + .quad 0x052656e194eb9a72 + .quad 0x2f4dcbfd184f4e2f + .quad 0x406f8db1c482e18b + .quad 0x9e630d2c7f191ee4 + .quad 0x4fbf8301bc3ff670 + .quad 0x787d8e4e7afb73c4 + .quad 0x50d83d5be8f58fa5 + + // 2^28 * 8 * G + + .quad 0x85683916c11a1897 + .quad 0x2d69a4efe506d008 + .quad 0x39af1378f664bd01 + .quad 0x65942131361517c6 + .quad 0xc0accf90b4d3b66d + .quad 0xa7059de561732e60 + .quad 0x033d1f7870c6b0ba + .quad 0x584161cd26d946e4 + .quad 0xbbf2b1a072d27ca2 + .quad 0xbf393c59fbdec704 + .quad 0xe98dbbcee262b81e + .quad 0x02eebd0b3029b589 + + // 2^32 * 1 * G + + .quad 0x61368756a60dac5f + .quad 0x17e02f6aebabdc57 + .quad 0x7f193f2d4cce0f7d + .quad 0x20234a7789ecdcf0 + .quad 0x8765b69f7b85c5e8 + .quad 0x6ff0678bd168bab2 + .quad 0x3a70e77c1d330f9b + .quad 0x3a5f6d51b0af8e7c + .quad 0x76d20db67178b252 + .quad 0x071c34f9d51ed160 + .quad 0xf62a4a20b3e41170 + .quad 0x7cd682353cffe366 + + // 2^32 * 2 * G + + .quad 0x0be1a45bd887fab6 + .quad 0x2a846a32ba403b6e + .quad 0xd9921012e96e6000 + .quad 0x2838c8863bdc0943 + .quad 0xa665cd6068acf4f3 + .quad 0x42d92d183cd7e3d3 + .quad 0x5759389d336025d9 + .quad 0x3ef0253b2b2cd8ff + .quad 0xd16bb0cf4a465030 + .quad 0xfa496b4115c577ab + .quad 0x82cfae8af4ab419d + .quad 0x21dcb8a606a82812 + + // 2^32 * 3 * G + + .quad 0x5c6004468c9d9fc8 + .quad 0x2540096ed42aa3cb + .quad 0x125b4d4c12ee2f9c + .quad 0x0bc3d08194a31dab + .quad 0x9a8d00fabe7731ba + .quad 0x8203607e629e1889 + .quad 0xb2cc023743f3d97f + .quad 0x5d840dbf6c6f678b + .quad 0x706e380d309fe18b + .quad 0x6eb02da6b9e165c7 + .quad 0x57bbba997dae20ab + .quad 0x3a4276232ac196dd + + // 2^32 * 4 * G + + .quad 0x4b42432c8a7084fa + .quad 0x898a19e3dfb9e545 + .quad 0xbe9f00219c58e45d + .quad 0x1ff177cea16debd1 + .quad 0x3bf8c172db447ecb + .quad 0x5fcfc41fc6282dbd + .quad 0x80acffc075aa15fe + .quad 0x0770c9e824e1a9f9 + .quad 0xcf61d99a45b5b5fd + .quad 0x860984e91b3a7924 + .quad 0xe7300919303e3e89 + .quad 0x39f264fd41500b1e + + // 2^32 * 5 * G + + .quad 0xa7ad3417dbe7e29c + .quad 0xbd94376a2b9c139c + .quad 0xa0e91b8e93597ba9 + .quad 0x1712d73468889840 + .quad 0xd19b4aabfe097be1 + .quad 0xa46dfce1dfe01929 + .quad 0xc3c908942ca6f1ff + .quad 0x65c621272c35f14e + .quad 0xe72b89f8ce3193dd + .quad 0x4d103356a125c0bb + .quad 0x0419a93d2e1cfe83 + .quad 0x22f9800ab19ce272 + + // 2^32 * 6 * G + + .quad 0x605a368a3e9ef8cb + .quad 0xe3e9c022a5504715 + .quad 0x553d48b05f24248f + .quad 0x13f416cd647626e5 + .quad 0x42029fdd9a6efdac + .quad 0xb912cebe34a54941 + .quad 0x640f64b987bdf37b + .quad 0x4171a4d38598cab4 + .quad 0xfa2758aa99c94c8c + .quad 0x23006f6fb000b807 + .quad 0xfbd291ddadda5392 + .quad 0x508214fa574bd1ab + + // 2^32 * 7 * G + + .quad 0xc20269153ed6fe4b + .quad 0xa65a6739511d77c4 + .quad 0xcbde26462c14af94 + .quad 0x22f960ec6faba74b + .quad 0x461a15bb53d003d6 + .quad 0xb2102888bcf3c965 + .quad 0x27c576756c683a5a + .quad 0x3a7758a4c86cb447 + .quad 0x548111f693ae5076 + .quad 0x1dae21df1dfd54a6 + .quad 0x12248c90f3115e65 + .quad 0x5d9fd15f8de7f494 + + // 2^32 * 8 * G + + .quad 0x031408d36d63727f + .quad 0x6a379aefd7c7b533 + .quad 0xa9e18fc5ccaee24b + .quad 0x332f35914f8fbed3 + .quad 0x3f244d2aeed7521e + .quad 0x8e3a9028432e9615 + .quad 0xe164ba772e9c16d4 + .quad 0x3bc187fa47eb98d8 + .quad 0x6d470115ea86c20c + .quad 0x998ab7cb6c46d125 + .quad 0xd77832b53a660188 + .quad 0x450d81ce906fba03 + + // 2^36 * 1 * G + + .quad 0xf8ae4d2ad8453902 + .quad 0x7018058ee8db2d1d + .quad 0xaab3995fc7d2c11e + .quad 0x53b16d2324ccca79 + .quad 0x23264d66b2cae0b5 + .quad 0x7dbaed33ebca6576 + .quad 0x030ebed6f0d24ac8 + .quad 0x2a887f78f7635510 + .quad 0x2a23b9e75c012d4f + .quad 0x0c974651cae1f2ea + .quad 0x2fb63273675d70ca + .quad 0x0ba7250b864403f5 + + // 2^36 * 2 * G + + .quad 0xbb0d18fd029c6421 + .quad 0xbc2d142189298f02 + .quad 0x8347f8e68b250e96 + .quad 0x7b9f2fe8032d71c9 + .quad 0xdd63589386f86d9c + .quad 0x61699176e13a85a4 + .quad 0x2e5111954eaa7d57 + .quad 0x32c21b57fb60bdfb + .quad 0xd87823cd319e0780 + .quad 0xefc4cfc1897775c5 + .quad 0x4854fb129a0ab3f7 + .quad 0x12c49d417238c371 + + // 2^36 * 3 * G + + .quad 0x0950b533ffe83769 + .quad 0x21861c1d8e1d6bd1 + .quad 0xf022d8381302e510 + .quad 0x2509200c6391cab4 + .quad 0x09b3a01783799542 + .quad 0x626dd08faad5ee3f + .quad 0xba00bceeeb70149f + .quad 0x1421b246a0a444c9 + .quad 0x4aa43a8e8c24a7c7 + .quad 0x04c1f540d8f05ef5 + .quad 0xadba5e0c0b3eb9dc + .quad 0x2ab5504448a49ce3 + + // 2^36 * 4 * G + + .quad 0x2ed227266f0f5dec + .quad 0x9824ee415ed50824 + .quad 0x807bec7c9468d415 + .quad 0x7093bae1b521e23f + .quad 0xdc07ac631c5d3afa + .quad 0x58615171f9df8c6c + .quad 0x72a079d89d73e2b0 + .quad 0x7301f4ceb4eae15d + .quad 0x6409e759d6722c41 + .quad 0xa674e1cf72bf729b + .quad 0xbc0a24eb3c21e569 + .quad 0x390167d24ebacb23 + + // 2^36 * 5 * G + + .quad 0x27f58e3bba353f1c + .quad 0x4c47764dbf6a4361 + .quad 0xafbbc4e56e562650 + .quad 0x07db2ee6aae1a45d + .quad 0xd7bb054ba2f2120b + .quad 0xe2b9ceaeb10589b7 + .quad 0x3fe8bac8f3c0edbe + .quad 0x4cbd40767112cb69 + .quad 0x0b603cc029c58176 + .quad 0x5988e3825cb15d61 + .quad 0x2bb61413dcf0ad8d + .quad 0x7b8eec6c74183287 + + // 2^36 * 6 * G + + .quad 0xe4ca40782cd27cb0 + .quad 0xdaf9c323fbe967bd + .quad 0xb29bd34a8ad41e9e + .quad 0x72810497626ede4d + .quad 0x32fee570fc386b73 + .quad 0xda8b0141da3a8cc7 + .quad 0x975ffd0ac8968359 + .quad 0x6ee809a1b132a855 + .quad 0x9444bb31fcfd863a + .quad 0x2fe3690a3e4e48c5 + .quad 0xdc29c867d088fa25 + .quad 0x13bd1e38d173292e + + // 2^36 * 7 * G + + .quad 0xd32b4cd8696149b5 + .quad 0xe55937d781d8aab7 + .quad 0x0bcb2127ae122b94 + .quad 0x41e86fcfb14099b0 + .quad 0x223fb5cf1dfac521 + .quad 0x325c25316f554450 + .quad 0x030b98d7659177ac + .quad 0x1ed018b64f88a4bd + .quad 0x3630dfa1b802a6b0 + .quad 0x880f874742ad3bd5 + .quad 0x0af90d6ceec5a4d4 + .quad 0x746a247a37cdc5d9 + + // 2^36 * 8 * G + + .quad 0xd531b8bd2b7b9af6 + .quad 0x5005093537fc5b51 + .quad 0x232fcf25c593546d + .quad 0x20a365142bb40f49 + .quad 0x6eccd85278d941ed + .quad 0x2254ae83d22f7843 + .quad 0xc522d02e7bbfcdb7 + .quad 0x681e3351bff0e4e2 + .quad 0x8b64b59d83034f45 + .quad 0x2f8b71f21fa20efb + .quad 0x69249495ba6550e4 + .quad 0x539ef98e45d5472b + + // 2^40 * 1 * G + + .quad 0x6e7bb6a1a6205275 + .quad 0xaa4f21d7413c8e83 + .quad 0x6f56d155e88f5cb2 + .quad 0x2de25d4ba6345be1 + .quad 0xd074d8961cae743f + .quad 0xf86d18f5ee1c63ed + .quad 0x97bdc55be7f4ed29 + .quad 0x4cbad279663ab108 + .quad 0x80d19024a0d71fcd + .quad 0xc525c20afb288af8 + .quad 0xb1a3974b5f3a6419 + .quad 0x7d7fbcefe2007233 + + // 2^40 * 2 * G + + .quad 0xfaef1e6a266b2801 + .quad 0x866c68c4d5739f16 + .quad 0xf68a2fbc1b03762c + .quad 0x5975435e87b75a8d + .quad 0xcd7c5dc5f3c29094 + .quad 0xc781a29a2a9105ab + .quad 0x80c61d36421c3058 + .quad 0x4f9cd196dcd8d4d7 + .quad 0x199297d86a7b3768 + .quad 0xd0d058241ad17a63 + .quad 0xba029cad5c1c0c17 + .quad 0x7ccdd084387a0307 + + // 2^40 * 3 * G + + .quad 0xdca6422c6d260417 + .quad 0xae153d50948240bd + .quad 0xa9c0c1b4fb68c677 + .quad 0x428bd0ed61d0cf53 + .quad 0x9b0c84186760cc93 + .quad 0xcdae007a1ab32a99 + .quad 0xa88dec86620bda18 + .quad 0x3593ca848190ca44 + .quad 0x9213189a5e849aa7 + .quad 0xd4d8c33565d8facd + .quad 0x8c52545b53fdbbd1 + .quad 0x27398308da2d63e6 + + // 2^40 * 4 * G + + .quad 0x42c38d28435ed413 + .quad 0xbd50f3603278ccc9 + .quad 0xbb07ab1a79da03ef + .quad 0x269597aebe8c3355 + .quad 0xb9a10e4c0a702453 + .quad 0x0fa25866d57d1bde + .quad 0xffb9d9b5cd27daf7 + .quad 0x572c2945492c33fd + .quad 0xc77fc745d6cd30be + .quad 0xe4dfe8d3e3baaefb + .quad 0xa22c8830aa5dda0c + .quad 0x7f985498c05bca80 + + // 2^40 * 5 * G + + .quad 0x3849ce889f0be117 + .quad 0x8005ad1b7b54a288 + .quad 0x3da3c39f23fc921c + .quad 0x76c2ec470a31f304 + .quad 0xd35615520fbf6363 + .quad 0x08045a45cf4dfba6 + .quad 0xeec24fbc873fa0c2 + .quad 0x30f2653cd69b12e7 + .quad 0x8a08c938aac10c85 + .quad 0x46179b60db276bcb + .quad 0xa920c01e0e6fac70 + .quad 0x2f1273f1596473da + + // 2^40 * 6 * G + + .quad 0x4739fc7c8ae01e11 + .quad 0xfd5274904a6aab9f + .quad 0x41d98a8287728f2e + .quad 0x5d9e572ad85b69f2 + .quad 0x30488bd755a70bc0 + .quad 0x06d6b5a4f1d442e7 + .quad 0xead1a69ebc596162 + .quad 0x38ac1997edc5f784 + .quad 0x0666b517a751b13b + .quad 0x747d06867e9b858c + .quad 0xacacc011454dde49 + .quad 0x22dfcd9cbfe9e69c + + // 2^40 * 7 * G + + .quad 0x8ddbd2e0c30d0cd9 + .quad 0xad8e665facbb4333 + .quad 0x8f6b258c322a961f + .quad 0x6b2916c05448c1c7 + .quad 0x56ec59b4103be0a1 + .quad 0x2ee3baecd259f969 + .quad 0x797cb29413f5cd32 + .quad 0x0fe9877824cde472 + .quad 0x7edb34d10aba913b + .quad 0x4ea3cd822e6dac0e + .quad 0x66083dff6578f815 + .quad 0x4c303f307ff00a17 + + // 2^40 * 8 * G + + .quad 0xd30a3bd617b28c85 + .quad 0xc5d377b739773bea + .quad 0xc6c6e78c1e6a5cbf + .quad 0x0d61b8f78b2ab7c4 + .quad 0x29fc03580dd94500 + .quad 0xecd27aa46fbbec93 + .quad 0x130a155fc2e2a7f8 + .quad 0x416b151ab706a1d5 + .quad 0x56a8d7efe9c136b0 + .quad 0xbd07e5cd58e44b20 + .quad 0xafe62fda1b57e0ab + .quad 0x191a2af74277e8d2 + + // 2^44 * 1 * G + + .quad 0xd550095bab6f4985 + .quad 0x04f4cd5b4fbfaf1a + .quad 0x9d8e2ed12a0c7540 + .quad 0x2bc24e04b2212286 + .quad 0x09d4b60b2fe09a14 + .quad 0xc384f0afdbb1747e + .quad 0x58e2ea8978b5fd6e + .quad 0x519ef577b5e09b0a + .quad 0x1863d7d91124cca9 + .quad 0x7ac08145b88a708e + .quad 0x2bcd7309857031f5 + .quad 0x62337a6e8ab8fae5 + + // 2^44 * 2 * G + + .quad 0x4bcef17f06ffca16 + .quad 0xde06e1db692ae16a + .quad 0x0753702d614f42b0 + .quad 0x5f6041b45b9212d0 + .quad 0xd1ab324e1b3a1273 + .quad 0x18947cf181055340 + .quad 0x3b5d9567a98c196e + .quad 0x7fa00425802e1e68 + .quad 0x7d531574028c2705 + .quad 0x80317d69db0d75fe + .quad 0x30fface8ef8c8ddd + .quad 0x7e9de97bb6c3e998 + + // 2^44 * 3 * G + + .quad 0x1558967b9e6585a3 + .quad 0x97c99ce098e98b92 + .quad 0x10af149b6eb3adad + .quad 0x42181fe8f4d38cfa + .quad 0xf004be62a24d40dd + .quad 0xba0659910452d41f + .quad 0x81c45ee162a44234 + .quad 0x4cb829d8a22266ef + .quad 0x1dbcaa8407b86681 + .quad 0x081f001e8b26753b + .quad 0x3cd7ce6a84048e81 + .quad 0x78af11633f25f22c + + // 2^44 * 4 * G + + .quad 0x8416ebd40b50babc + .quad 0x1508722628208bee + .quad 0xa3148fafb9c1c36d + .quad 0x0d07daacd32d7d5d + .quad 0x3241c00e7d65318c + .quad 0xe6bee5dcd0e86de7 + .quad 0x118b2dc2fbc08c26 + .quad 0x680d04a7fc603dc3 + .quad 0xf9c2414a695aa3eb + .quad 0xdaa42c4c05a68f21 + .quad 0x7c6c23987f93963e + .quad 0x210e8cd30c3954e3 + + // 2^44 * 5 * G + + .quad 0xac4201f210a71c06 + .quad 0x6a65e0aef3bfb021 + .quad 0xbc42c35c393632f7 + .quad 0x56ea8db1865f0742 + .quad 0x2b50f16137fe6c26 + .quad 0xe102bcd856e404d8 + .quad 0x12b0f1414c561f6b + .quad 0x51b17bc8d028ec91 + .quad 0xfff5fb4bcf535119 + .quad 0xf4989d79df1108a0 + .quad 0xbdfcea659a3ba325 + .quad 0x18a11f1174d1a6f2 + + // 2^44 * 6 * G + + .quad 0x407375ab3f6bba29 + .quad 0x9ec3b6d8991e482e + .quad 0x99c80e82e55f92e9 + .quad 0x307c13b6fb0c0ae1 + .quad 0xfbd63cdad27a5f2c + .quad 0xf00fc4bc8aa106d7 + .quad 0x53fb5c1a8e64a430 + .quad 0x04eaabe50c1a2e85 + .quad 0x24751021cb8ab5e7 + .quad 0xfc2344495c5010eb + .quad 0x5f1e717b4e5610a1 + .quad 0x44da5f18c2710cd5 + + // 2^44 * 7 * G + + .quad 0x033cc55ff1b82eb5 + .quad 0xb15ae36d411cae52 + .quad 0xba40b6198ffbacd3 + .quad 0x768edce1532e861f + .quad 0x9156fe6b89d8eacc + .quad 0xe6b79451e23126a1 + .quad 0xbd7463d93944eb4e + .quad 0x726373f6767203ae + .quad 0xe305ca72eb7ef68a + .quad 0x662cf31f70eadb23 + .quad 0x18f026fdb4c45b68 + .quad 0x513b5384b5d2ecbd + + // 2^44 * 8 * G + + .quad 0x46d46280c729989e + .quad 0x4b93fbd05368a5dd + .quad 0x63df3f81d1765a89 + .quad 0x34cebd64b9a0a223 + .quad 0x5e2702878af34ceb + .quad 0x900b0409b946d6ae + .quad 0x6512ebf7dabd8512 + .quad 0x61d9b76988258f81 + .quad 0xa6c5a71349b7d94b + .quad 0xa3f3d15823eb9446 + .quad 0x0416fbd277484834 + .quad 0x69d45e6f2c70812f + + // 2^48 * 1 * G + + .quad 0xce16f74bc53c1431 + .quad 0x2b9725ce2072edde + .quad 0xb8b9c36fb5b23ee7 + .quad 0x7e2e0e450b5cc908 + .quad 0x9fe62b434f460efb + .quad 0xded303d4a63607d6 + .quad 0xf052210eb7a0da24 + .quad 0x237e7dbe00545b93 + .quad 0x013575ed6701b430 + .quad 0x231094e69f0bfd10 + .quad 0x75320f1583e47f22 + .quad 0x71afa699b11155e3 + + // 2^48 * 2 * G + + .quad 0x65ce6f9b3953b61d + .quad 0xc65839eaafa141e6 + .quad 0x0f435ffda9f759fe + .quad 0x021142e9c2b1c28e + .quad 0xea423c1c473b50d6 + .quad 0x51e87a1f3b38ef10 + .quad 0x9b84bf5fb2c9be95 + .quad 0x00731fbc78f89a1c + .quad 0xe430c71848f81880 + .quad 0xbf960c225ecec119 + .quad 0xb6dae0836bba15e3 + .quad 0x4c4d6f3347e15808 + + // 2^48 * 3 * G + + .quad 0x18f7eccfc17d1fc9 + .quad 0x6c75f5a651403c14 + .quad 0xdbde712bf7ee0cdf + .quad 0x193fddaaa7e47a22 + .quad 0x2f0cddfc988f1970 + .quad 0x6b916227b0b9f51b + .quad 0x6ec7b6c4779176be + .quad 0x38bf9500a88f9fa8 + .quad 0x1fd2c93c37e8876f + .quad 0xa2f61e5a18d1462c + .quad 0x5080f58239241276 + .quad 0x6a6fb99ebf0d4969 + + // 2^48 * 4 * G + + .quad 0x6a46c1bb560855eb + .quad 0x2416bb38f893f09d + .quad 0xd71d11378f71acc1 + .quad 0x75f76914a31896ea + .quad 0xeeb122b5b6e423c6 + .quad 0x939d7010f286ff8e + .quad 0x90a92a831dcf5d8c + .quad 0x136fda9f42c5eb10 + .quad 0xf94cdfb1a305bdd1 + .quad 0x0f364b9d9ff82c08 + .quad 0x2a87d8a5c3bb588a + .quad 0x022183510be8dcba + + // 2^48 * 5 * G + + .quad 0x4af766385ead2d14 + .quad 0xa08ed880ca7c5830 + .quad 0x0d13a6e610211e3d + .quad 0x6a071ce17b806c03 + .quad 0x9d5a710143307a7f + .quad 0xb063de9ec47da45f + .quad 0x22bbfe52be927ad3 + .quad 0x1387c441fd40426c + .quad 0xb5d3c3d187978af8 + .quad 0x722b5a3d7f0e4413 + .quad 0x0d7b4848bb477ca0 + .quad 0x3171b26aaf1edc92 + + // 2^48 * 6 * G + + .quad 0xa92f319097564ca8 + .quad 0xff7bb84c2275e119 + .quad 0x4f55fe37a4875150 + .quad 0x221fd4873cf0835a + .quad 0xa60db7d8b28a47d1 + .quad 0xa6bf14d61770a4f1 + .quad 0xd4a1f89353ddbd58 + .quad 0x6c514a63344243e9 + .quad 0x2322204f3a156341 + .quad 0xfb73e0e9ba0a032d + .quad 0xfce0dd4c410f030e + .quad 0x48daa596fb924aaa + + // 2^48 * 7 * G + + .quad 0x6eca8e665ca59cc7 + .quad 0xa847254b2e38aca0 + .quad 0x31afc708d21e17ce + .quad 0x676dd6fccad84af7 + .quad 0x14f61d5dc84c9793 + .quad 0x9941f9e3ef418206 + .quad 0xcdf5b88f346277ac + .quad 0x58c837fa0e8a79a9 + .quad 0x0cf9688596fc9058 + .quad 0x1ddcbbf37b56a01b + .quad 0xdcc2e77d4935d66a + .quad 0x1c4f73f2c6a57f0a + + // 2^48 * 8 * G + + .quad 0x0e7a4fbd305fa0bb + .quad 0x829d4ce054c663ad + .quad 0xf421c3832fe33848 + .quad 0x795ac80d1bf64c42 + .quad 0xb36e706efc7c3484 + .quad 0x73dfc9b4c3c1cf61 + .quad 0xeb1d79c9781cc7e5 + .quad 0x70459adb7daf675c + .quad 0x1b91db4991b42bb3 + .quad 0x572696234b02dcca + .quad 0x9fdf9ee51f8c78dc + .quad 0x5fe162848ce21fd3 + + // 2^52 * 1 * G + + .quad 0xe2790aae4d077c41 + .quad 0x8b938270db7469a3 + .quad 0x6eb632dc8abd16a2 + .quad 0x720814ecaa064b72 + .quad 0x315c29c795115389 + .quad 0xd7e0e507862f74ce + .quad 0x0c4a762185927432 + .quad 0x72de6c984a25a1e4 + .quad 0xae9ab553bf6aa310 + .quad 0x050a50a9806d6e1b + .quad 0x92bb7403adff5139 + .quad 0x0394d27645be618b + + // 2^52 * 2 * G + + .quad 0x4d572251857eedf4 + .quad 0xe3724edde19e93c5 + .quad 0x8a71420e0b797035 + .quad 0x3b3c833687abe743 + .quad 0xf5396425b23545a4 + .quad 0x15a7a27e98fbb296 + .quad 0xab6c52bc636fdd86 + .quad 0x79d995a8419334ee + .quad 0xcd8a8ea61195dd75 + .quad 0xa504d8a81dd9a82f + .quad 0x540dca81a35879b6 + .quad 0x60dd16a379c86a8a + + // 2^52 * 3 * G + + .quad 0x35a2c8487381e559 + .quad 0x596ffea6d78082cb + .quad 0xcb9771ebdba7b653 + .quad 0x5a08b5019b4da685 + .quad 0x3501d6f8153e47b8 + .quad 0xb7a9675414a2f60c + .quad 0x112ee8b6455d9523 + .quad 0x4e62a3c18112ea8a + .quad 0xc8d4ac04516ab786 + .quad 0x595af3215295b23d + .quad 0xd6edd234db0230c1 + .quad 0x0929efe8825b41cc + + // 2^52 * 4 * G + + .quad 0x5f0601d1cbd0f2d3 + .quad 0x736e412f6132bb7f + .quad 0x83604432238dde87 + .quad 0x1e3a5272f5c0753c + .quad 0x8b3172b7ad56651d + .quad 0x01581b7a3fabd717 + .quad 0x2dc94df6424df6e4 + .quad 0x30376e5d2c29284f + .quad 0xd2918da78159a59c + .quad 0x6bdc1cd93f0713f3 + .quad 0x565f7a934acd6590 + .quad 0x53daacec4cb4c128 + + // 2^52 * 5 * G + + .quad 0x4ca73bd79cc8a7d6 + .quad 0x4d4a738f47e9a9b2 + .quad 0xf4cbf12942f5fe00 + .quad 0x01a13ff9bdbf0752 + .quad 0x99852bc3852cfdb0 + .quad 0x2cc12e9559d6ed0b + .quad 0x70f9e2bf9b5ac27b + .quad 0x4f3b8c117959ae99 + .quad 0x55b6c9c82ff26412 + .quad 0x1ac4a8c91fb667a8 + .quad 0xd527bfcfeb778bf2 + .quad 0x303337da7012a3be + + // 2^52 * 6 * G + + .quad 0x955422228c1c9d7c + .quad 0x01fac1371a9b340f + .quad 0x7e8d9177925b48d7 + .quad 0x53f8ad5661b3e31b + .quad 0x976d3ccbfad2fdd1 + .quad 0xcb88839737a640a8 + .quad 0x2ff00c1d6734cb25 + .quad 0x269ff4dc789c2d2b + .quad 0x0c003fbdc08d678d + .quad 0x4d982fa37ead2b17 + .quad 0xc07e6bcdb2e582f1 + .quad 0x296c7291df412a44 + + // 2^52 * 7 * G + + .quad 0x7903de2b33daf397 + .quad 0xd0ff0619c9a624b3 + .quad 0x8a1d252b555b3e18 + .quad 0x2b6d581c52e0b7c0 + .quad 0xdfb23205dab8b59e + .quad 0x465aeaa0c8092250 + .quad 0xd133c1189a725d18 + .quad 0x2327370261f117d1 + .quad 0x3d0543d3623e7986 + .quad 0x679414c2c278a354 + .quad 0xae43f0cc726196f6 + .quad 0x7836c41f8245eaba + + // 2^52 * 8 * G + + .quad 0xe7a254db49e95a81 + .quad 0x5192d5d008b0ad73 + .quad 0x4d20e5b1d00afc07 + .quad 0x5d55f8012cf25f38 + .quad 0xca651e848011937c + .quad 0xc6b0c46e6ef41a28 + .quad 0xb7021ba75f3f8d52 + .quad 0x119dff99ead7b9fd + .quad 0x43eadfcbf4b31d4d + .quad 0xc6503f7411148892 + .quad 0xfeee68c5060d3b17 + .quad 0x329293b3dd4a0ac8 + + // 2^56 * 1 * G + + .quad 0x4e59214fe194961a + .quad 0x49be7dc70d71cd4f + .quad 0x9300cfd23b50f22d + .quad 0x4789d446fc917232 + .quad 0x2879852d5d7cb208 + .quad 0xb8dedd70687df2e7 + .quad 0xdc0bffab21687891 + .quad 0x2b44c043677daa35 + .quad 0x1a1c87ab074eb78e + .quad 0xfac6d18e99daf467 + .quad 0x3eacbbcd484f9067 + .quad 0x60c52eef2bb9a4e4 + + // 2^56 * 2 * G + + .quad 0x0b5d89bc3bfd8bf1 + .quad 0xb06b9237c9f3551a + .quad 0x0e4c16b0d53028f5 + .quad 0x10bc9c312ccfcaab + .quad 0x702bc5c27cae6d11 + .quad 0x44c7699b54a48cab + .quad 0xefbc4056ba492eb2 + .quad 0x70d77248d9b6676d + .quad 0xaa8ae84b3ec2a05b + .quad 0x98699ef4ed1781e0 + .quad 0x794513e4708e85d1 + .quad 0x63755bd3a976f413 + + // 2^56 * 3 * G + + .quad 0xb55fa03e2ad10853 + .quad 0x356f75909ee63569 + .quad 0x9ff9f1fdbe69b890 + .quad 0x0d8cc1c48bc16f84 + .quad 0x3dc7101897f1acb7 + .quad 0x5dda7d5ec165bbd8 + .quad 0x508e5b9c0fa1020f + .quad 0x2763751737c52a56 + .quad 0x029402d36eb419a9 + .quad 0xf0b44e7e77b460a5 + .quad 0xcfa86230d43c4956 + .quad 0x70c2dd8a7ad166e7 + + // 2^56 * 4 * G + + .quad 0x656194509f6fec0e + .quad 0xee2e7ea946c6518d + .quad 0x9733c1f367e09b5c + .quad 0x2e0fac6363948495 + .quad 0x91d4967db8ed7e13 + .quad 0x74252f0ad776817a + .quad 0xe40982e00d852564 + .quad 0x32b8613816a53ce5 + .quad 0x79e7f7bee448cd64 + .quad 0x6ac83a67087886d0 + .quad 0xf89fd4d9a0e4db2e + .quad 0x4179215c735a4f41 + + // 2^56 * 5 * G + + .quad 0x8c7094e7d7dced2a + .quad 0x97fb8ac347d39c70 + .quad 0xe13be033a906d902 + .quad 0x700344a30cd99d76 + .quad 0xe4ae33b9286bcd34 + .quad 0xb7ef7eb6559dd6dc + .quad 0x278b141fb3d38e1f + .quad 0x31fa85662241c286 + .quad 0xaf826c422e3622f4 + .quad 0xc12029879833502d + .quad 0x9bc1b7e12b389123 + .quad 0x24bb2312a9952489 + + // 2^56 * 6 * G + + .quad 0xb1a8ed1732de67c3 + .quad 0x3cb49418461b4948 + .quad 0x8ebd434376cfbcd2 + .quad 0x0fee3e871e188008 + .quad 0x41f80c2af5f85c6b + .quad 0x687284c304fa6794 + .quad 0x8945df99a3ba1bad + .quad 0x0d1d2af9ffeb5d16 + .quad 0xa9da8aa132621edf + .quad 0x30b822a159226579 + .quad 0x4004197ba79ac193 + .quad 0x16acd79718531d76 + + // 2^56 * 7 * G + + .quad 0x72df72af2d9b1d3d + .quad 0x63462a36a432245a + .quad 0x3ecea07916b39637 + .quad 0x123e0ef6b9302309 + .quad 0xc959c6c57887b6ad + .quad 0x94e19ead5f90feba + .quad 0x16e24e62a342f504 + .quad 0x164ed34b18161700 + .quad 0x487ed94c192fe69a + .quad 0x61ae2cea3a911513 + .quad 0x877bf6d3b9a4de27 + .quad 0x78da0fc61073f3eb + + // 2^56 * 8 * G + + .quad 0x5bf15d28e52bc66a + .quad 0x2c47e31870f01a8e + .quad 0x2419afbc06c28bdd + .quad 0x2d25deeb256b173a + .quad 0xa29f80f1680c3a94 + .quad 0x71f77e151ae9e7e6 + .quad 0x1100f15848017973 + .quad 0x054aa4b316b38ddd + .quad 0xdfc8468d19267cb8 + .quad 0x0b28789c66e54daf + .quad 0x2aeb1d2a666eec17 + .quad 0x134610a6ab7da760 + + // 2^60 * 1 * G + + .quad 0xcaf55ec27c59b23f + .quad 0x99aeed3e154d04f2 + .quad 0x68441d72e14141f4 + .quad 0x140345133932a0a2 + .quad 0xd91430e0dc028c3c + .quad 0x0eb955a85217c771 + .quad 0x4b09e1ed2c99a1fa + .quad 0x42881af2bd6a743c + .quad 0x7bfec69aab5cad3d + .quad 0xc23e8cd34cb2cfad + .quad 0x685dd14bfb37d6a2 + .quad 0x0ad6d64415677a18 + + // 2^60 * 2 * G + + .quad 0x781a439e417becb5 + .quad 0x4ac5938cd10e0266 + .quad 0x5da385110692ac24 + .quad 0x11b065a2ade31233 + .quad 0x7914892847927e9f + .quad 0x33dad6ef370aa877 + .quad 0x1f8f24fa11122703 + .quad 0x5265ac2f2adf9592 + .quad 0x405fdd309afcb346 + .quad 0xd9723d4428e63f54 + .quad 0x94c01df05f65aaae + .quad 0x43e4dc3ae14c0809 + + // 2^60 * 3 * G + + .quad 0xbc12c7f1a938a517 + .quad 0x473028ab3180b2e1 + .quad 0x3f78571efbcd254a + .quad 0x74e534426ff6f90f + .quad 0xea6f7ac3adc2c6a3 + .quad 0xd0e928f6e9717c94 + .quad 0xe2d379ead645eaf5 + .quad 0x46dd8785c51ffbbe + .quad 0x709801be375c8898 + .quad 0x4b06dab5e3fd8348 + .quad 0x75880ced27230714 + .quad 0x2b09468fdd2f4c42 + + // 2^60 * 4 * G + + .quad 0x97c749eeb701cb96 + .quad 0x83f438d4b6a369c3 + .quad 0x62962b8b9a402cd9 + .quad 0x6976c7509888df7b + .quad 0x5b97946582ffa02a + .quad 0xda096a51fea8f549 + .quad 0xa06351375f77af9b + .quad 0x1bcfde61201d1e76 + .quad 0x4a4a5490246a59a2 + .quad 0xd63ebddee87fdd90 + .quad 0xd9437c670d2371fa + .quad 0x69e87308d30f8ed6 + + // 2^60 * 5 * G + + .quad 0x435a8bb15656beb0 + .quad 0xf8fac9ba4f4d5bca + .quad 0xb9b278c41548c075 + .quad 0x3eb0ef76e892b622 + .quad 0x0f80bf028bc80303 + .quad 0x6aae16b37a18cefb + .quad 0xdd47ea47d72cd6a3 + .quad 0x61943588f4ed39aa + .quad 0xd26e5c3e91039f85 + .quad 0xc0e9e77df6f33aa9 + .quad 0xe8968c5570066a93 + .quad 0x3c34d1881faaaddd + + // 2^60 * 6 * G + + .quad 0x3f9d2b5ea09f9ec0 + .quad 0x1dab3b6fb623a890 + .quad 0xa09ba3ea72d926c4 + .quad 0x374193513fd8b36d + .quad 0xbd5b0b8f2fffe0d9 + .quad 0x6aa254103ed24fb9 + .quad 0x2ac7d7bcb26821c4 + .quad 0x605b394b60dca36a + .quad 0xb4e856e45a9d1ed2 + .quad 0xefe848766c97a9a2 + .quad 0xb104cf641e5eee7d + .quad 0x2f50b81c88a71c8f + + // 2^60 * 7 * G + + .quad 0x31723c61fc6811bb + .quad 0x9cb450486211800f + .quad 0x768933d347995753 + .quad 0x3491a53502752fcd + .quad 0x2b552ca0a7da522a + .quad 0x3230b336449b0250 + .quad 0xf2c4c5bca4b99fb9 + .quad 0x7b2c674958074a22 + .quad 0xd55165883ed28cdf + .quad 0x12d84fd2d362de39 + .quad 0x0a874ad3e3378e4f + .quad 0x000d2b1f7c763e74 + + // 2^60 * 8 * G + + .quad 0x3d420811d06d4a67 + .quad 0xbefc048590e0ffe3 + .quad 0xf870c6b7bd487bde + .quad 0x6e2a7316319afa28 + .quad 0x9624778c3e94a8ab + .quad 0x0ad6f3cee9a78bec + .quad 0x948ac7810d743c4f + .quad 0x76627935aaecfccc + .quad 0x56a8ac24d6d59a9f + .quad 0xc8db753e3096f006 + .quad 0x477f41e68f4c5299 + .quad 0x588d851cf6c86114 + + // 2^64 * 1 * G + + .quad 0x51138ec78df6b0fe + .quad 0x5397da89e575f51b + .quad 0x09207a1d717af1b9 + .quad 0x2102fdba2b20d650 + .quad 0xcd2a65e777d1f515 + .quad 0x548991878faa60f1 + .quad 0xb1b73bbcdabc06e5 + .quad 0x654878cba97cc9fb + .quad 0x969ee405055ce6a1 + .quad 0x36bca7681251ad29 + .quad 0x3a1af517aa7da415 + .quad 0x0ad725db29ecb2ba + + // 2^64 * 2 * G + + .quad 0xdc4267b1834e2457 + .quad 0xb67544b570ce1bc5 + .quad 0x1af07a0bf7d15ed7 + .quad 0x4aefcffb71a03650 + .quad 0xfec7bc0c9b056f85 + .quad 0x537d5268e7f5ffd7 + .quad 0x77afc6624312aefa + .quad 0x4f675f5302399fd9 + .quad 0xc32d36360415171e + .quad 0xcd2bef118998483b + .quad 0x870a6eadd0945110 + .quad 0x0bccbb72a2a86561 + + // 2^64 * 3 * G + + .quad 0x185e962feab1a9c8 + .quad 0x86e7e63565147dcd + .quad 0xb092e031bb5b6df2 + .quad 0x4024f0ab59d6b73e + .quad 0x186d5e4c50fe1296 + .quad 0xe0397b82fee89f7e + .quad 0x3bc7f6c5507031b0 + .quad 0x6678fd69108f37c2 + .quad 0x1586fa31636863c2 + .quad 0x07f68c48572d33f2 + .quad 0x4f73cc9f789eaefc + .quad 0x2d42e2108ead4701 + + // 2^64 * 4 * G + + .quad 0x97f5131594dfd29b + .quad 0x6155985d313f4c6a + .quad 0xeba13f0708455010 + .quad 0x676b2608b8d2d322 + .quad 0x21717b0d0f537593 + .quad 0x914e690b131e064c + .quad 0x1bb687ae752ae09f + .quad 0x420bf3a79b423c6e + .quad 0x8138ba651c5b2b47 + .quad 0x8671b6ec311b1b80 + .quad 0x7bff0cb1bc3135b0 + .quad 0x745d2ffa9c0cf1e0 + + // 2^64 * 5 * G + + .quad 0xbf525a1e2bc9c8bd + .quad 0xea5b260826479d81 + .quad 0xd511c70edf0155db + .quad 0x1ae23ceb960cf5d0 + .quad 0x6036df5721d34e6a + .quad 0xb1db8827997bb3d0 + .quad 0xd3c209c3c8756afa + .quad 0x06e15be54c1dc839 + .quad 0x5b725d871932994a + .quad 0x32351cb5ceb1dab0 + .quad 0x7dc41549dab7ca05 + .quad 0x58ded861278ec1f7 + + // 2^64 * 6 * G + + .quad 0xd8173793f266c55c + .quad 0xc8c976c5cc454e49 + .quad 0x5ce382f8bc26c3a8 + .quad 0x2ff39de85485f6f9 + .quad 0x2dfb5ba8b6c2c9a8 + .quad 0x48eeef8ef52c598c + .quad 0x33809107f12d1573 + .quad 0x08ba696b531d5bd8 + .quad 0x77ed3eeec3efc57a + .quad 0x04e05517d4ff4811 + .quad 0xea3d7a3ff1a671cb + .quad 0x120633b4947cfe54 + + // 2^64 * 7 * G + + .quad 0x0b94987891610042 + .quad 0x4ee7b13cecebfae8 + .quad 0x70be739594f0a4c0 + .quad 0x35d30a99b4d59185 + .quad 0x82bd31474912100a + .quad 0xde237b6d7e6fbe06 + .quad 0xe11e761911ea79c6 + .quad 0x07433be3cb393bde + .quad 0xff7944c05ce997f4 + .quad 0x575d3de4b05c51a3 + .quad 0x583381fd5a76847c + .quad 0x2d873ede7af6da9f + + // 2^64 * 8 * G + + .quad 0x157a316443373409 + .quad 0xfab8b7eef4aa81d9 + .quad 0xb093fee6f5a64806 + .quad 0x2e773654707fa7b6 + .quad 0xaa6202e14e5df981 + .quad 0xa20d59175015e1f5 + .quad 0x18a275d3bae21d6c + .quad 0x0543618a01600253 + .quad 0x0deabdf4974c23c1 + .quad 0xaa6f0a259dce4693 + .quad 0x04202cb8a29aba2c + .quad 0x4b1443362d07960d + + // 2^68 * 1 * G + + .quad 0x47b837f753242cec + .quad 0x256dc48cc04212f2 + .quad 0xe222fbfbe1d928c5 + .quad 0x48ea295bad8a2c07 + .quad 0x299b1c3f57c5715e + .quad 0x96cb929e6b686d90 + .quad 0x3004806447235ab3 + .quad 0x2c435c24a44d9fe1 + .quad 0x0607c97c80f8833f + .quad 0x0e851578ca25ec5b + .quad 0x54f7450b161ebb6f + .quad 0x7bcb4792a0def80e + + // 2^68 * 2 * G + + .quad 0x8487e3d02bc73659 + .quad 0x4baf8445059979df + .quad 0xd17c975adcad6fbf + .quad 0x57369f0bdefc96b6 + .quad 0x1cecd0a0045224c2 + .quad 0x757f1b1b69e53952 + .quad 0x775b7a925289f681 + .quad 0x1b6cc62016736148 + .quad 0xf1a9990175638698 + .quad 0x353dd1beeeaa60d3 + .quad 0x849471334c9ba488 + .quad 0x63fa6e6843ade311 + + // 2^68 * 3 * G + + .quad 0xd15c20536597c168 + .quad 0x9f73740098d28789 + .quad 0x18aee7f13257ba1f + .quad 0x3418bfda07346f14 + .quad 0x2195becdd24b5eb7 + .quad 0x5e41f18cc0cd44f9 + .quad 0xdf28074441ca9ede + .quad 0x07073b98f35b7d67 + .quad 0xd03c676c4ce530d4 + .quad 0x0b64c0473b5df9f4 + .quad 0x065cef8b19b3a31e + .quad 0x3084d661533102c9 + + // 2^68 * 4 * G + + .quad 0xe1f6b79ebf8469ad + .quad 0x15801004e2663135 + .quad 0x9a498330af74181b + .quad 0x3ba2504f049b673c + .quad 0x9a6ce876760321fd + .quad 0x7fe2b5109eb63ad8 + .quad 0x00e7d4ae8ac80592 + .quad 0x73d86b7abb6f723a + .quad 0x0b52b5606dba5ab6 + .quad 0xa9134f0fbbb1edab + .quad 0x30a9520d9b04a635 + .quad 0x6813b8f37973e5db + + // 2^68 * 5 * G + + .quad 0x9854b054334127c1 + .quad 0x105d047882fbff25 + .quad 0xdb49f7f944186f4f + .quad 0x1768e838bed0b900 + .quad 0xf194ca56f3157e29 + .quad 0x136d35705ef528a5 + .quad 0xdd4cef778b0599bc + .quad 0x7d5472af24f833ed + .quad 0xd0ef874daf33da47 + .quad 0x00d3be5db6e339f9 + .quad 0x3f2a8a2f9c9ceece + .quad 0x5d1aeb792352435a + + // 2^68 * 6 * G + + .quad 0xf59e6bb319cd63ca + .quad 0x670c159221d06839 + .quad 0xb06d565b2150cab6 + .quad 0x20fb199d104f12a3 + .quad 0x12c7bfaeb61ba775 + .quad 0xb84e621fe263bffd + .quad 0x0b47a5c35c840dcf + .quad 0x7e83be0bccaf8634 + .quad 0x61943dee6d99c120 + .quad 0x86101f2e460b9fe0 + .quad 0x6bb2f1518ee8598d + .quad 0x76b76289fcc475cc + + // 2^68 * 7 * G + + .quad 0x791b4cc1756286fa + .quad 0xdbced317d74a157c + .quad 0x7e732421ea72bde6 + .quad 0x01fe18491131c8e9 + .quad 0x4245f1a1522ec0b3 + .quad 0x558785b22a75656d + .quad 0x1d485a2548a1b3c0 + .quad 0x60959eccd58fe09f + .quad 0x3ebfeb7ba8ed7a09 + .quad 0x49fdc2bbe502789c + .quad 0x44ebce5d3c119428 + .quad 0x35e1eb55be947f4a + + // 2^68 * 8 * G + + .quad 0xdbdae701c5738dd3 + .quad 0xf9c6f635b26f1bee + .quad 0x61e96a8042f15ef4 + .quad 0x3aa1d11faf60a4d8 + .quad 0x14fd6dfa726ccc74 + .quad 0x3b084cfe2f53b965 + .quad 0xf33ae4f552a2c8b4 + .quad 0x59aab07a0d40166a + .quad 0x77bcec4c925eac25 + .quad 0x1848718460137738 + .quad 0x5b374337fea9f451 + .quad 0x1865e78ec8e6aa46 + + // 2^72 * 1 * G + + .quad 0xccc4b7c7b66e1f7a + .quad 0x44157e25f50c2f7e + .quad 0x3ef06dfc713eaf1c + .quad 0x582f446752da63f7 + .quad 0x967c54e91c529ccb + .quad 0x30f6269264c635fb + .quad 0x2747aff478121965 + .quad 0x17038418eaf66f5c + .quad 0xc6317bd320324ce4 + .quad 0xa81042e8a4488bc4 + .quad 0xb21ef18b4e5a1364 + .quad 0x0c2a1c4bcda28dc9 + + // 2^72 * 2 * G + + .quad 0xd24dc7d06f1f0447 + .quad 0xb2269e3edb87c059 + .quad 0xd15b0272fbb2d28f + .quad 0x7c558bd1c6f64877 + .quad 0xedc4814869bd6945 + .quad 0x0d6d907dbe1c8d22 + .quad 0xc63bd212d55cc5ab + .quad 0x5a6a9b30a314dc83 + .quad 0xd0ec1524d396463d + .quad 0x12bb628ac35a24f0 + .quad 0xa50c3a791cbc5fa4 + .quad 0x0404a5ca0afbafc3 + + // 2^72 * 3 * G + + .quad 0x8c1f40070aa743d6 + .quad 0xccbad0cb5b265ee8 + .quad 0x574b046b668fd2de + .quad 0x46395bfdcadd9633 + .quad 0x62bc9e1b2a416fd1 + .quad 0xb5c6f728e350598b + .quad 0x04343fd83d5d6967 + .quad 0x39527516e7f8ee98 + .quad 0x117fdb2d1a5d9a9c + .quad 0x9c7745bcd1005c2a + .quad 0xefd4bef154d56fea + .quad 0x76579a29e822d016 + + // 2^72 * 4 * G + + .quad 0x45b68e7e49c02a17 + .quad 0x23cd51a2bca9a37f + .quad 0x3ed65f11ec224c1b + .quad 0x43a384dc9e05bdb1 + .quad 0x333cb51352b434f2 + .quad 0xd832284993de80e1 + .quad 0xb5512887750d35ce + .quad 0x02c514bb2a2777c1 + .quad 0x684bd5da8bf1b645 + .quad 0xfb8bd37ef6b54b53 + .quad 0x313916d7a9b0d253 + .quad 0x1160920961548059 + + // 2^72 * 5 * G + + .quad 0xb44d166929dacfaa + .quad 0xda529f4c8413598f + .quad 0xe9ef63ca453d5559 + .quad 0x351e125bc5698e0b + .quad 0x7a385616369b4dcd + .quad 0x75c02ca7655c3563 + .quad 0x7dc21bf9d4f18021 + .quad 0x2f637d7491e6e042 + .quad 0xd4b49b461af67bbe + .quad 0xd603037ac8ab8961 + .quad 0x71dee19ff9a699fb + .quad 0x7f182d06e7ce2a9a + + // 2^72 * 6 * G + + .quad 0x7a7c8e64ab0168ec + .quad 0xcb5a4a5515edc543 + .quad 0x095519d347cd0eda + .quad 0x67d4ac8c343e93b0 + .quad 0x09454b728e217522 + .quad 0xaa58e8f4d484b8d8 + .quad 0xd358254d7f46903c + .quad 0x44acc043241c5217 + .quad 0x1c7d6bbb4f7a5777 + .quad 0x8b35fed4918313e1 + .quad 0x4adca1c6c96b4684 + .quad 0x556d1c8312ad71bd + + // 2^72 * 7 * G + + .quad 0x17ef40e30c8d3982 + .quad 0x31f7073e15a3fa34 + .quad 0x4f21f3cb0773646e + .quad 0x746c6c6d1d824eff + .quad 0x81f06756b11be821 + .quad 0x0faff82310a3f3dd + .quad 0xf8b2d0556a99465d + .quad 0x097abe38cc8c7f05 + .quad 0x0c49c9877ea52da4 + .quad 0x4c4369559bdc1d43 + .quad 0x022c3809f7ccebd2 + .quad 0x577e14a34bee84bd + + // 2^72 * 8 * G + + .quad 0xf0e268ac61a73b0a + .quad 0xf2fafa103791a5f5 + .quad 0xc1e13e826b6d00e9 + .quad 0x60fa7ee96fd78f42 + .quad 0x94fecebebd4dd72b + .quad 0xf46a4fda060f2211 + .quad 0x124a5977c0c8d1ff + .quad 0x705304b8fb009295 + .quad 0xb63d1d354d296ec6 + .quad 0xf3c3053e5fad31d8 + .quad 0x670b958cb4bd42ec + .quad 0x21398e0ca16353fd + + // 2^76 * 1 * G + + .quad 0x216ab2ca8da7d2ef + .quad 0x366ad9dd99f42827 + .quad 0xae64b9004fdd3c75 + .quad 0x403a395b53909e62 + .quad 0x86c5fc16861b7e9a + .quad 0xf6a330476a27c451 + .quad 0x01667267a1e93597 + .quad 0x05ffb9cd6082dfeb + .quad 0xa617fa9ff53f6139 + .quad 0x60f2b5e513e66cb6 + .quad 0xd7a8beefb3448aa4 + .quad 0x7a2932856f5ea192 + + // 2^76 * 2 * G + + .quad 0x0b39d761b02de888 + .quad 0x5f550e7ed2414e1f + .quad 0xa6bfa45822e1a940 + .quad 0x050a2f7dfd447b99 + .quad 0xb89c444879639302 + .quad 0x4ae4f19350c67f2c + .quad 0xf0b35da8c81af9c6 + .quad 0x39d0003546871017 + .quad 0x437c3b33a650db77 + .quad 0x6bafe81dbac52bb2 + .quad 0xfe99402d2db7d318 + .quad 0x2b5b7eec372ba6ce + + // 2^76 * 3 * G + + .quad 0xb3bc4bbd83f50eef + .quad 0x508f0c998c927866 + .quad 0x43e76587c8b7e66e + .quad 0x0f7655a3a47f98d9 + .quad 0xa694404d613ac8f4 + .quad 0x500c3c2bfa97e72c + .quad 0x874104d21fcec210 + .quad 0x1b205fb38604a8ee + .quad 0x55ecad37d24b133c + .quad 0x441e147d6038c90b + .quad 0x656683a1d62c6fee + .quad 0x0157d5dc87e0ecae + + // 2^76 * 4 * G + + .quad 0xf2a7af510354c13d + .quad 0xd7a0b145aa372b60 + .quad 0x2869b96a05a3d470 + .quad 0x6528e42d82460173 + .quad 0x95265514d71eb524 + .quad 0xe603d8815df14593 + .quad 0x147cdf410d4de6b7 + .quad 0x5293b1730437c850 + .quad 0x23d0e0814bccf226 + .quad 0x92c745cd8196fb93 + .quad 0x8b61796c59541e5b + .quad 0x40a44df0c021f978 + + // 2^76 * 5 * G + + .quad 0xdaa869894f20ea6a + .quad 0xea14a3d14c620618 + .quad 0x6001fccb090bf8be + .quad 0x35f4e822947e9cf0 + .quad 0x86c96e514bc5d095 + .quad 0xf20d4098fca6804a + .quad 0x27363d89c826ea5d + .quad 0x39ca36565719cacf + .quad 0x97506f2f6f87b75c + .quad 0xc624aea0034ae070 + .quad 0x1ec856e3aad34dd6 + .quad 0x055b0be0e440e58f + + // 2^76 * 6 * G + + .quad 0x6469a17d89735d12 + .quad 0xdb6f27d5e662b9f1 + .quad 0x9fcba3286a395681 + .quad 0x363b8004d269af25 + .quad 0x4d12a04b6ea33da2 + .quad 0x57cf4c15e36126dd + .quad 0x90ec9675ee44d967 + .quad 0x64ca348d2a985aac + .quad 0x99588e19e4c4912d + .quad 0xefcc3b4e1ca5ce6b + .quad 0x4522ea60fa5b98d5 + .quad 0x7064bbab1de4a819 + + // 2^76 * 7 * G + + .quad 0xb919e1515a770641 + .quad 0xa9a2e2c74e7f8039 + .quad 0x7527250b3df23109 + .quad 0x756a7330ac27b78b + .quad 0xa290c06142542129 + .quad 0xf2e2c2aebe8d5b90 + .quad 0xcf2458db76abfe1b + .quad 0x02157ade83d626bf + .quad 0x3e46972a1b9a038b + .quad 0x2e4ee66a7ee03fb4 + .quad 0x81a248776edbb4ca + .quad 0x1a944ee88ecd0563 + + // 2^76 * 8 * G + + .quad 0xd5a91d1151039372 + .quad 0x2ed377b799ca26de + .quad 0xa17202acfd366b6b + .quad 0x0730291bd6901995 + .quad 0xbb40a859182362d6 + .quad 0xb99f55778a4d1abb + .quad 0x8d18b427758559f6 + .quad 0x26c20fe74d26235a + .quad 0x648d1d9fe9cc22f5 + .quad 0x66bc561928dd577c + .quad 0x47d3ed21652439d1 + .quad 0x49d271acedaf8b49 + + // 2^80 * 1 * G + + .quad 0x89f5058a382b33f3 + .quad 0x5ae2ba0bad48c0b4 + .quad 0x8f93b503a53db36e + .quad 0x5aa3ed9d95a232e6 + .quad 0x2798aaf9b4b75601 + .quad 0x5eac72135c8dad72 + .quad 0xd2ceaa6161b7a023 + .quad 0x1bbfb284e98f7d4e + .quad 0x656777e9c7d96561 + .quad 0xcb2b125472c78036 + .quad 0x65053299d9506eee + .quad 0x4a07e14e5e8957cc + + // 2^80 * 2 * G + + .quad 0x4ee412cb980df999 + .quad 0xa315d76f3c6ec771 + .quad 0xbba5edde925c77fd + .quad 0x3f0bac391d313402 + .quad 0x240b58cdc477a49b + .quad 0xfd38dade6447f017 + .quad 0x19928d32a7c86aad + .quad 0x50af7aed84afa081 + .quad 0x6e4fde0115f65be5 + .quad 0x29982621216109b2 + .quad 0x780205810badd6d9 + .quad 0x1921a316baebd006 + + // 2^80 * 3 * G + + .quad 0x89422f7edfb870fc + .quad 0x2c296beb4f76b3bd + .quad 0x0738f1d436c24df7 + .quad 0x6458df41e273aeb0 + .quad 0xd75aad9ad9f3c18b + .quad 0x566a0eef60b1c19c + .quad 0x3e9a0bac255c0ed9 + .quad 0x7b049deca062c7f5 + .quad 0xdccbe37a35444483 + .quad 0x758879330fedbe93 + .quad 0x786004c312c5dd87 + .quad 0x6093dccbc2950e64 + + // 2^80 * 4 * G + + .quad 0x1ff39a8585e0706d + .quad 0x36d0a5d8b3e73933 + .quad 0x43b9f2e1718f453b + .quad 0x57d1ea084827a97c + .quad 0x6bdeeebe6084034b + .quad 0x3199c2b6780fb854 + .quad 0x973376abb62d0695 + .quad 0x6e3180c98b647d90 + .quad 0xee7ab6e7a128b071 + .quad 0xa4c1596d93a88baa + .quad 0xf7b4de82b2216130 + .quad 0x363e999ddd97bd18 + + // 2^80 * 5 * G + + .quad 0x96a843c135ee1fc4 + .quad 0x976eb35508e4c8cf + .quad 0xb42f6801b58cd330 + .quad 0x48ee9b78693a052b + .quad 0x2f1848dce24baec6 + .quad 0x769b7255babcaf60 + .quad 0x90cb3c6e3cefe931 + .quad 0x231f979bc6f9b355 + .quad 0x5c31de4bcc2af3c6 + .quad 0xb04bb030fe208d1f + .quad 0xb78d7009c14fb466 + .quad 0x079bfa9b08792413 + + // 2^80 * 6 * G + + .quad 0xe3903a51da300df4 + .quad 0x843964233da95ab0 + .quad 0xed3cf12d0b356480 + .quad 0x038c77f684817194 + .quad 0xf3c9ed80a2d54245 + .quad 0x0aa08b7877f63952 + .quad 0xd76dac63d1085475 + .quad 0x1ef4fb159470636b + .quad 0x854e5ee65b167bec + .quad 0x59590a4296d0cdc2 + .quad 0x72b2df3498102199 + .quad 0x575ee92a4a0bff56 + + // 2^80 * 7 * G + + .quad 0xd4c080908a182fcf + .quad 0x30e170c299489dbd + .quad 0x05babd5752f733de + .quad 0x43d4e7112cd3fd00 + .quad 0x5d46bc450aa4d801 + .quad 0xc3af1227a533b9d8 + .quad 0x389e3b262b8906c2 + .quad 0x200a1e7e382f581b + .quad 0x518db967eaf93ac5 + .quad 0x71bc989b056652c0 + .quad 0xfe2b85d9567197f5 + .quad 0x050eca52651e4e38 + + // 2^80 * 8 * G + + .quad 0xc3431ade453f0c9c + .quad 0xe9f5045eff703b9b + .quad 0xfcd97ac9ed847b3d + .quad 0x4b0ee6c21c58f4c6 + .quad 0x97ac397660e668ea + .quad 0x9b19bbfe153ab497 + .quad 0x4cb179b534eca79f + .quad 0x6151c09fa131ae57 + .quad 0x3af55c0dfdf05d96 + .quad 0xdd262ee02ab4ee7a + .quad 0x11b2bb8712171709 + .quad 0x1fef24fa800f030b + + // 2^84 * 1 * G + + .quad 0xb496123a6b6c6609 + .quad 0xa750fe8580ab5938 + .quad 0xf471bf39b7c27a5f + .quad 0x507903ce77ac193c + .quad 0xff91a66a90166220 + .quad 0xf22552ae5bf1e009 + .quad 0x7dff85d87f90df7c + .quad 0x4f620ffe0c736fb9 + .quad 0x62f90d65dfde3e34 + .quad 0xcf28c592b9fa5fad + .quad 0x99c86ef9c6164510 + .quad 0x25d448044a256c84 + + // 2^84 * 2 * G + + .quad 0xbd68230ec7e9b16f + .quad 0x0eb1b9c1c1c5795d + .quad 0x7943c8c495b6b1ff + .quad 0x2f9faf620bbacf5e + .quad 0x2c7c4415c9022b55 + .quad 0x56a0d241812eb1fe + .quad 0xf02ea1c9d7b65e0d + .quad 0x4180512fd5323b26 + .quad 0xa4ff3e698a48a5db + .quad 0xba6a3806bd95403b + .quad 0x9f7ce1af47d5b65d + .quad 0x15e087e55939d2fb + + // 2^84 * 3 * G + + .quad 0x12207543745c1496 + .quad 0xdaff3cfdda38610c + .quad 0xe4e797272c71c34f + .quad 0x39c07b1934bdede9 + .quad 0x8894186efb963f38 + .quad 0x48a00e80dc639bd5 + .quad 0xa4e8092be96c1c99 + .quad 0x5a097d54ca573661 + .quad 0x2d45892b17c9e755 + .quad 0xd033fd7289308df8 + .quad 0x6c2fe9d9525b8bd9 + .quad 0x2edbecf1c11cc079 + + // 2^84 * 4 * G + + .quad 0x1616a4e3c715a0d2 + .quad 0x53623cb0f8341d4d + .quad 0x96ef5329c7e899cb + .quad 0x3d4e8dbba668baa6 + .quad 0xee0f0fddd087a25f + .quad 0x9c7531555c3e34ee + .quad 0x660c572e8fab3ab5 + .quad 0x0854fc44544cd3b2 + .quad 0x61eba0c555edad19 + .quad 0x24b533fef0a83de6 + .quad 0x3b77042883baa5f8 + .quad 0x678f82b898a47e8d + + // 2^84 * 5 * G + + .quad 0xb1491d0bd6900c54 + .quad 0x3539722c9d132636 + .quad 0x4db928920b362bc9 + .quad 0x4d7cd1fea68b69df + .quad 0x1e09d94057775696 + .quad 0xeed1265c3cd951db + .quad 0xfa9dac2b20bce16f + .quad 0x0f7f76e0e8d089f4 + .quad 0x36d9ebc5d485b00c + .quad 0xa2596492e4adb365 + .quad 0xc1659480c2119ccd + .quad 0x45306349186e0d5f + + // 2^84 * 6 * G + + .quad 0x94ddd0c1a6cdff1d + .quad 0x55f6f115e84213ae + .quad 0x6c935f85992fcf6a + .quad 0x067ee0f54a37f16f + .quad 0x96a414ec2b072491 + .quad 0x1bb2218127a7b65b + .quad 0x6d2849596e8a4af0 + .quad 0x65f3b08ccd27765f + .quad 0xecb29fff199801f7 + .quad 0x9d361d1fa2a0f72f + .quad 0x25f11d2375fd2f49 + .quad 0x124cefe80fe10fe2 + + // 2^84 * 7 * G + + .quad 0x4c126cf9d18df255 + .quad 0xc1d471e9147a63b6 + .quad 0x2c6d3c73f3c93b5f + .quad 0x6be3a6a2e3ff86a2 + .quad 0x1518e85b31b16489 + .quad 0x8faadcb7db710bfb + .quad 0x39b0bdf4a14ae239 + .quad 0x05f4cbea503d20c1 + .quad 0xce040e9ec04145bc + .quad 0xc71ff4e208f6834c + .quad 0xbd546e8dab8847a3 + .quad 0x64666aa0a4d2aba5 + + // 2^84 * 8 * G + + .quad 0x6841435a7c06d912 + .quad 0xca123c21bb3f830b + .quad 0xd4b37b27b1cbe278 + .quad 0x1d753b84c76f5046 + .quad 0xb0c53bf73337e94c + .quad 0x7cb5697e11e14f15 + .quad 0x4b84abac1930c750 + .quad 0x28dd4abfe0640468 + .quad 0x7dc0b64c44cb9f44 + .quad 0x18a3e1ace3925dbf + .quad 0x7a3034862d0457c4 + .quad 0x4c498bf78a0c892e + + // 2^88 * 1 * G + + .quad 0x37d653fb1aa73196 + .quad 0x0f9495303fd76418 + .quad 0xad200b09fb3a17b2 + .quad 0x544d49292fc8613e + .quad 0x22d2aff530976b86 + .quad 0x8d90b806c2d24604 + .quad 0xdca1896c4de5bae5 + .quad 0x28005fe6c8340c17 + .quad 0x6aefba9f34528688 + .quad 0x5c1bff9425107da1 + .quad 0xf75bbbcd66d94b36 + .quad 0x72e472930f316dfa + + // 2^88 * 2 * G + + .quad 0x2695208c9781084f + .quad 0xb1502a0b23450ee1 + .quad 0xfd9daea603efde02 + .quad 0x5a9d2e8c2733a34c + .quad 0x07f3f635d32a7627 + .quad 0x7aaa4d865f6566f0 + .quad 0x3c85e79728d04450 + .quad 0x1fee7f000fe06438 + .quad 0x765305da03dbf7e5 + .quad 0xa4daf2491434cdbd + .quad 0x7b4ad5cdd24a88ec + .quad 0x00f94051ee040543 + + // 2^88 * 3 * G + + .quad 0x8d356b23c3d330b2 + .quad 0xf21c8b9bb0471b06 + .quad 0xb36c316c6e42b83c + .quad 0x07d79c7e8beab10d + .quad 0xd7ef93bb07af9753 + .quad 0x583ed0cf3db766a7 + .quad 0xce6998bf6e0b1ec5 + .quad 0x47b7ffd25dd40452 + .quad 0x87fbfb9cbc08dd12 + .quad 0x8a066b3ae1eec29b + .quad 0x0d57242bdb1fc1bf + .quad 0x1c3520a35ea64bb6 + + // 2^88 * 4 * G + + .quad 0x80d253a6bccba34a + .quad 0x3e61c3a13838219b + .quad 0x90c3b6019882e396 + .quad 0x1c3d05775d0ee66f + .quad 0xcda86f40216bc059 + .quad 0x1fbb231d12bcd87e + .quad 0xb4956a9e17c70990 + .quad 0x38750c3b66d12e55 + .quad 0x692ef1409422e51a + .quad 0xcbc0c73c2b5df671 + .quad 0x21014fe7744ce029 + .quad 0x0621e2c7d330487c + + // 2^88 * 5 * G + + .quad 0xaf9860cc8259838d + .quad 0x90ea48c1c69f9adc + .quad 0x6526483765581e30 + .quad 0x0007d6097bd3a5bc + .quad 0xb7ae1796b0dbf0f3 + .quad 0x54dfafb9e17ce196 + .quad 0x25923071e9aaa3b4 + .quad 0x5d8e589ca1002e9d + .quad 0xc0bf1d950842a94b + .quad 0xb2d3c363588f2e3e + .quad 0x0a961438bb51e2ef + .quad 0x1583d7783c1cbf86 + + // 2^88 * 6 * G + + .quad 0xeceea2ef5da27ae1 + .quad 0x597c3a1455670174 + .quad 0xc9a62a126609167a + .quad 0x252a5f2e81ed8f70 + .quad 0x90034704cc9d28c7 + .quad 0x1d1b679ef72cc58f + .quad 0x16e12b5fbe5b8726 + .quad 0x4958064e83c5580a + .quad 0x0d2894265066e80d + .quad 0xfcc3f785307c8c6b + .quad 0x1b53da780c1112fd + .quad 0x079c170bd843b388 + + // 2^88 * 7 * G + + .quad 0x0506ece464fa6fff + .quad 0xbee3431e6205e523 + .quad 0x3579422451b8ea42 + .quad 0x6dec05e34ac9fb00 + .quad 0xcdd6cd50c0d5d056 + .quad 0x9af7686dbb03573b + .quad 0x3ca6723ff3c3ef48 + .quad 0x6768c0d7317b8acc + .quad 0x94b625e5f155c1b3 + .quad 0x417bf3a7997b7b91 + .quad 0xc22cbddc6d6b2600 + .quad 0x51445e14ddcd52f4 + + // 2^88 * 8 * G + + .quad 0x57502b4b3b144951 + .quad 0x8e67ff6b444bbcb3 + .quad 0xb8bd6927166385db + .quad 0x13186f31e39295c8 + .quad 0x893147ab2bbea455 + .quad 0x8c53a24f92079129 + .quad 0x4b49f948be30f7a7 + .quad 0x12e990086e4fd43d + .quad 0xf10c96b37fdfbb2e + .quad 0x9f9a935e121ceaf9 + .quad 0xdf1136c43a5b983f + .quad 0x77b2e3f05d3e99af + + // 2^92 * 1 * G + + .quad 0xfd0d75879cf12657 + .quad 0xe82fef94e53a0e29 + .quad 0xcc34a7f05bbb4be7 + .quad 0x0b251172a50c38a2 + .quad 0x9532f48fcc5cd29b + .quad 0x2ba851bea3ce3671 + .quad 0x32dacaa051122941 + .quad 0x478d99d9350004f2 + .quad 0x1d5ad94890bb02c0 + .quad 0x50e208b10ec25115 + .quad 0xa26a22894ef21702 + .quad 0x4dc923343b524805 + + // 2^92 * 2 * G + + .quad 0xe3828c400f8086b6 + .quad 0x3f77e6f7979f0dc8 + .quad 0x7ef6de304df42cb4 + .quad 0x5265797cb6abd784 + .quad 0x3ad3e3ebf36c4975 + .quad 0xd75d25a537862125 + .quad 0xe873943da025a516 + .quad 0x6bbc7cb4c411c847 + .quad 0x3c6f9cd1d4a50d56 + .quad 0xb6244077c6feab7e + .quad 0x6ff9bf483580972e + .quad 0x00375883b332acfb + + // 2^92 * 3 * G + + .quad 0x0001b2cd28cb0940 + .quad 0x63fb51a06f1c24c9 + .quad 0xb5ad8691dcd5ca31 + .quad 0x67238dbd8c450660 + .quad 0xc98bec856c75c99c + .quad 0xe44184c000e33cf4 + .quad 0x0a676b9bba907634 + .quad 0x669e2cb571f379d7 + .quad 0xcb116b73a49bd308 + .quad 0x025aad6b2392729e + .quad 0xb4793efa3f55d9b1 + .quad 0x72a1056140678bb9 + + // 2^92 * 4 * G + + .quad 0xa2b6812b1cc9249d + .quad 0x62866eee21211f58 + .quad 0x2cb5c5b85df10ece + .quad 0x03a6b259e263ae00 + .quad 0x0d8d2909e2e505b6 + .quad 0x98ca78abc0291230 + .quad 0x77ef5569a9b12327 + .quad 0x7c77897b81439b47 + .quad 0xf1c1b5e2de331cb5 + .quad 0x5a9f5d8e15fca420 + .quad 0x9fa438f17bd932b1 + .quad 0x2a381bf01c6146e7 + + // 2^92 * 5 * G + + .quad 0xac9b9879cfc811c1 + .quad 0x8b7d29813756e567 + .quad 0x50da4e607c70edfc + .quad 0x5dbca62f884400b6 + .quad 0xf7c0be32b534166f + .quad 0x27e6ca6419cf70d4 + .quad 0x934df7d7a957a759 + .quad 0x5701461dabdec2aa + .quad 0x2c6747402c915c25 + .quad 0x1bdcd1a80b0d340a + .quad 0x5e5601bd07b43f5f + .quad 0x2555b4e05539a242 + + // 2^92 * 6 * G + + .quad 0x6fc09f5266ddd216 + .quad 0xdce560a7c8e37048 + .quad 0xec65939da2df62fd + .quad 0x7a869ae7e52ed192 + .quad 0x78409b1d87e463d4 + .quad 0xad4da95acdfb639d + .quad 0xec28773755259b9c + .quad 0x69c806e9c31230ab + .quad 0x7b48f57414bb3f22 + .quad 0x68c7cee4aedccc88 + .quad 0xed2f936179ed80be + .quad 0x25d70b885f77bc4b + + // 2^92 * 7 * G + + .quad 0x4151c3d9762bf4de + .quad 0x083f435f2745d82b + .quad 0x29775a2e0d23ddd5 + .quad 0x138e3a6269a5db24 + .quad 0x98459d29bb1ae4d4 + .quad 0x56b9c4c739f954ec + .quad 0x832743f6c29b4b3e + .quad 0x21ea8e2798b6878a + .quad 0x87bef4b46a5a7b9c + .quad 0xd2299d1b5fc1d062 + .quad 0x82409818dd321648 + .quad 0x5c5abeb1e5a2e03d + + // 2^92 * 8 * G + + .quad 0x14722af4b73c2ddb + .quad 0xbc470c5f5a05060d + .quad 0x00943eac2581b02e + .quad 0x0e434b3b1f499c8f + .quad 0x02cde6de1306a233 + .quad 0x7b5a52a2116f8ec7 + .quad 0xe1c681f4c1163b5b + .quad 0x241d350660d32643 + .quad 0x6be4404d0ebc52c7 + .quad 0xae46233bb1a791f5 + .quad 0x2aec170ed25db42b + .quad 0x1d8dfd966645d694 + + // 2^96 * 1 * G + + .quad 0x296fa9c59c2ec4de + .quad 0xbc8b61bf4f84f3cb + .quad 0x1c7706d917a8f908 + .quad 0x63b795fc7ad3255d + .quad 0xd598639c12ddb0a4 + .quad 0xa5d19f30c024866b + .quad 0xd17c2f0358fce460 + .quad 0x07a195152e095e8a + .quad 0xa8368f02389e5fc8 + .quad 0x90433b02cf8de43b + .quad 0xafa1fd5dc5412643 + .quad 0x3e8fe83d032f0137 + + // 2^96 * 2 * G + + .quad 0x2f8b15b90570a294 + .quad 0x94f2427067084549 + .quad 0xde1c5ae161bbfd84 + .quad 0x75ba3b797fac4007 + .quad 0x08704c8de8efd13c + .quad 0xdfc51a8e33e03731 + .quad 0xa59d5da51260cde3 + .quad 0x22d60899a6258c86 + .quad 0x6239dbc070cdd196 + .quad 0x60fe8a8b6c7d8a9a + .quad 0xb38847bceb401260 + .quad 0x0904d07b87779e5e + + // 2^96 * 3 * G + + .quad 0xb4ce1fd4ddba919c + .quad 0xcf31db3ec74c8daa + .quad 0x2c63cc63ad86cc51 + .quad 0x43e2143fbc1dde07 + .quad 0xf4322d6648f940b9 + .quad 0x06952f0cbd2d0c39 + .quad 0x167697ada081f931 + .quad 0x6240aacebaf72a6c + .quad 0xf834749c5ba295a0 + .quad 0xd6947c5bca37d25a + .quad 0x66f13ba7e7c9316a + .quad 0x56bdaf238db40cac + + // 2^96 * 4 * G + + .quad 0x362ab9e3f53533eb + .quad 0x338568d56eb93d40 + .quad 0x9e0e14521d5a5572 + .quad 0x1d24a86d83741318 + .quad 0x1310d36cc19d3bb2 + .quad 0x062a6bb7622386b9 + .quad 0x7c9b8591d7a14f5c + .quad 0x03aa31507e1e5754 + .quad 0xf4ec7648ffd4ce1f + .quad 0xe045eaf054ac8c1c + .quad 0x88d225821d09357c + .quad 0x43b261dc9aeb4859 + + // 2^96 * 5 * G + + .quad 0xe55b1e1988bb79bb + .quad 0xa09ed07dc17a359d + .quad 0xb02c2ee2603dea33 + .quad 0x326055cf5b276bc2 + .quad 0x19513d8b6c951364 + .quad 0x94fe7126000bf47b + .quad 0x028d10ddd54f9567 + .quad 0x02b4d5e242940964 + .quad 0xb4a155cb28d18df2 + .quad 0xeacc4646186ce508 + .quad 0xc49cf4936c824389 + .quad 0x27a6c809ae5d3410 + + // 2^96 * 6 * G + + .quad 0x8ba6ebcd1f0db188 + .quad 0x37d3d73a675a5be8 + .quad 0xf22edfa315f5585a + .quad 0x2cb67174ff60a17e + .quad 0xcd2c270ac43d6954 + .quad 0xdd4a3e576a66cab2 + .quad 0x79fa592469d7036c + .quad 0x221503603d8c2599 + .quad 0x59eecdf9390be1d0 + .quad 0xa9422044728ce3f1 + .quad 0x82891c667a94f0f4 + .quad 0x7b1df4b73890f436 + + // 2^96 * 7 * G + + .quad 0xe492f2e0b3b2a224 + .quad 0x7c6c9e062b551160 + .quad 0x15eb8fe20d7f7b0e + .quad 0x61fcef2658fc5992 + .quad 0x5f2e221807f8f58c + .quad 0xe3555c9fd49409d4 + .quad 0xb2aaa88d1fb6a630 + .quad 0x68698245d352e03d + .quad 0xdbb15d852a18187a + .quad 0xf3e4aad386ddacd7 + .quad 0x44bae2810ff6c482 + .quad 0x46cf4c473daf01cf + + // 2^96 * 8 * G + + .quad 0x426525ed9ec4e5f9 + .quad 0x0e5eda0116903303 + .quad 0x72b1a7f2cbe5cadc + .quad 0x29387bcd14eb5f40 + .quad 0x213c6ea7f1498140 + .quad 0x7c1e7ef8392b4854 + .quad 0x2488c38c5629ceba + .quad 0x1065aae50d8cc5bb + .quad 0x1c2c4525df200d57 + .quad 0x5c3b2dd6bfca674a + .quad 0x0a07e7b1e1834030 + .quad 0x69a198e64f1ce716 + + // 2^100 * 1 * G + + .quad 0x7afcd613efa9d697 + .quad 0x0cc45aa41c067959 + .quad 0xa56fe104c1fada96 + .quad 0x3a73b70472e40365 + .quad 0x7b26e56b9e2d4734 + .quad 0xc4c7132b81c61675 + .quad 0xef5c9525ec9cde7f + .quad 0x39c80b16e71743ad + .quad 0x0f196e0d1b826c68 + .quad 0xf71ff0e24960e3db + .quad 0x6113167023b7436c + .quad 0x0cf0ea5877da7282 + + // 2^100 * 2 * G + + .quad 0x196c80a4ddd4ccbd + .quad 0x22e6f55d95f2dd9d + .quad 0xc75e33c740d6c71b + .quad 0x7bb51279cb3c042f + .quad 0xe332ced43ba6945a + .quad 0xde0b1361e881c05d + .quad 0x1ad40f095e67ed3b + .quad 0x5da8acdab8c63d5d + .quad 0xc4b6664a3a70159f + .quad 0x76194f0f0a904e14 + .quad 0xa5614c39a4096c13 + .quad 0x6cd0ff50979feced + + // 2^100 * 3 * G + + .quad 0xc0e067e78f4428ac + .quad 0x14835ab0a61135e3 + .quad 0xf21d14f338062935 + .quad 0x6390a4c8df04849c + .quad 0x7fecfabdb04ba18e + .quad 0xd0fc7bfc3bddbcf7 + .quad 0xa41d486e057a131c + .quad 0x641a4391f2223a61 + .quad 0xc5c6b95aa606a8db + .quad 0x914b7f9eb06825f1 + .quad 0x2a731f6b44fc9eff + .quad 0x30ddf38562705cfc + + // 2^100 * 4 * G + + .quad 0x4e3dcbdad1bff7f9 + .quad 0xc9118e8220645717 + .quad 0xbacccebc0f189d56 + .quad 0x1b4822e9d4467668 + .quad 0x33bef2bd68bcd52c + .quad 0xc649dbb069482ef2 + .quad 0xb5b6ee0c41cb1aee + .quad 0x5c294d270212a7e5 + .quad 0xab360a7f25563781 + .quad 0x2512228a480f7958 + .quad 0xc75d05276114b4e3 + .quad 0x222d9625d976fe2a + + // 2^100 * 5 * G + + .quad 0x1c717f85b372ace1 + .quad 0x81930e694638bf18 + .quad 0x239cad056bc08b58 + .quad 0x0b34271c87f8fff4 + .quad 0x0f94be7e0a344f85 + .quad 0xeb2faa8c87f22c38 + .quad 0x9ce1e75e4ee16f0f + .quad 0x43e64e5418a08dea + .quad 0x8155e2521a35ce63 + .quad 0xbe100d4df912028e + .quad 0xbff80bf8a57ddcec + .quad 0x57342dc96d6bc6e4 + + // 2^100 * 6 * G + + .quad 0xefeef065c8ce5998 + .quad 0xbf029510b5cbeaa2 + .quad 0x8c64a10620b7c458 + .quad 0x35134fb231c24855 + .quad 0xf3c3bcb71e707bf6 + .quad 0x351d9b8c7291a762 + .quad 0x00502e6edad69a33 + .quad 0x522f521f1ec8807f + .quad 0x272c1f46f9a3902b + .quad 0xc91ba3b799657bcc + .quad 0xae614b304f8a1c0e + .quad 0x7afcaad70b99017b + + // 2^100 * 7 * G + + .quad 0xc25ded54a4b8be41 + .quad 0x902d13e11bb0e2dd + .quad 0x41f43233cde82ab2 + .quad 0x1085faa5c3aae7cb + .quad 0xa88141ecef842b6b + .quad 0x55e7b14797abe6c5 + .quad 0x8c748f9703784ffe + .quad 0x5b50a1f7afcd00b7 + .quad 0x9b840f66f1361315 + .quad 0x18462242701003e9 + .quad 0x65ed45fae4a25080 + .quad 0x0a2862393fda7320 + + // 2^100 * 8 * G + + .quad 0x46ab13c8347cbc9d + .quad 0x3849e8d499c12383 + .quad 0x4cea314087d64ac9 + .quad 0x1f354134b1a29ee7 + .quad 0x960e737b6ecb9d17 + .quad 0xfaf24948d67ceae1 + .quad 0x37e7a9b4d55e1b89 + .quad 0x5cb7173cb46c59eb + .quad 0x4a89e68b82b7abf0 + .quad 0xf41cd9279ba6b7b9 + .quad 0x16e6c210e18d876f + .quad 0x7cacdb0f7f1b09c6 + + // 2^104 * 1 * G + + .quad 0x9062b2e0d91a78bc + .quad 0x47c9889cc8509667 + .quad 0x9df54a66405070b8 + .quad 0x7369e6a92493a1bf + .quad 0xe1014434dcc5caed + .quad 0x47ed5d963c84fb33 + .quad 0x70019576ed86a0e7 + .quad 0x25b2697bd267f9e4 + .quad 0x9d673ffb13986864 + .quad 0x3ca5fbd9415dc7b8 + .quad 0xe04ecc3bdf273b5e + .quad 0x1420683db54e4cd2 + + // 2^104 * 2 * G + + .quad 0xb478bd1e249dd197 + .quad 0x620c35005e58c102 + .quad 0xfb02d32fccbaac5c + .quad 0x60b63bebf508a72d + .quad 0x34eebb6fc1cc5ad0 + .quad 0x6a1b0ce99646ac8b + .quad 0xd3b0da49a66bde53 + .quad 0x31e83b4161d081c1 + .quad 0x97e8c7129e062b4f + .quad 0x49e48f4f29320ad8 + .quad 0x5bece14b6f18683f + .quad 0x55cf1eb62d550317 + + // 2^104 * 3 * G + + .quad 0x5879101065c23d58 + .quad 0x8b9d086d5094819c + .quad 0xe2402fa912c55fa7 + .quad 0x669a6564570891d4 + .quad 0x3076b5e37df58c52 + .quad 0xd73ab9dde799cc36 + .quad 0xbd831ce34913ee20 + .quad 0x1a56fbaa62ba0133 + .quad 0x943e6b505c9dc9ec + .quad 0x302557bba77c371a + .quad 0x9873ae5641347651 + .quad 0x13c4836799c58a5c + + // 2^104 * 4 * G + + .quad 0x423a5d465ab3e1b9 + .quad 0xfc13c187c7f13f61 + .quad 0x19f83664ecb5b9b6 + .quad 0x66f80c93a637b607 + .quad 0xc4dcfb6a5d8bd080 + .quad 0xdeebc4ec571a4842 + .quad 0xd4b2e883b8e55365 + .quad 0x50bdc87dc8e5b827 + .quad 0x606d37836edfe111 + .quad 0x32353e15f011abd9 + .quad 0x64b03ac325b73b96 + .quad 0x1dd56444725fd5ae + + // 2^104 * 5 * G + + .quad 0x8fa47ff83362127d + .quad 0xbc9f6ac471cd7c15 + .quad 0x6e71454349220c8b + .quad 0x0e645912219f732e + .quad 0xc297e60008bac89a + .quad 0x7d4cea11eae1c3e0 + .quad 0xf3e38be19fe7977c + .quad 0x3a3a450f63a305cd + .quad 0x078f2f31d8394627 + .quad 0x389d3183de94a510 + .quad 0xd1e36c6d17996f80 + .quad 0x318c8d9393a9a87b + + // 2^104 * 6 * G + + .quad 0xf2745d032afffe19 + .quad 0x0c9f3c497f24db66 + .quad 0xbc98d3e3ba8598ef + .quad 0x224c7c679a1d5314 + .quad 0x5d669e29ab1dd398 + .quad 0xfc921658342d9e3b + .quad 0x55851dfdf35973cd + .quad 0x509a41c325950af6 + .quad 0xbdc06edca6f925e9 + .quad 0x793ef3f4641b1f33 + .quad 0x82ec12809d833e89 + .quad 0x05bff02328a11389 + + // 2^104 * 7 * G + + .quad 0x3632137023cae00b + .quad 0x544acf0ad1accf59 + .quad 0x96741049d21a1c88 + .quad 0x780b8cc3fa2a44a7 + .quad 0x6881a0dd0dc512e4 + .quad 0x4fe70dc844a5fafe + .quad 0x1f748e6b8f4a5240 + .quad 0x576277cdee01a3ea + .quad 0x1ef38abc234f305f + .quad 0x9a577fbd1405de08 + .quad 0x5e82a51434e62a0d + .quad 0x5ff418726271b7a1 + + // 2^104 * 8 * G + + .quad 0x398e080c1789db9d + .quad 0xa7602025f3e778f5 + .quad 0xfa98894c06bd035d + .quad 0x106a03dc25a966be + .quad 0xe5db47e813b69540 + .quad 0xf35d2a3b432610e1 + .quad 0xac1f26e938781276 + .quad 0x29d4db8ca0a0cb69 + .quad 0xd9ad0aaf333353d0 + .quad 0x38669da5acd309e5 + .quad 0x3c57658ac888f7f0 + .quad 0x4ab38a51052cbefa + + // 2^108 * 1 * G + + .quad 0xdfdacbee4324c0e9 + .quad 0x054442883f955bb7 + .quad 0xdef7aaa8ea31609f + .quad 0x68aee70642287cff + .quad 0xf68fe2e8809de054 + .quad 0xe3bc096a9c82bad1 + .quad 0x076353d40aadbf45 + .quad 0x7b9b1fb5dea1959e + .quad 0xf01cc8f17471cc0c + .quad 0x95242e37579082bb + .quad 0x27776093d3e46b5f + .quad 0x2d13d55a28bd85fb + + // 2^108 * 2 * G + + .quad 0xfac5d2065b35b8da + .quad 0xa8da8a9a85624bb7 + .quad 0xccd2ca913d21cd0f + .quad 0x6b8341ee8bf90d58 + .quad 0xbf019cce7aee7a52 + .quad 0xa8ded2b6e454ead3 + .quad 0x3c619f0b87a8bb19 + .quad 0x3619b5d7560916d8 + .quad 0x3579f26b0282c4b2 + .quad 0x64d592f24fafefae + .quad 0xb7cded7b28c8c7c0 + .quad 0x6a927b6b7173a8d7 + + // 2^108 * 3 * G + + .quad 0x1f6db24f986e4656 + .quad 0x1021c02ed1e9105b + .quad 0xf8ff3fff2cc0a375 + .quad 0x1d2a6bf8c6c82592 + .quad 0x8d7040863ece88eb + .quad 0xf0e307a980eec08c + .quad 0xac2250610d788fda + .quad 0x056d92a43a0d478d + .quad 0x1b05a196fc3da5a1 + .quad 0x77d7a8c243b59ed0 + .quad 0x06da3d6297d17918 + .quad 0x66fbb494f12353f7 + + // 2^108 * 4 * G + + .quad 0x751a50b9d85c0fb8 + .quad 0xd1afdc258bcf097b + .quad 0x2f16a6a38309a969 + .quad 0x14ddff9ee5b00659 + .quad 0xd6d70996f12309d6 + .quad 0xdbfb2385e9c3d539 + .quad 0x46d602b0f7552411 + .quad 0x270a0b0557843e0c + .quad 0x61ff0640a7862bcc + .quad 0x81cac09a5f11abfe + .quad 0x9047830455d12abb + .quad 0x19a4bde1945ae873 + + // 2^108 * 5 * G + + .quad 0x9b9f26f520a6200a + .quad 0x64804443cf13eaf8 + .quad 0x8a63673f8631edd3 + .quad 0x72bbbce11ed39dc1 + .quad 0x40c709dec076c49f + .quad 0x657bfaf27f3e53f6 + .quad 0x40662331eca042c4 + .quad 0x14b375487eb4df04 + .quad 0xae853c94ab66dc47 + .quad 0xeb62343edf762d6e + .quad 0xf08e0e186fb2f7d1 + .quad 0x4f0b1c02700ab37a + + // 2^108 * 6 * G + + .quad 0xe1706787d81951fa + .quad 0xa10a2c8eb290c77b + .quad 0xe7382fa03ed66773 + .quad 0x0a4d84710bcc4b54 + .quad 0x79fd21ccc1b2e23f + .quad 0x4ae7c281453df52a + .quad 0xc8172ec9d151486b + .quad 0x68abe9443e0a7534 + .quad 0xda12c6c407831dcb + .quad 0x0da230d74d5c510d + .quad 0x4ab1531e6bd404e1 + .quad 0x4106b166bcf440ef + + // 2^108 * 7 * G + + .quad 0x02e57a421cd23668 + .quad 0x4ad9fb5d0eaef6fd + .quad 0x954e6727b1244480 + .quad 0x7f792f9d2699f331 + .quad 0xa485ccd539e4ecf2 + .quad 0x5aa3f3ad0555bab5 + .quad 0x145e3439937df82d + .quad 0x1238b51e1214283f + .quad 0x0b886b925fd4d924 + .quad 0x60906f7a3626a80d + .quad 0xecd367b4b98abd12 + .quad 0x2876beb1def344cf + + // 2^108 * 8 * G + + .quad 0xdc84e93563144691 + .quad 0x632fe8a0d61f23f4 + .quad 0x4caa800612a9a8d5 + .quad 0x48f9dbfa0e9918d3 + .quad 0xd594b3333a8a85f8 + .quad 0x4ea37689e78d7d58 + .quad 0x73bf9f455e8e351f + .quad 0x5507d7d2bc41ebb4 + .quad 0x1ceb2903299572fc + .quad 0x7c8ccaa29502d0ee + .quad 0x91bfa43411cce67b + .quad 0x5784481964a831e7 + + // 2^112 * 1 * G + + .quad 0xda7c2b256768d593 + .quad 0x98c1c0574422ca13 + .quad 0xf1a80bd5ca0ace1d + .quad 0x29cdd1adc088a690 + .quad 0xd6cfd1ef5fddc09c + .quad 0xe82b3efdf7575dce + .quad 0x25d56b5d201634c2 + .quad 0x3041c6bb04ed2b9b + .quad 0x0ff2f2f9d956e148 + .quad 0xade797759f356b2e + .quad 0x1a4698bb5f6c025c + .quad 0x104bbd6814049a7b + + // 2^112 * 2 * G + + .quad 0x51f0fd3168f1ed67 + .quad 0x2c811dcdd86f3bc2 + .quad 0x44dc5c4304d2f2de + .quad 0x5be8cc57092a7149 + .quad 0xa95d9a5fd67ff163 + .quad 0xe92be69d4cc75681 + .quad 0xb7f8024cde20f257 + .quad 0x204f2a20fb072df5 + .quad 0xc8143b3d30ebb079 + .quad 0x7589155abd652e30 + .quad 0x653c3c318f6d5c31 + .quad 0x2570fb17c279161f + + // 2^112 * 3 * G + + .quad 0x3efa367f2cb61575 + .quad 0xf5f96f761cd6026c + .quad 0xe8c7142a65b52562 + .quad 0x3dcb65ea53030acd + .quad 0x192ea9550bb8245a + .quad 0xc8e6fba88f9050d1 + .quad 0x7986ea2d88a4c935 + .quad 0x241c5f91de018668 + .quad 0x28d8172940de6caa + .quad 0x8fbf2cf022d9733a + .quad 0x16d7fcdd235b01d1 + .quad 0x08420edd5fcdf0e5 + + // 2^112 * 4 * G + + .quad 0xcdff20ab8362fa4a + .quad 0x57e118d4e21a3e6e + .quad 0xe3179617fc39e62b + .quad 0x0d9a53efbc1769fd + .quad 0x0358c34e04f410ce + .quad 0xb6135b5a276e0685 + .quad 0x5d9670c7ebb91521 + .quad 0x04d654f321db889c + .quad 0x5e7dc116ddbdb5d5 + .quad 0x2954deb68da5dd2d + .quad 0x1cb608173334a292 + .quad 0x4a7a4f2618991ad7 + + // 2^112 * 5 * G + + .quad 0xf4a718025fb15f95 + .quad 0x3df65f346b5c1b8f + .quad 0xcdfcf08500e01112 + .quad 0x11b50c4cddd31848 + .quad 0x24c3b291af372a4b + .quad 0x93da8270718147f2 + .quad 0xdd84856486899ef2 + .quad 0x4a96314223e0ee33 + .quad 0xa6e8274408a4ffd6 + .quad 0x738e177e9c1576d9 + .quad 0x773348b63d02b3f2 + .quad 0x4f4bce4dce6bcc51 + + // 2^112 * 6 * G + + .quad 0xa71fce5ae2242584 + .quad 0x26ea725692f58a9e + .quad 0xd21a09d71cea3cf4 + .quad 0x73fcdd14b71c01e6 + .quad 0x30e2616ec49d0b6f + .quad 0xe456718fcaec2317 + .quad 0x48eb409bf26b4fa6 + .quad 0x3042cee561595f37 + .quad 0x427e7079449bac41 + .quad 0x855ae36dbce2310a + .quad 0x4cae76215f841a7c + .quad 0x389e740c9a9ce1d6 + + // 2^112 * 7 * G + + .quad 0x64fcb3ae34dcb9ce + .quad 0x97500323e348d0ad + .quad 0x45b3f07d62c6381b + .quad 0x61545379465a6788 + .quad 0xc9bd78f6570eac28 + .quad 0xe55b0b3227919ce1 + .quad 0x65fc3eaba19b91ed + .quad 0x25c425e5d6263690 + .quad 0x3f3e06a6f1d7de6e + .quad 0x3ef976278e062308 + .quad 0x8c14f6264e8a6c77 + .quad 0x6539a08915484759 + + // 2^112 * 8 * G + + .quad 0xe9d21f74c3d2f773 + .quad 0xc150544125c46845 + .quad 0x624e5ce8f9b99e33 + .quad 0x11c5e4aac5cd186c + .quad 0xddc4dbd414bb4a19 + .quad 0x19b2bc3c98424f8e + .quad 0x48a89fd736ca7169 + .quad 0x0f65320ef019bd90 + .quad 0xd486d1b1cafde0c6 + .quad 0x4f3fe6e3163b5181 + .quad 0x59a8af0dfaf2939a + .quad 0x4cabc7bdec33072a + + // 2^116 * 1 * G + + .quad 0x16faa8fb532f7428 + .quad 0xdbd42ea046a4e272 + .quad 0x5337653b8b9ea480 + .quad 0x4065947223973f03 + .quad 0xf7c0a19c1a54a044 + .quad 0x4a1c5e2477bd9fbb + .quad 0xa6e3ca115af22972 + .quad 0x1819bb953f2e9e0d + .quad 0x498fbb795e042e84 + .quad 0x7d0dd89a7698b714 + .quad 0x8bfb0ba427fe6295 + .quad 0x36ba82e721200524 + + // 2^116 * 2 * G + + .quad 0xd60ecbb74245ec41 + .quad 0xfd9be89e34348716 + .quad 0xc9240afee42284de + .quad 0x4472f648d0531db4 + .quad 0xc8d69d0a57274ed5 + .quad 0x45ba803260804b17 + .quad 0xdf3cda102255dfac + .quad 0x77d221232709b339 + .quad 0x498a6d7064ad94d8 + .quad 0xa5b5c8fd9af62263 + .quad 0x8ca8ed0545c141f4 + .quad 0x2c63bec3662d358c + + // 2^116 * 3 * G + + .quad 0x7fe60d8bea787955 + .quad 0xb9dc117eb5f401b7 + .quad 0x91c7c09a19355cce + .quad 0x22692ef59442bedf + .quad 0x9a518b3a8586f8bf + .quad 0x9ee71af6cbb196f0 + .quad 0xaa0625e6a2385cf2 + .quad 0x1deb2176ddd7c8d1 + .quad 0x8563d19a2066cf6c + .quad 0x401bfd8c4dcc7cd7 + .quad 0xd976a6becd0d8f62 + .quad 0x67cfd773a278b05e + + // 2^116 * 4 * G + + .quad 0x8dec31faef3ee475 + .quad 0x99dbff8a9e22fd92 + .quad 0x512d11594e26cab1 + .quad 0x0cde561eec4310b9 + .quad 0x2d5fa9855a4e586a + .quad 0x65f8f7a449beab7e + .quad 0xaa074dddf21d33d3 + .quad 0x185cba721bcb9dee + .quad 0x93869da3f4e3cb41 + .quad 0xbf0392f540f7977e + .quad 0x026204fcd0463b83 + .quad 0x3ec91a769eec6eed + + // 2^116 * 5 * G + + .quad 0x1e9df75bf78166ad + .quad 0x4dfda838eb0cd7af + .quad 0xba002ed8c1eaf988 + .quad 0x13fedb3e11f33cfc + .quad 0x0fad2fb7b0a3402f + .quad 0x46615ecbfb69f4a8 + .quad 0xf745bcc8c5f8eaa6 + .quad 0x7a5fa8794a94e896 + .quad 0x52958faa13cd67a1 + .quad 0x965ee0818bdbb517 + .quad 0x16e58daa2e8845b3 + .quad 0x357d397d5499da8f + + // 2^116 * 6 * G + + .quad 0x1ebfa05fb0bace6c + .quad 0xc934620c1caf9a1e + .quad 0xcc771cc41d82b61a + .quad 0x2d94a16aa5f74fec + .quad 0x481dacb4194bfbf8 + .quad 0x4d77e3f1bae58299 + .quad 0x1ef4612e7d1372a0 + .quad 0x3a8d867e70ff69e1 + .quad 0x6f58cd5d55aff958 + .quad 0xba3eaa5c75567721 + .quad 0x75c123999165227d + .quad 0x69be1343c2f2b35e + + // 2^116 * 7 * G + + .quad 0x0e091d5ee197c92a + .quad 0x4f51019f2945119f + .quad 0x143679b9f034e99c + .quad 0x7d88112e4d24c696 + .quad 0x82bbbdac684b8de3 + .quad 0xa2f4c7d03fca0718 + .quad 0x337f92fbe096aaa8 + .quad 0x200d4d8c63587376 + .quad 0x208aed4b4893b32b + .quad 0x3efbf23ebe59b964 + .quad 0xd762deb0dba5e507 + .quad 0x69607bd681bd9d94 + + // 2^116 * 8 * G + + .quad 0xf6be021068de1ce1 + .quad 0xe8d518e70edcbc1f + .quad 0xe3effdd01b5505a5 + .quad 0x35f63353d3ec3fd0 + .quad 0x3b7f3bd49323a902 + .quad 0x7c21b5566b2c6e53 + .quad 0xe5ba8ff53a7852a7 + .quad 0x28bc77a5838ece00 + .quad 0x63ba78a8e25d8036 + .quad 0x63651e0094333490 + .quad 0x48d82f20288ce532 + .quad 0x3a31abfa36b57524 + + // 2^120 * 1 * G + + .quad 0x239e9624089c0a2e + .quad 0xc748c4c03afe4738 + .quad 0x17dbed2a764fa12a + .quad 0x639b93f0321c8582 + .quad 0xc08f788f3f78d289 + .quad 0xfe30a72ca1404d9f + .quad 0xf2778bfccf65cc9d + .quad 0x7ee498165acb2021 + .quad 0x7bd508e39111a1c3 + .quad 0x2b2b90d480907489 + .quad 0xe7d2aec2ae72fd19 + .quad 0x0edf493c85b602a6 + + // 2^120 * 2 * G + + .quad 0xaecc8158599b5a68 + .quad 0xea574f0febade20e + .quad 0x4fe41d7422b67f07 + .quad 0x403b92e3019d4fb4 + .quad 0x6767c4d284764113 + .quad 0xa090403ff7f5f835 + .quad 0x1c8fcffacae6bede + .quad 0x04c00c54d1dfa369 + .quad 0x4dc22f818b465cf8 + .quad 0x71a0f35a1480eff8 + .quad 0xaee8bfad04c7d657 + .quad 0x355bb12ab26176f4 + + // 2^120 * 3 * G + + .quad 0xa71e64cc7493bbf4 + .quad 0xe5bd84d9eca3b0c3 + .quad 0x0a6bc50cfa05e785 + .quad 0x0f9b8132182ec312 + .quad 0xa301dac75a8c7318 + .quad 0xed90039db3ceaa11 + .quad 0x6f077cbf3bae3f2d + .quad 0x7518eaf8e052ad8e + .quad 0xa48859c41b7f6c32 + .quad 0x0f2d60bcf4383298 + .quad 0x1815a929c9b1d1d9 + .quad 0x47c3871bbb1755c4 + + // 2^120 * 4 * G + + .quad 0x5144539771ec4f48 + .quad 0xf805b17dc98c5d6e + .quad 0xf762c11a47c3c66b + .quad 0x00b89b85764699dc + .quad 0xfbe65d50c85066b0 + .quad 0x62ecc4b0b3a299b0 + .quad 0xe53754ea441ae8e0 + .quad 0x08fea02ce8d48d5f + .quad 0x824ddd7668deead0 + .quad 0xc86445204b685d23 + .quad 0xb514cfcd5d89d665 + .quad 0x473829a74f75d537 + + // 2^120 * 5 * G + + .quad 0x82d2da754679c418 + .quad 0xe63bd7d8b2618df0 + .quad 0x355eef24ac47eb0a + .quad 0x2078684c4833c6b4 + .quad 0x23d9533aad3902c9 + .quad 0x64c2ddceef03588f + .quad 0x15257390cfe12fb4 + .quad 0x6c668b4d44e4d390 + .quad 0x3b48cf217a78820c + .quad 0xf76a0ab281273e97 + .quad 0xa96c65a78c8eed7b + .quad 0x7411a6054f8a433f + + // 2^120 * 6 * G + + .quad 0x4d659d32b99dc86d + .quad 0x044cdc75603af115 + .quad 0xb34c712cdcc2e488 + .quad 0x7c136574fb8134ff + .quad 0x579ae53d18b175b4 + .quad 0x68713159f392a102 + .quad 0x8455ecba1eef35f5 + .quad 0x1ec9a872458c398f + .quad 0xb8e6a4d400a2509b + .quad 0x9b81d7020bc882b4 + .quad 0x57e7cc9bf1957561 + .quad 0x3add88a5c7cd6460 + + // 2^120 * 7 * G + + .quad 0xab895770b635dcf2 + .quad 0x02dfef6cf66c1fbc + .quad 0x85530268beb6d187 + .quad 0x249929fccc879e74 + .quad 0x85c298d459393046 + .quad 0x8f7e35985ff659ec + .quad 0x1d2ca22af2f66e3a + .quad 0x61ba1131a406a720 + .quad 0xa3d0a0f116959029 + .quad 0x023b6b6cba7ebd89 + .quad 0x7bf15a3e26783307 + .quad 0x5620310cbbd8ece7 + + // 2^120 * 8 * G + + .quad 0x528993434934d643 + .quad 0xb9dbf806a51222f5 + .quad 0x8f6d878fc3f41c22 + .quad 0x37676a2a4d9d9730 + .quad 0x6646b5f477e285d6 + .quad 0x40e8ff676c8f6193 + .quad 0xa6ec7311abb594dd + .quad 0x7ec846f3658cec4d + .quad 0x9b5e8f3f1da22ec7 + .quad 0x130f1d776c01cd13 + .quad 0x214c8fcfa2989fb8 + .quad 0x6daaf723399b9dd5 + + // 2^124 * 1 * G + + .quad 0x591e4a5610628564 + .quad 0x2a4bb87ca8b4df34 + .quad 0xde2a2572e7a38e43 + .quad 0x3cbdabd9fee5046e + .quad 0x81aebbdd2cd13070 + .quad 0x962e4325f85a0e9e + .quad 0xde9391aacadffecb + .quad 0x53177fda52c230e6 + .quad 0xa7bc970650b9de79 + .quad 0x3d12a7fbc301b59b + .quad 0x02652e68d36ae38c + .quad 0x79d739835a6199dc + + // 2^124 * 2 * G + + .quad 0xd9354df64131c1bd + .quad 0x758094a186ec5822 + .quad 0x4464ee12e459f3c2 + .quad 0x6c11fce4cb133282 + .quad 0x21c9d9920d591737 + .quad 0x9bea41d2e9b46cd6 + .quad 0xe20e84200d89bfca + .quad 0x79d99f946eae5ff8 + .quad 0xf17b483568673205 + .quad 0x387deae83caad96c + .quad 0x61b471fd56ffe386 + .quad 0x31741195b745a599 + + // 2^124 * 3 * G + + .quad 0xe8d10190b77a360b + .quad 0x99b983209995e702 + .quad 0xbd4fdff8fa0247aa + .quad 0x2772e344e0d36a87 + .quad 0x17f8ba683b02a047 + .quad 0x50212096feefb6c8 + .quad 0x70139be21556cbe2 + .quad 0x203e44a11d98915b + .quad 0xd6863eba37b9e39f + .quad 0x105bc169723b5a23 + .quad 0x104f6459a65c0762 + .quad 0x567951295b4d38d4 + + // 2^124 * 4 * G + + .quad 0x535fd60613037524 + .quad 0xe210adf6b0fbc26a + .quad 0xac8d0a9b23e990ae + .quad 0x47204d08d72fdbf9 + .quad 0x07242eb30d4b497f + .quad 0x1ef96306b9bccc87 + .quad 0x37950934d8116f45 + .quad 0x05468d6201405b04 + .quad 0x00f565a9f93267de + .quad 0xcecfd78dc0d58e8a + .quad 0xa215e2dcf318e28e + .quad 0x4599ee919b633352 + + // 2^124 * 5 * G + + .quad 0xd3c220ca70e0e76b + .quad 0xb12bea58ea9f3094 + .quad 0x294ddec8c3271282 + .quad 0x0c3539e1a1d1d028 + .quad 0xac746d6b861ae579 + .quad 0x31ab0650f6aea9dc + .quad 0x241d661140256d4c + .quad 0x2f485e853d21a5de + .quad 0x329744839c0833f3 + .quad 0x6fe6257fd2abc484 + .quad 0x5327d1814b358817 + .quad 0x65712585893fe9bc + + // 2^124 * 6 * G + + .quad 0x9c102fb732a61161 + .quad 0xe48e10dd34d520a8 + .quad 0x365c63546f9a9176 + .quad 0x32f6fe4c046f6006 + .quad 0x81c29f1bd708ee3f + .quad 0xddcb5a05ae6407d0 + .quad 0x97aec1d7d2a3eba7 + .quad 0x1590521a91d50831 + .quad 0x40a3a11ec7910acc + .quad 0x9013dff8f16d27ae + .quad 0x1a9720d8abb195d4 + .quad 0x1bb9fe452ea98463 + + // 2^124 * 7 * G + + .quad 0xe9d1d950b3d54f9e + .quad 0x2d5f9cbee00d33c1 + .quad 0x51c2c656a04fc6ac + .quad 0x65c091ee3c1cbcc9 + .quad 0xcf5e6c95cc36747c + .quad 0x294201536b0bc30d + .quad 0x453ac67cee797af0 + .quad 0x5eae6ab32a8bb3c9 + .quad 0x7083661114f118ea + .quad 0x2b37b87b94349cad + .quad 0x7273f51cb4e99f40 + .quad 0x78a2a95823d75698 + + // 2^124 * 8 * G + + .quad 0xa2b072e95c8c2ace + .quad 0x69cffc96651e9c4b + .quad 0x44328ef842e7b42b + .quad 0x5dd996c122aadeb3 + .quad 0xb4f23c425ef83207 + .quad 0xabf894d3c9a934b5 + .quad 0xd0708c1339fd87f7 + .quad 0x1876789117166130 + .quad 0x925b5ef0670c507c + .quad 0x819bc842b93c33bf + .quad 0x10792e9a70dd003f + .quad 0x59ad4b7a6e28dc74 + + // 2^128 * 1 * G + + .quad 0x5f3a7562eb3dbe47 + .quad 0xf7ea38548ebda0b8 + .quad 0x00c3e53145747299 + .quad 0x1304e9e71627d551 + .quad 0x583b04bfacad8ea2 + .quad 0x29b743e8148be884 + .quad 0x2b1e583b0810c5db + .quad 0x2b5449e58eb3bbaa + .quad 0x789814d26adc9cfe + .quad 0x3c1bab3f8b48dd0b + .quad 0xda0fe1fff979c60a + .quad 0x4468de2d7c2dd693 + + // 2^128 * 2 * G + + .quad 0x51bb355e9419469e + .quad 0x33e6dc4c23ddc754 + .quad 0x93a5b6d6447f9962 + .quad 0x6cce7c6ffb44bd63 + .quad 0x4b9ad8c6f86307ce + .quad 0x21113531435d0c28 + .quad 0xd4a866c5657a772c + .quad 0x5da6427e63247352 + .quad 0x1a94c688deac22ca + .quad 0xb9066ef7bbae1ff8 + .quad 0x88ad8c388d59580f + .quad 0x58f29abfe79f2ca8 + + // 2^128 * 3 * G + + .quad 0xe90ecfab8de73e68 + .quad 0x54036f9f377e76a5 + .quad 0xf0495b0bbe015982 + .quad 0x577629c4a7f41e36 + .quad 0x4b5a64bf710ecdf6 + .quad 0xb14ce538462c293c + .quad 0x3643d056d50b3ab9 + .quad 0x6af93724185b4870 + .quad 0x3220024509c6a888 + .quad 0xd2e036134b558973 + .quad 0x83e236233c33289f + .quad 0x701f25bb0caec18f + + // 2^128 * 4 * G + + .quad 0xc3a8b0f8e4616ced + .quad 0xf700660e9e25a87d + .quad 0x61e3061ff4bca59c + .quad 0x2e0c92bfbdc40be9 + .quad 0x9d18f6d97cbec113 + .quad 0x844a06e674bfdbe4 + .quad 0x20f5b522ac4e60d6 + .quad 0x720a5bc050955e51 + .quad 0x0c3f09439b805a35 + .quad 0xe84e8b376242abfc + .quad 0x691417f35c229346 + .quad 0x0e9b9cbb144ef0ec + + // 2^128 * 5 * G + + .quad 0xfbbad48ffb5720ad + .quad 0xee81916bdbf90d0e + .quad 0xd4813152635543bf + .quad 0x221104eb3f337bd8 + .quad 0x8dee9bd55db1beee + .quad 0xc9c3ab370a723fb9 + .quad 0x44a8f1bf1c68d791 + .quad 0x366d44191cfd3cde + .quad 0x9e3c1743f2bc8c14 + .quad 0x2eda26fcb5856c3b + .quad 0xccb82f0e68a7fb97 + .quad 0x4167a4e6bc593244 + + // 2^128 * 6 * G + + .quad 0x643b9d2876f62700 + .quad 0x5d1d9d400e7668eb + .quad 0x1b4b430321fc0684 + .quad 0x7938bb7e2255246a + .quad 0xc2be2665f8ce8fee + .quad 0xe967ff14e880d62c + .quad 0xf12e6e7e2f364eee + .quad 0x34b33370cb7ed2f6 + .quad 0xcdc591ee8681d6cc + .quad 0xce02109ced85a753 + .quad 0xed7485c158808883 + .quad 0x1176fc6e2dfe65e4 + + // 2^128 * 7 * G + + .quad 0xb4af6cd05b9c619b + .quad 0x2ddfc9f4b2a58480 + .quad 0x3d4fa502ebe94dc4 + .quad 0x08fc3a4c677d5f34 + .quad 0xdb90e28949770eb8 + .quad 0x98fbcc2aacf440a3 + .quad 0x21354ffeded7879b + .quad 0x1f6a3e54f26906b6 + .quad 0x60a4c199d30734ea + .quad 0x40c085b631165cd6 + .quad 0xe2333e23f7598295 + .quad 0x4f2fad0116b900d1 + + // 2^128 * 8 * G + + .quad 0x44beb24194ae4e54 + .quad 0x5f541c511857ef6c + .quad 0xa61e6b2d368d0498 + .quad 0x445484a4972ef7ab + .quad 0x962cd91db73bb638 + .quad 0xe60577aafc129c08 + .quad 0x6f619b39f3b61689 + .quad 0x3451995f2944ee81 + .quad 0x9152fcd09fea7d7c + .quad 0x4a816c94b0935cf6 + .quad 0x258e9aaa47285c40 + .quad 0x10b89ca6042893b7 + + // 2^132 * 1 * G + + .quad 0x9b2a426e3b646025 + .quad 0x32127190385ce4cf + .quad 0xa25cffc2dd6dea45 + .quad 0x06409010bea8de75 + .quad 0xd67cded679d34aa0 + .quad 0xcc0b9ec0cc4db39f + .quad 0xa535a456e35d190f + .quad 0x2e05d9eaf61f6fef + .quad 0xc447901ad61beb59 + .quad 0x661f19bce5dc880a + .quad 0x24685482b7ca6827 + .quad 0x293c778cefe07f26 + + // 2^132 * 2 * G + + .quad 0x86809e7007069096 + .quad 0xaad75b15e4e50189 + .quad 0x07f35715a21a0147 + .quad 0x0487f3f112815d5e + .quad 0x16c795d6a11ff200 + .quad 0xcb70d0e2b15815c9 + .quad 0x89f293209b5395b5 + .quad 0x50b8c2d031e47b4f + .quad 0x48350c08068a4962 + .quad 0x6ffdd05351092c9a + .quad 0x17af4f4aaf6fc8dd + .quad 0x4b0553b53cdba58b + + // 2^132 * 3 * G + + .quad 0x9c65fcbe1b32ff79 + .quad 0xeb75ea9f03b50f9b + .quad 0xfced2a6c6c07e606 + .quad 0x35106cd551717908 + .quad 0xbf05211b27c152d4 + .quad 0x5ec26849bd1af639 + .quad 0x5e0b2caa8e6fab98 + .quad 0x054c8bdd50bd0840 + .quad 0x38a0b12f1dcf073d + .quad 0x4b60a8a3b7f6a276 + .quad 0xfed5ac25d3404f9a + .quad 0x72e82d5e5505c229 + + // 2^132 * 4 * G + + .quad 0x6b0b697ff0d844c8 + .quad 0xbb12f85cd979cb49 + .quad 0xd2a541c6c1da0f1f + .quad 0x7b7c242958ce7211 + .quad 0x00d9cdfd69771d02 + .quad 0x410276cd6cfbf17e + .quad 0x4c45306c1cb12ec7 + .quad 0x2857bf1627500861 + .quad 0x9f21903f0101689e + .quad 0xd779dfd3bf861005 + .quad 0xa122ee5f3deb0f1b + .quad 0x510df84b485a00d4 + + // 2^132 * 5 * G + + .quad 0xa54133bb9277a1fa + .quad 0x74ec3b6263991237 + .quad 0x1a3c54dc35d2f15a + .quad 0x2d347144e482ba3a + .quad 0x24b3c887c70ac15e + .quad 0xb0f3a557fb81b732 + .quad 0x9b2cde2fe578cc1b + .quad 0x4cf7ed0703b54f8e + .quad 0x6bd47c6598fbee0f + .quad 0x9e4733e2ab55be2d + .quad 0x1093f624127610c5 + .quad 0x4e05e26ad0a1eaa4 + + // 2^132 * 6 * G + + .quad 0xda9b6b624b531f20 + .quad 0x429a760e77509abb + .quad 0xdbe9f522e823cb80 + .quad 0x618f1856880c8f82 + .quad 0x1833c773e18fe6c0 + .quad 0xe3c4711ad3c87265 + .quad 0x3bfd3c4f0116b283 + .quad 0x1955875eb4cd4db8 + .quad 0x6da6de8f0e399799 + .quad 0x7ad61aa440fda178 + .quad 0xb32cd8105e3563dd + .quad 0x15f6beae2ae340ae + + // 2^132 * 7 * G + + .quad 0x862bcb0c31ec3a62 + .quad 0x810e2b451138f3c2 + .quad 0x788ec4b839dac2a4 + .quad 0x28f76867ae2a9281 + .quad 0xba9a0f7b9245e215 + .quad 0xf368612dd98c0dbb + .quad 0x2e84e4cbf220b020 + .quad 0x6ba92fe962d90eda + .quad 0x3e4df9655884e2aa + .quad 0xbd62fbdbdbd465a5 + .quad 0xd7596caa0de9e524 + .quad 0x6e8042ccb2b1b3d7 + + // 2^132 * 8 * G + + .quad 0xf10d3c29ce28ca6e + .quad 0xbad34540fcb6093d + .quad 0xe7426ed7a2ea2d3f + .quad 0x08af9d4e4ff298b9 + .quad 0x1530653616521f7e + .quad 0x660d06b896203dba + .quad 0x2d3989bc545f0879 + .quad 0x4b5303af78ebd7b0 + .quad 0x72f8a6c3bebcbde8 + .quad 0x4f0fca4adc3a8e89 + .quad 0x6fa9d4e8c7bfdf7a + .quad 0x0dcf2d679b624eb7 + + // 2^136 * 1 * G + + .quad 0x3d5947499718289c + .quad 0x12ebf8c524533f26 + .quad 0x0262bfcb14c3ef15 + .quad 0x20b878d577b7518e + .quad 0x753941be5a45f06e + .quad 0xd07caeed6d9c5f65 + .quad 0x11776b9c72ff51b6 + .quad 0x17d2d1d9ef0d4da9 + .quad 0x27f2af18073f3e6a + .quad 0xfd3fe519d7521069 + .quad 0x22e3b72c3ca60022 + .quad 0x72214f63cc65c6a7 + + // 2^136 * 2 * G + + .quad 0xb4e37f405307a693 + .quad 0xaba714d72f336795 + .quad 0xd6fbd0a773761099 + .quad 0x5fdf48c58171cbc9 + .quad 0x1d9db7b9f43b29c9 + .quad 0xd605824a4f518f75 + .quad 0xf2c072bd312f9dc4 + .quad 0x1f24ac855a1545b0 + .quad 0x24d608328e9505aa + .quad 0x4748c1d10c1420ee + .quad 0xc7ffe45c06fb25a2 + .quad 0x00ba739e2ae395e6 + + // 2^136 * 3 * G + + .quad 0x592e98de5c8790d6 + .quad 0xe5bfb7d345c2a2df + .quad 0x115a3b60f9b49922 + .quad 0x03283a3e67ad78f3 + .quad 0xae4426f5ea88bb26 + .quad 0x360679d984973bfb + .quad 0x5c9f030c26694e50 + .quad 0x72297de7d518d226 + .quad 0x48241dc7be0cb939 + .quad 0x32f19b4d8b633080 + .quad 0xd3dfc90d02289308 + .quad 0x05e1296846271945 + + // 2^136 * 4 * G + + .quad 0xba82eeb32d9c495a + .quad 0xceefc8fcf12bb97c + .quad 0xb02dabae93b5d1e0 + .quad 0x39c00c9c13698d9b + .quad 0xadbfbbc8242c4550 + .quad 0xbcc80cecd03081d9 + .quad 0x843566a6f5c8df92 + .quad 0x78cf25d38258ce4c + .quad 0x15ae6b8e31489d68 + .quad 0xaa851cab9c2bf087 + .quad 0xc9a75a97f04efa05 + .quad 0x006b52076b3ff832 + + // 2^136 * 5 * G + + .quad 0x29e0cfe19d95781c + .quad 0xb681df18966310e2 + .quad 0x57df39d370516b39 + .quad 0x4d57e3443bc76122 + .quad 0xf5cb7e16b9ce082d + .quad 0x3407f14c417abc29 + .quad 0xd4b36bce2bf4a7ab + .quad 0x7de2e9561a9f75ce + .quad 0xde70d4f4b6a55ecb + .quad 0x4801527f5d85db99 + .quad 0xdbc9c440d3ee9a81 + .quad 0x6b2a90af1a6029ed + + // 2^136 * 6 * G + + .quad 0x6923f4fc9ae61e97 + .quad 0x5735281de03f5fd1 + .quad 0xa764ae43e6edd12d + .quad 0x5fd8f4e9d12d3e4a + .quad 0x77ebf3245bb2d80a + .quad 0xd8301b472fb9079b + .quad 0xc647e6f24cee7333 + .quad 0x465812c8276c2109 + .quad 0x4d43beb22a1062d9 + .quad 0x7065fb753831dc16 + .quad 0x180d4a7bde2968d7 + .quad 0x05b32c2b1cb16790 + + // 2^136 * 7 * G + + .quad 0xc8c05eccd24da8fd + .quad 0xa1cf1aac05dfef83 + .quad 0xdbbeeff27df9cd61 + .quad 0x3b5556a37b471e99 + .quad 0xf7fca42c7ad58195 + .quad 0x3214286e4333f3cc + .quad 0xb6c29d0d340b979d + .quad 0x31771a48567307e1 + .quad 0x32b0c524e14dd482 + .quad 0xedb351541a2ba4b6 + .quad 0xa3d16048282b5af3 + .quad 0x4fc079d27a7336eb + + // 2^136 * 8 * G + + .quad 0x51c938b089bf2f7f + .quad 0x2497bd6502dfe9a7 + .quad 0xffffc09c7880e453 + .quad 0x124567cecaf98e92 + .quad 0xdc348b440c86c50d + .quad 0x1337cbc9cc94e651 + .quad 0x6422f74d643e3cb9 + .quad 0x241170c2bae3cd08 + .quad 0x3ff9ab860ac473b4 + .quad 0xf0911dee0113e435 + .quad 0x4ae75060ebc6c4af + .quad 0x3f8612966c87000d + + // 2^140 * 1 * G + + .quad 0x0c9c5303f7957be4 + .quad 0xa3c31a20e085c145 + .quad 0xb0721d71d0850050 + .quad 0x0aba390eab0bf2da + .quad 0x529fdffe638c7bf3 + .quad 0xdf2b9e60388b4995 + .quad 0xe027b34f1bad0249 + .quad 0x7bc92fc9b9fa74ed + .quad 0x9f97ef2e801ad9f9 + .quad 0x83697d5479afda3a + .quad 0xe906b3ffbd596b50 + .quad 0x02672b37dd3fb8e0 + + // 2^140 * 2 * G + + .quad 0x48b2ca8b260885e4 + .quad 0xa4286bec82b34c1c + .quad 0x937e1a2617f58f74 + .quad 0x741d1fcbab2ca2a5 + .quad 0xee9ba729398ca7f5 + .quad 0xeb9ca6257a4849db + .quad 0x29eb29ce7ec544e1 + .quad 0x232ca21ef736e2c8 + .quad 0xbf61423d253fcb17 + .quad 0x08803ceafa39eb14 + .quad 0xf18602df9851c7af + .quad 0x0400f3a049e3414b + + // 2^140 * 3 * G + + .quad 0xabce0476ba61c55b + .quad 0x36a3d6d7c4d39716 + .quad 0x6eb259d5e8d82d09 + .quad 0x0c9176e984d756fb + .quad 0x2efba412a06e7b06 + .quad 0x146785452c8d2560 + .quad 0xdf9713ebd67a91c7 + .quad 0x32830ac7157eadf3 + .quad 0x0e782a7ab73769e8 + .quad 0x04a05d7875b18e2c + .quad 0x29525226ebcceae1 + .quad 0x0d794f8383eba820 + + // 2^140 * 4 * G + + .quad 0xff35f5cb9e1516f4 + .quad 0xee805bcf648aae45 + .quad 0xf0d73c2bb93a9ef3 + .quad 0x097b0bf22092a6c2 + .quad 0x7be44ce7a7a2e1ac + .quad 0x411fd93efad1b8b7 + .quad 0x1734a1d70d5f7c9b + .quad 0x0d6592233127db16 + .quad 0xc48bab1521a9d733 + .quad 0xa6c2eaead61abb25 + .quad 0x625c6c1cc6cb4305 + .quad 0x7fc90fea93eb3a67 + + // 2^140 * 5 * G + + .quad 0x0408f1fe1f5c5926 + .quad 0x1a8f2f5e3b258bf4 + .quad 0x40a951a2fdc71669 + .quad 0x6598ee93c98b577e + .quad 0xc527deb59c7cb23d + .quad 0x955391695328404e + .quad 0xd64392817ccf2c7a + .quad 0x6ce97dabf7d8fa11 + .quad 0x25b5a8e50ef7c48f + .quad 0xeb6034116f2ce532 + .quad 0xc5e75173e53de537 + .quad 0x73119fa08c12bb03 + + // 2^140 * 6 * G + + .quad 0xed30129453f1a4cb + .quad 0xbce621c9c8f53787 + .quad 0xfacb2b1338bee7b9 + .quad 0x3025798a9ea8428c + .quad 0x7845b94d21f4774d + .quad 0xbf62f16c7897b727 + .quad 0x671857c03c56522b + .quad 0x3cd6a85295621212 + .quad 0x3fecde923aeca999 + .quad 0xbdaa5b0062e8c12f + .quad 0x67b99dfc96988ade + .quad 0x3f52c02852661036 + + // 2^140 * 7 * G + + .quad 0xffeaa48e2a1351c6 + .quad 0x28624754fa7f53d7 + .quad 0x0b5ba9e57582ddf1 + .quad 0x60c0104ba696ac59 + .quad 0x9258bf99eec416c6 + .quad 0xac8a5017a9d2f671 + .quad 0x629549ab16dea4ab + .quad 0x05d0e85c99091569 + .quad 0x051de020de9cbe97 + .quad 0xfa07fc56b50bcf74 + .quad 0x378cec9f0f11df65 + .quad 0x36853c69ab96de4d + + // 2^140 * 8 * G + + .quad 0x36d9b8de78f39b2d + .quad 0x7f42ed71a847b9ec + .quad 0x241cd1d679bd3fde + .quad 0x6a704fec92fbce6b + .quad 0x4433c0b0fac5e7be + .quad 0x724bae854c08dcbe + .quad 0xf1f24cc446978f9b + .quad 0x4a0aff6d62825fc8 + .quad 0xe917fb9e61095301 + .quad 0xc102df9402a092f8 + .quad 0xbf09e2f5fa66190b + .quad 0x681109bee0dcfe37 + + // 2^144 * 1 * G + + .quad 0x559a0cc9782a0dde + .quad 0x551dcdb2ea718385 + .quad 0x7f62865b31ef238c + .quad 0x504aa7767973613d + .quad 0x9c18fcfa36048d13 + .quad 0x29159db373899ddd + .quad 0xdc9f350b9f92d0aa + .quad 0x26f57eee878a19d4 + .quad 0x0cab2cd55687efb1 + .quad 0x5180d162247af17b + .quad 0x85c15a344f5a2467 + .quad 0x4041943d9dba3069 + + // 2^144 * 2 * G + + .quad 0xc3c0eeba43ebcc96 + .quad 0x8d749c9c26ea9caf + .quad 0xd9fa95ee1c77ccc6 + .quad 0x1420a1d97684340f + .quad 0x4b217743a26caadd + .quad 0x47a6b424648ab7ce + .quad 0xcb1d4f7a03fbc9e3 + .quad 0x12d931429800d019 + .quad 0x00c67799d337594f + .quad 0x5e3c5140b23aa47b + .quad 0x44182854e35ff395 + .quad 0x1b4f92314359a012 + + // 2^144 * 3 * G + + .quad 0x3e5c109d89150951 + .quad 0x39cefa912de9696a + .quad 0x20eae43f975f3020 + .quad 0x239b572a7f132dae + .quad 0x33cf3030a49866b1 + .quad 0x251f73d2215f4859 + .quad 0xab82aa4051def4f6 + .quad 0x5ff191d56f9a23f6 + .quad 0x819ed433ac2d9068 + .quad 0x2883ab795fc98523 + .quad 0xef4572805593eb3d + .quad 0x020c526a758f36cb + + // 2^144 * 4 * G + + .quad 0x779834f89ed8dbbc + .quad 0xc8f2aaf9dc7ca46c + .quad 0xa9524cdca3e1b074 + .quad 0x02aacc4615313877 + .quad 0xe931ef59f042cc89 + .quad 0x2c589c9d8e124bb6 + .quad 0xadc8e18aaec75997 + .quad 0x452cfe0a5602c50c + .quad 0x86a0f7a0647877df + .quad 0xbbc464270e607c9f + .quad 0xab17ea25f1fb11c9 + .quad 0x4cfb7d7b304b877b + + // 2^144 * 5 * G + + .quad 0x72b43d6cb89b75fe + .quad 0x54c694d99c6adc80 + .quad 0xb8c3aa373ee34c9f + .quad 0x14b4622b39075364 + .quad 0xe28699c29789ef12 + .quad 0x2b6ecd71df57190d + .quad 0xc343c857ecc970d0 + .quad 0x5b1d4cbc434d3ac5 + .quad 0xb6fb2615cc0a9f26 + .quad 0x3a4f0e2bb88dcce5 + .quad 0x1301498b3369a705 + .quad 0x2f98f71258592dd1 + + // 2^144 * 6 * G + + .quad 0x0c94a74cb50f9e56 + .quad 0x5b1ff4a98e8e1320 + .quad 0x9a2acc2182300f67 + .quad 0x3a6ae249d806aaf9 + .quad 0x2e12ae444f54a701 + .quad 0xfcfe3ef0a9cbd7de + .quad 0xcebf890d75835de0 + .quad 0x1d8062e9e7614554 + .quad 0x657ada85a9907c5a + .quad 0x1a0ea8b591b90f62 + .quad 0x8d0e1dfbdf34b4e9 + .quad 0x298b8ce8aef25ff3 + + // 2^144 * 7 * G + + .quad 0x2a927953eff70cb2 + .quad 0x4b89c92a79157076 + .quad 0x9418457a30a7cf6a + .quad 0x34b8a8404d5ce485 + .quad 0x837a72ea0a2165de + .quad 0x3fab07b40bcf79f6 + .quad 0x521636c77738ae70 + .quad 0x6ba6271803a7d7dc + .quad 0xc26eecb583693335 + .quad 0xd5a813df63b5fefd + .quad 0xa293aa9aa4b22573 + .quad 0x71d62bdd465e1c6a + + // 2^144 * 8 * G + + .quad 0x6533cc28d378df80 + .quad 0xf6db43790a0fa4b4 + .quad 0xe3645ff9f701da5a + .quad 0x74d5f317f3172ba4 + .quad 0xcd2db5dab1f75ef5 + .quad 0xd77f95cf16b065f5 + .quad 0x14571fea3f49f085 + .quad 0x1c333621262b2b3d + .quad 0xa86fe55467d9ca81 + .quad 0x398b7c752b298c37 + .quad 0xda6d0892e3ac623b + .quad 0x4aebcc4547e9d98c + + // 2^148 * 1 * G + + .quad 0x53175a7205d21a77 + .quad 0xb0c04422d3b934d4 + .quad 0xadd9f24bdd5deadc + .quad 0x074f46e69f10ff8c + .quad 0x0de9b204a059a445 + .quad 0xe15cb4aa4b17ad0f + .quad 0xe1bbec521f79c557 + .quad 0x2633f1b9d071081b + .quad 0xc1fb4177018b9910 + .quad 0xa6ea20dc6c0fe140 + .quad 0xd661f3e74354c6ff + .quad 0x5ecb72e6f1a3407a + + // 2^148 * 2 * G + + .quad 0xa515a31b2259fb4e + .quad 0x0960f3972bcac52f + .quad 0xedb52fec8d3454cb + .quad 0x382e2720c476c019 + .quad 0xfeeae106e8e86997 + .quad 0x9863337f98d09383 + .quad 0x9470480eaa06ebef + .quad 0x038b6898d4c5c2d0 + .quad 0xf391c51d8ace50a6 + .quad 0x3142d0b9ae2d2948 + .quad 0xdb4d5a1a7f24ca80 + .quad 0x21aeba8b59250ea8 + + // 2^148 * 3 * G + + .quad 0x24f13b34cf405530 + .quad 0x3c44ea4a43088af7 + .quad 0x5dd5c5170006a482 + .quad 0x118eb8f8890b086d + .quad 0x53853600f0087f23 + .quad 0x4c461879da7d5784 + .quad 0x6af303deb41f6860 + .quad 0x0a3c16c5c27c18ed + .quad 0x17e49c17cc947f3d + .quad 0xccc6eda6aac1d27b + .quad 0xdf6092ceb0f08e56 + .quad 0x4909b3e22c67c36b + + // 2^148 * 4 * G + + .quad 0x9c9c85ea63fe2e89 + .quad 0xbe1baf910e9412ec + .quad 0x8f7baa8a86fbfe7b + .quad 0x0fb17f9fef968b6c + .quad 0x59a16676706ff64e + .quad 0x10b953dd0d86a53d + .quad 0x5848e1e6ce5c0b96 + .quad 0x2d8b78e712780c68 + .quad 0x79d5c62eafc3902b + .quad 0x773a215289e80728 + .quad 0xc38ae640e10120b9 + .quad 0x09ae23717b2b1a6d + + // 2^148 * 5 * G + + .quad 0xbb6a192a4e4d083c + .quad 0x34ace0630029e192 + .quad 0x98245a59aafabaeb + .quad 0x6d9c8a9ada97faac + .quad 0x10ab8fa1ad32b1d0 + .quad 0xe9aced1be2778b24 + .quad 0xa8856bc0373de90f + .quad 0x66f35ddddda53996 + .quad 0xd27d9afb24997323 + .quad 0x1bb7e07ef6f01d2e + .quad 0x2ba7472df52ecc7f + .quad 0x03019b4f646f9dc8 + + // 2^148 * 6 * G + + .quad 0x04a186b5565345cd + .quad 0xeee76610bcc4116a + .quad 0x689c73b478fb2a45 + .quad 0x387dcbff65697512 + .quad 0xaf09b214e6b3dc6b + .quad 0x3f7573b5ad7d2f65 + .quad 0xd019d988100a23b0 + .quad 0x392b63a58b5c35f7 + .quad 0x4093addc9c07c205 + .quad 0xc565be15f532c37e + .quad 0x63dbecfd1583402a + .quad 0x61722b4aef2e032e + + // 2^148 * 7 * G + + .quad 0x0012aafeecbd47af + .quad 0x55a266fb1cd46309 + .quad 0xf203eb680967c72c + .quad 0x39633944ca3c1429 + .quad 0xd6b07a5581cb0e3c + .quad 0x290ff006d9444969 + .quad 0x08680b6a16dcda1f + .quad 0x5568d2b75a06de59 + .quad 0x8d0cb88c1b37cfe1 + .quad 0x05b6a5a3053818f3 + .quad 0xf2e9bc04b787d959 + .quad 0x6beba1249add7f64 + + // 2^148 * 8 * G + + .quad 0x1d06005ca5b1b143 + .quad 0x6d4c6bb87fd1cda2 + .quad 0x6ef5967653fcffe7 + .quad 0x097c29e8c1ce1ea5 + .quad 0x5c3cecb943f5a53b + .quad 0x9cc9a61d06c08df2 + .quad 0xcfba639a85895447 + .quad 0x5a845ae80df09fd5 + .quad 0x4ce97dbe5deb94ca + .quad 0x38d0a4388c709c48 + .quad 0xc43eced4a169d097 + .quad 0x0a1249fff7e587c3 + + // 2^152 * 1 * G + + .quad 0x12f0071b276d01c9 + .quad 0xe7b8bac586c48c70 + .quad 0x5308129b71d6fba9 + .quad 0x5d88fbf95a3db792 + .quad 0x0b408d9e7354b610 + .quad 0x806b32535ba85b6e + .quad 0xdbe63a034a58a207 + .quad 0x173bd9ddc9a1df2c + .quad 0x2b500f1efe5872df + .quad 0x58d6582ed43918c1 + .quad 0xe6ed278ec9673ae0 + .quad 0x06e1cd13b19ea319 + + // 2^152 * 2 * G + + .quad 0x40d0ad516f166f23 + .quad 0x118e32931fab6abe + .quad 0x3fe35e14a04d088e + .quad 0x3080603526e16266 + .quad 0x472baf629e5b0353 + .quad 0x3baa0b90278d0447 + .quad 0x0c785f469643bf27 + .quad 0x7f3a6a1a8d837b13 + .quad 0xf7e644395d3d800b + .quad 0x95a8d555c901edf6 + .quad 0x68cd7830592c6339 + .quad 0x30d0fded2e51307e + + // 2^152 * 3 * G + + .quad 0xe0594d1af21233b3 + .quad 0x1bdbe78ef0cc4d9c + .quad 0x6965187f8f499a77 + .quad 0x0a9214202c099868 + .quad 0x9cb4971e68b84750 + .quad 0xa09572296664bbcf + .quad 0x5c8de72672fa412b + .quad 0x4615084351c589d9 + .quad 0xbc9019c0aeb9a02e + .quad 0x55c7110d16034cae + .quad 0x0e6df501659932ec + .quad 0x3bca0d2895ca5dfe + + // 2^152 * 4 * G + + .quad 0x40f031bc3c5d62a4 + .quad 0x19fc8b3ecff07a60 + .quad 0x98183da2130fb545 + .quad 0x5631deddae8f13cd + .quad 0x9c688eb69ecc01bf + .quad 0xf0bc83ada644896f + .quad 0xca2d955f5f7a9fe2 + .quad 0x4ea8b4038df28241 + .quad 0x2aed460af1cad202 + .quad 0x46305305a48cee83 + .quad 0x9121774549f11a5f + .quad 0x24ce0930542ca463 + + // 2^152 * 5 * G + + .quad 0x1fe890f5fd06c106 + .quad 0xb5c468355d8810f2 + .quad 0x827808fe6e8caf3e + .quad 0x41d4e3c28a06d74b + .quad 0x3fcfa155fdf30b85 + .quad 0xd2f7168e36372ea4 + .quad 0xb2e064de6492f844 + .quad 0x549928a7324f4280 + .quad 0xf26e32a763ee1a2e + .quad 0xae91e4b7d25ffdea + .quad 0xbc3bd33bd17f4d69 + .quad 0x491b66dec0dcff6a + + // 2^152 * 6 * G + + .quad 0x98f5b13dc7ea32a7 + .quad 0xe3d5f8cc7e16db98 + .quad 0xac0abf52cbf8d947 + .quad 0x08f338d0c85ee4ac + .quad 0x75f04a8ed0da64a1 + .quad 0xed222caf67e2284b + .quad 0x8234a3791f7b7ba4 + .quad 0x4cf6b8b0b7018b67 + .quad 0xc383a821991a73bd + .quad 0xab27bc01df320c7a + .quad 0xc13d331b84777063 + .quad 0x530d4a82eb078a99 + + // 2^152 * 7 * G + + .quad 0x004c3630e1f94825 + .quad 0x7e2d78268cab535a + .quad 0xc7482323cc84ff8b + .quad 0x65ea753f101770b9 + .quad 0x6d6973456c9abf9e + .quad 0x257fb2fc4900a880 + .quad 0x2bacf412c8cfb850 + .quad 0x0db3e7e00cbfbd5b + .quad 0x3d66fc3ee2096363 + .quad 0x81d62c7f61b5cb6b + .quad 0x0fbe044213443b1a + .quad 0x02a4ec1921e1a1db + + // 2^152 * 8 * G + + .quad 0x5ce6259a3b24b8a2 + .quad 0xb8577acc45afa0b8 + .quad 0xcccbe6e88ba07037 + .quad 0x3d143c51127809bf + .quad 0xf5c86162f1cf795f + .quad 0x118c861926ee57f2 + .quad 0x172124851c063578 + .quad 0x36d12b5dec067fcf + .quad 0x126d279179154557 + .quad 0xd5e48f5cfc783a0a + .quad 0x36bdb6e8df179bac + .quad 0x2ef517885ba82859 + + // 2^156 * 1 * G + + .quad 0x88bd438cd11e0d4a + .quad 0x30cb610d43ccf308 + .quad 0xe09a0e3791937bcc + .quad 0x4559135b25b1720c + .quad 0x1ea436837c6da1e9 + .quad 0xf9c189af1fb9bdbe + .quad 0x303001fcce5dd155 + .quad 0x28a7c99ebc57be52 + .quad 0xb8fd9399e8d19e9d + .quad 0x908191cb962423ff + .quad 0xb2b948d747c742a3 + .quad 0x37f33226d7fb44c4 + + // 2^156 * 2 * G + + .quad 0x0dae8767b55f6e08 + .quad 0x4a43b3b35b203a02 + .quad 0xe3725a6e80af8c79 + .quad 0x0f7a7fd1705fa7a3 + .quad 0x33912553c821b11d + .quad 0x66ed42c241e301df + .quad 0x066fcc11104222fd + .quad 0x307a3b41c192168f + .quad 0x8eeb5d076eb55ce0 + .quad 0x2fc536bfaa0d925a + .quad 0xbe81830fdcb6c6e8 + .quad 0x556c7045827baf52 + + // 2^156 * 3 * G + + .quad 0x8e2b517302e9d8b7 + .quad 0xe3e52269248714e8 + .quad 0xbd4fbd774ca960b5 + .quad 0x6f4b4199c5ecada9 + .quad 0xb94b90022bf44406 + .quad 0xabd4237eff90b534 + .quad 0x7600a960faf86d3a + .quad 0x2f45abdac2322ee3 + .quad 0x61af4912c8ef8a6a + .quad 0xe58fa4fe43fb6e5e + .quad 0xb5afcc5d6fd427cf + .quad 0x6a5393281e1e11eb + + // 2^156 * 4 * G + + .quad 0xf3da5139a5d1ee89 + .quad 0x8145457cff936988 + .quad 0x3f622fed00e188c4 + .quad 0x0f513815db8b5a3d + .quad 0x0fff04fe149443cf + .quad 0x53cac6d9865cddd7 + .quad 0x31385b03531ed1b7 + .quad 0x5846a27cacd1039d + .quad 0x4ff5cdac1eb08717 + .quad 0x67e8b29590f2e9bc + .quad 0x44093b5e237afa99 + .quad 0x0d414bed8708b8b2 + + // 2^156 * 5 * G + + .quad 0xcfb68265fd0e75f6 + .quad 0xe45b3e28bb90e707 + .quad 0x7242a8de9ff92c7a + .quad 0x685b3201933202dd + .quad 0x81886a92294ac9e8 + .quad 0x23162b45d55547be + .quad 0x94cfbc4403715983 + .quad 0x50eb8fdb134bc401 + .quad 0xc0b73ec6d6b330cd + .quad 0x84e44807132faff1 + .quad 0x732b7352c4a5dee1 + .quad 0x5d7c7cf1aa7cd2d2 + + // 2^156 * 6 * G + + .quad 0xaf3b46bf7a4aafa2 + .quad 0xb78705ec4d40d411 + .quad 0x114f0c6aca7c15e3 + .quad 0x3f364faaa9489d4d + .quad 0x33d1013e9b73a562 + .quad 0x925cef5748ec26e1 + .quad 0xa7fce614dd468058 + .quad 0x78b0fad41e9aa438 + .quad 0xbf56a431ed05b488 + .quad 0xa533e66c9c495c7e + .quad 0xe8652baf87f3651a + .quad 0x0241800059d66c33 + + // 2^156 * 7 * G + + .quad 0xceb077fea37a5be4 + .quad 0xdb642f02e5a5eeb7 + .quad 0xc2e6d0c5471270b8 + .quad 0x4771b65538e4529c + .quad 0x28350c7dcf38ea01 + .quad 0x7c6cdbc0b2917ab6 + .quad 0xace7cfbe857082f7 + .quad 0x4d2845aba2d9a1e0 + .quad 0xbb537fe0447070de + .quad 0xcba744436dd557df + .quad 0xd3b5a3473600dbcb + .quad 0x4aeabbe6f9ffd7f8 + + // 2^156 * 8 * G + + .quad 0x4630119e40d8f78c + .quad 0xa01a9bc53c710e11 + .quad 0x486d2b258910dd79 + .quad 0x1e6c47b3db0324e5 + .quad 0x6a2134bcc4a9c8f2 + .quad 0xfbf8fd1c8ace2e37 + .quad 0x000ae3049911a0ba + .quad 0x046e3a616bc89b9e + .quad 0x14e65442f03906be + .quad 0x4a019d54e362be2a + .quad 0x68ccdfec8dc230c7 + .quad 0x7cfb7e3faf6b861c + + // 2^160 * 1 * G + + .quad 0x4637974e8c58aedc + .quad 0xb9ef22fbabf041a4 + .quad 0xe185d956e980718a + .quad 0x2f1b78fab143a8a6 + .quad 0x96eebffb305b2f51 + .quad 0xd3f938ad889596b8 + .quad 0xf0f52dc746d5dd25 + .quad 0x57968290bb3a0095 + .quad 0xf71ab8430a20e101 + .quad 0xf393658d24f0ec47 + .quad 0xcf7509a86ee2eed1 + .quad 0x7dc43e35dc2aa3e1 + + // 2^160 * 2 * G + + .quad 0x85966665887dd9c3 + .quad 0xc90f9b314bb05355 + .quad 0xc6e08df8ef2079b1 + .quad 0x7ef72016758cc12f + .quad 0x5a782a5c273e9718 + .quad 0x3576c6995e4efd94 + .quad 0x0f2ed8051f237d3e + .quad 0x044fb81d82d50a99 + .quad 0xc1df18c5a907e3d9 + .quad 0x57b3371dce4c6359 + .quad 0xca704534b201bb49 + .quad 0x7f79823f9c30dd2e + + // 2^160 * 3 * G + + .quad 0x8334d239a3b513e8 + .quad 0xc13670d4b91fa8d8 + .quad 0x12b54136f590bd33 + .quad 0x0a4e0373d784d9b4 + .quad 0x6a9c1ff068f587ba + .quad 0x0827894e0050c8de + .quad 0x3cbf99557ded5be7 + .quad 0x64a9b0431c06d6f0 + .quad 0x2eb3d6a15b7d2919 + .quad 0xb0b4f6a0d53a8235 + .quad 0x7156ce4389a45d47 + .quad 0x071a7d0ace18346c + + // 2^160 * 4 * G + + .quad 0xd3072daac887ba0b + .quad 0x01262905bfa562ee + .quad 0xcf543002c0ef768b + .quad 0x2c3bcc7146ea7e9c + .quad 0xcc0c355220e14431 + .quad 0x0d65950709b15141 + .quad 0x9af5621b209d5f36 + .quad 0x7c69bcf7617755d3 + .quad 0x07f0d7eb04e8295f + .quad 0x10db18252f50f37d + .quad 0xe951a9a3171798d7 + .quad 0x6f5a9a7322aca51d + + // 2^160 * 5 * G + + .quad 0x8ba1000c2f41c6c5 + .quad 0xc49f79c10cfefb9b + .quad 0x4efa47703cc51c9f + .quad 0x494e21a2e147afca + .quad 0xe729d4eba3d944be + .quad 0x8d9e09408078af9e + .quad 0x4525567a47869c03 + .quad 0x02ab9680ee8d3b24 + .quad 0xefa48a85dde50d9a + .quad 0x219a224e0fb9a249 + .quad 0xfa091f1dd91ef6d9 + .quad 0x6b5d76cbea46bb34 + + // 2^160 * 6 * G + + .quad 0x8857556cec0cd994 + .quad 0x6472dc6f5cd01dba + .quad 0xaf0169148f42b477 + .quad 0x0ae333f685277354 + .quad 0xe0f941171e782522 + .quad 0xf1e6ae74036936d3 + .quad 0x408b3ea2d0fcc746 + .quad 0x16fb869c03dd313e + .quad 0x288e199733b60962 + .quad 0x24fc72b4d8abe133 + .quad 0x4811f7ed0991d03e + .quad 0x3f81e38b8f70d075 + + // 2^160 * 7 * G + + .quad 0x7f910fcc7ed9affe + .quad 0x545cb8a12465874b + .quad 0xa8397ed24b0c4704 + .quad 0x50510fc104f50993 + .quad 0x0adb7f355f17c824 + .quad 0x74b923c3d74299a4 + .quad 0xd57c3e8bcbf8eaf7 + .quad 0x0ad3e2d34cdedc3d + .quad 0x6f0c0fc5336e249d + .quad 0x745ede19c331cfd9 + .quad 0xf2d6fd0009eefe1c + .quad 0x127c158bf0fa1ebe + + // 2^160 * 8 * G + + .quad 0xf6197c422e9879a2 + .quad 0xa44addd452ca3647 + .quad 0x9b413fc14b4eaccb + .quad 0x354ef87d07ef4f68 + .quad 0xdea28fc4ae51b974 + .quad 0x1d9973d3744dfe96 + .quad 0x6240680b873848a8 + .quad 0x4ed82479d167df95 + .quad 0xfee3b52260c5d975 + .quad 0x50352efceb41b0b8 + .quad 0x8808ac30a9f6653c + .quad 0x302d92d20539236d + + // 2^164 * 1 * G + + .quad 0x4c59023fcb3efb7c + .quad 0x6c2fcb99c63c2a94 + .quad 0xba4190e2c3c7e084 + .quad 0x0e545daea51874d9 + .quad 0x957b8b8b0df53c30 + .quad 0x2a1c770a8e60f098 + .quad 0xbbc7a670345796de + .quad 0x22a48f9a90c99bc9 + .quad 0x6b7dc0dc8d3fac58 + .quad 0x5497cd6ce6e42bfd + .quad 0x542f7d1bf400d305 + .quad 0x4159f47f048d9136 + + // 2^164 * 2 * G + + .quad 0x20ad660839e31e32 + .quad 0xf81e1bd58405be50 + .quad 0xf8064056f4dabc69 + .quad 0x14d23dd4ce71b975 + .quad 0x748515a8bbd24839 + .quad 0x77128347afb02b55 + .quad 0x50ba2ac649a2a17f + .quad 0x060525513ad730f1 + .quad 0xf2398e098aa27f82 + .quad 0x6d7982bb89a1b024 + .quad 0xfa694084214dd24c + .quad 0x71ab966fa32301c3 + + // 2^164 * 3 * G + + .quad 0x2dcbd8e34ded02fc + .quad 0x1151f3ec596f22aa + .quad 0xbca255434e0328da + .quad 0x35768fbe92411b22 + .quad 0xb1088a0702809955 + .quad 0x43b273ea0b43c391 + .quad 0xca9b67aefe0686ed + .quad 0x605eecbf8335f4ed + .quad 0x83200a656c340431 + .quad 0x9fcd71678ee59c2f + .quad 0x75d4613f71300f8a + .quad 0x7a912faf60f542f9 + + // 2^164 * 4 * G + + .quad 0xb204585e5edc1a43 + .quad 0x9f0e16ee5897c73c + .quad 0x5b82c0ae4e70483c + .quad 0x624a170e2bddf9be + .quad 0x253f4f8dfa2d5597 + .quad 0x25e49c405477130c + .quad 0x00c052e5996b1102 + .quad 0x33cb966e33bb6c4a + .quad 0x597028047f116909 + .quad 0x828ac41c1e564467 + .quad 0x70417dbde6217387 + .quad 0x721627aefbac4384 + + // 2^164 * 5 * G + + .quad 0x97d03bc38736add5 + .quad 0x2f1422afc532b130 + .quad 0x3aa68a057101bbc4 + .quad 0x4c946cf7e74f9fa7 + .quad 0xfd3097bc410b2f22 + .quad 0xf1a05da7b5cfa844 + .quad 0x61289a1def57ca74 + .quad 0x245ea199bb821902 + .quad 0xaedca66978d477f8 + .quad 0x1898ba3c29117fe1 + .quad 0xcf73f983720cbd58 + .quad 0x67da12e6b8b56351 + + // 2^164 * 6 * G + + .quad 0x7067e187b4bd6e07 + .quad 0x6e8f0203c7d1fe74 + .quad 0x93c6aa2f38c85a30 + .quad 0x76297d1f3d75a78a + .quad 0x2b7ef3d38ec8308c + .quad 0x828fd7ec71eb94ab + .quad 0x807c3b36c5062abd + .quad 0x0cb64cb831a94141 + .quad 0x3030fc33534c6378 + .quad 0xb9635c5ce541e861 + .quad 0x15d9a9bed9b2c728 + .quad 0x49233ea3f3775dcb + + // 2^164 * 7 * G + + .quad 0x629398fa8dbffc3a + .quad 0xe12fe52dd54db455 + .quad 0xf3be11dfdaf25295 + .quad 0x628b140dce5e7b51 + .quad 0x7b3985fe1c9f249b + .quad 0x4fd6b2d5a1233293 + .quad 0xceb345941adf4d62 + .quad 0x6987ff6f542de50c + .quad 0x47e241428f83753c + .quad 0x6317bebc866af997 + .quad 0xdabb5b433d1a9829 + .quad 0x074d8d245287fb2d + + // 2^164 * 8 * G + + .quad 0x8337d9cd440bfc31 + .quad 0x729d2ca1af318fd7 + .quad 0xa040a4a4772c2070 + .quad 0x46002ef03a7349be + .quad 0x481875c6c0e31488 + .quad 0x219429b2e22034b4 + .quad 0x7223c98a31283b65 + .quad 0x3420d60b342277f9 + .quad 0xfaa23adeaffe65f7 + .quad 0x78261ed45be0764c + .quad 0x441c0a1e2f164403 + .quad 0x5aea8e567a87d395 + + // 2^168 * 1 * G + + .quad 0x7813c1a2bca4283d + .quad 0xed62f091a1863dd9 + .quad 0xaec7bcb8c268fa86 + .quad 0x10e5d3b76f1cae4c + .quad 0x2dbc6fb6e4e0f177 + .quad 0x04e1bf29a4bd6a93 + .quad 0x5e1966d4787af6e8 + .quad 0x0edc5f5eb426d060 + .quad 0x5453bfd653da8e67 + .quad 0xe9dc1eec24a9f641 + .quad 0xbf87263b03578a23 + .quad 0x45b46c51361cba72 + + // 2^168 * 2 * G + + .quad 0xa9402abf314f7fa1 + .quad 0xe257f1dc8e8cf450 + .quad 0x1dbbd54b23a8be84 + .quad 0x2177bfa36dcb713b + .quad 0xce9d4ddd8a7fe3e4 + .quad 0xab13645676620e30 + .quad 0x4b594f7bb30e9958 + .quad 0x5c1c0aef321229df + .quad 0x37081bbcfa79db8f + .quad 0x6048811ec25f59b3 + .quad 0x087a76659c832487 + .quad 0x4ae619387d8ab5bb + + // 2^168 * 3 * G + + .quad 0x8ddbf6aa5344a32e + .quad 0x7d88eab4b41b4078 + .quad 0x5eb0eb974a130d60 + .quad 0x1a00d91b17bf3e03 + .quad 0x61117e44985bfb83 + .quad 0xfce0462a71963136 + .quad 0x83ac3448d425904b + .quad 0x75685abe5ba43d64 + .quad 0x6e960933eb61f2b2 + .quad 0x543d0fa8c9ff4952 + .quad 0xdf7275107af66569 + .quad 0x135529b623b0e6aa + + // 2^168 * 4 * G + + .quad 0x18f0dbd7add1d518 + .quad 0x979f7888cfc11f11 + .quad 0x8732e1f07114759b + .quad 0x79b5b81a65ca3a01 + .quad 0xf5c716bce22e83fe + .quad 0xb42beb19e80985c1 + .quad 0xec9da63714254aae + .quad 0x5972ea051590a613 + .quad 0x0fd4ac20dc8f7811 + .quad 0x9a9ad294ac4d4fa8 + .quad 0xc01b2d64b3360434 + .quad 0x4f7e9c95905f3bdb + + // 2^168 * 5 * G + + .quad 0x62674bbc5781302e + .quad 0xd8520f3989addc0f + .quad 0x8c2999ae53fbd9c6 + .quad 0x31993ad92e638e4c + .quad 0x71c8443d355299fe + .quad 0x8bcd3b1cdbebead7 + .quad 0x8092499ef1a49466 + .quad 0x1942eec4a144adc8 + .quad 0x7dac5319ae234992 + .quad 0x2c1b3d910cea3e92 + .quad 0x553ce494253c1122 + .quad 0x2a0a65314ef9ca75 + + // 2^168 * 6 * G + + .quad 0x2db7937ff7f927c2 + .quad 0xdb741f0617d0a635 + .quad 0x5982f3a21155af76 + .quad 0x4cf6e218647c2ded + .quad 0xcf361acd3c1c793a + .quad 0x2f9ebcac5a35bc3b + .quad 0x60e860e9a8cda6ab + .quad 0x055dc39b6dea1a13 + .quad 0xb119227cc28d5bb6 + .quad 0x07e24ebc774dffab + .quad 0xa83c78cee4a32c89 + .quad 0x121a307710aa24b6 + + // 2^168 * 7 * G + + .quad 0xe4db5d5e9f034a97 + .quad 0xe153fc093034bc2d + .quad 0x460546919551d3b1 + .quad 0x333fc76c7a40e52d + .quad 0xd659713ec77483c9 + .quad 0x88bfe077b82b96af + .quad 0x289e28231097bcd3 + .quad 0x527bb94a6ced3a9b + .quad 0x563d992a995b482e + .quad 0x3405d07c6e383801 + .quad 0x485035de2f64d8e5 + .quad 0x6b89069b20a7a9f7 + + // 2^168 * 8 * G + + .quad 0x812aa0416270220d + .quad 0x995a89faf9245b4e + .quad 0xffadc4ce5072ef05 + .quad 0x23bc2103aa73eb73 + .quad 0x4082fa8cb5c7db77 + .quad 0x068686f8c734c155 + .quad 0x29e6c8d9f6e7a57e + .quad 0x0473d308a7639bcf + .quad 0xcaee792603589e05 + .quad 0x2b4b421246dcc492 + .quad 0x02a1ef74e601a94f + .quad 0x102f73bfde04341a + + // 2^172 * 1 * G + + .quad 0xb5a2d50c7ec20d3e + .quad 0xc64bdd6ea0c97263 + .quad 0x56e89052c1ff734d + .quad 0x4929c6f72b2ffaba + .quad 0x358ecba293a36247 + .quad 0xaf8f9862b268fd65 + .quad 0x412f7e9968a01c89 + .quad 0x5786f312cd754524 + .quad 0x337788ffca14032c + .quad 0xf3921028447f1ee3 + .quad 0x8b14071f231bccad + .quad 0x4c817b4bf2344783 + + // 2^172 * 2 * G + + .quad 0x0ff853852871b96e + .quad 0xe13e9fab60c3f1bb + .quad 0xeefd595325344402 + .quad 0x0a37c37075b7744b + .quad 0x413ba057a40b4484 + .quad 0xba4c2e1a4f5f6a43 + .quad 0x614ba0a5aee1d61c + .quad 0x78a1531a8b05dc53 + .quad 0x6cbdf1703ad0562b + .quad 0x8ecf4830c92521a3 + .quad 0xdaebd303fd8424e7 + .quad 0x72ad82a42e5ec56f + + // 2^172 * 3 * G + + .quad 0x3f9e8e35bafb65f6 + .quad 0x39d69ec8f27293a1 + .quad 0x6cb8cd958cf6a3d0 + .quad 0x1734778173adae6d + .quad 0xc368939167024bc3 + .quad 0x8e69d16d49502fda + .quad 0xfcf2ec3ce45f4b29 + .quad 0x065f669ea3b4cbc4 + .quad 0x8a00aec75532db4d + .quad 0xb869a4e443e31bb1 + .quad 0x4a0f8552d3a7f515 + .quad 0x19adeb7c303d7c08 + + // 2^172 * 4 * G + + .quad 0xc720cb6153ead9a3 + .quad 0x55b2c97f512b636e + .quad 0xb1e35b5fd40290b1 + .quad 0x2fd9ccf13b530ee2 + .quad 0x9d05ba7d43c31794 + .quad 0x2470c8ff93322526 + .quad 0x8323dec816197438 + .quad 0x2852709881569b53 + .quad 0x07bd475b47f796b8 + .quad 0xd2c7b013542c8f54 + .quad 0x2dbd23f43b24f87e + .quad 0x6551afd77b0901d6 + + // 2^172 * 5 * G + + .quad 0x4546baaf54aac27f + .quad 0xf6f66fecb2a45a28 + .quad 0x582d1b5b562bcfe8 + .quad 0x44b123f3920f785f + .quad 0x68a24ce3a1d5c9ac + .quad 0xbb77a33d10ff6461 + .quad 0x0f86ce4425d3166e + .quad 0x56507c0950b9623b + .quad 0x1206f0b7d1713e63 + .quad 0x353fe3d915bafc74 + .quad 0x194ceb970ad9d94d + .quad 0x62fadd7cf9d03ad3 + + // 2^172 * 6 * G + + .quad 0xc6b5967b5598a074 + .quad 0x5efe91ce8e493e25 + .quad 0xd4b72c4549280888 + .quad 0x20ef1149a26740c2 + .quad 0x3cd7bc61e7ce4594 + .quad 0xcd6b35a9b7dd267e + .quad 0xa080abc84366ef27 + .quad 0x6ec7c46f59c79711 + .quad 0x2f07ad636f09a8a2 + .quad 0x8697e6ce24205e7d + .quad 0xc0aefc05ee35a139 + .quad 0x15e80958b5f9d897 + + // 2^172 * 7 * G + + .quad 0x25a5ef7d0c3e235b + .quad 0x6c39c17fbe134ee7 + .quad 0xc774e1342dc5c327 + .quad 0x021354b892021f39 + .quad 0x4dd1ed355bb061c4 + .quad 0x42dc0cef941c0700 + .quad 0x61305dc1fd86340e + .quad 0x56b2cc930e55a443 + .quad 0x1df79da6a6bfc5a2 + .quad 0x02f3a2749fde4369 + .quad 0xb323d9f2cda390a7 + .quad 0x7be0847b8774d363 + + // 2^172 * 8 * G + + .quad 0x8c99cc5a8b3f55c3 + .quad 0x0611d7253fded2a0 + .quad 0xed2995ff36b70a36 + .quad 0x1f699a54d78a2619 + .quad 0x1466f5af5307fa11 + .quad 0x817fcc7ded6c0af2 + .quad 0x0a6de44ec3a4a3fb + .quad 0x74071475bc927d0b + .quad 0xe77292f373e7ea8a + .quad 0x296537d2cb045a31 + .quad 0x1bd0653ed3274fde + .quad 0x2f9a2c4476bd2966 + + // 2^176 * 1 * G + + .quad 0xeb18b9ab7f5745c6 + .quad 0x023a8aee5787c690 + .quad 0xb72712da2df7afa9 + .quad 0x36597d25ea5c013d + .quad 0xa2b4dae0b5511c9a + .quad 0x7ac860292bffff06 + .quad 0x981f375df5504234 + .quad 0x3f6bd725da4ea12d + .quad 0x734d8d7b106058ac + .quad 0xd940579e6fc6905f + .quad 0x6466f8f99202932d + .quad 0x7b7ecc19da60d6d0 + + // 2^176 * 2 * G + + .quad 0x78c2373c695c690d + .quad 0xdd252e660642906e + .quad 0x951d44444ae12bd2 + .quad 0x4235ad7601743956 + .quad 0x6dae4a51a77cfa9b + .quad 0x82263654e7a38650 + .quad 0x09bbffcd8f2d82db + .quad 0x03bedc661bf5caba + .quad 0x6258cb0d078975f5 + .quad 0x492942549189f298 + .quad 0xa0cab423e2e36ee4 + .quad 0x0e7ce2b0cdf066a1 + + // 2^176 * 3 * G + + .quad 0xc494643ac48c85a3 + .quad 0xfd361df43c6139ad + .quad 0x09db17dd3ae94d48 + .quad 0x666e0a5d8fb4674a + .quad 0xfea6fedfd94b70f9 + .quad 0xf130c051c1fcba2d + .quad 0x4882d47e7f2fab89 + .quad 0x615256138aeceeb5 + .quad 0x2abbf64e4870cb0d + .quad 0xcd65bcf0aa458b6b + .quad 0x9abe4eba75e8985d + .quad 0x7f0bc810d514dee4 + + // 2^176 * 4 * G + + .quad 0xb9006ba426f4136f + .quad 0x8d67369e57e03035 + .quad 0xcbc8dfd94f463c28 + .quad 0x0d1f8dbcf8eedbf5 + .quad 0x83ac9dad737213a0 + .quad 0x9ff6f8ba2ef72e98 + .quad 0x311e2edd43ec6957 + .quad 0x1d3a907ddec5ab75 + .quad 0xba1693313ed081dc + .quad 0x29329fad851b3480 + .quad 0x0128013c030321cb + .quad 0x00011b44a31bfde3 + + // 2^176 * 5 * G + + .quad 0x3fdfa06c3fc66c0c + .quad 0x5d40e38e4dd60dd2 + .quad 0x7ae38b38268e4d71 + .quad 0x3ac48d916e8357e1 + .quad 0x16561f696a0aa75c + .quad 0xc1bf725c5852bd6a + .quad 0x11a8dd7f9a7966ad + .quad 0x63d988a2d2851026 + .quad 0x00120753afbd232e + .quad 0xe92bceb8fdd8f683 + .quad 0xf81669b384e72b91 + .quad 0x33fad52b2368a066 + + // 2^176 * 6 * G + + .quad 0x540649c6c5e41e16 + .quad 0x0af86430333f7735 + .quad 0xb2acfcd2f305e746 + .quad 0x16c0f429a256dca7 + .quad 0x8d2cc8d0c422cfe8 + .quad 0x072b4f7b05a13acb + .quad 0xa3feb6e6ecf6a56f + .quad 0x3cc355ccb90a71e2 + .quad 0xe9b69443903e9131 + .quad 0xb8a494cb7a5637ce + .quad 0xc87cd1a4baba9244 + .quad 0x631eaf426bae7568 + + // 2^176 * 7 * G + + .quad 0xb3e90410da66fe9f + .quad 0x85dd4b526c16e5a6 + .quad 0xbc3d97611ef9bf83 + .quad 0x5599648b1ea919b5 + .quad 0x47d975b9a3700de8 + .quad 0x7280c5fbe2f80552 + .quad 0x53658f2732e45de1 + .quad 0x431f2c7f665f80b5 + .quad 0xd6026344858f7b19 + .quad 0x14ab352fa1ea514a + .quad 0x8900441a2090a9d7 + .quad 0x7b04715f91253b26 + + // 2^176 * 8 * G + + .quad 0x83edbd28acf6ae43 + .quad 0x86357c8b7d5c7ab4 + .quad 0xc0404769b7eb2c44 + .quad 0x59b37bf5c2f6583f + .quad 0xb376c280c4e6bac6 + .quad 0x970ed3dd6d1d9b0b + .quad 0xb09a9558450bf944 + .quad 0x48d0acfa57cde223 + .quad 0xb60f26e47dabe671 + .quad 0xf1d1a197622f3a37 + .quad 0x4208ce7ee9960394 + .quad 0x16234191336d3bdb + + // 2^180 * 1 * G + + .quad 0xf19aeac733a63aef + .quad 0x2c7fba5d4442454e + .quad 0x5da87aa04795e441 + .quad 0x413051e1a4e0b0f5 + .quad 0x852dd1fd3d578bbe + .quad 0x2b65ce72c3286108 + .quad 0x658c07f4eace2273 + .quad 0x0933f804ec38ab40 + .quad 0xa7ab69798d496476 + .quad 0x8121aadefcb5abc8 + .quad 0xa5dc12ef7b539472 + .quad 0x07fd47065e45351a + + // 2^180 * 2 * G + + .quad 0xc8583c3d258d2bcd + .quad 0x17029a4daf60b73f + .quad 0xfa0fc9d6416a3781 + .quad 0x1c1e5fba38b3fb23 + .quad 0x304211559ae8e7c3 + .quad 0xf281b229944882a5 + .quad 0x8a13ac2e378250e4 + .quad 0x014afa0954ba48f4 + .quad 0xcb3197001bb3666c + .quad 0x330060524bffecb9 + .quad 0x293711991a88233c + .quad 0x291884363d4ed364 + + // 2^180 * 3 * G + + .quad 0x033c6805dc4babfa + .quad 0x2c15bf5e5596ecc1 + .quad 0x1bc70624b59b1d3b + .quad 0x3ede9850a19f0ec5 + .quad 0xfb9d37c3bc1ab6eb + .quad 0x02be14534d57a240 + .quad 0xf4d73415f8a5e1f6 + .quad 0x5964f4300ccc8188 + .quad 0xe44a23152d096800 + .quad 0x5c08c55970866996 + .quad 0xdf2db60a46affb6e + .quad 0x579155c1f856fd89 + + // 2^180 * 4 * G + + .quad 0x96324edd12e0c9ef + .quad 0x468b878df2420297 + .quad 0x199a3776a4f573be + .quad 0x1e7fbcf18e91e92a + .quad 0xb5f16b630817e7a6 + .quad 0x808c69233c351026 + .quad 0x324a983b54cef201 + .quad 0x53c092084a485345 + .quad 0xd2d41481f1cbafbf + .quad 0x231d2db6716174e5 + .quad 0x0b7d7656e2a55c98 + .quad 0x3e955cd82aa495f6 + + // 2^180 * 5 * G + + .quad 0xe48f535e3ed15433 + .quad 0xd075692a0d7270a3 + .quad 0x40fbd21daade6387 + .quad 0x14264887cf4495f5 + .quad 0xab39f3ef61bb3a3f + .quad 0x8eb400652eb9193e + .quad 0xb5de6ecc38c11f74 + .quad 0x654d7e9626f3c49f + .quad 0xe564cfdd5c7d2ceb + .quad 0x82eeafded737ccb9 + .quad 0x6107db62d1f9b0ab + .quad 0x0b6baac3b4358dbb + + // 2^180 * 6 * G + + .quad 0x7ae62bcb8622fe98 + .quad 0x47762256ceb891af + .quad 0x1a5a92bcf2e406b4 + .quad 0x7d29401784e41501 + .quad 0x204abad63700a93b + .quad 0xbe0023d3da779373 + .quad 0xd85f0346633ab709 + .quad 0x00496dc490820412 + .quad 0x1c74b88dc27e6360 + .quad 0x074854268d14850c + .quad 0xa145fb7b3e0dcb30 + .quad 0x10843f1b43803b23 + + // 2^180 * 7 * G + + .quad 0xc5f90455376276dd + .quad 0xce59158dd7645cd9 + .quad 0x92f65d511d366b39 + .quad 0x11574b6e526996c4 + .quad 0xd56f672de324689b + .quad 0xd1da8aedb394a981 + .quad 0xdd7b58fe9168cfed + .quad 0x7ce246cd4d56c1e8 + .quad 0xb8f4308e7f80be53 + .quad 0x5f3cb8cb34a9d397 + .quad 0x18a961bd33cc2b2c + .quad 0x710045fb3a9af671 + + // 2^180 * 8 * G + + .quad 0x73f93d36101b95eb + .quad 0xfaef33794f6f4486 + .quad 0x5651735f8f15e562 + .quad 0x7fa3f19058b40da1 + .quad 0xa03fc862059d699e + .quad 0x2370cfa19a619e69 + .quad 0xc4fe3b122f823deb + .quad 0x1d1b056fa7f0844e + .quad 0x1bc64631e56bf61f + .quad 0xd379ab106e5382a3 + .quad 0x4d58c57e0540168d + .quad 0x566256628442d8e4 + + // 2^184 * 1 * G + + .quad 0xb9e499def6267ff6 + .quad 0x7772ca7b742c0843 + .quad 0x23a0153fe9a4f2b1 + .quad 0x2cdfdfecd5d05006 + .quad 0xdd499cd61ff38640 + .quad 0x29cd9bc3063625a0 + .quad 0x51e2d8023dd73dc3 + .quad 0x4a25707a203b9231 + .quad 0x2ab7668a53f6ed6a + .quad 0x304242581dd170a1 + .quad 0x4000144c3ae20161 + .quad 0x5721896d248e49fc + + // 2^184 * 2 * G + + .quad 0x0b6e5517fd181bae + .quad 0x9022629f2bb963b4 + .quad 0x5509bce932064625 + .quad 0x578edd74f63c13da + .quad 0x285d5091a1d0da4e + .quad 0x4baa6fa7b5fe3e08 + .quad 0x63e5177ce19393b3 + .quad 0x03c935afc4b030fd + .quad 0x997276c6492b0c3d + .quad 0x47ccc2c4dfe205fc + .quad 0xdcd29b84dd623a3c + .quad 0x3ec2ab590288c7a2 + + // 2^184 * 3 * G + + .quad 0xa1a0d27be4d87bb9 + .quad 0xa98b4deb61391aed + .quad 0x99a0ddd073cb9b83 + .quad 0x2dd5c25a200fcace + .quad 0xa7213a09ae32d1cb + .quad 0x0f2b87df40f5c2d5 + .quad 0x0baea4c6e81eab29 + .quad 0x0e1bf66c6adbac5e + .quad 0xe2abd5e9792c887e + .quad 0x1a020018cb926d5d + .quad 0xbfba69cdbaae5f1e + .quad 0x730548b35ae88f5f + + // 2^184 * 4 * G + + .quad 0xc43551a3cba8b8ee + .quad 0x65a26f1db2115f16 + .quad 0x760f4f52ab8c3850 + .quad 0x3043443b411db8ca + .quad 0x805b094ba1d6e334 + .quad 0xbf3ef17709353f19 + .quad 0x423f06cb0622702b + .quad 0x585a2277d87845dd + .quad 0xa18a5f8233d48962 + .quad 0x6698c4b5ec78257f + .quad 0xa78e6fa5373e41ff + .quad 0x7656278950ef981f + + // 2^184 * 5 * G + + .quad 0x38c3cf59d51fc8c0 + .quad 0x9bedd2fd0506b6f2 + .quad 0x26bf109fab570e8f + .quad 0x3f4160a8c1b846a6 + .quad 0xe17073a3ea86cf9d + .quad 0x3a8cfbb707155fdc + .quad 0x4853e7fc31838a8e + .quad 0x28bbf484b613f616 + .quad 0xf2612f5c6f136c7c + .quad 0xafead107f6dd11be + .quad 0x527e9ad213de6f33 + .quad 0x1e79cb358188f75d + + // 2^184 * 6 * G + + .quad 0x013436c3eef7e3f1 + .quad 0x828b6a7ffe9e10f8 + .quad 0x7ff908e5bcf9defc + .quad 0x65d7951b3a3b3831 + .quad 0x77e953d8f5e08181 + .quad 0x84a50c44299dded9 + .quad 0xdc6c2d0c864525e5 + .quad 0x478ab52d39d1f2f4 + .quad 0x66a6a4d39252d159 + .quad 0xe5dde1bc871ac807 + .quad 0xb82c6b40a6c1c96f + .quad 0x16d87a411a212214 + + // 2^184 * 7 * G + + .quad 0xb3bd7e5a42066215 + .quad 0x879be3cd0c5a24c1 + .quad 0x57c05db1d6f994b7 + .quad 0x28f87c8165f38ca6 + .quad 0xfba4d5e2d54e0583 + .quad 0xe21fafd72ebd99fa + .quad 0x497ac2736ee9778f + .quad 0x1f990b577a5a6dde + .quad 0xa3344ead1be8f7d6 + .quad 0x7d1e50ebacea798f + .quad 0x77c6569e520de052 + .quad 0x45882fe1534d6d3e + + // 2^184 * 8 * G + + .quad 0x6669345d757983d6 + .quad 0x62b6ed1117aa11a6 + .quad 0x7ddd1857985e128f + .quad 0x688fe5b8f626f6dd + .quad 0xd8ac9929943c6fe4 + .quad 0xb5f9f161a38392a2 + .quad 0x2699db13bec89af3 + .quad 0x7dcf843ce405f074 + .quad 0x6c90d6484a4732c0 + .quad 0xd52143fdca563299 + .quad 0xb3be28c3915dc6e1 + .quad 0x6739687e7327191b + + // 2^188 * 1 * G + + .quad 0x9f65c5ea200814cf + .quad 0x840536e169a31740 + .quad 0x8b0ed13925c8b4ad + .quad 0x0080dbafe936361d + .quad 0x8ce5aad0c9cb971f + .quad 0x1156aaa99fd54a29 + .quad 0x41f7247015af9b78 + .quad 0x1fe8cca8420f49aa + .quad 0x72a1848f3c0cc82a + .quad 0x38c560c2877c9e54 + .quad 0x5004e228ce554140 + .quad 0x042418a103429d71 + + // 2^188 * 2 * G + + .quad 0x899dea51abf3ff5f + .quad 0x9b93a8672fc2d8ba + .quad 0x2c38cb97be6ebd5c + .quad 0x114d578497263b5d + .quad 0x58e84c6f20816247 + .quad 0x8db2b2b6e36fd793 + .quad 0x977182561d484d85 + .quad 0x0822024f8632abd7 + .quad 0xb301bb7c6b1beca3 + .quad 0x55393f6dc6eb1375 + .quad 0x910d281097b6e4eb + .quad 0x1ad4548d9d479ea3 + + // 2^188 * 3 * G + + .quad 0xcd5a7da0389a48fd + .quad 0xb38fa4aa9a78371e + .quad 0xc6d9761b2cdb8e6c + .quad 0x35cf51dbc97e1443 + .quad 0xa06fe66d0fe9fed3 + .quad 0xa8733a401c587909 + .quad 0x30d14d800df98953 + .quad 0x41ce5876c7b30258 + .quad 0x59ac3bc5d670c022 + .quad 0xeae67c109b119406 + .quad 0x9798bdf0b3782fda + .quad 0x651e3201fd074092 + + // 2^188 * 4 * G + + .quad 0xd63d8483ef30c5cf + .quad 0x4cd4b4962361cc0c + .quad 0xee90e500a48426ac + .quad 0x0af51d7d18c14eeb + .quad 0xa57ba4a01efcae9e + .quad 0x769f4beedc308a94 + .quad 0xd1f10eeb3603cb2e + .quad 0x4099ce5e7e441278 + .quad 0x1ac98e4f8a5121e9 + .quad 0x7dae9544dbfa2fe0 + .quad 0x8320aa0dd6430df9 + .quad 0x667282652c4a2fb5 + + // 2^188 * 5 * G + + .quad 0x874621f4d86bc9ab + .quad 0xb54c7bbe56fe6fea + .quad 0x077a24257fadc22c + .quad 0x1ab53be419b90d39 + .quad 0xada8b6e02946db23 + .quad 0x1c0ce51a7b253ab7 + .quad 0x8448c85a66dd485b + .quad 0x7f1fc025d0675adf + .quad 0xd8ee1b18319ea6aa + .quad 0x004d88083a21f0da + .quad 0x3bd6aa1d883a4f4b + .quad 0x4db9a3a6dfd9fd14 + + // 2^188 * 6 * G + + .quad 0x8ce7b23bb99c0755 + .quad 0x35c5d6edc4f50f7a + .quad 0x7e1e2ed2ed9b50c3 + .quad 0x36305f16e8934da1 + .quad 0xd95b00bbcbb77c68 + .quad 0xddbc846a91f17849 + .quad 0x7cf700aebe28d9b3 + .quad 0x5ce1285c85d31f3e + .quad 0x31b6972d98b0bde8 + .quad 0x7d920706aca6de5b + .quad 0xe67310f8908a659f + .quad 0x50fac2a6efdf0235 + + // 2^188 * 7 * G + + .quad 0xf3d3a9f35b880f5a + .quad 0xedec050cdb03e7c2 + .quad 0xa896981ff9f0b1a2 + .quad 0x49a4ae2bac5e34a4 + .quad 0x295b1c86f6f449bc + .quad 0x51b2e84a1f0ab4dd + .quad 0xc001cb30aa8e551d + .quad 0x6a28d35944f43662 + .quad 0x28bb12ee04a740e0 + .quad 0x14313bbd9bce8174 + .quad 0x72f5b5e4e8c10c40 + .quad 0x7cbfb19936adcd5b + + // 2^188 * 8 * G + + .quad 0xa311ddc26b89792d + .quad 0x1b30b4c6da512664 + .quad 0x0ca77b4ccf150859 + .quad 0x1de443df1b009408 + .quad 0x8e793a7acc36e6e0 + .quad 0xf9fab7a37d586eed + .quad 0x3a4f9692bae1f4e4 + .quad 0x1c14b03eff5f447e + .quad 0x19647bd114a85291 + .quad 0x57b76cb21034d3af + .quad 0x6329db440f9d6dfa + .quad 0x5ef43e586a571493 + + // 2^192 * 1 * G + + .quad 0xef782014385675a6 + .quad 0xa2649f30aafda9e8 + .quad 0x4cd1eb505cdfa8cb + .quad 0x46115aba1d4dc0b3 + .quad 0xa66dcc9dc80c1ac0 + .quad 0x97a05cf41b38a436 + .quad 0xa7ebf3be95dbd7c6 + .quad 0x7da0b8f68d7e7dab + .quad 0xd40f1953c3b5da76 + .quad 0x1dac6f7321119e9b + .quad 0x03cc6021feb25960 + .quad 0x5a5f887e83674b4b + + // 2^192 * 2 * G + + .quad 0x8f6301cf70a13d11 + .quad 0xcfceb815350dd0c4 + .quad 0xf70297d4a4bca47e + .quad 0x3669b656e44d1434 + .quad 0x9e9628d3a0a643b9 + .quad 0xb5c3cb00e6c32064 + .quad 0x9b5302897c2dec32 + .quad 0x43e37ae2d5d1c70c + .quad 0x387e3f06eda6e133 + .quad 0x67301d5199a13ac0 + .quad 0xbd5ad8f836263811 + .quad 0x6a21e6cd4fd5e9be + + // 2^192 * 3 * G + + .quad 0xf1c6170a3046e65f + .quad 0x58712a2a00d23524 + .quad 0x69dbbd3c8c82b755 + .quad 0x586bf9f1a195ff57 + .quad 0xef4129126699b2e3 + .quad 0x71d30847708d1301 + .quad 0x325432d01182b0bd + .quad 0x45371b07001e8b36 + .quad 0xa6db088d5ef8790b + .quad 0x5278f0dc610937e5 + .quad 0xac0349d261a16eb8 + .quad 0x0eafb03790e52179 + + // 2^192 * 4 * G + + .quad 0x960555c13748042f + .quad 0x219a41e6820baa11 + .quad 0x1c81f73873486d0c + .quad 0x309acc675a02c661 + .quad 0x5140805e0f75ae1d + .quad 0xec02fbe32662cc30 + .quad 0x2cebdf1eea92396d + .quad 0x44ae3344c5435bb3 + .quad 0x9cf289b9bba543ee + .quad 0xf3760e9d5ac97142 + .quad 0x1d82e5c64f9360aa + .quad 0x62d5221b7f94678f + + // 2^192 * 5 * G + + .quad 0x524c299c18d0936d + .quad 0xc86bb56c8a0c1a0c + .quad 0xa375052edb4a8631 + .quad 0x5c0efde4bc754562 + .quad 0x7585d4263af77a3c + .quad 0xdfae7b11fee9144d + .quad 0xa506708059f7193d + .quad 0x14f29a5383922037 + .quad 0xdf717edc25b2d7f5 + .quad 0x21f970db99b53040 + .quad 0xda9234b7c3ed4c62 + .quad 0x5e72365c7bee093e + + // 2^192 * 6 * G + + .quad 0x575bfc074571217f + .quad 0x3779675d0694d95b + .quad 0x9a0a37bbf4191e33 + .quad 0x77f1104c47b4eabc + .quad 0x7d9339062f08b33e + .quad 0x5b9659e5df9f32be + .quad 0xacff3dad1f9ebdfd + .quad 0x70b20555cb7349b7 + .quad 0xbe5113c555112c4c + .quad 0x6688423a9a881fcd + .quad 0x446677855e503b47 + .quad 0x0e34398f4a06404a + + // 2^192 * 7 * G + + .quad 0xb67d22d93ecebde8 + .quad 0x09b3e84127822f07 + .quad 0x743fa61fb05b6d8d + .quad 0x5e5405368a362372 + .quad 0x18930b093e4b1928 + .quad 0x7de3e10e73f3f640 + .quad 0xf43217da73395d6f + .quad 0x6f8aded6ca379c3e + .quad 0xe340123dfdb7b29a + .quad 0x487b97e1a21ab291 + .quad 0xf9967d02fde6949e + .quad 0x780de72ec8d3de97 + + // 2^192 * 8 * G + + .quad 0x0ae28545089ae7bc + .quad 0x388ddecf1c7f4d06 + .quad 0x38ac15510a4811b8 + .quad 0x0eb28bf671928ce4 + .quad 0x671feaf300f42772 + .quad 0x8f72eb2a2a8c41aa + .quad 0x29a17fd797373292 + .quad 0x1defc6ad32b587a6 + .quad 0xaf5bbe1aef5195a7 + .quad 0x148c1277917b15ed + .quad 0x2991f7fb7ae5da2e + .quad 0x467d201bf8dd2867 + + // 2^196 * 1 * G + + .quad 0x7906ee72f7bd2e6b + .quad 0x05d270d6109abf4e + .quad 0x8d5cfe45b941a8a4 + .quad 0x44c218671c974287 + .quad 0x745f9d56296bc318 + .quad 0x993580d4d8152e65 + .quad 0xb0e5b13f5839e9ce + .quad 0x51fc2b28d43921c0 + .quad 0x1b8fd11795e2a98c + .quad 0x1c4e5ee12b6b6291 + .quad 0x5b30e7107424b572 + .quad 0x6e6b9de84c4f4ac6 + + // 2^196 * 2 * G + + .quad 0xdff25fce4b1de151 + .quad 0xd841c0c7e11c4025 + .quad 0x2554b3c854749c87 + .quad 0x2d292459908e0df9 + .quad 0x6b7c5f10f80cb088 + .quad 0x736b54dc56e42151 + .quad 0xc2b620a5c6ef99c4 + .quad 0x5f4c802cc3a06f42 + .quad 0x9b65c8f17d0752da + .quad 0x881ce338c77ee800 + .quad 0xc3b514f05b62f9e3 + .quad 0x66ed5dd5bec10d48 + + // 2^196 * 3 * G + + .quad 0x7d38a1c20bb2089d + .quad 0x808334e196ccd412 + .quad 0xc4a70b8c6c97d313 + .quad 0x2eacf8bc03007f20 + .quad 0xf0adf3c9cbca047d + .quad 0x81c3b2cbf4552f6b + .quad 0xcfda112d44735f93 + .quad 0x1f23a0c77e20048c + .quad 0xf235467be5bc1570 + .quad 0x03d2d9020dbab38c + .quad 0x27529aa2fcf9e09e + .quad 0x0840bef29d34bc50 + + // 2^196 * 4 * G + + .quad 0x796dfb35dc10b287 + .quad 0x27176bcd5c7ff29d + .quad 0x7f3d43e8c7b24905 + .quad 0x0304f5a191c54276 + .quad 0xcd54e06b7f37e4eb + .quad 0x8cc15f87f5e96cca + .quad 0xb8248bb0d3597dce + .quad 0x246affa06074400c + .quad 0x37d88e68fbe45321 + .quad 0x86097548c0d75032 + .quad 0x4e9b13ef894a0d35 + .quad 0x25a83cac5753d325 + + // 2^196 * 5 * G + + .quad 0x10222f48eed8165e + .quad 0x623fc1234b8bcf3a + .quad 0x1e145c09c221e8f0 + .quad 0x7ccfa59fca782630 + .quad 0x9f0f66293952b6e2 + .quad 0x33db5e0e0934267b + .quad 0xff45252bd609fedc + .quad 0x06be10f5c506e0c9 + .quad 0x1a9615a9b62a345f + .quad 0x22050c564a52fecc + .quad 0xa7a2788528bc0dfe + .quad 0x5e82770a1a1ee71d + + // 2^196 * 6 * G + + .quad 0x35425183ad896a5c + .quad 0xe8673afbe78d52f6 + .quad 0x2c66f25f92a35f64 + .quad 0x09d04f3b3b86b102 + .quad 0xe802e80a42339c74 + .quad 0x34175166a7fffae5 + .quad 0x34865d1f1c408cae + .quad 0x2cca982c605bc5ee + .quad 0xfd2d5d35197dbe6e + .quad 0x207c2eea8be4ffa3 + .quad 0x2613d8db325ae918 + .quad 0x7a325d1727741d3e + + // 2^196 * 7 * G + + .quad 0xd036b9bbd16dfde2 + .quad 0xa2055757c497a829 + .quad 0x8e6cc966a7f12667 + .quad 0x4d3b1a791239c180 + .quad 0xecd27d017e2a076a + .quad 0xd788689f1636495e + .quad 0x52a61af0919233e5 + .quad 0x2a479df17bb1ae64 + .quad 0x9e5eee8e33db2710 + .quad 0x189854ded6c43ca5 + .quad 0xa41c22c592718138 + .quad 0x27ad5538a43a5e9b + + // 2^196 * 8 * G + + .quad 0x2746dd4b15350d61 + .quad 0xd03fcbc8ee9521b7 + .quad 0xe86e365a138672ca + .quad 0x510e987f7e7d89e2 + .quad 0xcb5a7d638e47077c + .quad 0x8db7536120a1c059 + .quad 0x549e1e4d8bedfdcc + .quad 0x080153b7503b179d + .quad 0xdda69d930a3ed3e3 + .quad 0x3d386ef1cd60a722 + .quad 0xc817ad58bdaa4ee6 + .quad 0x23be8d554fe7372a + + // 2^200 * 1 * G + + .quad 0x95fe919a74ef4fad + .quad 0x3a827becf6a308a2 + .quad 0x964e01d309a47b01 + .quad 0x71c43c4f5ba3c797 + .quad 0xbc1ef4bd567ae7a9 + .quad 0x3f624cb2d64498bd + .quad 0xe41064d22c1f4ec8 + .quad 0x2ef9c5a5ba384001 + .quad 0xb6fd6df6fa9e74cd + .quad 0xf18278bce4af267a + .quad 0x8255b3d0f1ef990e + .quad 0x5a758ca390c5f293 + + // 2^200 * 2 * G + + .quad 0xa2b72710d9462495 + .quad 0x3aa8c6d2d57d5003 + .quad 0xe3d400bfa0b487ca + .quad 0x2dbae244b3eb72ec + .quad 0x8ce0918b1d61dc94 + .quad 0x8ded36469a813066 + .quad 0xd4e6a829afe8aad3 + .quad 0x0a738027f639d43f + .quad 0x980f4a2f57ffe1cc + .quad 0x00670d0de1839843 + .quad 0x105c3f4a49fb15fd + .quad 0x2698ca635126a69c + + // 2^200 * 3 * G + + .quad 0xe765318832b0ba78 + .quad 0x381831f7925cff8b + .quad 0x08a81b91a0291fcc + .quad 0x1fb43dcc49caeb07 + .quad 0x2e3d702f5e3dd90e + .quad 0x9e3f0918e4d25386 + .quad 0x5e773ef6024da96a + .quad 0x3c004b0c4afa3332 + .quad 0x9aa946ac06f4b82b + .quad 0x1ca284a5a806c4f3 + .quad 0x3ed3265fc6cd4787 + .quad 0x6b43fd01cd1fd217 + + // 2^200 * 4 * G + + .quad 0xc7a75d4b4697c544 + .quad 0x15fdf848df0fffbf + .quad 0x2868b9ebaa46785a + .quad 0x5a68d7105b52f714 + .quad 0xb5c742583e760ef3 + .quad 0x75dc52b9ee0ab990 + .quad 0xbf1427c2072b923f + .quad 0x73420b2d6ff0d9f0 + .quad 0xaf2cf6cb9e851e06 + .quad 0x8f593913c62238c4 + .quad 0xda8ab89699fbf373 + .quad 0x3db5632fea34bc9e + + // 2^200 * 5 * G + + .quad 0xf46eee2bf75dd9d8 + .quad 0x0d17b1f6396759a5 + .quad 0x1bf2d131499e7273 + .quad 0x04321adf49d75f13 + .quad 0x2e4990b1829825d5 + .quad 0xedeaeb873e9a8991 + .quad 0xeef03d394c704af8 + .quad 0x59197ea495df2b0e + .quad 0x04e16019e4e55aae + .quad 0xe77b437a7e2f92e9 + .quad 0xc7ce2dc16f159aa4 + .quad 0x45eafdc1f4d70cc0 + + // 2^200 * 6 * G + + .quad 0x698401858045d72b + .quad 0x4c22faa2cf2f0651 + .quad 0x941a36656b222dc6 + .quad 0x5a5eebc80362dade + .quad 0xb60e4624cfccb1ed + .quad 0x59dbc292bd5c0395 + .quad 0x31a09d1ddc0481c9 + .quad 0x3f73ceea5d56d940 + .quad 0xb7a7bfd10a4e8dc6 + .quad 0xbe57007e44c9b339 + .quad 0x60c1207f1557aefa + .quad 0x26058891266218db + + // 2^200 * 7 * G + + .quad 0x59f704a68360ff04 + .quad 0xc3d93fde7661e6f4 + .quad 0x831b2a7312873551 + .quad 0x54ad0c2e4e615d57 + .quad 0x4c818e3cc676e542 + .quad 0x5e422c9303ceccad + .quad 0xec07cccab4129f08 + .quad 0x0dedfa10b24443b8 + .quad 0xee3b67d5b82b522a + .quad 0x36f163469fa5c1eb + .quad 0xa5b4d2f26ec19fd3 + .quad 0x62ecb2baa77a9408 + + // 2^200 * 8 * G + + .quad 0xe5ed795261152b3d + .quad 0x4962357d0eddd7d1 + .quad 0x7482c8d0b96b4c71 + .quad 0x2e59f919a966d8be + .quad 0x92072836afb62874 + .quad 0x5fcd5e8579e104a5 + .quad 0x5aad01adc630a14a + .quad 0x61913d5075663f98 + .quad 0x0dc62d361a3231da + .quad 0xfa47583294200270 + .quad 0x02d801513f9594ce + .quad 0x3ddbc2a131c05d5c + + // 2^204 * 1 * G + + .quad 0x3f50a50a4ffb81ef + .quad 0xb1e035093bf420bf + .quad 0x9baa8e1cc6aa2cd0 + .quad 0x32239861fa237a40 + .quad 0xfb735ac2004a35d1 + .quad 0x31de0f433a6607c3 + .quad 0x7b8591bfc528d599 + .quad 0x55be9a25f5bb050c + .quad 0x0d005acd33db3dbf + .quad 0x0111b37c80ac35e2 + .quad 0x4892d66c6f88ebeb + .quad 0x770eadb16508fbcd + + // 2^204 * 2 * G + + .quad 0x8451f9e05e4e89dd + .quad 0xc06302ffbc793937 + .quad 0x5d22749556a6495c + .quad 0x09a6755ca05603fb + .quad 0xf1d3b681a05071b9 + .quad 0x2207659a3592ff3a + .quad 0x5f0169297881e40e + .quad 0x16bedd0e86ba374e + .quad 0x5ecccc4f2c2737b5 + .quad 0x43b79e0c2dccb703 + .quad 0x33e008bc4ec43df3 + .quad 0x06c1b840f07566c0 + + // 2^204 * 3 * G + + .quad 0x7688a5c6a388f877 + .quad 0x02a96c14deb2b6ac + .quad 0x64c9f3431b8c2af8 + .quad 0x3628435554a1eed6 + .quad 0x69ee9e7f9b02805c + .quad 0xcbff828a547d1640 + .quad 0x3d93a869b2430968 + .quad 0x46b7b8cd3fe26972 + .quad 0xe9812086fe7eebe0 + .quad 0x4cba6be72f515437 + .quad 0x1d04168b516efae9 + .quad 0x5ea1391043982cb9 + + // 2^204 * 4 * G + + .quad 0x49125c9cf4702ee1 + .quad 0x4520b71f8b25b32d + .quad 0x33193026501fef7e + .quad 0x656d8997c8d2eb2b + .quad 0x6f2b3be4d5d3b002 + .quad 0xafec33d96a09c880 + .quad 0x035f73a4a8bcc4cc + .quad 0x22c5b9284662198b + .quad 0xcb58c8fe433d8939 + .quad 0x89a0cb2e6a8d7e50 + .quad 0x79ca955309fbbe5a + .quad 0x0c626616cd7fc106 + + // 2^204 * 5 * G + + .quad 0x1ffeb80a4879b61f + .quad 0x6396726e4ada21ed + .quad 0x33c7b093368025ba + .quad 0x471aa0c6f3c31788 + .quad 0x8fdfc379fbf454b1 + .quad 0x45a5a970f1a4b771 + .quad 0xac921ef7bad35915 + .quad 0x42d088dca81c2192 + .quad 0x8fda0f37a0165199 + .quad 0x0adadb77c8a0e343 + .quad 0x20fbfdfcc875e820 + .quad 0x1cf2bea80c2206e7 + + // 2^204 * 6 * G + + .quad 0xc2ddf1deb36202ac + .quad 0x92a5fe09d2e27aa5 + .quad 0x7d1648f6fc09f1d3 + .quad 0x74c2cc0513bc4959 + .quad 0x982d6e1a02c0412f + .quad 0x90fa4c83db58e8fe + .quad 0x01c2f5bcdcb18bc0 + .quad 0x686e0c90216abc66 + .quad 0x1fadbadba54395a7 + .quad 0xb41a02a0ae0da66a + .quad 0xbf19f598bba37c07 + .quad 0x6a12b8acde48430d + + // 2^204 * 7 * G + + .quad 0xf8daea1f39d495d9 + .quad 0x592c190e525f1dfc + .quad 0xdb8cbd04c9991d1b + .quad 0x11f7fda3d88f0cb7 + .quad 0x793bdd801aaeeb5f + .quad 0x00a2a0aac1518871 + .quad 0xe8a373a31f2136b4 + .quad 0x48aab888fc91ef19 + .quad 0x041f7e925830f40e + .quad 0x002d6ca979661c06 + .quad 0x86dc9ff92b046a2e + .quad 0x760360928b0493d1 + + // 2^204 * 8 * G + + .quad 0x21bb41c6120cf9c6 + .quad 0xeab2aa12decda59b + .quad 0xc1a72d020aa48b34 + .quad 0x215d4d27e87d3b68 + .quad 0xb43108e5695a0b05 + .quad 0x6cb00ee8ad37a38b + .quad 0x5edad6eea3537381 + .quad 0x3f2602d4b6dc3224 + .quad 0xc8b247b65bcaf19c + .quad 0x49779dc3b1b2c652 + .quad 0x89a180bbd5ece2e2 + .quad 0x13f098a3cec8e039 + + // 2^208 * 1 * G + + .quad 0x9adc0ff9ce5ec54b + .quad 0x039c2a6b8c2f130d + .quad 0x028007c7f0f89515 + .quad 0x78968314ac04b36b + .quad 0xf3aa57a22796bb14 + .quad 0x883abab79b07da21 + .quad 0xe54be21831a0391c + .quad 0x5ee7fb38d83205f9 + .quad 0x538dfdcb41446a8e + .quad 0xa5acfda9434937f9 + .quad 0x46af908d263c8c78 + .quad 0x61d0633c9bca0d09 + + // 2^208 * 2 * G + + .quad 0x63744935ffdb2566 + .quad 0xc5bd6b89780b68bb + .quad 0x6f1b3280553eec03 + .quad 0x6e965fd847aed7f5 + .quad 0xada328bcf8fc73df + .quad 0xee84695da6f037fc + .quad 0x637fb4db38c2a909 + .quad 0x5b23ac2df8067bdc + .quad 0x9ad2b953ee80527b + .quad 0xe88f19aafade6d8d + .quad 0x0e711704150e82cf + .quad 0x79b9bbb9dd95dedc + + // 2^208 * 3 * G + + .quad 0xebb355406a3126c2 + .quad 0xd26383a868c8c393 + .quad 0x6c0c6429e5b97a82 + .quad 0x5065f158c9fd2147 + .quad 0xd1997dae8e9f7374 + .quad 0xa032a2f8cfbb0816 + .quad 0xcd6cba126d445f0a + .quad 0x1ba811460accb834 + .quad 0x708169fb0c429954 + .quad 0xe14600acd76ecf67 + .quad 0x2eaab98a70e645ba + .quad 0x3981f39e58a4faf2 + + // 2^208 * 4 * G + + .quad 0x18fb8a7559230a93 + .quad 0x1d168f6960e6f45d + .quad 0x3a85a94514a93cb5 + .quad 0x38dc083705acd0fd + .quad 0xc845dfa56de66fde + .quad 0xe152a5002c40483a + .quad 0xe9d2e163c7b4f632 + .quad 0x30f4452edcbc1b65 + .quad 0x856d2782c5759740 + .quad 0xfa134569f99cbecc + .quad 0x8844fc73c0ea4e71 + .quad 0x632d9a1a593f2469 + + // 2^208 * 5 * G + + .quad 0xf6bb6b15b807cba6 + .quad 0x1823c7dfbc54f0d7 + .quad 0xbb1d97036e29670b + .quad 0x0b24f48847ed4a57 + .quad 0xbf09fd11ed0c84a7 + .quad 0x63f071810d9f693a + .quad 0x21908c2d57cf8779 + .quad 0x3a5a7df28af64ba2 + .quad 0xdcdad4be511beac7 + .quad 0xa4538075ed26ccf2 + .quad 0xe19cff9f005f9a65 + .quad 0x34fcf74475481f63 + + // 2^208 * 6 * G + + .quad 0xc197e04c789767ca + .quad 0xb8714dcb38d9467d + .quad 0x55de888283f95fa8 + .quad 0x3d3bdc164dfa63f7 + .quad 0xa5bb1dab78cfaa98 + .quad 0x5ceda267190b72f2 + .quad 0x9309c9110a92608e + .quad 0x0119a3042fb374b0 + .quad 0x67a2d89ce8c2177d + .quad 0x669da5f66895d0c1 + .quad 0xf56598e5b282a2b0 + .quad 0x56c088f1ede20a73 + + // 2^208 * 7 * G + + .quad 0x336d3d1110a86e17 + .quad 0xd7f388320b75b2fa + .quad 0xf915337625072988 + .quad 0x09674c6b99108b87 + .quad 0x581b5fac24f38f02 + .quad 0xa90be9febae30cbd + .quad 0x9a2169028acf92f0 + .quad 0x038b7ea48359038f + .quad 0x9f4ef82199316ff8 + .quad 0x2f49d282eaa78d4f + .quad 0x0971a5ab5aef3174 + .quad 0x6e5e31025969eb65 + + // 2^208 * 8 * G + + .quad 0xb16c62f587e593fb + .quad 0x4999eddeca5d3e71 + .quad 0xb491c1e014cc3e6d + .quad 0x08f5114789a8dba8 + .quad 0x3304fb0e63066222 + .quad 0xfb35068987acba3f + .quad 0xbd1924778c1061a3 + .quad 0x3058ad43d1838620 + .quad 0x323c0ffde57663d0 + .quad 0x05c3df38a22ea610 + .quad 0xbdc78abdac994f9a + .quad 0x26549fa4efe3dc99 + + // 2^212 * 1 * G + + .quad 0x738b38d787ce8f89 + .quad 0xb62658e24179a88d + .quad 0x30738c9cf151316d + .quad 0x49128c7f727275c9 + .quad 0x04dbbc17f75396b9 + .quad 0x69e6a2d7d2f86746 + .quad 0xc6409d99f53eabc6 + .quad 0x606175f6332e25d2 + .quad 0x4021370ef540e7dd + .quad 0x0910d6f5a1f1d0a5 + .quad 0x4634aacd5b06b807 + .quad 0x6a39e6356944f235 + + // 2^212 * 2 * G + + .quad 0x96cd5640df90f3e7 + .quad 0x6c3a760edbfa25ea + .quad 0x24f3ef0959e33cc4 + .quad 0x42889e7e530d2e58 + .quad 0x1da1965774049e9d + .quad 0xfbcd6ea198fe352b + .quad 0xb1cbcd50cc5236a6 + .quad 0x1f5ec83d3f9846e2 + .quad 0x8efb23c3328ccb75 + .quad 0xaf42a207dd876ee9 + .quad 0x20fbdadc5dfae796 + .quad 0x241e246b06bf9f51 + + // 2^212 * 3 * G + + .quad 0x29e68e57ad6e98f6 + .quad 0x4c9260c80b462065 + .quad 0x3f00862ea51ebb4b + .quad 0x5bc2c77fb38d9097 + .quad 0x7eaafc9a6280bbb8 + .quad 0x22a70f12f403d809 + .quad 0x31ce40bb1bfc8d20 + .quad 0x2bc65635e8bd53ee + .quad 0xe8d5dc9fa96bad93 + .quad 0xe58fb17dde1947dc + .quad 0x681532ea65185fa3 + .quad 0x1fdd6c3b034a7830 + + // 2^212 * 4 * G + + .quad 0x0a64e28c55dc18fe + .quad 0xe3df9e993399ebdd + .quad 0x79ac432370e2e652 + .quad 0x35ff7fc33ae4cc0e + .quad 0x9c13a6a52dd8f7a9 + .quad 0x2dbb1f8c3efdcabf + .quad 0x961e32405e08f7b5 + .quad 0x48c8a121bbe6c9e5 + .quad 0xfc415a7c59646445 + .quad 0xd224b2d7c128b615 + .quad 0x6035c9c905fbb912 + .quad 0x42d7a91274429fab + + // 2^212 * 5 * G + + .quad 0x4e6213e3eaf72ed3 + .quad 0x6794981a43acd4e7 + .quad 0xff547cde6eb508cb + .quad 0x6fed19dd10fcb532 + .quad 0xa9a48947933da5bc + .quad 0x4a58920ec2e979ec + .quad 0x96d8800013e5ac4c + .quad 0x453692d74b48b147 + .quad 0xdd775d99a8559c6f + .quad 0xf42a2140df003e24 + .quad 0x5223e229da928a66 + .quad 0x063f46ba6d38f22c + + // 2^212 * 6 * G + + .quad 0xd2d242895f536694 + .quad 0xca33a2c542939b2c + .quad 0x986fada6c7ddb95c + .quad 0x5a152c042f712d5d + .quad 0x39843cb737346921 + .quad 0xa747fb0738c89447 + .quad 0xcb8d8031a245307e + .quad 0x67810f8e6d82f068 + .quad 0x3eeb8fbcd2287db4 + .quad 0x72c7d3a301a03e93 + .quad 0x5473e88cbd98265a + .quad 0x7324aa515921b403 + + // 2^212 * 7 * G + + .quad 0x857942f46c3cbe8e + .quad 0xa1d364b14730c046 + .quad 0x1c8ed914d23c41bf + .quad 0x0838e161eef6d5d2 + .quad 0xad23f6dae82354cb + .quad 0x6962502ab6571a6d + .quad 0x9b651636e38e37d1 + .quad 0x5cac5005d1a3312f + .quad 0x8cc154cce9e39904 + .quad 0x5b3a040b84de6846 + .quad 0xc4d8a61cb1be5d6e + .quad 0x40fb897bd8861f02 + + // 2^212 * 8 * G + + .quad 0x84c5aa9062de37a1 + .quad 0x421da5000d1d96e1 + .quad 0x788286306a9242d9 + .quad 0x3c5e464a690d10da + .quad 0xe57ed8475ab10761 + .quad 0x71435e206fd13746 + .quad 0x342f824ecd025632 + .quad 0x4b16281ea8791e7b + .quad 0xd1c101d50b813381 + .quad 0xdee60f1176ee6828 + .quad 0x0cb68893383f6409 + .quad 0x6183c565f6ff484a + + // 2^216 * 1 * G + + .quad 0x741d5a461e6bf9d6 + .quad 0x2305b3fc7777a581 + .quad 0xd45574a26474d3d9 + .quad 0x1926e1dc6401e0ff + .quad 0xdb468549af3f666e + .quad 0xd77fcf04f14a0ea5 + .quad 0x3df23ff7a4ba0c47 + .quad 0x3a10dfe132ce3c85 + .quad 0xe07f4e8aea17cea0 + .quad 0x2fd515463a1fc1fd + .quad 0x175322fd31f2c0f1 + .quad 0x1fa1d01d861e5d15 + + // 2^216 * 2 * G + + .quad 0xcc8055947d599832 + .quad 0x1e4656da37f15520 + .quad 0x99f6f7744e059320 + .quad 0x773563bc6a75cf33 + .quad 0x38dcac00d1df94ab + .quad 0x2e712bddd1080de9 + .quad 0x7f13e93efdd5e262 + .quad 0x73fced18ee9a01e5 + .quad 0x06b1e90863139cb3 + .quad 0xa493da67c5a03ecd + .quad 0x8d77cec8ad638932 + .quad 0x1f426b701b864f44 + + // 2^216 * 3 * G + + .quad 0xefc9264c41911c01 + .quad 0xf1a3b7b817a22c25 + .quad 0x5875da6bf30f1447 + .quad 0x4e1af5271d31b090 + .quad 0xf17e35c891a12552 + .quad 0xb76b8153575e9c76 + .quad 0xfa83406f0d9b723e + .quad 0x0b76bb1b3fa7e438 + .quad 0x08b8c1f97f92939b + .quad 0xbe6771cbd444ab6e + .quad 0x22e5646399bb8017 + .quad 0x7b6dd61eb772a955 + + // 2^216 * 4 * G + + .quad 0xb7adc1e850f33d92 + .quad 0x7998fa4f608cd5cf + .quad 0xad962dbd8dfc5bdb + .quad 0x703e9bceaf1d2f4f + .quad 0x5730abf9ab01d2c7 + .quad 0x16fb76dc40143b18 + .quad 0x866cbe65a0cbb281 + .quad 0x53fa9b659bff6afe + .quad 0x6c14c8e994885455 + .quad 0x843a5d6665aed4e5 + .quad 0x181bb73ebcd65af1 + .quad 0x398d93e5c4c61f50 + + // 2^216 * 5 * G + + .quad 0x1c4bd16733e248f3 + .quad 0xbd9e128715bf0a5f + .quad 0xd43f8cf0a10b0376 + .quad 0x53b09b5ddf191b13 + .quad 0xc3877c60d2e7e3f2 + .quad 0x3b34aaa030828bb1 + .quad 0x283e26e7739ef138 + .quad 0x699c9c9002c30577 + .quad 0xf306a7235946f1cc + .quad 0x921718b5cce5d97d + .quad 0x28cdd24781b4e975 + .quad 0x51caf30c6fcdd907 + + // 2^216 * 6 * G + + .quad 0xa60ba7427674e00a + .quad 0x630e8570a17a7bf3 + .quad 0x3758563dcf3324cc + .quad 0x5504aa292383fdaa + .quad 0x737af99a18ac54c7 + .quad 0x903378dcc51cb30f + .quad 0x2b89bc334ce10cc7 + .quad 0x12ae29c189f8e99a + .quad 0xa99ec0cb1f0d01cf + .quad 0x0dd1efcc3a34f7ae + .quad 0x55ca7521d09c4e22 + .quad 0x5fd14fe958eba5ea + + // 2^216 * 7 * G + + .quad 0xb5dc2ddf2845ab2c + .quad 0x069491b10a7fe993 + .quad 0x4daaf3d64002e346 + .quad 0x093ff26e586474d1 + .quad 0x3c42fe5ebf93cb8e + .quad 0xbedfa85136d4565f + .quad 0xe0f0859e884220e8 + .quad 0x7dd73f960725d128 + .quad 0xb10d24fe68059829 + .quad 0x75730672dbaf23e5 + .quad 0x1367253ab457ac29 + .quad 0x2f59bcbc86b470a4 + + // 2^216 * 8 * G + + .quad 0x83847d429917135f + .quad 0xad1b911f567d03d7 + .quad 0x7e7748d9be77aad1 + .quad 0x5458b42e2e51af4a + .quad 0x7041d560b691c301 + .quad 0x85201b3fadd7e71e + .quad 0x16c2e16311335585 + .quad 0x2aa55e3d010828b1 + .quad 0xed5192e60c07444f + .quad 0x42c54e2d74421d10 + .quad 0x352b4c82fdb5c864 + .quad 0x13e9004a8a768664 + + // 2^220 * 1 * G + + .quad 0xcbb5b5556c032bff + .quad 0xdf7191b729297a3a + .quad 0xc1ff7326aded81bb + .quad 0x71ade8bb68be03f5 + .quad 0x1e6284c5806b467c + .quad 0xc5f6997be75d607b + .quad 0x8b67d958b378d262 + .quad 0x3d88d66a81cd8b70 + .quad 0x8b767a93204ed789 + .quad 0x762fcacb9fa0ae2a + .quad 0x771febcc6dce4887 + .quad 0x343062158ff05fb3 + + // 2^220 * 2 * G + + .quad 0xe05da1a7e1f5bf49 + .quad 0x26457d6dd4736092 + .quad 0x77dcb07773cc32f6 + .quad 0x0a5d94969cdd5fcd + .quad 0xfce219072a7b31b4 + .quad 0x4d7adc75aa578016 + .quad 0x0ec276a687479324 + .quad 0x6d6d9d5d1fda4beb + .quad 0x22b1a58ae9b08183 + .quad 0xfd95d071c15c388b + .quad 0xa9812376850a0517 + .quad 0x33384cbabb7f335e + + // 2^220 * 3 * G + + .quad 0x3c6fa2680ca2c7b5 + .quad 0x1b5082046fb64fda + .quad 0xeb53349c5431d6de + .quad 0x5278b38f6b879c89 + .quad 0x33bc627a26218b8d + .quad 0xea80b21fc7a80c61 + .quad 0x9458b12b173e9ee6 + .quad 0x076247be0e2f3059 + .quad 0x52e105f61416375a + .quad 0xec97af3685abeba4 + .quad 0x26e6b50623a67c36 + .quad 0x5cf0e856f3d4fb01 + + // 2^220 * 4 * G + + .quad 0xf6c968731ae8cab4 + .quad 0x5e20741ecb4f92c5 + .quad 0x2da53be58ccdbc3e + .quad 0x2dddfea269970df7 + .quad 0xbeaece313db342a8 + .quad 0xcba3635b842db7ee + .quad 0xe88c6620817f13ef + .quad 0x1b9438aa4e76d5c6 + .quad 0x8a50777e166f031a + .quad 0x067b39f10fb7a328 + .quad 0x1925c9a6010fbd76 + .quad 0x6df9b575cc740905 + + // 2^220 * 5 * G + + .quad 0x42c1192927f6bdcf + .quad 0x8f91917a403d61ca + .quad 0xdc1c5a668b9e1f61 + .quad 0x1596047804ec0f8d + .quad 0xecdfc35b48cade41 + .quad 0x6a88471fb2328270 + .quad 0x740a4a2440a01b6a + .quad 0x471e5796003b5f29 + .quad 0xda96bbb3aced37ac + .quad 0x7a2423b5e9208cea + .quad 0x24cc5c3038aebae2 + .quad 0x50c356afdc5dae2f + + // 2^220 * 6 * G + + .quad 0x09dcbf4341c30318 + .quad 0xeeba061183181dce + .quad 0xc179c0cedc1e29a1 + .quad 0x1dbf7b89073f35b0 + .quad 0xcfed9cdf1b31b964 + .quad 0xf486a9858ca51af3 + .quad 0x14897265ea8c1f84 + .quad 0x784a53dd932acc00 + .quad 0x2d99f9df14fc4920 + .quad 0x76ccb60cc4499fe5 + .quad 0xa4132cbbe5cf0003 + .quad 0x3f93d82354f000ea + + // 2^220 * 7 * G + + .quad 0x8183e7689e04ce85 + .quad 0x678fb71e04465341 + .quad 0xad92058f6688edac + .quad 0x5da350d3532b099a + .quad 0xeaac12d179e14978 + .quad 0xff923ff3bbebff5e + .quad 0x4af663e40663ce27 + .quad 0x0fd381a811a5f5ff + .quad 0xf256aceca436df54 + .quad 0x108b6168ae69d6e8 + .quad 0x20d986cb6b5d036c + .quad 0x655957b9fee2af50 + + // 2^220 * 8 * G + + .quad 0xaea8b07fa902030f + .quad 0xf88c766af463d143 + .quad 0x15b083663c787a60 + .quad 0x08eab1148267a4a8 + .quad 0xbdc1409bd002d0ac + .quad 0x66660245b5ccd9a6 + .quad 0x82317dc4fade85ec + .quad 0x02fe934b6ad7df0d + .quad 0xef5cf100cfb7ea74 + .quad 0x22897633a1cb42ac + .quad 0xd4ce0c54cef285e2 + .quad 0x30408c048a146a55 + + // 2^224 * 1 * G + + .quad 0x739d8845832fcedb + .quad 0xfa38d6c9ae6bf863 + .quad 0x32bc0dcab74ffef7 + .quad 0x73937e8814bce45e + .quad 0xbb2e00c9193b877f + .quad 0xece3a890e0dc506b + .quad 0xecf3b7c036de649f + .quad 0x5f46040898de9e1a + .quad 0xb9037116297bf48d + .quad 0xa9d13b22d4f06834 + .quad 0xe19715574696bdc6 + .quad 0x2cf8a4e891d5e835 + + // 2^224 * 2 * G + + .quad 0x6d93fd8707110f67 + .quad 0xdd4c09d37c38b549 + .quad 0x7cb16a4cc2736a86 + .quad 0x2049bd6e58252a09 + .quad 0x2cb5487e17d06ba2 + .quad 0x24d2381c3950196b + .quad 0xd7659c8185978a30 + .quad 0x7a6f7f2891d6a4f6 + .quad 0x7d09fd8d6a9aef49 + .quad 0xf0ee60be5b3db90b + .quad 0x4c21b52c519ebfd4 + .quad 0x6011aadfc545941d + + // 2^224 * 3 * G + + .quad 0x5f67926dcf95f83c + .quad 0x7c7e856171289071 + .quad 0xd6a1e7f3998f7a5b + .quad 0x6fc5cc1b0b62f9e0 + .quad 0x63ded0c802cbf890 + .quad 0xfbd098ca0dff6aaa + .quad 0x624d0afdb9b6ed99 + .quad 0x69ce18b779340b1e + .quad 0xd1ef5528b29879cb + .quad 0xdd1aae3cd47e9092 + .quad 0x127e0442189f2352 + .quad 0x15596b3ae57101f1 + + // 2^224 * 4 * G + + .quad 0x462739d23f9179a2 + .quad 0xff83123197d6ddcf + .quad 0x1307deb553f2148a + .quad 0x0d2237687b5f4dda + .quad 0x09ff31167e5124ca + .quad 0x0be4158bd9c745df + .quad 0x292b7d227ef556e5 + .quad 0x3aa4e241afb6d138 + .quad 0x2cc138bf2a3305f5 + .quad 0x48583f8fa2e926c3 + .quad 0x083ab1a25549d2eb + .quad 0x32fcaa6e4687a36c + + // 2^224 * 5 * G + + .quad 0x7bc56e8dc57d9af5 + .quad 0x3e0bd2ed9df0bdf2 + .quad 0xaac014de22efe4a3 + .quad 0x4627e9cefebd6a5c + .quad 0x3207a4732787ccdf + .quad 0x17e31908f213e3f8 + .quad 0xd5b2ecd7f60d964e + .quad 0x746f6336c2600be9 + .quad 0x3f4af345ab6c971c + .quad 0xe288eb729943731f + .quad 0x33596a8a0344186d + .quad 0x7b4917007ed66293 + + // 2^224 * 6 * G + + .quad 0x2d85fb5cab84b064 + .quad 0x497810d289f3bc14 + .quad 0x476adc447b15ce0c + .quad 0x122ba376f844fd7b + .quad 0x54341b28dd53a2dd + .quad 0xaa17905bdf42fc3f + .quad 0x0ff592d94dd2f8f4 + .quad 0x1d03620fe08cd37d + .quad 0xc20232cda2b4e554 + .quad 0x9ed0fd42115d187f + .quad 0x2eabb4be7dd479d9 + .quad 0x02c70bf52b68ec4c + + // 2^224 * 7 * G + + .quad 0xa287ec4b5d0b2fbb + .quad 0x415c5790074882ca + .quad 0xe044a61ec1d0815c + .quad 0x26334f0a409ef5e0 + .quad 0xace532bf458d72e1 + .quad 0x5be768e07cb73cb5 + .quad 0x56cf7d94ee8bbde7 + .quad 0x6b0697e3feb43a03 + .quad 0xb6c8f04adf62a3c0 + .quad 0x3ef000ef076da45d + .quad 0x9c9cb95849f0d2a9 + .quad 0x1cc37f43441b2fae + + // 2^224 * 8 * G + + .quad 0x508f565a5cc7324f + .quad 0xd061c4c0e506a922 + .quad 0xfb18abdb5c45ac19 + .quad 0x6c6809c10380314a + .quad 0xd76656f1c9ceaeb9 + .quad 0x1c5b15f818e5656a + .quad 0x26e72832844c2334 + .quad 0x3a346f772f196838 + .quad 0xd2d55112e2da6ac8 + .quad 0xe9bd0331b1e851ed + .quad 0x960746dd8ec67262 + .quad 0x05911b9f6ef7c5d0 + + // 2^228 * 1 * G + + .quad 0xe9dcd756b637ff2d + .quad 0xec4c348fc987f0c4 + .quad 0xced59285f3fbc7b7 + .quad 0x3305354793e1ea87 + .quad 0x01c18980c5fe9f94 + .quad 0xcd656769716fd5c8 + .quad 0x816045c3d195a086 + .quad 0x6e2b7f3266cc7982 + .quad 0xcc802468f7c3568f + .quad 0x9de9ba8219974cb3 + .quad 0xabb7229cb5b81360 + .quad 0x44e2017a6fbeba62 + + // 2^228 * 2 * G + + .quad 0xc4c2a74354dab774 + .quad 0x8e5d4c3c4eaf031a + .quad 0xb76c23d242838f17 + .quad 0x749a098f68dce4ea + .quad 0x87f82cf3b6ca6ecd + .quad 0x580f893e18f4a0c2 + .quad 0x058930072604e557 + .quad 0x6cab6ac256d19c1d + .quad 0xdcdfe0a02cc1de60 + .quad 0x032665ff51c5575b + .quad 0x2c0c32f1073abeeb + .quad 0x6a882014cd7b8606 + + // 2^228 * 3 * G + + .quad 0xa52a92fea4747fb5 + .quad 0xdc12a4491fa5ab89 + .quad 0xd82da94bb847a4ce + .quad 0x4d77edce9512cc4e + .quad 0xd111d17caf4feb6e + .quad 0x050bba42b33aa4a3 + .quad 0x17514c3ceeb46c30 + .quad 0x54bedb8b1bc27d75 + .quad 0x77c8e14577e2189c + .quad 0xa3e46f6aff99c445 + .quad 0x3144dfc86d335343 + .quad 0x3a96559e7c4216a9 + + // 2^228 * 4 * G + + .quad 0x12550d37f42ad2ee + .quad 0x8b78e00498a1fbf5 + .quad 0x5d53078233894cb2 + .quad 0x02c84e4e3e498d0c + .quad 0x4493896880baaa52 + .quad 0x4c98afc4f285940e + .quad 0xef4aa79ba45448b6 + .quad 0x5278c510a57aae7f + .quad 0xa54dd074294c0b94 + .quad 0xf55d46b8df18ffb6 + .quad 0xf06fecc58dae8366 + .quad 0x588657668190d165 + + // 2^228 * 5 * G + + .quad 0xd47712311aef7117 + .quad 0x50343101229e92c7 + .quad 0x7a95e1849d159b97 + .quad 0x2449959b8b5d29c9 + .quad 0xbf5834f03de25cc3 + .quad 0xb887c8aed6815496 + .quad 0x5105221a9481e892 + .quad 0x6760ed19f7723f93 + .quad 0x669ba3b7ac35e160 + .quad 0x2eccf73fba842056 + .quad 0x1aec1f17c0804f07 + .quad 0x0d96bc031856f4e7 + + // 2^228 * 6 * G + + .quad 0x3318be7775c52d82 + .quad 0x4cb764b554d0aab9 + .quad 0xabcf3d27cc773d91 + .quad 0x3bf4d1848123288a + .quad 0xb1d534b0cc7505e1 + .quad 0x32cd003416c35288 + .quad 0xcb36a5800762c29d + .quad 0x5bfe69b9237a0bf8 + .quad 0x183eab7e78a151ab + .quad 0xbbe990c999093763 + .quad 0xff717d6e4ac7e335 + .quad 0x4c5cddb325f39f88 + + // 2^228 * 7 * G + + .quad 0xc0f6b74d6190a6eb + .quad 0x20ea81a42db8f4e4 + .quad 0xa8bd6f7d97315760 + .quad 0x33b1d60262ac7c21 + .quad 0x57750967e7a9f902 + .quad 0x2c37fdfc4f5b467e + .quad 0xb261663a3177ba46 + .quad 0x3a375e78dc2d532b + .quad 0x8141e72f2d4dddea + .quad 0xe6eafe9862c607c8 + .quad 0x23c28458573cafd0 + .quad 0x46b9476f4ff97346 + + // 2^228 * 8 * G + + .quad 0x0c1ffea44f901e5c + .quad 0x2b0b6fb72184b782 + .quad 0xe587ff910114db88 + .quad 0x37130f364785a142 + .quad 0x1215505c0d58359f + .quad 0x2a2013c7fc28c46b + .quad 0x24a0a1af89ea664e + .quad 0x4400b638a1130e1f + .quad 0x3a01b76496ed19c3 + .quad 0x31e00ab0ed327230 + .quad 0x520a885783ca15b1 + .quad 0x06aab9875accbec7 + + // 2^232 * 1 * G + + .quad 0xc1339983f5df0ebb + .quad 0xc0f3758f512c4cac + .quad 0x2cf1130a0bb398e1 + .quad 0x6b3cecf9aa270c62 + .quad 0x5349acf3512eeaef + .quad 0x20c141d31cc1cb49 + .quad 0x24180c07a99a688d + .quad 0x555ef9d1c64b2d17 + .quad 0x36a770ba3b73bd08 + .quad 0x624aef08a3afbf0c + .quad 0x5737ff98b40946f2 + .quad 0x675f4de13381749d + + // 2^232 * 2 * G + + .quad 0x0e2c52036b1782fc + .quad 0x64816c816cad83b4 + .quad 0xd0dcbdd96964073e + .quad 0x13d99df70164c520 + .quad 0xa12ff6d93bdab31d + .quad 0x0725d80f9d652dfe + .quad 0x019c4ff39abe9487 + .quad 0x60f450b882cd3c43 + .quad 0x014b5ec321e5c0ca + .quad 0x4fcb69c9d719bfa2 + .quad 0x4e5f1c18750023a0 + .quad 0x1c06de9e55edac80 + + // 2^232 * 3 * G + + .quad 0x990f7ad6a33ec4e2 + .quad 0x6608f938be2ee08e + .quad 0x9ca143c563284515 + .quad 0x4cf38a1fec2db60d + .quad 0xffd52b40ff6d69aa + .quad 0x34530b18dc4049bb + .quad 0x5e4a5c2fa34d9897 + .quad 0x78096f8e7d32ba2d + .quad 0xa0aaaa650dfa5ce7 + .quad 0xf9c49e2a48b5478c + .quad 0x4f09cc7d7003725b + .quad 0x373cad3a26091abe + + // 2^232 * 4 * G + + .quad 0xb294634d82c9f57c + .quad 0x1fcbfde124934536 + .quad 0x9e9c4db3418cdb5a + .quad 0x0040f3d9454419fc + .quad 0xf1bea8fb89ddbbad + .quad 0x3bcb2cbc61aeaecb + .quad 0x8f58a7bb1f9b8d9d + .quad 0x21547eda5112a686 + .quad 0xdefde939fd5986d3 + .quad 0xf4272c89510a380c + .quad 0xb72ba407bb3119b9 + .quad 0x63550a334a254df4 + + // 2^232 * 5 * G + + .quad 0x6507d6edb569cf37 + .quad 0x178429b00ca52ee1 + .quad 0xea7c0090eb6bd65d + .quad 0x3eea62c7daf78f51 + .quad 0x9bba584572547b49 + .quad 0xf305c6fae2c408e0 + .quad 0x60e8fa69c734f18d + .quad 0x39a92bafaa7d767a + .quad 0x9d24c713e693274e + .quad 0x5f63857768dbd375 + .quad 0x70525560eb8ab39a + .quad 0x68436a0665c9c4cd + + // 2^232 * 6 * G + + .quad 0xbc0235e8202f3f27 + .quad 0xc75c00e264f975b0 + .quad 0x91a4e9d5a38c2416 + .quad 0x17b6e7f68ab789f9 + .quad 0x1e56d317e820107c + .quad 0xc5266844840ae965 + .quad 0xc1e0a1c6320ffc7a + .quad 0x5373669c91611472 + .quad 0x5d2814ab9a0e5257 + .quad 0x908f2084c9cab3fc + .quad 0xafcaf5885b2d1eca + .quad 0x1cb4b5a678f87d11 + + // 2^232 * 7 * G + + .quad 0xb664c06b394afc6c + .quad 0x0c88de2498da5fb1 + .quad 0x4f8d03164bcad834 + .quad 0x330bca78de7434a2 + .quad 0x6b74aa62a2a007e7 + .quad 0xf311e0b0f071c7b1 + .quad 0x5707e438000be223 + .quad 0x2dc0fd2d82ef6eac + .quad 0x982eff841119744e + .quad 0xf9695e962b074724 + .quad 0xc58ac14fbfc953fb + .quad 0x3c31be1b369f1cf5 + + // 2^232 * 8 * G + + .quad 0xb0f4864d08948aee + .quad 0x07dc19ee91ba1c6f + .quad 0x7975cdaea6aca158 + .quad 0x330b61134262d4bb + .quad 0xc168bc93f9cb4272 + .quad 0xaeb8711fc7cedb98 + .quad 0x7f0e52aa34ac8d7a + .quad 0x41cec1097e7d55bb + .quad 0xf79619d7a26d808a + .quad 0xbb1fd49e1d9e156d + .quad 0x73d7c36cdba1df27 + .quad 0x26b44cd91f28777d + + // 2^236 * 1 * G + + .quad 0x300a9035393aa6d8 + .quad 0x2b501131a12bb1cd + .quad 0x7b1ff677f093c222 + .quad 0x4309c1f8cab82bad + .quad 0xaf44842db0285f37 + .quad 0x8753189047efc8df + .quad 0x9574e091f820979a + .quad 0x0e378d6069615579 + .quad 0xd9fa917183075a55 + .quad 0x4bdb5ad26b009fdc + .quad 0x7829ad2cd63def0e + .quad 0x078fc54975fd3877 + + // 2^236 * 2 * G + + .quad 0x87dfbd1428878f2d + .quad 0x134636dd1e9421a1 + .quad 0x4f17c951257341a3 + .quad 0x5df98d4bad296cb8 + .quad 0xe2004b5bb833a98a + .quad 0x44775dec2d4c3330 + .quad 0x3aa244067eace913 + .quad 0x272630e3d58e00a9 + .quad 0xf3678fd0ecc90b54 + .quad 0xf001459b12043599 + .quad 0x26725fbc3758b89b + .quad 0x4325e4aa73a719ae + + // 2^236 * 3 * G + + .quad 0x657dc6ef433c3493 + .quad 0x65375e9f80dbf8c3 + .quad 0x47fd2d465b372dae + .quad 0x4966ab79796e7947 + .quad 0xed24629acf69f59d + .quad 0x2a4a1ccedd5abbf4 + .quad 0x3535ca1f56b2d67b + .quad 0x5d8c68d043b1b42d + .quad 0xee332d4de3b42b0a + .quad 0xd84e5a2b16a4601c + .quad 0x78243877078ba3e4 + .quad 0x77ed1eb4184ee437 + + // 2^236 * 4 * G + + .quad 0xbfd4e13f201839a0 + .quad 0xaeefffe23e3df161 + .quad 0xb65b04f06b5d1fe3 + .quad 0x52e085fb2b62fbc0 + .quad 0x185d43f89e92ed1a + .quad 0xb04a1eeafe4719c6 + .quad 0x499fbe88a6f03f4f + .quad 0x5d8b0d2f3c859bdd + .quad 0x124079eaa54cf2ba + .quad 0xd72465eb001b26e7 + .quad 0x6843bcfdc97af7fd + .quad 0x0524b42b55eacd02 + + // 2^236 * 5 * G + + .quad 0xfd0d5dbee45447b0 + .quad 0x6cec351a092005ee + .quad 0x99a47844567579cb + .quad 0x59d242a216e7fa45 + .quad 0xbc18dcad9b829eac + .quad 0x23ae7d28b5f579d0 + .quad 0xc346122a69384233 + .quad 0x1a6110b2e7d4ac89 + .quad 0x4f833f6ae66997ac + .quad 0x6849762a361839a4 + .quad 0x6985dec1970ab525 + .quad 0x53045e89dcb1f546 + + // 2^236 * 6 * G + + .quad 0xcb8bb346d75353db + .quad 0xfcfcb24bae511e22 + .quad 0xcba48d40d50ae6ef + .quad 0x26e3bae5f4f7cb5d + .quad 0x84da3cde8d45fe12 + .quad 0xbd42c218e444e2d2 + .quad 0xa85196781f7e3598 + .quad 0x7642c93f5616e2b2 + .quad 0x2323daa74595f8e4 + .quad 0xde688c8b857abeb4 + .quad 0x3fc48e961c59326e + .quad 0x0b2e73ca15c9b8ba + + // 2^236 * 7 * G + + .quad 0xd6bb4428c17f5026 + .quad 0x9eb27223fb5a9ca7 + .quad 0xe37ba5031919c644 + .quad 0x21ce380db59a6602 + .quad 0x0e3fbfaf79c03a55 + .quad 0x3077af054cbb5acf + .quad 0xd5c55245db3de39f + .quad 0x015e68c1476a4af7 + .quad 0xc1d5285220066a38 + .quad 0x95603e523570aef3 + .quad 0x832659a7226b8a4d + .quad 0x5dd689091f8eedc9 + + // 2^236 * 8 * G + + .quad 0xcbac84debfd3c856 + .quad 0x1624c348b35ff244 + .quad 0xb7f88dca5d9cad07 + .quad 0x3b0e574da2c2ebe8 + .quad 0x1d022591a5313084 + .quad 0xca2d4aaed6270872 + .quad 0x86a12b852f0bfd20 + .quad 0x56e6c439ad7da748 + .quad 0xc704ff4942bdbae6 + .quad 0x5e21ade2b2de1f79 + .quad 0xe95db3f35652fad8 + .quad 0x0822b5378f08ebc1 + + // 2^240 * 1 * G + + .quad 0x51f048478f387475 + .quad 0xb25dbcf49cbecb3c + .quad 0x9aab1244d99f2055 + .quad 0x2c709e6c1c10a5d6 + .quad 0xe1b7f29362730383 + .quad 0x4b5279ffebca8a2c + .quad 0xdafc778abfd41314 + .quad 0x7deb10149c72610f + .quad 0xcb62af6a8766ee7a + .quad 0x66cbec045553cd0e + .quad 0x588001380f0be4b5 + .quad 0x08e68e9ff62ce2ea + + // 2^240 * 2 * G + + .quad 0x34ad500a4bc130ad + .quad 0x8d38db493d0bd49c + .quad 0xa25c3d98500a89be + .quad 0x2f1f3f87eeba3b09 + .quad 0x2f2d09d50ab8f2f9 + .quad 0xacb9218dc55923df + .quad 0x4a8f342673766cb9 + .quad 0x4cb13bd738f719f5 + .quad 0xf7848c75e515b64a + .quad 0xa59501badb4a9038 + .quad 0xc20d313f3f751b50 + .quad 0x19a1e353c0ae2ee8 + + // 2^240 * 3 * G + + .quad 0x7d1c7560bafa05c3 + .quad 0xb3e1a0a0c6e55e61 + .quad 0xe3529718c0d66473 + .quad 0x41546b11c20c3486 + .quad 0xb42172cdd596bdbd + .quad 0x93e0454398eefc40 + .quad 0x9fb15347b44109b5 + .quad 0x736bd3990266ae34 + .quad 0x85532d509334b3b4 + .quad 0x46fd114b60816573 + .quad 0xcc5f5f30425c8375 + .quad 0x412295a2b87fab5c + + // 2^240 * 4 * G + + .quad 0x19c99b88f57ed6e9 + .quad 0x5393cb266df8c825 + .quad 0x5cee3213b30ad273 + .quad 0x14e153ebb52d2e34 + .quad 0x2e655261e293eac6 + .quad 0x845a92032133acdb + .quad 0x460975cb7900996b + .quad 0x0760bb8d195add80 + .quad 0x413e1a17cde6818a + .quad 0x57156da9ed69a084 + .quad 0x2cbf268f46caccb1 + .quad 0x6b34be9bc33ac5f2 + + // 2^240 * 5 * G + + .quad 0xf3df2f643a78c0b2 + .quad 0x4c3e971ef22e027c + .quad 0xec7d1c5e49c1b5a3 + .quad 0x2012c18f0922dd2d + .quad 0x11fc69656571f2d3 + .quad 0xc6c9e845530e737a + .quad 0xe33ae7a2d4fe5035 + .quad 0x01b9c7b62e6dd30b + .quad 0x880b55e55ac89d29 + .quad 0x1483241f45a0a763 + .quad 0x3d36efdfc2e76c1f + .quad 0x08af5b784e4bade8 + + // 2^240 * 6 * G + + .quad 0x283499dc881f2533 + .quad 0x9d0525da779323b6 + .quad 0x897addfb673441f4 + .quad 0x32b79d71163a168d + .quad 0xe27314d289cc2c4b + .quad 0x4be4bd11a287178d + .quad 0x18d528d6fa3364ce + .quad 0x6423c1d5afd9826e + .quad 0xcc85f8d9edfcb36a + .quad 0x22bcc28f3746e5f9 + .quad 0xe49de338f9e5d3cd + .quad 0x480a5efbc13e2dcc + + // 2^240 * 7 * G + + .quad 0x0b51e70b01622071 + .quad 0x06b505cf8b1dafc5 + .quad 0x2c6bb061ef5aabcd + .quad 0x47aa27600cb7bf31 + .quad 0xb6614ce442ce221f + .quad 0x6e199dcc4c053928 + .quad 0x663fb4a4dc1cbe03 + .quad 0x24b31d47691c8e06 + .quad 0x2a541eedc015f8c3 + .quad 0x11a4fe7e7c693f7c + .quad 0xf0af66134ea278d6 + .quad 0x545b585d14dda094 + + // 2^240 * 8 * G + + .quad 0x67bf275ea0d43a0f + .quad 0xade68e34089beebe + .quad 0x4289134cd479e72e + .quad 0x0f62f9c332ba5454 + .quad 0x6204e4d0e3b321e1 + .quad 0x3baa637a28ff1e95 + .quad 0x0b0ccffd5b99bd9e + .quad 0x4d22dc3e64c8d071 + .quad 0xfcb46589d63b5f39 + .quad 0x5cae6a3f57cbcf61 + .quad 0xfebac2d2953afa05 + .quad 0x1c0fa01a36371436 + + // 2^244 * 1 * G + + .quad 0xe7547449bc7cd692 + .quad 0x0f9abeaae6f73ddf + .quad 0x4af01ca700837e29 + .quad 0x63ab1b5d3f1bc183 + .quad 0xc11ee5e854c53fae + .quad 0x6a0b06c12b4f3ff4 + .quad 0x33540f80e0b67a72 + .quad 0x15f18fc3cd07e3ef + .quad 0x32750763b028f48c + .quad 0x06020740556a065f + .quad 0xd53bd812c3495b58 + .quad 0x08706c9b865f508d + + // 2^244 * 2 * G + + .quad 0xf37ca2ab3d343dff + .quad 0x1a8c6a2d80abc617 + .quad 0x8e49e035d4ccffca + .quad 0x48b46beebaa1d1b9 + .quad 0xcc991b4138b41246 + .quad 0x243b9c526f9ac26b + .quad 0xb9ef494db7cbabbd + .quad 0x5fba433dd082ed00 + .quad 0x9c49e355c9941ad0 + .quad 0xb9734ade74498f84 + .quad 0x41c3fed066663e5c + .quad 0x0ecfedf8e8e710b3 + + // 2^244 * 3 * G + + .quad 0x76430f9f9cd470d9 + .quad 0xb62acc9ba42f6008 + .quad 0x1898297c59adad5e + .quad 0x7789dd2db78c5080 + .quad 0x744f7463e9403762 + .quad 0xf79a8dee8dfcc9c9 + .quad 0x163a649655e4cde3 + .quad 0x3b61788db284f435 + .quad 0xb22228190d6ef6b2 + .quad 0xa94a66b246ce4bfa + .quad 0x46c1a77a4f0b6cc7 + .quad 0x4236ccffeb7338cf + + // 2^244 * 4 * G + + .quad 0x8497404d0d55e274 + .quad 0x6c6663d9c4ad2b53 + .quad 0xec2fb0d9ada95734 + .quad 0x2617e120cdb8f73c + .quad 0x3bd82dbfda777df6 + .quad 0x71b177cc0b98369e + .quad 0x1d0e8463850c3699 + .quad 0x5a71945b48e2d1f1 + .quad 0x6f203dd5405b4b42 + .quad 0x327ec60410b24509 + .quad 0x9c347230ac2a8846 + .quad 0x77de29fc11ffeb6a + + // 2^244 * 5 * G + + .quad 0xb0ac57c983b778a8 + .quad 0x53cdcca9d7fe912c + .quad 0x61c2b854ff1f59dc + .quad 0x3a1a2cf0f0de7dac + .quad 0x835e138fecced2ca + .quad 0x8c9eaf13ea963b9a + .quad 0xc95fbfc0b2160ea6 + .quad 0x575e66f3ad877892 + .quad 0x99803a27c88fcb3a + .quad 0x345a6789275ec0b0 + .quad 0x459789d0ff6c2be5 + .quad 0x62f882651e70a8b2 + + // 2^244 * 6 * G + + .quad 0x085ae2c759ff1be4 + .quad 0x149145c93b0e40b7 + .quad 0xc467e7fa7ff27379 + .quad 0x4eeecf0ad5c73a95 + .quad 0x6d822986698a19e0 + .quad 0xdc9821e174d78a71 + .quad 0x41a85f31f6cb1f47 + .quad 0x352721c2bcda9c51 + .quad 0x48329952213fc985 + .quad 0x1087cf0d368a1746 + .quad 0x8e5261b166c15aa5 + .quad 0x2d5b2d842ed24c21 + + // 2^244 * 7 * G + + .quad 0x02cfebd9ebd3ded1 + .quad 0xd45b217739021974 + .quad 0x7576f813fe30a1b7 + .quad 0x5691b6f9a34ef6c2 + .quad 0x5eb7d13d196ac533 + .quad 0x377234ecdb80be2b + .quad 0xe144cffc7cf5ae24 + .quad 0x5226bcf9c441acec + .quad 0x79ee6c7223e5b547 + .quad 0x6f5f50768330d679 + .quad 0xed73e1e96d8adce9 + .quad 0x27c3da1e1d8ccc03 + + // 2^244 * 8 * G + + .quad 0x7eb9efb23fe24c74 + .quad 0x3e50f49f1651be01 + .quad 0x3ea732dc21858dea + .quad 0x17377bd75bb810f9 + .quad 0x28302e71630ef9f6 + .quad 0xc2d4a2032b64cee0 + .quad 0x090820304b6292be + .quad 0x5fca747aa82adf18 + .quad 0x232a03c35c258ea5 + .quad 0x86f23a2c6bcb0cf1 + .quad 0x3dad8d0d2e442166 + .quad 0x04a8933cab76862b + + // 2^248 * 1 * G + + .quad 0xd2c604b622943dff + .quad 0xbc8cbece44cfb3a0 + .quad 0x5d254ff397808678 + .quad 0x0fa3614f3b1ca6bf + .quad 0x69082b0e8c936a50 + .quad 0xf9c9a035c1dac5b6 + .quad 0x6fb73e54c4dfb634 + .quad 0x4005419b1d2bc140 + .quad 0xa003febdb9be82f0 + .quad 0x2089c1af3a44ac90 + .quad 0xf8499f911954fa8e + .quad 0x1fba218aef40ab42 + + // 2^248 * 2 * G + + .quad 0xab549448fac8f53e + .quad 0x81f6e89a7ba63741 + .quad 0x74fd6c7d6c2b5e01 + .quad 0x392e3acaa8c86e42 + .quad 0x4f3e57043e7b0194 + .quad 0xa81d3eee08daaf7f + .quad 0xc839c6ab99dcdef1 + .quad 0x6c535d13ff7761d5 + .quad 0x4cbd34e93e8a35af + .quad 0x2e0781445887e816 + .quad 0x19319c76f29ab0ab + .quad 0x25e17fe4d50ac13b + + // 2^248 * 3 * G + + .quad 0x0a289bd71e04f676 + .quad 0x208e1c52d6420f95 + .quad 0x5186d8b034691fab + .quad 0x255751442a9fb351 + .quad 0x915f7ff576f121a7 + .quad 0xc34a32272fcd87e3 + .quad 0xccba2fde4d1be526 + .quad 0x6bba828f8969899b + .quad 0xe2d1bc6690fe3901 + .quad 0x4cb54a18a0997ad5 + .quad 0x971d6914af8460d4 + .quad 0x559d504f7f6b7be4 + + // 2^248 * 4 * G + + .quad 0xa7738378b3eb54d5 + .quad 0x1d69d366a5553c7c + .quad 0x0a26cf62f92800ba + .quad 0x01ab12d5807e3217 + .quad 0x9c4891e7f6d266fd + .quad 0x0744a19b0307781b + .quad 0x88388f1d6061e23b + .quad 0x123ea6a3354bd50e + .quad 0x118d189041e32d96 + .quad 0xb9ede3c2d8315848 + .quad 0x1eab4271d83245d9 + .quad 0x4a3961e2c918a154 + + // 2^248 * 5 * G + + .quad 0x71dc3be0f8e6bba0 + .quad 0xd6cef8347effe30a + .quad 0xa992425fe13a476a + .quad 0x2cd6bce3fb1db763 + .quad 0x0327d644f3233f1e + .quad 0x499a260e34fcf016 + .quad 0x83b5a716f2dab979 + .quad 0x68aceead9bd4111f + .quad 0x38b4c90ef3d7c210 + .quad 0x308e6e24b7ad040c + .quad 0x3860d9f1b7e73e23 + .quad 0x595760d5b508f597 + + // 2^248 * 6 * G + + .quad 0x6129bfe104aa6397 + .quad 0x8f960008a4a7fccb + .quad 0x3f8bc0897d909458 + .quad 0x709fa43edcb291a9 + .quad 0x882acbebfd022790 + .quad 0x89af3305c4115760 + .quad 0x65f492e37d3473f4 + .quad 0x2cb2c5df54515a2b + .quad 0xeb0a5d8c63fd2aca + .quad 0xd22bc1662e694eff + .quad 0x2723f36ef8cbb03a + .quad 0x70f029ecf0c8131f + + // 2^248 * 7 * G + + .quad 0x461307b32eed3e33 + .quad 0xae042f33a45581e7 + .quad 0xc94449d3195f0366 + .quad 0x0b7d5d8a6c314858 + .quad 0x2a6aafaa5e10b0b9 + .quad 0x78f0a370ef041aa9 + .quad 0x773efb77aa3ad61f + .quad 0x44eca5a2a74bd9e1 + .quad 0x25d448327b95d543 + .quad 0x70d38300a3340f1d + .quad 0xde1c531c60e1c52b + .quad 0x272224512c7de9e4 + + // 2^248 * 8 * G + + .quad 0x1abc92af49c5342e + .quad 0xffeed811b2e6fad0 + .quad 0xefa28c8dfcc84e29 + .quad 0x11b5df18a44cc543 + .quad 0xbf7bbb8a42a975fc + .quad 0x8c5c397796ada358 + .quad 0xe27fc76fcdedaa48 + .quad 0x19735fd7f6bc20a6 + .quad 0xe3ab90d042c84266 + .quad 0xeb848e0f7f19547e + .quad 0x2503a1d065a497b9 + .quad 0x0fef911191df895f diff --git a/cbits/s2n/x86_att/p256_montjadd.S b/cbits/s2n/x86_att/p256_montjadd.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/p256_montjadd.S @@ -0,0 +1,593 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Point addition on NIST curve P-256 in Montgomery-Jacobian coordinates +// +// extern void p256_montjadd(uint64_t p3[static 12], const uint64_t p1[static 12], +// const uint64_t p2[static 12]); +// +// Does p3 := p1 + p2 where all points are regarded as Jacobian triples with +// each coordinate in the Montgomery domain, i.e. x' = (2^256 * x) mod p_256. +// A Jacobian triple (x',y',z') represents affine point (x/z^2,y/z^3). +// +// Standard x86-64 ABI: RDI = p3, RSI = p1, RDX = p2 +// Microsoft x64 ABI: RCX = p3, RDX = p1, R8 = p2 +// ---------------------------------------------------------------------------- +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_montjadd) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_montjadd) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_montjadd) + .text + +// Size of individual field elements + +#define NUMSIZE 32 + +// Pointer-offset pairs for inputs and outputs +// These assume %rdi = p3, %rsi = p1 and %rbp = p2, +// which needs to be set up explicitly before use. +// The first two hold initially, and the second is +// set up by copying the initial %rdx input to %rbp. +// Thereafter, no code macro modifies any of them. + +#define x_1 0(%rsi) +#define y_1 NUMSIZE(%rsi) +#define z_1 (2*NUMSIZE)(%rsi) + +#define x_2 0(%rbp) +#define y_2 NUMSIZE(%rbp) +#define z_2 (2*NUMSIZE)(%rbp) + +#define x_3 0(%rdi) +#define y_3 NUMSIZE(%rdi) +#define z_3 (2*NUMSIZE)(%rdi) + +// Pointer-offset pairs for temporaries, with some aliasing +// NSPACE is the total stack needed for these temporaries + +#define z1sq (NUMSIZE*0)(%rsp) +#define ww (NUMSIZE*0)(%rsp) +#define resx (NUMSIZE*0)(%rsp) + +#define yd (NUMSIZE*1)(%rsp) +#define y2a (NUMSIZE*1)(%rsp) + +#define x2a (NUMSIZE*2)(%rsp) +#define zzx2 (NUMSIZE*2)(%rsp) + +#define zz (NUMSIZE*3)(%rsp) +#define t1 (NUMSIZE*3)(%rsp) + +#define t2 (NUMSIZE*4)(%rsp) +#define x1a (NUMSIZE*4)(%rsp) +#define zzx1 (NUMSIZE*4)(%rsp) +#define resy (NUMSIZE*4)(%rsp) + +#define xd (NUMSIZE*5)(%rsp) +#define z2sq (NUMSIZE*5)(%rsp) +#define resz (NUMSIZE*5)(%rsp) + +#define y1a (NUMSIZE*6)(%rsp) + +#define NSPACE NUMSIZE*7 + +// Corresponds exactly to bignum_montmul_p256 + +#define montmul_p256(P0,P1,P2) \ + xorl %r13d, %r13d ; \ + movq P2, %rdx ; \ + mulxq P1, %r8, %r9 ; \ + mulxq 0x8+P1, %rbx, %r10 ; \ + adcq %rbx, %r9 ; \ + mulxq 0x10+P1, %rbx, %r11 ; \ + adcq %rbx, %r10 ; \ + mulxq 0x18+P1, %rbx, %r12 ; \ + adcq %rbx, %r11 ; \ + adcq %r13, %r12 ; \ + movq 0x8+P2, %rdx ; \ + xorl %r14d, %r14d ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x18+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcq %r14, %r13 ; \ + xorl %r15d, %r15d ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + notq %rdx; \ + leaq 0x2(%rdx), %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcxq %r15, %r13 ; \ + adoxq %r15, %r14 ; \ + adcq %r15, %r14 ; \ + movq 0x10+P2, %rdx ; \ + xorl %r8d, %r8d ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adoxq %r8, %r14 ; \ + mulxq 0x18+P1, %rax, %rbx ; \ + adcq %rax, %r13 ; \ + adcq %rbx, %r14 ; \ + adcq %r8, %r15 ; \ + movq 0x18+P2, %rdx ; \ + xorl %r9d, %r9d ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + adoxq %r9, %r15 ; \ + mulxq 0x18+P1, %rax, %rbx ; \ + adcq %rax, %r14 ; \ + adcq %rbx, %r15 ; \ + adcq %r9, %r8 ; \ + xorl %r9d, %r9d ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + notq %rdx; \ + leaq 0x2(%rdx), %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r14 ; \ + adoxq %rbx, %r15 ; \ + adcxq %r9, %r15 ; \ + adoxq %r9, %r8 ; \ + adcq %r9, %r8 ; \ + movl $0x1, %ecx ; \ + addq %r12, %rcx ; \ + decq %rdx; \ + adcq %r13, %rdx ; \ + decq %r9; \ + movq %r9, %rax ; \ + adcq %r14, %r9 ; \ + movl $0xfffffffe, %r11d ; \ + adcq %r15, %r11 ; \ + adcq %r8, %rax ; \ + cmovbq %rcx, %r12 ; \ + cmovbq %rdx, %r13 ; \ + cmovbq %r9, %r14 ; \ + cmovbq %r11, %r15 ; \ + movq %r12, P0 ; \ + movq %r13, 0x8+P0 ; \ + movq %r14, 0x10+P0 ; \ + movq %r15, 0x18+P0 + +// Corresponds exactly to bignum_montsqr_p256 except for +// register tweaks to avoid modifying %rbp. + +#define montsqr_p256(P0,P1) \ + movq P1, %rdx ; \ + mulxq %rdx, %r8, %r15 ; \ + mulxq 0x8+P1, %r9, %r10 ; \ + mulxq 0x18+P1, %r11, %r12 ; \ + movq 0x10+P1, %rdx ; \ + mulxq 0x18+P1, %r13, %r14 ; \ + xorl %ecx, %ecx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + movq 0x18+P1, %rdx ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcxq %rcx, %r13 ; \ + adoxq %rcx, %r14 ; \ + adcq %rcx, %r14 ; \ + xorl %ecx, %ecx ; \ + adcxq %r9, %r9 ; \ + adoxq %r15, %r9 ; \ + movq 0x8+P1, %rdx ; \ + mulxq %rdx, %rax, %rdx ; \ + adcxq %r10, %r10 ; \ + adoxq %rax, %r10 ; \ + adcxq %r11, %r11 ; \ + adoxq %rdx, %r11 ; \ + movq 0x10+P1, %rdx ; \ + mulxq %rdx, %rax, %rdx ; \ + adcxq %r12, %r12 ; \ + adoxq %rax, %r12 ; \ + adcxq %r13, %r13 ; \ + adoxq %rdx, %r13 ; \ + movq 0x18+P1, %rdx ; \ + mulxq %rdx, %rax, %r15 ; \ + adcxq %r14, %r14 ; \ + adoxq %rax, %r14 ; \ + adcxq %rcx, %r15 ; \ + adoxq %rcx, %r15 ; \ + xorl %ecx, %ecx ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + movabsq $0xffffffff00000001, %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcxq %rcx, %r13 ; \ + movl %ecx, %r9d ; \ + adoxq %rcx, %r9 ; \ + adcxq %rcx, %r9 ; \ + addq %r9, %r14 ; \ + adcq %rcx, %r15 ; \ + movl %ecx, %r8d ; \ + adcq %rcx, %r8 ; \ + xorl %ecx, %ecx ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + movabsq $0xffffffff00000001, %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r14 ; \ + adoxq %rbx, %r15 ; \ + adcxq %rcx, %r15 ; \ + adoxq %rcx, %r8 ; \ + adcq %rcx, %r8 ; \ + movl $0x1, %ebx ; \ + addq %r12, %rbx ; \ + leaq -0x1(%rdx), %rdx ; \ + adcq %r13, %rdx ; \ + leaq -0x1(%rcx), %rcx ; \ + movq %rcx, %rax ; \ + adcq %r14, %rcx ; \ + movl $0xfffffffe, %r11d ; \ + adcq %r15, %r11 ; \ + adcq %r8, %rax ; \ + cmovbq %rbx, %r12 ; \ + cmovbq %rdx, %r13 ; \ + cmovbq %rcx, %r14 ; \ + cmovbq %r11, %r15 ; \ + movq %r12, P0 ; \ + movq %r13, 0x8+P0 ; \ + movq %r14, 0x10+P0 ; \ + movq %r15, 0x18+P0 + +// Almost-Montgomery variant which we use when an input to other muls +// with the other argument fully reduced (which is always safe). +// Again, the basic squaring code is tweaked to avoid modifying %rbp. + +#define amontsqr_p256(P0,P1) \ + movq P1, %rdx ; \ + mulxq %rdx, %r8, %r15 ; \ + mulxq 0x8+P1, %r9, %r10 ; \ + mulxq 0x18+P1, %r11, %r12 ; \ + movq 0x10+P1, %rdx ; \ + mulxq 0x18+P1, %r13, %r14 ; \ + xorl %ecx, %ecx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + movq 0x18+P1, %rdx ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcxq %rcx, %r13 ; \ + adoxq %rcx, %r14 ; \ + adcq %rcx, %r14 ; \ + xorl %ecx, %ecx ; \ + adcxq %r9, %r9 ; \ + adoxq %r15, %r9 ; \ + movq 0x8+P1, %rdx ; \ + mulxq %rdx, %rax, %rdx ; \ + adcxq %r10, %r10 ; \ + adoxq %rax, %r10 ; \ + adcxq %r11, %r11 ; \ + adoxq %rdx, %r11 ; \ + movq 0x10+P1, %rdx ; \ + mulxq %rdx, %rax, %rdx ; \ + adcxq %r12, %r12 ; \ + adoxq %rax, %r12 ; \ + adcxq %r13, %r13 ; \ + adoxq %rdx, %r13 ; \ + movq 0x18+P1, %rdx ; \ + mulxq %rdx, %rax, %r15 ; \ + adcxq %r14, %r14 ; \ + adoxq %rax, %r14 ; \ + adcxq %rcx, %r15 ; \ + adoxq %rcx, %r15 ; \ + xorl %ecx, %ecx ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + movabsq $0xffffffff00000001, %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcxq %rcx, %r13 ; \ + movl %ecx, %r9d ; \ + adoxq %rcx, %r9 ; \ + adcxq %rcx, %r9 ; \ + addq %r9, %r14 ; \ + adcq %rcx, %r15 ; \ + movl %ecx, %r8d ; \ + adcq %rcx, %r8 ; \ + xorl %ecx, %ecx ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + movabsq $0xffffffff00000001, %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r14 ; \ + adoxq %rbx, %r15 ; \ + adcxq %rcx, %r15 ; \ + adoxq %rcx, %r8 ; \ + adcq %rcx, %r8 ; \ + movl $0x1, %r8d ; \ + leaq -0x1(%rdx), %rdx ; \ + leaq -0x1(%rcx), %rax ; \ + movl $0xfffffffe, %r11d ; \ + cmovzq %rcx, %r8 ; \ + cmovzq %rcx, %rdx ; \ + cmovzq %rcx, %rax ; \ + cmovzq %rcx, %r11 ; \ + addq %r8, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq %rax, %r14 ; \ + adcq %r11, %r15 ; \ + movq %r12, P0 ; \ + movq %r13, 0x8+P0 ; \ + movq %r14, 0x10+P0 ; \ + movq %r15, 0x18+P0 + +// Corresponds exactly to bignum_sub_p256 + +#define sub_p256(P0,P1,P2) \ + movq P1, %rax ; \ + subq P2, %rax ; \ + movq 0x8+P1, %rcx ; \ + sbbq 0x8+P2, %rcx ; \ + movq 0x10+P1, %r8 ; \ + sbbq 0x10+P2, %r8 ; \ + movq 0x18+P1, %r9 ; \ + sbbq 0x18+P2, %r9 ; \ + movl $0xffffffff, %r10d ; \ + sbbq %r11, %r11 ; \ + xorq %rdx, %rdx ; \ + andq %r11, %r10 ; \ + subq %r10, %rdx ; \ + addq %r11, %rax ; \ + movq %rax, P0 ; \ + adcq %r10, %rcx ; \ + movq %rcx, 0x8+P0 ; \ + adcq $0x0, %r8 ; \ + movq %r8, 0x10+P0 ; \ + adcq %rdx, %r9 ; \ + movq %r9, 0x18+P0 + +// Additional macros to help with final multiplexing + +#define load4(r0,r1,r2,r3,P) \ + movq P, r0 ; \ + movq 8+P, r1 ; \ + movq 16+P, r2 ; \ + movq 24+P, r3 + +#define store4(P,r0,r1,r2,r3) \ + movq r0, P ; \ + movq r1, 8+P ; \ + movq r2, 16+P ; \ + movq r3, 24+P + +#define czload4(r0,r1,r2,r3,P) \ + cmovzq P, r0 ; \ + cmovzq 8+P, r1 ; \ + cmovzq 16+P, r2 ; \ + cmovzq 24+P, r3 + +#define muxload4(r0,r1,r2,r3,P0,P1,P2) \ + movq P0, r0 ; \ + cmovbq P1, r0 ; \ + cmovnbe P2, r0 ; \ + movq 8+P0, r1 ; \ + cmovbq 8+P1, r1 ; \ + cmovnbe 8+P2, r1 ; \ + movq 16+P0, r2 ; \ + cmovbq 16+P1, r2 ; \ + cmovnbe 16+P2, r2 ; \ + movq 24+P0, r3 ; \ + cmovbq 24+P1, r3 ; \ + cmovnbe 24+P2, r3 + +S2N_BN_SYMBOL(p256_montjadd): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx +#endif + +// Save registers and make room on stack for temporary variables +// Put the input y in %rbp where it lasts as long as it's needed. + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + + CFI_DEC_RSP(NSPACE) + + movq %rdx, %rbp + +// Main code, just a sequence of basic field operations +// 12 * multiply + 4 * square + 7 * subtract + + amontsqr_p256(z1sq,z_1) + amontsqr_p256(z2sq,z_2) + + montmul_p256(y1a,z_2,y_1) + montmul_p256(y2a,z_1,y_2) + + montmul_p256(x2a,z1sq,x_2) + montmul_p256(x1a,z2sq,x_1) + montmul_p256(y2a,z1sq,y2a) + montmul_p256(y1a,z2sq,y1a) + + sub_p256(xd,x2a,x1a) + sub_p256(yd,y2a,y1a) + + amontsqr_p256(zz,xd) + montsqr_p256(ww,yd) + + montmul_p256(zzx1,zz,x1a) + montmul_p256(zzx2,zz,x2a) + + sub_p256(resx,ww,zzx1) + sub_p256(t1,zzx2,zzx1) + + montmul_p256(xd,xd,z_1) + + sub_p256(resx,resx,zzx2) + + sub_p256(t2,zzx1,resx) + + montmul_p256(t1,t1,y1a) + + montmul_p256(resz,xd,z_2) + montmul_p256(t2,yd,t2) + + sub_p256(resy,t2,t1) + +// Load in the z coordinates of the inputs to check for P1 = 0 and P2 = 0 +// The condition codes get set by a comparison (P2 != 0) - (P1 != 0) +// So "NBE" <=> ~(CF \/ ZF) <=> P1 = 0 /\ ~(P2 = 0) +// and "B" <=> CF <=> ~(P1 = 0) /\ P2 = 0 +// and "Z" <=> ZF <=> (P1 = 0 <=> P2 = 0) + + load4(%r8,%r9,%r10,%r11,z_1) + + movq %r8, %rax + movq %r9, %rdx + orq %r10, %rax + orq %r11, %rdx + orq %rdx, %rax + negq %rax + sbbq %rax, %rax + + load4(%r12,%r13,%r14,%r15,z_2) + + movq %r12, %rbx + movq %r13, %rdx + orq %r14, %rbx + orq %r15, %rdx + orq %rdx, %rbx + negq %rbx + sbbq %rbx, %rbx + + cmpq %rax, %rbx + +// Multiplex the outputs accordingly, re-using the z's in registers + + cmovbq %r8, %r12 + cmovbq %r9, %r13 + cmovbq %r10, %r14 + cmovbq %r11, %r15 + + czload4(%r12,%r13,%r14,%r15,resz) + + muxload4(%rax,%rbx,%rcx,%rdx,resx,x_1,x_2) + muxload4(%r8,%r9,%r10,%r11,resy,y_1,y_2) + +// Finally store back the multiplexed values + + store4(x_3,%rax,%rbx,%rcx,%rdx) + store4(y_3,%r8,%r9,%r10,%r11) + store4(z_3,%r12,%r13,%r14,%r15) + +// Restore stack and registers + + CFI_INC_RSP(NSPACE) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_montjadd) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/p256_montjadd_alt.S b/cbits/s2n/x86_att/p256_montjadd_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/p256_montjadd_alt.S @@ -0,0 +1,579 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Point addition on NIST curve P-256 in Montgomery-Jacobian coordinates +// +// extern void p256_montjadd_alt(uint64_t p3[static 12], +// const uint64_t p1[static 12], +// const uint64_t p2[static 12]); +// +// Does p3 := p1 + p2 where all points are regarded as Jacobian triples with +// each coordinate in the Montgomery domain, i.e. x' = (2^256 * x) mod p_256. +// A Jacobian triple (x',y',z') represents affine point (x/z^2,y/z^3). +// +// Standard x86-64 ABI: RDI = p3, RSI = p1, RDX = p2 +// Microsoft x64 ABI: RCX = p3, RDX = p1, R8 = p2 +// ---------------------------------------------------------------------------- +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_montjadd_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_montjadd_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_montjadd_alt) + .text + +// Size of individual field elements + +#define NUMSIZE 32 + +// Pointer-offset pairs for inputs and outputs +// These assume %rdi = p3, %rsi = p1 and %rbp = p2, +// which needs to be set up explicitly before use. +// The first two hold initially, and the second is +// set up by copying the initial %rdx input to %rbp. +// Thereafter, no code macro modifies any of them. + +#define x_1 0(%rsi) +#define y_1 NUMSIZE(%rsi) +#define z_1 (2*NUMSIZE)(%rsi) + +#define x_2 0(%rbp) +#define y_2 NUMSIZE(%rbp) +#define z_2 (2*NUMSIZE)(%rbp) + +#define x_3 0(%rdi) +#define y_3 NUMSIZE(%rdi) +#define z_3 (2*NUMSIZE)(%rdi) + +// Pointer-offset pairs for temporaries, with some aliasing +// NSPACE is the total stack needed for these temporaries + +#define z1sq (NUMSIZE*0)(%rsp) +#define ww (NUMSIZE*0)(%rsp) +#define resx (NUMSIZE*0)(%rsp) + +#define yd (NUMSIZE*1)(%rsp) +#define y2a (NUMSIZE*1)(%rsp) + +#define x2a (NUMSIZE*2)(%rsp) +#define zzx2 (NUMSIZE*2)(%rsp) + +#define zz (NUMSIZE*3)(%rsp) +#define t1 (NUMSIZE*3)(%rsp) + +#define t2 (NUMSIZE*4)(%rsp) +#define x1a (NUMSIZE*4)(%rsp) +#define zzx1 (NUMSIZE*4)(%rsp) +#define resy (NUMSIZE*4)(%rsp) + +#define xd (NUMSIZE*5)(%rsp) +#define z2sq (NUMSIZE*5)(%rsp) +#define resz (NUMSIZE*5)(%rsp) + +#define y1a (NUMSIZE*6)(%rsp) + +#define NSPACE NUMSIZE*7 + +// Corresponds exactly to bignum_montmul_p256_alt + +#define montmul_p256(P0,P1,P2) \ + movq P2, %rbx ; \ + movq P1, %rax ; \ + mulq %rbx; \ + movq %rax, %r8 ; \ + movq %rdx, %r9 ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + xorl %r10d, %r10d ; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + movq 0x10+P1, %rax ; \ + mulq %rbx; \ + xorl %r11d, %r11d ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + movq 0x18+P1, %rax ; \ + mulq %rbx; \ + xorl %r12d, %r12d ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + movq 0x8+P2, %rbx ; \ + xorl %r13d, %r13d ; \ + movq P1, %rax ; \ + mulq %rbx; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %r14, %r14 ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + subq %r14, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %r14, %r14 ; \ + movq 0x10+P1, %rax ; \ + mulq %rbx; \ + subq %r14, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %r14, %r14 ; \ + movq 0x18+P1, %rax ; \ + mulq %rbx; \ + subq %r14, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + xorl %r14d, %r14d ; \ + movq $0x100000000, %rbx ; \ + movq %r8, %rax ; \ + mulq %rbx; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %r15, %r15 ; \ + movq %r9, %rax ; \ + mulq %rbx; \ + subq %r15, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %r15, %r15 ; \ + notq %rbx; \ + leaq 0x2(%rbx), %rbx ; \ + movq %r8, %rax ; \ + mulq %rbx; \ + subq %r15, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %r15, %r15 ; \ + movq %r9, %rax ; \ + mulq %rbx; \ + subq %r15, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq %r14, %r14 ; \ + movq 0x10+P2, %rbx ; \ + xorl %r15d, %r15d ; \ + movq P1, %rax ; \ + mulq %rbx; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %r8, %r8 ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + subq %r8, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %r8, %r8 ; \ + movq 0x10+P1, %rax ; \ + mulq %rbx; \ + subq %r8, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + sbbq %r8, %r8 ; \ + movq 0x18+P1, %rax ; \ + mulq %rbx; \ + subq %r8, %rdx ; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq %r15, %r15 ; \ + movq 0x18+P2, %rbx ; \ + xorl %r8d, %r8d ; \ + movq P1, %rax ; \ + mulq %rbx; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %r9, %r9 ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + subq %r9, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + sbbq %r9, %r9 ; \ + movq 0x10+P1, %rax ; \ + mulq %rbx; \ + subq %r9, %rdx ; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + sbbq %r9, %r9 ; \ + movq 0x18+P1, %rax ; \ + mulq %rbx; \ + subq %r9, %rdx ; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + adcq %r8, %r8 ; \ + xorl %r9d, %r9d ; \ + movq $0x100000000, %rbx ; \ + movq %r10, %rax ; \ + mulq %rbx; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %rcx, %rcx ; \ + movq %r11, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + sbbq %rcx, %rcx ; \ + notq %rbx; \ + leaq 0x2(%rbx), %rbx ; \ + movq %r10, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + sbbq %rcx, %rcx ; \ + movq %r11, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + adcq %r9, %r8 ; \ + movl $0x1, %ecx ; \ + addq %r12, %rcx ; \ + decq %rbx; \ + adcq %r13, %rbx ; \ + decq %r9; \ + movq %r9, %rax ; \ + adcq %r14, %r9 ; \ + movl $0xfffffffe, %r11d ; \ + adcq %r15, %r11 ; \ + adcq %r8, %rax ; \ + cmovbq %rcx, %r12 ; \ + cmovbq %rbx, %r13 ; \ + cmovbq %r9, %r14 ; \ + cmovbq %r11, %r15 ; \ + movq %r12, P0 ; \ + movq %r13, 0x8+P0 ; \ + movq %r14, 0x10+P0 ; \ + movq %r15, 0x18+P0 + +// Corresponds exactly to bignum_montsqr_p256_alt + +#define montsqr_p256(P0,P1) \ + movq P1, %rax ; \ + movq %rax, %rbx ; \ + mulq %rax; \ + movq %rax, %r8 ; \ + movq %rdx, %r15 ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + movq %rax, %r9 ; \ + movq %rdx, %r10 ; \ + movq 0x18+P1, %rax ; \ + movq %rax, %r13 ; \ + mulq %rbx; \ + movq %rax, %r11 ; \ + movq %rdx, %r12 ; \ + movq 0x10+P1, %rax ; \ + movq %rax, %rbx ; \ + mulq %r13; \ + movq %rax, %r13 ; \ + movq %rdx, %r14 ; \ + movq P1, %rax ; \ + mulq %rbx; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rcx, %rcx ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %rcx, %rcx ; \ + movq 0x18+P1, %rbx ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + xorl %ecx, %ecx ; \ + addq %r9, %r9 ; \ + adcq %r10, %r10 ; \ + adcq %r11, %r11 ; \ + adcq %r12, %r12 ; \ + adcq %r13, %r13 ; \ + adcq %r14, %r14 ; \ + adcq %rcx, %rcx ; \ + movq 0x8+P1, %rax ; \ + mulq %rax; \ + addq %r15, %r9 ; \ + adcq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %r15, %r15 ; \ + movq 0x10+P1, %rax ; \ + mulq %rax; \ + negq %r15; \ + adcq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + sbbq %r15, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq %rax; \ + negq %r15; \ + adcq %rax, %r14 ; \ + adcq %rcx, %rdx ; \ + movq %rdx, %r15 ; \ + movq $0x100000000, %rbx ; \ + movq %r8, %rax ; \ + mulq %rbx; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %rcx, %rcx ; \ + movq %r9, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rcx, %rcx ; \ + notq %rbx; \ + leaq 0x2(%rbx), %rbx ; \ + movq %r8, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %rcx, %rcx ; \ + xorl %r8d, %r8d ; \ + movq %r9, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq %r8, %r14 ; \ + adcq %r8, %r15 ; \ + adcq %r8, %r8 ; \ + movq $0x100000000, %rbx ; \ + movq %r10, %rax ; \ + mulq %rbx; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %rcx, %rcx ; \ + movq %r11, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + sbbq %rcx, %rcx ; \ + notq %rbx; \ + leaq 0x2(%rbx), %rbx ; \ + movq %r10, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + sbbq %rcx, %rcx ; \ + xorl %r9d, %r9d ; \ + movq %r11, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + adcq %r9, %r8 ; \ + movl $0x1, %ecx ; \ + addq %r12, %rcx ; \ + leaq -0x1(%rbx), %rbx ; \ + adcq %r13, %rbx ; \ + leaq -0x1(%r9), %r9 ; \ + movq %r9, %rax ; \ + adcq %r14, %r9 ; \ + movl $0xfffffffe, %r11d ; \ + adcq %r15, %r11 ; \ + adcq %r8, %rax ; \ + cmovbq %rcx, %r12 ; \ + cmovbq %rbx, %r13 ; \ + cmovbq %r9, %r14 ; \ + cmovbq %r11, %r15 ; \ + movq %r12, P0 ; \ + movq %r13, 0x8+P0 ; \ + movq %r14, 0x10+P0 ; \ + movq %r15, 0x18+P0 + +// Corresponds exactly to bignum_sub_p256 + +#define sub_p256(P0,P1,P2) \ + movq P1, %rax ; \ + subq P2, %rax ; \ + movq 0x8+P1, %rcx ; \ + sbbq 0x8+P2, %rcx ; \ + movq 0x10+P1, %r8 ; \ + sbbq 0x10+P2, %r8 ; \ + movq 0x18+P1, %r9 ; \ + sbbq 0x18+P2, %r9 ; \ + movl $0xffffffff, %r10d ; \ + sbbq %r11, %r11 ; \ + xorq %rdx, %rdx ; \ + andq %r11, %r10 ; \ + subq %r10, %rdx ; \ + addq %r11, %rax ; \ + movq %rax, P0 ; \ + adcq %r10, %rcx ; \ + movq %rcx, 0x8+P0 ; \ + adcq $0x0, %r8 ; \ + movq %r8, 0x10+P0 ; \ + adcq %rdx, %r9 ; \ + movq %r9, 0x18+P0 + +// Additional macros to help with final multiplexing + +#define load4(r0,r1,r2,r3,P) \ + movq P, r0 ; \ + movq 8+P, r1 ; \ + movq 16+P, r2 ; \ + movq 24+P, r3 + +#define store4(P,r0,r1,r2,r3) \ + movq r0, P ; \ + movq r1, 8+P ; \ + movq r2, 16+P ; \ + movq r3, 24+P + +#define czload4(r0,r1,r2,r3,P) \ + cmovzq P, r0 ; \ + cmovzq 8+P, r1 ; \ + cmovzq 16+P, r2 ; \ + cmovzq 24+P, r3 + +#define muxload4(r0,r1,r2,r3,P0,P1,P2) \ + movq P0, r0 ; \ + cmovbq P1, r0 ; \ + cmovnbe P2, r0 ; \ + movq 8+P0, r1 ; \ + cmovbq 8+P1, r1 ; \ + cmovnbe 8+P2, r1 ; \ + movq 16+P0, r2 ; \ + cmovbq 16+P1, r2 ; \ + cmovnbe 16+P2, r2 ; \ + movq 24+P0, r3 ; \ + cmovbq 24+P1, r3 ; \ + cmovnbe 24+P2, r3 + +S2N_BN_SYMBOL(p256_montjadd_alt): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx +#endif + +// Save registers and make room on stack for temporary variables +// Put the input y in %rbp where it lasts as long as it's needed. + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + + CFI_DEC_RSP(NSPACE) + + movq %rdx, %rbp + +// Main code, just a sequence of basic field operations +// 12 * multiply + 4 * square + 7 * subtract + + montsqr_p256(z1sq,z_1) + montsqr_p256(z2sq,z_2) + + montmul_p256(y1a,z_2,y_1) + montmul_p256(y2a,z_1,y_2) + + montmul_p256(x2a,z1sq,x_2) + montmul_p256(x1a,z2sq,x_1) + montmul_p256(y2a,z1sq,y2a) + montmul_p256(y1a,z2sq,y1a) + + sub_p256(xd,x2a,x1a) + sub_p256(yd,y2a,y1a) + + montsqr_p256(zz,xd) + montsqr_p256(ww,yd) + + montmul_p256(zzx1,zz,x1a) + montmul_p256(zzx2,zz,x2a) + + sub_p256(resx,ww,zzx1) + sub_p256(t1,zzx2,zzx1) + + montmul_p256(xd,xd,z_1) + + sub_p256(resx,resx,zzx2) + + sub_p256(t2,zzx1,resx) + + montmul_p256(t1,t1,y1a) + + montmul_p256(resz,xd,z_2) + montmul_p256(t2,yd,t2) + + sub_p256(resy,t2,t1) + +// Load in the z coordinates of the inputs to check for P1 = 0 and P2 = 0 +// The condition codes get set by a comparison (P2 != 0) - (P1 != 0) +// So "NBE" <=> ~(CF \/ ZF) <=> P1 = 0 /\ ~(P2 = 0) +// and "B" <=> CF <=> ~(P1 = 0) /\ P2 = 0 +// and "Z" <=> ZF <=> (P1 = 0 <=> P2 = 0) + + load4(%r8,%r9,%r10,%r11,z_1) + + movq %r8, %rax + movq %r9, %rdx + orq %r10, %rax + orq %r11, %rdx + orq %rdx, %rax + negq %rax + sbbq %rax, %rax + + load4(%r12,%r13,%r14,%r15,z_2) + + movq %r12, %rbx + movq %r13, %rdx + orq %r14, %rbx + orq %r15, %rdx + orq %rdx, %rbx + negq %rbx + sbbq %rbx, %rbx + + cmpq %rax, %rbx + +// Multiplex the outputs accordingly, re-using the z's in registers + + cmovbq %r8, %r12 + cmovbq %r9, %r13 + cmovbq %r10, %r14 + cmovbq %r11, %r15 + + czload4(%r12,%r13,%r14,%r15,resz) + + muxload4(%rax,%rbx,%rcx,%rdx,resx,x_1,x_2) + muxload4(%r8,%r9,%r10,%r11,resy,y_1,y_2) + +// Finally store back the multiplexed values + + store4(x_3,%rax,%rbx,%rcx,%rdx) + store4(y_3,%r8,%r9,%r10,%r11) + store4(z_3,%r12,%r13,%r14,%r15) + +// Restore stack and registers + + CFI_INC_RSP(NSPACE) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_montjadd_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/p256_montjdouble.S b/cbits/s2n/x86_att/p256_montjdouble.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/p256_montjdouble.S @@ -0,0 +1,634 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Point doubling on NIST curve P-256 in Montgomery-Jacobian coordinates +// +// extern void p256_montjdouble(uint64_t p3[static 12], +// const uint64_t p1[static 12]); +// +// Does p3 := 2 * p1 where all points are regarded as Jacobian triples with +// each coordinate in the Montgomery domain, i.e. x' = (2^256 * x) mod p_256. +// A Jacobian triple (x',y',z') represents affine point (x/z^2,y/z^3). +// +// Standard x86-64 ABI: RDI = p3, RSI = p1 +// Microsoft x64 ABI: RCX = p3, RDX = p1 +// ---------------------------------------------------------------------------- +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_montjdouble) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_montjdouble) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_montjdouble) + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Pointer-offset pairs for inputs and outputs +// These assume %rdi = p3, %rsi = p1, which is true when the +// arguments come in initially and is not disturbed throughout. + +#define x_1 0(%rsi) +#define y_1 NUMSIZE(%rsi) +#define z_1 (2*NUMSIZE)(%rsi) + +#define x_3 0(%rdi) +#define y_3 NUMSIZE(%rdi) +#define z_3 (2*NUMSIZE)(%rdi) + +// Pointer-offset pairs for temporaries, with some aliasing +// NSPACE is the total stack needed for these temporaries + +#define z2 (NUMSIZE*0)(%rsp) +#define y4 (NUMSIZE*0)(%rsp) + +#define y2 (NUMSIZE*1)(%rsp) + +#define t1 (NUMSIZE*2)(%rsp) + +#define t2 (NUMSIZE*3)(%rsp) +#define x2p (NUMSIZE*3)(%rsp) +#define dx2 (NUMSIZE*3)(%rsp) + +#define xy2 (NUMSIZE*4)(%rsp) + +#define x4p (NUMSIZE*5)(%rsp) +#define d (NUMSIZE*5)(%rsp) + +#define NSPACE NUMSIZE*6 + +// Corresponds exactly to bignum_montmul_p256 + +#define montmul_p256(P0,P1,P2) \ + xorl %r13d, %r13d ; \ + movq P2, %rdx ; \ + mulxq P1, %r8, %r9 ; \ + mulxq 0x8+P1, %rbx, %r10 ; \ + adcq %rbx, %r9 ; \ + mulxq 0x10+P1, %rbx, %r11 ; \ + adcq %rbx, %r10 ; \ + mulxq 0x18+P1, %rbx, %r12 ; \ + adcq %rbx, %r11 ; \ + adcq %r13, %r12 ; \ + movq 0x8+P2, %rdx ; \ + xorl %r14d, %r14d ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x18+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcq %r14, %r13 ; \ + xorl %r15d, %r15d ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + notq %rdx; \ + leaq 0x2(%rdx), %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcxq %r15, %r13 ; \ + adoxq %r15, %r14 ; \ + adcq %r15, %r14 ; \ + movq 0x10+P2, %rdx ; \ + xorl %r8d, %r8d ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adoxq %r8, %r14 ; \ + mulxq 0x18+P1, %rax, %rbx ; \ + adcq %rax, %r13 ; \ + adcq %rbx, %r14 ; \ + adcq %r8, %r15 ; \ + movq 0x18+P2, %rdx ; \ + xorl %r9d, %r9d ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + adoxq %r9, %r15 ; \ + mulxq 0x18+P1, %rax, %rbx ; \ + adcq %rax, %r14 ; \ + adcq %rbx, %r15 ; \ + adcq %r9, %r8 ; \ + xorl %r9d, %r9d ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + notq %rdx; \ + leaq 0x2(%rdx), %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r14 ; \ + adoxq %rbx, %r15 ; \ + adcxq %r9, %r15 ; \ + adoxq %r9, %r8 ; \ + adcq %r9, %r8 ; \ + movl $0x1, %ecx ; \ + addq %r12, %rcx ; \ + decq %rdx; \ + adcq %r13, %rdx ; \ + decq %r9; \ + movq %r9, %rax ; \ + adcq %r14, %r9 ; \ + movl $0xfffffffe, %r11d ; \ + adcq %r15, %r11 ; \ + adcq %r8, %rax ; \ + cmovbq %rcx, %r12 ; \ + cmovbq %rdx, %r13 ; \ + cmovbq %r9, %r14 ; \ + cmovbq %r11, %r15 ; \ + movq %r12, P0 ; \ + movq %r13, 0x8+P0 ; \ + movq %r14, 0x10+P0 ; \ + movq %r15, 0x18+P0 + +// Corresponds exactly to bignum_montsqr_p256 + +#define montsqr_p256(P0,P1) \ + movq P1, %rdx ; \ + mulxq %rdx, %r8, %r15 ; \ + mulxq 0x8+P1, %r9, %r10 ; \ + mulxq 0x18+P1, %r11, %r12 ; \ + movq 0x10+P1, %rdx ; \ + mulxq 0x18+P1, %r13, %r14 ; \ + xorl %ebp, %ebp ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + movq 0x18+P1, %rdx ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcxq %rbp, %r13 ; \ + adoxq %rbp, %r14 ; \ + adcq %rbp, %r14 ; \ + xorl %ebp, %ebp ; \ + adcxq %r9, %r9 ; \ + adoxq %r15, %r9 ; \ + movq 0x8+P1, %rdx ; \ + mulxq %rdx, %rax, %rdx ; \ + adcxq %r10, %r10 ; \ + adoxq %rax, %r10 ; \ + adcxq %r11, %r11 ; \ + adoxq %rdx, %r11 ; \ + movq 0x10+P1, %rdx ; \ + mulxq %rdx, %rax, %rdx ; \ + adcxq %r12, %r12 ; \ + adoxq %rax, %r12 ; \ + adcxq %r13, %r13 ; \ + adoxq %rdx, %r13 ; \ + movq 0x18+P1, %rdx ; \ + mulxq %rdx, %rax, %r15 ; \ + adcxq %r14, %r14 ; \ + adoxq %rax, %r14 ; \ + adcxq %rbp, %r15 ; \ + adoxq %rbp, %r15 ; \ + xorl %ebp, %ebp ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + movabsq $0xffffffff00000001, %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcxq %rbp, %r13 ; \ + movl %ebp, %r9d ; \ + adoxq %rbp, %r9 ; \ + adcxq %rbp, %r9 ; \ + addq %r9, %r14 ; \ + adcq %rbp, %r15 ; \ + movl %ebp, %r8d ; \ + adcq %rbp, %r8 ; \ + xorl %ebp, %ebp ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + movabsq $0xffffffff00000001, %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r14 ; \ + adoxq %rbx, %r15 ; \ + adcxq %rbp, %r15 ; \ + adoxq %rbp, %r8 ; \ + adcq %rbp, %r8 ; \ + movl $0x1, %ecx ; \ + addq %r12, %rcx ; \ + leaq -0x1(%rdx), %rdx ; \ + adcq %r13, %rdx ; \ + leaq -0x1(%rbp), %rbp ; \ + movq %rbp, %rax ; \ + adcq %r14, %rbp ; \ + movl $0xfffffffe, %r11d ; \ + adcq %r15, %r11 ; \ + adcq %r8, %rax ; \ + cmovbq %rcx, %r12 ; \ + cmovbq %rdx, %r13 ; \ + cmovbq %rbp, %r14 ; \ + cmovbq %r11, %r15 ; \ + movq %r12, P0 ; \ + movq %r13, 0x8+P0 ; \ + movq %r14, 0x10+P0 ; \ + movq %r15, 0x18+P0 + +// Corresponds exactly to bignum_sub_p256 + +#define sub_p256(P0,P1,P2) \ + movq P1, %rax ; \ + subq P2, %rax ; \ + movq 0x8+P1, %rcx ; \ + sbbq 0x8+P2, %rcx ; \ + movq 0x10+P1, %r8 ; \ + sbbq 0x10+P2, %r8 ; \ + movq 0x18+P1, %r9 ; \ + sbbq 0x18+P2, %r9 ; \ + movl $0xffffffff, %r10d ; \ + sbbq %r11, %r11 ; \ + xorq %rdx, %rdx ; \ + andq %r11, %r10 ; \ + subq %r10, %rdx ; \ + addq %r11, %rax ; \ + movq %rax, P0 ; \ + adcq %r10, %rcx ; \ + movq %rcx, 0x8+P0 ; \ + adcq $0x0, %r8 ; \ + movq %r8, 0x10+P0 ; \ + adcq %rdx, %r9 ; \ + movq %r9, 0x18+P0 + +// Corresponds exactly to bignum_add_p256 + +#define add_p256(P0,P1,P2) \ + xorq %r11, %r11 ; \ + movq P1, %rax ; \ + addq P2, %rax ; \ + movq 0x8+P1, %rcx ; \ + adcq 0x8+P2, %rcx ; \ + movq 0x10+P1, %r8 ; \ + adcq 0x10+P2, %r8 ; \ + movq 0x18+P1, %r9 ; \ + adcq 0x18+P2, %r9 ; \ + adcq %r11, %r11 ; \ + subq $0xffffffffffffffff, %rax ; \ + movl $0xffffffff, %r10d ; \ + sbbq %r10, %rcx ; \ + sbbq $0x0, %r8 ; \ + movq $0xffffffff00000001, %rdx ; \ + sbbq %rdx, %r9 ; \ + sbbq $0x0, %r11 ; \ + andq %r11, %r10 ; \ + andq %r11, %rdx ; \ + addq %r11, %rax ; \ + movq %rax, P0 ; \ + adcq %r10, %rcx ; \ + movq %rcx, 0x8+P0 ; \ + adcq $0x0, %r8 ; \ + movq %r8, 0x10+P0 ; \ + adcq %rdx, %r9 ; \ + movq %r9, 0x18+P0 + +// A weak version of add that only guarantees sum in 4 digits + +#define weakadd_p256(P0,P1,P2) \ + movq P1, %rax ; \ + addq P2, %rax ; \ + movq 0x8+P1, %rcx ; \ + adcq 0x8+P2, %rcx ; \ + movq 0x10+P1, %r8 ; \ + adcq 0x10+P2, %r8 ; \ + movq 0x18+P1, %r9 ; \ + adcq 0x18+P2, %r9 ; \ + movl $0xffffffff, %r10d ; \ + sbbq %r11, %r11 ; \ + xorq %rdx, %rdx ; \ + andq %r11, %r10 ; \ + subq %r10, %rdx ; \ + subq %r11, %rax ; \ + movq %rax, P0 ; \ + sbbq %r10, %rcx ; \ + movq %rcx, 0x8+P0 ; \ + sbbq $0x0, %r8 ; \ + movq %r8, 0x10+P0 ; \ + sbbq %rdx, %r9 ; \ + movq %r9, 0x18+P0 + +// P0 = C * P1 - D * P2 computed as d * (p_256 - P2) + c * P1 +// Quotient estimation is done just as q = h + 1 as in bignum_triple_p256 +// This also applies to the other functions following. + +#define cmsub_p256(P0,C,P1,D,P2) \ + /* First (%r11;%r10;%r9;%r8) = p_256 - P2 */ \ + movq $0xffffffffffffffff, %r8 ; \ + xorl %r10d, %r10d ; \ + subq P2, %r8 ; \ + movq $0x00000000ffffffff, %r9 ; \ + sbbq 0x8+P2, %r9 ; \ + sbbq 0x10+P2, %r10 ; \ + movq $0xffffffff00000001, %r11 ; \ + sbbq 0x18+P2, %r11 ; \ + /* (%r12;%r11;%r10;%r9;%r8) = D * (p_256 - P2) */ \ + xorl %r12d, %r12d ; \ + movq $D, %rdx ; \ + mulxq %r8, %r8, %rax ; \ + mulxq %r9, %r9, %rcx ; \ + addq %rax, %r9 ; \ + mulxq %r10, %r10, %rax ; \ + adcq %rcx, %r10 ; \ + mulxq %r11, %r11, %rcx ; \ + adcq %rax, %r11 ; \ + adcq %rcx, %r12 ; \ + /* (%rdx;%r11;%r10;%r9;%r8) = 2^256 + C * P1 + D * (p_256 - P2) */ \ + movq $C, %rdx ; \ + xorl %eax, %eax ; \ + mulxq P1, %rax, %rcx ; \ + adcxq %rax, %r8 ; \ + adoxq %rcx, %r9 ; \ + mulxq 0x8+P1, %rax, %rcx ; \ + adcxq %rax, %r9 ; \ + adoxq %rcx, %r10 ; \ + mulxq 0x10+P1, %rax, %rcx ; \ + adcxq %rax, %r10 ; \ + adoxq %rcx, %r11 ; \ + mulxq 0x18+P1, %rax, %rdx ; \ + adcxq %rax, %r11 ; \ + adoxq %r12, %rdx ; \ + adcq $1, %rdx ; \ + /* Now the tail for modular reduction from tripling */ \ + addq %rdx, %r8 ; \ + movq $0x100000000, %rax ; \ + mulxq %rax, %rax, %rcx ; \ + sbbq $0x0, %rax ; \ + sbbq $0x0, %rcx ; \ + subq %rax, %r9 ; \ + sbbq %rcx, %r10 ; \ + movq $0xffffffff00000001, %rax ; \ + mulxq %rax, %rax, %rcx ; \ + sbbq %rax, %r11 ; \ + sbbq %rcx, %rdx ; \ + decq %rdx; \ + movl $0xffffffff, %eax ; \ + andq %rdx, %rax ; \ + xorl %ecx, %ecx ; \ + subq %rax, %rcx ; \ + addq %rdx, %r8 ; \ + movq %r8, P0 ; \ + adcq %rax, %r9 ; \ + movq %r9, 0x8+P0 ; \ + adcq $0x0, %r10 ; \ + movq %r10, 0x10+P0 ; \ + adcq %rcx, %r11 ; \ + movq %r11, 0x18+P0 + +// P0 = 3 * P1 - 8 * P2, computed as (p_256 - P2) << 3 + 3 * P1 + +#define cmsub38_p256(P0,P1,P2) \ + /* First (%r11;%r10;%r9;%r8) = p_256 - P2 */ \ + movq $0xffffffffffffffff, %r8 ; \ + xorl %r10d, %r10d ; \ + subq P2, %r8 ; \ + movq $0x00000000ffffffff, %r9 ; \ + sbbq 0x8+P2, %r9 ; \ + sbbq 0x10+P2, %r10 ; \ + movq $0xffffffff00000001, %r11 ; \ + sbbq 0x18+P2, %r11 ; \ + /* (%r12;%r11;%r10;%r9;%r8) = (p_256 - P2) << 3 */ \ + movq %r11, %r12 ; \ + shldq $3, %r10, %r11 ; \ + shldq $3, %r9, %r10 ; \ + shldq $3, %r8, %r9 ; \ + shlq $3, %r8 ; \ + shrq $61, %r12 ; \ + /* (%rdx;%r11;%r10;%r9;%r8) = 2^256 + 3 * P1 + 8 * (p_256 - P2) */ \ + movq $3, %rdx ; \ + xorl %eax, %eax ; \ + mulxq P1, %rax, %rcx ; \ + adcxq %rax, %r8 ; \ + adoxq %rcx, %r9 ; \ + mulxq 0x8+P1, %rax, %rcx ; \ + adcxq %rax, %r9 ; \ + adoxq %rcx, %r10 ; \ + mulxq 0x10+P1, %rax, %rcx ; \ + adcxq %rax, %r10 ; \ + adoxq %rcx, %r11 ; \ + mulxq 0x18+P1, %rax, %rdx ; \ + adcxq %rax, %r11 ; \ + adoxq %r12, %rdx ; \ + adcq $1, %rdx ; \ + /* Now the tail for modular reduction from tripling */ \ + addq %rdx, %r8 ; \ + movq $0x100000000, %rax ; \ + mulxq %rax, %rax, %rcx ; \ + sbbq $0x0, %rax ; \ + sbbq $0x0, %rcx ; \ + subq %rax, %r9 ; \ + sbbq %rcx, %r10 ; \ + movq $0xffffffff00000001, %rax ; \ + mulxq %rax, %rax, %rcx ; \ + sbbq %rax, %r11 ; \ + sbbq %rcx, %rdx ; \ + decq %rdx; \ + movl $0xffffffff, %eax ; \ + andq %rdx, %rax ; \ + xorl %ecx, %ecx ; \ + subq %rax, %rcx ; \ + addq %rdx, %r8 ; \ + movq %r8, P0 ; \ + adcq %rax, %r9 ; \ + movq %r9, 0x8+P0 ; \ + adcq $0x0, %r10 ; \ + movq %r10, 0x10+P0 ; \ + adcq %rcx, %r11 ; \ + movq %r11, 0x18+P0 + +// P0 = 4 * P1 - P2, by direct subtraction of P2, +// since the quotient estimate still works safely +// for initial value > -p_256 + +#define cmsub41_p256(P0,P1,P2) \ + movq 0x18+P1, %r11 ; \ + movq %r11, %rdx ; \ + movq 0x10+P1, %r10 ; \ + shldq $2, %r10, %r11 ; \ + movq 0x8+P1, %r9 ; \ + shldq $2, %r9, %r10 ; \ + movq P1, %r8 ; \ + shldq $2, %r8, %r9 ; \ + shlq $2, %r8 ; \ + shrq $62, %rdx ; \ + addq $1, %rdx ; \ + subq P2, %r8 ; \ + sbbq 0x8+P2, %r9 ; \ + sbbq 0x10+P2, %r10 ; \ + sbbq 0x18+P2, %r11 ; \ + sbbq $0, %rdx ; \ + /* Now the tail for modular reduction from tripling */ \ + addq %rdx, %r8 ; \ + movq $0x100000000, %rax ; \ + mulxq %rax, %rax, %rcx ; \ + sbbq $0x0, %rax ; \ + sbbq $0x0, %rcx ; \ + subq %rax, %r9 ; \ + sbbq %rcx, %r10 ; \ + movq $0xffffffff00000001, %rax ; \ + mulxq %rax, %rax, %rcx ; \ + sbbq %rax, %r11 ; \ + sbbq %rcx, %rdx ; \ + decq %rdx; \ + movl $0xffffffff, %eax ; \ + andq %rdx, %rax ; \ + xorl %ecx, %ecx ; \ + subq %rax, %rcx ; \ + addq %rdx, %r8 ; \ + movq %r8, P0 ; \ + adcq %rax, %r9 ; \ + movq %r9, 0x8+P0 ; \ + adcq $0x0, %r10 ; \ + movq %r10, 0x10+P0 ; \ + adcq %rcx, %r11 ; \ + movq %r11, 0x18+P0 + +S2N_BN_SYMBOL(p256_montjdouble): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Save registers and make room on stack for temporary variables + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + + CFI_DEC_RSP(NSPACE) + +// Main code, just a sequence of basic field operations + +// z2 = z^2 +// y2 = y^2 + + montsqr_p256(z2,z_1) + montsqr_p256(y2,y_1) + +// x2p = x^2 - z^4 = (x + z^2) * (x - z^2) + + sub_p256(t2,x_1,z2) + weakadd_p256(t1,x_1,z2) + montmul_p256(x2p,t1,t2) + +// t1 = y + z +// xy2 = x * y^2 +// x4p = x2p^2 + + add_p256(t1,y_1,z_1) + montmul_p256(xy2,x_1,y2) + montsqr_p256(x4p,x2p) + +// t1 = (y + z)^2 + + montsqr_p256(t1,t1) + +// d = 12 * xy2 - 9 * x4p +// t1 = y^2 + 2 * y * z + + cmsub_p256(d,12,xy2,9,x4p) + sub_p256(t1,t1,z2) + +// y4 = y^4 + + montsqr_p256(y4,y2) + +// dx2 = d * x2p + + montmul_p256(dx2,d,x2p) + +// z_3' = 2 * y * z + + sub_p256(z_3,t1,y2) + +// x' = 4 * xy2 - d + + cmsub41_p256(x_3,xy2,d) + +// y' = 3 * dx2 - 8 * y4 + + cmsub38_p256(y_3,dx2,y4) + +// Restore stack and registers + + CFI_INC_RSP(NSPACE) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_montjdouble) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/p256_montjdouble_alt.S b/cbits/s2n/x86_att/p256_montjdouble_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/p256_montjdouble_alt.S @@ -0,0 +1,747 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Point doubling on NIST curve P-256 in Montgomery-Jacobian coordinates +// +// extern void p256_montjdouble_alt(uint64_t p3[static 12], +// const uint64_t p1[static 12]); +// +// Does p3 := 2 * p1 where all points are regarded as Jacobian triples with +// each coordinate in the Montgomery domain, i.e. x' = (2^256 * x) mod p_256. +// A Jacobian triple (x',y',z') represents affine point (x/z^2,y/z^3). +// +// Standard x86-64 ABI: RDI = p3, RSI = p1 +// Microsoft x64 ABI: RCX = p3, RDX = p1 +// ---------------------------------------------------------------------------- +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_montjdouble_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_montjdouble_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_montjdouble_alt) + .text + .balign 4 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Pointer-offset pairs for inputs and outputs +// These assume %rdi = p3, %rsi = p1, which is true when the +// arguments come in initially and is not disturbed throughout. + +#define x_1 0(%rsi) +#define y_1 NUMSIZE(%rsi) +#define z_1 (2*NUMSIZE)(%rsi) + +#define x_3 0(%rdi) +#define y_3 NUMSIZE(%rdi) +#define z_3 (2*NUMSIZE)(%rdi) + +// Pointer-offset pairs for temporaries, with some aliasing +// NSPACE is the total stack needed for these temporaries + +#define z2 (NUMSIZE*0)(%rsp) +#define y4 (NUMSIZE*0)(%rsp) + +#define y2 (NUMSIZE*1)(%rsp) + +#define t1 (NUMSIZE*2)(%rsp) + +#define t2 (NUMSIZE*3)(%rsp) +#define x2p (NUMSIZE*3)(%rsp) +#define dx2 (NUMSIZE*3)(%rsp) + +#define xy2 (NUMSIZE*4)(%rsp) + +#define x4p (NUMSIZE*5)(%rsp) +#define d (NUMSIZE*5)(%rsp) + +#define NSPACE NUMSIZE*6 + +// Corresponds exactly to bignum_montmul_p256_alt + +#define montmul_p256(P0,P1,P2) \ + movq P2, %rbx ; \ + movq P1, %rax ; \ + mulq %rbx; \ + movq %rax, %r8 ; \ + movq %rdx, %r9 ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + xorl %r10d, %r10d ; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + movq 0x10+P1, %rax ; \ + mulq %rbx; \ + xorl %r11d, %r11d ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + movq 0x18+P1, %rax ; \ + mulq %rbx; \ + xorl %r12d, %r12d ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + movq 0x8+P2, %rbx ; \ + xorl %r13d, %r13d ; \ + movq P1, %rax ; \ + mulq %rbx; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %r14, %r14 ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + subq %r14, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %r14, %r14 ; \ + movq 0x10+P1, %rax ; \ + mulq %rbx; \ + subq %r14, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %r14, %r14 ; \ + movq 0x18+P1, %rax ; \ + mulq %rbx; \ + subq %r14, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + xorl %r14d, %r14d ; \ + movq $0x100000000, %rbx ; \ + movq %r8, %rax ; \ + mulq %rbx; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %r15, %r15 ; \ + movq %r9, %rax ; \ + mulq %rbx; \ + subq %r15, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %r15, %r15 ; \ + notq %rbx; \ + leaq 0x2(%rbx), %rbx ; \ + movq %r8, %rax ; \ + mulq %rbx; \ + subq %r15, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %r15, %r15 ; \ + movq %r9, %rax ; \ + mulq %rbx; \ + subq %r15, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq %r14, %r14 ; \ + movq 0x10+P2, %rbx ; \ + xorl %r15d, %r15d ; \ + movq P1, %rax ; \ + mulq %rbx; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %r8, %r8 ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + subq %r8, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %r8, %r8 ; \ + movq 0x10+P1, %rax ; \ + mulq %rbx; \ + subq %r8, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + sbbq %r8, %r8 ; \ + movq 0x18+P1, %rax ; \ + mulq %rbx; \ + subq %r8, %rdx ; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq %r15, %r15 ; \ + movq 0x18+P2, %rbx ; \ + xorl %r8d, %r8d ; \ + movq P1, %rax ; \ + mulq %rbx; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %r9, %r9 ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + subq %r9, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + sbbq %r9, %r9 ; \ + movq 0x10+P1, %rax ; \ + mulq %rbx; \ + subq %r9, %rdx ; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + sbbq %r9, %r9 ; \ + movq 0x18+P1, %rax ; \ + mulq %rbx; \ + subq %r9, %rdx ; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + adcq %r8, %r8 ; \ + xorl %r9d, %r9d ; \ + movq $0x100000000, %rbx ; \ + movq %r10, %rax ; \ + mulq %rbx; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %rcx, %rcx ; \ + movq %r11, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + sbbq %rcx, %rcx ; \ + notq %rbx; \ + leaq 0x2(%rbx), %rbx ; \ + movq %r10, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + sbbq %rcx, %rcx ; \ + movq %r11, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + adcq %r9, %r8 ; \ + movl $0x1, %ecx ; \ + addq %r12, %rcx ; \ + decq %rbx; \ + adcq %r13, %rbx ; \ + decq %r9; \ + movq %r9, %rax ; \ + adcq %r14, %r9 ; \ + movl $0xfffffffe, %r11d ; \ + adcq %r15, %r11 ; \ + adcq %r8, %rax ; \ + cmovbq %rcx, %r12 ; \ + cmovbq %rbx, %r13 ; \ + cmovbq %r9, %r14 ; \ + cmovbq %r11, %r15 ; \ + movq %r12, P0 ; \ + movq %r13, 0x8+P0 ; \ + movq %r14, 0x10+P0 ; \ + movq %r15, 0x18+P0 + +// Corresponds exactly to bignum_montsqr_p256_alt + +#define montsqr_p256(P0,P1) \ + movq P1, %rax ; \ + movq %rax, %rbx ; \ + mulq %rax; \ + movq %rax, %r8 ; \ + movq %rdx, %r15 ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + movq %rax, %r9 ; \ + movq %rdx, %r10 ; \ + movq 0x18+P1, %rax ; \ + movq %rax, %r13 ; \ + mulq %rbx; \ + movq %rax, %r11 ; \ + movq %rdx, %r12 ; \ + movq 0x10+P1, %rax ; \ + movq %rax, %rbx ; \ + mulq %r13; \ + movq %rax, %r13 ; \ + movq %rdx, %r14 ; \ + movq P1, %rax ; \ + mulq %rbx; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rcx, %rcx ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %rcx, %rcx ; \ + movq 0x18+P1, %rbx ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + xorl %ecx, %ecx ; \ + addq %r9, %r9 ; \ + adcq %r10, %r10 ; \ + adcq %r11, %r11 ; \ + adcq %r12, %r12 ; \ + adcq %r13, %r13 ; \ + adcq %r14, %r14 ; \ + adcq %rcx, %rcx ; \ + movq 0x8+P1, %rax ; \ + mulq %rax; \ + addq %r15, %r9 ; \ + adcq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %r15, %r15 ; \ + movq 0x10+P1, %rax ; \ + mulq %rax; \ + negq %r15; \ + adcq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + sbbq %r15, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq %rax; \ + negq %r15; \ + adcq %rax, %r14 ; \ + adcq %rcx, %rdx ; \ + movq %rdx, %r15 ; \ + movq $0x100000000, %rbx ; \ + movq %r8, %rax ; \ + mulq %rbx; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %rcx, %rcx ; \ + movq %r9, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rcx, %rcx ; \ + notq %rbx; \ + leaq 0x2(%rbx), %rbx ; \ + movq %r8, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %rcx, %rcx ; \ + xorl %r8d, %r8d ; \ + movq %r9, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq %r8, %r14 ; \ + adcq %r8, %r15 ; \ + adcq %r8, %r8 ; \ + movq $0x100000000, %rbx ; \ + movq %r10, %rax ; \ + mulq %rbx; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %rcx, %rcx ; \ + movq %r11, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + sbbq %rcx, %rcx ; \ + notq %rbx; \ + leaq 0x2(%rbx), %rbx ; \ + movq %r10, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + sbbq %rcx, %rcx ; \ + xorl %r9d, %r9d ; \ + movq %r11, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + adcq %r9, %r8 ; \ + movl $0x1, %ecx ; \ + addq %r12, %rcx ; \ + leaq -0x1(%rbx), %rbx ; \ + adcq %r13, %rbx ; \ + leaq -0x1(%r9), %r9 ; \ + movq %r9, %rax ; \ + adcq %r14, %r9 ; \ + movl $0xfffffffe, %r11d ; \ + adcq %r15, %r11 ; \ + adcq %r8, %rax ; \ + cmovbq %rcx, %r12 ; \ + cmovbq %rbx, %r13 ; \ + cmovbq %r9, %r14 ; \ + cmovbq %r11, %r15 ; \ + movq %r12, P0 ; \ + movq %r13, 0x8+P0 ; \ + movq %r14, 0x10+P0 ; \ + movq %r15, 0x18+P0 + +// Corresponds exactly to bignum_sub_p256 + +#define sub_p256(P0,P1,P2) \ + movq P1, %rax ; \ + subq P2, %rax ; \ + movq 0x8+P1, %rcx ; \ + sbbq 0x8+P2, %rcx ; \ + movq 0x10+P1, %r8 ; \ + sbbq 0x10+P2, %r8 ; \ + movq 0x18+P1, %r9 ; \ + sbbq 0x18+P2, %r9 ; \ + movl $0xffffffff, %r10d ; \ + sbbq %r11, %r11 ; \ + xorq %rdx, %rdx ; \ + andq %r11, %r10 ; \ + subq %r10, %rdx ; \ + addq %r11, %rax ; \ + movq %rax, P0 ; \ + adcq %r10, %rcx ; \ + movq %rcx, 0x8+P0 ; \ + adcq $0x0, %r8 ; \ + movq %r8, 0x10+P0 ; \ + adcq %rdx, %r9 ; \ + movq %r9, 0x18+P0 + +// Corresponds exactly to bignum_add_p256 + +#define add_p256(P0,P1,P2) \ + xorq %r11, %r11 ; \ + movq P1, %rax ; \ + addq P2, %rax ; \ + movq 0x8+P1, %rcx ; \ + adcq 0x8+P2, %rcx ; \ + movq 0x10+P1, %r8 ; \ + adcq 0x10+P2, %r8 ; \ + movq 0x18+P1, %r9 ; \ + adcq 0x18+P2, %r9 ; \ + adcq %r11, %r11 ; \ + subq $0xffffffffffffffff, %rax ; \ + movl $0xffffffff, %r10d ; \ + sbbq %r10, %rcx ; \ + sbbq $0x0, %r8 ; \ + movq $0xffffffff00000001, %rdx ; \ + sbbq %rdx, %r9 ; \ + sbbq $0x0, %r11 ; \ + andq %r11, %r10 ; \ + andq %r11, %rdx ; \ + addq %r11, %rax ; \ + movq %rax, P0 ; \ + adcq %r10, %rcx ; \ + movq %rcx, 0x8+P0 ; \ + adcq $0x0, %r8 ; \ + movq %r8, 0x10+P0 ; \ + adcq %rdx, %r9 ; \ + movq %r9, 0x18+P0 + +// A weak version of add that only guarantees sum in 4 digits + +#define weakadd_p256(P0,P1,P2) \ + movq P1, %rax ; \ + addq P2, %rax ; \ + movq 0x8+P1, %rcx ; \ + adcq 0x8+P2, %rcx ; \ + movq 0x10+P1, %r8 ; \ + adcq 0x10+P2, %r8 ; \ + movq 0x18+P1, %r9 ; \ + adcq 0x18+P2, %r9 ; \ + movl $0xffffffff, %r10d ; \ + sbbq %r11, %r11 ; \ + xorq %rdx, %rdx ; \ + andq %r11, %r10 ; \ + subq %r10, %rdx ; \ + subq %r11, %rax ; \ + movq %rax, P0 ; \ + sbbq %r10, %rcx ; \ + movq %rcx, 0x8+P0 ; \ + sbbq $0x0, %r8 ; \ + movq %r8, 0x10+P0 ; \ + sbbq %rdx, %r9 ; \ + movq %r9, 0x18+P0 + +// P0 = C * P1 - D * P2 computed as d * (p_256 - P2) + c * P1 +// Quotient estimation is done just as q = h + 1 as in bignum_triple_p256_alt. +// This also applies to the other functions following. + +#define cmsub_p256(P0,C,P1,D,P2) \ + /* First (%r12;%r11;%r10;%r9) = p_256 - P2 */ \ + movq $0xffffffffffffffff, %r9 ; \ + xorl %r11d, %r11d ; \ + subq P2, %r9 ; \ + movq $0x00000000ffffffff, %r10 ; \ + sbbq 0x8+P2, %r10 ; \ + sbbq 0x10+P2, %r11 ; \ + movq $0xffffffff00000001, %r12 ; \ + sbbq 0x18+P2, %r12 ; \ + /* (%r12;%r11;%r10;%r9;%r8) = D * (p_256 - P2) */ \ + movq $D, %rcx ; \ + movq %r9, %rax ; \ + mulq %rcx; \ + movq %rax, %r8 ; \ + movq %rdx, %r9 ; \ + movq %r10, %rax ; \ + xorl %r10d, %r10d ; \ + mulq %rcx; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + movq %r11, %rax ; \ + xorl %r11d, %r11d ; \ + mulq %rcx; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + movq %r12, %rax ; \ + xorl %r12d, %r12d ; \ + mulq %rcx; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + /* (%rcx;%r11;%r10;%r9;%r8) = 2^256 + C * P1 + D * (p_256 - P2) */ \ + movl $C, %ecx ; \ + movq P1, %rax ; \ + mulq %rcx; \ + addq %rax, %r8 ; \ + adcq %rdx, %r9 ; \ + sbbq %rbx, %rbx ; \ + movq 0x8+P1, %rax ; \ + mulq %rcx; \ + subq %rbx, %rdx ; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %rbx, %rbx ; \ + movq 0x10+P1, %rax ; \ + mulq %rcx; \ + subq %rbx, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rbx, %rbx ; \ + movq 0x18+P1, %rax ; \ + mulq %rcx; \ + subq %rbx, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + leaq 1(%r12), %rcx ; \ + /* Now the tail for modular reduction from tripling */ \ + movq $0xffffffff00000001, %rax ; \ + mulq %rcx; \ + movq %rcx, %rbx ; \ + shlq $0x20, %rbx ; \ + addq %rcx, %r8 ; \ + sbbq $0x0, %rbx ; \ + subq %rbx, %r9 ; \ + sbbq $0x0, %r10 ; \ + sbbq %rax, %r11 ; \ + sbbq %rdx, %rcx ; \ + decq %rcx; \ + movl $0xffffffff, %eax ; \ + andq %rcx, %rax ; \ + xorl %edx, %edx ; \ + subq %rax, %rdx ; \ + addq %rcx, %r8 ; \ + movq %r8, P0 ; \ + adcq %rax, %r9 ; \ + movq %r9, 0x8+P0 ; \ + adcq $0x0, %r10 ; \ + movq %r10, 0x10+P0 ; \ + adcq %rdx, %r11 ; \ + movq %r11, 0x18+P0 + +// P0 = 3 * P1 - 8 * P2, computed as (p_256 - P2) << 3 + 3 * P1 + +#define cmsub38_p256(P0,P1,P2) \ + /* First (%r11;%r10;%r9;%r8) = p_256 - P2 */ \ + movq $0xffffffffffffffff, %r8 ; \ + xorl %r10d, %r10d ; \ + subq P2, %r8 ; \ + movq $0x00000000ffffffff, %r9 ; \ + sbbq 0x8+P2, %r9 ; \ + sbbq 0x10+P2, %r10 ; \ + movq $0xffffffff00000001, %r11 ; \ + sbbq 0x18+P2, %r11 ; \ + /* (%r12;%r11;%r10;%r9;%r8) = (p_256 - P2) << 3 */ \ + movq %r11, %r12 ; \ + shldq $3, %r10, %r11 ; \ + shldq $3, %r9, %r10 ; \ + shldq $3, %r8, %r9 ; \ + shlq $3, %r8 ; \ + shrq $61, %r12 ; \ + /* (%rcx;%r11;%r10;%r9;%r8) = 2^256 + 3 * P1 + 8 * (p_256 - P2) */ \ + movl $3, %ecx ; \ + movq P1, %rax ; \ + mulq %rcx; \ + addq %rax, %r8 ; \ + adcq %rdx, %r9 ; \ + sbbq %rbx, %rbx ; \ + movq 0x8+P1, %rax ; \ + mulq %rcx; \ + subq %rbx, %rdx ; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %rbx, %rbx ; \ + movq 0x10+P1, %rax ; \ + mulq %rcx; \ + subq %rbx, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rbx, %rbx ; \ + movq 0x18+P1, %rax ; \ + mulq %rcx; \ + subq %rbx, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + leaq 1(%r12), %rcx ; \ + /* Now the tail for modular reduction from tripling */ \ + movq $0xffffffff00000001, %rax ; \ + mulq %rcx; \ + movq %rcx, %rbx ; \ + shlq $0x20, %rbx ; \ + addq %rcx, %r8 ; \ + sbbq $0x0, %rbx ; \ + subq %rbx, %r9 ; \ + sbbq $0x0, %r10 ; \ + sbbq %rax, %r11 ; \ + sbbq %rdx, %rcx ; \ + decq %rcx; \ + movl $0xffffffff, %eax ; \ + andq %rcx, %rax ; \ + xorl %edx, %edx ; \ + subq %rax, %rdx ; \ + addq %rcx, %r8 ; \ + movq %r8, P0 ; \ + adcq %rax, %r9 ; \ + movq %r9, 0x8+P0 ; \ + adcq $0x0, %r10 ; \ + movq %r10, 0x10+P0 ; \ + adcq %rdx, %r11 ; \ + movq %r11, 0x18+P0 + +// P0 = 4 * P1 - P2, by direct subtraction of P2, +// since the quotient estimate still works safely +// for initial value > -p_256 + +#define cmsub41_p256(P0,P1,P2) \ + movq 0x18+P1, %r11 ; \ + movq %r11, %rcx ; \ + movq 0x10+P1, %r10 ; \ + shldq $2, %r10, %r11 ; \ + movq 0x8+P1, %r9 ; \ + shldq $2, %r9, %r10 ; \ + movq P1, %r8 ; \ + shldq $2, %r8, %r9 ; \ + shlq $2, %r8 ; \ + shrq $62, %rcx ; \ + addq $1, %rcx ; \ + subq P2, %r8 ; \ + sbbq 0x8+P2, %r9 ; \ + sbbq 0x10+P2, %r10 ; \ + sbbq 0x18+P2, %r11 ; \ + sbbq $0, %rcx ; \ + /* Now the tail for modular reduction from tripling */ \ + movq $0xffffffff00000001, %rax ; \ + mulq %rcx; \ + movq %rcx, %rbx ; \ + shlq $0x20, %rbx ; \ + addq %rcx, %r8 ; \ + sbbq $0x0, %rbx ; \ + subq %rbx, %r9 ; \ + sbbq $0x0, %r10 ; \ + sbbq %rax, %r11 ; \ + sbbq %rdx, %rcx ; \ + decq %rcx; \ + movl $0xffffffff, %eax ; \ + andq %rcx, %rax ; \ + xorl %edx, %edx ; \ + subq %rax, %rdx ; \ + addq %rcx, %r8 ; \ + movq %r8, P0 ; \ + adcq %rax, %r9 ; \ + movq %r9, 0x8+P0 ; \ + adcq $0x0, %r10 ; \ + movq %r10, 0x10+P0 ; \ + adcq %rdx, %r11 ; \ + movq %r11, 0x18+P0 + +S2N_BN_SYMBOL(p256_montjdouble_alt): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi +#endif + +// Save registers and make room on stack for temporary variables + + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + + CFI_DEC_RSP(NSPACE) + +// Main code, just a sequence of basic field operations + +// z2 = z^2 +// y2 = y^2 + + montsqr_p256(z2,z_1) + montsqr_p256(y2,y_1) + +// x2p = x^2 - z^4 = (x + z^2) * (x - z^2) + + sub_p256(t2,x_1,z2) + weakadd_p256(t1,x_1,z2) + montmul_p256(x2p,t1,t2) + +// t1 = y + z +// xy2 = x * y^2 +// x4p = x2p^2 + + add_p256(t1,y_1,z_1) + montmul_p256(xy2,x_1,y2) + montsqr_p256(x4p,x2p) + +// t1 = (y + z)^2 + + montsqr_p256(t1,t1) + +// d = 12 * xy2 - 9 * x4p +// t1 = y^2 + 2 * y * z + + cmsub_p256(d,12,xy2,9,x4p) + sub_p256(t1,t1,z2) + +// y4 = y^4 + + montsqr_p256(y4,y2) + +// dx2 = d * x2p + + montmul_p256(dx2,d,x2p) + +// z_3' = 2 * y * z + + sub_p256(z_3,t1,y2) + +// x' = 4 * xy2 - d + + cmsub41_p256(x_3,xy2,d) + +// y' = 3 * dx2 - 8 * y4 + + cmsub38_p256(y_3,dx2,y4) + +// Restore stack and registers + + CFI_INC_RSP(NSPACE) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_montjdouble_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/p256_montjmixadd.S b/cbits/s2n/x86_att/p256_montjmixadd.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/p256_montjmixadd.S @@ -0,0 +1,567 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Point mixed addition on NIST curve P-256 in Montgomery-Jacobian coordinates +// +// extern void p256_montjmixadd(uint64_t p3[static 12], +// const uint64_t p1[static 12], +// const uint64_t p2[static 8]); +// +// Does p3 := p1 + p2 where all points are regarded as Jacobian triples with +// each coordinate in the Montgomery domain, i.e. x' = (2^256 * x) mod p_256. +// A Jacobian triple (x',y',z') represents affine point (x/z^2,y/z^3). +// The "mixed" part means that p2 only has x and y coordinates, with the +// implicit z coordinate assumed to be the identity. +// +// Standard x86-64 ABI: RDI = p3, RSI = p1, RDX = p2 +// Microsoft x64 ABI: RCX = p3, RDX = p1, R8 = p2 +// ---------------------------------------------------------------------------- +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_montjmixadd) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_montjmixadd) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_montjmixadd) + .text + +// Size of individual field elements + +#define NUMSIZE 32 + +// Pointer-offset pairs for inputs and outputs. +// These assume %rdi = p3, %rsi = p1 and %rbp = p2, +// which needs to be set up explicitly before use. +// The first two hold initially, and the second is +// set up by copying the initial %rdx input to %rbp. +// Thereafter, no code macro modifies any of them. + +#define x_1 0(%rsi) +#define y_1 NUMSIZE(%rsi) +#define z_1 (2*NUMSIZE)(%rsi) + +#define x_2 0(%rbp) +#define y_2 NUMSIZE(%rbp) + +#define x_3 0(%rdi) +#define y_3 NUMSIZE(%rdi) +#define z_3 (2*NUMSIZE)(%rdi) + +// Pointer-offset pairs for temporaries, with some aliasing. +// NSPACE is the total stack needed for all temporaries. + +#define zp2 (NUMSIZE*0)(%rsp) +#define ww (NUMSIZE*0)(%rsp) +#define resx (NUMSIZE*0)(%rsp) + +#define yd (NUMSIZE*1)(%rsp) +#define y2a (NUMSIZE*1)(%rsp) + +#define x2a (NUMSIZE*2)(%rsp) +#define zzx2 (NUMSIZE*2)(%rsp) + +#define zz (NUMSIZE*3)(%rsp) +#define t1 (NUMSIZE*3)(%rsp) + +#define t2 (NUMSIZE*4)(%rsp) +#define zzx1 (NUMSIZE*4)(%rsp) +#define resy (NUMSIZE*4)(%rsp) + +#define xd (NUMSIZE*5)(%rsp) +#define resz (NUMSIZE*5)(%rsp) + +#define NSPACE NUMSIZE*6 + +// Corresponds exactly to bignum_montmul_p256 + +#define montmul_p256(P0,P1,P2) \ + xorl %r13d, %r13d ; \ + movq P2, %rdx ; \ + mulxq P1, %r8, %r9 ; \ + mulxq 0x8+P1, %rbx, %r10 ; \ + adcq %rbx, %r9 ; \ + mulxq 0x10+P1, %rbx, %r11 ; \ + adcq %rbx, %r10 ; \ + mulxq 0x18+P1, %rbx, %r12 ; \ + adcq %rbx, %r11 ; \ + adcq %r13, %r12 ; \ + movq 0x8+P2, %rdx ; \ + xorl %r14d, %r14d ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x18+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcq %r14, %r13 ; \ + xorl %r15d, %r15d ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + notq %rdx; \ + leaq 0x2(%rdx), %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcxq %r15, %r13 ; \ + adoxq %r15, %r14 ; \ + adcq %r15, %r14 ; \ + movq 0x10+P2, %rdx ; \ + xorl %r8d, %r8d ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adoxq %r8, %r14 ; \ + mulxq 0x18+P1, %rax, %rbx ; \ + adcq %rax, %r13 ; \ + adcq %rbx, %r14 ; \ + adcq %r8, %r15 ; \ + movq 0x18+P2, %rdx ; \ + xorl %r9d, %r9d ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + mulxq 0x10+P1, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + adoxq %r9, %r15 ; \ + mulxq 0x18+P1, %rax, %rbx ; \ + adcq %rax, %r14 ; \ + adcq %rbx, %r15 ; \ + adcq %r9, %r8 ; \ + xorl %r9d, %r9d ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + notq %rdx; \ + leaq 0x2(%rdx), %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r14 ; \ + adoxq %rbx, %r15 ; \ + adcxq %r9, %r15 ; \ + adoxq %r9, %r8 ; \ + adcq %r9, %r8 ; \ + movl $0x1, %ecx ; \ + addq %r12, %rcx ; \ + decq %rdx; \ + adcq %r13, %rdx ; \ + decq %r9; \ + movq %r9, %rax ; \ + adcq %r14, %r9 ; \ + movl $0xfffffffe, %r11d ; \ + adcq %r15, %r11 ; \ + adcq %r8, %rax ; \ + cmovbq %rcx, %r12 ; \ + cmovbq %rdx, %r13 ; \ + cmovbq %r9, %r14 ; \ + cmovbq %r11, %r15 ; \ + movq %r12, P0 ; \ + movq %r13, 0x8+P0 ; \ + movq %r14, 0x10+P0 ; \ + movq %r15, 0x18+P0 + +// Corresponds exactly to bignum_montsqr_p256 except for +// register tweaks to avoid modifying %rbp. + +#define montsqr_p256(P0,P1) \ + movq P1, %rdx ; \ + mulxq %rdx, %r8, %r15 ; \ + mulxq 0x8+P1, %r9, %r10 ; \ + mulxq 0x18+P1, %r11, %r12 ; \ + movq 0x10+P1, %rdx ; \ + mulxq 0x18+P1, %r13, %r14 ; \ + xorl %ecx, %ecx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + movq 0x18+P1, %rdx ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcxq %rcx, %r13 ; \ + adoxq %rcx, %r14 ; \ + adcq %rcx, %r14 ; \ + xorl %ecx, %ecx ; \ + adcxq %r9, %r9 ; \ + adoxq %r15, %r9 ; \ + movq 0x8+P1, %rdx ; \ + mulxq %rdx, %rax, %rdx ; \ + adcxq %r10, %r10 ; \ + adoxq %rax, %r10 ; \ + adcxq %r11, %r11 ; \ + adoxq %rdx, %r11 ; \ + movq 0x10+P1, %rdx ; \ + mulxq %rdx, %rax, %rdx ; \ + adcxq %r12, %r12 ; \ + adoxq %rax, %r12 ; \ + adcxq %r13, %r13 ; \ + adoxq %rdx, %r13 ; \ + movq 0x18+P1, %rdx ; \ + mulxq %rdx, %rax, %r15 ; \ + adcxq %r14, %r14 ; \ + adoxq %rax, %r14 ; \ + adcxq %rcx, %r15 ; \ + adoxq %rcx, %r15 ; \ + xorl %ecx, %ecx ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + movabsq $0xffffffff00000001, %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcxq %rcx, %r13 ; \ + movl %ecx, %r9d ; \ + adoxq %rcx, %r9 ; \ + adcxq %rcx, %r9 ; \ + addq %r9, %r14 ; \ + adcq %rcx, %r15 ; \ + movl %ecx, %r8d ; \ + adcq %rcx, %r8 ; \ + xorl %ecx, %ecx ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + movabsq $0xffffffff00000001, %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r14 ; \ + adoxq %rbx, %r15 ; \ + adcxq %rcx, %r15 ; \ + adoxq %rcx, %r8 ; \ + adcq %rcx, %r8 ; \ + movl $0x1, %ebx ; \ + addq %r12, %rbx ; \ + leaq -0x1(%rdx), %rdx ; \ + adcq %r13, %rdx ; \ + leaq -0x1(%rcx), %rcx ; \ + movq %rcx, %rax ; \ + adcq %r14, %rcx ; \ + movl $0xfffffffe, %r11d ; \ + adcq %r15, %r11 ; \ + adcq %r8, %rax ; \ + cmovbq %rbx, %r12 ; \ + cmovbq %rdx, %r13 ; \ + cmovbq %rcx, %r14 ; \ + cmovbq %r11, %r15 ; \ + movq %r12, P0 ; \ + movq %r13, 0x8+P0 ; \ + movq %r14, 0x10+P0 ; \ + movq %r15, 0x18+P0 + +// Almost-Montgomery variant which we use when an input to other muls +// with the other argument fully reduced (which is always safe). +// Again, the basic squaring code is tweaked to avoid modifying %rbp. + +#define amontsqr_p256(P0,P1) \ + movq P1, %rdx ; \ + mulxq %rdx, %r8, %r15 ; \ + mulxq 0x8+P1, %r9, %r10 ; \ + mulxq 0x18+P1, %r11, %r12 ; \ + movq 0x10+P1, %rdx ; \ + mulxq 0x18+P1, %r13, %r14 ; \ + xorl %ecx, %ecx ; \ + mulxq P1, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + movq 0x18+P1, %rdx ; \ + mulxq 0x8+P1, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcxq %rcx, %r13 ; \ + adoxq %rcx, %r14 ; \ + adcq %rcx, %r14 ; \ + xorl %ecx, %ecx ; \ + adcxq %r9, %r9 ; \ + adoxq %r15, %r9 ; \ + movq 0x8+P1, %rdx ; \ + mulxq %rdx, %rax, %rdx ; \ + adcxq %r10, %r10 ; \ + adoxq %rax, %r10 ; \ + adcxq %r11, %r11 ; \ + adoxq %rdx, %r11 ; \ + movq 0x10+P1, %rdx ; \ + mulxq %rdx, %rax, %rdx ; \ + adcxq %r12, %r12 ; \ + adoxq %rax, %r12 ; \ + adcxq %r13, %r13 ; \ + adoxq %rdx, %r13 ; \ + movq 0x18+P1, %rdx ; \ + mulxq %rdx, %rax, %r15 ; \ + adcxq %r14, %r14 ; \ + adoxq %rax, %r14 ; \ + adcxq %rcx, %r15 ; \ + adoxq %rcx, %r15 ; \ + xorl %ecx, %ecx ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r9 ; \ + adoxq %rbx, %r10 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r10 ; \ + adoxq %rbx, %r11 ; \ + movabsq $0xffffffff00000001, %rdx ; \ + mulxq %r8, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r9, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + adcxq %rcx, %r13 ; \ + movl %ecx, %r9d ; \ + adoxq %rcx, %r9 ; \ + adcxq %rcx, %r9 ; \ + addq %r9, %r14 ; \ + adcq %rcx, %r15 ; \ + movl %ecx, %r8d ; \ + adcq %rcx, %r8 ; \ + xorl %ecx, %ecx ; \ + movabsq $0x100000000, %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r11 ; \ + adoxq %rbx, %r12 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r12 ; \ + adoxq %rbx, %r13 ; \ + movabsq $0xffffffff00000001, %rdx ; \ + mulxq %r10, %rax, %rbx ; \ + adcxq %rax, %r13 ; \ + adoxq %rbx, %r14 ; \ + mulxq %r11, %rax, %rbx ; \ + adcxq %rax, %r14 ; \ + adoxq %rbx, %r15 ; \ + adcxq %rcx, %r15 ; \ + adoxq %rcx, %r8 ; \ + adcq %rcx, %r8 ; \ + movl $0x1, %r8d ; \ + leaq -0x1(%rdx), %rdx ; \ + leaq -0x1(%rcx), %rax ; \ + movl $0xfffffffe, %r11d ; \ + cmovzq %rcx, %r8 ; \ + cmovzq %rcx, %rdx ; \ + cmovzq %rcx, %rax ; \ + cmovzq %rcx, %r11 ; \ + addq %r8, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq %rax, %r14 ; \ + adcq %r11, %r15 ; \ + movq %r12, P0 ; \ + movq %r13, 0x8+P0 ; \ + movq %r14, 0x10+P0 ; \ + movq %r15, 0x18+P0 + +// Corresponds exactly to bignum_sub_p256 + +#define sub_p256(P0,P1,P2) \ + movq P1, %rax ; \ + subq P2, %rax ; \ + movq 0x8+P1, %rcx ; \ + sbbq 0x8+P2, %rcx ; \ + movq 0x10+P1, %r8 ; \ + sbbq 0x10+P2, %r8 ; \ + movq 0x18+P1, %r9 ; \ + sbbq 0x18+P2, %r9 ; \ + movl $0xffffffff, %r10d ; \ + sbbq %r11, %r11 ; \ + xorq %rdx, %rdx ; \ + andq %r11, %r10 ; \ + subq %r10, %rdx ; \ + addq %r11, %rax ; \ + movq %rax, P0 ; \ + adcq %r10, %rcx ; \ + movq %rcx, 0x8+P0 ; \ + adcq $0x0, %r8 ; \ + movq %r8, 0x10+P0 ; \ + adcq %rdx, %r9 ; \ + movq %r9, 0x18+P0 + +// Additional macros to help with final multiplexing + +#define testzero4(P) \ + movq P, %rax ; \ + movq 8+P, %rdx ; \ + orq 16+P, %rax ; \ + orq 24+P, %rdx ; \ + orq %rdx, %rax + +#define mux4(r0,r1,r2,r3,PNE,PEQ) \ + movq PNE, r0 ; \ + movq PEQ, %rax ; \ + cmovzq %rax, r0 ; \ + movq 8+PNE, r1 ; \ + movq 8+PEQ, %rax ; \ + cmovzq %rax, r1 ; \ + movq 16+PNE, r2 ; \ + movq 16+PEQ, %rax ; \ + cmovzq %rax, r2 ; \ + movq 24+PNE, r3 ; \ + movq 24+PEQ, %rax ; \ + cmovzq %rax, r3 + +#define load4(r0,r1,r2,r3,P) \ + movq P, r0 ; \ + movq 8+P, r1 ; \ + movq 16+P, r2 ; \ + movq 24+P, r3 + +#define store4(P,r0,r1,r2,r3) \ + movq r0, P ; \ + movq r1, 8+P ; \ + movq r2, 16+P ; \ + movq r3, 24+P + +S2N_BN_SYMBOL(p256_montjmixadd): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx +#endif + +// Save registers and make room on stack for temporary variables +// Put the input y in %rbp where it lasts as long as it's needed. + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + + CFI_DEC_RSP(NSPACE) + + movq %rdx, %rbp + +// Main code, just a sequence of basic field operations +// 8 * multiply + 3 * square + 7 * subtract + + amontsqr_p256(zp2,z_1) + + montmul_p256(y2a,z_1,y_2) + montmul_p256(x2a,zp2,x_2) + montmul_p256(y2a,zp2,y2a) + + sub_p256(xd,x2a,x_1) + + sub_p256(yd,y2a,y_1) + + amontsqr_p256(zz,xd) + montsqr_p256(ww,yd) + + montmul_p256(zzx1,zz,x_1) + montmul_p256(zzx2,zz,x2a) + + sub_p256(resx,ww,zzx1) + sub_p256(t1,zzx2,zzx1) + + montmul_p256(resz,xd,z_1) + + sub_p256(resx,resx,zzx2) + + sub_p256(t2,zzx1,resx) + + montmul_p256(t1,t1,y_1) + montmul_p256(t2,yd,t2) + + sub_p256(resy,t2,t1) + +// Test if z_1 = 0 to decide if p1 = 0 (up to projective equivalence) + + testzero4(z_1) + +// Multiplex: if p1 <> 0 just copy the computed result from the staging area. +// If p1 = 0 then return the point p2 augmented with a z = 1 coordinate (in +// Montgomery form so not the simple constant 1 but rather 2^256 - p_256), +// hence giving 0 + p2 = p2 for the final result. + + mux4(%r8,%r9,%r10,%r11,resx,x_2) + mux4(%r12,%r13,%r14,%r15,resy,y_2) + + store4(x_3,%r8,%r9,%r10,%r11) + store4(y_3,%r12,%r13,%r14,%r15) + + load4(%r8,%r9,%r10,%r11,resz) + movl $1, %eax + cmovzq %rax, %r8 + movq $0xffffffff00000000, %rax + cmovzq %rax, %r9 + movq $0xffffffffffffffff, %rax + cmovzq %rax, %r10 + movl $0x00000000fffffffe, %eax + cmovzq %rax, %r11 + + store4(z_3,%r8,%r9,%r10,%r11) + +// Restore stack and registers + + CFI_INC_RSP(NSPACE) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_montjmixadd) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/p256_montjmixadd_alt.S b/cbits/s2n/x86_att/p256_montjmixadd_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/p256_montjmixadd_alt.S @@ -0,0 +1,552 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Point mixed addition on NIST curve P-256 in Montgomery-Jacobian coordinates +// +// extern void p256_montjmixadd_alt(uint64_t p3[static 12], +// const uint64_t p1[static 12], +// const uint64_t p2[static 8]); +// +// Does p3 := p1 + p2 where all points are regarded as Jacobian triples with +// each coordinate in the Montgomery domain, i.e. x' = (2^256 * x) mod p_256. +// A Jacobian triple (x',y',z') represents affine point (x/z^2,y/z^3). +// The "mixed" part means that p2 only has x and y coordinates, with the +// implicit z coordinate assumed to be the identity. +// +// Standard x86-64 ABI: RDI = p3, RSI = p1, RDX = p2 +// Microsoft x64 ABI: RCX = p3, RDX = p1, R8 = p2 +// ---------------------------------------------------------------------------- +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_montjmixadd_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_montjmixadd_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_montjmixadd_alt) + .text + +// Size of individual field elements + +#define NUMSIZE 32 + +// Pointer-offset pairs for inputs and outputs. +// These assume %rdi = p3, %rsi = p1 and %rbp = p2, +// which needs to be set up explicitly before use. +// The first two hold initially, and the second is +// set up by copying the initial %rdx input to %rbp. +// Thereafter, no code macro modifies any of them. + +#define x_1 0(%rsi) +#define y_1 NUMSIZE(%rsi) +#define z_1 (2*NUMSIZE)(%rsi) + +#define x_2 0(%rbp) +#define y_2 NUMSIZE(%rbp) + +#define x_3 0(%rdi) +#define y_3 NUMSIZE(%rdi) +#define z_3 (2*NUMSIZE)(%rdi) + +// Pointer-offset pairs for temporaries, with some aliasing. +// NSPACE is the total stack needed for all temporaries. + +#define zp2 (NUMSIZE*0)(%rsp) +#define ww (NUMSIZE*0)(%rsp) +#define resx (NUMSIZE*0)(%rsp) + +#define yd (NUMSIZE*1)(%rsp) +#define y2a (NUMSIZE*1)(%rsp) + +#define x2a (NUMSIZE*2)(%rsp) +#define zzx2 (NUMSIZE*2)(%rsp) + +#define zz (NUMSIZE*3)(%rsp) +#define t1 (NUMSIZE*3)(%rsp) + +#define t2 (NUMSIZE*4)(%rsp) +#define zzx1 (NUMSIZE*4)(%rsp) +#define resy (NUMSIZE*4)(%rsp) + +#define xd (NUMSIZE*5)(%rsp) +#define resz (NUMSIZE*5)(%rsp) + +#define NSPACE NUMSIZE*6 + +// Corresponds exactly to bignum_montmul_p256_alt + +#define montmul_p256(P0,P1,P2) \ + movq P2, %rbx ; \ + movq P1, %rax ; \ + mulq %rbx; \ + movq %rax, %r8 ; \ + movq %rdx, %r9 ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + xorl %r10d, %r10d ; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + movq 0x10+P1, %rax ; \ + mulq %rbx; \ + xorl %r11d, %r11d ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + movq 0x18+P1, %rax ; \ + mulq %rbx; \ + xorl %r12d, %r12d ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + movq 0x8+P2, %rbx ; \ + xorl %r13d, %r13d ; \ + movq P1, %rax ; \ + mulq %rbx; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %r14, %r14 ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + subq %r14, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %r14, %r14 ; \ + movq 0x10+P1, %rax ; \ + mulq %rbx; \ + subq %r14, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %r14, %r14 ; \ + movq 0x18+P1, %rax ; \ + mulq %rbx; \ + subq %r14, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + xorl %r14d, %r14d ; \ + movq $0x100000000, %rbx ; \ + movq %r8, %rax ; \ + mulq %rbx; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %r15, %r15 ; \ + movq %r9, %rax ; \ + mulq %rbx; \ + subq %r15, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %r15, %r15 ; \ + notq %rbx; \ + leaq 0x2(%rbx), %rbx ; \ + movq %r8, %rax ; \ + mulq %rbx; \ + subq %r15, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %r15, %r15 ; \ + movq %r9, %rax ; \ + mulq %rbx; \ + subq %r15, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq %r14, %r14 ; \ + movq 0x10+P2, %rbx ; \ + xorl %r15d, %r15d ; \ + movq P1, %rax ; \ + mulq %rbx; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %r8, %r8 ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + subq %r8, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %r8, %r8 ; \ + movq 0x10+P1, %rax ; \ + mulq %rbx; \ + subq %r8, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + sbbq %r8, %r8 ; \ + movq 0x18+P1, %rax ; \ + mulq %rbx; \ + subq %r8, %rdx ; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + adcq %r15, %r15 ; \ + movq 0x18+P2, %rbx ; \ + xorl %r8d, %r8d ; \ + movq P1, %rax ; \ + mulq %rbx; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %r9, %r9 ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + subq %r9, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + sbbq %r9, %r9 ; \ + movq 0x10+P1, %rax ; \ + mulq %rbx; \ + subq %r9, %rdx ; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + sbbq %r9, %r9 ; \ + movq 0x18+P1, %rax ; \ + mulq %rbx; \ + subq %r9, %rdx ; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + adcq %r8, %r8 ; \ + xorl %r9d, %r9d ; \ + movq $0x100000000, %rbx ; \ + movq %r10, %rax ; \ + mulq %rbx; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %rcx, %rcx ; \ + movq %r11, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + sbbq %rcx, %rcx ; \ + notq %rbx; \ + leaq 0x2(%rbx), %rbx ; \ + movq %r10, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + sbbq %rcx, %rcx ; \ + movq %r11, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + adcq %r9, %r8 ; \ + movl $0x1, %ecx ; \ + addq %r12, %rcx ; \ + decq %rbx; \ + adcq %r13, %rbx ; \ + decq %r9; \ + movq %r9, %rax ; \ + adcq %r14, %r9 ; \ + movl $0xfffffffe, %r11d ; \ + adcq %r15, %r11 ; \ + adcq %r8, %rax ; \ + cmovbq %rcx, %r12 ; \ + cmovbq %rbx, %r13 ; \ + cmovbq %r9, %r14 ; \ + cmovbq %r11, %r15 ; \ + movq %r12, P0 ; \ + movq %r13, 0x8+P0 ; \ + movq %r14, 0x10+P0 ; \ + movq %r15, 0x18+P0 + +// Corresponds exactly to bignum_montsqr_p256_alt + +#define montsqr_p256(P0,P1) \ + movq P1, %rax ; \ + movq %rax, %rbx ; \ + mulq %rax; \ + movq %rax, %r8 ; \ + movq %rdx, %r15 ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + movq %rax, %r9 ; \ + movq %rdx, %r10 ; \ + movq 0x18+P1, %rax ; \ + movq %rax, %r13 ; \ + mulq %rbx; \ + movq %rax, %r11 ; \ + movq %rdx, %r12 ; \ + movq 0x10+P1, %rax ; \ + movq %rax, %rbx ; \ + mulq %r13; \ + movq %rax, %r13 ; \ + movq %rdx, %r14 ; \ + movq P1, %rax ; \ + mulq %rbx; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rcx, %rcx ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %rcx, %rcx ; \ + movq 0x18+P1, %rbx ; \ + movq 0x8+P1, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq $0x0, %r14 ; \ + xorl %ecx, %ecx ; \ + addq %r9, %r9 ; \ + adcq %r10, %r10 ; \ + adcq %r11, %r11 ; \ + adcq %r12, %r12 ; \ + adcq %r13, %r13 ; \ + adcq %r14, %r14 ; \ + adcq %rcx, %rcx ; \ + movq 0x8+P1, %rax ; \ + mulq %rax; \ + addq %r15, %r9 ; \ + adcq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %r15, %r15 ; \ + movq 0x10+P1, %rax ; \ + mulq %rax; \ + negq %r15; \ + adcq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + sbbq %r15, %r15 ; \ + movq 0x18+P1, %rax ; \ + mulq %rax; \ + negq %r15; \ + adcq %rax, %r14 ; \ + adcq %rcx, %rdx ; \ + movq %rdx, %r15 ; \ + movq $0x100000000, %rbx ; \ + movq %r8, %rax ; \ + mulq %rbx; \ + addq %rax, %r9 ; \ + adcq %rdx, %r10 ; \ + sbbq %rcx, %rcx ; \ + movq %r9, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r10 ; \ + adcq %rdx, %r11 ; \ + sbbq %rcx, %rcx ; \ + notq %rbx; \ + leaq 0x2(%rbx), %rbx ; \ + movq %r8, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %rcx, %rcx ; \ + xorl %r8d, %r8d ; \ + movq %r9, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + adcq %r8, %r14 ; \ + adcq %r8, %r15 ; \ + adcq %r8, %r8 ; \ + movq $0x100000000, %rbx ; \ + movq %r10, %rax ; \ + mulq %rbx; \ + addq %rax, %r11 ; \ + adcq %rdx, %r12 ; \ + sbbq %rcx, %rcx ; \ + movq %r11, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r12 ; \ + adcq %rdx, %r13 ; \ + sbbq %rcx, %rcx ; \ + notq %rbx; \ + leaq 0x2(%rbx), %rbx ; \ + movq %r10, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r13 ; \ + adcq %rdx, %r14 ; \ + sbbq %rcx, %rcx ; \ + xorl %r9d, %r9d ; \ + movq %r11, %rax ; \ + mulq %rbx; \ + subq %rcx, %rdx ; \ + addq %rax, %r14 ; \ + adcq %rdx, %r15 ; \ + adcq %r9, %r8 ; \ + movl $0x1, %ecx ; \ + addq %r12, %rcx ; \ + leaq -0x1(%rbx), %rbx ; \ + adcq %r13, %rbx ; \ + leaq -0x1(%r9), %r9 ; \ + movq %r9, %rax ; \ + adcq %r14, %r9 ; \ + movl $0xfffffffe, %r11d ; \ + adcq %r15, %r11 ; \ + adcq %r8, %rax ; \ + cmovbq %rcx, %r12 ; \ + cmovbq %rbx, %r13 ; \ + cmovbq %r9, %r14 ; \ + cmovbq %r11, %r15 ; \ + movq %r12, P0 ; \ + movq %r13, 0x8+P0 ; \ + movq %r14, 0x10+P0 ; \ + movq %r15, 0x18+P0 + +// Corresponds exactly to bignum_sub_p256 + +#define sub_p256(P0,P1,P2) \ + movq P1, %rax ; \ + subq P2, %rax ; \ + movq 0x8+P1, %rcx ; \ + sbbq 0x8+P2, %rcx ; \ + movq 0x10+P1, %r8 ; \ + sbbq 0x10+P2, %r8 ; \ + movq 0x18+P1, %r9 ; \ + sbbq 0x18+P2, %r9 ; \ + movl $0xffffffff, %r10d ; \ + sbbq %r11, %r11 ; \ + xorq %rdx, %rdx ; \ + andq %r11, %r10 ; \ + subq %r10, %rdx ; \ + addq %r11, %rax ; \ + movq %rax, P0 ; \ + adcq %r10, %rcx ; \ + movq %rcx, 0x8+P0 ; \ + adcq $0x0, %r8 ; \ + movq %r8, 0x10+P0 ; \ + adcq %rdx, %r9 ; \ + movq %r9, 0x18+P0 + +// Additional macros to help with final multiplexing + +#define testzero4(P) \ + movq P, %rax ; \ + movq 8+P, %rdx ; \ + orq 16+P, %rax ; \ + orq 24+P, %rdx ; \ + orq %rdx, %rax + +#define mux4(r0,r1,r2,r3,PNE,PEQ) \ + movq PNE, r0 ; \ + movq PEQ, %rax ; \ + cmovzq %rax, r0 ; \ + movq 8+PNE, r1 ; \ + movq 8+PEQ, %rax ; \ + cmovzq %rax, r1 ; \ + movq 16+PNE, r2 ; \ + movq 16+PEQ, %rax ; \ + cmovzq %rax, r2 ; \ + movq 24+PNE, r3 ; \ + movq 24+PEQ, %rax ; \ + cmovzq %rax, r3 + +#define load4(r0,r1,r2,r3,P) \ + movq P, r0 ; \ + movq 8+P, r1 ; \ + movq 16+P, r2 ; \ + movq 24+P, r3 + +#define store4(P,r0,r1,r2,r3) \ + movq r0, P ; \ + movq r1, 8+P ; \ + movq r2, 16+P ; \ + movq r3, 24+P + +S2N_BN_SYMBOL(p256_montjmixadd_alt): + CFI_START + _CET_ENDBR + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx +#endif + +// Save registers and make room on stack for temporary variables +// Put the input y in %rbp where it lasts as long as it's needed. + + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + + CFI_DEC_RSP(NSPACE) + + movq %rdx, %rbp + +// Main code, just a sequence of basic field operations +// 8 * multiply + 3 * square + 7 * subtract + + montsqr_p256(zp2,z_1) + + montmul_p256(y2a,z_1,y_2) + montmul_p256(x2a,zp2,x_2) + montmul_p256(y2a,zp2,y2a) + + sub_p256(xd,x2a,x_1) + + sub_p256(yd,y2a,y_1) + + montsqr_p256(zz,xd) + montsqr_p256(ww,yd) + + montmul_p256(zzx1,zz,x_1) + montmul_p256(zzx2,zz,x2a) + + sub_p256(resx,ww,zzx1) + sub_p256(t1,zzx2,zzx1) + + montmul_p256(resz,xd,z_1) + + sub_p256(resx,resx,zzx2) + + sub_p256(t2,zzx1,resx) + + montmul_p256(t1,t1,y_1) + montmul_p256(t2,yd,t2) + + sub_p256(resy,t2,t1) + +// Test if z_1 = 0 to decide if p1 = 0 (up to projective equivalence) + + testzero4(z_1) + +// Multiplex: if p1 <> 0 just copy the computed result from the staging area. +// If p1 = 0 then return the point p2 augmented with a z = 1 coordinate (in +// Montgomery form so not the simple constant 1 but rather 2^256 - p_256), +// hence giving 0 + p2 = p2 for the final result. + + mux4(%r8,%r9,%r10,%r11,resx,x_2) + mux4(%r12,%r13,%r14,%r15,resy,y_2) + + store4(x_3,%r8,%r9,%r10,%r11) + store4(y_3,%r12,%r13,%r14,%r15) + + load4(%r8,%r9,%r10,%r11,resz) + movl $1, %eax + cmovzq %rax, %r8 + movq $0xffffffff00000000, %rax + cmovzq %rax, %r9 + movq $0xffffffffffffffff, %rax + cmovzq %rax, %r10 + movl $0x00000000fffffffe, %eax + cmovzq %rax, %r11 + + store4(z_3,%r8,%r9,%r10,%r11) + +// Restore stack and registers + + CFI_INC_RSP(NSPACE) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + +#if WINDOWS_ABI + CFI_POP(%rsi) + CFI_POP(%rdi) +#endif + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_montjmixadd_alt) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/p256_scalarmul.S b/cbits/s2n/x86_att/p256_scalarmul.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/p256_scalarmul.S @@ -0,0 +1,6823 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Scalar multiplication for P-256 +// Input scalar[4], point[8]; output res[8] +// +// extern void p256_scalarmul +// (uint64_t res[static 8],const uint64_t scalar[static 4], +// const uint64_t point[static 8]); +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-256, returns the point (X,Y) = n * P. The input and output +// are affine points, and in the case of the point at infinity as +// the result, (0,0) is returned. +// +// Standard x86-64 ABI: RDI = res, RSI = scalar, RDX = point +// Microsoft x64 ABI: RCX = res, RDX = scalar, R8 = point +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_scalarmul) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_scalarmul) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_scalarmul) + + .text + .balign 32 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Intermediate variables on the stack. The last z2, z3 values can +// safely be overlaid on the table, which is no longer needed at the end. +// Uppercase syntactic variants make x86_att version simpler to generate + +#define SCALARB (0*NUMSIZE) +#define scalarb (0*NUMSIZE)(%rsp) +#define ACC (1*NUMSIZE) +#define acc (1*NUMSIZE)(%rsp) +#define TABENT (4*NUMSIZE) +#define tabent (4*NUMSIZE)(%rsp) + +#define TAB (7*NUMSIZE) +#define tab (7*NUMSIZE)(%rsp) + +#define Z2 (7*NUMSIZE) +#define z2 (7*NUMSIZE)(%rsp) +#define Z3 (8*NUMSIZE) +#define z3 (8*NUMSIZE)(%rsp) + +#define res (31*NUMSIZE)(%rsp) + +#define NSPACE 32*NUMSIZE + +S2N_BN_SYMBOL(p256_scalarmul): + CFI_START + _CET_ENDBR + +// The Windows version literally calls the standard ABI version. +// This simplifies the proofs since subroutine offsets are fixed. + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + CFI_CALL(Lp256_scalarmul_standard) + CFI_POP(%rsi) + CFI_POP(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_scalarmul) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_standard) + +Lp256_scalarmul_standard: + CFI_START +#endif + +// Real start of the standard ABI code. + + CFI_PUSH(%r15) + CFI_PUSH(%r14) + CFI_PUSH(%r13) + CFI_PUSH(%r12) + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + + CFI_DEC_RSP(NSPACE) + +// Preserve the "res" and "point" input arguments. We load and process the +// scalar immediately so we don't bother preserving that input argument. +// Also, "point" is only needed early on and so its register gets re-used. + + movq %rdx, %rbx + movq %rdi, res + +// Load the digits of group order n_256 = [%r15;%r14;%r13;%r12] + + movq $0xf3b9cac2fc632551, %r12 + movq $0xbce6faada7179e84, %r13 + movq $0xffffffffffffffff, %r14 + movq $0xffffffff00000000, %r15 + +// First, reduce the input scalar mod n_256, i.e. conditionally subtract n_256 + + movq (%rsi), %r8 + subq %r12, %r8 + movq 8(%rsi), %r9 + sbbq %r13, %r9 + movq 16(%rsi), %r10 + sbbq %r14, %r10 + movq 24(%rsi), %r11 + sbbq %r15, %r11 + + cmovcq (%rsi), %r8 + cmovcq 8(%rsi), %r9 + cmovcq 16(%rsi), %r10 + cmovcq 24(%rsi), %r11 + +// Now if the top bit of the reduced scalar is set, negate it mod n_256, +// i.e. do n |-> n_256 - n. Remember the sign in %rbp so we can +// correspondingly negate the point below. + + subq %r8, %r12 + sbbq %r9, %r13 + sbbq %r10, %r14 + sbbq %r11, %r15 + + movq %r11, %rbp + shrq $63, %rbp + cmovnzq %r12, %r8 + cmovnzq %r13, %r9 + cmovnzq %r14, %r10 + cmovnzq %r15, %r11 + +// In either case then add the recoding constant 0x08888...888 to allow +// signed digits. + + movq $0x8888888888888888, %rax + addq %rax, %r8 + adcq %rax, %r9 + adcq %rax, %r10 + adcq %rax, %r11 + btc $63, %r11 + + movq %r8, SCALARB(%rsp) + movq %r9, SCALARB+8(%rsp) + movq %r10, SCALARB+16(%rsp) + movq %r11, SCALARB+24(%rsp) + +// Set the tab[0] table entry to Montgomery-Jacobian point = 1 * P +// The z coordinate is just the Montgomery form of the constant 1. + + leaq TAB(%rsp), %rdi + movq %rbx, %rsi + CFI_CALL(Lp256_scalarmul_local_tomont_p256) + + leaq 32(%rbx), %rsi + leaq TAB+32(%rsp), %rdi + CFI_CALL(Lp256_scalarmul_local_tomont_p256) + + movl $1, %eax + movq %rax, TAB+64(%rsp) + movq $0xffffffff00000000, %rdx + movq %rdx, TAB+72(%rsp) + subq $2, %rax + movq %rax, TAB+80(%rsp) + movq $0x00000000fffffffe, %rax + movq %rax, TAB+88(%rsp) + +// If the top bit of the scalar was set, negate (y coordinate of) the point + + movq TAB+32(%rsp), %r12 + movq TAB+40(%rsp), %r13 + movq TAB+48(%rsp), %r14 + movq TAB+56(%rsp), %r15 + + xorl %r10d, %r10d + leaq -1(%r10), %r8 + movq $0x00000000ffffffff, %r11 + movq %r11, %r9 + negq %r11 + + subq %r12, %r8 + sbbq %r13, %r9 + sbbq %r14, %r10 + sbbq %r15, %r11 + + testq %rbp, %rbp + cmovzq %r12, %r8 + cmovzq %r13, %r9 + cmovzq %r14, %r10 + cmovzq %r15, %r11 + + movq %r8, TAB+32(%rsp) + movq %r9, TAB+40(%rsp) + movq %r10, TAB+48(%rsp) + movq %r11, TAB+56(%rsp) + +// Compute and record tab[1] = 2 * p, ..., tab[7] = 8 * P + + leaq TAB+96*1(%rsp), %rdi + leaq TAB(%rsp), %rsi + CFI_CALL(Lp256_scalarmul_local_p256_montjdouble) + + leaq TAB+96*2(%rsp), %rdi + leaq TAB+96*1(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp256_scalarmul_local_p256_montjmixadd) + + leaq TAB+96*3(%rsp), %rdi + leaq TAB+96*1(%rsp), %rsi + CFI_CALL(Lp256_scalarmul_local_p256_montjdouble) + + leaq TAB+96*4(%rsp), %rdi + leaq TAB+96*3(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp256_scalarmul_local_p256_montjmixadd) + + leaq TAB+96*5(%rsp), %rdi + leaq TAB+96*2(%rsp), %rsi + CFI_CALL(Lp256_scalarmul_local_p256_montjdouble) + + leaq TAB+96*6(%rsp), %rdi + leaq TAB+96*5(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp256_scalarmul_local_p256_montjmixadd) + + leaq TAB+96*7(%rsp), %rdi + leaq TAB+96*3(%rsp), %rsi + CFI_CALL(Lp256_scalarmul_local_p256_montjdouble) + +// Set up accumulator as table entry for top 4 bits (constant-time indexing) + + movq SCALARB+24(%rsp), %rdi + shrq $60, %rdi + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + xorl %r13d, %r13d + xorl %r14d, %r14d + xorl %r15d, %r15d + + .set I, 1 +.rep 8 + cmpq $I, %rdi + + cmovzq TAB+96*(I-1)(%rsp), %rax + cmovzq TAB+96*(I-1)+8(%rsp), %rbx + cmovzq TAB+96*(I-1)+16(%rsp), %rcx + cmovzq TAB+96*(I-1)+24(%rsp), %rdx + cmovzq TAB+96*(I-1)+32(%rsp), %r8 + cmovzq TAB+96*(I-1)+40(%rsp), %r9 + cmovzq TAB+96*(I-1)+48(%rsp), %r10 + cmovzq TAB+96*(I-1)+56(%rsp), %r11 + cmovzq TAB+96*(I-1)+64(%rsp), %r12 + cmovzq TAB+96*(I-1)+72(%rsp), %r13 + cmovzq TAB+96*(I-1)+80(%rsp), %r14 + cmovzq TAB+96*(I-1)+88(%rsp), %r15 + .set I, (I+1) +.endr + movq %rax, ACC(%rsp) + movq %rbx, ACC+8(%rsp) + movq %rcx, ACC+16(%rsp) + movq %rdx, ACC+24(%rsp) + movq %r8, ACC+32(%rsp) + movq %r9, ACC+40(%rsp) + movq %r10, ACC+48(%rsp) + movq %r11, ACC+56(%rsp) + movq %r12, ACC+64(%rsp) + movq %r13, ACC+72(%rsp) + movq %r14, ACC+80(%rsp) + movq %r15, ACC+88(%rsp) + +// Main loop over size-4 bitfield + + movl $252, %ebp + +Lp256_scalarmul_loop: + subq $4, %rbp + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp256_scalarmul_local_p256_montjdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp256_scalarmul_local_p256_montjdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp256_scalarmul_local_p256_montjdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp256_scalarmul_local_p256_montjdouble) + + movq %rbp, %rax + shrq $6, %rax + movq (%rsp,%rax,8), %rdi + movq %rbp, %rcx + shrq %cl, %rdi + andq $15, %rdi + + subq $8, %rdi + sbbq %rsi, %rsi // %rsi = sign of digit (-1 = negative) + xorq %rsi, %rdi + subq %rsi, %rdi // %rdi = absolute value of digit + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + xorl %r13d, %r13d + xorl %r14d, %r14d + xorl %r15d, %r15d + + .set I, 1 +.rep 8 + cmpq $I, %rdi + + cmovzq TAB+96*(I-1)(%rsp), %rax + cmovzq TAB+96*(I-1)+8(%rsp), %rbx + cmovzq TAB+96*(I-1)+16(%rsp), %rcx + cmovzq TAB+96*(I-1)+24(%rsp), %rdx + cmovzq TAB+96*(I-1)+32(%rsp), %r8 + cmovzq TAB+96*(I-1)+40(%rsp), %r9 + cmovzq TAB+96*(I-1)+48(%rsp), %r10 + cmovzq TAB+96*(I-1)+56(%rsp), %r11 + cmovzq TAB+96*(I-1)+64(%rsp), %r12 + cmovzq TAB+96*(I-1)+72(%rsp), %r13 + cmovzq TAB+96*(I-1)+80(%rsp), %r14 + cmovzq TAB+96*(I-1)+88(%rsp), %r15 + .set I, (I+1) +.endr + + movq %r12, TABENT+64(%rsp) + movq %r13, TABENT+72(%rsp) + movq %r14, TABENT+80(%rsp) + movq %r15, TABENT+88(%rsp) + + xorl %r14d, %r14d + leaq -1(%r14), %r12 + movq $0x00000000ffffffff, %r15 + movq %r15, %r13 + negq %r15 + + subq %r8, %r12 + sbbq %r9, %r13 + sbbq %r10, %r14 + sbbq %r11, %r15 + + testq %rsi, %rsi + cmovnzq %r12, %r8 + cmovnzq %r13, %r9 + cmovnzq %r14, %r10 + cmovnzq %r15, %r11 + + movq %rax, TABENT(%rsp) + movq %rbx, TABENT+8(%rsp) + movq %rcx, TABENT+16(%rsp) + movq %rdx, TABENT+24(%rsp) + + movq %r8, TABENT+32(%rsp) + movq %r9, TABENT+40(%rsp) + movq %r10, TABENT+48(%rsp) + movq %r11, TABENT+56(%rsp) + + leaq TABENT(%rsp), %rdx + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp256_scalarmul_local_p256_montjadd) + + testq %rbp, %rbp + jne Lp256_scalarmul_loop + +// Let z2 = 1/z^2 and z3 = 1/z^3, both without Montgomery form + + leaq Z2(%rsp), %rdi + leaq ACC+64(%rsp), %rsi + CFI_CALL(Lp256_scalarmul_local_montsqr_p256) + + leaq Z3(%rsp), %rdi + leaq ACC+64(%rsp), %rsi + leaq Z2(%rsp), %rdx + CFI_CALL(Lp256_scalarmul_local_montmul_p256) + + leaq Z2(%rsp), %rdi + leaq Z3(%rsp), %rsi + CFI_CALL(Lp256_scalarmul_local_demont_p256) + + leaq Z3(%rsp), %rdi + leaq Z2(%rsp), %rsi + CFI_CALL(Lp256_scalarmul_local_inv_p256) + + leaq Z2(%rsp), %rdi + leaq ACC+64(%rsp), %rsi + leaq Z3(%rsp), %rdx + CFI_CALL(Lp256_scalarmul_local_montmul_p256) + +// Convert back from Jacobian (X, Y, Z) |-> (X/Z^2, Y/Z^3) + + movq res, %rdi + leaq ACC(%rsp), %rsi + leaq Z2(%rsp), %rdx + movq %rdi, %rbx + CFI_CALL(Lp256_scalarmul_local_montmul_p256) + + leaq 32(%rbx), %rdi + leaq ACC+32(%rsp), %rsi + leaq Z3(%rsp), %rdx + CFI_CALL(Lp256_scalarmul_local_montmul_p256) + +// Restore stack and registers and return + + CFI_INC_RSP(NSPACE) + CFI_POP(%rbx) + CFI_POP(%rbp) + CFI_POP(%r12) + CFI_POP(%r13) + CFI_POP(%r14) + CFI_POP(%r15) + CFI_RET + +#if WINDOWS_ABI +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_standard) +#else +S2N_BN_SIZE_DIRECTIVE(p256_scalarmul) +#endif + +// Local copies of subroutines, complete clones at the moment + +Lp256_scalarmul_local_demont_p256: + CFI_START + CFI_PUSH(%rbx) + movq (%rsi), %r8 + movq 0x8(%rsi), %r9 + movq 0x10(%rsi), %r10 + movq 0x18(%rsi), %r11 + xorq %rbx, %rbx + xorq %rsi, %rsi + movq $0x100000000, %rdx + mulxq %r8, %rax, %rcx + adcxq %rax, %r9 + adoxq %rcx, %r10 + mulxq %r9, %rax, %rcx + adcxq %rax, %r10 + adoxq %rcx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rcx + adcxq %rax, %r11 + adoxq %rcx, %rbx + mulxq %r9, %rax, %rcx + adcxq %rax, %rbx + adoxq %rcx, %rsi + movl $0x0, %r8d + adcxq %r8, %rsi + xorq %r9, %r9 + movq $0x100000000, %rdx + mulxq %r10, %rax, %rcx + adcxq %rax, %r11 + adoxq %rcx, %rbx + mulxq %r11, %rax, %rcx + adcxq %rax, %rbx + adoxq %rcx, %rsi + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rcx + adcxq %rax, %rsi + adoxq %rcx, %r8 + mulxq %r11, %rax, %rcx + adcxq %rax, %r8 + adoxq %rcx, %r9 + movl $0x0, %r10d + adcxq %r10, %r9 + movq %rbx, (%rdi) + movq %rsi, 0x8(%rdi) + movq %r8, 0x10(%rdi) + movq %r9, 0x18(%rdi) + CFI_POP(%rbx) + CFI_RET + +Lp256_scalarmul_local_inv_p256: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(240) + movq %rdi, 0xe0(%rsp) + xorl %ecx, %ecx + movl $0xffffffff, %edx + movq %rdx, %rbx + leaq -0x1(%rcx), %rax + negq %rdx + movq %rax, (%rsp) + movq %rbx, 0x8(%rsp) + movq %rcx, 0x10(%rsp) + movq %rdx, 0x18(%rsp) + movq %rcx, 0x20(%rsp) + movq (%rsi), %r8 + movq 0x8(%rsi), %r9 + movq 0x10(%rsi), %r10 + movq 0x18(%rsi), %r11 + leaq 0x1(%rcx), %rax + addq %r8, %rax + leaq -0x1(%rdx), %rbx + adcq %r9, %rbx + notq %rcx + adcq %r10, %rcx + notq %rdx + adcq %r11, %rdx + cmovaeq %r8, %rax + cmovaeq %r9, %rbx + cmovaeq %r10, %rcx + cmovaeq %r11, %rdx + movq %rax, 0x28(%rsp) + movq %rbx, 0x30(%rsp) + movq %rcx, 0x38(%rsp) + movq %rdx, 0x40(%rsp) + xorl %eax, %eax + movq %rax, 0x48(%rsp) + xorl %eax, %eax + movq %rax, 0x50(%rsp) + movq %rax, 0x58(%rsp) + movq %rax, 0x60(%rsp) + movq %rax, 0x68(%rsp) + movq $0x4000000000000, %rcx + movq %rcx, 0x78(%rsp) + movq %rax, 0x80(%rsp) + movq %rax, 0x88(%rsp) + movq %rax, 0x90(%rsp) + movq $0xa, 0xb0(%rsp) + movq $0x1, 0xb8(%rsp) + jmp Lp256_scalarmul_inv_midloop +Lp256_scalarmul_inv_loop: + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + movq %r8, %rax + andq %r9, %rax + movq %r10, %rdi + andq %r11, %rdi + addq %rax, %rdi + movq %rdi, 0xa0(%rsp) + movq %r12, %rax + andq %r13, %rax + movq %r14, %rsi + andq %r15, %rsi + addq %rax, %rsi + movq %rsi, 0xa8(%rsp) + xorl %ebx, %ebx + movq (%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x28(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rdi + adcq %rdx, %rbx + xorl %ebp, %ebp + movq (%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x28(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + xorl %ecx, %ecx + movq 0x8(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x30(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + shrdq $0x3b, %rbx, %rdi + movq %rdi, (%rsp) + xorl %edi, %edi + movq 0x8(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rbp + adcq %rdx, %rdi + movq 0x30(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rdi + shrdq $0x3b, %rbp, %rsi + movq %rsi, 0x28(%rsp) + xorl %esi, %esi + movq 0x10(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rsi + movq 0x38(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rsi + shrdq $0x3b, %rcx, %rbx + movq %rbx, 0x8(%rsp) + xorl %ebx, %ebx + movq 0x10(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x38(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rdi + adcq %rdx, %rbx + shrdq $0x3b, %rdi, %rbp + movq %rbp, 0x30(%rsp) + movq 0x18(%rsp), %rax + xorq %r9, %rax + movq 0x20(%rsp), %rbp + xorq %r9, %rbp + andq %r8, %rbp + negq %rbp + mulq %r8 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x40(%rsp), %rax + xorq %r11, %rax + movq 0x48(%rsp), %rdx + xorq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbp + mulq %r10 + addq %rax, %rsi + adcq %rdx, %rbp + shrdq $0x3b, %rsi, %rcx + movq %rcx, 0x10(%rsp) + shrdq $0x3b, %rbp, %rsi + sarq $0x3b, %rbp + movq 0x18(%rsp), %rax + movq %rsi, 0x18(%rsp) + movq 0x20(%rsp), %rsi + movq %rbp, 0x20(%rsp) + xorq %r13, %rax + xorq %r13, %rsi + andq %r12, %rsi + negq %rsi + mulq %r12 + addq %rax, %rbx + adcq %rdx, %rsi + movq 0x40(%rsp), %rax + xorq %r15, %rax + movq 0x48(%rsp), %rdx + xorq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rsi + mulq %r14 + addq %rax, %rbx + adcq %rdx, %rsi + shrdq $0x3b, %rbx, %rdi + movq %rdi, 0x38(%rsp) + shrdq $0x3b, %rsi, %rbx + movq %rbx, 0x40(%rsp) + sarq $0x3b, %rsi + movq %rsi, 0x48(%rsp) + movq 0xa0(%rsp), %rbx + movq 0xa8(%rsp), %rbp + xorl %ecx, %ecx + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x78(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x50(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x50(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x78(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x78(%rsp) + xorl %ebx, %ebx + movq 0x58(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x80(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + xorl %ebp, %ebp + movq 0x58(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, 0x58(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x80(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, 0x80(%rsp) + xorl %ecx, %ecx + movq 0x60(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x88(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x60(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x60(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x88(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x88(%rsp) + movq 0x68(%rsp), %rax + xorq %r9, %rax + movq %r9, %rbx + andq %r8, %rbx + negq %rbx + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x90(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbx + mulq %r10 + addq %rax, %rcx + adcq %rbx, %rdx + movq 0x68(%rsp), %rax + movq %rcx, 0x68(%rsp) + movq %rdx, 0x70(%rsp) + xorq %r13, %rax + movq %r13, %rcx + andq %r12, %rcx + negq %rcx + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rcx + movq 0x90(%rsp), %rax + xorq %r15, %rax + movq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rcx + mulq %r14 + addq %rax, %rsi + adcq %rcx, %rdx + movq %rsi, 0x90(%rsp) + movq %rdx, 0x98(%rsp) + movq $0xe000000000000000, %r8 + addq 0x50(%rsp), %r8 + movq $0xffffffffffffffff, %r9 + adcq 0x58(%rsp), %r9 + movq $0x1fffffff, %r10 + adcq 0x60(%rsp), %r10 + movq $0x2000000000000000, %r11 + adcq 0x68(%rsp), %r11 + movq $0x1fffffffe0000000, %r12 + adcq 0x70(%rsp), %r12 + movq %r8, %rbx + shlq $0x20, %rbx + movq $0xffffffff00000001, %rax + mulq %r8 + shrq $0x20, %r8 + addq %rbx, %r9 + adcq %r8, %r10 + adcq %rax, %r11 + adcq %rdx, %r12 + sbbq %rax, %rax + movl $0xffffffff, %ebx + andq %rax, %rbx + movq $0xffffffff00000001, %rdx + andq %rax, %rdx + subq %rax, %r9 + movq %r9, 0x50(%rsp) + sbbq %rbx, %r10 + movq %r10, 0x58(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x60(%rsp) + sbbq %rdx, %r12 + movq %r12, 0x68(%rsp) + movq $0xe000000000000000, %r8 + addq 0x78(%rsp), %r8 + movq $0xffffffffffffffff, %r9 + adcq 0x80(%rsp), %r9 + movq $0x1fffffff, %r10 + adcq 0x88(%rsp), %r10 + movq $0x2000000000000000, %r11 + adcq 0x90(%rsp), %r11 + movq $0x1fffffffe0000000, %r12 + adcq 0x98(%rsp), %r12 + movq %r8, %rbx + shlq $0x20, %rbx + movq $0xffffffff00000001, %rax + mulq %r8 + shrq $0x20, %r8 + addq %rbx, %r9 + adcq %r8, %r10 + adcq %rax, %r11 + adcq %rdx, %r12 + sbbq %rax, %rax + movl $0xffffffff, %ebx + andq %rax, %rbx + movq $0xffffffff00000001, %rdx + andq %rax, %rdx + subq %rax, %r9 + movq %r9, 0x78(%rsp) + sbbq %rbx, %r10 + movq %r10, 0x80(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x88(%rsp) + sbbq %rdx, %r12 + movq %r12, 0x90(%rsp) +Lp256_scalarmul_inv_midloop: + movq 0xb8(%rsp), %rsi + movq (%rsp), %rdx + movq 0x28(%rsp), %rcx + movq %rdx, %rbx + andq $0xfffff, %rbx + movq $0xfffffe0000000000, %rax + orq %rax, %rbx + andq $0xfffff, %rcx + movq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + xorl %ebp, %ebp + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %rdx + leaq (%rcx,%rax), %rdi + shlq $0x16, %rdx + shlq $0x16, %rdi + sarq $0x2b, %rdx + sarq $0x2b, %rdi + movq $0x20000100000, %rax + leaq (%rbx,%rax), %rbx + leaq (%rcx,%rax), %rcx + sarq $0x2a, %rbx + sarq $0x2a, %rcx + movq %rdx, 0xc0(%rsp) + movq %rbx, 0xc8(%rsp) + movq %rdi, 0xd0(%rsp) + movq %rcx, 0xd8(%rsp) + movq (%rsp), %r12 + imulq %r12, %rdi + imulq %rdx, %r12 + movq 0x28(%rsp), %r13 + imulq %r13, %rbx + imulq %rcx, %r13 + addq %rbx, %r12 + addq %rdi, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r10 + shlq $0x16, %r8 + shlq $0x16, %r10 + sarq $0x2b, %r8 + sarq $0x2b, %r10 + movq $0x20000100000, %rax + leaq (%rbx,%rax), %r15 + leaq (%rcx,%rax), %r11 + sarq $0x2a, %r15 + sarq $0x2a, %r11 + movq %r13, %rbx + movq %r12, %rcx + imulq %r8, %r12 + imulq %r15, %rbx + addq %rbx, %r12 + imulq %r11, %r13 + imulq %r10, %rcx + addq %rcx, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movq $0xc000000000000000, %rax + orq %rax, %rcx + movq 0xc0(%rsp), %rax + imulq %r8, %rax + movq 0xd0(%rsp), %rdx + imulq %r15, %rdx + imulq 0xc8(%rsp), %r8 + imulq 0xd8(%rsp), %r15 + addq %r8, %r15 + leaq (%rax,%rdx), %r9 + movq 0xc0(%rsp), %rax + imulq %r10, %rax + movq 0xd0(%rsp), %rdx + imulq %r11, %rdx + imulq 0xc8(%rsp), %r10 + imulq 0xd8(%rsp), %r11 + addq %r10, %r11 + leaq (%rax,%rdx), %r13 + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r12 + shlq $0x15, %r8 + shlq $0x15, %r12 + sarq $0x2b, %r8 + sarq $0x2b, %r12 + movq $0x20000100000, %rax + leaq (%rbx,%rax), %r10 + leaq (%rcx,%rax), %r14 + sarq $0x2b, %r10 + sarq $0x2b, %r14 + movq %r9, %rax + imulq %r8, %rax + movq %r13, %rdx + imulq %r10, %rdx + imulq %r15, %r8 + imulq %r11, %r10 + addq %r8, %r10 + leaq (%rax,%rdx), %r8 + movq %r9, %rax + imulq %r12, %rax + movq %r13, %rdx + imulq %r14, %rdx + imulq %r15, %r12 + imulq %r11, %r14 + addq %r12, %r14 + leaq (%rax,%rdx), %r12 + movq %rsi, 0xb8(%rsp) + decq 0xb0(%rsp) + jne Lp256_scalarmul_inv_loop + movq (%rsp), %rax + movq 0x28(%rsp), %rcx + imulq %r8, %rax + imulq %r10, %rcx + addq %rcx, %rax + sarq $0x3f, %rax + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + xorq %rax, %r9 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + xorq %rax, %r11 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + xorq %rax, %r13 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + xorq %rax, %r15 + movq %r8, %rax + andq %r9, %rax + movq %r10, %r12 + andq %r11, %r12 + addq %rax, %r12 + xorl %r13d, %r13d + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x78(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movq 0x58(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r13 + adcq %rdx, %r14 + movq 0x80(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq 0x60(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x88(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x68(%rsp), %rax + xorq %r9, %rax + andq %r8, %r9 + negq %r9 + mulq %r8 + addq %rax, %r15 + adcq %rdx, %r9 + movq 0x90(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %r9 + mulq %r10 + addq %rax, %r15 + adcq %rdx, %r9 + movq %r12, 0x50(%rsp) + movq %r13, 0x58(%rsp) + movq %r14, 0x60(%rsp) + movq %r15, 0x68(%rsp) + movq %r9, 0x70(%rsp) + movq $0xe000000000000000, %r8 + addq 0x50(%rsp), %r8 + movq $0xffffffffffffffff, %r9 + adcq 0x58(%rsp), %r9 + movq $0x1fffffff, %r10 + adcq 0x60(%rsp), %r10 + movq $0x2000000000000000, %r11 + adcq 0x68(%rsp), %r11 + movq $0x1fffffffe0000000, %r12 + adcq 0x70(%rsp), %r12 + movq %r8, %rbx + shlq $0x20, %rbx + movq $0xffffffff00000001, %rax + mulq %r8 + shrq $0x20, %r8 + addq %rbx, %r9 + adcq %r8, %r10 + adcq %rax, %r11 + adcq %rdx, %r12 + sbbq %rax, %rax + movl $0xffffffff, %ebx + andq %rax, %rbx + movq $0xffffffff00000001, %rdx + andq %rax, %rdx + subq %rax, %r9 + movq %r9, 0x50(%rsp) + sbbq %rbx, %r10 + movq %r10, 0x58(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x60(%rsp) + sbbq %rdx, %r12 + movq %r12, 0x68(%rsp) + movq 0x50(%rsp), %r8 + movq 0x58(%rsp), %r9 + movq 0x60(%rsp), %r10 + movq 0x68(%rsp), %r11 + movl $0x1, %eax + movl $0xffffffff, %ebx + leaq -0x2(%rax), %rcx + leaq -0x1(%rbx), %rdx + notq %rbx + addq %r8, %rax + adcq %r9, %rbx + adcq %r10, %rcx + adcq %r11, %rdx + cmovaeq %r8, %rax + cmovaeq %r9, %rbx + cmovaeq %r10, %rcx + cmovaeq %r11, %rdx + movq 0xe0(%rsp), %rdi + movq %rax, (%rdi) + movq %rbx, 0x8(%rdi) + movq %rcx, 0x10(%rdi) + movq %rdx, 0x18(%rdi) + CFI_INC_RSP(240) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +Lp256_scalarmul_local_montmul_p256: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + movq %rdx, %rcx + xorl %r13d, %r13d + movq (%rcx), %rdx + mulxq (%rsi), %r8, %r9 + mulxq 0x8(%rsi), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x10(%rsi), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x18(%rsi), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x8(%rcx), %rdx + xorl %r14d, %r14d + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x10(%rcx), %rdx + xorl %r8d, %r8d + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x18(%rsi), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x18(%rcx), %rdx + xorl %r9d, %r9d + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x18(%rsi), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + CFI_RET + +Lp256_scalarmul_local_montsqr_p256: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + movq (%rsi), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0x8(%rsi), %r9, %r10 + mulxq 0x18(%rsi), %r11, %r12 + movq 0x10(%rsi), %rdx + mulxq 0x18(%rsi), %r13, %r14 + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x18(%rsi), %rdx + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rbp, %r13 + adoxq %rbp, %r14 + adcq %rbp, %r14 + xorl %ebp, %ebp + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0x8(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x10(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x18(%rsi), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rbp, %r15 + adoxq %rbp, %r15 + xorl %ebp, %ebp + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rbp, %r13 + movl %ebp, %r9d + adoxq %rbp, %r9 + adcxq %rbp, %r9 + addq %r9, %r14 + adcq %rbp, %r15 + movl %ebp, %r8d + adcq %rbp, %r8 + xorl %ebp, %ebp + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %r8 + adcq %rbp, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rdx), %rdx + adcq %r13, %rdx + leaq -0x1(%rbp), %rbp + movq %rbp, %rax + adcq %r14, %rbp + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %rbp, %r14 + cmovbq %r11, %r15 + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +Lp256_scalarmul_local_tomont_p256: + CFI_START + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + xorq %r13, %r13 + movl $0x3, %edx + mulxq (%rsi), %r8, %r9 + mulxq 0x8(%rsi), %rcx, %r10 + adcxq %rcx, %r9 + mulxq 0x10(%rsi), %rcx, %r11 + adcxq %rcx, %r10 + mulxq 0x18(%rsi), %rcx, %r12 + adcxq %rcx, %r11 + adcxq %r13, %r12 + movq $0xfffffffbffffffff, %rdx + xorq %r14, %r14 + mulxq (%rsi), %rax, %rcx + adcxq %rax, %r9 + adoxq %rcx, %r10 + mulxq 0x8(%rsi), %rax, %rcx + adcxq %rax, %r10 + adoxq %rcx, %r11 + mulxq 0x10(%rsi), %rax, %rcx + adcxq %rax, %r11 + adoxq %rcx, %r12 + mulxq 0x18(%rsi), %rax, %rcx + adcxq %rax, %r12 + adoxq %rcx, %r13 + adcq %r14, %r13 + xorq %r15, %r15 + movq $0x100000000, %rdx + mulxq %r8, %rax, %rcx + adcxq %rax, %r9 + adoxq %rcx, %r10 + mulxq %r9, %rax, %rcx + adcxq %rax, %r10 + adoxq %rcx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rcx + adcxq %rax, %r11 + adoxq %rcx, %r12 + mulxq %r9, %rax, %rcx + adcxq %rax, %r12 + adoxq %rcx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcxq %r15, %r14 + movq $0xfffffffffffffffe, %rdx + xorq %r8, %r8 + mulxq (%rsi), %rax, %rcx + adcxq %rax, %r10 + adoxq %rcx, %r11 + mulxq 0x8(%rsi), %rax, %rcx + adcxq %rax, %r11 + adoxq %rcx, %r12 + mulxq 0x10(%rsi), %rax, %rcx + adcxq %rax, %r12 + adoxq %rcx, %r13 + mulxq 0x18(%rsi), %rax, %rcx + adcxq %rax, %r13 + adoxq %rcx, %r14 + adcxq %r8, %r14 + adoxq %r8, %r15 + adcxq %r8, %r15 + movq $0x4fffffffd, %rdx + xorq %r9, %r9 + mulxq (%rsi), %rax, %rcx + adcxq %rax, %r11 + adoxq %rcx, %r12 + mulxq 0x8(%rsi), %rax, %rcx + adcxq %rax, %r12 + adoxq %rcx, %r13 + mulxq 0x10(%rsi), %rax, %rcx + adcxq %rax, %r13 + adoxq %rcx, %r14 + mulxq 0x18(%rsi), %rax, %rcx + adcxq %rax, %r14 + adoxq %rcx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcxq %r9, %r8 + xorq %r9, %r9 + movq $0x100000000, %rdx + mulxq %r10, %rax, %rcx + adcxq %rax, %r11 + adoxq %rcx, %r12 + mulxq %r11, %rax, %rcx + adcxq %rax, %r12 + adoxq %rcx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rcx + adcxq %rax, %r13 + adoxq %rcx, %r14 + mulxq %r11, %rax, %rcx + adcxq %rax, %r14 + adoxq %rcx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcxq %r9, %r8 + movl $0xffffffff, %edx + movq $0xffffffff00000001, %rcx + movq $0xfffffffffffffffe, %rax + subq %r12, %rax + movq %rdx, %rax + sbbq %r13, %rax + movl $0x0, %eax + sbbq %r14, %rax + movq %rcx, %rax + sbbq %r15, %rax + movl $0x0, %eax + sbbq %r8, %rax + andq %rax, %rdx + andq %rax, %rcx + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq $0x0, %r14 + sbbq %rcx, %r15 + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_RET + +Lp256_scalarmul_local_p256_montjadd: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(224) + movq %rdx, %rbp + movq 0x40(%rsi), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0x48(%rsi), %r9, %r10 + mulxq 0x58(%rsi), %r11, %r12 + movq 0x50(%rsi), %rdx + mulxq 0x58(%rsi), %r13, %r14 + xorl %ecx, %ecx + mulxq 0x40(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x58(%rsi), %rdx + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + adoxq %rcx, %r14 + adcq %rcx, %r14 + xorl %ecx, %ecx + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0x48(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x50(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x58(%rsi), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rcx, %r15 + adoxq %rcx, %r15 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + movl %ecx, %r9d + adoxq %rcx, %r9 + adcxq %rcx, %r9 + addq %r9, %r14 + adcq %rcx, %r15 + movl %ecx, %r8d + adcq %rcx, %r8 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rcx, %r15 + adoxq %rcx, %r8 + adcq %rcx, %r8 + movl $0x1, %r8d + leaq -0x1(%rdx), %rdx + leaq -0x1(%rcx), %rax + movl $0xfffffffe, %r11d + cmoveq %rcx, %r8 + cmoveq %rcx, %rdx + cmoveq %rcx, %rax + cmoveq %rcx, %r11 + addq %r8, %r12 + adcq %rdx, %r13 + adcq %rax, %r14 + adcq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + movq 0x40(%rbp), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0x48(%rbp), %r9, %r10 + mulxq 0x58(%rbp), %r11, %r12 + movq 0x50(%rbp), %rdx + mulxq 0x58(%rbp), %r13, %r14 + xorl %ecx, %ecx + mulxq 0x40(%rbp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x48(%rbp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x58(%rbp), %rdx + mulxq 0x48(%rbp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + adoxq %rcx, %r14 + adcq %rcx, %r14 + xorl %ecx, %ecx + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0x48(%rbp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x50(%rbp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x58(%rbp), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rcx, %r15 + adoxq %rcx, %r15 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + movl %ecx, %r9d + adoxq %rcx, %r9 + adcxq %rcx, %r9 + addq %r9, %r14 + adcq %rcx, %r15 + movl %ecx, %r8d + adcq %rcx, %r8 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rcx, %r15 + adoxq %rcx, %r8 + adcq %rcx, %r8 + movl $0x1, %r8d + leaq -0x1(%rdx), %rdx + leaq -0x1(%rcx), %rax + movl $0xfffffffe, %r11d + cmoveq %rcx, %r8 + cmoveq %rcx, %rdx + cmoveq %rcx, %rax + cmoveq %rcx, %r11 + addq %r8, %r12 + adcq %rdx, %r13 + adcq %rax, %r14 + adcq %r11, %r15 + movq %r12, 0xa0(%rsp) + movq %r13, 0xa8(%rsp) + movq %r14, 0xb0(%rsp) + movq %r15, 0xb8(%rsp) + xorl %r13d, %r13d + movq 0x20(%rsi), %rdx + mulxq 0x40(%rbp), %r8, %r9 + mulxq 0x48(%rbp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x50(%rbp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x58(%rbp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x28(%rsi), %rdx + xorl %r14d, %r14d + mulxq 0x40(%rbp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x48(%rbp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x50(%rbp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x58(%rbp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x30(%rsi), %rdx + xorl %r8d, %r8d + mulxq 0x40(%rbp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x48(%rbp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x50(%rbp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x58(%rbp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x38(%rsi), %rdx + xorl %r9d, %r9d + mulxq 0x40(%rbp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x48(%rbp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x50(%rbp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x58(%rbp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0xc0(%rsp) + movq %r13, 0xc8(%rsp) + movq %r14, 0xd0(%rsp) + movq %r15, 0xd8(%rsp) + xorl %r13d, %r13d + movq 0x20(%rbp), %rdx + mulxq 0x40(%rsi), %r8, %r9 + mulxq 0x48(%rsi), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x50(%rsi), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x58(%rsi), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x28(%rbp), %rdx + xorl %r14d, %r14d + mulxq 0x40(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x50(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x58(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x30(%rbp), %rdx + xorl %r8d, %r8d + mulxq 0x40(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x50(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x58(%rsi), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x38(%rbp), %rdx + xorl %r9d, %r9d + mulxq 0x40(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x50(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x58(%rsi), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x20(%rsp) + movq %r13, 0x28(%rsp) + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + xorl %r13d, %r13d + movq 0x0(%rbp), %rdx + mulxq (%rsp), %r8, %r9 + mulxq 0x8(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x10(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x18(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x8(%rbp), %rdx + xorl %r14d, %r14d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x10(%rbp), %rdx + xorl %r8d, %r8d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x18(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x18(%rbp), %rdx + xorl %r9d, %r9d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x18(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x40(%rsp) + movq %r13, 0x48(%rsp) + movq %r14, 0x50(%rsp) + movq %r15, 0x58(%rsp) + xorl %r13d, %r13d + movq (%rsi), %rdx + mulxq 0xa0(%rsp), %r8, %r9 + mulxq 0xa8(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0xb0(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0xb8(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x8(%rsi), %rdx + xorl %r14d, %r14d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xb8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x10(%rsi), %rdx + xorl %r8d, %r8d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x18(%rsi), %rdx + xorl %r9d, %r9d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + xorl %r13d, %r13d + movq 0x20(%rsp), %rdx + mulxq (%rsp), %r8, %r9 + mulxq 0x8(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x10(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x18(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x28(%rsp), %rdx + xorl %r14d, %r14d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x30(%rsp), %rdx + xorl %r8d, %r8d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x18(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x38(%rsp), %rdx + xorl %r9d, %r9d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x18(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x20(%rsp) + movq %r13, 0x28(%rsp) + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + xorl %r13d, %r13d + movq 0xc0(%rsp), %rdx + mulxq 0xa0(%rsp), %r8, %r9 + mulxq 0xa8(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0xb0(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0xb8(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0xc8(%rsp), %rdx + xorl %r14d, %r14d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xb8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0xd0(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0xd8(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0xc0(%rsp) + movq %r13, 0xc8(%rsp) + movq %r14, 0xd0(%rsp) + movq %r15, 0xd8(%rsp) + movq 0x40(%rsp), %rax + subq 0x80(%rsp), %rax + movq 0x48(%rsp), %rcx + sbbq 0x88(%rsp), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x90(%rsp), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x98(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0xa0(%rsp) + adcq %r10, %rcx + movq %rcx, 0xa8(%rsp) + adcq $0x0, %r8 + movq %r8, 0xb0(%rsp) + adcq %rdx, %r9 + movq %r9, 0xb8(%rsp) + movq 0x20(%rsp), %rax + subq 0xc0(%rsp), %rax + movq 0x28(%rsp), %rcx + sbbq 0xc8(%rsp), %rcx + movq 0x30(%rsp), %r8 + sbbq 0xd0(%rsp), %r8 + movq 0x38(%rsp), %r9 + sbbq 0xd8(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x20(%rsp) + adcq %r10, %rcx + movq %rcx, 0x28(%rsp) + adcq $0x0, %r8 + movq %r8, 0x30(%rsp) + adcq %rdx, %r9 + movq %r9, 0x38(%rsp) + movq 0xa0(%rsp), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0xa8(%rsp), %r9, %r10 + mulxq 0xb8(%rsp), %r11, %r12 + movq 0xb0(%rsp), %rdx + mulxq 0xb8(%rsp), %r13, %r14 + xorl %ecx, %ecx + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0xb8(%rsp), %rdx + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + adoxq %rcx, %r14 + adcq %rcx, %r14 + xorl %ecx, %ecx + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0xa8(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0xb0(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0xb8(%rsp), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rcx, %r15 + adoxq %rcx, %r15 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + movl %ecx, %r9d + adoxq %rcx, %r9 + adcxq %rcx, %r9 + addq %r9, %r14 + adcq %rcx, %r15 + movl %ecx, %r8d + adcq %rcx, %r8 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rcx, %r15 + adoxq %rcx, %r8 + adcq %rcx, %r8 + movl $0x1, %r8d + leaq -0x1(%rdx), %rdx + leaq -0x1(%rcx), %rax + movl $0xfffffffe, %r11d + cmoveq %rcx, %r8 + cmoveq %rcx, %rdx + cmoveq %rcx, %rax + cmoveq %rcx, %r11 + addq %r8, %r12 + adcq %rdx, %r13 + adcq %rax, %r14 + adcq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + movq 0x20(%rsp), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0x28(%rsp), %r9, %r10 + mulxq 0x38(%rsp), %r11, %r12 + movq 0x30(%rsp), %rdx + mulxq 0x38(%rsp), %r13, %r14 + xorl %ecx, %ecx + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x38(%rsp), %rdx + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + adoxq %rcx, %r14 + adcq %rcx, %r14 + xorl %ecx, %ecx + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0x28(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x30(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x38(%rsp), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rcx, %r15 + adoxq %rcx, %r15 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + movl %ecx, %r9d + adoxq %rcx, %r9 + adcxq %rcx, %r9 + addq %r9, %r14 + adcq %rcx, %r15 + movl %ecx, %r8d + adcq %rcx, %r8 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rcx, %r15 + adoxq %rcx, %r8 + adcq %rcx, %r8 + movl $0x1, %ebx + addq %r12, %rbx + leaq -0x1(%rdx), %rdx + adcq %r13, %rdx + leaq -0x1(%rcx), %rcx + movq %rcx, %rax + adcq %r14, %rcx + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rbx, %r12 + cmovbq %rdx, %r13 + cmovbq %rcx, %r14 + cmovbq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + xorl %r13d, %r13d + movq 0x80(%rsp), %rdx + mulxq 0x60(%rsp), %r8, %r9 + mulxq 0x68(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x70(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x78(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x88(%rsp), %rdx + xorl %r14d, %r14d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x78(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x90(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x98(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + xorl %r13d, %r13d + movq 0x40(%rsp), %rdx + mulxq 0x60(%rsp), %r8, %r9 + mulxq 0x68(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x70(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x78(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x48(%rsp), %rdx + xorl %r14d, %r14d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x78(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x50(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x58(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x40(%rsp) + movq %r13, 0x48(%rsp) + movq %r14, 0x50(%rsp) + movq %r15, 0x58(%rsp) + movq (%rsp), %rax + subq 0x80(%rsp), %rax + movq 0x8(%rsp), %rcx + sbbq 0x88(%rsp), %rcx + movq 0x10(%rsp), %r8 + sbbq 0x90(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x98(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, (%rsp) + adcq %r10, %rcx + movq %rcx, 0x8(%rsp) + adcq $0x0, %r8 + movq %r8, 0x10(%rsp) + adcq %rdx, %r9 + movq %r9, 0x18(%rsp) + movq 0x40(%rsp), %rax + subq 0x80(%rsp), %rax + movq 0x48(%rsp), %rcx + sbbq 0x88(%rsp), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x90(%rsp), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x98(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x60(%rsp) + adcq %r10, %rcx + movq %rcx, 0x68(%rsp) + adcq $0x0, %r8 + movq %r8, 0x70(%rsp) + adcq %rdx, %r9 + movq %r9, 0x78(%rsp) + xorl %r13d, %r13d + movq 0x40(%rsi), %rdx + mulxq 0xa0(%rsp), %r8, %r9 + mulxq 0xa8(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0xb0(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0xb8(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x48(%rsi), %rdx + xorl %r14d, %r14d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xb8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x50(%rsi), %rdx + xorl %r8d, %r8d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x58(%rsi), %rdx + xorl %r9d, %r9d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0xa0(%rsp) + movq %r13, 0xa8(%rsp) + movq %r14, 0xb0(%rsp) + movq %r15, 0xb8(%rsp) + movq (%rsp), %rax + subq 0x40(%rsp), %rax + movq 0x8(%rsp), %rcx + sbbq 0x48(%rsp), %rcx + movq 0x10(%rsp), %r8 + sbbq 0x50(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x58(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, (%rsp) + adcq %r10, %rcx + movq %rcx, 0x8(%rsp) + adcq $0x0, %r8 + movq %r8, 0x10(%rsp) + adcq %rdx, %r9 + movq %r9, 0x18(%rsp) + movq 0x80(%rsp), %rax + subq (%rsp), %rax + movq 0x88(%rsp), %rcx + sbbq 0x8(%rsp), %rcx + movq 0x90(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x98(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x80(%rsp) + adcq %r10, %rcx + movq %rcx, 0x88(%rsp) + adcq $0x0, %r8 + movq %r8, 0x90(%rsp) + adcq %rdx, %r9 + movq %r9, 0x98(%rsp) + xorl %r13d, %r13d + movq 0xc0(%rsp), %rdx + mulxq 0x60(%rsp), %r8, %r9 + mulxq 0x68(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x70(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x78(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0xc8(%rsp), %rdx + xorl %r14d, %r14d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x78(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0xd0(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0xd8(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + xorl %r13d, %r13d + movq 0x40(%rbp), %rdx + mulxq 0xa0(%rsp), %r8, %r9 + mulxq 0xa8(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0xb0(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0xb8(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x48(%rbp), %rdx + xorl %r14d, %r14d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xb8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x50(%rbp), %rdx + xorl %r8d, %r8d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x58(%rbp), %rdx + xorl %r9d, %r9d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0xa0(%rsp) + movq %r13, 0xa8(%rsp) + movq %r14, 0xb0(%rsp) + movq %r15, 0xb8(%rsp) + xorl %r13d, %r13d + movq 0x80(%rsp), %rdx + mulxq 0x20(%rsp), %r8, %r9 + mulxq 0x28(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x30(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x38(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x88(%rsp), %rdx + xorl %r14d, %r14d + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x38(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x90(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x38(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x98(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x38(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq 0x80(%rsp), %rax + subq 0x60(%rsp), %rax + movq 0x88(%rsp), %rcx + sbbq 0x68(%rsp), %rcx + movq 0x90(%rsp), %r8 + sbbq 0x70(%rsp), %r8 + movq 0x98(%rsp), %r9 + sbbq 0x78(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x80(%rsp) + adcq %r10, %rcx + movq %rcx, 0x88(%rsp) + adcq $0x0, %r8 + movq %r8, 0x90(%rsp) + adcq %rdx, %r9 + movq %r9, 0x98(%rsp) + movq 0x40(%rsi), %r8 + movq 0x48(%rsi), %r9 + movq 0x50(%rsi), %r10 + movq 0x58(%rsi), %r11 + movq %r8, %rax + movq %r9, %rdx + orq %r10, %rax + orq %r11, %rdx + orq %rdx, %rax + negq %rax + sbbq %rax, %rax + movq 0x40(%rbp), %r12 + movq 0x48(%rbp), %r13 + movq 0x50(%rbp), %r14 + movq 0x58(%rbp), %r15 + movq %r12, %rbx + movq %r13, %rdx + orq %r14, %rbx + orq %r15, %rdx + orq %rdx, %rbx + negq %rbx + sbbq %rbx, %rbx + cmpq %rax, %rbx + cmovbq %r8, %r12 + cmovbq %r9, %r13 + cmovbq %r10, %r14 + cmovbq %r11, %r15 + cmoveq 0xa0(%rsp), %r12 + cmoveq 0xa8(%rsp), %r13 + cmoveq 0xb0(%rsp), %r14 + cmoveq 0xb8(%rsp), %r15 + movq (%rsp), %rax + cmovbq (%rsi), %rax + cmova 0x0(%rbp), %rax + movq 0x8(%rsp), %rbx + cmovbq 0x8(%rsi), %rbx + cmova 0x8(%rbp), %rbx + movq 0x10(%rsp), %rcx + cmovbq 0x10(%rsi), %rcx + cmova 0x10(%rbp), %rcx + movq 0x18(%rsp), %rdx + cmovbq 0x18(%rsi), %rdx + cmova 0x18(%rbp), %rdx + movq 0x80(%rsp), %r8 + cmovbq 0x20(%rsi), %r8 + cmova 0x20(%rbp), %r8 + movq 0x88(%rsp), %r9 + cmovbq 0x28(%rsi), %r9 + cmova 0x28(%rbp), %r9 + movq 0x90(%rsp), %r10 + cmovbq 0x30(%rsi), %r10 + cmova 0x30(%rbp), %r10 + movq 0x98(%rsp), %r11 + cmovbq 0x38(%rsi), %r11 + cmova 0x38(%rbp), %r11 + movq %rax, (%rdi) + movq %rbx, 0x8(%rdi) + movq %rcx, 0x10(%rdi) + movq %rdx, 0x18(%rdi) + movq %r8, 0x20(%rdi) + movq %r9, 0x28(%rdi) + movq %r10, 0x30(%rdi) + movq %r11, 0x38(%rdi) + movq %r12, 0x40(%rdi) + movq %r13, 0x48(%rdi) + movq %r14, 0x50(%rdi) + movq %r15, 0x58(%rdi) + CFI_INC_RSP(224) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +Lp256_scalarmul_local_p256_montjdouble: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(192) + movq 0x40(%rsi), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0x48(%rsi), %r9, %r10 + mulxq 0x58(%rsi), %r11, %r12 + movq 0x50(%rsi), %rdx + mulxq 0x58(%rsi), %r13, %r14 + xorl %ebp, %ebp + mulxq 0x40(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x58(%rsi), %rdx + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rbp, %r13 + adoxq %rbp, %r14 + adcq %rbp, %r14 + xorl %ebp, %ebp + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0x48(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x50(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x58(%rsi), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rbp, %r15 + adoxq %rbp, %r15 + xorl %ebp, %ebp + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rbp, %r13 + movl %ebp, %r9d + adoxq %rbp, %r9 + adcxq %rbp, %r9 + addq %r9, %r14 + adcq %rbp, %r15 + movl %ebp, %r8d + adcq %rbp, %r8 + xorl %ebp, %ebp + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %r8 + adcq %rbp, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rdx), %rdx + adcq %r13, %rdx + leaq -0x1(%rbp), %rbp + movq %rbp, %rax + adcq %r14, %rbp + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %rbp, %r14 + cmovbq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + movq 0x20(%rsi), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0x28(%rsi), %r9, %r10 + mulxq 0x38(%rsi), %r11, %r12 + movq 0x30(%rsi), %rdx + mulxq 0x38(%rsi), %r13, %r14 + xorl %ebp, %ebp + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x38(%rsi), %rdx + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rbp, %r13 + adoxq %rbp, %r14 + adcq %rbp, %r14 + xorl %ebp, %ebp + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0x28(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x30(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x38(%rsi), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rbp, %r15 + adoxq %rbp, %r15 + xorl %ebp, %ebp + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rbp, %r13 + movl %ebp, %r9d + adoxq %rbp, %r9 + adcxq %rbp, %r9 + addq %r9, %r14 + adcq %rbp, %r15 + movl %ebp, %r8d + adcq %rbp, %r8 + xorl %ebp, %ebp + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %r8 + adcq %rbp, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rdx), %rdx + adcq %r13, %rdx + leaq -0x1(%rbp), %rbp + movq %rbp, %rax + adcq %r14, %rbp + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %rbp, %r14 + cmovbq %r11, %r15 + movq %r12, 0x20(%rsp) + movq %r13, 0x28(%rsp) + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq (%rsi), %rax + subq (%rsp), %rax + movq 0x8(%rsi), %rcx + sbbq 0x8(%rsp), %rcx + movq 0x10(%rsi), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x18(%rsi), %r9 + sbbq 0x18(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x60(%rsp) + adcq %r10, %rcx + movq %rcx, 0x68(%rsp) + adcq $0x0, %r8 + movq %r8, 0x70(%rsp) + adcq %rdx, %r9 + movq %r9, 0x78(%rsp) + movq (%rsi), %rax + addq (%rsp), %rax + movq 0x8(%rsi), %rcx + adcq 0x8(%rsp), %rcx + movq 0x10(%rsi), %r8 + adcq 0x10(%rsp), %r8 + movq 0x18(%rsi), %r9 + adcq 0x18(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + subq %r11, %rax + movq %rax, 0x40(%rsp) + sbbq %r10, %rcx + movq %rcx, 0x48(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x50(%rsp) + sbbq %rdx, %r9 + movq %r9, 0x58(%rsp) + xorl %r13d, %r13d + movq 0x60(%rsp), %rdx + mulxq 0x40(%rsp), %r8, %r9 + mulxq 0x48(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x50(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x58(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x68(%rsp), %rdx + xorl %r14d, %r14d + mulxq 0x40(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x48(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x50(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x58(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x70(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0x40(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x48(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x50(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x58(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x78(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0x40(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x48(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x50(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x58(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + xorq %r11, %r11 + movq 0x20(%rsi), %rax + addq 0x40(%rsi), %rax + movq 0x28(%rsi), %rcx + adcq 0x48(%rsi), %rcx + movq 0x30(%rsi), %r8 + adcq 0x50(%rsi), %r8 + movq 0x38(%rsi), %r9 + adcq 0x58(%rsi), %r9 + adcq %r11, %r11 + subq $0xffffffffffffffff, %rax + movl $0xffffffff, %r10d + sbbq %r10, %rcx + sbbq $0x0, %r8 + movq $0xffffffff00000001, %rdx + sbbq %rdx, %r9 + sbbq $0x0, %r11 + andq %r11, %r10 + andq %r11, %rdx + addq %r11, %rax + movq %rax, 0x40(%rsp) + adcq %r10, %rcx + movq %rcx, 0x48(%rsp) + adcq $0x0, %r8 + movq %r8, 0x50(%rsp) + adcq %rdx, %r9 + movq %r9, 0x58(%rsp) + xorl %r13d, %r13d + movq 0x20(%rsp), %rdx + mulxq (%rsi), %r8, %r9 + mulxq 0x8(%rsi), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x10(%rsi), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x18(%rsi), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x28(%rsp), %rdx + xorl %r14d, %r14d + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x30(%rsp), %rdx + xorl %r8d, %r8d + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x18(%rsi), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x38(%rsp), %rdx + xorl %r9d, %r9d + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x18(%rsi), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq 0x60(%rsp), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0x68(%rsp), %r9, %r10 + mulxq 0x78(%rsp), %r11, %r12 + movq 0x70(%rsp), %rdx + mulxq 0x78(%rsp), %r13, %r14 + xorl %ebp, %ebp + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x78(%rsp), %rdx + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rbp, %r13 + adoxq %rbp, %r14 + adcq %rbp, %r14 + xorl %ebp, %ebp + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0x68(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x70(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x78(%rsp), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rbp, %r15 + adoxq %rbp, %r15 + xorl %ebp, %ebp + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rbp, %r13 + movl %ebp, %r9d + adoxq %rbp, %r9 + adcxq %rbp, %r9 + addq %r9, %r14 + adcq %rbp, %r15 + movl %ebp, %r8d + adcq %rbp, %r8 + xorl %ebp, %ebp + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %r8 + adcq %rbp, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rdx), %rdx + adcq %r13, %rdx + leaq -0x1(%rbp), %rbp + movq %rbp, %rax + adcq %r14, %rbp + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %rbp, %r14 + cmovbq %r11, %r15 + movq %r12, 0xa0(%rsp) + movq %r13, 0xa8(%rsp) + movq %r14, 0xb0(%rsp) + movq %r15, 0xb8(%rsp) + movq 0x40(%rsp), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0x48(%rsp), %r9, %r10 + mulxq 0x58(%rsp), %r11, %r12 + movq 0x50(%rsp), %rdx + mulxq 0x58(%rsp), %r13, %r14 + xorl %ebp, %ebp + mulxq 0x40(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x48(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x58(%rsp), %rdx + mulxq 0x48(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rbp, %r13 + adoxq %rbp, %r14 + adcq %rbp, %r14 + xorl %ebp, %ebp + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0x48(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x50(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x58(%rsp), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rbp, %r15 + adoxq %rbp, %r15 + xorl %ebp, %ebp + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rbp, %r13 + movl %ebp, %r9d + adoxq %rbp, %r9 + adcxq %rbp, %r9 + addq %r9, %r14 + adcq %rbp, %r15 + movl %ebp, %r8d + adcq %rbp, %r8 + xorl %ebp, %ebp + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %r8 + adcq %rbp, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rdx), %rdx + adcq %r13, %rdx + leaq -0x1(%rbp), %rbp + movq %rbp, %rax + adcq %r14, %rbp + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %rbp, %r14 + cmovbq %r11, %r15 + movq %r12, 0x40(%rsp) + movq %r13, 0x48(%rsp) + movq %r14, 0x50(%rsp) + movq %r15, 0x58(%rsp) + movq $0xffffffffffffffff, %r8 + xorl %r10d, %r10d + subq 0xa0(%rsp), %r8 + movq $0xffffffff, %r9 + sbbq 0xa8(%rsp), %r9 + sbbq 0xb0(%rsp), %r10 + movq $0xffffffff00000001, %r11 + sbbq 0xb8(%rsp), %r11 + xorl %r12d, %r12d + movq $0x9, %rdx + mulxq %r8, %r8, %rax + mulxq %r9, %r9, %rcx + addq %rax, %r9 + mulxq %r10, %r10, %rax + adcq %rcx, %r10 + mulxq %r11, %r11, %rcx + adcq %rax, %r11 + adcq %rcx, %r12 + movq $0xc, %rdx + xorl %eax, %eax + mulxq 0x80(%rsp), %rax, %rcx + adcxq %rax, %r8 + adoxq %rcx, %r9 + mulxq 0x88(%rsp), %rax, %rcx + adcxq %rax, %r9 + adoxq %rcx, %r10 + mulxq 0x90(%rsp), %rax, %rcx + adcxq %rax, %r10 + adoxq %rcx, %r11 + mulxq 0x98(%rsp), %rax, %rdx + adcxq %rax, %r11 + adoxq %r12, %rdx + adcq $0x1, %rdx + addq %rdx, %r8 + movq $0x100000000, %rax + mulxq %rax, %rax, %rcx + sbbq $0x0, %rax + sbbq $0x0, %rcx + subq %rax, %r9 + sbbq %rcx, %r10 + movq $0xffffffff00000001, %rax + mulxq %rax, %rax, %rcx + sbbq %rax, %r11 + sbbq %rcx, %rdx + decq %rdx + movl $0xffffffff, %eax + andq %rdx, %rax + xorl %ecx, %ecx + subq %rax, %rcx + addq %rdx, %r8 + movq %r8, 0xa0(%rsp) + adcq %rax, %r9 + movq %r9, 0xa8(%rsp) + adcq $0x0, %r10 + movq %r10, 0xb0(%rsp) + adcq %rcx, %r11 + movq %r11, 0xb8(%rsp) + movq 0x40(%rsp), %rax + subq (%rsp), %rax + movq 0x48(%rsp), %rcx + sbbq 0x8(%rsp), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x40(%rsp) + adcq %r10, %rcx + movq %rcx, 0x48(%rsp) + adcq $0x0, %r8 + movq %r8, 0x50(%rsp) + adcq %rdx, %r9 + movq %r9, 0x58(%rsp) + movq 0x20(%rsp), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0x28(%rsp), %r9, %r10 + mulxq 0x38(%rsp), %r11, %r12 + movq 0x30(%rsp), %rdx + mulxq 0x38(%rsp), %r13, %r14 + xorl %ebp, %ebp + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x38(%rsp), %rdx + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rbp, %r13 + adoxq %rbp, %r14 + adcq %rbp, %r14 + xorl %ebp, %ebp + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0x28(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x30(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x38(%rsp), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rbp, %r15 + adoxq %rbp, %r15 + xorl %ebp, %ebp + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rbp, %r13 + movl %ebp, %r9d + adoxq %rbp, %r9 + adcxq %rbp, %r9 + addq %r9, %r14 + adcq %rbp, %r15 + movl %ebp, %r8d + adcq %rbp, %r8 + xorl %ebp, %ebp + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %r8 + adcq %rbp, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rdx), %rdx + adcq %r13, %rdx + leaq -0x1(%rbp), %rbp + movq %rbp, %rax + adcq %r14, %rbp + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %rbp, %r14 + cmovbq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + xorl %r13d, %r13d + movq 0x60(%rsp), %rdx + mulxq 0xa0(%rsp), %r8, %r9 + mulxq 0xa8(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0xb0(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0xb8(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x68(%rsp), %rdx + xorl %r14d, %r14d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xb8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x70(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x78(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + movq 0x40(%rsp), %rax + subq 0x20(%rsp), %rax + movq 0x48(%rsp), %rcx + sbbq 0x28(%rsp), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x30(%rsp), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x38(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x40(%rdi) + adcq %r10, %rcx + movq %rcx, 0x48(%rdi) + adcq $0x0, %r8 + movq %r8, 0x50(%rdi) + adcq %rdx, %r9 + movq %r9, 0x58(%rdi) + movq 0x98(%rsp), %r11 + movq %r11, %rdx + movq 0x90(%rsp), %r10 + shldq $0x2, %r10, %r11 + movq 0x88(%rsp), %r9 + shldq $0x2, %r9, %r10 + movq 0x80(%rsp), %r8 + shldq $0x2, %r8, %r9 + shlq $0x2, %r8 + shrq $0x3e, %rdx + addq $0x1, %rdx + subq 0xa0(%rsp), %r8 + sbbq 0xa8(%rsp), %r9 + sbbq 0xb0(%rsp), %r10 + sbbq 0xb8(%rsp), %r11 + sbbq $0x0, %rdx + addq %rdx, %r8 + movq $0x100000000, %rax + mulxq %rax, %rax, %rcx + sbbq $0x0, %rax + sbbq $0x0, %rcx + subq %rax, %r9 + sbbq %rcx, %r10 + movq $0xffffffff00000001, %rax + mulxq %rax, %rax, %rcx + sbbq %rax, %r11 + sbbq %rcx, %rdx + decq %rdx + movl $0xffffffff, %eax + andq %rdx, %rax + xorl %ecx, %ecx + subq %rax, %rcx + addq %rdx, %r8 + movq %r8, (%rdi) + adcq %rax, %r9 + movq %r9, 0x8(%rdi) + adcq $0x0, %r10 + movq %r10, 0x10(%rdi) + adcq %rcx, %r11 + movq %r11, 0x18(%rdi) + movq $0xffffffffffffffff, %r8 + xorl %r10d, %r10d + subq (%rsp), %r8 + movq $0xffffffff, %r9 + sbbq 0x8(%rsp), %r9 + sbbq 0x10(%rsp), %r10 + movq $0xffffffff00000001, %r11 + sbbq 0x18(%rsp), %r11 + movq %r11, %r12 + shldq $0x3, %r10, %r11 + shldq $0x3, %r9, %r10 + shldq $0x3, %r8, %r9 + shlq $0x3, %r8 + shrq $0x3d, %r12 + movq $0x3, %rdx + xorl %eax, %eax + mulxq 0x60(%rsp), %rax, %rcx + adcxq %rax, %r8 + adoxq %rcx, %r9 + mulxq 0x68(%rsp), %rax, %rcx + adcxq %rax, %r9 + adoxq %rcx, %r10 + mulxq 0x70(%rsp), %rax, %rcx + adcxq %rax, %r10 + adoxq %rcx, %r11 + mulxq 0x78(%rsp), %rax, %rdx + adcxq %rax, %r11 + adoxq %r12, %rdx + adcq $0x1, %rdx + addq %rdx, %r8 + movq $0x100000000, %rax + mulxq %rax, %rax, %rcx + sbbq $0x0, %rax + sbbq $0x0, %rcx + subq %rax, %r9 + sbbq %rcx, %r10 + movq $0xffffffff00000001, %rax + mulxq %rax, %rax, %rcx + sbbq %rax, %r11 + sbbq %rcx, %rdx + decq %rdx + movl $0xffffffff, %eax + andq %rdx, %rax + xorl %ecx, %ecx + subq %rax, %rcx + addq %rdx, %r8 + movq %r8, 0x20(%rdi) + adcq %rax, %r9 + movq %r9, 0x28(%rdi) + adcq $0x0, %r10 + movq %r10, 0x30(%rdi) + adcq %rcx, %r11 + movq %r11, 0x38(%rdi) + CFI_INC_RSP(192) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +Lp256_scalarmul_local_p256_montjmixadd: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(192) + movq %rdx, %rbp + movq 0x40(%rsi), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0x48(%rsi), %r9, %r10 + mulxq 0x58(%rsi), %r11, %r12 + movq 0x50(%rsi), %rdx + mulxq 0x58(%rsi), %r13, %r14 + xorl %ecx, %ecx + mulxq 0x40(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x58(%rsi), %rdx + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + adoxq %rcx, %r14 + adcq %rcx, %r14 + xorl %ecx, %ecx + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0x48(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x50(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x58(%rsi), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rcx, %r15 + adoxq %rcx, %r15 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + movl %ecx, %r9d + adoxq %rcx, %r9 + adcxq %rcx, %r9 + addq %r9, %r14 + adcq %rcx, %r15 + movl %ecx, %r8d + adcq %rcx, %r8 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rcx, %r15 + adoxq %rcx, %r8 + adcq %rcx, %r8 + movl $0x1, %r8d + leaq -0x1(%rdx), %rdx + leaq -0x1(%rcx), %rax + movl $0xfffffffe, %r11d + cmoveq %rcx, %r8 + cmoveq %rcx, %rdx + cmoveq %rcx, %rax + cmoveq %rcx, %r11 + addq %r8, %r12 + adcq %rdx, %r13 + adcq %rax, %r14 + adcq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + xorl %r13d, %r13d + movq 0x20(%rbp), %rdx + mulxq 0x40(%rsi), %r8, %r9 + mulxq 0x48(%rsi), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x50(%rsi), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x58(%rsi), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x28(%rbp), %rdx + xorl %r14d, %r14d + mulxq 0x40(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x50(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x58(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x30(%rbp), %rdx + xorl %r8d, %r8d + mulxq 0x40(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x50(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x58(%rsi), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x38(%rbp), %rdx + xorl %r9d, %r9d + mulxq 0x40(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x50(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x58(%rsi), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x20(%rsp) + movq %r13, 0x28(%rsp) + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + xorl %r13d, %r13d + movq 0x0(%rbp), %rdx + mulxq (%rsp), %r8, %r9 + mulxq 0x8(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x10(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x18(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x8(%rbp), %rdx + xorl %r14d, %r14d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x10(%rbp), %rdx + xorl %r8d, %r8d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x18(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x18(%rbp), %rdx + xorl %r9d, %r9d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x18(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x40(%rsp) + movq %r13, 0x48(%rsp) + movq %r14, 0x50(%rsp) + movq %r15, 0x58(%rsp) + xorl %r13d, %r13d + movq 0x20(%rsp), %rdx + mulxq (%rsp), %r8, %r9 + mulxq 0x8(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x10(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x18(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x28(%rsp), %rdx + xorl %r14d, %r14d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x30(%rsp), %rdx + xorl %r8d, %r8d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x18(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x38(%rsp), %rdx + xorl %r9d, %r9d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x18(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x20(%rsp) + movq %r13, 0x28(%rsp) + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq 0x40(%rsp), %rax + subq (%rsi), %rax + movq 0x48(%rsp), %rcx + sbbq 0x8(%rsi), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x10(%rsi), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x18(%rsi), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0xa0(%rsp) + adcq %r10, %rcx + movq %rcx, 0xa8(%rsp) + adcq $0x0, %r8 + movq %r8, 0xb0(%rsp) + adcq %rdx, %r9 + movq %r9, 0xb8(%rsp) + movq 0x20(%rsp), %rax + subq 0x20(%rsi), %rax + movq 0x28(%rsp), %rcx + sbbq 0x28(%rsi), %rcx + movq 0x30(%rsp), %r8 + sbbq 0x30(%rsi), %r8 + movq 0x38(%rsp), %r9 + sbbq 0x38(%rsi), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x20(%rsp) + adcq %r10, %rcx + movq %rcx, 0x28(%rsp) + adcq $0x0, %r8 + movq %r8, 0x30(%rsp) + adcq %rdx, %r9 + movq %r9, 0x38(%rsp) + movq 0xa0(%rsp), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0xa8(%rsp), %r9, %r10 + mulxq 0xb8(%rsp), %r11, %r12 + movq 0xb0(%rsp), %rdx + mulxq 0xb8(%rsp), %r13, %r14 + xorl %ecx, %ecx + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0xb8(%rsp), %rdx + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + adoxq %rcx, %r14 + adcq %rcx, %r14 + xorl %ecx, %ecx + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0xa8(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0xb0(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0xb8(%rsp), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rcx, %r15 + adoxq %rcx, %r15 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + movl %ecx, %r9d + adoxq %rcx, %r9 + adcxq %rcx, %r9 + addq %r9, %r14 + adcq %rcx, %r15 + movl %ecx, %r8d + adcq %rcx, %r8 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rcx, %r15 + adoxq %rcx, %r8 + adcq %rcx, %r8 + movl $0x1, %r8d + leaq -0x1(%rdx), %rdx + leaq -0x1(%rcx), %rax + movl $0xfffffffe, %r11d + cmoveq %rcx, %r8 + cmoveq %rcx, %rdx + cmoveq %rcx, %rax + cmoveq %rcx, %r11 + addq %r8, %r12 + adcq %rdx, %r13 + adcq %rax, %r14 + adcq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + movq 0x20(%rsp), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0x28(%rsp), %r9, %r10 + mulxq 0x38(%rsp), %r11, %r12 + movq 0x30(%rsp), %rdx + mulxq 0x38(%rsp), %r13, %r14 + xorl %ecx, %ecx + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x38(%rsp), %rdx + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + adoxq %rcx, %r14 + adcq %rcx, %r14 + xorl %ecx, %ecx + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0x28(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x30(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x38(%rsp), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rcx, %r15 + adoxq %rcx, %r15 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + movl %ecx, %r9d + adoxq %rcx, %r9 + adcxq %rcx, %r9 + addq %r9, %r14 + adcq %rcx, %r15 + movl %ecx, %r8d + adcq %rcx, %r8 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rcx, %r15 + adoxq %rcx, %r8 + adcq %rcx, %r8 + movl $0x1, %ebx + addq %r12, %rbx + leaq -0x1(%rdx), %rdx + adcq %r13, %rdx + leaq -0x1(%rcx), %rcx + movq %rcx, %rax + adcq %r14, %rcx + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rbx, %r12 + cmovbq %rdx, %r13 + cmovbq %rcx, %r14 + cmovbq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + xorl %r13d, %r13d + movq (%rsi), %rdx + mulxq 0x60(%rsp), %r8, %r9 + mulxq 0x68(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x70(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x78(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x8(%rsi), %rdx + xorl %r14d, %r14d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x78(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x10(%rsi), %rdx + xorl %r8d, %r8d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x18(%rsi), %rdx + xorl %r9d, %r9d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + xorl %r13d, %r13d + movq 0x40(%rsp), %rdx + mulxq 0x60(%rsp), %r8, %r9 + mulxq 0x68(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x70(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x78(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x48(%rsp), %rdx + xorl %r14d, %r14d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x78(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x50(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x58(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x40(%rsp) + movq %r13, 0x48(%rsp) + movq %r14, 0x50(%rsp) + movq %r15, 0x58(%rsp) + movq (%rsp), %rax + subq 0x80(%rsp), %rax + movq 0x8(%rsp), %rcx + sbbq 0x88(%rsp), %rcx + movq 0x10(%rsp), %r8 + sbbq 0x90(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x98(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, (%rsp) + adcq %r10, %rcx + movq %rcx, 0x8(%rsp) + adcq $0x0, %r8 + movq %r8, 0x10(%rsp) + adcq %rdx, %r9 + movq %r9, 0x18(%rsp) + movq 0x40(%rsp), %rax + subq 0x80(%rsp), %rax + movq 0x48(%rsp), %rcx + sbbq 0x88(%rsp), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x90(%rsp), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x98(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x60(%rsp) + adcq %r10, %rcx + movq %rcx, 0x68(%rsp) + adcq $0x0, %r8 + movq %r8, 0x70(%rsp) + adcq %rdx, %r9 + movq %r9, 0x78(%rsp) + xorl %r13d, %r13d + movq 0x40(%rsi), %rdx + mulxq 0xa0(%rsp), %r8, %r9 + mulxq 0xa8(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0xb0(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0xb8(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x48(%rsi), %rdx + xorl %r14d, %r14d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xb8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x50(%rsi), %rdx + xorl %r8d, %r8d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x58(%rsi), %rdx + xorl %r9d, %r9d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0xa0(%rsp) + movq %r13, 0xa8(%rsp) + movq %r14, 0xb0(%rsp) + movq %r15, 0xb8(%rsp) + movq (%rsp), %rax + subq 0x40(%rsp), %rax + movq 0x8(%rsp), %rcx + sbbq 0x48(%rsp), %rcx + movq 0x10(%rsp), %r8 + sbbq 0x50(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x58(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, (%rsp) + adcq %r10, %rcx + movq %rcx, 0x8(%rsp) + adcq $0x0, %r8 + movq %r8, 0x10(%rsp) + adcq %rdx, %r9 + movq %r9, 0x18(%rsp) + movq 0x80(%rsp), %rax + subq (%rsp), %rax + movq 0x88(%rsp), %rcx + sbbq 0x8(%rsp), %rcx + movq 0x90(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x98(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x80(%rsp) + adcq %r10, %rcx + movq %rcx, 0x88(%rsp) + adcq $0x0, %r8 + movq %r8, 0x90(%rsp) + adcq %rdx, %r9 + movq %r9, 0x98(%rsp) + xorl %r13d, %r13d + movq 0x20(%rsi), %rdx + mulxq 0x60(%rsp), %r8, %r9 + mulxq 0x68(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x70(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x78(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x28(%rsi), %rdx + xorl %r14d, %r14d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x78(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x30(%rsi), %rdx + xorl %r8d, %r8d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x38(%rsi), %rdx + xorl %r9d, %r9d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + xorl %r13d, %r13d + movq 0x80(%rsp), %rdx + mulxq 0x20(%rsp), %r8, %r9 + mulxq 0x28(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x30(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x38(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x88(%rsp), %rdx + xorl %r14d, %r14d + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x38(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x90(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x38(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x98(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x38(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq 0x80(%rsp), %rax + subq 0x60(%rsp), %rax + movq 0x88(%rsp), %rcx + sbbq 0x68(%rsp), %rcx + movq 0x90(%rsp), %r8 + sbbq 0x70(%rsp), %r8 + movq 0x98(%rsp), %r9 + sbbq 0x78(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x80(%rsp) + adcq %r10, %rcx + movq %rcx, 0x88(%rsp) + adcq $0x0, %r8 + movq %r8, 0x90(%rsp) + adcq %rdx, %r9 + movq %r9, 0x98(%rsp) + movq 0x40(%rsi), %rax + movq 0x48(%rsi), %rdx + orq 0x50(%rsi), %rax + orq 0x58(%rsi), %rdx + orq %rdx, %rax + movq (%rsp), %r8 + movq 0x0(%rbp), %rax + cmoveq %rax, %r8 + movq 0x8(%rsp), %r9 + movq 0x8(%rbp), %rax + cmoveq %rax, %r9 + movq 0x10(%rsp), %r10 + movq 0x10(%rbp), %rax + cmoveq %rax, %r10 + movq 0x18(%rsp), %r11 + movq 0x18(%rbp), %rax + cmoveq %rax, %r11 + movq 0x80(%rsp), %r12 + movq 0x20(%rbp), %rax + cmoveq %rax, %r12 + movq 0x88(%rsp), %r13 + movq 0x28(%rbp), %rax + cmoveq %rax, %r13 + movq 0x90(%rsp), %r14 + movq 0x30(%rbp), %rax + cmoveq %rax, %r14 + movq 0x98(%rsp), %r15 + movq 0x38(%rbp), %rax + cmoveq %rax, %r15 + movq %r8, (%rdi) + movq %r9, 0x8(%rdi) + movq %r10, 0x10(%rdi) + movq %r11, 0x18(%rdi) + movq %r12, 0x20(%rdi) + movq %r13, 0x28(%rdi) + movq %r14, 0x30(%rdi) + movq %r15, 0x38(%rdi) + movq 0xa0(%rsp), %r8 + movq 0xa8(%rsp), %r9 + movq 0xb0(%rsp), %r10 + movq 0xb8(%rsp), %r11 + movl $0x1, %eax + cmoveq %rax, %r8 + movq $0xffffffff00000000, %rax + cmoveq %rax, %r9 + movq $0xffffffffffffffff, %rax + cmoveq %rax, %r10 + movl $0xfffffffe, %eax + cmoveq %rax, %r11 + movq %r8, 0x40(%rdi) + movq %r9, 0x48(%rdi) + movq %r10, 0x50(%rdi) + movq %r11, 0x58(%rdi) + CFI_INC_RSP(192) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/p256_scalarmul_alt.S b/cbits/s2n/x86_att/p256_scalarmul_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/p256_scalarmul_alt.S @@ -0,0 +1,8672 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Scalar multiplication for P-256 +// Input scalar[4], point[8]; output res[8] +// +// extern void p256_scalarmul_alt +// (uint64_t res[static 8],const uint64_t scalar[static 4], +// const uint64_t point[static 8]); +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-256, returns the point (X,Y) = n * P. The input and output +// are affine points, and in the case of the point at infinity as +// the result, (0,0) is returned. +// +// Standard x86-64 ABI: RDI = res, RSI = scalar, RDX = point +// Microsoft x64 ABI: RCX = res, RDX = scalar, R8 = point +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_scalarmul_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_scalarmul_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_scalarmul_alt) + + .text + .balign 32 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Intermediate variables on the stack. The last z2, z3 values can +// safely be overlaid on the table, which is no longer needed at the end. +// Uppercase syntactic variants make x86_att version simpler to generate + +#define SCALARB (0*NUMSIZE) +#define scalarb (0*NUMSIZE)(%rsp) +#define ACC (1*NUMSIZE) +#define acc (1*NUMSIZE)(%rsp) +#define TABENT (4*NUMSIZE) +#define tabent (4*NUMSIZE)(%rsp) + +#define TAB (7*NUMSIZE) +#define tab (7*NUMSIZE)(%rsp) + +#define Z2 (7*NUMSIZE) +#define z2 (7*NUMSIZE)(%rsp) +#define Z3 (8*NUMSIZE) +#define z3 (8*NUMSIZE)(%rsp) + +#define res (31*NUMSIZE)(%rsp) + +#define NSPACE 32*NUMSIZE + +S2N_BN_SYMBOL(p256_scalarmul_alt): + CFI_START + _CET_ENDBR + +// The Windows version literally calls the standard ABI version. +// This simplifies the proofs since subroutine offsets are fixed. + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + CFI_CALL(Lp256_scalarmul_alt_standard) + CFI_POP(%rsi) + CFI_POP(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_scalarmul_alt) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmul_alt_standard) + +Lp256_scalarmul_alt_standard: + CFI_START +#endif + +// Real start of the standard ABI code. + + CFI_PUSH(%r15) + CFI_PUSH(%r14) + CFI_PUSH(%r13) + CFI_PUSH(%r12) + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + + CFI_DEC_RSP(NSPACE) + +// Preserve the "res" and "point" input arguments. We load and process the +// scalar immediately so we don't bother preserving that input argument. +// Also, "point" is only needed early on and so its register gets re-used. + + movq %rdx, %rbx + movq %rdi, res + +// Load the digits of group order n_256 = [%r15;%r14;%r13;%r12] + + movq $0xf3b9cac2fc632551, %r12 + movq $0xbce6faada7179e84, %r13 + movq $0xffffffffffffffff, %r14 + movq $0xffffffff00000000, %r15 + +// First, reduce the input scalar mod n_256, i.e. conditionally subtract n_256 + + movq (%rsi), %r8 + subq %r12, %r8 + movq 8(%rsi), %r9 + sbbq %r13, %r9 + movq 16(%rsi), %r10 + sbbq %r14, %r10 + movq 24(%rsi), %r11 + sbbq %r15, %r11 + + cmovcq (%rsi), %r8 + cmovcq 8(%rsi), %r9 + cmovcq 16(%rsi), %r10 + cmovcq 24(%rsi), %r11 + +// Now if the top bit of the reduced scalar is set, negate it mod n_256, +// i.e. do n |-> n_256 - n. Remember the sign in %rbp so we can +// correspondingly negate the point below. + + subq %r8, %r12 + sbbq %r9, %r13 + sbbq %r10, %r14 + sbbq %r11, %r15 + + movq %r11, %rbp + shrq $63, %rbp + cmovnzq %r12, %r8 + cmovnzq %r13, %r9 + cmovnzq %r14, %r10 + cmovnzq %r15, %r11 + +// In either case then add the recoding constant 0x08888...888 to allow +// signed digits. + + movq $0x8888888888888888, %rax + addq %rax, %r8 + adcq %rax, %r9 + adcq %rax, %r10 + adcq %rax, %r11 + btc $63, %r11 + + movq %r8, SCALARB(%rsp) + movq %r9, SCALARB+8(%rsp) + movq %r10, SCALARB+16(%rsp) + movq %r11, SCALARB+24(%rsp) + +// Set the tab[0] table entry to Montgomery-Jacobian point = 1 * P +// The z coordinate is just the Montgomery form of the constant 1. + + leaq TAB(%rsp), %rdi + movq %rbx, %rsi + CFI_CALL(Lp256_scalarmul_alt_local_tomont_p256) + + leaq 32(%rbx), %rsi + leaq TAB+32(%rsp), %rdi + CFI_CALL(Lp256_scalarmul_alt_local_tomont_p256) + + movl $1, %eax + movq %rax, TAB+64(%rsp) + movq $0xffffffff00000000, %rdx + movq %rdx, TAB+72(%rsp) + subq $2, %rax + movq %rax, TAB+80(%rsp) + movq $0x00000000fffffffe, %rax + movq %rax, TAB+88(%rsp) + +// If the top bit of the scalar was set, negate (y coordinate of) the point + + movq TAB+32(%rsp), %r12 + movq TAB+40(%rsp), %r13 + movq TAB+48(%rsp), %r14 + movq TAB+56(%rsp), %r15 + + xorl %r10d, %r10d + leaq -1(%r10), %r8 + movq $0x00000000ffffffff, %r11 + movq %r11, %r9 + negq %r11 + + subq %r12, %r8 + sbbq %r13, %r9 + sbbq %r14, %r10 + sbbq %r15, %r11 + + testq %rbp, %rbp + cmovzq %r12, %r8 + cmovzq %r13, %r9 + cmovzq %r14, %r10 + cmovzq %r15, %r11 + + movq %r8, TAB+32(%rsp) + movq %r9, TAB+40(%rsp) + movq %r10, TAB+48(%rsp) + movq %r11, TAB+56(%rsp) + +// Compute and record tab[1] = 2 * p, ..., tab[7] = 8 * P + + leaq TAB+96*1(%rsp), %rdi + leaq TAB(%rsp), %rsi + CFI_CALL(Lp256_scalarmul_alt_local_p256_montjdouble) + + leaq TAB+96*2(%rsp), %rdi + leaq TAB+96*1(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp256_scalarmul_alt_local_p256_montjmixadd) + + leaq TAB+96*3(%rsp), %rdi + leaq TAB+96*1(%rsp), %rsi + CFI_CALL(Lp256_scalarmul_alt_local_p256_montjdouble) + + leaq TAB+96*4(%rsp), %rdi + leaq TAB+96*3(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp256_scalarmul_alt_local_p256_montjmixadd) + + leaq TAB+96*5(%rsp), %rdi + leaq TAB+96*2(%rsp), %rsi + CFI_CALL(Lp256_scalarmul_alt_local_p256_montjdouble) + + leaq TAB+96*6(%rsp), %rdi + leaq TAB+96*5(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp256_scalarmul_alt_local_p256_montjmixadd) + + leaq TAB+96*7(%rsp), %rdi + leaq TAB+96*3(%rsp), %rsi + CFI_CALL(Lp256_scalarmul_alt_local_p256_montjdouble) + +// Set up accumulator as table entry for top 4 bits (constant-time indexing) + + movq SCALARB+24(%rsp), %rdi + shrq $60, %rdi + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + xorl %r13d, %r13d + xorl %r14d, %r14d + xorl %r15d, %r15d + + .set I, 1 +.rep 8 + cmpq $I, %rdi + + cmovzq TAB+96*(I-1)(%rsp), %rax + cmovzq TAB+96*(I-1)+8(%rsp), %rbx + cmovzq TAB+96*(I-1)+16(%rsp), %rcx + cmovzq TAB+96*(I-1)+24(%rsp), %rdx + cmovzq TAB+96*(I-1)+32(%rsp), %r8 + cmovzq TAB+96*(I-1)+40(%rsp), %r9 + cmovzq TAB+96*(I-1)+48(%rsp), %r10 + cmovzq TAB+96*(I-1)+56(%rsp), %r11 + cmovzq TAB+96*(I-1)+64(%rsp), %r12 + cmovzq TAB+96*(I-1)+72(%rsp), %r13 + cmovzq TAB+96*(I-1)+80(%rsp), %r14 + cmovzq TAB+96*(I-1)+88(%rsp), %r15 + .set I, (I+1) +.endr + movq %rax, ACC(%rsp) + movq %rbx, ACC+8(%rsp) + movq %rcx, ACC+16(%rsp) + movq %rdx, ACC+24(%rsp) + movq %r8, ACC+32(%rsp) + movq %r9, ACC+40(%rsp) + movq %r10, ACC+48(%rsp) + movq %r11, ACC+56(%rsp) + movq %r12, ACC+64(%rsp) + movq %r13, ACC+72(%rsp) + movq %r14, ACC+80(%rsp) + movq %r15, ACC+88(%rsp) + +// Main loop over size-4 bitfield + + movl $252, %ebp + +Lp256_scalarmul_alt_loop: + subq $4, %rbp + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp256_scalarmul_alt_local_p256_montjdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp256_scalarmul_alt_local_p256_montjdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp256_scalarmul_alt_local_p256_montjdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp256_scalarmul_alt_local_p256_montjdouble) + + movq %rbp, %rax + shrq $6, %rax + movq (%rsp,%rax,8), %rdi + movq %rbp, %rcx + shrq %cl, %rdi + andq $15, %rdi + + subq $8, %rdi + sbbq %rsi, %rsi // %rsi = sign of digit (-1 = negative) + xorq %rsi, %rdi + subq %rsi, %rdi // %rdi = absolute value of digit + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + xorl %r13d, %r13d + xorl %r14d, %r14d + xorl %r15d, %r15d + + .set I, 1 +.rep 8 + cmpq $I, %rdi + + cmovzq TAB+96*(I-1)(%rsp), %rax + cmovzq TAB+96*(I-1)+8(%rsp), %rbx + cmovzq TAB+96*(I-1)+16(%rsp), %rcx + cmovzq TAB+96*(I-1)+24(%rsp), %rdx + cmovzq TAB+96*(I-1)+32(%rsp), %r8 + cmovzq TAB+96*(I-1)+40(%rsp), %r9 + cmovzq TAB+96*(I-1)+48(%rsp), %r10 + cmovzq TAB+96*(I-1)+56(%rsp), %r11 + cmovzq TAB+96*(I-1)+64(%rsp), %r12 + cmovzq TAB+96*(I-1)+72(%rsp), %r13 + cmovzq TAB+96*(I-1)+80(%rsp), %r14 + cmovzq TAB+96*(I-1)+88(%rsp), %r15 + .set I, (I+1) +.endr + + movq %r12, TABENT+64(%rsp) + movq %r13, TABENT+72(%rsp) + movq %r14, TABENT+80(%rsp) + movq %r15, TABENT+88(%rsp) + + xorl %r14d, %r14d + leaq -1(%r14), %r12 + movq $0x00000000ffffffff, %r15 + movq %r15, %r13 + negq %r15 + + subq %r8, %r12 + sbbq %r9, %r13 + sbbq %r10, %r14 + sbbq %r11, %r15 + + testq %rsi, %rsi + cmovnzq %r12, %r8 + cmovnzq %r13, %r9 + cmovnzq %r14, %r10 + cmovnzq %r15, %r11 + + movq %rax, TABENT(%rsp) + movq %rbx, TABENT+8(%rsp) + movq %rcx, TABENT+16(%rsp) + movq %rdx, TABENT+24(%rsp) + + movq %r8, TABENT+32(%rsp) + movq %r9, TABENT+40(%rsp) + movq %r10, TABENT+48(%rsp) + movq %r11, TABENT+56(%rsp) + + leaq TABENT(%rsp), %rdx + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp256_scalarmul_alt_local_p256_montjadd) + + testq %rbp, %rbp + jne Lp256_scalarmul_alt_loop + +// Let z2 = 1/z^2 and z3 = 1/z^3, both without Montgomery form + + leaq Z2(%rsp), %rdi + leaq ACC+64(%rsp), %rsi + CFI_CALL(Lp256_scalarmul_alt_local_montsqr_p256) + + leaq Z3(%rsp), %rdi + leaq ACC+64(%rsp), %rsi + leaq Z2(%rsp), %rdx + CFI_CALL(Lp256_scalarmul_alt_local_montmul_p256) + + leaq Z2(%rsp), %rdi + leaq Z3(%rsp), %rsi + CFI_CALL(Lp256_scalarmul_alt_local_demont_p256) + + leaq Z3(%rsp), %rdi + leaq Z2(%rsp), %rsi + CFI_CALL(Lp256_scalarmul_alt_local_inv_p256) + + leaq Z2(%rsp), %rdi + leaq ACC+64(%rsp), %rsi + leaq Z3(%rsp), %rdx + CFI_CALL(Lp256_scalarmul_alt_local_montmul_p256) + +// Convert back from Jacobian (X, Y, Z) |-> (X/Z^2, Y/Z^3) + + movq res, %rdi + leaq ACC(%rsp), %rsi + leaq Z2(%rsp), %rdx + movq %rdi, %rbx + CFI_CALL(Lp256_scalarmul_alt_local_montmul_p256) + + leaq 32(%rbx), %rdi + leaq ACC+32(%rsp), %rsi + leaq Z3(%rsp), %rdx + CFI_CALL(Lp256_scalarmul_alt_local_montmul_p256) + +// Restore stack and registers and return + + CFI_INC_RSP(NSPACE) + CFI_POP(%rbx) + CFI_POP(%rbp) + CFI_POP(%r12) + CFI_POP(%r13) + CFI_POP(%r14) + CFI_POP(%r15) + CFI_RET + +#if WINDOWS_ABI +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmul_alt_standard) +#else +S2N_BN_SIZE_DIRECTIVE(p256_scalarmul_alt) +#endif + +// Local copies of subroutines, complete clones at the moment + +Lp256_scalarmul_alt_local_demont_p256: + CFI_START + movq (%rsi), %r8 + movq 0x8(%rsi), %r9 + movq 0x10(%rsi), %r10 + movq 0x18(%rsi), %r11 + movabsq $0x100000000, %rcx + movq %r8, %rax + mulq %rcx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rsi, %rsi + movq %r9, %rax + mulq %rcx + subq %rsi, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rsi, %rsi + negq %rcx + negq %rsi + incq %rcx + movq %r8, %rax + mulq %rcx + addq %rax, %r11 + adcq %rdx, %rsi + sbbq %r8, %r8 + negq %r8 + movq %r9, %rax + mulq %rcx + addq %rax, %rsi + adcq %rdx, %r8 + negq %rcx + incq %rcx + movq %r10, %rax + mulq %rcx + addq %rax, %r11 + adcq %rdx, %rsi + sbbq %r9, %r9 + movq %r11, %rax + mulq %rcx + subq %r9, %rdx + addq %rax, %rsi + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %rcx + negq %r9 + incq %rcx + movq %r10, %rax + mulq %rcx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rax + mulq %rcx + addq %rax, %r9 + adcq %rdx, %r10 + movq %rsi, (%rdi) + movq %r8, 0x8(%rdi) + movq %r9, 0x10(%rdi) + movq %r10, 0x18(%rdi) + CFI_RET + +Lp256_scalarmul_alt_local_inv_p256: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(240) + movq %rdi, 0xe0(%rsp) + xorl %ecx, %ecx + movl $0xffffffff, %edx + movq %rdx, %rbx + leaq -0x1(%rcx), %rax + negq %rdx + movq %rax, (%rsp) + movq %rbx, 0x8(%rsp) + movq %rcx, 0x10(%rsp) + movq %rdx, 0x18(%rsp) + movq %rcx, 0x20(%rsp) + movq (%rsi), %r8 + movq 0x8(%rsi), %r9 + movq 0x10(%rsi), %r10 + movq 0x18(%rsi), %r11 + leaq 0x1(%rcx), %rax + addq %r8, %rax + leaq -0x1(%rdx), %rbx + adcq %r9, %rbx + notq %rcx + adcq %r10, %rcx + notq %rdx + adcq %r11, %rdx + cmovaeq %r8, %rax + cmovaeq %r9, %rbx + cmovaeq %r10, %rcx + cmovaeq %r11, %rdx + movq %rax, 0x28(%rsp) + movq %rbx, 0x30(%rsp) + movq %rcx, 0x38(%rsp) + movq %rdx, 0x40(%rsp) + xorl %eax, %eax + movq %rax, 0x48(%rsp) + xorl %eax, %eax + movq %rax, 0x50(%rsp) + movq %rax, 0x58(%rsp) + movq %rax, 0x60(%rsp) + movq %rax, 0x68(%rsp) + movabsq $0x4000000000000, %rcx + movq %rcx, 0x78(%rsp) + movq %rax, 0x80(%rsp) + movq %rax, 0x88(%rsp) + movq %rax, 0x90(%rsp) + movq $0xa, 0xb0(%rsp) + movq $0x1, 0xb8(%rsp) + jmp Lp256_scalarmul_alt_inv_midloop +Lp256_scalarmul_alt_inv_loop: + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + movq %r8, %rax + andq %r9, %rax + movq %r10, %rdi + andq %r11, %rdi + addq %rax, %rdi + movq %rdi, 0xa0(%rsp) + movq %r12, %rax + andq %r13, %rax + movq %r14, %rsi + andq %r15, %rsi + addq %rax, %rsi + movq %rsi, 0xa8(%rsp) + xorl %ebx, %ebx + movq (%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x28(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rdi + adcq %rdx, %rbx + xorl %ebp, %ebp + movq (%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x28(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + xorl %ecx, %ecx + movq 0x8(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x30(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + shrdq $0x3b, %rbx, %rdi + movq %rdi, (%rsp) + xorl %edi, %edi + movq 0x8(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rbp + adcq %rdx, %rdi + movq 0x30(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rdi + shrdq $0x3b, %rbp, %rsi + movq %rsi, 0x28(%rsp) + xorl %esi, %esi + movq 0x10(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rsi + movq 0x38(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rsi + shrdq $0x3b, %rcx, %rbx + movq %rbx, 0x8(%rsp) + xorl %ebx, %ebx + movq 0x10(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x38(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rdi + adcq %rdx, %rbx + shrdq $0x3b, %rdi, %rbp + movq %rbp, 0x30(%rsp) + movq 0x18(%rsp), %rax + xorq %r9, %rax + movq 0x20(%rsp), %rbp + xorq %r9, %rbp + andq %r8, %rbp + negq %rbp + mulq %r8 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x40(%rsp), %rax + xorq %r11, %rax + movq 0x48(%rsp), %rdx + xorq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbp + mulq %r10 + addq %rax, %rsi + adcq %rdx, %rbp + shrdq $0x3b, %rsi, %rcx + movq %rcx, 0x10(%rsp) + shrdq $0x3b, %rbp, %rsi + sarq $0x3b, %rbp + movq 0x18(%rsp), %rax + movq %rsi, 0x18(%rsp) + movq 0x20(%rsp), %rsi + movq %rbp, 0x20(%rsp) + xorq %r13, %rax + xorq %r13, %rsi + andq %r12, %rsi + negq %rsi + mulq %r12 + addq %rax, %rbx + adcq %rdx, %rsi + movq 0x40(%rsp), %rax + xorq %r15, %rax + movq 0x48(%rsp), %rdx + xorq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rsi + mulq %r14 + addq %rax, %rbx + adcq %rdx, %rsi + shrdq $0x3b, %rbx, %rdi + movq %rdi, 0x38(%rsp) + shrdq $0x3b, %rsi, %rbx + movq %rbx, 0x40(%rsp) + sarq $0x3b, %rsi + movq %rsi, 0x48(%rsp) + movq 0xa0(%rsp), %rbx + movq 0xa8(%rsp), %rbp + xorl %ecx, %ecx + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x78(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x50(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x50(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x78(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x78(%rsp) + xorl %ebx, %ebx + movq 0x58(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x80(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + xorl %ebp, %ebp + movq 0x58(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, 0x58(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x80(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, 0x80(%rsp) + xorl %ecx, %ecx + movq 0x60(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x88(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x60(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x60(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x88(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x88(%rsp) + movq 0x68(%rsp), %rax + xorq %r9, %rax + movq %r9, %rbx + andq %r8, %rbx + negq %rbx + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x90(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbx + mulq %r10 + addq %rax, %rcx + adcq %rbx, %rdx + movq 0x68(%rsp), %rax + movq %rcx, 0x68(%rsp) + movq %rdx, 0x70(%rsp) + xorq %r13, %rax + movq %r13, %rcx + andq %r12, %rcx + negq %rcx + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rcx + movq 0x90(%rsp), %rax + xorq %r15, %rax + movq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rcx + mulq %r14 + addq %rax, %rsi + adcq %rcx, %rdx + movq %rsi, 0x90(%rsp) + movq %rdx, 0x98(%rsp) + movabsq $0xe000000000000000, %r8 + addq 0x50(%rsp), %r8 + movq $0xffffffffffffffff, %r9 + adcq 0x58(%rsp), %r9 + movq $0x1fffffff, %r10 + adcq 0x60(%rsp), %r10 + movabsq $0x2000000000000000, %r11 + adcq 0x68(%rsp), %r11 + movabsq $0x1fffffffe0000000, %r12 + adcq 0x70(%rsp), %r12 + movq %r8, %rbx + shlq $0x20, %rbx + movabsq $0xffffffff00000001, %rax + mulq %r8 + shrq $0x20, %r8 + addq %rbx, %r9 + adcq %r8, %r10 + adcq %rax, %r11 + adcq %rdx, %r12 + sbbq %rax, %rax + movl $0xffffffff, %ebx + andq %rax, %rbx + movabsq $0xffffffff00000001, %rdx + andq %rax, %rdx + subq %rax, %r9 + movq %r9, 0x50(%rsp) + sbbq %rbx, %r10 + movq %r10, 0x58(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x60(%rsp) + sbbq %rdx, %r12 + movq %r12, 0x68(%rsp) + movabsq $0xe000000000000000, %r8 + addq 0x78(%rsp), %r8 + movq $0xffffffffffffffff, %r9 + adcq 0x80(%rsp), %r9 + movq $0x1fffffff, %r10 + adcq 0x88(%rsp), %r10 + movabsq $0x2000000000000000, %r11 + adcq 0x90(%rsp), %r11 + movabsq $0x1fffffffe0000000, %r12 + adcq 0x98(%rsp), %r12 + movq %r8, %rbx + shlq $0x20, %rbx + movabsq $0xffffffff00000001, %rax + mulq %r8 + shrq $0x20, %r8 + addq %rbx, %r9 + adcq %r8, %r10 + adcq %rax, %r11 + adcq %rdx, %r12 + sbbq %rax, %rax + movl $0xffffffff, %ebx + andq %rax, %rbx + movabsq $0xffffffff00000001, %rdx + andq %rax, %rdx + subq %rax, %r9 + movq %r9, 0x78(%rsp) + sbbq %rbx, %r10 + movq %r10, 0x80(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x88(%rsp) + sbbq %rdx, %r12 + movq %r12, 0x90(%rsp) +Lp256_scalarmul_alt_inv_midloop: + movq 0xb8(%rsp), %rsi + movq (%rsp), %rdx + movq 0x28(%rsp), %rcx + movq %rdx, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + xorl %ebp, %ebp + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %rdx + leaq (%rcx,%rax), %rdi + shlq $0x16, %rdx + shlq $0x16, %rdi + sarq $0x2b, %rdx + sarq $0x2b, %rdi + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %rbx + leaq (%rcx,%rax), %rcx + sarq $0x2a, %rbx + sarq $0x2a, %rcx + movq %rdx, 0xc0(%rsp) + movq %rbx, 0xc8(%rsp) + movq %rdi, 0xd0(%rsp) + movq %rcx, 0xd8(%rsp) + movq (%rsp), %r12 + imulq %r12, %rdi + imulq %rdx, %r12 + movq 0x28(%rsp), %r13 + imulq %r13, %rbx + imulq %rcx, %r13 + addq %rbx, %r12 + addq %rdi, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r10 + shlq $0x16, %r8 + shlq $0x16, %r10 + sarq $0x2b, %r8 + sarq $0x2b, %r10 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r15 + leaq (%rcx,%rax), %r11 + sarq $0x2a, %r15 + sarq $0x2a, %r11 + movq %r13, %rbx + movq %r12, %rcx + imulq %r8, %r12 + imulq %r15, %rbx + addq %rbx, %r12 + imulq %r11, %r13 + imulq %r10, %rcx + addq %rcx, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq 0xc0(%rsp), %rax + imulq %r8, %rax + movq 0xd0(%rsp), %rdx + imulq %r15, %rdx + imulq 0xc8(%rsp), %r8 + imulq 0xd8(%rsp), %r15 + addq %r8, %r15 + leaq (%rax,%rdx), %r9 + movq 0xc0(%rsp), %rax + imulq %r10, %rax + movq 0xd0(%rsp), %rdx + imulq %r11, %rdx + imulq 0xc8(%rsp), %r10 + imulq 0xd8(%rsp), %r11 + addq %r10, %r11 + leaq (%rax,%rdx), %r13 + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r12 + shlq $0x15, %r8 + shlq $0x15, %r12 + sarq $0x2b, %r8 + sarq $0x2b, %r12 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r10 + leaq (%rcx,%rax), %r14 + sarq $0x2b, %r10 + sarq $0x2b, %r14 + movq %r9, %rax + imulq %r8, %rax + movq %r13, %rdx + imulq %r10, %rdx + imulq %r15, %r8 + imulq %r11, %r10 + addq %r8, %r10 + leaq (%rax,%rdx), %r8 + movq %r9, %rax + imulq %r12, %rax + movq %r13, %rdx + imulq %r14, %rdx + imulq %r15, %r12 + imulq %r11, %r14 + addq %r12, %r14 + leaq (%rax,%rdx), %r12 + movq %rsi, 0xb8(%rsp) + decq 0xb0(%rsp) + jne Lp256_scalarmul_alt_inv_loop + movq (%rsp), %rax + movq 0x28(%rsp), %rcx + imulq %r8, %rax + imulq %r10, %rcx + addq %rcx, %rax + sarq $0x3f, %rax + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + xorq %rax, %r9 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + xorq %rax, %r11 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + xorq %rax, %r13 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + xorq %rax, %r15 + movq %r8, %rax + andq %r9, %rax + movq %r10, %r12 + andq %r11, %r12 + addq %rax, %r12 + xorl %r13d, %r13d + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x78(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movq 0x58(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r13 + adcq %rdx, %r14 + movq 0x80(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq 0x60(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x88(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x68(%rsp), %rax + xorq %r9, %rax + andq %r8, %r9 + negq %r9 + mulq %r8 + addq %rax, %r15 + adcq %rdx, %r9 + movq 0x90(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %r9 + mulq %r10 + addq %rax, %r15 + adcq %rdx, %r9 + movq %r12, 0x50(%rsp) + movq %r13, 0x58(%rsp) + movq %r14, 0x60(%rsp) + movq %r15, 0x68(%rsp) + movq %r9, 0x70(%rsp) + movabsq $0xe000000000000000, %r8 + addq 0x50(%rsp), %r8 + movq $0xffffffffffffffff, %r9 + adcq 0x58(%rsp), %r9 + movq $0x1fffffff, %r10 + adcq 0x60(%rsp), %r10 + movabsq $0x2000000000000000, %r11 + adcq 0x68(%rsp), %r11 + movabsq $0x1fffffffe0000000, %r12 + adcq 0x70(%rsp), %r12 + movq %r8, %rbx + shlq $0x20, %rbx + movabsq $0xffffffff00000001, %rax + mulq %r8 + shrq $0x20, %r8 + addq %rbx, %r9 + adcq %r8, %r10 + adcq %rax, %r11 + adcq %rdx, %r12 + sbbq %rax, %rax + movl $0xffffffff, %ebx + andq %rax, %rbx + movabsq $0xffffffff00000001, %rdx + andq %rax, %rdx + subq %rax, %r9 + movq %r9, 0x50(%rsp) + sbbq %rbx, %r10 + movq %r10, 0x58(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x60(%rsp) + sbbq %rdx, %r12 + movq %r12, 0x68(%rsp) + movq 0x50(%rsp), %r8 + movq 0x58(%rsp), %r9 + movq 0x60(%rsp), %r10 + movq 0x68(%rsp), %r11 + movl $0x1, %eax + movl $0xffffffff, %ebx + leaq -0x2(%rax), %rcx + leaq -0x1(%rbx), %rdx + notq %rbx + addq %r8, %rax + adcq %r9, %rbx + adcq %r10, %rcx + adcq %r11, %rdx + cmovaeq %r8, %rax + cmovaeq %r9, %rbx + cmovaeq %r10, %rcx + cmovaeq %r11, %rdx + movq 0xe0(%rsp), %rdi + movq %rax, (%rdi) + movq %rbx, 0x8(%rdi) + movq %rcx, 0x10(%rdi) + movq %rdx, 0x18(%rdi) + CFI_INC_RSP(240) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +Lp256_scalarmul_alt_local_montmul_p256: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + movq %rdx, %rcx + movq (%rcx), %rbx + movq (%rsi), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x8(%rsi), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x10(%rsi), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x18(%rsi), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x8(%rcx), %rbx + xorl %r13d, %r13d + movq (%rsi), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x8(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x10(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x18(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x10(%rcx), %rbx + xorl %r15d, %r15d + movq (%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x8(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x10(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x18(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x18(%rcx), %rbx + xorl %r8d, %r8d + movq (%rsi), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x8(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x10(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x18(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + CFI_RET + +Lp256_scalarmul_alt_local_montsqr_p256: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + movq (%rsi), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0x8(%rsi), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x18(%rsi), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x10(%rsi), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq (%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0x8(%rsi), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0x18(%rsi), %rbx + movq 0x8(%rsi), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0x8(%rsi), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0x10(%rsi), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0x18(%rsi), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + CFI_RET + + +Lp256_scalarmul_alt_local_tomont_p256: + CFI_START + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + movl $0x3, %ecx + movq (%rsi), %rax + mulq %rcx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x8(%rsi), %rax + mulq %rcx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x10(%rsi), %rax + mulq %rcx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x18(%rsi), %rax + mulq %rcx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movabsq $0xfffffffbffffffff, %rcx + xorl %r13d, %r13d + movq (%rsi), %rax + mulq %rcx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x8(%rsi), %rax + mulq %rcx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x10(%rsi), %rax + mulq %rcx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x18(%rsi), %rax + mulq %rcx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rcx + movq %r8, %rax + mulq %rcx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rcx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rcx + leaq 0x2(%rcx), %rcx + movq %r8, %rax + mulq %rcx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rcx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq $0xfffffffffffffffe, %rcx + xorl %r15d, %r15d + movq (%rsi), %rax + mulq %rcx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x8(%rsi), %rax + mulq %rcx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x10(%rsi), %rax + mulq %rcx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x18(%rsi), %rax + mulq %rcx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movabsq $0x4fffffffd, %rcx + xorl %r8d, %r8d + movq (%rsi), %rax + mulq %rcx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x8(%rsi), %rax + mulq %rcx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x10(%rsi), %rax + mulq %rcx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x18(%rsi), %rax + mulq %rcx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rcx + movq %r10, %rax + mulq %rcx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq %r11, %rax + mulq %rcx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + notq %rcx + leaq 0x2(%rcx), %rcx + movq %r10, %rax + mulq %rcx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq %r11, %rax + mulq %rcx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + subq %r9, %r8 + xorl %edx, %edx + leaq -0x1(%rdx), %r9 + incq %rdx + addq %r12, %rdx + decq %rcx + adcq %r13, %rcx + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rdx, %r12 + cmovbq %rcx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_RET + +Lp256_scalarmul_alt_local_p256_montjadd: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(224) + movq %rdx, %rbp + movq 0x40(%rsi), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0x48(%rsi), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x58(%rsi), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x50(%rsi), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq 0x40(%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0x48(%rsi), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0x58(%rsi), %rbx + movq 0x48(%rsi), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0x48(%rsi), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0x50(%rsi), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0x58(%rsi), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + movq 0x40(%rbp), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0x48(%rbp), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x58(%rbp), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x50(%rbp), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq 0x40(%rbp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0x48(%rbp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0x58(%rbp), %rbx + movq 0x48(%rbp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0x48(%rbp), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0x50(%rbp), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0x58(%rbp), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0xa0(%rsp) + movq %r13, 0xa8(%rsp) + movq %r14, 0xb0(%rsp) + movq %r15, 0xb8(%rsp) + movq 0x20(%rsi), %rbx + movq 0x40(%rbp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x48(%rbp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x50(%rbp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x58(%rbp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x28(%rsi), %rbx + xorl %r13d, %r13d + movq 0x40(%rbp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x48(%rbp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x50(%rbp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x58(%rbp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x30(%rsi), %rbx + xorl %r15d, %r15d + movq 0x40(%rbp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x48(%rbp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x50(%rbp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x58(%rbp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x38(%rsi), %rbx + xorl %r8d, %r8d + movq 0x40(%rbp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x48(%rbp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x50(%rbp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x58(%rbp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0xc0(%rsp) + movq %r13, 0xc8(%rsp) + movq %r14, 0xd0(%rsp) + movq %r15, 0xd8(%rsp) + movq 0x20(%rbp), %rbx + movq 0x40(%rsi), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x48(%rsi), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x50(%rsi), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x58(%rsi), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x28(%rbp), %rbx + xorl %r13d, %r13d + movq 0x40(%rsi), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x48(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x50(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x58(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x30(%rbp), %rbx + xorl %r15d, %r15d + movq 0x40(%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x48(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x50(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x58(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x38(%rbp), %rbx + xorl %r8d, %r8d + movq 0x40(%rsi), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x48(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x50(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x58(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x20(%rsp) + movq %r13, 0x28(%rsp) + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq 0x0(%rbp), %rbx + movq (%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x10(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x18(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x8(%rbp), %rbx + xorl %r13d, %r13d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x10(%rbp), %rbx + xorl %r15d, %r15d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x18(%rbp), %rbx + xorl %r8d, %r8d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x40(%rsp) + movq %r13, 0x48(%rsp) + movq %r14, 0x50(%rsp) + movq %r15, 0x58(%rsp) + movq (%rsi), %rbx + movq 0xa0(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0xb0(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0xb8(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x8(%rsi), %rbx + xorl %r13d, %r13d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x10(%rsi), %rbx + xorl %r15d, %r15d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x18(%rsi), %rbx + xorl %r8d, %r8d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq 0x20(%rsp), %rbx + movq (%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x10(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x18(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x28(%rsp), %rbx + xorl %r13d, %r13d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x30(%rsp), %rbx + xorl %r15d, %r15d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x38(%rsp), %rbx + xorl %r8d, %r8d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x20(%rsp) + movq %r13, 0x28(%rsp) + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq 0xc0(%rsp), %rbx + movq 0xa0(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0xb0(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0xb8(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0xc8(%rsp), %rbx + xorl %r13d, %r13d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0xd0(%rsp), %rbx + xorl %r15d, %r15d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0xd8(%rsp), %rbx + xorl %r8d, %r8d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0xc0(%rsp) + movq %r13, 0xc8(%rsp) + movq %r14, 0xd0(%rsp) + movq %r15, 0xd8(%rsp) + movq 0x40(%rsp), %rax + subq 0x80(%rsp), %rax + movq 0x48(%rsp), %rcx + sbbq 0x88(%rsp), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x90(%rsp), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x98(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0xa0(%rsp) + adcq %r10, %rcx + movq %rcx, 0xa8(%rsp) + adcq $0x0, %r8 + movq %r8, 0xb0(%rsp) + adcq %rdx, %r9 + movq %r9, 0xb8(%rsp) + movq 0x20(%rsp), %rax + subq 0xc0(%rsp), %rax + movq 0x28(%rsp), %rcx + sbbq 0xc8(%rsp), %rcx + movq 0x30(%rsp), %r8 + sbbq 0xd0(%rsp), %r8 + movq 0x38(%rsp), %r9 + sbbq 0xd8(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x20(%rsp) + adcq %r10, %rcx + movq %rcx, 0x28(%rsp) + adcq $0x0, %r8 + movq %r8, 0x30(%rsp) + adcq %rdx, %r9 + movq %r9, 0x38(%rsp) + movq 0xa0(%rsp), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0xa8(%rsp), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0xb8(%rsp), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0xb0(%rsp), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0xa8(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0xb8(%rsp), %rbx + movq 0xa8(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0xa8(%rsp), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0xb0(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0xb8(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + movq 0x20(%rsp), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0x28(%rsp), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x38(%rsp), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x30(%rsp), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq 0x20(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0x28(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0x38(%rsp), %rbx + movq 0x28(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0x28(%rsp), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0x30(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0x38(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + movq 0x80(%rsp), %rbx + movq 0x60(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x70(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x78(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x88(%rsp), %rbx + xorl %r13d, %r13d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x90(%rsp), %rbx + xorl %r15d, %r15d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x98(%rsp), %rbx + xorl %r8d, %r8d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq 0x40(%rsp), %rbx + movq 0x60(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x70(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x78(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x48(%rsp), %rbx + xorl %r13d, %r13d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x50(%rsp), %rbx + xorl %r15d, %r15d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x58(%rsp), %rbx + xorl %r8d, %r8d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x40(%rsp) + movq %r13, 0x48(%rsp) + movq %r14, 0x50(%rsp) + movq %r15, 0x58(%rsp) + movq (%rsp), %rax + subq 0x80(%rsp), %rax + movq 0x8(%rsp), %rcx + sbbq 0x88(%rsp), %rcx + movq 0x10(%rsp), %r8 + sbbq 0x90(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x98(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, (%rsp) + adcq %r10, %rcx + movq %rcx, 0x8(%rsp) + adcq $0x0, %r8 + movq %r8, 0x10(%rsp) + adcq %rdx, %r9 + movq %r9, 0x18(%rsp) + movq 0x40(%rsp), %rax + subq 0x80(%rsp), %rax + movq 0x48(%rsp), %rcx + sbbq 0x88(%rsp), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x90(%rsp), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x98(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x60(%rsp) + adcq %r10, %rcx + movq %rcx, 0x68(%rsp) + adcq $0x0, %r8 + movq %r8, 0x70(%rsp) + adcq %rdx, %r9 + movq %r9, 0x78(%rsp) + movq 0x40(%rsi), %rbx + movq 0xa0(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0xb0(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0xb8(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x48(%rsi), %rbx + xorl %r13d, %r13d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x50(%rsi), %rbx + xorl %r15d, %r15d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x58(%rsi), %rbx + xorl %r8d, %r8d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0xa0(%rsp) + movq %r13, 0xa8(%rsp) + movq %r14, 0xb0(%rsp) + movq %r15, 0xb8(%rsp) + movq (%rsp), %rax + subq 0x40(%rsp), %rax + movq 0x8(%rsp), %rcx + sbbq 0x48(%rsp), %rcx + movq 0x10(%rsp), %r8 + sbbq 0x50(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x58(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, (%rsp) + adcq %r10, %rcx + movq %rcx, 0x8(%rsp) + adcq $0x0, %r8 + movq %r8, 0x10(%rsp) + adcq %rdx, %r9 + movq %r9, 0x18(%rsp) + movq 0x80(%rsp), %rax + subq (%rsp), %rax + movq 0x88(%rsp), %rcx + sbbq 0x8(%rsp), %rcx + movq 0x90(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x98(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x80(%rsp) + adcq %r10, %rcx + movq %rcx, 0x88(%rsp) + adcq $0x0, %r8 + movq %r8, 0x90(%rsp) + adcq %rdx, %r9 + movq %r9, 0x98(%rsp) + movq 0xc0(%rsp), %rbx + movq 0x60(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x70(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x78(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0xc8(%rsp), %rbx + xorl %r13d, %r13d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0xd0(%rsp), %rbx + xorl %r15d, %r15d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0xd8(%rsp), %rbx + xorl %r8d, %r8d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + movq 0x40(%rbp), %rbx + movq 0xa0(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0xb0(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0xb8(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x48(%rbp), %rbx + xorl %r13d, %r13d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x50(%rbp), %rbx + xorl %r15d, %r15d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x58(%rbp), %rbx + xorl %r8d, %r8d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0xa0(%rsp) + movq %r13, 0xa8(%rsp) + movq %r14, 0xb0(%rsp) + movq %r15, 0xb8(%rsp) + movq 0x80(%rsp), %rbx + movq 0x20(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x28(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x30(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x38(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x88(%rsp), %rbx + xorl %r13d, %r13d + movq 0x20(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x30(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x38(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x90(%rsp), %rbx + xorl %r15d, %r15d + movq 0x20(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x30(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x38(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x98(%rsp), %rbx + xorl %r8d, %r8d + movq 0x20(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x30(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x38(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq 0x80(%rsp), %rax + subq 0x60(%rsp), %rax + movq 0x88(%rsp), %rcx + sbbq 0x68(%rsp), %rcx + movq 0x90(%rsp), %r8 + sbbq 0x70(%rsp), %r8 + movq 0x98(%rsp), %r9 + sbbq 0x78(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x80(%rsp) + adcq %r10, %rcx + movq %rcx, 0x88(%rsp) + adcq $0x0, %r8 + movq %r8, 0x90(%rsp) + adcq %rdx, %r9 + movq %r9, 0x98(%rsp) + movq 0x40(%rsi), %r8 + movq 0x48(%rsi), %r9 + movq 0x50(%rsi), %r10 + movq 0x58(%rsi), %r11 + movq %r8, %rax + movq %r9, %rdx + orq %r10, %rax + orq %r11, %rdx + orq %rdx, %rax + negq %rax + sbbq %rax, %rax + movq 0x40(%rbp), %r12 + movq 0x48(%rbp), %r13 + movq 0x50(%rbp), %r14 + movq 0x58(%rbp), %r15 + movq %r12, %rbx + movq %r13, %rdx + orq %r14, %rbx + orq %r15, %rdx + orq %rdx, %rbx + negq %rbx + sbbq %rbx, %rbx + cmpq %rax, %rbx + cmovbq %r8, %r12 + cmovbq %r9, %r13 + cmovbq %r10, %r14 + cmovbq %r11, %r15 + cmoveq 0xa0(%rsp), %r12 + cmoveq 0xa8(%rsp), %r13 + cmoveq 0xb0(%rsp), %r14 + cmoveq 0xb8(%rsp), %r15 + movq (%rsp), %rax + cmovbq (%rsi), %rax + cmova 0x0(%rbp), %rax + movq 0x8(%rsp), %rbx + cmovbq 0x8(%rsi), %rbx + cmova 0x8(%rbp), %rbx + movq 0x10(%rsp), %rcx + cmovbq 0x10(%rsi), %rcx + cmova 0x10(%rbp), %rcx + movq 0x18(%rsp), %rdx + cmovbq 0x18(%rsi), %rdx + cmova 0x18(%rbp), %rdx + movq 0x80(%rsp), %r8 + cmovbq 0x20(%rsi), %r8 + cmova 0x20(%rbp), %r8 + movq 0x88(%rsp), %r9 + cmovbq 0x28(%rsi), %r9 + cmova 0x28(%rbp), %r9 + movq 0x90(%rsp), %r10 + cmovbq 0x30(%rsi), %r10 + cmova 0x30(%rbp), %r10 + movq 0x98(%rsp), %r11 + cmovbq 0x38(%rsi), %r11 + cmova 0x38(%rbp), %r11 + movq %rax, (%rdi) + movq %rbx, 0x8(%rdi) + movq %rcx, 0x10(%rdi) + movq %rdx, 0x18(%rdi) + movq %r8, 0x20(%rdi) + movq %r9, 0x28(%rdi) + movq %r10, 0x30(%rdi) + movq %r11, 0x38(%rdi) + movq %r12, 0x40(%rdi) + movq %r13, 0x48(%rdi) + movq %r14, 0x50(%rdi) + movq %r15, 0x58(%rdi) + CFI_INC_RSP(224) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +Lp256_scalarmul_alt_local_p256_montjdouble: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(192) + movq 0x40(%rsi), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0x48(%rsi), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x58(%rsi), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x50(%rsi), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq 0x40(%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0x48(%rsi), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0x58(%rsi), %rbx + movq 0x48(%rsi), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0x48(%rsi), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0x50(%rsi), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0x58(%rsi), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + movq 0x20(%rsi), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0x28(%rsi), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x38(%rsi), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x30(%rsi), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq 0x20(%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0x28(%rsi), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0x38(%rsi), %rbx + movq 0x28(%rsi), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0x28(%rsi), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0x30(%rsi), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0x38(%rsi), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x20(%rsp) + movq %r13, 0x28(%rsp) + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq (%rsi), %rax + subq (%rsp), %rax + movq 0x8(%rsi), %rcx + sbbq 0x8(%rsp), %rcx + movq 0x10(%rsi), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x18(%rsi), %r9 + sbbq 0x18(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x60(%rsp) + adcq %r10, %rcx + movq %rcx, 0x68(%rsp) + adcq $0x0, %r8 + movq %r8, 0x70(%rsp) + adcq %rdx, %r9 + movq %r9, 0x78(%rsp) + movq (%rsi), %rax + addq (%rsp), %rax + movq 0x8(%rsi), %rcx + adcq 0x8(%rsp), %rcx + movq 0x10(%rsi), %r8 + adcq 0x10(%rsp), %r8 + movq 0x18(%rsi), %r9 + adcq 0x18(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + subq %r11, %rax + movq %rax, 0x40(%rsp) + sbbq %r10, %rcx + movq %rcx, 0x48(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x50(%rsp) + sbbq %rdx, %r9 + movq %r9, 0x58(%rsp) + movq 0x60(%rsp), %rbx + movq 0x40(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x48(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x50(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x58(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x68(%rsp), %rbx + xorl %r13d, %r13d + movq 0x40(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x48(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x50(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x58(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x70(%rsp), %rbx + xorl %r15d, %r15d + movq 0x40(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x48(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x50(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x58(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x78(%rsp), %rbx + xorl %r8d, %r8d + movq 0x40(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x48(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x50(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x58(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + xorq %r11, %r11 + movq 0x20(%rsi), %rax + addq 0x40(%rsi), %rax + movq 0x28(%rsi), %rcx + adcq 0x48(%rsi), %rcx + movq 0x30(%rsi), %r8 + adcq 0x50(%rsi), %r8 + movq 0x38(%rsi), %r9 + adcq 0x58(%rsi), %r9 + adcq %r11, %r11 + subq $0xffffffffffffffff, %rax + movl $0xffffffff, %r10d + sbbq %r10, %rcx + sbbq $0x0, %r8 + movabsq $0xffffffff00000001, %rdx + sbbq %rdx, %r9 + sbbq $0x0, %r11 + andq %r11, %r10 + andq %r11, %rdx + addq %r11, %rax + movq %rax, 0x40(%rsp) + adcq %r10, %rcx + movq %rcx, 0x48(%rsp) + adcq $0x0, %r8 + movq %r8, 0x50(%rsp) + adcq %rdx, %r9 + movq %r9, 0x58(%rsp) + movq 0x20(%rsp), %rbx + movq (%rsi), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x8(%rsi), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x10(%rsi), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x18(%rsi), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x28(%rsp), %rbx + xorl %r13d, %r13d + movq (%rsi), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x8(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x10(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x18(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x30(%rsp), %rbx + xorl %r15d, %r15d + movq (%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x8(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x10(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x18(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x38(%rsp), %rbx + xorl %r8d, %r8d + movq (%rsi), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x8(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x10(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x18(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq 0x60(%rsp), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0x68(%rsp), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x78(%rsp), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x70(%rsp), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0x68(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0x78(%rsp), %rbx + movq 0x68(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0x68(%rsp), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0x70(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0x78(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0xa0(%rsp) + movq %r13, 0xa8(%rsp) + movq %r14, 0xb0(%rsp) + movq %r15, 0xb8(%rsp) + movq 0x40(%rsp), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0x48(%rsp), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x58(%rsp), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x50(%rsp), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq 0x40(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0x48(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0x58(%rsp), %rbx + movq 0x48(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0x48(%rsp), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0x50(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0x58(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x40(%rsp) + movq %r13, 0x48(%rsp) + movq %r14, 0x50(%rsp) + movq %r15, 0x58(%rsp) + movq $0xffffffffffffffff, %r9 + xorl %r11d, %r11d + subq 0xa0(%rsp), %r9 + movabsq $0xffffffff, %r10 + sbbq 0xa8(%rsp), %r10 + sbbq 0xb0(%rsp), %r11 + movabsq $0xffffffff00000001, %r12 + sbbq 0xb8(%rsp), %r12 + movq $0x9, %rcx + movq %r9, %rax + mulq %rcx + movq %rax, %r8 + movq %rdx, %r9 + movq %r10, %rax + xorl %r10d, %r10d + mulq %rcx + addq %rax, %r9 + adcq %rdx, %r10 + movq %r11, %rax + xorl %r11d, %r11d + mulq %rcx + addq %rax, %r10 + adcq %rdx, %r11 + movq %r12, %rax + xorl %r12d, %r12d + mulq %rcx + addq %rax, %r11 + adcq %rdx, %r12 + movl $0xc, %ecx + movq 0x80(%rsp), %rax + mulq %rcx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %rbx, %rbx + movq 0x88(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rbx, %rbx + movq 0x90(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbx, %rbx + movq 0x98(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + leaq 0x1(%r12), %rcx + movabsq $0xffffffff00000001, %rax + mulq %rcx + movq %rcx, %rbx + shlq $0x20, %rbx + addq %rcx, %r8 + sbbq $0x0, %rbx + subq %rbx, %r9 + sbbq $0x0, %r10 + sbbq %rax, %r11 + sbbq %rdx, %rcx + decq %rcx + movl $0xffffffff, %eax + andq %rcx, %rax + xorl %edx, %edx + subq %rax, %rdx + addq %rcx, %r8 + movq %r8, 0xa0(%rsp) + adcq %rax, %r9 + movq %r9, 0xa8(%rsp) + adcq $0x0, %r10 + movq %r10, 0xb0(%rsp) + adcq %rdx, %r11 + movq %r11, 0xb8(%rsp) + movq 0x40(%rsp), %rax + subq (%rsp), %rax + movq 0x48(%rsp), %rcx + sbbq 0x8(%rsp), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x40(%rsp) + adcq %r10, %rcx + movq %rcx, 0x48(%rsp) + adcq $0x0, %r8 + movq %r8, 0x50(%rsp) + adcq %rdx, %r9 + movq %r9, 0x58(%rsp) + movq 0x20(%rsp), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0x28(%rsp), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x38(%rsp), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x30(%rsp), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq 0x20(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0x28(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0x38(%rsp), %rbx + movq 0x28(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0x28(%rsp), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0x30(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0x38(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + movq 0x60(%rsp), %rbx + movq 0xa0(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0xb0(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0xb8(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x68(%rsp), %rbx + xorl %r13d, %r13d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x70(%rsp), %rbx + xorl %r15d, %r15d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x78(%rsp), %rbx + xorl %r8d, %r8d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + movq 0x40(%rsp), %rax + subq 0x20(%rsp), %rax + movq 0x48(%rsp), %rcx + sbbq 0x28(%rsp), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x30(%rsp), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x38(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x40(%rdi) + adcq %r10, %rcx + movq %rcx, 0x48(%rdi) + adcq $0x0, %r8 + movq %r8, 0x50(%rdi) + adcq %rdx, %r9 + movq %r9, 0x58(%rdi) + movq 0x98(%rsp), %r11 + movq %r11, %rcx + movq 0x90(%rsp), %r10 + shldq $0x2, %r10, %r11 + movq 0x88(%rsp), %r9 + shldq $0x2, %r9, %r10 + movq 0x80(%rsp), %r8 + shldq $0x2, %r8, %r9 + shlq $0x2, %r8 + shrq $0x3e, %rcx + addq $0x1, %rcx + subq 0xa0(%rsp), %r8 + sbbq 0xa8(%rsp), %r9 + sbbq 0xb0(%rsp), %r10 + sbbq 0xb8(%rsp), %r11 + sbbq $0x0, %rcx + movabsq $0xffffffff00000001, %rax + mulq %rcx + movq %rcx, %rbx + shlq $0x20, %rbx + addq %rcx, %r8 + sbbq $0x0, %rbx + subq %rbx, %r9 + sbbq $0x0, %r10 + sbbq %rax, %r11 + sbbq %rdx, %rcx + decq %rcx + movl $0xffffffff, %eax + andq %rcx, %rax + xorl %edx, %edx + subq %rax, %rdx + addq %rcx, %r8 + movq %r8, (%rdi) + adcq %rax, %r9 + movq %r9, 0x8(%rdi) + adcq $0x0, %r10 + movq %r10, 0x10(%rdi) + adcq %rdx, %r11 + movq %r11, 0x18(%rdi) + movq $0xffffffffffffffff, %r8 + xorl %r10d, %r10d + subq (%rsp), %r8 + movabsq $0xffffffff, %r9 + sbbq 0x8(%rsp), %r9 + sbbq 0x10(%rsp), %r10 + movabsq $0xffffffff00000001, %r11 + sbbq 0x18(%rsp), %r11 + movq %r11, %r12 + shldq $0x3, %r10, %r11 + shldq $0x3, %r9, %r10 + shldq $0x3, %r8, %r9 + shlq $0x3, %r8 + shrq $0x3d, %r12 + movl $0x3, %ecx + movq 0x60(%rsp), %rax + mulq %rcx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %rbx, %rbx + movq 0x68(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rbx, %rbx + movq 0x70(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbx, %rbx + movq 0x78(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + leaq 0x1(%r12), %rcx + movabsq $0xffffffff00000001, %rax + mulq %rcx + movq %rcx, %rbx + shlq $0x20, %rbx + addq %rcx, %r8 + sbbq $0x0, %rbx + subq %rbx, %r9 + sbbq $0x0, %r10 + sbbq %rax, %r11 + sbbq %rdx, %rcx + decq %rcx + movl $0xffffffff, %eax + andq %rcx, %rax + xorl %edx, %edx + subq %rax, %rdx + addq %rcx, %r8 + movq %r8, 0x20(%rdi) + adcq %rax, %r9 + movq %r9, 0x28(%rdi) + adcq $0x0, %r10 + movq %r10, 0x30(%rdi) + adcq %rdx, %r11 + movq %r11, 0x38(%rdi) + CFI_INC_RSP(192) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + CFI_RET + +Lp256_scalarmul_alt_local_p256_montjmixadd: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(192) + movq %rdx, %rbp + movq 0x40(%rsi), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0x48(%rsi), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x58(%rsi), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x50(%rsi), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq 0x40(%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0x48(%rsi), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0x58(%rsi), %rbx + movq 0x48(%rsi), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0x48(%rsi), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0x50(%rsi), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0x58(%rsi), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + movq 0x20(%rbp), %rbx + movq 0x40(%rsi), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x48(%rsi), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x50(%rsi), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x58(%rsi), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x28(%rbp), %rbx + xorl %r13d, %r13d + movq 0x40(%rsi), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x48(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x50(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x58(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x30(%rbp), %rbx + xorl %r15d, %r15d + movq 0x40(%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x48(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x50(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x58(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x38(%rbp), %rbx + xorl %r8d, %r8d + movq 0x40(%rsi), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x48(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x50(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x58(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x20(%rsp) + movq %r13, 0x28(%rsp) + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq 0x0(%rbp), %rbx + movq (%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x10(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x18(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x8(%rbp), %rbx + xorl %r13d, %r13d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x10(%rbp), %rbx + xorl %r15d, %r15d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x18(%rbp), %rbx + xorl %r8d, %r8d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x40(%rsp) + movq %r13, 0x48(%rsp) + movq %r14, 0x50(%rsp) + movq %r15, 0x58(%rsp) + movq 0x20(%rsp), %rbx + movq (%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x10(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x18(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x28(%rsp), %rbx + xorl %r13d, %r13d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x30(%rsp), %rbx + xorl %r15d, %r15d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x38(%rsp), %rbx + xorl %r8d, %r8d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x20(%rsp) + movq %r13, 0x28(%rsp) + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq 0x40(%rsp), %rax + subq (%rsi), %rax + movq 0x48(%rsp), %rcx + sbbq 0x8(%rsi), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x10(%rsi), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x18(%rsi), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0xa0(%rsp) + adcq %r10, %rcx + movq %rcx, 0xa8(%rsp) + adcq $0x0, %r8 + movq %r8, 0xb0(%rsp) + adcq %rdx, %r9 + movq %r9, 0xb8(%rsp) + movq 0x20(%rsp), %rax + subq 0x20(%rsi), %rax + movq 0x28(%rsp), %rcx + sbbq 0x28(%rsi), %rcx + movq 0x30(%rsp), %r8 + sbbq 0x30(%rsi), %r8 + movq 0x38(%rsp), %r9 + sbbq 0x38(%rsi), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x20(%rsp) + adcq %r10, %rcx + movq %rcx, 0x28(%rsp) + adcq $0x0, %r8 + movq %r8, 0x30(%rsp) + adcq %rdx, %r9 + movq %r9, 0x38(%rsp) + movq 0xa0(%rsp), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0xa8(%rsp), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0xb8(%rsp), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0xb0(%rsp), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0xa8(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0xb8(%rsp), %rbx + movq 0xa8(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0xa8(%rsp), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0xb0(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0xb8(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + movq 0x20(%rsp), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0x28(%rsp), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x38(%rsp), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x30(%rsp), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq 0x20(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0x28(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0x38(%rsp), %rbx + movq 0x28(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0x28(%rsp), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0x30(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0x38(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + movq (%rsi), %rbx + movq 0x60(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x70(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x78(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x8(%rsi), %rbx + xorl %r13d, %r13d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x10(%rsi), %rbx + xorl %r15d, %r15d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x18(%rsi), %rbx + xorl %r8d, %r8d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq 0x40(%rsp), %rbx + movq 0x60(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x70(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x78(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x48(%rsp), %rbx + xorl %r13d, %r13d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x50(%rsp), %rbx + xorl %r15d, %r15d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x58(%rsp), %rbx + xorl %r8d, %r8d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x40(%rsp) + movq %r13, 0x48(%rsp) + movq %r14, 0x50(%rsp) + movq %r15, 0x58(%rsp) + movq (%rsp), %rax + subq 0x80(%rsp), %rax + movq 0x8(%rsp), %rcx + sbbq 0x88(%rsp), %rcx + movq 0x10(%rsp), %r8 + sbbq 0x90(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x98(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, (%rsp) + adcq %r10, %rcx + movq %rcx, 0x8(%rsp) + adcq $0x0, %r8 + movq %r8, 0x10(%rsp) + adcq %rdx, %r9 + movq %r9, 0x18(%rsp) + movq 0x40(%rsp), %rax + subq 0x80(%rsp), %rax + movq 0x48(%rsp), %rcx + sbbq 0x88(%rsp), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x90(%rsp), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x98(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x60(%rsp) + adcq %r10, %rcx + movq %rcx, 0x68(%rsp) + adcq $0x0, %r8 + movq %r8, 0x70(%rsp) + adcq %rdx, %r9 + movq %r9, 0x78(%rsp) + movq 0x40(%rsi), %rbx + movq 0xa0(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0xb0(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0xb8(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x48(%rsi), %rbx + xorl %r13d, %r13d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x50(%rsi), %rbx + xorl %r15d, %r15d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x58(%rsi), %rbx + xorl %r8d, %r8d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0xa0(%rsp) + movq %r13, 0xa8(%rsp) + movq %r14, 0xb0(%rsp) + movq %r15, 0xb8(%rsp) + movq (%rsp), %rax + subq 0x40(%rsp), %rax + movq 0x8(%rsp), %rcx + sbbq 0x48(%rsp), %rcx + movq 0x10(%rsp), %r8 + sbbq 0x50(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x58(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, (%rsp) + adcq %r10, %rcx + movq %rcx, 0x8(%rsp) + adcq $0x0, %r8 + movq %r8, 0x10(%rsp) + adcq %rdx, %r9 + movq %r9, 0x18(%rsp) + movq 0x80(%rsp), %rax + subq (%rsp), %rax + movq 0x88(%rsp), %rcx + sbbq 0x8(%rsp), %rcx + movq 0x90(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x98(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x80(%rsp) + adcq %r10, %rcx + movq %rcx, 0x88(%rsp) + adcq $0x0, %r8 + movq %r8, 0x90(%rsp) + adcq %rdx, %r9 + movq %r9, 0x98(%rsp) + movq 0x20(%rsi), %rbx + movq 0x60(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x70(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x78(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x28(%rsi), %rbx + xorl %r13d, %r13d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x30(%rsi), %rbx + xorl %r15d, %r15d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x38(%rsi), %rbx + xorl %r8d, %r8d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + movq 0x80(%rsp), %rbx + movq 0x20(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x28(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x30(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x38(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x88(%rsp), %rbx + xorl %r13d, %r13d + movq 0x20(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x30(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x38(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x90(%rsp), %rbx + xorl %r15d, %r15d + movq 0x20(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x30(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x38(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x98(%rsp), %rbx + xorl %r8d, %r8d + movq 0x20(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x30(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x38(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq 0x80(%rsp), %rax + subq 0x60(%rsp), %rax + movq 0x88(%rsp), %rcx + sbbq 0x68(%rsp), %rcx + movq 0x90(%rsp), %r8 + sbbq 0x70(%rsp), %r8 + movq 0x98(%rsp), %r9 + sbbq 0x78(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x80(%rsp) + adcq %r10, %rcx + movq %rcx, 0x88(%rsp) + adcq $0x0, %r8 + movq %r8, 0x90(%rsp) + adcq %rdx, %r9 + movq %r9, 0x98(%rsp) + movq 0x40(%rsi), %rax + movq 0x48(%rsi), %rdx + orq 0x50(%rsi), %rax + orq 0x58(%rsi), %rdx + orq %rdx, %rax + movq (%rsp), %r8 + movq 0x0(%rbp), %rax + cmoveq %rax, %r8 + movq 0x8(%rsp), %r9 + movq 0x8(%rbp), %rax + cmoveq %rax, %r9 + movq 0x10(%rsp), %r10 + movq 0x10(%rbp), %rax + cmoveq %rax, %r10 + movq 0x18(%rsp), %r11 + movq 0x18(%rbp), %rax + cmoveq %rax, %r11 + movq 0x80(%rsp), %r12 + movq 0x20(%rbp), %rax + cmoveq %rax, %r12 + movq 0x88(%rsp), %r13 + movq 0x28(%rbp), %rax + cmoveq %rax, %r13 + movq 0x90(%rsp), %r14 + movq 0x30(%rbp), %rax + cmoveq %rax, %r14 + movq 0x98(%rsp), %r15 + movq 0x38(%rbp), %rax + cmoveq %rax, %r15 + movq %r8, (%rdi) + movq %r9, 0x8(%rdi) + movq %r10, 0x10(%rdi) + movq %r11, 0x18(%rdi) + movq %r12, 0x20(%rdi) + movq %r13, 0x28(%rdi) + movq %r14, 0x30(%rdi) + movq %r15, 0x38(%rdi) + movq 0xa0(%rsp), %r8 + movq 0xa8(%rsp), %r9 + movq 0xb0(%rsp), %r10 + movq 0xb8(%rsp), %r11 + movl $0x1, %eax + cmoveq %rax, %r8 + movabsq $0xffffffff00000000, %rax + cmoveq %rax, %r9 + movq $0xffffffffffffffff, %rax + cmoveq %rax, %r10 + movl $0xfffffffe, %eax + cmoveq %rax, %r11 + movq %r8, 0x40(%rdi) + movq %r9, 0x48(%rdi) + movq %r10, 0x50(%rdi) + movq %r11, 0x58(%rdi) + CFI_INC_RSP(192) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/p256_scalarmulbase.S b/cbits/s2n/x86_att/p256_scalarmulbase.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/p256_scalarmulbase.S @@ -0,0 +1,3571 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Scalar multiplication for precomputed point on NIST curve P-256 +// Input scalar[4], blocksize, table[]; output res[8] +// +// extern void p256_scalarmulbase +// (uint64_t res[static 8], +// const uint64_t scalar[static 4], +// uint64_t blocksize, +// const uint64_t *table); +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-256, the input argument "table" is expected to be a table of +// multiples of the point P in Montgomery-affine form, with each block +// corresponding to "blocksize" bits of the scalar as follows, where +// B = 2^{blocksize-1} (e.g. B = 8 for blocksize = 4): +// +// For each i,j with blocksize * i <= 256 and 1 <= j <= B +// the multiple 2^{blocksize * i} * j * P is stored at +// tab[8 * (B * i + (j - 1))], considered as uint64_t pointers +// or tab + 64 * (B * i + (j - 1)) as byte pointers. +// +// Standard x86-64 ABI: RDI = res, RSI = scalar, RDX = blocksize, RCX = table +// Microsoft x64 ABI: RCX = res, RDX = scalar, R8 = blocksize, R9 = table +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_scalarmulbase) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_scalarmulbase) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_scalarmulbase) + + + .text + .balign 32 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Intermediate variables on the stack. The last z2, z3 values can +// safely be overlaid on "nacc", which is no longer needed at the end. +// Uppercase syntactic variants make x86_att version simpler to generate + +#define RSCALAR (0*NUMSIZE) +#define ACC (1*NUMSIZE) +#define NACC (4*NUMSIZE) +#define TABENT (7*NUMSIZE) +#define Z2 (4*NUMSIZE) +#define Z3 (5*NUMSIZE) + +#define rscalar RSCALAR(%rsp) +#define acc ACC(%rsp) +#define nacc NACC(%rsp) +#define tabent TABENT(%rsp) + +#define z2 Z2(%rsp) +#define z3 Z3(%rsp) + +#define res (9*NUMSIZE)(%rsp) +#define blocksize (9*NUMSIZE+8)(%rsp) +#define table (9*NUMSIZE+16)(%rsp) +#define i (9*NUMSIZE+24)(%rsp) +#define bf (9*NUMSIZE+32)(%rsp) +#define cf (9*NUMSIZE+40)(%rsp) +#define j (9*NUMSIZE+48)(%rsp) + +#define NSPACE 11*NUMSIZE + +S2N_BN_SYMBOL(p256_scalarmulbase): + CFI_START + _CET_ENDBR + +// The Windows version literally calls the standard ABI version. +// This simplifies the proofs since subroutine offsets are fixed. + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + movq %r9, %rcx + CFI_CALL(Lp256_scalarmulbase_standard) + CFI_POP(%rsi) + CFI_POP(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_scalarmulbase) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_standard) + +Lp256_scalarmulbase_standard: + CFI_START +#endif + +// Real start of the standard ABI code. + + CFI_PUSH(%r15) + CFI_PUSH(%r14) + CFI_PUSH(%r13) + CFI_PUSH(%r12) + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + + CFI_DEC_RSP(NSPACE) + +// Preserve the input arguments except the scalar, since that gets absorbed +// immediately. The "table" value subsequently gets shifted up each iteration +// of the loop, while "res" and "blocksize" are static throughout. + + movq %rdi, res + movq %rdx, blocksize + movq %rcx, table + +// Load the digits of group order n_256 = [%r15;%r14;%r13;%r12] + + movq $0xf3b9cac2fc632551, %r12 + movq $0xbce6faada7179e84, %r13 + movq $0xffffffffffffffff, %r14 + movq $0xffffffff00000000, %r15 + +// First, reduce the input scalar mod n_256, i.e. conditionally subtract n_256 +// Store it to "rscalar" (reduced scalar) + + movq (%rsi), %r8 + subq %r12, %r8 + movq 8(%rsi), %r9 + sbbq %r13, %r9 + movq 16(%rsi), %r10 + sbbq %r14, %r10 + movq 24(%rsi), %r11 + sbbq %r15, %r11 + + cmovcq (%rsi), %r8 + cmovcq 8(%rsi), %r9 + cmovcq 16(%rsi), %r10 + cmovcq 24(%rsi), %r11 + + movq %r8, RSCALAR(%rsp) + movq %r9, RSCALAR+8(%rsp) + movq %r10, RSCALAR+16(%rsp) + movq %r11, RSCALAR+24(%rsp) + +// Initialize the accumulator to all zeros and the "carry flag" cf to 0 + + xorl %eax, %eax + + movq %rax, ACC(%rsp) + movq %rax, ACC+8(%rsp) + movq %rax, ACC+16(%rsp) + movq %rax, ACC+24(%rsp) + movq %rax, ACC+32(%rsp) + movq %rax, ACC+40(%rsp) + movq %rax, ACC+48(%rsp) + movq %rax, ACC+56(%rsp) + movq %rax, ACC+64(%rsp) + movq %rax, ACC+72(%rsp) + movq %rax, ACC+80(%rsp) + movq %rax, ACC+88(%rsp) + + movq %rax, cf + +// Main loop over {i >= 0 | blocksize * i <= 256}. Note the non-strict +// inequality, to allow top carry for any choices of blocksize. + + movq %rax, i + +Lp256_scalarmulbase_loop: + +// The next raw bitfield is bf = bitfield(blocksize * i,blocksize) + cf, +// adding in the deferred carry cf. We then shift the whole scalar right +// by blocksize so we can keep picking bitfield(0,blocksize). + + movq RSCALAR(%rsp), %r8 + movq RSCALAR+8(%rsp), %r9 + movq RSCALAR+16(%rsp), %r10 + movq RSCALAR+24(%rsp), %r11 + + movq blocksize, %rcx + movl $1, %eax + shlq %cl, %rax + decq %rax + andq %r8, %rax + + shrdq %cl, %r9, %r8 + shrdq %cl, %r10, %r9 + shrdq %cl, %r11, %r10 + shrq %cl, %r11 + + addq cf, %rax + movq %rax, bf + + movq %r8, RSCALAR(%rsp) + movq %r9, RSCALAR+8(%rsp) + movq %r10, RSCALAR+16(%rsp) + movq %r11, RSCALAR+24(%rsp) + +// Now if bf <= B we just select entry j, unnegated and set cf = 0. +// If bf > B we set j = 2 * B - bf and negate the j'th entry, setting cf = 1. +// In either case we ultimately add bf, in the latter case with deferred +// carry as 2 * B - (2 * B - bf) = bf. + + movl $1, %eax + movq blocksize, %rcx + shlq %cl, %rax + movq %rax, %rbx + shrq $1, %rax + + subq bf, %rbx + cmpq bf, %rax + + cmovncq bf, %rbx + sbbq %rax, %rax + movq %rbx, j + negq %rax + movq %rax, cf + +// Load table entry j - 1 for nonzero j in constant-time style. + + movq blocksize, %rcx + decq %rcx + movl $1, %esi + shlq %cl, %rsi + movq j, %r12 + movq table, %rbp + +Lp256_scalarmulbase_tabloop: + subq $1, %r12 + cmovzq (%rbp), %rax + cmovzq 8(%rbp), %rbx + cmovzq 16(%rbp), %rcx + cmovzq 24(%rbp), %rdx + cmovzq 32(%rbp), %r8 + cmovzq 40(%rbp), %r9 + cmovzq 48(%rbp), %r10 + cmovzq 56(%rbp), %r11 + + addq $64, %rbp + decq %rsi + jnz Lp256_scalarmulbase_tabloop + + movq %rbp, table + +// Before storing back, optionally negate the y coordinate of the table entry + + xorl %r14d, %r14d + leaq -1(%r14), %r12 + movq $0x00000000ffffffff, %r15 + movq %r15, %r13 + negq %r15 + + subq %r8, %r12 + sbbq %r9, %r13 + sbbq %r10, %r14 + sbbq %r11, %r15 + + movq %rax, TABENT(%rsp) + movq %rbx, TABENT+8(%rsp) + movq %rcx, TABENT+16(%rsp) + movq %rdx, TABENT+24(%rsp) + + movq cf, %rax + testq %rax, %rax + cmovnzq %r12, %r8 + cmovnzq %r13, %r9 + cmovnzq %r14, %r10 + cmovnzq %r15, %r11 + + movq %r8, TABENT+32(%rsp) + movq %r9, TABENT+40(%rsp) + movq %r10, TABENT+48(%rsp) + movq %r11, TABENT+56(%rsp) + +// Add the adjusted table point to the accumulator + + leaq NACC(%rsp), %rdi + leaq ACC(%rsp), %rsi + leaq TABENT(%rsp), %rdx + CFI_CALL(Lp256_scalarmulbase_local_p256_montjmixadd) + +// However, only commit that update to the accumulator if j is nonzero, +// because the mixed addition function does not handle this case directly, +// and in any case we didn't choose the table entry appropriately. + + movq j, %rax + testq %rax, %rax + + movq ACC(%rsp), %rax + cmovnzq NACC(%rsp), %rax + movq %rax, ACC(%rsp) + + movq ACC+8(%rsp), %rax + cmovnzq NACC+8(%rsp), %rax + movq %rax, ACC+8(%rsp) + + movq ACC+16(%rsp), %rax + cmovnzq NACC+16(%rsp), %rax + movq %rax, ACC+16(%rsp) + + movq ACC+24(%rsp), %rax + cmovnzq NACC+24(%rsp), %rax + movq %rax, ACC+24(%rsp) + + movq ACC+32(%rsp), %rax + cmovnzq NACC+32(%rsp), %rax + movq %rax, ACC+32(%rsp) + + movq ACC+40(%rsp), %rax + cmovnzq NACC+40(%rsp), %rax + movq %rax, ACC+40(%rsp) + + movq ACC+48(%rsp), %rax + cmovnzq NACC+48(%rsp), %rax + movq %rax, ACC+48(%rsp) + + movq ACC+56(%rsp), %rax + cmovnzq NACC+56(%rsp), %rax + movq %rax, ACC+56(%rsp) + + movq ACC+64(%rsp), %rax + cmovnzq NACC+64(%rsp), %rax + movq %rax, ACC+64(%rsp) + + movq ACC+72(%rsp), %rax + cmovnzq NACC+72(%rsp), %rax + movq %rax, ACC+72(%rsp) + + movq ACC+80(%rsp), %rax + cmovnzq NACC+80(%rsp), %rax + movq %rax, ACC+80(%rsp) + + movq ACC+88(%rsp), %rax + cmovnzq NACC+88(%rsp), %rax + movq %rax, ACC+88(%rsp) + +// Loop while blocksize * i <= 256 + + movq i, %rax + incq %rax + movq %rax, i + + imulq blocksize, %rax + cmpq $257, %rax + jc Lp256_scalarmulbase_loop + +// That's the end of the main loop, and we just need to translate +// back from the Jacobian representation to affine. First of all, +// let z2 = 1/z^2 and z3 = 1/z^3, both without Montgomery form + + leaq Z2(%rsp), %rdi + leaq ACC+64(%rsp), %rsi + CFI_CALL(Lp256_scalarmulbase_local_montsqr_p256) + + leaq Z3(%rsp), %rdi + leaq ACC+64(%rsp), %rsi + leaq Z2(%rsp), %rdx + CFI_CALL(Lp256_scalarmulbase_local_montmul_p256) + + leaq Z2(%rsp), %rdi + leaq Z3(%rsp), %rsi + CFI_CALL(Lp256_scalarmulbase_local_demont_p256) + + leaq Z3(%rsp), %rdi + leaq Z2(%rsp), %rsi + CFI_CALL(Lp256_scalarmulbase_local_inv_p256) + + leaq Z2(%rsp), %rdi + leaq ACC+64(%rsp), %rsi + leaq Z3(%rsp), %rdx + CFI_CALL(Lp256_scalarmulbase_local_montmul_p256) + +// Convert back from Jacobian (X, Y, Z) |-> (X/Z^2, Y/Z^3) + + movq res, %rdi + leaq ACC(%rsp), %rsi + leaq Z2(%rsp), %rdx + movq %rdi, %rbx + CFI_CALL(Lp256_scalarmulbase_local_montmul_p256) + + leaq 32(%rbx), %rdi + leaq ACC+32(%rsp), %rsi + leaq Z3(%rsp), %rdx + CFI_CALL(Lp256_scalarmulbase_local_montmul_p256) + +// Restore stack and registers and return + + CFI_INC_RSP(NSPACE) + CFI_POP(%rbx) + CFI_POP(%rbp) + CFI_POP(%r12) + CFI_POP(%r13) + CFI_POP(%r14) + CFI_POP(%r15) + CFI_RET + +#if WINDOWS_ABI +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_standard) +#else +S2N_BN_SIZE_DIRECTIVE(p256_scalarmulbase) +#endif + +// Local copies of subroutines, complete clones at the moment + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_local_demont_p256) + +Lp256_scalarmulbase_local_demont_p256: + CFI_START + CFI_PUSH(%rbx) + movq (%rsi), %r8 + movq 0x8(%rsi), %r9 + movq 0x10(%rsi), %r10 + movq 0x18(%rsi), %r11 + xorq %rbx, %rbx + xorq %rsi, %rsi + movq $0x100000000, %rdx + mulxq %r8, %rax, %rcx + adcxq %rax, %r9 + adoxq %rcx, %r10 + mulxq %r9, %rax, %rcx + adcxq %rax, %r10 + adoxq %rcx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rcx + adcxq %rax, %r11 + adoxq %rcx, %rbx + mulxq %r9, %rax, %rcx + adcxq %rax, %rbx + adoxq %rcx, %rsi + movl $0x0, %r8d + adcxq %r8, %rsi + xorq %r9, %r9 + movq $0x100000000, %rdx + mulxq %r10, %rax, %rcx + adcxq %rax, %r11 + adoxq %rcx, %rbx + mulxq %r11, %rax, %rcx + adcxq %rax, %rbx + adoxq %rcx, %rsi + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rcx + adcxq %rax, %rsi + adoxq %rcx, %r8 + mulxq %r11, %rax, %rcx + adcxq %rax, %r8 + adoxq %rcx, %r9 + movl $0x0, %r10d + adcxq %r10, %r9 + movq %rbx, (%rdi) + movq %rsi, 0x8(%rdi) + movq %r8, 0x10(%rdi) + movq %r9, 0x18(%rdi) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_local_demont_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_local_inv_p256) + +Lp256_scalarmulbase_local_inv_p256: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(240) + movq %rdi, 0xe0(%rsp) + xorl %ecx, %ecx + movl $0xffffffff, %edx + movq %rdx, %rbx + leaq -0x1(%rcx), %rax + negq %rdx + movq %rax, (%rsp) + movq %rbx, 0x8(%rsp) + movq %rcx, 0x10(%rsp) + movq %rdx, 0x18(%rsp) + movq %rcx, 0x20(%rsp) + movq (%rsi), %r8 + movq 0x8(%rsi), %r9 + movq 0x10(%rsi), %r10 + movq 0x18(%rsi), %r11 + leaq 0x1(%rcx), %rax + addq %r8, %rax + leaq -0x1(%rdx), %rbx + adcq %r9, %rbx + notq %rcx + adcq %r10, %rcx + notq %rdx + adcq %r11, %rdx + cmovaeq %r8, %rax + cmovaeq %r9, %rbx + cmovaeq %r10, %rcx + cmovaeq %r11, %rdx + movq %rax, 0x28(%rsp) + movq %rbx, 0x30(%rsp) + movq %rcx, 0x38(%rsp) + movq %rdx, 0x40(%rsp) + xorl %eax, %eax + movq %rax, 0x48(%rsp) + xorl %eax, %eax + movq %rax, 0x50(%rsp) + movq %rax, 0x58(%rsp) + movq %rax, 0x60(%rsp) + movq %rax, 0x68(%rsp) + movq $0x4000000000000, %rcx + movq %rcx, 0x78(%rsp) + movq %rax, 0x80(%rsp) + movq %rax, 0x88(%rsp) + movq %rax, 0x90(%rsp) + movq $0xa, 0xb0(%rsp) + movq $0x1, 0xb8(%rsp) + jmp Lp256_scalarmulbase_inv_midloop +Lp256_scalarmulbase_inv_loop: + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + movq %r8, %rax + andq %r9, %rax + movq %r10, %rdi + andq %r11, %rdi + addq %rax, %rdi + movq %rdi, 0xa0(%rsp) + movq %r12, %rax + andq %r13, %rax + movq %r14, %rsi + andq %r15, %rsi + addq %rax, %rsi + movq %rsi, 0xa8(%rsp) + xorl %ebx, %ebx + movq (%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x28(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rdi + adcq %rdx, %rbx + xorl %ebp, %ebp + movq (%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x28(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + xorl %ecx, %ecx + movq 0x8(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x30(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + shrdq $0x3b, %rbx, %rdi + movq %rdi, (%rsp) + xorl %edi, %edi + movq 0x8(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rbp + adcq %rdx, %rdi + movq 0x30(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rdi + shrdq $0x3b, %rbp, %rsi + movq %rsi, 0x28(%rsp) + xorl %esi, %esi + movq 0x10(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rsi + movq 0x38(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rsi + shrdq $0x3b, %rcx, %rbx + movq %rbx, 0x8(%rsp) + xorl %ebx, %ebx + movq 0x10(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x38(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rdi + adcq %rdx, %rbx + shrdq $0x3b, %rdi, %rbp + movq %rbp, 0x30(%rsp) + movq 0x18(%rsp), %rax + xorq %r9, %rax + movq 0x20(%rsp), %rbp + xorq %r9, %rbp + andq %r8, %rbp + negq %rbp + mulq %r8 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x40(%rsp), %rax + xorq %r11, %rax + movq 0x48(%rsp), %rdx + xorq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbp + mulq %r10 + addq %rax, %rsi + adcq %rdx, %rbp + shrdq $0x3b, %rsi, %rcx + movq %rcx, 0x10(%rsp) + shrdq $0x3b, %rbp, %rsi + sarq $0x3b, %rbp + movq 0x18(%rsp), %rax + movq %rsi, 0x18(%rsp) + movq 0x20(%rsp), %rsi + movq %rbp, 0x20(%rsp) + xorq %r13, %rax + xorq %r13, %rsi + andq %r12, %rsi + negq %rsi + mulq %r12 + addq %rax, %rbx + adcq %rdx, %rsi + movq 0x40(%rsp), %rax + xorq %r15, %rax + movq 0x48(%rsp), %rdx + xorq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rsi + mulq %r14 + addq %rax, %rbx + adcq %rdx, %rsi + shrdq $0x3b, %rbx, %rdi + movq %rdi, 0x38(%rsp) + shrdq $0x3b, %rsi, %rbx + movq %rbx, 0x40(%rsp) + sarq $0x3b, %rsi + movq %rsi, 0x48(%rsp) + movq 0xa0(%rsp), %rbx + movq 0xa8(%rsp), %rbp + xorl %ecx, %ecx + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x78(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x50(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x50(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x78(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x78(%rsp) + xorl %ebx, %ebx + movq 0x58(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x80(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + xorl %ebp, %ebp + movq 0x58(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, 0x58(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x80(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, 0x80(%rsp) + xorl %ecx, %ecx + movq 0x60(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x88(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x60(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x60(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x88(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x88(%rsp) + movq 0x68(%rsp), %rax + xorq %r9, %rax + movq %r9, %rbx + andq %r8, %rbx + negq %rbx + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x90(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbx + mulq %r10 + addq %rax, %rcx + adcq %rbx, %rdx + movq 0x68(%rsp), %rax + movq %rcx, 0x68(%rsp) + movq %rdx, 0x70(%rsp) + xorq %r13, %rax + movq %r13, %rcx + andq %r12, %rcx + negq %rcx + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rcx + movq 0x90(%rsp), %rax + xorq %r15, %rax + movq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rcx + mulq %r14 + addq %rax, %rsi + adcq %rcx, %rdx + movq %rsi, 0x90(%rsp) + movq %rdx, 0x98(%rsp) + movq $0xe000000000000000, %r8 + addq 0x50(%rsp), %r8 + movq $0xffffffffffffffff, %r9 + adcq 0x58(%rsp), %r9 + movq $0x1fffffff, %r10 + adcq 0x60(%rsp), %r10 + movq $0x2000000000000000, %r11 + adcq 0x68(%rsp), %r11 + movq $0x1fffffffe0000000, %r12 + adcq 0x70(%rsp), %r12 + movq %r8, %rbx + shlq $0x20, %rbx + movq $0xffffffff00000001, %rax + mulq %r8 + shrq $0x20, %r8 + addq %rbx, %r9 + adcq %r8, %r10 + adcq %rax, %r11 + adcq %rdx, %r12 + sbbq %rax, %rax + movl $0xffffffff, %ebx + andq %rax, %rbx + movq $0xffffffff00000001, %rdx + andq %rax, %rdx + subq %rax, %r9 + movq %r9, 0x50(%rsp) + sbbq %rbx, %r10 + movq %r10, 0x58(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x60(%rsp) + sbbq %rdx, %r12 + movq %r12, 0x68(%rsp) + movq $0xe000000000000000, %r8 + addq 0x78(%rsp), %r8 + movq $0xffffffffffffffff, %r9 + adcq 0x80(%rsp), %r9 + movq $0x1fffffff, %r10 + adcq 0x88(%rsp), %r10 + movq $0x2000000000000000, %r11 + adcq 0x90(%rsp), %r11 + movq $0x1fffffffe0000000, %r12 + adcq 0x98(%rsp), %r12 + movq %r8, %rbx + shlq $0x20, %rbx + movq $0xffffffff00000001, %rax + mulq %r8 + shrq $0x20, %r8 + addq %rbx, %r9 + adcq %r8, %r10 + adcq %rax, %r11 + adcq %rdx, %r12 + sbbq %rax, %rax + movl $0xffffffff, %ebx + andq %rax, %rbx + movq $0xffffffff00000001, %rdx + andq %rax, %rdx + subq %rax, %r9 + movq %r9, 0x78(%rsp) + sbbq %rbx, %r10 + movq %r10, 0x80(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x88(%rsp) + sbbq %rdx, %r12 + movq %r12, 0x90(%rsp) +Lp256_scalarmulbase_inv_midloop: + movq 0xb8(%rsp), %rsi + movq (%rsp), %rdx + movq 0x28(%rsp), %rcx + movq %rdx, %rbx + andq $0xfffff, %rbx + movq $0xfffffe0000000000, %rax + orq %rax, %rbx + andq $0xfffff, %rcx + movq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + xorl %ebp, %ebp + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %rdx + leaq (%rcx,%rax), %rdi + shlq $0x16, %rdx + shlq $0x16, %rdi + sarq $0x2b, %rdx + sarq $0x2b, %rdi + movq $0x20000100000, %rax + leaq (%rbx,%rax), %rbx + leaq (%rcx,%rax), %rcx + sarq $0x2a, %rbx + sarq $0x2a, %rcx + movq %rdx, 0xc0(%rsp) + movq %rbx, 0xc8(%rsp) + movq %rdi, 0xd0(%rsp) + movq %rcx, 0xd8(%rsp) + movq (%rsp), %r12 + imulq %r12, %rdi + imulq %rdx, %r12 + movq 0x28(%rsp), %r13 + imulq %r13, %rbx + imulq %rcx, %r13 + addq %rbx, %r12 + addq %rdi, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r10 + shlq $0x16, %r8 + shlq $0x16, %r10 + sarq $0x2b, %r8 + sarq $0x2b, %r10 + movq $0x20000100000, %rax + leaq (%rbx,%rax), %r15 + leaq (%rcx,%rax), %r11 + sarq $0x2a, %r15 + sarq $0x2a, %r11 + movq %r13, %rbx + movq %r12, %rcx + imulq %r8, %r12 + imulq %r15, %rbx + addq %rbx, %r12 + imulq %r11, %r13 + imulq %r10, %rcx + addq %rcx, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movq $0xc000000000000000, %rax + orq %rax, %rcx + movq 0xc0(%rsp), %rax + imulq %r8, %rax + movq 0xd0(%rsp), %rdx + imulq %r15, %rdx + imulq 0xc8(%rsp), %r8 + imulq 0xd8(%rsp), %r15 + addq %r8, %r15 + leaq (%rax,%rdx), %r9 + movq 0xc0(%rsp), %rax + imulq %r10, %rax + movq 0xd0(%rsp), %rdx + imulq %r11, %rdx + imulq 0xc8(%rsp), %r10 + imulq 0xd8(%rsp), %r11 + addq %r10, %r11 + leaq (%rax,%rdx), %r13 + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r12 + shlq $0x15, %r8 + shlq $0x15, %r12 + sarq $0x2b, %r8 + sarq $0x2b, %r12 + movq $0x20000100000, %rax + leaq (%rbx,%rax), %r10 + leaq (%rcx,%rax), %r14 + sarq $0x2b, %r10 + sarq $0x2b, %r14 + movq %r9, %rax + imulq %r8, %rax + movq %r13, %rdx + imulq %r10, %rdx + imulq %r15, %r8 + imulq %r11, %r10 + addq %r8, %r10 + leaq (%rax,%rdx), %r8 + movq %r9, %rax + imulq %r12, %rax + movq %r13, %rdx + imulq %r14, %rdx + imulq %r15, %r12 + imulq %r11, %r14 + addq %r12, %r14 + leaq (%rax,%rdx), %r12 + movq %rsi, 0xb8(%rsp) + decq 0xb0(%rsp) + jne Lp256_scalarmulbase_inv_loop + movq (%rsp), %rax + movq 0x28(%rsp), %rcx + imulq %r8, %rax + imulq %r10, %rcx + addq %rcx, %rax + sarq $0x3f, %rax + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + xorq %rax, %r9 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + xorq %rax, %r11 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + xorq %rax, %r13 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + xorq %rax, %r15 + movq %r8, %rax + andq %r9, %rax + movq %r10, %r12 + andq %r11, %r12 + addq %rax, %r12 + xorl %r13d, %r13d + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x78(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movq 0x58(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r13 + adcq %rdx, %r14 + movq 0x80(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq 0x60(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x88(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x68(%rsp), %rax + xorq %r9, %rax + andq %r8, %r9 + negq %r9 + mulq %r8 + addq %rax, %r15 + adcq %rdx, %r9 + movq 0x90(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %r9 + mulq %r10 + addq %rax, %r15 + adcq %rdx, %r9 + movq %r12, 0x50(%rsp) + movq %r13, 0x58(%rsp) + movq %r14, 0x60(%rsp) + movq %r15, 0x68(%rsp) + movq %r9, 0x70(%rsp) + movq $0xe000000000000000, %r8 + addq 0x50(%rsp), %r8 + movq $0xffffffffffffffff, %r9 + adcq 0x58(%rsp), %r9 + movq $0x1fffffff, %r10 + adcq 0x60(%rsp), %r10 + movq $0x2000000000000000, %r11 + adcq 0x68(%rsp), %r11 + movq $0x1fffffffe0000000, %r12 + adcq 0x70(%rsp), %r12 + movq %r8, %rbx + shlq $0x20, %rbx + movq $0xffffffff00000001, %rax + mulq %r8 + shrq $0x20, %r8 + addq %rbx, %r9 + adcq %r8, %r10 + adcq %rax, %r11 + adcq %rdx, %r12 + sbbq %rax, %rax + movl $0xffffffff, %ebx + andq %rax, %rbx + movq $0xffffffff00000001, %rdx + andq %rax, %rdx + subq %rax, %r9 + movq %r9, 0x50(%rsp) + sbbq %rbx, %r10 + movq %r10, 0x58(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x60(%rsp) + sbbq %rdx, %r12 + movq %r12, 0x68(%rsp) + movq 0x50(%rsp), %r8 + movq 0x58(%rsp), %r9 + movq 0x60(%rsp), %r10 + movq 0x68(%rsp), %r11 + movl $0x1, %eax + movl $0xffffffff, %ebx + leaq -0x2(%rax), %rcx + leaq -0x1(%rbx), %rdx + notq %rbx + addq %r8, %rax + adcq %r9, %rbx + adcq %r10, %rcx + adcq %r11, %rdx + cmovaeq %r8, %rax + cmovaeq %r9, %rbx + cmovaeq %r10, %rcx + cmovaeq %r11, %rdx + movq 0xe0(%rsp), %rdi + movq %rax, (%rdi) + movq %rbx, 0x8(%rdi) + movq %rcx, 0x10(%rdi) + movq %rdx, 0x18(%rdi) + CFI_INC_RSP(240) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_local_inv_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_local_montmul_p256) + +Lp256_scalarmulbase_local_montmul_p256: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + movq %rdx, %rcx + xorl %r13d, %r13d + movq (%rcx), %rdx + mulxq (%rsi), %r8, %r9 + mulxq 0x8(%rsi), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x10(%rsi), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x18(%rsi), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x8(%rcx), %rdx + xorl %r14d, %r14d + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x10(%rcx), %rdx + xorl %r8d, %r8d + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x18(%rsi), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x18(%rcx), %rdx + xorl %r9d, %r9d + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x18(%rsi), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_local_montmul_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_local_montsqr_p256) + +Lp256_scalarmulbase_local_montsqr_p256: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + movq (%rsi), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0x8(%rsi), %r9, %r10 + mulxq 0x18(%rsi), %r11, %r12 + movq 0x10(%rsi), %rdx + mulxq 0x18(%rsi), %r13, %r14 + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x18(%rsi), %rdx + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rbp, %r13 + adoxq %rbp, %r14 + adcq %rbp, %r14 + xorl %ebp, %ebp + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0x8(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x10(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x18(%rsi), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rbp, %r15 + adoxq %rbp, %r15 + xorl %ebp, %ebp + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rbp, %r13 + movl %ebp, %r9d + adoxq %rbp, %r9 + adcxq %rbp, %r9 + addq %r9, %r14 + adcq %rbp, %r15 + movl %ebp, %r8d + adcq %rbp, %r8 + xorl %ebp, %ebp + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %r8 + adcq %rbp, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rdx), %rdx + adcq %r13, %rdx + leaq -0x1(%rbp), %rbp + movq %rbp, %rax + adcq %r14, %rbp + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %rbp, %r14 + cmovbq %r11, %r15 + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_local_montsqr_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_local_p256_montjmixadd) + +Lp256_scalarmulbase_local_p256_montjmixadd: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(192) + movq %rdx, %rbp + movq 0x40(%rsi), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0x48(%rsi), %r9, %r10 + mulxq 0x58(%rsi), %r11, %r12 + movq 0x50(%rsi), %rdx + mulxq 0x58(%rsi), %r13, %r14 + xorl %ecx, %ecx + mulxq 0x40(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x58(%rsi), %rdx + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + adoxq %rcx, %r14 + adcq %rcx, %r14 + xorl %ecx, %ecx + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0x48(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x50(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x58(%rsi), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rcx, %r15 + adoxq %rcx, %r15 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + movl %ecx, %r9d + adoxq %rcx, %r9 + adcxq %rcx, %r9 + addq %r9, %r14 + adcq %rcx, %r15 + movl %ecx, %r8d + adcq %rcx, %r8 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rcx, %r15 + adoxq %rcx, %r8 + adcq %rcx, %r8 + movl $0x1, %r8d + leaq -0x1(%rdx), %rdx + leaq -0x1(%rcx), %rax + movl $0xfffffffe, %r11d + cmoveq %rcx, %r8 + cmoveq %rcx, %rdx + cmoveq %rcx, %rax + cmoveq %rcx, %r11 + addq %r8, %r12 + adcq %rdx, %r13 + adcq %rax, %r14 + adcq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + xorl %r13d, %r13d + movq 0x20(%rbp), %rdx + mulxq 0x40(%rsi), %r8, %r9 + mulxq 0x48(%rsi), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x50(%rsi), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x58(%rsi), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x28(%rbp), %rdx + xorl %r14d, %r14d + mulxq 0x40(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x50(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x58(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x30(%rbp), %rdx + xorl %r8d, %r8d + mulxq 0x40(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x50(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x58(%rsi), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x38(%rbp), %rdx + xorl %r9d, %r9d + mulxq 0x40(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x50(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x58(%rsi), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x20(%rsp) + movq %r13, 0x28(%rsp) + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + xorl %r13d, %r13d + movq 0x0(%rbp), %rdx + mulxq (%rsp), %r8, %r9 + mulxq 0x8(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x10(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x18(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x8(%rbp), %rdx + xorl %r14d, %r14d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x10(%rbp), %rdx + xorl %r8d, %r8d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x18(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x18(%rbp), %rdx + xorl %r9d, %r9d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x18(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x40(%rsp) + movq %r13, 0x48(%rsp) + movq %r14, 0x50(%rsp) + movq %r15, 0x58(%rsp) + xorl %r13d, %r13d + movq 0x20(%rsp), %rdx + mulxq (%rsp), %r8, %r9 + mulxq 0x8(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x10(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x18(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x28(%rsp), %rdx + xorl %r14d, %r14d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x30(%rsp), %rdx + xorl %r8d, %r8d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x18(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x38(%rsp), %rdx + xorl %r9d, %r9d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x18(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x20(%rsp) + movq %r13, 0x28(%rsp) + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq 0x40(%rsp), %rax + subq (%rsi), %rax + movq 0x48(%rsp), %rcx + sbbq 0x8(%rsi), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x10(%rsi), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x18(%rsi), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0xa0(%rsp) + adcq %r10, %rcx + movq %rcx, 0xa8(%rsp) + adcq $0x0, %r8 + movq %r8, 0xb0(%rsp) + adcq %rdx, %r9 + movq %r9, 0xb8(%rsp) + movq 0x20(%rsp), %rax + subq 0x20(%rsi), %rax + movq 0x28(%rsp), %rcx + sbbq 0x28(%rsi), %rcx + movq 0x30(%rsp), %r8 + sbbq 0x30(%rsi), %r8 + movq 0x38(%rsp), %r9 + sbbq 0x38(%rsi), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x20(%rsp) + adcq %r10, %rcx + movq %rcx, 0x28(%rsp) + adcq $0x0, %r8 + movq %r8, 0x30(%rsp) + adcq %rdx, %r9 + movq %r9, 0x38(%rsp) + movq 0xa0(%rsp), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0xa8(%rsp), %r9, %r10 + mulxq 0xb8(%rsp), %r11, %r12 + movq 0xb0(%rsp), %rdx + mulxq 0xb8(%rsp), %r13, %r14 + xorl %ecx, %ecx + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0xb8(%rsp), %rdx + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + adoxq %rcx, %r14 + adcq %rcx, %r14 + xorl %ecx, %ecx + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0xa8(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0xb0(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0xb8(%rsp), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rcx, %r15 + adoxq %rcx, %r15 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + movl %ecx, %r9d + adoxq %rcx, %r9 + adcxq %rcx, %r9 + addq %r9, %r14 + adcq %rcx, %r15 + movl %ecx, %r8d + adcq %rcx, %r8 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rcx, %r15 + adoxq %rcx, %r8 + adcq %rcx, %r8 + movl $0x1, %r8d + leaq -0x1(%rdx), %rdx + leaq -0x1(%rcx), %rax + movl $0xfffffffe, %r11d + cmoveq %rcx, %r8 + cmoveq %rcx, %rdx + cmoveq %rcx, %rax + cmoveq %rcx, %r11 + addq %r8, %r12 + adcq %rdx, %r13 + adcq %rax, %r14 + adcq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + movq 0x20(%rsp), %rdx + mulxq %rdx, %r8, %r15 + mulxq 0x28(%rsp), %r9, %r10 + mulxq 0x38(%rsp), %r11, %r12 + movq 0x30(%rsp), %rdx + mulxq 0x38(%rsp), %r13, %r14 + xorl %ecx, %ecx + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x38(%rsp), %rdx + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + adoxq %rcx, %r14 + adcq %rcx, %r14 + xorl %ecx, %ecx + adcxq %r9, %r9 + adoxq %r15, %r9 + movq 0x28(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x30(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x38(%rsp), %rdx + mulxq %rdx, %rax, %r15 + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %rcx, %r15 + adoxq %rcx, %r15 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq $0xffffffff00000001, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %rcx, %r13 + movl %ecx, %r9d + adoxq %rcx, %r9 + adcxq %rcx, %r9 + addq %r9, %r14 + adcq %rcx, %r15 + movl %ecx, %r8d + adcq %rcx, %r8 + xorl %ecx, %ecx + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq $0xffffffff00000001, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rcx, %r15 + adoxq %rcx, %r8 + adcq %rcx, %r8 + movl $0x1, %ebx + addq %r12, %rbx + leaq -0x1(%rdx), %rdx + adcq %r13, %rdx + leaq -0x1(%rcx), %rcx + movq %rcx, %rax + adcq %r14, %rcx + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rbx, %r12 + cmovbq %rdx, %r13 + cmovbq %rcx, %r14 + cmovbq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + xorl %r13d, %r13d + movq (%rsi), %rdx + mulxq 0x60(%rsp), %r8, %r9 + mulxq 0x68(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x70(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x78(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x8(%rsi), %rdx + xorl %r14d, %r14d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x78(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x10(%rsi), %rdx + xorl %r8d, %r8d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x18(%rsi), %rdx + xorl %r9d, %r9d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + xorl %r13d, %r13d + movq 0x40(%rsp), %rdx + mulxq 0x60(%rsp), %r8, %r9 + mulxq 0x68(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x70(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x78(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x48(%rsp), %rdx + xorl %r14d, %r14d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x78(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x50(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x58(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x40(%rsp) + movq %r13, 0x48(%rsp) + movq %r14, 0x50(%rsp) + movq %r15, 0x58(%rsp) + movq (%rsp), %rax + subq 0x80(%rsp), %rax + movq 0x8(%rsp), %rcx + sbbq 0x88(%rsp), %rcx + movq 0x10(%rsp), %r8 + sbbq 0x90(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x98(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, (%rsp) + adcq %r10, %rcx + movq %rcx, 0x8(%rsp) + adcq $0x0, %r8 + movq %r8, 0x10(%rsp) + adcq %rdx, %r9 + movq %r9, 0x18(%rsp) + movq 0x40(%rsp), %rax + subq 0x80(%rsp), %rax + movq 0x48(%rsp), %rcx + sbbq 0x88(%rsp), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x90(%rsp), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x98(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x60(%rsp) + adcq %r10, %rcx + movq %rcx, 0x68(%rsp) + adcq $0x0, %r8 + movq %r8, 0x70(%rsp) + adcq %rdx, %r9 + movq %r9, 0x78(%rsp) + xorl %r13d, %r13d + movq 0x40(%rsi), %rdx + mulxq 0xa0(%rsp), %r8, %r9 + mulxq 0xa8(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0xb0(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0xb8(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x48(%rsi), %rdx + xorl %r14d, %r14d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xb8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x50(%rsi), %rdx + xorl %r8d, %r8d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x58(%rsi), %rdx + xorl %r9d, %r9d + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0xa0(%rsp) + movq %r13, 0xa8(%rsp) + movq %r14, 0xb0(%rsp) + movq %r15, 0xb8(%rsp) + movq (%rsp), %rax + subq 0x40(%rsp), %rax + movq 0x8(%rsp), %rcx + sbbq 0x48(%rsp), %rcx + movq 0x10(%rsp), %r8 + sbbq 0x50(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x58(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, (%rsp) + adcq %r10, %rcx + movq %rcx, 0x8(%rsp) + adcq $0x0, %r8 + movq %r8, 0x10(%rsp) + adcq %rdx, %r9 + movq %r9, 0x18(%rsp) + movq 0x80(%rsp), %rax + subq (%rsp), %rax + movq 0x88(%rsp), %rcx + sbbq 0x8(%rsp), %rcx + movq 0x90(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x98(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x80(%rsp) + adcq %r10, %rcx + movq %rcx, 0x88(%rsp) + adcq $0x0, %r8 + movq %r8, 0x90(%rsp) + adcq %rdx, %r9 + movq %r9, 0x98(%rsp) + xorl %r13d, %r13d + movq 0x20(%rsi), %rdx + mulxq 0x60(%rsp), %r8, %r9 + mulxq 0x68(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x70(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x78(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x28(%rsi), %rdx + xorl %r14d, %r14d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x78(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x30(%rsi), %rdx + xorl %r8d, %r8d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x38(%rsi), %rdx + xorl %r9d, %r9d + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x70(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x78(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + xorl %r13d, %r13d + movq 0x80(%rsp), %rdx + mulxq 0x20(%rsp), %r8, %r9 + mulxq 0x28(%rsp), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x30(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x38(%rsp), %rbx, %r12 + adcq %rbx, %r11 + adcq %r13, %r12 + movq 0x88(%rsp), %rdx + xorl %r14d, %r14d + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x38(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcq %r14, %r13 + xorl %r15d, %r15d + movq $0x100000000, %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq %r9, %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r8, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r9, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adcxq %r15, %r13 + adoxq %r15, %r14 + adcq %r15, %r14 + movq 0x90(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + adoxq %r8, %r14 + mulxq 0x38(%rsp), %rax, %rbx + adcq %rax, %r13 + adcq %rbx, %r14 + adcq %r8, %r15 + movq 0x98(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x28(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r9, %r15 + mulxq 0x38(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r9, %r8 + xorl %r9d, %r9d + movq $0x100000000, %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq %r11, %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + notq %rdx + leaq 0x2(%rdx), %rdx + mulxq %r10, %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq %r11, %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %r9, %r15 + adoxq %r9, %r8 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rdx + adcq %r13, %rdx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rdx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq 0x80(%rsp), %rax + subq 0x60(%rsp), %rax + movq 0x88(%rsp), %rcx + sbbq 0x68(%rsp), %rcx + movq 0x90(%rsp), %r8 + sbbq 0x70(%rsp), %r8 + movq 0x98(%rsp), %r9 + sbbq 0x78(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x80(%rsp) + adcq %r10, %rcx + movq %rcx, 0x88(%rsp) + adcq $0x0, %r8 + movq %r8, 0x90(%rsp) + adcq %rdx, %r9 + movq %r9, 0x98(%rsp) + movq 0x40(%rsi), %rax + movq 0x48(%rsi), %rdx + orq 0x50(%rsi), %rax + orq 0x58(%rsi), %rdx + orq %rdx, %rax + movq (%rsp), %r8 + movq 0x0(%rbp), %rax + cmoveq %rax, %r8 + movq 0x8(%rsp), %r9 + movq 0x8(%rbp), %rax + cmoveq %rax, %r9 + movq 0x10(%rsp), %r10 + movq 0x10(%rbp), %rax + cmoveq %rax, %r10 + movq 0x18(%rsp), %r11 + movq 0x18(%rbp), %rax + cmoveq %rax, %r11 + movq 0x80(%rsp), %r12 + movq 0x20(%rbp), %rax + cmoveq %rax, %r12 + movq 0x88(%rsp), %r13 + movq 0x28(%rbp), %rax + cmoveq %rax, %r13 + movq 0x90(%rsp), %r14 + movq 0x30(%rbp), %rax + cmoveq %rax, %r14 + movq 0x98(%rsp), %r15 + movq 0x38(%rbp), %rax + cmoveq %rax, %r15 + movq %r8, (%rdi) + movq %r9, 0x8(%rdi) + movq %r10, 0x10(%rdi) + movq %r11, 0x18(%rdi) + movq %r12, 0x20(%rdi) + movq %r13, 0x28(%rdi) + movq %r14, 0x30(%rdi) + movq %r15, 0x38(%rdi) + movq 0xa0(%rsp), %r8 + movq 0xa8(%rsp), %r9 + movq 0xb0(%rsp), %r10 + movq 0xb8(%rsp), %r11 + movl $0x1, %eax + cmoveq %rax, %r8 + movq $0xffffffff00000000, %rax + cmoveq %rax, %r9 + movq $0xffffffffffffffff, %rax + cmoveq %rax, %r10 + movl $0xfffffffe, %eax + cmoveq %rax, %r11 + movq %r8, 0x40(%rdi) + movq %r9, 0x48(%rdi) + movq %r10, 0x50(%rdi) + movq %r11, 0x58(%rdi) + CFI_INC_RSP(192) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_local_p256_montjmixadd) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/p256_scalarmulbase_alt.S b/cbits/s2n/x86_att/p256_scalarmulbase_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/p256_scalarmulbase_alt.S @@ -0,0 +1,4212 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Scalar multiplication for precomputed point on NIST curve P-256 +// Input scalar[4], blocksize, table[]; output res[8] +// +// extern void p256_scalarmulbase_alt +// (uint64_t res[static 8], +// const uint64_t scalar[static 4], +// uint64_t blocksize, +// const uint64_t *table); +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-256, the input argument "table" is expected to be a table of +// multiples of the point P in Montgomery-affine form, with each block +// corresponding to "blocksize" bits of the scalar as follows, where +// B = 2^{blocksize-1} (e.g. B = 8 for blocksize = 4): +// +// For each i,j with blocksize * i <= 256 and 1 <= j <= B +// the multiple 2^{blocksize * i} * j * P is stored at +// tab[8 * (B * i + (j - 1))], considered as uint64_t pointers +// or tab + 64 * (B * i + (j - 1)) as byte pointers. +// +// Standard x86-64 ABI: RDI = res, RSI = scalar, RDX = blocksize, RCX = table +// Microsoft x64 ABI: RCX = res, RDX = scalar, R8 = blocksize, R9 = table +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p256_scalarmulbase_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p256_scalarmulbase_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p256_scalarmulbase_alt) + + + .text + .balign 32 + +// Size of individual field elements + +#define NUMSIZE 32 + +// Intermediate variables on the stack. The last z2, z3 values can +// safely be overlaid on "nacc", which is no longer needed at the end. +// Uppercase syntactic variants make x86_att version simpler to generate + +#define RSCALAR (0*NUMSIZE) +#define ACC (1*NUMSIZE) +#define NACC (4*NUMSIZE) +#define TABENT (7*NUMSIZE) +#define Z2 (4*NUMSIZE) +#define Z3 (5*NUMSIZE) + +#define rscalar RSCALAR(%rsp) +#define acc ACC(%rsp) +#define nacc NACC(%rsp) +#define tabent TABENT(%rsp) + +#define z2 Z2(%rsp) +#define z3 Z3(%rsp) + +#define res (9*NUMSIZE)(%rsp) +#define blocksize (9*NUMSIZE+8)(%rsp) +#define table (9*NUMSIZE+16)(%rsp) +#define i (9*NUMSIZE+24)(%rsp) +#define bf (9*NUMSIZE+32)(%rsp) +#define cf (9*NUMSIZE+40)(%rsp) +#define j (9*NUMSIZE+48)(%rsp) + +#define NSPACE 11*NUMSIZE + +S2N_BN_SYMBOL(p256_scalarmulbase_alt): + CFI_START + _CET_ENDBR + +// The Windows version literally calls the standard ABI version. +// This simplifies the proofs since subroutine offsets are fixed. + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + movq %r9, %rcx + CFI_CALL(Lp256_scalarmulbase_alt_standard) + CFI_POP(%rsi) + CFI_POP(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p256_scalarmulbase_alt) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_alt_standard) + +Lp256_scalarmulbase_alt_standard: + CFI_START +#endif + +// Real start of the standard ABI code. + + CFI_PUSH(%r15) + CFI_PUSH(%r14) + CFI_PUSH(%r13) + CFI_PUSH(%r12) + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + + CFI_DEC_RSP(NSPACE) + +// Preserve the input arguments except the scalar, since that gets absorbed +// immediately. The "table" value subsequently gets shifted up each iteration +// of the loop, while "res" and "blocksize" are static throughout. + + movq %rdi, res + movq %rdx, blocksize + movq %rcx, table + +// Load the digits of group order n_256 = [%r15;%r14;%r13;%r12] + + movq $0xf3b9cac2fc632551, %r12 + movq $0xbce6faada7179e84, %r13 + movq $0xffffffffffffffff, %r14 + movq $0xffffffff00000000, %r15 + +// First, reduce the input scalar mod n_256, i.e. conditionally subtract n_256 +// Store it to "rscalar" (reduced scalar) + + movq (%rsi), %r8 + subq %r12, %r8 + movq 8(%rsi), %r9 + sbbq %r13, %r9 + movq 16(%rsi), %r10 + sbbq %r14, %r10 + movq 24(%rsi), %r11 + sbbq %r15, %r11 + + cmovcq (%rsi), %r8 + cmovcq 8(%rsi), %r9 + cmovcq 16(%rsi), %r10 + cmovcq 24(%rsi), %r11 + + movq %r8, RSCALAR(%rsp) + movq %r9, RSCALAR+8(%rsp) + movq %r10, RSCALAR+16(%rsp) + movq %r11, RSCALAR+24(%rsp) + +// Initialize the accumulator to all zeros and the "carry flag" cf to 0 + + xorl %eax, %eax + + movq %rax, ACC(%rsp) + movq %rax, ACC+8(%rsp) + movq %rax, ACC+16(%rsp) + movq %rax, ACC+24(%rsp) + movq %rax, ACC+32(%rsp) + movq %rax, ACC+40(%rsp) + movq %rax, ACC+48(%rsp) + movq %rax, ACC+56(%rsp) + movq %rax, ACC+64(%rsp) + movq %rax, ACC+72(%rsp) + movq %rax, ACC+80(%rsp) + movq %rax, ACC+88(%rsp) + + movq %rax, cf + +// Main loop over {i >= 0 | blocksize * i <= 256}. Note the non-strict +// inequality, to allow top carry for any choices of blocksize. + + movq %rax, i + +Lp256_scalarmulbase_alt_loop: + +// The next raw bitfield is bf = bitfield(blocksize * i,blocksize) + cf, +// adding in the deferred carry cf. We then shift the whole scalar right +// by blocksize so we can keep picking bitfield(0,blocksize). + + movq RSCALAR(%rsp), %r8 + movq RSCALAR+8(%rsp), %r9 + movq RSCALAR+16(%rsp), %r10 + movq RSCALAR+24(%rsp), %r11 + + movq blocksize, %rcx + movl $1, %eax + shlq %cl, %rax + decq %rax + andq %r8, %rax + + shrdq %cl, %r9, %r8 + shrdq %cl, %r10, %r9 + shrdq %cl, %r11, %r10 + shrq %cl, %r11 + + addq cf, %rax + movq %rax, bf + + movq %r8, RSCALAR(%rsp) + movq %r9, RSCALAR+8(%rsp) + movq %r10, RSCALAR+16(%rsp) + movq %r11, RSCALAR+24(%rsp) + +// Now if bf <= B we just select entry j, unnegated and set cf = 0. +// If bf > B we set j = 2 * B - bf and negate the j'th entry, setting cf = 1. +// In either case we ultimately add bf, in the latter case with deferred +// carry as 2 * B - (2 * B - bf) = bf. + + movl $1, %eax + movq blocksize, %rcx + shlq %cl, %rax + movq %rax, %rbx + shrq $1, %rax + + subq bf, %rbx + cmpq bf, %rax + + cmovncq bf, %rbx + sbbq %rax, %rax + movq %rbx, j + negq %rax + movq %rax, cf + +// Load table entry j - 1 for nonzero j in constant-time style. + + movq blocksize, %rcx + decq %rcx + movl $1, %esi + shlq %cl, %rsi + movq j, %r12 + movq table, %rbp + +Lp256_scalarmulbase_alt_tabloop: + subq $1, %r12 + cmovzq (%rbp), %rax + cmovzq 8(%rbp), %rbx + cmovzq 16(%rbp), %rcx + cmovzq 24(%rbp), %rdx + cmovzq 32(%rbp), %r8 + cmovzq 40(%rbp), %r9 + cmovzq 48(%rbp), %r10 + cmovzq 56(%rbp), %r11 + + addq $64, %rbp + decq %rsi + jnz Lp256_scalarmulbase_alt_tabloop + + movq %rbp, table + +// Before storing back, optionally negate the y coordinate of the table entry + + xorl %r14d, %r14d + leaq -1(%r14), %r12 + movq $0x00000000ffffffff, %r15 + movq %r15, %r13 + negq %r15 + + subq %r8, %r12 + sbbq %r9, %r13 + sbbq %r10, %r14 + sbbq %r11, %r15 + + movq %rax, TABENT(%rsp) + movq %rbx, TABENT+8(%rsp) + movq %rcx, TABENT+16(%rsp) + movq %rdx, TABENT+24(%rsp) + + movq cf, %rax + testq %rax, %rax + cmovnzq %r12, %r8 + cmovnzq %r13, %r9 + cmovnzq %r14, %r10 + cmovnzq %r15, %r11 + + movq %r8, TABENT+32(%rsp) + movq %r9, TABENT+40(%rsp) + movq %r10, TABENT+48(%rsp) + movq %r11, TABENT+56(%rsp) + +// Add the adjusted table point to the accumulator + + leaq NACC(%rsp), %rdi + leaq ACC(%rsp), %rsi + leaq TABENT(%rsp), %rdx + CFI_CALL(Lp256_scalarmulbase_alt_local_p256_montjmixadd) + +// However, only commit that update to the accumulator if j is nonzero, +// because the mixed addition function does not handle this case directly, +// and in any case we didn't choose the table entry appropriately. + + movq j, %rax + testq %rax, %rax + + movq ACC(%rsp), %rax + cmovnzq NACC(%rsp), %rax + movq %rax, ACC(%rsp) + + movq ACC+8(%rsp), %rax + cmovnzq NACC+8(%rsp), %rax + movq %rax, ACC+8(%rsp) + + movq ACC+16(%rsp), %rax + cmovnzq NACC+16(%rsp), %rax + movq %rax, ACC+16(%rsp) + + movq ACC+24(%rsp), %rax + cmovnzq NACC+24(%rsp), %rax + movq %rax, ACC+24(%rsp) + + movq ACC+32(%rsp), %rax + cmovnzq NACC+32(%rsp), %rax + movq %rax, ACC+32(%rsp) + + movq ACC+40(%rsp), %rax + cmovnzq NACC+40(%rsp), %rax + movq %rax, ACC+40(%rsp) + + movq ACC+48(%rsp), %rax + cmovnzq NACC+48(%rsp), %rax + movq %rax, ACC+48(%rsp) + + movq ACC+56(%rsp), %rax + cmovnzq NACC+56(%rsp), %rax + movq %rax, ACC+56(%rsp) + + movq ACC+64(%rsp), %rax + cmovnzq NACC+64(%rsp), %rax + movq %rax, ACC+64(%rsp) + + movq ACC+72(%rsp), %rax + cmovnzq NACC+72(%rsp), %rax + movq %rax, ACC+72(%rsp) + + movq ACC+80(%rsp), %rax + cmovnzq NACC+80(%rsp), %rax + movq %rax, ACC+80(%rsp) + + movq ACC+88(%rsp), %rax + cmovnzq NACC+88(%rsp), %rax + movq %rax, ACC+88(%rsp) + +// Loop while blocksize * i <= 256 + + movq i, %rax + incq %rax + movq %rax, i + + imulq blocksize, %rax + cmpq $257, %rax + jc Lp256_scalarmulbase_alt_loop + +// That's the end of the main loop, and we just need to translate +// back from the Jacobian representation to affine. First of all, +// let z2 = 1/z^2 and z3 = 1/z^3, both without Montgomery form + + leaq Z2(%rsp), %rdi + leaq ACC+64(%rsp), %rsi + CFI_CALL(Lp256_scalarmulbase_alt_local_montsqr_p256) + + leaq Z3(%rsp), %rdi + leaq ACC+64(%rsp), %rsi + leaq Z2(%rsp), %rdx + CFI_CALL(Lp256_scalarmulbase_alt_local_montmul_p256) + + leaq Z2(%rsp), %rdi + leaq Z3(%rsp), %rsi + CFI_CALL(Lp256_scalarmulbase_alt_local_demont_p256) + + leaq Z3(%rsp), %rdi + leaq Z2(%rsp), %rsi + CFI_CALL(Lp256_scalarmulbase_alt_local_inv_p256) + + leaq Z2(%rsp), %rdi + leaq ACC+64(%rsp), %rsi + leaq Z3(%rsp), %rdx + CFI_CALL(Lp256_scalarmulbase_alt_local_montmul_p256) + +// Convert back from Jacobian (X, Y, Z) |-> (X/Z^2, Y/Z^3) + + movq res, %rdi + leaq ACC(%rsp), %rsi + leaq Z2(%rsp), %rdx + movq %rdi, %rbx + CFI_CALL(Lp256_scalarmulbase_alt_local_montmul_p256) + + leaq 32(%rbx), %rdi + leaq ACC+32(%rsp), %rsi + leaq Z3(%rsp), %rdx + CFI_CALL(Lp256_scalarmulbase_alt_local_montmul_p256) + +// Restore stack and registers and return + + CFI_INC_RSP(NSPACE) + CFI_POP(%rbx) + CFI_POP(%rbp) + CFI_POP(%r12) + CFI_POP(%r13) + CFI_POP(%r14) + CFI_POP(%r15) + CFI_RET + +#if WINDOWS_ABI +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_alt_standard) +#else +S2N_BN_SIZE_DIRECTIVE(p256_scalarmulbase_alt) +#endif + +// Local copies of subroutines, complete clones at the moment + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_alt_local_demont_p256) + +Lp256_scalarmulbase_alt_local_demont_p256: + CFI_START + movq (%rsi), %r8 + movq 0x8(%rsi), %r9 + movq 0x10(%rsi), %r10 + movq 0x18(%rsi), %r11 + movabsq $0x100000000, %rcx + movq %r8, %rax + mulq %rcx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rsi, %rsi + movq %r9, %rax + mulq %rcx + subq %rsi, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rsi, %rsi + negq %rcx + negq %rsi + incq %rcx + movq %r8, %rax + mulq %rcx + addq %rax, %r11 + adcq %rdx, %rsi + sbbq %r8, %r8 + negq %r8 + movq %r9, %rax + mulq %rcx + addq %rax, %rsi + adcq %rdx, %r8 + negq %rcx + incq %rcx + movq %r10, %rax + mulq %rcx + addq %rax, %r11 + adcq %rdx, %rsi + sbbq %r9, %r9 + movq %r11, %rax + mulq %rcx + subq %r9, %rdx + addq %rax, %rsi + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %rcx + negq %r9 + incq %rcx + movq %r10, %rax + mulq %rcx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rax + mulq %rcx + addq %rax, %r9 + adcq %rdx, %r10 + movq %rsi, (%rdi) + movq %r8, 0x8(%rdi) + movq %r9, 0x10(%rdi) + movq %r10, 0x18(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_alt_local_demont_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_alt_local_inv_p256) + +Lp256_scalarmulbase_alt_local_inv_p256: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(240) + movq %rdi, 0xe0(%rsp) + xorl %ecx, %ecx + movl $0xffffffff, %edx + movq %rdx, %rbx + leaq -0x1(%rcx), %rax + negq %rdx + movq %rax, (%rsp) + movq %rbx, 0x8(%rsp) + movq %rcx, 0x10(%rsp) + movq %rdx, 0x18(%rsp) + movq %rcx, 0x20(%rsp) + movq (%rsi), %r8 + movq 0x8(%rsi), %r9 + movq 0x10(%rsi), %r10 + movq 0x18(%rsi), %r11 + leaq 0x1(%rcx), %rax + addq %r8, %rax + leaq -0x1(%rdx), %rbx + adcq %r9, %rbx + notq %rcx + adcq %r10, %rcx + notq %rdx + adcq %r11, %rdx + cmovaeq %r8, %rax + cmovaeq %r9, %rbx + cmovaeq %r10, %rcx + cmovaeq %r11, %rdx + movq %rax, 0x28(%rsp) + movq %rbx, 0x30(%rsp) + movq %rcx, 0x38(%rsp) + movq %rdx, 0x40(%rsp) + xorl %eax, %eax + movq %rax, 0x48(%rsp) + xorl %eax, %eax + movq %rax, 0x50(%rsp) + movq %rax, 0x58(%rsp) + movq %rax, 0x60(%rsp) + movq %rax, 0x68(%rsp) + movabsq $0x4000000000000, %rcx + movq %rcx, 0x78(%rsp) + movq %rax, 0x80(%rsp) + movq %rax, 0x88(%rsp) + movq %rax, 0x90(%rsp) + movq $0xa, 0xb0(%rsp) + movq $0x1, 0xb8(%rsp) + jmp Lp256_scalarmulbase_alt_inv_midloop +Lp256_scalarmulbase_alt_inv_loop: + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + movq %r8, %rax + andq %r9, %rax + movq %r10, %rdi + andq %r11, %rdi + addq %rax, %rdi + movq %rdi, 0xa0(%rsp) + movq %r12, %rax + andq %r13, %rax + movq %r14, %rsi + andq %r15, %rsi + addq %rax, %rsi + movq %rsi, 0xa8(%rsp) + xorl %ebx, %ebx + movq (%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x28(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rdi + adcq %rdx, %rbx + xorl %ebp, %ebp + movq (%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x28(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + xorl %ecx, %ecx + movq 0x8(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x30(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + shrdq $0x3b, %rbx, %rdi + movq %rdi, (%rsp) + xorl %edi, %edi + movq 0x8(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rbp + adcq %rdx, %rdi + movq 0x30(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rdi + shrdq $0x3b, %rbp, %rsi + movq %rsi, 0x28(%rsp) + xorl %esi, %esi + movq 0x10(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rsi + movq 0x38(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rsi + shrdq $0x3b, %rcx, %rbx + movq %rbx, 0x8(%rsp) + xorl %ebx, %ebx + movq 0x10(%rsp), %rax + xorq %r13, %rax + mulq %r12 + addq %rax, %rdi + adcq %rdx, %rbx + movq 0x38(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rdi + adcq %rdx, %rbx + shrdq $0x3b, %rdi, %rbp + movq %rbp, 0x30(%rsp) + movq 0x18(%rsp), %rax + xorq %r9, %rax + movq 0x20(%rsp), %rbp + xorq %r9, %rbp + andq %r8, %rbp + negq %rbp + mulq %r8 + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x40(%rsp), %rax + xorq %r11, %rax + movq 0x48(%rsp), %rdx + xorq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbp + mulq %r10 + addq %rax, %rsi + adcq %rdx, %rbp + shrdq $0x3b, %rsi, %rcx + movq %rcx, 0x10(%rsp) + shrdq $0x3b, %rbp, %rsi + sarq $0x3b, %rbp + movq 0x18(%rsp), %rax + movq %rsi, 0x18(%rsp) + movq 0x20(%rsp), %rsi + movq %rbp, 0x20(%rsp) + xorq %r13, %rax + xorq %r13, %rsi + andq %r12, %rsi + negq %rsi + mulq %r12 + addq %rax, %rbx + adcq %rdx, %rsi + movq 0x40(%rsp), %rax + xorq %r15, %rax + movq 0x48(%rsp), %rdx + xorq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rsi + mulq %r14 + addq %rax, %rbx + adcq %rdx, %rsi + shrdq $0x3b, %rbx, %rdi + movq %rdi, 0x38(%rsp) + shrdq $0x3b, %rsi, %rbx + movq %rbx, 0x40(%rsp) + sarq $0x3b, %rsi + movq %rsi, 0x48(%rsp) + movq 0xa0(%rsp), %rbx + movq 0xa8(%rsp), %rbp + xorl %ecx, %ecx + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x78(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x50(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x50(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x78(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x78(%rsp) + xorl %ebx, %ebx + movq 0x58(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x80(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rcx + adcq %rdx, %rbx + xorl %ebp, %ebp + movq 0x58(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rcx, 0x58(%rsp) + addq %rax, %rsi + adcq %rdx, %rbp + movq 0x80(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rsi + adcq %rdx, %rbp + movq %rsi, 0x80(%rsp) + xorl %ecx, %ecx + movq 0x60(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %rbx + adcq %rdx, %rcx + movq 0x88(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %rbx + adcq %rdx, %rcx + xorl %esi, %esi + movq 0x60(%rsp), %rax + xorq %r13, %rax + mulq %r12 + movq %rbx, 0x60(%rsp) + addq %rax, %rbp + adcq %rdx, %rsi + movq 0x88(%rsp), %rax + xorq %r15, %rax + mulq %r14 + addq %rax, %rbp + adcq %rdx, %rsi + movq %rbp, 0x88(%rsp) + movq 0x68(%rsp), %rax + xorq %r9, %rax + movq %r9, %rbx + andq %r8, %rbx + negq %rbx + mulq %r8 + addq %rax, %rcx + adcq %rdx, %rbx + movq 0x90(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %rbx + mulq %r10 + addq %rax, %rcx + adcq %rbx, %rdx + movq 0x68(%rsp), %rax + movq %rcx, 0x68(%rsp) + movq %rdx, 0x70(%rsp) + xorq %r13, %rax + movq %r13, %rcx + andq %r12, %rcx + negq %rcx + mulq %r12 + addq %rax, %rsi + adcq %rdx, %rcx + movq 0x90(%rsp), %rax + xorq %r15, %rax + movq %r15, %rdx + andq %r14, %rdx + subq %rdx, %rcx + mulq %r14 + addq %rax, %rsi + adcq %rcx, %rdx + movq %rsi, 0x90(%rsp) + movq %rdx, 0x98(%rsp) + movabsq $0xe000000000000000, %r8 + addq 0x50(%rsp), %r8 + movq $0xffffffffffffffff, %r9 + adcq 0x58(%rsp), %r9 + movq $0x1fffffff, %r10 + adcq 0x60(%rsp), %r10 + movabsq $0x2000000000000000, %r11 + adcq 0x68(%rsp), %r11 + movabsq $0x1fffffffe0000000, %r12 + adcq 0x70(%rsp), %r12 + movq %r8, %rbx + shlq $0x20, %rbx + movabsq $0xffffffff00000001, %rax + mulq %r8 + shrq $0x20, %r8 + addq %rbx, %r9 + adcq %r8, %r10 + adcq %rax, %r11 + adcq %rdx, %r12 + sbbq %rax, %rax + movl $0xffffffff, %ebx + andq %rax, %rbx + movabsq $0xffffffff00000001, %rdx + andq %rax, %rdx + subq %rax, %r9 + movq %r9, 0x50(%rsp) + sbbq %rbx, %r10 + movq %r10, 0x58(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x60(%rsp) + sbbq %rdx, %r12 + movq %r12, 0x68(%rsp) + movabsq $0xe000000000000000, %r8 + addq 0x78(%rsp), %r8 + movq $0xffffffffffffffff, %r9 + adcq 0x80(%rsp), %r9 + movq $0x1fffffff, %r10 + adcq 0x88(%rsp), %r10 + movabsq $0x2000000000000000, %r11 + adcq 0x90(%rsp), %r11 + movabsq $0x1fffffffe0000000, %r12 + adcq 0x98(%rsp), %r12 + movq %r8, %rbx + shlq $0x20, %rbx + movabsq $0xffffffff00000001, %rax + mulq %r8 + shrq $0x20, %r8 + addq %rbx, %r9 + adcq %r8, %r10 + adcq %rax, %r11 + adcq %rdx, %r12 + sbbq %rax, %rax + movl $0xffffffff, %ebx + andq %rax, %rbx + movabsq $0xffffffff00000001, %rdx + andq %rax, %rdx + subq %rax, %r9 + movq %r9, 0x78(%rsp) + sbbq %rbx, %r10 + movq %r10, 0x80(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x88(%rsp) + sbbq %rdx, %r12 + movq %r12, 0x90(%rsp) +Lp256_scalarmulbase_alt_inv_midloop: + movq 0xb8(%rsp), %rsi + movq (%rsp), %rdx + movq 0x28(%rsp), %rcx + movq %rdx, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + xorl %ebp, %ebp + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %rdx + leaq (%rcx,%rax), %rdi + shlq $0x16, %rdx + shlq $0x16, %rdi + sarq $0x2b, %rdx + sarq $0x2b, %rdi + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %rbx + leaq (%rcx,%rax), %rcx + sarq $0x2a, %rbx + sarq $0x2a, %rcx + movq %rdx, 0xc0(%rsp) + movq %rbx, 0xc8(%rsp) + movq %rdi, 0xd0(%rsp) + movq %rcx, 0xd8(%rsp) + movq (%rsp), %r12 + imulq %r12, %rdi + imulq %rdx, %r12 + movq 0x28(%rsp), %r13 + imulq %r13, %rbx + imulq %rcx, %r13 + addq %rbx, %r12 + addq %rdi, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r10 + shlq $0x16, %r8 + shlq $0x16, %r10 + sarq $0x2b, %r8 + sarq $0x2b, %r10 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r15 + leaq (%rcx,%rax), %r11 + sarq $0x2a, %r15 + sarq $0x2a, %r11 + movq %r13, %rbx + movq %r12, %rcx + imulq %r8, %r12 + imulq %r15, %rbx + addq %rbx, %r12 + imulq %r11, %r13 + imulq %r10, %rcx + addq %rcx, %r13 + sarq $0x14, %r12 + sarq $0x14, %r13 + movq %r12, %rbx + andq $0xfffff, %rbx + movabsq $0xfffffe0000000000, %rax + orq %rax, %rbx + movq %r13, %rcx + andq $0xfffff, %rcx + movabsq $0xc000000000000000, %rax + orq %rax, %rcx + movq 0xc0(%rsp), %rax + imulq %r8, %rax + movq 0xd0(%rsp), %rdx + imulq %r15, %rdx + imulq 0xc8(%rsp), %r8 + imulq 0xd8(%rsp), %r15 + addq %r8, %r15 + leaq (%rax,%rdx), %r9 + movq 0xc0(%rsp), %rax + imulq %r10, %rax + movq 0xd0(%rsp), %rdx + imulq %r11, %rdx + imulq 0xc8(%rsp), %r10 + imulq 0xd8(%rsp), %r11 + addq %r10, %r11 + leaq (%rax,%rdx), %r13 + movq $0xfffffffffffffffe, %rax + movl $0x2, %edx + movq %rbx, %rdi + movq %rax, %r8 + testq %rsi, %rsi + cmovs %rbp, %r8 + testq $0x1, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + cmovs %rbp, %r8 + movq %rbx, %rdi + testq %rdx, %rcx + cmoveq %rbp, %r8 + cmoveq %rbp, %rdi + sarq $1, %rcx + xorq %r8, %rdi + xorq %r8, %rsi + btq $0x3f, %r8 + cmovbq %rcx, %rbx + movq %rax, %r8 + subq %rax, %rsi + leaq (%rcx,%rdi), %rcx + sarq $1, %rcx + movl $0x100000, %eax + leaq (%rbx,%rax), %r8 + leaq (%rcx,%rax), %r12 + shlq $0x15, %r8 + shlq $0x15, %r12 + sarq $0x2b, %r8 + sarq $0x2b, %r12 + movabsq $0x20000100000, %rax + leaq (%rbx,%rax), %r10 + leaq (%rcx,%rax), %r14 + sarq $0x2b, %r10 + sarq $0x2b, %r14 + movq %r9, %rax + imulq %r8, %rax + movq %r13, %rdx + imulq %r10, %rdx + imulq %r15, %r8 + imulq %r11, %r10 + addq %r8, %r10 + leaq (%rax,%rdx), %r8 + movq %r9, %rax + imulq %r12, %rax + movq %r13, %rdx + imulq %r14, %rdx + imulq %r15, %r12 + imulq %r11, %r14 + addq %r12, %r14 + leaq (%rax,%rdx), %r12 + movq %rsi, 0xb8(%rsp) + decq 0xb0(%rsp) + jne Lp256_scalarmulbase_alt_inv_loop + movq (%rsp), %rax + movq 0x28(%rsp), %rcx + imulq %r8, %rax + imulq %r10, %rcx + addq %rcx, %rax + sarq $0x3f, %rax + movq %r8, %r9 + sarq $0x3f, %r9 + xorq %r9, %r8 + subq %r9, %r8 + xorq %rax, %r9 + movq %r10, %r11 + sarq $0x3f, %r11 + xorq %r11, %r10 + subq %r11, %r10 + xorq %rax, %r11 + movq %r12, %r13 + sarq $0x3f, %r13 + xorq %r13, %r12 + subq %r13, %r12 + xorq %rax, %r13 + movq %r14, %r15 + sarq $0x3f, %r15 + xorq %r15, %r14 + subq %r15, %r14 + xorq %rax, %r15 + movq %r8, %rax + andq %r9, %rax + movq %r10, %r12 + andq %r11, %r12 + addq %rax, %r12 + xorl %r13d, %r13d + movq 0x50(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x78(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movq 0x58(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r13 + adcq %rdx, %r14 + movq 0x80(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq 0x60(%rsp), %rax + xorq %r9, %rax + mulq %r8 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x88(%rsp), %rax + xorq %r11, %rax + mulq %r10 + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x68(%rsp), %rax + xorq %r9, %rax + andq %r8, %r9 + negq %r9 + mulq %r8 + addq %rax, %r15 + adcq %rdx, %r9 + movq 0x90(%rsp), %rax + xorq %r11, %rax + movq %r11, %rdx + andq %r10, %rdx + subq %rdx, %r9 + mulq %r10 + addq %rax, %r15 + adcq %rdx, %r9 + movq %r12, 0x50(%rsp) + movq %r13, 0x58(%rsp) + movq %r14, 0x60(%rsp) + movq %r15, 0x68(%rsp) + movq %r9, 0x70(%rsp) + movabsq $0xe000000000000000, %r8 + addq 0x50(%rsp), %r8 + movq $0xffffffffffffffff, %r9 + adcq 0x58(%rsp), %r9 + movq $0x1fffffff, %r10 + adcq 0x60(%rsp), %r10 + movabsq $0x2000000000000000, %r11 + adcq 0x68(%rsp), %r11 + movabsq $0x1fffffffe0000000, %r12 + adcq 0x70(%rsp), %r12 + movq %r8, %rbx + shlq $0x20, %rbx + movabsq $0xffffffff00000001, %rax + mulq %r8 + shrq $0x20, %r8 + addq %rbx, %r9 + adcq %r8, %r10 + adcq %rax, %r11 + adcq %rdx, %r12 + sbbq %rax, %rax + movl $0xffffffff, %ebx + andq %rax, %rbx + movabsq $0xffffffff00000001, %rdx + andq %rax, %rdx + subq %rax, %r9 + movq %r9, 0x50(%rsp) + sbbq %rbx, %r10 + movq %r10, 0x58(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x60(%rsp) + sbbq %rdx, %r12 + movq %r12, 0x68(%rsp) + movq 0x50(%rsp), %r8 + movq 0x58(%rsp), %r9 + movq 0x60(%rsp), %r10 + movq 0x68(%rsp), %r11 + movl $0x1, %eax + movl $0xffffffff, %ebx + leaq -0x2(%rax), %rcx + leaq -0x1(%rbx), %rdx + notq %rbx + addq %r8, %rax + adcq %r9, %rbx + adcq %r10, %rcx + adcq %r11, %rdx + cmovaeq %r8, %rax + cmovaeq %r9, %rbx + cmovaeq %r10, %rcx + cmovaeq %r11, %rdx + movq 0xe0(%rsp), %rdi + movq %rax, (%rdi) + movq %rbx, 0x8(%rdi) + movq %rcx, 0x10(%rdi) + movq %rdx, 0x18(%rdi) + CFI_INC_RSP(240) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_alt_local_inv_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_alt_local_montmul_p256) + +Lp256_scalarmulbase_alt_local_montmul_p256: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + movq %rdx, %rcx + movq (%rcx), %rbx + movq (%rsi), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x8(%rsi), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x10(%rsi), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x18(%rsi), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x8(%rcx), %rbx + xorl %r13d, %r13d + movq (%rsi), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x8(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x10(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x18(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x10(%rcx), %rbx + xorl %r15d, %r15d + movq (%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x8(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x10(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x18(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x18(%rcx), %rbx + xorl %r8d, %r8d + movq (%rsi), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x8(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x10(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x18(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_alt_local_montmul_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_alt_local_montsqr_p256) + +Lp256_scalarmulbase_alt_local_montsqr_p256: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + movq (%rsi), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0x8(%rsi), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x18(%rsi), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x10(%rsi), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq (%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0x8(%rsi), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0x18(%rsi), %rbx + movq 0x8(%rsi), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0x8(%rsi), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0x10(%rsi), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0x18(%rsi), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, (%rdi) + movq %r13, 0x8(%rdi) + movq %r14, 0x10(%rdi) + movq %r15, 0x18(%rdi) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_alt_local_montsqr_p256) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp256_scalarmulbase_alt_local_p256_montjmixadd) + +Lp256_scalarmulbase_alt_local_p256_montjmixadd: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(192) + movq %rdx, %rbp + movq 0x40(%rsi), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0x48(%rsi), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x58(%rsi), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x50(%rsi), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq 0x40(%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0x48(%rsi), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0x58(%rsi), %rbx + movq 0x48(%rsi), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0x48(%rsi), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0x50(%rsi), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0x58(%rsi), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + movq 0x20(%rbp), %rbx + movq 0x40(%rsi), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x48(%rsi), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x50(%rsi), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x58(%rsi), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x28(%rbp), %rbx + xorl %r13d, %r13d + movq 0x40(%rsi), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x48(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x50(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x58(%rsi), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x30(%rbp), %rbx + xorl %r15d, %r15d + movq 0x40(%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x48(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x50(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x58(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x38(%rbp), %rbx + xorl %r8d, %r8d + movq 0x40(%rsi), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x48(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x50(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x58(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x20(%rsp) + movq %r13, 0x28(%rsp) + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq 0x0(%rbp), %rbx + movq (%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x10(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x18(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x8(%rbp), %rbx + xorl %r13d, %r13d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x10(%rbp), %rbx + xorl %r15d, %r15d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x18(%rbp), %rbx + xorl %r8d, %r8d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x40(%rsp) + movq %r13, 0x48(%rsp) + movq %r14, 0x50(%rsp) + movq %r15, 0x58(%rsp) + movq 0x20(%rsp), %rbx + movq (%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x10(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x18(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x28(%rsp), %rbx + xorl %r13d, %r13d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x30(%rsp), %rbx + xorl %r15d, %r15d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x38(%rsp), %rbx + xorl %r8d, %r8d + movq (%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x20(%rsp) + movq %r13, 0x28(%rsp) + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq 0x40(%rsp), %rax + subq (%rsi), %rax + movq 0x48(%rsp), %rcx + sbbq 0x8(%rsi), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x10(%rsi), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x18(%rsi), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0xa0(%rsp) + adcq %r10, %rcx + movq %rcx, 0xa8(%rsp) + adcq $0x0, %r8 + movq %r8, 0xb0(%rsp) + adcq %rdx, %r9 + movq %r9, 0xb8(%rsp) + movq 0x20(%rsp), %rax + subq 0x20(%rsi), %rax + movq 0x28(%rsp), %rcx + sbbq 0x28(%rsi), %rcx + movq 0x30(%rsp), %r8 + sbbq 0x30(%rsi), %r8 + movq 0x38(%rsp), %r9 + sbbq 0x38(%rsi), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x20(%rsp) + adcq %r10, %rcx + movq %rcx, 0x28(%rsp) + adcq $0x0, %r8 + movq %r8, 0x30(%rsp) + adcq %rdx, %r9 + movq %r9, 0x38(%rsp) + movq 0xa0(%rsp), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0xa8(%rsp), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0xb8(%rsp), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0xb0(%rsp), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0xa8(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0xb8(%rsp), %rbx + movq 0xa8(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0xa8(%rsp), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0xb0(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0xb8(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + movq 0x20(%rsp), %rax + movq %rax, %rbx + mulq %rax + movq %rax, %r8 + movq %rdx, %r15 + movq 0x28(%rsp), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x38(%rsp), %rax + movq %rax, %r13 + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x30(%rsp), %rax + movq %rax, %rbx + mulq %r13 + movq %rax, %r13 + movq %rdx, %r14 + movq 0x20(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + movq 0x28(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq 0x38(%rsp), %rbx + movq 0x28(%rsp), %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorl %ecx, %ecx + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %rcx, %rcx + movq 0x28(%rsp), %rax + mulq %rax + addq %r15, %r9 + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + movq 0x30(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %r15, %r15 + movq 0x38(%rsp), %rax + mulq %rax + negq %r15 + adcq %rax, %r14 + adcq %rcx, %rdx + movq %rdx, %r15 + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rcx, %rcx + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + xorl %r8d, %r8d + movq %r9, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r8, %r14 + adcq %r8, %r15 + adcq %r8, %r8 + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + xorl %r9d, %r9d + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + leaq -0x1(%rbx), %rbx + adcq %r13, %rbx + leaq -0x1(%r9), %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, (%rsp) + movq %r13, 0x8(%rsp) + movq %r14, 0x10(%rsp) + movq %r15, 0x18(%rsp) + movq (%rsi), %rbx + movq 0x60(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x70(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x78(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x8(%rsi), %rbx + xorl %r13d, %r13d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x10(%rsi), %rbx + xorl %r15d, %r15d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x18(%rsi), %rbx + xorl %r8d, %r8d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq 0x40(%rsp), %rbx + movq 0x60(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x70(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x78(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x48(%rsp), %rbx + xorl %r13d, %r13d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x50(%rsp), %rbx + xorl %r15d, %r15d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x58(%rsp), %rbx + xorl %r8d, %r8d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x40(%rsp) + movq %r13, 0x48(%rsp) + movq %r14, 0x50(%rsp) + movq %r15, 0x58(%rsp) + movq (%rsp), %rax + subq 0x80(%rsp), %rax + movq 0x8(%rsp), %rcx + sbbq 0x88(%rsp), %rcx + movq 0x10(%rsp), %r8 + sbbq 0x90(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x98(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, (%rsp) + adcq %r10, %rcx + movq %rcx, 0x8(%rsp) + adcq $0x0, %r8 + movq %r8, 0x10(%rsp) + adcq %rdx, %r9 + movq %r9, 0x18(%rsp) + movq 0x40(%rsp), %rax + subq 0x80(%rsp), %rax + movq 0x48(%rsp), %rcx + sbbq 0x88(%rsp), %rcx + movq 0x50(%rsp), %r8 + sbbq 0x90(%rsp), %r8 + movq 0x58(%rsp), %r9 + sbbq 0x98(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x60(%rsp) + adcq %r10, %rcx + movq %rcx, 0x68(%rsp) + adcq $0x0, %r8 + movq %r8, 0x70(%rsp) + adcq %rdx, %r9 + movq %r9, 0x78(%rsp) + movq 0x40(%rsi), %rbx + movq 0xa0(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0xb0(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0xb8(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x48(%rsi), %rbx + xorl %r13d, %r13d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x50(%rsi), %rbx + xorl %r15d, %r15d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x58(%rsi), %rbx + xorl %r8d, %r8d + movq 0xa0(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0xa0(%rsp) + movq %r13, 0xa8(%rsp) + movq %r14, 0xb0(%rsp) + movq %r15, 0xb8(%rsp) + movq (%rsp), %rax + subq 0x40(%rsp), %rax + movq 0x8(%rsp), %rcx + sbbq 0x48(%rsp), %rcx + movq 0x10(%rsp), %r8 + sbbq 0x50(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x58(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, (%rsp) + adcq %r10, %rcx + movq %rcx, 0x8(%rsp) + adcq $0x0, %r8 + movq %r8, 0x10(%rsp) + adcq %rdx, %r9 + movq %r9, 0x18(%rsp) + movq 0x80(%rsp), %rax + subq (%rsp), %rax + movq 0x88(%rsp), %rcx + sbbq 0x8(%rsp), %rcx + movq 0x90(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x98(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x80(%rsp) + adcq %r10, %rcx + movq %rcx, 0x88(%rsp) + adcq $0x0, %r8 + movq %r8, 0x90(%rsp) + adcq %rdx, %r9 + movq %r9, 0x98(%rsp) + movq 0x20(%rsi), %rbx + movq 0x60(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x70(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x78(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x28(%rsi), %rbx + xorl %r13d, %r13d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x30(%rsi), %rbx + xorl %r15d, %r15d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x38(%rsi), %rbx + xorl %r8d, %r8d + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x68(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x70(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x78(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x60(%rsp) + movq %r13, 0x68(%rsp) + movq %r14, 0x70(%rsp) + movq %r15, 0x78(%rsp) + movq 0x80(%rsp), %rbx + movq 0x20(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x28(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x30(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x38(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x88(%rsp), %rbx + xorl %r13d, %r13d + movq 0x20(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r14, %r14 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r14, %r14 + movq 0x30(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r14, %r14 + movq 0x38(%rsp), %rax + mulq %rbx + subq %r14, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + xorl %r14d, %r14d + movabsq $0x100000000, %rbx + movq %r8, %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r15, %r15 + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r8, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r15, %r15 + movq %r9, %rax + mulq %rbx + subq %r15, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x90(%rsp), %rbx + xorl %r15d, %r15d + movq 0x20(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x30(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x38(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x98(%rsp), %rbx + xorl %r8d, %r8d + movq 0x20(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x30(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x38(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + xorl %r9d, %r9d + movabsq $0x100000000, %rbx + movq %r10, %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rcx, %rcx + notq %rbx + leaq 0x2(%rbx), %rbx + movq %r10, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rcx, %rcx + movq %r11, %rax + mulq %rbx + subq %rcx, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r9, %r8 + movl $0x1, %ecx + addq %r12, %rcx + decq %rbx + adcq %r13, %rbx + decq %r9 + movq %r9, %rax + adcq %r14, %r9 + movl $0xfffffffe, %r11d + adcq %r15, %r11 + adcq %r8, %rax + cmovbq %rcx, %r12 + cmovbq %rbx, %r13 + cmovbq %r9, %r14 + cmovbq %r11, %r15 + movq %r12, 0x80(%rsp) + movq %r13, 0x88(%rsp) + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq 0x80(%rsp), %rax + subq 0x60(%rsp), %rax + movq 0x88(%rsp), %rcx + sbbq 0x68(%rsp), %rcx + movq 0x90(%rsp), %r8 + sbbq 0x70(%rsp), %r8 + movq 0x98(%rsp), %r9 + sbbq 0x78(%rsp), %r9 + movl $0xffffffff, %r10d + sbbq %r11, %r11 + xorq %rdx, %rdx + andq %r11, %r10 + subq %r10, %rdx + addq %r11, %rax + movq %rax, 0x80(%rsp) + adcq %r10, %rcx + movq %rcx, 0x88(%rsp) + adcq $0x0, %r8 + movq %r8, 0x90(%rsp) + adcq %rdx, %r9 + movq %r9, 0x98(%rsp) + movq 0x40(%rsi), %rax + movq 0x48(%rsi), %rdx + orq 0x50(%rsi), %rax + orq 0x58(%rsi), %rdx + orq %rdx, %rax + movq (%rsp), %r8 + movq 0x0(%rbp), %rax + cmoveq %rax, %r8 + movq 0x8(%rsp), %r9 + movq 0x8(%rbp), %rax + cmoveq %rax, %r9 + movq 0x10(%rsp), %r10 + movq 0x10(%rbp), %rax + cmoveq %rax, %r10 + movq 0x18(%rsp), %r11 + movq 0x18(%rbp), %rax + cmoveq %rax, %r11 + movq 0x80(%rsp), %r12 + movq 0x20(%rbp), %rax + cmoveq %rax, %r12 + movq 0x88(%rsp), %r13 + movq 0x28(%rbp), %rax + cmoveq %rax, %r13 + movq 0x90(%rsp), %r14 + movq 0x30(%rbp), %rax + cmoveq %rax, %r14 + movq 0x98(%rsp), %r15 + movq 0x38(%rbp), %rax + cmoveq %rax, %r15 + movq %r8, (%rdi) + movq %r9, 0x8(%rdi) + movq %r10, 0x10(%rdi) + movq %r11, 0x18(%rdi) + movq %r12, 0x20(%rdi) + movq %r13, 0x28(%rdi) + movq %r14, 0x30(%rdi) + movq %r15, 0x38(%rdi) + movq 0xa0(%rsp), %r8 + movq 0xa8(%rsp), %r9 + movq 0xb0(%rsp), %r10 + movq 0xb8(%rsp), %r11 + movl $0x1, %eax + cmoveq %rax, %r8 + movabsq $0xffffffff00000000, %rax + cmoveq %rax, %r9 + movq $0xffffffffffffffff, %rax + cmoveq %rax, %r10 + movl $0xfffffffe, %eax + cmoveq %rax, %r11 + movq %r8, 0x40(%rdi) + movq %r9, 0x48(%rdi) + movq %r10, 0x50(%rdi) + movq %r11, 0x58(%rdi) + CFI_INC_RSP(192) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp256_scalarmulbase_alt_local_p256_montjmixadd) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/p384_montjscalarmul.S b/cbits/s2n/x86_att/p384_montjscalarmul.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/p384_montjscalarmul.S @@ -0,0 +1,7418 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Montgomery-Jacobian form scalar multiplication for P-384 +// Input scalar[6], point[18]; output res[18] +// +// extern void p384_montjscalarmul +// (uint64_t res[static 18], +// const uint64_t scalar[static 6], +// const uint64_t point[static 18]); +// +// This function is a variant of its affine point version p384_scalarmul. +// Here, input and output points are assumed to be in Jacobian form with +// their coordinates in the Montgomery domain. Thus, if priming indicates +// Montgomery form, x' = (2^384 * x) mod p_384 etc., each point argument +// is a triple (x',y',z') representing the affine point (x/z^2,y/z^3) when +// z' is nonzero or the point at infinity (group identity) if z' = 0. +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-384, returns a representation of n * P. If the result is the +// point at infinity (either because the input point was or because the +// scalar was a multiple of p_384) then the output is guaranteed to +// represent the point at infinity, i.e. to have its z coordinate zero. +// +// Standard x86-64 ABI: RDI = res, RSI = scalar, RDX = point +// Microsoft x64 ABI: RCX = res, RDX = scalar, R8 = point +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p384_montjscalarmul) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p384_montjscalarmul) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p384_montjscalarmul) + + + .text + .balign 32 + +// Size of individual field elements + +#define NUMSIZE 48 +#define JACSIZE (3*NUMSIZE) + +// Intermediate variables on the stack. +// The table is 16 entries, each of size JACSIZE = 3 * NUMSIZE +// Uppercase syntactic variants make x86_att version simpler to generate. + +#define SCALARB (0*NUMSIZE) +#define scalarb (0*NUMSIZE)(%rsp) +#define ACC (1*NUMSIZE) +#define acc (1*NUMSIZE)(%rsp) +#define TABENT (4*NUMSIZE) +#define tabent (4*NUMSIZE)(%rsp) + +#define TAB (7*NUMSIZE) +#define tab (7*NUMSIZE)(%rsp) + +#define res (55*NUMSIZE)(%rsp) + +#define NSPACE 56*NUMSIZE + +// Avoid using .rep for the sake of the BoringSSL/AWS-LC delocator, +// which doesn't accept repetitions, assembler macros etc. + +#define selectblock_xz(I) \ + cmpq $I, %rdi ; \ + cmovzq TAB+JACSIZE*(I-1)(%rsp), %rax ; \ + cmovzq TAB+JACSIZE*(I-1)+8(%rsp), %rbx ; \ + cmovzq TAB+JACSIZE*(I-1)+16(%rsp), %rcx ; \ + cmovzq TAB+JACSIZE*(I-1)+24(%rsp), %rdx ; \ + cmovzq TAB+JACSIZE*(I-1)+32(%rsp), %r8 ; \ + cmovzq TAB+JACSIZE*(I-1)+40(%rsp), %r9 ; \ + cmovzq TAB+JACSIZE*(I-1)+96(%rsp), %r10 ; \ + cmovzq TAB+JACSIZE*(I-1)+104(%rsp), %r11 ; \ + cmovzq TAB+JACSIZE*(I-1)+112(%rsp), %r12 ; \ + cmovzq TAB+JACSIZE*(I-1)+120(%rsp), %r13 ; \ + cmovzq TAB+JACSIZE*(I-1)+128(%rsp), %r14 ; \ + cmovzq TAB+JACSIZE*(I-1)+136(%rsp), %r15 + +#define selectblock_y(I) \ + cmpq $I, %rdi ; \ + cmovzq TAB+JACSIZE*(I-1)+48(%rsp), %rax ; \ + cmovzq TAB+JACSIZE*(I-1)+56(%rsp), %rbx ; \ + cmovzq TAB+JACSIZE*(I-1)+64(%rsp), %rcx ; \ + cmovzq TAB+JACSIZE*(I-1)+72(%rsp), %rdx ; \ + cmovzq TAB+JACSIZE*(I-1)+80(%rsp), %r8 ; \ + cmovzq TAB+JACSIZE*(I-1)+88(%rsp), %r9 + +S2N_BN_SYMBOL(p384_montjscalarmul): + CFI_START + _CET_ENDBR + +// The Windows version literally calls the standard ABI version. +// This simplifies the proofs since subroutine offsets are fixed. + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + CFI_CALL(Lp384_montjscalarmul_standard) + CFI_POP(%rsi) + CFI_POP(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p384_montjscalarmul) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp384_montjscalarmul_standard) + +Lp384_montjscalarmul_standard: + CFI_START +#endif + +// Real start of the standard ABI code. + + CFI_PUSH(%r15) + CFI_PUSH(%r14) + CFI_PUSH(%r13) + CFI_PUSH(%r12) + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + + CFI_DEC_RSP(NSPACE) + +// Preserve the "res" input argument; others get processed early. + + movq %rdi, res + +// Reduce the input scalar mod n_384, i.e. conditionally subtract n_384. +// Store it to "scalarb". + + movq (%rsi), %r8 + movq $0xecec196accc52973, %rax + subq %rax, %r8 + movq 8(%rsi), %r9 + movq $0x581a0db248b0a77a, %rax + sbbq %rax, %r9 + movq 16(%rsi), %r10 + movq $0xc7634d81f4372ddf, %rax + sbbq %rax, %r10 + movq 24(%rsi), %r11 + movq $0xffffffffffffffff, %rax + sbbq %rax, %r11 + movq 32(%rsi), %r12 + sbbq %rax, %r12 + movq 40(%rsi), %r13 + sbbq %rax, %r13 + + cmovcq (%rsi), %r8 + cmovcq 8(%rsi), %r9 + cmovcq 16(%rsi), %r10 + cmovcq 24(%rsi), %r11 + cmovcq 32(%rsi), %r12 + cmovcq 40(%rsi), %r13 + + movq %r8, SCALARB(%rsp) + movq %r9, SCALARB+8(%rsp) + movq %r10, SCALARB+16(%rsp) + movq %r11, SCALARB+24(%rsp) + movq %r12, SCALARB+32(%rsp) + movq %r13, SCALARB+40(%rsp) + +// Set the tab[0] table entry to the input point = 1 * P + + movq (%rdx), %rax + movq %rax, TAB(%rsp) + movq 8(%rdx), %rax + movq %rax, TAB+8(%rsp) + movq 16(%rdx), %rax + movq %rax, TAB+16(%rsp) + movq 24(%rdx), %rax + movq %rax, TAB+24(%rsp) + movq 32(%rdx), %rax + movq %rax, TAB+32(%rsp) + movq 40(%rdx), %rax + movq %rax, TAB+40(%rsp) + + movq 48(%rdx), %rax + movq %rax, TAB+48(%rsp) + movq 56(%rdx), %rax + movq %rax, TAB+56(%rsp) + movq 64(%rdx), %rax + movq %rax, TAB+64(%rsp) + movq 72(%rdx), %rax + movq %rax, TAB+72(%rsp) + movq 80(%rdx), %rax + movq %rax, TAB+80(%rsp) + movq 88(%rdx), %rax + movq %rax, TAB+88(%rsp) + + movq 96(%rdx), %rax + movq %rax, TAB+96(%rsp) + movq 104(%rdx), %rax + movq %rax, TAB+104(%rsp) + movq 112(%rdx), %rax + movq %rax, TAB+112(%rsp) + movq 120(%rdx), %rax + movq %rax, TAB+120(%rsp) + movq 128(%rdx), %rax + movq %rax, TAB+128(%rsp) + movq 136(%rdx), %rax + movq %rax, TAB+136(%rsp) + +// Compute and record tab[1] = 2 * p, ..., tab[15] = 16 * P + + leaq TAB+JACSIZE*1(%rsp), %rdi + leaq TAB(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_p384_montjdouble) + + leaq TAB+JACSIZE*2(%rsp), %rdi + leaq TAB+JACSIZE*1(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp384_montjscalarmul_p384_montjadd) + + leaq TAB+JACSIZE*3(%rsp), %rdi + leaq TAB+JACSIZE*1(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_p384_montjdouble) + + leaq TAB+JACSIZE*4(%rsp), %rdi + leaq TAB+JACSIZE*3(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp384_montjscalarmul_p384_montjadd) + + leaq TAB+JACSIZE*5(%rsp), %rdi + leaq TAB+JACSIZE*2(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_p384_montjdouble) + + leaq TAB+JACSIZE*6(%rsp), %rdi + leaq TAB+JACSIZE*5(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp384_montjscalarmul_p384_montjadd) + + leaq TAB+JACSIZE*7(%rsp), %rdi + leaq TAB+JACSIZE*3(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_p384_montjdouble) + + leaq TAB+JACSIZE*8(%rsp), %rdi + leaq TAB+JACSIZE*7(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp384_montjscalarmul_p384_montjadd) + + leaq TAB+JACSIZE*9(%rsp), %rdi + leaq TAB+JACSIZE*4(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_p384_montjdouble) + + leaq TAB+JACSIZE*10(%rsp), %rdi + leaq TAB+JACSIZE*9(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp384_montjscalarmul_p384_montjadd) + + leaq TAB+JACSIZE*11(%rsp), %rdi + leaq TAB+JACSIZE*5(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_p384_montjdouble) + + leaq TAB+JACSIZE*12(%rsp), %rdi + leaq TAB+JACSIZE*11(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp384_montjscalarmul_p384_montjadd) + + leaq TAB+JACSIZE*13(%rsp), %rdi + leaq TAB+JACSIZE*6(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_p384_montjdouble) + + leaq TAB+JACSIZE*14(%rsp), %rdi + leaq TAB+JACSIZE*13(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp384_montjscalarmul_p384_montjadd) + + leaq TAB+JACSIZE*15(%rsp), %rdi + leaq TAB+JACSIZE*7(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_p384_montjdouble) + +// Add the recoding constant sum_i(16 * 32^i) to the scalar to allow signed +// digits. The digits of the constant, in lowest-to-highest order, are as +// follows; they are generated dynamically to use fewer large constant loads. +// +// 0x0842108421084210 +// 0x1084210842108421 +// 0x2108421084210842 +// 0x4210842108421084 +// 0x8421084210842108 +// 0x0842108421084210 + + movq $0x1084210842108421, %rax + movq %rax, %rcx + shrq $1, %rax + movq SCALARB(%rsp), %r8 + addq %rax, %r8 + movq SCALARB+8(%rsp), %r9 + adcq %rcx, %r9 + leaq (%rcx,%rcx), %rcx + movq SCALARB+16(%rsp), %r10 + adcq %rcx, %r10 + leaq (%rcx,%rcx), %rcx + movq SCALARB+24(%rsp), %r11 + adcq %rcx, %r11 + leaq (%rcx,%rcx), %rcx + movq SCALARB+32(%rsp), %r12 + adcq %rcx, %r12 + movq SCALARB+40(%rsp), %r13 + adcq %rax, %r13 + sbbq %rdi, %rdi + negq %rdi + +// Record the top bitfield in %rdi then shift the whole scalar left 4 bits +// to align the top of the next bitfield with the MSB (bits 379..383). + + shldq $4, %r13, %rdi + shldq $4, %r12, %r13 + shldq $4, %r11, %r12 + shldq $4, %r10, %r11 + shldq $4, %r9, %r10 + shldq $4, %r8, %r9 + shlq $4, %r8 + + movq %r8, SCALARB(%rsp) + movq %r9, SCALARB+8(%rsp) + movq %r10, SCALARB+16(%rsp) + movq %r11, SCALARB+24(%rsp) + movq %r12, SCALARB+32(%rsp) + movq %r13, SCALARB+40(%rsp) + +// Initialize the accumulator to the corresponding entry using constant-time +// lookup in the table. This top digit, uniquely, is not recoded so there is +// no sign adjustment to make. On the x86 integer side we don't have enough +// registers to hold all the fields; this could be better done with SIMD +// registers anyway. So we do x and z coordinates in one sweep, y in another +// (this is a rehearsal for below where we might need to negate the y). + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + xorl %r13d, %r13d + xorl %r14d, %r14d + xorl %r15d, %r15d + + selectblock_xz(1) + selectblock_xz(2) + selectblock_xz(3) + selectblock_xz(4) + selectblock_xz(5) + selectblock_xz(6) + selectblock_xz(7) + selectblock_xz(8) + selectblock_xz(9) + selectblock_xz(10) + selectblock_xz(11) + selectblock_xz(12) + selectblock_xz(13) + selectblock_xz(14) + selectblock_xz(15) + selectblock_xz(16) + + movq %rax, ACC(%rsp) + movq %rbx, ACC+8(%rsp) + movq %rcx, ACC+16(%rsp) + movq %rdx, ACC+24(%rsp) + movq %r8, ACC+32(%rsp) + movq %r9, ACC+40(%rsp) + movq %r10, ACC+96(%rsp) + movq %r11, ACC+104(%rsp) + movq %r12, ACC+112(%rsp) + movq %r13, ACC+120(%rsp) + movq %r14, ACC+128(%rsp) + movq %r15, ACC+136(%rsp) + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + + selectblock_y(1) + selectblock_y(2) + selectblock_y(3) + selectblock_y(4) + selectblock_y(5) + selectblock_y(6) + selectblock_y(7) + selectblock_y(8) + selectblock_y(9) + selectblock_y(10) + selectblock_y(11) + selectblock_y(12) + selectblock_y(13) + selectblock_y(14) + selectblock_y(15) + selectblock_y(16) + + movq %rax, ACC+48(%rsp) + movq %rbx, ACC+56(%rsp) + movq %rcx, ACC+64(%rsp) + movq %rdx, ACC+72(%rsp) + movq %r8, ACC+80(%rsp) + movq %r9, ACC+88(%rsp) + +// Main loop over size-5 bitfields: double 5 times then add signed digit +// At each stage we shift the scalar left by 5 bits so we can simply pick +// the top 5 bits as the bitfield, saving some fiddle over indexing. + + movl $380, %ebp + +Lp384_montjscalarmul_mainloop: + subq $5, %rbp + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp384_montjscalarmul_p384_montjdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp384_montjscalarmul_p384_montjdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp384_montjscalarmul_p384_montjdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp384_montjscalarmul_p384_montjdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp384_montjscalarmul_p384_montjdouble) + +// Choose the bitfield and adjust it to sign and magnitude + + movq SCALARB(%rsp), %r8 + movq SCALARB+8(%rsp), %r9 + movq SCALARB+16(%rsp), %r10 + movq SCALARB+24(%rsp), %r11 + movq SCALARB+32(%rsp), %r12 + movq SCALARB+40(%rsp), %r13 + + movq %r13, %rdi + shrq $59, %rdi + shldq $5, %r12, %r13 + shldq $5, %r11, %r12 + shldq $5, %r10, %r11 + shldq $5, %r9, %r10 + shldq $5, %r8, %r9 + shlq $5, %r8 + + movq %r8, SCALARB(%rsp) + movq %r9, SCALARB+8(%rsp) + movq %r10, SCALARB+16(%rsp) + movq %r11, SCALARB+24(%rsp) + movq %r12, SCALARB+32(%rsp) + movq %r13, SCALARB+40(%rsp) + + subq $16, %rdi + sbbq %rsi, %rsi // %rsi = sign of digit (-1 = negative) + xorq %rsi, %rdi + subq %rsi, %rdi // %rdi = absolute value of digit + +// Conditionally select the table entry tab[i-1] = i * P in constant time +// Again, this is done in two sweeps, first doing x and z then y. + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + xorl %r13d, %r13d + xorl %r14d, %r14d + xorl %r15d, %r15d + + selectblock_xz(1) + selectblock_xz(2) + selectblock_xz(3) + selectblock_xz(4) + selectblock_xz(5) + selectblock_xz(6) + selectblock_xz(7) + selectblock_xz(8) + selectblock_xz(9) + selectblock_xz(10) + selectblock_xz(11) + selectblock_xz(12) + selectblock_xz(13) + selectblock_xz(14) + selectblock_xz(15) + selectblock_xz(16) + + movq %rax, TABENT(%rsp) + movq %rbx, TABENT+8(%rsp) + movq %rcx, TABENT+16(%rsp) + movq %rdx, TABENT+24(%rsp) + movq %r8, TABENT+32(%rsp) + movq %r9, TABENT+40(%rsp) + movq %r10, TABENT+96(%rsp) + movq %r11, TABENT+104(%rsp) + movq %r12, TABENT+112(%rsp) + movq %r13, TABENT+120(%rsp) + movq %r14, TABENT+128(%rsp) + movq %r15, TABENT+136(%rsp) + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + + selectblock_y(1) + selectblock_y(2) + selectblock_y(3) + selectblock_y(4) + selectblock_y(5) + selectblock_y(6) + selectblock_y(7) + selectblock_y(8) + selectblock_y(9) + selectblock_y(10) + selectblock_y(11) + selectblock_y(12) + selectblock_y(13) + selectblock_y(14) + selectblock_y(15) + selectblock_y(16) + +// Store it to "tabent" with the y coordinate optionally negated. +// This is done carefully to give coordinates < p_384 even in +// the degenerate case y = 0 (when z = 0 for points on the curve). +// The digits of the prime p_384 are generated dynamically from +// the zeroth via not/lea to reduce the number of constant loads. + + movq %rax, %r10 + orq %rbx, %r10 + movq %rcx, %r11 + orq %rdx, %r11 + movq %r8, %r12 + orq %r9, %r12 + orq %r11, %r10 + orq %r12, %r10 + cmovzq %r10, %rsi + + movl $0xffffffff, %r10d + movq %r10, %r11 + notq %r11 + leaq (%r10,%r11), %r13 + subq %rax, %r10 + leaq -1(%r13), %r12 + sbbq %rbx, %r11 + movq %r13, %r14 + sbbq %rcx, %r12 + sbbq %rdx, %r13 + movq %r14, %r15 + sbbq %r8, %r14 + sbbq %r9, %r15 + + testq %rsi, %rsi + cmovnzq %r10, %rax + cmovnzq %r11, %rbx + cmovnzq %r12, %rcx + cmovnzq %r13, %rdx + cmovnzq %r14, %r8 + cmovnzq %r15, %r9 + + movq %rax, TABENT+48(%rsp) + movq %rbx, TABENT+56(%rsp) + movq %rcx, TABENT+64(%rsp) + movq %rdx, TABENT+72(%rsp) + movq %r8, TABENT+80(%rsp) + movq %r9, TABENT+88(%rsp) + +// Add to the accumulator + + leaq TABENT(%rsp), %rdx + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp384_montjscalarmul_p384_montjadd) + + testq %rbp, %rbp + jne Lp384_montjscalarmul_mainloop + +// That's the end of the main loop, and we just need to copy the +// result in "acc" to the output. + + movq res, %rdi + movq ACC(%rsp), %rax + movq %rax, (%rdi) + movq ACC+8(%rsp), %rax + movq %rax, 8(%rdi) + movq ACC+16(%rsp), %rax + movq %rax, 16(%rdi) + movq ACC+24(%rsp), %rax + movq %rax, 24(%rdi) + movq ACC+32(%rsp), %rax + movq %rax, 32(%rdi) + movq ACC+40(%rsp), %rax + movq %rax, 40(%rdi) + movq ACC+48(%rsp), %rax + movq %rax, 48(%rdi) + movq ACC+56(%rsp), %rax + movq %rax, 56(%rdi) + movq ACC+64(%rsp), %rax + movq %rax, 64(%rdi) + movq ACC+72(%rsp), %rax + movq %rax, 72(%rdi) + movq ACC+80(%rsp), %rax + movq %rax, 80(%rdi) + movq ACC+88(%rsp), %rax + movq %rax, 88(%rdi) + movq ACC+96(%rsp), %rax + movq %rax, 96(%rdi) + movq ACC+104(%rsp), %rax + movq %rax, 104(%rdi) + movq ACC+112(%rsp), %rax + movq %rax, 112(%rdi) + movq ACC+120(%rsp), %rax + movq %rax, 120(%rdi) + movq ACC+128(%rsp), %rax + movq %rax, 128(%rdi) + movq ACC+136(%rsp), %rax + movq %rax, 136(%rdi) + +// Restore stack and registers and return + + CFI_INC_RSP(NSPACE) + CFI_POP(%rbx) + CFI_POP(%rbp) + CFI_POP(%r12) + CFI_POP(%r13) + CFI_POP(%r14) + CFI_POP(%r15) + CFI_RET + +#if WINDOWS_ABI +S2N_BN_SIZE_DIRECTIVE(Lp384_montjscalarmul_standard) +#else +S2N_BN_SIZE_DIRECTIVE(p384_montjscalarmul) +#endif + +// Local copies of subroutines, complete clones at the moment + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp384_montjscalarmul_p384_montjadd) + +Lp384_montjscalarmul_p384_montjadd: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(352) + movq %rsi, 0x150(%rsp) + movq %rdx, 0x158(%rsp) + movq 0x60(%rsi), %rdx + mulxq 0x68(%rsi), %r9, %r10 + mulxq 0x78(%rsi), %r11, %r12 + mulxq 0x88(%rsi), %r13, %r14 + movq 0x78(%rsi), %rdx + mulxq 0x80(%rsi), %r15, %rcx + xorl %ebp, %ebp + movq 0x70(%rsi), %rdx + mulxq 0x60(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x68(%rsi), %rdx + mulxq 0x78(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x80(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x88(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %rcx + adcq %rbp, %rcx + xorl %ebp, %ebp + movq 0x80(%rsi), %rdx + mulxq 0x60(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq 0x70(%rsi), %rdx + mulxq 0x78(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x80(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x88(%rsi), %rax, %rdx + adcxq %rax, %r15 + adoxq %rdx, %rcx + movq 0x88(%rsi), %rdx + mulxq 0x80(%rsi), %rbx, %rbp + mulxq 0x78(%rsi), %rax, %rdx + adcxq %rax, %rcx + adoxq %rdx, %rbx + movl $0x0, %eax + adcxq %rax, %rbx + adoxq %rax, %rbp + adcq %rax, %rbp + xorq %rax, %rax + movq 0x60(%rsi), %rdx + mulxq 0x60(%rsi), %r8, %rax + adcxq %r9, %r9 + adoxq %rax, %r9 + movq 0x68(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x70(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x78(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %r15, %r15 + adoxq %rdx, %r15 + movq 0x80(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %rcx, %rcx + adoxq %rax, %rcx + adcxq %rbx, %rbx + adoxq %rdx, %rbx + movq 0x88(%rsi), %rdx + mulxq %rdx, %rax, %rsi + adcxq %rbp, %rbp + adoxq %rax, %rbp + movl $0x0, %eax + adcxq %rax, %rsi + adoxq %rax, %rsi + movq %rbx, (%rsp) + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r8, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r8 + addq %rbx, %rax + adcq %rdx, %r8 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r9 + sbbq %r8, %r10 + sbbq %rbx, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rdx, %r8 + sbbq $0x0, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r9, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r9 + addq %rbx, %rax + adcq %rdx, %r9 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r10 + sbbq %r9, %r11 + sbbq %rbx, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rdx, %r9 + sbbq $0x0, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r10, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r10 + addq %rbx, %rax + adcq %rdx, %r10 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r11 + sbbq %r10, %r12 + sbbq %rbx, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rdx, %r10 + sbbq $0x0, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r11, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r11 + addq %rbx, %rax + adcq %rdx, %r11 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r12 + sbbq %r11, %r13 + sbbq %rbx, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rdx, %r11 + sbbq $0x0, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r12, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r12 + addq %rbx, %rax + adcq %rdx, %r12 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r13 + sbbq %r12, %r8 + sbbq %rbx, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rdx, %r12 + sbbq $0x0, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r13, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r13 + addq %rbx, %rax + adcq %rdx, %r13 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r8 + sbbq %r13, %r9 + sbbq %rbx, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rdx, %r13 + sbbq $0x0, %r13 + movq (%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rsi + movl $0x0, %r8d + movabsq $0xffffffff00000001, %rax + movl $0xffffffff, %r9d + movl $0x1, %r10d + cmovaeq %r8, %rax + cmovaeq %r8, %r9 + cmovaeq %r8, %r10 + addq %rax, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r8, %rbx + adcq %r8, %rbp + adcq %r8, %rsi + movq %r14, (%rsp) + movq %r15, 0x8(%rsp) + movq %rcx, 0x10(%rsp) + movq %rbx, 0x18(%rsp) + movq %rbp, 0x20(%rsp) + movq %rsi, 0x28(%rsp) + movq 0x158(%rsp), %rsi + movq 0x60(%rsi), %rdx + mulxq 0x68(%rsi), %r9, %r10 + mulxq 0x78(%rsi), %r11, %r12 + mulxq 0x88(%rsi), %r13, %r14 + movq 0x78(%rsi), %rdx + mulxq 0x80(%rsi), %r15, %rcx + xorl %ebp, %ebp + movq 0x70(%rsi), %rdx + mulxq 0x60(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x68(%rsi), %rdx + mulxq 0x78(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x80(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x88(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %rcx + adcq %rbp, %rcx + xorl %ebp, %ebp + movq 0x80(%rsi), %rdx + mulxq 0x60(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq 0x70(%rsi), %rdx + mulxq 0x78(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x80(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x88(%rsi), %rax, %rdx + adcxq %rax, %r15 + adoxq %rdx, %rcx + movq 0x88(%rsi), %rdx + mulxq 0x80(%rsi), %rbx, %rbp + mulxq 0x78(%rsi), %rax, %rdx + adcxq %rax, %rcx + adoxq %rdx, %rbx + movl $0x0, %eax + adcxq %rax, %rbx + adoxq %rax, %rbp + adcq %rax, %rbp + xorq %rax, %rax + movq 0x60(%rsi), %rdx + mulxq 0x60(%rsi), %r8, %rax + adcxq %r9, %r9 + adoxq %rax, %r9 + movq 0x68(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x70(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x78(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %r15, %r15 + adoxq %rdx, %r15 + movq 0x80(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %rcx, %rcx + adoxq %rax, %rcx + adcxq %rbx, %rbx + adoxq %rdx, %rbx + movq 0x88(%rsi), %rdx + mulxq %rdx, %rax, %rsi + adcxq %rbp, %rbp + adoxq %rax, %rbp + movl $0x0, %eax + adcxq %rax, %rsi + adoxq %rax, %rsi + movq %rbx, 0xf0(%rsp) + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r8, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r8 + addq %rbx, %rax + adcq %rdx, %r8 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r9 + sbbq %r8, %r10 + sbbq %rbx, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rdx, %r8 + sbbq $0x0, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r9, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r9 + addq %rbx, %rax + adcq %rdx, %r9 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r10 + sbbq %r9, %r11 + sbbq %rbx, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rdx, %r9 + sbbq $0x0, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r10, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r10 + addq %rbx, %rax + adcq %rdx, %r10 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r11 + sbbq %r10, %r12 + sbbq %rbx, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rdx, %r10 + sbbq $0x0, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r11, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r11 + addq %rbx, %rax + adcq %rdx, %r11 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r12 + sbbq %r11, %r13 + sbbq %rbx, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rdx, %r11 + sbbq $0x0, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r12, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r12 + addq %rbx, %rax + adcq %rdx, %r12 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r13 + sbbq %r12, %r8 + sbbq %rbx, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rdx, %r12 + sbbq $0x0, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r13, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r13 + addq %rbx, %rax + adcq %rdx, %r13 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r8 + sbbq %r13, %r9 + sbbq %rbx, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rdx, %r13 + sbbq $0x0, %r13 + movq 0xf0(%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rsi + movl $0x0, %r8d + movabsq $0xffffffff00000001, %rax + movl $0xffffffff, %r9d + movl $0x1, %r10d + cmovaeq %r8, %rax + cmovaeq %r8, %r9 + cmovaeq %r8, %r10 + addq %rax, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r8, %rbx + adcq %r8, %rbp + adcq %r8, %rsi + movq %r14, 0xf0(%rsp) + movq %r15, 0xf8(%rsp) + movq %rcx, 0x100(%rsp) + movq %rbx, 0x108(%rsp) + movq %rbp, 0x110(%rsp) + movq %rsi, 0x118(%rsp) + movq 0x150(%rsp), %rsi + movq 0x158(%rsp), %rcx + movq 0x30(%rsi), %rdx + xorl %r15d, %r15d + mulxq 0x60(%rcx), %r8, %r9 + mulxq 0x68(%rcx), %rbx, %r10 + addq %rbx, %r9 + mulxq 0x70(%rcx), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x78(%rcx), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0x80(%rcx), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0x88(%rcx), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r8, %rbx + adcq %r8, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rbx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rdx + addq %rdx, %r14 + adcq $0x0, %r15 + movq 0x38(%rsi), %rdx + xorl %r8d, %r8d + mulxq 0x60(%rcx), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x68(%rcx), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x70(%rcx), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x78(%rcx), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x80(%rcx), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r8, %r15 + mulxq 0x88(%rcx), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r9, %rbx + adcq %r9, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rbx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rdx + addq %rdx, %r15 + adcq $0x0, %r8 + movq 0x40(%rsi), %rdx + xorl %r9d, %r9d + mulxq 0x60(%rcx), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rcx), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x70(%rcx), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x78(%rcx), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x80(%rcx), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adoxq %r9, %r8 + mulxq 0x88(%rcx), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r10, %rbx + adcq %r10, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rbx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rdx + addq %rdx, %r8 + adcq $0x0, %r9 + movq 0x48(%rsi), %rdx + xorl %r10d, %r10d + mulxq 0x60(%rcx), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x68(%rcx), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x70(%rcx), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x78(%rcx), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x80(%rcx), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + adoxq %r10, %r9 + mulxq 0x88(%rcx), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r11, %rbx + adcq %r11, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rbx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rdx + addq %rdx, %r9 + adcq $0x0, %r10 + movq 0x50(%rsi), %rdx + xorl %r11d, %r11d + mulxq 0x60(%rcx), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x68(%rcx), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x70(%rcx), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x78(%rcx), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x80(%rcx), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + adoxq %r11, %r10 + mulxq 0x88(%rcx), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r12, %rbx + adcq %r12, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rbx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rdx + addq %rdx, %r10 + adcq $0x0, %r11 + movq 0x58(%rsi), %rdx + xorl %r12d, %r12d + mulxq 0x60(%rcx), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x68(%rcx), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x70(%rcx), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x78(%rcx), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x80(%rcx), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + adoxq %r12, %r11 + mulxq 0x88(%rcx), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r13, %rbx + adcq %r13, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rbx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rdx + addq %rdx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x120(%rsp) + movq %r15, 0x128(%rsp) + movq %r8, 0x130(%rsp) + movq %r9, 0x138(%rsp) + movq %r10, 0x140(%rsp) + movq %r11, 0x148(%rsp) + movq 0x150(%rsp), %rsi + movq 0x158(%rsp), %rcx + movq 0x30(%rcx), %rdx + xorl %r15d, %r15d + mulxq 0x60(%rsi), %r8, %r9 + mulxq 0x68(%rsi), %rbx, %r10 + addq %rbx, %r9 + mulxq 0x70(%rsi), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x78(%rsi), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0x80(%rsi), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0x88(%rsi), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r8, %rbx + adcq %r8, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rbx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rdx + addq %rdx, %r14 + adcq $0x0, %r15 + movq 0x38(%rcx), %rdx + xorl %r8d, %r8d + mulxq 0x60(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x68(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x70(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x78(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x80(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r8, %r15 + mulxq 0x88(%rsi), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r9, %rbx + adcq %r9, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rbx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rdx + addq %rdx, %r15 + adcq $0x0, %r8 + movq 0x40(%rcx), %rdx + xorl %r9d, %r9d + mulxq 0x60(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x70(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x78(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x80(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adoxq %r9, %r8 + mulxq 0x88(%rsi), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r10, %rbx + adcq %r10, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rbx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rdx + addq %rdx, %r8 + adcq $0x0, %r9 + movq 0x48(%rcx), %rdx + xorl %r10d, %r10d + mulxq 0x60(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x68(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x70(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x78(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x80(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + adoxq %r10, %r9 + mulxq 0x88(%rsi), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r11, %rbx + adcq %r11, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rbx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rdx + addq %rdx, %r9 + adcq $0x0, %r10 + movq 0x50(%rcx), %rdx + xorl %r11d, %r11d + mulxq 0x60(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x68(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x70(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x78(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x80(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + adoxq %r11, %r10 + mulxq 0x88(%rsi), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r12, %rbx + adcq %r12, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rbx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rdx + addq %rdx, %r10 + adcq $0x0, %r11 + movq 0x58(%rcx), %rdx + xorl %r12d, %r12d + mulxq 0x60(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x68(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x70(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x78(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x80(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + adoxq %r12, %r11 + mulxq 0x88(%rsi), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r13, %rbx + adcq %r13, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rbx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rdx + addq %rdx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq %r8, 0x40(%rsp) + movq %r9, 0x48(%rsp) + movq %r10, 0x50(%rsp) + movq %r11, 0x58(%rsp) + movq 0x158(%rsp), %rcx + movq (%rcx), %rdx + xorl %r15d, %r15d + mulxq (%rsp), %r8, %r9 + mulxq 0x8(%rsp), %rbx, %r10 + addq %rbx, %r9 + mulxq 0x10(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x18(%rsp), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0x20(%rsp), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0x28(%rsp), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r8, %rbx + adcq %r8, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rbx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rdx + addq %rdx, %r14 + adcq $0x0, %r15 + movq 0x8(%rcx), %rdx + xorl %r8d, %r8d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r8, %r15 + mulxq 0x28(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r9, %rbx + adcq %r9, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rbx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rdx + addq %rdx, %r15 + adcq $0x0, %r8 + movq 0x10(%rcx), %rdx + xorl %r9d, %r9d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adoxq %r9, %r8 + mulxq 0x28(%rsp), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r10, %rbx + adcq %r10, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rbx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rdx + addq %rdx, %r8 + adcq $0x0, %r9 + movq 0x18(%rcx), %rdx + xorl %r10d, %r10d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + adoxq %r10, %r9 + mulxq 0x28(%rsp), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r11, %rbx + adcq %r11, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rbx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rdx + addq %rdx, %r9 + adcq $0x0, %r10 + movq 0x20(%rcx), %rdx + xorl %r11d, %r11d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + adoxq %r11, %r10 + mulxq 0x28(%rsp), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r12, %rbx + adcq %r12, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rbx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rdx + addq %rdx, %r10 + adcq $0x0, %r11 + movq 0x28(%rcx), %rdx + xorl %r12d, %r12d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + adoxq %r12, %r11 + mulxq 0x28(%rsp), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r13, %rbx + adcq %r13, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rbx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rdx + addq %rdx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x60(%rsp) + movq %r15, 0x68(%rsp) + movq %r8, 0x70(%rsp) + movq %r9, 0x78(%rsp) + movq %r10, 0x80(%rsp) + movq %r11, 0x88(%rsp) + movq 0x150(%rsp), %rsi + movq (%rsi), %rdx + xorl %r15d, %r15d + mulxq 0xf0(%rsp), %r8, %r9 + mulxq 0xf8(%rsp), %rbx, %r10 + addq %rbx, %r9 + mulxq 0x100(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x108(%rsp), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0x110(%rsp), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0x118(%rsp), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r8, %rbx + adcq %r8, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rbx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rdx + addq %rdx, %r14 + adcq $0x0, %r15 + movq 0x8(%rsi), %rdx + xorl %r8d, %r8d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r8, %r15 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r9, %rbx + adcq %r9, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rbx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rdx + addq %rdx, %r15 + adcq $0x0, %r8 + movq 0x10(%rsi), %rdx + xorl %r9d, %r9d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adoxq %r9, %r8 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r10, %rbx + adcq %r10, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rbx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rdx + addq %rdx, %r8 + adcq $0x0, %r9 + movq 0x18(%rsi), %rdx + xorl %r10d, %r10d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + adoxq %r10, %r9 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r11, %rbx + adcq %r11, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rbx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rdx + addq %rdx, %r9 + adcq $0x0, %r10 + movq 0x20(%rsi), %rdx + xorl %r11d, %r11d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + adoxq %r11, %r10 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r12, %rbx + adcq %r12, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rbx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rdx + addq %rdx, %r10 + adcq $0x0, %r11 + movq 0x28(%rsi), %rdx + xorl %r12d, %r12d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + adoxq %r12, %r11 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r13, %rbx + adcq %r13, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rbx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rdx + addq %rdx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0xc0(%rsp) + movq %r15, 0xc8(%rsp) + movq %r8, 0xd0(%rsp) + movq %r9, 0xd8(%rsp) + movq %r10, 0xe0(%rsp) + movq %r11, 0xe8(%rsp) + movq 0x30(%rsp), %rdx + xorl %r15d, %r15d + mulxq (%rsp), %r8, %r9 + mulxq 0x8(%rsp), %rbx, %r10 + addq %rbx, %r9 + mulxq 0x10(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x18(%rsp), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0x20(%rsp), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0x28(%rsp), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r8, %rbx + adcq %r8, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rbx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rdx + addq %rdx, %r14 + adcq $0x0, %r15 + movq 0x38(%rsp), %rdx + xorl %r8d, %r8d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r8, %r15 + mulxq 0x28(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r9, %rbx + adcq %r9, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rbx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rdx + addq %rdx, %r15 + adcq $0x0, %r8 + movq 0x40(%rsp), %rdx + xorl %r9d, %r9d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adoxq %r9, %r8 + mulxq 0x28(%rsp), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r10, %rbx + adcq %r10, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rbx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rdx + addq %rdx, %r8 + adcq $0x0, %r9 + movq 0x48(%rsp), %rdx + xorl %r10d, %r10d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + adoxq %r10, %r9 + mulxq 0x28(%rsp), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r11, %rbx + adcq %r11, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rbx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rdx + addq %rdx, %r9 + adcq $0x0, %r10 + movq 0x50(%rsp), %rdx + xorl %r11d, %r11d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + adoxq %r11, %r10 + mulxq 0x28(%rsp), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r12, %rbx + adcq %r12, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rbx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rdx + addq %rdx, %r10 + adcq $0x0, %r11 + movq 0x58(%rsp), %rdx + xorl %r12d, %r12d + mulxq (%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x8(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x10(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x18(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x20(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + adoxq %r12, %r11 + mulxq 0x28(%rsp), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r13, %rbx + adcq %r13, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rbx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rdx + addq %rdx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq %r8, 0x40(%rsp) + movq %r9, 0x48(%rsp) + movq %r10, 0x50(%rsp) + movq %r11, 0x58(%rsp) + movq 0x120(%rsp), %rdx + xorl %r15d, %r15d + mulxq 0xf0(%rsp), %r8, %r9 + mulxq 0xf8(%rsp), %rbx, %r10 + addq %rbx, %r9 + mulxq 0x100(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x108(%rsp), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0x110(%rsp), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0x118(%rsp), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r8, %rbx + adcq %r8, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rbx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rdx + addq %rdx, %r14 + adcq $0x0, %r15 + movq 0x128(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r8, %r15 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r9, %rbx + adcq %r9, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rbx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rdx + addq %rdx, %r15 + adcq $0x0, %r8 + movq 0x130(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adoxq %r9, %r8 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r10, %rbx + adcq %r10, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rbx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rdx + addq %rdx, %r8 + adcq $0x0, %r9 + movq 0x138(%rsp), %rdx + xorl %r10d, %r10d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + adoxq %r10, %r9 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r11, %rbx + adcq %r11, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rbx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rdx + addq %rdx, %r9 + adcq $0x0, %r10 + movq 0x140(%rsp), %rdx + xorl %r11d, %r11d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + adoxq %r11, %r10 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r12, %rbx + adcq %r12, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rbx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rdx + addq %rdx, %r10 + adcq $0x0, %r11 + movq 0x148(%rsp), %rdx + xorl %r12d, %r12d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + adoxq %r12, %r11 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r13, %rbx + adcq %r13, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rbx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rdx + addq %rdx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x120(%rsp) + movq %r15, 0x128(%rsp) + movq %r8, 0x130(%rsp) + movq %r9, 0x138(%rsp) + movq %r10, 0x140(%rsp) + movq %r11, 0x148(%rsp) + movq 0x60(%rsp), %rax + subq 0xc0(%rsp), %rax + movq 0x68(%rsp), %rdx + sbbq 0xc8(%rsp), %rdx + movq 0x70(%rsp), %r8 + sbbq 0xd0(%rsp), %r8 + movq 0x78(%rsp), %r9 + sbbq 0xd8(%rsp), %r9 + movq 0x80(%rsp), %r10 + sbbq 0xe0(%rsp), %r10 + movq 0x88(%rsp), %r11 + sbbq 0xe8(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %esi + andq %rsi, %rcx + xorq %rsi, %rsi + subq %rcx, %rsi + subq %rsi, %rax + movq %rax, 0xf0(%rsp) + sbbq %rcx, %rdx + movq %rdx, 0xf8(%rsp) + sbbq %rax, %rax + andq %rsi, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0x100(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x108(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x110(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x118(%rsp) + movq 0x30(%rsp), %rax + subq 0x120(%rsp), %rax + movq 0x38(%rsp), %rdx + sbbq 0x128(%rsp), %rdx + movq 0x40(%rsp), %r8 + sbbq 0x130(%rsp), %r8 + movq 0x48(%rsp), %r9 + sbbq 0x138(%rsp), %r9 + movq 0x50(%rsp), %r10 + sbbq 0x140(%rsp), %r10 + movq 0x58(%rsp), %r11 + sbbq 0x148(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %esi + andq %rsi, %rcx + xorq %rsi, %rsi + subq %rcx, %rsi + subq %rsi, %rax + movq %rax, 0x30(%rsp) + sbbq %rcx, %rdx + movq %rdx, 0x38(%rsp) + sbbq %rax, %rax + andq %rsi, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0x40(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x48(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x50(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x58(%rsp) + movq 0xf0(%rsp), %rdx + mulxq 0xf8(%rsp), %r9, %r10 + mulxq 0x108(%rsp), %r11, %r12 + mulxq 0x118(%rsp), %r13, %r14 + movq 0x108(%rsp), %rdx + mulxq 0x110(%rsp), %r15, %rcx + xorl %ebp, %ebp + movq 0x100(%rsp), %rdx + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0xf8(%rsp), %rdx + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x118(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %rcx + adcq %rbp, %rcx + xorl %ebp, %ebp + movq 0x110(%rsp), %rdx + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq 0x100(%rsp), %rdx + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x118(%rsp), %rax, %rdx + adcxq %rax, %r15 + adoxq %rdx, %rcx + movq 0x118(%rsp), %rdx + mulxq 0x110(%rsp), %rbx, %rbp + mulxq 0x108(%rsp), %rax, %rdx + adcxq %rax, %rcx + adoxq %rdx, %rbx + movl $0x0, %eax + adcxq %rax, %rbx + adoxq %rax, %rbp + adcq %rax, %rbp + xorq %rax, %rax + movq 0xf0(%rsp), %rdx + mulxq 0xf0(%rsp), %r8, %rax + adcxq %r9, %r9 + adoxq %rax, %r9 + movq 0xf8(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x100(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x108(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %r15, %r15 + adoxq %rdx, %r15 + movq 0x110(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %rcx, %rcx + adoxq %rax, %rcx + adcxq %rbx, %rbx + adoxq %rdx, %rbx + movq 0x118(%rsp), %rdx + mulxq %rdx, %rax, %rsi + adcxq %rbp, %rbp + adoxq %rax, %rbp + movl $0x0, %eax + adcxq %rax, %rsi + adoxq %rax, %rsi + movq %rbx, 0x90(%rsp) + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r8, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r8 + addq %rbx, %rax + adcq %rdx, %r8 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r9 + sbbq %r8, %r10 + sbbq %rbx, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rdx, %r8 + sbbq $0x0, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r9, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r9 + addq %rbx, %rax + adcq %rdx, %r9 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r10 + sbbq %r9, %r11 + sbbq %rbx, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rdx, %r9 + sbbq $0x0, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r10, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r10 + addq %rbx, %rax + adcq %rdx, %r10 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r11 + sbbq %r10, %r12 + sbbq %rbx, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rdx, %r10 + sbbq $0x0, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r11, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r11 + addq %rbx, %rax + adcq %rdx, %r11 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r12 + sbbq %r11, %r13 + sbbq %rbx, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rdx, %r11 + sbbq $0x0, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r12, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r12 + addq %rbx, %rax + adcq %rdx, %r12 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r13 + sbbq %r12, %r8 + sbbq %rbx, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rdx, %r12 + sbbq $0x0, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r13, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r13 + addq %rbx, %rax + adcq %rdx, %r13 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r8 + sbbq %r13, %r9 + sbbq %rbx, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rdx, %r13 + sbbq $0x0, %r13 + movq 0x90(%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rsi + movl $0x0, %r8d + movabsq $0xffffffff00000001, %rax + movl $0xffffffff, %r9d + movl $0x1, %r10d + cmovaeq %r8, %rax + cmovaeq %r8, %r9 + cmovaeq %r8, %r10 + addq %rax, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r8, %rbx + adcq %r8, %rbp + adcq %r8, %rsi + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq %rcx, 0xa0(%rsp) + movq %rbx, 0xa8(%rsp) + movq %rbp, 0xb0(%rsp) + movq %rsi, 0xb8(%rsp) + movq 0x30(%rsp), %rdx + mulxq 0x38(%rsp), %r9, %r10 + mulxq 0x48(%rsp), %r11, %r12 + mulxq 0x58(%rsp), %r13, %r14 + movq 0x48(%rsp), %rdx + mulxq 0x50(%rsp), %r15, %rcx + xorl %ebp, %ebp + movq 0x40(%rsp), %rdx + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x38(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x38(%rsp), %rdx + mulxq 0x48(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x50(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x58(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %rcx + adcq %rbp, %rcx + xorl %ebp, %ebp + movq 0x50(%rsp), %rdx + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq 0x40(%rsp), %rdx + mulxq 0x48(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x50(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x58(%rsp), %rax, %rdx + adcxq %rax, %r15 + adoxq %rdx, %rcx + movq 0x58(%rsp), %rdx + mulxq 0x50(%rsp), %rbx, %rbp + mulxq 0x48(%rsp), %rax, %rdx + adcxq %rax, %rcx + adoxq %rdx, %rbx + movl $0x0, %eax + adcxq %rax, %rbx + adoxq %rax, %rbp + adcq %rax, %rbp + xorq %rax, %rax + movq 0x30(%rsp), %rdx + mulxq 0x30(%rsp), %r8, %rax + adcxq %r9, %r9 + adoxq %rax, %r9 + movq 0x38(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x40(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x48(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %r15, %r15 + adoxq %rdx, %r15 + movq 0x50(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %rcx, %rcx + adoxq %rax, %rcx + adcxq %rbx, %rbx + adoxq %rdx, %rbx + movq 0x58(%rsp), %rdx + mulxq %rdx, %rax, %rsi + adcxq %rbp, %rbp + adoxq %rax, %rbp + movl $0x0, %eax + adcxq %rax, %rsi + adoxq %rax, %rsi + movq %rbx, (%rsp) + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r8, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r8 + addq %rbx, %rax + adcq %rdx, %r8 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r9 + sbbq %r8, %r10 + sbbq %rbx, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rdx, %r8 + sbbq $0x0, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r9, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r9 + addq %rbx, %rax + adcq %rdx, %r9 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r10 + sbbq %r9, %r11 + sbbq %rbx, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rdx, %r9 + sbbq $0x0, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r10, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r10 + addq %rbx, %rax + adcq %rdx, %r10 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r11 + sbbq %r10, %r12 + sbbq %rbx, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rdx, %r10 + sbbq $0x0, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r11, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r11 + addq %rbx, %rax + adcq %rdx, %r11 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r12 + sbbq %r11, %r13 + sbbq %rbx, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rdx, %r11 + sbbq $0x0, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r12, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r12 + addq %rbx, %rax + adcq %rdx, %r12 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r13 + sbbq %r12, %r8 + sbbq %rbx, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rdx, %r12 + sbbq $0x0, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r13, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r13 + addq %rbx, %rax + adcq %rdx, %r13 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r8 + sbbq %r13, %r9 + sbbq %rbx, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rdx, %r13 + sbbq $0x0, %r13 + movq (%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rsi + movl $0x0, %r8d + adcq %r8, %r8 + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %r9d + adcq %r15, %r9 + movl $0x1, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rsi, %r13 + adcq $0x0, %r8 + cmovneq %rax, %r14 + cmovneq %r9, %r15 + cmovneq %r10, %rcx + cmovneq %r11, %rbx + cmovneq %r12, %rbp + cmovneq %r13, %rsi + movq %r14, (%rsp) + movq %r15, 0x8(%rsp) + movq %rcx, 0x10(%rsp) + movq %rbx, 0x18(%rsp) + movq %rbp, 0x20(%rsp) + movq %rsi, 0x28(%rsp) + movq 0xc0(%rsp), %rdx + xorl %r15d, %r15d + mulxq 0x90(%rsp), %r8, %r9 + mulxq 0x98(%rsp), %rbx, %r10 + addq %rbx, %r9 + mulxq 0xa0(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0xa8(%rsp), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0xb0(%rsp), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0xb8(%rsp), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r8, %rbx + adcq %r8, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rbx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rdx + addq %rdx, %r14 + adcq $0x0, %r15 + movq 0xc8(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r8, %r15 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r9, %rbx + adcq %r9, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rbx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rdx + addq %rdx, %r15 + adcq $0x0, %r8 + movq 0xd0(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adoxq %r9, %r8 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r10, %rbx + adcq %r10, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rbx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rdx + addq %rdx, %r8 + adcq $0x0, %r9 + movq 0xd8(%rsp), %rdx + xorl %r10d, %r10d + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + adoxq %r10, %r9 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r11, %rbx + adcq %r11, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rbx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rdx + addq %rdx, %r9 + adcq $0x0, %r10 + movq 0xe0(%rsp), %rdx + xorl %r11d, %r11d + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + adoxq %r11, %r10 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r12, %rbx + adcq %r12, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rbx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rdx + addq %rdx, %r10 + adcq $0x0, %r11 + movq 0xe8(%rsp), %rdx + xorl %r12d, %r12d + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + adoxq %r12, %r11 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r13, %rbx + adcq %r13, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rbx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rdx + addq %rdx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0xc0(%rsp) + movq %r15, 0xc8(%rsp) + movq %r8, 0xd0(%rsp) + movq %r9, 0xd8(%rsp) + movq %r10, 0xe0(%rsp) + movq %r11, 0xe8(%rsp) + movq 0x60(%rsp), %rdx + xorl %r15d, %r15d + mulxq 0x90(%rsp), %r8, %r9 + mulxq 0x98(%rsp), %rbx, %r10 + addq %rbx, %r9 + mulxq 0xa0(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0xa8(%rsp), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0xb0(%rsp), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0xb8(%rsp), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r8, %rbx + adcq %r8, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rbx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rdx + addq %rdx, %r14 + adcq $0x0, %r15 + movq 0x68(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r8, %r15 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r9, %rbx + adcq %r9, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rbx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rdx + addq %rdx, %r15 + adcq $0x0, %r8 + movq 0x70(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adoxq %r9, %r8 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r10, %rbx + adcq %r10, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rbx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rdx + addq %rdx, %r8 + adcq $0x0, %r9 + movq 0x78(%rsp), %rdx + xorl %r10d, %r10d + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + adoxq %r10, %r9 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r11, %rbx + adcq %r11, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rbx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rdx + addq %rdx, %r9 + adcq $0x0, %r10 + movq 0x80(%rsp), %rdx + xorl %r11d, %r11d + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + adoxq %r11, %r10 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r12, %rbx + adcq %r12, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rbx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rdx + addq %rdx, %r10 + adcq $0x0, %r11 + movq 0x88(%rsp), %rdx + xorl %r12d, %r12d + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + adoxq %r12, %r11 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r13, %rbx + adcq %r13, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rbx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rdx + addq %rdx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x60(%rsp) + movq %r15, 0x68(%rsp) + movq %r8, 0x70(%rsp) + movq %r9, 0x78(%rsp) + movq %r10, 0x80(%rsp) + movq %r11, 0x88(%rsp) + movq (%rsp), %rax + subq 0xc0(%rsp), %rax + movq 0x8(%rsp), %rdx + sbbq 0xc8(%rsp), %rdx + movq 0x10(%rsp), %r8 + sbbq 0xd0(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0xd8(%rsp), %r9 + movq 0x20(%rsp), %r10 + sbbq 0xe0(%rsp), %r10 + movq 0x28(%rsp), %r11 + sbbq 0xe8(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %esi + andq %rsi, %rcx + xorq %rsi, %rsi + subq %rcx, %rsi + subq %rsi, %rax + movq %rax, (%rsp) + sbbq %rcx, %rdx + movq %rdx, 0x8(%rsp) + sbbq %rax, %rax + andq %rsi, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0x10(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x18(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x20(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x28(%rsp) + movq 0x60(%rsp), %rax + subq 0xc0(%rsp), %rax + movq 0x68(%rsp), %rdx + sbbq 0xc8(%rsp), %rdx + movq 0x70(%rsp), %r8 + sbbq 0xd0(%rsp), %r8 + movq 0x78(%rsp), %r9 + sbbq 0xd8(%rsp), %r9 + movq 0x80(%rsp), %r10 + sbbq 0xe0(%rsp), %r10 + movq 0x88(%rsp), %r11 + sbbq 0xe8(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %esi + andq %rsi, %rcx + xorq %rsi, %rsi + subq %rcx, %rsi + subq %rsi, %rax + movq %rax, 0x90(%rsp) + sbbq %rcx, %rdx + movq %rdx, 0x98(%rsp) + sbbq %rax, %rax + andq %rsi, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0xa0(%rsp) + sbbq $0x0, %r9 + movq %r9, 0xa8(%rsp) + sbbq $0x0, %r10 + movq %r10, 0xb0(%rsp) + sbbq $0x0, %r11 + movq %r11, 0xb8(%rsp) + movq 0x150(%rsp), %rsi + movq 0x60(%rsi), %rdx + xorl %r15d, %r15d + mulxq 0xf0(%rsp), %r8, %r9 + mulxq 0xf8(%rsp), %rbx, %r10 + addq %rbx, %r9 + mulxq 0x100(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x108(%rsp), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0x110(%rsp), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0x118(%rsp), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r8, %rbx + adcq %r8, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rbx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rdx + addq %rdx, %r14 + adcq $0x0, %r15 + movq 0x68(%rsi), %rdx + xorl %r8d, %r8d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r8, %r15 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r9, %rbx + adcq %r9, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rbx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rdx + addq %rdx, %r15 + adcq $0x0, %r8 + movq 0x70(%rsi), %rdx + xorl %r9d, %r9d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adoxq %r9, %r8 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r10, %rbx + adcq %r10, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rbx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rdx + addq %rdx, %r8 + adcq $0x0, %r9 + movq 0x78(%rsi), %rdx + xorl %r10d, %r10d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + adoxq %r10, %r9 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r11, %rbx + adcq %r11, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rbx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rdx + addq %rdx, %r9 + adcq $0x0, %r10 + movq 0x80(%rsi), %rdx + xorl %r11d, %r11d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + adoxq %r11, %r10 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r12, %rbx + adcq %r12, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rbx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rdx + addq %rdx, %r10 + adcq $0x0, %r11 + movq 0x88(%rsi), %rdx + xorl %r12d, %r12d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + adoxq %r12, %r11 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r13, %rbx + adcq %r13, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rbx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rdx + addq %rdx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0xf0(%rsp) + movq %r15, 0xf8(%rsp) + movq %r8, 0x100(%rsp) + movq %r9, 0x108(%rsp) + movq %r10, 0x110(%rsp) + movq %r11, 0x118(%rsp) + movq (%rsp), %rax + subq 0x60(%rsp), %rax + movq 0x8(%rsp), %rdx + sbbq 0x68(%rsp), %rdx + movq 0x10(%rsp), %r8 + sbbq 0x70(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x78(%rsp), %r9 + movq 0x20(%rsp), %r10 + sbbq 0x80(%rsp), %r10 + movq 0x28(%rsp), %r11 + sbbq 0x88(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %esi + andq %rsi, %rcx + xorq %rsi, %rsi + subq %rcx, %rsi + subq %rsi, %rax + movq %rax, (%rsp) + sbbq %rcx, %rdx + movq %rdx, 0x8(%rsp) + sbbq %rax, %rax + andq %rsi, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0x10(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x18(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x20(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x28(%rsp) + movq 0xc0(%rsp), %rax + subq (%rsp), %rax + movq 0xc8(%rsp), %rdx + sbbq 0x8(%rsp), %rdx + movq 0xd0(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0xd8(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movq 0xe0(%rsp), %r10 + sbbq 0x20(%rsp), %r10 + movq 0xe8(%rsp), %r11 + sbbq 0x28(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %esi + andq %rsi, %rcx + xorq %rsi, %rsi + subq %rcx, %rsi + subq %rsi, %rax + movq %rax, 0xc0(%rsp) + sbbq %rcx, %rdx + movq %rdx, 0xc8(%rsp) + sbbq %rax, %rax + andq %rsi, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0xd0(%rsp) + sbbq $0x0, %r9 + movq %r9, 0xd8(%rsp) + sbbq $0x0, %r10 + movq %r10, 0xe0(%rsp) + sbbq $0x0, %r11 + movq %r11, 0xe8(%rsp) + movq 0x120(%rsp), %rdx + xorl %r15d, %r15d + mulxq 0x90(%rsp), %r8, %r9 + mulxq 0x98(%rsp), %rbx, %r10 + addq %rbx, %r9 + mulxq 0xa0(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0xa8(%rsp), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0xb0(%rsp), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0xb8(%rsp), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r8, %rbx + adcq %r8, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rbx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rdx + addq %rdx, %r14 + adcq $0x0, %r15 + movq 0x128(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r8, %r15 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r9, %rbx + adcq %r9, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rbx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rdx + addq %rdx, %r15 + adcq $0x0, %r8 + movq 0x130(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adoxq %r9, %r8 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r10, %rbx + adcq %r10, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rbx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rdx + addq %rdx, %r8 + adcq $0x0, %r9 + movq 0x138(%rsp), %rdx + xorl %r10d, %r10d + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + adoxq %r10, %r9 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r11, %rbx + adcq %r11, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rbx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rdx + addq %rdx, %r9 + adcq $0x0, %r10 + movq 0x140(%rsp), %rdx + xorl %r11d, %r11d + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + adoxq %r11, %r10 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r12, %rbx + adcq %r12, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rbx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rdx + addq %rdx, %r10 + adcq $0x0, %r11 + movq 0x148(%rsp), %rdx + xorl %r12d, %r12d + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + adoxq %r12, %r11 + mulxq 0xb8(%rsp), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r13, %rbx + adcq %r13, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rbx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rdx + addq %rdx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq %r8, 0xa0(%rsp) + movq %r9, 0xa8(%rsp) + movq %r10, 0xb0(%rsp) + movq %r11, 0xb8(%rsp) + movq 0x158(%rsp), %rcx + movq 0x60(%rcx), %rdx + xorl %r15d, %r15d + mulxq 0xf0(%rsp), %r8, %r9 + mulxq 0xf8(%rsp), %rbx, %r10 + addq %rbx, %r9 + mulxq 0x100(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x108(%rsp), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0x110(%rsp), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0x118(%rsp), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r8, %rbx + adcq %r8, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rbx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rdx + addq %rdx, %r14 + adcq $0x0, %r15 + movq 0x68(%rcx), %rdx + xorl %r8d, %r8d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r8, %r15 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r9, %rbx + adcq %r9, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rbx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rdx + addq %rdx, %r15 + adcq $0x0, %r8 + movq 0x70(%rcx), %rdx + xorl %r9d, %r9d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adoxq %r9, %r8 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r10, %rbx + adcq %r10, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rbx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rdx + addq %rdx, %r8 + adcq $0x0, %r9 + movq 0x78(%rcx), %rdx + xorl %r10d, %r10d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + adoxq %r10, %r9 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r11, %rbx + adcq %r11, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rbx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rdx + addq %rdx, %r9 + adcq $0x0, %r10 + movq 0x80(%rcx), %rdx + xorl %r11d, %r11d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + adoxq %r11, %r10 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r12, %rbx + adcq %r12, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rbx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rdx + addq %rdx, %r10 + adcq $0x0, %r11 + movq 0x88(%rcx), %rdx + xorl %r12d, %r12d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + adoxq %r12, %r11 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r13, %rbx + adcq %r13, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rbx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rdx + addq %rdx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0xf0(%rsp) + movq %r15, 0xf8(%rsp) + movq %r8, 0x100(%rsp) + movq %r9, 0x108(%rsp) + movq %r10, 0x110(%rsp) + movq %r11, 0x118(%rsp) + movq 0xc0(%rsp), %rdx + xorl %r15d, %r15d + mulxq 0x30(%rsp), %r8, %r9 + mulxq 0x38(%rsp), %rbx, %r10 + addq %rbx, %r9 + mulxq 0x40(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x48(%rsp), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0x50(%rsp), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0x58(%rsp), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r8, %rbx + adcq %r8, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rbx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rdx + addq %rdx, %r14 + adcq $0x0, %r15 + movq 0xc8(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x38(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x40(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x48(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x50(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r8, %r15 + mulxq 0x58(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r9, %rbx + adcq %r9, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rbx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rdx + addq %rdx, %r15 + adcq $0x0, %r8 + movq 0xd0(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x38(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x40(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x48(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x50(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adoxq %r9, %r8 + mulxq 0x58(%rsp), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r10, %rbx + adcq %r10, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rbx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rdx + addq %rdx, %r8 + adcq $0x0, %r9 + movq 0xd8(%rsp), %rdx + xorl %r10d, %r10d + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x38(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x40(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x48(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x50(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + adoxq %r10, %r9 + mulxq 0x58(%rsp), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r11, %rbx + adcq %r11, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rbx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rdx + addq %rdx, %r9 + adcq $0x0, %r10 + movq 0xe0(%rsp), %rdx + xorl %r11d, %r11d + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x38(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x40(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x48(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x50(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + adoxq %r11, %r10 + mulxq 0x58(%rsp), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r12, %rbx + adcq %r12, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rbx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rdx + addq %rdx, %r10 + adcq $0x0, %r11 + movq 0xe8(%rsp), %rdx + xorl %r12d, %r12d + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x38(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x40(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x48(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x50(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + adoxq %r12, %r11 + mulxq 0x58(%rsp), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r13, %rbx + adcq %r13, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rbx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rdx + addq %rdx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0xc0(%rsp) + movq %r15, 0xc8(%rsp) + movq %r8, 0xd0(%rsp) + movq %r9, 0xd8(%rsp) + movq %r10, 0xe0(%rsp) + movq %r11, 0xe8(%rsp) + movq 0xc0(%rsp), %rax + subq 0x90(%rsp), %rax + movq 0xc8(%rsp), %rdx + sbbq 0x98(%rsp), %rdx + movq 0xd0(%rsp), %r8 + sbbq 0xa0(%rsp), %r8 + movq 0xd8(%rsp), %r9 + sbbq 0xa8(%rsp), %r9 + movq 0xe0(%rsp), %r10 + sbbq 0xb0(%rsp), %r10 + movq 0xe8(%rsp), %r11 + sbbq 0xb8(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %esi + andq %rsi, %rcx + xorq %rsi, %rsi + subq %rcx, %rsi + subq %rsi, %rax + movq %rax, 0xc0(%rsp) + sbbq %rcx, %rdx + movq %rdx, 0xc8(%rsp) + sbbq %rax, %rax + andq %rsi, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0xd0(%rsp) + sbbq $0x0, %r9 + movq %r9, 0xd8(%rsp) + sbbq $0x0, %r10 + movq %r10, 0xe0(%rsp) + sbbq $0x0, %r11 + movq %r11, 0xe8(%rsp) + movq 0x158(%rsp), %rcx + movq 0x60(%rcx), %r8 + movq 0x68(%rcx), %r9 + movq 0x70(%rcx), %r10 + movq 0x78(%rcx), %r11 + movq 0x80(%rcx), %rbx + movq 0x88(%rcx), %rbp + movq %r8, %rax + movq %r9, %rdx + orq %r10, %rax + orq %r11, %rdx + orq %rbx, %rax + orq %rbp, %rdx + orq %rdx, %rax + negq %rax + sbbq %rax, %rax + movq 0x150(%rsp), %rsi + movq 0x60(%rsi), %r12 + movq 0x68(%rsi), %r13 + movq 0x70(%rsi), %r14 + movq 0x78(%rsi), %r15 + movq 0x80(%rsi), %rdx + movq 0x88(%rsi), %rcx + cmoveq %r12, %r8 + cmoveq %r13, %r9 + cmoveq %r14, %r10 + cmoveq %r15, %r11 + cmoveq %rdx, %rbx + cmoveq %rcx, %rbp + orq %r13, %r12 + orq %r15, %r14 + orq %rcx, %rdx + orq %r14, %r12 + orq %r12, %rdx + negq %rdx + sbbq %rdx, %rdx + cmpq %rdx, %rax + cmoveq 0xf0(%rsp), %r8 + cmoveq 0xf8(%rsp), %r9 + cmoveq 0x100(%rsp), %r10 + cmoveq 0x108(%rsp), %r11 + cmoveq 0x110(%rsp), %rbx + cmoveq 0x118(%rsp), %rbp + movq %r8, 0xf0(%rsp) + movq %r9, 0xf8(%rsp) + movq %r10, 0x100(%rsp) + movq %r11, 0x108(%rsp) + movq %rbx, 0x110(%rsp) + movq %rbp, 0x118(%rsp) + movq 0x158(%rsp), %rcx + movq 0x150(%rsp), %rsi + movq (%rsp), %r8 + cmovbq (%rsi), %r8 + cmova (%rcx), %r8 + movq 0x8(%rsp), %r9 + cmovbq 0x8(%rsi), %r9 + cmova 0x8(%rcx), %r9 + movq 0x10(%rsp), %r10 + cmovbq 0x10(%rsi), %r10 + cmova 0x10(%rcx), %r10 + movq 0x18(%rsp), %r11 + cmovbq 0x18(%rsi), %r11 + cmova 0x18(%rcx), %r11 + movq 0x20(%rsp), %rbx + cmovbq 0x20(%rsi), %rbx + cmova 0x20(%rcx), %rbx + movq 0x28(%rsp), %rbp + cmovbq 0x28(%rsi), %rbp + cmova 0x28(%rcx), %rbp + movq 0xc0(%rsp), %r12 + cmovbq 0x30(%rsi), %r12 + cmova 0x30(%rcx), %r12 + movq 0xc8(%rsp), %r13 + cmovbq 0x38(%rsi), %r13 + cmova 0x38(%rcx), %r13 + movq 0xd0(%rsp), %r14 + cmovbq 0x40(%rsi), %r14 + cmova 0x40(%rcx), %r14 + movq 0xd8(%rsp), %r15 + cmovbq 0x48(%rsi), %r15 + cmova 0x48(%rcx), %r15 + movq 0xe0(%rsp), %rdx + cmovbq 0x50(%rsi), %rdx + cmova 0x50(%rcx), %rdx + movq 0xe8(%rsp), %rax + cmovbq 0x58(%rsi), %rax + cmova 0x58(%rcx), %rax + movq %r8, (%rdi) + movq %r9, 0x8(%rdi) + movq %r10, 0x10(%rdi) + movq %r11, 0x18(%rdi) + movq %rbx, 0x20(%rdi) + movq %rbp, 0x28(%rdi) + movq 0xf0(%rsp), %r8 + movq 0xf8(%rsp), %r9 + movq 0x100(%rsp), %r10 + movq 0x108(%rsp), %r11 + movq 0x110(%rsp), %rbx + movq 0x118(%rsp), %rbp + movq %r12, 0x30(%rdi) + movq %r13, 0x38(%rdi) + movq %r14, 0x40(%rdi) + movq %r15, 0x48(%rdi) + movq %rdx, 0x50(%rdi) + movq %rax, 0x58(%rdi) + movq %r8, 0x60(%rdi) + movq %r9, 0x68(%rdi) + movq %r10, 0x70(%rdi) + movq %r11, 0x78(%rdi) + movq %rbx, 0x80(%rdi) + movq %rbp, 0x88(%rdi) + CFI_INC_RSP(352) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp384_montjscalarmul_p384_montjadd) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp384_montjscalarmul_p384_montjdouble) + +Lp384_montjscalarmul_p384_montjdouble: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(344) + movq %rdi, 0x150(%rsp) + movq 0x60(%rsi), %rdx + mulxq 0x68(%rsi), %r9, %r10 + mulxq 0x78(%rsi), %r11, %r12 + mulxq 0x88(%rsi), %r13, %r14 + movq 0x78(%rsi), %rdx + mulxq 0x80(%rsi), %r15, %rcx + xorl %ebp, %ebp + movq 0x70(%rsi), %rdx + mulxq 0x60(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x68(%rsi), %rdx + mulxq 0x78(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x80(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x88(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %rcx + adcq %rbp, %rcx + xorl %ebp, %ebp + movq 0x80(%rsi), %rdx + mulxq 0x60(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq 0x70(%rsi), %rdx + mulxq 0x78(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x80(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x88(%rsi), %rax, %rdx + adcxq %rax, %r15 + adoxq %rdx, %rcx + movq 0x88(%rsi), %rdx + mulxq 0x80(%rsi), %rbx, %rbp + mulxq 0x78(%rsi), %rax, %rdx + adcxq %rax, %rcx + adoxq %rdx, %rbx + movl $0x0, %eax + adcxq %rax, %rbx + adoxq %rax, %rbp + adcq %rax, %rbp + xorq %rax, %rax + movq 0x60(%rsi), %rdx + mulxq 0x60(%rsi), %r8, %rax + adcxq %r9, %r9 + adoxq %rax, %r9 + movq 0x68(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x70(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x78(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %r15, %r15 + adoxq %rdx, %r15 + movq 0x80(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %rcx, %rcx + adoxq %rax, %rcx + adcxq %rbx, %rbx + adoxq %rdx, %rbx + movq 0x88(%rsi), %rdx + mulxq %rdx, %rax, %rdi + adcxq %rbp, %rbp + adoxq %rax, %rbp + movl $0x0, %eax + adcxq %rax, %rdi + adoxq %rax, %rdi + movq %rbx, (%rsp) + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r8, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r8 + addq %rbx, %rax + adcq %rdx, %r8 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r9 + sbbq %r8, %r10 + sbbq %rbx, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rdx, %r8 + sbbq $0x0, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r9, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r9 + addq %rbx, %rax + adcq %rdx, %r9 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r10 + sbbq %r9, %r11 + sbbq %rbx, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rdx, %r9 + sbbq $0x0, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r10, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r10 + addq %rbx, %rax + adcq %rdx, %r10 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r11 + sbbq %r10, %r12 + sbbq %rbx, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rdx, %r10 + sbbq $0x0, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r11, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r11 + addq %rbx, %rax + adcq %rdx, %r11 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r12 + sbbq %r11, %r13 + sbbq %rbx, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rdx, %r11 + sbbq $0x0, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r12, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r12 + addq %rbx, %rax + adcq %rdx, %r12 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r13 + sbbq %r12, %r8 + sbbq %rbx, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rdx, %r12 + sbbq $0x0, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r13, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r13 + addq %rbx, %rax + adcq %rdx, %r13 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r8 + sbbq %r13, %r9 + sbbq %rbx, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rdx, %r13 + sbbq $0x0, %r13 + movq (%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rdi + movl $0x0, %r8d + adcq %r8, %r8 + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %r9d + adcq %r15, %r9 + movl $0x1, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rdi, %r13 + adcq $0x0, %r8 + cmovneq %rax, %r14 + cmovneq %r9, %r15 + cmovneq %r10, %rcx + cmovneq %r11, %rbx + cmovneq %r12, %rbp + cmovneq %r13, %rdi + movq %r14, (%rsp) + movq %r15, 0x8(%rsp) + movq %rcx, 0x10(%rsp) + movq %rbx, 0x18(%rsp) + movq %rbp, 0x20(%rsp) + movq %rdi, 0x28(%rsp) + movq 0x30(%rsi), %rdx + mulxq 0x38(%rsi), %r9, %r10 + mulxq 0x48(%rsi), %r11, %r12 + mulxq 0x58(%rsi), %r13, %r14 + movq 0x48(%rsi), %rdx + mulxq 0x50(%rsi), %r15, %rcx + xorl %ebp, %ebp + movq 0x40(%rsi), %rdx + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x38(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x38(%rsi), %rdx + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x50(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x58(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %rcx + adcq %rbp, %rcx + xorl %ebp, %ebp + movq 0x50(%rsi), %rdx + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq 0x40(%rsi), %rdx + mulxq 0x48(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x50(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x58(%rsi), %rax, %rdx + adcxq %rax, %r15 + adoxq %rdx, %rcx + movq 0x58(%rsi), %rdx + mulxq 0x50(%rsi), %rbx, %rbp + mulxq 0x48(%rsi), %rax, %rdx + adcxq %rax, %rcx + adoxq %rdx, %rbx + movl $0x0, %eax + adcxq %rax, %rbx + adoxq %rax, %rbp + adcq %rax, %rbp + xorq %rax, %rax + movq 0x30(%rsi), %rdx + mulxq 0x30(%rsi), %r8, %rax + adcxq %r9, %r9 + adoxq %rax, %r9 + movq 0x38(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x40(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x48(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %r15, %r15 + adoxq %rdx, %r15 + movq 0x50(%rsi), %rdx + mulxq %rdx, %rax, %rdx + adcxq %rcx, %rcx + adoxq %rax, %rcx + adcxq %rbx, %rbx + adoxq %rdx, %rbx + movq 0x58(%rsi), %rdx + mulxq %rdx, %rax, %rdi + adcxq %rbp, %rbp + adoxq %rax, %rbp + movl $0x0, %eax + adcxq %rax, %rdi + adoxq %rax, %rdi + movq %rbx, 0x30(%rsp) + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r8, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r8 + addq %rbx, %rax + adcq %rdx, %r8 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r9 + sbbq %r8, %r10 + sbbq %rbx, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rdx, %r8 + sbbq $0x0, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r9, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r9 + addq %rbx, %rax + adcq %rdx, %r9 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r10 + sbbq %r9, %r11 + sbbq %rbx, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rdx, %r9 + sbbq $0x0, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r10, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r10 + addq %rbx, %rax + adcq %rdx, %r10 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r11 + sbbq %r10, %r12 + sbbq %rbx, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rdx, %r10 + sbbq $0x0, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r11, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r11 + addq %rbx, %rax + adcq %rdx, %r11 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r12 + sbbq %r11, %r13 + sbbq %rbx, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rdx, %r11 + sbbq $0x0, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r12, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r12 + addq %rbx, %rax + adcq %rdx, %r12 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r13 + sbbq %r12, %r8 + sbbq %rbx, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rdx, %r12 + sbbq $0x0, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r13, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r13 + addq %rbx, %rax + adcq %rdx, %r13 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r8 + sbbq %r13, %r9 + sbbq %rbx, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rdx, %r13 + sbbq $0x0, %r13 + movq 0x30(%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rdi + movl $0x0, %r8d + adcq %r8, %r8 + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %r9d + adcq %r15, %r9 + movl $0x1, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rdi, %r13 + adcq $0x0, %r8 + cmovneq %rax, %r14 + cmovneq %r9, %r15 + cmovneq %r10, %rcx + cmovneq %r11, %rbx + cmovneq %r12, %rbp + cmovneq %r13, %rdi + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq %rcx, 0x40(%rsp) + movq %rbx, 0x48(%rsp) + movq %rbp, 0x50(%rsp) + movq %rdi, 0x58(%rsp) + movq (%rsi), %rax + addq (%rsp), %rax + movq 0x8(%rsi), %rcx + adcq 0x8(%rsp), %rcx + movq 0x10(%rsi), %r8 + adcq 0x10(%rsp), %r8 + movq 0x18(%rsi), %r9 + adcq 0x18(%rsp), %r9 + movq 0x20(%rsi), %r10 + adcq 0x20(%rsp), %r10 + movq 0x28(%rsi), %r11 + adcq 0x28(%rsp), %r11 + sbbq %rdx, %rdx + movl $0x1, %ebx + andq %rdx, %rbx + movl $0xffffffff, %ebp + andq %rbp, %rdx + xorq %rbp, %rbp + subq %rdx, %rbp + addq %rbp, %rax + movq %rax, 0xf0(%rsp) + adcq %rdx, %rcx + movq %rcx, 0xf8(%rsp) + adcq %rbx, %r8 + movq %r8, 0x100(%rsp) + adcq $0x0, %r9 + movq %r9, 0x108(%rsp) + adcq $0x0, %r10 + movq %r10, 0x110(%rsp) + adcq $0x0, %r11 + movq %r11, 0x118(%rsp) + movq (%rsi), %rax + subq (%rsp), %rax + movq 0x8(%rsi), %rdx + sbbq 0x8(%rsp), %rdx + movq 0x10(%rsi), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x18(%rsi), %r9 + sbbq 0x18(%rsp), %r9 + movq 0x20(%rsi), %r10 + sbbq 0x20(%rsp), %r10 + movq 0x28(%rsi), %r11 + sbbq 0x28(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %ebx + andq %rbx, %rcx + xorq %rbx, %rbx + subq %rcx, %rbx + subq %rbx, %rax + movq %rax, 0xc0(%rsp) + sbbq %rcx, %rdx + movq %rdx, 0xc8(%rsp) + sbbq %rax, %rax + andq %rbx, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0xd0(%rsp) + sbbq $0x0, %r9 + movq %r9, 0xd8(%rsp) + sbbq $0x0, %r10 + movq %r10, 0xe0(%rsp) + sbbq $0x0, %r11 + movq %r11, 0xe8(%rsp) + movq 0xc0(%rsp), %rdx + xorl %r15d, %r15d + mulxq 0xf0(%rsp), %r8, %r9 + mulxq 0xf8(%rsp), %rbx, %r10 + addq %rbx, %r9 + mulxq 0x100(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x108(%rsp), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0x110(%rsp), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0x118(%rsp), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r8, %rbx + adcq %r8, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rbx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rdx + addq %rdx, %r14 + adcq $0x0, %r15 + movq 0xc8(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r8, %r15 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r9, %rbx + adcq %r9, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rbx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rdx + addq %rdx, %r15 + adcq $0x0, %r8 + movq 0xd0(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adoxq %r9, %r8 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r10, %rbx + adcq %r10, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rbx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rdx + addq %rdx, %r8 + adcq $0x0, %r9 + movq 0xd8(%rsp), %rdx + xorl %r10d, %r10d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + adoxq %r10, %r9 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r11, %rbx + adcq %r11, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rbx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rdx + addq %rdx, %r9 + adcq $0x0, %r10 + movq 0xe0(%rsp), %rdx + xorl %r11d, %r11d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + adoxq %r11, %r10 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r12, %rbx + adcq %r12, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rbx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rdx + addq %rdx, %r10 + adcq $0x0, %r11 + movq 0xe8(%rsp), %rdx + xorl %r12d, %r12d + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + adoxq %r12, %r11 + mulxq 0x118(%rsp), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r13, %rbx + adcq %r13, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rbx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rdx + addq %rdx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x60(%rsp) + movq %r15, 0x68(%rsp) + movq %r8, 0x70(%rsp) + movq %r9, 0x78(%rsp) + movq %r10, 0x80(%rsp) + movq %r11, 0x88(%rsp) + movq 0x30(%rsi), %rax + addq 0x60(%rsi), %rax + movq 0x38(%rsi), %rcx + adcq 0x68(%rsi), %rcx + movq 0x40(%rsi), %r8 + adcq 0x70(%rsi), %r8 + movq 0x48(%rsi), %r9 + adcq 0x78(%rsi), %r9 + movq 0x50(%rsi), %r10 + adcq 0x80(%rsi), %r10 + movq 0x58(%rsi), %r11 + adcq 0x88(%rsi), %r11 + movl $0x0, %edx + adcq %rdx, %rdx + movabsq $0xffffffff00000001, %rbp + addq %rbp, %rax + movl $0xffffffff, %ebp + adcq %rbp, %rcx + adcq $0x1, %r8 + adcq $0x0, %r9 + adcq $0x0, %r10 + adcq $0x0, %r11 + adcq $0xffffffffffffffff, %rdx + movl $0x1, %ebx + andq %rdx, %rbx + andq %rbp, %rdx + xorq %rbp, %rbp + subq %rdx, %rbp + subq %rbp, %rax + movq %rax, 0xf0(%rsp) + sbbq %rdx, %rcx + movq %rcx, 0xf8(%rsp) + sbbq %rbx, %r8 + movq %r8, 0x100(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x108(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x110(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x118(%rsp) + movq 0x60(%rsp), %rdx + mulxq 0x68(%rsp), %r9, %r10 + mulxq 0x78(%rsp), %r11, %r12 + mulxq 0x88(%rsp), %r13, %r14 + movq 0x78(%rsp), %rdx + mulxq 0x80(%rsp), %r15, %rcx + xorl %ebp, %ebp + movq 0x70(%rsp), %rdx + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x68(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x68(%rsp), %rdx + mulxq 0x78(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x80(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x88(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %rcx + adcq %rbp, %rcx + xorl %ebp, %ebp + movq 0x80(%rsp), %rdx + mulxq 0x60(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq 0x70(%rsp), %rdx + mulxq 0x78(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x80(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x88(%rsp), %rax, %rdx + adcxq %rax, %r15 + adoxq %rdx, %rcx + movq 0x88(%rsp), %rdx + mulxq 0x80(%rsp), %rbx, %rbp + mulxq 0x78(%rsp), %rax, %rdx + adcxq %rax, %rcx + adoxq %rdx, %rbx + movl $0x0, %eax + adcxq %rax, %rbx + adoxq %rax, %rbp + adcq %rax, %rbp + xorq %rax, %rax + movq 0x60(%rsp), %rdx + mulxq 0x60(%rsp), %r8, %rax + adcxq %r9, %r9 + adoxq %rax, %r9 + movq 0x68(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x70(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x78(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %r15, %r15 + adoxq %rdx, %r15 + movq 0x80(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %rcx, %rcx + adoxq %rax, %rcx + adcxq %rbx, %rbx + adoxq %rdx, %rbx + movq 0x88(%rsp), %rdx + mulxq %rdx, %rax, %rdi + adcxq %rbp, %rbp + adoxq %rax, %rbp + movl $0x0, %eax + adcxq %rax, %rdi + adoxq %rax, %rdi + movq %rbx, 0x120(%rsp) + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r8, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r8 + addq %rbx, %rax + adcq %rdx, %r8 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r9 + sbbq %r8, %r10 + sbbq %rbx, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rdx, %r8 + sbbq $0x0, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r9, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r9 + addq %rbx, %rax + adcq %rdx, %r9 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r10 + sbbq %r9, %r11 + sbbq %rbx, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rdx, %r9 + sbbq $0x0, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r10, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r10 + addq %rbx, %rax + adcq %rdx, %r10 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r11 + sbbq %r10, %r12 + sbbq %rbx, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rdx, %r10 + sbbq $0x0, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r11, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r11 + addq %rbx, %rax + adcq %rdx, %r11 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r12 + sbbq %r11, %r13 + sbbq %rbx, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rdx, %r11 + sbbq $0x0, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r12, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r12 + addq %rbx, %rax + adcq %rdx, %r12 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r13 + sbbq %r12, %r8 + sbbq %rbx, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rdx, %r12 + sbbq $0x0, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r13, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r13 + addq %rbx, %rax + adcq %rdx, %r13 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r8 + sbbq %r13, %r9 + sbbq %rbx, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rdx, %r13 + sbbq $0x0, %r13 + movq 0x120(%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rdi + movl $0x0, %r8d + adcq %r8, %r8 + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %r9d + adcq %r15, %r9 + movl $0x1, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rdi, %r13 + adcq $0x0, %r8 + cmovneq %rax, %r14 + cmovneq %r9, %r15 + cmovneq %r10, %rcx + cmovneq %r11, %rbx + cmovneq %r12, %rbp + cmovneq %r13, %rdi + movq %r14, 0x120(%rsp) + movq %r15, 0x128(%rsp) + movq %rcx, 0x130(%rsp) + movq %rbx, 0x138(%rsp) + movq %rbp, 0x140(%rsp) + movq %rdi, 0x148(%rsp) + movq 0x30(%rsp), %rdx + xorl %r15d, %r15d + mulxq (%rsi), %r8, %r9 + mulxq 0x8(%rsi), %rbx, %r10 + addq %rbx, %r9 + mulxq 0x10(%rsi), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x18(%rsi), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0x20(%rsi), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0x28(%rsi), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r8, %rbx + adcq %r8, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rbx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rdx + addq %rdx, %r14 + adcq $0x0, %r15 + movq 0x38(%rsp), %rdx + xorl %r8d, %r8d + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r8, %r15 + mulxq 0x28(%rsi), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r9, %rbx + adcq %r9, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rbx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rdx + addq %rdx, %r15 + adcq $0x0, %r8 + movq 0x40(%rsp), %rdx + xorl %r9d, %r9d + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adoxq %r9, %r8 + mulxq 0x28(%rsi), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r10, %rbx + adcq %r10, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rbx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rdx + addq %rdx, %r8 + adcq $0x0, %r9 + movq 0x48(%rsp), %rdx + xorl %r10d, %r10d + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + adoxq %r10, %r9 + mulxq 0x28(%rsi), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r11, %rbx + adcq %r11, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rbx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rdx + addq %rdx, %r9 + adcq $0x0, %r10 + movq 0x50(%rsp), %rdx + xorl %r11d, %r11d + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + adoxq %r11, %r10 + mulxq 0x28(%rsi), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r12, %rbx + adcq %r12, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rbx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rdx + addq %rdx, %r10 + adcq $0x0, %r11 + movq 0x58(%rsp), %rdx + xorl %r12d, %r12d + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + adoxq %r12, %r11 + mulxq 0x28(%rsi), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r13, %rbx + adcq %r13, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rbx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rdx + addq %rdx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq %r8, 0xa0(%rsp) + movq %r9, 0xa8(%rsp) + movq %r10, 0xb0(%rsp) + movq %r11, 0xb8(%rsp) + movq 0xf0(%rsp), %rdx + mulxq 0xf8(%rsp), %r9, %r10 + mulxq 0x108(%rsp), %r11, %r12 + mulxq 0x118(%rsp), %r13, %r14 + movq 0x108(%rsp), %rdx + mulxq 0x110(%rsp), %r15, %rcx + xorl %ebp, %ebp + movq 0x100(%rsp), %rdx + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0xf8(%rsp), %rdx + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x118(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %rcx + adcq %rbp, %rcx + xorl %ebp, %ebp + movq 0x110(%rsp), %rdx + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq 0x100(%rsp), %rdx + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x118(%rsp), %rax, %rdx + adcxq %rax, %r15 + adoxq %rdx, %rcx + movq 0x118(%rsp), %rdx + mulxq 0x110(%rsp), %rbx, %rbp + mulxq 0x108(%rsp), %rax, %rdx + adcxq %rax, %rcx + adoxq %rdx, %rbx + movl $0x0, %eax + adcxq %rax, %rbx + adoxq %rax, %rbp + adcq %rax, %rbp + xorq %rax, %rax + movq 0xf0(%rsp), %rdx + mulxq 0xf0(%rsp), %r8, %rax + adcxq %r9, %r9 + adoxq %rax, %r9 + movq 0xf8(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x100(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x108(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %r15, %r15 + adoxq %rdx, %r15 + movq 0x110(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %rcx, %rcx + adoxq %rax, %rcx + adcxq %rbx, %rbx + adoxq %rdx, %rbx + movq 0x118(%rsp), %rdx + mulxq %rdx, %rax, %rdi + adcxq %rbp, %rbp + adoxq %rax, %rbp + movl $0x0, %eax + adcxq %rax, %rdi + adoxq %rax, %rdi + movq %rbx, 0xc0(%rsp) + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r8, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r8 + addq %rbx, %rax + adcq %rdx, %r8 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r9 + sbbq %r8, %r10 + sbbq %rbx, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rdx, %r8 + sbbq $0x0, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r9, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r9 + addq %rbx, %rax + adcq %rdx, %r9 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r10 + sbbq %r9, %r11 + sbbq %rbx, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rdx, %r9 + sbbq $0x0, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r10, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r10 + addq %rbx, %rax + adcq %rdx, %r10 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r11 + sbbq %r10, %r12 + sbbq %rbx, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rdx, %r10 + sbbq $0x0, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r11, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r11 + addq %rbx, %rax + adcq %rdx, %r11 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r12 + sbbq %r11, %r13 + sbbq %rbx, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rdx, %r11 + sbbq $0x0, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r12, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r12 + addq %rbx, %rax + adcq %rdx, %r12 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r13 + sbbq %r12, %r8 + sbbq %rbx, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rdx, %r12 + sbbq $0x0, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r13, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r13 + addq %rbx, %rax + adcq %rdx, %r13 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r8 + sbbq %r13, %r9 + sbbq %rbx, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rdx, %r13 + sbbq $0x0, %r13 + movq 0xc0(%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rdi + movl $0x0, %r8d + adcq %r8, %r8 + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %r9d + adcq %r15, %r9 + movl $0x1, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rdi, %r13 + adcq $0x0, %r8 + cmovneq %rax, %r14 + cmovneq %r9, %r15 + cmovneq %r10, %rcx + cmovneq %r11, %rbx + cmovneq %r12, %rbp + cmovneq %r13, %rdi + movq %r14, 0xc0(%rsp) + movq %r15, 0xc8(%rsp) + movq %rcx, 0xd0(%rsp) + movq %rbx, 0xd8(%rsp) + movq %rbp, 0xe0(%rsp) + movq %rdi, 0xe8(%rsp) + movabsq $0xffffffff, %r8 + subq 0x120(%rsp), %r8 + movabsq $0xffffffff00000000, %r9 + sbbq 0x128(%rsp), %r9 + movq $0xfffffffffffffffe, %r10 + sbbq 0x130(%rsp), %r10 + movq $0xffffffffffffffff, %r11 + sbbq 0x138(%rsp), %r11 + movq $0xffffffffffffffff, %r12 + sbbq 0x140(%rsp), %r12 + movq $0xffffffffffffffff, %r13 + sbbq 0x148(%rsp), %r13 + movq $0x9, %rdx + mulxq %r8, %r8, %rax + mulxq %r9, %r9, %rcx + addq %rax, %r9 + mulxq %r10, %r10, %rax + adcq %rcx, %r10 + mulxq %r11, %r11, %rcx + adcq %rax, %r11 + mulxq %r12, %r12, %rax + adcq %rcx, %r12 + mulxq %r13, %r13, %r14 + adcq %rax, %r13 + adcq $0x1, %r14 + xorl %ecx, %ecx + movq $0xc, %rdx + mulxq 0x90(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x98(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0xa0(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0xa8(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0xb0(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0xb8(%rsp), %rax, %rdx + adcxq %rax, %r13 + adoxq %r14, %rdx + adcxq %rcx, %rdx + xorq %rcx, %rcx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rax, %rcx + adcxq %rax, %r8 + adoxq %rcx, %r9 + movl $0xffffffff, %eax + mulxq %rax, %rax, %rcx + adcxq %rax, %r9 + adoxq %rcx, %r10 + adcxq %rdx, %r10 + movl $0x0, %eax + movl $0x0, %ecx + adoxq %rax, %rax + adcq %rax, %r11 + adcq %rcx, %r12 + adcq %rcx, %r13 + adcq %rcx, %rcx + subq $0x1, %rcx + movl $0xffffffff, %edx + xorq %rax, %rax + andq %rcx, %rdx + subq %rdx, %rax + andq $0x1, %rcx + subq %rax, %r8 + movq %r8, 0x120(%rsp) + sbbq %rdx, %r9 + movq %r9, 0x128(%rsp) + sbbq %rcx, %r10 + movq %r10, 0x130(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x138(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x140(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x148(%rsp) + movq 0xc0(%rsp), %rax + subq (%rsp), %rax + movq 0xc8(%rsp), %rdx + sbbq 0x8(%rsp), %rdx + movq 0xd0(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0xd8(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movq 0xe0(%rsp), %r10 + sbbq 0x20(%rsp), %r10 + movq 0xe8(%rsp), %r11 + sbbq 0x28(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %ebx + andq %rbx, %rcx + xorq %rbx, %rbx + subq %rcx, %rbx + subq %rbx, %rax + movq %rax, 0xf0(%rsp) + sbbq %rcx, %rdx + movq %rdx, 0xf8(%rsp) + sbbq %rax, %rax + andq %rbx, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0x100(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x108(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x110(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x118(%rsp) + movq 0x30(%rsp), %rdx + mulxq 0x38(%rsp), %r9, %r10 + mulxq 0x48(%rsp), %r11, %r12 + mulxq 0x58(%rsp), %r13, %r14 + movq 0x48(%rsp), %rdx + mulxq 0x50(%rsp), %r15, %rcx + xorl %ebp, %ebp + movq 0x40(%rsp), %rdx + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x38(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq 0x38(%rsp), %rdx + mulxq 0x48(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x50(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x58(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adcxq %rbp, %r15 + adoxq %rbp, %rcx + adcq %rbp, %rcx + xorl %ebp, %ebp + movq 0x50(%rsp), %rdx + mulxq 0x30(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq 0x40(%rsp), %rdx + mulxq 0x48(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x50(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x58(%rsp), %rax, %rdx + adcxq %rax, %r15 + adoxq %rdx, %rcx + movq 0x58(%rsp), %rdx + mulxq 0x50(%rsp), %rbx, %rbp + mulxq 0x48(%rsp), %rax, %rdx + adcxq %rax, %rcx + adoxq %rdx, %rbx + movl $0x0, %eax + adcxq %rax, %rbx + adoxq %rax, %rbp + adcq %rax, %rbp + xorq %rax, %rax + movq 0x30(%rsp), %rdx + mulxq 0x30(%rsp), %r8, %rax + adcxq %r9, %r9 + adoxq %rax, %r9 + movq 0x38(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r10, %r10 + adoxq %rax, %r10 + adcxq %r11, %r11 + adoxq %rdx, %r11 + movq 0x40(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r12, %r12 + adoxq %rax, %r12 + adcxq %r13, %r13 + adoxq %rdx, %r13 + movq 0x48(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %r14, %r14 + adoxq %rax, %r14 + adcxq %r15, %r15 + adoxq %rdx, %r15 + movq 0x50(%rsp), %rdx + mulxq %rdx, %rax, %rdx + adcxq %rcx, %rcx + adoxq %rax, %rcx + adcxq %rbx, %rbx + adoxq %rdx, %rbx + movq 0x58(%rsp), %rdx + mulxq %rdx, %rax, %rdi + adcxq %rbp, %rbp + adoxq %rax, %rbp + movl $0x0, %eax + adcxq %rax, %rdi + adoxq %rax, %rdi + movq %rbx, 0xc0(%rsp) + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r8, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r8 + addq %rbx, %rax + adcq %rdx, %r8 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r9 + sbbq %r8, %r10 + sbbq %rbx, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rdx, %r8 + sbbq $0x0, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r9, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r9 + addq %rbx, %rax + adcq %rdx, %r9 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r10 + sbbq %r9, %r11 + sbbq %rbx, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rdx, %r9 + sbbq $0x0, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r10, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r10 + addq %rbx, %rax + adcq %rdx, %r10 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r11 + sbbq %r10, %r12 + sbbq %rbx, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rdx, %r10 + sbbq $0x0, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r11, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r11 + addq %rbx, %rax + adcq %rdx, %r11 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r12 + sbbq %r11, %r13 + sbbq %rbx, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rdx, %r11 + sbbq $0x0, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r12, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r12 + addq %rbx, %rax + adcq %rdx, %r12 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r13 + sbbq %r12, %r8 + sbbq %rbx, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rdx, %r12 + sbbq $0x0, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %r13, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %rbx, %r13 + addq %rbx, %rax + adcq %rdx, %r13 + movl $0x0, %ebx + adcq %rbx, %rbx + subq %rax, %r8 + sbbq %r13, %r9 + sbbq %rbx, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rdx, %r13 + sbbq $0x0, %r13 + movq 0xc0(%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rdi + movl $0x0, %r8d + adcq %r8, %r8 + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %r9d + adcq %r15, %r9 + movl $0x1, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rdi, %r13 + adcq $0x0, %r8 + cmovneq %rax, %r14 + cmovneq %r9, %r15 + cmovneq %r10, %rcx + cmovneq %r11, %rbx + cmovneq %r12, %rbp + cmovneq %r13, %rdi + movq %r14, 0xc0(%rsp) + movq %r15, 0xc8(%rsp) + movq %rcx, 0xd0(%rsp) + movq %rbx, 0xd8(%rsp) + movq %rbp, 0xe0(%rsp) + movq %rdi, 0xe8(%rsp) + movq 0x150(%rsp), %rdi + movq 0xf0(%rsp), %rax + subq 0x30(%rsp), %rax + movq 0xf8(%rsp), %rdx + sbbq 0x38(%rsp), %rdx + movq 0x100(%rsp), %r8 + sbbq 0x40(%rsp), %r8 + movq 0x108(%rsp), %r9 + sbbq 0x48(%rsp), %r9 + movq 0x110(%rsp), %r10 + sbbq 0x50(%rsp), %r10 + movq 0x118(%rsp), %r11 + sbbq 0x58(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %ebx + andq %rbx, %rcx + xorq %rbx, %rbx + subq %rcx, %rbx + subq %rbx, %rax + movq %rax, 0x60(%rdi) + sbbq %rcx, %rdx + movq %rdx, 0x68(%rdi) + sbbq %rax, %rax + andq %rbx, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0x70(%rdi) + sbbq $0x0, %r9 + movq %r9, 0x78(%rdi) + sbbq $0x0, %r10 + movq %r10, 0x80(%rdi) + sbbq $0x0, %r11 + movq %r11, 0x88(%rdi) + movq 0x60(%rsp), %rdx + xorl %r15d, %r15d + mulxq 0x120(%rsp), %r8, %r9 + mulxq 0x128(%rsp), %rbx, %r10 + addq %rbx, %r9 + mulxq 0x130(%rsp), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x138(%rsp), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0x140(%rsp), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0x148(%rsp), %rbx, %r14 + adcq %rbx, %r13 + adcq %r15, %r14 + movq %r8, %rdx + shlq $0x20, %rdx + addq %r8, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r8, %rbx + adcq %r8, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rbx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rdx + addq %rdx, %r14 + adcq $0x0, %r15 + movq 0x68(%rsp), %rdx + xorl %r8d, %r8d + mulxq 0x120(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x128(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x130(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x138(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x140(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + adoxq %r8, %r15 + mulxq 0x148(%rsp), %rax, %rbx + adcq %rax, %r14 + adcq %rbx, %r15 + adcq %r8, %r8 + movq %r9, %rdx + shlq $0x20, %rdx + addq %r9, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r9, %rbx + adcq %r9, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rbx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rdx + addq %rdx, %r15 + adcq $0x0, %r8 + movq 0x70(%rsp), %rdx + xorl %r9d, %r9d + mulxq 0x120(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x128(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x130(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x138(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x140(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + adoxq %r9, %r8 + mulxq 0x148(%rsp), %rax, %rbx + adcq %rax, %r15 + adcq %rbx, %r8 + adcq %r9, %r9 + movq %r10, %rdx + shlq $0x20, %rdx + addq %r10, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r10, %rbx + adcq %r10, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rbx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rdx + addq %rdx, %r8 + adcq $0x0, %r9 + movq 0x78(%rsp), %rdx + xorl %r10d, %r10d + mulxq 0x120(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x128(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x130(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x138(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x140(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + adoxq %r10, %r9 + mulxq 0x148(%rsp), %rax, %rbx + adcq %rax, %r8 + adcq %rbx, %r9 + adcq %r10, %r10 + movq %r11, %rdx + shlq $0x20, %rdx + addq %r11, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r11, %rbx + adcq %r11, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rbx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rdx + addq %rdx, %r9 + adcq $0x0, %r10 + movq 0x80(%rsp), %rdx + xorl %r11d, %r11d + mulxq 0x120(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x128(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x130(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x138(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x140(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + adoxq %r11, %r10 + mulxq 0x148(%rsp), %rax, %rbx + adcq %rax, %r9 + adcq %rbx, %r10 + adcq %r11, %r11 + movq %r12, %rdx + shlq $0x20, %rdx + addq %r12, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r12, %rbx + adcq %r12, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rbx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rdx + addq %rdx, %r10 + adcq $0x0, %r11 + movq 0x88(%rsp), %rdx + xorl %r12d, %r12d + mulxq 0x120(%rsp), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x128(%rsp), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x130(%rsp), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x138(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x140(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + adoxq %r12, %r11 + mulxq 0x148(%rsp), %rax, %rbx + adcq %rax, %r10 + adcq %rbx, %r11 + adcq %r12, %r12 + movq %r13, %rdx + shlq $0x20, %rdx + addq %r13, %rdx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rbx, %rax + movl $0xffffffff, %ebx + mulxq %rbx, %r13, %rbx + adcq %r13, %rax + adcq %rdx, %rbx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rbx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rdx + addq %rdx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0xf0(%rsp) + movq %r15, 0xf8(%rsp) + movq %r8, 0x100(%rsp) + movq %r9, 0x108(%rsp) + movq %r10, 0x110(%rsp) + movq %r11, 0x118(%rsp) + movq 0xb8(%rsp), %rdx + movq %rdx, %r13 + shrq $0x3e, %rdx + movq 0xb0(%rsp), %r12 + shldq $0x2, %r12, %r13 + movq 0xa8(%rsp), %r11 + shldq $0x2, %r11, %r12 + movq 0xa0(%rsp), %r10 + shldq $0x2, %r10, %r11 + movq 0x98(%rsp), %r9 + shldq $0x2, %r9, %r10 + movq 0x90(%rsp), %r8 + shldq $0x2, %r8, %r9 + shlq $0x2, %r8 + addq $0x1, %rdx + subq 0x120(%rsp), %r8 + sbbq 0x128(%rsp), %r9 + sbbq 0x130(%rsp), %r10 + sbbq 0x138(%rsp), %r11 + sbbq 0x140(%rsp), %r12 + sbbq 0x148(%rsp), %r13 + sbbq $0x0, %rdx + xorq %rcx, %rcx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rax, %rcx + adcxq %rax, %r8 + adoxq %rcx, %r9 + movl $0xffffffff, %eax + mulxq %rax, %rax, %rcx + adcxq %rax, %r9 + adoxq %rcx, %r10 + adcxq %rdx, %r10 + movl $0x0, %eax + movl $0x0, %ecx + adoxq %rax, %rax + adcq %rax, %r11 + adcq %rcx, %r12 + adcq %rcx, %r13 + adcq %rcx, %rcx + subq $0x1, %rcx + movl $0xffffffff, %edx + xorq %rax, %rax + andq %rcx, %rdx + subq %rdx, %rax + andq $0x1, %rcx + subq %rax, %r8 + movq %r8, (%rdi) + sbbq %rdx, %r9 + movq %r9, 0x8(%rdi) + sbbq %rcx, %r10 + movq %r10, 0x10(%rdi) + sbbq $0x0, %r11 + movq %r11, 0x18(%rdi) + sbbq $0x0, %r12 + movq %r12, 0x20(%rdi) + sbbq $0x0, %r13 + movq %r13, 0x28(%rdi) + movabsq $0xffffffff, %r8 + subq 0xc0(%rsp), %r8 + movabsq $0xffffffff00000000, %r9 + sbbq 0xc8(%rsp), %r9 + movq $0xfffffffffffffffe, %r10 + sbbq 0xd0(%rsp), %r10 + movq $0xffffffffffffffff, %r11 + sbbq 0xd8(%rsp), %r11 + movq $0xffffffffffffffff, %r12 + sbbq 0xe0(%rsp), %r12 + movq $0xffffffffffffffff, %r13 + sbbq 0xe8(%rsp), %r13 + movq %r13, %r14 + shrq $0x3d, %r14 + shldq $0x3, %r12, %r13 + shldq $0x3, %r11, %r12 + shldq $0x3, %r10, %r11 + shldq $0x3, %r9, %r10 + shldq $0x3, %r8, %r9 + shlq $0x3, %r8 + addq $0x1, %r14 + xorl %ecx, %ecx + movq $0x3, %rdx + mulxq 0xf0(%rsp), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0xf8(%rsp), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x100(%rsp), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x108(%rsp), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x110(%rsp), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x118(%rsp), %rax, %rdx + adcxq %rax, %r13 + adoxq %r14, %rdx + adcxq %rcx, %rdx + xorq %rcx, %rcx + movabsq $0xffffffff00000001, %rax + mulxq %rax, %rax, %rcx + adcxq %rax, %r8 + adoxq %rcx, %r9 + movl $0xffffffff, %eax + mulxq %rax, %rax, %rcx + adcxq %rax, %r9 + adoxq %rcx, %r10 + adcxq %rdx, %r10 + movl $0x0, %eax + movl $0x0, %ecx + adoxq %rax, %rax + adcq %rax, %r11 + adcq %rcx, %r12 + adcq %rcx, %r13 + adcq %rcx, %rcx + subq $0x1, %rcx + movl $0xffffffff, %edx + xorq %rax, %rax + andq %rcx, %rdx + subq %rdx, %rax + andq $0x1, %rcx + subq %rax, %r8 + movq %r8, 0x30(%rdi) + sbbq %rdx, %r9 + movq %r9, 0x38(%rdi) + sbbq %rcx, %r10 + movq %r10, 0x40(%rdi) + sbbq $0x0, %r11 + movq %r11, 0x48(%rdi) + sbbq $0x0, %r12 + movq %r12, 0x50(%rdi) + sbbq $0x0, %r13 + movq %r13, 0x58(%rdi) + CFI_INC_RSP(344) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp384_montjscalarmul_p384_montjdouble) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/p384_montjscalarmul_alt.S b/cbits/s2n/x86_att/p384_montjscalarmul_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/p384_montjscalarmul_alt.S @@ -0,0 +1,9440 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Montgomery-Jacobian form scalar multiplication for P-384 +// Input scalar[6], point[18]; output res[18] +// +// extern void p384_montjscalarmul_alt +// (uint64_t res[static 18], +// const uint64_t scalar[static 6], +// const uint64_t point[static 18]); +// +// This function is a variant of its affine point version p384_scalarmul. +// Here, input and output points are assumed to be in Jacobian form with +// their coordinates in the Montgomery domain. Thus, if priming indicates +// Montgomery form, x' = (2^384 * x) mod p_384 etc., each point argument +// is a triple (x',y',z') representing the affine point (x/z^2,y/z^3) when +// z' is nonzero or the point at infinity (group identity) if z' = 0. +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-384, returns a representation of n * P. If the result is the +// point at infinity (either because the input point was or because the +// scalar was a multiple of p_384) then the output is guaranteed to +// represent the point at infinity, i.e. to have its z coordinate zero. +// +// Standard x86-64 ABI: RDI = res, RSI = scalar, RDX = point +// Microsoft x64 ABI: RCX = res, RDX = scalar, R8 = point +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p384_montjscalarmul_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p384_montjscalarmul_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p384_montjscalarmul_alt) + + + .text + .balign 32 + +// Size of individual field elements + +#define NUMSIZE 48 +#define JACSIZE (3*NUMSIZE) + +// Intermediate variables on the stack. +// The table is 16 entries, each of size JACSIZE = 3 * NUMSIZE +// Uppercase syntactic variants make x86_att version simpler to generate. + +#define SCALARB (0*NUMSIZE) +#define scalarb (0*NUMSIZE)(%rsp) +#define ACC (1*NUMSIZE) +#define acc (1*NUMSIZE)(%rsp) +#define TABENT (4*NUMSIZE) +#define tabent (4*NUMSIZE)(%rsp) + +#define TAB (7*NUMSIZE) +#define tab (7*NUMSIZE)(%rsp) + +#define res (55*NUMSIZE)(%rsp) + +#define NSPACE 56*NUMSIZE + +// Avoid using .rep for the sake of the BoringSSL/AWS-LC delocator, +// which doesn't accept repetitions, assembler macros etc. + +#define selectblock_xz(I) \ + cmpq $I, %rdi ; \ + cmovzq TAB+JACSIZE*(I-1)(%rsp), %rax ; \ + cmovzq TAB+JACSIZE*(I-1)+8(%rsp), %rbx ; \ + cmovzq TAB+JACSIZE*(I-1)+16(%rsp), %rcx ; \ + cmovzq TAB+JACSIZE*(I-1)+24(%rsp), %rdx ; \ + cmovzq TAB+JACSIZE*(I-1)+32(%rsp), %r8 ; \ + cmovzq TAB+JACSIZE*(I-1)+40(%rsp), %r9 ; \ + cmovzq TAB+JACSIZE*(I-1)+96(%rsp), %r10 ; \ + cmovzq TAB+JACSIZE*(I-1)+104(%rsp), %r11 ; \ + cmovzq TAB+JACSIZE*(I-1)+112(%rsp), %r12 ; \ + cmovzq TAB+JACSIZE*(I-1)+120(%rsp), %r13 ; \ + cmovzq TAB+JACSIZE*(I-1)+128(%rsp), %r14 ; \ + cmovzq TAB+JACSIZE*(I-1)+136(%rsp), %r15 + +#define selectblock_y(I) \ + cmpq $I, %rdi ; \ + cmovzq TAB+JACSIZE*(I-1)+48(%rsp), %rax ; \ + cmovzq TAB+JACSIZE*(I-1)+56(%rsp), %rbx ; \ + cmovzq TAB+JACSIZE*(I-1)+64(%rsp), %rcx ; \ + cmovzq TAB+JACSIZE*(I-1)+72(%rsp), %rdx ; \ + cmovzq TAB+JACSIZE*(I-1)+80(%rsp), %r8 ; \ + cmovzq TAB+JACSIZE*(I-1)+88(%rsp), %r9 + +S2N_BN_SYMBOL(p384_montjscalarmul_alt): + CFI_START + _CET_ENDBR + +// The Windows version literally calls the standard ABI version. +// This simplifies the proofs since subroutine offsets are fixed. + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + CFI_CALL(Lp384_montjscalarmul_alt_standard) + CFI_POP(%rsi) + CFI_POP(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p384_montjscalarmul_alt) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp384_montjscalarmul_alt_standard) + +Lp384_montjscalarmul_alt_standard: + CFI_START +#endif + +// Real start of the standard ABI code. + + CFI_PUSH(%r15) + CFI_PUSH(%r14) + CFI_PUSH(%r13) + CFI_PUSH(%r12) + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + + CFI_DEC_RSP(NSPACE) + +// Preserve the "res" input argument; others get processed early. + + movq %rdi, res + +// Reduce the input scalar mod n_384, i.e. conditionally subtract n_384. +// Store it to "scalarb". + + movq (%rsi), %r8 + movq $0xecec196accc52973, %rax + subq %rax, %r8 + movq 8(%rsi), %r9 + movq $0x581a0db248b0a77a, %rax + sbbq %rax, %r9 + movq 16(%rsi), %r10 + movq $0xc7634d81f4372ddf, %rax + sbbq %rax, %r10 + movq 24(%rsi), %r11 + movq $0xffffffffffffffff, %rax + sbbq %rax, %r11 + movq 32(%rsi), %r12 + sbbq %rax, %r12 + movq 40(%rsi), %r13 + sbbq %rax, %r13 + + cmovcq (%rsi), %r8 + cmovcq 8(%rsi), %r9 + cmovcq 16(%rsi), %r10 + cmovcq 24(%rsi), %r11 + cmovcq 32(%rsi), %r12 + cmovcq 40(%rsi), %r13 + + movq %r8, SCALARB(%rsp) + movq %r9, SCALARB+8(%rsp) + movq %r10, SCALARB+16(%rsp) + movq %r11, SCALARB+24(%rsp) + movq %r12, SCALARB+32(%rsp) + movq %r13, SCALARB+40(%rsp) + +// Set the tab[0] table entry to the input point = 1 * P + + movq (%rdx), %rax + movq %rax, TAB(%rsp) + movq 8(%rdx), %rax + movq %rax, TAB+8(%rsp) + movq 16(%rdx), %rax + movq %rax, TAB+16(%rsp) + movq 24(%rdx), %rax + movq %rax, TAB+24(%rsp) + movq 32(%rdx), %rax + movq %rax, TAB+32(%rsp) + movq 40(%rdx), %rax + movq %rax, TAB+40(%rsp) + + movq 48(%rdx), %rax + movq %rax, TAB+48(%rsp) + movq 56(%rdx), %rax + movq %rax, TAB+56(%rsp) + movq 64(%rdx), %rax + movq %rax, TAB+64(%rsp) + movq 72(%rdx), %rax + movq %rax, TAB+72(%rsp) + movq 80(%rdx), %rax + movq %rax, TAB+80(%rsp) + movq 88(%rdx), %rax + movq %rax, TAB+88(%rsp) + + movq 96(%rdx), %rax + movq %rax, TAB+96(%rsp) + movq 104(%rdx), %rax + movq %rax, TAB+104(%rsp) + movq 112(%rdx), %rax + movq %rax, TAB+112(%rsp) + movq 120(%rdx), %rax + movq %rax, TAB+120(%rsp) + movq 128(%rdx), %rax + movq %rax, TAB+128(%rsp) + movq 136(%rdx), %rax + movq %rax, TAB+136(%rsp) + +// Compute and record tab[1] = 2 * p, ..., tab[15] = 16 * P + + leaq TAB+JACSIZE*1(%rsp), %rdi + leaq TAB(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjdouble) + + leaq TAB+JACSIZE*2(%rsp), %rdi + leaq TAB+JACSIZE*1(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjadd) + + leaq TAB+JACSIZE*3(%rsp), %rdi + leaq TAB+JACSIZE*1(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjdouble) + + leaq TAB+JACSIZE*4(%rsp), %rdi + leaq TAB+JACSIZE*3(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjadd) + + leaq TAB+JACSIZE*5(%rsp), %rdi + leaq TAB+JACSIZE*2(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjdouble) + + leaq TAB+JACSIZE*6(%rsp), %rdi + leaq TAB+JACSIZE*5(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjadd) + + leaq TAB+JACSIZE*7(%rsp), %rdi + leaq TAB+JACSIZE*3(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjdouble) + + leaq TAB+JACSIZE*8(%rsp), %rdi + leaq TAB+JACSIZE*7(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjadd) + + leaq TAB+JACSIZE*9(%rsp), %rdi + leaq TAB+JACSIZE*4(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjdouble) + + leaq TAB+JACSIZE*10(%rsp), %rdi + leaq TAB+JACSIZE*9(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjadd) + + leaq TAB+JACSIZE*11(%rsp), %rdi + leaq TAB+JACSIZE*5(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjdouble) + + leaq TAB+JACSIZE*12(%rsp), %rdi + leaq TAB+JACSIZE*11(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjadd) + + leaq TAB+JACSIZE*13(%rsp), %rdi + leaq TAB+JACSIZE*6(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjdouble) + + leaq TAB+JACSIZE*14(%rsp), %rdi + leaq TAB+JACSIZE*13(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjadd) + + leaq TAB+JACSIZE*15(%rsp), %rdi + leaq TAB+JACSIZE*7(%rsp), %rsi + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjdouble) + +// Add the recoding constant sum_i(16 * 32^i) to the scalar to allow signed +// digits. The digits of the constant, in lowest-to-highest order, are as +// follows; they are generated dynamically to use fewer large constant loads. +// +// 0x0842108421084210 +// 0x1084210842108421 +// 0x2108421084210842 +// 0x4210842108421084 +// 0x8421084210842108 +// 0x0842108421084210 + + movq $0x1084210842108421, %rax + movq %rax, %rcx + shrq $1, %rax + movq SCALARB(%rsp), %r8 + addq %rax, %r8 + movq SCALARB+8(%rsp), %r9 + adcq %rcx, %r9 + leaq (%rcx,%rcx), %rcx + movq SCALARB+16(%rsp), %r10 + adcq %rcx, %r10 + leaq (%rcx,%rcx), %rcx + movq SCALARB+24(%rsp), %r11 + adcq %rcx, %r11 + leaq (%rcx,%rcx), %rcx + movq SCALARB+32(%rsp), %r12 + adcq %rcx, %r12 + movq SCALARB+40(%rsp), %r13 + adcq %rax, %r13 + sbbq %rdi, %rdi + negq %rdi + +// Record the top bitfield in %rdi then shift the whole scalar left 4 bits +// to align the top of the next bitfield with the MSB (bits 379..383). + + shldq $4, %r13, %rdi + shldq $4, %r12, %r13 + shldq $4, %r11, %r12 + shldq $4, %r10, %r11 + shldq $4, %r9, %r10 + shldq $4, %r8, %r9 + shlq $4, %r8 + + movq %r8, SCALARB(%rsp) + movq %r9, SCALARB+8(%rsp) + movq %r10, SCALARB+16(%rsp) + movq %r11, SCALARB+24(%rsp) + movq %r12, SCALARB+32(%rsp) + movq %r13, SCALARB+40(%rsp) + +// Initialize the accumulator to the corresponding entry using constant-time +// lookup in the table. This top digit, uniquely, is not recoded so there is +// no sign adjustment to make. On the x86 integer side we don't have enough +// registers to hold all the fields; this could be better done with SIMD +// registers anyway. So we do x and z coordinates in one sweep, y in another +// (this is a rehearsal for below where we might need to negate the y). + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + xorl %r13d, %r13d + xorl %r14d, %r14d + xorl %r15d, %r15d + + selectblock_xz(1) + selectblock_xz(2) + selectblock_xz(3) + selectblock_xz(4) + selectblock_xz(5) + selectblock_xz(6) + selectblock_xz(7) + selectblock_xz(8) + selectblock_xz(9) + selectblock_xz(10) + selectblock_xz(11) + selectblock_xz(12) + selectblock_xz(13) + selectblock_xz(14) + selectblock_xz(15) + selectblock_xz(16) + + movq %rax, ACC(%rsp) + movq %rbx, ACC+8(%rsp) + movq %rcx, ACC+16(%rsp) + movq %rdx, ACC+24(%rsp) + movq %r8, ACC+32(%rsp) + movq %r9, ACC+40(%rsp) + movq %r10, ACC+96(%rsp) + movq %r11, ACC+104(%rsp) + movq %r12, ACC+112(%rsp) + movq %r13, ACC+120(%rsp) + movq %r14, ACC+128(%rsp) + movq %r15, ACC+136(%rsp) + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + + selectblock_y(1) + selectblock_y(2) + selectblock_y(3) + selectblock_y(4) + selectblock_y(5) + selectblock_y(6) + selectblock_y(7) + selectblock_y(8) + selectblock_y(9) + selectblock_y(10) + selectblock_y(11) + selectblock_y(12) + selectblock_y(13) + selectblock_y(14) + selectblock_y(15) + selectblock_y(16) + + movq %rax, ACC+48(%rsp) + movq %rbx, ACC+56(%rsp) + movq %rcx, ACC+64(%rsp) + movq %rdx, ACC+72(%rsp) + movq %r8, ACC+80(%rsp) + movq %r9, ACC+88(%rsp) + +// Main loop over size-5 bitfields: double 5 times then add signed digit +// At each stage we shift the scalar left by 5 bits so we can simply pick +// the top 5 bits as the bitfield, saving some fiddle over indexing. + + movl $380, %ebp + +Lp384_montjscalarmul_alt_mainloop: + subq $5, %rbp + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjdouble) + +// Choose the bitfield and adjust it to sign and magnitude + + movq SCALARB(%rsp), %r8 + movq SCALARB+8(%rsp), %r9 + movq SCALARB+16(%rsp), %r10 + movq SCALARB+24(%rsp), %r11 + movq SCALARB+32(%rsp), %r12 + movq SCALARB+40(%rsp), %r13 + + movq %r13, %rdi + shrq $59, %rdi + shldq $5, %r12, %r13 + shldq $5, %r11, %r12 + shldq $5, %r10, %r11 + shldq $5, %r9, %r10 + shldq $5, %r8, %r9 + shlq $5, %r8 + + movq %r8, SCALARB(%rsp) + movq %r9, SCALARB+8(%rsp) + movq %r10, SCALARB+16(%rsp) + movq %r11, SCALARB+24(%rsp) + movq %r12, SCALARB+32(%rsp) + movq %r13, SCALARB+40(%rsp) + + subq $16, %rdi + sbbq %rsi, %rsi // %rsi = sign of digit (-1 = negative) + xorq %rsi, %rdi + subq %rsi, %rdi // %rdi = absolute value of digit + +// Conditionally select the table entry tab[i-1] = i * P in constant time +// Again, this is done in two sweeps, first doing x and z then y. + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + xorl %r13d, %r13d + xorl %r14d, %r14d + xorl %r15d, %r15d + +selectblock_xz(1) + selectblock_xz(2) + selectblock_xz(3) + selectblock_xz(4) + selectblock_xz(5) + selectblock_xz(6) + selectblock_xz(7) + selectblock_xz(8) + selectblock_xz(9) + selectblock_xz(10) + selectblock_xz(11) + selectblock_xz(12) + selectblock_xz(13) + selectblock_xz(14) + selectblock_xz(15) + selectblock_xz(16) + + movq %rax, TABENT(%rsp) + movq %rbx, TABENT+8(%rsp) + movq %rcx, TABENT+16(%rsp) + movq %rdx, TABENT+24(%rsp) + movq %r8, TABENT+32(%rsp) + movq %r9, TABENT+40(%rsp) + movq %r10, TABENT+96(%rsp) + movq %r11, TABENT+104(%rsp) + movq %r12, TABENT+112(%rsp) + movq %r13, TABENT+120(%rsp) + movq %r14, TABENT+128(%rsp) + movq %r15, TABENT+136(%rsp) + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + + selectblock_y(1) + selectblock_y(2) + selectblock_y(3) + selectblock_y(4) + selectblock_y(5) + selectblock_y(6) + selectblock_y(7) + selectblock_y(8) + selectblock_y(9) + selectblock_y(10) + selectblock_y(11) + selectblock_y(12) + selectblock_y(13) + selectblock_y(14) + selectblock_y(15) + selectblock_y(16) + +// Store it to "tabent" with the y coordinate optionally negated. +// This is done carefully to give coordinates < p_384 even in +// the degenerate case y = 0 (when z = 0 for points on the curve). +// The digits of the prime p_384 are generated dynamically from +// the zeroth via not/lea to reduce the number of constant loads. + + movq %rax, %r10 + orq %rbx, %r10 + movq %rcx, %r11 + orq %rdx, %r11 + movq %r8, %r12 + orq %r9, %r12 + orq %r11, %r10 + orq %r12, %r10 + cmovzq %r10, %rsi + + movl $0xffffffff, %r10d + movq %r10, %r11 + notq %r11 + leaq (%r10,%r11), %r13 + subq %rax, %r10 + leaq -1(%r13), %r12 + sbbq %rbx, %r11 + movq %r13, %r14 + sbbq %rcx, %r12 + sbbq %rdx, %r13 + movq %r14, %r15 + sbbq %r8, %r14 + sbbq %r9, %r15 + + testq %rsi, %rsi + cmovnzq %r10, %rax + cmovnzq %r11, %rbx + cmovnzq %r12, %rcx + cmovnzq %r13, %rdx + cmovnzq %r14, %r8 + cmovnzq %r15, %r9 + + movq %rax, TABENT+48(%rsp) + movq %rbx, TABENT+56(%rsp) + movq %rcx, TABENT+64(%rsp) + movq %rdx, TABENT+72(%rsp) + movq %r8, TABENT+80(%rsp) + movq %r9, TABENT+88(%rsp) + +// Add to the accumulator + + leaq TABENT(%rsp), %rdx + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp384_montjscalarmul_alt_p384_montjadd) + + testq %rbp, %rbp + jne Lp384_montjscalarmul_alt_mainloop + +// That's the end of the main loop, and we just need to copy the +// result in "acc" to the output. + + movq res, %rdi + movq ACC(%rsp), %rax + movq %rax, (%rdi) + movq ACC+8(%rsp), %rax + movq %rax, 8(%rdi) + movq ACC+16(%rsp), %rax + movq %rax, 16(%rdi) + movq ACC+24(%rsp), %rax + movq %rax, 24(%rdi) + movq ACC+32(%rsp), %rax + movq %rax, 32(%rdi) + movq ACC+40(%rsp), %rax + movq %rax, 40(%rdi) + movq ACC+48(%rsp), %rax + movq %rax, 48(%rdi) + movq ACC+56(%rsp), %rax + movq %rax, 56(%rdi) + movq ACC+64(%rsp), %rax + movq %rax, 64(%rdi) + movq ACC+72(%rsp), %rax + movq %rax, 72(%rdi) + movq ACC+80(%rsp), %rax + movq %rax, 80(%rdi) + movq ACC+88(%rsp), %rax + movq %rax, 88(%rdi) + movq ACC+96(%rsp), %rax + movq %rax, 96(%rdi) + movq ACC+104(%rsp), %rax + movq %rax, 104(%rdi) + movq ACC+112(%rsp), %rax + movq %rax, 112(%rdi) + movq ACC+120(%rsp), %rax + movq %rax, 120(%rdi) + movq ACC+128(%rsp), %rax + movq %rax, 128(%rdi) + movq ACC+136(%rsp), %rax + movq %rax, 136(%rdi) + +// Restore stack and registers and return + + CFI_INC_RSP(NSPACE) + CFI_POP(%rbx) + CFI_POP(%rbp) + CFI_POP(%r12) + CFI_POP(%r13) + CFI_POP(%r14) + CFI_POP(%r15) + CFI_RET + +#if WINDOWS_ABI +S2N_BN_SIZE_DIRECTIVE(Lp384_montjscalarmul_alt_standard) +#else +S2N_BN_SIZE_DIRECTIVE(p384_montjscalarmul_alt) +#endif + +// Local copies of subroutines, complete clones at the moment + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp384_montjscalarmul_alt_p384_montjadd) + +Lp384_montjscalarmul_alt_p384_montjadd: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(352) + movq %rsi, 0x150(%rsp) + movq %rdx, 0x158(%rsp) + movq 0x60(%rsi), %rbx + movq 0x68(%rsi), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x78(%rsi), %rax + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x88(%rsi), %rax + mulq %rbx + movq %rax, %r13 + movq %rdx, %r14 + movq 0x78(%rsi), %rax + mulq 0x80(%rsi) + movq %rax, %r15 + movq %rdx, %rcx + movq 0x70(%rsi), %rbx + movq 0x60(%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x68(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rbp, %rbp + movq 0x68(%rsi), %rbx + movq 0x78(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x80(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x88(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %rcx + movq 0x80(%rsi), %rbx + movq 0x60(%rsi), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x70(%rsi), %rbx + movq 0x78(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x80(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x88(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r15 + adcq %rdx, %rcx + sbbq %rbp, %rbp + xorl %ebx, %ebx + movq 0x78(%rsi), %rax + mulq 0x88(%rsi) + subq %rbp, %rdx + xorl %ebp, %ebp + addq %rax, %rcx + adcq %rdx, %rbx + adcl %ebp, %ebp + movq 0x80(%rsi), %rax + mulq 0x88(%rsi) + addq %rax, %rbx + adcq %rdx, %rbp + xorl %r8d, %r8d + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %r15, %r15 + adcq %rcx, %rcx + adcq %rbx, %rbx + adcq %rbp, %rbp + adcl %r8d, %r8d + movq 0x60(%rsi), %rax + mulq %rax + movq %r8, (%rsp) + movq %rax, %r8 + movq 0x68(%rsi), %rax + movq %rbp, 0x8(%rsp) + addq %rdx, %r9 + sbbq %rbp, %rbp + mulq %rax + negq %rbp + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x70(%rsi), %rax + mulq %rax + negq %rbp + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x78(%rsi), %rax + mulq %rax + negq %rbp + adcq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x80(%rsi), %rax + mulq %rax + negq %rbp + adcq %rax, %rcx + adcq %rdx, %rbx + sbbq %rbp, %rbp + movq 0x88(%rsi), %rax + mulq %rax + negq %rbp + adcq 0x8(%rsp), %rax + adcq (%rsp), %rdx + movq %rax, %rbp + movq %rdx, %rsi + movq %rbx, (%rsp) + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r8 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r8, %r9 + sbbq %rdx, %r10 + sbbq %rax, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rbx, %r8 + sbbq $0x0, %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r9 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r9, %r10 + sbbq %rdx, %r11 + sbbq %rax, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rbx, %r9 + sbbq $0x0, %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r10 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r10, %r11 + sbbq %rdx, %r12 + sbbq %rax, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rbx, %r10 + sbbq $0x0, %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r11 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r11, %r12 + sbbq %rdx, %r13 + sbbq %rax, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rbx, %r11 + sbbq $0x0, %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r12 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r12, %r13 + sbbq %rdx, %r8 + sbbq %rax, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rbx, %r12 + sbbq $0x0, %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r13 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r13, %r8 + sbbq %rdx, %r9 + sbbq %rax, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rbx, %r13 + sbbq $0x0, %r13 + movq (%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rsi + movl $0x0, %r8d + adcq %r8, %r8 + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %r9d + adcq %r15, %r9 + movl $0x1, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rsi, %r13 + adcq $0x0, %r8 + cmovneq %rax, %r14 + cmovneq %r9, %r15 + cmovneq %r10, %rcx + cmovneq %r11, %rbx + cmovneq %r12, %rbp + cmovneq %r13, %rsi + movq %r14, (%rsp) + movq %r15, 0x8(%rsp) + movq %rcx, 0x10(%rsp) + movq %rbx, 0x18(%rsp) + movq %rbp, 0x20(%rsp) + movq %rsi, 0x28(%rsp) + movq 0x158(%rsp), %rsi + movq 0x60(%rsi), %rbx + movq 0x68(%rsi), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x78(%rsi), %rax + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x88(%rsi), %rax + mulq %rbx + movq %rax, %r13 + movq %rdx, %r14 + movq 0x78(%rsi), %rax + mulq 0x80(%rsi) + movq %rax, %r15 + movq %rdx, %rcx + movq 0x70(%rsi), %rbx + movq 0x60(%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x68(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rbp, %rbp + movq 0x68(%rsi), %rbx + movq 0x78(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x80(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x88(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %rcx + movq 0x80(%rsi), %rbx + movq 0x60(%rsi), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x70(%rsi), %rbx + movq 0x78(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x80(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x88(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r15 + adcq %rdx, %rcx + sbbq %rbp, %rbp + xorl %ebx, %ebx + movq 0x78(%rsi), %rax + mulq 0x88(%rsi) + subq %rbp, %rdx + xorl %ebp, %ebp + addq %rax, %rcx + adcq %rdx, %rbx + adcl %ebp, %ebp + movq 0x80(%rsi), %rax + mulq 0x88(%rsi) + addq %rax, %rbx + adcq %rdx, %rbp + xorl %r8d, %r8d + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %r15, %r15 + adcq %rcx, %rcx + adcq %rbx, %rbx + adcq %rbp, %rbp + adcl %r8d, %r8d + movq 0x60(%rsi), %rax + mulq %rax + movq %r8, 0xf0(%rsp) + movq %rax, %r8 + movq 0x68(%rsi), %rax + movq %rbp, 0xf8(%rsp) + addq %rdx, %r9 + sbbq %rbp, %rbp + mulq %rax + negq %rbp + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x70(%rsi), %rax + mulq %rax + negq %rbp + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x78(%rsi), %rax + mulq %rax + negq %rbp + adcq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x80(%rsi), %rax + mulq %rax + negq %rbp + adcq %rax, %rcx + adcq %rdx, %rbx + sbbq %rbp, %rbp + movq 0x88(%rsi), %rax + mulq %rax + negq %rbp + adcq 0xf8(%rsp), %rax + adcq 0xf0(%rsp), %rdx + movq %rax, %rbp + movq %rdx, %rsi + movq %rbx, 0xf0(%rsp) + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r8 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r8, %r9 + sbbq %rdx, %r10 + sbbq %rax, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rbx, %r8 + sbbq $0x0, %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r9 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r9, %r10 + sbbq %rdx, %r11 + sbbq %rax, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rbx, %r9 + sbbq $0x0, %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r10 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r10, %r11 + sbbq %rdx, %r12 + sbbq %rax, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rbx, %r10 + sbbq $0x0, %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r11 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r11, %r12 + sbbq %rdx, %r13 + sbbq %rax, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rbx, %r11 + sbbq $0x0, %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r12 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r12, %r13 + sbbq %rdx, %r8 + sbbq %rax, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rbx, %r12 + sbbq $0x0, %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r13 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r13, %r8 + sbbq %rdx, %r9 + sbbq %rax, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rbx, %r13 + sbbq $0x0, %r13 + movq 0xf0(%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rsi + movl $0x0, %r8d + adcq %r8, %r8 + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %r9d + adcq %r15, %r9 + movl $0x1, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rsi, %r13 + adcq $0x0, %r8 + cmovneq %rax, %r14 + cmovneq %r9, %r15 + cmovneq %r10, %rcx + cmovneq %r11, %rbx + cmovneq %r12, %rbp + cmovneq %r13, %rsi + movq %r14, 0xf0(%rsp) + movq %r15, 0xf8(%rsp) + movq %rcx, 0x100(%rsp) + movq %rbx, 0x108(%rsp) + movq %rbp, 0x110(%rsp) + movq %rsi, 0x118(%rsp) + movq 0x150(%rsp), %rsi + movq 0x158(%rsp), %rcx + movq 0x30(%rsi), %rbx + movq 0x60(%rcx), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x68(%rcx), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x70(%rcx), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x78(%rcx), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x80(%rcx), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x88(%rcx), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r8, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rdx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rbx + addq %rbx, %r14 + adcq $0x0, %r15 + movq 0x38(%rsi), %rbx + movq 0x60(%rcx), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r8, %r8 + movq 0x68(%rcx), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x70(%rcx), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x78(%rcx), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x80(%rcx), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r8, %r8 + movq 0x88(%rcx), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r8, %r8 + negq %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r9, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rdx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rbx + addq %rbx, %r15 + adcq $0x0, %r8 + movq 0x40(%rsi), %rbx + movq 0x60(%rcx), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r9, %r9 + movq 0x68(%rcx), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x70(%rcx), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x78(%rcx), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x80(%rcx), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + movq 0x88(%rcx), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r10, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rdx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rbx + addq %rbx, %r8 + adcq $0x0, %r9 + movq 0x48(%rsi), %rbx + movq 0x60(%rcx), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r10, %r10 + movq 0x68(%rcx), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r10, %r10 + movq 0x70(%rcx), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r10, %r10 + movq 0x78(%rcx), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r10, %r10 + movq 0x80(%rcx), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r10, %r10 + movq 0x88(%rcx), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r11, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rbx + addq %rbx, %r9 + adcq $0x0, %r10 + movq 0x50(%rsi), %rbx + movq 0x60(%rcx), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r11, %r11 + movq 0x68(%rcx), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r11, %r11 + movq 0x70(%rcx), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r11, %r11 + movq 0x78(%rcx), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r11, %r11 + movq 0x80(%rcx), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r11, %r11 + movq 0x88(%rcx), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r11, %r11 + negq %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r12, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rdx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rbx + addq %rbx, %r10 + adcq $0x0, %r11 + movq 0x58(%rsi), %rbx + movq 0x60(%rcx), %rax + mulq %rbx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r12, %r12 + movq 0x68(%rcx), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r12, %r12 + movq 0x70(%rcx), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r12, %r12 + movq 0x78(%rcx), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r12, %r12 + movq 0x80(%rcx), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r12, %r12 + movq 0x88(%rcx), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r12, %r12 + negq %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r13, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rdx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rbx + addq %rbx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x120(%rsp) + movq %r15, 0x128(%rsp) + movq %r8, 0x130(%rsp) + movq %r9, 0x138(%rsp) + movq %r10, 0x140(%rsp) + movq %r11, 0x148(%rsp) + movq 0x150(%rsp), %rsi + movq 0x158(%rsp), %rcx + movq 0x30(%rcx), %rbx + movq 0x60(%rsi), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x68(%rsi), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x70(%rsi), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x78(%rsi), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x80(%rsi), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x88(%rsi), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r8, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rdx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rbx + addq %rbx, %r14 + adcq $0x0, %r15 + movq 0x38(%rcx), %rbx + movq 0x60(%rsi), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r8, %r8 + movq 0x68(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x70(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x78(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x80(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r8, %r8 + movq 0x88(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r8, %r8 + negq %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r9, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rdx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rbx + addq %rbx, %r15 + adcq $0x0, %r8 + movq 0x40(%rcx), %rbx + movq 0x60(%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r9, %r9 + movq 0x68(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x70(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x78(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x80(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + movq 0x88(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r10, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rdx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rbx + addq %rbx, %r8 + adcq $0x0, %r9 + movq 0x48(%rcx), %rbx + movq 0x60(%rsi), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r10, %r10 + movq 0x68(%rsi), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r10, %r10 + movq 0x70(%rsi), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r10, %r10 + movq 0x78(%rsi), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r10, %r10 + movq 0x80(%rsi), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r10, %r10 + movq 0x88(%rsi), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r11, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rbx + addq %rbx, %r9 + adcq $0x0, %r10 + movq 0x50(%rcx), %rbx + movq 0x60(%rsi), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r11, %r11 + movq 0x68(%rsi), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r11, %r11 + movq 0x70(%rsi), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r11, %r11 + movq 0x78(%rsi), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r11, %r11 + movq 0x80(%rsi), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r11, %r11 + movq 0x88(%rsi), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r11, %r11 + negq %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r12, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rdx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rbx + addq %rbx, %r10 + adcq $0x0, %r11 + movq 0x58(%rcx), %rbx + movq 0x60(%rsi), %rax + mulq %rbx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r12, %r12 + movq 0x68(%rsi), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r12, %r12 + movq 0x70(%rsi), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r12, %r12 + movq 0x78(%rsi), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r12, %r12 + movq 0x80(%rsi), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r12, %r12 + movq 0x88(%rsi), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r12, %r12 + negq %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r13, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rdx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rbx + addq %rbx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq %r8, 0x40(%rsp) + movq %r9, 0x48(%rsp) + movq %r10, 0x50(%rsp) + movq %r11, 0x58(%rsp) + movq 0x158(%rsp), %rcx + movq (%rcx), %rbx + movq (%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x10(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x18(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x20(%rsp), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x28(%rsp), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r8, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rdx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rbx + addq %rbx, %r14 + adcq $0x0, %r15 + movq 0x8(%rcx), %rbx + movq (%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r8, %r8 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x20(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r8, %r8 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r8, %r8 + negq %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r9, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rdx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rbx + addq %rbx, %r15 + adcq $0x0, %r8 + movq 0x10(%rcx), %rbx + movq (%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r9, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x20(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r10, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rdx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rbx + addq %rbx, %r8 + adcq $0x0, %r9 + movq 0x18(%rcx), %rbx + movq (%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r10, %r10 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r10, %r10 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r10, %r10 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r10, %r10 + movq 0x20(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r10, %r10 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r11, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rbx + addq %rbx, %r9 + adcq $0x0, %r10 + movq 0x20(%rcx), %rbx + movq (%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r11, %r11 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r11, %r11 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r11, %r11 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r11, %r11 + movq 0x20(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r11, %r11 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r11, %r11 + negq %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r12, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rdx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rbx + addq %rbx, %r10 + adcq $0x0, %r11 + movq 0x28(%rcx), %rbx + movq (%rsp), %rax + mulq %rbx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r12, %r12 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r12, %r12 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r12, %r12 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r12, %r12 + movq 0x20(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r12, %r12 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r12, %r12 + negq %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r13, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rdx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rbx + addq %rbx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x60(%rsp) + movq %r15, 0x68(%rsp) + movq %r8, 0x70(%rsp) + movq %r9, 0x78(%rsp) + movq %r10, 0x80(%rsp) + movq %r11, 0x88(%rsp) + movq 0x150(%rsp), %rsi + movq (%rsi), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0xf8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x100(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x108(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x110(%rsp), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x118(%rsp), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r8, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rdx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rbx + addq %rbx, %r14 + adcq $0x0, %r15 + movq 0x8(%rsi), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r8, %r8 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r8, %r8 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r8, %r8 + negq %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r9, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rdx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rbx + addq %rbx, %r15 + adcq $0x0, %r8 + movq 0x10(%rsi), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r9, %r9 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r10, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rdx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rbx + addq %rbx, %r8 + adcq $0x0, %r9 + movq 0x18(%rsi), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r10, %r10 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r10, %r10 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r10, %r10 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r10, %r10 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r10, %r10 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r11, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rbx + addq %rbx, %r9 + adcq $0x0, %r10 + movq 0x20(%rsi), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r11, %r11 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r11, %r11 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r11, %r11 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r11, %r11 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r11, %r11 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r11, %r11 + negq %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r12, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rdx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rbx + addq %rbx, %r10 + adcq $0x0, %r11 + movq 0x28(%rsi), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r12, %r12 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r12, %r12 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r12, %r12 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r12, %r12 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r12, %r12 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r12, %r12 + negq %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r13, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rdx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rbx + addq %rbx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0xc0(%rsp) + movq %r15, 0xc8(%rsp) + movq %r8, 0xd0(%rsp) + movq %r9, 0xd8(%rsp) + movq %r10, 0xe0(%rsp) + movq %r11, 0xe8(%rsp) + movq 0x30(%rsp), %rbx + movq (%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x10(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x18(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x20(%rsp), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x28(%rsp), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r8, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rdx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rbx + addq %rbx, %r14 + adcq $0x0, %r15 + movq 0x38(%rsp), %rbx + movq (%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r8, %r8 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x20(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r8, %r8 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r8, %r8 + negq %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r9, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rdx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rbx + addq %rbx, %r15 + adcq $0x0, %r8 + movq 0x40(%rsp), %rbx + movq (%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r9, %r9 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x20(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r10, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rdx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rbx + addq %rbx, %r8 + adcq $0x0, %r9 + movq 0x48(%rsp), %rbx + movq (%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r10, %r10 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r10, %r10 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r10, %r10 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r10, %r10 + movq 0x20(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r10, %r10 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r11, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rbx + addq %rbx, %r9 + adcq $0x0, %r10 + movq 0x50(%rsp), %rbx + movq (%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r11, %r11 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r11, %r11 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r11, %r11 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r11, %r11 + movq 0x20(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r11, %r11 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r11, %r11 + negq %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r12, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rdx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rbx + addq %rbx, %r10 + adcq $0x0, %r11 + movq 0x58(%rsp), %rbx + movq (%rsp), %rax + mulq %rbx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r12, %r12 + movq 0x8(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r12, %r12 + movq 0x10(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r12, %r12 + movq 0x18(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r12, %r12 + movq 0x20(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r12, %r12 + movq 0x28(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r12, %r12 + negq %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r13, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rdx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rbx + addq %rbx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq %r8, 0x40(%rsp) + movq %r9, 0x48(%rsp) + movq %r10, 0x50(%rsp) + movq %r11, 0x58(%rsp) + movq 0x120(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0xf8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x100(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x108(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x110(%rsp), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x118(%rsp), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r8, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rdx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rbx + addq %rbx, %r14 + adcq $0x0, %r15 + movq 0x128(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r8, %r8 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r8, %r8 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r8, %r8 + negq %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r9, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rdx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rbx + addq %rbx, %r15 + adcq $0x0, %r8 + movq 0x130(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r9, %r9 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r10, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rdx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rbx + addq %rbx, %r8 + adcq $0x0, %r9 + movq 0x138(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r10, %r10 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r10, %r10 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r10, %r10 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r10, %r10 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r10, %r10 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r11, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rbx + addq %rbx, %r9 + adcq $0x0, %r10 + movq 0x140(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r11, %r11 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r11, %r11 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r11, %r11 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r11, %r11 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r11, %r11 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r11, %r11 + negq %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r12, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rdx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rbx + addq %rbx, %r10 + adcq $0x0, %r11 + movq 0x148(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r12, %r12 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r12, %r12 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r12, %r12 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r12, %r12 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r12, %r12 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r12, %r12 + negq %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r13, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rdx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rbx + addq %rbx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x120(%rsp) + movq %r15, 0x128(%rsp) + movq %r8, 0x130(%rsp) + movq %r9, 0x138(%rsp) + movq %r10, 0x140(%rsp) + movq %r11, 0x148(%rsp) + movq 0x60(%rsp), %rax + subq 0xc0(%rsp), %rax + movq 0x68(%rsp), %rdx + sbbq 0xc8(%rsp), %rdx + movq 0x70(%rsp), %r8 + sbbq 0xd0(%rsp), %r8 + movq 0x78(%rsp), %r9 + sbbq 0xd8(%rsp), %r9 + movq 0x80(%rsp), %r10 + sbbq 0xe0(%rsp), %r10 + movq 0x88(%rsp), %r11 + sbbq 0xe8(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %esi + andq %rsi, %rcx + xorq %rsi, %rsi + subq %rcx, %rsi + subq %rsi, %rax + movq %rax, 0xf0(%rsp) + sbbq %rcx, %rdx + movq %rdx, 0xf8(%rsp) + sbbq %rax, %rax + andq %rsi, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0x100(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x108(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x110(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x118(%rsp) + movq 0x30(%rsp), %rax + subq 0x120(%rsp), %rax + movq 0x38(%rsp), %rdx + sbbq 0x128(%rsp), %rdx + movq 0x40(%rsp), %r8 + sbbq 0x130(%rsp), %r8 + movq 0x48(%rsp), %r9 + sbbq 0x138(%rsp), %r9 + movq 0x50(%rsp), %r10 + sbbq 0x140(%rsp), %r10 + movq 0x58(%rsp), %r11 + sbbq 0x148(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %esi + andq %rsi, %rcx + xorq %rsi, %rsi + subq %rcx, %rsi + subq %rsi, %rax + movq %rax, 0x30(%rsp) + sbbq %rcx, %rdx + movq %rdx, 0x38(%rsp) + sbbq %rax, %rax + andq %rsi, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0x40(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x48(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x50(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x58(%rsp) + movq 0xf0(%rsp), %rbx + movq 0xf8(%rsp), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x108(%rsp), %rax + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x118(%rsp), %rax + mulq %rbx + movq %rax, %r13 + movq %rdx, %r14 + movq 0x108(%rsp), %rax + mulq 0x110(%rsp) + movq %rax, %r15 + movq %rdx, %rcx + movq 0x100(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0xf8(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rbp, %rbp + movq 0xf8(%rsp), %rbx + movq 0x108(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x110(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x118(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %rcx + movq 0x110(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x100(%rsp), %rbx + movq 0x108(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x110(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x118(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r15 + adcq %rdx, %rcx + sbbq %rbp, %rbp + xorl %ebx, %ebx + movq 0x108(%rsp), %rax + mulq 0x118(%rsp) + subq %rbp, %rdx + xorl %ebp, %ebp + addq %rax, %rcx + adcq %rdx, %rbx + adcl %ebp, %ebp + movq 0x110(%rsp), %rax + mulq 0x118(%rsp) + addq %rax, %rbx + adcq %rdx, %rbp + xorl %r8d, %r8d + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %r15, %r15 + adcq %rcx, %rcx + adcq %rbx, %rbx + adcq %rbp, %rbp + adcl %r8d, %r8d + movq 0xf0(%rsp), %rax + mulq %rax + movq %r8, 0x90(%rsp) + movq %rax, %r8 + movq 0xf8(%rsp), %rax + movq %rbp, 0x98(%rsp) + addq %rdx, %r9 + sbbq %rbp, %rbp + mulq %rax + negq %rbp + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x100(%rsp), %rax + mulq %rax + negq %rbp + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x108(%rsp), %rax + mulq %rax + negq %rbp + adcq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x110(%rsp), %rax + mulq %rax + negq %rbp + adcq %rax, %rcx + adcq %rdx, %rbx + sbbq %rbp, %rbp + movq 0x118(%rsp), %rax + mulq %rax + negq %rbp + adcq 0x98(%rsp), %rax + adcq 0x90(%rsp), %rdx + movq %rax, %rbp + movq %rdx, %rsi + movq %rbx, 0x90(%rsp) + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r8 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r8, %r9 + sbbq %rdx, %r10 + sbbq %rax, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rbx, %r8 + sbbq $0x0, %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r9 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r9, %r10 + sbbq %rdx, %r11 + sbbq %rax, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rbx, %r9 + sbbq $0x0, %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r10 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r10, %r11 + sbbq %rdx, %r12 + sbbq %rax, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rbx, %r10 + sbbq $0x0, %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r11 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r11, %r12 + sbbq %rdx, %r13 + sbbq %rax, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rbx, %r11 + sbbq $0x0, %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r12 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r12, %r13 + sbbq %rdx, %r8 + sbbq %rax, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rbx, %r12 + sbbq $0x0, %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r13 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r13, %r8 + sbbq %rdx, %r9 + sbbq %rax, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rbx, %r13 + sbbq $0x0, %r13 + movq 0x90(%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rsi + movl $0x0, %r8d + adcq %r8, %r8 + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %r9d + adcq %r15, %r9 + movl $0x1, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rsi, %r13 + adcq $0x0, %r8 + cmovneq %rax, %r14 + cmovneq %r9, %r15 + cmovneq %r10, %rcx + cmovneq %r11, %rbx + cmovneq %r12, %rbp + cmovneq %r13, %rsi + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq %rcx, 0xa0(%rsp) + movq %rbx, 0xa8(%rsp) + movq %rbp, 0xb0(%rsp) + movq %rsi, 0xb8(%rsp) + movq 0x30(%rsp), %rbx + movq 0x38(%rsp), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x48(%rsp), %rax + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x58(%rsp), %rax + mulq %rbx + movq %rax, %r13 + movq %rdx, %r14 + movq 0x48(%rsp), %rax + mulq 0x50(%rsp) + movq %rax, %r15 + movq %rdx, %rcx + movq 0x40(%rsp), %rbx + movq 0x30(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x38(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rbp, %rbp + movq 0x38(%rsp), %rbx + movq 0x48(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x50(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x58(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %rcx + movq 0x50(%rsp), %rbx + movq 0x30(%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x40(%rsp), %rbx + movq 0x48(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x50(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x58(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r15 + adcq %rdx, %rcx + sbbq %rbp, %rbp + xorl %ebx, %ebx + movq 0x48(%rsp), %rax + mulq 0x58(%rsp) + subq %rbp, %rdx + xorl %ebp, %ebp + addq %rax, %rcx + adcq %rdx, %rbx + adcl %ebp, %ebp + movq 0x50(%rsp), %rax + mulq 0x58(%rsp) + addq %rax, %rbx + adcq %rdx, %rbp + xorl %r8d, %r8d + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %r15, %r15 + adcq %rcx, %rcx + adcq %rbx, %rbx + adcq %rbp, %rbp + adcl %r8d, %r8d + movq 0x30(%rsp), %rax + mulq %rax + movq %r8, (%rsp) + movq %rax, %r8 + movq 0x38(%rsp), %rax + movq %rbp, 0x8(%rsp) + addq %rdx, %r9 + sbbq %rbp, %rbp + mulq %rax + negq %rbp + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x40(%rsp), %rax + mulq %rax + negq %rbp + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x48(%rsp), %rax + mulq %rax + negq %rbp + adcq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x50(%rsp), %rax + mulq %rax + negq %rbp + adcq %rax, %rcx + adcq %rdx, %rbx + sbbq %rbp, %rbp + movq 0x58(%rsp), %rax + mulq %rax + negq %rbp + adcq 0x8(%rsp), %rax + adcq (%rsp), %rdx + movq %rax, %rbp + movq %rdx, %rsi + movq %rbx, (%rsp) + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r8 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r8, %r9 + sbbq %rdx, %r10 + sbbq %rax, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rbx, %r8 + sbbq $0x0, %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r9 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r9, %r10 + sbbq %rdx, %r11 + sbbq %rax, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rbx, %r9 + sbbq $0x0, %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r10 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r10, %r11 + sbbq %rdx, %r12 + sbbq %rax, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rbx, %r10 + sbbq $0x0, %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r11 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r11, %r12 + sbbq %rdx, %r13 + sbbq %rax, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rbx, %r11 + sbbq $0x0, %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r12 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r12, %r13 + sbbq %rdx, %r8 + sbbq %rax, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rbx, %r12 + sbbq $0x0, %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r13 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r13, %r8 + sbbq %rdx, %r9 + sbbq %rax, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rbx, %r13 + sbbq $0x0, %r13 + movq (%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rsi + movl $0x0, %r8d + adcq %r8, %r8 + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %r9d + adcq %r15, %r9 + movl $0x1, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rsi, %r13 + adcq $0x0, %r8 + cmovneq %rax, %r14 + cmovneq %r9, %r15 + cmovneq %r10, %rcx + cmovneq %r11, %rbx + cmovneq %r12, %rbp + cmovneq %r13, %rsi + movq %r14, (%rsp) + movq %r15, 0x8(%rsp) + movq %rcx, 0x10(%rsp) + movq %rbx, 0x18(%rsp) + movq %rbp, 0x20(%rsp) + movq %rsi, 0x28(%rsp) + movq 0xc0(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x98(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0xa0(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0xa8(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0xb0(%rsp), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0xb8(%rsp), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r8, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rdx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rbx + addq %rbx, %r14 + adcq $0x0, %r15 + movq 0xc8(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r8, %r8 + movq 0x98(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0xa0(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r8, %r8 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r8, %r8 + negq %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r9, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rdx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rbx + addq %rbx, %r15 + adcq $0x0, %r8 + movq 0xd0(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r9, %r9 + movq 0x98(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0xa0(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r10, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rdx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rbx + addq %rbx, %r8 + adcq $0x0, %r9 + movq 0xd8(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r10, %r10 + movq 0x98(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r10, %r10 + movq 0xa0(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r10, %r10 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r10, %r10 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r10, %r10 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r11, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rbx + addq %rbx, %r9 + adcq $0x0, %r10 + movq 0xe0(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r11, %r11 + movq 0x98(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r11, %r11 + movq 0xa0(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r11, %r11 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r11, %r11 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r11, %r11 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r11, %r11 + negq %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r12, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rdx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rbx + addq %rbx, %r10 + adcq $0x0, %r11 + movq 0xe8(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r12, %r12 + movq 0x98(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r12, %r12 + movq 0xa0(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r12, %r12 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r12, %r12 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r12, %r12 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r12, %r12 + negq %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r13, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rdx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rbx + addq %rbx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0xc0(%rsp) + movq %r15, 0xc8(%rsp) + movq %r8, 0xd0(%rsp) + movq %r9, 0xd8(%rsp) + movq %r10, 0xe0(%rsp) + movq %r11, 0xe8(%rsp) + movq 0x60(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x98(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0xa0(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0xa8(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0xb0(%rsp), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0xb8(%rsp), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r8, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rdx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rbx + addq %rbx, %r14 + adcq $0x0, %r15 + movq 0x68(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r8, %r8 + movq 0x98(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0xa0(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r8, %r8 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r8, %r8 + negq %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r9, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rdx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rbx + addq %rbx, %r15 + adcq $0x0, %r8 + movq 0x70(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r9, %r9 + movq 0x98(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0xa0(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r10, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rdx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rbx + addq %rbx, %r8 + adcq $0x0, %r9 + movq 0x78(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r10, %r10 + movq 0x98(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r10, %r10 + movq 0xa0(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r10, %r10 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r10, %r10 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r10, %r10 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r11, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rbx + addq %rbx, %r9 + adcq $0x0, %r10 + movq 0x80(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r11, %r11 + movq 0x98(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r11, %r11 + movq 0xa0(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r11, %r11 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r11, %r11 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r11, %r11 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r11, %r11 + negq %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r12, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rdx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rbx + addq %rbx, %r10 + adcq $0x0, %r11 + movq 0x88(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r12, %r12 + movq 0x98(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r12, %r12 + movq 0xa0(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r12, %r12 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r12, %r12 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r12, %r12 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r12, %r12 + negq %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r13, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rdx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rbx + addq %rbx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x60(%rsp) + movq %r15, 0x68(%rsp) + movq %r8, 0x70(%rsp) + movq %r9, 0x78(%rsp) + movq %r10, 0x80(%rsp) + movq %r11, 0x88(%rsp) + movq (%rsp), %rax + subq 0xc0(%rsp), %rax + movq 0x8(%rsp), %rdx + sbbq 0xc8(%rsp), %rdx + movq 0x10(%rsp), %r8 + sbbq 0xd0(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0xd8(%rsp), %r9 + movq 0x20(%rsp), %r10 + sbbq 0xe0(%rsp), %r10 + movq 0x28(%rsp), %r11 + sbbq 0xe8(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %esi + andq %rsi, %rcx + xorq %rsi, %rsi + subq %rcx, %rsi + subq %rsi, %rax + movq %rax, (%rsp) + sbbq %rcx, %rdx + movq %rdx, 0x8(%rsp) + sbbq %rax, %rax + andq %rsi, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0x10(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x18(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x20(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x28(%rsp) + movq 0x60(%rsp), %rax + subq 0xc0(%rsp), %rax + movq 0x68(%rsp), %rdx + sbbq 0xc8(%rsp), %rdx + movq 0x70(%rsp), %r8 + sbbq 0xd0(%rsp), %r8 + movq 0x78(%rsp), %r9 + sbbq 0xd8(%rsp), %r9 + movq 0x80(%rsp), %r10 + sbbq 0xe0(%rsp), %r10 + movq 0x88(%rsp), %r11 + sbbq 0xe8(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %esi + andq %rsi, %rcx + xorq %rsi, %rsi + subq %rcx, %rsi + subq %rsi, %rax + movq %rax, 0x90(%rsp) + sbbq %rcx, %rdx + movq %rdx, 0x98(%rsp) + sbbq %rax, %rax + andq %rsi, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0xa0(%rsp) + sbbq $0x0, %r9 + movq %r9, 0xa8(%rsp) + sbbq $0x0, %r10 + movq %r10, 0xb0(%rsp) + sbbq $0x0, %r11 + movq %r11, 0xb8(%rsp) + movq 0x150(%rsp), %rsi + movq 0x60(%rsi), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0xf8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x100(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x108(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x110(%rsp), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x118(%rsp), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r8, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rdx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rbx + addq %rbx, %r14 + adcq $0x0, %r15 + movq 0x68(%rsi), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r8, %r8 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r8, %r8 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r8, %r8 + negq %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r9, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rdx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rbx + addq %rbx, %r15 + adcq $0x0, %r8 + movq 0x70(%rsi), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r9, %r9 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r10, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rdx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rbx + addq %rbx, %r8 + adcq $0x0, %r9 + movq 0x78(%rsi), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r10, %r10 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r10, %r10 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r10, %r10 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r10, %r10 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r10, %r10 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r11, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rbx + addq %rbx, %r9 + adcq $0x0, %r10 + movq 0x80(%rsi), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r11, %r11 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r11, %r11 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r11, %r11 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r11, %r11 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r11, %r11 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r11, %r11 + negq %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r12, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rdx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rbx + addq %rbx, %r10 + adcq $0x0, %r11 + movq 0x88(%rsi), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r12, %r12 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r12, %r12 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r12, %r12 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r12, %r12 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r12, %r12 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r12, %r12 + negq %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r13, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rdx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rbx + addq %rbx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0xf0(%rsp) + movq %r15, 0xf8(%rsp) + movq %r8, 0x100(%rsp) + movq %r9, 0x108(%rsp) + movq %r10, 0x110(%rsp) + movq %r11, 0x118(%rsp) + movq (%rsp), %rax + subq 0x60(%rsp), %rax + movq 0x8(%rsp), %rdx + sbbq 0x68(%rsp), %rdx + movq 0x10(%rsp), %r8 + sbbq 0x70(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x78(%rsp), %r9 + movq 0x20(%rsp), %r10 + sbbq 0x80(%rsp), %r10 + movq 0x28(%rsp), %r11 + sbbq 0x88(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %esi + andq %rsi, %rcx + xorq %rsi, %rsi + subq %rcx, %rsi + subq %rsi, %rax + movq %rax, (%rsp) + sbbq %rcx, %rdx + movq %rdx, 0x8(%rsp) + sbbq %rax, %rax + andq %rsi, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0x10(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x18(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x20(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x28(%rsp) + movq 0xc0(%rsp), %rax + subq (%rsp), %rax + movq 0xc8(%rsp), %rdx + sbbq 0x8(%rsp), %rdx + movq 0xd0(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0xd8(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movq 0xe0(%rsp), %r10 + sbbq 0x20(%rsp), %r10 + movq 0xe8(%rsp), %r11 + sbbq 0x28(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %esi + andq %rsi, %rcx + xorq %rsi, %rsi + subq %rcx, %rsi + subq %rsi, %rax + movq %rax, 0xc0(%rsp) + sbbq %rcx, %rdx + movq %rdx, 0xc8(%rsp) + sbbq %rax, %rax + andq %rsi, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0xd0(%rsp) + sbbq $0x0, %r9 + movq %r9, 0xd8(%rsp) + sbbq $0x0, %r10 + movq %r10, 0xe0(%rsp) + sbbq $0x0, %r11 + movq %r11, 0xe8(%rsp) + movq 0x120(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x98(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0xa0(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0xa8(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0xb0(%rsp), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0xb8(%rsp), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r8, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rdx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rbx + addq %rbx, %r14 + adcq $0x0, %r15 + movq 0x128(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r8, %r8 + movq 0x98(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0xa0(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r8, %r8 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r8, %r8 + negq %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r9, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rdx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rbx + addq %rbx, %r15 + adcq $0x0, %r8 + movq 0x130(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r9, %r9 + movq 0x98(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0xa0(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r10, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rdx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rbx + addq %rbx, %r8 + adcq $0x0, %r9 + movq 0x138(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r10, %r10 + movq 0x98(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r10, %r10 + movq 0xa0(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r10, %r10 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r10, %r10 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r10, %r10 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r11, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rbx + addq %rbx, %r9 + adcq $0x0, %r10 + movq 0x140(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r11, %r11 + movq 0x98(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r11, %r11 + movq 0xa0(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r11, %r11 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r11, %r11 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r11, %r11 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r11, %r11 + negq %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r12, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rdx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rbx + addq %rbx, %r10 + adcq $0x0, %r11 + movq 0x148(%rsp), %rbx + movq 0x90(%rsp), %rax + mulq %rbx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r12, %r12 + movq 0x98(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r12, %r12 + movq 0xa0(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r12, %r12 + movq 0xa8(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r12, %r12 + movq 0xb0(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r12, %r12 + movq 0xb8(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r12, %r12 + negq %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r13, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rdx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rbx + addq %rbx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq %r8, 0xa0(%rsp) + movq %r9, 0xa8(%rsp) + movq %r10, 0xb0(%rsp) + movq %r11, 0xb8(%rsp) + movq 0x158(%rsp), %rcx + movq 0x60(%rcx), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0xf8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x100(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x108(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x110(%rsp), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x118(%rsp), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r8, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rdx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rbx + addq %rbx, %r14 + adcq $0x0, %r15 + movq 0x68(%rcx), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r8, %r8 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r8, %r8 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r8, %r8 + negq %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r9, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rdx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rbx + addq %rbx, %r15 + adcq $0x0, %r8 + movq 0x70(%rcx), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r9, %r9 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r10, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rdx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rbx + addq %rbx, %r8 + adcq $0x0, %r9 + movq 0x78(%rcx), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r10, %r10 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r10, %r10 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r10, %r10 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r10, %r10 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r10, %r10 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r11, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rbx + addq %rbx, %r9 + adcq $0x0, %r10 + movq 0x80(%rcx), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r11, %r11 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r11, %r11 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r11, %r11 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r11, %r11 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r11, %r11 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r11, %r11 + negq %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r12, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rdx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rbx + addq %rbx, %r10 + adcq $0x0, %r11 + movq 0x88(%rcx), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r12, %r12 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r12, %r12 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r12, %r12 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r12, %r12 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r12, %r12 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r12, %r12 + negq %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r13, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rdx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rbx + addq %rbx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0xf0(%rsp) + movq %r15, 0xf8(%rsp) + movq %r8, 0x100(%rsp) + movq %r9, 0x108(%rsp) + movq %r10, 0x110(%rsp) + movq %r11, 0x118(%rsp) + movq 0xc0(%rsp), %rbx + movq 0x30(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x38(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x40(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x48(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x50(%rsp), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x58(%rsp), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r8, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rdx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rbx + addq %rbx, %r14 + adcq $0x0, %r15 + movq 0xc8(%rsp), %rbx + movq 0x30(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r8, %r8 + movq 0x38(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x40(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x48(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x50(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r8, %r8 + movq 0x58(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r8, %r8 + negq %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r9, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rdx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rbx + addq %rbx, %r15 + adcq $0x0, %r8 + movq 0xd0(%rsp), %rbx + movq 0x30(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r9, %r9 + movq 0x38(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x40(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x48(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x50(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + movq 0x58(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r10, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rdx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rbx + addq %rbx, %r8 + adcq $0x0, %r9 + movq 0xd8(%rsp), %rbx + movq 0x30(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r10, %r10 + movq 0x38(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r10, %r10 + movq 0x40(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r10, %r10 + movq 0x48(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r10, %r10 + movq 0x50(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r10, %r10 + movq 0x58(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r11, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rbx + addq %rbx, %r9 + adcq $0x0, %r10 + movq 0xe0(%rsp), %rbx + movq 0x30(%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r11, %r11 + movq 0x38(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r11, %r11 + movq 0x40(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r11, %r11 + movq 0x48(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r11, %r11 + movq 0x50(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r11, %r11 + movq 0x58(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r11, %r11 + negq %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r12, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rdx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rbx + addq %rbx, %r10 + adcq $0x0, %r11 + movq 0xe8(%rsp), %rbx + movq 0x30(%rsp), %rax + mulq %rbx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r12, %r12 + movq 0x38(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r12, %r12 + movq 0x40(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r12, %r12 + movq 0x48(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r12, %r12 + movq 0x50(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r12, %r12 + movq 0x58(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r12, %r12 + negq %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r13, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rdx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rbx + addq %rbx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0xc0(%rsp) + movq %r15, 0xc8(%rsp) + movq %r8, 0xd0(%rsp) + movq %r9, 0xd8(%rsp) + movq %r10, 0xe0(%rsp) + movq %r11, 0xe8(%rsp) + movq 0xc0(%rsp), %rax + subq 0x90(%rsp), %rax + movq 0xc8(%rsp), %rdx + sbbq 0x98(%rsp), %rdx + movq 0xd0(%rsp), %r8 + sbbq 0xa0(%rsp), %r8 + movq 0xd8(%rsp), %r9 + sbbq 0xa8(%rsp), %r9 + movq 0xe0(%rsp), %r10 + sbbq 0xb0(%rsp), %r10 + movq 0xe8(%rsp), %r11 + sbbq 0xb8(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %esi + andq %rsi, %rcx + xorq %rsi, %rsi + subq %rcx, %rsi + subq %rsi, %rax + movq %rax, 0xc0(%rsp) + sbbq %rcx, %rdx + movq %rdx, 0xc8(%rsp) + sbbq %rax, %rax + andq %rsi, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0xd0(%rsp) + sbbq $0x0, %r9 + movq %r9, 0xd8(%rsp) + sbbq $0x0, %r10 + movq %r10, 0xe0(%rsp) + sbbq $0x0, %r11 + movq %r11, 0xe8(%rsp) + movq 0x158(%rsp), %rcx + movq 0x60(%rcx), %r8 + movq 0x68(%rcx), %r9 + movq 0x70(%rcx), %r10 + movq 0x78(%rcx), %r11 + movq 0x80(%rcx), %rbx + movq 0x88(%rcx), %rbp + movq %r8, %rax + movq %r9, %rdx + orq %r10, %rax + orq %r11, %rdx + orq %rbx, %rax + orq %rbp, %rdx + orq %rdx, %rax + negq %rax + sbbq %rax, %rax + movq 0x150(%rsp), %rsi + movq 0x60(%rsi), %r12 + movq 0x68(%rsi), %r13 + movq 0x70(%rsi), %r14 + movq 0x78(%rsi), %r15 + movq 0x80(%rsi), %rdx + movq 0x88(%rsi), %rcx + cmoveq %r12, %r8 + cmoveq %r13, %r9 + cmoveq %r14, %r10 + cmoveq %r15, %r11 + cmoveq %rdx, %rbx + cmoveq %rcx, %rbp + orq %r13, %r12 + orq %r15, %r14 + orq %rcx, %rdx + orq %r14, %r12 + orq %r12, %rdx + negq %rdx + sbbq %rdx, %rdx + cmpq %rdx, %rax + cmoveq 0xf0(%rsp), %r8 + cmoveq 0xf8(%rsp), %r9 + cmoveq 0x100(%rsp), %r10 + cmoveq 0x108(%rsp), %r11 + cmoveq 0x110(%rsp), %rbx + cmoveq 0x118(%rsp), %rbp + movq %r8, 0xf0(%rsp) + movq %r9, 0xf8(%rsp) + movq %r10, 0x100(%rsp) + movq %r11, 0x108(%rsp) + movq %rbx, 0x110(%rsp) + movq %rbp, 0x118(%rsp) + movq 0x158(%rsp), %rcx + movq 0x150(%rsp), %rsi + movq (%rsp), %r8 + cmovbq (%rsi), %r8 + cmova (%rcx), %r8 + movq 0x8(%rsp), %r9 + cmovbq 0x8(%rsi), %r9 + cmova 0x8(%rcx), %r9 + movq 0x10(%rsp), %r10 + cmovbq 0x10(%rsi), %r10 + cmova 0x10(%rcx), %r10 + movq 0x18(%rsp), %r11 + cmovbq 0x18(%rsi), %r11 + cmova 0x18(%rcx), %r11 + movq 0x20(%rsp), %rbx + cmovbq 0x20(%rsi), %rbx + cmova 0x20(%rcx), %rbx + movq 0x28(%rsp), %rbp + cmovbq 0x28(%rsi), %rbp + cmova 0x28(%rcx), %rbp + movq 0xc0(%rsp), %r12 + cmovbq 0x30(%rsi), %r12 + cmova 0x30(%rcx), %r12 + movq 0xc8(%rsp), %r13 + cmovbq 0x38(%rsi), %r13 + cmova 0x38(%rcx), %r13 + movq 0xd0(%rsp), %r14 + cmovbq 0x40(%rsi), %r14 + cmova 0x40(%rcx), %r14 + movq 0xd8(%rsp), %r15 + cmovbq 0x48(%rsi), %r15 + cmova 0x48(%rcx), %r15 + movq 0xe0(%rsp), %rdx + cmovbq 0x50(%rsi), %rdx + cmova 0x50(%rcx), %rdx + movq 0xe8(%rsp), %rax + cmovbq 0x58(%rsi), %rax + cmova 0x58(%rcx), %rax + movq %r8, (%rdi) + movq %r9, 0x8(%rdi) + movq %r10, 0x10(%rdi) + movq %r11, 0x18(%rdi) + movq %rbx, 0x20(%rdi) + movq %rbp, 0x28(%rdi) + movq 0xf0(%rsp), %r8 + movq 0xf8(%rsp), %r9 + movq 0x100(%rsp), %r10 + movq 0x108(%rsp), %r11 + movq 0x110(%rsp), %rbx + movq 0x118(%rsp), %rbp + movq %r12, 0x30(%rdi) + movq %r13, 0x38(%rdi) + movq %r14, 0x40(%rdi) + movq %r15, 0x48(%rdi) + movq %rdx, 0x50(%rdi) + movq %rax, 0x58(%rdi) + movq %r8, 0x60(%rdi) + movq %r9, 0x68(%rdi) + movq %r10, 0x70(%rdi) + movq %r11, 0x78(%rdi) + movq %rbx, 0x80(%rdi) + movq %rbp, 0x88(%rdi) + CFI_INC_RSP(352) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp384_montjscalarmul_alt_p384_montjadd) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp384_montjscalarmul_alt_p384_montjdouble) + +Lp384_montjscalarmul_alt_p384_montjdouble: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(344) + movq %rdi, 0x150(%rsp) + movq 0x60(%rsi), %rbx + movq 0x68(%rsi), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x78(%rsi), %rax + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x88(%rsi), %rax + mulq %rbx + movq %rax, %r13 + movq %rdx, %r14 + movq 0x78(%rsi), %rax + mulq 0x80(%rsi) + movq %rax, %r15 + movq %rdx, %rcx + movq 0x70(%rsi), %rbx + movq 0x60(%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x68(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rbp, %rbp + movq 0x68(%rsi), %rbx + movq 0x78(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x80(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x88(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %rcx + movq 0x80(%rsi), %rbx + movq 0x60(%rsi), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x70(%rsi), %rbx + movq 0x78(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x80(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x88(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r15 + adcq %rdx, %rcx + sbbq %rbp, %rbp + xorl %ebx, %ebx + movq 0x78(%rsi), %rax + mulq 0x88(%rsi) + subq %rbp, %rdx + xorl %ebp, %ebp + addq %rax, %rcx + adcq %rdx, %rbx + adcl %ebp, %ebp + movq 0x80(%rsi), %rax + mulq 0x88(%rsi) + addq %rax, %rbx + adcq %rdx, %rbp + xorl %r8d, %r8d + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %r15, %r15 + adcq %rcx, %rcx + adcq %rbx, %rbx + adcq %rbp, %rbp + adcl %r8d, %r8d + movq 0x60(%rsi), %rax + mulq %rax + movq %r8, (%rsp) + movq %rax, %r8 + movq 0x68(%rsi), %rax + movq %rbp, 0x8(%rsp) + addq %rdx, %r9 + sbbq %rbp, %rbp + mulq %rax + negq %rbp + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x70(%rsi), %rax + mulq %rax + negq %rbp + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x78(%rsi), %rax + mulq %rax + negq %rbp + adcq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x80(%rsi), %rax + mulq %rax + negq %rbp + adcq %rax, %rcx + adcq %rdx, %rbx + sbbq %rbp, %rbp + movq 0x88(%rsi), %rax + mulq %rax + negq %rbp + adcq 0x8(%rsp), %rax + adcq (%rsp), %rdx + movq %rax, %rbp + movq %rdx, %rdi + movq %rbx, (%rsp) + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r8 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r8, %r9 + sbbq %rdx, %r10 + sbbq %rax, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rbx, %r8 + sbbq $0x0, %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r9 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r9, %r10 + sbbq %rdx, %r11 + sbbq %rax, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rbx, %r9 + sbbq $0x0, %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r10 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r10, %r11 + sbbq %rdx, %r12 + sbbq %rax, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rbx, %r10 + sbbq $0x0, %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r11 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r11, %r12 + sbbq %rdx, %r13 + sbbq %rax, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rbx, %r11 + sbbq $0x0, %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r12 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r12, %r13 + sbbq %rdx, %r8 + sbbq %rax, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rbx, %r12 + sbbq $0x0, %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r13 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r13, %r8 + sbbq %rdx, %r9 + sbbq %rax, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rbx, %r13 + sbbq $0x0, %r13 + movq (%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rdi + movl $0x0, %r8d + adcq %r8, %r8 + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %r9d + adcq %r15, %r9 + movl $0x1, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rdi, %r13 + adcq $0x0, %r8 + cmovneq %rax, %r14 + cmovneq %r9, %r15 + cmovneq %r10, %rcx + cmovneq %r11, %rbx + cmovneq %r12, %rbp + cmovneq %r13, %rdi + movq %r14, (%rsp) + movq %r15, 0x8(%rsp) + movq %rcx, 0x10(%rsp) + movq %rbx, 0x18(%rsp) + movq %rbp, 0x20(%rsp) + movq %rdi, 0x28(%rsp) + movq 0x30(%rsi), %rbx + movq 0x38(%rsi), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x48(%rsi), %rax + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x58(%rsi), %rax + mulq %rbx + movq %rax, %r13 + movq %rdx, %r14 + movq 0x48(%rsi), %rax + mulq 0x50(%rsi) + movq %rax, %r15 + movq %rdx, %rcx + movq 0x40(%rsi), %rbx + movq 0x30(%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x38(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rbp, %rbp + movq 0x38(%rsi), %rbx + movq 0x48(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x50(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x58(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %rcx + movq 0x50(%rsi), %rbx + movq 0x30(%rsi), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x40(%rsi), %rbx + movq 0x48(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x50(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x58(%rsi), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r15 + adcq %rdx, %rcx + sbbq %rbp, %rbp + xorl %ebx, %ebx + movq 0x48(%rsi), %rax + mulq 0x58(%rsi) + subq %rbp, %rdx + xorl %ebp, %ebp + addq %rax, %rcx + adcq %rdx, %rbx + adcl %ebp, %ebp + movq 0x50(%rsi), %rax + mulq 0x58(%rsi) + addq %rax, %rbx + adcq %rdx, %rbp + xorl %r8d, %r8d + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %r15, %r15 + adcq %rcx, %rcx + adcq %rbx, %rbx + adcq %rbp, %rbp + adcl %r8d, %r8d + movq 0x30(%rsi), %rax + mulq %rax + movq %r8, 0x30(%rsp) + movq %rax, %r8 + movq 0x38(%rsi), %rax + movq %rbp, 0x38(%rsp) + addq %rdx, %r9 + sbbq %rbp, %rbp + mulq %rax + negq %rbp + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x40(%rsi), %rax + mulq %rax + negq %rbp + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x48(%rsi), %rax + mulq %rax + negq %rbp + adcq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x50(%rsi), %rax + mulq %rax + negq %rbp + adcq %rax, %rcx + adcq %rdx, %rbx + sbbq %rbp, %rbp + movq 0x58(%rsi), %rax + mulq %rax + negq %rbp + adcq 0x38(%rsp), %rax + adcq 0x30(%rsp), %rdx + movq %rax, %rbp + movq %rdx, %rdi + movq %rbx, 0x30(%rsp) + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r8 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r8, %r9 + sbbq %rdx, %r10 + sbbq %rax, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rbx, %r8 + sbbq $0x0, %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r9 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r9, %r10 + sbbq %rdx, %r11 + sbbq %rax, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rbx, %r9 + sbbq $0x0, %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r10 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r10, %r11 + sbbq %rdx, %r12 + sbbq %rax, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rbx, %r10 + sbbq $0x0, %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r11 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r11, %r12 + sbbq %rdx, %r13 + sbbq %rax, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rbx, %r11 + sbbq $0x0, %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r12 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r12, %r13 + sbbq %rdx, %r8 + sbbq %rax, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rbx, %r12 + sbbq $0x0, %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r13 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r13, %r8 + sbbq %rdx, %r9 + sbbq %rax, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rbx, %r13 + sbbq $0x0, %r13 + movq 0x30(%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rdi + movl $0x0, %r8d + adcq %r8, %r8 + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %r9d + adcq %r15, %r9 + movl $0x1, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rdi, %r13 + adcq $0x0, %r8 + cmovneq %rax, %r14 + cmovneq %r9, %r15 + cmovneq %r10, %rcx + cmovneq %r11, %rbx + cmovneq %r12, %rbp + cmovneq %r13, %rdi + movq %r14, 0x30(%rsp) + movq %r15, 0x38(%rsp) + movq %rcx, 0x40(%rsp) + movq %rbx, 0x48(%rsp) + movq %rbp, 0x50(%rsp) + movq %rdi, 0x58(%rsp) + movq (%rsi), %rax + addq (%rsp), %rax + movq 0x8(%rsi), %rcx + adcq 0x8(%rsp), %rcx + movq 0x10(%rsi), %r8 + adcq 0x10(%rsp), %r8 + movq 0x18(%rsi), %r9 + adcq 0x18(%rsp), %r9 + movq 0x20(%rsi), %r10 + adcq 0x20(%rsp), %r10 + movq 0x28(%rsi), %r11 + adcq 0x28(%rsp), %r11 + sbbq %rdx, %rdx + movl $0x1, %ebx + andq %rdx, %rbx + movl $0xffffffff, %ebp + andq %rbp, %rdx + xorq %rbp, %rbp + subq %rdx, %rbp + addq %rbp, %rax + movq %rax, 0xf0(%rsp) + adcq %rdx, %rcx + movq %rcx, 0xf8(%rsp) + adcq %rbx, %r8 + movq %r8, 0x100(%rsp) + adcq $0x0, %r9 + movq %r9, 0x108(%rsp) + adcq $0x0, %r10 + movq %r10, 0x110(%rsp) + adcq $0x0, %r11 + movq %r11, 0x118(%rsp) + movq (%rsi), %rax + subq (%rsp), %rax + movq 0x8(%rsi), %rdx + sbbq 0x8(%rsp), %rdx + movq 0x10(%rsi), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x18(%rsi), %r9 + sbbq 0x18(%rsp), %r9 + movq 0x20(%rsi), %r10 + sbbq 0x20(%rsp), %r10 + movq 0x28(%rsi), %r11 + sbbq 0x28(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %ebx + andq %rbx, %rcx + xorq %rbx, %rbx + subq %rcx, %rbx + subq %rbx, %rax + movq %rax, 0xc0(%rsp) + sbbq %rcx, %rdx + movq %rdx, 0xc8(%rsp) + sbbq %rax, %rax + andq %rbx, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0xd0(%rsp) + sbbq $0x0, %r9 + movq %r9, 0xd8(%rsp) + sbbq $0x0, %r10 + movq %r10, 0xe0(%rsp) + sbbq $0x0, %r11 + movq %r11, 0xe8(%rsp) + movq 0xc0(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0xf8(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x100(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x108(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x110(%rsp), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x118(%rsp), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r8, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rdx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rbx + addq %rbx, %r14 + adcq $0x0, %r15 + movq 0xc8(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r8, %r8 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r8, %r8 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r8, %r8 + negq %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r9, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rdx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rbx + addq %rbx, %r15 + adcq $0x0, %r8 + movq 0xd0(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r9, %r9 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r10, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rdx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rbx + addq %rbx, %r8 + adcq $0x0, %r9 + movq 0xd8(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r10, %r10 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r10, %r10 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r10, %r10 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r10, %r10 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r10, %r10 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r11, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rbx + addq %rbx, %r9 + adcq $0x0, %r10 + movq 0xe0(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r11, %r11 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r11, %r11 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r11, %r11 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r11, %r11 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r11, %r11 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r11, %r11 + negq %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r12, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rdx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rbx + addq %rbx, %r10 + adcq $0x0, %r11 + movq 0xe8(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r12, %r12 + movq 0xf8(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r12, %r12 + movq 0x100(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r12, %r12 + movq 0x108(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r12, %r12 + movq 0x110(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r12, %r12 + movq 0x118(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r12, %r12 + negq %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r13, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rdx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rbx + addq %rbx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x60(%rsp) + movq %r15, 0x68(%rsp) + movq %r8, 0x70(%rsp) + movq %r9, 0x78(%rsp) + movq %r10, 0x80(%rsp) + movq %r11, 0x88(%rsp) + movq 0x30(%rsi), %rax + addq 0x60(%rsi), %rax + movq 0x38(%rsi), %rcx + adcq 0x68(%rsi), %rcx + movq 0x40(%rsi), %r8 + adcq 0x70(%rsi), %r8 + movq 0x48(%rsi), %r9 + adcq 0x78(%rsi), %r9 + movq 0x50(%rsi), %r10 + adcq 0x80(%rsi), %r10 + movq 0x58(%rsi), %r11 + adcq 0x88(%rsi), %r11 + movl $0x0, %edx + adcq %rdx, %rdx + movabsq $0xffffffff00000001, %rbp + addq %rbp, %rax + movl $0xffffffff, %ebp + adcq %rbp, %rcx + adcq $0x1, %r8 + adcq $0x0, %r9 + adcq $0x0, %r10 + adcq $0x0, %r11 + adcq $0xffffffffffffffff, %rdx + movl $0x1, %ebx + andq %rdx, %rbx + andq %rbp, %rdx + xorq %rbp, %rbp + subq %rdx, %rbp + subq %rbp, %rax + movq %rax, 0xf0(%rsp) + sbbq %rdx, %rcx + movq %rcx, 0xf8(%rsp) + sbbq %rbx, %r8 + movq %r8, 0x100(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x108(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x110(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x118(%rsp) + movq 0x60(%rsp), %rbx + movq 0x68(%rsp), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x78(%rsp), %rax + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x88(%rsp), %rax + mulq %rbx + movq %rax, %r13 + movq %rdx, %r14 + movq 0x78(%rsp), %rax + mulq 0x80(%rsp) + movq %rax, %r15 + movq %rdx, %rcx + movq 0x70(%rsp), %rbx + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x68(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rbp, %rbp + movq 0x68(%rsp), %rbx + movq 0x78(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x80(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x88(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %rcx + movq 0x80(%rsp), %rbx + movq 0x60(%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x70(%rsp), %rbx + movq 0x78(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x80(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x88(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r15 + adcq %rdx, %rcx + sbbq %rbp, %rbp + xorl %ebx, %ebx + movq 0x78(%rsp), %rax + mulq 0x88(%rsp) + subq %rbp, %rdx + xorl %ebp, %ebp + addq %rax, %rcx + adcq %rdx, %rbx + adcl %ebp, %ebp + movq 0x80(%rsp), %rax + mulq 0x88(%rsp) + addq %rax, %rbx + adcq %rdx, %rbp + xorl %r8d, %r8d + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %r15, %r15 + adcq %rcx, %rcx + adcq %rbx, %rbx + adcq %rbp, %rbp + adcl %r8d, %r8d + movq 0x60(%rsp), %rax + mulq %rax + movq %r8, 0x120(%rsp) + movq %rax, %r8 + movq 0x68(%rsp), %rax + movq %rbp, 0x128(%rsp) + addq %rdx, %r9 + sbbq %rbp, %rbp + mulq %rax + negq %rbp + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x70(%rsp), %rax + mulq %rax + negq %rbp + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x78(%rsp), %rax + mulq %rax + negq %rbp + adcq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x80(%rsp), %rax + mulq %rax + negq %rbp + adcq %rax, %rcx + adcq %rdx, %rbx + sbbq %rbp, %rbp + movq 0x88(%rsp), %rax + mulq %rax + negq %rbp + adcq 0x128(%rsp), %rax + adcq 0x120(%rsp), %rdx + movq %rax, %rbp + movq %rdx, %rdi + movq %rbx, 0x120(%rsp) + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r8 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r8, %r9 + sbbq %rdx, %r10 + sbbq %rax, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rbx, %r8 + sbbq $0x0, %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r9 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r9, %r10 + sbbq %rdx, %r11 + sbbq %rax, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rbx, %r9 + sbbq $0x0, %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r10 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r10, %r11 + sbbq %rdx, %r12 + sbbq %rax, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rbx, %r10 + sbbq $0x0, %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r11 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r11, %r12 + sbbq %rdx, %r13 + sbbq %rax, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rbx, %r11 + sbbq $0x0, %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r12 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r12, %r13 + sbbq %rdx, %r8 + sbbq %rax, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rbx, %r12 + sbbq $0x0, %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r13 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r13, %r8 + sbbq %rdx, %r9 + sbbq %rax, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rbx, %r13 + sbbq $0x0, %r13 + movq 0x120(%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rdi + movl $0x0, %r8d + adcq %r8, %r8 + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %r9d + adcq %r15, %r9 + movl $0x1, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rdi, %r13 + adcq $0x0, %r8 + cmovneq %rax, %r14 + cmovneq %r9, %r15 + cmovneq %r10, %rcx + cmovneq %r11, %rbx + cmovneq %r12, %rbp + cmovneq %r13, %rdi + movq %r14, 0x120(%rsp) + movq %r15, 0x128(%rsp) + movq %rcx, 0x130(%rsp) + movq %rbx, 0x138(%rsp) + movq %rbp, 0x140(%rsp) + movq %rdi, 0x148(%rsp) + movq 0x30(%rsp), %rbx + movq (%rsi), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x8(%rsi), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x10(%rsi), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x18(%rsi), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x20(%rsi), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x28(%rsi), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r8, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rdx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rbx + addq %rbx, %r14 + adcq $0x0, %r15 + movq 0x38(%rsp), %rbx + movq (%rsi), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r8, %r8 + movq 0x8(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x10(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x18(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x20(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r8, %r8 + movq 0x28(%rsi), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r8, %r8 + negq %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r9, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rdx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rbx + addq %rbx, %r15 + adcq $0x0, %r8 + movq 0x40(%rsp), %rbx + movq (%rsi), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r9, %r9 + movq 0x8(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x10(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x18(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x20(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + movq 0x28(%rsi), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r10, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rdx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rbx + addq %rbx, %r8 + adcq $0x0, %r9 + movq 0x48(%rsp), %rbx + movq (%rsi), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r10, %r10 + movq 0x8(%rsi), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r10, %r10 + movq 0x10(%rsi), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r10, %r10 + movq 0x18(%rsi), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r10, %r10 + movq 0x20(%rsi), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r10, %r10 + movq 0x28(%rsi), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r11, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rbx + addq %rbx, %r9 + adcq $0x0, %r10 + movq 0x50(%rsp), %rbx + movq (%rsi), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r11, %r11 + movq 0x8(%rsi), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r11, %r11 + movq 0x10(%rsi), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r11, %r11 + movq 0x18(%rsi), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r11, %r11 + movq 0x20(%rsi), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r11, %r11 + movq 0x28(%rsi), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r11, %r11 + negq %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r12, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rdx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rbx + addq %rbx, %r10 + adcq $0x0, %r11 + movq 0x58(%rsp), %rbx + movq (%rsi), %rax + mulq %rbx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r12, %r12 + movq 0x8(%rsi), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r12, %r12 + movq 0x10(%rsi), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r12, %r12 + movq 0x18(%rsi), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r12, %r12 + movq 0x20(%rsi), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r12, %r12 + movq 0x28(%rsi), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r12, %r12 + negq %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r13, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rdx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rbx + addq %rbx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0x90(%rsp) + movq %r15, 0x98(%rsp) + movq %r8, 0xa0(%rsp) + movq %r9, 0xa8(%rsp) + movq %r10, 0xb0(%rsp) + movq %r11, 0xb8(%rsp) + movq 0xf0(%rsp), %rbx + movq 0xf8(%rsp), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x108(%rsp), %rax + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x118(%rsp), %rax + mulq %rbx + movq %rax, %r13 + movq %rdx, %r14 + movq 0x108(%rsp), %rax + mulq 0x110(%rsp) + movq %rax, %r15 + movq %rdx, %rcx + movq 0x100(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0xf8(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rbp, %rbp + movq 0xf8(%rsp), %rbx + movq 0x108(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x110(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x118(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %rcx + movq 0x110(%rsp), %rbx + movq 0xf0(%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x100(%rsp), %rbx + movq 0x108(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x110(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x118(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r15 + adcq %rdx, %rcx + sbbq %rbp, %rbp + xorl %ebx, %ebx + movq 0x108(%rsp), %rax + mulq 0x118(%rsp) + subq %rbp, %rdx + xorl %ebp, %ebp + addq %rax, %rcx + adcq %rdx, %rbx + adcl %ebp, %ebp + movq 0x110(%rsp), %rax + mulq 0x118(%rsp) + addq %rax, %rbx + adcq %rdx, %rbp + xorl %r8d, %r8d + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %r15, %r15 + adcq %rcx, %rcx + adcq %rbx, %rbx + adcq %rbp, %rbp + adcl %r8d, %r8d + movq 0xf0(%rsp), %rax + mulq %rax + movq %r8, 0xc0(%rsp) + movq %rax, %r8 + movq 0xf8(%rsp), %rax + movq %rbp, 0xc8(%rsp) + addq %rdx, %r9 + sbbq %rbp, %rbp + mulq %rax + negq %rbp + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x100(%rsp), %rax + mulq %rax + negq %rbp + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x108(%rsp), %rax + mulq %rax + negq %rbp + adcq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x110(%rsp), %rax + mulq %rax + negq %rbp + adcq %rax, %rcx + adcq %rdx, %rbx + sbbq %rbp, %rbp + movq 0x118(%rsp), %rax + mulq %rax + negq %rbp + adcq 0xc8(%rsp), %rax + adcq 0xc0(%rsp), %rdx + movq %rax, %rbp + movq %rdx, %rdi + movq %rbx, 0xc0(%rsp) + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r8 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r8, %r9 + sbbq %rdx, %r10 + sbbq %rax, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rbx, %r8 + sbbq $0x0, %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r9 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r9, %r10 + sbbq %rdx, %r11 + sbbq %rax, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rbx, %r9 + sbbq $0x0, %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r10 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r10, %r11 + sbbq %rdx, %r12 + sbbq %rax, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rbx, %r10 + sbbq $0x0, %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r11 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r11, %r12 + sbbq %rdx, %r13 + sbbq %rax, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rbx, %r11 + sbbq $0x0, %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r12 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r12, %r13 + sbbq %rdx, %r8 + sbbq %rax, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rbx, %r12 + sbbq $0x0, %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r13 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r13, %r8 + sbbq %rdx, %r9 + sbbq %rax, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rbx, %r13 + sbbq $0x0, %r13 + movq 0xc0(%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rdi + movl $0x0, %r8d + adcq %r8, %r8 + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %r9d + adcq %r15, %r9 + movl $0x1, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rdi, %r13 + adcq $0x0, %r8 + cmovneq %rax, %r14 + cmovneq %r9, %r15 + cmovneq %r10, %rcx + cmovneq %r11, %rbx + cmovneq %r12, %rbp + cmovneq %r13, %rdi + movq %r14, 0xc0(%rsp) + movq %r15, 0xc8(%rsp) + movq %rcx, 0xd0(%rsp) + movq %rbx, 0xd8(%rsp) + movq %rbp, 0xe0(%rsp) + movq %rdi, 0xe8(%rsp) + movabsq $0xffffffff, %r9 + subq 0x120(%rsp), %r9 + movabsq $0xffffffff00000000, %r10 + sbbq 0x128(%rsp), %r10 + movq $0xfffffffffffffffe, %r11 + sbbq 0x130(%rsp), %r11 + movq $0xffffffffffffffff, %r12 + sbbq 0x138(%rsp), %r12 + movq $0xffffffffffffffff, %r13 + sbbq 0x140(%rsp), %r13 + movq $0xffffffffffffffff, %r14 + sbbq 0x148(%rsp), %r14 + movq $0x9, %rcx + movq %r9, %rax + mulq %rcx + movq %rax, %r8 + movq %rdx, %r9 + movq %r10, %rax + xorl %r10d, %r10d + mulq %rcx + addq %rax, %r9 + adcq %rdx, %r10 + movq %r11, %rax + xorl %r11d, %r11d + mulq %rcx + addq %rax, %r10 + adcq %rdx, %r11 + movq %r12, %rax + xorl %r12d, %r12d + mulq %rcx + addq %rax, %r11 + adcq %rdx, %r12 + movq %r13, %rax + xorl %r13d, %r13d + mulq %rcx + addq %rax, %r12 + adcq %rdx, %r13 + movq %r14, %rax + movl $0x1, %r14d + mulq %rcx + addq %rax, %r13 + adcq %rdx, %r14 + movl $0xc, %ecx + movq 0x90(%rsp), %rax + mulq %rcx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %rbx, %rbx + movq 0x98(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rbx, %rbx + movq 0xa0(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbx, %rbx + movq 0xa8(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rbx, %rbx + movq 0xb0(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbx, %rbx + movq 0xb8(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + movabsq $0xffffffff00000001, %rax + mulq %r14 + addq %rax, %r8 + adcq %rdx, %r9 + adcq %r14, %r10 + movq %r14, %rax + sbbq %rcx, %rcx + movl $0xffffffff, %edx + negq %rcx + mulq %rdx + addq %rax, %r9 + adcq %rdx, %r10 + adcq %rcx, %r11 + adcq $0x0, %r12 + adcq $0x0, %r13 + sbbq %rcx, %rcx + notq %rcx + movl $0xffffffff, %edx + xorq %rax, %rax + andq %rcx, %rdx + subq %rdx, %rax + andq $0x1, %rcx + subq %rax, %r8 + movq %r8, 0x120(%rsp) + sbbq %rdx, %r9 + movq %r9, 0x128(%rsp) + sbbq %rcx, %r10 + movq %r10, 0x130(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x138(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x140(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x148(%rsp) + movq 0xc0(%rsp), %rax + subq (%rsp), %rax + movq 0xc8(%rsp), %rdx + sbbq 0x8(%rsp), %rdx + movq 0xd0(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0xd8(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movq 0xe0(%rsp), %r10 + sbbq 0x20(%rsp), %r10 + movq 0xe8(%rsp), %r11 + sbbq 0x28(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %ebx + andq %rbx, %rcx + xorq %rbx, %rbx + subq %rcx, %rbx + subq %rbx, %rax + movq %rax, 0xf0(%rsp) + sbbq %rcx, %rdx + movq %rdx, 0xf8(%rsp) + sbbq %rax, %rax + andq %rbx, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0x100(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x108(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x110(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x118(%rsp) + movq 0x30(%rsp), %rbx + movq 0x38(%rsp), %rax + mulq %rbx + movq %rax, %r9 + movq %rdx, %r10 + movq 0x48(%rsp), %rax + mulq %rbx + movq %rax, %r11 + movq %rdx, %r12 + movq 0x58(%rsp), %rax + mulq %rbx + movq %rax, %r13 + movq %rdx, %r14 + movq 0x48(%rsp), %rax + mulq 0x50(%rsp) + movq %rax, %r15 + movq %rdx, %rcx + movq 0x40(%rsp), %rbx + movq 0x30(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x38(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rbp, %rbp + movq 0x38(%rsp), %rbx + movq 0x48(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x50(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x58(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %rcx + movq 0x50(%rsp), %rbx + movq 0x30(%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x40(%rsp), %rbx + movq 0x48(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x50(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x58(%rsp), %rax + mulq %rbx + subq %rbp, %rdx + addq %rax, %r15 + adcq %rdx, %rcx + sbbq %rbp, %rbp + xorl %ebx, %ebx + movq 0x48(%rsp), %rax + mulq 0x58(%rsp) + subq %rbp, %rdx + xorl %ebp, %ebp + addq %rax, %rcx + adcq %rdx, %rbx + adcl %ebp, %ebp + movq 0x50(%rsp), %rax + mulq 0x58(%rsp) + addq %rax, %rbx + adcq %rdx, %rbp + xorl %r8d, %r8d + addq %r9, %r9 + adcq %r10, %r10 + adcq %r11, %r11 + adcq %r12, %r12 + adcq %r13, %r13 + adcq %r14, %r14 + adcq %r15, %r15 + adcq %rcx, %rcx + adcq %rbx, %rbx + adcq %rbp, %rbp + adcl %r8d, %r8d + movq 0x30(%rsp), %rax + mulq %rax + movq %r8, 0xc0(%rsp) + movq %rax, %r8 + movq 0x38(%rsp), %rax + movq %rbp, 0xc8(%rsp) + addq %rdx, %r9 + sbbq %rbp, %rbp + mulq %rax + negq %rbp + adcq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x40(%rsp), %rax + mulq %rax + negq %rbp + adcq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x48(%rsp), %rax + mulq %rax + negq %rbp + adcq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x50(%rsp), %rax + mulq %rax + negq %rbp + adcq %rax, %rcx + adcq %rdx, %rbx + sbbq %rbp, %rbp + movq 0x58(%rsp), %rax + mulq %rax + negq %rbp + adcq 0xc8(%rsp), %rax + adcq 0xc0(%rsp), %rdx + movq %rax, %rbp + movq %rdx, %rdi + movq %rbx, 0xc0(%rsp) + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r8 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r8, %r9 + sbbq %rdx, %r10 + sbbq %rax, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + movq %rbx, %r8 + sbbq $0x0, %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r9 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r9, %r10 + sbbq %rdx, %r11 + sbbq %rax, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r8 + movq %rbx, %r9 + sbbq $0x0, %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r10 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r10, %r11 + sbbq %rdx, %r12 + sbbq %rax, %r13 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + movq %rbx, %r10 + sbbq $0x0, %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r11 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r11, %r12 + sbbq %rdx, %r13 + sbbq %rax, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + movq %rbx, %r11 + sbbq $0x0, %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r12 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r12, %r13 + sbbq %rdx, %r8 + sbbq %rax, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %r11 + movq %rbx, %r12 + sbbq $0x0, %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %rax, %r13 + movl $0x0, %eax + adcq %rbx, %rdx + adcl %eax, %eax + subq %r13, %r8 + sbbq %rdx, %r9 + sbbq %rax, %r10 + sbbq $0x0, %r11 + sbbq $0x0, %r12 + movq %rbx, %r13 + sbbq $0x0, %r13 + movq 0xc0(%rsp), %rbx + addq %r8, %r14 + adcq %r9, %r15 + adcq %r10, %rcx + adcq %r11, %rbx + adcq %r12, %rbp + adcq %r13, %rdi + movl $0x0, %r8d + adcq %r8, %r8 + xorq %r11, %r11 + xorq %r12, %r12 + xorq %r13, %r13 + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %r9d + adcq %r15, %r9 + movl $0x1, %r10d + adcq %rcx, %r10 + adcq %rbx, %r11 + adcq %rbp, %r12 + adcq %rdi, %r13 + adcq $0x0, %r8 + cmovneq %rax, %r14 + cmovneq %r9, %r15 + cmovneq %r10, %rcx + cmovneq %r11, %rbx + cmovneq %r12, %rbp + cmovneq %r13, %rdi + movq %r14, 0xc0(%rsp) + movq %r15, 0xc8(%rsp) + movq %rcx, 0xd0(%rsp) + movq %rbx, 0xd8(%rsp) + movq %rbp, 0xe0(%rsp) + movq %rdi, 0xe8(%rsp) + movq 0x150(%rsp), %rdi + movq 0xf0(%rsp), %rax + subq 0x30(%rsp), %rax + movq 0xf8(%rsp), %rdx + sbbq 0x38(%rsp), %rdx + movq 0x100(%rsp), %r8 + sbbq 0x40(%rsp), %r8 + movq 0x108(%rsp), %r9 + sbbq 0x48(%rsp), %r9 + movq 0x110(%rsp), %r10 + sbbq 0x50(%rsp), %r10 + movq 0x118(%rsp), %r11 + sbbq 0x58(%rsp), %r11 + sbbq %rcx, %rcx + movl $0xffffffff, %ebx + andq %rbx, %rcx + xorq %rbx, %rbx + subq %rcx, %rbx + subq %rbx, %rax + movq %rax, 0x60(%rdi) + sbbq %rcx, %rdx + movq %rdx, 0x68(%rdi) + sbbq %rax, %rax + andq %rbx, %rcx + negq %rax + sbbq %rcx, %r8 + movq %r8, 0x70(%rdi) + sbbq $0x0, %r9 + movq %r9, 0x78(%rdi) + sbbq $0x0, %r10 + movq %r10, 0x80(%rdi) + sbbq $0x0, %r11 + movq %r11, 0x88(%rdi) + movq 0x60(%rsp), %rbx + movq 0x120(%rsp), %rax + mulq %rbx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x128(%rsp), %rax + mulq %rbx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x130(%rsp), %rax + mulq %rbx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x138(%rsp), %rax + mulq %rbx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x140(%rsp), %rax + mulq %rbx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x148(%rsp), %rax + mulq %rbx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + xorl %r15d, %r15d + movq %r8, %rbx + shlq $0x20, %rbx + addq %r8, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r8 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r8, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r9 + sbbq %rdx, %r10 + sbbq %rbp, %r11 + sbbq $0x0, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %rbx + addq %rbx, %r14 + adcq $0x0, %r15 + movq 0x68(%rsp), %rbx + movq 0x120(%rsp), %rax + mulq %rbx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r8, %r8 + movq 0x128(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r8, %r8 + movq 0x130(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r8, %r8 + movq 0x138(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r8, %r8 + movq 0x140(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r8, %r8 + movq 0x148(%rsp), %rax + mulq %rbx + subq %r8, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r8, %r8 + negq %r8 + movq %r9, %rbx + shlq $0x20, %rbx + addq %r9, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r9 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r9, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r10 + sbbq %rdx, %r11 + sbbq %rbp, %r12 + sbbq $0x0, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %rbx + addq %rbx, %r15 + adcq $0x0, %r8 + movq 0x70(%rsp), %rbx + movq 0x120(%rsp), %rax + mulq %rbx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r9, %r9 + movq 0x128(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r9, %r9 + movq 0x130(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r9, %r9 + movq 0x138(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r9, %r9 + movq 0x140(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r9, %r9 + movq 0x148(%rsp), %rax + mulq %rbx + subq %r9, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r9, %r9 + negq %r9 + movq %r10, %rbx + shlq $0x20, %rbx + addq %r10, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r10 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r10, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r11 + sbbq %rdx, %r12 + sbbq %rbp, %r13 + sbbq $0x0, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %rbx + addq %rbx, %r8 + adcq $0x0, %r9 + movq 0x78(%rsp), %rbx + movq 0x120(%rsp), %rax + mulq %rbx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %r10, %r10 + movq 0x128(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r10, %r10 + movq 0x130(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r10, %r10 + movq 0x138(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r10, %r10 + movq 0x140(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r10, %r10 + movq 0x148(%rsp), %rax + mulq %rbx + subq %r10, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r10, %r10 + negq %r10 + movq %r11, %rbx + shlq $0x20, %rbx + addq %r11, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r11 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r11, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r12 + sbbq %rdx, %r13 + sbbq %rbp, %r14 + sbbq $0x0, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %rbx + addq %rbx, %r9 + adcq $0x0, %r10 + movq 0x80(%rsp), %rbx + movq 0x120(%rsp), %rax + mulq %rbx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %r11, %r11 + movq 0x128(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r11, %r11 + movq 0x130(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r11, %r11 + movq 0x138(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r11, %r11 + movq 0x140(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r11, %r11 + movq 0x148(%rsp), %rax + mulq %rbx + subq %r11, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r11, %r11 + negq %r11 + movq %r12, %rbx + shlq $0x20, %rbx + addq %r12, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r12 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r12, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r13 + sbbq %rdx, %r14 + sbbq %rbp, %r15 + sbbq $0x0, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %rbx + addq %rbx, %r10 + adcq $0x0, %r11 + movq 0x88(%rsp), %rbx + movq 0x120(%rsp), %rax + mulq %rbx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %r12, %r12 + movq 0x128(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %r12, %r12 + movq 0x130(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + sbbq %r12, %r12 + movq 0x138(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %r12, %r12 + movq 0x140(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %r12, %r12 + movq 0x148(%rsp), %rax + mulq %rbx + subq %r12, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %r12, %r12 + negq %r12 + movq %r13, %rbx + shlq $0x20, %rbx + addq %r13, %rbx + xorl %ebp, %ebp + movabsq $0xffffffff00000001, %rax + mulq %rbx + movq %rdx, %r13 + movabsq $0xffffffff, %rax + mulq %rbx + addq %r13, %rax + adcq %rbx, %rdx + adcl %ebp, %ebp + subq %rax, %r14 + sbbq %rdx, %r15 + sbbq %rbp, %r8 + sbbq $0x0, %r9 + sbbq $0x0, %r10 + sbbq $0x0, %rbx + addq %rbx, %r11 + adcq $0x0, %r12 + xorl %edx, %edx + xorl %ebp, %ebp + xorl %r13d, %r13d + movabsq $0xffffffff00000001, %rax + addq %r14, %rax + movl $0xffffffff, %ebx + adcq %r15, %rbx + movl $0x1, %ecx + adcq %r8, %rcx + adcq %r9, %rdx + adcq %r10, %rbp + adcq %r11, %r13 + adcq $0x0, %r12 + cmovneq %rax, %r14 + cmovneq %rbx, %r15 + cmovneq %rcx, %r8 + cmovneq %rdx, %r9 + cmovneq %rbp, %r10 + cmovneq %r13, %r11 + movq %r14, 0xf0(%rsp) + movq %r15, 0xf8(%rsp) + movq %r8, 0x100(%rsp) + movq %r9, 0x108(%rsp) + movq %r10, 0x110(%rsp) + movq %r11, 0x118(%rsp) + movq 0xb8(%rsp), %rcx + movq %rcx, %r13 + shrq $0x3e, %rcx + movq 0xb0(%rsp), %r12 + shldq $0x2, %r12, %r13 + movq 0xa8(%rsp), %r11 + shldq $0x2, %r11, %r12 + movq 0xa0(%rsp), %r10 + shldq $0x2, %r10, %r11 + movq 0x98(%rsp), %r9 + shldq $0x2, %r9, %r10 + movq 0x90(%rsp), %r8 + shldq $0x2, %r8, %r9 + shlq $0x2, %r8 + addq $0x1, %rcx + subq 0x120(%rsp), %r8 + sbbq 0x128(%rsp), %r9 + sbbq 0x130(%rsp), %r10 + sbbq 0x138(%rsp), %r11 + sbbq 0x140(%rsp), %r12 + sbbq 0x148(%rsp), %r13 + sbbq $0x0, %rcx + movabsq $0xffffffff00000001, %rax + mulq %rcx + addq %rax, %r8 + adcq %rdx, %r9 + adcq %rcx, %r10 + movq %rcx, %rax + sbbq %rcx, %rcx + movl $0xffffffff, %edx + negq %rcx + mulq %rdx + addq %rax, %r9 + adcq %rdx, %r10 + adcq %rcx, %r11 + adcq $0x0, %r12 + adcq $0x0, %r13 + sbbq %rcx, %rcx + notq %rcx + movl $0xffffffff, %edx + xorq %rax, %rax + andq %rcx, %rdx + subq %rdx, %rax + andq $0x1, %rcx + subq %rax, %r8 + movq %r8, (%rdi) + sbbq %rdx, %r9 + movq %r9, 0x8(%rdi) + sbbq %rcx, %r10 + movq %r10, 0x10(%rdi) + sbbq $0x0, %r11 + movq %r11, 0x18(%rdi) + sbbq $0x0, %r12 + movq %r12, 0x20(%rdi) + sbbq $0x0, %r13 + movq %r13, 0x28(%rdi) + movabsq $0xffffffff, %r8 + subq 0xc0(%rsp), %r8 + movabsq $0xffffffff00000000, %r9 + sbbq 0xc8(%rsp), %r9 + movq $0xfffffffffffffffe, %r10 + sbbq 0xd0(%rsp), %r10 + movq $0xffffffffffffffff, %r11 + sbbq 0xd8(%rsp), %r11 + movq $0xffffffffffffffff, %r12 + sbbq 0xe0(%rsp), %r12 + movq $0xffffffffffffffff, %r13 + sbbq 0xe8(%rsp), %r13 + movq %r13, %r14 + shrq $0x3d, %r14 + shldq $0x3, %r12, %r13 + shldq $0x3, %r11, %r12 + shldq $0x3, %r10, %r11 + shldq $0x3, %r9, %r10 + shldq $0x3, %r8, %r9 + shlq $0x3, %r8 + addq $0x1, %r14 + movl $0x3, %ecx + movq 0xf0(%rsp), %rax + mulq %rcx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %rbx, %rbx + movq 0xf8(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rbx, %rbx + movq 0x100(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbx, %rbx + movq 0x108(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rbx, %rbx + movq 0x110(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbx, %rbx + movq 0x118(%rsp), %rax + mulq %rcx + subq %rbx, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + movabsq $0xffffffff00000001, %rax + mulq %r14 + addq %rax, %r8 + adcq %rdx, %r9 + adcq %r14, %r10 + movq %r14, %rax + sbbq %rcx, %rcx + movl $0xffffffff, %edx + negq %rcx + mulq %rdx + addq %rax, %r9 + adcq %rdx, %r10 + adcq %rcx, %r11 + adcq $0x0, %r12 + adcq $0x0, %r13 + sbbq %rcx, %rcx + notq %rcx + movl $0xffffffff, %edx + xorq %rax, %rax + andq %rcx, %rdx + subq %rdx, %rax + andq $0x1, %rcx + subq %rax, %r8 + movq %r8, 0x30(%rdi) + sbbq %rdx, %r9 + movq %r9, 0x38(%rdi) + sbbq %rcx, %r10 + movq %r10, 0x40(%rdi) + sbbq $0x0, %r11 + movq %r11, 0x48(%rdi) + sbbq $0x0, %r12 + movq %r12, 0x50(%rdi) + sbbq $0x0, %r13 + movq %r13, 0x58(%rdi) + CFI_INC_RSP(344) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp384_montjscalarmul_alt_p384_montjdouble) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/p521_jscalarmul.S b/cbits/s2n/x86_att/p521_jscalarmul.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/p521_jscalarmul.S @@ -0,0 +1,2505 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Jacobian form scalar multiplication for P-521 +// Input scalar[9], point[27]; output res[27] +// +// extern void p521_jscalarmul +// (uint64_t res[static 27], +// const uint64_t scalar[static 9], +// const uint64_t point[static 27]); +// +// This function is a variant of its affine point version p521_scalarmul. +// Here, input and output points are assumed to be in Jacobian form with +// a triple (x,y,z) representing the affine point (x/z^2,y/z^3) when +// z is nonzero or the point at infinity (group identity) if z = 0. +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-521, returns a representation of n * P. If the result is the +// point at infinity (either because the input point was or because the +// scalar was a multiple of p_521) then the output is guaranteed to +// represent the point at infinity, i.e. to have its z coordinate zero. +// +// Standard x86-64 ABI: RDI = res, RSI = scalar, RDX = point +// Microsoft x64 ABI: RCX = res, RDX = scalar, R8 = point +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p521_jscalarmul) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p521_jscalarmul) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p521_jscalarmul) + + + .text + .balign 32 + +// Size of individual field elements + +#define NUMSIZE 72 +#define JACSIZE (3*NUMSIZE) + +// Intermediate variables on the stack. +// The table is 16 entries, each of size JACSIZE = 3 * NUMSIZE +// Uppercase syntactic variants make x86_att version simpler to generate. + +#define SCALARB (0*NUMSIZE) +#define scalarb (0*NUMSIZE)(%rsp) +#define ACC (1*NUMSIZE) +#define acc (1*NUMSIZE)(%rsp) +#define TABENT (4*NUMSIZE) +#define tabent (4*NUMSIZE)(%rsp) + +#define TAB (7*NUMSIZE) +#define tab (7*NUMSIZE)(%rsp) + +#define res (55*NUMSIZE)(%rsp) + +#define NSPACE 56*NUMSIZE + +// Avoid using .rep for the sake of the BoringSSL/AWS-LC delocator, +// which doesn't accept repetitions, assembler macros etc. + +#define selectblock(I,C) \ + cmpq $I, %rdi ; \ + cmovzq TAB+JACSIZE*(I-1)+C*NUMSIZE(%rsp), %rax ; \ + cmovzq TAB+JACSIZE*(I-1)+8+C*NUMSIZE(%rsp), %rbx ; \ + cmovzq TAB+JACSIZE*(I-1)+16+C*NUMSIZE(%rsp), %rcx ; \ + cmovzq TAB+JACSIZE*(I-1)+24+C*NUMSIZE(%rsp), %rdx ; \ + cmovzq TAB+JACSIZE*(I-1)+32+C*NUMSIZE(%rsp), %r8 ; \ + cmovzq TAB+JACSIZE*(I-1)+40+C*NUMSIZE(%rsp), %r9 ; \ + cmovzq TAB+JACSIZE*(I-1)+48+C*NUMSIZE(%rsp), %r10 ; \ + cmovzq TAB+JACSIZE*(I-1)+56+C*NUMSIZE(%rsp), %r11 ; \ + cmovzq TAB+JACSIZE*(I-1)+64+C*NUMSIZE(%rsp), %r12 + +S2N_BN_SYMBOL(p521_jscalarmul): + CFI_START + _CET_ENDBR + +// The Windows version literally calls the standard ABI version. +// This simplifies the proofs since subroutine offsets are fixed. + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + CFI_CALL(Lp521_jscalarmul_standard) + CFI_POP(%rsi) + CFI_POP(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p521_jscalarmul) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_standard) + +Lp521_jscalarmul_standard: + CFI_START +#endif + +// Real start of the standard ABI code. + + CFI_PUSH(%r15) + CFI_PUSH(%r14) + CFI_PUSH(%r13) + CFI_PUSH(%r12) + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + + CFI_DEC_RSP(NSPACE) + +// Preserve the "res" input argument; others get processed early. + + movq %rdi, res + +// Reduce the input scalar mod n_521 and store it to "scalarb". + + movq %rdx, %rbx + leaq SCALARB(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_bignum_mod_n521_9) + +// Set the tab[0] table entry to the input point = 1 * P, but also +// reduce all coordinates modulo p. In principle we assume reduction +// as a precondition, but this reduces the scope for surprise, e.g. +// making sure that any input with z = 0 is treated as zero, even +// if the other coordinates are not in fact reduced. + + leaq TAB(%rsp), %rdi + movq %rbx, %rsi + CFI_CALL(Lp521_jscalarmul_bignum_mod_p521_9) + + leaq TAB+NUMSIZE(%rsp), %rdi + leaq NUMSIZE(%rbx), %rsi + CFI_CALL(Lp521_jscalarmul_bignum_mod_p521_9) + + leaq TAB+2*NUMSIZE(%rsp), %rdi + leaq 2*NUMSIZE(%rbx), %rsi + CFI_CALL(Lp521_jscalarmul_bignum_mod_p521_9) + +// If bit 520 of the scalar is set, then negate the scalar mod n_521, +// i.e. do scalar |-> n_521 - scalar, and also the point to compensate +// by negating its y coordinate. This further step is not needed by +// the indexing scheme (the top window is only a couple of bits either +// way), but is convenient to exclude a problem with the specific value +// scalar = n_521 - 18, where the last Jacobian addition is of the form +// (n_521 - 9) * P + -(9 * P) and hence is a degenerate doubling case. + + xorl %eax, %eax + notq %rax + movq $0xbb6fb71e91386409, %r8 + subq SCALARB(%rsp), %r8 + movq $0x3bb5c9b8899c47ae, %r9 + sbbq SCALARB+8(%rsp), %r9 + movq $0x7fcc0148f709a5d0, %r10 + sbbq SCALARB+16(%rsp), %r10 + movq $0x51868783bf2f966b, %r11 + sbbq SCALARB+24(%rsp), %r11 + leaq -5(%rax), %r12 + sbbq SCALARB+32(%rsp), %r12 + movq %rax, %r13 + sbbq SCALARB+40(%rsp), %r13 + movq %rax, %r14 + sbbq SCALARB+48(%rsp), %r14 + movq %rax, %r15 + sbbq SCALARB+56(%rsp), %r15 + movq $0x1ff, %rax + movq SCALARB+64(%rsp), %rcx + sbbq %rcx, %rax + + btq $8, %rcx + sbbq %rcx, %rcx + + cmovncq SCALARB(%rsp), %r8 + cmovncq SCALARB+8(%rsp), %r9 + cmovncq SCALARB+16(%rsp), %r10 + cmovncq SCALARB+24(%rsp), %r11 + cmovncq SCALARB+32(%rsp), %r12 + cmovncq SCALARB+40(%rsp), %r13 + cmovncq SCALARB+48(%rsp), %r14 + cmovncq SCALARB+56(%rsp), %r15 + cmovncq SCALARB+64(%rsp), %rax + + movq %r8, SCALARB(%rsp) + movq %r9, SCALARB+8(%rsp) + movq %r10, SCALARB+16(%rsp) + movq %r11, SCALARB+24(%rsp) + movq %r12, SCALARB+32(%rsp) + movq %r13, SCALARB+40(%rsp) + movq %r14, SCALARB+48(%rsp) + movq %r15, SCALARB+56(%rsp) + movq %rax, SCALARB+64(%rsp) + + movq TAB+NUMSIZE(%rsp), %r8 + movq TAB+NUMSIZE+8(%rsp), %r9 + movq TAB+NUMSIZE+16(%rsp), %r10 + movq TAB+NUMSIZE+24(%rsp), %r11 + movq TAB+NUMSIZE+32(%rsp), %r12 + movq TAB+NUMSIZE+40(%rsp), %r13 + movq TAB+NUMSIZE+48(%rsp), %r14 + movq TAB+NUMSIZE+56(%rsp), %r15 + movq TAB+NUMSIZE+64(%rsp), %rax + + movq %r8, %rbx + movq %r12, %rbp + orq %r9, %rbx + orq %r13, %rbp + orq %r10, %rbx + orq %r14, %rbp + orq %r11, %rbx + orq %r15, %rbp + orq %rbp, %rbx + orq %rax, %rbx + cmovzq %rbx, %rcx + + xorq %rcx, %r8 + xorq %rcx, %r9 + xorq %rcx, %r10 + xorq %rcx, %r11 + xorq %rcx, %r12 + xorq %rcx, %r13 + xorq %rcx, %r14 + xorq %rcx, %r15 + andq $0x1FF, %rcx + xorq %rcx, %rax + + movq %r8, TAB+NUMSIZE(%rsp) + movq %r9, TAB+NUMSIZE+8(%rsp) + movq %r10, TAB+NUMSIZE+16(%rsp) + movq %r11, TAB+NUMSIZE+24(%rsp) + movq %r12, TAB+NUMSIZE+32(%rsp) + movq %r13, TAB+NUMSIZE+40(%rsp) + movq %r14, TAB+NUMSIZE+48(%rsp) + movq %r15, TAB+NUMSIZE+56(%rsp) + movq %rax, TAB+NUMSIZE+64(%rsp) + +// Compute and record tab[1] = 2 * p, ..., tab[15] = 16 * P + + leaq TAB+JACSIZE*1(%rsp), %rdi + leaq TAB(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_jdouble) + + leaq TAB+JACSIZE*2(%rsp), %rdi + leaq TAB+JACSIZE*1(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp521_jscalarmul_jadd) + + leaq TAB+JACSIZE*3(%rsp), %rdi + leaq TAB+JACSIZE*1(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_jdouble) + + leaq TAB+JACSIZE*4(%rsp), %rdi + leaq TAB+JACSIZE*3(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp521_jscalarmul_jadd) + + leaq TAB+JACSIZE*5(%rsp), %rdi + leaq TAB+JACSIZE*2(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_jdouble) + + leaq TAB+JACSIZE*6(%rsp), %rdi + leaq TAB+JACSIZE*5(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp521_jscalarmul_jadd) + + leaq TAB+JACSIZE*7(%rsp), %rdi + leaq TAB+JACSIZE*3(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_jdouble) + + leaq TAB+JACSIZE*8(%rsp), %rdi + leaq TAB+JACSIZE*7(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp521_jscalarmul_jadd) + + leaq TAB+JACSIZE*9(%rsp), %rdi + leaq TAB+JACSIZE*4(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_jdouble) + + leaq TAB+JACSIZE*10(%rsp), %rdi + leaq TAB+JACSIZE*9(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp521_jscalarmul_jadd) + + leaq TAB+JACSIZE*11(%rsp), %rdi + leaq TAB+JACSIZE*5(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_jdouble) + + leaq TAB+JACSIZE*12(%rsp), %rdi + leaq TAB+JACSIZE*11(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp521_jscalarmul_jadd) + + leaq TAB+JACSIZE*13(%rsp), %rdi + leaq TAB+JACSIZE*6(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_jdouble) + + leaq TAB+JACSIZE*14(%rsp), %rdi + leaq TAB+JACSIZE*13(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp521_jscalarmul_jadd) + + leaq TAB+JACSIZE*15(%rsp), %rdi + leaq TAB+JACSIZE*7(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_jdouble) + +// Add the recoding constant sum_i(16 * 32^i) to the scalar to allow signed +// digits. The digits of the constant, in lowest-to-highest order, are as +// follows; they are generated dynamically to use fewer large constant loads. +// +// 0x0842108421084210 %rax +// 0x1084210842108421 %rbx +// 0x2108421084210842 %rbx<<1 +// 0x4210842108421084 %rbx<<2 +// 0x8421084210842108 %rbx<<3 +// 0x0842108421084210 %rax +// 0x1084210842108421 %rbx +// 0x2108421084210842 %rbx<<1 +// 0x0000000000000084 + + movq $0x1084210842108421, %rax + movq %rax, %rbx + shrq $1, %rax + movq SCALARB(%rsp), %r8 + addq %rax, %r8 + movq SCALARB+8(%rsp), %r9 + adcq %rbx, %r9 + leaq (%rbx,%rbx), %rcx + movq SCALARB+16(%rsp), %r10 + adcq %rcx, %r10 + leaq (%rcx,%rcx), %rcx + movq SCALARB+24(%rsp), %r11 + adcq %rcx, %r11 + leaq (%rcx,%rcx), %rcx + movq SCALARB+32(%rsp), %r12 + adcq %rcx, %r12 + movq SCALARB+40(%rsp), %r13 + adcq %rax, %r13 + movq SCALARB+48(%rsp), %r14 + adcq %rbx, %r14 + movq SCALARB+56(%rsp), %r15 + leaq (%rbx,%rbx), %rcx + adcq %rcx, %r15 + movq SCALARB+64(%rsp), %rax + adcq $0x84, %rax + +// Because of the initial reduction the top bitfield (>= bits 520) is <= 1, +// i.e. just a single bit. Record that in %rdi, then shift the whole +// scalar left 56 bits to align the top of the next bitfield with the MSB +// (bits 571..575). + + movq %rax, %rdi + shrq $8, %rdi + shldq $56, %r15, %rax + shldq $56, %r14, %r15 + shldq $56, %r13, %r14 + shldq $56, %r12, %r13 + shldq $56, %r11, %r12 + shldq $56, %r10, %r11 + shldq $56, %r9, %r10 + shldq $56, %r8, %r9 + shlq $56, %r8 + + movq %r8, SCALARB(%rsp) + movq %r9, SCALARB+8(%rsp) + movq %r10, SCALARB+16(%rsp) + movq %r11, SCALARB+24(%rsp) + movq %r12, SCALARB+32(%rsp) + movq %r13, SCALARB+40(%rsp) + movq %r14, SCALARB+48(%rsp) + movq %r15, SCALARB+56(%rsp) + movq %rax, SCALARB+64(%rsp) + +// According to the top bit, initialize the accumulator to P or 0. This top +// digit, uniquely, is not recoded so there is no sign adjustment to make. +// We only really need to adjust the z coordinate to zero, but do all three. + + xorl %ecx, %ecx + testq %rdi, %rdi + + movq TAB(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC(%rsp) + movq TAB+8(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+8(%rsp) + movq TAB+16(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+16(%rsp) + movq TAB+24(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+24(%rsp) + movq TAB+32(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+32(%rsp) + movq TAB+40(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+40(%rsp) + movq TAB+48(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+48(%rsp) + movq TAB+56(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+56(%rsp) + movq TAB+64(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+64(%rsp) + movq TAB+72(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+72(%rsp) + movq TAB+80(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+80(%rsp) + movq TAB+88(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+88(%rsp) + movq TAB+96(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+96(%rsp) + movq TAB+104(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+104(%rsp) + movq TAB+112(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+112(%rsp) + movq TAB+120(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+120(%rsp) + movq TAB+128(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+128(%rsp) + movq TAB+136(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+136(%rsp) + movq TAB+144(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+144(%rsp) + movq TAB+152(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+152(%rsp) + movq TAB+160(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+160(%rsp) + movq TAB+168(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+168(%rsp) + movq TAB+176(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+176(%rsp) + movq TAB+184(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+184(%rsp) + movq TAB+192(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+192(%rsp) + movq TAB+200(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+200(%rsp) + movq TAB+208(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+208(%rsp) + +// Main loop over size-5 bitfields: double 5 times then add signed digit +// At each stage we shift the scalar left by 5 bits so we can simply pick +// the top 5 bits as the bitfield, saving some fiddle over indexing. + + movl $520, %ebp + +Lp521_jscalarmul_mainloop: + subq $5, %rbp + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_jdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_jdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_jdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_jdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_jdouble) + +// Choose the bitfield and adjust it to sign and magnitude + + movq SCALARB(%rsp), %r8 + movq SCALARB+8(%rsp), %r9 + movq SCALARB+16(%rsp), %r10 + movq SCALARB+24(%rsp), %r11 + movq SCALARB+32(%rsp), %r12 + movq SCALARB+40(%rsp), %r13 + movq SCALARB+48(%rsp), %r14 + movq SCALARB+56(%rsp), %r15 + movq SCALARB+64(%rsp), %rax + + + movq %rax, %rdi + shrq $59, %rdi + + shldq $5, %r15, %rax + shldq $5, %r14, %r15 + shldq $5, %r13, %r14 + shldq $5, %r12, %r13 + shldq $5, %r11, %r12 + shldq $5, %r10, %r11 + shldq $5, %r9, %r10 + shldq $5, %r8, %r9 + shlq $5, %r8 + + movq %r8, SCALARB(%rsp) + movq %r9, SCALARB+8(%rsp) + movq %r10, SCALARB+16(%rsp) + movq %r11, SCALARB+24(%rsp) + movq %r12, SCALARB+32(%rsp) + movq %r13, SCALARB+40(%rsp) + movq %r14, SCALARB+48(%rsp) + movq %r15, SCALARB+56(%rsp) + movq %rax, SCALARB+64(%rsp) + + subq $16, %rdi + sbbq %rsi, %rsi // %rsi = sign of digit (-1 = negative) + xorq %rsi, %rdi + subq %rsi, %rdi // %rdi = absolute value of digit + +// Conditionally select the table entry tab[i-1] = i * P in constant time +// Again, this is done in separate sweeps per coordinate, doing y last. + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + selectblock(1,0) + selectblock(2,0) + selectblock(3,0) + selectblock(4,0) + selectblock(5,0) + selectblock(6,0) + selectblock(7,0) + selectblock(8,0) + selectblock(9,0) + selectblock(10,0) + selectblock(11,0) + selectblock(12,0) + selectblock(13,0) + selectblock(14,0) + selectblock(15,0) + selectblock(16,0) + movq %rax, TABENT(%rsp) + movq %rbx, TABENT+8(%rsp) + movq %rcx, TABENT+16(%rsp) + movq %rdx, TABENT+24(%rsp) + movq %r8, TABENT+32(%rsp) + movq %r9, TABENT+40(%rsp) + movq %r10, TABENT+48(%rsp) + movq %r11, TABENT+56(%rsp) + movq %r12, TABENT+64(%rsp) + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + selectblock(1,2) + selectblock(2,2) + selectblock(3,2) + selectblock(4,2) + selectblock(5,2) + selectblock(6,2) + selectblock(7,2) + selectblock(8,2) + selectblock(9,2) + selectblock(10,2) + selectblock(11,2) + selectblock(12,2) + selectblock(13,2) + selectblock(14,2) + selectblock(15,2) + selectblock(16,2) + movq %rax, TABENT+2*NUMSIZE(%rsp) + movq %rbx, TABENT+2*NUMSIZE+8(%rsp) + movq %rcx, TABENT+2*NUMSIZE+16(%rsp) + movq %rdx, TABENT+2*NUMSIZE+24(%rsp) + movq %r8, TABENT+2*NUMSIZE+32(%rsp) + movq %r9, TABENT+2*NUMSIZE+40(%rsp) + movq %r10, TABENT+2*NUMSIZE+48(%rsp) + movq %r11, TABENT+2*NUMSIZE+56(%rsp) + movq %r12, TABENT+2*NUMSIZE+64(%rsp) + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + selectblock(1,1) + selectblock(2,1) + selectblock(3,1) + selectblock(4,1) + selectblock(5,1) + selectblock(6,1) + selectblock(7,1) + selectblock(8,1) + selectblock(9,1) + selectblock(10,1) + selectblock(11,1) + selectblock(12,1) + selectblock(13,1) + selectblock(14,1) + selectblock(15,1) + selectblock(16,1) + +// Store it to "tabent" with the y coordinate optionally negated. +// This is done carefully to give coordinates < p_521 even in +// the degenerate case y = 0 (when z = 0 for points on the curve). + + movq %rax, %r13 + orq %rbx, %r13 + movq %rcx, %r14 + orq %rdx, %r14 + movq %r8, %r15 + orq %r9, %r15 + movq %r10, %rdi + orq %r11, %rdi + orq %r14, %r13 + orq %rdi, %r15 + orq %r12, %r15 + orq %r15, %r13 + cmovzq %r13, %rsi + + xorq %rsi, %rax + xorq %rsi, %rbx + xorq %rsi, %rcx + xorq %rsi, %rdx + xorq %rsi, %r8 + xorq %rsi, %r9 + xorq %rsi, %r10 + xorq %rsi, %r11 + andq $0x1FF, %rsi + xorq %rsi, %r12 + + movq %rax, TABENT+NUMSIZE(%rsp) + movq %rbx, TABENT+NUMSIZE+8(%rsp) + movq %rcx, TABENT+NUMSIZE+16(%rsp) + movq %rdx, TABENT+NUMSIZE+24(%rsp) + movq %r8, TABENT+NUMSIZE+32(%rsp) + movq %r9, TABENT+NUMSIZE+40(%rsp) + movq %r10, TABENT+NUMSIZE+48(%rsp) + movq %r11, TABENT+NUMSIZE+56(%rsp) + movq %r12, TABENT+NUMSIZE+64(%rsp) + +// Add to the accumulator + + leaq TABENT(%rsp), %rdx + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_jadd) + + testq %rbp, %rbp + jne Lp521_jscalarmul_mainloop + +// That's the end of the main loop, and we just need to copy the +// result in "acc" to the output. + + movq res, %rdi + movq ACC(%rsp), %rax + movq %rax, (%rdi) + movq ACC+8(%rsp), %rax + movq %rax, 8(%rdi) + movq ACC+16(%rsp), %rax + movq %rax, 16(%rdi) + movq ACC+24(%rsp), %rax + movq %rax, 24(%rdi) + movq ACC+32(%rsp), %rax + movq %rax, 32(%rdi) + movq ACC+40(%rsp), %rax + movq %rax, 40(%rdi) + movq ACC+48(%rsp), %rax + movq %rax, 48(%rdi) + movq ACC+56(%rsp), %rax + movq %rax, 56(%rdi) + movq ACC+64(%rsp), %rax + movq %rax, 64(%rdi) + movq ACC+72(%rsp), %rax + movq %rax, 72(%rdi) + movq ACC+80(%rsp), %rax + movq %rax, 80(%rdi) + movq ACC+88(%rsp), %rax + movq %rax, 88(%rdi) + movq ACC+96(%rsp), %rax + movq %rax, 96(%rdi) + movq ACC+104(%rsp), %rax + movq %rax, 104(%rdi) + movq ACC+112(%rsp), %rax + movq %rax, 112(%rdi) + movq ACC+120(%rsp), %rax + movq %rax, 120(%rdi) + movq ACC+128(%rsp), %rax + movq %rax, 128(%rdi) + movq ACC+136(%rsp), %rax + movq %rax, 136(%rdi) + movq ACC+144(%rsp), %rax + movq %rax, 144(%rdi) + movq ACC+152(%rsp), %rax + movq %rax, 152(%rdi) + movq ACC+160(%rsp), %rax + movq %rax, 160(%rdi) + movq ACC+168(%rsp), %rax + movq %rax, 168(%rdi) + movq ACC+176(%rsp), %rax + movq %rax, 176(%rdi) + movq ACC+184(%rsp), %rax + movq %rax, 184(%rdi) + movq ACC+192(%rsp), %rax + movq %rax, 192(%rdi) + movq ACC+200(%rsp), %rax + movq %rax, 200(%rdi) + movq ACC+208(%rsp), %rax + movq %rax, 208(%rdi) + +// Restore stack and registers and return + + CFI_INC_RSP(NSPACE) + CFI_POP(%rbx) + CFI_POP(%rbp) + CFI_POP(%r12) + CFI_POP(%r13) + CFI_POP(%r14) + CFI_POP(%r15) + CFI_RET + +#if WINDOWS_ABI +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_standard) +#else +S2N_BN_SIZE_DIRECTIVE(p521_jscalarmul) +#endif + +// Local copies of subroutines, complete clones at the moment + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_bignum_mod_p521_9) + +Lp521_jscalarmul_bignum_mod_p521_9: + CFI_START + CFI_PUSH(%rbx) + movq 0x40(%rsi), %rax + movl $0x1ff, %edx + andq %rax, %rdx + shrq $0x9, %rax + stc + adcq (%rsi), %rax + movq 0x8(%rsi), %rcx + adcq $0x0, %rcx + movq 0x10(%rsi), %r8 + adcq $0x0, %r8 + movq 0x18(%rsi), %r9 + adcq $0x0, %r9 + movq 0x20(%rsi), %r10 + adcq $0x0, %r10 + movq 0x28(%rsi), %r11 + adcq $0x0, %r11 + movq 0x30(%rsi), %rbx + adcq $0x0, %rbx + movq 0x38(%rsi), %rsi + adcq $0x0, %rsi + adcq $0x0, %rdx + cmpq $0x200, %rdx + sbbq $0x0, %rax + movq %rax, (%rdi) + sbbq $0x0, %rcx + movq %rcx, 0x8(%rdi) + sbbq $0x0, %r8 + movq %r8, 0x10(%rdi) + sbbq $0x0, %r9 + movq %r9, 0x18(%rdi) + sbbq $0x0, %r10 + movq %r10, 0x20(%rdi) + sbbq $0x0, %r11 + movq %r11, 0x28(%rdi) + sbbq $0x0, %rbx + movq %rbx, 0x30(%rdi) + sbbq $0x0, %rsi + movq %rsi, 0x38(%rdi) + sbbq $0x0, %rdx + andq $0x1ff, %rdx + movq %rdx, 0x40(%rdi) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_bignum_mod_p521_9) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_bignum_mod_n521_9) + +Lp521_jscalarmul_bignum_mod_n521_9: + CFI_START + movq 0x40(%rsi), %rdx + movq $0xfffffffffffffe00, %rax + orq %rdx, %rax + movq %rax, 0x40(%rdi) + shrq $0x9, %rdx + addq $0x1, %rdx + movq $0x449048e16ec79bf7, %r9 + mulxq %r9, %rax, %rcx + adcxq (%rsi), %rax + movq %rax, (%rdi) + movq $0xc44a36477663b851, %r10 + mulxq %r10, %rax, %r8 + adcxq 0x8(%rsi), %rax + adoxq %rcx, %rax + movq %rax, 0x8(%rdi) + movq $0x8033feb708f65a2f, %r11 + mulxq %r11, %rax, %rcx + adcxq 0x10(%rsi), %rax + adoxq %r8, %rax + movq %rax, 0x10(%rdi) + movq $0xae79787c40d06994, %rax + mulxq %rax, %rax, %r8 + adcxq 0x18(%rsi), %rax + adoxq %rcx, %rax + movq %rax, 0x18(%rdi) + movl $0x5, %eax + mulxq %rax, %rax, %rcx + adcxq 0x20(%rsi), %rax + adoxq %r8, %rax + movq %rax, 0x20(%rdi) + movq %rcx, %rax + adoxq %rcx, %rcx + adcq 0x28(%rsi), %rcx + movq %rcx, 0x28(%rdi) + movq 0x30(%rsi), %rcx + adcq %rax, %rcx + movq %rcx, 0x30(%rdi) + movq 0x38(%rsi), %rcx + adcq %rax, %rcx + movq %rcx, 0x38(%rdi) + movq 0x40(%rdi), %rcx + adcq %rax, %rcx + cmc + sbbq %rdx, %rdx + andq %rdx, %r9 + andq %rdx, %r10 + andq %rdx, %r11 + movq $0xae79787c40d06994, %r8 + andq %rdx, %r8 + andl $0x5, %edx + subq %r9, (%rdi) + sbbq %r10, 0x8(%rdi) + sbbq %r11, 0x10(%rdi) + sbbq %r8, 0x18(%rdi) + sbbq %rdx, 0x20(%rdi) + sbbq %rax, 0x28(%rdi) + sbbq %rax, 0x30(%rdi) + sbbq %rax, 0x38(%rdi) + sbbl %eax, %ecx + andl $0x1ff, %ecx + movq %rcx, 0x40(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_bignum_mod_n521_9) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_jadd) + +Lp521_jscalarmul_jadd: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(528) + movq %rdi, 0x1f8(%rsp) + movq %rsi, 0x200(%rsp) + movq %rdx, 0x208(%rsp) + movq 0x200(%rsp), %rsi + leaq 0x90(%rsi), %rsi + leaq (%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_sqr_p521) + movq 0x208(%rsp), %rdi + leaq 0x90(%rdi), %rsi + leaq 0x168(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_sqr_p521) + movq 0x200(%rsp), %rsi + movq 0x208(%rsp), %rdi + leaq 0x48(%rsi), %rdx + leaq 0x90(%rdi), %rsi + leaq 0x1b0(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_mul_p521) + movq 0x200(%rsp), %rsi + movq 0x208(%rsp), %rdi + leaq 0x48(%rdi), %rdx + leaq 0x90(%rsi), %rsi + leaq 0x48(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_mul_p521) + movq 0x208(%rsp), %rdi + leaq (%rdi), %rdx + leaq (%rsp), %rsi + leaq 0x90(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_mul_p521) + movq 0x200(%rsp), %rsi + leaq (%rsi), %rdx + leaq 0x168(%rsp), %rsi + leaq 0x120(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_mul_p521) + leaq 0x48(%rsp), %rdx + leaq (%rsp), %rsi + leaq 0x48(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_mul_p521) + leaq 0x1b0(%rsp), %rdx + leaq 0x168(%rsp), %rsi + leaq 0x1b0(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_mul_p521) + movq 0x90(%rsp), %rax + subq 0x120(%rsp), %rax + movq 0x98(%rsp), %rdx + sbbq 0x128(%rsp), %rdx + movq 0xa0(%rsp), %r8 + sbbq 0x130(%rsp), %r8 + movq 0xa8(%rsp), %r9 + sbbq 0x138(%rsp), %r9 + movq 0xb0(%rsp), %r10 + sbbq 0x140(%rsp), %r10 + movq 0xb8(%rsp), %r11 + sbbq 0x148(%rsp), %r11 + movq 0xc0(%rsp), %r12 + sbbq 0x150(%rsp), %r12 + movq 0xc8(%rsp), %r13 + sbbq 0x158(%rsp), %r13 + movq 0xd0(%rsp), %r14 + sbbq 0x160(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0x168(%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x170(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x178(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x180(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x188(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x190(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x198(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x1a0(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x1a8(%rsp) + movq 0x48(%rsp), %rax + subq 0x1b0(%rsp), %rax + movq 0x50(%rsp), %rdx + sbbq 0x1b8(%rsp), %rdx + movq 0x58(%rsp), %r8 + sbbq 0x1c0(%rsp), %r8 + movq 0x60(%rsp), %r9 + sbbq 0x1c8(%rsp), %r9 + movq 0x68(%rsp), %r10 + sbbq 0x1d0(%rsp), %r10 + movq 0x70(%rsp), %r11 + sbbq 0x1d8(%rsp), %r11 + movq 0x78(%rsp), %r12 + sbbq 0x1e0(%rsp), %r12 + movq 0x80(%rsp), %r13 + sbbq 0x1e8(%rsp), %r13 + movq 0x88(%rsp), %r14 + sbbq 0x1f0(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0x48(%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x50(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x58(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x60(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x68(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x70(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x78(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x80(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x88(%rsp) + leaq 0x168(%rsp), %rsi + leaq 0xd8(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_sqr_p521) + leaq 0x48(%rsp), %rsi + leaq (%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_sqr_p521) + leaq 0x120(%rsp), %rdx + leaq 0xd8(%rsp), %rsi + leaq 0x120(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_mul_p521) + leaq 0x90(%rsp), %rdx + leaq 0xd8(%rsp), %rsi + leaq 0x90(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_mul_p521) + movq (%rsp), %rax + subq 0x120(%rsp), %rax + movq 0x8(%rsp), %rdx + sbbq 0x128(%rsp), %rdx + movq 0x10(%rsp), %r8 + sbbq 0x130(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x138(%rsp), %r9 + movq 0x20(%rsp), %r10 + sbbq 0x140(%rsp), %r10 + movq 0x28(%rsp), %r11 + sbbq 0x148(%rsp), %r11 + movq 0x30(%rsp), %r12 + sbbq 0x150(%rsp), %r12 + movq 0x38(%rsp), %r13 + sbbq 0x158(%rsp), %r13 + movq 0x40(%rsp), %r14 + sbbq 0x160(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, (%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x8(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x10(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x18(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x20(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x28(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x30(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x38(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x40(%rsp) + movq 0x90(%rsp), %rax + subq 0x120(%rsp), %rax + movq 0x98(%rsp), %rdx + sbbq 0x128(%rsp), %rdx + movq 0xa0(%rsp), %r8 + sbbq 0x130(%rsp), %r8 + movq 0xa8(%rsp), %r9 + sbbq 0x138(%rsp), %r9 + movq 0xb0(%rsp), %r10 + sbbq 0x140(%rsp), %r10 + movq 0xb8(%rsp), %r11 + sbbq 0x148(%rsp), %r11 + movq 0xc0(%rsp), %r12 + sbbq 0x150(%rsp), %r12 + movq 0xc8(%rsp), %r13 + sbbq 0x158(%rsp), %r13 + movq 0xd0(%rsp), %r14 + sbbq 0x160(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0xd8(%rsp) + sbbq $0x0, %rdx + movq %rdx, 0xe0(%rsp) + sbbq $0x0, %r8 + movq %r8, 0xe8(%rsp) + sbbq $0x0, %r9 + movq %r9, 0xf0(%rsp) + sbbq $0x0, %r10 + movq %r10, 0xf8(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x100(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x108(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x110(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x118(%rsp) + movq 0x200(%rsp), %rsi + leaq 0x90(%rsi), %rdx + leaq 0x168(%rsp), %rsi + leaq 0x168(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_mul_p521) + movq (%rsp), %rax + subq 0x90(%rsp), %rax + movq 0x8(%rsp), %rdx + sbbq 0x98(%rsp), %rdx + movq 0x10(%rsp), %r8 + sbbq 0xa0(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0xa8(%rsp), %r9 + movq 0x20(%rsp), %r10 + sbbq 0xb0(%rsp), %r10 + movq 0x28(%rsp), %r11 + sbbq 0xb8(%rsp), %r11 + movq 0x30(%rsp), %r12 + sbbq 0xc0(%rsp), %r12 + movq 0x38(%rsp), %r13 + sbbq 0xc8(%rsp), %r13 + movq 0x40(%rsp), %r14 + sbbq 0xd0(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, (%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x8(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x10(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x18(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x20(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x28(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x30(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x38(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x40(%rsp) + movq 0x120(%rsp), %rax + subq (%rsp), %rax + movq 0x128(%rsp), %rdx + sbbq 0x8(%rsp), %rdx + movq 0x130(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x138(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movq 0x140(%rsp), %r10 + sbbq 0x20(%rsp), %r10 + movq 0x148(%rsp), %r11 + sbbq 0x28(%rsp), %r11 + movq 0x150(%rsp), %r12 + sbbq 0x30(%rsp), %r12 + movq 0x158(%rsp), %r13 + sbbq 0x38(%rsp), %r13 + movq 0x160(%rsp), %r14 + sbbq 0x40(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0x120(%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x128(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x130(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x138(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x140(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x148(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x150(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x158(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x160(%rsp) + leaq 0x1b0(%rsp), %rdx + leaq 0xd8(%rsp), %rsi + leaq 0xd8(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_mul_p521) + movq 0x208(%rsp), %rdi + leaq 0x90(%rdi), %rdx + leaq 0x168(%rsp), %rsi + leaq 0x168(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_mul_p521) + leaq 0x120(%rsp), %rdx + leaq 0x48(%rsp), %rsi + leaq 0x120(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_mul_p521) + movq 0x120(%rsp), %rax + subq 0xd8(%rsp), %rax + movq 0x128(%rsp), %rdx + sbbq 0xe0(%rsp), %rdx + movq 0x130(%rsp), %r8 + sbbq 0xe8(%rsp), %r8 + movq 0x138(%rsp), %r9 + sbbq 0xf0(%rsp), %r9 + movq 0x140(%rsp), %r10 + sbbq 0xf8(%rsp), %r10 + movq 0x148(%rsp), %r11 + sbbq 0x100(%rsp), %r11 + movq 0x150(%rsp), %r12 + sbbq 0x108(%rsp), %r12 + movq 0x158(%rsp), %r13 + sbbq 0x110(%rsp), %r13 + movq 0x160(%rsp), %r14 + sbbq 0x118(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0x120(%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x128(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x130(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x138(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x140(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x148(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x150(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x158(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x160(%rsp) + movq 0x200(%rsp), %rsi + movq 0x90(%rsi), %r8 + movq 0x98(%rsi), %r9 + movq 0xa0(%rsi), %r10 + movq 0xa8(%rsi), %r11 + movq 0xb0(%rsi), %r12 + movq 0xb8(%rsi), %r13 + movq 0xc0(%rsi), %r14 + movq 0xc8(%rsi), %r15 + movq 0xd0(%rsi), %rbp + orq %r9, %r8 + orq %r11, %r10 + orq %r13, %r12 + orq %r15, %r14 + orq %r10, %r8 + orq %r14, %r12 + orq %rbp, %r8 + orq %r12, %r8 + negq %r8 + sbbq %rax, %rax + movq 0x208(%rsp), %rdi + movq 0x90(%rdi), %r8 + movq 0x98(%rdi), %r9 + movq 0xa0(%rdi), %r10 + movq 0xa8(%rdi), %r11 + movq 0xb0(%rdi), %r12 + movq 0xb8(%rdi), %r13 + movq 0xc0(%rdi), %r14 + movq 0xc8(%rdi), %r15 + movq 0xd0(%rdi), %rbp + orq %r9, %r8 + orq %r11, %r10 + orq %r13, %r12 + orq %r15, %r14 + orq %r10, %r8 + orq %r14, %r12 + orq %rbp, %r8 + orq %r12, %r8 + negq %r8 + sbbq %rdx, %rdx + cmpq %rax, %rdx + movq 0x120(%rsp), %r8 + cmovbq 0x48(%rsi), %r8 + cmova 0x48(%rdi), %r8 + movq 0x128(%rsp), %r9 + cmovbq 0x50(%rsi), %r9 + cmova 0x50(%rdi), %r9 + movq 0x130(%rsp), %r10 + cmovbq 0x58(%rsi), %r10 + cmova 0x58(%rdi), %r10 + movq 0x138(%rsp), %r11 + cmovbq 0x60(%rsi), %r11 + cmova 0x60(%rdi), %r11 + movq 0x140(%rsp), %r12 + cmovbq 0x68(%rsi), %r12 + cmova 0x68(%rdi), %r12 + movq 0x148(%rsp), %r13 + cmovbq 0x70(%rsi), %r13 + cmova 0x70(%rdi), %r13 + movq 0x150(%rsp), %r14 + cmovbq 0x78(%rsi), %r14 + cmova 0x78(%rdi), %r14 + movq 0x158(%rsp), %r15 + cmovbq 0x80(%rsi), %r15 + cmova 0x80(%rdi), %r15 + movq 0x160(%rsp), %rbp + cmovbq 0x88(%rsi), %rbp + cmova 0x88(%rdi), %rbp + movq %r8, 0x120(%rsp) + movq %r9, 0x128(%rsp) + movq %r10, 0x130(%rsp) + movq %r11, 0x138(%rsp) + movq %r12, 0x140(%rsp) + movq %r13, 0x148(%rsp) + movq %r14, 0x150(%rsp) + movq %r15, 0x158(%rsp) + movq %rbp, 0x160(%rsp) + movq 0x168(%rsp), %r8 + cmovbq 0x90(%rsi), %r8 + cmova 0x90(%rdi), %r8 + movq 0x170(%rsp), %r9 + cmovbq 0x98(%rsi), %r9 + cmova 0x98(%rdi), %r9 + movq 0x178(%rsp), %r10 + cmovbq 0xa0(%rsi), %r10 + cmova 0xa0(%rdi), %r10 + movq 0x180(%rsp), %r11 + cmovbq 0xa8(%rsi), %r11 + cmova 0xa8(%rdi), %r11 + movq 0x188(%rsp), %r12 + cmovbq 0xb0(%rsi), %r12 + cmova 0xb0(%rdi), %r12 + movq 0x190(%rsp), %r13 + cmovbq 0xb8(%rsi), %r13 + cmova 0xb8(%rdi), %r13 + movq 0x198(%rsp), %r14 + cmovbq 0xc0(%rsi), %r14 + cmova 0xc0(%rdi), %r14 + movq 0x1a0(%rsp), %r15 + cmovbq 0xc8(%rsi), %r15 + cmova 0xc8(%rdi), %r15 + movq 0x1a8(%rsp), %rbp + cmovbq 0xd0(%rsi), %rbp + cmova 0xd0(%rdi), %rbp + movq %r8, 0x168(%rsp) + movq %r9, 0x170(%rsp) + movq %r10, 0x178(%rsp) + movq %r11, 0x180(%rsp) + movq %r12, 0x188(%rsp) + movq %r13, 0x190(%rsp) + movq %r14, 0x198(%rsp) + movq %r15, 0x1a0(%rsp) + movq %rbp, 0x1a8(%rsp) + movq (%rsp), %r8 + cmovbq (%rsi), %r8 + cmova (%rdi), %r8 + movq 0x8(%rsp), %r9 + cmovbq 0x8(%rsi), %r9 + cmova 0x8(%rdi), %r9 + movq 0x10(%rsp), %r10 + cmovbq 0x10(%rsi), %r10 + cmova 0x10(%rdi), %r10 + movq 0x18(%rsp), %r11 + cmovbq 0x18(%rsi), %r11 + cmova 0x18(%rdi), %r11 + movq 0x20(%rsp), %r12 + cmovbq 0x20(%rsi), %r12 + cmova 0x20(%rdi), %r12 + movq 0x28(%rsp), %r13 + cmovbq 0x28(%rsi), %r13 + cmova 0x28(%rdi), %r13 + movq 0x30(%rsp), %r14 + cmovbq 0x30(%rsi), %r14 + cmova 0x30(%rdi), %r14 + movq 0x38(%rsp), %r15 + cmovbq 0x38(%rsi), %r15 + cmova 0x38(%rdi), %r15 + movq 0x40(%rsp), %rbp + cmovbq 0x40(%rsi), %rbp + cmova 0x40(%rdi), %rbp + movq 0x1f8(%rsp), %rdi + movq %r8, (%rdi) + movq %r9, 0x8(%rdi) + movq %r10, 0x10(%rdi) + movq %r11, 0x18(%rdi) + movq %r12, 0x20(%rdi) + movq %r13, 0x28(%rdi) + movq %r14, 0x30(%rdi) + movq %r15, 0x38(%rdi) + movq %rbp, 0x40(%rdi) + movq 0x120(%rsp), %rax + movq %rax, 0x48(%rdi) + movq 0x128(%rsp), %rax + movq %rax, 0x50(%rdi) + movq 0x130(%rsp), %rax + movq %rax, 0x58(%rdi) + movq 0x138(%rsp), %rax + movq %rax, 0x60(%rdi) + movq 0x140(%rsp), %rax + movq %rax, 0x68(%rdi) + movq 0x148(%rsp), %rax + movq %rax, 0x70(%rdi) + movq 0x150(%rsp), %rax + movq %rax, 0x78(%rdi) + movq 0x158(%rsp), %rax + movq %rax, 0x80(%rdi) + movq 0x160(%rsp), %rax + movq %rax, 0x88(%rdi) + movq 0x168(%rsp), %rax + movq %rax, 0x90(%rdi) + movq 0x170(%rsp), %rax + movq %rax, 0x98(%rdi) + movq 0x178(%rsp), %rax + movq %rax, 0xa0(%rdi) + movq 0x180(%rsp), %rax + movq %rax, 0xa8(%rdi) + movq 0x188(%rsp), %rax + movq %rax, 0xb0(%rdi) + movq 0x190(%rsp), %rax + movq %rax, 0xb8(%rdi) + movq 0x198(%rsp), %rax + movq %rax, 0xc0(%rdi) + movq 0x1a0(%rsp), %rax + movq %rax, 0xc8(%rdi) + movq 0x1a8(%rsp), %rax + movq %rax, 0xd0(%rdi) + CFI_INC_RSP(528) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_jadd) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_jdouble) + +Lp521_jscalarmul_jdouble: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(520) + movq %rdi, 0x1f8(%rsp) + movq %rsi, 0x200(%rsp) + movq 0x200(%rsp), %rdi + leaq 0x90(%rdi), %rsi + leaq (%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_sqr_p521) + movq 0x200(%rsp), %rdi + leaq 0x48(%rdi), %rsi + leaq 0x48(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_sqr_p521) + movq 0x200(%rsp), %rdi + stc + movq (%rdi), %rax + adcq (%rsp), %rax + movq 0x8(%rdi), %rbx + adcq 0x8(%rsp), %rbx + movq 0x10(%rdi), %r8 + adcq 0x10(%rsp), %r8 + movq 0x18(%rdi), %r9 + adcq 0x18(%rsp), %r9 + movq 0x20(%rdi), %r10 + adcq 0x20(%rsp), %r10 + movq 0x28(%rdi), %r11 + adcq 0x28(%rsp), %r11 + movq 0x30(%rdi), %r12 + adcq 0x30(%rsp), %r12 + movq 0x38(%rdi), %r13 + adcq 0x38(%rsp), %r13 + movq 0x40(%rdi), %r14 + adcq 0x40(%rsp), %r14 + movq $0x200, %rdx + andq %r14, %rdx + cmpq $0x200, %rdx + sbbq $0x0, %rax + movq %rax, 0x168(%rsp) + sbbq $0x0, %rbx + movq %rbx, 0x170(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x178(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x180(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x188(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x190(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x198(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x1a0(%rsp) + sbbq %rdx, %r14 + movq %r14, 0x1a8(%rsp) + movq 0x200(%rsp), %rdi + movq (%rdi), %rax + subq (%rsp), %rax + movq 0x8(%rdi), %rdx + sbbq 0x8(%rsp), %rdx + movq 0x10(%rdi), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x18(%rdi), %r9 + sbbq 0x18(%rsp), %r9 + movq 0x20(%rdi), %r10 + sbbq 0x20(%rsp), %r10 + movq 0x28(%rdi), %r11 + sbbq 0x28(%rsp), %r11 + movq 0x30(%rdi), %r12 + sbbq 0x30(%rsp), %r12 + movq 0x38(%rdi), %r13 + sbbq 0x38(%rsp), %r13 + movq 0x40(%rdi), %r14 + sbbq 0x40(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0x120(%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x128(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x130(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x138(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x140(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x148(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x150(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x158(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x160(%rsp) + leaq 0x120(%rsp), %rdx + leaq 0x168(%rsp), %rsi + leaq 0x90(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_mul_p521) + movq 0x200(%rsp), %rdi + stc + movq 0x48(%rdi), %rax + adcq 0x90(%rdi), %rax + movq 0x50(%rdi), %rbx + adcq 0x98(%rdi), %rbx + movq 0x58(%rdi), %r8 + adcq 0xa0(%rdi), %r8 + movq 0x60(%rdi), %r9 + adcq 0xa8(%rdi), %r9 + movq 0x68(%rdi), %r10 + adcq 0xb0(%rdi), %r10 + movq 0x70(%rdi), %r11 + adcq 0xb8(%rdi), %r11 + movq 0x78(%rdi), %r12 + adcq 0xc0(%rdi), %r12 + movq 0x80(%rdi), %r13 + adcq 0xc8(%rdi), %r13 + movq 0x88(%rdi), %r14 + adcq 0xd0(%rdi), %r14 + movq $0x200, %rdx + andq %r14, %rdx + cmpq $0x200, %rdx + sbbq $0x0, %rax + movq %rax, 0x168(%rsp) + sbbq $0x0, %rbx + movq %rbx, 0x170(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x178(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x180(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x188(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x190(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x198(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x1a0(%rsp) + sbbq %rdx, %r14 + movq %r14, 0x1a8(%rsp) + leaq 0x90(%rsp), %rsi + leaq 0x1b0(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_sqr_p521) + movq 0x200(%rsp), %rdi + leaq 0x48(%rsp), %rdx + leaq (%rdi), %rsi + leaq 0xd8(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_mul_p521) + leaq 0x168(%rsp), %rsi + leaq 0x120(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_sqr_p521) + movq $0x9, %rdx + movq 0x1f0(%rsp), %rbx + xorq $0x1ff, %rbx + movq 0x1b0(%rsp), %rax + notq %rax + mulxq %rax, %r8, %r9 + movq 0x1b8(%rsp), %rax + notq %rax + mulxq %rax, %rax, %r10 + addq %rax, %r9 + movq 0x1c0(%rsp), %rax + notq %rax + mulxq %rax, %rax, %r11 + adcq %rax, %r10 + movq 0x1c8(%rsp), %rax + notq %rax + mulxq %rax, %rax, %r12 + adcq %rax, %r11 + movq 0x1d0(%rsp), %rax + notq %rax + mulxq %rax, %rax, %r13 + adcq %rax, %r12 + movq 0x1d8(%rsp), %rax + notq %rax + mulxq %rax, %rax, %r14 + adcq %rax, %r13 + movq 0x1e0(%rsp), %rax + notq %rax + mulxq %rax, %rax, %r15 + adcq %rax, %r14 + movq 0x1e8(%rsp), %rax + notq %rax + mulxq %rax, %rax, %rcx + adcq %rax, %r15 + mulxq %rbx, %rbx, %rax + adcq %rcx, %rbx + xorl %eax, %eax + movq $0xc, %rdx + mulxq 0xd8(%rsp), %rax, %rcx + adcxq %rax, %r8 + adoxq %rcx, %r9 + mulxq 0xe0(%rsp), %rax, %rcx + adcxq %rax, %r9 + adoxq %rcx, %r10 + mulxq 0xe8(%rsp), %rax, %rcx + adcxq %rax, %r10 + adoxq %rcx, %r11 + mulxq 0xf0(%rsp), %rax, %rcx + adcxq %rax, %r11 + adoxq %rcx, %r12 + mulxq 0xf8(%rsp), %rax, %rcx + adcxq %rax, %r12 + adoxq %rcx, %r13 + mulxq 0x100(%rsp), %rax, %rcx + adcxq %rax, %r13 + adoxq %rcx, %r14 + mulxq 0x108(%rsp), %rax, %rcx + adcxq %rax, %r14 + adoxq %rcx, %r15 + mulxq 0x110(%rsp), %rax, %rcx + adcxq %rax, %r15 + adoxq %rcx, %rbx + mulxq 0x118(%rsp), %rax, %rcx + adcxq %rax, %rbx + movq %r9, %rax + andq %r10, %rax + andq %r11, %rax + andq %r12, %rax + andq %r13, %rax + andq %r14, %rax + andq %r15, %rax + movq %rbx, %rdx + shrq $0x9, %rdx + orq $0xfffffffffffffe00, %rbx + leaq 0x1(%rdx), %rcx + addq %r8, %rcx + movl $0x0, %ecx + adcq %rcx, %rax + movq %rbx, %rax + adcq %rcx, %rax + adcq %rdx, %r8 + movq %r8, 0x1b0(%rsp) + adcq %rcx, %r9 + movq %r9, 0x1b8(%rsp) + adcq %rcx, %r10 + movq %r10, 0x1c0(%rsp) + adcq %rcx, %r11 + movq %r11, 0x1c8(%rsp) + adcq %rcx, %r12 + movq %r12, 0x1d0(%rsp) + adcq %rcx, %r13 + movq %r13, 0x1d8(%rsp) + adcq %rcx, %r14 + movq %r14, 0x1e0(%rsp) + adcq %rcx, %r15 + movq %r15, 0x1e8(%rsp) + adcq %rcx, %rbx + andq $0x1ff, %rbx + movq %rbx, 0x1f0(%rsp) + movq 0x120(%rsp), %rax + subq (%rsp), %rax + movq 0x128(%rsp), %rdx + sbbq 0x8(%rsp), %rdx + movq 0x130(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x138(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movq 0x140(%rsp), %r10 + sbbq 0x20(%rsp), %r10 + movq 0x148(%rsp), %r11 + sbbq 0x28(%rsp), %r11 + movq 0x150(%rsp), %r12 + sbbq 0x30(%rsp), %r12 + movq 0x158(%rsp), %r13 + sbbq 0x38(%rsp), %r13 + movq 0x160(%rsp), %r14 + sbbq 0x40(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0x168(%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x170(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x178(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x180(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x188(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x190(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x198(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x1a0(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x1a8(%rsp) + leaq 0x48(%rsp), %rsi + leaq 0x120(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_sqr_p521) + movq 0x1f8(%rsp), %rdi + movq 0x168(%rsp), %rax + subq 0x48(%rsp), %rax + movq 0x170(%rsp), %rdx + sbbq 0x50(%rsp), %rdx + movq 0x178(%rsp), %r8 + sbbq 0x58(%rsp), %r8 + movq 0x180(%rsp), %r9 + sbbq 0x60(%rsp), %r9 + movq 0x188(%rsp), %r10 + sbbq 0x68(%rsp), %r10 + movq 0x190(%rsp), %r11 + sbbq 0x70(%rsp), %r11 + movq 0x198(%rsp), %r12 + sbbq 0x78(%rsp), %r12 + movq 0x1a0(%rsp), %r13 + sbbq 0x80(%rsp), %r13 + movq 0x1a8(%rsp), %r14 + sbbq 0x88(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0x90(%rdi) + sbbq $0x0, %rdx + movq %rdx, 0x98(%rdi) + sbbq $0x0, %r8 + movq %r8, 0xa0(%rdi) + sbbq $0x0, %r9 + movq %r9, 0xa8(%rdi) + sbbq $0x0, %r10 + movq %r10, 0xb0(%rdi) + sbbq $0x0, %r11 + movq %r11, 0xb8(%rdi) + sbbq $0x0, %r12 + movq %r12, 0xc0(%rdi) + sbbq $0x0, %r13 + movq %r13, 0xc8(%rdi) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0xd0(%rdi) + leaq 0x90(%rsp), %rdx + leaq 0x1b0(%rsp), %rsi + leaq 0x168(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_mul_p521) + movq 0x1f8(%rsp), %rdi + movq 0x118(%rsp), %rbx + movq 0x110(%rsp), %r15 + shldq $0x2, %r15, %rbx + movq 0x108(%rsp), %r14 + shldq $0x2, %r14, %r15 + movq 0x100(%rsp), %r13 + shldq $0x2, %r13, %r14 + movq 0xf8(%rsp), %r12 + shldq $0x2, %r12, %r13 + movq 0xf0(%rsp), %r11 + shldq $0x2, %r11, %r12 + movq 0xe8(%rsp), %r10 + shldq $0x2, %r10, %r11 + movq 0xe0(%rsp), %r9 + shldq $0x2, %r9, %r10 + movq 0xd8(%rsp), %r8 + shldq $0x2, %r8, %r9 + shlq $0x2, %r8 + movq 0x1f0(%rsp), %rcx + xorq $0x1ff, %rcx + movq 0x1b0(%rsp), %rax + notq %rax + addq %rax, %r8 + movq 0x1b8(%rsp), %rax + notq %rax + adcq %rax, %r9 + movq 0x1c0(%rsp), %rax + notq %rax + adcq %rax, %r10 + movq 0x1c8(%rsp), %rax + notq %rax + adcq %rax, %r11 + movq 0x1d0(%rsp), %rax + notq %rax + adcq %rax, %r12 + movq 0x1d8(%rsp), %rax + notq %rax + adcq %rax, %r13 + movq 0x1e0(%rsp), %rax + notq %rax + adcq %rax, %r14 + movq 0x1e8(%rsp), %rax + notq %rax + adcq %rax, %r15 + adcq %rcx, %rbx + movq %r9, %rax + andq %r10, %rax + andq %r11, %rax + andq %r12, %rax + andq %r13, %rax + andq %r14, %rax + andq %r15, %rax + movq %rbx, %rdx + shrq $0x9, %rdx + orq $0xfffffffffffffe00, %rbx + leaq 0x1(%rdx), %rcx + addq %r8, %rcx + movl $0x0, %ecx + adcq %rcx, %rax + movq %rbx, %rax + adcq %rcx, %rax + adcq %rdx, %r8 + movq %r8, (%rdi) + adcq %rcx, %r9 + movq %r9, 0x8(%rdi) + adcq %rcx, %r10 + movq %r10, 0x10(%rdi) + adcq %rcx, %r11 + movq %r11, 0x18(%rdi) + adcq %rcx, %r12 + movq %r12, 0x20(%rdi) + adcq %rcx, %r13 + movq %r13, 0x28(%rdi) + adcq %rcx, %r14 + movq %r14, 0x30(%rdi) + adcq %rcx, %r15 + movq %r15, 0x38(%rdi) + adcq %rcx, %rbx + andq $0x1ff, %rbx + movq %rbx, 0x40(%rdi) + movq 0x1f8(%rsp), %rdi + movq 0x160(%rsp), %rbx + xorq $0x1ff, %rbx + movq 0x158(%rsp), %r15 + notq %r15 + shldq $0x3, %r15, %rbx + movq 0x150(%rsp), %r14 + notq %r14 + shldq $0x3, %r14, %r15 + movq 0x148(%rsp), %r13 + notq %r13 + shldq $0x3, %r13, %r14 + movq 0x140(%rsp), %r12 + notq %r12 + shldq $0x3, %r12, %r13 + movq 0x138(%rsp), %r11 + notq %r11 + shldq $0x3, %r11, %r12 + movq 0x130(%rsp), %r10 + notq %r10 + shldq $0x3, %r10, %r11 + movq 0x128(%rsp), %r9 + notq %r9 + shldq $0x3, %r9, %r10 + movq 0x120(%rsp), %r8 + notq %r8 + shldq $0x3, %r8, %r9 + shlq $0x3, %r8 + movq $0x3, %rdx + xorl %eax, %eax + mulxq 0x168(%rsp), %rax, %rcx + adcxq %rax, %r8 + adoxq %rcx, %r9 + mulxq 0x170(%rsp), %rax, %rcx + adcxq %rax, %r9 + adoxq %rcx, %r10 + mulxq 0x178(%rsp), %rax, %rcx + adcxq %rax, %r10 + adoxq %rcx, %r11 + mulxq 0x180(%rsp), %rax, %rcx + adcxq %rax, %r11 + adoxq %rcx, %r12 + mulxq 0x188(%rsp), %rax, %rcx + adcxq %rax, %r12 + adoxq %rcx, %r13 + mulxq 0x190(%rsp), %rax, %rcx + adcxq %rax, %r13 + adoxq %rcx, %r14 + mulxq 0x198(%rsp), %rax, %rcx + adcxq %rax, %r14 + adoxq %rcx, %r15 + mulxq 0x1a0(%rsp), %rax, %rcx + adcxq %rax, %r15 + adoxq %rcx, %rbx + mulxq 0x1a8(%rsp), %rax, %rcx + adcxq %rax, %rbx + movq %r9, %rax + andq %r10, %rax + andq %r11, %rax + andq %r12, %rax + andq %r13, %rax + andq %r14, %rax + andq %r15, %rax + movq %rbx, %rdx + shrq $0x9, %rdx + orq $0xfffffffffffffe00, %rbx + leaq 0x1(%rdx), %rcx + addq %r8, %rcx + movl $0x0, %ecx + adcq %rcx, %rax + movq %rbx, %rax + adcq %rcx, %rax + adcq %rdx, %r8 + movq %r8, 0x48(%rdi) + adcq %rcx, %r9 + movq %r9, 0x50(%rdi) + adcq %rcx, %r10 + movq %r10, 0x58(%rdi) + adcq %rcx, %r11 + movq %r11, 0x60(%rdi) + adcq %rcx, %r12 + movq %r12, 0x68(%rdi) + adcq %rcx, %r13 + movq %r13, 0x70(%rdi) + adcq %rcx, %r14 + movq %r14, 0x78(%rdi) + adcq %rcx, %r15 + movq %r15, 0x80(%rdi) + adcq %rcx, %rbx + andq $0x1ff, %rbx + movq %rbx, 0x88(%rdi) + CFI_INC_RSP(520) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_jdouble) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_mul_p521) + +Lp521_jscalarmul_mul_p521: + CFI_START + CFI_DEC_RSP(64) + movq %rdx, %rcx + xorl %ebp, %ebp + movq (%rcx), %rdx + mulxq (%rsi), %r8, %r9 + movq %r8, (%rsp) + mulxq 0x8(%rsi), %rbx, %r10 + adcq %rbx, %r9 + mulxq 0x10(%rsi), %rbx, %r11 + adcq %rbx, %r10 + mulxq 0x18(%rsi), %rbx, %r12 + adcq %rbx, %r11 + mulxq 0x20(%rsi), %rbx, %r13 + adcq %rbx, %r12 + mulxq 0x28(%rsi), %rbx, %r14 + adcq %rbx, %r13 + mulxq 0x30(%rsi), %rbx, %r15 + adcq %rbx, %r14 + mulxq 0x38(%rsi), %rbx, %r8 + adcq %rbx, %r15 + adcq %rbp, %r8 + movq 0x8(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + movq %r9, 0x8(%rsp) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x38(%rsi), %rax, %r9 + adcxq %rax, %r8 + adoxq %rbp, %r9 + adcq %rbp, %r9 + movq 0x10(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + movq %r10, 0x10(%rsp) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x38(%rsi), %rax, %r10 + adcxq %rax, %r9 + adoxq %rbp, %r10 + adcq %rbp, %r10 + movq 0x18(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + movq %r11, 0x18(%rsp) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x38(%rsi), %rax, %r11 + adcxq %rax, %r10 + adoxq %rbp, %r11 + adcq %rbp, %r11 + movq 0x20(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + movq %r12, 0x20(%rsp) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x38(%rsi), %rax, %r12 + adcxq %rax, %r11 + adoxq %rbp, %r12 + adcq %rbp, %r12 + movq 0x28(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + movq %r13, 0x28(%rsp) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x38(%rsi), %rax, %r13 + adcxq %rax, %r12 + adoxq %rbp, %r13 + adcq %rbp, %r13 + movq 0x30(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + movq %r14, 0x30(%rsp) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x38(%rsi), %rax, %r14 + adcxq %rax, %r13 + adoxq %rbp, %r14 + adcq %rbp, %r14 + movq 0x38(%rcx), %rdx + xorl %ebp, %ebp + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %r8 + movq %r15, 0x38(%rsp) + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x38(%rsi), %rax, %r15 + adcxq %rax, %r14 + adoxq %rbp, %r15 + adcq %rbp, %r15 + movq 0x40(%rsi), %rdx + xorl %ebp, %ebp + mulxq (%rcx), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x8(%rcx), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x10(%rcx), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x18(%rcx), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x20(%rcx), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x28(%rcx), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x30(%rcx), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x38(%rcx), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbp, %rbx + adcq %rbx, %rbp + movq 0x40(%rcx), %rdx + xorl %eax, %eax + mulxq (%rsi), %rax, %rbx + adcxq %rax, %r8 + adoxq %rbx, %r9 + mulxq 0x8(%rsi), %rax, %rbx + adcxq %rax, %r9 + adoxq %rbx, %r10 + mulxq 0x10(%rsi), %rax, %rbx + adcxq %rax, %r10 + adoxq %rbx, %r11 + mulxq 0x18(%rsi), %rax, %rbx + adcxq %rax, %r11 + adoxq %rbx, %r12 + mulxq 0x20(%rsi), %rax, %rbx + adcxq %rax, %r12 + adoxq %rbx, %r13 + mulxq 0x28(%rsi), %rax, %rbx + adcxq %rax, %r13 + adoxq %rbx, %r14 + mulxq 0x30(%rsi), %rax, %rbx + adcxq %rax, %r14 + adoxq %rbx, %r15 + mulxq 0x38(%rsi), %rax, %rbx + adcxq %rax, %r15 + adoxq %rbx, %rbp + mulxq 0x40(%rsi), %rax, %rbx + adcq %rax, %rbp + movq %r8, %rax + andq $0x1ff, %rax + shrdq $0x9, %r9, %r8 + shrdq $0x9, %r10, %r9 + shrdq $0x9, %r11, %r10 + shrdq $0x9, %r12, %r11 + shrdq $0x9, %r13, %r12 + shrdq $0x9, %r14, %r13 + shrdq $0x9, %r15, %r14 + shrdq $0x9, %rbp, %r15 + shrq $0x9, %rbp + addq %rax, %rbp + stc + adcq (%rsp), %r8 + adcq 0x8(%rsp), %r9 + adcq 0x10(%rsp), %r10 + adcq 0x18(%rsp), %r11 + adcq 0x20(%rsp), %r12 + adcq 0x28(%rsp), %r13 + adcq 0x30(%rsp), %r14 + adcq 0x38(%rsp), %r15 + adcq $0xfffffffffffffe00, %rbp + cmc + sbbq $0x0, %r8 + movq %r8, (%rdi) + sbbq $0x0, %r9 + movq %r9, 0x8(%rdi) + sbbq $0x0, %r10 + movq %r10, 0x10(%rdi) + sbbq $0x0, %r11 + movq %r11, 0x18(%rdi) + sbbq $0x0, %r12 + movq %r12, 0x20(%rdi) + sbbq $0x0, %r13 + movq %r13, 0x28(%rdi) + sbbq $0x0, %r14 + movq %r14, 0x30(%rdi) + sbbq $0x0, %r15 + movq %r15, 0x38(%rdi) + sbbq $0x0, %rbp + andq $0x1ff, %rbp + movq %rbp, 0x40(%rdi) + CFI_INC_RSP(64) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_mul_p521) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_sqr_p521) + +Lp521_jscalarmul_sqr_p521: + CFI_START + CFI_DEC_RSP(64) + xorl %ebp, %ebp + movq (%rsi), %rdx + mulxq 0x8(%rsi), %r9, %rax + movq %r9, 0x8(%rsp) + mulxq 0x10(%rsi), %r10, %rcx + adcxq %rax, %r10 + movq %r10, 0x10(%rsp) + mulxq 0x18(%rsi), %r11, %rax + adcxq %rcx, %r11 + mulxq 0x20(%rsi), %r12, %rcx + adcxq %rax, %r12 + mulxq 0x28(%rsi), %r13, %rax + adcxq %rcx, %r13 + mulxq 0x30(%rsi), %r14, %rcx + adcxq %rax, %r14 + mulxq 0x38(%rsi), %r15, %r8 + adcxq %rcx, %r15 + adcxq %rbp, %r8 + xorl %ebp, %ebp + movq 0x8(%rsi), %rdx + mulxq 0x10(%rsi), %rax, %rcx + adcxq %rax, %r11 + adoxq %rcx, %r12 + movq %r11, 0x18(%rsp) + mulxq 0x18(%rsi), %rax, %rcx + adcxq %rax, %r12 + adoxq %rcx, %r13 + movq %r12, 0x20(%rsp) + mulxq 0x20(%rsi), %rax, %rcx + adcxq %rax, %r13 + adoxq %rcx, %r14 + mulxq 0x28(%rsi), %rax, %rcx + adcxq %rax, %r14 + adoxq %rcx, %r15 + mulxq 0x30(%rsi), %rax, %rcx + adcxq %rax, %r15 + adoxq %rcx, %r8 + mulxq 0x38(%rsi), %rax, %r9 + adcxq %rax, %r8 + adoxq %rbp, %r9 + movq 0x20(%rsi), %rdx + mulxq 0x28(%rsi), %rax, %r10 + adcxq %rax, %r9 + adoxq %rbp, %r10 + adcxq %rbp, %r10 + xorl %ebp, %ebp + movq 0x10(%rsi), %rdx + mulxq 0x18(%rsi), %rax, %rcx + adcxq %rax, %r13 + adoxq %rcx, %r14 + movq %r13, 0x28(%rsp) + mulxq 0x20(%rsi), %rax, %rcx + adcxq %rax, %r14 + adoxq %rcx, %r15 + movq %r14, 0x30(%rsp) + mulxq 0x28(%rsi), %rax, %rcx + adcxq %rax, %r15 + adoxq %rcx, %r8 + mulxq 0x30(%rsi), %rax, %rcx + adcxq %rax, %r8 + adoxq %rcx, %r9 + mulxq 0x38(%rsi), %rax, %rcx + adcxq %rax, %r9 + adoxq %rcx, %r10 + movq 0x30(%rsi), %rdx + mulxq 0x20(%rsi), %rax, %r11 + adcxq %rax, %r10 + adoxq %rbp, %r11 + mulxq 0x28(%rsi), %rax, %r12 + adcxq %rax, %r11 + adoxq %rbp, %r12 + adcxq %rbp, %r12 + xorl %ebp, %ebp + movq 0x18(%rsi), %rdx + mulxq 0x20(%rsi), %rax, %rcx + adcxq %rax, %r15 + adoxq %rcx, %r8 + movq %r15, 0x38(%rsp) + mulxq 0x28(%rsi), %rax, %rcx + adcxq %rax, %r8 + adoxq %rcx, %r9 + mulxq 0x30(%rsi), %rax, %rcx + adcxq %rax, %r9 + adoxq %rcx, %r10 + mulxq 0x38(%rsi), %rax, %rcx + adcxq %rax, %r10 + adoxq %rcx, %r11 + movq 0x38(%rsi), %rdx + mulxq 0x20(%rsi), %rax, %rcx + adcxq %rax, %r11 + adoxq %rcx, %r12 + mulxq 0x28(%rsi), %rax, %r13 + adcxq %rax, %r12 + adoxq %rbp, %r13 + mulxq 0x30(%rsi), %rax, %r14 + adcxq %rax, %r13 + adoxq %rbp, %r14 + adcxq %rbp, %r14 + xorl %ebp, %ebp + movq (%rsi), %rdx + mulxq %rdx, %rax, %rcx + movq %rax, (%rsp) + movq 0x8(%rsp), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 0x8(%rsp) + movq 0x10(%rsp), %rax + movq 0x8(%rsi), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %rax, %rax + adoxq %rdx, %rax + movq %rax, 0x10(%rsp) + movq 0x18(%rsp), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 0x18(%rsp) + movq 0x20(%rsp), %rax + movq 0x10(%rsi), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %rax, %rax + adoxq %rdx, %rax + movq %rax, 0x20(%rsp) + movq 0x28(%rsp), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 0x28(%rsp) + movq 0x30(%rsp), %rax + movq 0x18(%rsi), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %rax, %rax + adoxq %rdx, %rax + movq %rax, 0x30(%rsp) + movq 0x38(%rsp), %rax + adcxq %rax, %rax + adoxq %rcx, %rax + movq %rax, 0x38(%rsp) + movq 0x20(%rsi), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %r8, %r8 + adoxq %rdx, %r8 + adcxq %r9, %r9 + adoxq %rcx, %r9 + movq 0x28(%rsi), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %r10, %r10 + adoxq %rdx, %r10 + adcxq %r11, %r11 + adoxq %rcx, %r11 + movq 0x30(%rsi), %rdx + mulxq %rdx, %rdx, %rcx + adcxq %r12, %r12 + adoxq %rdx, %r12 + adcxq %r13, %r13 + adoxq %rcx, %r13 + movq 0x38(%rsi), %rdx + mulxq %rdx, %rdx, %r15 + adcxq %r14, %r14 + adoxq %rdx, %r14 + adcxq %rbp, %r15 + adoxq %rbp, %r15 + movq 0x40(%rsi), %rdx + movq %rdx, %rbp + imulq %rbp, %rbp + addq %rdx, %rdx + mulxq (%rsi), %rax, %rcx + adcxq %rax, %r8 + adoxq %rcx, %r9 + mulxq 0x8(%rsi), %rax, %rcx + adcxq %rax, %r9 + adoxq %rcx, %r10 + mulxq 0x10(%rsi), %rax, %rcx + adcxq %rax, %r10 + adoxq %rcx, %r11 + mulxq 0x18(%rsi), %rax, %rcx + adcxq %rax, %r11 + adoxq %rcx, %r12 + mulxq 0x20(%rsi), %rax, %rcx + adcxq %rax, %r12 + adoxq %rcx, %r13 + mulxq 0x28(%rsi), %rax, %rcx + adcxq %rax, %r13 + adoxq %rcx, %r14 + mulxq 0x30(%rsi), %rax, %rcx + adcxq %rax, %r14 + adoxq %rcx, %r15 + mulxq 0x38(%rsi), %rax, %rcx + adcxq %rax, %r15 + adoxq %rcx, %rbp + adcq $0x0, %rbp + movq %r8, %rax + andq $0x1ff, %rax + shrdq $0x9, %r9, %r8 + shrdq $0x9, %r10, %r9 + shrdq $0x9, %r11, %r10 + shrdq $0x9, %r12, %r11 + shrdq $0x9, %r13, %r12 + shrdq $0x9, %r14, %r13 + shrdq $0x9, %r15, %r14 + shrdq $0x9, %rbp, %r15 + shrq $0x9, %rbp + addq %rax, %rbp + stc + adcq (%rsp), %r8 + adcq 0x8(%rsp), %r9 + adcq 0x10(%rsp), %r10 + adcq 0x18(%rsp), %r11 + adcq 0x20(%rsp), %r12 + adcq 0x28(%rsp), %r13 + adcq 0x30(%rsp), %r14 + adcq 0x38(%rsp), %r15 + adcq $0xfffffffffffffe00, %rbp + cmc + sbbq $0x0, %r8 + movq %r8, (%rdi) + sbbq $0x0, %r9 + movq %r9, 0x8(%rdi) + sbbq $0x0, %r10 + movq %r10, 0x10(%rdi) + sbbq $0x0, %r11 + movq %r11, 0x18(%rdi) + sbbq $0x0, %r12 + movq %r12, 0x20(%rdi) + sbbq $0x0, %r13 + movq %r13, 0x28(%rdi) + sbbq $0x0, %r14 + movq %r14, 0x30(%rdi) + sbbq $0x0, %r15 + movq %r15, 0x38(%rdi) + sbbq $0x0, %rbp + andq $0x1ff, %rbp + movq %rbp, 0x40(%rdi) + CFI_INC_RSP(64) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_sqr_p521) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/s2n/x86_att/p521_jscalarmul_alt.S b/cbits/s2n/x86_att/p521_jscalarmul_alt.S new file mode 100644 --- /dev/null +++ b/cbits/s2n/x86_att/p521_jscalarmul_alt.S @@ -0,0 +1,2850 @@ +// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 OR ISC OR MIT-0 + +// ---------------------------------------------------------------------------- +// Jacobian form scalar multiplication for P-521 +// Input scalar[9], point[27]; output res[27] +// +// extern void p521_jscalarmul_alt +// (uint64_t res[static 27], +// const uint64_t scalar[static 9], +// const uint64_t point[static 27]); +// +// This function is a variant of its affine point version p521_scalarmul. +// Here, input and output points are assumed to be in Jacobian form with +// a triple (x,y,z) representing the affine point (x/z^2,y/z^3) when +// z is nonzero or the point at infinity (group identity) if z = 0. +// +// Given scalar = n and point = P, assumed to be on the NIST elliptic +// curve P-521, returns a representation of n * P. If the result is the +// point at infinity (either because the input point was or because the +// scalar was a multiple of p_521) then the output is guaranteed to +// represent the point at infinity, i.e. to have its z coordinate zero. +// +// Standard x86-64 ABI: RDI = res, RSI = scalar, RDX = point +// Microsoft x64 ABI: RCX = res, RDX = scalar, R8 = point +// ---------------------------------------------------------------------------- + +#include "_internal_s2n_bignum_x86_att.h" + + + S2N_BN_SYM_VISIBILITY_DIRECTIVE(p521_jscalarmul_alt) + S2N_BN_FUNCTION_TYPE_DIRECTIVE(p521_jscalarmul_alt) + S2N_BN_SYM_PRIVACY_DIRECTIVE(p521_jscalarmul_alt) + + + .text + .balign 32 + +// Size of individual field elements + +#define NUMSIZE 72 +#define JACSIZE (3*NUMSIZE) + +// Intermediate variables on the stack. +// The table is 16 entries, each of size JACSIZE = 3 * NUMSIZE +// Uppercase syntactic variants make x86_att version simpler to generate. + +#define SCALARB (0*NUMSIZE) +#define scalarb (0*NUMSIZE)(%rsp) +#define ACC (1*NUMSIZE) +#define acc (1*NUMSIZE)(%rsp) +#define TABENT (4*NUMSIZE) +#define tabent (4*NUMSIZE)(%rsp) + +#define TAB (7*NUMSIZE) +#define tab (7*NUMSIZE)(%rsp) + +#define res (55*NUMSIZE)(%rsp) + +#define NSPACE 56*NUMSIZE + +// Avoid using .rep for the sake of the BoringSSL/AWS-LC delocator, +// which doesn't accept repetitions, assembler macros etc. + +#define selectblock(I,C) \ + cmpq $I, %rdi ; \ + cmovzq TAB+JACSIZE*(I-1)+C*NUMSIZE(%rsp), %rax ; \ + cmovzq TAB+JACSIZE*(I-1)+8+C*NUMSIZE(%rsp), %rbx ; \ + cmovzq TAB+JACSIZE*(I-1)+16+C*NUMSIZE(%rsp), %rcx ; \ + cmovzq TAB+JACSIZE*(I-1)+24+C*NUMSIZE(%rsp), %rdx ; \ + cmovzq TAB+JACSIZE*(I-1)+32+C*NUMSIZE(%rsp), %r8 ; \ + cmovzq TAB+JACSIZE*(I-1)+40+C*NUMSIZE(%rsp), %r9 ; \ + cmovzq TAB+JACSIZE*(I-1)+48+C*NUMSIZE(%rsp), %r10 ; \ + cmovzq TAB+JACSIZE*(I-1)+56+C*NUMSIZE(%rsp), %r11 ; \ + cmovzq TAB+JACSIZE*(I-1)+64+C*NUMSIZE(%rsp), %r12 + +S2N_BN_SYMBOL(p521_jscalarmul_alt): + CFI_START + _CET_ENDBR + +// The Windows version literally calls the standard ABI version. +// This simplifies the proofs since subroutine offsets are fixed. + +#if WINDOWS_ABI + CFI_PUSH(%rdi) + CFI_PUSH(%rsi) + movq %rcx, %rdi + movq %rdx, %rsi + movq %r8, %rdx + CFI_CALL(Lp521_jscalarmul_alt_standard) + CFI_POP(%rsi) + CFI_POP(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(p521_jscalarmul_alt) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_alt_standard) + +Lp521_jscalarmul_alt_standard: + CFI_START +#endif + +// Real start of the standard ABI code. + + CFI_PUSH(%r15) + CFI_PUSH(%r14) + CFI_PUSH(%r13) + CFI_PUSH(%r12) + CFI_PUSH(%rbp) + CFI_PUSH(%rbx) + + CFI_DEC_RSP(NSPACE) + +// Preserve the "res" input argument; others get processed early. + + movq %rdi, res + +// Reduce the input scalar mod n_521 and store it to "scalarb". + + movq %rdx, %rbx + leaq SCALARB(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_bignum_mod_n521_9) + +// Set the tab[0] table entry to the input point = 1 * P, but also +// reduce all coordinates modulo p. In principle we assume reduction +// as a precondition, but this reduces the scope for surprise, e.g. +// making sure that any input with z = 0 is treated as zero, even +// if the other coordinates are not in fact reduced. + + leaq TAB(%rsp), %rdi + movq %rbx, %rsi + CFI_CALL(Lp521_jscalarmul_alt_bignum_mod_p521_9) + + leaq TAB+NUMSIZE(%rsp), %rdi + leaq NUMSIZE(%rbx), %rsi + CFI_CALL(Lp521_jscalarmul_alt_bignum_mod_p521_9) + + leaq TAB+2*NUMSIZE(%rsp), %rdi + leaq 2*NUMSIZE(%rbx), %rsi + CFI_CALL(Lp521_jscalarmul_alt_bignum_mod_p521_9) + +// If bit 520 of the scalar is set, then negate the scalar mod n_521, +// i.e. do scalar |-> n_521 - scalar, and also the point to compensate +// by negating its y coordinate. This further step is not needed by +// the indexing scheme (the top window is only a couple of bits either +// way), but is convenient to exclude a problem with the specific value +// scalar = n_521 - 18, where the last Jacobian addition is of the form +// (n_521 - 9) * P + -(9 * P) and hence is a degenerate doubling case. + + xorl %eax, %eax + notq %rax + movq $0xbb6fb71e91386409, %r8 + subq SCALARB(%rsp), %r8 + movq $0x3bb5c9b8899c47ae, %r9 + sbbq SCALARB+8(%rsp), %r9 + movq $0x7fcc0148f709a5d0, %r10 + sbbq SCALARB+16(%rsp), %r10 + movq $0x51868783bf2f966b, %r11 + sbbq SCALARB+24(%rsp), %r11 + leaq -5(%rax), %r12 + sbbq SCALARB+32(%rsp), %r12 + movq %rax, %r13 + sbbq SCALARB+40(%rsp), %r13 + movq %rax, %r14 + sbbq SCALARB+48(%rsp), %r14 + movq %rax, %r15 + sbbq SCALARB+56(%rsp), %r15 + movq $0x1ff, %rax + movq SCALARB+64(%rsp), %rcx + sbbq %rcx, %rax + + btq $8, %rcx + sbbq %rcx, %rcx + + cmovncq SCALARB(%rsp), %r8 + cmovncq SCALARB+8(%rsp), %r9 + cmovncq SCALARB+16(%rsp), %r10 + cmovncq SCALARB+24(%rsp), %r11 + cmovncq SCALARB+32(%rsp), %r12 + cmovncq SCALARB+40(%rsp), %r13 + cmovncq SCALARB+48(%rsp), %r14 + cmovncq SCALARB+56(%rsp), %r15 + cmovncq SCALARB+64(%rsp), %rax + + movq %r8, SCALARB(%rsp) + movq %r9, SCALARB+8(%rsp) + movq %r10, SCALARB+16(%rsp) + movq %r11, SCALARB+24(%rsp) + movq %r12, SCALARB+32(%rsp) + movq %r13, SCALARB+40(%rsp) + movq %r14, SCALARB+48(%rsp) + movq %r15, SCALARB+56(%rsp) + movq %rax, SCALARB+64(%rsp) + + movq TAB+NUMSIZE(%rsp), %r8 + movq TAB+NUMSIZE+8(%rsp), %r9 + movq TAB+NUMSIZE+16(%rsp), %r10 + movq TAB+NUMSIZE+24(%rsp), %r11 + movq TAB+NUMSIZE+32(%rsp), %r12 + movq TAB+NUMSIZE+40(%rsp), %r13 + movq TAB+NUMSIZE+48(%rsp), %r14 + movq TAB+NUMSIZE+56(%rsp), %r15 + movq TAB+NUMSIZE+64(%rsp), %rax + + movq %r8, %rbx + movq %r12, %rbp + orq %r9, %rbx + orq %r13, %rbp + orq %r10, %rbx + orq %r14, %rbp + orq %r11, %rbx + orq %r15, %rbp + orq %rbp, %rbx + orq %rax, %rbx + cmovzq %rbx, %rcx + + xorq %rcx, %r8 + xorq %rcx, %r9 + xorq %rcx, %r10 + xorq %rcx, %r11 + xorq %rcx, %r12 + xorq %rcx, %r13 + xorq %rcx, %r14 + xorq %rcx, %r15 + andq $0x1FF, %rcx + xorq %rcx, %rax + + movq %r8, TAB+NUMSIZE(%rsp) + movq %r9, TAB+NUMSIZE+8(%rsp) + movq %r10, TAB+NUMSIZE+16(%rsp) + movq %r11, TAB+NUMSIZE+24(%rsp) + movq %r12, TAB+NUMSIZE+32(%rsp) + movq %r13, TAB+NUMSIZE+40(%rsp) + movq %r14, TAB+NUMSIZE+48(%rsp) + movq %r15, TAB+NUMSIZE+56(%rsp) + movq %rax, TAB+NUMSIZE+64(%rsp) + +// Compute and record tab[1] = 2 * p, ..., tab[15] = 16 * P + + leaq TAB+JACSIZE*1(%rsp), %rdi + leaq TAB(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_alt_jdouble) + + leaq TAB+JACSIZE*2(%rsp), %rdi + leaq TAB+JACSIZE*1(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp521_jscalarmul_alt_jadd) + + leaq TAB+JACSIZE*3(%rsp), %rdi + leaq TAB+JACSIZE*1(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_alt_jdouble) + + leaq TAB+JACSIZE*4(%rsp), %rdi + leaq TAB+JACSIZE*3(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp521_jscalarmul_alt_jadd) + + leaq TAB+JACSIZE*5(%rsp), %rdi + leaq TAB+JACSIZE*2(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_alt_jdouble) + + leaq TAB+JACSIZE*6(%rsp), %rdi + leaq TAB+JACSIZE*5(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp521_jscalarmul_alt_jadd) + + leaq TAB+JACSIZE*7(%rsp), %rdi + leaq TAB+JACSIZE*3(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_alt_jdouble) + + leaq TAB+JACSIZE*8(%rsp), %rdi + leaq TAB+JACSIZE*7(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp521_jscalarmul_alt_jadd) + + leaq TAB+JACSIZE*9(%rsp), %rdi + leaq TAB+JACSIZE*4(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_alt_jdouble) + + leaq TAB+JACSIZE*10(%rsp), %rdi + leaq TAB+JACSIZE*9(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp521_jscalarmul_alt_jadd) + + leaq TAB+JACSIZE*11(%rsp), %rdi + leaq TAB+JACSIZE*5(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_alt_jdouble) + + leaq TAB+JACSIZE*12(%rsp), %rdi + leaq TAB+JACSIZE*11(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp521_jscalarmul_alt_jadd) + + leaq TAB+JACSIZE*13(%rsp), %rdi + leaq TAB+JACSIZE*6(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_alt_jdouble) + + leaq TAB+JACSIZE*14(%rsp), %rdi + leaq TAB+JACSIZE*13(%rsp), %rsi + leaq TAB(%rsp), %rdx + CFI_CALL(Lp521_jscalarmul_alt_jadd) + + leaq TAB+JACSIZE*15(%rsp), %rdi + leaq TAB+JACSIZE*7(%rsp), %rsi + CFI_CALL(Lp521_jscalarmul_alt_jdouble) + +// Add the recoding constant sum_i(16 * 32^i) to the scalar to allow signed +// digits. The digits of the constant, in lowest-to-highest order, are as +// follows; they are generated dynamically to use fewer large constant loads. +// +// 0x0842108421084210 %rax +// 0x1084210842108421 %rbx +// 0x2108421084210842 %rbx<<1 +// 0x4210842108421084 %rbx<<2 +// 0x8421084210842108 %rbx<<3 +// 0x0842108421084210 %rax +// 0x1084210842108421 %rbx +// 0x2108421084210842 %rbx<<1 +// 0x0000000000000084 + + movq $0x1084210842108421, %rax + movq %rax, %rbx + shrq $1, %rax + movq SCALARB(%rsp), %r8 + addq %rax, %r8 + movq SCALARB+8(%rsp), %r9 + adcq %rbx, %r9 + leaq (%rbx,%rbx), %rcx + movq SCALARB+16(%rsp), %r10 + adcq %rcx, %r10 + leaq (%rcx,%rcx), %rcx + movq SCALARB+24(%rsp), %r11 + adcq %rcx, %r11 + leaq (%rcx,%rcx), %rcx + movq SCALARB+32(%rsp), %r12 + adcq %rcx, %r12 + movq SCALARB+40(%rsp), %r13 + adcq %rax, %r13 + movq SCALARB+48(%rsp), %r14 + adcq %rbx, %r14 + movq SCALARB+56(%rsp), %r15 + leaq (%rbx,%rbx), %rcx + adcq %rcx, %r15 + movq SCALARB+64(%rsp), %rax + adcq $0x84, %rax + +// Because of the initial reduction the top bitfield (>= bits 520) is <= 1, +// i.e. just a single bit. Record that in %rdi, then shift the whole +// scalar left 56 bits to align the top of the next bitfield with the MSB +// (bits 571..575). + + movq %rax, %rdi + shrq $8, %rdi + shldq $56, %r15, %rax + shldq $56, %r14, %r15 + shldq $56, %r13, %r14 + shldq $56, %r12, %r13 + shldq $56, %r11, %r12 + shldq $56, %r10, %r11 + shldq $56, %r9, %r10 + shldq $56, %r8, %r9 + shlq $56, %r8 + + movq %r8, SCALARB(%rsp) + movq %r9, SCALARB+8(%rsp) + movq %r10, SCALARB+16(%rsp) + movq %r11, SCALARB+24(%rsp) + movq %r12, SCALARB+32(%rsp) + movq %r13, SCALARB+40(%rsp) + movq %r14, SCALARB+48(%rsp) + movq %r15, SCALARB+56(%rsp) + movq %rax, SCALARB+64(%rsp) + +// According to the top bit, initialize the accumulator to P or 0. This top +// digit, uniquely, is not recoded so there is no sign adjustment to make. +// We only really need to adjust the z coordinate to zero, but do all three. + + xorl %ecx, %ecx + testq %rdi, %rdi + + movq TAB(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC(%rsp) + movq TAB+8(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+8(%rsp) + movq TAB+16(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+16(%rsp) + movq TAB+24(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+24(%rsp) + movq TAB+32(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+32(%rsp) + movq TAB+40(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+40(%rsp) + movq TAB+48(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+48(%rsp) + movq TAB+56(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+56(%rsp) + movq TAB+64(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+64(%rsp) + movq TAB+72(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+72(%rsp) + movq TAB+80(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+80(%rsp) + movq TAB+88(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+88(%rsp) + movq TAB+96(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+96(%rsp) + movq TAB+104(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+104(%rsp) + movq TAB+112(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+112(%rsp) + movq TAB+120(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+120(%rsp) + movq TAB+128(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+128(%rsp) + movq TAB+136(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+136(%rsp) + movq TAB+144(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+144(%rsp) + movq TAB+152(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+152(%rsp) + movq TAB+160(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+160(%rsp) + movq TAB+168(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+168(%rsp) + movq TAB+176(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+176(%rsp) + movq TAB+184(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+184(%rsp) + movq TAB+192(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+192(%rsp) + movq TAB+200(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+200(%rsp) + movq TAB+208(%rsp), %rax + cmovzq %rcx, %rax + movq %rax, ACC+208(%rsp) + +// Main loop over size-5 bitfields: double 5 times then add signed digit +// At each stage we shift the scalar left by 5 bits so we can simply pick +// the top 5 bits as the bitfield, saving some fiddle over indexing. + + movl $520, %ebp + +Lp521_jscalarmul_alt_mainloop: + subq $5, %rbp + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_jdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_jdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_jdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_jdouble) + + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_jdouble) + +// Choose the bitfield and adjust it to sign and magnitude + + movq SCALARB(%rsp), %r8 + movq SCALARB+8(%rsp), %r9 + movq SCALARB+16(%rsp), %r10 + movq SCALARB+24(%rsp), %r11 + movq SCALARB+32(%rsp), %r12 + movq SCALARB+40(%rsp), %r13 + movq SCALARB+48(%rsp), %r14 + movq SCALARB+56(%rsp), %r15 + movq SCALARB+64(%rsp), %rax + + + movq %rax, %rdi + shrq $59, %rdi + + shldq $5, %r15, %rax + shldq $5, %r14, %r15 + shldq $5, %r13, %r14 + shldq $5, %r12, %r13 + shldq $5, %r11, %r12 + shldq $5, %r10, %r11 + shldq $5, %r9, %r10 + shldq $5, %r8, %r9 + shlq $5, %r8 + + movq %r8, SCALARB(%rsp) + movq %r9, SCALARB+8(%rsp) + movq %r10, SCALARB+16(%rsp) + movq %r11, SCALARB+24(%rsp) + movq %r12, SCALARB+32(%rsp) + movq %r13, SCALARB+40(%rsp) + movq %r14, SCALARB+48(%rsp) + movq %r15, SCALARB+56(%rsp) + movq %rax, SCALARB+64(%rsp) + + subq $16, %rdi + sbbq %rsi, %rsi // %rsi = sign of digit (-1 = negative) + xorq %rsi, %rdi + subq %rsi, %rdi // %rdi = absolute value of digit + +// Conditionally select the table entry tab[i-1] = i * P in constant time +// Again, this is done in separate sweeps per coordinate, doing y last. + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + selectblock(1,0) + selectblock(2,0) + selectblock(3,0) + selectblock(4,0) + selectblock(5,0) + selectblock(6,0) + selectblock(7,0) + selectblock(8,0) + selectblock(9,0) + selectblock(10,0) + selectblock(11,0) + selectblock(12,0) + selectblock(13,0) + selectblock(14,0) + selectblock(15,0) + selectblock(16,0) + movq %rax, TABENT(%rsp) + movq %rbx, TABENT+8(%rsp) + movq %rcx, TABENT+16(%rsp) + movq %rdx, TABENT+24(%rsp) + movq %r8, TABENT+32(%rsp) + movq %r9, TABENT+40(%rsp) + movq %r10, TABENT+48(%rsp) + movq %r11, TABENT+56(%rsp) + movq %r12, TABENT+64(%rsp) + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + selectblock(1,2) + selectblock(2,2) + selectblock(3,2) + selectblock(4,2) + selectblock(5,2) + selectblock(6,2) + selectblock(7,2) + selectblock(8,2) + selectblock(9,2) + selectblock(10,2) + selectblock(11,2) + selectblock(12,2) + selectblock(13,2) + selectblock(14,2) + selectblock(15,2) + selectblock(16,2) + movq %rax, TABENT+2*NUMSIZE(%rsp) + movq %rbx, TABENT+2*NUMSIZE+8(%rsp) + movq %rcx, TABENT+2*NUMSIZE+16(%rsp) + movq %rdx, TABENT+2*NUMSIZE+24(%rsp) + movq %r8, TABENT+2*NUMSIZE+32(%rsp) + movq %r9, TABENT+2*NUMSIZE+40(%rsp) + movq %r10, TABENT+2*NUMSIZE+48(%rsp) + movq %r11, TABENT+2*NUMSIZE+56(%rsp) + movq %r12, TABENT+2*NUMSIZE+64(%rsp) + + xorl %eax, %eax + xorl %ebx, %ebx + xorl %ecx, %ecx + xorl %edx, %edx + xorl %r8d, %r8d + xorl %r9d, %r9d + xorl %r10d, %r10d + xorl %r11d, %r11d + xorl %r12d, %r12d + selectblock(1,1) + selectblock(2,1) + selectblock(3,1) + selectblock(4,1) + selectblock(5,1) + selectblock(6,1) + selectblock(7,1) + selectblock(8,1) + selectblock(9,1) + selectblock(10,1) + selectblock(11,1) + selectblock(12,1) + selectblock(13,1) + selectblock(14,1) + selectblock(15,1) + selectblock(16,1) + +// Store it to "tabent" with the y coordinate optionally negated. +// This is done carefully to give coordinates < p_521 even in +// the degenerate case y = 0 (when z = 0 for points on the curve). + + movq %rax, %r13 + orq %rbx, %r13 + movq %rcx, %r14 + orq %rdx, %r14 + movq %r8, %r15 + orq %r9, %r15 + movq %r10, %rdi + orq %r11, %rdi + orq %r14, %r13 + orq %rdi, %r15 + orq %r12, %r15 + orq %r15, %r13 + cmovzq %r13, %rsi + + xorq %rsi, %rax + xorq %rsi, %rbx + xorq %rsi, %rcx + xorq %rsi, %rdx + xorq %rsi, %r8 + xorq %rsi, %r9 + xorq %rsi, %r10 + xorq %rsi, %r11 + andq $0x1FF, %rsi + xorq %rsi, %r12 + + movq %rax, TABENT+NUMSIZE(%rsp) + movq %rbx, TABENT+NUMSIZE+8(%rsp) + movq %rcx, TABENT+NUMSIZE+16(%rsp) + movq %rdx, TABENT+NUMSIZE+24(%rsp) + movq %r8, TABENT+NUMSIZE+32(%rsp) + movq %r9, TABENT+NUMSIZE+40(%rsp) + movq %r10, TABENT+NUMSIZE+48(%rsp) + movq %r11, TABENT+NUMSIZE+56(%rsp) + movq %r12, TABENT+NUMSIZE+64(%rsp) + +// Add to the accumulator + + leaq TABENT(%rsp), %rdx + leaq ACC(%rsp), %rsi + leaq ACC(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_jadd) + + testq %rbp, %rbp + jne Lp521_jscalarmul_alt_mainloop + +// That's the end of the main loop, and we just need to copy the +// result in "acc" to the output. + + movq res, %rdi + movq ACC(%rsp), %rax + movq %rax, (%rdi) + movq ACC+8(%rsp), %rax + movq %rax, 8(%rdi) + movq ACC+16(%rsp), %rax + movq %rax, 16(%rdi) + movq ACC+24(%rsp), %rax + movq %rax, 24(%rdi) + movq ACC+32(%rsp), %rax + movq %rax, 32(%rdi) + movq ACC+40(%rsp), %rax + movq %rax, 40(%rdi) + movq ACC+48(%rsp), %rax + movq %rax, 48(%rdi) + movq ACC+56(%rsp), %rax + movq %rax, 56(%rdi) + movq ACC+64(%rsp), %rax + movq %rax, 64(%rdi) + movq ACC+72(%rsp), %rax + movq %rax, 72(%rdi) + movq ACC+80(%rsp), %rax + movq %rax, 80(%rdi) + movq ACC+88(%rsp), %rax + movq %rax, 88(%rdi) + movq ACC+96(%rsp), %rax + movq %rax, 96(%rdi) + movq ACC+104(%rsp), %rax + movq %rax, 104(%rdi) + movq ACC+112(%rsp), %rax + movq %rax, 112(%rdi) + movq ACC+120(%rsp), %rax + movq %rax, 120(%rdi) + movq ACC+128(%rsp), %rax + movq %rax, 128(%rdi) + movq ACC+136(%rsp), %rax + movq %rax, 136(%rdi) + movq ACC+144(%rsp), %rax + movq %rax, 144(%rdi) + movq ACC+152(%rsp), %rax + movq %rax, 152(%rdi) + movq ACC+160(%rsp), %rax + movq %rax, 160(%rdi) + movq ACC+168(%rsp), %rax + movq %rax, 168(%rdi) + movq ACC+176(%rsp), %rax + movq %rax, 176(%rdi) + movq ACC+184(%rsp), %rax + movq %rax, 184(%rdi) + movq ACC+192(%rsp), %rax + movq %rax, 192(%rdi) + movq ACC+200(%rsp), %rax + movq %rax, 200(%rdi) + movq ACC+208(%rsp), %rax + movq %rax, 208(%rdi) + +// Restore stack and registers and return + + CFI_INC_RSP(NSPACE) + CFI_POP(%rbx) + CFI_POP(%rbp) + CFI_POP(%r12) + CFI_POP(%r13) + CFI_POP(%r14) + CFI_POP(%r15) + CFI_RET + +#if WINDOWS_ABI +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_alt_standard) +#else +S2N_BN_SIZE_DIRECTIVE(p521_jscalarmul_alt) +#endif + +// Local copies of subroutines, complete clones at the moment + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_alt_bignum_mod_p521_9) + +Lp521_jscalarmul_alt_bignum_mod_p521_9: + CFI_START + CFI_PUSH(%rbx) + movq 0x40(%rsi), %rax + movl $0x1ff, %edx + andq %rax, %rdx + shrq $0x9, %rax + stc + adcq (%rsi), %rax + movq 0x8(%rsi), %rcx + adcq $0x0, %rcx + movq 0x10(%rsi), %r8 + adcq $0x0, %r8 + movq 0x18(%rsi), %r9 + adcq $0x0, %r9 + movq 0x20(%rsi), %r10 + adcq $0x0, %r10 + movq 0x28(%rsi), %r11 + adcq $0x0, %r11 + movq 0x30(%rsi), %rbx + adcq $0x0, %rbx + movq 0x38(%rsi), %rsi + adcq $0x0, %rsi + adcq $0x0, %rdx + cmpq $0x200, %rdx + sbbq $0x0, %rax + movq %rax, (%rdi) + sbbq $0x0, %rcx + movq %rcx, 0x8(%rdi) + sbbq $0x0, %r8 + movq %r8, 0x10(%rdi) + sbbq $0x0, %r9 + movq %r9, 0x18(%rdi) + sbbq $0x0, %r10 + movq %r10, 0x20(%rdi) + sbbq $0x0, %r11 + movq %r11, 0x28(%rdi) + sbbq $0x0, %rbx + movq %rbx, 0x30(%rdi) + sbbq $0x0, %rsi + movq %rsi, 0x38(%rdi) + sbbq $0x0, %rdx + andq $0x1ff, %rdx + movq %rdx, 0x40(%rdi) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_alt_bignum_mod_p521_9) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_alt_bignum_mod_n521_9) + +Lp521_jscalarmul_alt_bignum_mod_n521_9: + CFI_START + movq 0x40(%rsi), %rcx + movq $0xfffffffffffffe00, %rax + orq %rcx, %rax + movq %rax, 0x40(%rdi) + shrq $0x9, %rcx + addq $0x1, %rcx + movq $0x449048e16ec79bf7, %rax + mulq %rcx + movq %rax, %r8 + movq %rdx, %r9 + movq $0xc44a36477663b851, %rax + mulq %rcx + xorq %r10, %r10 + addq %rax, %r9 + adcq %rdx, %r10 + movq $0x8033feb708f65a2f, %rax + mulq %rcx + xorq %r11, %r11 + addq %rax, %r10 + adcq %rdx, %r11 + movq $0xae79787c40d06994, %rax + mulq %rcx + imulq $0x5, %rcx, %rcx + addq %rax, %r11 + adcq %rdx, %rcx + sbbq %rdx, %rdx + negq %rdx + xorl %eax, %eax + addq (%rsi), %r8 + movq %r8, (%rdi) + adcq 0x8(%rsi), %r9 + movq %r9, 0x8(%rdi) + adcq 0x10(%rsi), %r10 + movq %r10, 0x10(%rdi) + adcq 0x18(%rsi), %r11 + movq %r11, 0x18(%rdi) + adcq 0x20(%rsi), %rcx + movq %rcx, 0x20(%rdi) + adcq 0x28(%rsi), %rdx + movq %rdx, 0x28(%rdi) + movq 0x30(%rsi), %rdx + adcq %rax, %rdx + movq %rdx, 0x30(%rdi) + movq 0x38(%rsi), %rdx + adcq %rax, %rdx + movq %rdx, 0x38(%rdi) + movq 0x40(%rdi), %rcx + adcq %rax, %rcx + cmc + sbbq %rdx, %rdx + movq $0x449048e16ec79bf7, %r8 + andq %rdx, %r8 + movq $0xc44a36477663b851, %r9 + andq %rdx, %r9 + movq $0x8033feb708f65a2f, %r10 + andq %rdx, %r10 + movq $0xae79787c40d06994, %r11 + andq %rdx, %r11 + andq $0x5, %rdx + subq %r8, (%rdi) + sbbq %r9, 0x8(%rdi) + sbbq %r10, 0x10(%rdi) + sbbq %r11, 0x18(%rdi) + sbbq %rdx, 0x20(%rdi) + sbbq %rax, 0x28(%rdi) + sbbq %rax, 0x30(%rdi) + sbbq %rax, 0x38(%rdi) + sbbl %eax, %ecx + andl $0x1ff, %ecx + movq %rcx, 0x40(%rdi) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_alt_bignum_mod_n521_9) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_alt_jadd) + +Lp521_jscalarmul_alt_jadd: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(528) + movq %rdi, 0x1f8(%rsp) + movq %rsi, 0x200(%rsp) + movq %rdx, 0x208(%rsp) + movq 0x200(%rsp), %rsi + leaq 0x90(%rsi), %rsi + leaq (%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_sqr_p521) + movq 0x208(%rsp), %rdi + leaq 0x90(%rdi), %rsi + leaq 0x168(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_sqr_p521) + movq 0x200(%rsp), %rsi + movq 0x208(%rsp), %rdi + leaq 0x48(%rsi), %rdx + leaq 0x90(%rdi), %rsi + leaq 0x1b0(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_mul_p521) + movq 0x200(%rsp), %rsi + movq 0x208(%rsp), %rdi + leaq 0x48(%rdi), %rdx + leaq 0x90(%rsi), %rsi + leaq 0x48(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_mul_p521) + movq 0x208(%rsp), %rdi + leaq (%rdi), %rdx + leaq (%rsp), %rsi + leaq 0x90(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_mul_p521) + movq 0x200(%rsp), %rsi + leaq (%rsi), %rdx + leaq 0x168(%rsp), %rsi + leaq 0x120(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_mul_p521) + leaq 0x48(%rsp), %rdx + leaq (%rsp), %rsi + leaq 0x48(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_mul_p521) + leaq 0x1b0(%rsp), %rdx + leaq 0x168(%rsp), %rsi + leaq 0x1b0(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_mul_p521) + movq 0x90(%rsp), %rax + subq 0x120(%rsp), %rax + movq 0x98(%rsp), %rdx + sbbq 0x128(%rsp), %rdx + movq 0xa0(%rsp), %r8 + sbbq 0x130(%rsp), %r8 + movq 0xa8(%rsp), %r9 + sbbq 0x138(%rsp), %r9 + movq 0xb0(%rsp), %r10 + sbbq 0x140(%rsp), %r10 + movq 0xb8(%rsp), %r11 + sbbq 0x148(%rsp), %r11 + movq 0xc0(%rsp), %r12 + sbbq 0x150(%rsp), %r12 + movq 0xc8(%rsp), %r13 + sbbq 0x158(%rsp), %r13 + movq 0xd0(%rsp), %r14 + sbbq 0x160(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0x168(%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x170(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x178(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x180(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x188(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x190(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x198(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x1a0(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x1a8(%rsp) + movq 0x48(%rsp), %rax + subq 0x1b0(%rsp), %rax + movq 0x50(%rsp), %rdx + sbbq 0x1b8(%rsp), %rdx + movq 0x58(%rsp), %r8 + sbbq 0x1c0(%rsp), %r8 + movq 0x60(%rsp), %r9 + sbbq 0x1c8(%rsp), %r9 + movq 0x68(%rsp), %r10 + sbbq 0x1d0(%rsp), %r10 + movq 0x70(%rsp), %r11 + sbbq 0x1d8(%rsp), %r11 + movq 0x78(%rsp), %r12 + sbbq 0x1e0(%rsp), %r12 + movq 0x80(%rsp), %r13 + sbbq 0x1e8(%rsp), %r13 + movq 0x88(%rsp), %r14 + sbbq 0x1f0(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0x48(%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x50(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x58(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x60(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x68(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x70(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x78(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x80(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x88(%rsp) + leaq 0x168(%rsp), %rsi + leaq 0xd8(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_sqr_p521) + leaq 0x48(%rsp), %rsi + leaq (%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_sqr_p521) + leaq 0x120(%rsp), %rdx + leaq 0xd8(%rsp), %rsi + leaq 0x120(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_mul_p521) + leaq 0x90(%rsp), %rdx + leaq 0xd8(%rsp), %rsi + leaq 0x90(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_mul_p521) + movq (%rsp), %rax + subq 0x120(%rsp), %rax + movq 0x8(%rsp), %rdx + sbbq 0x128(%rsp), %rdx + movq 0x10(%rsp), %r8 + sbbq 0x130(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0x138(%rsp), %r9 + movq 0x20(%rsp), %r10 + sbbq 0x140(%rsp), %r10 + movq 0x28(%rsp), %r11 + sbbq 0x148(%rsp), %r11 + movq 0x30(%rsp), %r12 + sbbq 0x150(%rsp), %r12 + movq 0x38(%rsp), %r13 + sbbq 0x158(%rsp), %r13 + movq 0x40(%rsp), %r14 + sbbq 0x160(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, (%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x8(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x10(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x18(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x20(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x28(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x30(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x38(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x40(%rsp) + movq 0x90(%rsp), %rax + subq 0x120(%rsp), %rax + movq 0x98(%rsp), %rdx + sbbq 0x128(%rsp), %rdx + movq 0xa0(%rsp), %r8 + sbbq 0x130(%rsp), %r8 + movq 0xa8(%rsp), %r9 + sbbq 0x138(%rsp), %r9 + movq 0xb0(%rsp), %r10 + sbbq 0x140(%rsp), %r10 + movq 0xb8(%rsp), %r11 + sbbq 0x148(%rsp), %r11 + movq 0xc0(%rsp), %r12 + sbbq 0x150(%rsp), %r12 + movq 0xc8(%rsp), %r13 + sbbq 0x158(%rsp), %r13 + movq 0xd0(%rsp), %r14 + sbbq 0x160(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0xd8(%rsp) + sbbq $0x0, %rdx + movq %rdx, 0xe0(%rsp) + sbbq $0x0, %r8 + movq %r8, 0xe8(%rsp) + sbbq $0x0, %r9 + movq %r9, 0xf0(%rsp) + sbbq $0x0, %r10 + movq %r10, 0xf8(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x100(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x108(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x110(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x118(%rsp) + movq 0x200(%rsp), %rsi + leaq 0x90(%rsi), %rdx + leaq 0x168(%rsp), %rsi + leaq 0x168(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_mul_p521) + movq (%rsp), %rax + subq 0x90(%rsp), %rax + movq 0x8(%rsp), %rdx + sbbq 0x98(%rsp), %rdx + movq 0x10(%rsp), %r8 + sbbq 0xa0(%rsp), %r8 + movq 0x18(%rsp), %r9 + sbbq 0xa8(%rsp), %r9 + movq 0x20(%rsp), %r10 + sbbq 0xb0(%rsp), %r10 + movq 0x28(%rsp), %r11 + sbbq 0xb8(%rsp), %r11 + movq 0x30(%rsp), %r12 + sbbq 0xc0(%rsp), %r12 + movq 0x38(%rsp), %r13 + sbbq 0xc8(%rsp), %r13 + movq 0x40(%rsp), %r14 + sbbq 0xd0(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, (%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x8(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x10(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x18(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x20(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x28(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x30(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x38(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x40(%rsp) + movq 0x120(%rsp), %rax + subq (%rsp), %rax + movq 0x128(%rsp), %rdx + sbbq 0x8(%rsp), %rdx + movq 0x130(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x138(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movq 0x140(%rsp), %r10 + sbbq 0x20(%rsp), %r10 + movq 0x148(%rsp), %r11 + sbbq 0x28(%rsp), %r11 + movq 0x150(%rsp), %r12 + sbbq 0x30(%rsp), %r12 + movq 0x158(%rsp), %r13 + sbbq 0x38(%rsp), %r13 + movq 0x160(%rsp), %r14 + sbbq 0x40(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0x120(%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x128(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x130(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x138(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x140(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x148(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x150(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x158(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x160(%rsp) + leaq 0x1b0(%rsp), %rdx + leaq 0xd8(%rsp), %rsi + leaq 0xd8(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_mul_p521) + movq 0x208(%rsp), %rdi + leaq 0x90(%rdi), %rdx + leaq 0x168(%rsp), %rsi + leaq 0x168(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_mul_p521) + leaq 0x120(%rsp), %rdx + leaq 0x48(%rsp), %rsi + leaq 0x120(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_mul_p521) + movq 0x120(%rsp), %rax + subq 0xd8(%rsp), %rax + movq 0x128(%rsp), %rdx + sbbq 0xe0(%rsp), %rdx + movq 0x130(%rsp), %r8 + sbbq 0xe8(%rsp), %r8 + movq 0x138(%rsp), %r9 + sbbq 0xf0(%rsp), %r9 + movq 0x140(%rsp), %r10 + sbbq 0xf8(%rsp), %r10 + movq 0x148(%rsp), %r11 + sbbq 0x100(%rsp), %r11 + movq 0x150(%rsp), %r12 + sbbq 0x108(%rsp), %r12 + movq 0x158(%rsp), %r13 + sbbq 0x110(%rsp), %r13 + movq 0x160(%rsp), %r14 + sbbq 0x118(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0x120(%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x128(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x130(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x138(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x140(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x148(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x150(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x158(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x160(%rsp) + movq 0x200(%rsp), %rsi + movq 0x90(%rsi), %r8 + movq 0x98(%rsi), %r9 + movq 0xa0(%rsi), %r10 + movq 0xa8(%rsi), %r11 + movq 0xb0(%rsi), %r12 + movq 0xb8(%rsi), %r13 + movq 0xc0(%rsi), %r14 + movq 0xc8(%rsi), %r15 + movq 0xd0(%rsi), %rbp + orq %r9, %r8 + orq %r11, %r10 + orq %r13, %r12 + orq %r15, %r14 + orq %r10, %r8 + orq %r14, %r12 + orq %rbp, %r8 + orq %r12, %r8 + negq %r8 + sbbq %rax, %rax + movq 0x208(%rsp), %rdi + movq 0x90(%rdi), %r8 + movq 0x98(%rdi), %r9 + movq 0xa0(%rdi), %r10 + movq 0xa8(%rdi), %r11 + movq 0xb0(%rdi), %r12 + movq 0xb8(%rdi), %r13 + movq 0xc0(%rdi), %r14 + movq 0xc8(%rdi), %r15 + movq 0xd0(%rdi), %rbp + orq %r9, %r8 + orq %r11, %r10 + orq %r13, %r12 + orq %r15, %r14 + orq %r10, %r8 + orq %r14, %r12 + orq %rbp, %r8 + orq %r12, %r8 + negq %r8 + sbbq %rdx, %rdx + cmpq %rax, %rdx + movq 0x120(%rsp), %r8 + cmovbq 0x48(%rsi), %r8 + cmova 0x48(%rdi), %r8 + movq 0x128(%rsp), %r9 + cmovbq 0x50(%rsi), %r9 + cmova 0x50(%rdi), %r9 + movq 0x130(%rsp), %r10 + cmovbq 0x58(%rsi), %r10 + cmova 0x58(%rdi), %r10 + movq 0x138(%rsp), %r11 + cmovbq 0x60(%rsi), %r11 + cmova 0x60(%rdi), %r11 + movq 0x140(%rsp), %r12 + cmovbq 0x68(%rsi), %r12 + cmova 0x68(%rdi), %r12 + movq 0x148(%rsp), %r13 + cmovbq 0x70(%rsi), %r13 + cmova 0x70(%rdi), %r13 + movq 0x150(%rsp), %r14 + cmovbq 0x78(%rsi), %r14 + cmova 0x78(%rdi), %r14 + movq 0x158(%rsp), %r15 + cmovbq 0x80(%rsi), %r15 + cmova 0x80(%rdi), %r15 + movq 0x160(%rsp), %rbp + cmovbq 0x88(%rsi), %rbp + cmova 0x88(%rdi), %rbp + movq %r8, 0x120(%rsp) + movq %r9, 0x128(%rsp) + movq %r10, 0x130(%rsp) + movq %r11, 0x138(%rsp) + movq %r12, 0x140(%rsp) + movq %r13, 0x148(%rsp) + movq %r14, 0x150(%rsp) + movq %r15, 0x158(%rsp) + movq %rbp, 0x160(%rsp) + movq 0x168(%rsp), %r8 + cmovbq 0x90(%rsi), %r8 + cmova 0x90(%rdi), %r8 + movq 0x170(%rsp), %r9 + cmovbq 0x98(%rsi), %r9 + cmova 0x98(%rdi), %r9 + movq 0x178(%rsp), %r10 + cmovbq 0xa0(%rsi), %r10 + cmova 0xa0(%rdi), %r10 + movq 0x180(%rsp), %r11 + cmovbq 0xa8(%rsi), %r11 + cmova 0xa8(%rdi), %r11 + movq 0x188(%rsp), %r12 + cmovbq 0xb0(%rsi), %r12 + cmova 0xb0(%rdi), %r12 + movq 0x190(%rsp), %r13 + cmovbq 0xb8(%rsi), %r13 + cmova 0xb8(%rdi), %r13 + movq 0x198(%rsp), %r14 + cmovbq 0xc0(%rsi), %r14 + cmova 0xc0(%rdi), %r14 + movq 0x1a0(%rsp), %r15 + cmovbq 0xc8(%rsi), %r15 + cmova 0xc8(%rdi), %r15 + movq 0x1a8(%rsp), %rbp + cmovbq 0xd0(%rsi), %rbp + cmova 0xd0(%rdi), %rbp + movq %r8, 0x168(%rsp) + movq %r9, 0x170(%rsp) + movq %r10, 0x178(%rsp) + movq %r11, 0x180(%rsp) + movq %r12, 0x188(%rsp) + movq %r13, 0x190(%rsp) + movq %r14, 0x198(%rsp) + movq %r15, 0x1a0(%rsp) + movq %rbp, 0x1a8(%rsp) + movq (%rsp), %r8 + cmovbq (%rsi), %r8 + cmova (%rdi), %r8 + movq 0x8(%rsp), %r9 + cmovbq 0x8(%rsi), %r9 + cmova 0x8(%rdi), %r9 + movq 0x10(%rsp), %r10 + cmovbq 0x10(%rsi), %r10 + cmova 0x10(%rdi), %r10 + movq 0x18(%rsp), %r11 + cmovbq 0x18(%rsi), %r11 + cmova 0x18(%rdi), %r11 + movq 0x20(%rsp), %r12 + cmovbq 0x20(%rsi), %r12 + cmova 0x20(%rdi), %r12 + movq 0x28(%rsp), %r13 + cmovbq 0x28(%rsi), %r13 + cmova 0x28(%rdi), %r13 + movq 0x30(%rsp), %r14 + cmovbq 0x30(%rsi), %r14 + cmova 0x30(%rdi), %r14 + movq 0x38(%rsp), %r15 + cmovbq 0x38(%rsi), %r15 + cmova 0x38(%rdi), %r15 + movq 0x40(%rsp), %rbp + cmovbq 0x40(%rsi), %rbp + cmova 0x40(%rdi), %rbp + movq 0x1f8(%rsp), %rdi + movq %r8, (%rdi) + movq %r9, 0x8(%rdi) + movq %r10, 0x10(%rdi) + movq %r11, 0x18(%rdi) + movq %r12, 0x20(%rdi) + movq %r13, 0x28(%rdi) + movq %r14, 0x30(%rdi) + movq %r15, 0x38(%rdi) + movq %rbp, 0x40(%rdi) + movq 0x120(%rsp), %rax + movq %rax, 0x48(%rdi) + movq 0x128(%rsp), %rax + movq %rax, 0x50(%rdi) + movq 0x130(%rsp), %rax + movq %rax, 0x58(%rdi) + movq 0x138(%rsp), %rax + movq %rax, 0x60(%rdi) + movq 0x140(%rsp), %rax + movq %rax, 0x68(%rdi) + movq 0x148(%rsp), %rax + movq %rax, 0x70(%rdi) + movq 0x150(%rsp), %rax + movq %rax, 0x78(%rdi) + movq 0x158(%rsp), %rax + movq %rax, 0x80(%rdi) + movq 0x160(%rsp), %rax + movq %rax, 0x88(%rdi) + movq 0x168(%rsp), %rax + movq %rax, 0x90(%rdi) + movq 0x170(%rsp), %rax + movq %rax, 0x98(%rdi) + movq 0x178(%rsp), %rax + movq %rax, 0xa0(%rdi) + movq 0x180(%rsp), %rax + movq %rax, 0xa8(%rdi) + movq 0x188(%rsp), %rax + movq %rax, 0xb0(%rdi) + movq 0x190(%rsp), %rax + movq %rax, 0xb8(%rdi) + movq 0x198(%rsp), %rax + movq %rax, 0xc0(%rdi) + movq 0x1a0(%rsp), %rax + movq %rax, 0xc8(%rdi) + movq 0x1a8(%rsp), %rax + movq %rax, 0xd0(%rdi) + CFI_INC_RSP(528) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_alt_jadd) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_alt_jdouble) + +Lp521_jscalarmul_alt_jdouble: + CFI_START + CFI_PUSH(%rbx) + CFI_PUSH(%rbp) + CFI_PUSH(%r12) + CFI_PUSH(%r13) + CFI_PUSH(%r14) + CFI_PUSH(%r15) + CFI_DEC_RSP(520) + movq %rdi, 0x1f8(%rsp) + movq %rsi, 0x200(%rsp) + movq 0x200(%rsp), %rdi + leaq 0x90(%rdi), %rsi + leaq (%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_sqr_p521) + movq 0x200(%rsp), %rdi + leaq 0x48(%rdi), %rsi + leaq 0x48(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_sqr_p521) + movq 0x200(%rsp), %rdi + stc + movq (%rdi), %rax + adcq (%rsp), %rax + movq 0x8(%rdi), %rbx + adcq 0x8(%rsp), %rbx + movq 0x10(%rdi), %r8 + adcq 0x10(%rsp), %r8 + movq 0x18(%rdi), %r9 + adcq 0x18(%rsp), %r9 + movq 0x20(%rdi), %r10 + adcq 0x20(%rsp), %r10 + movq 0x28(%rdi), %r11 + adcq 0x28(%rsp), %r11 + movq 0x30(%rdi), %r12 + adcq 0x30(%rsp), %r12 + movq 0x38(%rdi), %r13 + adcq 0x38(%rsp), %r13 + movq 0x40(%rdi), %r14 + adcq 0x40(%rsp), %r14 + movq $0x200, %rdx + andq %r14, %rdx + cmpq $0x200, %rdx + sbbq $0x0, %rax + movq %rax, 0x168(%rsp) + sbbq $0x0, %rbx + movq %rbx, 0x170(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x178(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x180(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x188(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x190(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x198(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x1a0(%rsp) + sbbq %rdx, %r14 + movq %r14, 0x1a8(%rsp) + movq 0x200(%rsp), %rdi + movq (%rdi), %rax + subq (%rsp), %rax + movq 0x8(%rdi), %rdx + sbbq 0x8(%rsp), %rdx + movq 0x10(%rdi), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x18(%rdi), %r9 + sbbq 0x18(%rsp), %r9 + movq 0x20(%rdi), %r10 + sbbq 0x20(%rsp), %r10 + movq 0x28(%rdi), %r11 + sbbq 0x28(%rsp), %r11 + movq 0x30(%rdi), %r12 + sbbq 0x30(%rsp), %r12 + movq 0x38(%rdi), %r13 + sbbq 0x38(%rsp), %r13 + movq 0x40(%rdi), %r14 + sbbq 0x40(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0x120(%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x128(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x130(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x138(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x140(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x148(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x150(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x158(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x160(%rsp) + leaq 0x120(%rsp), %rdx + leaq 0x168(%rsp), %rsi + leaq 0x90(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_mul_p521) + movq 0x200(%rsp), %rdi + stc + movq 0x48(%rdi), %rax + adcq 0x90(%rdi), %rax + movq 0x50(%rdi), %rbx + adcq 0x98(%rdi), %rbx + movq 0x58(%rdi), %r8 + adcq 0xa0(%rdi), %r8 + movq 0x60(%rdi), %r9 + adcq 0xa8(%rdi), %r9 + movq 0x68(%rdi), %r10 + adcq 0xb0(%rdi), %r10 + movq 0x70(%rdi), %r11 + adcq 0xb8(%rdi), %r11 + movq 0x78(%rdi), %r12 + adcq 0xc0(%rdi), %r12 + movq 0x80(%rdi), %r13 + adcq 0xc8(%rdi), %r13 + movq 0x88(%rdi), %r14 + adcq 0xd0(%rdi), %r14 + movq $0x200, %rdx + andq %r14, %rdx + cmpq $0x200, %rdx + sbbq $0x0, %rax + movq %rax, 0x168(%rsp) + sbbq $0x0, %rbx + movq %rbx, 0x170(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x178(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x180(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x188(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x190(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x198(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x1a0(%rsp) + sbbq %rdx, %r14 + movq %r14, 0x1a8(%rsp) + leaq 0x90(%rsp), %rsi + leaq 0x1b0(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_sqr_p521) + movq 0x200(%rsp), %rdi + leaq 0x48(%rsp), %rdx + leaq (%rdi), %rsi + leaq 0xd8(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_mul_p521) + leaq 0x168(%rsp), %rsi + leaq 0x120(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_sqr_p521) + movq $0x9, %rcx + movq 0x1b0(%rsp), %rax + notq %rax + mulq %rcx + movq %rax, %r8 + movq %rdx, %r9 + movq 0x1b8(%rsp), %rax + notq %rax + mulq %rcx + xorl %r10d, %r10d + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x1c0(%rsp), %rax + notq %rax + mulq %rcx + xorl %r11d, %r11d + addq %rax, %r10 + adcq %rdx, %r11 + movq 0x1c8(%rsp), %rax + notq %rax + mulq %rcx + xorl %r12d, %r12d + addq %rax, %r11 + adcq %rdx, %r12 + movq 0x1d0(%rsp), %rax + notq %rax + mulq %rcx + xorl %r13d, %r13d + addq %rax, %r12 + adcq %rdx, %r13 + movq 0x1d8(%rsp), %rax + notq %rax + mulq %rcx + xorl %r14d, %r14d + addq %rax, %r13 + adcq %rdx, %r14 + movq 0x1e0(%rsp), %rax + notq %rax + mulq %rcx + xorl %r15d, %r15d + addq %rax, %r14 + adcq %rdx, %r15 + movq 0x1e8(%rsp), %rax + notq %rax + mulq %rcx + xorl %ebx, %ebx + addq %rax, %r15 + adcq %rdx, %rbx + movq 0x1f0(%rsp), %rax + xorq $0x1ff, %rax + imulq %rcx, %rax + addq %rax, %rbx + xorl %eax, %eax + movl $0xc, %ecx + movq 0xd8(%rsp), %rax + mulq %rcx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %rbp, %rbp + movq 0xe0(%rsp), %rax + mulq %rcx + subq %rbp, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rbp, %rbp + movq 0xe8(%rsp), %rax + mulq %rcx + subq %rbp, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0xf0(%rsp), %rax + mulq %rcx + subq %rbp, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rbp, %rbp + movq 0xf8(%rsp), %rax + mulq %rcx + subq %rbp, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x100(%rsp), %rax + mulq %rcx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x108(%rsp), %rax + mulq %rcx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x110(%rsp), %rax + mulq %rcx + subq %rbp, %rdx + addq %rax, %r15 + adcq %rdx, %rbx + movq 0x118(%rsp), %rax + imulq %rcx, %rax + addq %rax, %rbx + movq %r9, %rax + andq %r10, %rax + andq %r11, %rax + andq %r12, %rax + andq %r13, %rax + andq %r14, %rax + andq %r15, %rax + movq %rbx, %rdx + shrq $0x9, %rdx + orq $0xfffffffffffffe00, %rbx + leaq 0x1(%rdx), %rcx + addq %r8, %rcx + movl $0x0, %ecx + adcq %rcx, %rax + movq %rbx, %rax + adcq %rcx, %rax + adcq %rdx, %r8 + movq %r8, 0x1b0(%rsp) + adcq %rcx, %r9 + movq %r9, 0x1b8(%rsp) + adcq %rcx, %r10 + movq %r10, 0x1c0(%rsp) + adcq %rcx, %r11 + movq %r11, 0x1c8(%rsp) + adcq %rcx, %r12 + movq %r12, 0x1d0(%rsp) + adcq %rcx, %r13 + movq %r13, 0x1d8(%rsp) + adcq %rcx, %r14 + movq %r14, 0x1e0(%rsp) + adcq %rcx, %r15 + movq %r15, 0x1e8(%rsp) + adcq %rcx, %rbx + andq $0x1ff, %rbx + movq %rbx, 0x1f0(%rsp) + movq 0x120(%rsp), %rax + subq (%rsp), %rax + movq 0x128(%rsp), %rdx + sbbq 0x8(%rsp), %rdx + movq 0x130(%rsp), %r8 + sbbq 0x10(%rsp), %r8 + movq 0x138(%rsp), %r9 + sbbq 0x18(%rsp), %r9 + movq 0x140(%rsp), %r10 + sbbq 0x20(%rsp), %r10 + movq 0x148(%rsp), %r11 + sbbq 0x28(%rsp), %r11 + movq 0x150(%rsp), %r12 + sbbq 0x30(%rsp), %r12 + movq 0x158(%rsp), %r13 + sbbq 0x38(%rsp), %r13 + movq 0x160(%rsp), %r14 + sbbq 0x40(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0x168(%rsp) + sbbq $0x0, %rdx + movq %rdx, 0x170(%rsp) + sbbq $0x0, %r8 + movq %r8, 0x178(%rsp) + sbbq $0x0, %r9 + movq %r9, 0x180(%rsp) + sbbq $0x0, %r10 + movq %r10, 0x188(%rsp) + sbbq $0x0, %r11 + movq %r11, 0x190(%rsp) + sbbq $0x0, %r12 + movq %r12, 0x198(%rsp) + sbbq $0x0, %r13 + movq %r13, 0x1a0(%rsp) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0x1a8(%rsp) + leaq 0x48(%rsp), %rsi + leaq 0x120(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_sqr_p521) + movq 0x1f8(%rsp), %rdi + movq 0x168(%rsp), %rax + subq 0x48(%rsp), %rax + movq 0x170(%rsp), %rdx + sbbq 0x50(%rsp), %rdx + movq 0x178(%rsp), %r8 + sbbq 0x58(%rsp), %r8 + movq 0x180(%rsp), %r9 + sbbq 0x60(%rsp), %r9 + movq 0x188(%rsp), %r10 + sbbq 0x68(%rsp), %r10 + movq 0x190(%rsp), %r11 + sbbq 0x70(%rsp), %r11 + movq 0x198(%rsp), %r12 + sbbq 0x78(%rsp), %r12 + movq 0x1a0(%rsp), %r13 + sbbq 0x80(%rsp), %r13 + movq 0x1a8(%rsp), %r14 + sbbq 0x88(%rsp), %r14 + sbbq $0x0, %rax + movq %rax, 0x90(%rdi) + sbbq $0x0, %rdx + movq %rdx, 0x98(%rdi) + sbbq $0x0, %r8 + movq %r8, 0xa0(%rdi) + sbbq $0x0, %r9 + movq %r9, 0xa8(%rdi) + sbbq $0x0, %r10 + movq %r10, 0xb0(%rdi) + sbbq $0x0, %r11 + movq %r11, 0xb8(%rdi) + sbbq $0x0, %r12 + movq %r12, 0xc0(%rdi) + sbbq $0x0, %r13 + movq %r13, 0xc8(%rdi) + sbbq $0x0, %r14 + andq $0x1ff, %r14 + movq %r14, 0xd0(%rdi) + leaq 0x90(%rsp), %rdx + leaq 0x1b0(%rsp), %rsi + leaq 0x168(%rsp), %rdi + CFI_CALL(Lp521_jscalarmul_alt_mul_p521) + movq 0x1f8(%rsp), %rdi + movq 0x118(%rsp), %rbx + movq 0x110(%rsp), %r15 + shldq $0x2, %r15, %rbx + movq 0x108(%rsp), %r14 + shldq $0x2, %r14, %r15 + movq 0x100(%rsp), %r13 + shldq $0x2, %r13, %r14 + movq 0xf8(%rsp), %r12 + shldq $0x2, %r12, %r13 + movq 0xf0(%rsp), %r11 + shldq $0x2, %r11, %r12 + movq 0xe8(%rsp), %r10 + shldq $0x2, %r10, %r11 + movq 0xe0(%rsp), %r9 + shldq $0x2, %r9, %r10 + movq 0xd8(%rsp), %r8 + shldq $0x2, %r8, %r9 + shlq $0x2, %r8 + movq 0x1f0(%rsp), %rcx + xorq $0x1ff, %rcx + movq 0x1b0(%rsp), %rax + notq %rax + addq %rax, %r8 + movq 0x1b8(%rsp), %rax + notq %rax + adcq %rax, %r9 + movq 0x1c0(%rsp), %rax + notq %rax + adcq %rax, %r10 + movq 0x1c8(%rsp), %rax + notq %rax + adcq %rax, %r11 + movq 0x1d0(%rsp), %rax + notq %rax + adcq %rax, %r12 + movq 0x1d8(%rsp), %rax + notq %rax + adcq %rax, %r13 + movq 0x1e0(%rsp), %rax + notq %rax + adcq %rax, %r14 + movq 0x1e8(%rsp), %rax + notq %rax + adcq %rax, %r15 + adcq %rcx, %rbx + movq %r9, %rax + andq %r10, %rax + andq %r11, %rax + andq %r12, %rax + andq %r13, %rax + andq %r14, %rax + andq %r15, %rax + movq %rbx, %rdx + shrq $0x9, %rdx + orq $0xfffffffffffffe00, %rbx + leaq 0x1(%rdx), %rcx + addq %r8, %rcx + movl $0x0, %ecx + adcq %rcx, %rax + movq %rbx, %rax + adcq %rcx, %rax + adcq %rdx, %r8 + movq %r8, (%rdi) + adcq %rcx, %r9 + movq %r9, 0x8(%rdi) + adcq %rcx, %r10 + movq %r10, 0x10(%rdi) + adcq %rcx, %r11 + movq %r11, 0x18(%rdi) + adcq %rcx, %r12 + movq %r12, 0x20(%rdi) + adcq %rcx, %r13 + movq %r13, 0x28(%rdi) + adcq %rcx, %r14 + movq %r14, 0x30(%rdi) + adcq %rcx, %r15 + movq %r15, 0x38(%rdi) + adcq %rcx, %rbx + andq $0x1ff, %rbx + movq %rbx, 0x40(%rdi) + movq 0x1f8(%rsp), %rdi + movq 0x160(%rsp), %rbx + xorq $0x1ff, %rbx + movq 0x158(%rsp), %r15 + notq %r15 + shldq $0x3, %r15, %rbx + movq 0x150(%rsp), %r14 + notq %r14 + shldq $0x3, %r14, %r15 + movq 0x148(%rsp), %r13 + notq %r13 + shldq $0x3, %r13, %r14 + movq 0x140(%rsp), %r12 + notq %r12 + shldq $0x3, %r12, %r13 + movq 0x138(%rsp), %r11 + notq %r11 + shldq $0x3, %r11, %r12 + movq 0x130(%rsp), %r10 + notq %r10 + shldq $0x3, %r10, %r11 + movq 0x128(%rsp), %r9 + notq %r9 + shldq $0x3, %r9, %r10 + movq 0x120(%rsp), %r8 + notq %r8 + shldq $0x3, %r8, %r9 + shlq $0x3, %r8 + movl $0x3, %ecx + movq 0x168(%rsp), %rax + mulq %rcx + addq %rax, %r8 + adcq %rdx, %r9 + sbbq %rbp, %rbp + movq 0x170(%rsp), %rax + mulq %rcx + subq %rbp, %rdx + addq %rax, %r9 + adcq %rdx, %r10 + sbbq %rbp, %rbp + movq 0x178(%rsp), %rax + mulq %rcx + subq %rbp, %rdx + addq %rax, %r10 + adcq %rdx, %r11 + sbbq %rbp, %rbp + movq 0x180(%rsp), %rax + mulq %rcx + subq %rbp, %rdx + addq %rax, %r11 + adcq %rdx, %r12 + sbbq %rbp, %rbp + movq 0x188(%rsp), %rax + mulq %rcx + subq %rbp, %rdx + addq %rax, %r12 + adcq %rdx, %r13 + sbbq %rbp, %rbp + movq 0x190(%rsp), %rax + mulq %rcx + subq %rbp, %rdx + addq %rax, %r13 + adcq %rdx, %r14 + sbbq %rbp, %rbp + movq 0x198(%rsp), %rax + mulq %rcx + subq %rbp, %rdx + addq %rax, %r14 + adcq %rdx, %r15 + sbbq %rbp, %rbp + movq 0x1a0(%rsp), %rax + mulq %rcx + subq %rbp, %rdx + addq %rax, %r15 + adcq %rdx, %rbx + movq 0x1a8(%rsp), %rax + imulq %rcx, %rax + addq %rax, %rbx + movq %r9, %rax + andq %r10, %rax + andq %r11, %rax + andq %r12, %rax + andq %r13, %rax + andq %r14, %rax + andq %r15, %rax + movq %rbx, %rdx + shrq $0x9, %rdx + orq $0xfffffffffffffe00, %rbx + leaq 0x1(%rdx), %rcx + addq %r8, %rcx + movl $0x0, %ecx + adcq %rcx, %rax + movq %rbx, %rax + adcq %rcx, %rax + adcq %rdx, %r8 + movq %r8, 0x48(%rdi) + adcq %rcx, %r9 + movq %r9, 0x50(%rdi) + adcq %rcx, %r10 + movq %r10, 0x58(%rdi) + adcq %rcx, %r11 + movq %r11, 0x60(%rdi) + adcq %rcx, %r12 + movq %r12, 0x68(%rdi) + adcq %rcx, %r13 + movq %r13, 0x70(%rdi) + adcq %rcx, %r14 + movq %r14, 0x78(%rdi) + adcq %rcx, %r15 + movq %r15, 0x80(%rdi) + adcq %rcx, %rbx + andq $0x1ff, %rbx + movq %rbx, 0x88(%rdi) + CFI_INC_RSP(520) + CFI_POP(%r15) + CFI_POP(%r14) + CFI_POP(%r13) + CFI_POP(%r12) + CFI_POP(%rbp) + CFI_POP(%rbx) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_alt_jdouble) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_alt_mul_p521) + +Lp521_jscalarmul_alt_mul_p521: + CFI_START + CFI_DEC_RSP(72) + movq %rdx, %rcx + movq (%rsi), %rax + mulq (%rcx) + movq %rax, (%rsp) + movq %rdx, %r9 + xorq %r10, %r10 + xorq %r11, %r11 + movq (%rsi), %rax + mulq 0x8(%rcx) + addq %rax, %r9 + adcq %rdx, %r10 + movq 0x8(%rsi), %rax + mulq (%rcx) + addq %rax, %r9 + adcq %rdx, %r10 + adcq %r11, %r11 + movq %r9, 0x8(%rsp) + xorq %r12, %r12 + movq (%rsi), %rax + mulq 0x10(%rcx) + addq %rax, %r10 + adcq %rdx, %r11 + adcq %r12, %r12 + movq 0x8(%rsi), %rax + mulq 0x8(%rcx) + addq %rax, %r10 + adcq %rdx, %r11 + adcq $0x0, %r12 + movq 0x10(%rsi), %rax + mulq (%rcx) + addq %rax, %r10 + adcq %rdx, %r11 + adcq $0x0, %r12 + movq %r10, 0x10(%rsp) + xorq %r13, %r13 + movq (%rsi), %rax + mulq 0x18(%rcx) + addq %rax, %r11 + adcq %rdx, %r12 + adcq %r13, %r13 + movq 0x8(%rsi), %rax + mulq 0x10(%rcx) + addq %rax, %r11 + adcq %rdx, %r12 + adcq $0x0, %r13 + movq 0x10(%rsi), %rax + mulq 0x8(%rcx) + addq %rax, %r11 + adcq %rdx, %r12 + adcq $0x0, %r13 + movq 0x18(%rsi), %rax + mulq (%rcx) + addq %rax, %r11 + adcq %rdx, %r12 + adcq $0x0, %r13 + movq %r11, 0x18(%rsp) + xorq %r14, %r14 + movq (%rsi), %rax + mulq 0x20(%rcx) + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x8(%rsi), %rax + mulq 0x18(%rcx) + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + movq 0x10(%rsi), %rax + mulq 0x10(%rcx) + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + movq 0x18(%rsi), %rax + mulq 0x8(%rcx) + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + movq 0x20(%rsi), %rax + mulq (%rcx) + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + movq %r12, 0x20(%rsp) + xorq %r15, %r15 + movq (%rsi), %rax + mulq 0x28(%rcx) + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x8(%rsi), %rax + mulq 0x20(%rcx) + addq %rax, %r13 + adcq %rdx, %r14 + adcq $0x0, %r15 + movq 0x10(%rsi), %rax + mulq 0x18(%rcx) + addq %rax, %r13 + adcq %rdx, %r14 + adcq $0x0, %r15 + movq 0x18(%rsi), %rax + mulq 0x10(%rcx) + addq %rax, %r13 + adcq %rdx, %r14 + adcq $0x0, %r15 + movq 0x20(%rsi), %rax + mulq 0x8(%rcx) + addq %rax, %r13 + adcq %rdx, %r14 + adcq $0x0, %r15 + movq 0x28(%rsi), %rax + mulq (%rcx) + addq %rax, %r13 + adcq %rdx, %r14 + adcq $0x0, %r15 + movq %r13, 0x28(%rsp) + xorq %r8, %r8 + movq (%rsi), %rax + mulq 0x30(%rcx) + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + movq 0x8(%rsi), %rax + mulq 0x28(%rcx) + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %r8 + movq 0x10(%rsi), %rax + mulq 0x20(%rcx) + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %r8 + movq 0x18(%rsi), %rax + mulq 0x18(%rcx) + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %r8 + movq 0x20(%rsi), %rax + mulq 0x10(%rcx) + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %r8 + movq 0x28(%rsi), %rax + mulq 0x8(%rcx) + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %r8 + movq 0x30(%rsi), %rax + mulq (%rcx) + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %r8 + movq %r14, 0x30(%rsp) + xorq %r9, %r9 + movq (%rsi), %rax + mulq 0x38(%rcx) + addq %rax, %r15 + adcq %rdx, %r8 + adcq %r9, %r9 + movq 0x8(%rsi), %rax + mulq 0x30(%rcx) + addq %rax, %r15 + adcq %rdx, %r8 + adcq $0x0, %r9 + movq 0x10(%rsi), %rax + mulq 0x28(%rcx) + addq %rax, %r15 + adcq %rdx, %r8 + adcq $0x0, %r9 + movq 0x18(%rsi), %rax + mulq 0x20(%rcx) + addq %rax, %r15 + adcq %rdx, %r8 + adcq $0x0, %r9 + movq 0x20(%rsi), %rax + mulq 0x18(%rcx) + addq %rax, %r15 + adcq %rdx, %r8 + adcq $0x0, %r9 + movq 0x28(%rsi), %rax + mulq 0x10(%rcx) + addq %rax, %r15 + adcq %rdx, %r8 + adcq $0x0, %r9 + movq 0x30(%rsi), %rax + mulq 0x8(%rcx) + addq %rax, %r15 + adcq %rdx, %r8 + adcq $0x0, %r9 + movq 0x38(%rsi), %rax + mulq (%rcx) + addq %rax, %r15 + adcq %rdx, %r8 + adcq $0x0, %r9 + movq %r15, 0x38(%rsp) + xorq %r10, %r10 + movq (%rsi), %rax + mulq 0x40(%rcx) + addq %rax, %r8 + adcq %rdx, %r9 + adcq %r10, %r10 + movq 0x8(%rsi), %rax + mulq 0x38(%rcx) + addq %rax, %r8 + adcq %rdx, %r9 + adcq $0x0, %r10 + movq 0x10(%rsi), %rax + mulq 0x30(%rcx) + addq %rax, %r8 + adcq %rdx, %r9 + adcq $0x0, %r10 + movq 0x18(%rsi), %rax + mulq 0x28(%rcx) + addq %rax, %r8 + adcq %rdx, %r9 + adcq $0x0, %r10 + movq 0x20(%rsi), %rax + mulq 0x20(%rcx) + addq %rax, %r8 + adcq %rdx, %r9 + adcq $0x0, %r10 + movq 0x28(%rsi), %rax + mulq 0x18(%rcx) + addq %rax, %r8 + adcq %rdx, %r9 + adcq $0x0, %r10 + movq 0x30(%rsi), %rax + mulq 0x10(%rcx) + addq %rax, %r8 + adcq %rdx, %r9 + adcq $0x0, %r10 + movq 0x38(%rsi), %rax + mulq 0x8(%rcx) + addq %rax, %r8 + adcq %rdx, %r9 + adcq $0x0, %r10 + movq 0x40(%rsi), %rax + mulq (%rcx) + addq %rax, %r8 + adcq %rdx, %r9 + adcq $0x0, %r10 + movq %r8, 0x40(%rsp) + xorq %r11, %r11 + movq 0x8(%rsi), %rax + mulq 0x40(%rcx) + addq %rax, %r9 + adcq %rdx, %r10 + adcq %r11, %r11 + movq 0x10(%rsi), %rax + mulq 0x38(%rcx) + addq %rax, %r9 + adcq %rdx, %r10 + adcq $0x0, %r11 + movq 0x18(%rsi), %rax + mulq 0x30(%rcx) + addq %rax, %r9 + adcq %rdx, %r10 + adcq $0x0, %r11 + movq 0x20(%rsi), %rax + mulq 0x28(%rcx) + addq %rax, %r9 + adcq %rdx, %r10 + adcq $0x0, %r11 + movq 0x28(%rsi), %rax + mulq 0x20(%rcx) + addq %rax, %r9 + adcq %rdx, %r10 + adcq $0x0, %r11 + movq 0x30(%rsi), %rax + mulq 0x18(%rcx) + addq %rax, %r9 + adcq %rdx, %r10 + adcq $0x0, %r11 + movq 0x38(%rsi), %rax + mulq 0x10(%rcx) + addq %rax, %r9 + adcq %rdx, %r10 + adcq $0x0, %r11 + movq 0x40(%rsi), %rax + mulq 0x8(%rcx) + addq %rax, %r9 + adcq %rdx, %r10 + adcq $0x0, %r11 + xorq %r12, %r12 + movq 0x10(%rsi), %rax + mulq 0x40(%rcx) + addq %rax, %r10 + adcq %rdx, %r11 + adcq %r12, %r12 + movq 0x18(%rsi), %rax + mulq 0x38(%rcx) + addq %rax, %r10 + adcq %rdx, %r11 + adcq $0x0, %r12 + movq 0x20(%rsi), %rax + mulq 0x30(%rcx) + addq %rax, %r10 + adcq %rdx, %r11 + adcq $0x0, %r12 + movq 0x28(%rsi), %rax + mulq 0x28(%rcx) + addq %rax, %r10 + adcq %rdx, %r11 + adcq $0x0, %r12 + movq 0x30(%rsi), %rax + mulq 0x20(%rcx) + addq %rax, %r10 + adcq %rdx, %r11 + adcq $0x0, %r12 + movq 0x38(%rsi), %rax + mulq 0x18(%rcx) + addq %rax, %r10 + adcq %rdx, %r11 + adcq $0x0, %r12 + movq 0x40(%rsi), %rax + mulq 0x10(%rcx) + addq %rax, %r10 + adcq %rdx, %r11 + adcq $0x0, %r12 + xorq %r13, %r13 + movq 0x18(%rsi), %rax + mulq 0x40(%rcx) + addq %rax, %r11 + adcq %rdx, %r12 + adcq %r13, %r13 + movq 0x20(%rsi), %rax + mulq 0x38(%rcx) + addq %rax, %r11 + adcq %rdx, %r12 + adcq $0x0, %r13 + movq 0x28(%rsi), %rax + mulq 0x30(%rcx) + addq %rax, %r11 + adcq %rdx, %r12 + adcq $0x0, %r13 + movq 0x30(%rsi), %rax + mulq 0x28(%rcx) + addq %rax, %r11 + adcq %rdx, %r12 + adcq $0x0, %r13 + movq 0x38(%rsi), %rax + mulq 0x20(%rcx) + addq %rax, %r11 + adcq %rdx, %r12 + adcq $0x0, %r13 + movq 0x40(%rsi), %rax + mulq 0x18(%rcx) + addq %rax, %r11 + adcq %rdx, %r12 + adcq $0x0, %r13 + xorq %r14, %r14 + movq 0x20(%rsi), %rax + mulq 0x40(%rcx) + addq %rax, %r12 + adcq %rdx, %r13 + adcq %r14, %r14 + movq 0x28(%rsi), %rax + mulq 0x38(%rcx) + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + movq 0x30(%rsi), %rax + mulq 0x30(%rcx) + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + movq 0x38(%rsi), %rax + mulq 0x28(%rcx) + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + movq 0x40(%rsi), %rax + mulq 0x20(%rcx) + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + xorq %r15, %r15 + movq 0x28(%rsi), %rax + mulq 0x40(%rcx) + addq %rax, %r13 + adcq %rdx, %r14 + adcq %r15, %r15 + movq 0x30(%rsi), %rax + mulq 0x38(%rcx) + addq %rax, %r13 + adcq %rdx, %r14 + adcq $0x0, %r15 + movq 0x38(%rsi), %rax + mulq 0x30(%rcx) + addq %rax, %r13 + adcq %rdx, %r14 + adcq $0x0, %r15 + movq 0x40(%rsi), %rax + mulq 0x28(%rcx) + addq %rax, %r13 + adcq %rdx, %r14 + adcq $0x0, %r15 + xorq %r8, %r8 + movq 0x30(%rsi), %rax + mulq 0x40(%rcx) + addq %rax, %r14 + adcq %rdx, %r15 + adcq %r8, %r8 + movq 0x38(%rsi), %rax + mulq 0x38(%rcx) + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %r8 + movq 0x40(%rsi), %rax + mulq 0x30(%rcx) + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %r8 + movq 0x38(%rsi), %rax + mulq 0x40(%rcx) + addq %rax, %r15 + adcq %rdx, %r8 + movq 0x40(%rsi), %rax + mulq 0x38(%rcx) + addq %rax, %r15 + adcq %rdx, %r8 + movq 0x40(%rsi), %rax + imulq 0x40(%rcx), %rax + addq %r8, %rax + movq 0x40(%rsp), %r8 + movq %r8, %rdx + andq $0x1ff, %rdx + shrdq $0x9, %r9, %r8 + shrdq $0x9, %r10, %r9 + shrdq $0x9, %r11, %r10 + shrdq $0x9, %r12, %r11 + shrdq $0x9, %r13, %r12 + shrdq $0x9, %r14, %r13 + shrdq $0x9, %r15, %r14 + shrdq $0x9, %rax, %r15 + shrq $0x9, %rax + addq %rax, %rdx + stc + adcq (%rsp), %r8 + adcq 0x8(%rsp), %r9 + adcq 0x10(%rsp), %r10 + adcq 0x18(%rsp), %r11 + adcq 0x20(%rsp), %r12 + adcq 0x28(%rsp), %r13 + adcq 0x30(%rsp), %r14 + adcq 0x38(%rsp), %r15 + adcq $0xfffffffffffffe00, %rdx + cmc + sbbq $0x0, %r8 + movq %r8, (%rdi) + sbbq $0x0, %r9 + movq %r9, 0x8(%rdi) + sbbq $0x0, %r10 + movq %r10, 0x10(%rdi) + sbbq $0x0, %r11 + movq %r11, 0x18(%rdi) + sbbq $0x0, %r12 + movq %r12, 0x20(%rdi) + sbbq $0x0, %r13 + movq %r13, 0x28(%rdi) + sbbq $0x0, %r14 + movq %r14, 0x30(%rdi) + sbbq $0x0, %r15 + movq %r15, 0x38(%rdi) + sbbq $0x0, %rdx + andq $0x1ff, %rdx + movq %rdx, 0x40(%rdi) + CFI_INC_RSP(72) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_alt_mul_p521) + +S2N_BN_FUNCTION_TYPE_DIRECTIVE(Lp521_jscalarmul_alt_sqr_p521) + +Lp521_jscalarmul_alt_sqr_p521: + CFI_START + CFI_DEC_RSP(72) + movq (%rsi), %rax + mulq %rax + movq %rax, (%rsp) + movq %rdx, %r9 + xorq %r10, %r10 + xorq %r11, %r11 + movq (%rsi), %rax + mulq 0x8(%rsi) + addq %rax, %rax + adcq %rdx, %rdx + adcq $0x0, %r11 + addq %rax, %r9 + adcq %rdx, %r10 + adcq $0x0, %r11 + movq %r9, 0x8(%rsp) + xorq %r12, %r12 + movq 0x8(%rsi), %rax + mulq %rax + addq %rax, %r10 + adcq %rdx, %r11 + adcq $0x0, %r12 + movq (%rsi), %rax + mulq 0x10(%rsi) + addq %rax, %rax + adcq %rdx, %rdx + adcq $0x0, %r12 + addq %rax, %r10 + adcq %rdx, %r11 + adcq $0x0, %r12 + movq %r10, 0x10(%rsp) + movq (%rsi), %rax + mulq 0x18(%rsi) + xorq %r13, %r13 + movq %rax, %rbx + movq %rdx, %rcx + movq 0x8(%rsi), %rax + mulq 0x10(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r13 + addq %rbx, %rbx + adcq %rcx, %rcx + adcq %r13, %r13 + addq %rbx, %r11 + adcq %rcx, %r12 + adcq $0x0, %r13 + movq %r11, 0x18(%rsp) + movq (%rsi), %rax + mulq 0x20(%rsi) + xorq %r14, %r14 + movq %rax, %rbx + movq %rdx, %rcx + movq 0x8(%rsi), %rax + mulq 0x18(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r14 + addq %rbx, %rbx + adcq %rcx, %rcx + adcq %r14, %r14 + addq %rbx, %r12 + adcq %rcx, %r13 + adcq $0x0, %r14 + movq 0x10(%rsi), %rax + mulq %rax + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + movq %r12, 0x20(%rsp) + movq (%rsi), %rax + mulq 0x28(%rsi) + xorq %r15, %r15 + movq %rax, %rbx + movq %rdx, %rcx + movq 0x8(%rsi), %rax + mulq 0x20(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r15 + movq 0x10(%rsi), %rax + mulq 0x18(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r15 + addq %rbx, %rbx + adcq %rcx, %rcx + adcq %r15, %r15 + addq %rbx, %r13 + adcq %rcx, %r14 + adcq $0x0, %r15 + movq %r13, 0x28(%rsp) + movq (%rsi), %rax + mulq 0x30(%rsi) + xorq %r8, %r8 + movq %rax, %rbx + movq %rdx, %rcx + movq 0x8(%rsi), %rax + mulq 0x28(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r8 + movq 0x10(%rsi), %rax + mulq 0x20(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r8 + addq %rbx, %rbx + adcq %rcx, %rcx + adcq %r8, %r8 + addq %rbx, %r14 + adcq %rcx, %r15 + adcq $0x0, %r8 + movq 0x18(%rsi), %rax + mulq %rax + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %r8 + movq %r14, 0x30(%rsp) + movq (%rsi), %rax + mulq 0x38(%rsi) + xorq %r9, %r9 + movq %rax, %rbx + movq %rdx, %rcx + movq 0x8(%rsi), %rax + mulq 0x30(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r9 + movq 0x10(%rsi), %rax + mulq 0x28(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r9 + movq 0x18(%rsi), %rax + mulq 0x20(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r9 + addq %rbx, %rbx + adcq %rcx, %rcx + adcq %r9, %r9 + addq %rbx, %r15 + adcq %rcx, %r8 + adcq $0x0, %r9 + movq %r15, 0x38(%rsp) + movq (%rsi), %rax + mulq 0x40(%rsi) + xorq %r10, %r10 + movq %rax, %rbx + movq %rdx, %rcx + movq 0x8(%rsi), %rax + mulq 0x38(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r10 + movq 0x10(%rsi), %rax + mulq 0x30(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r10 + movq 0x18(%rsi), %rax + mulq 0x28(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r10 + addq %rbx, %rbx + adcq %rcx, %rcx + adcq %r10, %r10 + addq %rbx, %r8 + adcq %rcx, %r9 + adcq $0x0, %r10 + movq 0x20(%rsi), %rax + mulq %rax + addq %rax, %r8 + adcq %rdx, %r9 + adcq $0x0, %r10 + movq %r8, 0x40(%rsp) + movq 0x8(%rsi), %rax + mulq 0x40(%rsi) + xorq %r11, %r11 + movq %rax, %rbx + movq %rdx, %rcx + movq 0x10(%rsi), %rax + mulq 0x38(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r11 + movq 0x18(%rsi), %rax + mulq 0x30(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r11 + movq 0x20(%rsi), %rax + mulq 0x28(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r11 + addq %rbx, %rbx + adcq %rcx, %rcx + adcq %r11, %r11 + addq %rbx, %r9 + adcq %rcx, %r10 + adcq $0x0, %r11 + movq 0x10(%rsi), %rax + mulq 0x40(%rsi) + xorq %r12, %r12 + movq %rax, %rbx + movq %rdx, %rcx + movq 0x18(%rsi), %rax + mulq 0x38(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r12 + movq 0x20(%rsi), %rax + mulq 0x30(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r12 + addq %rbx, %rbx + adcq %rcx, %rcx + adcq %r12, %r12 + addq %rbx, %r10 + adcq %rcx, %r11 + adcq $0x0, %r12 + movq 0x28(%rsi), %rax + mulq %rax + addq %rax, %r10 + adcq %rdx, %r11 + adcq $0x0, %r12 + movq 0x18(%rsi), %rax + mulq 0x40(%rsi) + xorq %r13, %r13 + movq %rax, %rbx + movq %rdx, %rcx + movq 0x20(%rsi), %rax + mulq 0x38(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r13 + movq 0x28(%rsi), %rax + mulq 0x30(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r13 + addq %rbx, %rbx + adcq %rcx, %rcx + adcq %r13, %r13 + addq %rbx, %r11 + adcq %rcx, %r12 + adcq $0x0, %r13 + movq 0x20(%rsi), %rax + mulq 0x40(%rsi) + xorq %r14, %r14 + movq %rax, %rbx + movq %rdx, %rcx + movq 0x28(%rsi), %rax + mulq 0x38(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r14 + addq %rbx, %rbx + adcq %rcx, %rcx + adcq %r14, %r14 + addq %rbx, %r12 + adcq %rcx, %r13 + adcq $0x0, %r14 + movq 0x30(%rsi), %rax + mulq %rax + addq %rax, %r12 + adcq %rdx, %r13 + adcq $0x0, %r14 + movq 0x28(%rsi), %rax + mulq 0x40(%rsi) + xorq %r15, %r15 + movq %rax, %rbx + movq %rdx, %rcx + movq 0x30(%rsi), %rax + mulq 0x38(%rsi) + addq %rax, %rbx + adcq %rdx, %rcx + adcq $0x0, %r15 + addq %rbx, %rbx + adcq %rcx, %rcx + adcq %r15, %r15 + addq %rbx, %r13 + adcq %rcx, %r14 + adcq $0x0, %r15 + xorq %r8, %r8 + movq 0x38(%rsi), %rax + mulq %rax + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %r8 + movq 0x30(%rsi), %rax + mulq 0x40(%rsi) + addq %rax, %rax + adcq %rdx, %rdx + adcq $0x0, %r8 + addq %rax, %r14 + adcq %rdx, %r15 + adcq $0x0, %r8 + movq 0x38(%rsi), %rax + mulq 0x40(%rsi) + addq %rax, %rax + adcq %rdx, %rdx + addq %rax, %r15 + adcq %rdx, %r8 + movq 0x40(%rsi), %rax + imulq %rax, %rax + addq %r8, %rax + movq 0x40(%rsp), %r8 + movq %r8, %rdx + andq $0x1ff, %rdx + shrdq $0x9, %r9, %r8 + shrdq $0x9, %r10, %r9 + shrdq $0x9, %r11, %r10 + shrdq $0x9, %r12, %r11 + shrdq $0x9, %r13, %r12 + shrdq $0x9, %r14, %r13 + shrdq $0x9, %r15, %r14 + shrdq $0x9, %rax, %r15 + shrq $0x9, %rax + addq %rax, %rdx + stc + adcq (%rsp), %r8 + adcq 0x8(%rsp), %r9 + adcq 0x10(%rsp), %r10 + adcq 0x18(%rsp), %r11 + adcq 0x20(%rsp), %r12 + adcq 0x28(%rsp), %r13 + adcq 0x30(%rsp), %r14 + adcq 0x38(%rsp), %r15 + adcq $0xfffffffffffffe00, %rdx + cmc + sbbq $0x0, %r8 + movq %r8, (%rdi) + sbbq $0x0, %r9 + movq %r9, 0x8(%rdi) + sbbq $0x0, %r10 + movq %r10, 0x10(%rdi) + sbbq $0x0, %r11 + movq %r11, 0x18(%rdi) + sbbq $0x0, %r12 + movq %r12, 0x20(%rdi) + sbbq $0x0, %r13 + movq %r13, 0x28(%rdi) + sbbq $0x0, %r14 + movq %r14, 0x30(%rdi) + sbbq $0x0, %r15 + movq %r15, 0x38(%rdi) + sbbq $0x0, %rdx + andq $0x1ff, %rdx + movq %rdx, 0x40(%rdi) + CFI_INC_RSP(72) + CFI_RET + +S2N_BN_SIZE_DIRECTIVE(Lp521_jscalarmul_alt_sqr_p521) + +#if defined(__linux__) && defined(__ELF__) +.section .note.GNU-stack, "", %progbits +#endif diff --git a/cbits/sha1_armv8.c b/cbits/sha1_armv8.c --- a/cbits/sha1_armv8.c +++ b/cbits/sha1_armv8.c @@ -13,21 +13,14 @@ #include #include -#if defined(__linux__) -#include -#include -#endif +#include "crypton_cpu.h" /* * The instructions are an extension, so a translation unit compiled for * baseline ARMv8-A may not use them; see sha256_armv8.c for the whole of that * argument. */ -#ifdef WITH_TARGET_ATTRIBUTES -#define TARGET_ARMV8_CRYPTO __attribute__((target("+crypto"))) -#else -#define TARGET_ARMV8_CRYPTO -#endif +#include "crypton_armv8_target.h" /* * A group of four rounds, and the schedule that goes with it. @@ -55,7 +48,7 @@ * One 64-byte block. `state` is the five words of chaining value in host * order, `buf` the block as it arrived, which SHA-1 reads big-endian. */ -TARGET_ARMV8_CRYPTO +CRYPTON_TARGET_ARMV8_CRYPTO void crypton_sha1_armv8_do_chunks(uint32_t state[5], const uint8_t *data, uint32_t blocks) { @@ -72,15 +65,15 @@ e0 = state[4]; for (; blocks > 0; blocks--, data += 64) { - const uint32_t *buf = (const uint32_t *) data; + const uint8_t *buf = data; abcd_prev = abcd; e_prev = e0; - m0 = vreinterpretq_u32_u8(vrev32q_u8(vreinterpretq_u8_u32(vld1q_u32(buf)))); - m1 = vreinterpretq_u32_u8(vrev32q_u8(vreinterpretq_u8_u32(vld1q_u32(buf + 4)))); - m2 = vreinterpretq_u32_u8(vrev32q_u8(vreinterpretq_u8_u32(vld1q_u32(buf + 8)))); - m3 = vreinterpretq_u32_u8(vrev32q_u8(vreinterpretq_u8_u32(vld1q_u32(buf + 12)))); + m0 = vreinterpretq_u32_u8(vrev32q_u8(vld1q_u8(buf))); + m1 = vreinterpretq_u32_u8(vrev32q_u8(vld1q_u8(buf + 16))); + m2 = vreinterpretq_u32_u8(vrev32q_u8(vld1q_u8(buf + 32))); + m3 = vreinterpretq_u32_u8(vrev32q_u8(vld1q_u8(buf + 48))); wk0 = vaddq_u32(m0, k0); wk1 = vaddq_u32(m1, k0); @@ -151,7 +144,7 @@ } /* the one-block form, for the partial block a message ends with */ -void crypton_sha1_armv8_do_chunk(uint32_t state[5], const uint32_t buf[16]) +void crypton_sha1_armv8_do_chunk(uint32_t state[5], const uint8_t buf[64]) { crypton_sha1_armv8_do_chunks(state, (const uint8_t *) buf, 1); } @@ -163,11 +156,5 @@ */ int crypton_sha1_armv8_available(void) { -#if defined(__APPLE__) - return 1; -#elif defined(__linux__) - return (getauxval(AT_HWCAP) & HWCAP_SHA1) != 0; -#else - return 0; -#endif + return (crypton_arm_features() & CRYPTON_ARM_SHA1) != 0; } diff --git a/cbits/sha1_x86.c b/cbits/sha1_x86.c --- a/cbits/sha1_x86.c +++ b/cbits/sha1_x86.c @@ -168,7 +168,7 @@ } /* the one-block form, for the partial block a message ends with */ -void crypton_sha1_x86_do_chunk(uint32_t state[5], const uint32_t buf[16]) +void crypton_sha1_x86_do_chunk(uint32_t state[5], const uint8_t buf[64]) { crypton_sha1_x86_do_chunks(state, (const uint8_t *) buf, 1); } diff --git a/cbits/sha256_armv8.c b/cbits/sha256_armv8.c --- a/cbits/sha256_armv8.c +++ b/cbits/sha256_armv8.c @@ -12,10 +12,7 @@ #include #include -#if defined(__linux__) -#include -#include -#endif +#include "crypton_cpu.h" /* * The SHA-2 instructions are an extension, so a translation unit compiled for @@ -29,11 +26,7 @@ * * "+crypto" rather than "crypto": GCC rejects the latter. */ -#ifdef WITH_TARGET_ATTRIBUTES -#define TARGET_ARMV8_CRYPTO __attribute__((target("+crypto"))) -#else -#define TARGET_ARMV8_CRYPTO -#endif +#include "crypton_armv8_target.h" static const uint32_t K[64] = { 0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, @@ -58,8 +51,8 @@ * One 64-byte block. `state` is the eight words of chaining value in host * order, `buf` the block as it arrived, which SHA-256 reads big-endian. */ -TARGET_ARMV8_CRYPTO -void crypton_sha256_armv8_do_chunk(uint32_t state[8], const uint32_t buf[16]) +CRYPTON_TARGET_ARMV8_CRYPTO +void crypton_sha256_armv8_do_chunk(uint32_t state[8], const uint8_t buf[64]) { uint32x4_t abcd, efgh, abcd_prev, efgh_prev, abcd_save, tmp; uint32x4_t m0, m1, m2, m3; @@ -68,10 +61,10 @@ abcd_prev = abcd = vld1q_u32(state); efgh_prev = efgh = vld1q_u32(state + 4); - m0 = vreinterpretq_u32_u8(vrev32q_u8(vreinterpretq_u8_u32(vld1q_u32(buf)))); - m1 = vreinterpretq_u32_u8(vrev32q_u8(vreinterpretq_u8_u32(vld1q_u32(buf + 4)))); - m2 = vreinterpretq_u32_u8(vrev32q_u8(vreinterpretq_u8_u32(vld1q_u32(buf + 8)))); - m3 = vreinterpretq_u32_u8(vrev32q_u8(vreinterpretq_u8_u32(vld1q_u32(buf + 12)))); + m0 = vreinterpretq_u32_u8(vrev32q_u8(vld1q_u8(buf))); + m1 = vreinterpretq_u32_u8(vrev32q_u8(vld1q_u8(buf + 16))); + m2 = vreinterpretq_u32_u8(vrev32q_u8(vld1q_u8(buf + 32))); + m3 = vreinterpretq_u32_u8(vrev32q_u8(vld1q_u8(buf + 48))); /* twelve groups of four rounds that also extend the schedule ... */ for (i = 0; i < 48; i += 16) { @@ -135,11 +128,5 @@ */ int crypton_sha256_armv8_available(void) { -#if defined(__APPLE__) - return 1; -#elif defined(__linux__) - return (getauxval(AT_HWCAP) & HWCAP_SHA2) != 0; -#else - return 0; -#endif + return (crypton_arm_features() & CRYPTON_ARM_SHA2) != 0; } diff --git a/cbits/sha3_armv8.c b/cbits/sha3_armv8.c --- a/cbits/sha3_armv8.c +++ b/cbits/sha3_armv8.c @@ -25,12 +25,7 @@ #include #include -#if defined(__APPLE__) -#include -#elif defined(__linux__) -#include -#include -#endif +#include "crypton_cpu.h" /* * The SHA-3 instructions are an ARMv8.2 extension, so a translation unit @@ -38,11 +33,7 @@ * of that argument. The flag for a build without attributes already asks for * "+sha3", which the SHA-512 path needed. */ -#ifdef WITH_TARGET_ATTRIBUTES -#define TARGET_ARMV8_SHA3 __attribute__((target("+sha3"))) -#else -#define TARGET_ARMV8_SHA3 -#endif +#include "crypton_armv8_target.h" static const uint64_t rc[24] = { 0x0000000000000001ULL, 0x0000000000008082ULL, 0x800000000000808aULL, @@ -130,7 +121,7 @@ } while (0) /* the twenty-four rounds over the state, in place */ -TARGET_ARMV8_SHA3 +CRYPTON_TARGET_ARMV8_SHA3 void crypton_sha3_armv8_permute(uint64_t state[25]) { uint64x2_t a[25], b[25], c[5], d[5]; @@ -161,16 +152,5 @@ */ int crypton_sha3_armv8_available(void) { -#if defined(__APPLE__) - int v = 0; - size_t n = sizeof(v); - - if (sysctlbyname("hw.optional.arm.FEAT_SHA3", &v, &n, NULL, 0) != 0) - return 0; - return v != 0; -#elif defined(__linux__) - return (getauxval(AT_HWCAP) & HWCAP_SHA3) != 0; -#else - return 0; -#endif + return (crypton_arm_features() & CRYPTON_ARM_SHA3) != 0; } diff --git a/cbits/sha512_armv8.c b/cbits/sha512_armv8.c --- a/cbits/sha512_armv8.c +++ b/cbits/sha512_armv8.c @@ -14,25 +14,14 @@ #include #include -#if defined(__linux__) -#include -#include -#endif -#if defined(__APPLE__) -#include -#include -#endif +#include "crypton_cpu.h" /* * The instructions are an extension, so a translation unit compiled for * baseline ARMv8-A may not use them; mark the function that does. The * SHA-512 instructions live behind "+sha3" in both GCC and clang. */ -#ifdef WITH_TARGET_ATTRIBUTES -#define TARGET_ARMV8_SHA3 __attribute__((target("+sha3"))) -#else -#define TARGET_ARMV8_SHA3 -#endif +#include "crypton_armv8_target.h" static const uint64_t K[80] = { 0x428a2f98d728ae22ULL, 0x7137449123ef65cdULL, 0xb5c0fbcfec4d3b2fULL, @@ -72,8 +61,8 @@ * two rounds and rotates which pair plays which part, so four steps return * to the start; a group of eight steps is one pass over the schedule. */ -TARGET_ARMV8_SHA3 -void crypton_sha512_armv8_do_chunk(uint64_t state[8], const uint64_t buf[16]) +CRYPTON_TARGET_ARMV8_SHA3 +void crypton_sha512_armv8_do_chunk(uint64_t state[8], const uint8_t buf[128]) { uint64x2_t ab, cd, ef, gh, ab0, cd0, ef0, gh0; uint64x2_t s[8]; @@ -86,7 +75,7 @@ for (i = 0; i < 8; i++) s[i] = vreinterpretq_u64_u8(vrev64q_u8( - vld1q_u8((const uint8_t *) (buf + 2 * i)))); + vld1q_u8(buf + 16 * i))); /* two rounds; A, B, C, D is a rotation of gh, ef, cd, ab */ #define RND(A, B, C, D, sv, ki) \ @@ -146,16 +135,5 @@ */ int crypton_sha512_armv8_available(void) { -#if defined(__APPLE__) - int v = 0; - size_t n = sizeof(v); - - if (sysctlbyname("hw.optional.arm.FEAT_SHA512", &v, &n, NULL, 0) != 0) - return 0; - return v != 0; -#elif defined(__linux__) - return (getauxval(AT_HWCAP) & HWCAP_SHA512) != 0; -#else - return 0; -#endif + return (crypton_arm_features() & CRYPTON_ARM_SHA512) != 0; } diff --git a/cbits/tests/bearssl_diff.c b/cbits/tests/bearssl_diff.c new file mode 100644 --- /dev/null +++ b/cbits/tests/bearssl_diff.c @@ -0,0 +1,432 @@ +/* + * crypton's portable AES and GHASH against the BearSSL they are written on. + * + * This began as the migration check: for one commit, cbits/aes/generic.c and + * cbits/aes/gf.c were still the table-driven implementations, and this said + * the vendored code computed what they computed before they were replaced. + * + * Since the replacement the two sides are no longer independent, and what is + * left is still worth checking: everything in generic.c and gf.c is now + * crypton's own glue -- a schedule kept compressed and carried through + * memcpy, the interleave-and-ortho idiom around a single block, three of four + * lanes left idle, and a GHASH entry that reaches the same multiply by handing + * it a block of zeros. Each of those is somewhere a mistake would live, and + * each is compared here against calling BearSSL directly. + * + * FIPS-197's own vectors come first either way, so that agreement means AES + * rather than two callers agreeing on something that is not. + * + * Run with an argument to corrupt results on purpose and see the comparison + * notice: a differential test that cannot fail has said nothing. + */ +#include +#include +#include +#include + +#include "bearssl/inner.h" +#include "crypton_aes.h" +#include "aes/generic.h" +#include "aes/gf.h" +#include "aes/block128.h" + +/* + * crypton_aes.c's table, which is a global, and the three key sizes of the + * two CTR entries this looks for in it. Searching rather than indexing + * because the index is an enum private to that file. + */ +#define BRANCH_TABLE_SEARCH 64 +extern void *crypton_aes_branch_table[]; + +/* the two GCM implementations, which crypton_aes.c declares but no header + * does: in this build the generic one reaches the portable block function + * too, so the two have to agree exactly, tag and all */ +void crypton_aes_generic_gcm_encrypt(uint8_t *, aes_gcm *, aes_key *, uint8_t *, uint32_t); +void crypton_aes_generic_gcm_decrypt(uint8_t *, aes_gcm *, aes_key *, uint8_t *, uint32_t); +void crypton_aes_bitsliced_gcm_encrypt(uint8_t *, aes_gcm *, aes_key *, uint8_t *, uint32_t); +void crypton_aes_bitsliced_gcm_decrypt(uint8_t *, aes_gcm *, aes_key *, uint8_t *, uint32_t); + +/* ---- the vendored code, one block at a time, as aes_ct64_cbcenc.c does ---- */ + +static void bear_key(uint64_t *comp_skey, unsigned *nr, + const uint8_t *key, size_t len) +{ + *nr = br_aes_ct64_keysched(comp_skey, key, len); +} + +static void bear_block(uint8_t *out, const uint64_t *comp_skey, unsigned nr, + const uint8_t *in, int decrypt) +{ + uint64_t sk_exp[120]; + uint32_t w[4]; + uint64_t q[8]; + + br_aes_ct64_skey_expand(sk_exp, nr, comp_skey); + w[0] = br_dec32le(in); + w[1] = br_dec32le(in + 4); + w[2] = br_dec32le(in + 8); + w[3] = br_dec32le(in + 12); + memset(q, 0, sizeof q); + br_aes_ct64_interleave_in(&q[0], &q[4], w); + br_aes_ct64_ortho(q); + if (decrypt) + br_aes_ct64_bitslice_decrypt(nr, sk_exp, q); + else + br_aes_ct64_bitslice_encrypt(nr, sk_exp, q); + br_aes_ct64_ortho(q); + br_aes_ct64_interleave_out(w, q[0], q[4]); + br_enc32le(out, w[0]); + br_enc32le(out + 4, w[1]); + br_enc32le(out + 8, w[2]); + br_enc32le(out + 12, w[3]); +} + +/* ---- crypton's GHASH, driven the way crypton_aes.c drives it ---- */ + +static void crypton_ghash(uint8_t *y, const uint8_t *h, + const uint8_t *data, size_t len) +{ + table_4bit ht; + block128 acc; + size_t i; + + crypton_aes_generic_hinit(ht, (const block128 *) h); + block128_zero(&acc); + for (i = 0; i < len; i += 16) { + block128_xor_bytes(&acc, data + i, 16); + crypton_aes_generic_gf_mul(&acc, ht); + } + memcpy(y, &acc, 16); +} + +/* ---- the comparison ---- */ + +static int failures; +static int sabotage; + +static void same(const char *what, const uint8_t *a, const uint8_t *b, size_t n) +{ + if (memcmp(a, b, n) != 0) { + size_t i; + + printf(" MISMATCH %s\n bearssl ", what); + for (i = 0; i < n; i++) printf("%02x", a[i]); + printf("\n crypton "); + for (i = 0; i < n; i++) printf("%02x", b[i]); + printf("\n"); + failures++; + } +} + +static uint32_t rnd_state = 1; +static uint8_t rnd(void) +{ + rnd_state = rnd_state * 1103515245u + 12345u; + return (uint8_t)(rnd_state >> 16); +} +static void rnd_fill(uint8_t *p, size_t n) +{ + while (n--) *p++ = rnd(); +} + +int main(int argc, char **argv) +{ + /* FIPS-197 C.1, C.2, C.3 */ + static const uint8_t pt[16] = { + 0x00,0x11,0x22,0x33,0x44,0x55,0x66,0x77, + 0x88,0x99,0xaa,0xbb,0xcc,0xdd,0xee,0xff }; + static const uint8_t k128[16] = { + 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 }; + static const uint8_t k192[24] = { + 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 }; + static const uint8_t k256[32] = { + 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15, + 16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31 }; + static const uint8_t c128[16] = { + 0x69,0xc4,0xe0,0xd8,0x6a,0x7b,0x04,0x30, + 0xd8,0xcd,0xb7,0x80,0x70,0xb4,0xc5,0x5a }; + static const uint8_t c192[16] = { + 0xdd,0xa9,0x7c,0xa4,0x86,0x4c,0xdf,0xe0, + 0x6e,0xaf,0x70,0xa0,0xec,0x0d,0x71,0x91 }; + static const uint8_t c256[16] = { + 0x8e,0xa2,0xb7,0xca,0x51,0x67,0x45,0xbf, + 0xea,0xfc,0x49,0x90,0x4b,0x49,0x60,0x89 }; + const uint8_t *keys[3] = { k128, k192, k256 }; + const uint8_t *cts[3] = { c128, c192, c256 }; + size_t klens[3] = { 16, 24, 32 }; + uint64_t comp[30]; + unsigned nr; + uint8_t a[16], b[16]; + int i, round; + + sabotage = argc > 1; + printf("== FIPS-197, so that agreement means AES ==\n"); + for (i = 0; i < 3; i++) { + bear_key(comp, &nr, keys[i], klens[i]); + bear_block(a, comp, nr, pt, 0); + if (sabotage && i == 1) a[0] ^= 1; + same("bearssl vs FIPS-197", a, cts[i], 16); + bear_block(b, comp, nr, cts[i], 1); + same("bearssl decrypt vs plaintext", b, pt, 16); + } + + printf("== crypton's glue vs BearSSL direct, 2000 random keys and blocks ==\n"); + for (round = 0; round < 2000; round++) { + uint8_t key[32], in[16], e1[16], e2[16], d1[16], d2[16]; + size_t kl = klens[round % 3]; + aes_key ck; + + rnd_fill(key, kl); + rnd_fill(in, 16); + + bear_key(comp, &nr, key, kl); + bear_block(e1, comp, nr, in, 0); + bear_block(d1, comp, nr, in, 1); + + crypton_aes_generic_init(&ck, key, (uint8_t) kl); + crypton_aes_generic_encrypt_block((aes_block *) e2, &ck, + (aes_block *) in); + crypton_aes_generic_decrypt_block((aes_block *) d2, &ck, + (aes_block *) in); + if (sabotage && round == 7) e1[3] ^= 0x10; + same("encrypt", e1, e2, 16); + same("decrypt", d1, d2, 16); + } + + printf("== GHASH: crypton's entries vs br_ghash_ctmul64, 500 messages ==\n"); + for (round = 0; round < 500; round++) { + uint8_t h[16], data[256], y1[16], y2[16]; + size_t len = 16u * (size_t)(1 + (round % 16)); + + rnd_fill(h, 16); + rnd_fill(data, len); + + memset(y1, 0, 16); + br_ghash_ctmul64(y1, h, data, len); + crypton_ghash(y2, h, data, len); + if (sabotage && round == 3) y1[15] ^= 0x80; + same("ghash", y1, y2, 16); + } + + printf("== gf_mul4: the four-block entry against the same four blocks ==\n"); + for (round = 0; round < 500; round++) { + uint8_t h[16], data[64], y1[16]; + table_4bit ht; + block128 acc; + + rnd_fill(h, 16); + rnd_fill(data, sizeof data); + + memset(y1, 0, 16); + br_ghash_ctmul64(y1, h, data, sizeof data); + + crypton_aes_generic_hinit(ht, (const block128 *) h); + block128_zero(&acc); + crypton_aes_generic_gf_mul4(&acc, (const block128 *) data, ht); + if (sabotage && round == 11) y1[0] ^= 0x40; + same("gf_mul4", y1, (const uint8_t *) &acc, 16); + } + + /* + * And that the wide entries are the ones installed. Nothing else + * here would notice if they were not: the entries they replace are + * correct too, just a block at a time, so every answer above would + * be the same and only the speed would be gone. + */ + /* + * The Haskell suite reaches the four-block pass, but thinly: its + * vectors are mostly a block or three long, and breaking that pass + * alone fails sixteen of its examples where breaking every pass + * fails seven hundred. So the lane logic and the counter are + * covered here instead, where the lengths can be chosen. + */ + printf("== many blocks at a pass against one at a time ==\n"); + for (round = 0; round < 200; round++) { + uint8_t key[32], in[16 * 9], wide[16 * 9], single[16 * 9]; + size_t kl = klens[round % 3]; + uint32_t nb = 1 + (round % 9); + aes_sched sched; + aes_key ck; + uint32_t i; + int dec = round & 1; + + rnd_fill(key, kl); + rnd_fill(in, nb * 16); + crypton_aes_generic_init(&ck, key, (uint8_t) kl); + + crypton_aes_generic_schedule(&sched, &ck); + crypton_aes_generic_blocks(wide, in, nb, &sched, dec); + + for (i = 0; i < nb; i++) { + if (dec) + crypton_aes_generic_decrypt_block( + (aes_block *) (single + 16 * i), &ck, + (aes_block *) (in + 16 * i)); + else + crypton_aes_generic_encrypt_block( + (aes_block *) (single + 16 * i), &ck, + (aes_block *) (in + 16 * i)); + } + if (sabotage && round == 5) wide[16] ^= 2; + same("wide pass", wide, single, nb * 16); + } + + printf("== the four-block CTR against one block at a time ==\n"); + for (round = 0; round < 200; round++) { + uint8_t key[32], iv[16], in[200], got[200], want[200]; + size_t kl = klens[round % 3]; + /* lengths that land on, before and after a group of four */ + uint32_t len = 1 + (round % 200); + aes_key ck; + aes_block counter, ks; + uint32_t done, n, i; + int c32 = round & 1; + + rnd_fill(key, kl); + rnd_fill(iv, 16); + rnd_fill(in, len); + crypton_aes_generic_init(&ck, key, (uint8_t) kl); + + if (c32) + crypton_aes_bitsliced_encrypt_c32(got, &ck, + (aes_block *) iv, in, len); + else + crypton_aes_bitsliced_encrypt_ctr(got, &ck, + (aes_block *) iv, in, len); + + block128_copy(&counter, (block128 *) iv); + for (done = 0; done < len; done += 16) { + crypton_aes_generic_encrypt_block(&ks, &ck, &counter); + n = len - done < 16 ? len - done : 16; + for (i = 0; i < n; i++) + want[done + i] = ((uint8_t *) &ks)[i] + ^ in[done + i]; + if (c32) + block128_inc32_le(&counter); + else + block128_inc_be(&counter); + } + if (sabotage && round == 9) got[len - 1] ^= 4; + same("ctr", got, want, len); + } + + printf("== XTS four at a pass against one at a time ==\n"); + for (round = 0; round < 200; round++) { + uint8_t key[32], key2[32], du[16], in[16 * 11]; + uint8_t got[16 * 11], want[16 * 11]; + size_t kl = klens[round % 3]; + uint32_t nb = 1 + (round % 11); + uint32_t spoint = round % 3; + aes_key k1, k2; + aes_block tweak; + uint32_t i, j; + int dec = round & 1; + + rnd_fill(key, kl); + rnd_fill(key2, kl); + rnd_fill(du, 16); + rnd_fill(in, nb * 16); + crypton_aes_initkey(&k1, key, (uint8_t) kl); + crypton_aes_initkey(&k2, key2, (uint8_t) kl); + + { + aes_block d; + memcpy(&d, du, 16); + if (dec) + crypton_aes_decrypt_xts((aes_block *) got, &k1, &k2, + &d, spoint, (aes_block *) in, nb); + else + crypton_aes_encrypt_xts((aes_block *) got, &k1, &k2, + &d, spoint, (aes_block *) in, nb); + } + + /* the same thing a block at a time */ + memcpy(&tweak, du, 16); + crypton_aes_generic_encrypt_block(&tweak, &k2, &tweak); + for (j = 0; j < spoint; j++) + crypton_aes_generic_gf_mulx((block128 *) &tweak); + for (i = 0; i < nb; i++) { + aes_block t; + + block128_vxor(&t, (block128 *) (in + 16 * i), &tweak); + if (dec) + crypton_aes_generic_decrypt_block(&t, &k1, &t); + else + crypton_aes_generic_encrypt_block(&t, &k1, &t); + block128_vxor((block128 *) (want + 16 * i), &t, &tweak); + crypton_aes_generic_gf_mulx((block128 *) &tweak); + } + if (sabotage && round == 17) got[16] ^= 8; + same("xts", got, want, nb * 16); + } + + printf("== the four-block GCM against the one-block GCM ==\n"); + for (round = 0; round < 200; round++) { + uint8_t key[32], iv[12], in[300], ga[300], gb[300]; + size_t kl = klens[round % 3]; + uint32_t len = 1 + (round % 300); + aes_key ck; + aes_gcm g1, g2; + int dec = round & 1; + + rnd_fill(key, kl); + rnd_fill(iv, sizeof iv); + rnd_fill(in, len); + crypton_aes_initkey(&ck, key, (uint8_t) kl); + crypton_aes_gcm_init(&g1, &ck, iv, sizeof iv); + memcpy(&g2, &g1, sizeof g1); + + if (dec) { + crypton_aes_generic_gcm_decrypt(ga, &g1, &ck, in, len); + crypton_aes_bitsliced_gcm_decrypt(gb, &g2, &ck, in, len); + } else { + crypton_aes_generic_gcm_encrypt(ga, &g1, &ck, in, len); + crypton_aes_bitsliced_gcm_encrypt(gb, &g2, &ck, in, len); + } + if (sabotage && round == 13) gb[0] ^= 1; + same("gcm text", ga, gb, len); + /* the running GHASH and counter, which the tag is made from */ + same("gcm state", (const uint8_t *) &g1, (const uint8_t *) &g2, + sizeof g1); + } + + printf("== which implementation the build chose ==\n"); +#if !defined(WITH_AESNI) && !defined(WITH_ARMV8_CRYPTO) + /* + * With no accelerator compiled in, crypton_aes.c does not read the + * branch table at all -- its GET_ macros name the portable entries + * directly -- so there is nothing here to look at, and which + * implementation runs is settled by the preprocessor. Scanning the + * table in this build was a check that passed while telling nothing, + * which is how the four-block CTR came to be written, installed, and + * never called. + */ + printf(" named at compile time; the table is not read in this build\n"); + (void) crypton_aes_branch_table; + if (sabotage) { /* nothing to corrupt here */ } +#else + { + int i, ctr = 0, c32 = 0; + + for (i = 0; i < BRANCH_TABLE_SEARCH; i++) { + if (crypton_aes_branch_table[i] == + (void *) crypton_aes_bitsliced_encrypt_ctr) + ctr++; + if (crypton_aes_branch_table[i] == + (void *) crypton_aes_bitsliced_encrypt_c32) + c32++; + } + printf(" CTR entries %d, C32 entries %d\n", ctr, c32); + if (sabotage) { ctr = 0; } + if (ctr != 3 || c32 != 3) { + printf(" MISMATCH the portable CTR entries were not installed\n"); + failures++; + } + } +#endif + + printf("%s: %d mismatch(es)%s\n", + failures ? "FAIL" : "ok", failures, + sabotage ? " (sabotage was asked for)" : ""); + return failures != 0; +} diff --git a/cbits/tests/ct/README b/cbits/tests/ct/README new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/README @@ -0,0 +1,88 @@ +Does the code that handles a secret run in time independent of it? + +memcheck already follows undefined bytes through arithmetic and reports the +moment one of them decides a branch or an address. That is the same question, +so each driver here declares its secret undefined and runs under valgrind: +every report names a place where a secret reached a branch or an index. The +technique is Adam Langley's ctgrind. + +ct_canary.c is the calibration. It branches on a secret and indexes a table +with one, so it must report; if it does not, the marking is not reaching the +code and the silence of every other driver in that run means nothing. Round +five learned this the hard way with ThreadSanitizer, which saw nothing through +GHC's runtime including a deliberate race. + +What is marked secret in each driver is the thing the caller would call a +private key, and nothing else. A modulus, a peer's public key, a nonce and a +message are public and stay defined; so does each answer, which is declared +public again before anything looks at it. + + canary a branch and a table index on a secret -- must report + powm crypton_powm_sec, the exponent being an RSA private key + p256 both scalar multiplications and the scalar inversion + x25519 the scalar + ed25519 the private key, through signing + decaf Ed448 signing and X448, both private scalars + chapoly the ChaCha20 key, the plaintext, and the Poly1305 key + aes the AES key and the plaintext, through ECB and GCM + aes_armv8 the same driver again, on AArch64, against the instructions + aes_x86ni the same driver again, on x86-64, against AES-NI + +One driver, built three times, against the portable C and against each of +the two instruction sets. All three must report nothing at all -- not +"nothing outside known.txt", nothing. The portable AES and GHASH are +bitsliced and table-free; AESE, AESMC, PMULL, AESENC and PCLMULQDQ look +nothing up. None of the three has anywhere for a secret to decide a branch +or an address. + +This entry used to read the other way round. The portable implementation +was table-driven -- an S-box indexed by a byte of the state, and Shoup's +4-bit table for the GHASH -- so the aes driver was *required* to report, and +known.txt carried four entries saying so. That requirement was also what +kept the pair honest: if the portable build had quietly taken an accelerated +path it would have fallen silent, and the silence would have failed it. + +With the tables gone that check goes too, so the question is now asked +outright rather than read off a leak. Each build of the driver prints which +implementation its dispatch took, and run.sh requires the answer it expects +before it looks at anything else. A build that takes the wrong path fails +on that line rather than passing by saying nothing. + +The x86-64 build is new with the rewrite. Before it, AES-NI and PCLMULQDQ +had never been put to this harness at all: their being constant time was an +inference from the instruction specifications rather than a measurement of +crypton's code. The AArch64 build came earlier, when the GHASH was +rewritten and it was noticed that the harness had only ever run on x86-64. + +What round eight found +---------------------- + +Of the eight drivers, five were silent: the RSA exponentiation, X25519, +Ed25519, ChaCha20 and Poly1305 never let a private key decide a branch or an +address. The AES driver reported from the tables, as it was built to. + +The other two reported, five places between them, and every one of them an +assert(): + + crypton_p256_modmul assert(top <= 1), assert(top == 0) + crypton_gf_448_strong_reduce two asserts on a carry and a borrow + crypton_gf_invert assert(ret), that what was inverted had an + inverse + +The first four check an invariant of a reduction rather than anything about +the data, so they hold whatever the input is. The fifth holds because the two +callers that ask for it are inverting a projective z, which is never zero for +a point on the curve. Either way the branch goes the same way every time and +no timing follows from it. They are reported at all because +crypton's C is compiled without NDEBUG, so assert() is live in a released +library. Twenty-eight assertions ship that way, none of them with a side +effect, and defining NDEBUG measured no faster on P-256, so whether to keep +them is a question about what a library should do when an internal invariant +fails -- abort the process, or carry on -- rather than one about speed. They +are listed in known.txt and the job passes with them. + +The decision was to keep them: an internal invariant that fails in a +cryptographic library is better met with an abort than with a wrong answer +carried onwards. So this is settled rather than open, and the five entries in +known.txt are permanent. What is not permanent is anything else appearing +beside them. diff --git a/cbits/tests/ct/cpu_stub.c b/cbits/tests/ct/cpu_stub.c new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/cpu_stub.c @@ -0,0 +1,19 @@ +/* + * One symbol, so that cbits/crypton_cpu.c can be linked on its own. + * + * The ML-KEM backend asks crypton_cpu.c whether this processor has AVX2. + * That file also carries crypton_cpu_option, which the Haskell side calls + * and which reaches into cbits/crypton_aes.c, so the linker wants + * crypton_aes_cpu_init as well -- and behind it the whole AES build, which + * has nothing to do with what is being measured here. This stands in for + * it. Nothing in an ML-KEM driver calls it; if anything ever did, zeroes + * say "no AES instructions", which is the safe answer. + */ +#include +#include "crypton_cpu.h" + +uint8_t *crypton_aes_cpu_init(void) +{ + static uint8_t none[CPU_OPTION_COUNT]; + return none; +} diff --git a/cbits/tests/ct/ct.h b/cbits/tests/ct/ct.h new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/ct.h @@ -0,0 +1,53 @@ +/* Marking secrets for valgrind. + * + * memcheck already follows undefined bytes through arithmetic and complains + * the moment one decides a branch or an address. That is the same question + * as "does this run in time independent of the secret", so a secret declared + * undefined turns memcheck into a checker for it. The technique is Adam + * Langley's ctgrind. + * + * Without CRYPTON_CT_VALGRIND the macros vanish and the drivers still build + * and run, which is how they are kept honest on a machine with no valgrind. + */ +#ifndef CRYPTON_TESTS_CT_H +#define CRYPTON_TESTS_CT_H + +#ifdef CRYPTON_CT_VALGRIND +#include +/* this memory is a secret: report any branch or index that depends on it */ +#define CT_SECRET(p, n) VALGRIND_MAKE_MEM_UNDEFINED((p), (n)) +/* and this is the answer, which the caller is allowed to look at */ +#define CT_PUBLIC(p, n) VALGRIND_MAKE_MEM_DEFINED((p), (n)) +#else +#define CT_SECRET(p, n) ((void)(p), (void)(n)) +#define CT_PUBLIC(p, n) ((void)(p), (void)(n)) +#endif + +#include +#include + +/* A deterministic filler, so that a report names the same operation on every + * run. It is not random and does not need to be. */ +static uint64_t ct_s0 = 0x243f6a8885a308d3ULL, ct_s1 = 0x13198a2e03707344ULL; +static uint64_t ct_rnd(void) { + uint64_t x = ct_s0, y = ct_s1; + ct_s0 = y; + x ^= x << 23; + ct_s1 = x ^ y ^ (x >> 17) ^ (y >> 26); + return ct_s1 + y; +} +static void ct_fill(void *p, size_t n) { + uint8_t *q = (uint8_t *)p; + size_t i; + for (i = 0; i < n; i++) q[i] = (uint8_t)(ct_rnd() >> 24); +} +/* Look at the answer, so that nothing above is optimized away. Whatever is + * handed here has been declared public first. */ +static void ct_sink(const void *p, size_t n) { + const uint8_t *q = (const uint8_t *)p; + size_t i; + uint8_t acc = 0; + for (i = 0; i < n; i++) acc ^= q[i]; + if (acc == 0xa5 && n == (size_t)-1) printf("unreachable\n"); +} +#endif diff --git a/cbits/tests/ct/ct_aes.c b/cbits/tests/ct/ct_aes.c new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/ct_aes.c @@ -0,0 +1,46 @@ +/* AES, and AES-GCM. The key is the secret, and so is the plaintext. + * + * Whether this reports depends on which implementation the machine selected. + * AES-NI and the ARMv8 instructions do not look anything up; the generic C + * is table-driven and is variable-time by construction, which is a property + * of that code rather than a defect in it. See cbits/tests/ct/README. */ +#include "tests/ct/ct.h" +#include +#include +#include "crypton_aes.h" +#include "crypton_cpu.h" + +/* crypton_aes.c fills this; run.sh reads the line below to tell the three + * builds of this driver apart. They are all silent now, so which one took + * which implementation cannot be read off a leak any more. */ +uint8_t *crypton_aes_cpu_init(void); + +int main(void) { + aes_key k; + aes_gcm_key gk; + uint8_t key[32], pt[256], ct[256 + 16], iv[12]; + + printf("dispatch aes=%d pclmul=%d\n", + crypton_aes_cpu_init()[CPU_AESNI] != 0, + crypton_aes_cpu_init()[CPU_PCLMUL] != 0); + + ct_fill(key, sizeof key); + ct_fill(pt, sizeof pt); + ct_fill(iv, sizeof iv); + CT_SECRET(key, sizeof key); + CT_SECRET(pt, sizeof pt); + + crypton_aes_initkey(&k, key, sizeof key); + crypton_aes_encrypt_ecb((aes_block *)ct, &k, (aes_block *)pt, + sizeof pt / 16); + CT_PUBLIC(ct, sizeof pt); + ct_sink(ct, sizeof pt); + + crypton_aes_gcm_key_init(&gk, &k); + /* the output takes the ciphertext and then the tag */ + crypton_aes_gcm_full_encrypt(ct, &gk, &k, iv, sizeof iv, NULL, 0, + pt, sizeof pt, 16); + CT_PUBLIC(ct, sizeof ct); + ct_sink(ct, sizeof ct); + return 0; +} diff --git a/cbits/tests/ct/ct_aes_armv8.c b/cbits/tests/ct/ct_aes_armv8.c new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/ct_aes_armv8.c @@ -0,0 +1,12 @@ +/* The same driver as ct_aes.c, built against the AArch64 implementation + * instead of the table-driven C. + * + * AESE, AESMC and PMULL look nothing up and branch on nothing, so this one + * must report nothing at all -- not "nothing unknown", nothing. The + * table-driven run of the same driver is what keeps that honest: if the + * marking stopped reaching the code, that run would fall silent and fail, + * and a silence here would mean no more than a silence there. + * + * Until this existed the constant-time harness ran only on x86-64, so + * crypton's AArch64 AES and GHASH had never been put to it. */ +#include "ct_aes.c" diff --git a/cbits/tests/ct/ct_canary.c b/cbits/tests/ct/ct_canary.c new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/ct_canary.c @@ -0,0 +1,21 @@ +/* The calibration. This one is deliberately not constant time: it branches + * on a secret byte and indexes a table with another. If it reports nothing, + * the marking is not reaching the code and every other driver's silence in + * this run means nothing either -- which is the whole reason it is here. */ +#include "tests/ct/ct.h" + +static const uint8_t table[256] = {1}; + +int main(void) { + uint8_t secret[32], out[2]; + + ct_fill(secret, sizeof secret); + CT_SECRET(secret, sizeof secret); + + out[0] = secret[0] & 1 ? 0x5a : 0xa5; /* a branch on the secret */ + out[1] = table[secret[1]]; /* an address from the secret */ + + CT_PUBLIC(out, sizeof out); + ct_sink(out, sizeof out); + return 0; +} diff --git a/cbits/tests/ct/ct_chapoly.c b/cbits/tests/ct/ct_chapoly.c new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/ct_chapoly.c @@ -0,0 +1,33 @@ +/* ChaCha20 and Poly1305. The key is the secret, and so is the plaintext. */ +#include "tests/ct/ct.h" +#include +#include "crypton_chacha.h" +#include "crypton_poly1305.h" + +int main(void) { + crypton_chacha_context ctx; + poly1305_ctx pctx; + poly1305_key pkey; + poly1305_mac mac; + uint8_t key[32], iv[12], pt[256], ct[256]; + + ct_fill(key, sizeof key); + ct_fill(iv, sizeof iv); + ct_fill(pt, sizeof pt); + ct_fill(pkey, sizeof pkey); + CT_SECRET(key, sizeof key); + CT_SECRET(pt, sizeof pt); + CT_SECRET(pkey, sizeof pkey); + + crypton_chacha_init(&ctx, 20, sizeof key, key, sizeof iv, iv); + crypton_chacha_combine(ct, &ctx, pt, sizeof pt); + CT_PUBLIC(ct, sizeof ct); /* the ciphertext goes on the wire */ + ct_sink(ct, sizeof ct); + + crypton_poly1305_init(&pctx, &pkey); + crypton_poly1305_update(&pctx, ct, sizeof ct); + crypton_poly1305_finalize(mac, &pctx); + CT_PUBLIC(mac, sizeof mac); + ct_sink(mac, sizeof mac); + return 0; +} diff --git a/cbits/tests/ct/ct_decaf.c b/cbits/tests/ct/ct_decaf.c new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/ct_decaf.c @@ -0,0 +1,36 @@ +/* X448 and Ed448. The scalar and the private key are the secrets. */ +#include "tests/ct/ct.h" +#include +#include "decaf/ed448.h" +#include "decaf/point_448.h" + +int main(void) { + uint8_t priv[CRYPTON_DECAF_EDDSA_448_PRIVATE_BYTES]; + uint8_t pub[CRYPTON_DECAF_EDDSA_448_PUBLIC_BYTES]; + uint8_t sig[CRYPTON_DECAF_EDDSA_448_SIGNATURE_BYTES]; + uint8_t xs[CRYPTON_DECAF_X448_PRIVATE_BYTES]; + uint8_t xb[CRYPTON_DECAF_X448_PUBLIC_BYTES]; + uint8_t xo[CRYPTON_DECAF_X448_PUBLIC_BYTES]; + uint8_t msg[64]; + + ct_fill(priv, sizeof priv); + ct_fill(msg, sizeof msg); + ct_fill(xs, sizeof xs); + ct_fill(xb, sizeof xb); + CT_SECRET(priv, sizeof priv); + CT_SECRET(xs, sizeof xs); + + crypton_decaf_ed448_derive_public_key(pub, priv); + CT_PUBLIC(pub, sizeof pub); + crypton_decaf_ed448_sign(sig, priv, pub, msg, sizeof msg, 0, NULL, 0); + CT_PUBLIC(sig, sizeof sig); + ct_sink(sig, sizeof sig); + + crypton_decaf_x448_derive_public_key(xo, xs); + CT_PUBLIC(xo, sizeof xo); + ct_sink(xo, sizeof xo); + (void)crypton_decaf_x448(xo, xb, xs); + CT_PUBLIC(xo, sizeof xo); + ct_sink(xo, sizeof xo); + return 0; +} diff --git a/cbits/tests/ct/ct_ed25519.c b/cbits/tests/ct/ct_ed25519.c new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/ct_ed25519.c @@ -0,0 +1,23 @@ +/* Ed25519 signing. The private key is the secret; the message is not. */ +#include "tests/ct/ct.h" +#include +#include "ed25519/ed25519.h" + +int main(void) { + ed25519_secret_key sk; + ed25519_public_key pk; + ed25519_signature sig; + uint8_t msg[64]; + + ct_fill(sk, sizeof sk); + ct_fill(msg, sizeof msg); + CT_SECRET(sk, sizeof sk); + + crypton_ed25519_publickey(sk, pk); + CT_PUBLIC(pk, sizeof pk); + + crypton_ed25519_sign(msg, sizeof msg, sk, pk, sig); + CT_PUBLIC(sig, sizeof sig); + ct_sink(sig, sizeof sig); + return 0; +} diff --git a/cbits/tests/ct/ct_mlkem.c b/cbits/tests/ct/ct_mlkem.c new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/ct_mlkem.c @@ -0,0 +1,45 @@ +/* ML-KEM-768, which is the parameter set TLS uses. The seed and the + * decapsulation key are secret; the encapsulation key and the ciphertext are + * published, and the shared secret is the answer the caller asked for. + * + * mlkem-native's AArch64 and x86-64 assembly carries a HOL-Light proof of + * secret-independent timing, so a report from the backend build means the + * proof does not cover what crypton built, or that crypton reached it + * wrongly. The portable C has no such proof, which is the reason to ask it + * the question at all. */ +#include "tests/ct/ct.h" + +int crypton_mlkem768_keypair_derand(uint8_t *pk, uint8_t *sk, + const uint8_t *coins); +int crypton_mlkem768_enc_derand(uint8_t *ct, uint8_t *ss, const uint8_t *pk, + const uint8_t *coins); +int crypton_mlkem768_dec(uint8_t *ss, const uint8_t *ct, const uint8_t *sk); + +int main(void) { + uint8_t pk[1184], sk[2400], ct[1088], ss[32], ss2[32]; + uint8_t kcoins[64], ecoins[32]; + + ct_fill(kcoins, sizeof kcoins); + ct_fill(ecoins, sizeof ecoins); + + CT_SECRET(kcoins, sizeof kcoins); + if (crypton_mlkem768_keypair_derand(pk, sk, kcoins) != 0) return 1; + CT_PUBLIC(pk, sizeof pk); + ct_sink(pk, sizeof pk); + + /* The message encapsulation draws is as secret as the key it derives. */ + CT_SECRET(ecoins, sizeof ecoins); + if (crypton_mlkem768_enc_derand(ct, ss, pk, ecoins) != 0) return 1; + CT_PUBLIC(ct, sizeof ct); + CT_PUBLIC(ss, sizeof ss); + ct_sink(ct, sizeof ct); + ct_sink(ss, sizeof ss); + + /* sk is still undefined here: decapsulation is the half that runs on the + * long-lived secret, and the half an attacker can drive by sending + * ciphertexts of their choosing. */ + if (crypton_mlkem768_dec(ss2, ct, sk) != 0) return 1; + CT_PUBLIC(ss2, sizeof ss2); + ct_sink(ss2, sizeof ss2); + return 0; +} diff --git a/cbits/tests/ct/ct_p256.c b/cbits/tests/ct/ct_p256.c new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/ct_p256.c @@ -0,0 +1,42 @@ +/* The two P-256 scalar multiplications and the scalar inversion, all of + * which take the private key as the scalar. */ +#include "tests/ct/ct.h" +#include +#include "p256/p256.h" + +void crypton_p256e_point_mul(const crypton_p256_int *n, + const crypton_p256_int *ix, const crypton_p256_int *iy, + crypton_p256_int *ox, crypton_p256_int *oy); +void crypton_p256e_scalar_invert(const crypton_p256_int *a, + crypton_p256_int *b); + +int main(void) { + crypton_p256_int n, px, py, ox, oy, inv, one; + int i; + + /* a public point to be multiplied: the generator's 0x9e3779b9 multiple */ + crypton_p256_init(&one); + P256_DIGIT(&one, 0) = 0x9e3779b9u; + crypton_p256_base_point_mul(&one, &px, &py); + + for (i = 0; i < P256_NDIGITS; i++) + P256_DIGIT(&n, i) = (crypton_p256_digit)ct_rnd(); + crypton_p256_mod(&crypton_SECP256r1_n, &n, &n); + + CT_SECRET(&n, sizeof n); + + crypton_p256_base_point_mul(&n, &ox, &oy); + CT_PUBLIC(&ox, sizeof ox); + CT_PUBLIC(&oy, sizeof oy); + ct_sink(&ox, sizeof ox); + + crypton_p256e_point_mul(&n, &px, &py, &ox, &oy); + CT_PUBLIC(&ox, sizeof ox); + CT_PUBLIC(&oy, sizeof oy); + ct_sink(&oy, sizeof oy); + + crypton_p256e_scalar_invert(&n, &inv); + CT_PUBLIC(&inv, sizeof inv); + ct_sink(&inv, sizeof inv); + return 0; +} diff --git a/cbits/tests/ct/ct_powm.c b/cbits/tests/ct/ct_powm.c new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/ct_powm.c @@ -0,0 +1,58 @@ +/* The windowed modular exponentiation, which is what an RSA private key + * operation runs. The exponent is the secret it is built to hide. */ +#include "tests/ct/ct.h" +#include +#include "crypton_powm.h" + +/* A 2048-bit modulus unless the build says otherwise: run.sh builds it at + * the other sizes the vendored multiplication takes as well, since those + * are different code. */ +#ifndef CT_POWM_BYTES +#define CT_POWM_BYTES 256 +#endif + +#ifdef CT_POWM_COUNT_S2N +/* run.sh links this build with -Wl,--wrap=bignum_emontredc_8n, so every + * reduction the s2n-bignum path does comes through here and is counted. + * The build then says which implementation it ran, rather than leaving it + * to be inferred: under valgrind the processor's answers are valgrind's, + * and a build that quietly took the C would pass here while checking what + * the plain powm build already has. */ +#include +uint64_t __real_bignum_emontredc_8n(uint64_t k, uint64_t *z, + const uint64_t *m, uint64_t w); +static unsigned long s2n_reductions; +uint64_t __wrap_bignum_emontredc_8n(uint64_t k, uint64_t *z, + const uint64_t *m, uint64_t w) +{ + s2n_reductions++; + return __real_bignum_emontredc_8n(k, z, m, w); +} +#endif + +int main(void) { + enum { LEN = CT_POWM_BYTES }; + uint8_t out[LEN], base[LEN], mod[LEN], exp[LEN]; + + ct_fill(base, sizeof base); + ct_fill(mod, sizeof mod); + ct_fill(exp, sizeof exp); + mod[0] |= 0x80; /* full width */ + mod[LEN - 1] |= 1; /* and odd, which is what it wants */ + base[0] &= 0x7f; /* below the modulus */ + + /* The exponent is the private key. The base is the ciphertext, which an + * attacker chooses and already knows, so it stays public. */ + CT_SECRET(exp, sizeof exp); + + if (crypton_powm_sec(out, base, LEN, exp, LEN, mod, LEN) != 0) { + printf("powm_sec refused\n"); + return 1; + } + CT_PUBLIC(out, sizeof out); + ct_sink(out, sizeof out); +#ifdef CT_POWM_COUNT_S2N + printf("dispatch powm_s2n=%lu\n", s2n_reductions); +#endif + return 0; +} diff --git a/cbits/tests/ct/ct_x25519.c b/cbits/tests/ct/ct_x25519.c new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/ct_x25519.c @@ -0,0 +1,25 @@ +/* X25519. The scalar is the private key; the base point is the peer's + * public key and is not secret. */ +#include "tests/ct/ct.h" +#include + +void crypton_curve25519_donna(uint8_t *mypublic, const uint8_t *secret, + const uint8_t *basepoint); + +int main(void) { + uint8_t sec[32], base[32], out[32]; + static const uint8_t g[32] = {9}; + + ct_fill(sec, sizeof sec); + ct_fill(base, sizeof base); + CT_SECRET(sec, sizeof sec); + + crypton_curve25519_donna(out, sec, g); /* the public key */ + CT_PUBLIC(out, sizeof out); + ct_sink(out, sizeof out); + + crypton_curve25519_donna(out, sec, base); /* the shared secret */ + CT_PUBLIC(out, sizeof out); + ct_sink(out, sizeof out); + return 0; +} diff --git a/cbits/tests/ct/known.txt b/cbits/tests/ct/known.txt new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/known.txt @@ -0,0 +1,66 @@ +# Places where a secret reaches a branch that are known, understood and not +# defects. A driver reporting only these passes; anything else fails. +# +# Every entry is "file:line what it is". Keep it short: a long one means +# something has been accepted that should have been fixed. + +# assert() on a value derived from the secret. The asserted condition holds +# on every input -- these check an internal invariant of the reduction, not +# anything about the data -- so the branch goes the same way every time and +# no timing follows from it. They are reported because crypton's C is built +# without NDEBUG, so assert() is live in a released library. See the round +# eight notes in cbits/tests/ct/README. +p256.c:200 assert(top <= 1) in crypton_p256_modmul +p256.c:204 assert(top == 0) in crypton_p256_modmul +f_generic.c:94 assert on the borrow in crypton_gf_448_strong_reduce +f_generic.c:106 assert on the carry in crypton_gf_448_strong_reduce + +# The same thing for a different reason. crypton_gf_invert asserts that what +# it inverted had an inverse, and the two callers that ask for the assertion +# are inverting a projective z, which is never zero for a point on the curve. +# So this one holds because of what the callers pass rather than because of +# arithmetic, and it too goes the same way on every valid input. +decaf.c:136 assert(ret) in crypton_gf_invert + +# AArch64 only. gcc keeps a carry in the flags and takes it out with `cset` +# or `cinc`, where on x86-64 it uses `adc` and the carry never leaves the +# data path. memcheck calls `cset` a conditional move and reports it, and +# it attributes the report to the branch that ends the block rather than to +# the `cset` itself -- so the site it names is a loop back-edge, not the +# instruction that touched the secret. +# +# Checked by disassembling the address memcheck named, in a -no-pie build so +# that its addresses and objdump's agree. At every one of these the branch +# reads flags from a `cmp` against a loop counter or a pointer bound, both +# public; the only instructions consuming the secret's flags are `cset` and +# `cinc`, which do not branch and take the same time either way. In decaf's +# lookup the secret only reaches a `dup` and a NEON `and`/`orr`. +# +# 400f60 cmp x3, #0x20 <- public: four digits of 8 bytes +# 400f64 b.ne 400f3c <- what memcheck names +# 404c6c cmp x3, x6 <- public: j against n_table +# 404c70 b.ne 404c30 <- what memcheck names +addM@p256.c the carry of addM's loop, which is taken without a +# branch. Named by function rather than by line because +# the line is the compiler's choice and the function is +# not. Three compilers were asked: AArch64 names 147 and +# 149, gcc 11.5 on x86-64 names 147, 148 and 149, and the +# gcc 14 the CI uses names none of them -- and all three +# name addM. +# +# On AArch64 the only instructions consuming the secret's +# flags are cset and cinc, which do not branch. On x86-64, +# disassembled in a -no-pie build, the body between the +# loop label and its closing branch is +# +# 401ad6 mov (%rax,%rdx,1),%rbx +# 401aeb and %rcx,%rbx <- the secret, as a mask +# 401af1 add %r12,%r8 +# 401af4 adc %r13,%r9 <- carried, not branched on +# 401afd mov %rsi,0x120(%rsp,%rdx,1) +# 401b0e cmp $0x20,%rdx <- public: the loop offset +# 401b12 jne 401ad6 <- the only branch +# +# one branch, on a public counter; no conditional move, and +# no address the secret reaches. +constant_time.h:150 decaf's constant-time lookup, over j < n_table diff --git a/cbits/tests/ct/run.sh b/cbits/tests/ct/run.sh new file mode 100644 --- /dev/null +++ b/cbits/tests/ct/run.sh @@ -0,0 +1,365 @@ +#!/bin/sh +# Does the code that handles a secret run in time independent of it? +# +# memcheck already follows undefined bytes through arithmetic and reports the +# moment one decides a branch or an address. That is the same question, so +# each driver declares its secret undefined and runs under valgrind: every +# report names a place where the secret reached a branch or an index. +# The technique is Adam Langley's ctgrind. +# +# Without valgrind the drivers are still built and run, which says only that +# the plumbing is right -- it checks nothing about timing, and says so. +# +# Usage: cbits/tests/ct/run.sh [build-dir] +set -eu + +root=$(CDPATH= cd -- "$(dirname -- "$0")/../../.." && pwd) +out=${1:-$(mktemp -d)} +cc=${CC:-cc} +mkdir -p "$out" +cd "$root" + +D=cbits/decaf +decaf_src="$D/ed448goldilocks/decaf_all.c $D/ed448goldilocks/eddsa.c + $D/ed448goldilocks/scalar.c $D/p448/f_arithmetic.c + $D/p448/f_generic.c $D/utils.c $D/p448/arch_ref64/f_impl.c + cbits/crypton_sha3.c" +decaf_inc="-DCRYPTON_DECAF_WORD_BITS=64 -I$D/include -I$D/p448 + -I$D/include/arch_ref64 -I$D/p448/arch_ref64" + +# The portable C, not whatever the machine happens to offer, and the BearSSL +# it is written on. All three builds below have to be silent, so what tells +# them apart is which implementation each one's dispatch chose -- which the +# driver prints and run_one checks, since otherwise a build that quietly took +# the wrong path would pass by saying nothing. +aes_src="cbits/crypton_aes.c cbits/aes/generic.c cbits/aes/gf.c + cbits/bearssl/aes_ct64.c cbits/bearssl/aes_ct64_enc.c + cbits/bearssl/aes_ct64_dec.c cbits/bearssl/ghash_ctmul64.c + cbits/bearssl/dec32le.c" + +# The same driver again against the instructions, on each architecture that +# has them. +armv8_src="$aes_src cbits/aes/armv8.c cbits/crypton_cpu.c" +armv8_inc="-DWITH_ARMV8_CRYPTO -march=armv8-a+crypto -Icbits/aes" + +x86ni_src="$aes_src cbits/aes/x86ni.c cbits/crypton_cpu.c + cbits/aes/gcm_vaes_x86.c cbits/aes/gcm_vaes512_x86.c" +x86ni_inc="-DWITH_AESNI -DWITH_PCLMUL -maes -mpclmul -mssse3 -msse4.1 -Icbits/aes" + +status=0 +have_valgrind=no +ct_define= +if command -v valgrind > /dev/null 2>&1; then + have_valgrind=yes + ct_define=-DCRYPTON_CT_VALGRIND +fi + +# MLK_CONFIG_CT_TESTING_ENABLED turns on mlkem-native's own secret and +# declassify annotations, which are in the source at the points the algorithm +# says a value becomes public. The important one is the public seed: it is +# derived from the key generation seed, so it starts out secret, and the +# matrix it then seeds is generated by rejection sampling, which branches on +# the bytes it draws. Upstream declassifies the seed first -- rho travels in +# the encapsulation key and is not secret -- and without that the driver +# reports a leak that is not one. Those points are maintained by the people +# whose proofs cover this code, so they are a better answer than marking by +# hand from outside. +# +# Only with valgrind, since the macros include valgrind/memcheck.h. +mlkem_src="cbits/mlkem/crypton_mlkem.c" +mlkem_inc="-Icbits/mlkem${ct_define:+ -DMLK_CONFIG_CT_TESTING_ENABLED}" +# And the hand-written backend, where the architecture has one. Both builds +# have to be silent: unlike the AES pair below there is no variable-time +# implementation here for one of them to expose. +# +# crypton_cpu.c because the backend asks it whether this processor has the +# instructions before using them; without it the link fails on the +# capability function in cbits/mlkem/crypton_mlkem.h. cpu_stub.c is the one +# symbol that file needs from the AES build, which is not wanted here. +mlkem_native_src="$mlkem_src cbits/mlkem/crypton_mlkem_asm.S cbits/crypton_cpu.c \ + cbits/tests/ct/cpu_stub.c" +mlkem_native_inc="$mlkem_inc -DCRYPTON_MLKEM_NATIVE_BACKEND" + + +# Does this processor have AES instructions at all? +# +# Asked of the system rather than of crypton, because the question being +# settled is whether crypton found what is there -- and a witness that comes +# from the code under test cannot answer that. "Cannot tell" counts as +# having them, so an unfamiliar system turns into a loud failure rather than +# a silent skip. +# +# CRYPTON_CT_CPUINFO is for testing this function itself; nothing else sets +# it. +machine_has_aes() { + cpuinfo=${CRYPTON_CT_CPUINFO:-/proc/cpuinfo} + if [ -r "$cpuinfo" ]; then + # "aes" in Features on ARM, in flags on x86; -w so that a model + # name containing the letters does not answer for the flags + grep -qw aes "$cpuinfo" + return + fi + if [ "$(uname -s)" = Darwin ]; then + [ "$(sysctl -n hw.optional.arm.FEAT_AES 2>/dev/null)" = 1 ] && return 0 + [ "$(sysctl -n hw.optional.aes 2>/dev/null)" = 1 ] && return 0 + return 1 + fi + return 0 +} + +# run_one [driver] +# +# The driver defaults to ct_.c. It is given separately where one +# driver is built twice -- the same question asked of the portable C and of +# the hand-written backend -- rather than copying the file to a second name. +run_one() { + name=$1; srcs=$2; inc=$3; drv=${4:-$1} + # shellcheck disable=SC2086 + $cc -O2 -g $ct_define -Icbits -Icbits/include64 $inc \ + -o "$out/$name" "cbits/tests/ct/ct_$drv.c" $srcs 2> "$out/$name.cc" || { + echo "FAIL $name did not build"; sed -n '1,12p' "$out/$name.cc"; status=1; return + } + # Which implementation did this build's dispatch actually take? The + # driver says, and the three AES builds want different answers. This + # used to be inferred: the portable build was required to report, + # because the tables leaked, and silence meant it had taken an + # accelerated path instead. The tables are gone and all three are + # silent now, so the question is asked outright rather than read off a + # leak that no longer happens. + case $name in + aes | aes_armv8 | aes_x86ni) + want=0 + [ "$name" = aes ] || want=1 + got=$("$out/$name" 2>/dev/null | sed -n 's/^dispatch aes=\([01]\).*/\1/p') + if [ "$got" != "$want" ]; then + # A processor with no AES instructions is not a failure of + # this driver; there is simply nothing here to measure, and + # the portable one above has already measured what does run. + # Boards in this position are common -- the Raspberry Pi 2, + # 3 and 4 among them, on either word size. + if [ "$want" = 1 ] && [ "$got" = 0 ] && ! machine_has_aes; then + echo "skip $name: this processor has no AES instructions," + echo " so there is nothing here to measure" + return + fi + echo "FAIL $name: dispatch took aes=$got, wanted aes=$want --" + echo " this build is not running the implementation it is named for" + status=1 + return + fi + ;; + powm_s2n_*) + got=$("$out/$name" 2>/dev/null | sed -n 's/^dispatch powm_s2n=\([0-9]*\).*/\1/p') + if [ -z "$got" ]; then + echo "FAIL $name: it did not run to the end on this processor" + status=1 + return + elif [ "$got" -eq 0 ]; then + echo "FAIL $name: the s2n-bignum reduction ran 0 times --" + echo " this build checked the C again, not the path it is named for" + status=1 + return + fi + ;; + esac + if [ "$have_valgrind" = no ]; then + "$out/$name" > /dev/null 2>&1 && echo "built $name (no valgrind here; nothing checked)" \ + || { echo "FAIL $name did not run"; status=1; } + return + fi + # in a subshell, so that a driver valgrind kills is not announced by + # this shell as well -- the log says what happened + ( valgrind --error-exitcode=0 --track-origins=yes --num-callers=20 \ + --log-file="$out/$name.log" "$out/$name" > /dev/null 2>&1 ) \ + 2> /dev/null || true + # A driver valgrind could not run to the end reports nothing, and nothing + # used to read as a pass. An instruction valgrind cannot decode is the + # likely one: the vendored ADX assembly needs a valgrind that knows it. + if grep -q "unhandled instruction\|Illegal instruction\|SIGILL" "$out/$name.log"; then + case $name in + powm_s2n_*) + echo "skip $name: this valgrind cannot decode the vendored assembly's" + echo " instructions, so it cannot run the path this build is for" + ;; + *) + echo "FAIL $name: valgrind could not run it to the end" + grep -m3 "unhandled instruction\|Illegal instruction\|SIGILL" "$out/$name.log" | sed 's/^/ /' + status=1 + ;; + esac + return + fi + n=$(grep -c "^==[0-9]*== \(Conditional jump\|Use of uninitialised\)" "$out/$name.log" || true) + # Which places did it name? Only the frame the report is against -- the + # first "at" line under the complaint -- is the place; the "by" lines + # below it are how the code got there and are not themselves branching + # on anything. Nor is the "at" line under "Uninitialised value was + # created by", which --track-origins prints to say where the value came + # from: that frame is a stack allocation, not a branch, and taking it + # for one put a function's opening brace on the list. + # + # A site is kept as "function@file:line", and known.txt may name any of + # the three widths: that exact line, the function, or the whole file. + # The function is there because the line is the compiler's choice and + # not the code's: gcc 14 names no line of addM's loop at all, gcc 11.5 + # on x86-64 names three, and clang on AArch64 named two of those three. + # What the disassembly establishes is a property of the function, so + # that is what an entry should be able to say. + sites=$(awk ' + /^==[0-9]*== (Conditional jump|Use of uninitialised)/ { want = 1; next } + want && /^==[0-9]*== at 0x/ { print; want = 0 } + ' "$out/$name.log" | + sed -n 's/^==[0-9]*== at 0x[0-9A-Fa-f]*: \([A-Za-z_0-9.]*\) (\([^)]*\))$/\1@\2/p' | + grep -v '@ct_' | sort -u) + unknown= + for site in $sites; do + funcfile=${site%%:*} + fileline=${site#*@} + file=${fileline%%:*} + if grep -q "^$fileline[[:space:]]" cbits/tests/ct/known.txt || + grep -q "^$funcfile[[:space:]]" cbits/tests/ct/known.txt || + grep -q "^$file[[:space:]]" cbits/tests/ct/known.txt; then + continue + fi + unknown="$unknown $site" + done + + case $name in + canary) + # the calibration: silence here would mean the marking never reached + # the code, and every other zero in this run would be worthless + if [ "$n" -eq 0 ]; then + echo "FAIL canary: the deliberately leaky driver reported nothing," + echo " so the marking is not reaching the code and nothing below counts" + status=1 + else + echo "ok canary: reported $n, so the marking works" + fi + ;; + aes | aes_armv8 | aes_x86ni) + # Nothing here looks anything up: the portable AES and GHASH are + # bitsliced, and the two instruction sets branch on nothing. So + # anything at all is a finding and known.txt does not apply -- + # until the tables went this entry read the other way round, with + # the portable build required to report. + if [ "$n" -eq 0 ]; then + echo "ok $name: the secret decided nothing" + else + echo "FAIL $name: $n report(s) from AES or GHASH," + echo " which should branch on nothing:" + for site in $sites; do echo " $site"; done + sed -n '/Conditional jump\|Use of uninitialised/,/^==[0-9]*== $/p' \ + "$out/$name.log" | head -30 | sed 's/^/ /' + status=1 + fi + ;; + *) + if [ "$n" -eq 0 ]; then + echo "ok $name: the secret decided nothing" + elif [ -z "$unknown" ]; then + echo "ok $name: $n report(s), all known -- $(echo "$sites" | tr '\n' ' ')" + else + echo "REPORT $name: the secret decided a branch or an address" + echo " somewhere not listed in cbits/tests/ct/known.txt:" + for site in $unknown; do echo " $site"; done + sed -n '/Conditional jump\|Use of uninitialised/,/^==[0-9]*== $/p' \ + "$out/$name.log" | head -30 | sed 's/^/ /' + status=1 + fi + ;; + esac +} + +# The calibration first: it must report, or nothing below means anything. +run_one canary "" "" + +run_one powm "cbits/crypton_powm.c" "" + +# The same exponentiation through s2n-bignum, which is what RSA signing runs +# on x86-64 with ADX. The build above is the C: it does not define +# CRYPTON_S2N_BIGNUM, so from the release the assembly went in until this +# was added, the path that actually runs was never put through this +# harness. Each size is a different routine -- Karatsuba at 1024 and 2048, +# a padded operand through the 2048-bit one at 1536, schoolbook at 3072 and +# 4096 -- so each is built. +# +# ADX is assumed by CRYPTON_CPU_TESTING_ADX, and only where the processor +# reports it: under valgrind the processor's answers are valgrind's, and +# what this is for is the assembly, not valgrind's opinion of the +# processor. The driver counts the reductions it did through the assembly, +# and a count of nothing fails the build rather than passing it. +case $(uname -m) in +x86_64 | amd64) + if grep -qw adx "${CRYPTON_CT_CPUINFO:-/proc/cpuinfo}" 2>/dev/null; then + s2n=cbits/s2n/x86_att + powm_s2n_src="cbits/crypton_powm.c cbits/crypton_cpu.c cbits/tests/ct/cpu_stub.c + $s2n/bignum_montifier.S $s2n/bignum_emontredc_8n.S + $s2n/bignum_kmul_16_32.S $s2n/bignum_ksqr_16_32.S + $s2n/bignum_kmul_32_64.S $s2n/bignum_ksqr_32_64.S + $s2n/bignum_sqr_8_16.S $s2n/bignum_mul_8_16.S + $s2n/bignum_mul.S $s2n/bignum_sqr.S" + powm_s2n_inc="-Icbits/s2n/include -DCRYPTON_S2N_BIGNUM + -DCRYPTON_CPU_TESTING_ADX -DCT_POWM_COUNT_S2N + -Wl,--wrap=bignum_emontredc_8n" + for bytes in 128 192 256 384 512; do + run_one "powm_s2n_$((bytes * 8))" "$powm_s2n_src" \ + "$powm_s2n_inc -DCT_POWM_BYTES=$bytes" powm + done + else + echo "skip powm_s2n: this processor reports no ADX, so the vendored" + echo " multiplication cannot run here" + fi + ;; +aarch64 | arm64) + # the generated squaring is separate code at each size; 2048 is above + for bytes in 128 192; do + run_one "powm_$((bytes * 8))" "cbits/crypton_powm.c" \ + "-DCT_POWM_BYTES=$bytes" powm + done + ;; +esac +run_one p256 "cbits/p256/p256.c cbits/p256/p256_ec.c" "" +run_one x25519 "cbits/curve25519/curve25519-donna-c64.c" "" +run_one ed25519 "cbits/ed25519/ed25519.c cbits/crypton_sha512.c" "-Icbits/ed25519" +run_one decaf "$decaf_src" "$decaf_inc" +run_one chapoly "cbits/crypton_chacha.c cbits/crypton_poly1305.c" "" +run_one mlkem "$mlkem_src" "$mlkem_inc" +run_one aes "$aes_src" "" + +# Only where the instructions exist. Elsewhere there is nothing to measure +# and the build would not even compile. +case $(uname -m) in +aarch64 | arm64) + run_one aes_armv8 "$armv8_src" "$armv8_inc" aes + ;; +x86_64 | amd64) + run_one aes_x86ni "$x86ni_src" "$x86ni_inc" aes + ;; +esac + +# The ML-KEM backend, on both architectures that have one. x86-64 wants the +# flags its own build system passes, or the AVX2 sources do not assemble. +# +# The backend now asks the processor before it runs, so on x86-64 without +# AVX2 this build measures the portable C under a name that says otherwise. +# Say so and skip, rather than pass while checking the thing the plain +# mlkem build above has already checked. +case $(uname -m) in +aarch64 | arm64) + run_one mlkem_native "$mlkem_native_src" "$mlkem_native_inc" mlkem + ;; +x86_64 | amd64) + if grep -qw avx2 "${CRYPTON_CT_CPUINFO:-/proc/cpuinfo}" 2>/dev/null; then + run_one mlkem_native "$mlkem_native_src" "$mlkem_native_inc -mavx2 -mbmi2" mlkem + else + echo "skip mlkem_native: this processor has no AVX2, so the backend" + echo " would fall back to the C the mlkem build above measured" + fi + ;; +esac + +if [ "$have_valgrind" = no ]; then + echo "skip no valgrind here, so none of the above was checked" + exit 0 +fi +exit $status diff --git a/cbits/tests/endian/README b/cbits/tests/endian/README new file mode 100644 --- /dev/null +++ b/cbits/tests/endian/README @@ -0,0 +1,27 @@ +Does this C give the same answers on a big-endian machine? + +The cabal file lists s390x and ppc64 among the architectures it builds for. +Neither is in the CI matrix, and neither had ever compiled this code, let +alone run it. Eighteen files read their input through the loaders in +crypton_align.h -- which were rewritten from word-typed casts to memcpy in +#256 -- and eight more decide something from the byte order themselves. That +is the largest body of untested endianness-sensitive code in the tree, and the +riskiest part of it is the most recently written. + +The two sides cannot be compared in one run the way the 32-bit harness +compares two builds, because only one endianness exists on the machine doing +the comparing. So the answers are frozen instead: vectors.txt is what this +code gives on a little-endian host, and check mode reads it back and compares. +Any machine can run check mode, and a big-endian one that disagrees says so +line by line. + + cbits/tests/endian/run.sh generate write vectors.txt from this machine + cbits/tests/endian/run.sh check this machine against vectors.txt + +Generate mode is for a maintainer on a little-endian machine after adding a +primitive, and the file it writes is committed. It is not run in CI, where +only check mode makes sense. + +The first few lines of vectors.txt are published test vectors rather than +whatever this code happened to produce -- SHA-256 of "abc" and the like -- +so that a mistake on the generating host is caught rather than frozen in. diff --git a/cbits/tests/endian/endian.c b/cbits/tests/endian/endian.c new file mode 100644 --- /dev/null +++ b/cbits/tests/endian/endian.c @@ -0,0 +1,322 @@ +/* What this C answers, so that a big-endian machine can be asked the same. + * + * Every primitive here reads its input a word at a time, or writes its output + * that way, or both -- which is the step that goes wrong when the byte order + * changes. Each one is fed the same deterministic bytes at several lengths, + * including lengths either side of its block, since the tail is where the + * length is packed in and where the byte order shows. + */ +#include +#include +#include +#include + +#include "crypton_md4.h" +#include "crypton_md5.h" +#include "crypton_sha1.h" +#include "crypton_sha256.h" +#include "crypton_sha512.h" +#include "crypton_sha3.h" +#include "crypton_ripemd.h" +#include "crypton_skein256.h" +#include "crypton_skein512.h" +#include "crypton_tiger.h" +#include "crypton_whirlpool.h" +#include "crypton_chacha.h" +#include "crypton_salsa.h" +#include "crypton_poly1305.h" +#include "crypton_aes.h" +#include "aes/gf.h" +#include "aes/block128.h" + +/* The skein headers spell the prefix "cryponite", which nothing defines. */ +void crypton_skein256_init(struct skein256_ctx *ctx, uint32_t hashlen); +void crypton_skein256_update(struct skein256_ctx *ctx, const uint8_t *data, uint32_t len); +void crypton_skein256_finalize(struct skein256_ctx *ctx, uint32_t hashlen, uint8_t *out); +void crypton_skein512_init(struct skein512_ctx *ctx, uint32_t hashlen); +void crypton_skein512_update(struct skein512_ctx *ctx, const uint8_t *data, uint32_t len); +void crypton_skein512_finalize(struct skein512_ctx *ctx, uint32_t hashlen, uint8_t *out); + +static int generating; +static FILE *vf; +static int failures, checked; + +/* one answer: named, and either written out or compared with what was */ +static void answer(const char *name, const uint8_t *out, size_t n) { + char got[512], want[512], label[128]; + size_t i; + for (i = 0; i < n && i * 2 + 2 < sizeof got; i++) + snprintf(got + i * 2, 3, "%02x", out[i]); + got[n * 2] = 0; + /* an answer of no bytes still has to be a token, or the reader below + takes the next line's name for this line's answer and everything + after it is compared against the wrong thing */ + if (n == 0) strcpy(got, "-"); + if (generating) { + fprintf(vf, "%s %s\n", name, got); + return; + } + if (fscanf(vf, "%127s %511s", label, want) != 2) { + printf("FAIL %s: vectors.txt ended early\n", name); + failures++; + return; + } + checked++; + if (strcmp(label, name) != 0) { + printf("FAIL out of step: expected %s, vectors.txt has %s\n", name, label); + failures++; + } else if (strcmp(got, want) != 0) { + printf("FAIL %s\n little-endian %s\n this machine %s\n", name, want, got); + failures++; + } +} + +/* the input: deterministic, and at lengths either side of every block size */ +static const size_t lengths[] = {0, 1, 3, 55, 56, 63, 64, 65, 111, 112, + 127, 128, 129, 135, 136, 255, 256, 1000}; +static uint8_t buf[1024]; +static void fill(void) { + size_t i; + for (i = 0; i < sizeof buf; i++) buf[i] = (uint8_t)(i * 7 + (i >> 5) * 31); +} + +#define HASH(nm, ctxt, initcall, updcall, fincall, outlen) \ + do { \ + size_t li; \ + for (li = 0; li < sizeof lengths / sizeof *lengths; li++) { \ + ctxt ctx; \ + uint8_t out[outlen]; \ + char nmbuf[128]; \ + initcall; \ + updcall; \ + fincall; \ + snprintf(nmbuf, sizeof nmbuf, "%s/%zu", nm, lengths[li]); \ + answer(nmbuf, out, outlen); \ + } \ + } while (0) + +/* + * AES, which reaches further into the byte order than the hashes above do. + * + * The portable implementation keeps its schedule as 64-bit words and reads + * its input through br_dec32le; the GHASH beside it reads H and the + * accumulator as big-endian words; the counter modes carry a counter that is + * incremented big-endian and stored little-endian in one case and the other + * way round in another; and XTS doubles its tweak in GF(2^128) through + * cpu_to_le64. Every one of those is a place where a big-endian machine can + * differ, and none of them was asked about here until now. + * + * The entries called are the public ones, so this is whichever + * implementation the build installed -- which, for the build this harness + * makes, is the portable one. That is the one a big-endian machine runs: + * crypton has no AES instructions for s390x, and the POWER8 ones are + * little-endian only. + */ +static void aes_answers(void) { + static const uint8_t keylens[] = {16, 24, 32}; + /* multiples of the block, for the modes that take whole blocks */ + static const uint32_t blocks[] = {1, 2, 4, 7}; + /* and byte counts, including a partial block, for the ones that do not */ + static const uint32_t bytes[] = {0, 1, 15, 16, 17, 64, 100}; + uint8_t key[32], key2[32], iv[16], out[128], tmp[128]; + char nm[128]; + size_t ki, li; + uint32_t i; + + for (i = 0; i < 32; i++) { key[i] = (uint8_t)(i * 3 + 1); + key2[i] = (uint8_t)(i * 5 + 2); } + for (i = 0; i < 16; i++) iv[i] = (uint8_t)(i * 11 + 7); + + for (ki = 0; ki < sizeof keylens / sizeof *keylens; ki++) { + uint8_t kl = keylens[ki]; + aes_key k, k2; + aes_gcm_key gk; + + crypton_aes_initkey(&k, key, kl); + crypton_aes_initkey(&k2, key2, kl); + crypton_aes_gcm_key_init(&gk, &k); + + for (li = 0; li < sizeof blocks / sizeof *blocks; li++) { + uint32_t nb = blocks[li]; + aes_block ivb; + + crypton_aes_encrypt_ecb((aes_block *)out, &k, (aes_block *)buf, nb); + snprintf(nm, sizeof nm, "aes%u-ecb/%u", kl * 8, nb); + answer(nm, out, nb * 16); + + crypton_aes_decrypt_ecb((aes_block *)out, &k, (aes_block *)buf, nb); + snprintf(nm, sizeof nm, "aes%u-ecbd/%u", kl * 8, nb); + answer(nm, out, nb * 16); + + memcpy(&ivb, iv, 16); + crypton_aes_encrypt_cbc((aes_block *)out, &k, &ivb, (aes_block *)buf, nb); + snprintf(nm, sizeof nm, "aes%u-cbc/%u", kl * 8, nb); + answer(nm, out, nb * 16); + + memcpy(&ivb, iv, 16); + crypton_aes_decrypt_cbc((aes_block *)out, &k, &ivb, (aes_block *)buf, nb); + snprintf(nm, sizeof nm, "aes%u-cbcd/%u", kl * 8, nb); + answer(nm, out, nb * 16); + + memcpy(&ivb, iv, 16); + crypton_aes_encrypt_xts((aes_block *)out, &k, &k2, &ivb, 0, + (aes_block *)buf, nb); + snprintf(nm, sizeof nm, "aes%u-xts/%u", kl * 8, nb); + answer(nm, out, nb * 16); + + memcpy(&ivb, iv, 16); + crypton_aes_decrypt_xts((aes_block *)out, &k, &k2, &ivb, 0, + (aes_block *)buf, nb); + snprintf(nm, sizeof nm, "aes%u-xtsd/%u", kl * 8, nb); + answer(nm, out, nb * 16); + + /* a starting point too, since that is extra tweak doubling */ + memcpy(&ivb, iv, 16); + crypton_aes_encrypt_xts((aes_block *)out, &k, &k2, &ivb, 3, + (aes_block *)buf, nb); + snprintf(nm, sizeof nm, "aes%u-xts-sp3/%u", kl * 8, nb); + answer(nm, out, nb * 16); + } + + for (li = 0; li < sizeof bytes / sizeof *bytes; li++) { + uint32_t n = bytes[li]; + aes_block ivb; + + memcpy(&ivb, iv, 16); + crypton_aes_encrypt_ctr(out, &k, &ivb, buf, n); + snprintf(nm, sizeof nm, "aes%u-ctr/%u", kl * 8, n); + answer(nm, out, n); + + /* the tag goes after the ciphertext, so this answers for both */ + crypton_aes_gcm_full_encrypt(out, &gk, &k, iv, 12, buf, 13, + buf, n, 16); + snprintf(nm, sizeof nm, "aes%u-gcm/%u", kl * 8, n); + answer(nm, out, n + 16); + } + } + + /* GHASH and POLYVAL on their own, which the modes above reach only + * through whatever length they were given */ + { + table_4bit ht; + block128 acc; + aes_polyval pv; + + crypton_aes_generic_hinit(ht, (const block128 *)buf); + memcpy(&acc, buf + 16, 16); + crypton_aes_generic_gf_mul(&acc, ht); + answer("ghash-mul", (const uint8_t *)&acc, 16); + + crypton_aes_generic_hinit(ht, (const block128 *)buf); + memcpy(&acc, buf + 16, 16); + crypton_aes_generic_gf_mul4(&acc, (const block128 *)(buf + 32), ht); + answer("ghash-mul4", (const uint8_t *)&acc, 16); + + memcpy(tmp, buf, 16); + crypton_aes_polyval_init(&pv, (const aes_block *)tmp); + crypton_aes_polyval_update(&pv, buf + 16, 64); + crypton_aes_polyval_finalize(&pv, (aes_block *)out); + answer("polyval", out, 16); + } +} + +int main(int argc, char **argv) { + generating = (argc > 1 && strcmp(argv[1], "generate") == 0); + vf = fopen(argc > 2 ? argv[2] : "cbits/tests/endian/vectors.txt", + generating ? "w" : "r"); + if (!vf) { printf("cannot open vectors.txt\n"); return 2; } + fill(); + + HASH("md4", struct md4_ctx, crypton_md4_init(&ctx), + crypton_md4_update(&ctx, buf, (uint32_t)lengths[li]), + crypton_md4_finalize(&ctx, out), 16); + HASH("md5", struct md5_ctx, crypton_md5_init(&ctx), + crypton_md5_update(&ctx, buf, (uint32_t)lengths[li]), + crypton_md5_finalize(&ctx, out), 16); + HASH("sha1", struct sha1_ctx, crypton_sha1_init(&ctx), + crypton_sha1_update(&ctx, buf, (uint32_t)lengths[li]), + crypton_sha1_finalize(&ctx, out), 20); + HASH("sha256", struct sha256_ctx, crypton_sha256_init(&ctx), + crypton_sha256_update(&ctx, buf, (uint32_t)lengths[li]), + crypton_sha256_finalize(&ctx, out), 32); + HASH("sha512", struct sha512_ctx, crypton_sha512_init(&ctx), + crypton_sha512_update(&ctx, buf, (uint32_t)lengths[li]), + crypton_sha512_finalize(&ctx, out), 64); + /* sha3's context ends in a flexible buffer whose width comes from the + * hash length, so it is not a plain automatic variable like the rest. */ + { + size_t li; + for (li = 0; li < sizeof lengths / sizeof *lengths; li++) { + uint8_t space[SHA3_CTX_BUF_MAX_SIZE]; + struct sha3_ctx *ctx = (struct sha3_ctx *)space; + uint8_t out[32]; + char nmbuf[128]; + crypton_sha3_init(ctx, 256); + crypton_sha3_update(ctx, buf, (uint32_t)lengths[li]); + crypton_sha3_finalize(ctx, 256, out); + snprintf(nmbuf, sizeof nmbuf, "sha3-256/%zu", lengths[li]); + answer(nmbuf, out, sizeof out); + } + } + HASH("ripemd160", struct ripemd160_ctx, crypton_ripemd160_init(&ctx), + crypton_ripemd160_update(&ctx, buf, (uint32_t)lengths[li]), + crypton_ripemd160_finalize(&ctx, out), 20); + HASH("skein256", struct skein256_ctx, crypton_skein256_init(&ctx, 256), + crypton_skein256_update(&ctx, buf, (uint32_t)lengths[li]), + crypton_skein256_finalize(&ctx, 256, out), 32); + HASH("skein512", struct skein512_ctx, crypton_skein512_init(&ctx, 512), + crypton_skein512_update(&ctx, buf, (uint32_t)lengths[li]), + crypton_skein512_finalize(&ctx, 512, out), 64); + HASH("tiger", struct tiger_ctx, crypton_tiger_init(&ctx), + crypton_tiger_update(&ctx, buf, (uint32_t)lengths[li]), + crypton_tiger_finalize(&ctx, out), 24); + HASH("whirlpool", struct whirlpool_ctx, crypton_whirlpool_init(&ctx), + crypton_whirlpool_update(&ctx, buf, (uint32_t)lengths[li]), + crypton_whirlpool_finalize(&ctx, out), 64); + + /* The stream ciphers and the one-time authenticator. Each reads its key + * and its input a word at a time and writes its output the same way, so + * the byte order shows in the answer rather than in a length field. */ + { + size_t li; + for (li = 0; li < sizeof lengths / sizeof *lengths; li++) { + crypton_chacha_context cctx; + crypton_salsa_context sctx; + poly1305_ctx pctx; + poly1305_key pkey; + poly1305_mac mac; + uint8_t key[32], iv[8], outbuf[1024]; + char nmbuf[128]; + size_t i; + + for (i = 0; i < sizeof key; i++) key[i] = (uint8_t)(i * 11 + 3); + for (i = 0; i < sizeof iv; i++) iv[i] = (uint8_t)(i * 5 + 1); + + crypton_chacha_init(&cctx, 20, sizeof key, key, sizeof iv, iv); + crypton_chacha_combine(outbuf, &cctx, buf, (uint32_t)lengths[li]); + snprintf(nmbuf, sizeof nmbuf, "chacha20/%zu", lengths[li]); + answer(nmbuf, outbuf, lengths[li] < 64 ? lengths[li] : 64); + + crypton_salsa_init(&sctx, 20, sizeof key, key, sizeof iv, iv); + crypton_salsa_combine(outbuf, &sctx, buf, (uint32_t)lengths[li]); + snprintf(nmbuf, sizeof nmbuf, "salsa20/%zu", lengths[li]); + answer(nmbuf, outbuf, lengths[li] < 64 ? lengths[li] : 64); + + for (i = 0; i < sizeof pkey; i++) pkey[i] = (uint8_t)(i * 13 + 7); + crypton_poly1305_init(&pctx, &pkey); + crypton_poly1305_update(&pctx, buf, (uint32_t)lengths[li]); + crypton_poly1305_finalize(mac, &pctx); + snprintf(nmbuf, sizeof nmbuf, "poly1305/%zu", lengths[li]); + answer(nmbuf, mac, sizeof mac); + } + } + + aes_answers(); + + if (!generating && failures == 0) + printf("ok %d answers match the little-endian ones\n", checked); + else if (!generating) + printf("FAIL %d of %d answers differ\n", failures, checked); + fclose(vf); + return failures != 0; +} diff --git a/cbits/tests/endian/run.sh b/cbits/tests/endian/run.sh new file mode 100644 --- /dev/null +++ b/cbits/tests/endian/run.sh @@ -0,0 +1,62 @@ +#!/bin/sh +# Does this C give the same answers on a big-endian machine? +# +# s390x and ppc64 are among the architectures the cabal file builds for, and +# neither is in the matrix. Eighteen files read their input through the +# loaders in crypton_align.h -- rewritten from word-typed casts to memcpy in +# #256 -- and eight more decide something from the byte order themselves. +# +# AES is here as well, and it reaches further into the byte order than the +# hashes do: a schedule kept as 64-bit words, a GHASH that reads H and its +# accumulator as big-endian words, counters incremented one way and stored +# the other, and an XTS tweak doubled through cpu_to_le64. It is the +# portable implementation that answers, which is what a big-endian machine +# runs: crypton has no AES instructions for s390x, and the POWER8 ones are +# little-endian only. +# +# The two sides cannot be compared in one run the way the 32-bit harness +# compares two builds, because the machine doing the comparing has only one +# byte order. So the answers are frozen: vectors.txt is what this code gives +# on a little-endian host, and check mode reads it back. +# +# Usage: cbits/tests/endian/run.sh [generate] [build-dir] +set -eu + +root=$(CDPATH= cd -- "$(dirname -- "$0")/../../.." && pwd) +mode=check +if [ "${1:-}" = generate ]; then mode=generate; shift; fi +out=${1:-$(mktemp -d)} +cc=${CC:-cc} +mkdir -p "$out" +cd "$root" + +srcs="cbits/crypton_md4.c cbits/crypton_md5.c cbits/crypton_sha1.c + cbits/crypton_sha256.c cbits/crypton_sha512.c cbits/crypton_sha3.c + cbits/crypton_ripemd.c cbits/crypton_skein256.c cbits/crypton_skein512.c + cbits/crypton_tiger.c cbits/crypton_whirlpool.c + cbits/crypton_chacha.c cbits/crypton_salsa.c cbits/crypton_poly1305.c + cbits/crypton_aes.c cbits/aes/generic.c cbits/aes/gf.c + cbits/bearssl/aes_ct64.c cbits/bearssl/aes_ct64_enc.c + cbits/bearssl/aes_ct64_dec.c cbits/bearssl/ghash_ctmul64.c + cbits/bearssl/dec32le.c" + +# Generating is done under the sanitizers, since a driver that writes out of +# bounds would otherwise freeze whatever it happened to leave behind. That +# is not hypothetical: sha3's context ends in a flexible buffer and the first +# draft of the driver put it on the stack as a plain struct. +san= +if [ "$mode" = generate ]; then + san="-fsanitize=address,undefined -fno-sanitize-recover=all" +fi + +# shellcheck disable=SC2086 +$cc -O2 -g $san -Icbits -Icbits/aes -Icbits/include64 -o "$out/endian" \ + cbits/tests/endian/endian.c $srcs + +if [ "$mode" = generate ]; then + "$out/endian" generate cbits/tests/endian/vectors.txt + echo "wrote $(wc -l < cbits/tests/endian/vectors.txt | tr -d ' ') answers" + echo "commit cbits/tests/endian/vectors.txt" +else + "$out/endian" check cbits/tests/endian/vectors.txt +fi diff --git a/cbits/tests/endian/vectors.txt b/cbits/tests/endian/vectors.txt new file mode 100644 --- /dev/null +++ b/cbits/tests/endian/vectors.txt @@ -0,0 +1,381 @@ +md4/0 31d6cfe0d16ae931b73c59d7e0c089c0 +md4/1 47c61a0fa8738ba77308a8a600f88e4b +md4/3 5dc60555aff84f4f4d75c487477bd40e +md4/55 7fb3b8a921fbb273575603f44876e276 +md4/56 150c4579f2a8f07aa321b3825b46cbf3 +md4/63 a6fc40dc3217aaf3bdd437886ee24644 +md4/64 e29ccc2f2131c0d40572b6dbf243369b +md4/65 9e8345b86c1dc8ed95f1a591bd01a02a +md4/111 815d46faeafcc449d668b476b28d16b4 +md4/112 bb70170948d31fae99063ea7ce0be725 +md4/127 c6f47dc12fd474f894fa9f51e043a109 +md4/128 1379e209df9100a5b5c0d3c3068baf23 +md4/129 bf3a7c34766fa54bfa6335e8fd2d89b5 +md4/135 0985ede04e702bd38e6c10955120c3dc +md4/136 46e572e144b3130b4993660116ed1d29 +md4/255 adc67d84a7b2ccfe7e5d541a4c55aaae +md4/256 1ef108f24aac5e2df49d8e692a68cfae +md4/1000 0ae32726214fc6eb8133de6ec0e3ab03 +md5/0 d41d8cd98f00b204e9800998ecf8427e +md5/1 93b885adfe0da089cdf634904fd59f71 +md5/3 ed41c1aca5ad5feb033df37822dae4b7 +md5/55 c051c9637f919672c309560032dee7ab +md5/56 0f4e8d49afd96d05c011692366b0e92f +md5/63 39b5ef10a4f0648885ac75ec5fbfbd9e +md5/64 56bce76bade2e3259fd44ec9592893f3 +md5/65 a41bf4e0f25f0705ead601e8f5481dd1 +md5/111 39ffc0fb12a340f0f0cef28c7c5d73f2 +md5/112 9330636dc725a2bea4026e2ffcbf5e01 +md5/127 bb7911f7538cf8139325caf596135fa0 +md5/128 89248d055d67dc618b5fa8e6a7936747 +md5/129 e0295e53676fb80f77c499a6f747705d +md5/135 80c0d9601f6b0371c533d601deac9d15 +md5/136 5d49aa8ae1bb7f6d426897800b281547 +md5/255 38c3929ff959900de1f79decd32110ee +md5/256 def8272a2a5a237a4590c8abc83261c5 +md5/1000 6d1e3c683bc932d465a43307a4dfb791 +sha1/0 da39a3ee5e6b4b0d3255bfef95601890afd80709 +sha1/1 5ba93c9db0cff93f52b521d7420e43f6eda2784f +sha1/3 75550941124b46eb4161d17ac200c05c4fc03ce7 +sha1/55 4574696c8a057e3b8169f6946e96fe694ac1f63d +sha1/56 cc1ecce36813cd3f62aa8b6b00b9ca127619fc41 +sha1/63 fa09a13f3c17facdcf2fe69a63023e5b5bdb7ab4 +sha1/64 cb8d5827b951666a2fd890cea3fc5afbe527b6da +sha1/65 7264c8d694493c5f81fb537c6085387878aa1571 +sha1/111 50d721885aca3ebeeaaec214d7bed12303fdbbb9 +sha1/112 ecace0ac5661c929fbad2810d081988461f9375d +sha1/127 9039e73e76720f39656f5b10aa188d9135a0dd4e +sha1/128 cfb672f8266b134afdbc79b79988d9a2aa2c511f +sha1/129 f9ccb7eb40a100fa075f0f1992e61bc775d8c882 +sha1/135 9cec88751569f9d70ef6bb764fac3a2f2ec93a51 +sha1/136 2125c7780f72d171f651cdb70efa0f5011a45c1d +sha1/255 c9069eb027c7b6de5d2385779003c0c3b81152f4 +sha1/256 d397a655930755132ebc094b6353b23dfee0f155 +sha1/1000 7f0b25e59adefabd5323e7fba76def0018847494 +sha256/0 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 +sha256/1 6e340b9cffb37a989ca544e6bb780a2c78901d3fb33738768511a30617afa01d +sha256/3 b361d0f9a938a2bb4fbdc9c21dc5a859788041b0040919d8a811c1888184f4df +sha256/55 8c9d785249c00ba19b5c8c6d0df51b3da614b8af732fe566909d47897f580f79 +sha256/56 e458987ff5bc3a0e1d2d84fb1cab06a8d8e7bcec2d5b6fe43498280b93d65003 +sha256/63 e3c8148d3ba2928b012cc55b325faa2d2bec48d319fed4ea2a3791150ebbfb60 +sha256/64 e9c6e41c26bd6bedc6a8e61bc7a02b85818c632875643db2c5c5db65eac2d0b8 +sha256/65 f0960adca320b44bc378b2839ad6b0dbfd2d7d98fb12802f7fb823ab9c424f93 +sha256/111 ecf5dd1da8d2b562cbdb69e6714ad568483f1b0a5b79c65e07d4b1807190fc90 +sha256/112 51221cdabbf22da7d38a3ec00200cd011bab7487bdf6947bb3356bc0c94bdea8 +sha256/127 75df9f55cf2ec2201b5bdb9671df418dfd6b7fdcff8dd85b768872949ef0863c +sha256/128 fa52030abda28172c20bc098a70e13e4a764979b7f536a332f6ea29867c0dcb3 +sha256/129 e08048e99eb6c53cb09d709071f1f1eda4d52cbfbff9dd4e756ce4c939a50498 +sha256/135 4cf50d0a4148ed5147501e6ef09835c775da2ac903e0760288bdfa03454fa923 +sha256/136 a571b968a9479df4ddb5bedcaa1cde191864b5fee12d969407056eeb2a39aeb0 +sha256/255 eb6337ccc19a34a3ac4fb140185955cfaaab38440deb2ff87ed6f22ff1fa0239 +sha256/256 0617cb171a31c804506afbbec540a7a1d17953017a883fc7ff2bc913d77bf2e7 +sha256/1000 7fd5abf2b68d694aa89ca16c6a325450c83ad0b07cdbe3c86dbd55db684623b6 +sha512/0 cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e +sha512/1 b8244d028981d693af7b456af8efa4cad63d282e19ff14942c246e50d9351d22704a802a71c3580b6370de4ceb293c324a8423342557d4e5c38438f0e36910ee +sha512/3 9d1d055e0ac16db8b8b3b6c44692743b855129eb6b9fb9706ac623cf5c086ce4983ae494e5abeafc77a05e5e95f87c96bf8bd9434bccace808732165a43a6412 +sha512/55 541042668dfe41adb3801f723e7c02886e5f1b4fda826bfe4d2ae2a2749ad9f2a764509f74018e8b050c6e822283546066fa6aa09389a50d4b66c0890ad146cd +sha512/56 a09c32a7c3a410c136ef656933a616366394455b85ec7ad284f1048d7472d68cea43de3df7f24b384ebcd724ee552325dfef830619022a189ef2c36fa702693c +sha512/63 4d6719f704edfd162acd129079a6692dfcbd99497394aa8703667288e3fca6bc1bc45f95616458ae43b7529ffd08d7b06aa5b5a8193051aeaec23cedc444dc96 +sha512/64 dce432552c3db0b0628a924dd1fb617d7ccff63dd7643aaba7f0d97629c6c22cb8e9f24eea742b4aaa5b72f4772ca387a1175ef2bc3f6d2653b3a9479ef35fa5 +sha512/65 0c627ddbf62632ac97c3b955db49181fec702992f0a0bd07b553f226075133db40532c90854b1447675ced18f950fa2a2207d6b6db1757b34374e0f5f3900a3b +sha512/111 993f2d3b273771809640eb7cd2f2830681faaec2d2ea91375b6467b5b04b0581e02262d9715201c7b5c1abcb3f5a9f64d4f2e8f18961a25153d3b6c99a81f501 +sha512/112 186430ceef81d4724e1196c32a956fff9c11676064711432f9b828001c488cd3beaf8f35c3283887bbdbed1915c5dce0b298d9c88ccf8113ad5e1f0f6121993c +sha512/127 1dae2ca13d511448d2fda00b57f03481a3c01a51273c756f6dd4934a2c271f0096960b3188a2cf1c485327958785abdfcc876e17ad4f16a66d311a81270a8b40 +sha512/128 dfe931ea5f331a50c1f2e2b8ef5e9d417bac976f13c1439a1655a134669a64d4f9c33ad540233a43796ea6266eae953a6b99d49fd88f27d4f342c0fe43ae42ca +sha512/129 eca5ea3ff8033c59d5a891c5d0e2cfb099d4cab8bcea7f4878342fa60a10f26cb3e570870932c69b6b3419e865f9dd4e0281a132861c54ea21947d3bbd599aa1 +sha512/135 000bb1bda38edd5e935b1f4a865ff74cacc325af1c8485ddd04a0f4e3f859ac5c1733f32af5c6646cb94dbb4678d8704d022fbfcea8f2dbcbc41e5cdc5921fed +sha512/136 0b0ce141f1a6a6d27522b26a2683aac47b04e99d64e29472969391fe5521dd0d7e6898c84b007103ee908dc73f7acb7cb1c4a4d4b418aaa3edba51cdd9e3de60 +sha512/255 45bf80a489f2579063e0890396d5bcbea66469e93ccf0e04cf933e7b10e7e9826ec0376d706b0f601599f69dc8629a1bdb4de1c9b03890ea0ac7889db258ffef +sha512/256 8e287890cf2beb5414ff4035dc579fc41b8d9ef21febb0a40e3d7c310f291ed3c2588a637bd314ea3b73ac398d0183a0dca8e4cc830cc59f39b9567427766bba +sha512/1000 8f61fbcd0e1f611d5f996c4c21002f9311ca0e4f4a101fd9843596a5693230e40e6f3fbe59bbdfd5ece955563234939d2a1ec809f6572ac06119f82a5c1a90e1 +sha3-256/0 a7ffc6f8bf1ed76651c14756a061d662f580ff4de43b49fa82d80a4b80f8434a +sha3-256/1 5d53469f20fef4f8eab52b88044ede69c77a6a68a60728609fc4a65ff531e7d0 +sha3-256/3 aa6c85d1a41110dd488f4a35719cf4bbcf2634d7c50401958ad41097096c7760 +sha3-256/55 81fe59ad864db751440e1251d7792a74d1af93ff48e24b25b65639f97bf48bf2 +sha3-256/56 129a430d7d7f2ad866b0da79f1c50a8a0f80f248c21a280eb3528cdff3b6f26c +sha3-256/63 d305b528e64c7e3c88cc1a937c152c169ad07871cb359e71f478ca6c1ad20c7e +sha3-256/64 c47f2570ff2e79c00bcbd0674a2ee9e0eed07c0c5db8c3a1aa68357aa39abe97 +sha3-256/65 5b73115520ad1b600cae25ebdbd2510037ea11e158c7ec8d2b63b9945b32981a +sha3-256/111 2aa0850d3ed0fa9d5e8f00053f0f4f8248dc24577925d27c82e7584a49da39a9 +sha3-256/112 c031eec2ab94c67081f5729f8718dc048d3a17daed923fc6d247f495ca66acf7 +sha3-256/127 92927f72784017dfc937c60ced80e2bba7c59abe9367e8bbf95b76afd9123d04 +sha3-256/128 10cab47e12f96037ac9d1d962a2031fd9e3bdf345cb5ae4827ae19ab1c50d077 +sha3-256/129 fb32880265355aa1082038604b8b633933f16c650dd402546e5f92db425a6663 +sha3-256/135 009f270e8f6dcaa6a7af6134429c49068a372ae93ed022bfd39eb665b3b226ee +sha3-256/136 efde869e19bad5db360fe59a0ea91abe35ea8e0fc87aadcef11d480cb63d1d7e +sha3-256/255 1998ee5adc07d9b0f3c87bb2520dac580b48cb1e65ee14959634f4e25e78618e +sha3-256/256 71d78d32c485425571f393927366001e60ea0d404be34137b9bfad1c065293d0 +sha3-256/1000 671cf26630a648c48c2eee1125dbcdc3f0b0ee7ea5fa5e4311efe7b536db03ea +ripemd160/0 9c1185a5c5e9fc54612808977ee8f548b2258d31 +ripemd160/1 c81b94933420221a7ac004a90242d8b1d3e5070d +ripemd160/3 1a51cd060324236aef3ca82f633638c3d34693f7 +ripemd160/55 79dd03427a79bff167b170be1fd01edaa69c48da +ripemd160/56 8fd66d9a2808b45155c2532c5be09a49db1138b7 +ripemd160/63 9fed7522e7537e1ec07d1d3fb04fe528b94db59d +ripemd160/64 bb8e58b9b697f8a6001318e6dd7b97fdb96cbbb2 +ripemd160/65 9f4b683e9ffcef970435986daab885750258af82 +ripemd160/111 c46b3c4192a0b9db5b6f327dca3d2979536a51b8 +ripemd160/112 4146822be4f0fb524965b46e18b405400e4a077e +ripemd160/127 f000414f29cfd8c047cc4e00594ba4448baae0a1 +ripemd160/128 2864a0a1a3f701a27dd29eb3b08e6eb98705e6da +ripemd160/129 5d12f9405bf021ac7edd672d9b05f0be6c40b5c3 +ripemd160/135 87667a9b70214ed9552e049c9e173f774e7045a6 +ripemd160/136 34e83a91054cac23d5bee365e346bd9592e3ad80 +ripemd160/255 fd4fa53a24ed05cac5746fe7ad4a51aa7b0f906f +ripemd160/256 4d29394d45cb3a350fb04abfd7abefd98d5dbc71 +ripemd160/1000 09d36b8fc9c87979223ad021c0e5bed386f1c502 +skein256/0 c8877087da56e072870daa843f176e9453115929094c3a40c463a196c29bf7ba +skein256/1 34e2b65bf0be667ca5deba82c37cb253eb9f8474f3426ba622a25219fd182433 +skein256/3 175623dce1b0cb69bed8a38af2ec2d8f81e2ca1edac0423bc765b90a89eb693f +skein256/55 acb6ea7280880ee8814bf2228fba49c9860bea7bcdd18f5fa9c93471d3af4585 +skein256/56 87edff7a077c3a72ff2ad09dad290dcb156aaacf472d11451231cfb48577960d +skein256/63 3a74537a726a4987042e8cf3c285396b94cdca3297f3c618af0035efd2d77cd0 +skein256/64 31e6401745b92aa5cabbb55a032b9285e9a978fd7ee0a7756903190bf31dffc2 +skein256/65 ec66f99651b0d7d4cfead397b4d3d2284fe0d3cde3df2fc00f7faea0d5e83eb0 +skein256/111 a67fc7fbb5babe99f92ccd808bf61af941a69ea064b01c145ab8b8a51c65dc83 +skein256/112 c738917c7e782f30c1bb7af09e9593f5e12ba339353f10a12c83d9054bd81c53 +skein256/127 c3ba1dfa5b9dd1e80a203026213e34b5408663a895f91a09e719f9dcb187f7b4 +skein256/128 81188e87abc45684a64ca1591259830f758a5c8b11a7144a0c58fbab6923bd7e +skein256/129 f6c477673becfbbbdc6b6510649f94423b91d5b159181d8cdb4d60d5aadb9c9e +skein256/135 b5c7ba0a7265390af1982e3c61d15d30cf6f25b6c0452d6bbb932de85cf8d1fc +skein256/136 e982d50fc214234541c6b1779bacf6c6d592d6c88275635f3fe9f4517c04a3e9 +skein256/255 e9a0544f7252e46be8e3dedf5a9db4f502f01183a6c593cf25e7fad93ac5e888 +skein256/256 625327ea0bc0edcb43b70f0b979742cf223f55ae15bf021f3636d50141b23853 +skein256/1000 23b8bbd0a122b07e9ccf552a0763e7aaead3ef25b74cea8282ea0c0fd24de7b9 +skein512/0 bc5b4c50925519c290cc634277ae3d6257212395cba733bbad37a4af0fa06af41fca7903d06564fea7a2d3730dbdb80c1f85562dfcc070334ea4d1d9e72cba7a +skein512/1 40285f433699a1d8c799b276ccf18010c9dc9d418b0e8a4ed987b44c61c01c5ccbcc0977b1d34a4d3665d20e12716df934d208fea6607f74968ed86be3c99832 +skein512/3 cf8bc9d2f25e5a8633d564fd73ffc76308819232a13a1dece60ce8e81651e5bd7569c95cb4b5772c1cfa885acadfdbab5bb94b28b01421770f278b1d8968c053 +skein512/55 523aecc435dcb541f5be26dd5935132b1f40bc78abd8aaa2b87f29202837e3b55ad576ffeb1692a47b6e0c5a2bb0fb669262751900dcf5c63dd654043ed934a7 +skein512/56 81641af4c81d0f52f0345c1e742a8c52a28fc09b22ad132c5599fb50c77f5c4f310d3a213fe7a1768711df13361a60c67e0ab6587592b30aa01433a79b134260 +skein512/63 5aa0c9c865d56c0dc8e6430290d3e3d6c785912c1ec2961dc4a928efefe4faa4711ae95b976db1b3d1af33cb4992dd84e35ce204371d357fc03754f9eb3dc25b +skein512/64 2ebf29904f1d770a56fa50408927dc948bb28bed094aa735940653b204b2e23adb18932893cb4f927be7d0b86eed144f7cbccb02afacb47c834b4fdcafce5dac +skein512/65 debfe997c41e391d00143289d49c807b1b763569b29b013ff58b015172859eaa202b223f17c357506765c2ec43f561c339d7b184b1982c1d44a3c4b3933f8d62 +skein512/111 f338e3b93dea8713b80f3d4f61f7cea1bb6b3c71ba90b0798e5178ba0c6431c68e34c6a6759c0c1181ab5ca765c17d3a200b4a440c262b92e2158b87a5576908 +skein512/112 4d6372e5f65a3462730a9db06eda6b6e2b35672abbc624cba58ec17440bd0d922a25d5923685d4625ca955c88da134353df92638d5cd3c3f9f1500138c639191 +skein512/127 c9094df9b9e0bfdb94a36b291fa26150a7d0ab2d5c90e9364c58ee1144eee6c8939333ed796af9ee0f84286f46748002cca1a9c9a6f84f3d3a71fa6562afece1 +skein512/128 9a69ca7d3222bb63d0ffa9b153ec118356e8c18b59adfb23376836b2d9f1a4684117000594261c91fbfef549f188cf831bbd27142b5af4df8be1f80ca0490d9b +skein512/129 cbef633515a50c561518d1a7a200de7cb269ea17a2861a852d65002562174a5b471adbd0e45ab4317827b2756f2be2b897da906456a062927d9e14839eae7ede +skein512/135 116b23ac95a0b499a62e571703f43af7e79c8de5450bc793964cdd1c64158eaa7ec463ce67ab1b3fd23c5b653ffa09015efbc225f753c0a5964687a16c25fe30 +skein512/136 12bc711a66f7f09f7ad2994e666c7064c3c9a1695a8fe6bc0092cf035e11192aab53c006e409449f0971fd68f4248253477d099ba2d85a4ada064bbe91c53558 +skein512/255 dd7f2f871bcdd425b07c691325b808aca79551e661dc9a72b95d5f059ae0c1e7039f64d10bea14b4c2fc251e5a7ce11b4e9400503fd7e3684d4eefcf2ceca425 +skein512/256 234270864103c6fed1dc8249bf6c91f5660f5438a6cec17c8f97dfbcde0e4a9ffc3a971f4e23d58db01fab96621a13f41ca4914883f7a149401a3d347161532c +skein512/1000 3631c9fcf9af7bd2c9e6b474bf56de55261f002c5a2d0ecb478a6024a56366efba38e4b6936d3a5cdc8f7800dc39250520dded0b3583719c2b3159a9ea326b25 +tiger/0 3293ac630c13f0245f92bbb1766e16167a4e58492dde73f3 +tiger/1 5d9ed00a030e638bdb753a6a24fb900e5a63b8e73e6c25b6 +tiger/3 53b5b6bd6bab3c1c63a8d8c62572e9a3502eae8e80d2143d +tiger/55 3aa4132b6dce81714c8b76a2b69eb25a870ca99101196d4b +tiger/56 28a5e8fd6f9b87ec0b9d2e46a94c9663dbe7d81eefb3fe48 +tiger/63 f919691beddf54789d147663465b2ec8f93d7c3999504aa1 +tiger/64 ff2896adeb7883a369da8e71ffa30c6ae0892bd7e4f3b943 +tiger/65 7d72e2840ba3895c4bfc9120daf8858f48741fb5dc64e4ce +tiger/111 e30fec5c5b5e862b7b085fa7f71c3e6d687fac374149beb7 +tiger/112 0e8f76b3168b6ce849055b405685334c75c6e10fa0ee3727 +tiger/127 01287abaaa42f9150c6fa88dc921fb061a564b6371f67dba +tiger/128 4a4c0ee2b099109011e95568165ab9216fd2987c0f852a38 +tiger/129 628843804c1769616bc717659fdcf6d21a73f0e6a0ea1eca +tiger/135 ad945cec97c5ec4565d2f5ebe5d28c297cadc3f90b072526 +tiger/136 322c7537dac6383fe8b55d4fcdd4888574601cd5123b511f +tiger/255 2bdd7d685a587f31bf202cfd2e1d90e5d53d043b66e773f2 +tiger/256 d8adfc2e0f57a67f8088edb54e2b1f8a93783ffc5aa17833 +tiger/1000 79b772b89f0536af5cf5f34b26ce13ced821d05b468dcdce +whirlpool/0 19fa61d75522a4669b44e39c1d2e1726c530232130d407f89afee0964997f7a73e83be698b288febcf88e3e03c4f0757ea8964e59b63d93708b138cc42a66eb3 +whirlpool/1 4d9444c212955963d425a410176fccfb74161e6839692b4c11fde2ed6eb559efe0560c39a7b61d5a8bcabd6817a3135af80f342a4942ccaae745abddfb6afed0 +whirlpool/3 a83205596a19327369464c5a98896a274ebea82da9ff93e69ddde9eb7ee90757a2173fd3bf10f6d2b7c241acd0ac23f5f1b414b51d33b26f75e63831f303324f +whirlpool/55 ad06ca5facbc8457391c279446818876cc2302eaddd9d39576aa39dff825fa9d0fe455bb245c260b7b7672193607cefbefb41a18706fd2fac2a7e1a2fb8f0cfa +whirlpool/56 6b9e3c1a6714ba482d06f5da0f11c85bb7436f7aebc334e3aeb36745b4cb5d561b27fab0bd59979ede3cca617610c9dccbd0a41b1779e2875ca75761c68e43c0 +whirlpool/63 67da72d6fa8713cba84cca7be5ac53f8ab9d3551a934c7f3376ecebd468d944359772aa83005d815ff74bc26dfe6f22ebbb299b13fbe8f899961d471c76ce871 +whirlpool/64 d893e82c6bf61b2423e8c09e0308d7776b1b1d9556b629cac8c4b82508d6b11891cd2a31312b5d01b640a70acbf94c44c546cdb17389b006943bce62a1e9bd83 +whirlpool/65 6d4593ad291898521736fc63d758a5d5789d9fa3b521cd188f191cd5899b14669006d1d8e6611baa5f782e65aa3d46c47b734b6feca0041a3ada84741e4a57e8 +whirlpool/111 87c6b4fc91580b9ad15689f0e7a9c5378b13646ede124af2cb3cb8124a74e64946214720669f0e2fbd1f23b62980f8d3f19ecda1d26723f7a12f4bef2c6082c4 +whirlpool/112 94826949d4fda3d38d1627bc667261460c8b46e7a11fd24de6116643059e6e02d8271af809e4a4535fd4de8af11c13efad4cea8afe1c348deb10b7c362fd5f23 +whirlpool/127 008f8b8b3dffba832750f542b2a6a3f8ff81547d48715851116c20e475f93893fc7e4ed7a2feaf7595e26b30fde6805335a326b70591ca6d913b66961ea8c5a5 +whirlpool/128 c6dc5ef8e9f449b75b35875c043def87c37ff7f8078044f6cd020fcd311fa5655ba433ef9fd4b6007a0f310f88f63aaba0649175ceb851b0537856c1adf43fa6 +whirlpool/129 2f53c6ddab30118a451bd8dee8e12ee7443e8f3a4132453cc5d74948c008048b5b99ce51f2795e158d0758ccaa6db80f6f1d876533336f7d41f7a098512eb2ca +whirlpool/135 5835a5cfd59d0f75b0af7ee2b667adcf66168dd6d3f710d895211802e8798fa80a6bcaf05f63473245227332d3983fd6580bc90afd56caebe7470ed95c37b081 +whirlpool/136 0356408b498a01e39ea4966dffc809df366747b79fce4c978bd4cc59c7d6765910b88cfea9ac63b1fc1720bee5c45efb51b22cf7047203f0840076d3c2eb2b24 +whirlpool/255 23a0a3cea1f2d971c467993b40582618a8b75acbb6c3bd7fd76e93ae8d0cd8a6797838daffb13cb21ebb07f8ed3729c77abf83ad4c17c2a5866b46bfd327e1f6 +whirlpool/256 e13d036a12baf02ddcb79354e0e4b6e808eed3102864591d57250fbdfc7822208771af0d0a99b65d355c1c05cd09f57874c14bf1731ecaa24350596d1db0b7ad +whirlpool/1000 10d2452c3f4d5c4e124cc6e4542c258d7c122da116650fdd894a599df8bd9673b801301794560c2d2b186d131e46a0e4ddf765590b33a1638351727fa9104f27 +chacha20/0 - +salsa20/0 - +poly1305/0 d7e4f1fe0b1825323f4c596673808d9a +chacha20/1 06 +salsa20/1 9e +poly1305/1 e1eb05201a506d8741b8d5ef79203e58 +chacha20/3 064429 +salsa20/3 9ee7f7 +poly1305/3 453fee0552873db241b6aa62a2c52813 +chacha20/55 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dd +salsa20/55 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b +poly1305/55 269c983cc75a974291409f4ae5d403cc +chacha20/56 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dddc +salsa20/56 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b02 +poly1305/56 02b5d7816f3c5bb108e08629f814cce3 +chacha20/63 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dddc8a54dbf245d2bc +salsa20/63 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b029db7b8819cafae +poly1305/63 9f0dcc8932e4f6637ee198bf2c8000d6 +chacha20/64 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dddc8a54dbf245d2bcff +salsa20/64 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b029db7b8819cafae17 +poly1305/64 6da04a74098654ca6392a1c62040b4cf +chacha20/65 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dddc8a54dbf245d2bcff +salsa20/65 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b029db7b8819cafae17 +poly1305/65 db28069b3ca255821238da8b086fe27a +chacha20/111 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dddc8a54dbf245d2bcff +salsa20/111 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b029db7b8819cafae17 +poly1305/111 f10eb1e2364e6ed3ccc9326d9e56ed64 +chacha20/112 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dddc8a54dbf245d2bcff +salsa20/112 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b029db7b8819cafae17 +poly1305/112 183852995a777f1af0f29760c17fa63a +chacha20/127 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dddc8a54dbf245d2bcff +salsa20/127 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b029db7b8819cafae17 +poly1305/127 234fcf18eb82f1b2821f113a22fab7f9 +chacha20/128 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dddc8a54dbf245d2bcff +salsa20/128 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b029db7b8819cafae17 +poly1305/128 3f8f2a778d4f490bb17809b1dcdf50e5 +chacha20/129 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dddc8a54dbf245d2bcff +salsa20/129 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b029db7b8819cafae17 +poly1305/129 e3fe11911fea16b2b3bf2be93d22c7d7 +chacha20/135 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dddc8a54dbf245d2bcff +salsa20/135 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b029db7b8819cafae17 +poly1305/135 4520d5997340683ccd21a2804631d700 +chacha20/136 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dddc8a54dbf245d2bcff +salsa20/136 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b029db7b8819cafae17 +poly1305/136 848568032e1e248045e53031e8219393 +chacha20/255 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dddc8a54dbf245d2bcff +salsa20/255 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b029db7b8819cafae17 +poly1305/255 8ae46792e840f083d98f7e78f34a8f9d +chacha20/256 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dddc8a54dbf245d2bcff +salsa20/256 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b029db7b8819cafae17 +poly1305/256 427f7cd821633cc0993956cf3a7cf26c +chacha20/1000 064429ccbd6ef85054af07f8dee3a283e75e844df9a59b1b76addd7d41fb554946f5dd12b49b97f4a72f05b0b41c557e8c69da034ae9dddc8a54dbf245d2bcff +salsa20/1000 9ee7f7b5774db507d8e78fe7398594de9b40444a5301c14dfbc675e6310f9aa659e9cca9892563df9386a60607a2e18781d35d4897ea2b029db7b8819cafae17 +poly1305/1000 c403a63071ea2f2e732472db8e7b79c0 +aes128-ecb/1 51d0767be0142b54f38422f098a979c1 +aes128-ecbd/1 0772f11dd1e53677251c6d2b42c12a84 +aes128-cbc/1 53423fee1b0b9dda9006158bb0e3c6e6 +aes128-cbcd/1 0060ec35e2db7f237a7618abc9578b28 +aes128-xts/1 2634e65251d6fed878a1b3546061cd8f +aes128-xtsd/1 9d22db2b665e917344f60d1561481854 +aes128-xts-sp3/1 1d40e186532c4338b08ebb40039c49b6 +aes128-ecb/2 51d0767be0142b54f38422f098a979c1f5e0ee3fe17beef5b33b72d8f60ec17c +aes128-ecbd/2 0772f11dd1e53677251c6d2b42c12a843d8abf116e0bd219a05d2c1d89e8faee +aes128-cbc/2 53423fee1b0b9dda9006158bb0e3c6e6b1acc41a9ff16d75c2170e3242346760 +aes128-cbcd/2 0060ec35e2db7f237a7618abc9578b283d8db1047228f82898626a50ddb39887 +aes128-xts/2 2634e65251d6fed878a1b3546061cd8ff2e82679d8d7ade78233b89a76958855 +aes128-xtsd/2 9d22db2b665e917344f60d1561481854c8d8c3fe1890b52db7ca720cd253192d +aes128-xts-sp3/2 1d40e186532c4338b08ebb40039c49b6cc2b2b42b37e9412005737634804ad97 +aes128-ecb/4 51d0767be0142b54f38422f098a979c1f5e0ee3fe17beef5b33b72d8f60ec17c527ee4edb0bd6e9346d1148c3763613b5112f9a67b30c06fd511e725279957b9 +aes128-ecbd/4 0772f11dd1e53677251c6d2b42c12a843d8abf116e0bd219a05d2c1d89e8faeed21af82debb90e26d30fd338766a0ff47234606001901e7888e61a64468495be +aes128-cbc/4 53423fee1b0b9dda9006158bb0e3c6e6b1acc41a9ff16d75c2170e324234676075d60f392793fd0623859fffb40b23585955ff09a2c20c02b9c2baf084d6943b +aes128-cbcd/4 0060ec35e2db7f237a7618abc9578b283d8db1047228f82898626a50ddb39887a26d86a8672a94877ba06585b2a1dd2d8d326d741ab23748bfd85f2815def4d6 +aes128-xts/4 2634e65251d6fed878a1b3546061cd8ff2e82679d8d7ade78233b89a76958855da809daab8adf228232779804848093c5aa478a2d7e0c11b58dfbfd19b94ea36 +aes128-xtsd/4 9d22db2b665e917344f60d1561481854c8d8c3fe1890b52db7ca720cd253192dedd68b0854e3340afb2d8a34942bcd5479bed9f04ee52769479c2192544eaa2f +aes128-xts-sp3/4 1d40e186532c4338b08ebb40039c49b6cc2b2b42b37e9412005737634804ad970343c410aa37b63b6595f3e6e8bbcb8bc8581ec02491cb1e3df920165a8d05c7 +aes128-ecb/7 51d0767be0142b54f38422f098a979c1f5e0ee3fe17beef5b33b72d8f60ec17c527ee4edb0bd6e9346d1148c3763613b5112f9a67b30c06fd511e725279957b977a15ad7bbc6a54c141a2d07c4b233000adedca5ef085da03709e978ed1aca3500e2bd1f0ad7a29ade429446d2ddfb93 +aes128-ecbd/7 0772f11dd1e53677251c6d2b42c12a843d8abf116e0bd219a05d2c1d89e8faeed21af82debb90e26d30fd338766a0ff47234606001901e7888e61a64468495be2de8cb16ce03066f65758235de9386de764ddfa85c70fae9b4c80eed9b53a216d5c34dcea96fa27e88fe002306a3548d +aes128-cbc/7 53423fee1b0b9dda9006158bb0e3c6e6b1acc41a9ff16d75c2170e324234676075d60f392793fd0623859fffb40b23585955ff09a2c20c02b9c2baf084d6943b84e8cef5ab838dc0864175cab14a1026f6230f8df62be8d6794dfb9aba0622e62e6a358aab88832f2b18d4819eca4f91 +aes128-cbcd/7 0060ec35e2db7f237a7618abc9578b283d8db1047228f82898626a50ddb39887a26d86a8672a94877ba06585b2a1dd2d8d326d741ab23748bfd85f2815def4d6429eb69245919fcfc2db37891d5957068848d3bb4651d2c682f54aa6c90ac271bbb6314d23fe3ae12e53b498c46a845a +aes128-xts/7 2634e65251d6fed878a1b3546061cd8ff2e82679d8d7ade78233b89a76958855da809daab8adf228232779804848093c5aa478a2d7e0c11b58dfbfd19b94ea3651d66c7adda6172d3e8672cc65cce7dbd8d9d6f32bf8f087867527f0a8128fdd80ed3a97398f66577f2041342c585e72 +aes128-xtsd/7 9d22db2b665e917344f60d1561481854c8d8c3fe1890b52db7ca720cd253192dedd68b0854e3340afb2d8a34942bcd5479bed9f04ee52769479c2192544eaa2fa426c177c9b397f0bb380e3991c69479759294d63723eeb04aae18e48f84fa213dcf254eb1d532252c6328a8fdd8976f +aes128-xts-sp3/7 1d40e186532c4338b08ebb40039c49b6cc2b2b42b37e9412005737634804ad970343c410aa37b63b6595f3e6e8bbcb8bc8581ec02491cb1e3df920165a8d05c7fa1c26910201c2ac91dab7ffcb1995c57cfc19a1a8ffbb3e11debe1e9de63147bd99f1f0f9af66d827b18a427ed02d32 +aes128-ctr/0 - +aes128-gcm/0 91527d0ffa250047dc10c74aaf7dc630 +aes128-ctr/1 f3 +aes128-gcm/1 110be12ced502543dc2b098ce214c704a9 +aes128-ctr/15 f3e218ff08f7128f6c9e088df658bf +aes128-gcm/15 11e696e7a098883e54b19200262e669d12ac829a6a3eed6933b267f13690e7 +aes128-ctr/16 f3e218ff08f7128f6c9e088df658bfcc +aes128-gcm/16 11e696e7a098883e54b19200262e6622968a6e12ea8c1d162967acbccc339980 +aes128-ctr/17 f3e218ff08f7128f6c9e088df658bfccea +aes128-gcm/17 11e696e7a098883e54b19200262e6622dd540e6e9683dd7026b3b24579826c0593 +aes128-ctr/64 f3e218ff08f7128f6c9e088df658bfccea245fbfae1cb12ed69d5f01a5a6d7f161f8ae682ba4555a088d89fe91fdfb2890935fe9d0ae19f0ac4f73e19d26d1cc +aes128-gcm/64 11e696e7a098883e54b19200262e6622dd7f1ac8145981a493fdaea4641bf49da46e0d311c3daf1665fd5fdf025965d9312f324d4c646d3f3f85ab13d236716fdb10935bd1c334f13b56a98aa09ef476 +aes128-ctr/100 f3e218ff08f7128f6c9e088df658bfccea245fbfae1cb12ed69d5f01a5a6d7f161f8ae682ba4555a088d89fe91fdfb2890935fe9d0ae19f0ac4f73e19d26d1cc8c30a3d0749027d7cff23bb8dbccb1a8540feb89f5ca9447a5cdef585d59125c58b73b22 +aes128-gcm/100 11e696e7a098883e54b19200262e6622dd7f1ac8145981a493fdaea4641bf49da46e0d311c3daf1665fd5fdf025965d9312f324d4c646d3f3f85ab13d236716fc40f23dfc154e04b18dfc89ae50a386551eab4dd02434c526f9fc45a278b21a485347381adbe43ff0386975d584a98adacbde34a +aes192-ecb/1 b7bd8a4e647f413adc38c4464e4434b6 +aes192-ecbd/1 4308adb49827c2741aabb7c40c5b63a5 +aes192-cbc/1 9fdd2c700fd7e916fd89805f18439a27 +aes192-cbcd/1 441ab09cab198b2045c1c24487cdc209 +aes192-xts/1 c2eea1b9f5f0fd170168a3e2af1d059a +aes192-xtsd/1 5a3a19588f9700d5482eae9dd316051d +aes192-xts-sp3/1 f1dba9fc3e3d43d3a46608690e98b0e8 +aes192-ecb/2 b7bd8a4e647f413adc38c4464e4434b67d307330822566b908545f9e8de18c1a +aes192-ecbd/2 4308adb49827c2741aabb7c40c5b63a5e6015605e3e6c8df51f2de769db588e2 +aes192-cbc/2 9fdd2c700fd7e916fd89805f18439a2754e55a17706f184fe6b2185ef87eb075 +aes192-cbcd/2 441ab09cab198b2045c1c24487cdc209e6065810ffc5e2ee69cd983bc9eeea8b +aes192-xts/2 c2eea1b9f5f0fd170168a3e2af1d059af4c8308582684da17d5b6d631459d6d5 +aes192-xtsd/2 5a3a19588f9700d5482eae9dd316051dab8c3b6155002903c90664413bd96653 +aes192-xts-sp3/2 f1dba9fc3e3d43d3a46608690e98b0e823173a551464b1a3ef6095560902a48c +aes192-ecb/4 b7bd8a4e647f413adc38c4464e4434b67d307330822566b908545f9e8de18c1ac8d6dd6f7bee6e99e4682aef343f7cd0ea985b24e2368b8d1acd3b0bf164a30b +aes192-ecbd/4 4308adb49827c2741aabb7c40c5b63a5e6015605e3e6c8df51f2de769db588e2f4a1ce5088ed9eee6f721f73986dcf23ffc534ac5d16cb7010015dc1b50330ce +aes192-cbc/4 9fdd2c700fd7e916fd89805f18439a2754e55a17706f184fe6b2185ef87eb075061361ddd895a18991e3068ef563dcea087fc1517810a4ac222a391215026095 +aes192-cbcd/4 441ab09cab198b2045c1c24487cdc209e6065810ffc5e2ee69cd983bc9eeea8b84d6b0d5047e044fc7dda9ce5ca61dfa00c339b84634e240273f188de65951a6 +aes192-xts/4 c2eea1b9f5f0fd170168a3e2af1d059af4c8308582684da17d5b6d631459d6d52e43efd39bfc3ebbbbf189a4bfb1071bbc3851a67841dc4206c4f0f86fca0fed +aes192-xtsd/4 5a3a19588f9700d5482eae9dd316051dab8c3b6155002903c90664413bd9665351b41f0a6b8b2afd38a19274800e86739bb50da065cc97f9818895e062c6be6c +aes192-xts-sp3/4 f1dba9fc3e3d43d3a46608690e98b0e823173a551464b1a3ef6095560902a48cdc267dd87440e9a910bedc98758b512799ad24644966748404717b701ad70804 +aes192-ecb/7 b7bd8a4e647f413adc38c4464e4434b67d307330822566b908545f9e8de18c1ac8d6dd6f7bee6e99e4682aef343f7cd0ea985b24e2368b8d1acd3b0bf164a30b225dbf0a85eb782bbaf30c02286d91d271bbfb08e60fc12977e7eea142abc72bed4ade399f048bfa5d5a48f02ac47e22 +aes192-ecbd/7 4308adb49827c2741aabb7c40c5b63a5e6015605e3e6c8df51f2de769db588e2f4a1ce5088ed9eee6f721f73986dcf23ffc534ac5d16cb7010015dc1b50330ce1b0b2582b28d8f6174ee21d4cefdc31632f8ca413704961b7da50fdc2de61e730ab37b8b3148660f36335f6d4a01d85a +aes192-cbc/7 9fdd2c700fd7e916fd89805f18439a2754e55a17706f184fe6b2185ef87eb075061361ddd895a18991e3068ef563dcea087fc1517810a4ac222a391215026095582b2b9c34d2b9b40ef88f9ec4459fc671f5ecb0d23defc6ea99f6075174381b861dec529ac5a49c9c55efae85aca707 +aes192-cbcd/7 441ab09cab198b2045c1c24487cdc209e6065810ffc5e2ee69cd983bc9eeea8b84d6b0d5047e044fc7dda9ce5ca61dfa00c339b84634e240273f188de65951a6747d5806391f16c1d34094680d3712ceccfdc6522d25be344b984b977fbf7e1464c60708bbd9fe90909eebd688c8088d +aes192-xts/7 c2eea1b9f5f0fd170168a3e2af1d059af4c8308582684da17d5b6d631459d6d52e43efd39bfc3ebbbbf189a4bfb1071bbc3851a67841dc4206c4f0f86fca0fedf90c2617695920f2f5611aab587198b0263b555f0a93d1580c22822b6fe3451e31e5f67c9526f5128d87bc586e61a255 +aes192-xtsd/7 5a3a19588f9700d5482eae9dd316051dab8c3b6155002903c90664413bd9665351b41f0a6b8b2afd38a19274800e86739bb50da065cc97f9818895e062c6be6cfa00838607d2c4a55642a4973aac6fbdd5e9c514e69e8569d1559acde57fc207a5709b8f0eff9a93ad7dc6c170fda2c0 +aes192-xts-sp3/7 f1dba9fc3e3d43d3a46608690e98b0e823173a551464b1a3ef6095560902a48cdc267dd87440e9a910bedc98758b512799ad24644966748404717b701ad70804506ba3f939ed54c43dc721d3532f83cf267aa4c32d9543edb71de9548dacd5171e76bef732fdad2c49ebb781d639a445 +aes192-ctr/0 - +aes192-gcm/0 b03daf3f9346095b3083d09391d47cee +aes192-ctr/1 72 +aes192-gcm/1 bdc01f6ed1a9e3dc9be2c4d9e8d8d3b584 +aes192-ctr/15 72138e46596ca4b6de00c2aeadc2c1 +aes192-gcm/15 bd19761039873f6946e29ba643ee41c6ca3791dc7497012fce9f48631f599c +aes192-ctr/16 72138e46596ca4b6de00c2aeadc2c17c +aes192-gcm/16 bd19761039873f6946e29ba643ee4175c85f71850653f05f2410354e2e93cdc9 +aes192-ctr/17 72138e46596ca4b6de00c2aeadc2c17c02 +aes192-gcm/17 bd19761039873f6946e29ba643ee4175b3e87ccbca9af385c12d31382f6e721151 +aes192-ctr/64 72138e46596ca4b6de00c2aeadc2c17c02df852e3c7abed9e3cd5a94d24c71074945c7835771862ec94dbc9c8ba47ef305131b7f3933a150025d8a796c5889b2 +aes192-gcm/64 bd19761039873f6946e29ba643ee4175b3c8f8e942762e364f67333b85f6c220999554b2d6e78a47fc789581a39e89a4b6a5539491894c85fbba17f03e183e3b27736f65aaba8f86ba884618e2c75b83 +aes192-ctr/100 72138e46596ca4b6de00c2aeadc2c17c02df852e3c7abed9e3cd5a94d24c71074945c7835771862ec94dbc9c8ba47ef305131b7f3933a150025d8a796c5889b2aea7d364add8934a993f8532bae1b59ec0c4d75bf59dca8e265de412ed73052188706192 +aes192-gcm/100 bd19761039873f6946e29ba643ee4175b3c8f8e942762e364f67333b85f6c220999554b2d6e78a47fc789581a39e89a4b6a5539491894c85fbba17f03e183e3bb42b4d0d362fc35b7719b76c299e857e82b8b127e9cbc732559c26b6338ef92f153271feb0462a5c14552476a3a41aae5c275596 +aes256-ecb/1 791925c38818304f047d756d78ef8f13 +aes256-ecbd/1 831e330b93ed108f70c15c463224b5c6 +aes256-cbc/1 6b1dfef4ab6d3fd0a32a3c3a7c9603d5 +aes256-cbcd/1 840c2e23a0d359db2fab29c6b9b2146a +aes256-xts/1 b90a69094799c4e679fc829084d41055 +aes256-xtsd/1 18db4b08618f8573974e5c85d0954302 +aes256-xts-sp3/1 d5f9ef046dd7e688938aab18d3d68ce5 +aes256-ecb/2 791925c38818304f047d756d78ef8f139da36087c3b26165970cf0987eee7d70 +aes256-ecbd/2 831e330b93ed108f70c15c463224b5c6f2f7d848bb39a82dd65b4db54a6f54ff +aes256-cbc/2 6b1dfef4ab6d3fd0a32a3c3a7c9603d5090fa2b707cd96fea1132c143d5683fa +aes256-cbcd/2 840c2e23a0d359db2fab29c6b9b2146af2f0d65da71a821cee640bf81e343696 +aes256-xts/2 b90a69094799c4e679fc829084d4105597001399d1f01a458a64913345271c27 +aes256-xtsd/2 18db4b08618f8573974e5c85d0954302511a3cdde7128329882a30bd28169980 +aes256-xts-sp3/2 d5f9ef046dd7e688938aab18d3d68ce5724313716ca706a5b7a914024b26cfc1 +aes256-ecb/4 791925c38818304f047d756d78ef8f139da36087c3b26165970cf0987eee7d70aff88ab2d21b91927e27057d2f91f923cd1e35c75f2b3111adc0fd520ae368b4 +aes256-ecbd/4 831e330b93ed108f70c15c463224b5c6f2f7d848bb39a82dd65b4db54a6f54ff6c4c59c5bd048efc1c7cf8353e29eae59fd397d041edd80e0c4a41824a3b7bc1 +aes256-cbc/4 6b1dfef4ab6d3fd0a32a3c3a7c9603d5090fa2b707cd96fea1132c143d5683fae5e8441b18737ef3581adef0e182aea3b94a6e5e4925464a95ddb6b409f19e88 +aes256-cbcd/4 840c2e23a0d359db2fab29c6b9b2146af2f0d65da71a821cee640bf81e3436961c3b27403197145db4d34e88fae2383c60d59ac45acff13e3b7404ce19611aa9 +aes256-xts/4 b90a69094799c4e679fc829084d4105597001399d1f01a458a64913345271c27261700fa455603c6d5370d3867bf3213717c7241850cc16b414cf4598472f9e7 +aes256-xtsd/4 18db4b08618f8573974e5c85d0954302511a3cdde7128329882a30bd28169980f52cc7d5d434263c1c22188e735a49d6130e0683a55e636739ba7abb743e7b09 +aes256-xts-sp3/4 d5f9ef046dd7e688938aab18d3d68ce5724313716ca706a5b7a914024b26cfc1b47c048f8548cc7a80a22a9dcaf25065efd935f4730020a085cfd489b8a98cab +aes256-ecb/7 791925c38818304f047d756d78ef8f139da36087c3b26165970cf0987eee7d70aff88ab2d21b91927e27057d2f91f923cd1e35c75f2b3111adc0fd520ae368b4b6d87226c642479229919a9c47bfca56ca5607a049cd7f423fb0556511f5db4bb8f7165904e83e4da028f316252e9f5d +aes256-ecbd/7 831e330b93ed108f70c15c463224b5c6f2f7d848bb39a82dd65b4db54a6f54ff6c4c59c5bd048efc1c7cf8353e29eae59fd397d041edd80e0c4a41824a3b7bc1eaa8daa1a6f40e115cfd93684f1b7ad8b66b09a303b2488f93eaac36762e571d4f00c320962c9dc1607e8d884579f306 +aes256-cbc/7 6b1dfef4ab6d3fd0a32a3c3a7c9603d5090fa2b707cd96fea1132c143d5683fae5e8441b18737ef3581adef0e182aea3b94a6e5e4925464a95ddb6b409f19e88f104a783b152e599e90b85fc60f57aef7ee95c1608290490adf2688b4683b2d70a6b16291cb5a49008dd74dbf8c16191 +aes256-cbcd/7 840c2e23a0d359db2fab29c6b9b2146af2f0d65da71a821cee640bf81e3436961c3b27403197145db4d34e88fae2383c60d59ac45acff13e3b7404ce19611aa985dea7252d6697b1fb5326d48cd1ab00486e05b0199360a0a5d7e87d2477377a2175bfa31cbd055ec6d3393387b023d1 +aes256-xts/7 b90a69094799c4e679fc829084d4105597001399d1f01a458a64913345271c27261700fa455603c6d5370d3867bf3213717c7241850cc16b414cf4598472f9e77778636d8d94ec90c8bc220a7075ab56b4758635686c3985c243224397844894defd980efc5ce05ed451cec1af9583e9 +aes256-xtsd/7 18db4b08618f8573974e5c85d0954302511a3cdde7128329882a30bd28169980f52cc7d5d434263c1c22188e735a49d6130e0683a55e636739ba7abb743e7b09cdc794b975bdb545b6a5a69fde322ceb0f3da5d8a1dd3103dee190f9fb65d8a0e86ddffed9358c0872862ae921e7ba75 +aes256-xts-sp3/7 d5f9ef046dd7e688938aab18d3d68ce5724313716ca706a5b7a914024b26cfc1b47c048f8548cc7a80a22a9dcaf25065efd935f4730020a085cfd489b8a98cab5e7eb266eb52f17b6f78bcb0996b96fc7ca3a671856b32fe2e8c9341c50a70c2d40906b8d450fed98e6a5f6a4d5f84ab +aes256-ctr/0 - +aes256-gcm/0 8316aabedf3935666f9d8b2dd6b5b356 +aes256-ctr/1 21 +aes256-gcm/1 7b8fcd0cf59d4bccd66dd123de4c1c7aa4 +aes256-ctr/15 219430a80fe1a08ed512ff1c9cee61 +aes256-gcm/15 7bfc609c1f245b464a8e0763c5e8b7a1dba82dac1ce33ecb0350b7d9e582df +aes256-ctr/16 219430a80fe1a08ed512ff1c9cee61dc +aes256-gcm/16 7bfc609c1f245b464a8e0763c5e8b7f9ae001617c759469bd602cd92dd4bf7e9 +aes256-ctr/17 219430a80fe1a08ed512ff1c9cee61dc52 +aes256-gcm/17 7bfc609c1f245b464a8e0763c5e8b7f96b70cf1b59a4a56949e6437f0b9b7b257b +aes256-ctr/64 219430a80fe1a08ed512ff1c9cee61dc52867258efbdf3943f48d30ef7da12c7bfb56c293beeddc92b1c270b7110c0e2db805ddad464f0eac9d46e7109001374 +aes256-gcm/64 7bfc609c1f245b464a8e0763c5e8b7f96bc3d807f39fd255cbf14006baf6bbccb5d2509a2b3cba5a4c7ee680ee09979e810f94882ff5c0b6e572d397aef389ec72e6a3f5eb4be76004c10d8c5881492a +aes256-ctr/100 219430a80fe1a08ed512ff1c9cee61dc52867258efbdf3943f48d30ef7da12c7bfb56c293beeddc92b1c270b7110c0e2db805ddad464f0eac9d46e71090013745c52cea313ba7efea2eac9da963dcf1bc484a1c1924fd8f91fd8bf774dc77b472356c522 +aes256-gcm/100 7bfc609c1f245b464a8e0763c5e8b7f96bc3d807f39fd255cbf14006baf6bbccb5d2509a2b3cba5a4c7ee680ee09979e810f94882ff5c0b6e572d397aef389ecb7bd0a0a1f4d638d3e2557ca92beb2bf4228499ac6cc38d20e0539cd4658ed2a7d24f4ab32d4b6ea69814b4c2718fd1dcdb7e16a +ghash-mul c65b128f165974f311df03df3486ce71 +ghash-mul4 6bfd46cd3d314cac2d75ead0e96092c7 +polyval bbc31d0761373f0b2737904b9a76fb27 diff --git a/cbits/tests/fuzz/README b/cbits/tests/fuzz/README new file mode 100644 --- /dev/null +++ b/cbits/tests/fuzz/README @@ -0,0 +1,35 @@ +Generated bytes fed to the code that parses what comes off the wire. + +Rounds one to ten asked prepared questions. This one asks whether anything +breaks on input nobody chose. What is looked for is memory safety rather +than wrong answers: these inputs are meant to be rejected, and the question is +how they are rejected. + +The harnesses are the places where bytes from elsewhere are taken apart: + + ed25519 a public key and a signature, both the sender's to choose + decaf point decoding, EdDSA public key decoding, Ed448 verification + p256 a point checked for being on the curve, then multiplied + aead AES-GCM decryption, with the iv, aad and tag lengths from the wire + canary the calibration + +fuzz_canary reads one byte past a buffer when the input opens with four +particular bytes. Four is the point: one chance in 2^32 puts it out of reach +of throwing random input at the harness, and well within reach of a fuzzer +that watches which comparisons it got past. So a campaign that does not +report the canary is not exploring, and the silence of the others would mean +nothing. Round five believed a ThreadSanitizer zero that meant nothing and +round ten twice believed a scrubbing zero that meant nothing; this is cheaper +than learning it a third time. + +The corpus is seeded with inputs each harness accepts -- a signature that +verifies, the same with one bit of the message moved, a point that is on the +curve, a ciphertext that authenticates -- so that a campaign starts from the +far side of the checks rather than in front of them. They were generated by +running the primitives, and checked: the valid signature verifies, the +tampered one does not, the point is on the curve. + +Where clang has no libFuzzer runtime -- Apple's does not -- run.sh replays the +corpus and a deterministic stream through standalone.c instead. That says the +harnesses build and run clean under the sanitizers. It explores nothing, and +it says so rather than reporting a pass. diff --git a/cbits/tests/fuzz/corpus/aead-authentic.bin b/cbits/tests/fuzz/corpus/aead-authentic.bin new file mode 100644 Binary files /dev/null and b/cbits/tests/fuzz/corpus/aead-authentic.bin differ diff --git a/cbits/tests/fuzz/corpus/ed25519-tampered.bin b/cbits/tests/fuzz/corpus/ed25519-tampered.bin new file mode 100644 Binary files /dev/null and b/cbits/tests/fuzz/corpus/ed25519-tampered.bin differ diff --git a/cbits/tests/fuzz/corpus/ed25519-valid.bin b/cbits/tests/fuzz/corpus/ed25519-valid.bin new file mode 100644 Binary files /dev/null and b/cbits/tests/fuzz/corpus/ed25519-valid.bin differ diff --git a/cbits/tests/fuzz/corpus/p256-on-curve.bin b/cbits/tests/fuzz/corpus/p256-on-curve.bin new file mode 100644 Binary files /dev/null and b/cbits/tests/fuzz/corpus/p256-on-curve.bin differ diff --git a/cbits/tests/fuzz/fuzz.h b/cbits/tests/fuzz/fuzz.h new file mode 100644 --- /dev/null +++ b/cbits/tests/fuzz/fuzz.h @@ -0,0 +1,34 @@ +/* Feeding generated bytes to the code that parses what comes off the wire. + * + * Each harness is an LLVMFuzzerTestOneInput, so a real fuzzer drives it. + * Where there is no fuzzer -- Apple's clang ships no libFuzzer runtime -- + * standalone.c supplies a main that replays the committed corpus and then a + * deterministic stream of its own, which says the harness is wired up + * correctly and nothing about coverage. + * + * What is being looked for is memory safety, not wrong answers: these inputs + * are meant to be rejected, and the question is how they are rejected. + */ +#ifndef CRYPTON_TESTS_FUZZ_H +#define CRYPTON_TESTS_FUZZ_H + +#include +#include +#include + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size); + +/* Carve a fixed-width field out of the input. Short input means the harness + * returns rather than reading past the end, which is the harness's own bug to + * avoid and not the library's to report. */ +static int fz_take(const uint8_t **p, size_t *left, void *out, size_t n) +{ + if (*left < n) + return 0; + memcpy(out, *p, n); + *p += n; + *left -= n; + return 1; +} + +#endif diff --git a/cbits/tests/fuzz/fuzz_aead.c b/cbits/tests/fuzz/fuzz_aead.c new file mode 100644 --- /dev/null +++ b/cbits/tests/fuzz/fuzz_aead.c @@ -0,0 +1,60 @@ +/* AES-GCM decryption, where the lengths and the tag are the sender's to + * choose. The key is not attacker-controlled and is fixed here; what varies + * is everything that arrives with the message. */ +#include "tests/fuzz/fuzz.h" +#include +#include "crypton_aes.h" + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) +{ + static const uint8_t key[16] = { + 0x9e, 0x37, 0x79, 0xb9, 0x7f, 0x4a, 0x7c, 0x15, + 0xf3, 0x9c, 0xc0, 0x60, 0x5c, 0xed, 0xc8, 0x34, + }; + aes_key k; + aes_gcm_key gk; + uint8_t ivlen, aadlen, taglen; + const uint8_t *p = data; + size_t left = size; + uint8_t *out; + + if (!fz_take(&p, &left, &ivlen, 1)) + return 0; + if (!fz_take(&p, &left, &aadlen, 1)) + return 0; + if (!fz_take(&p, &left, &taglen, 1)) + return 0; + + /* the three lengths are the sender's, so they are taken as they come, + * short of asking for more bytes than arrived */ + if (left < (size_t)ivlen + aadlen) + return 0; + taglen = (uint8_t)(taglen % 17); /* 0..16, as the API allows */ + + { + const uint8_t *iv = p; + const uint8_t *aad = p + ivlen; + const uint8_t *ct = p + ivlen + aadlen; + size_t rest = left - ivlen - aadlen; + const uint8_t *tag; + size_t ctlen; + + /* the tag arrives with the message, so it comes off the end */ + if (rest < taglen) + return 0; + ctlen = rest - taglen; + tag = ct + ctlen; + + out = (uint8_t *)malloc(ctlen + 1); + if (!out) + return 0; + crypton_aes_initkey(&k, (uint8_t *)key, sizeof key); + crypton_aes_gcm_key_init(&gk, &k); + (void)crypton_aes_gcm_full_decrypt(out, &gk, &k, (uint8_t *)iv, ivlen, + (uint8_t *)aad, aadlen, + (uint8_t *)ct, (uint32_t)ctlen, + tag, taglen); + free(out); + } + return 0; +} diff --git a/cbits/tests/fuzz/fuzz_canary.c b/cbits/tests/fuzz/fuzz_canary.c new file mode 100644 --- /dev/null +++ b/cbits/tests/fuzz/fuzz_canary.c @@ -0,0 +1,41 @@ +/* The calibration. This harness reads one byte past a buffer when the input + * opens with a four-byte marker. Four bytes is the point: one chance in + * 2^32 puts it out of reach of throwing random input at the harness, while a + * fuzzer that watches which comparisons it got past finds it in seconds. So + * a campaign that does not report this one is not fuzzing, and the silence of + * the harnesses beside it means nothing. + * + * Round five believed a ThreadSanitizer zero that meant nothing, and round + * ten twice believed a scrubbing zero that meant nothing. This is cheaper + * than learning it a third time. + * + * Replaying the corpus is not expected to reach it, and run.sh says so. + */ +#include "tests/fuzz/fuzz.h" +#include + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) +{ + uint8_t *buf; + int r; + + if (size < 5) + return 0; + if (data[0] != 0xde || data[1] != 0xad) + return 0; + if (data[2] != 0xbe || data[3] != 0xef) + return 0; + + /* On the heap, not the stack. A compiler that can see the size of a + * local can also see that reading past it is undefined and remove the + * read, which is what the first attempt at this did: the campaign found + * nothing because by then there was nothing left to find. It cannot + * reason that way about what malloc returned. */ + buf = (uint8_t *)malloc(16); + if (!buf) + return 0; + memset(buf, 0, 16); + r = buf[16] == data[4] ? 1 : 0; /* deliberately one past the end */ + free(buf); + return r; +} diff --git a/cbits/tests/fuzz/fuzz_decaf.c b/cbits/tests/fuzz/fuzz_decaf.c new file mode 100644 --- /dev/null +++ b/cbits/tests/fuzz/fuzz_decaf.c @@ -0,0 +1,40 @@ +/* Decoding a point and verifying an Ed448 signature, both from bytes that + * came from somewhere else. point_decode validates; the EdDSA decode is the + * first thing a verifier does with a public key. */ +#include "tests/fuzz/fuzz.h" +#include "decaf/point_448.h" +#include "decaf/ed448.h" + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) +{ + uint8_t ser[CRYPTON_DECAF_448_SER_BYTES]; + uint8_t pub[CRYPTON_DECAF_EDDSA_448_PUBLIC_BYTES]; + uint8_t sig[CRYPTON_DECAF_EDDSA_448_SIGNATURE_BYTES]; + crypton_decaf_448_point_t pt; + const uint8_t *p = data; + size_t left = size; + uint8_t selector; + + if (!fz_take(&p, &left, &selector, 1)) + return 0; + + if (selector & 1) { + if (!fz_take(&p, &left, ser, sizeof ser)) + return 0; + (void)crypton_decaf_448_point_decode(pt, ser, selector & 2 ? 1 : 0); + return 0; + } + + if (!fz_take(&p, &left, pub, sizeof pub)) + return 0; + if (selector & 2) { + (void)crypton_decaf_448_point_decode_like_eddsa_and_mul_by_ratio(pt, pub); + return 0; + } + if (!fz_take(&p, &left, sig, sizeof sig)) + return 0; + /* a context of at most 255 bytes, as the API takes a uint8_t length */ + (void)crypton_decaf_ed448_verify(sig, pub, p, left, selector & 4 ? 1 : 0, + NULL, 0); + return 0; +} diff --git a/cbits/tests/fuzz/fuzz_ed25519.c b/cbits/tests/fuzz/fuzz_ed25519.c new file mode 100644 --- /dev/null +++ b/cbits/tests/fuzz/fuzz_ed25519.c @@ -0,0 +1,22 @@ +/* Ed25519 signature verification: the public key and the signature are both + * whatever the sender chose, and both are decoded before anything is + * checked. */ +#include "tests/fuzz/fuzz.h" +#include "ed25519/ed25519.h" + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) +{ + ed25519_public_key pk; + ed25519_signature sig; + const uint8_t *p = data; + size_t left = size; + + if (!fz_take(&p, &left, pk, sizeof pk)) + return 0; + if (!fz_take(&p, &left, sig, sizeof sig)) + return 0; + + /* whatever is left is the message */ + (void)crypton_ed25519_sign_open(p, left, pk, sig); + return 0; +} diff --git a/cbits/tests/fuzz/fuzz_p256.c b/cbits/tests/fuzz/fuzz_p256.c new file mode 100644 --- /dev/null +++ b/cbits/tests/fuzz/fuzz_p256.c @@ -0,0 +1,33 @@ +/* A P-256 point as it arrives: two field elements, checked for being on the + * curve, and then multiplied if they are. */ +#include "tests/fuzz/fuzz.h" +#include "p256/p256.h" + +void crypton_p256e_point_mul(const crypton_p256_int *n, + const crypton_p256_int *ix, const crypton_p256_int *iy, + crypton_p256_int *ox, crypton_p256_int *oy); + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) +{ + uint8_t xb[P256_NBYTES], yb[P256_NBYTES], nb[P256_NBYTES]; + crypton_p256_int x, y, n, ox, oy; + const uint8_t *p = data; + size_t left = size; + + if (!fz_take(&p, &left, xb, sizeof xb)) + return 0; + if (!fz_take(&p, &left, yb, sizeof yb)) + return 0; + if (!fz_take(&p, &left, nb, sizeof nb)) + return 0; + + crypton_p256_from_bin(xb, &x); + crypton_p256_from_bin(yb, &y); + crypton_p256_from_bin(nb, &n); + + if (crypton_p256_is_valid_point(&x, &y)) { + crypton_p256_mod(&crypton_SECP256r1_n, &n, &n); + crypton_p256e_point_mul(&n, &x, &y, &ox, &oy); + } + return 0; +} diff --git a/cbits/tests/fuzz/run.sh b/cbits/tests/fuzz/run.sh new file mode 100644 --- /dev/null +++ b/cbits/tests/fuzz/run.sh @@ -0,0 +1,135 @@ +#!/bin/sh +# Generated bytes fed to the code that parses what comes off the wire. +# +# Each harness is an LLVMFuzzerTestOneInput. With a clang that has the +# libFuzzer runtime, each gets a bounded campaign under ASan and UBSan; with +# one that does not -- Apple's, for instance -- standalone.c replays the +# corpus and a deterministic stream instead, which says the harnesses are +# wired up and nothing about coverage, and says so. +# +# fuzz_canary reads one byte past a buffer when the input opens with four +# particular bytes. One chance in 2^32 puts that out of reach of random +# input and well within reach of a fuzzer that watches which comparisons it +# got past, so a campaign that does not report it is not fuzzing, and the +# silence of the others would mean nothing. +# +# Usage: cbits/tests/fuzz/run.sh [seconds-per-harness] [build-dir] +set -eu + +root=$(CDPATH= cd -- "$(dirname -- "$0")/../../.." && pwd) +secs=${1:-30} +out=${2:-$(mktemp -d)} +cc=${CC:-cc} +mkdir -p "$out" +cd "$root" + +D=cbits/decaf +san="-fsanitize=address,undefined -fno-sanitize-recover=all" + +sources_for() { + case $1 in + canary) echo "" ;; + ed25519) echo "cbits/ed25519/ed25519.c cbits/crypton_sha512.c" ;; + p256) echo "cbits/p256/p256.c cbits/p256/p256_ec.c" ;; + aead) echo "cbits/crypton_aes.c cbits/aes/generic.c cbits/aes/gf.c + cbits/bearssl/aes_ct64.c cbits/bearssl/aes_ct64_enc.c + cbits/bearssl/aes_ct64_dec.c cbits/bearssl/ghash_ctmul64.c + cbits/bearssl/dec32le.c" ;; + decaf) echo "$D/ed448goldilocks/decaf_all.c $D/ed448goldilocks/eddsa.c + $D/ed448goldilocks/scalar.c $D/p448/f_arithmetic.c + $D/p448/f_generic.c $D/utils.c + $D/p448/arch_ref64/f_impl.c cbits/crypton_sha3.c" ;; + esac +} +includes_for() { + case $1 in + ed25519) echo "-Icbits/ed25519" ;; + decaf) echo "-DCRYPTON_DECAF_WORD_BITS=64 -I$D/include -I$D/p448 + -I$D/include/arch_ref64 -I$D/p448/arch_ref64" ;; + *) echo "" ;; + esac +} + +# Is there a libFuzzer to drive these? +have_fuzzer=no +printf '#include \n#include \nint LLVMFuzzerTestOneInput(const uint8_t *d, size_t n){(void)d;(void)n;return 0;}\n' \ + > "$out/probe.c" +if $cc -fsanitize=fuzzer,address -o "$out/probe" "$out/probe.c" 2>/dev/null; then + have_fuzzer=yes +fi + +status=0 +harnesses="canary ed25519 decaf p256 aead" + +for h in $harnesses; do + # shellcheck disable=SC2046,SC2086 + $cc -O1 -g $san $(test $have_fuzzer = yes && echo -fsanitize=fuzzer) \ + -Icbits -Icbits/include64 $(includes_for $h) \ + -o "$out/fuzz_$h" "cbits/tests/fuzz/fuzz_$h.c" \ + $(test $have_fuzzer = no && echo cbits/tests/fuzz/standalone.c) \ + $(sources_for $h) 2> "$out/$h.cc" || { + echo "FAIL $h did not build"; sed -n '1,12p' "$out/$h.cc"; status=1; continue + } + + if [ $have_fuzzer = no ]; then + "$out/fuzz_$h" cbits/tests/fuzz/corpus 20000 > "$out/$h.log" 2>&1 || true + if grep -qE "ERROR: |runtime error:" "$out/$h.log"; then + echo "FOUND $h: the sanitizers reported on replayed input" + grep -m1 -E "ERROR: |runtime error:" "$out/$h.log" | sed 's/^/ /' + status=1 + else + echo "ran $h ($(tail -1 "$out/$h.log"))" + fi + continue + fi + + mkdir -p "$out/corpus-$h" + cp cbits/tests/fuzz/corpus/* "$out/corpus-$h/" 2>/dev/null || true + # -use_value_profile records the operands of comparisons, which is how a + # fuzzer gets past a check for particular bytes rather than waiting for + # them to come up at random. The canary is exactly that check, and the + # parsers here are full of them. + "$out/fuzz_$h" "$out/corpus-$h" -max_total_time="$secs" \ + -use_value_profile=1 -print_final_stats=1 \ + > "$out/$h.log" 2>&1 || true + + # What a find looks like. AddressSanitizer writes "ERROR:", but + # UndefinedBehaviorSanitizer writes "runtime error:" and nothing else, + # so a detector that waits for "ERROR:" reads a campaign that found the + # canary as one that found nothing -- which is what the first three + # attempts at this file did. libFuzzer's own line is the one that + # covers every case: it writes the input out whatever reported. + found=no + grep -qE "Test unit written to|ERROR: |runtime error:|deadly signal" \ + "$out/$h.log" && found=yes + + if [ "$h" = canary ]; then + if [ $found = no ]; then + echo "FAIL canary: the harness that reads out of bounds on a" + echo " four-byte marker was not found in ${secs}s, so this" + echo " campaign is not exploring and nothing below counts." + echo " What it did do:" + tail -12 "$out/$h.log" | sed 's/^/ /' + status=1 + else + echo "ok canary: found, so the campaign explores" + fi + continue + fi + + if [ $found = yes ]; then + echo "FOUND $h: the sanitizers reported" + grep -m1 -E "ERROR: |runtime error:|deadly signal" "$out/$h.log" | + sed 's/^/ /' + sed -n '/#0 /,/#8 /p' "$out/$h.log" | head -12 | sed 's/^/ /' + status=1 + else + echo "ok $h: $(grep -oE 'stat::number_of_executed_units: *[0-9]+' \ + "$out/$h.log" | grep -oE '[0-9]+' | tail -1) inputs, nothing reported" + fi +done + +if [ $have_fuzzer = no ]; then + echo "skip $cc has no libFuzzer, so the corpus was replayed and nothing explored" +fi +exit $status diff --git a/cbits/tests/fuzz/standalone.c b/cbits/tests/fuzz/standalone.c new file mode 100644 --- /dev/null +++ b/cbits/tests/fuzz/standalone.c @@ -0,0 +1,66 @@ +/* A main for the harnesses, for where there is no fuzzer. + * + * Apple's clang ships no libFuzzer runtime, so this replays the committed + * corpus and then a deterministic stream of generated inputs. That says the + * harness is wired up and that the sanitizers are clean on what it feeds -- + * it is not a fuzzing campaign and does not explore anything. The campaign + * runs in CI, where clang has the runtime. + */ +#include +#include +#include +#include +#include + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size); + +static uint64_t s0 = 0x243f6a8885a308d3ULL, s1 = 0x13198a2e03707344ULL; +static uint64_t rnd(void) +{ + uint64_t x = s0, y = s1; + s0 = y; + x ^= x << 23; + s1 = x ^ y ^ (x >> 17) ^ (y >> 26); + return s1 + y; +} + +int main(int argc, char **argv) +{ + static uint8_t buf[4096]; + long rounds = argc > 2 ? strtol(argv[2], NULL, 10) : 20000; + long i; + int replayed = 0; + + if (argc > 1) { + DIR *d = opendir(argv[1]); + struct dirent *e; + if (d) { + while ((e = readdir(d)) != NULL) { + char path[1024]; + FILE *f; + size_t n; + if (e->d_name[0] == '.') + continue; + snprintf(path, sizeof path, "%s/%s", argv[1], e->d_name); + f = fopen(path, "rb"); + if (!f) + continue; + n = fread(buf, 1, sizeof buf, f); + fclose(f); + LLVMFuzzerTestOneInput(buf, n); + replayed++; + } + closedir(d); + } + } + + for (i = 0; i < rounds; i++) { + size_t n = (size_t)(rnd() % sizeof buf); + size_t j; + for (j = 0; j < n; j++) + buf[j] = (uint8_t)(rnd() >> 24); + LLVMFuzzerTestOneInput(buf, n); + } + printf("replayed %d corpus input(s), then %ld generated\n", replayed, rounds); + return 0; +} diff --git a/cbits/tests/perf/floors.txt b/cbits/tests/perf/floors.txt new file mode 100644 --- /dev/null +++ b/cbits/tests/perf/floors.txt @@ -0,0 +1,22 @@ +# A primitive that falls off its accelerated path does not get a little +# slower, it falls off a cliff: #274 took SHA-256 from 3396 MB/s to 644 on an +# Apple M4, a factor of five, and no test noticed because the answers stayed +# right. +# +# Absolute numbers cannot be checked here. ubuntu-latest is not one machine +# -- EPYC 7763, EPYC 9V74, Xeon 8370C and Xeon 6973P-C have all turned up, and +# the ones with AVX-512 differ from the ones without by more than twofold on +# AES-GCM. So each line is a ratio between two primitives measured in the +# same run on the same machine, with a floor generous enough that only a cliff +# reaches it. +# +# faster slower floor what it would catch +# +# Measured for the floors: an M4 gives 1.00, 0.56, 4.04 and 0.58 for the four +# below; an EPYC 7763 gives 0.94, 0.46, 2.75 and 0.55. Every floor is less +# than half of both, and #274 put the first at 0.19. + +sha256 sha1 0.40 SHA-256 off the SHA-2 instructions +sha512 sha1 0.20 SHA-512 off its assembly +aes128gcm chachapoly 1.00 AES-GCM off AES-NI or the ARMv8 AES instructions +sha3-256 sha512 0.25 SHA-3 off the SHA-3 instructions diff --git a/cbits/tests/perf/run.sh b/cbits/tests/perf/run.sh new file mode 100644 --- /dev/null +++ b/cbits/tests/perf/run.sh @@ -0,0 +1,102 @@ +#!/bin/sh +# Watching for a primitive that has fallen off its accelerated path. +# +# #274 took SHA-256 from 3396 MB/s to 644 on an Apple M4 and shipped, because +# the answers were right and only the speed was wrong. No test can catch +# that; this is what does. +# +# What it checks is ratios between primitives measured in the same run, not +# absolute throughput, because the runner is not the same machine twice -- +# see cbits/tests/perf/floors.txt. A ratio is only useful against a cliff; +# this will not notice a few per cent, and is not meant to. +# +# The last thing it does is build the library again with the AES acceleration +# turned off and check that the AES line then fails. Without that, a run +# where the measurement had quietly stopped working would look exactly like a +# run where everything was fast. +# +# Usage: cbits/tests/perf/run.sh [build-dir] +set -eu + +root=$(CDPATH= cd -- "$(dirname -- "$0")/../../.." && pwd) +out=${1:-$(mktemp -d)} +cc=${CC:-cc} +mkdir -p "$out" +cd "$root" + +# The harness links against the library as cabal built it, so that what is +# measured is the configuration the package actually ships. +build_harness() { + builddir=$1; bin=$2; shift 2 + cabal build lib:crypton --builddir="$builddir" -v0 "$@" > /dev/null + archive=$(find "$builddir" -name 'libHScrypton-*.a' | head -1) + test -n "$archive" || { echo "no library archive under $builddir"; exit 1; } + $cc -O3 -Icbits -Icbits/include64 -Icbits/include32 \ + -o "$bin" cbits/tests/perf/throughput.c "$archive" 2>/dev/null || + $cc -O3 -Icbits -Icbits/include64 \ + -o "$bin" cbits/tests/perf/throughput.c "$archive" +} + +measure() { + best=0 + for _ in 1 2 3; do + v=$("$1" "$2" 2>/dev/null || echo 0) + best=$(awk -v a="$best" -v b="$v" 'BEGIN{print (b>a)?b:a}') + done + echo "$best" +} + +# Every ratio in floors.txt, against the binary named. Prints one line each +# and returns the number that were under the floor. +check() { + bin=$1; label=$2; quiet=${3:-no} + bad=0 + while read -r fast slow floor _rest; do + case "$fast" in ''|\#*) continue ;; esac + a=$(measure "$bin" "$fast") + b=$(measure "$bin" "$slow") + r=$(awk -v a="$a" -v b="$b" 'BEGIN{printf "%.2f", (b>0)?a/b:0}') + under=$(awk -v r="$r" -v f="$floor" 'BEGIN{print (r +#include +#include +#include +#include + +#include "crypton_aes.h" +#include "crypton_sha1.h" +#include "crypton_sha256.h" +#include "crypton_sha512.h" +#include "crypton_sha3.h" +#include "crypton_chacha.h" +#include "crypton_poly1305.h" + +#define LEN 16384 +#define REPS 12 + +static uint8_t inb[LEN], outb[LEN + 64]; +static uint64_t sink; + +static double now_us(void) +{ + struct timespec ts; + clock_gettime(CLOCK_MONOTONIC, &ts); + return ts.tv_sec * 1e6 + ts.tv_nsec / 1e3; +} + +static void bench_gcm(int keybits, int iters) +{ + aes_key key; + aes_gcm gcm; + uint8_t kb[32], iv[12], tag[16]; + int i; + for (i = 0; i < 32; i++) kb[i] = (uint8_t)(i * 7); + for (i = 0; i < 12; i++) iv[i] = (uint8_t)(i + 3); + crypton_aes_initkey(&key, kb, keybits / 8); + crypton_aes_gcm_init(&gcm, &key, iv, 12); + for (i = 0; i < iters; i++) { + crypton_aes_gcm_encrypt(outb, &gcm, &key, inb, LEN); + sink += outb[i & 1023]; + } + crypton_aes_gcm_finish(tag, &gcm, &key); + sink += tag[0]; +} + +static void bench_chachapoly(int iters) +{ + crypton_chacha_context ctx; + poly1305_ctx pctx; + poly1305_key pkey; + poly1305_mac mac; + uint8_t kb[32], iv[12]; + int i; + for (i = 0; i < 32; i++) kb[i] = (uint8_t)(i * 5); + for (i = 0; i < 12; i++) iv[i] = (uint8_t)(i + 1); + memcpy(&pkey, kb, sizeof(pkey)); + for (i = 0; i < iters; i++) { + crypton_chacha_init(&ctx, 20, 32, kb, 12, iv); + crypton_chacha_combine(outb, &ctx, inb, LEN); + crypton_poly1305_init(&pctx, &pkey); + crypton_poly1305_update(&pctx, outb, LEN); + crypton_poly1305_finalize(mac, &pctx); + sink += mac[0] + outb[i & 1023]; + } +} + +static void bench_sha1(int iters) +{ + struct sha1_ctx c; + uint8_t out[20]; + int i; + for (i = 0; i < iters; i++) { + crypton_sha1_init(&c); + crypton_sha1_update(&c, inb, LEN); + crypton_sha1_finalize(&c, out); + sink += out[0]; + } +} + +static void bench_sha256(int iters) +{ + struct sha256_ctx c; + uint8_t out[32]; + int i; + for (i = 0; i < iters; i++) { + crypton_sha256_init(&c); + crypton_sha256_update(&c, inb, LEN); + crypton_sha256_finalize(&c, out); + sink += out[0]; + } +} + +static void bench_sha512(int iters) +{ + struct sha512_ctx c; + uint8_t out[64]; + int i; + for (i = 0; i < iters; i++) { + crypton_sha512_init(&c); + crypton_sha512_update(&c, inb, LEN); + crypton_sha512_finalize(&c, out); + sink += out[0]; + } +} + +static void bench_sha3(int iters) +{ + /* sha3_ctx ends in a flexible array the caller provides room for. */ + uint8_t raw[SHA3_CTX_BUF_MAX_SIZE]; + struct sha3_ctx *c = (struct sha3_ctx *)raw; + uint8_t out[32]; + int i; + for (i = 0; i < iters; i++) { + crypton_sha3_init(c, 256); + crypton_sha3_update(c, inb, LEN); + crypton_sha3_finalize(c, 256, out); + sink += out[0]; + } +} + +int main(int argc, char **argv) +{ + const char *algo = argc > 1 ? argv[1] : "sha256"; + int iters = 2000, r, i; + double best = 1e30; + + for (i = 0; i < LEN; i++) inb[i] = (uint8_t)(i * 17 + 3); + + for (r = 0; r < REPS; r++) { + double t0 = now_us(), t1; + if (!strcmp(algo, "aes128gcm")) bench_gcm(128, iters); + else if (!strcmp(algo, "aes256gcm")) bench_gcm(256, iters); + else if (!strcmp(algo, "chachapoly")) bench_chachapoly(iters); + else if (!strcmp(algo, "sha1")) bench_sha1(iters); + else if (!strcmp(algo, "sha256")) bench_sha256(iters); + else if (!strcmp(algo, "sha512")) bench_sha512(iters); + else if (!strcmp(algo, "sha3-256")) bench_sha3(iters); + else { fprintf(stderr, "unknown algo %s\n", algo); return 2; } + t1 = now_us(); + if (r > 1 && t1 - t0 < best) best = t1 - t0; + } + if (sink == 0) return 3; + printf("%.1f\n", (double)LEN * iters / best); /* bytes/us == MB/s */ + return 0; +} diff --git a/cbits/tests/ppc8_diff.c b/cbits/tests/ppc8_diff.c new file mode 100644 --- /dev/null +++ b/cbits/tests/ppc8_diff.c @@ -0,0 +1,270 @@ +/* + * The POWER8 path against the portable one. + * + * crypton's own test suite exercises whichever implementation the machine + * installed, so on a POWER8 it never runs the portable code and nothing + * compares the two. CI has no POWER8 either. This does the comparison in + * one process, with both compiled in, and runs under qemu-ppc64le. + * + * powerpc64le-linux-gnu-gcc -O2 -static -Icbits -Icbits/aes \ + * -DWITH_PPC8_CRYPTO -o ppc8_diff \ + * cbits/tests/ppc8_diff.c cbits/crypton_aes.c cbits/aes/generic.c \ + * cbits/aes/gf.c cbits/aes/ppc8.c cbits/crypton_cpu.c \ + * cbits/asm/aesp8-ppc-linux64le.S cbits/asm/ghashp8-ppc-linux64le.S \ + * cbits/bearssl/*.c + * qemu-ppc64le-static ./ppc8_diff + * + * Given any argument it corrupts a result in each section, so that the + * comparison can be seen to notice: one that cannot fail has said nothing. + * + * One trap in writing this, worth naming because it looks like a crash in + * the code under test. Several of the generic mode loops reach the block + * function through the branch table, which in this process holds the POWER8 + * entries -- so calling them as the reference hands a portable key to the + * POWER8 block function. The references below are therefore the entries + * that reach cbits/aes/generic.c directly, and XTS, whose generic loop takes + * its tweak through the table, is written out here instead. + */ +#include +#include +#include +#include + +#include "crypton_aes.h" +#include "aes/generic.h" +#include "aes/gf.h" +#include "aes/block128.h" +#include "aes/ppc8.h" + +/* The portable mode loops this compares against. crypton_aes.c defines + * them and declares them only to itself, at the top of that file, so they + * are declared again here. They were once in a header; when they left it, + * this file -- built by hand and by nobody since -- went on calling them + * undeclared, which C has not allowed since C99 and gcc 14 refuses. */ +void crypton_aes_generic_encrypt_ecb(aes_block *output, aes_key *key, + aes_block *input, uint32_t nb_blocks); +void crypton_aes_generic_decrypt_ecb(aes_block *output, aes_key *key, + aes_block *input, uint32_t nb_blocks); +void crypton_aes_generic_encrypt_cbc(aes_block *output, aes_key *key, + aes_block *iv, aes_block *input, + uint32_t nb_blocks); +void crypton_aes_generic_decrypt_cbc(aes_block *output, aes_key *key, + aes_block *iv, aes_block *input, + uint32_t nb_blocks); + +/* the portable CTR, which reaches generic.c rather than the branch table */ +void crypton_aes_bitsliced_encrypt_ctr(uint8_t *output, aes_key *key, + aes_block *iv, uint8_t *input, + uint32_t len); + +static int failures; +static int sabotage; + +static void same(const char *what, const uint8_t *a, const uint8_t *b, size_t n) +{ + size_t i; + + if (memcmp(a, b, n) == 0) + return; + printf(" MISMATCH %s\n ppc8 ", what); + for (i = 0; i < n && i < 32; i++) printf("%02x", a[i]); + printf("\n portable "); + for (i = 0; i < n && i < 32; i++) printf("%02x", b[i]); + printf("\n"); + failures++; +} + +static uint32_t rnd_state = 1; +static uint8_t rnd(void) +{ + rnd_state = rnd_state * 1103515245u + 12345u; + return (uint8_t) (rnd_state >> 16); +} +static void rnd_fill(uint8_t *p, size_t n) +{ + while (n--) *p++ = rnd(); +} + +/* + * XTS with the portable block function, written out because the generic + * loop takes its tweak through the branch table. + */ +static void xts_reference(uint8_t *out, aes_key *k1, aes_key *k2, + const uint8_t *dataunit, uint32_t spoint, + const uint8_t *in, uint32_t nb, int decrypt) +{ + block128 tweak, t; + uint32_t i; + + memcpy(&tweak, dataunit, 16); + crypton_aes_generic_encrypt_block(&tweak, k2, &tweak); + while (spoint-- > 0) + crypton_aes_generic_gf_mulx(&tweak); + + for (i = 0; i < nb; i++) { + block128_vxor(&t, (const block128 *) (in + 16 * i), &tweak); + if (decrypt) + crypton_aes_generic_decrypt_block(&t, k1, &t); + else + crypton_aes_generic_encrypt_block(&t, k1, &t); + block128_vxor((block128 *) (out + 16 * i), &t, &tweak); + crypton_aes_generic_gf_mulx(&tweak); + } +} + +/* the two inits write different things into the same aes_key, so each side + * gets its own */ +static void keys(aes_key *p8, aes_key *gen, const uint8_t *k, uint8_t len) +{ + memset(p8, 0, sizeof *p8); + memset(gen, 0, sizeof *gen); + p8->nbr = gen->nbr = len == 16 ? 10 : len == 24 ? 12 : 14; + crypton_aes_ppc8_init(p8, (uint8_t *) k, len); + crypton_aes_generic_init(gen, (uint8_t *) k, len); +} + +int main(int argc, char **argv) +{ + static const uint8_t klens[3] = { 16, 24, 32 }; + int round; + + setvbuf(stdout, NULL, _IONBF, 0); /* so a crash does not eat the section it was in */ + sabotage = argc > 1; + + printf("== the dispatch would take: aes=%d ==\n", + crypton_aes_ppc8_available()); + + printf("== ECB and CBC, both directions ==\n"); + for (round = 0; round < 300; round++) { + uint8_t key[32], in[16 * 9], a[16 * 9], b[16 * 9], iv[16]; + uint8_t len = klens[round % 3]; + uint32_t nb = 1 + (round % 9); + aes_key kp, kg; + + rnd_fill(key, len); + rnd_fill(in, nb * 16); + rnd_fill(iv, 16); + keys(&kp, &kg, key, len); + + crypton_aes_ppc8_encrypt_ecb((aes_block *) a, &kp, (aes_block *) in, nb); + crypton_aes_generic_encrypt_ecb((aes_block *) b, &kg, (aes_block *) in, nb); + if (sabotage && round == 3) a[0] ^= 1; + same("ecb encrypt", a, b, nb * 16); + + crypton_aes_ppc8_decrypt_ecb((aes_block *) a, &kp, (aes_block *) in, nb); + crypton_aes_generic_decrypt_ecb((aes_block *) b, &kg, (aes_block *) in, nb); + same("ecb decrypt", a, b, nb * 16); + + { + aes_block iv1, iv2; + memcpy(&iv1, iv, 16); memcpy(&iv2, iv, 16); + crypton_aes_ppc8_encrypt_cbc((aes_block *) a, &kp, &iv1, (aes_block *) in, nb); + crypton_aes_generic_encrypt_cbc((aes_block *) b, &kg, &iv2, (aes_block *) in, nb); + same("cbc encrypt", a, b, nb * 16); + + memcpy(&iv1, iv, 16); memcpy(&iv2, iv, 16); + crypton_aes_ppc8_decrypt_cbc((aes_block *) a, &kp, &iv1, (aes_block *) in, nb); + crypton_aes_generic_decrypt_cbc((aes_block *) b, &kg, &iv2, (aes_block *) in, nb); + same("cbc decrypt", a, b, nb * 16); + } + } + + /* + * CTR, and the one place the two counters have to be reconciled by + * hand. crypton counts over all 128 bits and the assembly over the + * low 32, so the work is handed over in runs that stop where that word + * wraps. A buffer that never reaches the wrap exercises none of that, + * so the IV here is placed a few blocks before it on purpose. + */ + printf("== CTR, including the 32-bit counter wrapping ==\n"); + for (round = 0; round < 200; round++) { + uint8_t key[32], in[600], a[600], b[600], iv[16]; + uint8_t len = klens[round % 3]; + uint32_t n = 1 + (round % 600); + aes_key kp, kg; + aes_block iv1, iv2; + uint32_t before = round % 5; /* blocks left before the wrap */ + + rnd_fill(key, len); + rnd_fill(in, n); + rnd_fill(iv, 16); + /* put the low word within a few blocks of wrapping, and for a + * third of the rounds exactly on it */ + iv[12] = 0xff; iv[13] = 0xff; iv[14] = 0xff; + iv[15] = (uint8_t) (0x100 - before - 1); + keys(&kp, &kg, key, len); + + memcpy(&iv1, iv, 16); memcpy(&iv2, iv, 16); + crypton_aes_ppc8_encrypt_ctr(a, &kp, &iv1, in, n); + crypton_aes_bitsliced_encrypt_ctr(b, &kg, &iv2, in, n); + if (sabotage && round == 7) a[n - 1] ^= 2; + same("ctr across the wrap", a, b, n); + } + + printf("== XTS, both directions and a starting point ==\n"); + for (round = 0; round < 200; round++) { + uint8_t k1[32], k2[32], in[16 * 11], a[16 * 11], b[16 * 11], du[16]; + uint8_t len = klens[round % 3]; + uint32_t nb = 1 + (round % 11); + uint32_t spoint = round % 4; + aes_key p1, g1, p2, g2; + aes_block d1; + + rnd_fill(k1, len); + rnd_fill(k2, len); + rnd_fill(in, nb * 16); + rnd_fill(du, 16); + keys(&p1, &g1, k1, len); + keys(&p2, &g2, k2, len); + + memcpy(&d1, du, 16); + crypton_aes_ppc8_encrypt_xts((aes_block *) a, &p1, &p2, &d1, spoint, + (aes_block *) in, nb); + xts_reference(b, &g1, &g2, du, spoint, in, nb, 0); + /* a[0], not a further block: at this round nb is 1 and anything + * past the first block is outside what same() is given */ + if (sabotage && round == 11) a[0] ^= 4; + same("xts encrypt", a, b, nb * 16); + + memcpy(&d1, du, 16); + crypton_aes_ppc8_decrypt_xts((aes_block *) a, &p1, &p2, &d1, spoint, + (aes_block *) in, nb); + xts_reference(b, &g1, &g2, du, spoint, in, nb, 1); + same("xts decrypt", a, b, nb * 16); + } + + /* + * GHASH, where the two arguments do not take the same convention and + * ppc8.c has to swap one of them. Two different H and a non-zero + * accumulator, because a symmetric H or a zero start can hide a wrong + * convention. + */ + printf("== GHASH ==\n"); + for (round = 0; round < 300; round++) { + uint8_t h[16], data[64], start[16]; + table_4bit tp, tg; + block128 ap, ag; + + rnd_fill(h, 16); + rnd_fill(data, sizeof data); + rnd_fill(start, 16); + + crypton_aes_ppc8_hinit(tp, (const block128 *) h); + crypton_aes_generic_hinit(tg, (const block128 *) h); + + memcpy(&ap, start, 16); memcpy(&ag, start, 16); + crypton_aes_ppc8_gf_mul4(&ap, (const block128 *) data, tp); + crypton_aes_generic_gf_mul4(&ag, (const block128 *) data, tg); + if (sabotage && round == 5) ((uint8_t *) &ap)[0] ^= 8; + same("gf_mul4", (const uint8_t *) &ap, (const uint8_t *) &ag, 16); + + memcpy(&ap, start, 16); memcpy(&ag, start, 16); + crypton_aes_ppc8_gf_mul(&ap, tp); + crypton_aes_generic_gf_mul(&ag, tg); + same("gf_mul", (const uint8_t *) &ap, (const uint8_t *) &ag, 16); + } + + printf("%s: %d mismatch(es)%s\n", failures ? "FAIL" : "ok", failures, + sabotage ? " (sabotage was asked for)" : ""); + return failures != 0; +} diff --git a/cbits/tests/scrub/README b/cbits/tests/scrub/README new file mode 100644 --- /dev/null +++ b/cbits/tests/scrub/README @@ -0,0 +1,53 @@ +What is left in memory after a secret has been through it. + +Round eight asked whether a secret can be read from the time a primitive +takes. This asks whether it can be read from the memory afterwards. The +stack below a returning function is not erased -- it is simply no longer +addressed -- and a context handed back to a caller is whatever the primitive +left in it, so a later core file, crash dump or swapped page can carry a key +away long after the caller believes it is done with it. + +Each probe paints the stack with a filler, runs a primitive on an +unmistakable secret, copies the painted region away before anything can +disturb it, and looks for the pattern. Two things are asked separately: + + scratch what the primitive left below the caller's frame, in its own + working memory + context what it left in the context object the caller still holds, which + is on the heap here as it is in crypton + +The first probe keeps the secret on purpose and has to be found. That is not +ceremony: the first two versions of this file reported that nothing was ever +left behind, including by that probe, and both times the search was at fault. +Once because the probed function was inlined into its caller, so its locals +sat in a live frame rather than an abandoned one; once because the loop that +copies the region away was turned into a call to memcpy, whose own frame +landed exactly on the evidence. + +What this can say and what it cannot +------------------------------------ + +It finds a secret that survives *as it was handed over*. It cannot find one +that survives transformed, and the difference matters: Poly1305 reports +nothing, but its finalize clears nothing either -- the key is clamped into r +and pad, so it is still there and simply not byte-for-byte what was passed in. +Read "no verbatim copy" as exactly that and no more. + +What it found +------------- + +Nothing survives in any primitive's own scratch. The RSA exponentiation is +clean, which is worth saying because crypton_powm_sec memsets its table and +then frees it, and a compiler is entitled to drop a store to memory that is +about to die. clang at -O2 keeps it -- the bzero is still there in the +assembly, two instructions before the free -- but that is the compiler's +choice rather than a guarantee, and explicit_bzero exists for this. + +Three contexts keep the secret as it was given: SHA-256, SHA-512 and +ChaCha20. The hash ones hold the buffered message, the ChaCha one holds the +key. Nothing clears any of them, in the C or in the Haskell above it, where +Context is Bytes rather than ScrubbedBytes -- and hashFinalize copies the +context before finalizing, so there are two of them per digest. They are +listed in known.txt; whether to clear them is a question about what a context +means after it is finished with, and about the cost of scrubbing every hash, +rather than something to settle here. diff --git a/cbits/tests/scrub/known.txt b/cbits/tests/scrub/known.txt new file mode 100644 --- /dev/null +++ b/cbits/tests/scrub/known.txt @@ -0,0 +1,16 @@ +# Places that keep a secret as it was handed over, which are known and not +# treated as defects here. A probe reporting only these passes; anything +# else fails. + +# A hash context buffers the message a block at a time, so hashing a secret +# leaves it in the context. Nothing clears it: crypton_sha256_finalize and +# its siblings compute the digest and return, and the Haskell side holds the +# context as Bytes rather than ScrubbedBytes, so it is not cleared there +# either. hashFinalize copies the context before finalizing it, so there are +# two such objects per digest rather than one. +sha256 context +sha512 context + +# crypton_chacha_init copies the key into the state, where it stays for the +# life of the context. +chacha20 context diff --git a/cbits/tests/scrub/run.sh b/cbits/tests/scrub/run.sh new file mode 100644 --- /dev/null +++ b/cbits/tests/scrub/run.sh @@ -0,0 +1,30 @@ +#!/bin/sh +# What is left on the stack after a secret has been through it. +# +# The stack below a returning function is not erased -- it is simply no longer +# addressed -- so whatever a primitive kept there stays until something else +# writes over it, and a later core file or swapped page carries it away. +# +# The driver paints the stack, runs each primitive on an unmistakable secret, +# copies the painted region away before anything can disturb it, and looks for +# the pattern. The first probe keeps the secret on purpose and has to be +# found; if it is not, the search is looking somewhere the calls do not use +# and no other result counts. +# +# Not run under the sanitizers: reading the stack below the current frame is +# exactly what this does, and ASan calls that an error. +# +# Usage: cbits/tests/scrub/run.sh [build-dir] +set -eu + +root=$(CDPATH= cd -- "$(dirname -- "$0")/../../.." && pwd) +out=${1:-$(mktemp -d)} +cc=${CC:-cc} +mkdir -p "$out" +cd "$root" + +$cc -O2 -Icbits -Icbits/include64 -o "$out/scrub" cbits/tests/scrub/scrub.c \ + cbits/crypton_sha256.c cbits/crypton_sha512.c cbits/crypton_chacha.c \ + cbits/crypton_poly1305.c cbits/crypton_powm.c + +"$out/scrub" diff --git a/cbits/tests/scrub/scrub.c b/cbits/tests/scrub/scrub.c new file mode 100644 --- /dev/null +++ b/cbits/tests/scrub/scrub.c @@ -0,0 +1,271 @@ +/* What is left on the stack after a secret has been through it. + * + * The stack below a returning function is not erased -- it is simply no + * longer addressed -- so whatever the function kept there stays until + * something else writes over it. For a hash context or a key schedule that + * means a copy of the key can outlive every object the caller thinks holds + * it, and a later crash dump, core file or swapped page carries it away. + * + * So: paint the stack with a filler, run the primitive on a secret made of + * an unmistakable pattern, copy the painted region somewhere else before + * anything can disturb it, and look for the pattern in the copy. A hit is a + * place where the secret outlived the call. + * + * The region examined is below this file's own frame, so the driver's own + * copy of the secret is not what is being found. + */ +#include +#include +#include +#include + +/* Each probed function needs a frame of its own: inlined into probe, its + * locals would sit in a live frame rather than an abandoned one, and finding + * them there would mean nothing. That cost the first attempt at this file. */ +#if defined(__GNUC__) || defined(__clang__) +#define NOINLINE __attribute__((noinline)) +#else +#define NOINLINE +#endif + +/* 32 bytes that nothing else would produce */ +static const uint8_t SECRET[32] = { + 0x9e, 0x37, 0x79, 0xb9, 0x7f, 0x4a, 0x7c, 0x15, + 0xf3, 0x9c, 0xc0, 0x60, 0x5c, 0xed, 0xc8, 0x34, + 0x1a, 0x2e, 0x8f, 0x5b, 0xd7, 0x06, 0x41, 0x92, + 0xc3, 0x58, 0xe6, 0x70, 0xb1, 0x24, 0xaf, 0x8d, +}; + +#define REGION (512 * 1024) +#define FILLER 0x5a + +/* Write the filler over the stack the primitive is about to use. Recursion + * rather than one large frame, so that the compiler cannot decide the whole + * thing is dead and skip it. */ +static void paint(int depth) { + volatile uint8_t pad[8192]; + size_t i; + for (i = 0; i < sizeof pad; i++) pad[i] = FILLER; + if (depth > 0) paint(depth - 1); +} + +/* How many times the pattern appears in the copy. A run of the filler is + * what is expected; anything else is what was left behind. */ +static int count_hits(const uint8_t *hay, size_t n, const uint8_t *needle, + size_t m) { + size_t i; + int hits = 0; + if (n < m) return 0; + for (i = 0; i + m <= n; i++) + if (hay[i] == needle[0] && memcmp(hay + i, needle, m) == 0) hits++; + return hits; +} + +static uint8_t *snapshot; +static int failures, checked; + +/* The context a primitive is handed, on the heap where crypton's own callers + * put it. After the call it is looked at directly: whatever is still in it + * is what the caller is left holding. */ +static void *ctx_mem; +static size_t ctx_len; + +/* known.txt lists the places that keep the secret and are understood. A + * probe that reports only those passes; anything else is new and fails. */ +static int is_known(const char *name, const char *what) { + char line[256], want[128]; + FILE *f = fopen("cbits/tests/scrub/known.txt", "r"); + int found = 0; + if (!f) return 0; + snprintf(want, sizeof want, "%s %s", name, what); + while (fgets(line, sizeof line, f)) { + char a[64], b[64]; + if (line[0] == '#' || sscanf(line, "%63s %63s", a, b) != 2) continue; + if (strcmp(a, name) == 0 && strcmp(b, what) == 0) { found = 1; break; } + } + fclose(f); + return found; +} + +/* Run one primitive and report what it left. The callback is handed the + * secret and is expected to use it and return. */ +static void probe(const char *name, void (*run)(const uint8_t *, size_t), + size_t look_for) { + volatile uint8_t here; + const uint8_t *low; + int hits; + + paint(24); /* about 200 KB of filler */ + run(SECRET, sizeof SECRET); + + /* Everything below this frame is what the call used. Copied with a + * loop rather than memcpy: a call pushes a frame exactly where the one + * being examined was, and would erase the top of the evidence before it + * could be read. That is how the first version of this reported that + * nothing was ever left behind, including by the canary. */ + low = (const uint8_t *)&here - REGION; + { + /* volatile, or the compiler recognises the loop and emits memcpy -- + * which is the call this loop exists to avoid. That cost the first + * two attempts at this file. */ + const volatile uint8_t *v = low; + size_t k; + for (k = 0; k < REGION; k++) snapshot[k] = v[k]; + } + + hits = count_hits(snapshot, REGION, SECRET, look_for); + checked++; + if (strcmp(name, "canary") == 0) { + if (hits == 0) { + printf("FAIL canary: the driver that keeps the secret on purpose\n" + " was not found, so nothing below this line counts\n"); + failures++; + } else { + printf("ok canary: found %d, so the search works\n", hits); + } + return; + } + /* two separate questions: what the primitive left in its own scratch, + and what it left in the context its caller still holds */ + if (hits == 0) { + printf("ok %-9s scratch: no verbatim copy below the frame\n", name); + } else if (is_known(name, "scratch")) { + printf("note %-9s scratch: %d verbatim copy(ies), known\n", name, hits); + } else { + printf("LEFT %-9s scratch: %d verbatim copy(ies) below the frame,\n" + " and cbits/tests/scrub/known.txt does not list it\n", + name, hits); + failures++; + } + if (ctx_len) { + int chits = count_hits((const uint8_t *)ctx_mem, ctx_len, SECRET, + look_for); + /* No hit is not the same as no secret. A primitive that stores the + key transformed -- Poly1305 clamps it into r and pad -- keeps key + material the search cannot see. All this can say is whether the + bytes survive as they were handed over. */ + if (chits == 0) { + printf("ok %-9s context: no verbatim copy of the secret\n", + name); + } else if (is_known(name, "context")) { + printf("note %-9s context: %d verbatim copy(ies), known\n", + name, chits); + } else { + printf("LEFT %-9s context: %d verbatim copy(ies) of the secret,\n" + " and cbits/tests/scrub/known.txt does not list it\n", + name, chits); + failures++; + } + ctx_len = 0; + } +} + +/* ---- the primitives ---------------------------------------------------- */ + +#include "crypton_sha256.h" +#include "crypton_sha512.h" +#include "crypton_chacha.h" +#include "crypton_poly1305.h" +#include "crypton_powm.h" + +NOINLINE static void run_sha256(const uint8_t *s, size_t n) { + struct sha256_ctx *ctx = ctx_mem; + uint8_t out[32]; + ctx_len = sizeof *ctx; + crypton_sha256_init(ctx); + crypton_sha256_update(ctx, s, (uint32_t)n); + crypton_sha256_finalize(ctx, out); + if (out[0] == 0xff && out[31] == 0xff) printf("unreachable\n"); +} + +NOINLINE static void run_sha512(const uint8_t *s, size_t n) { + struct sha512_ctx *ctx = ctx_mem; + uint8_t out[64]; + ctx_len = sizeof *ctx; + crypton_sha512_init(ctx); + crypton_sha512_update(ctx, s, (uint32_t)n); + crypton_sha512_finalize(ctx, out); + if (out[0] == 0xff && out[63] == 0xff) printf("unreachable\n"); +} + +NOINLINE static void run_chacha(const uint8_t *s, size_t n) { + crypton_chacha_context *ctx = ctx_mem; + uint8_t iv[8] = {1, 2, 3, 4, 5, 6, 7, 8}; + uint8_t out[64]; + ctx_len = sizeof *ctx; + crypton_chacha_init(ctx, 20, (uint32_t)n, s, sizeof iv, iv); + crypton_chacha_combine(out, ctx, out, sizeof out); + if (out[0] == 0xff && out[63] == 0xff) printf("unreachable\n"); +} + +NOINLINE static void run_poly1305(const uint8_t *s, size_t n) { + poly1305_ctx *ctx = ctx_mem; + poly1305_mac mac; + uint8_t msg[64]; + /* the key is handed over where it already lies, so that nothing of it is + put on this frame by the driver rather than by the library */ + (void)n; + memset(msg, 0x11, sizeof msg); + ctx_len = sizeof *ctx; + crypton_poly1305_init(ctx, (poly1305_key *)(void *)(uintptr_t)s); + crypton_poly1305_update(ctx, msg, sizeof msg); + crypton_poly1305_finalize(mac, ctx); + if (mac[0] == 0xff && mac[15] == 0xff) printf("unreachable\n"); +} + +/* The RSA private-key exponentiation. Its scratch is on the heap, but the + * windows it selects and the accumulators pass through the stack. */ +static uint8_t *powm_out, *powm_base, *powm_mod, *powm_exp; +NOINLINE static void run_powm(const uint8_t *s, size_t n) { + /* Everything is on the heap: the exponent because it is the secret and + must not be put on this frame by the driver, the rest to keep the + frame small enough that what is found below it came from the library. */ + enum { LEN = 128 }; + uint8_t *out = powm_out, *base = powm_base, *mod = powm_mod; + (void)s; (void)n; + if (crypton_powm_sec(out, base, LEN, powm_exp, LEN, mod, LEN) != 0) + printf("powm_sec refused\n"); + if (out[0] == 0xff && out[LEN - 1] == 0xff) printf("unreachable\n"); +} + +/* The calibration. This one keeps the secret on the stack on purpose, so it + * has to be found; if it is not, the painting or the snapshot is looking + * somewhere the calls do not use and every "ok" below means nothing. */ +NOINLINE static void run_canary(const uint8_t *s, size_t n) { + volatile uint8_t copy[128]; + size_t i; + for (i = 0; i < n && i < sizeof copy; i++) copy[i] = s[i]; + if (copy[0] == 0xff && copy[31] == 0xff) printf("unreachable\n"); +} + +int main(void) { + size_t i; + snapshot = malloc(REGION); + ctx_mem = malloc(4096); + powm_out = malloc(128); powm_base = malloc(128); + powm_mod = malloc(128); powm_exp = malloc(128); + if (!snapshot || !ctx_mem || !powm_out || !powm_base || !powm_mod || !powm_exp) return 2; + for (i = 0; i < 128; i++) { + powm_base[i] = (uint8_t)(i * 3 + 1); + powm_mod[i] = (uint8_t)(i * 5 + 7); + powm_exp[i] = SECRET[i % sizeof SECRET]; + } + powm_mod[0] |= 0x80; + powm_mod[127] |= 1; + powm_base[0] &= 0x7f; + + probe("canary", run_canary, 32); + probe("sha256", run_sha256, 32); + probe("sha512", run_sha512, 32); + probe("chacha20", run_chacha, 32); + probe("poly1305", run_poly1305, 16); + probe("powm_sec", run_powm, 32); + + free(snapshot); + if (failures) + printf("\n%d place(s) keep the secret and are not in known.txt\n", + failures); + else + printf("\nnothing keeps the secret that known.txt does not name\n"); + return failures != 0; +} diff --git a/cbits/tests/sysdrg/run.sh b/cbits/tests/sysdrg/run.sh new file mode 100644 --- /dev/null +++ b/cbits/tests/sysdrg/run.sh @@ -0,0 +1,22 @@ +#!/bin/sh +# Does the generator behind MonadRandom IO survive the things that break a +# generator: a second thread, a reseed, and a fork? +# +# None of this can be asked from Haskell alone. A forkIO thread is not an +# operating system thread, and the Haskell test suite cannot fork the way a +# child process must for the question to mean anything. +# +# Usage: cbits/tests/sysdrg/run.sh [build-dir] +set -eu + +cbits=$(cd "$(dirname "$0")/../.." && pwd) +out=${1:-$(mktemp -d)} + +${CC:-cc} -O2 -Wall -Wextra -DCRYPTON_SYSDRG_TESTING -I"$cbits" -o "$out/sysdrg" \ + "$cbits/tests/sysdrg/sysdrg.c" \ + "$cbits/crypton_sysdrg.c" \ + "$cbits/crypton_sysrandom.c" \ + "$cbits/crypton_chacha.c" \ + "$cbits/crypton_sha512.c" + +"$out/sysdrg" diff --git a/cbits/tests/sysdrg/sysdrg.c b/cbits/tests/sysdrg/sysdrg.c new file mode 100644 --- /dev/null +++ b/cbits/tests/sysdrg/sysdrg.c @@ -0,0 +1,140 @@ +#include +#include +#include +#include +#include +#include +#include + +void crypton_sysdrg_test_key(uint8_t out[32]); +void crypton_sysdrg_test_lock(void); +void crypton_sysdrg_test_unlock(void); +int crypton_sysdrg_bytes(uint8_t *out, int len); +uint32_t crypton_sysdrg_generation(void); +uint64_t crypton_sysdrg_thread_used(void); +static uint64_t used_in_thread; + +static int fail = 0; +static void check(const char *what, int ok) { + printf("%-52s %s\n", what, ok ? "ok" : "FAIL"); + if (!ok) fail = 1; +} + +/* Holds the process generator's lock for a moment, so that a fork can be + * made to happen while it is held. Releasing after a delay rather than on + * demand is deliberate: a prepare handler has to be able to take the lock, + * and a holder that waited to be told would stop the fork instead of the + * child. */ +static pthread_mutex_t gate = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t gate_cv = PTHREAD_COND_INITIALIZER; +static int holding = 0; + +static void *lock_holder(void *arg) { + (void) arg; + crypton_sysdrg_test_lock(); + pthread_mutex_lock(&gate); + holding = 1; + pthread_cond_broadcast(&gate_cv); + pthread_mutex_unlock(&gate); + usleep(200000); + crypton_sysdrg_test_unlock(); + return NULL; +} + +static void *thread_body(void *arg) { + uint8_t *out = (uint8_t *) arg; + check("a second OS thread gets bytes", crypton_sysdrg_bytes(out, 32) == 32); + used_in_thread = crypton_sysdrg_thread_used(); + return NULL; +} + +int main(void) { + uint8_t a[32], b[32], big[4096]; + memset(a, 0, 32); memset(b, 0, 32); + + check("asking for 32 bytes gives 32", crypton_sysdrg_bytes(a, 32) == 32); + check("asking again gives something else", + crypton_sysdrg_bytes(b, 32) == 32 && memcmp(a, b, 32) != 0); + check("a 4096-byte request is filled", crypton_sysdrg_bytes(big, 4096) == 4096); + check("zero length is fine", crypton_sysdrg_bytes(a, 0) == 0); + + /* two OS threads must not share a stream. The main thread has already + * produced over 4 KiB by here, so a shared generator would show it. */ + uint8_t t1[32], t2[32]; + pthread_t p1, p2; + pthread_create(&p1, NULL, thread_body, t1); + pthread_join(p1, NULL); + pthread_create(&p2, NULL, thread_body, t2); + pthread_join(p2, NULL); + check("two OS threads do not produce the same bytes", memcmp(t1, t2, 32) != 0); + check("a new OS thread starts its own stream, not the main one's", + used_in_thread == 32); + + /* crossing the per-thread reseed limit (1 MiB) must not repeat */ + uint8_t before[32], after[32]; + crypton_sysdrg_bytes(before, 32); + for (int i = 0; i < 1100; i++) { uint8_t junk[1024]; crypton_sysdrg_bytes(junk, 1024); } + crypton_sysdrg_bytes(after, 32); + check("output after a reseed differs from before", memcmp(before, after, 32) != 0); + + /* fork: parent and child must diverge */ + int fds[2]; + if (pipe(fds) != 0) { perror("pipe"); return 2; } + uint8_t parent[32], child[32]; + pid_t pid = fork(); + if (pid == 0) { + close(fds[0]); + crypton_sysdrg_bytes(child, 32); + ssize_t w = write(fds[1], child, 32); + _exit(w == 32 ? 0 : 1); + } + close(fds[1]); + crypton_sysdrg_bytes(parent, 32); + ssize_t r = read(fds[0], child, 32); + int status = 0; waitpid(pid, &status, 0); + check("the child was able to produce bytes", r == 32 && status == 0); + check("parent and child do not produce the same bytes", memcmp(parent, child, 32) != 0); + check("the fork was noticed", crypton_sysdrg_generation() == 0); + + /* Backtracking resistance. The key that produced a draw is replaced + * once the bytes are out, so a state read afterwards is not the state + * that made them and cannot be wound back to remake them. Comparing + * the key before and against the key after is the whole of it: without + * the rekey it is the same key and the counter alone says where to + * start. */ + uint8_t key_before[32], key_after[32], drawn[32]; + memset(key_before, 0, 32); memset(key_after, 0, 32); + crypton_sysdrg_test_key(key_before); + crypton_sysdrg_bytes(drawn, 32); + crypton_sysdrg_test_key(key_after); + check("a draw replaces the key that made it", + memcmp(key_before, key_after, 32) != 0); + + /* A fork while another thread holds the process generator's lock. The + * child's first draw has to reseed, the generation having changed, and + * reseeding takes that lock. With no prepare and parent handlers the + * child inherits it locked and waits on it for ever -- the thread that + * would release it did not come across the fork. The alarm is what + * turns that into a failure rather than a hung test run. */ + pthread_t holder; + if (pthread_create(&holder, NULL, lock_holder, NULL) != 0) { + perror("pthread_create"); return 2; + } + pthread_mutex_lock(&gate); + while (!holding) pthread_cond_wait(&gate_cv, &gate); + pthread_mutex_unlock(&gate); + + pid_t locked_pid = fork(); + if (locked_pid == 0) { + uint8_t c[32]; + alarm(5); + _exit(crypton_sysdrg_bytes(c, 32) == 32 ? 0 : 1); + } + pthread_join(holder, NULL); + int locked_status = 0; + waitpid(locked_pid, &locked_status, 0); + check("a child forked while the lock was held can draw", + WIFEXITED(locked_status) && WEXITSTATUS(locked_status) == 0); + + return fail; +} diff --git a/cbits/tests/width/ed448_width.c b/cbits/tests/width/ed448_width.c new file mode 100644 --- /dev/null +++ b/cbits/tests/width/ed448_width.c @@ -0,0 +1,87 @@ +/* Ed448 and X448 asked of the 32-bit field arithmetic and of the 64-bit one. + Only the f_impl.c under p448/arch_32 or p448/arch_ref64, and the two arch + include directories, differ between the builds; everything above them is + the same source. */ +#include +#include +#include +#include "decaf/ed448.h" +#include "decaf/point_448.h" + +static uint64_t s0 = 0x243f6a8885a308d3ULL, s1 = 0x13198a2e03707344ULL; +static uint64_t rnd(void) { + uint64_t x = s0, y = s1; + s0 = y; x ^= x << 23; + s1 = x ^ y ^ (x >> 17) ^ (y >> 26); + return s1 + y; +} +static void fill(uint8_t *p, size_t n) { + for (size_t i = 0; i < n; i++) p[i] = (uint8_t)(rnd() >> 24); +} +static void show(const char *t, const uint8_t *b, size_t n) { + printf("%s ", t); + for (size_t i = 0; i < n; i++) printf("%02x", b[i]); + printf("\n"); +} + +int main(void) { + uint8_t priv[CRYPTON_DECAF_EDDSA_448_PRIVATE_BYTES]; + uint8_t pub[CRYPTON_DECAF_EDDSA_448_PUBLIC_BYTES]; + uint8_t sig[CRYPTON_DECAF_EDDSA_448_SIGNATURE_BYTES]; + uint8_t msg[256], ctx[8]; + uint8_t xs[CRYPTON_DECAF_X448_PRIVATE_BYTES]; + uint8_t xb[CRYPTON_DECAF_X448_PUBLIC_BYTES]; + uint8_t xo[CRYPTON_DECAF_X448_PUBLIC_BYTES]; + + /* the scalars and points worth naming */ + static const uint8_t edge[3] = {0x00, 0x01, 0xff}; + for (unsigned e = 0; e < 3; e++) { + memset(priv, edge[e], sizeof priv); + crypton_decaf_ed448_derive_public_key(pub, priv); + show("epub", pub, sizeof pub); + crypton_decaf_ed448_sign(sig, priv, pub, (const uint8_t *)"", 0, 0, NULL, 0); + show("esig", sig, sizeof sig); + printf("everify=%d\n", + crypton_decaf_ed448_verify(sig, pub, (const uint8_t *)"", 0, 0, NULL, 0)); + + memset(xs, edge[e], sizeof xs); + crypton_decaf_x448_derive_public_key(xo, xs); + show("xpub", xo, sizeof xo); + memset(xb, edge[e], sizeof xb); + printf("x448=%d\n", crypton_decaf_x448(xo, xb, xs)); + show("xsh", xo, sizeof xo); + } + + for (int it = 0; it < 512; it++) { + size_t mlen = (size_t)(rnd() % sizeof msg); + uint8_t clen = (uint8_t)(rnd() % sizeof ctx); + fill(priv, sizeof priv); + fill(msg, sizeof msg); + fill(ctx, sizeof ctx); + + crypton_decaf_ed448_derive_public_key(pub, priv); + show("pub", pub, sizeof pub); + crypton_decaf_ed448_sign(sig, priv, pub, msg, mlen, 0, ctx, clen); + show("sig", sig, sizeof sig); + printf("ok=%d bad=%d\n", + crypton_decaf_ed448_verify(sig, pub, msg, mlen, 0, ctx, clen), + crypton_decaf_ed448_verify(sig, pub, msg, mlen, 1, ctx, clen)); + /* a signature with one bit moved has to fail on both builds alike */ + sig[(size_t)(rnd() % sizeof sig)] ^= 1; + printf("tampered=%d\n", + crypton_decaf_ed448_verify(sig, pub, msg, mlen, 0, ctx, clen)); + + fill(xs, sizeof xs); + crypton_decaf_x448_derive_public_key(xb, xs); + show("xp", xb, sizeof xb); + uint8_t xs2[CRYPTON_DECAF_X448_PRIVATE_BYTES], xb2[CRYPTON_DECAF_X448_PUBLIC_BYTES]; + uint8_t sh1[CRYPTON_DECAF_X448_PUBLIC_BYTES], sh2[CRYPTON_DECAF_X448_PUBLIC_BYTES]; + fill(xs2, sizeof xs2); + crypton_decaf_x448_derive_public_key(xb2, xs2); + int r1 = crypton_decaf_x448(sh1, xb2, xs); + int r2 = crypton_decaf_x448(sh2, xb, xs2); + printf("r=%d,%d agree=%d\n", r1, r2, memcmp(sh1, sh2, sizeof sh1) == 0); + show("sh", sh1, sizeof sh1); + } + return 0; +} diff --git a/cbits/tests/width/p256_width.c b/cbits/tests/width/p256_width.c new file mode 100644 --- /dev/null +++ b/cbits/tests/width/p256_width.c @@ -0,0 +1,144 @@ +/* Exercise the public P-256 API and print everything it answers, so that the + 32-bit build and the 64-bit build can be compared byte for byte. Every + value crosses the boundary as big-endian bytes, which is the one form the + two representations agree on by construction. */ +#include +#include +#include +#include "p256/p256.h" + +/* This family has no header at all -- the Haskell side declares it through + the FFI -- so it is declared here. */ +void crypton_p256e_point_mul(const crypton_p256_int *n, + const crypton_p256_int *in_x, const crypton_p256_int *in_y, + crypton_p256_int *out_x, crypton_p256_int *out_y); +void crypton_p256e_point_add( + const crypton_p256_int *in_x1, const crypton_p256_int *in_y1, + const crypton_p256_int *in_x2, const crypton_p256_int *in_y2, + crypton_p256_int *out_x, crypton_p256_int *out_y); +void crypton_p256e_point_negate( + const crypton_p256_int *in_x, const crypton_p256_int *in_y, + crypton_p256_int *out_x, crypton_p256_int *out_y); +void crypton_p256e_modadd(const crypton_p256_int *MOD, + const crypton_p256_int *a, const crypton_p256_int *b, crypton_p256_int *c); +void crypton_p256e_modsub(const crypton_p256_int *MOD, + const crypton_p256_int *a, const crypton_p256_int *b, crypton_p256_int *c); +void crypton_p256e_scalar_invert(const crypton_p256_int *a, crypton_p256_int *b); + +static uint64_t s0 = 0x243f6a8885a308d3ULL, s1 = 0x13198a2e03707344ULL; +static uint64_t rnd(void) { /* xoroshiro-ish, deterministic */ + uint64_t x = s0, y = s1; + s0 = y; + x ^= x << 23; + s1 = x ^ y ^ (x >> 17) ^ (y >> 26); + return s1 + y; +} +static void rnd_bytes(uint8_t *p, int n) { + for (int i = 0; i < n; i++) p[i] = (uint8_t)(rnd() >> 24); +} +static void show(const char *tag, const crypton_p256_int *v) { + uint8_t b[P256_NBYTES]; + crypton_p256_to_bin(v, b); + printf("%s ", tag); + for (int i = 0; i < P256_NBYTES; i++) printf("%02x", b[i]); + printf("\n"); +} +static void from_hex(const char *h, crypton_p256_int *out) { + uint8_t b[P256_NBYTES]; + for (int i = 0; i < P256_NBYTES; i++) { + unsigned v; sscanf(h + 2 * i, "%2x", &v); b[i] = (uint8_t)v; + } + crypton_p256_from_bin(b, out); +} + +int main(void) { + crypton_p256_int n, x, y, x2, y2, r, a, b, n2tmp; + uint8_t buf[P256_NBYTES]; + + show("order", &crypton_SECP256r1_n); + show("prime", &crypton_SECP256r1_p); + show("bparam", &crypton_SECP256r1_b); + + /* the scalars worth naming: zero, one, the order and its neighbours, and + the all-bits-set shapes the comb recoding has to single out */ + static const char *corners[] = { + "0000000000000000000000000000000000000000000000000000000000000000", + "0000000000000000000000000000000000000000000000000000000000000001", + "0000000000000000000000000000000000000000000000000000000000000002", + "ffffffff00000000ffffffffffffffffbce6faada7179e84f3b9cac2fc632551", /* n */ + "ffffffff00000000ffffffffffffffffbce6faada7179e84f3b9cac2fc632550", /* n-1 */ + "ffffffff00000000ffffffffffffffffbce6faada7179e84f3b9cac2fc632552", /* n+1 */ + "7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "0000000000000000000000000000000000000000000000000000000000000000", + }; + for (unsigned i = 0; i < sizeof corners / sizeof *corners; i++) { + from_hex(corners[i], &n); + crypton_p256_base_point_mul(&n, &x, &y); + printf("corner%u valid=%d\n", i, crypton_p256_is_valid_point(&x, &y)); + show(" cx", &x); show(" cy", &y); + } + + for (int it = 0; it < 1024; it++) { + rnd_bytes(buf, P256_NBYTES); + crypton_p256_from_bin(buf, &n); + crypton_p256_mod(&crypton_SECP256r1_n, &n, &n); + show("n", &n); + + crypton_p256_base_point_mul(&n, &x, &y); + printf("valid=%d zero=%d odd=%d even=%d\n", + crypton_p256_is_valid_point(&x, &y), + crypton_p256_is_zero(&x), crypton_p256_is_odd(&y), + crypton_p256_is_even(&y)); + show("x", &x); show("y", &y); + + /* n2 * (that point), then n1*G + n2*P through the vartime pair */ + rnd_bytes(buf, P256_NBYTES); + crypton_p256_from_bin(buf, &a); + crypton_p256_mod(&crypton_SECP256r1_n, &a, &a); + crypton_p256e_point_mul(&a, &x, &y, &x2, &y2); + show("px", &x2); show("py", &y2); + printf("pvalid=%d\n", crypton_p256_is_valid_point(&x2, &y2)); + + rnd_bytes(buf, P256_NBYTES); + crypton_p256_from_bin(buf, &b); + crypton_p256_mod(&crypton_SECP256r1_n, &b, &b); + crypton_p256_points_mul_vartime(&a, &b, &x, &y, &x2, &y2); + show("vx", &x2); show("vy", &y2); + + /* the integer side: every arithmetic entry point the header offers */ + crypton_p256_modmul(&crypton_SECP256r1_n, &a, 0, &b, &r); show("mul", &r); + crypton_p256_modmul(&crypton_SECP256r1_p, &a, 1, &b, &r); show("mulc", &r); + crypton_p256e_scalar_invert(&a, &r); show("inv", &r); + crypton_p256e_modadd(&crypton_SECP256r1_n, &a, &b, &r); show("madd", &r); + crypton_p256e_modsub(&crypton_SECP256r1_n, &a, &b, &r); show("msub", &r); + crypton_p256e_point_add(&x, &y, &x2, &y2, &r, &n2tmp); show("ax", &r); show("ay", &n2tmp); + crypton_p256e_point_negate(&x, &y, &r, &n2tmp); show("gx", &r); show("gy", &n2tmp); + crypton_p256_modinv_vartime(&crypton_SECP256r1_n, &a, &r); show("invv", &r); + printf("cmp=%d add=%d sub=%d addd=%d\n", + crypton_p256_cmp(&a, &b), + crypton_p256_add(&a, &b, &r), + crypton_p256_sub(&a, &b, &r), + crypton_p256_add_d(&a, 0x9e3779b9u, &r)); + show("sum", &r); + /* the shifts are defined as n % P256_BITSPERDIGIT, which is 32 on one + build and 64 on the other, so keep the ask inside both */ + /* a shift of nothing has to be the number itself. Held here rather + than left to the random amounts below, because zero is the amount + that used to shift a digit by its own width. */ + crypton_p256_shl(&a, 0, &r); + printf("shl0=%d ", crypton_p256_cmp(&a, &r) == 0); + crypton_p256_shr(&a, 0, &r); + printf("shr0=%d\n", crypton_p256_cmp(&a, &r) == 0); + for (int s = 0; s < 3; s++) { + int k = (int)(rnd() % 32); + printf("shl%d=%d\n", k, (int)(crypton_p256_shl(&a, k, &r) & 0xff)); + show("shl", &r); + crypton_p256_shr(&a, k, &r); show("shr", &r); + } + for (int bit = 0; bit < 256; bit += 37) + printf("bit%d=%d ", bit, crypton_p256_get_bit(&a, bit)); + printf("\n"); + } + return 0; +} diff --git a/cbits/tests/width/run.sh b/cbits/tests/width/run.sh new file mode 100644 --- /dev/null +++ b/cbits/tests/width/run.sh @@ -0,0 +1,103 @@ +#!/bin/sh +# The three implementations that only a 32-bit architecture receives, asked +# the same questions as the 64-bit ones they stand in for. +# +# cbits/include32/p256 against cbits/include64/p256 +# cbits/curve25519-donna.c against curve25519-donna-c64.c +# cbits/decaf/p448/arch_32 against cbits/decaf/p448/arch_ref64 +# +# No job in the matrix is 32-bit, so until this ran, none of the left column +# had ever been compiled, let alone executed. The two sides of each pair +# define the same symbols, so they cannot share a binary: each driver is built +# twice and the two outputs compared. +# +# With a compiler that can target 32-bit x86 -- gcc-multilib on the Linux +# runner -- the 32-bit side is built a second time as a real 32-bit binary, +# which is the only way to see what the narrower int, size_t and pointer do. +# +# Usage: cbits/tests/width/run.sh [build-dir] +set -eu + +root=$(CDPATH= cd -- "$(dirname -- "$0")/../../.." && pwd) +out=${1:-$(mktemp -d)} +cc=${CC:-cc} +mkdir -p "$out" +cd "$root" + +D=cbits/decaf +decaf_common="$D/ed448goldilocks/decaf_all.c $D/ed448goldilocks/eddsa.c + $D/ed448goldilocks/scalar.c $D/p448/f_arithmetic.c + $D/p448/f_generic.c $D/utils.c cbits/crypton_sha3.c" + +# name, the flags and sources for the 64-bit side, then for the 32-bit side +build() { + # shellcheck disable=SC2086 + $cc -O2 -Wno-deprecated-declarations $EXTRA -o "$out/$1" $2 2>&1 | + grep -vE "^$|deprecated" || true + test -x "$out/$1" || { echo "did not build: $1"; exit 1; } +} + +status=0 +compare() { + if cmp -s "$out/$1.txt" "$out/$2.txt"; then + echo "ok $3 ($(wc -l < "$out/$1.txt" | tr -d ' ') lines)" + else + echo "FAIL $3" + diff "$out/$1.txt" "$out/$2.txt" | head -20 + status=1 + fi +} + +for width in 64 32; do + case $width in + 64) p256_inc=cbits/include64; donna=cbits/curve25519/curve25519-donna-c64.c; arch=arch_ref64 ;; + 32) p256_inc=cbits/include32; donna=cbits/curve25519/curve25519-donna.c; arch=arch_32 ;; + esac + + # decaf decides its word size twice and from two different things: the + # field limbs from ARCH_WORD_BITS, which follows the arch directory + # picked above, and the scalar limbs from CRYPTON_DECAF_WORD_BITS, which + # common.h reads off the host compiler. On a 32-bit machine -- the only + # place cabal asks for arch_32 -- both come out 32. Here the host is + # 64-bit whichever side is being built, so say which is wanted rather + # than compile a half-32-bit-half-64-bit library and compare that. + + EXTRA="-Icbits -I$p256_inc" + build "p256_$width" "cbits/tests/width/p256_width.c cbits/p256/p256.c cbits/p256/p256_ec.c" + + EXTRA="-Icbits" + build "x25519_$width" "cbits/tests/width/x25519_width.c $donna" + + EXTRA="-DCRYPTON_DECAF_WORD_BITS=$width -Icbits -I$D/include -I$D/p448 -I$D/include/$arch -I$D/p448/$arch" + build "ed448_$width" "cbits/tests/width/ed448_width.c $decaf_common $D/p448/$arch/f_impl.c" + + for t in p256 x25519 ed448; do + "$out/${t}_$width" > "$out/${t}_$width.txt" + done +done + +for t in p256 x25519 ed448; do + compare "${t}_64" "${t}_32" "$t: the 32-bit implementation answers what the 64-bit one answers" +done + +# The same sources again, this time actually narrow. Only the 32-bit side is +# tried: the 64-bit P-256 wants __uint128_t, which a 32-bit target has not got, +# which is the whole reason the 32-bit side exists. +printf 'int main(void){return 0;}\n' > "$out/probe.c" +if $cc -m32 -o "$out/probe" "$out/probe.c" 2>/dev/null && "$out/probe"; then + EXTRA="-m32 -Icbits -Icbits/include32" + build p256_m32 "cbits/tests/width/p256_width.c cbits/p256/p256.c cbits/p256/p256_ec.c" + EXTRA="-m32 -Icbits" + build x25519_m32 "cbits/tests/width/x25519_width.c cbits/curve25519/curve25519-donna.c" + EXTRA="-m32 -DCRYPTON_DECAF_WORD_BITS=32 -Icbits -I$D/include -I$D/p448 -I$D/include/arch_32 -I$D/p448/arch_32" + build ed448_m32 "cbits/tests/width/ed448_width.c $decaf_common $D/p448/arch_32/f_impl.c" + + for t in p256 x25519 ed448; do + "$out/${t}_m32" > "$out/${t}_m32.txt" + compare "${t}_32" "${t}_m32" "$t: a 32-bit host answers what a 64-bit host answers" + done +else + echo "skip $cc cannot build and run a 32-bit binary here; the comparison above still ran" +fi + +exit $status diff --git a/cbits/tests/width/x25519_width.c b/cbits/tests/width/x25519_width.c new file mode 100644 --- /dev/null +++ b/cbits/tests/width/x25519_width.c @@ -0,0 +1,74 @@ +/* The same X25519 asked of the 32-bit donna and the 64-bit donna. Both files + define crypton_curve25519_donna, so they cannot share a binary: build twice + and compare. */ +#include +#include +#include + +void crypton_curve25519_donna(uint8_t *mypublic, const uint8_t *secret, + const uint8_t *basepoint); + +static uint64_t s0 = 0x243f6a8885a308d3ULL, s1 = 0x13198a2e03707344ULL; +static uint64_t rnd(void) { + uint64_t x = s0, y = s1; + s0 = y; x ^= x << 23; + s1 = x ^ y ^ (x >> 17) ^ (y >> 26); + return s1 + y; +} +static void show(const char *t, const uint8_t *b) { + printf("%s ", t); + for (int i = 0; i < 32; i++) printf("%02x", b[i]); + printf("\n"); +} + +int main(void) { + uint8_t sec[32], base[32], out[32]; + /* the named points: the generator, zero, one, the low-order points and + the all-ones field element that reduces to nothing */ + static const uint8_t corners[][32] = { + {9}, + {0}, + {1}, + {0xe0,0xeb,0x7a,0x7c,0x3b,0x41,0xb8,0xae,0x16,0x56,0xe3,0xfa,0xf1,0x9f, + 0xc4,0x6a,0xda,0x09,0x8d,0xeb,0x9c,0x32,0xb1,0xfd,0x86,0x62,0x05,0x16, + 0x5f,0x49,0xb8,0x00}, + {0x5f,0x9c,0x95,0xbc,0xa3,0x50,0x8c,0x24,0xb1,0xd0,0xb1,0x55,0x9c,0x83, + 0xef,0x5b,0x04,0x44,0x5c,0xc4,0x58,0x1c,0x8e,0x86,0xd8,0x22,0x4e,0xdd, + 0xd0,0x9f,0x11,0x57}, + {0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff}, + {0xec,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff, + 0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0xff,0x7f}, + }; + for (unsigned c = 0; c < sizeof corners / sizeof *corners; c++) { + memset(sec, 0, 32); + sec[0] = (uint8_t)(0x40 + c); sec[31] = 0x40; + crypton_curve25519_donna(out, sec, corners[c]); + show("corner", out); + /* and the scalars with every clamped bit at an edge */ + memset(sec, 0xff, 32); sec[0] = 0xf8; sec[31] = 0x7f; + crypton_curve25519_donna(out, sec, corners[c]); + show("cmax", out); + memset(sec, 0x00, 32); sec[31] = 0x40; + crypton_curve25519_donna(out, sec, corners[c]); + show("cmin", out); + } + for (int it = 0; it < 2048; it++) { + for (int i = 0; i < 32; i++) sec[i] = (uint8_t)(rnd() >> 24); + for (int i = 0; i < 32; i++) base[i] = (uint8_t)(rnd() >> 24); + crypton_curve25519_donna(out, sec, base); + show("r", out); + /* and a round trip: the shared secret both sides should agree on */ + uint8_t pa[32], pb[32], sa[32], sb[32], g[32] = {9}; + uint8_t s2[32]; + for (int i = 0; i < 32; i++) s2[i] = (uint8_t)(rnd() >> 24); + crypton_curve25519_donna(pa, sec, g); + crypton_curve25519_donna(pb, s2, g); + crypton_curve25519_donna(sa, sec, pb); + crypton_curve25519_donna(sb, s2, pa); + printf("agree=%d\n", memcmp(sa, sb, 32) == 0); + show("sa", sa); + } + return 0; +} diff --git a/crypton.cabal b/crypton.cabal --- a/crypton.cabal +++ b/crypton.cabal @@ -1,9 +1,28 @@ cabal-version: 3.0 name: crypton -version: 2.0.1 -license: BSD-3-Clause -license-file: LICENSE -copyright: Vincent Hanquez +version: 2.2.5 +-- crypton's own code is BSD-3-Clause. The parts of +-- cbits/aes/gcm_fused_x86.c that follow picotls's fusion are MIT, and the +-- vendored s2n-bignum assembly in cbits/s2n is taken under ISC, and the +-- constant-time AES and GHASH in cbits/bearssl are BearSSL's under MIT; +-- each has its licence beside it, and they are listed below. The CRYPTOGAMS +-- assembly in cbits/asm is taken under its BSD-3-Clause option, and the +-- AArch64 multiply-accumulate loop in cbits/crypton_bignum.h follows Go's, +-- which is BSD-3-Clause too; the first term already covers both. +license: BSD-3-Clause AND MIT AND ISC +license-files: + LICENSE + cbits/LICENSE.go + cbits/aes/LICENSE.fusion + cbits/asm/LICENSE.cryptogams + cbits/s2n/LICENSE + cbits/bearssl/LICENSE + cbits/mlkem/LICENSE + cbits/mldsa/LICENSE +copyright: + 2006-2022 Vincent Hanquez and contributors, + 2023-2026 Kazu Yamamoto + maintainer: Kazu Yamamoto author: Vincent Hanquez stability: experimental @@ -12,43 +31,26 @@ homepage: https://github.com/kazu-yamamoto/crypton bug-reports: https://github.com/kazu-yamamoto/crypton/issues -synopsis: Cryptography Primitives sink -description: - A repository of cryptographic primitives. - . - * Symmetric ciphers: AES, DES, 3DES, CAST5, Blowfish, Twofish, Camellia, RC4, Salsa, XSalsa, ChaCha. - . - * Hash: SHA1, SHA2, SHA3, SHAKE, MD2, MD4, MD5, Keccak, Skein, Ripemd, Tiger, Whirlpool, Blake2 - . - * MAC: HMAC, KMAC, Poly1305 - . - * Asymmetric crypto: DSA, RSA, DH, ECDH, ECDSA, ECC, Curve25519, Curve448, Ed25519, Ed448 - . - * Key Derivation Function: PBKDF2, Scrypt, HKDF, Argon2, BCrypt, BCryptPBKDF - . - * Cryptographic Random generation: System Entropy, Deterministic Random Generator - . - * Data related: Anti-Forensic Information Splitter (AFIS) - . - If anything cryptographic related is missing from here, submit - a pull request to have it added. This package strives to be a - cryptographic kitchen sink that provides cryptography for everyone. - . - Evaluate the security related to your requirements before using. - . - Read "Crypto.Tutorial" for a quick start guide. +synopsis: Cryptography Primitives +description: `crypton` is a low-level cryptography library. + To achieve high performance, it utilizes C and assembly + language to define FFI bindings, structuring them + in a way that makes them easy to use. category: Cryptography build-type: Simple extra-source-files: cbits/*.h cbits/aes/*.h + cbits/curve25519/*.h cbits/aes/armv8_impl.c cbits/aes/x86ni_impl.c - cbits/asm/LICENSE.cryptogams cbits/asm/README.md cbits/asm/aesni-gcm-x86_64.pl cbits/asm/arm-xlate.pl + cbits/asm/aesp8-ppc.pl + cbits/asm/ghashp8-ppc.pl + cbits/asm/ppc-xlate.pl cbits/asm/arm_arch.h cbits/asm/chacha-armv8.pl cbits/asm/chacha-x86_64.pl @@ -67,6 +69,70 @@ cbits/blake2/ref/*.h cbits/blake2/sse/*.h cbits/crypton_hash_prefix.c + cbits/crypton_mont.py + cbits/crypton_powm_ifma.py + cbits/mlkem/*.S + cbits/mlkem/*.c + cbits/mlkem/*.h + cbits/mlkem/*.md + cbits/mlkem/*.sh + cbits/mlkem/COMMIT + cbits/mlkem/LICENSE + cbits/mlkem/src/*.c + cbits/mlkem/src/*.h + cbits/mlkem/src/*.inc + cbits/mlkem/src/fips202/*.c + cbits/mlkem/src/fips202/*.h + cbits/mlkem/src/fips202/native/*.h + cbits/mlkem/src/fips202/native/aarch64/*.h + cbits/mlkem/src/fips202/native/aarch64/src/*.S + cbits/mlkem/src/fips202/native/aarch64/src/*.c + cbits/mlkem/src/fips202/native/aarch64/src/*.h + cbits/mlkem/src/fips202/native/x86_64/*.h + cbits/mlkem/src/fips202/native/x86_64/src/*.S + cbits/mlkem/src/fips202/native/x86_64/src/*.c + cbits/mlkem/src/fips202/native/x86_64/src/*.h + cbits/mlkem/src/native/*.h + cbits/mlkem/src/native/aarch64/*.h + cbits/mlkem/src/native/aarch64/*.md + cbits/mlkem/src/native/aarch64/src/*.S + cbits/mlkem/src/native/aarch64/src/*.c + cbits/mlkem/src/native/aarch64/src/*.h + cbits/mlkem/src/native/x86_64/*.h + cbits/mlkem/src/native/x86_64/*.md + cbits/mlkem/src/native/x86_64/src/*.S + cbits/mlkem/src/native/x86_64/src/*.c + cbits/mlkem/src/native/x86_64/src/*.h + cbits/mldsa/*.S + cbits/mldsa/*.c + cbits/mldsa/*.h + cbits/mldsa/*.md + cbits/mldsa/*.sh + cbits/mldsa/COMMIT + cbits/mldsa/LICENSE + cbits/mldsa/src/*.c + cbits/mldsa/src/*.h + cbits/mldsa/src/*.inc + cbits/mldsa/src/fips202/*.c + cbits/mldsa/src/fips202/*.h + cbits/mldsa/src/fips202/native/*.h + cbits/mldsa/src/fips202/native/aarch64/*.h + cbits/mldsa/src/fips202/native/aarch64/src/*.S + cbits/mldsa/src/fips202/native/aarch64/src/*.c + cbits/mldsa/src/fips202/native/aarch64/src/*.h + cbits/mldsa/src/fips202/native/x86_64/*.h + cbits/mldsa/src/fips202/native/x86_64/src/*.S + cbits/mldsa/src/fips202/native/x86_64/src/*.c + cbits/mldsa/src/fips202/native/x86_64/src/*.h + cbits/mldsa/src/native/*.h + cbits/mldsa/src/native/aarch64/*.h + cbits/mldsa/src/native/aarch64/src/*.S + cbits/mldsa/src/native/aarch64/src/*.c + cbits/mldsa/src/native/aarch64/src/*.h + cbits/mldsa/src/native/x86_64/*.h + cbits/mldsa/src/native/x86_64/src/*.S + cbits/mldsa/src/native/x86_64/src/*.c + cbits/mldsa/src/native/x86_64/src/*.h cbits/decaf/ed448goldilocks/decaf.c cbits/decaf/ed448goldilocks/decaf_tables.c cbits/decaf/include/*.h @@ -79,7 +145,48 @@ cbits/ed25519/*.h cbits/include32/p256/*.h cbits/include64/p256/*.h + cbits/s2n/COMMIT + cbits/s2n/README.md + cbits/s2n/arm/*.S + cbits/p256/*.h + cbits/p256/gen_base_table.py + cbits/s2n/import.sh + cbits/s2n/include/*.h + cbits/s2n/x86_att/*.S + cbits/s2n/x86_att/*.h + cbits/bearssl/README.md + cbits/bearssl/VERSION + cbits/bearssl/import.sh + cbits/bearssl/inner.h + cbits/tests/*.c + cbits/tests/ct/*.c + cbits/tests/ct/*.h + cbits/tests/ct/known.txt + cbits/tests/ct/README + cbits/tests/ct/run.sh + cbits/tests/endian/*.c + cbits/tests/endian/README + cbits/tests/endian/run.sh + cbits/tests/endian/vectors.txt + cbits/tests/fuzz/*.c + cbits/tests/fuzz/*.h + cbits/tests/fuzz/README + cbits/tests/fuzz/corpus/*.bin + cbits/tests/fuzz/run.sh + cbits/tests/perf/*.c + cbits/tests/perf/floors.txt + cbits/tests/perf/run.sh + cbits/tests/scrub/*.c + cbits/tests/scrub/README + cbits/tests/scrub/known.txt + cbits/tests/scrub/run.sh + cbits/tests/sysdrg/*.c + cbits/tests/sysdrg/run.sh + cbits/tests/width/*.c + cbits/tests/width/run.sh tests/*.hs + tests/tutorial/extract.awk + tests/tutorial/run.sh extra-doc-files: CHANGELOG.md @@ -95,12 +202,6 @@ manual: True -flag support_rdrand - description: - allow compilation with RDRAND on system and architecture that supports it - - manual: True - flag support_pclmuldq description: Allow compilation with pclmuldq on architecture that supports it @@ -108,10 +209,22 @@ manual: True flag support_sse - description: Use SSE optimized version of (BLAKE2, ARGON2) + description: + Use SSE optimized version of (BLAKE2, ARGON2) on i386. x86-64 takes + them anyway; every other architecture has no SSE to offer and ignores + this flag, rather than failing to compile the sources. + default: False manual: True +flag support_s2n_bignum + description: + Use the vendored s2n-bignum assembly for the NIST prime curves on + x86-64 and AArch64. See cbits/s2n/README.md. + + default: True + manual: True + flag integer-gmp description: Whether or not to use GMP for some functions manual: True @@ -143,11 +256,13 @@ library exposed-modules: Crypto.Cipher.AES + Crypto.Cipher.AES.GCM Crypto.Cipher.AESGCMSIV Crypto.Cipher.Blowfish Crypto.Cipher.Camellia Crypto.Cipher.CAST5 Crypto.Cipher.ChaCha + Crypto.Cipher.ChaCha.Poly1305 Crypto.Cipher.ChaChaPoly1305 Crypto.Cipher.DES Crypto.Cipher.RC4 @@ -167,6 +282,7 @@ Crypto.Hash Crypto.Hash.Algorithms Crypto.Hash.IO + Crypto.KEM Crypto.KDF.Argon2 Crypto.KDF.BCrypt Crypto.KDF.BCryptPBKDF @@ -205,6 +321,8 @@ Crypto.PubKey.Ed25519 Crypto.PubKey.Ed448 Crypto.PubKey.EdDSA + Crypto.PubKey.MLKEM + Crypto.PubKey.MLDSA Crypto.PubKey.MaskGenFunction Crypto.PubKey.Rabin.Basic Crypto.PubKey.Rabin.Modified @@ -212,6 +330,7 @@ Crypto.PubKey.Rabin.RW Crypto.PubKey.Rabin.Types Crypto.PubKey.RSA + Crypto.PubKey.RSA.Blinded Crypto.PubKey.RSA.OAEP Crypto.PubKey.RSA.PKCS15 Crypto.PubKey.RSA.Prim @@ -236,6 +355,8 @@ -- limb counts, which are public. cc-options: -std=gnu99 -O3 c-sources: + cbits/curve25519/x25519.c + cbits/crypton_modinv.c cbits/argon2/argon2.c cbits/crypton_blake2b.c cbits/crypton_blake2bp.c @@ -244,7 +365,19 @@ cbits/crypton_blowfish.c cbits/crypton_camellia.c cbits/crypton_chacha.c + cbits/crypton_chachapoly.c cbits/crypton_cpu.c + cbits/crypton_sysdrg.c + cbits/crypton_sysrandom.c + -- BearSSL's constant-time AES and GHASH, which cbits/aes/generic.c + -- and cbits/aes/gf.c are written on top of. Unconditional because + -- those two are: every one of the three AES branches below names + -- them, accelerated or not. + cbits/bearssl/aes_ct64.c + cbits/bearssl/aes_ct64_dec.c + cbits/bearssl/aes_ct64_enc.c + cbits/bearssl/dec32le.c + cbits/bearssl/ghash_ctmul64.c cbits/crypton_des.c cbits/crypton_ecc.c cbits/crypton_f2m.c @@ -255,6 +388,7 @@ cbits/crypton_pbkdf2.c cbits/crypton_poly1305.c cbits/crypton_powm.c + cbits/crypton_powm_ifma.c cbits/crypton_rc4.c cbits/crypton_ripemd.c cbits/crypton_salsa.c @@ -269,10 +403,13 @@ cbits/crypton_whirlpool.c cbits/crypton_xsalsa.c cbits/ed25519/ed25519.c + cbits/ed25519/ed25519_s2n.c cbits/p256/p256.c cbits/p256/p256_ec.c other-modules: + Crypto.Number.ModArithmetic.Pub + Crypto.PubKey.RSA.Blinded.Internal Crypto.Cipher.AES.Primitive Crypto.Cipher.Blowfish.Primitive Crypto.Cipher.Camellia.Primitive @@ -319,6 +456,7 @@ Crypto.Internal.ECC Crypto.Internal.Endian Crypto.Internal.Imports + Crypto.Internal.Poly1305 Crypto.Internal.Nat Crypto.Internal.WordArray Crypto.Internal.Words @@ -329,6 +467,7 @@ Crypto.Random.Entropy.Source Crypto.Random.HmacDRG Crypto.Random.Probabilistic + Crypto.Random.SysDRG Crypto.Random.SystemDRG default-language: Haskell2010 @@ -339,12 +478,11 @@ ghc-options: -Wall -fwarn-tabs -optc-O3 build-depends: base >=4.13 && <5, - bytestring, - primitive >=0.9, - deepseq, - base16 >=1.0, - bytestring, - text, + bytestring <0.13, + primitive >=0.9 && <0.10, + deepseq <1.6, + base16 >=1.0 && <1.1, + text <2.2, ram >=0.20.1 && <0.23 if flag(old_toolchain_inliner) @@ -368,6 +506,17 @@ include-dirs: cbits/decaf/include/arch_ref64 cbits/decaf/p448/arch_ref64 + -- decaf sizes its field limbs from the arch directory just chosen and + -- its scalar limbs from a separate macro that cbits/decaf/include/ + -- decaf/common.h works out from the compiler, by asking after + -- __x86_64__ and the width of uint_fast32_t. The two answers need + -- not agree: on Apple Silicon uint_fast32_t is four bytes, so the + -- same aarch64 CPU takes 64-bit field limbs and 32-bit scalar limbs, + -- where on Linux it takes 64-bit for both. Ed448 signing pays 5.5% + -- for that (20.69 us against 19.56 on an M4). The architecture is + -- already decided here, so decide this with it. + cc-options: -DCRYPTON_DECAF_WORD_BITS=64 + else c-sources: cbits/decaf/ed448goldilocks/decaf_all.c @@ -379,6 +528,7 @@ cbits/decaf/utils.c include-dirs: cbits/decaf/include/arch_32 cbits/decaf/p448/arch_32 + cc-options: -DCRYPTON_DECAF_WORD_BITS=32 if ((((((((arch(x86_64) || arch(aarch64)) || arch(loongarch64)) || arch(ppc64le)) || arch(riscv64)) || arch(s390x)) || arch(alpha)) || arch(ppc64)) || arch(sparc64)) c-sources: cbits/curve25519/curve25519-donna-c64.c @@ -447,11 +597,180 @@ cbits/asm/sha256-x86_64-elf.S cbits/asm/sha512-x86_64-elf.S - if ((flag(support_rdrand) && (arch(i386) || arch(x86_64))) && !os(windows)) - cpp-options: -DSUPPORT_RDRAND - c-sources: cbits/crypton_rdrand.c - other-modules: Crypto.Random.Entropy.RDRand + -- AWS's s2n-bignum: hand-written, formally verified, constant-time + -- assembly for the NIST prime curves, two and a half to three times + -- faster than the C it replaces here. Apache-2.0 OR ISC OR MIT-0, so + -- unlike OpenSSL's and BoringSSL's ecp_nistz256 it can be used. Both + -- variants of each routine are built and chosen between in + -- cbits/p256/p256_s2n.c -- see cbits/s2n/README.md for why the question + -- is a different one on the two architectures. + -- + -- On Windows, x86-64 only. The vendored files carry the ELF directives + -- .type and .size, which neither of Windows' assemblers accepts, and + -- assume the System V calling convention unless WINDOWS_ABI is set; + -- cbits/s2n/x86_att/_internal_s2n_bignum_x86_att.h, which every one of + -- them reaches first, empties the first and sets the second on _WIN32, + -- so the vendored files themselves stay as upstream wrote them. The + -- AArch64 files have the same directives and no such header yet, and no + -- runner to check one, so Windows on ARM keeps the C. + if flag(support_s2n_bignum) && (arch(x86_64) || (arch(aarch64) && !os(windows))) + cc-options: -DCRYPTON_S2N_BIGNUM + include-dirs: cbits/s2n/include + c-sources: + cbits/crypton_ecc_s2n.c + cbits/p256/p256_base_table.c + cbits/p256/p256_s2n.c + cbits/p256/p256_verify.c + cbits/p256/p256_wnaf_table.c + if arch(aarch64) + asm-sources: + cbits/s2n/arm/bignum_deamont_p384.S + cbits/s2n/arm/bignum_montifier.S + cbits/s2n/arm/bignum_modinv.S + cbits/s2n/arm/curve25519_x25519.S + cbits/s2n/arm/curve25519_x25519_alt.S + cbits/s2n/arm/curve25519_x25519base.S + cbits/s2n/arm/curve25519_x25519base_alt.S + cbits/s2n/arm/edwards25519_encode.S + cbits/s2n/arm/edwards25519_scalarmulbase.S + cbits/s2n/arm/edwards25519_scalarmulbase_alt.S + cbits/s2n/arm/bignum_inv_p521.S + cbits/s2n/arm/bignum_montinv_p384.S + cbits/s2n/arm/bignum_montmul_p384.S + cbits/s2n/arm/bignum_montmul_p384_alt.S + cbits/s2n/arm/bignum_montsqr_p384.S + cbits/s2n/arm/bignum_montsqr_p384_alt.S + cbits/s2n/arm/bignum_mul_p521.S + cbits/s2n/arm/bignum_mul_p521_alt.S + cbits/s2n/arm/bignum_sqr_p521.S + cbits/s2n/arm/bignum_sqr_p521_alt.S + cbits/s2n/arm/bignum_tomont_p384.S + cbits/s2n/arm/bignum_demont_p256.S + cbits/s2n/arm/bignum_neg_p256.S + cbits/s2n/arm/bignum_tomont_p256.S + cbits/s2n/arm/p256_montjadd.S + cbits/s2n/arm/p256_montjadd_alt.S + cbits/s2n/arm/p256_montjdouble.S + cbits/s2n/arm/p256_montjdouble_alt.S + cbits/s2n/arm/p256_montjmixadd.S + cbits/s2n/arm/p256_montjmixadd_alt.S + cbits/s2n/arm/p256_scalarmul.S + cbits/s2n/arm/p256_scalarmul_alt.S + cbits/s2n/arm/p256_scalarmulbase.S + cbits/s2n/arm/p256_scalarmulbase_alt.S + cbits/s2n/arm/p384_montjscalarmul.S + cbits/s2n/arm/p384_montjscalarmul_alt.S + cbits/s2n/arm/p521_jscalarmul.S + cbits/s2n/arm/p521_jscalarmul_alt.S + + else + asm-sources: + cbits/s2n/x86_att/bignum_deamont_p384.S + cbits/s2n/x86_att/bignum_modinv.S + cbits/s2n/x86_att/curve25519_x25519.S + cbits/s2n/x86_att/curve25519_x25519_alt.S + cbits/s2n/x86_att/curve25519_x25519base.S + cbits/s2n/x86_att/curve25519_x25519base_alt.S + cbits/s2n/x86_att/edwards25519_encode.S + cbits/s2n/x86_att/edwards25519_scalarmulbase.S + cbits/s2n/x86_att/edwards25519_scalarmulbase_alt.S + cbits/s2n/x86_att/bignum_deamont_p384_alt.S + cbits/s2n/x86_att/bignum_inv_p521.S + cbits/s2n/x86_att/bignum_montinv_p384.S + cbits/s2n/x86_att/bignum_montmul_p384.S + cbits/s2n/x86_att/bignum_montmul_p384_alt.S + cbits/s2n/x86_att/bignum_montsqr_p384.S + cbits/s2n/x86_att/bignum_montsqr_p384_alt.S + cbits/s2n/x86_att/bignum_mul_p521.S + cbits/s2n/x86_att/bignum_mul_p521_alt.S + cbits/s2n/x86_att/bignum_sqr_p521.S + cbits/s2n/x86_att/bignum_sqr_p521_alt.S + cbits/s2n/x86_att/bignum_tomont_p384.S + cbits/s2n/x86_att/bignum_tomont_p384_alt.S + cbits/s2n/x86_att/bignum_demont_p256.S + cbits/s2n/x86_att/bignum_demont_p256_alt.S + cbits/s2n/x86_att/bignum_neg_p256.S + cbits/s2n/x86_att/bignum_tomont_p256.S + cbits/s2n/x86_att/bignum_tomont_p256_alt.S + cbits/s2n/x86_att/p256_montjadd.S + cbits/s2n/x86_att/p256_montjadd_alt.S + cbits/s2n/x86_att/p256_montjdouble.S + cbits/s2n/x86_att/p256_montjdouble_alt.S + cbits/s2n/x86_att/p256_montjmixadd.S + cbits/s2n/x86_att/p256_montjmixadd_alt.S + cbits/s2n/x86_att/p256_scalarmul.S + cbits/s2n/x86_att/p256_scalarmul_alt.S + cbits/s2n/x86_att/p256_scalarmulbase.S + cbits/s2n/x86_att/p256_scalarmulbase_alt.S + + asm-sources: + cbits/s2n/x86_att/bignum_montifier.S + cbits/s2n/x86_att/p384_montjscalarmul.S + cbits/s2n/x86_att/p384_montjscalarmul_alt.S + cbits/s2n/x86_att/p521_jscalarmul.S + cbits/s2n/x86_att/p521_jscalarmul_alt.S + + -- Modular exponentiation at RSA sizes, x86-64 only: on AArch64 + -- crypton's C measures faster than these, so nothing is taken + -- for it. All of them want ADX, which the run-time check in + -- cbits/crypton_powm.c asks about before using any. + -- Not on macOS: Apple's assembler reads $0 in bignum_emontredc_8n's + -- macros as a macro argument and rejects the file. + if !os(osx) + asm-sources: + cbits/s2n/x86_att/bignum_emontredc_8n.S + cbits/s2n/x86_att/bignum_mul.S + cbits/s2n/x86_att/bignum_sqr.S + cbits/s2n/x86_att/bignum_kmul_16_32.S + cbits/s2n/x86_att/bignum_kmul_32_64.S + cbits/s2n/x86_att/bignum_ksqr_16_32.S + cbits/s2n/x86_att/bignum_ksqr_32_64.S + cbits/s2n/x86_att/bignum_sqr_8_16.S + cbits/s2n/x86_att/bignum_mul_8_16.S + + -- The PQ Code Package's mlkem-native and mldsa-native, vendored under + -- cbits/mlkem and cbits/mldsa. Apache-2.0 OR ISC OR MIT, the same + -- three-way form as s2n-bignum above, and by the same authors, so + -- crypton takes them on the same terms. The portable C is built + -- everywhere; the hand-written backends are added below where the + -- architecture has them. + -- + -- Each wrapper includes its amalgamation once per parameter set, which + -- is how one build offers all three. See cbits/mlkem/crypton_mlkem.c. + c-sources: + cbits/mlkem/crypton_mlkem.c + cbits/mldsa/crypton_mldsa.c + include-dirs: + cbits/mlkem + cbits/mldsa + + -- Windows is left out of the backends for the reason s2n-bignum is: + -- the x86-64 assembly is written for the SysV ABI -- upstream guards + -- its own backend header on that -- and the AArch64 files carry ELF and + -- Mach-O directives and nothing for COFF. Windows gets the portable C, + -- which is the same code every other architecture gets. + if ((arch(x86_64) || arch(aarch64)) && !os(windows)) + cc-options: + -DCRYPTON_MLKEM_NATIVE_BACKEND -DCRYPTON_MLDSA_NATIVE_BACKEND + asm-options: + -DCRYPTON_MLKEM_NATIVE_BACKEND -DCRYPTON_MLDSA_NATIVE_BACKEND + asm-sources: + cbits/mlkem/crypton_mlkem_asm.S + cbits/mldsa/crypton_mldsa_asm.S + + -- What the two packages' own build systems pass for the x86-64 + -- backend, and only for the assembly: cc-options in a conditional + -- are the component's, not the block's, so -mavx2 here would raise + -- the baseline of every C file in the library -- Ed25519, Argon2, + -- P-256 and the rest -- and gcc would vectorise code that asks for + -- no such thing, with nothing to check at runtime. The backends' + -- own AVX2 is all in the .S. AArch64 needs nothing beyond the + -- baseline for the arithmetic, and takes the armv8.4 Keccak from + -- the -march the block below already sets. + if arch(x86_64) + asm-options: -mavx2 -mbmi2 + if (flag(support_aesni) && arch(aarch64)) cc-options: -DWITH_ARMV8_CRYPTO c-sources: @@ -463,6 +782,52 @@ if !flag(use_target_attributes) cc-options: -march=armv8-a+crypto + -- The same instructions, reached from AArch32. cbits/aes/armv8.c needs + -- nothing changed to compile here: every intrinsic it uses exists in both + -- execution states, including the PMULL2 it takes through + -- vmull_high_p64, which the compiler lowers to a pair of VMULL.P64. What + -- differs is how the operating system reports the instructions -- + -- AArch32 has filled AT_HWCAP with older features and puts these in + -- AT_HWCAP2 -- which cbits/crypton_cpu.c now asks for. + -- + -- -mfpu is passed whatever use_target_attributes says, unlike the branch + -- above. AArch32 names an FPU rather than an architecture extension and + -- the two compilers spell the function attribute for it differently; a + -- global option is safe here, since a compiler emits these instructions + -- where an intrinsic asks for them and nowhere else. + -- + -- A soft-float ARM that cannot take -mfpu fails loudly at the first C + -- file rather than quietly, and -f-support_aesni is the way out. + if (flag(support_aesni) && arch(arm)) + cc-options: -DWITH_ARMV8_CRYPTO -mfpu=crypto-neon-fp-armv8 + c-sources: + cbits/aes/generic.c + cbits/aes/gf.c + cbits/aes/armv8.c + cbits/crypton_aes.c + + -- POWER8's vector AES and vector carry-less multiply, through the + -- CRYPTOGAMS assembly in cbits/asm. No cc-options beyond the define: + -- cbits/aes/ppc8.c calls that assembly rather than using intrinsics of + -- its own, and the assembly says .machine "any", so the assembler takes + -- the instructions without being told about the processor. + -- + -- Little-endian only. The generator emits a big-endian flavour from the + -- same source, and it is not checked in: nothing here has been able to + -- run it, and an AES path that has never been executed is not one to + -- ship. cbits/asm/generate.sh says what adding it would take. + if (flag(support_aesni) && arch(ppc64le)) + cc-options: -DWITH_PPC8_CRYPTO + c-sources: + cbits/aes/generic.c + cbits/aes/gf.c + cbits/aes/ppc8.c + cbits/crypton_aes.c + + asm-sources: + cbits/asm/aesp8-ppc-linux64le.S + cbits/asm/ghashp8-ppc-linux64le.S + if arch(aarch64) cc-options: -DWITH_ARMV8_SHA1 -DWITH_ARMV8_SHA2 -DWITH_ARMV8_SHA3 @@ -516,7 +881,18 @@ cbits/asm/sha1-armv8-linux64.S cbits/asm/sha256-armv8-linux64.S - if ((flag(support_aesni) && (((os(linux) || os(freebsd)) || os(osx)) || os(windows))) && (arch(i386) || arch(x86_64))) + -- No operating system condition. The processor is asked with cpuid, + -- which no kernel is involved in, and crypton_cpu.h turns USE_AESNI on + -- from __i386__ or __x86_64__ alone -- so nothing here was ever decided + -- by which system it is. The list that used to stand here named linux, + -- freebsd and osx when it arrived from cipher-aes in 2014 and gained + -- windows later, each time because someone wanted that one system; it + -- was never a statement that the rest could not do this. What it did + -- instead was send OpenBSD, NetBSD, DragonFly, Solaris and illumos down + -- the table-driven fallback on processors that have had AES-NI since + -- 2010. The object format still has a say further down, where the GCM + -- assembly is chosen, and there ELF is the default. + if (flag(support_aesni) && (arch(i386) || arch(x86_64))) cc-options: -DWITH_AESNI c-sources: cbits/aes/generic.c @@ -543,8 +919,25 @@ -- checked in per object format since it comes from a -- generator. if arch(x86_64) - cc-options: -DWITH_X86_GCM_ASM - c-sources: cbits/aes/gcm_x86_asm.c + cc-options: -DWITH_X86_GCM_ASM -DWITH_GCM_FUSED + c-sources: + cbits/aes/gcm_fused_x86.c + -- AES-GCM in the 256-bit form of the same two + -- instructions, two blocks to each, which is worth + -- twice the throughput where the processor has them -- + -- Zen 3 and Ice Lake onwards. Nothing to borrow: the + -- wide AES-GCM in OpenSSL and BoringSSL is Apache-2.0, + -- s2n-bignum has no GCM, and the assembly above is + -- 128-bit throughout. crypton_cpu.c asks the + -- processor before any of it runs. + cbits/aes/gcm_vaes_x86.c + -- and the 512-bit form, four blocks to each, which Ice + -- Lake and Zen 4 onwards have. Same story about + -- borrowing, and the same run-time question -- with + -- three more bits of XCR0 in it, since these need the + -- operating system to save the AVX-512 state. + cbits/aes/gcm_vaes512_x86.c + cbits/aes/gcm_x86_asm.c if os(osx) asm-sources: cbits/asm/aesni-gcm-x86_64-macosx.S @@ -555,13 +948,18 @@ else asm-sources: cbits/asm/aesni-gcm-x86_64-elf.S - else + -- Neither of the two branches above. This was an `else`, which pairs with + -- the x86 `if` alone and so fired on AArch64 as well, where the ARMv8 + -- branch had already named every file it names. Cabal drops the repeats, + -- so nothing was built twice, but the line read as the fallback for a + -- platform with no AES instructions and was not one. + if !((flag(support_aesni) && (arch(aarch64) || arch(arm))) || (flag(support_aesni) && (arch(i386) || arch(x86_64))) || (flag(support_aesni) && arch(ppc64le))) c-sources: cbits/aes/generic.c cbits/aes/gf.c cbits/crypton_aes.c - if (arch(x86_64) || flag(support_sse)) + if (arch(x86_64) || (flag(support_sse) && arch(i386))) c-sources: cbits/blake2/sse/blake2b.c cbits/blake2/sse/blake2bp.c @@ -579,7 +977,7 @@ include-dirs: cbits/blake2/ref - if (arch(x86_64) || flag(support_sse)) + if (arch(x86_64) || (flag(support_sse) && arch(i386))) cpp-options: -DSUPPORT_SSE if arch(i386) @@ -589,13 +987,15 @@ cpp-options: -DWINDOWS other-modules: Crypto.Random.Entropy.Windows extra-libraries: advapi32 - build-depends: Win32 + build-depends: Win32 <2.15 else - other-modules: Crypto.Random.Entropy.Unix + other-modules: + Crypto.Random.Entropy.SysRandom + Crypto.Random.Entropy.Unix if (impl(ghc >=0) && flag(integer-gmp)) - build-depends: integer-gmp + build-depends: integer-gmp <1.2 if flag(support_deepseq) cpp-options: -DWITH_DEEPSEQ_SUPPORT @@ -665,6 +1065,10 @@ PubKey.DSASpec PubKey.ECCSpec PubKey.ECDSASpec + PubKey.MLKEMSpec + PubKey.MLKEMVectors + PubKey.MLDSASpec + PubKey.MLDSAVectors PubKey.ElGamalSpec PubKey.MGF1Spec PubKey.OAEPSpec @@ -673,7 +1077,9 @@ PubKey.RabinSpec PubKey.SecrecySpec PubKey.RSASpec + EntropySpec RuntimeSpec + SysDRGSpec StreamCipher.ChaChaPoly1305Spec StreamCipher.ChaChaSpec StreamCipher.RC4Spec @@ -694,6 +1100,45 @@ default-language: Haskell2010 ghc-options: -Wall -fno-warn-orphans -fno-warn-missing-signatures -rtsopts + -threaded "-with-rtsopts=-N2" + +-- The generator behind MonadRandom and a fork made from Haskell. Its own +-- executables because the suite above runs with -N2, where GHC does not +-- support forkProcess, and because the answer may differ between the +-- threaded runtime and the one without it -- so it is built for both. +-- Not on Windows, which has no fork and no unix package. +test-suite test-forkprocess-threaded + type: exitcode-stdio-1.0 + main-is: ForkProcess.hs + hs-source-dirs: tests/forkprocess + default-language: Haskell2010 + ghc-options: + -Wall -rtsopts -threaded "-with-rtsopts=-N1" + + if os(windows) + buildable: False + else + build-depends: + base >=4.13 && <5, + bytestring, + crypton, + unix + +test-suite test-forkprocess-unthreaded + type: exitcode-stdio-1.0 + main-is: ForkProcess.hs + hs-source-dirs: tests/forkprocess + default-language: Haskell2010 + ghc-options: -Wall -rtsopts + + if os(windows) + buildable: False + else + build-depends: + base >=4.13 && <5, + bytestring, + crypton, + unix benchmark bench-crypton type: exitcode-stdio-1.0 diff --git a/tests/BlockCipher/AESSpec.hs b/tests/BlockCipher/AESSpec.hs --- a/tests/BlockCipher/AESSpec.hs +++ b/tests/BlockCipher/AESSpec.hs @@ -3,6 +3,7 @@ module BlockCipher.AESSpec (spec) where import BlockCipher +import Control.Exception (evaluate) import qualified Crypto.Cipher.AES as AES import Crypto.Cipher.Types import Crypto.Error @@ -20,6 +21,10 @@ import qualified BlockCipher.AES.GCMLong as KATGCMLong import qualified BlockCipher.AES.OCB3 as KATOCB3 import qualified BlockCipher.AES.XTS as KATXTS +import qualified Crypto.Cipher.AES.GCM as GCM +import Data.Bits (xor) +import Foreign.Marshal.Alloc (allocaBytes) +import Foreign.Ptr (castPtr) {- instance Show AES.AES where @@ -200,8 +205,8 @@ ++ "-byte AAD, " ++ show ptlen ++ "-byte message" - ) $ - case klen of + ) + $ case klen of 16 -> run (undefined :: AES.AES128) v 24 -> run (undefined :: AES.AES192) v _ -> run (undefined :: AES.AES256) v @@ -215,13 +220,172 @@ digest ciphertext `shouldBe` ctHash aeadSimpleDecrypt aead aad ciphertext authTag `shouldBe` Just plaintext where - cipher = throwCryptoError (cipherInit (KATGCMLong.gcmKey klen)) `asTypeOf` cipherWitness + cipher = + throwCryptoError (cipherInit (KATGCMLong.gcmKey klen)) `asTypeOf` cipherWitness aead = throwCryptoError (aeadInit AEAD_GCM cipher KATGCMLong.gcmIV) aad = KATGCMLong.gcmAAD aadlen plaintext = KATGCMLong.gcmPlaintext ptlen (authTag, ciphertext) = aeadSimpleEncrypt aead aad plaintext 16 digest bs = BA.convert (hash bs :: Digest SHA256) :: ByteString +-- | Crypto.Cipher.AES.GCM builds the key part of the state once and does a +-- whole message in one call. It has to answer exactly what the general +-- interface answers, so it is run over the same vectors, and a tampered +-- message has to come back as Nothing rather than as plaintext. +oneShotTests :: Spec +oneShotTests = describe "Crypto.Cipher.AES.GCM" $ do + describe "agrees with the general interface" $ do + run "AES-128" KATGCM.vectors_aes128_enc + run "AES-192" KATGCM.vectors_aes192_enc + run "AES-256" KATGCM.vectors_aes256_enc + describe "decryptWithTag hands back the tag encrypt made" $ do + runTag "AES-128" KATGCM.vectors_aes128_enc + runTag "AES-192" KATGCM.vectors_aes192_enc + runTag "AES-256" KATGCM.vectors_aes256_enc + it "decryptWithTag gives a different tag for a tampered ciphertext" $ + let ctx = ctx16 + sealed = GCM.encrypt ctx iv16 B.empty message 16 :: B.ByteString + body = B.take (B.length sealed - 16) sealed + tag = AuthTag (BA.convert (B.drop (B.length sealed - 16) sealed)) + (_, tag') = + GCM.decryptWithTag ctx iv16 B.empty (flipFirst body) 16 + :: (B.ByteString, AuthTag) + in tag' `shouldSatisfy` (/= tag) + describe "refuses a message that was interfered with" $ do + it "a flipped bit in the tag" $ tamper (\(c, t) -> (c, flipFirst t)) + it "a flipped bit in the ciphertext" $ tamper (\(c, t) -> (flipFirst c, t)) + it "refuses input shorter than the tag" $ + (GCM.decrypt ctx16 iv16 B.empty (B.replicate 8 0) 16 :: Maybe B.ByteString) + `shouldBe` Nothing + it "refuses a ciphertext with no authentication tag" $ + let sealed = GCM.encrypt ctx16 iv16 header message 16 :: B.ByteString + body = B.take (B.length sealed - 16) sealed + in (GCM.decrypt ctx16 iv16 header body 0 :: Maybe B.ByteString) + `shouldBe` Nothing + it "does not authenticate an altered ciphertext with a zero-length tag" $ + let sealed = GCM.encrypt ctx16 iv16 header message 16 :: B.ByteString + body = B.take (B.length sealed - 16) sealed + in (GCM.decrypt ctx16 iv16 header (flipFirst body) 0 :: Maybe B.ByteString) + `shouldBe` Nothing + -- Shorter than four bytes the tag authenticates next to nothing, and + -- longer than sixteen there is no more tag for GCM to give. + describe "refuses a tag length outside 4 to 16 bytes" $ + forM_ [0, 3, 17 :: Int] $ \taglen -> describe (show taglen) $ do + it "encrypt" $ + evaluate (GCM.encrypt ctx16 iv16 header message taglen :: B.ByteString) + `shouldThrow` (== CryptoError_AuthenticationTagSizeInvalid) + it "decrypt" $ + (GCM.decrypt ctx16 iv16 header plainSealed taglen :: Maybe B.ByteString) + `shouldBe` Nothing + it "decryptWithTag" $ + evaluate + ( GCM.decryptWithTag ctx16 iv16 header message taglen + :: (B.ByteString, AuthTag) + ) + `shouldThrow` (== CryptoError_AuthenticationTagSizeInvalid) + it "encryptWithMask" $ + withMaskTag taglen 0 `shouldReturn` Nothing + -- SP 800-38D 5.2.1.1 wants at least one byte of IV. With none, GCM's + -- pre-counter block is zero and the tag of a message is + -- GHASH_H(A, C) XOR E(K, 0^128) -- and E(K, 0^128) is the GHASH key H + -- itself. One full tag therefore gives H away, H belongs to the key + -- rather than to the nonce, and with it a tag can be forged for any + -- message under any nonce the key has been used with, twelve-byte ones + -- included. The general interface has refused the empty IV since + -- f98cff3 and these four did not. + describe "refuses a nonce of no bytes" $ do + it "encrypt" $ + evaluate (GCM.encrypt ctx16 B.empty header message 16 :: B.ByteString) + `shouldThrow` (== CryptoError_IvSizeInvalid) + -- a message that really is sealed under the empty nonce, so that this + -- says something whether or not `encrypt` refuses one: before the + -- check, `decrypt` returned the sixteen bytes of plaintext for it + it "decrypt, on a message that is genuine under it" $ + (GCM.decrypt ctx16 B.empty header emptyNonceSealed 16 :: Maybe B.ByteString) + `shouldBe` Nothing + it "decryptWithTag" $ + evaluate + ( GCM.decryptWithTag ctx16 B.empty header message 16 + :: (B.ByteString, AuthTag) + ) + `shouldThrow` (== CryptoError_IvSizeInvalid) + it "encryptWithMask, writing nothing" $ + withMaskIv B.empty 16 4 `shouldReturn` Nothing + -- and only the empty one: SP 800-38D allows every length from one byte up, + -- so this must not become a check for twelve + it "takes a nonce of one byte" $ + let iv1 = B.singleton 0x77 + sealed = GCM.encrypt ctx16 iv1 header message 16 :: B.ByteString + in (GCM.decrypt ctx16 iv1 header sealed 16 :: Maybe B.ByteString) + `shouldBe` Just message + describe "header protection" $ do + it "writes the ciphertext encrypt gives" $ + withMask 4 `shouldReturn` Just (plainSealed, expectedMask 4) + it "and at another offset" $ + withMask 0 `shouldReturn` Just (plainSealed, expectedMask 0) + it "refuses a sample that does not fit, writing nothing" $ do + withMask (B.length plainSealed - 15) `shouldReturn` Nothing + withMask (-1) `shouldReturn` Nothing + where + run name vs = + it name $ + [ (key, iv) + | (key, iv, aad, input, out, taglen, tag) <- vs + , let ctx = throwCryptoError (GCM.newContext key) + , let sealed = GCM.encrypt ctx iv aad input taglen :: B.ByteString + , sealed /= out `B.append` tag + || GCM.decrypt ctx iv aad sealed taglen /= Just input + ] + `shouldBe` [] + -- The tag decryptWithTag computes has to be the one encrypt appended, and + -- the body it returns the one decrypt returns, over the same vectors. + runTag name vs = + it name $ + [ (key, iv) + | (key, iv, aad, input, out, taglen, tag) <- vs + , let ctx = throwCryptoError (GCM.newContext key) + , let (body, tag') = + GCM.decryptWithTag ctx iv aad out taglen + :: (B.ByteString, AuthTag) + , body /= input || tag' /= AuthTag (BA.convert tag) + ] + `shouldBe` [] + ctx16 = throwCryptoError (GCM.newContext (B.replicate 16 0x2b)) + iv16 = B.replicate 12 0x77 + -- header protection keeps a key of its own, as QUIC does + hpKeyBytes = B.replicate 16 0x9c + hpKey = throwCryptoError (GCM.newHeaderKey hpKeyBytes) + hpAes = throwCryptoError (cipherInit hpKeyBytes) :: AES.AES128 + message = "a packet payload" :: B.ByteString + header = "\x40\x01\x02\x03" :: B.ByteString + plainSealed = GCM.encrypt ctx16 iv16 header message 16 :: B.ByteString + -- ctx16, no nonce at all, `header` as the additional data and `message` + -- as the plaintext, taken from the tree before the check went in + emptyNonceSealed = + "\x09\x25\xea\x53\x9b\x98\xf1\x0e\x02\x5f\x8a\x70\x8d\xf4\x6d\xb2\xf2\xce\x43\x31\x67\x12\x32\xef\x29\xcc\x69\x52\x12\x98\xd1\xf3" + :: B.ByteString + -- the buffers the caller owns, as a packet writer would have them + withMask = withMaskTag 16 + withMaskTag = withMaskIv iv16 + withMaskIv iv taglen off = + allocaBytes (B.length message + taglen) $ \outp -> + allocaBytes 16 $ \maskp -> do + ok <- GCM.encryptWithMask ctx16 hpKey iv header message taglen off outp maskp + if ok + then do + sealed <- B.packCStringLen (castPtr outp, B.length message + taglen) + mask <- B.packCStringLen (castPtr maskp, 16) + return (Just (sealed, mask)) + else return Nothing + expectedMask off = ecbEncrypt hpAes (B.take 16 (B.drop off plainSealed)) + flipFirst b = B.cons (B.head b `xor` 1) (B.tail b) + tamper f = + let sealed = GCM.encrypt ctx16 iv16 B.empty ("hello there" :: B.ByteString) 16 + (c, t) = B.splitAt (B.length sealed - 16) sealed + (c', t') = f (c, t) + in (GCM.decrypt ctx16 iv16 B.empty (c' `B.append` t') 16 :: Maybe B.ByteString) + `shouldBe` Nothing + spec :: Spec spec = do testBlockCipher128 kats128 (undefined :: AES.AES128) @@ -230,3 +394,4 @@ aeadIVLengthTests aeadTagLengthTests gcmLongTests + oneShotTests diff --git a/tests/ECDSASpec.hs b/tests/ECDSASpec.hs --- a/tests/ECDSASpec.hs +++ b/tests/ECDSASpec.hs @@ -79,8 +79,42 @@ ECC.signExtendedDigestWith k key digest >>= \s -> pure $ ECC.sign_s (ECC.signature s) <= n `div` 2 pure $ propertyHold [eqTest "normalized" (Just True) check] +-- | The deterministic nonce of RFC 6979, against the implementation in +-- Crypto.PubKey.ECC.ECDSA, which is itself held to the vectors in the RFC by +-- tests/PubKey/ECDSASpec.hs. Agreeing with it is agreeing with those. +propertyDeterministic + :: HashAlgorithm hash => hash -> Curve -> ArbitraryBS0_2901 -> Gen Bool +propertyDeterministic hashAlg (Curve c curve _) (ArbitraryBS0_2901 msg) = do + d <- arbitraryScalar curve + let prx = Just c -- using Maybe as Proxy + privECC = ECC.PrivateKey curve d + privECDSA = throwCryptoError $ ECDSA.scalarFromInteger prx d + pubECDSA = ECDSA.toPublic prx privECDSA + digest = hashWith hashAlg msg + kECC = ECC.deterministicNonce hashAlg privECC digest Just + kECDSA = + ECDSA.deterministicNonce prx hashAlg privECDSA digest Just + sigECDSA = ECDSA.signDeterministic prx hashAlg privECDSA hashAlg msg + sigWithK = fromJust $ ECDSA.signWith prx kECDSA privECDSA hashAlg msg + pure $ + propertyHold + [ eqTest "nonce" kECC (ECDSA.scalarToInteger prx kECDSA) + , eqTest "signature matches signWith" sigWithK sigECDSA + , eqTest + "signature verifies" + True + (ECDSA.verify prx hashAlg pubECDSA sigECDSA msg) + ] + spec :: Spec spec = do + modifyMaxSuccess (const 5) $ + describe "RFC 6979 deterministic nonce" $ do + prop "SHA1" $ propertyDeterministic SHA1 + prop "SHA224" $ propertyDeterministic SHA224 + prop "SHA256" $ propertyDeterministic SHA256 + prop "SHA384" $ propertyDeterministic SHA384 + prop "SHA512" $ propertyDeterministic SHA512 modifyMaxSuccess (const 5) $ describe "verification" $ do prop "SHA1" $ propertyECDSA SHA1 diff --git a/tests/EntropySpec.hs b/tests/EntropySpec.hs new file mode 100644 --- /dev/null +++ b/tests/EntropySpec.hs @@ -0,0 +1,58 @@ +-- | The system source of entropy. +-- +-- There is nothing here to check the bytes against -- they are supposed to +-- be unpredictable, and a test that said otherwise would be a test of the +-- kernel. What can be checked is that the right number of them comes back, +-- which is where an implementation of the call with a limit per request +-- goes wrong: @getentropy(3)@ refuses more than 256 bytes at a time, so a +-- backend that forgets to loop answers a short buffer for anything larger. +module EntropySpec (spec) where + +import Control.Exception (try) +import qualified Data.ByteString as BS +import Data.Maybe (catMaybes) +import Foreign.Marshal.Alloc (allocaBytes) +import Test.Hspec + +import Crypto.Random.Entropy (EntropyError (..), getEntropy) +import Crypto.Random.Entropy.Unsafe (gatherBackend, replenish, supportedBackends) + +spec :: Spec +spec = describe "the system entropy source" $ do + mapM_ lengthCase [0, 1, 31, 32, 255, 256, 257, 512, 1000] + + it "does not answer the same thing twice" $ do + a <- getEntropy 64 :: IO BS.ByteString + b <- getEntropy 64 :: IO BS.ByteString + a `shouldNotBe` b + + it "is not answering a constant" $ do + b <- getEntropy 1024 :: IO BS.ByteString + BS.length (BS.filter (== BS.head b) b) `shouldSatisfy` (< 64) + + -- getEntropy would not notice: replenish tops a short answer up from + -- the next backend, so the buffer comes back full either way and the + -- system call is quietly replaced by the device file. The backend has + -- to be asked on its own. + it "the best backend fills a buffer larger than one request on its own" $ do + bs <- catMaybes `fmap` sequence supportedBackends + case bs of + [] -> expectationFailure "no source of entropy on this system" + (b : _) -> do + n <- allocaBytes 1000 $ \ptr -> gatherBackend b ptr 1000 + n `shouldBe` 1000 + + -- A system with no source of entropy is the one failure this library + -- cannot answer, and it used to be an `error`, which a caller could not + -- tell from a bug. There is no way to take the sources away from a + -- running machine, but replenish can be handed the empty list, which is + -- the same question. + it "says so with an exception when there is no source at all" $ do + r <- allocaBytes 16 $ \ptr -> try (replenish 16 [] ptr) + r `shouldBe` Left NoEntropySource + +lengthCase :: Int -> Spec +lengthCase n = + it ("gives back the " ++ show n ++ " bytes asked for") $ do + b <- getEntropy n :: IO BS.ByteString + BS.length b `shouldBe` n diff --git a/tests/HashSpec.hs b/tests/HashSpec.hs --- a/tests/HashSpec.hs +++ b/tests/HashSpec.hs @@ -135,13 +135,16 @@ , "a8f04b0f7201a0d728101c9d26525b31764a3493fcd8458f" ] ) - , {- - , ("Skein256-160", HashAlg Skein256_160, [ - "ff800bed6d2044ee9d604a674e3fda50d9b24a72", - "3265703c166aa3e0d7da070b9cf1b1a5953f0a77", - "17b29aa1424b3ec022505bd215ff73fd2e6d1e5a" ]) - -} - + , + ( "Skein256-160" + , HashAlg (Skein256 :: Skein256 160) + , + [ "ff800bed6d2044ee9d604a674e3fda50d9b24a72" + , "3265703c166aa3e0d7da070b9cf1b1a5953f0a77" + , "17b29aa1424b3ec022505bd215ff73fd2e6d1e5a" + ] + ) + , ( "Skein256-256" , HashAlg Skein256_256 , @@ -150,13 +153,16 @@ , "fb2f2f2deed0e1dd7ee2b91cee34e2d1c22072e1f5eaee288c35a0723eb653cd" ] ) - , {- - , ("Skein512-160", HashAlg Skein512_160, [ - "49daf1ccebb3544bc93cb5019ba91b0eea8876ee", - "826325ee55a6dd18c3b2dbbc9c10420f5475975e", - "7544ec7a35712ec953f02b0d0c86641cae4eb6e5" ]) - -} - + , + ( "Skein512-160" + , HashAlg (Skein512 :: Skein512 160) + , + [ "49daf1ccebb3544bc93cb5019ba91b0eea8876ee" + , "826325ee55a6dd18c3b2dbbc9c10420f5475975e" + , "7544ec7a35712ec953f02b0d0c86641cae4eb6e5" + ] + ) + , ( "Skein512-384" , HashAlg Skein512_384 , @@ -563,9 +569,41 @@ Nothing -> error ("invalid Nat: " ++ show n) Just (SomeNat p) -> convert (hashEmpty p) +-- | The Skein types with the size in their name and the ones that take it as +-- a type parameter are the same function, and the parameter also takes the +-- sizes that have no name of their own. +skeinNatTests :: Spec +skeinNatTests = describe "Skein with the digest size as a type parameter" $ do + describe "agrees with the type of that name" $ do + it "Skein256 224" $ same (Skein256 :: Skein256 224) Skein256_224 + it "Skein256 256" $ same (Skein256 :: Skein256 256) Skein256_256 + it "Skein512 224" $ same (Skein512 :: Skein512 224) Skein512_224 + it "Skein512 256" $ same (Skein512 :: Skein512 256) Skein512_256 + it "Skein512 384" $ same (Skein512 :: Skein512 384) Skein512_384 + it "Skein512 512" $ same (Skein512 :: Skein512 512) Skein512_512 + describe "takes a size no named type offers" $ do + it "8 bits" $ len (Skein512 :: Skein512 8) `shouldBe` 1 + it "1024 bits" $ len (Skein512 :: Skein512 1024) `shouldBe` 128 + it "8192 bits" $ len (Skein512 :: Skein512 8192) `shouldBe` 1024 + it "rounds a size that is not a whole number of bytes up" $ do + len (Skein512 :: Skein512 100) `shouldBe` 13 + len (Skein256 :: Skein256 1) `shouldBe` 1 + -- the length goes into the configuration block, so it changes the chaining + -- value the message is hashed from: a longer digest is not an extension of + -- a shorter one, which is the opposite of how SHAKE behaves + it "answers a different size with an unrelated digest, not a longer one" $ do + let short = convert (hashWith (Skein512 :: Skein512 256) v1) :: ByteString + long = convert (hashWith (Skein512 :: Skein512 512) v1) :: ByteString + B.take (B.length short) long `shouldNotBe` short + where + same a b = map (h a) vectors `shouldBe` map (h b) vectors + h alg m = convert (hashWith alg m) :: ByteString + len alg = B.length (convert (hashWith alg v1) :: ByteString) + spec :: Spec spec = do describe "KATs" $ mapM_ makeTestAlg expected + skeinNatTests describe "KATs over several blocks" $ mapM_ (makeTestAlgWith longVectors) expectedLong describe "Chunking" $ mapM_ makeTestChunk expected diff --git a/tests/KDF/PBKDF2Spec.hs b/tests/KDF/PBKDF2Spec.hs --- a/tests/KDF/PBKDF2Spec.hs +++ b/tests/KDF/PBKDF2Spec.hs @@ -110,7 +110,18 @@ `shouldBe` refused PBKDF2.tryFastPBKDF2_SHA512 (PBKDF2.Parameters 1 (-1)) badPass badSalt `shouldBe` refused + -- Zero is not rejected: asking for no key is asking for no work, and + -- that is what the slow path has always answered. The fast ones go + -- straight to C, where `assert(out && nout)` took the process down + -- on a length the caller chose -- a library built without NDEBUG + -- keeps its assertions. All four agree now. + it "derives nothing when asked for nothing" $ do + slow none `shouldBe` "" + fast1 none `shouldBe` "" + fast256 none `shouldBe` "" + fast512 none `shouldBe` "" where + none = PBKDF2.Parameters 1 0 badPrf = PBKDF2.prfHMAC SHA256 badPass = "password" :: ByteString badSalt = "salt" :: ByteString diff --git a/tests/NumberSpec.hs b/tests/NumberSpec.hs --- a/tests/NumberSpec.hs +++ b/tests/NumberSpec.hs @@ -95,6 +95,24 @@ bigPrime = 2 ^ (512 :: Int) - 569 bigComposite = (2 ^ (256 :: Int) - 189) * (2 ^ (256 :: Int) - 357) +-- | The two sizes at which the exponentiation hands its multiplication to +-- s2n-bignum's assembly: a modulus of exactly sixteen or thirty-two 64-bit +-- limbs, which is 1024 or 2048 bits -- the halves a CRT exponentiation works +-- in for RSA-2048 and RSA-4096, and nothing else. A property over moduli of +-- no particular size reaches that path only by accident, and the one below +-- with "a modulus a key would have" is 1025 bits, one limb too wide. +modulus1024, modulus2048 :: Integer +modulus1024 = bit 1023 .|. (bigPrime * bigComposite `mod` bit 1023) .|. 1 +modulus2048 = + bit 2047 .|. ((bigPrime * bigComposite) ^ (2 :: Int) `mod` bit 2047) .|. 1 + +-- | A number of up to this many bytes. 'QAInteger' stops at thirty-two, +-- which is too narrow for either of these: the base has to be able to fill a +-- modulus and to overflow it, and the exponent's length is what the window +-- walks. +wideOf :: Int -> Gen Integer +wideOf bytes = BE.os2ip <$> arbitraryBSof 0 bytes + -- the index is threaded through so that repeated calls cannot be shared askAgain :: Int -> Integer -> Bool askAgain i n = i `seq` primalityTestMillerRabin 1 n @@ -174,6 +192,31 @@ \(QAInteger b) (QAInteger e) -> let m = 2 * bigPrime * bigComposite + 1 -- odd, and 1025 bits in expSafe b (abs e) m === expFast b (abs e) m + prop "agrees with the fast one at the two sizes with assembly behind them" $ + forAll (elements [modulus1024, modulus2048]) $ \m -> + forAll (wideOf 300) $ \b -> + forAll (wideOf 256) $ \e -> + expSafe b e m === expFast b e m + prop "agrees with the fast one whatever the exponent's length" $ + forAll (choose (0, 129)) $ \bytes -> + forAll (wideOf bytes) $ \e -> + forAll (wideOf 128) $ \b -> + expSafe b e modulus1024 === expFast b e modulus1024 + -- the C takes RSA's exponent at these sizes on some machines; plain + -- squaring shares no multiplication with it + prop "agrees with plain squaring on RSA's exponent at every key size" $ + forAll (elements [128, 192, 256, 384, 512]) $ \bytes -> + forAll (wideOf bytes) $ \m' -> + forAll (wideOf bytes) $ \b -> + let m = m' .|. bit (8 * bytes - 1) .|. 1 + in expFast b 65537 m === naivePow (b `mod` m) 65537 m + -- the fast one keeps R^2 for the last few moduli; asking under more of + -- them than it keeps, round and round, would show one used for another + it "keeps each modulus' constant to that modulus" $ do + let ms = [modulus2048 - 2 * 0x10001 * fromIntegral k | k <- [0 .. 11 :: Int]] + bs = [modulus2048 `div` (3 + fromIntegral k) | k <- [0 .. 11 :: Int]] + asked = concat (replicate 3 (zip bs ms)) + [(b, m) | (b, m) <- asked, expFast b 65537 m /= expSafe b 65537 m] `shouldBe` [] where safely (b, e, m) = expSafe b e m fastly (b, e, m) = expFast b e m @@ -239,6 +282,10 @@ numBytes i == byteCount i prop "num-bytes-small" $ \() -> map numBytes [0, 1, 255, 256, 257, 65535, 65536] == [0, 1, 1, 2, 2, 2, 3] + -- the magnitude, which is what GMP counts, and what the fallback used to + -- divide by 256 forever looking for a quotient of zero + prop "num-bits-negative" $ \(Positive i) -> + numBits (negate i) == numBits i prop "generate-param" $ \testDRG (Int1_2901 bits) -> let r = withTestDRG testDRG $ generateParams bits (Just SetHighest) False in r >= 0 && numBits r == bits && testBit r (bits - 1) diff --git a/tests/PubKey/DSASpec.hs b/tests/PubKey/DSASpec.hs --- a/tests/PubKey/DSASpec.hs +++ b/tests/PubKey/DSASpec.hs @@ -386,19 +386,27 @@ , DSA.public_params = pgq vector } -doSignatureTest hashAlg i vector = it (show i) (actual `shouldBe` expected) +-- Both routes, and each against the vector rather than against the other: +-- they share everything below the hash, so a test that only made them agree +-- would pass with that shared part broken. +doSignatureTest hashAlg i vector = describe (show i) $ do + it "signWith" $ + DSA.signWith (k vector) key hashAlg (msg vector) `shouldBe` expected + it "signDigestWith" $ + DSA.signDigestWith (k vector) key (hashWith hashAlg (msg vector)) + `shouldBe` expected where + key = vectorToPrivate vector expected = Just $ DSA.Signature (r vector) (s vector) - actual = DSA.signWith (k vector) (vectorToPrivate vector) hashAlg (msg vector) -doVerifyTest hashAlg i vector = it (show i) (actual `shouldBe` True) +doVerifyTest hashAlg i vector = describe (show i) $ do + it "verify" $ + DSA.verify hashAlg pub sig (msg vector) `shouldBe` True + it "verifyDigest" $ + DSA.verifyDigest pub sig (hashWith hashAlg (msg vector)) `shouldBe` True where - actual = - DSA.verify - hashAlg - (vectorToPublic vector) - (DSA.Signature (r vector) (s vector)) - (msg vector) + pub = vectorToPublic vector + sig = DSA.Signature (r vector) (s vector) -- | Both sign and verify invert a value modulo q with 'fromJust'. The -- inverse does not exist when the value shares a factor with q, and neither diff --git a/tests/PubKey/ElGamalSpec.hs b/tests/PubKey/ElGamalSpec.hs --- a/tests/PubKey/ElGamalSpec.hs +++ b/tests/PubKey/ElGamalSpec.hs @@ -3,7 +3,7 @@ module PubKey.ElGamalSpec (spec) where import Crypto.Error -import Crypto.Hash (SHA256 (..)) +import Crypto.Hash (SHA256 (..), hashWith) import qualified Crypto.PubKey.DH as DH import qualified Crypto.PubKey.ElGamal as ElGamal import Crypto.Random (drgNewTest, withDRG) @@ -103,6 +103,35 @@ `shouldBe` False it "rejects a signature with r out of range" $ ElGamal.verify params pub SHA256 msg (ElGamal.Signature 0 1) `shouldBe` False + -- The digest route and the hash-value route, crossed: a signature made + -- by one has to verify under the other. Each route alone would pass + -- with its own arguments in the wrong order; crossing them would not. + it "verifies under the digest route what the hash route signed" $ + case ElGamal.signWith k params priv SHA256 msg of + Nothing -> expectationFailure "expected a signature" + Just sig -> + ElGamal.verifyDigest params pub (hashWith SHA256 msg) sig + `shouldBe` True + it "verifies under the hash route what the digest route signed" $ + case ElGamal.signDigestWith k params priv (hashWith SHA256 msg) of + Nothing -> expectationFailure "expected a signature" + Just sig -> ElGamal.verify params pub SHA256 msg sig `shouldBe` True + it "rejects a digest of another message" $ + case ElGamal.signDigestWith k params priv (hashWith SHA256 msg) of + Nothing -> expectationFailure "expected a signature" + Just sig -> + ElGamal.verifyDigest + params + pub + (hashWith SHA256 ("other" :: ByteString)) + sig + `shouldBe` False + it "verifies what signDigest signs when it draws k itself" $ + let (sig, _) = + withDRG + (drgNewTest (1, 2, 3, 4, 5)) + (ElGamal.signDigest params priv (hashWith SHA256 msg)) + in ElGamal.verifyDigest params pub (hashWith SHA256 msg) sig `shouldBe` True -- 'sign' draws a blinder for the inversion of k, so it takes a path -- 'signWith' does not: the inverse comes back from a different number -- than the one wanted, times the blinder diff --git a/tests/PubKey/MGF1Spec.hs b/tests/PubKey/MGF1Spec.hs --- a/tests/PubKey/MGF1Spec.hs +++ b/tests/PubKey/MGF1Spec.hs @@ -10,7 +10,11 @@ import Data.ByteString.Char8 () import Crypto.Hash +import qualified Data.ByteArray as BA +import Crypto.Number.Serialize (i2ospOf_) import Crypto.PubKey.MaskGenFunction +import Test.Hspec.QuickCheck (prop) +import Test.QuickCheck (choose, forAll) import Utils @@ -32,5 +36,26 @@ } ] +-- | RFC 8017's definition through the pure hash API, block by block, which +-- shares nothing with mgf1's in-place contexts or with the one-shot hash. +reference :: HashAlgorithm a => a -> ByteString -> Int -> ByteString +reference alg sd len = B.take len $ B.concat [block c | c <- [0 .. blocks - 1]] + where + blocks = (len + hashDigestSize alg - 1) `div` hashDigestSize alg + block c = BA.convert (hashFinalize (hashUpdates (hashInitWith alg) [sd, i2ospOf_ 4 (fromIntegral c)])) :: ByteString + spec :: Spec -spec = zipWithM_ doMGFTest [katZero ..] vectorsMGF +spec = do + zipWithM_ doMGFTest [katZero ..] vectorsMGF + describe "against the definition" $ + mapM_ agree + [ ("SHA1", \s n -> (mgf1 SHA1 s n, reference SHA1 s n)) + , ("SHA256", \s n -> (mgf1 SHA256 s n, reference SHA256 s n)) + , ("SHA512", \s n -> (mgf1 SHA512 s n, reference SHA512 s n)) + , ("SHA3_256", \s n -> (mgf1 SHA3_256 s n, reference SHA3_256 s n)) + , ("Blake2b_512", \s n -> (mgf1 Blake2b_512 s n, reference Blake2b_512 s n)) + ] + where + agree (name, f) = prop name $ + forAll (arbitraryBSof 0 100) $ \s -> forAll (choose (0, 300)) $ \n -> + let (got, want) = f s n in got == (want :: ByteString) diff --git a/tests/PubKey/MLDSASpec.hs b/tests/PubKey/MLDSASpec.hs new file mode 100644 --- /dev/null +++ b/tests/PubKey/MLDSASpec.hs @@ -0,0 +1,251 @@ +{-# LANGUAGE RankNTypes #-} +{-# LANGUAGE ScopedTypeVariables #-} + +-- | ML-DSA against NIST's ACVP vectors, and against itself. +-- +-- The vectors are the point. A signature this module makes and then verifies +-- says only that its two halves agree; the vectors say the signature is the +-- one FIPS 204 asks for, which is what a peer will check it against. +module PubKey.MLDSASpec (spec) where + +import qualified Data.ByteArray as B +import Data.ByteArray.Encoding (Base (Base16), convertFromBase) +import qualified Data.ByteString as BS +import Data.Proxy (Proxy (..)) +import Control.Monad (forM_, when) +import Data.List (nub) +import Test.Hspec hiding (context) + +import Crypto.Error +import Crypto.PubKey.MLDSA + +import Imports () +import PubKey.MLDSAVectors + +hex :: String -> BS.ByteString +hex s = case convertFromBase Base16 (BS.pack (map (fromIntegral . fromEnum) s)) of + Left e -> error ("bad hex in a test vector: " ++ e) + Right b -> b + +withSet :: String -> (forall p. MLDSA p => Proxy p -> r) -> r +withSet "ML-DSA-44" k = k (Proxy :: Proxy MLDSA44) +withSet "ML-DSA-65" k = k (Proxy :: Proxy MLDSA65) +withSet "ML-DSA-87" k = k (Proxy :: Proxy MLDSA87) +withSet s _ = error ("unknown parameter set in a test vector: " ++ s) + +ctxOf :: String -> Context +ctxOf "" = emptyContext +ctxOf s = case context (hex s) of + CryptoPassed c -> c + CryptoFailed e -> error (show e) + +spec :: Spec +spec = do + describe "ACVP keyGen" $ + mapM_ keyGenCase keyGenVectors + describe "ACVP sigGen" $ + mapM_ sigGenCase sigGenVectors + describe "what a signature is bound to" $ + mapM_ bindingCase sigGenVectors + describe "ACVP sigGen, the external-mu interface" $ + mapM_ extMuCase extMuVectors + describe "the message representative" $ + mapM_ muCase sigGenVectors + describe "the message representative, a piece at a time" $ + mapM_ muStreamCase sigGenVectors + describe "the seed a key pair came from" $ do + seedKeeps "ML-DSA-44" (Proxy :: Proxy MLDSA44) + seedKeeps "ML-DSA-65" (Proxy :: Proxy MLDSA65) + seedKeeps "ML-DSA-87" (Proxy :: Proxy MLDSA87) + describe "round trip" $ do + roundTrip "ML-DSA-44" (Proxy :: Proxy MLDSA44) + roundTrip "ML-DSA-65" (Proxy :: Proxy MLDSA65) + roundTrip "ML-DSA-87" (Proxy :: Proxy MLDSA87) + +keyGenCase :: KeyGenVector -> Spec +keyGenCase v = + it (kgSet v ++ " tcId " ++ show (kgId v)) $ + withSet (kgSet v) $ \p -> + case keyPairFromSeed p (hex (kgSeed v)) of + CryptoFailed e -> expectationFailure (show e) + CryptoPassed (vk, sk) -> do + B.convert vk `shouldBe` hex (kgPk v) + B.convert sk `shouldBe` hex (kgSk v) + +sigGenCase :: SigGenVector -> Spec +sigGenCase v = + it (label v) $ + withSet (sgSet v) $ \(_ :: Proxy p) -> + case signingKey (hex (sgSk v)) :: CryptoFailable (SigningKey p) of + CryptoFailed e -> expectationFailure (show e) + CryptoPassed sk -> + let ctx = ctxOf (sgContext v) + msg = hex (sgMessage v) + got + | sgDeterministic v = + CryptoPassed (signDeterministic sk ctx msg) + | otherwise = signWith sk ctx msg (hex (sgRnd v)) + in case got of + CryptoFailed e -> expectationFailure (show e) + CryptoPassed s -> + B.convert s `shouldBe` hex (sgSignature v) + +-- | The vector's own signature, verified -- and then the three ways it should +-- stop verifying. Signing and verifying with this module alone could agree +-- on a wrong domain prefix; this pins the prefix to the vector's signature +-- and then shows the context is really part of it. +bindingCase :: SigGenVector -> Spec +bindingCase v = + it (label v) $ + withSet (sgSet v) $ \(_ :: Proxy p) -> + case ( signingKey (hex (sgSk v)) :: CryptoFailable (SigningKey p) + , signature (hex (sgSignature v)) :: CryptoFailable (Signature p) + ) of + (CryptoPassed sk, CryptoPassed sig) -> do + let vk = toPublic sk + ctx = ctxOf (sgContext v) + msg = hex (sgMessage v) + verify vk ctx msg sig `shouldBe` True + verify vk ctx (flipFirst msg) sig `shouldBe` False + verify vk (otherContext (sgContext v)) msg sig `shouldBe` False + case signature (flipFirst (hex (sgSignature v))) of + CryptoPassed bad -> verify vk ctx msg bad `shouldBe` False + CryptoFailed e -> expectationFailure (show e) + (CryptoFailed e, _) -> expectationFailure (show e) + (_, CryptoFailed e) -> expectationFailure (show e) + where + flipFirst b + | BS.null b = BS.singleton 1 + | otherwise = BS.cons (BS.head b `seq` BS.head b + 1) (BS.tail b) + -- any context other than the one it was signed under + otherContext "" = ctxOf "00" + otherContext _ = emptyContext + +-- Signing a representative the vector supplies. +extMuCase :: ExtMuVector -> Spec +extMuCase v = + it (xmSet v ++ " tcId " ++ show (xmId v) ++ det) $ + withSet (xmSet v) $ \(_ :: Proxy p) -> + case ( signingKey (hex (xmSk v)) :: CryptoFailable (SigningKey p) + , mu (hex (xmMu v)) + ) of + (CryptoPassed sk, CryptoPassed m) -> do + let got + | xmDeterministic v = + CryptoPassed (signExternalMuDeterministic sk m) + | otherwise = signExternalMuWith sk m (hex (xmRnd v)) + case got of + CryptoFailed e -> expectationFailure (show e) + CryptoPassed sig -> do + B.convert sig `shouldBe` hex (xmSignature v) + verifyExternalMu (toPublic sk) m sig `shouldBe` True + (CryptoFailed e, _) -> expectationFailure (show e) + (_, CryptoFailed e) -> expectationFailure (show e) + where + det = if xmDeterministic v then ", deterministic" else ", hedged" + +-- messageRepresentative, against a vector that never mentions mu. +-- +-- The vectors for the external-mu interface supply the representative, so +-- using them would only say that signing it works, not that this computes +-- the right one. Taking a vector from the ordinary interface and computing +-- the representative from its key, context and message does say that: the +-- signature has to come out the same as the one the vector gives for +-- signing that message directly. +muCase :: SigGenVector -> Spec +muCase v = + it (label v) $ + withSet (sgSet v) $ \(_ :: Proxy p) -> + case signingKey (hex (sgSk v)) :: CryptoFailable (SigningKey p) of + CryptoFailed e -> expectationFailure (show e) + CryptoPassed sk -> do + let vk = toPublic sk + ctx = ctxOf (sgContext v) + msg = hex (sgMessage v) + m = messageRepresentative vk ctx msg + B.length m `shouldBe` muSize + let viaMu = B.convert (signExternalMuDeterministic sk m) + direct = B.convert (signDeterministic sk ctx msg) + (viaMu :: BS.ByteString) `shouldBe` direct + -- and for the deterministic vectors it is the + -- signature the vector itself gives + when (sgDeterministic v) $ + viaMu `shouldBe` hex (sgSignature v) + +-- The streaming form, against the same vectors. +-- +-- Where the message is cut must not matter, so every cut is tried: none, +-- at the front, at the back, at thirds, and one byte at a time. All of +-- them have to give what 'messageRepresentative' gives for the whole +-- message, and -- for the deterministic vectors -- signing that has to +-- give the signature the vector itself holds. Without that last step the +-- test would only say two of this module's paths agree with each other. +muStreamCase :: SigGenVector -> Spec +muStreamCase v = + it (label v) $ + withSet (sgSet v) $ \(_ :: Proxy p) -> + case signingKey (hex (sgSk v)) :: CryptoFailable (SigningKey p) of + CryptoFailed e -> expectationFailure (show e) + CryptoPassed sk -> do + let vk = toPublic sk + ctx = ctxOf (sgContext v) + msg = hex (sgMessage v) + whole = B.convert (messageRepresentative vk ctx msg) + n = BS.length msg + cuts = nub [0, 1, n `div` 3, n `div` 2, n - 1, n] + chunked :: [BS.ByteString] -> BS.ByteString + chunked cs = + B.convert (muFinalize (muUpdates (muInit vk ctx) cs)) + forM_ (filter (\i -> i >= 0 && i <= n) cuts) $ \i -> + let (a, b) = BS.splitAt i msg + in chunked [a, b] `shouldBe` (whole :: BS.ByteString) + chunked (map BS.singleton (BS.unpack msg)) + `shouldBe` (whole :: BS.ByteString) + -- an empty piece is not a piece + chunked [BS.empty, msg, BS.empty] + `shouldBe` (whole :: BS.ByteString) + let streamed = + muFinalize $ + muUpdate (muUpdate (muInit vk ctx) (BS.take 1 msg)) $ + BS.drop 1 msg + when (sgDeterministic v) $ + (B.convert (signExternalMuDeterministic sk streamed) :: BS.ByteString) + `shouldBe` hex (sgSignature v) + +-- The seed generateKeyPairAndSeed hands back has to be the one the pair +-- was derived from: expanding it again has to give that very pair, not +-- merely some pair. Two generated pairs also have to differ. +seedKeeps :: MLDSA p => String -> Proxy p -> Spec +seedKeeps name p = + it (name ++ ": the seed comes back and rebuilds the pair") $ do + (vk, sk, seed) <- generateKeyPairAndSeed p + B.length seed `shouldBe` seedSize + case keyPairFromSeed p seed of + CryptoPassed (vk', sk') -> do + (B.convert vk' :: BS.ByteString) `shouldBe` B.convert vk + (B.convert sk' :: BS.ByteString) `shouldBe` B.convert sk + CryptoFailed e -> expectationFailure (show e) + (vk2, _, _) <- generateKeyPairAndSeed p + (B.convert vk2 :: BS.ByteString) `shouldNotBe` B.convert vk + +roundTrip :: MLDSA p => String -> Proxy p -> Spec +roundTrip name p = + it (name ++ ": a signature this module makes, it verifies") $ + case keyPairFromSeed p (BS.replicate 32 5) of + CryptoFailed e -> expectationFailure (show e) + CryptoPassed (vk, sk) -> do + let msg = BS.pack [1 .. 40] + ctx = ctxOf "aabb" + sig = signDeterministic sk ctx msg + verify vk ctx msg sig `shouldBe` True + toPublic sk `shouldBe` vk + -- the same key and message twice give the same signature + signDeterministic sk ctx msg `shouldBe` sig + +label :: SigGenVector -> String +label v = + sgSet v + ++ " tcId " + ++ show (sgId v) + ++ (if sgDeterministic v then ", deterministic" else ", hedged") + ++ (if null (sgContext v) then ", no context" else ", with a context") diff --git a/tests/PubKey/MLDSAVectors.hs b/tests/PubKey/MLDSAVectors.hs new file mode 100644 --- /dev/null +++ b/tests/PubKey/MLDSAVectors.hs @@ -0,0 +1,207 @@ +-- | ACVP test vectors for ML-DSA, from NIST's ACVP-Server. +-- +-- Generated from gen-val/json-files/ML-DSA-{keyGen,sigGen}-FIPS204. The +-- signature vectors are the pure, external-interface groups, which is what +-- this module implements. +-- +-- Every combination -- deterministic or hedged, with a context string or +-- without -- is taken at ML-DSA-44, which is the smallest, and the ones with +-- a context at the other two. That exercises each path through the Haskell +-- at 44 and would still catch a size constant that is wrong at 65 or 87, +-- without carrying four times the vectors for it. +-- +-- They are here rather than downloaded when the suite runs, so that it needs +-- no network and a release tarball carries what it was tested against. +module PubKey.MLDSAVectors ( + KeyGenVector (..), + SigGenVector (..), + ExtMuVector (..), + keyGenVectors, + sigGenVectors, + extMuVectors, +) where + +-- | A seed in, the key pair out. +data KeyGenVector = KeyGenVector + { kgSet :: String + , kgId :: Int + , kgSeed :: String + , kgPk :: String + , kgSk :: String + } + +-- | A signing key, a message and a context in, the signature out. When +-- 'sgDeterministic' the randomness is zeroes and 'sgRnd' is empty; otherwise +-- 'sgRnd' is the randomness the vector was produced with. +data SigGenVector = SigGenVector + { sgSet :: String + , sgId :: Int + , sgDeterministic :: Bool + , sgSk :: String + , sgMessage :: String + , sgContext :: String + , sgRnd :: String + , sgSignature :: String + } + +keyGenVectors :: [KeyGenVector] +keyGenVectors = + [ KeyGenVector + "ML-DSA-44" + 1 + "7194B13C95231010AFD2C909992BD2003BA6F437C3886BDBE3F6B867A14BA161" + "0B89806F0EEC39F2891116152ED4319D4260DFB8AC0710765BD497E6E1DE17783CF81E435A412EABEF5DB3AF5D15867BBB4C60F8CF98BA31BAD6D41A5F8EB0C11B632C3F19D844A223C353BD182883DCF13B5C97823D0C0E6902DB25AD8D344A37F59F4AFACA5BC8874792DA1E6A3EAE742AB7034B20A4AB75A93BCA4B68002DD242CED348920B7E5ABF645A0E2E79617BCB3EE7BA972B3E718D3EFFC59B1869814BA3F526927477B12BF25CBAD8B04B09905FDAD3820715A8B9A905DE1CD65EFF6B0B0886305EFB6CFEEC9E90B5EF9A5AAEC45C753298E8DF9B017CE0FEC9B7431B20775CE8CB11F1F42D1D9FE936D0803196E71ADDC26CC430CC3B69760C7CCAFAB7651E21BAA28F92BBFF1C4A6EEF156D6F08F80B5E3B6FC943E6E984378B90888D09A6EA38B0BA86A3446211452E076DC9F65620014205D5271C7A44FEC3CC5375EB246AFFC11B26CAFB8B96CEE3A68E31642E3D69B9130795F25ED818EBB211CD8BE648ADB5C8A120C8186017727FBCAB31C7425C08FE9195DE6BDBADA5778D727EE5CDE0674FACB7AB81786357B529C71DDB24DF770E8E95E5F3112BD297B352CB91B08ED1097A98E87BD7CE4235B8DD42292CD4C59D87C1F0FF00734AA22D7CAE4361ADC47742C897601048526702538828BA3C3A959990C0E99463FD22417E147FF2DAA74C0C8D3A06E9703A2E160590086DB8011A3D9CEC5AE6348706F87CB2379632CE56E660A0BA1B30E3846C5B5C6C0339DD993E543A5322AF5A11FC7040A2DF23A0B43E882D7A0FF4431A723BBB918AFF7F14BC045CBE94BCAB27AE3109147B588665EF486006562B1297016EFDE787B46237060EE431E0F011166F916AA0789A7647103B7400A1CBCF0E22BD7B6DD2BB3EC51EC98F0EC6A5BAA4CDC83F993D302F8FA849F2046B78AA32F0B3751885ECB941799E250E6546DCA5C20C24845190F239EDC20DDA77353D555DE61509CA6D3C6DC3195BBC6F1703CB03EAD5E7FCBCF5D196E9AB71522408E11D6337C74F9A31EB22AD084A19132BF72E7076A9743ED070ABA78789791824E050CD27694C2648263D1200811FA1B81A00B8FC09CB7A338795E54F6598D7753395F05C60E6EBA9630912B7AA8CAAB3017565DEF72C7929F4E7736C2B8043FEB448801E2DED704E834294B69F6A109C0968214FDC5C3FF0D1B1555D617E16DF61829231962C59B22A10FE400F8B8CB2A3F19FB4B2E8D087F22687506E7F0D061857D1C1789C7F55B899FF4B322982D64BD0AA751D5BEE320B135C7F5DDCD5E6245B57DD22F44042F2BA6DE942365A59FD0C6B0F20C07B71277C6EE7DD9D225032605AED1D3CF8242EB85C33A0AFC3AB42764088D8F4A80FAF804CD84360B2055181E58A0B5AD4C367ABC667982045AD0FD7E048AF8C326D5DB60233302B107E515B15B0F90E5F348C54192B559B4C0A86CDF0719387EA3FF6B1D60B324A98963C56927E2B8DD5A39AC792AEB85EBDBD8DC34B395C2B4DEF4D853AC21A7660348EA8C96C943DE0BAFF3AA6849179E5EF2BAA1731C81C605BEC3860FC4A6A08CC9F75BDE9533511780FF1E0B01D34C0DC3EB80A7E2F52A7A4B815DDA98EA775DFE0C5B3D419B05934DDA05A9616C0978CC99CC8D7B68227BD846419D765956C3D7AA811CE60AF22DF322FEF0DCE38C4278E0237F1D29EF139E201C8ECB4D36E79910D06C5CA4CAA8C2886B96DE6EDD40D2499E30EB942F22BEBF6ED5C8E37DF9557E74D67DC467BAAFA68F1CE37C8BD9B3A4F9DE71670128125AA16ACA7232239575E1C6819C820AD16832F23647DD53C5740A8552F86901AA4F883EFD5A3EFD7C3BF458C5122712D44BE43306C9B8264" + "0B89806F0EEC39F2891116152ED4319D4260DFB8AC0710765BD497E6E1DE17786CDEC899F1C6534284585DDA4DF03E45E4D39B4526015A7B3D65F8BF875452560DB3223594A1FCE8DB48C8F1793611A17FCC0006FFEA26CF7094D8325037288F8AF7D062833B71B8C0F06108442786E3DE59D649162273EF179AEADBBD48CC981CA86463082A0020220CC10003B160C0B82C20A70409C521649820E3A84409C689832621E20832C1B04C1C8911429420193789C12462E4C661643600D8484A0998601902621B104CC2A4495A1600444690A3800CD3889098A49024C3901A344D594052614411248101C416261308881C818083A86142200D23262A1B2501D0422464C4911315094092641B071208956C513830A20831CBB84153240E99C41111A1241A297003A74011B744A2442DDAC0041C21611222684B126821C3104082600A2352C43890CC967063206553486D84124C0239328A0640112689E49841114871D22652E3C2240A98889802891B8031E4089140B404534286411826D102929C946180C84801134118348509B76984180D0A404AE2340E0181809C04662481299AA424A2B8445C38491AC20DD9B045A286491C864011036C13B78044B2618498700A1989A00024E1C2488CB09063B8100A408C18171009B76C1108249136404BC29104A96901B525044802C300319C380E0A26229CC4841A04306184201A1300A1940460982912924D12144AE1068D80B62D1AA5855C240849A20401B56880B449D9B21163240A98424414B40401264282008C0B04651B21409A488494866D18B380C4A801020326CC44491922208A364264048D1205814AB20401426A5886201034110039295A98654138080CC964D3849021A788E0088901386012222859488290C20C538430823626818051A4C871819431E0386ED3A22121186E8B027151240ED13689903408A3480A0B34894C040608314C81280400894503920D010208E0C011D1041288482E01280E08C161C11820C4C84800242A24C545D80020C8368498C885D4B40CC3062942B00D98487002A7899B303152284E10860C23B124901829C2C84462B269431824098621D2062989020A1CA76023C00448426CA4902CE09681C2C66D23286C40206A4404210C3405C980814B0892A0162151B001C1148D413849DC968C212244C4A00D18049240083209126A00177109118114962402C22D109591A1084D0B316C01466ACA8291CA1252D3B23018094D44C4845490411FB45680A1B2685FF6F4075D76E422EE9B1DD39048A2C6B0C1C441689316AEE550C179A8E55B87627071ED299FCBFACFD13CFBE61A9F87BE0579A19357C4E7B0F125D354DCEC0CA0EAEB5E116FBD87EFB3049ADE6F28921D56ED84487CAD51BD84F7C07FF460B09B4201E7B9DF1801DA2771C2DA20D2B687A44D4D1F49DA38BCF9AE30713ACD86E32F52FA8735DCDEDF1CDA4A5C2C8880784A5B7C3AC395FF8C4BD1A0303868CD3E6D1F7EF258F38F5CF560AB9E8357272222123821D25C141258A6B132F9F99D014584BFE23AD4957F6692CD7E8327CF66E581A598C4FF3C7CBF5316B3AB028FF82E8BC3250F250E6A451996BFAA00A1458A86F8304F2A839DD1B929EAE4E53E919AEA13BA09569B14148ECE44CB27650EAE5FB352061C7301D8DD9CB5156BC15DE1F578AD95D6505CCFD485D99867D48BC6910A491856D30B17FD8952B774A70F57A24458CD9B18D0A222BC5B307A34EE347106A9B76609505E7C81495F88D8DA05D742188F01820EBB8AC559FF3417A33E6CD4FDA1D60C1A6D37C3D27F51717645ADEE9020F10748F7CDB0D5B142F465C54FE5D70CB787EB47B8741A162BF373AFA1C8DB3985900C6A8B9035006CEDB7EB9854D3C50E1C30C8A6B34269D85D7C683EDB1BE1455ECE1C9768EA9C9A140036E8EAD9A19D9F167E52CA10DA5FA7FE2BED7C0ADB0A45C642AC02ECB5B1C5199AD5D6227CB4F506A973D696908C15791513783736FFFE5A47395CE2E7CE1C42E7F6541825A2BDE5617F53B5155AC30E3EC43BB4EF5AD727ACE5A5ADCC7F036A1BB606F7C943C112B372DB92832639DB2F488A3ED64E43609DD93B43F38DB939F6BAE61E3E44772929E65F43D739A061AE3272021A220387A43BE3A985AD713999F75E040DF53DA81801BF165052A68179E6BB1FD4F624C31EDAB74F6E2E7EFC31EDA78103BFCB32B837BA07C5D37E922440DE741BE0029BE98DC86739324D73E62B3FD09B9EE00EF8DBCFD0ED687C5269BF3A4F84AFE2B8FB52BBD118E718DF5972038CDAB018CF8AC7D6785C958AC5B9B23785DE1A90B9BE64279015FCE8C36C87453E392DA20CC72C533D115BC0F53A385A0DF6127B3A81592552B7CF0E8AF3797869683FF0C42D2A189C04442966B37CA321A4DBF02067447D50D09F92E4E63A272A97E0460FF1BFBF82F61412BBBDEAE83D0843F5B38E10A53DC5CA86A4C6AEB17601FB560A8852BC60C25767C2489F95143FCF75B648E814374BA98DFA753D08A53F02377F551C9AA374301CE7C84B1A48E6F750794E5386361815155053175DB12E29EF9D49920D7704CC343DA6021249479E5E5405F3BEDE4DBD612009C34E659C3C9D7C59DA97D104F47D4B314D1C0E2414F8746CA658A9731C18A0C89F61E11F617B197093ADDEFA42CA0DF723F93A12ED83C352B05F8F5039B2C8D321C4992D0DF249BC5148E0C27519430A9E70A5CC24B8E0217D6F9BD04737A4FCF7351C7670269DAD9DA97858A1FA9DA23DFAB215172BFF72962F62406D2C5747CA0F273EF8F31761F99CAF2F417685F971C3415FD1C79D7A4E75EC50A6B7B795A35BC45EDAA824EE71E651830C96AC2905C1EAF817B8E833C9BE77242A6B43FFFBC108DAC12631EBC86BC06BD7E506F827B142F03476357C9AE5B98B447DBE0F408C75535FD9341FA3693F2887FACE3B72CECFD62CBF908319A22336AC43E57E2A46C42B28B6E55D7075B5CFED7D8C0A8A1A4ADAAE79B4A09B7CCFC04FD99FEBE3A0DE1F8A9B43F97A2F93CD758AD546C0A1C3801E3BCFE1ADC246E7A34044CF56BFBDF5A5035C2B8D1E19A3A7243C225BE23EBA7FFF8E052C3845310F4B2B7393BDC15B766BA0B3CB45BD0A1CC693C947A5A964DD39287828EE86EB6C2DB9D8967EDF4B75C78EF4B34561EA1A9D93BB8E1209381E9D1F2C0E61EDBAB542E07E2C3C71F4841A3F2117F26B608CA244C46663C3FBD6A6A20F55C8C778C585BAC5DBEFDE74A7EFA8658D95B12EE9412BC8CC24333BB2A3E994E887A8140FE482EDFECA89E887531A536BC13FC44AF7B595B06E6B122E59A324992C553D6278AF277E5C545B126105D1A180D2CF769ABBCD9B8DB72330E6548521FF4569C674E60D35923B86F0166CD24D8AC7FA4F49743E7E2C90BCF3E66955C6F5CA430024902C536D0E0F5D3A637C033A3BA6F9778475C455E440A5E03B485F7C8263F5D007A8A1B3DEF7AE943DD38633715B50A2E76228521EB1D0CAAEAB48951C1E395DF94F9A63313DBCCF1A6BE8C0954388AEBB0AE472E741A8006DC0299F5E4073E89A3D097512321BA8037C391CBEE65977354B3739CB04FAE7663D86E9CE04BEF14D3615B9DF81AEA3E4" + , KeyGenVector + "ML-DSA-65" + 26 + "A991FD42B071D49C48AE3E75C647459E0DAAD1E1BA356A04801912D3294BCFF8" + "36DB0B5DCE98BD190CB139E80B71B49C7D7040B71C5A1F3412C46BDE939192B1B57CCB88AC2714C1240CB0EB62C689E031AEA3D9F3EB3ED7BFA45931D288DCAE3413199B31A7032560DCE8A61E195D13A1440615C2F3AA7DD28C5B1B742BFA400052186721F13D3DF9DCFAEE348B10D66913C7148913E085E1A4A03C659398DADC6A8E0E0C1A7F9F44D30436DB90FD65A6AB8F36137338255653BAAE8DA21526A333426DBD9F76CCE0F43212643E854D772018B35CE726BCAAA5AB0651BAF8C122E13929BB35B6E4963DF2595FDC7237CDAA7234BF776B07F353CCDBA12AD3E025138E3492D7F8E929DB55DC23E23075F66D57A10492E6A10AE7B758ACC2291CA18BA1CA07A5B574AB6D8AAC18B9524990AD2F110225B7D82F696300A660A166AD35B3C57ECBAB77117C79656FA8AE2A19A7DEFB2AFD2AF54683D043BE0F933B8EAE0D591448ED55D00068CD9FE10B067FCFAAC53AEDB1E9B667E36C4E30231F85C7AA0A474AF2FA4776226F4479555E155528D78B98183CBDF7FAE4E7301140F163EB71E991D15FAD4A0D2F25A5A62FA2E9BCC823CC2927662E40C538213DED9E2DF508E911E4924E507A50861FBB050EDBBF56D937206F8FBC6F4CEAD4CD10D06B73AADCD4AA39703A7A2BFFAE68B7BAA47341B699DA9F3B167D4D90EFEE0A07EE3529A3B5E8648B9CB07EE973E1D8DCCF1D16E95092C4A0184CCB4902D6086D9F444ACA5FA45F43CA91B351E82585989FFCBD6D2C3471D6B8593AA46F29D0DD9B44E8AA4D8F9A0BC886BB7982C56AAB11E23BFDBD8BC674732FADACACAE25FA416B2D0CC7743827293336507DA4B14C1F0AA2E929AF975466DADC89A016F33A0CCA2D5C08114CF04B02358805A772536432C44DBE9886130D2D3A0FFA0E175875A2207686F5E562B879EB2957573AB706B942468C20CC69BC566D29D9F151F3CFAE71CC97CE4A30722D4679FC1C089B5009935931EE60AAC5496B0FD5F24E514C0E20FA1DCC7729184A50FC85FAD1D2F32F715FBF55666E49F5A19761F2DD1AA5D1A33C7916EB6A794981C0334176ABC493EF30D9EAEAAD42E705989DCFCDEB578529A700BD14076A348A2062D6483CC63CD7F55136587AAC0E531A06EB2DE74E61CFCBBCE18F2ADA5A741F683BF101F71432EE659DD1508E0C8FB2400E0CCBE435DA3466D543D3EF5BA369E125C0B84D855EFE6D4A22FF929A7A7A984E448D23871E09B88A0BC3F3B7DA55DD2EDFB5A6DAA102819FF50CD4DCCD0A95D2F27354668065D4A56C31FB18B92B2A8DB2C6453BAA9333AEA6EABB1BD6411D584DD5900262057A707F81CC5137DBDD9AC1079BB98DA78A8E4BD1B2E0546C2B3D956FCC280D37D855E31F1E4315B387A742280F057F3219EAE512884AE7EC4D2E3A72265B1D0163FBCBF616B2E289B0EAF9C63437D50B7B50CE408F5B4562F2ABF510C19F5E8A0ACE264DB6E0F2A69A7D0B4A5E62A2B964F08C8FFE9C295F5773BDD7FAB054A13822D428FAE28AE5E4ADC9D9F6E4DFFC457A3E49F0BCF62B32961C4667B60960452AFD917FDD00D954FA30C8533E5629F90AF85948DC1BAD889F91832DDF9B738254C9E7939726C37AB4557C2CE363C1391816C467537B471E5985E8084C277B62BE514922D352E20689EABB3EB91C343F36E77B152D5E85AFD088F4D02E7024B248A7420F58C7EBBEB480CAE39B56164F5ACD37A4F56B3DB6E1CC6B7C8C96CD3C44A69D9AC99175257BAB7FD83C5B574B5C9702C0FD13A5B176C60F82D2DFFF50C2AF25D96E0F8D27EC818D499E479B9642AED4A4A0E6AF5F14CC5E1299EABAE055EF3C763D1E350E2D76E92CEE47A4233368466A298AFB4CA108A325D2A4F8B79F21EE7349C1C186ECD7897F9886CF27EC01B05388870484867F84BAE2C016D04A3762241907C4DE207798DD125A2CEBB6C2982F779E04117BDD65CD7FF0361A59D3EC05F6D903B6D15554BEEFB6D40D96A0D4B37AE76C69C1B9592088B7DB878F95ABEDCB5FD5423ED93DF1B27D01A4DC9F4438E7C55F35B0AEB7395B08E1ECBF15CB2B61D043C0454AEAEF2D487093FA0D7DE3FC6CAF084B6A0F15A5CB05D9340D4E6763983DC45B7828539C77A60D5E081A03FE29949D916392B6D989B4C8C047E3635A76BA88AC18A7A18CCFF5C7E06B02A43D2DFF169FA449739E382BD020E0963C14A9ACAE6B6561C722D2BDA183F33EE6A904DF0207F5B098E56335CC063F9640C4997F593218D502F6B382354C73979E93C4B1B2471965FFA5A0DC8EE8ECA9F5697E7EF08DC0EEFD9AD75CD4122194B450201DFD73CDA46A7B2478DF66129FBB9C75B774213F9615BD990F8D07501FED440FD25D6CB912B8ECFA678D887A4EE28677E6E0491D49EFC7A3B34B9815C5C22983DA280D0AFDE2324F5281BC8B796DDACFD82723BBD9AA34B0C96075B36848591E47B80086897846FA76D092BC8BC6200837BFB5545039F8602B7EA49F63C0C3B8317EEEB7612F8E818DDE09E43C7DA76FD2FF6847906A45DA3D993E8EAED9FB3B1E579D8CC6900C89522AAEB0B4A80DA1E66AB8DFD62DFE4E4D77A3A77E5BD669207C70AA8537DD6D80A647B0420D79531A7456052C3C989F0F08DE3D343C40067680B39ECE95A17AAC8A622D1D5D95B38CA0F11D94E5B0A7634EEF4055517ACE79F0DF1D7C172E0246ABB2AB6B135EE1A38A3B84F86FD7C3CAF178CD4446D0B554256AD45C657E1192070ABA7DF480F489EBDF9753A79CCBC6AA893913C5F1271F1C6035" + "36DB0B5DCE98BD190CB139E80B71B49C7D7040B71C5A1F3412C46BDE939192B133824E8FA472BEAD7A4D3BF6DDC323BB37E0D8D50854522DBBFF2890D5FD9736EC39B00C9031E14E972D23B06640342D1A78FA5DF0EF12AA3236A50AC6E7BD8FB80C402EC98F22A4E2BB6E9E26E516E7E01BAAD986920061E5DAD927E4E0061A3866606015637231056063365487762585526887675878487477638236280364817056370582625626040150855601281021415375658360081814003238344132863410087754120308825448505583635861483488522781450115530724037288452180645532851376425811578032786032168836177052271783100201556873405285855840687261141784114410655808751130742173183267801325883830046480283005852511571077126111413273338305581727825323153811440104652051222210226383420042340322308056166352828043247278611463585302027847233562568428033166345443003065880210047365367315160748730010865734738305232222523752820711625615767042435315314503041375164660366121405180624802071164334230625718652833518622521854441353358223121352540300388484252536762043855528882400160748523278371774111874135436141440273310385340855586032187104381475057571182780222731565348534387571046724673888254604354385508308043028205873204570401180432163575734240531768606715214426833325634520763763723518740465145177515257128367252225710368810875341411301258348454232761048362867773260383180381738162335055515482041865381233334185878354587451337064781024554443887677673632652073866018425757626437708228375720344830144464713214160163503461453087771277385166532247065330167588150054362558464618363012667170148045255155505852527457605055186031538645654218602787417881724640247712454302818075713048757672721263432274814710043321053707037323076550400607427725460304754370164874174660112813541057331681152601054103866853711065347808675867280645076661507844584617431350801506735705655588557763038770722137306354434225726148356748145448405337674501367155743323121430777237601375131842781822541637737684866136033562874806077454652343344523376688865207837368261460663174085715881486277771105030663307516560865370605141151427575626557305834181601010676611140375585358407327605130777628845428075077042880047212574387827502731387663076364011400711625371477843100456442114008255062481181022270487235380661013734700364144346574117201004664624034383102461407855152244654114385024337324032207628818203135837764028525504622646060546321503604141638311777672736251736116485482155586726005155741501257214560667436087333810604210000768248328716753662713101822523612840613402468265615133652047073250607757568023357276561280717402160118308041601228257128106430611343883016085704472043883082027544828877406838744175438740321103025587868268600854465810776568573488081068834626204671561152001437524050818730116108867426350618732585768271733354605757872840002565184766480704586166333368630416538633768348004707041106457232184732732723056625067668604641302261520414588670616402767446637810213634724784500051171174854851110503637815161314655084024111051400267461562547284356765843660727856543172543618510316827312736388341513864280454700853333375821431615420043800422886563456470388125547022771880188550286833688023527412111250433820145308E4EE45B2C9CA523459C49BF1604ACBACCA7FEC3ED15FBD497EEA72B009E7EAD6EEFF7DAD65C246DC26DCBB2973BD2D305993E843581C4B2A4FF45ACC882CBAE751E63E925F84EA3FDF722FBBD9AE92F18CC4B26F188BB6B4254497704531041139AC623381B1117F51DEB204938C82B278E01AE4969EB587E51E5CCACD12F60E4AA2B5367A2CB34D2D442C4E124A3BA2CBA083FAE8CE72F39C7513D80574A06C02383451E653DA50C2FE71DCBE1609CAB5F6F0D1B7AAF21844466FA1AF79C671216792FCC2018E633EF173FB4CF7E767E1705DB9939EA2C185DC3AEC7BED66EE13CD8814D790287CCBF859EB646E58EBEBA9D081F811B653A25C8FAD64DD6FCBD6290E8F55320C51474E18882BCA350A33768B9DB737FFD1B7CA5BAF2579526438CB086A6C9ABC3FD387907A6D35CCA79F6F1C9801709EA1D02DCE59676E8C002E464A660F9E410E81E75A62F59A80E95C058EE950BF0804C5E8A0ADA205D71EC2E6E4F929AB17B47708BA00E4BF64452BEE8483FBB68627D57475D86922547B7CCEB7F7A7EFF98F46FD6451346E1314F8F694BA16C27234EEC848FB7EDB442D845E55825AB01B829DE26F76C275D5609BE4C36212F9837CCEB4628CA7899F214C9B06E7DB5EB9BC24522475DE460994A89C583A9D3C885C9B74C7D6271B8702778E84DF19F13015655C9D06804BB1237A17A93E19BD46232FAFF39D6E1A6ACE3495022C265638FC97B59B86A38EAA8F8A03BBCAFA35EED6576D109396203E19DCF1D151836CC7BF46C818B9959A53F0502464BA199F400E412B30902264F1E4909F52F2031D9EA8396C2F735A2DA81A389A792031418F2BF73A767F42DB0EE14FE1CBEA5A99B5B95C6137FA4621DEF9B4F6069CA4709242D15C797CDBB71EE507B22FBE95825DA5AA4D9BEAADC3C898AB122629EA717C6934E5A4170DE7EEB5D608684DA36A318F18E559F1DC9D18104271913EFB0EE8CD2379FF1B31C575C71D90A54E61185E1399E1AD710B4E960A7B41C88CE2F4889B0B2C7B3C208A1F14D750DB138FCE86FADD7072296155022AB24665C95A7719E92A1CF9FAEF424F90E945C905D7908013A0E0A5B9C7819BEC35B06DAD1A146E75D345DE5D18565A1D30C862C122BA4E87CF21CC63B87347B3ED90015EDC1B5D3125956BFAD93335495A74107B8AF62B50B4C56B48E2A3293F56DC7CBC09F2F6FBDE9A96CAAA2787892E3D04238D9F988125D8BEE2EB6299EC5C678AFE42D3441BF6E4B22EB6975E6DEC06AA507AA7C787D70EB97B60DB056D658F7404FA409939B8F4A321B7722F0FCA1404E79FFCD87D2A6EE13D508B6A193AF891E587E4F0E502D68784C55E9662B0905D912BDA6C678D14E5424781E0E86FB2BE43D7618DA355CDE8617F7254870F84E4641E7E16FDF3B6DB86B60BD873E8D432B95F53C4119DEC7A711CC8F2BF901E38B692A54E7ED9D8A2DBC4CF5F271F4BB1A4678E6A19F370486984045A1B4F8521C9A6F811D49BF7C182E8175F75CC1421486ABD75130329EC1C87594055A2F0773EBB3CE3DFE073DE8955F15EA3C10D75B6F67AC2D639C3575E7C0BCEC031F902E90235BA405DA6B2570EC9A0635AF9A3EB05CAFF00AB4767DF1C5B0C01DF45569D2F11C58061C848FA8D94CA4E9671FF94A440FBA63956EBD924E2EAA2A2CA5E21FA24ECCE4A1B82CEC6F55E5E2E6B1C2416E4CD1C2FA0DEC1B07C6E415A75075662CB6FFAE761AB7FE771EA54BC5F4A53ACC7B64A545A25627950057F336B9751FCCE479820BCC2CB25A92781AF13CE3572A0CF0D946798D44BFE0BFB5BAB4786E6E024971A1FF320B8C931329F8FD12E17E794058A809D849B333CD98D66FBBC18C216D0F01ACDA3E50F712691099336897D449AC15E77BA6FD29C6042D50ECDDEA4E9D578E2054B56B2645C97990B87251AD09B76F029DAB7498404C9294ACF6535E65B74D0BF345DFAC95FE020308A931BCE8635C78A2FC0F67910C63633E5FE9004846ED9EFC411235028EABDBD4FF10788FBB081AB51187F30020091486241900DA3BADC323AFF56DB7BE0DA078C69E535422E04FFD8EA8661C559BC311BD8D26EA8486832D839536F59B79E1D0443B938DD621758CB2EBBF41F87290590042913C761650481AFF1ED1F5A3E10B4C9D2B198DF8F2EF9FA7EB53FBAB3E04450678770FCEC45ED3B78591E9F643A53702C404A1EA13638455E43427A807607F7091E4FD7E3FD29901026CA4C724C1CBD2CC4C9F50AB48D02C451ECDB5E8E9020D214F3DAD92A6321F0C2B25C4D20FE09E10A4BC26B578C09E288A0796F97C04C45370AD42E1B920671355A6DFF3A10C3C9EA96F569A90E9380F5EE4A91EED1CBB644D890F0D755425AB16A92687D6E194B17374C4CCBBF924043BCF84CC6956835DE8A2A42E91E139169EC1B54AD1B7C0FB1488C8E98AE3EB2D5ACA80E1790C49EFCB98E50FE03EF3ED2040D10DAB1B3B69DA0A903EFB56686199AC2863CD04032A78290AAC30E600FB8E1DAFBC8F54F0E485BB47212A46EB81505890C67905D6900572F1693CF75E2143872B420C0348AD08CE8BA2F3FDAC84926B9E8FA40231E6EE7BE6CE6F115A62A57825ECDD19670FE5C236477245701D30208099C4430F053C46911EB51E94819F59404C90F7282E149E60321921D0D197F8A73E1D7BAD4F63659ADD3BBFFB5C2B71A4C3A9A0CE5ADD6DC9181C31A040B1D143891663D0361255BDCEB5A6EF9A414ABCE158030F8664B1C42242ADBAB2112EAF420699D1EC1FA2BFC9ECCA599F4A1795F7CF8593057C835127700A6703C6786BB75F1002DADB09CD7097E5FF978000115FA6278B912258E945E37461D00D9D5827C231A76B47FA58F68A363705E3B8109D364C89E510D84367D2092C501399250F482719F9C5F9D68C7896650AA91DDD17D0FA75160A2586C5F6846E411D7CE9D040114360FD948705FB60FF93C6C1320240DFDD132360C5B8F99384DB7E9147FD93D0D745A763A48A76EA7E9E06AA0BAC02CCF341FA505354606750D3D4F60B28FFF171F5B656C231640D82498BDFB874773274CA568CE2B2B824CE5793B86EF02AA7CEE6A027F27C89BE27C519C0BB356359A98A1C011551CC106A78E6D675CE68FCA86AF5F6F28B9DBF9B464A26836A328F6AAD6B01243E7BE9EE2BB6DB1FCC2E8B52CD64AC5D62000976A78BE4948F36513C48C8BCC0205DCBFA3F7F1018986A2734205690E26485E24188F6EA5BE3EB5E13BCE8BB2E748943D7BB0E30125011E074ADF4ABA8F1C24962372BCDA4622794D3C04FD99F5428EB2F45C829048E117CF5E3E53C881A8B3EB740B5BE99C6AA7CAFD7B5190A19FA4117EE8B0C9D1E501DB54C1C65C30E62E7787987722AC90708ADFE64784FE5D89C07FC59E01EBC02EF6E3FB86F44403A25AC4A276AB8A639867E9C13A3E7CB127743E5DE2695AC25AD2BCD8036BC601D663EEA17FBC26958671A116EC5055C53EA1AD7CD5F0685B20BA2BFDD699B83B9866190FC9364D34E3EC342AE3881B934047D577" + , KeyGenVector + "ML-DSA-87" + 51 + "A16F5B0796703E2D1A0140A35CBF36EFABE70E752BA59B6A9A0E9C4B05302F73" + "A5787E8044248F3F85AAC54E9469FC98F1B1138CC127B120F9946C80B96E3D89CCFE38C995645D4B6A559EACB2AFB81621D765C6E42E73031D44CBE74D322C7B16249576EB4C500253538D1A2C6B408E681B93B9014E3147DFBECF9D9858F7E8635F8598BA6847127D216A888FFB1636CC761616A0389C39A4245695DDE0C86CCF8A3BC5A50EA6FDBCC0A34457D4DEFE35F775C5993685AEF2237C31912A619FF804AFE8AC3418C13502820AEE5249D6E577EE0B2A5E3E8DA2DD30F514A076B50556D7581BA1F9B2E4671756A63065C20EBDA6EE2C33C9D97AB14C5F2204FC5359ADB2BDAA3AAB7AE1DACFF18A67D801BCB8F054BBC444F0FD0001A7908EBDCDF2B84F3C026EEC1282498D31AC33AE6A309ACAB17B70DC9F0EFBE52648D1AD2A4CD5964DC619B66CDC9D35EDA7A3DC21C729B9929024DB8B852DFDF102A086845702FD249EE43B54D2D033ABA95110C4F0A66EEAEE78F3C41D5D792A37D1D2299252A5498A44CA354F6F37FDC2F3B72B1A8378A5BA8B4997556E5A6F4125AD946BD4FC402C4320B27111BC204B8B5448F43F7E77A8166A48137D85584BEFE2D9C85CCCD9BBF3F8A4E05930180AFDD697DA82ED9F1069150FFC76578C941CDBCC5BD2ECB7D6ABF9E68327DF51C26C8B42CB1DD8BDA98A82C4C6BA6A991E651BDFF68F0A62D5DBFEA020D4303E0C53D474CB11D553C5BEE1156917D72AC2A6CCC4EC1C775D41EE660E2485A45AF5A7CA083DDDCC3EE4FFFC5E77BA97CC4473303C77D8B6FDB35E2A20627BEBBE327DD2D1AD1F880CA8EAE1E0067A9093929E5AB18D406A518EDD8C1E0F0AB07736F55FCECF4A3B2ADCE1CE3E080B58DDF85A2262D0803A7E5B4E485E642BA533E2EC7A43F9E8DB20F75292ADAC704395469408C15641A9C28B83E8C1C799FAE0652F51369978F4C089FE15DD78C5F560CD28F5F75FE0A39A60A61AEF6C7D802141E9809E7ACA68A38BE9BDF5312258704F8B11AF4262220CB55641FE95DF83EC9F786D6E69202C91EC4CAA4E38A21C3CC609C28B8F65552FE8334850858BBB30B837D874867EAD330A1F5B4D7F6BEC4748BE54A782D5B21A192BC00A7F2240FBD900460785D1971A94C587493D21CDA249676EDAC6C865147E269488B9CA78FAEFDC778FF60E79735DD5539879182424B054FAE8A9E153BFFD6957C533AEDC105E43AD7626312C8D229327D3E72AA9644BF3E2C9A08ABF807CF3A472C7DAF0AC4290A9E8F88D07AE8FEDB8A4B218C3EBBDBE53882781F2DE034B17FEFE69302B4975CF43E03645DC53BD355AF988B3A3D2431BF9D9A865750051EED7BAEFD1BAD4939C7EC293509A5851A145F79DCBEEFD195571AC2172AC6036812B4DC8040D186B8984CF9DC24F8F765166C6B2E8389DD24ED63CEE951442861669BDA0622BD90B041DC477C01A0D95D547E07A892CE1F26275ABC6F97702E03B776E2CA71E3D0EBB88D1ADF591122E6F0EC95A6CFBB976AE64BD0C7F074CB6E78E644DEE8200E2D626435907B9134000DBC3B50271F5A8F254D2CF02DA039D458E80FA13A33567AA9B374B47B799D0FF1A14CA92A50EFA192EA1641FE29A380E11386528B7248D6DE6AA90E1C9744713B768264B72A13CCAA94F3E19ADA5129E0D8155EC1B267E2CDC7F0E7A08F203D9A18F8C8D18529709EF746A21D5FEC36EB547A2FD4490092AD0F06C55EE0A1CB104E2C3C3CF2BCBCBFFEAEC744A13E37310962CE40E072E7ECAE223943A03002077D3D96C7B4C3FB1E2C9EE9C5222A63252F26372BC2BD94507CE5728CEB7A0AA33527E0662B4561D1BD255806450772EABB7ED40F787A2E3C664FA6BC3BE9BCD84FC7B42F16F94CC56CFB67297E475177E19E51010CFB74BA996BA1C3D793EA010C99901E2223B375E364BA193772B329D5D05AED959EBB924B698F0657AF43EEE8E0D54D15C93511209AAA9E10251BF81AD8B467D8FAEE2A440782C372F55A62CFEF408803E4B3BD8EFB94061EAB525BD31957779B89EB1C75AB97F278B3EAF99A05686E04873D7A685868D1C0F4510ADAB0B267FE4D3CB70C35B295AFC671B60C6575B60EEB99756E7B204A24D7095DF277BE18668E0F1AA5621D16F204575F76C3B13385BAF61AFF7F36D56111FD88FF093BEC4FFC26BA63720660B5BD209A9C14C0AC6B4E08B38FF580C18E39A22A9AC36912892537B0FD42800445DBEF1A03D2D8624C1B125519927C282EEFAC4AFC16128B07456FBC99D34C783AE08EBB0C46915971429FE64E448F1678BF76C7C20F91AD80E213E39AD89962AEA44E06EAB351C9B3A5859D5A884DE0CD767260CB60A96F508C62B731047CB1F3CA6AE28742771F4EC3BE45DF500F0132B6E947D2468AAB9410BC1F581328471B3E53E8E29A0794F4EA8BDF9FDB3922CECBD2AAAF75A2AF4C7CEBA03382332D39DFB770559789708930E4C77966C7364E2A4D762C122BE3FCBD37276EA6194071BB7C18E2627524F3AB2BE7C0C6E59F62D1F075E1951DD3352F6CA9762F243F6691FE6A9DCE4278B178F688E433B990272000F23C92F74A1CFF1429BDF1C3FE1B9AA1C0E7A58ABFFBB54D3F38ED93B11FB12233CFF48C812A227747849012F7BB85529D87459E11DCA9F7A9C3242054DB6BB93B48A47D69BA791B2A5D694A8776B22B8881BB2D192AB0A45AEC71A7171F844E3B1C4149D118EFA899E88D96A296D5EB811AF923AFD1A92E0C71464B90C8E5EC24954FE8C9BFB518A308B3D301D8D6C62E5AD40F63BCE24699AF0BBD2E48FD9AA7C3A1C597731700E3D86E16AD36A67BC031BF381B82CF40B1B235F51727913BBC311A186FDC23EE267739740CC57C36BE05F9331EA40279C34C44FB8FDFCC7E9CDA54C394CEF9C17F8529A56E3BCCBA327CABD07F5F9C567B70DFBD9D7139E39B7E1A493482321B11C881BA9D44CBABEA64F1AF18C18B3B5F98FF4A95FD907E112AED857B033BB7E0FF9466CF5FC691CCC1142BFB8CE81BAD7DB554409B79E23D0A41063857E4AA05583525E69EDAA017F6A7FA96C25A38E7A7D01E22F96ADBB683B4EB8A487202426D68D297B30BE0A803D4FCA1E037DB3BD63A2F06EFD68BD4D7BEAD8338D36AC1425163B3C739B86AC9D967ED54DE24CA58BC129CFF73585B667DC33F32BC4A4D9E106076849BC2574F0E5AB2C0B5CBDBCA59BB644BC86F3339B23B3B12959FC7887F7291136343E54F4A56C344DC5BE641DBCE62AED4D4BB958C4D54D7196DB4ACB8A1F43F88CDBB758546F44C59293501E0569F00B97FC244507C248031D07EE4DBB6A97270B772791E758E582989D93124AB82F15A9DCA469C1A0E98CC75FF683430036F16B4E94E94DECA06EBE7FE2B8E9C0A690AAD7A91877799FCF24CE275844A63E68F1A5C799BC83C7F5384CD9322E20B619D39D0031482FFED8224F6C522C8533CB29C04AF154D920D747D81616EA219E358385AA9FDB8E94A7EE5B53F2CC31B3A7BAC787E54AB9536FC42A3E369043C6F5C11D0F7D452852C3FB3F1845942186044385FB9E482962B1DAEBD2B3DF125D1A61843F71A272E3A1D97AAB97DE5831C56D16A6A6620F8C6CD0F4F1E41AFE6895BA3664D23A03EAB3531126E87335F4BCBBF2E39C47B5A58BA15066F79717D8296667553ECD0991F14D42F8934D753929F146E4B58D00A3E0139D66" + "A5787E8044248F3F85AAC54E9469FC98F1B1138CC127B120F9946C80B96E3D898CFB9219F5616156C30E9ED4C2C060256A1248DC7B0BA16B8A37C9412493D4DD2674A8FF7151B493FB452DAA920A5FEB5CE5E8D70677735C0543AEC2BB161C07E2C1065FC5CCA8EA34EA53C7D879A57F60BEF6D318280489CD568229DEA833EA03326C82482218104DCA1070C3280111834900B2691B3544E408061935600398441BB509C19285C3200C8A488AD4842D4AC08881B24811C851589845A1C80521424C58248E0C298899A46584A220114606D4C04018B704E2C085C91668A330621AA090D9183154A40421086E189408D34604DC24264204125B0221541401D4A240242206822892889890643884C11671C2340423978D4408449008450184510B31690B30900217641388001885711C89650AC770D3C870D0A021E124015C202A94864090425063142890C211044528002552642046982841E3346D220202D4006A2319500A40814884204C02529A468C1AA028C32002DA14041B208C01286CD4106493108CD3966CD32652E148224842005042889B4470DAB468E2805101262052404108328254A0601247255A928C000632819849D9386AC846848AB28121A16D88A085092530C2908D9C4040D14222D9A440E1045043165219016524354C1108608B4689100551E4008D03B161D306060A2049491850E1C888580691480621E486704042918B46108A128123189214474562281193C64904B66DCC2008A002208C284E90A84103B9640A058980881014957160A621591864C2029019154A13278D21904804800902440901A29001C750D3262913C9448C88082433629994518400118AB0700882255924404288099A407210B26CA12011912485A000520840600946899388911B46111B906C84002D1B1642122012CC3042CB806988126C8010884C9064C9125262886889B48190922123B711C1228C12B88908069210044513094502068C01B6508B0800148869004028620245D42204A0C68088B0004B128000388ED040844CB65103169221A249D4824101408D02923058B4241104650B0044134586C8A40C238610C3208EC9404A2393050CC08553164C11170ADBB64C593804D13040C8442601326019268A434465E03460E4308D22416403484D09384890900954800818104D82C27021C55123377043340408818C88C661D0404A4224880C26490481314A94211C1025040391D93089E0063181348D24388E84A82584888C4A40669A220A48120D1B012699266814C74CD24291244332228920980668A006260849890BB340A4086AD4040D58268DC1C07182484ECCA84CE0380D01166208C8409C046DC2A2898B0629C106312386300A9365E3347280A4616486711AA50121A581CC006ACCC02CC108644B304D931231104090914246024048DB182E611691D34800CC34890C888898182114018611A3601906211A053208044D4A40105B444914429111384413110442462223366D44B22559B0650B161163C62DD22621188408D2A66C0B120E92904510955053186919B16920A06524436ED9C6480126600CA02CA3062110C62C94C68118128141262408439009310D0A276A09040909482883B480609800208824199730A1B461D4382AD08680182026248984A4B661C0462D4AC60D09C97019C50408438224888091C8851AB50D44B08DD416289A8224C0864C03246C200546182912A046654224281110501A1961C9B48082A2810B0708CAA06011A920E216655A103094266A0BC52522046DC08091A232724C4621DBC068181701633085D44089890086CB2204224650D4168000432688844C2280911845722400054BB42C81B02908256AD8402D00A43088341010153009323100404E13C31123A420114921432860C1206D51C62DD1108A19339108312D5B346984022644A064C11681944465189500DBC04C9A2850C2183218B869D2A28100B131648061D0126D14A3242283711C0880C80050A1402E0A838C9B9028D296510C144DA1940880420810320A09198891B609D9B67123B44C903468D286891C906C00248A52362418418EE0C64580426C1CC77118124882184D502272D1384C0C14881B187100299121934544842D9B8210DC28285B2052C1346CE4802500448164044CC9222E14435010474293206519B83080122D1C006163108924C34CF692DDF651A0F08DDE43395656847184F33E137C04F3426909179E9DEF4A914BCBEB9A4530ADAED1ADBEC4D3AFFBB57A70BCC7FD6BE902F7B19A13B6255BA27AF9CA8028D67403BE3E5DC168244A989165899D0E759104B072114F7B6CC107D7514235468DD88420616EEDEA618B8B6CF692981087ECD26EB7C049BA29E1918456154B4B62D7F2054B2DFA8D56BDA46605726C47EDC9A3F82EC1AA41235888846C3C428815D4607EADEBD81AB7B72E6A5D92ABE542E2C9F1C64E3B54FDBECAC29DF0C45E0BAE8AEB2427073355A2C4DFDB8B633C77E9D438DC8C4786AD5FDABD817AA011C18ED90F4B6A0542AD5EE5B42224283392AF775B1122B8CF27B126AD2D806CBA48DD6D6D0FC97A3F123E4A3964F579A6345C2468D9F8C0A5388BEEEB15657A150D7C73D4EC6395CD3F24DDCCCE2FDE77F6335A22A79463E071050FCA1DA34E04E224B4C6759C54E08359A7B228F7F7936D45452B9A8E793BBD87EDA4F6D81189AA5501B8957CD54B9ED4690814811AAD2AFB3DD8E9464A829FD82CF4D941BC86A749C72F82AF36AFE766812F0367DCFA395B9A5D2A38EEC569EDAA793164080188540B070E1F9E3C2921C2115E456ED7B3C7EB8E4063517133CEA544E4B81A3AB24CBC4B58D2BEBC6B7094ACF1A8E3DC22198D1E85B42E80A09A138023D1FB2C9D0C4650CF2D3BE3E5D43F57DA4CA25F23D0E58373161810BFEDAB0DDECF0F1BAFD5229E07C5BF15E52AF391876BD2751C57F8D8BEA0AC50E110178DB04456CFB1FB7B174F5BD3471C5CB572B5B052F0525225E694DE9588A8372831139A5EB41B6A58AB0E590186472C151209F2D2BF99D6D9E583BBFEE0B2E2A5E26265884CF6D2B0F498442E6CFF79A7333FB0835391145F4075758B04DFF4B3F86B9AD8BE10FF01E4DDF067B6E66EA96CC9D335B36A4C9ECDCB89A591CA4834B5ABEF15560CCD78396568851B4BA0ECDB7831C5B27AC459ACA7D307D97868A8613636A3D7845BA2110781D200C2AF170983C5D773379D350F70F48DDD3B37D02E750FD0BEAC3951EE52C831BBED1B26EC3B086CC5E901EFB1B7392B2643FF7328210CAD3CC0529D9909569B7A77E7B459425E1E6E717155B3A7CBF2A62F8C5436F4668F7EDC652189FB54FE49885CDE2B35F7B7BAB41E460F99B3621D73E1C07DEE392661A0066BC40002422F2E3C0CCE1ABA02B1ACC0B4792F3BB9950DB6638E89CF9467CC716FD31BF08C70EF7B0695197656F1F07EC01797D84D332DBC440E8C8B48F3B4399EA8DE593FCD36A8C4AE69C275E69CB7C2F80F284B0A05B25F8BA4597CB56B18C1150F4127BA1054CAD65CEEBEAC54ED8210A61CAF049AD212995FF924157D8B6827F73AA46FF5534C7187BA24705B68F7C6C7998E60F54DD77038D4902CD923D137ED447C157E647AE91C85096A8A3A13B4C7829CC4BB5B05C1BDC2673ADA69ED599DD9F379182C7B7DE21DA504C7C2283B1E10AC2AE9EC092B95027F27F1364401B3CF9E6E2270DF1B607CBD7DA8F9AB36D340C9057D7A38DE7F7197DD75A4E24BA5525ACCFF7B6E0BE21136E08D6AED502864B8B6DEFB690AEA7ADBF36D75D37437693F279E57AB7E8048E031AB029EA799A45C46FF8491087372C787E36C9093A0926B15D383B4F8CA26201272F97F91AEA95B05BFF56D1D35E047E4D19B234082FB13B754BF87F9A998D94D81AB5EC9B7DB5F182DF41F9D64264BFCACC3810919F9DF780FE00EAFDE884A70F17E8913DDD14BDE92D915544DFC830690A980BECA539DCE75CBFAADC80291D6DAAD7C871FEDEE7DBE20DC7ABCC56E0D4F317EA57791EF8DA9569C9C4DB52DC5135E7806A2821788D7E3BB43C44C827AFFF07302D21D14AB0F47B0E74A0973E05471259BFD650787B7CAADE4CD7AE155F01867C624DF65A0F47F65F675B48AF6671C39AEAA0F1660D5EFCADE3212BD70FAC0AC342EA12E686E79CFDBC4C0068FAAF6B801C5E9B6C687C0BA9E883CEFC3D59263458736C6FD358ECE9EADBEB5DA31D03A87F3A5EF581947E80116696097F916808CE634FD5455296BD8D9DD899CE19C34ADE6CFCEE97F3D8930AB5D24F33706AF6A315C6B4021756F8D1CA3C5ADBDBED71D0CCF3AC15B75AE7BEDE62F08760D4EF03D584674B7E4B98CAAF9B05904CE4A30896D42C78D31C5D3E548D1AAC12BA1B4AF6D5DEEE2BEC4B9242E9CE340D2B5C94197C736BA15FA2D40FB9ADCB840CE8B905E05EA928FC1CEF280C68A2BD3FE1FBD4ED47443DA3F42F482DB6BD467724EBA128FAE13E28F325FFAC53061B9BD009CB00100B2CF0998F8CFC498AB723FB093FA5F68B573595E50518EF184FB04DB336FB9335781BF7A9CCB5BEF71AC131F61A87FF2A07A2131BC40003FA7170F7F43D10BB53368EF7CBAED8208F5ED1DB135E5232E06DB1B394E88D99D8E9E4997326DB9F5367F0DA6F26B24F8E7F772E4D23E0F4BE11C7F91E7F643DD932772887F50A82DC5C2D3B916E612B53AA402731E99A401BBFBCF5D3182B3D85126288C160EC5A442A8F95D4929756CFA5DE4B98CCF93BB9EDF0C5E6717D52651EAA58633ABABC4B10C0A0EA14EF796C3724312E97E4D2A7265E006A325D27517F11547B91BAF56AD11268BFFD0E864962830200F1D85F9571AFFADBBF24FBF69F1E0A658E4B3F97726ADA13D7DEF371855A535826BE77AAB5F03D520E3C3F3AAA83DF603019B635B047D58210BADCF9252857A49AF7909942D793069F441B1133C855D71F9C5702E0DEB6EC2E406A617219F928F9ADD14848B73AA14AF4C22F9607814A53BD832147060C797345D9017653F166723F484A4CB271DAA3CBF64D70FDF7110C1C4864BA12414F23C06C79AE70C0F3973716C114C7C64C8537D76188DDC0B5E5F99CDAC1B93A6D67444C3238F586322083EAACA34B5A791E74957FD3EED8F1A0A297B4F72880F9F08618F0BB92A599797C206190973E9DAF2FDF0B9C06F6BF70603B9D2F9233A14B14F263F0F9E0B200342D4099EFF7C405D18DE9FF10D3A7CED4867BB12EE110984CB51F46ACC1A63FD4A8577EB6B6E0A35805390882538F203F8C96E6E9E44F9FC0CF891BCA0D3D49ABDAE3E0CF6EBD1C5CB231EDEC40BAF1D8C63828B41033D5DE389ABAD6FF95E497AF0753E798172FB638B93D49D96766A1EC78EDFB13A8C6862DAA245A59D3C36C408349C62AAB197D9308420CDE0A26F77D9DB8BC64CED2349FA0199A977520442CC787F9432CD9F07279D19AC77D0D3B35B6E04508108EDE9842EF22AE5DD3D8B96D8DFBD775B2EB150FD70ED80F3F3259B289203265F7A4E26BC7855562481A7705CEB34F1B3F917A28CAC6EFE47ED3ECC170C7FF29961EEF0299ACCCB6696606AAC9699D7992099C6BB2B288EB17B5907D78319A15D0CAFB8CE96B654BE2F1200B722164A4A5A3289D314B46BAD462EC95E9D069F82652C3C4095314B3504DF63463D0CF2F2D746CE2F883B8C6A7805DBCC94004CDE0B5A3A79C3549B3C312A426CBF1FE3E8D46CCF15375959D8DE784C725653152DB3B97917A1951F6FDF313FC593CD8425D6A4EE55F321D27CCE899C4AF8C83EC25A95669267080F6CDC4B16DCB0E2D9BAD8FFD1F31590BFC5C82C80BEE12143C210B6C104F3E173438D6CBE0F42E6F2EB6A2750AF3A1380D2125D8EDE383E6162065745400B90AF9682B2BCE9FEB2CA3D187B148A3A592772D4BF135105C938FDC7904A799A5DB2D7503515A54EF84F5DA5FAD4D83369652A32EADDD555ADEB6B3218BF2932C54AC95B552D12E27BA99BBC9119645594EAFB5BA859AA77683E294D3CC12E8A8A88BC8892675F3AA1E166C027A0686B33323EA45A9274D51DB772B15C5D06A66E179FF5AB8952B2D3CF78627B47F9A8911847F59F99387970FFC27DDA6A24DAEF8AD873665C9B7D5C85FA2DEC1FCCE52E680D3622B930B42597F09B72DCAA7C9661306243531AC1A20A3963DADA81974DD2C22AB535FF2D87870C9B73CD95D64C41F4F40C0884D980FDDE9445F3ACCCF66D9D5AA44DDF823448F1162A99CA017FF44B309550D1793CF3BE8DF7E6FCEF5F440127799D6E6411C4C8FDD31E4D8CA550F0226817309E233B7EF9AFF3D1E65132740C79263B62BCDFA974682D3C4314D9033E295D0D712C2881F6A0E1EDEBEAF922B6E410CC9A1D2DD9A0CAE142914C4845B3AC60CB9D71852EC35AD7FE74CDFB1D1F4E5560A77FAE212BCDE1D6AE651AFF3C6B18185F8F6D32933DC524A1C5BE33146C8C6FA37B01F722354B9DFAFC8A4F769CD7B304A3C088366C4C7BC6D8ED6283E1555B4A2F7F6CF9C462AB5B4D52D8CFE937FA855F008ECE0C2C02EE3787CFDF8C16200619FA7F956F41696CA78CC0D691C55B018870D26F236FD30785D7AEA9F90F16491488E3F615ED6AC390DC14E5FC8032D23579C24AFA29241830C4D2628D790179B792CAACB5EF865997DA83A60AC8305B09C133003121F6B3F0CBABCB74A62A7DD2AFDD1501E2AB594C6E3FBA51D83D68A68A835C726CE42D966215C0BA4BC7774A439A1D6DD12B4578F5E713CF500EC7D456095EAF9BBFF8A6779DE5136BD8980D90301A1584F221989794CD651057B021A9D6E15C030FDF04EBE6BCC85353EA23A4C1EF70BAD07846D81B948761DD83D18B09CFCB09E487A518BE97733E799E8A55083BA3409B9139AB44144F6E283D7532284FFA52A1CE1EAD335F5AFE051FC50A" + ] + +sigGenVectors :: [SigGenVector] +sigGenVectors = + [ SigGenVector + "ML-DSA-44" + 5 + True + "9CE7AE44A9A0ACC05723F4A0DEC292C44AC1F951E1463B748B65C453632142D7115B92334F0D082E57F5E6E1CD28A2D83B6260EEC4929DFF296019A16E64285E884C5E92340A7D60B459ACF8868409B0B57E70FBF78BCD8FF6D437C5491ECC07D19CCE70FAFF8AFF70B7F88CF3EC5622A6775DF8D549759D979A458453E15A77534421C1108210496149382059866D4106511C9209DB820823130661226249304621444C22076DDC902442020D8C804C21C00823248D2115651B9341032522D010844812491BC04901052614420CCC263014274DC1B66092C02594102E8B220C531672D0C640A2A0680031448B24288B92245A0282A100900CA4080C3672188905C28011CB30321A8900230408D004095C322CD1346C11935021C06C60A08003962D224132C9883114148D80A66192C2655140650449501CA26122492C63328608144C0191288B48629410682212901C3680A2328A5A188280C8659AA62D1CB1918C487042C685C9169113286E4B3201CCB4494B204220268C62C0281316864422505012088C062902B2609A862D08218902046C18936889026E0BB071040980E41012DC325010A8815C44612347018326469A8208C2462CCC462A1AC741C1082D09A56440A009621666223049591661D98804CA080219468404A640539045CA184C5CC081DCA64D42A62421A9285814865904310A411119C484A4406498842422B70543349002476D84188AA4446A23B3900849301CA6208AC68409A211D0C070CC2272824045219671611672D83629A4A42418C2218A16011425108C90299B4025C24892DA4001C30045E0044512256AA34068632410DA488AA306421C9808194022A0263111A609E18091C3140A4C2685E4B80C2280050A452861204D11396048B28C1418210C11651302690CA8099C822D4A2825619204C0149010455210932520A56CD144808C10805C365223452691929019178AA1420C8A384541164CC9802053B02D0C3484D1C4219498211B454991B08C1CB98844C0215844318B36800804015C980893044D41B40C9832501BB88108837052B22C50963198086C203624D342895B064A62A87199044A610808191601043531CAB428D3044D1A178CD4A86DCC120D19980C83466014A24083C86518008D991210C406649106065C3488CAB668943445A2C43012B020CC9665A1262E08872824488E40426C133824180110044521CC828D2444501B8085C4B224C44451C8B26DC13481DA304198B620BFB2957FBD886D99CED5B18DE21AD8438A531ADD8FAD11035ADBABB41390D6E7F6771026ED82AB331C70124EEE1D3CF44F319FCF489C2B5FBB87C4D8F68913C6D689BECCC38CBD54488FB340E1848FB96E4D5B18C21EE9E4778C04C57B809835DAAE90FB98DBE672195290E49A9CADCDDC7EAE1197DBDCAF287A382F6C4CD6F45034388053AA8B13B9BA082778F1F4C689FBD09BDB6497DD6F46AC0F73AE73A6A30E5F9FC363E34BE57E71C7D593DC73664633DC642D779AFD2D1E61722CE477BDBC79F8FDDB23252BB28F39D1218DF783A5F28A10224AA89F92A5212198EB2F252FE4EDBA12EB3BF384B871E457EBB06D30E1A3203994A65FD34E543EA06C55EFD90B11EFEA3211A93EE663A0D81BF41B0EE8A8A1E7B5BD51CB44CDAA4E7ED9BF537000CB06BEBE5C12B3AA254472897E07CB2C4EBEA64450E3DBF2DE5B6C21310D96ABBA04463C1B215BE1BB1893063498769975509D7A15B7FA2D9F28242436D68F538F492F55A6DE5F94E35AC6174E5E77CEEB379EBABA1B76A5E85ABF01E67E69AC54CF004445316AE3D95061A5C71548CC822A1888C3587E8FAC74668C876104D8B0BE84D97C7643BDAA884A724271589616A8C2FA683647785F20384BE9A66BB52DE5A0EE74FBBB2B35EA18DBB14E0F7BD71AF19745AADA829E78CC79E3C3DF1F5B96AA366223F3F31F87838D91A2ADE2770ACB9C7FDBC5D19CDBFD635B631F341F44AF5FD90994DB7572F2D4D71E24A2FB4C7EA198B5CC38C4D8614B71D9150C59810B851CB922900F4F262C9F43DF69D6316ADABE40EA37AD7C20163F66BE50B7F8C68D5B1987F2479A1A572F7EFDB34C2724F09CB44777ECF6FF1E20438139D74470ED6D40369140A7A22B03596A6B63B5EDF78EB42E084DEBD08B6B1C9279FAE6AC596176899879824F5E8C357CBECE77E8294DBFC4EDFB991D5753AE2F3756C2DC03FE34E314D95F68420E526B0AD52B96B1D2A212EB1133D327A07B7A26371AFE8A3681B12B999DF4716D652630170533CCDBD5846B72F745FB7EC661DC12900874474FB3EDC5A1BD78243E9CF45C997CDF9F3CCF0CE4A395ECB736A256D3EBD8BB296380C72B0207ED7B0BDE9BCEC76D02B8B4FA0944E9B2BCAED9BEC4C19176755BCE73776BF7FB8460ACFF7A67934FA76BB4F8A1C6900316073FBF3E2A81B5CF99A9512F9B0C3781C075445ECDB344758A339FE2484D45E80B1D024FC940BA6460E768A3E949E1C53F8EE0B3F79AEB895EE8DD046F98123475E846FF3412BE7DBACB37F3B1589ABFDADA96471BE56698534FBF0398381625E3A60A6A15C0B1333773ECC9C831525A1C3CE9C291361592978137FB33313D9DE592D8EC292EBBFDFC84CAFE790A992DD4A94426408857C2A5A3284654A8320F065AA4B1C9EF409CB14BD22ADBC54D9C34053B5D91A7C7B408B0A4551B3C67E8CC73D696AA9711336D62CDB93993D16AC5CD34AFAF60AC5707E26440CDEEFB92FC836F93F3C93F7E9B54D745C87236D85E4614A838793BB7360B747BF56B146C309DBBE90C84010F07052556FD9C9675AC0661E5302205EF46564B0795CCAC0F3713B8DB19B0124B10BF534215385C1BE4C08E02A13F06155BCFC64EAFB44BAA1ED3D1FE1AB0C3CE4967A4DD8EBE2EDA5F611AC71FDCAAA865E47CE1B62C199B487A5BF34FFA886FC4E3326B186E23BA519103CCB4C418CBF9A12D1D2373CCDE405380ED27EFD47902F786BB6BF6C407A87EC0FFA060B74029C326806AF28F71777101C64516A187B039F66C937B4B01245B0B284775A063F29199F581DFEF7519E2BD00DB1B23CB6427C40DE16D473B4EFDA1071C014B7E25D2D5411E6E8060AB2174498BFE159E9435E74ED44346EE38012A7A9010620D2244CA21AC03CFFFAED7195D9C8C4F7CC120F3150D25983C7C080B80207353492DC534CF0FFC8892F6FDA0EFED4A4932FD6B4FAC5A2198488DD128FF64296DE42AFE9C605E06C338A69460A39EAFE3A9768B2A9D8AAF812E9A82ACCAD1DE6469CFD24B16CF6D546018FB7CF56A0383D0F83C65CB53A26B7C4B75267F4BE18632D4494C2AB703E087F37634C822A5FB003D560F44EE5957114C7FFBD1BD133886FB5C47739E8CA7DB7C72EDCBC6D1377BF5E8D6BBE08467BB26BA359B6FE38EB4D0DC4DA29CFE68CD570B472E6E08B2991F28BE8406D470869C4F53BFA0E2302EC097B060D68057393EABF3B13D010BEB9865E0341E5EFF46EA25AF98627A04687BFE9346E1CF6C087F4F9DF6FCAAFB6B6CA9356F626FD7191836ACD57A4AD5FDA57F4E9E5BBC4A43F2501366B043352A3F80E2A6B3099190DDBE193C37D2AD9D952810D88A0EDFA9" + "661880719B09183546633AFDCBDB764955F5080D3171C9B57F2C2BAF3B84EB4B3434307D97081F960C11853BF06BFD15061688E9749DC29796177ABAA02546CB9F2A68C8F07D60DDDF7DAAF30860CDF9E6CBD482508A759DFD20FBF8E6FF13B7048D9BBD64E4E1DB99D974EA3ABFB7911060B9B4DC4B593AE8CDADD159F9B65367E5B3BFF1DD5AA080B1CEAAC45E6AAAF155F19C527B9CB84D8D9BBF84618D67833949A569A9E601FD0C2F7904A6C8838C3232E760398CB08A1D69D8DE57ACAF2AD2518CE2F7DC243C52D6DF2B27D5B66579163110B83AD4C62B9C46CD2253D3EC5AA0DA5D2DFDA263429F9A0E52DC276336FB322A990ACBACAEBAE09A4067DBECE432074F39ED61AC40CB1A84E682800C378634FE40B71E62F0134B8EAAE32F5D238190D3E85FEA6BF973E2B86CBCC0F9498A7E6FC4FF6E7506EFAE23595A1F23838740A77CCA96B5F5F9FDAF2D25529E37A4E9C907B2E464280D9E7F15E064F7220FDDBE5AF79E33548521A9639783A46FC303BE52413261459B9E1BFEEB3B17A26146B0CAC1623B623C0B39132AFFC7299C6A50F7075968BFA780BC3201705FBF35D3625EAC41D913FBC5ABAAF208C658D11C6E4AFA4D75C4A45C0B9B763628E8B52C1CF05798450AB44A457C1D0D1CB31B3369295815EB52C4B9F7F14B8E5E4042C8B70EBCE18B3B46C65FE69F3A7D9C4E5729D4776AB06D3AD8D2E2D48BD11430E4B79AA96A0A4407E1940882262A979CCB7D1CA28F25F9636178C626921CDACFE20CCAD11A67CB74CC649D6314CFB39FF975F166A0F63DAF88B511367E116C7FF71D51DACA9111BCE3BD99CDD4811873AE4538FF9BD2EC34E694622B24AF31653172E52D4262E531748C388D2A2443EBE036E5B0769DF1DD4A943CAFC385491C4AC4F1186CF9A994F6AE9D8DB037C8234325B16ECB7C29238A5226EBF7DBE0111AE8C4F247871091A5B12CA0C07C30707F338AA39764040EB6399F7D8B4E46224F5D09E0A84857CEC948D2D08AE05C2AD0B13A3DFD2F6F24C96C3A9F55D9DA7781B7811DA70122BA7188D30688CD7BF1D8F55765C6C8DAD07AB1E1CA02C8569C4F5B9E06D2341F27170105D700EAD5AB373D67AB7629D320BD5351C72839BB7660D3F05E79B22F3FA1CD8F9AE89724CC780A04BE3FB67040BA17B90273C1F5A79BED6F24B64DB53CFCA662E76A8D377655C5881EE05764CFF8A7AC2FEE2BE603B5F9271F0E79910028E515A38669E76646A72D5AC4AAA0C20D7D9E474EDE21114AE85052F37D1E831F4DC98119714D96EFFA0D103D31EC064B1B9D2CFDB14DF1672F15D300D76054B86A8BB0B1ECDDABAA9BF99BD089405FBB512954D2D7EDABFD7A6EF71CF832E896DE9A7E90A857756F6F04EB5D4B6CD62631CB32C0F63252E81C5808624C1AAE8656E73C5F67918784E94A042AD6C85682D04437A7B93F7D1BF9E5222FDB5EB9742EC125A2EEA76B6972ECFF744E4CCE3F88E3C0068FDD487C892B07D2FBFA70E8BE5276A407DF65914BC78402BB074600B81D2A4B4778C7A4662D815439B631F6EEBA7B944D865DB22F087F5782AEA90B7F40892CF15FA33AE7C42B6492959ACA1E06B06513C641414A1BAAEAB3AF69919214E9D3814FFD9F1F8E5B8EC7A7CEDAF33206EB403C2069130A7CEEFEC2C84352F30C606F8DFD25F1F1A13330544BC60AA9AD7BF2BC859E7558617E74EE91C0C18BE3136DD883A0AEF9A7C5DB95C17589F0DD8F838ACF36067A39F22EAC1E98B77CC8F1CAD4B7E7383106C2F28B19AE4B8F2A9387483D8A7DADE825C0BCD2B5BB9951F4430C04B38465D7C1154CE3EF9FF9AE0161DA3208700CFECF883B4C10F21B0ED67BB8A8F9908C6DBA50AE967C7C71600CCCC4D0CA6FCCACEB069540F11F286FABC9606000067BC532A4A2D51DF80564C82D6B3961C7D836DBB8AE7556BE125D26CB8F81E13A8E78062698A2A6A8659A7CECFE047C4E3A3B4B419A8E7E81ECCA47497D82928151C731E8889C02237876472CF7438859A6A44E21033301BF608EC29AC61E1EACC67330875D22F3885D4E749B4EDF0FAF650D689C2F59F5814E8F0B7B245F72ED12950C07F2EE670E407D7BF9B4504F6C09E7FA380C3042D37A2375C5313305D84829EC9E4B1997604674CC712C70A5C4543DAF2A38BA16FE3358208A4EAD830DEEA8695914A4A9B81232F2222E752B9BADC372B441F380B1830042F5D2A9923897FBE6580B5287C7F1F01EB86B9AFE764DF333E1E61A09D50074AC9B6B9BA930EB5EBD2DEB788FE0A458CF4AC980262F9CAE907CD97E975429D2D09A322C20C13BE73991CA9B24DE6F0C5F7D6F32A7D00FDEC88C5CE51824F087C91056FFED04D63E7D7151350E643AEC61F1A288DCF2E327991F30F7FFB9588AC7D9FF5ECBD1FE02064C258BF8A9FEF8835C9E70AFB860C96F21D0EB06DD8AA2FC4ADF0D5CBD48006251403DBB52496A0B36D48A5B55D4A7E65C1C13D16154468AE0DD4214092E57E6FAC5835F47258C35A24C5A231483D98CF3168B152FFA28D5A084E1DE7F0D5891F89FDF2F619BFCE83243CC8929E70EC2BCC22161C05086D9F4E5FB430EBE796ED6840B2346964E15784B65B5D41DBA9CC8F09DF87B2E1503C325C5B0B0753498F8A0A6E483D7B25ABB9C90D55349FFBA95ACB94357720E7D698FFBF8F009B20921B9FED3D74F488233F56B8AAB7E92BF1EB5B9627E5652F7747E34CB59E64F26273C54E6081E704E0F3899A96A82FB9D5A55DF3F2FD945769F07F1479E57BACD9ED50DA71CED14FDF5E93CC4CCF1FF65C6D22214B3EEFB34E470D41C1B65384BFD669B69E8C2707D69932D2B0E2AE773240DADD6A2B7343A3EF56A8C6B7E9023154BEFB365237373A9C5CCE7283A2A4EFC3F9AAC894D96EC1A018D5C361943367A55663979E4B3BFBA362439A44E3BCC52A0A5AEB40B797B84689719E22EAACDFF703663067CEFA2ED4FB802C81B55B21A708AE0E742A6B1B96CB66886A10E7338EC55F900563CDC818DF15ABC33E73AD11EBAF1A1E3BA546A37B942597030DFAA7535B05626D1D4612BC4A19711E873AF0FD9B780B8E86E9CAB1FD22A9D08452F896CA5A6974AAAD63AA98A5157008A09E4E061B65F68161ADB47A0D0A70F0BB615F9979BF1578F718F16E8CE034AE44CF07939A8BE5A28210797FE26BC3AED23788E9B3B4C7B831D88201188891EB2DA6D5C0B94328344B035E41674751892A09F72ECB52452A8FB0E6B2C020B5D051E0C743F4EDA8C956544BF8DF5779189B5231CE68F3CBE4EACABE621C8E98870C2D98D06FFCDB5BD8501297473FE54E22F34158B7C20DFAC12740C049811978EB8338C4959C1BDCAAF99066EE8D0C2B9AF7E325F7741A34F23E6C7BD9864568C6F114435B8036062938DCC20700292DA31A468EF9E06A0FD68444CB92351577BB976707BABF45E186DD166183E0C946FA0EB0FEFE01003DF8DF02A161BCE33F9A50A2D047133C5409449FFA1E62DED935B6887564BFC86E67BE2AE7034F2616D724C452B9FEFD728415AEC67F67EC13B605F097C3BA546EA36CCF4495B079B80B0F620A8694EF264E4C3FC92E7BCB36F2B0A78D073126392F5AE799254BCEB3EECD324AAEEED45D92F7C8F9F383B99AF5F480647468754D5A10E95AF2721E0746334CE806C12B83251BE57C7E705A08A5DA1B335304E73BA5491BDA71F9798874376A4D9C6F750137E3796B942A8665EB5B546BFA824EA070D9478BD26376AE7789A704D635640A4615C591B56814FCC4EE3DE897E1021B3C4082F069BB26FD853EC3A9275B542933FC4D0CAD8AF103D9B146F960FBDD3B22CE503A6EA7F68312632CB66BEEBE0BE79ED4752C34E99C7AE198604E3960A1E52F97D5673BC81861693187177BE572218532C1ADB26ABAA3C76626C067FC43EA44225150EF911A60823796EF8D63E11868D83A7D5AE60770F8B4F85638AA98F011047B5CFDDAF9CFB02C93C9EBA368CFCC5F6B00FC64FECF7F4F3BE950CA9D15F17292E623AD77C3D936E249B586C0249E5920C5B09A72B3EAAF0A65AED2F692764950BBD7F718ED252BAB7F18E96E89B4EABC576D035F423E6AED3C9A9CFF74A60B231410851F5AA08DC91FECB12ED7EBA502AE57AC9DA95D6140E751C97AEE4B882047B14" + "" + "" + "DD36EBCC94CD8834E17132B10458E43557357C09A5B855592ECB412BC27480F212C4105527CD9919AB9D2BCC2EB74B5BE49F964B7B302213CEF8DBF1617554F90BE7547809E1E061C0494430FA8B07972BFF6999DC645DC2ABE9059470DF537F6B312771688DEAA173FF211BE5E7D8F4819E3F45419BF79BB51600986351B62290C9C70157C0F34968FF50DDC771458EF8CBD9F61A70B821BD834E4ABA0ACB642C3A0B83904E3CCCB095807BD3CF1CBE20573DD9C3E044EA605614995FEB203B84720DDBEFBCD14E23564C4DD046E8C6F6FAD8638E0B2BC8967360E3FCF632715CF1C34DB2A7C183699BF44DECA5FCC427CB90819A1A1B233275D90C35C7889E35DB6F725D9C8812358E306B942127723A6A2E1E93ED1D2E2BCCFF996EB10F36D5B595FEB3A41AEDA88C489CDD4984E96079A29371CDA0C7CDB2FCD99F92BB4C8527E5A8471DA0187B507E7FC4830F93D2FCDBD1C07E6D542B6706EC7B1CBC5691FC1585558758014FA15BB47A44BE8E6E1FB8773E57137198150B86BD2D557A4FD08889EEDA4F0FF3F77056620786B306F843207942F6544688F949EDD984C8D2386795BB1E3B5A4F0016755EA9556277C8BF3A85F6E38D89F8473565722E8C80CB0CA1FE15C99A8C9D501841C316050EAEE5935A400899378C576A6826539EF7995AD73B37F7F6FC972E3663759D9C7DB2099747C3373969A1315EF3987BC8CAD92CAE546EE776BEA23C018B169F04D5D5A229CA19025837E20A0A948D147A9F4A68CD7D5805D7F65F0370DD9A7924A564921CDDF3E6F62111EE7EB7082B4A77AF80C5BDDBB496E3E1F8940620C2515B49F211EB3BD5D5E0B06F554E279F672C7812E82E8E1C80867A0F0762F9ED3DE4B22A24E1D24B4C315CE98F33F5754EAA1E98EDDC50A93BF5DD14A72AE454B61AC9C548062AAEF2E8996433DACF8E2238D31B434DE66688ED47069465DEDD6EE2C0E66DB2B24FF3C89A0FE8F012EBF6778C03BD9246B6354229D2ED78F4CA5E96B0A774CA9B78469373E5BE91BABA862B5A60F06889BF4346D93E08C27685CA5106A1D3F621EB7466C2CB4BA29F3ED03B06FE01F720146C5FCBB2A2F2943C9A48F965CC574DB9F36799783F728BD807FB78CBDE55595C14A714166598E3B5F78469A9E6BD38432C8B802FCF0A1287E75577EA1CCB9069A9E7C86AFA5B2126BBB17C1FF8AE4465A5A975F651FECD18C9012954AAFA9F5108310F2E05D401BC75E20D5E56305A9C2C663B805231FBA93B99D39E03B29EF65C91B253F0F80EA1930D5CDFA907797AE8120E100ED6F8D7A1831D95EAA28C9FAA3D9BD272D3BE1FB6C5AD393A515D842FDFBC78BF240435A2A477EEB2A9F03FEEB74CCC1D1E8C1D16ACB01017D64D9D6BDF43564D5B87F1F9C0850C90B745162C875F6542DC91E6B1BE2E2419A731033C8289AFEC90075C0DBB3482DDB3B3F45F04AFCF5062C3ABEAB65C63CE882353B3880C6CCC2705147CB9C1AC42D3783F06D1FE216BE45CFB87A67D230A5586CFE24DB1F682DF0F14105FE5015B664A898D9B14F517D1302A020ED92F887D3AC57D416572D44AC036509B13186C9317E0E9A97912FE7E2EC2DD62637CFE24390F51337776F6D62437BF832964C3D3697D411BDD3B1715423E5B71672980802CA2D00373AF55073687E65ADDD398AFA41A7F7423EA601839081D635AC5BE70036C2D0920978CE709560D4CA292DEE56D9F094C8153A9EB94D93300B3174A400BFDA080F66F39B06FC4DF062B58DF729FF3065B7D720C1AFB2D4ADAD20099505B6FC8F7DD3FDF1AECACE3E446A899D9D747F329D2FAB1BF9EBF73D83AB7D4E4BC70F5921539075BBEDFCC80598C50A76EA11BE3748A2558D85D34ED7B51C9153CD19E5C184A4515D1D1577AE078CF9E345C0DCF41C9773C6D5F25B7D8BA39C297FF2AF9D162C12E283A213FBB4FDD2D9A42E12254472E51DBD3B5E5D16DF16ABC48B9DA8D34D4784A62E9E7D57E9BB215DDE58B45EC65E2466BCB1F5210EF830EA1878713D2465E54C133486FA91C2F5C215AB029F1C1613FC85DE03F26F964A2C4636F55ABAD106D96B4634BB11AADF825D212400CF7059E189AE4E80B373AEA6C8D0CAEA1F545810A6D3C5B1012322BB2C730EED76F357AE2A03F3EDD4CB18DFA56643523B6F6528E83E31851579C2A214AB224A75DD98FA8383F028194E2BF4EEC96DC0EB9F2B31D9D66E66F50C84CB827744CC0D58EA770F2EE5ACA58B9594F42F43464C252102C3475FAC5E11A79E29F1365F5E72747B95FD99CF5FFA099A2A3FE6A43F8DC1ACA0D81361B4DE017C61F338F768D63CC24DCF7C4F334028A27CDA1265E7FC6703D63FFAA0C6DA10C623D8DC2AE72F4C134AB9EC1C8FC5C90CB6D63111594F164A40335FE3EFF9BD6E81969A8221D5FF3CFFDB54D260774E02A47A4A2708B639F4A1593267C5CEA8412F61A5780DAAFB89D04E5112B6C2C709B9279D01BEE877878EA15C14DB7DEFF1D64D74AC6F204730E9F24DE21194DDB88F0069B3F1E2014FF838A7B4BF6615E7C68D03638720651365EACECAE7A02B007E059935FA99F0C250E8FEC507E02EDEE75DB93601C5F217ED7626A5D43C92DF6AE8A51E1DA5F5368C897E996D0A986A0C63F0D20965EF56936714558F962F43293FA65F3673DCE9C13554919D8D698F91414BB85D0D47EB43160BDF26879092C6D3502219B5E499FA14D1ADBAF53D8B16A1823FC5CECE854E337BAFEDA725CA1C00E3C6F79EE2CFB8B87CC45EDD624162DA00CD3368597641A639219A77C62CA721ABAB7DBD28206ECB25E89A9C759DB2E181A5DC115602F3AF0453B1B04E6B449510FE42CA9E50234117858A0F3DA30221CF2850931A1118094820D77C5B4BE7013B723D477326F8B3172467A03BF2CB557ECE13DFCFFDB47FA6B16659336FB8873082E14E14D558BD4830337265E52A95F5A9A0210A737034B9C5061C1F9035EB0792444C5D21016E7CA779D7612DC05DA6660BC39192D87486D8D388083E0BFA40593ED8CE52D90F95B5E8D67CEED92B690860434B73D20244FC14A678B53F313F3806DA39F9128429F99997E51B1904CF64FEC3789E112AF100B77D7EC2495847721AB3E0E1BE7165E4C29F6B4267354B7BD95BA2448FEDB4AE4DCA5B43519E4C5356EB5CCA226148517847751204A764A8A8C4C073C788F98A2D1B57E9EF2946F7CDBB042AEB7E4782F9893140D865630F525C816FE74D98C22C124DC8ABF54F31068189B3D67458A9300BDAFF9A875484C6091BC4870702E38B1AE2CB4AE688EA324629CBB8006222F466283898AB8D3D6FC02051F212B405D8D8F91929597A0A4C4D2D5EA070A193E4550626674819DDCDFF3FCFF1121424F5C64666A80878C93E6EDEEEFF1F70000000000000000000000000000000C1F2F41" + , SigGenVector + "ML-DSA-44" + 1 + True + "D10A7DAADC675BFFE61DA485642DA00930CA6206C420755A687F6F9AFDF547D19DD48D7F417E56ED9A333584E14284D4B7B354A4A9162F6BD09664115F9BF3649A997870E0609AEC65F23FDD5884926E5914FC9A03A0D2081D9234C5A230F2A767A0BE128F439DEB23341D1A67FFED52E6146464C19E662C1C8E770744D8FFC612114D1C1222222664D3102AA23092DBC4305C202619A690DCB26C13040C0A2126C9382E1A0330012685229585CC226ED3844DD12012C224712242691A294922C428094988422268DC8065C4122E18A14060444C9946285C904DC3C2701403315AC64CD2148DA0064A9C3441DB288CE1A24588268448B42051440CC318051AB890193351A3C484609630999230E1246C01879058C82C14108C0BA44023282824880103117260C220A2C685D8442CC304324AB420800222DA8621C328261A054009480C42C60591182242962949482009266504A52980B42023A44823444D242081A292241017800B930418036AC0286910C9490C220414460512024958406099B610134004DA8045D13660A1266689022400054012A95084344A08B4294B222062802C1CB08419846504220059366501153161068619B02C1C3904C8A028D4340A8340400316860494000A164C20B1294A188C51A49023212E4B3644D2B80821C5640B194162026992802012B66011990D400201531472A290246030691B342264964800B62019080952008C538208020384C9B28460A86802263122B621A186898302066310250C132CA242728AB20C224671E1264451A82D230321A2202CCCA8611C498C514661E3900581C01159306E0244601117004BB2404430709348024444611A922D194781C4A0906346920A304E19280E244842E1308EC2C04C0C974808088283006D082631D090050841419B308EC9C4315416300B875082C2850BA909D9364D9918666146701928890B1391939089DA060823898D5C8444209980D88625002266232048E0960441926161002D8B986888C26011A1289192445A101064361018380921076580481212B68C084924583009232691C29450A1168419286C9C82491318240837842292811A304621B820114142E1C4109BA02450A4650129411319915C806D943870621021093411013429811084529830C3446413098A0CA30849262909B69184280C11158A08C90CCAA490D8B84D22C38521392EC4A86D1C1826C11070434208589201C4C2498044655A22685102924934040380013E561E982B8CA565AEA3EF00D4E294F5E7478BF127A82E36CF32813D6A2ABF054B56E3D0B1B52E45E11F2B1EAD362995C40675AE48304C1BB2534A1BA1949045FCB52C3E6A41FAF37FDF22104F68C09436A0CEE08A42FAEC891A417E0DCBF284172AC4F94F2890DC1D66FDF1CE8246B2C237153020B445F7F6B34651B3F380D1DCEDC4C7182962748C24E2280618CC7D98D45B57AF73241235A3C9492EB0B9F4293948FF40BAB03533CCF612023410FC19C3B33986E5C5B4892A657E32F79544AA1FA18271F4B6A92B888C968FA4B45196F0CCF15C4779E41573F2984CC5C34B57F9D3C758544A9163B9023AD4279DBA755B8092F311C4662F33DC7B14F903B2AB102E4F21D59637D192CB8DD60A650EDFE248D4452662CE148A5EAEA25664D5FD90B03A41DF219633E83C614D354DD509190A547E90E9C648902E64AAD4E086A9C7B29090658C1524BFF42AEFD64DD571DFDD2285FE62D9876FA77D885DB1A9D4D0622ABC73BA4B929D6E2FC98334EA4D4B2C190C7376F6AAC912A38E9241F7D08F4058161D8FFEA1970A53E2FA135AEBDD221470D36225A2A23C1E8922951CAB2A0E0A2628991EF72044A5EB715B3C7538655F34FD0F03002B9C15079CC9F8BF1E17531105E6E8387B70C14E3AD6C9F1D83354903F564E53E888DC152165C6C684868C150FDFF73BC39E65350B09D54DE51FCFA9D43E1A36FD0420642A78C628ED92D0734C30C2A544AEAFA81344AEE122B2D57987F244BA36C33890234BEAA43DFEAFFC8DE0EDB74393C458C775991D0C120CA1790DF28168E3F607C4CEE3359014800D8EB6DF570D591C9FE5D3E3B5EED9AC196BC61696926AB48B7F61A41D1D236517B01638508DB17ED4834F3EABAEDC5B0E31270119F6319670DE5715B99129EFCD92C83206C8086459452BCDB9808C114038510055494CFB652526B6763BD58B932E3B5A1FD24AE3F26E15F44B609CF97A204FEB07EF9AF3519795508424A1ECB061A3DFFB121BDAD5CD44B519FB7C49E9FB234AD76EF3CEB54EDFB13D1E7FF9DB99261BB39513EE6138FD5527CE67A56F2F688AA0437F1C4FC50670FB49268E48360538C0C21454F79666751887C5F98C495ADEFA4089E33FA1806019DCC25F6A20BEC3A95AC2525BC5A424DCD0D0DDCF88A59536889D836D2397E9B72A9DF58356567C87059B8FBB830BC490215F3B7FD7F01576F09A815DB55B527A9668ED2FF0AC4AA6D4FCEE920F953D292538014DBC57DA9B01F3E522F228E6A200E22A31E28DB9CCD5E672ABC99EDC5950909A02FDEA10B92C3A1D4383E492A731B1E9DEFE6E1DCA4625387E96BAFE38A02A3ED15B1637A3F72BEEF52AFD2AD74386D7797657C450D36842CCA8F3E300BA699C5C8AE0F067D584B9CA4AE44AF156B3F4DF7C409C6A1FF20008B95CB4D20D505A722299A5F96F0857B435602F166AE8C51ADB1B0B65C115193FD66D51B7B4E94FA659D78468FDE4C4BDA59DD5B7A45D9BD178D7BB3ACFF94CECFEACB3CBCC93BCD0C44E064AC885A604F6192030192BCC7CE7BBF8B221DC377F2548E4EF2F86E5762B92C619C63F575AD547CF9E32E8FB145AD80CDE6FA28473635681C97BDCCB85D6F15FF0C567DF09B5A2E2683CA3FC8096BD1FB6E36DF32B4CAA02270AB35D1BB2DBAA7970A6273E2BB4FAB09AEB66D40BAA51D1146B206BE11598053E379424A64DD7EF818DC7A4894B6B12DCCDA271A269748E062674B50F762B4401FD54288E46952ADEF25D30BEDFF1C8FB153A55A98C83E12FE0876C4F52D5E9B2CD986B71C67F46E05AB0201FE635582909730DEC0DD2A4B1FDA1BF6BD7FB118E5490BA54FD01D394E1E1B80B4287B785C9226190198D8BECB7296D85422632716C5E7384213DBC3C27A8305B80F5B3E91874020A24123253E3014F4333A296A2F61C859D34E980B0960E4628D2D1B1DA13F603EADAA5A6901BFDE1F8D1CB92DC0DF91A8D6FC7A65736516A3E2FB041B535B095DF6278A39DB66CCF0B365ECE9C9317CB05899B90F7E4696564A627CC9D025EDA8C54DF41CDCA82F95E247AD577DB90C50F579EA555C9A4F7F538526925B30FD876E85660C669FF4771A8748FD6B9AD434E19F10DC6E28C3902BA2CF5516C7870DD03F83D1098142AF43E29C6CA8C6E945D3AF983EE7C15C3C87044691BE225AA422EE600D06F18A6FD394317FF10C391CBA1789202E4D6853CC27518B849F5FF16AEBE26DF3F85A005F2A77001A925D1C67825CC64FE0092D90C2B28541F9E43871F77055BB2FDF0130C5254B03912BCB937E1E0BF2180D0F48A84A2CF093D5270EDFED0AB141A608C8A52BC6C4BA34128FF377A4BB046D90CF79B" + "636C19A0652537F75E3931F27C7CF6025D78CC759B8AF4595563A320CEF0FD67FEA0883A4564A78D0C9401014BA1DC69124B77C5636EDDEBA65F050DB3438C603A7D9359E6094ED43D30E378759BB85A7156635A5137E492F96E439339E5BEF00D779C03FC4DC076A6C8FC4700D659EAF33EF2713399D7D6808FA6A680F49D3A77E44FCF85AB9964C797F4280E6421A014C388D61F36D45C4722AF95B1870D9AF10AAC22766EAE7B1512A708F61FF82F25582E8D8E8E442406C53E3426AF766F9FEA47079447F843511665645ADB228557D54428F6AD85391140696FB100927A7601E319A44F056A81253B14C756647035F035B829ED708204EC5699B17A688D2B2DBABE4B58768C8E6E156730892668DA1BE91A8EBDA730C02E711BDC94D8968D6B9EEEBE3E8B1F0AC376D11E31B17607332D8911F5F9C91B79FDD677804F053F3398E1002D9B07A4FA0F8786C2AABD05E89C2B9918CA254D9E1B5A46301A1A4AFD94C8EFB75EA35FC6A2DB1B466736D44407919B9A02D4FC87985ED1FBCE47D62BCDEEF97E020C7A37C69BECC7ADE6FDED561577F04517EB8E47F1C3D9DB25628FA2C82C64242A597E82E1B1ED6243795C38AD2031D679F3F9521EA85B8AFC1877884F0A2642359A76464D6A39113DACEF4D4ED5F49DEADEADEF6F99BFA0A0CBED1AC32B0F33C6AA0DA1BD1BB426E368ECDD49FB56E9333BB35F3DECED433B346202911C7A2A8A3948A4BF27D89E3DEADC5CDDFAD502CD7E373E55996C0F4FCDC896BAF211BFC5C9D81FFE24686A7909F8AAFB3C1004CA1CF4A3DB3FF4625847C9279252A73694D799EDFFA01FC222137E49FBC55619F45AE86128A6965EB7D7BF1DE78A7DF752F46F5256055FB37E1D4A729EAB3A1865B0B1104AFA2A0429EFF35E539DE06C861CF535DDBBC9A66A325B53ABC04DAB3612BBC7F60414472A4D0D1FACCAE5DA71F533EC56702A2078A8145E415452370E9F2839785161B9094CA608DC414F7F919DA8FEF75C4A5E4B341958BD1E94D056E961FB49777819F32B49A1CF76549100D27CD625A820709B71B0F16B55E552ED757F30D34F510E74CDA618CCE5D0FC2584ADA3708246D51A344A9A86ECE79A4415879F737916AE75ED586C7D3BA5E60C36DD52EF29687515F2AAE9FDC6FBCC39245471CCDC3302D69DCDD8F75CC7C51E486181B820BE9887537CA77CEC4B42C64FC84500C32E6DB89672AEC2F0C9929927D64809C11C02695435FA49EC99494B8FC8C1EE61ECF5916A43D42328EDBD4EC693153696D28F11B7A7ECB9C0D7FC6B95D6AFAD18582D3D0DDFDB502DB91141EFBB21E904DF223E76D9478F290271469402DC392EAC09E87D1A15B91105EAFE3616BC311E916D7210DD5A2141AB572399EC5618B74757ABEC407B7151331CD1FB883A81260A33EB1A94818658F628F88C67EC2ED7E6877097E23C517D87069164B31869A3E38A851B2EFE3FB8EF8E2891F8F957C4818413F1DD65B08F4E1918D33975E61EB9F185BFB1AF5CFA794EFD93CBA097A0D9CBD99A372395694936CA8AF8B806EC21F431909931905A1E4A1573CEFAC618A4118B40620813EB811EF0410A7B448EC1CB8E92B83BB1640B3CA24A0F16873A78D4053D46C8B8AF87EACC4A959B97C195248955353FEDAA4ED2081AB79AD64D6886DCA13CDAB24B21C3D1A6E6FF3713575AE5E564F0C055764218F914D9725991B249CCAFEE1B7B0AEE46F59463EC66CC1CF42AF46700B2538B905756BC3967B86509D36603304B84D94D64A313E1069A6EB64D412BBF9A85B0D0DF3AA0555752E909756987D5826A3556B3028563236BCAF8AB148C51C2DB735A3B859A81DA8C50EA06364CD0921ACBDA93ED9A0EE0EB557ABB7AE2A83CF040C13A19EF4D44C9434D11664E79E585728EBCC942029CAE742AD3DF4A1EDA625055B0325605A887A4D5F037BD551629590599BC0A8D45F0F3911F97D5E89E3384CEBB65A23F8B0C4AC65688F2904094C2A9C8E9525F310BBA5271E211F34EFA5DE2C8FDE9D9D222EFEC1C3B9900426574318A24E3FC164DCCA43E75627BF86702F08835F6B5C54DCDACE569ACFDB0F9C7D45D0A5A50EAE618BEB2AE26A0CFD455C8394AFA0A39D23BE575CEE22E0B7F536516314F4E303B3A824030870132437BC798B87D045FDD05AD588A8C256958EF3AB059E44A9CED81B64E59E83F80946F01F34B87FBBA04BFFA7CEC3BB1F189DC296F0796A2456F131FB337091ED88199943B7DAE45825F4246FC29B9B00D35EE5777C0C11929AB7FA290095AFC4ABB23359C49E6DFE153EA3121BF4EC411A4290E3453D9FBB736DE225D8F986C9063054AEDCA54374322AB51BC785007CEDD312975EB6050051100F0812A028CA4A6A971C71CA142CD9FF994A3DF0CB7265F4C601B3D123AD5CD668F0D21D73DB64F30C83076671A3D4D668825351D6CA00D125867B9DBD56FDAE82F36DE8F33281FCAC14CACC539B7B319F7128114BEFC9D6E9D55FF5C81107D75F6AF1FCE5644790A3AF4190A0EAB6757C33AEB53E4D0984E0535D4AC3259BEDA56BB70A00F7753E36566E5564AA2A3710231B53FEAD43067502E45AA27DF701F4A64ADB21C69887879D6271538E87D4ADFFC8CA278083F046E8D0A98EFCDB98D7C1D3FC7ED836D8ED42E4829D89FF45BC63577E16CEC3D892B9B3689F85DA6C20000B42D44437F57DDF88FB6926A9FA41603F5968FBA8D304A830B6DDB4B8664E4DA9478430B5EBCCB079CF91D880D370812D47FC18B9D9578F838A2004B5F7AF2AD3E9FA2A8B6EB82B7212C50062883250BB93BC57911237A3720FAF7E073ED34EC00460C3243C14D9AF8D3AE17CC99F2308274389466F2F34BD2E56A04867A8AD0441C9F1BB3A87FF1CF2C6B056E4D1C8BFE7D5BFEBD79099F61C0DAC3B31CC764D81D64E924F0AEDD72F60D7091550BADA81D204DDE63EDA906C30B6285F2E8DC9DFD5A79FAC5A81ECB3962AB2210636D2260B59BF9646133975FAECD0F7917D5A12F80F5C22D0E5EB919B140978508B5911ADB9DC78A0C1DF21DA7082462387C7AF9302BC5F487FEFE7D25058254513E68D32F47F691CED25B46E94A2B0CDCDF1F3394E8C37188C162D22F965C6764DC68D215004157B00FE1F49C4EB5EAD02D13826CE631B9117DD3719FA7A1AFC6F7949F34FC3AF0618B1AC6A1D4EE5C320279D045235EF87C9CCF50F1A3E3CAD582C98BAAE8EC4AFE59A5CBDFAD56E0CEC415837176328F833E872146394A0756F973F7B5BD204885EB4778B3E13703BE15893BAB3AC4497A7E48CA30723545D158AD16B114F657FFC0BD35BFAC0ECB09B8947C7E1B60030F250BAB14200843210624C1AD40BC529107560B3ED36E862A5CEB39B6564CF61A5A566D67875F56CCFB092D323928F4F24AD2CB127B5E669272337377D53F2B3457FAD634AE9C64D269869C3A01F85ED0620EF582814E15318B1C04EC32C05398432F2B97F8560040554C619090F1AF7620A53728D86E638392CF8BAFDDB91FCB172C2BB06F7E759937CDF1E4D2A70016CBE1A6518B6AC56FDF9B166EF0380D4631333790087B57C6F6C2939DDE431FA83E7BE7EAFF50FE4FBD342BC7456C5CCA6BCF28F14F3E92B3CBA6A18AB983A40B635BDB4C027915592587CF9B2E236D984C4D21834E830419C8EA6415AA24D78DD05AEBB205A9178D53777454585B9D30BC4BAE901075654F0EE1F61A0669D19ADB28F7CC5429E4FBDD32F8B4D212FABE046B7F7BBEB2B5C7263FDBE9A4F5BF4B2B4767531AA29B3BC76C43C63D8B8FEACB9A0FC6B5FDE1427B116EEF64AB6721F8354061FDC659143134789FF531D9942F40950C17B8816CA8641974B6CC6F3F0C39F81620BE2A60A7277E366989A7958624BCD3EB735BF04C601F2826003BB06AE1299DC760B8EB13808073A9CECCC88C4074F17CDA2135D7BF76D8A93EE1279A2705355134D6FB1539AD0CA933B0C338C9660727D5D86F177AD19D5BCE24C18D29A659920591F3489CC6554A38B0A85CD2A7CE006E504081973BE3271E98DC8EF84559DC0D95E098667678165906525EF8BCF3AAB3B9D282826727E52959BE15DDAE518B0EA119C55181BA40D3C3E8BE200C81DDADB93D49D0677D265B311920B64BD942C5736A7999A064F003A2EA008D8317437A7561B6F116FE781AB266B671BDE3BDC37642374FFB520E702E60CFE098D9C1E88BDE9A099DB807ABBA75ECAFD41E6372CB986644FFCF590F6A54A91EAA33909F158B902055895CDC86371E556279C4D77F79142878D6731CFEEF93F9958EE2D9CA4E2899F6E2EBDFC1DF1B72EB57FFFD4D3FAD8C0D5AEE3D24860241681DBBA1AA0AAC627712162C062183661497A20131FB1751308CE377B09FAB0D0FF10145452F0A8E5930F96590DE19F1D29FDDD0931233696301D1B49806927177A70B1F125605F167361364657A5202407C7D431022B41A11139B4B2072C46D955FDFE3E05A6D4FE2EEF2482C447DFF4CF2265D7327D0578AEB797FA5E8DF5A7F40A8E37A9FF467D9F06BE17A3F058A93413300CC44AD3D435A83D4C146404ECF6C8832CE104180F2F3A1206514B10EC03BC6E636AFC7087277F4BB1FE75A7A11D1F52B4DC6C559286AAF944C20B43715A3BC1CC1521FC598D6F8946A69EAA287EE411A4BC6B17C57873956745FDE6B0B5E240A89E26953EB41F1FF421CE161C124F42A8BC09639AE46FE2AA50B0689D53BB04BCBACC67C19CDDAE1005FC256B2DE617BE9B3A7BCAAEFB7505E8220C622F15E05729340229FDDDE337C402965975DACC6D1297B334123DF0BCE665AF70057212AC459B35FA533A465E7E8E089FB85B7B879211D441B225F26F2E5554442C61B5DC925EA4E1626E7EF6D9756A04AD1F124AA9CBE1CB4EB4FAFA8C9858C2212FC3D892A5BECAA6255383DD22F6BD107F0787278563F03A587241295B06B5E25C19D0227A83A4A649C01DCC1C13C0F61445835E4B1E4AFFF653164A551476332FAFF0E3FCC1660255C245C063E69D6ECCA63DC139D5824D0DCCF3D095FA788F8071B2B9828D9C4206CD33DD6B48CCEA33AC44CD9406A4D259737189037CDF89AFA10920D0C11B1163C1C7EEA7DBB08F56074D658418309992837AADAF0013DEABDBA92BC812E18101C8E5D4904755C4111099274B7B6548CCD143B2422CEA59A668260D9FD19FEF2E98D0E9610D7AE635BCCA8959BA36C1E948FD68021019E2B246D21B2D9B70EBA9E2E1724D4E91CA4945C25C1FAC907873439910BD49D2000B2E3802D7B42C0808EA25FA32BA157269BAB1D4101510EA49A9CF599389C5DFED142C62C10B84FB19E2713096EA2242AD3CAA8319BF243790F90E5CAF45FA8A1ABCE454FCC2AF044DCB93445CA1E358067EAF8E13D08CC1792925701FC2B6BD26CDE27A4228C14CBABE63A6809D50C6791E2F65F31157FD37268DBE1F69C2A17213E1AD685C7625E4A2AF6BDDC3F806E744E9D27438C1087B4AD6B85EA5EF6A1A316A78A4380B52F0A39EB429C3480DEEEC640ED3ECE2EC8C85E573AF76D98366D92F15D1D6E34B43BDBF67154D9E60CE862F2839463349574C4F87834F00B00AB401C82406941E049D5517264A28DC1D78B2B026E4376891532C65E4F7E898C4E5B14E29ABD97146560C70BED49C83B2348F7BD7D595CBF22B63FD77DA64111E03EBB2CDDB109CE48A85BCA039940540A4405918F69CA0988F293A48BE0BDC12BCAA52D9ED27F29D986274863E4B1B65BCAB83A676CDB856C59FF5DDAE25570F1BFAFB54F9B1FC27F6DFD6BEF3763874F6E86E050AE80DA80F5C9BD95FF96D7F2177FC76C0124E1AB08E8E8475C7B3517945C71B9478770861BB7747DCACB685545C86A6C7B1CACDF4028D71E06F6376947886067EC1CBA82EC4289AD0422D3E007EF3D94E57C9FF5C45DD602095CC8A46579A09EAB6F44835C50E91A11703C172AE07ACF9B9CE2E7260CD7309F16D2D55D9A1E4139FDC182BED930C864C6019B8E7618DB31253F83BADA9B6DF2035C8C184DE1130BCA8D43BCACF5DA1F9DC1C279C934FCA3EA223EAD290CC7407D2FB4B3506525887B64C272BCC55464EDACC6A1DCEA21B710DD7D736A8E6F8DFFFF810159C8CC63639E64EAEB0C806796A83C9999C740FF300D7C1DC98827DF4FAE7E3C2DEB0C294660681CE951C58CFA46FC916DBE3B7E3BAFEC66F213C4CBBB4E3CB1987BAE923AAA58D5457BAE1D736E308D54E32D8B5F17B7DC7D47D49F35AE98986F38FB2BD451759B4A9B96F38B6AADC6B2F97954E6ABE994A682E3AF45A370DB6F39872034E387CE37143749AB5E5F61F61942A68908D2F659300E5B1FACAF81E44151EACBA3DFF5A0035CB8A243C4B4DE035BF424551DC12452C414D828D293CCD98C70133D64520E67FCA7C9F7491E45E683F23F7C5B71B265282FF6EE5C9DAB26B0EB61099B62F59B7180C7CF264D82BA0334C57DB97CC5EE79AFA059ED7FE7C477F044CBD9481B4970F9F920602B14C889E36D44612216F4B63A51B7AC65956AA028FB3B160B9E707927A0D0717C29CB674CBF83E988EC732B5AD777B9957010C96A9C8AF77B15496757CD4D95424E8EB20A4C1D765ED43DB2F3B62B5405C846753612BF0ADCD5B36935E1BEC69E66E1862C7D81DBE646DB2AA666CD2F7F7E481693ECFC24D4BA2F8C2803BD3AFFE418C0E5ABD34EDED961323694C40C61DB6DF31CBDAAB726C731FFCB5FB4477912BE6B722E0BEC7165D33EA422EB06BAFBEBDC4B364CBE508BEFF09845B4B2170555219B7E074DB8B829697BFB4315B8C9458012CC4DCE345AEA7A7CC3A59BCF4F39E738A3D0FF7BFEA9980581183A066BB43DE1830151ED8DC56B7E834B62512FEA933549816C072703141D3A5CF9C568FEC825F6CBB9E36E31B910D7267F12C7D3627BDDA20534DA33AC3DE5BF1E47D643A5035E2924916571D84FB5473AD340F9DD59A3D0080D02274F1A5C1F5546806F5812FAC00BA52C054BE6B0075B294E095C3A069FB62E559FC36ACE5197B1588856117188DD9D33E4F85AFF5DADC5911D6CF4D0ADCFD780C99AD75D3F89255F4B445DCCEC47BAFD2DE7D57637F8EF7720A95777F7FC9CE623FDB942DBCA1BB96F91F74D24E63A75E3EBCD49B5FA7E82DDB3285A4395F1429282B227BD9DE9FA318A71B57ACD2A48049B0CE83E2E68D62576E0C4C54C44E046721B4F2E3F10EFD40C781ED0A3F8BD78EDA7877637B4ADD2E166CE8DBE06AAA84259738BA236AC238220F3DC3BB88B446D82CCF7ED239FE5A6F265EB3372F4FF5559882E3D2BFBB99E7599D79A54C4465102FFB206FF02CB87B1B69DEF4A981BF88D3000FEDC476D35D2F6EFEE27975C905436CC8EED8D8559870B8598318C956587FF31B0549B87CE8EA094855B5293364ECE7CE0FDA3F8BFE1559A6415FBD6667B4B41BBCDAB5FD116C932E54C67AE226F9B9407F311B0C8BBFE5BB46D9D98C086E0EC0A7503D1E2D66FF1AE6CB3BE3C090702AEBC5C476855B6BE9F8D354107789E990AA3B72A724950643A79067F799EE9E9AEA5105F1FEB90C19F751B333AAA6BB4EB2D5C26F2B5A7BF1B7C32F8175633BAC16FEB722C8AC667CAA9DF19096CE06D721DD5AD59AB5999183DE9E5525A6B7F8F249F1A18DDD1C1DAB5D14B4FBFFC0462855382D5C2DF618A0450FE4CA01C9BF52FD9057042F3C69FAB70380CE39E683B35866FF660AE682A105D0BA3716C4AEEDD2FED28383AE8F44AE3842B0971E810F36FB8EB0B79241D013EAB2BE710832F9C427ECCAE5CABEE727A7085CB6A68BB89F3B3F49CE7315CFB9C84E3285B6EBA685096722D7103A76454D23CD520B98621D48655CAEC7F32B5AB3DB2030F4345B646A585F18A900F497EE01B19E607384E88F36CA024000ED587FCBE814C254570EC22DD59070E1540277ACAFA55BAB169BEFE2955B331D824FE8CFF9FD137C349DB70D49771F74DB24820D2FC240C62879C8FB8106E21E1C0B94F5054C05E60F977EF07B7414DF8770F987694F292734FE1C0FD31303A38E22815E56A6F5C4BBFB5C48E985C444799C0E94BA70309D69FBAC0B10C1D16BE068763FB89E7B9605A407A9EC9312D0F92CC1A47EF40BF46239ECC24D0B78B39538DC0C66CD614B408D88C664A2D756CDE90689756694CC64DD0D29D9BC239E6A6C5CB3E9E0C9FFB6371B13B070C16F63C68E7099D32FE1FCC5ED20BD17725BA1493736123B1D5C59906C95588192520DBA415C17D4D7A1A26EFF073D239012DC73A542C2DF29A512A6457184C250EBDD1A545D3A5EDF468BFA054BE03FF2AF01E167A97D6F6C52177BAAE71B63ABB7AADF472FD2A204341D12A23F313120CD9D50D5207A0BBB144A26FF3E82FC593E39368D00FD800FE701D1A52873B68906AA3DD374A1D4EE27D034BFD2890615C24B25E5BA47CAFB70D576F568339B5909F9884498BEA4641EF0C4DBF8F0D56E0749D8C3DCB0828EEEC6921905E3EEFCAC3DB0A7F70D5FAB0EF2964CD6B9C202EE8CE947204786BAA17F51E869A366B53DAB95BB03C3FAE081090B536B240E40BF0BBC6CA91C9831E8C46BE1FDD0B40A560160D4FA710EAAF98263C8796F7F0D514FC91DBDBCC6AE70802C126EE462E4C184C2EA316F8078197A296FBB18C204E6DAC650C5758F18FB92448AF9281512E21ED14B7465E827909E934CD5B230C51B154A84074503A2C0474019972106BCB6FF21FBDDDD2A216EDDA5D9285F604BBBDBAC0FD4CC12041ABF28AE8747054667BB9C360DFC90B7252F5029592E51F8E5D7509C677E6853352FE87E945F40BEF6FEF78A5846230C68137E4D2F5734032DBCFDC7F60C456F35E48C20F908CA936C7F708D218FA5E3AF5698ACDDAE47B150E1776EE2AD3E777C290A0E39C25300C05BFB671884E4951BE0753473C1984FF2C5E35C099AFF57D689021624C1D4148B1C7E65FE3A87343D80F9D20514F08E4F2EC01DD6EE1D3EC1E9919FC9222615C28A49744FD77EB2D3CA3A26C9B1416818CB484AE7A6AED480BE4D6CFD5B0E77B7B393568D19599769B7B485B2ADED0F568EFAC7A9DDC955E0C68D996EE87A984FFCD4B6B097F70392F218B10C8F9B1E57BC4567F35E748C6A43FE742CEC898268B1055DBC44CB9BE6EE262E5A33A9DF3AAA85E86AD359554A90FB7C33DDF1B91072E4A66ED5B8DDA4B03B5CB3CF1D55C032521EA599F54D6774337512605C223AD9ED77997A119CC92B826020C0D97E060DEC9FF02C92FDFA7E20706281F927105C1D8D4D2363BCC2424B3887EFCB97E495F71B877C7E58672AAB0D26D3DF04540A5FEC58BE140D29640C7E74FA18957B575441865D317048764B61309D20CEC0C" + "8C1F0F14834390D53E370F974037A24DCC7752B210A6387513EF685897E846E14ED2E6F0548224497CB32EC2CD4A6CD1FEF802E81768B7730F6396A88C0D886DF9DDE1AC09A146CE5F691BC71947D8AA97F565205540BF307EF0919A5084B298" + "" + "1951500245BAC3685E1C31EE20418604A0B693FC396F47EBA26D6A241493285F9D96A8A0972E031C07B4D56DF2CCBB688A1733E5882DCB8092EAB8055B8F0FCD8BECD771B1A95201FF9A8EEE6A14298E4165CC0523E9FE223022E92BDD480429A46D7E0D9E82BC200E8060C9260C05847F5B9E738193749053CA4CD71BBA1D11A2CCD3D3F857FB830836BCF0B09F2F8C4BE2E86C73F8353F65C2504A5A6F79E8503C148E17477BC45A3F688AD9BFD7CCB82A1576FD07B715304CE9B5091729BE74F6EC06646FE2E814933B3317D2E299B6B80E8FAFBD6CC710E3B4A2E913990B2C7E08C3D300B2C9F62F3AE322F274A85E3DC6F2307653A083B3B9134088AEAB5A97B5EEB565C12A11B19DBB299DC25A9CAF201231BC06492AAF381E49FC840A3FAAFA99E4F48CE06C5BCEA8D9C069349BB7ACB98B74AEF15B998994FFB80997B93672CBED2156D6B2DFF819D9B58DFA26D712D09A7DFF5006F25E5D1CE889FEA8D73B309680C6C473861E9A309A863127EDA77F3AB0B8B76AFD113555A2CF24124E1265CEA11E4A11E78F58255E3928B12F0223B88202A56AB2174CF0F95C43DCA78A99E8571FF088159CE6511278ADAEB03C7CCF5280099D21555E8D7D96C1CDC9F9CD8C3DF20CF5F0757E8A978BADA93D336955847F50BCD794CD784720196FE51EF4034F59D86B6D64EC977695EED60BEC329D236853922F0FB3216A1A5B1C65F27B2111D8DA01F2E7EF52F9325600F7F107B3A78CBCB7922B62681EA86632EC0E249F024D2A91F9C8A004BAB1D812ECD1037BED6122EE0989D7011D00398A4D9850C5C50F9FA75196DFCD0893CD6EB0C34377DBB83A5186046E803A9546E42785BDE21161D2DFA517BF3F6B539452848CA071C7AE57F15E7A6E1AF7BBF83AFD3131594C8829A870E738EC3AAB66A1652967B072C2B0917542D8B12FD6AD86714CB70EAA71F2BD70B3191A13E07EC8C04B8652E3420BFDAA30683EFE94D91390653866F4BA0215578FA737CFE07024AFF1C614A31A34398178A2D6D468871CFB0865591D6A919C9AF45D7C441651A754016EC3F137240237D0E856A2A883767F7907425F4B3FACCF3D102A728EFCEBEE43C3E995E54C40A2203E3FA0A4567DCBD8172B6BF03F5DEDB47658674AD339DB2249890648AAFDBB4F630D23DF6BC0AC13C7B8F206ACCFB40DBDDB8A350572402B62C81B1D7A01AE847AA9017828A64662C81A6B140B0390F2BE3E53592C051F268E842B2A467EE380F3E79951CF8EE23E54F40AA90CB2F18EB3B763F968A5453C65B85C07FAB395B002F16C785419732DDC8B00B720C776D3CDB890A1919ADF11ADBE680FCC1D8FD4F5308F8DFBF173FD44BD68052021C8CE8BED130F0132E1AE2AA6D9444C53FE676F85D4818F39707208300CC51343D0173BC100A388FA6233123D89D3DFE6D469110EED0BC3C9AF89E8FD59736A18F69743CC120E4471210E4B2F9D1E4A1B94BBE6D5FA9C3A4980739A22EBDD0F1838A3BA4137E1C344A887F70B44302A85AFA432BEDB6C6B24B9E6339AD83A0948A7625CE4BE6A0CA40685DC7EA673E404793F75FA440B8DEFE7C5784B21C814BE5CBD0E0CD5B84F5246133BDC18ED711E88FC7F18A7526A9BA62670C90FE9FEC6F293C6F62DEBA57F1AB6C6D07FE55334F0250B11624139567EF451C36A1C26691C8856B39387DC2AB4C8AF32DED57BEB84BFDA47B06503F9D34C9B36FA4521EAD4253A7BFCE0B7E96928F78F3F089C374B8735EEE30784EB1C47242C447802DE5C8D55EFA727787493641CD0AA169B18AAC14B2121595003AB3213C02903DFC36BE579AF3DD54E6F7A8651950F35034FC5C36941F5157508062C72AB95609478196F0CDD23C4364492000581EC91CCABEB2638E3D2B4877315B1C46D92BCFBD6C6C875708DA86A7C41311CCCE6A394F771F5E3A1486D5D0797B09EC838CCB0006BBDE6F3E0692CAB845F95E5CD341904DDA7F072C510C5C850ED1929594EED2B3427F67F4565EE030176C57371D4624FFE3096F82A83B7D8A5263DC697A6330061F4C56F99AAC21FA126846A24F09E50116AD3E622C1F36CF384CDBF49CF5EBCEC5C773CC6FC62C0CD65AC9320416319C1F718C3A67285D4AD40510C63E3F6C511A1B19647925C2D383EB4A774AFD80CD12C5698C6A1AA0A6158D888DA7FF26AE4E70743802232BA6D6477F952AA97F65B16C0F5B7FAC66EAB502DDDF4B4E0836F11A7521E3E3D8F273459B6AFC42E2835249B6FF6B7665A07120EC588114DECCC758C2B75321190309881C343600D0D9528B03E0044712A9F94E6AE6400373F6923238DC8038DC4A1D1C397D211361FE683C96DE856A8C9CD0F7135E8A217EF1CC498A0B3D24E946FC2F464E0E5352BB7F26347F10D72A2970DE8D77D99BE62AEEE801A10AF431CFE6A69FAF6BF7F0F400C598FF2865E22C2C3085D65C19C8DC1C09C2565F567E7B76EDA51AD53753E23483585153ABA59625BFF7E1425D9DC87CF7F6BBA1C7479249C187D855B4DEEBF106C6E00EF40B09BCC578EA9BA1ECC467E49CE6425973079941A380BB8BB3D1E3D734E7282E03B88A1FEBE0712CEA5625147FBAEB433E804BE741641A313E47FE64D07FEE0A7FF75E4E2DA7203B95E0D11F4F04CF2D1F13AFC248EC4E068F5D926148E624BD335C825ACEF5153D95865933EEEEB49C1B82681197F1A3DA24A6471543F0AB1E5C63AFE1244CC19222CEF14524CD982983391E8EE8CDD42C99E8D5CFB9E47447C5D048ED3E82AC3042D96ED7AA98C51EBF4507382CB111ACECB11D67D58D3F7ECBD1562EAE432D5D48EAEEF572391026506DB7CD46855B36A76018BCF970358FF392A0300FA38F543123112444FCAA445F17604F23DB8753BFCB2414E93390881527834FB303471E38BF6FE6AB773C4B2F60E2FF286F24D339A07373EF2A19E70C20FE5C305BCD167491BCA15FC934041263A62F4D4534D44EC7BAB6A79128556EC475EE99BEAE8D147B77640447142F59BBB53830B0229A22DAEDCAC05E29C9EF793BFFE6AED1B80340BE5B01E3F973E19A9309F68B1F30A7BE4364F5BA7E4AA21AF8DBECDB125FCAA2DCAD014A09A9087B6D0D728225E3CA08844B7DD62389BBC791D9412F3A520632326A9135DFD84A38491904430D0ECB147F1FF259CEB69263365A9D19F104C2D38CCB8B793F36E4AC9826BAA2B4E5DC604225E3C1F217BAB088DF18B043102965142CF2DF7B69D066774A91D628B0DCD7AE88AAE3A0F8143E77E2623C0ABD54A78DB5DB61E45B05E23F60ABFFDA34E6544D307A0C95773232A324D54656D8C90A7ACB5B9BBD0D3DADBE8F7FA2022393A3C45476C6E829DBDC5C6CDD1E0E4FAFE0C2A2D5D60849CACAFB2D4D6E3EF0517182B2C3F54577C839EA2A7DFE1E5EEF6FE0000000000001529374A" + , SigGenVector + "ML-DSA-65" + 31 + True + "AB5CABAB5C21B2C822F0B9B777C2368A89400F43F5E55F7C9263F985F3D21950787F6C89965BAC34ADCB36C46C81C45BB2A97A332E98D0FE9311A992A631F9D3B8B78F76F7E1BB4434A11362FD9D9D80C7DD6BBE6C06A5D80F635262DD8932FA1A7A5B5B2DF3592B3EB1FC7C483A4913FD98FEA25FDABC6488B6F06FD317283B14211703856180743131443863514707247734683531213288267203825188367650253033010245132773245512164055080418118558670651842813162354774387702523173555722023333318736338561136722838325612543303203125262618583380378123810785071284574673273362807333201836323335100351252046422406664246464022403363876337713142068301280017024462613008871685850447422654773856671440550247737614834867052541442350764640801302803557661822560380837144685726812260630284716130025674200750228877060667072317780080064180766854827147483667145776600118564604525704242004645600145066550307525132815561665870383704218473060657655835006403322475026884457376214661242802846787353147638277446300372376357756637004874784512753036872120436008171211632025840115482224775760013843117021585011451017044443783138306253462040241073763413520140002763605562252712755556287101007315167718673488502883773886888851604057350516157074717027315210787074625886757258833683007717740203166631557356472038447070286616653057341513426226176277575631386513371673510016324448782162250640772235047688201205574623017834676767243334308778062313104778541125612236261326266476227503446662864475738320575457568351716165601575281566112574746635002137326628503527078340641178863260660247611520743882658087061621017442722554722078018330314446115860453330145158321086187728214805555333230765568507856264835314005712501631506268325357220085347831614278188118248517083351760827377527874472153622727218732477062685630260500314826347214557282771682717140377380367525363238886707356080764521070026621863710300777041055245524147005231848572624241823462088050170583030268005663840457653542023818680383040780823236080748714750114520376272314006265656853766677727620843344012833126304433443670510022200847682711077578008241700385261578578537145855143540065185454253678525230742370061425207577423124878314522385765145820480122800164015447632441040423233602776058320154550062355066417517520613826187341803376572713184087136648210740244173030846412001103000873148355316781287028768507665120733888813770123264148332323841460123053610115763850577313176218431478238315358437425484146127588236132166765543574555712056371643767064181520207444313043233806463042308423442254811706817874661562274313577038060758242627064448431473447552260525704704724761188717331010717384238020076044771878138074082645284286572374337762466220263046614487234488816441183660488724608487607280008880481342078415731866733547824708507757707784406050033224018751004312507257574283816483017741538305710661710861226045787784510050332512313624508736526164783220226746820623370370264855282163738370715310310864006027575224138274567341783676248575650035115843414615403753587343467202080842758400841250154672384462040881335430770631831435868806246888106284601022478000835678170147030017458555567318068363537378183058005778280886837060626B884C0CD9D5CFD660BCA5015E9229EFE1917E75644FB75026A3B9AFE569A30DAE5E8A6286E624B994A4288578A58B542471A1A0583A3DEDD9CB058ABB1D8DCAB1E5AC4F11D7E0F8EBF2753CA7ABC63C29C63A278226C57004E462CEB2539B7299B2D51C4D588146214A91C3886FA73F6038502D95008D72F5DF8CBB1414E33700BC85862033B8B36DF80EFC28439397511F5E27645D0508B75491B2EE4B4101A189BCDB4783429FD0ACB888B35D4F5A6C9DE49E258D26C57763A78171566115F83685E3E7A456ABD6A25B57335ACB92C03ECC5276E3ED25D12CE50C447DA5E21B1A95EAC3D4DC9AB97354F64173398D699EF98B185C57DD71E9DA1AE359B300E2DFEC31D3B3AD390F3F994AA44F5B198A3E8007FDF627685561022F255E464520755AE436FFBCA77A22AFABBD57E9754891132B23494F5451CF6F97D4697FC58FA67C92C872ABD51ADA6342B58E5BAB451F08A440946173E154E81F6C5B62983BAFDE03BA39E50483374D7E0D8DF25B0B6CA35D78DC462DE4D0495D3C4F95C7750E5566684A3402364E1B0B5CBBC1DA8D7FCF99BBA296A46FCC7F1FA12AA663C838FAA2BC0D85245BB7B208E2463FF6D43EC7CC3E3230A5A658016E8A04AFF662BC9BE2EDD8F32C09A272E45709C91AD2475DBC0D128BE3AC06062B332C3131004B2F18414C11A3242AE1EB49C47160C7BDFF81510E6D31865FCBCC02DD6559CD6BFFEE54C44FCEBA57A77DA061636EE37138C7DD9F40900A41E814545B10FCE93FF6BAF5B8AF316D191216D9D460D4AA8517D5C4991A0A501D5E33DF909BA73E4A2E5FE3D06591B3FB45C51C88BDF128DB80623E031CFED0DA0DEFEC24F58F71FFD0300924079E0C9E40A60D50D3F271977A8EA269A6AA5BF26DBB02612F1755D65736381946B17F9529D27B11BC99FE6D010CB4849820F041928EF8929E897C290E9F3109850E276E026149437E8DEA6A3BFED7AE34C2F207CC2E480F4860D4BB0DF49B732D6B4B1BEF49D6196512789D06D2ABE40BB45EDFD3941BB8805B881591FD17178C24BEFB06F9F061F3A0DAAD19BAF4A1AB221DDD7D133B40B2840CD1A6BDFEBD6D1B681908E72AB3A817E29B58F062BF9F4F46F21EB71F97D6F055F74971F7AC4868AAA389B553972F5E09EE34A2CE319DA70C8111DF9751EE2C51BFB5B129DC1D23763FCB4961C4CBF1D67185170307D768942F0CE4051C556EE066E203B7914316A1064DA98660C6775CD5DEB9BC11DEF90A3D26DF0ECCED4A92AEDE273D2A5683653E460A808B7006C669A0F0ECA3534C27198FDF77B0C982881E5F158B2F5A0E5895FC03F508D0500ECB2259DF9A65783327D086F857D0DD815239994DDF65FDA2FC009535E5957E792C02D59EBFB0808D972F916A20CC8878C1F50A80122E2B6F1BD89BD92C28FA1518FA4C3DA8C6063D3E7968B677F2C00516142C4496C89208625856B1DF9182C50A07F64B66BB6BC75D945BA00F4F272E6DE2996A74C770892BC8084F11921B514B6166953B35CDA2F97BD49FD0F61F08F4B4ACDEBFE83C3BB5B31310977FC6A136FCB6374ED14E5101149464CE4AB5FAD47C81F23855BBA8B638E71B7861E13F828B228027C08CE4293E14592F5DA055C9D4F9383983B1FEECB4A5FA66760E9CDC9ECA0594775AC54E33E73BBD199F681B15EF27A9ECCEC9F312D1CCF6DC5CE3CC918345472160A73AEDDD5B3C83C53ECDF2F7E2A5A5BB9B0DC2E33D526C89FED05687894F3B3D7960B842C9212B3C209A221AC2F831FF34363BFEB965AE5F6E56871711C8FB6EA40523BB928079778031CD50C8465822EB1217DF12F8B604AA433F14BEEDE86A259C13B6D7ABA3DF45D018A77A1427D7683EC0CBAE6D8303D0FC7411ED3D5EAB3566CC088AC15F70412C4E3F43F500D08FFF7AAB7C477555EC7A4E566083FBADDC4638B56094B8FC9D9A5FC9F97815FD3FE79330ABAC930C93647324DB652689AAD9FAC5E8DB86DB191B23195A8FC25BEB8C7A2786C2C9F7B722828FAAE2ABD419932FDAEB0DC6B489129F5AB66166007C5DAC004D9D7CA1B8FF5E866AE738AE10513AD4DB6C9101A12D4663075363855A32E92C4A8C03167B21447F025424AF620C7F485BA5AA55DB0596FE4370E23A92BCF4101D6C6469EAB0FAEB9B1D0EED2754548F2892C3397416CDDBA9AA119BC635C1037EDB3276A202B1D51C0A2485922586FC27A02AD6F1A5CA28D98F8FC6BDAF164DC26C05247935F23BF8EF504798E30B154A665BEB837BAFEC6A19486CE2DF6DD544B0C20688A37DE9E5E0B3BF93A70A865AD0635040E358507B0E9D43907ADA991005496A1B887EEB5B8B03CB9D22159FF7AEA0A8901FEA51048A5C975DAF858D142A63FB5140A424ECDEBF2564DF4B56D9D4DE86820DF38246B20F65A880193560C484F459283F0783EFC9CF8969AB00A619718455F662EF800F12CFFCDB252E0C29F8ED425ADCB3FB73E255A0CF950868F1F4DA6ED9DA31B0398B4C98A31043E1BB88BCE90F877DD45E533A1C0AD1C62D40D1441D3034F6E7F97785C7C348E20550E6565D627012C05CA145B670997AA8FA0F68EA6FEED9CAA398D9248832309B0A5DEED6B220C0B1FC04AAC894B5FCE311DF67281FB3A645AFBBAF2114F7C8C9B241397CC5BCEE1BBE0A1B2FA9167D58A39534BA0BBBC828C40686EEDD9ACE35CEF2AEA5330E733CBCDA7E9D869615688BD284EDA1BE04EF7C5654E2E6439EDD572756219E4A5CA3846503FAA367AAB83003CD74F3BA470195B2FF538BA32E9BDA8C78AEF0881DE83ECDD95DBDE9BB95F5B4B81F4FA30B341B4EADADEEC8E33B772F58D554CA02DCAFECAA81A75330EAE5F59519E0C1524F60283C11CD2303FA233F22767F81DB1234DC5E590457E9A54ABC76970B0276183671E446A470A8E66BF3C672C32A3D1097F07B3897BBF2CCA4060BA7B80744DA7301D50F834A1179468AE326FF59625AB90F2DB491B92E8FDD82A278B415594FC09A3DB3E0F2B7BB001B400ED1CDDC81B3132AAFAE1FE662A1B3AC4ADD170E7158708A2E62DE526394DC73E210B28FB3F0B5C23FB2E316F6C385CE4219160C013354D702571082986B2D2F0CCB923F4098CEB6F332209B4DC3B2EE888239E90F88E101C543A3A3C501CA4772FE2D704098D5FB21BACD34B4F422A4FA5A7BF2EA3112D5459621EA1F028B107054DAA4846ABDFEE0D6862C7F445FCBA1B45BEDC8CFBDA29B5309F0FFE9D74172EC4CA52FEEF7FE4EAC3D5C632BCCBEAAEA53D63E372046EE2C6434597B15237B004342885112BA8DFCD75640B00C0DF3A4726532B9D7326414C5DE1B823D0A2DC14335935866B2AAAF2959DFAD4275D88452EBA5DB38466C289D316C7465EA29C63F89270FC66E5C7F487DFA9B840B483F8EE39F9A8F717B31FE3299C37F7BBA99F4BFE0FF8DBDFDC90AB0267582B3DB78B0B2F2611B1FF93A77187F02ECBC2C6AD260707F9F922B80FDDE0FB196FD919620A23822B4AE08F46B4C865A3548DD0CF90572836DF5881840D1812C5EEDB" + "653983BA2B10B9B29DBEE1BBA6B98BE849D882CA19C29E0176C469498734AEEDA6A02C930D287649ECD2FA198B131084BE91B54110D97E91CC90E2A15AA58597D76DCEB73E4A79FE1A92A497546B925965C3EF0CCC865A75576ADD862690DC906A4E841D6D7648AD9E81C76B4ED5FCA91709323636F0E65C8B3B2E86F81780E2FEF26D5015A96397FA25FDB2E2E6B085F0D59EA672DA82D89E99C7D00E42D39B34AA51FF5BEC7FB5B0A411B9C884F6520C3CC8CBB2B48C4FF1C93C945F143C0BDEC537D838BCC2B9A1515BFF2A74A1DB2E26E44B4DB2606B838ABC1DA620060BA43B2D69602479F187167E131F0FD9ADE6E2B56620E1470C2F97FDB2AE49CF1F657A0F042288BEE48D052C95BE9002803C6A6775406B86B34050F21E0906F6B3AB3DA4219016184D58F6D2A694875C21B506B9A4FD7B5C099B9F1A3AD617DA5FABC01C1BB2B787F60E64F19D7521535A8AC835E366A174C85856D9180D26B2499347C8E8E2E7EAEBEA1F467243B56074B8286188616EA773A9FF039CA2721655D63453D5960B83EC5096510D151DC75CF37B8A237C887EE61D5FF365D33A813C2D5C8F0B1253F0424CDD10DF54C4D29C64D6D724AE43B1663F6529874DCFC96176A6D2E8BA2C32CAA038672DEEBE4C47E641359D9CF7E2FF444A37451E917BB30BFF9EA00CB9DD9A74731C9143F5B077AA4818C983D5611447DEA298C6A73DBCCAAA239F36F690FF0AB872FD0C2ADEE6E2FEEC5C4D455E2E9ED4979CB9E852E67028BB21DC927696A4660FE10300E8DE7B5B80198EAEE585CB0031AC8C3565199976FDF409594163187A0A32CB2A2ED08024D11E02EC5D6D1DC47B1EF58E10AF99DA0B32D168034D798306EB1C024A471E273A9DE4FFF4A4A9CE5D0CA7BBE61E66273515714DF8BAC2B61DF791073C7BD022DADCAC77D6CEF07C80C923C503977A97F2587D1B69F804E615BBCDE5212A9881D1AB5361B5587FB331DB46FDA1D772EBDCB231D4FCA2CD2D3E62CF56812157C6C7C59C7683564A242C7ABE37264249E5114F0679F05F466E21D7B8007CAC2DEF126C40EBBC843BE32B22334211EE04F6624C02AF116A3307B059FABF39764660BCCF262B39AC3214FF18E32466F6C71B96F276C30DE9BF39987997EE983AFB0F469A38D9099BE8D8101099D1A3CB80474792F935858CFBF5A830C62184422498E987F0E73D0FAD8F27FC5008DF68695E9B1C30BDD2E597378176E9B72A10D4601878390C2A1B637850CC1B1CC47D6B92E8F13259A23D1023941A02A188FD5B431914854BFA5150CD76D3FAE3082B51841771B58653ED749EC3335E7AE9D94C4AAF2348150283049D78864717DCD356F295EC7AD5480B67612A88CEFEEFC853BDEBD6B75F1EE7709246FE92BA4F9E2692324D433C973721668366FD73154B9C0A4197B60265893FD929120C6A71EAC827FBE4498BD39455A1D91DE85EC19AA4536803A517C622767CD816D1ECD9FA836CC4BE0DBA9E050EE6AF7D24DC0518F0FAFFE665F5568CE6089EE890E17A041FFEF9EE76B194A339CDF8FBA4AD2A156469F84CF1F8F97C5A3B70A15ED8D207E1532A4FF16A81E847F0DE5E72B0C856AAFDC063E85C574A4DEB58F899DB94182B6BAC6D04D41225CCE0A849424668B334097DF2F0FE728DC02F23048F4EC8F03921A4DA64FC6027EFB1615AEA187971EFE84662CCCD3AD045116964FC01D72E3B9361BCB7579C7A6B49C02BCFB79E2A932006677C04839BDD5C310177CB71AFDC301288430D91F54FC3664F69182D7712E12F6EB1B45F4E1CA5C9494CEC3DD1992943FC2FB10BC38EB71BDB529F40A7A57EEDB59D0BD9202D41394ED020AD6A9EB8EEAABFD2F6FF96CD798BD48C8F56F39969FA57DC8A88D302CF25B0A1900843EF67E2F534410F7944912793AD22FC66EAF06129F05A069BE4DE00E26F4427F50D38F789F640E4FB92EAB1E3266CB202B194F5C985167BE6F1D936183CA263F6744EFF5F25AEFADFB9E73F6C41F7741AF73C84D33B9C430D971C37F1209EE2D257F2DA9B86A93F36673065C923130328CA56B61F74BBE94FE9860864E461101FB0637F441F3F70AE5BF88E24EB2C93A0811B5330BD9335D737FADAF7579B2FF55451F6B7B384441E8774B2A9922049C47D600013576571A61A33C8645DF8220EA2204535F01100812FAA962E394F0AFF5E38EF7AA9D1169A86E6C9B40673C91DA3ECF99D20A8FC6DDAA9E46DFFF78DE4E8CED82716C6FA9BBE66DDED7FDB2B14189526C734F3CE365F0640A36CD40064D448903188E1B3CAB32D90EFDF5C741B4D5141005C612C78006008CA90AEC84E55BD49F3F6786659968329616CCE9768D0D052352356221D3C86AAEE3B4A4DDD8EFDBE76C445EB9F8764577E7F69FFAA4E4314DD8B574C13330D38E218EDBEED02A8CA6C78E5ABE5AEDD8C6F6A55" + "32ACB4CD9FD5F046155769897EDB150CE0" + "" + "EABA6286EF3BA82B537455C41A33532C9CF71BCE65634BED72D74769D4049D6C3516FC0683DC654F0395BA0CF857C9D025C40B40DA0E32C5558BC1DAF06674B2D6779969BE02ACD6522A3222577E1997E2C7078A295C2DF27CDD942025B35FE65DE175902D62BE5141F22AB604C0B8A09F3BEFFCA336E9CF64D4747D8381BF8827C7D3CFCBBCFEF58516EF7E9F9B0B58E501B0B3C494C171B4FB9044C3D8F977CD00244C25D594721E7824406079B9502161DC25D1FDA0389BCDC2375F9974DFFB99CCAD95CE1B1099CA0B11B638AD8ABD3F1F708FB31AD29DD06C0594E4DDD14E1C934AFA772B3727A20583015DF38D09DDBDB71E53202669A68CA1D5199FC4384B76E98D8F731D8817FBBAD361D73EA4B0FA2204A2D0C573D4283CA41A2F1B7B2AB43E61C80B1D363ED35140A18E96EC857A0AA3D4FEC11823F385578CA9D90F11A59DC471F4D399B46DA5454C4216A6D319603C867DEAFA731EB81D8A45F67CD53719F7B289CF090AF5DB7069B6DBDB014B3448E7ABE999AC29AC7943E434E70D1D9BF0DDCA7DF95F407F0751A9BEDE275750F8927988347EA7550FE4753A7A85FAAC2C3A39A65FA20C01E53851806494E29EED7C0015AF6168CD5E15DFD1CB394E2DF6B46B496F454404D1DA659B09AB797BF48839F456C6F7BD594489811E341E7C03D074D8B84925A47BCF46044299D1FE7528E7E78642C55007D10B040E208AEB4E3A38C3B3CC757A17F160949D082073BE235D0817A51B68C2FD5442AD973C907A3C44750F827A6473ED0C3CB016D6D82F61D639AD015964EDDD9055CAACCA1C71B7D9AE5ED8FB3B3AD01EC6B08EEEE87D6759284E60917AD337C5F615478EF78B0A6EDB672F2BDFD95906598CC05B21D171CD1C35C198AA5CB931D7F769DE115CB6499218060CF8CF3C4695F17C3983061FC71EA223763A0D6544E149A112EC36ACDE5DB1E0F5429E72F417BF550AD743D77C6376D6FCDACD97DB708F4947FC322DB62B00F95B2BBD10C247A471409287D956916B2A79120ADB1A2C403C9FA979FDCC4D858B61B15FD589E8FC3EB7406D8BF67C7CDC8A22E110FA50E1EA1905196CC8AB9E3E434EED6647512EBBBC7F64F786E37E7A1AE27C49FF34ABC5164BF25ADAF66D5394F22483617EC2DE6DBF6A90D4C1085E92A8EB8E51BA1E03EAE41EF192154BE87D8607B997793A0D861F7E146844D989D3AD192470535EE576F0DA16BA8FE39786F7169CDD05AFEB8A0689C7F21D20D1CBDB572122DCA4AF4C6BF4F8CE11CFD632FA3778BEAC46A17F9DD1C27992CA7B8136E9C224F4376B8C4A042B2282079C60251FBF7B6C6E49E82F6DAAD47F917CC20A693F04E3F4F399F7BB1E3B489F6880AB90B379DCEC0162176962C77649C8B1DBC23B487CAE5B075EF611EE29A995A2F914234D973352095C4B41525546E239DE18DED572C64FE528A4AA6705DEE5046E6953D76FA4300B6286AC91947D3C890DF7D7FF57B33740771DCEC8DAFF878BA9DE76742AB7D07C91BEA969F6B584D15E5A322BF63077BCE4C40F1E93651D091CB92C0D2AAE1525D7FAE70B1A515B351D8229C3849582043F9C614E865A683B29B5D3C1EE426F052EF8868096408345C12C5EDC7253AC8F3F5CB9D70EC096F7ACED9868BD3ED2F521CFFD6B0D2B77FFACC261421CCDFD8B2768DC90F23DC94ED55E96EDE45015F4FA137A3661AEC4E35B2E591229DB8D19BD53619909233E6AB49E2B9A64D68314CA263CF59DF9CDE57F52366680BD45D5A0A4C9D41EEB72A774AA72983070AAB67DE710DA1A9C8804249207C62C425A40852A25402BAA69DC8500EEE992858705581A161FC2298B5BA04641D13AF1A7D279575B0A8167CC2ADA9FC41CAEEF5916564E372B0C175C530EEE60BBF0D86F24A855BB7CF9BDF4D8BC8144BA90E3D015C604751E358F46223EC2E5E26D3B63A9A178222B01669D30C4B6A0309556E6952FB61BD379E794F4DB7CC72B1449663AFA908E61915E5BA91F8926332E5807BD1B11683AC08CEE214B7AB80CB3DEEE143CFA61214C70A732C3EB68E67A2A7A6A651D75D6F0F0437783395EED4F32370A635F3BED9F399CBA16FE027B624E549ED6AB3179940DE0DA3F1D7EB9F9254264E8089EEF823854A49817185F75323B5305FC34DB8BBF8734869782AEA671D8DCC5606ADA903258E9DE147516A4B11C2F700E2A7D906895362DB97945BB64D6D81266975BE4C8CDBF0F78F316D925032D9C79FEC2B6ECE659402703209E3DAB3EDDBC4829F2BABB101DA2D53E55FF20E958B22EC4C626D139F9039A332C940E333BD4768377B448CAD8768552A32388E61AAA64D73057A917B511BEDB1CBABBA00EDDCD04079AE02F19946FB29034D523AB7A267FE9B0128F02EE7E703AE4553148A3B9996A06071C7FDEBAF15C663CA538961749171C5C1EE2CCEC3DE81433E0317EF8FC50A4FC215BDEB24AD9E7BBAB7F95D4651200C03F3633386896A7A5C6FD1E76D9CB92680791EA8E90DC4D53EAE013A4C1ECD06DFCA11A017E8EF908E940C6274765852B3CD14EB9814A4DB4CDA5C980B89B9F8134BDFE9C13E4C9275CAB41CB17AF1C4DB7502FAA4ADCBB773F89FA97E5A947A380F2DF4D246110EEF244DA451D2F2E172775FF9B67BC4215C5573D425B20638363F016C8DDB5FBBBC0874A3BFE87F6CC76C3034FC2CC408098881B06C709270939638FE2E395CCC33933AF5744E0CACDE1C8C5E2F31F0D9B9B5CA81D195A8C7437A57B77D465A49F4CBCD928B06F3FD1A013615B35514177C5D133399070FD407BD2FC30F380775A7F812B08E74E9F8C009C45711EB5F5024B6558D6E4DA6E2FB30ADD4784E556E482DFD0FE2401E50E21954655504336F211647696C87CB5950ED77BBA05FD9756A176E68D9564321BC929E018C0BB1567C90769344DE824FF54FE8663528E2B197F2CE4F80AF50146F11B4AC02B0806EC8EDBFFE92A3C913B771105D0055B6430FED4C9DC4AD27944E9D692DE1F01EAFCCB2E93CF4E6B459F6F961BED82C91B45A3D8AA0BFE11E0498D26011CF9AAC2FF306F5A7AE11AFDC419FF744BA1280B60946FC3AF38674CA30B72386D6E498ABD3704E3317840CFDDC1A63583BAB0E097CBFDADD701348646EA5F756C64D61B7A227613076E7A7D59A11B9C397520B864A2ED59570B5F5D7CFCFF50BF0703B4940FE60ABD134322F080EAD5FCAC94CAE41785C3FB8E99A6F5E7BADB261C0AFD7598C59D6B7DF2D308196EE9096274049E962553B102A2A8796A5D4CE739534867F1C9E9B1F5B0FCA613986D828642ECBD278C70391881C2E65D9E411DEC96BB11A61CE85A636981E6B005B65414912EF0168F625888F95A2425888B181B2876C0A1C5646F7D06B156E0ACF39785784F4078CECEAFE9AEFB6BED561CBAE07CEE010B1003AE4D0E5C1AC7C0D6E51C3A3B49B8E0667A3A0DD1A9594E7457E8CB884E42312EE1F87DE49D95011288AA4E6DD7FC018C87D24FE9411908430B6820351B78263FB42D5427AF700BE9FE705C9C6F5904614085291A47840E50BE77E504FE6571C0C1DA6C02DBB37DD6210B313D5A4082A03AB1812A11064196129BE8E703C51FD924DFC933FA64E5D3903E77B41024A7E5AE32A5DCFDA95242AB8B22F0AC5AEF1A235308F1A79FA546316524AE8C952731E839C1D9D6F2432DFCA06C91DEAAE862CC70B5BD04E8BDF26623C70126F7D223C8ABE59C2201BBE6111E573D3D718A920E32267338549A758C2705E9D0128DABD4E3B75A107DEA57F98ABD9876A9EFF78BF710B1641FA7B7A1AC7EF6A5258DD28DDFC4B6BF6B9F9C351F0A7400F6929BEDE58FE812BB3382B84F00087F9BDAEA9506B2E5748C2B6C81D91D320B3C238C184B48D75F79ED228B4B5071324FFD4F6727303AF60FD385C8A4D0432C7EE3223E5EB3CAC2F7ADCF9BD0A2BDFCAB32B9F12C18D3351F4D4F13155ECDCF81FE2C0FD200124542EA70988310589BBF2B6527D23F2AFC3B98742C16EB66B6FE3B3BAB26E951BAED4D026BD7D41B63B32533A67EA03054213E8C9B10BF240CE0988FA76E16625B5F19AB652F2C114612A6A3CC56B1B41F460B94F234515F9928E9FD7301F6D9DFA809389A8D8E8C455D088733B429E9AA27224487C116A789B548440C9C4D8119B114D7FE0D960FA03DA19BCB80D50F211A0EAB0DAC5305CCFE40878F85FEC8C1387617732C3A2BED6373B52B69956BB06EDFAD5702007F54F3815884B874AA8DA45303326E983ABCCD6D009ED63942010424A68AEAFF00807455C48D672305A0D6AAE491E2C4AA98E6350C5DEACD498A5AA8ECD0C6E727FD74DE88D65AC32D89416643D07529803CA0094DBF133E29262D2C6A5A9D2E2E7A8FFA7DC058E8EBB77957254470F55779BBF1F4276F4974F9BE5C12F7AEE1753B3371BA7EF5C6D0695C828BD8C0137104CA6882FEBCC5407189A27B1E42869F4C72198ABC22D2670F906C17595224992CED4141F037410052231457C1347042A4ED7BDC0E3678C0EBA062C4C33799C519CDD91D082149DEB9C679238EB20D3E224C0BEA4E022062F262DB9724DB86B6C6E35EB2453F3015AD53FF22E3C11E2C2C465CC63E52747FD5E10B8CBCBFCFDA102324577484A7A9C0D3D4DEE822233576ABD30000000000000000000000000000000000000001050B111E24" + , SigGenVector + "ML-DSA-87" + 61 + True + "99A25F4EFA0AABBE2D4FA7D9A415008E4C08F09965E4D0E6311005B5D77BBB90C3B9713A4D33EF04C82826481963E3DAEC7127AFB3627EF1B36F5EDB0C0900EEFB2C12FDE8F42C16442EAE18A59C1D4205E5F3A73C74AE510BF74C8AC96D691EBB8BADF50CCD253A975580BB60A065E7032A1B7548F570F62C75F3AC5EC5934A082969210832E232061A077081020808C964222511A0A884900000CC02680202444C3492108100A0109142906909277110272EE0268AC2322E182526509004D0C46153C20CC9220E5CA0648B948C44122501A38894B2101C3080A2804DA2022A61386E5B160CCA142684280801A74903C465E1401281826D83102189428464166C834632A38600C2846420875021A93053A470204522C8942C01C7411292058C90119C86041C041194C84524108C5B0472DA842C22355018394C149144E1000923B32CE33650E4408CA2328CC8104C0C81445A488541B821C034061BB66560846163A451C1360A0311052312918A9008244052883222228981DB924121196E14860C1CC54550C070CC0451D130101B4668D8A46190042D60B405A00030D0B66C081704000688580888C930800A232C5C40099B206AD9B4880405290A443050360642942911093284302918170D9384711C41849CC484A2A02463B011E108214316210C2386C1900D01917124A10518C84C2141481040720A116019B42418C0201B192849C6080CC1090B0449A1289021174D8A82459C046A03A81011432914A7699A3449C9486A89242C08800194203292264EDC402D81926900936522C46910198C60126C8AC26D2215866196804910924C2689E0366D089251A4103194C84D08034403156A60B885CB860814C05090826D1C4246DB4844C8420109B9112424059448519224048CC22914174AA1962894804DC0880510A909D89448C4A2281CA3885A0229228691A4346E52120502A7091C42802180510A8768C928908C04455A801194B6215C000A2435815B3044DB0412002140C3C4704CA665112820008700228149009601142909802869D3366413B069D3345054320224826C0421825B82690907401990881C43091C862104812460289151226814012484B0301C8808C3166550809020B781193322620408D2366C801865D31011121252C9960104044810A42950345214824951B60498A604D30860CA102221374DDC442C0CB6000A4348C24851DB4266939221D8A671C38884C032509920880903021C184810A189E2944114B2451217700B046442C86DD13850A1B2100A0460DB420D09A50D4B04929A084599822DC24481A136291C370640A48812488E044450588288D3A269004946A226494C866504220E13B70453120E59462C12B78800B890E1444924470A01082D1C844CA226891304901B88251339111A86054A042C13012902878C1B8045599885C2168913A465134288C1920014480903974C419224040230D3B63064905183C28CD3844D132111DA806504A6214C0228D330710938500A460614234582B86C4A08695A328C81902951262484086809414A20262993B2699C308C42260249C6719AA205923269D938306038618A1821603631DB9025E42265DAC86C11296E82107211468C12994043C28DD0A051CAA48DA09204D8260424002923C644CAB489138988E3208C5B221260C66D82284280049043922804300619025103376254A28898362111094E11496412096C42404D01317261967181244D11B5482145490BA56D0C124A11044894302024970009936D510462910006598881E3024214C1050111085C245143A80C82C66849B2611329629B3426A2164AE4300E1CC38083142DCBC48C03390109B78CD83831C1044A40462A0996651C87690AC568A2327089A40D4396309830618A4445582288611228A3C249C2B409DBA61113488A581091D23204011089214502544864C28605A44462048340DB46725C00848B408D20B025D3264A8AB44012144EE2200A42B86150228618B1216004481C4588214910140050C2248D994460DC0285521070D0B2900136528BB048E2826091226961326C0B485124878DE3348E1CC8840B825114988D09816958A621E33269913669232745D2A08512095158122C810082DA1604A0300A84C62D24200A5828291A4051198325528211C4B00044268E12414D0003289C0266024809589671D4262879E58A1AD5DA28F03197F51EA970E46DD935B0F548844DD2D11AB2E9900EAFF23F55A77AC1AC50568CC37CE42DA518EBD55CCFE9C598DECFC7F022DB948602C922EC0B47C8A894CAADE5641F8610EF5EA8FF11A184A2660B220DBEC802EBA96F0F3BD7916CDCFDBFA1C350D326D37E847E99BE53BA36C5E1B7489D8348F14DFB098D349A71A5C7BCEE4EAA725B5C919E2F969D3922421903FC16192F5C5B3435235B45E9A7EDFC1418AEC2EED9F772746684F7A326CC76C67BCD3EECE0A2B8A7A3D3184AD146B0771B73097A0EDD3B377287742E98EB3C98D226C490C30E5F35F7A3431DFA32BE938D5B307B2014DC2CC1FE25ACE88068F64E287904AE488F0EF5139B7F9BB258236415D2F280C483AAAD6C3514F898E3591F2D24148E360DF03CD11CF6D85812BB295493001D28832CCFAD5BA68E65F7A00362D2423FDBFF61D7A49C7334BDB896FBC3AFA019FD58C9D13D1F758763897F2120CDCF8573E87A690EC99F09F29AE282EE66EE9F402F363219EB3B3030A18E12DCACB71E1A766E7D68587EA02E34587EE559C495A5C9EBA24382061FB32223DD8EE519E6736624E58FE6340B2C884500420A11E420D67F277CC2953A12370523699744957BF08307448CC85B5ECD1F1AF01AB66AA272A6B859E87036362D353EB18B357D7C7F640BE9C24317736C8691EF7186A7AB3398414AFAA6D3B607D75FC20E0D88828EE6C6337C2E47740D0FC1DBFCE99E722786FD5C9D7B4EF54F86FD0809EA798D956E0663AD4CE52D606E1E6D91087AEECAC21CF158A9447675103AED30343DE6F901B1637D60D4E1F06DECBDB7FFF66CEDC29E4089D3BC9A5FAE5095788A53E114F4A92AE259179FB47C1149B536D37E8E2AF55638AA70E3F12891645559D7DC82A2DE76A95BB68393A2BADCF70B57430B9AD17A964D429A5C8A033913B7A9DA87C8582974A271851AF2E21BABB064E027FD0518E40F0E6A156A55A66BF318BD2E5DB7A29C64EF02EE0FCB00FB9215CC04234D100F071F441F56D654E64761912A2A9193D20D47E136D271B77E6410AC3D202CD27850A7B1360CF6A1C5CAABF393208F5DD0C0D70BCA90F9D67F760CC4E129A024DA880A91E0FC62B35AB31D6EAF4E1E8AF2410025B5A1266F17928D36D8808B611C8F8DCB4535199FCC41E4A5B2A6FCDCB3896616C0FA03BE4CFE30DFFC40AE721820A7033D32AF047C918DD804477670ECFE764D9DC3797E001DF50C0E11F230FD84D29740C8C11A3DB2DEEF40197E4029645884B6FFCD5BD0196DA6A8287519FA3AADC2F966772F2654508FB9CB94E378DF69492F2C892DC4267D2879826246D26F7EF35E2E2F32CCE7C724BC681A226F503E8666EB85C19DB49CAC3E62BC8CCA9284D9CE8EDB2EF78D64A2AFEF48D258BFD4AF1C955E0FADFD879079F4C91222DC4C614148F382E17575C98241816203C81FA1E7B123B3DBC7EC0AD1FFE5B70EDB61F185C3A7B640C868FE2A2BF465831E46F2DF91A580DAFAAE6D15E893C58EC7A0A5A56B13B5056AA6DDE734A1CF15DD5057B278ABDE46DEF562A26D71516E7C97DF9B43D9C7BBD089CCD86B03E2F1533C14B643B06EC8C7F7D97FEFEDE2FCCAB110952DA4354F6DCB4D2EE07CA50079B19E6BCBF124D570F3CB164738FB9F975D1FD163601D6B59E7C14ED0057616F99D25F17DE2C2A8B97EF8C39E92C032D0471C09F6AC132F47D84F51F9186BBB49D295CCE822045D81D1DFD2CAC83BE029BB239680462663A458FB63B2446216D5453CE3941A96D090C85C87218F3D3D90F4ABEDD0ABF67D57191FF51C36B2291B7F16A0FB3E63C69CDC64726D87513F072297721E29C06D1C8CF91F18F914BAFD55CB1F88F952B17678CBC371954CB21C425E8CD0F6746C7D3CA468AB646B26A2A35FBCC317A1D264D6E96D1B41A3A08DBCFC10BAF3D22D28F5F370E402F3BD5CC6D706BEE8D3DED5430F7DCDCB5FA970A4D25FEA422D9CB1779B0C5D577331437A077C17E659752C71C574B843C0FE09E549BA09D83CFA6497E5ECFB0E3F4A6166CFA58BB0088BA841841BC57308FB1BEE99110EEA8447D236622059FEF08FB2685CF9D5ABACBF66E15CBF1A4874732531AACC855358A3DDC62305104BD8EFC0904A7ECBF40B9C221E21CC562051E584AC013565FE28A3141B724F4473022C2258103FF4B5D869BF115C56B16226A178586217E3B06B456E42FA8D73E5CCD2E5D9C45F4E224DEC7203A57F0E7D34866CEEC856E1EDF421359A4A03EDD7817CEE7162EB8B6296B50B88267D9775CB32346CDA913B0F338215BCC478B315C90F4C87B37EAB0D8A94725F2BD2442F567A56A097340A08093D3515EAF1C3D3B4D886C41734928C1044B2A685488A7A1598438D2D16A9E0ABDA13E59E190C7568E32FC70E7D24DA8CC85A9E2DE53C63DE6DB66C877B54B8769D68C05309D4FD9BD5EB65543399502C43BC1572052722EF4619667A026C77F208511B9C2C7A2C01D5E3F0650DA404779D06897F9B18F0E131ABD3739D0AFC656B00561AC02BAF7DA05EE153ACB2CE1B4C269B386CC3C45444C54B797E702A87D959B1A231EA8BCBB2BB1500BE7CDA3637B1F71DB9CC56256983A7E3048D181DC5DA074DF32E5A23C9DF67D7D48AD0C35895F10599C30D039361AB8BBB47155F55476DC85B9C3D0E519C316065CD2D07459EC44AB3363F8F95D7A48DA7BD67B99E13A96D05F338F7DF89D1EBCA98D0A466138B775DCB4F64559731EFB320BD5698E05C7FA9741CB40A532FE0F4A3A3534591644409093A8B51336AF9F799304C53B41C617FD43075070A34D6FDBB547E9323C0E0A9B7C0E05B3322149859B7E840457681941274636AADEFB8127A4D4151F72BB14D549E68B7A8CDE3346424AB3AD24431B423CDCA7BB331E14228EF817516DE713A8FCF8505093411ED82500FC28C0B3BEFC7EE800D09358D82B5721CD375C537C8FE33A88730F70CC2718BAEEA7632CFA1192C1B76D2F47B3CF4ED70B964E3B5A043FAE86C5DAD02311A3D52D699BA99E2FFC4586C0B29ACA2D1615F53F62E66F339BD5B50F56D5865B17F4FE2BCB877959D2197FA62BD368F1289ADDDFE624190D1ED089E78589C35FFE21E2E655BD9E18B87E687384DA72A07E2A2E98E8011E3824F6973F2BB41333B530E11F13ADF6D061F14EB45BEBE6091BE94F4829D3B5B1E5159DEA37E7EFF7A1D65BA8988C77A8CFF8113E91B8039B07D31895D9F8F7EEA8B82D47ED57D9B2DEA83DC49BF4DA8863D9932ED5524C89438643DAF7298B89F7AAECA1C91D22BEC74EBE000E9E89013CEA06FCAD5B8C65C8E9107DD169104C0405463ECAD60C1BD4CBD7D1E5B24256E6834BEA9FD6EA38DE92C861869205FE984AA342EBD773CBDAF2C48EF7D1B142F3BF4C984ED7B278DA28A76F0F984180BA679DF0909520F05CCF1CEBE1BA49C7F5881FD9F3751E8D98F728947C85850EC297ECDE30F82A2E330D0E3D80ED2D8A210456892DB71B93B0864CDDC1E8F93E6DB4533E30F1603502A13832C698D48B7CE9E86D9F5DE1CFA0AD0207D2D246C70AF35F6C95215DFB55292546AF41DC44979D912D5C014ABDF4E15675C6C54E0EEE74CFA8C7A268CE06C4384631C485161F9F14EEC2539CFB48BDD437E9A7F7071C7638A0B5903EFA202F3A2B81BE0742EB646F827D4E11AAEC1F7D86C6FA28E03B67CBF2C564875CB1350E06C5FDC34014B897215AA2AA10D45EDB0108AE375B1AD3326692EF8DDFB6D646BCE4D8F6C3D9B0F35FC56441BA648E9E1027C86C8DFC39CCFEC3C7105D0827BB46889822AE2582E7E6CA8ABA1E614EBED27405980716D52C0FDEE04481F0A6EF690A53F92868891B2A0AC0B551E2E078B3C652840318C3DB50F72C80EBD288E9445B4193287E20588B91419C6E5BE719729D4EE34AA1EE72E00DD132C79BDB42CF66AEECB03F528F91279940F5F01BFF55BC2DA5816B5D029A1303D684F995740B534F2A17030EF2BA46ADF484E004CEEBA7EF4C7F63EA55397B7B1E59EC07A093CDBD5BDE7EC65B55593FA4B8F3964A74B326E46BB10535328431FEAFD3B8B9208E54C8AB7BE0B0F367AC95DDDD2E0F27EFF470A9F98F00C05D78CD28A5173AE67F95FD625B56248696C95A5DF46894CA012FF9651902A2A820BB8E435EAE5E731F989422B5EE485B06FA6E8BFC41A332CDF1DAA1C4DFA183ABBBA74FF1A6EDD80500B22708ABA6F212C89FF04D3FE26E06E30623D223C4A7A3D98F88E1EF3B5505E493863A02ADFD6303E0BDE99040EE2D182C87227BDA89EB2145733D786F8A285DAF40BBBB106750434B985CACEC62E27A5168DAECB3945F92D027E456B26589EB0C0AB5314A0FD6B4D9BD4C1AF458978E9B8A764052FF57E6E1F3DB8AB1182566C4A86AE3747FCEE4E97F98B9CEB9FEFDE7325EEC7781A7BE9EE1617AF4728A4EBAF98148026FAB9B8735165CE7F51AEF5EF879C699853491838988E11C62DC19CF16F9ABFC6340CC90C06992C4B7F3D00E79975317E11DF3F7D8F3D874C7693C17CC3539D9E45A07E088501AC1FE0376473AD3F1AAA86843F9B23BE92EB5996DFB34D03FE8B193F6D452AA3B2B8662746B05A4B5DAC285EB7A6010CEB91DF3DE424C6A8B9F141907427B56C1A5A20A787C69AF99747B428EF729A64943FC55935A7291C6AD2349D3EEEAA2C382C3D91BE2FE11C67A41A5CFC1469C2B59F3C756B30A8" + "ECBC9DB7AAF9BF043EF2C011DC4A74E9577533A2C213B186F64BF4CFBBEE94190C9CCA44818B5F6E4CB98B5C520DE70073357307B00ECEB6011F4309670CBEC94F121EC34AE81FCB7DE4B190CAB2A53A555CDC79B8E32B2A60152CA057DD52FE4A560FF7473D5C4FAF2CEBDC392ECC2DC09AE7BA0A09205E096500A637CEF769C0EABF96E065D53C16E683D942359061BC646F4E4612E909EDF05C1D79723632052D11CA8F88922FE92DCCF3130E4F66837A196E01FD96D2B54E1F7295BF69E2DA97E7919C14DBD5D2714CDCE49E580C63F1DCE35510CAEDBEAEDE6B5D64A13509B0926C1AB258605E6F7FCBEB558093B97DD736F473936C030243F4743A44513D3732643112893B28D157E87DCDF002A352A5A2B6DF6C3B15C37EA4ED8032B558FD99DB5823E55F1F8E1661AFA83081A73175A6EF9D4ED6128A881771A9C93E330691AE4E983418F8943F05B67FC69DDBF04EB7BF65BBE9ABE8738C3CDEF45F94CCE075460C71F5E8350BAB86FABE8617C4F665095E0A147BD7BAC08D1D4D0440C6007505DD61D1555815CA7937C166A801A98AE334269541E652A8DEC74EFEFF85E248616787C04ED99930F34BFACE809987DC1B4D89220F62DC8D9EB0ABEF880916740641FF30DDC5D76D5736F71D73F05889B6D60715F9E98A619865852647F166EA696EBFD9233EB22362820C99C374F223D6F2E0EFF37FE2C61B226F90E83AEF63CFBA9266142F7C97C5FB94DBAFA58407F56E60375436A5FCD5D408A0D6781EF75EBB3F477CE892372A3525997BB79A23C2460806314E1E0F020E4EA766A7CCEA94109349C9474DFD7491857345A70A237B6F47A363B208F105088FC9E607C9AFA86A4086DB16930AE96BB9393F231791D835AE2F567555D5966E5F5C5136692ECBC5975C39E258299F9D18B8889E5F6465E0004F3594743FC77A68343F33A7022442D89F887C71D884E3C4CBB51C4FFFA29F90F8398D70A36420D9ABBBDD0D6468CED5D79E0204F5D93ADDA638D85FA12524124D9710D0F91A76A18D724B12628AFCA315BE7424B4EC270909878A1A06AD4CF6CE4097600CE3934CC2B7F11DE243653052DF0DED1BF352E4F81CB38D6639597E8F2D56C7BA4F8322AB901707D11F26BD15F1545F2CDB23250752AF8F053EDCDABEC33C08017A22A73E3AC83F185937A7ACB9CC43141842CC365741DDF6C15D74CF2F6109C7D276E13568776EA85A7C492FA55163B3AC0A20CC04DB775B570FB6A9719994A3674DFD1C7FEF1E380ED111CF9D34057806E568C4047015D093612486B07D796B4CE764ABC9699815CE243C19AB0BBD7C966626EB05E18ACCF9C7861805EC5C4BEABDDA5AB19CDF2CD06EA58E3C0E9719792A730DC8642926411548A896698373B37928432536A60A73EE4F2E08CCE141701C81886010C51201FF9157A6521B209CAD5A1E500836FBB2D41D8D772A26D36DEEFBB18920210E98784B5B3159DE917CC6582E9F4F306BCD0BA9811E5D8B4C5BC2B085E497E6F5DCFA6E807303065B563CEA027781951E73DF497016FD0806F8BF5F37B534CD67C8D0B0356DA44B167BBF7F2763830CBAFA65B9B2FFAFC58D2FDC24BF1A821BD5634E86833B90842D914820DC8A38E21D7F79BAD00931B54F54ECC5F2D1DD5F9C9B0A5F46A18C33B536C3E3B5FE1979F5CEF69D7A1EB6DFC39427106F45BAF075FD1FF6042DB0141CF3D886E9F54A76645C0BADC2535CBB910E8DF7B5A9278EBCADD9FF757ED6017BB8F0D6DA48EE518BDDB997D3779D4FFD9574DA2E2CCC0DCAB3CED846D6541DF3DF2460A2595005E9BB2382B6F2418B8620D6C99BCAE0A01893302A4DD80CAF5F6F8A1503246FD35E0FA40FCDE5A1FBDE1CB15385B48D7EF42884122DAF158ECBF16857EE1FA837735B7D2948D5C99DA55EDA6367D8A0C8C77834AC5BE22FDC39EDE496824759B880182B267C55F629D4E0F55C8C4E7CAF87EB9F48D729509F82FA7E3FFE9E7D9BBA66B2E38728B367C00CD5AC81B1D7B72CE23400E2141F33FCB34F30BCB735AA8C770E6306A892BCEB815CF65C4D52D12A3BF633480912C9CAB98C1AE59FE91D0332F9D48302F323167D09F4DEE9EC88CD34CFA5D538DD854B9265FACCA939F5763AE049D51F06FAE1CC6C268F5D5B2FB358C31F945F58E37BCFF38AFCD4EF83A399523918A0E1915364052D1B59543E1C61CEBB2CDD8EC874F8E9B6B0595EE6FA887FE49C78918114F0FC2328CDDC6585A4E35BBAE90B237DEEB2AA22F58150882DFBF7BEDAE300BD89A538010CB726A30BAC32A9E714F57F3EBF7F784F0BB928FD30DFF47F7132D68D42CFDF8B613D6E2AF92DAFAA4EB80CEF772F0DBA8DFF5884C3624A25519C395BDDF89F21E5F66ECC39064E633EEC607A04461CAA8DB485E1F0C482CF0AA7314891EE97AF32113593DA9A6E13E14849453254CD83E5CA4974A7D2321B8A4867A81A30AC883293CED1C90174D7282CB422DF0C1871504B783E76C94201E3FF954F729740956B28524BC51DCF5E08BF3EE9D3151A1F97D93F5AC618B085DE94EF985F7D9667168738FA2CB04B8FA63F56264F2195D298F4CC09FE571DA4237FFB11AD759D5FE68D0B68E8B31464E33DB2E4BDD304DAA59C0EFDDC7DF521F021AD41142ECD582313FA54FB3A6EA7F2A986F3790EC5D0C0103F7ACEC073E1E207A3575A060E69312BF282A4C5EAE417DDBD4F5ED729947807472298FE8416969F19E61F8684B0F9E5CAA2A23ED70832D77953D60980FCEDC2DDDFB6074FFBCD95C07BCE1DBC90319459DFD3175655E6FE2AE7FA52AB571AE804DC17B64453C59DB9F993BE075DBF74A582619BE4F29EF495FB5B1499878E543AD57A7B05D616BB7AB709F5497CFA85FB94DA39AC55BD7E571C8C5B0E4055C2EFDD3A01E2BB087FBB0DF0A170F61022576954EE4A7D3274D75D360D56D2C96B597AEC5A9E52EF508324B7DEC995DC8E29E32ED7ABAE3DDB37D853D20B8BA472ECBA364D853A058541009D87596A4D2B2F4298FB45B5258FC77294A4D8305020AC104816C00CD1C2DE738DD13E044DCC7C2B025BAC508334D0F75956FCE4F9CED1DB8CA636CA57329E454CC8E9F68721C99189D1FF21FC38B77EE0173E6F072A3C76C7C525165D06FD027DA8893AB3B10F98C6040F354598A1B1BD94D054341BEDD5029DA796B98F95990744BA9DEA7513B16C" + "75A3050569757BF2A4D6E23458A194E146D60FBD1DED4A00B5DF1A0390FB8415BDA71A73B18229B8152A60C122E49860E0E50E062A56A48F17B787922D911C05980518F10175C755DD3A14C27C61B73D714795AC011C1BD6981BE878D706F22AECDD22DA220AA775920F23452D3CABFF2F5F981FBB09E2BBDF3BFBA8EA5965007F41FF2860E79D0A6650B5E3D7" + "" + "84BB2375AC1F8349002B6E1A353BD2A816A5C7823C134E38063F639310DE2046AAC3226BC934643652970A2396969F41E3DFDA53BC5925D6F6DBBD35B34F582A8C8C384AF9F23DFC05136D731BF4E782D05ED1F488C5B93111098BE233F389652CA8FF71DAD22879900F43D2EE1FB9EAAF3A06D9181178AB557C757767D7F8C9989E578B2DF961F2EA268B65A54AEB86ABF61404E0D45F6AF908DAD5611515C732A89AF830A851CB35868B9428B74951331B852BE78E9EAD81224C3B8FFB647B989C2D23D068736DE910B9CE2E0C06A3A5BABCE718C910A75C58350FE08E3A5FA14997362FA858087597CBA396F6627CCA12D36801563F20593BAC5D91CDE8DAE33872530D5ECEE00F605BC69C08F007AF0FCBE8834975C2EA7858514DBE71672229ADFF6A600C5BF75AFECF5C9DF763B78241864CA275937D6B14E35D2BBC203ED733BB34BF8D571EACD3666A52E481B5147E76BC83564CA5DB1A50B7B98119EAA217C674707B4C6CC21E024373A858803C5228A54E0357C6EB1297B09AD934E5C631A185B2E4DC4ABDEB540A32EF8EAF190242CF252CD08173A0495DFDA0353184FF50A4E209912C472E0DABB8F9ACC469B31A590EAC417220AE24E5314CB73D1C5AB4800222C9FEC31E420E5A5019108C9A4C0E88056C5F2A4D49BEB45404E807B9142B497C13CDC79BB08B655181628D41D825ABCDF441999D894D023E71FB6BCBCDE770B04E39D2F46509997A8272DF040E44E4741882E69C8D0236F111B15EDD8C2E7642BC81A87AC8EAB602A9AF92EBE98EB731DAE4F67AB76BFB825794716C621AC32FD6E26DD74FD2B3DACB85BB4D3D198A5EABFF530D9B4536C4B1E2D75E425EACF80546D6CCDA92267633D7071E6B61494B7F16D6693E821D1BCA4E044BF20D7D14490F49B46FCF7527184BBEE2C5929FEE409003C07B691031BFC94BBBFD07969801BED449DAFE4E435B0917A166FB1A2C458A6F8E21B3896DC0977D81B552ADA473A132D121FC160E5F155F0C08CCFBA0BF9AA76037F3BE5B4E4867D65D19FD61BBB3ED1A3F8AB67E429CBFD282BFEA052B8AD8F27540339CB06B5321F9E3E8DB5E804B02129C4438BC9E9801AA9326387739F988FE2FF2A962E6205945791CAFC7EB95F40BD5BB3DFF2F8BC8737AAEAD94830BE4C910B93393DA8D36BD01677729ACEB47FC1ACCF43AB55549D6AB97BAAD9BD731FAE62BB18789CE483CDB0424979A83F9A0C96A2AFEB164D954A8F69A1BF27C1D6F53D91370D37871D04080C295FD0923174BEB6250F9B397903F77533DE97BDD50DC58DB9617AF8FDB8FEDF975DB8AD8555CDCB8F72E095E8FE50BDBBE40659D7AAAA23DC112AC3B7DAC5C30B4C8BE57DC9D9A5D9AFB90E3D58850DA24D59C81F21D1EE5C8645999D8B1F480C13B602E91918FBA53C2335C9F2EFEA10CD278DE5FBAAE0D75D3C28BA43A0E7A8A7ED756E6D6045D84C272FE544DC3EF4D415070D1EBD120E519DB0E63150BD000AE2785033AAC5A52B7F4388D0154F4ED447B255086001EAEFC807AEF6D5D0AF5CEA3DD99E53A89D0905DDE9E661B937D14A991731F7F40B3AD56339CF0F1FAA923F68F1E357713D0D8A443FA35DA84F09A81390601696F83D76A6FDEFBCDA0D93435C86D4B1D57A925BC56A0AC9A0114FF5B632AEAA5F9780889EBC81B72970E0E0AA287976008CF9E8F96672209B9E1449BE7CFC32F177942BA8EB16C0D7E7CDEC6596AA103795574CEB25C04F1FF6702828B863239AB3C0F641CA494E32A25449C12F50F90BB7DA9627964E3D891B5470D0589C33C38A8CB44E87C7ADECC55DCF39F04F5E5AE0B4BF3DBA4E3792EA38D877F8955C71BFD1A4D47B60B2D305B61BC22C64688CCADDEB565642E0FA09845ADF39F80032669456AB6DC0B7FEF7F921E8F6B46775F176FD84652543A5664810A80F73CB5FA6CCC470A677385BED654BB11A45E3B69C4C231B2388890EFEA4082DCF97BDFFD784DC33C98A68CEF1ED3E887320AD21CD5577E68BCE8E6055BC5720A3147E377DA5D808B5BFB4A774DEB84EC771BE43CCDECA8CFEFE97D7BA39EDF421D4D2D1D6F55F623B3627F1661B1705D6DF720A9ECDB2A4162D5F2AD06A5EBE8F078EFA0EC0E42FAFA1C84BA1642304D940023C436B8629A730FDCC69C3A135812503A0295237822445097E8E9FF307C130B93224D786D6F3C44DEB31B50435027089782BAFCF2C4A9E6ADDAA6D5154FD68675623747219557E8113E95E8E8066F2349C6DB8AE99B4CBD08300BAD57CE20D1DC58BEFCAF6CDF38567ED72692122A607CD3404929C6B7B03CCBFA45E14F0F78EA1E34E2A82282D06344FC4DD0E640CFBD5046A280CD9960AA28C2DFAEAFA0654E06F7E8F70E6E74E3394FE3153A8200F5EBC29BDAA36A358A567096D4760218DD2CAC69586D6F2FB51C5EDE9CAA619BFF217F79FB1EAE672ACBB18B92FFE86A9D5AAAA1FE108D4C1F9A09F578BE3E5F884BBD910116D509517F34EEC8FF7D988F1EB9D336BB37C327CA1D624254B9BB18BFCFFD1084119C593EC98F8F12ABC1CA123463374FE770A0C8D212F5946D9CB01AD5F62A17CD848B2EC11762FC28CBE9F4B44B9AC92ADFE80C38CC7FD4C8EEECB9759CBD1D503CE5A5A18B94C47E3C5CF643DBA4EB4B63B7F5817DA772696B72DCA1C362B6E21585FE06FAA0DEC5C031AADCB91D73AAEC9F0EDACA3FEA9CDD3E9507014D3E207AE7E8525E41BF2A553236886BB6FD71A23A18CB3BA84007D25F980ED72E80CE3FC34A18081B0761023060CFAB59334DE767DB5BD46B31F149B27FAC368E838AE181A9BD4A50442B8F758B45DAA0690E9B51659CFC7E77C0F418EF0C95078F06869477BBFBA3EF6FFDB3C41448A7A5568978214DE3EF9D34AF8D4F6E806B17428E082F2420523A2C488814AD5FDB79AD806CC86AB8055F4CB92EC5756B2C875FCF1EFEC5808400FEA666EE705C063258C3586B09A96876FD14E78AB8176D3F6DC45384E52ABE816663A510C1EC4A2E754FD470F2CEEEFC9A9AA2D05D5D8740C46F689B7A6D27CED79321042A901699381F81F5443B7F66F5E2C19DA2AAC116F83AEC22288D24CECE84B0E4D97CE515A5E254C4904B54C5037A1F1DB2E580710CF38B91FB2C56C194C2282221DEE8D37BA8A73A5325FB9337FA229328D0792A35112B984F8A46DFDC43F42C836B73132834408456814222D431DE346712FDA8DE18B105002A6CE0ED88A6766A2B9644C822DC82627B699EC402075158FBD68A9C628377EDA2FC8FEE6CBFC33EA7EBE500E7DD884884A1EE0F052F37F7415B57E20FFBE63DDFBBF5550707023F285EDD26631EDD4CCC3B9DF41864769BE515AEFC2234A3C6A187A1FF375457EF4EF10DCBC69E48C135B4D3BC4273EC85E56E973F4C7E831FBCC6B187595BCB71D18688180D97E69B14999B202995082C327D14C328DE8096B76AFA178637E7A1611B7D7638E7747BDF99DFF69AAE2F73580780E4FAC8A0648C945353C2B42A20AC5B96767D194B1549DB4B0A843005ECF273BF4D25DDB64DC97A1E3A160643D942A85DAFB510B379397F665F8480A0D6EF5CE47AD28E0D95533F10A91FBA7265997B5402A8B652242BA70354930D88E6CC146A4F86E7C669978F5F56281023157F57D93A0336CE96F966A7F3A594D7D682B60D10BE99568F28DC47CCAEA73471BFFC2301AD5854DB5554218CC3A86AD1AFD4D031A25BCF3F7E79210313EC71531EC04024E57104A3C0C08FD4CB1E4A85E4104ECDA2007F345218960C1981C886D0DB7CAF690C9B475CCF15C93FD34B0581C27CEBEF01D8913CFF92ECBDDF215E029DF77434FAB30106FF7918B385DD8BB15B2DA9994EAB2B32C0905CB5E39793A7C630F298681DF1E6C505B9FEC598D8192AEE35D50FD3A5D6B6DE43D36309B9695B992F79FFF0BA50E5E3F26C76154D661CA89FE0541C4C4F05A10C9CAA82A0A1C9648235ABA5AFBBE21AF2E3309F0CB1124FA5EAD3FBA70D99C125B3F18DF5F34A212CE288AF72A83115CCFA1226DC41972DABB57ED97175BB0E0F1F717C37F95896D2BA5CCF469FE1A1724F9CE742FA8AABA33CF3A697C74CBA1BA14BC574AE514BEA4202D5CA2E679DEA386FC42D3AB48C5B9E59A641E8775A3DCCD26298570F46A4E49EE1E39F95351AF87FEEA90D5874D4B04851C389A8540D1543C6D4015D6D934483520253004783AC52DAF1F3D5908DAF37C728777EBB38C9475225F0CCE8D83B736BAA064C193B0BD0DC6EAD4947BD207AE82A34707740BDDD55D369DD9B22952F2956AE210B9F8C6E206B66BEAC1017418126199C7EBCD0B58196DEA0495C51146D32D230BB7E7095D8F41005C4F73C688099CD8171C78B2823603D0F6B508226644DA1481190C62562D19F1CFDA2087A78EA6BDD7E3081B2B9F0F53CDD48423AF20537EF50748E5B770EE659D28C6459CDEB095D986E43E9AA12726A00BC197497DC3293C741F39AB7AF406396E0B3892FC2ACC994454D89C1125ABE04A48329F475C7A09EA98FEE58543220A5AB803125F64A74574E78AE5E42773EB8C5B17346EF857E5940732A03E20123DEC77BB4DC2751620D52EA1F3A1FD35DE458CCBB9FEF2B2E1A655F1B1BECA0C003B8FF6C9BBE0AFD4BB36B78E0EC1116EBF3E8B63C62837A8A806C4E28019E9D600113AF6FA777F1F03F2E1687B875E3DE7570488B98FD9F40365352068759B2172FA3114011A9B101EA2D47245F115563A327C6A96A96428F2144A12169315C49668ABDDA755D973F9CF2229193C27CD4D3DD5E6B071DFE16BA301DF94A6EB2DAF3CAA8E19133C26F2A5440AB30481467E1371F3CBA8C852C01C3BD63D87E96707E45A9CE04FE2C3201B5B5FD7F0221A893108045889BD6C1557A3AE6A1C38EB4D17E4129791CD22402E3B546A2BA772E853866082B84CF9096A1915857D83A6D5B7E4F8CE3B7CB0C286D136172F7AB828BA37853F7FD709881E25F08EADAF83C69D2A984BE852FEBE104C140C57D9E06530A37CA9C7FA68F60F568ABF71F564D6CBE149E96FA71EDC70E04C1EEF47D0920B9DC01BFBB5B49A19970BC92AFC84C69CD56A9B6FCD0A359241D006A976322E8186ADD9450B20FC2BD2C2EC624981998E84D180AA8CBCB9E33295023B8E794F2E5D6FBE18E38F8DCDD99687032CEAF69E9D438E06FCAA3505674049B4E9AE866A87AB40BE7982788CAB4EE8DD16E637AA598EA01CB80A82C38FB29CFD744F3F5DFC5D4EBEAF7A3F8B9763456EC38E7B3E059CA8BC9DB593F51296D63ABD4DCA17C3F047A278719B50CD23CDD88735C64067B1582E94E05B08D77577F7B77987621FE76BAAD86FABA5604EBBC062A1260C476881E742BA6C37F6DDA5954E70F7A5B89C4D4D7E58FD94304E665AF0D0A486764E73D24F147A527FBA83D85CEFE2D41E70C139D1BCC94AF6A4E6887E5FB37C4ACC14B570685EDEA335E578D5FE72FA46B91791336759255193C9335EBF487A5B5C4FA2FC6F6DA112E0103F731D25ED9967AB42ED8573B984ADC715B1035DE162A3CEC7695A2AD6994A4982AA438C1A2E0F4DD4DAF820C09DFFAAB04AA2B7CBE51BCAC08E43F32D8C1E03C72550C8AAB28C62294C0E6580E83C2009B89BE531D5744D03F4E2D982E2E6DDBD784E03362BA0264F546FA222FF0D8E736BB9A82FA743AC69CC25E24A003074BA2CA7469AD6ADF9E36C9382BA9870F9DD698E09BA3A47207130838A9CFB7BA405896CC9EE1020B68CED84CB7DBABE018F53F1ED15342FEC459B0EDA444B7A014955FD5808B054F355735C727C58EC471A53352B526177B5EA267CD920EEB7056D21674E204A699A8FF36200BAD2B2D4EF9D5C503B7B0EDA3F210FA0294DC9BF4A6C24E64CA31B43F7BB2B7B72F3C299FE2458F600EBEC0F54CE569CFF662F5CBA589C2EACEA1F455D7B98C37C14FB4DCE7C0466794C5608F71BF8BF870F2CDB045946A88422E5CD5D9D6C2651B22D47C2B8FA76E9731AF48AB92C6E046E19ED062ED8B9548989E1E211CF8CFB636700125A7B439CC62C65DD830B45049AC93D1E687FBA0F1B2A3022E3342847D93A9B67B55493329E24964268A6C2E1F95048D07ED74BFD6A66A98196092956BA41EF2A895C76E58583DD2A4EEEA727BC80AF6FA699407FAA850C4202D1E9932B4FB6A3F8EC80371ADAF895B833EBCCB755C9C27FF57DF1B35386225C086518F6E32ADDDF1CF3794CE99105BEFE22C10460894C3B7F5305734167F46304E4C2503364BA0FEFCB61F0327B5C47F3EE5F758FC6C94D83BF6F32DEB3C2E8D924CE2699AFB77FFE0FED15D6DC6285FD78D85DA7A50ACB239DB50FBFA2296CE0F60C006E352BB8337A999CF596895F28E133BFCBD26802AA2C252E2C2396A219CAF6F876B0718D309A20454573A885D32429FDFB2052A80B241668A6757B523735FCAD73A4222303B4664707EA1CC2B425051537D85939AA0C010899EA1AAC7D8F0325078808692D4F4F7FE5EA0A3B8BBCDD1EFFD1A73C6CE7992CCE763696CC00000000000000000000000000000000009141C262F33373B" + , SigGenVector + "ML-DSA-44" + 189 + False + "12B197B30DF3DFCB40A94519CE8A60E7961EB743D3B59BD7EAEB80AB3D6EE61875CC49126B841A81168F7EC8BB06FD2B1EF7686AC6818CA3E9217620CE14F33D2CA2E31D93E1783F0F3FFEBE5E8FB7BB02FB6F505B642C0A136176C2EB01E75056A3C144042C95110903244FA3064617B56E25A2CC5681305BB68B6B78C3B410534245042682A0108909B764DA484482362C59148DD2908902250D5A3808004026A040260CA0288084111442009C968C2026245B2626D904458C40311C86700CA49004098C10228C8B468CCA32480987319BB6282181210023329900201BB748D126620C02600C164860026A42421094B2410A4026638010D180090C1828990428C220281A282AC3326D511089C1A28C62964591864C19384E9C02301B470E18A008214462E0900D82364149100E102911E332869CC80CE41460C4C28103024A9444461C2146119329994452C914458B36659CC00182A40899100C4326681931515816682441510030722186411A276D9C900C03876923B865E0026C883602DA344988A0705B4411893430D124655A4405123031C1A630E4000E02198D1244828CC8859B80455A9420D1222840A20002B980109561128231838288910051A4447002A844C8B83011C268E3822954C48D940232D0C28022B76C13048001044DA486500A3680DA346642360E5B169008146A04354C80C8411CA431898645644670CB4046C9366A4342320A874C1BC06C189369908801CC809143B66410054910040864C42404B220E3444D9C388920397203120C802861122909E3122602835053163191487223080A18924011B66D1C4144E1380409B6619844408938600C190210C02D61840D2221511BA48824496122B80DE3089162800812C29080C040934486DB446E64382003971023160D92226EC8964CC410068C1292CC286924B94109B189833030649010CA18060C4805C2284141B26054A0281200651A38700BA88C033760A4C44C20332193A04959004124B480014965C408214326295390644B0462008845A400250B8269190281D9268564080619917019398EE1B4416080845B4224D846324AA245E49808C9C6615C36612380911C1392DBB46412C31090486690204682222A20C1811C093104254ADA862489846198422841382A592602924042C4061164C2111832818A222D1A134021B87122A76DCA8608418061A384051981301123508CA600214784C0146819070A81300E64B0814404500B982811A864460669052632944F28CAD8E6C611C5C675A12ACD3AF1C8C21CF9BCC169B8D99BBB02279DA7A03AFB5CE1F4CDBFD673EB202FF38A9A47190FDC14C4265D4C194C7B098A8B9009BBDE9D7F5E5B60CA36980CED2AA531166FB243CE9F7A352F5CE63976169072DF0D78D352020B5F0C2CCCC873972CA6299EDC8311A5CCF2B7CE1F56F169E3C1EA636D4D2EA39B8B3C62E2950B95F18CB6B4569102AAB689E2E7308A9B8DB89391434E075249EA11C7B2688921D0364004652A3070FF888C23A9900ED8AC10C2BE89686E2B5D47CBE8355B89238D16D0CE724C377B30E249CAED1431586FFC4BDFE5A47B76022C31840F3617391C3A98BC2A7FBE040B5DC2D3B42DF401881652096DD434B0B742128A51333C9D7E245B9CDE1AA0BAB68A12CEA3918838397E652F2C11D999B399C1EFDF94FDFECEF6421DEB77A41214D362E9A38434E37477C666E5C1402327C0D8BF14512004AC262664A450C83ED9540C51A1D65E0A7CEA08A3771151885CFEABFE0953C5A938820D879FBD01FA4FD295AA90E80EFF986218AC0D5640CB1F38DC525A11CFCA5069B31042B4BF85FE7E66C450672B699B4E1032FB64F5B84DFF731B92174AF862063DCD8F2EE49BE4D7DAFA8481EFD231DCF09DB17B99BB52A52439C628513D2ACEAABE5532F0EFAC2A98CC4A4DA9CA9F8CE7A26D2F504CC029CA7CC8FBEE52025BEE43ADDF8778BD0F0B9D08F48A22C8777ACCD3639411AD65F3CACF1DD9C848E588E121CB0D5C797D595F07D08CEA32729A256A5F30FD2D8D3F1F222930F3D2BA8F9EA3EC325F2AA5A91EE933EB0A81B83DFAD26D0754350267AC7102202E7941149127C462FD1208E0D09BAE6AB921D179BE850B51C74ED8D09C8543E960FB8F8C1805CCDC9B83E7A97773A1BF8A794A497606BB39F1320EC841333228B8BD95272C2582068B1F6479C6D2EBB10D5875715DC80C08AFDA55792C0427034B1C4B03EAAA50FFA02C6B1B3813FC67C31CA9EC2B17F58934E3B350325E6667F163B88DD6368B153DC2935B6F45EEF4364808D6FDC04AD8E86EC2C5BAF15CE54E4F1D1649033BBD029E464416D1C35B66553C8053BFBF573BB052D1F9BE015EB6C1882FB4EA0F38AA3711C79AF4D816FB5A89B6E3E61772FEFEBCDD1389371FCDF52DDF06DEC2372EB76B5875305349754195BA3D40BEFC242049A01460633A36F680EB457E121A3086712C46516FF4BD34601C1E7936F18ACF893C2BF6115EBE6B2E5F5F3AE2811A87807BE4289AED0188884BF265CBC2AA78FE68508C45B0E9ABC2D4C466F908F4DEDB0F061350FBC9A70926678FC331DE6747E00BBDF9E5C7529B406A846E4E8D7731240C6F4AE32E76153DF6098C413796338BDDD9C89C4ED3B73D502E28644B4B3C899438326C023D0BBC70AA4C000E4F46A477CC2AB25DFEB0D946ECB2199F2625623DC213AAE84B66DE408A8A5E38A064A336A8C949EB23F0AB13A8A406A002DF990873FFF6893894DFEDCA41947D2D3A52F8626841D793D9065886F4A0251293B7AEB621EE2CF05420E0EAE263F53BBD8A995EAA023DF770330E0826BF63C291B8A769ACA8601B21FDCFA13AE0ACDC409944C8403FE04385D5487BA9E7FE611AF00CE164EBDC2599029C3B6FEB6A8FFEC418721A06072AB45442F41D86A83B24230B6958D7205A49CA5D9ADD08D2413B1CE9EF1465994CE960C12ACC3B4C27986916932C41AFE880C4DB470D83FEB6AD8FCFD8C5FB7CAC4A6F075994FE8F1913E49CDF66908D5089F7EC5AAD03B3C9A29A462975C62AB9CC9CA78EE0DD702A3408FA9F0D1BC37D2FF67E3FA93F926D542E6C8B5F75F36B62849A9B94C7574F4A5E6DB8660E0637B5BAD413B9EFB54D93DC0BE64862D18FCA111671F264C17CEDB6EBAFD3C2FB211FA5E3C918A7400898FC6CBEC172FBE0068565B574A1E409B9F788C52A89C2FD3BF7D804C8C6D1C98BC535374592207829B482A5C41B464C7EECDF398B45036472E69CE4D6C1E933AD2453F3F2F48DD75E80AF753CA68AC5E0FDE466E6EFAE4E6E9C3ABEDC3C8BB9E8D684E6512C4F10496475A3C133D35937BDAAE817FB9A4095500EF4C8B021FC447FC7A5DAB968B312AAC6FF45265F261226657DC3478080BDA20751FCE39774AD549383A3E0CE807B89DEDD425E326EF297F40212C348BE138DB230B86FDBD9A1B92E0E62FF68E6B9DDA679FCC1215E493C9DDCAD3D453E2D681D8138F2A6104BC56D5DA1DD901667E292AE8AC50BA8F230EF395C5A8B19598C42823E65677E2517E9540D85784354B8CE1BBDD2331F020DC5BF23B5E7602C8A728C57E0EDBE890359A947EA34DFF9B74203A5E1502AB8A58757A2" + "E9E8B80328C3F6724B9500731E5C9DFD261E78714CD6241BD0B05CAD6AD74AB4C89BD17E98B62E23052272827A9B9F60F50D641C1A5B2EBA6750B956E7FB94BE187A0FBEBD920B8A3B92CD7B908B6B8A0643A9FE95622FD8598E11A34169B6D29D030EE625A6A49E0CFA7F4A33884879F4FBC09DCD32C12660356461311884BBDC0003D9045F85C6E176305689900DBEF043BBC62592D3A1DC1245AB6C85153D5009DB404639BCAFC1E52FA0D39D38A6E0D1E74655D5C4B7414912C00F47AC101703F4C237ED8FD789119CE94A14EBEED355F219962D4145CB122ED61046C15DCF7AB5BE8C0CFCD3EA7D74961B8FCA6F202206D0A57B8BCDC281CE33C07A54FEA8B77915830B11DA26979421FA03C79946EF5164446915499039C99EE6226A7F9F6E99E229105E71874B3ABAC41B068B1FB4552BF9F46C419AA326E27D99256052583D008FE74E9C1A4A58B1ABAFECB61B46370B82BCD8DD6F28C978AD79BCE0FCE569B1D222D257C0D8E8D1D1DC6A641BD35B39792145BB0A0F23613E0E74031C2A8D1613088A7780B12DC878E2EC928A0887227CD90A89FB7DB95C9A5309EDA161B4BD6FF56CA2F5D81384AB4139C36BBDB447FC1664D1E7FD0B7FAA7F4F9336D80AB2A558503F508338E0AAF28607FD63DCEAE63FC6AB135916BF35BC262EB9BD773249362747A79B87C3C1772AFEBC413CD1AE0951AD0B53E8CCA0F7DBD746F958D4F043493A358AF337ECD1FEFB25B6B710946D281192687692705CD29AD37394C07350B57452D13ED61AE31581E559287292AF4BDBBACC25A6B79B0B73600D1B0253E4544E2949C0DF49E2DEB70E2D2D7BCF32A8EF76E7A4C89FDEBD12B087B9F89B23AF13D4F8A53A9EC1C35F94C48475ADBBFB1906B62602F82947CE835EA00C41727C92A2DB37CB4CAFA59CEA9CF2E6AC40705194FB6B17B652EF0CC4F1E9445DADFC87AA30B51E0D8F1472CEDC54E7360BBF6DA2FEBA3EF926C358FA445B612174793E8428F16CAE134F4E39653E7AABE9511F520BB8D6EC80AF9C9C64315DDE4ED80C57FFD1872A12785AD5A6BEA9C360BD2CA239AFFA81D03E55AF1C7E266D25F5F80E01DA6816F359020539A72AD55EB81B8A9F82D446E70E3267E77A83147E947CD194077B7B47CB240EF6E18F02A63C299F92A402A1C78D167C56AFE21767F8E7C9A9D0128858AD2FA2249CD3FE8B8817C992EB0C655E25A2A790D329776E48AAC9DF1B8ABD05BC8B9E0A61C76A34A7507F27506012C1C6E5348E1744E34C3A04EB0608A5DC083F19F3FDCE2C445A31BFD5B675B27234B1D284F793FD1FAF896D36A5219BF37C07F5D9C395C3944BDDC78B37F46CB971A90DA2FC21E3FD19DC88C162772121BDF6C9232F231C4253CEA6DA9016FB374BCFAEC74358EA462A24FCDFD71762F1B758A2332CB7891F7151FAE6670E2216BC5FD9C569D97A5F29F6DAB9428D934EB40DE9B32CB897249418A0BDB7133C05DE3F6D6A6616FFA10BCEEBC4DC1E38110FB2F9A413005529CA3FDC697A852D14BAB7D77B9FA301AEB2FCB382E2626470811A43E528F7BCB6BA0C0D67C5B19DBF51CD26408FB92C5298229BD0B61036D8A7A11A18447EE2E0F86734CBF599D04ECD6A93193A0C129C3287640BD8B41DEFD8030195FF6B29E4E7604DA1555FDAFD2D7485D4000FDCA9361A836E63CA4145098A2AD097FDC2523F932816B548B7B6F5C8D519F9277A03D49893C2BD6978107702EA0D15D0C5028D7FFB7A12B041C9DD41D7DBDAB847F49740A05C0050D13B0E3BC7F71294F7CDA4AFD2411B114374F493128FA4864B3EF9FDA10D9D69B260C0178C5625365DEFD17B6225F741F1394D3EC5B2B092529A92EC2E7083C94806D5CA830D4B90B3BA46C222F621DE2B286D25D6E670872A4424D7E7D8F1A3B2F472840323695F5E2324363ADBD6777279C51F1336BDBB8BD9EDEC6DA9B4D7F0809EF96E9457F147815B93A1FB61183091819BB0E22350EDE0185B14B1E1BAB6C6319BF40709F41576023787AA9CA435029E4E646A9D221B920177886D6749B8E23E3EA0961BCFE0E5064E053FE3EF58A1C87CFB9A9CF404D2310FD6E9D91BC407766A9AF445FD7E99E8005FA2F44CA2D86062882B512F4DD832E5949968C9E665F0F98B85C7B90351A74CC9EACD37B7C5A93FB2C081B40102F061AD484F26E5B2EA2CD9AC124B17132209E052313E3AE02EE3C04E7BD192C9A69DD1DFCC255A889EFA280717E6A8698E2D92137EA79FC241796EAAAE84AC1333F092BAB4A641B41C5EF9D1007B5A9C8D6ED328E4C5BA395885850DDA77E798CC2882718235D8615676D938E0F847C6874821951C793BB66115F08C07972BF597DAC94742E1E5AF04EFA11F7C1C4E48128DA5056D7F729570C45368E5DC51F79655552E405A542C4E8F148E0EAB7AB57DD569588EC14BED090AA1960E333E197EAFCBF6D1CB1E7BE633C7B86C7B9F115B629B24F6CFA6614B26FA5ED53B6015488109AEBDD51718F9602835239614036977D16F48879A885E88CCE8E417DC0E69EA4F807F09892CD1E4AB915705E9180FDF76B149C9FF980189B74702E8FDA4F17F3A0CAF23E3DF291B38481F9C1116CFCED1E1BCCD8DA3F6FEA462E9B348092CCA2D681BB8E7E1C1327FD76DF697A5EBA0FA4AF22A6F1FE756C59605A412A9F2A53EB642FCB7CA19C30C2CED75FABC95A906FB5D3BAF6B817065D28200A4CD5F6C4D67B22C4FF29C5885A295F18F4D284897A9717116400DF1E391EF53229E65A11D7DC223042C53786D737BBD5C64015426A49F0A6E7CE9108AB759A26B0ACB005FF25691A24FED36B3E0EE005098A83C335A5EED9FC493740E6472F16307595C73CC5504A135AF84923FCF02F1F875FCA9BCA9C4663FCDB4F7F7C751D3807683E3DFF8C59451F8DD1FC42D6C798A6CD6B64571D98A3C3C6C92EA569912CF49BFAC488D5DCE2B421B953301A2AC5F9E4D91DA201C3F22CB703DC7F1D76C8E102067D9CC18D8FDB0D762B1FC55AD793483C1C22823F0B0997861655B7F03A5CC9289AE1CE0E3600FBAFD11CDD89B2CD59847F57D392099CF83BAAF78E7B7AEE8308A978282117B793EFB48BDBF345328D94D132DECD657290B312D5C5C9F0B02BBD0022CC30623AB74E153904AA500B1160B18F3BA69B932B75B88A6BCF05A928C9362C0E56A34F240CD97A5090E8494D7EA4CB73198951D90E896F7F864790B865C9FCE39D89C63C3498B5087491B4D52810A36FCE1EE2B619C9882CDF73BCC1E1EEE245CD716E12049981FC4F7B9296A63EE3430972A7580FBA64FB9BCAC5C695048417285B141B63ACD26576F2708EB523E7A53A51C676061AA0D18CC1CEE6FEF8EEB00A57BA9DE42D94DFF8AD72A08BF1CD6AB1C4CF83B49E7EE2E8F839088CDB12E9A30C70828EA979F9D58B0474FD89D9C5DA4BB211E281BC7AEA39375E177A5AFD12703A55C797B673D6A2156109592C20565082E7D83559DD62381A351D40ABEAFC1DCE19BA20265D238A474C6F1EB2F47E44AEC365471AE089F485FDE4E9FE2DEE848FCC5DAA74F523E83EE325E3C1E8360F2F1B7E8A6F7BEA2BB42F254A45527D6D2030F6E903AD501D4DD4C2F50C4E13AB482A4C7B9B1EB586B56147C51261CCF1E83231E5937E396B87099925029F8CFBF437BC0254559F52169A574D08D726E6F9C594994A07E2F43C90C12A60CC111CAE40AC3948B6C78CCB8BED7662660C483409F113BC3B4C907104C92BD25AEE0400C9B3052F5AE0FAE950C496A85408F5BCA3855ED011E66155DB9439B1B0DD801285DA6B6BC8FBC958C135929CA26C154E7ADCAFEAF089DB351EDE1B2449BC2EE35139047947609953704F59FD1B8023F6C8836DDFC0E15B9A53B2CC524F39A82B32BE1CC652EDC1E8AC60DC5C242EF7EB206A6DF1D751A17E0F6757194935B44C9B7E2C036B445126754531626F5B01AACC1588038643996EC1881AA404472DDDD33BA725D9B3F0B86DE43570B063061A1B6B506703977842A405C75BFCBF1AAAB5F1D565752D161C8C9B9B0E863D6498E417632EE1E3CFA8911C42EFD07D43F8F2BBBE24FBEA318E8A325AD330196EE6E5A7D940B80D666416D14FA6AB905ED77EDB27E5245A30BD681B3C5847D679E0C2EE803F44C2D41AA0D27F3C979999C69AA2B9B9FA21C575D71E038C281736860DE7403566E97A28951216E9C007DB7BA7546F8E6C87C80323E718BEEB693CC7CC3F347305311AFA311641C224F95CEFDCFEF554DF02C5325A457671E3AB3ED6E44BE32CE7A86A9CB94E9121CA1258AA690C5A2A301103AF5C2C1B7FDDC448C2480BE48D93C5EC37192575E2A2215E75C554462C6D86D2C8C976E2D3EF83C22F21335318981DFEAF28C7A112E9DD264EF00AF39FE379F59B147C9B927514FEBA485FBFC8605E70C8E439965311F745650957F8DE3E0A7EE10BC37708D6E2E8CF2AC30A2F323FA6F8D9319A625AF15E9B7572755DE4198D528EE1F8174759FA8EABF1547082CAD78EB5DCF7BB5D0DC2F3DACE52D16184B07F439CFA09BE5CC5D0FCF7702DF53959D843D6E29498C0EF78FBB6741D2AE7761" + "" + "C4AD982F61A2A27834455A6F81545BDD7E13AC4C1B8B40F7C298B31BAF23FCFE" + "6AFD52ED68DA3355C947923F4902356A23F8E20D9C855C49766742AE6FDC471C8D3D37F6C47FDCDC2749652A3E5072E02DED5F71416C231C9D9F38D1569D0AF1A878EF33CCA66BC7F4890ECEA257B3510C1D7F2C3238508E099E6D34B743F23507F4641D20721747F574BEE8DE44C0F76A413749C0E83F97A94D13EC884FBED8407373D0508D9B0DC09F6CB60CF0202D19C8D65816F000D17986112CD7098DFF4AE5950A4FBBB3B76CB222426BEBC4AC2CAEA2CD833F1BE908A1E83BD7BD4720D6986D889A8E7DF6D546C138C462E7857927941BA6FF07BEBE11965FC503D8C39390613B1F0BF6736A02A52F03DC11C3227D4528C45A397BA45516FD5BBC66C0905CC517228E070412407B9A0627BF8EEA1BE53737B69371F42C5CE14F54016B80370FFB86DB3DA9C514D70B203107FBA69BD92DBD9703B3F2D5D0285789164E0CACB232E80685F217548C1A9C73C4DCA585E56B8834484CEBDB85859086071B35BDC872BE938AB2B10DF58E2663767C7B49A164838D0EB48C79E4BB90EB0D43D7F9FA040D2362161A5DCEFE23A9C3129DE9F3E8CBA330A9A50FA3C9203BA24D9E03E0D81761A09CA409D1394E22DF182B40BC0181875ECCE17F99013A4E1765D8C6D25FB9E7E8B977A50D1C1C6B72CB1A2089EEB3FC80CBE19023CBBA297A1A04DF965E726F797C7C85EC0AA5E0D20D13C0F4CCB28CDCCA61ED076B96C6CFEB8ED03A210EFDE0F98361A166964CC5D822199C3F1EEB5C83831C99CC77192844644212967D343F244895D26AB38B90C5B4392C620E6E12ECA393FFC79F76A9EAD00DC4F5ECBF64541E01513ED28D7B2C7ED5DB59DD02CEF66ADBFA04876D78DD834218242F73FCBF0D02991D41304F52010B4C8AE5251F564CE2E7035FE3D7401167869DB6A35FAF987B1B0AFA97E50481EAA15F32E236B9FB9705266B9084070AEA0BC70D3E1D3EF2A85EAAB4DB1B4DEBA0EB026B8A0A3BF5182872F1013C418D1BD70F4FDD7EA2459CEC89705AC1F0A25714E94A595BB21057AC334E4E6C6403E93D10D92D758787305723A79830E3DA45C4A284220687FFA67232DB8045A4AEF7F97936211CDB55EB2AD1ED7EF45C7BDCFCA9221A53AB1710BFAB8242B48FE6F9428F7F6092812801819BC57BAE8B0BE4A5EFE52D5754D9E5765D81F622D8B796C5E900ED87D96B60C3033FE3EA116A254E9B7604F18A364BB7CF519D3830DE845CA33C2AB08F657DBAB933E407B1C226889EE0C7D7B0E0CACB57501E107E5DC77AA88B6DD5F52BD30960F0AD3E30CD99AB3FD5F65DFC4E8D953DD4E6E34F2F6E9CC1FA8AC3486E744CCA14191E33D762125AA32D714D6BA6D40F7E488D33522543FFD7719040DDCDA1B330A492665DCBF9CB794BC60D7FAA389A6DC988CD84DA68CCB87C92D1DA9C94A2670BEB431B7FEF5A9DA4D9FD9812800B86229C212D5EB426EB821CD0FB9E79650EA51A195B5E67E68EE8E89BF78DC861BB28398A27A43C614B5E5C51B5C7F4843399EFD5222CB239887C8B8894E424EBE6A6C4460C791FD4684A2F4A1A0EDEE37AC0DBF87FBF0DA617611409671597E1BDF9BC03F8AC87D5366B91AB607506B04B52E85BCC8E8EE228455928B956B774C760EDDE33F3C0DB5B7A049652D5267D714EDB61E83D9C0E512131035838D21FD047F6C5AF91594D3ECC575C265ED96E2734F521386502F096D6415EE62B38ADAA73CB1F2C6AD48275A7EBC2DC260A8F415843D8AF54BACA40E7434F943671B491AF0DD312610423E5A48D48165EDCCAA9113BA35E507D565A5E72A85191C172B172DA82F13F5C927E093E9BC9AB886A7EEAB34275E4927A07C5DFE73920BF329722BD98F6A7B1094C52971031D4790FBE8AA7311209765DE096A114A3EA7F4B84E8EC8114AF83FF48D1C8936615CAF59228902BBD679163911203A7374B3F75C06F32609FBEFF5CBDA7C99569D768A9A4A1963223C5BEE342866DAEE1A02DE154A3B0510CDE0C66C04AC53B21B6A9C59EA3119C8F51CD6477540757D944E670519796E899BDE4BC3D5ABBA8EB7DAF318A98C62DACC7DCEAB56BFC5F61B95EF517FE95F08FB1ED5216AA4DFAB13C87F591447F4AE0D29F6DEC4BB8D51C69482E80C111745822CF110AFA5195982A48295BE60F432399B699E48BE01EF1D56E24E0A6F60766E9C89FA8B4055186F65C6DB9B4D4DB4B8C252850FF7B0199D2AC13DF3C8E89BCCEF2A788EC8C0AD4FAF558AF99D737047DCF5964E4506D9457B1AFC3E0211DDAB72A96F7CD5AD0192562931A54499FACCA77C96426B5C2EF785674779D1AD48C276A7F7113A0401319D77DC7F4B11A5AF0CEC95B5AFACA4D5448F81F4961157E3F9A0E1CF7590F979E6CDD2BDDE0A618720ED897A6DEC017038931ED10FD036E9864C15AB78E5AB45363AD88C496A1E850FA4D4CE10A80DE239DA898C1497370819D2AA37A3052F07D7A45AC2A5540CF6CD25449D6A7D16C458663852280C64E9886F3EA2689B3623D9996E62807B602A79B41D9EA4A4BDB9E13BDCB46B18547141B5A499B5B6B241DD07A97D0AD62BF6A50E552B057BC762F79648074A932507746D34F6EDC8088F06DB4F79C4E702F8E31A9F7932D1A5470B64C302566E61516ECFECBF74CE039EACEEE9B8BDB689540764F87553EDD408E2F9121F42F468968DB159BBC84217A55856ACEFF8E6D6F88280096E12F030DAD1EDFAB26D3C3BB21AC567BF97B145BD7752A619C0AB4E5D26966C32806C32B5D47E498F65DB729B4FE57A2CDCEAD3362A58801AE66CEABD3553DD510EAAEFB48FBC83A9F87CBFAE105812A14CA11F1DA5D9B9B0E40937565585A184D17A2F828A7E38E54FD5187E239CAA55BFEBE8DD3223D1E735DA7BFA0058F013FD0D22EA7DE3197AC9CAD773B540D9C7D536A9DC7650B4F0C8552A47748BBA5D5AEB750D2329BDA43DBD003DAADA8F6D90BED30C13AEEA16D2A8176EED21FAD808A79B4B8E4A8C3BCD229A66898243AB1CAF873A7D66E1B28B03157248646F0E3E90FA1A25DED784AD7D6FC896DD175E49B94A0B04BDE4E36F1E3BF24EB2C2F440C69C1EBAA4E9D2882373C9EBCD3884A0531CD4E06FD130C4A08BD2DB7C4EC4F15B0A0DF29CC6CE02699C988CC9FDF3BDDE0E166BECCA29A06B2733B1701E2079EDA98B306C4E16566BDCB56F667DDDF3DAA7123282E2DFAC95483329FF2824EE34BF4600C706D07E40C7198619CB55C533381B4FC57D4B24382DD3F844498CEC0FD573D353011E0FBB78A30252EF7B5310A5D57A66E2304884A299BCEBCB3D67A54B480474D6612D7E4091833677693A0A5A9AFBEC1C5CBCCD9E3E5F2020A1027282E313C5170B0B2B9BED0EAF602101617202125343F5053757C849DA5B0CAD0DEEEF216222D506A9CA3F0F10000000000000000000000000013243A43" + , SigGenVector + "ML-DSA-44" + 181 + False + "4F0B44AD5705B42A1FCC6D854D514AA342412A3252CB075379E40BA37555F9AB7B25F5F52143938DD768FF03E906C4180E9E04A252A1E70614046067ED83C0CD9A6267DDCA3CFE1CFFEC3B0787C7DC62614A3CD909D4168616D10B5B55B43193E9892C47E87F207956DE5BE1B426B939EA60A213DE78E0FC15671C6402C6360393140621C0801B95310A48022013210C06125B126609185184061009444111098ED91030C2008A8B006191464908B408891831C0B00C82A4254210105B44850A1185593226A34864244180519085D8C8509CB84093184E0C98654198446188490C06324BB47080042D09214899A448D208420008918C986819402CCB8844C1968CD120494B180A91468D4A040A00134D1A3101441242E3360A40187122C30984A20859284211180883805120C509040472DB9210DC8069A1300C42806024344C0CC84C2203680188084C24099A160A88A21100872C9C9851D8A26D222926A32464CB0625DB364D5422528930924C823014137100B82CCC164D61B6091841629CC08142906904C09001A72104C441233286493084C22644C24612CB860C818089140751DB16525C428020854900B1458CC01122172010048C91300E0BC72599040C5B043113C80C43A80561326A19C3449BC48C59088CC2B611DC98300C838124916C84282A898688D394115BC22518344200A00423282421B431603486A4164162948DC2C4080A03300494214C208A1A316451101051A60943304D048684149804E11880C1960099240004126A9BB06850981024454001A050138528028771224161C8104C634084D802282096849A144DE292210B21805C207012202449346C5BB82D02C6118038211411214408681014059C188ECB129160346EA0C291CB329258086D8C942C8C86304C24060C3350093852220931209140980861110404C8C66099186E181169939449133224C4A4514B84111A883043863058164111822C898051C34848CA2641023285D21069C4282AC3446CC8020DA0A28414898D83466012B32461B02D40B24CC91864C2344E04958D088451D806098414686318882185100B074859C4314CA081E1268222B924D2426522386251A20C1442800A86880247420C928D083606D180518C4024230872091784E042099802611B084251288644282C0BB849239985D11050822045E300124122861A39400B464444388060A0650A034841206E1A246C9118526104514B2870D824500AB63184B04C93148E6B6CFD9581DA4BF7B459221782451EA37D40B77CD24B96305AA08DB5E905E5C19D0113470AEA5F7AE91A5266DE20C590D70EC550BC213DC60571C40F09CB09F718F383B7A9BF4776DC8A34222685887FB13036A47EF0053C1986743082632C371BF00431D2569BDA36C0322099CAFF139D60A1966564FF4A315B1A8404714236481F3415E1E85BFFE410A6FD4F3F69294B9A430B18285E86A3F5E6D76995A2ABBCE6915C893E72F83E4B057D75AF4FCC2AB7088C2539026995C6DF9AFC20D182105602E4A6C8F0F8779B6E26119F35862D6FFA53F22D5A82C41F197F69DB992615338644106B382BE0D34FF3B1D02DB62B9019915F0BA1C76F038D094EE8EDB7733914E31C97CD2188CA3958E67B997270E87B22036B5308AF0ECBE08197D188D5EB1C43CA21B62EE5D3E84E88B3EE4048F71068688E8809AE26238F185C6CF21BBC78337EA403A60DD2D792F9632461210E4CA6BC98524CED9BFD43901E4B57B040568D9027A6BC1CC80AA4C549FEAEEE2C85803661E007FC26ED4D58833C6CF336BA326478D260A0A31E5AEAEA35A42F924FC0F8647EBC969037A93E88D56DC6D4831FD91E596C9B6A9A5F799289FA5EDE65A5F1DBE8CA65030669ACC683BEDE86799E5A26753183F5806C60D5B681FB11341BE8474C5DE12B76B5A114886E3F1BA675BC8D50E6BAAB9361D9535164319FEC594BF7D81B13F404174583198901EDE1BE03C933C652636D83C797E34A9FCB427B9CEEE2D5430AD3529C977BE2847B0570CFD97DD357C83905AD69BD6465672F9B2A84B771C49D52EF64BF4BC96FC08AD6A31849DD2CF27B0B02D467A23D7022B754391D9A0E5476873D1E30089517FD6C24B65743EC9DE9DFAE2729B5223D5BA13BDECDDF5D3A79F208E6541C7B55A6A7EEECB711D4FF91ED0B565C5C03F4446E078B4CF1DA1DC66F3ADB2C4B6BFE1C7EDFC3C547B63E2A03DF7D10CE6EFEF8E5FD76EDE61969B25B86BE4E6D11973DE0EFD00274EED9D4F85FC7798555E9460DE2A1AF5B030AB2C151110CAC4907F9BA940D2C8001D45E8212E47C7D9739DACA9BA9763A518F7340AAB19B2449E2526D2F08717B576E16690AB9AD6C68E94154529933EF5D526B784AD916E12CCB35A59F0FD32363E50B02339463C5AAD71DC3F474CCAD4A9C39EB1AC7E4E2D5845987D576842D7A19BD4BAFA22F0235D3C1E806362A71565D729333930469384B18C0EA242F2544DBE345D66C4E61A3006E13EF5274E629DA7F78AC6C05B73DD271B46DF592BA677354832FA29E53CDD99088708CBBB987B053D8592E417928DD5879A440A516AA46B98DA9FAEC09C91F15E20B9A20B8164D53481D9D7DE8C64B19473E662CF263583BE28D9B17B5949387D864D6757A5060B6D634FCA96E3A339AA89CD5E2130E90F6482F242D17A5C1375DDA4124A72B7B379033E7E79D4729E193C990A26F7539D86B59C0315A02C4C4B3F8DC1536273D9AFFEA07404EC6C06FC5414149AEB5C999443F390D5166AFC7956DF6D8FE00D50F09335554D0B3E6490865B06084DBC2EED2DF33763D9B8C28669B54E551D6197B6142947A36F08EF27A86664084D422373D36583C48C3C6EBFE5688AD84E68DBE8297906DD0698D394CA88065BA03C1BB812111146E25A91787D22AF88B409A5566BEB570F3616D089408188EEB6691B76BAA6FF69FFA81FCD6779DA047BAC0A83A847537196E9D35779EBF3942AE77B55C1D8238D7F791D7771324882EC059E00C214CFD13F04AC8086F9B4F09399BA2AF5E57723979D77DE247B594FA3E72C68FFEFAB4BB7999FB250F90769FE5F480DDA8A0F091E74BB5A75781CF858BD593EAF3C7F11B5742A03EF21A66811E3CE7148DA4CDDBFD085436881E74BE776C0549EDF0B516D36F9968B3B622089CB98A98991E7812E226B4443A6F34808AE9FA4EC1730E0D1DF49C609D6C060B6B7259FF451260F597EADCE3B7C0915CF8B7EC95E11B261F6849F7AB88976134D88A8BD0F4458897DA503B042DBA50CC5BBDB5EC55D31F7AA351BC3BBB5F09BAB2D81EE6BC6266B8BDDE4C4D32501ACCB1B46CA9B22A2F7D1EE37A50AD8D2CCCA9AB8359AB560B7D050505721F402756C594E7C9CA4791C71F75C046AF4B8F0B3640B2D24049D860DAB0C76463648AC302E9E85B16AEE6ECBA4BCE48896D3F8F544F08946032082402F658693F4EB3C2C481F24C1FBDA21A98CEB3FB040EE38622642FD7F7C4DF63482864484C959D7C787856DC946E12946C0C0A180F10827B248A9EEFA17F9CF6F4E2339E32EFF5194A5A89F792860E3C7BB1563C38F6F9FC28C26BCABCB4151410CADCA6651BBD170C885D5860BEE74C" + "6EC7BBEA6D449E00773C9CD7589887C23DE59D95BB43CA246FC29052A85E6D602120C7EBFE953A276900C492719AE827BF04A706179D3C36B2B97FA76B08003BCF0E7240BF47C6E5ECB532F87A7CDEEF6AFC5AA9993D6E93618732C47A26DD9CF83E0F508952231AAD414486319DBDE57EFACE49A4F87ADF5EFC93E2617426303B649538615CE93845FFA1141CCD90F46A9439A05719008D7671AC738B68194F556153C1912879B79A2562A586F1500B6F1E4B7BC47C918F262760DB14845EA7A428056DD8FF12B50FFD1CE9A826B5D74DC1F7E90FBD785BCE2CC1A4DD89B5B70E0ACB3BC89ECC67075A383CD760BBE7CC3E65CA3B6DB26F63CD0BA4671A10483AF349640E8115B28B3356C12DDB28DAD028BFEED2EFAB586FD9BDDD51589B5D1C6618FF3A8C108B2C37BAE6C5214920AB5CFC02782CAE52F384F11E0D41EA451A1BE377FCD436CD2B381A80429EDC60F8CA1E3B9EB57900DD14D4715A3BC447B8FC85A28242EBCA989BB8F22C8C67C7F54B5AD188A342A0BE65B9E89ECF68E32A470DA966D2AC529CA7DFE055386597FD7240C31EEE8EC6A5EC742E958871777B0D2A9223643A350E78C01769E88C33944F700F47E4F155DA310DA008BBCB5DBE49C8E26BFAE66D8187A70F4469F7B38D7DE960E530C144F0968D0AD1BBBDB6880481619B1AB07EAF553F0828E1EB683BDF5CC57154613FAA63FD1B7996C2C18FCFF12FD13392EA425BB5466F6BA7EDB206A0947D4D867A3452942D04F3D9649CAF73D0CF9D6EDC73DA13F7CE96923E91010F52FB5F71039FC1CD400F06C4565283DD02A939EFC1589CF4EF50253C0293A434C9492BE879B21D09BF5FC77868C9641D0B9826E2C5CB587EE3D92420A9812D07886E8EBA6F8F75288CFEBB19D77FCD1371CFBD2F9B2CA258039B77EFDBA9087A799EF835DCD5ECC8C7C10F68235B1A5BC8FFC9B1FCA6A681A36B8F5C583DE99B67DE75F1DB95123B81B9D541AE46C393F8CF2CC5446F59EE90D4CBEF106D829D03BDAEE6EE122BF5C40E21559437566F5BA0CADADAEC5915C56303078BD749A20F6A7904B51E33C2AA07B29E841F5294DC56C65A1092077CC2630522D4D6FC76ACA674E7CB32C521FADA6DE2691B79CB86D8DE874DF648845145367B402D9C17EA1BB9FEEAD86DBADE1450581759C1F9DF455071EEE79266A45FBF38ECA4354A33843C2A145810D649CC6EDF5CECE3107E231F7578CF9C2646CE56FBAEB7874DF4AC45FA6A3F1CFEBBC8418A0DD834FA5A6DEF73F12AF7787FA66A3156CC9E8B577CB891E73E7E6D9710A070481BCCB0FFADC74D42DEC9AC3E2D39E9ADB30E5B3889A5DA8934A822BB3252BAD635A5B88F17E56528D714FD5641C58F603AA0E9CA76A5EED35CC8E412F02557DB20EE04BC9A234D9DD9D2DE7779365119BA12E0892DDAC5396252B365823EFAB68C768D93DE49731B2916A53CDF50260F2E6650EADCE352AD82E2060D3E77B47E0EBEC8A0467F7F0AFDBC6D7F88471C6960F1DD36BD09755417D3135E5470D897962D83FCB986010B9D0EF45138698095415B7D160E4BAC92BA8227478A1D19F10EA6E909DC5EC5DE08EC81699C2FD63F1B9E7B6D61E63D8036605A7E650AF71E7790800FA99C33BDC9DFFD4CCBE2F315008C51823B90818FB592A5101B8EBD06384D4C8F45D7CB6C20B7FEE375F313DB70760D22102F6CC41FC6E774085375C8C2723FDFBE638B0EAF61B1A855A1559C07AD6391578078772C0BF3D1E67ACED15F5484CA901B89D40864A288A798D01297CDEFD725D51C463DCFFA4838947351EA584025B3EC6AF684896673C6B64727532B6559CCB92AA07A37B6FA120E5E6ED8693EEEC0233EFC26BB905076967EF4CA0D5696024BBB29470149F2E891B215DADD0A45AA1134A9C268A587E272A2C32A9F8CC523384CE3AFE900F5CD0164CC4373C2185AA89356D27ACF4F2F6A1247F616F34D5DF6889CF565EE3E67062492B7D4F1F5B999D91DECE9BBACD541489D2767FCF1931BA035697CB133C6803F8E26A49078073AAFF3078668FD2CF119A798B5BEA7954A36474CF930698CE0A455E7E89EEFE9E152E60F913BCA63ED2F99E63E146AD62776C4937A595A20BF844350CCEB5B8BFFE5AC1F95CE0B0FCAC2A65450281024F521D24329EFCF8E727BDE9C4B94259383E90E4F385FF7F9B9EE8249E7CD9B9D218EE5F17EDE740E35E57F8DC8C5FE4D0B8D21EBC632B2DA1350229EFD0473F208A66149E23120A4D5D581EB3A0B4ADC8C804E5ABFA34483D8C7239E834AE4B6D0BA9813A18ACE5AE7E08F0AB58BA237EFA6E8E2BA43ACA0BA03D4B08B088564DF3676DD423A827463099D6C44FE86AC78AFA708AEC4ED4697DB4D24FF3163854CD3093C371BE697AF96A6A789437B0827D3E2EF8ED237E972AEF829D1310B09B127B0BF06AE805B9E35DAE5AD5FF48F041F550A3A14ADFC930FEAB77685CC897C53077ED380277DDE1D043BCECAF85D666FAE9EBB885CB9A6833F2E2C5E78A270A9B0A6BD5AFFB2BE119297CD5380D33E1716EF283D332DF9C5D9D332E68F71B4D18E4F9D37757C5ED6DBEFB5567E7F567214BBD02D74FFF94EDD32D75E899A0048ED4CD3D74552D3E61C9C3C2D7BD6F8B031BB8090B9A21E6227E38948637B73181C302C7730882E954C89CF93612031667B3ADAFCB31D791AC3045F74E81F70EC309E4DAD3D74EB0BB833AD32BCE26686577167D368A3B76ADE69EB1C93E7C9FABC8C75C8A789D8E30D42A06A875784AF20F35BB1192C78A566F5361657F94E589E35894091867AB22381FFE1481E5F14804215E13F672F45CC074F213CFB640D23E888E76B04F235D59A18497245ACB5167211BC067141261CF54483BC4F4D3BCA7226F5D3768791FF1C53818D4C3B118A0B718A372F7F4ADE5C5DF295F152291CA695AD5637C188FC49EB86BB1F8279E7DA750E796AE1601F7C2C95FF84DAEAA20FA010C623EB001AEAF4188F9FC60AF15C7670DCF438BCB75CD04137DB73F0617B25822A74D65E4B5103B12698BCF0BE637172743C66C892CB859C8136511E9C1BB14C8F856E62CFEBAD42BDCA883CF9527888FA100B13C7B40799D7A54AF894C17CAFD5B54237EC6B8380502D7D99645D42CC2FE5DF18CA8701D1BC5BCE526CFE703A64EC993B1EA86ED929D69BC986C3F4A5F8B7F704FBE09B6FF161DF5B44AC57EBF7045D59033A8326EADAF966F6F3339BB712FEC3EA683FC1469C76A2CF59B4223C92BCBA5FACB8F77AE16D3AA1E20C6703AFB5F8CE937DB4BEBC05BB31FE0858159B48035A374A6D0BCD57681CFD7D4C58A7A0D7B1C9BD849516F1A919C3C4D24989381F2E640FA48667C28B1813CBB0661F7FF6D051E14EBA776769DCD1D4CD596F7BE7ADE46BD79E7129931778FCFD80CD7BEE911E41C470E5049A3CAD16B0FADA0414104B7DE10D101412165133BF11A417F281CFA75D848C322BC579E8B3833FE54B370DA197BFFCBD397CC550A311CA13256099EFE0253B21B2E33DDECB4A91D767F5458924CA788C71AFCD99A53E1669A44432445BA7DA91C0326C104B6E6D06156B8FB97A6E5858A5F597BAB6FC541EF3894C3950A643F6AEBFF77D221D81086A8495FA5D0AF2F17D83EB9F9D8950B8AF8E2DCD76D78BAB65EDC46D5FD6DB9094344307B403AD2156C310DC52A9E3732C5753FEAD153DAD8C49FE0B2C49B48E5F3D4881FC7D0F038DDBE2B48517F563F8F0B614643BCDDCB64085C3073AFD397895F3470DDB63E36BB83A7BF2F4ED110D054529832C245385555D5F607ABCC757D5A8732902693D193A41898F523114A0E53EC302540E20091C4A76197C1DAD42C9CE24088DF74BC9477D924D2D70CFCB77DC7E28FE9025F9B001204E338596CC8EB7FF22FD8B7C6466C2927908D00A3D98A99F13EA3BF9C9F021930A5BB18CDAF79A1E38AD677516F06863F85A151A430B7B00BF608D67ACEFAD8973A16DCCCABBED8F221416B89868FFA151E8E9767A953477287C47C886BD334E6600DE1592BA4CE1D18FB07883A37C06281424E03E138951B6FF86169CBE1ED38D9A5C146F27B576C8288A04E7EB77AE92E486A490C0E9C2144E3D483015095C939763FA520FEAAFB9B5A027D69EA0F347B72CA61AB26AF195AF235AD66F2B694F61A287DFBE66BF8BA157FBB17E7138A814B75B7F59" + "A4DCE602B7B73A674D3332C4AA54FB922C2855DC6613DBBC2222E050AB032FB6F6A4E3BA5D63C19CFA4C9D045105E74EBB613C14A0156B88974366795435B3EA3783C992A3B7F4693CF5F1D19C73FF899B34865FED673ABF37867251CDFD92EC1D3977518E838341C02A7BDAD4A20F09953476E622D123AED4581526F9B594B8CFC673AB7D95BD121BA83011160E174CAD6B0661B661B391E4F4E64C959FEAF13C92E9ECEEBD759C952D48D56307A3BBA4AAC6E3A6" + "D61F6A291543B6FC8431F98FBFCA3798471EE5892894E2C56E6BB639E4E91FC6" + "118731919121CEE068E6AB60F58D6BB18AC9AAF829DE59678A5437B8A6FC73AAB197FE1FB58CD531646F33D98E499A3BACD214A8FA9BF1EBA5E2A4A464C6443F71E2C3818381E5E07542C42AC52349597C99BB2C8B05CED48E8B4DA83D13AFD074701FFEC81870352C7BBA63789EBC134A6728317F7E1494E39BCD042D07DFD22656C94154417B6B2A801FE935F4287765C76C421798EFB4D4B8FB29BA0FEB09A7ED5830A782EB2477F93DF646680FAC735FB89F6B6222C46976892EE6D58EFDE51C07D56FF7570F58674FFCA68ABF8BD64DD78590F9862034C166B51EC8FE62AF8B34B2349E0787CBF71DC5696D958F5CAD9A19B0C1F18A58F882B2E3709DF999EF0E1894A417525A93A8FF3C5527414F9E58CDD9A0FEB63B6CC64F68AC13813B2237B7D5DB24C783EFD846A22A2682AC4396209C00425B41357E8F3ADB2FA4D463BED66252D18E67491E0623035B73EE2B11A475F6B00DBCAE0EA7BB22AE9B38C78915F4ABA57AF11B303292862667C22B7FAE7DE3F417F51609049AE3E90E6F7E07DB4E6994BAD80135BE281F1EF369DC0DD065AF4AFB7799EF16D739E8CC721A3C42C671EB805148E4958B6A3C38E6CDB6876651339A5DF9A6C461B5D6DAA4DCF58E9D6CAC0E07190A03D217D38FBA7F2C54FA8E3F04CDB7421D7461A7995F1BD1A6426C553ACAB6778075EF93B94B8038E9FACD85BE0E7188AAD76C9E8D2194CA18C37127CE3C4938FA9D7DC4BE2E4BDC39A0DB7C25086C77F2920F20F6723BB78255B68BE360CC05B9DC473E885534EE988C0DA48DBE075CD38A6B077A23E6507F6CAC5C4017A4C6F56FCA61AFFC9122512A32E50FE393D6E20A39EA1EC1ED1CFFAD7F8BB357B623D4B176ACCE9AA8C2F9FFB83AC62C551B6737F62977B30FF0707CDEACEC614733E6FA0FF2F686ABAB6E4BF356302B41EDBDA77E8F2083B8B6748F7800A7706BFF515C466F6B4779C64EB51ABF6506713A7A268D7A06685306EF7EE9147089A382DE8ACCF6FCB797309F364E8011902FA700B398F14361A1A9331709EF81DE3D66C01C3DA273368C2B523D7647A9321C3CFE831953FE6DEE13612367EB68D3A76546B3C22A9F77B58B1D274E9E88E433115ADF2E9520D56496CA9085960415670574E9A7940A5BD6BA9F87F376C9289FA7584BA2D1E501FACC5F31986315AD7ED8AF5E54BDD0B88EBCCAA5DCCD2ACDA93A81C34DD7B5BD10A4125B76431777C168DD34068918D7332D72F8A77892233043C36CDE46D22968FA1CAC4F44BACF0735BE0CD0F6126755D82CC89054F9CD68CF24BBD4E6C384756988877ED834CAB9082518DAD0F097B01EC8AB498D9C107986ED7865153E96B480986D3803729FC9302685D40D68026E1932908ACF5BC39CD88959DFD29EB1CD81B49B513D48E5B0628BCE04AE4767182A88BA8CEF07CF06D62BDF8E109F6C3D492D06AC60709E29022480BB7974DE4A0529B93FC0D73ED71128A2C1FF4F6E1B3C25D4DB7D8B2BE7FCA2BEC4711ACD865852B462D703E64B41C2700FD2CF477C4AD4B07A1434E366C1C559F85EB6A6365E130FF5E00E0D2D859B4CFD56D8AF44730A80690EE62825A389442AFE24A118269B05D6D47CB7F6AFD14DCD1EE9CF565E487429A934C55553329731D95891B770BBF2C6369D336B5FD55B2D7982FCF42C426B2344F61CE2C0CAF81122258ECA1DBCE1DDD227393241824D0643F39E7B1D5FE88721F1AE404564AAE283B9D38FFEDFD24989F0F8CC02CD237DBED3C45A6F89236A70568D3CE7E44464F1E4CBA6D8C878C8995590917E2AA981AD1F5058CC097CA3C2A9AFF44A0E6A77E97D5C61812DCD6107DAF6710A80800B727EF70F38956B6C85D320532FF8DA597843327E18F289489D7F0B1D9248E2ED2B93356E10D6F1356E01B26D904F4C49B2D7333DFC40DA8CFDA2063253B47C0D665A2B62F5DAF647F82E66CC5439E564DDFA80EDAA786421F735B12B2665FF6680DA2415478CF2B50A80C75BAD48EEB726634BB294CFDA96F74274B2AE62764631AD27824DF474194A7A925BF01CC77E6D81A84B9513B243755E1C3D09F6CFC255CD3DE03C24F7F7B7A50035C71BB1C39ECEA722C4F5A435659715B56659C3FA438FD012793AEE18F46930CB12262698B9380495C17D6FE8330B44E85F155DEBDF8C33AFBF8F8E2071FD2C80698C1622599453A9D1BCF3DB65CA5B13FB18CA753E6A269C26C89642E3EF2E93D7C0CE1FB24142FD814A963AE921709CBE2F8DD0F6B53EF6A48124C128C78979DC734B7C57643C407F4A3584B019B109B3653A0797482DE384A390B362E2147086DCFFC6CBDE28A21EAA4E36E36A618CBDE52D83C58B19BD3CCDFCBBCA35F04AAFEDFC2B337E4E07A1D88C1F7CC2C46075FF9924697F8529546570E0454683B565FDA985F284D490A619E1AAF4ABB58109CBF32323EBA831CDEE8A8B3AF53B36C2EC3022C42EF1C7D652AAF48CFF3CC28E9CE5B2F53705B21008F02219A0302C9CE784AAA28B662BA69DE62FB82A7A68EB99773427A1A41D40CF265C1D8DBD5CB1953FEAED7666655AD8ACCA80C2FCD5B7B508C9C946B88AF20499D94A545E9BC5832283ABA90A05A8AD4993216450C193E977945B76481A6FD57608CDADF84266290847C79FB6D50787D4BCC89AC9D69E4E42C328DABA581CE63FEBE8E5AFEEC66BD60299FB217BBDC97F7990F57D96E28F77B9A68443A82730CAA9FC719AE18051C4ED7C4419BF408BC12EB06CE17D470C6C423E780C0D4B39F788B6C6191D55975243611593706B8F9674006BD910FCCD5AE7402D283D55C06135E181FB7413FD44F7F425625753BFE8C4D6C7616BD01770BE0E40CD958FC0925C1DB16946DBC765653040C70421D62119ED1D4CA24E6CA7591D21E7AE68A9733AD4DAE6A3CBC17B2690AAEC3F905B996F19BEA89CFF2E4293A58317EEB4DA598552C5CA0C20487ACF183806AB16ECE1BF66DF386DF017E7B0642DC09962DE6614440C5A5951E8DD3418CCB361652234831CB77E7347DBE66BD20193BAF96F411817C8948C77FE270212296E820207FA3B6A00E08E79B001BB50E15F355F9FC9C38C1E5370A773F1E5088A94679F779CEF24BD46EF75975974DA7D850FB6B7953CB033D6A85ECB3D845B4F952DCE4D94D059B660369B89540F2983247329E124167F75F34701AF58026A9A1BAE6323011536445834CF7EB850949AC554F7FB577815EA0BF6311A101B4472DA510770412F4432FCD5BD3E0EAC5DBC38D44DDC41DD7612847BA3EF100682606D3C035D2C7137C3AD80B738EAABA32009143839434547565D6B77859FA8ABC1DADCE8F000070B0D13323B444B50565F72878CBDBEC9CFE1F2FAFC080C0E181E4E60638092B2BABBC7F2FC022C6A7190B1DCE800000000000000000000000000142B3B43" + , SigGenVector + "ML-DSA-65" + 211 + False + "E6B018E6AF97B7BC031EAEFF5BB7C1D7C0C6E7E99D26AD7982F086CA164303F6857FE824886494CCEB2E1C7587233A21D9C8A43FE72F40F20E1E818C8875B269D1BD79E35D99A737A319E23A8189513D4BD01622D8CE54D16F8244EB67579F3326D48CAD8CD0FF398CBF13C1558BBD5B7BBB5EF336A1DB3CA39A17F50901AB4410546146805704484358188205343435775365810270412131180120804507122731361844382271558373873051543234275801878474425612127621855656268420574057661416687326147410720130174200357401066161187212366571811631032327077866171217341246476726488814638881856524337016118600454057346067071241063471426071771181387157225771038224611454543507238074544358714676315450370406233228852676285487851613250216780243315303852746722221267061383400540165272717237145024412101130477147401646123346714143866483754524580775745646056406784613870312457081148550445178785753747030246330541703511363322802113418051867220145112651521711847627786376187625330522214146417637572781032524638355330808834430100508127570078647218173670788543156561783427270087104457227862447518405306045631358161143706677445471706857158277838470416001514348233055235474306367746306584858881446634574105224055654378563144221713308235170238816811148416782557832356155671153773077107663225447246348070087284655835238177033651453374455731644213518706823435831482305221638071802083671384801564824843855128020176782254420673527524130873768432883358566016316667787357548555273731662457578572041873282322876752467807201134511670363387024468408688034812801550164446607863361326382201621827443146007325157447537161850132777370067228671601838171260831210812527771055555538568801658558338613061204114885742780807405144442473621250216321238464008508862531043175741026205865871418778717100423113447570168844364438850352318808466746275126886100878428242827203378003384142420744703045567851451525113121745661565328753358113122830888284463123041284155552874803225364021351253068801288287026756771071230376126506116003840157202842525252503735710245302836186531770662353503546446707301026817745738057541368114851523218684786552455728430863877464845330271136242743378515740874658375740631112803847438375622448428055324317716870205254842008140535706651126561838260488262213371427256255852634674544256033837887547061355025705612378634720778260123421753476764446835403304681351632715172501060164545265123704413483037842183260620617774701418657003712652237760751234168548714186604556524067243733376755014625701381487162624270371215361786555383436444578686684421773521446823352864541637044256001806416130056340346168258514180007674305780860710272831053777171655664201562160667735123514053181357670031362703626178428082750078844008477601704304553840834241207434338084782584327003485001276630325441003823004201117452600113414832302365617264860284454112831688327220663760671655810875478227082201380178603633441772572184088070456012558817628876304818107064203607742878257274663004067626006527675164102838071063810712251531826554345142674230544807613105842256787572125455314257264333446771614160740181810482648667807858703616655212757355365416633073628052170138606517641430640383080634826336455853021248A49639FFE6F8A4C45E5D69BF9F59297252502474B719B44EEA80C22059C4A5DF6D6C17CBAD88FB62A5CDC8FCD9A9E9280C6FCE15D9F936F118D359DD39586A62F522DE7054767B95980B60931A563ACE1C42399B86B2345195A1A768824A0F77E702A8595C371FE878186D785504413DC15BA846D808ED0297F6F4855BD220E1B88664711F4534B09C8B0E23AD755C319102377304B95DD147D20AA6C263F84F2AEF78EF18109679648B56D184940D0CBF06E326C2FED26317B5A6C6DD6BE829621A49B3765328C26A8833571B4A43429669C7CE4630CEB50BA2C7403382C9956ED5368F00908E3C5675DC7C6FCAAE0D4B7266D1F39A615BEE858211BAAA768F8C6E10ABBDBA99AFE4F4FF3271D03444A536FAA2953B98CF810360767D45B5E9C743622590373A657CB01EB640524B01082822BF6559013A7862EA3098F36193C27D3053FB763C033BE38BD56D7E572020BF960D3A65274715CF79886ED50D4173F035BA4BC4A935D5F410EBE186924489EC3F24CF4FCB90F36AA331A2BB90DAA0E4430C29F07B485396B404AA3B72DCF05A9DEB1C142DDE42F05D5BA7D38015899AB96C4DA40012DD618F01A90F87D49F8D33F88B2A4705CDA69318587CDBECBEAE8B244B249FB8AFFA76460498B0CF64023791166812B268EF800478638250B4C639899ED0120562AE75FC95045AF9A8D5468196E28D97D60F4D09085DE52282240B2035686DD690B5ABECB6645DB35C9C0644CBF8F765629D4214AD52DF0BFD39D44491619DF8EAA98D32FF26D2DC9A8ED17648ACE6F6084163B7D32C7959E305DDA431F3332EBE697405011854D0EB9582975DB5C47AECF0DAA8DF3141E9E5A2E73A90D2B37C3E49B2DF79170EC0A10A58DCE1E106813BAE25C856961F373198D21D8D1A4D3F60102ADA354199386A36B3575E32F71EAB1159231F3EAB635B328091B5565E33AE9BAC192A0FAACA59ED96195CD41EB879E1F5FEF6B279A0931C2454A7DFCDBAB42175F701F1C59D4A57FB77C0FBB6631C28A6A2E6FBADA0E3356F5C899F7A3F9F1BF6B3BAC836C311C2C3415861C389133150C2055D36324544717FDBE90123FE8C61A6F2B59C51A025E3ECB2CC4B88110029501B7F26385DC31ED6B53192C522AB908FF118F777E71EFD061209999372B534E99EF360A35BB9A2168BD3CDB7701B930CD858DE89696A225B5E0280509C7BC97919B0BF58C54FE80712E40FC65C12ED652394D34BE60DC3E63606D550FE83CA72E0A1791294EFEEAF07BE46A5939A316605C5CEB744FE16C83813169C646529C9924691B4070B96E7E6C526716C69C43DE63A000FFB960E193F6656028CEE50ABD0A9E396434149D7016EF6B17D7DDDBE170061941F54B04BDE58937B8CCB87C61294A10649C46D5D6814D107B46D7421F4FCE12740972631CE40C55081E375B2D8D7F4069684FCDE88335C9C07D38DF2F4169E30CC764AA63D8CCC032100780B4554B748C9F3ACCF4AD25D9CDD2DFB457C42EF106D6B4C6CC68BC5E6FE20AA8A18ECC59C9DA093495B9F3BEA9759C0917DDA624D15955E4D62DFE80CAA39A9695261C01DEEF3B540BCC18384ADC710740EEFE79702AF54E70F3A499F644031291848068AD2A601643EFFA8E3105352C69F2F99A6E83309131E21B095852725D01761071B99B996E60DB1581981972D789E634731E75B5265B6394E99F2216D5C99E27458D9E4D20DBA1B9AA8A6EF88A3A70C3304051FC3F40CE03630969576A9C4DDCE130E4B5BFFECB92FB8D0ED64135B49848A98ACCF6F24F875A8174D081B538F8BD2F44CCEF08F4EFB89BED967C52D455D47C1568B9FF1E2F277788EDF835401FFAD8011AED103A66671E8232C25511060A8A96EE48D47DED33023F17A130402F37E2406C2C06803F75C83A66589568B5F9E78F885D18A81CD3FB646DA199D8936C2FF62B03AD77EB32A8250BC5F142E9F85E943117E0A825408C419C35B8EFCF93687B7D6CCAB2889F487E3C9B86FDE42320576289B08F1AD136ECD4AAA81143A876C6806E3BF23A1C0A8899711B1F366018844E2D667F74F999403580EE382718A439495A6D3F3A39913028749604E40CCFCFD57ADA9B831E5F40E275B680A9B665EBC92A6D50A1317E45C4F548897F7881E3877B08C71C163DBC79FC698310DABE54CBFEC7EA4C9372C1A960FCBB1A7DFA2523F4B1E1CD12F54E9E25A1C09DB29B799ECDB25E569A03D3F2D6A773765EAACB6645CE7FE5B7630DA3A2F382AE20FCD3F7DAF1FAFD2AB3C5CBEAC5ABDDD0B0E0C8489FC3A07659D64CE083901D7ACE838EF1B07859CE07EAE436C6B23735E2AB5C5E3A1C60D21B9B274CB91BCA35BE9CC63986B5E2BE885C695A61BC4214CBF71BB05C8F26CA976B6682DDC2CF4834F0FC1365F62CF32E94EDAE076ADF29F0603AB20F13957732AE319894838B84EDAA3070FEBC23BA5B8A1671F1092AFBFF5387CD0C509212D7A708DC913CDC33F21D966F6CF1484B8A334EDBA35C1D31A8B77E4B48324E60CA6BC3B09D2C7ED44CADFAC3B3F1509DC3A29F96D9AE9F06F9C0983BCBCEAC138DA27686E0BC75153FF494D961263128DDB7DEDA6224D91E1068B2D5E273FD30FEDB5060EE2506061DD4A48F87B35934697EBABCDFED5AD739116ECA251B60FBF43C8FBFAC2922F48AFC15F7694869AFD7D4539FE180FF060CF9910D09CF755417596E666BA0783AFACF0107BC0F692753D693C770EAF6C19836D65641267932431DC8A5074A9D2EF71F6D1AEF29F16371B287214537321F175EFE58AFC03D9B78BF393402430732F6D6BA0800C65E7FEE999880DF44E882E776C033319E0BAE3EBD914ED1B761B6277F2626F5D45259B4A47733CED5E81E23725371ABA6217066C687494A9EDEF3E02B6804EE8ECA634A0E9C5ADC53C54084479E742D01118E62C9A4695C8207574E5073BBC229D662CDFA951D9A71C351448427D8604F7C507C3BA072E345BBEED7AEF59FCF40D285559926AB3E089A4D318F57D5C58054A597EB7326F2D2180221866BEDB7D4DA45CAD1A43AC059394A9E042BC29F689245895C07E8DED4EE9B590159164BF906DCBBB4421804A1899B4F6675B281F80C3CBF15CC19D8633A1A7E52F82258A73042B6CB6BA97ADA269265E341BFD3EB89AED9460E4E010ADB401ED8BF2CA68A79CAF20F1313E7774200F1FCD41F650A0BD6C4FA7171E9B6AF5811DA077E9516A14F08A3B167BA13214B6250FCFA439F8DB8820CA2D8460D0F9119CAFD9E04D674A0F1EB851E03D1DFB78FAC5F05128FE409372BC174C0DA5EADA226C74B23324F9C62998D5C467621EFACB47CFFEE2AB9E6E550925CB200EC2C90E361917C79E81E77E7F9D15143AFF61607857B063F15CDA5F7AA26A20759A7F4CD55D59387D4F9F00EF0B02C598A8375662C6DC2115740289455DD24BC4A21A2C3B093458B4D9E2D7DEC5C5B306E296D4F1D8C40578E57D0BC30F3F881081EFDD3A5F8D59F68971E48EF3BDB50373D235B48DAA0ABD394E6889AB348856BE53" + "7837B09F687F6DAE4DE61176A74E30B6E2A992EB2FE65318DEC63F79837A238D4640C2F11143F5D9638335B7FBAF19715A82A8D97CF4233B28C2E25D58D132839D0D25C0C08A398B820FE052217CDAC124984BB5FDC09B6C01827B4BCE78E943D65734D6713D09A538A599186025BDA40DB1D5421508DAD1E396FEC92F72311C274077DF21E873F9DA60BED7917E03FF4CABBDFF9410844E3755BEE437E305F67D2B9663250A50CBB16A539A561127A17B35AD3468AE48BEA49DEA71B324642DFA14E9B8207E35F9E026AEDDCF29EB21C771089C10FF0B67E4B6E15A0430F1604722C34133EE1283D1BE3474152C7486F7EEA2A484AC4FBE551CA7320207B7149AC40A44B0E21069158D4860BE61838CE9E462BC448C2451667424435828111A1EF3D599B95E6A59AAE730A9FEB6327A5F005B1321D607705EC3A21B3CBA5046374616F1DBE38C44F783924CFC8215098F60FDA96F486134E4B07C46A262DD3D342D2C13BFC2DB5DAACA5B820AEC21DA09457D1AF78574635AD5B4B640370F53069C391D5D1E4E0D7BF52EE3ACBB099867BD7DB2B5E4D96D4B89320DCF5653D8AE073E1485889887F733FFD1F2BFED8CE2840D9410BD5B45B0BEA6F5A545407BC721A39948EC1C74518D6F102A8A2CF17FD85310D11AFA51A18600A1FDEA146744295C1EB16D653594DFAE0A0FFCCCB160A3E0BAAC84A524405406AD22B15400977F0DF63C8C955F92FF94763E3AA6A834225351EAACE6512098AB46CDF7352956ED44B6A500D3B3F5CA6693D6C5BDEB9AD4A558DAAAEEDC35B2F610408A2AF9936B3CFB25417CD97A2DFCD9881C0417A8EBD0F7F1C99492E96AF721D94EE1F1A3EEB0DADC65D76BCC96216F3DD8462DD4D05E217BCEE743451961FDAC486F9B9C39B1CE4AD8E7070D4BC6D80DDAAC4B67A36A49EA7E2A55BDFAB3C46DE4B7926E2E49E923AED13E19489FA623A53EF9DDCFE263DBDECAE622D57D1B170150ACF2EA7D4BD4891A49074627B4644D2A3A16CBC5BD91A97DEC791D733EF8D88440FD04F34EDF54C4A38D730C58CB4304951475F49C414DB0E5395D9F992CA7BD149EF2F869E33EB00A68070B428EB64A19B6A0E6B5ED2333F66DD68D905E2DAA98B59B0F155DBB35FA2BE1C075441EBCC6EDFEC6D74072AFAD0755241839DCEC287C4F604A7D37177D7727289CD13A4C92EFC76C4B5627E815E68C4C7438AD22537E91B090B0AB8E59491CFF2988953954CA5DC65C2F6ABA2DE011F60B9160997650B0320B8E16EFB5239CB61054C826DC97DDD5D2247304B19D3FA6A061B1F5FF2C53F2AF71D1319F2FEACF341C576C86B157BC8660C6CA4360A4DCC0B51FA0F3D1D268C04FD83354643598EFB6655579FF5BC002A4C6D195144188A52596D4C8DA869A71D16CE27259614D315BDF85E84A015689E87B796B43392D314A8DB6B266F4CF3B4D9F7B56E5A97D1E3215CE4273D233DB483BBDA9C2E7114CC2C71BD6807F462A21F9BB7DDD9FC4A6243B535486F0F965643631AD1FF135081D58F5AA4FA159856A54326D38971279E8DC9AF7F02EA5B8716A78DA04B07F5466F315EF18AD82BF3244832BA86BCFDC10365E9E459BFC32480E57D1608CC9ED80DBD8087422B0477BDF92EC60EEC7C93CD966CE9E002F3EFB8912C9F5FD9EC530E31970340BB57704FD9E74D9F7BB5AD84DCFFF51FAD19A4B0D132A99B5441A546CE41C489F170BD8B8C86F9A7A4B0FB3AC7E3A18F331150062104C178E2FCC6C39EF25066F075687E7F2830FD1FD5B592BB9C982B1534391EB8C50F69FF3CB3BD74198D6A43A6C7500EB6B36C3008CB206EB0B5A6BA4AED1C5CA41DC7D57735329460F5EB06244371335265F25085C2C68CD4D1247F482E79F4838E97180AB274B913F1F2CF57F54FE534EEE8B4D19FEB3B35C287EBFE468663C5330B7428A6123B2AD3F0610DFC5F5CCD12C5CE923F278EF2BFBE2CED9F8D046300CBA7520CF7FABD7AF4092EEDFAF3EF757B7B5AE96490DA5D712090DFB62EDD1F5EF8F04EC2489E10B806D001DC73E05EFCA3BE6A23B209BBE62445961B6581C40B0B51DB42EC1939D57E9B6E5CC071C6B6AC4537F2E81AB0D4332AEB72739F43752CC25E3AA70B31EBA1424E62DDFF110FC809AAB8F6F0DD699C824E2DF4EB27BD0E4C4BD1818EC056FE3EE3C5845D568158E06DAB4C51C178A3EFC97752301A9507C2B0347069E035A4F70AC869714965F9905E1F703F7FBE80CC4B61B4754E08F74D46CDB34011A6FFEDDACC104234EF418DECA03911E24E8073849210E0138FC31B0AAE609AE3D6409721628E0022155E5C51D805111401EBA7589726623E214058CC4257934BA355F1B14D13CDF5D90ABAF45F0163E89F5D6231BC3E39D7F1D4453083D93CC052AB633BEDA071742D5386D10E54126670DCA57F27F5F70681EF61BE87FB596FEBB9587177D4C534DC9A27A344CBB75392DD2BDD392889AB1CEB10B2D54E0E5506CBACB5B7B543ACB9DEA7B19113E1A292370C9640502B986F0026CB718F13744BC8B7266E499AAF637C3D64B0F8BB43A69231E8C25A1DBDF21B0F5254B718F73368AA5C579834C42FCEE06E1E2DA36E528B729A7646ECE6251EBE931DCA6E53DA5B8264671DDF192CA83FAF4042DCB0BE1D1CC30AE342C5CD017A26A1517C6F7E20A0FADE518B7F5E7D7DB90CA9F79F456FF49A29DC848B4650B5F29D2D969A614C5937E21210A2B746AF9F6E991682F4075447FBEDBBC768830CFEF442B5FE0CEC42232CEBA699A7E077E593ED1E535FBA35532BA386B37F761388BBC8833F2C437C8CEC763D295A7FEB82912B3D7C1BC8B78771E23189693EA96A15011993631900D1692A1C1741A7356D2CFFE218E03889C55BF6483D24D5A7088D8ED09FDD187283442DB4A58E138DCA1F666F103726E8F590E6ABE5AC54FF1DCFAA2ADF7154C39C00294FC92E2ABC07EBE48223F366A2B85E41E490A482AC83E6D2C03A1B69BEA18A5286D9EF37377587506848D8E5F08FE446392FF64DFE97032F0065903358443D6B09E3F6870E8771DAEEA319AFB2F748507D73BDEC0C94467D60EDCEB95065F381F89BED38E051B9F71BF2D23514641CDF1EF0995BD635F6F72CE6E8C88E8DDDB0E82E13DFD6575106CF9422038B736459A65D372E16DBB01C6BB3903158898B3DF66DB878CDC32C461EB027964262BABD18414209BBC42E90A6D192F2AD6FB6F72A646646308C9AF856E2F2456C56FDFEB63DA11F70A5A25CA3E8F558BB833900F593FD306170FAC950B2BF5E20D2799C3A4E85D670AC8B17A4C7B72B45FBAB16411317E5F3284D3C1421CBFBCD47246B143B5466997A240D89E068AEBD1EEB1CBE7CA5553D8DD78EA51F5094BA04F80D76644A31186982E6684573CAB6E25B74B67600C5549ADF5E6C6CC02C509C52FDAB1F910883A192BF41F9E00CEC33FB653BB1210D02932D077CDDD2AB28C4D058526FFEA2578725E773035871924C2C4F07E8F130E4A55DA9C1E88FB99915C29A6B82491883035E2CF215A2504CEEE00C876BB0EB6F4BDA9B7881A5A5A4180D256395608248B3BAC8D4508CCF30C62B679E33D1963B363FDC0481CC3DCC228D7ED4E32668467E405901FC21AF17EF2D34BF71D83186D2089A0DA6F0891CBCB4F60F94AE6E0ADAD96675687833C04D73051AD8EA81289D03834131332FE4B5B17A3FE1FC9AE4381D99F300CE9A6549CB40D6D53ECE26E4E7FC3F82962874DC6733F2DD8D9B8E59DA49A7017600993305E759A7FC4BE9B447BD7D7AE4BE81B4AB790157DD7B3124F8C32128827EA25577144127ED1045788BD048CEB0772D0B72397028660057570F36C841C47584908C3DF211C9F3EE2B91775834C2306BB114E59E170F2E8369261B1BDB3BE6EFE4050AFF1C2831F110408F56F9798D622B01F428A46999AB7B4A2A43BB043EAAE860D1B2345B6DB53D9A07A0AAA9DD8379C62009E6DF40C5C74F5A89DD6F8B17F13A00C70CF7384B1BDAF7F52F50EAD5D3F7B3461C2FD4F937DEB96C968980259F36F1958FA18B2D1C3C39BBC8789470ABF1036D952A4EF2CF8078882444774596B07DC3D186D6D76548606B129479A066EDB11DCF26E60CFAD31C5688F0658CA74C504228FB5E6D1ED2848B2739C5EBE7018899D12E91F6F296BF3FE3E5B9F9D397A4E87060A1EF77824C511DE8A3BA03004C32EA39144C24D0A2DAEAD5EF5B239273CAE3D8EC334472D2E57AC6C7A571AA05D340DD9C475298B16C93F6524A67B717D59E155418E38D10867066E4E64D46C13D1057124B7801D037E1DFB054F9534C5FAD04FC32D7163A5329144922D0934C544B4B9DB416C4EA8A482B2F1DE0499AC999B579899B1C88AE1B3910628C186310EDA0AE841CE615169E974F6FC4459C3976D54B33A43A04F9DFAAAA2BEB8911978AF3537C83019EE4F92DC3A0065858B554FBEBADBFFE04412D2195BBDB0E34623745B81CFAE13562830B31B88B321B6566753A2FEB58ED43105E6DB87421E03AD64ED221BDBF3A9F1BD6F69D4114AF9AEA09DDBB391AA1097775DC257" + "270E7D9A16E4AFF615F9DF70DA06779F6A92B2ECBEB0D5BF24B305AC03B393C296BB521E3CB2135FA97B8DB279315F0269A611FF4286167A2CBA4D2A445871A2AB59A23A08E0467B91" + "418A9C20FD0D9D9AE226E14CA79C4881ADBCBDDCDDBA0CABFCBDA35E6B7FEF5A" + "D7D02545BDD400F453F23809842DEE38F54B858B06D970C1583235A426ABCA9D1D9158E3D10D3F9C02E35EB715246C0809F1DC24FD83860583984FEEC382CA5D91674B338D4260EEDE26312C9C2373CECA6331D9A898FA12683A943AEE312D978124D0D7AEA85AC38A21B74B57BD6B089D49F0999FE3178ED222CC45C2E95315D32776B1837EA1390F9FDB59B9BC90F5A11DE59065381BF0844CDE39C17F272CB836DE98040711E51C37A696B2026D8F38FAB751E946AD1761FE74FFC89026726706F1CB0EA28095555A06F322D97E2FB821A07D8508D2A3DC58F91E0867D5C81FACD5600D206C5C02DB181F881E2D72540B24D317E4FC4EC7AC65AA2B8F350BF4C22F5322332FD130405A134A3D0191180659AB26506E23082FF93932EFBE7B9ACFBB08035B94E16DD475D0487334047A81E7F1FB86168A3D7DEA446A4779F1D7293D7D40EECEDEF1E183AFF67F7E368628ED5D957426EE960973C24AAEF9432A3912F0884A333B495EF3345A753194E2911306EDAE80888A59028C8D59C14C0077123A167BFD2500EC86A3059C3C5149BED1AA69A5C1459E0A05B201E5A24003C20D637FD556C49A5DA08A96D896F198B3F4A14B337CA7AA06F57B3795131E7FC0E664FCDB7251CCAD56E2AB29EE9FFF23076F256098BBC10949E2CCF66AB6AF1708AAF25A93C91F694315F2FB572FA4ECD7AD4E1222FEE9A3BF85F2AB73DC13C22AD463AD8BDE69E14F03A6B410A79F5552D601A02E6965F15BB9BF30B4DD7BC9AC8D2D33A806E40555BDA85BD595C5B5CB436805C9327B96102D1FC96E7D876E2605B37C914FB2A102B8BC66948F4D27F68E18699405A08D9C7ECFFDDA4AC46DAFE89B913C59758AF6309102113848AB59562C552FDF752FB8A0823AAA0610A3D1D6751F6FA75A9F5205CD84B4A91438059E2A6F89FA7BDC96E4C701EDD93415093E305F610B3F59A7C25E2C8D1923377B94BE62234D8C0DE1B359309B7F0A1272444FEF393E08B6F24CED7B2F62E8FAD92614045E0E4B16DA142F21A7F73C554141D301EA133500D97C492C724C07FDCA987AFCEAF09BE96E816E0BE557882B6E98EB90D3398B5B2A87FB582FA4F82A8A7AFD5612546758EB8AE06A848A2E9A0EC6E6B06874512C141E3B3FC3926A4B87C21B16679898DA727B8AE3CA833E23A6B6F98A2B07362693C6D8B5A8C449E74C35D9C19F4DEA8B74B3306CED16876902ACB51B26A7478A32FC65D26EF893513E70D639E098886196272B7AB7993934D9D0EA27A0DF2272F4AF8D0E092530430666D8CBC3595B1ECB82A8485FFE44A64B1704C85F8A01B0F6B7C608B0F57960A833F1FE4193CF77FE5AE3D3BBA1AC171F7B678049D5F72690638952435D823AF1F6BAF403AB28DF6CE94172DE72A4743926D9A105CD322C226A6CDFF0D6A43B8EE940723FC2B8BD3543F8456D0B3C077AC4FA6BBDD6CFDEA9D26DC60DB166602C53B5C9A2E0ED1608BE76AB60F835AB9FFCABE777CCF34DF8BDC22BECB50D30D9EC0DA2F862C7516296B6FA4B07F4E840FC80A38A717740CC217DDA72DD2B3846566B0339D5934ACEA2A2F746550D9CB05EA7A54A40E4F636C11E9DB6C9D35AE5EAEEF1CD49598CB4681714F81051B89C73944E60F65FC4E75730C07F66D77A372AB69B6425467E101CE3156084068367D94F0BE3E07289B4C188E256D5FF166FBF66D37CE86B78B03A08E542249436C553C872BE65A1B97109FE59392546F82771A50AD2208FF3B8A831F645E95957B0CBC4980BBC162FDE352AD65F8F8A512D2FBE8DEA4A826FEB547C866587F9DC2568BAB4BAE6B3A50FD4D197DE3B44B387251F20389C45303B27C529BEA6DCE01DAB7C2EB2EC6A3888610FC201684F8EA567B4AC4C5DC3FA2260EC9BF0919E1658D22EA12298E033F44BE4100A6A9514D439F1D8D00825C49E3AB0D7CC769F3029B2DF5DF740AEB7F7D9ECDC01282DE66026E74C33E3F7526242F635D1666B257FAD8F657E9A70C1A551657122BECE6BC388FC864147C0D59453BA490EBEE70707FADB41B8ED0FD2D16D48FE3EC3967B0662539B77ACCDAD7C0173AFF1D82CE76B2DB028EA62A2ADF55555D38B6E129A316AE2AF5AB37E72CD6036E53824F48BC2F0575C9863169F9678FF20B189554D736A6DF964480D3A7C76DDEF048D144FFFC0646648E89DBD40F3CDEC9EE01A19D4D62B4FF1E65460DE8A8284FC86750F5F519E303EA103FB7795E9344649C66E53347EDD927DFBEA055AD89040B6BE41E7C74223DAA115ECEC52FFBC0392D3D2715C7BBF2BBC04F8162E5924072A69C9F79E75EC39D6191FD6F3197DFE77CD7147D76AC39BB5F73C7EC1CB02B3CE8C074AF59420075CE2B0E53C0195D50DF4EADF5765D8CCDFDD7ABAB7D635E06A6513840C35E611999EBAE529D45326B58253BDE5FB75ECD58E43A98DB72C50AF5A72D9CE91BE63BF68B8EAB8C8E2433A70AE9A20F6CCDEFB53754EFCADDE54844BC2D9D81A30CB5640184573CE3E3DD8ADA6855B51F52818B1E45152E8EEFF94122EE906E985DBE8F84110B1F34DFF67A184B5C10B54E093387ECCF71AAB4E698E315CEE4349A8E372AFD276EFB1640E1209F32F63E66B237D375CF0A48627D480FC4F186D48B52BEF1682A132AAA029CEB07D4EB95E6EDFEE8935E7E6D589A14F12B7DC5770BFB8529DCF9A8EFE9C9F76A09E61B7D7DE156B1A11061A3FB2F672428D722CD3077603A0DCDCA5589FC5015B7323ED53155F0A95FBC6B2F928D0B94D41D6EA992550938DAC99CA34C736D59461023141C5945FB3FF9352ED63AD2C56D9B3BAEBCDF560D3523CC0D5D7B49F85E51F233CB0FE7FEF274AA5135A3730BDD1EB74773E7EC97A711DF4E2485501B1CA0ECF6BDBE1970C91E08943A72718A5A60C468D45744D946CE129B6D26AAC4986CFD951BB5FB813EE46E4AC36CB5C5B5E663B91A5C6482C89BB7127886D50E8590E2133CEF820C44E6CB43EF43D33014180D9A315DCFF17659D9DBDB8F57763D9CCF4A7DCD2492DDD1821C3517B200DEE234DF7354A35657CD5960DBFF1E3E733EAC19768931A29E5992FF0B6BED6944BAD1299D5AAAB500A0DD0ECE9211020BF7C3A9F822C73596C3CF95B4EFD0C148D9DB83526E3A46D19E9A83222570E182B5A90DB1242682765F807E36D7421BC77E3243176A44B9E312E14907669C9E1ED51A9DAF5D0B4609FBD111FDE00BECD49BDD06E616FEC70E05F5AD107DB887587C2DAA02DFA161F74EAA5D5E8E3EB18F452BAFD7E0FF1E5D652B798EF0AD19EEA74DDFDA7C4585C670E299D1C512DF09DAF58F69267E38D5D565ADE68A6AA351C05AA02A2580CC3B8A7D5F483785E38F16F8A5784DE0E6A654E8128AD2E68C423635748783B899ABCD8DCDEBB4FB68F190BBF9C6887C136CE5A13D08F8B13ECF9DD547C0134A0C247C22EE2B3E5050CD9E3C77CA1DAAA4A5453F944A1823CC877F76E49C1D45ED71C094AB83F9E7E4B778913BCE4D075D060D7CC2D006995E1FD664D9277C81D22D6E72E697BC1DFC9829FC3FEFD1C0996CE9C631ECDE78CE8112AB61BFB22D7C4834029AEE178DB838A92F2A0B27852280B55239AAE70D03C902999FD09A24EF1EB79890678E6D8927ED04FEAADE840A387A3BC40BC7BBF0FC382258D3D155DD04D2D7B020D2A39A0E66D16E850A91F78194381A3EC1D5491FFF53A26E691EA7BD39BCD23CF6649CD389F34545A5279D2608F79811580C5F803FAE740A7782A8956E675F8CCE4E6DD88505A36A50F13F879221C4A9418D270D8C187AD0630464FC0629E14BE0BA06DAC26F71F5DC2B3413A9ACEB8124B44155A849EE5D9C719262E4319ED149E964F909A9E71F3D7C43973D5A268FA681A3B9E0F0C2FB5E7E4DAC3CDD54AF9170E2C5BA391FF1F1441FE32C81C70B3A3A4F2CB7B06A50E0224C74B5F0D3D7E946C32C6A3CF6D7609BDDBF386229AA1E850A0C011ACA09FC0CB7FE2A26F876702162834D5976643863BA05BF1B9E97E3DD8F73C096F9095A604BD5F4D896B24A4CE7AECC7B5BF85E6736C84BA1EA6196CAC705898AA13ED75D9007762B91D175A1DFF3E4F381E5011ADDA8FDECBDC4CBB0C60E0B87D1688989015DB2FB38EFF3A77C4D27DBE871D889BB52F814233A1D63ABD98B0E57ED1A28813709BBFD3730CB68A5A7DA7B4F1F0CE5794F6E3C13C172E994AF6C986622A681F3C2C1FEA081AD3366865E40FBE23F1F2D4DE7EF5C11F710BC544AD34E362CBC5683E95377A9C66A5DC5843F92666BF4D950653313B2D1EAF106929F378A9BD8E78091FAF7719002FD048D5A12FCDEF8E0A6A531A0B767BC5ABB39CF3E26C8C8CEE8382A2129E2CF7DAB882C4217D07977C652404AD5E93FA10E7F5F02D87E0D1AD01BEE6E54348599AA82BBC5F1530CCAA9E39A15F94AF8A16ECB6DA606837603536E7D597E5785C73E3552F37F299D9F208A20AC2B6B5C1D0AD74CCE5D8EE8A7914C8C8AA6278275F08CDC7BB79EE7CBFA44A59F177EC8A3B5386FD7988AB128C6DC2D89148F4349BAE854AD421C57A2FFFEAE88378C9784B9C841382A9EEB8931E18D74B3726AA41A551AC3D8149030A23B1723FCB105C727597BF06194C6E78ACC33E91AED20CB6C7DFFCFF1D477779929DA3E2E43941A0EA00000000000000000000000000000000000000060D11172024" + , SigGenVector + "ML-DSA-87" + 241 + False + "326A32B5141B532480E5754C8C9588BF99B10C99DDF705C3A4C8BCB1BCDC5D1759511C267A392B96B5AC1A1D7B5A394F56DB0A3AAFD9ED590278A61157DE9D17A3FF17C115F241960267E257BF284B0682F43A63EEB5335CCBE697BE9D1397CE9E34F487926056A95D69931DF38C19F6771A72DD2D1C391191034F525A9895FC941064A0142101474A10315260342408151201162D1BB148C0028112268CA402520025528B04441AA80C9A2485209985E2184DA2864148B08CD9488558464924062624184214A28D21908413054044824C080968E3489163304D422810621608A336910A478C43283204B500E3226112A644E0022CC0243219A5310B394A18B31120204AE4326D64104E2320700B19699020089946688B425040482D881049C9A2889202711B198050261003C981A10620134532CCB68D14013003812921B74854468440466419909159262910A4299A2491C00682DC06408B864DDB164123842D99C4611B28291418645CB04890801104C68423452883422A0B9101A08090D8C425E2940CDBB64909A8614A8011E2362092344824866CC83044C1808942C285129010C140610B474D84428162381113232003C0895B346EE3308A20A8112446020B035298B0114230651415802422911941524C2204A120422197451A121119150981C00C2108315222716138865322000C0664C02680E2A8681B248AC122861130091A9360C2262021452991402020400842A80D2400711C200009898822B86812C4088AA240C8346219882808406D94082621115113994012A56D14372D14C911DB288C84122ADA9080C3C2504940081AC1092406111A078002118D541804234421643286884449CA360910C8841930241A452EC41488632228E21471C8060D84A200C02222828830CC1029CB9424D0A06D23060899B200A0224512266A089021111989549691C2482D19013248200E1C064000328ED4C64094164E01118018920962B05159002683B8001309610A10210C940859086A14B16DDB268504042D1398606202809C942D808451001630D0420C08104A04435201434A89024113466DA02490D33245039844229371909430C1387152C880A21870DAB080D00204CCA48C1C120063B24188A889504481CC400E51C21020B4001CA32DE2B04D5B14854AA88514186A24A3084C42720A91041494694984608BB42551846CE4C66D0088490220829328200B3226DCC02514004C5140919C920CCB462E830071931292E0228063401209254863308912248E4AC46511400052A68061960DA0B8242139604C9249D3A66CCA2090E2922CE4C865E2306E64C0510CC63004238E11307148A48441162888208250B291D01622CC0450DB0230821640932471008970C04050D326409CC86101134A1A344012463091200C40048E5C848819062E52A86121174ED238491A4111C244604B04880B414DE22070630450C9466940846011296C21086840060D11A3445A906CC3425091243189468ED0068A14174951342204316642022660082850262884208C1187290CC0284B204C51428D039431909465111922922824CCA06018154C89C4908AC66C99921061A42412B30D1C38061CA168D31292CC9800039831044565C30202E3A0810C860C1487308A362681469258B664C2467113128682482EC0326DC34871921484E3C069DC306D18A6649C2012248730A1B2652028705A4005033190A0C02091222D8240899A0000128028C188054932910125511B3600CB824CC0264E18100609A1819B980D13418C1B864CE2964810B790989441D82052DB100414046E8BB289111068DB322E4A028E12A43050C46192C44914492A9B388160842199A20519288291A4314A041152A885C044658C964520C781080448DB386864048A09B624D4C660C812400A264C91B000A4262E09098E02378ED03689DA46902337618C202AD3206D59145192388154B051D2906D129900432809CBC62122C67014B209D8468121C28CC132250305058046410487605C3491CA2651CA9688033291D4182019946112B044114581803285239901223024102771541645C91065CA104112892DC2360402922D801282CC309288B61048220D98288562828010258D0C342918810421403083327264C64489340A0B15808A207012A14940A20C4096051A0531C28481019609889450093184484445749697E96509975B51BEFD2AA63C2E98952B5B50EEBE3D7624A254838172E568BE420B6AA3D6554BE087A08BB09D14C15C41806F8F5309E1776107389849650A0841058F1CAD023359319DEE38DAF85122F2FF6DFC3B9DD444F2BB1AE2CF888B8F15576B1480C9727126BAE6D527E54316EBF6AB31196E4EBE365FCF6006921EA3AF403AD3731A35F31D0E24D82DF96AA2F7BACB0184E6708DA086A96A987C1BCE93764EC70B2B9D0B413F668A66D24AD8CAB0B10B01D1CE57EE28E4F44AE01A2C150AB1FFCAB46468E9BC342EB36BF242F242393E7554A9C96A96EFC2F7E3213EE084803F76A7DADFFCE63E90369D7155E7219D183D261B62C12FED80E4F8C950355B8106C869BAB9627517BF9E34E9886233855AC266FCEFD3928628F95747D3BEEB9D70CC12808EBB10E2A3235C8CA7CB45C31E2EC381010BB6702548CC41950BF85AB06430C2E39AB35A5108E13701E190AE90C95B8657A3F63D90DABC2B9D395E73DE2D47AAD0330890DB2324AE8E1A87C397B4E1D9ECA14CB4DB88AE67644AFFDC9067F40E0D4607446E1B351113E67D6D7F4D3E5394EDD5A0F7608334BEEF67BA4FDCDD73FD16A7B5D276696D132A65D40797E85C2819B8141C73451D2C4D69227FC282CB5A4C8CB9E0B1F6B7DBF57ADF9D5BF01EA0B05030656175D9EE00873295F34E3B85C64F3CFB28B3402050A0BDCC88E6D64799D208F8A43575BA5FE84C1DACCAD5870813336676291295A58477C86FD419465E3CA56AA3153427AF83154F9450166336764D7A59677ABCF09270BB5B36E53012EBF25DE6711F13CDAD210ED2B3AE190255ED1B4B1DEB79629603265F15F2667409E6E6527302F42B4ECBE78092D814C73E2565FDD1CA9CE4424E3F4C99DC3FA82D2DC1D64E90F1655DD8B9AECE4595CCBF00D9CA5326BA0B3659539AA68ADA482158AEE3148D7E83001DA0E56DE8348E8F01FCB857E6BB9C590CBF1E6B2076E43B992BB4DB601C6171126656F8E3172B7F796C47D010A6176D93E353ED0AA1747F0F8952E6AEF7F902013A2483111CD5B5DD3AEACBC76BAE1E900A125DF4ECED2D98A381DEFE7C9CAE562FE859F49C4333200B799B9CC20FDDF8622C822E33C05164A85A99074427EA725E8C4189C70F573091A0B4A472F05BBD2650F7A20F325259D4E49975BF523051EC37B47101148A56E30A2C2FECEEA6924E3033C4FB0D9D70D397FE9A9F5A32BC84BF819402EB050F9D3410F8960CA45C31F98AA88C765316831F38465F239D911C8A17642624C31811A3031ACEDF6E27D5D9C9931416AE688A465BD07A38D38AE8BAA1D1C0B568B1A3DC6F16E9D83324A959E2CEA6A6426C89323EBCD4E3E7317CE74C556271A488A49BEFEDD5C2F8F67DABAE474511AF90E58E4C509155BCF4755E2CAE74552A9A83115989EB6ECF103B2EC4909616596B9A0631B2D6D6B1AECF57F30546F5B1237FDE644402AA8763C023D2430A0E58C22400E2855266A70B56EEC792B2A458DABE7C835165A11C368C331FE3FA7CA8832BD54C01F668E7377D59D74010B0DEA6387937F29D7A878CBB5C1D5738B6822D0E8696D17F23CD2E7A50AE1181268C4F584B0D2E246B1C2FF029B20720ECA596D0E4C946C015DE8D7EF370FD016CED707558D52B6223177D65843052359F488B5423110D69E1CF7204D76019F5ECEEA89C10EEC07DC00B0F9C57EEFF7A74064496C906CF59F42DA843EF00C3A19EBE15462322CBCCCF09516B853384C7FFAB23995EB8A4BEEA9AF0560A3476C2886AA6E1467C59B84458D9D90DA0FA7552C4D0FDCB330CFCE0B05A53B41E75958A162D997A17F48F307B5F8872A13C03D1BE785464FE1E5DDABA564D3A50947DF15AB65675D895DC2BE8B5C346E433B2DAC739C010856680504E9BE75FD01090C9858F9BB49E27E2D176DFAFFF3D953E9E3DAE031159813AF05372EB0B9AF203485E1F929995F62C1AE3769EDFE724BF38A3F88FFC8CA4378FDE0FFC1631E6A878D2232C67EEF63A7A3F5B0ECF08D2B3C5E232E8A28A995264EF6E63655D5CD31B0F17E4D525E76AEB58B715AF30F3E8891823FEFA51DCE32958984B61D7E2C488AD70200D80BA16CCDF2D96559B718F64CA63113A442803A776E81890D89B171FB05945C0D7539DB748540811A6396C9D07E10384E74D42375A81913E7D0B61C796CA54CC025BDA26CFD043C0DA42FCE5355CB6E0F03554909911245B13D5987BC70B6188DAE392EF23C79590BC06A14151730D89BA936BF20EAD62237C9DB120EB91956F0861C92C37C7800447BFFACD3FA50B8CD1043CCF164AAAD2F128AB030E112CDC36A8AA1C8D60DB2BD9AC72754E508E5D803AF95C73C1EC2F6C5D736A3D3FCA4793E0D03BBA67E0DD5D4443440813606E9F9FDA3241E24A183DC3DC7B86712EF0E2387991B7F65FEC9BD856B90706DD5C15687FA2ACCE230E72F4795E624BBB13AF7D38B95DB1E8DC9D52A906C786187A62786372A956DF86963EDCC0BB4B54F3A06202138A0D186F54A7F69F8D05DC06A36ADBA2C23AC91EE66D89B5A468CD52145C30A2B07412158092F5FD828526E17D747EB9F7CAED977377F561CA76A22A9305E8C36D11092A0E00F016723C1FEAFCBAEA0F88583732B1E1CD5EBC40FE32018EE953EBE566949EF39A9265FE5786606F699189D0E5A7791269C9E8AB21B53DD77320C228DC06BAFD6EA1EF7DBABF582AD09FC752E768B88DCC61D9BB7122B47E49C2179467149081A02065EDE3DAEB9B54A436B7BB6AE9902226EEF366CCA066E6F1214450A4FFE1EA78E900626973C33614B487530BFBCFF2906D3A91D70F6D00770FC5E17E37928236EFC885ABC62D78CD1485CD11F2417292FD6DF4BF5FCF5A24F53450471757EF22333661405230FA161BB98E32E6D692769F3EBEA117A725E2FD1DC30825A87EC44949BA05FE4709A517F17151CF43D694651B2781293C1A4E358F84FD258A658210A979BE5EE6A127153908F3D3A4A65513C5698E6D559130DCD75877AC8E654FB5B5CC027CCA158F46C356864667B7E0B73636A9BFEB6F53E4B7C8B51BF50F4D8E376F8AFD92EAAB741F09E4058A68F71E217BD10DA64AC60A31C1BD20D98E29311B9A305229D4C6DEF150E7796C8FC2CA0F786553449E1AAB04A61FF395E4DD31B75718FE13338CC617E4D2DF00363B56142F2C09F4D443A1B9AF0EF21EF343CF8B8515FDEC8BAD535FA14BEDCB66E00AF5334F11C3CC192351729543D03EDCD6DAC2306F8B9C926A176AF6B36EEEDA116C361646A3AF6B3F95148C3CE14E3C8CE3D44E642CF5C03D9ED7E7BAB56F1CEE0DFDF5DDB161CA9207FA84D3DE47E967A308C56BF9FB13E73D3A8B233AD7D4850D24B24456E2600E169D87DE3523AD2CB607357B4BC339EE36A6F9CD3882720B156FCC1DDC24D09FA12CF03223017E32FF176D0BB71506F01D551A0790492D6C765E00CB601A9C36B9CC72D5D54B1EABD6092824AC54E8B49E954D0A5A163B48256267133BBF10322C92127BE6B6586456B78DD1DF6B5EECAFF7B2CCE8FEB38FDFD28C674C706B729E53B19226B905064262C918145F1738984552BF99788D5BD26836D07455CCC2EF1542280589BF3BF871BD67860F35C27CA8D7046089BC25082050C2F4611492912C03E54DEC0C436C01CE20337DAC3E9CBCE14630162C709B7A4B59597B0939319BD7BED910859691148F2F774184E258F5455DA242DD32837512D12EFD110A765AD6219C0EB1EB4CAEC299894C8802A12B5E7605AC2A2ABDD15B1EF9797F1FDB29D852FDEC92B4BB82B459FB33A1DE3662C1645965B2DBF51483A97A4E9BD430C5AAA815ED66589A836B96BBF86810931506ED0026F949F0DFF9C1B3162F2D890C99E63D5C970A3594A3661A10FF225C4B44E4E2B678C27735E924F09091CC3A97A64E74C178659EDAE4290C88CB30FE190EEEA3ECB53C8565C3D13D99747F3914A296FE3CF0F3E5F28E151A5F6ED5D357D5726451165FC3C07955878AA86DD4709A08F5DF5FD885431AF4C83CB5153E7C75004EED5EFBDE2FCFB1FF4F13F2A119D585FABE0CB5FE6339F682AE7FD3FB20FB1CAD4DCDD8A1BC5909EDC5E3BEDA7D1B117E498E0939F87703CB923DF9DA37058FF9500C4CB9069343A36038CB67AC9AE5DB8787F1DED5ECE9DBA7231FED0C897A47E0C7E4B50B18B3D7F04BE7B5C0D6587CA5E084B38B2151D1BBB98F6515A1AF48E0E31C915C7A6586D723E1211F061702C0D38E50EE4A8C269BEED63742F2A65D0B621694F222504D822A9909EC45A889FDE0A0A04EF3D862D5B757F624060EB51708912B2510A001B689BE87823041F764A6E62828E0DFAA1D5910273293C5642368593ACED4AD1BFA9B41C1C9920BB86AF0779040631E919103309A69A4B4B114871F0965680AC69A09DBAE204CF9D15D5AFB1D11D3BFEB49FEB864C7798CE12C4141FD0950AAEB549E63BAB02F0159EC9BB7A20F4630B1DF088830B53FDD415BFB891F15815416C4857250F9073C0D0DDFEF4609B178C6DFAD676642EBAE50C8731767EDFD1B9F8DCBD595C81F8FCFB624C5A9DD359C9DC48422044193632F9D29A1E2343A85363746DB79ED8939E527EB8F983CD5E6761652BA65233F6882FC6AB93C838CD8710B39B2F40B272655B238D2ABC52508095533DF374FE08630CE29939198C51568304D2DED9E6CE4C4F17A7E985E03B623" + "F0136C8C07CFC92954E22DD375C72DF823EAEC62D6A7EC30ED5348C2C58E901B2EBEFF5A3F6190321E667084F8C221C61E31FBDABCCCABF06EABEE9801E78FC5672596968D4C291EC5A82E81975BDBDBEB49AA5553F741C4548DA7688056DA4CE808E0D49E0D0FC8864FD2FE91007D16136EEEDC3478CD47FA34A39D15B06E98A6C0821571EF4AD925DECD213F3BF9F82ABC18EEEE810D6A1D52C3E8E735A7547112398AF8A8D22C5DDC5D41E2EA99BD1D8AD3BC04DFAAE04A7ABD441CA7B68320B00CD9F91783FD781DCE986157E5DA96DF909DB9C8057DEB33A27CA54B492C75BF99C8C5309DAE288C84095C9614EDF7638F66D32B42BA78D20997C5AB89FD272B2E7544DC3D254564C8FB827E4FF5CB67282F497157A435DEA40355E312F1880787AC8BDF541F92D8D72BCF77991DD55F7BD89D1204AFD7BA5599FCD0FE1D8F98731B9B2555473FC5414E202F5E8A78A1D35401A289FB2223D1E12694E5D426F75B2E0211BFA9B37C7A260E27D2A99C7C2DE1A42415EAC6762C5ACE09C3BAEC4F75F3D9FB06A5837878168D50724F7EA0EAEA4ED1249CEB342909B742222C9207778076514987667BB3C8C0001EF07FE39CF03656C163CFEC4422826D2136413916AC40F1331B9021401035F4883056AC76D8786366A9B4B871A96CDAA8A78BE8BD4930759CF6BC15549CFF761B4196B1FC2B14AD1FFE834721DDE8EF30E5BE408651351EE153D78B859762E9F0D69559999DBF66E7123AF26812C9738242ACB786EF4CB19B2F4EBC92FCBF59F1EB59EA516A31DD9C4430D202D04E300F698845F053D549C0E6C4CA7A5BAB17E43CC3B643D8DD0A879D0B5B390D975FFAC9183D14A396B5A29CC32E74B87BA7A069DF9A3048F927A69CE8870877E684143983F5034B81C4D7F4FF3C0083A6702513D22091AC987DBE1B8995676DA9A1D9C4B11B28A0BB738A7146C9C5B843303250F5B218C049C654AC95BEBF0B351EBDC5F543B4D9C01AC6EBE0DB72ED2954D948A19D8C6666CE0866F4E9BA815E6A4CB9B6EF0D1C79D3EE81187E8F512C4996503F1D7682D874B402602DACAF9CDF696F2D9F37C0A51966182064DB1BD16994A1995760C8DA7A59EEAFB236DFEB0EE6F39960BBC0EB2FA7A0E550FC316264B5970D98BD83D203D0E86023EF4DB94C681C963D808700A13A56DA33CBAA62B9119FFFF02B542B7D3119E8BB62268581D3B1FA5FEB7404B58739EDE800041DB8A3C24975CF8DFE637C82B1843C87DA273FDA37E962067C5B14CE7E2AB7A700E39E25423C4EC55C0B9B5D6568CF18840563A49000E7BBDDE6FF973A2B9999C0E2EB809CC18D3ECCD0DB1FA540EC7CF24EFA17C4778AEB633AE6CF7A1A95382B419E52E4C1677F1D271C1AEB4BF6C4B67AB2D064C4DA5E35CF420340AA65E8EBC76B9B04A3F6A41F5FDC9ED6BFBAA63E277AE002800096DCA2F6DB6686DEFAC6F2DAA6F235EB305AB550645BB7012BABA98A1FA42CDBC56E5DD0F0D451596FC43856D8EB664FC9461E0CDE466EF3AD93CA2CC0F6A2321DCBC60C41FE314F679396D24E6594F6D86980C1D649A68B7A9ECE2224B9ED069686BC6CE23EB94908F55685D1E9927CB5EFE6AEFC39B9D937A84BFB473A45BD1868BCE1E24AE5BE187CFB8F23DE33C69A165C6977066E52689EB6C8FCD1C60385796305AF2C9311585DB46B8C2178EE7048CD0D3DCA377C6E6B0E755B3E1BCCE888B19BCDA40E67249A34AE82DDBE76131206189EAE8B361430A9BD94B7AB8C0B36484A978C0177606CBCC9014CAC1829A32C07CB270330D34827CE1B6FD3E06335FA2FF5E719401AD240A2DF5E7F3CFFC52D68980FC206663D32B1CD1B49A10E9C288088ACDD560BC5178D2E6A6E05FAF4EA6510E27C8CB583DBD33D250631606F0C4E647930EA2B3E4B159C788A421070D6D34CAA111AEA8DF9B996099457DAB306B9AC7F76D0C7370F22FD661AAF3D72DCD27CE0E46628B8CDDA4AF8213A60487CB628C7B4ED4253B3311039D3B76844B2B844F49F56692E80616CB9247B68205BD01E82F3EF2A20C71C374E94A9193A936E748A5BBB189A381CB6DF0D096FEB7FA4E3EDF1FCB226429FB0B7F1A24A37407AC39366E8C53EBD107EA6156DBF77D429F0DEE15AB45701DA7220FAD6ABB7C2CC4DCD8143147083C447A653B401B5BFAB8E9C590C048E4744D77B58480305B6D15A16237F2262C64AEAE5E8FC9BD856CB68FEDB61891DE3704CA3A5DABB6EAA30A2EE8D578128A969FEE329E7FE94E87C5613B6AD3659C0E86EA4A7CBCBC42D14CDE7908355ACDF1FAB11BA49055114F36F719EE88277183372E3046047557AB82C2E39138858EEF5FC39254E8C32CB7204EE20467D5A4307FB19DB43A1DB58D160467CD0C419E290B71B43C67E144C4E26A63CA026DBFFA9FEFE2D6BE8A51140FFA462D881F118418F47217C0442052984C4E9B21B5875C21FFDC2DD5B71D7D9984708843DD012105858F89C9D1FA7C1DB88DDC07FA98708289FA99302719987E3BF9CAB03E1B16184685CA3609E62C991C44FF4F9BA9834F30B65EE329779B5FC593A00E939852A7DD67B18DE7B138E8EDC2B7B2695E11DC085E5890B2E38FF813B1F742CF1A5BD40758ADBD859BAD995711D6A96FAF01CCE67F28EF6C6A007B253719954A9FFAC46C27616B1AF85B214DB50AC36EA53F2D4741D2C9DEC7FB78C3C8B5D0BCFB5E51D5454EAEDC490EC948F7B757D2D443DB17C680A8E1D009BAC63DB429B19428F51CED97A27056F3C7EAC4A9E872B1B687F27774D5E06B3F7CE7BF05C000F98DE2E02A19995C36429CAC6104115CF179CC0DC5234F77092F3676E3D8B2E3A038FEE5F6D9EC8DB44D15B6FBEF0F91A660C071E97049F863E9BA16975D994BF9888B70A5C7BDB1EBB57E0F2BB360B335DDC48D24BC7C3BB9A4ADB48BD833324C316A5CA2456F6AEDF52E65ABF3AD119E45EEDA8FE58D63E52DC61532ABA71A1B2A02C8C4FF0914A85472B21051697FC9D05032E25E6C5F0A3F4BAB5A17BF60A74EB7056366F1FE78C138FC34879330D9103CF20C4E58D22F662FB6ED71E5E140B749A382329FF6C26C714B209BB529E6391D40090017E0F23938036DC45AFA0C5987F9B18969E66035F2243F4B1B67699D7B486BF24AC8A0FB2CFC18F03CC480B0863BF32EC76171893243C8C8EC2F3550F3AA88EBAA92957B2878EAEB6F9289CAED2B6A40B86D3804241684D043D97417D78882C68F6DEBF07CBB9BFF65959FC7241C9260370CE97661185996931236B47840710F59BDD77F8BA14D91E3A91C7DAD27F27A8DFC585897DC7466ED77A1BA5D7FA872134CEF1687205C9F557152E57F370E34CC842410C2945C38C46E9FE8309313368E9BB4AB5CBEFEB20D2AECE41548C58D5FD24D60A6371DD7E84BD585D269BA3265682E44B45E1719F4DA57D06C3474A9E47241FAD578AEBCD0790963D8F39C550B592889E2B9FF91DD22E954E3EE40E1AFBE796ECEB59B8DF200FEB3FFE43CEE3F3BBEBD2624586DE100B06106A0CDCF2CF94815D1CB90CA065EBE3B17D0A28D83FFF5B1AD7DEAF7C44B3A6F8FA1ECA04C0684A261C0DE24FD6C2B112C2CB91B603002E34119D5B19BB9D6AFF09E26316FDA67C0FC12BD86E346CB833F76B1AAFB2AFDB5D35C8AA3D780B52F57B16BB8FF5A97DDD7A4466DE5E63FDBBDBD3250D92E615C47259BF7B83F753270A37D2322893061D79AD95C8744DB7F4442D35CB335C2C4D49E422FCC6F78A100CDA2B165842E9109F586ABF335FA8945686720D086002364D2A7412A4A576E46DC9271DE10E7BB73B3E2B35937A78BDF6C6A23CDBF7D934F70D24D3F9254ECA8EFE39AC1C9F5BF9C24E05E294F94372FE6E37F27004E2A67544BC3C073E054120BF2FD6EC772B2DB91241BD9E4E78B3492B11FF676F8ABED4262348CC140C040BA4310CAD8D9E106D9E61FAC3B6CC988C8951134211E651CBDD648A8899CC1B0D04746BE7043D9B3F54104DC0559B38B0ABE19930190880101810EA28DD657DEC22505EEF7D446E22812C4A91F669DCC88C67ED17B93D5E6363E92B735F1C6C9FA1442A3263C51898F9526D776CE3F944834A4449041F314C83C2B495DC6775EFA4B775F0AE6012D1B1F136EA8AE126187F08DAD40B10561634D8937696CA8A2F3CF85992FC784C5E1F505A5A36652416394B606697B1E7FE497268FABE31210B4C6D4FAFF85F64DF6E7186F20EA0889DD252A8B8D6D69D8CA751F26157BBCFA54369FABF02F768E104363C343E8E4F3BD3C0B58ED76575462D52A660416815ABA6C5815EDB5C5D88E07F1D4B1DCADF4DB233CA6C66756BC4C7D61B607F4961228159FF1EDC016E9D2C1C0442293D1D49B325AAA28B9605A5E35D604AFA8AA4E5C247883A1392735DD555036C1B4E21282116EA242F236A0A9412B04F5E44F330881F914B653F5F9D4AD9C5F5A09AB794ACBF8FEDBA2D055393ED7A7230BD369A4AA406D2E2CEE7B003A4513476FDF70E85E10D95DE3ED48FF2731A4CF062429B0E3BF298A83C75BA556D2B8EBCF7C6AD3A6AD5725945D53D3CAF0F58883C1E2F330E2B3DAA32795A5366DBB47B103E01CCA3880F9175A93A789DCA2296C9213022A79FFBF99350C37966E8037F4DDB07D1A621A96C41D6D20925D0FBF5F2C03C6A5A02571810A7EE4CF5506848B35F4899C01557068B8A24FEB70BA0819DB8C9CF31B6DE9674C7E6FD2735F665D58A14C65199ADBC8F6E213898CBDFF7A51582EBB63DD3C7259A326DD2E774ACC3BDC9970246971FA4800C057E0330A635B8E59CE2B9239E21160CE8B9523C9C739F3CC97114B08AB264A5D9DCF5DE3182EBC00E84E84CFF863BABB2D149844A59E8FA5C35B391F5C769842EB9AADACDC9DD323D9C96856EFB717D030A77F436935F83B6C2A70FB5285CA509564871033EB52AC57F17A9DC9FBD034468BA526D73DA762BAD1883F3A3B1554673252BFFB6E5BD5CE56711098A31CF459AB2667EA3182E814728E9FB120F2D9121340E6E039DF7CF38F585A25CBE5397149502D043777A81942E9DAEB5F5516B096DC8DEC8E3DB9D83A9F51E99C2B0E4D79EF1637A37FFE912D652DE3B7B792E131F650DA22B3605F97D0742618D58E60DD82A2E23658A6F7B6A09FEE8D41A8BEC56ACF2A9BEBC01CD15C905A67A541CD53BB356066584BF110ECC5F2C031AC283AEBCB5AA2D21C02E3094A8259C7D47C9350203969E39A80C5620CA7FEB3261BDEEA223C21D52D73E49F4400E911532D94B4A5E65D8B125D17ED3FC05600A2271A2393BCE5692139467DB47C30B27DF52252A890EA304F729717A5531E63F1A91B74B3670685704D458026ACFDA964CD404E2457A9CDABDD8AAA3265B676C96D1AB225A58FBE4356BA525E960DDA2CC97D5BC9E7D11C5B412A431559DC3A9FA41624772A72E2291A260564050ED610AFF40EA0462AEF9191DC9887A8C8795C1E576D6C1E8A10D81DA1F4C762AE37E9A1EBEDC32858E7118242F13B4B0FDB0EFD68023CE8780C9DC726F92D3C61D49317E39783D766B4208445E41B27D7FF16DF212947314FCD397FC539EC796B57666E19133A16AA3026BF437CA5B17C62F103729893C842BC2B2DAE2E078D737400FB95978AC9028977C4F189ED7AEAE79DCCC75FFD7C52FBF5A3EC4C9011F5183930F780A90A2FB8536004E6C43850176A4E4FFCA5BCA4F88C3E91AA8165E98CBBB5C23D924525DDC96D0C9264DF3A974E6DA2899EA0CE0B6E89D5CEE7F9156FF279C5FECFAE850CC7EEAD73B81382520D899396C5EC0671B491DC1751EDCC66D0B4A5F7314810E20468659522729D4A9127BCA45FF614A8710736E09089752D83CFF6555E1389CA3DF39CDA947788EB826C90C71B4026D70A040ABB025EE59142FA0CAB6E2FB731EED7E486DD2EAC7BC2494F762AB827806B647505456892227A6A0940E34B24C1E2BB7F42D0923B8F1950239471600B4C3AFFE692E199ED6E1AB51908ADC5DBE5DDC0D62604628051F766AF32D6ACE79372B706CF310282B8E183A550E432E5B8930A4158F37E0331CF17A57452DEE1B015FD4B837E6EF02F19D9A407945F69ACDB0211B86CFD80BE4A853E2131AC82959B39B05DEFB0BAA4B2BE56843C4DB6FD48E710E5647CF9C64B2B443913B2DD705DE0555B5CB283FBE7763556C3DBE98EBE51D41F1CE075E0950D0073BCA22AF8052A5025DC755F1CF63DC5EA6D9ED34BAEF0EB4EAAE29ABE0631D0ABCF8A39BD60A2E5A727888FA9908F01B79E8C1275A8C00DE66F8B9AFAC187BE6C2EEF9C07E5646680047C1BACD74ACA45ED5C35C49DFF8B6B921F11C722FA23433566E46AC4FF13D440B075342B9FF58152BDF4E4CAE3C3134A50F6F938B73079B0474CDCCC2D03E24100A86102D5BC99185A8E5373D39A7243B087C8D154D4F9118809E8291DDBD11178B8555616C3D229D85E0DA05201BA3F75FAA7FE255AF9660E87D6A7E66166682EB5758DC9E84E5885EE3C4E3B07A96EB1DE9A84340FD6FE462F4E552C681804AC48A7FB607C0E008EDC0DFC494F855CDD030FFE6602A0D6B0E5F9D7E8CD3483DA759DC0547661C1CCF096E4D687D44FF7371250FBF27A3A18FC71C23DDB0115C0E4742D8F8C83FF53FBE0E03CC9D322A837B2E9D2F5685E931D5CD875B5CDBDE0D3AEB9AA8C77AA10F55664411D4765D4E787FB98E1E97F9E2EA9F736A4E4AA60320D842FA80A20234BCBCC65068D3D14837D5A602DFFEE57DE51B4ACAAF5ECB1BB6ACE5A5BF95241E81988C0414DFFE346A22C91A665CB0057E018FA5C184CC49C0DB2F66083B6B00F00D5E8F03637F75C10C4E7CA88CF7E513EEE12A95142CE02890689354BF41B99F71A65E41C3F45F2C25BAED64F5D9286E84254AF50A72966BA053F7C3AB9B7BF9B37182B466693938D63508AB97807B882C3EF03A3A6D10F35BC4ABFD2C36612D6585E5706154AD746084621B65F22F4C5024FA0CC8E07E99E2E40914DE99BD38506D9DC7CCCCAC264BB91BB906F2D765D88997E96FB29B034CB23D0E7BF92D7B3E690D3D5708D5E70E6ABE64E3CB4C1FF2850D0B15FBA25EABF5CFA2B4BA900A1DFF845311CAEBE695ACE5111046F81C55F1C90C044B7BC2BC1F0742EE6B94F37BCFABDCE7A110BB92115690BE5E7DB238BE0E48AE0CD9C8DB6BF521857DF8EC3DA3708DF930DA643A7719DD13562EB1418DC0E658D297EB5DA40CC2064A33ABEF6AE7EC10BE40F8A5661B45F8366CE82DFB6CB929E7572590804C392880CA55BD1214803F5CACAD46E9CEAFF8276CA9DD3542D0A0F0BE04EB2649BD6B86E17415146C64E04A1E70CB0CA9E7EC424CD93DB8C00AF4E0692977F5CB5DCC78B9F0D1BAF14400A9629E4036B0091A35F9B68415362577B22F07586D0E43E85EA77C15C41CE5F9D0EF73DD38BD23E7C2978538B4BC73F5CFB5EDA0EA15F01D40D6FE5A98CCB1BA316A40892BF42CA0CFF26E5D2B83BF5C0115E276D7166310799FD2655EF2D930F5B1BB7318750BB12BC5B64E2669BC123B5C4D72FDA76CEDDE59FF37A55865854194C7E896FAB94E88DF1D4304426978D678BDC4781AC5BD99526ED1C493904809BA9190E2938AF820C58DC264053F4C2AC3CA36A06E40D224AA7B21F3D91415A28149438E943CA1AF5D32B9EDEA67147361ABD8A933C952FE1F251F343A55537F7AACCB1AB2ADD69C736969C13FF249CAAF5B492C93904E4C64F63C6BD91519C6F4F8DF3431C4CEAAD86C398215B7F45C90F949531C487759E0FDDE8158AE08ADEC7F78344EDAC3F8AE66750BEA2F494278951AE2C205466D2D2D0EF4BC7D104D7723B25FF96DF1AADB0B5863B5A487EFF19BFF69FC80D09EDF72A140849E0406C92AB0711F3BDE77FE166293A325E0CE6E9393C183DA782795D19BD8E4F0C03D9FFB899673F682855D824374A7666F7A4A853045DCA40152C12367A1C03BFA418B0AF912A02B549FD86F65852D394BFD725B13DE324C48990D29C1C7955AF5C20870AEF263E2D65E646391243D0F7C82EAFDFCB5FE02B239FD5DF47689B8026A94A295A9BC35FA6F1111E1E48545F6029F4A8097C589659E1A6A285E51E76F4647176933111BCD6B95C13F63A1CCA425BC465EF8FE4A81FED6A7C58AA18F31517582B12A6AB248FD437FE147883B45DFF5C9EE8239E4C8F941D4E65A1911EA7609C79FE6AC99EFAAC897DDC743C3D3BC01DE9C8B063A1914695EC35A5D10942147250A5BF846BE9155F5BC52F93257052F436BAE33319C1323982D6A2A459F85E9D7FA9BC802006C8DCFBDA34D3CAD96C8EE9FFD967EF841C1D35DF9B34B6012990A8FA6C81FF3CF40012D2C725EE78191E0B58B573890797DC2BBBCD1416EBAFB300235BE3E767D79F73626F876037D8F2F3A21055C429B3F4DE53F9D6DC6772825A276DB76B0CC01BBE79AD3CFC76301592998B896E09EA6DDAB6E0B783961920AF6BFD904D6B8C7F3B7E6FAE978EBBCF0B7D6CE58811EE57C7BB22F78BFB9A1E2834CBB8353B930F7F039B4C51A218FDAE6CA32DA431556B1432CBD06B0B82D6CAC150A08C6A3BBEAB1116ED8D028A939B9E030ED254DF78E5581270EA3E1095F1284EEC45A48BA17B758283AA4AC336A3B61E8B9173FF19AD9655181BA7242D94C13840393E49AA199472C16E8D1F4A602E281E5BB5572968882A9F5EA545FE46F110C9952354CBC46A30F8AD5791D825B3BD51957B1EA58ACD639E0A1E045A8D2029005544FE2D8FCD896A6AD1F6DC1624AD52FFD31E6FF1760E8350E4895DD49BCA9433B7FCD316BD34B3EB001B8950C482936BD8E657B45CB8521804EAC471647A765B479098B8193BE63AD2CCB9C826983BA90AB24F1225EBC450B4EC42936F86A7C9E3AFE8F02B9E2E28087DA021091704577E33F80C346A0B3090C4463763EEB872EDC2DF3AEFB9ECF2CE837C211C55150B23A68227F9858CC2DAD8BA0880ECAB84AEBC762E12380E005BF191B12CE6094ADCD7240F63572F63E0F162EDBA38F46044A91725AFF1D257B5F57F28F96DB44DC66441754CE02DC5D2262FE48FB3F5CB2A8B2EE133CFBF1E5C70FF911D018A00D5F2A1B0041EEB638A4AE55F103EC8A5869C3A737E9A40682445943853B800C775301358697C67550EAD7B7437D363E0A943492DD90F30F95B0B2C6B63EA1105CF2D22741F71E8CFBA1372BF939B7340B0F82264BDDFF4A7A432DA424C2168715D0F1FAB2414E8DBA806FDCDA6E53C255D789CDC6B6E04DEC84698F84D90C6B7D129A4CFB0C0B94DC48DD2C2AECCFE4D3B17E312D9499B01DED06BDD62AD598AC9AC8AA532674D408B6616067FD3AF3423FBA5A71B90546BCED7370E0144B3EA8442965DF12238D0BF0EA33DB569E905DE7D121687DDCD6FFA8BC77B480C4F2EA3CB6058E98DB146123D74E8F05EF786428F0A506357A5E49B8FEBB2685" + "FEABDB56BF7B4E5DC3966E89DB7C72447844D26198A0A062B82E3EC678845BC90F847D4E30B170728BBD1ACDC38C2E35327AD4192CAFED7CFA0C0730EEDA1BDE6332160AB1A5BE4D5DFF10861C9B6FBF5E2DA09363BFFB258E2FF4EFFECE729B443668334C8D62D30E37D828D47BABEEFC05FE60755F14CC6562971479CD843627E501C78A6BC55D8E16F3778B618825446DBD8205998146FEF29E12EAF0EA02AE64895DA1EF5CB98804FAF7A513BE0375" + "86616EC2771AC1C55B85A900DCEC382CA02A257A2C61D400F66C024FBE63A068" + "35616474832D3BAB031CD04BC85E46776F16B7C3166FED646066C3195CCDF135F23D583CCCCCA20119A1A428421312962ECD86A23C68040BF76149323BC550030BDC20F8E0226830835225EA2488206D44A7675373B8DA215225EF037DC91298A05D87927D8F2500BE4F71C1ACB3BE7665D41E50DCE9A989D2329BFD0A1563ACF0C102E4C89B0E608F14F5FBC6B65779F4E3B67E3BD11A8058270B5AB6075A5202D9A1A417691F020C99F1BA1A76741A8C3B417524C1D62E6046A47DAAE1385B8EB9DF70D08A4F4B73C502E5E9AC23553D353D5481B7CC481145A233F14CF5DA79D4131B143F38E86B3D4CF9AEAAC0C3852889D0D9728C8366E61CC0A9B6B0765A03908E2D94293C58B461E83BCF7473686A27B885B0F3FF9B88799B73476E4B6A1ACE17F9568EE7D86CA3D45ACE12974EF43C1BB680B31FBE9C41449359C9A6FC048ABF087C23FEFDBDC6BFFABCD698CE693F4CC4DA64511A90DB6C4246DFBF093A540DF96707CE128FF871B61F4C01028CA3459895C65740C62D0EFAF878065ADAD6D6FF545311D5C76716DF5D438F8EDA960C0DE457309E3814BCEC08AE6669AE099A24AEB757587E53928E5427A28656AC3EAEA93AC9C38FE5298723B8AF69F547708DE6D87368BAD585700DFA3C937A5AB4BC7D2E2821EF338076E34B7C0A9559BB64D165054C6C6A554DA4CB5D85E4B47F47DB094B65049E53299004CECF85AA9F2EE7DA6EDBBEFE28BEDAA1FCF63F66B174BD95808B875BEE6BA89331BAD8337B09035F8BE6B53A2E8B4105994D2BBC0DFF833ABBAFD0D1405975B898464977168010A8E395265C24F94B214372E21905BFEEF12763B6AD0288AB134C4F87366AE8EF85F90BA0A38E9EB27579BCF548FCABC3CB4883577D71FCD9514E135BD2188F148B418F17E13E6ACBDCCC672EF46AA0D732EFF6D3503D6184ABE1093D4705644F7FD1DF85AF7F4A934834276409C2D506209F4B366CAA5168A0B8370E8335D30673556A356965DEC6ACDDD4832DF7EDA404C3E5ADA9C17421BA911D505407221A5531D92A6BFB25709A3D1C4B8A746E77F51907120448FB8B9F71D58FEBD81CDED3A51D3C80D0817631F0B29577E8348BA611A5EE6B053E5B094887E1DF2E141984580E502199967FEA6610F3C8A2A689D3BCCFB2E0975F7200F4F7EA4BEA70601E6BD9FA47F4BA12CC946E4B2B3508FBC6EED8584D9C2E801C1F7BB14FA20D481E8A5D511F61DC27D403994EF23FE3DD8B561054DB53D18A3407129FE6F17F3468A97B9E8484946B988ED269299353AE962236230F1B1DF460F823BB41EEF87193036B519F72257908BB4835DD92F9BED4FB226FA35986A0BED4A616378CD9CCA7C9E50BE7461EFE38889A3BEEFD0C93FF6949FD2813E9BBB0466C8ECBEC808AEC6A8B87FC144B01D4F3F106D9F2A1DA977949277A0A3B7421934D554871EBFE48DE7EC7FB5F2915FE6891FE9A1B97DC0E4FB323C8FF8BA9142F6BF48C888CF1AD8A5C1BE1A2639B1A52F4000F712554E90BB0B0E73369A1E5D747C71DA32CEC244B6D0B554B7492692D4FBD0141A0198B2EFE64F7EED9957149D431626E4D3B3EA48D3D83D3070F58C0BAB7DACFE8FCC337EBC44D4D2A29A7A39671E0B9AF1CEAC5B9D1004AEAFD2BFBFCFD5FFED1A9A2DA5B55562ACC3574BA04D22C3B0303E58918E469A9CE039505468E6789D7A42F7B6D12780E01679FF00D05A71B8D99712299E0C82BD21E6019D91669EFABAE51091D52776686D3B76F964809CE1F0477589933274581EB3A4BA331E9834F722CB72AFB9264A7DE2733F2C2C5E5664EF31EC7587B92784020E22158E00D8D38209C74EDE8D40F3317C2ABC748CAC82B094BF1561AC0D54B7E1654A04F5C956F674F6555FC387320A2840F2B963EC33F0A57DFBB33F6A5F8C1FB760483B3D05C0EC2AAC2AAA1FF6FE10202337333463761DF07581E0E65AF0E1C00FC1612D4E41AC04FDEB52120321C77457A2CB6AE18068DC5261218865C5F78FD2C5DAD6832002F7B7535FDD19CBC8E2B0794737F4B3B79F625B25DF9CC831FEA38B1F72C43DE95597BC23BE974ADA3F714E6E77EC88027BEBCF15A0E8000224808D4F770DC8867BF0BEC045B68C22FCC4BD889FC27D88A22AB99C70C2B63870C5EB7B976C9882F368A91B9AAB617FFC2474DF61A8C9F99B29616CAFCC7678E99BAB978BA1B843109E2FAB9AB15050134207072D6E895128E6A030A25178D350642DE5AD1322EB3F2687B18070DC24DE3A803E5792782D4A91151CF332A566B6250FE975AFF9ECF8665D0C96C78016D13BF12B90A82D3D1EE45E449273D3CDA9564E451C0B0A63C715CDCD9A9028F0B9973015947C5B015E872C96C054E1ABBAFC65734D34E107AFF670145086652F86B78C27C5AB56F91B047F446D0965C406E4560D56E969BCEAA1E4C535E5F8A50B2FEFC83D43B2B5C082C469E2962B14351E89539479550B0BF1777F0BE3BF1B27C8299B4E3D9E52035E8B156C1E779547D513D2313C25367A0CAD45049BC98CCE4BFAE54D46C4B397FFD0D7E8E93575D458A8330AC69B204043F6A5F4D49FBFA77DF0EE8CBBA28D8BFE92117DB29DC91DBC96F91685031E4331AB39FE493DC7AF777F6C2380C278A77DFF0B35E47DD3A902FADEBB69821D58F4851557E0BEE576D8BD8C1CB21F0EB296C13209425A773A9B3E218A7ED99FBC602BEEB48B141B699FA9DB640BD0153232193C9C545D4601DB6F1B81D1ED5A3FE0C914E28EC4834BBCB3BC8BFA9B67E3556D591A6100C4CED5E1F3025DDCD4DFCDD9D6C40FC39AE851F8D64F98168A4E5F7ADCD13726AF8EDB24F93ED2D151AA080B4DE9234CCAE209CC3E4605C1600423FA61F872DA07EA21EAA1A8E572F271A2409F4A70FCFAB8B9BC28CE765953650997E600B781E91A15E7C2CFC584A66DE7B477F2DF8404349294F85221D3053E78C03CDC07AD1501E7029EB9366FC523A237F5058991F319DFAD59C85F2C8678E4A36BC58FF85BBA569FE29EB717D51467983D7337C27116656E9ACC9A82B9871FC553472F48705D4BF438318928311AEA9449896BB09381E2A85A3C4863747551EE7B7CE978043E2595A1694F2ECF6F3665871CDE2517F493D2CFC6AD82AF0F160E1A3B94DE751C5AD7FFD03FB70B219BF303F2F4ECDCD7C7D5040257175527829CA29AD28ABC434B58ED461D97041E05D1D359C9048A3E48BFF328F132AD9FEF25D19FF2334BE5E9EDDA27758A3D6B352785A8752216F6DC15FEF79BC1329EE03673F3282A0CEC3C1D42A60B236288E9D11E55B9E553924164E1EC536F990D97FD8F6B010A2201900A7A4DC73D6792B48C8913A4954DFDD437A46D44049D8980B7450338832C79CF8D5488ED68D17B7E678AF26464E5429781CBFAFE24EA60B77C1BD8213AD21569C04870F60D7B44C10AF7B483C6BCA620D933A42FF94C87D241BA5FC8E8F817754245551D6EC485CD0B2A60584EE7D76DD0414CC2CBD51F4B11FE84D74604B2870B6E5A357EC9ECEADEF1150356378E73F9A841C88F016AD23FB064B9024B24749B821827F38FA55E425999A2F91A930D4B92240F375D295650933A5C11DD9AD8C7AB8F27EAC1831FF79DAC38F4F3F41F6721EEF780A8A4DC750CE160494D0F9C5297CF4718010F8CC36F423B3B7362870E980BBFDD5C5B51C4E59BC08DCA7BD85FE22E792399AA049B03618703E807DB28CD0D6C4602B35E5C864345CAFF421BD37CCACF89945B1D92FCDB763CFEA5A16AB8CA92FCE5D5A5226A7951C8DB0AC6F3F1F02C65A7446487500636ABA4AED155A42A50E36C71D3F6A00AEB2198F27AE9D25B821B6EA3F0C096859955A2F1BDCDE091F19B28023C654420997B058972FF5418380329211E26580FB1D0431F11C6059CC696453E05405B6CC1A5C7894C2F0992388D8FC46FE3BC9A6071564ABE820E366F2667E0F913C287A5C339FF07865489235D43457847B075C872945427729A431CE362BBB8BC66FF4763150E1E64B806CA0F12A9B8F085F61346ACB69ED7A68B8402F8EC18273950504293D69FE1165383A18AE91F9CA3F59E90D888651512C915C70F9EE21983C8A956413EA58C3E409E6E23CFE9379E2C1D5D80F74E892947E76B55D74B8716FB8C77B5A4FC8F481D8F2B7BE8A301FEF6470B5957E74C219BC850F5BE6F7EEF41C8DFC6B304F4DC1C59C17F858E5D2E3BFEFBB9FA24C06FC0E41969D7E1894FE626AC055629952D9BEF2D1BEFB888155A5BF387E741323F1D1ECC54B660C379BEEEB18BC69A33D79867C01A963D2DDE2B46C068A9CF788C2AB988BAF2661BD371AA49CE332145E1DE859798EA8200C13577F0A952F60652FE7B993BADF1A4D9C980BB21B96F0294B4263478F9951C96EBFE6E28023C78CA21C6E16C9D0FF4B045B60A9DCB9FFDE4D3620FEBEF5CF4CF0290FA595E8882BDB79574EFA6442451D206084603830514BDA2A859F66752B15BC0E5B1A9E26F1CA8D350B25941D6AA291CA1EE0C575269AE52ECFE4D7EA52C6159AC4C7EA63D27E8EDA208CC7D64224AC78C4E2AAAC1C114A20E6BA98A3F86A47B07C9CF5DD99FF97FC1DCC2E2D3C7CC4A79775C0944E036852CA524A720773CD527099F989F08E61C29B6C4E09D90B84825D3CCD329D22B1E5EC42DE800D5760851D5B7EE3CE84684386F95D67C7237E9AC1DFABF44BFC5EB91340362E0825116715838E46123D3D4273582146E423437BDCFB340A444441B0732C1FF6698396AD4EAC39803165DF4E174CE979A61AF41F30D3EBE022902350F521EDB1C0083A4D1F25585EF6D6EF296AA39AD31B8DD43AF711846890BD7EB0DBC6C5B5BEA5E612DC1197030AE1D7C0F5A8AF2E4EAF298B59145D58B5BBEB1E98C197E7E02270D7C88C2AC6BC6F40D766E62226B7E66CED9E58F47AE2FFA9083F04BD810754DCF1A628EB0C7FCC404BDCFFCD48094D1ACB6D42DFD74CBC4FF36BB5100C6D858119F705B0A10BFA98AEE43B12CF27C36B600B309B0A8B0177A2551A7B9A7B0FCFEE8CF3E8C9339F7A37BB517E3A63F83C5ABE83DF98E1D8BADEE09633467CC5915A8DC4C9E00A3FBD915A5487B2F7949CDAD6BD84C31A32FF3DB9E4DD867B2BB417DAE07F23FC428A7D143C62ECE97C7CC5762308D8E5C16FA8BF853D47CEAE94D2FAFD90AB5D16114D16E1C74778AD26CCF493314214DBEA1DC3C17D294BC87FF10E0C756E4330CF400FCB601FB3952E94CD9D06264C00DE6C6E2BE44E6F93D5E39EC56B0925001783ADF1CBB104223D3C8950CDD5638A834E3B08875EE85C3D579111F34D4B7B8172A7F164E0C44A203FE765B89C5A008E504F25F7DAED8CDEC86E2DA93525C17F6D500B4DFB6F2D2CA677BD071E2C2F5629C91D9BE5EDA44D313E64510ADFEFC72DAED7161344C32D0381E416E87F4C8E56569D0AFC7C4FC7F9C3B3D50BFA698ADD10089ED1F91A8A0CBA4BF6785E7E24FCDAE61C7E7C28AF496E4D74FF0E55B01C13A2E9A8A73A7F2AA70B1065B0B2E439EAC2CEFCE3F6B8A8F0535523D43A36F791450BE8A040DAB814A329CE5230815AA1068554AFBFA9A45B05B634801DADDF2A84B79D72C322EE429F2DF6495C86083E6E733AF3BDA3B846970AB5AD8E704F855879F90BD1D41614E37415AAB711F9D7A285F570BE24148D22ECB8A15E461183FA5540BBE4ED3405F477E6EE6DA37D2CC0FD6F3BDEBB4559BFBBE97B77AA3B66507EAEF7AE18C764B120403ED7B1C80F992A012AC69C0F193FEA90A6A629AF7945C1728820091012B2D30DDBF9E0CCDD08CD9045B14905BE7D3F1DC8127EB8EA82C9174659BF3A9AFB0608C4D5B1F77547B62A5EE576FF967A32D0363E84B722DD8F1D2E436A1CD7B408C0BF16A6D185070CD2A9DDD6011B88B02E7E0C5EC9D5243BED5A7A41F54B6CD24034AE47DAF2DA58898BB2626408509D04B2303915F29F8990A4AB18FB6E1A500FDB71B2375167DCA15A15E48904CD8B2826C6E92FBAD5F61C2467CB22FE76F165BDED3C41A048C81FEEBAAF14FBBF77FB118AE1EF1C5439545804A3C3D7EA22615205328EEB02E382999873E1DF83F30F1698B6C6534DD079FDD25DF617984C59287B4E73C15A0D3B0DE362EC2E67EC12CCFEA891DED0C99FDA7A3C90E096838E7507F6BC86E7BC8BBA1A782ED685D993393F37F5F7B176B478DA4469BBB3F6E3BE87B94E2E3E9E564B7ED33E0C1ABC690317776FD55EE9106B97BA4C4CFE79199DDF2327E7F2A94DB016F100409E224695471D0E048584A4B0270CA66612B7F2480B1987B3674F280C445F027E387558A3E26C684CA8FBE28CAFA72E1D558202D9213FEF1D8292BBB9D9B69105EB3987D9159BD15E7C6321C1E1D7E066F04025F0B9A819E20CB3C34B43581E603753BA1897AD8223D94D7DF1C451F7097D486A0C2BE10C8CAF3CDC063F5E375C0D748789C35190B2C9CC0D5275B6C6E609447888B3B9BFC1E2E31C51C3D609466ACB181A30348BC4D3DBED062335718AACBC000000000000000000000000000000000000000000000000000004090F191D212A31" + ] + +-- | ACVP vectors for the external-mu interface: a message +-- representative in, a signature out. +data ExtMuVector = ExtMuVector + { xmSet :: String + , xmId :: Int + , xmDeterministic :: Bool + , xmSk :: String + , xmMu :: String + , xmRnd :: String + , xmSignature :: String + } + +extMuVectors :: [ExtMuVector] +extMuVectors = + [ ExtMuVector + "ML-DSA-44" + 91 + True + "6BEB6817BDEF24413265898BAEAB86C94BF2FA5533CCFC0C0D42E92F34B5E7B5AAA3E9C5F83E71B3A69C77FFD8802D23CA07A53EEDC12560CEE54F317C9247E499FF7DC3FA2DE3963DE39F405088D8DB8C977BFFE9E46E5F7A1FB02A000452A5EFB967C1258A339E5FB15ADB410A5CC77FFA26F3491F6DD0F54D46BF9B3946E208A86103C740CA146C114812C40490D9844804A048522640A4122A242085C3106A23068DD9A205502891A0961061128D991680C2A86803B28004C14C21B64964B429D4968589B200C03622048240A0068E231644E1240CD434711908680A4304802830E43424E2A2899C042103340898006582266D1A396E00862D14A42012A52D98B8440A944011B52149368A09034D44066C18B8701A454901B5885A324414038218B08123462E22A82140A26493448159C60051464A1A124520461061B2019A08498B246DD9A86920B5694318611246454C282A08A0254A88494B8684203630109708E3840C521241021461132571E048858C02884412890C12920CB24D12280D1204421CB70564165208178251942823B83063206549028ECB1842D4080522C1601B252A040609C4227049162A0A062210B36C1C30211017800895108A9005402828840064810671521431C0466603880049A00008C32DC09489DA024A19288A83484A13409014080D0B1629124092D20862404289DBC86DD298898838005BB80D21126D440869194188941030E2202589444A8BA06004070409142D22166C24062A0C430088B68D43808C094012544665111784E244611A086D4A140981069024B80DDBB0719B848812A38008362E023488D0846003114111C14981B660C81672C9064D1A136D23C708D1340042064982C009D84428D0B268E2222E589068A2320000362851262D1B262D9C948DC4C491C9A2841137900C40259C2850218751C226468302820A0452DA3405901420D2142E8388219B302290184851166212245292A20800372288484DC2446619080D0A05821C34269096910A06044B1211203009C9866908852898204D48B84448008402B60101418E1C182190B00589940913224E1025328CB644D840241B474153C22C613601C08880A348288210801B09648038619930318C220C10C41014A54C60A8081C338554205111A289D4204DA348441845410A118620C16C60340641348960902910B68154320010105261284820B50859006D23C7916486458942856134890A972958404E92266D58C440AA56B190495FD1285669BF1F0F6BEBA53A52AFF58C90ED5DC9396F108BD5A8BF6F4DA9D8CA99F0CF13F1DABE503DF3612AEEFA0F5F8521C9B98536FDC1A135960FD342AF86B8AF34A7496E723FBA8DD8E4B5E9668EFC7DAEC509C42908036346C46029C74C3C401243D303117DEFCD85FA2D33A8E8F09BB402F1C33DE1BBC4C604A3F289A8210F7104E2C6E1FBC1EEF2ED3DCA16DF6264FEAB235E3A75FD800C4FD028239456BFB8A141F90B573897FCC6A2BE6E8FE61509911733980861917C915BDD8D42BB8C36BEBF80EB82920539DFE783094691767AF801FA93DDD3BBD6041951075F504D3EDBC580F506506469FD6C2F75992C2DD1C988B50A3096DF7702AB254340D6B06EC8D06F8D67FB3E78083B1E31098A5E0F02F8A7C0C110A99B8D0D619BC1E7A5C7998C7801DB708499244AF761FABAC5691035F97C4115DC6940F6F51BEA5EB878E296B166E1DE0AA5D7A3C86B6DE58EE4F21048A46436BE55A57C483016E4AF2FEF7EF51ECAF42FC40961B63070961CEF97CD5FC63372EFF708D9F9398A06CB279A5F5C1EE3D30CA952B04B2503C31AA4784D279E249C5B7DC2235ADF5E0ED0B561EC1D585065B73F1299A082D535BF0613808870D5CF8AF39C14DE9E385FDCB95A5661ADC0BC7CBAA4FF3589B3412A1D324FF3563CA895BC17A6CB2DAF83B704BDBBD72EF48D34400B2698CE3FE0EBC525D542CFD2C6D0B955DBEEFE69A0407D2DD475400947179E583BCD11B521BA9D2578586847C7C41EB50FE0018CD200515CD7165E711DA5B4A9DFFAF6C27E2FF8F8C18B9CEAD63DE207FAB139E8DD6E4F759020E8B35C0E2CBF1287F67BADC2312A3A9238F87324ED45216ECA8D2504B7A359B2A9277474BDF67D0DA9E5EFFE4C0D9F891E57CD78A946FFE6C0F4CA6C9BDC4949D7F0957072C4764DD914FFB3D24A03026A9DC3C619294456AF6429C33256417932D9C15C47F082409760262F896BDA28E1E3CF9BD7407289485AC0F8832177E79098F602E98582AB2463546A671CB329C6DB14F51A6B51FF74A76DD0A91314BA2B691FD314668BB8B1EE2C8E2BC7B9E8C97E8D6105EE3DB67AAA7ECD9FADF7BF2AB94EBD4DAAB42749EA118634C729D0926A413EECE91C4689563F8825D5C61F50CBB3177A20C293E5E667F346F22C48460A650494C17839EA5D388D212D9E1A7DDDA0B975EA546977BA2175339E7F994DE50AF65DF644DC48DBC9641CE2816AD53D0BC9757BD8F209DE8A65726C489532DAF43E4F51F0CC66B45A1F7DF7055CB524F968B8A3FAD56140204D2F8C123180E7004DC65EB587320C72CB89CC3A7FA4F60B907649F2E6A2BBAC02BE39906B669E18FAC21A9470276813A391C385958089D8FDEFA383DF21C8BD054C48F418391343DFC9424E749C55FE0D9CBD7E0917F01ADFFD622ADE02D7D811BD7F64925A67D7288FC8791422F0D44D4DE8E740B7722DA907A772849BE8C3D624FA2A4C1786927754C1796E1E305F781CBCF4F78C06326CEDB433BE9F9CF7A05D0D387ED62B1DD6F7FD90A0BB9451D3C7F7D5CF5DC741157D7DFC572886B81954CC7217F2EB1BCEF02CDC034DFEC787585806E78C81F8CF63DEE8A872ADA2247CDC57AE9E210D54B4A611683C2806D572CCE09DAB6395FE6DC3051078CE9EA3058B4E9657E6C28144C4C54CC35AD4D99CA224E20B840BC1F8E9F983C1C3CCD93E7CBB3D8D47C4F6EB97F9BA0E3CCF259ABA488684D9B60F0A796C80F7F4B9DC12029C12782F9EA0F1E78CC3DCD0E0F6267CAE3A10CD2C57DB71044926A3BDAACA6D9835A941FFAFF43107D7C05EC42F09F30EAD88F863689EAAEDF4DF9107F21BC39993988B3A907F5AD5D89535CFC2011560A15BCD1BD5175C09E972528C778BC6325B87D79434BBEFBC813C0565A959C687E0718577A91B2F031B9A278D80AF8F749B9ECB855802202AAA6B6586087D6B2C939C81840180F5070739990C2FFC913F8CDCB55A2E965BB186DAC8D1738AFF04046EA10849FAAA627CEF16798CFCEC8377A937CFDBB1804F86FA1815EE4D32B429AE016224ED67D310F7006A3109AAD22798828CFFBC0BB65C2AE5EB0A7FEDC7B9E2F6D810E5538BC20AEF00C531CE5468BB49D97976540310462560AAB008A50C7BD12E73F0813A3089C1E0C24C54DE1DFE81637E4A45511002FA4EE12629EDC887C4FC401EEEE8AB22987B79C7FE07BB938D9707F971D22F5817F5B65CFCEB62A0033CAD9B83F56AE7E4F0F09AE8D091A380CD4A13533ED9498A4A903501CBC0ECD4C787D06ED61917C6E2119A79EAD8C9C21E6C25E18C8ED2A7BC82E039BCD9DA09D8DCEA9D47E4C610CCB31FC92ABDF99BA06237" + "3E240AEF582154F8AB1879A5B6E0DC69A5A214DA86BA5585B4268C68B5449A81E20B8A8CBAD23E37EE42C9E51A4892D76859143FA70B51C9B0C4C19758433A74" + "" + "3327AA270451D3E5AAA3E1E27A8414138D9270C1B44710F249693B205F890350C970BC577E0D2E9FFE641147A7E877CA86D6B31C64A8D80B9CB55A3FB1EE6C09E0E78F7E01C3A576D92FE1C26D745BB194EFBB375840CC916595DFF2BC0219642439E9A85D64EDE176AE94E51CF6F9B0506E2595C934DDF7432103CA8DC93DD3E2087D9FDD0746C8E14255194ABF853D9BFC7A6D421C726D2ADBBC2AF70D2CEC2A59A12A59CBA49EA33E015EF457FCEE9D3E4A2759816267283E0C2E9850C36E1192EBC92252F11B0AE6C029F5F1F8E9844C4A1D5252C6969B42F231461992391F97067F2D98CCAF750A80E582D5992E684E7D415C486C28CE519A3506FB2CBA2467EF083DE349D5CFD8DBB84BDEF507F0AFD6E9682A465D4116531552E12127CBD721CFE10C18C20904085AD05D8BABE2ED9C215B1F83FC425225843D767DFFACBCA2E9E58E388B689E208D0723E6124DD8CE9E31D1CCBEE25987EDAF7BE55E1E0D8436963C98C99A11D32FEF8440C44CDB83DEA2481527DFA349C2B9E2F3527D0887F1F718B758A7AC3163007ABBCB77BF6C1CCFC23DC155DC1B0BA876F34776D35A2ECC44F4C7D996C4807965F2E3584FD4CDD8BD50F134A78AE672E4D7CC10ABB16892F68D83486FABFCCED9316CA17EF5109559A97D84B6B0835E36E0635DC29E5159F872B3B15227E28B7E2971371444D1918301A7A0BC6D682DD3567D22B055D5B339A3C09C6272448D52E4E9EB59007FACF2FEC44559966599D1EA6772A9B08134EDB7EC533AD84709A4811465D46EB26C8A5F7AAF648AD43ABCA302BB85274984F8B6FB8EC663FA0338AD8F2584A0BB0070ECF58E3D8AAC634BB1B8ED7CDAA8EFBD461F49513B1538FEBE6CD55DFCA881366C3F146836BD4EACA6FC31D56814C058731DDEB45DF78F86615FE2A10683361F6D76BABB9D0653CD51C7D81DA0BF05E937C35F75B87C9B684840159A7EFA578F770E2F777FA7C190F9DC54BCF65690327142612F7F9D1AABDC846B72419752A9DBA8D3271DE3DA8D35B065091E9B601B00285FA7162431540617000B949509E24E6CA4913B316F157A5EEA97AFD9E82489B26DD8213E0F1E62D5A96BBC94C8ED9B4E1DCD57AF129BE73FAACF704ACBF5E1277D23760BE2DD4718A744F5B450410165E61F4793D9CDF639985F1DE12C981BD2A4C7167E681C7AEC4D016B6839488F9BF8A77F9A9795894E48DB3D7456F3DC7F620E8A2266720D1C3C325377D9528EC4C82BD54EF806BFD7B001E87E6FCAB79011FD9827ED7C9D954D192FC7748C2DFC2D3190110201AC8C84E1B72287793CF57941CF555BEEF8F6C61D08F0EA3094949384277BA95E2CC47CC627E071ABADE48077423610FA37BE30E264E40C9F58A59F0944FEE0D15983A2B31F72F591ED3B04B1FD5B82EBC6B730A358638FA97E67C4D1A79DE19C73B9A561A70DD221E94807A840A2138FE88F5E083FACC31E6A08C2729810A22C6653324094D2EA278D5D3EE266EF10B4CD79C00791CF63FD16CAD0EAD7EC9BE7010A80F8FADABBD98740CD52138DED90443EA770ECF2B0187DD33DACF6F4F034D364042B80A58156A976B94DA79436B58C6726A7A55AD58D9C85234E79CDDE7058E318AF3084DFC203E23AC33E4691AE923A0CB17C61B2432DED3F61D74D7940B8A1AD7BAE8DC524EFE024A0EB178508EE4B55B864F78BCB5E6FEE4B04BD6AD94D8C366F7BEAF8C161F286750EEAC00FD79C041BE0E424680FB29439BB8A0027AE1EEA94D11E32C07EF01745A335DA969FD77E3E8B2471441E150C40E66979DBE238801AC27F623EE12D9854662E310A613C9FB0F10D4821F98B6AC5C0F2CC61202992FC10B0460C82A943AD7D87832FF11D4FFF800CC0E7F91659C53E3ED9740C7CC24BE4684EFAB1E929A78F2B73B48ECA5249B56B320F5B07D398112985D60E04C6A83F2FB9F042700042AA3252108C32770D149A7E905E18DAE0073B3E2694489B5FD9097F3A1D0D77B4CF9B7763EAFB6BC8F01C13624F4AE7EB22A2E22ECDDE934F6105547254792B95EA91458562AB39AD8FCDE3247C91DFE3C5037FABB273468A9D7C9082066A7519B2141D021AFA8CF3EA64439F019FE18184179F04AD744FB9A1E65F39EC1DF77C275497EE07F666F3200D4F07F0E9ACAAA721B543D965D162BCC82EAE574B272443297C5DBBE68565D22CEDAF11CD07E2129A5F0B6580A1BACE3A2C69946010683C046A8779A0F8AB08E813CD14481B0826C8463AB06B30F23CD6A289FF20C68C9A0A0C0B73F6EBD4BB31EB6FD11FE4650BD5769FAB9A2CF71233DE64C7FBA84B10C2B172F674B3ED2D85BED4A7701B785593F2729C3D4E83C46B8211D11CFCF33A9198AC9086FDA7C97BA07C69072427815076A39A825BD53572DAE9ADDF9D4DE6A67B51B8C785BB2402021D369AC58D42BC09C4D59C3CC1ACF41966D2B19165723D9836F50FEFD4CB42A8FA3156B96423B76FA40A3120840D874A0B7B991720AC95F15D1B856D8F4275C7C136257F49057B78CAE101C97360D3A90510598F99021F44243F62151B13FAB61BA3947B9F605DD7C850FE9860F6B9A10843DEEA99ECE225C40A5DA1CDD026612A3088566F798E59082221924F823CAEDBEAEDC3B12BA29ED400387AE33F201DDAE901B21E3343870AD91C7C9CCEADF8DEC329EFB9F2FF8EC1F8E103A508C8E1E938EEADD307455153F892C612B4597D569AE72250EEE17DF0B97353C05F766278E91951A7314D05C52BB32A4D35E752BD799F11A84E5C1EB4CB9EFC83CABA48B74BBEF5D53B796E93B0178E72A46E3A7010A900D4AD3A20724F84F3CA8CCD5720EDC4A3E65523AE5F167451EDBB5623738FFB626D6287A249C763F6498F5180F79F1A9137D72CB52497581217D0FD0BBB05183F63EE788DD01CAFCCDA55BDADDD882EA716915A888C6E040F4A29DF30ED6424330FCCE93E3C4B204C1286F357AD0501662AE6EF6A56F795E65505C7ADB776714E26549B41EFE8520846E46FDE5B1A06DEA4774477213D8DBB410B6A2DC9063F04D48E7AEA33E03248178F025CA5480A91279FDBD57639D28FF8117455E8EBBA89A0E983ADB00145477F29EACAE2F778FAA5BB2DB971812C7272931097011B0943EA530B5D3FCE1EC77B30A2C1A31DE96239594E63519CA451BA614388E65E590385A0EFB6404EA6B8905852812F4D6AC7C45F15179449FBAAD3564ADB018B777317A1E2E940EA29CE40615BFBE3C319F3CAB0597C5DF6BB11E729379B2239DF06856CDC3FD1DCA85D28A8CCCA86D0FE0B9970B22D23E7141E27282E3A515770717F939699A0A1ADBCEBFE02101D2E303A4345465887A0A9BBBDC2D5DEE4F412313D41798590EB152C373A3E44525A8F9BBDCBCEEBF000000000000000000000000000000000000013272F3E" + , ExtMuVector + "ML-DSA-44" + 271 + False + "2A85D5D03CAC6C8977EAF1C19847F8BA8D53E5D1C4A9CA2D02C2C4A7E9838BBAE95FC78DB5A422E1F747EE3CC59290B084383C52BC03906295CDCFD8D5FAD5507813FAE6B7DD5F3289528AF354ABB98056A8C233F49EC8653D02DED6CBB7B43DCED8921EA9F2A03A0EC1B1D409CA80A0CF4825F804CED52E3445A56E3235ADBB0C276D03450E42842998126D08983053484CD4906CC1160104196909804854122962362E88868DC284099B046A1BC65002B91043A06D4824480013265340289B1090CA34120AA441D1A690C8302E01122D21860004B4900B050D4B32841AB5258C344A8202429C088519936542C68D1B19444A08260C2660802806D1000E08236843984403341202290494368E94368C44A640233792DB1871E4B27119238201B68408986881984863280209266C4C201193C41093C2659C944801486658862C0CB42D0180441B498699346C13994C00454C9B428ECC186E00849123044DE2042692A020DB002D202140028929C80082C0340448246C5C026A04164A22116A84846151166641328811A165DA006964120CE0406C13312E8B048941A269003785042189D49660C02662CA206103A22542861113C469D2B089DBA8501AB48DCCA6885C12420437452101904A048D19A70440982913414951A610C9024491288D63328604448E9A047149B8648A127298840519B66DD01252E2822CC8464A23913089262ECBA08951042003490D21458A63202400410561C088A324908B1646D48668032492D000290428269A088C8CB409C80808A0068193C00D02C6891401520BC849C4B84124B851CA420958A851042841D144229908500B214214300C08989118076DC9080ED4A0711CC649A4C62D0B336603262DE4340E1187840800621A427040B26194169144B82C010142598640D330240C487019491121966CC2002884862C843240121548D236006300861C3888142922944820D00221E0320604108209B08C031541902020E4B845913021C0228A5AA22589346A1C14105826810BA61148263222B86858483162A6851196290131219CA63001188D90007209B44D13202504C76C89B461014089134125C8C40188C800021465DBC0604B82016084008C48298240728B88718AC48500254424146C61C41109388EC148851B1752D4144A8022222392491B36420492711AC3410C0129A11270DA0411E330925A2290C0042D1A928512459060242D8824061041405BC42484C4906122885BB828431621475D2170672CEE466D30FF84A2A6D043AA46635A5516F1C0AC3FFF2AADFF5C05A1A164CB409A1AD5F417E49BB01C94F5168D8136EDB7CBE1C61E5C19ED45CB0DC45B1D0C2C98A9C05033C374269453BFC458847BC9F3EE88BCBA992D46F5ABB7F05A8A3E924424899C3808C05007D64FEA4D8DFB3CCA275E7CD3830E225F3F4D8F162CA01618840AAEF7346204775E6EBED692939618DA470A7B9FF793E08B04589F8D6310AC73A30BAF9B786D798151462422010861E0AF7E2219EBC08BE234FB05F80AE6A04EDFE89BD59469DDCCC21E2E4CD3F72951C17A6F178A33368DE11BF1BDB1EC6ACBD73BDD8E058B8E0BE35FF72502858EA819BFB6541E6550B5B305C73B90B83C83EA143E9E871775E449A39929E2CCCBC71F867D4D0022606D778F32C437A649D9CDEC131F2A8B06FDBCDC8E7326A5FF973AA60E4E6BEAB5033F37481EF4ADEF1A86AA3F8519B541561D7473B5AAB33224E9A7F1685843A80718C5A6E6D810FD695F07F36EB725FCA7AAD654668089D0A77DB39F841F8D24474E17B28718EE1FBB1C6DDDE71BD8E6812463CF91542EA42B70C0CE545AB2B9D693CDF1726278F5DF850085802F16F7A6DC0CC694AD07BDDC90BFA4B1E429161D667896B51014439643D689CFB93F7201CF676B792CB18D023CDA85A7CD5F6DC4497E834D4B5D416A6D55E56B844D33732FEDC6EFBE9F92CA0C73D6BE804128601413DDD55EA965F9828EF7AC40A1F94B96B0620C57ADDED7FCBC0A2EE199B69B0C6F2E458EEE4912CBFC9346051596D67F4A7667C879C990EB205E087B3EC30152B196959D3AAACB7AF8F41A03786C3A5EE7DC0605B232B84329CB85C2563FEDEFDF894D78267A7CEDE96EA9FC19EDDB6A3336B9F355EADF9F37521DA91D214FC212AFA67DEE0FCF4B16BE5847981C1A72531F4C5122841F877E8B182C9EBDE70BD60AE59F977552686DAD425C9534C8B8F358A5AEDFE3447BDEA63ED18705C73DF335A1AB5F0B2F0CEB8DA38F7CD93F346AF92400665F7D0553FB322F4A5E35521D69FE425E434546301943621000972B3E4FCFFF86C2175AC509D123239CAA9FEE418A8E375F2BB92DCEEC5C8055E33820FFBF4D3F5E67260E4C6EB761DBE22A97255B66F6C31DE9E77D8B0671CF3573B1F644B7ADC8A4387EE2564D634A7D748B25EEA78D1C60D606B47450BB0478323D471003DCC5B12EFFD72B30A5EB21F600DBBE672D377EC64C942DC5E9B4160967560F7C0DD5931BBF876AB1E596C97FDE14F2E1D86F8FFB54FC65CB1AFA5C92344F1CACAB1A7662728E813546188A3E0EFAC48E60CFC55418481F88221A371FD66B41CBD2B901153136A007C7D55DD25A73AC5B5393027117CDF37D380DEA394754E9A4C10FBF43466FB892D3C119E9D486BE7453E963FDDC069790A6FE5230D91E28A9DD420F9F8B6EB112E15FFFFEA67D8C3DD133017D08A48C24D426673D8FB445000C8A2E9A015AAB250F1D7AAA6C7B34E7871FBD01416238C417B097F7BD7C9359A5544CD68537C231F3C49EE996FE0E45A76E1705B041805415D3F73919901BF7055053F91F5B446C49254F3AC5616FD695E661EC73C70916EFA75E1C23EB94C5DB4692CBB2BE32244A79204E935D526DFC13652F3B7F7B01F0B6E8458AAC49927502E8388E5EE5369FA238A5CCEEEE70B6E6D446FCFE6016620DAC08A69F97EF8784AFFF9F28F0F68024F366F38A395B0639A959164D3CFABC1F720B8A259BF98A41E8E548BA7F5892619304E7C78643251176226AC8B598A248CFBE8BAD6AA9E6606C52C2919860323D2A7CDE1B9673801575F6BEE95A1E05952AA9D4664B74BB8AB9A1B11AA71B19442E9997CE05B2B840C26FB85116597F47BD03DD1DFA08560D37CCD6B5DEE4E351C683B0D1C42BE627DD7027A5922185AECACA3131FA98BE62A2636485A66E45FC09A04AEC79F88BFB8CAF25F564FC47893EAAB3BFA8F5BDF24CE293484BBDC8822588458A26FCC3594B539CF6D547016F564D3EBAA0FCEA8CA431D438C5E3D99B66B4148C9A8A194FBDB2D6521F9075C658ABC50564F0039D3C779CB89FE7C197D1CA07E74DC3741D02C5DD8F0A7618A9EF8B2385192720979AC0023D75322F7B899B6C7302C8802822ECBB488EAB2D8AB0987B7666484DAC4944E980644024E03C1736622099A0F1785C5DF972C3E97390DDE0480A17BE2501115DF0A656ED9FCBEBDBFFD88E9BC5E1E9D4E369A4A67C4D9399FA011ACCB8D90B07B80E3262F9E06D3A9C3CB1BB09A8C11426548115CF5ABE4191C2D28A4566690BEF7FD8417BB1F6510C7F5B12ABD396E40336E7CBD0A96FEACB960A38BABCD0B2FBFA7810CF5743" + "1D24714C69DBF0C970D540CB572020257499D927CF3B7C99DBF0EBE667FAAC0FA3ABCB36B2AF351C8D013FB4E708B101667847EE18782CE95A23FB9EF77DCAAF" + "288F870FF69E89D6CED754AB158C3A9B42653103384E2CC0167F104C4209D580" + "5C04A9FD9BD3757969F5099D5627D3B0A1F2CA39DD4ED72EE355E568EE8CEA68B91EF0AC2E0003C8B84F03FA40E5C2D74287D7CEDB6C2E14787C6E1E6006915753F9F3795533053E50CD771B689CAA446F96C3A53265568320921690848AD3EC7209B2B05545AD470E49B8C226ABE1E8DFE4D5D55DDEB72BD2871580C9802145E13358AE3E5A4DF855E26C5FEE86C41E032325138BCCC1E58911988F5C4AE8149FF0C1EC880567C475A445C5802763E0D770877F4195475D3A34FD39B95735BA6F71B2D4DA59999EBDFA0DF2D989C8AFE7404D0C82DEC7643C6E0FAECB0D7FB5D44CFBA0DA77BEA14279623424CB9B3DD41823B02383E0C7394169B562C012B28C5382EBE4EAD06A1DCA7CF5E2EB816E33129736CE65255EB4EE02028835E4C8C3F626D3FB1C97F05A7E65C71471188F59C4684F768A9E6C7FB31DCD7726F301E4695D06D52D5D91C80471150117AB5E4688EA333D44D3D440E60D580830812E69716B5AD30CA8D2F0C7A8502B7068008917039B930767AA9A4A54B59E24B6CC89854EF9F430A3EA6EC8FFD9D3A14171011AA8CA53C77293EC81EDE75DE60FB193B427B2DC67F095496391E31CC0D75D53818DF88A5A10EDFF994ABA36F4D6922EF185C543FBEDE5FBB445BF84DD2D99D6CC8C845F390872EE5A6EC08359FF38D41E0017A8FF20C720BFA1697435C5ED16A852C77150DDD9333DD5D99C53ED91D1E3DFD4E1F5F1977678C83BD0C1741109500C71FC29B97ADE88DD1A2E95C15CBB10748504F1D54ABDE13FDFB39EDDF4CFA9ECE08FDB9A0D54C2FCEE36D506768414D393AD6A4106916B213DCF71A028DA5FD43B6F115ACC6F46D406F9AC36C305617E76E22225B6E79A1B1EC82005F492D08692A127EB77CF798DBDBC93068E70E128BAAAC1129BD69736B88D55361FBC4DAAC8A8192B59199F802F8B80F2C2875E88777A823682255730C0E0D3260A1F6779B17C806EB17594A19FC681C8CEEC7E71EB3A4FB3D276A988FBE0563BC0E10D34AE24FA0C17CBE6B3A2833E02E44E56BE07EC592E8BB24F3843D8176320391E13F1BE33F810CCC6FA486D3340F98940B89E1DBB3BB8C3EB360D450ACA8C5D2E097E86A3820E8A9722E9AA65CCE411E879BE9FB25006AE83CFABB4B9317D99D76937F15C45AC2143BB3797F14678D60E4DF904852A06FD9B1979504DB89FDEBCC9484D1723814854856B2599DE24D9478FE6E43E2C023F982709A8850059BDF05F0E716FD368EDF0A85151F932E4BED7D4E0687A852DFF40B116F0164C03593B09AE2F1F069FABE592F2DC41B19F579B20CDB22527147A6097206C7C76E1DEDD595B7E1F0F94CFDB4F55A5654E4D544946843BC5EEE859A4F2ECA0C765D0992C6A8D7CBD7B63556D7DD7D18CC067EEE0D50D1F7ACAB5E8588E1EFD785DB02F8473D9712C86A4768B8472DD31A8A0F5EF15E63C8222BBD79570FFF5941DF891548C397404A60C6CCAC68A9ECFAA98BFB2D5C84FA806F76BAAE8918A64C9B0C042294EBC7AB28F5A395AE7D359DB23FAFDAD37E0DCA95D609636B2F3BC1A05643251EE389E40403FC75E5657515C1741DA629F68307BC1AD220E69EF62AA8114CDD633CB33479AF35A671015ED06496271578D6D67BFEAEC7B3850E3472DE32D086E2FBA936F7CFBD24603C2B4734CA228C9B60F46BDFF9A58F8404D66A12D2A69D24924D93AAC8E802ED5DA9FE14DBFE1480D72198DA4DDE65AD9559259016FDE9C3B16A11EDD8CB0758BDF2F969E3445A651796F259D0A66A4D732363F381A691F1E2444ACAC20AB0E0B96C4618FEE33B7117B43CFBAC0247072105F34E76CA3C34EC4614F08B18437EE01835295A627BDC1F9650CB3AB394D47928AF3B8E5AA9E574D1F27A47FD1BDEBD49FBC5DA8EE6B9C897DD23834C82A039F48EBCEF771C18301C789073C457AF59FB5D5200D0EF62EB055C6C08DE0DF25B2E34ABE200F3D6C9B1977A89A671668AB4B10ACE8E78A10379FBBBBC19364AD7EC795AE26C207329B7C4D0233132D21679FDEA77F275B0CACF1EB6D6FFE2D42AA0D8A750CA441731D812BCF098032C83A0B5FEFA7E86B10205BA254E1B789B59D98CEC272A179A695EA3948B1597BCF2BEB653332FCB5554671EFC0375CD907D281F03108BD52ADD05DB18561B6AE5A30859BBDA23329BE4BACF3DE9B0891412D85AED9D2983CB0D434BB6BE70B43F139BE09A02A7AE3461179D19200FA016FB9396199C181BB3D947F2BCA6E2ADC899EC2ADD4FAA68ACFC400338EF7A3EDB8451F0D9D55B8875488EB940FB897C5EA9F42A833DE2BC2592F0A2A4976A55DEBDC3C7516195D54A0E9035A97B54B6B45CCAFF2A32546A394499C7B42BFFFD7BB84EA7262AC8B86951C75009146546F4E4320BAC89FE08195B14AD089A6B396A46CC5ADC509AEF6DF3B83194689258C66C18D27DC006F1ADC1C74F47575F64C92D1C772402548A130250BF68F0CD669DE41A8218DE7D7A31FCF69D3D4365B150C866941128204BB92E03BE1E98FA1C4E9D2A57ECE743ED18C04987D62EF3D1E7F5DE6E8D2896F17D204BEC7CF2AB6A1645B8B8912CB085347F9A104C105F76D83DA9B24A8382A28403D3EB9C81102E11FFB94D478C4E9598371ADD165181A9CE70483D912356AA062B8C0A3CEA544D12AAC653BD0C797EA23BEB8B035C78A708539F1694E72B2F7A4B3AD60F14A4FBFEB118FFBFDC8F53657C06007CEF8A64AA1DECCFF14C64D1807141850354998CEE19758F0C7E386E5F9649D8D6C964A329281B8348E9D7A59C2546F95F91820EA619DA1700DE87B9FFFFA959A94D7F9F298DDCEA6D48D0159F9A6BE63D84CBF04551D0DEDDE5F3A6B662F586DE92D2B6368E96C9311205C5B33C6EB1AC5F49D3A90A0361FEAC68DE73D32BE91DD33F00133680A1CE1F36D73F0FA4EEA975E4078ED4C6E6E6E81BE55C5DA1DEA6DE0D81DC8024D3C74508F8E64E97834BF6D3D98B5A1F64F18F38BE13FEB356785E37FD3E8D1E25B51CB69B1066E6A76296FC1B070694012B4C513EB88FAD48E0DD28C384639E73894AF1D0EE58DD2184860CEA9579FC8D5E7AC69A8CF85656BD08999C7FAC4E6CE811F2F96222D9C45C91DF101417E2FE0B5C84ACFBD14CA27109CCFB5CCB4E5AA0F5D93B077BB46221AF87550B9C3C53CB43AD525D0939FB950B9973A3C452C42952EC4FDDA481210F26CBC61B314C41E6EC23F8693440470A1C6E88C479C32158A415AC6E8C21B8C12A146D8977ACD7EF1480248D5FE3C58511CF6BD53E045E313C3DD0367E082B14B6189FDC46334FDD2082548497B95A1BCBFC5DBF8FA1120344858596F7280939FDCE1E9F0262A44464B4E545658606D708890A3A6B1BCCD124F6685888F9BB6B9D4EAEBF700000000000000000000000000000000000000000D1C2F3C" + , ExtMuVector + "ML-DSA-65" + 121 + True + "AB7A4EC2C0AFC8498CF7925E3D60A93B6B6EB419BECD5EBCEE1C2AEACFADFEEA8A2B94ED05A43494E1AD8DD53CB6F5F5549C70EBBE3733A30C19D613B3458A3CE9A87DED7D27E65ED4B9C3994D0B53344EEAB0096715CD3BD062FC5E493DE2181BE3B5FD96B924D3FAD5B8FF6C77DCBC3DA1E9C027A1E679333A5E35AB8B004A71586147841003264887731527328242807853235432038134051624202346086281176674880504571712611408565817276707863652441688234632531151414526434617442002301057453217218367123071021625107782412184428103038308130725463267333656581642752387685224571011883663382427660465565077413126127088487371543152880358858055051321376430111706840712724011545160338687522458026627145875285466703575688838580725232114023676876528480353628014710087842704177204183453420133664574077156886430715815673571314212318682224003872761102200576220438207405870026044121220200405883612301645714386406554274811828087515712770137862343607458325464163356456426615881684413768834457427451266667600867488673163253071556440082208035245580538754132375068757580665877131711486083864852472865210463525434668568031038764681301800841364027417747453562101681174623311603614120344231073470172104555805401341184128332224726687437860070812682822613011600714074403761324886008624433871350674831163132028630005855118412075743812074472750267308047510812847348845324822742168524417756540033734323600025665083521001221317575617605841284204586125815083553413152117710830802537755523165610726814433568321687118558661788005721053672766307677133313243572455050333471710188355120823878375658652570245425113815240146136810168632252410380760825757811251637523116158642342881660218647256242603011173254121750446688073286831603534178815551641581108763652568816886585304108446403036118130251384038481425412270328168587704453773656063074177375202314481714308072344401132260444773371278212485536412335073463664611471813806370751155852353663864561740764065218333351808330238148832376144483802846675306210781571511883028755774567757023810855211164202637348323001427744707070526046047426584777272851434153757335183865451567108672466843235573268086586212333318753488316753775012228237556036055660815377356285625313636214862420805678175736270058525615080488388070884533847462441678711815703458235013652682614481455234613860664080054443642163031825603244451230373708462650551003371800327257012130288034210067206634560675565806768871325875382837323747630882351666287123354045786817244850237114283035884267331780642008756187156471037281528463313237550204487316442105658023144310606503758314377232117268378327374822115484276110277355308480518408664808353217107527618428323183230735430353214402450801445187612124235810535657576412370863712814167847778253250212325721772781614438328108773880615462073865072481506564745133245713750614866018706440633682058014728641814430204722287080214700430471427261478316565412850287855315764540074753325287577144387387460452020077684135743551700380331863856257066685864103828167481406622455436571500500225684218255500620135162856367085467247548812666261868304358145002476357835523564632807373376612363648656084410211042107500014524013518714666368834284772016188331990D6F86BED8A287C3246AD255FF4F4B8BB43C0EAFCC23FF5D1F0416877E341575E759406EEEA21ACDD7247D7A543CA079E027EB613EF2C1764782EBE086F6BF4AF2EADC49BDA16968BCFC2003C97888DDC76C1DF1EE69C543208A43A8C28AD3F0D0C184BDC220FDC51B5F32EEF57C5B16BB695CBCE6D6C9CC6A5B217F48DF4AF809463B387C15776E2054B646237CCDB7B3909659B359F947ACE82AF58AE9ACAC5AB4AF47E5A6B7CE42FF257C629DD89B178EB218F3E57B5A7995FF41F68B75EB39AA5C77BF0097FDEDDFA89BA5F51C762C991944F40C8FF7D38CC5CBA92B7DD709322AEC9A050C524540C853CB5A690447207315DFC962C4E389253754908A93EF6FE5DB5CAA6ED428CA80C3F81F3C5544BA8376FF6BCE9050AA9475B411D983EADDF8031F9C6D7062E7B02EA2DDC0A9F1C3248329E602CBBF7080C66D42AFCBE8F628DBF1F2DAEDDB65C3DA527266B4846AE8EEA52C419C91CFCF42AE3621F87AAE6BCBB91FB49E3CE906B39ED3DD354308FBF373A6E636962A7023A6EF05512FE325C24DD6A120FD60737FEE47E4263B86FE28B46BD3FE6054D3B2EF88B7E20223138D9E6080B8880AC9BFD3EF9746A32E0402BA298520EA0C5877222550F1CB80303FD4E5B1D06404D54958A2C30902A454EDD2F62A4F3F2C85E953840E17463F600D7B19EED1E2D15CA109BDD9EC0435A6BA5A0F1B46C94BADDD1FABE95BC8F1F997A5ECC3A69F6F70EEC3D3AED3573635BAA621E756E5CC5BF72D2625CB079E80AB8359CD0BF8B70A431945EDC3553E0B49FE7079783E7EB28E8710A67B2810C6B78031E2FC5DF61C0499ABE789F1CCE15B549443B1B5B6CAC0C495C600D8877C5A4AEBF76953CF98C55720F425390EA4FFDBC94047D89C9AEBA3268FFD953BD5BAEAE8F40EC91C41E9F89F40D610EECF5FD56E543F2C1AB9B5280A4265D83576DA1DAA606D5ED9442767EE63B3D79BBA0BF68ABECC637BDAF99F367F19E82E148A0803BF29318FDA961C415087E3DFBD070BD96F5480390BD05112BEA81A104BFEDDCEC6EB107A38A2498A64DBB6B9B77453782E06A75658B0056DF1D71E44471A616B51D24EA6D942C23E1F6F50AFEE0E1FD94E74EBB42718EE0BEF962F73FFFB1CF073CFA448B1EFFD41FC3249EC9A249BC1ECC5635089BD8E2978DCF329A75417E34A5FD1B78A6C983E6DA4552DB236F5A4B018B6CBED747C20B3404F6F7A2F2D3FD7E43A3D078D2AD1EE0B28DC9E59E593073137C2F2891E3FB7FC7F9FDC0C1F5DFB7E83BAFA2D28EAE363EED414BF38FECC8976ED6DDCBB3513CC75302C849559C5880975247C55459B4ACA61E5D1855EDCA14D6C91AA479849045EF8D30A05ADEB605C2D50940524374C332C8846030CBABE1EA48FEB1265ECF913369F32EF308459C9177768BDB0F9F69A3DD171748F481D4C266BC13C9A4611F6D5B666CDFABFF13D1794FFD2438514960F209F23FF967C22A99E0D341BF61EBB494E3C27A45E4E674F854C05DFDD3C61616A00A3564037B6A1D665F99FE955322967C30E37D077C731CCD03F213B0B66FB8900F5C5C64A800E367C861C941F74D233D5C01938872E2B29C7C55F50ADB38D453A35EDE8B19AFC2662230E8FF719D87036751FEB3C310E08749A1ED60B791BB268B362BDFADF3FDEB4EE75C89A87E3837E1714C29BD86651CA0724CB26744A551FC0F4B60B57B37FB420D15F0735DBFAC893BC321B63241CCCC789395EB6E545FD6F77B087E08C3053E9D65D29F6F4418D02939A2A197126A75F6EEE32514BC270ABD990E30CEDDCC7B31D77898AA3D4BFB84C6D3539C894930544E52D107176731BF48EDA32C6855C04094977FADD763903361EFE04141058856B9C089DE450D84B6CEB08F8B3D924844658C74B4A520726B6789E116C0D7E91024CEE1625302B2C6E12E0127B01D0FC5EEB9ED4D9F5BB1676E9C37FEF16A6B2849D09AD5369F5AE49AF1ADE0AD6B5E248064AE29BD57B00F3301491B34280B2A671536F72EA305E896E637E8B662593602AE928EF0A3EFE3E510A193FC18E1A386E4388BAAA278E1D6C9C6891D19DC082E084C9157ACE6060AAC13F08B0ACEA4AAAEE3B53DD0E2E6B2FA12EAFB3751316DAF990B2282119CFAD216F0D567F0B4B67EAA3A784592D311FA8EBF889A8F3E828F41311CA8A96554543644A8694EFEEC080D596C0E6C84676F448CA0C3865F687D4A9C56320C36F32B058B061377C8FA02E07987EE67991E5FD825BC54A457338B82154B24980FEC04919755B777C19F5A8827DEED048353F7A5FD8FCADF061335839558067010BF9621BAFCE5BE0AB9C5EA2B33FF21D0319B4A1862866B92C179B262245CE52D8C931FAB32E5858470FBF29F9FD452B7BE84E6D71558495D1EB40C896D14C3C833DE0007F87F2D512345C9609FD6F72C312E883A4D0F2D23F1343C3A2EA446F18EF67FB6B5F77080391E5E580BC0AB0F78FD05A3F0E3F9281612DF9B7ED63A47665D519E865BF20E46CE98D04B147D5E711D5C6D46E3071FDFEC710F39C2745273D2296395CE21AB4C423BA46E02142BA4EA2021E1110B1B2F65920CB7A137E89C3238A9BE7A01F8094DC22EDCF126197D5FFA771561CE8D72A47693D883E2BA502B59E312FA2854EE898B3BB0D098755704B6070AF9653F89464E4BA8E281CA7C1F4525F61ECAA8ACF516A631379B1167B669671906A9BCBACF61EBA9FBE2B843057750A27C9AB60E78141B281C6A106B1608A2A76FEDFEE895B89ED53CC2EAAC7FBC4BA00B87B929BF50F043FF776340D633FB6ECEDA3C02BCCF791918841D0EC39905591E3A76C7FB32317C42C13B73E16CEA5796834D15E5CB5973C3504E1C1FDD693AD486134CDB358F76AEF38C1B9BE67D5FB150D130B53CA80325F0D9F4AD1452817645144AE1233D92162021227656967D76F3872639DA284EFE048490FDC7581FABDD0850B9680C74940982D9C94BAB52600CD46E1FFDF0182104209CA009C658F455404AA5EFB3E6463D80F4A76BF13768612542388CA2857D852135E1E7D102BC99AFAB2649DEA08DB41933EC07EA61B1728BE37848130E4EBD5135ECA073E8990E71EF52E7CC78BED909B346AC651BF8AD885C36FCF0E6F4E4C6E245B0A5BDA75D51D5593F31DA7D42B40B6B26A78237CC98A62BC7AE21D06E7BD68066A9CA9D4E507064911B954BE137CE706641ACB4F182E085ED9BFFD903D670A29CC484220F8D13B56B05545D5911F4C8A2513166AA0AD1816BF186B7FE7F2CE393515BF4402A8BFEB4836CB902281BD6D1959CB12A63CD5AD1D9D0F5EBB42C0ECD49A9BCC54A4B4C5DDAE10387F7D1802C07AEB93F3E89EB641981DF126F500585B85D23ABC8FCB5F05C32675CC345DA250F3872DD5328E551FE09EBC41E2E886D591CB2EA48E8B6FE3D09C396A56994149503E84031AD9DD600933742749D8CDC0A0738FBE40D9932871CEEBE57EEB0670FFA67108C7CFABF809B38C2A6074B3CDC4F5149F225086BAC3F80767D0605DF799450C147" + "5524A9CFA4CCD49DFB40A32FBFCA132E14DD58127E556C0F1C517EA0E9494DD96B803217CC34A89ECCB25CE151D18F9989F1940CB12065883C76044C6BA05057" + "" + "018EC7BBA66F8156D1132131F909F72ABF83122C0C9EDBBA032A851CF1195ED1EC8752DCC7C72AF830299B75B2E7AF49C3AC3CD03F266739C723B3B245BCEFAE6CA6BC627F2706103586E72A0FB0AC5168693210AF8081938A0E77801EB745A130AF21D513EF75B55352C515894DD8C35208A9CD6955D3B193E54D742835820DD2FF3110CCEE17D92B23ADDB85A3FC0A4B2B615ACCA4E3078626F9904B42649B7084A756F8518C3E0C3ADACD3984668282C3918087C10B4D4DD1100063B3756AC224717F3B60DC4C9B8FF1A5D5DD1F19BB0144349E67A28F0C6D42C0B13A8F83FE16418BD58E8C06752D0BB59A1BB9F091261C28B7E9506F85C5D2B3555CB62BC763E60BE24A6894BC08540ED2380E410852D336A135B4822F52C1FAB1810F7E8531676CEAE926A1247E607FCD16738D1F565328968F166A490441F8CD3FC1C6C6D2D86BFE5A209C923C99BA701FB89B9C719BB69CC8C32076CCA9FF19D961F8A2C2B1C01697719768E134ADF6B28E5F20CBFDECDF2157A961EA2253B7BE3C5B2BD8B1381705866B3BD9FFA9EEAEFA56EF022FB83459ECA7342314B8CEF9161838257AA60C0483CE652D1F58516F929D9016C75DAA1DEB2616AD41A7E8BEBE9E0A834A5D60CA4C26F1F50C1A494B573917F4A95323C4006D3668D416AEFAA79D6FCAB96A92EF6ECF63DBA0CC7480D16DD591B11EF359D998A49F27E5287C9AA16EBDDCD3127C757F25B3221C19ECBF6A7203ECEF69F584A43C8382D513035D361E0797594FBC85AF686D6D6BDA7F2A42AA01E8AFFA32486BA3CC00A84B716C20792DC8CEE14226CC4E0A15A97225FD66C74ECB8D2A5BD22A04E958A20862120BBBAE092DD2361569F846B84066DA972E3CC5F0042BE682BDF58738A45FFDFE50CFD6FE165A2B42A9054C0F256B42532BBF83775D47DDD9AA4BCE6145175FF8F3D4EC56EDCE2F9D152A9DF7001674A10DE4A0225E144ECDF0B6DB906FD63C44F5CCD159C17B6B0644CEFF5F7E509CA5FED88EE51404951E0A1B8E2F7C6A8A2740FD304A68B8D072F4B350E1A487421A600F1D8D93771F18784A4390561828A1B1BF5EE5854DF461D58496DE217E269871F9D80F432F15C87529F04ADA33DBF3497036D978F1D93D37B7C56415D7F3BD95D5FD520E1AA3595794FB51481A838A97E08C71E950865170DC1B9FEE84F6B2E9DABA41CD3970613EBA01F7369BF68BC78AD3B21CA90F2E00350F2428DD3B1EB3136902C98C13C493D36CE3BA7F80901E01210E47198AD524808DB36F74F106A96480168B8D8B981BBD635B9077EB463C59DC791F4113DE6013F2BD5B6897AC7F04E7A58938E684BEEBEEAE0EA93A6FB4809231D4770972EF987D5F6B361EE16F895936965C6B981684794385CD0028375E8CB5C11B505D371DA33418457173E368BA11361BA461742B4E118C6242293F12E56A34EB0DE9446C45FA0BF591BCCB527A8434F67DC3D53633B50907E76A002671517434D2655F7EA1B5089F628047256BF9D079C0A1A73A951404B6BBDB9B2BD5979EB948494EC73D3B840080CFB79B3354A9D35D1B9D1542F10B44F8E55A6FB037C92A61AF475D1D6007712407F43C1E8EF609FCE19EADCEF79E4C2CFC5B28CEA0226D9BF0029C70746BF4FB752B28B7D3516E6D86F17F8CD5A9CB94A384389384A9975CFA5A4960AF5397AB9AD38EED806AE8F1E19D2840D25766615DA9FEF4D6C52C25AB9F9740C063FC8DEA4312A1EE7CA299C73F7AC7A1544844F27A3FCC29A92BC544167C128C7564C178DE48AE9807E62B4C5DBF5C0C19607D8D9F9221FFD04659CC9CC3C9DED4A1792EA787638402CDA2995200418EE3F827F976B73A53D256305D8FD1744FAACB4DFD58D305BE14CCFC2E11635FCD72937C4D4DE3C095EB179E21ABE5D6F9906D96EE97395568343AF974A529E80ED22CCACF48C1B39E170696E495C9E9DB40AFCFF1043D38C093113F0BA8E30A121D93756DB7A3498980D527EA12D95CDBE609216705829332B0030E3D0C15934EB6547937A0CD9536A1BCA184E187955CA94C6E5F7F55C7883291BD932C69CD46D7BC6082B58CFB374B044F068E5F308285F5F33C70F53E9F92B55C6711265DCE124A6E6DDEF65724FE2B291B282906C06C7212AE353BC2356E54DBAE8B213693D8452C1D0AA2AE8AE8A4409F6A03674462D791C28E957C06C457AC158A51ACDF773CDFF273DB0543233A1E196FE51F30CE1A6C9C9A3DF081BA5D2CCF3066BC8F173648748FAD50D06FA7458DB7448EC4116A438A87CE43C37FEA690A2A619187964EF2165C50C1A860B6FFFCD815EDA7FADF1964AF736C9FBAC7D37ABE03A8D2C534B476044B0C5E643621EFA866A637727E6034CACED93686862A7985783E418187F47D567CCF81FEE7954EAEB3284652F24B84A1980AB1D26D878D8757EB8F721F8D126CE6D5F77518165524E6AEA9141464FD23FE62F44A0AE64038E7DFC01C6E481DD05B5E8636A9D34242E99A4C0438C5B9592D912158ADC1B633AD698DC31930E99DB8F21D5F4271C94E7D261938F6291BDFCA06579FD8D77409288BE72095113F8BD07A1F5D1171940BFCC15432FFBE0035492790070458A25466411D53D65329500AAEABC9BA7272DE27AC6E95383C3D88434A1CDC1CD260E660FFF808D7E59ABD7D0B8235210633CD67B56376A04325A3AFAD3A919B3D8FABA1E68246B72D87D1C898FDDDF8D255EA27E0487E8E9C068F1045977D86595393E183AB938C5A95E5B8A15C3CC8B526A28FEDD01AEF2FE244D79EAE1C2FEC714261195809A9C69268DCC4C50B42D0AE8C21DEFB76110F0FDD3E49189F61FEB94FD7DE0A1F91E780105702479F04071DD085B060C2DD388FDC068F4AEBEFD55D332CC0F6970EE3E289BF43A02782B6E09DDF5DA004DA158BBD37B7146B8F889B5D75825D2CE4D00A871C71494185DAC3CC5BE31C9F1881A20B2F1F36B957A606A5BD79600CBE862DBA025118252AAB049E961D0783839924C19B0B0BA65392414719AA79B714DBE3578B71512E908DD70B90D083987296914A562CF2444B0B005C4E9B9F6E615D7323DC1AB464EE78858D4D9CBEB79556CFCE0CC4D78B6CEF3BE34FC8D0F7EA4FDDFAEA18E6BDE8FB44EEECDBC19E21F53241880994D5CB39DF4D254ED3EC74E6756B24F2FFF205E2E3F6E223FD1B1DAC16A65181EC992303621F81E1AF4A9285A952387455451489D1EC3A4DEB5668249C03BC33F6984F56F0BE9C880082476EAC7B1AE2804962C9427FF9E5D9FFBEFB9DCD2A02B0EB1C25DF1FA18CC5EA1C58E7378B8CCCE2F760232B896506C8FAB679BFBF7A773E42399092EA5C038E8E62972A3AE82656DB7B64E096603542DBD96A8C83EBCA0683052EF1C3CE5B46DDEF8D71CAEE66B0E09F011A546F7017F1E406B52C5D04F3D81A451229DFDF50381EF7D482E182246FEA11CAB88ED6C3C8CF5AF53AA9B0ADFAD32F4A256D626BDB2AB15DB686DC8C421EBCABEA0ABB79DA7C86FC922DC1C2E62F94A644D6007A4FDEF59DA86A35B1887C7B56B98C84EA8CB82A425CA0018160E9F1EC9CEDBA6CF0312B0D4734421DC3F621B8FF37F0ED45F098E3FA772CEDD2195DA11070227005765E05E75FBC5504BE7C2327D1A63635515D22747AD76C9CA3FDB8E9EE9FD911C076C74FABBFC119A46EFA3C4E903207E1218CE012B316E49FCD3127F37C5FE66A6E5869248B1B3F0E3CFAC235B7DE2945CBEB8DDDFCAC20FC857C914C84FA1D65A2ABB5927F145251EE5ECC41D0FAACAA03B668EB015A313303BA3822905A28E5002E4211456A5B4BF27130585CFD3FCB3A261D400097557239894763843B4490D616FB3E0B6BC6FE8C77B75197C1B6247A11E34BC6C0B5430241D00378ADAFDF315B28B6759949D6DC5E4DE1D08F739AD1A4FD07BCA030E9EF1C153BD1760BECB965B72EB1F567D82A40DC1D4161CF58BBFCEFC2A9A9988F5B019BCC15B3FB30983FD2786C7022FB0943AB487DD2FF1A35A8930C832DFF2DFA4C6310DF5B71154C0D8C8C2A4F0539ED22563B4F327DD331DF5D687243D19E4522FAA94E01B655BE918A689EE554882D59BFDF5A4807677396EF9F5F9DB016AC1CAFFA9AE38E95A15EE8D2EDFDB1F955A6A5D9320CFC730C846346D237D498A52CF7D25A37A378EDE231C4525F5240CABD265007BD552422594852E012074429CBF0BBF5DC3F606EC4B9939E5BF51310C1A516F91929344EEB05B21D2D1AA38C35300CDE0258B788A7EB57129D10EAEF9B51A4172DD52D1E7914F733094E81FF946EEF167E140DDF25F9DBDE233E546117BB756718F518086A1BE2D5F3EF7DC7BD9B493B9E30CCFCE9691EFEBB65616D8F0ABA8CE96647219A00883B1334A987D0E89C44F6F4A15726A40DA3E9E29FE61FB5424F5C6973170D587E5D889FF69DAD268F4B94EA3C86ED54BD27E2B9E10AD1C3FBD5FCFC68B498946AFE657A09778DF702BACB9029539AF69C99EC98F812D649FA9C8DEAB13B1E24C2F51E75FD70C24FF362E3D8E7E9F875D53749A3B4D94AB5CF253445C6BE8BB5CA527323039479981620A8F5A938D50ED2C1A9E81CEE3AEA77B4AD2B6A804A0A373EDE5791090C2757515E6489A3EE086F79A9AFC5CBFB32485355838EA5A8000000000000000000000000000000000000000000000004060A101820" + , ExtMuVector + "ML-DSA-65" + 301 + False + "99259B67E253EB22326B0F5D4276DFDD07588F2B010C8E41411B4E623C5B285BD7ECD0946A042B660557B1B650B0278211AD31AD25F201C7D66C68E3A2F9FA1FB79DC085922321AA2178CD1F344B1FD15C7366437536D6A0E62A43CB8976D1278558CEDBB7FC4CDC746B672FFFC0E7FF04AC23D687220CDBFD7716B7B2BE9C9F5436323341012526304751178784346354365266180061628256170683080376663588271672205532066537888455555837188244235611837077113032884571828175365720315653671642517885454551186606283882111214507411706518530748408458285027548020617185522486542012656151033036576776481822653305681111408641468800328783753311405150841354627800236743238324667881314056888427280106004300471873781412407667330251485114213472437866200462624111676084424122420241628383324437266525571212631618328558184042356162646345533522433386175883753613518047263585633710375032253800672176338312313224278467128421703484676561805032627285502234822773727261634710517585168858033325601477732738157266560675587102358688207618013435383308381230688344875331488524658761065123178852442682377470844447516187082365612141562280774428557010442087230266105188305426640311356205671274687140685726088410821575822473311063264540066515166154771028561777041736023803604564263258874616801067122655484713216415023351736088832234173620757877602810662818768866080385153155677438331173708481742271715564642260546385855548682378055584806350051175707686853464276567171402087733133363367701666630136524613272365728070673034580783046516786707671183412461402862261522746001248735226566323436125738848364012175786708636780361103401567284266708760561716371228328724465630322187444828444846388366178526314683218720337676406448178140646332057560460462665715538430347478842808513837484458441388562330201310540041762325027877541288877645584531203550116266428773606586364012454415403475817461184778247854741301057125044512851326048465878142078204521002034105441474712483574337442586323207351126877371506537330662521247020627446643602555175710131522456662784888783264756057458371027773008662488024542552558240076050222228711207666786500823435600803671106623756638432832600545226756344151616833027874344211476556401155070461064031065186663101041648167031818202442266843455164620843482212757625657610174771446766320413055648781578761382323823851422800810021518618006661864620611614485000856326102775028273151581516641506855366633608431287403410057785631844523706465466812815180274558866260060408275836082043441422344166706470184527407862020810670688028327364746604441327303228614526873146341447182114484064436363637377033224220461350533578431100056500546883254113652322723242314433640646643878847125082155657747022147783170867631237216203306864758401117524684187224865862181271456617230442540462478244680148542628180568125480385506385546087007411545122137366785881812513550003430470082042580301773104346104627673216635057133262163340445681380360138641831685654410115218106540322261472130612772503862676523325725454574705127516652837676654184837142885158761564366705132532683042413427827156683823133016152881142313806311547256036683474676621614427053028120007124164356421445281555253013218288067861504B976F2F81B60D8FB3682E72F4B9D63BE886E62045C88982888CB64D14A9AB7E84C7C4128671B28F66057E232FCFF40B1FAE2E5188786C20C4F6BDF05515B77C9C474EA1D002387EDD61C30CEC50B9C3F34A66B82296583CD659BBF22CBDCEBEA4FC7CC29BE2E51E22C6D7DFA18001226ED0FEC499C53BE21E4C6C39B2FFA98611477E98176E8BE0C1AB567B1625157B152559FECE36B2B78F0857D5C7A717177AFB7DF768CC13C02BAAC553C68FD0FC103562491F576D69B19F17428254853768151F8AD91DB98F549BFB452CDE66E650A4CA4516C3DF33CB7A3432FD5A76D786376C1BC51C29FAA456EB60727EAE059A29766BC07F58075A95FFBE997CB9441493335060F2E8853BE632466D465B576A5DFF1B234091AD48365C2AD3C762083C5326E19C0A6BF55FF3036AFB1393ACF6DA5C9DC377004DEB454C0E06A80725BE1DE947DD1606D8FF3E3F509FB5BD78D1F57152FC86B49698FEEBA48E99E9A1EABED2598A9A21E46C0E26D82BA83B4E908A67A37E7E922DDB16EE9EF9D74E3F9CAEB9D56008456A0D1E5A26B5CD67C2B7DB29C0E55DF937EEAB8E309C795D7E6BCEF8342FAA5125FCC4AF8BA61CFFDAFCBC5725A32DAFE0034B70D995AC3EFF70B849E7915BEEFA9519DDEB8762F57F449DB2E79A25065536680978890EC8ED471D4DC01D29D0388E1EAE8EC94200AFE7A34D19E66CE10389F74B4D447D06446460547AAFD109439E84FB223A0FE5E446E938383C0615E60A68D6F63D5F9BF69CB674344945D2D79EB5ACBF751A01765F8625F3E42E2C319FF5AFD81FCE7789BDF2B0F6054F377EBCBF57F97A1734263730D14666C0F4510461973D57169BAA92C36EFBC9D18FCF707C49716238937FCF8FF3A7122E7C5B0F7589DA38E5421C318BB9A8CE7A2EB2A8A760272E60C0427F51D3089CB36119C459FB86186D11EF2E6E19F43C9EDC45D6B8523BB175C63669876E24B6280A0EC3B44B902033667DD495DB6B9534A7BF05B2359F19876D9747553CB176C8E56A15012EB6D0A1B2F865B5C8BD717687DDD1B29DCA7622A4D914A60559AE5DFDE29C0F51534F44C4FC2352C1EAAC3023528A49858223A4A449A3B8A66FD798DFA0534751E22233AB53E6F6DFDF8EA73DA82B5FC9B22BEF2DDDEBC498CF397EF2A2E3F8095BAC3456A327B306F2FF96F9269FB420E32D4AA2963E807ADC1AB1AEB731C35CF3E7DD35202F1B100024677184BBDFF48D3805A62616C5A8F1ADCAF353810C50FEB699DC10872728552BF2362183641671D34E821BF620C1B990A45B973B388CF18860980C8F469CF955B6C8905A8539C913D1731E1B4EF058C893EB9A2A3AD6016BA5DB29DCE869CF461F38C3524D11EBA59ED19F278267B04DF1023576EE9FD2652101F7D9E5C0B918E07312B7D8682DF8B5263ED17A08D7F071E343D970234348ED546D2E9440AB0BC3AC4DDDDBF914F036AEC9084E1095CC34475CF67F661A6D8256FFAEC8A3FACA34F1F281F0FFC440DCA840055AC65C9602E0BB349599CF07DDF14F446E08DBE1E2AEDB401C15937BB218E708F912D4EDB974D7B99D00112C22067B3A410A58EBD90BF815FBD90A69652A0350F3840D88FFD12A22108C93DD8557CCF10D881FCCEB4AF521DFD993EE199C621D13BA8274EBAD07AEF3D47DC418670D57F36687A7FF8E6261724BC0CB57B3440647BFB35946BACFC1B583EE83519C3AC189B09D002B26146B682BFCB0E60EDA634850E60DF239F7AD509C34F85ACFFD18A2F38572607F48008B023B32FDF03B6BA8B05A3DBDA1EDF7D56E4720DD1B70AB5FA014B34669CB086242F695C0EF459B88726525D0338CEB4F5C5F002797F5C89D4E925A378F252CB7361057F678C003B491C6A52A90FAA2F99B418C00C6779F089DA552624E4E6E5AEC6022C4C5BF36109CC8A7DA3AF08629C72B9804004B4F2E7ECDA48DC381E625BD67D6B3E3BCDC937BD1284D9981F7C0F340AE4FBE356E0E419E7DF1F097ECCAB1C764EC3D01C00B79A5936C69B8A0F297CA39BE32D44940C0E231E811636894C575F3B4AC2E4F6110237E784E7EA72D88EE4162B6BC606DDFE7D7A38AAF83CD6497B531CDE9BC267B270B139579539ABCF73491509909BCDBC358FE2BB1273250A7029896632DBDCE54C92726C4804050835C0DFC37C21EF762F4D5BF9C8FB506ABCF13644609B05ED1B8D7E8DC48FAD5D1ADE9394F0099DA60F317AB542C7180F3FC263CB946740BB4207AB55224BB3E80CB6120666D4C7645CC06778F7FD1AB4B64FC7D6F2CAEFA172DB2DB123058D6D9600E1117CB2B3F2680C5F9692B4B1D0879622176E0503FE9FAE96A241760F02B68269E231E01AEE22167A783028AD5C43B0B3CA03528869D293F5E303B9192D7775F2E9E6AEDDBE979ABCA0C591791B36AEE307AB04D0FAA8CBFB469210094B211B4A14A60739931E0EB012D2E0639CCDBA3ACE8FB26AA8AEE54DA0E5FA466B506DB7BD9B38A1F38DD440A4EC166EB8C094BCFBEC30584BB7FF47C2D0B40D62E2F925BB12F52DE35D352DAD1963266DDB131DC1D74AE29B848E1424F3CF0FC3A57A9FE829B30223284A9276CC23C034F96D829D833F995315B4A2548DED2A728C081D3AB464C615D80898416E7BA007FCA2F903A21CA44B4A24786C5468596661238FF70AF25723BC67B07BC8E90C4A716A7C292A76018C971F5C0D8915018A4EA545AABCC003D27461272A1A5DB9366131F660DBDA5B5E687905580C598C1BDA6ACB29812C7F15BEE3ECF6D3D316315C444A546345BD82E926E694096B681406722A8C9D7DAD233FABD0892309804B8116C922D58540370D3ABBF856C282DD54198E4098834559F1B21FDFBF2B81B8D242D78E1CD635466796FA6661984F0ADFB1934AE46047B149587FC5E9C9115009096048258AEFFE02349FC5C71F484D438D225AA2306E5CDAE809944476FC842B10E3440E84C7B03445D0F6DAC79E9A8CDDAD265F5E305A8D47F7B329032C3F3B26C0318E2992FB79038253EF9D8241983769BD739AC35C64FCEB0834F904DD4CBCB5D55D4AB38E1106A35CE7C43E4EC2C7C18EEBC2EC8341E3727F2AFD719D6183D26AD35CE5BD6F498A3C7C61D80F98BB79AAB2F582944D574905FFDAF82042C0E0D72FF9F308D94DAE78F65869C25E879B62BEDE1E32CC919E3E432A1BB7F4264EC7A9E515469469CE21F1281D4F1FBFA5591C209AEE129E3656BE64135B5A977A49A9BBB475E1FF6742A4CDA1302BEB3B9022FFD770B57CDE8079E8B59B8389D310F4754486B3527DDCD9A2434892987E8330FDA910D22CD4E7432A8D809F433B807CD75F8F391B8FA0ACC2D168650929810D8611CA818190EAD237A8F715CCA170FDA02050130210F5A61AA51C789F2499DAECCDA83A0E215D464CD1ADD237B89E280D75F881ECAF7B9CAC5400A21863BDC16584D085A762594C4684D17D04DB806C04911EC82337658037038E363688D393FB84BA37F9A208A2686F6D687B24890F0814BB36C9A6FE7304F8BCEF7D43EF" + "DAD12B228F03FCC52986A9E70C5BB0CF6CF15BC8430544709F7007DC2CEA0724FB2DDA5FF6A2CF19B2D1AA31311AE26872615E2FBA3E7BB43021C45F1CAF97D3" + "447DE003E066C1807A13217FAE2636DEBEEA8EA4EDC17A72619535732896434D" + "4D5FAFEA677654549C97BA59CCB6A8DFF6618629386CC4CC33516D652D9088AFAF466F34966128F7A10640D93116C0736C69FBF6C2F1E12F36E53E2008AD030CF543D9783B0675C964295D22E27A14C0CECD941DD5A26FFBFEC9588FF88C58758D66DFBFEB47561603B5123D99789D678B94F6F7D122BA621A1F7366EB18B114E22CF0B5CA3A09E47C2891618A9B1970DD5B18EBFF1C897C8A1E34A7ACE7021C982C135CF6876686B927A9A6BB0CD6F6530A5001534386D81296E96BE0AEBBA410D9CD45F14FFCD9C6E6CD461A20D9D8078DF048969BF1D7903B55BC5CBE5562EA9CBFC12907ED05832195011F3A762F0EA33EBE8DE46A75D04EA64B4DFA1162F046A436D739E1655B98BF08C542771A5D154E911E14D012AA0F2070DE7FC4F60B835A600E46E8B4903909349DFCD45E6CCA19D7CCCEFAC60BD0844DBEE74EF21C738863918C048900429A1BFB5A223C75A9C958217F599EA54600696F8D66C2249D878F348A8756AB3E973675A46A9DF1B660E6400FBA9DAE48DACDC40A2D298D4554CD2354819451C8E017D2738AC0A0DB39082D07EFAFDFE3B343F59400AA63734119E83C5B6A14FA96DEAAC024CC9AD00E6E92A17F1B54A6DFC8B380945B356160240FA272D6082FC886EDDECFA9A21B2D7F42B10D86B404BCA3BC4987885A4F7F9A07BE46BE245BA39B36202ECC76228F6DAD289B7ADE83C20CDB021E8402563381EA427F5EFF7D41E4BF8C67B106096AC1EA8025FCB58F72A351334FF0BA39592F6E2E72116F0D05D375815A5445BF28A4DF060F4C09C4935B016CFC66C41EAB03E35B6189C4C0B0C425F311A22B14422B50B3238E053136AAD0BA350D742695F35066203BA9135D6F0EF06CA65F0E7E0BB53E01A82FE6CF944B7516F9B19D92A0684DD97672C70C589AB87847B2339067F1BC0C452C1A97EA8A82E9BAAE384829816BBD622F310510BCD40A7D0714CBAAEB52C9681A6FF2E16D74EEDA2C52C2B0DDA560AB225B99EC0093E0E4A49FF748A733740E81BAA7A0BFB6328CC236880D4B5CD634FDBE0EB24717D53C055EA4E41B7BDA27AE7644643513B30917455BB6C456D96FE4419CDA05116B67D4870888BC2FDFD8548FAB73758DD52A64F8EE112EC365E6F6DCF0B373818868A0745C0836D265D69704BF6E7116E538510A699550066244C84C7357DBF0E375EDDC39CFE9B37DC5D26CDCF88B978DEC77D31ECCEDFD1C2D649B4C364FB732D5674ED6E3975FCEC1FBBD6BE68A8C41094D1AF52D34663CF6DAE4C1C40B0D1FE474530F50508845E714782187A77851630A2C1A4572A9B2710C67ED44D09979680825AA6582BA2D6902D2F30DDC21A9797D361BEA010FC081DF1876A65FAC518FD0F51F4C92DAAD50207C4B1401E0165C36CE6AD8EE2BF51FDF2DE387844D0F38C02E381836105A291937BC3019AD5660D159CB13BC06BFDE6B26B7225BDAC775D63B5CEFC2372451ED27DB9DCD6EF6C5ED7A14517F5F20307765BDDA233FB1680182701AD9DD95E241AD8B869B3236AFB234B200C491C75903B63F33A1E978060A89E7A648C50CB74AEE751B49AC756D82CA388B15936F6B542010B230BF72CF5564E83BE7AC8834EDF793BC15A795B41AF36F035A36D226E18801FDBC4A8B7C9741E7135D900CEA7A174013FE81E7BF0ED72FBDB35A891F727E18639B7C2DE022541FA2E3F9BB1329C9E696F845CE4056C7966F2E49E56980B6A2F3D0DF967F486A59218EAE71AE746E8567441D8F707F622B3B159CC91D9263BC15658FB0441582EAD3FDD46601BCB3A5EB71EE64A95A4FBB11A64F3059B66B25CFE847416BCAA4CAF6B614F5BCCC40B990F1E565855CBE13766C7C7CD465091EA125FD715B3E30CFB4CC3DDDD3379C0307B3D875EC894409A1A30A93B8EC08F194BB94E06AC39E79F68C3A6BE8AA5986F21DEACAC7CC3DEE8C15223D27746DB20A18588CE30AC0CBA03E6555E68F54CDD570FD8AE59B9C8E152F9617A1129EE0EA04D0D7281960DE5806CA578E3B98D86A6204885FB5FB90FFB7F3E6E8388CFE5B21E829FDDF181670078FC8BF628920151B21DA77F8D52205E1F09A70F83ED69A38326378DFEA53B347DD7EA62E14837764BE04B6242B8DC7098F98C2C04B5C33D35438202348211360382045E8437E3E34B231AF5F5B0D3CCB786040C1B98470B8CAFDD2F489859971D8BAE5489CB7647AD1BA3EF85C7D2332621EC73C977D0092FC36930C65C825ACC30DADB5A512D76C39D3661660DC656EF9D60EF906461FC78314813B961D9A1D055C40A24EDB3AA64465AE8A680421D619D59751F75A4B50FD236976B619A91D0CFA022B6759E9972B0B0E019987C3FC2C3EA61F7B0E8625EEB07B1782AC7445578C5B189FD8C697062FACB18BE250396A5B5051170470F11B0F1E431BCC95896C2ADFD49E45FB92A78718035E892F4DECE6D0EB2D7A02DFA2B3733D3A538B0EE4B94D2D1807A2DB879E42781B3F748C9E0666CC098C478046AFA1792CEAA7AC5AA0A36A8DAF8C85E4BED9E040AA5EC33D1292E825811BD6936C53C3C1E332BC8080762AC51BE8EA98726E19AD342D269E76D5CDAE83A696AFD6DEE58916701B8FB0764B70638A24A872B44864175C1B9C4A137A5C46EBBB84913CBB39ADF92ABFF84C769689EC64E476CA225D7ACF8E4A712160783598F1B7EE4A79949EBD2D29F3892922D0A8023C0420E445687A64D07115851B27DFB09ED53F071409B8217A937363AD0E0629BC29B1CF1619A7C35CED703D10991F77DFD540244A708844DB97CAE24B0B2F59671A7DDF1DBBB635395A88E2222C3C104BE9E447504A578C08099A34EAEFF49E987D97B7F1081E4BEBC2CF0CC5F3C298C7D3A459536B80CFF4FEC23D493EAD435CCFE6F8EB220C8F66FC7C1C38BBAE36272D1EFBF8F5CE6F46DD7763CE06B1EB41D1FBFBAD8155EECEF416B4386781B27673DE69A316E59D512B3C1994F2361B0133E83C83FB43D210B95C5E583D211DC7655B1363B44D384EACB20E1DADCE6ABC87C15FBAEBF6529919078A02FB0C606AA4DF4493ADD707E3B3DCA40912831D2927A5F8444047A0F3FA567BD12FE85CD6B6E25C105251060E8996F14391C837366C61840143EE9FA7991FAD2419A137928876379ECC56B559D60C52360BBE2359EBBE17FD9C426D669DA482B7F85311121A52502172FF8A39F580C6B2F37C2EC20DE4E68287254B9EF8F2D1AD562FEB053ECEB243C69D4E7F78AC7E076161EB4432309FCCEC3728BE9F0FBE76AD21417D176769E8C8D9BA217A124A93DDB8736EDF3DFF4FE1F603187F2ADC40055B7560A9B339842387545815E61D75072BDAF161505B329126CA1360BEBCC9430681F92A32AD80B573E2FC1D38EBC03DAA5815FE951DFA02D160D90E36053D33A80676F086AFA9CE8A01BFBB169F10A54A0B1FEBAA1B5709078E090F21307DFC9A16A117038E5F3F2112467DBFE08A85D95FAAFD99C0CCEF5CB95CE099E3A7D1BEF0234853285271FFA365671060A4F6DBA320FB625E6A72D0D06AC12C0D050BBE14E7201C69BFF074432C89AAA91B802397494568E01EB6AB197B5AC6D2F3559A2FD97CBD37675F84EF382301ADA8C0E173D1E55624F22BDCD4DB6065D7A32704EF1687C8CA9BC3F9BB3A2B238C9EFD605BE3CE76EC4C9B2FF580BABAC04ED83A47EC403B112D1038210685B212F6EC0BD9B312F9205C5296709542B2CD72831960EBAD6D3CFA0ADE4255B659C2D2985A94FB2F2A139A64C4FB88428D5A6EB80242AD67C4BD82E72EFE740400DD22B1796F2967625FF2753E57EFB57ACEEEEAA07A4712E01EDD6C999B8E2315A900BD14F1B06CA15C5DC2A08532FD674D68FFBA62D49158393026417865E65E657A14597FBF884A8528DE60E2E0D0FB0265B40E7E85360D371F2EED22BB76754AD7A092FE93827F8BB162D430DDDB7B217C05A79CF7EA97569CC588180E37F73BA10D00AC752DE08891247B3F5CC1358588C6AE7F9F4F305A848B921588DAFEF5DF74F2100DFE432863536D15CD4ACCB2AF39B566BD8BB12EA0C8C17F84D936A97A175EF12C847DC255289F6EFCB47D9F6448D3B4FA21E5C8109C24524495D6F91FA0D7491FC14EA51A74C77DBBDC7B04B51683942CCE215A7553186430D7421ACE4262F3576E3C71A62EFCEE459B0C505EFEC6B5344FA11437F45FEFB16521591B814DDD79D3126509FE9C9B3E2B29E63B0024596655CBF9EE0E11DC2A738999ED00FEB5C3958DC0B631B45119FAD15BAC4E942A9115B26592C3A8523A95C3708321FBF235A2F7704306B7DCF1CE9692103D522F24C284BBFF3D6826B258DAD52628D8D33FF86FE34F29C25E06144CEAF9507E3817A52A0C57D32A25FCEE50DF4FB7CE22F5F3AD674EB6DA9288C20997F0CDD3575DC94CFBD56173DDF0F5D730AD4BF074D413426EEC6EC21A94E3577D4B12AE165A07DCA3BF22BF3A3096AA9DA127881FF746B5903908A73DAB1F9DF7A1F16A87E0CD90EFEDF3BB079A63BDC45DCBE4C38A25A14F258CB9282A29849DB88C708CCC4E953DE205D857D6FE177DFCFC95230349E841DEDCE24E3FAFFB848EA5FC1632183F10293545676A77A9CBD61B1C79ADB5C5D5F52B49E4F32A6D858CB5F52851647C81D1E5F705186E7A99BECBF500000000000000000000000A12161C242C" + , ExtMuVector + "ML-DSA-87" + 151 + True + "C2AB8038C46B135AA07098E5CC4168A670837D3878BC52B24CC5AFACEB488E1EBFB207207D8CA0415CB502F25F08AB7150B736EB694A35CD4D9B8E02F416029A07155C0DD309BA69F23D20D63F617A51888419347D5D85F24F252A3C0D6220AC962443E186FE559C81B85A6A2300253924E9BB7C269DDBD5B174EE5FF60126C0CC864D1C31515092910388111C4386D316601842701042712109711B8149D1C26112080118B861093560539081E12290A3324D603289248269D334099B842948924CA2360813C800121270D8B48018032CA0121048C42D00426418A23102252421296D92A27100B48190426100C3299AA26050B881941021081051E030624C1889D4947020B9715C482948982882B08C91000C10A30898142880A08552068D8A04415324060306056282405A104E63140650282D9AC468983211D1863014884C61042E11476214318ADA342903488E94808183B245E2400AE216660C94851236109B2008230242E3440962964C23C86D4B26310A45928806059B360D040086E006510A486593104D00358E94283219B72518282E0AB121E0906C61126A9B2626E19671420691CC185013462113324C1C90054224104B04001816848CA09099B42152A809509044904470011281843861182804E0404951B668191024D2B040120586DB94204B2284A244019C98811B158D5B042542408C908669A44470949228E29250504248004312DB1044D910621929321289801CC68010A2845110514A140A43A03121A871DC242ECA128183C845E4846C9190209282414B20851A088A504446A43442404889DCC48111C851DA22301AC010E38091641626601212D8246ACBA64980340CA4480219A94CE1028DD018511A096A92468812C30963388EA23811421626043306D9422C00C48D13342A14B21122450D5120420BB665A4022A84089284906983824DE1388E59B0808A32109CC80C1A2661D2102D44482D2005099B328A0400089AB4850C9010DC866CE100249B100688384050488249444E4C944504A391C840501B150280204E60C4805A8065D3B4481B165113C8000C380A094960C8082DD0126588004108A74018C36C518280C1B82C944006C1C63014232C0294210B95200332465446860C31902083508C800CDC407162480D04864D134982CA346E5C040048A87021B5211CB68C9C004912862C13064E21027023392DD092241B040CD43249E3C85064C240C0A65050B81009B7009BC2091C11909006484818886138640AC26D44108C8C18300A40124B9890989070931808530252810684A390402421882380301293012309291BA745D8262E8148661443490409326242852084858C2211E036710009689A946D82162113938841486A54960803032E1AB864C3380C633625D1202E63B4302121651A354844C26024436602207143368C8B22825216281B1126D4B26820183018A43122C589D9186E44002622414E433066E3A2019B483111442252868C0C4308D428111C0132919085414086E4B829542288142492C38651228691C4C01093948412070C6426020B2601CB420159488ACB422953324ACBB4911C3724C23640D2C84894C48923B2608914490B36091A016692B84912332690C8911285915834525B36491808328A2472CA204A53261209A48D23A950810664CAA66D941482CA444D89001143B030422832E2A65012B400DBC80961088020C4512005400B282CE414480AC08C4A306120C4515110529B0266C418302003501127061BB304C2C86C23A9809BB2684CB00800B7480A840C5B9050011991503266D91230014701CA8661E09421810631000652C2286124A305200964E0102010261061868009C640A0066483208001A22C1CC74CD998440B3151582289E2346E4412322115619C80251A08614012821A020C5BC64923052ACC10508A363048124E4C886C01B5091BA788CB3650C034909C482D10424A88382C12344559A0488208921BC670CC88819A200422478A939861800400DA148E50266AA3B6614A02459B1009E1C840E41680E39629D3A8905B326E43A204038251E3086919250004246452A2088104859122719B402613922443846C99B481031386CA322E1B266E58A62102420211860D90C48840100E13236E98320D483041E0487092424A18A6042382511896048A3088180889DB242008182E00C9488B9645F0738C2672FA24FB0F493286EE777C0AA428814994C07322341DDD11E6938E932390D3786ED3F2B23B3AB38FEDB1A144317B12C3F9E5E5BEF0CF921FB3C711CE739C2F4A10DE27B409EAC0CA85E80555BCE15011995DE017C7E4FACA100E2F1B1A0F2485248E5E230596441AD87620D2CC4C70ABCA187E9F0C116D61B8074BAE8FE2D164E899BE2B5621D8A7A07D11179E7A98C31B6F2D8BEB87B7C7066F10E7AF5ADC086F24B4417C25A969805AF6E2BFE48FC60DFE7E169748701AD8FFBBF31778EDF33DC81AFCCFE48CC5033E8D9CCB6FCD35B35E6A1259DF79A3F27454F319364E29DFC92B79A55D40E1AAB1802DE0B8FCE2CBDE7E7D641AA4B36DABF68B851693C59FBE18CA64FFDF3A5671EEEC952C8C82D9C8E07790FC44B957D4590B25BC02DB0816F507439225F14D56493F993A741C5901739A1A98C1062C23A3911B017601BCBF126B363E5D0FA665EA9FB4D34B70A7F08611FBA7AF4F2438A741C31AAECA0809ADF919F2A000DA0E0411323D773FBEEC24B07756F37C9C70A9E7DD76A7EA309AFDB0AB259686715052D3673262377F96FE38A543B40CF51DD8D8A4C4E6E3CD98EAFC08F8B383228BD7406F16A4DB483C571D6518BE9AF4FC34706C83351819E3C622D4E84B61923699335AF0080FCD5BA95E74BB8D33B6476FBFEAC97C167AF09242AD7E99778B9FDC86C9B4C2FE7AA2E0A57D186A9AB176089E069F9DD472652383D507244A81D87C99495E2787206196670C8FA799941BF1EDC81A8354E2CD2777B59A250C5524BDE7449AE0BC11A1D985615BD98B36494BC450C78A231E98289BF11E43313F121F22B5E34ED459D812BBFAFDE9A74A684509702E02FB048D6ABDADC167537D40E4F11B4F996C17C5DBF2919033FE9E00C99C536C70ECF539F3CB12B8AC33682A3405C1B9A984C07C6113A63ED143AB977E01696EEC625204BAEC4910C7AB585299A22E7298AB06FC880A126E2D5E01DD4DB2E915F327FED476FF95E8091469A5A5E89A00CB02800E5ACE958292F6290C0B1C1CAD512915B227CBC4BB958E4B098A40B2414F90620CFC83221B4A83A72D7FA9BD932D24FD58099352C7092F0CE95A1F797049812968D54ACB0765B716EB97D78503C46F8A94301D75DB577548BC8B6624074807F46AB8EEAE283E9386A0B8FB196E2DC8FB740C7C3B993FAF1464B442BC4E5C6FB39CC322603EC4E91EBBFE6974F7487CA167D1E0772B8A05DB7CE1742E85C00FE8C2CD35F6AC2ECDC2F18C6AC6B8B9A5A572123E284DDEBECCB9F8C3C46113FE8B4148E68EDE4288DB1784D1EE904A654083EDB82F9E071A602996FBD1FDA144767A463D6FACF600220648D1C0DC9124347129F0319015DC36BDF23C493A8031B4F2931D4AF8AA9A4FF7C90F28392C525F45E35C4C04CF85246E4333AE668B084A57AF4F080FEF93C0CC61C0D64B5690D3F99F5685DCBC42A4AC71F48968A1D6DB66D780C04E27CA5401AB23D8A7C7D7F8DA76D69748CFBA49DEC7548736B55724DCE1AB3B4523828B12BF040C834585E6B183154EEBDE8EB9573D03B0E5B22E0F0A6AA5096A0EF1888DE67BEEE1A52C4220ED798C0ABAD5E86A2D89582358AC296B15FF994CCC08F1A8EE8DBFB84484ED0E100236E826D98CD8DF6D0209CFB3A1E6719DC4432E2C9A1215180F72FF76100BBFBFA6670EB64C63AF31EE808152C8B45402B85681A7A0F3F507382B4A17582E542799D55D7E26461D862BB15BA6E04FF73E542FF377383D1D668BA21A0F06241625771FF57D82D41A661544AE5B654FB19A7E17A4CD53A25A7EE16407F5547D2FB7EA8479CF1110D9486D282ED2906829019C452C3EAE64828DCCAC6EDB11267B87FBEB0B59F51A6BEFFAF5ACA032495E4CB53021DDDDD8165D116547D4CD9DCCE492C4614C0102358AE258667D55DBB422B88A303852A0E0CA85D053ABF3F41D19DC8BB08D0B84D6C433C062D55B6A36AD34D9714790B6048A31416C1989208D964C2EE0FC4F24DCB6FC146FCC64F79640742E6D099091F266F8A48A5090A87A5573DF829CBFF09DFF23E4BB9B369FA4CDB524CDF3BFEE1C7BA6E02AE31199C762CC5B35E5F55A13DB20193F437FE80AD17DF5E2009089A23B5F19BF445656E72E94A4F33E710E5BABE9E5FBEBCD8C6C0D60473D1DF50EDF466A2705C64835BE9E57E554151ECEE5018FC29FA5F184B784192EAD8A4619BA16A20E2925D1D1C85404DDEB1F14F178648993056ACA0E43FDFF2ED95EF7779AC153EE9B1E61DE4C8D93432DC524D8C13D709E7294826315FE38AB66859EB8C81AD8FABF6FA75A189A8963977DB0C457E9CCBB54984B07E2638449C18997A976BCEBB907591C944C1D0C0BCEC1DC3F73AEE13F2E23125D2DEA7AFC36EA8D9D6E8C65342B5BAE295CFAC0DB1AEE40E99863F4B8D3DF99BB69DF9983DD1FC1CC24E213AB91778F9379FBAE001ADA48D7E1A2C6766F5D94E19C2CF0E41CC3133061EF2745C11D5CF5C93B3F182D41C48566BA9D2BFAB05325BDDD0704CBB9268069534BAAE5204A7CBC5C5DA2300A1AD3D531C230D924C47DAB8BE125A14D3D13F050F58CE8C63C0944D6ECDD348675C5C7E910263203CA3A1A75D3C9311C516691F0A8F342BCE49549B2B70BAF8E53958E66123996F236A0D432F482FC5DE63DCE8ADFED89CCAE6CC4D5BF82C714DBE38A63C9272439C63D798534C39519783BBE8234B510564BEB13E6F4A415939441ECFE7480FB9B32F781ADA1BBDE9BA87FDFC572B1FC6573D56E4D4761EC046C98AFE4C020244E119609AB7B7A537464C4DE449E2B87C37C84747522914B7A5970AD16127724009AA2117353A098F4C7CA206ACD6ADC511FCEBA92B73340F3225DAE1785E57F97F7130A4C4D950982439D9CC6617F8B46C8F81D921C4EAF69031E21139BC0A473A2FFD134A48AF56880EDF19079C6B86D8E74CC436652CFB630C91BD3321A7E50B1AA351C3E81F13F3DF81F16E53361CD8560D62300BB3A09735DBEA5A819C3DF5E34CA306C2AC29734508CB6CB7FC7DCF55C87E09ED6C6B87BDF00C84B9B93F57AC4E4840057AD5360A71C005B71A5A4E80CA1B9FCE26590A489F322BA5C30E7B1D2E18E33997CED864A05428237807137C318241DD951FD685E0B9908C4C1BD86F0CB5E4C4C5D91E4E95C9194573FAF3B6DB50A345A64B39480D70AAF590763D9202470592D78F7906E23616D6517C65D77556ED087411EF6732246949BC260059430316F91D5F9185DC71795F705CD9EDBA4C72F33FE9C08E5C1F64CCA074F292A711A80BAC0DD53BD1D82149EC470661FD63D6393FDCF01F2A946ADCBE0AEF6117B51D9915D8F186CD506D62D5008D66C28044D558B52001EC212714C59098F51D5060F8A510E9C30DE4B37D4A1BD0F8F9666380F75C5C34A31993B82814995D2BB7B2FDB0C35EB3B61F7D050AAB401A7D1A6E29776C1C90BE0BF36A8004CB37A817197BB5BEE7E56DDF08235F284F74A3224E35ED7495E6B4C9ABECF340AE50CC459FA77E646FAEC3BC451400209228B4C4A780AC1C5AEA00085DA44C75BE261B8434E8B84D4245B6FAF19983B96338EA4CBF1DD17D17E753EEC23AD80702088B4E01D4812AA1F8C72352B528C457D70E7FF19338A7152FED920A97D7D62F1E2C9F93C5A8FACF7F417E08B1CD94ADEDDA43AC8F68D5AACD31D61180EA69B36AB123E4E23BE1F380D3EF41BA3627424537F42C79D19EC316216CF602379B00F53E71F9E807190729E23210750631A20608AF474E50295311466F854AA8F2053F3ECBF2847CC0411F6AF4B132CF8E2444B059D8E8EA7FFB634CB7808D90BE3DDDB5269BE030658AD390FB9B74C6E40952DF6FAAFFAA1811E6BD621B3EADF77A566213F60885574D87C2B7A085E7CECAFAB3FF0B3737EB76C2A65907BCA040AE43667927E8BD53215DA42D5494AB76AF43EA96BC5841356B6F520A513AA1C8228F14AFA4091D02589456AE7DDCC47DD320073D1C8CC931A56F6D930372317A4FAECDBADFE551611BDDF1D84C4920F76C08CD75FDB9351BAD5210AFCA72CA72F29990A50217A0F89EDF36B5676761DD576D5F4ED387E002444C6E92B0FF6341C580DDD6530FEAC68C49B41CD8C57CBA44D34716F70A91B5CE7C579F832E21316694FC180C1D0C31A64FC5D23DF80E3DAC1033300A1C61F9CC22B8083922E7B7EC75348DD000B256024F4F77B3F1E5EFC8A08ACCF2FB58D8891911C8F2B5D1E666493EFF44926968C05263A4DBB1C1E9044B2983F3CC9D6137086CCD9F4D5024FB73077DC79A50A42D1453850BC2B9F04351AE5580DA750E66333902F9190E99598546627AD7C22DB7DC24D1F8E538D350179712F24B1CCEBF8535D2099E802B1DCA98A67A410FD8B158B31240F71C1751759852139A03D210C1C633263716426C1B4B7439E8CA004665045FF2BC90A65C566C04657DBAC89237217E689740BD2203717EDA2D05E9A41C078112F01A87D1D94491824882097DD05A72E9B78C92E17F0E1EB274C2991D3542864BED0A05ABFE0EE7217B07E3FEAFDDAA85276F27B650D5BD530387ABC568AB7169622C0F2276587E9A1EB1C6AA91A6B421C15813C583295AD45E38650D2C6A0667D37A3044D8142CA63B0FE942BF386ED59DDDF232CDD2E96D52CA59EDAD2018D04F3D30A9E1F5FC4A5293DFFE8F2BEF27D3A0FF48B49FDEE3C2B4EFE2A800BAD" + "F5F6B274070FA8CED053B71F1F9FA37CFAD56BC040D9F89F60FB565BC36E34017037B1108D6C97B7F98A63FBA0C16E20FCFEC8A3177981A60740E19EC6A25934" + "" + "2BEFF7FC531C100CCD447C941016866E76ADA7D820049488259BEE24927DE880F48CF336B230E87196AFB6B479312F98DE970C1EB693FB70B7AC6311754BCF050075AD1D9431C237B0544BEBEBEB478101E64B03DADCFF6DBE7E70675E2E40CDCF6217BE73D618F2E9FCA20F98D1C6E9DDB633C172993E1B110F394A5207F27579E7B07F6CEFCBDD82B6852024A073AC3F75C2BDDA00297DFE760F73E0BCB1904B91C76BF5FDACABD6EB278A271A6603740537D7BBD2D6BFD4201A8541334CC34DFC10C27F4CF31F5DC5C96BC8027A64BD0E982FB52013A1CC60D9343540EFF87B90916252D366D64559399A47EA51DF746038340F0566D10C261E22F0151D319C846C32C064F5BB9669F74189135DDDA548709775A4152564447E606572527D3CB9515379D8B49E5BE053225B3DEBF3827BB590DBA73B50BB53A67C5055AA3BE1459701E45C1B9E6CC1F9BDD4985970C68CC22B923F9131ED6829972A9FC7A8549C5E31DD97C9BEA8CCCFE34F692BE66A97D63CD432F010845D792AE12741BE1C2114288846CFC96B662C86C9DACA5A80B08D87FBC6F2C3606FE76C9F0B7565B4D0FA025FE4861DAA33961DEAD0A68B53AF8FE912BD4D11C994075A58D903811240560E0EB7AD9E10E7206A6B8442BD9437D4846E8C5A3C807831CD1627A4E776DDBECCAD5646FA86766211DC2DB56515A5032D948A1E9D3704725B3786BA3638B398D1301B723B4C5B8A3A04B3C255B9E62AFE0DA6D94BA8E53BBDF478AB35EBF308E2B84BFC1872360AD1476E13C5BD264C8B8B4FDC81CD6EE0EA0D893F0CEA2C395AAA14B090107AEBE749D04F25AE57096091BE98B3ABD5422534C1E1C1DB65B8302204CB4BA4DE7CB5E447C373179A5AF4E7B8A13579279EBF91C955EBD46261D5E0635D5C2E648A268010863A630EDA4D7F834BEA5C4385870B5EB5D239E83D831593A7E37FC384C24DF00A862240E69C01FE1E36A0C194BA579FDA06E4DC1DA8B6C020066FB4273520D6D2C3300768F6E696CDE46733580E5DD16B9418C4A6787AA5AB964386A735C9F2D78511DE53E006460748CE501915444056E3AD1E2FC55C0B2372F82D38EBA136C2088B0873C14C6F8C03D241F1EFA99A9B2339C672BD1ADD5EA959BFEEB01BB51B2C7E870D98C54BA4A5E903509AB1CDE32CAA50C1120AEC7DE5C63CBB5BE7A3AA97FDFD2E1511388F669EF8ABEE57285A1A746F9FAA7B310FA0324F76E2C3D9EAD55000C7001FF129493B7A0E4A1B149C94871B67BE7BDFD4D0C3DFB7DCD4943A6C584690A860FAB36A7AA1CF01B8E33486AC7C59240CEDEFE849F33AE9EB4ECE145FBEC8B0C1CA2D6DE1D9F54084E5CE775D4AB8DB278B9B896CC4605F7DEAD5B0AD25E22DD63CC48EBDD5B4D36BAB58521962F1D8C6D2F792AF96B4460D4FDAE1A46E235BCE5571385AA283EA5F2DF7A47B00834FE11C9AAF165FB378A0A64EE2F4D777DB0A45417B77B8D5D497F95672B4ADD7DA98815D1148902590BC27F5243092C8FCACD8F6918A58691983257582DB4B0017F58E6537061750875DACCE9A3618641B58E03E238F6F2D38C668752A0128D44EFDB846E208646B0E556C932827B37814160A614CB19FC51ABA8AFA5D064E2BAD048ADBE3C18E7955B1F101C8460F9981F5CAD22556D148D36BB6A7802ADF0563EFBE26976744D9A1DE4ECA311B73D3389845D3B0BA6CA5AE6B166E30FE199D3CF5DF1B44BAE2821D1CF54E21FF74FE36DFC91E3607E5E5207AC7ED474C1094975C3C83A3496C7028DCE07B674FEDA56B548C86B5EDBA50BF8BA9BF954FE06CC524970B644E92696F66B48FAFD5B31BE8DCAB022496A322AA2ACA1920268F3B614224C0D870DD59E4BB95C400189FA2B9ACD4E2984067F325364B60FCD1B42990AA665F7849B1A9C1CB8B08182FF747D15DC339D4303D526D243E8FDAF3D34062B22DF72F29B894072F43C204DCB7095DBA92B3D71FE6F7DA9BFE13863835F1E4ED03BFC88A4339EA2AF778DBEDEB61766A7A1FEB1C96DA8628C5F8F54DD75BFE1AA014D1E100BF12ECC3F1945584450AB9D7A74EFA3B804EE444B040FD7C5FE785E54D5640B8C18B13F8F36824AAB7176715E09C428A75AA9967B16A3BF9191AE5F73FCE4ED2D2AD7ED1A1F25FD693CED39FD7AA7AD3F7BF88865A6250FE50A7E5537D0B4CF45ED7C35A5933C72F6D5DD03CBFB58D1CB285FDCD6EDC57DA80C84F44AB9154902D1B693BA8358E172E9771663CC6B8F374916F4CDEFB8EF33F297EDBEF928496C7D26F98F4F96B7FA62A2FF1F3D4D98FE3B20E8660867D55EB5E60C374C9DBB3C1AAED656D606CB1A609ABF090D7F62EA4C08BEF832E19B5F327CB4B01C7FA3C0915EF3CF0E5B002E370FF4C6BC0693DCE6E9B999D19254B206655F767A48BE46DE6B1F3331F70F0E21B889DE390C2EFA43E98427FE05111CBE03B05B83A2CD5974C7A60773E37F5CEE08497EB2C685ADFAD2156273AB22971740B81906FDF12A134A3CA1DB7EFB856E23824AFDBFA3AEA78928C47A0F978530C0D75EF7B38F3A510294CE47E1C5B25C6CC40A3E40984266844B78EF2EF2D1119670141DBB8EF92CADA1FD72B59FA1D74AF82BC30BB20D338EAE371C0D30A9D162C31F09CF4F43909BD62A0CF8F4C528BFEC99796793C865E4965C328E4E8685EEB4D33B0BE2F25C453F9BEA11615744CE80FE68319A49229FA56C6C1C869684CAB447EBD1D1C08A94D324D704C4602A4B563092D6C636CAE27D17C00E7E8486FD7B86592A4A9D2EE93BBDCB564EDA2A1212BDA368A5F9AB7698C985D13043968DF9F1386CD151B3DC0A38A7C0835D1D6C3FD689ABD3562E3E22BA0B551E90CBE186DC8E54757DDC446E9B813FDCB7945E38D2C0BEC200DC5363AD73E5BE43385862AD2EE57D40B0A56F33D2BB38E21234DB51467E77DC133ABE0537A37310A7BF3D56CA6E73E28887C5EDA1E86B75DC785267CB9ACD2C0C6BD54DD12418087091D95D1E8E8372C58D5AE95B698856964CBDB8AFFF3D19E0AED06B1FE368AD5F293CD1B9BBB067EC993A5EED6A580B82E82B749B656B0F4A1AB9BC92F81CE08B6D9AFAABAF67F590A46FC2625E79B859A02133F0365F2D9EBD7EA241913ADEDD9F13674A69653E3C36D1AB33356F8A6C1967E81D4AA702F9D28AD2D68A62336278482B02E3A758C158D795CAB8CDF2604F8FF21A5A6699CC065FDB10BA9D1E4F46F8AE352F3B94EA4B7C4B5F6C6AB1440606FC153A07B558B53E8552DDFAAB0778A7C69CD83760D1EFADEC9878CB924A1DD4FFC23B558376C322D243458B11061598460011E2363C882F4AD992763D9678438651847AFBBFC540E274F173E0CFABD0D28C69D1CF7250544EF60FA032293660BF65E6CF39D8D5D26B9E6C7A654EEF1A9DCDEFB642FF56C189FC2B0A55A672FB421A46BAF463BC2C4336E3EC93CAED4F9CF593541122E4055E70D74B68AC85EBFED88EF015333514167537020A1F04A159D9E4B5F708516FA220CD19B6D42A733BA6DD677131D459C08495F85C974B7FFA5D3A2A6E3352C06E681D9778E580F8F638919AD904F68F4CD6DC869EB51258695AA858A8FD4250EFD36FEED39373B80137D19A5EBE88AB77E34D630C171EDFFD0CE649EA0991AF9108CC8C855165FC2C34E8BADF84FEAD48649A7249CAC4248F1C02111223C308B978F0E0D3C504613723D03FB00DC4A39B4343EC0627C6BD16DE006BD158E29F6CE63469DF8C3365F68E76DA2496653DEDC801B3CE1F0B8D8C27F71CADD6462E0298BBFE782E819D511EBC72CBFF516CDC761089B44F5EA7926DDCAA5501715C74D24B901834C7C6DE59239EB202B1EE365F152C4B0AB149691DCEE542B34F870408346D7BA325566E43B4DFD8B7C74294BB8571AF781D5C257BA7EE3489F3149441921D30E62ABFE3A5C5CB51200C587E2702452D5B3B1AF2A5E56B59970607A1EB91D9091935E5E5DBCEE0A56656595BBEA88DA24686EEDFBF58DC67FA78CE05B59B21D28FE68BC9C1E580293C597F7B3BBE700C121ACCEFE250467176AACDFF4E679E1F074525729C245B124F922575DB32F2A9E12F512AB421917289CEFCBC53BF6C5C36F58F04EC6545350AEC82F6851ECD811BC3870E7596CD5FD5A3DF3160E554A20F07D0A003B589ABF23BF03671325C1ED0B6472B2A29A47E38029387ADC62CF5193FA49DA2043CBA460037BE46A965D12E63DAF479FEF850A43D8E1E5DFDEC01B7D0D3ED0A224D27BA32C4A3E4CF196DEDF7DE08FB5F959AEE1EA34A9F2FA3D2A2565B438CEB0819CA5594314CEC9297DA8C8A2360A55D4EDDCF67B68E2D70BD9A8BDF3EC98AF11C4C21A861DD9E9D5BDD7A78542682BA7F5266B21EBF36B8626BC2BC4FBD3718BA00DACC6F67225C38CAAFB6BC79FA49F0298D9B90AD95CA741387E12569F2A3EDFA04285FBDB804C08318976D57D25C5522213AEF6C89BDE012628E9BFFC7EE67A47994EC02659E057E64CB028590ED120833386CCE5AD2EDB82E8D1EA5BD8C2E254F7FEBDBDD05298F667291B91F7E5E606E752DA5E7DB37216344DE2BF1D05BA2C040D829352A114E893C9927956944CDCB32DFA8DD6922D394F38E2A46EE4E8E59C3A71E1C7378150A19D7F50E3D00F5A7B49A61E8439B2C29758F421D56A588D588492C2A0621BBFD87F887F001E216EB3BEC22123D3FFF9704FEEC08C71189471139D312D20BA68532FDF384C4597EDD49C075A5E61C5DA5A288426A0D9495525BD4E1B9FFB07C545266B1FACC5BC424742C3AC443504EB69EC56179C8004C1B9DE16E1244A3A662B156E04AEA9025A2385727C6162A60540F6E2CA851F93692CBAAFB23C4CCD6EAA18325DE4310F997927E2C0D69EFC1E5C42E11329471356835688FA7EFDF6BB12AB5983062B4AF5F729FEF5443B6811EAF41E526151663742F3026C961CEB882D56B65EB8B62785E1DFF34F88B40B178BF5BB84D958EC1A980C9F7DB58D98D2D46901D4599DAF1D94162E5B83C0A473E9DE98F1D6CB290FC87B677A6376E6139A0395F61751262CB7BC136F96DB0FB1A0D3285031497BEBF49CDBCF74393CCE395E89225BD684DB9AED5623630BAF564E319EEBFFF943D92F6F9CEA9E792E180BFFE3F799DF825A48CA8DAE4A511E1834139644F3BCE31EB8BF05E957904A8208A4380157E44148AA6226B5FCCAF7183BD41045BCBDC0E263376D85418A260AA640C4735CEF48BA5E8DAA08042CFD402F9A0F42B6C13D58B6FA3C05CE8CFDD92F98FA281A158E6B942C61F967292D74A3C893BD725DA10D3053208F530F4330C5DF76B14D3451B5F7014DBD56EAA7BF3C2AD848857E17012D0BA0FFC4F92F106D0306B8840CA8A83912126285AE3F737D696BA267E60808C82108A78B1D6638E547B3DA1D2B8F6495C9E6FBBFB3C8BA66B5427A6BCF0AB70109BC4E26D6B4EA0F5FBCC75C0754D115F929B729F98C03EE585ADD062796554991EF3F5964752EFB03FDB2BEC4EB3C20732F2A355FD1BBC7247F2AA129A59C1936E6EDB78BCB4EA103F876492ACD4333310FA0794038D2027E99377EF33C5B188F8C89C2C790683BEE55CC0BB9C3E1BB0ADA6D3FB5DDF16DB5A3797D5818367586161A8D97FD746DF2455D609DDEE42AC6907A3F5F48ECAD3DD7EAC202A54007A287418CB8F15BE5B3014953CF5EB3A6E59FCC623CDED91C259A5A60A89B9CC2A86B7B88C7AC4CB4FD8FF2B773D4AF44388F644576D2EBEFC315FDE36076BC711F4803258AB493C516004D4AF110B9B66CBC7E120D6876D97A09668D5E4FB63452C06BE936EDAC134B4232BBA8B29710C19ADD644448338FEAA240AC23A0E1AA434377BDE5C5499243A89A196C97F67D76C8304C186D8CCE329E7281636B9880573160583D904F204EF96F825D0F382C8DF600C17868313194972256CC3C7BE3727B2D559F5FCCDB0A37F65E9B4EE9176592C2AADA40F41F93680DF6D85DADB0270E3595E4F717D2936C0A3AF68B2D2349A2AC24CB795E82B80EE2DF6CC05FBE5AB4EEF2BE089EF64D8E84282ADA273F3F79FF624F8823010F6228B78E499625CF9A110002511DE710C9010158625F1AB8A89D195FAD5DD02693555CC9F68EB7F4A8D9E0DE2AE90ECD37032370200D7AAECDDC6BA88EDE19A3265BDA88F97A25399DC34594B7D2D84AB00875C3DE01366F7CC7316CE9D832732D0696B49057668AF4A3A082C526F8D4F296ED1F03E196D4A8F9ABBF604547125D4BCA9DEF2AB417CB34A3A81931CC7A9B474939EA54EA3E53F74DCF114F583B5A2946577D2E87120762493BDDF9BBD03EFC2E5035862C8F5F6C44D25ABBE65B580DD3F880287CB2DC1878BB9BD6046BAB6C0B1F89B5094B42802F5EFEC6B9301FB85477F9AD219E5DFAFC1EA9E1D04681AF62EC9108FBAFED944623D562F41CDD07903E816257E7C179233E447BA9AFD0F117494A8A8C99ABC1105D69C439637A7FC0D5D7DCE4F5F802405C8A9DCDE1F606292C536EF4385E90C1C8CBCCDEF3FD25475889C10000000000000000000000000000000810141F272D373C" + , ExtMuVector + "ML-DSA-87" + 331 + False + "4B6CE73916E642641A4F515BAC5C66D2587A570AF8D2044270D54EF27D2C847A9C232B0605C2AF78CC3C05BD9F9DEDCE35F8BD9F0EADE9418637A71334FFC3D816AC02D910AD0667F4A5D0C04F49E082B4EE0F4AA24990AFB631283E34FC1E99F56F96E6D69AD62C875862E4CADDB993C238D2D139685E7575EBFB06F8DDE3B50085654A360C108765C9823010A96850200A09B700211442A1266008B388CAA08C21430E04101113C20120338DDC826053228410192009488A0139011BB41141482621494D81B0298A064C40C260D2168CC14422D9B04503488018A0459842062149200C02641B121022412ED9B20490C66C64B44024278861A80014C990108585040302DCA669A0120643482C53920808085000456E40060919C2092349680A368D82848598228902962149066D1C084C4A0402D0C000D94002DC902814B08408A868420006120861D2168844801153B085CB922D8124055B064D58448594C444D0846453A42524228554A4915A264C5C1086C3468612206011B56112313240888C4A3291D026890A940DC096241A456823B688D92630DBC471110872E4849010156EDCC6415926319A38868A822413B92CA4484619873013A004193584A03630232111E420314842446142696314510A056DD3420D498869E00282CA488D21168D4C042080266ED0206142320D24A640CA3051E18844821286D0944CC0880C4C38054B060861968410124E19922C18464014226CD3125203307020340A0B8810DA14500B012810B0290A3572C1460162062CC9B82DC8A05122410503852908074A5A860C1A962DE38428A2422494A02D62020601A108D1420153A229C1C46DD3483221166A1400308938041C386063A225129208DC000D81302D044889E20421611204CC046CA1226A13488A10476A23C36088C87001B56923B4291112920B25326332689AB04CD8C20C93924100376588A20DCA8440839801C9084DC0C288C90606E3A8509220719A924800B0890018064498201A3322E3384C091368204520A3140152465163842CCBA4511AB3200338510C32100181814C142E13206292866C433422E29208C9B40961C450C8148C09B66CE11648420222A082886320229C448C222712C3065094B605880445092369D0262A9A00214B4049138410D4163202130E9C882CD88044A3206DDC428464B6114912881A920450442A14401011238C2347299C46910A960DC3968CDB804D23B840D2863162A60C50288E9414101048840BC4245C82801C30828A32429492500A874911B724104325C8422D20B36020C18818064DDBB6094C4664C2002809B301C28471C10288049160183531D08005C04051D49665483601DC4865E3487204A089A3A00181B648989231C9122890202D5BC421102262910485DA3840188408DA449122450103078E032040E4322C148568E39665DB226519098ADA02495144269CB08448A86D13086A41446113B70DE036100C11808012328C2265A0A045501872003150812685194849412012100432D4B2482328400A0292D2128981384222482DCBA4001C08711A392C2124860A160022B569CBB60544120CE008612328729A444D03042DC4380590C66114A07089064D6428120A13129B006881C8414CB24C101740C0404D00860152308C943044020566D9204D9130310808321B960D834684C4120A41B060201410108308941805C40025E09884D1004208A205A2008D8BA421514226083769CB180654426021C925D10409A484314A200C99888509491124B180209991614648D0246C133426823842111584D9A268A482912130520A164DCA402E11456D48C225DC080222A92113394A09B44041924853B664DA44495A26060C1242944808A424299324248218300A348D64322513A190A240011B1820940650843085584649CAA809C10248A406840CB231E3922409A76C0CA8680B256EC918218AA0619124321CB9800B2512232624C3A881D3A44D91082C93268018A024224924DA486282060C9008429C46621C47924B3649D3C46420476863389012928404A80DC4182CC1A28D19367064128C899284094072D444259C90802224450CB94163484CDA902DC3144923C0051B3085000809E046905026510CA38D4188644C148211490A08109110A84520A90124186A18B5641A867151C460D080095A0446D124681B36915C40492B5E0EF07A11E1CB556039E2CEBA7BA8B9C353399B5E267BD91F6A25DEADB18F543C36A8E597BE36F6D88C75D810AADB19B7436AE309899F561439CC97A4F65331DBB737A398935C5C47D06E52F15394AA235C72D81283C063A3E6C14C2D9E0D355443D645827B8A07DA9D3B9DF49EF158630D6DDDA82C3F05EBFBD4359FA752A6280D61F3DB5DF226D5661653ADA02F5EC341506D117FB39365E0C2F3ECA47CF9B98780741759D3BC5EDDF27AECD11960356B15B8B2169C6400B7C388F28B00E8691B10802E6BCEF22B4297016BE40F89CD05AC45801564C580FC7ABBC2249AD43F59457FE083698FB4A88EFB996DFDAFEF0FA02853B76310F2AE13D6D77DF08B425AFBDE363F6F84529B47645EB3E9E669FEA21A1054D16B814E8712B75035110D5ED2E850E1992326C2794EE8B91401C76C78F29B6BE9337D75C2673FEB2D86A882E2ADDB65FC7ED14C0DBBEC4C01E339146988EBD6E689BFF4EC0B5361EE2B39F5DD504F9411321F227200A4870F7D46A9FB12B1202AC11914715A1FFAE19537CA1A552754213DB50C5C400864661C5C0E274863A31251EE5D8AC68B9A5C1B0CA63F03C542F922F3F902878BC0076A4F7759B59EE3800DCE029FFEEF5709221F26DA979977076B845C3259F2E7942DB2B1E7C7B377AF4ED7C8256719ACA3C9EE79A999F2FD3F7A4977B03B2C04F4DF00868EC208DDE1B9E64C46EEADDC03C374596F573CBA9863A6FFEE60C5F22D0565A63A4DCD037AD5899533BA3808F7BCB9A85109E88DFC84F82B76E0A2429E6E8B1E5522501ED4AC2B14D0ADEAB6953E1956DD49A2C88D76689C36229C89764395770E71DD58DC87A920BA8CCB9AAFA3C1794ACE4C8556D5F7C2D67678B781817678F9CE5619AD773E00D1F3CD0C28FF52366E6FFBFC2C453341A38CA0DC331C023977091DF6D0967DE437C9B0C9C04798390405C2AE028A93FD56A5C4F6493754A2B15453F2C39F71C5857A0756E61466D2E3AB1F4339DF4020B76FF1B0C363950B88414E8A5607E08B21E3BF7DA749FD5FDF4AA6B8ED5E43943E8052CB422585946F977D9AC37CF7D11210C9218A5764160D2AED79544677D1C5832A12B482973BF618DCEC5D6AF5C73DB0FF126E5724B3C8867B336EC236420E60BAA3AF36B1B2C32D27D28CB6D879F66A55B05A6A34943045813A9CBD262EC79B007DD37963A609808EA8F69F4C320946F6746FC65E659AC1C774F9205ADB08C15970A9B7A16ACB20A0A1B236FE81248E516AACCBB98B500CA425400912CCA8C0789CFB29C3674B766EDDE528119E5EB9C8363BAA8E42D537698B95D7CD08D8007A529A04AD452971B0FC5B176DEB613D6636290E025E058175D1ECF9DD0FFC122D8D10D6C1A0560E649299FA9D4F94C371A9DF1671870C0AD037108DB26871CDA9B9CDC4375C01BF91E1D18712509B75D9B1A05A87636AE952515B2BD2531BC948974DFB8EB940563C7EB71F0FE584A89BE0641A8F0D27AC7DE144CBCAC175790E5BBC541369A0585576059459BB2F32A556270A01E6899769B0C49A246539F839D41A509413FADFF9467B0219FA3E2D1A3D224404AA510D4D469544A8BAF6D562DB1ACD4D772FBF16E4B6FBBF5AC55A2BA1A81953859DA0D11827DE6A37FF2F38F2D0BDEB1575CE2C92B431B3361629E6C5F1AF24642A7D0FF9651AEBCAC0DACD1F7F8C19034E12E70350FC89170179FAABDDB4834D1C029D94844825C10A2040DF3CC247FE60FCEEC731DC5E7C932E020C905538C6BCF97B64E9250EDBFA785275359E8EBB196796EB708FF826E4FAF382D0BD15B8A4A87413E7EE5E66321AF6BE7273298FE67E3679904571EE55F4A748620FD7541EAA7C3D1046499FEF901E5870BF92B0284248F3CD3E1BBA3D0CFFC2B4EEEC1AB6F09865BE105ABE3F41E6D9D08DFA23FEA6476E2DE1100E3C4384CFE21DB7AB9B2FF5BF98A5BC2944BBA01B46D87CF2A4CCC7337F8882FF0D7898F9F0948D3E05859D7F355A4CB74964DAB276D32A1CA45EF61E1D200744B80C591F0D282386FBC2026FDE95BAE97D9FB5B9B5757C78D16CE6D599B9DAF1EFE16F21F9465342E25090516BD45DB7B4E6FEF79C1E0CD41675344BDAE19AE263686B77E1D2A6717FDC0F57E16B7B33D5DAA0B82EDA1B4BB910FE0B8BBAC1809D02681612E455B174CC59BA4EE88AD97DE936E31A1A2C69194BAD8A2DC489C01D4831C03DC36FAF18370FE7D78B7F0C216945300D22EB7DD8794736A7D3A2F28D86C7082B6E089A50C87A501939E5709686FDD2EFB764C546E962C161BAE37B7292322560F60EA95A6A1D83D028523B2F27B3DEB249FACF5AA95AF0C1913450036FA4DAEB3124ECD21C753775CDF9F974820843D3B2A5BA9430FB3DE5A528C42D1CF5829E0F9D76C68882BDF7D82E7A215C1E578B4BE338715772E11514C4B277EF0A43A2ED913612C09CA42A41FA68AE77ADD3792B2737034D69D84CA5D20CBC88970482BFFEC8848F1E8AFA051E6A3D79035ED583A62382007C6BE5D1F8DD506152E9C0943D9CAD1CF72EBC07FB74874877432DE374C500866D75AF1ADACAC71A8963A5ADB69E68FCAB2DA6D1D35C880F8CA4B3913EEE6308E3E47237B783A596787404D0D8049F277A17A38687ADB6514A8C7D9553D5C425416F14075FDFA63941CCFF85DA0D491779505100E9ACD817E492178E78746F28A4D2069F18F10F6CB374FD4981FB1BB098F066BB1CC1501B2FCFD3DF75521645412FAF5154B4B1973285035200BC35747AD83E057DC1059429A7192C953110222CD5C5F219D55B160A490AC98D4DD4297F1B5B9F68AE84FDD913B66A9A1056EDB1D2534F5BDC0F8A1D9F5F63E812913C7BD81C57071B1F08769CACFFCA2E9D867FF2B29FE50DE92C7A657542DE0CCFCD9C7B8FA605E0FBEF58CB165BB2DC505F015BA9F39FE10CFEC48C9E7E541685320D009D47C149AA64506F5C1C0F218C9529D6954A328FFBF7DBF43CB574190FDCB33477FB0B44FCD0A6BA14BA657CBB0AF439D64B84FFEE9FEAAD50C94FCE914CC5FB8DDAB8CCF6B08DB1D361B62351ECDAB2476ACEA0E3F5DD5C4D22913C1A3FE10A3761C42FDF39CD49FF0641D0EE8463A188C5E6E30A00D724CFD87D10BB576163ADFE83AB2C8A6BC3D9D70A81AC4DABC5E828C249C10C48F79B325E24B679FFAD23395A177D2F3BDE3D249392D0AF18F19D445C6AF288E87DD0F76445060D4617F778FB9CD3B201FAFB87D507A9EE84E822D9ACE3F7BB56CD6A85B33C9E83AE56D99AB270802A2CB32816B39675836576A298B9C8F009719B75190B09EDED6D6996C4A295EE4568F7ADB3F0EC6EE2F6AABA235DFEE6CAAD71D8353D31EA74EBB01F718F87A0831236FA7239E173A38CBCA77F042E69E616985733525C614320B78C0F04B266D3DC5806D444D1EB66433EF8CEE169CE7DB358A7CEFFC7FEBC33AEBA6B0BF2940BFFAE4576CAC3A84ED7A7A3E7853C51417E5017A5D8555F71FF75AC71AC44CC5B147746613EC1AA5556FF7B08D133DFD4D082AF8761EA406185C8B45AAD5C1EB7946B1A06CDE00708B17635A2F06FC0E7B45D90EC98E9735913C1F1C223C177FD1D53F139704D56F923D0A107BBBE0F05B5B6A4BCEE19BDC56AE6CD9F4F3EAF0425F0E6E018133DE9B1D5C94F380E321FD26563AD8156C9B5362FAE55F32B7AD4756AC8B80D1AF58C88D8BCBF35E1BE7D7DC09E02C87680A94158CF8CFA3726EB13E57AF13916F0A184E94D0C81688022B691D4F925451883525B05142F0ABEE8416A27A2CAA6F9708C1CF6738E0F3EECECB695B6159BA3902A23064638E4D47A8DFFD036D4EC0EDD4DE980C1B28613ED785A5EF80524C20AE6B4C8F8F53BA65B661A0A6178EF0B008B2D20268C1C38D9C56B1FB7C466E5B113709517D36341E8A18D67C4A70674E0BEE94FD48954C8D0BB2494F578CD05ACFC7313C9ADB836D9F22C566B5248B9B7349D04CC722F6F6697FFE085CB153AB7EBAB6B8D8741A0B9D7E32015D1090747D2BC5D0E73BA49BD860C3CC74C3CE838B13AE3BBEE2391C1492131172FEC746F7022AA26D680F56CE0CA8B91D34C7BB70ACC5F60A67FCDC439421796AC51BC7E67D1777C5EDF51C84B7AEC50E06AAA8C91A78F448AEE98B4ECC3A6D8BC9527D57BDEC4356D24D1672F9904636C25C91ADFF0669C9813EB468594A6768537E222E5AB01362C36CE52395CC2C23DEA3EA3BF1867DD6A78CDD21F6E0D410D4B80D309460218DA4AE070239E5BC0E2D4D13D44308D6766D317F44483E84EB22967F7E568901E3A39F53E8916CB9596F979765C53B855B67C45546F63B27BBFAA3E9ABBC61D6969AF88686BB7DFF76FAED2DC2FB4F305D28835839F3F6288C531FFDA62273870B0FE60888508C5D8CC0C1AEA80C48EB01BB5662C7B112151A4A11853374C333C21F48C0B05053E8198B14DB7F1290976F065ED5958E5405B349BA52C2D0555A01A54AA371FF3D941B6178895BC4D5D46339F5D62BB5ABCED8663038DC42492BEEDB8E5CAE9398E83A7DD83A21389878AFC3CF2537B7D6556B46D7C02D423CA9ECC1EF41C5317E688783CC6575B6C92F1CC0D2A4159EAF7E404209F875C382F11ADA2282726D2A06C4EA7BEC61A9D06EF02A35E7C4F322CB4750899E927B73CC454BCF3D24D76FA409D6A4B82083C6BFA38FC4A5F7C22388BFDFADF88CFB2AC9D07337F7C3246CB" + "1CF7A0F0D4ABDB6F8200DE6B356B82F46CF3677DA321BEFBF0DF8E49C7AFA48678407FA519534EEDAE92F661C539D03E2EC9E3EBA6AC429C9CB9442398890279" + "5B000AE26E572CA3A70C0B142224E75A8531EED4A2CA48F0B0E4013447203401" + "612B292C10779A6FD1D1414A8DC3A3A1ADD4B13FB18EA56D479A3E10733CC018F3BA25994127D2B84AB1936E9E0A7CD33ADFAEB76BE5B039B9199C6F915C296CEC7A6871845FEEF86C04E615D05833CD9A3058D2138F5456E79372FA7E7DF78B0C81CEB88D9B0387B8A1C717FC707C9148EE02F12024144147E82DEAF075DEF11D1D3FF16D78719121566FD03195C099DE3777B71FB2DD51E063FC85FD72824BDF8C25BD9FD18067F0470D7C7660FEE73D4D71FCF9310FE5E0512BFF0799486B3ADFF0996CC69758C22ACDEC29CB2E75F8399E2513C57263DDA5D2E2E0253958807444A042E97C4A3BDDF913320A582FA41143789E2DC6EE27ABF532AD06E501B44C8769B568CABA12707C16B3DDCD9A0B9C8232993BA5495F8A2D87EA886E25A739289512AE6C62F53101CC2EF21A7778BFF3F7B4591B698D84BB373761DBAA7A908C3297162B9C5AC72D302F1503B1EF62C8732507A7EB5E207892F20987D5F49E7CC98584BCFC8A0ED8855A07EDE9F7F6B0167EE0A88C8EC932756D769B319FF2E8A83175F4260A5C3D20394A26D8DA271C215783A8DB2B1811A04F7A3AED3AF55CA932E0E32BBDE0596885CF6A533C12BE49386255A0D69734B88573D1E5F9F79A68D723EA699FEE5859E9C3EB3A9E408B92E89871839F571F2DFE4549CF3C95533E14EDE4DF931ED4460FFC3B0D9B1C88EC321035CB055072B5A644783365836FB4A13DDF185CE80F7658991F61C1F99D6B4D4919F1889F385B6BD523C48C154FEAE1C531EEB1EE34569CAD4396AAE0540EC9AF3FC1476F8E4F1F29ABCB2557C8AF43A7EDC51F2460EE135317BE25826ACAA669D4552E03CE4A9F02D27ACBF600DCFADF705342221116B2F38EA4C540D3DFE16D74B0CAF3B1B954635C00EB981F33EDFB836F46607723EC8237EECF11C57DF8F7FD6CFC4819013909FF43D7996A3DB5BC5B217D440D4E455042D5658B4E27BF76DBDB7306E279C358B7A9D22BA18B7ADDA564D6F2489AF68F11090747DD1ED22C03FCB816F6BF2767D95CF1D1E6197087A7F872EA2A84AACD36CF5374AAC0DEBE7EAC3B7D63407CFE576BA13DDB837227E1D97FD05E50586857985F1F2EC18F61A36F39B77883483A8E2DBD07049DEE6C8214479610C8C9F013DAFEEDADA87444734B8321A6D1996C26199D79F1BA59022AE5CA29348456EF19AEDE6BEA6379EF8EC0147BC1494559FA2642E1E33225CD8BE28DBBB714364F7D9E5CACFD72A1B84B1A4A596A87A52025B7234C4BD69ADB1318BF6E49BB4BF172E725BF6DD08B3AE150E8C3EFB27F1ECB6248EC0E7F1B3C717AF5E3A9D04BFF37B93B6E51C8933FF0FFE09B6ECE237081586417E110811EE994E6F7164D9E264638E72B8E07398C41C323ECFD897852A0515E199048E8DB1D926AA17626CE8543A429C0895D5CAAF83A4C7A9C18B466FD77A617D8CA15899803E64EF265AB378D7027DDED4A8A0C4AB11F804955C1440A71BAD2496DD12807A01CF78AD0FF8FD495E9814FE11A22370852A0C85FE89EDF099D4C8B96F2CDB0751BDA7EB5AFC733F8FE3AD08343FA68D384497858CCEDBD7080156CDE68D826827DB12B83A5BB533AAAD5306F25FD1EB735D8EDC036BD339CDBEA44655EB384C8DB1D7DCE9DAE578275CA24FEDA5D0EB15179075EC05FF458B55E7FB1C1E9706AE9D83973DBF89FE59E8CABF0AFB792D88805CE5E2908B2EC6B8FD8790360DB6E3722AF0BB3049A95CF7FF5D4290006F6D7B077F7172B0D4BEB84282EA4AA6559788748BCEB16EACBFEFEC89436B7337551DF1450A679C15E7493263CB252C1F0288BE5B46253170C4870AF0197F8D387135E0A6C6E8513204FBDCBA477D3C9A3E7708A986A26215F3115F004992D12587A68901571FEF1DEE1560D7780348B3157F0E59524DA89F88DAE639EBB735DA6C807A56F25BB95F4AA4363A7F89429C221B106C6864847C744B0D44A4112724805827244BB06952AB995EECB749544866EB0EEA9AB34D54EFF252ADDE20B210BF14F7ED753615BFEB9D8293C2975319C38DC907EE48A206BA99356DF5BE324D6524E0D05F2ED29B17C15EB2BB334E0B98586A56BF9934973EE6879C6C1C5DDDDB01290217B4E1CD7468D704C739EF88A91EFB0A4DCA6894EDCF36067CC28E301E02DE233233C41110BF4EC710A2FC26AD47B0109581B78FB05D038786D8407DB96AD459DEE5BA1941C7340251FD2381D2EC7F911DF1AF8DB258CF144DD83D69EAEC22CF582B783F5757C8A9F15D20FB2CD27DF7A9E5E610B1F4FBB8019F82FC2EB9A27E85EE1F73B69BA6D4BA6313E906451978497DF208B72AA3630F096D74AF418A2E6E3FA5A4C25E2D97D501903724FE3346EF8FDCA42FD018BDC7B1FBEAD58B90B2CDEBE702A3CEAD2ED936BC3535607842FCB6DCFF7FD768AAE8322BD3C32E3F791378BCB77C63D87D112ECE628D570143A628C5A896113BD56B5C645DFABFD818926ECED4CAA6D2ECEAFC6C7B4DC95E3D11A1CF536A9D9B4D5FD410F08B0D1C011F01BD5F57EB8CD965A07673F9A5DE98702B460FD62CEABBFB509652520A5BB9B89ACC8F35EBE56463220E0C6B43B95C4C4B0C37492E72D20216B876CBA74A8E7C2B53FEF54E01BA7269A8AEADA4F2006DD279A38F24445C45B2083BDEBC9740BEA4601AC8F68E1FE1F1E7582E711EDBED1B4A2469BAD204FC38E8F4190529AAF9C0BBA9CE922ACFD7EEAD758625AE5AD2F16DEDC441CBA9E254378B4C85271C254C4996035BC8A0495D57B3ECCDD373ED2BD6DEDBB82F3AC637F697F7DA9ECA5ED2079258D7CA89DE2656352CC33FF176DB0CDEB1DE69C30A6E6527B686F5429BCACB296A1572C0674E2F6EDCCC98181EE07F76D15CC0744AE4ED1FD96E91E105E8D376FE97416834A018451D9E686439907E278BBAE676591AA67B5E072F0B39D2AAD4F897D4FC2A1A4DC31F22C598AE2A9FCBE82AF016D3CFCCD1A924A3EEB7364122599F8E86A36BD44E4EF2C61572DB8D2DD131810E7D5F1238F60D13286D314F7A826AE63F24DAADF1F6F4EC5C49541FB9F4D0FC7941BA6AF2C5B04589CE36F29D8B6C31FD747B3D1489DA004063BD381737BDB40376DCCE16D4B48B3D42EE7BD484F1343FF6E2FD3BC83C2A61DAC88A5E8351A94B26A51A289236DB4E16E3D5BE3CCBA00592664DAB25F299FD22FF870C5EB36C73336D5D15CC0435FA46BB98004958B61887060F147FFC09DE787F6E3D7B3C6767880016343778597580160C783265803088878DCFFB808C9DA851706BB4614122B5A530B55AE0CA2B8C276CFC00822EF2048AD946AD0C81352569D98977A2B83236C8DF7CC50BD4A1C57AD4D16CD225AAB2720D583593D65E6400610739612C00E58DEB8A464A3ECA51289C637F34D1511840F4DC45D11CC9C2A104FCD5F3E4384C2229E4108EF532726ED297EB25333466BA727BE652C76C62136480EE437FEE57E1A492045BD7D0EFBF841DAD6540D620F7854A3B379771FBAAEECFC0BFE9DFF266C654433BF51E2786845A2A92720AB20DC28D4B7D0FD937F3CD35E4447109F9D55CE52D203C821DF78D8807D656F47A26ABAFDFC19FAAFE9FD1FD0A23C72995E90EE17E13E8ED8A1998C4A5BB90BD899C1A737CD8524CD76CE559804E13E1471805F5EDEF4C1DB312C7B712935F54EEE80874519A35A5437B5C8010FAF8F2535AC26C8B13AD3398298F1F14BB0A71ED147618D8AF724B4DE41F72BEAAB27072FC4B150FAC4FF6E12ACAE8309AC652350A523FB54B816ACD2B4B9E670EE916A8CAC58021139FF2F5A47A9B9E3C00196D9A5A0BC9B79C542CEB82C02993BC645F9E6BCEAA6E01E799824D9119AE426F961ABD2132C4D4782D670DC3C241BC8AED847CC4012341EFA9113882DD455A1D0CEC6D971857C17A187BAC5E2719C467907BB888B8F73111282E9B703AC88FF509B992DDC0FA6FAE5D1B44AEB2F8798C8D93E2B76D1623FC85C9C20B76466EDCD920F843504D94C393017F14403D25386D988C9EA648A4A3E786E976A9EAFDB6493B40AC695AAB88CD89990714A4DE95AAF9836C36F3BCA919218BD2779968445F31E71AF2AF863AE9C21BC63E3D26243B3FAC925B9536CC338959499531E41CF36815EDBA8965F5DEF30D6D70421E2BEECD0ACFEAA5935E1F0699A2D05C1948B04B65EF4668A7E2C25F7B73D78298C86A56FB0BC088A255B19671BCE09BF238E69FB09D84A8513359DA87F35AEABCE8AA3D770B315CC34619ACE898BCE91F1D8304AAEAE350005ECD810F069E775399A600CAD68F53D01305A2E896441E37DC0410CF431DB61DD9BB608DF8D570CEA939557EAF6A74C620EC3007EA3EC5B0892CFAC774323A69BFDFDEC2279228041A32A2F843A027BCE265C46441A9330AFEF6B8BE2FDC8577BC2F195EFCA4FF02C02C54A748E74315692F9AF4FDAB7EF42A119D7EEB9F6DD648DE965F2D78F119EEA6E62852A6669267D572B7163A67E33A607EBF1AD09BFDAF9D4808A936E21A746D251D66DB86ADF72B89300925BB6EDD4868455F157613A1CE30ABD1C26D79B4D93FD9003CB4F4A3E994C69F2760DC521BC5D19B0FFF44AB4032F65AB5627F238E40D6DA327F56897C12598080FBDEE9C830914BB08B324D74F4439292E319B0B67FE634FDE10823B47E4413FC65ACC6BA581E1E452C580CE8EAD13DF8C3EC76DC5BCF742418DBD6B0C9803176EE4E0C6F849C93CA623C44055A95203E5344FA73948BB85790B7EBE10026E86043BEA93E16903FF1485964ACA76A1F4B6935B357ABB5E180C8117F82935A4A1A2EA5F8ED8E2A3D75AAA5FCC6A67C96A02BADB6B1AC2AAF369240B718117EFC8C8CF8FAABE3F3EC343148DEC8294B5A31B9D2EC2B9B38182B9BFCFBF0CC06169EC86849F64AF0B14704FFE142AB5217F224605EC5E4B9B4C7F59A14B6C1FDADEF35533B33BAD2398E2A61ADEFBED2E3EE6025AA2A573B0F125EC6E63DECE39C4C34E746F9165981D8A38F53CEE2908A874716953671C6AA7B06A1855AE6B62243CFF4844786C2F1F5D825EA746BD5F2C93A90DB32F56497707512BFCD5D79544DA19D0999570883774BE1317FC9679FC24C7C2E9125714C437D071984F88EEE976C05981959A4ACC6BD2C0D574B23057795CB8DF64893748AE29411AF6314A4CE9E0286812EECF0958714D9892A92E02ADF25DFACB6AB1EC02A83682AC75435BCF4ACBAEDA7D4840D4FD28983C1811BD1DACB26588B9120F0ED4A58269F153B8FA01238525EF544CBCF7F5FCE2AF2C3133E796B336BD951EB2A663D13D084B6B01F8C3B8F5DAB6F53DC090C202C066D2E13ECAB8E45053ECA2CE74AC936B55587C18B641D40649CB7EA0814CE495020AA35D1157395669CBB476C99171433696DF098F64AFBD51C21A67438F2D3796E4DCAEE07AF68B422969476E8DF327F94B669052B2935A4473B847CCD698F936C08DB5FF53E4B0AB55DFAB1DEADED56B624D861E019BF0B658489AFD1EECE990AA574E06068FF80C44D9CAAD4689E1DFA23C56981F5D4BEF9841C71EC70805C241E89A6C4948088691C3572BF20C91364CE37F0038F38A5B4AB47A6EC9863709397750745F1A8A427B61744553DEBD84451B84F3FABFE12185011E0BAE2275EB4079E927AD1FD0DB0999A5B24E8D3F5DE9974D7621094850789A5C2DE48802FDE6157F60EFD9DC7AEACB58B4004CD3AA9A9D430FED8C99FE05E9797B89092B5511D1BD15562F39A2B602A0C3A65D12683086DDBB4D5110A26F3FBCC2EE24200C9DA5D30DA39AAE9ACA70CC1C00CB335C9C3BF68E22BD1FEF1BC1256B79ACDD4CA8BCF5D788729F932E3CC793B475488FD204379D8FF5F92CC34641B6A40A404F26F0561C70C8E8196CC3870D7A551F521A2B9C7A5D778A592426E8FBFE0606AAD44D8DA6B237D1EF3ABF929D920A1040EF9BA599B1C0E9C239C293347AEECAD751134F2128B2651DC7EB253539B710DF120E6195E606EE79ACAB150829560D5003DDE113301AD097A3F5A5AD0BD9D9C30F7644665997E009A24AC414EE83843110E918D1503788D5254F38D03C273D82C3A26FA4B2458CDD837A1193E59BA1E6870E1F3DE6F86BDA7F324A3D75A8F0D73A2B6BE50D8D6322DCE308AD5451A3CF466DB178383659F726CF6601C59A7A9C48CEC0D398FCBF1697FB35F3661C4E65A438F138DB917D9E770799063E6E1D3FEC55A458CF602D456393DEDE66953CA6A92D77BE003A780E8CF00D23A3C02F0580EE7A7F4DCC41766F6B7164F9095893A840DAC93864B362D7568E917CA1BF232225E9CB768B731C90FD519A37D0CB1CFC306D215471DC23E78CE1440853CC6E2B98BA6EEA6FB63ABE99638600E8994D69C4E62ED548D9AB78007B35D97BA139A2D37F157FBA37EE323D289CD771F16C75C7B9B39286A3483105F1119C50E40C5A082330509AA8FE0E193E729E1329344346475B8AB6BE17607FAD212A2B5171878E959FC3F86DC5787FABDEE0FF4F5E696D85DB000000000000000000000000000000000000000000000000070C161A25272D33" + ] diff --git a/tests/PubKey/MLKEMSpec.hs b/tests/PubKey/MLKEMSpec.hs new file mode 100644 --- /dev/null +++ b/tests/PubKey/MLKEMSpec.hs @@ -0,0 +1,184 @@ +{-# LANGUAGE RankNTypes #-} +{-# LANGUAGE ScopedTypeVariables #-} + +-- | ML-KEM against NIST's ACVP vectors, and against itself. +-- +-- The vectors are the point: a round trip only says the two halves of one +-- implementation agree with each other, which they would even if both were +-- wrong in the same way. +module PubKey.MLKEMSpec (spec) where + +import qualified Data.ByteArray as B +import Data.ByteArray.Encoding (Base (Base16), convertFromBase) +import qualified Data.ByteString as BS +import Data.Word (Word8) +import Data.Proxy (Proxy (..)) +import Test.Hspec +import Test.Hspec.QuickCheck (prop) + +import Crypto.Error +import Crypto.PubKey.MLKEM + +import Imports () +import PubKey.MLKEMVectors + +hex :: String -> BS.ByteString +hex s = case convertFromBase Base16 (BS.pack (map (fromIntegral . fromEnum) s)) of + Left e -> error ("bad hex in a test vector: " ++ e) + Right b -> b + +-- | Run an action for whichever parameter set a vector names. The set is a +-- type, so there is no way to pass it as a value; this is the one place that +-- turns the name back into one. +withSet + :: String + -> (forall p. MLKEM p => Proxy p -> r) + -> r +withSet "ML-KEM-512" k = k (Proxy :: Proxy MLKEM512) +withSet "ML-KEM-768" k = k (Proxy :: Proxy MLKEM768) +withSet "ML-KEM-1024" k = k (Proxy :: Proxy MLKEM1024) +withSet s _ = error ("unknown parameter set in a test vector: " ++ s) + +spec :: Spec +spec = do + describe "ACVP keyGen" $ + mapM_ keyGenCase keyGenVectors + describe "ACVP encapsulation" $ + mapM_ encapCase encapVectors + describe "ACVP decapsulation" $ + mapM_ decapCase decapVectors + describe "ACVP encapsulation key check (FIPS 203 7.2)" $ + mapM_ (checkCase "encapsulation key" encapsulationKeyOf) ekCheckVectors + describe "ACVP decapsulation key check (FIPS 203 7.3)" $ + mapM_ (checkCase "decapsulation key" decapsulationKeyOf) dkCheckVectors + describe "the seed a key pair came from" $ do + seedKeeps "ML-KEM-512" (Proxy :: Proxy MLKEM512) + seedKeeps "ML-KEM-768" (Proxy :: Proxy MLKEM768) + seedKeeps "ML-KEM-1024" (Proxy :: Proxy MLKEM1024) + describe "round trip" $ do + roundTrip "ML-KEM-512" (Proxy :: Proxy MLKEM512) + roundTrip "ML-KEM-768" (Proxy :: Proxy MLKEM768) + roundTrip "ML-KEM-1024" (Proxy :: Proxy MLKEM1024) + describe "a ciphertext that was not meant for this key" $ do + implicitRejection "ML-KEM-512" (Proxy :: Proxy MLKEM512) + implicitRejection "ML-KEM-768" (Proxy :: Proxy MLKEM768) + implicitRejection "ML-KEM-1024" (Proxy :: Proxy MLKEM1024) + +keyGenCase :: KeyGenVector -> Spec +keyGenCase v = + it (kgSet v ++ " tcId " ++ show (kgId v)) $ + withSet (kgSet v) $ \p -> + case keyPairFromSeed p (hex (kgD v) `BS.append` hex (kgZ v)) of + CryptoFailed e -> expectationFailure (show e) + CryptoPassed (ek, dk) -> do + B.convert ek `shouldBe` hex (kgEk v) + B.convert dk `shouldBe` hex (kgDk v) + +encapCase :: EncapVector -> Spec +encapCase v = + it (enSet v ++ " tcId " ++ show (enId v)) $ + withSet (enSet v) $ \(p :: Proxy p) -> + case encapsulationKey (hex (enEk v)) :: CryptoFailable (EncapsulationKey p) of + CryptoFailed e -> expectationFailure (show e) + CryptoPassed ek -> case encapsulateWith p ek (B.convert (hex (enM v))) of + CryptoFailed e -> expectationFailure (show e) + CryptoPassed (ct, ss) -> do + B.convert ct `shouldBe` hex (enC v) + B.convert ss `shouldBe` hex (enK v) + +decapCase :: DecapVector -> Spec +decapCase v = + it (deSet v ++ " tcId " ++ show (deId v)) $ + withSet (deSet v) $ \(p :: Proxy p) -> + case ( decapsulationKey (hex (deDk v)) :: CryptoFailable (DecapsulationKey p) + , ciphertext (hex (deC v)) :: CryptoFailable (Ciphertext p) + ) of + (CryptoPassed dk, CryptoPassed ct) -> + case decapsulate p dk ct of + CryptoPassed ss -> B.convert ss `shouldBe` hex (deK v) + CryptoFailed e -> expectationFailure (show e) + (CryptoFailed e, _) -> expectationFailure (show e) + (_, CryptoFailed e) -> expectationFailure (show e) + +-- | The two key checks, each given a key the vector says to accept and one it +-- says to refuse. A constructor that accepted everything would pass the +-- first and fail the second. +checkCase + :: String + -> (forall p. MLKEM p => Proxy p -> BS.ByteString -> Bool) + -> KeyCheckVector + -> Spec +checkCase what accepts v = + it (ckSet v ++ " tcId " ++ show (ckId v) ++ verdict) $ + withSet (ckSet v) (\p -> accepts p (hex (ckKey v))) `shouldBe` ckPasses v + where + verdict + | ckPasses v = " (a sound " ++ what ++ ")" + | otherwise = " (" ++ what ++ " the standard refuses)" + +encapsulationKeyOf :: MLKEM p => Proxy p -> BS.ByteString -> Bool +encapsulationKeyOf (_ :: Proxy p) bs = + case encapsulationKey bs :: CryptoFailable (EncapsulationKey p) of + CryptoPassed _ -> True + CryptoFailed _ -> False + +decapsulationKeyOf :: MLKEM p => Proxy p -> BS.ByteString -> Bool +decapsulationKeyOf (_ :: Proxy p) bs = + case decapsulationKey bs :: CryptoFailable (DecapsulationKey p) of + CryptoPassed _ -> True + CryptoFailed _ -> False + +-- The seed and the coins are drawn as lists of bytes and padded to the +-- lengths the entry points want; there is no Arbitrary ByteString in scope +-- and one is not worth adding for this. +-- The seed generateKeyPairAndSeed hands back has to be the one the pair +-- was derived from: expanding it again has to give that very pair, not +-- merely some pair. Two generated pairs also have to differ. +seedKeeps :: MLKEM p => String -> Proxy p -> Spec +seedKeeps name p = + it (name ++ ": the seed comes back and rebuilds the pair") $ do + (ek, dk, seed) <- generateKeyPairAndSeed p + B.length seed `shouldBe` seedSize + case keyPairFromSeed p seed of + CryptoPassed (ek', dk') -> do + (B.convert ek' :: BS.ByteString) `shouldBe` B.convert ek + (B.convert dk' :: BS.ByteString) `shouldBe` B.convert dk + CryptoFailed e -> expectationFailure (show e) + (ek2, _, _) <- generateKeyPairAndSeed p + (B.convert ek2 :: BS.ByteString) `shouldNotBe` B.convert ek + +roundTrip :: MLKEM p => String -> Proxy p -> Spec +roundTrip name p = + prop (name ++ ": the two sides agree") $ \(seedBytes :: [Word8]) coinBytes -> + let pad n bs = BS.take n (bs `BS.append` BS.replicate n 0) + d = pad 64 (BS.pack seedBytes) + m = pad 32 (BS.pack coinBytes) + in case keyPairFromSeed p d of + CryptoFailed e -> error (show e) + CryptoPassed (ek, dk) -> case encapsulateWith p ek (B.convert m) of + CryptoFailed e -> error (show e) + CryptoPassed (ct, ss) -> + (B.convert <$> decapsulate p dk ct) + == CryptoPassed (B.convert ss :: BS.ByteString) + +-- | Decapsulating a ciphertext made for a different key answers something, +-- and that something is not the other key pair's secret. ML-KEM rejects +-- implicitly, so there is no error to look for -- only a secret that does not +-- match, which is what a caller would see. +implicitRejection :: MLKEM p => String -> Proxy p -> Spec +implicitRejection name p = + it (name ++ ": answers a secret that does not match") $ do + let seedA = BS.replicate 64 7 + seedB = BS.replicate 64 9 + m = BS.replicate 32 3 + case (keyPairFromSeed p seedA, keyPairFromSeed p seedB) of + (CryptoPassed (ekA, _), CryptoPassed (_, dkB)) -> + case encapsulateWith p ekA (B.convert m) of + CryptoPassed (ct, ss) -> + case decapsulate p dkB ct of + CryptoPassed ss' -> + (B.convert ss' :: BS.ByteString) + `shouldNotBe` B.convert ss + CryptoFailed e -> expectationFailure (show e) + CryptoFailed e -> expectationFailure (show e) + _ -> expectationFailure "could not derive the two key pairs" diff --git a/tests/PubKey/MLKEMVectors.hs b/tests/PubKey/MLKEMVectors.hs new file mode 100644 --- /dev/null +++ b/tests/PubKey/MLKEMVectors.hs @@ -0,0 +1,195 @@ +-- | ACVP test vectors for ML-KEM, from NIST's ACVP-Server. +-- +-- Generated from gen-val/json-files/ML-KEM-{keyGen,encapDecap}-FIPS203, one +-- vector per parameter set per operation, and for the key checks one that +-- must be accepted and one that must be refused. They are here rather than +-- downloaded when the suite runs, so that it needs no network and a release +-- tarball carries what it was tested against. +module PubKey.MLKEMVectors ( + KeyGenVector (..), + EncapVector (..), + DecapVector (..), + KeyCheckVector (..), + keyGenVectors, + encapVectors, + decapVectors, + ekCheckVectors, + dkCheckVectors, +) where + +-- | The two seed halves in, the key pair out. +data KeyGenVector = KeyGenVector + { kgSet :: String + , kgId :: Int + , kgD :: String + , kgZ :: String + , kgEk :: String + , kgDk :: String + } + +-- | An encapsulation key and the message in, ciphertext and secret out. +data EncapVector = EncapVector + { enSet :: String + , enId :: Int + , enEk :: String + , enM :: String + , enC :: String + , enK :: String + } + +-- | A decapsulation key and a ciphertext in, the secret out. +data DecapVector = DecapVector + { deSet :: String + , deId :: Int + , deDk :: String + , deC :: String + , deK :: String + } + +-- | A key that the checks of FIPS 203 7.2 and 7.3 must accept or refuse. +data KeyCheckVector = KeyCheckVector + { ckSet :: String + , ckId :: Int + , ckKey :: String + , ckPasses :: Bool + } + +keyGenVectors :: [KeyGenVector] +keyGenVectors = + [ KeyGenVector + "ML-KEM-512" + 1 + "47B893474672BA92E4B12EE44FB32953AF8E8503B5FB471D1614FB8A021A660A" + "1F8CB39E9E30BC458A0DC5408884B1187FB217018DF760FA57317703B844A0A9" + "28266A088B3482439BCA01AFB7CA5C6136A979B5159985A9484B36B679A5F7B9819EB63577891F7BB9CB98413CCC434ADC79A16D6AB3076569CE6291C59B5D64612A7FB0C15013200BC8BEBB03A570174B5E4363AED86EB02A220D281FB5457F0A549FC5051D49A6B2015259A2C3084F405E1769952260675586A584904059275A265234EF3ABF88C171A80898FC783358BBC9803C8789027D917C9EBACBC568CC18DE84C85454B94249586C0C6E2B8A16FA789C51212DD1728EE9B8C6C40528BF93826FA82368419623032AF27B5694305816811D3CA85805100E9C1A9621E5089E54CB47F5A8FEA0B49EF81C6B5187F48924C7947D6B61697A4A8A18452EF803336AD4BE503275BCACC03C181405F7B1DC9B47FB169EB37BBE27E29C763A4E52B9A42520388CF09B8EDBCDF41CCF6537190E6156C37CC1AAC63C0F90CE78D0B9B190C548D71B6F26CC8F585EA14004B5B30AAA100B2ADC1263828833B24E46163B41446F98C882092A39941867B80632E2097674A793935227DB0B8577E03A69C50A514C7473C892E3FBA7C4316BDABC952A70644176687D4191323BAD93D85A3CA250868C0747E6C44F6126C874AFBEC0BDD4503CB2C59A69816E7D4109941467579A1FFE6A4F50FA379051729DAB6E2F61432F15BE67D667C7CC1054742B2B953078A5CF88D9133087309D88C61DA240D99C59137329907B47865321ECD5564E987333B4CB607B0AFCA86769DC95B2F921357213FCB80C3B152918E9BAB2228C0A1B77897AC68CE55088165F87F397DA9790873B62C5383C0CCC370F0267CBE195651CCF336182C22AC3924B76C9E779B7A271D166B6D24B84242B7E73CC723F764039F6C851744034C3304DB0C091A5764FDC9D593556FF734B82A87CCBC38CA99564D988BBD2D1BF071BB160722D365104FB27610651A8ED817F2742A6B5A1273A61ACAF4460B0AB1456A9922351400A1C7D95D856D6E3370622C9C4164BC6B401435624A98B95CAEB274F34CE92038D785068CDD8CF44C38D84ACB2C466A2756C870EE78C26E738CC451002304EB8C90AB24B6463EB124D779F937A2E3692611D2E34D57B36CC4B2CD3B31FF485C6684D408B972E0D5CA7D2224AAE4E" + "89C31D05611AAAB258F78BC2DE0A80D5914BF80C376A990D33CB97F4F2077CE12D2DAC559DE3400B0622754A2E814730BB7C2B401A076CEC9524654DDAC661B2F2123EA64D3BD727CA42C8D2725475BC0AD4B698E61001D031105897CA746249D24538CD63849E874EB9449FBAE979CA2A3F357B7D87F112FB16AB8BA20BDD315688231B21A4083277663113BC70A806E772917FA95743A01F138BC7BD5CB3B21CBF8EE301EB0BCE71A4BAAC3907B469CBEDE767A55AA194930B4A2B227E633CF33A3FC715454A32873B717FE6A21C01989133E42113D985F46807A26B0DE105CD9A897334D9816075C3149A1919CC2DBED56A23F3192CDB68F6E305852B8B864BACB5349CFD98419FFC9486C6B07F788CBC79B7680441B0DB1DBD92AEDFF49DB6F2AE52B95FA9B966C6CC08714A131770903859B823D2CF9731A36D8795B60585F8F1BF1A2C4E544407D5C1359810BF99A7374511A663EC1B192B83E27C4F4B6C3387692E4D756A235C14D151056E829FB79936A0C8A52EE128F0F369F8EB7390783788037CE32B021E57C45EB968CD5B9A04B02F89149778653F208C14C2E786D4544749572BF0D9A31B13BBAD60B6756B9379FC2E1660C58ED373C957309EA213213A5988CC59B1C37153811F7271BC1E729AB1629714D1955F706796556213B08E3F766867D1CC02648BE0574BB8818D0978578C562FFB6795DC065037129C5E4738B5960FE7E2A22570C36571A4A7A0A5187609F8D27714CA299309509C7C8AC2776E6814A262359B971B1A153CAB81858D96350F9F2CCC9884C8F5509E237655B4B2B22E75C6919A82EB56B095E3C52B55583F6855E186CB75DA3A20640762922062721D5C1AC51B87B10633BF43860814C89CE16BBF66667C2101BD27BA80969C78DC487678989EF5AB22D9B42679590E1855C94A9A90B4E021333A4532C7AE238358646094B598AA431486D0B7BBD82C5C4EBB6195680C3E5AA806B155AF7004318867FE55B1996353495034DF459624F4B04B51167A5148499846A36C605AD85B997674DF241D8EC59E56E22CCDB342F535292AA444B69C3E7A252128266A088B3482439BCA01AFB7CA5C6136A979B5159985A9484B36B679A5F7B9819EB63577891F7BB9CB98413CCC434ADC79A16D6AB3076569CE6291C59B5D64612A7FB0C15013200BC8BEBB03A570174B5E4363AED86EB02A220D281FB5457F0A549FC5051D49A6B2015259A2C3084F405E1769952260675586A584904059275A265234EF3ABF88C171A80898FC783358BBC9803C8789027D917C9EBACBC568CC18DE84C85454B94249586C0C6E2B8A16FA789C51212DD1728EE9B8C6C40528BF93826FA82368419623032AF27B5694305816811D3CA85805100E9C1A9621E5089E54CB47F5A8FEA0B49EF81C6B5187F48924C7947D6B61697A4A8A18452EF803336AD4BE503275BCACC03C181405F7B1DC9B47FB169EB37BBE27E29C763A4E52B9A42520388CF09B8EDBCDF41CCF6537190E6156C37CC1AAC63C0F90CE78D0B9B190C548D71B6F26CC8F585EA14004B5B30AAA100B2ADC1263828833B24E46163B41446F98C882092A39941867B80632E2097674A793935227DB0B8577E03A69C50A514C7473C892E3FBA7C4316BDABC952A70644176687D4191323BAD93D85A3CA250868C0747E6C44F6126C874AFBEC0BDD4503CB2C59A69816E7D4109941467579A1FFE6A4F50FA379051729DAB6E2F61432F15BE67D667C7CC1054742B2B953078A5CF88D9133087309D88C61DA240D99C59137329907B47865321ECD5564E987333B4CB607B0AFCA86769DC95B2F921357213FCB80C3B152918E9BAB2228C0A1B77897AC68CE55088165F87F397DA9790873B62C5383C0CCC370F0267CBE195651CCF336182C22AC3924B76C9E779B7A271D166B6D24B84242B7E73CC723F764039F6C851744034C3304DB0C091A5764FDC9D593556FF734B82A87CCBC38CA99564D988BBD2D1BF071BB160722D365104FB27610651A8ED817F2742A6B5A1273A61ACAF4460B0AB1456A9922351400A1C7D95D856D6E3370622C9C4164BC6B401435624A98B95CAEB274F34CE92038D785068CDD8CF44C38D84ACB2C466A2756C870EE78C26E738CC451002304EB8C90AB24B6463EB124D779F937A2E3692611D2E34D57B36CC4B2CD3B31FF485C6684D408B972E0D5CA7D2224AAE4E3A389831056ED8FD81476869245782689C84B3CE90FE6A9E78D0A380FD6A15731F8CB39E9E30BC458A0DC5408884B1187FB217018DF760FA57317703B844A0A9" + , KeyGenVector + "ML-KEM-768" + 26 + "E582B7D75E6C80B05AE392A1FC9F7153B12390FD99930368CC67A768BAEBC8A0" + "1CDACB8740C0B87C4A379575F187B367CBFA3B300BF591B109F79816E9CBE8F0" + "28C793778741B80B02B4339F2AA4347255B099F17264E1B8CC0A2C7C2A1A79F7997B907FD0496C6E6C8AD7714F5F339D75F11F625591A869BE1175AE47F05FD4313468232BA6957D7807B824F445AC99A0D568AB1AD54DCA8249D1482E61275F52248C77F61A4248753188CD1794CD0A465EC0DC4B025985C461B74E76286E4C37E77405695CC9FD0654374B427A20343AEC0FF1A187768273BFC4905472A1DA387F14559D6CE87313F6A5B6138434539F9A13684055B177E543F8B40F432ABD7CC49989A50A9084C660913F45A8593B17499BC4CF936C2BC1851421CB986808A0EF30AFE97AAB5B8B8EB3F0B3506A95B91563A0E57DB7231044987EF141BDAB3537C316AD16F17805A81F29329879A94E96157E4B7447F7D59603B21BD896CC47B7CD4E232322EB9C5D2215696BCFFCA3A04EFCC4C5D9CC39AC9A6E8700D38C244B0169E7FA1FE81B4B10365E74E6A1F7F756D11ACDC84043F81006D62995376C22535958FEB53F78117EE0F61C4C862640D06DC57A2B8BE62A41A642AF3BC63F6BAC98BBBBFF70570F37B8F8D9572F2735657A6C98F96CAF57A849868720B2640B8BB2732237A1F984C18872D10289CE43C952C9257E06529AEB76AFD127B17596FD25C5216C9CABD9B18EFC50E87BBB04568BB7D5C4E9288C006483AF5912E19108573700BD10CD77224B80659EA75AA74270B33AC4008B738BFEE271E78658C8742FF13C96AD0781A03C7576CA26DD58B52980BA58C0505E446AFA140CDCEA0490DB1F9B18815D4314B2459CACC562441C91F4084E5426C88E632CF7482E79907911D06473260835D7B85E7856A829AEA0381707B939CE86882CC09C4448C6AE94A9C303107C5667EEFB8DF7763CC21189A3C590C40AA51F491503A7935EC08F4FC300CBE607ED8C9100C29FBF45584B13C8D780069337AEC76C36CEB70373E2AB6E7B934B466F53FB32EAF040055496B8540E23A2A277E534468608D5EC0F8D38CEA5BBB806C1BF4F164F6AC826FE733F95461E29DCC11200C0AADA1B8332023EAB329718CE25CC0A09555903F3578BBC863B1752CA94365DA556DF54C3B7E05CBB7115FBC1B6C57A172C31B9906560C8FB54F3C563A2256CC073243B8179B4A28D60E086CF51082EE429272996F0AABE03BA0EAFD3C8E7D954BD0933E2F60ED0C32CEDE7B820A28E48F3CA3C40913CCCAE2337ABFC59843F08C9863325D65A4E9E15C1F46172B118B2B5EB0F1D5158A00134F27B085488C3A0621FE4E5678698250FB74EE5152E3E35A66544A05D279EA99131FBC15165060B90F88EEB7B20892A4DE4CB1683495BD7DA037966B47CC040F1764C5DEB06B5499D4267391CEBBB47F734D8539E39528436A1858182854BF20B1F93279AFB706464C65CCC5AE099B37CC03556C26ABF4C3F8B9BA3A936707211A49A59B268F5284F7970C77612719450377417428C4BA47C9CA115CF95304C4759C5D8859B44985C06A6C924689237BA320D610960D61C53E85431789E67A40113F167FF93429C264F6CABC95448C903437D39A6577BE0CF0012852AA476351A9046A110A1A625A3D74C910B78BCE9CFCA735E4F91B8A4C57DBE489E849446098AACF73070AEE638FCC8896473D3C159D3AFB4B687B40DFBF371A9C2644B605187B71A14BC4C8678FE8247" + "3808B98D9A093C7853B0B814D1CA5F392677D3D0A38F81C852F95B9A69B374A24588C0ADD5B510BE567C5A24688EC91ED0F28BB4C86978C09793795C36B94E5DD8498EB4353FF40EA87B17E921B5B4CBA08E5B7BE5A9C8BC69AC5AC3075DD947D04B8695097AC39790A0C8ABE11200C9F136ED7B0C10077E36111C1F139D9BD27142993F5883925C4918413CB8043962A28567397A1C7E003BAC30644055155CE9464E623FC5E2160CD1143FD2C90C03A08D0B333FBB40A308CACD81618674B406809790B2538D30431F064F65EBBA7C41B2A53146A3ABCA66754CAE591534045C1D640D6308472EBCA660A8A7D372470D3869115B8860B577311980B2C66B8352229ABC17B0C31F58E5A5EBDBCA6E3B157D9CA79EE890778A4CF28429C2A82C80DC9C524B0864E385E355BD4E65732D395E464070F61828E57C3E1BE2A50052C4FF1951F5D0C89720987D8941F085871113A91F518FC79B7189056990B2447BF54FC170C932E0A4E0F7A262542F56D1B49EEBC392106D93C77DE9186DFBB824E73B2F7FC796B312875B43414877AFB356214492C19C748DD381BC7A9237BF097CC606A03119A240A5536AC7844F6A79D0E2B821B68D96542450179BCCA231F2DA68C5EB6D118B99A1B66EAA566C78A9009008B0D66155BA4839F8E518C650177DEC170EB09E8BF6A89905320590C642B801CF5B5151C2AF3E7271CB9961C65A5BD17479429A31BE9081F2767D94816C16D1B04772012382B689AB2D3FB9BDC66547BFEB23052600BB369771494D9C914EC93A066ABC9611940B947310DA04197312425D736A1933B785C95DC430791E42CD691C79BA63BE06EC80765C9C07053EAC706697F21720C672B9803DF9935532197B0485BBE42B0DD16561AC0605E9DA73C189C9E29ABA3AECA19C21621B209326418543C44B88843EB3720A2CEC3DB52C59D6507EE5CB03C268B31FC4BF115695EB0B8AD3F11FEDEA1A5D0724A5D5284D0C10E9844D0D32BB5735448421BC5317650BEAC4829B234B787339812F2316ACE8C22C42D6346214934FC0B49ECCBDF8DA19AEE64DE9628C4F7AB3E512A399B0C227B677AC4A69891A6874F6641D2CC95460B751E17E434C924B2947D806856665696E3CC4DB9553B81606C31C3800B48756A073BF685EEA20899C176769E902DB971827D153B9C33516E959C6B469841946C0D921A371A5DE9740C6AB9CA272B5850BA8753CA023A460EBF7BD573C0745F40B90F105EE17C19B6832E019B80F8858BB515F7C709E68F29C1375B22567AFE7B528F4431A94B553DF825CDEB84B4EA9296EAB9AD66271EEC5AEF6F79509A20A182279FDF92F87FB6E7509968D22CA750B5056974841B9654E5716BDC33A2C6A116A4117FA757A1D22710668412B8878E134B70FE32158A2317FBC62F01371296AA42E33C903E0C439F19684B11E2F911F7FB79860C8800F9CA146EABE29DB07237AABB503A9BE2AA7263A0626C162A27537775792B2E7B0FD347929934AD8F521D4159059F611312AB903879490059BE8B38920E2CB4A256B8B35783E909346D13E9888BEB9350369F7C1A8501331110C651621A616B365A026D1CC47DF440C9E650DD0C0BFC8295439114528C793778741B80B02B4339F2AA4347255B099F17264E1B8CC0A2C7C2A1A79F7997B907FD0496C6E6C8AD7714F5F339D75F11F625591A869BE1175AE47F05FD4313468232BA6957D7807B824F445AC99A0D568AB1AD54DCA8249D1482E61275F52248C77F61A4248753188CD1794CD0A465EC0DC4B025985C461B74E76286E4C37E77405695CC9FD0654374B427A20343AEC0FF1A187768273BFC4905472A1DA387F14559D6CE87313F6A5B6138434539F9A13684055B177E543F8B40F432ABD7CC49989A50A9084C660913F45A8593B17499BC4CF936C2BC1851421CB986808A0EF30AFE97AAB5B8B8EB3F0B3506A95B91563A0E57DB7231044987EF141BDAB3537C316AD16F17805A81F29329879A94E96157E4B7447F7D59603B21BD896CC47B7CD4E232322EB9C5D2215696BCFFCA3A04EFCC4C5D9CC39AC9A6E8700D38C244B0169E7FA1FE81B4B10365E74E6A1F7F756D11ACDC84043F81006D62995376C22535958FEB53F78117EE0F61C4C862640D06DC57A2B8BE62A41A642AF3BC63F6BAC98BBBBFF70570F37B8F8D9572F2735657A6C98F96CAF57A849868720B2640B8BB2732237A1F984C18872D10289CE43C952C9257E06529AEB76AFD127B17596FD25C5216C9CABD9B18EFC50E87BBB04568BB7D5C4E9288C006483AF5912E19108573700BD10CD77224B80659EA75AA74270B33AC4008B738BFEE271E78658C8742FF13C96AD0781A03C7576CA26DD58B52980BA58C0505E446AFA140CDCEA0490DB1F9B18815D4314B2459CACC562441C91F4084E5426C88E632CF7482E79907911D06473260835D7B85E7856A829AEA0381707B939CE86882CC09C4448C6AE94A9C303107C5667EEFB8DF7763CC21189A3C590C40AA51F491503A7935EC08F4FC300CBE607ED8C9100C29FBF45584B13C8D780069337AEC76C36CEB70373E2AB6E7B934B466F53FB32EAF040055496B8540E23A2A277E534468608D5EC0F8D38CEA5BBB806C1BF4F164F6AC826FE733F95461E29DCC11200C0AADA1B8332023EAB329718CE25CC0A09555903F3578BBC863B1752CA94365DA556DF54C3B7E05CBB7115FBC1B6C57A172C31B9906560C8FB54F3C563A2256CC073243B8179B4A28D60E086CF51082EE429272996F0AABE03BA0EAFD3C8E7D954BD0933E2F60ED0C32CEDE7B820A28E48F3CA3C40913CCCAE2337ABFC59843F08C9863325D65A4E9E15C1F46172B118B2B5EB0F1D5158A00134F27B085488C3A0621FE4E5678698250FB74EE5152E3E35A66544A05D279EA99131FBC15165060B90F88EEB7B20892A4DE4CB1683495BD7DA037966B47CC040F1764C5DEB06B5499D4267391CEBBB47F734D8539E39528436A1858182854BF20B1F93279AFB706464C65CCC5AE099B37CC03556C26ABF4C3F8B9BA3A936707211A49A59B268F5284F7970C77612719450377417428C4BA47C9CA115CF95304C4759C5D8859B44985C06A6C924689237BA320D610960D61C53E85431789E67A40113F167FF93429C264F6CABC95448C903437D39A6577BE0CF0012852AA476351A9046A110A1A625A3D74C910B78BCE9CFCA735E4F91B8A4C57DBE489E849446098AACF73070AEE638FCC8896473D3C159D3AFB4B687B40DFBF371A9C2644B605187B71A14BC4C8678FE824781E66EF5A7A221619F6A64039CC369843E10DF5C859F6959CC3FD8E5272330FD1CDACB8740C0B87C4A379575F187B367CBFA3B300BF591B109F79816E9CBE8F0" + , KeyGenVector + "ML-KEM-1024" + 51 + "F3A706FAF090C03DB506863AB0B20BD8A1627956318E88C67EB875E8E7266009" + "35D2BC43DD1CC879F765BF2A0C5E297889DDE910E57E2BB0EAE417B90AB7A275" + "8D0923CA8A2DA2B4146EC25321122B8A5AA8AFE0C03415273008A46EE83031E98AAAA125ABC75D3B30322560C197E75DD0E48A348099F7B2144D7B8A8660A4A97BCF19C0583BD9BB2123033CD7BB5A14B08B817831A673A28170F5F6443C0551913A327CBA18C3A053C4040250403B70AB9588832403AA0FC37665E04980FE1602E7D2715D9CBC00515DF432A4F5B32B3BC92AE3F31700166D498123E94576509B712B18491B1435EE7AB7AEB1AD30D72348C3CC083ABE24A8B12097BF32F792476288EECC3BF630ADCDAC6ACA7950D9839501A448500742BAE37F109203A809B2B960A307E25347A32C3EAB79288173A878789B296E9E8C1C28C5BB3AC472601C9765F7B77225A810C7B85370BEF4A5B079D2015ADA54236B8F33840675F9B2EB427A1B5974CD5C61B24010886C5A7BDA5BBED974AF7217F3338AD719CB308A8BCB1B6D6ED2A1643736C29095E8A8452A3A36C7BB5AE58CBFDC61529466A90F454ED6895B0861083DD1371999B2F559A3A487CF59A074FB49215EA6A6BE656F9AF17B121A2447CB7985590E9738842B899BA57AC311810AE2D9794F37483DD6BCCB64AF6D56588AE94665961C025C3AA2861974C236BCA4BD8FF5509F7AB774593E7C5549E57C2F18D15C0515094AD9A0DFAA0601E524F8231156B627BB25A0DAE04DACD0A66C041CEF400583FC13BAE640291A39A5C5CA8BCA1AD5C683CDD8290891A76940817DD8C9F52678780548E37A05806600801426DBB950C3B2BA34E24CC77864DD91B39F1408C716A69DF63342854E50A245FB50977B9410DED2C93F86B1F9D5A78B87BF81E51CA620A7E8566B19AB700964A40E3266415228D432156E5CBDF52364A90483A55C39B3FB16FC7465A3F8AC801B70B9FB28B583444BA5C1A73722D417A9D6D9B7DEB08BC6B330FF27CF61AB8831E27758C64AF3B12150CB7B33ABC29858106D63686D8762459ABF9413850AE53ED6313F76F83D0FB8AB34374E7DF693E4A1B3E5A8AD0CE820AFE1CF401ACDE650A8101B0946022D52178E19613C42B88B07CC04EAA81DFB28AC9DC076236B67219A30F8F945DD57BD2F335C52D59372308D38993467DB53DA3382B74867B616481BD0091A2232C1116DC88A589DB9107224A681008C67C589186A6929549BEEF92253DB02B0C8AA9F9C875A670266C72BCBDB4F5625043703C1A0457395832E4C335180462ED2220C59E7361903C107D85457F6CD82EB820D0855D97675C2E0151CDB73C2885DDB7849D74541580124E890116A65BC068093B57914E20C937C60A3EB25576F1A976A9583839B672144CD4A45C3477A45C29B4E0BC2BDBD206585C9B7A7741C8B6B5793A92797A15AE7A5B73A74B2971463634BA52AA792AF05530730B6D0A89A346156B733677932BD36593A7496130CC458DCC5CA987C21960604EC8A8C5396056680CBF3F1AAC4F401AA5029FB2150434BB4706C31A2D54E4297939FA7C9C6F85700613CEB65C7F03AC56EB86E2D27CCC6DCB7B9394DCDB942FF222D86958A996C0CB6A8A44F97A70441C95FA71250116EEC20863C0B5A643458788AB001F8869D909922F51EE547A1E889255B3A0599C65842E5AB8D73872F053BC62392EA53896D328102D460BF1609583C22C3B43780EC6DAD0319EB4A5A65B4756C3CB40EAA935183BF8BD46ABE76BA46E199103A5313C3235F49C915E097BCA804DE680781D8365731BEAC6789A9203FB8787C4C070E00A13A6722A66A28236DB179825653D33CCF898B72C6B8450D97AFD3276BB13340519CBEDA708D12A858F54C49F4547195B7788A9150B2649E36AA394121926D568A488B16D3557B2A32AF57D11FC3373F80A28C0723273D362502E7C428AB44D3CBABF9EA585FD1BD0C9846556A1E196B78CF951592984A0A8487A78C2317D7ACA4118E1049750A0788F0D66AD9E48E34731130ABA0B427360A856D96D80B3F028FDD3ABA9035C10106BA1C0934BED36C6D7C7434249654EA89FC22137F4AB903653B75FB25B6F01635E6CC7D39CF1508690562826B49B6FFC59E0DD35022E541F8BA0D304AA5B4E20606907C424395666C54ABC2B8FB009847C86317685000C231215C8C15945860F6A85DDB98A8C3A527F2749D3C027E694E8F0B0F0FA454913AADB635AADD452F7128BF7752569669A8B93290EB92E78F6ADFF23E89F57F3890753B51F12F3F3A8A654E677847" + "6CBBB491D7B3A3439F53C9027BD727A0E1C78004B4CDB66760E50F1BF97FFF0139F82862F1F3178A077B6AF77735CB802E91316BC207B9F1485FF7A868891E003BB9C6558BE6052C0826AF1CB5836B94C7D5D714EEFC7CFD983E9884B71348499DCA05DEF33B49D1C14BBBB95AB195BD72219B18BD0C704275CC94AB06653BBB9E0EEC74181903A0081AAE068B81EBAE8925335B8134B1480A60F758282C2F011B909641266774513A19090BC672EB3A0708CB577F134151C155C1E560D9B335A35A4D75F223502752CD250C9A7778EFC76A52709813A24679B987A6172F1714641EA6902D364F96357E92B44275EBCCE7F6207BB0A215E98B3AA92DAA46A39AD9132F3874C6239391E716BA600E42BC7084425051F865756A89DAB46A76104E183A12031B99BE86257B583C2E88104FEC3BFCAA50F238BEEC20ABF31219685885F2C3CE2F80C09B874E73A8712D35B8EE819D84B1C59A37947C0B9E25F066EE030ED28172FD5914B562408BA9B9DE9BAB2B95A4F30C22ECE458E133371A693D9458058265A75AD6CA5707617B38A1B6F396FB199CC1C77100741DE37500F1BBB10B9204EEB7C3F5B91A1CC91B6D4072AF410957358ACB0625488B9F6A9915758B7CC9B4AFE7D5AD6A95231A7A711B3A8996E23AFB3695B45B6D919C771D7685D90A7550A9B5B1E7538A365451A5A815009E49326C6541CAF0BA685056BF82EA4AB504161B555568583802800554A9480602437B0B9677FAC638A2508154A658E0623C62666E078D21A5A545A818D66C9547B9749458B735A4AD51812950E8A625A1B2A753ABBBA16415006300AD32D34C8B0B3B2AF4D1CFBDE08B4F06C70DE4BA28FB3763745CCE133DD2E488F89A51E7A586A86AC928867B0541438AF5645DE3211AA94ACA4A1095896EBC363C374A85565C1EBA030B5EA21B9B8A140E131ED07580D5B8590446988DA65939A823A84661B26CA12461AFE373519F43C13987A96A9069CD6B9177B205B25B5197DBCDE66C6457892C1DF96FE50AB7EF3B3DAB0A87436B3451476C44AA22AEF918AE5A1A8AE46920263032E3BC5DEBCFEE43B8AA290CEDF06966939EE6CC7C6AF83CB9D90CC889AB8B5A3D4C224E73916AE057BF5455528CACBED31B554043C6B0EA196854461D2644461470EFC55080E326FED49D62B5A4BB06B14F505EB4486C9A1B033B927836DA656C2475E5B8B823986DCA99B298512D94B77FE0588DBB96BA53A67369B82C2BEAB2CCC0103E549317A99DD97B598FA0A506184EB55C96B17302533C7EC3334E3CEC2BBB22BE0F89442840884FAB4A766A41BC8061D4ABCAC5190C729309AC6151BD434B818BC1E7D0CD435C9CA6DC7182228FF93867134A5CC65C6167C4A8BB830E7FEB863C15BF87B259EB0518FF5248792A625F37C589589799746D36197DC63AABD134A00EF1B49C79A2DF67B28E6B6E989CC9D0577EA5AA1B2FE64B322C7A8B241D503523D875CCDC479D28E3C01BD8383DCB2B684281E6949171F518EA97009237B5CD42AC8FC8A571716EB784B336529DB67728499988B73B80EF5C6D33F3B42ADA7A0A36A7C6A8A0DFB804B3D7BB05803957B913827A16F52ACDD7650EE2971A85DC21F6A94A5B31988E4438A9C1733AB8C48C32653FE800BF81BA66658CD2001ACF3A83E3A0426AB6C3B2600A53621277A43D723AB96758B120F120248C15EDF4294FC38F96EC490BD52094655841D9355038A379D8B666875171F39218E21D37023F5108B6D15389F07C1241E15071C43CD7D160EB5341B719C2055424EF6CBF66815F84943C9481CD1DE685A74C5B9DB37A0713952FD4A67BEC8CA7496F6C9C5FB84A8EAFE68D54654C6BAC9C3AB712C2BC32C0CAB072C18DE3CABBD9224DFAA305DFDB3F85019D7B988577B40F27514822DB42D87489502C53CCE6A095B486D9143D6D160B1C7313BF288913657698117EB145C0C9C55D66C15BF1A2B268F6A5FA259139551ABFE3468F1CC49C01B4232543DEF0BF9AE503B998536310751A8BC302B4B4440B835233797C967E7F9308155B27BA63C65E08BDD1946828196B30C042CDA0B56E4008215AC51F373604F5930BB66297C32BE41A162CA21471CC86657B5CB5307D89091DA9A5137F1C1433AB8AC1D59DE421C7C17A883EAB99BA4714F841998D0923CA8A2DA2B4146EC25321122B8A5AA8AFE0C03415273008A46EE83031E98AAAA125ABC75D3B30322560C197E75DD0E48A348099F7B2144D7B8A8660A4A97BCF19C0583BD9BB2123033CD7BB5A14B08B817831A673A28170F5F6443C0551913A327CBA18C3A053C4040250403B70AB9588832403AA0FC37665E04980FE1602E7D2715D9CBC00515DF432A4F5B32B3BC92AE3F31700166D498123E94576509B712B18491B1435EE7AB7AEB1AD30D72348C3CC083ABE24A8B12097BF32F792476288EECC3BF630ADCDAC6ACA7950D9839501A448500742BAE37F109203A809B2B960A307E25347A32C3EAB79288173A878789B296E9E8C1C28C5BB3AC472601C9765F7B77225A810C7B85370BEF4A5B079D2015ADA54236B8F33840675F9B2EB427A1B5974CD5C61B24010886C5A7BDA5BBED974AF7217F3338AD719CB308A8BCB1B6D6ED2A1643736C29095E8A8452A3A36C7BB5AE58CBFDC61529466A90F454ED6895B0861083DD1371999B2F559A3A487CF59A074FB49215EA6A6BE656F9AF17B121A2447CB7985590E9738842B899BA57AC311810AE2D9794F37483DD6BCCB64AF6D56588AE94665961C025C3AA2861974C236BCA4BD8FF5509F7AB774593E7C5549E57C2F18D15C0515094AD9A0DFAA0601E524F8231156B627BB25A0DAE04DACD0A66C041CEF400583FC13BAE640291A39A5C5CA8BCA1AD5C683CDD8290891A76940817DD8C9F52678780548E37A05806600801426DBB950C3B2BA34E24CC77864DD91B39F1408C716A69DF63342854E50A245FB50977B9410DED2C93F86B1F9D5A78B87BF81E51CA620A7E8566B19AB700964A40E3266415228D432156E5CBDF52364A90483A55C39B3FB16FC7465A3F8AC801B70B9FB28B583444BA5C1A73722D417A9D6D9B7DEB08BC6B330FF27CF61AB8831E27758C64AF3B12150CB7B33ABC29858106D63686D8762459ABF9413850AE53ED6313F76F83D0FB8AB34374E7DF693E4A1B3E5A8AD0CE820AFE1CF401ACDE650A8101B0946022D52178E19613C42B88B07CC04EAA81DFB28AC9DC076236B67219A30F8F945DD57BD2F335C52D59372308D38993467DB53DA3382B74867B616481BD0091A2232C1116DC88A589DB9107224A681008C67C589186A6929549BEEF92253DB02B0C8AA9F9C875A670266C72BCBDB4F5625043703C1A0457395832E4C335180462ED2220C59E7361903C107D85457F6CD82EB820D0855D97675C2E0151CDB73C2885DDB7849D74541580124E890116A65BC068093B57914E20C937C60A3EB25576F1A976A9583839B672144CD4A45C3477A45C29B4E0BC2BDBD206585C9B7A7741C8B6B5793A92797A15AE7A5B73A74B2971463634BA52AA792AF05530730B6D0A89A346156B733677932BD36593A7496130CC458DCC5CA987C21960604EC8A8C5396056680CBF3F1AAC4F401AA5029FB2150434BB4706C31A2D54E4297939FA7C9C6F85700613CEB65C7F03AC56EB86E2D27CCC6DCB7B9394DCDB942FF222D86958A996C0CB6A8A44F97A70441C95FA71250116EEC20863C0B5A643458788AB001F8869D909922F51EE547A1E889255B3A0599C65842E5AB8D73872F053BC62392EA53896D328102D460BF1609583C22C3B43780EC6DAD0319EB4A5A65B4756C3CB40EAA935183BF8BD46ABE76BA46E199103A5313C3235F49C915E097BCA804DE680781D8365731BEAC6789A9203FB8787C4C070E00A13A6722A66A28236DB179825653D33CCF898B72C6B8450D97AFD3276BB13340519CBEDA708D12A858F54C49F4547195B7788A9150B2649E36AA394121926D568A488B16D3557B2A32AF57D11FC3373F80A28C0723273D362502E7C428AB44D3CBABF9EA585FD1BD0C9846556A1E196B78CF951592984A0A8487A78C2317D7ACA4118E1049750A0788F0D66AD9E48E34731130ABA0B427360A856D96D80B3F028FDD3ABA9035C10106BA1C0934BED36C6D7C7434249654EA89FC22137F4AB903653B75FB25B6F01635E6CC7D39CF1508690562826B49B6FFC59E0DD35022E541F8BA0D304AA5B4E20606907C424395666C54ABC2B8FB009847C86317685000C231215C8C15945860F6A85DDB98A8C3A527F2749D3C027E694E8F0B0F0FA454913AADB635AADD452F7128BF7752569669A8B93290EB92E78F6ADFF23E89F57F3890753B51F12F3F3A8A654E6778479370FE5B05DDC92C939F62CBDE4C0FEA36F45CD20C5748CF3AC891A4C260449635D2BC43DD1CC879F765BF2A0C5E297889DDE910E57E2BB0EAE417B90AB7A275" + ] + +encapVectors :: [EncapVector] +encapVectors = + [ EncapVector + "ML-KEM-512" + 1 + "17E5129B2029F3281987D6624725B64C51CF8DCA3562372BACB7AE15FA9F2FF6AB47659B7D305B61F55F571315FF69AA49E1388100319A650E86C59BA3024C3DEC83D4AAAB661C452F6CB6A8D2638C133C599045329CCC8F677D24683DF1146EE1C7318C3763A47ACFF81A03927B9BAC5A49DC285C4EE204C4D72DBB1C97FE53C622E621338669FCFAC1E30A36F0A9769BBB2BA408787EE629CEC383F29AAAABD46E22133F339C08E29B82C4FAAF0F676E1C2B04377975DC3A3B246488EDD1636C58ABCD65B8198B6FA8A8475EF541DAF34B5ADB9DB0589D8958B62A88930EEB1C0C352B1E57BC7882C89EFA318457813F2B55B96881E7F75C3DC97357681E54553AE099095AC38B34C00199952602C481C0F1CF74B550CE7C4F6C5876AFF076DD921942DB377E1749F0306F77443A2F058D785854C7E32B67F49BCB99A5E18AAB607C649B892B4DA7CC7AF31557149F02B19460FB49E5051A7251ADE0083CDC1B0B0CA9633220C2B3C532FA2CBC0DC6CFFCD4455E0005D06CAFC727C778375CB67AC461B3627A653C843BAEAEF866C7F67746262FF6D76536255C89045A172CBFAA123D6ECA3E56FC922F437A14F78D1ADB54B3E54E8DD530450B8500E15A54C97471CB26AB437394480282BA6B7A786C28857BAF387725BB4342B4C3ABE4CBF90612B04C8BDC1599C2D4C66B80B1A1941440040A42AABE03E1CCB0B4190780B6603BB69EC199B063826BEAA5447C88427C6BD7A06E4164A490B955152979B8F5BC9417598AF01BCC371577189E1775165B9610B9F75AEEA277018B1C2C04391115B6F353AFEECBAA18AA4001DB94FCDC5F46EA520E9836BC5A4576B58C59FC3F75634946D8CC1B1081CE93ABA8566304169332FA316D4BA675A086AD87937963511E4A1845EA38689B564E11295672098CF01D5D4BAB0338121128A682B048E6AA31B83C65FF861E123010433A3C7607AE5E90C438067F637B44ADF8798CD0980EC83BDAEB4D427A8F8D88C519E52542F07734A645A2A5BD4D6521A4B64A96A10F35002780169B35E35F01CA74FE6207B8B475EBC079647CC10AEC3C29683E071D87B82ABBDD6D369E326E475325ED5AE7ED232B37F49388C06A740D421204" + "DCACFE4DE1C115DA106ACD1EEFEAFDC7F0F4E5707453EE2D6B0D69D34CC0EF4A" + "1C3204A5A2C031077459E24A179FC80F8833E19F36E7AC0D3071BFBD2D48FCF1352B96EFD0FA3195B44A27EC575B2794909E4089421E56409AD00CF472680F438E0A6D39E88FE6B938EF722C7B7F75F714264C8F22C528A63985C75D2412278B137ACD29003CAD1711A2637C630164507B7D3C0ACD1DD3BA6E689411DF6D3EEE410EC8C93EF27BC82019C3943B85E645519BEC1105D4738388C7A5452A67880DE88D65C1626A55A4565B5C26B20BBFC33F2DCECF938149D8B58B19DFB5F451C3A9FB5AB3DC486C435F5397B6E32416A9306D9869B91231ADFCC9A4AD3D956EC49832C3EF2A6ED50638F6633D5A8FA7BB7B04EF45FF9B57D6EA7B771D57C3E5B9BF96E03ABD601BB46E5CE3E104233E7BE642B082A610FCEBEA684C516AA39A051DDB87025CA849816E177C17C10115939B98B5D9F95323328CB5250C4E38B7E932481D20BCBC66B0BECB3DC1AA196F5FE207B28F36344A3C00F2FBE179878D6C7981467FC2DF70D079088F09B4D2CB56D5DEDE593A77B31930E8F6138DFC7F9A7295FB372C1A33713610B2BE51F311E7CE6050EC276E5C3A50790EE4CB81511052AB659DD54F4BAF211EACDC2A4E987C7E2EB8B384F06813C2542BF765C3E6B142968BF3C66414D01A205964E743777040A6A9879C84681B6F6B3F2F2B0B0455952A99A21CD609C1FD7BE71BA6A1C9589445DC69CF1D4937763AC640DF853FD3E6DC9143467E473FF4B89975F24CCA51506830C279AD8BF806611E836F405E8C1FC36208E99ACB764316BD7C42D2E248B1DC3D5C551AB77FC168296853129C31F5722707791FD5F9AAABC4FD4C82A99970A93F274FD0D6252EDCAEDC076BF5984E6E28056BB831FF289CB0F6EE87BC4816712DA0F322A51765F5F230D0DE72CBB4714C9CFE56C1BD727DD7D3F7C1B5B83029C81A22B8FD430B137003B63F29B710A1A2C549F9AB0B5AC8A123935A1A3BF627690252AEF65F20681E2E9A664321CDF5FDE1943C196BA2F04E42E35F62CE77198F01A747A14D57D7ED46259189A6AA3363FC178A8CC5499F033F9B8D90D327AAFAB7DE16831E8325D337AB3DCDF1" + "2D74374B55D29AA585E144A29BA4F0A96537A73B4F176C5527075F66E38E8858" + , EncapVector + "ML-KEM-768" + 26 + "9C6839354086919BC7B8A547FF18B523075FD1706856400952489E4CD13ED8B17A817A3076E0947B860EC9558BD9D3C94DF5418A7A7301763999C09996715CE32AACFEF767A44969925949CC9B8120CB78953BCF6FB8282427CB3F4CBBC50AB23274493A8BA7E5C09088160FDA731B55980867912BAC53125B27073BB502AFAC5E74973E0F3601DC50CE08CC7F3FA44CE331A43F9673BE745295E693077264A8E238F0A912C0A59C44D561E2A8BD831541FE869B3110B051100DB3517F87510ECAD8651CE4998BB648002BA41A5351FA2A4ED94A58CB707214A59F8876176CF82FCEB99364971C7C0896E8B62C0544463FF101983C2045981CCD9173EB33BA18483ED915BD11EA950DB3A8787C105263242E3A64C748B8B1A5947880C5BF3896E08B4B9916A39FD70C2FF8BD041A2B18E251D16A276E38100A613EA96A86CB3652DCC83F2D31CD9F8B79E42541A66752D191436CF5B5D8350EE7835CDB122855392C1DA5C33FDA12AD1028F2806861A2AB9C63785FD11E220346322C250936AB819254734A8DC9241971AC90CDA6696ADB90D1E327DC977841ABA38324C8FB2092784942003A0808E5546D034592F03BACC19E3138C9070CCEA80A933D536C49E4771091AF8A70207D474574C64B9F1C8E47224174061117E98C18897A7B1B3C4478C6558C321A89A84B6C508947CB61C89204FBCA43BCAA1D688F768B02E50BABEBF0C5FB8962B2355973C4193FA92E0CE4B9F8473E154828C986BF63E1CD985C6F0462C9B4374594D226C5E99B8A844978DB8DD2839E27138AACF75E917635CEC4B407A54F184C22F62981B9F4552FB334FF2B7FF68B470FD61B993C32581CC0F39881A5A6A769A60297F4B0579ACBF52A8ABB143885886245CA2636E20498959E4F2857AB96706E6409C4164BA88C7193E568E4446120215A6D8499C6B4BC559628A9F481C7DB51E2E10DF1F4339A647B4CCC0405B79F70A90CA9A8AD07789222BA6E66C3AFF42B61BAA637AF15126C88B2E210A7CAD9CB08CBB5123A7F9BB4AD63DAAEA6B43F9AABA561212823CB898A676C967B2D64F4B6333439F8993BD7B19687248926E077A9700431A59157B9668EE022B6D31343F15C08F14488877DFB30A552C06F96A16F354B9A3F19090B32A043B43256BA5B1088B56A4125588C8746D4099EA21C4F33AD8C5C7C32286DEE2CA76FC2651D57AEE9D847824AC609B5BFD5C87316C2AFE0E153D6326AF89987ECB70CF0A0A77FA000E0E0ACC2685E2317429C3A7513F42873CB98F669703AE9146919226DB58D6C2B93B6F50C7CDAB677B505945A0F2523886C1B82A6A7CC7EBA9251096D88360D7EF6473ACBCD359A0D20414B95C209B2413B1483A3A061386377B28E3896BCD8C5E2692681CC803E1694EFB6CE353082FFCA4079EA9BB7B2025BCB68990466ED2803AB11516D80325DC4CDE3DA9E020653E084C45F837C88711DF3AA6075741C1283121C8B1C691013969A62B2563E556206CF0810848691C7C408D018617A7300AD943E189A9B44512EF13B669831392F1329E2AC7C623AA044223079139B7F1C213E93682ED86165468377D40884310163A5539F27745E667ADF506896BA933B002E4D50B2497ECFF09D0BBCA4F7E6F9DB9E10C643D23701BD6385E163CF71C1E919A6E20A" + "4E77596168711E913965D8175AC3BD76AAB08B7F9385A02AE883CF6C6E17DD81" + "0385E8044D17E2B96B3F50ED28C2502216322AC33F69D2CE34F0A11E9B3DE339AEBE98283A6010A34BDC98B0E5BBE142A38575DA305066331D8D11D161F512B4B56B60FD049DFEE3771473253A0310CD842C71CB7734474BFBDBBADD2B9B87C37804D8558B00D77B1F79F0FD2C94552A8D3B19FDD6A5511C4EBEEA40AA839A489AEC636B510BF51BF3782F24F53F752EE2DB9FB4964E8379D0FA70603CB8C2C7293BC35A56B818257B65FBE6B2C98178B61D7CEB58C7E1A9100650A945C739E3603C188BA3C179823950C6D874D7FC8F8DDB7A9EE941F4A78A0F8A1EAE080D4D77893AAEFB0DB452659B0275D171736390A1B23661C69058869A8C4D0BB46D1655858162012A057C7C067FD035D47D5D627E3CD27CD3CA021A34F8DB6695C7E581ECE67D898482D509E4DF61DCE5BBDFAAC89465A9209C547B4263E9155940F8FCB1DA3A4A7D25DF2969F31644741A8C6FE8065D5F176214A4DB8FE76BF41869B0F22E1F5CF5C34A5067B1D9F39F3AB4B0B9F6330D7CF6EC7477B53D9223FA70E4C680D49B32883D92B50C53CF9E61915437CF57ECA49D6BBC88C64CCB222D4F20227CCCD4F7E7C21F95CE74D6AB5DECE17F600AD1715248648E4D1AF6FE41A02023EB5CFF871EC654756B7D156E601E82827E17DD0FE4F5357136120A810506A40F28A0B860C3CE75BF7796653327939B507043AB8BE0175A9B312319818063AC40302EB42452DA0B1CB9E3C2E43C49D740A03AC1B9057FB44EFF830D548D12585F50DBAF7404C2DFA7F4403DB8B7CEE65DF528F575C7F552D0632553FC6C90A3B09C1114492392C165E9B1B0A10E9556C72CDE2BB23DF6DA784521249EEC542012EC33599B97B95FF5234158A6D553B3BBAE9704E630D64A66D658F74F27C8C30237E87686FB04551BB128D917305173E2788883352A0216FD8D0035D748C9B42A600073ADBA73F851BB5ECA287AD325A3A3561769868C56E6D84FB661ED5A41649342EA342F82BE528FD9BB4F492452C0A9426ED8677113EC6F1CAC51233F0520425C3B27ACFAA0D25A07BC2CA01115C90432ED14A1F4DDA693C6E704A16BECE8972D8218B62CEC4F255058988B224D86F2BDE4B221A3C672CB744CA8F9158A4E5C97D0C56AF5DFC1C5F0191F64CB6D7AB86B14432ECEB5A3ED77A8D3EA651A07C9A0CF95E631F41C2AB29191F66DD9A73271F47A455999C5FC8391550BBB346CA1BCAD598A32A50CB52281156874F36DE9D8EA5860D99703EA3F4907C640B47A75A285B4347845CD6C94407995F569FD8A4B12461D8236E4015AE000E2ACAF2BD9F8D28377EC0D9E4CAE714C481C5ABAE37448B3F84A291B84B4213C8FE1F6E0FDFA1A6D1A52C974F795C5947F667A8DBE162983EC0C71C0F0237E6ED8BCB7AC9E4C73B5A1ED85FD5E0FA2810C842D484D6F9DFCC8472F05683D9D1144EF6C36294BD06A019EFEC202FC46288EBA2ADE82AA3363FAF32B9B027C75A7D615845CF2EC538141F1746009737600F697CC8D90C80C1E19F1AB2C08646C0E3958" + "79D74F6C6C2D916BEC47BD828FD9B67295A37F54927FAB1263C0D122F1C6F1ED" + , EncapVector + "ML-KEM-1024" + 51 + "2191ABB6D6BEEE29C5780758A970349879B61A028DEEA5404731292346C81EEB1D17766AFBCAA68C867D91132F34A494E28CAA767241B50902F4825771865FC8D633736248963A253DD52C1A07C7177CB6DF74C43C3C74D7133DA9DC915FE14B5BC30B86153E7B86BE04189F4CE8CEB6E5CB69808D35640AC50335B1633162E28260300646153BE7B26A84C1C565956F8577C95CFA80B68978CD7A79E49C7E808B21CDB460ED8388C4859670333B69980B2F9A0F78D7A5B9D8554893A0B5C52447B15318828989816F43406ABFA4950607A5A57CBA62F5C4340AC3D294191687C7D2B60EF2D83F6BEC15B1618129B78577447BD1E36A87142B9C552C8CD2304F35B81B3BC5196C3840C0346D513EC8862BA14A172C051402F7398707A6133C8E88E5B57F9B9EEC18AED1404EA1C47BE5169E5D558E262A231A75925D25C499650C8AC031BE580CFB185A0E0362CE2194C53713A630CB59600C1942908FF5142B02364B74B3F5474301AB2DA4188DCA640961142F28584EBE810683C046B59CC2CC07025DA63A5E8C5E99E24FBAE183329886279C2D58D0B8CAC1C7B3033F05E765C75414748687FF03C00B36709D41C5968528F178CD03B813B69054D3E20FB6A04A713B1B169AB5F0369810FC5CD967A866B70A2D87514605794930BE9B7B5D49D08F5CB08D84FC5BEECB994DD2A5FC10AEEB4CCE1ED251FD4815A9D06EEA6A77F18C56ACABB1F7C478DD52218B7BBF9770A51C3AC96402548A1BB22DD0C29F422330057179EC2D91E1AEE27BBCB1EB4671324265E28676090896E332535BA2E9169C54113F19A45229C2448078AE08F7213F347244527EF726855CA054AB7A1ED1A40A285A60D04C00CCFA8FD318CF182705DDB888DF652669B3224CF54ED6D00DB5121331B40C7E35B1945925E5AC1503A999899317A2370020D81873330327DB7CC1D430E32C6570FCCFFEEA90A871A4B04CA9A8065AB20A6459F1806221A5482802F14A68893B111607C4E085A5A6410C027C1BF5FA33ADECCEA576211F272CC942ACACC879DE398477780A26E7545ED693EDD66AD24772B0C0C4B033A73ED94E1D02C384D1A288A78F5D8838B5AC0B37855DC59B987EC9B0F5C6868B0319F3372024FC15B247144F925D3EDC453FB8551EC1CA734A37A3972B8CC9B84A217A4D0BC0D08C815359ABA59A07EED998133587E062CF735A8521A18AC2059E3F142F12CCC5AE024CA33B8E4A285BE5A3CED460C2C3675B956294EF2C61C96A88441665D56B2029688EFE4540DC35ABF5957BF9477F58277851719547510F22D01A13D0047C5B88372A654C732AFF395DBF825829A06E6AE6923296218AB9AD33D0B07A0B6A3F3B3DD8428A1F82B72A921853D94C5D400494494F0D168148B5392F386B16C6056D6B8D767B7F4D918A9384C855C8B4DE8450EB6CCC2DF18528CBA0A3552D5704683AA3AAF52751FD5A17801A76A58A75038533C1576E8668B5D290CEB20CB1FF91317D9B418311C11E617D10B4B0E9055AEF5CC6EBF8B91F0244ED9854FDE463E55112A238038750CD016735613B9C461996B90A3B4D4B0D8B8825EC087FB138AB868A04CB3C9249B5B118F81ED06B99E6B521F206A32A4013E876C6C80B36AF39AC197A21B78B5F2713AA45C74335F32676C2AC2902710EC42AF5984687A672E0466831C7B436300B30351795B22B3A939347A7B744831BAFE78C3BF09A8224A958568F75D71F822941DFD64D8969ADA5B73C847272B5914491BBCE5F0C09EF501BABD95190F93201AB413E833F7924B696D5C2B62272245932A6D9C35E1670AA3BC57CD41B8D61A52AAA6D80BB774F922ECB41CD58469C4DCAC698F6730890237602064B148650C17E6EDCB3713382FE755E46F3ABE0C6C45184AE5C5C861E35A0E2BAB2B4E8A4775584351170152512C7E99E96F285B916605B53B91D6C501F51A93129483046CB2DFA16D4A540ED2B6AB18BBDFBA2B201BAA20297AB1FE7133757C07DC041971AA4219157D77628F31AAF520650A2250ADAD1524E58268D21CC62E2469C8425E86359DBCAA066ACB466E06947A1C144E53EE0DC930432C806C493B2BB25757B1E2C775A27A896EC25B58E8A78D4DAC91815A1F3D90635EA25AE13CEFA77877DC0C2F0678ADE307243B438C54AA69DDA539B8A41C1B3C98C77FCEB2C0BC2025A272D927943EA338CDF32A0DEC8B187DF5C" + "2F1E2CA7BD72AF847CAC38CDEFC4D345909D7517543EDF32E2FC491BA05EB5C3" + "D892E0948544D020ECE56F00C6495B9BD1C469AC0F134002C864D10CF61C6C7887890276A7546310EA077741D83428F22B60E2A40ED8BBF5D9227893BB0B7417DF4380323426EBC9744EF35A1BB6DAB1181FF0B677E8C9B6574360994F96EB87C3524E15E468283169D90D8A994BED0DA9CD778CA239CA6C225390221FD408A3EAE541A032D714F3D078DD1EC722CA83FFFC92A416F46FC1E8710C3F8E9CD452C016F483985A5C1D951FE2B03C4AC2C9A0BA71F6FFA4B29EDE76DF69CAD594D10618B94E783FDDAF11CC801ED5036FEB4A70D071079944B7D9F1FFBB98F123DC34A52A40DE57E079A1F8B180DA9E6ECAA47A111F3C054DD9563D7E74B95AC65AB453AFE0BF0DCEC5578FE6FBC2D9EB91932799EA64B2DFA95B2C9C6E931FFB0A5A499DDC42C6D6B563D5A6C1F34DBCB36F57E6C10C69FEADC70A3D01F5F3715F6BEA0D3E2E7AC2B04E01CA80A2E187BB7D9EFA5EE257B9C3CAEA3947D785CA56A76D0B6692A69418935DFC36FD9B26758F5E1043D5C2A6D5CC1556AED592BBCD652F4024BF324910F8CAA2415171F3EC8B1CC494F2B519CBD2B13D317672DE60F0A3D400E6079F85C3CDAD0DA5A17F2EFACCEC03D3C2E85B063AC3CB29F2FC4850705B8D472E35DC12D06A7D021DB302CA883DC7D4A57B7A9E1D023960D4EA5E39C9D5BF328B8E4AC0AFC3A6C990F598F373059AD28EDBEB90412472C637877EA82E7105CAF467A15CE961695F2FF04C750E277E78C1CE9E490A5C45168936BCF79261B95FB20AFCCDCE790E0AEDDE204DA4F6734700FC1CE05239F4EAD0A496FD8F733049EF9B6ECC54D5C635B56D6FCAC7ABA1BA32D707D888B820CE3E794C2562798210D726C6C5DB3D22A214F3F2A1F2477FCB77377AF41D0F3F0AC635E7BF0674CEBB672EA93B9504EFA8C5AAFB7458E41A1D85C28841877FF2F444005CD1C1E4227573E8151184786489C7CC39549F8A29B9C4D68BE38F24081FE239C5D0E14AD5E361720836A72A99CE88E47BF3E6A1A9EB57CDC3724CB4789FB88C135CA9CCD94CCC9BD73EE8A796BFEA36E894ED27B0791E8AD23FCDA5AE7C167D5774AA00468D97BD0FD2B9BD78D01E1E39A66994A369212D95D7886BE00F2B7A1F187C85B873F33CB7D3E4C23993F7BC9FF7ABBC6FD9D5C54F06E5F31B79DA992BC0F2AA79A36BD00483CA0ADD2DCB2BE36C27DC913668F030128E82281E5E66E87C75DCE3FC60431CE6A19C6C45E19A2D84430E945034B7DA6C506EF009493AF22F701088E01C20ACB21972FF28D284D2A57A69D108161938F028011BA985E2B925532B52EED878646579183EB0EF3FB57BE1E05C1BD2B68D0E4A9CE104E0ECDFB30268E6C007812FD88498D97795447E7F88CE8A3BCD161B1E34DCF77413B778F88A3AD66F247F7664D0F6AF31EC1DADC00FEC09D6AC63FC8A802C3752484B5B43555381043FDD30EF5D7DCF4F5B68EB89F08A0FE5ECB65D6D9418CA5283CA398D3129CD3B44EDF5E3568C953CE2B66A28B473CAB0C43918A05C5A4358D1392604495D06B395187CAA6B36050B436B218485466CB6643EDA7AC66AAEEA758715A22A8849066879CC966DA7E0F7E843A1E234920AECF2A4FED2F78C69A6C103A5A535D77976B1E40BB0D75D6E370212017734F2B0BE9F3F87C5583634EA6A998D8FE9CE5E5905F2A6ABB9D635425D06731C1227EB634A3603D081CB1A7C2B0C1685942EFD6F65992DD39AA67CD954FA0310A0F5866AC6121E27E349D5C2ADF37A672C1DFB021A855000F0C0C29489926B997930B20DF641120EF8605CFD9482EEA9344918AEC689F580A94508F318E63EADC3EB7486B9FCD7A92BA16F5E02CF78EF73F528FAE3A43D451C58261A82B3735E09D4F4679CE112803505006CBAE6EAF641A6F66B0C51EE90095735D87AD88C3D19CD7888248FCC3872939605F205976EF6C6AA52F8316E9DEF841594373FCD177F0E24D8975653A29F738FA8F0D457F2D72B7C00A4B7A66FB080F705A8BD599314354E65842AF598B8A2A40F6CB3208762AC6CD467D06A2E987A7AF72E355BEA297DA87BC9250EBE8D8F85CF292200A21D93475BA46BE78C17C1605C10B97659859F08F980114955E68361E180C98015AA46A776070C4F2B328A903A70AD226742F0022279179FD2530390F190E3008F7202C83A6A15866DF848C8A150D12287451DEC8EE7F04C1C3121E4BBA687D7" + "5087E3B0C90BF601DD6501E071270EFF8683621E9F5D67A7A668E50C4F460A75" + ] + +decapVectors :: [DecapVector] +decapVectors = + [ DecapVector + "ML-KEM-512" + 76 + "B5589FAD1B1ED2EA0F5869628F8777567062E7C4A06C619635B90C44BC3B0B19ABA7D01A90594CFB8AAFD823329D502A08A5CB10A7827D6B9687634BB1E3805AB22370683871E78A2D5AC9CA582C7B01C1306309BEB9C2F9761552265D6EB67DC635B9025ACF510C6CE8D1801CB81E47434ADE02C00318C4BBE349F751C75A80225F807C7FB976F5BB5478F8056773795DB90F8A762EDBCB1CD2FA53F69A5C44B726E5B5926F92366351ADCBFACF4CD9492B216638F4959E3635A3918BCD9AA59242667DD886BA9B53061421AC3206A68349357A0DB7B1B40C2B6BB41676FAA77E05B3B8C57C549CAA2BA36B6540AA060129B7F5A7C59DB17E5A9708D6C88AC4B3C8B6DB1234A75319931AD3B6608031BA1A45772DBA8C39C13C24CBA719382CF6231AE064C127B782C5781AEB16CB7C7767EE608D37986CEF629FF5796F9ECAB3A0CCC8F4E0687F32BE1426A198B00CBBD2A99616268B5A3B700A1990A75158B668EE9637F3F7032DA19B7F7185B5B70DA163792A79238F8AB3755AB4A387CDB6A7905C17CBC259AEF9B534D03860CDFCB8C4349465B8847C741B0D522AF5D976E2D70CB7350A150A1217481C73CC5CDD6393AAC9BB6C68770819001AF7455EA2687C637AC43AB3812981A511BACBF23DCCD3252D1860A143B4939523C92C94A213BCED384D79555FECE80176F739C955C8283668CC1208B8B08FCA3027BB7494BFFA6B2DF317BD36408FA0580D010DD4376EE6D91A0BFB70AD3C086D24A9991A53C5FCABF04647A7D777EE428F749C309782C2B83049B0961889012478981F16F83B6048C0E2B1C67F739B4D48256AC23341984EC19B7286CC301A7C205AF19B4DD5C3FF59A00C75048992BBA539369295BBB4FC5245F2598B25B935A445EB6CB9305408FEC37BBF8C3E74BAA1BED452A4D77607BA45DEC85F93A30A4E33960893AE6165955141284C335D90C4799E279E1A997A8C93B3696377F0941382B41104B84A0C95882B006435272609023E705415B697B2AD9AC5C16AB5CC343EB293334A298CDCE4BF4E680D92E3CBF9E38684522CD496A237CA72F453AF6CA07EC754A67B21437BE714F2361291B7B76FCC0B740ACB3C1C71EA20149FE5C27948A09852C2A3EA773FA41FA70804AE74A344E7474BCC909A2893A97A89A9D755E7495A9DF0B292259B5B904FFD8249AA5590027B91DEA4209A156994A4AF3F6143FA599869C2C293C9446A431C439A92A94CC1C5977232A47EDAF110202CCB5C881AB81C29D30CBA381C316EA8016CD912FE7C8A3515CC6C6223BD988768D562FE72578C26896D5B03E1D920B574295E88BC3DF657EE000D53B58A28773264730EC50A70EBCC42506939F185B566700CB8D01253F52287CCC02036008F43CAE36BB71ACA892690A548F832314CBA13B118E6608CE584932B58284EF38970589508DA06079679697B8BD8B2345B332CA0115EDFAB5A7F9A0356A3C3F102092F705FBD69016FF9A13B0CA21854A0ABC53D00189FC57B2494E7977BEB352B24BB8F338FD30A4E70C4379851A295A16D8AD0100DB46ED772A99799198D9739E1F543F140324243767D71070BA303177498B029608F894160787F3DE62CE65B74278B5FFA24A65B89A59097A1FF437ECA2C431CC70CA0418141C36574D730CB64A234DB88F826112F694FFCABAD91A93DCF9BBCBD6071EE0C6A500BB6C78A1D5D362A45046700C09CDE0360C9A9754C4B5C26FABB0F420ABA265460F7729B625C5B6B7ACF97766B9032F4A429CC76C8B03C7168F7037E8101830C8B838B5350D2A9CC575AA88C3A673A81FEE3CA79436D2499C280C1A52E0C5086B5C3C209410214AD4315704545B74B98C13AC3131234212FF344C3514C876A96006D5DFA01BED741BDD76260C026C29E78138851319BC71F12A66B2D362066A1C4B2028303B91445AAB80426AE9465AA25859B28F1402179CA3D1C073CE876AEF5B25F964149506C15CAB4698C91AC94ABFBE69DACF06323E9A6E151CA6C019F49078E60C41837B1763BD87EBCE040A2D7A3D4964839E4AB07D121ED3A2E23F75AB6086161603C20B7BFC063CBD00A7392E26D7F697C17BA0C62231A3E34A72C0BC958093C62471B7D1A8B2F7520A0581E809048B4798C0F1A0A2083306D6C42233202F2101C1DB61A58086B24558B17D5BEFC7542572B81E9880D3F164D708BC3C357546DDDC39C71C3CEEEAB482ED67E61CFDEFAAEDD3027B700DE0C72DD84D43F2B81F309EC9D9E0069DFB3705C76B01E9A8DB4B66437487DCAE2E3319165983C937CD3E95C39A9" + "5EB39C88DB720B7C6B1DBE08047717F870671DB16328EF96F8EE9827AD694CC9BC662E697D2D30103851239332CE9B2A3653BD98C3FAAB41DC5445FA844E05E4D189D4B8E6E3FECA09A81975AA0585E1B179808678F22944108AF19FCFBF1C58BA76FDA508449EA6B3312F8184D1E568C1A4B1C188B04E654978EE9E7A10299C45EEEBAD0A24884E12BBB05F0B9575ED5C8BACDEC1AB16A59EBBE2B7A677559140C9CB00E18B4B5A0C51A4A92A61537DAA2360525D699DE4F30DB0B4D841B708B92E38E1E07F3A2EC124E6C82DF062B110F48262BA95A50956B7430EED0041B295FB84C38BB2B96CF1C85E88D3BD4DD5AF8E1ABA963FCB9D9004F5CCB7FA15F2D8781BE9CC19B77EA6F397372F9B6C9B375342530F4FD4E3B934E52B60C20FAB1DAFED0E6C72FAD8783B62852000A24BB81273BBF1C25EDD879058C5EEF67E3A0DA14E937564A508E9475FF3B25B5687046684760DEC75132D14FA471D54F4BE9ACE2AE9088678F44104230582ED466FBD8BA53150EBDAF126F417183689A24539358B00C4EABFCCEC170C87975B3DFB9FF6267E81B44C93CC9451A3F9F7B913AFDE043748C5F7F36FAB63C801B0BC5042C3D44D836770E9058E04F3571B04C0C87D38B17DD1F7ECEDA701B0532CF35A096F8237083834AED2B2ABA4DF1CFDCAEAABE69DBCFB9B4838C8EA1A9A3CA8FFC75AD9B9C0366517D782F412946539CDF0D02304AD60BEABFF9D9BE1F75ED39BBF2F10FFE061F2828DBCE798DE6685A3F3F52477B361B1A42DB1C0922E8D2D545A6AD54F0CAC86F6AFAF7B381DBCA54330F13BC19F7B698603F844170B90658A9A82AC7D582AE798DB6EDA9087E1E7EFF6CB903558A289F988D5CB5D2C52DB0ECA5898E3AC15B136640A38ACC68F1A7BE53C54F0E1EE66556663E204BA8399633348EF38C7B1FF500AC53958D103B984E540732459E860BA5F065B466FAED85DA80716FFF2F57371D9F9D09EBFE3A088B4C2C44E1F03C73A97830269A41419D2BFA48FCC16523745E67C38E4BE7344C4AAEFE9BD667CF4741E5F16607680378DF0D2EB60DB6DB6AA66A53B63A03E52CA" + "1EB644EBFD75877D0CA481E1A3B95E7C461B81E0E3DF5A42699775C7BDA3B004" + , DecapVector + "ML-KEM-768" + 86 + "859384DFC0C207A6466CCBBC507C26DFECA413E06E46E30CF18528577B76C108C31E1B6EFA8B41044A4FB2152D3FF391BC6443B57060AE655B0EFCBABC889032ECC986D2A144153C8189505C37C6BB1471E5EA8C1521770D732E88A1A3B0624E5A2A42661C1743677591296B81447169635160F80D61F31C420A5D35F5ADFEA37E4254BE39741E197056082C1BD9390A030B2A63FC37DFA561AE091DD9DA11649BB7406464838BC23C872423B7515E3444FB862BA7D51BECABCF9B458F302905CFF983D00578AD89686721432221782BA7A134E052BBA7879A36654C081E4C3B615A1C764C128AFBD79EF0D816C7E5AF135A51448C14CF3C6172E022741B889DA485C208899190BB2CB8090BD2B7F8213213E090B27A9C3C56BF87024BD2E3001CEA031BE20E959BC1A75281B8A3639F487C12F40786E5458B14747D965E0E294947D6A487D44938D50829D0723A910806B9376E76480141960757CF37E2AC03B4A6767A68381A69F6B45C274C62B800403C1C992D0B84D336560D6B4BFED4A38EDCCA3C61699D31CB6BB88C10909CC6081D1BC9A549F604373576F2B5B14D453EEC5C7673B5A39FFA8B6A223972019F0EE7300BC82DA77414E4413648B68FC8E7B51DC6CBF083098C1A2F2CC680E52A9E7A8964EACB2DF18B8DA1D603317298B797BC17524EA03376BA3B410FF1B1B23526D3EB7A8255380793C5D938861E4983D4D23CAD3A34BB256434DCC315C40C8D0A9D2132A595B78DFBB655F9F2361A379047706400E5CFA1654AC779841C0767AC8C1DD27254599137D9F3355328CD34E581E1E1BDD64C082EAA29A7F71610B65766882CA804106CE0BC5E20630A44BC5FF87DB491C2CF7746B3150BA03ABD907096D8004FA20384FB154F553659BBC451C09C0D42A43035994CC24B4F798BC7C961C6C64C92F2C63E87149D89E98221A9337F0A973226557056952BBC05ABBA46C7EC6CE1B2721A0487687C16CFF6B50CD5BD59443CA765353848AA42323EBED3776C014D60D8A6B6D6B31DE26F11C15822991249F76C3130735D2C22D5D4B4550A3428004349073E3F296A286C3A0D6572C383B3FD277439C85A03B42B584C97B5779A2D675B6DE024949438AA8700AEEBC9FFCA10C8B327CA6B917B899D3ED817C4543CDF288B489C2EF3AA2F95F310B0688E3E81AA83005571812A5267781AC21C92F7C41F359DE74877C19124FA462B3A181BD885A3C3534F16D99AFEA4BB6A434CE177C99020C880EB368FD211261172F0BB7AE4270FC9AA6DFDB5985A1C069F1815E9277BEE24CDB1ECBA0CF2C3C48100F540A24E4050F9B376868098805136E673150C957B73EA3E7BBAB076B459D2157E9E501734500BD2B574B9C563A7CA8AD83B6CFBEC76727B1D3C61C289958FC8805AE6E4455A5257E8903854E35112B7CD46185EA16339DC0838E118971CBA5A78025F86749433A38AB79B58E6E6885003470E208735177F82D738C3F7A52D848B742A75D115A528F4915078632FBB639145668B504C30ABB96929A75FEC5DEAE90973B2431D685F74C98D36DB263BBB6C73D71974993E13B50242167110262A8EB406C987481EAACD44069D53C2B967003DD2DCB979D81A97F62351344BC40A6B203083918C40AA459EA2E20892514749556294534CB3E3B948F1C0F7A941FDE49874C609973A4E845B112456974B3AAF9877B3159B7ED0C4C423889C8A8B1C5B33409A8C6B26F573A6668A5FE0C74AD0CA28304D1C7A498A93BC08921D3943042F986438ABCC61A358C6482839A524845B04FFCA16CB741F7DB5B0AFD7B9874B6B53D156B3E036B3D41E963B27AC0C87B411436471591001635E6440903A0B38302A92C8777F264DE5290DD1E511BC7C0A68E3266882CEDD30ACA8928596B7AFFB28B94D8711E31A8C9E5AC2A9567B55630D5FF8627E52386408A1BB3BC2D6B90A7C33C04AAA60E9046E10D89B41FBB8E3E26CBC27C1C062411436CC22D637DD021CB93B6972CB7AEE5C98AD64006B0CD070B0AD99D467158C85852B88666BA1DE29CFA3047396A5BDDC76B1AFABA62D622E17F846E17487A167CA9B45070309526EE08791647CEE81B7AFF21CA97C7D09A7B0A2880DB0110461F168A8F5BAFA045864373217B05ED1223FD0B00A1D7810E341C15CE69567E8265110918B857F2546BEB5761DE3849FDE971E2CC649C363C0EF296E2DC94B07499869851BA3032248E4A628294D626C6F38C42530A07952D6B44B650112F77C8DF5C5939102090C8188DB61F3713E93C4A22A61C91FEA79EA06BB1344A5CCD63812F540A8907FFCF62FBDD7344A744095BC5DF65047DCF8362A38BFF92B5B6098893AE96286F4199F857DC5351B962204807B9421602A0F00222A82B6F4785DAF303354F9C097CB74A0723CBFDA0553C4B0B4454E1ED11C6926396BE9AD5CD16011D333A1A880E123934243542776A0848B0CCC442BCB117C24982C42D10510683E7D534AC47C6631ACB187039D098B7DE551B868A28FFC42112253AEF0F86F4C358C778017B30C0C3510254E6ABF50664FB54C197BACBB790C32A935783093C88927B65C5C696C695CAB4A845B262219C639C4F07DCF663914094B1DB58DD04105D7C6BC32B6392D481BF954ABACAC7570CB30BDF265E955AC1DF3A356AB59C89B0CC6DB7ECF4185F81BA9AF7BA8CA47BF69A278681C1C8216CABB4887A366204118764669209D502BE63BC8F21429E7434B381B55C95C53DA864E333C13E7818644788A381215A4954CF3A038DAE27CA4684B130268436CC22FA100EBD843D6D9A20DD0B8C4BC8D56899FD3E4272CA1301AC9429986977A9A4910A785990629DFFC7EF2093AAA646741825E53AC6C43B962B5624A4D31A5E054436A2B7B880460A52062E4C5C9B2E72225DC2355D44DD6B22A18CA6A05094BF8D3B2A0C58C13777BBE448FD0182B876A596E595AD4E79DB6F0C1B4E1435507802E2B539EDC25820B23E7127655E11B34A5AAFF25C11A7B617ED7083650C9030462A79AB77BE63780A45F8CA96DCB15358E56AEDC31360B2030010282CA67CDE5169993FA808EE475BDA0AF49C216AF9506330C9B0384BD2C3C51B814CFF09B3C530CA754E67966C22E44F68650B7117F78235D8425929816526B0A46F647A85A7929B49D677A38418368091590FD1276EE9629E63944DB4770B7C22E30B2CFFBA5B6D46A9A1217B7FA44B54B8266F07554B3EA970CC30D1E92C03F41CE2A2093F58A831A4353E02393CB8969A582D55545EC39A61104BA2E5BF082D0D9573988CC26089B222718D29BCFB365365BD9529C55AABA5635850809B305614091247E8BB47315503A07337ED40CA79D3C8B8D240D3AF38C8A7A65ED18047B92600FF09EB509820F0E" + "F8FFAE03CA26E156AF3AA562E6F84159596D3A985AE42DA4E45171BA7C8A1CFEE2549EFDDC414B27FF697920BFC8A006BE1651B0ECD18F91C91F1DF9E6332178C3BA45407099C0C5C223E9C88D4BED9742D8A3E1251C6AF4B03CF79F2671D8746E7C86EB270F0C54B3362F691387539F5E7B16680AB51C1F93D098D40CB8698827AE9EB1A85F52199589D561A0FB1140FAD888AB0B03FCE03A9D1D1069F0FFE274362F3F1B656E402E3075EFC29CFDCD3D3A4163F74CE6B1566CF4626D14544A2A33DDF5860612AF5D337C6B55A7EAE01422CA01882BAA591F9EDDA6976E2BFDF945277B4C145CDBEA5F22B636DD9761F8EDC518C33223FD46D771E39756B26DFD35798D3AA071F90D2250220D4869C59B3CAD020B1E786EB8C856D2AB33FFA1605FB61CB91C64DE695FCC1D61BA06941B5336CB67E648B4197AF2B297F4EED2DF996E213D63E0BFB54A86C47B286A9D5059580DAD4007E4DC6FB944A53717F325D94EB6A514C13EFC445F8E4EBFFDE3D7B00B955AEBCE826E4E9568C5E4EB69E0F3BE52C7B441CB7F09EE441BC01058183ACCDE0A9473881BF09C992A86C6511319EE5AF76FA80079DE11EA7C8B9D0947C308345A6586A8EDA5D00AA7A26346C3E2C90C1709E52EE7CA292E89E1C22B5447449AE508BA0FE2C801E3A0F6D12E22C93BD98D19CE199AA055D209431FDDC46BEAB5DB20FAD6B53B93E1E723785EBC29D486FB240D6A4813683E2E71FF05B4F53EB57C068C2C35F0B0DDABD8E17B48F8477747D1D97AFF5434D1F09CE1DD325DC2BB5F79523DD4185DD38D3629B6E83F76257BBBDCD6963B1EDFFBDF4456C6A6D2641F035D08A7EE3A7F1C046CD3BE550AB93853FC689913E0E136A83BA15F948D8F229CFD6E12D19C2FDDC9F17DFDE4B28A7EAC09043F1D4F37DDD4923151E837D7C9C9FC4DC9DE1576686B50F6934474402D53439D45A015BA6558C3032571E1B3263BC7E6CAD549D1050FDD49F9B5CB2024CF543B6FB706B636106B54E938F8EA664D9CBD75FF7042ED9D81546BDA255D177D7335A7DF06B5D0EDC98C6678318F7131ED7564D5344DF54802241144502EEDCB057BE3CA1D3FF8C65FA7487521DC9D4DB49265DFF8158B4CE2A1893399BCB4BBFCD2C3F7D9C2286EEDCD76518026D6586F10A79E02477EFD8FF93931B01210C6AD0F75D23A62F68044200597DD277CEBAB9B5C21C74C5CD74B63A8B6D59789F5C50CACE6676C93990DB2F7B4F39E00F455AC002C12367D7BFD0CB0327EF077C45382F0454255B2E11A942254608BC41B890172BE38E3EDD20ED028C37006AA6ED5907C78614748E53B7EE3242A7C1000552EDDB67A5436FE8808D53699E9BB3C190F5B966A1FDBB1DA68E8FC630D841C63F5FBE7161BAA0670BF970C3E617AAAC12C83FF3C1D19761B7421685F1F6C9C93EB35900E54F10744D90F4D444E06C45E2A9FAEF1A96F0B752D4DF1DF86A9503D8D99CDF8A62622184A919E19DF8230E7017D5F411A89DA304B67E63CC3E3D901F998F563CA9BD4C9BE" + "34CFAE7F2CA3B0B9C3E06AFEED554C053F6E51D875D3BD3FF0EDA2086EE79F3A" + , DecapVector + "ML-KEM-1024" + 96 + "36BB5D4B256DCFCB14028B1EFD5A74E5606617950DA40B2408CB2C0462A222D4373A7A290FCB2997DAACAE414DD8C3AA7B38773C79C949059CF15B12FA8C522C538FDDE905E798A3E7D5A6F27C93330B8243A12983E5C83880971827B01C2A03B4FABE4B15071AD124A5A00C69D98AFD6BC63F305693B50040595995E99AD375BBD58989CDD2AA92E95532EA88336C5DD9F6C4A1B254FBCC4921B96DE593C3134A2219B7A192E87309BBC6EF95499D594B30A4661F809D03297B6BCB2EA894B724083115B0C6E6C2AF2D239AB69574EA95A746CB23A0DC429EF42A79234CAD8B9E85597BEEC196E85870CA1415CAC40213B64CA34A8247595C89FBC19AD73B715AB3990B7E9691C9E5C767D0780E4E3026DC172494BB4ED87311D20896571608B065224B2C08D3B754837C0B8AD8CDAC7775A8158A0FA93872C972F4048DE1E9C9A8E33D59397E8DE188251A6286BA2CE6F258D1F8A99E78C464F576FCE80A999C6475EC03BC597014DA6084F2C97EB88795F28CDB27437FB11F67977BE1F865722B85C3371D4534CDDD396BD6B12298F4287A7CAA87542AFADCC70A222FC1830182831BD1319D980BA03937B2CEE6C2417A766E841609D8417E9C9D8DFCBC956217A5EC02DEBA6BA6717A0C71009AA663DD4CAFD5FB5142EA14FB52C72FAB33B0070FB38C9C9670323DEC00D9EB9F02F64BF4E65D86C2C216799093040F6FFB80FDF6BDCC649F1F39BCD2C613221B528803A75E16BB5B8B067EBA7765738AF0647DA1026DEA6717CBE4116BB28DFF91C224243C1A72350165C30184AD901A9425538D255B72FD834C8E0090CB0289C2F79EF7997CC2AA5928D453198BC392F6524A88B5AE67B96A1C1782267F044A0AA0D8A011332CB8D34E1C6CC149A4A6A3B99621A308818322FD336EDFB867E4495EC9D92B4BC70F7B170A0D28B3EC365E5DA02C7E50373B3059248A439E0A7A0B3A656863785CA749E7419A9B1595EE28B7DF838F1F2BAF02A065B1242C17DA7974C5332BD7724C77644E88549FB572BA8964979979B5A7901CF05823B4B985E4B3D531C21620689F036646D528DBCCB4645B72EC920FD0F5426614BC469664A48A9EE0B5C7D9A52164E3163C299441E11DD246731180C9BD70219171712C7BB371B213FC976236043F9367569561CC00F033D473809B3C49FA3725F111907015C6214C63347C4B8672980DD856E30B55BDB7B3D22554242393AB54314715CEBB43AB224A911B090DF1CC5E6950504ED063BAB0A0D72A5C0FA8A1824962DE9CB685745659ECADBDC3B335627DDB5959FED1B66108AF4F5548F151ACBB3336A94C59A0A6B28D94B9359AC7C766CD7B9015C3B99EF45280470C37E985CCE280ACABE67AFC54151239125845B94FA26656C4CAD63B0494416464F4C7FE5936BC284E9AE388FA844EF71227D1AA0E872BC6FDE92F894C26E885087523CFB56C7440222150A62BB27C067173025BA2953999BB3746094D546A6AE03BAF8630FB3BC6F57239E7A35C20691FBE1611E6C46EFC25B77029ABAA947D81C286C4816E24F35D883B4D315A529A593421BCC5765B34EF6C4D416C3CD95443F9BA2FA248659EE77AA201987F6A5EBDA78D307559A874963B1927198B3524518ACD6618F2D25AFA4B11C4B396C9111A19398D55079EF62054A812C37FE81DB4565F67038B9EDA7A4B63B640252701844917E8CC7B3B1EDE433D0356A2C0EA7E2DF3C237498231A65E8E87874BD78487EBC1CE800C7D1B56476C220A8BCFBA568BA20B773C993A98302C084B2BA25297049841355A16FF098C420A2220837BA6C89B44356B810576760316254646B0A5A41754A18AF17D998541400CAC0DC830AD9616F4F17840A9CE2D69B3BBE8CF9D60B01B00AC7EC8B1E08B3D886A804B4A4F2FE30CEE898E42109F3A16C91FCA5D1C328B90F00DA0996F2365BE54927D93F83F8F319D8FFC375A5126356A94C6B6C3DEC98CC9F2CF0089B3A6B878C73BBEF04A4442B1B0FBE273A994BFB6E88EF627B6FAE415C37691C9D30E10B45A83E02AC3F16A5B1249E8B15C0304AB6F3482477C44A6FA8B5DB797FCBA271FC1544CA333A6144E72B02E929CAB8FB6011EB66399B8AF76CB188AAC3EE43B992482C4585934A5075428FB9966ABA67B414114E379A05C7F81C351F6E18EEB8C3042E4A8DAD0CED9A28CC492A2573590B15A914CD100BA119939610C6CD77D73FCB5BBBC50609152B3946E2D5969BDA6586A7B26BD4284F41269DEC3C124FC3C73A0A369818D9A491931607AFBC85B54410444F9A426C9491099CB4AD40EDCAB36F6924764403ADE37288FEAC93C2544F16ACB41F0355E2491FD948FB8774A62C0AB9FD3BDF48948C94206E386BDF8D463FAE547B580B1D9531D48E11D1DF6539A7911B5577F7D8B55B17547306CCDF4224950776A2C90ADCE2673983C3FF286260FC6B08138CDC93CA3297873AA9443C7165021D767D3C132B1A66B413B8BD3849E8F6547665601C776A558209FFCFC84EF1415DCD6091F12C790F1CFA2C5CC9C170BB7D0072AE997FF443D853B021B1999E1686AC6580EA6A0A946A2B417C17EEF967A51CBCAB67992A16A8C12623CA327AC7FAC5E4B616844E28F43BA45AD9252C695C4EE6707D60C10085917A3D5CE9D46CC9EA52A7759827FD64571B7B4630554946796B308AB65680FB23091515683D1CB815221B294FA403D79124A9989EB7402F9990712E91D78C89AE45B33647A8FC11C5770AB738A3573E2C32E485A07DD0A009DF27B4456834776ADE5C454E5C70D00CC455115169E8C5D9AC35844E5126E784596B191CACC1C8FF6C2C0E371A81726930808CAF3680CA194D0156BB1662BE1928D3B3A432F102EE9C8BA950C3CD05B394320C568474D78DC7E75862A155CC73CF2A093C87A96653FC1B1288FB130CAA41500541E34725E612C475740079BEA8EF227BF1269B9F0F86E2CA929C3D611CE52552859C7CFCA9BB0F125E2CA1BC1CC67EFF49E27EA6B087700B8723C9DC0175A424D1578458D0BCE007C5858361279B0C76E62A03E1977B880552396320976040D9155DBE7A4AE2AAE1A312B010CC430B310CDA08EBBE690E0A102D24BC508B2A31F61171EFC3C80E0038C20469436776C65656F189A972544529CCCE3D917EEEC4912F779E3435A890000042468F93A3D1F104780D26FFAAB53AA33BEA1A3103C3A51A86C6A7E178BB65C8145F6A41EFB6DCA856FE4847AC574476DD768271031E2E8A70220CAACA92EC136120DE09FFA31C034C0A243EB48762286B1B2218ED2B34C551572B6BF741A9D0D8A31BC1678067A3FD13995CB559325B29C39A78C7798C3806682D3B0C810F5740D438A1861C8B5F11343B9B21A11694BB1049DCC0C872B2575911DE0E8997BD19F2DD7B7D2AC501098CC54444ABCF1C266721122A6ACAC20B17DC5CA3F9B5089136E6EDB6061382E96D08B27D869003B8794BA83A50B1E2E72910699BF02513B35A1630F0BC2E8A666087787DD3B0B10529AF8499F6314039B827786B71CA5DC706DD0610DD220EAF024D577CE14F80649B86EF5F839ECF23F5C595223A294DD9BC0CB7BB59F039082E68B57FBB325911C2D8CAC1C694CDCE01D9E890561B43FEA7BB5E2E00C6D3A650D9838801810BDF5013A8B95E9D76B329C1F2A0A40C4964FC52823E55B2AF6412136314937F98150CA4A2679444773B66CC679519815746B74728A0C9B8346187106AA2185669023A1983A38580505D04E4847B2AA2A4127125013DB519F141B5FE640B06935C2B673819A665D9039848803128204C4B223533C9A3F12C9553C8A4161490FCC5B51CA2B6A81385169C38C78ADA8BA7C42877138A50FBFC5863E4967DE01593597CFC7110A0B9C82506C9FDC7BC4D4D6B245751CCEE2119BE688EA0985C5647294A3B9ABD07296473B70578FBDB0C4C5F16EBE034375818A63E408141B57129093FAD761C54A99133111707656811A2351319C04B99259946FE197B7E89283659623BA26185F887680943C1CB88341420C6C32CE2B5B6304FA99BD35A2D7CAB3E39060056BB4D756A78050835F301AC355BA7A412227027DFD2C681440AF9C81664A9C4DC5955F4BBC2479888A7536A9199199AC6710B495949CA82BB9B1A2B06823BB1A79C6162CFB6A8B8A8893152928A169B9801765DEB2AB890BA2D32C5C4409912C86C12C08972CB3A042BBA1B76675987279067C288FAB89DDD93A75368888A40C2C9A20B60A0FCEE8495DABB859B797C7B763CDB9CEB405213B7BB45F134CB6CB106CD7252EA069FEB250138018FC4A5071473CA81C6A495A46AEF27210D556FABC5D12281CA6D03BEC1E62E762B589499CC3DEB96E2A1119663DB91A6A308E2360A02413CCF01948541C54D5589829053E8513A916F58E7318073CC06A308A53FCBA7D197E3A2DED7516D56C84BA0B51B7FFA0E5D77342681EB856686B73D7942C37D8E2C8" + "4F18ED0D9FF40079EC3A6D67F903ADEE4D5CAF5B4194D969CA222537EECD10ABFF3CE7CDA292B2F632F65A1C73E8CE44B6D7890E123987A8C08FF23F7345313294FB6B3E8E382EAFB352958637A6A01E0839DD1218F1BCA9167894383BE7F651AB0C0E74CA9B19DFA157DB8656E5C4367DED62F7B4432F43F5BA02F394271D79F1AC3DC4B1CAED50723730B337EC50303182ED297AA3908CC28EEA64E80819CB1509FE54302E3E03EBC39BDF0C00A755650AF3610AF7834FFFFEF5E6A2F7B66D68634E5B1B76F2482652B785DC8051494BCE4F41867BAA07EE88C9680E19139EB9BC5128FD5A348E01EE2A7178B3B29BD0C6D6096A7DC3059DA039E2AEB64D5DDE1154BFDFCFAEE6CB6CCDDA5039C64EC3ED3A2CD58FC4F7A4228995CB240B1BA31CBEA59BF6483E100C3816BADAE7414F0E744451FF1FA0912D5C0A2472B1B7C0ABDFB175424DB48AB6F7BE4DB871788AF54960D244035D0E3FFA2FDBCF143C22001D4D7D3CE3BA848562C9CEB024D5DFB8F07689E72B0F2AD31D05890BF96F46002E204EDA434511DAA17385B81E0C513CC36F9D53A6ADB36C1B23C2E37CAC31E5997E1AC3102D4FC2C3A5B3BA3C9772541FCFFF57AE162F5F55FE21DC47A66DA2AB98233DE8EE3058635FE5F3315E7BFBA87E8DD2280930240945E972CC29CCA19060439D317C146DFF758FB183D66542ED00F2841C203095E9020D0F10F416C0B7E641F4B02A9F71F599E42678C85CEACE3EF586EC1B4C4C62D21F2CC75CFA7F58A525CB6A55868EAA31043596E2985D52F32938B304678A89C72D3B91AEB6943A4B34541DA42E1060F6DC03AB7D6AA90B2EC7A4D8B2E1FCE969A8614765F8212A08BCEAFFF50A48C2A522E1A41FC2C2CC89827830D34AD556DBCAB5E69121C22DD035437FC6AD0DD95C0E22916D5284E833CB84286DDFC47DCBE9758B980D78E4DC294A3CE5A1175F7ECBE2D103801089CA744F2BBCEE7212F64B0CCFF10D4E9643027B942134FF2B50DB9084D66857C4DF62B2C6919D9DE84D51170EFC7942B3871FD536D26310FF39AE099E0FA562FB13B0418FFDFD398657259A822A458E3688B42D492117F16E291F913566CFD6CCB097C0C45C2E5416B7D3707E93FEC60037937B3930D1BB6A6CCA5418C1D26083B3286D00EB0AED6C2B7E77028FC70896789AB29F424D91D393F1F63ED8E0D02D3A19F16B80481CAD1809BF92DB925D11651E7DC1B0499E1853865B708D79446166A1C25EDE6FFCD277689B298DA836BF52D4A885C6E8F4320AD031FFBD676378093C0F854236733B002E55636D44A5DB9ECD7358CA24A3EF7638EF03B4B5D8E79DEF61221FDE892987860DFCED702D4CA1536348FCCD99FA58C1031D934CDE4593538474B2EBC8248BD4C4792B2B63B16A152970007574969F34CACB80DC978240C9755903E9F7951ACA608B682FA60A90A8282B3E54862B60CEC8EEB737FBC31B6511052856083CAD928D2E41F226C89A3DE0FC36797714471216EA0FF8A3DC1EEE64E24AFC46A16A818B4FB212410975034F5486A4E84078F59DAD65E5D1EBD37021EEDE2B459041D4250468459D535CA974683CD4584B62B45756EF0D2C0E0F2AF7B946597DCB86129279078C4FEA6563EF1E378BEF179D65B1C8B0432660AF5AFB461478B9A9CBA980925E8035CAFE83C3DBAD397F2BDFAAA08BA05C623115CDEA9A62129D271F237D33CF1266B38F122CD7A365956B9A08D611646CBDA81AD64CF096344F9B905386E7A231526D4C5DE966263B7C8299FC7AC8DAE2CC188AC205F60A14AA108C394260A4630FCCD65AA9F56FB26425B02C6319C305B042EB9E10DBA41221299BD6DD4E02CE5E4085DB6439EF118BB45B6D4C086AF359A3C4B79E9B8BCB93CAB682EB050C745380B65E8775ACDD27F97E934A059B3D663E35BEDC977FBCBBECB3D10C2D8AE2CD8B1940968FF7CE054EA7EEBD0E81E65A372CCD133E098867EDE82DE7C58D99E908B416584645710E2B013A849FC15C209931918CBFB0FCC35D87CA63315719CF8DE8025682F123B6A4288EC654438C6D75522A7E0B80F59BBB44AB8DFCB5372597590AB7644DD1ED74162F5FF389B417981FE46B27396508B28965BC564F8B45D68DDEED1CBB0B3A50EAD14087F3B9E5D8ABD007BC38258D127EC34413DFC4F41EE0B549E106F77C1696DBF04F31A2DCA4691E71D8BDECF4F2721D98E1C8" + "5D1350C0937B6222662F3D0A6194B8F1B60D4AEAF94FB8426C91078B6C12041D" + ] + +ekCheckVectors :: [KeyCheckVector] +ekCheckVectors = + [ KeyCheckVector + "ML-KEM-512" + 116 + "8B0C8F08A734CF1801A9CB0BF1538EA8AB5754D3A618D4531AE041FB054AA46A1E70DB84D63B722BC69CB4785CEBD791745B6C85367289447E3D266374F30F517B5512C02FFDA63B519CB361F1BFF4738A38A0AB824A232CF332D717169A947FFDA532DBE9BDFB0372146056B97A066568C5AF83554BD818BB6C206695C3ADF29950023321A215DC98A531BC310BD39C22A56ECB047B53E62D6EA837DEB77974492CCD2314B6B21B11E170C4583992A3974A921420B9A033B953C69B651E46A310813891FAB0D420B147E7933C316DEFB8BBD3A9BA3883182B4827721644CA969D5563372DE3C5B9224EBDE4344BA7146198489DC20B6C079036DA8C1020918BF293B3AC0EA9F646B8003D516C394F36A6ECFC92027A92F7EB6369C280D284BAACA3838C03840CB1BEFEACCE0F04454959CE459C62E9790BCAB7CDEB37644841AE66186E9D415FFC548A34E61B75E2A204204803159AC57A50AE209A513337A7420DFAC60D47655FCF239FCB3C9111C6552C273E42E39C8CC180A14927697A3673687556A9A4640ACA90C4C66F00654A13AC7A5C25DEB2427B651B4376C6DD506258E255C7BB8D8B312EAF8552C74598D7D4B7FE388FEE92A12E3381450B948BABB0D8C351538188E7ACACDDFA38A26205C71C4CF72C06A50C83CEA12918E03E95781D03033C22717898C63BFD8A9BA3AC312E61799AE30211777D3FFC33A06ACEAA171180580BE8458A1DE556D8F027D4F95867D6C151404269C215E8C0A42D3A2829A92BF737785B6BC1620578BE53803D8B2DB2A626335561120339761214FE90267CF803C63C27350561E785CC02990B3FD7ABE193B3F1037FED871929F649BCF40D3B0C4161499DEB8324A9375D9A262D65617868D73F519CB59BE36B1B24B14C31BCBF08571A777224C45BC0B73279AA9B6B740A351845D86676C0B333FDA586BC89895DB9113ECB58FBD73540A6A8DD14978915A1D7A3164A378512DBC0B5535152CC0BC5D667470BC6946250E07162135555F0298EAB570188C1C1857A55499807FD53191775BFBB9ACFD38563FDFC2D64666569F94DC2B1BAD1D82824729C8D16292176E97AE6D8E30FE1BF83F44762D689613D105183EE6AA7933B" + True + , KeyCheckVector + "ML-KEM-512" + 117 + "024DC3A47081953A8C5691A1FAD560C7653820B4413B266882F6A2D7229765B2972347330112BF65E82F4A8C7837272B426A4E3305A8774621872B50B270C9C2314A631144E851A410E083F843575469899D102BD911108B90C475210D9EC35C86502F47D97BB0F65DD2EC0482D5000BC462C32C1B3BA769C6D407E8739850F9814035A8114AA405C086FC738EDA7A8B63CAC736E1B88C0A8F42A6A4F795A628C339C6902C73F6359952BDC3344AD8633C3BC7BE540CBAA2753EF10301ACC8059739649A7A6A23DB341845878243137B2B7DB13B5F866544BAE8C22B972AB812624E4B0648201AAFCB4652197891086C7CB0B99BE95D488AC0F74274ED63870B405A64C338400810C324536B9CA3689A81F9A553BA1450E0C10E3503935A7616D07BA9C87AA269F8526DDC41F732119FA941064C55AC7319A514BE51A587DBA0A5256A53A4868B34B80426A8A8316928A6A57EE7EA2B706C5E5ED37EF7B68D5BE75726480EA9A01C5FD8439949B85AE374486472C9A7970A84B08E997440613EB2E3AEF03B400AB06DB0B27D019576AF6984724479E78A79BBF14494016308D1713DFA9677F94C562C684ECC7CB9137D24F0AE06FA563BE66BFB648FB3AAB64203BB12CA3FBE0AC8A2272C6B26B9378B7A1693A4BFEBA351D18CBE3A838E299E0BB6C77DD13CA9791DE2D989FC61B6A0E99733C72A18575555FC7F99221127BC8272505E5B489FAB663E1A848C497BC3C6EBC8D5C23E2AA45DFF832F75894AEA99135EFC65FE5B2A1FCBC291A39040F00F01569780F2435028114C59964903247421B3CB72C85484340FCC404AA368F405CF7AB9913F81073B585793A1AB1DEA39F452BFAD241DC808367C26075409343B6C8FEB08A9BDD83FAD348FAD5B4A1E7B7EF87376DB7076992C481B0BB6CCC63C2E82CF65701503BC3B22B4BCC1F186909140B1CB19CCB863253BC2A9BA9567F53CB4F7A8A4406CC7176CC68A334BA892F816BE2DF672E4CB6628DA62F961396A1B3F50C42F764C33109A34D2E94DBD1CC43A8284AFAA79199334E54080E238A333D307655A91C07252A8224F1E457298E15EE0966B5C7C6C4824A02F117E6807B25F4F4D993F9B45B086B3D43AE092" + False + , KeyCheckVector + "ML-KEM-768" + 136 + "0616CAC3A41E537C5B0AE0A6113215A0AA8D2B12855107232D532C4A494F0661C3AC0A417DB950F0F38B47912EFF36615F2311CB3438B825426EE20D2624759AE95D54155EB11CB41AC81EAD028C1EBAA6297A208E9C0418F3954BC981E84080967145A67922764964BFA413C359C398F12889C0429477647B53BE55268E40EA76FAE5AA92720083EC7117C2B14D4A986D9948AF818E67144148DCC85F5704C3924B6640365F5076EFB1AEEC692DD0C57EE2493DF0CB3AC7341CAD495B71553487A868D7E3A26EBA81B93C1867840E7D6064D77B20F08303E3533466446BEC697996B2CA973720E50A3B09DAAC1CD06D5B73077CD803181692ABF80A8FFC415F5089CEAB51BDD60375955FA46CAF282B9E240B3FAA998B5E84A9F5D935214C5DE1A97BF45619CF632ED9EA0ED2E09C6151AA211AA381B57F1537B8802C41F3E5085A4A6D191386A6F6A6506105021C878C4803728228071C3B11E571DB15CF709518C79286D835B462EB7CCBB00E420142DD44C7CEB201DF83343BA28357B9313FFBC21523182C6443C2DAB19F0903C14CC3816787A15C06076A8535536B15E44A00B8B76A0BB2597959C503B1B7753837C9A15A415636F1012404C61E2979F2E3B26CC8294DE779BDFB3C985970F0D64343F0104061C23844B0B81B7F26C895587164A59469087B80F373A882002C8C73B901D61E529682ABC86F87A33E6AF985C0C43ADFA0364F8290C2A923CF7B60D072AF8C94495F439520EB81E4576485002415079AB705CB4C5B3F652B605F915DABA81D71EC667B03C0A7F77F3C7AB188641F3A334C3E8030977BB52FD4C54F39BAA4153E49C832F836940EB3CAEAE0BDAE634241718671D43E57CA24CBE6081ED28FCB180A419297138A4E90A49269A502A940B3F2CB5EFE1261F17B376F839805951413E3100C84C17A576F731C6FFDDA02C69652FFA40E324804C916CF3D511905626A1AD4CF25B52CAC5B0F919BBF19DC37431365F15C6B523540F81321F8A9C02109CAFBC4B7E0C19FC89B9080AAC952306F6632866EFA5CB0D0C53C49921F5996516A7BD95149890A4FD7BB7382033DFD2B5B5D010D859CC79C9A68E41855744832CE2B5916F301B7A5A627E3858B9512CBE4AFBD9049A229972520673EFC9ADBEA700334725F7CC256FC0B74EAAF0CF1BAD802BFD57A737B52748C009F46DB4A74611DA8F8A549D6BAAEC6AC51A14422C1834D199E30D210FDE062402B7ED2A46DE6A54075D1C15E8082A11CCE4FE19289A85A309A371F620398E169C75B7E4405623A5901A7F94DE7B1C240713688E36ED81C971BCB50A5E17B053B37A2D79E724CBBC2EB856FB66B997CC94D8B17C4A9729338022E1636E63BA7DD66BA288793AC72BF1EF36CB9439B19F9A2861340DB711909EBA6DD4C2219347CD68435C6B634101C4F52CA08DC3C939D509FC5382DD5512210609D51246F2D832DEC00BDF4C1AD89B45333E14403AB5A2C51CCC353427EEC746284B8ACB963AC765290417E7CF9CE9849940D1189AD55A12E3C90DE718F1FA0621F7389360B2F5B6923F841B0D58A9A5CA0CF8AF4BE7BB689E2039141FA7E67938630B55BD65C0630F1405E6C760D890EC64752DBA36B23F483770BA563B368A916CFDA68FDA5D35991B632AF5C8D8F2CACA112F926B6" + True + , KeyCheckVector + "ML-KEM-768" + 137 + "026DBCA9319A20DCBD8AA0AE721344710493091C20DE43273A821043B0CA062A37AF3A4DBE33B503D01375BA11F06A4F10C7A961746F68456B013449F3E416D0E63A40740A1123C5C1051FFDC8AC8BF9C5EA1246CA16A9F7FC1C8E750CD0C40F18584A473A0719591F1040C2D7DB72ADD962AC278A96B4836C1B6F2C33B6ACC30E4865487B1CB2EA609CEF60CD7715715ECC1D0A541A7286962C26ADBAD87F216C5C6073C579153371A767172784F5EABAE70773102B0728D422B1A8B51E09AC56F8A11F4B40F70B9D7E7380130C977563C6F50BA07641C3633363D102A11D3956CB363ECDFAC9E82467715319DE73BD82931803B09B9E52315F3B9E90D634E7B43A097B020F20613FE42AD941A88E7461D831CCE093CDB88331E8283CC0E18BE0C40F89523EBF09AD35C5C4C44708D4E028E6269BBD7721C8F5C2DFC5570F6CB9721B6D9BC303D7C525646539A809200BF3CC30F902EAC04471FB79744CB82BEACC31D97EEC934CAA13BD607717C811241E4379762464713630D4FCADC9E890A6A96450A786B5849108602951960AC9DA84F10A97B04B66E7304CF234804EDAAB8C9355BF05AF2C6346A694C87A90013779904E183D8CF7BDA6950B58FA4A78195BCC7659BB040E6A642741B90E4E27292340A7224B2CD6D53F21311A497516030606EB206EE9625FB0DB621BA10EFC604F2F696D11E667EFB24319193E16C45E317BA0E7120A0012CDD33A62356135F3E18AF837CA868294E7A92F2670492C0996538360B375294E47313AF78418AA262C96CC2986332F7107AFD50C53443C97161858363BB821625BA209DD40AAC3BC48F4C8069A2535040436F799421DA205F5B729F31A7E1469883D07BAC1B62A6CBC7811646B51134CF27B307FA3A61CF9CC0B6393C0859F91ABC734EC9AA573375674134DE65BFEDB39F5E546CBF0CD98FC5C16B36408EC058A5142035ABAE46594BB2A5FB1BACFB648515B6672AD04A88239BCFE4C4DCAB768AB9294CDA37716E62FC08C4E0C1ABA58565919E19B17B795F3A37A3475A499131FEE57B7AAF8B7FAE64207D4B2E32632E4632920278EDB864D61D4B1E8C62A2BD42DC980475E1CCF9061C64B573BAD7BA2F799C5A37054EF3586BE94056A49712CE94743C3236937053974C9B8E274F6D7A0B0D427F565A37C9AACB0242E7C2C86D01630C8627373EA951325476C1A617A845749582E93C724B4C032B8106FE4F101D99BAA5E4B9F3DFA8205279C2927834B6412B73C84E9F3A5E398AA5F0456DE89A10F381D840C19EF92B4FFD96DA90566C51AB5E8EB7ADC5444A76B59A61CBFDE1B889E744D1E389B90EB5BF1888E7C42ACCF28C5C34A63E0E36E64E99636B07AE6255F1BF980A887CD65F2467EAB30071533F4D0B1F454811C3303176A9E8D4140F57B538D89B794D7C069050637F9A7ED6BADB2A910064B21CDDA813FD4CF2CDCCDF58CBAD33C3777A930EB9AA4A2BB1CD22C7B3BA17B6905A8FE96875DE00C34757202581823250133915ACC659559215E5490263975225F29AC8ACA24CC6888A4D77927687A69344A57AC819E7780A7ABA782BA0AF0886826D20040B537719C87D99AA4B4601ACFA7CFA61BBA07ECC9C3A75CE3B01E0A163F1B8ED4D9AD2AE714111A25290CDD167D42F6F56B767583F78CAC" + False + , KeyCheckVector + "ML-KEM-1024" + 156 + "521CA0F404B0DC436D02EC4D4C783A9FE60A027716754CC83A478EF9B292C71B1B4B78038B582944AA857FEC42ECA10F1D42CF97C6790F4B26152C5328279CC5A0AC9C369DFEB82A79E498398B54357C5B6B7AA3935A5BF456865C93669E0721ACC1C1233BC75ADB9A2845BEFAD66045D93A9A1A22024688DA726B38BB84D8C246A7E67F909A5329715212EA2F8D3A4631645887E0A32DB14957A39DBDB550395352944B7D86368BD4B41F11BBABDA9360B5F7160D8A123439B6E327CE6A55BDF08ACDFA1B71C382ABCFE9209B8C960CD95C7EE8B1A9A98CAFA4242FF25D957A82EFAA926E99372203381F801E0FA1898855A1A228C533F295E342122574936CE4925DB87EEEC11ED7306B0340A44B37C1C819ACDC474ADC004FC6CB2C578A2B089CC4894BC53BC4830F3B54D73AA2C10CA7E46C322AF962E3AA6E45287538B920EDFA71356B0BAD9A0D1E0855701CA4C0E23F66AB3C7A4615A5076848D564A9A154A00958CDC9152D61A9915484E65154C4D33BF29285A2632AE16C81040B2C3618718F807FA0821DA33A7B165379EC3757ECC21FB9A08DDAD77A125476E6266D42E20EFBD99488192DD735A196A5B2D310CD10E3CF711B441C642887CBCF5133B056EB8A1E5089540BA2E4404F25B71FD787BACC2BCA73E0B9DE76CEE2664AECF5499A687C7C167F08836A15B832E69B66A1509725720543A123311159063137F25711250016AE983684112A0E5B0AC03AC28468606ACC5CDE2450AF086E386149C76015CA6ABC228948D8029B224B63A0A485518753E9271B74DB35EB080B839A174326A57E332C24F3506EE748D1C56835799EECE13C16E53D3A3966A17548526B0D9132C22788687C8BCF3288784B31539D615344ABCE04258B6F218E48A78A4732B91F75455024ACD237A3B2F143E447533786889693AC3B96492D58765ECA406317085DC659BF893B7C11CE4655B2452A389CD912BB98A63DFA37DFBB71BD58A5CD6A6094114101940CAB2B90EF215A0A356098AB06EF068002E165BC21ABAA68B58891445D434139746CB54402972B12DC75B53C471ED6951498DA464CC4BFA76B2C5E8165D6D1CC76C3506261052E411B1DB34BDB6CBBC4A8CB0EFB980691AF6508CA8AE97518EA0E689747D2CA89FDD44F017C49FDCC6DFB10465C8412B175697F052940909BE126C5D35C0E642B6D251B7D526C11407962DEDC06C8DC138DE5233C5C976265A8DA7B3683C4467497592E8BCC741449AAAC41050BAD867C36872C2119A4C7DB34067A943F0C951168252EA0FB5FC6D769F940BFFDE7C46DAB183E161E02046CC708CE817046EA8063F469148D2336432B36EFC266077AA88C85A26606AAF7EB5AB6F3B64ED95FAC5C3C6DA380D0DC93A30C437A14816C25499B29132D492C3792C1880865B3670857B1758A97B14A29B569844775444C915707E2870B79CA8695759719A34D73062F51560FFF19684F1048FF9730E2669443EA8635F303214A629B71A04E6042B06584E6386F3B396304535BB8356240353EC8EA8158C01565154BAEA1509DE657625C2731A01BF741C5242BA7B909BDD6D47175681B92898AC8872F013852546A982081895729A369F40A70073B7A59BFEE3460982307BDB163D3A05664A91961B12EB4248C190B8D980392BA0ACCD9705B4D3BA54535426F10099FA8198537ADB66656B2D2843BE15F42B93FB4E521280031B27798AFAB68EA084EC0767FB5C76F3A129D15E985AA18C899A9532DF21D3366024D9C214B338DA5891BCAF287C4940E5B3837F66A4AEFD3C4A8406532FB8FB1CB8E45691BC1E960C976501FD532AB164D20A3368B0780C71A0E68C76A59F6CFC7F6663798B102C0A9FFA90C18E33121D127F6467E49471681B186A752B63B593C02F85F18384BB8F6087E59AE3EF879178534DE3361AC4A03448486BAFC394D369F450C234208377833A5B9768E13EB4C855C7893020A8091A8A9709A13693CF44CB842A747274273989C28FE3CC0753573BFFC565BA3BE02D7348FE98CC9BC616AFAC99308805ED0A1480888168790F4FB5417F27BB7F96EC963611D2CB468A98C508140CF5267C90A21E07BC471064EFAA4476D0B95C9A16EBDC34965B3B219DC759AF65392CA3CD486A2B9744AAF5BB33CB727E5BBA176CC4FC28398CD1EF5EEE6429066EDF3FBA01694C6AFF0F246418712A4BE53E1C13A75" + True + , KeyCheckVector + "ML-KEM-1024" + 159 + "020D58896B0BC35C2D70DA8992C424899A8BA9414A1A8319A70659FCC6A3F2B794E84B753B0C8822ABCE063106F4A8706358A0A422AF8FF287EAC05A3D8AA84339996763A8F7A9833D2980F19754D18C37E8C7B46425BF8CC1A75CBB7E1DF5AC70E2056AD37F79F4AB83F87E5E32AA2E36CCA999AA9A4C82E81CC94B4AB471785661349F0FD832297A3CAB38CEFDB45FB020495AD1BC1108186BF4B31D8354422B21AE719809762132E076E5A5B692F40C704ACF1D3904D690AF79F217EA38A8545281F3309CD6EC565A5921FDEC938853A63EBA0A1E8A68187CB5A3D639D35439CE5A80985BBB9C7A9762EC14D3EB5501380D7C298F9FA177EB5A510B04D0CA7A19C763ABE4494A72A5BFC985BC403739DD8A9A13681DD39BCC93E965E4A09E44B97690C36D0D58B4F02126E4A0A07E6AC10E58C24B554ED89AAEB15C49FF53C16433835DD72FB4F69458D1BC99330891F1BE9AB172242412FF819F82377914B269B7120137375BE39B9D32112326EA5CFB5A071B479A73E0532B643B53E58ACF1A987473A3AB017A66DA9AC5946097679BBADAB1A0827783E07F7BC198903215A6777FA3B77DC264CE37636AAA48A5F4208C03636E71C7746083BB7D403035390DD78118FDFC2F7DF282287C96AFE8774A20CAA6DB69104038EDA590C648C69511811E9C28D52C3540721428336EE3D7732A287E51109BA8589227864F5C27C5DD07556B33CB4842340E5C5B027464A74734F088A591EBBC3CAA0899359E75C395A6A57670B04BB38619BBB82A27C85F063CB286160CA46C372604B794D4202EA16F8A757E9DD8A54194C3384850D58213C7533CADA33DD6704F082B4EDFA67FAFA2CAB0C89897FC4C8F2255B839CD653C4F03A728F0D25CF2B99AFDD0419C515DBDBCCA0FC6947B0A0C936AB1F2F290DCE5474183AD4DC85218EA13AFC5B671926842A15D7C367D04EA97890B371297491D21720DE148F9780866E1414EC6844EDA97AE951AFB0AB07FE61B1B856209DB6DD522792C03332645A63C460EC595AC5C49B1DE1980DDE8B4BD5505FFA7CC4749C42F0694BF3ABE4CC6ACF8A7235766513319915B58522487C7F9EB9C2754863A40036E28332265C2A4FC420D95AF87063C244233D5C27CA5B0261669386B624B5AFC992CAB773EC948B8957079B3B3FD27C16CF969005B27B4C438B7269FE317024814AE56862D18362FA47A0035339A485528540BCD27A288489C80AB890BC3992BE39249E507CCC92A250EE631297AB19E6C39AC446EA5D54C675368BD5BC4002463D8CB9A294C3E427A2940BA5F00B97FB2B1714CB6880F5B6A13F868FB936E23FC4B7EB563E2F1C5FB421A449715A1F8CA0D3C6AF726CAA5D93C255BBCB7E60843A54EBA470B32D19E3088C17AE5BEC5408ED9758639C2AE50FCC6C5471B638210F8BB1509691B312082DDA85C91B31517D78D4CF96CD94335494A66B8EC3F9B2478DC264A1B7462DEEA1C44CC0D357C00604ABCC2437404771640D01FA1A9A73E900EB0B7CC8CB3A5EF621EA0118796AC7907612F339654CB519B50A04B6C38A1AA7464E34A8E9B9029A4C1687149B0959070CCF4744A03951B4AB9F6B88D85487E2929338CC6CAC39651EF7C9B802657938B7C368C2B6B0539EC0C152EC95AB702662C21685748A968C68C150A96A9050BF3C597ADE0699930C9780287D8811A68A74AEEC8C335D4666C64AEB257C508A45E32D7A6CF86448B81486296CFEE29C3CA7995FB9B55A8D62C6BD3BE1C267627F4754DFCC1A7818051BB0448219794B87089B8969C72B01FFC168703CD11992383552E14197AB1C99AAF848231579EDDC305D4942B27957700ED5AD44797E0B5953AE06EB25CB0CDC94C6B67A423CC6700D1604E362FB35105A8E6A00ED6587A759B98E37BD358C9FFCC4465F9C5D945A93954550A2B45A6CA90A3593E07163D7A8AADA9848924B135C4911224B52FB5C2823F499F98D0238811B86B55A0D444A2B14B885A0B23F510B5B80280830B5DC705122595578FE076FF859B5FE0878E7B17932A3B99AC58D9A86812AB2006073B8D80BB940931CCFBA269D319D68CC6B1FB5F6EBC288AD208379364EFA73384A8A441C1759BCA9D90016024EB2316D50E8042CB74F81320D089F6D08008007B7031A02BF773BB1C620A73B3A1579FFDBCAEF6652400008B53F560F13D5DCC0B716895116434F2EE28" + False + ] + +dkCheckVectors :: [KeyCheckVector] +dkCheckVectors = + [ KeyCheckVector + "ML-KEM-512" + 108 + "6933455E456332A43690482EE948557981B259004078A024D7A56F45065E238918A9BC0B1817A3BA2451A5EC8EA3D2BE78528CCE1CACACA6027CB409464A74D6DB110C175095C32750E247000B4712E223C9C60455E97D4CF2CA49E85778274C24819A7F55A33E9536DC6C4E56B73429F7BDDDD71FDAA10DF9C33AD9E4656D945DF13A74F7166EFB4B0BED081D7A8A96440A6C9BC8A18E4792B25468ED10764A6C48A2EA61C786B44B06046A1A5C6AE1745B510F8974772EB30455E409E908BE563564F9AC04D79C535705C679038018983786780BB67A5246378B5E71BF00CCA66B4A98CB6A67D29B26C343C67C47A788617395E0B5A6EAC09264830BF811C87C1327F6766A96158E9813E6BA5BBD30592935834191751DB50FF66BC1A807041207144A21242B3222D1B9C37629CE4E717E442394AD15B07597C25EC096E70738D1671AED7768A059A1503144F8CB6A27A667BFEBB458D4970DA2B375D021161A9A868146555AABBF3B5FF8EA671E842F041BB48A1B2AF564166545CD03A7B473B4304449B90F934605B2407B80019DA440A81C64B47649D276B98A43CCE361560B671E7AFBA6F084BC9979718D8A4993CAAED1C6CE1494B264090992298EDDA7B2EFD181EAA76123A333DED9017157AADD9A421BC8173A62B6AAB030FB00A4CCA61ECD6B7D6095AD56FCA21C4A7538D15B652A824576663A95AE0255BB2293CFAADB6019404F052749AAB545A855316F6C865930A44F5B0F6850B0BCB5A7865A8147ACAF4E92BE72437658011B8739543C0C9A637BA99CFC48B8CBC55BB63978142BD079B8CB1808B9FBA4A41B5ED9F05B74658D92C1AAF83639F6C385B5E3396DE92CBF5AC7A65B3FE216B9F3AA183A05C693833312FC0103C216BBAC6444D75B4F50054490140164CCFF96CCE0483FC750CCA75303D657317DB387538A1EC9BB5D0C3615FCEC760DC5BB655CC2FC876D76FB838DF00CCCCBBC48E246F5287149E67AEDF5614E233A37D6BEA5235A95050AB8E8A5D9067860127A8D473BEE0AA4032761F75A5618894ACC7CB4D43C51C34009AAA2940829250323902CB4C21827C5CB034C05569677B92121303EBE773A58F83A794BA4642149F5F5762923695456BF1B27A6D4375B47F618D61643D4F59FEFC750C90ACE66108D95217FABE10397F4ADFC6145BE81775683175DD4381EF640938171B2478703E026EA021D47334506A86348676C4BA1ABE7D84DCBB268A3A2BF17897ED5ACC36E699862093DD25A20453342C9AB4947BCC41111A7FEB60302B0209E16925EFAA61C79570C918ADA33B65887039F62AF4414BF80530651A03085B249F721AAE2410FCA8C484E7B641AF7251ABC6F8AB67F28BA82FF6C51F85395BE5927A2434C7E512206F9923F1817DB3690E2315760CBA546C397136127F0B40E6ED267FC57703D7A5A3339C4E0977E9BC79E95D888DD131770762E5AE63224A9941699292DF8C4E27B41238C7D5B652E1E72890D1A6164CC3EF74A3AC26632B05986A00A533820B97B0458BA2185AEE8AA04C6B7ED2B8D62553652984BDF27022870652E8787BAC8ABEA70BF748343605CB6050143D49694A9459165FAA3ED20806D647BEB3C3962F49D0265AC05864749A63C90D303AF31C0E441077A1187B25C8054A35D03DC2AEC86C8E7CB10661319280C85826BA92AF37F20F11833A8616C64195327CEC6C651DF0B78E4B624E2EA38C499A257CA9C39D93C01C46864A32F7F60026A28C42ECC00C41C064F4C26C4F1AC13487B3CA68A082CB8F8E64C662A6445F853BF79C529035EEFCB685F83A4C2047C93895B7197CAA9A22A57020CE5698B48303A48C0B42AA302030412961381E25C2BF3866447E407A7E659FB54BD81C86E990415D635B9B3F56A5B9229FDE42DC5F6B25A08189FC8278E9C7240D555BA4A1E4BD75278B120021BB850AAC2DF6A00EED00A2F938A3BB014BC31A371A199D01A62ED05520A8C76E5E855D81B023ABC8298F82A39A2441792C5016329CAE0021B1325D32AA6EAFC6310758CBD284B7691C6D58BA97C418D88B0C3A8A909C1837917266B2B71A99E192E33671A58693D2898BC10512C1CE724F7457F2BEC9AE4E8CF1131538DE75E94C5BB1ABABBCA0058F8C3400DC478B4756377F997891AB35DD68D63F42ECD930A0AEB44C9353CBB09F5BCFD3D90171F38506F35F792D6BE40B9EEABF876C7D44CD0C5309448EC6FFFE28D5843BAF9D8CDB84BB499DE1F20B4627059BC0895E8302139A293DF352A9F8F7C1D3D7D581E12E15D5096F2535FA36A510B5B74" + True + , KeyCheckVector + "ML-KEM-512" + 106 + "7E1587F4F113783B6ED920916A17762B997D0975BC7623B1228000FD178A4B28144463C4C838C9B3D960B1355DBF512018094663C42D7922B8CD04C7FB44695E5B9E2DE46DC0C836AF62756DA16EDBD022A8581C0C655AE491A0D5176023C58AED4B5D6E7A43F3E33D7BC17A39766A0535A355BBBE0D7905BDF01EE0F7015BA1326DC093D4635BDE5155EA6831EF564752507C40476922A5A40516949998B971688A21405C6987BA15F897572AAA641B8064710C9251A9FA203FB1EBC1E8A6C6D6A46291C4165796C7E7B37D31B198F1FB6CA9121B9FB6B2DA6CA973F2C115DCAC598530996524DB29A585040DF5A407C126075252209657B693F718A0F57E2971A567CB25CD932AAB92B233C53BFAEC8738E4B3FB835240937B61F89B81C3250FBB056E4213E822B16323A164D38B7A727388CCA7026C1D519647FD038495823C600563270528FE94336114A3ECD1BF2A3CBA2927312CC5BEB644A0E4E2B16C24658830412ED43C2974341BF8086CB316B681CAB26A90D0EAAD4D3147C8BA47FFF1CA57B96C8D9C019692A6BB6698311C886E2694682B7801F89ECAB9B53DEB779D98033F214895B37C48307A809A528138A174CB3C7763C730840AD1D69FF6961339A04A88146AE64304B19147F6E27FBF7B5AFA0098C674034AFCC4C2443B75ABCA6D643CD72240B7D5C78D18A2AF9608587C43683BA9F9237A3660BC0F288167D6AB5BC27B9925975A50BB7EB7CEAA9B7EEE97A227A09C32D96CBCBB4E0834671E0C076FE5C9C8812E7123ADFAC49F8AC1C3426413AD5202E09C5FDB98582A5C1D0D50A4AC3A32E6B4361BC42D7D3A4417C0B8D4993DE51BBA1FE58B11B05644812232A7CBD786141E39190D6AB59EA79D0B09677654567BC257AE2B476D412ACCB640B87537C067BCC2052EDE550387A34039F910A9776FDB01B44A5A949CF6C3805C351782CDFB3A5D4504093599C0796833C3D55FEA9042E1A24183285B8AC460EC4C17F7C7A79B9B15D458952BBC8D7508C22ACAA0E5CA8E0F01BD2778725487460293050C22A0812C2D19D1AC9E9313F1B82F95B2191444B2DB66B571A2AD2DB831FFB151AE8147820C42A255CE6A762833A5147C4969AD18C2490928775C092379C1A6AC999BF051AE201E05120F8B82756952B6B8F347B520AE37C218EF9A630169885173C731919468C12C7E2BB34B1797A86789BCF11FAF45BFA120741B64916DC0600D067FC172403158B2F8616F876A9E603147B50971BE00460B480E68AB7697B1CEEFEA53840825F31592A9C1AB579CA4E6071D4DE8A62BB9C068E3BE30C01E6EA7A5CC3387C3C6939FF592F8F9ADF776ACA74318D33935FC831628C414E0112AF7139AFA079E7C704E444599F4906B3B07034638B01C5C89B522249B04153540839A8A01EAEBAB55DA64A4DBCE12F5A71BE4AE7C37B2EED09830041497A577F039CA04CC41EC70C6B4720AA5216D2BD292714510666B5AD7C047A1665973010A728A0498D2469AB89B26B53BEC845426DCA7246A7E8B74C429C574584B442D9A7D8CEA4F1B9959E722BF79A14D75E0203594624518A66CB8BFE161268A288252BBCD7F448B7ABA1ADA59529E12C790C263ADE51A5EE7647E76B4283933093B40EBDBB29C8B7F7C527761E1C11B260CE56A2E985CBBEC3422B76BA2CD4484F5973589081AB5BA001B75841D4730C6A19EF77AA2547A997AC10E310B0FD31A8D5A04B07C43330CA183D2886422DB6C2C502FE202462541CA30F522C66C3000ED83932588D4208A3FA9938BF8A7B204257D145737E94AC6D167DBA344DD0476EB717C55A31C9ABB6C03F82969032A49B01DDAA59DAD883527326CCDEA9FD776827C4A84F2E5B6184C8C23222392574CC5D59F3BE81C8B62386AB095550C13521BBFAED914C2D69F707A5DE6EA174E215C8DA76444BC2ED4686256D80E04FCAFD9DBACF9C904BDF347C00B58581120F557AB4BD3ACAE4169C5AB1854C13294051817564D1DFA367B55B1A1004511198775996A0AEC34FA6155B191A9B1489AB8E99DF5C35B24AB711E051A6B147574A951A67615D36C790CA88641D63E252426F6F162436C0B3EAC8BB952462513454040CD5C3B607F05D0A8362B52AB078CA26DECC06F74B39B5049A55DECA588C86386515EBE513A4FDA34949AA74CD7D801A5A0F65A36DEA426D8BC4A188321FDD70DA24920E847D0A871197C91A41421A544F4E158636F19AE7A0C21B61B1E96B4F80D7BBF3F46F13714AA6095BF961808E27395C6E5B280F285D96BDA1D977112493164A14FBA1368E0" + False + , KeyCheckVector + "ML-KEM-768" + 126 + "F629720F4168EEBC9772E040CF5885DF694EDFF1A2291133470163774BB709D93331139A60A812C70338C74970B90C38D5C16CA25245BA0847BE1CCFB6F97907971EF3DA8188EA5B765A8F6DA13CE85CB6183CC4C9A9C39EA1085D69C88AFB2CD4011114C47C67CCB73D863FC1C45D0DAC2CC48142F595578BB99D4881587A31C2FF03613AC97F76A90ABEE451C4A471E9E5820BB9A340E378AC3B8084C00BFD309BEFE14279DA8C38F540F94431175AAAA1671E29A0CA7B491DB6DB7A72933F921019CF5B8D0ED64A0803C264EAB0224362F32B4C0B12451919C1C95378BDA6B55225ABFA6C43DA0978CB830978226E68375D40BA94DA10302F888FDE8173CE21B080A86028007D2DB743FB4773ABE679515221C48A78B73014BAF5384601425E165680979C65871C7D3B3A78478E1C772A792C1F1891700068BF50C000AB1804D3D76619F600B24770EE356A807A2ACC07004EC9428443AB3345B7C2E36AA134533A727272B097F21CAF2619CD569C3BD5B431282B7ED2874FEAC4B585326421BC12B8623D724A7F9516A4217458F3E365EC42CD772A0534158FD5247F1BC92685B9AD60B9672F07005769CE4A61773293BD48298CE54397C2435E0DEA4DF39B9AADEBC98437617C0B8664DC87748170E240188068B765E3B9693CAAB1500AEE64BAA023701D5C647A844E1477A665514047537F9741B270BB46914534F84303ACAC915434C4E7B6921D6071875B358D56BABD49501766B863F929D045A71A168B9DE16F23996CA1C1CA630CB59B692DE30A0B1E975429AB136EB020437B9300E696AD868F5A92092278606C31168CF21C549033C1445996887306934EBA76519D021C67D2B893734A873403E9B4A91C7839C5320FBB6BABF551C0B410B8F84CCE1D0C0411B6A9765662E7CB4E5F9378D8E4515D847A950A551422A6779529436C49C97B6A703B9582C06A6B31387A94BA568A31BF6268E4134D1671C42A03949EF10A6380B07A60556343CB61087DBC5505FD9512A814334D46915DB7B858B51F2EF80DE446C5E2EBA473967ABA6627C3F066F3353B7DDBBE9553546849400C63BD3BF7A515700DDC874629525DA56A25355352117B8E25494F62383CD8A5AEA34522917A62E4B04A128A7234A33094448AD99782367810D8B7758E3985E0E3BC62203E367726337A9DA1909C4F1690634AC199407B65B9CD2F83C71B688FF3242929A107FA1383F141569A32C631C097D459381AD730258474CBD87889280DAF5A6E9CF45ED2E78E85B052551745085A02664046ED87667331C2FCBB474CB6289BC85A3E0AAD72A949C9458AE2B9586880BDCA52306CA483B3E711FB3409D4AA7CFCEB7CA212191B378865AA7360C629C7E375689BA16E70176189A3B0120EDA33617EE83C65C2347D8B6696EA3B93C49470570E6FC3275A59A58221753EEAAB3DE687A395C7F68824CEB61CBD2C9C1D641113574258A6C5A37BABB039C1C3042FC1796593E43F19948ECB8ABBF6D56DE00B18FF1B94E3AB9AB560AC5F96A7FBF267CF8B64A6B87DFF8409B6AC26FA06718A302BA81504329A3975E456FA86BCA8BA665BE899C8261CD7094904D97F77B8BBFB63BECFE38F582842D1C304D2094601901EC45B32AB391E263282A6C86802E1BDE88C2624823202460EF0E94F782C9CA4542248540217B462CC4671C6793DB3B83D225791BA89700152B34833969701814E502A206C9230740AA1B35B6DBC9132BCBA84EB7AB9AB06FF0512ECF87821A68C4A2CC3936BA945D6848F685ECD718222BCA9BD4B2754D1AB00B70E7ECB4F6E18C38D214F8FCC28E379846FF268BE758558CB56FDD9A2C3A64AF5B1ACCAFC1E3FC09C1F259E00D36F1F4A8C0B473FAE069AA6DCCD54E5503EC79278D8AA28A43262D80C34F6087790250B471ECA8A31F59B2D5FF59699A41CE8A72BA59368473B9A94B4078C178B0A30018036B4235B2737217D18CBC76AC99F62DB8141888C7393C608015DD9249F82B119EFF1545F355C028CAC0D04B60BD702079151150125FD68439C599F61B1A2EA75733733334BA15E6554A7D68CA83C122C4AE999264296EB60047F481AC84A1D6D128CD9B607BCF3A0FFACC78F7B889B73C6BD186CE776408DF89C2AAA6F78929EC378CC2B5C6D40773AFA21BC45DB2EED041EE0C6849AB677A8C6BEB3806B52C527A13B7D065C82F9D02FB861802B36BD3AE11B6298B5A197B3F5F85A5BCC55FB398DE11165B8CACEE26608E17C538A6BC40742CB2141CC1E15527AFC54AA4B16854A7057F1CC54CC0EB795A110FC59B5C53AB6638C999234D3E113B5D83EFDE9164AE604FA97869F5335EAB2288BC650ECF99A16C461F9146A5D047D41C76286B91DBED5BA6F63B036EA6B6F23C5B0EC193F32869307193590281FC39D6112BD41CA5372847EFCA3C47808C249CC9FA6E062362AAD8D2861B2F72509D827239B1966E7011AEA546537C27EC86390B4A126389B7F556FFF575433B70ED7F57EBACB079BF16D46E42492D98C56C7466184B22879A1BC0714B37A3CA7439948BA920B9CA0D5F8CF62FA0827D3055817BE0317B4F7A29925137268002AE577C6C8C15415F22B2E43C4345121F76C01FB9C159E623FB766AD2591412D9A2880A75D8F356904044E066C4A39381462A2A6A1AC8C4AE6B7942B413730BD32344971928DE4C634CE262C77983722AA03A15308D042A3F4A824B48710D511070331A7CC1517C6024B22A35C2103928EE2AAEE7238F123177647BB5CBB708927CD01E52A21B282CEA82F161520AC8697D637491A48236DF100EFC7A86E41B518B0B8F306B8B82581259B3E4966625C499B25498A1F546A64D286BC1BC88200A9F2241FB907C4CC962844347E58533171A090AC7C86A2D48E908C75CE0A25A28B81C9BB5423390AC9A1447FDB6CF4D68CF7FAB10F416BA3823141A088D0C8ADF928984DD0A5D495A96D51A2925C09B02B9DA2959BCBCB3C629B177A382C75F3A1D6D0A50220B82DF815AF75C9CE4A875E723C0931908C639E5892C06C4526A2103A9EF569B2250608624A8758CFE8D3A3F2DA6B64759F279750C7F199A2303F684A826B72BCBB9A8CC78280D93AC1009B58413085104156D2DA65AA8B669E6466E3F4CDB907BF75761417411268E4AD94610ABCEA560E564D65509B48945CAE43333C4B08363C22DA8477510A895B5B0443F839C67B4220A3C080E29052D8BE57C31858DA73A281508F6615BFCA2AA591A6DB290562FFA88EF2AD586F6FE0BA6643106CC1090EC080571F6AA33F728780246BFE2E3074BAD30297D6125220FF1E66C629986EA940BEE294B4FFC62908F6CAE073F555DF3658E58EAF77AC1D1737B49BE4B697E88F76D4326179BCC5120FE458C51C" + True + , KeyCheckVector + "ML-KEM-768" + 128 + "43A825CEC4A9FAD91BE6F1A402A2C6AE197595064374C62B7BEB6D8DC9C9B69C386AFC6D1615BD7F9461D7C8516B5248F7FBC8FF153EBA1A938C2CA4526A4B98A3C267D758DBEABA0741B42908149148BC442C8096BB12064B031089C551603604298DD786529843A098A67FD2B4315F265F794A2508984538C319F5E7AFB83CB69DA043D2031BD2DC2D940C899A10897033587746933D5C312A171DA98BCBCB777A22F300333A2A61535B40819706240173941BDAA3866DC03C3238465505A7C57CAA5A1685656265EC2B1DE6946C4A8B299A77511D5A1FF2475349FC2DF4E6765939B7258C2CA2C7213AC357A9E72F7E50032AC59621C10945E1B04376C5A46B84B2E7C838B20B0B3324072B0780C3814E59AF30BC9C22121F7519AE9BB2B8820602953A61FF3111151143D37C640E020549985ED863B202D7C0C7D067B5C184960574E7C8991832539BD34A68A816E2E8AE8F8BB596BCBC7B044D6EF16D05915AF12BBABAA069EC177174333780140D8949AD85B0C6CAD7679421A146F601F9A5AA56212A6B97AE81264CB8B167277347C6619B89360CE6297F5C938AFE702AA1097A70267057C69AA2F6662473447112A9AD22B15B5C6A90D28754434A07F16B79EABD129298AAA5AF6F913EFFABCF2E1886433357939053377C0DE4FA167468CE7AA7CD4700C2BEE14744A432FEC92A6D84304A78C697E1660A848D00B722EB202EB0E623B17C68E797B732C491800BC78D126EFE34207A9251D6B6316A457C1BF55AA0CC78B9D30C0C4382D901881E7C52B7E85D12830971848D95C36D9DB18FADB0C8946051C12BA02CBCC557170531F99FDDA5B7199B41D3074746B968EBEC9F2DC07D1E3B9ADF6B050F148D19030C04705D999130A9F36B8E5C4B66ACA585C052B21396E0C03D49A7A661A8C77A29508BD32D4A515D9C515DC259B5D965349608A1DD942837C09677D055160C6E3C8B700BF8155FE22186730E6A8AA97A7788DA5523A98C8283C61331052E6D80660E8821C8C4C19A7888C37663413C05B9B11C63455E8AE51DECA93710040001475E1168CA63434451A46DD946CCD8589FDC4A5CA161064435528BFA2617A8669CD9C2F393B26CDA492B1337B90484B1610A6E3CACE044106CBC5C5798B7178CC3D62B166D6722F6AC9720E918D0E0275A63172BC76ADAC957896653D7BB3380AC0EF99786478CB457FC80A28C8656691D92A3AC68F721182CA376E5B471B1C9D6667EE6470721940E63E49C0F41216702B3327924E70A03BFB5B278B0675DEC9057577581D8B0A2A2379C9861C59578F3969152D689E9AB3673F66F1E38CE6EA938D59C405B5C6A330589D6559022D373E9284F57A727749A44B1277F7FE45200CD0B31843B16D759B997973A8AA603DC61D3C62850C78EA40C1FA3DCCD6C84915FA987DBE838D5EC018EC5B527FC2ADAC0753453CA90B62E91071BC5026751718253B0BC84D3C74FBC71012224A59776DADB39E3A12D0DFB2F83D125AA0754990790BEF598CAEC547B845E16779399E827FC6B1756C29F29E91E572A3F05F53A81314C77BCA26C042350F75B42B9A629C1C45328BE769A50851A8EB2263A1425C11E59ADA0472F34526696053843BCA6288A9C9AFACC9225A08FE598C1CA31C7C7B5B3C04D8F562263666B723AB1B47927CDB7AFC704B422911952F48069F04ADBAC60867A0A8C683F5AA30D7D7CAD28A845811552E17C34BD297F19B9BAAD72A45858924EB95F3D29CD95269C55D7560A41B85A1204681049CAD04F08025C5DBC2D9D72725D67CC99413A898751011287F2C74485A6A7159878F66040CF859BABD901A21A57D4FB865BE940F75295CBE3283906C45C2C388B3A5C47BC56C7D56C4DB6ADFE8065EC7AC6781CBDA9BB1A89D211C6C68B9A4202033B109FC99B3DF191EBB523ACB34247379B5935CE4A8A56961149B7C918C2CB801B148817151A317370A7809A33068C89B4B1AC831103391AF0BA3463D2A45EE35ABE4A06753C8D2B862ED1B04047A21F8829421AE773EDFA753032271BE67DC4FCC8EC73338F295C8A67C3D5B2A5A36C6E1F65386341CB7042B504EB3B957944438A05A36CA9A1E40C0D4600F5543A15B99DBA91222FE5769F07318315B58B5CB1DB6596D93C8B6B704B12C1BBBAE6145329B92E3A966B2C7C38643A0A2C38D4824642AB90C654C57CFA96819B3AF81034F9CC6582F477E8A58342A36A6AC776FFF183FEB03D6B34CEE9211F96A77E05FCBAA0783E2F95301AB99D8356411CB6196BA2BFE6476B7719B50F2AB2BC3C92E4FA0A77121D3967CFAF9542084249557399A8EB8E7D79315F82788A614E59807C5BC1A3DD8C09980C2FB33295A31B4728EA218C3929347B73C35A5A2686B4584715017CABD0E5A35C8C87D5332C937583F4242A385A7FB67AAA77815137AC991B4656D8D93619071B57836A928326144027483799396238984C73D966055861AE19F76424999E213885D9D967C0CC828A307376F41DE38B7B57DA66355677D0FCBC4BB15E35413D7C3C35B1A9A389CBC4120415014135B135641F060B14C8946456A603181227121C5925C3A0C70210FB03FDACA7DBB73AF14AB794800A7D2BB27D8B2DCB9CB9C356A0C4E38B815CA3910036DC9748045B403968486EBA5FC2A67947B92D4F247D6DC686DC28C2E76648BD89201E273121C6B286C9B3B686406EB69FC5C458E8C7AFDB00C873E4AEDD307904600EA46981816C5B87B10260E165358147B3827C380A8AA849310B9389CAF9145711BC22E74BAF280D56991D346A1645A12E37FACC66C12700DC39CB8621B6C5C001A6181B42B2527173FFD0A91CD15719839F2703A68DC82A7BA083A051834D393F18F01C897248C1649897BBA2BEB8B0DEE66A39AB4164483041B7333EFC9D54352BB4659CCE581D447C77263C5E4DF74F5727B8392ABF2503739C5C98CD5C3CC1E562F41A1FE0664E07C17A2636959A51B17EB1919D1942CFCAA3A32A554D499B1205A60676033BCBCA8056978F98293A683E80E06D6E261F6185C8861312C642C04BD93962912EAF56672A019163DC91F0D844ADF88212F51F43601EB99B043BAAA31A9279489BB0B629A137187317802F571391C88B2E5E757A17D63EB0374F40B9A918D742A9851420B65355E3C441E048A9D51857B64E411AA217575ED8957A9F0247A7F071289BBAEDC70D6CC24E1F273646C0B9832B6DC4861638CCA434D6341C8645F5E561187AB9A478B65BF5A3EE96D7846D34AC145DB03B70BEF620D782AEF1A9D2983B3BB086D37A486C5C020B73C2544B1EA91CE5CD5AA22348DCCAF9B62F405960918CB2C22EDF7DF58216FD7F1903AEB2B10CFEFFE0ED3EF73B980E6D0C500E9898103BDE8C9D5B" + False + , KeyCheckVector + "ML-KEM-1024" + 148 + "31C48008D52C5365473A1512E7813452784A12410BAA3313EC0C0B4769C6E52CC31776074B2282E035B611AA1AC8DA6DFE63C0A1941B473644162288E53A5EC9C59037999DB3E8C6C7857CA57C42C7501182740E7C041A7404620A9C76F0F611D010703E720CB1004C20E27274B44C0CC3967C8566BB07442A40CD670C0C6869AECD22A57DA32059C19256CABF02F16AF25A3E2C876CB575C12B13871A37B61A00176885B08B158C6B86C02A15539DC264A951C9FAFB2F26DC1C65F3040EAABBA5186A2315944E4B7AE93489E896BCF8F677FB7AA24A180BFACCC19F80678EFC64F8638280B64E1FCBBC4E2672475C5B4F4689E2E6A0B16B262214BE03138364341E774466C5F52C14A5645CDBC5691C9C4D7329C596089EA8B10128BC01E8AB31A56708CA7185FAC4E8E68EF8B8096E6351E19AB2FB2A7E0EB35282198B8BCB3783C05F71B65FE7F389AC166D3E09AB767A14B3575153801A7B38C3F1B085FB28777D326954A66FDE72CCAD43C92E971E2365563F987669E84CDFD43EF9312C5458BEDF9C5EDC757F01365144BA2F4DB954FB2A6168B46F534A0277CC00390B81AB07164A14A830B69ADDB5011D37BE8F7332CBB5AC3E1C042E4373E56B6180459785F08F6311405348990FE907C6F9213321CE23591ECA58180384CEB5B876FC46BE59C559F77AAC21D22BF384901430CB5FF24D4CC2B4AAA0B9A95A4F2EFA60D396040D4315C83911220947A9E0CA7D2C081CC6810370602FE38B1AA9711A87B4331B7F0D4576D1F20A16F0B6EDE5767356419739777C5380FA0411C1052581F412E0C050AB9393D997BEA1E51AE6E32539795E66C46D9C5A04A5B56A8EF808D8AB8CA7D82609E727CC1029FD8002E4E551A7CB05DB27841BDC3B634922B1000D4395AD76039A80745EE72A4A6FA12BEDE61A35D58AE7752B3924CAD762351E598C10354574532583C07B82B4BFACF88C18D12674EB1F3F28CF25DA43BB6541CA6691579651828372C90067E379A982E623BFD062087138D31957F05136ADE1173E0390A62B07EC061918E34089543B9D1C4170F40B1A562C6A11402CA00A3715A41697BF5D8C2E570C5F6AC89D04698F9241CD135C2FA69771A3B9CF1BB793D1E6C945C97C7665658753C31A00778F72C89E30BC2AF746AD23AC21B40E299B57B5D44E55335EDC0A3D1EA42D8C7855166B4CD0229F6CB2A4F9435F412137AA904BCD0ACCEA09B8ED10C9922594BD756E70873B1AC52A575C1210D0A4CD80AB11B7200BC373F6C208D725605207C04D8492F8C9C5E6C1CFF023AE8BA547BE1743582044E5D0C4F1F20534A0A62717AD60D91A6C7A03BCE2721B7503E75581A23BBD8F1848C00AB5509466F144B8FFD5C5EC1370EC8B7C960755A645C4CF23BF4CE9A19E3134D2E5830706910DA7B0CD9AB92D145F2591CA2DC982E6E66716B7754A82BB335709A5249F88BAA71C895BFF403340CBCF70768CAA563F5E9348F4B3C6F29B4C77141FC5D52AA532287876590B15AAFDA8289B9C8FD4EBB0EF8944BB99A0CF83661D78727854B0950A29715A4AA8732B65A08E6BC81F123358CBC3C2E4E93FD2307895597D6D66B16ED139E8B2B260B63E14720BDEA963F8B537A85A1A455BB176D310F9855F6AB019A4D919EB1453C8CA1028260C64513584108DF2D56B6451172B674270744328CACE2E167359FB594106630C887973B35776EC50FD030641312A623908DEC6C81FA573D13C475981ABC5D30F77D4B74999209F4A565C1233E3CB702897379D39B8B68A43C4199D7F64889281BCDFA47ECDB1312D981CB9AB8AAD764AB6EB88216CAE97D93D75E3CF1E654A21209246023B42376344D909C651BA92A4A9CF1CB03A522632B41EC27C8F180CD05A230A2F60A12110C8D832372EF1A1A3BA1769FB49888728BD080DC2E476971756A4E8AAD5C18AA807D0D708714CAB2BF4272C458C20DAE8B089A60945B4B5C5697F31BC4C4DFB1494830323D962C8520A6F7C6C11F8613BE83C22B362A352B47B2B0838392A2DC4745068211E238DC385211BF048ECB2478946C9C2B837CBD0604FB241FDC0445008013DA6480B826EF484AF939C4A399101D380CB55C17E64B081C201D016261407A6C95C570E7BB08C79BA4BBBB92260388342789E7D7784A0E80E602A7EFD97CF79202F38E8A1C0595DD59C97B7B559D344517FD472C8B673FE5012CA935E6DD59E70614ED7C86244132F8771AD2D1A9A884570F40AB82317BBFCC5879844A6C0C757236C94E8D6923906C065B675B54053176B4C6AD2AAB8A5AEC323CCD68A5AF8DB0AE94BB4D431CA1EF198851BBFF64B4EF96C6D0EA045F82C9B7D48337D2A3F1EA7185C2881A4E267DFF8407748C03FCAC286F302A16173128396D76A9ACB217F29353AA87001EA9663D8F48B1E4B0D9AD6CBF81A6B600B763B75493577B85694AF8FA2729323CF5341961F423770217F34466A9115929447AECFD741E39C6796C5A132E831AF3CC2B70A434524C764C9C783EA1D4F269BA091575139C092D91BC46C4402CA0C65B29588B8A34E7C5744FB38189279A507CC9A810588F8166F95833183618AE2470BB25AE12B330AECB4F296B11E8C0EE2E34F12E2ACD0C891CC136D033A17E218A7FFAA7D96807267B083C1CAA8E1D162283AAFD98C5AC1657883CCCDA04A9AC8428B5EA6BFA4B04336D6A6D222B79714CC9696C8CD62B286C04F7AF69F0FDB7ADFC999DA613BC8F6A156161C2E135E7656938EF7C80489B0DFB7AEE086C9E66446EF1173569A8A7E23A0B82B92BC50250CF577236199322B783D6B8C5AB82C9C378716B231C4850ACFCC131F03CA51055A0030CFDCA874CA08BB6E3462FE517DD5F282EC0C5DAEC64414C4534C264D68DB5C2085CA25644059B2AFFCA64BB1D8762A293D42B79E4A9BAAE8D73E1012C114702D2BD19629D646B5300B0500167F3C9B1A7145FCD95BC712BBEC42731D321D7DC8C4C361C271DB1863759B147BA14A524690625823DB8772CA0E2A45AB7C734A1FF0951156A305D70B458AB422A6557AB423641266464A88799987B72572E60A79D0732C84049683310EE475B695556C4A65615651734F56A90D25178D604CD058230E3338FA94003FFC48AE600D2660212D648FA8062B38182006030857882B26E62BF8A23CD5D275B5517F4EA736A298402F155B2141A3AA8A74E5F69678185EB4334FBC380980E850A1399AFA0C4EDB172EFE326BCAFAA3A0DCC07575455C084442E2B5D0105C39B234B409A5BDA7C33C2B5D2994B7CFB90CFA23BA2A65C28E255F0BA3728671AB9E410642D1812D0969C88C721D867E36B8B4808CBD1898B05B563FFC23A306136100975AED0A0DBC1AA8E823C7FC402821F65E31972C3FDCCC96F55B16C3AD48BC4A9CAA05B4C0CB7E863A47296D3C727D7B3B90491364430CCC4756A2203150958770FE354E0F8803D1A64E75A309F7EA01260345AA1C3E5FFB863BAA4B0E5768F20B7D906766F8B86ADD6066CC485FA5C9141D16039E5562E433B511092C1D18C50F7186642152C6986100BA728C136DC28B237E2C8347B28E3FB07C0AE48A42F955476340BD5CBC145860CA6449C597920575AC49C0317F2926BEB2BA7396BD73864B86490E1EBC8A03475A9FF082A7F967F509C8F8BB0BFD4C98174B0B36D06CAC5789919C02B3C0162AD4C4DF639207416BBB5071E5F692668B5D40AC2131988535C68E68C227810C0B68E1B72103B61D98A32A7A05AFE1BB737627955A2276921703A5B0FDE38E415C4E752784FCBBC432A53BC4FC0DB52A46C3D0C3014041C870C00041C183D21D5BF277ADBCAA73D6241CEB018E5550A8E24FA5CAC9310B6735EB5D245239F35B06E803697186680B6AA8BA1730B4CBA23E105565F37B2B16070C3B0E77A965CF577F3C3A82A5192C4C3BB52373C54034B872604ED4A833F9E18E389A4BCCCC6C5991CB49F097771CABD8AA6D53020875080E91D198F93C2EFD276C86586A9E53AC1AA119851C06BE12127AAC3D662AB5325B729F38CAC505A3D0E135CD9A38A8069A7A2A76C5B39B30C7AAACD666EB061093B5C41E720B580A2914552AF509AE2035329D28050C5016AA06AE5EC62E17D84C9024494049816AB32DCF204015A092AEF05857794861175B19A72C9687B6E9EA385963BE3E6A197627359728BF47D4CEDA32779EC51B2F6BC8287241C7B39474805B5AD13DB6C47627AA40F91CCB6CD062DDC58FEA6A22EC98A65F73A0054B3E211ACA6904BC01A9A04B042AC128910B793448A98D19EC1A48DCC451B00F66629AF4DBC2C24101F9480C390226F7B184FF307BBA63B6A1D5B1FA4281AF4C358052CDFDE37B07B2C7627429859EFC72C31BFCA310E4A63E25B10BD4A57CA6A2611E141A8C0CFDE7A700EA3B849017A99F79273DC7FC7702D269197D22112996A9AAAD6803C7CF2AC8132D84A9B5016EFD34D64543E136993D98D12FDA" + True + , KeyCheckVector + "ML-KEM-1024" + 146 + "B94C8F189B7CEC12B6981C64BE0509E6A2C6F687445732862C984690FACF0AD6817C863F4AA45F9D87C0ED432EB1318354A60BA8EA823E9838FC1B08A9D7C5569976BC624378882CAD589586C95B27C6448819AD8F59A10B60B645BA69C9479F604747D2DA20827B6255AC01CDEB71F5B157729872CCB5A1DDDC715D0768B104AFBF279024366E92954FF45A06DE32C87818033267611B2B9F20A2246DF979FDE2C595929910803A2EB8B4193B029FD51594015A2CB48DEDE90D8913B118F50C9CE4AD1A6B3FDA8B6EECE6673EFB3F7B229CAC7B5B23E86ABCBC2E58E5CACE626FFE040D2EC83A4D457EE014B780727474E51B93A71FEF1545D3C913384A6375E62AC54404809C2BA3245678014D636A435AE50FEB45124AF16518851F2B121FBB982170E5AB454B53AD8A9D3802A400D13A8A724A8A1B9F6F49C30D059695D520BA73A370FAB8F042A3714AC400714F66328E0CD473E15B71AA62773637BDC4B29BBA1A07EE15BC73839B7F08B9EBD4CB36D2427E3CCC92266F19EBA7B6737FCC58848BF0AF174613A4316D7933651FB39625496C26CA9705E48BA105046E8A226DB181543A0C26B455482618460680F522C92F3549F0B87A14915D0FF15C6BA5862C556122311A4DDB07B5248C3ED995F4AC80D015B3E6A71C97226BA61C189A460D6D368F587825EC51C50669269B8B8872B9933FC1BAB5472D90AC8D36E453242826F7A7319FC8C262795947C2810104B98023AC84E854BA59A3319C6EE6905E36D0C3B0B41DFB393CE378046FE292097C528EBB72B91B20C637C531D393BFC43D54692AE4D281E9E637137463A0701D286AA89ED464499A1458913C586A2FAB90506B8097E061C0B710B68EBCACD8353C7CCC0AD37A0EEF32A4CD1232B69AAFCE9AA5620B953BE52EC7594D68D57976EB1B38CC677885840A463C72C3CF17445AE4202AEFFA4A7EE0275134162C4BC3A82BAC03864D39C9406DF72EE60218D83C0E0EDC067A1592D93A8FA32B84CA85BC51D15BFFBA17F7D740143A068879362DE00669A0CF7EF70C604299DC3CB52F023E56F544E3E35CFC9739A64A9392C0AD1D927D62F09D96418F3D926E227C26E02CB1E2C10B89E373A1D515D39C18F259212E5C14A7678A3A761F18E488619C46D2A21590270FE511ACB7C3BB45F9C699E63596DA08827458142791893070D6A130182430665A3152F62328DC4C2294C874F293331604298675A4C2C31F79ADEF67C85329082DDBB222DA4DD0C11B6FD8C9234ACAF4D40679C82BD0479EEA17CCFFD209409C5C2AF148A87C006B65B39436BA5CC6440812ABF7B08364764BEA966982979EB55684D1433E71A471FFF73AD423554514C144CC4377BA07197CA72EA6CC69E215488CB91C7980EF74CBB5E780931A34CDE964DBF21E3BC266556281897530361BB3F8B82CB5415FFFA1CF8E32B12E2124DCE774D6D692E0F803CA357536101CA5944D4E1A063679B0937ABE6BA20A8CCC75FDC03AD4BA2F61522CB8F565B05232E2FC104AE09EA156898E8B2D3D0A7FFBF195E9D1A623E73AF87181B4D932AB259B7D2B3814E721D499950D184E56F109CB1106E9E1062888B6D3185DC30AAA952946C81B6FB83B507773877F7C84EF142C235241E2901D72CC8A6F481A19721A9E877BE36369B4F614E598698083906061074D6055DF372510C543458CCBD9F10881D209176396EDDCB8118B3E5A22A989A56C67912721B290868A75F6D820802A0FCBF00AEE135139D8A1868745FDC22D06F05748D323875187FF298E2D37240AD93EF7A8A3CB462665209DE6F1990828830A65446E2A4C30958BE3AA2D4F192D673A8E99497084544122A56BE4F6BF6BD72E98894E5B696A6A19CCA2C1C79FA976384C3670B8465C2ACBDB46C1B5881F5BFA3480675F83F5535606CB393934A6DBA30469282A3127B5B77BD7709350091FFF453CF8F76370F847ABA2B78131093FE4B699D342BE220E085458E150B081B81FCDD35CDFAC605FBC4F9757C24D7AC542E90EA4E9255B5B29F78479AB36AF1BD691D005AB401269BC9395C9986ED9A2017E3818B1F87DF9846F1EA1A18EB681A0E561F9F379214A6CE44334CDC5C4D489436A5850B272499D32165D10CC8B6577965C56BB55034181BCD7C81488BB9FB1DC8D13C1536870BFD4868F136CBC490B62C79741B273ABA970742D7012AA2165C297A844AB4C27414360B5CDF0E29D31B5CD802CA3B8149B00DA23F7E213A736017C2B9D8846C12E4A5DB12B57E83168D8D6BEF82CBA020116758CB06BF694FA186000458BDA4358C537ADBFAC277FEA5C02E1C8B79998AF6C4F1A6CA47085022CA7A6CAAB0780254FD8C0CB78223D86B8656E56231E9A28F0612747E2075E25B602895FE7C498197A62541420CF839C31B347F478BCED325336BCAEBEE986D52C7AA5B5CA56723F1641CA3480A040D1267A407FE3E92D2C8023DE00846AD79B0AA89CC1F99965E88DBDE06B66415E36D2120E2400F435C3145B57EB28C62C248FEE401525FB3BE6491E02D4156EA38C819AC59D55CAAE14A7D1D3B9E2CB57B7BB58704A8C2D239E622A0D15A589916517768387E21408E0E81A01216A5D4A636686321B919E7019A1697B886E81A376391EDAD6C5E8044F6BAC63A84B02104221F19AA8E3468579A0A156897FEFA66A11A37FDF46088EA586AF8804A9854884597A7327A733315E020693DBC62BF4A33E1DEB74E91437EA30670E7430B7E1596611A10C3595C3AA14F6337B55CA5D0B953068D52EAEB7415BF18E88D7905945CCF3140A75263A25D931F802702EEB40A944BA6AF03FA2509ACFE753EC60CDB0A845CA5131EAD4401E2C8D0A32306FFCC7DD189FA367AB6A4A9A6DD1806BA70AA4440BC80B99D0C91C7931871B436F94B39B1D04448D18085C5AA946E642CE04143670BF7F8514A0797C6FA71BE4BB1416A0B3248075251CB1A5B79E36862AF4947C76D82CE8363B89139808C4CBD14B3C4F1700160B61E5648FB954C25009BACFA265F1A0564673AF000120D93A2D4E211CAACB1D69713D4E183268803163C859A71B92D9117FF807089F47940C57955E3343506762B56272AEE182C719A5478444F29513C75AB44E3C4F0658111473CAB33933C5E2999BD01C5DCB71F1F82F6CA57B9EB6376C388C20064944E3AA803574FD816C119A312140117BA492A3D3204541335C082CAD440F0B22BC55A65E2B852E3EB1042ED046E8A9B755773425C920D2F4260CB29D1DB76A879B36283BCFA8598F799302DA5C14C5AC03230A338EE3C8497899A92CBE645B3D9E91499F441FAA5211D87162E76262BD795409559B294A0453F028A3C737962AA6B9969D8B45A934C7CF22FC2EF51C661541202085554BBB5AA2B207D161A75B7831000B8A532C76C1D01FF3A359212BBF79137B42839A217B009702192A92470F4B3DA4A9397E51CEB6F883C5168DF122076CA783C0C2BBBF572680E5268CFBA41E8024EC58A133A8934C46553302753F39AE7E4B9311A984CDC68360293D94D8A6D857318D5B257291685C5068311839A489485C64AFB7430376108502279FD47042B7216E8F7616EBB408CE4C3876548AFDA6B31596C21EF096B92782D5D745885106C3770E1E04C1DA2A027912137CBA3FFD2632B808AD0BFA8F7D2C9AB4E24245C8131AC5167684190583AD527A25BD8369FD747F29528C61287EEED97D6CCBBF791AC8A90AB9A5E19F57482A89E3A9DE9C00CCBB9884C80F7CC7680B654FE1A3CA5A160431A1C42CA445AF7A4D60CCA7F6A27A7B919C6D70B030025D86341943897296D61B0C8414A696C95F70337C54BDD04C08A3FBC5F9835B9A080496A1308E03A88E76973C85A9DA0C0340B95C29228DE1FB3D4CD96A9204848A92A3C912B0BD8817F6212242D06CFE803E91B656661A51A6A1841DB00F1F8A02518A971F61199F93712C306CB944C52B469B04E76389E71543896A410979009A2A9F09CC58C0868294A72EA8A0FF96A43F543E5E96AA1934545F755382859F962615B8B546D5609B59157DB4CA4D5FB87366E03ED93A2D3E223590A3B9F03C95327A35D65C5B957546D3F0745A31A3AA012719340F13992D8BACB18153A83D59295BC497EFC5C875C4542B430285F54FE245558538B15CFC03B69889E2EA1B1E11CA89D165C9ECC1278C2D5E878C82C13D5AFB7C2DEA9D71437924D97D698180C0B626E683A2AD3BCA5652C9E3021ADF322BBDCB6D9C762D0A12270505A811E3C27FB16A9493047755163DF828E0C8B0CAB4B32EF17C784B0BC1A1922967112122792B20A2917A7FC46016556611416675AB9C5AFF696F80D48B659ACCE23FF815C80547A9762CC0D85C3F2501DD291BA56692474A2E13B3EEB40A73D7BA5E222A09506341F00EAA4E856209AC3525E24569BEA949B03B755A36972911F56028D4EA3F21B1537B186287A191FA400689D618C2C2C20CC091EF65" + False + ] diff --git a/tests/PubKey/P256Spec.hs b/tests/PubKey/P256Spec.hs --- a/tests/PubKey/P256Spec.hs +++ b/tests/PubKey/P256Spec.hs @@ -157,6 +157,21 @@ [ eqTest "scalarZero" P256.scalarZero inv0 , eqTest "scalarN" P256.scalarZero invN ] + -- The same two for the variable-time inverse, which is exported and + -- which scalarFromBinary will happily hand a zero. It used not to + -- return at all on that: in the binary extended Euclid below it, zero + -- stays even and is halved forever, and the loop's only exit is in + -- the branch both operands must be odd to reach. Not even + -- System.Timeout gets a program out of that, the hang being inside a + -- foreign call. The properties above step around it with a + -- precondition; this one walks into it. + prop "inv-zero" $ + let inv0 = P256.scalarInv P256.scalarZero + invN = P256.scalarInv P256.scalarN + in propertyHold + [ eqTest "scalarZero" P256.scalarZero inv0 + , eqTest "scalarN" P256.scalarZero invN + ] describe "point" $ do prop "marshalling" $ \rx ry -> let p = P256.pointFromIntegers (unP256 rx, unP256 ry) @@ -191,6 +206,19 @@ prop "point-add-inverse" propertyPointAddInverse prop "point-negate" propertyPointNegate prop "point-mul" propertyPointMul + -- A signed window can reach the last addition with the accumulator + -- equal to the very point it is adding, which the formulas cannot + -- do: they answer the infinity where the truth is twice that point. + -- Which scalars do it depends on the window and on the order mod 64; + -- for the five-bit window here it is 30 alone, and the sweep that + -- found it covered every scalar below a million and every one within + -- a million of the order. The neighbours are here because they are + -- the family it came from. + describe "point-mul-small-scalars" $ + sequence_ + [ it (show k) (casePointMulSmall k) + | k <- [1 .. 70] ++ [2 ^ (32 :: Int), 2 ^ (64 :: Int)] + ] prop "infinity" $ let gN = P256.toPoint P256.scalarN g1 = P256.pointBase @@ -198,7 +226,18 @@ [ eqTest "zero" True (P256.pointIsAtInfinity gN) , eqTest "base" False (P256.pointIsAtInfinity g1) ] + -- The variable-point multiplication is what the vendored assembly + -- replaces where there is any, so say out loud what the two ends of + -- the scalar range do on a point that is not the base one. + it "point-mul-order" $ + P256.pointIsAtInfinity (P256.pointMul P256.scalarN point7) + `shouldBe` True + it "point-mul-order-minus-one" $ + P256.pointMul (unP256Scalar (P256Scalar (curveN - 1))) point7 + `shouldBe` P256.pointNegate point7 where + point7 = P256.toPoint (unP256Scalar (P256Scalar 7)) + casePointIsValid pointTuple = let s = P256.pointFromIntegers pointTuple in P256.pointIsValid s `shouldBe` True @@ -239,6 +278,13 @@ [ eqTest "p256" pR (P256.pointMul (unP256Scalar s) p) , eqTest "ecc" peR (pointP256ToECC pR) ] + + -- k * (7 * G), against the reference implementation. + casePointMulSmall k = + let base = P256.toPoint (unP256Scalar (P256Scalar 7)) + baseE = ECC.pointMul curve 7 curveGen + got = P256.pointMul (unP256Scalar (P256Scalar k)) base + in ECC.pointMul curve k baseE `propertyEq` pointP256ToECC got pointInfinity :: P256.Point pointInfinity = P256.pointFromIntegers (0, 0) diff --git a/tests/PubKey/PSSSpec.hs b/tests/PubKey/PSSSpec.hs --- a/tests/PubKey/PSSSpec.hs +++ b/tests/PubKey/PSSSpec.hs @@ -2,9 +2,14 @@ module PubKey.PSSSpec (spec) where +import Crypto.Number.Basic (numBits) import Crypto.Number.Serialize (i2ospOf_, os2ip) import Crypto.PubKey.RSA +import Crypto.PubKey.RSA.Prim (dp, ep) import qualified Crypto.PubKey.RSA.PSS as PSS +import qualified Data.ByteString as B +import qualified Data.Bits as Bits +import Data.Word (Word8) import Imports @@ -494,10 +499,61 @@ , os2ip sg + modulus < 2 ^ (8 * k) ] +-- | RFC 8017 9.1.2 step 6: the leftmost @8*emLen - emBits@ bits of the +-- leftmost octet of maskedDB have to be zero. Step 9 clears them in DB, +-- and clearing is not checking -- an encoding with one of them set used to +-- verify as though it were sound, because the bit that made it wrong was +-- thrown away before anything looked at it. +-- +-- Only the signer can produce such a thing, since it takes the private key +-- to sign a chosen encoding, so this is conformance rather than forgery. +-- The vectors are walked for one whose altered encoding stays below the +-- modulus, as the signature range tests above do, because an encoding at or +-- past it says nothing. +step6Tests :: Spec +step6Tests = describe "an encoding with a bit outside emBits set" $ do + it "the honest signature verifies, key 1024" $ + verifies rsaKey1 (fst (altered rsaKey1 vectorsKey1)) `shouldBe` True + it "and the altered one does not, key 1024" $ + verifies rsaKey1 (snd (altered rsaKey1 vectorsKey1)) `shouldBe` False + it "the honest signature verifies, key 1026" $ + verifies rsaKey3 (fst (altered rsaKey3 vectorsKey3)) `shouldBe` True + it "and the altered one does not, key 1026" $ + verifies rsaKey3 (snd (altered rsaKey3 vectorsKey3)) `shouldBe` False + it "key 1025 has no bits outside emBits to set" $ + forbidden (numBits (public_n (private_pub rsaKey2))) `shouldBe` 0 + where + verifies key (v, sg) = + PSS.verify PSS.defaultPSSParamsSHA1 (private_pub key) (message v) sg + + -- the bits of the leftmost octet the standard requires to be zero + forbidden bits = Bits.complement mask + where + mask = if sh > 0 then 0xff `Bits.shiftR` (8 - sh) else 0xff :: Word8 + sh = (bits - 1) Bits..&. 0x7 + + -- the first vector whose encoding, with a forbidden bit set, is still + -- below the modulus, paired as (honest, altered) + altered key vs = firstVector + [ ((v, signature v), (v, dp Nothing key em')) + | v <- vs + , let pub = private_pub key + em = ep pub (signature v) + bit = lowestSet (forbidden (numBits (public_n pub))) + em' = B.cons (B.head em Bits..|. bit) (B.tail em) + , B.head em Bits..&. forbidden (numBits (public_n pub)) == 0 + , os2ip em' < public_n pub + ] + + -- the forbidden bit worth setting is the lowest of them: it is the one + -- that adds least, and an encoding at or past the modulus proves nothing + lowestSet w = minimum ([2 ^ i | i <- [0 .. 7 :: Int], Bits.testBit w i]) + spec :: Spec spec = describe "RSA-PSS" $ do signatureRangeTests + step6Tests describe "signature internal" $ do doSignTest rsaKeyInt katZero vectorInt describe "verify internal" $ do diff --git a/tests/PubKey/RSASpec.hs b/tests/PubKey/RSASpec.hs --- a/tests/PubKey/RSASpec.hs +++ b/tests/PubKey/RSASpec.hs @@ -1,15 +1,26 @@ +-- The properties below hold the new entry points against sign, signSafer +-- and verify, which are deprecated as of this release. Comparing against +-- them is the point, so the warning is off here and nowhere else. +{-# OPTIONS_GHC -Wno-deprecations #-} +{-# LANGUAGE ExistentialQuantification #-} {-# LANGUAGE OverloadedStrings #-} module PubKey.RSASpec (spec) where +import Control.Concurrent (forkIO) +import Control.Concurrent.MVar import Crypto.Hash import Crypto.Number.ModArithmetic (inverse) import Crypto.Number.Serialize (i2osp, i2ospOf_, os2ip) import qualified Crypto.PubKey.RSA as RSA +import qualified Crypto.PubKey.RSA.Blinded as RSA +import qualified Crypto.PubKey.RSA.OAEP as OAEP import qualified Crypto.PubKey.RSA.PKCS15 as RSA +import qualified Crypto.PubKey.RSA.PSS as PSS import Crypto.PubKey.RSA.Prim (ep) import qualified Crypto.PubKey.RSA.Prim as Prim import qualified Data.ByteString as B +import Data.Bits ((.|.)) import Data.Either import Imports @@ -130,20 +141,26 @@ -- and @s + n@ verify just as well as @s@ itself. doMalleabilityTest :: Show a => a -> VectorRSA -> Spec doMalleabilityTest i vector = - describe (show i) $ do + describe (show i) $ mapM_ checks [("verify", verify'), ("verifyDigest", verifyDigest')] + where + -- Both routes, and absolutely rather than against each other: they + -- share the checks of RFC 8017, so a test that only made them agree + -- would pass with those checks gone. + checks (name, v) = describe name $ do it "the signature itself verifies" $ - verify' s `shouldBe` True + v s `shouldBe` True it "a leading zero octet is rejected" $ - verify' (B.cons 0 s) `shouldBe` False + v (B.cons 0 s) `shouldBe` False it "a trailing zero octet is rejected" $ - verify' (B.snoc s 0) `shouldBe` False + v (B.snoc s 0) `shouldBe` False it "s + n is rejected" $ - verify' (i2osp (os2ip s + n vector)) `shouldBe` False + v (i2osp (os2ip s + n vector)) `shouldBe` False it "an empty signature is rejected" $ - verify' B.empty `shouldBe` False - where + v B.empty `shouldBe` False s = fromRight (error "doMalleabilityTest") $ sig vector verify' = RSA.verify (Just SHA1) (vectorToPublic vector) (msg vector) + verifyDigest' = + RSA.verifyDigest (vectorToPublic vector) (hashWith SHA1 (msg vector)) -- | The checks RFC 8017 section 7.2.2 puts on an EME-PKCS1-v1_5 block: the -- leading @00 02@, a padding string of at least eight nonzero octets, and the @@ -225,6 +242,68 @@ , os2ip ct + modulus < 2 ^ (8 * k) ] +-- | The public-key operation takes bytes to the C and back where it can, and +-- the 'Integer' path where it cannot: a representative not below the +-- modulus, or one of another length. Either way the answer is what plain +-- squaring gives, which shares nothing with the C. +publicOperationTests :: Spec +publicOperationTests = describe "public-key operation" $ do + prop "agrees with plain squaring" $ + forAll (elements [128, 256, 384]) $ \k -> + forAll (arbitraryBS k) $ \nb -> + forAll (arbitraryBS k) $ \m -> + let pk = keyOf k nb in ep pk m === expected pk m + prop "agrees at the edges of the modulus and of the length" $ + forAll (elements [128, 256, 384]) $ \k -> + forAll (arbitraryBS k) $ \nb -> + let pk = keyOf k nb + nn = RSA.public_n pk + edges = + [i2ospOf_ k x | x <- [0, 1, nn - 1, nn, nn + 1], x < 2 ^ (8 * k)] + ++ [i2ospOf_ (k - 1) (nn `div` 256), B.cons 0 (i2ospOf_ k (nn - 1))] + in [m | m <- edges, ep pk m /= expected pk m] === [] + where + -- an odd modulus of exactly k bytes, which is all the operation needs + keyOf k nb = + RSA.PublicKey k (os2ip nb .|. (2 ^ (8 * k - 1)) .|. 1) 0x10001 + -- b^65537 as b squared sixteen times, times b + expected pk m = i2ospOf_ (RSA.public_size pk) (squared 16 b * b `mod` nn) :: ByteString + where + nn = RSA.public_n pk + b = os2ip m `mod` nn + squared :: Int -> Integer -> Integer + squared 0 x = x + squared i x = squared (i - 1) (x * x `mod` nn) + +-- | A key whose CRT values are off by one stands in for a fault in one half +-- of the computation, which would otherwise hand out a signature whose gcd +-- with n is a prime. Every private-key path has to answer as the sound key. +crtFaultTests :: Spec +crtFaultTests = describe "a fault in one CRT half" $ + forM_ faults $ \(name, bad) -> describe name $ do + it "signs what the sound key signs" $ + RSA.sign Nothing (Just SHA256) bad m `shouldBe` RSA.sign Nothing (Just SHA256) key m + prop "signs with a blinder what the sound key signs" $ \testDRG -> + let blinder = withTestDRG testDRG $ RSA.generateBlinder (RSA.public_n pub) + in RSA.sign (Just blinder) (Just SHA256) bad m + === RSA.sign Nothing (Just SHA256) key m + prop "signs through signSafer what the sound key signs" $ \testDRG -> + withTestDRG testDRG (RSA.signSafer (Just SHA256) bad m) + === RSA.sign Nothing (Just SHA256) key m + it "decrypts what it was given" $ do + c <- either (error . show) id <$> RSA.encrypt pub m + RSA.decrypt Nothing bad c `shouldBe` (Right m :: Either RSA.Error ByteString) + where + vector = firstVector vectorsSHA1 + key = vectorToPrivate vector + pub = vectorToPublic vector + m = "a message to sign" :: ByteString + faults = + [ ("dP", key{RSA.private_dP = RSA.private_dP key + 1}) + , ("dQ", key{RSA.private_dQ = RSA.private_dQ key + 1}) + , ("qinv", key{RSA.private_qinv = RSA.private_qinv key + 1}) + ] + -- | Building a key from its two primes has to arrive at the key the vectors -- carry -- the private exponent, both of its halves, and the inverse of one -- prime modulo the other, which is the part worked out without the extended @@ -287,6 +366,239 @@ ] exponents = [3, 5, 17, 257, 65537, 9, 15, 2] + +-- | Every algorithm with a 'RSA.HashAlgorithmASN1' instance, so that a +-- property quantifies over the lot of them rather than over the two or +-- three somebody thought of. Adding an instance and forgetting to add it +-- here leaves a hole, which is why the count is asserted below. +data SomeHashASN1 + = forall hashAlg. + (RSA.HashAlgorithmASN1 hashAlg, Show hashAlg) => + SomeHashASN1 hashAlg + +instance Show SomeHashASN1 where + show (SomeHashASN1 h) = show h + +instance Arbitrary SomeHashASN1 where + arbitrary = elements allHashASN1 + +allHashASN1 :: [SomeHashASN1] +allHashASN1 = + [ SomeHashASN1 MD2 + , SomeHashASN1 MD5 + , SomeHashASN1 SHA1 + , SomeHashASN1 SHA224 + , SomeHashASN1 SHA256 + , SomeHashASN1 SHA384 + , SomeHashASN1 SHA512 + , SomeHashASN1 SHA512t_224 + , SomeHashASN1 SHA512t_256 + , SomeHashASN1 SHA3_224 + , SomeHashASN1 SHA3_256 + , SomeHashASN1 SHA3_384 + , SomeHashASN1 SHA3_512 + , SomeHashASN1 RIPEMD160 + ] + +-- | Bytes of the length a DigestInfo has. The ones this stands in for run +-- from 34 bytes for MD5 to 83 for SHA-512, and 'RSA.padSignature' refuses +-- anything that does not leave room for the padding, so the wide generator +-- used elsewhere here would be discarded almost every time. +newtype ArbitraryDigestInfo = ArbitraryDigestInfo ByteString + deriving (Show, Eq) + +instance Arbitrary ArbitraryDigestInfo where + arbitrary = ArbitraryDigestInfo `fmap` arbitraryBSof 0 200 + +-- | The digest a message has under an algorithm picked at runtime, which is +-- what the existential leaves us able to say. +hashOf :: RSA.HashAlgorithmASN1 hashAlg => hashAlg -> ByteString -> Digest hashAlg +hashOf = hashWith + +-- | The operations that take a 'Digest' or a DigestInfo, against the ones +-- that take @Maybe hashAlg@. Two routes to one signature drift apart +-- unless something holds them together, so for every instance and every +-- message these must answer alike -- on a genuine signature, on a tampered +-- one, and on one of the wrong length. +digestOperationTests :: Spec +digestOperationTests = describe "operations taking a digest" $ do + it "covers every HashAlgorithmASN1 instance" $ + length allHashASN1 `shouldBe` 14 + + prop "signs what sign signs" $ \(SomeHashASN1 h) (ArbitraryBS0_2901 m) -> + RSA.signDigest Nothing key (hashOf h m) + === RSA.sign Nothing (Just h) key m + + prop "verifies what verify verifies" $ + \(SomeHashASN1 h) (ArbitraryBS0_2901 m) -> + case RSA.sign Nothing (Just h) key m of + Left err -> counterexample (show err) False + Right s -> + let tampered = B.snoc (B.init s) (B.last s + 1) + longer = B.snoc s 0 + dg = hashOf h m + in conjoin + [ RSA.verifyDigest pub dg s === RSA.verify (Just h) pub m s + , RSA.verifyDigest pub dg tampered + === RSA.verify (Just h) pub m tampered + , RSA.verifyDigest pub dg longer + === RSA.verify (Just h) pub m longer + ] + + prop "accepts its own signature and rejects a changed one" $ + \(SomeHashASN1 h) (ArbitraryBS0_2901 m) -> + let dg = hashOf h m + in case RSA.signDigest Nothing key dg of + Left err -> counterexample (show err) False + Right s -> + let tampered = B.snoc (B.init s) (B.last s + 1) + in property (RSA.verifyDigest pub dg s) + .&&. property (not (RSA.verifyDigest pub dg tampered)) + + prop "blinding leaves the signature where it was" $ + \(SomeHashASN1 h) (ArbitraryBS0_2901 m) testDRG -> + let dg = hashOf h m + blinder = withTestDRG testDRG $ RSA.generateBlinder (RSA.public_n pub) + in RSA.signDigest (Just blinder) key dg === RSA.signDigest Nothing key dg + + prop "signSaferDigest signs what signDigest signs" $ + \(SomeHashASN1 h) (ArbitraryBS0_2901 m) testDRG -> + let dg = hashOf h m + in withTestDRG testDRG (RSA.signSaferDigest key dg) + === RSA.signDigest Nothing key dg + + -- The DigestInfo pair take no HashAlgorithmASN1 constraint, since + -- nothing of theirs hashes or encodes. sign and verify carry it even + -- for this path, and with the variable appearing nowhere else the + -- caller has to name an algorithm it is not using -- which is the + -- annotation below, and the wart these replace. + prop "signs what sign Nothing signs" $ \(ArbitraryDigestInfo di) -> + RSA.signDigestInfo Nothing key di + === RSA.sign Nothing (Nothing :: Maybe SHA256) key di + + prop "verifies what verify Nothing verifies" $ \(ArbitraryDigestInfo di) -> + case RSA.signDigestInfo Nothing key di of + Left err -> counterexample (show err) False + Right s -> + conjoin + [ RSA.verifyDigestInfo pub di s + === RSA.verify (Nothing :: Maybe SHA256) pub di s + , property (RSA.verifyDigestInfo pub di s) + ] + + prop "signSaferDigestInfo signs what signDigestInfo signs" $ + \(ArbitraryDigestInfo di) testDRG -> + withTestDRG testDRG (RSA.signSaferDigestInfo key di) + === RSA.signDigestInfo Nothing key di + where + vector = firstVector vectorsSHA1 + key = vectorToPrivate vector + pub = vectorToPublic vector + +-- | A 'RSA.BlindedKey' carries one blinding pair, each operation returning +-- the key with it squared and a new one drawn every 32 uses; the @Safer@ +-- functions keep one per key for the last few keys. None of it may change +-- an answer, so every answer is held against the unblinded one, over more +-- uses than one pair lasts. +blindedKeyTests :: Spec +blindedKeyTests = describe "BlindedKey" $ do + it "signs what signDigest Nothing signs, across refreshes" $ + threaded 100 $ \bk i -> do + let dg = hashWith SHA256 (B.replicate i 0x61) + (got, bk') <- RSA.signDigestBlinded bk dg + got `shouldBe` RSA.signDigest Nothing key dg + return bk' + + it "signs a DigestInfo as signDigestInfo Nothing does" $ + threaded 40 $ \bk i -> do + let di = B.replicate (20 + i `mod` 10) (fromIntegral i) + (got, bk') <- RSA.signDigestInfoBlinded bk di + got `shouldBe` RSA.signDigestInfo Nothing key di + return bk' + + it "decrypts what it was given" $ + threaded 40 $ \bk i -> do + let m = B.replicate i 0x5a + c <- either (error . show) id <$> RSA.encrypt pub m + (got, bk') <- RSA.decryptBlinded bk c + got `shouldBe` (Right m :: Either RSA.Error ByteString) + return bk' + + it "refuses what decrypt refuses" $ do + let c = B.replicate (RSA.public_size pub + 1) 1 + (got, _) <- RSA.decryptBlinded (RSA.blindedKey key) c + got `shouldBe` (Left RSA.MessageSizeIncorrect :: Either RSA.Error ByteString) + + it "makes PSS signatures that verify" $ + threaded 40 $ \bk i -> do + let m = B.replicate i 0x33 + params = PSS.defaultPSSParams SHA256 + (s, bk') <- PSS.signBlinded params bk m + either (error . show) (PSS.verify params pub m) s `shouldBe` True + return bk' + + it "decrypts OAEP" $ + threaded 40 $ \bk i -> do + let m = B.replicate i 0x77 + params = OAEP.defaultOAEPParams SHA256 + c <- either (error . show) id <$> OAEP.encrypt params pub m + (got, bk') <- OAEP.decryptBlinded params bk c + got `shouldBe` Right m + return bk' + + it "works for a key without its CRT values" $ do + let bare = key {RSA.private_p = 0, RSA.private_q = 0} + go _ 41 = return () + go bk i = do + let dg = hashWith SHA256 (B.replicate i 0x62) + (got, bk') <- RSA.signDigestBlinded bk dg + got `shouldBe` RSA.signDigest Nothing key dg + go bk' (i + 1) + go (RSA.blindedKey bare) (1 :: Int) + + -- The randomness comes from the monad, so a pure generator will do. + prop "runs in a pure generator" $ \testDRG -> + let go bk i acc + | i > (40 :: Int) = return (reverse acc) + | otherwise = do + (s, bk') <- RSA.signDigestBlinded bk (hashWith SHA256 (B.replicate i 0x64)) + go bk' (i + 1) (s : acc) + sigs = withTestDRG testDRG (go (RSA.blindedKey key) 1 []) + in sigs === [RSA.signDigest Nothing key (hashWith SHA256 (B.replicate i 0x64)) | i <- [1 .. 40]] + + it "gives every thread a right answer through the Safer functions" $ do + dones <- forM [1 .. 8 :: Int] $ \t -> do + done <- newEmptyMVar + _ <- forkIO $ do + oks <- forM [1 .. 30 :: Int] $ \i -> do + let dg = hashWith SHA256 (B.pack [fromIntegral t, fromIntegral i]) + got <- RSA.signSaferDigest key dg + return $! got == RSA.signDigest Nothing key dg + putMVar done (and oks) + return done + results <- mapM takeMVar dones + results `shouldBe` replicate 8 True + + -- More keys than the Safer functions remember, interleaved, so that the + -- table both hits and evicts. + it "keeps the Safer functions right across more keys than it remembers" $ do + let keys = key : [snd (withTestDRG (TestDRG (s', 2, 3, 4, 5)) (RSA.generate 64 65537)) | s' <- [1 .. 5]] + forM_ [1 .. 12 :: Int] $ \i -> + forM_ keys $ \k -> do + let dg = hashWith SHA256 (B.replicate i 0x63) + got <- RSA.signSaferDigest k dg + got `shouldBe` RSA.signDigest Nothing k dg + where + vector = firstVector vectorsSHA1 + key = vectorToPrivate vector + pub = vectorToPublic vector + -- run count operations, each handed the key the one before returned + threaded count op = go (RSA.blindedKey key) (1 :: Int) + where + go bk i + | i > count = return () + | otherwise = op bk i >>= \bk' -> go bk' (i + 1) + spec :: Spec spec = do keyGenerationTests @@ -302,5 +614,9 @@ sequence_ $ zipWith doMalleabilityTest [katZero ..] $ filter vectorHasSignature vectorsSHA1 + digestOperationTests + blindedKeyTests unpadTests ciphertextRangeTests + publicOperationTests + crtFaultTests diff --git a/tests/PubKey/RabinSpec.hs b/tests/PubKey/RabinSpec.hs --- a/tests/PubKey/RabinSpec.hs +++ b/tests/PubKey/RabinSpec.hs @@ -161,13 +161,22 @@ (message vector) (BRabin.Signature ((os2ip $ padding vector), (signature vector))) -doModifiedRabinSignTest key i vector = it (show i) (actual `shouldBe` Right (signature vector)) +-- Both routes, and each against the vector rather than against the other: +-- they share everything below the hash, so a test that only made them agree +-- would pass with the shared part broken. +doModifiedRabinSignTest key i vector = describe (show i) $ do + it "sign" $ MRabin.sign key SHA1 (message vector) `shouldBe` expected + it "signDigest" $ + MRabin.signDigest key (hashWith SHA1 (message vector)) `shouldBe` expected where - actual = MRabin.sign key SHA1 (message vector) + expected = Right (signature vector) -doModifiedRabinVerifyTest key i vector = it (show i) (actual `shouldBe` True) - where - actual = MRabin.verify key SHA1 (message vector) (signature vector) +doModifiedRabinVerifyTest key i vector = describe (show i) $ do + it "verify" $ + MRabin.verify key SHA1 (message vector) (signature vector) `shouldBe` True + it "verifyDigest" $ + MRabin.verifyDigest key (hashWith SHA1 (message vector)) (signature vector) + `shouldBe` True doRwEncryptTest key i vector = it (show i) (actual `shouldBe` Right (cipherText vector)) where @@ -182,13 +191,22 @@ where actual = RW.decrypt (OAEP.defaultOAEPParams SHA1) key (cipherText vector) -doRwSignTest key i vector = it (show i) (actual `shouldBe` Right (signature vector)) +-- Both routes, and each against the vector rather than against the other: +-- they share everything below the hash, so a test that only made them agree +-- would pass with the shared part broken. +doRwSignTest key i vector = describe (show i) $ do + it "sign" $ RW.sign key SHA1 (message vector) `shouldBe` expected + it "signDigest" $ + RW.signDigest key (hashWith SHA1 (message vector)) `shouldBe` expected where - actual = RW.sign key SHA1 (message vector) + expected = Right (signature vector) -doRwVerifyTest key i vector = it (show i) (actual `shouldBe` True) - where - actual = RW.verify key SHA1 (message vector) (signature vector) +doRwVerifyTest key i vector = describe (show i) $ do + it "verify" $ + RW.verify key SHA1 (message vector) (signature vector) `shouldBe` True + it "verifyDigest" $ + RW.verifyDigest key (hashWith SHA1 (message vector)) (signature vector) + `shouldBe` True -- | Squaring and the square roots that undo it both work modulo n, so a value -- at or above the modulus behaves exactly like the value it reduces to, and so diff --git a/tests/RuntimeSpec.hs b/tests/RuntimeSpec.hs --- a/tests/RuntimeSpec.hs +++ b/tests/RuntimeSpec.hs @@ -1,7 +1,72 @@ +-- | What 'processorOptions' says about the machine it is running on. +-- +-- A test cannot know what processor it is on, so it cannot check that the +-- answer is right. What it can check is that the answer is well formed: +-- that every name is distinct, that nothing is reported under a name from +-- another architecture, and that the two questions which do not name one +-- agree with the list. The list is printed as well, because a CI log that +-- says what each runner reported is the only way the detection itself gets +-- looked at. module RuntimeSpec (spec) where -import Crypto.System.CPU +import Data.List (nub, sort) import Test.Hspec +import Crypto.System.CPU ( + ProcessorOption (..), + hasAESAcceleration, + hasGHASHAcceleration, + processorOptions, + ) + +-- | Every name this module gives, which is also the set 'Show' has to +-- cover. +x86Options :: [ProcessorOption] +x86Options = + [AESNI, PCLMUL, SSSE3, AVX, AVX2, SHANI, MOVBE, ADX, VAES, VAES512] + +armOptions :: [ProcessorOption] +armOptions = [NEON, ARMAES, ARMPMULL, ARMSHA1, ARMSHA2, ARMSHA512] + +ppcOptions :: [ProcessorOption] +ppcOptions = [PPCAES, PPCVPMSUM] + spec :: Spec -spec = it "CPU" $ putStrLn (show processorOptions) +spec = describe "processorOptions" $ do + it "CPU" $ putStrLn (show processorOptions) + + it "gives every name a number of its own" $ + -- two patterns sharing a number would make one of them unreachable + -- and the other print under the wrong name + length (nub (x86Options ++ armOptions ++ ppcOptions)) + `shouldBe` length (x86Options ++ armOptions ++ ppcOptions) + + it "has a name for every option it names" $ + -- Show falls back to "ProcessorOption n" for what it does not know, + -- which is for values from a newer release, not for these + filter (startsWith "ProcessorOption " . show) (x86Options ++ armOptions ++ ppcOptions) + `shouldBe` [] + + it "reports each option at most once, in order" $ do + processorOptions `shouldBe` sort processorOptions + nub processorOptions `shouldBe` processorOptions + + it "does not mix one architecture's names with another's" $ do + let reported g = any (`elem` g) processorOptions + -- a machine reporting two of these groups is the + -- AArch64-says-AESNI fault this replaced + length (filter reported [x86Options, armOptions, ppcOptions]) + `shouldSatisfy` (<= 1) + + -- Half a check, and which half depends on the machine: where the + -- processor has AES this fails if the answer is broken to False and + -- passes if it is broken to True, and the other way round on a machine + -- without it. Both were tried. + it "answers the architecture-free questions from the same list" $ do + hasAESAcceleration + `shouldBe` any (`elem` processorOptions) [AESNI, ARMAES, PPCAES] + hasGHASHAcceleration + `shouldBe` any (`elem` processorOptions) [PCLMUL, ARMPMULL, PPCVPMSUM] + +startsWith :: String -> String -> Bool +startsWith p s = take (length p) s == p diff --git a/tests/StreamCipher/ChaChaPoly1305Spec.hs b/tests/StreamCipher/ChaChaPoly1305Spec.hs --- a/tests/StreamCipher/ChaChaPoly1305Spec.hs +++ b/tests/StreamCipher/ChaChaPoly1305Spec.hs @@ -2,12 +2,14 @@ module StreamCipher.ChaChaPoly1305Spec where +import qualified Crypto.Cipher.ChaCha.Poly1305 as One import qualified Crypto.Cipher.ChaChaPoly1305 as CP import Crypto.Cipher.Types import Crypto.Error import Imports import MAC.Poly1305Spec () +import Data.Bits (xor) import qualified Data.ByteArray as B (convert) import qualified Data.ByteString as B @@ -89,8 +91,39 @@ CryptoPassed st -> aeadSimpleDecrypt st a5aad a5cipher (AuthTag $ B.convert a5tag) _ -> Nothing +-- | The key is checked once, where it is made, and initializing cannot fail +-- after that. +keyTests :: Spec +keyTests = describe "key" $ do + it "takes thirty-two bytes" $ + passed (CP.key (B.replicate 32 0x41)) `shouldBe` True + it "refuses any other length" $ + [n | n <- [0, 1, 16, 31, 33, 64], passed (CP.key (B.replicate n 0x41))] + `shouldBe` [] + it "says which error" $ + -- Key has no Show, on purpose: it is key material + errorOf (CP.key (B.replicate 31 0x41)) + `shouldBe` Just CryptoError_KeySizeInvalid + it "and the AEAD entry point reports the same thing" $ + errorOf + (CP.aeadChacha20poly1305Init (B.replicate 31 0x41) (B.replicate 12 0x42)) + `shouldBe` Just CryptoError_KeySizeInvalid + it "a key that was taken initializes without an error case" $ do + let k = throwCryptoError (CP.key (B.replicate 32 0x41)) + n = throwCryptoError (CP.nonce12 (B.replicate 12 0x42)) + st = CP.initialize k n + (out, st') = CP.encrypt ("hello" :: B.ByteString) (CP.finalizeAAD st) + B.length out `shouldBe` 5 + B.length (B.convert (CP.finalize st') :: B.ByteString) `shouldBe` 16 + where + passed (CryptoPassed _) = True + passed (CryptoFailed _) = False + errorOf (CryptoFailed e) = Just e + errorOf (CryptoPassed _) = Nothing + spec :: Spec spec = do + keyTests it "V1" runEncrypt it "V1-decrypt" runDecrypt it "V1-extended" runEncryptX @@ -98,11 +131,13 @@ it "nonce increment" runNonceInc it "RFC8439 A5 enc" rfc8439encrypt it "RFC8439 A5 dec" rfc8439decrypt + oneShotTests where runEncrypt = let ini = - throwCryptoError $ - CP.initialize key (throwCryptoError $ CP.nonce8 constant iv) + CP.initialize + (throwCryptoError $ CP.key key) + (throwCryptoError $ CP.nonce8 constant iv) afterAAD = CP.finalizeAAD (CP.appendAAD aad ini) (out, afterEncrypt) = CP.encrypt plaintext afterAAD outtag = CP.finalize afterEncrypt @@ -112,7 +147,9 @@ ] runEncryptX = let ini = - throwCryptoError $ CP.initializeX key (throwCryptoError $ CP.nonce24 ivX) + CP.initializeX + (throwCryptoError $ CP.key key) + (throwCryptoError $ CP.nonce24 ivX) afterAAD = CP.finalizeAAD (CP.appendAAD aad ini) (out, afterEncrypt) = CP.encrypt plaintext afterAAD outtag = CP.finalize afterEncrypt @@ -123,8 +160,9 @@ runDecrypt = let ini = - throwCryptoError $ - CP.initialize key (throwCryptoError $ CP.nonce8 constant iv) + CP.initialize + (throwCryptoError $ CP.key key) + (throwCryptoError $ CP.nonce8 constant iv) afterAAD = CP.finalizeAAD (CP.appendAAD aad ini) (out, afterDecrypt) = CP.decrypt ciphertext afterAAD outtag = CP.finalize afterDecrypt @@ -135,7 +173,9 @@ runDecryptX = let ini = - throwCryptoError $ CP.initializeX key (throwCryptoError $ CP.nonce24 ivX) + CP.initializeX + (throwCryptoError $ CP.key key) + (throwCryptoError $ CP.nonce24 ivX) afterAAD = CP.finalizeAAD (CP.appendAAD aad ini) (out, afterDecrypt) = CP.decrypt ciphertextX afterAAD outtag = CP.finalize afterDecrypt @@ -165,3 +205,66 @@ B.convert . CP.incrementNonce $ n10 ] + +-- | Crypto.Cipher.ChaCha.Poly1305 does a whole message in one call where +-- Crypto.Cipher.ChaChaPoly1305 does it in steps. It has to answer exactly +-- what the steps answer, and what RFC 8439 prints. +oneShotTests :: Spec +oneShotTests = describe "Crypto.Cipher.ChaCha.Poly1305" $ do + it "RFC 8439 2.8.2, twelve-byte nonce" $ + propertyHoldCase + [ eqTest "ciphertext" ciphertext (B.take (B.length ciphertext) sealed) + , eqTest "tag" tag (B.drop (B.length ciphertext) sealed) + ] + it "decrypt undoes encrypt" $ + One.decrypt ctx nonce12 aad sealed 16 `shouldBe` Just plaintext + it "refuses a flipped bit in the ciphertext" $ + One.decrypt ctx nonce12 aad (flipHead sealed) 16 + `shouldBe` (Nothing :: Maybe B.ByteString) + it "refuses a flipped bit in the tag" $ + One.decrypt ctx nonce12 aad (flipLast sealed) 16 + `shouldBe` (Nothing :: Maybe B.ByteString) + it "refuses input shorter than the tag" $ + One.decrypt ctx nonce12 aad (B.replicate 8 0) 16 + `shouldBe` (Nothing :: Maybe B.ByteString) + it "refuses a ciphertext with no authentication tag" $ + One.decrypt ctx nonce12 aad ciphertext 0 + `shouldBe` (Nothing :: Maybe B.ByteString) + it "does not authenticate an altered ciphertext with a zero-length tag" $ + One.decrypt ctx nonce12 aad (flipHead ciphertext) 0 + `shouldBe` (Nothing :: Maybe B.ByteString) + it "refuses a nonce that is not twelve bytes" $ do + One.decrypt ctx (B.replicate 10 0) aad sealed 16 + `shouldBe` (Nothing :: Maybe B.ByteString) + -- eight is the other ChaCha construction, not this AEAD + One.decrypt ctx (B.replicate 8 0) aad sealed 16 + `shouldBe` (Nothing :: Maybe B.ByteString) + it "decryptWithTag hands back the tag encrypt made" $ + case One.decryptWithTag ctx nonce12 aad (B.take (B.length ciphertext) sealed) 16 of + CryptoFailed e -> expectationFailure (show e) + CryptoPassed (body, t) -> + propertyHoldCase + [ eqTest "plaintext" plaintext body + , eqTest "tag" (AuthTag (B.convert tag)) t + ] + it "agrees with the step-at-a-time interface over a different message" $ + let msg = "another message, of a length that is not a multiple of 16" :: B.ByteString + ad = "\x01\x02\x03" :: B.ByteString + ini = CP.initialize (throwCryptoError $ CP.key key) + (throwCryptoError $ CP.nonce12 nonce12) + afterAAD = CP.finalizeAAD (CP.appendAAD ad ini) + (out, afterEnc) = CP.encrypt msg afterAAD + t = CP.finalize afterEnc + one = throwCryptoError (One.encrypt ctx nonce12 ad msg 16) :: B.ByteString + in propertyHoldCase + [ eqTest "ciphertext" out (B.take (B.length out) one) + , eqTest "tag" (B.convert t :: B.ByteString) (B.drop (B.length out) one) + ] + where + ctx = throwCryptoError (One.newContext key) + -- the same nonce the step interface builds from constant and iv + nonce12 = constant `B.append` iv + sealed = throwCryptoError (One.encrypt ctx nonce12 aad plaintext 16) :: B.ByteString + flipHead bs = B.cons (B.head bs `xor` 1) (B.tail bs) + flipLast bs = + B.snoc (B.init bs) (B.last bs `xor` 1) diff --git a/tests/SysDRGSpec.hs b/tests/SysDRGSpec.hs new file mode 100644 --- /dev/null +++ b/tests/SysDRGSpec.hs @@ -0,0 +1,100 @@ +-- | The generator behind 'MonadRandom' for 'IO'. +-- +-- What can be asked from Haskell is narrow. A @forkIO@ thread is not an +-- operating system thread, and this process cannot fork, so the questions +-- that matter most -- does a new operating system thread start its own +-- stream, does a child after @fork@ diverge from its parent -- are in +-- @cbits\/tests\/sysdrg@ instead. +-- +-- What is left is still worth asking: the generator keeps a lock and a +-- thread-local slot, and it is reached through a @safe@ foreign call, so +-- many Haskell threads drawing at once is exactly the shape that deadlocks +-- or hands two of them the same bytes. That needs @-threaded@ and more +-- than one capability, which is why the suite has them. +module SysDRGSpec (spec) where + +import Control.Concurrent ( + forkIO, + getNumCapabilities, + setNumCapabilities, + yield, + ) +import Control.Concurrent.MVar (newEmptyMVar, putMVar, takeMVar) +import Control.Exception (finally) +import Control.Monad (forM, forM_) +import qualified Data.ByteString as BS +import Data.List (group, nub, sort) +import Test.Hspec + +import Crypto.Random (getRandomBytes) + +spec :: Spec +spec = describe "the generator behind MonadRandom IO" $ do + it "runs on more than one capability, or the rest proves little" $ do + n <- getNumCapabilities + n `shouldSatisfy` (> 1) + + mapM_ lengthCase [0, 1, 32, 1000, 4096] + + it "gives every thread something different" $ do + -- plainly, rather than through async, which is not a dependency here + boxes <- forM [1 .. 256 :: Int] $ \_ -> do + box <- newEmptyMVar + _ <- forkIO $ do + b <- getRandomBytes 32 :: IO BS.ByteString + putMVar box b + return box + bss <- mapM takeMVar boxes + length (nub bss) `shouldBe` 256 + + it "keeps the streams apart while setNumCapabilities changes underneath" $ do + -- The state is held against the operating system thread rather than + -- against the capability, which is what makes this safe: a forkIO + -- thread moves between capabilities, and a capability is served by + -- different worker threads over its life, so state held against one + -- would be shared by threads running at the same time. Raising the + -- count makes the runtime create worker threads, each of which has + -- to start its own stream; lowering it disables capabilities under + -- threads that are drawing. + n0 <- getNumCapabilities + let flips = concat (replicate 3 [1, 2, min 8 (n0 * 2), n0]) + threads = 64 :: Int + draws = 16 :: Int + bss <- + ( do + flipped <- newEmptyMVar + _ <- forkIO $ do + forM_ flips $ \n -> setNumCapabilities n >> yield + putMVar flipped () + boxes <- forM [1 .. threads] $ \_ -> do + box <- newEmptyMVar + _ <- forkIO $ do + bs <- forM [1 .. draws] $ \_ -> do + b <- getRandomBytes 32 :: IO BS.ByteString + yield + return b + putMVar box bs + return box + bss <- concat <$> mapM takeMVar boxes + takeMVar flipped + return bss + ) + `finally` setNumCapabilities n0 + -- sorted and grouped rather than nub, which is quadratic and this + -- list is long enough for that to show + length bss `shouldBe` threads * draws + length (group (sort bss)) `shouldBe` threads * draws + + it "does not repeat across a reseed" $ do + -- the per-thread generator reseeds after a mebibyte + first <- getRandomBytes 32 :: IO BS.ByteString + forM_ [1 .. 300 :: Int] $ \_ -> + (getRandomBytes 4096 :: IO BS.ByteString) >>= \b -> b `seq` return () + second <- getRandomBytes 32 :: IO BS.ByteString + second `shouldNotBe` first + +lengthCase :: Int -> Spec +lengthCase n = + it ("gives back the " ++ show n ++ " bytes asked for") $ do + b <- getRandomBytes n :: IO BS.ByteString + BS.length b `shouldBe` n diff --git a/tests/forkprocess/ForkProcess.hs b/tests/forkprocess/ForkProcess.hs new file mode 100644 --- /dev/null +++ b/tests/forkprocess/ForkProcess.hs @@ -0,0 +1,81 @@ +-- | Does the generator behind 'MonadRandom' notice a fork made from +-- Haskell? +-- +-- @cbits\/tests\/sysdrg@ asks the same question of @fork(2)@ called from C, +-- in a process with no runtime system in it at all. That leaves the case +-- anyone actually meets untested: 'forkProcess', with the runtime's own +-- threads about. The handler is registered with @pthread_atfork@, which +-- libc runs for every @fork(2)@ whoever calls it, so it should reach this +-- too -- should, which is why this is a test and not a comment. +-- +-- It cannot live in the main suite. That one runs with @-N2@, where GHC +-- says 'forkProcess' is not supported; this needs its own runtime options, +-- and is built twice, once threaded with one capability and once not +-- threaded at all. Both are configurations GHC supports 'forkProcess' in. +-- +-- The check is that parent and child disagree. Without fork detection the +-- child carries on the parent's stream, so the next block each of them +-- draws is the same block -- they would agree exactly, which is the fault. +module Main (main) where + +import Control.Concurrent (getNumCapabilities, rtsSupportsBoundThreads) +import Control.Monad (when) +import qualified Data.ByteString as B +import System.Exit (exitFailure) +import System.IO (hClose, hFlush, hPutStrLn, stderr, stdout) +import System.Posix.IO (closeFd, createPipe, fdToHandle) +import System.Posix.Process (ProcessStatus (..), forkProcess, getProcessStatus) + +import Crypto.Random (getRandomBytes) + +draw :: IO B.ByteString +draw = getRandomBytes 32 + +main :: IO () +main = do + caps <- getNumCapabilities + putStrLn $ + "threaded: " + ++ show rtsSupportsBoundThreads + ++ ", capabilities: " + ++ show caps + -- GHC supports forkProcess with -threaded only while one capability is + -- in use. Saying so here means a change to the runtime options shows + -- up as a failure rather than as a test that quietly means nothing. + when (rtsSupportsBoundThreads && caps /= 1) $ + die "this test needs one capability when threaded" + + -- Before the fork, or the child inherits whatever is still in the + -- buffer and writes it out again when it exits. + hFlush stdout + + -- Draw once first, so that both sides inherit a generator that has been + -- seeded. A child of an unseeded one would seed itself for the first + -- time and differ for that reason instead of this one. + _ <- draw + + (readEnd, writeEnd) <- createPipe + pid <- forkProcess $ do + closeFd readEnd + b <- draw + h <- fdToHandle writeEnd + B.hPut h b + hClose h + closeFd writeEnd + hr <- fdToHandle readEnd + fromChild <- B.hGet hr 32 + hClose hr + status <- getProcessStatus True False pid + + fromParent <- draw + + case status of + Just (Exited _) -> return () + other -> die ("the child did not exit cleanly: " ++ show other) + when (B.length fromChild /= 32) $ + die ("the child sent " ++ show (B.length fromChild) ++ " bytes, not 32") + when (fromChild == fromParent) $ + die "parent and child drew the same bytes: the fork went unnoticed" + putStrLn "parent and child drew different bytes" + where + die msg = hPutStrLn stderr ("FAIL: " ++ msg) >> exitFailure diff --git a/tests/tutorial/extract.awk b/tests/tutorial/extract.awk new file mode 100644 --- /dev/null +++ b/tests/tutorial/extract.awk @@ -0,0 +1,53 @@ +# Pull every code block out of a Haddock module and write each one as a +# compilable Haskell module. +# +# A block is a maximal run of lines beginning "-- >", which is what Haddock +# renders as a code sample. The run is ended by any line that is not one, +# so two samples separated by a line of prose are two modules and cannot +# refer to each other -- which is the point: each sample has to stand on its +# own, because that is how a reader will copy it. +# +# The name comes from the "-- $section" marker the block sits under, so a +# compiler message names the section of the tutorial that is wrong rather +# than a number. The module header goes after any LANGUAGE pragmas, since +# those have to come first. +# +# Writes one file per block into the directory given as -v out=, and prints +# each module name on stdout. + +/^-- \$[A-Za-z_][A-Za-z0-9_]*$/ { + section = substr($0, 5) + nth = 0 + next +} + +/^-- >/ { + if (!inblock) { + inblock = 1 + nlines = 0 + nth++ + mod = "Example_" section "_" nth + file = out "/" mod ".hs" + } + # "-- > foo" carries a space that is not part of the code; "-- >" alone + # is a blank line inside the block. + line = (length($0) > 5) ? substr($0, 6) : "" + lines[++nlines] = line + next +} + +{ if (inblock) flush() } + +END { if (inblock) flush() } + +function flush( i, k) { + # Pragmas, then the header, then the rest. + k = 1 + while (k <= nlines && (lines[k] ~ /^\{-#/ || lines[k] == "")) k++ + for (i = 1; i < k; i++) print lines[i] > file + print "module " mod " where" > file + for (i = k; i <= nlines; i++) print lines[i] > file + close(file) + print mod + inblock = 0 +} diff --git a/tests/tutorial/run.sh b/tests/tutorial/run.sh new file mode 100644 --- /dev/null +++ b/tests/tutorial/run.sh @@ -0,0 +1,126 @@ +#!/bin/sh +# Does the tutorial still compile? +# +# Crypto/Tutorial.hs is the one place in the package whose code the compiler +# never sees: every example in it is a Haddock block, so an API change +# silently leaves it wrong and the next reader copies something that does +# not build. That has happened -- taking a checked key in Poly1305 made the +# tutorial's crypto_box stop type-checking, and it was noticed by reading +# the module rather than by anything here. +# +# So the blocks are pulled out and type-checked. Each one becomes a module +# of its own, because each one is a thing a reader copies whole; a block +# that needs a definition from the block above it will fail here, which is +# the right answer. +# +# Checked against this tree rather than against whatever crypton is +# installed: cabal repl has the project's library as its home package, so a +# tutorial written for an API this working tree has not got cannot pass by +# finding the API in a released version on the machine. -fno-code because +# nothing is run -- the question is only whether it compiles. +# +# -Wall as well, and a warning in a tutorial counts as a failure: an unused +# import is three words a reader will paste into their own file. Only the +# extracted modules are held to it, not the library loaded beside them. +# +# $GHC names a compiler other than the one on the path, and $BUILDDIR a +# build directory to keep that compiler's products out of the usual one. +# CI has one compiler per job and sets neither; locally they are how an +# example is asked whether it also builds on the oldest GHC the package +# supports, which is the reader this is most likely to have failed. +# +# The first module is wrong on purpose and has to be reported, or the run +# proves nothing: a repl that failed to start, a ghci whose message format +# changed, or an extractor that produced no modules would otherwise all look +# like a tutorial that compiles. +set -eu + +cd "$(dirname "$0")/../.." +tutorial=Crypto/Tutorial.hs + +work=$(mktemp -d) +trap 'rm -rf "$work"' EXIT INT TERM + +modules=$(awk -v out="$work" -f tests/tutorial/extract.awk "$tutorial") +count=$(printf '%s\n' "$modules" | grep -c . || true) +if [ "$count" -eq 0 ]; then + echo "FAIL no code blocks found in $tutorial" + exit 1 +fi +echo "$count code blocks in $tutorial" + +# The deliberate one. hashWith returns a Digest, and has since the module +# was written, so this is the shape of every break this harness is for: the +# tutorial says something the library no longer agrees with. +cat > "$work/Calibration.hs" <<'HS' +module Calibration where +import Crypto.Hash (SHA1 (..), hashWith) +thisIsNotADigest :: Int +thisIsNotADigest = hashWith SHA1 "the harness has to report this" +HS + +{ + echo ':!echo @@ Calibration' + echo ":load $work/Calibration.hs" + for m in $modules; do + echo ":!echo @@ $m" + echo ":load $work/$m.hs" + done + echo ':quit' +} > "$work/script" + +# Not cabal's -v0: that reaches ghci too and takes the "Ok, N modules +# loaded." line with it, which is what is read below. +cabal repl crypton ${GHC:+-w "$GHC"} ${BUILDDIR:+--builddir="$BUILDDIR"} \ + --repl-options=-i"$work" --repl-options=-fno-code \ + --repl-options=-Wall --repl-options=-Wno-missing-home-modules \ + < "$work/script" > "$work/log" 2>&1 || true + +awk -v expected="$count" ' + # ghci writes its prompt before the echo, so the marker is at the end + # of the line rather than the start of it. + /@@ [A-Za-z_]/ { mod = $NF; verdict[mod] = "no answer"; order[++n] = mod; next } + mod == "" { next } + /^Ok, [0-9]+ modules? loaded\.$/ { verdict[mod] = "ok"; next } + /^Failed, [0-9]+ modules? loaded\.$/ { verdict[mod] = "failed"; next } + # Only the extracted modules are held to -Wall; the library is loaded + # beside them and is not what this is asking about. + /Example_[A-Za-z0-9_]*\.hs:[0-9]+:[0-9]+: warning:/ { warned[mod] = 1 } + # Everything ghci said about a module before it gave its verdict, minus + # the progress lines, which are the bulk of it and say nothing. + /^(ghci> )?\[ *[0-9]+ of [0-9]+\] Compiling/ { next } + { if (verdict[mod] == "no answer" && $0 != "") detail[mod] = detail[mod] $0 "\n" } + END { + bad = 0 + for (i = 1; i <= n; i++) { + m = order[i] + want = (m == "Calibration") ? "failed" : "ok" + got = verdict[m] + if (got == want && want == "ok" && warned[m]) got = "warned" + if (got == want) { + printf "ok %s%s\n", m, (m == "Calibration" ? " (reported, as it must be)" : "") + } else { + bad++ + printf "FAIL %s: expected %s, got %s\n", m, want, got + printf "%s", detail[m] + } + } + # Nothing answered, or not everything did: the run itself did not + # happen the way this script assumes, so the log is worth seeing. + if (n != expected + 1) { + printf "FAIL %d modules answered, %d expected\n", n - 1, expected + exit 2 + } + exit (bad > 0) + } +' "$work/log" || { + status=$? + if [ "$status" -eq 2 ]; then + echo + echo "--- the last of the log ---" + tail -60 "$work/log" + fi + exit 1 +} + +echo "every example in $tutorial compiles"